2024-08-19T14:00:04.690 INFO:root:teuthology version: 1.2.0 2024-08-19T14:00:04.711 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-08-19T14:00:04.805 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-08-18_21:00:03-rados-squid-distro-default-smithi/7859709 branch: squid description: rados/thrash-old-clients/{0-distro$/{centos_9.stream} 0-size-min-size-overrides/3-size-2-min-size 1-install/quincy backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/fastclose rados thrashers/morepggrow thrashosds-health workloads/radosbench} email: ceph-qa@ceph.com first_in_suite: false job_id: '7859709' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-08-18_21:00:03-rados-squid-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 30 - volumes: count: 4 size: 10 - volumes: size: 50 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: client.2: debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 1 ms inject socket failures: 2500 ms tcp read timeout: 5 osd_pool_default_min_size: 2 osd_pool_default_size: 3 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon scrub interval: 300 osd: bluestore zero block detection: true debug ms: 1 debug osd: 20 filestore queue throttle high multiple: 2 filestore queue throttle max multiple: 10 journal throttle high multiple: 2 journal throttle max multiple: 10 osd backoff on peering: true osd blocked scrub grace period: 3600 osd debug verify cached snaps: true osd debug verify missing on start: true osd max backfills: 9 osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd op queue: debug_random osd op queue cut off: debug_random osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(MON_DOWN\) - \(OSD_SLOW_PING_TIME - but it is still running - objects unfound and apparently lost - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - mons down - mon down - out of quorum - noscrub - nodeep-scrub sha1: de09dd018de4baddfa6ca4d43bc8fb784cf75833 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: de09dd018de4baddfa6ca4d43bc8fb784cf75833 selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 thrashosds: min_in: 4 workunit: branch: squid sha1: 78215f5a1cf5eaf321d506a66539942e12851f34 owner: scheduled_teuthology@teuthology priority: 700 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - - mon.c - osd.8 - osd.9 - osd.10 - osd.11 - client.2 seed: 3241 sha1: de09dd018de4baddfa6ca4d43bc8fb784cf75833 sleep_before_teardown: 0 subset: 18710/100000 suite: rados suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_78215f5a1cf5eaf321d506a66539942e12851f34/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 78215f5a1cf5eaf321d506a66539942e12851f34 targets: smithi031.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKtAJQjqSS+rnoiFfzSObTuqyhiV8jmeO/zkqfomlOTLop6eEUBGbxT4ojHKvUPOKyxwCZ6h9sdt0bQbt2keegI= smithi032.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP/jaLwp995VURmbqhwQoCGvk+96QX57IHWFcnx3RNwwOS69Ps5aMOEs/BOBRkIsvdrAIqNxa1uly0gXkWolrPA= smithi045.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBSwbd9s+O5vUfGXhVhBXZxNGOlTuA0myxQQl/7g0PWTVxV77+fi7xD8kbaq8XhjBOvTXHeHEjgbyOTHn6b2cRM= tasks: - install: branch: quincy exclude_packages: - ceph-mgr-dashboard - ceph-mgr-diskprediction-local - ceph-mgr-rook - ceph-mgr-cephadm - ceph-base-debuginfo - ceph-common-debuginfo - ceph-immutable-object-cache-debuginfo - ceph-radosgw-debuginfo - ceph-test-debuginfo - ceph-base-debuginfo - ceph-mgr-debuginfo - ceph-mds-debuginfo - ceph-mon-debuginfo - ceph-osd-debuginfo - ceph-fuse-debuginfo - librados-devel-debuginfo - libcephfs2-debuginfo - librados2-debuginfo - librbd1-debuginfo - python3-cephfs-debuginfo - python3-rados-debuginfo - python3-rbd-debuginfo - python3-rgw-debuginfo - rbd-fuse-debuginfo - rbd-mirror-debuginfo - rbd-nbd-debuginfo - ceph-volume - cephadm: conf: mon: auth allow insecure global id reclaim: true - exec: mon.a: - while ! ceph balancer status ; do sleep 1 ; done - ceph balancer mode crush-compat - ceph balancer on - thrashosds: chance_pgnum_grow: 3 chance_pgpnum_fix: 1 timeout: 1200 - full_sequential: - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 67db276914d29505c9707d04f3a3b8fb9d8aee4b timestamp: 2024-08-18_21:00:03 tube: smithi user: teuthology verbose: false verify_ceph_hash: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-08-19T14:00:04.806 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_78215f5a1cf5eaf321d506a66539942e12851f34/qa; will attempt to use it 2024-08-19T14:00:04.806 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_78215f5a1cf5eaf321d506a66539942e12851f34/qa/tasks 2024-08-19T14:00:04.806 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-08-19T14:00:04.808 INFO:teuthology.task.internal:Saving configuration 2024-08-19T14:00:04.824 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-08-19T14:00:04.831 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-08-19T14:00:04.861 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi031.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-08-18_21:00:03-rados-squid-distro-default-smithi/7859709', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-08-19 13:55:23.924615', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKtAJQjqSS+rnoiFfzSObTuqyhiV8jmeO/zkqfomlOTLop6eEUBGbxT4ojHKvUPOKyxwCZ6h9sdt0bQbt2keegI='} 2024-08-19T14:00:04.889 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi032.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-08-18_21:00:03-rados-squid-distro-default-smithi/7859709', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-08-19 13:55:23.925832', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP/jaLwp995VURmbqhwQoCGvk+96QX57IHWFcnx3RNwwOS69Ps5aMOEs/BOBRkIsvdrAIqNxa1uly0gXkWolrPA='} 2024-08-19T14:00:04.917 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi045.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-08-18_21:00:03-rados-squid-distro-default-smithi/7859709', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-08-19 13:55:23.926808', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBSwbd9s+O5vUfGXhVhBXZxNGOlTuA0myxQQl/7g0PWTVxV77+fi7xD8kbaq8XhjBOvTXHeHEjgbyOTHn6b2cRM='} 2024-08-19T14:00:04.918 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-08-19T14:00:04.924 INFO:teuthology.task.internal:roles: ubuntu@smithi031.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2024-08-19T14:00:04.925 INFO:teuthology.task.internal:roles: ubuntu@smithi032.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1'] 2024-08-19T14:00:04.925 INFO:teuthology.task.internal:roles: ubuntu@smithi045.front.sepia.ceph.com - ['mon.c', 'osd.8', 'osd.9', 'osd.10', 'osd.11', 'client.2'] 2024-08-19T14:00:04.925 INFO:teuthology.run_tasks:Running task console_log... 2024-08-19T14:00:05.025 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f7fbf987ac0>, signals=[15]) 2024-08-19T14:00:05.025 INFO:teuthology.run_tasks:Running task internal.connect... 2024-08-19T14:00:05.031 INFO:teuthology.task.internal:Opening connections... 2024-08-19T14:00:05.031 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi031.front.sepia.ceph.com 2024-08-19T14:00:05.033 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-19T14:00:05.124 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi032.front.sepia.ceph.com 2024-08-19T14:00:05.125 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-19T14:00:05.210 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi045.front.sepia.ceph.com 2024-08-19T14:00:05.211 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-19T14:00:05.293 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-08-19T14:00:05.300 DEBUG:teuthology.orchestra.run.smithi031:> uname -m 2024-08-19T14:00:05.319 INFO:teuthology.orchestra.run.smithi031.stdout:x86_64 2024-08-19T14:00:05.319 DEBUG:teuthology.orchestra.run.smithi031:> cat /etc/os-release 2024-08-19T14:00:05.377 INFO:teuthology.orchestra.run.smithi031.stdout:NAME="CentOS Stream" 2024-08-19T14:00:05.377 INFO:teuthology.orchestra.run.smithi031.stdout:VERSION="9" 2024-08-19T14:00:05.377 INFO:teuthology.orchestra.run.smithi031.stdout:ID="centos" 2024-08-19T14:00:05.377 INFO:teuthology.orchestra.run.smithi031.stdout:ID_LIKE="rhel fedora" 2024-08-19T14:00:05.377 INFO:teuthology.orchestra.run.smithi031.stdout:VERSION_ID="9" 2024-08-19T14:00:05.377 INFO:teuthology.orchestra.run.smithi031.stdout:PLATFORM_ID="platform:el9" 2024-08-19T14:00:05.377 INFO:teuthology.orchestra.run.smithi031.stdout:PRETTY_NAME="CentOS Stream 9" 2024-08-19T14:00:05.377 INFO:teuthology.orchestra.run.smithi031.stdout:ANSI_COLOR="0;31" 2024-08-19T14:00:05.378 INFO:teuthology.orchestra.run.smithi031.stdout:LOGO="fedora-logo-icon" 2024-08-19T14:00:05.378 INFO:teuthology.orchestra.run.smithi031.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-08-19T14:00:05.378 INFO:teuthology.orchestra.run.smithi031.stdout:HOME_URL="https://centos.org/" 2024-08-19T14:00:05.378 INFO:teuthology.orchestra.run.smithi031.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-08-19T14:00:05.378 INFO:teuthology.orchestra.run.smithi031.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-08-19T14:00:05.378 INFO:teuthology.orchestra.run.smithi031.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-08-19T14:00:05.379 INFO:teuthology.lock.ops:Updating smithi031.front.sepia.ceph.com on lock server 2024-08-19T14:00:05.409 DEBUG:teuthology.orchestra.run.smithi032:> uname -m 2024-08-19T14:00:05.429 INFO:teuthology.orchestra.run.smithi032.stdout:x86_64 2024-08-19T14:00:05.429 DEBUG:teuthology.orchestra.run.smithi032:> cat /etc/os-release 2024-08-19T14:00:05.487 INFO:teuthology.orchestra.run.smithi032.stdout:NAME="CentOS Stream" 2024-08-19T14:00:05.487 INFO:teuthology.orchestra.run.smithi032.stdout:VERSION="9" 2024-08-19T14:00:05.487 INFO:teuthology.orchestra.run.smithi032.stdout:ID="centos" 2024-08-19T14:00:05.487 INFO:teuthology.orchestra.run.smithi032.stdout:ID_LIKE="rhel fedora" 2024-08-19T14:00:05.487 INFO:teuthology.orchestra.run.smithi032.stdout:VERSION_ID="9" 2024-08-19T14:00:05.487 INFO:teuthology.orchestra.run.smithi032.stdout:PLATFORM_ID="platform:el9" 2024-08-19T14:00:05.487 INFO:teuthology.orchestra.run.smithi032.stdout:PRETTY_NAME="CentOS Stream 9" 2024-08-19T14:00:05.487 INFO:teuthology.orchestra.run.smithi032.stdout:ANSI_COLOR="0;31" 2024-08-19T14:00:05.487 INFO:teuthology.orchestra.run.smithi032.stdout:LOGO="fedora-logo-icon" 2024-08-19T14:00:05.487 INFO:teuthology.orchestra.run.smithi032.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-08-19T14:00:05.487 INFO:teuthology.orchestra.run.smithi032.stdout:HOME_URL="https://centos.org/" 2024-08-19T14:00:05.488 INFO:teuthology.orchestra.run.smithi032.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-08-19T14:00:05.488 INFO:teuthology.orchestra.run.smithi032.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-08-19T14:00:05.488 INFO:teuthology.orchestra.run.smithi032.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-08-19T14:00:05.488 INFO:teuthology.lock.ops:Updating smithi032.front.sepia.ceph.com on lock server 2024-08-19T14:00:05.518 DEBUG:teuthology.orchestra.run.smithi045:> uname -m 2024-08-19T14:00:05.536 INFO:teuthology.orchestra.run.smithi045.stdout:x86_64 2024-08-19T14:00:05.537 DEBUG:teuthology.orchestra.run.smithi045:> cat /etc/os-release 2024-08-19T14:00:05.595 INFO:teuthology.orchestra.run.smithi045.stdout:NAME="CentOS Stream" 2024-08-19T14:00:05.595 INFO:teuthology.orchestra.run.smithi045.stdout:VERSION="9" 2024-08-19T14:00:05.595 INFO:teuthology.orchestra.run.smithi045.stdout:ID="centos" 2024-08-19T14:00:05.595 INFO:teuthology.orchestra.run.smithi045.stdout:ID_LIKE="rhel fedora" 2024-08-19T14:00:05.595 INFO:teuthology.orchestra.run.smithi045.stdout:VERSION_ID="9" 2024-08-19T14:00:05.595 INFO:teuthology.orchestra.run.smithi045.stdout:PLATFORM_ID="platform:el9" 2024-08-19T14:00:05.595 INFO:teuthology.orchestra.run.smithi045.stdout:PRETTY_NAME="CentOS Stream 9" 2024-08-19T14:00:05.595 INFO:teuthology.orchestra.run.smithi045.stdout:ANSI_COLOR="0;31" 2024-08-19T14:00:05.595 INFO:teuthology.orchestra.run.smithi045.stdout:LOGO="fedora-logo-icon" 2024-08-19T14:00:05.596 INFO:teuthology.orchestra.run.smithi045.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-08-19T14:00:05.596 INFO:teuthology.orchestra.run.smithi045.stdout:HOME_URL="https://centos.org/" 2024-08-19T14:00:05.596 INFO:teuthology.orchestra.run.smithi045.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-08-19T14:00:05.596 INFO:teuthology.orchestra.run.smithi045.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-08-19T14:00:05.596 INFO:teuthology.orchestra.run.smithi045.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-08-19T14:00:05.596 INFO:teuthology.lock.ops:Updating smithi045.front.sepia.ceph.com on lock server 2024-08-19T14:00:05.626 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-08-19T14:00:05.636 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-08-19T14:00:05.642 INFO:teuthology.task.internal:Checking for old test directory... 2024-08-19T14:00:05.642 DEBUG:teuthology.orchestra.run.smithi031:> test '!' -e /home/ubuntu/cephtest 2024-08-19T14:00:05.645 DEBUG:teuthology.orchestra.run.smithi032:> test '!' -e /home/ubuntu/cephtest 2024-08-19T14:00:05.647 DEBUG:teuthology.orchestra.run.smithi045:> test '!' -e /home/ubuntu/cephtest 2024-08-19T14:00:05.664 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-08-19T14:00:05.669 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-08-19T14:00:05.670 DEBUG:teuthology.orchestra.run.smithi031:> test -z $(ls -A /var/lib/ceph) 2024-08-19T14:00:05.702 DEBUG:teuthology.orchestra.run.smithi032:> test -z $(ls -A /var/lib/ceph) 2024-08-19T14:00:05.705 DEBUG:teuthology.orchestra.run.smithi045:> test -z $(ls -A /var/lib/ceph) 2024-08-19T14:00:05.740 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-08-19T14:00:05.843 INFO:teuthology.run_tasks:Running task kernel... 2024-08-19T14:00:05.859 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-08-19T14:00:05.859 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}, 'mon.c': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-08-19T14:00:05.859 DEBUG:teuthology.orchestra.run.smithi031:> test -f /run/.containerenv -o -f /.dockerenv 2024-08-19T14:00:05.860 DEBUG:teuthology.orchestra.run.smithi032:> test -f /run/.containerenv -o -f /.dockerenv 2024-08-19T14:00:05.860 DEBUG:teuthology.orchestra.run.smithi045:> test -f /run/.containerenv -o -f /.dockerenv 2024-08-19T14:00:05.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:00:05.878 DEBUG:teuthology.orchestra.run.smithi032:> uname -r 2024-08-19T14:00:05.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:00:05.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:00:05.879 DEBUG:teuthology.orchestra.run.smithi045:> uname -r 2024-08-19T14:00:05.880 DEBUG:teuthology.orchestra.run.smithi031:> uname -r 2024-08-19T14:00:05.936 INFO:teuthology.orchestra.run.smithi032.stdout:5.14.0-480.el9.x86_64 2024-08-19T14:00:05.936 INFO:teuthology.task.kernel:Running kernel on smithi032: 5.14.0-480.el9.x86_64 2024-08-19T14:00:05.936 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum install -y kernel 2024-08-19T14:00:05.937 INFO:teuthology.orchestra.run.smithi031.stdout:5.14.0-480.el9.x86_64 2024-08-19T14:00:05.937 INFO:teuthology.task.kernel:Running kernel on smithi031: 5.14.0-480.el9.x86_64 2024-08-19T14:00:05.938 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum install -y kernel 2024-08-19T14:00:05.939 INFO:teuthology.orchestra.run.smithi045.stdout:5.14.0-480.el9.x86_64 2024-08-19T14:00:05.939 INFO:teuthology.task.kernel:Running kernel on smithi045: 5.14.0-480.el9.x86_64 2024-08-19T14:00:05.939 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum install -y kernel 2024-08-19T14:00:08.183 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.2 MB 00:00 2024-08-19T14:00:08.212 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 8.2 MB 00:00 2024-08-19T14:00:08.262 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.2 MB 00:00 2024-08-19T14:00:11.200 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 9 - AppStream 11 MB/s | 20 MB 00:01 2024-08-19T14:00:13.862 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - AppStream 4.5 MB/s | 20 MB 00:04 2024-08-19T14:00:15.909 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 9 - CRB 7.7 MB/s | 6.6 MB 00:00 2024-08-19T14:00:18.210 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 9 - Extras packages 18 kB/s | 18 kB 00:01 2024-08-19T14:00:18.388 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.6 MB 00:00 2024-08-19T14:00:19.464 INFO:teuthology.orchestra.run.smithi032.stdout:Extra Packages for Enterprise Linux 20 MB/s | 22 MB 00:01 2024-08-19T14:00:19.930 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - Extras packages 73 kB/s | 18 kB 00:00 2024-08-19T14:00:20.934 INFO:teuthology.orchestra.run.smithi045.stdout:Extra Packages for Enterprise Linux 25 MB/s | 22 MB 00:00 2024-08-19T14:00:24.978 INFO:teuthology.orchestra.run.smithi032.stdout:lab-extras 38 kB/s | 1.7 kB 00:00 2024-08-19T14:00:26.352 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - AppStream 1.2 MB/s | 20 MB 00:16 2024-08-19T14:00:26.615 INFO:teuthology.orchestra.run.smithi045.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-08-19T14:00:26.959 INFO:teuthology.orchestra.run.smithi032.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-08-19T14:00:26.959 INFO:teuthology.orchestra.run.smithi032.stdout:Package kernel-5.14.0-480.el9.x86_64 is already installed. 2024-08-19T14:00:27.024 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-08-19T14:00:27.025 INFO:teuthology.orchestra.run.smithi032.stdout:Nothing to do. 2024-08-19T14:00:27.025 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-08-19T14:00:27.147 DEBUG:teuthology.orchestra.run.smithi032:> echo no | sudo yum reinstall kernel || true 2024-08-19T14:00:27.680 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:00:03 ago on Mon 19 Aug 2024 02:00:24 PM UTC. 2024-08-19T14:00:27.835 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-08-19T14:00:27.859 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-08-19T14:00:27.859 INFO:teuthology.orchestra.run.smithi032.stdout: Package Architecture Version Repository Size 2024-08-19T14:00:27.859 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-08-19T14:00:27.859 INFO:teuthology.orchestra.run.smithi032.stdout:Reinstalling: 2024-08-19T14:00:27.859 INFO:teuthology.orchestra.run.smithi032.stdout: kernel x86_64 5.14.0-480.el9 baseos 1.5 M 2024-08-19T14:00:27.859 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-19T14:00:27.859 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-08-19T14:00:27.859 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-08-19T14:00:27.859 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-19T14:00:27.859 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 1.5 M 2024-08-19T14:00:27.860 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 0 2024-08-19T14:00:27.860 INFO:teuthology.orchestra.run.smithi032.stderr:Operation aborted. 2024-08-19T14:00:27.912 INFO:teuthology.orchestra.run.smithi032.stdout:Is this ok [y/N]: 2024-08-19T14:00:27.912 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum reinstall -y kernel || true 2024-08-19T14:00:28.440 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:00:04 ago on Mon 19 Aug 2024 02:00:24 PM UTC. 2024-08-19T14:00:28.568 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-08-19T14:00:28.569 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-08-19T14:00:28.569 INFO:teuthology.orchestra.run.smithi032.stdout: Package Architecture Version Repository Size 2024-08-19T14:00:28.569 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-08-19T14:00:28.569 INFO:teuthology.orchestra.run.smithi032.stdout:Reinstalling: 2024-08-19T14:00:28.569 INFO:teuthology.orchestra.run.smithi032.stdout: kernel x86_64 5.14.0-480.el9 baseos 1.5 M 2024-08-19T14:00:28.569 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-19T14:00:28.569 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-08-19T14:00:28.569 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-08-19T14:00:28.569 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-19T14:00:28.570 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 1.5 M 2024-08-19T14:00:28.570 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 0 2024-08-19T14:00:28.570 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-08-19T14:00:28.638 INFO:teuthology.orchestra.run.smithi045.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-08-19T14:00:28.639 INFO:teuthology.orchestra.run.smithi045.stdout:Package kernel-5.14.0-480.el9.x86_64 is already installed. 2024-08-19T14:00:28.722 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-08-19T14:00:28.723 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2024-08-19T14:00:28.723 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-08-19T14:00:28.828 DEBUG:teuthology.orchestra.run.smithi045:> echo no | sudo yum reinstall kernel || true 2024-08-19T14:00:29.105 INFO:teuthology.orchestra.run.smithi032.stdout:kernel-5.14.0-480.el9.x86_64.rpm 3.5 MB/s | 1.5 MB 00:00 2024-08-19T14:00:29.105 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-08-19T14:00:29.105 INFO:teuthology.orchestra.run.smithi032.stdout:Total 2.8 MB/s | 1.5 MB 00:00 2024-08-19T14:00:29.142 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-08-19T14:00:29.359 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:03 ago on Mon 19 Aug 2024 02:00:26 PM UTC. 2024-08-19T14:00:29.390 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-08-19T14:00:29.391 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-08-19T14:00:29.416 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-08-19T14:00:29.417 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-08-19T14:00:29.497 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-08-19T14:00:29.503 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-08-19T14:00:29.503 INFO:teuthology.orchestra.run.smithi045.stdout: Package Architecture Version Repository Size 2024-08-19T14:00:29.503 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-08-19T14:00:29.503 INFO:teuthology.orchestra.run.smithi045.stdout:Reinstalling: 2024-08-19T14:00:29.503 INFO:teuthology.orchestra.run.smithi045.stdout: kernel x86_64 5.14.0-480.el9 baseos 1.5 M 2024-08-19T14:00:29.503 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-08-19T14:00:29.503 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-08-19T14:00:29.503 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-08-19T14:00:29.503 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-08-19T14:00:29.504 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 1.5 M 2024-08-19T14:00:29.504 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 0 2024-08-19T14:00:29.504 INFO:teuthology.orchestra.run.smithi045.stderr:Operation aborted. 2024-08-19T14:00:29.560 INFO:teuthology.orchestra.run.smithi045.stdout:Is this ok [y/N]: 2024-08-19T14:00:29.560 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum reinstall -y kernel || true 2024-08-19T14:00:29.584 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-08-19T14:00:29.665 INFO:teuthology.orchestra.run.smithi032.stdout: Reinstalling : kernel-5.14.0-480.el9.x86_64 1/2 2024-08-19T14:00:29.665 INFO:teuthology.orchestra.run.smithi032.stdout: Cleanup : kernel-5.14.0-480.el9.x86_64 2/2 2024-08-19T14:00:30.088 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:04 ago on Mon 19 Aug 2024 02:00:26 PM UTC. 2024-08-19T14:00:30.215 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-08-19T14:00:30.215 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-08-19T14:00:30.216 INFO:teuthology.orchestra.run.smithi045.stdout: Package Architecture Version Repository Size 2024-08-19T14:00:30.216 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-08-19T14:00:30.216 INFO:teuthology.orchestra.run.smithi045.stdout:Reinstalling: 2024-08-19T14:00:30.216 INFO:teuthology.orchestra.run.smithi045.stdout: kernel x86_64 5.14.0-480.el9 baseos 1.5 M 2024-08-19T14:00:30.216 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-08-19T14:00:30.216 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-08-19T14:00:30.216 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-08-19T14:00:30.216 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-08-19T14:00:30.216 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 1.5 M 2024-08-19T14:00:30.216 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 0 2024-08-19T14:00:30.216 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-08-19T14:00:30.368 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: kernel-5.14.0-480.el9.x86_64 2/2 2024-08-19T14:00:30.368 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : kernel-5.14.0-480.el9.x86_64 1/2 2024-08-19T14:00:30.875 INFO:teuthology.orchestra.run.smithi045.stdout:kernel-5.14.0-480.el9.x86_64.rpm 3.8 MB/s | 1.5 MB 00:00 2024-08-19T14:00:30.875 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-08-19T14:00:30.876 INFO:teuthology.orchestra.run.smithi045.stdout:Total 2.3 MB/s | 1.5 MB 00:00 2024-08-19T14:00:30.924 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-08-19T14:00:31.067 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : kernel-5.14.0-480.el9.x86_64 2/2 2024-08-19T14:00:31.067 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-19T14:00:31.067 INFO:teuthology.orchestra.run.smithi032.stdout:Reinstalled: 2024-08-19T14:00:31.068 INFO:teuthology.orchestra.run.smithi032.stdout: kernel-5.14.0-480.el9.x86_64 2024-08-19T14:00:31.068 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-19T14:00:31.068 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-08-19T14:00:31.086 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - CRB 7.9 MB/s | 6.6 MB 00:00 2024-08-19T14:00:31.156 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-08-19T14:00:31.156 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-08-19T14:00:31.181 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-08-19T14:00:31.182 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-08-19T14:00:31.251 DEBUG:teuthology.orchestra.run.smithi032:> rpm -q kernel | sort -rV | head -n 1 2024-08-19T14:00:31.307 INFO:teuthology.orchestra.run.smithi032.stdout:kernel-5.14.0-480.el9.x86_64 2024-08-19T14:00:31.307 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-480.el9.x86_64 2024-08-19T14:00:31.307 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-08-19T14:00:31.308 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-08-19T14:00:31.308 DEBUG:teuthology.orchestra.run.smithi032:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-08-19T14:00:31.336 INFO:teuthology.orchestra.run.smithi032.stdout:ttyS1 2024-08-19T14:00:31.344 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-08-19T14:00:31.367 DEBUG:teuthology.parallel:result is None 2024-08-19T14:00:31.436 INFO:teuthology.orchestra.run.smithi045.stdout: Reinstalling : kernel-5.14.0-480.el9.x86_64 1/2 2024-08-19T14:00:31.436 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : kernel-5.14.0-480.el9.x86_64 2/2 2024-08-19T14:00:32.182 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: kernel-5.14.0-480.el9.x86_64 2/2 2024-08-19T14:00:32.183 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : kernel-5.14.0-480.el9.x86_64 1/2 2024-08-19T14:00:32.662 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - Extras packages 62 kB/s | 18 kB 00:00 2024-08-19T14:00:32.869 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : kernel-5.14.0-480.el9.x86_64 2/2 2024-08-19T14:00:32.870 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-08-19T14:00:32.870 INFO:teuthology.orchestra.run.smithi045.stdout:Reinstalled: 2024-08-19T14:00:32.870 INFO:teuthology.orchestra.run.smithi045.stdout: kernel-5.14.0-480.el9.x86_64 2024-08-19T14:00:32.870 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-08-19T14:00:32.870 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-08-19T14:00:33.063 DEBUG:teuthology.orchestra.run.smithi045:> rpm -q kernel | sort -rV | head -n 1 2024-08-19T14:00:33.120 INFO:teuthology.orchestra.run.smithi045.stdout:kernel-5.14.0-480.el9.x86_64 2024-08-19T14:00:33.120 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-480.el9.x86_64 2024-08-19T14:00:33.120 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-08-19T14:00:33.121 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2024-08-19T14:00:33.121 DEBUG:teuthology.orchestra.run.smithi045:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-08-19T14:00:33.148 INFO:teuthology.orchestra.run.smithi045.stdout:ttyS1 2024-08-19T14:00:33.174 DEBUG:teuthology.parallel:result is None 2024-08-19T14:00:33.832 INFO:teuthology.orchestra.run.smithi031.stdout:Extra Packages for Enterprise Linux 21 MB/s | 22 MB 00:01 2024-08-19T14:00:39.397 INFO:teuthology.orchestra.run.smithi031.stdout:lab-extras 38 kB/s | 1.7 kB 00:00 2024-08-19T14:00:41.378 INFO:teuthology.orchestra.run.smithi031.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-08-19T14:00:41.379 INFO:teuthology.orchestra.run.smithi031.stdout:Package kernel-5.14.0-480.el9.x86_64 is already installed. 2024-08-19T14:00:41.451 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-08-19T14:00:41.452 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2024-08-19T14:00:41.452 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-08-19T14:00:41.556 DEBUG:teuthology.orchestra.run.smithi031:> echo no | sudo yum reinstall kernel || true 2024-08-19T14:00:42.131 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:00:03 ago on Mon 19 Aug 2024 02:00:39 PM UTC. 2024-08-19T14:00:42.273 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-08-19T14:00:42.278 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-08-19T14:00:42.278 INFO:teuthology.orchestra.run.smithi031.stdout: Package Architecture Version Repository Size 2024-08-19T14:00:42.278 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-08-19T14:00:42.278 INFO:teuthology.orchestra.run.smithi031.stdout:Reinstalling: 2024-08-19T14:00:42.278 INFO:teuthology.orchestra.run.smithi031.stdout: kernel x86_64 5.14.0-480.el9 baseos 1.5 M 2024-08-19T14:00:42.278 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:00:42.278 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-08-19T14:00:42.278 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-08-19T14:00:42.279 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:00:42.279 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 1.5 M 2024-08-19T14:00:42.279 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 0 2024-08-19T14:00:42.279 INFO:teuthology.orchestra.run.smithi031.stderr:Operation aborted. 2024-08-19T14:00:42.335 INFO:teuthology.orchestra.run.smithi031.stdout:Is this ok [y/N]: 2024-08-19T14:00:42.336 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum reinstall -y kernel || true 2024-08-19T14:00:42.868 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:00:03 ago on Mon 19 Aug 2024 02:00:39 PM UTC. 2024-08-19T14:00:42.994 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-08-19T14:00:42.995 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-08-19T14:00:42.995 INFO:teuthology.orchestra.run.smithi031.stdout: Package Architecture Version Repository Size 2024-08-19T14:00:42.996 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-08-19T14:00:42.996 INFO:teuthology.orchestra.run.smithi031.stdout:Reinstalling: 2024-08-19T14:00:42.996 INFO:teuthology.orchestra.run.smithi031.stdout: kernel x86_64 5.14.0-480.el9 baseos 1.5 M 2024-08-19T14:00:42.996 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:00:42.996 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-08-19T14:00:42.996 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-08-19T14:00:42.996 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:00:42.996 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 1.5 M 2024-08-19T14:00:42.996 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 0 2024-08-19T14:00:42.996 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-08-19T14:00:43.899 INFO:teuthology.orchestra.run.smithi031.stdout:kernel-5.14.0-480.el9.x86_64.rpm 1.8 MB/s | 1.5 MB 00:00 2024-08-19T14:00:43.899 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-08-19T14:00:43.899 INFO:teuthology.orchestra.run.smithi031.stdout:Total 1.7 MB/s | 1.5 MB 00:00 2024-08-19T14:00:43.935 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-08-19T14:00:44.184 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-08-19T14:00:44.184 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-08-19T14:00:44.210 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-08-19T14:00:44.210 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-08-19T14:00:44.360 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-08-19T14:00:44.450 INFO:teuthology.orchestra.run.smithi031.stdout: Reinstalling : kernel-5.14.0-480.el9.x86_64 1/2 2024-08-19T14:00:44.450 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : kernel-5.14.0-480.el9.x86_64 2/2 2024-08-19T14:00:45.108 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: kernel-5.14.0-480.el9.x86_64 2/2 2024-08-19T14:00:45.108 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : kernel-5.14.0-480.el9.x86_64 1/2 2024-08-19T14:00:45.869 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : kernel-5.14.0-480.el9.x86_64 2/2 2024-08-19T14:00:45.869 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:00:45.869 INFO:teuthology.orchestra.run.smithi031.stdout:Reinstalled: 2024-08-19T14:00:45.869 INFO:teuthology.orchestra.run.smithi031.stdout: kernel-5.14.0-480.el9.x86_64 2024-08-19T14:00:45.869 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:00:45.869 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-08-19T14:00:46.085 DEBUG:teuthology.orchestra.run.smithi031:> rpm -q kernel | sort -rV | head -n 1 2024-08-19T14:00:46.142 INFO:teuthology.orchestra.run.smithi031.stdout:kernel-5.14.0-480.el9.x86_64 2024-08-19T14:00:46.142 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-480.el9.x86_64 2024-08-19T14:00:46.142 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-08-19T14:00:46.142 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-08-19T14:00:46.143 DEBUG:teuthology.orchestra.run.smithi031:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-08-19T14:00:46.177 INFO:teuthology.orchestra.run.smithi031.stdout:ttyS1 2024-08-19T14:00:46.201 DEBUG:teuthology.parallel:result is None 2024-08-19T14:00:46.201 INFO:teuthology.run_tasks:Running task internal.base... 2024-08-19T14:00:46.210 INFO:teuthology.task.internal:Creating test directory... 2024-08-19T14:00:46.211 DEBUG:teuthology.orchestra.run.smithi031:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-08-19T14:00:46.244 DEBUG:teuthology.orchestra.run.smithi032:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-08-19T14:00:46.247 DEBUG:teuthology.orchestra.run.smithi045:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-08-19T14:00:46.265 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-08-19T14:00:46.273 INFO:teuthology.run_tasks:Running task internal.archive... 2024-08-19T14:00:46.282 INFO:teuthology.task.internal:Creating archive directory... 2024-08-19T14:00:46.282 DEBUG:teuthology.orchestra.run.smithi031:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-08-19T14:00:46.299 DEBUG:teuthology.orchestra.run.smithi032:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-08-19T14:00:46.306 DEBUG:teuthology.orchestra.run.smithi045:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-08-19T14:00:46.338 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-08-19T14:00:46.346 INFO:teuthology.task.internal:Enabling coredump saving... 2024-08-19T14:00:46.347 DEBUG:teuthology.orchestra.run.smithi031:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-08-19T14:00:46.365 DEBUG:teuthology.orchestra.run.smithi032:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-08-19T14:00:46.371 DEBUG:teuthology.orchestra.run.smithi045:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-08-19T14:00:46.391 INFO:teuthology.orchestra.run.smithi031.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T14:00:46.401 INFO:teuthology.orchestra.run.smithi032.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T14:00:46.403 INFO:teuthology.orchestra.run.smithi031.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T14:00:46.412 INFO:teuthology.orchestra.run.smithi032.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T14:00:46.428 INFO:teuthology.orchestra.run.smithi045.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T14:00:46.440 INFO:teuthology.orchestra.run.smithi045.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T14:00:46.442 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-08-19T14:00:46.450 INFO:teuthology.task.internal:Configuring sudo... 2024-08-19T14:00:46.450 DEBUG:teuthology.orchestra.run.smithi031:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-08-19T14:00:46.452 DEBUG:teuthology.orchestra.run.smithi032:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-08-19T14:00:46.462 DEBUG:teuthology.orchestra.run.smithi045:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-08-19T14:00:46.490 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-08-19T14:00:46.505 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-08-19T14:00:46.505 DEBUG:teuthology.orchestra.run.smithi031:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-08-19T14:00:46.518 DEBUG:teuthology.orchestra.run.smithi032:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-08-19T14:00:46.530 DEBUG:teuthology.orchestra.run.smithi045:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-08-19T14:00:46.547 DEBUG:teuthology.orchestra.run.smithi031:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-08-19T14:00:46.623 DEBUG:teuthology.orchestra.run.smithi031:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-08-19T14:00:46.664 DEBUG:teuthology.orchestra.run.smithi031:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-08-19T14:00:46.752 DEBUG:teuthology.orchestra.run.smithi031:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-08-19T14:00:46.785 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-08-19T14:00:46.785 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-08-19T14:00:46.852 DEBUG:teuthology.orchestra.run.smithi032:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-08-19T14:00:46.904 DEBUG:teuthology.orchestra.run.smithi032:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-08-19T14:00:46.944 DEBUG:teuthology.orchestra.run.smithi032:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-08-19T14:00:47.038 DEBUG:teuthology.orchestra.run.smithi032:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-08-19T14:00:47.072 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-19T14:00:47.072 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-08-19T14:00:47.139 DEBUG:teuthology.orchestra.run.smithi045:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-08-19T14:00:47.196 DEBUG:teuthology.orchestra.run.smithi045:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-08-19T14:00:47.241 DEBUG:teuthology.orchestra.run.smithi045:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-08-19T14:00:47.334 DEBUG:teuthology.orchestra.run.smithi045:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-08-19T14:00:47.370 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-08-19T14:00:47.370 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-08-19T14:00:47.438 DEBUG:teuthology.orchestra.run.smithi031:> sudo service rsyslog restart 2024-08-19T14:00:47.441 DEBUG:teuthology.orchestra.run.smithi032:> sudo service rsyslog restart 2024-08-19T14:00:47.444 DEBUG:teuthology.orchestra.run.smithi045:> sudo service rsyslog restart 2024-08-19T14:00:47.477 INFO:teuthology.orchestra.run.smithi031.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-08-19T14:00:47.499 INFO:teuthology.orchestra.run.smithi032.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-08-19T14:00:47.548 INFO:teuthology.orchestra.run.smithi045.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-08-19T14:00:47.907 INFO:teuthology.run_tasks:Running task internal.timer... 2024-08-19T14:00:47.914 INFO:teuthology.task.internal:Starting timer... 2024-08-19T14:00:47.914 INFO:teuthology.run_tasks:Running task pcp... 2024-08-19T14:00:47.926 INFO:teuthology.run_tasks:Running task selinux... 2024-08-19T14:00:47.933 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-08-19T14:00:47.934 DEBUG:teuthology.orchestra.run.smithi031:> sudo service auditd rotate 2024-08-19T14:00:48.046 INFO:teuthology.orchestra.run.smithi031.stdout:Rotating logs: 2024-08-19T14:00:48.048 DEBUG:teuthology.orchestra.run.smithi032:> sudo service auditd rotate 2024-08-19T14:00:48.105 INFO:teuthology.orchestra.run.smithi032.stdout:Rotating logs: 2024-08-19T14:00:48.107 DEBUG:teuthology.orchestra.run.smithi045:> sudo service auditd rotate 2024-08-19T14:00:48.173 INFO:teuthology.orchestra.run.smithi045.stdout:Rotating logs: 2024-08-19T14:00:48.177 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-08-19T14:00:48.177 DEBUG:teuthology.orchestra.run.smithi031:> /usr/sbin/getenforce 2024-08-19T14:00:48.196 INFO:teuthology.orchestra.run.smithi031.stdout:Permissive 2024-08-19T14:00:48.197 DEBUG:teuthology.orchestra.run.smithi032:> /usr/sbin/getenforce 2024-08-19T14:00:48.234 INFO:teuthology.orchestra.run.smithi032.stdout:Permissive 2024-08-19T14:00:48.235 DEBUG:teuthology.orchestra.run.smithi045:> /usr/sbin/getenforce 2024-08-19T14:00:48.266 INFO:teuthology.orchestra.run.smithi045.stdout:Permissive 2024-08-19T14:00:48.266 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi031.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi032.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi045.front.sepia.ceph.com': 'permissive'} 2024-08-19T14:00:48.267 DEBUG:teuthology.orchestra.run.smithi031:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-08-19T14:00:48.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:00:48.307 DEBUG:teuthology.orchestra.run.smithi032:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-08-19T14:00:48.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:00:48.335 DEBUG:teuthology.orchestra.run.smithi045:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-08-19T14:00:48.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:00:48.377 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-08-19T14:00:48.377 DEBUG:teuthology.orchestra.run.smithi031:> sudo /usr/sbin/setenforce permissive 2024-08-19T14:00:48.403 DEBUG:teuthology.orchestra.run.smithi032:> sudo /usr/sbin/setenforce permissive 2024-08-19T14:00:48.430 DEBUG:teuthology.orchestra.run.smithi045:> sudo /usr/sbin/setenforce permissive 2024-08-19T14:00:48.460 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-08-19T14:00:49.473 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-08-19T14:00:49.485 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-08-19T14:00:49.486 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi031.front.sepia.ceph.com,smithi032.front.sepia.ceph.com,smithi045.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-08-19T14:02:37.968 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi031.front.sepia.ceph.com'), Remote(name='ubuntu@smithi032.front.sepia.ceph.com'), Remote(name='ubuntu@smithi045.front.sepia.ceph.com')] 2024-08-19T14:02:37.970 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-08-19T14:02:37.971 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-19T14:02:38.060 DEBUG:teuthology.orchestra.run.smithi031:> true 2024-08-19T14:02:38.146 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-08-19T14:02:38.146 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com' 2024-08-19T14:02:38.147 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-19T14:02:38.234 DEBUG:teuthology.orchestra.run.smithi032:> true 2024-08-19T14:02:38.322 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi032.front.sepia.ceph.com' 2024-08-19T14:02:38.323 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi045.front.sepia.ceph.com' 2024-08-19T14:02:38.323 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-19T14:02:38.427 DEBUG:teuthology.orchestra.run.smithi045:> true 2024-08-19T14:02:38.511 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi045.front.sepia.ceph.com' 2024-08-19T14:02:38.512 INFO:teuthology.run_tasks:Running task clock... 2024-08-19T14:02:38.524 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-08-19T14:02:38.524 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-08-19T14:02:38.525 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-19T14:02:38.527 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-08-19T14:02:38.528 DEBUG:teuthology.orchestra.run.smithi032:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-19T14:02:38.530 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-08-19T14:02:38.530 DEBUG:teuthology.orchestra.run.smithi045:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-19T14:02:38.557 INFO:teuthology.orchestra.run.smithi031.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-08-19T14:02:38.561 INFO:teuthology.orchestra.run.smithi032.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-08-19T14:02:38.571 INFO:teuthology.orchestra.run.smithi031.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-08-19T14:02:38.576 INFO:teuthology.orchestra.run.smithi032.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-08-19T14:02:38.583 INFO:teuthology.orchestra.run.smithi045.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-08-19T14:02:38.598 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: ntpd: command not found 2024-08-19T14:02:38.598 INFO:teuthology.orchestra.run.smithi045.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-08-19T14:02:38.602 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: ntpd: command not found 2024-08-19T14:02:38.608 INFO:teuthology.orchestra.run.smithi031.stdout:506 Cannot talk to daemon 2024-08-19T14:02:38.614 INFO:teuthology.orchestra.run.smithi032.stdout:506 Cannot talk to daemon 2024-08-19T14:02:38.623 INFO:teuthology.orchestra.run.smithi031.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-08-19T14:02:38.624 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: ntpd: command not found 2024-08-19T14:02:38.628 INFO:teuthology.orchestra.run.smithi032.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-08-19T14:02:38.636 INFO:teuthology.orchestra.run.smithi045.stdout:506 Cannot talk to daemon 2024-08-19T14:02:38.638 INFO:teuthology.orchestra.run.smithi031.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-08-19T14:02:38.643 INFO:teuthology.orchestra.run.smithi032.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-08-19T14:02:38.651 INFO:teuthology.orchestra.run.smithi045.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-08-19T14:02:38.665 INFO:teuthology.orchestra.run.smithi045.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-08-19T14:02:38.696 INFO:teuthology.orchestra.run.smithi031.stderr:bash: line 1: ntpq: command not found 2024-08-19T14:02:38.699 INFO:teuthology.orchestra.run.smithi031.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-08-19T14:02:38.699 INFO:teuthology.orchestra.run.smithi031.stdout:=============================================================================== 2024-08-19T14:02:38.699 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-19T14:02:38.699 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-19T14:02:38.699 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-19T14:02:38.700 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-19T14:02:38.706 INFO:teuthology.orchestra.run.smithi032.stderr:bash: line 1: ntpq: command not found 2024-08-19T14:02:38.709 INFO:teuthology.orchestra.run.smithi032.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-08-19T14:02:38.710 INFO:teuthology.orchestra.run.smithi032.stdout:=============================================================================== 2024-08-19T14:02:38.710 INFO:teuthology.orchestra.run.smithi032.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-19T14:02:38.710 INFO:teuthology.orchestra.run.smithi032.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-19T14:02:38.710 INFO:teuthology.orchestra.run.smithi032.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-19T14:02:38.710 INFO:teuthology.orchestra.run.smithi032.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-19T14:02:38.720 INFO:teuthology.orchestra.run.smithi045.stderr:bash: line 1: ntpq: command not found 2024-08-19T14:02:38.723 INFO:teuthology.orchestra.run.smithi045.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-08-19T14:02:38.723 INFO:teuthology.orchestra.run.smithi045.stdout:=============================================================================== 2024-08-19T14:02:38.723 INFO:teuthology.orchestra.run.smithi045.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-19T14:02:38.724 INFO:teuthology.orchestra.run.smithi045.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-19T14:02:38.724 INFO:teuthology.orchestra.run.smithi045.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-19T14:02:38.724 INFO:teuthology.orchestra.run.smithi045.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-19T14:02:38.725 INFO:teuthology.run_tasks:Running task install... 2024-08-19T14:02:38.733 DEBUG:teuthology.task.install:project ceph 2024-08-19T14:02:38.733 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'de09dd018de4baddfa6ca4d43bc8fb784cf75833'}} 2024-08-19T14:02:38.733 DEBUG:teuthology.task.install:config {'branch': 'quincy', 'exclude_packages': ['ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo', 'ceph-volume'], 'flavor': 'default', 'sha1': 'de09dd018de4baddfa6ca4d43bc8fb784cf75833'} 2024-08-19T14:02:38.733 INFO:teuthology.task.install:Using flavor: default 2024-08-19T14:02:38.740 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-08-19T14:02:38.740 INFO:teuthology.task.install:extra packages: [] 2024-08-19T14:02:38.741 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo', 'ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'de09dd018de4baddfa6ca4d43bc8fb784cf75833', 'tag': None, 'wait_for_package': False} 2024-08-19T14:02:38.741 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-08-19T14:02:38.741 INFO:teuthology.packaging:ref: None 2024-08-19T14:02:38.742 INFO:teuthology.packaging:tag: None 2024-08-19T14:02:38.742 INFO:teuthology.packaging:branch: quincy 2024-08-19T14:02:38.742 INFO:teuthology.packaging:sha1: de09dd018de4baddfa6ca4d43bc8fb784cf75833 2024-08-19T14:02:38.742 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-08-19T14:02:38.744 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo', 'ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'de09dd018de4baddfa6ca4d43bc8fb784cf75833', 'tag': None, 'wait_for_package': False} 2024-08-19T14:02:38.745 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-08-19T14:02:38.745 INFO:teuthology.packaging:ref: None 2024-08-19T14:02:38.745 INFO:teuthology.packaging:tag: None 2024-08-19T14:02:38.745 INFO:teuthology.packaging:branch: quincy 2024-08-19T14:02:38.745 INFO:teuthology.packaging:sha1: de09dd018de4baddfa6ca4d43bc8fb784cf75833 2024-08-19T14:02:38.745 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-08-19T14:02:38.746 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo', 'ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'de09dd018de4baddfa6ca4d43bc8fb784cf75833', 'tag': None, 'wait_for_package': False} 2024-08-19T14:02:38.746 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-08-19T14:02:38.746 INFO:teuthology.packaging:ref: None 2024-08-19T14:02:38.746 INFO:teuthology.packaging:tag: None 2024-08-19T14:02:38.746 INFO:teuthology.packaging:branch: quincy 2024-08-19T14:02:38.746 INFO:teuthology.packaging:sha1: de09dd018de4baddfa6ca4d43bc8fb784cf75833 2024-08-19T14:02:38.747 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-08-19T14:02:38.972 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/quincy/df62bdccd9a13bf566b27a250eddfdd89be3072d/centos/9/flavors/default/ 2024-08-19T14:02:38.972 INFO:teuthology.task.install.rpm:Package version is 17.2.7-1406.gdf62bdcc 2024-08-19T14:02:39.008 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/quincy/df62bdccd9a13bf566b27a250eddfdd89be3072d/centos/9/flavors/default/ 2024-08-19T14:02:39.008 INFO:teuthology.task.install.rpm:Package version is 17.2.7-1406.gdf62bdcc 2024-08-19T14:02:39.032 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/quincy/df62bdccd9a13bf566b27a250eddfdd89be3072d/centos/9/flavors/default/ 2024-08-19T14:02:39.033 INFO:teuthology.task.install.rpm:Package version is 17.2.7-1406.gdf62bdcc 2024-08-19T14:02:39.114 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/quincy/df62bdccd9a13bf566b27a250eddfdd89be3072d/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/quincy/df62bdccd9a13bf566b27a250eddfdd89be3072d/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/quincy/df62bdccd9a13bf566b27a250eddfdd89be3072d/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-08-19T14:02:39.114 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-08-19T14:02:39.114 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-08-19T14:02:39.132 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/quincy/df62bdccd9a13bf566b27a250eddfdd89be3072d/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/quincy/df62bdccd9a13bf566b27a250eddfdd89be3072d/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/quincy/df62bdccd9a13bf566b27a250eddfdd89be3072d/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-08-19T14:02:39.132 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-19T14:02:39.132 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-08-19T14:02:39.141 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-08-19T14:02:39.141 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-08-19T14:02:39.141 INFO:teuthology.packaging:ref: None 2024-08-19T14:02:39.141 INFO:teuthology.packaging:tag: None 2024-08-19T14:02:39.142 INFO:teuthology.packaging:branch: quincy 2024-08-19T14:02:39.142 INFO:teuthology.packaging:sha1: de09dd018de4baddfa6ca4d43bc8fb784cf75833 2024-08-19T14:02:39.142 DEBUG:teuthology.orchestra.run.smithi031:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-08-19T14:02:39.150 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/quincy/df62bdccd9a13bf566b27a250eddfdd89be3072d/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/quincy/df62bdccd9a13bf566b27a250eddfdd89be3072d/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/quincy/df62bdccd9a13bf566b27a250eddfdd89be3072d/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-08-19T14:02:39.150 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-08-19T14:02:39.150 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-08-19T14:02:39.160 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-08-19T14:02:39.160 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-08-19T14:02:39.160 INFO:teuthology.packaging:ref: None 2024-08-19T14:02:39.160 INFO:teuthology.packaging:tag: None 2024-08-19T14:02:39.160 INFO:teuthology.packaging:branch: quincy 2024-08-19T14:02:39.160 INFO:teuthology.packaging:sha1: de09dd018de4baddfa6ca4d43bc8fb784cf75833 2024-08-19T14:02:39.161 DEBUG:teuthology.orchestra.run.smithi032:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-08-19T14:02:39.176 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-08-19T14:02:39.177 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-08-19T14:02:39.177 INFO:teuthology.packaging:ref: None 2024-08-19T14:02:39.177 INFO:teuthology.packaging:tag: None 2024-08-19T14:02:39.177 INFO:teuthology.packaging:branch: quincy 2024-08-19T14:02:39.177 INFO:teuthology.packaging:sha1: de09dd018de4baddfa6ca4d43bc8fb784cf75833 2024-08-19T14:02:39.178 DEBUG:teuthology.orchestra.run.smithi045:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-08-19T14:02:39.207 DEBUG:teuthology.orchestra.run.smithi031:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-08-19T14:02:39.228 DEBUG:teuthology.orchestra.run.smithi032:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-08-19T14:02:39.244 DEBUG:teuthology.orchestra.run.smithi045:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-08-19T14:02:39.282 DEBUG:teuthology.orchestra.run.smithi031:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-08-19T14:02:39.306 DEBUG:teuthology.orchestra.run.smithi032:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-08-19T14:02:39.321 DEBUG:teuthology.orchestra.run.smithi045:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-08-19T14:02:39.346 INFO:teuthology.orchestra.run.smithi031.stdout:check_obsoletes = 1 2024-08-19T14:02:39.348 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum clean all 2024-08-19T14:02:39.372 INFO:teuthology.orchestra.run.smithi032.stdout:check_obsoletes = 1 2024-08-19T14:02:39.374 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum clean all 2024-08-19T14:02:39.385 INFO:teuthology.orchestra.run.smithi045.stdout:check_obsoletes = 1 2024-08-19T14:02:39.386 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum clean all 2024-08-19T14:02:39.579 INFO:teuthology.orchestra.run.smithi031.stdout:45 files removed 2024-08-19T14:02:39.605 INFO:teuthology.orchestra.run.smithi032.stdout:45 files removed 2024-08-19T14:02:39.607 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-radosgw 2024-08-19T14:02:39.617 INFO:teuthology.orchestra.run.smithi045.stdout:45 files removed 2024-08-19T14:02:39.632 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph-radosgw 2024-08-19T14:02:39.644 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-radosgw 2024-08-19T14:02:40.154 INFO:teuthology.orchestra.run.smithi031.stdout:ceph packages for x86_64 289 kB/s | 77 kB 00:00 2024-08-19T14:02:40.177 INFO:teuthology.orchestra.run.smithi032.stdout:ceph packages for x86_64 292 kB/s | 77 kB 00:00 2024-08-19T14:02:40.187 INFO:teuthology.orchestra.run.smithi045.stdout:ceph packages for x86_64 296 kB/s | 77 kB 00:00 2024-08-19T14:02:40.455 INFO:teuthology.orchestra.run.smithi031.stdout:ceph noarch packages 42 kB/s | 11 kB 00:00 2024-08-19T14:02:40.460 INFO:teuthology.orchestra.run.smithi032.stdout:ceph noarch packages 45 kB/s | 11 kB 00:00 2024-08-19T14:02:40.471 INFO:teuthology.orchestra.run.smithi045.stdout:ceph noarch packages 45 kB/s | 11 kB 00:00 2024-08-19T14:02:40.720 INFO:teuthology.orchestra.run.smithi032.stdout:ceph source packages 8.0 kB/s | 1.8 kB 00:00 2024-08-19T14:02:40.730 INFO:teuthology.orchestra.run.smithi031.stdout:ceph source packages 7.5 kB/s | 1.8 kB 00:00 2024-08-19T14:02:40.763 INFO:teuthology.orchestra.run.smithi045.stdout:ceph source packages 7.0 kB/s | 1.8 kB 00:00 2024-08-19T14:02:41.247 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - BaseOS 17 MB/s | 8.2 MB 00:00 2024-08-19T14:02:41.453 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.2 MB 00:00 2024-08-19T14:02:41.722 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - BaseOS 8.9 MB/s | 8.2 MB 00:00 2024-08-19T14:02:44.064 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - AppStream 17 MB/s | 20 MB 00:01 2024-08-19T14:02:44.613 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 9 - AppStream 9.8 MB/s | 20 MB 00:02 2024-08-19T14:02:49.815 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - AppStream 2.7 MB/s | 20 MB 00:07 2024-08-19T14:02:50.663 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 9 - CRB 3.0 MB/s | 6.6 MB 00:02 2024-08-19T14:02:52.372 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 9 - Extras packages 41 kB/s | 18 kB 00:00 2024-08-19T14:02:53.452 INFO:teuthology.orchestra.run.smithi032.stdout:Extra Packages for Enterprise Linux 24 MB/s | 22 MB 00:00 2024-08-19T14:02:54.398 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - CRB 1.0 MB/s | 6.6 MB 00:06 2024-08-19T14:02:54.582 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - CRB 7.4 MB/s | 6.6 MB 00:00 2024-08-19T14:02:55.907 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - Extras packages 72 kB/s | 18 kB 00:00 2024-08-19T14:02:56.827 INFO:teuthology.orchestra.run.smithi045.stdout:Extra Packages for Enterprise Linux 28 MB/s | 22 MB 00:00 2024-08-19T14:02:56.849 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - Extras packages 18 kB/s | 18 kB 00:01 2024-08-19T14:02:57.761 INFO:teuthology.orchestra.run.smithi031.stdout:Extra Packages for Enterprise Linux 29 MB/s | 22 MB 00:00 2024-08-19T14:02:59.041 INFO:teuthology.orchestra.run.smithi032.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-08-19T14:03:01.305 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-08-19T14:03:01.307 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-08-19T14:03:01.307 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repository Size 2024-08-19T14:03:01.307 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-08-19T14:03:01.307 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-08-19T14:03:01.307 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-radosgw x86_64 2:17.2.7-1406.gdf62bdcc.el9 ceph 14 M 2024-08-19T14:03:01.307 INFO:teuthology.orchestra.run.smithi032.stdout:Upgrading: 2024-08-19T14:03:01.307 INFO:teuthology.orchestra.run.smithi032.stdout: librados2 x86_64 2:17.2.7-1406.gdf62bdcc.el9 ceph 3.1 M 2024-08-19T14:03:01.307 INFO:teuthology.orchestra.run.smithi032.stdout: librbd1 x86_64 2:17.2.7-1406.gdf62bdcc.el9 ceph 2.9 M 2024-08-19T14:03:01.308 INFO:teuthology.orchestra.run.smithi032.stdout:Installing dependencies: 2024-08-19T14:03:01.308 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-base x86_64 2:17.2.7-1406.gdf62bdcc.el9 ceph 5.0 M 2024-08-19T14:03:01.308 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-common x86_64 2:17.2.7-1406.gdf62bdcc.el9 ceph 22 M 2024-08-19T14:03:01.308 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-selinux x86_64 2:17.2.7-1406.gdf62bdcc.el9 ceph 25 k 2024-08-19T14:03:01.308 INFO:teuthology.orchestra.run.smithi032.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-08-19T14:03:01.308 INFO:teuthology.orchestra.run.smithi032.stdout: libcephfs2 x86_64 2:17.2.7-1406.gdf62bdcc.el9 ceph 636 k 2024-08-19T14:03:01.308 INFO:teuthology.orchestra.run.smithi032.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-08-19T14:03:01.308 INFO:teuthology.orchestra.run.smithi032.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-08-19T14:03:01.308 INFO:teuthology.orchestra.run.smithi032.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-08-19T14:03:01.308 INFO:teuthology.orchestra.run.smithi032.stdout: libradosstriper1 x86_64 2:17.2.7-1406.gdf62bdcc.el9 ceph 457 k 2024-08-19T14:03:01.308 INFO:teuthology.orchestra.run.smithi032.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-08-19T14:03:01.309 INFO:teuthology.orchestra.run.smithi032.stdout: librgw2 x86_64 2:17.2.7-1406.gdf62bdcc.el9 ceph 4.9 M 2024-08-19T14:03:01.309 INFO:teuthology.orchestra.run.smithi032.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-08-19T14:03:01.309 INFO:teuthology.orchestra.run.smithi032.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-08-19T14:03:01.309 INFO:teuthology.orchestra.run.smithi032.stdout: python3-ceph-argparse x86_64 2:17.2.7-1406.gdf62bdcc.el9 ceph 45 k 2024-08-19T14:03:01.309 INFO:teuthology.orchestra.run.smithi032.stdout: python3-ceph-common x86_64 2:17.2.7-1406.gdf62bdcc.el9 ceph 120 k 2024-08-19T14:03:01.309 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cephfs x86_64 2:17.2.7-1406.gdf62bdcc.el9 ceph 182 k 2024-08-19T14:03:01.309 INFO:teuthology.orchestra.run.smithi032.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-08-19T14:03:01.309 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rados x86_64 2:17.2.7-1406.gdf62bdcc.el9 ceph 367 k 2024-08-19T14:03:01.309 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rbd x86_64 2:17.2.7-1406.gdf62bdcc.el9 ceph 343 k 2024-08-19T14:03:01.309 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rgw x86_64 2:17.2.7-1406.gdf62bdcc.el9 ceph 116 k 2024-08-19T14:03:01.309 INFO:teuthology.orchestra.run.smithi032.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-08-19T14:03:01.309 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-19T14:03:01.309 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-08-19T14:03:01.309 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-08-19T14:03:01.310 INFO:teuthology.orchestra.run.smithi032.stdout:Install 22 Packages 2024-08-19T14:03:01.310 INFO:teuthology.orchestra.run.smithi032.stdout:Upgrade 2 Packages 2024-08-19T14:03:01.310 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-19T14:03:01.310 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 58 M 2024-08-19T14:03:01.310 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-08-19T14:03:01.884 INFO:teuthology.orchestra.run.smithi032.stdout:(1/24): ceph-base-17.2.7-1406.gdf62bdcc.el9.x86 10 MB/s | 5.0 MB 00:00 2024-08-19T14:03:01.951 INFO:teuthology.orchestra.run.smithi032.stdout:(2/24): ceph-selinux-17.2.7-1406.gdf62bdcc.el9. 377 kB/s | 25 kB 00:00 2024-08-19T14:03:02.059 INFO:teuthology.orchestra.run.smithi032.stdout:(3/24): libcephfs2-17.2.7-1406.gdf62bdcc.el9.x8 5.7 MB/s | 636 kB 00:00 2024-08-19T14:03:02.160 INFO:teuthology.orchestra.run.smithi032.stdout:(4/24): libradosstriper1-17.2.7-1406.gdf62bdcc. 4.5 MB/s | 457 kB 00:00 2024-08-19T14:03:02.318 INFO:teuthology.orchestra.run.smithi032.stdout:(5/24): ceph-radosgw-17.2.7-1406.gdf62bdcc.el9. 16 MB/s | 14 MB 00:00 2024-08-19T14:03:02.360 INFO:teuthology.orchestra.run.smithi032.stdout:(6/24): python3-ceph-argparse-17.2.7-1406.gdf62 1.1 MB/s | 45 kB 00:00 2024-08-19T14:03:02.446 INFO:teuthology.orchestra.run.smithi032.stdout:(7/24): librgw2-17.2.7-1406.gdf62bdcc.el9.x86_6 17 MB/s | 4.9 MB 00:00 2024-08-19T14:03:02.469 INFO:teuthology.orchestra.run.smithi032.stdout:(8/24): python3-ceph-common-17.2.7-1406.gdf62bd 1.1 MB/s | 120 kB 00:00 2024-08-19T14:03:02.494 INFO:teuthology.orchestra.run.smithi032.stdout:(9/24): python3-cephfs-17.2.7-1406.gdf62bdcc.el 3.5 MB/s | 182 kB 00:00 2024-08-19T14:03:02.519 INFO:teuthology.orchestra.run.smithi032.stdout:(10/24): python3-rados-17.2.7-1406.gdf62bdcc.el 7.2 MB/s | 367 kB 00:00 2024-08-19T14:03:02.545 INFO:teuthology.orchestra.run.smithi032.stdout:(11/24): python3-rbd-17.2.7-1406.gdf62bdcc.el9. 6.7 MB/s | 343 kB 00:00 2024-08-19T14:03:02.570 INFO:teuthology.orchestra.run.smithi032.stdout:(12/24): python3-rgw-17.2.7-1406.gdf62bdcc.el9. 2.3 MB/s | 116 kB 00:00 2024-08-19T14:03:02.608 INFO:teuthology.orchestra.run.smithi045.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-08-19T14:03:02.787 INFO:teuthology.orchestra.run.smithi032.stdout:(13/24): ceph-common-17.2.7-1406.gdf62bdcc.el9. 16 MB/s | 22 MB 00:01 2024-08-19T14:03:02.904 INFO:teuthology.orchestra.run.smithi032.stdout:(14/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 446 kB/s | 160 kB 00:00 2024-08-19T14:03:02.929 INFO:teuthology.orchestra.run.smithi032.stdout:(15/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 126 kB/s | 45 kB 00:00 2024-08-19T14:03:02.971 INFO:teuthology.orchestra.run.smithi032.stdout:(16/24): python3-prettytable-0.7.2-27.el9.noarc 1.0 MB/s | 42 kB 00:00 2024-08-19T14:03:02.996 INFO:teuthology.orchestra.run.smithi032.stdout:(17/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.1 MB/s | 292 kB 00:00 2024-08-19T14:03:03.058 INFO:teuthology.orchestra.run.smithi032.stdout:(18/24): librdkafka-1.6.1-102.el9.x86_64.rpm 2.4 MB/s | 662 kB 00:00 2024-08-19T14:03:03.097 INFO:teuthology.orchestra.run.smithi032.stdout:(19/24): liboath-2.6.7-2.el9.x86_64.rpm 484 kB/s | 49 kB 00:00 2024-08-19T14:03:03.122 INFO:teuthology.orchestra.run.smithi032.stdout:(20/24): gperftools-libs-2.9.1-2.el9.x86_64.rpm 2.0 MB/s | 309 kB 00:00 2024-08-19T14:03:03.148 INFO:teuthology.orchestra.run.smithi032.stdout:(21/24): libunwind-1.6.2-1.el9.x86_64.rpm 745 kB/s | 67 kB 00:00 2024-08-19T14:03:03.223 INFO:teuthology.orchestra.run.smithi032.stdout:(22/24): thrift-0.15.0-2.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-08-19T14:03:03.290 INFO:teuthology.orchestra.run.smithi032.stdout:(23/24): librados2-17.2.7-1406.gdf62bdcc.el9.x8 18 MB/s | 3.1 MB 00:00 2024-08-19T14:03:03.340 INFO:teuthology.orchestra.run.smithi032.stdout:(24/24): librbd1-17.2.7-1406.gdf62bdcc.el9.x86_ 15 MB/s | 2.9 MB 00:00 2024-08-19T14:03:03.345 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-08-19T14:03:03.345 INFO:teuthology.orchestra.run.smithi032.stdout:Total 29 MB/s | 58 MB 00:02 2024-08-19T14:03:03.408 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-08-19T14:03:03.419 INFO:teuthology.orchestra.run.smithi031.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-08-19T14:03:03.443 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-08-19T14:03:03.443 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-08-19T14:03:03.902 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-08-19T14:03:03.902 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-08-19T14:03:04.912 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-08-19T14:03:04.914 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-08-19T14:03:04.914 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2024-08-19T14:03:04.914 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-08-19T14:03:04.915 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-08-19T14:03:04.915 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-radosgw x86_64 2:17.2.7-1406.gdf62bdcc.el9 ceph 14 M 2024-08-19T14:03:04.915 INFO:teuthology.orchestra.run.smithi045.stdout:Upgrading: 2024-08-19T14:03:04.915 INFO:teuthology.orchestra.run.smithi045.stdout: librados2 x86_64 2:17.2.7-1406.gdf62bdcc.el9 ceph 3.1 M 2024-08-19T14:03:04.915 INFO:teuthology.orchestra.run.smithi045.stdout: librbd1 x86_64 2:17.2.7-1406.gdf62bdcc.el9 ceph 2.9 M 2024-08-19T14:03:04.915 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2024-08-19T14:03:04.915 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-base x86_64 2:17.2.7-1406.gdf62bdcc.el9 ceph 5.0 M 2024-08-19T14:03:04.915 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-common x86_64 2:17.2.7-1406.gdf62bdcc.el9 ceph 22 M 2024-08-19T14:03:04.915 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-selinux x86_64 2:17.2.7-1406.gdf62bdcc.el9 ceph 25 k 2024-08-19T14:03:04.915 INFO:teuthology.orchestra.run.smithi045.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-08-19T14:03:04.915 INFO:teuthology.orchestra.run.smithi045.stdout: libcephfs2 x86_64 2:17.2.7-1406.gdf62bdcc.el9 ceph 636 k 2024-08-19T14:03:04.915 INFO:teuthology.orchestra.run.smithi045.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-08-19T14:03:04.916 INFO:teuthology.orchestra.run.smithi045.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-08-19T14:03:04.916 INFO:teuthology.orchestra.run.smithi045.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-08-19T14:03:04.916 INFO:teuthology.orchestra.run.smithi045.stdout: libradosstriper1 x86_64 2:17.2.7-1406.gdf62bdcc.el9 ceph 457 k 2024-08-19T14:03:04.916 INFO:teuthology.orchestra.run.smithi045.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-08-19T14:03:04.916 INFO:teuthology.orchestra.run.smithi045.stdout: librgw2 x86_64 2:17.2.7-1406.gdf62bdcc.el9 ceph 4.9 M 2024-08-19T14:03:04.916 INFO:teuthology.orchestra.run.smithi045.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-08-19T14:03:04.916 INFO:teuthology.orchestra.run.smithi045.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-08-19T14:03:04.916 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ceph-argparse x86_64 2:17.2.7-1406.gdf62bdcc.el9 ceph 45 k 2024-08-19T14:03:04.916 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ceph-common x86_64 2:17.2.7-1406.gdf62bdcc.el9 ceph 120 k 2024-08-19T14:03:04.916 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cephfs x86_64 2:17.2.7-1406.gdf62bdcc.el9 ceph 182 k 2024-08-19T14:03:04.916 INFO:teuthology.orchestra.run.smithi045.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-08-19T14:03:04.916 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rados x86_64 2:17.2.7-1406.gdf62bdcc.el9 ceph 367 k 2024-08-19T14:03:04.916 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rbd x86_64 2:17.2.7-1406.gdf62bdcc.el9 ceph 343 k 2024-08-19T14:03:04.917 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rgw x86_64 2:17.2.7-1406.gdf62bdcc.el9 ceph 116 k 2024-08-19T14:03:04.917 INFO:teuthology.orchestra.run.smithi045.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-08-19T14:03:04.917 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-08-19T14:03:04.917 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-08-19T14:03:04.917 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-08-19T14:03:04.917 INFO:teuthology.orchestra.run.smithi045.stdout:Install 22 Packages 2024-08-19T14:03:04.917 INFO:teuthology.orchestra.run.smithi045.stdout:Upgrade 2 Packages 2024-08-19T14:03:04.917 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-08-19T14:03:04.917 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 58 M 2024-08-19T14:03:04.918 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-08-19T14:03:05.141 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-08-19T14:03:05.494 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-08-19T14:03:05.576 INFO:teuthology.orchestra.run.smithi045.stdout:(1/24): ceph-base-17.2.7-1406.gdf62bdcc.el9.x86 11 MB/s | 5.0 MB 00:00 2024-08-19T14:03:05.592 INFO:teuthology.orchestra.run.smithi032.stdout: Upgrading : librados2-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2/26 2024-08-19T14:03:05.634 INFO:teuthology.orchestra.run.smithi045.stdout:(2/24): ceph-selinux-17.2.7-1406.gdf62bdcc.el9. 433 kB/s | 25 kB 00:00 2024-08-19T14:03:05.679 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-08-19T14:03:05.680 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: librados2-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2/26 2024-08-19T14:03:05.681 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-08-19T14:03:05.681 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2024-08-19T14:03:05.681 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-08-19T14:03:05.681 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-08-19T14:03:05.682 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-radosgw x86_64 2:17.2.7-1406.gdf62bdcc.el9 ceph 14 M 2024-08-19T14:03:05.682 INFO:teuthology.orchestra.run.smithi031.stdout:Upgrading: 2024-08-19T14:03:05.682 INFO:teuthology.orchestra.run.smithi031.stdout: librados2 x86_64 2:17.2.7-1406.gdf62bdcc.el9 ceph 3.1 M 2024-08-19T14:03:05.682 INFO:teuthology.orchestra.run.smithi031.stdout: librbd1 x86_64 2:17.2.7-1406.gdf62bdcc.el9 ceph 2.9 M 2024-08-19T14:03:05.682 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2024-08-19T14:03:05.683 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-base x86_64 2:17.2.7-1406.gdf62bdcc.el9 ceph 5.0 M 2024-08-19T14:03:05.683 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-common x86_64 2:17.2.7-1406.gdf62bdcc.el9 ceph 22 M 2024-08-19T14:03:05.683 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-selinux x86_64 2:17.2.7-1406.gdf62bdcc.el9 ceph 25 k 2024-08-19T14:03:05.683 INFO:teuthology.orchestra.run.smithi031.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-08-19T14:03:05.683 INFO:teuthology.orchestra.run.smithi031.stdout: libcephfs2 x86_64 2:17.2.7-1406.gdf62bdcc.el9 ceph 636 k 2024-08-19T14:03:05.683 INFO:teuthology.orchestra.run.smithi031.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-08-19T14:03:05.684 INFO:teuthology.orchestra.run.smithi031.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-08-19T14:03:05.684 INFO:teuthology.orchestra.run.smithi031.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-08-19T14:03:05.684 INFO:teuthology.orchestra.run.smithi031.stdout: libradosstriper1 x86_64 2:17.2.7-1406.gdf62bdcc.el9 ceph 457 k 2024-08-19T14:03:05.684 INFO:teuthology.orchestra.run.smithi031.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-08-19T14:03:05.684 INFO:teuthology.orchestra.run.smithi031.stdout: librgw2 x86_64 2:17.2.7-1406.gdf62bdcc.el9 ceph 4.9 M 2024-08-19T14:03:05.684 INFO:teuthology.orchestra.run.smithi031.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-08-19T14:03:05.685 INFO:teuthology.orchestra.run.smithi031.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-08-19T14:03:05.685 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ceph-argparse x86_64 2:17.2.7-1406.gdf62bdcc.el9 ceph 45 k 2024-08-19T14:03:05.685 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ceph-common x86_64 2:17.2.7-1406.gdf62bdcc.el9 ceph 120 k 2024-08-19T14:03:05.685 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cephfs x86_64 2:17.2.7-1406.gdf62bdcc.el9 ceph 182 k 2024-08-19T14:03:05.685 INFO:teuthology.orchestra.run.smithi031.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-08-19T14:03:05.685 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rados x86_64 2:17.2.7-1406.gdf62bdcc.el9 ceph 367 k 2024-08-19T14:03:05.685 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rbd x86_64 2:17.2.7-1406.gdf62bdcc.el9 ceph 343 k 2024-08-19T14:03:05.685 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rgw x86_64 2:17.2.7-1406.gdf62bdcc.el9 ceph 116 k 2024-08-19T14:03:05.685 INFO:teuthology.orchestra.run.smithi031.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-08-19T14:03:05.685 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:03:05.685 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-08-19T14:03:05.685 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-08-19T14:03:05.686 INFO:teuthology.orchestra.run.smithi031.stdout:Install 22 Packages 2024-08-19T14:03:05.686 INFO:teuthology.orchestra.run.smithi031.stdout:Upgrade 2 Packages 2024-08-19T14:03:05.686 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:03:05.686 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 58 M 2024-08-19T14:03:05.686 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-08-19T14:03:05.718 INFO:teuthology.orchestra.run.smithi045.stdout:(3/24): libcephfs2-17.2.7-1406.gdf62bdcc.el9.x8 7.5 MB/s | 636 kB 00:00 2024-08-19T14:03:05.765 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-rados-2:17.2.7-1406.gdf62bdcc.el9.x86_64 3/26 2024-08-19T14:03:05.793 INFO:teuthology.orchestra.run.smithi045.stdout:(4/24): libradosstriper1-17.2.7-1406.gdf62bdcc. 6.0 MB/s | 457 kB 00:00 2024-08-19T14:03:05.802 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libcephfs2-2:17.2.7-1406.gdf62bdcc.el9.x86_64 4/26 2024-08-19T14:03:05.891 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: libcephfs2-2:17.2.7-1406.gdf62bdcc.el9.x86_64 4/26 2024-08-19T14:03:05.936 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-08-19T14:03:05.965 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-08-19T14:03:06.177 INFO:teuthology.orchestra.run.smithi045.stdout:(5/24): ceph-common-17.2.7-1406.gdf62bdcc.el9.x 21 MB/s | 22 MB 00:01 2024-08-19T14:03:06.260 INFO:teuthology.orchestra.run.smithi045.stdout:(6/24): librgw2-17.2.7-1406.gdf62bdcc.el9.x86_6 11 MB/s | 4.9 MB 00:00 2024-08-19T14:03:06.377 INFO:teuthology.orchestra.run.smithi045.stdout:(7/24): python3-ceph-common-17.2.7-1406.gdf62bd 1.0 MB/s | 120 kB 00:00 2024-08-19T14:03:06.426 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-08-19T14:03:06.437 INFO:teuthology.orchestra.run.smithi031.stdout:(1/24): ceph-base-17.2.7-1406.gdf62bdcc.el9.x86 7.7 MB/s | 5.0 MB 00:00 2024-08-19T14:03:06.444 INFO:teuthology.orchestra.run.smithi045.stdout:(8/24): python3-cephfs-17.2.7-1406.gdf62bdcc.el 2.7 MB/s | 182 kB 00:00 2024-08-19T14:03:06.455 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : librgw2-2:17.2.7-1406.gdf62bdcc.el9.x86_64 8/26 2024-08-19T14:03:06.504 INFO:teuthology.orchestra.run.smithi031.stdout:(2/24): ceph-selinux-17.2.7-1406.gdf62bdcc.el9. 378 kB/s | 25 kB 00:00 2024-08-19T14:03:06.517 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: librgw2-2:17.2.7-1406.gdf62bdcc.el9.x86_64 8/26 2024-08-19T14:03:06.527 INFO:teuthology.orchestra.run.smithi045.stdout:(9/24): python3-rados-17.2.7-1406.gdf62bdcc.el9 4.3 MB/s | 367 kB 00:00 2024-08-19T14:03:06.546 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-08-19T14:03:06.578 INFO:teuthology.orchestra.run.smithi045.stdout:(10/24): python3-ceph-argparse-17.2.7-1406.gdf6 112 kB/s | 45 kB 00:00 2024-08-19T14:03:06.592 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-ceph-argparse-2:17.2.7-1406.gdf62bdcc.el9. 10/26 2024-08-19T14:03:06.596 INFO:teuthology.orchestra.run.smithi031.stdout:(3/24): libcephfs2-17.2.7-1406.gdf62bdcc.el9.x8 6.8 MB/s | 636 kB 00:00 2024-08-19T14:03:06.620 INFO:teuthology.orchestra.run.smithi045.stdout:(11/24): python3-rbd-17.2.7-1406.gdf62bdcc.el9. 3.6 MB/s | 343 kB 00:00 2024-08-19T14:03:06.628 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-cephfs-2:17.2.7-1406.gdf62bdcc.el9.x86_64 11/26 2024-08-19T14:03:06.653 INFO:teuthology.orchestra.run.smithi045.stdout:(12/24): python3-rgw-17.2.7-1406.gdf62bdcc.el9. 1.5 MB/s | 116 kB 00:00 2024-08-19T14:03:06.688 INFO:teuthology.orchestra.run.smithi031.stdout:(4/24): libradosstriper1-17.2.7-1406.gdf62bdcc. 4.9 MB/s | 457 kB 00:00 2024-08-19T14:03:06.698 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-rgw-2:17.2.7-1406.gdf62bdcc.el9.x86_64 12/26 2024-08-19T14:03:06.724 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libradosstriper1-2:17.2.7-1406.gdf62bdcc.el9.x86_6 13/26 2024-08-19T14:03:06.770 INFO:teuthology.orchestra.run.smithi045.stdout:(13/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 388 kB/s | 45 kB 00:00 2024-08-19T14:03:06.790 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: libradosstriper1-2:17.2.7-1406.gdf62bdcc.el9.x86_6 13/26 2024-08-19T14:03:06.795 INFO:teuthology.orchestra.run.smithi045.stdout:(14/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 913 kB/s | 160 kB 00:00 2024-08-19T14:03:06.829 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-08-19T14:03:06.858 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/26 2024-08-19T14:03:06.890 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-08-19T14:03:06.904 INFO:teuthology.orchestra.run.smithi045.stdout:(15/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.6 MB/s | 292 kB 00:00 2024-08-19T14:03:07.004 INFO:teuthology.orchestra.run.smithi045.stdout:(16/24): python3-prettytable-0.7.2-27.el9.noarc 423 kB/s | 42 kB 00:00 2024-08-19T14:03:07.038 INFO:teuthology.orchestra.run.smithi045.stdout:(17/24): librdkafka-1.6.1-102.el9.x86_64.rpm 2.4 MB/s | 662 kB 00:00 2024-08-19T14:03:07.088 INFO:teuthology.orchestra.run.smithi045.stdout:(18/24): gperftools-libs-2.9.1-2.el9.x86_64.rpm 3.6 MB/s | 309 kB 00:00 2024-08-19T14:03:07.113 INFO:teuthology.orchestra.run.smithi045.stdout:(19/24): liboath-2.6.7-2.el9.x86_64.rpm 666 kB/s | 49 kB 00:00 2024-08-19T14:03:07.139 INFO:teuthology.orchestra.run.smithi045.stdout:(20/24): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-08-19T14:03:07.196 INFO:teuthology.orchestra.run.smithi031.stdout:(5/24): ceph-common-17.2.7-1406.gdf62bdcc.el9.x 16 MB/s | 22 MB 00:01 2024-08-19T14:03:07.199 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-08-19T14:03:07.214 INFO:teuthology.orchestra.run.smithi045.stdout:(21/24): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-08-19T14:03:07.236 INFO:teuthology.orchestra.run.smithi032.stdout: Upgrading : librbd1-2:17.2.7-1406.gdf62bdcc.el9.x86_64 18/26 2024-08-19T14:03:07.272 INFO:teuthology.orchestra.run.smithi031.stdout:(6/24): librgw2-17.2.7-1406.gdf62bdcc.el9.x86_6 8.4 MB/s | 4.9 MB 00:00 2024-08-19T14:03:07.297 INFO:teuthology.orchestra.run.smithi031.stdout:(7/24): python3-ceph-argparse-17.2.7-1406.gdf62 451 kB/s | 45 kB 00:00 2024-08-19T14:03:07.322 INFO:teuthology.orchestra.run.smithi031.stdout:(8/24): python3-ceph-common-17.2.7-1406.gdf62bd 2.3 MB/s | 120 kB 00:00 2024-08-19T14:03:07.331 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: librbd1-2:17.2.7-1406.gdf62bdcc.el9.x86_64 18/26 2024-08-19T14:03:07.332 INFO:teuthology.orchestra.run.smithi045.stdout:(22/24): librados2-17.2.7-1406.gdf62bdcc.el9.x8 16 MB/s | 3.1 MB 00:00 2024-08-19T14:03:07.347 INFO:teuthology.orchestra.run.smithi031.stdout:(9/24): python3-cephfs-17.2.7-1406.gdf62bdcc.el 3.5 MB/s | 182 kB 00:00 2024-08-19T14:03:07.372 INFO:teuthology.orchestra.run.smithi031.stdout:(10/24): python3-rados-17.2.7-1406.gdf62bdcc.el 7.1 MB/s | 367 kB 00:00 2024-08-19T14:03:07.398 INFO:teuthology.orchestra.run.smithi031.stdout:(11/24): python3-rbd-17.2.7-1406.gdf62bdcc.el9. 6.7 MB/s | 343 kB 00:00 2024-08-19T14:03:07.403 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-rbd-2:17.2.7-1406.gdf62bdcc.el9.x86_64 19/26 2024-08-19T14:03:07.423 INFO:teuthology.orchestra.run.smithi031.stdout:(12/24): python3-rgw-17.2.7-1406.gdf62bdcc.el9. 2.3 MB/s | 116 kB 00:00 2024-08-19T14:03:07.430 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-ceph-common-2:17.2.7-1406.gdf62bdcc.el9.x8 20/26 2024-08-19T14:03:07.506 INFO:teuthology.orchestra.run.smithi045.stdout:(23/24): ceph-radosgw-17.2.7-1406.gdf62bdcc.el9 6.0 MB/s | 14 MB 00:02 2024-08-19T14:03:07.565 INFO:teuthology.orchestra.run.smithi045.stdout:(24/24): librbd1-17.2.7-1406.gdf62bdcc.el9.x86_ 8.4 MB/s | 2.9 MB 00:00 2024-08-19T14:03:07.565 INFO:teuthology.orchestra.run.smithi031.stdout:(13/24): ceph-radosgw-17.2.7-1406.gdf62bdcc.el9 8.1 MB/s | 14 MB 00:01 2024-08-19T14:03:07.570 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-08-19T14:03:07.571 INFO:teuthology.orchestra.run.smithi045.stdout:Total 22 MB/s | 58 MB 00:02 2024-08-19T14:03:07.636 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-08-19T14:03:07.673 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-08-19T14:03:07.674 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-08-19T14:03:07.681 INFO:teuthology.orchestra.run.smithi031.stdout:(14/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 175 kB/s | 45 kB 00:00 2024-08-19T14:03:07.732 INFO:teuthology.orchestra.run.smithi031.stdout:(15/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 480 kB/s | 160 kB 00:00 2024-08-19T14:03:07.799 INFO:teuthology.orchestra.run.smithi031.stdout:(16/24): python3-prettytable-0.7.2-27.el9.noarc 634 kB/s | 42 kB 00:00 2024-08-19T14:03:07.824 INFO:teuthology.orchestra.run.smithi031.stdout:(17/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.0 MB/s | 292 kB 00:00 2024-08-19T14:03:07.875 INFO:teuthology.orchestra.run.smithi031.stdout:(18/24): gperftools-libs-2.9.1-2.el9.x86_64.rpm 4.0 MB/s | 309 kB 00:00 2024-08-19T14:03:07.891 INFO:teuthology.orchestra.run.smithi031.stdout:(19/24): liboath-2.6.7-2.el9.x86_64.rpm 826 kB/s | 49 kB 00:00 2024-08-19T14:03:07.925 INFO:teuthology.orchestra.run.smithi031.stdout:(20/24): librdkafka-1.6.1-102.el9.x86_64.rpm 1.8 MB/s | 662 kB 00:00 2024-08-19T14:03:07.950 INFO:teuthology.orchestra.run.smithi031.stdout:(21/24): libunwind-1.6.2-1.el9.x86_64.rpm 893 kB/s | 67 kB 00:00 2024-08-19T14:03:08.025 INFO:teuthology.orchestra.run.smithi031.stdout:(22/24): thrift-0.15.0-2.el9.x86_64.rpm 12 MB/s | 1.6 MB 00:00 2024-08-19T14:03:08.134 INFO:teuthology.orchestra.run.smithi031.stdout:(23/24): librados2-17.2.7-1406.gdf62bdcc.el9.x8 15 MB/s | 3.1 MB 00:00 2024-08-19T14:03:08.136 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-08-19T14:03:08.137 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-08-19T14:03:08.184 INFO:teuthology.orchestra.run.smithi031.stdout:(24/24): librbd1-17.2.7-1406.gdf62bdcc.el9.x86_ 13 MB/s | 2.9 MB 00:00 2024-08-19T14:03:08.189 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-08-19T14:03:08.189 INFO:teuthology.orchestra.run.smithi031.stdout:Total 23 MB/s | 58 MB 00:02 2024-08-19T14:03:08.254 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-08-19T14:03:08.290 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-08-19T14:03:08.290 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-08-19T14:03:08.753 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-08-19T14:03:08.753 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-08-19T14:03:09.407 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-08-19T14:03:09.803 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-08-19T14:03:09.867 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : librados2-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2/26 2024-08-19T14:03:09.963 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librados2-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2/26 2024-08-19T14:03:10.008 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-08-19T14:03:10.048 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-rados-2:17.2.7-1406.gdf62bdcc.el9.x86_64 3/26 2024-08-19T14:03:10.056 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-common-2:17.2.7-1406.gdf62bdcc.el9.x86_64 21/26 2024-08-19T14:03:10.077 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libcephfs2-2:17.2.7-1406.gdf62bdcc.el9.x86_64 4/26 2024-08-19T14:03:10.094 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-common-2:17.2.7-1406.gdf62bdcc.el9.x86_64 21/26 2024-08-19T14:03:10.164 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: libcephfs2-2:17.2.7-1406.gdf62bdcc.el9.x86_64 4/26 2024-08-19T14:03:10.211 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-08-19T14:03:10.239 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-08-19T14:03:10.404 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-08-19T14:03:10.470 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : librados2-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2/26 2024-08-19T14:03:10.566 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librados2-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2/26 2024-08-19T14:03:10.653 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-rados-2:17.2.7-1406.gdf62bdcc.el9.x86_64 3/26 2024-08-19T14:03:10.663 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-common-2:17.2.7-1406.gdf62bdcc.el9.x86_64 21/26 2024-08-19T14:03:10.680 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libcephfs2-2:17.2.7-1406.gdf62bdcc.el9.x86_64 4/26 2024-08-19T14:03:10.698 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-base-2:17.2.7-1406.gdf62bdcc.el9.x86_64 22/26 2024-08-19T14:03:10.702 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-08-19T14:03:10.729 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : librgw2-2:17.2.7-1406.gdf62bdcc.el9.x86_64 8/26 2024-08-19T14:03:10.768 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: libcephfs2-2:17.2.7-1406.gdf62bdcc.el9.x86_64 4/26 2024-08-19T14:03:10.789 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-base-2:17.2.7-1406.gdf62bdcc.el9.x86_64 22/26 2024-08-19T14:03:10.789 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-08-19T14:03:10.789 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-08-19T14:03:10.790 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-19T14:03:10.791 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librgw2-2:17.2.7-1406.gdf62bdcc.el9.x86_64 8/26 2024-08-19T14:03:10.814 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-08-19T14:03:10.821 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-08-19T14:03:10.824 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-selinux-2:17.2.7-1406.gdf62bdcc.el9.x86_64 23/26 2024-08-19T14:03:10.843 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-08-19T14:03:10.867 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-ceph-argparse-2:17.2.7-1406.gdf62bdcc.el9. 10/26 2024-08-19T14:03:10.903 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cephfs-2:17.2.7-1406.gdf62bdcc.el9.x86_64 11/26 2024-08-19T14:03:10.982 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-rgw-2:17.2.7-1406.gdf62bdcc.el9.x86_64 12/26 2024-08-19T14:03:11.007 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libradosstriper1-2:17.2.7-1406.gdf62bdcc.el9.x86_6 13/26 2024-08-19T14:03:11.055 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: libradosstriper1-2:17.2.7-1406.gdf62bdcc.el9.x86_6 13/26 2024-08-19T14:03:11.095 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-08-19T14:03:11.124 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/26 2024-08-19T14:03:11.157 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-08-19T14:03:11.296 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-08-19T14:03:11.332 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : librgw2-2:17.2.7-1406.gdf62bdcc.el9.x86_64 8/26 2024-08-19T14:03:11.395 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librgw2-2:17.2.7-1406.gdf62bdcc.el9.x86_64 8/26 2024-08-19T14:03:11.424 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-08-19T14:03:11.466 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-08-19T14:03:11.469 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-ceph-argparse-2:17.2.7-1406.gdf62bdcc.el9. 10/26 2024-08-19T14:03:11.511 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : librbd1-2:17.2.7-1406.gdf62bdcc.el9.x86_64 18/26 2024-08-19T14:03:11.521 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cephfs-2:17.2.7-1406.gdf62bdcc.el9.x86_64 11/26 2024-08-19T14:03:11.600 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-rgw-2:17.2.7-1406.gdf62bdcc.el9.x86_64 12/26 2024-08-19T14:03:11.613 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librbd1-2:17.2.7-1406.gdf62bdcc.el9.x86_64 18/26 2024-08-19T14:03:11.626 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libradosstriper1-2:17.2.7-1406.gdf62bdcc.el9.x86_6 13/26 2024-08-19T14:03:11.675 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: libradosstriper1-2:17.2.7-1406.gdf62bdcc.el9.x86_6 13/26 2024-08-19T14:03:11.686 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-rbd-2:17.2.7-1406.gdf62bdcc.el9.x86_64 19/26 2024-08-19T14:03:11.713 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-ceph-common-2:17.2.7-1406.gdf62bdcc.el9.x8 20/26 2024-08-19T14:03:11.714 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-08-19T14:03:11.743 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/26 2024-08-19T14:03:11.776 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-08-19T14:03:12.084 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-08-19T14:03:12.130 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : librbd1-2:17.2.7-1406.gdf62bdcc.el9.x86_64 18/26 2024-08-19T14:03:12.225 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librbd1-2:17.2.7-1406.gdf62bdcc.el9.x86_64 18/26 2024-08-19T14:03:12.297 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-rbd-2:17.2.7-1406.gdf62bdcc.el9.x86_64 19/26 2024-08-19T14:03:12.324 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-ceph-common-2:17.2.7-1406.gdf62bdcc.el9.x8 20/26 2024-08-19T14:03:14.377 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-common-2:17.2.7-1406.gdf62bdcc.el9.x86_64 21/26 2024-08-19T14:03:14.418 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-common-2:17.2.7-1406.gdf62bdcc.el9.x86_64 21/26 2024-08-19T14:03:14.955 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-common-2:17.2.7-1406.gdf62bdcc.el9.x86_64 21/26 2024-08-19T14:03:14.987 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-base-2:17.2.7-1406.gdf62bdcc.el9.x86_64 22/26 2024-08-19T14:03:15.041 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-common-2:17.2.7-1406.gdf62bdcc.el9.x86_64 21/26 2024-08-19T14:03:15.082 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-base-2:17.2.7-1406.gdf62bdcc.el9.x86_64 22/26 2024-08-19T14:03:15.082 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-08-19T14:03:15.082 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-08-19T14:03:15.083 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-08-19T14:03:15.087 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-common-2:17.2.7-1406.gdf62bdcc.el9.x86_64 21/26 2024-08-19T14:03:15.115 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-selinux-2:17.2.7-1406.gdf62bdcc.el9.x86_64 23/26 2024-08-19T14:03:15.623 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-common-2:17.2.7-1406.gdf62bdcc.el9.x86_64 21/26 2024-08-19T14:03:15.657 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-base-2:17.2.7-1406.gdf62bdcc.el9.x86_64 22/26 2024-08-19T14:03:15.749 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-base-2:17.2.7-1406.gdf62bdcc.el9.x86_64 22/26 2024-08-19T14:03:15.749 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-08-19T14:03:15.749 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-08-19T14:03:15.749 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:03:15.783 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-selinux-2:17.2.7-1406.gdf62bdcc.el9.x86_64 23/26 2024-08-19T14:03:20.739 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-selinux-2:17.2.7-1406.gdf62bdcc.el9.x86_64 23/26 2024-08-19T14:03:20.739 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /sys 2024-08-19T14:03:20.740 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /proc 2024-08-19T14:03:20.740 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /mnt 2024-08-19T14:03:20.740 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /var/tmp 2024-08-19T14:03:20.740 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /home 2024-08-19T14:03:20.740 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /root 2024-08-19T14:03:20.740 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /tmp 2024-08-19T14:03:20.740 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-19T14:03:22.092 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-radosgw-2:17.2.7-1406.gdf62bdcc.el9.x86_64 24/26 2024-08-19T14:03:22.153 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-1406.gdf62bdcc.el9.x86_64 24/26 2024-08-19T14:03:22.154 INFO:teuthology.orchestra.run.smithi032.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-19T14:03:22.154 INFO:teuthology.orchestra.run.smithi032.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-08-19T14:03:22.154 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-08-19T14:03:22.154 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-08-19T14:03:22.154 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-19T14:03:22.155 INFO:teuthology.orchestra.run.smithi032.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-08-19T14:03:22.246 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-08-19T14:03:22.246 INFO:teuthology.orchestra.run.smithi032.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-08-19T14:03:23.613 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-08-19T14:03:23.613 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-base-2:17.2.7-1406.gdf62bdcc.el9.x86_64 1/26 2024-08-19T14:03:23.613 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-common-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2/26 2024-08-19T14:03:23.613 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-radosgw-2:17.2.7-1406.gdf62bdcc.el9.x86_64 3/26 2024-08-19T14:03:23.613 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-selinux-2:17.2.7-1406.gdf62bdcc.el9.x86_64 4/26 2024-08-19T14:03:23.614 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libcephfs2-2:17.2.7-1406.gdf62bdcc.el9.x86_64 5/26 2024-08-19T14:03:23.614 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libradosstriper1-2:17.2.7-1406.gdf62bdcc.el9.x86_6 6/26 2024-08-19T14:03:23.614 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librgw2-2:17.2.7-1406.gdf62bdcc.el9.x86_64 7/26 2024-08-19T14:03:23.614 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-ceph-argparse-2:17.2.7-1406.gdf62bdcc.el9. 8/26 2024-08-19T14:03:23.614 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-ceph-common-2:17.2.7-1406.gdf62bdcc.el9.x8 9/26 2024-08-19T14:03:23.614 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-cephfs-2:17.2.7-1406.gdf62bdcc.el9.x86_64 10/26 2024-08-19T14:03:23.614 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-rados-2:17.2.7-1406.gdf62bdcc.el9.x86_64 11/26 2024-08-19T14:03:23.614 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-rbd-2:17.2.7-1406.gdf62bdcc.el9.x86_64 12/26 2024-08-19T14:03:23.614 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-rgw-2:17.2.7-1406.gdf62bdcc.el9.x86_64 13/26 2024-08-19T14:03:23.614 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-08-19T14:03:23.614 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-08-19T14:03:23.614 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-08-19T14:03:23.615 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-08-19T14:03:23.615 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-08-19T14:03:23.615 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 19/26 2024-08-19T14:03:23.615 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-08-19T14:03:23.615 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-08-19T14:03:23.615 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-08-19T14:03:23.615 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librados2-2:17.2.7-1406.gdf62bdcc.el9.x86_64 23/26 2024-08-19T14:03:23.615 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-08-19T14:03:23.615 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librbd1-2:17.2.7-1406.gdf62bdcc.el9.x86_64 25/26 2024-08-19T14:03:24.361 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-08-19T14:03:24.361 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-19T14:03:24.361 INFO:teuthology.orchestra.run.smithi032.stdout:Upgraded: 2024-08-19T14:03:24.361 INFO:teuthology.orchestra.run.smithi032.stdout: librados2-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2024-08-19T14:03:24.361 INFO:teuthology.orchestra.run.smithi032.stdout: librbd1-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2024-08-19T14:03:24.361 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-08-19T14:03:24.361 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-base-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2024-08-19T14:03:24.361 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-common-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2024-08-19T14:03:24.362 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-radosgw-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2024-08-19T14:03:24.362 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-selinux-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2024-08-19T14:03:24.362 INFO:teuthology.orchestra.run.smithi032.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-08-19T14:03:24.362 INFO:teuthology.orchestra.run.smithi032.stdout: libcephfs2-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2024-08-19T14:03:24.362 INFO:teuthology.orchestra.run.smithi032.stdout: liboath-2.6.7-2.el9.x86_64 2024-08-19T14:03:24.362 INFO:teuthology.orchestra.run.smithi032.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-08-19T14:03:24.362 INFO:teuthology.orchestra.run.smithi032.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-08-19T14:03:24.362 INFO:teuthology.orchestra.run.smithi032.stdout: libradosstriper1-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2024-08-19T14:03:24.362 INFO:teuthology.orchestra.run.smithi032.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-08-19T14:03:24.362 INFO:teuthology.orchestra.run.smithi032.stdout: librgw2-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2024-08-19T14:03:24.362 INFO:teuthology.orchestra.run.smithi032.stdout: libunwind-1.6.2-1.el9.x86_64 2024-08-19T14:03:24.362 INFO:teuthology.orchestra.run.smithi032.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-08-19T14:03:24.363 INFO:teuthology.orchestra.run.smithi032.stdout: python3-ceph-argparse-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2024-08-19T14:03:24.363 INFO:teuthology.orchestra.run.smithi032.stdout: python3-ceph-common-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2024-08-19T14:03:24.363 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cephfs-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2024-08-19T14:03:24.363 INFO:teuthology.orchestra.run.smithi032.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-08-19T14:03:24.363 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rados-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2024-08-19T14:03:24.363 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rbd-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2024-08-19T14:03:24.363 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rgw-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2024-08-19T14:03:24.363 INFO:teuthology.orchestra.run.smithi032.stdout: thrift-0.15.0-2.el9.x86_64 2024-08-19T14:03:24.363 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-19T14:03:24.363 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-08-19T14:03:24.639 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph-test 2024-08-19T14:03:24.884 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-selinux-2:17.2.7-1406.gdf62bdcc.el9.x86_64 23/26 2024-08-19T14:03:24.885 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /sys 2024-08-19T14:03:24.885 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /proc 2024-08-19T14:03:24.885 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /mnt 2024-08-19T14:03:24.885 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /var/tmp 2024-08-19T14:03:24.885 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /home 2024-08-19T14:03:24.885 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /root 2024-08-19T14:03:24.885 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /tmp 2024-08-19T14:03:24.885 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-08-19T14:03:25.212 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:00:26 ago on Mon 19 Aug 2024 02:02:59 PM UTC. 2024-08-19T14:03:25.350 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-08-19T14:03:25.351 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-08-19T14:03:25.351 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repository Size 2024-08-19T14:03:25.351 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-08-19T14:03:25.351 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-08-19T14:03:25.351 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-test x86_64 2:17.2.7-1406.gdf62bdcc.el9 ceph 48 M 2024-08-19T14:03:25.351 INFO:teuthology.orchestra.run.smithi032.stdout:Installing dependencies: 2024-08-19T14:03:25.351 INFO:teuthology.orchestra.run.smithi032.stdout: libcephsqlite x86_64 2:17.2.7-1406.gdf62bdcc.el9 ceph 163 k 2024-08-19T14:03:25.351 INFO:teuthology.orchestra.run.smithi032.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-08-19T14:03:25.351 INFO:teuthology.orchestra.run.smithi032.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-08-19T14:03:25.351 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-19T14:03:25.351 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-08-19T14:03:25.351 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-08-19T14:03:25.352 INFO:teuthology.orchestra.run.smithi032.stdout:Install 4 Packages 2024-08-19T14:03:25.352 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-19T14:03:25.352 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 48 M 2024-08-19T14:03:25.352 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 193 M 2024-08-19T14:03:25.352 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-08-19T14:03:25.542 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-selinux-2:17.2.7-1406.gdf62bdcc.el9.x86_64 23/26 2024-08-19T14:03:25.542 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /sys 2024-08-19T14:03:25.542 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /proc 2024-08-19T14:03:25.542 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /mnt 2024-08-19T14:03:25.542 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /var/tmp 2024-08-19T14:03:25.542 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /home 2024-08-19T14:03:25.542 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /root 2024-08-19T14:03:25.542 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /tmp 2024-08-19T14:03:25.542 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:03:25.843 INFO:teuthology.orchestra.run.smithi032.stdout:(1/4): libcephsqlite-17.2.7-1406.gdf62bdcc.el9. 697 kB/s | 163 kB 00:00 2024-08-19T14:03:25.885 INFO:teuthology.orchestra.run.smithi032.stdout:(2/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.1 MB/s | 304 kB 00:00 2024-08-19T14:03:26.003 INFO:teuthology.orchestra.run.smithi032.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 398 kB/s | 64 kB 00:00 2024-08-19T14:03:26.253 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-radosgw-2:17.2.7-1406.gdf62bdcc.el9.x86_64 24/26 2024-08-19T14:03:26.332 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-1406.gdf62bdcc.el9.x86_64 24/26 2024-08-19T14:03:26.332 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-19T14:03:26.332 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-08-19T14:03:26.332 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-08-19T14:03:26.332 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-08-19T14:03:26.332 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-08-19T14:03:26.334 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-08-19T14:03:26.406 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-08-19T14:03:26.407 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-08-19T14:03:26.880 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-radosgw-2:17.2.7-1406.gdf62bdcc.el9.x86_64 24/26 2024-08-19T14:03:26.941 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-1406.gdf62bdcc.el9.x86_64 24/26 2024-08-19T14:03:26.941 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-19T14:03:26.941 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-08-19T14:03:26.941 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-08-19T14:03:26.941 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-08-19T14:03:26.942 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:03:26.943 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-08-19T14:03:27.032 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-08-19T14:03:27.033 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-08-19T14:03:27.179 INFO:teuthology.orchestra.run.smithi032.stdout:(4/4): ceph-test-17.2.7-1406.gdf62bdcc.el9.x86_ 31 MB/s | 48 MB 00:01 2024-08-19T14:03:27.180 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-08-19T14:03:27.181 INFO:teuthology.orchestra.run.smithi032.stdout:Total 26 MB/s | 48 MB 00:01 2024-08-19T14:03:27.209 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-08-19T14:03:27.230 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-08-19T14:03:27.230 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-08-19T14:03:27.612 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-08-19T14:03:27.612 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-08-19T14:03:27.700 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-08-19T14:03:27.700 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-base-2:17.2.7-1406.gdf62bdcc.el9.x86_64 1/26 2024-08-19T14:03:27.700 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-common-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2/26 2024-08-19T14:03:27.700 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-radosgw-2:17.2.7-1406.gdf62bdcc.el9.x86_64 3/26 2024-08-19T14:03:27.700 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-selinux-2:17.2.7-1406.gdf62bdcc.el9.x86_64 4/26 2024-08-19T14:03:27.700 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libcephfs2-2:17.2.7-1406.gdf62bdcc.el9.x86_64 5/26 2024-08-19T14:03:27.700 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libradosstriper1-2:17.2.7-1406.gdf62bdcc.el9.x86_6 6/26 2024-08-19T14:03:27.700 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librgw2-2:17.2.7-1406.gdf62bdcc.el9.x86_64 7/26 2024-08-19T14:03:27.700 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-ceph-argparse-2:17.2.7-1406.gdf62bdcc.el9. 8/26 2024-08-19T14:03:27.700 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-ceph-common-2:17.2.7-1406.gdf62bdcc.el9.x8 9/26 2024-08-19T14:03:27.700 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cephfs-2:17.2.7-1406.gdf62bdcc.el9.x86_64 10/26 2024-08-19T14:03:27.700 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-rados-2:17.2.7-1406.gdf62bdcc.el9.x86_64 11/26 2024-08-19T14:03:27.701 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-rbd-2:17.2.7-1406.gdf62bdcc.el9.x86_64 12/26 2024-08-19T14:03:27.701 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-rgw-2:17.2.7-1406.gdf62bdcc.el9.x86_64 13/26 2024-08-19T14:03:27.701 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-08-19T14:03:27.701 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-08-19T14:03:27.701 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-08-19T14:03:27.701 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-08-19T14:03:27.701 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-08-19T14:03:27.701 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 19/26 2024-08-19T14:03:27.701 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-08-19T14:03:27.701 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-08-19T14:03:27.701 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-08-19T14:03:27.701 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librados2-2:17.2.7-1406.gdf62bdcc.el9.x86_64 23/26 2024-08-19T14:03:27.702 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-08-19T14:03:27.702 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librbd1-2:17.2.7-1406.gdf62bdcc.el9.x86_64 25/26 2024-08-19T14:03:28.167 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-08-19T14:03:28.255 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-08-19T14:03:28.295 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-08-19T14:03:28.322 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libcephsqlite-2:17.2.7-1406.gdf62bdcc.el9.x86_64 3/4 2024-08-19T14:03:28.345 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-08-19T14:03:28.346 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-base-2:17.2.7-1406.gdf62bdcc.el9.x86_64 1/26 2024-08-19T14:03:28.346 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-common-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2/26 2024-08-19T14:03:28.346 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-radosgw-2:17.2.7-1406.gdf62bdcc.el9.x86_64 3/26 2024-08-19T14:03:28.346 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-selinux-2:17.2.7-1406.gdf62bdcc.el9.x86_64 4/26 2024-08-19T14:03:28.346 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libcephfs2-2:17.2.7-1406.gdf62bdcc.el9.x86_64 5/26 2024-08-19T14:03:28.346 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libradosstriper1-2:17.2.7-1406.gdf62bdcc.el9.x86_6 6/26 2024-08-19T14:03:28.346 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librgw2-2:17.2.7-1406.gdf62bdcc.el9.x86_64 7/26 2024-08-19T14:03:28.346 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-ceph-argparse-2:17.2.7-1406.gdf62bdcc.el9. 8/26 2024-08-19T14:03:28.346 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-ceph-common-2:17.2.7-1406.gdf62bdcc.el9.x8 9/26 2024-08-19T14:03:28.346 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cephfs-2:17.2.7-1406.gdf62bdcc.el9.x86_64 10/26 2024-08-19T14:03:28.347 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-rados-2:17.2.7-1406.gdf62bdcc.el9.x86_64 11/26 2024-08-19T14:03:28.347 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-rbd-2:17.2.7-1406.gdf62bdcc.el9.x86_64 12/26 2024-08-19T14:03:28.347 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-rgw-2:17.2.7-1406.gdf62bdcc.el9.x86_64 13/26 2024-08-19T14:03:28.347 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-08-19T14:03:28.347 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-08-19T14:03:28.347 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-08-19T14:03:28.347 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-08-19T14:03:28.347 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-08-19T14:03:28.347 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 19/26 2024-08-19T14:03:28.347 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-08-19T14:03:28.347 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-08-19T14:03:28.347 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-08-19T14:03:28.347 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librados2-2:17.2.7-1406.gdf62bdcc.el9.x86_64 23/26 2024-08-19T14:03:28.348 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-08-19T14:03:28.348 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librbd1-2:17.2.7-1406.gdf62bdcc.el9.x86_64 25/26 2024-08-19T14:03:28.417 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-08-19T14:03:28.417 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-08-19T14:03:28.418 INFO:teuthology.orchestra.run.smithi045.stdout:Upgraded: 2024-08-19T14:03:28.418 INFO:teuthology.orchestra.run.smithi045.stdout: librados2-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2024-08-19T14:03:28.418 INFO:teuthology.orchestra.run.smithi045.stdout: librbd1-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2024-08-19T14:03:28.418 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-08-19T14:03:28.418 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-base-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2024-08-19T14:03:28.418 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-common-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2024-08-19T14:03:28.418 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-radosgw-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2024-08-19T14:03:28.418 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-selinux-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2024-08-19T14:03:28.418 INFO:teuthology.orchestra.run.smithi045.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-08-19T14:03:28.418 INFO:teuthology.orchestra.run.smithi045.stdout: libcephfs2-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2024-08-19T14:03:28.418 INFO:teuthology.orchestra.run.smithi045.stdout: liboath-2.6.7-2.el9.x86_64 2024-08-19T14:03:28.418 INFO:teuthology.orchestra.run.smithi045.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-08-19T14:03:28.419 INFO:teuthology.orchestra.run.smithi045.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-08-19T14:03:28.419 INFO:teuthology.orchestra.run.smithi045.stdout: libradosstriper1-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2024-08-19T14:03:28.419 INFO:teuthology.orchestra.run.smithi045.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-08-19T14:03:28.419 INFO:teuthology.orchestra.run.smithi045.stdout: librgw2-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2024-08-19T14:03:28.419 INFO:teuthology.orchestra.run.smithi045.stdout: libunwind-1.6.2-1.el9.x86_64 2024-08-19T14:03:28.419 INFO:teuthology.orchestra.run.smithi045.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-08-19T14:03:28.419 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ceph-argparse-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2024-08-19T14:03:28.419 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ceph-common-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2024-08-19T14:03:28.419 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cephfs-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2024-08-19T14:03:28.419 INFO:teuthology.orchestra.run.smithi045.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-08-19T14:03:28.419 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rados-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2024-08-19T14:03:28.419 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rbd-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2024-08-19T14:03:28.419 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rgw-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2024-08-19T14:03:28.420 INFO:teuthology.orchestra.run.smithi045.stdout: thrift-0.15.0-2.el9.x86_64 2024-08-19T14:03:28.420 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-08-19T14:03:28.420 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-08-19T14:03:28.696 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-test 2024-08-19T14:03:29.022 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-08-19T14:03:29.023 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:03:29.023 INFO:teuthology.orchestra.run.smithi031.stdout:Upgraded: 2024-08-19T14:03:29.023 INFO:teuthology.orchestra.run.smithi031.stdout: librados2-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2024-08-19T14:03:29.023 INFO:teuthology.orchestra.run.smithi031.stdout: librbd1-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2024-08-19T14:03:29.023 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-08-19T14:03:29.023 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-base-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2024-08-19T14:03:29.023 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-common-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2024-08-19T14:03:29.023 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-radosgw-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2024-08-19T14:03:29.023 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-selinux-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2024-08-19T14:03:29.023 INFO:teuthology.orchestra.run.smithi031.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-08-19T14:03:29.024 INFO:teuthology.orchestra.run.smithi031.stdout: libcephfs2-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2024-08-19T14:03:29.024 INFO:teuthology.orchestra.run.smithi031.stdout: liboath-2.6.7-2.el9.x86_64 2024-08-19T14:03:29.024 INFO:teuthology.orchestra.run.smithi031.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-08-19T14:03:29.024 INFO:teuthology.orchestra.run.smithi031.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-08-19T14:03:29.024 INFO:teuthology.orchestra.run.smithi031.stdout: libradosstriper1-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2024-08-19T14:03:29.024 INFO:teuthology.orchestra.run.smithi031.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-08-19T14:03:29.024 INFO:teuthology.orchestra.run.smithi031.stdout: librgw2-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2024-08-19T14:03:29.024 INFO:teuthology.orchestra.run.smithi031.stdout: libunwind-1.6.2-1.el9.x86_64 2024-08-19T14:03:29.024 INFO:teuthology.orchestra.run.smithi031.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-08-19T14:03:29.024 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ceph-argparse-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2024-08-19T14:03:29.024 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ceph-common-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2024-08-19T14:03:29.024 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cephfs-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2024-08-19T14:03:29.024 INFO:teuthology.orchestra.run.smithi031.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-08-19T14:03:29.025 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rados-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2024-08-19T14:03:29.025 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rbd-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2024-08-19T14:03:29.025 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rgw-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2024-08-19T14:03:29.025 INFO:teuthology.orchestra.run.smithi031.stdout: thrift-0.15.0-2.el9.x86_64 2024-08-19T14:03:29.025 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:03:29.025 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-08-19T14:03:29.303 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:27 ago on Mon 19 Aug 2024 02:03:02 PM UTC. 2024-08-19T14:03:29.437 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-08-19T14:03:29.438 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-08-19T14:03:29.438 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2024-08-19T14:03:29.438 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-08-19T14:03:29.438 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-08-19T14:03:29.438 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-test x86_64 2:17.2.7-1406.gdf62bdcc.el9 ceph 48 M 2024-08-19T14:03:29.438 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2024-08-19T14:03:29.438 INFO:teuthology.orchestra.run.smithi045.stdout: libcephsqlite x86_64 2:17.2.7-1406.gdf62bdcc.el9 ceph 163 k 2024-08-19T14:03:29.438 INFO:teuthology.orchestra.run.smithi045.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-08-19T14:03:29.438 INFO:teuthology.orchestra.run.smithi045.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-08-19T14:03:29.439 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-08-19T14:03:29.439 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-08-19T14:03:29.439 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-08-19T14:03:29.439 INFO:teuthology.orchestra.run.smithi045.stdout:Install 4 Packages 2024-08-19T14:03:29.439 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-08-19T14:03:29.439 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 48 M 2024-08-19T14:03:29.439 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 193 M 2024-08-19T14:03:29.440 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-08-19T14:03:29.474 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-test 2024-08-19T14:03:29.874 INFO:teuthology.orchestra.run.smithi045.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.7 MB/s | 304 kB 00:00 2024-08-19T14:03:29.933 INFO:teuthology.orchestra.run.smithi045.stdout:(2/4): libcephsqlite-17.2.7-1406.gdf62bdcc.el9. 687 kB/s | 163 kB 00:00 2024-08-19T14:03:29.975 INFO:teuthology.orchestra.run.smithi045.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 634 kB/s | 64 kB 00:00 2024-08-19T14:03:30.028 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:00:27 ago on Mon 19 Aug 2024 02:03:03 PM UTC. 2024-08-19T14:03:30.168 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-08-19T14:03:30.169 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-08-19T14:03:30.169 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2024-08-19T14:03:30.169 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-08-19T14:03:30.169 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-08-19T14:03:30.169 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-test x86_64 2:17.2.7-1406.gdf62bdcc.el9 ceph 48 M 2024-08-19T14:03:30.169 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2024-08-19T14:03:30.169 INFO:teuthology.orchestra.run.smithi031.stdout: libcephsqlite x86_64 2:17.2.7-1406.gdf62bdcc.el9 ceph 163 k 2024-08-19T14:03:30.169 INFO:teuthology.orchestra.run.smithi031.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-08-19T14:03:30.169 INFO:teuthology.orchestra.run.smithi031.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-08-19T14:03:30.169 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:03:30.170 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-08-19T14:03:30.170 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-08-19T14:03:30.170 INFO:teuthology.orchestra.run.smithi031.stdout:Install 4 Packages 2024-08-19T14:03:30.170 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:03:30.170 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 48 M 2024-08-19T14:03:30.171 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 193 M 2024-08-19T14:03:30.171 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-08-19T14:03:30.727 INFO:teuthology.orchestra.run.smithi031.stdout:(1/4): libcephsqlite-17.2.7-1406.gdf62bdcc.el9. 359 kB/s | 163 kB 00:00 2024-08-19T14:03:30.904 INFO:teuthology.orchestra.run.smithi031.stdout:(2/4): socat-1.7.4.1-6.el9.x86_64.rpm 482 kB/s | 304 kB 00:00 2024-08-19T14:03:31.080 INFO:teuthology.orchestra.run.smithi031.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 180 kB/s | 64 kB 00:00 2024-08-19T14:03:31.283 INFO:teuthology.orchestra.run.smithi045.stdout:(4/4): ceph-test-17.2.7-1406.gdf62bdcc.el9.x86_ 30 MB/s | 48 MB 00:01 2024-08-19T14:03:31.285 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-08-19T14:03:31.286 INFO:teuthology.orchestra.run.smithi045.stdout:Total 26 MB/s | 48 MB 00:01 2024-08-19T14:03:31.313 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-08-19T14:03:31.334 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-08-19T14:03:31.334 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-08-19T14:03:31.719 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-08-19T14:03:31.720 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-08-19T14:03:32.261 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-08-19T14:03:32.311 INFO:teuthology.orchestra.run.smithi031.stdout:(4/4): ceph-test-17.2.7-1406.gdf62bdcc.el9.x86_ 24 MB/s | 48 MB 00:02 2024-08-19T14:03:32.313 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-08-19T14:03:32.314 INFO:teuthology.orchestra.run.smithi031.stdout:Total 23 MB/s | 48 MB 00:02 2024-08-19T14:03:32.341 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-08-19T14:03:32.342 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-08-19T14:03:32.362 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-08-19T14:03:32.362 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-08-19T14:03:32.383 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-08-19T14:03:32.410 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libcephsqlite-2:17.2.7-1406.gdf62bdcc.el9.x86_64 3/4 2024-08-19T14:03:32.745 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-08-19T14:03:32.745 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-08-19T14:03:32.951 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: libcephsqlite-2:17.2.7-1406.gdf62bdcc.el9.x86_64 3/4 2024-08-19T14:03:33.021 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-test-2:17.2.7-1406.gdf62bdcc.el9.x86_64 4/4 2024-08-19T14:03:33.294 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-08-19T14:03:33.379 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-08-19T14:03:33.419 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-08-19T14:03:33.446 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libcephsqlite-2:17.2.7-1406.gdf62bdcc.el9.x86_64 3/4 2024-08-19T14:03:35.018 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-test-2:17.2.7-1406.gdf62bdcc.el9.x86_64 4/4 2024-08-19T14:03:35.018 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-test-2:17.2.7-1406.gdf62bdcc.el9.x86_64 1/4 2024-08-19T14:03:35.018 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libcephsqlite-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2/4 2024-08-19T14:03:35.019 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-08-19T14:03:35.320 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-08-19T14:03:35.320 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-19T14:03:35.321 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-08-19T14:03:35.321 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-test-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2024-08-19T14:03:35.321 INFO:teuthology.orchestra.run.smithi032.stdout: libcephsqlite-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2024-08-19T14:03:35.321 INFO:teuthology.orchestra.run.smithi032.stdout: socat-1.7.4.1-6.el9.x86_64 2024-08-19T14:03:35.321 INFO:teuthology.orchestra.run.smithi032.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-08-19T14:03:35.321 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-19T14:03:35.321 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-08-19T14:03:35.593 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph 2024-08-19T14:03:36.136 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:00:37 ago on Mon 19 Aug 2024 02:02:59 PM UTC. 2024-08-19T14:03:36.273 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-08-19T14:03:36.275 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-08-19T14:03:36.276 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repository Size 2024-08-19T14:03:36.276 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-08-19T14:03:36.276 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-08-19T14:03:36.276 INFO:teuthology.orchestra.run.smithi032.stdout: ceph x86_64 2:17.2.7-1406.gdf62bdcc.el9 ceph 6.5 k 2024-08-19T14:03:36.276 INFO:teuthology.orchestra.run.smithi032.stdout:Installing dependencies: 2024-08-19T14:03:36.276 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mds x86_64 2:17.2.7-1406.gdf62bdcc.el9 ceph 2.1 M 2024-08-19T14:03:36.276 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr x86_64 2:17.2.7-1406.gdf62bdcc.el9 ceph 1.4 M 2024-08-19T14:03:36.276 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-modules-core noarch 2:17.2.7-1406.gdf62bdcc.el9 ceph-noarch 240 k 2024-08-19T14:03:36.276 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mon x86_64 2:17.2.7-1406.gdf62bdcc.el9 ceph 3.9 M 2024-08-19T14:03:36.276 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-osd x86_64 2:17.2.7-1406.gdf62bdcc.el9 ceph 15 M 2024-08-19T14:03:36.276 INFO:teuthology.orchestra.run.smithi032.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-08-19T14:03:36.276 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-08-19T14:03:36.277 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-08-19T14:03:36.277 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-08-19T14:03:36.277 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-08-19T14:03:36.277 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-08-19T14:03:36.277 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-08-19T14:03:36.277 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-collections 2024-08-19T14:03:36.277 INFO:teuthology.orchestra.run.smithi032.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-08-19T14:03:36.277 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-08-19T14:03:36.277 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-08-19T14:03:36.277 INFO:teuthology.orchestra.run.smithi032.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-08-19T14:03:36.277 INFO:teuthology.orchestra.run.smithi032.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-08-19T14:03:36.277 INFO:teuthology.orchestra.run.smithi032.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-08-19T14:03:36.277 INFO:teuthology.orchestra.run.smithi032.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-08-19T14:03:36.278 INFO:teuthology.orchestra.run.smithi032.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-08-19T14:03:36.278 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-08-19T14:03:36.278 INFO:teuthology.orchestra.run.smithi032.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-08-19T14:03:36.278 INFO:teuthology.orchestra.run.smithi032.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-08-19T14:03:36.278 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-08-19T14:03:36.278 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-08-19T14:03:36.278 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-08-19T14:03:36.278 INFO:teuthology.orchestra.run.smithi032.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-08-19T14:03:36.278 INFO:teuthology.orchestra.run.smithi032.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-08-19T14:03:36.278 INFO:teuthology.orchestra.run.smithi032.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-08-19T14:03:36.278 INFO:teuthology.orchestra.run.smithi032.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-08-19T14:03:36.278 INFO:teuthology.orchestra.run.smithi032.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-08-19T14:03:36.278 INFO:teuthology.orchestra.run.smithi032.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-08-19T14:03:36.279 INFO:teuthology.orchestra.run.smithi032.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-08-19T14:03:36.279 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-19T14:03:36.279 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-08-19T14:03:36.279 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-08-19T14:03:36.279 INFO:teuthology.orchestra.run.smithi032.stdout:Install 34 Packages 2024-08-19T14:03:36.279 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-19T14:03:36.280 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 27 M 2024-08-19T14:03:36.280 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 96 M 2024-08-19T14:03:36.280 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-08-19T14:03:36.557 INFO:teuthology.orchestra.run.smithi032.stdout:(1/34): ceph-17.2.7-1406.gdf62bdcc.el9.x86_64.r 44 kB/s | 6.5 kB 00:00 2024-08-19T14:03:36.732 INFO:teuthology.orchestra.run.smithi032.stdout:(2/34): ceph-mds-17.2.7-1406.gdf62bdcc.el9.x86_ 6.4 MB/s | 2.1 MB 00:00 2024-08-19T14:03:36.765 INFO:teuthology.orchestra.run.smithi032.stdout:(3/34): ceph-mgr-17.2.7-1406.gdf62bdcc.el9.x86_ 4.0 MB/s | 1.4 MB 00:00 2024-08-19T14:03:36.849 INFO:teuthology.orchestra.run.smithi032.stdout:(4/34): ceph-mgr-modules-core-17.2.7-1406.gdf62 2.8 MB/s | 240 kB 00:00 2024-08-19T14:03:36.957 INFO:teuthology.orchestra.run.smithi032.stdout:(5/34): ceph-mon-17.2.7-1406.gdf62bdcc.el9.x86_ 9.8 MB/s | 3.9 MB 00:00 2024-08-19T14:03:37.027 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: libcephsqlite-2:17.2.7-1406.gdf62bdcc.el9.x86_64 3/4 2024-08-19T14:03:37.101 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-test-2:17.2.7-1406.gdf62bdcc.el9.x86_64 4/4 2024-08-19T14:03:37.108 INFO:teuthology.orchestra.run.smithi032.stdout:(6/34): python3-cffi-1.14.5-5.el9.x86_64.rpm 980 kB/s | 253 kB 00:00 2024-08-19T14:03:37.275 INFO:teuthology.orchestra.run.smithi032.stdout:(7/34): ceph-osd-17.2.7-1406.gdf62bdcc.el9.x86_ 28 MB/s | 15 MB 00:00 2024-08-19T14:03:37.300 INFO:teuthology.orchestra.run.smithi032.stdout:(8/34): python3-ply-3.11-14.el9.noarch.rpm 555 kB/s | 106 kB 00:00 2024-08-19T14:03:37.334 INFO:teuthology.orchestra.run.smithi032.stdout:(9/34): python3-cryptography-36.0.1-4.el9.x86_6 3.3 MB/s | 1.2 MB 00:00 2024-08-19T14:03:37.359 INFO:teuthology.orchestra.run.smithi032.stdout:(10/34): python3-requests-2.25.1-8.el9.noarch.r 2.1 MB/s | 125 kB 00:00 2024-08-19T14:03:37.384 INFO:teuthology.orchestra.run.smithi032.stdout:(11/34): python3-urllib3-1.26.5-6.el9.noarch.rp 4.2 MB/s | 215 kB 00:00 2024-08-19T14:03:37.418 INFO:teuthology.orchestra.run.smithi032.stdout:(12/34): python3-pycparser-2.20-6.el9.noarch.rp 947 kB/s | 135 kB 00:00 2024-08-19T14:03:37.652 INFO:teuthology.orchestra.run.smithi032.stdout:(13/34): python3-markupsafe-1.1.1-12.el9.x86_64 130 kB/s | 35 kB 00:00 2024-08-19T14:03:37.835 INFO:teuthology.orchestra.run.smithi032.stdout:(14/34): python3-pytz-2021.1-5.el9.noarch.rpm 278 kB/s | 51 kB 00:00 2024-08-19T14:03:37.861 INFO:teuthology.orchestra.run.smithi032.stdout:(15/34): python3-packaging-20.9-5.el9.noarch.rp 174 kB/s | 77 kB 00:00 2024-08-19T14:03:37.886 INFO:teuthology.orchestra.run.smithi032.stdout:(16/34): python3-mako-1.1.4-6.el9.noarch.rpm 327 kB/s | 172 kB 00:00 2024-08-19T14:03:37.936 INFO:teuthology.orchestra.run.smithi032.stdout:(17/34): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 580 kB/s | 43 kB 00:00 2024-08-19T14:03:37.962 INFO:teuthology.orchestra.run.smithi032.stdout:(18/34): python3-toml-0.10.2-6.el9.noarch.rpm 330 kB/s | 42 kB 00:00 2024-08-19T14:03:37.995 INFO:teuthology.orchestra.run.smithi032.stdout:(19/34): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.5 MB/s | 172 kB 00:00 2024-08-19T14:03:38.029 INFO:teuthology.orchestra.run.smithi032.stdout:(20/34): python3-jaraco-classes-3.2.1-5.el9.noa 568 kB/s | 18 kB 00:00 2024-08-19T14:03:38.042 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: libcephsqlite-2:17.2.7-1406.gdf62bdcc.el9.x86_64 3/4 2024-08-19T14:03:38.062 INFO:teuthology.orchestra.run.smithi032.stdout:(21/34): python3-jaraco-8.2.1-3.el9.noarch.rpm 106 kB/s | 11 kB 00:00 2024-08-19T14:03:38.088 INFO:teuthology.orchestra.run.smithi032.stdout:(22/34): python3-jaraco-collections-3.0.0-8.el9 396 kB/s | 23 kB 00:00 2024-08-19T14:03:38.113 INFO:teuthology.orchestra.run.smithi032.stdout:(23/34): python3-cherrypy-18.6.1-2.el9.noarch.r 2.0 MB/s | 358 kB 00:00 2024-08-19T14:03:38.119 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-test-2:17.2.7-1406.gdf62bdcc.el9.x86_64 4/4 2024-08-19T14:03:38.138 INFO:teuthology.orchestra.run.smithi032.stdout:(24/34): python3-jaraco-functools-3.5.0-2.el9.n 258 kB/s | 19 kB 00:00 2024-08-19T14:03:38.163 INFO:teuthology.orchestra.run.smithi032.stdout:(25/34): python3-jaraco-text-3.2.0-6.el9.noarch 261 kB/s | 20 kB 00:00 2024-08-19T14:03:38.180 INFO:teuthology.orchestra.run.smithi032.stdout:(26/34): python3-logutils-0.3.5-21.el9.noarch.r 691 kB/s | 46 kB 00:00 2024-08-19T14:03:38.205 INFO:teuthology.orchestra.run.smithi032.stdout:(27/34): python3-more-itertools-8.12.0-2.el9.no 1.1 MB/s | 79 kB 00:00 2024-08-19T14:03:38.223 INFO:teuthology.orchestra.run.smithi032.stdout:(28/34): python3-portend-3.1.0-2.el9.noarch.rpm 388 kB/s | 16 kB 00:00 2024-08-19T14:03:38.248 INFO:teuthology.orchestra.run.smithi032.stdout:(29/34): python3-pyOpenSSL-21.0.0-1.el9.noarch. 2.1 MB/s | 90 kB 00:00 2024-08-19T14:03:38.274 INFO:teuthology.orchestra.run.smithi032.stdout:(30/34): python3-tempora-5.0.0-2.el9.noarch.rpm 714 kB/s | 36 kB 00:00 2024-08-19T14:03:38.299 INFO:teuthology.orchestra.run.smithi032.stdout:(31/34): python3-pecan-1.4.2-3.el9.noarch.rpm 2.0 MB/s | 272 kB 00:00 2024-08-19T14:03:38.324 INFO:teuthology.orchestra.run.smithi032.stdout:(32/34): python3-zc-lockfile-2.0-10.el9.noarch. 805 kB/s | 20 kB 00:00 2024-08-19T14:03:38.349 INFO:teuthology.orchestra.run.smithi032.stdout:(33/34): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-08-19T14:03:38.374 INFO:teuthology.orchestra.run.smithi032.stdout:(34/34): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.2 MB/s | 427 kB 00:00 2024-08-19T14:03:38.378 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-08-19T14:03:38.378 INFO:teuthology.orchestra.run.smithi032.stdout:Total 13 MB/s | 27 MB 00:02 2024-08-19T14:03:38.510 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-08-19T14:03:38.549 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-08-19T14:03:38.549 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-08-19T14:03:38.833 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-08-19T14:03:38.834 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-08-19T14:03:39.280 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-test-2:17.2.7-1406.gdf62bdcc.el9.x86_64 4/4 2024-08-19T14:03:39.280 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-test-2:17.2.7-1406.gdf62bdcc.el9.x86_64 1/4 2024-08-19T14:03:39.280 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libcephsqlite-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2/4 2024-08-19T14:03:39.280 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-08-19T14:03:39.596 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-08-19T14:03:39.596 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-08-19T14:03:39.596 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-08-19T14:03:39.597 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-test-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2024-08-19T14:03:39.597 INFO:teuthology.orchestra.run.smithi045.stdout: libcephsqlite-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2024-08-19T14:03:39.597 INFO:teuthology.orchestra.run.smithi045.stdout: socat-1.7.4.1-6.el9.x86_64 2024-08-19T14:03:39.597 INFO:teuthology.orchestra.run.smithi045.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-08-19T14:03:39.597 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-08-19T14:03:39.597 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-08-19T14:03:39.875 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph 2024-08-19T14:03:40.194 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-08-19T14:03:40.240 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-test-2:17.2.7-1406.gdf62bdcc.el9.x86_64 4/4 2024-08-19T14:03:40.240 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-test-2:17.2.7-1406.gdf62bdcc.el9.x86_64 1/4 2024-08-19T14:03:40.240 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libcephsqlite-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2/4 2024-08-19T14:03:40.240 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-08-19T14:03:40.261 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/34 2024-08-19T14:03:40.285 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/34 2024-08-19T14:03:40.318 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/34 2024-08-19T14:03:40.415 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/34 2024-08-19T14:03:40.425 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:38 ago on Mon 19 Aug 2024 02:03:02 PM UTC. 2024-08-19T14:03:40.529 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-08-19T14:03:40.529 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:03:40.530 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-08-19T14:03:40.530 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-test-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2024-08-19T14:03:40.530 INFO:teuthology.orchestra.run.smithi031.stdout: libcephsqlite-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2024-08-19T14:03:40.530 INFO:teuthology.orchestra.run.smithi031.stdout: socat-1.7.4.1-6.el9.x86_64 2024-08-19T14:03:40.530 INFO:teuthology.orchestra.run.smithi031.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-08-19T14:03:40.530 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:03:40.530 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-08-19T14:03:40.557 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-08-19T14:03:40.559 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-08-19T14:03:40.559 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2024-08-19T14:03:40.560 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-08-19T14:03:40.560 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-08-19T14:03:40.560 INFO:teuthology.orchestra.run.smithi045.stdout: ceph x86_64 2:17.2.7-1406.gdf62bdcc.el9 ceph 6.5 k 2024-08-19T14:03:40.560 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2024-08-19T14:03:40.560 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mds x86_64 2:17.2.7-1406.gdf62bdcc.el9 ceph 2.1 M 2024-08-19T14:03:40.560 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr x86_64 2:17.2.7-1406.gdf62bdcc.el9 ceph 1.4 M 2024-08-19T14:03:40.560 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-modules-core noarch 2:17.2.7-1406.gdf62bdcc.el9 ceph-noarch 240 k 2024-08-19T14:03:40.560 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mon x86_64 2:17.2.7-1406.gdf62bdcc.el9 ceph 3.9 M 2024-08-19T14:03:40.560 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-osd x86_64 2:17.2.7-1406.gdf62bdcc.el9 ceph 15 M 2024-08-19T14:03:40.560 INFO:teuthology.orchestra.run.smithi045.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-08-19T14:03:40.560 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-08-19T14:03:40.560 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-08-19T14:03:40.561 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-08-19T14:03:40.561 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-08-19T14:03:40.561 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-08-19T14:03:40.561 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-08-19T14:03:40.561 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-collections 2024-08-19T14:03:40.561 INFO:teuthology.orchestra.run.smithi045.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-08-19T14:03:40.561 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-08-19T14:03:40.561 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-08-19T14:03:40.561 INFO:teuthology.orchestra.run.smithi045.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-08-19T14:03:40.561 INFO:teuthology.orchestra.run.smithi045.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-08-19T14:03:40.561 INFO:teuthology.orchestra.run.smithi045.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-08-19T14:03:40.561 INFO:teuthology.orchestra.run.smithi045.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-08-19T14:03:40.561 INFO:teuthology.orchestra.run.smithi045.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-08-19T14:03:40.562 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-08-19T14:03:40.562 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-08-19T14:03:40.562 INFO:teuthology.orchestra.run.smithi045.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-08-19T14:03:40.562 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-08-19T14:03:40.562 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-08-19T14:03:40.562 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-08-19T14:03:40.562 INFO:teuthology.orchestra.run.smithi045.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-08-19T14:03:40.562 INFO:teuthology.orchestra.run.smithi045.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-08-19T14:03:40.562 INFO:teuthology.orchestra.run.smithi045.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-08-19T14:03:40.562 INFO:teuthology.orchestra.run.smithi045.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-08-19T14:03:40.562 INFO:teuthology.orchestra.run.smithi045.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-08-19T14:03:40.562 INFO:teuthology.orchestra.run.smithi045.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-08-19T14:03:40.563 INFO:teuthology.orchestra.run.smithi045.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-08-19T14:03:40.563 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-08-19T14:03:40.563 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-08-19T14:03:40.563 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-08-19T14:03:40.563 INFO:teuthology.orchestra.run.smithi045.stdout:Install 34 Packages 2024-08-19T14:03:40.563 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-08-19T14:03:40.564 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 27 M 2024-08-19T14:03:40.564 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 96 M 2024-08-19T14:03:40.564 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-08-19T14:03:40.625 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/34 2024-08-19T14:03:40.672 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/34 2024-08-19T14:03:40.715 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/34 2024-08-19T14:03:40.744 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/34 2024-08-19T14:03:40.778 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/34 2024-08-19T14:03:40.794 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph 2024-08-19T14:03:40.812 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/34 2024-08-19T14:03:40.845 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/34 2024-08-19T14:03:40.888 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/34 2024-08-19T14:03:40.922 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/34 2024-08-19T14:03:40.940 INFO:teuthology.orchestra.run.smithi045.stdout:(1/34): ceph-17.2.7-1406.gdf62bdcc.el9.x86_64.r 57 kB/s | 6.5 kB 00:00 2024-08-19T14:03:40.972 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/34 2024-08-19T14:03:41.011 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-packaging-20.9-5.el9.noarch 15/34 2024-08-19T14:03:41.106 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 16/34 2024-08-19T14:03:41.124 INFO:teuthology.orchestra.run.smithi045.stdout:(2/34): ceph-mds-17.2.7-1406.gdf62bdcc.el9.x86_ 7.0 MB/s | 2.1 MB 00:00 2024-08-19T14:03:41.174 INFO:teuthology.orchestra.run.smithi045.stdout:(3/34): ceph-mgr-17.2.7-1406.gdf62bdcc.el9.x86_ 4.1 MB/s | 1.4 MB 00:00 2024-08-19T14:03:41.257 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 17/34 2024-08-19T14:03:41.259 INFO:teuthology.orchestra.run.smithi045.stdout:(4/34): ceph-mgr-modules-core-17.2.7-1406.gdf62 2.8 MB/s | 240 kB 00:00 2024-08-19T14:03:41.326 INFO:teuthology.orchestra.run.smithi045.stdout:(5/34): ceph-mon-17.2.7-1406.gdf62bdcc.el9.x86_ 10 MB/s | 3.9 MB 00:00 2024-08-19T14:03:41.340 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 18/34 2024-08-19T14:03:41.341 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:00:38 ago on Mon 19 Aug 2024 02:03:03 PM UTC. 2024-08-19T14:03:41.399 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 19/34 2024-08-19T14:03:41.436 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 20/34 2024-08-19T14:03:41.477 INFO:teuthology.orchestra.run.smithi045.stdout:(6/34): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.1 MB/s | 253 kB 00:00 2024-08-19T14:03:41.486 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-08-19T14:03:41.489 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-08-19T14:03:41.489 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2024-08-19T14:03:41.489 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-08-19T14:03:41.489 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-08-19T14:03:41.489 INFO:teuthology.orchestra.run.smithi031.stdout: ceph x86_64 2:17.2.7-1406.gdf62bdcc.el9 ceph 6.5 k 2024-08-19T14:03:41.489 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2024-08-19T14:03:41.489 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mds x86_64 2:17.2.7-1406.gdf62bdcc.el9 ceph 2.1 M 2024-08-19T14:03:41.489 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr x86_64 2:17.2.7-1406.gdf62bdcc.el9 ceph 1.4 M 2024-08-19T14:03:41.489 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-modules-core noarch 2:17.2.7-1406.gdf62bdcc.el9 ceph-noarch 240 k 2024-08-19T14:03:41.489 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mon x86_64 2:17.2.7-1406.gdf62bdcc.el9 ceph 3.9 M 2024-08-19T14:03:41.489 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-osd x86_64 2:17.2.7-1406.gdf62bdcc.el9 ceph 15 M 2024-08-19T14:03:41.489 INFO:teuthology.orchestra.run.smithi031.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-08-19T14:03:41.490 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-08-19T14:03:41.490 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-08-19T14:03:41.490 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-08-19T14:03:41.490 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-08-19T14:03:41.490 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-08-19T14:03:41.490 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-08-19T14:03:41.490 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-collections 2024-08-19T14:03:41.490 INFO:teuthology.orchestra.run.smithi031.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-08-19T14:03:41.490 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-08-19T14:03:41.490 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-08-19T14:03:41.490 INFO:teuthology.orchestra.run.smithi031.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-08-19T14:03:41.490 INFO:teuthology.orchestra.run.smithi031.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-08-19T14:03:41.490 INFO:teuthology.orchestra.run.smithi031.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-08-19T14:03:41.491 INFO:teuthology.orchestra.run.smithi031.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-08-19T14:03:41.491 INFO:teuthology.orchestra.run.smithi031.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-08-19T14:03:41.491 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-08-19T14:03:41.491 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-08-19T14:03:41.491 INFO:teuthology.orchestra.run.smithi031.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-08-19T14:03:41.491 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-08-19T14:03:41.491 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-08-19T14:03:41.491 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-08-19T14:03:41.491 INFO:teuthology.orchestra.run.smithi031.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-08-19T14:03:41.491 INFO:teuthology.orchestra.run.smithi031.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-08-19T14:03:41.491 INFO:teuthology.orchestra.run.smithi031.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-08-19T14:03:41.491 INFO:teuthology.orchestra.run.smithi031.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-08-19T14:03:41.491 INFO:teuthology.orchestra.run.smithi031.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-08-19T14:03:41.492 INFO:teuthology.orchestra.run.smithi031.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-08-19T14:03:41.492 INFO:teuthology.orchestra.run.smithi031.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-08-19T14:03:41.492 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:03:41.492 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-08-19T14:03:41.492 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-08-19T14:03:41.492 INFO:teuthology.orchestra.run.smithi031.stdout:Install 34 Packages 2024-08-19T14:03:41.492 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:03:41.493 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 27 M 2024-08-19T14:03:41.493 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 96 M 2024-08-19T14:03:41.493 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-08-19T14:03:41.495 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-ply-3.11-14.el9.noarch 21/34 2024-08-19T14:03:41.559 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 22/34 2024-08-19T14:03:41.618 INFO:teuthology.orchestra.run.smithi045.stdout:(7/34): ceph-osd-17.2.7-1406.gdf62bdcc.el9.x86_ 31 MB/s | 15 MB 00:00 2024-08-19T14:03:41.644 INFO:teuthology.orchestra.run.smithi045.stdout:(8/34): python3-ply-3.11-14.el9.noarch.rpm 639 kB/s | 106 kB 00:00 2024-08-19T14:03:41.677 INFO:teuthology.orchestra.run.smithi045.stdout:(9/34): python3-cryptography-36.0.1-4.el9.x86_6 3.5 MB/s | 1.2 MB 00:00 2024-08-19T14:03:41.727 INFO:teuthology.orchestra.run.smithi045.stdout:(10/34): python3-requests-2.25.1-8.el9.noarch.r 1.5 MB/s | 125 kB 00:00 2024-08-19T14:03:41.769 INFO:teuthology.orchestra.run.smithi045.stdout:(11/34): python3-urllib3-1.26.5-6.el9.noarch.rp 2.3 MB/s | 215 kB 00:00 2024-08-19T14:03:41.772 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 23/34 2024-08-19T14:03:41.794 INFO:teuthology.orchestra.run.smithi045.stdout:(12/34): python3-pycparser-2.20-6.el9.noarch.rp 769 kB/s | 135 kB 00:00 2024-08-19T14:03:41.832 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 24/34 2024-08-19T14:03:41.853 INFO:teuthology.orchestra.run.smithi045.stdout:(13/34): python3-mako-1.1.4-6.el9.noarch.rpm 1.3 MB/s | 172 kB 00:00 2024-08-19T14:03:41.870 INFO:teuthology.orchestra.run.smithi045.stdout:(14/34): python3-markupsafe-1.1.1-12.el9.x86_64 346 kB/s | 35 kB 00:00 2024-08-19T14:03:41.896 INFO:teuthology.orchestra.run.smithi045.stdout:(15/34): python3-pytz-2021.1-5.el9.noarch.rpm 1.2 MB/s | 51 kB 00:00 2024-08-19T14:03:41.914 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 25/34 2024-08-19T14:03:41.921 INFO:teuthology.orchestra.run.smithi045.stdout:(16/34): python3-packaging-20.9-5.el9.noarch.rp 612 kB/s | 77 kB 00:00 2024-08-19T14:03:41.938 INFO:teuthology.orchestra.run.smithi045.stdout:(17/34): python3-toml-0.10.2-6.el9.noarch.rpm 617 kB/s | 42 kB 00:00 2024-08-19T14:03:41.980 INFO:teuthology.orchestra.run.smithi045.stdout:(18/34): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 520 kB/s | 43 kB 00:00 2024-08-19T14:03:41.997 INFO:teuthology.orchestra.run.smithi045.stdout:(19/34): python3-jaraco-8.2.1-3.el9.noarch.rpm 738 kB/s | 11 kB 00:00 2024-08-19T14:03:42.013 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 26/34 2024-08-19T14:03:42.021 INFO:teuthology.orchestra.run.smithi045.stdout:(20/34): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.7 MB/s | 172 kB 00:00 2024-08-19T14:03:42.038 INFO:teuthology.orchestra.run.smithi045.stdout:(21/34): python3-jaraco-classes-3.2.1-5.el9.noa 424 kB/s | 18 kB 00:00 2024-08-19T14:03:42.048 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/34 2024-08-19T14:03:42.053 INFO:teuthology.orchestra.run.smithi031.stdout:(1/34): ceph-17.2.7-1406.gdf62bdcc.el9.x86_64.r 47 kB/s | 6.5 kB 00:00 2024-08-19T14:03:42.055 INFO:teuthology.orchestra.run.smithi045.stdout:(22/34): python3-jaraco-collections-3.0.0-8.el9 691 kB/s | 23 kB 00:00 2024-08-19T14:03:42.081 INFO:teuthology.orchestra.run.smithi045.stdout:(23/34): python3-jaraco-functools-3.5.0-2.el9.n 464 kB/s | 19 kB 00:00 2024-08-19T14:03:42.106 INFO:teuthology.orchestra.run.smithi045.stdout:(24/34): python3-jaraco-text-3.2.0-6.el9.noarch 391 kB/s | 20 kB 00:00 2024-08-19T14:03:42.131 INFO:teuthology.orchestra.run.smithi045.stdout:(25/34): python3-logutils-0.3.5-21.el9.noarch.r 920 kB/s | 46 kB 00:00 2024-08-19T14:03:42.156 INFO:teuthology.orchestra.run.smithi045.stdout:(26/34): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-08-19T14:03:42.171 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 28/34 2024-08-19T14:03:42.182 INFO:teuthology.orchestra.run.smithi045.stdout:(27/34): python3-cherrypy-18.6.1-2.el9.noarch.r 1.5 MB/s | 358 kB 00:00 2024-08-19T14:03:42.207 INFO:teuthology.orchestra.run.smithi045.stdout:(28/34): python3-portend-3.1.0-2.el9.noarch.rpm 327 kB/s | 16 kB 00:00 2024-08-19T14:03:42.211 INFO:teuthology.orchestra.run.smithi031.stdout:(2/34): ceph-mgr-17.2.7-1406.gdf62bdcc.el9.x86_ 4.7 MB/s | 1.4 MB 00:00 2024-08-19T14:03:42.232 INFO:teuthology.orchestra.run.smithi045.stdout:(29/34): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.8 MB/s | 90 kB 00:00 2024-08-19T14:03:42.258 INFO:teuthology.orchestra.run.smithi045.stdout:(30/34): python3-tempora-5.0.0-2.el9.noarch.rpm 711 kB/s | 36 kB 00:00 2024-08-19T14:03:42.262 INFO:teuthology.orchestra.run.smithi031.stdout:(3/34): ceph-mds-17.2.7-1406.gdf62bdcc.el9.x86_ 6.0 MB/s | 2.1 MB 00:00 2024-08-19T14:03:42.283 INFO:teuthology.orchestra.run.smithi045.stdout:(31/34): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-08-19T14:03:42.308 INFO:teuthology.orchestra.run.smithi045.stdout:(32/34): python3-zc-lockfile-2.0-10.el9.noarch. 803 kB/s | 20 kB 00:00 2024-08-19T14:03:42.326 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-1406.gdf62bdcc.el9. 29/34 2024-08-19T14:03:42.333 INFO:teuthology.orchestra.run.smithi045.stdout:(33/34): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-08-19T14:03:42.345 INFO:teuthology.orchestra.run.smithi031.stdout:(4/34): ceph-mgr-modules-core-17.2.7-1406.gdf62 2.8 MB/s | 240 kB 00:00 2024-08-19T14:03:42.358 INFO:teuthology.orchestra.run.smithi045.stdout:(34/34): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.2 MB/s | 427 kB 00:00 2024-08-19T14:03:42.359 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mgr-2:17.2.7-1406.gdf62bdcc.el9.x86_64 30/34 2024-08-19T14:03:42.363 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-08-19T14:03:42.363 INFO:teuthology.orchestra.run.smithi045.stdout:Total 15 MB/s | 27 MB 00:01 2024-08-19T14:03:42.385 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-mgr-2:17.2.7-1406.gdf62bdcc.el9.x86_64 30/34 2024-08-19T14:03:42.385 INFO:teuthology.orchestra.run.smithi032.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-19T14:03:42.385 INFO:teuthology.orchestra.run.smithi032.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-08-19T14:03:42.385 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-08-19T14:03:42.385 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-08-19T14:03:42.385 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-19T14:03:42.488 INFO:teuthology.orchestra.run.smithi031.stdout:(5/34): ceph-mon-17.2.7-1406.gdf62bdcc.el9.x86_ 9.0 MB/s | 3.9 MB 00:00 2024-08-19T14:03:42.492 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-08-19T14:03:42.513 INFO:teuthology.orchestra.run.smithi031.stdout:(6/34): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.5 MB/s | 253 kB 00:00 2024-08-19T14:03:42.532 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-08-19T14:03:42.532 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-08-19T14:03:42.588 INFO:teuthology.orchestra.run.smithi031.stdout:(7/34): python3-ply-3.11-14.el9.noarch.rpm 1.4 MB/s | 106 kB 00:00 2024-08-19T14:03:42.755 INFO:teuthology.orchestra.run.smithi031.stdout:(8/34): ceph-osd-17.2.7-1406.gdf62bdcc.el9.x86_ 28 MB/s | 15 MB 00:00 2024-08-19T14:03:42.780 INFO:teuthology.orchestra.run.smithi031.stdout:(9/34): python3-pycparser-2.20-6.el9.noarch.rpm 705 kB/s | 135 kB 00:00 2024-08-19T14:03:42.815 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-08-19T14:03:42.816 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-08-19T14:03:42.880 INFO:teuthology.orchestra.run.smithi031.stdout:(10/34): python3-urllib3-1.26.5-6.el9.noarch.rp 2.1 MB/s | 215 kB 00:00 2024-08-19T14:03:42.905 INFO:teuthology.orchestra.run.smithi031.stdout:(11/34): python3-requests-2.25.1-8.el9.noarch.r 831 kB/s | 125 kB 00:00 2024-08-19T14:03:42.947 INFO:teuthology.orchestra.run.smithi031.stdout:(12/34): python3-cryptography-36.0.1-4.el9.x86_ 2.7 MB/s | 1.2 MB 00:00 2024-08-19T14:03:43.014 INFO:teuthology.orchestra.run.smithi031.stdout:(13/34): python3-markupsafe-1.1.1-12.el9.x86_64 321 kB/s | 35 kB 00:00 2024-08-19T14:03:43.081 INFO:teuthology.orchestra.run.smithi031.stdout:(14/34): python3-packaging-20.9-5.el9.noarch.rp 578 kB/s | 77 kB 00:00 2024-08-19T14:03:43.098 INFO:teuthology.orchestra.run.smithi031.stdout:(15/34): python3-pytz-2021.1-5.el9.noarch.rpm 610 kB/s | 51 kB 00:00 2024-08-19T14:03:43.123 INFO:teuthology.orchestra.run.smithi031.stdout:(16/34): python3-mako-1.1.4-6.el9.noarch.rpm 709 kB/s | 172 kB 00:00 2024-08-19T14:03:43.149 INFO:teuthology.orchestra.run.smithi031.stdout:(17/34): python3-toml-0.10.2-6.el9.noarch.rpm 619 kB/s | 42 kB 00:00 2024-08-19T14:03:43.199 INFO:teuthology.orchestra.run.smithi031.stdout:(18/34): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 433 kB/s | 43 kB 00:00 2024-08-19T14:03:43.224 INFO:teuthology.orchestra.run.smithi031.stdout:(19/34): python3-jaraco-8.2.1-3.el9.noarch.rpm 468 kB/s | 11 kB 00:00 2024-08-19T14:03:43.249 INFO:teuthology.orchestra.run.smithi031.stdout:(20/34): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.3 MB/s | 172 kB 00:00 2024-08-19T14:03:43.275 INFO:teuthology.orchestra.run.smithi031.stdout:(21/34): python3-jaraco-classes-3.2.1-5.el9.noa 351 kB/s | 18 kB 00:00 2024-08-19T14:03:43.300 INFO:teuthology.orchestra.run.smithi031.stdout:(22/34): python3-jaraco-collections-3.0.0-8.el9 461 kB/s | 23 kB 00:00 2024-08-19T14:03:43.325 INFO:teuthology.orchestra.run.smithi031.stdout:(23/34): python3-jaraco-functools-3.5.0-2.el9.n 389 kB/s | 19 kB 00:00 2024-08-19T14:03:43.350 INFO:teuthology.orchestra.run.smithi031.stdout:(24/34): python3-jaraco-text-3.2.0-6.el9.noarch 393 kB/s | 20 kB 00:00 2024-08-19T14:03:43.376 INFO:teuthology.orchestra.run.smithi031.stdout:(25/34): python3-logutils-0.3.5-21.el9.noarch.r 922 kB/s | 46 kB 00:00 2024-08-19T14:03:43.401 INFO:teuthology.orchestra.run.smithi031.stdout:(26/34): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-08-19T14:03:43.426 INFO:teuthology.orchestra.run.smithi031.stdout:(27/34): python3-cherrypy-18.6.1-2.el9.noarch.r 1.3 MB/s | 358 kB 00:00 2024-08-19T14:03:43.452 INFO:teuthology.orchestra.run.smithi031.stdout:(28/34): python3-portend-3.1.0-2.el9.noarch.rpm 328 kB/s | 16 kB 00:00 2024-08-19T14:03:43.477 INFO:teuthology.orchestra.run.smithi031.stdout:(29/34): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.8 MB/s | 90 kB 00:00 2024-08-19T14:03:43.502 INFO:teuthology.orchestra.run.smithi031.stdout:(30/34): python3-tempora-5.0.0-2.el9.noarch.rpm 713 kB/s | 36 kB 00:00 2024-08-19T14:03:43.527 INFO:teuthology.orchestra.run.smithi031.stdout:(31/34): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-08-19T14:03:43.552 INFO:teuthology.orchestra.run.smithi031.stdout:(32/34): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-08-19T14:03:43.578 INFO:teuthology.orchestra.run.smithi031.stdout:(33/34): python3-zc-lockfile-2.0-10.el9.noarch. 398 kB/s | 20 kB 00:00 2024-08-19T14:03:43.603 INFO:teuthology.orchestra.run.smithi031.stdout:(34/34): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.2 MB/s | 427 kB 00:00 2024-08-19T14:03:43.607 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-08-19T14:03:43.608 INFO:teuthology.orchestra.run.smithi031.stdout:Total 13 MB/s | 27 MB 00:02 2024-08-19T14:03:43.741 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-08-19T14:03:43.770 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-osd-2:17.2.7-1406.gdf62bdcc.el9.x86_64 31/34 2024-08-19T14:03:43.781 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-08-19T14:03:43.781 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-08-19T14:03:43.794 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-osd-2:17.2.7-1406.gdf62bdcc.el9.x86_64 31/34 2024-08-19T14:03:43.794 INFO:teuthology.orchestra.run.smithi032.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-19T14:03:43.795 INFO:teuthology.orchestra.run.smithi032.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-08-19T14:03:43.795 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-08-19T14:03:43.795 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-08-19T14:03:43.795 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-19T14:03:44.065 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-08-19T14:03:44.065 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-08-19T14:03:44.185 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mon-2:17.2.7-1406.gdf62bdcc.el9.x86_64 32/34 2024-08-19T14:03:44.209 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-mon-2:17.2.7-1406.gdf62bdcc.el9.x86_64 32/34 2024-08-19T14:03:44.209 INFO:teuthology.orchestra.run.smithi032.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-19T14:03:44.210 INFO:teuthology.orchestra.run.smithi032.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-08-19T14:03:44.210 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-08-19T14:03:44.210 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-08-19T14:03:44.210 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-19T14:03:44.227 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-08-19T14:03:44.286 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/34 2024-08-19T14:03:44.319 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/34 2024-08-19T14:03:44.352 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/34 2024-08-19T14:03:44.419 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mds-2:17.2.7-1406.gdf62bdcc.el9.x86_64 33/34 2024-08-19T14:03:44.442 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-mds-2:17.2.7-1406.gdf62bdcc.el9.x86_64 33/34 2024-08-19T14:03:44.443 INFO:teuthology.orchestra.run.smithi032.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-19T14:03:44.443 INFO:teuthology.orchestra.run.smithi032.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-08-19T14:03:44.443 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-08-19T14:03:44.443 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-08-19T14:03:44.443 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-19T14:03:44.448 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/34 2024-08-19T14:03:44.509 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-2:17.2.7-1406.gdf62bdcc.el9.x86_64 34/34 2024-08-19T14:03:44.524 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/34 2024-08-19T14:03:44.572 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/34 2024-08-19T14:03:44.615 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/34 2024-08-19T14:03:44.644 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/34 2024-08-19T14:03:44.678 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/34 2024-08-19T14:03:44.712 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/34 2024-08-19T14:03:44.753 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/34 2024-08-19T14:03:44.796 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/34 2024-08-19T14:03:44.830 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/34 2024-08-19T14:03:44.879 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/34 2024-08-19T14:03:44.919 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-packaging-20.9-5.el9.noarch 15/34 2024-08-19T14:03:45.020 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 16/34 2024-08-19T14:03:45.172 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 17/34 2024-08-19T14:03:45.255 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 18/34 2024-08-19T14:03:45.306 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 19/34 2024-08-19T14:03:45.344 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 20/34 2024-08-19T14:03:45.402 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-ply-3.11-14.el9.noarch 21/34 2024-08-19T14:03:45.459 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-08-19T14:03:45.466 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 22/34 2024-08-19T14:03:45.522 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/34 2024-08-19T14:03:45.555 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/34 2024-08-19T14:03:45.588 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/34 2024-08-19T14:03:45.661 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 23/34 2024-08-19T14:03:45.684 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/34 2024-08-19T14:03:45.714 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 24/34 2024-08-19T14:03:45.760 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/34 2024-08-19T14:03:45.795 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 25/34 2024-08-19T14:03:45.807 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/34 2024-08-19T14:03:45.842 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/34 2024-08-19T14:03:45.872 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/34 2024-08-19T14:03:45.894 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 26/34 2024-08-19T14:03:45.906 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/34 2024-08-19T14:03:45.929 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/34 2024-08-19T14:03:45.948 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/34 2024-08-19T14:03:45.989 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/34 2024-08-19T14:03:46.032 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/34 2024-08-19T14:03:46.049 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 28/34 2024-08-19T14:03:46.066 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/34 2024-08-19T14:03:46.115 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/34 2024-08-19T14:03:46.130 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-2:17.2.7-1406.gdf62bdcc.el9.x86_64 34/34 2024-08-19T14:03:46.130 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-2:17.2.7-1406.gdf62bdcc.el9.x86_64 1/34 2024-08-19T14:03:46.130 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mds-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2/34 2024-08-19T14:03:46.130 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mgr-2:17.2.7-1406.gdf62bdcc.el9.x86_64 3/34 2024-08-19T14:03:46.130 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mon-2:17.2.7-1406.gdf62bdcc.el9.x86_64 4/34 2024-08-19T14:03:46.131 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-osd-2:17.2.7-1406.gdf62bdcc.el9.x86_64 5/34 2024-08-19T14:03:46.131 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-1406.gdf62bdcc.el9. 6/34 2024-08-19T14:03:46.131 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/34 2024-08-19T14:03:46.131 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/34 2024-08-19T14:03:46.131 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/34 2024-08-19T14:03:46.131 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/34 2024-08-19T14:03:46.131 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/34 2024-08-19T14:03:46.131 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/34 2024-08-19T14:03:46.131 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/34 2024-08-19T14:03:46.131 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/34 2024-08-19T14:03:46.131 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 15/34 2024-08-19T14:03:46.131 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/34 2024-08-19T14:03:46.131 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/34 2024-08-19T14:03:46.132 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 18/34 2024-08-19T14:03:46.132 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 19/34 2024-08-19T14:03:46.132 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 20/34 2024-08-19T14:03:46.132 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 21/34 2024-08-19T14:03:46.132 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 22/34 2024-08-19T14:03:46.132 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 23/34 2024-08-19T14:03:46.132 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 24/34 2024-08-19T14:03:46.132 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 25/34 2024-08-19T14:03:46.134 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 26/34 2024-08-19T14:03:46.134 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 27/34 2024-08-19T14:03:46.134 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 28/34 2024-08-19T14:03:46.134 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 29/34 2024-08-19T14:03:46.134 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 30/34 2024-08-19T14:03:46.134 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 31/34 2024-08-19T14:03:46.134 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 32/34 2024-08-19T14:03:46.134 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 33/34 2024-08-19T14:03:46.155 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-packaging-20.9-5.el9.noarch 15/34 2024-08-19T14:03:46.206 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-1406.gdf62bdcc.el9. 29/34 2024-08-19T14:03:46.231 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 16/34 2024-08-19T14:03:46.232 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-2:17.2.7-1406.gdf62bdcc.el9.x86_64 30/34 2024-08-19T14:03:46.257 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mgr-2:17.2.7-1406.gdf62bdcc.el9.x86_64 30/34 2024-08-19T14:03:46.257 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-19T14:03:46.257 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-08-19T14:03:46.257 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-08-19T14:03:46.257 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-08-19T14:03:46.258 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-08-19T14:03:46.382 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 17/34 2024-08-19T14:03:46.532 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 18/34 2024-08-19T14:03:46.583 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 19/34 2024-08-19T14:03:46.621 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 20/34 2024-08-19T14:03:46.679 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-ply-3.11-14.el9.noarch 21/34 2024-08-19T14:03:46.743 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 22/34 2024-08-19T14:03:46.940 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 23/34 2024-08-19T14:03:46.991 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 24/34 2024-08-19T14:03:47.024 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 34/34 2024-08-19T14:03:47.024 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-19T14:03:47.024 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-08-19T14:03:47.024 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2024-08-19T14:03:47.024 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mds-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2024-08-19T14:03:47.024 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2024-08-19T14:03:47.024 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-modules-core-2:17.2.7-1406.gdf62bdcc.el9.noarch 2024-08-19T14:03:47.024 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mon-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2024-08-19T14:03:47.024 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-osd-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2024-08-19T14:03:47.024 INFO:teuthology.orchestra.run.smithi032.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-08-19T14:03:47.025 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-08-19T14:03:47.025 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-08-19T14:03:47.025 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-08-19T14:03:47.025 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-08-19T14:03:47.025 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-08-19T14:03:47.025 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-08-19T14:03:47.025 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-08-19T14:03:47.025 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-08-19T14:03:47.025 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-08-19T14:03:47.025 INFO:teuthology.orchestra.run.smithi032.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-08-19T14:03:47.025 INFO:teuthology.orchestra.run.smithi032.stdout: python3-mako-1.1.4-6.el9.noarch 2024-08-19T14:03:47.025 INFO:teuthology.orchestra.run.smithi032.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-08-19T14:03:47.025 INFO:teuthology.orchestra.run.smithi032.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-08-19T14:03:47.026 INFO:teuthology.orchestra.run.smithi032.stdout: python3-packaging-20.9-5.el9.noarch 2024-08-19T14:03:47.026 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-08-19T14:03:47.026 INFO:teuthology.orchestra.run.smithi032.stdout: python3-ply-3.11-14.el9.noarch 2024-08-19T14:03:47.026 INFO:teuthology.orchestra.run.smithi032.stdout: python3-portend-3.1.0-2.el9.noarch 2024-08-19T14:03:47.026 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-08-19T14:03:47.026 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pycparser-2.20-6.el9.noarch 2024-08-19T14:03:47.026 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pytz-2021.1-5.el9.noarch 2024-08-19T14:03:47.026 INFO:teuthology.orchestra.run.smithi032.stdout: python3-requests-2.25.1-8.el9.noarch 2024-08-19T14:03:47.026 INFO:teuthology.orchestra.run.smithi032.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-08-19T14:03:47.026 INFO:teuthology.orchestra.run.smithi032.stdout: python3-toml-0.10.2-6.el9.noarch 2024-08-19T14:03:47.026 INFO:teuthology.orchestra.run.smithi032.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-08-19T14:03:47.026 INFO:teuthology.orchestra.run.smithi032.stdout: python3-webob-1.8.7-6.el9.noarch 2024-08-19T14:03:47.027 INFO:teuthology.orchestra.run.smithi032.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-08-19T14:03:47.027 INFO:teuthology.orchestra.run.smithi032.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-08-19T14:03:47.027 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-19T14:03:47.027 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-08-19T14:03:47.072 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 25/34 2024-08-19T14:03:47.171 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 26/34 2024-08-19T14:03:47.206 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/34 2024-08-19T14:03:47.330 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 28/34 2024-08-19T14:03:47.362 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph-base 2024-08-19T14:03:47.492 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-1406.gdf62bdcc.el9. 29/34 2024-08-19T14:03:47.524 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-2:17.2.7-1406.gdf62bdcc.el9.x86_64 30/34 2024-08-19T14:03:47.549 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-2:17.2.7-1406.gdf62bdcc.el9.x86_64 30/34 2024-08-19T14:03:47.550 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-19T14:03:47.550 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-08-19T14:03:47.550 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-08-19T14:03:47.550 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-08-19T14:03:47.550 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:03:47.657 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-osd-2:17.2.7-1406.gdf62bdcc.el9.x86_64 31/34 2024-08-19T14:03:47.686 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-osd-2:17.2.7-1406.gdf62bdcc.el9.x86_64 31/34 2024-08-19T14:03:47.686 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-19T14:03:47.686 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-08-19T14:03:47.686 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-08-19T14:03:47.686 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-08-19T14:03:47.686 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-08-19T14:03:47.918 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:00:48 ago on Mon 19 Aug 2024 02:02:59 PM UTC. 2024-08-19T14:03:48.004 INFO:teuthology.orchestra.run.smithi032.stdout:Package ceph-base-2:17.2.7-1406.gdf62bdcc.el9.x86_64 is already installed. 2024-08-19T14:03:48.049 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-08-19T14:03:48.050 INFO:teuthology.orchestra.run.smithi032.stdout:Nothing to do. 2024-08-19T14:03:48.050 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-08-19T14:03:48.075 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mon-2:17.2.7-1406.gdf62bdcc.el9.x86_64 32/34 2024-08-19T14:03:48.099 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mon-2:17.2.7-1406.gdf62bdcc.el9.x86_64 32/34 2024-08-19T14:03:48.100 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-19T14:03:48.100 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-08-19T14:03:48.100 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-08-19T14:03:48.100 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-08-19T14:03:48.100 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-08-19T14:03:48.102 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install cephadm 2024-08-19T14:03:48.309 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mds-2:17.2.7-1406.gdf62bdcc.el9.x86_64 33/34 2024-08-19T14:03:48.333 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mds-2:17.2.7-1406.gdf62bdcc.el9.x86_64 33/34 2024-08-19T14:03:48.333 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-19T14:03:48.334 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-08-19T14:03:48.334 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-08-19T14:03:48.334 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-08-19T14:03:48.334 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-08-19T14:03:48.577 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-2:17.2.7-1406.gdf62bdcc.el9.x86_64 34/34 2024-08-19T14:03:48.657 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:00:49 ago on Mon 19 Aug 2024 02:02:59 PM UTC. 2024-08-19T14:03:48.794 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-08-19T14:03:48.794 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-08-19T14:03:48.795 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repository Size 2024-08-19T14:03:48.795 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-08-19T14:03:48.795 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-08-19T14:03:48.795 INFO:teuthology.orchestra.run.smithi032.stdout: cephadm noarch 2:17.2.7-1406.gdf62bdcc.el9 ceph-noarch 83 k 2024-08-19T14:03:48.795 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-19T14:03:48.795 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-08-19T14:03:48.795 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-08-19T14:03:48.795 INFO:teuthology.orchestra.run.smithi032.stdout:Install 1 Package 2024-08-19T14:03:48.795 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-19T14:03:48.795 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 83 k 2024-08-19T14:03:48.795 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 375 k 2024-08-19T14:03:48.795 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-08-19T14:03:48.942 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-osd-2:17.2.7-1406.gdf62bdcc.el9.x86_64 31/34 2024-08-19T14:03:48.971 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-osd-2:17.2.7-1406.gdf62bdcc.el9.x86_64 31/34 2024-08-19T14:03:48.972 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-19T14:03:48.972 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-08-19T14:03:48.972 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-08-19T14:03:48.972 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-08-19T14:03:48.972 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:03:48.985 INFO:teuthology.orchestra.run.smithi032.stdout:cephadm-17.2.7-1406.gdf62bdcc.el9.noarch.rpm 436 kB/s | 83 kB 00:00 2024-08-19T14:03:48.986 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-08-19T14:03:48.986 INFO:teuthology.orchestra.run.smithi032.stdout:Total 432 kB/s | 83 kB 00:00 2024-08-19T14:03:48.986 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-08-19T14:03:48.992 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-08-19T14:03:48.992 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-08-19T14:03:48.994 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-08-19T14:03:48.994 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-08-19T14:03:49.082 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-08-19T14:03:49.360 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mon-2:17.2.7-1406.gdf62bdcc.el9.x86_64 32/34 2024-08-19T14:03:49.384 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mon-2:17.2.7-1406.gdf62bdcc.el9.x86_64 32/34 2024-08-19T14:03:49.384 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-19T14:03:49.384 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-08-19T14:03:49.384 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-08-19T14:03:49.384 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-08-19T14:03:49.384 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:03:49.544 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: cephadm-2:17.2.7-1406.gdf62bdcc.el9.noarch 1/1 2024-08-19T14:03:49.594 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mds-2:17.2.7-1406.gdf62bdcc.el9.x86_64 33/34 2024-08-19T14:03:49.618 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mds-2:17.2.7-1406.gdf62bdcc.el9.x86_64 33/34 2024-08-19T14:03:49.618 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-19T14:03:49.618 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-08-19T14:03:49.618 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-08-19T14:03:49.618 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-08-19T14:03:49.618 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:03:49.641 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : cephadm-2:17.2.7-1406.gdf62bdcc.el9.noarch 1/1 2024-08-19T14:03:49.683 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-2:17.2.7-1406.gdf62bdcc.el9.x86_64 34/34 2024-08-19T14:03:49.975 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: cephadm-2:17.2.7-1406.gdf62bdcc.el9.noarch 1/1 2024-08-19T14:03:50.017 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-2:17.2.7-1406.gdf62bdcc.el9.x86_64 34/34 2024-08-19T14:03:50.017 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-2:17.2.7-1406.gdf62bdcc.el9.x86_64 1/34 2024-08-19T14:03:50.018 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mds-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2/34 2024-08-19T14:03:50.018 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-2:17.2.7-1406.gdf62bdcc.el9.x86_64 3/34 2024-08-19T14:03:50.018 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mon-2:17.2.7-1406.gdf62bdcc.el9.x86_64 4/34 2024-08-19T14:03:50.018 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-osd-2:17.2.7-1406.gdf62bdcc.el9.x86_64 5/34 2024-08-19T14:03:50.018 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-1406.gdf62bdcc.el9. 6/34 2024-08-19T14:03:50.018 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/34 2024-08-19T14:03:50.018 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/34 2024-08-19T14:03:50.018 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/34 2024-08-19T14:03:50.018 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/34 2024-08-19T14:03:50.018 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/34 2024-08-19T14:03:50.018 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/34 2024-08-19T14:03:50.018 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/34 2024-08-19T14:03:50.018 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/34 2024-08-19T14:03:50.019 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 15/34 2024-08-19T14:03:50.019 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/34 2024-08-19T14:03:50.019 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/34 2024-08-19T14:03:50.019 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 18/34 2024-08-19T14:03:50.019 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 19/34 2024-08-19T14:03:50.019 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 20/34 2024-08-19T14:03:50.019 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 21/34 2024-08-19T14:03:50.019 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 22/34 2024-08-19T14:03:50.019 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 23/34 2024-08-19T14:03:50.019 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 24/34 2024-08-19T14:03:50.019 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 25/34 2024-08-19T14:03:50.019 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 26/34 2024-08-19T14:03:50.019 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 27/34 2024-08-19T14:03:50.019 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 28/34 2024-08-19T14:03:50.020 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 29/34 2024-08-19T14:03:50.020 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 30/34 2024-08-19T14:03:50.020 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 31/34 2024-08-19T14:03:50.020 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 32/34 2024-08-19T14:03:50.020 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 33/34 2024-08-19T14:03:50.226 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : cephadm-2:17.2.7-1406.gdf62bdcc.el9.noarch 1/1 2024-08-19T14:03:50.227 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-19T14:03:50.227 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-08-19T14:03:50.227 INFO:teuthology.orchestra.run.smithi032.stdout: cephadm-2:17.2.7-1406.gdf62bdcc.el9.noarch 2024-08-19T14:03:50.227 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-19T14:03:50.227 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-08-19T14:03:50.431 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph-immutable-object-cache 2024-08-19T14:03:50.981 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:00:51 ago on Mon 19 Aug 2024 02:02:59 PM UTC. 2024-08-19T14:03:51.033 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 34/34 2024-08-19T14:03:51.034 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-08-19T14:03:51.034 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-08-19T14:03:51.034 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2024-08-19T14:03:51.034 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mds-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2024-08-19T14:03:51.034 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2024-08-19T14:03:51.034 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-modules-core-2:17.2.7-1406.gdf62bdcc.el9.noarch 2024-08-19T14:03:51.034 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mon-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2024-08-19T14:03:51.034 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-osd-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2024-08-19T14:03:51.034 INFO:teuthology.orchestra.run.smithi045.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-08-19T14:03:51.034 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-08-19T14:03:51.034 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-08-19T14:03:51.034 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-08-19T14:03:51.035 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-08-19T14:03:51.035 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-08-19T14:03:51.035 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-08-19T14:03:51.035 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-08-19T14:03:51.035 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-08-19T14:03:51.035 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-08-19T14:03:51.035 INFO:teuthology.orchestra.run.smithi045.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-08-19T14:03:51.035 INFO:teuthology.orchestra.run.smithi045.stdout: python3-mako-1.1.4-6.el9.noarch 2024-08-19T14:03:51.035 INFO:teuthology.orchestra.run.smithi045.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-08-19T14:03:51.035 INFO:teuthology.orchestra.run.smithi045.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-08-19T14:03:51.035 INFO:teuthology.orchestra.run.smithi045.stdout: python3-packaging-20.9-5.el9.noarch 2024-08-19T14:03:51.035 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-08-19T14:03:51.035 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ply-3.11-14.el9.noarch 2024-08-19T14:03:51.036 INFO:teuthology.orchestra.run.smithi045.stdout: python3-portend-3.1.0-2.el9.noarch 2024-08-19T14:03:51.036 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-08-19T14:03:51.036 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pycparser-2.20-6.el9.noarch 2024-08-19T14:03:51.036 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pytz-2021.1-5.el9.noarch 2024-08-19T14:03:51.036 INFO:teuthology.orchestra.run.smithi045.stdout: python3-requests-2.25.1-8.el9.noarch 2024-08-19T14:03:51.036 INFO:teuthology.orchestra.run.smithi045.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-08-19T14:03:51.036 INFO:teuthology.orchestra.run.smithi045.stdout: python3-toml-0.10.2-6.el9.noarch 2024-08-19T14:03:51.036 INFO:teuthology.orchestra.run.smithi045.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-08-19T14:03:51.036 INFO:teuthology.orchestra.run.smithi045.stdout: python3-webob-1.8.7-6.el9.noarch 2024-08-19T14:03:51.036 INFO:teuthology.orchestra.run.smithi045.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-08-19T14:03:51.036 INFO:teuthology.orchestra.run.smithi045.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-08-19T14:03:51.036 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-08-19T14:03:51.036 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-08-19T14:03:51.111 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-08-19T14:03:51.112 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-08-19T14:03:51.112 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repo Size 2024-08-19T14:03:51.112 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-08-19T14:03:51.112 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-08-19T14:03:51.112 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-1406.gdf62bdcc.el9 ceph 137 k 2024-08-19T14:03:51.113 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-19T14:03:51.113 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-08-19T14:03:51.113 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-08-19T14:03:51.113 INFO:teuthology.orchestra.run.smithi032.stdout:Install 1 Package 2024-08-19T14:03:51.113 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-19T14:03:51.113 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 137 k 2024-08-19T14:03:51.113 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 379 k 2024-08-19T14:03:51.113 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-08-19T14:03:51.263 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-2:17.2.7-1406.gdf62bdcc.el9.x86_64 34/34 2024-08-19T14:03:51.264 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-2:17.2.7-1406.gdf62bdcc.el9.x86_64 1/34 2024-08-19T14:03:51.264 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mds-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2/34 2024-08-19T14:03:51.264 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-2:17.2.7-1406.gdf62bdcc.el9.x86_64 3/34 2024-08-19T14:03:51.264 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mon-2:17.2.7-1406.gdf62bdcc.el9.x86_64 4/34 2024-08-19T14:03:51.264 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-osd-2:17.2.7-1406.gdf62bdcc.el9.x86_64 5/34 2024-08-19T14:03:51.264 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-1406.gdf62bdcc.el9. 6/34 2024-08-19T14:03:51.264 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/34 2024-08-19T14:03:51.264 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/34 2024-08-19T14:03:51.264 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/34 2024-08-19T14:03:51.264 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/34 2024-08-19T14:03:51.265 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/34 2024-08-19T14:03:51.265 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/34 2024-08-19T14:03:51.265 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/34 2024-08-19T14:03:51.265 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/34 2024-08-19T14:03:51.265 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 15/34 2024-08-19T14:03:51.265 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/34 2024-08-19T14:03:51.265 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/34 2024-08-19T14:03:51.265 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 18/34 2024-08-19T14:03:51.265 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 19/34 2024-08-19T14:03:51.265 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 20/34 2024-08-19T14:03:51.265 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 21/34 2024-08-19T14:03:51.265 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 22/34 2024-08-19T14:03:51.265 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 23/34 2024-08-19T14:03:51.266 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 24/34 2024-08-19T14:03:51.266 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 25/34 2024-08-19T14:03:51.267 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 26/34 2024-08-19T14:03:51.267 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 27/34 2024-08-19T14:03:51.267 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 28/34 2024-08-19T14:03:51.267 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 29/34 2024-08-19T14:03:51.267 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 30/34 2024-08-19T14:03:51.267 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 31/34 2024-08-19T14:03:51.267 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 32/34 2024-08-19T14:03:51.267 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 33/34 2024-08-19T14:03:51.309 INFO:teuthology.orchestra.run.smithi032.stdout:ceph-immutable-object-cache-17.2.7-1406.gdf62bd 704 kB/s | 137 kB 00:00 2024-08-19T14:03:51.309 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-08-19T14:03:51.309 INFO:teuthology.orchestra.run.smithi032.stdout:Total 698 kB/s | 137 kB 00:00 2024-08-19T14:03:51.310 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-08-19T14:03:51.317 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-08-19T14:03:51.318 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-08-19T14:03:51.376 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-08-19T14:03:51.376 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-08-19T14:03:51.381 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-base 2024-08-19T14:03:51.524 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-08-19T14:03:51.609 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-1406.gdf62bdcc. 1/1 2024-08-19T14:03:51.635 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-1406.gdf62bdcc. 1/1 2024-08-19T14:03:51.636 INFO:teuthology.orchestra.run.smithi032.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-19T14:03:51.636 INFO:teuthology.orchestra.run.smithi032.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-08-19T14:03:51.636 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-19T14:03:51.933 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:49 ago on Mon 19 Aug 2024 02:03:02 PM UTC. 2024-08-19T14:03:52.019 INFO:teuthology.orchestra.run.smithi045.stdout:Package ceph-base-2:17.2.7-1406.gdf62bdcc.el9.x86_64 is already installed. 2024-08-19T14:03:52.063 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-08-19T14:03:52.064 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2024-08-19T14:03:52.064 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-08-19T14:03:52.123 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install cephadm 2024-08-19T14:03:52.271 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 34/34 2024-08-19T14:03:52.271 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:03:52.271 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-08-19T14:03:52.271 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2024-08-19T14:03:52.272 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mds-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2024-08-19T14:03:52.272 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2024-08-19T14:03:52.272 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-modules-core-2:17.2.7-1406.gdf62bdcc.el9.noarch 2024-08-19T14:03:52.272 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mon-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2024-08-19T14:03:52.272 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-osd-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2024-08-19T14:03:52.272 INFO:teuthology.orchestra.run.smithi031.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-08-19T14:03:52.272 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-08-19T14:03:52.272 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-08-19T14:03:52.272 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-08-19T14:03:52.272 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-08-19T14:03:52.272 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-08-19T14:03:52.272 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-08-19T14:03:52.272 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-08-19T14:03:52.273 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-08-19T14:03:52.273 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-08-19T14:03:52.273 INFO:teuthology.orchestra.run.smithi031.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-08-19T14:03:52.273 INFO:teuthology.orchestra.run.smithi031.stdout: python3-mako-1.1.4-6.el9.noarch 2024-08-19T14:03:52.273 INFO:teuthology.orchestra.run.smithi031.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-08-19T14:03:52.273 INFO:teuthology.orchestra.run.smithi031.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-08-19T14:03:52.273 INFO:teuthology.orchestra.run.smithi031.stdout: python3-packaging-20.9-5.el9.noarch 2024-08-19T14:03:52.273 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-08-19T14:03:52.273 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ply-3.11-14.el9.noarch 2024-08-19T14:03:52.273 INFO:teuthology.orchestra.run.smithi031.stdout: python3-portend-3.1.0-2.el9.noarch 2024-08-19T14:03:52.273 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-08-19T14:03:52.273 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pycparser-2.20-6.el9.noarch 2024-08-19T14:03:52.273 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pytz-2021.1-5.el9.noarch 2024-08-19T14:03:52.274 INFO:teuthology.orchestra.run.smithi031.stdout: python3-requests-2.25.1-8.el9.noarch 2024-08-19T14:03:52.274 INFO:teuthology.orchestra.run.smithi031.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-08-19T14:03:52.274 INFO:teuthology.orchestra.run.smithi031.stdout: python3-toml-0.10.2-6.el9.noarch 2024-08-19T14:03:52.274 INFO:teuthology.orchestra.run.smithi031.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-08-19T14:03:52.274 INFO:teuthology.orchestra.run.smithi031.stdout: python3-webob-1.8.7-6.el9.noarch 2024-08-19T14:03:52.274 INFO:teuthology.orchestra.run.smithi031.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-08-19T14:03:52.274 INFO:teuthology.orchestra.run.smithi031.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-08-19T14:03:52.274 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:03:52.274 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-08-19T14:03:52.514 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-1406.gdf62bdcc. 1/1 2024-08-19T14:03:52.514 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-19T14:03:52.515 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-08-19T14:03:52.515 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-immutable-object-cache-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2024-08-19T14:03:52.515 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-19T14:03:52.515 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-08-19T14:03:52.630 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-base 2024-08-19T14:03:52.669 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:50 ago on Mon 19 Aug 2024 02:03:02 PM UTC. 2024-08-19T14:03:52.746 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph-mgr 2024-08-19T14:03:52.799 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-08-19T14:03:52.800 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-08-19T14:03:52.800 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2024-08-19T14:03:52.800 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-08-19T14:03:52.800 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-08-19T14:03:52.800 INFO:teuthology.orchestra.run.smithi045.stdout: cephadm noarch 2:17.2.7-1406.gdf62bdcc.el9 ceph-noarch 83 k 2024-08-19T14:03:52.800 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-08-19T14:03:52.800 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-08-19T14:03:52.801 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-08-19T14:03:52.801 INFO:teuthology.orchestra.run.smithi045.stdout:Install 1 Package 2024-08-19T14:03:52.801 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-08-19T14:03:52.801 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 83 k 2024-08-19T14:03:52.801 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 375 k 2024-08-19T14:03:52.801 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-08-19T14:03:52.990 INFO:teuthology.orchestra.run.smithi045.stdout:cephadm-17.2.7-1406.gdf62bdcc.el9.noarch.rpm 438 kB/s | 83 kB 00:00 2024-08-19T14:03:52.991 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-08-19T14:03:52.991 INFO:teuthology.orchestra.run.smithi045.stdout:Total 434 kB/s | 83 kB 00:00 2024-08-19T14:03:52.991 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-08-19T14:03:52.999 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-08-19T14:03:52.999 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-08-19T14:03:52.999 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-08-19T14:03:52.999 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-08-19T14:03:53.064 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-08-19T14:03:53.179 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:00:50 ago on Mon 19 Aug 2024 02:03:03 PM UTC. 2024-08-19T14:03:53.265 INFO:teuthology.orchestra.run.smithi031.stdout:Package ceph-base-2:17.2.7-1406.gdf62bdcc.el9.x86_64 is already installed. 2024-08-19T14:03:53.295 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:00:54 ago on Mon 19 Aug 2024 02:02:59 PM UTC. 2024-08-19T14:03:53.308 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-08-19T14:03:53.309 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2024-08-19T14:03:53.309 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-08-19T14:03:53.373 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install cephadm 2024-08-19T14:03:53.383 INFO:teuthology.orchestra.run.smithi032.stdout:Package ceph-mgr-2:17.2.7-1406.gdf62bdcc.el9.x86_64 is already installed. 2024-08-19T14:03:53.428 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-08-19T14:03:53.429 INFO:teuthology.orchestra.run.smithi032.stdout:Nothing to do. 2024-08-19T14:03:53.429 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-08-19T14:03:53.464 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: cephadm-2:17.2.7-1406.gdf62bdcc.el9.noarch 1/1 2024-08-19T14:03:53.484 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph-fuse 2024-08-19T14:03:53.561 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : cephadm-2:17.2.7-1406.gdf62bdcc.el9.noarch 1/1 2024-08-19T14:03:53.907 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: cephadm-2:17.2.7-1406.gdf62bdcc.el9.noarch 1/1 2024-08-19T14:03:53.917 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:00:50 ago on Mon 19 Aug 2024 02:03:03 PM UTC. 2024-08-19T14:03:54.032 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:00:55 ago on Mon 19 Aug 2024 02:02:59 PM UTC. 2024-08-19T14:03:54.046 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-08-19T14:03:54.047 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-08-19T14:03:54.047 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2024-08-19T14:03:54.047 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-08-19T14:03:54.047 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-08-19T14:03:54.047 INFO:teuthology.orchestra.run.smithi031.stdout: cephadm noarch 2:17.2.7-1406.gdf62bdcc.el9 ceph-noarch 83 k 2024-08-19T14:03:54.047 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:03:54.047 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-08-19T14:03:54.047 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-08-19T14:03:54.047 INFO:teuthology.orchestra.run.smithi031.stdout:Install 1 Package 2024-08-19T14:03:54.048 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:03:54.048 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 83 k 2024-08-19T14:03:54.048 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 375 k 2024-08-19T14:03:54.048 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-08-19T14:03:54.157 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : cephadm-2:17.2.7-1406.gdf62bdcc.el9.noarch 1/1 2024-08-19T14:03:54.158 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-08-19T14:03:54.158 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-08-19T14:03:54.158 INFO:teuthology.orchestra.run.smithi045.stdout: cephadm-2:17.2.7-1406.gdf62bdcc.el9.noarch 2024-08-19T14:03:54.158 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-08-19T14:03:54.158 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-08-19T14:03:54.162 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-08-19T14:03:54.163 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-08-19T14:03:54.163 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repository Size 2024-08-19T14:03:54.163 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-08-19T14:03:54.163 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-08-19T14:03:54.163 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-fuse x86_64 2:17.2.7-1406.gdf62bdcc.el9 ceph 813 k 2024-08-19T14:03:54.164 INFO:teuthology.orchestra.run.smithi032.stdout:Installing dependencies: 2024-08-19T14:03:54.164 INFO:teuthology.orchestra.run.smithi032.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-08-19T14:03:54.164 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-19T14:03:54.164 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-08-19T14:03:54.164 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-08-19T14:03:54.164 INFO:teuthology.orchestra.run.smithi032.stdout:Install 2 Packages 2024-08-19T14:03:54.164 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-19T14:03:54.165 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 893 k 2024-08-19T14:03:54.165 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 2.6 M 2024-08-19T14:03:54.165 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-08-19T14:03:54.233 INFO:teuthology.orchestra.run.smithi031.stdout:cephadm-17.2.7-1406.gdf62bdcc.el9.noarch.rpm 448 kB/s | 83 kB 00:00 2024-08-19T14:03:54.234 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-08-19T14:03:54.234 INFO:teuthology.orchestra.run.smithi031.stdout:Total 444 kB/s | 83 kB 00:00 2024-08-19T14:03:54.234 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-08-19T14:03:54.239 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-08-19T14:03:54.239 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-08-19T14:03:54.241 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-08-19T14:03:54.241 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-08-19T14:03:54.319 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-08-19T14:03:54.378 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-immutable-object-cache 2024-08-19T14:03:54.554 INFO:teuthology.orchestra.run.smithi032.stdout:(1/2): ceph-fuse-17.2.7-1406.gdf62bdcc.el9.x86_ 2.8 MB/s | 813 kB 00:00 2024-08-19T14:03:54.789 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: cephadm-2:17.2.7-1406.gdf62bdcc.el9.noarch 1/1 2024-08-19T14:03:54.895 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : cephadm-2:17.2.7-1406.gdf62bdcc.el9.noarch 1/1 2024-08-19T14:03:54.921 INFO:teuthology.orchestra.run.smithi032.stdout:(2/2): fuse-2.9.9-16.el9.x86_64.rpm 122 kB/s | 80 kB 00:00 2024-08-19T14:03:54.922 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-08-19T14:03:54.922 INFO:teuthology.orchestra.run.smithi032.stdout:Total 1.2 MB/s | 893 kB 00:00 2024-08-19T14:03:54.927 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:52 ago on Mon 19 Aug 2024 02:03:02 PM UTC. 2024-08-19T14:03:54.936 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-08-19T14:03:54.961 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-08-19T14:03:54.961 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-08-19T14:03:55.038 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-08-19T14:03:55.042 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-08-19T14:03:55.061 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-08-19T14:03:55.061 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-08-19T14:03:55.061 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repo Size 2024-08-19T14:03:55.061 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-08-19T14:03:55.061 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-08-19T14:03:55.062 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-1406.gdf62bdcc.el9 ceph 137 k 2024-08-19T14:03:55.062 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-08-19T14:03:55.062 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-08-19T14:03:55.062 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-08-19T14:03:55.062 INFO:teuthology.orchestra.run.smithi045.stdout:Install 1 Package 2024-08-19T14:03:55.062 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-08-19T14:03:55.063 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 137 k 2024-08-19T14:03:55.063 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 379 k 2024-08-19T14:03:55.063 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-08-19T14:03:55.216 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-08-19T14:03:55.231 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: cephadm-2:17.2.7-1406.gdf62bdcc.el9.noarch 1/1 2024-08-19T14:03:55.279 INFO:teuthology.orchestra.run.smithi045.stdout:ceph-immutable-object-cache-17.2.7-1406.gdf62bd 637 kB/s | 137 kB 00:00 2024-08-19T14:03:55.279 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-08-19T14:03:55.280 INFO:teuthology.orchestra.run.smithi045.stdout:Total 633 kB/s | 137 kB 00:00 2024-08-19T14:03:55.280 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-08-19T14:03:55.287 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-08-19T14:03:55.287 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-08-19T14:03:55.346 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-08-19T14:03:55.346 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-08-19T14:03:55.353 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-08-19T14:03:55.436 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-fuse-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2/2 2024-08-19T14:03:55.466 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : cephadm-2:17.2.7-1406.gdf62bdcc.el9.noarch 1/1 2024-08-19T14:03:55.466 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:03:55.466 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-08-19T14:03:55.466 INFO:teuthology.orchestra.run.smithi031.stdout: cephadm-2:17.2.7-1406.gdf62bdcc.el9.noarch 2024-08-19T14:03:55.466 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:03:55.466 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-08-19T14:03:55.488 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-08-19T14:03:55.546 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-1406.gdf62bdcc. 1/1 2024-08-19T14:03:55.571 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-1406.gdf62bdcc. 1/1 2024-08-19T14:03:55.571 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-19T14:03:55.571 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-08-19T14:03:55.571 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-08-19T14:03:55.656 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-immutable-object-cache 2024-08-19T14:03:56.095 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-fuse-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2/2 2024-08-19T14:03:56.095 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-fuse-2:17.2.7-1406.gdf62bdcc.el9.x86_64 1/2 2024-08-19T14:03:56.201 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:00:53 ago on Mon 19 Aug 2024 02:03:03 PM UTC. 2024-08-19T14:03:56.324 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-08-19T14:03:56.324 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-19T14:03:56.324 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-08-19T14:03:56.324 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-fuse-2:17.2.7-1406.gdf62bdcc.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-08-19T14:03:56.324 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-19T14:03:56.324 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-08-19T14:03:56.331 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-08-19T14:03:56.332 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-08-19T14:03:56.332 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repo Size 2024-08-19T14:03:56.332 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-08-19T14:03:56.332 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-08-19T14:03:56.332 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-1406.gdf62bdcc.el9 ceph 137 k 2024-08-19T14:03:56.332 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:03:56.332 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-08-19T14:03:56.333 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-08-19T14:03:56.333 INFO:teuthology.orchestra.run.smithi031.stdout:Install 1 Package 2024-08-19T14:03:56.333 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:03:56.333 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 137 k 2024-08-19T14:03:56.333 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 379 k 2024-08-19T14:03:56.333 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-08-19T14:03:56.452 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-1406.gdf62bdcc. 1/1 2024-08-19T14:03:56.452 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-08-19T14:03:56.452 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-08-19T14:03:56.452 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-immutable-object-cache-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2024-08-19T14:03:56.453 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-08-19T14:03:56.453 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-08-19T14:03:56.554 INFO:teuthology.orchestra.run.smithi031.stdout:ceph-immutable-object-cache-17.2.7-1406.gdf62bd 623 kB/s | 137 kB 00:00 2024-08-19T14:03:56.555 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-08-19T14:03:56.555 INFO:teuthology.orchestra.run.smithi031.stdout:Total 618 kB/s | 137 kB 00:00 2024-08-19T14:03:56.555 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-08-19T14:03:56.564 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-08-19T14:03:56.564 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-08-19T14:03:56.569 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install librados-devel 2024-08-19T14:03:56.622 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-08-19T14:03:56.623 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-08-19T14:03:56.696 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-mgr 2024-08-19T14:03:56.777 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-08-19T14:03:56.847 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-1406.gdf62bdcc. 1/1 2024-08-19T14:03:56.872 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-1406.gdf62bdcc. 1/1 2024-08-19T14:03:56.872 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-19T14:03:56.872 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-08-19T14:03:56.872 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:03:57.116 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:00:58 ago on Mon 19 Aug 2024 02:02:59 PM UTC. 2024-08-19T14:03:57.248 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-08-19T14:03:57.249 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-08-19T14:03:57.249 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repo Size 2024-08-19T14:03:57.249 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-08-19T14:03:57.249 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-08-19T14:03:57.249 INFO:teuthology.orchestra.run.smithi032.stdout: librados-devel x86_64 2:17.2.7-1406.gdf62bdcc.el9 ceph 126 k 2024-08-19T14:03:57.249 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-19T14:03:57.250 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-08-19T14:03:57.250 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-08-19T14:03:57.250 INFO:teuthology.orchestra.run.smithi032.stdout:Install 1 Package 2024-08-19T14:03:57.250 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-19T14:03:57.250 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 126 k 2024-08-19T14:03:57.250 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 448 k 2024-08-19T14:03:57.250 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-08-19T14:03:57.250 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:55 ago on Mon 19 Aug 2024 02:03:02 PM UTC. 2024-08-19T14:03:57.337 INFO:teuthology.orchestra.run.smithi045.stdout:Package ceph-mgr-2:17.2.7-1406.gdf62bdcc.el9.x86_64 is already installed. 2024-08-19T14:03:57.380 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-08-19T14:03:57.381 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2024-08-19T14:03:57.381 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-08-19T14:03:57.443 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-fuse 2024-08-19T14:03:57.443 INFO:teuthology.orchestra.run.smithi032.stdout:librados-devel-17.2.7-1406.gdf62bdcc.el9.x86_64 654 kB/s | 126 kB 00:00 2024-08-19T14:03:57.444 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-08-19T14:03:57.444 INFO:teuthology.orchestra.run.smithi032.stdout:Total 648 kB/s | 126 kB 00:00 2024-08-19T14:03:57.444 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-08-19T14:03:57.450 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-08-19T14:03:57.450 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-08-19T14:03:57.509 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-08-19T14:03:57.510 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-08-19T14:03:57.665 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-08-19T14:03:57.748 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-1406.gdf62bdcc. 1/1 2024-08-19T14:03:57.748 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:03:57.748 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-08-19T14:03:57.748 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-immutable-object-cache-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2024-08-19T14:03:57.749 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:03:57.749 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-08-19T14:03:57.782 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : librados-devel-2:17.2.7-1406.gdf62bdcc.el9.x86_64 1/1 2024-08-19T14:03:57.961 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-mgr 2024-08-19T14:03:57.993 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:55 ago on Mon 19 Aug 2024 02:03:02 PM UTC. 2024-08-19T14:03:58.125 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-08-19T14:03:58.125 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-08-19T14:03:58.125 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2024-08-19T14:03:58.125 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-08-19T14:03:58.125 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-08-19T14:03:58.125 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-fuse x86_64 2:17.2.7-1406.gdf62bdcc.el9 ceph 813 k 2024-08-19T14:03:58.126 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2024-08-19T14:03:58.126 INFO:teuthology.orchestra.run.smithi045.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-08-19T14:03:58.126 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-08-19T14:03:58.126 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-08-19T14:03:58.126 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-08-19T14:03:58.126 INFO:teuthology.orchestra.run.smithi045.stdout:Install 2 Packages 2024-08-19T14:03:58.126 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-08-19T14:03:58.127 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 893 k 2024-08-19T14:03:58.127 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 2.6 M 2024-08-19T14:03:58.127 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-08-19T14:03:58.172 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: librados-devel-2:17.2.7-1406.gdf62bdcc.el9.x86_64 1/1 2024-08-19T14:03:58.364 INFO:teuthology.orchestra.run.smithi045.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 566 kB/s | 80 kB 00:00 2024-08-19T14:03:58.400 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librados-devel-2:17.2.7-1406.gdf62bdcc.el9.x86_64 1/1 2024-08-19T14:03:58.401 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-19T14:03:58.401 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-08-19T14:03:58.401 INFO:teuthology.orchestra.run.smithi032.stdout: librados-devel-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2024-08-19T14:03:58.401 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-19T14:03:58.401 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-08-19T14:03:58.498 INFO:teuthology.orchestra.run.smithi045.stdout:(2/2): ceph-fuse-17.2.7-1406.gdf62bdcc.el9.x86_ 2.9 MB/s | 813 kB 00:00 2024-08-19T14:03:58.498 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-08-19T14:03:58.499 INFO:teuthology.orchestra.run.smithi045.stdout:Total 2.3 MB/s | 893 kB 00:00 2024-08-19T14:03:58.510 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-08-19T14:03:58.521 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:00:55 ago on Mon 19 Aug 2024 02:03:03 PM UTC. 2024-08-19T14:03:58.533 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-08-19T14:03:58.533 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-08-19T14:03:58.604 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install libcephfs2 2024-08-19T14:03:58.605 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-08-19T14:03:58.606 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-08-19T14:03:58.613 INFO:teuthology.orchestra.run.smithi031.stdout:Package ceph-mgr-2:17.2.7-1406.gdf62bdcc.el9.x86_64 is already installed. 2024-08-19T14:03:58.657 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-08-19T14:03:58.657 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2024-08-19T14:03:58.658 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-08-19T14:03:58.726 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-fuse 2024-08-19T14:03:58.918 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-08-19T14:03:59.064 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-08-19T14:03:59.129 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-fuse-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2/2 2024-08-19T14:03:59.147 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:01:00 ago on Mon 19 Aug 2024 02:02:59 PM UTC. 2024-08-19T14:03:59.233 INFO:teuthology.orchestra.run.smithi032.stdout:Package libcephfs2-2:17.2.7-1406.gdf62bdcc.el9.x86_64 is already installed. 2024-08-19T14:03:59.273 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:00:56 ago on Mon 19 Aug 2024 02:03:03 PM UTC. 2024-08-19T14:03:59.276 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-08-19T14:03:59.277 INFO:teuthology.orchestra.run.smithi032.stdout:Nothing to do. 2024-08-19T14:03:59.277 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-08-19T14:03:59.354 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install libcephfs-devel 2024-08-19T14:03:59.403 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-08-19T14:03:59.404 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-08-19T14:03:59.404 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2024-08-19T14:03:59.404 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-08-19T14:03:59.404 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-08-19T14:03:59.404 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-fuse x86_64 2:17.2.7-1406.gdf62bdcc.el9 ceph 813 k 2024-08-19T14:03:59.404 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2024-08-19T14:03:59.404 INFO:teuthology.orchestra.run.smithi031.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-08-19T14:03:59.404 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:03:59.405 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-08-19T14:03:59.405 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-08-19T14:03:59.405 INFO:teuthology.orchestra.run.smithi031.stdout:Install 2 Packages 2024-08-19T14:03:59.405 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:03:59.405 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 893 k 2024-08-19T14:03:59.405 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 2.6 M 2024-08-19T14:03:59.406 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-08-19T14:03:59.591 INFO:teuthology.orchestra.run.smithi031.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 573 kB/s | 80 kB 00:00 2024-08-19T14:03:59.716 INFO:teuthology.orchestra.run.smithi031.stdout:(2/2): ceph-fuse-17.2.7-1406.gdf62bdcc.el9.x86_ 3.0 MB/s | 813 kB 00:00 2024-08-19T14:03:59.717 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-08-19T14:03:59.717 INFO:teuthology.orchestra.run.smithi031.stdout:Total 2.8 MB/s | 893 kB 00:00 2024-08-19T14:03:59.728 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-08-19T14:03:59.751 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-08-19T14:03:59.751 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-08-19T14:03:59.786 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-fuse-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2/2 2024-08-19T14:03:59.787 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-fuse-2:17.2.7-1406.gdf62bdcc.el9.x86_64 1/2 2024-08-19T14:03:59.825 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-08-19T14:03:59.825 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-08-19T14:03:59.896 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:01:00 ago on Mon 19 Aug 2024 02:02:59 PM UTC. 2024-08-19T14:04:00.005 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-08-19T14:04:00.027 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-08-19T14:04:00.027 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-08-19T14:04:00.028 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repo Size 2024-08-19T14:04:00.028 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-08-19T14:04:00.028 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-08-19T14:04:00.028 INFO:teuthology.orchestra.run.smithi032.stdout: libcephfs-devel x86_64 2:17.2.7-1406.gdf62bdcc.el9 ceph 25 k 2024-08-19T14:04:00.028 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-19T14:04:00.028 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-08-19T14:04:00.028 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-08-19T14:04:00.028 INFO:teuthology.orchestra.run.smithi032.stdout:Install 1 Package 2024-08-19T14:04:00.028 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-19T14:04:00.028 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 25 k 2024-08-19T14:04:00.028 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 110 k 2024-08-19T14:04:00.029 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-08-19T14:04:00.029 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-08-19T14:04:00.029 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-08-19T14:04:00.029 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-08-19T14:04:00.029 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-fuse-2:17.2.7-1406.gdf62bdcc.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-08-19T14:04:00.029 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-08-19T14:04:00.029 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-08-19T14:04:00.149 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-08-19T14:04:00.181 INFO:teuthology.orchestra.run.smithi032.stdout:libcephfs-devel-17.2.7-1406.gdf62bdcc.el9.x86_6 164 kB/s | 25 kB 00:00 2024-08-19T14:04:00.181 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-08-19T14:04:00.182 INFO:teuthology.orchestra.run.smithi032.stdout:Total 162 kB/s | 25 kB 00:00 2024-08-19T14:04:00.182 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-08-19T14:04:00.186 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-08-19T14:04:00.186 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-08-19T14:04:00.187 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-08-19T14:04:00.187 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-08-19T14:04:00.214 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-fuse-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2/2 2024-08-19T14:04:00.250 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install librados-devel 2024-08-19T14:04:00.273 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-08-19T14:04:00.377 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libcephfs-devel-2:17.2.7-1406.gdf62bdcc.el9.x86_64 1/1 2024-08-19T14:04:00.743 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-1406.gdf62bdcc.el9.x86_64 1/1 2024-08-19T14:04:00.801 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:58 ago on Mon 19 Aug 2024 02:03:02 PM UTC. 2024-08-19T14:04:00.883 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-fuse-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2/2 2024-08-19T14:04:00.883 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-fuse-2:17.2.7-1406.gdf62bdcc.el9.x86_64 1/2 2024-08-19T14:04:00.932 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-08-19T14:04:00.933 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-08-19T14:04:00.933 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repo Size 2024-08-19T14:04:00.933 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-08-19T14:04:00.933 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-08-19T14:04:00.933 INFO:teuthology.orchestra.run.smithi045.stdout: librados-devel x86_64 2:17.2.7-1406.gdf62bdcc.el9 ceph 126 k 2024-08-19T14:04:00.933 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-08-19T14:04:00.934 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-08-19T14:04:00.934 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-08-19T14:04:00.934 INFO:teuthology.orchestra.run.smithi045.stdout:Install 1 Package 2024-08-19T14:04:00.934 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-08-19T14:04:00.934 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 126 k 2024-08-19T14:04:00.934 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 448 k 2024-08-19T14:04:00.934 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-08-19T14:04:00.959 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libcephfs-devel-2:17.2.7-1406.gdf62bdcc.el9.x86_64 1/1 2024-08-19T14:04:00.959 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-19T14:04:00.959 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-08-19T14:04:00.959 INFO:teuthology.orchestra.run.smithi032.stdout: libcephfs-devel-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2024-08-19T14:04:00.959 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-19T14:04:00.959 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-08-19T14:04:01.109 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install librados2 2024-08-19T14:04:01.131 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-08-19T14:04:01.131 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:04:01.131 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-08-19T14:04:01.131 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-fuse-2:17.2.7-1406.gdf62bdcc.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-08-19T14:04:01.131 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:04:01.132 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-08-19T14:04:01.143 INFO:teuthology.orchestra.run.smithi045.stdout:librados-devel-17.2.7-1406.gdf62bdcc.el9.x86_64 603 kB/s | 126 kB 00:00 2024-08-19T14:04:01.144 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-08-19T14:04:01.144 INFO:teuthology.orchestra.run.smithi045.stdout:Total 599 kB/s | 126 kB 00:00 2024-08-19T14:04:01.144 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-08-19T14:04:01.150 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-08-19T14:04:01.150 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-08-19T14:04:01.211 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-08-19T14:04:01.211 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-08-19T14:04:01.356 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-08-19T14:04:01.393 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install librados-devel 2024-08-19T14:04:01.457 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : librados-devel-2:17.2.7-1406.gdf62bdcc.el9.x86_64 1/1 2024-08-19T14:04:01.659 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:01:02 ago on Mon 19 Aug 2024 02:02:59 PM UTC. 2024-08-19T14:04:01.746 INFO:teuthology.orchestra.run.smithi032.stdout:Package librados2-2:17.2.7-1406.gdf62bdcc.el9.x86_64 is already installed. 2024-08-19T14:04:01.789 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-08-19T14:04:01.790 INFO:teuthology.orchestra.run.smithi032.stdout:Nothing to do. 2024-08-19T14:04:01.790 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-08-19T14:04:01.847 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install librbd1 2024-08-19T14:04:01.932 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librados-devel-2:17.2.7-1406.gdf62bdcc.el9.x86_64 1/1 2024-08-19T14:04:01.949 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:00:58 ago on Mon 19 Aug 2024 02:03:03 PM UTC. 2024-08-19T14:04:02.082 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-08-19T14:04:02.083 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-08-19T14:04:02.083 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repo Size 2024-08-19T14:04:02.083 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-08-19T14:04:02.083 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-08-19T14:04:02.083 INFO:teuthology.orchestra.run.smithi031.stdout: librados-devel x86_64 2:17.2.7-1406.gdf62bdcc.el9 ceph 126 k 2024-08-19T14:04:02.083 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:04:02.083 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-08-19T14:04:02.083 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-08-19T14:04:02.083 INFO:teuthology.orchestra.run.smithi031.stdout:Install 1 Package 2024-08-19T14:04:02.084 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:04:02.084 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 126 k 2024-08-19T14:04:02.084 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 448 k 2024-08-19T14:04:02.084 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-08-19T14:04:02.184 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librados-devel-2:17.2.7-1406.gdf62bdcc.el9.x86_64 1/1 2024-08-19T14:04:02.185 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-08-19T14:04:02.185 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-08-19T14:04:02.185 INFO:teuthology.orchestra.run.smithi045.stdout: librados-devel-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2024-08-19T14:04:02.185 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-08-19T14:04:02.185 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-08-19T14:04:02.286 INFO:teuthology.orchestra.run.smithi031.stdout:librados-devel-17.2.7-1406.gdf62bdcc.el9.x86_64 623 kB/s | 126 kB 00:00 2024-08-19T14:04:02.287 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-08-19T14:04:02.287 INFO:teuthology.orchestra.run.smithi031.stdout:Total 619 kB/s | 126 kB 00:00 2024-08-19T14:04:02.287 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-08-19T14:04:02.293 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-08-19T14:04:02.293 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-08-19T14:04:02.352 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-08-19T14:04:02.352 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-08-19T14:04:02.387 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:01:03 ago on Mon 19 Aug 2024 02:02:59 PM UTC. 2024-08-19T14:04:02.403 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install libcephfs2 2024-08-19T14:04:02.474 INFO:teuthology.orchestra.run.smithi032.stdout:Package librbd1-2:17.2.7-1406.gdf62bdcc.el9.x86_64 is already installed. 2024-08-19T14:04:02.494 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-08-19T14:04:02.518 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-08-19T14:04:02.519 INFO:teuthology.orchestra.run.smithi032.stdout:Nothing to do. 2024-08-19T14:04:02.520 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-08-19T14:04:02.577 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install python3-rados 2024-08-19T14:04:02.582 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : librados-devel-2:17.2.7-1406.gdf62bdcc.el9.x86_64 1/1 2024-08-19T14:04:02.953 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:00 ago on Mon 19 Aug 2024 02:03:02 PM UTC. 2024-08-19T14:04:02.999 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librados-devel-2:17.2.7-1406.gdf62bdcc.el9.x86_64 1/1 2024-08-19T14:04:03.039 INFO:teuthology.orchestra.run.smithi045.stdout:Package libcephfs2-2:17.2.7-1406.gdf62bdcc.el9.x86_64 is already installed. 2024-08-19T14:04:03.083 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-08-19T14:04:03.084 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2024-08-19T14:04:03.084 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-08-19T14:04:03.117 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:01:04 ago on Mon 19 Aug 2024 02:02:59 PM UTC. 2024-08-19T14:04:03.165 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install libcephfs-devel 2024-08-19T14:04:03.204 INFO:teuthology.orchestra.run.smithi032.stdout:Package python3-rados-2:17.2.7-1406.gdf62bdcc.el9.x86_64 is already installed. 2024-08-19T14:04:03.248 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-08-19T14:04:03.249 INFO:teuthology.orchestra.run.smithi032.stdout:Nothing to do. 2024-08-19T14:04:03.249 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-08-19T14:04:03.255 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librados-devel-2:17.2.7-1406.gdf62bdcc.el9.x86_64 1/1 2024-08-19T14:04:03.255 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:04:03.255 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-08-19T14:04:03.255 INFO:teuthology.orchestra.run.smithi031.stdout: librados-devel-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2024-08-19T14:04:03.255 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:04:03.255 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-08-19T14:04:03.306 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install python3-rgw 2024-08-19T14:04:03.538 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install libcephfs2 2024-08-19T14:04:03.706 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:01 ago on Mon 19 Aug 2024 02:03:02 PM UTC. 2024-08-19T14:04:03.838 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-08-19T14:04:03.839 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-08-19T14:04:03.839 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repo Size 2024-08-19T14:04:03.839 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-08-19T14:04:03.839 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-08-19T14:04:03.839 INFO:teuthology.orchestra.run.smithi045.stdout: libcephfs-devel x86_64 2:17.2.7-1406.gdf62bdcc.el9 ceph 25 k 2024-08-19T14:04:03.839 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-08-19T14:04:03.840 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-08-19T14:04:03.840 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-08-19T14:04:03.840 INFO:teuthology.orchestra.run.smithi045.stdout:Install 1 Package 2024-08-19T14:04:03.840 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-08-19T14:04:03.840 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 25 k 2024-08-19T14:04:03.840 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 110 k 2024-08-19T14:04:03.840 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-08-19T14:04:03.847 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:01:04 ago on Mon 19 Aug 2024 02:02:59 PM UTC. 2024-08-19T14:04:03.935 INFO:teuthology.orchestra.run.smithi032.stdout:Package python3-rgw-2:17.2.7-1406.gdf62bdcc.el9.x86_64 is already installed. 2024-08-19T14:04:03.980 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-08-19T14:04:03.981 INFO:teuthology.orchestra.run.smithi032.stdout:Nothing to do. 2024-08-19T14:04:03.981 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-08-19T14:04:03.996 INFO:teuthology.orchestra.run.smithi045.stdout:libcephfs-devel-17.2.7-1406.gdf62bdcc.el9.x86_6 160 kB/s | 25 kB 00:00 2024-08-19T14:04:03.997 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-08-19T14:04:03.997 INFO:teuthology.orchestra.run.smithi045.stdout:Total 159 kB/s | 25 kB 00:00 2024-08-19T14:04:03.998 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-08-19T14:04:04.000 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-08-19T14:04:04.000 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-08-19T14:04:04.002 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-08-19T14:04:04.002 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-08-19T14:04:04.035 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install python3-cephfs 2024-08-19T14:04:04.088 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:01 ago on Mon 19 Aug 2024 02:03:03 PM UTC. 2024-08-19T14:04:04.175 INFO:teuthology.orchestra.run.smithi031.stdout:Package libcephfs2-2:17.2.7-1406.gdf62bdcc.el9.x86_64 is already installed. 2024-08-19T14:04:04.189 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-08-19T14:04:04.219 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-08-19T14:04:04.220 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2024-08-19T14:04:04.220 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-08-19T14:04:04.290 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install libcephfs-devel 2024-08-19T14:04:04.300 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libcephfs-devel-2:17.2.7-1406.gdf62bdcc.el9.x86_64 1/1 2024-08-19T14:04:04.584 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:01:05 ago on Mon 19 Aug 2024 02:02:59 PM UTC. 2024-08-19T14:04:04.619 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-1406.gdf62bdcc.el9.x86_64 1/1 2024-08-19T14:04:04.671 INFO:teuthology.orchestra.run.smithi032.stdout:Package python3-cephfs-2:17.2.7-1406.gdf62bdcc.el9.x86_64 is already installed. 2024-08-19T14:04:04.715 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-08-19T14:04:04.716 INFO:teuthology.orchestra.run.smithi032.stdout:Nothing to do. 2024-08-19T14:04:04.717 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-08-19T14:04:04.791 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install python3-rbd 2024-08-19T14:04:04.825 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libcephfs-devel-2:17.2.7-1406.gdf62bdcc.el9.x86_64 1/1 2024-08-19T14:04:04.826 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-08-19T14:04:04.826 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-08-19T14:04:04.826 INFO:teuthology.orchestra.run.smithi045.stdout: libcephfs-devel-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2024-08-19T14:04:04.826 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-08-19T14:04:04.826 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-08-19T14:04:04.839 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:01 ago on Mon 19 Aug 2024 02:03:03 PM UTC. 2024-08-19T14:04:04.969 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-08-19T14:04:04.970 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-08-19T14:04:04.970 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repo Size 2024-08-19T14:04:04.970 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-08-19T14:04:04.971 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-08-19T14:04:04.971 INFO:teuthology.orchestra.run.smithi031.stdout: libcephfs-devel x86_64 2:17.2.7-1406.gdf62bdcc.el9 ceph 25 k 2024-08-19T14:04:04.971 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:04:04.971 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-08-19T14:04:04.971 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-08-19T14:04:04.971 INFO:teuthology.orchestra.run.smithi031.stdout:Install 1 Package 2024-08-19T14:04:04.971 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:04:04.971 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 25 k 2024-08-19T14:04:04.971 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 110 k 2024-08-19T14:04:04.971 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-08-19T14:04:04.984 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install librados2 2024-08-19T14:04:05.114 INFO:teuthology.orchestra.run.smithi031.stdout:libcephfs-devel-17.2.7-1406.gdf62bdcc.el9.x86_6 174 kB/s | 25 kB 00:00 2024-08-19T14:04:05.115 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-08-19T14:04:05.115 INFO:teuthology.orchestra.run.smithi031.stdout:Total 173 kB/s | 25 kB 00:00 2024-08-19T14:04:05.115 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-08-19T14:04:05.119 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-08-19T14:04:05.119 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-08-19T14:04:05.120 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-08-19T14:04:05.120 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-08-19T14:04:05.201 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-08-19T14:04:05.310 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libcephfs-devel-2:17.2.7-1406.gdf62bdcc.el9.x86_64 1/1 2024-08-19T14:04:05.331 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:01:06 ago on Mon 19 Aug 2024 02:02:59 PM UTC. 2024-08-19T14:04:05.417 INFO:teuthology.orchestra.run.smithi032.stdout:Package python3-rbd-2:17.2.7-1406.gdf62bdcc.el9.x86_64 is already installed. 2024-08-19T14:04:05.461 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-08-19T14:04:05.462 INFO:teuthology.orchestra.run.smithi032.stdout:Nothing to do. 2024-08-19T14:04:05.462 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-08-19T14:04:05.515 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install rbd-fuse 2024-08-19T14:04:05.529 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:03 ago on Mon 19 Aug 2024 02:03:02 PM UTC. 2024-08-19T14:04:05.616 INFO:teuthology.orchestra.run.smithi045.stdout:Package librados2-2:17.2.7-1406.gdf62bdcc.el9.x86_64 is already installed. 2024-08-19T14:04:05.660 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-08-19T14:04:05.661 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2024-08-19T14:04:05.661 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-08-19T14:04:05.710 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-1406.gdf62bdcc.el9.x86_64 1/1 2024-08-19T14:04:05.721 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install librbd1 2024-08-19T14:04:05.921 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libcephfs-devel-2:17.2.7-1406.gdf62bdcc.el9.x86_64 1/1 2024-08-19T14:04:05.921 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:04:05.921 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-08-19T14:04:05.921 INFO:teuthology.orchestra.run.smithi031.stdout: libcephfs-devel-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2024-08-19T14:04:05.921 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:04:05.921 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-08-19T14:04:06.061 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:01:07 ago on Mon 19 Aug 2024 02:02:59 PM UTC. 2024-08-19T14:04:06.086 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install librados2 2024-08-19T14:04:06.191 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-08-19T14:04:06.191 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-08-19T14:04:06.191 INFO:teuthology.orchestra.run.smithi032.stdout: Package Architecture Version Repository Size 2024-08-19T14:04:06.192 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-08-19T14:04:06.192 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-08-19T14:04:06.192 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-fuse x86_64 2:17.2.7-1406.gdf62bdcc.el9 ceph 82 k 2024-08-19T14:04:06.192 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-19T14:04:06.192 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-08-19T14:04:06.192 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-08-19T14:04:06.192 INFO:teuthology.orchestra.run.smithi032.stdout:Install 1 Package 2024-08-19T14:04:06.192 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-19T14:04:06.193 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 82 k 2024-08-19T14:04:06.193 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 222 k 2024-08-19T14:04:06.193 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-08-19T14:04:06.262 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:04 ago on Mon 19 Aug 2024 02:03:02 PM UTC. 2024-08-19T14:04:06.348 INFO:teuthology.orchestra.run.smithi045.stdout:Package librbd1-2:17.2.7-1406.gdf62bdcc.el9.x86_64 is already installed. 2024-08-19T14:04:06.376 INFO:teuthology.orchestra.run.smithi032.stdout:rbd-fuse-17.2.7-1406.gdf62bdcc.el9.x86_64.rpm 447 kB/s | 82 kB 00:00 2024-08-19T14:04:06.376 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-08-19T14:04:06.377 INFO:teuthology.orchestra.run.smithi032.stdout:Total 444 kB/s | 82 kB 00:00 2024-08-19T14:04:06.377 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-08-19T14:04:06.384 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-08-19T14:04:06.384 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-08-19T14:04:06.391 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-08-19T14:04:06.392 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2024-08-19T14:04:06.393 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-08-19T14:04:06.443 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-08-19T14:04:06.443 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-08-19T14:04:06.454 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install python3-rados 2024-08-19T14:04:06.583 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-08-19T14:04:06.626 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:03 ago on Mon 19 Aug 2024 02:03:03 PM UTC. 2024-08-19T14:04:06.682 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : rbd-fuse-2:17.2.7-1406.gdf62bdcc.el9.x86_64 1/1 2024-08-19T14:04:06.711 INFO:teuthology.orchestra.run.smithi031.stdout:Package librados2-2:17.2.7-1406.gdf62bdcc.el9.x86_64 is already installed. 2024-08-19T14:04:06.754 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-08-19T14:04:06.755 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2024-08-19T14:04:06.755 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-08-19T14:04:06.823 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install librbd1 2024-08-19T14:04:07.003 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:05 ago on Mon 19 Aug 2024 02:03:02 PM UTC. 2024-08-19T14:04:07.038 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: rbd-fuse-2:17.2.7-1406.gdf62bdcc.el9.x86_64 1/1 2024-08-19T14:04:07.089 INFO:teuthology.orchestra.run.smithi045.stdout:Package python3-rados-2:17.2.7-1406.gdf62bdcc.el9.x86_64 is already installed. 2024-08-19T14:04:07.133 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-08-19T14:04:07.134 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2024-08-19T14:04:07.134 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-08-19T14:04:07.196 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install python3-rgw 2024-08-19T14:04:07.291 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : rbd-fuse-2:17.2.7-1406.gdf62bdcc.el9.x86_64 1/1 2024-08-19T14:04:07.291 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-19T14:04:07.291 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-08-19T14:04:07.291 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-fuse-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2024-08-19T14:04:07.291 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-19T14:04:07.291 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-08-19T14:04:07.367 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:04 ago on Mon 19 Aug 2024 02:03:03 PM UTC. 2024-08-19T14:04:07.453 INFO:teuthology.orchestra.run.smithi031.stdout:Package librbd1-2:17.2.7-1406.gdf62bdcc.el9.x86_64 is already installed. 2024-08-19T14:04:07.496 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-08-19T14:04:07.497 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2024-08-19T14:04:07.498 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-08-19T14:04:07.511 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install rbd-mirror 2024-08-19T14:04:07.562 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install python3-rados 2024-08-19T14:04:07.739 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:05 ago on Mon 19 Aug 2024 02:03:02 PM UTC. 2024-08-19T14:04:07.827 INFO:teuthology.orchestra.run.smithi045.stdout:Package python3-rgw-2:17.2.7-1406.gdf62bdcc.el9.x86_64 is already installed. 2024-08-19T14:04:07.872 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-08-19T14:04:07.873 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2024-08-19T14:04:07.873 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-08-19T14:04:07.932 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install python3-cephfs 2024-08-19T14:04:08.056 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:01:09 ago on Mon 19 Aug 2024 02:02:59 PM UTC. 2024-08-19T14:04:08.104 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:05 ago on Mon 19 Aug 2024 02:03:03 PM UTC. 2024-08-19T14:04:08.191 INFO:teuthology.orchestra.run.smithi031.stdout:Package python3-rados-2:17.2.7-1406.gdf62bdcc.el9.x86_64 is already installed. 2024-08-19T14:04:08.191 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-08-19T14:04:08.192 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-08-19T14:04:08.192 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repo Size 2024-08-19T14:04:08.192 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-08-19T14:04:08.192 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-08-19T14:04:08.192 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-mirror x86_64 2:17.2.7-1406.gdf62bdcc.el9 ceph 2.9 M 2024-08-19T14:04:08.192 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-19T14:04:08.192 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-08-19T14:04:08.193 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-08-19T14:04:08.193 INFO:teuthology.orchestra.run.smithi032.stdout:Install 1 Package 2024-08-19T14:04:08.193 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-19T14:04:08.193 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 2.9 M 2024-08-19T14:04:08.193 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 12 M 2024-08-19T14:04:08.193 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-08-19T14:04:08.235 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-08-19T14:04:08.236 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2024-08-19T14:04:08.236 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-08-19T14:04:08.303 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install python3-rgw 2024-08-19T14:04:08.474 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:06 ago on Mon 19 Aug 2024 02:03:02 PM UTC. 2024-08-19T14:04:08.522 INFO:teuthology.orchestra.run.smithi032.stdout:rbd-mirror-17.2.7-1406.gdf62bdcc.el9.x86_64.rpm 9.0 MB/s | 2.9 MB 00:00 2024-08-19T14:04:08.522 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-08-19T14:04:08.523 INFO:teuthology.orchestra.run.smithi032.stdout:Total 8.9 MB/s | 2.9 MB 00:00 2024-08-19T14:04:08.523 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-08-19T14:04:08.533 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-08-19T14:04:08.533 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-08-19T14:04:08.561 INFO:teuthology.orchestra.run.smithi045.stdout:Package python3-cephfs-2:17.2.7-1406.gdf62bdcc.el9.x86_64 is already installed. 2024-08-19T14:04:08.605 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-08-19T14:04:08.606 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2024-08-19T14:04:08.606 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-08-19T14:04:08.609 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-08-19T14:04:08.610 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-08-19T14:04:08.663 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install python3-rbd 2024-08-19T14:04:08.850 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:05 ago on Mon 19 Aug 2024 02:03:03 PM UTC. 2024-08-19T14:04:08.939 INFO:teuthology.orchestra.run.smithi031.stdout:Package python3-rgw-2:17.2.7-1406.gdf62bdcc.el9.x86_64 is already installed. 2024-08-19T14:04:08.982 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-08-19T14:04:08.983 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2024-08-19T14:04:08.983 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-08-19T14:04:09.035 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-08-19T14:04:09.045 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install python3-cephfs 2024-08-19T14:04:09.098 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : rbd-mirror-2:17.2.7-1406.gdf62bdcc.el9.x86_64 1/1 2024-08-19T14:04:09.124 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: rbd-mirror-2:17.2.7-1406.gdf62bdcc.el9.x86_64 1/1 2024-08-19T14:04:09.124 INFO:teuthology.orchestra.run.smithi032.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-19T14:04:09.124 INFO:teuthology.orchestra.run.smithi032.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-08-19T14:04:09.124 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-08-19T14:04:09.124 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-08-19T14:04:09.124 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-19T14:04:09.206 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:07 ago on Mon 19 Aug 2024 02:03:02 PM UTC. 2024-08-19T14:04:09.293 INFO:teuthology.orchestra.run.smithi045.stdout:Package python3-rbd-2:17.2.7-1406.gdf62bdcc.el9.x86_64 is already installed. 2024-08-19T14:04:09.336 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-08-19T14:04:09.337 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2024-08-19T14:04:09.337 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-08-19T14:04:09.396 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install rbd-fuse 2024-08-19T14:04:09.587 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:06 ago on Mon 19 Aug 2024 02:03:03 PM UTC. 2024-08-19T14:04:09.673 INFO:teuthology.orchestra.run.smithi031.stdout:Package python3-cephfs-2:17.2.7-1406.gdf62bdcc.el9.x86_64 is already installed. 2024-08-19T14:04:09.717 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-08-19T14:04:09.718 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2024-08-19T14:04:09.719 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-08-19T14:04:09.790 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install python3-rbd 2024-08-19T14:04:09.941 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:07 ago on Mon 19 Aug 2024 02:03:02 PM UTC. 2024-08-19T14:04:10.072 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-08-19T14:04:10.073 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-08-19T14:04:10.073 INFO:teuthology.orchestra.run.smithi045.stdout: Package Architecture Version Repository Size 2024-08-19T14:04:10.073 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-08-19T14:04:10.074 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-08-19T14:04:10.074 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-fuse x86_64 2:17.2.7-1406.gdf62bdcc.el9 ceph 82 k 2024-08-19T14:04:10.074 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-08-19T14:04:10.074 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-08-19T14:04:10.074 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-08-19T14:04:10.074 INFO:teuthology.orchestra.run.smithi045.stdout:Install 1 Package 2024-08-19T14:04:10.074 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-08-19T14:04:10.074 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 82 k 2024-08-19T14:04:10.074 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 222 k 2024-08-19T14:04:10.074 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-08-19T14:04:10.192 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : rbd-mirror-2:17.2.7-1406.gdf62bdcc.el9.x86_64 1/1 2024-08-19T14:04:10.192 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-19T14:04:10.192 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-08-19T14:04:10.192 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-mirror-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2024-08-19T14:04:10.192 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-19T14:04:10.193 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-08-19T14:04:10.273 INFO:teuthology.orchestra.run.smithi045.stdout:rbd-fuse-17.2.7-1406.gdf62bdcc.el9.x86_64.rpm 411 kB/s | 82 kB 00:00 2024-08-19T14:04:10.274 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-08-19T14:04:10.274 INFO:teuthology.orchestra.run.smithi045.stdout:Total 408 kB/s | 82 kB 00:00 2024-08-19T14:04:10.274 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-08-19T14:04:10.281 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-08-19T14:04:10.282 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-08-19T14:04:10.336 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:07 ago on Mon 19 Aug 2024 02:03:03 PM UTC. 2024-08-19T14:04:10.341 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-08-19T14:04:10.341 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-08-19T14:04:10.418 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install rbd-nbd 2024-08-19T14:04:10.422 INFO:teuthology.orchestra.run.smithi031.stdout:Package python3-rbd-2:17.2.7-1406.gdf62bdcc.el9.x86_64 is already installed. 2024-08-19T14:04:10.466 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-08-19T14:04:10.470 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2024-08-19T14:04:10.470 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-08-19T14:04:10.481 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-08-19T14:04:10.539 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install rbd-fuse 2024-08-19T14:04:10.578 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : rbd-fuse-2:17.2.7-1406.gdf62bdcc.el9.x86_64 1/1 2024-08-19T14:04:10.967 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:01:11 ago on Mon 19 Aug 2024 02:02:59 PM UTC. 2024-08-19T14:04:11.077 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:08 ago on Mon 19 Aug 2024 02:03:03 PM UTC. 2024-08-19T14:04:11.084 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: rbd-fuse-2:17.2.7-1406.gdf62bdcc.el9.x86_64 1/1 2024-08-19T14:04:11.097 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-08-19T14:04:11.097 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-08-19T14:04:11.098 INFO:teuthology.orchestra.run.smithi032.stdout: Package Architecture Version Repository Size 2024-08-19T14:04:11.098 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-08-19T14:04:11.098 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-08-19T14:04:11.098 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-nbd x86_64 2:17.2.7-1406.gdf62bdcc.el9 ceph 165 k 2024-08-19T14:04:11.098 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-19T14:04:11.098 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-08-19T14:04:11.098 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-08-19T14:04:11.098 INFO:teuthology.orchestra.run.smithi032.stdout:Install 1 Package 2024-08-19T14:04:11.098 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-19T14:04:11.099 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 165 k 2024-08-19T14:04:11.099 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 477 k 2024-08-19T14:04:11.099 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-08-19T14:04:11.207 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-08-19T14:04:11.208 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-08-19T14:04:11.208 INFO:teuthology.orchestra.run.smithi031.stdout: Package Architecture Version Repository Size 2024-08-19T14:04:11.208 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-08-19T14:04:11.208 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-08-19T14:04:11.208 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-fuse x86_64 2:17.2.7-1406.gdf62bdcc.el9 ceph 82 k 2024-08-19T14:04:11.209 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:04:11.209 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-08-19T14:04:11.209 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-08-19T14:04:11.209 INFO:teuthology.orchestra.run.smithi031.stdout:Install 1 Package 2024-08-19T14:04:11.209 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:04:11.209 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 82 k 2024-08-19T14:04:11.209 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 222 k 2024-08-19T14:04:11.209 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-08-19T14:04:11.309 INFO:teuthology.orchestra.run.smithi032.stdout:rbd-nbd-17.2.7-1406.gdf62bdcc.el9.x86_64.rpm 786 kB/s | 165 kB 00:00 2024-08-19T14:04:11.310 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-08-19T14:04:11.310 INFO:teuthology.orchestra.run.smithi032.stdout:Total 779 kB/s | 165 kB 00:00 2024-08-19T14:04:11.311 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-08-19T14:04:11.319 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-08-19T14:04:11.319 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-08-19T14:04:11.359 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : rbd-fuse-2:17.2.7-1406.gdf62bdcc.el9.x86_64 1/1 2024-08-19T14:04:11.359 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-08-19T14:04:11.359 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-08-19T14:04:11.359 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-fuse-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2024-08-19T14:04:11.359 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-08-19T14:04:11.360 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-08-19T14:04:11.375 INFO:teuthology.orchestra.run.smithi031.stdout:rbd-fuse-17.2.7-1406.gdf62bdcc.el9.x86_64.rpm 492 kB/s | 82 kB 00:00 2024-08-19T14:04:11.375 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-08-19T14:04:11.376 INFO:teuthology.orchestra.run.smithi031.stdout:Total 489 kB/s | 82 kB 00:00 2024-08-19T14:04:11.376 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-08-19T14:04:11.378 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-08-19T14:04:11.378 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-08-19T14:04:11.383 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-08-19T14:04:11.383 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-08-19T14:04:11.443 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-08-19T14:04:11.443 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-08-19T14:04:11.527 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-08-19T14:04:11.594 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install rbd-mirror 2024-08-19T14:04:11.599 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-08-19T14:04:11.615 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : rbd-nbd-2:17.2.7-1406.gdf62bdcc.el9.x86_64 1/1 2024-08-19T14:04:11.688 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : rbd-fuse-2:17.2.7-1406.gdf62bdcc.el9.x86_64 1/1 2024-08-19T14:04:12.000 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: rbd-nbd-2:17.2.7-1406.gdf62bdcc.el9.x86_64 1/1 2024-08-19T14:04:12.046 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: rbd-fuse-2:17.2.7-1406.gdf62bdcc.el9.x86_64 1/1 2024-08-19T14:04:12.143 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:10 ago on Mon 19 Aug 2024 02:03:02 PM UTC. 2024-08-19T14:04:12.241 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : rbd-nbd-2:17.2.7-1406.gdf62bdcc.el9.x86_64 1/1 2024-08-19T14:04:12.241 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-19T14:04:12.241 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-08-19T14:04:12.241 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-nbd-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2024-08-19T14:04:12.241 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-19T14:04:12.242 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-08-19T14:04:12.273 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-08-19T14:04:12.274 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-08-19T14:04:12.274 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repo Size 2024-08-19T14:04:12.274 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-08-19T14:04:12.274 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-08-19T14:04:12.275 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-mirror x86_64 2:17.2.7-1406.gdf62bdcc.el9 ceph 2.9 M 2024-08-19T14:04:12.275 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-08-19T14:04:12.275 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-08-19T14:04:12.275 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-08-19T14:04:12.275 INFO:teuthology.orchestra.run.smithi045.stdout:Install 1 Package 2024-08-19T14:04:12.275 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-08-19T14:04:12.275 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 2.9 M 2024-08-19T14:04:12.275 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 12 M 2024-08-19T14:04:12.275 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-08-19T14:04:12.312 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : rbd-fuse-2:17.2.7-1406.gdf62bdcc.el9.x86_64 1/1 2024-08-19T14:04:12.312 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:04:12.312 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-08-19T14:04:12.312 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-fuse-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2024-08-19T14:04:12.312 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:04:12.312 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-08-19T14:04:12.474 DEBUG:teuthology.parallel:result is None 2024-08-19T14:04:12.509 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install rbd-mirror 2024-08-19T14:04:12.609 INFO:teuthology.orchestra.run.smithi045.stdout:rbd-mirror-17.2.7-1406.gdf62bdcc.el9.x86_64.rpm 8.8 MB/s | 2.9 MB 00:00 2024-08-19T14:04:12.610 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-08-19T14:04:12.610 INFO:teuthology.orchestra.run.smithi045.stdout:Total 8.8 MB/s | 2.9 MB 00:00 2024-08-19T14:04:12.610 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-08-19T14:04:12.620 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-08-19T14:04:12.620 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-08-19T14:04:12.696 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-08-19T14:04:12.696 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-08-19T14:04:13.055 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:10 ago on Mon 19 Aug 2024 02:03:03 PM UTC. 2024-08-19T14:04:13.124 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-08-19T14:04:13.176 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : rbd-mirror-2:17.2.7-1406.gdf62bdcc.el9.x86_64 1/1 2024-08-19T14:04:13.185 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-08-19T14:04:13.186 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-08-19T14:04:13.186 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repo Size 2024-08-19T14:04:13.186 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-08-19T14:04:13.186 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-08-19T14:04:13.186 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-mirror x86_64 2:17.2.7-1406.gdf62bdcc.el9 ceph 2.9 M 2024-08-19T14:04:13.187 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:04:13.187 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-08-19T14:04:13.187 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-08-19T14:04:13.187 INFO:teuthology.orchestra.run.smithi031.stdout:Install 1 Package 2024-08-19T14:04:13.187 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:04:13.187 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 2.9 M 2024-08-19T14:04:13.188 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 12 M 2024-08-19T14:04:13.188 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-08-19T14:04:13.201 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: rbd-mirror-2:17.2.7-1406.gdf62bdcc.el9.x86_64 1/1 2024-08-19T14:04:13.201 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-19T14:04:13.201 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-08-19T14:04:13.201 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-08-19T14:04:13.201 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-08-19T14:04:13.201 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-08-19T14:04:13.519 INFO:teuthology.orchestra.run.smithi031.stdout:rbd-mirror-17.2.7-1406.gdf62bdcc.el9.x86_64.rpm 8.9 MB/s | 2.9 MB 00:00 2024-08-19T14:04:13.520 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-08-19T14:04:13.520 INFO:teuthology.orchestra.run.smithi031.stdout:Total 8.8 MB/s | 2.9 MB 00:00 2024-08-19T14:04:13.520 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-08-19T14:04:13.530 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-08-19T14:04:13.530 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-08-19T14:04:13.609 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-08-19T14:04:13.609 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-08-19T14:04:14.030 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-08-19T14:04:14.078 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : rbd-mirror-2:17.2.7-1406.gdf62bdcc.el9.x86_64 1/1 2024-08-19T14:04:14.103 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: rbd-mirror-2:17.2.7-1406.gdf62bdcc.el9.x86_64 1/1 2024-08-19T14:04:14.103 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-19T14:04:14.103 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-08-19T14:04:14.104 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-08-19T14:04:14.104 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-08-19T14:04:14.104 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:04:14.243 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : rbd-mirror-2:17.2.7-1406.gdf62bdcc.el9.x86_64 1/1 2024-08-19T14:04:14.243 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-08-19T14:04:14.243 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-08-19T14:04:14.244 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-mirror-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2024-08-19T14:04:14.244 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-08-19T14:04:14.244 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-08-19T14:04:14.481 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install rbd-nbd 2024-08-19T14:04:15.033 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:13 ago on Mon 19 Aug 2024 02:03:02 PM UTC. 2024-08-19T14:04:15.162 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : rbd-mirror-2:17.2.7-1406.gdf62bdcc.el9.x86_64 1/1 2024-08-19T14:04:15.162 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:04:15.162 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-08-19T14:04:15.162 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-mirror-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2024-08-19T14:04:15.162 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:04:15.162 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-08-19T14:04:15.164 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-08-19T14:04:15.164 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-08-19T14:04:15.165 INFO:teuthology.orchestra.run.smithi045.stdout: Package Architecture Version Repository Size 2024-08-19T14:04:15.165 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-08-19T14:04:15.165 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-08-19T14:04:15.165 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-nbd x86_64 2:17.2.7-1406.gdf62bdcc.el9 ceph 165 k 2024-08-19T14:04:15.165 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-08-19T14:04:15.165 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-08-19T14:04:15.165 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-08-19T14:04:15.165 INFO:teuthology.orchestra.run.smithi045.stdout:Install 1 Package 2024-08-19T14:04:15.165 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-08-19T14:04:15.166 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 165 k 2024-08-19T14:04:15.166 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 477 k 2024-08-19T14:04:15.166 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-08-19T14:04:15.370 INFO:teuthology.orchestra.run.smithi045.stdout:rbd-nbd-17.2.7-1406.gdf62bdcc.el9.x86_64.rpm 807 kB/s | 165 kB 00:00 2024-08-19T14:04:15.371 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-08-19T14:04:15.371 INFO:teuthology.orchestra.run.smithi045.stdout:Total 803 kB/s | 165 kB 00:00 2024-08-19T14:04:15.371 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-08-19T14:04:15.379 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-08-19T14:04:15.379 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-08-19T14:04:15.387 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install rbd-nbd 2024-08-19T14:04:15.438 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-08-19T14:04:15.438 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-08-19T14:04:15.582 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-08-19T14:04:15.678 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : rbd-nbd-2:17.2.7-1406.gdf62bdcc.el9.x86_64 1/1 2024-08-19T14:04:15.939 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:12 ago on Mon 19 Aug 2024 02:03:03 PM UTC. 2024-08-19T14:04:16.051 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: rbd-nbd-2:17.2.7-1406.gdf62bdcc.el9.x86_64 1/1 2024-08-19T14:04:16.069 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-08-19T14:04:16.070 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-08-19T14:04:16.070 INFO:teuthology.orchestra.run.smithi031.stdout: Package Architecture Version Repository Size 2024-08-19T14:04:16.070 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-08-19T14:04:16.070 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-08-19T14:04:16.070 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-nbd x86_64 2:17.2.7-1406.gdf62bdcc.el9 ceph 165 k 2024-08-19T14:04:16.070 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:04:16.070 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-08-19T14:04:16.070 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-08-19T14:04:16.070 INFO:teuthology.orchestra.run.smithi031.stdout:Install 1 Package 2024-08-19T14:04:16.071 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:04:16.071 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 165 k 2024-08-19T14:04:16.071 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 477 k 2024-08-19T14:04:16.071 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-08-19T14:04:16.271 INFO:teuthology.orchestra.run.smithi031.stdout:rbd-nbd-17.2.7-1406.gdf62bdcc.el9.x86_64.rpm 825 kB/s | 165 kB 00:00 2024-08-19T14:04:16.272 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-08-19T14:04:16.272 INFO:teuthology.orchestra.run.smithi031.stdout:Total 819 kB/s | 165 kB 00:00 2024-08-19T14:04:16.272 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-08-19T14:04:16.280 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-08-19T14:04:16.281 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-08-19T14:04:16.301 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : rbd-nbd-2:17.2.7-1406.gdf62bdcc.el9.x86_64 1/1 2024-08-19T14:04:16.301 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-08-19T14:04:16.301 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-08-19T14:04:16.301 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-nbd-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2024-08-19T14:04:16.301 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-08-19T14:04:16.301 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-08-19T14:04:16.340 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-08-19T14:04:16.340 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-08-19T14:04:16.491 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-08-19T14:04:16.551 DEBUG:teuthology.parallel:result is None 2024-08-19T14:04:16.601 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : rbd-nbd-2:17.2.7-1406.gdf62bdcc.el9.x86_64 1/1 2024-08-19T14:04:16.994 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: rbd-nbd-2:17.2.7-1406.gdf62bdcc.el9.x86_64 1/1 2024-08-19T14:04:17.231 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : rbd-nbd-2:17.2.7-1406.gdf62bdcc.el9.x86_64 1/1 2024-08-19T14:04:17.231 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:04:17.231 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-08-19T14:04:17.231 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-nbd-2:17.2.7-1406.gdf62bdcc.el9.x86_64 2024-08-19T14:04:17.231 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:04:17.232 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-08-19T14:04:17.453 DEBUG:teuthology.parallel:result is None 2024-08-19T14:04:17.453 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-08-19T14:04:17.453 INFO:teuthology.packaging:ref: None 2024-08-19T14:04:17.454 INFO:teuthology.packaging:tag: None 2024-08-19T14:04:17.454 INFO:teuthology.packaging:branch: quincy 2024-08-19T14:04:17.454 INFO:teuthology.packaging:sha1: de09dd018de4baddfa6ca4d43bc8fb784cf75833 2024-08-19T14:04:17.454 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-08-19T14:04:17.680 DEBUG:teuthology.orchestra.run.smithi031:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-08-19T14:04:17.702 INFO:teuthology.orchestra.run.smithi031.stdout:17.2.7-1406.gdf62bdcc.el9 2024-08-19T14:04:17.703 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1406.gdf62bdcc.el9 2024-08-19T14:04:17.703 INFO:teuthology.task.install:The correct ceph version 17.2.7-1406.gdf62bdcc is installed. 2024-08-19T14:04:17.705 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-08-19T14:04:17.705 INFO:teuthology.packaging:ref: None 2024-08-19T14:04:17.705 INFO:teuthology.packaging:tag: None 2024-08-19T14:04:17.705 INFO:teuthology.packaging:branch: quincy 2024-08-19T14:04:17.705 INFO:teuthology.packaging:sha1: de09dd018de4baddfa6ca4d43bc8fb784cf75833 2024-08-19T14:04:17.705 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-08-19T14:04:17.916 DEBUG:teuthology.orchestra.run.smithi032:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-08-19T14:04:17.942 INFO:teuthology.orchestra.run.smithi032.stdout:17.2.7-1406.gdf62bdcc.el9 2024-08-19T14:04:17.942 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1406.gdf62bdcc.el9 2024-08-19T14:04:17.942 INFO:teuthology.task.install:The correct ceph version 17.2.7-1406.gdf62bdcc is installed. 2024-08-19T14:04:17.945 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-08-19T14:04:17.945 INFO:teuthology.packaging:ref: None 2024-08-19T14:04:17.945 INFO:teuthology.packaging:tag: None 2024-08-19T14:04:17.945 INFO:teuthology.packaging:branch: quincy 2024-08-19T14:04:17.945 INFO:teuthology.packaging:sha1: de09dd018de4baddfa6ca4d43bc8fb784cf75833 2024-08-19T14:04:17.945 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-08-19T14:04:18.155 DEBUG:teuthology.orchestra.run.smithi045:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-08-19T14:04:18.182 INFO:teuthology.orchestra.run.smithi045.stdout:17.2.7-1406.gdf62bdcc.el9 2024-08-19T14:04:18.182 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1406.gdf62bdcc.el9 2024-08-19T14:04:18.182 INFO:teuthology.task.install:The correct ceph version 17.2.7-1406.gdf62bdcc is installed. 2024-08-19T14:04:18.184 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-08-19T14:04:18.184 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-08-19T14:04:18.184 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-08-19T14:04:18.213 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-19T14:04:18.213 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-08-19T14:04:18.243 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-08-19T14:04:18.243 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-08-19T14:04:18.273 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-08-19T14:04:18.274 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-08-19T14:04:18.274 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/usr/bin/daemon-helper 2024-08-19T14:04:18.299 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-08-19T14:04:18.363 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-19T14:04:18.363 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/usr/bin/daemon-helper 2024-08-19T14:04:18.394 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-08-19T14:04:18.463 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-08-19T14:04:18.463 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/usr/bin/daemon-helper 2024-08-19T14:04:18.492 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-08-19T14:04:18.562 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-08-19T14:04:18.563 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-08-19T14:04:18.563 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/usr/bin/adjust-ulimits 2024-08-19T14:04:18.587 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-08-19T14:04:18.652 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-19T14:04:18.652 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/usr/bin/adjust-ulimits 2024-08-19T14:04:18.682 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-08-19T14:04:18.749 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-08-19T14:04:18.749 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/usr/bin/adjust-ulimits 2024-08-19T14:04:18.779 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-08-19T14:04:18.848 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-08-19T14:04:18.849 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-08-19T14:04:18.849 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/usr/bin/stdin-killer 2024-08-19T14:04:18.874 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-08-19T14:04:18.938 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-19T14:04:18.938 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/usr/bin/stdin-killer 2024-08-19T14:04:18.965 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-08-19T14:04:19.032 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-08-19T14:04:19.032 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/usr/bin/stdin-killer 2024-08-19T14:04:19.061 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-08-19T14:04:19.129 INFO:teuthology.run_tasks:Running task cephadm... 2024-08-19T14:04:19.250 INFO:tasks.cephadm:Config: {'conf': {'mon': {'auth allow insecure global id reclaim': True, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'client.2': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 2500, 'ms tcp read timeout': 5, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'osd': {'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'filestore queue throttle high multiple': 2, 'filestore queue throttle max multiple': 10, 'journal throttle high multiple': 2, 'journal throttle max multiple': 10, 'osd backoff on peering': True, 'osd blocked scrub grace period': 3600, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd max backfills': 9, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(MON_DOWN\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub'], 'sha1': 'de09dd018de4baddfa6ca4d43bc8fb784cf75833'} 2024-08-19T14:04:19.251 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 2024-08-19T14:04:19.251 INFO:tasks.cephadm:Cluster fsid is ed7d1c6e-5e33-11ef-bccf-c7b262605968 2024-08-19T14:04:19.252 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-08-19T14:04:19.252 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.31', 'mon.b': '172.21.15.32', 'mon.c': '172.21.15.45'} 2024-08-19T14:04:19.252 INFO:tasks.cephadm:First mon is mon.a on smithi031 2024-08-19T14:04:19.253 INFO:tasks.cephadm:First mgr is y 2024-08-19T14:04:19.253 INFO:tasks.cephadm:Normalizing hostnames... 2024-08-19T14:04:19.253 DEBUG:teuthology.orchestra.run.smithi031:> sudo hostname $(hostname -s) 2024-08-19T14:04:19.281 DEBUG:teuthology.orchestra.run.smithi032:> sudo hostname $(hostname -s) 2024-08-19T14:04:19.307 DEBUG:teuthology.orchestra.run.smithi045:> sudo hostname $(hostname -s) 2024-08-19T14:04:19.376 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-08-19T14:04:19.376 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=de09dd018de4baddfa6ca4d43bc8fb784cf75833 2024-08-19T14:04:19.531 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'de09dd018de4baddfa6ca4d43bc8fb784cf75833', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/55869/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.1.0-1382-gde09dd01', 'node_name': '172.21.5.40+adami10', 'job_name': 'ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.1.0-1382.gde09dd01'}, 'url': 'https://1.chacra.ceph.com/r/ceph/squid/de09dd018de4baddfa6ca4d43bc8fb784cf75833/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-08-18 16:22:46.893995', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'squid', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/squid/de09dd018de4baddfa6ca4d43bc8fb784cf75833/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-08-19T14:04:19.673 INFO:tasks.util.chacra:got chacra host 1.chacra.ceph.com, ref squid, sha1 de09dd018de4baddfa6ca4d43bc8fb784cf75833 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=de09dd018de4baddfa6ca4d43bc8fb784cf75833 2024-08-19T14:04:19.675 INFO:tasks.cephadm:Discovered cachra url: https://1.chacra.ceph.com/binaries/ceph/squid/de09dd018de4baddfa6ca4d43bc8fb784cf75833/centos/9/x86_64/flavors/default/cephadm 2024-08-19T14:04:19.675 INFO:tasks.cephadm:Downloading cephadm from url: https://1.chacra.ceph.com/binaries/ceph/squid/de09dd018de4baddfa6ca4d43bc8fb784cf75833/centos/9/x86_64/flavors/default/cephadm 2024-08-19T14:04:19.675 DEBUG:teuthology.orchestra.run.smithi031:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/squid/de09dd018de4baddfa6ca4d43bc8fb784cf75833/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-08-19T14:04:20.062 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r--. 1 ubuntu ubuntu 784608 Aug 19 14:04 /home/ubuntu/cephtest/cephadm 2024-08-19T14:04:20.062 DEBUG:teuthology.orchestra.run.smithi032:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/squid/de09dd018de4baddfa6ca4d43bc8fb784cf75833/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-08-19T14:04:20.461 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-r--r--. 1 ubuntu ubuntu 784608 Aug 19 14:04 /home/ubuntu/cephtest/cephadm 2024-08-19T14:04:20.462 DEBUG:teuthology.orchestra.run.smithi045:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/squid/de09dd018de4baddfa6ca4d43bc8fb784cf75833/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-08-19T14:04:20.838 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-r--r--. 1 ubuntu ubuntu 784608 Aug 19 14:04 /home/ubuntu/cephtest/cephadm 2024-08-19T14:04:20.839 DEBUG:teuthology.orchestra.run.smithi031:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-08-19T14:04:20.860 DEBUG:teuthology.orchestra.run.smithi032:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-08-19T14:04:20.880 DEBUG:teuthology.orchestra.run.smithi045:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-08-19T14:04:20.912 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 on all hosts... 2024-08-19T14:04:20.913 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 pull 2024-08-19T14:04:20.915 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 pull 2024-08-19T14:04:20.923 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 pull 2024-08-19T14:04:22.413 INFO:teuthology.orchestra.run.smithi032.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833... 2024-08-19T14:04:22.509 INFO:teuthology.orchestra.run.smithi031.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833... 2024-08-19T14:04:22.523 INFO:teuthology.orchestra.run.smithi045.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833... 2024-08-19T14:04:51.897 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:04:51.898 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph_version": "ceph version 19.1.0-1382-gde09dd01 (de09dd018de4baddfa6ca4d43bc8fb784cf75833) squid (rc)", 2024-08-19T14:04:51.898 INFO:teuthology.orchestra.run.smithi031.stdout: "image_id": "ea7a5189f30bad3aca9524788e2965d2368d7ce786df09d4edcd8c3edf9cefba", 2024-08-19T14:04:51.899 INFO:teuthology.orchestra.run.smithi031.stdout: "repo_digests": [ 2024-08-19T14:04:51.899 INFO:teuthology.orchestra.run.smithi031.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9" 2024-08-19T14:04:51.899 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:04:51.899 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:05:09.827 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:05:09.827 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph_version": "ceph version 19.1.0-1382-gde09dd01 (de09dd018de4baddfa6ca4d43bc8fb784cf75833) squid (rc)", 2024-08-19T14:05:09.827 INFO:teuthology.orchestra.run.smithi045.stdout: "image_id": "ea7a5189f30bad3aca9524788e2965d2368d7ce786df09d4edcd8c3edf9cefba", 2024-08-19T14:05:09.827 INFO:teuthology.orchestra.run.smithi045.stdout: "repo_digests": [ 2024-08-19T14:05:09.827 INFO:teuthology.orchestra.run.smithi045.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9" 2024-08-19T14:05:09.828 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:05:09.828 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:05:10.255 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:05:10.255 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph_version": "ceph version 19.1.0-1382-gde09dd01 (de09dd018de4baddfa6ca4d43bc8fb784cf75833) squid (rc)", 2024-08-19T14:05:10.255 INFO:teuthology.orchestra.run.smithi032.stdout: "image_id": "ea7a5189f30bad3aca9524788e2965d2368d7ce786df09d4edcd8c3edf9cefba", 2024-08-19T14:05:10.255 INFO:teuthology.orchestra.run.smithi032.stdout: "repo_digests": [ 2024-08-19T14:05:10.255 INFO:teuthology.orchestra.run.smithi032.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9" 2024-08-19T14:05:10.255 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:05:10.255 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:05:10.281 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /etc/ceph 2024-08-19T14:05:10.313 DEBUG:teuthology.orchestra.run.smithi032:> sudo mkdir -p /etc/ceph 2024-08-19T14:05:10.343 DEBUG:teuthology.orchestra.run.smithi045:> sudo mkdir -p /etc/ceph 2024-08-19T14:05:10.373 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 777 /etc/ceph 2024-08-19T14:05:10.403 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 777 /etc/ceph 2024-08-19T14:05:10.427 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 777 /etc/ceph 2024-08-19T14:05:10.455 INFO:tasks.cephadm:Writing seed config... 2024-08-19T14:05:10.456 INFO:tasks.cephadm: override: [mon] auth allow insecure global id reclaim = True 2024-08-19T14:05:10.457 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-08-19T14:05:10.457 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-08-19T14:05:10.457 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-08-19T14:05:10.457 INFO:tasks.cephadm: override: [mon] mon scrub interval = 300 2024-08-19T14:05:10.457 INFO:tasks.cephadm: override: [client.2] debug ms = 1 2024-08-19T14:05:10.457 INFO:tasks.cephadm: override: [client.2] debug objecter = 20 2024-08-19T14:05:10.457 INFO:tasks.cephadm: override: [client.2] debug rados = 20 2024-08-19T14:05:10.457 INFO:tasks.cephadm: override: [global] mon client directed command retry = 5 2024-08-19T14:05:10.457 INFO:tasks.cephadm: override: [global] mon election default strategy = 1 2024-08-19T14:05:10.457 INFO:tasks.cephadm: override: [global] ms inject socket failures = 2500 2024-08-19T14:05:10.457 INFO:tasks.cephadm: override: [global] ms tcp read timeout = 5 2024-08-19T14:05:10.457 INFO:tasks.cephadm: override: [global] osd_pool_default_min_size = 2 2024-08-19T14:05:10.457 INFO:tasks.cephadm: override: [global] osd_pool_default_size = 3 2024-08-19T14:05:10.458 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-08-19T14:05:10.458 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-08-19T14:05:10.458 INFO:tasks.cephadm: override: [osd] bluestore zero block detection = True 2024-08-19T14:05:10.458 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-08-19T14:05:10.458 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-08-19T14:05:10.458 INFO:tasks.cephadm: override: [osd] filestore queue throttle high multiple = 2 2024-08-19T14:05:10.458 INFO:tasks.cephadm: override: [osd] filestore queue throttle max multiple = 10 2024-08-19T14:05:10.458 INFO:tasks.cephadm: override: [osd] journal throttle high multiple = 2 2024-08-19T14:05:10.458 INFO:tasks.cephadm: override: [osd] journal throttle max multiple = 10 2024-08-19T14:05:10.458 INFO:tasks.cephadm: override: [osd] osd backoff on peering = True 2024-08-19T14:05:10.458 INFO:tasks.cephadm: override: [osd] osd blocked scrub grace period = 3600 2024-08-19T14:05:10.458 INFO:tasks.cephadm: override: [osd] osd debug verify cached snaps = True 2024-08-19T14:05:10.458 INFO:tasks.cephadm: override: [osd] osd debug verify missing on start = True 2024-08-19T14:05:10.458 INFO:tasks.cephadm: override: [osd] osd max backfills = 9 2024-08-19T14:05:10.459 INFO:tasks.cephadm: override: [osd] osd max markdown count = 1000 2024-08-19T14:05:10.459 INFO:tasks.cephadm: override: [osd] osd mclock override recovery settings = True 2024-08-19T14:05:10.459 INFO:tasks.cephadm: override: [osd] osd mclock profile = high_recovery_ops 2024-08-19T14:05:10.459 INFO:tasks.cephadm: override: [osd] osd op queue = debug_random 2024-08-19T14:05:10.459 INFO:tasks.cephadm: override: [osd] osd op queue cut off = debug_random 2024-08-19T14:05:10.460 INFO:tasks.cephadm: override: [osd] osd scrub max interval = 120 2024-08-19T14:05:10.460 INFO:tasks.cephadm: override: [osd] osd scrub min interval = 60 2024-08-19T14:05:10.460 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-08-19T14:05:10.460 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-08-19T14:05:10.460 DEBUG:teuthology.orchestra.run.smithi031:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-08-19T14:05:10.479 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = ed7d1c6e-5e33-11ef-bccf-c7b262605968 mon client directed command retry = 5 mon election default strategy = 1 ms inject socket failures = 2500 ms tcp read timeout = 5 osd_pool_default_min_size = 2 osd_pool_default_size = 3 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 120 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = True osd debug verify missing on start = True osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore zero block detection = True debug ms = 1 debug osd = 20 filestore queue throttle high multiple = 2 filestore queue throttle max multiple = 10 journal throttle high multiple = 2 journal throttle max multiple = 10 osd backoff on peering = True osd blocked scrub grace period = 3600 osd max backfills = 9 osd max markdown count = 1000 osd mclock override recovery settings = True osd scrub min interval = 60 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false auth allow insecure global id reclaim = True debug mon = 20 debug ms = 1 debug paxos = 20 mon scrub interval = 300 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client.2] debug ms = 1 debug objecter = 20 debug rados = 20 2024-08-19T14:05:10.480 DEBUG:teuthology.orchestra.run.smithi031:mon.a> sudo journalctl -f -n 0 -u ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968@mon.a.service 2024-08-19T14:05:10.522 DEBUG:teuthology.orchestra.run.smithi031:mgr.y> sudo journalctl -f -n 0 -u ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968@mgr.y.service 2024-08-19T14:05:10.565 INFO:tasks.cephadm:Bootstrapping... 2024-08-19T14:05:10.566 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 -v bootstrap --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.31 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-08-19T14:05:10.780 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-08-19T14:05:10.781 INFO:teuthology.orchestra.run.smithi031.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833', '-v', 'bootstrap', '--fsid', 'ed7d1c6e-5e33-11ef-bccf-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.31', '--skip-admin-label'] 2024-08-19T14:05:10.781 INFO:teuthology.orchestra.run.smithi031.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-08-19T14:05:10.781 INFO:teuthology.orchestra.run.smithi031.stdout:Verifying podman|docker is present... 2024-08-19T14:05:10.806 INFO:teuthology.orchestra.run.smithi031.stdout:/bin/podman: stdout 5.1.2 2024-08-19T14:05:10.806 INFO:teuthology.orchestra.run.smithi031.stdout:Verifying lvm2 is present... 2024-08-19T14:05:10.806 INFO:teuthology.orchestra.run.smithi031.stdout:Verifying time synchronization is in place... 2024-08-19T14:05:10.813 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-08-19T14:05:10.813 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-08-19T14:05:10.820 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-08-19T14:05:10.820 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout inactive 2024-08-19T14:05:10.827 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout enabled 2024-08-19T14:05:10.833 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout active 2024-08-19T14:05:10.834 INFO:teuthology.orchestra.run.smithi031.stdout:Unit chronyd.service is enabled and running 2024-08-19T14:05:10.834 INFO:teuthology.orchestra.run.smithi031.stdout:Repeating the final host check... 2024-08-19T14:05:10.858 INFO:teuthology.orchestra.run.smithi031.stdout:/bin/podman: stdout 5.1.2 2024-08-19T14:05:10.858 INFO:teuthology.orchestra.run.smithi031.stdout:podman (/bin/podman) version 5.1.2 is present 2024-08-19T14:05:10.858 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl is present 2024-08-19T14:05:10.858 INFO:teuthology.orchestra.run.smithi031.stdout:lvcreate is present 2024-08-19T14:05:10.864 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-08-19T14:05:10.865 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-08-19T14:05:10.871 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-08-19T14:05:10.871 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout inactive 2024-08-19T14:05:10.877 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout enabled 2024-08-19T14:05:10.884 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout active 2024-08-19T14:05:10.885 INFO:teuthology.orchestra.run.smithi031.stdout:Unit chronyd.service is enabled and running 2024-08-19T14:05:10.885 INFO:teuthology.orchestra.run.smithi031.stdout:Host looks OK 2024-08-19T14:05:10.885 INFO:teuthology.orchestra.run.smithi031.stdout:Cluster fsid: ed7d1c6e-5e33-11ef-bccf-c7b262605968 2024-08-19T14:05:10.885 INFO:teuthology.orchestra.run.smithi031.stdout:Acquiring lock 140591677492720 on /run/cephadm/ed7d1c6e-5e33-11ef-bccf-c7b262605968.lock 2024-08-19T14:05:10.885 INFO:teuthology.orchestra.run.smithi031.stdout:Lock 140591677492720 acquired on /run/cephadm/ed7d1c6e-5e33-11ef-bccf-c7b262605968.lock 2024-08-19T14:05:10.885 INFO:teuthology.orchestra.run.smithi031.stdout:Verifying IP 172.21.15.31 port 3300 ... 2024-08-19T14:05:10.885 INFO:teuthology.orchestra.run.smithi031.stdout:Verifying IP 172.21.15.31 port 6789 ... 2024-08-19T14:05:10.885 INFO:teuthology.orchestra.run.smithi031.stdout:Base mon IP(s) is [172.21.15.31:3300, 172.21.15.31:6789], mon addrv is [v2:172.21.15.31:3300,v1:172.21.15.31:6789] 2024-08-19T14:05:10.892 INFO:teuthology.orchestra.run.smithi031.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.31 metric 100 2024-08-19T14:05:10.892 INFO:teuthology.orchestra.run.smithi031.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.31 metric 100 2024-08-19T14:05:10.892 INFO:teuthology.orchestra.run.smithi031.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-08-19T14:05:10.892 INFO:teuthology.orchestra.run.smithi031.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-08-19T14:05:10.892 INFO:teuthology.orchestra.run.smithi031.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-08-19T14:05:10.893 INFO:teuthology.orchestra.run.smithi031.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-08-19T14:05:10.894 INFO:teuthology.orchestra.run.smithi031.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-08-19T14:05:10.894 INFO:teuthology.orchestra.run.smithi031.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-08-19T14:05:10.894 INFO:teuthology.orchestra.run.smithi031.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-08-19T14:05:10.894 INFO:teuthology.orchestra.run.smithi031.stdout:/sbin/ip: stdout inet6 fe80::79a3:ddd2:5126:891c/64 scope link noprefixroute 2024-08-19T14:05:10.894 INFO:teuthology.orchestra.run.smithi031.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-08-19T14:05:10.894 INFO:teuthology.orchestra.run.smithi031.stdout:Mon IP `172.21.15.31` is in CIDR network `172.21.0.0/20` 2024-08-19T14:05:10.894 INFO:teuthology.orchestra.run.smithi031.stdout:Mon IP `172.21.15.31` is in CIDR network `172.21.0.0/20` 2024-08-19T14:05:10.894 INFO:teuthology.orchestra.run.smithi031.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-08-19T14:05:10.894 INFO:teuthology.orchestra.run.smithi031.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-08-19T14:05:10.895 INFO:teuthology.orchestra.run.smithi031.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833... 2024-08-19T14:05:11.382 INFO:teuthology.orchestra.run.smithi031.stdout:/bin/podman: stdout ea7a5189f30bad3aca9524788e2965d2368d7ce786df09d4edcd8c3edf9cefba 2024-08-19T14:05:11.382 INFO:teuthology.orchestra.run.smithi031.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833... 2024-08-19T14:05:11.383 INFO:teuthology.orchestra.run.smithi031.stdout:/bin/podman: stderr Getting image source signatures 2024-08-19T14:05:11.383 INFO:teuthology.orchestra.run.smithi031.stdout:/bin/podman: stderr Copying blob sha256:7556061847b3cc957dc118ff2516cc1cd809c4272dcb16074c5c7fe28afc4774 2024-08-19T14:05:11.383 INFO:teuthology.orchestra.run.smithi031.stdout:/bin/podman: stderr Copying blob sha256:e8b54c863393b7a8216a71737771b73b16a8e43ec7acf927c7faa175c255e551 2024-08-19T14:05:11.383 INFO:teuthology.orchestra.run.smithi031.stdout:/bin/podman: stderr Copying config sha256:ea7a5189f30bad3aca9524788e2965d2368d7ce786df09d4edcd8c3edf9cefba 2024-08-19T14:05:11.383 INFO:teuthology.orchestra.run.smithi031.stdout:/bin/podman: stderr Writing manifest to image destination 2024-08-19T14:05:12.308 INFO:teuthology.orchestra.run.smithi031.stdout:ceph: stdout ceph version 19.1.0-1382-gde09dd01 (de09dd018de4baddfa6ca4d43bc8fb784cf75833) squid (rc) 2024-08-19T14:05:12.309 INFO:teuthology.orchestra.run.smithi031.stdout:Ceph version: ceph version 19.1.0-1382-gde09dd01 (de09dd018de4baddfa6ca4d43bc8fb784cf75833) squid (rc) 2024-08-19T14:05:12.309 INFO:teuthology.orchestra.run.smithi031.stdout:Extracting ceph user uid/gid from container image... 2024-08-19T14:05:13.156 INFO:teuthology.orchestra.run.smithi031.stdout:stat: stdout 167 167 2024-08-19T14:05:13.156 INFO:teuthology.orchestra.run.smithi031.stdout:Creating initial keys... 2024-08-19T14:05:14.077 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph-authtool: stdout AQAZUcNmXcSdGhAAGZVczOabju07BcKnJ20m+g== 2024-08-19T14:05:14.999 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph-authtool: stdout AQAaUcNmIK19FxAA7Q8ZG8t5gi5mpW7njdLEWg== 2024-08-19T14:05:15.890 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph-authtool: stdout AQAbUcNmFYdUERAA+EBsYB5x92W/1MQl+eiJIA== 2024-08-19T14:05:15.890 INFO:teuthology.orchestra.run.smithi031.stdout:Creating initial monmap... 2024-08-19T14:05:16.693 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-08-19T14:05:16.693 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-08-19T14:05:16.693 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to ed7d1c6e-5e33-11ef-bccf-c7b262605968 2024-08-19T14:05:16.693 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-08-19T14:05:16.693 INFO:teuthology.orchestra.run.smithi031.stdout:monmaptool for a [v2:172.21.15.31:3300,v1:172.21.15.31:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-08-19T14:05:16.693 INFO:teuthology.orchestra.run.smithi031.stdout:setting min_mon_release = quincy 2024-08-19T14:05:16.693 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/monmaptool: set fsid to ed7d1c6e-5e33-11ef-bccf-c7b262605968 2024-08-19T14:05:16.693 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-08-19T14:05:16.694 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:05:16.694 INFO:teuthology.orchestra.run.smithi031.stdout:Creating mon... 2024-08-19T14:05:17.589 INFO:teuthology.orchestra.run.smithi031.stdout:create mon.a on 2024-08-19T14:05:17.813 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-08-19T14:05:18.078 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-08-19T14:05:18.565 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968.target → /etc/systemd/system/ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968.target. 2024-08-19T14:05:18.565 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968.target → /etc/systemd/system/ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968.target. 2024-08-19T14:05:19.518 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968@mon.a 2024-08-19T14:05:19.519 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stderr Failed to reset failed state of unit ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968@mon.a.service: Unit ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968@mon.a.service not loaded. 2024-08-19T14:05:19.770 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968.target.wants/ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968@mon.a.service → /etc/systemd/system/ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968@.service. 2024-08-19T14:05:20.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:20 smithi031 podman[15939]: 2024-08-19 14:05:19.920133455 +0000 UTC m=+0.019992673 image pull ea7a5189f30bad3aca9524788e2965d2368d7ce786df09d4edcd8c3edf9cefba quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 2024-08-19T14:05:21.018 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:20 smithi031 podman[15939]: 2024-08-19 14:05:20.85585402 +0000 UTC m=+0.955713208 container create 929e337bd3348aca2edeb60258bcbc4cd2dce4595e3e4d6a870d50cbc270cd78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mon-a, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, org.label-schema.build-date=20240731, RELEASE=squid-de09dd0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=) 2024-08-19T14:05:23.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:22 smithi031 podman[15939]: 2024-08-19 14:05:22.792486059 +0000 UTC m=+2.892345249 container init 929e337bd3348aca2edeb60258bcbc4cd2dce4595e3e4d6a870d50cbc270cd78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mon-a, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=squid-de09dd0, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0) 2024-08-19T14:05:23.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:22 smithi031 podman[15939]: 2024-08-19 14:05:22.796606309 +0000 UTC m=+2.896465502 container start 929e337bd3348aca2edeb60258bcbc4cd2dce4595e3e4d6a870d50cbc270cd78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mon-a, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, RELEASE=squid-de09dd0, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-08-19T14:05:23.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:22 smithi031 ceph-mon[15967]: mkfs ed7d1c6e-5e33-11ef-bccf-c7b262605968 2024-08-19T14:05:23.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:22 smithi031 ceph-mon[15967]: mon.a is new leader, mons a in quorum (ranks 0) 2024-08-19T14:05:23.695 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-08-19T14:05:23.695 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout disabled 2024-08-19T14:05:23.704 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-08-19T14:05:23.704 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout inactive 2024-08-19T14:05:23.704 INFO:teuthology.orchestra.run.smithi031.stdout:firewalld.service is not enabled 2024-08-19T14:05:23.704 INFO:teuthology.orchestra.run.smithi031.stdout:Not possible to enable service . firewalld.service is not available 2024-08-19T14:05:23.704 INFO:teuthology.orchestra.run.smithi031.stdout:Waiting for mon to start... 2024-08-19T14:05:23.704 INFO:teuthology.orchestra.run.smithi031.stdout:Waiting for mon... 2024-08-19T14:05:24.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:23 smithi031 bash[15939]: 929e337bd3348aca2edeb60258bcbc4cd2dce4595e3e4d6a870d50cbc270cd78 2024-08-19T14:05:24.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:23 smithi031 systemd[1]: Started Ceph mon.a for ed7d1c6e-5e33-11ef-bccf-c7b262605968. 2024-08-19T14:05:24.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:23 smithi031 ceph-mon[15967]: mon.a is new leader, mons a in quorum (ranks 0) 2024-08-19T14:05:24.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:23 smithi031 ceph-mon[15967]: monmap epoch 1 2024-08-19T14:05:24.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:23 smithi031 ceph-mon[15967]: fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 2024-08-19T14:05:24.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:23 smithi031 ceph-mon[15967]: last_changed 2024-08-19T14:05:16.209575+0000 2024-08-19T14:05:24.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:23 smithi031 ceph-mon[15967]: created 2024-08-19T14:05:16.209575+0000 2024-08-19T14:05:24.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:23 smithi031 ceph-mon[15967]: min_mon_release 19 (squid) 2024-08-19T14:05:24.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:23 smithi031 ceph-mon[15967]: election_strategy: 1 2024-08-19T14:05:24.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:23 smithi031 ceph-mon[15967]: 0: [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] mon.a 2024-08-19T14:05:24.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:23 smithi031 ceph-mon[15967]: fsmap 2024-08-19T14:05:24.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:23 smithi031 ceph-mon[15967]: osdmap e1: 0 total, 0 up, 0 in 2024-08-19T14:05:24.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:23 smithi031 ceph-mon[15967]: mgrmap e1: no daemons active 2024-08-19T14:05:26.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:26 smithi031 ceph-mon[15967]: from='client.? 172.21.15.31:0/113025899' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-08-19T14:05:28.693 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout cluster: 2024-08-19T14:05:28.693 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout id: ed7d1c6e-5e33-11ef-bccf-c7b262605968 2024-08-19T14:05:28.694 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-08-19T14:05:28.694 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 2024-08-19T14:05:28.694 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout services: 2024-08-19T14:05:28.694 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 3s) 2024-08-19T14:05:28.694 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-08-19T14:05:28.694 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-08-19T14:05:28.694 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 2024-08-19T14:05:28.694 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout data: 2024-08-19T14:05:28.694 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-08-19T14:05:28.694 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-08-19T14:05:28.694 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-08-19T14:05:28.694 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout pgs: 2024-08-19T14:05:28.694 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 2024-08-19T14:05:28.695 INFO:teuthology.orchestra.run.smithi031.stdout:mon is available 2024-08-19T14:05:28.695 INFO:teuthology.orchestra.run.smithi031.stdout:Assimilating anything we can from ceph.conf... 2024-08-19T14:05:29.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:29 smithi031 ceph-mon[15967]: from='client.? 172.21.15.31:0/1586666036' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-08-19T14:05:29.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:29 smithi031 ceph-mon[15967]: from='client.? 172.21.15.31:0/1586666036' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-08-19T14:05:29.924 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 2024-08-19T14:05:29.924 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout [global] 2024-08-19T14:05:29.924 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout fsid = ed7d1c6e-5e33-11ef-bccf-c7b262605968 2024-08-19T14:05:29.924 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.31:3300,v1:172.21.15.31:6789] 2024-08-19T14:05:29.924 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-08-19T14:05:29.924 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-08-19T14:05:29.924 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-08-19T14:05:29.924 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout ms_tcp_read_timeout = 5 2024-08-19T14:05:29.925 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-08-19T14:05:29.925 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 2024-08-19T14:05:29.925 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout [mgr] 2024-08-19T14:05:29.925 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-08-19T14:05:29.925 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 2024-08-19T14:05:29.925 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout [osd] 2024-08-19T14:05:29.925 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout filestore_queue_throttle_high_multiple = 2 2024-08-19T14:05:29.925 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout filestore_queue_throttle_max_multiple = 10 2024-08-19T14:05:29.925 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-08-19T14:05:29.925 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-08-19T14:05:29.925 INFO:teuthology.orchestra.run.smithi031.stdout:Generating new minimal ceph.conf... 2024-08-19T14:05:30.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:30 smithi031 ceph-mon[15967]: from='client.? 172.21.15.31:0/2183249064' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:05:30.986 INFO:teuthology.orchestra.run.smithi031.stdout:Restarting the monitor... 2024-08-19T14:05:31.312 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:31 smithi031 systemd[1]: Stopping Ceph mon.a for ed7d1c6e-5e33-11ef-bccf-c7b262605968... 2024-08-19T14:05:31.589 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:31 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mon-a[15963]: 2024-08-19T14:05:31.310+0000 7f868ea70640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-08-19T14:05:31.589 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:31 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mon-a[15963]: 2024-08-19T14:05:31.310+0000 7f868ea70640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-08-19T14:05:31.590 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:31 smithi031 podman[16210]: 2024-08-19 14:05:31.342329896 +0000 UTC m=+0.144665888 container died 929e337bd3348aca2edeb60258bcbc4cd2dce4595e3e4d6a870d50cbc270cd78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, RELEASE=squid-de09dd0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD) 2024-08-19T14:05:31.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:31 smithi031 podman[16210]: 2024-08-19 14:05:31.677132548 +0000 UTC m=+0.479468539 container cleanup 929e337bd3348aca2edeb60258bcbc4cd2dce4595e3e4d6a870d50cbc270cd78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mon-a, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, RELEASE=squid-de09dd0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True) 2024-08-19T14:05:31.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:31 smithi031 bash[16210]: ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mon-a 2024-08-19T14:05:31.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:31 smithi031 podman[16225]: 2024-08-19 14:05:31.836427305 +0000 UTC m=+0.495297474 container remove 929e337bd3348aca2edeb60258bcbc4cd2dce4595e3e4d6a870d50cbc270cd78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mon-a, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-de09dd0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-08-19T14:05:32.270 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 systemd[1]: ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968@mon.a.service: Deactivated successfully. 2024-08-19T14:05:32.271 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 systemd[1]: Stopped Ceph mon.a for ed7d1c6e-5e33-11ef-bccf-c7b262605968. 2024-08-19T14:05:32.271 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 systemd[1]: Starting Ceph mon.a for ed7d1c6e-5e33-11ef-bccf-c7b262605968... 2024-08-19T14:05:32.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 podman[16317]: 2024-08-19 14:05:32.27081043 +0000 UTC m=+0.110795641 container create 268c97faa6a34ab0a8ee48d8e11506efe1fadf98fee0a8802e9dcda667f18086 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mon-a, RELEASE=squid-de09dd0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-08-19T14:05:32.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 podman[16317]: 2024-08-19 14:05:32.176946583 +0000 UTC m=+0.016931766 image pull ea7a5189f30bad3aca9524788e2965d2368d7ce786df09d4edcd8c3edf9cefba quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 2024-08-19T14:05:32.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 podman[16317]: 2024-08-19 14:05:32.472807362 +0000 UTC m=+0.312792546 container init 268c97faa6a34ab0a8ee48d8e11506efe1fadf98fee0a8802e9dcda667f18086 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mon-a, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=squid-de09dd0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, ceph=True, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2) 2024-08-19T14:05:32.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 podman[16317]: 2024-08-19 14:05:32.475711277 +0000 UTC m=+0.315696451 container start 268c97faa6a34ab0a8ee48d8e11506efe1fadf98fee0a8802e9dcda667f18086 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mon-a, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-de09dd0, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-08-19T14:05:32.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: set uid:gid to 167:167 (ceph:ceph) 2024-08-19T14:05:32.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: ceph version 19.1.0-1382-gde09dd01 (de09dd018de4baddfa6ca4d43bc8fb784cf75833) squid (rc), process ceph-mon, pid 2 2024-08-19T14:05:32.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: pidfile_write: ignore empty --pid-file 2024-08-19T14:05:32.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: load: jerasure load: lrc 2024-08-19T14:05:32.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: RocksDB version: 7.9.2 2024-08-19T14:05:32.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Git sha 0 2024-08-19T14:05:32.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Compile date 2024-08-18 15:19:56 2024-08-19T14:05:32.558 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: DB SUMMARY 2024-08-19T14:05:32.558 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: DB Session ID: ZLCUZ05LIW319S605Q0B 2024-08-19T14:05:32.558 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: CURRENT file: CURRENT 2024-08-19T14:05:32.558 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: IDENTITY file: IDENTITY 2024-08-19T14:05:32.558 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-08-19T14:05:32.558 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-08-19T14:05:32.558 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 97724 ; 2024-08-19T14:05:32.558 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.error_if_exists: 0 2024-08-19T14:05:32.558 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.create_if_missing: 0 2024-08-19T14:05:32.558 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.paranoid_checks: 1 2024-08-19T14:05:32.558 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.flush_verify_memtable_count: 1 2024-08-19T14:05:32.558 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-08-19T14:05:32.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-08-19T14:05:32.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.env: 0x5592fdae17a0 2024-08-19T14:05:32.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.fs: PosixFileSystem 2024-08-19T14:05:32.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.info_log: 0x5592fee5d500 2024-08-19T14:05:32.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.max_file_opening_threads: 16 2024-08-19T14:05:32.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.statistics: (nil) 2024-08-19T14:05:32.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.use_fsync: 0 2024-08-19T14:05:32.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.max_log_file_size: 0 2024-08-19T14:05:32.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-08-19T14:05:32.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.log_file_time_to_roll: 0 2024-08-19T14:05:32.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.keep_log_file_num: 1000 2024-08-19T14:05:32.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.recycle_log_file_num: 0 2024-08-19T14:05:32.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.allow_fallocate: 1 2024-08-19T14:05:32.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.allow_mmap_reads: 0 2024-08-19T14:05:32.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.allow_mmap_writes: 0 2024-08-19T14:05:32.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.use_direct_reads: 0 2024-08-19T14:05:32.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-08-19T14:05:32.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.create_missing_column_families: 0 2024-08-19T14:05:32.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.db_log_dir: 2024-08-19T14:05:32.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.wal_dir: 2024-08-19T14:05:32.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.table_cache_numshardbits: 6 2024-08-19T14:05:32.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.WAL_ttl_seconds: 0 2024-08-19T14:05:32.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.WAL_size_limit_MB: 0 2024-08-19T14:05:32.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-08-19T14:05:32.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-08-19T14:05:32.561 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.is_fd_close_on_exec: 1 2024-08-19T14:05:32.561 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.advise_random_on_open: 1 2024-08-19T14:05:32.561 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.db_write_buffer_size: 0 2024-08-19T14:05:32.561 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.write_buffer_manager: 0x5592fee6cdc0 2024-08-19T14:05:32.561 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-08-19T14:05:32.561 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-08-19T14:05:32.561 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.use_adaptive_mutex: 0 2024-08-19T14:05:32.561 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.rate_limiter: (nil) 2024-08-19T14:05:32.561 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-08-19T14:05:32.561 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.wal_recovery_mode: 2 2024-08-19T14:05:32.563 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.enable_thread_tracking: 0 2024-08-19T14:05:32.563 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.enable_pipelined_write: 0 2024-08-19T14:05:32.563 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.unordered_write: 0 2024-08-19T14:05:32.563 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-08-19T14:05:32.563 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-08-19T14:05:32.563 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-08-19T14:05:32.563 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-08-19T14:05:32.563 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.row_cache: None 2024-08-19T14:05:32.564 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.wal_filter: None 2024-08-19T14:05:32.564 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-08-19T14:05:32.564 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.allow_ingest_behind: 0 2024-08-19T14:05:32.564 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.two_write_queues: 0 2024-08-19T14:05:32.564 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.manual_wal_flush: 0 2024-08-19T14:05:32.564 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.wal_compression: 0 2024-08-19T14:05:32.564 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.atomic_flush: 0 2024-08-19T14:05:32.564 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-08-19T14:05:32.564 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.persist_stats_to_disk: 0 2024-08-19T14:05:32.564 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.write_dbid_to_manifest: 0 2024-08-19T14:05:32.564 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.log_readahead_size: 0 2024-08-19T14:05:32.564 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-08-19T14:05:32.564 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.best_efforts_recovery: 0 2024-08-19T14:05:32.565 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-08-19T14:05:32.565 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-08-19T14:05:32.565 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.allow_data_in_errors: 0 2024-08-19T14:05:32.565 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.db_host_id: __hostname__ 2024-08-19T14:05:32.565 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.enforce_single_del_contracts: true 2024-08-19T14:05:32.565 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.max_background_jobs: 2 2024-08-19T14:05:32.565 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.max_background_compactions: -1 2024-08-19T14:05:32.565 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.max_subcompactions: 1 2024-08-19T14:05:32.565 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-08-19T14:05:32.565 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-08-19T14:05:32.566 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.delayed_write_rate : 16777216 2024-08-19T14:05:32.566 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.max_total_wal_size: 0 2024-08-19T14:05:32.566 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-08-19T14:05:32.566 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.stats_dump_period_sec: 600 2024-08-19T14:05:32.566 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.stats_persist_period_sec: 600 2024-08-19T14:05:32.566 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-08-19T14:05:32.566 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.max_open_files: -1 2024-08-19T14:05:32.566 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.bytes_per_sync: 0 2024-08-19T14:05:32.567 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.wal_bytes_per_sync: 0 2024-08-19T14:05:32.567 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.strict_bytes_per_sync: 0 2024-08-19T14:05:32.567 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.compaction_readahead_size: 0 2024-08-19T14:05:32.567 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.max_background_flushes: -1 2024-08-19T14:05:32.567 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Compression algorithms supported: 2024-08-19T14:05:32.567 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: kZSTD supported: 0 2024-08-19T14:05:32.567 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: kXpressCompression supported: 0 2024-08-19T14:05:32.567 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: kBZip2Compression supported: 0 2024-08-19T14:05:32.567 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-08-19T14:05:32.567 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: kLZ4Compression supported: 1 2024-08-19T14:05:32.567 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: kZlibCompression supported: 1 2024-08-19T14:05:32.567 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: kLZ4HCCompression supported: 1 2024-08-19T14:05:32.567 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: kSnappyCompression supported: 1 2024-08-19T14:05:32.567 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Fast CRC32 supported: Supported on x86 2024-08-19T14:05:32.568 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: DMutex implementation: pthread_mutex_t 2024-08-19T14:05:32.568 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-08-19T14:05:32.568 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-08-19T14:05:32.568 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-08-19T14:05:32.568 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.merge_operator: 2024-08-19T14:05:32.568 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.compaction_filter: None 2024-08-19T14:05:32.568 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.compaction_filter_factory: None 2024-08-19T14:05:32.568 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.sst_partitioner_factory: None 2024-08-19T14:05:32.568 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.memtable_factory: SkipListFactory 2024-08-19T14:05:32.568 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.table_factory: BlockBasedTable 2024-08-19T14:05:32.568 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5592fee5d120) 2024-08-19T14:05:32.568 INFO:journalctl@ceph.mon.a.smithi031.stdout: cache_index_and_filter_blocks: 1 2024-08-19T14:05:32.568 INFO:journalctl@ceph.mon.a.smithi031.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-08-19T14:05:32.568 INFO:journalctl@ceph.mon.a.smithi031.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-08-19T14:05:32.569 INFO:journalctl@ceph.mon.a.smithi031.stdout: pin_top_level_index_and_filter: 1 2024-08-19T14:05:32.569 INFO:journalctl@ceph.mon.a.smithi031.stdout: index_type: 0 2024-08-19T14:05:32.569 INFO:journalctl@ceph.mon.a.smithi031.stdout: data_block_index_type: 0 2024-08-19T14:05:32.569 INFO:journalctl@ceph.mon.a.smithi031.stdout: index_shortening: 1 2024-08-19T14:05:32.569 INFO:journalctl@ceph.mon.a.smithi031.stdout: data_block_hash_table_util_ratio: 0.750000 2024-08-19T14:05:32.569 INFO:journalctl@ceph.mon.a.smithi031.stdout: checksum: 4 2024-08-19T14:05:32.569 INFO:journalctl@ceph.mon.a.smithi031.stdout: no_block_cache: 0 2024-08-19T14:05:32.569 INFO:journalctl@ceph.mon.a.smithi031.stdout: block_cache: 0x5592fee51090 2024-08-19T14:05:32.569 INFO:journalctl@ceph.mon.a.smithi031.stdout: block_cache_name: BinnedLRUCache 2024-08-19T14:05:32.569 INFO:journalctl@ceph.mon.a.smithi031.stdout: block_cache_options: 2024-08-19T14:05:32.569 INFO:journalctl@ceph.mon.a.smithi031.stdout: capacity : 536870912 2024-08-19T14:05:32.569 INFO:journalctl@ceph.mon.a.smithi031.stdout: num_shard_bits : 4 2024-08-19T14:05:32.569 INFO:journalctl@ceph.mon.a.smithi031.stdout: strict_capacity_limit : 0 2024-08-19T14:05:32.569 INFO:journalctl@ceph.mon.a.smithi031.stdout: high_pri_pool_ratio: 0.000 2024-08-19T14:05:32.570 INFO:journalctl@ceph.mon.a.smithi031.stdout: block_cache_compressed: (nil) 2024-08-19T14:05:32.570 INFO:journalctl@ceph.mon.a.smithi031.stdout: persistent_cache: (nil) 2024-08-19T14:05:32.570 INFO:journalctl@ceph.mon.a.smithi031.stdout: block_size: 4096 2024-08-19T14:05:32.570 INFO:journalctl@ceph.mon.a.smithi031.stdout: block_size_deviation: 10 2024-08-19T14:05:32.570 INFO:journalctl@ceph.mon.a.smithi031.stdout: block_restart_interval: 16 2024-08-19T14:05:32.570 INFO:journalctl@ceph.mon.a.smithi031.stdout: index_block_restart_interval: 1 2024-08-19T14:05:32.570 INFO:journalctl@ceph.mon.a.smithi031.stdout: metadata_block_size: 4096 2024-08-19T14:05:32.570 INFO:journalctl@ceph.mon.a.smithi031.stdout: partition_filters: 0 2024-08-19T14:05:32.570 INFO:journalctl@ceph.mon.a.smithi031.stdout: use_delta_encoding: 1 2024-08-19T14:05:32.570 INFO:journalctl@ceph.mon.a.smithi031.stdout: filter_policy: bloomfilter 2024-08-19T14:05:32.570 INFO:journalctl@ceph.mon.a.smithi031.stdout: whole_key_filtering: 1 2024-08-19T14:05:32.570 INFO:journalctl@ceph.mon.a.smithi031.stdout: verify_compression: 0 2024-08-19T14:05:32.570 INFO:journalctl@ceph.mon.a.smithi031.stdout: read_amp_bytes_per_bit: 0 2024-08-19T14:05:32.571 INFO:journalctl@ceph.mon.a.smithi031.stdout: format_version: 5 2024-08-19T14:05:32.571 INFO:journalctl@ceph.mon.a.smithi031.stdout: enable_index_compression: 1 2024-08-19T14:05:32.571 INFO:journalctl@ceph.mon.a.smithi031.stdout: block_align: 0 2024-08-19T14:05:32.571 INFO:journalctl@ceph.mon.a.smithi031.stdout: max_auto_readahead_size: 262144 2024-08-19T14:05:32.571 INFO:journalctl@ceph.mon.a.smithi031.stdout: prepopulate_block_cache: 0 2024-08-19T14:05:32.571 INFO:journalctl@ceph.mon.a.smithi031.stdout: initial_auto_readahead_size: 8192 2024-08-19T14:05:32.571 INFO:journalctl@ceph.mon.a.smithi031.stdout: num_file_reads_for_auto_readahead: 2 2024-08-19T14:05:32.571 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.write_buffer_size: 33554432 2024-08-19T14:05:32.571 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.max_write_buffer_number: 2 2024-08-19T14:05:32.571 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.compression: NoCompression 2024-08-19T14:05:32.571 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.bottommost_compression: Disabled 2024-08-19T14:05:32.571 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.prefix_extractor: nullptr 2024-08-19T14:05:32.571 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-08-19T14:05:32.571 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.num_levels: 7 2024-08-19T14:05:32.572 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-08-19T14:05:32.572 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-08-19T14:05:32.572 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-08-19T14:05:32.572 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-08-19T14:05:32.572 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-08-19T14:05:32.572 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-08-19T14:05:32.572 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-08-19T14:05:32.572 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-08-19T14:05:32.572 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-08-19T14:05:32.572 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-08-19T14:05:32.572 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-08-19T14:05:32.572 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-08-19T14:05:32.572 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.compression_opts.window_bits: -14 2024-08-19T14:05:32.572 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.compression_opts.level: 32767 2024-08-19T14:05:32.573 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.compression_opts.strategy: 0 2024-08-19T14:05:32.573 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-08-19T14:05:32.573 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-08-19T14:05:32.573 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-08-19T14:05:32.573 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-08-19T14:05:32.573 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.compression_opts.enabled: false 2024-08-19T14:05:32.573 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-08-19T14:05:32.573 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-08-19T14:05:32.573 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-08-19T14:05:32.573 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-08-19T14:05:32.573 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.target_file_size_base: 67108864 2024-08-19T14:05:32.573 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.target_file_size_multiplier: 1 2024-08-19T14:05:32.573 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-08-19T14:05:32.573 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-08-19T14:05:32.574 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-08-19T14:05:32.574 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-08-19T14:05:32.574 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-08-19T14:05:32.574 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-08-19T14:05:32.574 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-08-19T14:05:32.574 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-08-19T14:05:32.574 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-08-19T14:05:32.574 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-08-19T14:05:32.574 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-08-19T14:05:32.574 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-08-19T14:05:32.574 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-08-19T14:05:32.574 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.arena_block_size: 1048576 2024-08-19T14:05:32.574 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-08-19T14:05:32.574 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-08-19T14:05:32.575 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.disable_auto_compactions: 0 2024-08-19T14:05:32.575 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-08-19T14:05:32.575 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-08-19T14:05:32.575 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-08-19T14:05:32.575 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-08-19T14:05:32.575 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-08-19T14:05:32.575 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-08-19T14:05:32.575 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-08-19T14:05:32.575 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-08-19T14:05:32.575 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-08-19T14:05:32.575 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-08-19T14:05:32.575 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-08-19T14:05:32.575 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.inplace_update_support: 0 2024-08-19T14:05:32.575 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.inplace_update_num_locks: 10000 2024-08-19T14:05:32.576 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-08-19T14:05:32.576 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-08-19T14:05:32.576 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.memtable_huge_page_size: 0 2024-08-19T14:05:32.576 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.bloom_locality: 0 2024-08-19T14:05:32.576 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.max_successive_merges: 0 2024-08-19T14:05:32.576 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.optimize_filters_for_hits: 0 2024-08-19T14:05:32.576 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.paranoid_file_checks: 0 2024-08-19T14:05:32.576 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.force_consistency_checks: 1 2024-08-19T14:05:32.576 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.report_bg_io_stats: 0 2024-08-19T14:05:32.576 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.ttl: 2592000 2024-08-19T14:05:32.576 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.periodic_compaction_seconds: 0 2024-08-19T14:05:32.576 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-08-19T14:05:32.576 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-08-19T14:05:32.576 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.enable_blob_files: false 2024-08-19T14:05:32.577 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.min_blob_size: 0 2024-08-19T14:05:32.577 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.blob_file_size: 268435456 2024-08-19T14:05:32.577 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.blob_compression_type: NoCompression 2024-08-19T14:05:32.577 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.enable_blob_garbage_collection: false 2024-08-19T14:05:32.577 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-08-19T14:05:32.577 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-08-19T14:05:32.577 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-08-19T14:05:32.577 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.blob_file_starting_level: 0 2024-08-19T14:05:32.577 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-08-19T14:05:32.577 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-08-19T14:05:32.577 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-08-19T14:05:32.577 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 9f625d5e-9d12-499a-890d-b26c4c912d18 2024-08-19T14:05:32.577 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: EVENT_LOG_v1 {"time_micros": 1724076332501690, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-08-19T14:05:32.577 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-08-19T14:05:32.578 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: EVENT_LOG_v1 {"time_micros": 1724076332502670, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 93306, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 298, "table_properties": {"data_size": 91352, "index_size": 274, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 645, "raw_key_size": 12416, "raw_average_key_size": 49, "raw_value_size": 84506, "raw_average_value_size": 334, "num_data_blocks": 12, "num_entries": 253, "num_filter_entries": 253, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1724076332, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "9f625d5e-9d12-499a-890d-b26c4c912d18", "db_session_id": "ZLCUZ05LIW319S605Q0B", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-08-19T14:05:32.578 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: EVENT_LOG_v1 {"time_micros": 1724076332502773, "job": 1, "event": "recovery_finished"} 2024-08-19T14:05:32.578 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-08-19T14:05:32.578 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-08-19T14:05:32.578 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5592fee7ee00 2024-08-19T14:05:32.578 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: DB pointer 0x5592fef78000 2024-08-19T14:05:32.578 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-08-19T14:05:32.578 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-08-19T14:05:32.578 INFO:journalctl@ceph.mon.a.smithi031.stdout: ** DB Stats ** 2024-08-19T14:05:32.578 INFO:journalctl@ceph.mon.a.smithi031.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-08-19T14:05:32.578 INFO:journalctl@ceph.mon.a.smithi031.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-08-19T14:05:32.578 INFO:journalctl@ceph.mon.a.smithi031.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-08-19T14:05:32.578 INFO:journalctl@ceph.mon.a.smithi031.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-08-19T14:05:32.579 INFO:journalctl@ceph.mon.a.smithi031.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-08-19T14:05:32.579 INFO:journalctl@ceph.mon.a.smithi031.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-08-19T14:05:32.579 INFO:journalctl@ceph.mon.a.smithi031.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-08-19T14:05:32.579 INFO:journalctl@ceph.mon.a.smithi031.stdout: 2024-08-19T14:05:32.579 INFO:journalctl@ceph.mon.a.smithi031.stdout: ** Compaction Stats [default] ** 2024-08-19T14:05:32.579 INFO:journalctl@ceph.mon.a.smithi031.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-08-19T14:05:32.579 INFO:journalctl@ceph.mon.a.smithi031.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-08-19T14:05:32.579 INFO:journalctl@ceph.mon.a.smithi031.stdout: L0 2/0 92.98 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 138.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-08-19T14:05:32.579 INFO:journalctl@ceph.mon.a.smithi031.stdout: Sum 2/0 92.98 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 138.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-08-19T14:05:32.579 INFO:journalctl@ceph.mon.a.smithi031.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 138.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-08-19T14:05:32.579 INFO:journalctl@ceph.mon.a.smithi031.stdout: 2024-08-19T14:05:32.579 INFO:journalctl@ceph.mon.a.smithi031.stdout: ** Compaction Stats [default] ** 2024-08-19T14:05:32.579 INFO:journalctl@ceph.mon.a.smithi031.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-08-19T14:05:32.579 INFO:journalctl@ceph.mon.a.smithi031.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-08-19T14:05:32.580 INFO:journalctl@ceph.mon.a.smithi031.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 138.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-08-19T14:05:32.580 INFO:journalctl@ceph.mon.a.smithi031.stdout: 2024-08-19T14:05:32.580 INFO:journalctl@ceph.mon.a.smithi031.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-08-19T14:05:32.580 INFO:journalctl@ceph.mon.a.smithi031.stdout: 2024-08-19T14:05:32.580 INFO:journalctl@ceph.mon.a.smithi031.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-08-19T14:05:32.580 INFO:journalctl@ceph.mon.a.smithi031.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-08-19T14:05:32.580 INFO:journalctl@ceph.mon.a.smithi031.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-08-19T14:05:32.580 INFO:journalctl@ceph.mon.a.smithi031.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-08-19T14:05:32.580 INFO:journalctl@ceph.mon.a.smithi031.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-08-19T14:05:32.580 INFO:journalctl@ceph.mon.a.smithi031.stdout: AddFile(Keys): cumulative 0, interval 0 2024-08-19T14:05:32.580 INFO:journalctl@ceph.mon.a.smithi031.stdout: Cumulative compaction: 0.00 GB write, 28.77 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-08-19T14:05:32.580 INFO:journalctl@ceph.mon.a.smithi031.stdout: Interval compaction: 0.00 GB write, 28.77 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-08-19T14:05:32.580 INFO:journalctl@ceph.mon.a.smithi031.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-08-19T14:05:32.581 INFO:journalctl@ceph.mon.a.smithi031.stdout: Block cache BinnedLRUCache@0x5592fee51090#2 capacity: 512.00 MB usage: 1.30 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-08-19T14:05:32.581 INFO:journalctl@ceph.mon.a.smithi031.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.83 KB,0.000157952%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2024-08-19T14:05:32.581 INFO:journalctl@ceph.mon.a.smithi031.stdout: 2024-08-19T14:05:32.581 INFO:journalctl@ceph.mon.a.smithi031.stdout: ** File Read Latency Histogram By Level [default] ** 2024-08-19T14:05:32.581 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: starting mon.a rank 0 at public addrs [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] at bind addrs [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 2024-08-19T14:05:32.581 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: mon.a@-1(???) e1 preinit fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 2024-08-19T14:05:32.581 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: mon.a@-1(???).mds e1 new map 2024-08-19T14:05:32.581 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: mon.a@-1(???).mds e1 print_map 2024-08-19T14:05:32.581 INFO:journalctl@ceph.mon.a.smithi031.stdout: e1 2024-08-19T14:05:32.581 INFO:journalctl@ceph.mon.a.smithi031.stdout: btime 2024-08-19T14:05:22:841940+0000 2024-08-19T14:05:32.581 INFO:journalctl@ceph.mon.a.smithi031.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-08-19T14:05:32.581 INFO:journalctl@ceph.mon.a.smithi031.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-08-19T14:05:32.581 INFO:journalctl@ceph.mon.a.smithi031.stdout: legacy client fscid: -1 2024-08-19T14:05:32.581 INFO:journalctl@ceph.mon.a.smithi031.stdout: 2024-08-19T14:05:32.582 INFO:journalctl@ceph.mon.a.smithi031.stdout: No filesystems configured 2024-08-19T14:05:32.582 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-08-19T14:05:32.582 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-08-19T14:05:32.582 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-08-19T14:05:32.582 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-08-19T14:05:32.582 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-08-19T14:05:32.582 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 bash[16317]: 268c97faa6a34ab0a8ee48d8e11506efe1fadf98fee0a8802e9dcda667f18086 2024-08-19T14:05:32.582 INFO:teuthology.orchestra.run.smithi031.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-08-19T14:05:32.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 systemd[1]: Started Ceph mon.a for ed7d1c6e-5e33-11ef-bccf-c7b262605968. 2024-08-19T14:05:32.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: mon.a is new leader, mons a in quorum (ranks 0) 2024-08-19T14:05:32.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: monmap epoch 1 2024-08-19T14:05:32.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 2024-08-19T14:05:32.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: last_changed 2024-08-19T14:05:16.209575+0000 2024-08-19T14:05:32.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: created 2024-08-19T14:05:16.209575+0000 2024-08-19T14:05:32.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: min_mon_release 19 (squid) 2024-08-19T14:05:32.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: election_strategy: 1 2024-08-19T14:05:32.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: 0: [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] mon.a 2024-08-19T14:05:32.808 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: fsmap 2024-08-19T14:05:32.808 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: osdmap e1: 0 total, 0 up, 0 in 2024-08-19T14:05:32.808 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:32 smithi031 ceph-mon[16336]: mgrmap e1: no daemons active 2024-08-19T14:05:33.584 INFO:teuthology.orchestra.run.smithi031.stdout:Wrote config to /etc/ceph/ceph.conf 2024-08-19T14:05:33.605 INFO:teuthology.orchestra.run.smithi031.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-08-19T14:05:33.605 INFO:teuthology.orchestra.run.smithi031.stdout:Creating mgr... 2024-08-19T14:05:33.606 INFO:teuthology.orchestra.run.smithi031.stdout:Verifying port 0.0.0.0:9283 ... 2024-08-19T14:05:33.606 INFO:teuthology.orchestra.run.smithi031.stdout:Verifying port 0.0.0.0:8765 ... 2024-08-19T14:05:33.968 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968@mgr.y 2024-08-19T14:05:33.968 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stderr Failed to reset failed state of unit ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968@mgr.y.service: Unit ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968@mgr.y.service not loaded. 2024-08-19T14:05:34.170 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968.target.wants/ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968@mgr.y.service → /etc/systemd/system/ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968@.service. 2024-08-19T14:05:34.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:34 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2534930147' entity='client.admin' 2024-08-19T14:05:34.366 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:34 smithi031 systemd[1]: Starting Ceph mgr.y for ed7d1c6e-5e33-11ef-bccf-c7b262605968... 2024-08-19T14:05:34.685 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:34 smithi031 podman[16547]: 2024-08-19 14:05:34.408017888 +0000 UTC m=+0.111296589 container create 5f1a20479194529c4b79d74a77cd8c89219f09873bb8b1c20a302a9886e97f60 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.36.0, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=squid-de09dd0) 2024-08-19T14:05:34.685 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:34 smithi031 podman[16547]: 2024-08-19 14:05:34.314006253 +0000 UTC m=+0.017284925 image pull ea7a5189f30bad3aca9524788e2965d2368d7ce786df09d4edcd8c3edf9cefba quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 2024-08-19T14:05:34.685 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:34 smithi031 podman[16547]: 2024-08-19 14:05:34.610063802 +0000 UTC m=+0.313342475 container init 5f1a20479194529c4b79d74a77cd8c89219f09873bb8b1c20a302a9886e97f60 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=squid-de09dd0, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image) 2024-08-19T14:05:34.685 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:34 smithi031 podman[16547]: 2024-08-19 14:05:34.612743506 +0000 UTC m=+0.316022166 container start 5f1a20479194529c4b79d74a77cd8c89219f09873bb8b1c20a302a9886e97f60 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=squid-de09dd0) 2024-08-19T14:05:34.685 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:34 smithi031 bash[16547]: 5f1a20479194529c4b79d74a77cd8c89219f09873bb8b1c20a302a9886e97f60 2024-08-19T14:05:34.699 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-08-19T14:05:34.699 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout disabled 2024-08-19T14:05:34.707 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-08-19T14:05:34.707 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout inactive 2024-08-19T14:05:34.707 INFO:teuthology.orchestra.run.smithi031.stdout:firewalld.service is not enabled 2024-08-19T14:05:34.707 INFO:teuthology.orchestra.run.smithi031.stdout:Not possible to enable service . firewalld.service is not available 2024-08-19T14:05:34.715 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-08-19T14:05:34.716 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout disabled 2024-08-19T14:05:34.724 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-08-19T14:05:34.724 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout inactive 2024-08-19T14:05:34.724 INFO:teuthology.orchestra.run.smithi031.stdout:firewalld.service is not enabled 2024-08-19T14:05:34.724 INFO:teuthology.orchestra.run.smithi031.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-08-19T14:05:34.724 INFO:teuthology.orchestra.run.smithi031.stdout:Waiting for mgr to start... 2024-08-19T14:05:34.724 INFO:teuthology.orchestra.run.smithi031.stdout:Waiting for mgr... 2024-08-19T14:05:34.946 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:34 smithi031 systemd[1]: Started Ceph mgr.y for ed7d1c6e-5e33-11ef-bccf-c7b262605968. 2024-08-19T14:05:34.947 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:34 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: 2024-08-19T14:05:34.832+0000 7f5c93fc2040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-08-19T14:05:34.947 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:34 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: 2024-08-19T14:05:34.901+0000 7f5c93fc2040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-08-19T14:05:35.197 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:35 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: 2024-08-19T14:05:35.053+0000 7f5c93fc2040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-08-19T14:05:35.506 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:35 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2038056923' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-08-19T14:05:35.507 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:35 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: 2024-08-19T14:05:35.225+0000 7f5c93fc2040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-08-19T14:05:35.507 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:35 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: 2024-08-19T14:05:35.374+0000 7f5c93fc2040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-08-19T14:05:35.507 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:35 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: 2024-08-19T14:05:35.504+0000 7f5c93fc2040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-08-19T14:05:35.753 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 2024-08-19T14:05:35.756 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout { 2024-08-19T14:05:35.756 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "fsid": "ed7d1c6e-5e33-11ef-bccf-c7b262605968", 2024-08-19T14:05:35.756 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "health": { 2024-08-19T14:05:35.756 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-08-19T14:05:35.756 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-08-19T14:05:35.756 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-08-19T14:05:35.756 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-08-19T14:05:35.756 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-08-19T14:05:35.756 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-08-19T14:05:35.756 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 0 2024-08-19T14:05:35.756 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout ], 2024-08-19T14:05:35.756 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-08-19T14:05:35.757 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "a" 2024-08-19T14:05:35.757 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout ], 2024-08-19T14:05:35.757 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-08-19T14:05:35.757 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "monmap": { 2024-08-19T14:05:35.757 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-08-19T14:05:35.757 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-08-19T14:05:35.757 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-08-19T14:05:35.757 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-08-19T14:05:35.757 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-08-19T14:05:35.757 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-08-19T14:05:35.757 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-08-19T14:05:35.757 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-08-19T14:05:35.757 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-08-19T14:05:35.757 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-08-19T14:05:35.758 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-08-19T14:05:35.758 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-08-19T14:05:35.761 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-08-19T14:05:35.761 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-08-19T14:05:35.761 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-08-19T14:05:35.761 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-08-19T14:05:35.761 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-08-19T14:05:35.761 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-08-19T14:05:35.761 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-08-19T14:05:35.761 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-08-19T14:05:35.762 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-08-19T14:05:35.762 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-08-19T14:05:35.762 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-08-19T14:05:35.762 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-08-19T14:05:35.762 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-08-19T14:05:35.762 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "btime": "2024-08-19T14:05:22:841940+0000", 2024-08-19T14:05:35.762 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-08-19T14:05:35.762 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-08-19T14:05:35.762 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-08-19T14:05:35.762 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-08-19T14:05:35.762 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "available": false, 2024-08-19T14:05:35.762 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-08-19T14:05:35.763 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "modules": [ 2024-08-19T14:05:35.763 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "iostat", 2024-08-19T14:05:35.763 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "nfs", 2024-08-19T14:05:35.763 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "restful" 2024-08-19T14:05:35.763 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout ], 2024-08-19T14:05:35.763 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "services": {} 2024-08-19T14:05:35.763 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-08-19T14:05:35.763 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-08-19T14:05:35.763 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-08-19T14:05:35.763 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "modified": "2024-08-19T14:05:22.842711+0000", 2024-08-19T14:05:35.763 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "services": {} 2024-08-19T14:05:35.763 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-08-19T14:05:35.763 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-08-19T14:05:35.763 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout } 2024-08-19T14:05:35.763 INFO:teuthology.orchestra.run.smithi031.stdout:mgr not available, waiting (1/15)... 2024-08-19T14:05:35.866 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:35 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: 2024-08-19T14:05:35.635+0000 7f5c93fc2040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-08-19T14:05:35.866 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:35 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: 2024-08-19T14:05:35.694+0000 7f5c93fc2040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-08-19T14:05:36.300 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:36 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: 2024-08-19T14:05:36.010+0000 7f5c93fc2040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-08-19T14:05:36.300 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:36 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: 2024-08-19T14:05:36.074+0000 7f5c93fc2040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-08-19T14:05:36.300 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:36 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: 2024-08-19T14:05:36.143+0000 7f5c93fc2040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-08-19T14:05:36.300 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:36 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: 2024-08-19T14:05:36.298+0000 7f5c93fc2040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-08-19T14:05:36.616 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:36 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: 2024-08-19T14:05:36.359+0000 7f5c93fc2040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-08-19T14:05:36.616 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:36 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: 2024-08-19T14:05:36.427+0000 7f5c93fc2040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-08-19T14:05:37.036 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:36 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: 2024-08-19T14:05:36.620+0000 7f5c93fc2040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-08-19T14:05:37.036 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:36 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: 2024-08-19T14:05:36.683+0000 7f5c93fc2040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-08-19T14:05:37.036 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:36 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: 2024-08-19T14:05:36.744+0000 7f5c93fc2040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-08-19T14:05:37.036 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:36 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: 2024-08-19T14:05:36.814+0000 7f5c93fc2040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-08-19T14:05:37.366 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:37 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: 2024-08-19T14:05:37.034+0000 7f5c93fc2040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-08-19T14:05:37.366 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:37 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: 2024-08-19T14:05:37.095+0000 7f5c93fc2040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-08-19T14:05:37.978 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:37 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: 2024-08-19T14:05:37.644+0000 7f5c93fc2040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-08-19T14:05:38.236 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:37 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: 2024-08-19T14:05:37.980+0000 7f5c93fc2040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-08-19T14:05:38.515 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:38 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3675740824' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-08-19T14:05:38.515 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:38 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: 2024-08-19T14:05:38.422+0000 7f5c93fc2040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-08-19T14:05:38.515 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:38 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: 2024-08-19T14:05:38.480+0000 7f5c93fc2040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-08-19T14:05:38.774 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:38 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: 2024-08-19T14:05:38.543+0000 7f5c93fc2040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-08-19T14:05:38.789 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 2024-08-19T14:05:38.790 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout { 2024-08-19T14:05:38.790 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "fsid": "ed7d1c6e-5e33-11ef-bccf-c7b262605968", 2024-08-19T14:05:38.790 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "health": { 2024-08-19T14:05:38.790 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-08-19T14:05:38.790 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-08-19T14:05:38.791 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-08-19T14:05:38.791 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-08-19T14:05:38.791 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-08-19T14:05:38.791 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-08-19T14:05:38.791 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 0 2024-08-19T14:05:38.792 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout ], 2024-08-19T14:05:38.792 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-08-19T14:05:38.792 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "a" 2024-08-19T14:05:38.792 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout ], 2024-08-19T14:05:38.792 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-08-19T14:05:38.793 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "monmap": { 2024-08-19T14:05:38.793 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-08-19T14:05:38.793 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-08-19T14:05:38.793 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-08-19T14:05:38.793 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-08-19T14:05:38.793 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-08-19T14:05:38.794 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-08-19T14:05:38.794 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-08-19T14:05:38.794 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-08-19T14:05:38.794 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-08-19T14:05:38.794 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-08-19T14:05:38.795 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-08-19T14:05:38.795 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-08-19T14:05:38.795 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-08-19T14:05:38.795 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-08-19T14:05:38.795 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-08-19T14:05:38.795 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-08-19T14:05:38.796 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-08-19T14:05:38.796 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-08-19T14:05:38.796 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-08-19T14:05:38.797 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-08-19T14:05:38.797 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-08-19T14:05:38.797 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-08-19T14:05:38.797 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-08-19T14:05:38.797 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-08-19T14:05:38.797 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-08-19T14:05:38.798 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "btime": "2024-08-19T14:05:22:841940+0000", 2024-08-19T14:05:38.798 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-08-19T14:05:38.798 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-08-19T14:05:38.798 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-08-19T14:05:38.798 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-08-19T14:05:38.799 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "available": false, 2024-08-19T14:05:38.799 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-08-19T14:05:38.799 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "modules": [ 2024-08-19T14:05:38.799 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "iostat", 2024-08-19T14:05:38.799 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "nfs", 2024-08-19T14:05:38.799 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "restful" 2024-08-19T14:05:38.800 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout ], 2024-08-19T14:05:38.800 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "services": {} 2024-08-19T14:05:38.800 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-08-19T14:05:38.800 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-08-19T14:05:38.800 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-08-19T14:05:38.801 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "modified": "2024-08-19T14:05:22.842711+0000", 2024-08-19T14:05:38.801 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "services": {} 2024-08-19T14:05:38.801 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-08-19T14:05:38.801 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-08-19T14:05:38.801 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout } 2024-08-19T14:05:38.801 INFO:teuthology.orchestra.run.smithi031.stdout:mgr not available, waiting (2/15)... 2024-08-19T14:05:40.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:39 smithi031 ceph-mon[16336]: Activating manager daemon y 2024-08-19T14:05:40.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:39 smithi031 ceph-mon[16336]: mgrmap e2: y(active, starting, since 0.00292953s) 2024-08-19T14:05:40.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:39 smithi031 ceph-mon[16336]: from='mgr.14100 172.21.15.31:0/3010541132' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-08-19T14:05:40.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:39 smithi031 ceph-mon[16336]: from='mgr.14100 172.21.15.31:0/3010541132' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-08-19T14:05:40.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:39 smithi031 ceph-mon[16336]: from='mgr.14100 172.21.15.31:0/3010541132' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-08-19T14:05:40.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:39 smithi031 ceph-mon[16336]: from='mgr.14100 172.21.15.31:0/3010541132' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:05:40.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:39 smithi031 ceph-mon[16336]: from='mgr.14100 172.21.15.31:0/3010541132' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-08-19T14:05:40.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:39 smithi031 ceph-mon[16336]: Manager daemon y is now available 2024-08-19T14:05:40.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:39 smithi031 ceph-mon[16336]: from='mgr.14100 172.21.15.31:0/3010541132' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-08-19T14:05:40.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:39 smithi031 ceph-mon[16336]: from='mgr.14100 172.21.15.31:0/3010541132' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-08-19T14:05:40.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:39 smithi031 ceph-mon[16336]: from='mgr.14100 172.21.15.31:0/3010541132' entity='mgr.y' 2024-08-19T14:05:40.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:39 smithi031 ceph-mon[16336]: from='mgr.14100 172.21.15.31:0/3010541132' entity='mgr.y' 2024-08-19T14:05:40.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:39 smithi031 ceph-mon[16336]: from='mgr.14100 172.21.15.31:0/3010541132' entity='mgr.y' 2024-08-19T14:05:41.771 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:41 smithi031 ceph-mon[16336]: mgrmap e3: y(active, since 1.00735s) 2024-08-19T14:05:41.771 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:41 smithi031 ceph-mon[16336]: from='client.14112 172.21.15.31:0/2929910784' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-08-19T14:05:42.044 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 2024-08-19T14:05:42.044 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout { 2024-08-19T14:05:42.044 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "fsid": "ed7d1c6e-5e33-11ef-bccf-c7b262605968", 2024-08-19T14:05:42.044 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "health": { 2024-08-19T14:05:42.045 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-08-19T14:05:42.045 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-08-19T14:05:42.045 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-08-19T14:05:42.045 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-08-19T14:05:42.045 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-08-19T14:05:42.045 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-08-19T14:05:42.045 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 0 2024-08-19T14:05:42.045 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout ], 2024-08-19T14:05:42.045 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-08-19T14:05:42.045 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "a" 2024-08-19T14:05:42.045 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout ], 2024-08-19T14:05:42.045 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-08-19T14:05:42.046 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "monmap": { 2024-08-19T14:05:42.046 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-08-19T14:05:42.046 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-08-19T14:05:42.046 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-08-19T14:05:42.046 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-08-19T14:05:42.046 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-08-19T14:05:42.046 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-08-19T14:05:42.046 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-08-19T14:05:42.046 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-08-19T14:05:42.046 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-08-19T14:05:42.046 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-08-19T14:05:42.046 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-08-19T14:05:42.047 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-08-19T14:05:42.047 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-08-19T14:05:42.047 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-08-19T14:05:42.050 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-08-19T14:05:42.050 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-08-19T14:05:42.050 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-08-19T14:05:42.050 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-08-19T14:05:42.051 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-08-19T14:05:42.051 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-08-19T14:05:42.051 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-08-19T14:05:42.051 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-08-19T14:05:42.051 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-08-19T14:05:42.051 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-08-19T14:05:42.051 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-08-19T14:05:42.051 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "btime": "2024-08-19T14:05:22:841940+0000", 2024-08-19T14:05:42.051 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-08-19T14:05:42.051 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-08-19T14:05:42.051 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-08-19T14:05:42.052 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-08-19T14:05:42.052 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "available": true, 2024-08-19T14:05:42.052 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-08-19T14:05:42.052 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "modules": [ 2024-08-19T14:05:42.052 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "iostat", 2024-08-19T14:05:42.052 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "nfs", 2024-08-19T14:05:42.052 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "restful" 2024-08-19T14:05:42.052 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout ], 2024-08-19T14:05:42.052 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "services": {} 2024-08-19T14:05:42.052 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-08-19T14:05:42.052 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-08-19T14:05:42.052 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-08-19T14:05:42.052 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "modified": "2024-08-19T14:05:22.842711+0000", 2024-08-19T14:05:42.053 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "services": {} 2024-08-19T14:05:42.053 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-08-19T14:05:42.053 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-08-19T14:05:42.053 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout } 2024-08-19T14:05:42.053 INFO:teuthology.orchestra.run.smithi031.stdout:mgr is available 2024-08-19T14:05:42.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:42 smithi031 ceph-mon[16336]: mgrmap e4: y(active, since 2s) 2024-08-19T14:05:42.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:42 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2574741211' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-08-19T14:05:43.099 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 2024-08-19T14:05:43.099 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout [global] 2024-08-19T14:05:43.099 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout fsid = ed7d1c6e-5e33-11ef-bccf-c7b262605968 2024-08-19T14:05:43.099 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.31:3300,v1:172.21.15.31:6789] 2024-08-19T14:05:43.099 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-08-19T14:05:43.099 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-08-19T14:05:43.099 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-08-19T14:05:43.099 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout ms_tcp_read_timeout = 5 2024-08-19T14:05:43.099 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-08-19T14:05:43.099 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 2024-08-19T14:05:43.100 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout [mgr] 2024-08-19T14:05:43.100 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-08-19T14:05:43.100 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 2024-08-19T14:05:43.100 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout [osd] 2024-08-19T14:05:43.100 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout filestore_queue_throttle_high_multiple = 2 2024-08-19T14:05:43.100 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout filestore_queue_throttle_max_multiple = 10 2024-08-19T14:05:43.100 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-08-19T14:05:43.100 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-08-19T14:05:43.100 INFO:teuthology.orchestra.run.smithi031.stdout:Enabling cephadm module... 2024-08-19T14:05:44.089 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:43 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/269103277' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-08-19T14:05:44.089 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:43 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: ignoring --setuser ceph since I am not root 2024-08-19T14:05:44.090 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:43 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: ignoring --setgroup ceph since I am not root 2024-08-19T14:05:44.090 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:44 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: 2024-08-19T14:05:44.020+0000 7fdfb09f3040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-08-19T14:05:44.090 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:44 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: 2024-08-19T14:05:44.088+0000 7fdfb09f3040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-08-19T14:05:44.351 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:44 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: 2024-08-19T14:05:44.238+0000 7fdfb09f3040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-08-19T14:05:44.616 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:44 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: 2024-08-19T14:05:44.410+0000 7fdfb09f3040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-08-19T14:05:44.616 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:44 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: 2024-08-19T14:05:44.559+0000 7fdfb09f3040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-08-19T14:05:45.014 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:44 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/269103277' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-08-19T14:05:45.014 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:44 smithi031 ceph-mon[16336]: mgrmap e5: y(active, since 4s) 2024-08-19T14:05:45.014 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:44 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: 2024-08-19T14:05:44.693+0000 7fdfb09f3040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-08-19T14:05:45.015 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:44 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: 2024-08-19T14:05:44.829+0000 7fdfb09f3040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-08-19T14:05:45.015 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:44 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: 2024-08-19T14:05:44.889+0000 7fdfb09f3040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-08-19T14:05:45.275 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:45 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: 2024-08-19T14:05:45.205+0000 7fdfb09f3040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-08-19T14:05:45.275 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:45 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: 2024-08-19T14:05:45.274+0000 7fdfb09f3040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-08-19T14:05:45.562 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:45 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: 2024-08-19T14:05:45.345+0000 7fdfb09f3040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-08-19T14:05:45.562 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:45 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: 2024-08-19T14:05:45.499+0000 7fdfb09f3040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-08-19T14:05:45.562 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:45 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: 2024-08-19T14:05:45.560+0000 7fdfb09f3040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-08-19T14:05:45.570 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout { 2024-08-19T14:05:45.570 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-08-19T14:05:45.570 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "available": true, 2024-08-19T14:05:45.571 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-08-19T14:05:45.571 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-08-19T14:05:45.571 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout } 2024-08-19T14:05:45.571 INFO:teuthology.orchestra.run.smithi031.stdout:Waiting for the mgr to restart... 2024-08-19T14:05:45.571 INFO:teuthology.orchestra.run.smithi031.stdout:Waiting for mgr epoch 5... 2024-08-19T14:05:45.816 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:45 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/802046693' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-08-19T14:05:45.816 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:45 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: 2024-08-19T14:05:45.636+0000 7fdfb09f3040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-08-19T14:05:46.116 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:45 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: 2024-08-19T14:05:45.831+0000 7fdfb09f3040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-08-19T14:05:46.116 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:45 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: 2024-08-19T14:05:45.894+0000 7fdfb09f3040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-08-19T14:05:46.116 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:45 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: 2024-08-19T14:05:45.957+0000 7fdfb09f3040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-08-19T14:05:46.116 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:46 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: 2024-08-19T14:05:46.029+0000 7fdfb09f3040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-08-19T14:05:46.616 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:46 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: 2024-08-19T14:05:46.247+0000 7fdfb09f3040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-08-19T14:05:46.616 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:46 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: 2024-08-19T14:05:46.308+0000 7fdfb09f3040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-08-19T14:05:47.116 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:46 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: 2024-08-19T14:05:46.841+0000 7fdfb09f3040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-08-19T14:05:47.592 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:47 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: 2024-08-19T14:05:47.160+0000 7fdfb09f3040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-08-19T14:05:47.866 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:47 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: 2024-08-19T14:05:47.591+0000 7fdfb09f3040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-08-19T14:05:47.866 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:47 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: 2024-08-19T14:05:47.649+0000 7fdfb09f3040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-08-19T14:05:47.866 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:47 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: 2024-08-19T14:05:47.710+0000 7fdfb09f3040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-08-19T14:05:49.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:48 smithi031 ceph-mon[16336]: Active manager daemon y restarted 2024-08-19T14:05:49.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:48 smithi031 ceph-mon[16336]: Activating manager daemon y 2024-08-19T14:05:49.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:48 smithi031 ceph-mon[16336]: osdmap e2: 0 total, 0 up, 0 in 2024-08-19T14:05:49.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:48 smithi031 ceph-mon[16336]: mgrmap e6: y(active, starting, since 0.00380484s) 2024-08-19T14:05:49.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:48 smithi031 ceph-mon[16336]: from='mgr.14118 172.21.15.31:0/3457336827' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:05:49.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:48 smithi031 ceph-mon[16336]: from='mgr.14118 172.21.15.31:0/3457336827' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-08-19T14:05:49.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:48 smithi031 ceph-mon[16336]: from='mgr.14118 172.21.15.31:0/3457336827' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-08-19T14:05:49.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:48 smithi031 ceph-mon[16336]: from='mgr.14118 172.21.15.31:0/3457336827' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-08-19T14:05:49.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:48 smithi031 ceph-mon[16336]: from='mgr.14118 172.21.15.31:0/3457336827' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-08-19T14:05:49.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:48 smithi031 ceph-mon[16336]: Manager daemon y is now available 2024-08-19T14:05:49.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:48 smithi031 ceph-mon[16336]: from='mgr.14118 172.21.15.31:0/3457336827' entity='mgr.y' 2024-08-19T14:05:49.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:48 smithi031 ceph-mon[16336]: from='mgr.14118 172.21.15.31:0/3457336827' entity='mgr.y' 2024-08-19T14:05:49.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:48 smithi031 ceph-mon[16336]: from='mgr.14118 172.21.15.31:0/3457336827' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:05:49.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:48 smithi031 ceph-mon[16336]: from='mgr.14118 172.21.15.31:0/3457336827' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:05:49.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:48 smithi031 ceph-mon[16336]: from='mgr.14118 172.21.15.31:0/3457336827' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-08-19T14:05:49.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:48 smithi031 ceph-mon[16336]: from='mgr.14118 172.21.15.31:0/3457336827' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-08-19T14:05:50.534 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout { 2024-08-19T14:05:50.534 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-08-19T14:05:50.534 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "initialized": true 2024-08-19T14:05:50.534 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout } 2024-08-19T14:05:50.534 INFO:teuthology.orchestra.run.smithi031.stdout:mgr epoch 5 is available 2024-08-19T14:05:50.534 INFO:teuthology.orchestra.run.smithi031.stdout:Setting orchestrator backend to cephadm... 2024-08-19T14:05:50.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:50 smithi031 ceph-mon[16336]: Found migration_current of "None". Setting to last migration. 2024-08-19T14:05:50.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:50 smithi031 ceph-mon[16336]: from='mgr.14118 172.21.15.31:0/3457336827' entity='mgr.y' 2024-08-19T14:05:50.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:50 smithi031 ceph-mon[16336]: from='mgr.14118 172.21.15.31:0/3457336827' entity='mgr.y' 2024-08-19T14:05:50.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:50 smithi031 ceph-mon[16336]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-08-19T14:05:50.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:50 smithi031 ceph-mon[16336]: mgrmap e7: y(active, since 1.00535s) 2024-08-19T14:05:51.796 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:51 smithi031 ceph-mon[16336]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-08-19T14:05:51.797 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:51 smithi031 ceph-mon[16336]: from='mgr.14118 172.21.15.31:0/3457336827' entity='mgr.y' 2024-08-19T14:05:51.797 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:51 smithi031 ceph-mon[16336]: from='mgr.14118 172.21.15.31:0/3457336827' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:05:52.069 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:51 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: 2024-08-19T14:05:51.795+0000 7fdf1d6cc640 -1 log_channel(cephadm) log [ERR] : [19/Aug/2024:14:05:51] ENGINE Error in HTTPServer.serve 2024-08-19T14:05:52.069 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:51 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: Traceback (most recent call last): 2024-08-19T14:05:52.069 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:51 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-08-19T14:05:52.069 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:51 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: self._connections.run(self.expiration_interval) 2024-08-19T14:05:52.069 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:51 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-08-19T14:05:52.069 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:51 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: self._run(expiration_interval) 2024-08-19T14:05:52.069 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:51 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-08-19T14:05:52.070 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:51 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: new_conn = self._from_server_socket(self.server.socket) 2024-08-19T14:05:52.070 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:51 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-08-19T14:05:52.070 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:51 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-08-19T14:05:52.070 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:51 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-08-19T14:05:52.070 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:51 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: s = self.context.wrap_socket( 2024-08-19T14:05:52.070 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:51 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-08-19T14:05:52.070 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:51 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: return self.sslsocket_class._create( 2024-08-19T14:05:52.070 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:51 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-08-19T14:05:52.070 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:51 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: self.do_handshake() 2024-08-19T14:05:52.070 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:51 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-08-19T14:05:52.070 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:51 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: self._sslobj.do_handshake() 2024-08-19T14:05:52.070 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:51 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-08-19T14:05:52.070 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:51 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: 2024-08-19T14:05:52.710 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:52 smithi031 ceph-mon[16336]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T14:05:52.710 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:52 smithi031 ceph-mon[16336]: mgrmap e8: y(active, since 2s) 2024-08-19T14:05:52.711 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:52 smithi031 ceph-mon[16336]: [19/Aug/2024:14:05:51] ENGINE Bus STARTING 2024-08-19T14:05:52.711 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:52 smithi031 ceph-mon[16336]: [19/Aug/2024:14:05:51] ENGINE Serving on https://172.21.15.31:7150 2024-08-19T14:05:52.711 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:52 smithi031 ceph-mon[16336]: [19/Aug/2024:14:05:51] ENGINE Error in HTTPServer.serve 2024-08-19T14:05:52.711 INFO:journalctl@ceph.mon.a.smithi031.stdout: Traceback (most recent call last): 2024-08-19T14:05:52.711 INFO:journalctl@ceph.mon.a.smithi031.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-08-19T14:05:52.711 INFO:journalctl@ceph.mon.a.smithi031.stdout: self._connections.run(self.expiration_interval) 2024-08-19T14:05:52.711 INFO:journalctl@ceph.mon.a.smithi031.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-08-19T14:05:52.711 INFO:journalctl@ceph.mon.a.smithi031.stdout: self._run(expiration_interval) 2024-08-19T14:05:52.711 INFO:journalctl@ceph.mon.a.smithi031.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-08-19T14:05:52.711 INFO:journalctl@ceph.mon.a.smithi031.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-08-19T14:05:52.711 INFO:journalctl@ceph.mon.a.smithi031.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-08-19T14:05:52.711 INFO:journalctl@ceph.mon.a.smithi031.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-08-19T14:05:52.712 INFO:journalctl@ceph.mon.a.smithi031.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-08-19T14:05:52.712 INFO:journalctl@ceph.mon.a.smithi031.stdout: s = self.context.wrap_socket( 2024-08-19T14:05:52.712 INFO:journalctl@ceph.mon.a.smithi031.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-08-19T14:05:52.712 INFO:journalctl@ceph.mon.a.smithi031.stdout: return self.sslsocket_class._create( 2024-08-19T14:05:52.712 INFO:journalctl@ceph.mon.a.smithi031.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-08-19T14:05:52.712 INFO:journalctl@ceph.mon.a.smithi031.stdout: self.do_handshake() 2024-08-19T14:05:52.712 INFO:journalctl@ceph.mon.a.smithi031.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-08-19T14:05:52.712 INFO:journalctl@ceph.mon.a.smithi031.stdout: self._sslobj.do_handshake() 2024-08-19T14:05:52.712 INFO:journalctl@ceph.mon.a.smithi031.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-08-19T14:05:52.712 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:52 smithi031 ceph-mon[16336]: from='mgr.14118 172.21.15.31:0/3457336827' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:05:52.997 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout value unchanged 2024-08-19T14:05:52.997 INFO:teuthology.orchestra.run.smithi031.stdout:Generating ssh key... 2024-08-19T14:05:53.508 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:53 smithi031 ceph-mon[16336]: [19/Aug/2024:14:05:51] ENGINE Serving on http://172.21.15.31:8765 2024-08-19T14:05:53.509 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:53 smithi031 ceph-mon[16336]: [19/Aug/2024:14:05:51] ENGINE Bus STARTED 2024-08-19T14:05:53.509 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:53 smithi031 ceph-mon[16336]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T14:05:54.203 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:53 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: Generating public/private rsa key pair. 2024-08-19T14:05:54.203 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:53 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: Your identification has been saved in /tmp/tmphzygu87g/key 2024-08-19T14:05:54.204 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:53 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: Your public key has been saved in /tmp/tmphzygu87g/key.pub 2024-08-19T14:05:54.204 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:53 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: The key fingerprint is: 2024-08-19T14:05:54.204 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:53 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: SHA256:+USACn5VdjgDKobj52L0VwsQK27o1hn9EFZs/yzDdCs ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968 2024-08-19T14:05:54.204 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:53 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: The key's randomart image is: 2024-08-19T14:05:54.204 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:53 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: +---[RSA 3072]----+ 2024-08-19T14:05:54.204 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:53 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: | . o=+.. | 2024-08-19T14:05:54.204 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:53 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: | .. o.+++o | 2024-08-19T14:05:54.204 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:53 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: |ooo+.=. .o. | 2024-08-19T14:05:54.204 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:53 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: |+oo.* . = . | 2024-08-19T14:05:54.204 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:53 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: |.= + + .S = . | 2024-08-19T14:05:54.204 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:53 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: |+ = o = .E + | 2024-08-19T14:05:54.204 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:53 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: | = = . o = | 2024-08-19T14:05:54.204 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:53 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: |o . . | 2024-08-19T14:05:54.205 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:53 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: | | 2024-08-19T14:05:54.205 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:05:53 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: +----[SHA256]-----+ 2024-08-19T14:05:54.460 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:54 smithi031 ceph-mon[16336]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T14:05:54.461 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:54 smithi031 ceph-mon[16336]: Generating ssh key... 2024-08-19T14:05:54.461 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:54 smithi031 ceph-mon[16336]: from='mgr.14118 172.21.15.31:0/3457336827' entity='mgr.y' 2024-08-19T14:05:54.461 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:54 smithi031 ceph-mon[16336]: from='mgr.14118 172.21.15.31:0/3457336827' entity='mgr.y' 2024-08-19T14:05:55.629 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDzYj8EIBzi7K0ZxURrNHT44hwvNCoQYr1Zvijc0ZjPAVRBz8xZ3EzgfaG7VjYpRN6Hg3w9XdxCiWVsZ5GqnCap1ZTF4+2iv6p+TyIX5uaLko2zKO6ASpw6qLwXK3rhogmodiqJGSqRPQcdckZobqPWm/lKPdlKm5ioLldzjoLLBTebpGmfXnlQXS/PXagirJZfNGhbN+2HIf7jl3ymeaWbuM9jxrXWPJ6tOfWktNPYWOuWSGxZPJo0c6bIU/uLd/Fp6HFEGwWhq5e/v/GFQAllYjYJieiDbh/ed9jXZCuL8sqL/6vGSaphOAPw5tv7ntbp987VEWo+oCvgHJnfI84Q5O8OBadRooCF8LnqQDJo9PXfnUJ4T0lV4WhW2gLmoxGfb4wVXhelhqY2ivj0M0d3kGodAbks9xYVB6fD5mG32ZIC4znudtZL6Zry3m8vfxCZe2F32Plcgudn2CeF3tRn6ND64P6MiEyL26KAOXIa8GgnEniSo9O3tVBsF8x/+/M= ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968 2024-08-19T14:05:55.629 INFO:teuthology.orchestra.run.smithi031.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-08-19T14:05:55.629 INFO:teuthology.orchestra.run.smithi031.stdout:Adding key to root@localhost authorized_keys... 2024-08-19T14:05:55.630 INFO:teuthology.orchestra.run.smithi031.stdout:Adding host smithi031... 2024-08-19T14:05:56.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:55 smithi031 ceph-mon[16336]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T14:05:57.258 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:56 smithi031 ceph-mon[16336]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi031", "addr": "172.21.15.31", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T14:05:58.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:05:57 smithi031 ceph-mon[16336]: Deploying cephadm binary to smithi031 2024-08-19T14:06:01.012 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout Added host 'smithi031' with addr '172.21.15.31' 2024-08-19T14:06:01.012 INFO:teuthology.orchestra.run.smithi031.stdout:Deploying unmanaged mon service... 2024-08-19T14:06:01.267 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:01 smithi031 ceph-mon[16336]: from='mgr.14118 172.21.15.31:0/3457336827' entity='mgr.y' 2024-08-19T14:06:01.267 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:01 smithi031 ceph-mon[16336]: Added host smithi031 2024-08-19T14:06:01.267 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:01 smithi031 ceph-mon[16336]: from='mgr.14118 172.21.15.31:0/3457336827' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:06:02.813 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:02 smithi031 ceph-mon[16336]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T14:06:02.813 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:02 smithi031 ceph-mon[16336]: Saving service mon spec with placement count:5 2024-08-19T14:06:02.813 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:02 smithi031 ceph-mon[16336]: from='mgr.14118 172.21.15.31:0/3457336827' entity='mgr.y' 2024-08-19T14:06:03.042 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-08-19T14:06:03.042 INFO:teuthology.orchestra.run.smithi031.stdout:Deploying unmanaged mgr service... 2024-08-19T14:06:03.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:03 smithi031 ceph-mon[16336]: from='mgr.14118 172.21.15.31:0/3457336827' entity='mgr.y' 2024-08-19T14:06:03.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:03 smithi031 ceph-mon[16336]: from='mgr.14118 172.21.15.31:0/3457336827' entity='mgr.y' 2024-08-19T14:06:03.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:03 smithi031 ceph-mon[16336]: from='mgr.14118 172.21.15.31:0/3457336827' entity='mgr.y' 2024-08-19T14:06:04.841 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:04 smithi031 ceph-mon[16336]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T14:06:04.841 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:04 smithi031 ceph-mon[16336]: Saving service mgr spec with placement count:2 2024-08-19T14:06:04.841 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:04 smithi031 ceph-mon[16336]: from='mgr.14118 172.21.15.31:0/3457336827' entity='mgr.y' 2024-08-19T14:06:04.843 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-08-19T14:06:06.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:06 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1893311481' entity='client.admin' 2024-08-19T14:06:06.870 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:06 smithi031 ceph-mon[16336]: from='mgr.14118 172.21.15.31:0/3457336827' entity='mgr.y' 2024-08-19T14:06:07.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:07 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/848554522' entity='client.admin' 2024-08-19T14:06:07.945 INFO:teuthology.orchestra.run.smithi031.stdout:Enabling the dashboard module... 2024-08-19T14:06:08.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:08 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/680089234' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-08-19T14:06:08.866 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:06:08 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: ignoring --setuser ceph since I am not root 2024-08-19T14:06:08.867 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:06:08 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: ignoring --setgroup ceph since I am not root 2024-08-19T14:06:08.867 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:06:08 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: 2024-08-19T14:06:08.848+0000 7ff4ee5b8040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-08-19T14:06:09.191 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:06:08 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: 2024-08-19T14:06:08.917+0000 7ff4ee5b8040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-08-19T14:06:09.191 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:06:09 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: 2024-08-19T14:06:09.066+0000 7ff4ee5b8040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-08-19T14:06:09.494 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:06:09 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: 2024-08-19T14:06:09.239+0000 7ff4ee5b8040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-08-19T14:06:09.494 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:06:09 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: 2024-08-19T14:06:09.388+0000 7ff4ee5b8040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-08-19T14:06:09.811 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:09 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/680089234' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-08-19T14:06:09.812 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:09 smithi031 ceph-mon[16336]: mgrmap e9: y(active, since 19s) 2024-08-19T14:06:09.812 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:06:09 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: 2024-08-19T14:06:09.522+0000 7ff4ee5b8040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-08-19T14:06:09.812 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:06:09 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: 2024-08-19T14:06:09.654+0000 7ff4ee5b8040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-08-19T14:06:09.812 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:06:09 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: 2024-08-19T14:06:09.713+0000 7ff4ee5b8040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-08-19T14:06:10.095 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:06:10 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: 2024-08-19T14:06:10.031+0000 7ff4ee5b8040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-08-19T14:06:10.095 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:06:10 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: 2024-08-19T14:06:10.094+0000 7ff4ee5b8040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-08-19T14:06:10.318 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout { 2024-08-19T14:06:10.318 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-08-19T14:06:10.318 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "available": true, 2024-08-19T14:06:10.318 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-08-19T14:06:10.318 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-08-19T14:06:10.318 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout } 2024-08-19T14:06:10.319 INFO:teuthology.orchestra.run.smithi031.stdout:Waiting for the mgr to restart... 2024-08-19T14:06:10.319 INFO:teuthology.orchestra.run.smithi031.stdout:Waiting for mgr epoch 9... 2024-08-19T14:06:10.366 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:06:10 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: 2024-08-19T14:06:10.165+0000 7ff4ee5b8040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-08-19T14:06:10.366 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:06:10 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: 2024-08-19T14:06:10.318+0000 7ff4ee5b8040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-08-19T14:06:10.648 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:10 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/977624437' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-08-19T14:06:10.649 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:06:10 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: 2024-08-19T14:06:10.383+0000 7ff4ee5b8040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-08-19T14:06:10.649 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:06:10 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: 2024-08-19T14:06:10.452+0000 7ff4ee5b8040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-08-19T14:06:10.649 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:06:10 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: 2024-08-19T14:06:10.647+0000 7ff4ee5b8040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-08-19T14:06:11.067 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:06:10 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: 2024-08-19T14:06:10.712+0000 7ff4ee5b8040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-08-19T14:06:11.067 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:06:10 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: 2024-08-19T14:06:10.776+0000 7ff4ee5b8040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-08-19T14:06:11.067 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:06:10 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: 2024-08-19T14:06:10.846+0000 7ff4ee5b8040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-08-19T14:06:11.067 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:06:11 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: 2024-08-19T14:06:11.066+0000 7ff4ee5b8040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-08-19T14:06:11.366 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:06:11 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: 2024-08-19T14:06:11.127+0000 7ff4ee5b8040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-08-19T14:06:11.984 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:06:11 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: 2024-08-19T14:06:11.662+0000 7ff4ee5b8040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-08-19T14:06:12.366 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:06:11 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: 2024-08-19T14:06:11.983+0000 7ff4ee5b8040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-08-19T14:06:12.866 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:06:12 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: 2024-08-19T14:06:12.413+0000 7ff4ee5b8040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-08-19T14:06:12.866 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:06:12 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: 2024-08-19T14:06:12.472+0000 7ff4ee5b8040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-08-19T14:06:12.866 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:06:12 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: 2024-08-19T14:06:12.534+0000 7ff4ee5b8040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-08-19T14:06:14.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:13 smithi031 ceph-mon[16336]: Active manager daemon y restarted 2024-08-19T14:06:14.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:13 smithi031 ceph-mon[16336]: Activating manager daemon y 2024-08-19T14:06:14.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:13 smithi031 ceph-mon[16336]: osdmap e3: 0 total, 0 up, 0 in 2024-08-19T14:06:14.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:13 smithi031 ceph-mon[16336]: mgrmap e10: y(active, starting, since 0.00327782s) 2024-08-19T14:06:14.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:13 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:06:14.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:13 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-08-19T14:06:14.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:13 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-08-19T14:06:14.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:13 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-08-19T14:06:14.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:13 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-08-19T14:06:14.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:13 smithi031 ceph-mon[16336]: Manager daemon y is now available 2024-08-19T14:06:14.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:13 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:06:14.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:13 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-08-19T14:06:14.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:13 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-08-19T14:06:15.508 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout { 2024-08-19T14:06:15.508 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-08-19T14:06:15.508 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "initialized": true 2024-08-19T14:06:15.508 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout } 2024-08-19T14:06:15.508 INFO:teuthology.orchestra.run.smithi031.stdout:mgr epoch 9 is available 2024-08-19T14:06:15.508 INFO:teuthology.orchestra.run.smithi031.stdout:Generating a dashboard self-signed certificate... 2024-08-19T14:06:15.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:15 smithi031 ceph-mon[16336]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-08-19T14:06:15.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:15 smithi031 ceph-mon[16336]: mgrmap e11: y(active, since 1.00578s) 2024-08-19T14:06:15.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:15 smithi031 ceph-mon[16336]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-08-19T14:06:15.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:15 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:06:16.706 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-08-19T14:06:16.706 INFO:teuthology.orchestra.run.smithi031.stdout:Creating initial admin user... 2024-08-19T14:06:16.892 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:06:16 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: 2024-08-19T14:06:16.671+0000 7ff454075640 -1 log_channel(cephadm) log [ERR] : [19/Aug/2024:14:06:16] ENGINE Error in HTTPServer.serve 2024-08-19T14:06:16.892 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:06:16 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: Traceback (most recent call last): 2024-08-19T14:06:16.892 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:06:16 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-08-19T14:06:16.892 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:06:16 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: self._connections.run(self.expiration_interval) 2024-08-19T14:06:16.892 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:06:16 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-08-19T14:06:16.892 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:06:16 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: self._run(expiration_interval) 2024-08-19T14:06:16.892 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:06:16 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-08-19T14:06:16.892 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:06:16 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: new_conn = self._from_server_socket(self.server.socket) 2024-08-19T14:06:16.892 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:06:16 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-08-19T14:06:16.892 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:06:16 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-08-19T14:06:16.892 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:06:16 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-08-19T14:06:16.893 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:06:16 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: s = self.context.wrap_socket( 2024-08-19T14:06:16.893 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:06:16 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-08-19T14:06:16.893 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:06:16 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: return self.sslsocket_class._create( 2024-08-19T14:06:16.893 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:06:16 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-08-19T14:06:16.893 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:06:16 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: self.do_handshake() 2024-08-19T14:06:16.893 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:06:16 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-08-19T14:06:16.893 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:06:16 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: self._sslobj.do_handshake() 2024-08-19T14:06:16.893 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:06:16 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-08-19T14:06:16.893 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:06:16 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: 2024-08-19T14:06:17.232 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:17 smithi031 ceph-mon[16336]: mgrmap e12: y(active, since 2s) 2024-08-19T14:06:17.232 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:17 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:06:17.232 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:17 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:06:17.232 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:17 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:06:17.232 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:17 smithi031 ceph-mon[16336]: [19/Aug/2024:14:06:16] ENGINE Bus STARTING 2024-08-19T14:06:17.232 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:17 smithi031 ceph-mon[16336]: [19/Aug/2024:14:06:16] ENGINE Serving on http://172.21.15.31:8765 2024-08-19T14:06:18.531 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$MOFRTlrzDpM0.xMn27l9muQYDz4wnMvKrWviuurbb/l/V/ZCLHddK", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1724076377, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-08-19T14:06:18.531 INFO:teuthology.orchestra.run.smithi031.stdout:Fetching dashboard port number... 2024-08-19T14:06:18.773 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:18 smithi031 ceph-mon[16336]: [19/Aug/2024:14:06:16] ENGINE Serving on https://172.21.15.31:7150 2024-08-19T14:06:18.773 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:18 smithi031 ceph-mon[16336]: [19/Aug/2024:14:06:16] ENGINE Bus STARTED 2024-08-19T14:06:18.773 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:18 smithi031 ceph-mon[16336]: [19/Aug/2024:14:06:16] ENGINE Error in HTTPServer.serve 2024-08-19T14:06:18.773 INFO:journalctl@ceph.mon.a.smithi031.stdout: Traceback (most recent call last): 2024-08-19T14:06:18.773 INFO:journalctl@ceph.mon.a.smithi031.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-08-19T14:06:18.773 INFO:journalctl@ceph.mon.a.smithi031.stdout: self._connections.run(self.expiration_interval) 2024-08-19T14:06:18.773 INFO:journalctl@ceph.mon.a.smithi031.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-08-19T14:06:18.773 INFO:journalctl@ceph.mon.a.smithi031.stdout: self._run(expiration_interval) 2024-08-19T14:06:18.774 INFO:journalctl@ceph.mon.a.smithi031.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-08-19T14:06:18.774 INFO:journalctl@ceph.mon.a.smithi031.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-08-19T14:06:18.774 INFO:journalctl@ceph.mon.a.smithi031.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-08-19T14:06:18.774 INFO:journalctl@ceph.mon.a.smithi031.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-08-19T14:06:18.774 INFO:journalctl@ceph.mon.a.smithi031.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-08-19T14:06:18.774 INFO:journalctl@ceph.mon.a.smithi031.stdout: s = self.context.wrap_socket( 2024-08-19T14:06:18.774 INFO:journalctl@ceph.mon.a.smithi031.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-08-19T14:06:18.774 INFO:journalctl@ceph.mon.a.smithi031.stdout: return self.sslsocket_class._create( 2024-08-19T14:06:18.774 INFO:journalctl@ceph.mon.a.smithi031.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-08-19T14:06:18.774 INFO:journalctl@ceph.mon.a.smithi031.stdout: self.do_handshake() 2024-08-19T14:06:18.774 INFO:journalctl@ceph.mon.a.smithi031.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-08-19T14:06:18.774 INFO:journalctl@ceph.mon.a.smithi031.stdout: self._sslobj.do_handshake() 2024-08-19T14:06:18.774 INFO:journalctl@ceph.mon.a.smithi031.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-08-19T14:06:18.775 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:18 smithi031 ceph-mon[16336]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T14:06:18.775 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:18 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:06:19.686 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 8443 2024-08-19T14:06:19.696 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-08-19T14:06:19.696 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout disabled 2024-08-19T14:06:19.704 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-08-19T14:06:19.704 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout inactive 2024-08-19T14:06:19.704 INFO:teuthology.orchestra.run.smithi031.stdout:firewalld.service is not enabled 2024-08-19T14:06:19.704 INFO:teuthology.orchestra.run.smithi031.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-08-19T14:06:19.706 INFO:teuthology.orchestra.run.smithi031.stdout:Ceph Dashboard is now available at: 2024-08-19T14:06:19.706 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:06:19.706 INFO:teuthology.orchestra.run.smithi031.stdout: URL: https://smithi031:8443/ 2024-08-19T14:06:19.706 INFO:teuthology.orchestra.run.smithi031.stdout: User: admin 2024-08-19T14:06:19.706 INFO:teuthology.orchestra.run.smithi031.stdout: Password: 8u8byqenaq 2024-08-19T14:06:19.706 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:06:19.706 INFO:teuthology.orchestra.run.smithi031.stdout:Saving cluster configuration to /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/config directory 2024-08-19T14:06:19.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:19 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2398945645' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-08-19T14:06:20.951 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-08-19T14:06:20.951 INFO:teuthology.orchestra.run.smithi031.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-08-19T14:06:20.951 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:06:20.951 INFO:teuthology.orchestra.run.smithi031.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-08-19T14:06:20.951 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:06:20.951 INFO:teuthology.orchestra.run.smithi031.stdout:Or, if you are only running a single cluster on this host: 2024-08-19T14:06:20.951 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:06:20.951 INFO:teuthology.orchestra.run.smithi031.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-08-19T14:06:20.951 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:06:20.951 INFO:teuthology.orchestra.run.smithi031.stdout:Please consider enabling telemetry to help improve Ceph: 2024-08-19T14:06:20.951 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:06:20.951 INFO:teuthology.orchestra.run.smithi031.stdout: ceph telemetry on 2024-08-19T14:06:20.952 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:06:20.952 INFO:teuthology.orchestra.run.smithi031.stdout:For more information see: 2024-08-19T14:06:20.952 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:06:20.952 INFO:teuthology.orchestra.run.smithi031.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-08-19T14:06:20.952 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:06:20.952 INFO:teuthology.orchestra.run.smithi031.stdout:Bootstrap complete. 2024-08-19T14:06:20.994 INFO:tasks.cephadm:Fetching config... 2024-08-19T14:06:20.995 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-08-19T14:06:20.995 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-08-19T14:06:21.011 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-08-19T14:06:21.011 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-08-19T14:06:21.011 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-08-19T14:06:21.067 INFO:tasks.cephadm:Fetching mon keyring... 2024-08-19T14:06:21.067 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-08-19T14:06:21.067 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/keyring of=/dev/stdout 2024-08-19T14:06:21.133 INFO:tasks.cephadm:Fetching pub ssh key... 2024-08-19T14:06:21.134 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-08-19T14:06:21.134 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-08-19T14:06:21.188 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-08-19T14:06:21.189 DEBUG:teuthology.orchestra.run.smithi031:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDzYj8EIBzi7K0ZxURrNHT44hwvNCoQYr1Zvijc0ZjPAVRBz8xZ3EzgfaG7VjYpRN6Hg3w9XdxCiWVsZ5GqnCap1ZTF4+2iv6p+TyIX5uaLko2zKO6ASpw6qLwXK3rhogmodiqJGSqRPQcdckZobqPWm/lKPdlKm5ioLldzjoLLBTebpGmfXnlQXS/PXagirJZfNGhbN+2HIf7jl3ymeaWbuM9jxrXWPJ6tOfWktNPYWOuWSGxZPJo0c6bIU/uLd/Fp6HFEGwWhq5e/v/GFQAllYjYJieiDbh/ed9jXZCuL8sqL/6vGSaphOAPw5tv7ntbp987VEWo+oCvgHJnfI84Q5O8OBadRooCF8LnqQDJo9PXfnUJ4T0lV4WhW2gLmoxGfb4wVXhelhqY2ivj0M0d3kGodAbks9xYVB6fD5mG32ZIC4znudtZL6Zry3m8vfxCZe2F32Plcgudn2CeF3tRn6ND64P6MiEyL26KAOXIa8GgnEniSo9O3tVBsF8x/+/M= ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-08-19T14:06:21.265 INFO:teuthology.orchestra.run.smithi031.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDzYj8EIBzi7K0ZxURrNHT44hwvNCoQYr1Zvijc0ZjPAVRBz8xZ3EzgfaG7VjYpRN6Hg3w9XdxCiWVsZ5GqnCap1ZTF4+2iv6p+TyIX5uaLko2zKO6ASpw6qLwXK3rhogmodiqJGSqRPQcdckZobqPWm/lKPdlKm5ioLldzjoLLBTebpGmfXnlQXS/PXagirJZfNGhbN+2HIf7jl3ymeaWbuM9jxrXWPJ6tOfWktNPYWOuWSGxZPJo0c6bIU/uLd/Fp6HFEGwWhq5e/v/GFQAllYjYJieiDbh/ed9jXZCuL8sqL/6vGSaphOAPw5tv7ntbp987VEWo+oCvgHJnfI84Q5O8OBadRooCF8LnqQDJo9PXfnUJ4T0lV4WhW2gLmoxGfb4wVXhelhqY2ivj0M0d3kGodAbks9xYVB6fD5mG32ZIC4znudtZL6Zry3m8vfxCZe2F32Plcgudn2CeF3tRn6ND64P6MiEyL26KAOXIa8GgnEniSo9O3tVBsF8x/+/M= ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968 2024-08-19T14:06:21.276 DEBUG:teuthology.orchestra.run.smithi032:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDzYj8EIBzi7K0ZxURrNHT44hwvNCoQYr1Zvijc0ZjPAVRBz8xZ3EzgfaG7VjYpRN6Hg3w9XdxCiWVsZ5GqnCap1ZTF4+2iv6p+TyIX5uaLko2zKO6ASpw6qLwXK3rhogmodiqJGSqRPQcdckZobqPWm/lKPdlKm5ioLldzjoLLBTebpGmfXnlQXS/PXagirJZfNGhbN+2HIf7jl3ymeaWbuM9jxrXWPJ6tOfWktNPYWOuWSGxZPJo0c6bIU/uLd/Fp6HFEGwWhq5e/v/GFQAllYjYJieiDbh/ed9jXZCuL8sqL/6vGSaphOAPw5tv7ntbp987VEWo+oCvgHJnfI84Q5O8OBadRooCF8LnqQDJo9PXfnUJ4T0lV4WhW2gLmoxGfb4wVXhelhqY2ivj0M0d3kGodAbks9xYVB6fD5mG32ZIC4znudtZL6Zry3m8vfxCZe2F32Plcgudn2CeF3tRn6ND64P6MiEyL26KAOXIa8GgnEniSo9O3tVBsF8x/+/M= ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-08-19T14:06:21.318 INFO:teuthology.orchestra.run.smithi032.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDzYj8EIBzi7K0ZxURrNHT44hwvNCoQYr1Zvijc0ZjPAVRBz8xZ3EzgfaG7VjYpRN6Hg3w9XdxCiWVsZ5GqnCap1ZTF4+2iv6p+TyIX5uaLko2zKO6ASpw6qLwXK3rhogmodiqJGSqRPQcdckZobqPWm/lKPdlKm5ioLldzjoLLBTebpGmfXnlQXS/PXagirJZfNGhbN+2HIf7jl3ymeaWbuM9jxrXWPJ6tOfWktNPYWOuWSGxZPJo0c6bIU/uLd/Fp6HFEGwWhq5e/v/GFQAllYjYJieiDbh/ed9jXZCuL8sqL/6vGSaphOAPw5tv7ntbp987VEWo+oCvgHJnfI84Q5O8OBadRooCF8LnqQDJo9PXfnUJ4T0lV4WhW2gLmoxGfb4wVXhelhqY2ivj0M0d3kGodAbks9xYVB6fD5mG32ZIC4znudtZL6Zry3m8vfxCZe2F32Plcgudn2CeF3tRn6ND64P6MiEyL26KAOXIa8GgnEniSo9O3tVBsF8x/+/M= ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968 2024-08-19T14:06:21.330 DEBUG:teuthology.orchestra.run.smithi045:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDzYj8EIBzi7K0ZxURrNHT44hwvNCoQYr1Zvijc0ZjPAVRBz8xZ3EzgfaG7VjYpRN6Hg3w9XdxCiWVsZ5GqnCap1ZTF4+2iv6p+TyIX5uaLko2zKO6ASpw6qLwXK3rhogmodiqJGSqRPQcdckZobqPWm/lKPdlKm5ioLldzjoLLBTebpGmfXnlQXS/PXagirJZfNGhbN+2HIf7jl3ymeaWbuM9jxrXWPJ6tOfWktNPYWOuWSGxZPJo0c6bIU/uLd/Fp6HFEGwWhq5e/v/GFQAllYjYJieiDbh/ed9jXZCuL8sqL/6vGSaphOAPw5tv7ntbp987VEWo+oCvgHJnfI84Q5O8OBadRooCF8LnqQDJo9PXfnUJ4T0lV4WhW2gLmoxGfb4wVXhelhqY2ivj0M0d3kGodAbks9xYVB6fD5mG32ZIC4znudtZL6Zry3m8vfxCZe2F32Plcgudn2CeF3tRn6ND64P6MiEyL26KAOXIa8GgnEniSo9O3tVBsF8x/+/M= ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-08-19T14:06:21.372 INFO:teuthology.orchestra.run.smithi045.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDzYj8EIBzi7K0ZxURrNHT44hwvNCoQYr1Zvijc0ZjPAVRBz8xZ3EzgfaG7VjYpRN6Hg3w9XdxCiWVsZ5GqnCap1ZTF4+2iv6p+TyIX5uaLko2zKO6ASpw6qLwXK3rhogmodiqJGSqRPQcdckZobqPWm/lKPdlKm5ioLldzjoLLBTebpGmfXnlQXS/PXagirJZfNGhbN+2HIf7jl3ymeaWbuM9jxrXWPJ6tOfWktNPYWOuWSGxZPJo0c6bIU/uLd/Fp6HFEGwWhq5e/v/GFQAllYjYJieiDbh/ed9jXZCuL8sqL/6vGSaphOAPw5tv7ntbp987VEWo+oCvgHJnfI84Q5O8OBadRooCF8LnqQDJo9PXfnUJ4T0lV4WhW2gLmoxGfb4wVXhelhqY2ivj0M0d3kGodAbks9xYVB6fD5mG32ZIC4znudtZL6Zry3m8vfxCZe2F32Plcgudn2CeF3tRn6ND64P6MiEyL26KAOXIa8GgnEniSo9O3tVBsF8x/+/M= ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968 2024-08-19T14:06:21.385 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-08-19T14:06:21.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:21 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/132218830' entity='client.admin' 2024-08-19T14:06:21.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:21 smithi031 ceph-mon[16336]: mgrmap e13: y(active, since 6s) 2024-08-19T14:06:21.635 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:06:23.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:23 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1612190052' entity='client.admin' 2024-08-19T14:06:23.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:23 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:06:23.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:23 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:06:23.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:23 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi031", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:06:23.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:23 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:06:23.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:23 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:06:23.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:23 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:06:23.574 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-08-19T14:06:23.574 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-08-19T14:06:23.884 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:06:25.304 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi032 2024-08-19T14:06:25.304 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-19T14:06:25.304 DEBUG:teuthology.orchestra.run.smithi032:> dd of=/etc/ceph/ceph.conf 2024-08-19T14:06:25.324 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-19T14:06:25.324 DEBUG:teuthology.orchestra.run.smithi032:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-08-19T14:06:25.382 INFO:tasks.cephadm:Adding host smithi032 to orchestrator... 2024-08-19T14:06:25.383 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph orch host add smithi032 2024-08-19T14:06:25.818 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:06:25.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:25 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:06:25.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:25 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:06:25.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:25 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:06:25.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:25 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:06:25.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:25 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:06:25.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:25 smithi031 ceph-mon[16336]: Updating smithi031:/etc/ceph/ceph.conf 2024-08-19T14:06:25.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:25 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:06:25.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:25 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:06:25.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:25 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:06:25.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:25 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:06:25.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:25 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:06:25.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:25 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:06:25.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:25 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:06:26.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:26 smithi031 ceph-mon[16336]: Updating smithi031:/var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-19T14:06:26.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:26 smithi031 ceph-mon[16336]: Updating smithi031:/etc/ceph/ceph.client.admin.keyring 2024-08-19T14:06:26.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:26 smithi031 ceph-mon[16336]: Updating smithi031:/var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/config/ceph.client.admin.keyring 2024-08-19T14:06:27.869 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:27 smithi031 ceph-mon[16336]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi032", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T14:06:28.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:28 smithi031 ceph-mon[16336]: Deploying cephadm binary to smithi032 2024-08-19T14:06:30.375 INFO:teuthology.orchestra.run.smithi031.stdout:Added host 'smithi032' with addr '172.21.15.32' 2024-08-19T14:06:30.954 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph orch host ls --format=json 2024-08-19T14:06:31.204 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:06:31.586 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:31 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:06:31.586 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:31 smithi031 ceph-mon[16336]: Added host smithi032 2024-08-19T14:06:31.586 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:31 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:06:31.586 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:31 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:06:31.586 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:31 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:06:31.860 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:06:31.860 INFO:teuthology.orchestra.run.smithi031.stdout:[{"addr": "172.21.15.31", "hostname": "smithi031", "labels": [], "status": ""}, {"addr": "172.21.15.32", "hostname": "smithi032", "labels": [], "status": ""}] 2024-08-19T14:06:32.477 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi045 2024-08-19T14:06:32.477 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-08-19T14:06:32.477 DEBUG:teuthology.orchestra.run.smithi045:> dd of=/etc/ceph/ceph.conf 2024-08-19T14:06:32.496 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-08-19T14:06:32.496 DEBUG:teuthology.orchestra.run.smithi045:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-08-19T14:06:32.554 INFO:tasks.cephadm:Adding host smithi045 to orchestrator... 2024-08-19T14:06:32.555 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph orch host add smithi045 2024-08-19T14:06:32.734 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:32 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:06:32.735 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:32 smithi031 ceph-mon[16336]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:06:32.820 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:06:34.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:33 smithi031 ceph-mon[16336]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi045", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T14:06:35.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:35 smithi031 ceph-mon[16336]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T14:06:35.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:35 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:06:35.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:35 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:06:35.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:35 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:06:35.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:35 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:06:35.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:35 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi032", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:06:35.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:35 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:06:35.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:35 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:06:35.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:35 smithi031 ceph-mon[16336]: Updating smithi032:/etc/ceph/ceph.conf 2024-08-19T14:06:35.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:35 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:06:35.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:35 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:06:35.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:35 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:06:36.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:36 smithi031 ceph-mon[16336]: Deploying cephadm binary to smithi045 2024-08-19T14:06:36.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:36 smithi031 ceph-mon[16336]: Updating smithi032:/var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-19T14:06:36.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:36 smithi031 ceph-mon[16336]: Updating smithi032:/etc/ceph/ceph.client.admin.keyring 2024-08-19T14:06:36.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:36 smithi031 ceph-mon[16336]: Updating smithi032:/var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/config/ceph.client.admin.keyring 2024-08-19T14:06:37.438 INFO:teuthology.orchestra.run.smithi031.stdout:Added host 'smithi045' with addr '172.21.15.45' 2024-08-19T14:06:37.761 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:37 smithi031 ceph-mon[16336]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T14:06:37.761 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:37 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:06:37.761 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:37 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:06:38.052 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph orch host ls --format=json 2024-08-19T14:06:38.305 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:06:38.755 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:38 smithi031 ceph-mon[16336]: Added host smithi045 2024-08-19T14:06:38.755 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:38 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:06:38.755 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:38 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:06:38.929 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:06:38.929 INFO:teuthology.orchestra.run.smithi031.stdout:[{"addr": "172.21.15.31", "hostname": "smithi031", "labels": [], "status": ""}, {"addr": "172.21.15.32", "hostname": "smithi032", "labels": [], "status": ""}, {"addr": "172.21.15.45", "hostname": "smithi045", "labels": [], "status": ""}] 2024-08-19T14:06:39.522 INFO:tasks.cephadm:Setting crush tunables to default 2024-08-19T14:06:39.522 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd crush tunables default 2024-08-19T14:06:39.772 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:06:39.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:39 smithi031 ceph-mon[16336]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T14:06:39.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:39 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:06:40.603 INFO:teuthology.orchestra.run.smithi031.stderr:adjusted tunables profile to default 2024-08-19T14:06:40.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:40 smithi031 ceph-mon[16336]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:06:40.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:40 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1213544896' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-08-19T14:06:41.161 INFO:tasks.cephadm:Adding mon.a on smithi031 2024-08-19T14:06:41.161 INFO:tasks.cephadm:Adding mon.b on smithi032 2024-08-19T14:06:41.162 INFO:tasks.cephadm:Adding mon.c on smithi045 2024-08-19T14:06:41.162 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph orch apply mon '3;smithi031:172.21.15.31=a;smithi032:172.21.15.32=b;smithi045:172.21.15.45=c' 2024-08-19T14:06:41.565 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /etc/ceph/ceph.conf 2024-08-19T14:06:41.612 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /etc/ceph/ceph.conf 2024-08-19T14:06:41.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:41 smithi031 ceph-mon[16336]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T14:06:41.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:41 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1213544896' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-08-19T14:06:41.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:41 smithi031 ceph-mon[16336]: osdmap e4: 0 total, 0 up, 0 in 2024-08-19T14:06:41.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:41 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:06:41.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:41 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:06:41.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:41 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:06:41.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:41 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:06:41.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:41 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi045", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:06:41.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:41 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:06:41.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:41 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:06:42.293 INFO:teuthology.orchestra.run.smithi045.stdout:Scheduled mon update... 2024-08-19T14:06:42.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:42 smithi031 ceph-mon[16336]: Updating smithi045:/etc/ceph/ceph.conf 2024-08-19T14:06:42.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:42 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:06:42.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:42 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:06:42.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:42 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:06:42.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:42 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:06:42.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:42 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-08-19T14:06:42.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:42 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:06:42.926 DEBUG:teuthology.orchestra.run.smithi032:mon.b> sudo journalctl -f -n 0 -u ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968@mon.b.service 2024-08-19T14:06:42.929 DEBUG:teuthology.orchestra.run.smithi045:mon.c> sudo journalctl -f -n 0 -u ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968@mon.c.service 2024-08-19T14:06:42.931 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-08-19T14:06:42.931 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph mon dump -f json 2024-08-19T14:06:43.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:43 smithi031 ceph-mon[16336]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T14:06:43.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:43 smithi031 ceph-mon[16336]: Updating smithi045:/var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-19T14:06:43.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:43 smithi031 ceph-mon[16336]: Updating smithi045:/etc/ceph/ceph.client.admin.keyring 2024-08-19T14:06:43.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:43 smithi031 ceph-mon[16336]: Updating smithi045:/var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/config/ceph.client.admin.keyring 2024-08-19T14:06:43.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:43 smithi031 ceph-mon[16336]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi031:172.21.15.31=a;smithi032:172.21.15.32=b;smithi045:172.21.15.45=c", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T14:06:43.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:43 smithi031 ceph-mon[16336]: Saving service mon spec with placement smithi031:172.21.15.31=a;smithi032:172.21.15.32=b;smithi045:172.21.15.45=c;count:3 2024-08-19T14:06:43.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:43 smithi031 ceph-mon[16336]: Deploying daemon mon.c on smithi045 2024-08-19T14:06:44.744 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-19T14:06:45.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:45 smithi031 ceph-mon[16336]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T14:06:46.069 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-08-19T14:06:46.070 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":1,"fsid":"ed7d1c6e-5e33-11ef-bccf-c7b262605968","modified":"2024-08-19T14:05:16.209575Z","created":"2024-08-19T14:05:16.209575Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:3300","nonce":0},{"type":"v1","addr":"172.21.15.31:6789","nonce":0}]},"addr":"172.21.15.31:6789/0","public_addr":"172.21.15.31:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-08-19T14:06:46.070 INFO:teuthology.orchestra.run.smithi045.stderr:dumped monmap epoch 1 2024-08-19T14:06:46.328 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:46 smithi045 systemd[1]: Starting Ceph mon.c for ed7d1c6e-5e33-11ef-bccf-c7b262605968... 2024-08-19T14:06:46.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:46 smithi031 ceph-mon[16336]: from='client.? 172.21.15.45:0/1173249320' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-08-19T14:06:47.131 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:46 smithi045 podman[20124]: 2024-08-19 14:06:46.870174656 +0000 UTC m=+0.113400030 container create 168d75d038216a2938bbf98b7348243fef43172e45f916a50eea621f2234813e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mon-c, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=squid-de09dd0, ceph=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0) 2024-08-19T14:06:47.132 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:46 smithi045 podman[20124]: 2024-08-19 14:06:46.774441653 +0000 UTC m=+0.017666981 image pull ea7a5189f30bad3aca9524788e2965d2368d7ce786df09d4edcd8c3edf9cefba quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9 2024-08-19T14:06:47.132 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 podman[20124]: 2024-08-19 14:06:47.063971522 +0000 UTC m=+0.307196860 container init 168d75d038216a2938bbf98b7348243fef43172e45f916a50eea621f2234813e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mon-c, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-de09dd0, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True) 2024-08-19T14:06:47.132 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 podman[20124]: 2024-08-19 14:06:47.06698172 +0000 UTC m=+0.310207039 container start 168d75d038216a2938bbf98b7348243fef43172e45f916a50eea621f2234813e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mon-c, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=squid-de09dd0, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, CEPH_POINT_RELEASE=) 2024-08-19T14:06:47.132 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: set uid:gid to 167:167 (ceph:ceph) 2024-08-19T14:06:47.132 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: ceph version 19.1.0-1382-gde09dd01 (de09dd018de4baddfa6ca4d43bc8fb784cf75833) squid (rc), process ceph-mon, pid 2 2024-08-19T14:06:47.132 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: pidfile_write: ignore empty --pid-file 2024-08-19T14:06:47.132 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: load: jerasure load: lrc 2024-08-19T14:06:47.132 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: RocksDB version: 7.9.2 2024-08-19T14:06:47.132 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Git sha 0 2024-08-19T14:06:47.132 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Compile date 2024-08-18 15:19:56 2024-08-19T14:06:47.132 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: DB SUMMARY 2024-08-19T14:06:47.133 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: DB Session ID: BTW1UXPMRAKFO0D7HD6Z 2024-08-19T14:06:47.133 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: CURRENT file: CURRENT 2024-08-19T14:06:47.133 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: IDENTITY file: IDENTITY 2024-08-19T14:06:47.133 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-08-19T14:06:47.133 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-08-19T14:06:47.133 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-08-19T14:06:47.135 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.error_if_exists: 0 2024-08-19T14:06:47.135 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.create_if_missing: 0 2024-08-19T14:06:47.135 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.paranoid_checks: 1 2024-08-19T14:06:47.135 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.flush_verify_memtable_count: 1 2024-08-19T14:06:47.135 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-08-19T14:06:47.135 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-08-19T14:06:47.135 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.env: 0x562d7a9127a0 2024-08-19T14:06:47.135 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.fs: PosixFileSystem 2024-08-19T14:06:47.135 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.info_log: 0x562d7bf43500 2024-08-19T14:06:47.136 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.max_file_opening_threads: 16 2024-08-19T14:06:47.136 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.statistics: (nil) 2024-08-19T14:06:47.136 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.use_fsync: 0 2024-08-19T14:06:47.136 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.max_log_file_size: 0 2024-08-19T14:06:47.136 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-08-19T14:06:47.136 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.log_file_time_to_roll: 0 2024-08-19T14:06:47.136 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.keep_log_file_num: 1000 2024-08-19T14:06:47.136 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.recycle_log_file_num: 0 2024-08-19T14:06:47.136 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.allow_fallocate: 1 2024-08-19T14:06:47.136 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.allow_mmap_reads: 0 2024-08-19T14:06:47.136 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.allow_mmap_writes: 0 2024-08-19T14:06:47.136 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.use_direct_reads: 0 2024-08-19T14:06:47.137 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-08-19T14:06:47.137 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.create_missing_column_families: 0 2024-08-19T14:06:47.137 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.db_log_dir: 2024-08-19T14:06:47.137 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.wal_dir: 2024-08-19T14:06:47.137 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.table_cache_numshardbits: 6 2024-08-19T14:06:47.137 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.WAL_ttl_seconds: 0 2024-08-19T14:06:47.137 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.WAL_size_limit_MB: 0 2024-08-19T14:06:47.137 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-08-19T14:06:47.137 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-08-19T14:06:47.137 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.is_fd_close_on_exec: 1 2024-08-19T14:06:47.137 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.advise_random_on_open: 1 2024-08-19T14:06:47.138 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.db_write_buffer_size: 0 2024-08-19T14:06:47.138 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.write_buffer_manager: 0x562d7bf52dc0 2024-08-19T14:06:47.138 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-08-19T14:06:47.138 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-08-19T14:06:47.138 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.use_adaptive_mutex: 0 2024-08-19T14:06:47.138 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.rate_limiter: (nil) 2024-08-19T14:06:47.138 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-08-19T14:06:47.138 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.wal_recovery_mode: 2 2024-08-19T14:06:47.138 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.enable_thread_tracking: 0 2024-08-19T14:06:47.138 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.enable_pipelined_write: 0 2024-08-19T14:06:47.138 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.unordered_write: 0 2024-08-19T14:06:47.139 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-08-19T14:06:47.139 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-08-19T14:06:47.139 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-08-19T14:06:47.139 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-08-19T14:06:47.139 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.row_cache: None 2024-08-19T14:06:47.139 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.wal_filter: None 2024-08-19T14:06:47.139 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-08-19T14:06:47.139 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.allow_ingest_behind: 0 2024-08-19T14:06:47.139 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.two_write_queues: 0 2024-08-19T14:06:47.139 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.manual_wal_flush: 0 2024-08-19T14:06:47.139 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.wal_compression: 0 2024-08-19T14:06:47.139 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.atomic_flush: 0 2024-08-19T14:06:47.139 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-08-19T14:06:47.139 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.persist_stats_to_disk: 0 2024-08-19T14:06:47.140 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.write_dbid_to_manifest: 0 2024-08-19T14:06:47.140 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.log_readahead_size: 0 2024-08-19T14:06:47.140 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-08-19T14:06:47.140 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.best_efforts_recovery: 0 2024-08-19T14:06:47.140 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-08-19T14:06:47.140 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-08-19T14:06:47.140 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.allow_data_in_errors: 0 2024-08-19T14:06:47.140 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.db_host_id: __hostname__ 2024-08-19T14:06:47.140 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.enforce_single_del_contracts: true 2024-08-19T14:06:47.140 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.max_background_jobs: 2 2024-08-19T14:06:47.140 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.max_background_compactions: -1 2024-08-19T14:06:47.140 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.max_subcompactions: 1 2024-08-19T14:06:47.140 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-08-19T14:06:47.141 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-08-19T14:06:47.141 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.delayed_write_rate : 16777216 2024-08-19T14:06:47.141 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.max_total_wal_size: 0 2024-08-19T14:06:47.141 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-08-19T14:06:47.141 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.stats_dump_period_sec: 600 2024-08-19T14:06:47.141 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.stats_persist_period_sec: 600 2024-08-19T14:06:47.141 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-08-19T14:06:47.141 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.max_open_files: -1 2024-08-19T14:06:47.141 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.bytes_per_sync: 0 2024-08-19T14:06:47.141 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.wal_bytes_per_sync: 0 2024-08-19T14:06:47.141 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.strict_bytes_per_sync: 0 2024-08-19T14:06:47.141 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.compaction_readahead_size: 0 2024-08-19T14:06:47.142 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.max_background_flushes: -1 2024-08-19T14:06:47.142 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Compression algorithms supported: 2024-08-19T14:06:47.142 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: kZSTD supported: 0 2024-08-19T14:06:47.142 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: kXpressCompression supported: 0 2024-08-19T14:06:47.142 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: kBZip2Compression supported: 0 2024-08-19T14:06:47.142 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-08-19T14:06:47.142 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: kLZ4Compression supported: 1 2024-08-19T14:06:47.142 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: kZlibCompression supported: 1 2024-08-19T14:06:47.142 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: kLZ4HCCompression supported: 1 2024-08-19T14:06:47.142 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: kSnappyCompression supported: 1 2024-08-19T14:06:47.142 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Fast CRC32 supported: Supported on x86 2024-08-19T14:06:47.142 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: DMutex implementation: pthread_mutex_t 2024-08-19T14:06:47.142 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-08-19T14:06:47.143 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-08-19T14:06:47.143 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-08-19T14:06:47.143 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.merge_operator: 2024-08-19T14:06:47.143 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.compaction_filter: None 2024-08-19T14:06:47.143 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.compaction_filter_factory: None 2024-08-19T14:06:47.143 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.sst_partitioner_factory: None 2024-08-19T14:06:47.143 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.memtable_factory: SkipListFactory 2024-08-19T14:06:47.143 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.table_factory: BlockBasedTable 2024-08-19T14:06:47.143 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x562d7bf43120) 2024-08-19T14:06:47.143 INFO:journalctl@ceph.mon.c.smithi045.stdout: cache_index_and_filter_blocks: 1 2024-08-19T14:06:47.143 INFO:journalctl@ceph.mon.c.smithi045.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-08-19T14:06:47.143 INFO:journalctl@ceph.mon.c.smithi045.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-08-19T14:06:47.144 INFO:journalctl@ceph.mon.c.smithi045.stdout: pin_top_level_index_and_filter: 1 2024-08-19T14:06:47.144 INFO:journalctl@ceph.mon.c.smithi045.stdout: index_type: 0 2024-08-19T14:06:47.144 INFO:journalctl@ceph.mon.c.smithi045.stdout: data_block_index_type: 0 2024-08-19T14:06:47.144 INFO:journalctl@ceph.mon.c.smithi045.stdout: index_shortening: 1 2024-08-19T14:06:47.144 INFO:journalctl@ceph.mon.c.smithi045.stdout: data_block_hash_table_util_ratio: 0.750000 2024-08-19T14:06:47.144 INFO:journalctl@ceph.mon.c.smithi045.stdout: checksum: 4 2024-08-19T14:06:47.144 INFO:journalctl@ceph.mon.c.smithi045.stdout: no_block_cache: 0 2024-08-19T14:06:47.144 INFO:journalctl@ceph.mon.c.smithi045.stdout: block_cache: 0x562d7bf371f0 2024-08-19T14:06:47.144 INFO:journalctl@ceph.mon.c.smithi045.stdout: block_cache_name: BinnedLRUCache 2024-08-19T14:06:47.144 INFO:journalctl@ceph.mon.c.smithi045.stdout: block_cache_options: 2024-08-19T14:06:47.144 INFO:journalctl@ceph.mon.c.smithi045.stdout: capacity : 536870912 2024-08-19T14:06:47.144 INFO:journalctl@ceph.mon.c.smithi045.stdout: num_shard_bits : 4 2024-08-19T14:06:47.144 INFO:journalctl@ceph.mon.c.smithi045.stdout: strict_capacity_limit : 0 2024-08-19T14:06:47.145 INFO:journalctl@ceph.mon.c.smithi045.stdout: high_pri_pool_ratio: 0.000 2024-08-19T14:06:47.145 INFO:journalctl@ceph.mon.c.smithi045.stdout: block_cache_compressed: (nil) 2024-08-19T14:06:47.145 INFO:journalctl@ceph.mon.c.smithi045.stdout: persistent_cache: (nil) 2024-08-19T14:06:47.145 INFO:journalctl@ceph.mon.c.smithi045.stdout: block_size: 4096 2024-08-19T14:06:47.145 INFO:journalctl@ceph.mon.c.smithi045.stdout: block_size_deviation: 10 2024-08-19T14:06:47.145 INFO:journalctl@ceph.mon.c.smithi045.stdout: block_restart_interval: 16 2024-08-19T14:06:47.145 INFO:journalctl@ceph.mon.c.smithi045.stdout: index_block_restart_interval: 1 2024-08-19T14:06:47.145 INFO:journalctl@ceph.mon.c.smithi045.stdout: metadata_block_size: 4096 2024-08-19T14:06:47.145 INFO:journalctl@ceph.mon.c.smithi045.stdout: partition_filters: 0 2024-08-19T14:06:47.145 INFO:journalctl@ceph.mon.c.smithi045.stdout: use_delta_encoding: 1 2024-08-19T14:06:47.145 INFO:journalctl@ceph.mon.c.smithi045.stdout: filter_policy: bloomfilter 2024-08-19T14:06:47.145 INFO:journalctl@ceph.mon.c.smithi045.stdout: whole_key_filtering: 1 2024-08-19T14:06:47.145 INFO:journalctl@ceph.mon.c.smithi045.stdout: verify_compression: 0 2024-08-19T14:06:47.145 INFO:journalctl@ceph.mon.c.smithi045.stdout: read_amp_bytes_per_bit: 0 2024-08-19T14:06:47.146 INFO:journalctl@ceph.mon.c.smithi045.stdout: format_version: 5 2024-08-19T14:06:47.146 INFO:journalctl@ceph.mon.c.smithi045.stdout: enable_index_compression: 1 2024-08-19T14:06:47.146 INFO:journalctl@ceph.mon.c.smithi045.stdout: block_align: 0 2024-08-19T14:06:47.146 INFO:journalctl@ceph.mon.c.smithi045.stdout: max_auto_readahead_size: 262144 2024-08-19T14:06:47.146 INFO:journalctl@ceph.mon.c.smithi045.stdout: prepopulate_block_cache: 0 2024-08-19T14:06:47.146 INFO:journalctl@ceph.mon.c.smithi045.stdout: initial_auto_readahead_size: 8192 2024-08-19T14:06:47.146 INFO:journalctl@ceph.mon.c.smithi045.stdout: num_file_reads_for_auto_readahead: 2 2024-08-19T14:06:47.146 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.write_buffer_size: 33554432 2024-08-19T14:06:47.146 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.max_write_buffer_number: 2 2024-08-19T14:06:47.146 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.compression: NoCompression 2024-08-19T14:06:47.146 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.bottommost_compression: Disabled 2024-08-19T14:06:47.146 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.prefix_extractor: nullptr 2024-08-19T14:06:47.146 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-08-19T14:06:47.147 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.num_levels: 7 2024-08-19T14:06:47.147 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-08-19T14:06:47.147 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-08-19T14:06:47.147 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-08-19T14:06:47.147 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-08-19T14:06:47.147 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-08-19T14:06:47.147 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-08-19T14:06:47.147 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-08-19T14:06:47.147 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-08-19T14:06:47.147 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-08-19T14:06:47.147 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-08-19T14:06:47.147 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-08-19T14:06:47.147 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-08-19T14:06:47.147 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.compression_opts.window_bits: -14 2024-08-19T14:06:47.148 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.compression_opts.level: 32767 2024-08-19T14:06:47.148 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.compression_opts.strategy: 0 2024-08-19T14:06:47.148 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-08-19T14:06:47.148 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-08-19T14:06:47.148 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-08-19T14:06:47.148 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-08-19T14:06:47.148 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.compression_opts.enabled: false 2024-08-19T14:06:47.148 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-08-19T14:06:47.148 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-08-19T14:06:47.148 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-08-19T14:06:47.148 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-08-19T14:06:47.148 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.target_file_size_base: 67108864 2024-08-19T14:06:47.148 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.target_file_size_multiplier: 1 2024-08-19T14:06:47.148 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-08-19T14:06:47.149 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-08-19T14:06:47.149 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-08-19T14:06:47.149 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-08-19T14:06:47.149 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-08-19T14:06:47.149 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-08-19T14:06:47.149 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-08-19T14:06:47.149 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-08-19T14:06:47.149 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-08-19T14:06:47.149 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-08-19T14:06:47.149 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-08-19T14:06:47.149 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-08-19T14:06:47.149 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-08-19T14:06:47.149 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.arena_block_size: 1048576 2024-08-19T14:06:47.149 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-08-19T14:06:47.149 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-08-19T14:06:47.150 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.disable_auto_compactions: 0 2024-08-19T14:06:47.150 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-08-19T14:06:47.150 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-08-19T14:06:47.150 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-08-19T14:06:47.150 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-08-19T14:06:47.150 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-08-19T14:06:47.150 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-08-19T14:06:47.150 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-08-19T14:06:47.150 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-08-19T14:06:47.150 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-08-19T14:06:47.150 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-08-19T14:06:47.150 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-08-19T14:06:47.150 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.inplace_update_support: 0 2024-08-19T14:06:47.151 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.inplace_update_num_locks: 10000 2024-08-19T14:06:47.151 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-08-19T14:06:47.151 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-08-19T14:06:47.151 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.memtable_huge_page_size: 0 2024-08-19T14:06:47.151 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.bloom_locality: 0 2024-08-19T14:06:47.151 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.max_successive_merges: 0 2024-08-19T14:06:47.151 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.optimize_filters_for_hits: 0 2024-08-19T14:06:47.151 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.paranoid_file_checks: 0 2024-08-19T14:06:47.151 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.force_consistency_checks: 1 2024-08-19T14:06:47.151 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.report_bg_io_stats: 0 2024-08-19T14:06:47.151 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.ttl: 2592000 2024-08-19T14:06:47.151 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.periodic_compaction_seconds: 0 2024-08-19T14:06:47.151 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-08-19T14:06:47.152 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-08-19T14:06:47.152 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.enable_blob_files: false 2024-08-19T14:06:47.152 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.min_blob_size: 0 2024-08-19T14:06:47.152 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.blob_file_size: 268435456 2024-08-19T14:06:47.152 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.blob_compression_type: NoCompression 2024-08-19T14:06:47.152 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.enable_blob_garbage_collection: false 2024-08-19T14:06:47.152 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-08-19T14:06:47.152 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-08-19T14:06:47.152 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-08-19T14:06:47.152 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.blob_file_starting_level: 0 2024-08-19T14:06:47.152 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-08-19T14:06:47.152 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-08-19T14:06:47.152 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-08-19T14:06:47.152 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: a74aced2-2819-41b7-a0db-808bdf7d3145 2024-08-19T14:06:47.152 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: EVENT_LOG_v1 {"time_micros": 1724076407093860, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-08-19T14:06:47.153 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-08-19T14:06:47.153 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: EVENT_LOG_v1 {"time_micros": 1724076407094266, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1648, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1724076407, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "a74aced2-2819-41b7-a0db-808bdf7d3145", "db_session_id": "BTW1UXPMRAKFO0D7HD6Z", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-08-19T14:06:47.153 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: EVENT_LOG_v1 {"time_micros": 1724076407094364, "job": 1, "event": "recovery_finished"} 2024-08-19T14:06:47.153 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-08-19T14:06:47.153 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-08-19T14:06:47.153 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x562d7bf64e00 2024-08-19T14:06:47.153 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: DB pointer 0x562d7c05e000 2024-08-19T14:06:47.153 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-08-19T14:06:47.153 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-08-19T14:06:47.153 INFO:journalctl@ceph.mon.c.smithi045.stdout: ** DB Stats ** 2024-08-19T14:06:47.153 INFO:journalctl@ceph.mon.c.smithi045.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-08-19T14:06:47.153 INFO:journalctl@ceph.mon.c.smithi045.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-08-19T14:06:47.153 INFO:journalctl@ceph.mon.c.smithi045.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-08-19T14:06:47.154 INFO:journalctl@ceph.mon.c.smithi045.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-08-19T14:06:47.154 INFO:journalctl@ceph.mon.c.smithi045.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-08-19T14:06:47.154 INFO:journalctl@ceph.mon.c.smithi045.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-08-19T14:06:47.154 INFO:journalctl@ceph.mon.c.smithi045.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-08-19T14:06:47.154 INFO:journalctl@ceph.mon.c.smithi045.stdout: 2024-08-19T14:06:47.154 INFO:journalctl@ceph.mon.c.smithi045.stdout: ** Compaction Stats [default] ** 2024-08-19T14:06:47.154 INFO:journalctl@ceph.mon.c.smithi045.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-08-19T14:06:47.154 INFO:journalctl@ceph.mon.c.smithi045.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-08-19T14:06:47.154 INFO:journalctl@ceph.mon.c.smithi045.stdout: L0 1/0 1.61 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-08-19T14:06:47.154 INFO:journalctl@ceph.mon.c.smithi045.stdout: Sum 1/0 1.61 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-08-19T14:06:47.154 INFO:journalctl@ceph.mon.c.smithi045.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-08-19T14:06:47.154 INFO:journalctl@ceph.mon.c.smithi045.stdout: 2024-08-19T14:06:47.154 INFO:journalctl@ceph.mon.c.smithi045.stdout: ** Compaction Stats [default] ** 2024-08-19T14:06:47.155 INFO:journalctl@ceph.mon.c.smithi045.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-08-19T14:06:47.155 INFO:journalctl@ceph.mon.c.smithi045.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-08-19T14:06:47.155 INFO:journalctl@ceph.mon.c.smithi045.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-08-19T14:06:47.155 INFO:journalctl@ceph.mon.c.smithi045.stdout: 2024-08-19T14:06:47.155 INFO:journalctl@ceph.mon.c.smithi045.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-08-19T14:06:47.155 INFO:journalctl@ceph.mon.c.smithi045.stdout: 2024-08-19T14:06:47.155 INFO:journalctl@ceph.mon.c.smithi045.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-08-19T14:06:47.155 INFO:journalctl@ceph.mon.c.smithi045.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-08-19T14:06:47.155 INFO:journalctl@ceph.mon.c.smithi045.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-08-19T14:06:47.155 INFO:journalctl@ceph.mon.c.smithi045.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-08-19T14:06:47.155 INFO:journalctl@ceph.mon.c.smithi045.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-08-19T14:06:47.155 INFO:journalctl@ceph.mon.c.smithi045.stdout: AddFile(Keys): cumulative 0, interval 0 2024-08-19T14:06:47.155 INFO:journalctl@ceph.mon.c.smithi045.stdout: Cumulative compaction: 0.00 GB write, 0.59 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-08-19T14:06:47.155 INFO:journalctl@ceph.mon.c.smithi045.stdout: Interval compaction: 0.00 GB write, 0.59 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-08-19T14:06:47.156 INFO:journalctl@ceph.mon.c.smithi045.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-08-19T14:06:47.156 INFO:journalctl@ceph.mon.c.smithi045.stdout: Block cache BinnedLRUCache@0x562d7bf371f0#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-08-19T14:06:47.156 INFO:journalctl@ceph.mon.c.smithi045.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-08-19T14:06:47.156 INFO:journalctl@ceph.mon.c.smithi045.stdout: 2024-08-19T14:06:47.156 INFO:journalctl@ceph.mon.c.smithi045.stdout: ** File Read Latency Histogram By Level [default] ** 2024-08-19T14:06:47.156 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-08-19T14:06:47.156 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: using public_addr v2:172.21.15.45:0/0 -> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-08-19T14:06:47.156 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: starting mon.c rank -1 at public addrs [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] at bind addrs [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 2024-08-19T14:06:47.156 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: mon.c@-1(???) e0 preinit fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 2024-08-19T14:06:47.156 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: mon.c@-1(synchronizing).mds e1 new map 2024-08-19T14:06:47.156 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: mon.c@-1(synchronizing).mds e1 print_map 2024-08-19T14:06:47.156 INFO:journalctl@ceph.mon.c.smithi045.stdout: e1 2024-08-19T14:06:47.156 INFO:journalctl@ceph.mon.c.smithi045.stdout: btime 2024-08-19T14:05:22:841940+0000 2024-08-19T14:06:47.157 INFO:journalctl@ceph.mon.c.smithi045.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-08-19T14:06:47.157 INFO:journalctl@ceph.mon.c.smithi045.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-08-19T14:06:47.157 INFO:journalctl@ceph.mon.c.smithi045.stdout: legacy client fscid: -1 2024-08-19T14:06:47.157 INFO:journalctl@ceph.mon.c.smithi045.stdout: 2024-08-19T14:06:47.157 INFO:journalctl@ceph.mon.c.smithi045.stdout: No filesystems configured 2024-08-19T14:06:47.157 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-08-19T14:06:47.157 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-08-19T14:06:47.157 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-08-19T14:06:47.157 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-08-19T14:06:47.157 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-08-19T14:06:47.157 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-08-19T14:06:47.157 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-08-19T14:06:47.157 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-08-19T14:06:47.157 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-08-19T14:06:47.157 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-08-19T14:06:47.158 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: Added host smithi045 2024-08-19T14:06:47.158 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:06:47.158 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:06:47.158 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T14:06:47.158 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:06:47.158 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:06:47.158 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1213544896' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-08-19T14:06:47.158 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T14:06:47.158 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1213544896' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-08-19T14:06:47.158 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: osdmap e4: 0 total, 0 up, 0 in 2024-08-19T14:06:47.158 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:06:47.158 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:06:47.158 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:06:47.158 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:06:47.159 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi045", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:06:47.159 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:06:47.159 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:06:47.159 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: Updating smithi045:/etc/ceph/ceph.conf 2024-08-19T14:06:47.159 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:06:47.159 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:06:47.159 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:06:47.159 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:06:47.159 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-08-19T14:06:47.159 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:06:47.159 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T14:06:47.159 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: Updating smithi045:/var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-19T14:06:47.159 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: Updating smithi045:/etc/ceph/ceph.client.admin.keyring 2024-08-19T14:06:47.159 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: Updating smithi045:/var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/config/ceph.client.admin.keyring 2024-08-19T14:06:47.159 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi031:172.21.15.31=a;smithi032:172.21.15.32=b;smithi045:172.21.15.45=c", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T14:06:47.160 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: Saving service mon spec with placement smithi031:172.21.15.31=a;smithi032:172.21.15.32=b;smithi045:172.21.15.45=c;count:3 2024-08-19T14:06:47.160 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: Deploying daemon mon.c on smithi045 2024-08-19T14:06:47.160 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T14:06:47.160 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: from='client.? 172.21.15.45:0/1173249320' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-08-19T14:06:47.160 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 ceph-mon[20143]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-08-19T14:06:47.160 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 bash[20124]: 168d75d038216a2938bbf98b7348243fef43172e45f916a50eea621f2234813e 2024-08-19T14:06:47.160 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:47 smithi045 systemd[1]: Started Ceph mon.c for ed7d1c6e-5e33-11ef-bccf-c7b262605968. 2024-08-19T14:06:47.626 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-08-19T14:06:47.626 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph mon dump -f json 2024-08-19T14:06:47.878 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:06:52.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:52 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:06:52.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:52 smithi045 ceph-mon[20143]: mon.a calling monitor election 2024-08-19T14:06:52.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:52 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:06:52.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:52 smithi045 ceph-mon[20143]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T14:06:52.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:52 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:06:52.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:52 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:06:52.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:52 smithi045 ceph-mon[20143]: mon.c calling monitor election 2024-08-19T14:06:52.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:52 smithi045 ceph-mon[20143]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T14:06:52.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:52 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:06:52.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:52 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:06:52.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:52 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:06:52.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:52 smithi045 ceph-mon[20143]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-08-19T14:06:52.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:52 smithi045 ceph-mon[20143]: monmap epoch 2 2024-08-19T14:06:52.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:52 smithi045 ceph-mon[20143]: fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 2024-08-19T14:06:52.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:52 smithi045 ceph-mon[20143]: last_changed 2024-08-19T14:06:47.117803+0000 2024-08-19T14:06:52.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:52 smithi045 ceph-mon[20143]: created 2024-08-19T14:05:16.209575+0000 2024-08-19T14:06:52.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:52 smithi045 ceph-mon[20143]: min_mon_release 19 (squid) 2024-08-19T14:06:52.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:52 smithi045 ceph-mon[20143]: election_strategy: 1 2024-08-19T14:06:52.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:52 smithi045 ceph-mon[20143]: 0: [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] mon.a 2024-08-19T14:06:52.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:52 smithi045 ceph-mon[20143]: 1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.c 2024-08-19T14:06:52.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:52 smithi045 ceph-mon[20143]: fsmap 2024-08-19T14:06:52.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:52 smithi045 ceph-mon[20143]: osdmap e4: 0 total, 0 up, 0 in 2024-08-19T14:06:52.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:52 smithi045 ceph-mon[20143]: mgrmap e13: y(active, since 38s) 2024-08-19T14:06:52.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:52 smithi045 ceph-mon[20143]: overall HEALTH_OK 2024-08-19T14:06:52.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:52 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:06:52.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:52 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:06:52.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:52 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:06:52.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:52 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-08-19T14:06:52.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:52 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:06:52.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:52 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:06:52.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:52 smithi031 ceph-mon[16336]: mon.a calling monitor election 2024-08-19T14:06:52.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:52 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:06:52.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:52 smithi031 ceph-mon[16336]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T14:06:52.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:52 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:06:52.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:52 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:06:52.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:52 smithi031 ceph-mon[16336]: mon.c calling monitor election 2024-08-19T14:06:52.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:52 smithi031 ceph-mon[16336]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T14:06:52.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:52 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:06:52.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:52 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:06:52.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:52 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:06:52.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:52 smithi031 ceph-mon[16336]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-08-19T14:06:52.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:52 smithi031 ceph-mon[16336]: monmap epoch 2 2024-08-19T14:06:52.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:52 smithi031 ceph-mon[16336]: fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 2024-08-19T14:06:52.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:52 smithi031 ceph-mon[16336]: last_changed 2024-08-19T14:06:47.117803+0000 2024-08-19T14:06:52.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:52 smithi031 ceph-mon[16336]: created 2024-08-19T14:05:16.209575+0000 2024-08-19T14:06:52.618 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:52 smithi031 ceph-mon[16336]: min_mon_release 19 (squid) 2024-08-19T14:06:52.618 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:52 smithi031 ceph-mon[16336]: election_strategy: 1 2024-08-19T14:06:52.618 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:52 smithi031 ceph-mon[16336]: 0: [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] mon.a 2024-08-19T14:06:52.618 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:52 smithi031 ceph-mon[16336]: 1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.c 2024-08-19T14:06:52.618 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:52 smithi031 ceph-mon[16336]: fsmap 2024-08-19T14:06:52.618 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:52 smithi031 ceph-mon[16336]: osdmap e4: 0 total, 0 up, 0 in 2024-08-19T14:06:52.618 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:52 smithi031 ceph-mon[16336]: mgrmap e13: y(active, since 38s) 2024-08-19T14:06:52.618 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:52 smithi031 ceph-mon[16336]: overall HEALTH_OK 2024-08-19T14:06:52.618 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:52 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:06:52.618 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:52 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:06:52.618 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:52 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:06:52.618 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:52 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-08-19T14:06:52.618 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:52 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:06:52.990 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-08-19T14:06:52.991 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":2,"fsid":"ed7d1c6e-5e33-11ef-bccf-c7b262605968","modified":"2024-08-19T14:06:47.117803Z","created":"2024-08-19T14:05:16.209575Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:3300","nonce":0},{"type":"v1","addr":"172.21.15.31:6789","nonce":0}]},"addr":"172.21.15.31:6789/0","public_addr":"172.21.15.31:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-08-19T14:06:52.991 INFO:teuthology.orchestra.run.smithi045.stderr:dumped monmap epoch 2 2024-08-19T14:06:53.294 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:53 smithi045 ceph-mon[20143]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T14:06:53.294 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:53 smithi045 ceph-mon[20143]: Deploying daemon mon.b on smithi032 2024-08-19T14:06:53.294 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:53 smithi045 ceph-mon[20143]: from='client.? 172.21.15.45:0/1534157212' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-08-19T14:06:53.294 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:53 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:06:53.538 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:53 smithi031 ceph-mon[16336]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T14:06:53.538 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:53 smithi031 ceph-mon[16336]: Deploying daemon mon.b on smithi032 2024-08-19T14:06:53.538 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:53 smithi031 ceph-mon[16336]: from='client.? 172.21.15.45:0/1534157212' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-08-19T14:06:53.538 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:53 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:06:54.589 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-08-19T14:06:54.589 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph mon dump -f json 2024-08-19T14:06:54.851 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:06:55.272 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:06:55 smithi045 ceph-mon[20143]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T14:06:55.517 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-08-19T14:06:55.517 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":2,"fsid":"ed7d1c6e-5e33-11ef-bccf-c7b262605968","modified":"2024-08-19T14:06:47.117803Z","created":"2024-08-19T14:05:16.209575Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:3300","nonce":0},{"type":"v1","addr":"172.21.15.31:6789","nonce":0}]},"addr":"172.21.15.31:6789/0","public_addr":"172.21.15.31:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-08-19T14:06:55.517 INFO:teuthology.orchestra.run.smithi045.stderr:dumped monmap epoch 2 2024-08-19T14:06:55.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:06:55 smithi031 ceph-mon[16336]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T14:06:55.711 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:06:55 smithi032 ceph-mon[20004]: mon.b@-1(synchronizing).mds e1 new map 2024-08-19T14:06:55.711 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:06:55 smithi032 ceph-mon[20004]: mon.b@-1(synchronizing).mds e1 print_map 2024-08-19T14:06:55.711 INFO:journalctl@ceph.mon.b.smithi032.stdout: e1 2024-08-19T14:06:55.711 INFO:journalctl@ceph.mon.b.smithi032.stdout: btime 2024-08-19T14:05:22:841940+0000 2024-08-19T14:06:55.711 INFO:journalctl@ceph.mon.b.smithi032.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-08-19T14:06:55.711 INFO:journalctl@ceph.mon.b.smithi032.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-08-19T14:06:55.711 INFO:journalctl@ceph.mon.b.smithi032.stdout: legacy client fscid: -1 2024-08-19T14:06:55.712 INFO:journalctl@ceph.mon.b.smithi032.stdout: 2024-08-19T14:06:55.712 INFO:journalctl@ceph.mon.b.smithi032.stdout: No filesystems configured 2024-08-19T14:06:55.712 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:06:55 smithi032 ceph-mon[20004]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-08-19T14:06:55.712 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:06:55 smithi032 ceph-mon[20004]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-08-19T14:06:55.712 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:06:55 smithi032 ceph-mon[20004]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-08-19T14:06:55.712 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:06:55 smithi032 ceph-mon[20004]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-08-19T14:06:55.712 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:06:55 smithi032 ceph-mon[20004]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-08-19T14:06:55.712 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:06:55 smithi032 ceph-mon[20004]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-08-19T14:06:55.712 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:06:55 smithi032 ceph-mon[20004]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-08-19T14:06:55.712 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:06:55 smithi032 ceph-mon[20004]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-08-19T14:06:55.712 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:06:55 smithi032 ceph-mon[20004]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-08-19T14:06:55.712 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:06:55 smithi032 ceph-mon[20004]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-08-19T14:06:55.712 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:06:55 smithi032 ceph-mon[20004]: Added host smithi045 2024-08-19T14:06:55.713 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:06:55 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:06:55.713 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:06:55 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:06:55.713 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:06:55 smithi032 ceph-mon[20004]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T14:06:55.713 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:06:55 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:06:55.713 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:06:55 smithi032 ceph-mon[20004]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:06:55.713 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:06:55 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1213544896' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-08-19T14:06:55.713 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:06:55 smithi032 ceph-mon[20004]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T14:06:55.713 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:06:55 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1213544896' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-08-19T14:06:55.713 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:06:55 smithi032 ceph-mon[20004]: osdmap e4: 0 total, 0 up, 0 in 2024-08-19T14:06:55.713 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:06:55 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:06:55.713 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:06:55 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:06:55.713 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:06:55 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:06:55.713 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:06:55 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:06:55.714 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:06:55 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi045", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:06:55.714 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:06:55 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:06:55.714 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:06:55 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:06:55.714 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:06:55 smithi032 ceph-mon[20004]: Updating smithi045:/etc/ceph/ceph.conf 2024-08-19T14:06:55.714 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:06:55 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:06:55.714 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:06:55 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:06:55.714 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:06:55 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:06:55.714 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:06:55 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:06:55.714 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:06:55 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-08-19T14:06:55.714 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:06:55 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:06:55.714 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:06:55 smithi032 ceph-mon[20004]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T14:06:55.714 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:06:55 smithi032 ceph-mon[20004]: Updating smithi045:/var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-19T14:06:55.714 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:06:55 smithi032 ceph-mon[20004]: Updating smithi045:/etc/ceph/ceph.client.admin.keyring 2024-08-19T14:06:55.715 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:06:55 smithi032 ceph-mon[20004]: Updating smithi045:/var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/config/ceph.client.admin.keyring 2024-08-19T14:06:55.715 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:06:55 smithi032 ceph-mon[20004]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi031:172.21.15.31=a;smithi032:172.21.15.32=b;smithi045:172.21.15.45=c", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T14:06:55.715 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:06:55 smithi032 ceph-mon[20004]: Saving service mon spec with placement smithi031:172.21.15.31=a;smithi032:172.21.15.32=b;smithi045:172.21.15.45=c;count:3 2024-08-19T14:06:55.715 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:06:55 smithi032 ceph-mon[20004]: Deploying daemon mon.c on smithi045 2024-08-19T14:06:55.715 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:06:55 smithi032 ceph-mon[20004]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T14:06:55.715 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:06:55 smithi032 ceph-mon[20004]: from='client.? 172.21.15.45:0/1173249320' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-08-19T14:06:55.715 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:06:55 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:06:55.715 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:06:55 smithi032 ceph-mon[20004]: mon.a calling monitor election 2024-08-19T14:06:55.715 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:06:55 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:06:55.715 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:06:55 smithi032 ceph-mon[20004]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T14:06:55.715 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:06:55 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:06:55.715 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:06:55 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:06:55.715 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:06:55 smithi032 ceph-mon[20004]: mon.c calling monitor election 2024-08-19T14:06:55.715 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:06:55 smithi032 ceph-mon[20004]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T14:06:55.716 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:06:55 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:06:55.716 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:06:55 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:06:55.716 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:06:55 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:06:55.716 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:06:55 smithi032 ceph-mon[20004]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-08-19T14:06:55.716 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:06:55 smithi032 ceph-mon[20004]: monmap epoch 2 2024-08-19T14:06:55.716 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:06:55 smithi032 ceph-mon[20004]: fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 2024-08-19T14:06:55.716 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:06:55 smithi032 ceph-mon[20004]: last_changed 2024-08-19T14:06:47.117803+0000 2024-08-19T14:06:55.716 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:06:55 smithi032 ceph-mon[20004]: created 2024-08-19T14:05:16.209575+0000 2024-08-19T14:06:55.716 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:06:55 smithi032 ceph-mon[20004]: min_mon_release 19 (squid) 2024-08-19T14:06:55.717 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:06:55 smithi032 ceph-mon[20004]: election_strategy: 1 2024-08-19T14:06:55.717 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:06:55 smithi032 ceph-mon[20004]: 0: [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] mon.a 2024-08-19T14:06:55.717 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:06:55 smithi032 ceph-mon[20004]: 1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.c 2024-08-19T14:06:55.717 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:06:55 smithi032 ceph-mon[20004]: fsmap 2024-08-19T14:06:55.717 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:06:55 smithi032 ceph-mon[20004]: osdmap e4: 0 total, 0 up, 0 in 2024-08-19T14:06:55.717 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:06:55 smithi032 ceph-mon[20004]: mgrmap e13: y(active, since 38s) 2024-08-19T14:06:55.717 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:06:55 smithi032 ceph-mon[20004]: overall HEALTH_OK 2024-08-19T14:06:55.717 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:06:55 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:06:55.717 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:06:55 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:06:55.717 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:06:55 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:06:55.717 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:06:55 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-08-19T14:06:55.717 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:06:55 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:06:55.717 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:06:55 smithi032 ceph-mon[20004]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T14:06:55.718 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:06:55 smithi032 ceph-mon[20004]: Deploying daemon mon.b on smithi032 2024-08-19T14:06:55.718 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:06:55 smithi032 ceph-mon[20004]: from='client.? 172.21.15.45:0/1534157212' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-08-19T14:06:55.718 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:06:55 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:06:55.718 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:06:55 smithi032 ceph-mon[20004]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T14:06:55.718 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:06:55 smithi032 ceph-mon[20004]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-08-19T14:06:56.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:06:55 smithi032 bash[19985]: 066967b48013e975d49dd757239af1a4e46f30fff847c4c9394bbd44a78939ff 2024-08-19T14:06:56.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:06:55 smithi032 systemd[1]: Started Ceph mon.b for ed7d1c6e-5e33-11ef-bccf-c7b262605968. 2024-08-19T14:06:57.105 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-08-19T14:06:57.105 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph mon dump -f json 2024-08-19T14:06:57.377 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:07:01.005 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-08-19T14:07:01.006 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":3,"fsid":"ed7d1c6e-5e33-11ef-bccf-c7b262605968","modified":"2024-08-19T14:06:55.715434Z","created":"2024-08-19T14:05:16.209575Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:3300","nonce":0},{"type":"v1","addr":"172.21.15.31:6789","nonce":0}]},"addr":"172.21.15.31:6789/0","public_addr":"172.21.15.31:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:3300","nonce":0},{"type":"v1","addr":"172.21.15.32:6789","nonce":0}]},"addr":"172.21.15.32:6789/0","public_addr":"172.21.15.32:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-08-19T14:07:01.006 INFO:teuthology.orchestra.run.smithi045.stderr:dumped monmap epoch 3 2024-08-19T14:07:01.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:00 smithi045 ceph-mon[20143]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T14:07:01.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:00 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:07:01.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:00 smithi045 ceph-mon[20143]: mon.c calling monitor election 2024-08-19T14:07:01.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:00 smithi045 ceph-mon[20143]: mon.a calling monitor election 2024-08-19T14:07:01.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:00 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:07:01.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:00 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:07:01.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:00 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:07:01.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:00 smithi045 ceph-mon[20143]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T14:07:01.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:00 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:07:01.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:00 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:07:01.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:00 smithi045 ceph-mon[20143]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T14:07:01.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:00 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:07:01.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:00 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:07:01.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:00 smithi045 ceph-mon[20143]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-08-19T14:07:01.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:00 smithi045 ceph-mon[20143]: monmap epoch 3 2024-08-19T14:07:01.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:00 smithi045 ceph-mon[20143]: fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 2024-08-19T14:07:01.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:00 smithi045 ceph-mon[20143]: last_changed 2024-08-19T14:06:55.715434+0000 2024-08-19T14:07:01.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:00 smithi045 ceph-mon[20143]: created 2024-08-19T14:05:16.209575+0000 2024-08-19T14:07:01.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:00 smithi045 ceph-mon[20143]: min_mon_release 19 (squid) 2024-08-19T14:07:01.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:00 smithi045 ceph-mon[20143]: election_strategy: 1 2024-08-19T14:07:01.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:00 smithi045 ceph-mon[20143]: 0: [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] mon.a 2024-08-19T14:07:01.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:00 smithi045 ceph-mon[20143]: 1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.c 2024-08-19T14:07:01.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:00 smithi045 ceph-mon[20143]: 2: [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] mon.b 2024-08-19T14:07:01.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:00 smithi045 ceph-mon[20143]: fsmap 2024-08-19T14:07:01.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:00 smithi045 ceph-mon[20143]: osdmap e4: 0 total, 0 up, 0 in 2024-08-19T14:07:01.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:00 smithi045 ceph-mon[20143]: mgrmap e13: y(active, since 47s) 2024-08-19T14:07:01.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:00 smithi045 ceph-mon[20143]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-08-19T14:07:01.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:00 smithi045 ceph-mon[20143]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-08-19T14:07:01.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:00 smithi045 ceph-mon[20143]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-08-19T14:07:01.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:00 smithi045 ceph-mon[20143]: mon.b (rank 2) addr [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] is down (out of quorum) 2024-08-19T14:07:01.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:00 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:01.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:00 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:01.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:00 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:01.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:00 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:01.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:00 smithi031 ceph-mon[16336]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T14:07:01.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:00 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:07:01.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:00 smithi031 ceph-mon[16336]: mon.c calling monitor election 2024-08-19T14:07:01.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:00 smithi031 ceph-mon[16336]: mon.a calling monitor election 2024-08-19T14:07:01.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:00 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:07:01.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:00 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:07:01.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:00 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:07:01.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:00 smithi031 ceph-mon[16336]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T14:07:01.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:00 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:07:01.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:00 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:07:01.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:00 smithi031 ceph-mon[16336]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T14:07:01.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:00 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:07:01.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:00 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:07:01.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:00 smithi031 ceph-mon[16336]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-08-19T14:07:01.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:00 smithi031 ceph-mon[16336]: monmap epoch 3 2024-08-19T14:07:01.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:00 smithi031 ceph-mon[16336]: fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 2024-08-19T14:07:01.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:00 smithi031 ceph-mon[16336]: last_changed 2024-08-19T14:06:55.715434+0000 2024-08-19T14:07:01.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:00 smithi031 ceph-mon[16336]: created 2024-08-19T14:05:16.209575+0000 2024-08-19T14:07:01.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:00 smithi031 ceph-mon[16336]: min_mon_release 19 (squid) 2024-08-19T14:07:01.118 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:00 smithi031 ceph-mon[16336]: election_strategy: 1 2024-08-19T14:07:01.118 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:00 smithi031 ceph-mon[16336]: 0: [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] mon.a 2024-08-19T14:07:01.118 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:00 smithi031 ceph-mon[16336]: 1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.c 2024-08-19T14:07:01.118 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:00 smithi031 ceph-mon[16336]: 2: [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] mon.b 2024-08-19T14:07:01.118 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:00 smithi031 ceph-mon[16336]: fsmap 2024-08-19T14:07:01.118 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:00 smithi031 ceph-mon[16336]: osdmap e4: 0 total, 0 up, 0 in 2024-08-19T14:07:01.118 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:00 smithi031 ceph-mon[16336]: mgrmap e13: y(active, since 47s) 2024-08-19T14:07:01.118 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:00 smithi031 ceph-mon[16336]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-08-19T14:07:01.118 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:00 smithi031 ceph-mon[16336]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-08-19T14:07:01.118 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:00 smithi031 ceph-mon[16336]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-08-19T14:07:01.118 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:00 smithi031 ceph-mon[16336]: mon.b (rank 2) addr [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] is down (out of quorum) 2024-08-19T14:07:01.118 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:00 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:01.118 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:00 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:01.118 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:00 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:01.119 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:00 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:01.568 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-08-19T14:07:01.568 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph config generate-minimal-conf 2024-08-19T14:07:01.832 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:07:01.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:01 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:07:01.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:01 smithi031 ceph-mon[16336]: from='client.? 172.21.15.45:0/1796772620' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-08-19T14:07:01.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:01 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:07:01.967 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:01 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:07:01.968 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:01 smithi045 ceph-mon[20143]: from='client.? 172.21.15.45:0/1796772620' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-08-19T14:07:01.968 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:01 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:07:02.462 INFO:teuthology.orchestra.run.smithi031.stdout:# minimal ceph.conf for ed7d1c6e-5e33-11ef-bccf-c7b262605968 2024-08-19T14:07:02.463 INFO:teuthology.orchestra.run.smithi031.stdout:[global] 2024-08-19T14:07:02.463 INFO:teuthology.orchestra.run.smithi031.stdout: fsid = ed7d1c6e-5e33-11ef-bccf-c7b262605968 2024-08-19T14:07:02.463 INFO:teuthology.orchestra.run.smithi031.stdout: mon_host = [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-08-19T14:07:03.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:02 smithi032 ceph-mon[20004]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T14:07:03.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:02 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:07:03.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:02 smithi032 ceph-mon[20004]: mon.c calling monitor election 2024-08-19T14:07:03.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:02 smithi032 ceph-mon[20004]: mon.a calling monitor election 2024-08-19T14:07:03.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:02 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:07:03.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:02 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:07:03.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:02 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:07:03.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:02 smithi032 ceph-mon[20004]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T14:07:03.009 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:02 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:07:03.009 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:02 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:07:03.009 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:02 smithi032 ceph-mon[20004]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T14:07:03.009 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:02 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:07:03.009 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:02 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:07:03.009 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:02 smithi032 ceph-mon[20004]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-08-19T14:07:03.009 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:02 smithi032 ceph-mon[20004]: monmap epoch 3 2024-08-19T14:07:03.009 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:02 smithi032 ceph-mon[20004]: fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 2024-08-19T14:07:03.009 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:02 smithi032 ceph-mon[20004]: last_changed 2024-08-19T14:06:55.715434+0000 2024-08-19T14:07:03.009 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:02 smithi032 ceph-mon[20004]: created 2024-08-19T14:05:16.209575+0000 2024-08-19T14:07:03.009 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:02 smithi032 ceph-mon[20004]: min_mon_release 19 (squid) 2024-08-19T14:07:03.010 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:02 smithi032 ceph-mon[20004]: election_strategy: 1 2024-08-19T14:07:03.010 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:02 smithi032 ceph-mon[20004]: 0: [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] mon.a 2024-08-19T14:07:03.010 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:02 smithi032 ceph-mon[20004]: 1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.c 2024-08-19T14:07:03.010 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:02 smithi032 ceph-mon[20004]: 2: [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] mon.b 2024-08-19T14:07:03.010 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:02 smithi032 ceph-mon[20004]: fsmap 2024-08-19T14:07:03.010 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:02 smithi032 ceph-mon[20004]: osdmap e4: 0 total, 0 up, 0 in 2024-08-19T14:07:03.010 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:02 smithi032 ceph-mon[20004]: mgrmap e13: y(active, since 47s) 2024-08-19T14:07:03.010 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:02 smithi032 ceph-mon[20004]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-08-19T14:07:03.010 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:02 smithi032 ceph-mon[20004]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-08-19T14:07:03.010 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:02 smithi032 ceph-mon[20004]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-08-19T14:07:03.011 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:02 smithi032 ceph-mon[20004]: mon.b (rank 2) addr [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] is down (out of quorum) 2024-08-19T14:07:03.011 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:02 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:03.011 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:02 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:03.011 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:02 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:03.011 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:02 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:03.011 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:02 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:07:03.011 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:02 smithi032 ceph-mon[20004]: from='client.? 172.21.15.45:0/1796772620' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-08-19T14:07:03.012 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:02 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:07:03.148 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-08-19T14:07:03.149 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-08-19T14:07:03.149 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/ceph/ceph.conf 2024-08-19T14:07:03.174 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-08-19T14:07:03.174 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-08-19T14:07:03.243 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-19T14:07:03.243 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/etc/ceph/ceph.conf 2024-08-19T14:07:03.270 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-19T14:07:03.270 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-08-19T14:07:03.335 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-08-19T14:07:03.335 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/etc/ceph/ceph.conf 2024-08-19T14:07:03.363 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-08-19T14:07:03.364 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-08-19T14:07:03.429 INFO:tasks.cephadm:Adding mgr.y on smithi031 2024-08-19T14:07:03.429 INFO:tasks.cephadm:Adding mgr.x on smithi032 2024-08-19T14:07:03.429 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph orch apply mgr '2;smithi031=y;smithi032=x' 2024-08-19T14:07:03.713 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:07:03.717 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:03 smithi031 ceph-mon[16336]: mon.b calling monitor election 2024-08-19T14:07:03.718 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:03 smithi031 ceph-mon[16336]: Updating smithi045:/var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-19T14:07:03.718 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:03 smithi031 ceph-mon[16336]: Updating smithi032:/var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-19T14:07:03.718 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:03 smithi031 ceph-mon[16336]: mon.b calling monitor election 2024-08-19T14:07:03.718 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:03 smithi031 ceph-mon[16336]: mon.c calling monitor election 2024-08-19T14:07:03.718 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:03 smithi031 ceph-mon[16336]: mon.a calling monitor election 2024-08-19T14:07:03.718 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:03 smithi031 ceph-mon[16336]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-08-19T14:07:03.718 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:03 smithi031 ceph-mon[16336]: monmap epoch 3 2024-08-19T14:07:03.718 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:03 smithi031 ceph-mon[16336]: fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 2024-08-19T14:07:03.718 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:03 smithi031 ceph-mon[16336]: last_changed 2024-08-19T14:06:55.715434+0000 2024-08-19T14:07:03.718 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:03 smithi031 ceph-mon[16336]: created 2024-08-19T14:05:16.209575+0000 2024-08-19T14:07:03.718 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:03 smithi031 ceph-mon[16336]: min_mon_release 19 (squid) 2024-08-19T14:07:03.718 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:03 smithi031 ceph-mon[16336]: election_strategy: 1 2024-08-19T14:07:03.718 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:03 smithi031 ceph-mon[16336]: 0: [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] mon.a 2024-08-19T14:07:03.719 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:03 smithi031 ceph-mon[16336]: 1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.c 2024-08-19T14:07:03.719 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:03 smithi031 ceph-mon[16336]: 2: [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] mon.b 2024-08-19T14:07:03.719 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:03 smithi031 ceph-mon[16336]: fsmap 2024-08-19T14:07:03.719 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:03 smithi031 ceph-mon[16336]: osdmap e4: 0 total, 0 up, 0 in 2024-08-19T14:07:03.719 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:03 smithi031 ceph-mon[16336]: mgrmap e13: y(active, since 49s) 2024-08-19T14:07:03.719 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:03 smithi031 ceph-mon[16336]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-08-19T14:07:03.719 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:03 smithi031 ceph-mon[16336]: Cluster is now healthy 2024-08-19T14:07:03.719 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:03 smithi031 ceph-mon[16336]: overall HEALTH_OK 2024-08-19T14:07:03.719 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:03 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:03.719 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:03 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:03.719 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:03 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:03.719 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:03 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:03.719 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:03 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:07:03.720 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:03 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:07:03.720 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:03 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:07:03.720 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:03 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:03.720 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:03 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:03.720 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:03 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:03.720 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:03 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:03.720 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:03 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:03.720 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:03 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:03.720 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:03 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:03.720 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:03 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:03.720 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:03 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-08-19T14:07:03.720 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:03 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-08-19T14:07:03.720 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:03 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:07:03.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:03 smithi032 ceph-mon[20004]: mon.b calling monitor election 2024-08-19T14:07:03.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:03 smithi032 ceph-mon[20004]: Updating smithi045:/var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-19T14:07:03.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:03 smithi032 ceph-mon[20004]: Updating smithi032:/var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-19T14:07:03.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:03 smithi032 ceph-mon[20004]: mon.b calling monitor election 2024-08-19T14:07:03.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:03 smithi032 ceph-mon[20004]: mon.c calling monitor election 2024-08-19T14:07:03.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:03 smithi032 ceph-mon[20004]: mon.a calling monitor election 2024-08-19T14:07:03.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:03 smithi032 ceph-mon[20004]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-08-19T14:07:03.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:03 smithi032 ceph-mon[20004]: monmap epoch 3 2024-08-19T14:07:03.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:03 smithi032 ceph-mon[20004]: fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 2024-08-19T14:07:03.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:03 smithi032 ceph-mon[20004]: last_changed 2024-08-19T14:06:55.715434+0000 2024-08-19T14:07:03.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:03 smithi032 ceph-mon[20004]: created 2024-08-19T14:05:16.209575+0000 2024-08-19T14:07:03.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:03 smithi032 ceph-mon[20004]: min_mon_release 19 (squid) 2024-08-19T14:07:03.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:03 smithi032 ceph-mon[20004]: election_strategy: 1 2024-08-19T14:07:03.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:03 smithi032 ceph-mon[20004]: 0: [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] mon.a 2024-08-19T14:07:03.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:03 smithi032 ceph-mon[20004]: 1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.c 2024-08-19T14:07:03.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:03 smithi032 ceph-mon[20004]: 2: [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] mon.b 2024-08-19T14:07:03.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:03 smithi032 ceph-mon[20004]: fsmap 2024-08-19T14:07:03.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:03 smithi032 ceph-mon[20004]: osdmap e4: 0 total, 0 up, 0 in 2024-08-19T14:07:03.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:03 smithi032 ceph-mon[20004]: mgrmap e13: y(active, since 49s) 2024-08-19T14:07:03.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:03 smithi032 ceph-mon[20004]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-08-19T14:07:03.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:03 smithi032 ceph-mon[20004]: Cluster is now healthy 2024-08-19T14:07:03.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:03 smithi032 ceph-mon[20004]: overall HEALTH_OK 2024-08-19T14:07:03.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:03 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:03.760 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:03 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:03.760 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:03 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:03.760 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:03 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:03.760 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:03 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:07:03.760 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:03 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:07:03.760 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:03 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:07:03.760 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:03 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:03.760 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:03 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:03.760 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:03 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:03.760 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:03 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:03.760 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:03 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:03.760 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:03 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:03.760 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:03 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:03.760 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:03 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:03.760 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:03 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-08-19T14:07:03.761 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:03 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-08-19T14:07:03.761 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:03 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:07:03.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:03 smithi045 ceph-mon[20143]: mon.b calling monitor election 2024-08-19T14:07:03.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:03 smithi045 ceph-mon[20143]: Updating smithi045:/var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-19T14:07:03.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:03 smithi045 ceph-mon[20143]: Updating smithi032:/var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-19T14:07:03.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:03 smithi045 ceph-mon[20143]: mon.b calling monitor election 2024-08-19T14:07:03.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:03 smithi045 ceph-mon[20143]: mon.c calling monitor election 2024-08-19T14:07:03.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:03 smithi045 ceph-mon[20143]: mon.a calling monitor election 2024-08-19T14:07:03.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:03 smithi045 ceph-mon[20143]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-08-19T14:07:03.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:03 smithi045 ceph-mon[20143]: monmap epoch 3 2024-08-19T14:07:03.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:03 smithi045 ceph-mon[20143]: fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 2024-08-19T14:07:03.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:03 smithi045 ceph-mon[20143]: last_changed 2024-08-19T14:06:55.715434+0000 2024-08-19T14:07:03.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:03 smithi045 ceph-mon[20143]: created 2024-08-19T14:05:16.209575+0000 2024-08-19T14:07:03.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:03 smithi045 ceph-mon[20143]: min_mon_release 19 (squid) 2024-08-19T14:07:03.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:03 smithi045 ceph-mon[20143]: election_strategy: 1 2024-08-19T14:07:03.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:03 smithi045 ceph-mon[20143]: 0: [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] mon.a 2024-08-19T14:07:03.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:03 smithi045 ceph-mon[20143]: 1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.c 2024-08-19T14:07:03.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:03 smithi045 ceph-mon[20143]: 2: [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] mon.b 2024-08-19T14:07:03.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:03 smithi045 ceph-mon[20143]: fsmap 2024-08-19T14:07:03.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:03 smithi045 ceph-mon[20143]: osdmap e4: 0 total, 0 up, 0 in 2024-08-19T14:07:03.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:03 smithi045 ceph-mon[20143]: mgrmap e13: y(active, since 49s) 2024-08-19T14:07:03.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:03 smithi045 ceph-mon[20143]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-08-19T14:07:03.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:03 smithi045 ceph-mon[20143]: Cluster is now healthy 2024-08-19T14:07:03.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:03 smithi045 ceph-mon[20143]: overall HEALTH_OK 2024-08-19T14:07:03.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:03 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:03.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:03 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:03.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:03 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:03.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:03 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:03.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:03 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:07:03.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:03 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:07:03.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:03 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:07:03.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:03 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:03.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:03 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:03.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:03 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:03.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:03 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:03.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:03 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:03.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:03 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:03.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:03 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:03.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:03 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:03.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:03 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-08-19T14:07:03.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:03 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-08-19T14:07:03.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:03 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:07:04.116 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:07:03 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: 2024-08-19T14:07:03.715+0000 7ff48d3f5640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-08-19T14:07:04.313 INFO:teuthology.orchestra.run.smithi045.stdout:Scheduled mgr update... 2024-08-19T14:07:04.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:04 smithi032 ceph-mon[20004]: Reconfiguring mon.a (unknown last config time)... 2024-08-19T14:07:04.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:04 smithi032 ceph-mon[20004]: Reconfiguring daemon mon.a on smithi031 2024-08-19T14:07:04.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:04 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:04.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:04 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:04.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:04 smithi045 ceph-mon[20143]: Reconfiguring mon.a (unknown last config time)... 2024-08-19T14:07:04.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:04 smithi045 ceph-mon[20143]: Reconfiguring daemon mon.a on smithi031 2024-08-19T14:07:04.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:04 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:04.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:04 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:04.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:04 smithi031 ceph-mon[16336]: Reconfiguring mon.a (unknown last config time)... 2024-08-19T14:07:04.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:04 smithi031 ceph-mon[16336]: Reconfiguring daemon mon.a on smithi031 2024-08-19T14:07:04.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:04 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:04.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:04 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:04.877 DEBUG:teuthology.orchestra.run.smithi032:mgr.x> sudo journalctl -f -n 0 -u ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968@mgr.x.service 2024-08-19T14:07:04.880 INFO:tasks.cephadm:Deploying OSDs... 2024-08-19T14:07:04.880 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-08-19T14:07:04.880 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/scratch_devs of=/dev/stdout 2024-08-19T14:07:04.897 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-08-19T14:07:04.897 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_1 2024-08-19T14:07:04.952 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-08-19T14:07:04.952 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-19T14:07:04.952 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 5h/5d Inode: 669 Links: 1 2024-08-19T14:07:04.952 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-19T14:07:04.953 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:device_t:s0 2024-08-19T14:07:04.953 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2024-08-19 14:06:24.161136982 +0000 2024-08-19T14:07:04.953 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2024-08-19 14:02:17.480697498 +0000 2024-08-19T14:07:04.953 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2024-08-19 14:02:17.480697498 +0000 2024-08-19T14:07:04.953 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: 2024-08-19 14:02:17.480697498 +0000 2024-08-19T14:07:04.953 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-08-19T14:07:05.018 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2024-08-19T14:07:05.018 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2024-08-19T14:07:05.018 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.000195956 s, 2.6 MB/s 2024-08-19T14:07:05.019 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-08-19T14:07:05.078 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_2 2024-08-19T14:07:05.135 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-08-19T14:07:05.135 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-19T14:07:05.135 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 5h/5d Inode: 690 Links: 1 2024-08-19T14:07:05.135 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-19T14:07:05.135 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:device_t:s0 2024-08-19T14:07:05.135 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2024-08-19 14:06:24.198136392 +0000 2024-08-19T14:07:05.135 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2024-08-19 14:02:17.810695440 +0000 2024-08-19T14:07:05.136 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2024-08-19 14:02:17.810695440 +0000 2024-08-19T14:07:05.136 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: 2024-08-19 14:02:17.810695440 +0000 2024-08-19T14:07:05.136 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-08-19T14:07:05.199 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2024-08-19T14:07:05.199 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2024-08-19T14:07:05.199 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.000192831 s, 2.7 MB/s 2024-08-19T14:07:05.200 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-08-19T14:07:05.256 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_3 2024-08-19T14:07:05.311 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-08-19T14:07:05.311 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-19T14:07:05.311 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 5h/5d Inode: 707 Links: 1 2024-08-19T14:07:05.311 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-19T14:07:05.311 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:device_t:s0 2024-08-19T14:07:05.312 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2024-08-19 14:06:24.232135848 +0000 2024-08-19T14:07:05.312 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2024-08-19 14:02:18.142693370 +0000 2024-08-19T14:07:05.312 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2024-08-19 14:02:18.142693370 +0000 2024-08-19T14:07:05.312 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: 2024-08-19 14:02:18.142693370 +0000 2024-08-19T14:07:05.312 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-08-19T14:07:05.374 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2024-08-19T14:07:05.374 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2024-08-19T14:07:05.374 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.000186686 s, 2.7 MB/s 2024-08-19T14:07:05.375 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-08-19T14:07:05.431 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_4 2024-08-19T14:07:05.487 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-08-19T14:07:05.487 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-19T14:07:05.487 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 5h/5d Inode: 726 Links: 1 2024-08-19T14:07:05.487 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-19T14:07:05.487 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:device_t:s0 2024-08-19T14:07:05.487 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2024-08-19 14:06:24.266135305 +0000 2024-08-19T14:07:05.487 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2024-08-19 14:02:18.462691375 +0000 2024-08-19T14:07:05.487 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2024-08-19 14:02:18.462691375 +0000 2024-08-19T14:07:05.487 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: 2024-08-19 14:02:18.462691375 +0000 2024-08-19T14:07:05.488 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-08-19T14:07:05.550 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2024-08-19T14:07:05.551 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2024-08-19T14:07:05.551 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.00020793 s, 2.5 MB/s 2024-08-19T14:07:05.552 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-08-19T14:07:05.608 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-19T14:07:05.608 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/scratch_devs of=/dev/stdout 2024-08-19T14:07:05.628 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-08-19T14:07:05.628 DEBUG:teuthology.orchestra.run.smithi032:> stat /dev/vg_nvme/lv_1 2024-08-19T14:07:05.687 INFO:teuthology.orchestra.run.smithi032.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-08-19T14:07:05.687 INFO:teuthology.orchestra.run.smithi032.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-19T14:07:05.687 INFO:teuthology.orchestra.run.smithi032.stdout:Device: 5h/5d Inode: 670 Links: 1 2024-08-19T14:07:05.687 INFO:teuthology.orchestra.run.smithi032.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-19T14:07:05.687 INFO:teuthology.orchestra.run.smithi032.stdout:Context: system_u:object_r:device_t:s0 2024-08-19T14:07:05.687 INFO:teuthology.orchestra.run.smithi032.stdout:Access: 2024-08-19 14:06:55.166727033 +0000 2024-08-19T14:07:05.687 INFO:teuthology.orchestra.run.smithi032.stdout:Modify: 2024-08-19 14:02:17.963575981 +0000 2024-08-19T14:07:05.687 INFO:teuthology.orchestra.run.smithi032.stdout:Change: 2024-08-19 14:02:17.963575981 +0000 2024-08-19T14:07:05.687 INFO:teuthology.orchestra.run.smithi032.stdout: Birth: 2024-08-19 14:02:17.963575981 +0000 2024-08-19T14:07:05.688 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-08-19T14:07:05.754 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records in 2024-08-19T14:07:05.754 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records out 2024-08-19T14:07:05.754 INFO:teuthology.orchestra.run.smithi032.stderr:512 bytes copied, 0.000105864 s, 4.8 MB/s 2024-08-19T14:07:05.755 DEBUG:teuthology.orchestra.run.smithi032:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-08-19T14:07:05.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:05 smithi032 ceph-mon[20004]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T14:07:05.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:05 smithi032 ceph-mon[20004]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi031=y;smithi032=x", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T14:07:05.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:05 smithi032 ceph-mon[20004]: Saving service mgr spec with placement smithi031=y;smithi032=x;count:2 2024-08-19T14:07:05.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:05 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:05.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:05 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:05.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:05 smithi032 ceph-mon[20004]: Reconfiguring mgr.y (unknown last config time)... 2024-08-19T14:07:05.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:05 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-08-19T14:07:05.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:05 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-08-19T14:07:05.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:05 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:07:05.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:05 smithi032 ceph-mon[20004]: Reconfiguring daemon mgr.y on smithi031 2024-08-19T14:07:05.775 DEBUG:teuthology.orchestra.run.smithi032:> stat /dev/vg_nvme/lv_2 2024-08-19T14:07:05.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:05 smithi045 ceph-mon[20143]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T14:07:05.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:05 smithi045 ceph-mon[20143]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi031=y;smithi032=x", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T14:07:05.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:05 smithi045 ceph-mon[20143]: Saving service mgr spec with placement smithi031=y;smithi032=x;count:2 2024-08-19T14:07:05.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:05 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:05.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:05 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:05.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:05 smithi045 ceph-mon[20143]: Reconfiguring mgr.y (unknown last config time)... 2024-08-19T14:07:05.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:05 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-08-19T14:07:05.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:05 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-08-19T14:07:05.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:05 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:07:05.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:05 smithi045 ceph-mon[20143]: Reconfiguring daemon mgr.y on smithi031 2024-08-19T14:07:05.832 INFO:teuthology.orchestra.run.smithi032.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-08-19T14:07:05.832 INFO:teuthology.orchestra.run.smithi032.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-19T14:07:05.833 INFO:teuthology.orchestra.run.smithi032.stdout:Device: 5h/5d Inode: 688 Links: 1 2024-08-19T14:07:05.833 INFO:teuthology.orchestra.run.smithi032.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-19T14:07:05.833 INFO:teuthology.orchestra.run.smithi032.stdout:Context: system_u:object_r:device_t:s0 2024-08-19T14:07:05.833 INFO:teuthology.orchestra.run.smithi032.stdout:Access: 2024-08-19 14:06:55.166727033 +0000 2024-08-19T14:07:05.833 INFO:teuthology.orchestra.run.smithi032.stdout:Modify: 2024-08-19 14:02:18.296572905 +0000 2024-08-19T14:07:05.833 INFO:teuthology.orchestra.run.smithi032.stdout:Change: 2024-08-19 14:02:18.296572905 +0000 2024-08-19T14:07:05.833 INFO:teuthology.orchestra.run.smithi032.stdout: Birth: 2024-08-19 14:02:18.296572905 +0000 2024-08-19T14:07:05.833 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-08-19T14:07:05.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:05 smithi031 ceph-mon[16336]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T14:07:05.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:05 smithi031 ceph-mon[16336]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi031=y;smithi032=x", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T14:07:05.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:05 smithi031 ceph-mon[16336]: Saving service mgr spec with placement smithi031=y;smithi032=x;count:2 2024-08-19T14:07:05.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:05 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:05.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:05 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:05.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:05 smithi031 ceph-mon[16336]: Reconfiguring mgr.y (unknown last config time)... 2024-08-19T14:07:05.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:05 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-08-19T14:07:05.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:05 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-08-19T14:07:05.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:05 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:07:05.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:05 smithi031 ceph-mon[16336]: Reconfiguring daemon mgr.y on smithi031 2024-08-19T14:07:05.898 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records in 2024-08-19T14:07:05.898 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records out 2024-08-19T14:07:05.898 INFO:teuthology.orchestra.run.smithi032.stderr:512 bytes copied, 0.000114111 s, 4.5 MB/s 2024-08-19T14:07:05.900 DEBUG:teuthology.orchestra.run.smithi032:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-08-19T14:07:05.957 DEBUG:teuthology.orchestra.run.smithi032:> stat /dev/vg_nvme/lv_3 2024-08-19T14:07:06.014 INFO:teuthology.orchestra.run.smithi032.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-08-19T14:07:06.015 INFO:teuthology.orchestra.run.smithi032.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-19T14:07:06.015 INFO:teuthology.orchestra.run.smithi032.stdout:Device: 5h/5d Inode: 708 Links: 1 2024-08-19T14:07:06.015 INFO:teuthology.orchestra.run.smithi032.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-19T14:07:06.015 INFO:teuthology.orchestra.run.smithi032.stdout:Context: system_u:object_r:device_t:s0 2024-08-19T14:07:06.015 INFO:teuthology.orchestra.run.smithi032.stdout:Access: 2024-08-19 14:06:55.167727014 +0000 2024-08-19T14:07:06.015 INFO:teuthology.orchestra.run.smithi032.stdout:Modify: 2024-08-19 14:02:18.638569745 +0000 2024-08-19T14:07:06.015 INFO:teuthology.orchestra.run.smithi032.stdout:Change: 2024-08-19 14:02:18.638569745 +0000 2024-08-19T14:07:06.015 INFO:teuthology.orchestra.run.smithi032.stdout: Birth: 2024-08-19 14:02:18.638569745 +0000 2024-08-19T14:07:06.015 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-08-19T14:07:06.079 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records in 2024-08-19T14:07:06.079 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records out 2024-08-19T14:07:06.079 INFO:teuthology.orchestra.run.smithi032.stderr:512 bytes copied, 0.000106748 s, 4.8 MB/s 2024-08-19T14:07:06.081 DEBUG:teuthology.orchestra.run.smithi032:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-08-19T14:07:06.136 DEBUG:teuthology.orchestra.run.smithi032:> stat /dev/vg_nvme/lv_4 2024-08-19T14:07:06.192 INFO:teuthology.orchestra.run.smithi032.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-08-19T14:07:06.192 INFO:teuthology.orchestra.run.smithi032.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-19T14:07:06.192 INFO:teuthology.orchestra.run.smithi032.stdout:Device: 5h/5d Inode: 729 Links: 1 2024-08-19T14:07:06.192 INFO:teuthology.orchestra.run.smithi032.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-19T14:07:06.192 INFO:teuthology.orchestra.run.smithi032.stdout:Context: system_u:object_r:device_t:s0 2024-08-19T14:07:06.192 INFO:teuthology.orchestra.run.smithi032.stdout:Access: 2024-08-19 14:06:55.167727014 +0000 2024-08-19T14:07:06.192 INFO:teuthology.orchestra.run.smithi032.stdout:Modify: 2024-08-19 14:02:18.957566798 +0000 2024-08-19T14:07:06.192 INFO:teuthology.orchestra.run.smithi032.stdout:Change: 2024-08-19 14:02:18.957566798 +0000 2024-08-19T14:07:06.193 INFO:teuthology.orchestra.run.smithi032.stdout: Birth: 2024-08-19 14:02:18.957566798 +0000 2024-08-19T14:07:06.193 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-08-19T14:07:06.257 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records in 2024-08-19T14:07:06.257 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records out 2024-08-19T14:07:06.257 INFO:teuthology.orchestra.run.smithi032.stderr:512 bytes copied, 9.8877e-05 s, 5.2 MB/s 2024-08-19T14:07:06.258 DEBUG:teuthology.orchestra.run.smithi032:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-08-19T14:07:06.314 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-08-19T14:07:06.314 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/scratch_devs of=/dev/stdout 2024-08-19T14:07:06.334 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-08-19T14:07:06.334 DEBUG:teuthology.orchestra.run.smithi045:> stat /dev/vg_nvme/lv_1 2024-08-19T14:07:06.392 INFO:teuthology.orchestra.run.smithi045.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-08-19T14:07:06.392 INFO:teuthology.orchestra.run.smithi045.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-19T14:07:06.392 INFO:teuthology.orchestra.run.smithi045.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-08-19T14:07:06.392 INFO:teuthology.orchestra.run.smithi045.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-19T14:07:06.392 INFO:teuthology.orchestra.run.smithi045.stdout:Context: system_u:object_r:device_t:s0 2024-08-19T14:07:06.392 INFO:teuthology.orchestra.run.smithi045.stdout:Access: 2024-08-19 14:06:46.137273983 +0000 2024-08-19T14:07:06.393 INFO:teuthology.orchestra.run.smithi045.stdout:Modify: 2024-08-19 14:02:18.100616474 +0000 2024-08-19T14:07:06.393 INFO:teuthology.orchestra.run.smithi045.stdout:Change: 2024-08-19 14:02:18.100616474 +0000 2024-08-19T14:07:06.393 INFO:teuthology.orchestra.run.smithi045.stdout: Birth: 2024-08-19 14:02:18.100616474 +0000 2024-08-19T14:07:06.393 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-08-19T14:07:06.461 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-08-19T14:07:06.461 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-08-19T14:07:06.461 INFO:teuthology.orchestra.run.smithi045.stderr:512 bytes copied, 0.00012237 s, 4.2 MB/s 2024-08-19T14:07:06.462 DEBUG:teuthology.orchestra.run.smithi045:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-08-19T14:07:06.521 DEBUG:teuthology.orchestra.run.smithi045:> stat /dev/vg_nvme/lv_2 2024-08-19T14:07:06.579 INFO:teuthology.orchestra.run.smithi045.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-08-19T14:07:06.579 INFO:teuthology.orchestra.run.smithi045.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-19T14:07:06.580 INFO:teuthology.orchestra.run.smithi045.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-08-19T14:07:06.580 INFO:teuthology.orchestra.run.smithi045.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-19T14:07:06.580 INFO:teuthology.orchestra.run.smithi045.stdout:Context: system_u:object_r:device_t:s0 2024-08-19T14:07:06.580 INFO:teuthology.orchestra.run.smithi045.stdout:Access: 2024-08-19 14:06:46.137273983 +0000 2024-08-19T14:07:06.580 INFO:teuthology.orchestra.run.smithi045.stdout:Modify: 2024-08-19 14:02:18.422615532 +0000 2024-08-19T14:07:06.580 INFO:teuthology.orchestra.run.smithi045.stdout:Change: 2024-08-19 14:02:18.422615532 +0000 2024-08-19T14:07:06.580 INFO:teuthology.orchestra.run.smithi045.stdout: Birth: 2024-08-19 14:02:18.422615532 +0000 2024-08-19T14:07:06.580 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-08-19T14:07:06.645 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-08-19T14:07:06.645 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-08-19T14:07:06.645 INFO:teuthology.orchestra.run.smithi045.stderr:512 bytes copied, 0.000122999 s, 4.2 MB/s 2024-08-19T14:07:06.646 DEBUG:teuthology.orchestra.run.smithi045:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-08-19T14:07:06.705 DEBUG:teuthology.orchestra.run.smithi045:> stat /dev/vg_nvme/lv_3 2024-08-19T14:07:06.762 INFO:teuthology.orchestra.run.smithi045.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-08-19T14:07:06.762 INFO:teuthology.orchestra.run.smithi045.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-19T14:07:06.762 INFO:teuthology.orchestra.run.smithi045.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-08-19T14:07:06.762 INFO:teuthology.orchestra.run.smithi045.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-19T14:07:06.763 INFO:teuthology.orchestra.run.smithi045.stdout:Context: system_u:object_r:device_t:s0 2024-08-19T14:07:06.763 INFO:teuthology.orchestra.run.smithi045.stdout:Access: 2024-08-19 14:06:46.138273968 +0000 2024-08-19T14:07:06.763 INFO:teuthology.orchestra.run.smithi045.stdout:Modify: 2024-08-19 14:02:18.742614596 +0000 2024-08-19T14:07:06.763 INFO:teuthology.orchestra.run.smithi045.stdout:Change: 2024-08-19 14:02:18.742614596 +0000 2024-08-19T14:07:06.763 INFO:teuthology.orchestra.run.smithi045.stdout: Birth: 2024-08-19 14:02:18.742614596 +0000 2024-08-19T14:07:06.763 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-08-19T14:07:06.827 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-08-19T14:07:06.827 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-08-19T14:07:06.828 INFO:teuthology.orchestra.run.smithi045.stderr:512 bytes copied, 0.000111392 s, 4.6 MB/s 2024-08-19T14:07:06.829 DEBUG:teuthology.orchestra.run.smithi045:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-08-19T14:07:06.886 DEBUG:teuthology.orchestra.run.smithi045:> stat /dev/vg_nvme/lv_4 2024-08-19T14:07:06.942 INFO:teuthology.orchestra.run.smithi045.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-08-19T14:07:06.942 INFO:teuthology.orchestra.run.smithi045.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-19T14:07:06.943 INFO:teuthology.orchestra.run.smithi045.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-08-19T14:07:06.943 INFO:teuthology.orchestra.run.smithi045.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-19T14:07:06.943 INFO:teuthology.orchestra.run.smithi045.stdout:Context: system_u:object_r:device_t:s0 2024-08-19T14:07:06.943 INFO:teuthology.orchestra.run.smithi045.stdout:Access: 2024-08-19 14:06:46.138273968 +0000 2024-08-19T14:07:06.943 INFO:teuthology.orchestra.run.smithi045.stdout:Modify: 2024-08-19 14:02:19.062613660 +0000 2024-08-19T14:07:06.943 INFO:teuthology.orchestra.run.smithi045.stdout:Change: 2024-08-19 14:02:19.062613660 +0000 2024-08-19T14:07:06.943 INFO:teuthology.orchestra.run.smithi045.stdout: Birth: 2024-08-19 14:02:19.062613660 +0000 2024-08-19T14:07:06.943 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-08-19T14:07:07.008 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-08-19T14:07:07.008 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-08-19T14:07:07.008 INFO:teuthology.orchestra.run.smithi045.stderr:512 bytes copied, 0.00010827 s, 4.7 MB/s 2024-08-19T14:07:07.009 DEBUG:teuthology.orchestra.run.smithi045:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-08-19T14:07:07.066 INFO:tasks.cephadm:Deploying osd.0 on smithi031 with /dev/vg_nvme/lv_4... 2024-08-19T14:07:07.067 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- lvm zap /dev/vg_nvme/lv_4 2024-08-19T14:07:07.172 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:06 smithi032 ceph-mon[20004]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T14:07:07.172 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:06 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:07.172 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:06 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:07.172 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:06 smithi032 ceph-mon[20004]: Reconfiguring mon.b (monmap changed)... 2024-08-19T14:07:07.172 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:06 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-08-19T14:07:07.173 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:06 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-08-19T14:07:07.173 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:06 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:07:07.173 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:06 smithi032 ceph-mon[20004]: Reconfiguring daemon mon.b on smithi032 2024-08-19T14:07:07.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:06 smithi045 ceph-mon[20143]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T14:07:07.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:06 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:07.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:06 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:07.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:06 smithi045 ceph-mon[20143]: Reconfiguring mon.b (monmap changed)... 2024-08-19T14:07:07.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:06 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-08-19T14:07:07.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:06 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-08-19T14:07:07.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:06 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:07:07.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:06 smithi045 ceph-mon[20143]: Reconfiguring daemon mon.b on smithi032 2024-08-19T14:07:07.331 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:07:07.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:06 smithi031 ceph-mon[16336]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T14:07:07.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:06 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:07.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:06 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:07.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:06 smithi031 ceph-mon[16336]: Reconfiguring mon.b (monmap changed)... 2024-08-19T14:07:07.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:06 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-08-19T14:07:07.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:06 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-08-19T14:07:07.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:06 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:07:07.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:06 smithi031 ceph-mon[16336]: Reconfiguring daemon mon.b on smithi032 2024-08-19T14:07:08.479 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:08 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:08.479 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:08 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:08.479 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:08 smithi031 ceph-mon[16336]: Reconfiguring mon.c (monmap changed)... 2024-08-19T14:07:08.480 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:08 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-08-19T14:07:08.480 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:08 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-08-19T14:07:08.480 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:08 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:07:08.480 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:08 smithi031 ceph-mon[16336]: Reconfiguring daemon mon.c on smithi045 2024-08-19T14:07:08.497 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:08 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:08.497 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:08 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:08.498 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:08 smithi045 ceph-mon[20143]: Reconfiguring mon.c (monmap changed)... 2024-08-19T14:07:08.498 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:08 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-08-19T14:07:08.498 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:08 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-08-19T14:07:08.498 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:08 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:07:08.498 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:08 smithi045 ceph-mon[20143]: Reconfiguring daemon mon.c on smithi045 2024-08-19T14:07:08.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:08 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:08.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:08 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:08.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:08 smithi032 ceph-mon[20004]: Reconfiguring mon.c (monmap changed)... 2024-08-19T14:07:08.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:08 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-08-19T14:07:08.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:08 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-08-19T14:07:08.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:08 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:07:08.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:08 smithi032 ceph-mon[20004]: Reconfiguring daemon mon.c on smithi045 2024-08-19T14:07:09.647 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:07:09.672 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph orch daemon add osd smithi031:vg_nvme/lv_4 2024-08-19T14:07:09.695 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:09 smithi031 ceph-mon[16336]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T14:07:09.695 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:09 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:09.695 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:09 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:09.696 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:09 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:07:09.696 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:09 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:07:09.696 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:09 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:07:09.696 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:09 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:09.696 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:09 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-08-19T14:07:09.696 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:09 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-08-19T14:07:09.696 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:09 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-08-19T14:07:09.696 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:09 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:07:09.696 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:09 smithi031 ceph-mon[16336]: Deploying daemon mgr.x on smithi032 2024-08-19T14:07:09.878 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:09 smithi032 ceph-mon[20004]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T14:07:09.878 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:09 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:09.878 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:09 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:09.879 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:09 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:07:09.879 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:09 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:07:09.879 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:09 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:07:09.879 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:09 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:09.879 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:09 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-08-19T14:07:09.879 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:09 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-08-19T14:07:09.879 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:09 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-08-19T14:07:09.879 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:09 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:07:09.879 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:09 smithi032 ceph-mon[20004]: Deploying daemon mgr.x on smithi032 2024-08-19T14:07:09.922 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:07:10.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:09 smithi045 ceph-mon[20143]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T14:07:10.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:09 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:10.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:09 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:10.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:09 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:07:10.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:09 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:07:10.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:09 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:07:10.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:09 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:10.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:09 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-08-19T14:07:10.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:09 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-08-19T14:07:10.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:09 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-08-19T14:07:10.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:09 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:07:10.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:09 smithi045 ceph-mon[20143]: Deploying daemon mgr.x on smithi032 2024-08-19T14:07:10.508 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Aug 19 14:07:10 smithi032 systemd[1]: Starting Ceph mgr.x for ed7d1c6e-5e33-11ef-bccf-c7b262605968... 2024-08-19T14:07:10.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:10 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-19T14:07:10.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:10 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-19T14:07:10.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:10 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:07:10.818 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:10 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-19T14:07:10.819 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:10 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-19T14:07:10.819 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:10 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:07:10.819 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Aug 19 14:07:10 smithi032 podman[21362]: 2024-08-19 14:07:10.546927613 +0000 UTC m=+0.109048481 container create 70be1fbca0718275dc69c5dde99d1c2f2eaa172ee9483fd4808a7a8efc6a3569 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-x, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-de09dd0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2) 2024-08-19T14:07:10.819 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Aug 19 14:07:10 smithi032 podman[21362]: 2024-08-19 14:07:10.455299383 +0000 UTC m=+0.017420261 image pull ea7a5189f30bad3aca9524788e2965d2368d7ce786df09d4edcd8c3edf9cefba quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9 2024-08-19T14:07:10.819 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Aug 19 14:07:10 smithi032 podman[21362]: 2024-08-19 14:07:10.790944523 +0000 UTC m=+0.353065391 container init 70be1fbca0718275dc69c5dde99d1c2f2eaa172ee9483fd4808a7a8efc6a3569 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-x, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-de09dd0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, org.label-schema.schema-version=1.0) 2024-08-19T14:07:10.819 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Aug 19 14:07:10 smithi032 podman[21362]: 2024-08-19 14:07:10.793937034 +0000 UTC m=+0.356057903 container start 70be1fbca0718275dc69c5dde99d1c2f2eaa172ee9483fd4808a7a8efc6a3569 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-x, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=squid-de09dd0, ceph=True) 2024-08-19T14:07:10.819 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Aug 19 14:07:10 smithi032 ceph-mgr[21383]: -- 172.21.15.32:0/2379828582 <== mon.2 v2:172.21.15.32:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555ec32d10e0 con 0x555ec32ae800 2024-08-19T14:07:11.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:10 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-19T14:07:11.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:10 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-19T14:07:11.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:10 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:07:11.091 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Aug 19 14:07:10 smithi032 bash[21362]: 70be1fbca0718275dc69c5dde99d1c2f2eaa172ee9483fd4808a7a8efc6a3569 2024-08-19T14:07:11.091 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Aug 19 14:07:10 smithi032 systemd[1]: Started Ceph mgr.x for ed7d1c6e-5e33-11ef-bccf-c7b262605968. 2024-08-19T14:07:11.092 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Aug 19 14:07:11 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-x[21379]: 2024-08-19T14:07:11.020+0000 7ff6a49c6040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-08-19T14:07:11.092 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Aug 19 14:07:11 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-x[21379]: 2024-08-19T14:07:11.090+0000 7ff6a49c6040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-08-19T14:07:11.422 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Aug 19 14:07:11 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-x[21379]: 2024-08-19T14:07:11.241+0000 7ff6a49c6040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-08-19T14:07:11.709 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:11 smithi032 ceph-mon[20004]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T14:07:11.710 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:11 smithi032 ceph-mon[20004]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T14:07:11.710 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:11 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:11.710 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:11 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:11.710 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:11 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:11.710 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:11 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:11.710 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:11 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:07:11.710 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Aug 19 14:07:11 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-x[21379]: 2024-08-19T14:07:11.421+0000 7ff6a49c6040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-08-19T14:07:11.710 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Aug 19 14:07:11 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-x[21379]: 2024-08-19T14:07:11.576+0000 7ff6a49c6040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-08-19T14:07:11.710 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Aug 19 14:07:11 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-x[21379]: 2024-08-19T14:07:11.708+0000 7ff6a49c6040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-08-19T14:07:11.836 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:11 smithi031 ceph-mon[16336]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T14:07:11.836 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:11 smithi031 ceph-mon[16336]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T14:07:11.836 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:11 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:11.836 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:11 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:11.836 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:11 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:11.836 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:11 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:11.836 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:11 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:07:12.007 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Aug 19 14:07:11 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-x[21379]: 2024-08-19T14:07:11.842+0000 7ff6a49c6040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-08-19T14:07:12.007 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Aug 19 14:07:11 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-x[21379]: 2024-08-19T14:07:11.901+0000 7ff6a49c6040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-08-19T14:07:12.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:11 smithi045 ceph-mon[20143]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T14:07:12.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:11 smithi045 ceph-mon[20143]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T14:07:12.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:11 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:12.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:11 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:12.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:11 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:12.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:11 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:12.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:11 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:07:12.507 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Aug 19 14:07:12 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-x[21379]: 2024-08-19T14:07:12.229+0000 7ff6a49c6040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-08-19T14:07:12.507 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Aug 19 14:07:12 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-x[21379]: 2024-08-19T14:07:12.294+0000 7ff6a49c6040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-08-19T14:07:12.507 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Aug 19 14:07:12 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-x[21379]: 2024-08-19T14:07:12.364+0000 7ff6a49c6040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-08-19T14:07:12.846 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Aug 19 14:07:12 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-x[21379]: 2024-08-19T14:07:12.519+0000 7ff6a49c6040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-08-19T14:07:12.846 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Aug 19 14:07:12 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-x[21379]: 2024-08-19T14:07:12.582+0000 7ff6a49c6040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-08-19T14:07:12.847 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Aug 19 14:07:12 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-x[21379]: 2024-08-19T14:07:12.651+0000 7ff6a49c6040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-08-19T14:07:12.847 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Aug 19 14:07:12 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-x[21379]: 2024-08-19T14:07:12.845+0000 7ff6a49c6040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-08-19T14:07:13.257 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Aug 19 14:07:12 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-x[21379]: 2024-08-19T14:07:12.908+0000 7ff6a49c6040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-08-19T14:07:13.257 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Aug 19 14:07:12 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-x[21379]: 2024-08-19T14:07:12.970+0000 7ff6a49c6040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-08-19T14:07:13.257 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Aug 19 14:07:13 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-x[21379]: 2024-08-19T14:07:13.040+0000 7ff6a49c6040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-08-19T14:07:13.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:13 smithi032 ceph-mon[20004]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T14:07:13.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:13 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:13.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:13 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:13.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:13 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:07:13.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:13 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:07:13.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:13 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:13.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:13 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:07:13.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:13 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:07:13.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:13 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:07:13.259 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:13 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:13.259 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:13 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1486502976' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d7a0fff3-23c6-4ee3-a275-a844862aabfb"}]: dispatch 2024-08-19T14:07:13.259 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:13 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1486502976' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d7a0fff3-23c6-4ee3-a275-a844862aabfb"}]': finished 2024-08-19T14:07:13.259 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:13 smithi032 ceph-mon[20004]: osdmap e5: 1 total, 0 up, 1 in 2024-08-19T14:07:13.259 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:13 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-08-19T14:07:13.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:13 smithi031 ceph-mon[16336]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T14:07:13.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:13 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:13.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:13 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:13.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:13 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:07:13.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:13 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:07:13.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:13 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:13.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:13 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:07:13.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:13 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:07:13.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:13 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:07:13.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:13 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:13.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:13 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1486502976' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d7a0fff3-23c6-4ee3-a275-a844862aabfb"}]: dispatch 2024-08-19T14:07:13.368 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:13 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1486502976' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d7a0fff3-23c6-4ee3-a275-a844862aabfb"}]': finished 2024-08-19T14:07:13.368 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:13 smithi031 ceph-mon[16336]: osdmap e5: 1 total, 0 up, 1 in 2024-08-19T14:07:13.368 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:13 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-08-19T14:07:13.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:13 smithi045 ceph-mon[20143]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T14:07:13.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:13 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:13.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:13 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:13.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:13 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:07:13.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:13 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:07:13.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:13 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:13.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:13 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:07:13.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:13 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:07:13.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:13 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:07:13.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:13 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:13.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:13 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1486502976' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d7a0fff3-23c6-4ee3-a275-a844862aabfb"}]: dispatch 2024-08-19T14:07:13.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:13 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1486502976' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d7a0fff3-23c6-4ee3-a275-a844862aabfb"}]': finished 2024-08-19T14:07:13.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:13 smithi045 ceph-mon[20143]: osdmap e5: 1 total, 0 up, 1 in 2024-08-19T14:07:13.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:13 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-08-19T14:07:13.757 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Aug 19 14:07:13 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-x[21379]: 2024-08-19T14:07:13.261+0000 7ff6a49c6040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-08-19T14:07:13.757 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Aug 19 14:07:13 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-x[21379]: 2024-08-19T14:07:13.323+0000 7ff6a49c6040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-08-19T14:07:14.200 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Aug 19 14:07:13 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-x[21379]: 2024-08-19T14:07:13.874+0000 7ff6a49c6040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-08-19T14:07:14.200 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:14 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/4039210093' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-19T14:07:14.200 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:14 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:14.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:14 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/4039210093' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-19T14:07:14.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:14 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:14.507 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Aug 19 14:07:14 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-x[21379]: 2024-08-19T14:07:14.199+0000 7ff6a49c6040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-08-19T14:07:14.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:14 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/4039210093' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-19T14:07:14.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:14 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:15.007 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Aug 19 14:07:14 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-x[21379]: 2024-08-19T14:07:14.631+0000 7ff6a49c6040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-08-19T14:07:15.007 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Aug 19 14:07:14 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-x[21379]: 2024-08-19T14:07:14.690+0000 7ff6a49c6040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-08-19T14:07:15.008 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Aug 19 14:07:14 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-x[21379]: 2024-08-19T14:07:14.752+0000 7ff6a49c6040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-08-19T14:07:15.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:15 smithi031 ceph-mon[16336]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T14:07:15.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:15 smithi032 ceph-mon[20004]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T14:07:15.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:15 smithi045 ceph-mon[20143]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T14:07:16.171 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:16 smithi031 ceph-mon[16336]: Standby manager daemon x started 2024-08-19T14:07:16.172 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:16 smithi031 ceph-mon[16336]: from='mgr.? 172.21.15.32:0/390459342' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-08-19T14:07:16.172 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:16 smithi031 ceph-mon[16336]: from='mgr.? 172.21.15.32:0/390459342' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-08-19T14:07:16.172 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:16 smithi031 ceph-mon[16336]: from='mgr.? 172.21.15.32:0/390459342' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-08-19T14:07:16.172 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:16 smithi031 ceph-mon[16336]: from='mgr.? 172.21.15.32:0/390459342' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-08-19T14:07:16.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:16 smithi032 ceph-mon[20004]: Standby manager daemon x started 2024-08-19T14:07:16.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:16 smithi032 ceph-mon[20004]: from='mgr.? 172.21.15.32:0/390459342' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-08-19T14:07:16.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:16 smithi032 ceph-mon[20004]: from='mgr.? 172.21.15.32:0/390459342' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-08-19T14:07:16.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:16 smithi032 ceph-mon[20004]: from='mgr.? 172.21.15.32:0/390459342' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-08-19T14:07:16.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:16 smithi032 ceph-mon[20004]: from='mgr.? 172.21.15.32:0/390459342' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-08-19T14:07:16.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:16 smithi045 ceph-mon[20143]: Standby manager daemon x started 2024-08-19T14:07:16.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:16 smithi045 ceph-mon[20143]: from='mgr.? 172.21.15.32:0/390459342' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-08-19T14:07:16.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:16 smithi045 ceph-mon[20143]: from='mgr.? 172.21.15.32:0/390459342' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-08-19T14:07:16.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:16 smithi045 ceph-mon[20143]: from='mgr.? 172.21.15.32:0/390459342' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-08-19T14:07:16.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:16 smithi045 ceph-mon[20143]: from='mgr.? 172.21.15.32:0/390459342' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-08-19T14:07:17.257 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:17 smithi031 ceph-mon[16336]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T14:07:17.257 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:17 smithi031 ceph-mon[16336]: mgrmap e14: y(active, since 62s), standbys: x 2024-08-19T14:07:17.257 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:17 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-08-19T14:07:17.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:17 smithi032 ceph-mon[20004]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T14:07:17.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:17 smithi032 ceph-mon[20004]: mgrmap e14: y(active, since 62s), standbys: x 2024-08-19T14:07:17.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:17 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-08-19T14:07:17.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:17 smithi045 ceph-mon[20143]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T14:07:17.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:17 smithi045 ceph-mon[20143]: mgrmap e14: y(active, since 62s), standbys: x 2024-08-19T14:07:17.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:17 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-08-19T14:07:19.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:19 smithi031 ceph-mon[16336]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T14:07:19.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:19 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-08-19T14:07:19.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:19 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:07:19.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:19 smithi032 ceph-mon[20004]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T14:07:19.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:19 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-08-19T14:07:19.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:19 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:07:19.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:19 smithi045 ceph-mon[20143]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T14:07:19.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:19 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-08-19T14:07:19.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:19 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:07:20.299 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:20 smithi031 ceph-mon[16336]: Deploying daemon osd.0 on smithi031 2024-08-19T14:07:20.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:20 smithi032 ceph-mon[20004]: Deploying daemon osd.0 on smithi031 2024-08-19T14:07:20.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:20 smithi045 ceph-mon[20143]: Deploying daemon osd.0 on smithi031 2024-08-19T14:07:21.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:21 smithi031 ceph-mon[16336]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T14:07:21.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:21 smithi032 ceph-mon[20004]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T14:07:21.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:21 smithi045 ceph-mon[20143]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T14:07:23.329 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:23 smithi031 ceph-mon[16336]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T14:07:23.348 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:23 smithi045 ceph-mon[20143]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T14:07:23.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:23 smithi032 ceph-mon[20004]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T14:07:24.339 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:24 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:07:24.339 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:24 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:24.339 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:24 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:24.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:24 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:07:24.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:24 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:24.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:24 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:24.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:24 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:07:24.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:24 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:24.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:24 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:25.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:25 smithi031 ceph-mon[16336]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T14:07:25.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:25 smithi032 ceph-mon[20004]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T14:07:25.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:25 smithi045 ceph-mon[20143]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T14:07:26.298 INFO:teuthology.orchestra.run.smithi031.stdout:Created osd(s) 0 on host 'smithi031' 2024-08-19T14:07:27.191 DEBUG:teuthology.orchestra.run.smithi031:osd.0> sudo journalctl -f -n 0 -u ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968@osd.0.service 2024-08-19T14:07:27.233 INFO:tasks.cephadm:Deploying osd.1 on smithi031 with /dev/vg_nvme/lv_3... 2024-08-19T14:07:27.234 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- lvm zap /dev/vg_nvme/lv_3 2024-08-19T14:07:27.530 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:07:27.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:27 smithi045 ceph-mon[20143]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T14:07:27.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:27 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:27.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:27 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:27.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:27 smithi045 ceph-mon[20143]: from='osd.0 [v2:172.21.15.31:6802/1108945097,v1:172.21.15.31:6803/1108945097]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-08-19T14:07:27.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:27 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:27.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:27 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:27.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:27 smithi031 ceph-mon[16336]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T14:07:27.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:27 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:27.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:27 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:27.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:27 smithi031 ceph-mon[16336]: from='osd.0 [v2:172.21.15.31:6802/1108945097,v1:172.21.15.31:6803/1108945097]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-08-19T14:07:27.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:27 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:27.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:27 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:27.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:27 smithi032 ceph-mon[20004]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T14:07:27.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:27 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:27.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:27 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:27.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:27 smithi032 ceph-mon[20004]: from='osd.0 [v2:172.21.15.31:6802/1108945097,v1:172.21.15.31:6803/1108945097]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-08-19T14:07:27.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:27 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:27.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:27 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:28.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:28 smithi031 ceph-mon[16336]: from='osd.0 [v2:172.21.15.31:6802/1108945097,v1:172.21.15.31:6803/1108945097]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-08-19T14:07:28.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:28 smithi031 ceph-mon[16336]: osdmap e6: 1 total, 0 up, 1 in 2024-08-19T14:07:28.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:28 smithi031 ceph-mon[16336]: from='osd.0 [v2:172.21.15.31:6802/1108945097,v1:172.21.15.31:6803/1108945097]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-08-19T14:07:28.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:28 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-08-19T14:07:28.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:28 smithi032 ceph-mon[20004]: from='osd.0 [v2:172.21.15.31:6802/1108945097,v1:172.21.15.31:6803/1108945097]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-08-19T14:07:28.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:28 smithi032 ceph-mon[20004]: osdmap e6: 1 total, 0 up, 1 in 2024-08-19T14:07:28.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:28 smithi032 ceph-mon[20004]: from='osd.0 [v2:172.21.15.31:6802/1108945097,v1:172.21.15.31:6803/1108945097]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-08-19T14:07:28.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:28 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-08-19T14:07:28.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:28 smithi045 ceph-mon[20143]: from='osd.0 [v2:172.21.15.31:6802/1108945097,v1:172.21.15.31:6803/1108945097]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-08-19T14:07:28.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:28 smithi045 ceph-mon[20143]: osdmap e6: 1 total, 0 up, 1 in 2024-08-19T14:07:28.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:28 smithi045 ceph-mon[20143]: from='osd.0 [v2:172.21.15.31:6802/1108945097,v1:172.21.15.31:6803/1108945097]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-08-19T14:07:28.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:28 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-08-19T14:07:28.956 INFO:journalctl@ceph.osd.0.smithi031.stdout:Aug 19 14:07:28 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-0[30082]: 2024-08-19T14:07:28.850+0000 7fd569261640 -1 osd.0 0 waiting for initial osdmap 2024-08-19T14:07:28.957 INFO:journalctl@ceph.osd.0.smithi031.stdout:Aug 19 14:07:28 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-0[30082]: 2024-08-19T14:07:28.852+0000 7fd561081640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-19T14:07:29.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:29 smithi031 ceph-mon[16336]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T14:07:29.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:29 smithi031 ceph-mon[16336]: from='osd.0 [v2:172.21.15.31:6802/1108945097,v1:172.21.15.31:6803/1108945097]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2024-08-19T14:07:29.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:29 smithi031 ceph-mon[16336]: osdmap e7: 1 total, 0 up, 1 in 2024-08-19T14:07:29.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:29 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-08-19T14:07:29.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:29 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-08-19T14:07:29.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:29 smithi031 ceph-mon[16336]: from='osd.0 [v2:172.21.15.31:6802/1108945097,v1:172.21.15.31:6803/1108945097]' entity='osd.0' 2024-08-19T14:07:29.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:29 smithi032 ceph-mon[20004]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T14:07:29.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:29 smithi032 ceph-mon[20004]: from='osd.0 [v2:172.21.15.31:6802/1108945097,v1:172.21.15.31:6803/1108945097]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2024-08-19T14:07:29.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:29 smithi032 ceph-mon[20004]: osdmap e7: 1 total, 0 up, 1 in 2024-08-19T14:07:29.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:29 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-08-19T14:07:29.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:29 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-08-19T14:07:29.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:29 smithi032 ceph-mon[20004]: from='osd.0 [v2:172.21.15.31:6802/1108945097,v1:172.21.15.31:6803/1108945097]' entity='osd.0' 2024-08-19T14:07:29.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:29 smithi045 ceph-mon[20143]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T14:07:29.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:29 smithi045 ceph-mon[20143]: from='osd.0 [v2:172.21.15.31:6802/1108945097,v1:172.21.15.31:6803/1108945097]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2024-08-19T14:07:29.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:29 smithi045 ceph-mon[20143]: osdmap e7: 1 total, 0 up, 1 in 2024-08-19T14:07:29.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:29 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-08-19T14:07:29.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:29 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-08-19T14:07:29.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:29 smithi045 ceph-mon[20143]: from='osd.0 [v2:172.21.15.31:6802/1108945097,v1:172.21.15.31:6803/1108945097]' entity='osd.0' 2024-08-19T14:07:30.565 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:30 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:07:30.565 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:30 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:07:30.565 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:30 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-08-19T14:07:30.565 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:30 smithi031 ceph-mon[16336]: osd.0 [v2:172.21.15.31:6802/1108945097,v1:172.21.15.31:6803/1108945097] boot 2024-08-19T14:07:30.565 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:30 smithi031 ceph-mon[16336]: osdmap e8: 1 total, 1 up, 1 in 2024-08-19T14:07:30.566 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:30 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-08-19T14:07:30.568 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:07:30.601 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph orch daemon add osd smithi031:vg_nvme/lv_3 2024-08-19T14:07:30.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:30 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:07:30.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:30 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:07:30.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:30 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-08-19T14:07:30.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:30 smithi032 ceph-mon[20004]: osd.0 [v2:172.21.15.31:6802/1108945097,v1:172.21.15.31:6803/1108945097] boot 2024-08-19T14:07:30.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:30 smithi032 ceph-mon[20004]: osdmap e8: 1 total, 1 up, 1 in 2024-08-19T14:07:30.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:30 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-08-19T14:07:30.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:30 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:07:30.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:30 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:07:30.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:30 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-08-19T14:07:30.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:30 smithi045 ceph-mon[20143]: osd.0 [v2:172.21.15.31:6802/1108945097,v1:172.21.15.31:6803/1108945097] boot 2024-08-19T14:07:30.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:30 smithi045 ceph-mon[20143]: osdmap e8: 1 total, 1 up, 1 in 2024-08-19T14:07:30.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:30 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-08-19T14:07:31.121 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:07:31.538 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:31 smithi031 ceph-mon[16336]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T14:07:31.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:31 smithi032 ceph-mon[20004]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T14:07:31.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:31 smithi045 ceph-mon[20143]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-19T14:07:32.727 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:32 smithi031 ceph-mon[16336]: osdmap e9: 1 total, 1 up, 1 in 2024-08-19T14:07:32.728 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:32 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-19T14:07:32.728 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:32 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-19T14:07:32.728 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:32 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:07:32.728 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:32 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:32.728 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:32 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:32.728 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:32 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:07:32.728 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:32 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:32.728 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:32 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:07:32.728 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:32 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:07:32.728 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:32 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:32.728 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:32 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:07:32.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:32 smithi032 ceph-mon[20004]: osdmap e9: 1 total, 1 up, 1 in 2024-08-19T14:07:32.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:32 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-19T14:07:32.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:32 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-19T14:07:32.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:32 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:07:32.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:32 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:32.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:32 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:32.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:32 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:07:32.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:32 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:32.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:32 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:07:32.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:32 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:07:32.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:32 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:32.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:32 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:07:32.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:32 smithi045 ceph-mon[20143]: osdmap e9: 1 total, 1 up, 1 in 2024-08-19T14:07:32.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:32 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-19T14:07:32.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:32 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-19T14:07:32.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:32 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:07:32.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:32 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:32.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:32 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:32.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:32 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:07:32.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:32 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:32.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:32 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:07:32.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:32 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:07:32.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:32 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:32.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:32 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:07:33.609 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:33 smithi031 ceph-mon[16336]: pgmap v38: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-19T14:07:33.609 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:33 smithi031 ceph-mon[16336]: from='client.14250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T14:07:33.609 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:33 smithi031 ceph-mon[16336]: Detected new or changed devices on smithi031 2024-08-19T14:07:33.609 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:33 smithi031 ceph-mon[16336]: Adjusting osd_memory_target on smithi031 to 17092M 2024-08-19T14:07:33.609 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:33 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:07:33.609 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:33 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:07:33.609 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:33 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:33.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:33 smithi032 ceph-mon[20004]: pgmap v38: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-19T14:07:33.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:33 smithi032 ceph-mon[20004]: from='client.14250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T14:07:33.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:33 smithi032 ceph-mon[20004]: Detected new or changed devices on smithi031 2024-08-19T14:07:33.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:33 smithi032 ceph-mon[20004]: Adjusting osd_memory_target on smithi031 to 17092M 2024-08-19T14:07:33.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:33 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:07:33.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:33 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:07:33.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:33 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:33.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:33 smithi045 ceph-mon[20143]: pgmap v38: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-19T14:07:33.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:33 smithi045 ceph-mon[20143]: from='client.14250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T14:07:33.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:33 smithi045 ceph-mon[20143]: Detected new or changed devices on smithi031 2024-08-19T14:07:33.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:33 smithi045 ceph-mon[20143]: Adjusting osd_memory_target on smithi031 to 17092M 2024-08-19T14:07:33.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:33 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:07:33.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:33 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:07:33.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:33 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:34.654 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:34 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/563921398' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "145d5bb4-74dd-440f-bcc9-7e9ebda9c5d6"}]: dispatch 2024-08-19T14:07:34.658 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:34 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/563921398' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "145d5bb4-74dd-440f-bcc9-7e9ebda9c5d6"}]': finished 2024-08-19T14:07:34.658 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:34 smithi031 ceph-mon[16336]: osdmap e10: 2 total, 1 up, 2 in 2024-08-19T14:07:34.658 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:34 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-08-19T14:07:34.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:34 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/563921398' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "145d5bb4-74dd-440f-bcc9-7e9ebda9c5d6"}]: dispatch 2024-08-19T14:07:34.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:34 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/563921398' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "145d5bb4-74dd-440f-bcc9-7e9ebda9c5d6"}]': finished 2024-08-19T14:07:34.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:34 smithi032 ceph-mon[20004]: osdmap e10: 2 total, 1 up, 2 in 2024-08-19T14:07:34.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:34 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-08-19T14:07:34.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:34 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/563921398' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "145d5bb4-74dd-440f-bcc9-7e9ebda9c5d6"}]: dispatch 2024-08-19T14:07:34.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:34 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/563921398' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "145d5bb4-74dd-440f-bcc9-7e9ebda9c5d6"}]': finished 2024-08-19T14:07:34.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:34 smithi045 ceph-mon[20143]: osdmap e10: 2 total, 1 up, 2 in 2024-08-19T14:07:34.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:34 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-08-19T14:07:35.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:35 smithi032 ceph-mon[20004]: pgmap v39: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-19T14:07:35.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:35 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3867054175' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-19T14:07:35.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:35 smithi045 ceph-mon[20143]: pgmap v39: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-19T14:07:35.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:35 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3867054175' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-19T14:07:35.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:35 smithi031 ceph-mon[16336]: pgmap v39: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-19T14:07:35.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:35 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3867054175' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-19T14:07:37.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:37 smithi032 ceph-mon[20004]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-19T14:07:37.772 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:37 smithi031 ceph-mon[16336]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-19T14:07:37.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:37 smithi045 ceph-mon[20143]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-19T14:07:39.437 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:39 smithi031 ceph-mon[16336]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-19T14:07:39.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:39 smithi032 ceph-mon[20004]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-19T14:07:39.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:39 smithi045 ceph-mon[20143]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-19T14:07:41.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:41 smithi032 ceph-mon[20004]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-19T14:07:41.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:41 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-08-19T14:07:41.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:41 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:07:41.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:41 smithi032 ceph-mon[20004]: Deploying daemon osd.1 on smithi031 2024-08-19T14:07:41.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:41 smithi045 ceph-mon[20143]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-19T14:07:41.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:41 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-08-19T14:07:41.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:41 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:07:41.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:41 smithi045 ceph-mon[20143]: Deploying daemon osd.1 on smithi031 2024-08-19T14:07:41.838 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:41 smithi031 ceph-mon[16336]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-19T14:07:41.838 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:41 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-08-19T14:07:41.838 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:41 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:07:41.838 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:41 smithi031 ceph-mon[16336]: Deploying daemon osd.1 on smithi031 2024-08-19T14:07:43.573 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:43 smithi031 ceph-mon[16336]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-19T14:07:43.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:43 smithi032 ceph-mon[20004]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-19T14:07:43.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:43 smithi045 ceph-mon[20143]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-19T14:07:45.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:45 smithi032 ceph-mon[20004]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-19T14:07:45.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:45 smithi045 ceph-mon[20143]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-19T14:07:45.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:45 smithi031 ceph-mon[16336]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-19T14:07:46.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:46 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:07:46.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:46 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:46.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:46 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:46.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:46 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:07:46.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:46 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:46.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:46 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:46.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:46 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:07:46.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:46 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:46.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:46 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:47.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:47 smithi031 ceph-mon[16336]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-19T14:07:48.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:47 smithi032 ceph-mon[20004]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-19T14:07:48.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:47 smithi045 ceph-mon[20143]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-19T14:07:49.012 INFO:teuthology.orchestra.run.smithi031.stdout:Created osd(s) 1 on host 'smithi031' 2024-08-19T14:07:49.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:48 smithi032 ceph-mon[20004]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-19T14:07:49.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:48 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:49.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:48 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:49.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:48 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:07:49.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:48 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:07:49.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:48 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:49.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:48 smithi032 ceph-mon[20004]: from='osd.1 [v2:172.21.15.31:6810/3236186806,v1:172.21.15.31:6811/3236186806]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-08-19T14:07:49.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:48 smithi045 ceph-mon[20143]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-19T14:07:49.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:48 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:49.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:48 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:49.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:48 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:07:49.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:48 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:07:49.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:48 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:49.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:48 smithi045 ceph-mon[20143]: from='osd.1 [v2:172.21.15.31:6810/3236186806,v1:172.21.15.31:6811/3236186806]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-08-19T14:07:49.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:48 smithi031 ceph-mon[16336]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-19T14:07:49.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:48 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:49.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:48 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:49.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:48 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:07:49.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:48 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:07:49.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:48 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:49.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:48 smithi031 ceph-mon[16336]: from='osd.1 [v2:172.21.15.31:6810/3236186806,v1:172.21.15.31:6811/3236186806]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-08-19T14:07:49.583 DEBUG:teuthology.orchestra.run.smithi031:osd.1> sudo journalctl -f -n 0 -u ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968@osd.1.service 2024-08-19T14:07:49.585 INFO:tasks.cephadm:Deploying osd.2 on smithi031 with /dev/vg_nvme/lv_2... 2024-08-19T14:07:49.586 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- lvm zap /dev/vg_nvme/lv_2 2024-08-19T14:07:50.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:49 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:07:50.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:49 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:50.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:49 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:50.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:49 smithi032 ceph-mon[20004]: from='osd.1 [v2:172.21.15.31:6810/3236186806,v1:172.21.15.31:6811/3236186806]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-08-19T14:07:50.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:49 smithi032 ceph-mon[20004]: osdmap e11: 2 total, 1 up, 2 in 2024-08-19T14:07:50.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:49 smithi032 ceph-mon[20004]: from='osd.1 [v2:172.21.15.31:6810/3236186806,v1:172.21.15.31:6811/3236186806]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-08-19T14:07:50.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:49 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-08-19T14:07:50.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:49 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:07:50.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:49 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:50.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:49 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:50.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:49 smithi045 ceph-mon[20143]: from='osd.1 [v2:172.21.15.31:6810/3236186806,v1:172.21.15.31:6811/3236186806]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-08-19T14:07:50.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:49 smithi045 ceph-mon[20143]: osdmap e11: 2 total, 1 up, 2 in 2024-08-19T14:07:50.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:49 smithi045 ceph-mon[20143]: from='osd.1 [v2:172.21.15.31:6810/3236186806,v1:172.21.15.31:6811/3236186806]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-08-19T14:07:50.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:49 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-08-19T14:07:50.356 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:07:50.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:49 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:07:50.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:49 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:50.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:49 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:50.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:49 smithi031 ceph-mon[16336]: from='osd.1 [v2:172.21.15.31:6810/3236186806,v1:172.21.15.31:6811/3236186806]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-08-19T14:07:50.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:49 smithi031 ceph-mon[16336]: osdmap e11: 2 total, 1 up, 2 in 2024-08-19T14:07:50.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:49 smithi031 ceph-mon[16336]: from='osd.1 [v2:172.21.15.31:6810/3236186806,v1:172.21.15.31:6811/3236186806]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-08-19T14:07:50.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:49 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-08-19T14:07:51.366 INFO:journalctl@ceph.osd.1.smithi031.stdout:Aug 19 14:07:51 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-1[35635]: 2024-08-19T14:07:51.017+0000 7fab76928640 -1 osd.1 0 waiting for initial osdmap 2024-08-19T14:07:51.367 INFO:journalctl@ceph.osd.1.smithi031.stdout:Aug 19 14:07:51 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-1[35635]: 2024-08-19T14:07:51.022+0000 7fab6e748640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-19T14:07:51.788 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:51 smithi031 ceph-mon[16336]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-19T14:07:51.788 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:51 smithi031 ceph-mon[16336]: from='osd.1 [v2:172.21.15.31:6810/3236186806,v1:172.21.15.31:6811/3236186806]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2024-08-19T14:07:51.789 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:51 smithi031 ceph-mon[16336]: osdmap e12: 2 total, 1 up, 2 in 2024-08-19T14:07:51.789 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:51 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-08-19T14:07:51.789 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:51 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-08-19T14:07:51.789 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:51 smithi031 ceph-mon[16336]: from='osd.1 [v2:172.21.15.31:6810/3236186806,v1:172.21.15.31:6811/3236186806]' entity='osd.1' 2024-08-19T14:07:52.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:51 smithi032 ceph-mon[20004]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-19T14:07:52.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:51 smithi032 ceph-mon[20004]: from='osd.1 [v2:172.21.15.31:6810/3236186806,v1:172.21.15.31:6811/3236186806]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2024-08-19T14:07:52.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:51 smithi032 ceph-mon[20004]: osdmap e12: 2 total, 1 up, 2 in 2024-08-19T14:07:52.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:51 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-08-19T14:07:52.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:51 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-08-19T14:07:52.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:51 smithi032 ceph-mon[20004]: from='osd.1 [v2:172.21.15.31:6810/3236186806,v1:172.21.15.31:6811/3236186806]' entity='osd.1' 2024-08-19T14:07:52.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:51 smithi045 ceph-mon[20143]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-19T14:07:52.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:51 smithi045 ceph-mon[20143]: from='osd.1 [v2:172.21.15.31:6810/3236186806,v1:172.21.15.31:6811/3236186806]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2024-08-19T14:07:52.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:51 smithi045 ceph-mon[20143]: osdmap e12: 2 total, 1 up, 2 in 2024-08-19T14:07:52.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:51 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-08-19T14:07:52.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:51 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-08-19T14:07:52.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:51 smithi045 ceph-mon[20143]: from='osd.1 [v2:172.21.15.31:6810/3236186806,v1:172.21.15.31:6811/3236186806]' entity='osd.1' 2024-08-19T14:07:52.693 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:52 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:07:52.693 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:52 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:07:52.694 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:52 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-08-19T14:07:52.694 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:52 smithi031 ceph-mon[16336]: osd.1 [v2:172.21.15.31:6810/3236186806,v1:172.21.15.31:6811/3236186806] boot 2024-08-19T14:07:52.694 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:52 smithi031 ceph-mon[16336]: osdmap e13: 2 total, 2 up, 2 in 2024-08-19T14:07:52.694 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:52 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-08-19T14:07:53.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:52 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:07:53.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:52 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:07:53.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:52 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-08-19T14:07:53.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:52 smithi032 ceph-mon[20004]: osd.1 [v2:172.21.15.31:6810/3236186806,v1:172.21.15.31:6811/3236186806] boot 2024-08-19T14:07:53.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:52 smithi032 ceph-mon[20004]: osdmap e13: 2 total, 2 up, 2 in 2024-08-19T14:07:53.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:52 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-08-19T14:07:53.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:52 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:07:53.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:52 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:07:53.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:52 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-08-19T14:07:53.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:52 smithi045 ceph-mon[20143]: osd.1 [v2:172.21.15.31:6810/3236186806,v1:172.21.15.31:6811/3236186806] boot 2024-08-19T14:07:53.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:52 smithi045 ceph-mon[20143]: osdmap e13: 2 total, 2 up, 2 in 2024-08-19T14:07:53.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:52 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-08-19T14:07:53.814 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:53 smithi031 ceph-mon[16336]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-19T14:07:53.814 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:53 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:53.814 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:53 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:53.814 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:53 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:07:53.814 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:53 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:07:53.814 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:53 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:53.814 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:53 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:07:53.814 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:53 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:07:53.815 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:53 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:53.908 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:53 smithi032 ceph-mon[20004]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-19T14:07:53.909 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:53 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:53.909 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:53 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:53.909 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:53 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:07:53.909 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:53 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:07:53.909 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:53 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:53.909 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:53 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:07:53.909 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:53 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:07:53.909 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:53 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:53.947 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:53 smithi045 ceph-mon[20143]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-19T14:07:53.947 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:53 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:53.947 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:53 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:53.948 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:53 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:07:53.948 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:53 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:07:53.948 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:53 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:53.948 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:53 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:07:53.948 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:53 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:07:53.948 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:53 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:07:54.379 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:07:54.404 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph orch daemon add osd smithi031:vg_nvme/lv_2 2024-08-19T14:07:54.658 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:07:54.769 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:54 smithi031 ceph-mon[16336]: Detected new or changed devices on smithi031 2024-08-19T14:07:54.769 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:54 smithi031 ceph-mon[16336]: Adjusting osd_memory_target on smithi031 to 8546M 2024-08-19T14:07:54.769 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:54 smithi031 ceph-mon[16336]: osdmap e14: 2 total, 2 up, 2 in 2024-08-19T14:07:55.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:54 smithi032 ceph-mon[20004]: Detected new or changed devices on smithi031 2024-08-19T14:07:55.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:54 smithi032 ceph-mon[20004]: Adjusting osd_memory_target on smithi031 to 8546M 2024-08-19T14:07:55.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:54 smithi032 ceph-mon[20004]: osdmap e14: 2 total, 2 up, 2 in 2024-08-19T14:07:55.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:54 smithi045 ceph-mon[20143]: Detected new or changed devices on smithi031 2024-08-19T14:07:55.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:54 smithi045 ceph-mon[20143]: Adjusting osd_memory_target on smithi031 to 8546M 2024-08-19T14:07:55.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:54 smithi045 ceph-mon[20143]: osdmap e14: 2 total, 2 up, 2 in 2024-08-19T14:07:55.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:55 smithi031 ceph-mon[16336]: pgmap v54: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-19T14:07:55.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:55 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-19T14:07:55.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:55 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-19T14:07:55.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:55 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:07:56.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:55 smithi032 ceph-mon[20004]: pgmap v54: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-19T14:07:56.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:55 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-19T14:07:56.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:55 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-19T14:07:56.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:55 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:07:56.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:55 smithi045 ceph-mon[20143]: pgmap v54: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-19T14:07:56.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:55 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-19T14:07:56.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:55 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-19T14:07:56.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:55 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:07:56.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:56 smithi031 ceph-mon[16336]: from='client.14277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T14:07:57.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:56 smithi032 ceph-mon[20004]: from='client.14277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T14:07:57.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:56 smithi045 ceph-mon[20143]: from='client.14277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T14:07:57.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:57 smithi031 ceph-mon[16336]: pgmap v55: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-19T14:07:58.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:57 smithi032 ceph-mon[20004]: pgmap v55: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-19T14:07:58.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:57 smithi045 ceph-mon[20143]: pgmap v55: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-19T14:07:58.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:58 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1194329710' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "376c18d0-4768-46a0-bb60-e79c88ca909c"}]: dispatch 2024-08-19T14:07:58.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:58 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1194329710' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "376c18d0-4768-46a0-bb60-e79c88ca909c"}]': finished 2024-08-19T14:07:58.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:58 smithi031 ceph-mon[16336]: osdmap e15: 3 total, 2 up, 3 in 2024-08-19T14:07:58.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:58 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-08-19T14:07:58.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:58 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2281404885' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-19T14:07:59.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:58 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1194329710' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "376c18d0-4768-46a0-bb60-e79c88ca909c"}]: dispatch 2024-08-19T14:07:59.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:58 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1194329710' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "376c18d0-4768-46a0-bb60-e79c88ca909c"}]': finished 2024-08-19T14:07:59.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:58 smithi032 ceph-mon[20004]: osdmap e15: 3 total, 2 up, 3 in 2024-08-19T14:07:59.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:58 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-08-19T14:07:59.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:58 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2281404885' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-19T14:07:59.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:58 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1194329710' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "376c18d0-4768-46a0-bb60-e79c88ca909c"}]: dispatch 2024-08-19T14:07:59.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:58 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1194329710' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "376c18d0-4768-46a0-bb60-e79c88ca909c"}]': finished 2024-08-19T14:07:59.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:58 smithi045 ceph-mon[20143]: osdmap e15: 3 total, 2 up, 3 in 2024-08-19T14:07:59.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:58 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-08-19T14:07:59.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:58 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2281404885' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-19T14:07:59.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:07:59 smithi031 ceph-mon[16336]: pgmap v57: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-19T14:08:00.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:07:59 smithi032 ceph-mon[20004]: pgmap v57: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-19T14:08:00.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:07:59 smithi045 ceph-mon[20143]: pgmap v57: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-19T14:08:01.854 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:01 smithi031 ceph-mon[16336]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-19T14:08:02.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:01 smithi032 ceph-mon[20004]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-19T14:08:02.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:01 smithi045 ceph-mon[20143]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-19T14:08:04.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:03 smithi032 ceph-mon[20004]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-19T14:08:04.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:03 smithi045 ceph-mon[20143]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-19T14:08:04.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:03 smithi031 ceph-mon[16336]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-19T14:08:04.751 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:04 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-08-19T14:08:04.751 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:04 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:08:05.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:04 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-08-19T14:08:05.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:04 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:08:05.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:04 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-08-19T14:08:05.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:04 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:08:05.975 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:05 smithi031 ceph-mon[16336]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-19T14:08:05.975 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:05 smithi031 ceph-mon[16336]: Deploying daemon osd.2 on smithi031 2024-08-19T14:08:06.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:05 smithi032 ceph-mon[20004]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-19T14:08:06.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:05 smithi032 ceph-mon[20004]: Deploying daemon osd.2 on smithi031 2024-08-19T14:08:06.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:05 smithi045 ceph-mon[20143]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-19T14:08:06.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:05 smithi045 ceph-mon[20143]: Deploying daemon osd.2 on smithi031 2024-08-19T14:08:08.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:07 smithi032 ceph-mon[20004]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-19T14:08:08.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:07 smithi045 ceph-mon[20143]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-19T14:08:08.075 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:07 smithi031 ceph-mon[16336]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-19T14:08:10.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:09 smithi032 ceph-mon[20004]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-19T14:08:10.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:09 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:08:10.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:09 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:10.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:09 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:10.008 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:09 smithi031 ceph-mon[16336]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-19T14:08:10.008 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:09 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:08:10.008 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:09 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:10.009 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:09 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:10.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:09 smithi045 ceph-mon[20143]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-19T14:08:10.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:09 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:08:10.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:09 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:10.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:09 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:12.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:11 smithi032 ceph-mon[20004]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-19T14:08:12.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:11 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:12.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:11 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:12.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:11 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:08:12.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:11 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:08:12.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:11 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:12.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:11 smithi045 ceph-mon[20143]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-19T14:08:12.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:11 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:12.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:11 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:12.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:11 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:08:12.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:11 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:08:12.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:11 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:12.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:11 smithi031 ceph-mon[16336]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-19T14:08:12.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:11 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:12.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:11 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:12.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:11 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:08:12.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:11 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:08:12.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:11 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:12.102 INFO:teuthology.orchestra.run.smithi031.stdout:Created osd(s) 2 on host 'smithi031' 2024-08-19T14:08:12.648 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:12 smithi031 ceph-mon[16336]: from='osd.2 [v2:172.21.15.31:6818/4045553387,v1:172.21.15.31:6819/4045553387]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-08-19T14:08:12.648 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:12 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:08:12.648 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:12 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:12.648 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:12 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:12.650 DEBUG:teuthology.orchestra.run.smithi031:osd.2> sudo journalctl -f -n 0 -u ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968@osd.2.service 2024-08-19T14:08:12.652 INFO:tasks.cephadm:Deploying osd.3 on smithi031 with /dev/vg_nvme/lv_1... 2024-08-19T14:08:12.652 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- lvm zap /dev/vg_nvme/lv_1 2024-08-19T14:08:13.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:12 smithi032 ceph-mon[20004]: from='osd.2 [v2:172.21.15.31:6818/4045553387,v1:172.21.15.31:6819/4045553387]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-08-19T14:08:13.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:12 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:08:13.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:12 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:13.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:12 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:13.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:12 smithi045 ceph-mon[20143]: from='osd.2 [v2:172.21.15.31:6818/4045553387,v1:172.21.15.31:6819/4045553387]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-08-19T14:08:13.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:12 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:08:13.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:12 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:13.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:12 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:13.961 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:13 smithi031 ceph-mon[16336]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-19T14:08:13.961 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:13 smithi031 ceph-mon[16336]: from='osd.2 [v2:172.21.15.31:6818/4045553387,v1:172.21.15.31:6819/4045553387]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-08-19T14:08:13.961 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:13 smithi031 ceph-mon[16336]: osdmap e16: 3 total, 2 up, 3 in 2024-08-19T14:08:13.961 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:13 smithi031 ceph-mon[16336]: from='osd.2 [v2:172.21.15.31:6818/4045553387,v1:172.21.15.31:6819/4045553387]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-08-19T14:08:13.961 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:13 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-08-19T14:08:13.962 INFO:journalctl@ceph.osd.2.smithi031.stdout:Aug 19 14:08:13 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-2[41419]: 2024-08-19T14:08:13.646+0000 7f9f490fe640 -1 osd.2 0 waiting for initial osdmap 2024-08-19T14:08:13.962 INFO:journalctl@ceph.osd.2.smithi031.stdout:Aug 19 14:08:13 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-2[41419]: 2024-08-19T14:08:13.650+0000 7f9f44712640 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-19T14:08:14.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:13 smithi032 ceph-mon[20004]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-19T14:08:14.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:13 smithi032 ceph-mon[20004]: from='osd.2 [v2:172.21.15.31:6818/4045553387,v1:172.21.15.31:6819/4045553387]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-08-19T14:08:14.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:13 smithi032 ceph-mon[20004]: osdmap e16: 3 total, 2 up, 3 in 2024-08-19T14:08:14.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:13 smithi032 ceph-mon[20004]: from='osd.2 [v2:172.21.15.31:6818/4045553387,v1:172.21.15.31:6819/4045553387]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-08-19T14:08:14.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:13 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-08-19T14:08:14.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:13 smithi045 ceph-mon[20143]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-19T14:08:14.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:13 smithi045 ceph-mon[20143]: from='osd.2 [v2:172.21.15.31:6818/4045553387,v1:172.21.15.31:6819/4045553387]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-08-19T14:08:14.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:13 smithi045 ceph-mon[20143]: osdmap e16: 3 total, 2 up, 3 in 2024-08-19T14:08:14.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:13 smithi045 ceph-mon[20143]: from='osd.2 [v2:172.21.15.31:6818/4045553387,v1:172.21.15.31:6819/4045553387]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-08-19T14:08:14.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:13 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-08-19T14:08:14.151 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:08:14.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:14 smithi031 ceph-mon[16336]: from='osd.2 [v2:172.21.15.31:6818/4045553387,v1:172.21.15.31:6819/4045553387]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2024-08-19T14:08:14.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:14 smithi031 ceph-mon[16336]: osdmap e17: 3 total, 2 up, 3 in 2024-08-19T14:08:14.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:14 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-08-19T14:08:14.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:14 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-08-19T14:08:15.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:14 smithi032 ceph-mon[20004]: from='osd.2 [v2:172.21.15.31:6818/4045553387,v1:172.21.15.31:6819/4045553387]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2024-08-19T14:08:15.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:14 smithi032 ceph-mon[20004]: osdmap e17: 3 total, 2 up, 3 in 2024-08-19T14:08:15.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:14 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-08-19T14:08:15.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:14 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-08-19T14:08:15.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:14 smithi045 ceph-mon[20143]: from='osd.2 [v2:172.21.15.31:6818/4045553387,v1:172.21.15.31:6819/4045553387]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2024-08-19T14:08:15.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:14 smithi045 ceph-mon[20143]: osdmap e17: 3 total, 2 up, 3 in 2024-08-19T14:08:15.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:14 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-08-19T14:08:15.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:14 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-08-19T14:08:15.945 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:15 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:08:15.945 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:15 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:08:15.945 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:15 smithi031 ceph-mon[16336]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-19T14:08:15.945 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:15 smithi031 ceph-mon[16336]: osd.2 [v2:172.21.15.31:6818/4045553387,v1:172.21.15.31:6819/4045553387] boot 2024-08-19T14:08:15.945 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:15 smithi031 ceph-mon[16336]: osdmap e18: 3 total, 3 up, 3 in 2024-08-19T14:08:15.945 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:15 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-08-19T14:08:15.945 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:15 smithi031 ceph-mon[16336]: Detected new or changed devices on smithi031 2024-08-19T14:08:15.945 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:15 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:15.945 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:15 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:15.945 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:15 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:08:15.946 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:15 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:08:15.946 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:15 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:08:15.946 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:15 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:08:15.946 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:15 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:08:15.946 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:15 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:08:15.946 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:15 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:08:15.946 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:15 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:15.946 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:15 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:08:15.946 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:15 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:08:15.946 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:15 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:15.946 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:15 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:08:15.947 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:15 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:08:15.947 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:15 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:08:15.947 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:15 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:16.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:15 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:08:16.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:15 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:08:16.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:15 smithi032 ceph-mon[20004]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-19T14:08:16.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:15 smithi032 ceph-mon[20004]: osd.2 [v2:172.21.15.31:6818/4045553387,v1:172.21.15.31:6819/4045553387] boot 2024-08-19T14:08:16.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:15 smithi032 ceph-mon[20004]: osdmap e18: 3 total, 3 up, 3 in 2024-08-19T14:08:16.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:15 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-08-19T14:08:16.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:15 smithi032 ceph-mon[20004]: Detected new or changed devices on smithi031 2024-08-19T14:08:16.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:15 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:16.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:15 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:16.009 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:15 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:08:16.009 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:15 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:08:16.009 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:15 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:08:16.009 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:15 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:08:16.009 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:15 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:08:16.009 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:15 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:08:16.009 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:15 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:08:16.009 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:15 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:16.009 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:15 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:08:16.009 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:15 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:08:16.009 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:15 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:16.009 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:15 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:08:16.009 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:15 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:08:16.010 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:15 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:08:16.010 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:15 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:16.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:15 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:08:16.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:15 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:08:16.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:15 smithi045 ceph-mon[20143]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-19T14:08:16.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:15 smithi045 ceph-mon[20143]: osd.2 [v2:172.21.15.31:6818/4045553387,v1:172.21.15.31:6819/4045553387] boot 2024-08-19T14:08:16.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:15 smithi045 ceph-mon[20143]: osdmap e18: 3 total, 3 up, 3 in 2024-08-19T14:08:16.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:15 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-08-19T14:08:16.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:15 smithi045 ceph-mon[20143]: Detected new or changed devices on smithi031 2024-08-19T14:08:16.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:15 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:16.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:15 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:16.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:15 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:08:16.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:15 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:08:16.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:15 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:08:16.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:15 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:08:16.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:15 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:08:16.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:15 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:08:16.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:15 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:08:16.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:15 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:16.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:15 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:08:16.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:15 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:08:16.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:15 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:16.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:15 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:08:16.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:15 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:08:16.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:15 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:08:16.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:15 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:16.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:16 smithi031 ceph-mon[16336]: Adjusting osd_memory_target on smithi031 to 5697M 2024-08-19T14:08:16.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:16 smithi031 ceph-mon[16336]: osdmap e19: 3 total, 3 up, 3 in 2024-08-19T14:08:16.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:16 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-08-19T14:08:17.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:16 smithi032 ceph-mon[20004]: Adjusting osd_memory_target on smithi031 to 5697M 2024-08-19T14:08:17.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:16 smithi032 ceph-mon[20004]: osdmap e19: 3 total, 3 up, 3 in 2024-08-19T14:08:17.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:16 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-08-19T14:08:17.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:16 smithi045 ceph-mon[20143]: Adjusting osd_memory_target on smithi031 to 5697M 2024-08-19T14:08:17.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:16 smithi045 ceph-mon[20143]: osdmap e19: 3 total, 3 up, 3 in 2024-08-19T14:08:17.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:16 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-08-19T14:08:17.244 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:08:17.270 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph orch daemon add osd smithi031:vg_nvme/lv_1 2024-08-19T14:08:17.524 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:08:17.895 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:17 smithi031 ceph-mon[16336]: pgmap v70: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-08-19T14:08:17.896 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:17 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-08-19T14:08:17.896 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:17 smithi031 ceph-mon[16336]: osdmap e20: 3 total, 3 up, 3 in 2024-08-19T14:08:17.896 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:17 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-08-19T14:08:18.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:17 smithi032 ceph-mon[20004]: pgmap v70: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-08-19T14:08:18.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:17 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-08-19T14:08:18.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:17 smithi032 ceph-mon[20004]: osdmap e20: 3 total, 3 up, 3 in 2024-08-19T14:08:18.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:17 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-08-19T14:08:18.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:17 smithi045 ceph-mon[20143]: pgmap v70: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-08-19T14:08:18.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:17 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-08-19T14:08:18.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:17 smithi045 ceph-mon[20143]: osdmap e20: 3 total, 3 up, 3 in 2024-08-19T14:08:18.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:17 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-08-19T14:08:18.467 INFO:journalctl@ceph.osd.0.smithi031.stdout:Aug 19 14:08:18 smithi031 sudo[45560]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-08-19T14:08:18.467 INFO:journalctl@ceph.osd.0.smithi031.stdout:Aug 19 14:08:18 smithi031 sudo[45560]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-19T14:08:18.467 INFO:journalctl@ceph.osd.0.smithi031.stdout:Aug 19 14:08:18 smithi031 sudo[45560]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-19T14:08:18.467 INFO:journalctl@ceph.osd.0.smithi031.stdout:Aug 19 14:08:18 smithi031 sudo[45560]: pam_unix(sudo:session): session closed for user root 2024-08-19T14:08:18.749 INFO:journalctl@ceph.osd.0.smithi031.stdout:Aug 19 14:08:18 smithi031 sudo[45662]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-08-19T14:08:18.749 INFO:journalctl@ceph.osd.0.smithi031.stdout:Aug 19 14:08:18 smithi031 sudo[45662]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-19T14:08:18.750 INFO:journalctl@ceph.osd.0.smithi031.stdout:Aug 19 14:08:18 smithi031 sudo[45662]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-19T14:08:18.750 INFO:journalctl@ceph.osd.0.smithi031.stdout:Aug 19 14:08:18 smithi031 sudo[45662]: pam_unix(sudo:session): session closed for user root 2024-08-19T14:08:18.750 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:18 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-08-19T14:08:18.750 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:18 smithi031 ceph-mon[16336]: osdmap e21: 3 total, 3 up, 3 in 2024-08-19T14:08:18.750 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:18 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-19T14:08:18.750 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:18 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-19T14:08:18.750 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:18 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:08:18.750 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:18 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:19.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:18 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-08-19T14:08:19.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:18 smithi032 ceph-mon[20004]: osdmap e21: 3 total, 3 up, 3 in 2024-08-19T14:08:19.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:18 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-19T14:08:19.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:18 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-19T14:08:19.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:18 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:08:19.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:18 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:19.014 INFO:journalctl@ceph.osd.1.smithi031.stdout:Aug 19 14:08:18 smithi031 sudo[45686]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-08-19T14:08:19.014 INFO:journalctl@ceph.osd.1.smithi031.stdout:Aug 19 14:08:18 smithi031 sudo[45686]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-19T14:08:19.015 INFO:journalctl@ceph.osd.1.smithi031.stdout:Aug 19 14:08:18 smithi031 sudo[45686]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-19T14:08:19.015 INFO:journalctl@ceph.osd.1.smithi031.stdout:Aug 19 14:08:18 smithi031 sudo[45686]: pam_unix(sudo:session): session closed for user root 2024-08-19T14:08:19.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:18 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-08-19T14:08:19.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:18 smithi045 ceph-mon[20143]: osdmap e21: 3 total, 3 up, 3 in 2024-08-19T14:08:19.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:18 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-19T14:08:19.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:18 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-19T14:08:19.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:18 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:08:19.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:18 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:19.277 INFO:journalctl@ceph.osd.1.smithi031.stdout:Aug 19 14:08:19 smithi031 sudo[45706]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-08-19T14:08:19.277 INFO:journalctl@ceph.osd.1.smithi031.stdout:Aug 19 14:08:19 smithi031 sudo[45706]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-19T14:08:19.277 INFO:journalctl@ceph.osd.1.smithi031.stdout:Aug 19 14:08:19 smithi031 sudo[45706]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-19T14:08:19.277 INFO:journalctl@ceph.osd.1.smithi031.stdout:Aug 19 14:08:19 smithi031 sudo[45706]: pam_unix(sudo:session): session closed for user root 2024-08-19T14:08:19.545 INFO:journalctl@ceph.osd.2.smithi031.stdout:Aug 19 14:08:19 smithi031 sudo[45709]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-08-19T14:08:19.546 INFO:journalctl@ceph.osd.2.smithi031.stdout:Aug 19 14:08:19 smithi031 sudo[45709]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-19T14:08:19.546 INFO:journalctl@ceph.osd.2.smithi031.stdout:Aug 19 14:08:19 smithi031 sudo[45709]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-19T14:08:19.546 INFO:journalctl@ceph.osd.2.smithi031.stdout:Aug 19 14:08:19 smithi031 sudo[45709]: pam_unix(sudo:session): session closed for user root 2024-08-19T14:08:19.806 INFO:journalctl@ceph.osd.2.smithi031.stdout:Aug 19 14:08:19 smithi031 sudo[45717]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-08-19T14:08:19.806 INFO:journalctl@ceph.osd.2.smithi031.stdout:Aug 19 14:08:19 smithi031 sudo[45717]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-19T14:08:19.806 INFO:journalctl@ceph.osd.2.smithi031.stdout:Aug 19 14:08:19 smithi031 sudo[45717]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-19T14:08:19.806 INFO:journalctl@ceph.osd.2.smithi031.stdout:Aug 19 14:08:19 smithi031 sudo[45717]: pam_unix(sudo:session): session closed for user root 2024-08-19T14:08:20.063 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:19 smithi031 sudo[45741]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-08-19T14:08:20.063 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:19 smithi031 sudo[45741]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-19T14:08:20.063 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:19 smithi031 sudo[45741]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-19T14:08:20.064 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:19 smithi031 sudo[45741]: pam_unix(sudo:session): session closed for user root 2024-08-19T14:08:20.347 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:20 smithi045 ceph-mon[20143]: pgmap v73: 1 pgs: 1 creating+peering; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-08-19T14:08:20.348 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:20 smithi045 ceph-mon[20143]: from='client.14310 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T14:08:20.348 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:20 smithi045 ceph-mon[20143]: osdmap e22: 3 total, 3 up, 3 in 2024-08-19T14:08:20.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:20 smithi031 sudo[45751]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-08-19T14:08:20.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:20 smithi031 sudo[45751]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-19T14:08:20.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:20 smithi031 sudo[45751]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-19T14:08:20.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:20 smithi031 sudo[45751]: pam_unix(sudo:session): session closed for user root 2024-08-19T14:08:20.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:20 smithi031 ceph-mon[16336]: pgmap v73: 1 pgs: 1 creating+peering; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-08-19T14:08:20.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:20 smithi031 ceph-mon[16336]: from='client.14310 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T14:08:20.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:20 smithi031 ceph-mon[16336]: osdmap e22: 3 total, 3 up, 3 in 2024-08-19T14:08:20.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:20 smithi032 ceph-mon[20004]: pgmap v73: 1 pgs: 1 creating+peering; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-08-19T14:08:20.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:20 smithi032 ceph-mon[20004]: from='client.14310 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T14:08:20.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:20 smithi032 ceph-mon[20004]: osdmap e22: 3 total, 3 up, 3 in 2024-08-19T14:08:20.615 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:20 smithi045 sudo[22101]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-08-19T14:08:20.615 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:20 smithi045 sudo[22101]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-19T14:08:20.615 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:20 smithi045 sudo[22101]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-19T14:08:20.615 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:20 smithi045 sudo[22101]: pam_unix(sudo:session): session closed for user root 2024-08-19T14:08:21.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:20 smithi045 sudo[22104]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-08-19T14:08:21.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:20 smithi045 sudo[22104]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-19T14:08:21.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:20 smithi045 sudo[22104]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-19T14:08:21.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:20 smithi045 sudo[22104]: pam_unix(sudo:session): session closed for user root 2024-08-19T14:08:21.164 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:20 smithi032 sudo[22052]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-08-19T14:08:21.165 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:20 smithi032 sudo[22052]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-19T14:08:21.165 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:20 smithi032 sudo[22052]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-19T14:08:21.165 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:20 smithi032 sudo[22052]: pam_unix(sudo:session): session closed for user root 2024-08-19T14:08:21.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:21 smithi032 sudo[22055]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-08-19T14:08:21.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:21 smithi032 sudo[22055]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-19T14:08:21.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:21 smithi032 sudo[22055]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-19T14:08:21.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:21 smithi032 sudo[22055]: pam_unix(sudo:session): session closed for user root 2024-08-19T14:08:21.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:21 smithi032 ceph-mon[20004]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-08-19T14:08:21.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:21 smithi032 ceph-mon[20004]: pgmap v75: 1 pgs: 1 creating+peering; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-08-19T14:08:21.509 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:21 smithi032 ceph-mon[20004]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-08-19T14:08:21.509 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:21 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:08:21.509 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:21 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:08:21.510 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:21 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:08:21.510 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:21 smithi032 ceph-mon[20004]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-08-19T14:08:21.510 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:21 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:08:21.510 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:21 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:08:21.510 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:21 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:08:21.510 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:21 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/883891663' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a787ea9f-b9ba-450f-b388-614cfba90634"}]: dispatch 2024-08-19T14:08:21.510 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:21 smithi032 ceph-mon[20004]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-08-19T14:08:21.510 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:21 smithi032 ceph-mon[20004]: mgrmap e15: y(active, since 2m), standbys: x 2024-08-19T14:08:21.510 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:21 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/883891663' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a787ea9f-b9ba-450f-b388-614cfba90634"}]': finished 2024-08-19T14:08:21.510 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:21 smithi032 ceph-mon[20004]: osdmap e23: 4 total, 3 up, 4 in 2024-08-19T14:08:21.510 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:21 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-08-19T14:08:21.511 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:21 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:21.511 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:21 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:08:21.511 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:21 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:08:21.511 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:21 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:08:21.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:21 smithi045 ceph-mon[20143]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-08-19T14:08:21.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:21 smithi045 ceph-mon[20143]: pgmap v75: 1 pgs: 1 creating+peering; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-08-19T14:08:21.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:21 smithi045 ceph-mon[20143]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-08-19T14:08:21.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:21 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:08:21.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:21 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:08:21.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:21 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:08:21.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:21 smithi045 ceph-mon[20143]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-08-19T14:08:21.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:21 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:08:21.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:21 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:08:21.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:21 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:08:21.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:21 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/883891663' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a787ea9f-b9ba-450f-b388-614cfba90634"}]: dispatch 2024-08-19T14:08:21.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:21 smithi045 ceph-mon[20143]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-08-19T14:08:21.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:21 smithi045 ceph-mon[20143]: mgrmap e15: y(active, since 2m), standbys: x 2024-08-19T14:08:21.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:21 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/883891663' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a787ea9f-b9ba-450f-b388-614cfba90634"}]': finished 2024-08-19T14:08:21.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:21 smithi045 ceph-mon[20143]: osdmap e23: 4 total, 3 up, 4 in 2024-08-19T14:08:21.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:21 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-08-19T14:08:21.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:21 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:21.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:21 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:08:21.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:21 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:08:21.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:21 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:08:21.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:21 smithi031 ceph-mon[16336]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-08-19T14:08:21.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:21 smithi031 ceph-mon[16336]: pgmap v75: 1 pgs: 1 creating+peering; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-08-19T14:08:21.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:21 smithi031 ceph-mon[16336]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-08-19T14:08:21.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:21 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:08:21.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:21 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:08:21.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:21 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:08:21.618 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:21 smithi031 ceph-mon[16336]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-08-19T14:08:21.618 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:21 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:08:21.618 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:21 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:08:21.618 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:21 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:08:21.618 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:21 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/883891663' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a787ea9f-b9ba-450f-b388-614cfba90634"}]: dispatch 2024-08-19T14:08:21.618 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:21 smithi031 ceph-mon[16336]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-08-19T14:08:21.618 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:21 smithi031 ceph-mon[16336]: mgrmap e15: y(active, since 2m), standbys: x 2024-08-19T14:08:21.618 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:21 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/883891663' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a787ea9f-b9ba-450f-b388-614cfba90634"}]': finished 2024-08-19T14:08:21.618 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:21 smithi031 ceph-mon[16336]: osdmap e23: 4 total, 3 up, 4 in 2024-08-19T14:08:21.618 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:21 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-08-19T14:08:21.618 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:21 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:21.618 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:21 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:08:21.618 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:21 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:08:21.618 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:21 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:08:22.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:22 smithi032 ceph-mon[20004]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-08-19T14:08:22.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:22 smithi032 ceph-mon[20004]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-08-19T14:08:22.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:22 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:22.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:22 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2684851884' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-19T14:08:22.518 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:22 smithi031 ceph-mon[16336]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-08-19T14:08:22.518 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:22 smithi031 ceph-mon[16336]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-08-19T14:08:22.518 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:22 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:22.518 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:22 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2684851884' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-19T14:08:22.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:22 smithi045 ceph-mon[20143]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-08-19T14:08:22.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:22 smithi045 ceph-mon[20143]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-08-19T14:08:22.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:22 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:22.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:22 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2684851884' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-19T14:08:23.447 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:23 smithi045 ceph-mon[20143]: pgmap v77: 1 pgs: 1 creating+peering; 0 B data, 81 MiB used, 268 GiB / 268 GiB avail 2024-08-19T14:08:23.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:23 smithi031 ceph-mon[16336]: pgmap v77: 1 pgs: 1 creating+peering; 0 B data, 81 MiB used, 268 GiB / 268 GiB avail 2024-08-19T14:08:23.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:23 smithi032 ceph-mon[20004]: pgmap v77: 1 pgs: 1 creating+peering; 0 B data, 81 MiB used, 268 GiB / 268 GiB avail 2024-08-19T14:08:25.459 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:25 smithi031 ceph-mon[16336]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-08-19T14:08:25.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:25 smithi032 ceph-mon[20004]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-08-19T14:08:25.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:25 smithi045 ceph-mon[20143]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-08-19T14:08:27.411 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:27 smithi031 ceph-mon[16336]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-08-19T14:08:27.411 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:27 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-08-19T14:08:27.411 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:27 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:08:27.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:27 smithi032 ceph-mon[20004]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-08-19T14:08:27.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:27 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-08-19T14:08:27.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:27 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:08:27.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:27 smithi045 ceph-mon[20143]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-08-19T14:08:27.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:27 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-08-19T14:08:27.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:27 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:08:28.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:28 smithi032 ceph-mon[20004]: Deploying daemon osd.3 on smithi031 2024-08-19T14:08:28.542 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:28 smithi031 ceph-mon[16336]: Deploying daemon osd.3 on smithi031 2024-08-19T14:08:28.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:28 smithi045 ceph-mon[20143]: Deploying daemon osd.3 on smithi031 2024-08-19T14:08:29.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:29 smithi031 ceph-mon[16336]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-08-19T14:08:29.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:29 smithi032 ceph-mon[20004]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-08-19T14:08:29.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:29 smithi045 ceph-mon[20143]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-08-19T14:08:31.313 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:31 smithi031 ceph-mon[16336]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-08-19T14:08:31.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:31 smithi032 ceph-mon[20004]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-08-19T14:08:31.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:31 smithi045 ceph-mon[20143]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-08-19T14:08:32.327 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:32 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:08:32.327 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:32 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:32.328 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:32 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:32.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:32 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:08:32.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:32 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:32.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:32 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:32.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:32 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:08:32.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:32 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:32.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:32 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:33.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:33 smithi031 ceph-mon[16336]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-08-19T14:08:34.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:33 smithi032 ceph-mon[20004]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-08-19T14:08:34.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:33 smithi045 ceph-mon[20143]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-08-19T14:08:35.155 INFO:teuthology.orchestra.run.smithi031.stdout:Created osd(s) 3 on host 'smithi031' 2024-08-19T14:08:35.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:35 smithi032 ceph-mon[20004]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-08-19T14:08:35.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:35 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:35.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:35 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:35.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:35 smithi032 ceph-mon[20004]: from='osd.3 [v2:172.21.15.31:6826/2288082716,v1:172.21.15.31:6827/2288082716]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-08-19T14:08:35.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:35 smithi032 ceph-mon[20004]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-08-19T14:08:35.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:35 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:08:35.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:35 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:08:35.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:35 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:35.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:35 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:08:35.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:35 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:35.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:35 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:35.763 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:35 smithi031 ceph-mon[16336]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-08-19T14:08:35.763 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:35 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:35.763 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:35 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:35.763 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:35 smithi031 ceph-mon[16336]: from='osd.3 [v2:172.21.15.31:6826/2288082716,v1:172.21.15.31:6827/2288082716]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-08-19T14:08:35.763 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:35 smithi031 ceph-mon[16336]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-08-19T14:08:35.763 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:35 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:08:35.763 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:35 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:08:35.763 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:35 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:35.763 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:35 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:08:35.763 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:35 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:35.763 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:35 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:35.790 DEBUG:teuthology.orchestra.run.smithi031:osd.3> sudo journalctl -f -n 0 -u ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968@osd.3.service 2024-08-19T14:08:35.792 INFO:tasks.cephadm:Deploying osd.4 on smithi032 with /dev/vg_nvme/lv_4... 2024-08-19T14:08:35.793 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- lvm zap /dev/vg_nvme/lv_4 2024-08-19T14:08:35.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:35 smithi045 ceph-mon[20143]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-08-19T14:08:35.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:35 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:35.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:35 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:35.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:35 smithi045 ceph-mon[20143]: from='osd.3 [v2:172.21.15.31:6826/2288082716,v1:172.21.15.31:6827/2288082716]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-08-19T14:08:35.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:35 smithi045 ceph-mon[20143]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-08-19T14:08:35.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:35 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:08:35.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:35 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:08:35.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:35 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:35.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:35 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:08:35.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:35 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:35.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:35 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:36.043 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:08:36.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:36 smithi032 ceph-mon[20004]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-08-19T14:08:36.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:36 smithi032 ceph-mon[20004]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-08-19T14:08:36.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:36 smithi032 ceph-mon[20004]: osdmap e24: 4 total, 3 up, 4 in 2024-08-19T14:08:36.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:36 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-08-19T14:08:36.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:36 smithi032 ceph-mon[20004]: from='osd.3 [v2:172.21.15.31:6826/2288082716,v1:172.21.15.31:6827/2288082716]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-08-19T14:08:36.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:36 smithi032 ceph-mon[20004]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-08-19T14:08:36.952 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:36 smithi031 ceph-mon[16336]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-08-19T14:08:36.952 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:36 smithi031 ceph-mon[16336]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-08-19T14:08:36.952 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:36 smithi031 ceph-mon[16336]: osdmap e24: 4 total, 3 up, 4 in 2024-08-19T14:08:36.952 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:36 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-08-19T14:08:36.953 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:36 smithi031 ceph-mon[16336]: from='osd.3 [v2:172.21.15.31:6826/2288082716,v1:172.21.15.31:6827/2288082716]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-08-19T14:08:36.953 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:36 smithi031 ceph-mon[16336]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-08-19T14:08:37.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:36 smithi045 ceph-mon[20143]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-08-19T14:08:37.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:36 smithi045 ceph-mon[20143]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-08-19T14:08:37.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:36 smithi045 ceph-mon[20143]: osdmap e24: 4 total, 3 up, 4 in 2024-08-19T14:08:37.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:36 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-08-19T14:08:37.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:36 smithi045 ceph-mon[20143]: from='osd.3 [v2:172.21.15.31:6826/2288082716,v1:172.21.15.31:6827/2288082716]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-08-19T14:08:37.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:36 smithi045 ceph-mon[20143]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-08-19T14:08:37.251 INFO:journalctl@ceph.osd.3.smithi031.stdout:Aug 19 14:08:37 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-3[47445]: 2024-08-19T14:08:37.121+0000 7fd36dc32640 -1 osd.3 0 waiting for initial osdmap 2024-08-19T14:08:37.251 INFO:journalctl@ceph.osd.3.smithi031.stdout:Aug 19 14:08:37 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-3[47445]: 2024-08-19T14:08:37.124+0000 7fd365a52640 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-19T14:08:37.820 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:37 smithi032 ceph-mon[20004]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2024-08-19T14:08:37.820 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:37 smithi032 ceph-mon[20004]: osdmap e25: 4 total, 3 up, 4 in 2024-08-19T14:08:37.821 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:37 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-08-19T14:08:37.821 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:37 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-08-19T14:08:37.821 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:37 smithi032 ceph-mon[20004]: from='osd.3 ' entity='osd.3' 2024-08-19T14:08:37.965 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:37 smithi031 ceph-mon[16336]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2024-08-19T14:08:37.965 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:37 smithi031 ceph-mon[16336]: osdmap e25: 4 total, 3 up, 4 in 2024-08-19T14:08:37.965 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:37 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-08-19T14:08:37.965 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:37 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-08-19T14:08:37.965 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:37 smithi031 ceph-mon[16336]: from='osd.3 ' entity='osd.3' 2024-08-19T14:08:38.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:37 smithi045 ceph-mon[20143]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2024-08-19T14:08:38.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:37 smithi045 ceph-mon[20143]: osdmap e25: 4 total, 3 up, 4 in 2024-08-19T14:08:38.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:37 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-08-19T14:08:38.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:37 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-08-19T14:08:38.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:37 smithi045 ceph-mon[20143]: from='osd.3 ' entity='osd.3' 2024-08-19T14:08:38.408 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-19T14:08:38.434 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph orch daemon add osd smithi032:vg_nvme/lv_4 2024-08-19T14:08:38.692 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:08:38.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:38 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:08:38.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:38 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:08:38.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:38 smithi031 ceph-mon[16336]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-08-19T14:08:38.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:38 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-08-19T14:08:38.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:38 smithi031 ceph-mon[16336]: osd.3 [v2:172.21.15.31:6826/2288082716,v1:172.21.15.31:6827/2288082716] boot 2024-08-19T14:08:38.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:38 smithi031 ceph-mon[16336]: osdmap e26: 4 total, 4 up, 4 in 2024-08-19T14:08:38.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:38 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-08-19T14:08:38.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:38 smithi031 ceph-mon[16336]: Detected new or changed devices on smithi031 2024-08-19T14:08:38.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:38 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:38.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:38 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:38.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:38 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:08:38.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:38 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:08:38.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:38 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:08:38.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:38 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:08:38.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:38 smithi031 ceph-mon[16336]: Adjusting osd_memory_target on smithi031 to 4273M 2024-08-19T14:08:38.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:38 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:38.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:38 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:08:38.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:38 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:08:38.868 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:38 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:38.962 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:38 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:08:38.962 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:38 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:08:38.962 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:38 smithi032 ceph-mon[20004]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-08-19T14:08:38.963 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:38 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-08-19T14:08:38.963 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:38 smithi032 ceph-mon[20004]: osd.3 [v2:172.21.15.31:6826/2288082716,v1:172.21.15.31:6827/2288082716] boot 2024-08-19T14:08:38.963 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:38 smithi032 ceph-mon[20004]: osdmap e26: 4 total, 4 up, 4 in 2024-08-19T14:08:38.963 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:38 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-08-19T14:08:38.963 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:38 smithi032 ceph-mon[20004]: Detected new or changed devices on smithi031 2024-08-19T14:08:38.963 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:38 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:38.963 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:38 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:38.963 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:38 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:08:38.963 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:38 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:08:38.963 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:38 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:08:38.963 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:38 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:08:38.963 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:38 smithi032 ceph-mon[20004]: Adjusting osd_memory_target on smithi031 to 4273M 2024-08-19T14:08:38.963 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:38 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:38.964 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:38 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:08:38.964 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:38 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:08:38.964 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:38 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:39.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:38 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:08:39.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:38 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:08:39.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:38 smithi045 ceph-mon[20143]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-08-19T14:08:39.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:38 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-08-19T14:08:39.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:38 smithi045 ceph-mon[20143]: osd.3 [v2:172.21.15.31:6826/2288082716,v1:172.21.15.31:6827/2288082716] boot 2024-08-19T14:08:39.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:38 smithi045 ceph-mon[20143]: osdmap e26: 4 total, 4 up, 4 in 2024-08-19T14:08:39.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:38 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-08-19T14:08:39.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:38 smithi045 ceph-mon[20143]: Detected new or changed devices on smithi031 2024-08-19T14:08:39.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:38 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:39.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:38 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:39.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:38 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:08:39.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:38 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:08:39.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:38 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:08:39.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:38 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:08:39.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:38 smithi045 ceph-mon[20143]: Adjusting osd_memory_target on smithi031 to 4273M 2024-08-19T14:08:39.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:38 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:39.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:38 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:08:39.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:38 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:08:39.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:38 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:39.916 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:39 smithi032 ceph-mon[20004]: from='client.24215 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T14:08:39.916 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:39 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-19T14:08:39.916 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:39 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-19T14:08:39.916 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:39 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:08:40.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:39 smithi045 ceph-mon[20143]: from='client.24215 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T14:08:40.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:39 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-19T14:08:40.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:39 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-19T14:08:40.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:39 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:08:40.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:39 smithi031 ceph-mon[16336]: from='client.24215 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T14:08:40.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:39 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-19T14:08:40.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:39 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-19T14:08:40.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:39 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:08:40.984 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:40 smithi032 ceph-mon[20004]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 113 MiB used, 357 GiB / 358 GiB avail 2024-08-19T14:08:40.984 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:40 smithi032 ceph-mon[20004]: osdmap e27: 4 total, 4 up, 4 in 2024-08-19T14:08:41.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:40 smithi045 ceph-mon[20143]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 113 MiB used, 357 GiB / 358 GiB avail 2024-08-19T14:08:41.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:40 smithi045 ceph-mon[20143]: osdmap e27: 4 total, 4 up, 4 in 2024-08-19T14:08:41.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:40 smithi031 ceph-mon[16336]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 113 MiB used, 357 GiB / 358 GiB avail 2024-08-19T14:08:41.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:40 smithi031 ceph-mon[16336]: osdmap e27: 4 total, 4 up, 4 in 2024-08-19T14:08:42.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:41 smithi032 ceph-mon[20004]: from='client.? 172.21.15.32:0/2003508004' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7859751a-12f5-4a57-a332-435360479d53"}]: dispatch 2024-08-19T14:08:42.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:41 smithi032 ceph-mon[20004]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7859751a-12f5-4a57-a332-435360479d53"}]: dispatch 2024-08-19T14:08:42.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:41 smithi032 ceph-mon[20004]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7859751a-12f5-4a57-a332-435360479d53"}]': finished 2024-08-19T14:08:42.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:41 smithi032 ceph-mon[20004]: osdmap e28: 5 total, 4 up, 5 in 2024-08-19T14:08:42.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:41 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-08-19T14:08:42.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:41 smithi045 ceph-mon[20143]: from='client.? 172.21.15.32:0/2003508004' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7859751a-12f5-4a57-a332-435360479d53"}]: dispatch 2024-08-19T14:08:42.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:41 smithi045 ceph-mon[20143]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7859751a-12f5-4a57-a332-435360479d53"}]: dispatch 2024-08-19T14:08:42.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:41 smithi045 ceph-mon[20143]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7859751a-12f5-4a57-a332-435360479d53"}]': finished 2024-08-19T14:08:42.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:41 smithi045 ceph-mon[20143]: osdmap e28: 5 total, 4 up, 5 in 2024-08-19T14:08:42.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:41 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-08-19T14:08:42.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:41 smithi031 ceph-mon[16336]: from='client.? 172.21.15.32:0/2003508004' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7859751a-12f5-4a57-a332-435360479d53"}]: dispatch 2024-08-19T14:08:42.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:41 smithi031 ceph-mon[16336]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7859751a-12f5-4a57-a332-435360479d53"}]: dispatch 2024-08-19T14:08:42.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:41 smithi031 ceph-mon[16336]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7859751a-12f5-4a57-a332-435360479d53"}]': finished 2024-08-19T14:08:42.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:41 smithi031 ceph-mon[16336]: osdmap e28: 5 total, 4 up, 5 in 2024-08-19T14:08:42.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:41 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-08-19T14:08:43.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:42 smithi032 ceph-mon[20004]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-08-19T14:08:43.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:42 smithi032 ceph-mon[20004]: from='client.? 172.21.15.32:0/2149088569' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-19T14:08:43.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:42 smithi045 ceph-mon[20143]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-08-19T14:08:43.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:42 smithi045 ceph-mon[20143]: from='client.? 172.21.15.32:0/2149088569' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-19T14:08:43.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:42 smithi031 ceph-mon[16336]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-08-19T14:08:43.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:42 smithi031 ceph-mon[16336]: from='client.? 172.21.15.32:0/2149088569' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-19T14:08:44.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:44 smithi031 ceph-mon[16336]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 358 GiB / 358 GiB avail 2024-08-19T14:08:44.964 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:44 smithi032 ceph-mon[20004]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 358 GiB / 358 GiB avail 2024-08-19T14:08:45.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:44 smithi045 ceph-mon[20143]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 358 GiB / 358 GiB avail 2024-08-19T14:08:46.951 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:46 smithi032 ceph-mon[20004]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-08-19T14:08:47.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:46 smithi045 ceph-mon[20143]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-08-19T14:08:47.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:46 smithi031 ceph-mon[16336]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-08-19T14:08:47.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:47 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-08-19T14:08:47.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:47 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:08:47.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:47 smithi032 ceph-mon[20004]: Deploying daemon osd.4 on smithi032 2024-08-19T14:08:48.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:47 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-08-19T14:08:48.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:47 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:08:48.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:47 smithi045 ceph-mon[20143]: Deploying daemon osd.4 on smithi032 2024-08-19T14:08:48.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:47 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-08-19T14:08:48.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:47 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:08:48.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:47 smithi031 ceph-mon[16336]: Deploying daemon osd.4 on smithi032 2024-08-19T14:08:48.877 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:48 smithi032 ceph-mon[20004]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-08-19T14:08:49.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:48 smithi045 ceph-mon[20143]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-08-19T14:08:49.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:48 smithi031 ceph-mon[16336]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-08-19T14:08:51.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:50 smithi032 ceph-mon[20004]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-08-19T14:08:51.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:50 smithi045 ceph-mon[20143]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-08-19T14:08:51.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:50 smithi031 ceph-mon[16336]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-08-19T14:08:53.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:52 smithi032 ceph-mon[20004]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-08-19T14:08:53.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:52 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:08:53.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:52 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:53.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:52 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:53.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:52 smithi045 ceph-mon[20143]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-08-19T14:08:53.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:52 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:08:53.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:52 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:53.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:52 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:53.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:52 smithi031 ceph-mon[16336]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-08-19T14:08:53.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:52 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:08:53.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:52 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:53.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:52 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:54.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:54 smithi032 ceph-mon[20004]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-08-19T14:08:55.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:54 smithi045 ceph-mon[20143]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-08-19T14:08:55.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:54 smithi031 ceph-mon[16336]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-08-19T14:08:56.001 INFO:teuthology.orchestra.run.smithi032.stdout:Created osd(s) 4 on host 'smithi032' 2024-08-19T14:08:56.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:56 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:08:56.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:56 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:08:56.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:56 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:56.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:56 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:56.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:56 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:08:56.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:56 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:08:56.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:56 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:56.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:56 smithi032 ceph-mon[20004]: from='osd.4 [v2:172.21.15.32:6800/3934739984,v1:172.21.15.32:6801/3934739984]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-08-19T14:08:56.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:56 smithi032 ceph-mon[20004]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-08-19T14:08:56.259 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:56 smithi032 ceph-mon[20004]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-08-19T14:08:56.259 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:56 smithi032 ceph-mon[20004]: from='osd.4 [v2:172.21.15.32:6800/3934739984,v1:172.21.15.32:6801/3934739984]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-08-19T14:08:56.259 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:56 smithi032 ceph-mon[20004]: osdmap e29: 5 total, 4 up, 5 in 2024-08-19T14:08:56.259 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:56 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-08-19T14:08:56.259 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:56 smithi032 ceph-mon[20004]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-08-19T14:08:56.259 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:56 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:08:56.259 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:56 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:56.259 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:56 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:56.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:56 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:08:56.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:56 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:08:56.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:56 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:56.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:56 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:56.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:56 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:08:56.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:56 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:08:56.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:56 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:56.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:56 smithi045 ceph-mon[20143]: from='osd.4 [v2:172.21.15.32:6800/3934739984,v1:172.21.15.32:6801/3934739984]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-08-19T14:08:56.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:56 smithi045 ceph-mon[20143]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-08-19T14:08:56.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:56 smithi045 ceph-mon[20143]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-08-19T14:08:56.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:56 smithi045 ceph-mon[20143]: from='osd.4 [v2:172.21.15.32:6800/3934739984,v1:172.21.15.32:6801/3934739984]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-08-19T14:08:56.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:56 smithi045 ceph-mon[20143]: osdmap e29: 5 total, 4 up, 5 in 2024-08-19T14:08:56.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:56 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-08-19T14:08:56.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:56 smithi045 ceph-mon[20143]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-08-19T14:08:56.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:56 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:08:56.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:56 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:56.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:56 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:56.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:56 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:08:56.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:56 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:08:56.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:56 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:56.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:56 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:56.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:56 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:08:56.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:56 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:08:56.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:56 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:56.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:56 smithi031 ceph-mon[16336]: from='osd.4 [v2:172.21.15.32:6800/3934739984,v1:172.21.15.32:6801/3934739984]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-08-19T14:08:56.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:56 smithi031 ceph-mon[16336]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-08-19T14:08:56.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:56 smithi031 ceph-mon[16336]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-08-19T14:08:56.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:56 smithi031 ceph-mon[16336]: from='osd.4 [v2:172.21.15.32:6800/3934739984,v1:172.21.15.32:6801/3934739984]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-08-19T14:08:56.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:56 smithi031 ceph-mon[16336]: osdmap e29: 5 total, 4 up, 5 in 2024-08-19T14:08:56.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:56 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-08-19T14:08:56.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:56 smithi031 ceph-mon[16336]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-08-19T14:08:56.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:56 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:08:56.618 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:56 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:56.618 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:56 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:08:56.792 DEBUG:teuthology.orchestra.run.smithi032:osd.4> sudo journalctl -f -n 0 -u ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968@osd.4.service 2024-08-19T14:08:56.795 INFO:tasks.cephadm:Deploying osd.5 on smithi032 with /dev/vg_nvme/lv_3... 2024-08-19T14:08:56.795 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- lvm zap /dev/vg_nvme/lv_3 2024-08-19T14:08:57.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:57 smithi032 ceph-mon[20004]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-08-19T14:08:57.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:57 smithi032 ceph-mon[20004]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]': finished 2024-08-19T14:08:57.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:57 smithi032 ceph-mon[20004]: osdmap e30: 5 total, 4 up, 5 in 2024-08-19T14:08:57.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:57 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-08-19T14:08:57.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:57 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:08:57.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:57 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:08:57.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:57 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:08:57.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:57 smithi045 ceph-mon[20143]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-08-19T14:08:57.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:57 smithi045 ceph-mon[20143]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]': finished 2024-08-19T14:08:57.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:57 smithi045 ceph-mon[20143]: osdmap e30: 5 total, 4 up, 5 in 2024-08-19T14:08:57.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:57 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-08-19T14:08:57.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:57 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:08:57.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:57 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:08:57.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:57 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:08:57.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:57 smithi031 ceph-mon[16336]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-08-19T14:08:57.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:57 smithi031 ceph-mon[16336]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]': finished 2024-08-19T14:08:57.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:57 smithi031 ceph-mon[16336]: osdmap e30: 5 total, 4 up, 5 in 2024-08-19T14:08:57.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:57 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-08-19T14:08:57.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:57 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:08:57.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:57 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:08:57.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:57 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:08:57.889 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:08:58.742 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:58 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:08:58.742 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:58 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:08:58.743 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:58 smithi032 ceph-mon[20004]: osd.4 [v2:172.21.15.32:6800/3934739984,v1:172.21.15.32:6801/3934739984] boot 2024-08-19T14:08:58.743 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:58 smithi032 ceph-mon[20004]: osdmap e31: 5 total, 5 up, 5 in 2024-08-19T14:08:58.743 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:58 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-08-19T14:08:58.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:58 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:08:58.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:58 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:08:58.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:58 smithi045 ceph-mon[20143]: osd.4 [v2:172.21.15.32:6800/3934739984,v1:172.21.15.32:6801/3934739984] boot 2024-08-19T14:08:58.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:58 smithi045 ceph-mon[20143]: osdmap e31: 5 total, 5 up, 5 in 2024-08-19T14:08:58.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:58 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-08-19T14:08:58.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:58 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:08:58.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:58 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:08:58.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:58 smithi031 ceph-mon[16336]: osd.4 [v2:172.21.15.32:6800/3934739984,v1:172.21.15.32:6801/3934739984] boot 2024-08-19T14:08:58.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:58 smithi031 ceph-mon[16336]: osdmap e31: 5 total, 5 up, 5 in 2024-08-19T14:08:58.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:58 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-08-19T14:08:59.823 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:59 smithi032 ceph-mon[20004]: pgmap v103: 1 pgs: 1 remapped+peering; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-08-19T14:08:59.824 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:08:59 smithi032 ceph-mon[20004]: osdmap e32: 5 total, 5 up, 5 in 2024-08-19T14:08:59.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:59 smithi031 ceph-mon[16336]: pgmap v103: 1 pgs: 1 remapped+peering; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-08-19T14:08:59.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:08:59 smithi031 ceph-mon[16336]: osdmap e32: 5 total, 5 up, 5 in 2024-08-19T14:09:00.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:59 smithi045 ceph-mon[20143]: pgmap v103: 1 pgs: 1 remapped+peering; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-08-19T14:09:00.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:08:59 smithi045 ceph-mon[20143]: osdmap e32: 5 total, 5 up, 5 in 2024-08-19T14:09:00.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:00 smithi031 ceph-mon[16336]: osdmap e33: 5 total, 5 up, 5 in 2024-08-19T14:09:00.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:00 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:09:00.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:00 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:09:00.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:00 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:09:00.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:00 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:09:00.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:00 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:09:00.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:00 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:09:00.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:00 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:09:00.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:00 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:09:00.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:00 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:09:00.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:00 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:09:00.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:00 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:09:00.971 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:00 smithi032 ceph-mon[20004]: osdmap e33: 5 total, 5 up, 5 in 2024-08-19T14:09:00.971 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:00 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:09:00.971 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:00 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:09:00.972 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:00 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:09:00.972 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:00 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:09:00.972 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:00 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:09:00.972 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:00 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:09:00.972 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:00 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:09:00.972 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:00 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:09:00.972 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:00 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:09:00.972 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:00 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:09:00.972 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:00 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:09:01.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:00 smithi045 ceph-mon[20143]: osdmap e33: 5 total, 5 up, 5 in 2024-08-19T14:09:01.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:00 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:09:01.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:00 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:09:01.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:00 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:09:01.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:00 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:09:01.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:00 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:09:01.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:00 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:09:01.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:00 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:09:01.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:00 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:09:01.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:00 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:09:01.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:00 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:09:01.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:00 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:09:01.267 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-19T14:09:01.295 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph orch daemon add osd smithi032:vg_nvme/lv_3 2024-08-19T14:09:01.549 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:09:01.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:01 smithi031 ceph-mon[16336]: pgmap v106: 1 pgs: 1 remapped+peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-08-19T14:09:01.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:01 smithi031 ceph-mon[16336]: Detected new or changed devices on smithi032 2024-08-19T14:09:01.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:01 smithi031 ceph-mon[16336]: Adjusting osd_memory_target on smithi032 to 17092M 2024-08-19T14:09:01.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:01 smithi031 ceph-mon[16336]: osdmap e34: 5 total, 5 up, 5 in 2024-08-19T14:09:01.891 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:01 smithi032 ceph-mon[20004]: pgmap v106: 1 pgs: 1 remapped+peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-08-19T14:09:01.891 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:01 smithi032 ceph-mon[20004]: Detected new or changed devices on smithi032 2024-08-19T14:09:01.891 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:01 smithi032 ceph-mon[20004]: Adjusting osd_memory_target on smithi032 to 17092M 2024-08-19T14:09:01.891 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:01 smithi032 ceph-mon[20004]: osdmap e34: 5 total, 5 up, 5 in 2024-08-19T14:09:02.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:01 smithi045 ceph-mon[20143]: pgmap v106: 1 pgs: 1 remapped+peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-08-19T14:09:02.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:01 smithi045 ceph-mon[20143]: Detected new or changed devices on smithi032 2024-08-19T14:09:02.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:01 smithi045 ceph-mon[20143]: Adjusting osd_memory_target on smithi032 to 17092M 2024-08-19T14:09:02.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:01 smithi045 ceph-mon[20143]: osdmap e34: 5 total, 5 up, 5 in 2024-08-19T14:09:02.823 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:02 smithi032 ceph-mon[20004]: osdmap e35: 5 total, 5 up, 5 in 2024-08-19T14:09:02.823 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:02 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-19T14:09:02.823 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:02 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-19T14:09:02.823 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:02 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:09:02.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:02 smithi031 ceph-mon[16336]: osdmap e35: 5 total, 5 up, 5 in 2024-08-19T14:09:02.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:02 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-19T14:09:02.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:02 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-19T14:09:02.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:02 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:09:03.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:02 smithi045 ceph-mon[20143]: osdmap e35: 5 total, 5 up, 5 in 2024-08-19T14:09:03.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:02 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-19T14:09:03.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:02 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-19T14:09:03.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:02 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:09:03.793 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:03 smithi032 ceph-mon[20004]: pgmap v109: 1 pgs: 1 remapped+peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-08-19T14:09:03.793 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:03 smithi032 ceph-mon[20004]: from='client.24242 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T14:09:03.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:03 smithi031 ceph-mon[16336]: pgmap v109: 1 pgs: 1 remapped+peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-08-19T14:09:03.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:03 smithi031 ceph-mon[16336]: from='client.24242 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T14:09:04.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:03 smithi045 ceph-mon[20143]: pgmap v109: 1 pgs: 1 remapped+peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-08-19T14:09:04.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:03 smithi045 ceph-mon[20143]: from='client.24242 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T14:09:04.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:04 smithi031 ceph-mon[16336]: from='client.? 172.21.15.32:0/3347560246' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "017f3eef-ac60-4b82-8350-919017cb4ccd"}]: dispatch 2024-08-19T14:09:04.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:04 smithi031 ceph-mon[16336]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "017f3eef-ac60-4b82-8350-919017cb4ccd"}]: dispatch 2024-08-19T14:09:04.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:04 smithi031 ceph-mon[16336]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "017f3eef-ac60-4b82-8350-919017cb4ccd"}]': finished 2024-08-19T14:09:04.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:04 smithi031 ceph-mon[16336]: osdmap e36: 6 total, 5 up, 6 in 2024-08-19T14:09:04.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:04 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-08-19T14:09:05.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:04 smithi032 ceph-mon[20004]: from='client.? 172.21.15.32:0/3347560246' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "017f3eef-ac60-4b82-8350-919017cb4ccd"}]: dispatch 2024-08-19T14:09:05.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:04 smithi032 ceph-mon[20004]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "017f3eef-ac60-4b82-8350-919017cb4ccd"}]: dispatch 2024-08-19T14:09:05.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:04 smithi032 ceph-mon[20004]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "017f3eef-ac60-4b82-8350-919017cb4ccd"}]': finished 2024-08-19T14:09:05.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:04 smithi032 ceph-mon[20004]: osdmap e36: 6 total, 5 up, 6 in 2024-08-19T14:09:05.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:04 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-08-19T14:09:05.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:04 smithi045 ceph-mon[20143]: from='client.? 172.21.15.32:0/3347560246' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "017f3eef-ac60-4b82-8350-919017cb4ccd"}]: dispatch 2024-08-19T14:09:05.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:04 smithi045 ceph-mon[20143]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "017f3eef-ac60-4b82-8350-919017cb4ccd"}]: dispatch 2024-08-19T14:09:05.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:04 smithi045 ceph-mon[20143]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "017f3eef-ac60-4b82-8350-919017cb4ccd"}]': finished 2024-08-19T14:09:05.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:04 smithi045 ceph-mon[20143]: osdmap e36: 6 total, 5 up, 6 in 2024-08-19T14:09:05.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:04 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-08-19T14:09:05.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:05 smithi031 ceph-mon[16336]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 100 KiB/s, 0 objects/s recovering 2024-08-19T14:09:05.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:05 smithi031 ceph-mon[16336]: from='client.? 172.21.15.32:0/405941565' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-19T14:09:06.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:05 smithi032 ceph-mon[20004]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 100 KiB/s, 0 objects/s recovering 2024-08-19T14:09:06.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:05 smithi032 ceph-mon[20004]: from='client.? 172.21.15.32:0/405941565' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-19T14:09:06.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:05 smithi045 ceph-mon[20143]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 100 KiB/s, 0 objects/s recovering 2024-08-19T14:09:06.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:05 smithi045 ceph-mon[20143]: from='client.? 172.21.15.32:0/405941565' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-19T14:09:07.839 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:07 smithi032 ceph-mon[20004]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-08-19T14:09:07.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:07 smithi031 ceph-mon[16336]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-08-19T14:09:08.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:07 smithi045 ceph-mon[20143]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-08-19T14:09:09.741 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:09 smithi032 ceph-mon[20004]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-08-19T14:09:09.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:09 smithi031 ceph-mon[16336]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-08-19T14:09:10.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:09 smithi045 ceph-mon[20143]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-08-19T14:09:10.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:10 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-08-19T14:09:10.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:10 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:09:10.881 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:10 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-08-19T14:09:10.882 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:10 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:09:11.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:10 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-08-19T14:09:11.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:10 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:09:11.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:11 smithi031 ceph-mon[16336]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-08-19T14:09:11.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:11 smithi031 ceph-mon[16336]: Deploying daemon osd.5 on smithi032 2024-08-19T14:09:11.883 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:11 smithi032 ceph-mon[20004]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-08-19T14:09:11.883 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:11 smithi032 ceph-mon[20004]: Deploying daemon osd.5 on smithi032 2024-08-19T14:09:12.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:11 smithi045 ceph-mon[20143]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-08-19T14:09:12.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:11 smithi045 ceph-mon[20143]: Deploying daemon osd.5 on smithi032 2024-08-19T14:09:12.866 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:12 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:09:12.866 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:12 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:09:12.867 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:12 smithi032 ceph-mon[20004]: osdmap e37: 6 total, 5 up, 6 in 2024-08-19T14:09:12.867 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:12 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-08-19T14:09:12.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:12 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:09:12.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:12 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:09:12.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:12 smithi031 ceph-mon[16336]: osdmap e37: 6 total, 5 up, 6 in 2024-08-19T14:09:12.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:12 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-08-19T14:09:13.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:12 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:09:13.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:12 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:09:13.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:12 smithi045 ceph-mon[20143]: osdmap e37: 6 total, 5 up, 6 in 2024-08-19T14:09:13.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:12 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-08-19T14:09:13.859 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:13 smithi032 ceph-mon[20004]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-08-19T14:09:13.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:13 smithi031 ceph-mon[16336]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-08-19T14:09:14.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:13 smithi045 ceph-mon[20143]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-08-19T14:09:15.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:15 smithi031 ceph-mon[16336]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-08-19T14:09:15.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:15 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:09:15.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:15 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:09:15.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:15 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:09:15.920 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:15 smithi032 ceph-mon[20004]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-08-19T14:09:15.920 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:15 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:09:15.920 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:15 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:09:15.920 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:15 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:09:16.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:15 smithi045 ceph-mon[20143]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-08-19T14:09:16.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:15 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:09:16.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:15 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:09:16.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:15 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:09:17.804 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:17 smithi032 ceph-mon[20004]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-08-19T14:09:17.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:17 smithi031 ceph-mon[16336]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-08-19T14:09:18.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:17 smithi045 ceph-mon[20143]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-08-19T14:09:18.400 INFO:teuthology.orchestra.run.smithi032.stdout:Created osd(s) 5 on host 'smithi032' 2024-08-19T14:09:19.016 DEBUG:teuthology.orchestra.run.smithi032:osd.5> sudo journalctl -f -n 0 -u ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968@osd.5.service 2024-08-19T14:09:19.019 INFO:tasks.cephadm:Deploying osd.6 on smithi032 with /dev/vg_nvme/lv_2... 2024-08-19T14:09:19.019 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- lvm zap /dev/vg_nvme/lv_2 2024-08-19T14:09:19.618 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:19 smithi032 ceph-mon[20004]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-08-19T14:09:19.618 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:19 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:09:19.618 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:19 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:09:19.618 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:19 smithi032 ceph-mon[20004]: from='osd.5 [v2:172.21.15.32:6808/526388985,v1:172.21.15.32:6809/526388985]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-08-19T14:09:19.618 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:19 smithi032 ceph-mon[20004]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-08-19T14:09:19.618 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:19 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:09:19.619 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:19 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:09:19.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:19 smithi045 ceph-mon[20143]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-08-19T14:09:19.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:19 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:09:19.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:19 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:09:19.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:19 smithi045 ceph-mon[20143]: from='osd.5 [v2:172.21.15.32:6808/526388985,v1:172.21.15.32:6809/526388985]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-08-19T14:09:19.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:19 smithi045 ceph-mon[20143]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-08-19T14:09:19.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:19 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:09:19.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:19 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:09:19.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:19 smithi031 ceph-mon[16336]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-08-19T14:09:19.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:19 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:09:19.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:19 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:09:19.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:19 smithi031 ceph-mon[16336]: from='osd.5 [v2:172.21.15.32:6808/526388985,v1:172.21.15.32:6809/526388985]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-08-19T14:09:19.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:19 smithi031 ceph-mon[16336]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-08-19T14:09:19.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:19 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:09:19.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:19 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:09:20.224 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:09:20.632 INFO:journalctl@ceph.osd.5.smithi032.stdout:Aug 19 14:09:20 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-5[29877]: 2024-08-19T14:09:20.410+0000 7f6c7bb26640 -1 osd.5 0 waiting for initial osdmap 2024-08-19T14:09:20.632 INFO:journalctl@ceph.osd.5.smithi032.stdout:Aug 19 14:09:20 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-5[29877]: 2024-08-19T14:09:20.414+0000 7f6c7794e640 -1 osd.5 39 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-19T14:09:20.633 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:20 smithi032 ceph-mon[20004]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-08-19T14:09:20.633 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:20 smithi032 ceph-mon[20004]: from='osd.5 [v2:172.21.15.32:6808/526388985,v1:172.21.15.32:6809/526388985]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-08-19T14:09:20.633 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:20 smithi032 ceph-mon[20004]: osdmap e38: 6 total, 5 up, 6 in 2024-08-19T14:09:20.633 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:20 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-08-19T14:09:20.633 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:20 smithi032 ceph-mon[20004]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-08-19T14:09:20.633 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:20 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:09:20.633 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:20 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:09:20.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:20 smithi045 ceph-mon[20143]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-08-19T14:09:20.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:20 smithi045 ceph-mon[20143]: from='osd.5 [v2:172.21.15.32:6808/526388985,v1:172.21.15.32:6809/526388985]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-08-19T14:09:20.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:20 smithi045 ceph-mon[20143]: osdmap e38: 6 total, 5 up, 6 in 2024-08-19T14:09:20.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:20 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-08-19T14:09:20.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:20 smithi045 ceph-mon[20143]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-08-19T14:09:20.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:20 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:09:20.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:20 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:09:20.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:20 smithi031 ceph-mon[16336]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-08-19T14:09:20.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:20 smithi031 ceph-mon[16336]: from='osd.5 [v2:172.21.15.32:6808/526388985,v1:172.21.15.32:6809/526388985]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-08-19T14:09:20.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:20 smithi031 ceph-mon[16336]: osdmap e38: 6 total, 5 up, 6 in 2024-08-19T14:09:20.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:20 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-08-19T14:09:20.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:20 smithi031 ceph-mon[16336]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-08-19T14:09:20.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:20 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:09:20.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:20 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:09:21.527 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:21 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:09:21.527 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:21 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:09:21.527 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:21 smithi032 ceph-mon[20004]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-08-19T14:09:21.527 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:21 smithi032 ceph-mon[20004]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]': finished 2024-08-19T14:09:21.527 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:21 smithi032 ceph-mon[20004]: osdmap e39: 6 total, 5 up, 6 in 2024-08-19T14:09:21.527 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:21 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-08-19T14:09:21.527 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:21 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-08-19T14:09:21.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:21 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:09:21.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:21 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:09:21.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:21 smithi045 ceph-mon[20143]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-08-19T14:09:21.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:21 smithi045 ceph-mon[20143]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]': finished 2024-08-19T14:09:21.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:21 smithi045 ceph-mon[20143]: osdmap e39: 6 total, 5 up, 6 in 2024-08-19T14:09:21.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:21 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-08-19T14:09:21.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:21 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-08-19T14:09:21.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:21 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:09:21.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:21 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:09:21.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:21 smithi031 ceph-mon[16336]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-08-19T14:09:21.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:21 smithi031 ceph-mon[16336]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]': finished 2024-08-19T14:09:21.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:21 smithi031 ceph-mon[16336]: osdmap e39: 6 total, 5 up, 6 in 2024-08-19T14:09:21.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:21 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-08-19T14:09:21.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:21 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-08-19T14:09:22.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:22 smithi032 ceph-mon[20004]: osd.5 [v2:172.21.15.32:6808/526388985,v1:172.21.15.32:6809/526388985] boot 2024-08-19T14:09:22.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:22 smithi032 ceph-mon[20004]: osdmap e40: 6 total, 6 up, 6 in 2024-08-19T14:09:22.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:22 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-08-19T14:09:22.768 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:22 smithi031 ceph-mon[16336]: osd.5 [v2:172.21.15.32:6808/526388985,v1:172.21.15.32:6809/526388985] boot 2024-08-19T14:09:22.768 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:22 smithi031 ceph-mon[16336]: osdmap e40: 6 total, 6 up, 6 in 2024-08-19T14:09:22.768 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:22 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-08-19T14:09:22.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:22 smithi045 ceph-mon[20143]: osd.5 [v2:172.21.15.32:6808/526388985,v1:172.21.15.32:6809/526388985] boot 2024-08-19T14:09:22.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:22 smithi045 ceph-mon[20143]: osdmap e40: 6 total, 6 up, 6 in 2024-08-19T14:09:22.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:22 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-08-19T14:09:23.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:23 smithi032 ceph-mon[20004]: pgmap v124: 1 pgs: 1 unknown; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-08-19T14:09:23.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:23 smithi032 ceph-mon[20004]: osdmap e41: 6 total, 6 up, 6 in 2024-08-19T14:09:23.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:23 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:09:23.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:23 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:09:23.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:23 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:09:23.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:23 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:09:23.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:23 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:09:23.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:23 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:09:23.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:23 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:09:23.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:23 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:09:23.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:23 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:09:23.509 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:23 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:09:23.509 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:23 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:09:23.509 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:23 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:09:23.509 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:23 smithi032 ceph-mon[20004]: osdmap e42: 6 total, 6 up, 6 in 2024-08-19T14:09:23.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:23 smithi045 ceph-mon[20143]: pgmap v124: 1 pgs: 1 unknown; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-08-19T14:09:23.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:23 smithi045 ceph-mon[20143]: osdmap e41: 6 total, 6 up, 6 in 2024-08-19T14:09:23.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:23 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:09:23.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:23 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:09:23.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:23 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:09:23.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:23 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:09:23.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:23 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:09:23.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:23 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:09:23.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:23 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:09:23.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:23 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:09:23.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:23 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:09:23.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:23 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:09:23.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:23 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:09:23.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:23 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:09:23.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:23 smithi045 ceph-mon[20143]: osdmap e42: 6 total, 6 up, 6 in 2024-08-19T14:09:23.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:23 smithi031 ceph-mon[16336]: pgmap v124: 1 pgs: 1 unknown; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-08-19T14:09:23.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:23 smithi031 ceph-mon[16336]: osdmap e41: 6 total, 6 up, 6 in 2024-08-19T14:09:23.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:23 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:09:23.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:23 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:09:23.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:23 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:09:23.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:23 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:09:23.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:23 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:09:23.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:23 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:09:23.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:23 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:09:23.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:23 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:09:23.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:23 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:09:23.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:23 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:09:23.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:23 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:09:23.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:23 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:09:23.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:23 smithi031 ceph-mon[16336]: osdmap e42: 6 total, 6 up, 6 in 2024-08-19T14:09:24.303 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-19T14:09:24.331 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph orch daemon add osd smithi032:vg_nvme/lv_2 2024-08-19T14:09:24.592 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:24 smithi032 ceph-mon[20004]: Detected new or changed devices on smithi032 2024-08-19T14:09:24.592 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:24 smithi032 ceph-mon[20004]: Adjusting osd_memory_target on smithi032 to 8546M 2024-08-19T14:09:24.595 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:09:24.698 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:24 smithi045 ceph-mon[20143]: Detected new or changed devices on smithi032 2024-08-19T14:09:24.698 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:24 smithi045 ceph-mon[20143]: Adjusting osd_memory_target on smithi032 to 8546M 2024-08-19T14:09:24.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:24 smithi031 ceph-mon[16336]: Detected new or changed devices on smithi032 2024-08-19T14:09:24.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:24 smithi031 ceph-mon[16336]: Adjusting osd_memory_target on smithi032 to 8546M 2024-08-19T14:09:25.694 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:25 smithi032 ceph-mon[20004]: pgmap v127: 1 pgs: 1 unknown; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-08-19T14:09:25.695 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:25 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-19T14:09:25.695 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:25 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-19T14:09:25.695 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:25 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:09:25.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:25 smithi045 ceph-mon[20143]: pgmap v127: 1 pgs: 1 unknown; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-08-19T14:09:25.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:25 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-19T14:09:25.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:25 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-19T14:09:25.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:25 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:09:25.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:25 smithi031 ceph-mon[16336]: pgmap v127: 1 pgs: 1 unknown; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-08-19T14:09:25.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:25 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-19T14:09:25.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:25 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-19T14:09:25.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:25 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:09:26.535 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:26 smithi032 ceph-mon[20004]: from='client.24269 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T14:09:26.535 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:26 smithi032 ceph-mon[20004]: osdmap e43: 6 total, 6 up, 6 in 2024-08-19T14:09:26.535 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:26 smithi032 ceph-mon[20004]: osdmap e44: 6 total, 6 up, 6 in 2024-08-19T14:09:26.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:26 smithi045 ceph-mon[20143]: from='client.24269 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T14:09:26.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:26 smithi045 ceph-mon[20143]: osdmap e43: 6 total, 6 up, 6 in 2024-08-19T14:09:26.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:26 smithi045 ceph-mon[20143]: osdmap e44: 6 total, 6 up, 6 in 2024-08-19T14:09:26.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:26 smithi031 ceph-mon[16336]: from='client.24269 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T14:09:26.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:26 smithi031 ceph-mon[16336]: osdmap e43: 6 total, 6 up, 6 in 2024-08-19T14:09:26.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:26 smithi031 ceph-mon[16336]: osdmap e44: 6 total, 6 up, 6 in 2024-08-19T14:09:27.717 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:27 smithi032 ceph-mon[20004]: pgmap v129: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 120 KiB/s, 0 objects/s recovering 2024-08-19T14:09:27.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:27 smithi045 ceph-mon[20143]: pgmap v129: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 120 KiB/s, 0 objects/s recovering 2024-08-19T14:09:27.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:27 smithi031 ceph-mon[16336]: pgmap v129: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 120 KiB/s, 0 objects/s recovering 2024-08-19T14:09:28.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:28 smithi032 ceph-mon[20004]: 1.0 scrub starts 2024-08-19T14:09:28.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:28 smithi032 ceph-mon[20004]: 1.0 scrub ok 2024-08-19T14:09:28.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:28 smithi032 ceph-mon[20004]: from='client.? 172.21.15.32:0/6869860' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "44aa8229-0268-49d7-bd57-3bf1f384e5e6"}]: dispatch 2024-08-19T14:09:28.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:28 smithi032 ceph-mon[20004]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "44aa8229-0268-49d7-bd57-3bf1f384e5e6"}]: dispatch 2024-08-19T14:09:28.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:28 smithi032 ceph-mon[20004]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "44aa8229-0268-49d7-bd57-3bf1f384e5e6"}]': finished 2024-08-19T14:09:28.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:28 smithi032 ceph-mon[20004]: osdmap e45: 7 total, 6 up, 7 in 2024-08-19T14:09:28.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:28 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-08-19T14:09:28.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:28 smithi032 ceph-mon[20004]: from='client.? 172.21.15.32:0/2908873024' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-19T14:09:28.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:28 smithi045 ceph-mon[20143]: 1.0 scrub starts 2024-08-19T14:09:28.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:28 smithi045 ceph-mon[20143]: 1.0 scrub ok 2024-08-19T14:09:28.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:28 smithi045 ceph-mon[20143]: from='client.? 172.21.15.32:0/6869860' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "44aa8229-0268-49d7-bd57-3bf1f384e5e6"}]: dispatch 2024-08-19T14:09:28.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:28 smithi045 ceph-mon[20143]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "44aa8229-0268-49d7-bd57-3bf1f384e5e6"}]: dispatch 2024-08-19T14:09:28.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:28 smithi045 ceph-mon[20143]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "44aa8229-0268-49d7-bd57-3bf1f384e5e6"}]': finished 2024-08-19T14:09:28.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:28 smithi045 ceph-mon[20143]: osdmap e45: 7 total, 6 up, 7 in 2024-08-19T14:09:28.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:28 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-08-19T14:09:28.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:28 smithi045 ceph-mon[20143]: from='client.? 172.21.15.32:0/2908873024' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-19T14:09:28.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:28 smithi031 ceph-mon[16336]: 1.0 scrub starts 2024-08-19T14:09:28.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:28 smithi031 ceph-mon[16336]: 1.0 scrub ok 2024-08-19T14:09:28.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:28 smithi031 ceph-mon[16336]: from='client.? 172.21.15.32:0/6869860' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "44aa8229-0268-49d7-bd57-3bf1f384e5e6"}]: dispatch 2024-08-19T14:09:28.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:28 smithi031 ceph-mon[16336]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "44aa8229-0268-49d7-bd57-3bf1f384e5e6"}]: dispatch 2024-08-19T14:09:28.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:28 smithi031 ceph-mon[16336]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "44aa8229-0268-49d7-bd57-3bf1f384e5e6"}]': finished 2024-08-19T14:09:28.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:28 smithi031 ceph-mon[16336]: osdmap e45: 7 total, 6 up, 7 in 2024-08-19T14:09:28.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:28 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-08-19T14:09:28.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:28 smithi031 ceph-mon[16336]: from='client.? 172.21.15.32:0/2908873024' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-19T14:09:29.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:29 smithi032 ceph-mon[20004]: pgmap v132: 1 pgs: 1 peering; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail; 121 KiB/s, 0 objects/s recovering 2024-08-19T14:09:29.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:29 smithi045 ceph-mon[20143]: pgmap v132: 1 pgs: 1 peering; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail; 121 KiB/s, 0 objects/s recovering 2024-08-19T14:09:29.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:29 smithi031 ceph-mon[16336]: pgmap v132: 1 pgs: 1 peering; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail; 121 KiB/s, 0 objects/s recovering 2024-08-19T14:09:31.646 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:31 smithi032 ceph-mon[20004]: pgmap v133: 1 pgs: 1 peering; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-08-19T14:09:31.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:31 smithi045 ceph-mon[20143]: pgmap v133: 1 pgs: 1 peering; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-08-19T14:09:31.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:31 smithi031 ceph-mon[16336]: pgmap v133: 1 pgs: 1 peering; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-08-19T14:09:33.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:33 smithi032 ceph-mon[20004]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-08-19T14:09:33.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:33 smithi045 ceph-mon[20143]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-08-19T14:09:33.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:33 smithi031 ceph-mon[16336]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-08-19T14:09:34.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:34 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-08-19T14:09:34.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:34 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:09:34.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:34 smithi032 ceph-mon[20004]: Deploying daemon osd.6 on smithi032 2024-08-19T14:09:34.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:34 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-08-19T14:09:34.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:34 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:09:34.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:34 smithi045 ceph-mon[20143]: Deploying daemon osd.6 on smithi032 2024-08-19T14:09:34.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:34 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-08-19T14:09:34.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:34 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:09:34.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:34 smithi031 ceph-mon[16336]: Deploying daemon osd.6 on smithi032 2024-08-19T14:09:35.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:35 smithi032 ceph-mon[20004]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-08-19T14:09:35.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:35 smithi045 ceph-mon[20143]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-08-19T14:09:35.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:35 smithi031 ceph-mon[16336]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-08-19T14:09:37.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:37 smithi032 ceph-mon[20004]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-08-19T14:09:37.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:37 smithi045 ceph-mon[20143]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-08-19T14:09:37.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:37 smithi031 ceph-mon[16336]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-08-19T14:09:38.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:38 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:09:38.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:38 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:09:38.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:38 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:09:38.814 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:38 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:09:38.814 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:38 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:09:38.814 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:38 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:09:38.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:38 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:09:38.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:38 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:09:38.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:38 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:09:39.724 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:39 smithi032 ceph-mon[20004]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-08-19T14:09:39.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:39 smithi045 ceph-mon[20143]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-08-19T14:09:39.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:39 smithi031 ceph-mon[16336]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-08-19T14:09:41.724 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:41 smithi032 ceph-mon[20004]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-08-19T14:09:41.725 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:41 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:09:41.725 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:41 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:09:41.725 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:41 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:09:41.725 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:41 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:09:41.725 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:41 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:09:41.725 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:41 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:09:41.725 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:41 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:09:41.725 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:41 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:09:41.725 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:41 smithi032 ceph-mon[20004]: from='osd.6 [v2:172.21.15.32:6816/1131035844,v1:172.21.15.32:6817/1131035844]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-08-19T14:09:41.725 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:41 smithi032 ceph-mon[20004]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-08-19T14:09:41.764 INFO:teuthology.orchestra.run.smithi032.stdout:Created osd(s) 6 on host 'smithi032' 2024-08-19T14:09:41.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:41 smithi045 ceph-mon[20143]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-08-19T14:09:41.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:41 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:09:41.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:41 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:09:41.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:41 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:09:41.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:41 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:09:41.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:41 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:09:41.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:41 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:09:41.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:41 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:09:41.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:41 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:09:41.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:41 smithi045 ceph-mon[20143]: from='osd.6 [v2:172.21.15.32:6816/1131035844,v1:172.21.15.32:6817/1131035844]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-08-19T14:09:41.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:41 smithi045 ceph-mon[20143]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-08-19T14:09:41.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:41 smithi031 ceph-mon[16336]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-08-19T14:09:41.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:41 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:09:41.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:41 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:09:41.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:41 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:09:41.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:41 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:09:41.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:41 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:09:41.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:41 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:09:41.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:41 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:09:41.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:41 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:09:41.868 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:41 smithi031 ceph-mon[16336]: from='osd.6 [v2:172.21.15.32:6816/1131035844,v1:172.21.15.32:6817/1131035844]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-08-19T14:09:41.868 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:41 smithi031 ceph-mon[16336]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-08-19T14:09:42.411 DEBUG:teuthology.orchestra.run.smithi032:osd.6> sudo journalctl -f -n 0 -u ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968@osd.6.service 2024-08-19T14:09:42.413 INFO:tasks.cephadm:Deploying osd.7 on smithi032 with /dev/vg_nvme/lv_1... 2024-08-19T14:09:42.414 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- lvm zap /dev/vg_nvme/lv_1 2024-08-19T14:09:42.645 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:42 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:09:42.645 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:42 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:09:42.646 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:42 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:09:42.646 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:42 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:09:42.646 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:42 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:09:42.646 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:42 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:09:42.646 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:42 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:09:42.646 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:42 smithi032 ceph-mon[20004]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-08-19T14:09:42.646 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:42 smithi032 ceph-mon[20004]: from='osd.6 [v2:172.21.15.32:6816/1131035844,v1:172.21.15.32:6817/1131035844]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-08-19T14:09:42.646 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:42 smithi032 ceph-mon[20004]: osdmap e46: 7 total, 6 up, 7 in 2024-08-19T14:09:42.646 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:42 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-08-19T14:09:42.646 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:42 smithi032 ceph-mon[20004]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-08-19T14:09:42.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:42 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:09:42.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:42 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:09:42.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:42 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:09:42.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:42 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:09:42.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:42 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:09:42.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:42 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:09:42.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:42 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:09:42.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:42 smithi045 ceph-mon[20143]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-08-19T14:09:42.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:42 smithi045 ceph-mon[20143]: from='osd.6 [v2:172.21.15.32:6816/1131035844,v1:172.21.15.32:6817/1131035844]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-08-19T14:09:42.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:42 smithi045 ceph-mon[20143]: osdmap e46: 7 total, 6 up, 7 in 2024-08-19T14:09:42.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:42 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-08-19T14:09:42.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:42 smithi045 ceph-mon[20143]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-08-19T14:09:42.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:42 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:09:42.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:42 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:09:42.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:42 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:09:42.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:42 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:09:42.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:42 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:09:42.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:42 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:09:42.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:42 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:09:42.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:42 smithi031 ceph-mon[16336]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-08-19T14:09:42.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:42 smithi031 ceph-mon[16336]: from='osd.6 [v2:172.21.15.32:6816/1131035844,v1:172.21.15.32:6817/1131035844]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-08-19T14:09:42.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:42 smithi031 ceph-mon[16336]: osdmap e46: 7 total, 6 up, 7 in 2024-08-19T14:09:42.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:42 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-08-19T14:09:42.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:42 smithi031 ceph-mon[16336]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-08-19T14:09:43.150 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:09:43.549 INFO:journalctl@ceph.osd.6.smithi032.stdout:Aug 19 14:09:43 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-6[35351]: 2024-08-19T14:09:43.433+0000 7f8d407e5640 -1 osd.6 0 waiting for initial osdmap 2024-08-19T14:09:43.549 INFO:journalctl@ceph.osd.6.smithi032.stdout:Aug 19 14:09:43 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-6[35351]: 2024-08-19T14:09:43.439+0000 7f8d3c60d640 -1 osd.6 47 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-19T14:09:43.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:43 smithi031 ceph-mon[16336]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-08-19T14:09:43.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:43 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:09:43.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:43 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:09:43.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:43 smithi031 ceph-mon[16336]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]': finished 2024-08-19T14:09:43.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:43 smithi031 ceph-mon[16336]: osdmap e47: 7 total, 6 up, 7 in 2024-08-19T14:09:43.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:43 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-08-19T14:09:43.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:43 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-08-19T14:09:43.888 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:43 smithi032 ceph-mon[20004]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-08-19T14:09:43.888 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:43 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:09:43.888 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:43 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:09:43.888 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:43 smithi032 ceph-mon[20004]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]': finished 2024-08-19T14:09:43.888 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:43 smithi032 ceph-mon[20004]: osdmap e47: 7 total, 6 up, 7 in 2024-08-19T14:09:43.888 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:43 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-08-19T14:09:43.888 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:43 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-08-19T14:09:44.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:43 smithi045 ceph-mon[20143]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-08-19T14:09:44.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:43 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:09:44.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:43 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:09:44.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:43 smithi045 ceph-mon[20143]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]': finished 2024-08-19T14:09:44.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:43 smithi045 ceph-mon[20143]: osdmap e47: 7 total, 6 up, 7 in 2024-08-19T14:09:44.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:43 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-08-19T14:09:44.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:43 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-08-19T14:09:45.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:45 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:09:45.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:45 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:09:45.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:45 smithi032 ceph-mon[20004]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-08-19T14:09:45.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:45 smithi032 ceph-mon[20004]: osd.6 [v2:172.21.15.32:6816/1131035844,v1:172.21.15.32:6817/1131035844] boot 2024-08-19T14:09:45.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:45 smithi032 ceph-mon[20004]: osdmap e48: 7 total, 7 up, 7 in 2024-08-19T14:09:45.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:45 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-08-19T14:09:45.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:45 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:09:45.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:45 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:09:45.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:45 smithi045 ceph-mon[20143]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-08-19T14:09:45.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:45 smithi045 ceph-mon[20143]: osd.6 [v2:172.21.15.32:6816/1131035844,v1:172.21.15.32:6817/1131035844] boot 2024-08-19T14:09:45.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:45 smithi045 ceph-mon[20143]: osdmap e48: 7 total, 7 up, 7 in 2024-08-19T14:09:45.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:45 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-08-19T14:09:45.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:45 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:09:45.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:45 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:09:45.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:45 smithi031 ceph-mon[16336]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-08-19T14:09:45.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:45 smithi031 ceph-mon[16336]: osd.6 [v2:172.21.15.32:6816/1131035844,v1:172.21.15.32:6817/1131035844] boot 2024-08-19T14:09:45.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:45 smithi031 ceph-mon[16336]: osdmap e48: 7 total, 7 up, 7 in 2024-08-19T14:09:45.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:45 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-08-19T14:09:46.624 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:46 smithi032 ceph-mon[20004]: osdmap e49: 7 total, 7 up, 7 in 2024-08-19T14:09:46.624 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:46 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:09:46.624 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:46 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:09:46.624 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:46 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:09:46.624 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:46 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:09:46.624 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:46 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:09:46.625 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:46 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:09:46.625 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:46 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:09:46.625 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:46 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:09:46.625 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:46 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:09:46.625 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:46 smithi032 ceph-mon[20004]: osdmap e50: 7 total, 7 up, 7 in 2024-08-19T14:09:46.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:46 smithi045 ceph-mon[20143]: osdmap e49: 7 total, 7 up, 7 in 2024-08-19T14:09:46.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:46 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:09:46.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:46 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:09:46.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:46 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:09:46.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:46 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:09:46.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:46 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:09:46.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:46 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:09:46.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:46 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:09:46.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:46 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:09:46.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:46 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:09:46.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:46 smithi045 ceph-mon[20143]: osdmap e50: 7 total, 7 up, 7 in 2024-08-19T14:09:46.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:46 smithi031 ceph-mon[16336]: osdmap e49: 7 total, 7 up, 7 in 2024-08-19T14:09:46.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:46 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:09:46.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:46 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:09:46.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:46 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:09:46.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:46 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:09:46.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:46 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:09:46.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:46 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:09:46.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:46 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:09:46.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:46 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:09:46.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:46 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:09:46.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:46 smithi031 ceph-mon[16336]: osdmap e50: 7 total, 7 up, 7 in 2024-08-19T14:09:47.653 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:47 smithi032 ceph-mon[20004]: pgmap v145: 1 pgs: 1 peering; 0 B data, 197 MiB used, 626 GiB / 626 GiB avail 2024-08-19T14:09:47.653 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:47 smithi032 ceph-mon[20004]: Detected new or changed devices on smithi032 2024-08-19T14:09:47.653 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:47 smithi032 ceph-mon[20004]: Adjusting osd_memory_target on smithi032 to 5697M 2024-08-19T14:09:47.656 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-19T14:09:47.683 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph orch daemon add osd smithi032:vg_nvme/lv_1 2024-08-19T14:09:47.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:47 smithi045 ceph-mon[20143]: pgmap v145: 1 pgs: 1 peering; 0 B data, 197 MiB used, 626 GiB / 626 GiB avail 2024-08-19T14:09:47.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:47 smithi045 ceph-mon[20143]: Detected new or changed devices on smithi032 2024-08-19T14:09:47.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:47 smithi045 ceph-mon[20143]: Adjusting osd_memory_target on smithi032 to 5697M 2024-08-19T14:09:47.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:47 smithi031 ceph-mon[16336]: pgmap v145: 1 pgs: 1 peering; 0 B data, 197 MiB used, 626 GiB / 626 GiB avail 2024-08-19T14:09:47.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:47 smithi031 ceph-mon[16336]: Detected new or changed devices on smithi032 2024-08-19T14:09:47.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:47 smithi031 ceph-mon[16336]: Adjusting osd_memory_target on smithi032 to 5697M 2024-08-19T14:09:47.943 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:09:48.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:48 smithi032 ceph-mon[20004]: osdmap e51: 7 total, 7 up, 7 in 2024-08-19T14:09:48.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:48 smithi045 ceph-mon[20143]: osdmap e51: 7 total, 7 up, 7 in 2024-08-19T14:09:48.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:48 smithi031 ceph-mon[16336]: osdmap e51: 7 total, 7 up, 7 in 2024-08-19T14:09:49.695 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:49 smithi032 ceph-mon[20004]: pgmap v148: 1 pgs: 1 peering; 0 B data, 197 MiB used, 626 GiB / 626 GiB avail 2024-08-19T14:09:49.695 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:49 smithi032 ceph-mon[20004]: osdmap e52: 7 total, 7 up, 7 in 2024-08-19T14:09:49.695 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:49 smithi032 ceph-mon[20004]: from='client.24296 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T14:09:49.695 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:49 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-19T14:09:49.695 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:49 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-19T14:09:49.695 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:49 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:09:49.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:49 smithi045 ceph-mon[20143]: pgmap v148: 1 pgs: 1 peering; 0 B data, 197 MiB used, 626 GiB / 626 GiB avail 2024-08-19T14:09:49.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:49 smithi045 ceph-mon[20143]: osdmap e52: 7 total, 7 up, 7 in 2024-08-19T14:09:49.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:49 smithi045 ceph-mon[20143]: from='client.24296 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T14:09:49.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:49 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-19T14:09:49.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:49 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-19T14:09:49.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:49 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:09:49.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:49 smithi031 ceph-mon[16336]: pgmap v148: 1 pgs: 1 peering; 0 B data, 197 MiB used, 626 GiB / 626 GiB avail 2024-08-19T14:09:49.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:49 smithi031 ceph-mon[16336]: osdmap e52: 7 total, 7 up, 7 in 2024-08-19T14:09:49.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:49 smithi031 ceph-mon[16336]: from='client.24296 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T14:09:49.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:49 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-19T14:09:49.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:49 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-19T14:09:49.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:49 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:09:51.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:51 smithi032 ceph-mon[20004]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-08-19T14:09:51.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:51 smithi032 ceph-mon[20004]: from='client.? 172.21.15.32:0/3561181213' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "263433e5-70b6-4192-86ff-dc2ad95927eb"}]: dispatch 2024-08-19T14:09:51.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:51 smithi032 ceph-mon[20004]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "263433e5-70b6-4192-86ff-dc2ad95927eb"}]: dispatch 2024-08-19T14:09:51.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:51 smithi032 ceph-mon[20004]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "263433e5-70b6-4192-86ff-dc2ad95927eb"}]': finished 2024-08-19T14:09:51.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:51 smithi032 ceph-mon[20004]: osdmap e53: 8 total, 7 up, 8 in 2024-08-19T14:09:51.760 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:51 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-08-19T14:09:51.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:51 smithi045 ceph-mon[20143]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-08-19T14:09:51.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:51 smithi045 ceph-mon[20143]: from='client.? 172.21.15.32:0/3561181213' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "263433e5-70b6-4192-86ff-dc2ad95927eb"}]: dispatch 2024-08-19T14:09:51.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:51 smithi045 ceph-mon[20143]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "263433e5-70b6-4192-86ff-dc2ad95927eb"}]: dispatch 2024-08-19T14:09:51.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:51 smithi045 ceph-mon[20143]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "263433e5-70b6-4192-86ff-dc2ad95927eb"}]': finished 2024-08-19T14:09:51.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:51 smithi045 ceph-mon[20143]: osdmap e53: 8 total, 7 up, 8 in 2024-08-19T14:09:51.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:51 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-08-19T14:09:51.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:51 smithi031 ceph-mon[16336]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-08-19T14:09:51.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:51 smithi031 ceph-mon[16336]: from='client.? 172.21.15.32:0/3561181213' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "263433e5-70b6-4192-86ff-dc2ad95927eb"}]: dispatch 2024-08-19T14:09:51.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:51 smithi031 ceph-mon[16336]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "263433e5-70b6-4192-86ff-dc2ad95927eb"}]: dispatch 2024-08-19T14:09:51.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:51 smithi031 ceph-mon[16336]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "263433e5-70b6-4192-86ff-dc2ad95927eb"}]': finished 2024-08-19T14:09:51.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:51 smithi031 ceph-mon[16336]: osdmap e53: 8 total, 7 up, 8 in 2024-08-19T14:09:51.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:51 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-08-19T14:09:52.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:52 smithi032 ceph-mon[20004]: from='client.? 172.21.15.32:0/3971285522' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-19T14:09:52.767 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:52 smithi031 ceph-mon[16336]: from='client.? 172.21.15.32:0/3971285522' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-19T14:09:52.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:52 smithi045 ceph-mon[20143]: from='client.? 172.21.15.32:0/3971285522' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-19T14:09:53.748 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:53 smithi032 ceph-mon[20004]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-08-19T14:09:53.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:53 smithi045 ceph-mon[20143]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-08-19T14:09:53.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:53 smithi031 ceph-mon[16336]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-08-19T14:09:55.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:55 smithi032 ceph-mon[20004]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-08-19T14:09:55.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:55 smithi045 ceph-mon[20143]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-08-19T14:09:55.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:55 smithi031 ceph-mon[16336]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-08-19T14:09:57.633 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:57 smithi032 ceph-mon[20004]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-08-19T14:09:57.633 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:57 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-08-19T14:09:57.633 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:57 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:09:57.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:57 smithi045 ceph-mon[20143]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-08-19T14:09:57.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:57 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-08-19T14:09:57.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:57 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:09:57.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:57 smithi031 ceph-mon[16336]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-08-19T14:09:57.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:57 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-08-19T14:09:57.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:57 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:09:58.593 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:58 smithi032 ceph-mon[20004]: Deploying daemon osd.7 on smithi032 2024-08-19T14:09:58.593 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:58 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:09:58.593 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:58 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:09:58.593 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:58 smithi032 ceph-mon[20004]: osdmap e54: 8 total, 7 up, 8 in 2024-08-19T14:09:58.593 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:58 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-08-19T14:09:58.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:58 smithi045 ceph-mon[20143]: Deploying daemon osd.7 on smithi032 2024-08-19T14:09:58.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:58 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:09:58.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:58 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:09:58.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:58 smithi045 ceph-mon[20143]: osdmap e54: 8 total, 7 up, 8 in 2024-08-19T14:09:58.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:58 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-08-19T14:09:58.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:58 smithi031 ceph-mon[16336]: Deploying daemon osd.7 on smithi032 2024-08-19T14:09:58.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:58 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:09:58.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:58 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:09:58.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:58 smithi031 ceph-mon[16336]: osdmap e54: 8 total, 7 up, 8 in 2024-08-19T14:09:58.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:58 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-08-19T14:09:59.600 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:09:59 smithi032 ceph-mon[20004]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-08-19T14:09:59.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:09:59 smithi045 ceph-mon[20143]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-08-19T14:09:59.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:09:59 smithi031 ceph-mon[16336]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-08-19T14:10:00.574 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:00 smithi032 ceph-mon[20004]: overall HEALTH_OK 2024-08-19T14:10:00.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:00 smithi045 ceph-mon[20143]: overall HEALTH_OK 2024-08-19T14:10:00.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:00 smithi031 ceph-mon[16336]: overall HEALTH_OK 2024-08-19T14:10:01.760 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:01 smithi032 ceph-mon[20004]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-08-19T14:10:01.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:01 smithi045 ceph-mon[20143]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-08-19T14:10:01.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:01 smithi031 ceph-mon[16336]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-08-19T14:10:02.657 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:02 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:10:02.657 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:02 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:02.657 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:02 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:02.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:02 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:10:02.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:02 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:02.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:02 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:02.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:02 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:10:02.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:02 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:02.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:02 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:03.689 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:03 smithi032 ceph-mon[20004]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-08-19T14:10:03.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:03 smithi045 ceph-mon[20143]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-08-19T14:10:03.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:03 smithi031 ceph-mon[16336]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-08-19T14:10:05.471 INFO:teuthology.orchestra.run.smithi032.stdout:Created osd(s) 7 on host 'smithi032' 2024-08-19T14:10:05.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:05 smithi032 ceph-mon[20004]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-08-19T14:10:05.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:05 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:05.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:05 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:05.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:05 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:10:05.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:05 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:10:05.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:05 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:10:05.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:05 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:10:05.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:05 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:10:05.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:05 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:05.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:05 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:10:05.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:05 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:10:05.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:05 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:10:05.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:05 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:05.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:05 smithi032 ceph-mon[20004]: from='osd.7 [v2:172.21.15.32:6824/2014939947,v1:172.21.15.32:6825/2014939947]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-08-19T14:10:05.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:05 smithi032 ceph-mon[20004]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-08-19T14:10:05.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:05 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:10:05.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:05 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:05.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:05 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:05.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:05 smithi045 ceph-mon[20143]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-08-19T14:10:05.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:05 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:05.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:05 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:05.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:05 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:10:05.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:05 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:10:05.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:05 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:10:05.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:05 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:10:05.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:05 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:10:05.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:05 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:05.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:05 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:10:05.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:05 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:10:05.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:05 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:10:05.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:05 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:05.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:05 smithi045 ceph-mon[20143]: from='osd.7 [v2:172.21.15.32:6824/2014939947,v1:172.21.15.32:6825/2014939947]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-08-19T14:10:05.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:05 smithi045 ceph-mon[20143]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-08-19T14:10:05.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:05 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:10:05.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:05 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:05.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:05 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:05.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:05 smithi031 ceph-mon[16336]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-08-19T14:10:05.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:05 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:05.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:05 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:05.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:05 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:10:05.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:05 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:10:05.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:05 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:10:05.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:05 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:10:05.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:05 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:10:05.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:05 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:05.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:05 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:10:05.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:05 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:10:05.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:05 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:10:05.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:05 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:05.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:05 smithi031 ceph-mon[16336]: from='osd.7 [v2:172.21.15.32:6824/2014939947,v1:172.21.15.32:6825/2014939947]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-08-19T14:10:05.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:05 smithi031 ceph-mon[16336]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-08-19T14:10:05.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:05 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:10:05.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:05 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:05.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:05 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:06.058 DEBUG:teuthology.orchestra.run.smithi032:osd.7> sudo journalctl -f -n 0 -u ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968@osd.7.service 2024-08-19T14:10:06.060 INFO:tasks.cephadm:Deploying osd.8 on smithi045 with /dev/vg_nvme/lv_4... 2024-08-19T14:10:06.061 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- lvm zap /dev/vg_nvme/lv_4 2024-08-19T14:10:06.331 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:10:06.747 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:06 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:10:06.747 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:06 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:10:06.747 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:06 smithi045 ceph-mon[20143]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-08-19T14:10:06.747 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:06 smithi045 ceph-mon[20143]: from='osd.7 [v2:172.21.15.32:6824/2014939947,v1:172.21.15.32:6825/2014939947]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-08-19T14:10:06.747 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:06 smithi045 ceph-mon[20143]: osdmap e55: 8 total, 7 up, 8 in 2024-08-19T14:10:06.747 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:06 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-08-19T14:10:06.748 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:06 smithi045 ceph-mon[20143]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-08-19T14:10:06.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:06 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:10:06.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:06 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:10:06.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:06 smithi032 ceph-mon[20004]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-08-19T14:10:06.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:06 smithi032 ceph-mon[20004]: from='osd.7 [v2:172.21.15.32:6824/2014939947,v1:172.21.15.32:6825/2014939947]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-08-19T14:10:06.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:06 smithi032 ceph-mon[20004]: osdmap e55: 8 total, 7 up, 8 in 2024-08-19T14:10:06.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:06 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-08-19T14:10:06.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:06 smithi032 ceph-mon[20004]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-08-19T14:10:06.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:06 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:10:06.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:06 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:10:06.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:06 smithi031 ceph-mon[16336]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-08-19T14:10:06.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:06 smithi031 ceph-mon[16336]: from='osd.7 [v2:172.21.15.32:6824/2014939947,v1:172.21.15.32:6825/2014939947]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-08-19T14:10:06.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:06 smithi031 ceph-mon[16336]: osdmap e55: 8 total, 7 up, 8 in 2024-08-19T14:10:06.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:06 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-08-19T14:10:06.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:06 smithi031 ceph-mon[16336]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-08-19T14:10:07.503 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:10:07 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7[41448]: 2024-08-19T14:10:07.186+0000 7f1329183640 -1 osd.7 0 waiting for initial osdmap 2024-08-19T14:10:07.503 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:10:07 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7[41448]: 2024-08-19T14:10:07.194+0000 7f132078f640 -1 osd.7 56 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-19T14:10:07.747 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:07 smithi045 ceph-mon[20143]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-08-19T14:10:07.747 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:07 smithi045 ceph-mon[20143]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]': finished 2024-08-19T14:10:07.747 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:07 smithi045 ceph-mon[20143]: osdmap e56: 8 total, 7 up, 8 in 2024-08-19T14:10:07.747 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:07 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-08-19T14:10:07.747 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:07 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-08-19T14:10:07.748 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:07 smithi045 ceph-mon[20143]: from='osd.7 ' entity='osd.7' 2024-08-19T14:10:07.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:07 smithi032 ceph-mon[20004]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-08-19T14:10:07.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:07 smithi032 ceph-mon[20004]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]': finished 2024-08-19T14:10:07.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:07 smithi032 ceph-mon[20004]: osdmap e56: 8 total, 7 up, 8 in 2024-08-19T14:10:07.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:07 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-08-19T14:10:07.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:07 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-08-19T14:10:07.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:07 smithi032 ceph-mon[20004]: from='osd.7 ' entity='osd.7' 2024-08-19T14:10:07.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:07 smithi031 ceph-mon[16336]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-08-19T14:10:07.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:07 smithi031 ceph-mon[16336]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]': finished 2024-08-19T14:10:07.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:07 smithi031 ceph-mon[16336]: osdmap e56: 8 total, 7 up, 8 in 2024-08-19T14:10:07.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:07 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-08-19T14:10:07.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:07 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-08-19T14:10:07.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:07 smithi031 ceph-mon[16336]: from='osd.7 ' entity='osd.7' 2024-08-19T14:10:08.722 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:08 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:10:08.722 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:08 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:10:08.722 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:08 smithi032 ceph-mon[20004]: osd.7 [v2:172.21.15.32:6824/2014939947,v1:172.21.15.32:6825/2014939947] boot 2024-08-19T14:10:08.722 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:08 smithi032 ceph-mon[20004]: osdmap e57: 8 total, 8 up, 8 in 2024-08-19T14:10:08.722 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:08 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-08-19T14:10:08.733 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:08 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:10:08.733 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:08 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:10:08.733 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:08 smithi045 ceph-mon[20143]: osd.7 [v2:172.21.15.32:6824/2014939947,v1:172.21.15.32:6825/2014939947] boot 2024-08-19T14:10:08.733 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:08 smithi045 ceph-mon[20143]: osdmap e57: 8 total, 8 up, 8 in 2024-08-19T14:10:08.733 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:08 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-08-19T14:10:08.737 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-08-19T14:10:08.775 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph orch daemon add osd smithi045:vg_nvme/lv_4 2024-08-19T14:10:08.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:08 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:10:08.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:08 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:10:08.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:08 smithi031 ceph-mon[16336]: osd.7 [v2:172.21.15.32:6824/2014939947,v1:172.21.15.32:6825/2014939947] boot 2024-08-19T14:10:08.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:08 smithi031 ceph-mon[16336]: osdmap e57: 8 total, 8 up, 8 in 2024-08-19T14:10:08.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:08 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-08-19T14:10:09.030 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:10:09.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:09 smithi045 ceph-mon[20143]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-08-19T14:10:09.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:09 smithi045 ceph-mon[20143]: osdmap e58: 8 total, 8 up, 8 in 2024-08-19T14:10:09.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:09 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:09.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:09 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:09.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:09 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:10:09.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:09 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:10:09.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:09 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:10:09.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:09 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:10:09.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:09 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:09.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:09 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:10:09.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:09 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:10:09.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:09 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:09.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:09 smithi031 ceph-mon[16336]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-08-19T14:10:09.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:09 smithi031 ceph-mon[16336]: osdmap e58: 8 total, 8 up, 8 in 2024-08-19T14:10:09.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:09 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:09.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:09 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:09.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:09 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:10:09.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:09 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:10:09.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:09 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:10:09.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:09 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:10:09.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:09 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:09.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:09 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:10:09.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:09 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:10:09.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:09 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:10.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:09 smithi032 ceph-mon[20004]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-08-19T14:10:10.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:09 smithi032 ceph-mon[20004]: osdmap e58: 8 total, 8 up, 8 in 2024-08-19T14:10:10.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:09 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:10.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:09 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:10.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:09 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:10:10.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:09 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:10:10.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:09 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:10:10.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:09 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:10:10.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:09 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:10.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:09 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:10:10.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:09 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:10:10.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:09 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:10.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:10 smithi045 ceph-mon[20143]: Detected new or changed devices on smithi032 2024-08-19T14:10:10.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:10 smithi045 ceph-mon[20143]: Adjusting osd_memory_target on smithi032 to 4273M 2024-08-19T14:10:10.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:10 smithi045 ceph-mon[20143]: osdmap e59: 8 total, 8 up, 8 in 2024-08-19T14:10:10.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:10 smithi045 ceph-mon[20143]: from='client.24244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T14:10:10.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:10 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-19T14:10:10.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:10 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-19T14:10:10.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:10 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:10:10.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:10 smithi031 ceph-mon[16336]: Detected new or changed devices on smithi032 2024-08-19T14:10:10.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:10 smithi031 ceph-mon[16336]: Adjusting osd_memory_target on smithi032 to 4273M 2024-08-19T14:10:10.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:10 smithi031 ceph-mon[16336]: osdmap e59: 8 total, 8 up, 8 in 2024-08-19T14:10:10.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:10 smithi031 ceph-mon[16336]: from='client.24244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T14:10:10.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:10 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-19T14:10:10.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:10 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-19T14:10:10.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:10 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:10:11.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:10 smithi032 ceph-mon[20004]: Detected new or changed devices on smithi032 2024-08-19T14:10:11.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:10 smithi032 ceph-mon[20004]: Adjusting osd_memory_target on smithi032 to 4273M 2024-08-19T14:10:11.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:10 smithi032 ceph-mon[20004]: osdmap e59: 8 total, 8 up, 8 in 2024-08-19T14:10:11.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:10 smithi032 ceph-mon[20004]: from='client.24244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T14:10:11.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:10 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-19T14:10:11.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:10 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-19T14:10:11.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:10 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:10:11.741 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:11 smithi045 ceph-mon[20143]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 612 MiB used, 715 GiB / 715 GiB avail 2024-08-19T14:10:11.741 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:11 smithi045 ceph-mon[20143]: osdmap e60: 8 total, 8 up, 8 in 2024-08-19T14:10:11.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:11 smithi031 ceph-mon[16336]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 612 MiB used, 715 GiB / 715 GiB avail 2024-08-19T14:10:11.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:11 smithi031 ceph-mon[16336]: osdmap e60: 8 total, 8 up, 8 in 2024-08-19T14:10:12.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:11 smithi032 ceph-mon[20004]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 612 MiB used, 715 GiB / 715 GiB avail 2024-08-19T14:10:12.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:11 smithi032 ceph-mon[20004]: osdmap e60: 8 total, 8 up, 8 in 2024-08-19T14:10:12.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:12 smithi031 ceph-mon[16336]: osdmap e61: 8 total, 8 up, 8 in 2024-08-19T14:10:12.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:12 smithi031 ceph-mon[16336]: from='client.? 172.21.15.45:0/4070918841' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2bbcad36-0c74-4528-9fb9-a8812afa1662"}]: dispatch 2024-08-19T14:10:12.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:12 smithi031 ceph-mon[16336]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2bbcad36-0c74-4528-9fb9-a8812afa1662"}]: dispatch 2024-08-19T14:10:12.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:12 smithi031 ceph-mon[16336]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2bbcad36-0c74-4528-9fb9-a8812afa1662"}]': finished 2024-08-19T14:10:12.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:12 smithi031 ceph-mon[16336]: osdmap e62: 9 total, 8 up, 9 in 2024-08-19T14:10:12.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:12 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-08-19T14:10:13.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:12 smithi032 ceph-mon[20004]: osdmap e61: 8 total, 8 up, 8 in 2024-08-19T14:10:13.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:12 smithi032 ceph-mon[20004]: from='client.? 172.21.15.45:0/4070918841' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2bbcad36-0c74-4528-9fb9-a8812afa1662"}]: dispatch 2024-08-19T14:10:13.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:12 smithi032 ceph-mon[20004]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2bbcad36-0c74-4528-9fb9-a8812afa1662"}]: dispatch 2024-08-19T14:10:13.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:12 smithi032 ceph-mon[20004]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2bbcad36-0c74-4528-9fb9-a8812afa1662"}]': finished 2024-08-19T14:10:13.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:12 smithi032 ceph-mon[20004]: osdmap e62: 9 total, 8 up, 9 in 2024-08-19T14:10:13.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:12 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-08-19T14:10:13.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:12 smithi045 ceph-mon[20143]: osdmap e61: 8 total, 8 up, 8 in 2024-08-19T14:10:13.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:12 smithi045 ceph-mon[20143]: from='client.? 172.21.15.45:0/4070918841' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2bbcad36-0c74-4528-9fb9-a8812afa1662"}]: dispatch 2024-08-19T14:10:13.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:12 smithi045 ceph-mon[20143]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2bbcad36-0c74-4528-9fb9-a8812afa1662"}]: dispatch 2024-08-19T14:10:13.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:12 smithi045 ceph-mon[20143]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2bbcad36-0c74-4528-9fb9-a8812afa1662"}]': finished 2024-08-19T14:10:13.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:12 smithi045 ceph-mon[20143]: osdmap e62: 9 total, 8 up, 9 in 2024-08-19T14:10:13.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:12 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-08-19T14:10:13.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:13 smithi031 ceph-mon[16336]: pgmap v170: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 612 MiB used, 715 GiB / 715 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-08-19T14:10:13.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:13 smithi031 ceph-mon[16336]: from='client.? 172.21.15.45:0/3363313748' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-19T14:10:13.867 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:10:13 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: 2024-08-19T14:10:13.678+0000 7ff487bea640 -1 calc_pg_upmaps abort due to max <= 0 2024-08-19T14:10:14.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:13 smithi032 ceph-mon[20004]: pgmap v170: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 612 MiB used, 715 GiB / 715 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-08-19T14:10:14.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:13 smithi032 ceph-mon[20004]: from='client.? 172.21.15.45:0/3363313748' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-19T14:10:14.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:13 smithi045 ceph-mon[20143]: pgmap v170: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 612 MiB used, 715 GiB / 715 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-08-19T14:10:14.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:13 smithi045 ceph-mon[20143]: from='client.? 172.21.15.45:0/3363313748' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-19T14:10:15.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:15 smithi031 ceph-mon[16336]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 125 KiB/s, 0 objects/s recovering 2024-08-19T14:10:15.939 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:15 smithi045 ceph-mon[20143]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 125 KiB/s, 0 objects/s recovering 2024-08-19T14:10:16.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:15 smithi032 ceph-mon[20004]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 125 KiB/s, 0 objects/s recovering 2024-08-19T14:10:17.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:17 smithi031 ceph-mon[16336]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-08-19T14:10:17.880 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:17 smithi045 ceph-mon[20143]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-08-19T14:10:18.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:17 smithi032 ceph-mon[20004]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-08-19T14:10:18.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:18 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-08-19T14:10:18.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:18 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:10:18.866 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:18 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-08-19T14:10:18.867 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:18 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:10:19.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:18 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-08-19T14:10:19.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:18 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:10:19.731 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:19 smithi045 ceph-mon[20143]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-08-19T14:10:19.731 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:19 smithi045 ceph-mon[20143]: Deploying daemon osd.8 on smithi045 2024-08-19T14:10:19.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:19 smithi031 ceph-mon[16336]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-08-19T14:10:19.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:19 smithi031 ceph-mon[16336]: Deploying daemon osd.8 on smithi045 2024-08-19T14:10:20.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:19 smithi032 ceph-mon[20004]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-08-19T14:10:20.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:19 smithi032 ceph-mon[20004]: Deploying daemon osd.8 on smithi045 2024-08-19T14:10:21.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:21 smithi031 ceph-mon[16336]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-08-19T14:10:22.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:21 smithi032 ceph-mon[20004]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-08-19T14:10:22.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:21 smithi045 ceph-mon[20143]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-08-19T14:10:23.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:23 smithi031 ceph-mon[16336]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-08-19T14:10:23.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:23 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:10:23.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:23 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:23.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:23 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:24.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:23 smithi032 ceph-mon[20004]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-08-19T14:10:24.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:23 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:10:24.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:23 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:24.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:23 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:24.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:23 smithi045 ceph-mon[20143]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-08-19T14:10:24.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:23 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:10:24.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:23 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:24.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:23 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:25.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:25 smithi031 ceph-mon[16336]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-08-19T14:10:25.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:25 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:25.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:25 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:25.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:25 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:10:25.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:25 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:10:25.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:25 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:26.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:25 smithi032 ceph-mon[20004]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-08-19T14:10:26.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:25 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:26.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:25 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:26.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:25 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:10:26.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:25 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:10:26.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:25 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:26.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:25 smithi045 ceph-mon[20143]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-08-19T14:10:26.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:25 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:26.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:25 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:26.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:25 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:10:26.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:25 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:10:26.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:25 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:26.378 INFO:teuthology.orchestra.run.smithi045.stdout:Created osd(s) 8 on host 'smithi045' 2024-08-19T14:10:26.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:26 smithi031 ceph-mon[16336]: from='osd.8 [v2:172.21.15.45:6800/1445140892,v1:172.21.15.45:6801/1445140892]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-08-19T14:10:26.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:26 smithi031 ceph-mon[16336]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-08-19T14:10:26.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:26 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:10:26.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:26 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:26.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:26 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:26.984 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:26 smithi045 ceph-mon[20143]: from='osd.8 [v2:172.21.15.45:6800/1445140892,v1:172.21.15.45:6801/1445140892]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-08-19T14:10:26.984 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:26 smithi045 ceph-mon[20143]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-08-19T14:10:26.984 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:26 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:10:26.984 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:26 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:26.984 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:26 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:27.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:26 smithi032 ceph-mon[20004]: from='osd.8 [v2:172.21.15.45:6800/1445140892,v1:172.21.15.45:6801/1445140892]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-08-19T14:10:27.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:26 smithi032 ceph-mon[20004]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-08-19T14:10:27.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:26 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:10:27.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:26 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:27.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:26 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:27.014 DEBUG:teuthology.orchestra.run.smithi045:osd.8> sudo journalctl -f -n 0 -u ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968@osd.8.service 2024-08-19T14:10:27.057 INFO:tasks.cephadm:Deploying osd.9 on smithi045 with /dev/vg_nvme/lv_3... 2024-08-19T14:10:27.057 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- lvm zap /dev/vg_nvme/lv_3 2024-08-19T14:10:27.840 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:27 smithi045 ceph-mon[20143]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-08-19T14:10:27.840 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:27 smithi045 ceph-mon[20143]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-08-19T14:10:27.840 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:27 smithi045 ceph-mon[20143]: osdmap e63: 9 total, 8 up, 9 in 2024-08-19T14:10:27.841 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:27 smithi045 ceph-mon[20143]: from='osd.8 [v2:172.21.15.45:6800/1445140892,v1:172.21.15.45:6801/1445140892]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-08-19T14:10:27.841 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:27 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-08-19T14:10:27.841 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:27 smithi045 ceph-mon[20143]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-08-19T14:10:27.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:27 smithi031 ceph-mon[16336]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-08-19T14:10:27.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:27 smithi031 ceph-mon[16336]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-08-19T14:10:27.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:27 smithi031 ceph-mon[16336]: osdmap e63: 9 total, 8 up, 9 in 2024-08-19T14:10:27.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:27 smithi031 ceph-mon[16336]: from='osd.8 [v2:172.21.15.45:6800/1445140892,v1:172.21.15.45:6801/1445140892]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-08-19T14:10:27.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:27 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-08-19T14:10:27.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:27 smithi031 ceph-mon[16336]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-08-19T14:10:28.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:27 smithi032 ceph-mon[20004]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-08-19T14:10:28.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:27 smithi032 ceph-mon[20004]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-08-19T14:10:28.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:27 smithi032 ceph-mon[20004]: osdmap e63: 9 total, 8 up, 9 in 2024-08-19T14:10:28.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:27 smithi032 ceph-mon[20004]: from='osd.8 [v2:172.21.15.45:6800/1445140892,v1:172.21.15.45:6801/1445140892]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-08-19T14:10:28.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:27 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-08-19T14:10:28.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:27 smithi032 ceph-mon[20004]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-08-19T14:10:28.091 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:10:27 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:10:27.871+0000 7f99538c7640 -1 osd.8 0 waiting for initial osdmap 2024-08-19T14:10:28.091 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:10:27 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:10:27.878+0000 7f994b6e7640 -1 osd.8 64 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-19T14:10:28.598 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:10:28.722 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:28 smithi045 ceph-mon[20143]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]': finished 2024-08-19T14:10:28.722 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:28 smithi045 ceph-mon[20143]: osdmap e64: 9 total, 8 up, 9 in 2024-08-19T14:10:28.722 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:28 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-08-19T14:10:28.722 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:28 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-08-19T14:10:28.722 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:28 smithi045 ceph-mon[20143]: from='osd.8 ' entity='osd.8' 2024-08-19T14:10:29.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:28 smithi032 ceph-mon[20004]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]': finished 2024-08-19T14:10:29.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:28 smithi032 ceph-mon[20004]: osdmap e64: 9 total, 8 up, 9 in 2024-08-19T14:10:29.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:28 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-08-19T14:10:29.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:28 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-08-19T14:10:29.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:28 smithi032 ceph-mon[20004]: from='osd.8 ' entity='osd.8' 2024-08-19T14:10:29.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:28 smithi031 ceph-mon[16336]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]': finished 2024-08-19T14:10:29.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:28 smithi031 ceph-mon[16336]: osdmap e64: 9 total, 8 up, 9 in 2024-08-19T14:10:29.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:28 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-08-19T14:10:29.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:28 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-08-19T14:10:29.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:28 smithi031 ceph-mon[16336]: from='osd.8 ' entity='osd.8' 2024-08-19T14:10:29.995 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:29 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:10:29.996 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:29 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:10:29.996 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:29 smithi045 ceph-mon[20143]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-08-19T14:10:29.996 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:29 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-08-19T14:10:29.996 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:29 smithi045 ceph-mon[20143]: osd.8 [v2:172.21.15.45:6800/1445140892,v1:172.21.15.45:6801/1445140892] boot 2024-08-19T14:10:29.996 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:29 smithi045 ceph-mon[20143]: osdmap e65: 9 total, 9 up, 9 in 2024-08-19T14:10:29.996 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:29 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-08-19T14:10:30.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:29 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:10:30.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:29 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:10:30.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:29 smithi032 ceph-mon[20004]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-08-19T14:10:30.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:29 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-08-19T14:10:30.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:29 smithi032 ceph-mon[20004]: osd.8 [v2:172.21.15.45:6800/1445140892,v1:172.21.15.45:6801/1445140892] boot 2024-08-19T14:10:30.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:29 smithi032 ceph-mon[20004]: osdmap e65: 9 total, 9 up, 9 in 2024-08-19T14:10:30.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:29 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-08-19T14:10:30.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:29 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:10:30.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:29 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:10:30.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:29 smithi031 ceph-mon[16336]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-08-19T14:10:30.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:29 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-08-19T14:10:30.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:29 smithi031 ceph-mon[16336]: osd.8 [v2:172.21.15.45:6800/1445140892,v1:172.21.15.45:6801/1445140892] boot 2024-08-19T14:10:30.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:29 smithi031 ceph-mon[16336]: osdmap e65: 9 total, 9 up, 9 in 2024-08-19T14:10:30.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:29 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-08-19T14:10:31.097 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:30 smithi045 ceph-mon[20143]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-08-19T14:10:31.097 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:30 smithi045 ceph-mon[20143]: osdmap e66: 9 total, 9 up, 9 in 2024-08-19T14:10:31.097 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:30 smithi045 ceph-mon[20143]: Detected new or changed devices on smithi045 2024-08-19T14:10:31.097 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:30 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:31.097 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:30 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:31.098 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:30 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:10:31.098 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:30 smithi045 ceph-mon[20143]: Adjusting osd_memory_target on smithi045 to 21188M 2024-08-19T14:10:31.098 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:30 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:31.098 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:30 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:10:31.098 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:30 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:10:31.098 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:30 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:31.098 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:30 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:10:31.098 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:30 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:10:31.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:30 smithi032 ceph-mon[20004]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-08-19T14:10:31.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:30 smithi032 ceph-mon[20004]: osdmap e66: 9 total, 9 up, 9 in 2024-08-19T14:10:31.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:30 smithi032 ceph-mon[20004]: Detected new or changed devices on smithi045 2024-08-19T14:10:31.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:30 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:31.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:30 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:31.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:30 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:10:31.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:30 smithi032 ceph-mon[20004]: Adjusting osd_memory_target on smithi045 to 21188M 2024-08-19T14:10:31.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:30 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:31.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:30 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:10:31.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:30 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:10:31.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:30 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:31.259 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:30 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:10:31.259 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:30 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:10:31.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:30 smithi031 ceph-mon[16336]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-08-19T14:10:31.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:30 smithi031 ceph-mon[16336]: osdmap e66: 9 total, 9 up, 9 in 2024-08-19T14:10:31.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:30 smithi031 ceph-mon[16336]: Detected new or changed devices on smithi045 2024-08-19T14:10:31.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:30 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:31.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:30 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:31.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:30 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:10:31.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:30 smithi031 ceph-mon[16336]: Adjusting osd_memory_target on smithi045 to 21188M 2024-08-19T14:10:31.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:30 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:31.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:30 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:10:31.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:30 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:10:31.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:30 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:31.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:30 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:10:31.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:30 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:10:31.866 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-08-19T14:10:31.901 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph orch daemon add osd smithi045:vg_nvme/lv_3 2024-08-19T14:10:32.157 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:10:32.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:31 smithi032 ceph-mon[20004]: osdmap e67: 9 total, 9 up, 9 in 2024-08-19T14:10:32.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:31 smithi045 ceph-mon[20143]: osdmap e67: 9 total, 9 up, 9 in 2024-08-19T14:10:32.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:31 smithi031 ceph-mon[16336]: osdmap e67: 9 total, 9 up, 9 in 2024-08-19T14:10:33.187 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:32 smithi045 ceph-mon[20143]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-08-19T14:10:33.188 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:32 smithi045 ceph-mon[20143]: osdmap e68: 9 total, 9 up, 9 in 2024-08-19T14:10:33.188 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:32 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-19T14:10:33.188 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:32 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-19T14:10:33.188 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:32 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:10:33.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:32 smithi032 ceph-mon[20004]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-08-19T14:10:33.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:32 smithi032 ceph-mon[20004]: osdmap e68: 9 total, 9 up, 9 in 2024-08-19T14:10:33.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:32 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-19T14:10:33.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:32 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-19T14:10:33.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:32 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:10:33.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:32 smithi031 ceph-mon[16336]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-08-19T14:10:33.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:32 smithi031 ceph-mon[16336]: osdmap e68: 9 total, 9 up, 9 in 2024-08-19T14:10:33.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:32 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-19T14:10:33.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:32 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-19T14:10:33.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:32 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:10:34.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:33 smithi032 ceph-mon[20004]: from='client.24271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T14:10:34.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:33 smithi032 ceph-mon[20004]: osdmap e69: 9 total, 9 up, 9 in 2024-08-19T14:10:34.274 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:33 smithi045 ceph-mon[20143]: from='client.24271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T14:10:34.274 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:33 smithi045 ceph-mon[20143]: osdmap e69: 9 total, 9 up, 9 in 2024-08-19T14:10:34.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:33 smithi031 ceph-mon[16336]: from='client.24271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T14:10:34.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:33 smithi031 ceph-mon[16336]: osdmap e69: 9 total, 9 up, 9 in 2024-08-19T14:10:35.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:34 smithi032 ceph-mon[20004]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-08-19T14:10:35.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:34 smithi045 ceph-mon[20143]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-08-19T14:10:35.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:34 smithi031 ceph-mon[16336]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-08-19T14:10:36.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:35 smithi032 ceph-mon[20004]: from='client.? 172.21.15.45:0/1001248071' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c8ee4277-3bf9-46bf-9cc0-152c1c4b9508"}]: dispatch 2024-08-19T14:10:36.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:35 smithi032 ceph-mon[20004]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c8ee4277-3bf9-46bf-9cc0-152c1c4b9508"}]: dispatch 2024-08-19T14:10:36.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:35 smithi032 ceph-mon[20004]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c8ee4277-3bf9-46bf-9cc0-152c1c4b9508"}]': finished 2024-08-19T14:10:36.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:35 smithi032 ceph-mon[20004]: osdmap e70: 10 total, 9 up, 10 in 2024-08-19T14:10:36.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:35 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-08-19T14:10:36.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:35 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:10:36.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:35 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:10:36.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:35 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:10:36.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:35 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:10:36.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:35 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:10:36.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:35 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:10:36.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:35 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:36.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:35 smithi032 ceph-mon[20004]: from='client.? 172.21.15.45:0/49441801' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-19T14:10:36.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:35 smithi045 ceph-mon[20143]: from='client.? 172.21.15.45:0/1001248071' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c8ee4277-3bf9-46bf-9cc0-152c1c4b9508"}]: dispatch 2024-08-19T14:10:36.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:35 smithi045 ceph-mon[20143]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c8ee4277-3bf9-46bf-9cc0-152c1c4b9508"}]: dispatch 2024-08-19T14:10:36.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:35 smithi045 ceph-mon[20143]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c8ee4277-3bf9-46bf-9cc0-152c1c4b9508"}]': finished 2024-08-19T14:10:36.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:35 smithi045 ceph-mon[20143]: osdmap e70: 10 total, 9 up, 10 in 2024-08-19T14:10:36.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:35 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-08-19T14:10:36.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:35 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:10:36.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:35 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:10:36.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:35 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:10:36.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:35 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:10:36.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:35 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:10:36.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:35 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:10:36.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:35 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:36.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:35 smithi045 ceph-mon[20143]: from='client.? 172.21.15.45:0/49441801' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-19T14:10:36.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:35 smithi031 ceph-mon[16336]: from='client.? 172.21.15.45:0/1001248071' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c8ee4277-3bf9-46bf-9cc0-152c1c4b9508"}]: dispatch 2024-08-19T14:10:36.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:35 smithi031 ceph-mon[16336]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c8ee4277-3bf9-46bf-9cc0-152c1c4b9508"}]: dispatch 2024-08-19T14:10:36.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:35 smithi031 ceph-mon[16336]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c8ee4277-3bf9-46bf-9cc0-152c1c4b9508"}]': finished 2024-08-19T14:10:36.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:35 smithi031 ceph-mon[16336]: osdmap e70: 10 total, 9 up, 10 in 2024-08-19T14:10:36.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:35 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-08-19T14:10:36.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:35 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:10:36.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:35 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:10:36.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:35 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:10:36.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:35 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:10:36.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:35 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:10:36.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:35 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:10:36.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:35 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:36.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:35 smithi031 ceph-mon[16336]: from='client.? 172.21.15.45:0/49441801' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-19T14:10:37.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:37 smithi031 ceph-mon[16336]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 107 KiB/s, 0 objects/s recovering 2024-08-19T14:10:37.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:37 smithi032 ceph-mon[20004]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 107 KiB/s, 0 objects/s recovering 2024-08-19T14:10:37.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:37 smithi045 ceph-mon[20143]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 107 KiB/s, 0 objects/s recovering 2024-08-19T14:10:39.362 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:39 smithi045 ceph-mon[20143]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-08-19T14:10:39.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:39 smithi031 ceph-mon[16336]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-08-19T14:10:39.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:39 smithi032 ceph-mon[20004]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-08-19T14:10:41.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:41 smithi031 ceph-mon[16336]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-08-19T14:10:41.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:41 smithi031 ceph-mon[16336]: 1.0 deep-scrub starts 2024-08-19T14:10:41.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:41 smithi031 ceph-mon[16336]: 1.0 deep-scrub ok 2024-08-19T14:10:41.381 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:41 smithi045 ceph-mon[20143]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-08-19T14:10:41.381 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:41 smithi045 ceph-mon[20143]: 1.0 deep-scrub starts 2024-08-19T14:10:41.381 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:41 smithi045 ceph-mon[20143]: 1.0 deep-scrub ok 2024-08-19T14:10:41.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:41 smithi032 ceph-mon[20004]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-08-19T14:10:41.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:41 smithi032 ceph-mon[20004]: 1.0 deep-scrub starts 2024-08-19T14:10:41.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:41 smithi032 ceph-mon[20004]: 1.0 deep-scrub ok 2024-08-19T14:10:42.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:42 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-08-19T14:10:42.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:42 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:10:42.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:42 smithi045 ceph-mon[20143]: Deploying daemon osd.9 on smithi045 2024-08-19T14:10:42.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:42 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-08-19T14:10:42.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:42 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:10:42.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:42 smithi031 ceph-mon[16336]: Deploying daemon osd.9 on smithi045 2024-08-19T14:10:42.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:42 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-08-19T14:10:42.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:42 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:10:42.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:42 smithi032 ceph-mon[20004]: Deploying daemon osd.9 on smithi045 2024-08-19T14:10:43.237 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:43 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:10:43.237 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:43 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:10:43.238 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:43 smithi045 ceph-mon[20143]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-08-19T14:10:43.238 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:43 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:10:43.238 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:43 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:10:43.238 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:43 smithi045 ceph-mon[20143]: osdmap e71: 10 total, 9 up, 10 in 2024-08-19T14:10:43.238 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:43 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-08-19T14:10:43.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:43 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:10:43.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:43 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:10:43.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:43 smithi031 ceph-mon[16336]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-08-19T14:10:43.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:43 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:10:43.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:43 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:10:43.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:43 smithi031 ceph-mon[16336]: osdmap e71: 10 total, 9 up, 10 in 2024-08-19T14:10:43.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:43 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-08-19T14:10:43.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:43 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:10:43.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:43 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:10:43.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:43 smithi032 ceph-mon[20004]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-08-19T14:10:43.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:43 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:10:43.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:43 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:10:43.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:43 smithi032 ceph-mon[20004]: osdmap e71: 10 total, 9 up, 10 in 2024-08-19T14:10:43.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:43 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-08-19T14:10:45.173 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:45 smithi045 ceph-mon[20143]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-08-19T14:10:45.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:45 smithi032 ceph-mon[20004]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-08-19T14:10:45.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:45 smithi031 ceph-mon[16336]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-08-19T14:10:47.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:47 smithi045 ceph-mon[20143]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-08-19T14:10:47.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:47 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:10:47.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:47 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:47.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:47 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:47.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:47 smithi032 ceph-mon[20004]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-08-19T14:10:47.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:47 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:10:47.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:47 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:47.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:47 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:47.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:47 smithi031 ceph-mon[16336]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-08-19T14:10:47.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:47 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:10:47.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:47 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:47.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:47 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:49.436 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:49 smithi045 ceph-mon[20143]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-08-19T14:10:49.436 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:49 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:49.437 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:49 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:49.437 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:49 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:10:49.437 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:49 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:10:49.437 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:49 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:49.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:49 smithi032 ceph-mon[20004]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-08-19T14:10:49.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:49 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:49.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:49 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:49.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:49 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:10:49.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:49 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:10:49.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:49 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:49.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:49 smithi031 ceph-mon[16336]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-08-19T14:10:49.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:49 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:49.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:49 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:49.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:49 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:10:49.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:49 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:10:49.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:49 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:49.774 INFO:teuthology.orchestra.run.smithi045.stdout:Created osd(s) 9 on host 'smithi045' 2024-08-19T14:10:50.366 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:50 smithi045 ceph-mon[20143]: from='osd.9 [v2:172.21.15.45:6808/1213787169,v1:172.21.15.45:6809/1213787169]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-08-19T14:10:50.366 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:50 smithi045 ceph-mon[20143]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-08-19T14:10:50.367 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:50 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:10:50.367 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:50 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:50.367 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:50 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:50.395 DEBUG:teuthology.orchestra.run.smithi045:osd.9> sudo journalctl -f -n 0 -u ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968@osd.9.service 2024-08-19T14:10:50.398 INFO:tasks.cephadm:Deploying osd.10 on smithi045 with /dev/vg_nvme/lv_2... 2024-08-19T14:10:50.398 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- lvm zap /dev/vg_nvme/lv_2 2024-08-19T14:10:50.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:50 smithi032 ceph-mon[20004]: from='osd.9 [v2:172.21.15.45:6808/1213787169,v1:172.21.15.45:6809/1213787169]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-08-19T14:10:50.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:50 smithi032 ceph-mon[20004]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-08-19T14:10:50.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:50 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:10:50.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:50 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:50.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:50 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:50.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:50 smithi031 ceph-mon[16336]: from='osd.9 [v2:172.21.15.45:6808/1213787169,v1:172.21.15.45:6809/1213787169]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-08-19T14:10:50.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:50 smithi031 ceph-mon[16336]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-08-19T14:10:50.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:50 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:10:50.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:50 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:50.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:50 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:51.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:51 smithi045 ceph-mon[20143]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-08-19T14:10:51.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:51 smithi045 ceph-mon[20143]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-08-19T14:10:51.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:51 smithi045 ceph-mon[20143]: from='osd.9 [v2:172.21.15.45:6808/1213787169,v1:172.21.15.45:6809/1213787169]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-08-19T14:10:51.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:51 smithi045 ceph-mon[20143]: osdmap e72: 10 total, 9 up, 10 in 2024-08-19T14:10:51.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:51 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-08-19T14:10:51.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:51 smithi045 ceph-mon[20143]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-08-19T14:10:51.297 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:10:51 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:10:51.137+0000 7f9988d21640 -1 osd.9 0 waiting for initial osdmap 2024-08-19T14:10:51.297 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:10:51 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:10:51.147+0000 7f9984b49640 -1 osd.9 73 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-19T14:10:51.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:51 smithi032 ceph-mon[20004]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-08-19T14:10:51.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:51 smithi032 ceph-mon[20004]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-08-19T14:10:51.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:51 smithi032 ceph-mon[20004]: from='osd.9 [v2:172.21.15.45:6808/1213787169,v1:172.21.15.45:6809/1213787169]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-08-19T14:10:51.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:51 smithi032 ceph-mon[20004]: osdmap e72: 10 total, 9 up, 10 in 2024-08-19T14:10:51.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:51 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-08-19T14:10:51.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:51 smithi032 ceph-mon[20004]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-08-19T14:10:51.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:51 smithi031 ceph-mon[16336]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-08-19T14:10:51.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:51 smithi031 ceph-mon[16336]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-08-19T14:10:51.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:51 smithi031 ceph-mon[16336]: from='osd.9 [v2:172.21.15.45:6808/1213787169,v1:172.21.15.45:6809/1213787169]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-08-19T14:10:51.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:51 smithi031 ceph-mon[16336]: osdmap e72: 10 total, 9 up, 10 in 2024-08-19T14:10:51.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:51 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-08-19T14:10:51.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:51 smithi031 ceph-mon[16336]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-08-19T14:10:51.624 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:10:52.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:52 smithi045 ceph-mon[20143]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]': finished 2024-08-19T14:10:52.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:52 smithi045 ceph-mon[20143]: osdmap e73: 10 total, 9 up, 10 in 2024-08-19T14:10:52.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:52 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-08-19T14:10:52.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:52 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-08-19T14:10:52.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:52 smithi032 ceph-mon[20004]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]': finished 2024-08-19T14:10:52.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:52 smithi032 ceph-mon[20004]: osdmap e73: 10 total, 9 up, 10 in 2024-08-19T14:10:52.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:52 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-08-19T14:10:52.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:52 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-08-19T14:10:52.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:52 smithi031 ceph-mon[16336]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]': finished 2024-08-19T14:10:52.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:52 smithi031 ceph-mon[16336]: osdmap e73: 10 total, 9 up, 10 in 2024-08-19T14:10:52.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:52 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-08-19T14:10:52.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:52 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-08-19T14:10:53.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:53 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:10:53.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:53 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:10:53.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:53 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:10:53.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:53 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:10:53.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:53 smithi045 ceph-mon[20143]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-08-19T14:10:53.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:53 smithi045 ceph-mon[20143]: osd.9 [v2:172.21.15.45:6808/1213787169,v1:172.21.15.45:6809/1213787169] boot 2024-08-19T14:10:53.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:53 smithi045 ceph-mon[20143]: osdmap e74: 10 total, 10 up, 10 in 2024-08-19T14:10:53.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:53 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-08-19T14:10:53.409 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:53 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:10:53.409 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:53 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:10:53.409 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:53 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:10:53.409 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:53 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:10:53.409 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:53 smithi032 ceph-mon[20004]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-08-19T14:10:53.409 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:53 smithi032 ceph-mon[20004]: osd.9 [v2:172.21.15.45:6808/1213787169,v1:172.21.15.45:6809/1213787169] boot 2024-08-19T14:10:53.409 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:53 smithi032 ceph-mon[20004]: osdmap e74: 10 total, 10 up, 10 in 2024-08-19T14:10:53.409 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:53 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-08-19T14:10:53.518 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:53 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:10:53.518 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:53 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:10:53.518 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:53 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:10:53.518 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:53 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:10:53.518 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:53 smithi031 ceph-mon[16336]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-08-19T14:10:53.518 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:53 smithi031 ceph-mon[16336]: osd.9 [v2:172.21.15.45:6808/1213787169,v1:172.21.15.45:6809/1213787169] boot 2024-08-19T14:10:53.518 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:53 smithi031 ceph-mon[16336]: osdmap e74: 10 total, 10 up, 10 in 2024-08-19T14:10:53.518 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:53 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-08-19T14:10:54.480 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:54 smithi045 ceph-mon[20143]: osdmap e75: 10 total, 10 up, 10 in 2024-08-19T14:10:54.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:54 smithi032 ceph-mon[20004]: osdmap e75: 10 total, 10 up, 10 in 2024-08-19T14:10:54.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:54 smithi031 ceph-mon[16336]: osdmap e75: 10 total, 10 up, 10 in 2024-08-19T14:10:55.663 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:55 smithi045 ceph-mon[20143]: Detected new or changed devices on smithi045 2024-08-19T14:10:55.663 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:55 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:55.663 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:55 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:55.663 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:55 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:10:55.663 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:55 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:10:55.663 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:55 smithi045 ceph-mon[20143]: Adjusting osd_memory_target on smithi045 to 10594M 2024-08-19T14:10:55.663 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:55 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:55.663 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:55 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:10:55.664 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:55 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:10:55.664 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:55 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:10:55.664 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:55 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:10:55.664 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:55 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:10:55.664 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:55 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:10:55.664 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:55 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:10:55.664 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:55 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:10:55.664 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:55 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:10:55.664 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:55 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:10:55.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:55 smithi031 ceph-mon[16336]: Detected new or changed devices on smithi045 2024-08-19T14:10:55.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:55 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:55.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:55 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:55.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:55 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:10:55.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:55 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:10:55.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:55 smithi031 ceph-mon[16336]: Adjusting osd_memory_target on smithi045 to 10594M 2024-08-19T14:10:55.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:55 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:55.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:55 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:10:55.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:55 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:10:55.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:55 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:10:55.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:55 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:10:55.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:55 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:10:55.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:55 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:10:55.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:55 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:10:55.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:55 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:10:55.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:55 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:10:55.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:55 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:10:56.001 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-08-19T14:10:56.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:55 smithi032 ceph-mon[20004]: Detected new or changed devices on smithi045 2024-08-19T14:10:56.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:55 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:56.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:55 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:56.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:55 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:10:56.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:55 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:10:56.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:55 smithi032 ceph-mon[20004]: Adjusting osd_memory_target on smithi045 to 10594M 2024-08-19T14:10:56.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:55 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:10:56.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:55 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:10:56.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:55 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:10:56.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:55 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:10:56.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:55 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:10:56.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:55 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:10:56.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:55 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:10:56.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:55 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:10:56.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:55 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:10:56.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:55 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:10:56.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:55 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:10:56.029 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph orch daemon add osd smithi045:vg_nvme/lv_2 2024-08-19T14:10:56.283 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:10:57.720 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:57 smithi045 ceph-mon[20143]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-08-19T14:10:57.721 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:57 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-19T14:10:57.721 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:57 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-19T14:10:57.721 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:57 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:10:57.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:57 smithi031 ceph-mon[16336]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-08-19T14:10:57.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:57 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-19T14:10:57.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:57 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-19T14:10:57.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:57 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:10:58.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:57 smithi032 ceph-mon[20004]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-08-19T14:10:58.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:57 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-19T14:10:58.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:57 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-19T14:10:58.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:57 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:10:58.662 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:58 smithi045 ceph-mon[20143]: from='client.24298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T14:10:58.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:58 smithi031 ceph-mon[16336]: from='client.24298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T14:10:59.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:58 smithi032 ceph-mon[20004]: from='client.24298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T14:10:59.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:59 smithi045 ceph-mon[20143]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-08-19T14:10:59.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:59 smithi045 ceph-mon[20143]: from='client.? 172.21.15.45:0/510390476' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "29031395-5f5c-40f8-ac90-9f6ac48e5c9f"}]: dispatch 2024-08-19T14:10:59.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:59 smithi045 ceph-mon[20143]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "29031395-5f5c-40f8-ac90-9f6ac48e5c9f"}]: dispatch 2024-08-19T14:10:59.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:59 smithi045 ceph-mon[20143]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "29031395-5f5c-40f8-ac90-9f6ac48e5c9f"}]': finished 2024-08-19T14:10:59.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:59 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-08-19T14:10:59.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:10:59 smithi045 ceph-mon[20143]: osdmap e76: 11 total, 10 up, 11 in 2024-08-19T14:10:59.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:59 smithi031 ceph-mon[16336]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-08-19T14:10:59.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:59 smithi031 ceph-mon[16336]: from='client.? 172.21.15.45:0/510390476' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "29031395-5f5c-40f8-ac90-9f6ac48e5c9f"}]: dispatch 2024-08-19T14:10:59.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:59 smithi031 ceph-mon[16336]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "29031395-5f5c-40f8-ac90-9f6ac48e5c9f"}]: dispatch 2024-08-19T14:10:59.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:59 smithi031 ceph-mon[16336]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "29031395-5f5c-40f8-ac90-9f6ac48e5c9f"}]': finished 2024-08-19T14:10:59.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:59 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-08-19T14:10:59.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:10:59 smithi031 ceph-mon[16336]: osdmap e76: 11 total, 10 up, 11 in 2024-08-19T14:11:00.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:59 smithi032 ceph-mon[20004]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-08-19T14:11:00.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:59 smithi032 ceph-mon[20004]: from='client.? 172.21.15.45:0/510390476' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "29031395-5f5c-40f8-ac90-9f6ac48e5c9f"}]: dispatch 2024-08-19T14:11:00.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:59 smithi032 ceph-mon[20004]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "29031395-5f5c-40f8-ac90-9f6ac48e5c9f"}]: dispatch 2024-08-19T14:11:00.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:59 smithi032 ceph-mon[20004]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "29031395-5f5c-40f8-ac90-9f6ac48e5c9f"}]': finished 2024-08-19T14:11:00.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:59 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-08-19T14:11:00.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:10:59 smithi032 ceph-mon[20004]: osdmap e76: 11 total, 10 up, 11 in 2024-08-19T14:11:00.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:00 smithi045 ceph-mon[20143]: from='client.? 172.21.15.45:0/2659699301' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-19T14:11:00.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:00 smithi031 ceph-mon[16336]: from='client.? 172.21.15.45:0/2659699301' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-19T14:11:01.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:00 smithi032 ceph-mon[20004]: from='client.? 172.21.15.45:0/2659699301' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-19T14:11:01.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:01 smithi045 ceph-mon[20143]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-08-19T14:11:01.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:01 smithi031 ceph-mon[16336]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-08-19T14:11:02.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:01 smithi032 ceph-mon[20004]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-08-19T14:11:03.673 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:03 smithi045 ceph-mon[20143]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-08-19T14:11:03.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:03 smithi031 ceph-mon[16336]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-08-19T14:11:04.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:03 smithi032 ceph-mon[20004]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-08-19T14:11:05.732 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:05 smithi045 ceph-mon[20143]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-08-19T14:11:05.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:05 smithi031 ceph-mon[16336]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-08-19T14:11:06.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:05 smithi032 ceph-mon[20004]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-08-19T14:11:06.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:06 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:11:06.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:06 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:11:06.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:06 smithi031 ceph-mon[16336]: osdmap e77: 11 total, 10 up, 11 in 2024-08-19T14:11:06.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:06 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-08-19T14:11:06.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:06 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-08-19T14:11:06.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:06 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:11:06.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:06 smithi031 ceph-mon[16336]: Deploying daemon osd.10 on smithi045 2024-08-19T14:11:06.933 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:06 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:11:06.933 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:06 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:11:06.933 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:06 smithi045 ceph-mon[20143]: osdmap e77: 11 total, 10 up, 11 in 2024-08-19T14:11:06.933 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:06 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-08-19T14:11:06.933 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:06 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-08-19T14:11:06.933 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:06 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:11:06.933 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:06 smithi045 ceph-mon[20143]: Deploying daemon osd.10 on smithi045 2024-08-19T14:11:07.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:06 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:11:07.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:06 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:11:07.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:06 smithi032 ceph-mon[20004]: osdmap e77: 11 total, 10 up, 11 in 2024-08-19T14:11:07.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:06 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-08-19T14:11:07.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:06 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-08-19T14:11:07.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:06 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:11:07.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:06 smithi032 ceph-mon[20004]: Deploying daemon osd.10 on smithi045 2024-08-19T14:11:07.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:07 smithi045 ceph-mon[20143]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-08-19T14:11:07.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:07 smithi031 ceph-mon[16336]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-08-19T14:11:08.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:07 smithi032 ceph-mon[20004]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-08-19T14:11:09.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:09 smithi031 ceph-mon[16336]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-08-19T14:11:09.941 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:09 smithi045 ceph-mon[20143]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-08-19T14:11:10.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:09 smithi032 ceph-mon[20004]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-08-19T14:11:10.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:10 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:11:10.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:10 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:11:10.980 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:10 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:11:10.980 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:10 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:11:11.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:10 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:11:11.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:10 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:11:11.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:11 smithi031 ceph-mon[16336]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-08-19T14:11:11.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:11 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:11:11.960 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:11 smithi045 ceph-mon[20143]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-08-19T14:11:11.961 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:11 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:11:12.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:11 smithi032 ceph-mon[20004]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-08-19T14:11:12.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:11 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:11:13.851 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:13 smithi045 ceph-mon[20143]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-08-19T14:11:13.851 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:13 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:11:13.851 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:13 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:11:13.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:13 smithi031 ceph-mon[16336]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-08-19T14:11:13.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:13 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:11:13.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:13 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:11:13.895 INFO:teuthology.orchestra.run.smithi045.stdout:Created osd(s) 10 on host 'smithi045' 2024-08-19T14:11:14.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:13 smithi032 ceph-mon[20004]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-08-19T14:11:14.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:13 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:11:14.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:13 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:11:14.514 DEBUG:teuthology.orchestra.run.smithi045:osd.10> sudo journalctl -f -n 0 -u ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968@osd.10.service 2024-08-19T14:11:14.517 INFO:tasks.cephadm:Deploying osd.11 on smithi045 with /dev/vg_nvme/lv_1... 2024-08-19T14:11:14.517 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- lvm zap /dev/vg_nvme/lv_1 2024-08-19T14:11:14.758 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:14 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:11:14.758 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:14 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:11:14.758 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:14 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:11:14.758 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:14 smithi045 ceph-mon[20143]: from='osd.10 [v2:172.21.15.45:6816/2461824621,v1:172.21.15.45:6817/2461824621]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-08-19T14:11:14.758 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:14 smithi045 ceph-mon[20143]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-08-19T14:11:14.758 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:14 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:11:14.758 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:14 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:11:14.759 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:14 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:11:14.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:14 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:11:14.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:14 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:11:14.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:14 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:11:14.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:14 smithi031 ceph-mon[16336]: from='osd.10 [v2:172.21.15.45:6816/2461824621,v1:172.21.15.45:6817/2461824621]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-08-19T14:11:14.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:14 smithi031 ceph-mon[16336]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-08-19T14:11:14.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:14 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:11:14.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:14 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:11:14.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:14 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:11:15.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:14 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:11:15.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:14 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:11:15.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:14 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:11:15.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:14 smithi032 ceph-mon[20004]: from='osd.10 [v2:172.21.15.45:6816/2461824621,v1:172.21.15.45:6817/2461824621]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-08-19T14:11:15.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:14 smithi032 ceph-mon[20004]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-08-19T14:11:15.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:14 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:11:15.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:14 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:11:15.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:14 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:11:15.251 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:11:15.696 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:15 smithi045 ceph-mon[20143]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-08-19T14:11:15.696 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:15 smithi045 ceph-mon[20143]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-08-19T14:11:15.696 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:15 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-08-19T14:11:15.696 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:15 smithi045 ceph-mon[20143]: osdmap e78: 11 total, 10 up, 11 in 2024-08-19T14:11:15.696 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:15 smithi045 ceph-mon[20143]: from='osd.10 [v2:172.21.15.45:6816/2461824621,v1:172.21.15.45:6817/2461824621]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-08-19T14:11:15.696 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:15 smithi045 ceph-mon[20143]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-08-19T14:11:15.696 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:15 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:11:15.696 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:15 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:11:15.696 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:15 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:11:15.696 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:15 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:11:15.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:15 smithi031 ceph-mon[16336]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-08-19T14:11:15.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:15 smithi031 ceph-mon[16336]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-08-19T14:11:15.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:15 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-08-19T14:11:15.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:15 smithi031 ceph-mon[16336]: osdmap e78: 11 total, 10 up, 11 in 2024-08-19T14:11:15.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:15 smithi031 ceph-mon[16336]: from='osd.10 [v2:172.21.15.45:6816/2461824621,v1:172.21.15.45:6817/2461824621]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-08-19T14:11:15.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:15 smithi031 ceph-mon[16336]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-08-19T14:11:15.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:15 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:11:15.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:15 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:11:15.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:15 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:11:15.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:15 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:11:16.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:15 smithi032 ceph-mon[20004]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-08-19T14:11:16.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:15 smithi032 ceph-mon[20004]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-08-19T14:11:16.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:15 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-08-19T14:11:16.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:15 smithi032 ceph-mon[20004]: osdmap e78: 11 total, 10 up, 11 in 2024-08-19T14:11:16.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:15 smithi032 ceph-mon[20004]: from='osd.10 [v2:172.21.15.45:6816/2461824621,v1:172.21.15.45:6817/2461824621]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-08-19T14:11:16.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:15 smithi032 ceph-mon[20004]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-08-19T14:11:16.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:15 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:11:16.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:15 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:11:16.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:15 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:11:16.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:15 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:11:16.025 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:11:15 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[35889]: 2024-08-19T14:11:15.856+0000 7fb75d53d640 -1 osd.10 0 waiting for initial osdmap 2024-08-19T14:11:16.025 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:11:15 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[35889]: 2024-08-19T14:11:15.865+0000 7fb75535d640 -1 osd.10 79 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-19T14:11:16.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:16 smithi031 ceph-mon[16336]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]': finished 2024-08-19T14:11:16.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:16 smithi031 ceph-mon[16336]: osdmap e79: 11 total, 10 up, 11 in 2024-08-19T14:11:16.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:16 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-08-19T14:11:16.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:16 smithi031 ceph-mon[16336]: from='osd.10 ' entity='osd.10' 2024-08-19T14:11:16.901 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:16 smithi045 ceph-mon[20143]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]': finished 2024-08-19T14:11:16.901 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:16 smithi045 ceph-mon[20143]: osdmap e79: 11 total, 10 up, 11 in 2024-08-19T14:11:16.901 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:16 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-08-19T14:11:16.901 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:16 smithi045 ceph-mon[20143]: from='osd.10 ' entity='osd.10' 2024-08-19T14:11:17.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:16 smithi032 ceph-mon[20004]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]': finished 2024-08-19T14:11:17.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:16 smithi032 ceph-mon[20004]: osdmap e79: 11 total, 10 up, 11 in 2024-08-19T14:11:17.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:16 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-08-19T14:11:17.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:16 smithi032 ceph-mon[20004]: from='osd.10 ' entity='osd.10' 2024-08-19T14:11:17.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:17 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:11:17.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:17 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:11:17.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:17 smithi031 ceph-mon[16336]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-08-19T14:11:17.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:17 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-08-19T14:11:17.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:17 smithi031 ceph-mon[16336]: osd.10 [v2:172.21.15.45:6816/2461824621,v1:172.21.15.45:6817/2461824621] boot 2024-08-19T14:11:17.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:17 smithi031 ceph-mon[16336]: osdmap e80: 11 total, 11 up, 11 in 2024-08-19T14:11:17.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:17 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-08-19T14:11:18.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:17 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:11:18.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:17 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:11:18.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:17 smithi032 ceph-mon[20004]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-08-19T14:11:18.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:17 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-08-19T14:11:18.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:17 smithi032 ceph-mon[20004]: osd.10 [v2:172.21.15.45:6816/2461824621,v1:172.21.15.45:6817/2461824621] boot 2024-08-19T14:11:18.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:17 smithi032 ceph-mon[20004]: osdmap e80: 11 total, 11 up, 11 in 2024-08-19T14:11:18.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:17 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-08-19T14:11:18.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:17 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:11:18.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:17 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:11:18.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:17 smithi045 ceph-mon[20143]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-08-19T14:11:18.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:17 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-08-19T14:11:18.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:17 smithi045 ceph-mon[20143]: osd.10 [v2:172.21.15.45:6816/2461824621,v1:172.21.15.45:6817/2461824621] boot 2024-08-19T14:11:18.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:17 smithi045 ceph-mon[20143]: osdmap e80: 11 total, 11 up, 11 in 2024-08-19T14:11:18.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:17 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-08-19T14:11:19.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:19 smithi032 ceph-mon[20004]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-19T14:11:19.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:19 smithi032 ceph-mon[20004]: Detected new or changed devices on smithi045 2024-08-19T14:11:19.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:19 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:11:19.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:19 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:11:19.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:19 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:11:19.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:19 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:11:19.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:19 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:11:19.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:19 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:11:19.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:19 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:11:19.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:19 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:11:19.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:19 smithi032 ceph-mon[20004]: Adjusting osd_memory_target on smithi045 to 7062M 2024-08-19T14:11:19.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:19 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:11:19.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:19 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:11:19.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:19 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:11:19.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:19 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:11:19.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:19 smithi032 ceph-mon[20004]: osdmap e81: 11 total, 11 up, 11 in 2024-08-19T14:11:19.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:19 smithi045 ceph-mon[20143]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-19T14:11:19.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:19 smithi045 ceph-mon[20143]: Detected new or changed devices on smithi045 2024-08-19T14:11:19.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:19 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:11:19.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:19 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:11:19.794 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:19 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:11:19.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:19 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:11:19.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:19 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:11:19.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:19 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:11:19.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:19 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:11:19.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:19 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:11:19.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:19 smithi045 ceph-mon[20143]: Adjusting osd_memory_target on smithi045 to 7062M 2024-08-19T14:11:19.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:19 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:11:19.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:19 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:11:19.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:19 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:11:19.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:19 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:11:19.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:19 smithi045 ceph-mon[20143]: osdmap e81: 11 total, 11 up, 11 in 2024-08-19T14:11:19.798 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-08-19T14:11:19.826 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph orch daemon add osd smithi045:vg_nvme/lv_1 2024-08-19T14:11:19.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:19 smithi031 ceph-mon[16336]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-19T14:11:19.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:19 smithi031 ceph-mon[16336]: Detected new or changed devices on smithi045 2024-08-19T14:11:19.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:19 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:11:19.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:19 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:11:19.870 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:19 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:11:19.870 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:19 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:11:19.870 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:19 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:11:19.871 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:19 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:11:19.871 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:19 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:11:19.871 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:19 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:11:19.871 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:19 smithi031 ceph-mon[16336]: Adjusting osd_memory_target on smithi045 to 7062M 2024-08-19T14:11:19.871 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:19 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:11:19.871 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:19 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:11:19.871 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:19 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:11:19.871 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:19 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:11:19.871 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:19 smithi031 ceph-mon[16336]: osdmap e81: 11 total, 11 up, 11 in 2024-08-19T14:11:20.081 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:11:21.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:21 smithi032 ceph-mon[20004]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-19T14:11:21.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:21 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:11:21.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:21 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:11:21.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:21 smithi032 ceph-mon[20004]: osdmap e82: 11 total, 11 up, 11 in 2024-08-19T14:11:21.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:21 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-19T14:11:21.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:21 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-19T14:11:21.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:21 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:11:21.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:21 smithi045 ceph-mon[20143]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-19T14:11:21.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:21 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:11:21.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:21 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:11:21.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:21 smithi045 ceph-mon[20143]: osdmap e82: 11 total, 11 up, 11 in 2024-08-19T14:11:21.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:21 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-19T14:11:21.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:21 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-19T14:11:21.793 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:21 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:11:21.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:21 smithi031 ceph-mon[16336]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-19T14:11:21.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:21 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:11:21.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:21 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:11:21.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:21 smithi031 ceph-mon[16336]: osdmap e82: 11 total, 11 up, 11 in 2024-08-19T14:11:21.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:21 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-19T14:11:21.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:21 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-19T14:11:21.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:21 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:11:22.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:22 smithi032 ceph-mon[20004]: from='client.24325 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T14:11:22.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:22 smithi032 ceph-mon[20004]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-19T14:11:22.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:22 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:11:22.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:22 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:11:22.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:22 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:11:22.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:22 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:11:22.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:22 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:11:22.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:22 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:11:22.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:22 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:11:22.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:22 smithi045 ceph-mon[20143]: from='client.24325 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T14:11:22.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:22 smithi045 ceph-mon[20143]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-19T14:11:22.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:22 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:11:22.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:22 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:11:22.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:22 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:11:22.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:22 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:11:22.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:22 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:11:22.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:22 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:11:22.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:22 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:11:22.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:22 smithi031 ceph-mon[16336]: from='client.24325 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T14:11:22.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:22 smithi031 ceph-mon[16336]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-19T14:11:22.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:22 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:11:22.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:22 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:11:22.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:22 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:11:22.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:22 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:11:22.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:22 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:11:22.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:22 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:11:22.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:22 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:11:23.703 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:23 smithi045 ceph-mon[20143]: from='client.? 172.21.15.45:0/2059995598' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5ef0db4b-0fef-4ea0-a19b-d3677e082c29"}]: dispatch 2024-08-19T14:11:23.703 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:23 smithi045 ceph-mon[20143]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5ef0db4b-0fef-4ea0-a19b-d3677e082c29"}]: dispatch 2024-08-19T14:11:23.703 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:23 smithi045 ceph-mon[20143]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5ef0db4b-0fef-4ea0-a19b-d3677e082c29"}]': finished 2024-08-19T14:11:23.703 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:23 smithi045 ceph-mon[20143]: osdmap e83: 12 total, 11 up, 12 in 2024-08-19T14:11:23.703 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:23 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-08-19T14:11:23.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:23 smithi032 ceph-mon[20004]: from='client.? 172.21.15.45:0/2059995598' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5ef0db4b-0fef-4ea0-a19b-d3677e082c29"}]: dispatch 2024-08-19T14:11:23.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:23 smithi032 ceph-mon[20004]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5ef0db4b-0fef-4ea0-a19b-d3677e082c29"}]: dispatch 2024-08-19T14:11:23.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:23 smithi032 ceph-mon[20004]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5ef0db4b-0fef-4ea0-a19b-d3677e082c29"}]': finished 2024-08-19T14:11:23.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:23 smithi032 ceph-mon[20004]: osdmap e83: 12 total, 11 up, 12 in 2024-08-19T14:11:23.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:23 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-08-19T14:11:23.767 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:23 smithi031 ceph-mon[16336]: from='client.? 172.21.15.45:0/2059995598' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5ef0db4b-0fef-4ea0-a19b-d3677e082c29"}]: dispatch 2024-08-19T14:11:23.767 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:23 smithi031 ceph-mon[16336]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5ef0db4b-0fef-4ea0-a19b-d3677e082c29"}]: dispatch 2024-08-19T14:11:23.767 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:23 smithi031 ceph-mon[16336]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5ef0db4b-0fef-4ea0-a19b-d3677e082c29"}]': finished 2024-08-19T14:11:23.767 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:23 smithi031 ceph-mon[16336]: osdmap e83: 12 total, 11 up, 12 in 2024-08-19T14:11:23.767 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:23 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-08-19T14:11:24.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:24 smithi032 ceph-mon[20004]: from='client.? 172.21.15.45:0/2962791247' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-19T14:11:24.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:24 smithi045 ceph-mon[20143]: from='client.? 172.21.15.45:0/2962791247' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-19T14:11:24.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:24 smithi031 ceph-mon[16336]: from='client.? 172.21.15.45:0/2962791247' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-19T14:11:25.712 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:25 smithi045 ceph-mon[20143]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-19T14:11:25.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:25 smithi032 ceph-mon[20004]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-19T14:11:25.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:25 smithi031 ceph-mon[16336]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-19T14:11:27.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:27 smithi045 ceph-mon[20143]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-19T14:11:27.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:27 smithi032 ceph-mon[20004]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-19T14:11:27.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:27 smithi031 ceph-mon[16336]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-19T14:11:29.752 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:29 smithi045 ceph-mon[20143]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-19T14:11:29.752 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:29 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-08-19T14:11:29.752 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:29 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:11:29.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:29 smithi032 ceph-mon[20004]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-19T14:11:29.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:29 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-08-19T14:11:29.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:29 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:11:29.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:29 smithi031 ceph-mon[16336]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-19T14:11:29.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:29 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-08-19T14:11:29.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:29 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:11:30.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:30 smithi032 ceph-mon[20004]: Deploying daemon osd.11 on smithi045 2024-08-19T14:11:30.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:30 smithi045 ceph-mon[20143]: Deploying daemon osd.11 on smithi045 2024-08-19T14:11:30.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:30 smithi031 ceph-mon[16336]: Deploying daemon osd.11 on smithi045 2024-08-19T14:11:31.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:31 smithi032 ceph-mon[20004]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-19T14:11:31.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:31 smithi045 ceph-mon[20143]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-19T14:11:31.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:31 smithi031 ceph-mon[16336]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-19T14:11:33.713 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:33 smithi045 ceph-mon[20143]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-19T14:11:33.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:33 smithi032 ceph-mon[20004]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-19T14:11:33.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:33 smithi031 ceph-mon[16336]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-19T14:11:34.729 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:34 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:11:34.729 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:34 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:11:34.729 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:34 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:11:34.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:34 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:11:34.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:34 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:11:34.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:34 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:11:34.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:34 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:11:34.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:34 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:11:34.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:34 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:11:35.665 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:35 smithi045 ceph-mon[20143]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-19T14:11:35.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:35 smithi032 ceph-mon[20004]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-19T14:11:35.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:35 smithi031 ceph-mon[16336]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-19T14:11:37.553 INFO:teuthology.orchestra.run.smithi045.stdout:Created osd(s) 11 on host 'smithi045' 2024-08-19T14:11:37.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:37 smithi032 ceph-mon[20004]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-19T14:11:37.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:37 smithi032 ceph-mon[20004]: from='osd.11 [v2:172.21.15.45:6824/2438396298,v1:172.21.15.45:6825/2438396298]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-08-19T14:11:37.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:37 smithi032 ceph-mon[20004]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-08-19T14:11:37.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:37 smithi045 ceph-mon[20143]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-19T14:11:37.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:37 smithi045 ceph-mon[20143]: from='osd.11 [v2:172.21.15.45:6824/2438396298,v1:172.21.15.45:6825/2438396298]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-08-19T14:11:37.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:37 smithi045 ceph-mon[20143]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-08-19T14:11:37.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:37 smithi031 ceph-mon[16336]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-19T14:11:37.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:37 smithi031 ceph-mon[16336]: from='osd.11 [v2:172.21.15.45:6824/2438396298,v1:172.21.15.45:6825/2438396298]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-08-19T14:11:37.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:37 smithi031 ceph-mon[16336]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-08-19T14:11:38.179 DEBUG:teuthology.orchestra.run.smithi045:osd.11> sudo journalctl -f -n 0 -u ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968@osd.11.service 2024-08-19T14:11:38.181 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-08-19T14:11:38.182 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd stat -f json 2024-08-19T14:11:38.460 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:11:38.747 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:38 smithi031 ceph-mon[16336]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-08-19T14:11:38.747 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:38 smithi031 ceph-mon[16336]: from='osd.11 [v2:172.21.15.45:6824/2438396298,v1:172.21.15.45:6825/2438396298]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-08-19T14:11:38.747 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:38 smithi031 ceph-mon[16336]: osdmap e84: 12 total, 11 up, 12 in 2024-08-19T14:11:38.747 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:38 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-08-19T14:11:38.748 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:38 smithi031 ceph-mon[16336]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-08-19T14:11:38.748 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:38 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:11:38.748 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:38 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:11:38.748 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:38 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:11:38.748 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:38 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:11:38.754 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:38 smithi045 ceph-mon[20143]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-08-19T14:11:38.754 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:38 smithi045 ceph-mon[20143]: from='osd.11 [v2:172.21.15.45:6824/2438396298,v1:172.21.15.45:6825/2438396298]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-08-19T14:11:38.755 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:38 smithi045 ceph-mon[20143]: osdmap e84: 12 total, 11 up, 12 in 2024-08-19T14:11:38.755 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:38 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-08-19T14:11:38.755 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:38 smithi045 ceph-mon[20143]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-08-19T14:11:38.755 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:38 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:11:38.755 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:38 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:11:38.755 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:38 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:11:38.755 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:38 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:11:38.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:38 smithi032 ceph-mon[20004]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-08-19T14:11:38.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:38 smithi032 ceph-mon[20004]: from='osd.11 [v2:172.21.15.45:6824/2438396298,v1:172.21.15.45:6825/2438396298]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-08-19T14:11:38.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:38 smithi032 ceph-mon[20004]: osdmap e84: 12 total, 11 up, 12 in 2024-08-19T14:11:38.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:38 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-08-19T14:11:38.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:38 smithi032 ceph-mon[20004]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-08-19T14:11:38.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:38 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:11:38.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:38 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:11:38.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:38 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:11:38.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:38 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:11:39.046 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:11:38 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:11:38.754+0000 7f3e0323f640 -1 osd.11 0 waiting for initial osdmap 2024-08-19T14:11:39.046 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:11:38 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:11:38.764+0000 7f3dfb05f640 -1 osd.11 85 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-19T14:11:39.128 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:11:39.689 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:39 smithi045 ceph-mon[20143]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-19T14:11:39.690 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:39 smithi045 ceph-mon[20143]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]': finished 2024-08-19T14:11:39.690 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:39 smithi045 ceph-mon[20143]: osdmap e85: 12 total, 11 up, 12 in 2024-08-19T14:11:39.690 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:39 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-08-19T14:11:39.690 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:39 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-08-19T14:11:39.690 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:39 smithi045 ceph-mon[20143]: from='osd.11 ' entity='osd.11' 2024-08-19T14:11:39.690 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:39 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3113342263' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-08-19T14:11:39.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:39 smithi032 ceph-mon[20004]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-19T14:11:39.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:39 smithi032 ceph-mon[20004]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]': finished 2024-08-19T14:11:39.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:39 smithi032 ceph-mon[20004]: osdmap e85: 12 total, 11 up, 12 in 2024-08-19T14:11:39.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:39 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-08-19T14:11:39.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:39 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-08-19T14:11:39.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:39 smithi032 ceph-mon[20004]: from='osd.11 ' entity='osd.11' 2024-08-19T14:11:39.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:39 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3113342263' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-08-19T14:11:39.835 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:39 smithi031 ceph-mon[16336]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-19T14:11:39.835 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:39 smithi031 ceph-mon[16336]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]': finished 2024-08-19T14:11:39.835 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:39 smithi031 ceph-mon[16336]: osdmap e85: 12 total, 11 up, 12 in 2024-08-19T14:11:39.835 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:39 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-08-19T14:11:39.835 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:39 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-08-19T14:11:39.835 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:39 smithi031 ceph-mon[16336]: from='osd.11 ' entity='osd.11' 2024-08-19T14:11:39.835 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:39 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3113342263' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-08-19T14:11:39.862 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":85,"num_osds":12,"num_up_osds":11,"osd_up_since":1724076676,"num_in_osds":12,"osd_in_since":1724076683,"num_remapped_pgs":0} 2024-08-19T14:11:40.555 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:40 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:11:40.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:40 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:11:40.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:40 smithi045 ceph-mon[20143]: osd.11 [v2:172.21.15.45:6824/2438396298,v1:172.21.15.45:6825/2438396298] boot 2024-08-19T14:11:40.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:40 smithi045 ceph-mon[20143]: osdmap e86: 12 total, 12 up, 12 in 2024-08-19T14:11:40.556 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:40 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-08-19T14:11:40.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:40 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:11:40.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:40 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:11:40.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:40 smithi032 ceph-mon[20004]: osd.11 [v2:172.21.15.45:6824/2438396298,v1:172.21.15.45:6825/2438396298] boot 2024-08-19T14:11:40.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:40 smithi032 ceph-mon[20004]: osdmap e86: 12 total, 12 up, 12 in 2024-08-19T14:11:40.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:40 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-08-19T14:11:40.864 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd stat -f json 2024-08-19T14:11:40.868 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:40 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:11:40.868 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:40 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:11:40.868 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:40 smithi031 ceph-mon[16336]: osd.11 [v2:172.21.15.45:6824/2438396298,v1:172.21.15.45:6825/2438396298] boot 2024-08-19T14:11:40.868 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:40 smithi031 ceph-mon[16336]: osdmap e86: 12 total, 12 up, 12 in 2024-08-19T14:11:40.868 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:40 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-08-19T14:11:41.133 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:11:41.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:41 smithi045 ceph-mon[20143]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-19T14:11:41.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:41 smithi045 ceph-mon[20143]: osdmap e87: 12 total, 12 up, 12 in 2024-08-19T14:11:41.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:41 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:11:41.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:41 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:11:41.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:41 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:11:41.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:41 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:11:41.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:41 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:11:41.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:41 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:11:41.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:41 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:11:41.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:41 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:11:41.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:41 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:11:41.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:41 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:11:41.827 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:11:41.828 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:41 smithi031 ceph-mon[16336]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-19T14:11:41.828 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:41 smithi031 ceph-mon[16336]: osdmap e87: 12 total, 12 up, 12 in 2024-08-19T14:11:41.828 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:41 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:11:41.828 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:41 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:11:41.828 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:41 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:11:41.828 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:41 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:11:41.828 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:41 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:11:41.828 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:41 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:11:41.828 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:41 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:11:41.829 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:41 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:11:41.829 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:41 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:11:41.829 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:41 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:11:42.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:41 smithi032 ceph-mon[20004]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-19T14:11:42.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:41 smithi032 ceph-mon[20004]: osdmap e87: 12 total, 12 up, 12 in 2024-08-19T14:11:42.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:41 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:11:42.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:41 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:11:42.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:41 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:11:42.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:41 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:11:42.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:41 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:11:42.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:41 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-08-19T14:11:42.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:41 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:11:42.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:41 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:11:42.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:41 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:11:42.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:41 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:11:42.515 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":88,"num_osds":12,"num_up_osds":12,"osd_up_since":1724076699,"num_in_osds":12,"osd_in_since":1724076683,"num_remapped_pgs":1} 2024-08-19T14:11:42.516 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd dump --format=json 2024-08-19T14:11:42.540 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:42 smithi031 ceph-mon[16336]: Detected new or changed devices on smithi045 2024-08-19T14:11:42.540 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:42 smithi031 ceph-mon[16336]: Adjusting osd_memory_target on smithi045 to 5297M 2024-08-19T14:11:42.540 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:42 smithi031 ceph-mon[16336]: osdmap e88: 12 total, 12 up, 12 in 2024-08-19T14:11:42.540 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:42 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:11:42.540 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:42 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:11:42.541 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:42 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:11:42.541 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:42 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:11:42.541 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:42 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2592279249' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-08-19T14:11:42.773 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:11:42.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:42 smithi045 ceph-mon[20143]: Detected new or changed devices on smithi045 2024-08-19T14:11:42.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:42 smithi045 ceph-mon[20143]: Adjusting osd_memory_target on smithi045 to 5297M 2024-08-19T14:11:42.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:42 smithi045 ceph-mon[20143]: osdmap e88: 12 total, 12 up, 12 in 2024-08-19T14:11:42.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:42 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:11:42.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:42 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:11:42.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:42 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:11:42.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:42 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:11:42.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:42 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2592279249' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-08-19T14:11:43.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:42 smithi032 ceph-mon[20004]: Detected new or changed devices on smithi045 2024-08-19T14:11:43.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:42 smithi032 ceph-mon[20004]: Adjusting osd_memory_target on smithi045 to 5297M 2024-08-19T14:11:43.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:42 smithi032 ceph-mon[20004]: osdmap e88: 12 total, 12 up, 12 in 2024-08-19T14:11:43.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:42 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:11:43.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:42 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:11:43.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:42 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:11:43.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:42 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:11:43.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:42 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2592279249' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-08-19T14:11:43.401 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:11:43.401 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":89,"fsid":"ed7d1c6e-5e33-11ef-bccf-c7b262605968","created":"2024-08-19T14:05:22.842169+0000","modified":"2024-08-19T14:11:42.513188+0000","last_up_change":"2024-08-19T14:11:39.491234+0000","last_in_change":"2024-08-19T14:11:23.117730+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-19T14:08:15.966221+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d7a0fff3-23c6-4ee3-a275-a844862aabfb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":1108945097},{"type":"v1","addr":"172.21.15.31:6803","nonce":1108945097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":1108945097},{"type":"v1","addr":"172.21.15.31:6805","nonce":1108945097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1108945097},{"type":"v1","addr":"172.21.15.31:6809","nonce":1108945097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":1108945097},{"type":"v1","addr":"172.21.15.31:6807","nonce":1108945097}]},"public_addr":"172.21.15.31:6803/1108945097","cluster_addr":"172.21.15.31:6805/1108945097","heartbeat_back_addr":"172.21.15.31:6809/1108945097","heartbeat_front_addr":"172.21.15.31:6807/1108945097","state":["exists","up"]},{"osd":1,"uuid":"145d5bb4-74dd-440f-bcc9-7e9ebda9c5d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":3236186806},{"type":"v1","addr":"172.21.15.31:6811","nonce":3236186806}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":3236186806},{"type":"v1","addr":"172.21.15.31:6813","nonce":3236186806}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":3236186806},{"type":"v1","addr":"172.21.15.31:6817","nonce":3236186806}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":3236186806},{"type":"v1","addr":"172.21.15.31:6815","nonce":3236186806}]},"public_addr":"172.21.15.31:6811/3236186806","cluster_addr":"172.21.15.31:6813/3236186806","heartbeat_back_addr":"172.21.15.31:6817/3236186806","heartbeat_front_addr":"172.21.15.31:6815/3236186806","state":["exists","up"]},{"osd":2,"uuid":"376c18d0-4768-46a0-bb60-e79c88ca909c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":4045553387},{"type":"v1","addr":"172.21.15.31:6819","nonce":4045553387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":4045553387},{"type":"v1","addr":"172.21.15.31:6821","nonce":4045553387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":4045553387},{"type":"v1","addr":"172.21.15.31:6825","nonce":4045553387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":4045553387},{"type":"v1","addr":"172.21.15.31:6823","nonce":4045553387}]},"public_addr":"172.21.15.31:6819/4045553387","cluster_addr":"172.21.15.31:6821/4045553387","heartbeat_back_addr":"172.21.15.31:6825/4045553387","heartbeat_front_addr":"172.21.15.31:6823/4045553387","state":["exists","up"]},{"osd":3,"uuid":"a787ea9f-b9ba-450f-b388-614cfba90634","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":2288082716},{"type":"v1","addr":"172.21.15.31:6827","nonce":2288082716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":2288082716},{"type":"v1","addr":"172.21.15.31:6829","nonce":2288082716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2288082716},{"type":"v1","addr":"172.21.15.31:6833","nonce":2288082716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":2288082716},{"type":"v1","addr":"172.21.15.31:6831","nonce":2288082716}]},"public_addr":"172.21.15.31:6827/2288082716","cluster_addr":"172.21.15.31:6829/2288082716","heartbeat_back_addr":"172.21.15.31:6833/2288082716","heartbeat_front_addr":"172.21.15.31:6831/2288082716","state":["exists","up"]},{"osd":4,"uuid":"7859751a-12f5-4a57-a332-435360479d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":3934739984},{"type":"v1","addr":"172.21.15.32:6801","nonce":3934739984}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":3934739984},{"type":"v1","addr":"172.21.15.32:6803","nonce":3934739984}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":3934739984},{"type":"v1","addr":"172.21.15.32:6807","nonce":3934739984}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":3934739984},{"type":"v1","addr":"172.21.15.32:6805","nonce":3934739984}]},"public_addr":"172.21.15.32:6801/3934739984","cluster_addr":"172.21.15.32:6803/3934739984","heartbeat_back_addr":"172.21.15.32:6807/3934739984","heartbeat_front_addr":"172.21.15.32:6805/3934739984","state":["exists","up"]},{"osd":5,"uuid":"017f3eef-ac60-4b82-8350-919017cb4ccd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":526388985},{"type":"v1","addr":"172.21.15.32:6809","nonce":526388985}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":526388985},{"type":"v1","addr":"172.21.15.32:6811","nonce":526388985}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":526388985},{"type":"v1","addr":"172.21.15.32:6815","nonce":526388985}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":526388985},{"type":"v1","addr":"172.21.15.32:6813","nonce":526388985}]},"public_addr":"172.21.15.32:6809/526388985","cluster_addr":"172.21.15.32:6811/526388985","heartbeat_back_addr":"172.21.15.32:6815/526388985","heartbeat_front_addr":"172.21.15.32:6813/526388985","state":["exists","up"]},{"osd":6,"uuid":"44aa8229-0268-49d7-bd57-3bf1f384e5e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":1131035844},{"type":"v1","addr":"172.21.15.32:6817","nonce":1131035844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":1131035844},{"type":"v1","addr":"172.21.15.32:6819","nonce":1131035844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":1131035844},{"type":"v1","addr":"172.21.15.32:6823","nonce":1131035844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":1131035844},{"type":"v1","addr":"172.21.15.32:6821","nonce":1131035844}]},"public_addr":"172.21.15.32:6817/1131035844","cluster_addr":"172.21.15.32:6819/1131035844","heartbeat_back_addr":"172.21.15.32:6823/1131035844","heartbeat_front_addr":"172.21.15.32:6821/1131035844","state":["exists","up"]},{"osd":7,"uuid":"263433e5-70b6-4192-86ff-dc2ad95927eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":2014939947},{"type":"v1","addr":"172.21.15.32:6825","nonce":2014939947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":2014939947},{"type":"v1","addr":"172.21.15.32:6827","nonce":2014939947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":2014939947},{"type":"v1","addr":"172.21.15.32:6831","nonce":2014939947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":2014939947},{"type":"v1","addr":"172.21.15.32:6829","nonce":2014939947}]},"public_addr":"172.21.15.32:6825/2014939947","cluster_addr":"172.21.15.32:6827/2014939947","heartbeat_back_addr":"172.21.15.32:6831/2014939947","heartbeat_front_addr":"172.21.15.32:6829/2014939947","state":["exists","up"]},{"osd":8,"uuid":"2bbcad36-0c74-4528-9fb9-a8812afa1662","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1445140892},{"type":"v1","addr":"172.21.15.45:6801","nonce":1445140892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1445140892},{"type":"v1","addr":"172.21.15.45:6803","nonce":1445140892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1445140892},{"type":"v1","addr":"172.21.15.45:6807","nonce":1445140892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1445140892},{"type":"v1","addr":"172.21.15.45:6805","nonce":1445140892}]},"public_addr":"172.21.15.45:6801/1445140892","cluster_addr":"172.21.15.45:6803/1445140892","heartbeat_back_addr":"172.21.15.45:6807/1445140892","heartbeat_front_addr":"172.21.15.45:6805/1445140892","state":["exists","up"]},{"osd":9,"uuid":"c8ee4277-3bf9-46bf-9cc0-152c1c4b9508","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1213787169},{"type":"v1","addr":"172.21.15.45:6809","nonce":1213787169}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1213787169},{"type":"v1","addr":"172.21.15.45:6811","nonce":1213787169}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1213787169},{"type":"v1","addr":"172.21.15.45:6815","nonce":1213787169}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1213787169},{"type":"v1","addr":"172.21.15.45:6813","nonce":1213787169}]},"public_addr":"172.21.15.45:6809/1213787169","cluster_addr":"172.21.15.45:6811/1213787169","heartbeat_back_addr":"172.21.15.45:6815/1213787169","heartbeat_front_addr":"172.21.15.45:6813/1213787169","state":["exists","up"]},{"osd":10,"uuid":"29031395-5f5c-40f8-ac90-9f6ac48e5c9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2461824621},{"type":"v1","addr":"172.21.15.45:6817","nonce":2461824621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2461824621},{"type":"v1","addr":"172.21.15.45:6819","nonce":2461824621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2461824621},{"type":"v1","addr":"172.21.15.45:6823","nonce":2461824621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2461824621},{"type":"v1","addr":"172.21.15.45:6821","nonce":2461824621}]},"public_addr":"172.21.15.45:6817/2461824621","cluster_addr":"172.21.15.45:6819/2461824621","heartbeat_back_addr":"172.21.15.45:6823/2461824621","heartbeat_front_addr":"172.21.15.45:6821/2461824621","state":["exists","up"]},{"osd":11,"uuid":"5ef0db4b-0fef-4ea0-a19b-d3677e082c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":86,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2438396298},{"type":"v1","addr":"172.21.15.45:6825","nonce":2438396298}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":2438396298},{"type":"v1","addr":"172.21.15.45:6827","nonce":2438396298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":2438396298},{"type":"v1","addr":"172.21.15.45:6831","nonce":2438396298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":2438396298},{"type":"v1","addr":"172.21.15.45:6829","nonce":2438396298}]},"public_addr":"172.21.15.45:6825/2438396298","cluster_addr":"172.21.15.45:6827/2438396298","heartbeat_back_addr":"172.21.15.45:6831/2438396298","heartbeat_front_addr":"172.21.15.45:6829/2438396298","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T14:11:15.004355+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T14:10:30.436717+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T14:10:41.958173+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T14:11:05.347211+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T14:11:20.435642+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T14:10:41.507554+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T14:10:51.578661+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T14:11:14.910669+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T14:10:27.030209+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T14:10:50.520674+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T14:11:14.756036+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T14:11:38.416189+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/1451745969":"2024-08-20T14:06:13.660657+0000","172.21.15.31:0/1515698918":"2024-08-20T14:06:13.660657+0000","172.21.15.31:6801/3297539242":"2024-08-20T14:06:13.660657+0000","172.21.15.31:6800/3297539242":"2024-08-20T14:06:13.660657+0000","172.21.15.31:0/2217423818":"2024-08-20T14:05:48.827576+0000","172.21.15.31:0/3353498281":"2024-08-20T14:06:13.660657+0000","172.21.15.31:0/4013594220":"2024-08-20T14:05:48.827576+0000","172.21.15.31:0/1654518065":"2024-08-20T14:05:48.827576+0000","172.21.15.31:6801/2368821258":"2024-08-20T14:05:48.827576+0000","172.21.15.31:6800/2368821258":"2024-08-20T14:05:48.827576+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-19T14:11:43.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:43 smithi045 ceph-mon[20143]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:11:43.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:43 smithi045 ceph-mon[20143]: osdmap e89: 12 total, 12 up, 12 in 2024-08-19T14:11:43.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:43 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1456513028' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-08-19T14:11:43.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:43 smithi031 ceph-mon[16336]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:11:43.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:43 smithi031 ceph-mon[16336]: osdmap e89: 12 total, 12 up, 12 in 2024-08-19T14:11:43.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:43 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1456513028' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-08-19T14:11:44.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:43 smithi032 ceph-mon[20004]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:11:44.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:43 smithi032 ceph-mon[20004]: osdmap e89: 12 total, 12 up, 12 in 2024-08-19T14:11:44.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:43 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1456513028' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-08-19T14:11:44.046 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-08-19T14:08:15.966221+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-08-19T14:11:44.047 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-08-19T14:11:44.342 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:11:44.715 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:44 smithi031 ceph-mon[16336]: osdmap e90: 12 total, 12 up, 12 in 2024-08-19T14:11:44.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:44 smithi045 ceph-mon[20143]: osdmap e90: 12 total, 12 up, 12 in 2024-08-19T14:11:44.976 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 1 2024-08-19T14:11:45.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:44 smithi032 ceph-mon[20004]: osdmap e90: 12 total, 12 up, 12 in 2024-08-19T14:11:45.574 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:45 smithi031 ceph-mon[16336]: pgmap v245: 1 pgs: 1 peering; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s, 0 objects/s recovering 2024-08-19T14:11:45.574 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:45 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1626700312' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-08-19T14:11:45.604 INFO:tasks.cephadm:Setting up client nodes... 2024-08-19T14:11:45.604 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-08-19T14:11:45.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:45 smithi045 ceph-mon[20143]: pgmap v245: 1 pgs: 1 peering; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s, 0 objects/s recovering 2024-08-19T14:11:45.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:45 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1626700312' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-08-19T14:11:45.860 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:11:46.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:45 smithi032 ceph-mon[20004]: pgmap v245: 1 pgs: 1 peering; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s, 0 objects/s recovering 2024-08-19T14:11:46.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:45 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1626700312' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-08-19T14:11:46.546 INFO:teuthology.orchestra.run.smithi031.stdout:[client.0] 2024-08-19T14:11:46.546 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQCiUsNmqFp5IBAA8mmDaQC7IkAwwKzfNdOodA== 2024-08-19T14:11:46.546 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:46 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:11:46.546 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:46 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:11:46.546 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:46 smithi031 ceph-mon[16336]: osdmap e91: 12 total, 12 up, 12 in 2024-08-19T14:11:46.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:46 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:11:46.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:46 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:11:46.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:46 smithi045 ceph-mon[20143]: osdmap e91: 12 total, 12 up, 12 in 2024-08-19T14:11:47.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:46 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:11:47.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:46 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:11:47.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:46 smithi032 ceph-mon[20004]: osdmap e91: 12 total, 12 up, 12 in 2024-08-19T14:11:47.118 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-08-19T14:11:47.118 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-08-19T14:11:47.118 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-08-19T14:11:47.155 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-08-19T14:11:47.427 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:11:47.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:47 smithi045 ceph-mon[20143]: pgmap v247: 1 pgs: 1 peering; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s, 0 objects/s recovering 2024-08-19T14:11:47.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:47 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1453821777' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-08-19T14:11:47.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:47 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1453821777' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-08-19T14:11:47.829 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:47 smithi032 ceph-mon[20004]: pgmap v247: 1 pgs: 1 peering; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s, 0 objects/s recovering 2024-08-19T14:11:47.829 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:47 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1453821777' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-08-19T14:11:47.829 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:47 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1453821777' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-08-19T14:11:47.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:47 smithi031 ceph-mon[16336]: pgmap v247: 1 pgs: 1 peering; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s, 0 objects/s recovering 2024-08-19T14:11:47.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:47 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1453821777' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-08-19T14:11:47.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:47 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1453821777' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-08-19T14:11:48.176 INFO:teuthology.orchestra.run.smithi032.stdout:[client.1] 2024-08-19T14:11:48.176 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQCkUsNmYC9BChAAP43wunVrbIuPQ4DHWn0cLg== 2024-08-19T14:11:48.807 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:48 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:11:48.807 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:48 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:11:48.808 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:48 smithi032 ceph-mon[20004]: osdmap e92: 12 total, 12 up, 12 in 2024-08-19T14:11:48.808 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:48 smithi032 ceph-mon[20004]: from='client.? 172.21.15.32:0/2614810716' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-08-19T14:11:48.808 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:48 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-08-19T14:11:48.808 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:48 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-08-19T14:11:48.838 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-19T14:11:48.838 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-08-19T14:11:48.838 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-08-19T14:11:48.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:48 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:11:48.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:48 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:11:48.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:48 smithi031 ceph-mon[16336]: osdmap e92: 12 total, 12 up, 12 in 2024-08-19T14:11:48.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:48 smithi031 ceph-mon[16336]: from='client.? 172.21.15.32:0/2614810716' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-08-19T14:11:48.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:48 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-08-19T14:11:48.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:48 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-08-19T14:11:48.878 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph auth get-or-create client.2 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-08-19T14:11:48.907 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:48 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:11:48.907 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:48 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:11:48.907 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:48 smithi045 ceph-mon[20143]: osdmap e92: 12 total, 12 up, 12 in 2024-08-19T14:11:48.908 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:48 smithi045 ceph-mon[20143]: from='client.? 172.21.15.32:0/2614810716' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-08-19T14:11:48.908 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:48 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-08-19T14:11:48.908 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:48 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-08-19T14:11:49.147 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:11:49.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:49 smithi045 ceph-mon[20143]: pgmap v249: 1 pgs: 1 peering; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s, 0 objects/s recovering 2024-08-19T14:11:49.866 INFO:teuthology.orchestra.run.smithi045.stdout:[client.2] 2024-08-19T14:11:49.866 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQClUsNmg5tfMxAAWNjfzgk3EdyIzhJs9y4Bew== 2024-08-19T14:11:49.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:49 smithi031 ceph-mon[16336]: pgmap v249: 1 pgs: 1 peering; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s, 0 objects/s recovering 2024-08-19T14:11:50.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:49 smithi032 ceph-mon[20004]: pgmap v249: 1 pgs: 1 peering; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s, 0 objects/s recovering 2024-08-19T14:11:50.623 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-08-19T14:11:50.623 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/etc/ceph/ceph.client.2.keyring 2024-08-19T14:11:50.624 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-08-19T14:11:50.704 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-08-19T14:11:50.705 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-08-19T14:11:50.705 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph mgr dump --format=json 2024-08-19T14:11:50.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:50 smithi031 ceph-mon[16336]: from='client.? 172.21.15.45:0/3681627299' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-08-19T14:11:50.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:50 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-08-19T14:11:50.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:50 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-08-19T14:11:50.970 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:11:50.987 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:50 smithi045 ceph-mon[20143]: from='client.? 172.21.15.45:0/3681627299' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-08-19T14:11:50.987 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:50 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-08-19T14:11:50.987 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:50 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-08-19T14:11:51.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:50 smithi032 ceph-mon[20004]: from='client.? 172.21.15.45:0/3681627299' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-08-19T14:11:51.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:50 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-08-19T14:11:51.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:50 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-08-19T14:11:51.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:51 smithi031 ceph-mon[16336]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:11:51.660 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:11:52.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:51 smithi032 ceph-mon[20004]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:11:52.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:51 smithi045 ceph-mon[20143]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:11:52.292 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":15,"flags":0,"active_gid":14150,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":532428460},{"type":"v1","addr":"172.21.15.31:6801","nonce":532428460}]},"active_addr":"172.21.15.31:6801/532428460","active_change":"2024-08-19T14:06:13.660762+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24107,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.31:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.31:0","nonce":296405905}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.31:0","nonce":611436414}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.31:0","nonce":2994076538}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.31:0","nonce":2685966395}]}]} 2024-08-19T14:11:52.297 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-08-19T14:11:52.297 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-08-19T14:11:52.297 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd dump --format=json 2024-08-19T14:11:52.550 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:11:52.828 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:52 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/4008556069' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-08-19T14:11:53.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:52 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/4008556069' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-08-19T14:11:53.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:52 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/4008556069' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-08-19T14:11:53.180 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:11:53.180 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":92,"fsid":"ed7d1c6e-5e33-11ef-bccf-c7b262605968","created":"2024-08-19T14:05:22.842169+0000","modified":"2024-08-19T14:11:47.545989+0000","last_up_change":"2024-08-19T14:11:39.491234+0000","last_in_change":"2024-08-19T14:11:23.117730+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-19T14:08:15.966221+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d7a0fff3-23c6-4ee3-a275-a844862aabfb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":1108945097},{"type":"v1","addr":"172.21.15.31:6803","nonce":1108945097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":1108945097},{"type":"v1","addr":"172.21.15.31:6805","nonce":1108945097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1108945097},{"type":"v1","addr":"172.21.15.31:6809","nonce":1108945097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":1108945097},{"type":"v1","addr":"172.21.15.31:6807","nonce":1108945097}]},"public_addr":"172.21.15.31:6803/1108945097","cluster_addr":"172.21.15.31:6805/1108945097","heartbeat_back_addr":"172.21.15.31:6809/1108945097","heartbeat_front_addr":"172.21.15.31:6807/1108945097","state":["exists","up"]},{"osd":1,"uuid":"145d5bb4-74dd-440f-bcc9-7e9ebda9c5d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":3236186806},{"type":"v1","addr":"172.21.15.31:6811","nonce":3236186806}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":3236186806},{"type":"v1","addr":"172.21.15.31:6813","nonce":3236186806}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":3236186806},{"type":"v1","addr":"172.21.15.31:6817","nonce":3236186806}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":3236186806},{"type":"v1","addr":"172.21.15.31:6815","nonce":3236186806}]},"public_addr":"172.21.15.31:6811/3236186806","cluster_addr":"172.21.15.31:6813/3236186806","heartbeat_back_addr":"172.21.15.31:6817/3236186806","heartbeat_front_addr":"172.21.15.31:6815/3236186806","state":["exists","up"]},{"osd":2,"uuid":"376c18d0-4768-46a0-bb60-e79c88ca909c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":4045553387},{"type":"v1","addr":"172.21.15.31:6819","nonce":4045553387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":4045553387},{"type":"v1","addr":"172.21.15.31:6821","nonce":4045553387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":4045553387},{"type":"v1","addr":"172.21.15.31:6825","nonce":4045553387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":4045553387},{"type":"v1","addr":"172.21.15.31:6823","nonce":4045553387}]},"public_addr":"172.21.15.31:6819/4045553387","cluster_addr":"172.21.15.31:6821/4045553387","heartbeat_back_addr":"172.21.15.31:6825/4045553387","heartbeat_front_addr":"172.21.15.31:6823/4045553387","state":["exists","up"]},{"osd":3,"uuid":"a787ea9f-b9ba-450f-b388-614cfba90634","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":2288082716},{"type":"v1","addr":"172.21.15.31:6827","nonce":2288082716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":2288082716},{"type":"v1","addr":"172.21.15.31:6829","nonce":2288082716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2288082716},{"type":"v1","addr":"172.21.15.31:6833","nonce":2288082716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":2288082716},{"type":"v1","addr":"172.21.15.31:6831","nonce":2288082716}]},"public_addr":"172.21.15.31:6827/2288082716","cluster_addr":"172.21.15.31:6829/2288082716","heartbeat_back_addr":"172.21.15.31:6833/2288082716","heartbeat_front_addr":"172.21.15.31:6831/2288082716","state":["exists","up"]},{"osd":4,"uuid":"7859751a-12f5-4a57-a332-435360479d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":3934739984},{"type":"v1","addr":"172.21.15.32:6801","nonce":3934739984}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":3934739984},{"type":"v1","addr":"172.21.15.32:6803","nonce":3934739984}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":3934739984},{"type":"v1","addr":"172.21.15.32:6807","nonce":3934739984}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":3934739984},{"type":"v1","addr":"172.21.15.32:6805","nonce":3934739984}]},"public_addr":"172.21.15.32:6801/3934739984","cluster_addr":"172.21.15.32:6803/3934739984","heartbeat_back_addr":"172.21.15.32:6807/3934739984","heartbeat_front_addr":"172.21.15.32:6805/3934739984","state":["exists","up"]},{"osd":5,"uuid":"017f3eef-ac60-4b82-8350-919017cb4ccd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":526388985},{"type":"v1","addr":"172.21.15.32:6809","nonce":526388985}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":526388985},{"type":"v1","addr":"172.21.15.32:6811","nonce":526388985}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":526388985},{"type":"v1","addr":"172.21.15.32:6815","nonce":526388985}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":526388985},{"type":"v1","addr":"172.21.15.32:6813","nonce":526388985}]},"public_addr":"172.21.15.32:6809/526388985","cluster_addr":"172.21.15.32:6811/526388985","heartbeat_back_addr":"172.21.15.32:6815/526388985","heartbeat_front_addr":"172.21.15.32:6813/526388985","state":["exists","up"]},{"osd":6,"uuid":"44aa8229-0268-49d7-bd57-3bf1f384e5e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":1131035844},{"type":"v1","addr":"172.21.15.32:6817","nonce":1131035844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":1131035844},{"type":"v1","addr":"172.21.15.32:6819","nonce":1131035844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":1131035844},{"type":"v1","addr":"172.21.15.32:6823","nonce":1131035844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":1131035844},{"type":"v1","addr":"172.21.15.32:6821","nonce":1131035844}]},"public_addr":"172.21.15.32:6817/1131035844","cluster_addr":"172.21.15.32:6819/1131035844","heartbeat_back_addr":"172.21.15.32:6823/1131035844","heartbeat_front_addr":"172.21.15.32:6821/1131035844","state":["exists","up"]},{"osd":7,"uuid":"263433e5-70b6-4192-86ff-dc2ad95927eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":2014939947},{"type":"v1","addr":"172.21.15.32:6825","nonce":2014939947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":2014939947},{"type":"v1","addr":"172.21.15.32:6827","nonce":2014939947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":2014939947},{"type":"v1","addr":"172.21.15.32:6831","nonce":2014939947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":2014939947},{"type":"v1","addr":"172.21.15.32:6829","nonce":2014939947}]},"public_addr":"172.21.15.32:6825/2014939947","cluster_addr":"172.21.15.32:6827/2014939947","heartbeat_back_addr":"172.21.15.32:6831/2014939947","heartbeat_front_addr":"172.21.15.32:6829/2014939947","state":["exists","up"]},{"osd":8,"uuid":"2bbcad36-0c74-4528-9fb9-a8812afa1662","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1445140892},{"type":"v1","addr":"172.21.15.45:6801","nonce":1445140892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1445140892},{"type":"v1","addr":"172.21.15.45:6803","nonce":1445140892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1445140892},{"type":"v1","addr":"172.21.15.45:6807","nonce":1445140892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1445140892},{"type":"v1","addr":"172.21.15.45:6805","nonce":1445140892}]},"public_addr":"172.21.15.45:6801/1445140892","cluster_addr":"172.21.15.45:6803/1445140892","heartbeat_back_addr":"172.21.15.45:6807/1445140892","heartbeat_front_addr":"172.21.15.45:6805/1445140892","state":["exists","up"]},{"osd":9,"uuid":"c8ee4277-3bf9-46bf-9cc0-152c1c4b9508","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1213787169},{"type":"v1","addr":"172.21.15.45:6809","nonce":1213787169}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1213787169},{"type":"v1","addr":"172.21.15.45:6811","nonce":1213787169}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1213787169},{"type":"v1","addr":"172.21.15.45:6815","nonce":1213787169}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1213787169},{"type":"v1","addr":"172.21.15.45:6813","nonce":1213787169}]},"public_addr":"172.21.15.45:6809/1213787169","cluster_addr":"172.21.15.45:6811/1213787169","heartbeat_back_addr":"172.21.15.45:6815/1213787169","heartbeat_front_addr":"172.21.15.45:6813/1213787169","state":["exists","up"]},{"osd":10,"uuid":"29031395-5f5c-40f8-ac90-9f6ac48e5c9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2461824621},{"type":"v1","addr":"172.21.15.45:6817","nonce":2461824621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2461824621},{"type":"v1","addr":"172.21.15.45:6819","nonce":2461824621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2461824621},{"type":"v1","addr":"172.21.15.45:6823","nonce":2461824621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2461824621},{"type":"v1","addr":"172.21.15.45:6821","nonce":2461824621}]},"public_addr":"172.21.15.45:6817/2461824621","cluster_addr":"172.21.15.45:6819/2461824621","heartbeat_back_addr":"172.21.15.45:6823/2461824621","heartbeat_front_addr":"172.21.15.45:6821/2461824621","state":["exists","up"]},{"osd":11,"uuid":"5ef0db4b-0fef-4ea0-a19b-d3677e082c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":86,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2438396298},{"type":"v1","addr":"172.21.15.45:6825","nonce":2438396298}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":2438396298},{"type":"v1","addr":"172.21.15.45:6827","nonce":2438396298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":2438396298},{"type":"v1","addr":"172.21.15.45:6831","nonce":2438396298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":2438396298},{"type":"v1","addr":"172.21.15.45:6829","nonce":2438396298}]},"public_addr":"172.21.15.45:6825/2438396298","cluster_addr":"172.21.15.45:6827/2438396298","heartbeat_back_addr":"172.21.15.45:6831/2438396298","heartbeat_front_addr":"172.21.15.45:6829/2438396298","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T14:11:15.004355+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T14:11:47.383044+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T14:10:41.958173+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T14:11:05.347211+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T14:11:20.435642+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T14:10:41.507554+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T14:10:51.578661+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T14:11:14.910669+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T14:11:45.368064+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T14:10:50.520674+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T14:11:14.756036+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T14:11:38.416189+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/1451745969":"2024-08-20T14:06:13.660657+0000","172.21.15.31:0/1515698918":"2024-08-20T14:06:13.660657+0000","172.21.15.31:6801/3297539242":"2024-08-20T14:06:13.660657+0000","172.21.15.31:6800/3297539242":"2024-08-20T14:06:13.660657+0000","172.21.15.31:0/2217423818":"2024-08-20T14:05:48.827576+0000","172.21.15.31:0/3353498281":"2024-08-20T14:06:13.660657+0000","172.21.15.31:0/4013594220":"2024-08-20T14:05:48.827576+0000","172.21.15.31:0/1654518065":"2024-08-20T14:05:48.827576+0000","172.21.15.31:6801/2368821258":"2024-08-20T14:05:48.827576+0000","172.21.15.31:6800/2368821258":"2024-08-20T14:05:48.827576+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-19T14:11:53.745 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:53 smithi031 ceph-mon[16336]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:11:53.745 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:53 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/4131788970' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-08-19T14:11:53.773 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-08-19T14:11:53.774 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd dump --format=json 2024-08-19T14:11:53.909 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:53 smithi032 ceph-mon[20004]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:11:53.909 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:53 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/4131788970' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-08-19T14:11:54.029 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:11:54.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:53 smithi045 ceph-mon[20143]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:11:54.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:53 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/4131788970' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-08-19T14:11:54.667 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:11:54.667 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":92,"fsid":"ed7d1c6e-5e33-11ef-bccf-c7b262605968","created":"2024-08-19T14:05:22.842169+0000","modified":"2024-08-19T14:11:47.545989+0000","last_up_change":"2024-08-19T14:11:39.491234+0000","last_in_change":"2024-08-19T14:11:23.117730+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-19T14:08:15.966221+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d7a0fff3-23c6-4ee3-a275-a844862aabfb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":1108945097},{"type":"v1","addr":"172.21.15.31:6803","nonce":1108945097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":1108945097},{"type":"v1","addr":"172.21.15.31:6805","nonce":1108945097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1108945097},{"type":"v1","addr":"172.21.15.31:6809","nonce":1108945097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":1108945097},{"type":"v1","addr":"172.21.15.31:6807","nonce":1108945097}]},"public_addr":"172.21.15.31:6803/1108945097","cluster_addr":"172.21.15.31:6805/1108945097","heartbeat_back_addr":"172.21.15.31:6809/1108945097","heartbeat_front_addr":"172.21.15.31:6807/1108945097","state":["exists","up"]},{"osd":1,"uuid":"145d5bb4-74dd-440f-bcc9-7e9ebda9c5d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":3236186806},{"type":"v1","addr":"172.21.15.31:6811","nonce":3236186806}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":3236186806},{"type":"v1","addr":"172.21.15.31:6813","nonce":3236186806}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":3236186806},{"type":"v1","addr":"172.21.15.31:6817","nonce":3236186806}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":3236186806},{"type":"v1","addr":"172.21.15.31:6815","nonce":3236186806}]},"public_addr":"172.21.15.31:6811/3236186806","cluster_addr":"172.21.15.31:6813/3236186806","heartbeat_back_addr":"172.21.15.31:6817/3236186806","heartbeat_front_addr":"172.21.15.31:6815/3236186806","state":["exists","up"]},{"osd":2,"uuid":"376c18d0-4768-46a0-bb60-e79c88ca909c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":4045553387},{"type":"v1","addr":"172.21.15.31:6819","nonce":4045553387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":4045553387},{"type":"v1","addr":"172.21.15.31:6821","nonce":4045553387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":4045553387},{"type":"v1","addr":"172.21.15.31:6825","nonce":4045553387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":4045553387},{"type":"v1","addr":"172.21.15.31:6823","nonce":4045553387}]},"public_addr":"172.21.15.31:6819/4045553387","cluster_addr":"172.21.15.31:6821/4045553387","heartbeat_back_addr":"172.21.15.31:6825/4045553387","heartbeat_front_addr":"172.21.15.31:6823/4045553387","state":["exists","up"]},{"osd":3,"uuid":"a787ea9f-b9ba-450f-b388-614cfba90634","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":2288082716},{"type":"v1","addr":"172.21.15.31:6827","nonce":2288082716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":2288082716},{"type":"v1","addr":"172.21.15.31:6829","nonce":2288082716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2288082716},{"type":"v1","addr":"172.21.15.31:6833","nonce":2288082716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":2288082716},{"type":"v1","addr":"172.21.15.31:6831","nonce":2288082716}]},"public_addr":"172.21.15.31:6827/2288082716","cluster_addr":"172.21.15.31:6829/2288082716","heartbeat_back_addr":"172.21.15.31:6833/2288082716","heartbeat_front_addr":"172.21.15.31:6831/2288082716","state":["exists","up"]},{"osd":4,"uuid":"7859751a-12f5-4a57-a332-435360479d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":3934739984},{"type":"v1","addr":"172.21.15.32:6801","nonce":3934739984}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":3934739984},{"type":"v1","addr":"172.21.15.32:6803","nonce":3934739984}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":3934739984},{"type":"v1","addr":"172.21.15.32:6807","nonce":3934739984}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":3934739984},{"type":"v1","addr":"172.21.15.32:6805","nonce":3934739984}]},"public_addr":"172.21.15.32:6801/3934739984","cluster_addr":"172.21.15.32:6803/3934739984","heartbeat_back_addr":"172.21.15.32:6807/3934739984","heartbeat_front_addr":"172.21.15.32:6805/3934739984","state":["exists","up"]},{"osd":5,"uuid":"017f3eef-ac60-4b82-8350-919017cb4ccd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":526388985},{"type":"v1","addr":"172.21.15.32:6809","nonce":526388985}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":526388985},{"type":"v1","addr":"172.21.15.32:6811","nonce":526388985}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":526388985},{"type":"v1","addr":"172.21.15.32:6815","nonce":526388985}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":526388985},{"type":"v1","addr":"172.21.15.32:6813","nonce":526388985}]},"public_addr":"172.21.15.32:6809/526388985","cluster_addr":"172.21.15.32:6811/526388985","heartbeat_back_addr":"172.21.15.32:6815/526388985","heartbeat_front_addr":"172.21.15.32:6813/526388985","state":["exists","up"]},{"osd":6,"uuid":"44aa8229-0268-49d7-bd57-3bf1f384e5e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":1131035844},{"type":"v1","addr":"172.21.15.32:6817","nonce":1131035844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":1131035844},{"type":"v1","addr":"172.21.15.32:6819","nonce":1131035844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":1131035844},{"type":"v1","addr":"172.21.15.32:6823","nonce":1131035844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":1131035844},{"type":"v1","addr":"172.21.15.32:6821","nonce":1131035844}]},"public_addr":"172.21.15.32:6817/1131035844","cluster_addr":"172.21.15.32:6819/1131035844","heartbeat_back_addr":"172.21.15.32:6823/1131035844","heartbeat_front_addr":"172.21.15.32:6821/1131035844","state":["exists","up"]},{"osd":7,"uuid":"263433e5-70b6-4192-86ff-dc2ad95927eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":2014939947},{"type":"v1","addr":"172.21.15.32:6825","nonce":2014939947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":2014939947},{"type":"v1","addr":"172.21.15.32:6827","nonce":2014939947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":2014939947},{"type":"v1","addr":"172.21.15.32:6831","nonce":2014939947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":2014939947},{"type":"v1","addr":"172.21.15.32:6829","nonce":2014939947}]},"public_addr":"172.21.15.32:6825/2014939947","cluster_addr":"172.21.15.32:6827/2014939947","heartbeat_back_addr":"172.21.15.32:6831/2014939947","heartbeat_front_addr":"172.21.15.32:6829/2014939947","state":["exists","up"]},{"osd":8,"uuid":"2bbcad36-0c74-4528-9fb9-a8812afa1662","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1445140892},{"type":"v1","addr":"172.21.15.45:6801","nonce":1445140892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1445140892},{"type":"v1","addr":"172.21.15.45:6803","nonce":1445140892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1445140892},{"type":"v1","addr":"172.21.15.45:6807","nonce":1445140892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1445140892},{"type":"v1","addr":"172.21.15.45:6805","nonce":1445140892}]},"public_addr":"172.21.15.45:6801/1445140892","cluster_addr":"172.21.15.45:6803/1445140892","heartbeat_back_addr":"172.21.15.45:6807/1445140892","heartbeat_front_addr":"172.21.15.45:6805/1445140892","state":["exists","up"]},{"osd":9,"uuid":"c8ee4277-3bf9-46bf-9cc0-152c1c4b9508","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1213787169},{"type":"v1","addr":"172.21.15.45:6809","nonce":1213787169}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1213787169},{"type":"v1","addr":"172.21.15.45:6811","nonce":1213787169}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1213787169},{"type":"v1","addr":"172.21.15.45:6815","nonce":1213787169}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1213787169},{"type":"v1","addr":"172.21.15.45:6813","nonce":1213787169}]},"public_addr":"172.21.15.45:6809/1213787169","cluster_addr":"172.21.15.45:6811/1213787169","heartbeat_back_addr":"172.21.15.45:6815/1213787169","heartbeat_front_addr":"172.21.15.45:6813/1213787169","state":["exists","up"]},{"osd":10,"uuid":"29031395-5f5c-40f8-ac90-9f6ac48e5c9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2461824621},{"type":"v1","addr":"172.21.15.45:6817","nonce":2461824621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2461824621},{"type":"v1","addr":"172.21.15.45:6819","nonce":2461824621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2461824621},{"type":"v1","addr":"172.21.15.45:6823","nonce":2461824621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2461824621},{"type":"v1","addr":"172.21.15.45:6821","nonce":2461824621}]},"public_addr":"172.21.15.45:6817/2461824621","cluster_addr":"172.21.15.45:6819/2461824621","heartbeat_back_addr":"172.21.15.45:6823/2461824621","heartbeat_front_addr":"172.21.15.45:6821/2461824621","state":["exists","up"]},{"osd":11,"uuid":"5ef0db4b-0fef-4ea0-a19b-d3677e082c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":86,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2438396298},{"type":"v1","addr":"172.21.15.45:6825","nonce":2438396298}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":2438396298},{"type":"v1","addr":"172.21.15.45:6827","nonce":2438396298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":2438396298},{"type":"v1","addr":"172.21.15.45:6831","nonce":2438396298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":2438396298},{"type":"v1","addr":"172.21.15.45:6829","nonce":2438396298}]},"public_addr":"172.21.15.45:6825/2438396298","cluster_addr":"172.21.15.45:6827/2438396298","heartbeat_back_addr":"172.21.15.45:6831/2438396298","heartbeat_front_addr":"172.21.15.45:6829/2438396298","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T14:11:15.004355+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T14:11:47.383044+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T14:10:41.958173+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T14:11:05.347211+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T14:11:20.435642+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T14:10:41.507554+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T14:10:51.578661+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T14:11:14.910669+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T14:11:45.368064+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T14:10:50.520674+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T14:11:14.756036+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T14:11:38.416189+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/1451745969":"2024-08-20T14:06:13.660657+0000","172.21.15.31:0/1515698918":"2024-08-20T14:06:13.660657+0000","172.21.15.31:6801/3297539242":"2024-08-20T14:06:13.660657+0000","172.21.15.31:6800/3297539242":"2024-08-20T14:06:13.660657+0000","172.21.15.31:0/2217423818":"2024-08-20T14:05:48.827576+0000","172.21.15.31:0/3353498281":"2024-08-20T14:06:13.660657+0000","172.21.15.31:0/4013594220":"2024-08-20T14:05:48.827576+0000","172.21.15.31:0/1654518065":"2024-08-20T14:05:48.827576+0000","172.21.15.31:6801/2368821258":"2024-08-20T14:05:48.827576+0000","172.21.15.31:6800/2368821258":"2024-08-20T14:05:48.827576+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-19T14:11:55.292 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-08-19T14:11:55.292 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-08-19T14:11:55.293 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-08-19T14:11:55.293 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-08-19T14:11:55.293 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-08-19T14:11:55.294 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-08-19T14:11:55.294 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-08-19T14:11:55.295 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-08-19T14:11:55.295 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-08-19T14:11:55.296 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-08-19T14:11:55.296 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-08-19T14:11:55.297 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-08-19T14:11:55.576 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:55 smithi031 ceph-mon[16336]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:11:55.576 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:55 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/241023553' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-08-19T14:11:55.847 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:55 smithi045 ceph-mon[20143]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:11:55.848 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:55 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/241023553' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-08-19T14:11:55.971 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:11:55.974 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:11:55.974 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:11:56.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:55 smithi032 ceph-mon[20004]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:11:56.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:55 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/241023553' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-08-19T14:11:56.012 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:11:56.109 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:11:56.109 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:11:56.710 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:56 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:11:56.711 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:56 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:11:56.870 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:56 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:11:56.870 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:56 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:11:57.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:56 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:11:57.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:56 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:11:57.044 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:11:57.044 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:11:57.044 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:11:57.045 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:11:57.046 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:11:57.047 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:11:57.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:57 smithi031 ceph-mon[16336]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:11:57.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:57 smithi031 ceph-mon[16336]: osdmap e93: 12 total, 12 up, 12 in 2024-08-19T14:11:57.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:57 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:11:57.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:57 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:11:58.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:57 smithi032 ceph-mon[20004]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:11:58.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:57 smithi032 ceph-mon[20004]: osdmap e93: 12 total, 12 up, 12 in 2024-08-19T14:11:58.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:57 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:11:58.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:57 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:11:58.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:57 smithi045 ceph-mon[20143]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:11:58.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:57 smithi045 ceph-mon[20143]: osdmap e93: 12 total, 12 up, 12 in 2024-08-19T14:11:58.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:57 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:11:58.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:57 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:11:58.645 INFO:teuthology.orchestra.run.smithi031.stdout:171798691873 2024-08-19T14:11:58.645 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-08-19T14:11:58.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:58 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:11:58.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:58 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:11:58.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:58 smithi031 ceph-mon[16336]: osdmap e94: 12 total, 12 up, 12 in 2024-08-19T14:11:59.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:58 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:11:59.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:58 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:11:59.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:58 smithi032 ceph-mon[20004]: osdmap e94: 12 total, 12 up, 12 in 2024-08-19T14:11:59.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:58 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:11:59.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:58 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:11:59.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:58 smithi045 ceph-mon[20143]: osdmap e94: 12 total, 12 up, 12 in 2024-08-19T14:11:59.400 INFO:teuthology.orchestra.run.smithi031.stdout:206158430236 2024-08-19T14:11:59.400 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-08-19T14:11:59.781 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:11:59 smithi031 ceph-mon[16336]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:12:00.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:11:59 smithi032 ceph-mon[20004]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:12:00.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:11:59 smithi045 ceph-mon[20143]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:12:00.093 INFO:teuthology.orchestra.run.smithi031.stdout:244813135896 2024-08-19T14:12:00.093 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-08-19T14:12:01.023 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:12:01.426 INFO:teuthology.orchestra.run.smithi031.stdout:343597383692 2024-08-19T14:12:01.426 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-08-19T14:12:01.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:01 smithi031 ceph-mon[16336]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:12:01.919 INFO:teuthology.orchestra.run.smithi031.stdout:133143986214 2024-08-19T14:12:01.920 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-08-19T14:12:02.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:01 smithi032 ceph-mon[20004]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:12:02.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:01 smithi045 ceph-mon[20143]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:12:03.188 INFO:teuthology.orchestra.run.smithi031.stdout:369367187463 2024-08-19T14:12:03.189 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-08-19T14:12:03.715 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:03 smithi031 ceph-mon[16336]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:12:03.715 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {auth=25,config=2,health=12,kv=61} crc {auth=1101443481,config=358384381,health=3897446751,kv=1777167106}) 2024-08-19T14:12:03.715 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=1391783278}) 2024-08-19T14:12:03.715 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {kv=94,logm=6} crc {kv=952528024,logm=2926248019}) 2024-08-19T14:12:03.715 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=963178822}) 2024-08-19T14:12:03.716 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1005077160}) 2024-08-19T14:12:03.716 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1050219514}) 2024-08-19T14:12:03.716 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3764259196}) 2024-08-19T14:12:03.716 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=27820230}) 2024-08-19T14:12:03.716 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3020014327}) 2024-08-19T14:12:03.716 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3954177880}) 2024-08-19T14:12:03.716 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1261178317}) 2024-08-19T14:12:03.716 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2069994730}) 2024-08-19T14:12:03.716 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2628432375}) 2024-08-19T14:12:03.716 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3442964405}) 2024-08-19T14:12:03.716 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1780536969}) 2024-08-19T14:12:03.716 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2200711492}) 2024-08-19T14:12:03.716 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=879794513}) 2024-08-19T14:12:03.716 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=349043006}) 2024-08-19T14:12:03.717 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4294713893}) 2024-08-19T14:12:03.717 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=32,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=44} crc {logm=3748162072,mdsmap=2474562478,mgr=2924679688,mgr_command_descs=1300230762,mgr_metadata=482233419,mgrstat=1585054585}) 2024-08-19T14:12:03.717 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=100} crc {mgrstat=860646894}) 2024-08-19T14:12:03.717 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=19,mon_config_key=81} crc {mgrstat=1149023864,mon_config_key=268930335}) 2024-08-19T14:12:03.717 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1196203817}) 2024-08-19T14:12:03.717 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=89,monmap=5,osd_metadata=6} crc {mon_config_key=3568880206,monmap=3193798142,osd_metadata=1272156132}) 2024-08-19T14:12:03.717 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {osd_metadata=6,osd_pg_creating=1,osdmap=93} crc {osd_metadata=369016071,osd_pg_creating=221095979,osdmap=2259984207}) 2024-08-19T14:12:03.717 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=98} crc {osdmap=2409524021}) 2024-08-19T14:12:04.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:03 smithi032 ceph-mon[20004]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:12:04.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {auth=25,config=2,health=12,kv=61} crc {auth=1101443481,config=358384381,health=3897446751,kv=1777167106}) 2024-08-19T14:12:04.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=1391783278}) 2024-08-19T14:12:04.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {kv=94,logm=6} crc {kv=952528024,logm=2926248019}) 2024-08-19T14:12:04.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=963178822}) 2024-08-19T14:12:04.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1005077160}) 2024-08-19T14:12:04.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1050219514}) 2024-08-19T14:12:04.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3764259196}) 2024-08-19T14:12:04.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=27820230}) 2024-08-19T14:12:04.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3020014327}) 2024-08-19T14:12:04.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3954177880}) 2024-08-19T14:12:04.009 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1261178317}) 2024-08-19T14:12:04.009 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2069994730}) 2024-08-19T14:12:04.009 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2628432375}) 2024-08-19T14:12:04.009 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3442964405}) 2024-08-19T14:12:04.009 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1780536969}) 2024-08-19T14:12:04.009 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2200711492}) 2024-08-19T14:12:04.009 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=879794513}) 2024-08-19T14:12:04.009 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=349043006}) 2024-08-19T14:12:04.009 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4294713893}) 2024-08-19T14:12:04.009 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=32,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=44} crc {logm=3748162072,mdsmap=2474562478,mgr=2924679688,mgr_command_descs=1300230762,mgr_metadata=482233419,mgrstat=1585054585}) 2024-08-19T14:12:04.009 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=100} crc {mgrstat=860646894}) 2024-08-19T14:12:04.009 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=19,mon_config_key=81} crc {mgrstat=1149023864,mon_config_key=268930335}) 2024-08-19T14:12:04.009 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1196203817}) 2024-08-19T14:12:04.010 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=89,monmap=5,osd_metadata=6} crc {mon_config_key=3568880206,monmap=3193798142,osd_metadata=1272156132}) 2024-08-19T14:12:04.010 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {osd_metadata=6,osd_pg_creating=1,osdmap=93} crc {osd_metadata=369016071,osd_pg_creating=221095979,osdmap=2259984207}) 2024-08-19T14:12:04.010 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=98} crc {osdmap=2409524021}) 2024-08-19T14:12:04.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:03 smithi045 ceph-mon[20143]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:12:04.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {auth=25,config=2,health=12,kv=61} crc {auth=1101443481,config=358384381,health=3897446751,kv=1777167106}) 2024-08-19T14:12:04.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=1391783278}) 2024-08-19T14:12:04.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {kv=94,logm=6} crc {kv=952528024,logm=2926248019}) 2024-08-19T14:12:04.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=963178822}) 2024-08-19T14:12:04.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1005077160}) 2024-08-19T14:12:04.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1050219514}) 2024-08-19T14:12:04.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3764259196}) 2024-08-19T14:12:04.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=27820230}) 2024-08-19T14:12:04.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3020014327}) 2024-08-19T14:12:04.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3954177880}) 2024-08-19T14:12:04.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1261178317}) 2024-08-19T14:12:04.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2069994730}) 2024-08-19T14:12:04.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2628432375}) 2024-08-19T14:12:04.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3442964405}) 2024-08-19T14:12:04.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1780536969}) 2024-08-19T14:12:04.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2200711492}) 2024-08-19T14:12:04.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=879794513}) 2024-08-19T14:12:04.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=349043006}) 2024-08-19T14:12:04.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4294713893}) 2024-08-19T14:12:04.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=32,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=44} crc {logm=3748162072,mdsmap=2474562478,mgr=2924679688,mgr_command_descs=1300230762,mgr_metadata=482233419,mgrstat=1585054585}) 2024-08-19T14:12:04.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=100} crc {mgrstat=860646894}) 2024-08-19T14:12:04.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=19,mon_config_key=81} crc {mgrstat=1149023864,mon_config_key=268930335}) 2024-08-19T14:12:04.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1196203817}) 2024-08-19T14:12:04.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=89,monmap=5,osd_metadata=6} crc {mon_config_key=3568880206,monmap=3193798142,osd_metadata=1272156132}) 2024-08-19T14:12:04.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {osd_metadata=6,osd_pg_creating=1,osdmap=93} crc {osd_metadata=369016071,osd_pg_creating=221095979,osdmap=2259984207}) 2024-08-19T14:12:04.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=98} crc {osdmap=2409524021}) 2024-08-19T14:12:04.697 INFO:teuthology.orchestra.run.smithi031.stdout:171798691874 2024-08-19T14:12:05.349 INFO:tasks.cephadm.ceph_manager.ceph:need seq 171798691873 got 171798691874 for osd.5 2024-08-19T14:12:05.350 DEBUG:teuthology.parallel:result is None 2024-08-19T14:12:05.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:05 smithi031 ceph-mon[16336]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:12:05.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:05 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1359061864' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-08-19T14:12:05.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:05 smithi031 ceph-mon[16336]: 1.0 scrub starts 2024-08-19T14:12:05.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:05 smithi031 ceph-mon[16336]: 1.0 scrub ok 2024-08-19T14:12:06.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:05 smithi032 ceph-mon[20004]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:12:06.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:05 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1359061864' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-08-19T14:12:06.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:05 smithi032 ceph-mon[20004]: 1.0 scrub starts 2024-08-19T14:12:06.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:05 smithi032 ceph-mon[20004]: 1.0 scrub ok 2024-08-19T14:12:06.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:05 smithi045 ceph-mon[20143]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:12:06.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:05 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1359061864' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-08-19T14:12:06.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:05 smithi045 ceph-mon[20143]: 1.0 scrub starts 2024-08-19T14:12:06.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:05 smithi045 ceph-mon[20143]: 1.0 scrub ok 2024-08-19T14:12:06.571 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:12:07.191 INFO:teuthology.orchestra.run.smithi031.stdout:111669149740 2024-08-19T14:12:07.191 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-08-19T14:12:07.701 INFO:teuthology.orchestra.run.smithi031.stdout:279172874263 2024-08-19T14:12:07.701 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-08-19T14:12:07.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:07 smithi031 ceph-mon[16336]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:12:08.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:07 smithi032 ceph-mon[20004]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:12:08.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:07 smithi045 ceph-mon[20143]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:12:08.133 INFO:teuthology.orchestra.run.smithi031.stdout:317827579922 2024-08-19T14:12:08.134 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-08-19T14:12:08.635 INFO:teuthology.orchestra.run.smithi031.stdout:77309411377 2024-08-19T14:12:08.635 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-08-19T14:12:08.955 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:12:08.958 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:12:08.958 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:12:09.653 INFO:teuthology.orchestra.run.smithi031.stdout:206158430238 2024-08-19T14:12:09.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:09 smithi031 ceph-mon[16336]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:12:10.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:09 smithi032 ceph-mon[20004]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:12:10.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:09 smithi045 ceph-mon[20143]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:12:10.059 INFO:teuthology.orchestra.run.smithi031.stdout:55834574902 2024-08-19T14:12:10.059 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-08-19T14:12:10.866 INFO:teuthology.orchestra.run.smithi031.stdout:244813135899 2024-08-19T14:12:10.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:10 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1572874120' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-08-19T14:12:11.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:10 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1572874120' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-08-19T14:12:11.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:10 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1572874120' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-08-19T14:12:11.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:11 smithi031 ceph-mon[16336]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:12:11.742 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:11 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2163218217' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-08-19T14:12:11.790 INFO:teuthology.orchestra.run.smithi031.stdout:34359738428 2024-08-19T14:12:11.790 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-08-19T14:12:12.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:11 smithi032 ceph-mon[20004]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:12:12.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:11 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2163218217' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-08-19T14:12:12.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:11 smithi045 ceph-mon[20143]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:12:12.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:11 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2163218217' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-08-19T14:12:12.196 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:12:12.281 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:12:12.282 INFO:tasks.cephadm.ceph_manager.ceph:need seq 206158430236 got 206158430238 for osd.6 2024-08-19T14:12:12.282 DEBUG:teuthology.parallel:result is None 2024-08-19T14:12:13.582 INFO:teuthology.orchestra.run.smithi031.stdout:317827579923 2024-08-19T14:12:13.689 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:13 smithi031 ceph-mon[16336]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:12:13.689 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:13 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/4179523685' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-08-19T14:12:13.758 INFO:teuthology.orchestra.run.smithi031.stdout:77309411379 2024-08-19T14:12:13.961 INFO:teuthology.orchestra.run.smithi031.stdout:343597383694 2024-08-19T14:12:14.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:13 smithi032 ceph-mon[20004]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:12:14.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:13 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/4179523685' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-08-19T14:12:14.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:13 smithi045 ceph-mon[20143]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:12:14.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:13 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/4179523685' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-08-19T14:12:14.535 INFO:teuthology.orchestra.run.smithi031.stdout:133143986216 2024-08-19T14:12:14.762 INFO:tasks.cephadm.ceph_manager.ceph:need seq 244813135896 got 244813135899 for osd.7 2024-08-19T14:12:14.763 DEBUG:teuthology.parallel:result is None 2024-08-19T14:12:14.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:14 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3464255409' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-08-19T14:12:14.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:14 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:12:14.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:14 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:12:14.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:14 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1555412055' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-08-19T14:12:14.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:14 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3074755294' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-08-19T14:12:15.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:14 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3464255409' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-08-19T14:12:15.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:14 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:12:15.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:14 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:12:15.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:14 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1555412055' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-08-19T14:12:15.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:14 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3074755294' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-08-19T14:12:15.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:14 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3464255409' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-08-19T14:12:15.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:14 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:12:15.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:14 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:12:15.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:14 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1555412055' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-08-19T14:12:15.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:14 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3074755294' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-08-19T14:12:15.549 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986214 got 133143986216 for osd.4 2024-08-19T14:12:15.549 DEBUG:teuthology.parallel:result is None 2024-08-19T14:12:15.819 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:15 smithi031 ceph-mon[16336]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:12:15.819 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:15 smithi031 ceph-mon[16336]: osdmap e95: 12 total, 12 up, 12 in 2024-08-19T14:12:16.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:15 smithi032 ceph-mon[20004]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:12:16.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:15 smithi032 ceph-mon[20004]: osdmap e95: 12 total, 12 up, 12 in 2024-08-19T14:12:16.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:15 smithi045 ceph-mon[20143]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:12:16.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:15 smithi045 ceph-mon[20143]: osdmap e95: 12 total, 12 up, 12 in 2024-08-19T14:12:16.152 INFO:tasks.cephadm.ceph_manager.ceph:need seq 317827579922 got 317827579923 for osd.9 2024-08-19T14:12:16.153 DEBUG:teuthology.parallel:result is None 2024-08-19T14:12:16.597 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411377 got 77309411379 for osd.2 2024-08-19T14:12:16.597 DEBUG:teuthology.parallel:result is None 2024-08-19T14:12:16.607 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:12:16.608 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:12:16.609 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:12:17.004 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:12:17.037 INFO:tasks.cephadm.ceph_manager.ceph:need seq 343597383692 got 343597383694 for osd.10 2024-08-19T14:12:17.037 DEBUG:teuthology.parallel:result is None 2024-08-19T14:12:17.739 INFO:teuthology.orchestra.run.smithi031.stdout:111669149743 2024-08-19T14:12:17.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:17 smithi031 ceph-mon[16336]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:12:18.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:17 smithi032 ceph-mon[20004]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:12:18.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:17 smithi045 ceph-mon[20143]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:12:18.361 INFO:teuthology.orchestra.run.smithi031.stdout:55834574904 2024-08-19T14:12:18.433 INFO:teuthology.orchestra.run.smithi031.stdout:369367187467 2024-08-19T14:12:18.853 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:18 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1846431860' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-08-19T14:12:18.853 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:18 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1329268055' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-08-19T14:12:18.853 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:18 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1693079138' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-08-19T14:12:19.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:18 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1846431860' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-08-19T14:12:19.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:18 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1329268055' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-08-19T14:12:19.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:18 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1693079138' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-08-19T14:12:19.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:18 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1846431860' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-08-19T14:12:19.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:18 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1329268055' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-08-19T14:12:19.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:18 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1693079138' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-08-19T14:12:19.271 INFO:tasks.cephadm.ceph_manager.ceph:need seq 369367187463 got 369367187467 for osd.11 2024-08-19T14:12:19.271 DEBUG:teuthology.parallel:result is None 2024-08-19T14:12:19.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:19 smithi031 ceph-mon[16336]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:12:19.930 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149740 got 111669149743 for osd.3 2024-08-19T14:12:19.930 DEBUG:teuthology.parallel:result is None 2024-08-19T14:12:20.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:19 smithi032 ceph-mon[20004]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:12:20.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:19 smithi045 ceph-mon[20143]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:12:20.576 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:12:20.606 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574902 got 55834574904 for osd.1 2024-08-19T14:12:20.606 DEBUG:teuthology.parallel:result is None 2024-08-19T14:12:21.222 INFO:teuthology.orchestra.run.smithi031.stdout:279172874267 2024-08-19T14:12:21.436 INFO:teuthology.orchestra.run.smithi031.stdout:34359738429 2024-08-19T14:12:21.803 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:21 smithi031 ceph-mon[16336]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:12:21.804 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:21 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1750644476' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-08-19T14:12:21.804 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:21 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3138483908' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-08-19T14:12:21.829 INFO:tasks.cephadm.ceph_manager.ceph:need seq 279172874263 got 279172874267 for osd.8 2024-08-19T14:12:21.830 DEBUG:teuthology.parallel:result is None 2024-08-19T14:12:22.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:21 smithi032 ceph-mon[20004]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:12:22.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:21 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1750644476' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-08-19T14:12:22.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:21 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3138483908' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-08-19T14:12:22.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:21 smithi045 ceph-mon[20143]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:12:22.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:21 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1750644476' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-08-19T14:12:22.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:21 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3138483908' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-08-19T14:12:22.649 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738428 got 34359738429 for osd.0 2024-08-19T14:12:22.649 DEBUG:teuthology.parallel:result is None 2024-08-19T14:12:22.649 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-08-19T14:12:22.649 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-19T14:12:22.907 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:12:23.545 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:12:23.545 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-08-19T14:12:23.863 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:23 smithi031 ceph-mon[16336]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:12:24.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:23 smithi032 ceph-mon[20004]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:12:24.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:23 smithi045 ceph-mon[20143]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:12:24.166 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":269,"stamp":"2024-08-19T14:12:21.729130+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":3,"kb":1124941824,"kb_used":362712,"kb_used_data":13512,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124579112,"statfs":{"total":1151940427776,"available":1151569010688,"internally_reserved":0,"allocated":13836288,"data_stored":7387152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":357414202},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001542"},"pg_stats":[{"pgid":"1.0","version":"70'139","reported_seq":347,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-08-19T14:12:15.558467+0000","last_change":"2024-08-19T14:12:04.853784+0000","last_active":"2024-08-19T14:12:15.558467+0000","last_peered":"2024-08-19T14:12:15.558467+0000","last_clean":"2024-08-19T14:12:15.558467+0000","last_became_active":"2024-08-19T14:11:43.524419+0000","last_became_peered":"2024-08-19T14:11:43.524419+0000","last_unstale":"2024-08-19T14:12:15.558467+0000","last_undegraded":"2024-08-19T14:12:15.558467+0000","last_fullsized":"2024-08-19T14:12:15.558467+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:12:04.853650+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:12:04.853650+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:13:34.151342+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187468,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32696,"kb_used_data":1516,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712456,"statfs":{"total":95995035648,"available":95961554944,"internally_reserved":0,"allocated":1552384,"data_stored":1012292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":80,"seq":343597383696,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":32176,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1019904,"data_stored":483364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":74,"seq":317827579925,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":28016,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1019904,"data_stored":483364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874267,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":32176,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1019904,"data_stored":483364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":57,"seq":244813135901,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32696,"kb_used_data":1516,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712456,"statfs":{"total":95995035648,"available":95961554944,"internally_reserved":0,"allocated":1552384,"data_stored":1012292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430241,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28536,"kb_used_data":1516,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716616,"statfs":{"total":95995035648,"available":95965814784,"internally_reserved":0,"allocated":1552384,"data_stored":1012292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691878,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":28016,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1019904,"data_stored":483364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986218,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":28016,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1019904,"data_stored":483364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149744,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":28016,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1019904,"data_stored":483364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411381,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":28016,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1019904,"data_stored":483364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574905,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":32176,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1019904,"data_stored":483364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738430,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":32176,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1019904,"data_stored":483364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T14:12:24.167 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-19T14:12:24.420 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:12:24.800 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:24 smithi031 ceph-mon[16336]: from='client.14661 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:12:24.800 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:24 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:12:24.801 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:24 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:12:25.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:24 smithi032 ceph-mon[20004]: from='client.14661 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:12:25.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:24 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:12:25.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:24 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:12:25.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:24 smithi045 ceph-mon[20143]: from='client.14661 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:12:25.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:24 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:12:25.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:24 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:12:25.062 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:12:25.062 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-08-19T14:12:25.811 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":271,"stamp":"2024-08-19T14:12:24.649566+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":3,"kb":1124941824,"kb_used":362712,"kb_used_data":13512,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124579112,"statfs":{"total":1151940427776,"available":1151569010688,"internally_reserved":0,"allocated":13836288,"data_stored":7387152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":357414202},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.022057"},"pg_stats":[{"pgid":"1.0","version":"70'139","reported_seq":347,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-08-19T14:12:15.558467+0000","last_change":"2024-08-19T14:12:04.853784+0000","last_active":"2024-08-19T14:12:15.558467+0000","last_peered":"2024-08-19T14:12:15.558467+0000","last_clean":"2024-08-19T14:12:15.558467+0000","last_became_active":"2024-08-19T14:11:43.524419+0000","last_became_peered":"2024-08-19T14:11:43.524419+0000","last_unstale":"2024-08-19T14:12:15.558467+0000","last_undegraded":"2024-08-19T14:12:15.558467+0000","last_fullsized":"2024-08-19T14:12:15.558467+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:12:04.853650+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:12:04.853650+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:13:34.151342+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187468,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32696,"kb_used_data":1516,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712456,"statfs":{"total":95995035648,"available":95961554944,"internally_reserved":0,"allocated":1552384,"data_stored":1012292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":80,"seq":343597383697,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":32176,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1019904,"data_stored":483364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":74,"seq":317827579926,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":28016,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1019904,"data_stored":483364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874268,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":32176,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1019904,"data_stored":483364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":57,"seq":244813135902,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32696,"kb_used_data":1516,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712456,"statfs":{"total":95995035648,"available":95961554944,"internally_reserved":0,"allocated":1552384,"data_stored":1012292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430242,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28536,"kb_used_data":1516,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716616,"statfs":{"total":95995035648,"available":95965814784,"internally_reserved":0,"allocated":1552384,"data_stored":1012292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691878,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":28016,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1019904,"data_stored":483364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986219,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":28016,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1019904,"data_stored":483364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149745,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":28016,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1019904,"data_stored":483364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411381,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":28016,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1019904,"data_stored":483364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574906,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":32176,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1019904,"data_stored":483364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738430,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":32176,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1019904,"data_stored":483364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T14:12:25.812 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-08-19T14:12:25.812 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-08-19T14:12:25.812 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-08-19T14:12:25.812 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph health --format=json 2024-08-19T14:12:25.947 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:25 smithi045 ceph-mon[20143]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:12:25.948 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:25 smithi045 ceph-mon[20143]: osdmap e96: 12 total, 12 up, 12 in 2024-08-19T14:12:26.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:25 smithi032 ceph-mon[20004]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:12:26.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:25 smithi032 ceph-mon[20004]: osdmap e96: 12 total, 12 up, 12 in 2024-08-19T14:12:26.069 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:12:26.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:25 smithi031 ceph-mon[16336]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:12:26.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:25 smithi031 ceph-mon[16336]: osdmap e96: 12 total, 12 up, 12 in 2024-08-19T14:12:26.759 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:12:26.759 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-08-19T14:12:26.759 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:26 smithi031 ceph-mon[16336]: from='client.14667 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:12:27.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:26 smithi032 ceph-mon[20004]: from='client.14667 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:12:27.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:26 smithi045 ceph-mon[20143]: from='client.14667 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:12:27.422 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-08-19T14:12:27.423 INFO:tasks.cephadm:Setup complete, yielding 2024-08-19T14:12:27.423 INFO:teuthology.run_tasks:Running task exec... 2024-08-19T14:12:27.433 INFO:teuthology.task.exec:Executing custom commands... 2024-08-19T14:12:27.433 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi031.front.sepia.ceph.com 2024-08-19T14:12:27.433 DEBUG:teuthology.orchestra.run.smithi031:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'while ! ceph balancer status ; do sleep 1 ; done' 2024-08-19T14:12:27.718 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:12:27.718 INFO:teuthology.orchestra.run.smithi031.stdout: "active": true, 2024-08-19T14:12:27.718 INFO:teuthology.orchestra.run.smithi031.stdout: "last_optimize_duration": "0:00:00.000285", 2024-08-19T14:12:27.718 INFO:teuthology.orchestra.run.smithi031.stdout: "last_optimize_started": "Mon Aug 19 14:12:13 2024", 2024-08-19T14:12:27.719 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": "upmap", 2024-08-19T14:12:27.719 INFO:teuthology.orchestra.run.smithi031.stdout: "no_optimization_needed": true, 2024-08-19T14:12:27.719 INFO:teuthology.orchestra.run.smithi031.stdout: "optimize_result": "Unable to find further optimization, or pool(s) pg_num is decreasing, or distribution is already perfect", 2024-08-19T14:12:27.719 INFO:teuthology.orchestra.run.smithi031.stdout: "plans": [] 2024-08-19T14:12:27.719 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:12:27.730 DEBUG:teuthology.orchestra.run.smithi031:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph balancer mode crush-compat' 2024-08-19T14:12:28.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:27 smithi032 ceph-mon[20004]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:12:28.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:27 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/966546072' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-08-19T14:12:28.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:27 smithi045 ceph-mon[20143]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:12:28.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:27 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/966546072' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-08-19T14:12:28.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:27 smithi031 ceph-mon[16336]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:12:28.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:27 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/966546072' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-08-19T14:12:28.686 DEBUG:teuthology.orchestra.run.smithi031:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph balancer on' 2024-08-19T14:12:29.003 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:28 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set create-compat", "format": "json"}]: dispatch 2024-08-19T14:12:29.005 INFO:teuthology.run_tasks:Running task thrashosds... 2024-08-19T14:12:29.015 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 3, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-08-19T14:12:29.015 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'squid'}, 'ceph': {'conf': {'client.2': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 2500, 'ms tcp read timeout': 5, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'filestore queue throttle high multiple': 2, 'filestore queue throttle max multiple': 10, 'journal throttle high multiple': 2, 'journal throttle max multiple': 10, 'osd backoff on peering': True, 'osd blocked scrub grace period': 3600, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd max backfills': 9, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(MON_DOWN\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub'], 'sha1': 'de09dd018de4baddfa6ca4d43bc8fb784cf75833'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'de09dd018de4baddfa6ca4d43bc8fb784cf75833'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'min_in': 4}, 'workunit': {'branch': 'squid', 'sha1': '78215f5a1cf5eaf321d506a66539942e12851f34'}} 2024-08-19T14:12:29.015 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 3, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0, 'min_in': 4} 2024-08-19T14:12:29.015 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-08-19T14:12:29.016 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:28 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set create-compat", "format": "json"}]: dispatch 2024-08-19T14:12:29.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:28 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set create-compat", "format": "json"}]: dispatch 2024-08-19T14:12:29.271 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:12:30.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:29 smithi032 ceph-mon[20004]: from='client.14679 -' entity='client.admin' cmd=[{"prefix": "balancer status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T14:12:30.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:29 smithi032 ceph-mon[20004]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:12:30.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:29 smithi032 ceph-mon[20004]: from='client.14685 -' entity='client.admin' cmd=[{"prefix": "balancer mode", "mode": "crush-compat", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T14:12:30.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:29 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set create-compat", "format": "json"}]': finished 2024-08-19T14:12:30.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:29 smithi032 ceph-mon[20004]: osdmap e97: 12 total, 12 up, 12 in 2024-08-19T14:12:30.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:29 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd crush dump", "format": "json"}]: dispatch 2024-08-19T14:12:30.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:29 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:12:30.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:29 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:12:30.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:29 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:12:30.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:29 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:12:30.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:29 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:12:30.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:29 smithi045 ceph-mon[20143]: from='client.14679 -' entity='client.admin' cmd=[{"prefix": "balancer status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T14:12:30.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:29 smithi045 ceph-mon[20143]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:12:30.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:29 smithi045 ceph-mon[20143]: from='client.14685 -' entity='client.admin' cmd=[{"prefix": "balancer mode", "mode": "crush-compat", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T14:12:30.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:29 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set create-compat", "format": "json"}]': finished 2024-08-19T14:12:30.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:29 smithi045 ceph-mon[20143]: osdmap e97: 12 total, 12 up, 12 in 2024-08-19T14:12:30.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:29 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd crush dump", "format": "json"}]: dispatch 2024-08-19T14:12:30.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:29 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:12:30.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:29 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:12:30.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:29 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:12:30.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:29 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:12:30.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:29 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:12:30.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:29 smithi031 ceph-mon[16336]: from='client.14679 -' entity='client.admin' cmd=[{"prefix": "balancer status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T14:12:30.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:29 smithi031 ceph-mon[16336]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:12:30.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:29 smithi031 ceph-mon[16336]: from='client.14685 -' entity='client.admin' cmd=[{"prefix": "balancer mode", "mode": "crush-compat", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T14:12:30.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:29 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set create-compat", "format": "json"}]': finished 2024-08-19T14:12:30.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:29 smithi031 ceph-mon[16336]: osdmap e97: 12 total, 12 up, 12 in 2024-08-19T14:12:30.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:29 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd crush dump", "format": "json"}]: dispatch 2024-08-19T14:12:30.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:29 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:12:30.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:29 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:12:30.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:29 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:12:30.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:29 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:12:30.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:29 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:12:30.704 INFO:tasks.thrashosds:Beginning thrashosds... 2024-08-19T14:12:30.704 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-08-19T14:12:30.704 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-19T14:12:30.958 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:12:31.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:30 smithi032 ceph-mon[20004]: from='client.14691 -' entity='client.admin' cmd=[{"prefix": "balancer on", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T14:12:31.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:30 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:12:31.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:30 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:12:31.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:30 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1012915416' entity='client.admin' 2024-08-19T14:12:31.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:30 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:12:31.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:30 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:12:31.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:30 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:12:31.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:30 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:12:31.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:30 smithi032 ceph-mon[20004]: osdmap e98: 12 total, 12 up, 12 in 2024-08-19T14:12:31.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:30 smithi045 ceph-mon[20143]: from='client.14691 -' entity='client.admin' cmd=[{"prefix": "balancer on", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T14:12:31.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:30 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:12:31.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:30 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:12:31.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:30 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1012915416' entity='client.admin' 2024-08-19T14:12:31.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:30 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:12:31.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:30 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:12:31.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:30 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:12:31.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:30 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:12:31.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:30 smithi045 ceph-mon[20143]: osdmap e98: 12 total, 12 up, 12 in 2024-08-19T14:12:31.078 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:30 smithi031 ceph-mon[16336]: from='client.14691 -' entity='client.admin' cmd=[{"prefix": "balancer on", "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T14:12:31.079 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:30 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:12:31.079 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:30 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:12:31.079 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:30 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1012915416' entity='client.admin' 2024-08-19T14:12:31.079 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:30 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:12:31.079 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:30 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:12:31.079 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:30 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:12:31.079 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:30 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:12:31.079 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:30 smithi031 ceph-mon[16336]: osdmap e98: 12 total, 12 up, 12 in 2024-08-19T14:12:31.580 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:12:31.581 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-08-19T14:12:32.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:31 smithi032 ceph-mon[20004]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:12:32.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:31 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:12:32.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:31 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:12:32.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:31 smithi045 ceph-mon[20143]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:12:32.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:31 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:12:32.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:31 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:12:32.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:31 smithi031 ceph-mon[16336]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:12:32.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:31 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:12:32.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:31 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:12:32.362 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":276,"stamp":"2024-08-19T14:12:30.138579+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":3,"kb":1124941824,"kb_used":362856,"kb_used_data":13656,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124578968,"statfs":{"total":1151940427776,"available":1151568863232,"internally_reserved":0,"allocated":13983744,"data_stored":7477352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":357414202},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.409175"},"pg_stats":[{"pgid":"1.0","version":"70'139","reported_seq":349,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-08-19T14:12:24.701128+0000","last_change":"2024-08-19T14:12:04.853784+0000","last_active":"2024-08-19T14:12:24.701128+0000","last_peered":"2024-08-19T14:12:24.701128+0000","last_clean":"2024-08-19T14:12:24.701128+0000","last_became_active":"2024-08-19T14:11:43.524419+0000","last_became_peered":"2024-08-19T14:11:43.524419+0000","last_unstale":"2024-08-19T14:12:24.701128+0000","last_undegraded":"2024-08-19T14:12:24.701128+0000","last_fullsized":"2024-08-19T14:12:24.701128+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:12:04.853650+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:12:04.853650+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:13:34.151342+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187469,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32708,"kb_used_data":1528,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712444,"statfs":{"total":95995035648,"available":95961542656,"internally_reserved":0,"allocated":1564672,"data_stored":1019669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":80,"seq":343597383698,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":32188,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712964,"statfs":{"total":95995035648,"available":95962075136,"internally_reserved":0,"allocated":1032192,"data_stored":490741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":74,"seq":317827579927,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":28028,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1032192,"data_stored":490741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874269,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":32200,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712952,"statfs":{"total":95995035648,"available":95962062848,"internally_reserved":0,"allocated":1044480,"data_stored":499794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":57,"seq":244813135903,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32708,"kb_used_data":1528,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712444,"statfs":{"total":95995035648,"available":95961542656,"internally_reserved":0,"allocated":1564672,"data_stored":1019669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430243,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28548,"kb_used_data":1528,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716604,"statfs":{"total":95995035648,"available":95965802496,"internally_reserved":0,"allocated":1564672,"data_stored":1019669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691879,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":28028,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1032192,"data_stored":490741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986220,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":28028,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1032192,"data_stored":490741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149746,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":28028,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1032192,"data_stored":490741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411382,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":28016,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1019904,"data_stored":483364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574907,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":32188,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712964,"statfs":{"total":95995035648,"available":95962075136,"internally_reserved":0,"allocated":1032192,"data_stored":490741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738431,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":32188,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712964,"statfs":{"total":95995035648,"available":95962075136,"internally_reserved":0,"allocated":1032192,"data_stored":490741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T14:12:32.363 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-19T14:12:32.614 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:12:32.952 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:32 smithi031 ceph-mon[16336]: from='client.14703 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:12:32.952 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:32 smithi031 ceph-mon[16336]: osdmap e99: 12 total, 12 up, 12 in 2024-08-19T14:12:33.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:32 smithi032 ceph-mon[20004]: from='client.14703 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:12:33.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:32 smithi032 ceph-mon[20004]: osdmap e99: 12 total, 12 up, 12 in 2024-08-19T14:12:33.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:32 smithi045 ceph-mon[20143]: from='client.14703 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:12:33.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:32 smithi045 ceph-mon[20143]: osdmap e99: 12 total, 12 up, 12 in 2024-08-19T14:12:33.215 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:12:33.215 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-08-19T14:12:33.864 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:33 smithi031 ceph-mon[16336]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:12:33.895 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":278,"stamp":"2024-08-19T14:12:31.730914+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":3,"kb":1124941824,"kb_used":362892,"kb_used_data":13692,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124578932,"statfs":{"total":1151940427776,"available":1151568826368,"internally_reserved":0,"allocated":14020608,"data_stored":7501419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":357414202},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001081"},"pg_stats":[{"pgid":"1.0","version":"70'139","reported_seq":349,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-08-19T14:12:24.701128+0000","last_change":"2024-08-19T14:12:04.853784+0000","last_active":"2024-08-19T14:12:24.701128+0000","last_peered":"2024-08-19T14:12:24.701128+0000","last_clean":"2024-08-19T14:12:24.701128+0000","last_became_active":"2024-08-19T14:11:43.524419+0000","last_became_peered":"2024-08-19T14:11:43.524419+0000","last_unstale":"2024-08-19T14:12:24.701128+0000","last_undegraded":"2024-08-19T14:12:24.701128+0000","last_fullsized":"2024-08-19T14:12:24.701128+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:12:04.853650+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:12:04.853650+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:13:34.151342+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187470,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32708,"kb_used_data":1528,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712444,"statfs":{"total":95995035648,"available":95961542656,"internally_reserved":0,"allocated":1564672,"data_stored":1019669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":80,"seq":343597383698,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":32188,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712964,"statfs":{"total":95995035648,"available":95962075136,"internally_reserved":0,"allocated":1032192,"data_stored":490741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":74,"seq":317827579927,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":28028,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1032192,"data_stored":490741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874269,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":32200,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712952,"statfs":{"total":95995035648,"available":95962062848,"internally_reserved":0,"allocated":1044480,"data_stored":499794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":57,"seq":244813135903,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32708,"kb_used_data":1528,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712444,"statfs":{"total":95995035648,"available":95961542656,"internally_reserved":0,"allocated":1564672,"data_stored":1019669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430243,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28548,"kb_used_data":1528,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716604,"statfs":{"total":95995035648,"available":95965802496,"internally_reserved":0,"allocated":1564672,"data_stored":1019669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691880,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":28052,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1056768,"data_stored":507431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986220,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":28028,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1032192,"data_stored":490741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149746,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":28028,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1032192,"data_stored":490741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411383,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":28028,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1032192,"data_stored":490741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574907,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":32188,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712964,"statfs":{"total":95995035648,"available":95962075136,"internally_reserved":0,"allocated":1032192,"data_stored":490741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738432,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":32188,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712964,"statfs":{"total":95995035648,"available":95962075136,"internally_reserved":0,"allocated":1032192,"data_stored":490741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T14:12:33.896 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-08-19T14:12:33.896 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd dump 2024-08-19T14:12:34.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:33 smithi032 ceph-mon[20004]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:12:34.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:33 smithi045 ceph-mon[20143]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:12:34.152 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:12:34.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-08-19T14:12:34.794+0000 7f8fad8c2640 0 --2- 172.21.15.31:0/4271743635 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8fa810bea0 0x7f8fa810e340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-08-19T14:12:34.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:34 smithi031 ceph-mon[16336]: from='client.14709 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:12:34.982 INFO:teuthology.orchestra.run.smithi031.stdout:epoch 100 2024-08-19T14:12:34.982 INFO:teuthology.orchestra.run.smithi031.stdout:fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 2024-08-19T14:12:34.983 INFO:teuthology.orchestra.run.smithi031.stdout:created 2024-08-19T14:05:22.842169+0000 2024-08-19T14:12:34.983 INFO:teuthology.orchestra.run.smithi031.stdout:modified 2024-08-19T14:12:34.677284+0000 2024-08-19T14:12:34.983 INFO:teuthology.orchestra.run.smithi031.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-08-19T14:12:34.983 INFO:teuthology.orchestra.run.smithi031.stdout:crush_version 27 2024-08-19T14:12:34.983 INFO:teuthology.orchestra.run.smithi031.stdout:full_ratio 0.95 2024-08-19T14:12:34.983 INFO:teuthology.orchestra.run.smithi031.stdout:backfillfull_ratio 0.9 2024-08-19T14:12:34.983 INFO:teuthology.orchestra.run.smithi031.stdout:nearfull_ratio 0.85 2024-08-19T14:12:34.983 INFO:teuthology.orchestra.run.smithi031.stdout:require_min_compat_client luminous 2024-08-19T14:12:34.983 INFO:teuthology.orchestra.run.smithi031.stdout:min_compat_client jewel 2024-08-19T14:12:34.983 INFO:teuthology.orchestra.run.smithi031.stdout:require_osd_release squid 2024-08-19T14:12:34.983 INFO:teuthology.orchestra.run.smithi031.stdout:stretch_mode_enabled false 2024-08-19T14:12:34.983 INFO:teuthology.orchestra.run.smithi031.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 22 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-08-19T14:12:34.984 INFO:teuthology.orchestra.run.smithi031.stdout:max_osd 12 2024-08-19T14:12:34.984 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0 up in weight 1 up_from 8 up_thru 59 down_at 0 last_clean_interval [0,0) [v2:172.21.15.31:6802/1108945097,v1:172.21.15.31:6803/1108945097] [v2:172.21.15.31:6804/1108945097,v1:172.21.15.31:6805/1108945097] exists,up d7a0fff3-23c6-4ee3-a275-a844862aabfb 2024-08-19T14:12:34.984 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1 up in weight 1 up_from 13 up_thru 34 down_at 0 last_clean_interval [0,0) [v2:172.21.15.31:6810/3236186806,v1:172.21.15.31:6811/3236186806] [v2:172.21.15.31:6812/3236186806,v1:172.21.15.31:6813/3236186806] exists,up 145d5bb4-74dd-440f-bcc9-7e9ebda9c5d6 2024-08-19T14:12:34.984 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.31:6818/4045553387,v1:172.21.15.31:6819/4045553387] [v2:172.21.15.31:6820/4045553387,v1:172.21.15.31:6821/4045553387] exists,up 376c18d0-4768-46a0-bb60-e79c88ca909c 2024-08-19T14:12:34.984 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3 up in weight 1 up_from 26 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.31:6826/2288082716,v1:172.21.15.31:6827/2288082716] [v2:172.21.15.31:6828/2288082716,v1:172.21.15.31:6829/2288082716] exists,up a787ea9f-b9ba-450f-b388-614cfba90634 2024-08-19T14:12:34.984 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4 up in weight 1 up_from 31 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.32:6800/3934739984,v1:172.21.15.32:6801/3934739984] [v2:172.21.15.32:6802/3934739984,v1:172.21.15.32:6803/3934739984] exists,up 7859751a-12f5-4a57-a332-435360479d53 2024-08-19T14:12:34.984 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5 up in weight 1 up_from 40 up_thru 43 down_at 0 last_clean_interval [0,0) [v2:172.21.15.32:6808/526388985,v1:172.21.15.32:6809/526388985] [v2:172.21.15.32:6810/526388985,v1:172.21.15.32:6811/526388985] exists,up 017f3eef-ac60-4b82-8350-919017cb4ccd 2024-08-19T14:12:34.984 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6 up in weight 1 up_from 48 up_thru 51 down_at 0 last_clean_interval [0,0) [v2:172.21.15.32:6816/1131035844,v1:172.21.15.32:6817/1131035844] [v2:172.21.15.32:6818/1131035844,v1:172.21.15.32:6819/1131035844] exists,up 44aa8229-0268-49d7-bd57-3bf1f384e5e6 2024-08-19T14:12:34.984 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7 up in weight 1 up_from 57 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.32:6824/2014939947,v1:172.21.15.32:6825/2014939947] [v2:172.21.15.32:6826/2014939947,v1:172.21.15.32:6827/2014939947] exists,up 263433e5-70b6-4192-86ff-dc2ad95927eb 2024-08-19T14:12:34.984 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8 up in weight 1 up_from 65 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.45:6800/1445140892,v1:172.21.15.45:6801/1445140892] [v2:172.21.15.45:6802/1445140892,v1:172.21.15.45:6803/1445140892] exists,up 2bbcad36-0c74-4528-9fb9-a8812afa1662 2024-08-19T14:12:34.984 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9 up in weight 1 up_from 74 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.45:6808/1213787169,v1:172.21.15.45:6809/1213787169] [v2:172.21.15.45:6810/1213787169,v1:172.21.15.45:6811/1213787169] exists,up c8ee4277-3bf9-46bf-9cc0-152c1c4b9508 2024-08-19T14:12:34.984 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10 up in weight 1 up_from 80 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.45:6816/2461824621,v1:172.21.15.45:6817/2461824621] [v2:172.21.15.45:6818/2461824621,v1:172.21.15.45:6819/2461824621] exists,up 29031395-5f5c-40f8-ac90-9f6ac48e5c9f 2024-08-19T14:12:34.985 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11 up in weight 1 up_from 86 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.45:6824/2438396298,v1:172.21.15.45:6825/2438396298] [v2:172.21.15.45:6826/2438396298,v1:172.21.15.45:6827/2438396298] exists,up 5ef0db4b-0fef-4ea0-a19b-d3677e082c29 2024-08-19T14:12:34.985 INFO:teuthology.orchestra.run.smithi031.stdout:blocklist 172.21.15.31:0/1451745969 expires 2024-08-20T14:06:13.660657+0000 2024-08-19T14:12:34.985 INFO:teuthology.orchestra.run.smithi031.stdout:blocklist 172.21.15.31:0/1515698918 expires 2024-08-20T14:06:13.660657+0000 2024-08-19T14:12:34.985 INFO:teuthology.orchestra.run.smithi031.stdout:blocklist 172.21.15.31:6801/3297539242 expires 2024-08-20T14:06:13.660657+0000 2024-08-19T14:12:34.985 INFO:teuthology.orchestra.run.smithi031.stdout:blocklist 172.21.15.31:6800/3297539242 expires 2024-08-20T14:06:13.660657+0000 2024-08-19T14:12:34.985 INFO:teuthology.orchestra.run.smithi031.stdout:blocklist 172.21.15.31:0/2217423818 expires 2024-08-20T14:05:48.827576+0000 2024-08-19T14:12:34.985 INFO:teuthology.orchestra.run.smithi031.stdout:blocklist 172.21.15.31:0/3353498281 expires 2024-08-20T14:06:13.660657+0000 2024-08-19T14:12:34.985 INFO:teuthology.orchestra.run.smithi031.stdout:blocklist 172.21.15.31:0/4013594220 expires 2024-08-20T14:05:48.827576+0000 2024-08-19T14:12:34.985 INFO:teuthology.orchestra.run.smithi031.stdout:blocklist 172.21.15.31:0/1654518065 expires 2024-08-20T14:05:48.827576+0000 2024-08-19T14:12:34.985 INFO:teuthology.orchestra.run.smithi031.stdout:blocklist 172.21.15.31:6801/2368821258 expires 2024-08-20T14:05:48.827576+0000 2024-08-19T14:12:34.986 INFO:teuthology.orchestra.run.smithi031.stdout:blocklist 172.21.15.31:6800/2368821258 expires 2024-08-20T14:05:48.827576+0000 2024-08-19T14:12:35.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:34 smithi032 ceph-mon[20004]: from='client.14709 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:12:35.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:34 smithi045 ceph-mon[20143]: from='client.14709 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:12:35.592 INFO:tasks.cephadm.ceph_manager.ceph:['osd.0 up in weight 1 up_from 8 up_thru 59 down_at 0 last_clean_interval [0,0) [v2:172.21.15.31:6802/1108945097,v1:172.21.15.31:6803/1108945097] [v2:172.21.15.31:6804/1108945097,v1:172.21.15.31:6805/1108945097] exists,up d7a0fff3-23c6-4ee3-a275-a844862aabfb', 'osd.1 up in weight 1 up_from 13 up_thru 34 down_at 0 last_clean_interval [0,0) [v2:172.21.15.31:6810/3236186806,v1:172.21.15.31:6811/3236186806] [v2:172.21.15.31:6812/3236186806,v1:172.21.15.31:6813/3236186806] exists,up 145d5bb4-74dd-440f-bcc9-7e9ebda9c5d6', 'osd.2 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.31:6818/4045553387,v1:172.21.15.31:6819/4045553387] [v2:172.21.15.31:6820/4045553387,v1:172.21.15.31:6821/4045553387] exists,up 376c18d0-4768-46a0-bb60-e79c88ca909c', 'osd.3 up in weight 1 up_from 26 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.31:6826/2288082716,v1:172.21.15.31:6827/2288082716] [v2:172.21.15.31:6828/2288082716,v1:172.21.15.31:6829/2288082716] exists,up a787ea9f-b9ba-450f-b388-614cfba90634', 'osd.4 up in weight 1 up_from 31 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.32:6800/3934739984,v1:172.21.15.32:6801/3934739984] [v2:172.21.15.32:6802/3934739984,v1:172.21.15.32:6803/3934739984] exists,up 7859751a-12f5-4a57-a332-435360479d53', 'osd.5 up in weight 1 up_from 40 up_thru 43 down_at 0 last_clean_interval [0,0) [v2:172.21.15.32:6808/526388985,v1:172.21.15.32:6809/526388985] [v2:172.21.15.32:6810/526388985,v1:172.21.15.32:6811/526388985] exists,up 017f3eef-ac60-4b82-8350-919017cb4ccd', 'osd.6 up in weight 1 up_from 48 up_thru 51 down_at 0 last_clean_interval [0,0) [v2:172.21.15.32:6816/1131035844,v1:172.21.15.32:6817/1131035844] [v2:172.21.15.32:6818/1131035844,v1:172.21.15.32:6819/1131035844] exists,up 44aa8229-0268-49d7-bd57-3bf1f384e5e6', 'osd.7 up in weight 1 up_from 57 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.32:6824/2014939947,v1:172.21.15.32:6825/2014939947] [v2:172.21.15.32:6826/2014939947,v1:172.21.15.32:6827/2014939947] exists,up 263433e5-70b6-4192-86ff-dc2ad95927eb', 'osd.8 up in weight 1 up_from 65 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.45:6800/1445140892,v1:172.21.15.45:6801/1445140892] [v2:172.21.15.45:6802/1445140892,v1:172.21.15.45:6803/1445140892] exists,up 2bbcad36-0c74-4528-9fb9-a8812afa1662', 'osd.9 up in weight 1 up_from 74 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.45:6808/1213787169,v1:172.21.15.45:6809/1213787169] [v2:172.21.15.45:6810/1213787169,v1:172.21.15.45:6811/1213787169] exists,up c8ee4277-3bf9-46bf-9cc0-152c1c4b9508', 'osd.10 up in weight 1 up_from 80 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.45:6816/2461824621,v1:172.21.15.45:6817/2461824621] [v2:172.21.15.45:6818/2461824621,v1:172.21.15.45:6819/2461824621] exists,up 29031395-5f5c-40f8-ac90-9f6ac48e5c9f', 'osd.11 up in weight 1 up_from 86 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.45:6824/2438396298,v1:172.21.15.45:6825/2438396298] [v2:172.21.15.45:6826/2438396298,v1:172.21.15.45:6827/2438396298] exists,up 5ef0db4b-0fef-4ea0-a19b-d3677e082c29'] 2024-08-19T14:12:35.593 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon mon.a config show 2024-08-19T14:12:35.844 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:12:35.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:35 smithi031 ceph-mon[16336]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:12:35.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:35 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:12:35.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:35 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:12:35.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:35 smithi031 ceph-mon[16336]: osdmap e100: 12 total, 12 up, 12 in 2024-08-19T14:12:35.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:35 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/4206406989' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-08-19T14:12:36.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:35 smithi032 ceph-mon[20004]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:12:36.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:35 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:12:36.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:35 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:12:36.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:35 smithi032 ceph-mon[20004]: osdmap e100: 12 total, 12 up, 12 in 2024-08-19T14:12:36.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:35 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/4206406989' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-08-19T14:12:36.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:35 smithi045 ceph-mon[20143]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:12:36.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:35 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:12:36.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:35 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:12:36.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:35 smithi045 ceph-mon[20143]: osdmap e100: 12 total, 12 up, 12 in 2024-08-19T14:12:36.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:35 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/4206406989' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-08-19T14:12:36.301 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:12:36.302 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mon.a", 2024-08-19T14:12:36.302 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster": "ceph", 2024-08-19T14:12:36.302 INFO:teuthology.orchestra.run.smithi031.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-08-19T14:12:36.302 INFO:teuthology.orchestra.run.smithi031.stdout: "admin_socket_mode": "", 2024-08-19T14:12:36.302 INFO:teuthology.orchestra.run.smithi031.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2024-08-19T14:12:36.302 INFO:teuthology.orchestra.run.smithi031.stdout: "auth_client_required": "cephx, none", 2024-08-19T14:12:36.302 INFO:teuthology.orchestra.run.smithi031.stdout: "auth_cluster_required": "cephx", 2024-08-19T14:12:36.302 INFO:teuthology.orchestra.run.smithi031.stdout: "auth_debug": "true", 2024-08-19T14:12:36.302 INFO:teuthology.orchestra.run.smithi031.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-08-19T14:12:36.302 INFO:teuthology.orchestra.run.smithi031.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-08-19T14:12:36.303 INFO:teuthology.orchestra.run.smithi031.stdout: "auth_service_required": "cephx", 2024-08-19T14:12:36.303 INFO:teuthology.orchestra.run.smithi031.stdout: "auth_service_ticket_ttl": "240.000000", 2024-08-19T14:12:36.303 INFO:teuthology.orchestra.run.smithi031.stdout: "auth_supported": "", 2024-08-19T14:12:36.303 INFO:teuthology.orchestra.run.smithi031.stdout: "bdev_aio": "true", 2024-08-19T14:12:36.303 INFO:teuthology.orchestra.run.smithi031.stdout: "bdev_aio_max_queue_depth": "1024", 2024-08-19T14:12:36.303 INFO:teuthology.orchestra.run.smithi031.stdout: "bdev_aio_poll_ms": "250", 2024-08-19T14:12:36.303 INFO:teuthology.orchestra.run.smithi031.stdout: "bdev_aio_reap_max": "16", 2024-08-19T14:12:36.303 INFO:teuthology.orchestra.run.smithi031.stdout: "bdev_async_discard": "false", 2024-08-19T14:12:36.303 INFO:teuthology.orchestra.run.smithi031.stdout: "bdev_block_size": "4096", 2024-08-19T14:12:36.303 INFO:teuthology.orchestra.run.smithi031.stdout: "bdev_debug_aio": "false", 2024-08-19T14:12:36.303 INFO:teuthology.orchestra.run.smithi031.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-08-19T14:12:36.303 INFO:teuthology.orchestra.run.smithi031.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-08-19T14:12:36.304 INFO:teuthology.orchestra.run.smithi031.stdout: "bdev_debug_inflight_ios": "false", 2024-08-19T14:12:36.304 INFO:teuthology.orchestra.run.smithi031.stdout: "bdev_enable_discard": "false", 2024-08-19T14:12:36.304 INFO:teuthology.orchestra.run.smithi031.stdout: "bdev_flock_retry": "3", 2024-08-19T14:12:36.304 INFO:teuthology.orchestra.run.smithi031.stdout: "bdev_flock_retry_interval": "0.100000", 2024-08-19T14:12:36.304 INFO:teuthology.orchestra.run.smithi031.stdout: "bdev_inject_crash": "0", 2024-08-19T14:12:36.304 INFO:teuthology.orchestra.run.smithi031.stdout: "bdev_inject_crash_flush_delay": "2", 2024-08-19T14:12:36.304 INFO:teuthology.orchestra.run.smithi031.stdout: "bdev_ioring": "false", 2024-08-19T14:12:36.304 INFO:teuthology.orchestra.run.smithi031.stdout: "bdev_ioring_hipri": "false", 2024-08-19T14:12:36.304 INFO:teuthology.orchestra.run.smithi031.stdout: "bdev_ioring_sqthread_poll": "false", 2024-08-19T14:12:36.304 INFO:teuthology.orchestra.run.smithi031.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-08-19T14:12:36.304 INFO:teuthology.orchestra.run.smithi031.stdout: "bdev_read_buffer_alignment": "4096", 2024-08-19T14:12:36.304 INFO:teuthology.orchestra.run.smithi031.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-08-19T14:12:36.305 INFO:teuthology.orchestra.run.smithi031.stdout: "bdev_type": "", 2024-08-19T14:12:36.305 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_alloc_size": "1048576", 2024-08-19T14:12:36.305 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_allocator": "hybrid", 2024-08-19T14:12:36.305 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_buffered_io": "true", 2024-08-19T14:12:36.305 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_check_for_zeros": "false", 2024-08-19T14:12:36.305 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_check_volume_selector_often": "false", 2024-08-19T14:12:36.305 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-08-19T14:12:36.305 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_compact_log_sync": "false", 2024-08-19T14:12:36.305 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-08-19T14:12:36.305 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-08-19T14:12:36.305 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_log_compact_min_size": "16777216", 2024-08-19T14:12:36.306 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_log_replay_check_allocations": "true", 2024-08-19T14:12:36.306 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_max_log_runway": "4194304", 2024-08-19T14:12:36.306 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_max_prefetch": "1048576", 2024-08-19T14:12:36.306 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_min_flush_size": "524288", 2024-08-19T14:12:36.306 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_min_log_runway": "1048576", 2024-08-19T14:12:36.306 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_replay_recovery": "false", 2024-08-19T14:12:36.306 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-08-19T14:12:36.306 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_shared_alloc_size": "65536", 2024-08-19T14:12:36.306 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_sync_write": "false", 2024-08-19T14:12:36.306 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-08-19T14:12:36.306 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-08-19T14:12:36.306 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-08-19T14:12:36.307 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_allocation_from_file": "true", 2024-08-19T14:12:36.307 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_allocator": "hybrid", 2024-08-19T14:12:36.307 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-08-19T14:12:36.307 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-08-19T14:12:36.307 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-08-19T14:12:36.307 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-08-19T14:12:36.307 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-08-19T14:12:36.307 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-08-19T14:12:36.307 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_blobid_prealloc": "10240", 2024-08-19T14:12:36.307 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_block_create": "true", 2024-08-19T14:12:36.307 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_block_db_create": "false", 2024-08-19T14:12:36.307 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_block_db_path": "", 2024-08-19T14:12:36.308 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_block_db_size": "0", 2024-08-19T14:12:36.308 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_block_path": "", 2024-08-19T14:12:36.308 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_block_preallocate_file": "false", 2024-08-19T14:12:36.308 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_block_size": "107374182400", 2024-08-19T14:12:36.308 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_block_wal_create": "false", 2024-08-19T14:12:36.308 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_block_wal_path": "", 2024-08-19T14:12:36.308 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_block_wal_size": "100663296", 2024-08-19T14:12:36.308 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_bluefs": "true", 2024-08-19T14:12:36.308 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-08-19T14:12:36.308 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_bluefs_env_mirror": "false", 2024-08-19T14:12:36.308 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-08-19T14:12:36.308 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-08-19T14:12:36.308 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-08-19T14:12:36.309 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-08-19T14:12:36.309 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-08-19T14:12:36.309 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-08-19T14:12:36.309 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_autotune": "true", 2024-08-19T14:12:36.309 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-08-19T14:12:36.309 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-08-19T14:12:36.309 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-08-19T14:12:36.309 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-08-19T14:12:36.309 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_size": "0", 2024-08-19T14:12:36.309 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-08-19T14:12:36.309 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-08-19T14:12:36.309 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-08-19T14:12:36.309 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-08-19T14:12:36.309 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_type": "2q", 2024-08-19T14:12:36.310 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-08-19T14:12:36.310 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_clone_cow": "true", 2024-08-19T14:12:36.310 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_compression_algorithm": "snappy", 2024-08-19T14:12:36.310 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_compression_max_blob_size": "0", 2024-08-19T14:12:36.310 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-08-19T14:12:36.310 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-08-19T14:12:36.310 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_compression_min_blob_size": "0", 2024-08-19T14:12:36.310 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-08-19T14:12:36.310 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-08-19T14:12:36.310 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_compression_mode": "none", 2024-08-19T14:12:36.310 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-08-19T14:12:36.310 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_csum_type": "crc32c", 2024-08-19T14:12:36.311 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_debug_enforce_settings": "default", 2024-08-19T14:12:36.311 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_debug_freelist": "false", 2024-08-19T14:12:36.311 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_debug_fsck_abort": "false", 2024-08-19T14:12:36.311 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-08-19T14:12:36.311 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-08-19T14:12:36.311 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_debug_inject_read_err": "false", 2024-08-19T14:12:36.311 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_debug_legacy_omap": "false", 2024-08-19T14:12:36.311 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-08-19T14:12:36.311 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-08-19T14:12:36.311 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-08-19T14:12:36.311 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-08-19T14:12:36.311 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_debug_prefill": "0.000000", 2024-08-19T14:12:36.311 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-08-19T14:12:36.312 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-08-19T14:12:36.312 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-08-19T14:12:36.312 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_debug_small_allocations": "0", 2024-08-19T14:12:36.312 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-08-19T14:12:36.312 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_default_buffered_read": "true", 2024-08-19T14:12:36.312 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_default_buffered_write": "false", 2024-08-19T14:12:36.312 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_deferred_batch_ops": "0", 2024-08-19T14:12:36.312 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-08-19T14:12:36.312 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-08-19T14:12:36.312 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_elastic_shared_blobs": "true", 2024-08-19T14:12:36.312 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-08-19T14:12:36.312 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-08-19T14:12:36.313 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-08-19T14:12:36.313 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-08-19T14:12:36.313 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-08-19T14:12:36.313 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_fail_eio": "false", 2024-08-19T14:12:36.313 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-08-19T14:12:36.313 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-08-19T14:12:36.313 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-08-19T14:12:36.313 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-08-19T14:12:36.313 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_fsck_on_mkfs": "true", 2024-08-19T14:12:36.313 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-08-19T14:12:36.313 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_fsck_on_mount": "false", 2024-08-19T14:12:36.313 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-08-19T14:12:36.313 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_fsck_on_umount": "false", 2024-08-19T14:12:36.313 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-08-19T14:12:36.313 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-08-19T14:12:36.314 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-08-19T14:12:36.314 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-08-19T14:12:36.314 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-08-19T14:12:36.314 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-08-19T14:12:36.314 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-08-19T14:12:36.314 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-08-19T14:12:36.314 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_ignore_data_csum": "false", 2024-08-19T14:12:36.314 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-08-19T14:12:36.314 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_kvbackend": "rocksdb", 2024-08-19T14:12:36.314 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-08-19T14:12:36.314 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-08-19T14:12:36.314 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_log_op_age": "5.000000", 2024-08-19T14:12:36.315 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_max_alloc_size": "0", 2024-08-19T14:12:36.315 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_max_blob_size": "0", 2024-08-19T14:12:36.315 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-08-19T14:12:36.315 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-08-19T14:12:36.315 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_max_defer_interval": "3.000000", 2024-08-19T14:12:36.315 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_max_deferred_txc": "32", 2024-08-19T14:12:36.315 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_min_alloc_size": "0", 2024-08-19T14:12:36.315 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-08-19T14:12:36.315 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-08-19T14:12:36.315 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_nid_prealloc": "1024", 2024-08-19T14:12:36.315 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_prefer_deferred_size": "0", 2024-08-19T14:12:36.315 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-08-19T14:12:36.315 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-08-19T14:12:36.315 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_qfsck_on_mount": "true", 2024-08-19T14:12:36.315 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_retry_disk_reads": "3", 2024-08-19T14:12:36.316 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_rocksdb_cf": "true", 2024-08-19T14:12:36.316 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-08-19T14:12:36.317 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-08-19T14:12:36.317 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_rocksdb_options_annex": "", 2024-08-19T14:12:36.317 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_spdk_coremask": "0x1", 2024-08-19T14:12:36.317 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_spdk_io_sleep": "5", 2024-08-19T14:12:36.317 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_spdk_max_io_completion": "0", 2024-08-19T14:12:36.317 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_spdk_mem": "512", 2024-08-19T14:12:36.317 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_sync_submit_transaction": "false", 2024-08-19T14:12:36.317 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_throttle_bytes": "67108864", 2024-08-19T14:12:36.317 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_throttle_cost_per_io": "0", 2024-08-19T14:12:36.317 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-08-19T14:12:36.318 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-08-19T14:12:36.318 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-08-19T14:12:36.318 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-08-19T14:12:36.318 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_tracing": "false", 2024-08-19T14:12:36.318 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-08-19T14:12:36.318 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-08-19T14:12:36.318 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_volume_selection_reserved": "0", 2024-08-19T14:12:36.318 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-08-19T14:12:36.318 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-08-19T14:12:36.318 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-08-19T14:12:36.318 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-08-19T14:12:36.318 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-08-19T14:12:36.318 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-08-19T14:12:36.319 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_zero_block_detection": "false", 2024-08-19T14:12:36.319 INFO:teuthology.orchestra.run.smithi031.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-08-19T14:12:36.319 INFO:teuthology.orchestra.run.smithi031.stdout: "cephfs_mirror_action_update_interval": "2", 2024-08-19T14:12:36.319 INFO:teuthology.orchestra.run.smithi031.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-08-19T14:12:36.319 INFO:teuthology.orchestra.run.smithi031.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-08-19T14:12:36.319 INFO:teuthology.orchestra.run.smithi031.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-08-19T14:12:36.319 INFO:teuthology.orchestra.run.smithi031.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-08-19T14:12:36.319 INFO:teuthology.orchestra.run.smithi031.stdout: "cephfs_mirror_mount_timeout": "10", 2024-08-19T14:12:36.319 INFO:teuthology.orchestra.run.smithi031.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-08-19T14:12:36.319 INFO:teuthology.orchestra.run.smithi031.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-08-19T14:12:36.319 INFO:teuthology.orchestra.run.smithi031.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-08-19T14:12:36.319 INFO:teuthology.orchestra.run.smithi031.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-08-19T14:12:36.319 INFO:teuthology.orchestra.run.smithi031.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-08-19T14:12:36.320 INFO:teuthology.orchestra.run.smithi031.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-08-19T14:12:36.320 INFO:teuthology.orchestra.run.smithi031.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-08-19T14:12:36.320 INFO:teuthology.orchestra.run.smithi031.stdout: "cephx_cluster_require_signatures": "false", 2024-08-19T14:12:36.320 INFO:teuthology.orchestra.run.smithi031.stdout: "cephx_cluster_require_version": "2", 2024-08-19T14:12:36.320 INFO:teuthology.orchestra.run.smithi031.stdout: "cephx_require_signatures": "false", 2024-08-19T14:12:36.320 INFO:teuthology.orchestra.run.smithi031.stdout: "cephx_require_version": "2", 2024-08-19T14:12:36.320 INFO:teuthology.orchestra.run.smithi031.stdout: "cephx_service_require_signatures": "false", 2024-08-19T14:12:36.320 INFO:teuthology.orchestra.run.smithi031.stdout: "cephx_service_require_version": "2", 2024-08-19T14:12:36.320 INFO:teuthology.orchestra.run.smithi031.stdout: "cephx_sign_messages": "true", 2024-08-19T14:12:36.320 INFO:teuthology.orchestra.run.smithi031.stdout: "chdir": "", 2024-08-19T14:12:36.320 INFO:teuthology.orchestra.run.smithi031.stdout: "client_acl_type": "", 2024-08-19T14:12:36.320 INFO:teuthology.orchestra.run.smithi031.stdout: "client_asio_thread_count": "2", 2024-08-19T14:12:36.320 INFO:teuthology.orchestra.run.smithi031.stdout: "client_cache_mid": "0.750000", 2024-08-19T14:12:36.321 INFO:teuthology.orchestra.run.smithi031.stdout: "client_cache_size": "16384", 2024-08-19T14:12:36.321 INFO:teuthology.orchestra.run.smithi031.stdout: "client_caps_release_delay": "5", 2024-08-19T14:12:36.321 INFO:teuthology.orchestra.run.smithi031.stdout: "client_check_pool_perm": "true", 2024-08-19T14:12:36.321 INFO:teuthology.orchestra.run.smithi031.stdout: "client_collect_and_send_global_metrics": "false", 2024-08-19T14:12:36.321 INFO:teuthology.orchestra.run.smithi031.stdout: "client_debug_force_sync_read": "false", 2024-08-19T14:12:36.321 INFO:teuthology.orchestra.run.smithi031.stdout: "client_debug_getattr_caps": "false", 2024-08-19T14:12:36.321 INFO:teuthology.orchestra.run.smithi031.stdout: "client_debug_inject_features": "", 2024-08-19T14:12:36.321 INFO:teuthology.orchestra.run.smithi031.stdout: "client_debug_inject_tick_delay": "0", 2024-08-19T14:12:36.321 INFO:teuthology.orchestra.run.smithi031.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-08-19T14:12:36.321 INFO:teuthology.orchestra.run.smithi031.stdout: "client_die_on_failed_remount": "false", 2024-08-19T14:12:36.321 INFO:teuthology.orchestra.run.smithi031.stdout: "client_dirsize_rbytes": "true", 2024-08-19T14:12:36.321 INFO:teuthology.orchestra.run.smithi031.stdout: "client_force_lazyio": "false", 2024-08-19T14:12:36.321 INFO:teuthology.orchestra.run.smithi031.stdout: "client_fs": "", 2024-08-19T14:12:36.321 INFO:teuthology.orchestra.run.smithi031.stdout: "client_inject_fixed_oldest_tid": "false", 2024-08-19T14:12:36.322 INFO:teuthology.orchestra.run.smithi031.stdout: "client_inject_release_failure": "false", 2024-08-19T14:12:36.322 INFO:teuthology.orchestra.run.smithi031.stdout: "client_max_inline_size": "4096", 2024-08-19T14:12:36.322 INFO:teuthology.orchestra.run.smithi031.stdout: "client_max_retries_on_remount_failure": "5", 2024-08-19T14:12:36.322 INFO:teuthology.orchestra.run.smithi031.stdout: "client_mds_namespace": "", 2024-08-19T14:12:36.322 INFO:teuthology.orchestra.run.smithi031.stdout: "client_metadata": "", 2024-08-19T14:12:36.322 INFO:teuthology.orchestra.run.smithi031.stdout: "client_mount_gid": "-1", 2024-08-19T14:12:36.322 INFO:teuthology.orchestra.run.smithi031.stdout: "client_mount_timeout": "300", 2024-08-19T14:12:36.322 INFO:teuthology.orchestra.run.smithi031.stdout: "client_mount_uid": "-1", 2024-08-19T14:12:36.322 INFO:teuthology.orchestra.run.smithi031.stdout: "client_mountpoint": "/", 2024-08-19T14:12:36.322 INFO:teuthology.orchestra.run.smithi031.stdout: "client_notify_timeout": "10", 2024-08-19T14:12:36.322 INFO:teuthology.orchestra.run.smithi031.stdout: "client_oc": "true", 2024-08-19T14:12:36.322 INFO:teuthology.orchestra.run.smithi031.stdout: "client_oc_max_dirty": "104857600", 2024-08-19T14:12:36.322 INFO:teuthology.orchestra.run.smithi031.stdout: "client_oc_max_dirty_age": "5.000000", 2024-08-19T14:12:36.323 INFO:teuthology.orchestra.run.smithi031.stdout: "client_oc_max_objects": "1000", 2024-08-19T14:12:36.323 INFO:teuthology.orchestra.run.smithi031.stdout: "client_oc_size": "209715200", 2024-08-19T14:12:36.323 INFO:teuthology.orchestra.run.smithi031.stdout: "client_oc_target_dirty": "8388608", 2024-08-19T14:12:36.323 INFO:teuthology.orchestra.run.smithi031.stdout: "client_permissions": "true", 2024-08-19T14:12:36.323 INFO:teuthology.orchestra.run.smithi031.stdout: "client_quota": "true", 2024-08-19T14:12:36.323 INFO:teuthology.orchestra.run.smithi031.stdout: "client_quota_df": "true", 2024-08-19T14:12:36.323 INFO:teuthology.orchestra.run.smithi031.stdout: "client_readahead_max_bytes": "0", 2024-08-19T14:12:36.323 INFO:teuthology.orchestra.run.smithi031.stdout: "client_readahead_max_periods": "4", 2024-08-19T14:12:36.323 INFO:teuthology.orchestra.run.smithi031.stdout: "client_readahead_min": "131072", 2024-08-19T14:12:36.323 INFO:teuthology.orchestra.run.smithi031.stdout: "client_reconnect_stale": "false", 2024-08-19T14:12:36.323 INFO:teuthology.orchestra.run.smithi031.stdout: "client_shutdown_timeout": "30", 2024-08-19T14:12:36.323 INFO:teuthology.orchestra.run.smithi031.stdout: "client_snapdir": ".snap", 2024-08-19T14:12:36.323 INFO:teuthology.orchestra.run.smithi031.stdout: "client_tick_interval": "1", 2024-08-19T14:12:36.324 INFO:teuthology.orchestra.run.smithi031.stdout: "client_trace": "", 2024-08-19T14:12:36.324 INFO:teuthology.orchestra.run.smithi031.stdout: "client_try_dentry_invalidate": "false", 2024-08-19T14:12:36.324 INFO:teuthology.orchestra.run.smithi031.stdout: "client_use_faked_inos": "false", 2024-08-19T14:12:36.324 INFO:teuthology.orchestra.run.smithi031.stdout: "client_use_random_mds": "false", 2024-08-19T14:12:36.324 INFO:teuthology.orchestra.run.smithi031.stdout: "clog_to_graylog": "false", 2024-08-19T14:12:36.324 INFO:teuthology.orchestra.run.smithi031.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-08-19T14:12:36.324 INFO:teuthology.orchestra.run.smithi031.stdout: "clog_to_graylog_port": "12201", 2024-08-19T14:12:36.324 INFO:teuthology.orchestra.run.smithi031.stdout: "clog_to_monitors": "default=true", 2024-08-19T14:12:36.324 INFO:teuthology.orchestra.run.smithi031.stdout: "clog_to_syslog": "false", 2024-08-19T14:12:36.324 INFO:teuthology.orchestra.run.smithi031.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-08-19T14:12:36.324 INFO:teuthology.orchestra.run.smithi031.stdout: "clog_to_syslog_level": "info", 2024-08-19T14:12:36.324 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "-", 2024-08-19T14:12:36.324 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_network": "", 2024-08-19T14:12:36.324 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_network_interface": "", 2024-08-19T14:12:36.325 INFO:teuthology.orchestra.run.smithi031.stdout: "compressor_zlib_isal": "false", 2024-08-19T14:12:36.325 INFO:teuthology.orchestra.run.smithi031.stdout: "compressor_zlib_level": "5", 2024-08-19T14:12:36.325 INFO:teuthology.orchestra.run.smithi031.stdout: "compressor_zlib_winsize": "-15", 2024-08-19T14:12:36.325 INFO:teuthology.orchestra.run.smithi031.stdout: "compressor_zstd_level": "1", 2024-08-19T14:12:36.325 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9", 2024-08-19T14:12:36.325 INFO:teuthology.orchestra.run.smithi031.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-08-19T14:12:36.325 INFO:teuthology.orchestra.run.smithi031.stdout: "crimson_alien_op_num_threads": "6", 2024-08-19T14:12:36.325 INFO:teuthology.orchestra.run.smithi031.stdout: "crimson_alien_thread_cpu_cores": "", 2024-08-19T14:12:36.325 INFO:teuthology.orchestra.run.smithi031.stdout: "crimson_osd_obc_lru_size": "512", 2024-08-19T14:12:36.325 INFO:teuthology.orchestra.run.smithi031.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-08-19T14:12:36.325 INFO:teuthology.orchestra.run.smithi031.stdout: "crimson_osd_stat_interval": "0", 2024-08-19T14:12:36.325 INFO:teuthology.orchestra.run.smithi031.stdout: "crimson_seastar_cpu_cores": "", 2024-08-19T14:12:36.325 INFO:teuthology.orchestra.run.smithi031.stdout: "crimson_seastar_num_threads": "0", 2024-08-19T14:12:36.325 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_location": "", 2024-08-19T14:12:36.326 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_location_hook": "", 2024-08-19T14:12:36.326 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_location_hook_timeout": "10", 2024-08-19T14:12:36.326 INFO:teuthology.orchestra.run.smithi031.stdout: "daemonize": "false", 2024-08-19T14:12:36.326 INFO:teuthology.orchestra.run.smithi031.stdout: "daos_pool": "tank", 2024-08-19T14:12:36.326 INFO:teuthology.orchestra.run.smithi031.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-08-19T14:12:36.326 INFO:teuthology.orchestra.run.smithi031.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-08-19T14:12:36.326 INFO:teuthology.orchestra.run.smithi031.stdout: "dbstore_db_name_prefix": "dbstore", 2024-08-19T14:12:36.326 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_alienstore": "0/5", 2024-08-19T14:12:36.326 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_allow_any_pool_priority": "false", 2024-08-19T14:12:36.326 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_asok": "1/5", 2024-08-19T14:12:36.326 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_asok_assert_abort": "false", 2024-08-19T14:12:36.326 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_asserts_on_shutdown": "true", 2024-08-19T14:12:36.327 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_auth": "1/5", 2024-08-19T14:12:36.327 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_bdev": "1/3", 2024-08-19T14:12:36.327 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_bluefs": "1/5", 2024-08-19T14:12:36.327 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_bluestore": "1/5", 2024-08-19T14:12:36.327 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_buffer": "0/1", 2024-08-19T14:12:36.327 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_ceph_exporter": "1/5", 2024-08-19T14:12:36.327 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_cephfs_mirror": "0/5", 2024-08-19T14:12:36.327 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_cephsqlite": "0/5", 2024-08-19T14:12:36.327 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_client": "0/5", 2024-08-19T14:12:36.327 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_compressor": "1/5", 2024-08-19T14:12:36.327 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_context": "0/1", 2024-08-19T14:12:36.327 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_crimson_interrupt": "0/5", 2024-08-19T14:12:36.327 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_crush": "1/1", 2024-08-19T14:12:36.327 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_crypto": "1/5", 2024-08-19T14:12:36.327 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_cyanstore": "0/5", 2024-08-19T14:12:36.328 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_deliberately_leak_memory": "false", 2024-08-19T14:12:36.328 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_disable_randomized_ping": "false", 2024-08-19T14:12:36.328 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_dpdk": "1/5", 2024-08-19T14:12:36.328 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_eventtrace": "1/5", 2024-08-19T14:12:36.328 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_filer": "0/1", 2024-08-19T14:12:36.328 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_filestore": "1/3", 2024-08-19T14:12:36.328 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_finisher": "1/1", 2024-08-19T14:12:36.328 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_fuse": "1/5", 2024-08-19T14:12:36.328 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_heartbeat_testing_span": "0", 2024-08-19T14:12:36.328 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_heartbeatmap": "1/5", 2024-08-19T14:12:36.328 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_immutable_obj_cache": "0/5", 2024-08-19T14:12:36.328 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_javaclient": "1/5", 2024-08-19T14:12:36.328 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_journal": "1/3", 2024-08-19T14:12:36.328 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_journaler": "0/5", 2024-08-19T14:12:36.329 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_kstore": "1/5", 2024-08-19T14:12:36.329 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_lockdep": "0/1", 2024-08-19T14:12:36.329 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_mclock": "1/5", 2024-08-19T14:12:36.329 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_mds": "1/5", 2024-08-19T14:12:36.329 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_mds_balancer": "1/5", 2024-08-19T14:12:36.329 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_mds_locker": "1/5", 2024-08-19T14:12:36.329 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_mds_log": "1/5", 2024-08-19T14:12:36.329 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_mds_log_expire": "1/5", 2024-08-19T14:12:36.329 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_mds_migrator": "1/5", 2024-08-19T14:12:36.329 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_mds_quiesce": "3/5", 2024-08-19T14:12:36.329 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_memstore": "1/5", 2024-08-19T14:12:36.329 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_mgr": "2/5", 2024-08-19T14:12:36.329 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_mgrc": "1/5", 2024-08-19T14:12:36.329 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_mon": "20/20", 2024-08-19T14:12:36.329 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_monc": "0/10", 2024-08-19T14:12:36.330 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_ms": "1/1", 2024-08-19T14:12:36.330 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_none": "0/5", 2024-08-19T14:12:36.330 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_objclass": "0/5", 2024-08-19T14:12:36.330 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_objectcacher": "0/5", 2024-08-19T14:12:36.330 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_objecter": "0/1", 2024-08-19T14:12:36.330 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_optracker": "0/5", 2024-08-19T14:12:36.330 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_osd": "1/5", 2024-08-19T14:12:36.330 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_paxos": "20/20", 2024-08-19T14:12:36.330 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_perfcounter": "1/5", 2024-08-19T14:12:36.330 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_prioritycache": "1/5", 2024-08-19T14:12:36.330 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_rados": "0/5", 2024-08-19T14:12:36.330 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_rbd": "0/5", 2024-08-19T14:12:36.330 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_rbd_mirror": "0/5", 2024-08-19T14:12:36.331 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_rbd_pwl": "0/5", 2024-08-19T14:12:36.331 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_rbd_replay": "0/5", 2024-08-19T14:12:36.331 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_refs": "0/0", 2024-08-19T14:12:36.331 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_reserver": "1/1", 2024-08-19T14:12:36.331 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_rgw": "1/5", 2024-08-19T14:12:36.331 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_rgw_access": "1/5", 2024-08-19T14:12:36.331 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_rgw_datacache": "1/5", 2024-08-19T14:12:36.331 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_rgw_dbstore": "1/5", 2024-08-19T14:12:36.331 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_rgw_flight": "1/5", 2024-08-19T14:12:36.331 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_rgw_lifecycle": "1/5", 2024-08-19T14:12:36.331 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_rgw_sync": "1/5", 2024-08-19T14:12:36.331 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_rocksdb": "4/5", 2024-08-19T14:12:36.331 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_seastore": "0/5", 2024-08-19T14:12:36.331 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_seastore_backref": "0/5", 2024-08-19T14:12:36.332 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_seastore_cache": "0/5", 2024-08-19T14:12:36.332 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_seastore_cleaner": "0/5", 2024-08-19T14:12:36.332 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_seastore_device": "0/5", 2024-08-19T14:12:36.332 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_seastore_epm": "0/5", 2024-08-19T14:12:36.332 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-08-19T14:12:36.332 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_seastore_journal": "0/5", 2024-08-19T14:12:36.332 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_seastore_lba": "0/5", 2024-08-19T14:12:36.332 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_seastore_odata": "0/5", 2024-08-19T14:12:36.332 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_seastore_omap": "0/5", 2024-08-19T14:12:36.332 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_seastore_onode": "0/5", 2024-08-19T14:12:36.332 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_seastore_t": "0/5", 2024-08-19T14:12:36.332 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_seastore_tm": "0/5", 2024-08-19T14:12:36.333 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_striper": "0/1", 2024-08-19T14:12:36.333 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_test": "0/5", 2024-08-19T14:12:36.333 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_throttle": "1/1", 2024-08-19T14:12:36.333 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_timer": "0/1", 2024-08-19T14:12:36.333 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_tp": "0/5", 2024-08-19T14:12:36.333 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_trace": "1/5", 2024-08-19T14:12:36.333 INFO:teuthology.orchestra.run.smithi031.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-08-19T14:12:36.333 INFO:teuthology.orchestra.run.smithi031.stdout: "device_failure_prediction_mode": "none", 2024-08-19T14:12:36.333 INFO:teuthology.orchestra.run.smithi031.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-08-19T14:12:36.333 INFO:teuthology.orchestra.run.smithi031.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-08-19T14:12:36.333 INFO:teuthology.orchestra.run.smithi031.stdout: "err_to_graylog": "false", 2024-08-19T14:12:36.333 INFO:teuthology.orchestra.run.smithi031.stdout: "err_to_journald": "false", 2024-08-19T14:12:36.333 INFO:teuthology.orchestra.run.smithi031.stdout: "err_to_stderr": "true", 2024-08-19T14:12:36.333 INFO:teuthology.orchestra.run.smithi031.stdout: "err_to_syslog": "false", 2024-08-19T14:12:36.334 INFO:teuthology.orchestra.run.smithi031.stdout: "event_tracing": "false", 2024-08-19T14:12:36.334 INFO:teuthology.orchestra.run.smithi031.stdout: "exporter_addr": "0.0.0.0", 2024-08-19T14:12:36.334 INFO:teuthology.orchestra.run.smithi031.stdout: "exporter_http_port": "9926", 2024-08-19T14:12:36.334 INFO:teuthology.orchestra.run.smithi031.stdout: "exporter_prio_limit": "5", 2024-08-19T14:12:36.334 INFO:teuthology.orchestra.run.smithi031.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-08-19T14:12:36.334 INFO:teuthology.orchestra.run.smithi031.stdout: "exporter_sort_metrics": "true", 2024-08-19T14:12:36.334 INFO:teuthology.orchestra.run.smithi031.stdout: "exporter_stats_period": "5", 2024-08-19T14:12:36.334 INFO:teuthology.orchestra.run.smithi031.stdout: "fake_statfs_for_testing": "0", 2024-08-19T14:12:36.334 INFO:teuthology.orchestra.run.smithi031.stdout: "fatal_signal_handlers": "true", 2024-08-19T14:12:36.334 INFO:teuthology.orchestra.run.smithi031.stdout: "filer_max_purge_ops": "10", 2024-08-19T14:12:36.334 INFO:teuthology.orchestra.run.smithi031.stdout: "filer_max_truncate_ops": "128", 2024-08-19T14:12:36.334 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_apply_finisher_threads": "1", 2024-08-19T14:12:36.335 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_blackhole": "false", 2024-08-19T14:12:36.335 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_btrfs_clone_range": "true", 2024-08-19T14:12:36.335 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_btrfs_snap": "true", 2024-08-19T14:12:36.335 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_caller_concurrency": "10", 2024-08-19T14:12:36.335 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_collect_device_partition_information": "true", 2024-08-19T14:12:36.335 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_commit_timeout": "600.000000", 2024-08-19T14:12:36.335 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_debug_inject_read_err": "false", 2024-08-19T14:12:36.335 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_debug_omap_check": "false", 2024-08-19T14:12:36.335 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_debug_random_read_err": "0.000000", 2024-08-19T14:12:36.335 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_debug_verify_split": "false", 2024-08-19T14:12:36.335 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_dump_file": "", 2024-08-19T14:12:36.335 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-08-19T14:12:36.335 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-08-19T14:12:36.336 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_fadvise": "true", 2024-08-19T14:12:36.336 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_fail_eio": "true", 2024-08-19T14:12:36.336 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_fd_cache_shards": "16", 2024-08-19T14:12:36.336 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_fd_cache_size": "128", 2024-08-19T14:12:36.336 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_fiemap": "false", 2024-08-19T14:12:36.336 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_fiemap_threshold": "4096", 2024-08-19T14:12:36.336 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-08-19T14:12:36.336 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_index_retry_probability": "0.000000", 2024-08-19T14:12:36.336 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_inject_stall": "0", 2024-08-19T14:12:36.336 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_journal_parallel": "false", 2024-08-19T14:12:36.336 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_journal_trailing": "false", 2024-08-19T14:12:36.336 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_journal_writeahead": "false", 2024-08-19T14:12:36.337 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_kill_at": "0", 2024-08-19T14:12:36.337 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-08-19T14:12:36.337 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_max_inline_xattr_size": "0", 2024-08-19T14:12:36.337 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-08-19T14:12:36.337 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-08-19T14:12:36.337 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-08-19T14:12:36.337 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_max_inline_xattrs": "0", 2024-08-19T14:12:36.337 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-08-19T14:12:36.337 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_max_inline_xattrs_other": "2", 2024-08-19T14:12:36.337 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-08-19T14:12:36.337 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_max_sync_interval": "5.000000", 2024-08-19T14:12:36.337 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_max_xattr_value_size": "0", 2024-08-19T14:12:36.337 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-08-19T14:12:36.337 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-08-19T14:12:36.337 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-08-19T14:12:36.338 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_merge_threshold": "-10", 2024-08-19T14:12:36.338 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_min_sync_interval": "0.010000", 2024-08-19T14:12:36.338 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_odsync_write": "false", 2024-08-19T14:12:36.338 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_omap_backend": "rocksdb", 2024-08-19T14:12:36.338 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_omap_backend_path": "", 2024-08-19T14:12:36.338 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_omap_header_cache_size": "1024", 2024-08-19T14:12:36.338 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_ondisk_finisher_threads": "1", 2024-08-19T14:12:36.338 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-08-19T14:12:36.338 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_op_thread_timeout": "60", 2024-08-19T14:12:36.338 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_op_threads": "2", 2024-08-19T14:12:36.338 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_punch_hole": "false", 2024-08-19T14:12:36.338 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-08-19T14:12:36.339 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-08-19T14:12:36.339 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-08-19T14:12:36.339 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-08-19T14:12:36.339 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-08-19T14:12:36.339 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_queue_max_bytes": "104857600", 2024-08-19T14:12:36.339 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-08-19T14:12:36.339 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-08-19T14:12:36.339 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-08-19T14:12:36.339 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_queue_max_ops": "50", 2024-08-19T14:12:36.339 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-08-19T14:12:36.339 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_seek_data_hole": "false", 2024-08-19T14:12:36.339 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_sloppy_crc": "false", 2024-08-19T14:12:36.339 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-08-19T14:12:36.340 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_splice": "false", 2024-08-19T14:12:36.340 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_split_multiple": "2", 2024-08-19T14:12:36.340 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_split_rand_factor": "20", 2024-08-19T14:12:36.340 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_update_to": "1000", 2024-08-19T14:12:36.340 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-08-19T14:12:36.340 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-08-19T14:12:36.340 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-08-19T14:12:36.340 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-08-19T14:12:36.340 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-08-19T14:12:36.340 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-08-19T14:12:36.340 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_wbthrottle_enable": "true", 2024-08-19T14:12:36.340 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-08-19T14:12:36.341 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-08-19T14:12:36.341 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-08-19T14:12:36.341 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-08-19T14:12:36.341 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-08-19T14:12:36.341 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-08-19T14:12:36.341 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_xfs_extsize": "false", 2024-08-19T14:12:36.341 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_zfs_snap": "false", 2024-08-19T14:12:36.341 INFO:teuthology.orchestra.run.smithi031.stdout: "fio_dir": "/tmp/fio", 2024-08-19T14:12:36.341 INFO:teuthology.orchestra.run.smithi031.stdout: "fsid": "ed7d1c6e-5e33-11ef-bccf-c7b262605968", 2024-08-19T14:12:36.341 INFO:teuthology.orchestra.run.smithi031.stdout: "fuse_allow_other": "true", 2024-08-19T14:12:36.341 INFO:teuthology.orchestra.run.smithi031.stdout: "fuse_atomic_o_trunc": "true", 2024-08-19T14:12:36.341 INFO:teuthology.orchestra.run.smithi031.stdout: "fuse_big_writes": "true", 2024-08-19T14:12:36.341 INFO:teuthology.orchestra.run.smithi031.stdout: "fuse_debug": "false", 2024-08-19T14:12:36.342 INFO:teuthology.orchestra.run.smithi031.stdout: "fuse_default_permissions": "false", 2024-08-19T14:12:36.342 INFO:teuthology.orchestra.run.smithi031.stdout: "fuse_disable_pagecache": "false", 2024-08-19T14:12:36.342 INFO:teuthology.orchestra.run.smithi031.stdout: "fuse_max_write": "0", 2024-08-19T14:12:36.342 INFO:teuthology.orchestra.run.smithi031.stdout: "fuse_multithreaded": "true", 2024-08-19T14:12:36.342 INFO:teuthology.orchestra.run.smithi031.stdout: "fuse_require_active_mds": "true", 2024-08-19T14:12:36.342 INFO:teuthology.orchestra.run.smithi031.stdout: "fuse_set_user_groups": "true", 2024-08-19T14:12:36.342 INFO:teuthology.orchestra.run.smithi031.stdout: "fuse_splice_move": "true", 2024-08-19T14:12:36.342 INFO:teuthology.orchestra.run.smithi031.stdout: "fuse_splice_read": "true", 2024-08-19T14:12:36.342 INFO:teuthology.orchestra.run.smithi031.stdout: "fuse_splice_write": "true", 2024-08-19T14:12:36.342 INFO:teuthology.orchestra.run.smithi031.stdout: "fuse_syncfs_on_mksnap": "true", 2024-08-19T14:12:36.342 INFO:teuthology.orchestra.run.smithi031.stdout: "fuse_use_invalidate_cb": "true", 2024-08-19T14:12:36.342 INFO:teuthology.orchestra.run.smithi031.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-08-19T14:12:36.342 INFO:teuthology.orchestra.run.smithi031.stdout: "gss_target_name": "ceph", 2024-08-19T14:12:36.343 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_file": "", 2024-08-19T14:12:36.343 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_inject_failure": "0", 2024-08-19T14:12:36.343 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_interval": "5", 2024-08-19T14:12:36.343 INFO:teuthology.orchestra.run.smithi031.stdout: "host": "smithi031", 2024-08-19T14:12:36.343 INFO:teuthology.orchestra.run.smithi031.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-08-19T14:12:36.343 INFO:teuthology.orchestra.run.smithi031.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-08-19T14:12:36.343 INFO:teuthology.orchestra.run.smithi031.stdout: "immutable_object_cache_max_size": "1073741824", 2024-08-19T14:12:36.343 INFO:teuthology.orchestra.run.smithi031.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-08-19T14:12:36.343 INFO:teuthology.orchestra.run.smithi031.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-08-19T14:12:36.343 INFO:teuthology.orchestra.run.smithi031.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-08-19T14:12:36.343 INFO:teuthology.orchestra.run.smithi031.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-08-19T14:12:36.343 INFO:teuthology.orchestra.run.smithi031.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-08-19T14:12:36.343 INFO:teuthology.orchestra.run.smithi031.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-08-19T14:12:36.343 INFO:teuthology.orchestra.run.smithi031.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-08-19T14:12:36.344 INFO:teuthology.orchestra.run.smithi031.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-08-19T14:12:36.344 INFO:teuthology.orchestra.run.smithi031.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-08-19T14:12:36.344 INFO:teuthology.orchestra.run.smithi031.stdout: "immutable_object_cache_watermark": "0.900000", 2024-08-19T14:12:36.344 INFO:teuthology.orchestra.run.smithi031.stdout: "inject_early_sigterm": "false", 2024-08-19T14:12:36.344 INFO:teuthology.orchestra.run.smithi031.stdout: "jaeger_agent_port": "6799", 2024-08-19T14:12:36.344 INFO:teuthology.orchestra.run.smithi031.stdout: "jaeger_tracing_enable": "false", 2024-08-19T14:12:36.344 INFO:teuthology.orchestra.run.smithi031.stdout: "journal_aio": "true", 2024-08-19T14:12:36.344 INFO:teuthology.orchestra.run.smithi031.stdout: "journal_align_min_size": "65536", 2024-08-19T14:12:36.344 INFO:teuthology.orchestra.run.smithi031.stdout: "journal_block_align": "true", 2024-08-19T14:12:36.344 INFO:teuthology.orchestra.run.smithi031.stdout: "journal_block_size": "4096", 2024-08-19T14:12:36.344 INFO:teuthology.orchestra.run.smithi031.stdout: "journal_dio": "true", 2024-08-19T14:12:36.344 INFO:teuthology.orchestra.run.smithi031.stdout: "journal_discard": "false", 2024-08-19T14:12:36.345 INFO:teuthology.orchestra.run.smithi031.stdout: "journal_force_aio": "false", 2024-08-19T14:12:36.345 INFO:teuthology.orchestra.run.smithi031.stdout: "journal_ignore_corruption": "false", 2024-08-19T14:12:36.345 INFO:teuthology.orchestra.run.smithi031.stdout: "journal_max_write_bytes": "10485760", 2024-08-19T14:12:36.345 INFO:teuthology.orchestra.run.smithi031.stdout: "journal_max_write_entries": "100", 2024-08-19T14:12:36.345 INFO:teuthology.orchestra.run.smithi031.stdout: "journal_replay_from": "0", 2024-08-19T14:12:36.345 INFO:teuthology.orchestra.run.smithi031.stdout: "journal_throttle_high_multiple": "0.000000", 2024-08-19T14:12:36.345 INFO:teuthology.orchestra.run.smithi031.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-08-19T14:12:36.345 INFO:teuthology.orchestra.run.smithi031.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-08-19T14:12:36.345 INFO:teuthology.orchestra.run.smithi031.stdout: "journal_throttle_max_multiple": "0.000000", 2024-08-19T14:12:36.345 INFO:teuthology.orchestra.run.smithi031.stdout: "journal_write_header_frequency": "0", 2024-08-19T14:12:36.345 INFO:teuthology.orchestra.run.smithi031.stdout: "journal_zero_on_create": "false", 2024-08-19T14:12:36.345 INFO:teuthology.orchestra.run.smithi031.stdout: "journaler_prefetch_periods": "10", 2024-08-19T14:12:36.345 INFO:teuthology.orchestra.run.smithi031.stdout: "journaler_prezero_periods": "5", 2024-08-19T14:12:36.345 INFO:teuthology.orchestra.run.smithi031.stdout: "journaler_write_head_interval": "15", 2024-08-19T14:12:36.346 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "", 2024-08-19T14:12:36.346 INFO:teuthology.orchestra.run.smithi031.stdout: "keyfile": "", 2024-08-19T14:12:36.346 INFO:teuthology.orchestra.run.smithi031.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-08-19T14:12:36.346 INFO:teuthology.orchestra.run.smithi031.stdout: "kstore_backend": "rocksdb", 2024-08-19T14:12:36.346 INFO:teuthology.orchestra.run.smithi031.stdout: "kstore_default_stripe_size": "65536", 2024-08-19T14:12:36.346 INFO:teuthology.orchestra.run.smithi031.stdout: "kstore_fsck_on_mount": "false", 2024-08-19T14:12:36.346 INFO:teuthology.orchestra.run.smithi031.stdout: "kstore_fsck_on_mount_deep": "true", 2024-08-19T14:12:36.346 INFO:teuthology.orchestra.run.smithi031.stdout: "kstore_max_bytes": "67108864", 2024-08-19T14:12:36.346 INFO:teuthology.orchestra.run.smithi031.stdout: "kstore_max_ops": "512", 2024-08-19T14:12:36.346 INFO:teuthology.orchestra.run.smithi031.stdout: "kstore_nid_prealloc": "1024", 2024-08-19T14:12:36.346 INFO:teuthology.orchestra.run.smithi031.stdout: "kstore_onode_map_size": "1024", 2024-08-19T14:12:36.346 INFO:teuthology.orchestra.run.smithi031.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-08-19T14:12:36.346 INFO:teuthology.orchestra.run.smithi031.stdout: "kstore_sync_submit_transaction": "false", 2024-08-19T14:12:36.347 INFO:teuthology.orchestra.run.smithi031.stdout: "kstore_sync_transaction": "false", 2024-08-19T14:12:36.347 INFO:teuthology.orchestra.run.smithi031.stdout: "librados_thread_count": "2", 2024-08-19T14:12:36.347 INFO:teuthology.orchestra.run.smithi031.stdout: "lockdep": "false", 2024-08-19T14:12:36.347 INFO:teuthology.orchestra.run.smithi031.stdout: "lockdep_force_backtrace": "false", 2024-08-19T14:12:36.347 INFO:teuthology.orchestra.run.smithi031.stdout: "log_coarse_timestamps": "true", 2024-08-19T14:12:36.347 INFO:teuthology.orchestra.run.smithi031.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-08-19T14:12:36.347 INFO:teuthology.orchestra.run.smithi031.stdout: "log_flush_on_exit": "false", 2024-08-19T14:12:36.347 INFO:teuthology.orchestra.run.smithi031.stdout: "log_graylog_host": "127.0.0.1", 2024-08-19T14:12:36.347 INFO:teuthology.orchestra.run.smithi031.stdout: "log_graylog_port": "12201", 2024-08-19T14:12:36.347 INFO:teuthology.orchestra.run.smithi031.stdout: "log_max_new": "1000", 2024-08-19T14:12:36.347 INFO:teuthology.orchestra.run.smithi031.stdout: "log_max_recent": "10000", 2024-08-19T14:12:36.347 INFO:teuthology.orchestra.run.smithi031.stdout: "log_stderr_prefix": "", 2024-08-19T14:12:36.347 INFO:teuthology.orchestra.run.smithi031.stdout: "log_stop_at_utilization": "0.970000", 2024-08-19T14:12:36.348 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": "true", 2024-08-19T14:12:36.348 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_graylog": "false", 2024-08-19T14:12:36.348 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_journald": "false", 2024-08-19T14:12:36.348 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_stderr": "false", 2024-08-19T14:12:36.348 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_syslog": "false", 2024-08-19T14:12:36.348 INFO:teuthology.orchestra.run.smithi031.stdout: "max_rotating_auth_attempts": "10", 2024-08-19T14:12:36.348 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-08-19T14:12:36.348 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_action_on_write_error": "1", 2024-08-19T14:12:36.348 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_alternate_name_max": "8192", 2024-08-19T14:12:36.348 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_asio_thread_count": "2", 2024-08-19T14:12:36.348 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_bal_export_pin": "true", 2024-08-19T14:12:36.348 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_bal_fragment_dirs": "true", 2024-08-19T14:12:36.349 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-08-19T14:12:36.349 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_bal_fragment_interval": "5", 2024-08-19T14:12:36.349 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_bal_fragment_size_max": "100000", 2024-08-19T14:12:36.349 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_bal_idle_threshold": "0.000000", 2024-08-19T14:12:36.349 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_bal_interval": "10", 2024-08-19T14:12:36.349 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_bal_max": "-1", 2024-08-19T14:12:36.349 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_bal_max_until": "-1", 2024-08-19T14:12:36.349 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_bal_merge_size": "50", 2024-08-19T14:12:36.349 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_bal_midchunk": "0.300000", 2024-08-19T14:12:36.349 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_bal_min_rebalance": "0.100000", 2024-08-19T14:12:36.349 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_bal_min_start": "0.200000", 2024-08-19T14:12:36.349 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_bal_minchunk": "0.001000", 2024-08-19T14:12:36.349 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_bal_mode": "0", 2024-08-19T14:12:36.350 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_bal_need_max": "1.200000", 2024-08-19T14:12:36.350 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_bal_need_min": "0.800000", 2024-08-19T14:12:36.350 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_bal_overload_epochs": "2", 2024-08-19T14:12:36.350 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-08-19T14:12:36.350 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_bal_sample_interval": "3.000000", 2024-08-19T14:12:36.350 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_bal_split_bits": "3", 2024-08-19T14:12:36.350 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_bal_split_rd": "25000.000000", 2024-08-19T14:12:36.350 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_bal_split_size": "10000", 2024-08-19T14:12:36.350 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_bal_split_wr": "10000.000000", 2024-08-19T14:12:36.350 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_bal_target_decay": "10.000000", 2024-08-19T14:12:36.350 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-08-19T14:12:36.350 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_beacon_grace": "15.000000", 2024-08-19T14:12:36.350 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_beacon_interval": "4.000000", 2024-08-19T14:12:36.351 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_beacon_mon_down_grace": "60", 2024-08-19T14:12:36.351 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_cache_memory_limit": "4294967296", 2024-08-19T14:12:36.351 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_cache_mid": "0.700000", 2024-08-19T14:12:36.351 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_cache_quiesce_decay_rate": "1.000000", 2024-08-19T14:12:36.351 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_cache_quiesce_delay": "0", 2024-08-19T14:12:36.351 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_cache_quiesce_sleep": "200", 2024-08-19T14:12:36.351 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_cache_quiesce_splitauth": "true", 2024-08-19T14:12:36.351 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_cache_quiesce_threshold": "524288", 2024-08-19T14:12:36.351 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_cache_release_free_interval": "10", 2024-08-19T14:12:36.351 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_cache_reservation": "0.050000", 2024-08-19T14:12:36.351 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-08-19T14:12:36.351 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_cache_trim_interval": "1", 2024-08-19T14:12:36.351 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_cache_trim_threshold": "262144", 2024-08-19T14:12:36.352 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-08-19T14:12:36.352 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-08-19T14:12:36.352 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_client_delegate_inos_pct": "50", 2024-08-19T14:12:36.352 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_client_prealloc_inos": "1000", 2024-08-19T14:12:36.352 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-08-19T14:12:36.352 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_connect_bootstrapping": "false", 2024-08-19T14:12:36.352 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_damage_table_max_entries": "10000", 2024-08-19T14:12:36.352 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-08-19T14:12:36.352 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_debug_auth_pins": "false", 2024-08-19T14:12:36.352 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_debug_frag": "false", 2024-08-19T14:12:36.352 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_debug_scatterstat": "false", 2024-08-19T14:12:36.352 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_debug_subtrees": "false", 2024-08-19T14:12:36.352 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_decay_halflife": "5.000000", 2024-08-19T14:12:36.353 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_default_dir_hash": "2", 2024-08-19T14:12:36.353 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_defer_session_stale": "true", 2024-08-19T14:12:36.353 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_deny_all_reconnect": "false", 2024-08-19T14:12:36.353 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_dir_keys_per_op": "16384", 2024-08-19T14:12:36.353 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_dir_max_commit_size": "10", 2024-08-19T14:12:36.353 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_dir_max_entries": "0", 2024-08-19T14:12:36.353 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_dir_prefetch": "true", 2024-08-19T14:12:36.353 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_dirstat_min_interval": "1.000000", 2024-08-19T14:12:36.353 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_dump_cache_after_rejoin": "false", 2024-08-19T14:12:36.353 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_dump_cache_on_map": "false", 2024-08-19T14:12:36.353 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_dump_cache_threshold_file": "0", 2024-08-19T14:12:36.353 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-08-19T14:12:36.353 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_early_reply": "true", 2024-08-19T14:12:36.354 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_enable_op_tracker": "true", 2024-08-19T14:12:36.354 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_enforce_unique_name": "true", 2024-08-19T14:12:36.354 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_export_ephemeral_distributed": "true", 2024-08-19T14:12:36.354 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-08-19T14:12:36.354 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_export_ephemeral_random": "true", 2024-08-19T14:12:36.354 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-08-19T14:12:36.354 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-08-19T14:12:36.354 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_forward_all_requests_to_auth": "false", 2024-08-19T14:12:36.354 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-08-19T14:12:36.354 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-08-19T14:12:36.354 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-08-19T14:12:36.354 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-08-19T14:12:36.355 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_health_cache_threshold": "1.500000", 2024-08-19T14:12:36.355 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_health_summarize_threshold": "10", 2024-08-19T14:12:36.355 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_heartbeat_grace": "15.000000", 2024-08-19T14:12:36.355 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_heartbeat_reset_grace": "1000", 2024-08-19T14:12:36.355 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_inject_health_dummy": "false", 2024-08-19T14:12:36.355 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-08-19T14:12:36.355 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_inject_migrator_session_race": "false", 2024-08-19T14:12:36.355 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-08-19T14:12:36.355 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-08-19T14:12:36.355 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-08-19T14:12:36.355 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_join_fs": "", 2024-08-19T14:12:36.355 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_journal_format": "1", 2024-08-19T14:12:36.355 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_kill_after_journal_logs_flushed": "false", 2024-08-19T14:12:36.356 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_kill_create_at": "0", 2024-08-19T14:12:36.356 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_kill_export_at": "0", 2024-08-19T14:12:36.356 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_kill_import_at": "0", 2024-08-19T14:12:36.356 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_kill_journal_at": "0", 2024-08-19T14:12:36.356 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_kill_journal_expire_at": "0", 2024-08-19T14:12:36.356 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_kill_journal_replay_at": "0", 2024-08-19T14:12:36.356 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_kill_link_at": "0", 2024-08-19T14:12:36.356 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_kill_mdstable_at": "0", 2024-08-19T14:12:36.356 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_kill_openc_at": "0", 2024-08-19T14:12:36.356 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_kill_rename_at": "0", 2024-08-19T14:12:36.356 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_kill_shutdown_at": "0", 2024-08-19T14:12:36.356 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_log_event_large_threshold": "524288", 2024-08-19T14:12:36.356 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_log_events_per_segment": "1024", 2024-08-19T14:12:36.356 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_log_major_segment_event_ratio": "12", 2024-08-19T14:12:36.357 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_log_max_events": "-1", 2024-08-19T14:12:36.357 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_log_max_segments": "128", 2024-08-19T14:12:36.357 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_log_pause": "false", 2024-08-19T14:12:36.357 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_log_segment_size": "0", 2024-08-19T14:12:36.357 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_log_skip_corrupt_events": "false", 2024-08-19T14:12:36.357 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_log_skip_unbounded_events": "false", 2024-08-19T14:12:36.357 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-08-19T14:12:36.357 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_log_trim_threshold": "128", 2024-08-19T14:12:36.357 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-08-19T14:12:36.357 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_log_warn_factor": "2.000000", 2024-08-19T14:12:36.357 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_max_caps_per_client": "1048576", 2024-08-19T14:12:36.357 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_max_completed_flushes": "100000", 2024-08-19T14:12:36.357 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_max_completed_requests": "100000", 2024-08-19T14:12:36.357 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_max_export_size": "20971520", 2024-08-19T14:12:36.358 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_max_file_recover": "32", 2024-08-19T14:12:36.358 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_max_purge_files": "64", 2024-08-19T14:12:36.358 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_max_purge_ops": "8192", 2024-08-19T14:12:36.358 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-08-19T14:12:36.358 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-08-19T14:12:36.358 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_max_snaps_per_dir": "100", 2024-08-19T14:12:36.358 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_metrics_update_interval": "2", 2024-08-19T14:12:36.358 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_min_caps_per_client": "100", 2024-08-19T14:12:36.358 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_min_caps_working_set": "10000", 2024-08-19T14:12:36.358 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-08-19T14:12:36.358 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_numa_node": "-1", 2024-08-19T14:12:36.358 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-08-19T14:12:36.358 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_op_complaint_time": "30.000000", 2024-08-19T14:12:36.359 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_op_history_duration": "600", 2024-08-19T14:12:36.359 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_op_history_size": "20", 2024-08-19T14:12:36.359 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_op_history_slow_op_size": "20", 2024-08-19T14:12:36.359 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_op_history_slow_op_threshold": "10", 2024-08-19T14:12:36.359 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_op_log_threshold": "5", 2024-08-19T14:12:36.359 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_ping_grace": "15", 2024-08-19T14:12:36.359 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_ping_interval": "5", 2024-08-19T14:12:36.359 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-08-19T14:12:36.359 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-08-19T14:12:36.359 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_recall_max_caps": "30000", 2024-08-19T14:12:36.359 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-08-19T14:12:36.359 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_recall_max_decay_threshold": "131072", 2024-08-19T14:12:36.359 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-08-19T14:12:36.360 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_recall_warning_threshold": "262144", 2024-08-19T14:12:36.360 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_reconnect_timeout": "45.000000", 2024-08-19T14:12:36.360 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_replay_interval": "1.000000", 2024-08-19T14:12:36.360 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-08-19T14:12:36.360 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-08-19T14:12:36.360 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_root_ino_gid": "0", 2024-08-19T14:12:36.360 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_root_ino_uid": "0", 2024-08-19T14:12:36.360 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-08-19T14:12:36.360 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_session_blocklist_on_evict": "true", 2024-08-19T14:12:36.360 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_session_blocklist_on_timeout": "true", 2024-08-19T14:12:36.360 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-08-19T14:12:36.360 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-08-19T14:12:36.360 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-08-19T14:12:36.360 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-08-19T14:12:36.361 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-08-19T14:12:36.361 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_session_metadata_threshold": "16777216", 2024-08-19T14:12:36.361 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-08-19T14:12:36.361 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_shutdown_check": "0", 2024-08-19T14:12:36.361 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_skip_ino": "0", 2024-08-19T14:12:36.361 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_sleep_rank_change": "0.000000", 2024-08-19T14:12:36.361 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_snap_max_uid": "4294967294", 2024-08-19T14:12:36.361 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_snap_min_uid": "0", 2024-08-19T14:12:36.361 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_snap_rstat": "false", 2024-08-19T14:12:36.361 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_standby_replay_damaged": "false", 2024-08-19T14:12:36.361 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_symlink_recovery": "true", 2024-08-19T14:12:36.361 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_task_status_update_interval": "2.000000", 2024-08-19T14:12:36.361 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_thrash_exports": "0", 2024-08-19T14:12:36.362 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_thrash_fragments": "0", 2024-08-19T14:12:36.362 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_tick_interval": "5.000000", 2024-08-19T14:12:36.362 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_valgrind_exit": "false", 2024-08-19T14:12:36.362 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_verify_backtrace": "1", 2024-08-19T14:12:36.362 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_verify_scatter": "false", 2024-08-19T14:12:36.362 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_wipe_ino_prealloc": "false", 2024-08-19T14:12:36.362 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_wipe_sessions": "false", 2024-08-19T14:12:36.362 INFO:teuthology.orchestra.run.smithi031.stdout: "mempool_debug": "false", 2024-08-19T14:12:36.362 INFO:teuthology.orchestra.run.smithi031.stdout: "memstore_debug_omit_block_device_write": "false", 2024-08-19T14:12:36.362 INFO:teuthology.orchestra.run.smithi031.stdout: "memstore_device_bytes": "1073741824", 2024-08-19T14:12:36.362 INFO:teuthology.orchestra.run.smithi031.stdout: "memstore_page_set": "false", 2024-08-19T14:12:36.362 INFO:teuthology.orchestra.run.smithi031.stdout: "memstore_page_size": "65536", 2024-08-19T14:12:36.362 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_client_bytes": "134217728", 2024-08-19T14:12:36.362 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_client_messages": "512", 2024-08-19T14:12:36.362 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-08-19T14:12:36.363 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_connect_retry_interval": "1.000000", 2024-08-19T14:12:36.363 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-08-19T14:12:36.363 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-08-19T14:12:36.363 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_disabled_modules": "", 2024-08-19T14:12:36.363 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_enable_op_tracker": "true", 2024-08-19T14:12:36.363 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-08-19T14:12:36.363 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_max_pg_num_change": "128", 2024-08-19T14:12:36.363 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_mds_bytes": "134217728", 2024-08-19T14:12:36.363 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_mds_messages": "128", 2024-08-19T14:12:36.363 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-08-19T14:12:36.363 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_mon_bytes": "134217728", 2024-08-19T14:12:36.363 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_mon_messages": "128", 2024-08-19T14:12:36.363 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_num_op_tracker_shard": "32", 2024-08-19T14:12:36.364 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_op_complaint_time": "30.000000", 2024-08-19T14:12:36.364 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_op_history_duration": "600", 2024-08-19T14:12:36.364 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_op_history_size": "20", 2024-08-19T14:12:36.364 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_op_history_slow_op_size": "20", 2024-08-19T14:12:36.364 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_op_history_slow_op_threshold": "10.000000", 2024-08-19T14:12:36.364 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_op_log_threshold": "5", 2024-08-19T14:12:36.364 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_osd_bytes": "536870912", 2024-08-19T14:12:36.364 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_osd_messages": "8192", 2024-08-19T14:12:36.364 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_pool": "true", 2024-08-19T14:12:36.364 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_service_beacon_grace": "60.000000", 2024-08-19T14:12:36.364 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_standby_modules": "true", 2024-08-19T14:12:36.364 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_stats_period": "5", 2024-08-19T14:12:36.364 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_stats_threshold": "5", 2024-08-19T14:12:36.365 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_tick_period": "2", 2024-08-19T14:12:36.365 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-08-19T14:12:36.365 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_accept_timeout_factor": "2.000000", 2024-08-19T14:12:36.365 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_allow_pool_delete": "true", 2024-08-19T14:12:36.365 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_allow_pool_size_one": "false", 2024-08-19T14:12:36.365 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_auth_validate_all_caps": "true", 2024-08-19T14:12:36.365 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-08-19T14:12:36.365 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-08-19T14:12:36.365 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_client_bytes": "104857600", 2024-08-19T14:12:36.365 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_client_directed_command_retry": "5", 2024-08-19T14:12:36.365 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_client_hunt_interval": "3.000000", 2024-08-19T14:12:36.365 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-08-19T14:12:36.365 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-08-19T14:12:36.366 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-08-19T14:12:36.366 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_client_hunt_parallel": "3", 2024-08-19T14:12:36.366 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_client_log_interval": "1.000000", 2024-08-19T14:12:36.366 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-08-19T14:12:36.366 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_client_ping_interval": "10.000000", 2024-08-19T14:12:36.366 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_client_ping_timeout": "30.000000", 2024-08-19T14:12:36.366 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_client_target_rank": "-1", 2024-08-19T14:12:36.366 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_clock_drift_allowed": "1.000000", 2024-08-19T14:12:36.366 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-08-19T14:12:36.366 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-08-19T14:12:36.366 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_cluster_log_file_level": "debug", 2024-08-19T14:12:36.366 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_cluster_log_to_file": "true", 2024-08-19T14:12:36.366 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_cluster_log_to_graylog": "false", 2024-08-19T14:12:36.367 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-08-19T14:12:36.367 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-08-19T14:12:36.367 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_cluster_log_to_journald": "true", 2024-08-19T14:12:36.367 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_cluster_log_to_stderr": "false", 2024-08-19T14:12:36.367 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-08-19T14:12:36.367 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-08-19T14:12:36.367 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_cluster_log_to_syslog_level": "info", 2024-08-19T14:12:36.367 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_compact_on_bootstrap": "false", 2024-08-19T14:12:36.367 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_compact_on_start": "false", 2024-08-19T14:12:36.367 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_compact_on_trim": "true", 2024-08-19T14:12:36.367 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_con_tracker_persist_interval": "10", 2024-08-19T14:12:36.367 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_con_tracker_score_halflife": "43200", 2024-08-19T14:12:36.367 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_config_key_max_entry_size": "65536", 2024-08-19T14:12:36.368 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_cpu_threads": "4", 2024-08-19T14:12:36.368 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_crush_min_required_version": "hammer", 2024-08-19T14:12:36.368 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_daemon_bytes": "419430400", 2024-08-19T14:12:36.368 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-08-19T14:12:36.368 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_data_avail_crit": "5", 2024-08-19T14:12:36.368 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_data_avail_warn": "5", 2024-08-19T14:12:36.368 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_data_size_warn": "16106127360", 2024-08-19T14:12:36.368 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_debug_block_osdmap_trim": "false", 2024-08-19T14:12:36.368 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-08-19T14:12:36.368 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_debug_dump_json": "false", 2024-08-19T14:12:36.368 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-08-19T14:12:36.368 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_debug_dump_transactions": "false", 2024-08-19T14:12:36.368 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_debug_extra_checks": "false", 2024-08-19T14:12:36.368 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-08-19T14:12:36.369 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-08-19T14:12:36.369 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_debug_no_require_reef": "false", 2024-08-19T14:12:36.369 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_debug_no_require_squid": "false", 2024-08-19T14:12:36.369 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-08-19T14:12:36.369 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_delta_reset_interval": "10.000000", 2024-08-19T14:12:36.369 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_dns_srv_name": "ceph-mon", 2024-08-19T14:12:36.369 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_down_mkfs_grace": "60", 2024-08-19T14:12:36.369 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_down_uptime_grace": "60", 2024-08-19T14:12:36.369 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_election_default_strategy": "1", 2024-08-19T14:12:36.369 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_election_timeout": "5.000000", 2024-08-19T14:12:36.369 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-08-19T14:12:36.369 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_elector_ping_divisor": "2", 2024-08-19T14:12:36.369 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_elector_ping_timeout": "2.000000", 2024-08-19T14:12:36.370 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_enable_op_tracker": "true", 2024-08-19T14:12:36.370 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_fake_pool_delete": "false", 2024-08-19T14:12:36.370 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_force_quorum_join": "false", 2024-08-19T14:12:36.370 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_fsmap_prune_threshold": "300", 2024-08-19T14:12:36.370 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_globalid_prealloc": "10000", 2024-08-19T14:12:36.370 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_health_detail_to_clog": "true", 2024-08-19T14:12:36.370 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_health_log_update_period": "5", 2024-08-19T14:12:36.370 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_health_max_detail": "50", 2024-08-19T14:12:36.370 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_health_to_clog": "true", 2024-08-19T14:12:36.370 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_health_to_clog_interval": "600", 2024-08-19T14:12:36.370 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-08-19T14:12:36.370 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_host": "[v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0]", 2024-08-19T14:12:36.370 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_host_override": "", 2024-08-19T14:12:36.371 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_initial_members": "", 2024-08-19T14:12:36.371 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-08-19T14:12:36.371 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-08-19T14:12:36.371 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-08-19T14:12:36.371 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-08-19T14:12:36.371 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_keyvaluedb": "rocksdb", 2024-08-19T14:12:36.371 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_lease": "5.000000", 2024-08-19T14:12:36.371 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-08-19T14:12:36.371 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-08-19T14:12:36.371 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_log_full_interval": "50", 2024-08-19T14:12:36.371 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_log_max": "10000", 2024-08-19T14:12:36.371 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_log_max_summary": "50", 2024-08-19T14:12:36.371 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_max_log_entries_per_event": "4096", 2024-08-19T14:12:36.371 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_max_log_epochs": "500", 2024-08-19T14:12:36.372 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_max_mdsmap_epochs": "500", 2024-08-19T14:12:36.372 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_max_mgrmap_epochs": "500", 2024-08-19T14:12:36.372 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_max_osd": "10000", 2024-08-19T14:12:36.372 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_max_pg_per_osd": "10000", 2024-08-19T14:12:36.372 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_max_pool_pg_num": "65536", 2024-08-19T14:12:36.372 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-08-19T14:12:36.372 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-08-19T14:12:36.372 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_mds_force_trim_to": "0", 2024-08-19T14:12:36.372 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_mds_skip_sanity": "false", 2024-08-19T14:12:36.372 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_memory_autotune": "true", 2024-08-19T14:12:36.372 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_memory_target": "2147483648", 2024-08-19T14:12:36.372 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_mgr_beacon_grace": "30", 2024-08-19T14:12:36.372 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-08-19T14:12:36.372 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_mgr_digest_period": "5", 2024-08-19T14:12:36.373 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_mgr_inactive_grace": "60", 2024-08-19T14:12:36.373 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_mgr_mkfs_grace": "240", 2024-08-19T14:12:36.373 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-08-19T14:12:36.373 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_min_osdmap_epochs": "500", 2024-08-19T14:12:36.373 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_op_complaint_time": "30", 2024-08-19T14:12:36.373 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_op_history_duration": "600", 2024-08-19T14:12:36.373 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_op_history_size": "20", 2024-08-19T14:12:36.373 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_op_history_slow_op_size": "20", 2024-08-19T14:12:36.373 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_op_history_slow_op_threshold": "10", 2024-08-19T14:12:36.373 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_op_log_threshold": "5", 2024-08-19T14:12:36.373 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-08-19T14:12:36.373 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-08-19T14:12:36.373 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-08-19T14:12:36.374 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_auto_mark_in": "false", 2024-08-19T14:12:36.374 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_auto_mark_new_in": "true", 2024-08-19T14:12:36.374 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-08-19T14:12:36.374 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-08-19T14:12:36.374 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_cache_size": "500", 2024-08-19T14:12:36.374 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_cache_size_min": "134217728", 2024-08-19T14:12:36.374 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_crush_smoke_test": "true", 2024-08-19T14:12:36.374 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_destroyed_out_interval": "600", 2024-08-19T14:12:36.374 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_down_out_interval": "600", 2024-08-19T14:12:36.374 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-08-19T14:12:36.374 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-08-19T14:12:36.374 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_force_trim_to": "0", 2024-08-19T14:12:36.374 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_full_ratio": "0.950000", 2024-08-19T14:12:36.375 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-08-19T14:12:36.375 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_laggy_halflife": "3600", 2024-08-19T14:12:36.375 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_laggy_max_interval": "300", 2024-08-19T14:12:36.375 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_laggy_weight": "0.300000", 2024-08-19T14:12:36.375 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-08-19T14:12:36.375 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_max_creating_pgs": "1024", 2024-08-19T14:12:36.375 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_max_initial_pgs": "1024", 2024-08-19T14:12:36.375 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_min_down_reporters": "2", 2024-08-19T14:12:36.375 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-08-19T14:12:36.375 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-08-19T14:12:36.375 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-08-19T14:12:36.375 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_prime_pg_temp": "true", 2024-08-19T14:12:36.375 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-08-19T14:12:36.375 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-08-19T14:12:36.376 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_report_timeout": "900", 2024-08-19T14:12:36.376 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-08-19T14:12:36.376 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-08-19T14:12:36.376 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_warn_num_repaired": "10", 2024-08-19T14:12:36.376 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_warn_op_age": "32.000000", 2024-08-19T14:12:36.376 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-08-19T14:12:36.376 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osdmap_full_prune_interval": "10", 2024-08-19T14:12:36.376 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osdmap_full_prune_min": "10000", 2024-08-19T14:12:36.376 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osdmap_full_prune_txsize": "100", 2024-08-19T14:12:36.376 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-08-19T14:12:36.376 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_pg_stuck_threshold": "60", 2024-08-19T14:12:36.376 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-08-19T14:12:36.377 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_pg_warn_min_objects": "10000", 2024-08-19T14:12:36.377 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_pg_warn_min_per_osd": "0", 2024-08-19T14:12:36.377 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-08-19T14:12:36.377 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_pool_quota_crit_threshold": "0", 2024-08-19T14:12:36.377 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_pool_quota_warn_threshold": "0", 2024-08-19T14:12:36.377 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_probe_timeout": "2.000000", 2024-08-19T14:12:36.377 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_reweight_max_change": "0.050000", 2024-08-19T14:12:36.377 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_reweight_max_osds": "4", 2024-08-19T14:12:36.377 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-08-19T14:12:36.377 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-08-19T14:12:36.379 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-08-19T14:12:36.379 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-08-19T14:12:36.379 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-08-19T14:12:36.379 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_scrub_interval": "300", 2024-08-19T14:12:36.379 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_scrub_max_keys": "100", 2024-08-19T14:12:36.379 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_scrub_timeout": "300", 2024-08-19T14:12:36.379 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_session_timeout": "300", 2024-08-19T14:12:36.379 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_smart_report_timeout": "5", 2024-08-19T14:12:36.380 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_stat_smooth_intervals": "6", 2024-08-19T14:12:36.380 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-08-19T14:12:36.380 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_stretch_pool_min_size": "2", 2024-08-19T14:12:36.380 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_stretch_pool_size": "4", 2024-08-19T14:12:36.380 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-08-19T14:12:36.380 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_subscribe_interval": "86400.000000", 2024-08-19T14:12:36.380 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_sync_debug": "false", 2024-08-19T14:12:36.380 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_sync_max_payload_keys": "2000", 2024-08-19T14:12:36.380 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_sync_max_payload_size": "1048576", 2024-08-19T14:12:36.380 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_sync_provider_kill_at": "0", 2024-08-19T14:12:36.381 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_sync_requester_kill_at": "0", 2024-08-19T14:12:36.381 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_sync_timeout": "60.000000", 2024-08-19T14:12:36.381 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_target_pg_per_osd": "100", 2024-08-19T14:12:36.381 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_tick_interval": "5", 2024-08-19T14:12:36.381 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_timecheck_interval": "300.000000", 2024-08-19T14:12:36.381 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-08-19T14:12:36.381 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_use_min_delay_socket": "false", 2024-08-19T14:12:36.381 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_warn_older_version_delay": "604800", 2024-08-19T14:12:36.381 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-08-19T14:12:36.381 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-08-19T14:12:36.381 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-08-19T14:12:36.381 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_warn_on_filestore_osds": "true", 2024-08-19T14:12:36.382 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-08-19T14:12:36.382 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-08-19T14:12:36.382 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-08-19T14:12:36.382 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_warn_on_misplaced": "false", 2024-08-19T14:12:36.382 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-08-19T14:12:36.382 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_warn_on_older_version": "true", 2024-08-19T14:12:36.382 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-08-19T14:12:36.382 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_warn_on_pool_no_app": "true", 2024-08-19T14:12:36.382 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_warn_on_pool_no_redundancy": "true", 2024-08-19T14:12:36.382 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-08-19T14:12:36.382 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-08-19T14:12:36.383 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-08-19T14:12:36.383 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_warn_on_too_few_osds": "false", 2024-08-19T14:12:36.383 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-08-19T14:12:36.383 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-08-19T14:12:36.383 INFO:teuthology.orchestra.run.smithi031.stdout: "monmap": "", 2024-08-19T14:12:36.383 INFO:teuthology.orchestra.run.smithi031.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-08-19T14:12:36.383 INFO:teuthology.orchestra.run.smithi031.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-08-19T14:12:36.383 INFO:teuthology.orchestra.run.smithi031.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-08-19T14:12:36.383 INFO:teuthology.orchestra.run.smithi031.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-08-19T14:12:36.383 INFO:teuthology.orchestra.run.smithi031.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-08-19T14:12:36.384 INFO:teuthology.orchestra.run.smithi031.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-08-19T14:12:36.384 INFO:teuthology.orchestra.run.smithi031.stdout: "motr_tracing_enabled": "false", 2024-08-19T14:12:36.384 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_async_op_threads": "3", 2024-08-19T14:12:36.384 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_async_rdma_buffer_size": "131072", 2024-08-19T14:12:36.384 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_async_rdma_cm": "false", 2024-08-19T14:12:36.384 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_async_rdma_device_name": "", 2024-08-19T14:12:36.384 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_async_rdma_dscp": "96", 2024-08-19T14:12:36.384 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-08-19T14:12:36.384 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_async_rdma_gid_idx": "0", 2024-08-19T14:12:36.384 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_async_rdma_local_gid": "", 2024-08-19T14:12:36.384 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_async_rdma_polling_us": "1000", 2024-08-19T14:12:36.384 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_async_rdma_port_num": "1", 2024-08-19T14:12:36.384 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-08-19T14:12:36.385 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-08-19T14:12:36.385 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_async_rdma_roce_ver": "1", 2024-08-19T14:12:36.385 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_async_rdma_send_buffers": "1024", 2024-08-19T14:12:36.385 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_async_rdma_sl": "3", 2024-08-19T14:12:36.385 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_async_rdma_support_srq": "true", 2024-08-19T14:12:36.385 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_async_rdma_type": "ib", 2024-08-19T14:12:36.385 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_async_reap_threshold": "5", 2024-08-19T14:12:36.385 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_bind_before_connect": "false", 2024-08-19T14:12:36.385 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_bind_ipv4": "true", 2024-08-19T14:12:36.385 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_bind_ipv6": "false", 2024-08-19T14:12:36.385 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_bind_msgr1": "true", 2024-08-19T14:12:36.385 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_bind_msgr2": "true", 2024-08-19T14:12:36.385 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_bind_port_max": "7568", 2024-08-19T14:12:36.386 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_bind_port_min": "6800", 2024-08-19T14:12:36.386 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_bind_prefer_ipv4": "false", 2024-08-19T14:12:36.386 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_bind_retry_count": "3", 2024-08-19T14:12:36.386 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_bind_retry_delay": "5", 2024-08-19T14:12:36.386 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_blackhole_client": "false", 2024-08-19T14:12:36.386 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_blackhole_mds": "false", 2024-08-19T14:12:36.386 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_blackhole_mgr": "false", 2024-08-19T14:12:36.386 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_blackhole_mon": "false", 2024-08-19T14:12:36.386 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_blackhole_osd": "false", 2024-08-19T14:12:36.386 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_client_mode": "crc secure", 2024-08-19T14:12:36.386 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_cluster_mode": "crc secure", 2024-08-19T14:12:36.386 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_cluster_type": "", 2024-08-19T14:12:36.386 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_compress_secure": "false", 2024-08-19T14:12:36.387 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_connection_idle_timeout": "900", 2024-08-19T14:12:36.387 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_connection_ready_timeout": "10", 2024-08-19T14:12:36.387 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_crc_data": "true", 2024-08-19T14:12:36.387 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_crc_header": "true", 2024-08-19T14:12:36.387 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_die_on_bad_msg": "false", 2024-08-19T14:12:36.387 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_die_on_bug": "true", 2024-08-19T14:12:36.387 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_die_on_old_message": "true", 2024-08-19T14:12:36.387 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_die_on_skipped_message": "false", 2024-08-19T14:12:36.387 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_die_on_unhandled_msg": "false", 2024-08-19T14:12:36.387 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-08-19T14:12:36.387 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_dpdk_coremask": "0xF", 2024-08-19T14:12:36.387 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-08-19T14:12:36.387 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_dpdk_devs_allowlist": "", 2024-08-19T14:12:36.387 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_dpdk_enable_tso": "true", 2024-08-19T14:12:36.388 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-08-19T14:12:36.388 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-08-19T14:12:36.388 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_dpdk_hugepages": "", 2024-08-19T14:12:36.388 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_dpdk_hw_flow_control": "true", 2024-08-19T14:12:36.388 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-08-19T14:12:36.388 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_dpdk_lro": "true", 2024-08-19T14:12:36.388 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_dpdk_memory_channel": "4", 2024-08-19T14:12:36.388 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-08-19T14:12:36.388 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_dpdk_pmd": "", 2024-08-19T14:12:36.389 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_dpdk_port_id": "0", 2024-08-19T14:12:36.389 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-08-19T14:12:36.389 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_dump_corrupt_message_level": "1", 2024-08-19T14:12:36.389 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_dump_on_send": "false", 2024-08-19T14:12:36.389 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_initial_backoff": "0.200000", 2024-08-19T14:12:36.389 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_inject_delay_max": "1.000000", 2024-08-19T14:12:36.389 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_inject_delay_probability": "0.000000", 2024-08-19T14:12:36.389 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_inject_delay_type": "", 2024-08-19T14:12:36.389 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_inject_internal_delays": "0.000000", 2024-08-19T14:12:36.389 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_inject_network_congestion": "0", 2024-08-19T14:12:36.389 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_inject_socket_failures": "2500", 2024-08-19T14:12:36.389 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_learn_addr_from_peer": "true", 2024-08-19T14:12:36.389 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_max_accept_failures": "4", 2024-08-19T14:12:36.390 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_max_backoff": "15.000000", 2024-08-19T14:12:36.390 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_mon_client_mode": "secure crc", 2024-08-19T14:12:36.390 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_mon_cluster_mode": "secure crc", 2024-08-19T14:12:36.390 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_mon_service_mode": "secure crc", 2024-08-19T14:12:36.390 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_osd_compress_min_size": "1024", 2024-08-19T14:12:36.390 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_osd_compress_mode": "none", 2024-08-19T14:12:36.390 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_osd_compression_algorithm": "snappy", 2024-08-19T14:12:36.390 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-08-19T14:12:36.390 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_pq_min_cost": "65536", 2024-08-19T14:12:36.390 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_public_type": "", 2024-08-19T14:12:36.390 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_service_mode": "crc secure", 2024-08-19T14:12:36.390 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_tcp_listen_backlog": "512", 2024-08-19T14:12:36.391 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_tcp_nodelay": "true", 2024-08-19T14:12:36.391 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-08-19T14:12:36.391 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_tcp_rcvbuf": "0", 2024-08-19T14:12:36.391 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_type": "async+posix", 2024-08-19T14:12:36.391 INFO:teuthology.orchestra.run.smithi031.stdout: "no_config_file": "false", 2024-08-19T14:12:36.391 INFO:teuthology.orchestra.run.smithi031.stdout: "objecter_completion_locks_per_session": "32", 2024-08-19T14:12:36.391 INFO:teuthology.orchestra.run.smithi031.stdout: "objecter_debug_inject_relock_delay": "false", 2024-08-19T14:12:36.391 INFO:teuthology.orchestra.run.smithi031.stdout: "objecter_inflight_op_bytes": "104857600", 2024-08-19T14:12:36.391 INFO:teuthology.orchestra.run.smithi031.stdout: "objecter_inflight_ops": "1024", 2024-08-19T14:12:36.391 INFO:teuthology.orchestra.run.smithi031.stdout: "objecter_inject_no_watch_ping": "false", 2024-08-19T14:12:36.391 INFO:teuthology.orchestra.run.smithi031.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-08-19T14:12:36.391 INFO:teuthology.orchestra.run.smithi031.stdout: "objecter_tick_interval": "5.000000", 2024-08-19T14:12:36.391 INFO:teuthology.orchestra.run.smithi031.stdout: "objecter_timeout": "10.000000", 2024-08-19T14:12:36.391 INFO:teuthology.orchestra.run.smithi031.stdout: "objectstore_blackhole": "false", 2024-08-19T14:12:36.392 INFO:teuthology.orchestra.run.smithi031.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-08-19T14:12:36.392 INFO:teuthology.orchestra.run.smithi031.stdout: "openssl_engine_opts": "", 2024-08-19T14:12:36.392 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_agent_delay_time": "5.000000", 2024-08-19T14:12:36.392 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_agent_hist_halflife": "1000", 2024-08-19T14:12:36.392 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_agent_max_low_ops": "2", 2024-08-19T14:12:36.392 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_agent_max_ops": "4", 2024-08-19T14:12:36.392 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-08-19T14:12:36.392 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_agent_quantize_effort": "0.100000", 2024-08-19T14:12:36.392 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_agent_slop": "0.020000", 2024-08-19T14:12:36.392 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-08-19T14:12:36.392 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_allow_recovery_below_min_size": "true", 2024-08-19T14:12:36.392 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_asio_thread_count": "2", 2024-08-19T14:12:36.392 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_async_recovery_min_cost": "100", 2024-08-19T14:12:36.392 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_auto_mark_unfound_lost": "false", 2024-08-19T14:12:36.393 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_backfill_retry_interval": "30.000000", 2024-08-19T14:12:36.393 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_backfill_scan_max": "512", 2024-08-19T14:12:36.393 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_backfill_scan_min": "64", 2024-08-19T14:12:36.393 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_backoff_on_degraded": "false", 2024-08-19T14:12:36.393 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_backoff_on_peering": "false", 2024-08-19T14:12:36.393 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_backoff_on_unfound": "true", 2024-08-19T14:12:36.393 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_beacon_report_interval": "300", 2024-08-19T14:12:36.393 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_bench_duration": "30", 2024-08-19T14:12:36.393 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-08-19T14:12:36.393 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_bench_max_block_size": "67108864", 2024-08-19T14:12:36.393 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_bench_small_size_max_iops": "100", 2024-08-19T14:12:36.393 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_blkin_trace_all": "false", 2024-08-19T14:12:36.393 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_blocked_scrub_grace_period": "120", 2024-08-19T14:12:36.393 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-08-19T14:12:36.394 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-08-19T14:12:36.394 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-08-19T14:12:36.394 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_check_for_log_corruption": "false", 2024-08-19T14:12:36.394 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-08-19T14:12:36.394 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-08-19T14:12:36.394 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-08-19T14:12:36.394 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-08-19T14:12:36.394 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_class_update_on_start": "true", 2024-08-19T14:12:36.394 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_client_message_cap": "256", 2024-08-19T14:12:36.394 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_client_message_size_cap": "524288000", 2024-08-19T14:12:36.394 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_client_op_priority": "63", 2024-08-19T14:12:36.394 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_client_watch_timeout": "30", 2024-08-19T14:12:36.394 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_command_max_records": "256", 2024-08-19T14:12:36.395 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_compact_on_start": "false", 2024-08-19T14:12:36.395 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-08-19T14:12:36.395 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_crush_chooseleaf_type": "1", 2024-08-19T14:12:36.395 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_crush_initial_weight": "-1.000000", 2024-08-19T14:12:36.395 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_crush_update_on_start": "true", 2024-08-19T14:12:36.395 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_crush_update_weight_set": "true", 2024-08-19T14:12:36.395 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-08-19T14:12:36.395 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-08-19T14:12:36.395 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-08-19T14:12:36.395 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_debug_drop_ping_duration": "0", 2024-08-19T14:12:36.395 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-08-19T14:12:36.395 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_debug_feed_pullee": "-1", 2024-08-19T14:12:36.395 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-08-19T14:12:36.395 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-08-19T14:12:36.395 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-08-19T14:12:36.396 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_debug_misdirected_ops": "false", 2024-08-19T14:12:36.396 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_debug_no_acting_change": "false", 2024-08-19T14:12:36.396 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_debug_no_purge_strays": "false", 2024-08-19T14:12:36.396 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_debug_op_order": "false", 2024-08-19T14:12:36.396 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_debug_pg_log_writeout": "false", 2024-08-19T14:12:36.396 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_debug_pretend_recovery_active": "false", 2024-08-19T14:12:36.396 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-08-19T14:12:36.396 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-08-19T14:12:36.396 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_debug_shutdown": "false", 2024-08-19T14:12:36.396 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-08-19T14:12:36.396 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-08-19T14:12:36.396 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_debug_trim_objects": "false", 2024-08-19T14:12:36.396 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_debug_verify_cached_snaps": "false", 2024-08-19T14:12:36.397 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_debug_verify_missing_on_start": "false", 2024-08-19T14:12:36.397 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_debug_verify_snaps": "false", 2024-08-19T14:12:36.397 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-08-19T14:12:36.397 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-08-19T14:12:36.397 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_deep_scrub_keys": "1024", 2024-08-19T14:12:36.397 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-08-19T14:12:36.397 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-08-19T14:12:36.397 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-08-19T14:12:36.397 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_deep_scrub_stride": "524288", 2024-08-19T14:12:36.397 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-08-19T14:12:36.397 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_default_data_pool_replay_window": "45", 2024-08-19T14:12:36.397 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_default_notify_timeout": "30", 2024-08-19T14:12:36.397 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_delete_sleep": "0.000000", 2024-08-19T14:12:36.398 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-08-19T14:12:36.398 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-08-19T14:12:36.398 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-08-19T14:12:36.398 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_discard_disconnected_ops": "true", 2024-08-19T14:12:36.398 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_enable_op_tracker": "true", 2024-08-19T14:12:36.398 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-08-19T14:12:36.398 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_extblkdev_plugins": "vdo", 2024-08-19T14:12:36.398 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-08-19T14:12:36.398 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-08-19T14:12:36.398 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_fast_info": "true", 2024-08-19T14:12:36.398 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_fast_shutdown": "true", 2024-08-19T14:12:36.398 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-08-19T14:12:36.399 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_fast_shutdown_timeout": "15", 2024-08-19T14:12:36.399 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-08-19T14:12:36.399 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-08-19T14:12:36.399 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-08-19T14:12:36.399 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_function_tracing": "false", 2024-08-19T14:12:36.399 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_heartbeat_grace": "20", 2024-08-19T14:12:36.399 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_heartbeat_interval": "6", 2024-08-19T14:12:36.399 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-08-19T14:12:36.399 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_heartbeat_min_peers": "10", 2024-08-19T14:12:36.399 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_heartbeat_min_size": "2000", 2024-08-19T14:12:36.399 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_heartbeat_stale": "600", 2024-08-19T14:12:36.399 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-08-19T14:12:36.399 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_hit_set_max_size": "100000", 2024-08-19T14:12:36.399 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_hit_set_min_size": "1000", 2024-08-19T14:12:36.400 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-08-19T14:12:36.400 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-08-19T14:12:36.400 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-08-19T14:12:36.400 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-08-19T14:12:36.400 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-08-19T14:12:36.400 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_journal_flush_on_shutdown": "true", 2024-08-19T14:12:36.400 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_journal_size": "5120", 2024-08-19T14:12:36.400 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_kill_backfill_at": "0", 2024-08-19T14:12:36.400 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_loop_before_reset_tphandle": "64", 2024-08-19T14:12:36.400 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_map_cache_size": "50", 2024-08-19T14:12:36.400 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_map_dedup": "true", 2024-08-19T14:12:36.400 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_map_message_max": "40", 2024-08-19T14:12:36.401 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_map_message_max_bytes": "10485760", 2024-08-19T14:12:36.401 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_map_share_max_epochs": "40", 2024-08-19T14:12:36.401 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_max_attr_name_len": "100", 2024-08-19T14:12:36.401 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_max_attr_size": "0", 2024-08-19T14:12:36.401 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_max_backfills": "1", 2024-08-19T14:12:36.401 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_max_markdown_count": "5", 2024-08-19T14:12:36.401 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_max_markdown_period": "600", 2024-08-19T14:12:36.401 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_max_object_name_len": "2048", 2024-08-19T14:12:36.401 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_max_object_namespace_len": "256", 2024-08-19T14:12:36.401 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_max_object_size": "134217728", 2024-08-19T14:12:36.401 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-08-19T14:12:36.401 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_max_omap_entries_per_request": "1024", 2024-08-19T14:12:36.401 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_max_pg_blocked_by": "16", 2024-08-19T14:12:36.401 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_max_pg_log_entries": "10000", 2024-08-19T14:12:36.402 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-08-19T14:12:36.402 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_max_pgls": "1024", 2024-08-19T14:12:36.402 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_max_push_cost": "8388608", 2024-08-19T14:12:36.402 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_max_push_objects": "10", 2024-08-19T14:12:36.402 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_max_scrubs": "3", 2024-08-19T14:12:36.402 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-08-19T14:12:36.402 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_max_trimming_pgs": "2", 2024-08-19T14:12:36.402 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_max_write_op_reply_len": "64", 2024-08-19T14:12:36.402 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_max_write_size": "90", 2024-08-19T14:12:36.402 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-08-19T14:12:36.402 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-08-19T14:12:36.402 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-08-19T14:12:36.403 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-08-19T14:12:36.403 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-08-19T14:12:36.403 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-08-19T14:12:36.403 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-08-19T14:12:36.403 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mclock_override_recovery_settings": "false", 2024-08-19T14:12:36.403 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mclock_profile": "balanced", 2024-08-19T14:12:36.403 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-08-19T14:12:36.403 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-08-19T14:12:36.403 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-08-19T14:12:36.403 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-08-19T14:12:36.403 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-08-19T14:12:36.403 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-08-19T14:12:36.403 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-08-19T14:12:36.403 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-08-19T14:12:36.404 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-08-19T14:12:36.404 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-08-19T14:12:36.404 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mclock_skip_benchmark": "false", 2024-08-19T14:12:36.404 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_memory_base": "805306368", 2024-08-19T14:12:36.404 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_memory_cache_min": "134217728", 2024-08-19T14:12:36.404 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-08-19T14:12:36.404 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-08-19T14:12:36.404 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_memory_target": "4294967296", 2024-08-19T14:12:36.404 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_memory_target_autotune": "false", 2024-08-19T14:12:36.404 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-08-19T14:12:36.404 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_min_pg_log_entries": "250", 2024-08-19T14:12:36.404 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_min_recovery_priority": "0", 2024-08-19T14:12:36.404 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mon_heartbeat_interval": "30", 2024-08-19T14:12:36.404 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-08-19T14:12:36.404 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mon_report_interval": "5", 2024-08-19T14:12:36.405 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mon_report_max_in_flight": "2", 2024-08-19T14:12:36.405 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-08-19T14:12:36.405 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_num_cache_shards": "32", 2024-08-19T14:12:36.405 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_num_op_tracker_shard": "32", 2024-08-19T14:12:36.405 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_numa_auto_affinity": "true", 2024-08-19T14:12:36.405 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_numa_node": "-1", 2024-08-19T14:12:36.405 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_numa_prefer_iface": "true", 2024-08-19T14:12:36.405 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-08-19T14:12:36.405 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_objecter_finishers": "1", 2024-08-19T14:12:36.405 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_objectstore": "bluestore", 2024-08-19T14:12:36.405 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_objectstore_fuse": "false", 2024-08-19T14:12:36.405 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_objectstore_tracing": "false", 2024-08-19T14:12:36.405 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_op_complaint_time": "30.000000", 2024-08-19T14:12:36.405 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_op_history_duration": "600", 2024-08-19T14:12:36.405 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_op_history_size": "20", 2024-08-19T14:12:36.406 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_op_history_slow_op_size": "20", 2024-08-19T14:12:36.406 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-08-19T14:12:36.406 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_op_log_threshold": "5", 2024-08-19T14:12:36.406 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_op_num_shards": "0", 2024-08-19T14:12:36.406 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_op_num_shards_hdd": "5", 2024-08-19T14:12:36.406 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_op_num_shards_ssd": "8", 2024-08-19T14:12:36.406 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_op_num_threads_per_shard": "0", 2024-08-19T14:12:36.406 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-08-19T14:12:36.406 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-08-19T14:12:36.406 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-08-19T14:12:36.406 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_op_pq_min_cost": "65536", 2024-08-19T14:12:36.406 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_op_queue": "mclock_scheduler", 2024-08-19T14:12:36.406 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_op_queue_cut_off": "high", 2024-08-19T14:12:36.406 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_op_thread_suicide_timeout": "150", 2024-08-19T14:12:36.406 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_op_thread_timeout": "15", 2024-08-19T14:12:36.407 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_open_classes_on_start": "true", 2024-08-19T14:12:36.407 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_os_flags": "0", 2024-08-19T14:12:36.407 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_peering_op_priority": "255", 2024-08-19T14:12:36.407 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pg_delete_cost": "1048576", 2024-08-19T14:12:36.407 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pg_delete_priority": "5", 2024-08-19T14:12:36.407 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-08-19T14:12:36.407 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-08-19T14:12:36.407 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pg_log_dups_tracked": "3000", 2024-08-19T14:12:36.407 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pg_log_trim_max": "10000", 2024-08-19T14:12:36.407 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pg_log_trim_min": "100", 2024-08-19T14:12:36.407 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-08-19T14:12:36.407 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pg_object_context_cache_count": "64", 2024-08-19T14:12:36.408 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-08-19T14:12:36.408 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-08-19T14:12:36.408 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-08-19T14:12:36.408 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-08-19T14:12:36.408 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-08-19T14:12:36.408 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-08-19T14:12:36.408 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-08-19T14:12:36.408 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-08-19T14:12:36.408 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pool_default_crimson": "false", 2024-08-19T14:12:36.408 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pool_default_crush_rule": "-1", 2024-08-19T14:12:36.408 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pool_default_ec_fast_read": "false", 2024-08-19T14:12:36.408 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-08-19T14:12:36.408 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pool_default_flag_bulk": "false", 2024-08-19T14:12:36.409 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-08-19T14:12:36.409 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pool_default_flag_nodelete": "false", 2024-08-19T14:12:36.409 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-08-19T14:12:36.409 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-08-19T14:12:36.409 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pool_default_flags": "0", 2024-08-19T14:12:36.409 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-08-19T14:12:36.409 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pool_default_min_size": "2", 2024-08-19T14:12:36.409 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-08-19T14:12:36.409 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pool_default_pg_num": "32", 2024-08-19T14:12:36.409 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pool_default_pgp_num": "0", 2024-08-19T14:12:36.409 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-08-19T14:12:36.409 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pool_default_read_ratio": "70", 2024-08-19T14:12:36.410 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pool_default_size": "3", 2024-08-19T14:12:36.410 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pool_default_type": "replicated", 2024-08-19T14:12:36.410 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-08-19T14:12:36.410 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pool_use_gmt_hitset": "true", 2024-08-19T14:12:36.410 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_push_per_object_cost": "1000", 2024-08-19T14:12:36.410 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_read_ec_check_for_errors": "false", 2024-08-19T14:12:36.410 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_recover_clone_overlap": "true", 2024-08-19T14:12:36.410 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_recover_clone_overlap_limit": "10", 2024-08-19T14:12:36.410 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_recovery_cost": "20971520", 2024-08-19T14:12:36.410 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_recovery_delay_start": "0.000000", 2024-08-19T14:12:36.410 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_recovery_max_active": "0", 2024-08-19T14:12:36.410 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_recovery_max_active_hdd": "3", 2024-08-19T14:12:36.411 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_recovery_max_active_ssd": "10", 2024-08-19T14:12:36.411 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_recovery_max_chunk": "8388608", 2024-08-19T14:12:36.411 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-08-19T14:12:36.411 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_recovery_max_single_start": "1", 2024-08-19T14:12:36.411 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_recovery_op_priority": "3", 2024-08-19T14:12:36.411 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_recovery_op_warn_multiple": "16", 2024-08-19T14:12:36.411 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_recovery_priority": "5", 2024-08-19T14:12:36.411 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_recovery_retry_interval": "30.000000", 2024-08-19T14:12:36.411 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_recovery_sleep": "0.000000", 2024-08-19T14:12:36.411 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-08-19T14:12:36.411 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-08-19T14:12:36.411 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-08-19T14:12:36.411 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_repair_during_recovery": "false", 2024-08-19T14:12:36.412 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_requested_scrub_priority": "120", 2024-08-19T14:12:36.412 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-08-19T14:12:36.412 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_rollback_to_cluster_snap": "", 2024-08-19T14:12:36.412 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_scrub_auto_repair": "false", 2024-08-19T14:12:36.412 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-08-19T14:12:36.412 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-08-19T14:12:36.412 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_scrub_begin_hour": "0", 2024-08-19T14:12:36.412 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_scrub_begin_week_day": "0", 2024-08-19T14:12:36.412 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_scrub_chunk_max": "25", 2024-08-19T14:12:36.412 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_scrub_chunk_min": "5", 2024-08-19T14:12:36.412 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_scrub_cost": "52428800", 2024-08-19T14:12:36.412 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-08-19T14:12:36.412 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_scrub_during_recovery": "false", 2024-08-19T14:12:36.413 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_scrub_end_hour": "0", 2024-08-19T14:12:36.413 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_scrub_end_week_day": "0", 2024-08-19T14:12:36.413 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_scrub_event_cost": "4096", 2024-08-19T14:12:36.413 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-08-19T14:12:36.413 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-08-19T14:12:36.413 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_scrub_invalid_stats": "true", 2024-08-19T14:12:36.413 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_scrub_load_threshold": "0.500000", 2024-08-19T14:12:36.413 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_scrub_max_interval": "604800.000000", 2024-08-19T14:12:36.413 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_scrub_max_preemptions": "5", 2024-08-19T14:12:36.413 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_scrub_min_interval": "86400.000000", 2024-08-19T14:12:36.413 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_scrub_priority": "5", 2024-08-19T14:12:36.413 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_scrub_sleep": "0.000000", 2024-08-19T14:12:36.414 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-08-19T14:12:36.414 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-08-19T14:12:36.414 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_shutdown_pgref_assert": "false", 2024-08-19T14:12:36.414 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_skip_data_digest": "false", 2024-08-19T14:12:36.414 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_smart_report_timeout": "5", 2024-08-19T14:12:36.414 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_snap_trim_cost": "1048576", 2024-08-19T14:12:36.414 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_snap_trim_priority": "5", 2024-08-19T14:12:36.414 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_snap_trim_sleep": "0.000000", 2024-08-19T14:12:36.414 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-08-19T14:12:36.414 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-08-19T14:12:36.414 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-08-19T14:12:36.414 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-08-19T14:12:36.414 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_stats_update_period_scrubbing": "15", 2024-08-19T14:12:36.415 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-08-19T14:12:36.415 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_target_transaction_size": "30", 2024-08-19T14:12:36.415 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-08-19T14:12:36.415 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-08-19T14:12:36.415 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-08-19T14:12:36.415 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-08-19T14:12:36.415 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-08-19T14:12:36.415 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-08-19T14:12:36.415 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-08-19T14:12:36.415 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_tier_default_cache_mode": "writeback", 2024-08-19T14:12:36.415 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-08-19T14:12:36.415 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-08-19T14:12:36.415 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_tracing": "false", 2024-08-19T14:12:36.415 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_use_stale_snap": "false", 2024-08-19T14:12:36.416 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-08-19T14:12:36.416 INFO:teuthology.orchestra.run.smithi031.stdout: "osdc_blkin_trace_all": "false", 2024-08-19T14:12:36.416 INFO:teuthology.orchestra.run.smithi031.stdout: "paxos_kill_at": "0", 2024-08-19T14:12:36.416 INFO:teuthology.orchestra.run.smithi031.stdout: "paxos_max_join_drift": "10", 2024-08-19T14:12:36.416 INFO:teuthology.orchestra.run.smithi031.stdout: "paxos_min": "500", 2024-08-19T14:12:36.416 INFO:teuthology.orchestra.run.smithi031.stdout: "paxos_min_wait": "0.050000", 2024-08-19T14:12:36.416 INFO:teuthology.orchestra.run.smithi031.stdout: "paxos_propose_interval": "1.000000", 2024-08-19T14:12:36.416 INFO:teuthology.orchestra.run.smithi031.stdout: "paxos_service_trim_max": "500", 2024-08-19T14:12:36.416 INFO:teuthology.orchestra.run.smithi031.stdout: "paxos_service_trim_max_multiplier": "20", 2024-08-19T14:12:36.416 INFO:teuthology.orchestra.run.smithi031.stdout: "paxos_service_trim_min": "250", 2024-08-19T14:12:36.416 INFO:teuthology.orchestra.run.smithi031.stdout: "paxos_stash_full_interval": "25", 2024-08-19T14:12:36.416 INFO:teuthology.orchestra.run.smithi031.stdout: "paxos_trim_max": "500", 2024-08-19T14:12:36.416 INFO:teuthology.orchestra.run.smithi031.stdout: "paxos_trim_min": "250", 2024-08-19T14:12:36.416 INFO:teuthology.orchestra.run.smithi031.stdout: "perf": "true", 2024-08-19T14:12:36.417 INFO:teuthology.orchestra.run.smithi031.stdout: "pid_file": "", 2024-08-19T14:12:36.417 INFO:teuthology.orchestra.run.smithi031.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-08-19T14:12:36.417 INFO:teuthology.orchestra.run.smithi031.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-08-19T14:12:36.417 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "-", 2024-08-19T14:12:36.417 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrv": "", 2024-08-19T14:12:36.417 INFO:teuthology.orchestra.run.smithi031.stdout: "public_bind_addr": "-", 2024-08-19T14:12:36.417 INFO:teuthology.orchestra.run.smithi031.stdout: "public_network": "", 2024-08-19T14:12:36.417 INFO:teuthology.orchestra.run.smithi031.stdout: "public_network_interface": "", 2024-08-19T14:12:36.417 INFO:teuthology.orchestra.run.smithi031.stdout: "qat_compressor_busy_polling": "false", 2024-08-19T14:12:36.417 INFO:teuthology.orchestra.run.smithi031.stdout: "qat_compressor_enabled": "false", 2024-08-19T14:12:36.417 INFO:teuthology.orchestra.run.smithi031.stdout: "qat_compressor_session_max_number": "256", 2024-08-19T14:12:36.417 INFO:teuthology.orchestra.run.smithi031.stdout: "rados_mon_op_timeout": "0", 2024-08-19T14:12:36.417 INFO:teuthology.orchestra.run.smithi031.stdout: "rados_osd_op_timeout": "0", 2024-08-19T14:12:36.417 INFO:teuthology.orchestra.run.smithi031.stdout: "rados_tracing": "false", 2024-08-19T14:12:36.418 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_atime_update_interval": "60", 2024-08-19T14:12:36.418 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-08-19T14:12:36.418 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_balance_parent_reads": "false", 2024-08-19T14:12:36.418 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_balance_snap_reads": "false", 2024-08-19T14:12:36.418 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_blkin_trace_all": "false", 2024-08-19T14:12:36.418 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_blocklist_expire_seconds": "0", 2024-08-19T14:12:36.418 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_blocklist_on_break_lock": "true", 2024-08-19T14:12:36.418 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_cache": "true", 2024-08-19T14:12:36.418 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_cache_block_writes_upfront": "false", 2024-08-19T14:12:36.418 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_cache_max_dirty": "25165824", 2024-08-19T14:12:36.418 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-08-19T14:12:36.418 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_cache_max_dirty_object": "0", 2024-08-19T14:12:36.418 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_cache_policy": "writearound", 2024-08-19T14:12:36.419 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_cache_size": "33554432", 2024-08-19T14:12:36.419 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_cache_target_dirty": "16777216", 2024-08-19T14:12:36.419 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-08-19T14:12:36.419 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_clone_copy_on_read": "false", 2024-08-19T14:12:36.419 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_compression_hint": "none", 2024-08-19T14:12:36.419 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_concurrent_management_ops": "10", 2024-08-19T14:12:36.419 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-08-19T14:12:36.419 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_default_clone_format": "auto", 2024-08-19T14:12:36.419 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_default_data_pool": "", 2024-08-19T14:12:36.419 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_default_features": "61", 2024-08-19T14:12:36.419 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_default_format": "2", 2024-08-19T14:12:36.419 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_default_map_options": "", 2024-08-19T14:12:36.419 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_default_order": "22", 2024-08-19T14:12:36.419 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_default_pool": "rbd", 2024-08-19T14:12:36.420 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-08-19T14:12:36.420 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_default_stripe_count": "0", 2024-08-19T14:12:36.420 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_default_stripe_unit": "0", 2024-08-19T14:12:36.420 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_disable_zero_copy_writes": "true", 2024-08-19T14:12:36.420 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_discard_granularity_bytes": "65536", 2024-08-19T14:12:36.420 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-08-19T14:12:36.420 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_enable_alloc_hint": "true", 2024-08-19T14:12:36.420 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-08-19T14:12:36.420 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_io_scheduler": "simple", 2024-08-19T14:12:36.420 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-08-19T14:12:36.420 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_journal_commit_age": "5.000000", 2024-08-19T14:12:36.420 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-08-19T14:12:36.420 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-08-19T14:12:36.421 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-08-19T14:12:36.421 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-08-19T14:12:36.421 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_journal_object_flush_interval": "0", 2024-08-19T14:12:36.421 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-08-19T14:12:36.421 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-08-19T14:12:36.421 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_journal_order": "24", 2024-08-19T14:12:36.421 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_journal_pool": "", 2024-08-19T14:12:36.421 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_journal_splay_width": "4", 2024-08-19T14:12:36.421 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_localize_parent_reads": "false", 2024-08-19T14:12:36.421 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_localize_snap_reads": "false", 2024-08-19T14:12:36.421 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-08-19T14:12:36.421 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-08-19T14:12:36.421 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-08-19T14:12:36.422 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-08-19T14:12:36.422 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-08-19T14:12:36.422 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-08-19T14:12:36.422 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_mirror_image_policy_type": "simple", 2024-08-19T14:12:36.422 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-08-19T14:12:36.422 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-08-19T14:12:36.422 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-08-19T14:12:36.422 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-08-19T14:12:36.422 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-08-19T14:12:36.422 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-08-19T14:12:36.422 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-08-19T14:12:36.422 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_mirror_memory_autotune": "true", 2024-08-19T14:12:36.422 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_mirror_memory_base": "805306368", 2024-08-19T14:12:36.423 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-08-19T14:12:36.423 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-08-19T14:12:36.423 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-08-19T14:12:36.423 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-08-19T14:12:36.423 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_mirror_memory_target": "4294967296", 2024-08-19T14:12:36.423 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-08-19T14:12:36.423 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-08-19T14:12:36.423 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-08-19T14:12:36.423 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_mirroring_delete_delay": "0", 2024-08-19T14:12:36.423 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-08-19T14:12:36.423 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_mirroring_replay_delay": "0", 2024-08-19T14:12:36.423 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-08-19T14:12:36.423 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-08-19T14:12:36.423 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_move_to_trash_on_remove": "false", 2024-08-19T14:12:36.424 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-08-19T14:12:36.424 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_mtime_update_interval": "60", 2024-08-19T14:12:36.424 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_non_blocking_aio": "true", 2024-08-19T14:12:36.424 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_op_thread_timeout": "60", 2024-08-19T14:12:36.424 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_op_threads": "1", 2024-08-19T14:12:36.424 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_parent_cache_enabled": "false", 2024-08-19T14:12:36.424 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_persistent_cache_mode": "disabled", 2024-08-19T14:12:36.424 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_persistent_cache_path": "/tmp", 2024-08-19T14:12:36.424 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_persistent_cache_size": "1073741824", 2024-08-19T14:12:36.424 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_plugins": "", 2024-08-19T14:12:36.424 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_qos_bps_burst": "0", 2024-08-19T14:12:36.424 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-08-19T14:12:36.424 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_qos_bps_limit": "0", 2024-08-19T14:12:36.425 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_qos_exclude_ops": "0", 2024-08-19T14:12:36.425 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_qos_iops_burst": "0", 2024-08-19T14:12:36.425 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-08-19T14:12:36.425 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_qos_iops_limit": "0", 2024-08-19T14:12:36.425 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_qos_read_bps_burst": "0", 2024-08-19T14:12:36.425 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-08-19T14:12:36.425 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_qos_read_bps_limit": "0", 2024-08-19T14:12:36.425 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_qos_read_iops_burst": "0", 2024-08-19T14:12:36.425 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-08-19T14:12:36.425 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_qos_read_iops_limit": "0", 2024-08-19T14:12:36.425 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_qos_schedule_tick_min": "50", 2024-08-19T14:12:36.425 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_qos_write_bps_burst": "0", 2024-08-19T14:12:36.425 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-08-19T14:12:36.426 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_qos_write_bps_limit": "0", 2024-08-19T14:12:36.426 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_qos_write_iops_burst": "0", 2024-08-19T14:12:36.426 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-08-19T14:12:36.426 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_qos_write_iops_limit": "0", 2024-08-19T14:12:36.426 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_quiesce_notification_attempts": "10", 2024-08-19T14:12:36.426 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_read_from_replica_policy": "default", 2024-08-19T14:12:36.426 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-08-19T14:12:36.426 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_readahead_max_bytes": "524288", 2024-08-19T14:12:36.426 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_readahead_trigger_requests": "10", 2024-08-19T14:12:36.426 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_request_timed_out_seconds": "30", 2024-08-19T14:12:36.426 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_skip_partial_discard": "true", 2024-08-19T14:12:36.426 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-08-19T14:12:36.427 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_tracing": "false", 2024-08-19T14:12:36.427 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_validate_names": "true", 2024-08-19T14:12:36.427 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_validate_pool": "true", 2024-08-19T14:12:36.427 INFO:teuthology.orchestra.run.smithi031.stdout: "restapi_base_url": "", 2024-08-19T14:12:36.427 INFO:teuthology.orchestra.run.smithi031.stdout: "restapi_log_level": "", 2024-08-19T14:12:36.427 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_account_default_quota_max_objects": "-1", 2024-08-19T14:12:36.427 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_account_default_quota_max_size": "-1", 2024-08-19T14:12:36.427 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_acl_grants_max_num": "100", 2024-08-19T14:12:36.427 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_admin_entry": "admin", 2024-08-19T14:12:36.427 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-08-19T14:12:36.427 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_backend_store": "rados", 2024-08-19T14:12:36.427 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_barbican_url": "", 2024-08-19T14:12:36.427 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_beast_enable_async": "true", 2024-08-19T14:12:36.428 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_bucket_counters_cache": "false", 2024-08-19T14:12:36.428 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-08-19T14:12:36.428 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-08-19T14:12:36.428 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-08-19T14:12:36.428 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_bucket_index_max_aio": "128", 2024-08-19T14:12:36.428 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-08-19T14:12:36.428 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-08-19T14:12:36.428 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_bucket_quota_ttl": "600", 2024-08-19T14:12:36.428 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-08-19T14:12:36.428 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_cache_enabled": "true", 2024-08-19T14:12:36.428 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_cache_expiry_interval": "900", 2024-08-19T14:12:36.428 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_cache_lru_size": "25000", 2024-08-19T14:12:36.428 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_config_store": "rados", 2024-08-19T14:12:36.429 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_content_length_compat": "false", 2024-08-19T14:12:36.429 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_copy_obj_progress": "true", 2024-08-19T14:12:36.429 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-08-19T14:12:36.429 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_cors_rules_max_num": "100", 2024-08-19T14:12:36.429 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_cross_domain_policy": "", 2024-08-19T14:12:36.429 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_default_encryption_key": "", 2024-08-19T14:12:36.429 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_kmip_addr": "", 2024-08-19T14:12:36.429 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_kmip_ca_path": "", 2024-08-19T14:12:36.429 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_kmip_client_cert": "", 2024-08-19T14:12:36.429 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_kmip_client_key": "", 2024-08-19T14:12:36.429 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-08-19T14:12:36.429 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_kmip_password": "", 2024-08-19T14:12:36.429 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-08-19T14:12:36.430 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_kmip_username": "", 2024-08-19T14:12:36.430 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_require_ssl": "true", 2024-08-19T14:12:36.430 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-08-19T14:12:36.430 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-08-19T14:12:36.430 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-08-19T14:12:36.430 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-08-19T14:12:36.430 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-08-19T14:12:36.430 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-08-19T14:12:36.430 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-08-19T14:12:36.430 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-08-19T14:12:36.430 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-08-19T14:12:36.430 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-08-19T14:12:36.431 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-08-19T14:12:36.431 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-08-19T14:12:36.431 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-08-19T14:12:36.431 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-08-19T14:12:36.431 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_suppress_logs": "true", 2024-08-19T14:12:36.431 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_vault_addr": "", 2024-08-19T14:12:36.431 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_vault_auth": "token", 2024-08-19T14:12:36.431 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_vault_namespace": "", 2024-08-19T14:12:36.431 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_vault_prefix": "", 2024-08-19T14:12:36.431 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-08-19T14:12:36.431 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-08-19T14:12:36.431 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-08-19T14:12:36.431 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-08-19T14:12:36.432 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_vault_token_file": "", 2024-08-19T14:12:36.432 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-08-19T14:12:36.432 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_curl_buffersize": "524288", 2024-08-19T14:12:36.432 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_curl_low_speed_limit": "1024", 2024-08-19T14:12:36.432 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_curl_low_speed_time": "300", 2024-08-19T14:12:36.432 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_curl_tcp_keepalive": "0", 2024-08-19T14:12:36.432 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-08-19T14:12:36.432 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-08-19T14:12:36.432 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-08-19T14:12:36.432 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-08-19T14:12:36.432 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-08-19T14:12:36.432 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_d3n_l1_fadvise": "4", 2024-08-19T14:12:36.432 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-08-19T14:12:36.432 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-08-19T14:12:36.433 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-08-19T14:12:36.433 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_d4n_host": "127.0.0.1", 2024-08-19T14:12:36.433 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_d4n_port": "6379", 2024-08-19T14:12:36.433 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-08-19T14:12:36.433 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_data_log_changes_size": "1000", 2024-08-19T14:12:36.433 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_data_log_num_shards": "128", 2024-08-19T14:12:36.433 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-08-19T14:12:36.433 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_data_log_window": "30", 2024-08-19T14:12:36.433 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_data_notify_interval_msec": "0", 2024-08-19T14:12:36.433 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_data_sync_poll_interval": "20", 2024-08-19T14:12:36.433 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_data_sync_spawn_window": "20", 2024-08-19T14:12:36.433 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-08-19T14:12:36.433 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-08-19T14:12:36.433 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-08-19T14:12:36.434 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_default_data_log_backing": "fifo", 2024-08-19T14:12:36.434 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-08-19T14:12:36.434 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_default_region_info_oid": "default.region", 2024-08-19T14:12:36.434 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-08-19T14:12:36.434 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-08-19T14:12:36.434 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_defer_to_bucket_acls": "", 2024-08-19T14:12:36.434 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-08-19T14:12:36.434 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-08-19T14:12:36.434 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-08-19T14:12:36.434 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-08-19T14:12:36.434 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-08-19T14:12:36.434 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-08-19T14:12:36.434 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-08-19T14:12:36.434 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-08-19T14:12:36.435 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_dmclock_data_res": "500.000000", 2024-08-19T14:12:36.435 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-08-19T14:12:36.435 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-08-19T14:12:36.435 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-08-19T14:12:36.435 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-08-19T14:12:36.435 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_dns_name": "", 2024-08-19T14:12:36.435 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_dns_s3website_name": "", 2024-08-19T14:12:36.435 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_dynamic_resharding": "true", 2024-08-19T14:12:36.435 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-08-19T14:12:36.435 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_enable_gc_threads": "true", 2024-08-19T14:12:36.435 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_enable_lc_threads": "true", 2024-08-19T14:12:36.435 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_enable_ops_log": "false", 2024-08-19T14:12:36.435 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_enable_quota_threads": "true", 2024-08-19T14:12:36.435 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_enable_static_website": "false", 2024-08-19T14:12:36.436 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_enable_usage_log": "false", 2024-08-19T14:12:36.436 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_enforce_swift_acls": "true", 2024-08-19T14:12:36.436 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_exit_timeout_secs": "120", 2024-08-19T14:12:36.436 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_expose_bucket": "false", 2024-08-19T14:12:36.436 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_extended_http_attrs": "", 2024-08-19T14:12:36.436 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_filter": "none", 2024-08-19T14:12:36.436 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-08-19T14:12:36.436 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_frontends": "beast port=7480", 2024-08-19T14:12:36.436 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_gc_max_concurrent_io": "10", 2024-08-19T14:12:36.436 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_gc_max_deferred": "50", 2024-08-19T14:12:36.436 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-08-19T14:12:36.436 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_gc_max_objs": "32", 2024-08-19T14:12:36.436 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_gc_max_queue_size": "134213632", 2024-08-19T14:12:36.436 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_gc_max_trim_chunk": "16", 2024-08-19T14:12:36.437 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_gc_obj_min_wait": "7200", 2024-08-19T14:12:36.437 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_gc_processor_max_time": "3600", 2024-08-19T14:12:36.437 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_gc_processor_period": "3600", 2024-08-19T14:12:36.437 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-08-19T14:12:36.437 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_get_obj_window_size": "16777216", 2024-08-19T14:12:36.437 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_healthcheck_disabling_path": "", 2024-08-19T14:12:36.437 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_ignore_get_invalid_range": "false", 2024-08-19T14:12:36.437 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_init_timeout": "300", 2024-08-19T14:12:36.437 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-08-19T14:12:36.437 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-08-19T14:12:36.437 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_kafka_connection_idle": "300", 2024-08-19T14:12:36.437 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_kafka_message_timeout": "5000", 2024-08-19T14:12:36.437 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_kafka_sleep_timeout": "10", 2024-08-19T14:12:36.438 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-08-19T14:12:36.438 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-08-19T14:12:36.438 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-08-19T14:12:36.438 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_keystone_admin_domain": "", 2024-08-19T14:12:36.438 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_keystone_admin_password": "", 2024-08-19T14:12:36.438 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_keystone_admin_password_path": "", 2024-08-19T14:12:36.438 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_keystone_admin_project": "", 2024-08-19T14:12:36.438 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_keystone_admin_tenant": "", 2024-08-19T14:12:36.438 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_keystone_admin_token": "", 2024-08-19T14:12:36.438 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_keystone_admin_token_path": "", 2024-08-19T14:12:36.438 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_keystone_admin_user": "", 2024-08-19T14:12:36.438 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_keystone_api_version": "2", 2024-08-19T14:12:36.439 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_keystone_barbican_domain": "", 2024-08-19T14:12:36.439 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_keystone_barbican_password": "", 2024-08-19T14:12:36.439 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_keystone_barbican_project": "", 2024-08-19T14:12:36.439 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_keystone_barbican_tenant": "", 2024-08-19T14:12:36.439 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_keystone_barbican_user": "", 2024-08-19T14:12:36.439 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-08-19T14:12:36.439 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_keystone_implicit_tenants": "false", 2024-08-19T14:12:36.439 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-08-19T14:12:36.439 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_keystone_service_token_enabled": "false", 2024-08-19T14:12:36.439 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_keystone_token_cache_size": "10000", 2024-08-19T14:12:36.439 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_keystone_url": "", 2024-08-19T14:12:36.439 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_keystone_verify_ssl": "true", 2024-08-19T14:12:36.439 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_lc_debug_interval": "-1", 2024-08-19T14:12:36.440 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_lc_lock_max_time": "90", 2024-08-19T14:12:36.440 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_lc_max_objs": "32", 2024-08-19T14:12:36.440 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_lc_max_rules": "1000", 2024-08-19T14:12:36.440 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_lc_max_worker": "3", 2024-08-19T14:12:36.440 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_lc_max_wp_worker": "3", 2024-08-19T14:12:36.440 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_lc_thread_delay": "0", 2024-08-19T14:12:36.440 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-08-19T14:12:36.440 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_ldap_dnattr": "uid", 2024-08-19T14:12:36.440 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-08-19T14:12:36.440 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_ldap_searchfilter": "", 2024-08-19T14:12:36.440 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-08-19T14:12:36.440 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_ldap_uri": "", 2024-08-19T14:12:36.440 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-08-19T14:12:36.441 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-08-19T14:12:36.441 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-08-19T14:12:36.441 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_log_http_headers": "", 2024-08-19T14:12:36.441 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_log_nonexistent_bucket": "false", 2024-08-19T14:12:36.441 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-08-19T14:12:36.441 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_log_object_name_utc": "false", 2024-08-19T14:12:36.441 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-08-19T14:12:36.441 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-08-19T14:12:36.441 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_max_attr_name_len": "0", 2024-08-19T14:12:36.441 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_max_attr_size": "0", 2024-08-19T14:12:36.441 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_max_attrs_num_in_req": "0", 2024-08-19T14:12:36.441 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_max_chunk_size": "4194304", 2024-08-19T14:12:36.441 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_max_concurrent_requests": "1024", 2024-08-19T14:12:36.442 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-08-19T14:12:36.442 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_max_dynamic_shards": "1999", 2024-08-19T14:12:36.442 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_max_listing_results": "1000", 2024-08-19T14:12:36.442 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_max_notify_retries": "10", 2024-08-19T14:12:36.442 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_max_objs_per_shard": "100000", 2024-08-19T14:12:36.442 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_max_put_param_size": "1048576", 2024-08-19T14:12:36.442 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_max_put_size": "5368709120", 2024-08-19T14:12:36.442 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_max_slo_entries": "1000", 2024-08-19T14:12:36.442 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_md_log_max_shards": "64", 2024-08-19T14:12:36.442 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_md_notify_interval_msec": "200", 2024-08-19T14:12:36.442 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_meta_sync_poll_interval": "20", 2024-08-19T14:12:36.442 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_meta_sync_spawn_window": "20", 2024-08-19T14:12:36.443 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-08-19T14:12:36.443 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_mp_lock_max_time": "600", 2024-08-19T14:12:36.443 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-08-19T14:12:36.443 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_multipart_min_part_size": "5242880", 2024-08-19T14:12:36.443 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-08-19T14:12:36.443 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-08-19T14:12:36.443 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_nfs_fhcache_size": "2017", 2024-08-19T14:12:36.443 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-08-19T14:12:36.443 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-08-19T14:12:36.443 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_nfs_lru_lanes": "5", 2024-08-19T14:12:36.443 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_nfs_max_gc": "300", 2024-08-19T14:12:36.443 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-08-19T14:12:36.443 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_nfs_run_gc_threads": "false", 2024-08-19T14:12:36.444 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_nfs_run_lc_threads": "false", 2024-08-19T14:12:36.444 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_nfs_run_quota_threads": "false", 2024-08-19T14:12:36.444 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_nfs_run_sync_thread": "false", 2024-08-19T14:12:36.444 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-08-19T14:12:36.444 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-08-19T14:12:36.444 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_num_async_rados_threads": "32", 2024-08-19T14:12:36.444 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_num_control_oids": "8", 2024-08-19T14:12:36.444 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_numa_node": "-1", 2024-08-19T14:12:36.444 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_obj_stripe_size": "4194304", 2024-08-19T14:12:36.444 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-08-19T14:12:36.444 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_objexp_chunk_size": "100", 2024-08-19T14:12:36.444 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_objexp_gc_interval": "600", 2024-08-19T14:12:36.444 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_objexp_hints_num_shards": "127", 2024-08-19T14:12:36.445 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-08-19T14:12:36.445 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-08-19T14:12:36.445 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_op_thread_timeout": "600", 2024-08-19T14:12:36.445 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_op_tracing": "false", 2024-08-19T14:12:36.445 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_opa_token": "", 2024-08-19T14:12:36.445 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_opa_url": "", 2024-08-19T14:12:36.445 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_opa_verify_ssl": "true", 2024-08-19T14:12:36.445 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-08-19T14:12:36.445 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-08-19T14:12:36.445 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_ops_log_rados": "false", 2024-08-19T14:12:36.445 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_ops_log_socket_path": "", 2024-08-19T14:12:36.445 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-08-19T14:12:36.445 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-08-19T14:12:36.446 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-08-19T14:12:36.446 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_period_push_interval": "2.000000", 2024-08-19T14:12:36.446 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_period_push_interval_max": "30.000000", 2024-08-19T14:12:36.446 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_period_root_pool": ".rgw.root", 2024-08-19T14:12:36.446 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-08-19T14:12:36.446 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-08-19T14:12:36.446 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_posix_cache_lanes": "3", 2024-08-19T14:12:36.446 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-08-19T14:12:36.446 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_posix_cache_max_buckets": "100", 2024-08-19T14:12:36.446 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_posix_cache_partitions": "3", 2024-08-19T14:12:36.446 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-08-19T14:12:36.446 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_print_continue": "true", 2024-08-19T14:12:36.446 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_print_prohibited_content_length": "false", 2024-08-19T14:12:36.447 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-08-19T14:12:36.447 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-08-19T14:12:36.447 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-08-19T14:12:36.447 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-08-19T14:12:36.447 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_rados_tracing": "false", 2024-08-19T14:12:36.447 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_realm": "", 2024-08-19T14:12:36.447 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_realm_id": "", 2024-08-19T14:12:36.447 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-08-19T14:12:36.447 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_region": "", 2024-08-19T14:12:36.447 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_region_root_pool": ".rgw.root", 2024-08-19T14:12:36.447 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_relaxed_region_enforcement": "false", 2024-08-19T14:12:36.447 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-08-19T14:12:36.447 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-08-19T14:12:36.448 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_request_uri": "", 2024-08-19T14:12:36.448 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_reshard_batch_size": "64", 2024-08-19T14:12:36.448 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-08-19T14:12:36.448 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_reshard_max_aio": "128", 2024-08-19T14:12:36.448 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_reshard_num_logs": "16", 2024-08-19T14:12:36.448 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_reshard_thread_interval": "600", 2024-08-19T14:12:36.448 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_resolve_cname": "false", 2024-08-19T14:12:36.448 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_rest_getusage_op_compat": "false", 2024-08-19T14:12:36.448 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_run_sync_thread": "true", 2024-08-19T14:12:36.448 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-08-19T14:12:36.448 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-08-19T14:12:36.448 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_s3_auth_use_keystone": "false", 2024-08-19T14:12:36.448 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_s3_auth_use_ldap": "false", 2024-08-19T14:12:36.448 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_s3_auth_use_rados": "true", 2024-08-19T14:12:36.449 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_s3_auth_use_sts": "false", 2024-08-19T14:12:36.449 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-08-19T14:12:36.449 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_s3_success_create_obj_status": "0", 2024-08-19T14:12:36.449 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-08-19T14:12:36.449 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_scheduler_type": "throttler", 2024-08-19T14:12:36.449 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_script_uri": "", 2024-08-19T14:12:36.449 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_service_provider_name": "", 2024-08-19T14:12:36.449 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-08-19T14:12:36.449 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_sts_client_id": "", 2024-08-19T14:12:36.449 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_sts_client_secret": "", 2024-08-19T14:12:36.449 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_sts_entry": "sts", 2024-08-19T14:12:36.449 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_sts_key": "sts", 2024-08-19T14:12:36.449 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_sts_max_session_duration": "43200", 2024-08-19T14:12:36.449 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_sts_min_session_duration": "900", 2024-08-19T14:12:36.450 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_sts_token_introspection_url": "", 2024-08-19T14:12:36.450 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_swift_account_in_url": "false", 2024-08-19T14:12:36.450 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_swift_auth_entry": "auth", 2024-08-19T14:12:36.450 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_swift_auth_url": "", 2024-08-19T14:12:36.450 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_swift_custom_header": "", 2024-08-19T14:12:36.450 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_swift_enforce_content_length": "false", 2024-08-19T14:12:36.450 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_swift_need_stats": "true", 2024-08-19T14:12:36.450 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_swift_tenant_name": "", 2024-08-19T14:12:36.450 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_swift_token_expiration": "86400", 2024-08-19T14:12:36.450 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_swift_url": "", 2024-08-19T14:12:36.450 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_swift_url_prefix": "swift", 2024-08-19T14:12:36.450 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_swift_versioning_enabled": "false", 2024-08-19T14:12:36.450 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-08-19T14:12:36.451 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-08-19T14:12:36.451 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_sync_lease_period": "120", 2024-08-19T14:12:36.451 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-08-19T14:12:36.451 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_sync_log_trim_interval": "1200", 2024-08-19T14:12:36.451 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-08-19T14:12:36.451 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-08-19T14:12:36.451 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-08-19T14:12:36.451 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_sync_obj_etag_verify": "false", 2024-08-19T14:12:36.451 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_sync_trace_history_size": "4096", 2024-08-19T14:12:36.451 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-08-19T14:12:36.451 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-08-19T14:12:36.451 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_thread_pool_size": "512", 2024-08-19T14:12:36.451 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_topic_persistency_max_retries": "0", 2024-08-19T14:12:36.452 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-08-19T14:12:36.452 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-08-19T14:12:36.452 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_topic_require_publish_policy": "false", 2024-08-19T14:12:36.452 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_torrent_comment": "", 2024-08-19T14:12:36.452 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_torrent_createby": "", 2024-08-19T14:12:36.452 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_torrent_encoding": "", 2024-08-19T14:12:36.452 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_torrent_flag": "false", 2024-08-19T14:12:36.452 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_torrent_max_size": "5368709120", 2024-08-19T14:12:36.452 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_torrent_origin": "", 2024-08-19T14:12:36.452 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_torrent_sha_unit": "524288", 2024-08-19T14:12:36.452 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_torrent_tracker": "", 2024-08-19T14:12:36.452 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_trust_forwarded_https": "false", 2024-08-19T14:12:36.452 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-08-19T14:12:36.452 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_usage_log_tick_interval": "30", 2024-08-19T14:12:36.453 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_usage_max_shards": "32", 2024-08-19T14:12:36.453 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_usage_max_user_shards": "1", 2024-08-19T14:12:36.453 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_use_opa_authz": "false", 2024-08-19T14:12:36.453 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_user_counters_cache": "false", 2024-08-19T14:12:36.453 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_user_counters_cache_size": "10000", 2024-08-19T14:12:36.453 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-08-19T14:12:36.453 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_user_default_quota_max_size": "-1", 2024-08-19T14:12:36.453 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_user_max_buckets": "1000", 2024-08-19T14:12:36.453 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_user_policies_max_num": "100", 2024-08-19T14:12:36.453 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-08-19T14:12:36.453 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-08-19T14:12:36.453 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_user_quota_sync_interval": "86400", 2024-08-19T14:12:36.453 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-08-19T14:12:36.453 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_user_unique_email": "true", 2024-08-19T14:12:36.454 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_verify_ssl": "true", 2024-08-19T14:12:36.454 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_website_routing_rules_max_num": "50", 2024-08-19T14:12:36.454 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_zone": "", 2024-08-19T14:12:36.454 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_zone_id": "", 2024-08-19T14:12:36.454 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-08-19T14:12:36.454 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_zonegroup": "", 2024-08-19T14:12:36.454 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_zonegroup_id": "", 2024-08-19T14:12:36.454 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-08-19T14:12:36.454 INFO:teuthology.orchestra.run.smithi031.stdout: "rgwlc_auto_session_clear": "true", 2024-08-19T14:12:36.454 INFO:teuthology.orchestra.run.smithi031.stdout: "rgwlc_skip_bucket_step": "false", 2024-08-19T14:12:36.454 INFO:teuthology.orchestra.run.smithi031.stdout: "rocksdb_block_size": "4096", 2024-08-19T14:12:36.454 INFO:teuthology.orchestra.run.smithi031.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-08-19T14:12:36.455 INFO:teuthology.orchestra.run.smithi031.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-08-19T14:12:36.455 INFO:teuthology.orchestra.run.smithi031.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-08-19T14:12:36.455 INFO:teuthology.orchestra.run.smithi031.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-08-19T14:12:36.455 INFO:teuthology.orchestra.run.smithi031.stdout: "rocksdb_cache_shard_bits": "4", 2024-08-19T14:12:36.455 INFO:teuthology.orchestra.run.smithi031.stdout: "rocksdb_cache_size": "536870912", 2024-08-19T14:12:36.455 INFO:teuthology.orchestra.run.smithi031.stdout: "rocksdb_cache_type": "binned_lru", 2024-08-19T14:12:36.455 INFO:teuthology.orchestra.run.smithi031.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-08-19T14:12:36.455 INFO:teuthology.orchestra.run.smithi031.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-08-19T14:12:36.455 INFO:teuthology.orchestra.run.smithi031.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-08-19T14:12:36.455 INFO:teuthology.orchestra.run.smithi031.stdout: "rocksdb_collect_compaction_stats": "false", 2024-08-19T14:12:36.455 INFO:teuthology.orchestra.run.smithi031.stdout: "rocksdb_collect_extended_stats": "false", 2024-08-19T14:12:36.455 INFO:teuthology.orchestra.run.smithi031.stdout: "rocksdb_collect_memory_stats": "false", 2024-08-19T14:12:36.455 INFO:teuthology.orchestra.run.smithi031.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-08-19T14:12:36.455 INFO:teuthology.orchestra.run.smithi031.stdout: "rocksdb_index_type": "binary_search", 2024-08-19T14:12:36.456 INFO:teuthology.orchestra.run.smithi031.stdout: "rocksdb_log_to_ceph_log": "true", 2024-08-19T14:12:36.456 INFO:teuthology.orchestra.run.smithi031.stdout: "rocksdb_metadata_block_size": "4096", 2024-08-19T14:12:36.456 INFO:teuthology.orchestra.run.smithi031.stdout: "rocksdb_partition_filters": "false", 2024-08-19T14:12:36.456 INFO:teuthology.orchestra.run.smithi031.stdout: "rocksdb_perf": "false", 2024-08-19T14:12:36.456 INFO:teuthology.orchestra.run.smithi031.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-08-19T14:12:36.456 INFO:teuthology.orchestra.run.smithi031.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-08-19T14:12:36.456 INFO:teuthology.orchestra.run.smithi031.stdout: "rotating_keys_renewal_timeout": "10", 2024-08-19T14:12:36.456 INFO:teuthology.orchestra.run.smithi031.stdout: "run_dir": "/var/run/ceph", 2024-08-19T14:12:36.456 INFO:teuthology.orchestra.run.smithi031.stdout: "seastore_block_create": "true", 2024-08-19T14:12:36.456 INFO:teuthology.orchestra.run.smithi031.stdout: "seastore_cache_lru_size": "67108864", 2024-08-19T14:12:36.456 INFO:teuthology.orchestra.run.smithi031.stdout: "seastore_cbjournal_size": "5368709120", 2024-08-19T14:12:36.456 INFO:teuthology.orchestra.run.smithi031.stdout: "seastore_data_delta_based_overwrite": "0", 2024-08-19T14:12:36.456 INFO:teuthology.orchestra.run.smithi031.stdout: "seastore_default_max_object_size": "16777216", 2024-08-19T14:12:36.456 INFO:teuthology.orchestra.run.smithi031.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-08-19T14:12:36.457 INFO:teuthology.orchestra.run.smithi031.stdout: "seastore_device_size": "53687091200", 2024-08-19T14:12:36.457 INFO:teuthology.orchestra.run.smithi031.stdout: "seastore_full_integrity_check": "false", 2024-08-19T14:12:36.457 INFO:teuthology.orchestra.run.smithi031.stdout: "seastore_journal_batch_capacity": "16", 2024-08-19T14:12:36.457 INFO:teuthology.orchestra.run.smithi031.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-08-19T14:12:36.457 INFO:teuthology.orchestra.run.smithi031.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-08-19T14:12:36.457 INFO:teuthology.orchestra.run.smithi031.stdout: "seastore_journal_iodepth_limit": "5", 2024-08-19T14:12:36.457 INFO:teuthology.orchestra.run.smithi031.stdout: "seastore_main_device_type": "SSD", 2024-08-19T14:12:36.457 INFO:teuthology.orchestra.run.smithi031.stdout: "seastore_max_concurrent_transactions": "8", 2024-08-19T14:12:36.457 INFO:teuthology.orchestra.run.smithi031.stdout: "seastore_max_data_allocation_size": "32768", 2024-08-19T14:12:36.457 INFO:teuthology.orchestra.run.smithi031.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-08-19T14:12:36.457 INFO:teuthology.orchestra.run.smithi031.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-08-19T14:12:36.457 INFO:teuthology.orchestra.run.smithi031.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-08-19T14:12:36.457 INFO:teuthology.orchestra.run.smithi031.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-08-19T14:12:36.457 INFO:teuthology.orchestra.run.smithi031.stdout: "seastore_segment_size": "67108864", 2024-08-19T14:12:36.458 INFO:teuthology.orchestra.run.smithi031.stdout: "set_keepcaps": "false", 2024-08-19T14:12:36.458 INFO:teuthology.orchestra.run.smithi031.stdout: "setgroup": "ceph", 2024-08-19T14:12:36.458 INFO:teuthology.orchestra.run.smithi031.stdout: "setuser": "ceph", 2024-08-19T14:12:36.458 INFO:teuthology.orchestra.run.smithi031.stdout: "setuser_match_path": "", 2024-08-19T14:12:36.458 INFO:teuthology.orchestra.run.smithi031.stdout: "target_max_misplaced_ratio": "0.050000", 2024-08-19T14:12:36.458 INFO:teuthology.orchestra.run.smithi031.stdout: "thp": "false", 2024-08-19T14:12:36.458 INFO:teuthology.orchestra.run.smithi031.stdout: "threadpool_default_timeout": "60", 2024-08-19T14:12:36.458 INFO:teuthology.orchestra.run.smithi031.stdout: "threadpool_empty_queue_max_wait": "2", 2024-08-19T14:12:36.458 INFO:teuthology.orchestra.run.smithi031.stdout: "throttler_perf_counter": "true" 2024-08-19T14:12:36.458 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:12:37.140 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-08-19T14:12:37.388 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:12:37.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:37 smithi031 ceph-mon[16336]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:12:37.815 INFO:teuthology.orchestra.run.smithi031.stdout:mon.a: {} 2024-08-19T14:12:37.815 INFO:teuthology.orchestra.run.smithi031.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-08-19T14:12:37.816 INFO:teuthology.orchestra.run.smithi031.stdout:mon.c: {} 2024-08-19T14:12:37.816 INFO:teuthology.orchestra.run.smithi031.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-08-19T14:12:37.825 INFO:teuthology.orchestra.run.smithi031.stdout:mon.b: {} 2024-08-19T14:12:37.826 INFO:teuthology.orchestra.run.smithi031.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-08-19T14:12:38.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:37 smithi032 ceph-mon[20004]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:12:38.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:37 smithi045 ceph-mon[20143]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:12:38.514 INFO:teuthology.run_tasks:Running task full_sequential... 2024-08-19T14:12:38.524 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-08-19T14:12:38.526 INFO:tasks.radosbench:Beginning radosbench... 2024-08-19T14:12:38.526 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_0 2024-08-19T14:12:38.526 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd pool create unique_pool_0 16 2024-08-19T14:12:38.527 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-08-19T14:12:38.527 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-08-19T14:12:38.527 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-19T14:12:38.527 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2024-08-19T14:12:38.527 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-08-19T14:12:38.527 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-08-19T14:12:38.528 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.4 config set filestore_inject_stall 3 2024-08-19T14:12:38.528 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-08-19T14:12:38.528 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:12:38.529 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-08-19T14:12:38.529 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:12:38.530 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-08-19T14:12:38.530 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-08-19T14:12:38.530 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-08-19T14:12:38.530 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-19T14:12:38.557 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:12:38.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:38.558 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:12:38.659 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:12:38.685 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:12:38.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:38.686 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:12:38.787 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:12:38.798 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:12:38.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:38.812 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:12:38.813 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:12:38.837 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:12:38.838 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:12:38.838 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:12:38.846 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:12:38.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:38 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2307199358' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-08-19T14:12:38.914 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:12:38.919 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:38 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2307199358' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-08-19T14:12:38.938 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:12:38.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:38.939 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:12:39.040 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:12:39.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:38 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2307199358' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-08-19T14:12:39.062 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:12:39.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:39.063 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:12:39.164 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:12:39.186 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:12:39.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:39.187 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:12:39.288 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:12:39.309 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:12:39.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:39.310 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:12:39.410 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:12:39.410 INFO:teuthology.orchestra.run.smithi032.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-08-19T14:12:39.410 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:12:39.411 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:12:39.434 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:12:39.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:39.435 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:12:39.536 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:12:39.556 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:12:39.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:39.557 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:12:39.658 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:12:39.683 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:12:39.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:39.684 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:12:39.785 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:12:39.802 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:39 smithi031 ceph-mon[16336]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:12:39.808 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:12:39.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:39.809 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:12:39.814 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:12:39.815 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-08-19T14:12:39.815 INFO:teuthology.orchestra.run.smithi031.stdout: "num_ops": 0 2024-08-19T14:12:39.815 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:12:39.910 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:12:39.935 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:12:39.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:39.936 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:12:40.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:39 smithi032 ceph-mon[20004]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:12:40.037 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:12:40.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:39 smithi045 ceph-mon[20143]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:12:40.057 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:12:40.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:40.058 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:12:40.159 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:12:40.181 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:12:40.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:40.182 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:12:40.283 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:12:40.304 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:12:40.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:40.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:12:40.406 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:12:40.431 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:12:40.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:40.432 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:12:40.533 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:12:40.553 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-08-19T14:12:40.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:40.556 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:12:40.556 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:12:40.656 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:12:40.682 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:12:40.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:40.683 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:12:40.783 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:12:40.806 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:12:40.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:40.807 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:12:40.834 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:40 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:12:40.834 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:40 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:12:40.834 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:40 smithi031 ceph-mon[16336]: osdmap e101: 12 total, 12 up, 12 in 2024-08-19T14:12:40.908 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:12:40.934 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:12:40.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:40.935 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:12:41.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:40 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:12:41.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:40 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:12:41.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:40 smithi032 ceph-mon[20004]: osdmap e101: 12 total, 12 up, 12 in 2024-08-19T14:12:41.036 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:12:41.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:40 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:12:41.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:40 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:12:41.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:40 smithi045 ceph-mon[20143]: osdmap e101: 12 total, 12 up, 12 in 2024-08-19T14:12:41.062 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:12:41.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:41.063 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:12:41.164 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:12:41.186 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:12:41.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:41.187 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:12:41.287 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:12:41.310 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:12:41.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:41.311 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:12:41.312 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:12:41.312 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:12:41.334 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:12:41.334 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:12:41.352 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:12:41.352 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:12:41.357 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:12:41.369 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:12:41.369 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:12:41.391 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:12:41.392 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:12:41.408 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:12:41.408 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:12:41.412 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:12:41.437 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:12:41.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:41.437 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:12:41.538 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:12:41.561 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:12:41.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:41.562 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:12:41.626 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:12:41.626 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:12:41.642 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:12:41.642 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:12:41.660 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:12:41.660 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:12:41.662 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:12:41.677 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:12:41.677 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:12:41.683 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:12:41.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:41.684 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:12:41.695 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:12:41.696 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:12:41.713 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:12:41.713 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:12:41.720 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'unique_pool_0' created 2024-08-19T14:12:41.722 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is set 2024-08-19T14:12:41.738 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:41 smithi031 ceph-mon[16336]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:12:41.738 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:41 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/753360257' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-08-19T14:12:41.738 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:41 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/4238334549' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:12:41.738 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:41 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:12:41.738 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:41 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3207460157' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:12:41.785 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:12:41.795 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:12:41.795 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:12:41.806 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:12:41.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:41.807 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:12:41.908 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:12:41.934 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:12:41.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:41.935 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:12:42.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:41 smithi032 ceph-mon[20004]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:12:42.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:41 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/753360257' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-08-19T14:12:42.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:41 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/4238334549' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:12:42.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:41 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:12:42.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:41 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3207460157' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:12:42.035 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:12:42.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:41 smithi045 ceph-mon[20143]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:12:42.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:41 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/753360257' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-08-19T14:12:42.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:41 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/4238334549' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:12:42.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:41 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:12:42.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:41 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3207460157' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:12:42.060 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:12:42.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:42.061 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:12:42.162 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:12:42.187 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:12:42.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:42.188 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:12:42.289 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:12:42.311 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:12:42.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:42.312 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:12:42.413 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:12:42.435 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:12:42.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:42.436 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:12:42.537 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:12:42.563 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:12:42.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:42.564 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:12:42.665 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:12:42.687 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:12:42.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:42.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:12:42.789 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:12:42.811 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:12:42.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:42.812 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:12:42.913 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:12:43.157 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:42 smithi031 ceph-mon[16336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:12:43.157 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:42 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/753360257' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-08-19T14:12:43.157 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:42 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:12:43.157 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:42 smithi031 ceph-mon[16336]: osdmap e102: 12 total, 12 up, 12 in 2024-08-19T14:12:43.158 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-08-19T14:12:43.159 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:42 smithi045 ceph-mon[20143]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:12:43.159 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:42 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/753360257' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-08-19T14:12:43.159 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:42 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:12:43.159 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:42 smithi045 ceph-mon[20143]: osdmap e102: 12 total, 12 up, 12 in 2024-08-19T14:12:43.159 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:42 smithi032 ceph-mon[20004]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:12:43.159 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:42 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/753360257' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-08-19T14:12:43.159 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:42 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:12:43.160 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:42 smithi032 ceph-mon[20004]: osdmap e102: 12 total, 12 up, 12 in 2024-08-19T14:12:43.184 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:12:43.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:43.185 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:12:43.286 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:12:43.307 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:12:43.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:43.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:12:43.409 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:12:43.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:43.431 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:12:43.432 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:12:43.533 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:12:43.558 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:12:43.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:43.559 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:12:43.660 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:12:43.681 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:12:43.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:43.682 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:12:43.783 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:12:43.808 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:12:43.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:43.809 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:12:43.909 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:12:43.930 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:12:43.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:43.931 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:12:43.936 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:43 smithi031 ceph-mon[16336]: pgmap v286: 17 pgs: 16 unknown, 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:12:43.936 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:43 smithi031 ceph-mon[16336]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-19T14:12:43.936 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:43 smithi031 ceph-mon[16336]: osdmap e103: 12 total, 12 up, 12 in 2024-08-19T14:12:44.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:43 smithi032 ceph-mon[20004]: pgmap v286: 17 pgs: 16 unknown, 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:12:44.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:43 smithi032 ceph-mon[20004]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-19T14:12:44.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:43 smithi032 ceph-mon[20004]: osdmap e103: 12 total, 12 up, 12 in 2024-08-19T14:12:44.032 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:12:44.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:43 smithi045 ceph-mon[20143]: pgmap v286: 17 pgs: 16 unknown, 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:12:44.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:43 smithi045 ceph-mon[20143]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-19T14:12:44.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:43 smithi045 ceph-mon[20143]: osdmap e103: 12 total, 12 up, 12 in 2024-08-19T14:12:44.057 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:12:44.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:44.058 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:12:44.159 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:12:44.184 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:12:44.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:44.185 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:12:44.286 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:12:44.312 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:12:44.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:44.313 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:12:44.331 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-08-19T14:12:44.413 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:12:44.436 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:12:44.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:44.438 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:12:44.439 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:12:44.460 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:12:44.539 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:12:44.564 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:12:44.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:44.565 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:12:44.666 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:12:44.692 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:12:44.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:44.693 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:12:44.794 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:12:44.820 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:12:44.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:44.821 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:12:44.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:44 smithi031 ceph-mon[16336]: osdmap e104: 12 total, 12 up, 12 in 2024-08-19T14:12:44.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:44 smithi031 ceph-mon[16336]: pgmap v289: 17 pgs: 2 active+clean, 2 creating+activating, 3 creating+peering, 10 unknown; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:12:44.906 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:12:44.907 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:12:44.921 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:12:44.946 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:12:44.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:44.947 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:12:45.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:44 smithi032 ceph-mon[20004]: osdmap e104: 12 total, 12 up, 12 in 2024-08-19T14:12:45.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:44 smithi032 ceph-mon[20004]: pgmap v289: 17 pgs: 2 active+clean, 2 creating+activating, 3 creating+peering, 10 unknown; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:12:45.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:44 smithi045 ceph-mon[20143]: osdmap e104: 12 total, 12 up, 12 in 2024-08-19T14:12:45.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:44 smithi045 ceph-mon[20143]: pgmap v289: 17 pgs: 2 active+clean, 2 creating+activating, 3 creating+peering, 10 unknown; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:12:45.048 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:12:45.066 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-08-19T14:12:45.066 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-19T14:12:45.067 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-08-19T14:12:45.067 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-08-19T14:12:45.067 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-08-19T14:12:45.067 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.5 config set filestore_inject_stall 3 2024-08-19T14:12:45.073 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:12:45.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:45.074 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:12:45.175 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:12:45.197 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:12:45.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:45.198 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:12:45.299 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:12:45.319 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:12:45.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:45.320 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:12:45.334 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:12:45.421 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:12:45.446 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:12:45.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:45.447 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:12:45.495 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-08-19T14:12:45.548 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:12:45.562 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:12:45.562 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:12:45.570 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:12:45.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:45.571 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:12:45.643 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-19T14:12:45.672 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:12:45.694 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:12:45.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:45.695 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:12:45.796 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:12:45.807 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:12:45.808 INFO:teuthology.orchestra.run.smithi032.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-08-19T14:12:45.808 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:12:45.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:45.821 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:12:45.821 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:12:45.922 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:12:45.943 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:12:45.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:45.944 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:12:46.045 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:12:46.070 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:12:46.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:46.071 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:12:46.172 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:12:46.196 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:12:46.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:46.197 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:12:46.298 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:12:46.320 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:12:46.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:46.321 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:12:46.422 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:12:46.448 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:12:46.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:46.449 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:12:46.515 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:12:46.515 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:12:46.515 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:12:46.516 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:12:46.534 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:12:46.534 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:12:46.550 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:12:46.553 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:12:46.553 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:12:46.570 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:12:46.570 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:12:46.572 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:12:46.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:46.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:12:46.586 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:12:46.587 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:12:46.603 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:12:46.603 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:12:46.620 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:12:46.620 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:12:46.637 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:12:46.637 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:12:46.654 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:12:46.654 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:12:46.672 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:12:46.673 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:12:46.674 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:12:46.690 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:12:46.690 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:12:46.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:46.697 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:12:46.698 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:12:46.707 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:12:46.707 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:12:46.795 INFO:teuthology.orchestra.run.smithi031.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-08-19T14:12:46.798 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:12:46.820 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:12:46.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:46.821 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:12:46.922 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:12:46.945 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:12:46.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:46.946 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:12:47.036 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:46 smithi031 ceph-mon[16336]: pgmap v290: 17 pgs: 10 active+clean, 2 creating+activating, 3 creating+peering, 2 unknown; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:12:47.036 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:46 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2405960911' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-08-19T14:12:47.037 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:46 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/978084492' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:12:47.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:46 smithi045 ceph-mon[20143]: pgmap v290: 17 pgs: 10 active+clean, 2 creating+activating, 3 creating+peering, 2 unknown; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:12:47.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:46 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2405960911' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-08-19T14:12:47.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:46 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/978084492' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:12:47.047 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:12:47.067 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:12:47.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:47.068 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:12:47.169 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:12:47.195 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:12:47.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:47.196 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:12:47.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:46 smithi032 ceph-mon[20004]: pgmap v290: 17 pgs: 10 active+clean, 2 creating+activating, 3 creating+peering, 2 unknown; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:12:47.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:46 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2405960911' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-08-19T14:12:47.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:46 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/978084492' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:12:47.296 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:12:47.321 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:12:47.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:47.322 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:12:47.423 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:12:47.445 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:12:47.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:47.446 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:12:47.547 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:12:47.573 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:12:47.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:47.574 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:12:47.642 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:12:47.642 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-08-19T14:12:47.642 INFO:teuthology.orchestra.run.smithi031.stdout: "num_ops": 0 2024-08-19T14:12:47.642 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:12:47.675 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:12:47.700 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:12:47.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:47.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:12:47.802 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:12:47.827 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:12:47.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:47.828 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:12:47.929 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:12:47.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:47.952 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:12:47.952 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:12:48.054 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:12:48.080 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:12:48.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:48.081 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:12:48.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:47 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2405960911' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-08-19T14:12:48.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:47 smithi031 ceph-mon[16336]: osdmap e105: 12 total, 12 up, 12 in 2024-08-19T14:12:48.182 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:12:48.207 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:12:48.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:48.208 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:12:48.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:47 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2405960911' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-08-19T14:12:48.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:47 smithi032 ceph-mon[20004]: osdmap e105: 12 total, 12 up, 12 in 2024-08-19T14:12:48.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:47 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2405960911' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-08-19T14:12:48.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:47 smithi045 ceph-mon[20143]: osdmap e105: 12 total, 12 up, 12 in 2024-08-19T14:12:48.310 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:12:48.335 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:12:48.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:48.336 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:12:48.337 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-08-19T14:12:48.437 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:12:48.462 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:12:48.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:48.463 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:12:48.564 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:12:48.590 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:12:48.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:48.591 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:12:48.692 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:12:48.717 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:12:48.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:48.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:12:48.819 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:12:48.844 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:12:48.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:48.845 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:12:48.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:48 smithi031 ceph-mon[16336]: pgmap v292: 17 pgs: 15 active+clean, 2 creating+activating; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:12:48.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:48 smithi031 ceph-mon[16336]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-19T14:12:48.947 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:12:48.971 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:12:48.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:48.972 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:12:49.073 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:12:49.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:49.099 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:12:49.099 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:12:49.200 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:12:49.226 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:12:49.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:49.227 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:12:49.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:48 smithi032 ceph-mon[20004]: pgmap v292: 17 pgs: 15 active+clean, 2 creating+activating; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:12:49.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:48 smithi032 ceph-mon[20004]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-19T14:12:49.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:48 smithi045 ceph-mon[20143]: pgmap v292: 17 pgs: 15 active+clean, 2 creating+activating; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:12:49.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:48 smithi045 ceph-mon[20143]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-19T14:12:49.329 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:12:49.354 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:12:49.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:49.355 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:12:49.455 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:12:49.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:49.482 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:12:49.482 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:12:49.568 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:12:49.583 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:12:49.605 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:12:49.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:49.606 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:12:49.707 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:12:49.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:49.728 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:12:49.729 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:12:49.767 DEBUG:teuthology.orchestra.run.smithi045:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_0 bench 90 write' 2024-08-19T14:12:49.770 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-08-19T14:12:49.770 INFO:teuthology.orchestra.run:waiting for 3000 2024-08-19T14:12:49.812 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is set 2024-08-19T14:12:49.829 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:hints = 1 2024-08-19T14:12:49.829 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-08-19T14:12:49.829 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Object prefix: benchmark_data_smithi045_46415 2024-08-19T14:12:49.829 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:12:49.830 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-19T14:12:49.830 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 0 0 0 0 0 0 - 0 2024-08-19T14:12:49.857 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:12:49.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:49.858 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:12:49.959 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:12:49.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:49.998 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:12:49.998 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:12:50.032 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:49 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1691617504' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:12:50.099 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:12:50.102 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:12:50.102 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-08-19T14:12:50.102 INFO:teuthology.orchestra.run.smithi031.stdout: "complaint_time": 30, 2024-08-19T14:12:50.102 INFO:teuthology.orchestra.run.smithi031.stdout: "num_blocked_ops": 0 2024-08-19T14:12:50.102 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:12:50.134 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:12:50.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:50.135 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:12:50.236 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:12:50.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:49 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1691617504' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:12:50.268 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:12:50.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:50.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:12:50.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:49 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1691617504' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:12:50.370 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:12:50.403 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:12:50.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:50.405 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:12:50.506 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:12:50.539 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:12:50.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:50.541 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:12:50.642 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:12:50.676 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:12:50.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:50.677 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:12:50.778 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:12:50.815 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:12:50.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:50.816 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:12:50.829 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 1 16 4625 4609 288.046 288.062 0.00224303 0.00346135 2024-08-19T14:12:50.917 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:12:50.945 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:12:50.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:50.946 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:12:51.048 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:12:51.084 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:12:51.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:51.085 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:12:51.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:50 smithi031 ceph-mon[16336]: pgmap v293: 17 pgs: 17 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:12:51.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:50 smithi031 ceph-mon[16336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:12:51.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:50 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1691617504' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:12:51.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:50 smithi031 ceph-mon[16336]: osdmap e106: 12 total, 12 up, 12 in 2024-08-19T14:12:51.185 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:12:51.223 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:12:51.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:51.224 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:12:51.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:50 smithi032 ceph-mon[20004]: pgmap v293: 17 pgs: 17 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:12:51.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:50 smithi032 ceph-mon[20004]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:12:51.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:50 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1691617504' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:12:51.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:50 smithi032 ceph-mon[20004]: osdmap e106: 12 total, 12 up, 12 in 2024-08-19T14:12:51.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:50 smithi045 ceph-mon[20143]: pgmap v293: 17 pgs: 17 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:12:51.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:50 smithi045 ceph-mon[20143]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:12:51.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:50 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1691617504' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:12:51.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:50 smithi045 ceph-mon[20143]: osdmap e106: 12 total, 12 up, 12 in 2024-08-19T14:12:51.326 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:12:51.359 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:12:51.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:51.360 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:12:51.394 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:12:51.431 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-08-19T14:12:51.432 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-19T14:12:51.432 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-08-19T14:12:51.433 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-19T14:12:51.462 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:12:51.530 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:12:51.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:51.531 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:12:51.632 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:12:51.660 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:12:51.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:51.661 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:12:51.763 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:12:51.798 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:12:51.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:51.799 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:12:51.815 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-08-19T14:12:51.829 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 2 16 8539 8523 266.321 244.625 0.00207911 0.00356811 2024-08-19T14:12:51.853 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:12:51.853 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:12:51.900 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:12:51.931 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:12:51.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:51.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:12:52.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:51 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:12:52.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:51 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:12:52.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:51 smithi032 ceph-mon[20004]: osdmap e107: 12 total, 12 up, 12 in 2024-08-19T14:12:52.033 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:12:52.058 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:12:52.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:52.059 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:12:52.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:51 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:12:52.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:51 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:12:52.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:51 smithi031 ceph-mon[16336]: osdmap e107: 12 total, 12 up, 12 in 2024-08-19T14:12:52.160 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:12:52.195 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:12:52.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:52.196 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:12:52.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:51 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:12:52.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:51 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:12:52.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:51 smithi045 ceph-mon[20143]: osdmap e107: 12 total, 12 up, 12 in 2024-08-19T14:12:52.296 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:12:52.329 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:12:52.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:52.330 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:12:52.431 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:12:52.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:52.454 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:12:52.454 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:12:52.556 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:12:52.586 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:12:52.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:52.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:12:52.688 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:12:52.734 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:12:52.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:52.735 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:12:52.830 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 3 16 11556 11540 240.397 188.562 0.00242866 0.00415164 2024-08-19T14:12:52.834 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:52 smithi031 ceph-mon[16336]: pgmap v296: 17 pgs: 17 active+clean; 110 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s wr, 291 op/s 2024-08-19T14:12:52.836 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:12:52.845 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-19T14:12:52.870 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:12:52.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:52.871 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:12:52.952 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:12:52.972 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:12:53.023 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:12:53.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:53.024 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:12:53.090 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:12:53.091 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-08-19T14:12:53.125 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:12:53.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:53.173 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:12:53.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:12:53.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:52 smithi032 ceph-mon[20004]: pgmap v296: 17 pgs: 17 active+clean; 110 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s wr, 291 op/s 2024-08-19T14:12:53.273 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:12:53.283 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:12:53.283 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:12:53.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:52 smithi045 ceph-mon[20143]: pgmap v296: 17 pgs: 17 active+clean; 110 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s wr, 291 op/s 2024-08-19T14:12:53.300 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:12:53.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:53.302 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:12:53.322 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:12:53.322 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:12:53.358 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:12:53.359 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:12:53.395 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:12:53.395 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:12:53.403 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:12:53.432 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:12:53.432 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:12:53.435 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:12:53.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:53.437 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:12:53.468 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:12:53.469 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:12:53.503 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:12:53.504 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:12:53.537 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:12:53.542 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:12:53.542 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:12:53.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:53.575 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:12:53.576 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:12:53.577 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:12:53.577 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:12:53.606 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:12:53.607 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:12:53.632 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:12:53.632 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:12:53.652 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:12:53.652 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:12:53.676 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:12:53.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:53.698 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:12:53.698 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:12:53.799 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:12:53.830 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 4 16 14404 14388 224.794 178 0.00682423 0.00444065 2024-08-19T14:12:53.831 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:12:53.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:53.832 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:12:53.933 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:12:53.962 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:12:53.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:53.963 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:12:53.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:53 smithi031 ceph-mon[16336]: from='client.24617 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:12:53.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:53 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/4168478961' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:12:54.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:53 smithi032 ceph-mon[20004]: from='client.24617 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:12:54.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:53 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/4168478961' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:12:54.064 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:12:54.099 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:12:54.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:54.100 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:12:54.201 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:12:54.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:54.231 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:12:54.231 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:12:54.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:53 smithi045 ceph-mon[20143]: from='client.24617 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:12:54.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:53 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/4168478961' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:12:54.332 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:12:54.363 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:12:54.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:54.364 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:12:54.419 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:12:54.419 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:12:54.465 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:12:54.498 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:12:54.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:54.500 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:12:54.600 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:12:54.635 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:12:54.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:54.636 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:12:54.737 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:12:54.770 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:12:54.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:54.772 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:12:54.830 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 5 16 18033 18017 225.194 226.812 0.00548197 0.00443374 2024-08-19T14:12:54.873 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:12:54.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:54.911 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:12:54.911 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:12:55.012 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:12:55.047 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:12:55.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:55.048 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:12:55.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:54 smithi031 ceph-mon[16336]: pgmap v297: 17 pgs: 17 active+clean; 376 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 54 MiB/s wr, 866 op/s 2024-08-19T14:12:55.149 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:12:55.182 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:12:55.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:55.183 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:12:55.196 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:54 smithi045 ceph-mon[20143]: pgmap v297: 17 pgs: 17 active+clean; 376 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 54 MiB/s wr, 866 op/s 2024-08-19T14:12:55.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:54 smithi032 ceph-mon[20004]: pgmap v297: 17 pgs: 17 active+clean; 376 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 54 MiB/s wr, 866 op/s 2024-08-19T14:12:55.284 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:12:55.319 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:12:55.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:55.320 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:12:55.379 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":296,"stamp":"2024-08-19T14:12:51.734473+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":115212832,"num_objects":1751,"num_object_clones":0,"num_object_copies":5253,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1751,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":1930,"num_write_kb":114874,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1888,"ondisk_log_size":1888,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":45,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":504872,"kb_used_data":147340,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124436952,"statfs":{"total":1151940427776,"available":1151423438848,"internally_reserved":0,"allocated":150876160,"data_stored":143897366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":365933882},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":114622464,"num_objects":1749,"num_object_clones":0,"num_object_copies":5247,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1749,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1749,"num_write_kb":111936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001058"},"pg_stats":[{"pgid":"2.f","version":"0'0","reported_seq":16,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-08-19T14:12:46.850062+0000","last_change":"2024-08-19T14:12:43.418438+0000","last_active":"2024-08-19T14:12:46.850062+0000","last_peered":"2024-08-19T14:12:46.850062+0000","last_clean":"2024-08-19T14:12:46.850062+0000","last_became_active":"2024-08-19T14:12:43.417591+0000","last_became_peered":"2024-08-19T14:12:43.417591+0000","last_unstale":"2024-08-19T14:12:46.850062+0000","last_undegraded":"2024-08-19T14:12:46.850062+0000","last_fullsized":"2024-08-19T14:12:46.850062+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:12:41.713780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:14:03.830426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.e","version":"106'214","reported_seq":231,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-08-19T14:12:50.665531+0000","last_change":"2024-08-19T14:12:42.963558+0000","last_active":"2024-08-19T14:12:50.665531+0000","last_peered":"2024-08-19T14:12:50.665531+0000","last_clean":"2024-08-19T14:12:50.665531+0000","last_became_active":"2024-08-19T14:12:42.963208+0000","last_became_peered":"2024-08-19T14:12:42.963208+0000","last_unstale":"2024-08-19T14:12:50.665531+0000","last_undegraded":"2024-08-19T14:12:50.665531+0000","last_fullsized":"2024-08-19T14:12:50.665531+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:12:41.713780+0000","objects_scrubbed":0,"log_size":214,"log_dups_size":0,"ondisk_log_size":214,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:14:05.735203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14024704,"num_objects":214,"num_object_clones":0,"num_object_copies":642,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":214,"num_write_kb":13696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9],"acting":[2,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":16,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-08-19T14:12:46.849856+0000","last_change":"2024-08-19T14:12:43.417948+0000","last_active":"2024-08-19T14:12:46.849856+0000","last_peered":"2024-08-19T14:12:46.849856+0000","last_clean":"2024-08-19T14:12:46.849856+0000","last_became_active":"2024-08-19T14:12:43.417506+0000","last_became_peered":"2024-08-19T14:12:43.417506+0000","last_unstale":"2024-08-19T14:12:46.849856+0000","last_undegraded":"2024-08-19T14:12:46.849856+0000","last_fullsized":"2024-08-19T14:12:46.849856+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:12:41.713780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:13:53.364424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.c","version":"106'199","reported_seq":217,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-08-19T14:12:50.497779+0000","last_change":"2024-08-19T14:12:43.039428+0000","last_active":"2024-08-19T14:12:50.497779+0000","last_peered":"2024-08-19T14:12:50.497779+0000","last_clean":"2024-08-19T14:12:50.497779+0000","last_became_active":"2024-08-19T14:12:43.038912+0000","last_became_peered":"2024-08-19T14:12:43.038912+0000","last_unstale":"2024-08-19T14:12:50.497779+0000","last_undegraded":"2024-08-19T14:12:50.497779+0000","last_fullsized":"2024-08-19T14:12:50.497779+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:12:41.713780+0000","objects_scrubbed":0,"log_size":199,"log_dups_size":0,"ondisk_log_size":199,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:14:01.252952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13041664,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":199,"num_write_kb":12736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,1],"acting":[11,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"70'139","reported_seq":367,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-08-19T14:12:46.796216+0000","last_change":"2024-08-19T14:12:04.853784+0000","last_active":"2024-08-19T14:12:46.796216+0000","last_peered":"2024-08-19T14:12:46.796216+0000","last_clean":"2024-08-19T14:12:46.796216+0000","last_became_active":"2024-08-19T14:11:43.524419+0000","last_became_peered":"2024-08-19T14:11:43.524419+0000","last_unstale":"2024-08-19T14:12:46.796216+0000","last_undegraded":"2024-08-19T14:12:46.796216+0000","last_fullsized":"2024-08-19T14:12:46.796216+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:12:04.853650+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:12:04.853650+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:13:34.151342+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"107'468","reported_seq":488,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-08-19T14:12:51.439604+0000","last_change":"2024-08-19T14:12:42.729380+0000","last_active":"2024-08-19T14:12:51.439604+0000","last_peered":"2024-08-19T14:12:51.439604+0000","last_clean":"2024-08-19T14:12:51.439604+0000","last_became_active":"2024-08-19T14:12:42.728978+0000","last_became_peered":"2024-08-19T14:12:42.728978+0000","last_unstale":"2024-08-19T14:12:51.439604+0000","last_undegraded":"2024-08-19T14:12:51.439604+0000","last_fullsized":"2024-08-19T14:12:51.439604+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:12:41.713780+0000","objects_scrubbed":0,"log_size":468,"log_dups_size":0,"ondisk_log_size":468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:14:01.040565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30670848,"num_objects":468,"num_object_clones":0,"num_object_copies":1404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":468,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":468,"num_write_kb":29952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":16,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-08-19T14:12:46.796126+0000","last_change":"2024-08-19T14:12:42.750113+0000","last_active":"2024-08-19T14:12:46.796126+0000","last_peered":"2024-08-19T14:12:46.796126+0000","last_clean":"2024-08-19T14:12:46.796126+0000","last_became_active":"2024-08-19T14:12:42.749752+0000","last_became_peered":"2024-08-19T14:12:42.749752+0000","last_unstale":"2024-08-19T14:12:46.796126+0000","last_undegraded":"2024-08-19T14:12:46.796126+0000","last_fullsized":"2024-08-19T14:12:46.796126+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:12:41.713780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:13:42.968083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,9],"acting":[7,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"106'234","reported_seq":252,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-08-19T14:12:50.650280+0000","last_change":"2024-08-19T14:12:43.039198+0000","last_active":"2024-08-19T14:12:50.650280+0000","last_peered":"2024-08-19T14:12:50.650280+0000","last_clean":"2024-08-19T14:12:50.650280+0000","last_became_active":"2024-08-19T14:12:43.038833+0000","last_became_peered":"2024-08-19T14:12:43.038833+0000","last_unstale":"2024-08-19T14:12:50.650280+0000","last_undegraded":"2024-08-19T14:12:50.650280+0000","last_fullsized":"2024-08-19T14:12:50.650280+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:12:41.713780+0000","objects_scrubbed":0,"log_size":234,"log_dups_size":0,"ondisk_log_size":234,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:13:59.308042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15335424,"num_objects":234,"num_object_clones":0,"num_object_copies":702,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":234,"num_write_kb":14976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"107'443","reported_seq":463,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-08-19T14:12:51.439636+0000","last_change":"2024-08-19T14:12:42.729159+0000","last_active":"2024-08-19T14:12:51.439636+0000","last_peered":"2024-08-19T14:12:51.439636+0000","last_clean":"2024-08-19T14:12:51.439636+0000","last_became_active":"2024-08-19T14:12:42.728710+0000","last_became_peered":"2024-08-19T14:12:42.728710+0000","last_unstale":"2024-08-19T14:12:51.439636+0000","last_undegraded":"2024-08-19T14:12:51.439636+0000","last_fullsized":"2024-08-19T14:12:51.439636+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:12:41.713780+0000","objects_scrubbed":0,"log_size":443,"log_dups_size":0,"ondisk_log_size":443,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:14:10.799819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29032448,"num_objects":443,"num_object_clones":0,"num_object_copies":1329,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":443,"num_write_kb":28352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":16,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-08-19T14:12:46.849200+0000","last_change":"2024-08-19T14:12:43.039576+0000","last_active":"2024-08-19T14:12:46.849200+0000","last_peered":"2024-08-19T14:12:46.849200+0000","last_clean":"2024-08-19T14:12:46.849200+0000","last_became_active":"2024-08-19T14:12:43.038924+0000","last_became_peered":"2024-08-19T14:12:43.038924+0000","last_unstale":"2024-08-19T14:12:46.849200+0000","last_undegraded":"2024-08-19T14:12:46.849200+0000","last_fullsized":"2024-08-19T14:12:46.849200+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:12:41.713780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:14:01.523601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,10],"acting":[1,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-08-19T14:12:46.850026+0000","last_change":"2024-08-19T14:12:43.418168+0000","last_active":"2024-08-19T14:12:46.850026+0000","last_peered":"2024-08-19T14:12:46.850026+0000","last_clean":"2024-08-19T14:12:46.850026+0000","last_became_active":"2024-08-19T14:12:43.417616+0000","last_became_peered":"2024-08-19T14:12:43.417616+0000","last_unstale":"2024-08-19T14:12:46.850026+0000","last_undegraded":"2024-08-19T14:12:46.850026+0000","last_fullsized":"2024-08-19T14:12:46.850026+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:12:41.713780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:13:59.305980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-08-19T14:12:46.849189+0000","last_change":"2024-08-19T14:12:42.963821+0000","last_active":"2024-08-19T14:12:46.849189+0000","last_peered":"2024-08-19T14:12:46.849189+0000","last_clean":"2024-08-19T14:12:46.849189+0000","last_became_active":"2024-08-19T14:12:42.963203+0000","last_became_peered":"2024-08-19T14:12:42.963203+0000","last_unstale":"2024-08-19T14:12:46.849189+0000","last_undegraded":"2024-08-19T14:12:46.849189+0000","last_fullsized":"2024-08-19T14:12:46.849189+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:12:41.713780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:14:01.457016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,3],"acting":[1,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-08-19T14:12:46.849838+0000","last_change":"2024-08-19T14:12:42.750098+0000","last_active":"2024-08-19T14:12:46.849838+0000","last_peered":"2024-08-19T14:12:46.849838+0000","last_clean":"2024-08-19T14:12:46.849838+0000","last_became_active":"2024-08-19T14:12:42.749703+0000","last_became_peered":"2024-08-19T14:12:42.749703+0000","last_unstale":"2024-08-19T14:12:46.849838+0000","last_undegraded":"2024-08-19T14:12:46.849838+0000","last_fullsized":"2024-08-19T14:12:46.849838+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:12:41.713780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:14:10.134149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,2],"acting":[6,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":16,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-08-19T14:12:46.796145+0000","last_change":"2024-08-19T14:12:43.039135+0000","last_active":"2024-08-19T14:12:46.796145+0000","last_peered":"2024-08-19T14:12:46.796145+0000","last_clean":"2024-08-19T14:12:46.796145+0000","last_became_active":"2024-08-19T14:12:43.038882+0000","last_became_peered":"2024-08-19T14:12:43.038882+0000","last_unstale":"2024-08-19T14:12:46.796145+0000","last_undegraded":"2024-08-19T14:12:46.796145+0000","last_fullsized":"2024-08-19T14:12:46.796145+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:12:41.713780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:13:55.854839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,0],"acting":[7,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":16,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-08-19T14:12:46.849177+0000","last_change":"2024-08-19T14:12:42.963504+0000","last_active":"2024-08-19T14:12:46.849177+0000","last_peered":"2024-08-19T14:12:46.849177+0000","last_clean":"2024-08-19T14:12:46.849177+0000","last_became_active":"2024-08-19T14:12:42.963114+0000","last_became_peered":"2024-08-19T14:12:42.963114+0000","last_unstale":"2024-08-19T14:12:46.849177+0000","last_undegraded":"2024-08-19T14:12:46.849177+0000","last_fullsized":"2024-08-19T14:12:46.849177+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:12:41.713780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:14:03.432018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.a","version":"106'191","reported_seq":209,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-08-19T14:12:50.495394+0000","last_change":"2024-08-19T14:12:42.963510+0000","last_active":"2024-08-19T14:12:50.495394+0000","last_peered":"2024-08-19T14:12:50.495394+0000","last_clean":"2024-08-19T14:12:50.495394+0000","last_became_active":"2024-08-19T14:12:42.963040+0000","last_became_peered":"2024-08-19T14:12:42.963040+0000","last_unstale":"2024-08-19T14:12:50.495394+0000","last_undegraded":"2024-08-19T14:12:50.495394+0000","last_fullsized":"2024-08-19T14:12:50.495394+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:12:41.713780+0000","objects_scrubbed":0,"log_size":191,"log_dups_size":0,"ondisk_log_size":191,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:14:08.585038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12517376,"num_objects":191,"num_object_clones":0,"num_object_copies":573,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":191,"num_write_kb":12224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,9],"acting":[11,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":16,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-08-19T14:12:46.850025+0000","last_change":"2024-08-19T14:12:42.729556+0000","last_active":"2024-08-19T14:12:46.850025+0000","last_peered":"2024-08-19T14:12:46.850025+0000","last_clean":"2024-08-19T14:12:46.850025+0000","last_became_active":"2024-08-19T14:12:42.729014+0000","last_became_peered":"2024-08-19T14:12:42.729014+0000","last_unstale":"2024-08-19T14:12:46.850025+0000","last_undegraded":"2024-08-19T14:12:46.850025+0000","last_fullsized":"2024-08-19T14:12:46.850025+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:12:41.713780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:13:51.929231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":114622464,"num_objects":1749,"num_object_clones":0,"num_object_copies":5247,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1749,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1749,"num_write_kb":111936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":214695936,"data_stored":214695936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1749,"ondisk_log_size":1749,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187474,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32816,"kb_used_data":1636,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712336,"statfs":{"total":95995035648,"available":95961432064,"internally_reserved":0,"allocated":1675264,"data_stored":1092142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":80,"seq":343597383702,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32284,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712868,"statfs":{"total":95995035648,"available":95961976832,"internally_reserved":0,"allocated":1130496,"data_stored":554922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":74,"seq":317827579931,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28136,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717016,"statfs":{"total":95995035648,"available":95966224384,"internally_reserved":0,"allocated":1142784,"data_stored":563214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874273,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32296,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712856,"statfs":{"total":95995035648,"available":95961964544,"internally_reserved":0,"allocated":1142784,"data_stored":563214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":57,"seq":244813135907,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32816,"kb_used_data":1636,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712336,"statfs":{"total":95995035648,"available":95961432064,"internally_reserved":0,"allocated":1675264,"data_stored":1092142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430247,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28660,"kb_used_data":1636,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716492,"statfs":{"total":95995035648,"available":95965687808,"internally_reserved":0,"allocated":1675264,"data_stored":1092142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691884,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":81412,"kb_used_data":50228,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93663740,"statfs":{"total":95995035648,"available":95911669760,"internally_reserved":0,"allocated":51433472,"data_stored":50845280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986224,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28140,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717012,"statfs":{"total":95995035648,"available":95966220288,"internally_reserved":0,"allocated":1142784,"data_stored":563214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149750,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28136,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717016,"statfs":{"total":95995035648,"available":95966224384,"internally_reserved":0,"allocated":1142784,"data_stored":563214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411387,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":86452,"kb_used_data":55272,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93658700,"statfs":{"total":95995035648,"available":95906508800,"internally_reserved":0,"allocated":56598528,"data_stored":56014622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574911,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32296,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712856,"statfs":{"total":95995035648,"available":95961964544,"internally_reserved":0,"allocated":1142784,"data_stored":563214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738436,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":61428,"kb_used_data":30248,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93683724,"statfs":{"total":95995035648,"available":95932133376,"internally_reserved":0,"allocated":30973952,"data_stored":30390046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":56492032,"data_stored":56492032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61014016,"data_stored":61014016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":59899904,"data_stored":59899904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":37289984,"data_stored":37289984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T14:12:55.380 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-08-19T14:12:55.380 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd pool set .mgr pg_num 11 2024-08-19T14:12:55.420 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:12:55.461 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:12:55.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:55.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:12:55.564 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:12:55.599 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:12:55.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:55.600 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:12:55.701 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:12:55.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:55.739 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:12:55.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:12:55.830 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 6 16 21396 21380 222.69 210.188 0.00177689 0.0044787 2024-08-19T14:12:55.840 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:12:55.861 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:12:55.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:55.862 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:12:55.944 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:12:55.963 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:12:55.969 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-08-19T14:12:55.984 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:12:55.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:55.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:12:56.086 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:12:56.114 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:12:56.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:56.115 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:12:56.216 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:12:56.262 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:12:56.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:56.263 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:12:56.364 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:12:56.399 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:12:56.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:56.400 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:12:56.501 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:12:56.530 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:12:56.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:56.531 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:12:56.555 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:12:56.621 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:12:56.631 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:12:56.656 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:12:56.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:56.657 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:12:56.758 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:12:56.794 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:12:56.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:56.795 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:12:56.830 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 7 16 23822 23806 212.536 151.625 0.0033513 0.00469937 2024-08-19T14:12:56.896 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:12:56.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:56.931 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:12:56.931 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:12:56.950 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:12:57.032 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:12:57.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:57.054 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:12:57.054 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:12:57.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:56 smithi031 ceph-mon[16336]: pgmap v298: 17 pgs: 17 active+clean; 914 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 114 MiB/s wr, 1.82k op/s 2024-08-19T14:12:57.155 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:12:57.177 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:12:57.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:57.177 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:12:57.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:56 smithi032 ceph-mon[20004]: pgmap v298: 17 pgs: 17 active+clean; 914 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 114 MiB/s wr, 1.82k op/s 2024-08-19T14:12:57.278 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:12:57.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:56 smithi045 ceph-mon[20143]: pgmap v298: 17 pgs: 17 active+clean; 914 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 114 MiB/s wr, 1.82k op/s 2024-08-19T14:12:57.313 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:12:57.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:57.315 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:12:57.416 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:12:57.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:57.453 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:12:57.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:12:57.554 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:12:57.588 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:12:57.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:57.589 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:12:57.690 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:12:57.724 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:12:57.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:57.725 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:12:57.826 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:12:57.830 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 8 16 26588 26572 207.576 172.875 0.00288399 0.00481252 2024-08-19T14:12:57.840 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is unset 2024-08-19T14:12:57.855 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:12:57.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:57.856 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:12:57.957 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:12:57.992 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:12:57.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:57.993 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:12:58.094 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:12:58.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:57 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2198193061' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:12:58.122 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:12:58.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:58.123 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:12:58.224 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:12:58.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:57 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2198193061' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:12:58.260 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:12:58.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:58.261 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:12:58.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:57 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2198193061' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:12:58.362 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:12:58.394 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:12:58.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:58.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:12:58.496 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:12:58.525 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:12:58.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:58.526 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:12:58.627 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:12:58.655 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:12:58.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:58.656 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:12:58.757 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:12:58.778 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:12:58.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:58.779 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:12:58.830 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 9 16 29540 29524 205.01 184.5 0.00633402 0.0048727 2024-08-19T14:12:58.851 INFO:teuthology.orchestra.run.smithi031.stderr:set pool 1 pg_num to 11 2024-08-19T14:12:58.880 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:12:58.917 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:12:58.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:58.918 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:12:59.018 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:12:59.018 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:12:59.019 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:12:59.048 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:12:59.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:59.049 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:12:59.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:58 smithi031 ceph-mon[16336]: pgmap v299: 17 pgs: 17 active+clean; 1.2 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 152 MiB/s wr, 2.42k op/s 2024-08-19T14:12:59.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:58 smithi031 ceph-mon[16336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:12:59.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:58 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2198193061' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:12:59.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:58 smithi031 ceph-mon[16336]: osdmap e108: 12 total, 12 up, 12 in 2024-08-19T14:12:59.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:58 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/63475761' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-08-19T14:12:59.150 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:12:59.180 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:12:59.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:59.181 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:12:59.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:58 smithi032 ceph-mon[20004]: pgmap v299: 17 pgs: 17 active+clean; 1.2 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 152 MiB/s wr, 2.42k op/s 2024-08-19T14:12:59.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:58 smithi032 ceph-mon[20004]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:12:59.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:58 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2198193061' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:12:59.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:58 smithi032 ceph-mon[20004]: osdmap e108: 12 total, 12 up, 12 in 2024-08-19T14:12:59.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:58 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/63475761' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-08-19T14:12:59.282 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:12:59.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:58 smithi045 ceph-mon[20143]: pgmap v299: 17 pgs: 17 active+clean; 1.2 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 152 MiB/s wr, 2.42k op/s 2024-08-19T14:12:59.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:58 smithi045 ceph-mon[20143]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:12:59.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:58 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2198193061' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:12:59.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:58 smithi045 ceph-mon[20143]: osdmap e108: 12 total, 12 up, 12 in 2024-08-19T14:12:59.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:58 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/63475761' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-08-19T14:12:59.313 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:12:59.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:59.314 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:12:59.415 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:12:59.450 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:12:59.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:59.451 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:12:59.552 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:12:59.572 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:12:59.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:59.573 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:12:59.674 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:12:59.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:59.708 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:12:59.708 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:12:59.809 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:12:59.830 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 10 16 32149 32133 200.814 163.062 0.0017991 0.00496155 2024-08-19T14:12:59.834 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:12:59.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:59.835 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:12:59.936 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:12:59.958 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:12:59.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:12:59.959 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:13:00.060 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:13:00.095 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:00.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:00.096 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:13:00.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:59 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/63475761' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-08-19T14:13:00.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:59 smithi031 ceph-mon[16336]: osdmap e109: 12 total, 12 up, 12 in 2024-08-19T14:13:00.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:12:59 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-08-19T14:13:00.196 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:13:00.218 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:00.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:00.219 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:13:00.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:59 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/63475761' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-08-19T14:13:00.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:59 smithi032 ceph-mon[20004]: osdmap e109: 12 total, 12 up, 12 in 2024-08-19T14:13:00.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:12:59 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-08-19T14:13:00.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:59 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/63475761' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-08-19T14:13:00.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:59 smithi045 ceph-mon[20143]: osdmap e109: 12 total, 12 up, 12 in 2024-08-19T14:13:00.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:12:59 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-08-19T14:13:00.320 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:13:00.342 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:00.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:00.344 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:13:00.445 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:13:00.477 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:00.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:00.478 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:13:00.571 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:13:00.579 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:13:00.605 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:00.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:00.606 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:13:00.707 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:13:00.740 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:00.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:00.741 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:13:00.830 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 11 16 34276 34260 194.643 132.938 0.00494995 0.00513287 2024-08-19T14:13:00.842 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:13:00.876 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:00.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:00.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:13:00.978 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:13:01.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:01.012 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:01.013 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:13:01.114 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:13:01.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:00 smithi031 ceph-mon[16336]: pgmap v302: 17 pgs: 17 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 188 MiB/s wr, 3.01k op/s 2024-08-19T14:13:01.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:00 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-08-19T14:13:01.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:00 smithi031 ceph-mon[16336]: osdmap e110: 12 total, 12 up, 12 in 2024-08-19T14:13:01.145 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:01.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:01.146 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:13:01.247 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:13:01.248 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-08-19T14:13:01.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:00 smithi032 ceph-mon[20004]: pgmap v302: 17 pgs: 17 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 188 MiB/s wr, 3.01k op/s 2024-08-19T14:13:01.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:00 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-08-19T14:13:01.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:00 smithi032 ceph-mon[20004]: osdmap e110: 12 total, 12 up, 12 in 2024-08-19T14:13:01.281 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:01.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:01.282 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:13:01.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:00 smithi045 ceph-mon[20143]: pgmap v302: 17 pgs: 17 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 188 MiB/s wr, 3.01k op/s 2024-08-19T14:13:01.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:00 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-08-19T14:13:01.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:00 smithi045 ceph-mon[20143]: osdmap e110: 12 total, 12 up, 12 in 2024-08-19T14:13:01.384 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:13:01.408 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:01.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:01.409 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:13:01.509 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:13:01.541 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:01.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:01.542 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:13:01.643 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:13:01.682 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:01.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:01.684 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:13:01.785 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:13:01.826 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:01.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:01.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:13:01.830 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 12 16 37099 37083 193.124 176.438 0.00650698 0.00517342 2024-08-19T14:13:01.857 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:13:01.857 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:13:01.895 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:13:01.895 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:13:01.928 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:13:01.928 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:13:01.930 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:13:01.957 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:01.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:01.958 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:13:01.961 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:13:01.961 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:13:01.991 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:13:01.991 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:13:02.025 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:13:02.025 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:13:02.057 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:13:02.058 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:13:02.058 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:13:02.089 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:02.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:02.090 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:13:02.093 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:13:02.093 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:13:02.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:01 smithi031 ceph-mon[16336]: osdmap e111: 12 total, 12 up, 12 in 2024-08-19T14:13:02.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:01 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-08-19T14:13:02.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:01 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2767265819' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:13:02.128 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:13:02.129 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:13:02.167 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:13:02.167 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:13:02.190 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:13:02.199 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:13:02.200 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:13:02.229 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:02.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:02.230 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:13:02.231 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:13:02.232 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:13:02.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:01 smithi032 ceph-mon[20004]: osdmap e111: 12 total, 12 up, 12 in 2024-08-19T14:13:02.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:01 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-08-19T14:13:02.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:01 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2767265819' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:13:02.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:01 smithi045 ceph-mon[20143]: osdmap e111: 12 total, 12 up, 12 in 2024-08-19T14:13:02.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:01 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-08-19T14:13:02.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:01 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2767265819' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:13:02.331 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:13:02.372 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:02.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:02.374 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:13:02.474 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:13:02.510 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:02.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:02.511 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:13:02.612 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:13:02.639 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:02.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:02.640 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:13:02.741 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:13:02.770 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:02.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:02.771 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:13:02.830 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 13 16 39853 39837 191.508 172.125 0.00261252 0.00521714 2024-08-19T14:13:02.872 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:13:02.907 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:02.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:02.908 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:13:03.009 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:13:03.033 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:03.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:03.034 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:13:03.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:02 smithi031 ceph-mon[16336]: 1.0 scrub starts 2024-08-19T14:13:03.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:02 smithi031 ceph-mon[16336]: 1.0 scrub ok 2024-08-19T14:13:03.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:02 smithi031 ceph-mon[16336]: pgmap v305: 27 pgs: 10 unknown, 17 active+clean; 1.9 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 177 MiB/s wr, 2.83k op/s 2024-08-19T14:13:03.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:02 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-08-19T14:13:03.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:02 smithi031 ceph-mon[16336]: osdmap e112: 12 total, 12 up, 12 in 2024-08-19T14:13:03.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:02 smithi031 ceph-mon[16336]: osdmap e113: 12 total, 12 up, 12 in 2024-08-19T14:13:03.135 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:13:03.156 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:03.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:03.157 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:13:03.194 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:13:03.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:02 smithi032 ceph-mon[20004]: 1.0 scrub starts 2024-08-19T14:13:03.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:02 smithi032 ceph-mon[20004]: 1.0 scrub ok 2024-08-19T14:13:03.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:02 smithi032 ceph-mon[20004]: pgmap v305: 27 pgs: 10 unknown, 17 active+clean; 1.9 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 177 MiB/s wr, 2.83k op/s 2024-08-19T14:13:03.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:02 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-08-19T14:13:03.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:02 smithi032 ceph-mon[20004]: osdmap e112: 12 total, 12 up, 12 in 2024-08-19T14:13:03.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:02 smithi032 ceph-mon[20004]: osdmap e113: 12 total, 12 up, 12 in 2024-08-19T14:13:03.258 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:13:03.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:03.296 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:03.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:13:03.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:02 smithi045 ceph-mon[20143]: 1.0 scrub starts 2024-08-19T14:13:03.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:02 smithi045 ceph-mon[20143]: 1.0 scrub ok 2024-08-19T14:13:03.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:02 smithi045 ceph-mon[20143]: pgmap v305: 27 pgs: 10 unknown, 17 active+clean; 1.9 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 177 MiB/s wr, 2.83k op/s 2024-08-19T14:13:03.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:02 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-08-19T14:13:03.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:02 smithi045 ceph-mon[20143]: osdmap e112: 12 total, 12 up, 12 in 2024-08-19T14:13:03.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:02 smithi045 ceph-mon[20143]: osdmap e113: 12 total, 12 up, 12 in 2024-08-19T14:13:03.397 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:13:03.430 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:03.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:03.431 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:13:03.532 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:13:03.554 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:03.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:03.555 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:13:03.656 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:13:03.680 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:03.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:03.681 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:13:03.783 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:13:03.784 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:13:03.784 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-08-19T14:13:03.784 INFO:teuthology.orchestra.run.smithi031.stdout: "complaint_time": 30, 2024-08-19T14:13:03.784 INFO:teuthology.orchestra.run.smithi031.stdout: "num_blocked_ops": 0 2024-08-19T14:13:03.784 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:13:03.824 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:03.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:03.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:13:03.830 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 14 16 41953 41937 187.203 131.25 0.00946961 0.00533687 2024-08-19T14:13:03.925 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:13:03.958 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:03.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:03.959 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:13:04.060 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:13:04.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:04.097 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:04.097 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:13:04.198 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:13:04.229 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:04.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:04.230 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:13:04.331 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:13:04.352 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:04.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:04.353 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:13:04.377 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-19T14:13:04.454 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:13:04.475 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:04.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:04.476 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:13:04.577 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:13:04.607 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:04.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:04.608 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:13:04.620 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-08-19T14:13:04.637 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-08-19T14:13:04.637 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-19T14:13:04.637 INFO:tasks.thrashosds.thrasher:Removing osd 10, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-08-19T14:13:04.637 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd out 10 2024-08-19T14:13:04.709 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:13:04.711 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:04 smithi031 ceph-mon[16336]: osdmap e114: 12 total, 12 up, 12 in 2024-08-19T14:13:04.778 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:04.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:04.780 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:13:04.781 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:13:04.830 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 15 16 43929 43913 182.955 123.5 0.0139923 0.00545973 2024-08-19T14:13:04.880 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:13:04.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:04.914 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:04.914 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:13:05.006 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:04 smithi032 ceph-mon[20004]: osdmap e114: 12 total, 12 up, 12 in 2024-08-19T14:13:05.014 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:13:05.041 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:05.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:05.043 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:13:05.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:04 smithi045 ceph-mon[20143]: osdmap e114: 12 total, 12 up, 12 in 2024-08-19T14:13:05.144 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:13:05.173 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:05.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:05.175 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:13:05.182 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:13:05.276 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:13:05.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:05.323 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:05.323 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:13:05.398 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:13:05.424 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:13:05.450 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:05.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:05.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:13:05.469 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:13:05.552 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:13:05.583 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:05.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:05.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:13:05.685 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:13:05.712 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:05.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:05.713 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:13:05.814 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:13:05.830 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 16 16 45996 45980 179.594 129.188 0.00789674 0.00556341 2024-08-19T14:13:05.844 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:05.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:05.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:13:05.846 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:05 smithi031 ceph-mon[16336]: pgmap v309: 27 pgs: 2 remapped+peering, 8 unknown, 17 active+clean; 2.2 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 158 MiB/s wr, 2.50k op/s 2024-08-19T14:13:05.846 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:05 smithi031 ceph-mon[16336]: osdmap e115: 12 total, 12 up, 12 in 2024-08-19T14:13:05.946 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:13:05.972 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:05.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:05.973 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:13:06.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:05 smithi032 ceph-mon[20004]: pgmap v309: 27 pgs: 2 remapped+peering, 8 unknown, 17 active+clean; 2.2 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 158 MiB/s wr, 2.50k op/s 2024-08-19T14:13:06.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:05 smithi032 ceph-mon[20004]: osdmap e115: 12 total, 12 up, 12 in 2024-08-19T14:13:06.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:05 smithi045 ceph-mon[20143]: pgmap v309: 27 pgs: 2 remapped+peering, 8 unknown, 17 active+clean; 2.2 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 158 MiB/s wr, 2.50k op/s 2024-08-19T14:13:06.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:05 smithi045 ceph-mon[20143]: osdmap e115: 12 total, 12 up, 12 in 2024-08-19T14:13:06.074 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:13:06.106 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:06.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:06.107 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:13:06.208 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:13:06.246 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:06.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:06.247 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:13:06.264 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:13:06.265 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 20, 2024-08-19T14:13:06.265 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 600, 2024-08-19T14:13:06.265 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [ 2024-08-19T14:13:06.265 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.265 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:4 2.e 2:74371dbc:::benchmark_data_smithi045_46415_object3:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-08-19T14:13:06.265 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:12:49.831347+0000", 2024-08-19T14:13:06.265 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 16.430234850000002, 2024-08-19T14:13:06.265 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.048835283, 2024-08-19T14:13:06.265 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:13:06.265 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:13:06.265 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:13:06.266 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:13:06.266 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:13:06.266 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:13:06.266 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 4 2024-08-19T14:13:06.266 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.266 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:13:06.266 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.266 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:13:06.266 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.831347+0000", 2024-08-19T14:13:06.266 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:13:06.266 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.266 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.266 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:13:06.266 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.831347+0000", 2024-08-19T14:13:06.267 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:13:06.267 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.267 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.267 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:13:06.267 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.831347+0000", 2024-08-19T14:13:06.267 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.8299999999999998e-07 2024-08-19T14:13:06.267 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.267 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.267 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:13:06.267 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.831497+0000", 2024-08-19T14:13:06.267 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00014990500000000001 2024-08-19T14:13:06.267 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.267 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.268 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:13:06.268 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.831512+0000", 2024-08-19T14:13:06.268 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4392e-05 2024-08-19T14:13:06.268 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.268 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.268 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:13:06.268 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.831517+0000", 2024-08-19T14:13:06.268 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0900000000000004e-06 2024-08-19T14:13:06.268 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.268 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.268 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:13:06.268 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.831568+0000", 2024-08-19T14:13:06.269 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1116000000000003e-05 2024-08-19T14:13:06.269 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.269 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.269 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op must wait for map", 2024-08-19T14:13:06.269 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.831573+0000", 2024-08-19T14:13:06.269 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.8829999999999997e-06 2024-08-19T14:13:06.269 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.269 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.269 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:13:06.269 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.832394+0000", 2024-08-19T14:13:06.269 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00082153 2024-08-19T14:13:06.269 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.269 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.269 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:13:06.270 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.832472+0000", 2024-08-19T14:13:06.270 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.8104999999999998e-05 2024-08-19T14:13:06.270 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.270 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.270 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,9", 2024-08-19T14:13:06.270 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.832592+0000", 2024-08-19T14:13:06.270 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000119917 2024-08-19T14:13:06.270 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.270 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.270 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:13:06.270 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.834718+0000", 2024-08-19T14:13:06.271 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0021254619999999998 2024-08-19T14:13:06.271 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.271 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.271 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:13:06.271 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.836047+0000", 2024-08-19T14:13:06.271 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.001328907 2024-08-19T14:13:06.271 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.271 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.271 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:13:06.271 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.879582+0000", 2024-08-19T14:13:06.271 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.043534974999999997 2024-08-19T14:13:06.271 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.272 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.272 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:13:06.272 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.879743+0000", 2024-08-19T14:13:06.272 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00016134400000000001 2024-08-19T14:13:06.272 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.272 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.272 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:13:06.272 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.880182+0000", 2024-08-19T14:13:06.272 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00043927400000000002 2024-08-19T14:13:06.272 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:13:06.272 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:13:06.272 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:13:06.272 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.272 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.272 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:5 2.e 2:79195de6:::benchmark_data_smithi045_46415_object4:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-08-19T14:13:06.273 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:12:49.831533+0000", 2024-08-19T14:13:06.273 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 16.430048261, 2024-08-19T14:13:06.273 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.049479087999999997, 2024-08-19T14:13:06.273 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:13:06.273 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:13:06.273 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:13:06.273 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:13:06.273 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:13:06.273 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:13:06.273 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 5 2024-08-19T14:13:06.273 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.273 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:13:06.273 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.273 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:13:06.274 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.831533+0000", 2024-08-19T14:13:06.274 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:13:06.274 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.274 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.274 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:13:06.274 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.831533+0000", 2024-08-19T14:13:06.274 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:13:06.274 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.274 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.274 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:13:06.274 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.831534+0000", 2024-08-19T14:13:06.275 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.7000000000000004e-07 2024-08-19T14:13:06.275 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.275 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.275 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:13:06.275 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.831617+0000", 2024-08-19T14:13:06.275 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.3312000000000005e-05 2024-08-19T14:13:06.275 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.275 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.275 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:13:06.275 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.831630+0000", 2024-08-19T14:13:06.275 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2719e-05 2024-08-19T14:13:06.275 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.275 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.275 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:13:06.275 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.831635+0000", 2024-08-19T14:13:06.276 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.6310000000000002e-06 2024-08-19T14:13:06.276 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.276 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.276 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:13:06.276 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.831671+0000", 2024-08-19T14:13:06.276 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.6146000000000003e-05 2024-08-19T14:13:06.276 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.276 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.276 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting_for_map not empty", 2024-08-19T14:13:06.276 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.831673+0000", 2024-08-19T14:13:06.276 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.367e-06 2024-08-19T14:13:06.276 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.276 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.276 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:13:06.276 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.833013+0000", 2024-08-19T14:13:06.276 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.001339806 2024-08-19T14:13:06.277 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.277 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.277 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:13:06.277 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.833112+0000", 2024-08-19T14:13:06.277 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.8714000000000004e-05 2024-08-19T14:13:06.277 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.277 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.277 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,9", 2024-08-19T14:13:06.277 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.833260+0000", 2024-08-19T14:13:06.277 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000148572 2024-08-19T14:13:06.277 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.277 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.277 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:13:06.277 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.834723+0000", 2024-08-19T14:13:06.277 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0014627069999999999 2024-08-19T14:13:06.277 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.278 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.278 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:13:06.278 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.836127+0000", 2024-08-19T14:13:06.278 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.001403605 2024-08-19T14:13:06.278 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.278 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.278 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:13:06.278 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.880957+0000", 2024-08-19T14:13:06.278 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.044829829000000002 2024-08-19T14:13:06.278 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.278 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.278 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:13:06.278 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.880988+0000", 2024-08-19T14:13:06.279 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.1161000000000002e-05 2024-08-19T14:13:06.279 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.279 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.279 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:13:06.279 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.881013+0000", 2024-08-19T14:13:06.279 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.4848999999999999e-05 2024-08-19T14:13:06.279 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:13:06.279 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:13:06.279 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:13:06.280 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.280 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.280 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:10 2.e 2:7213102a:::benchmark_data_smithi045_46415_object9:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-08-19T14:13:06.280 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:12:49.831650+0000", 2024-08-19T14:13:06.280 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 16.429931454999998, 2024-08-19T14:13:06.280 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.049457579000000002, 2024-08-19T14:13:06.280 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:13:06.280 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:13:06.280 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:13:06.280 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:13:06.280 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:13:06.280 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:13:06.280 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 10 2024-08-19T14:13:06.280 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.281 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:13:06.281 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.281 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:13:06.281 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.831650+0000", 2024-08-19T14:13:06.281 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:13:06.281 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.281 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.281 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:13:06.281 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.831650+0000", 2024-08-19T14:13:06.281 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:13:06.281 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.281 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.281 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:13:06.282 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.831651+0000", 2024-08-19T14:13:06.282 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.5499999999999998e-07 2024-08-19T14:13:06.282 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.282 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.282 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:13:06.282 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.831715+0000", 2024-08-19T14:13:06.282 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.4066999999999995e-05 2024-08-19T14:13:06.282 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.282 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.282 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:13:06.282 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.831722+0000", 2024-08-19T14:13:06.282 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.0899999999999999e-06 2024-08-19T14:13:06.282 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.283 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.283 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:13:06.283 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.831734+0000", 2024-08-19T14:13:06.283 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1569999999999999e-05 2024-08-19T14:13:06.283 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.283 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.283 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:13:06.283 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.831757+0000", 2024-08-19T14:13:06.283 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.3045999999999999e-05 2024-08-19T14:13:06.283 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.283 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.283 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting_for_map not empty", 2024-08-19T14:13:06.283 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.831759+0000", 2024-08-19T14:13:06.283 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.9609999999999999e-06 2024-08-19T14:13:06.284 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.284 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.284 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:13:06.284 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.833563+0000", 2024-08-19T14:13:06.284 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0018045159999999999 2024-08-19T14:13:06.284 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.284 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.284 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:13:06.284 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.833665+0000", 2024-08-19T14:13:06.284 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00010178700000000001 2024-08-19T14:13:06.284 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.284 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.284 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,9", 2024-08-19T14:13:06.284 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.833881+0000", 2024-08-19T14:13:06.285 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00021646400000000001 2024-08-19T14:13:06.285 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.285 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.285 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:13:06.285 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.834728+0000", 2024-08-19T14:13:06.285 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00084628299999999997 2024-08-19T14:13:06.285 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.285 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.285 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:13:06.285 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.836373+0000", 2024-08-19T14:13:06.285 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.001645405 2024-08-19T14:13:06.285 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.286 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.286 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:13:06.286 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.881051+0000", 2024-08-19T14:13:06.286 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.044677741999999999 2024-08-19T14:13:06.286 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.286 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.286 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:13:06.286 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.881084+0000", 2024-08-19T14:13:06.286 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.2981999999999997e-05 2024-08-19T14:13:06.286 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.286 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.287 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:13:06.287 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.881108+0000", 2024-08-19T14:13:06.287 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.4111e-05 2024-08-19T14:13:06.287 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:13:06.287 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:13:06.287 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:13:06.287 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.287 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.287 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:11 2.e 2:74bb3d92:::benchmark_data_smithi045_46415_object10:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-08-19T14:13:06.287 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:12:49.831743+0000", 2024-08-19T14:13:06.287 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 16.429839061999999, 2024-08-19T14:13:06.287 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.049447636000000003, 2024-08-19T14:13:06.288 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:13:06.288 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:13:06.288 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:13:06.288 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:13:06.288 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:13:06.288 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:13:06.288 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 11 2024-08-19T14:13:06.288 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.288 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:13:06.288 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.288 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:13:06.288 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.831743+0000", 2024-08-19T14:13:06.288 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:13:06.288 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.288 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.289 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:13:06.289 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.831743+0000", 2024-08-19T14:13:06.289 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:13:06.289 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.289 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.289 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:13:06.289 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.831743+0000", 2024-08-19T14:13:06.289 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.1699999999999999e-07 2024-08-19T14:13:06.289 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.289 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.289 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:13:06.289 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.831828+0000", 2024-08-19T14:13:06.289 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.4693999999999999e-05 2024-08-19T14:13:06.289 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.290 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.290 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:13:06.290 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.831835+0000", 2024-08-19T14:13:06.290 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.1629999999999999e-06 2024-08-19T14:13:06.290 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.290 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.290 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:13:06.290 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.831839+0000", 2024-08-19T14:13:06.290 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.8149999999999999e-06 2024-08-19T14:13:06.290 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.290 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.290 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:13:06.290 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.831865+0000", 2024-08-19T14:13:06.290 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.6635000000000002e-05 2024-08-19T14:13:06.291 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.291 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.291 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting_for_map not empty", 2024-08-19T14:13:06.291 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.831867+0000", 2024-08-19T14:13:06.291 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.9259999999999999e-06 2024-08-19T14:13:06.291 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.292 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.292 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:13:06.292 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.834168+0000", 2024-08-19T14:13:06.292 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0023005019999999998 2024-08-19T14:13:06.292 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.292 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.293 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:13:06.293 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.834284+0000", 2024-08-19T14:13:06.293 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00011576 2024-08-19T14:13:06.293 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.293 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.293 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,9", 2024-08-19T14:13:06.293 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.834409+0000", 2024-08-19T14:13:06.293 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012571699999999999 2024-08-19T14:13:06.293 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.293 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.293 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:13:06.293 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.834803+0000", 2024-08-19T14:13:06.294 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00039336200000000002 2024-08-19T14:13:06.294 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.294 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.294 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:13:06.294 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.836478+0000", 2024-08-19T14:13:06.294 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.001674954 2024-08-19T14:13:06.294 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.294 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.294 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:13:06.294 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.881139+0000", 2024-08-19T14:13:06.294 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.044661041999999998 2024-08-19T14:13:06.294 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.294 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.295 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:13:06.295 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.881168+0000", 2024-08-19T14:13:06.295 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.9026999999999998e-05 2024-08-19T14:13:06.295 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.295 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.295 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:13:06.295 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.881190+0000", 2024-08-19T14:13:06.295 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.2622000000000001e-05 2024-08-19T14:13:06.295 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:13:06.295 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:13:06.295 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:13:06.295 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.295 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.296 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.24580.0:3 2.3 e106/102 2:c78e558b:::benchmark_data_smithi045_46415_object2:head v 106'1, mlcod=0'0)", 2024-08-19T14:13:06.296 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:12:49.833633+0000", 2024-08-19T14:13:06.296 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 16.427948482000001, 2024-08-19T14:13:06.296 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.046850350999999998, 2024-08-19T14:13:06.296 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:13:06.296 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:13:06.296 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:13:06.296 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:13:06.296 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.296 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:13:06.296 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.833633+0000", 2024-08-19T14:13:06.296 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:13:06.296 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.297 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.297 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:13:06.297 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.833633+0000", 2024-08-19T14:13:06.297 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:13:06.297 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.297 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.297 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:13:06.297 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.833633+0000", 2024-08-19T14:13:06.297 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.4999999999999999e-07 2024-08-19T14:13:06.297 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.297 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.297 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:13:06.297 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.879434+0000", 2024-08-19T14:13:06.298 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.045800351000000003 2024-08-19T14:13:06.298 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.298 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.298 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:13:06.298 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.879459+0000", 2024-08-19T14:13:06.298 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.5415e-05 2024-08-19T14:13:06.298 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.298 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.298 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:13:06.298 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.879471+0000", 2024-08-19T14:13:06.298 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.134e-05 2024-08-19T14:13:06.298 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.298 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.299 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:13:06.299 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.879542+0000", 2024-08-19T14:13:06.299 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.1662999999999995e-05 2024-08-19T14:13:06.299 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.299 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.299 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:13:06.299 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.879568+0000", 2024-08-19T14:13:06.299 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.6227000000000001e-05 2024-08-19T14:13:06.299 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.299 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.299 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:13:06.299 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.880436+0000", 2024-08-19T14:13:06.299 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00086755800000000004 2024-08-19T14:13:06.299 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.300 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.300 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:13:06.300 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.880484+0000", 2024-08-19T14:13:06.300 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7546999999999997e-05 2024-08-19T14:13:06.300 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:13:06.300 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:13:06.300 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:13:06.300 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.300 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.301 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.24580.0:12 2.7 e106/102 2:ea64a4a4:::benchmark_data_smithi045_46415_object11:head v 106'1, mlcod=0'0)", 2024-08-19T14:13:06.301 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:12:49.835432+0000", 2024-08-19T14:13:06.301 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 16.426150024999998, 2024-08-19T14:13:06.301 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.045379990000000002, 2024-08-19T14:13:06.301 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:13:06.301 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:13:06.301 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:13:06.301 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:13:06.301 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.301 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:13:06.301 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.835432+0000", 2024-08-19T14:13:06.301 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:13:06.301 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.301 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.302 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:13:06.302 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.835432+0000", 2024-08-19T14:13:06.302 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:13:06.302 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.302 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.302 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:13:06.302 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.835432+0000", 2024-08-19T14:13:06.302 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.03e-07 2024-08-19T14:13:06.302 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.302 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.302 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:13:06.302 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.879599+0000", 2024-08-19T14:13:06.302 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.044167295000000002 2024-08-19T14:13:06.303 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.303 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.303 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:13:06.303 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.879614+0000", 2024-08-19T14:13:06.303 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4301000000000001e-05 2024-08-19T14:13:06.303 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.303 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.303 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:13:06.303 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.879620+0000", 2024-08-19T14:13:06.303 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.4999999999999996e-06 2024-08-19T14:13:06.303 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.303 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.303 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:13:06.303 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.879804+0000", 2024-08-19T14:13:06.304 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00018364499999999999 2024-08-19T14:13:06.304 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.304 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.304 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:13:06.304 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.880282+0000", 2024-08-19T14:13:06.304 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00047793400000000001 2024-08-19T14:13:06.304 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.304 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.304 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:13:06.304 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.880782+0000", 2024-08-19T14:13:06.304 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00050043500000000003 2024-08-19T14:13:06.304 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.304 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.304 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:13:06.305 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.880812+0000", 2024-08-19T14:13:06.305 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.9577000000000002e-05 2024-08-19T14:13:06.305 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:13:06.305 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:13:06.305 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:13:06.305 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.305 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.305 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:29 2.e 2:7027fbc6:::benchmark_data_smithi045_46415_object28:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-08-19T14:13:06.305 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:12:49.839129+0000", 2024-08-19T14:13:06.305 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 16.422452576000001, 2024-08-19T14:13:06.305 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.042268951999999999, 2024-08-19T14:13:06.305 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:13:06.305 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:13:06.305 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:13:06.305 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:13:06.306 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:13:06.306 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:13:06.306 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 29 2024-08-19T14:13:06.306 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.306 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:13:06.306 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.306 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:13:06.306 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.839129+0000", 2024-08-19T14:13:06.306 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:13:06.306 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.306 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.306 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:13:06.306 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.839129+0000", 2024-08-19T14:13:06.306 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:13:06.307 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.307 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.307 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:13:06.307 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.839130+0000", 2024-08-19T14:13:06.307 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.6000000000000004e-07 2024-08-19T14:13:06.307 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.307 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.307 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:13:06.307 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.839177+0000", 2024-08-19T14:13:06.307 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7502e-05 2024-08-19T14:13:06.307 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.307 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.307 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:13:06.308 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.839184+0000", 2024-08-19T14:13:06.308 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.0700000000000001e-06 2024-08-19T14:13:06.308 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.308 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.308 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:13:06.308 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.839188+0000", 2024-08-19T14:13:06.308 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.9160000000000003e-06 2024-08-19T14:13:06.308 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.308 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.308 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:13:06.308 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.839235+0000", 2024-08-19T14:13:06.308 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.6482e-05 2024-08-19T14:13:06.308 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.308 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.308 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:13:06.309 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.839342+0000", 2024-08-19T14:13:06.309 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00010746 2024-08-19T14:13:06.309 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.309 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.309 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,9", 2024-08-19T14:13:06.310 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.839492+0000", 2024-08-19T14:13:06.310 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000150075 2024-08-19T14:13:06.310 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.310 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.310 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:13:06.310 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.840274+0000", 2024-08-19T14:13:06.310 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00078173299999999995 2024-08-19T14:13:06.310 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.310 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.310 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:13:06.310 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.840514+0000", 2024-08-19T14:13:06.310 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00024001400000000001 2024-08-19T14:13:06.311 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.311 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.311 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:13:06.311 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.881348+0000", 2024-08-19T14:13:06.311 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.040833635 2024-08-19T14:13:06.311 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.311 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.311 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:13:06.311 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.881375+0000", 2024-08-19T14:13:06.311 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.7869e-05 2024-08-19T14:13:06.311 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.311 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.311 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:13:06.311 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.881398+0000", 2024-08-19T14:13:06.312 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.2636000000000001e-05 2024-08-19T14:13:06.312 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:13:06.312 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:13:06.312 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:13:06.312 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.312 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.312 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:5228 2.1 2:8c227e02:::benchmark_data_smithi045_46415_object5227:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-08-19T14:13:06.312 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:12:50.961869+0000", 2024-08-19T14:13:06.312 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 15.299713124, 2024-08-19T14:13:06.312 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.030052922999999999, 2024-08-19T14:13:06.312 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:13:06.312 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:13:06.312 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:13:06.313 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:13:06.313 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:13:06.313 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:13:06.313 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 5228 2024-08-19T14:13:06.313 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.313 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:13:06.313 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.313 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:13:06.313 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:50.961869+0000", 2024-08-19T14:13:06.313 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:13:06.313 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.313 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.313 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:13:06.313 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:50.961869+0000", 2024-08-19T14:13:06.314 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:13:06.314 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.314 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.314 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:13:06.314 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:50.961869+0000", 2024-08-19T14:13:06.314 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.5199999999999995e-07 2024-08-19T14:13:06.314 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.314 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.314 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:13:06.314 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:50.961956+0000", 2024-08-19T14:13:06.314 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.7027000000000006e-05 2024-08-19T14:13:06.314 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.314 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.315 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:13:06.315 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:50.961967+0000", 2024-08-19T14:13:06.315 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0448000000000001e-05 2024-08-19T14:13:06.315 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.315 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.315 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:13:06.315 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:50.961987+0000", 2024-08-19T14:13:06.315 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.9613e-05 2024-08-19T14:13:06.315 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.315 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.315 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:13:06.315 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:50.962334+0000", 2024-08-19T14:13:06.315 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00034776099999999998 2024-08-19T14:13:06.316 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.316 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.316 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:13:06.316 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:50.962443+0000", 2024-08-19T14:13:06.316 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000108486 2024-08-19T14:13:06.316 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.316 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.316 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,3", 2024-08-19T14:13:06.316 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:50.962585+0000", 2024-08-19T14:13:06.316 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00014193499999999999 2024-08-19T14:13:06.316 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.316 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.316 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:13:06.316 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:50.963427+0000", 2024-08-19T14:13:06.317 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00084245700000000001 2024-08-19T14:13:06.317 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.317 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.317 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:13:06.317 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:50.963598+0000", 2024-08-19T14:13:06.317 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000170767 2024-08-19T14:13:06.317 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.317 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.317 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:13:06.317 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:50.964250+0000", 2024-08-19T14:13:06.317 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00065191900000000004 2024-08-19T14:13:06.317 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.318 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.318 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:13:06.318 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:50.964338+0000", 2024-08-19T14:13:06.318 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.7861000000000005e-05 2024-08-19T14:13:06.318 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.318 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.318 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:13:06.318 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:50.991921+0000", 2024-08-19T14:13:06.318 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.027583797 2024-08-19T14:13:06.318 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:13:06.319 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:13:06.319 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:13:06.319 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.319 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.319 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.24580.0:5232 2.3 e107/102 2:cd7307b6:::benchmark_data_smithi045_46415_object5231:head v 107'327, mlcod=107'325)", 2024-08-19T14:13:06.319 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:12:50.963798+0000", 2024-08-19T14:13:06.319 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 15.297783554, 2024-08-19T14:13:06.319 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.028145843, 2024-08-19T14:13:06.319 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:13:06.319 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:13:06.319 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:13:06.319 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:13:06.319 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.319 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:13:06.320 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:50.963798+0000", 2024-08-19T14:13:06.320 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:13:06.320 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.320 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.320 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:13:06.320 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:50.963798+0000", 2024-08-19T14:13:06.320 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:13:06.320 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.320 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.320 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:13:06.320 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:50.963799+0000", 2024-08-19T14:13:06.320 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.1999999999999998e-07 2024-08-19T14:13:06.320 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.320 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.321 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:13:06.321 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:50.963857+0000", 2024-08-19T14:13:06.321 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.7716000000000001e-05 2024-08-19T14:13:06.321 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.321 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.321 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:13:06.321 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:50.963877+0000", 2024-08-19T14:13:06.321 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.0412e-05 2024-08-19T14:13:06.321 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.321 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.321 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:13:06.321 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:50.963883+0000", 2024-08-19T14:13:06.321 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.2360000000000001e-06 2024-08-19T14:13:06.322 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.322 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.322 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:13:06.322 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:50.963987+0000", 2024-08-19T14:13:06.322 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000103885 2024-08-19T14:13:06.322 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.322 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.322 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:13:06.322 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:50.964020+0000", 2024-08-19T14:13:06.322 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.3043000000000002e-05 2024-08-19T14:13:06.322 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.322 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.322 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:13:06.323 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:50.964672+0000", 2024-08-19T14:13:06.323 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00065210799999999998 2024-08-19T14:13:06.323 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.323 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.323 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:13:06.323 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:50.991944+0000", 2024-08-19T14:13:06.323 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.027271622999999998 2024-08-19T14:13:06.323 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:13:06.323 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:13:06.323 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:13:06.323 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.323 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.323 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.24580.0:5233 2.7 e107/102 2:e4558835:::benchmark_data_smithi045_46415_object5232:head v 107'322, mlcod=107'320)", 2024-08-19T14:13:06.324 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:12:50.964029+0000", 2024-08-19T14:13:06.324 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 15.297553083, 2024-08-19T14:13:06.324 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.028570974999999998, 2024-08-19T14:13:06.324 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:13:06.324 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:13:06.324 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:13:06.324 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:13:06.324 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.324 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:13:06.324 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:50.964029+0000", 2024-08-19T14:13:06.324 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:13:06.324 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.324 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.325 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:13:06.325 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:50.964029+0000", 2024-08-19T14:13:06.325 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:13:06.325 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.325 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.325 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:13:06.325 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:50.964029+0000", 2024-08-19T14:13:06.325 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.85e-07 2024-08-19T14:13:06.325 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.325 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.325 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:13:06.325 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:50.964142+0000", 2024-08-19T14:13:06.325 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00011280100000000001 2024-08-19T14:13:06.325 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.326 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.326 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:13:06.326 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:50.964153+0000", 2024-08-19T14:13:06.326 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0667e-05 2024-08-19T14:13:06.326 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.326 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.326 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:13:06.326 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:50.964159+0000", 2024-08-19T14:13:06.326 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.1999999999999999e-06 2024-08-19T14:13:06.326 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.326 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.326 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:13:06.326 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:50.964237+0000", 2024-08-19T14:13:06.327 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.7903000000000003e-05 2024-08-19T14:13:06.327 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.327 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.327 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:13:06.327 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:50.964264+0000", 2024-08-19T14:13:06.327 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.711e-05 2024-08-19T14:13:06.327 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.327 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.327 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:13:06.327 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:50.992558+0000", 2024-08-19T14:13:06.327 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.028294186999999998 2024-08-19T14:13:06.327 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.327 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.327 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:13:06.328 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:50.992600+0000", 2024-08-19T14:13:06.328 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.1322000000000002e-05 2024-08-19T14:13:06.328 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:13:06.328 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:13:06.328 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:13:06.328 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.328 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.328 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop_reply(client.24580.0:5228 2.1 e107/102 ondisk, result = 0)", 2024-08-19T14:13:06.328 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:12:50.964162+0000", 2024-08-19T14:13:06.328 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 15.297419996, 2024-08-19T14:13:06.328 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.027770928, 2024-08-19T14:13:06.328 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:13:06.328 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:13:06.328 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-08-19T14:13:06.329 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:13:06.329 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.329 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:13:06.329 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:50.964162+0000", 2024-08-19T14:13:06.329 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:13:06.329 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.329 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.329 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:13:06.329 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:50.964162+0000", 2024-08-19T14:13:06.329 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:13:06.329 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.329 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.329 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:13:06.329 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:50.964162+0000", 2024-08-19T14:13:06.330 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1600000000000001e-07 2024-08-19T14:13:06.330 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.330 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.330 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:13:06.330 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:50.964166+0000", 2024-08-19T14:13:06.330 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.6349999999999999e-06 2024-08-19T14:13:06.330 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.330 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.330 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:13:06.330 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:50.964174+0000", 2024-08-19T14:13:06.330 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.4780000000000008e-06 2024-08-19T14:13:06.330 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.330 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.331 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:13:06.331 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:50.964183+0000", 2024-08-19T14:13:06.331 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.9530000000000005e-06 2024-08-19T14:13:06.331 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.331 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.331 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:13:06.331 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:50.964236+0000", 2024-08-19T14:13:06.331 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.2358000000000003e-05 2024-08-19T14:13:06.331 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.331 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.331 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:13:06.331 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:50.964245+0000", 2024-08-19T14:13:06.331 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.9290000000000004e-06 2024-08-19T14:13:06.331 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.332 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.332 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:13:06.332 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:50.991933+0000", 2024-08-19T14:13:06.332 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.027688059000000001 2024-08-19T14:13:06.332 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:13:06.332 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:13:06.332 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:13:06.332 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.332 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.332 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:5239 2.e 2:74a8952a:::benchmark_data_smithi045_46415_object5238:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-08-19T14:13:06.332 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:12:50.964581+0000", 2024-08-19T14:13:06.332 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 15.297000562999999, 2024-08-19T14:13:06.332 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.028721312999999998, 2024-08-19T14:13:06.333 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:13:06.333 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:13:06.333 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:13:06.333 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:13:06.333 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:13:06.333 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:13:06.333 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 5239 2024-08-19T14:13:06.333 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.333 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:13:06.333 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.333 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:13:06.333 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:50.964581+0000", 2024-08-19T14:13:06.333 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:13:06.334 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.334 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.334 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:13:06.334 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:50.964581+0000", 2024-08-19T14:13:06.334 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:13:06.334 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.334 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.334 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:13:06.334 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:50.964582+0000", 2024-08-19T14:13:06.334 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.254e-06 2024-08-19T14:13:06.334 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.334 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.334 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:13:06.334 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:50.964659+0000", 2024-08-19T14:13:06.335 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.6544000000000004e-05 2024-08-19T14:13:06.335 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.335 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.335 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:13:06.335 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:50.991789+0000", 2024-08-19T14:13:06.335 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.027129617000000002 2024-08-19T14:13:06.335 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.335 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.335 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:13:06.335 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:50.991800+0000", 2024-08-19T14:13:06.335 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1187e-05 2024-08-19T14:13:06.335 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.335 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.335 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:13:06.336 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:50.991853+0000", 2024-08-19T14:13:06.336 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.3511999999999999e-05 2024-08-19T14:13:06.336 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.336 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.336 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:13:06.336 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:50.991987+0000", 2024-08-19T14:13:06.336 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000133638 2024-08-19T14:13:06.336 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.336 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.336 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,9", 2024-08-19T14:13:06.336 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:50.992124+0000", 2024-08-19T14:13:06.336 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013690600000000001 2024-08-19T14:13:06.336 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.336 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.337 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:13:06.337 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:50.993002+0000", 2024-08-19T14:13:06.337 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000877939 2024-08-19T14:13:06.337 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.337 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.337 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:13:06.337 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:50.993156+0000", 2024-08-19T14:13:06.337 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000153962 2024-08-19T14:13:06.337 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.337 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.337 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:13:06.337 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:50.993202+0000", 2024-08-19T14:13:06.337 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.6264000000000003e-05 2024-08-19T14:13:06.337 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.337 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.338 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:13:06.338 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:50.993259+0000", 2024-08-19T14:13:06.338 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.7442000000000003e-05 2024-08-19T14:13:06.338 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.338 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.338 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:13:06.338 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:50.993302+0000", 2024-08-19T14:13:06.338 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.3047999999999998e-05 2024-08-19T14:13:06.338 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:13:06.338 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:13:06.338 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:13:06.338 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.338 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.339 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.24580.0:5240 2.7 e107/102 2:e3205bfb:::benchmark_data_smithi045_46415_object5239:head v 107'323, mlcod=107'320)", 2024-08-19T14:13:06.339 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:12:50.965294+0000", 2024-08-19T14:13:06.339 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 15.296287504, 2024-08-19T14:13:06.339 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.027590762000000001, 2024-08-19T14:13:06.339 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:13:06.339 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:13:06.339 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:13:06.339 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:13:06.339 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.339 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:13:06.339 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:50.965294+0000", 2024-08-19T14:13:06.339 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:13:06.339 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.339 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.340 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:13:06.340 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:50.965294+0000", 2024-08-19T14:13:06.340 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:13:06.340 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.340 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.340 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:13:06.340 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:50.965295+0000", 2024-08-19T14:13:06.340 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.8999999999999995e-07 2024-08-19T14:13:06.340 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.340 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.340 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:13:06.340 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:50.965408+0000", 2024-08-19T14:13:06.340 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000113249 2024-08-19T14:13:06.341 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.341 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.341 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:13:06.341 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:50.991804+0000", 2024-08-19T14:13:06.341 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.026395827 2024-08-19T14:13:06.341 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.341 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.341 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:13:06.341 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:50.991811+0000", 2024-08-19T14:13:06.341 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.0500000000000003e-06 2024-08-19T14:13:06.341 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.341 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.341 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:13:06.341 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:50.991878+0000", 2024-08-19T14:13:06.341 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.6766000000000004e-05 2024-08-19T14:13:06.342 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.342 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.342 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:13:06.342 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:50.991894+0000", 2024-08-19T14:13:06.342 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6481e-05 2024-08-19T14:13:06.342 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.342 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.342 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:13:06.342 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:50.992827+0000", 2024-08-19T14:13:06.342 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00093302499999999998 2024-08-19T14:13:06.342 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.342 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.342 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:13:06.343 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:50.992885+0000", 2024-08-19T14:13:06.343 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.7473999999999999e-05 2024-08-19T14:13:06.343 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:13:06.343 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:13:06.343 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:13:06.343 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.343 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.343 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:9183 2.1 2:8c6b5143:::benchmark_data_smithi045_46415_object9182:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-08-19T14:13:06.343 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:12:52.037745+0000", 2024-08-19T14:13:06.343 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 14.223836226, 2024-08-19T14:13:06.343 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.048248580999999999, 2024-08-19T14:13:06.343 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:13:06.343 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:13:06.343 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:13:06.343 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:13:06.344 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:13:06.344 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:13:06.344 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 9183 2024-08-19T14:13:06.344 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.344 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:13:06.344 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.344 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:13:06.344 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:52.037745+0000", 2024-08-19T14:13:06.344 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:13:06.344 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.344 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.344 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:13:06.344 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:52.037745+0000", 2024-08-19T14:13:06.344 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:13:06.345 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.345 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.345 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:13:06.345 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:52.037746+0000", 2024-08-19T14:13:06.345 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.029e-06 2024-08-19T14:13:06.345 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.345 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.345 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:13:06.345 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:52.037808+0000", 2024-08-19T14:13:06.345 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.1181999999999998e-05 2024-08-19T14:13:06.345 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.345 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.345 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:13:06.345 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:52.037817+0000", 2024-08-19T14:13:06.346 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.0380000000000005e-06 2024-08-19T14:13:06.346 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.346 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.346 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:13:06.346 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:52.037822+0000", 2024-08-19T14:13:06.346 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1930000000000001e-06 2024-08-19T14:13:06.346 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.346 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.346 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:13:06.346 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:52.037877+0000", 2024-08-19T14:13:06.346 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.5157000000000001e-05 2024-08-19T14:13:06.346 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.347 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.347 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:13:06.347 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:52.037998+0000", 2024-08-19T14:13:06.347 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000120486 2024-08-19T14:13:06.347 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.347 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.347 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,3", 2024-08-19T14:13:06.347 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:52.038104+0000", 2024-08-19T14:13:06.347 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00010608200000000001 2024-08-19T14:13:06.347 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.347 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.347 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:13:06.347 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:52.038756+0000", 2024-08-19T14:13:06.348 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00065216000000000004 2024-08-19T14:13:06.348 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.348 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.348 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:13:06.348 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:52.085271+0000", 2024-08-19T14:13:06.348 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.046514940999999997 2024-08-19T14:13:06.348 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.348 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.348 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:13:06.348 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:52.085318+0000", 2024-08-19T14:13:06.348 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7105999999999999e-05 2024-08-19T14:13:06.348 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.348 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.348 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:13:06.349 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:52.085946+0000", 2024-08-19T14:13:06.349 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00062818200000000002 2024-08-19T14:13:06.349 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.349 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.349 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:13:06.349 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:52.085994+0000", 2024-08-19T14:13:06.349 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.8025000000000002e-05 2024-08-19T14:13:06.349 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:13:06.349 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:13:06.349 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:13:06.349 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.349 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.349 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop_reply(client.24580.0:9183 2.1 e107/102 ondisk, result = 0)", 2024-08-19T14:13:06.350 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:12:52.038765+0000", 2024-08-19T14:13:06.350 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 14.222816579, 2024-08-19T14:13:06.350 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.046514727999999998, 2024-08-19T14:13:06.350 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:13:06.350 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:13:06.350 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-08-19T14:13:06.351 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:13:06.351 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.351 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:13:06.351 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:52.038765+0000", 2024-08-19T14:13:06.351 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:13:06.351 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.351 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.351 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:13:06.352 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:52.038765+0000", 2024-08-19T14:13:06.352 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:13:06.352 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.352 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.352 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:13:06.352 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:52.038766+0000", 2024-08-19T14:13:06.352 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.5600000000000005e-07 2024-08-19T14:13:06.352 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.352 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.352 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:13:06.352 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:52.038769+0000", 2024-08-19T14:13:06.352 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.377e-06 2024-08-19T14:13:06.352 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.352 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.353 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:13:06.353 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:52.038777+0000", 2024-08-19T14:13:06.353 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.6960000000000005e-06 2024-08-19T14:13:06.353 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.353 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.353 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:13:06.353 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:52.038781+0000", 2024-08-19T14:13:06.353 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.5299999999999998e-06 2024-08-19T14:13:06.353 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.353 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.353 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:13:06.353 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:13:06.355 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:52.085244+0000", 2024-08-19T14:13:06.355 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.046462905999999998 2024-08-19T14:13:06.355 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.355 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.355 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:13:06.355 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:52.085262+0000", 2024-08-19T14:13:06.355 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.7405e-05 2024-08-19T14:13:06.355 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.355 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.355 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:13:06.355 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:52.085280+0000", 2024-08-19T14:13:06.355 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.8158000000000001e-05 2024-08-19T14:13:06.356 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:13:06.356 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:13:06.356 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:13:06.356 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.356 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.356 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop_reply(client.24580.0:9183 2.1 e107/102 ondisk, result = 0)", 2024-08-19T14:13:06.356 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:12:52.039130+0000", 2024-08-19T14:13:06.356 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 14.222451852000001, 2024-08-19T14:13:06.356 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.046875116000000001, 2024-08-19T14:13:06.356 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:13:06.356 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:13:06.356 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-08-19T14:13:06.356 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:13:06.356 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.356 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:13:06.357 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:52.039130+0000", 2024-08-19T14:13:06.357 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:13:06.357 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.357 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.357 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:13:06.357 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:52.039130+0000", 2024-08-19T14:13:06.357 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:13:06.357 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.357 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.358 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:13:06.358 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:52.039130+0000", 2024-08-19T14:13:06.358 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.2399999999999998e-07 2024-08-19T14:13:06.358 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.358 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.359 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:13:06.359 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:52.039134+0000", 2024-08-19T14:13:06.359 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7340000000000002e-06 2024-08-19T14:13:06.359 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.359 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.359 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:13:06.360 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:52.085234+0000", 2024-08-19T14:13:06.360 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.046100271999999998 2024-08-19T14:13:06.360 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.360 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.360 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:13:06.360 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:52.085251+0000", 2024-08-19T14:13:06.360 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6554000000000001e-05 2024-08-19T14:13:06.361 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.361 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.361 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:13:06.361 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:52.085307+0000", 2024-08-19T14:13:06.361 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.5977999999999998e-05 2024-08-19T14:13:06.361 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.361 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.362 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:13:06.362 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:52.085313+0000", 2024-08-19T14:13:06.362 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.2149999999999998e-06 2024-08-19T14:13:06.362 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.362 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.362 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:13:06.362 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:52.086005+0000", 2024-08-19T14:13:06.363 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00069173900000000003 2024-08-19T14:13:06.363 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:13:06.363 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:13:06.363 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:13:06.363 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.363 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.363 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.24580.0:9193 2.3 e107/102 2:c60fb45b:::benchmark_data_smithi045_46415_object9192:head v 107'602, mlcod=107'600)", 2024-08-19T14:13:06.364 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:12:52.041645+0000", 2024-08-19T14:13:06.364 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 14.219936930999999, 2024-08-19T14:13:06.364 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.044908590999999998, 2024-08-19T14:13:06.364 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:13:06.364 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:13:06.364 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:13:06.365 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:13:06.365 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.365 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:13:06.365 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:52.041645+0000", 2024-08-19T14:13:06.365 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:13:06.365 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.365 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.366 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:13:06.366 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:52.041645+0000", 2024-08-19T14:13:06.366 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:13:06.366 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.366 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.366 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:13:06.367 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:52.041645+0000", 2024-08-19T14:13:06.367 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.5799999999999999e-07 2024-08-19T14:13:06.367 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.367 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.367 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:13:06.368 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:52.041728+0000", 2024-08-19T14:13:06.368 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.2461999999999998e-05 2024-08-19T14:13:06.368 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.368 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.368 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:13:06.368 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:52.085353+0000", 2024-08-19T14:13:06.368 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.043625426000000002 2024-08-19T14:13:06.368 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.368 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.368 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:13:06.368 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:52.085361+0000", 2024-08-19T14:13:06.368 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.5660000000000001e-06 2024-08-19T14:13:06.368 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.369 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.369 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:13:06.369 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:52.085406+0000", 2024-08-19T14:13:06.369 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4888000000000001e-05 2024-08-19T14:13:06.369 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.369 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.369 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:13:06.369 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:52.085427+0000", 2024-08-19T14:13:06.369 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.1486999999999999e-05 2024-08-19T14:13:06.369 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.369 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.369 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:13:06.369 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:52.086519+0000", 2024-08-19T14:13:06.369 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.001092132 2024-08-19T14:13:06.370 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.370 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.370 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:13:06.370 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:52.086553+0000", 2024-08-19T14:13:06.370 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.3971999999999999e-05 2024-08-19T14:13:06.370 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:13:06.370 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:13:06.370 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:13:06.370 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.370 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.370 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:9197 2.1 2:8c219438:::benchmark_data_smithi045_46415_object9196:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-08-19T14:13:06.370 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:12:52.043059+0000", 2024-08-19T14:13:06.370 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 14.218522982, 2024-08-19T14:13:06.371 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.047505740999999997, 2024-08-19T14:13:06.371 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:13:06.371 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:13:06.371 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:13:06.371 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:13:06.371 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:13:06.371 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:13:06.371 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 9197 2024-08-19T14:13:06.371 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.371 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:13:06.371 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.371 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:13:06.371 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:52.043059+0000", 2024-08-19T14:13:06.371 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:13:06.372 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.372 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.372 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:13:06.372 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:52.043059+0000", 2024-08-19T14:13:06.372 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:13:06.372 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.372 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.372 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:13:06.372 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:52.043060+0000", 2024-08-19T14:13:06.372 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.9999999999999995e-07 2024-08-19T14:13:06.372 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.373 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.373 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:13:06.373 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:52.043097+0000", 2024-08-19T14:13:06.373 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7741999999999999e-05 2024-08-19T14:13:06.373 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.373 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.373 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:13:06.373 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:52.085248+0000", 2024-08-19T14:13:06.373 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.042150716999999997 2024-08-19T14:13:06.373 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.373 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.373 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:13:06.373 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:52.085262+0000", 2024-08-19T14:13:06.373 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3858e-05 2024-08-19T14:13:06.373 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.374 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.374 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:13:06.374 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:52.086034+0000", 2024-08-19T14:13:06.374 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00077186599999999996 2024-08-19T14:13:06.374 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.374 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.374 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:13:06.374 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:52.086145+0000", 2024-08-19T14:13:06.374 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000111223 2024-08-19T14:13:06.374 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.374 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.374 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,3", 2024-08-19T14:13:06.374 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:52.086276+0000", 2024-08-19T14:13:06.374 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013088800000000001 2024-08-19T14:13:06.375 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.375 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.375 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:13:06.375 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:52.087979+0000", 2024-08-19T14:13:06.375 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.001702788 2024-08-19T14:13:06.375 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.375 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.375 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:13:06.375 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:52.087990+0000", 2024-08-19T14:13:06.375 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1409999999999999e-05 2024-08-19T14:13:06.375 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.375 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.375 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:13:06.375 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:52.090465+0000", 2024-08-19T14:13:06.376 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0024745629999999999 2024-08-19T14:13:06.376 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.376 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.376 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:13:06.376 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:52.090519+0000", 2024-08-19T14:13:06.376 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.4240000000000002e-05 2024-08-19T14:13:06.376 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.376 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.376 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:13:06.376 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:52.090564+0000", 2024-08-19T14:13:06.376 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.5445999999999998e-05 2024-08-19T14:13:06.376 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:13:06.376 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:13:06.376 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:13:06.377 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.377 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.377 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:45090 2.e 2:7da9264d:::benchmark_data_smithi045_46415_object45089:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-08-19T14:13:06.377 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:13:05.392279+0000", 2024-08-19T14:13:06.377 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.86930220000000002, 2024-08-19T14:13:06.377 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.031050547000000001, 2024-08-19T14:13:06.377 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:13:06.377 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:13:06.377 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:13:06.377 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:13:06.377 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:13:06.377 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:13:06.377 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 45090 2024-08-19T14:13:06.377 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.378 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:13:06.378 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.378 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:13:06.378 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:05.392279+0000", 2024-08-19T14:13:06.378 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:13:06.378 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.378 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.378 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:13:06.378 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:05.392279+0000", 2024-08-19T14:13:06.378 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:13:06.378 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.378 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.378 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:13:06.378 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:05.392281+0000", 2024-08-19T14:13:06.379 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.494e-06 2024-08-19T14:13:06.379 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.379 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.379 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:13:06.379 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:05.392309+0000", 2024-08-19T14:13:06.379 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.8453e-05 2024-08-19T14:13:06.379 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.379 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.379 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:13:06.379 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:05.392320+0000", 2024-08-19T14:13:06.379 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0261000000000001e-05 2024-08-19T14:13:06.379 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.379 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.379 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:13:06.380 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:05.392326+0000", 2024-08-19T14:13:06.380 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.6259999999999997e-06 2024-08-19T14:13:06.380 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.380 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.380 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:13:06.380 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:05.392404+0000", 2024-08-19T14:13:06.380 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.7313999999999999e-05 2024-08-19T14:13:06.380 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.380 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.380 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:13:06.380 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:05.392539+0000", 2024-08-19T14:13:06.380 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000134933 2024-08-19T14:13:06.380 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.380 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.381 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,9", 2024-08-19T14:13:06.381 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:05.392682+0000", 2024-08-19T14:13:06.381 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0001438 2024-08-19T14:13:06.381 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.381 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.381 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:13:06.381 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:05.396009+0000", 2024-08-19T14:13:06.381 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.003326842 2024-08-19T14:13:06.381 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.381 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.381 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:13:06.381 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:05.400506+0000", 2024-08-19T14:13:06.381 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0044969600000000004 2024-08-19T14:13:06.381 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.382 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.382 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:13:06.382 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:05.423223+0000", 2024-08-19T14:13:06.382 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.022716755000000002 2024-08-19T14:13:06.382 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.382 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.382 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:13:06.382 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:05.423282+0000", 2024-08-19T14:13:06.382 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.8829000000000002e-05 2024-08-19T14:13:06.382 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.382 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.382 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:13:06.382 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:05.423330+0000", 2024-08-19T14:13:06.382 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.8279999999999999e-05 2024-08-19T14:13:06.383 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:13:06.383 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:13:06.383 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:13:06.383 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.383 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.383 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:45094 2.e 2:747ce0cd:::benchmark_data_smithi045_46415_object45093:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-08-19T14:13:06.383 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:13:05.393594+0000", 2024-08-19T14:13:06.383 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.867987803, 2024-08-19T14:13:06.383 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.030017827, 2024-08-19T14:13:06.383 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:13:06.383 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:13:06.383 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:13:06.383 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:13:06.383 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:13:06.384 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:13:06.384 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 45094 2024-08-19T14:13:06.384 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.384 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:13:06.384 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.384 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:13:06.384 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:05.393594+0000", 2024-08-19T14:13:06.384 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:13:06.384 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.384 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.384 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:13:06.385 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:05.393594+0000", 2024-08-19T14:13:06.385 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:13:06.385 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.385 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.385 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:13:06.385 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:05.393595+0000", 2024-08-19T14:13:06.385 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1990000000000001e-06 2024-08-19T14:13:06.385 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.385 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.385 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:13:06.385 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:05.393624+0000", 2024-08-19T14:13:06.385 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.8444000000000001e-05 2024-08-19T14:13:06.385 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.385 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.385 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:13:06.386 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:05.393637+0000", 2024-08-19T14:13:06.386 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3935e-05 2024-08-19T14:13:06.386 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.386 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.386 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:13:06.386 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:05.393645+0000", 2024-08-19T14:13:06.386 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.0910000000000004e-06 2024-08-19T14:13:06.386 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.386 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.386 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:13:06.387 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:05.394207+0000", 2024-08-19T14:13:06.387 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00056205100000000004 2024-08-19T14:13:06.387 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.387 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.387 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:13:06.387 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:05.394338+0000", 2024-08-19T14:13:06.387 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000131369 2024-08-19T14:13:06.387 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.387 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.387 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,9", 2024-08-19T14:13:06.387 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:05.394481+0000", 2024-08-19T14:13:06.387 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000142962 2024-08-19T14:13:06.387 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.387 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.387 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:13:06.388 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:05.397360+0000", 2024-08-19T14:13:06.388 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0028792010000000001 2024-08-19T14:13:06.388 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.388 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.388 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:13:06.388 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:05.401965+0000", 2024-08-19T14:13:06.388 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0046053159999999999 2024-08-19T14:13:06.388 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.388 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.388 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:13:06.388 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:05.423369+0000", 2024-08-19T14:13:06.388 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.021403741 2024-08-19T14:13:06.388 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.388 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.389 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:13:06.389 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:05.423574+0000", 2024-08-19T14:13:06.389 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00020492399999999999 2024-08-19T14:13:06.389 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:06.389 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:06.389 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:13:06.389 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:05.423612+0000", 2024-08-19T14:13:06.389 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7594e-05 2024-08-19T14:13:06.389 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:13:06.389 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:13:06.389 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:13:06.389 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:13:06.389 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:13:06.389 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:13:06.421 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:06.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:06.423 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:13:06.525 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:13:06.552 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:06.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:06.554 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:13:06.623 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is unset 2024-08-19T14:13:06.654 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:13:06.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:06.693 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:06.694 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:13:06.795 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:13:06.824 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:06.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:06.825 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:13:06.831 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 17 15 48234 48219 177.26 139.938 0.00333454 0.00563615 2024-08-19T14:13:06.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:06 smithi031 ceph-mon[16336]: osdmap e116: 12 total, 12 up, 12 in 2024-08-19T14:13:06.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:06 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2346534819' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:13:06.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:06 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:13:06.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:06 smithi031 ceph-mon[16336]: 1.9 scrub starts 2024-08-19T14:13:06.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:06 smithi031 ceph-mon[16336]: 1.9 scrub ok 2024-08-19T14:13:06.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:06 smithi031 ceph-mon[16336]: 1.7 scrub starts 2024-08-19T14:13:06.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:06 smithi031 ceph-mon[16336]: 1.7 scrub ok 2024-08-19T14:13:06.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:06 smithi031 ceph-mon[16336]: 1.8 scrub starts 2024-08-19T14:13:06.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:06 smithi031 ceph-mon[16336]: 1.8 scrub ok 2024-08-19T14:13:06.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:06 smithi031 ceph-mon[16336]: 1.a scrub starts 2024-08-19T14:13:06.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:06 smithi031 ceph-mon[16336]: 1.a scrub ok 2024-08-19T14:13:06.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:06 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mon-a[16332]: 2024-08-19T14:13:06.832+0000 7f2ff503e640 -1 mon.a@0(leader).osd e117 definitely_dead 0 2024-08-19T14:13:06.927 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:13:06.953 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:06.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:06.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:13:07.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:06 smithi032 ceph-mon[20004]: osdmap e116: 12 total, 12 up, 12 in 2024-08-19T14:13:07.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:06 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2346534819' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:13:07.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:06 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:13:07.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:06 smithi032 ceph-mon[20004]: 1.9 scrub starts 2024-08-19T14:13:07.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:06 smithi032 ceph-mon[20004]: 1.9 scrub ok 2024-08-19T14:13:07.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:06 smithi032 ceph-mon[20004]: 1.7 scrub starts 2024-08-19T14:13:07.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:06 smithi032 ceph-mon[20004]: 1.7 scrub ok 2024-08-19T14:13:07.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:06 smithi032 ceph-mon[20004]: 1.8 scrub starts 2024-08-19T14:13:07.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:06 smithi032 ceph-mon[20004]: 1.8 scrub ok 2024-08-19T14:13:07.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:06 smithi032 ceph-mon[20004]: 1.a scrub starts 2024-08-19T14:13:07.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:06 smithi032 ceph-mon[20004]: 1.a scrub ok 2024-08-19T14:13:07.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:06 smithi045 ceph-mon[20143]: osdmap e116: 12 total, 12 up, 12 in 2024-08-19T14:13:07.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:06 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2346534819' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:13:07.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:06 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:13:07.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:06 smithi045 ceph-mon[20143]: 1.9 scrub starts 2024-08-19T14:13:07.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:06 smithi045 ceph-mon[20143]: 1.9 scrub ok 2024-08-19T14:13:07.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:06 smithi045 ceph-mon[20143]: 1.7 scrub starts 2024-08-19T14:13:07.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:06 smithi045 ceph-mon[20143]: 1.7 scrub ok 2024-08-19T14:13:07.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:06 smithi045 ceph-mon[20143]: 1.8 scrub starts 2024-08-19T14:13:07.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:06 smithi045 ceph-mon[20143]: 1.8 scrub ok 2024-08-19T14:13:07.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:06 smithi045 ceph-mon[20143]: 1.a scrub starts 2024-08-19T14:13:07.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:06 smithi045 ceph-mon[20143]: 1.a scrub ok 2024-08-19T14:13:07.055 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:13:07.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:07.078 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:07.078 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:13:07.105 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:13:07.179 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:13:07.201 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:07.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:07.202 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:13:07.303 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:13:07.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:07.334 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:07.334 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:13:07.435 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:13:07.457 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:07.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:07.458 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:13:07.559 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:13:07.594 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:07.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:07.595 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:13:07.629 INFO:teuthology.orchestra.run.smithi031.stderr:marked out osd.10. 2024-08-19T14:13:07.645 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-08-19T14:13:07.695 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:13:07.716 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:07.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:07.717 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:13:07.818 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:13:07.831 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 18 16 49511 49495 171.843 79.75 0.00234101 0.00575692 2024-08-19T14:13:07.841 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:07.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:07.842 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:13:07.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:07 smithi031 ceph-mon[16336]: pgmap v312: 27 pgs: 1 peering, 7 active+undersized+remapped, 2 remapped+peering, 17 active+clean; 2.5 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 176 MiB/s wr, 2.81k op/s; 0 B/s, 0 objects/s recovering 2024-08-19T14:13:07.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:07 smithi031 ceph-mon[16336]: 1.1 scrub starts 2024-08-19T14:13:07.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:07 smithi031 ceph-mon[16336]: 1.1 scrub ok 2024-08-19T14:13:07.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:07 smithi031 ceph-mon[16336]: 1.3 scrub starts 2024-08-19T14:13:07.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:07 smithi031 ceph-mon[16336]: 1.3 scrub ok 2024-08-19T14:13:07.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:07 smithi031 ceph-mon[16336]: 1.4 scrub starts 2024-08-19T14:13:07.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:07 smithi031 ceph-mon[16336]: 1.4 scrub ok 2024-08-19T14:13:07.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:07 smithi031 ceph-mon[16336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:13:07.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:07 smithi031 ceph-mon[16336]: Cluster is now healthy 2024-08-19T14:13:07.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:07 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:13:07.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:07 smithi031 ceph-mon[16336]: osdmap e117: 12 total, 12 up, 12 in 2024-08-19T14:13:07.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:07 smithi031 ceph-mon[16336]: 1.2 scrub starts 2024-08-19T14:13:07.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:07 smithi031 ceph-mon[16336]: 1.2 scrub ok 2024-08-19T14:13:07.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:07 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1255409381' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["10"]}]: dispatch 2024-08-19T14:13:07.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:07 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["10"]}]: dispatch 2024-08-19T14:13:07.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:07 smithi031 ceph-mon[16336]: Client client.admin marked osd.10 out, while it was still marked up 2024-08-19T14:13:07.943 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:13:07.965 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:07.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:07.966 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:13:08.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:07 smithi032 ceph-mon[20004]: pgmap v312: 27 pgs: 1 peering, 7 active+undersized+remapped, 2 remapped+peering, 17 active+clean; 2.5 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 176 MiB/s wr, 2.81k op/s; 0 B/s, 0 objects/s recovering 2024-08-19T14:13:08.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:07 smithi032 ceph-mon[20004]: 1.1 scrub starts 2024-08-19T14:13:08.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:07 smithi032 ceph-mon[20004]: 1.1 scrub ok 2024-08-19T14:13:08.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:07 smithi032 ceph-mon[20004]: 1.3 scrub starts 2024-08-19T14:13:08.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:07 smithi032 ceph-mon[20004]: 1.3 scrub ok 2024-08-19T14:13:08.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:07 smithi032 ceph-mon[20004]: 1.4 scrub starts 2024-08-19T14:13:08.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:07 smithi032 ceph-mon[20004]: 1.4 scrub ok 2024-08-19T14:13:08.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:07 smithi032 ceph-mon[20004]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:13:08.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:07 smithi032 ceph-mon[20004]: Cluster is now healthy 2024-08-19T14:13:08.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:07 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:13:08.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:07 smithi032 ceph-mon[20004]: osdmap e117: 12 total, 12 up, 12 in 2024-08-19T14:13:08.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:07 smithi032 ceph-mon[20004]: 1.2 scrub starts 2024-08-19T14:13:08.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:07 smithi032 ceph-mon[20004]: 1.2 scrub ok 2024-08-19T14:13:08.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:07 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1255409381' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["10"]}]: dispatch 2024-08-19T14:13:08.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:07 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["10"]}]: dispatch 2024-08-19T14:13:08.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:07 smithi032 ceph-mon[20004]: Client client.admin marked osd.10 out, while it was still marked up 2024-08-19T14:13:08.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:07 smithi045 ceph-mon[20143]: pgmap v312: 27 pgs: 1 peering, 7 active+undersized+remapped, 2 remapped+peering, 17 active+clean; 2.5 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 176 MiB/s wr, 2.81k op/s; 0 B/s, 0 objects/s recovering 2024-08-19T14:13:08.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:07 smithi045 ceph-mon[20143]: 1.1 scrub starts 2024-08-19T14:13:08.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:07 smithi045 ceph-mon[20143]: 1.1 scrub ok 2024-08-19T14:13:08.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:07 smithi045 ceph-mon[20143]: 1.3 scrub starts 2024-08-19T14:13:08.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:07 smithi045 ceph-mon[20143]: 1.3 scrub ok 2024-08-19T14:13:08.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:07 smithi045 ceph-mon[20143]: 1.4 scrub starts 2024-08-19T14:13:08.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:07 smithi045 ceph-mon[20143]: 1.4 scrub ok 2024-08-19T14:13:08.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:07 smithi045 ceph-mon[20143]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:13:08.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:07 smithi045 ceph-mon[20143]: Cluster is now healthy 2024-08-19T14:13:08.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:07 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:13:08.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:07 smithi045 ceph-mon[20143]: osdmap e117: 12 total, 12 up, 12 in 2024-08-19T14:13:08.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:07 smithi045 ceph-mon[20143]: 1.2 scrub starts 2024-08-19T14:13:08.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:07 smithi045 ceph-mon[20143]: 1.2 scrub ok 2024-08-19T14:13:08.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:07 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1255409381' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["10"]}]: dispatch 2024-08-19T14:13:08.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:07 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["10"]}]: dispatch 2024-08-19T14:13:08.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:07 smithi045 ceph-mon[20143]: Client client.admin marked osd.10 out, while it was still marked up 2024-08-19T14:13:08.067 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:13:08.093 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:08.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:08.094 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:13:08.196 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:13:08.221 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:08.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:08.222 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:13:08.323 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:13:08.348 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:08.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:08.350 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:13:08.451 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:13:08.476 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:08.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:08.478 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:13:08.578 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:13:08.600 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:08.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:08.602 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:13:08.703 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:13:08.724 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:08.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:08.725 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:13:08.826 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:13:08.831 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 19 16 49511 49495 162.798 0 - 0.00575692 2024-08-19T14:13:08.848 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:08.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:08.849 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:13:08.865 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:08 smithi031 ceph-mon[16336]: 1.6 scrub starts 2024-08-19T14:13:08.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:08 smithi031 ceph-mon[16336]: 1.6 scrub ok 2024-08-19T14:13:08.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:08 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["10"]}]': finished 2024-08-19T14:13:08.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:08 smithi031 ceph-mon[16336]: osdmap e118: 12 total, 12 up, 11 in 2024-08-19T14:13:08.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:08 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/952453255' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:13:08.950 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:13:08.965 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:13:08.966 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:13:08.966 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:13:08.966 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:13:08.966 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:13:08.966 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:13:08.966 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:13:08.966 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:13:08.967 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:13:08.967 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:13:08.967 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:13:08.967 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:13:08.967 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:13:08.967 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:13:08.967 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:13:08.968 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:13:08.968 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:13:08.968 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:13:08.968 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:13:08.968 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:13:08.968 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:13:08.968 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:13:08.968 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:13:08.968 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:13:08.975 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:08.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:08.976 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:13:09.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:08 smithi032 ceph-mon[20004]: 1.6 scrub starts 2024-08-19T14:13:09.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:08 smithi032 ceph-mon[20004]: 1.6 scrub ok 2024-08-19T14:13:09.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:08 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["10"]}]': finished 2024-08-19T14:13:09.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:08 smithi032 ceph-mon[20004]: osdmap e118: 12 total, 12 up, 11 in 2024-08-19T14:13:09.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:08 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/952453255' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:13:09.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:08 smithi045 ceph-mon[20143]: 1.6 scrub starts 2024-08-19T14:13:09.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:08 smithi045 ceph-mon[20143]: 1.6 scrub ok 2024-08-19T14:13:09.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:08 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["10"]}]': finished 2024-08-19T14:13:09.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:08 smithi045 ceph-mon[20143]: osdmap e118: 12 total, 12 up, 11 in 2024-08-19T14:13:09.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:08 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/952453255' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:13:09.077 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:13:09.102 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:09.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:09.103 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:13:09.204 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:13:09.229 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:09.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:09.230 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:13:09.331 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:13:09.356 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:09.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:09.357 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:13:09.458 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:13:09.481 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:09.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:09.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:13:09.583 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:13:09.609 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:09.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:09.610 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:13:09.713 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:13:09.742 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:09.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:09.743 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:13:09.831 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:2024-08-19T14:13:09.832017+0000 min lat: 0.00106853 max lat: 2.15749 avg lat: 0.00644033 2024-08-19T14:13:09.831 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-19T14:13:09.831 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 20 16 49666 49650 155.142 4.84375 0.011161 0.00644033 2024-08-19T14:13:09.844 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:13:09.844 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:09 smithi031 ceph-mon[16336]: pgmap v315: 27 pgs: 2 peering, 2 active+undersized+remapped, 2 remapped+peering, 21 active+clean; 2.7 GiB data, 7.7 GiB used, 976 GiB / 983 GiB avail; 10 KiB/s rd, 145 MiB/s wr, 2.34k op/s; 268 KiB/s, 0 objects/s recovering 2024-08-19T14:13:09.845 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:09 smithi031 ceph-mon[16336]: osdmap e119: 12 total, 12 up, 11 in 2024-08-19T14:13:09.845 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:09 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:13:09.845 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:09 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:13:09.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:09.883 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:09.883 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:13:09.984 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:13:10.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:09 smithi032 ceph-mon[20004]: pgmap v315: 27 pgs: 2 peering, 2 active+undersized+remapped, 2 remapped+peering, 21 active+clean; 2.7 GiB data, 7.7 GiB used, 976 GiB / 983 GiB avail; 10 KiB/s rd, 145 MiB/s wr, 2.34k op/s; 268 KiB/s, 0 objects/s recovering 2024-08-19T14:13:10.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:09 smithi032 ceph-mon[20004]: osdmap e119: 12 total, 12 up, 11 in 2024-08-19T14:13:10.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:09 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:13:10.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:09 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:13:10.011 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:10.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:10.012 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:13:10.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:09 smithi045 ceph-mon[20143]: pgmap v315: 27 pgs: 2 peering, 2 active+undersized+remapped, 2 remapped+peering, 21 active+clean; 2.7 GiB data, 7.7 GiB used, 976 GiB / 983 GiB avail; 10 KiB/s rd, 145 MiB/s wr, 2.34k op/s; 268 KiB/s, 0 objects/s recovering 2024-08-19T14:13:10.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:09 smithi045 ceph-mon[20143]: osdmap e119: 12 total, 12 up, 11 in 2024-08-19T14:13:10.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:09 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:13:10.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:09 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:13:10.071 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-19T14:13:10.113 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:13:10.142 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:10.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:10.143 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:13:10.244 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:13:10.293 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:10.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:10.294 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:13:10.395 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:13:10.422 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:13:10.424 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:10.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:10.425 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:13:10.526 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:13:10.555 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:10.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:10.556 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:13:10.657 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:13:10.693 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:10.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:10.694 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:13:10.727 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:13:10.795 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:13:10.831 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 21 16 51142 51126 152.147 92.25 0.00882895 0.0065673 2024-08-19T14:13:10.835 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:10.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:10.836 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:13:10.937 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:13:10.966 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:10.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:10.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:13:10.969 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:10 smithi031 ceph-mon[16336]: osdmap e120: 12 total, 12 up, 11 in 2024-08-19T14:13:11.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:10 smithi032 ceph-mon[20004]: osdmap e120: 12 total, 12 up, 11 in 2024-08-19T14:13:11.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:10 smithi045 ceph-mon[20143]: osdmap e120: 12 total, 12 up, 11 in 2024-08-19T14:13:11.068 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:13:11.079 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:13:11.079 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:13:11.112 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:11.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:11.113 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:13:11.214 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:13:11.240 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:11.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:11.241 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:13:11.342 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:13:11.381 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:11.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:11.382 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:13:11.483 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:13:11.507 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:11.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:11.508 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:13:11.609 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:13:11.649 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:11.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:11.650 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:13:11.751 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:13:11.752 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:11 smithi031 ceph-mon[16336]: pgmap v318: 27 pgs: 3 peering, 2 remapped+peering, 2 unknown, 20 active+clean; 2.9 GiB data, 8.6 GiB used, 975 GiB / 983 GiB avail; 102 MiB/s wr, 1.62k op/s 2024-08-19T14:13:11.753 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:11 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:13:11.753 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:11 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:13:11.753 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:11 smithi031 ceph-mon[16336]: osdmap e121: 12 total, 12 up, 11 in 2024-08-19T14:13:11.780 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:11.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:11.781 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:13:11.831 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 22 16 52709 52693 149.682 97.9375 0.00787438 0.00667541 2024-08-19T14:13:11.882 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:13:11.909 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:11.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:11.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:13:12.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:11 smithi032 ceph-mon[20004]: pgmap v318: 27 pgs: 3 peering, 2 remapped+peering, 2 unknown, 20 active+clean; 2.9 GiB data, 8.6 GiB used, 975 GiB / 983 GiB avail; 102 MiB/s wr, 1.62k op/s 2024-08-19T14:13:12.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:11 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:13:12.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:11 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:13:12.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:11 smithi032 ceph-mon[20004]: osdmap e121: 12 total, 12 up, 11 in 2024-08-19T14:13:12.011 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:13:12.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:11 smithi045 ceph-mon[20143]: pgmap v318: 27 pgs: 3 peering, 2 remapped+peering, 2 unknown, 20 active+clean; 2.9 GiB data, 8.6 GiB used, 975 GiB / 983 GiB avail; 102 MiB/s wr, 1.62k op/s 2024-08-19T14:13:12.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:11 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:13:12.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:11 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:13:12.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:11 smithi045 ceph-mon[20143]: osdmap e121: 12 total, 12 up, 11 in 2024-08-19T14:13:12.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:12.052 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:12.052 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:13:12.093 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-08-19T14:13:12.153 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:13:12.179 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:12.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:12.180 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:13:12.281 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:13:12.325 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:12.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:12.326 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:13:12.359 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:13:12.428 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:13:12.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:12.508 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:12.509 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:13:12.610 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:13:12.620 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:13:12.634 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:12.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:12.635 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:13:12.736 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:13:12.764 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:12.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:12.765 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:13:12.832 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 23 15 53991 53976 146.661 80.1875 0.00907153 0.00681351 2024-08-19T14:13:12.866 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:13:12.894 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:12.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:12.896 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:13:12.921 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:12 smithi031 ceph-mon[16336]: osdmap e122: 12 total, 12 up, 11 in 2024-08-19T14:13:12.996 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:13:13.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:12 smithi032 ceph-mon[20004]: osdmap e122: 12 total, 12 up, 11 in 2024-08-19T14:13:13.023 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:13.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:13.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:13:13.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:12 smithi045 ceph-mon[20143]: osdmap e122: 12 total, 12 up, 11 in 2024-08-19T14:13:13.126 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:13:13.155 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:13.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:13.157 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:13:13.258 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:13:13.286 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:13.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:13.287 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:13:13.388 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:13:13.428 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:13.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:13.429 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:13:13.464 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:13:13.464 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:13:13.530 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:13:13.561 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:13.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:13.562 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:13:13.663 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:13:13.678 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is set 2024-08-19T14:13:13.690 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:13.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:13.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:13:13.792 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:13:13.818 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:13.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:13.819 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:13:13.831 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 24 16 55623 55607 144.797 101.938 0.00677346 0.00690029 2024-08-19T14:13:13.920 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:13:13.954 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:13.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:13.955 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:13:14.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:13 smithi032 ceph-mon[20004]: pgmap v321: 27 pgs: 3 peering, 2 remapped+peering, 2 unknown, 20 active+clean; 3.1 GiB data, 9.0 GiB used, 974 GiB / 983 GiB avail; 81 MiB/s wr, 1.29k op/s 2024-08-19T14:13:14.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:13 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1770189182' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:13:14.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:13 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:13:14.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:13 smithi045 ceph-mon[20143]: pgmap v321: 27 pgs: 3 peering, 2 remapped+peering, 2 unknown, 20 active+clean; 3.1 GiB data, 9.0 GiB used, 974 GiB / 983 GiB avail; 81 MiB/s wr, 1.29k op/s 2024-08-19T14:13:14.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:13 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1770189182' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:13:14.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:13 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:13:14.055 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:13:14.079 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:13 smithi031 ceph-mon[16336]: pgmap v321: 27 pgs: 3 peering, 2 remapped+peering, 2 unknown, 20 active+clean; 3.1 GiB data, 9.0 GiB used, 974 GiB / 983 GiB avail; 81 MiB/s wr, 1.29k op/s 2024-08-19T14:13:14.079 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:13 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1770189182' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:13:14.080 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:13 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:13:14.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:14.087 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:14.087 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:13:14.188 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:13:14.217 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:14.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:14.218 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:13:14.320 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:13:14.345 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:14.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:14.346 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:13:14.447 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:13:14.474 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:14.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:14.475 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:13:14.576 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:13:14.603 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:14.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:14.604 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:13:14.635 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:13:14.705 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:13:14.732 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:14.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:14.733 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:13:14.832 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 25 16 57424 57408 143.507 112.562 0.00419072 0.00696246 2024-08-19T14:13:14.834 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:13:14.863 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:14.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:14.865 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:13:14.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:14 smithi031 ceph-mon[16336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:13:14.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:14 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:13:14.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:14 smithi031 ceph-mon[16336]: osdmap e123: 12 total, 12 up, 11 in 2024-08-19T14:13:14.904 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11] out_osds: [10] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-08-19T14:13:14.904 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-19T14:13:14.905 INFO:tasks.thrashosds.thrasher:Removing osd 6, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11] 2024-08-19T14:13:14.905 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd out 6 2024-08-19T14:13:14.966 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:13:15.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:14 smithi032 ceph-mon[20004]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:13:15.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:14 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:13:15.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:14 smithi032 ceph-mon[20004]: osdmap e123: 12 total, 12 up, 11 in 2024-08-19T14:13:15.007 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:15.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:15.008 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:13:15.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:14 smithi045 ceph-mon[20143]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:13:15.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:14 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:13:15.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:14 smithi045 ceph-mon[20143]: osdmap e123: 12 total, 12 up, 11 in 2024-08-19T14:13:15.109 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:13:15.134 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:15.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:15.135 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:13:15.142 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-08-19T14:13:15.236 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:13:15.259 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:15.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:15.260 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:13:15.360 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:13:15.386 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:15.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:15.387 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:13:15.488 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:13:15.514 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:15.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:15.515 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:13:15.616 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:13:15.659 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:15.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:15.660 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:13:15.698 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:13:15.702 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:13:15.702 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:13:15.741 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:13:15.741 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:13:15.761 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:13:15.775 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:15 smithi031 ceph-mon[16336]: pgmap v323: 27 pgs: 2 active+recovering+undersized+remapped, 1 peering, 2 remapped+peering, 22 active+clean; 3.2 GiB data, 9.5 GiB used, 974 GiB / 983 GiB avail; 73 MiB/s wr, 1.16k op/s; 5785/155643 objects misplaced (3.717%); 5.4 MiB/s, 87 objects/s recovering 2024-08-19T14:13:15.776 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:15 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1783260005' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:13:15.776 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:13:15.777 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:13:15.777 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:13:15.787 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:15.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:15.788 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:13:15.814 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:13:15.814 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:13:15.831 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 26 15 58758 58743 141.197 83.4375 0.00355767 0.0070771 2024-08-19T14:13:15.845 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:13:15.845 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:13:15.878 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:13:15.878 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:13:15.889 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:13:15.911 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:13:15.912 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:13:15.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:15.923 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:15.923 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:13:15.948 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:13:15.948 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:13:15.981 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:13:15.981 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:13:16.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:15 smithi032 ceph-mon[20004]: pgmap v323: 27 pgs: 2 active+recovering+undersized+remapped, 1 peering, 2 remapped+peering, 22 active+clean; 3.2 GiB data, 9.5 GiB used, 974 GiB / 983 GiB avail; 73 MiB/s wr, 1.16k op/s; 5785/155643 objects misplaced (3.717%); 5.4 MiB/s, 87 objects/s recovering 2024-08-19T14:13:16.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:15 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1783260005' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:13:16.021 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:13:16.021 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:13:16.023 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:13:16.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:15 smithi045 ceph-mon[20143]: pgmap v323: 27 pgs: 2 active+recovering+undersized+remapped, 1 peering, 2 remapped+peering, 22 active+clean; 3.2 GiB data, 9.5 GiB used, 974 GiB / 983 GiB avail; 73 MiB/s wr, 1.16k op/s; 5785/155643 objects misplaced (3.717%); 5.4 MiB/s, 87 objects/s recovering 2024-08-19T14:13:16.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:15 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1783260005' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:13:16.049 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:16.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:16.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:13:16.057 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:13:16.057 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:13:16.089 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:13:16.089 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:13:16.152 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:13:16.178 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:16.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:16.179 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:13:16.280 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:13:16.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:16.319 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:16.319 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:13:16.420 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:13:16.461 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:16.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:16.462 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:13:16.563 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:13:16.592 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:16.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:16.593 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:13:16.605 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-19T14:13:16.694 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:13:16.697 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:16 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:13:16.698 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:16 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:13:16.698 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:16 smithi031 ceph-mon[16336]: osdmap e124: 12 total, 12 up, 11 in 2024-08-19T14:13:16.698 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:16 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:13:16.698 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:16 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:13:16.736 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:16.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:16.737 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:13:16.832 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 27 16 60318 60302 139.576 97.4375 0.00995822 0.00715903 2024-08-19T14:13:16.837 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:13:16.873 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:16.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:16.874 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:13:16.975 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:13:17.002 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:17.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:17.004 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:13:17.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:16 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:13:17.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:16 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:13:17.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:16 smithi032 ceph-mon[20004]: osdmap e124: 12 total, 12 up, 11 in 2024-08-19T14:13:17.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:16 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:13:17.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:16 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:13:17.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:16 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:13:17.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:16 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:13:17.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:16 smithi045 ceph-mon[20143]: osdmap e124: 12 total, 12 up, 11 in 2024-08-19T14:13:17.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:16 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:13:17.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:16 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:13:17.100 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:13:17.100 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 20, 2024-08-19T14:13:17.100 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 600, 2024-08-19T14:13:17.100 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [ 2024-08-19T14:13:17.100 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.100 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.24580.0:19 2.a e106/102 2:58ae6189:::benchmark_data_smithi045_46415_object18:head v 106'1, mlcod=0'0)", 2024-08-19T14:13:17.100 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:12:49.836929+0000", 2024-08-19T14:13:17.101 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 27.26140316, 2024-08-19T14:13:17.101 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.043536815999999999, 2024-08-19T14:13:17.101 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:13:17.101 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:13:17.101 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:13:17.101 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:13:17.101 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.101 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:13:17.101 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.836929+0000", 2024-08-19T14:13:17.101 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:13:17.101 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.101 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.101 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:13:17.101 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.836929+0000", 2024-08-19T14:13:17.102 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:13:17.102 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.102 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.102 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:13:17.102 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.836930+0000", 2024-08-19T14:13:17.102 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.153e-06 2024-08-19T14:13:17.102 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.102 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.102 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:13:17.102 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.879672+0000", 2024-08-19T14:13:17.102 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.042742044999999999 2024-08-19T14:13:17.102 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.102 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.102 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:13:17.103 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.879699+0000", 2024-08-19T14:13:17.103 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.6859000000000002e-05 2024-08-19T14:13:17.103 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.103 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.103 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:13:17.103 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.879709+0000", 2024-08-19T14:13:17.103 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.5489999999999995e-06 2024-08-19T14:13:17.103 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.103 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.103 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:13:17.103 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.879799+0000", 2024-08-19T14:13:17.103 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.0727000000000001e-05 2024-08-19T14:13:17.103 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.104 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.104 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:13:17.104 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.879821+0000", 2024-08-19T14:13:17.104 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.2103000000000002e-05 2024-08-19T14:13:17.104 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.104 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.104 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:13:17.104 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.880421+0000", 2024-08-19T14:13:17.104 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00059914699999999996 2024-08-19T14:13:17.104 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.104 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.104 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:13:17.104 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:49.880466+0000", 2024-08-19T14:13:17.105 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.5232999999999999e-05 2024-08-19T14:13:17.105 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:13:17.105 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:13:17.105 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:13:17.105 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.105 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.105 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.24580.0:12477 2.1 e107/102 2:8ac98501:::benchmark_data_smithi045_46415_object12476:head v 107'734, mlcod=107'732)", 2024-08-19T14:13:17.105 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:12:53.141192+0000", 2024-08-19T14:13:17.105 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 23.95714014, 2024-08-19T14:13:17.105 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.019820989000000001, 2024-08-19T14:13:17.105 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:13:17.105 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:13:17.105 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:13:17.105 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:13:17.106 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.106 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:13:17.106 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:53.141192+0000", 2024-08-19T14:13:17.106 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:13:17.106 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.106 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.106 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:13:17.106 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:53.141192+0000", 2024-08-19T14:13:17.106 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:13:17.106 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.106 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.106 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:13:17.106 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:53.141192+0000", 2024-08-19T14:13:17.107 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.9099999999999999e-07 2024-08-19T14:13:17.107 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.107 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.107 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:13:17.107 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:53.141214+0000", 2024-08-19T14:13:17.107 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.1695000000000002e-05 2024-08-19T14:13:17.107 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.107 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.107 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:13:17.107 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:53.141222+0000", 2024-08-19T14:13:17.107 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.2300000000000008e-06 2024-08-19T14:13:17.107 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.107 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.108 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:13:17.108 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:53.141228+0000", 2024-08-19T14:13:17.108 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1900000000000003e-06 2024-08-19T14:13:17.108 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.108 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.108 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:13:17.108 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:53.142872+0000", 2024-08-19T14:13:17.108 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0016448560000000001 2024-08-19T14:13:17.108 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.108 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.108 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:13:17.108 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:53.142893+0000", 2024-08-19T14:13:17.108 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.0434000000000001e-05 2024-08-19T14:13:17.109 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.109 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.109 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:13:17.109 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:53.160979+0000", 2024-08-19T14:13:17.109 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.018085878999999999 2024-08-19T14:13:17.109 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.109 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.109 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:13:17.109 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:53.161013+0000", 2024-08-19T14:13:17.109 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.4313999999999997e-05 2024-08-19T14:13:17.109 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:13:17.109 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:13:17.109 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:13:17.109 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.110 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.110 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.24580.0:12488 2.9 e107/102 2:9258fb20:::benchmark_data_smithi045_46415_object12487:head v 107'776, mlcod=107'774)", 2024-08-19T14:13:17.110 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:12:53.141402+0000", 2024-08-19T14:13:17.110 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 23.956930316000001, 2024-08-19T14:13:17.110 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.019573770000000001, 2024-08-19T14:13:17.110 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:13:17.110 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:13:17.110 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:13:17.110 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:13:17.110 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.110 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:13:17.110 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:53.141402+0000", 2024-08-19T14:13:17.110 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:13:17.111 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.111 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.111 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:13:17.111 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:53.141402+0000", 2024-08-19T14:13:17.111 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:13:17.111 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.111 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.111 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:13:17.111 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:53.141402+0000", 2024-08-19T14:13:17.111 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4099999999999999e-07 2024-08-19T14:13:17.111 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.111 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.111 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:13:17.112 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:53.141463+0000", 2024-08-19T14:13:17.112 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.0955000000000002e-05 2024-08-19T14:13:17.112 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.112 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.112 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:13:17.112 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:53.141472+0000", 2024-08-19T14:13:17.112 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.4230000000000005e-06 2024-08-19T14:13:17.112 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.112 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.112 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:13:17.112 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:53.141478+0000", 2024-08-19T14:13:17.112 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.1029999999999998e-06 2024-08-19T14:13:17.112 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.112 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.113 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:13:17.113 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:53.143937+0000", 2024-08-19T14:13:17.113 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0024596209999999999 2024-08-19T14:13:17.113 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.113 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.113 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:13:17.113 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:53.143957+0000", 2024-08-19T14:13:17.113 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.9751000000000001e-05 2024-08-19T14:13:17.113 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.113 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.113 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:13:17.113 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:53.160924+0000", 2024-08-19T14:13:17.113 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.016966933 2024-08-19T14:13:17.113 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.113 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.113 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:13:17.114 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:12:53.160976+0000", 2024-08-19T14:13:17.114 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1542999999999998e-05 2024-08-19T14:13:17.114 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:13:17.114 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:13:17.114 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:13:17.114 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.114 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.114 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.24580.0:45090 2.e e115/102 2:7da9264d:::benchmark_data_smithi045_46415_object45089:head v 115'2821, mlcod=115'2817)", 2024-08-19T14:13:17.114 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:13:05.396189+0000", 2024-08-19T14:13:17.114 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 11.70214367, 2024-08-19T14:13:17.114 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:13:17.116 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.025000402000000001, 2024-08-19T14:13:17.116 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:13:17.116 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:13:17.116 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:13:17.116 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:13:17.116 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.116 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:13:17.116 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:05.396189+0000", 2024-08-19T14:13:17.116 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:13:17.116 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.116 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.117 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:13:17.117 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:05.396189+0000", 2024-08-19T14:13:17.117 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:13:17.117 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.117 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.117 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:13:17.117 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:05.396189+0000", 2024-08-19T14:13:17.117 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.8999999999999998e-07 2024-08-19T14:13:17.117 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.117 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.117 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:13:17.117 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:05.396602+0000", 2024-08-19T14:13:17.117 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00041306600000000002 2024-08-19T14:13:17.117 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.118 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.118 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:13:17.118 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:05.396612+0000", 2024-08-19T14:13:17.118 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.9830000000000008e-06 2024-08-19T14:13:17.118 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.118 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.118 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:13:17.118 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:05.396618+0000", 2024-08-19T14:13:17.118 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.0549999999999996e-06 2024-08-19T14:13:17.118 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.118 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.118 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:13:17.119 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:05.403629+0000", 2024-08-19T14:13:17.119 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0070108560000000002 2024-08-19T14:13:17.119 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.119 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.119 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:13:17.119 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:05.403651+0000", 2024-08-19T14:13:17.119 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.2493e-05 2024-08-19T14:13:17.119 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.119 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.119 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:13:17.119 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:05.421028+0000", 2024-08-19T14:13:17.119 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.01737704 2024-08-19T14:13:17.119 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.120 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.120 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:13:17.120 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:05.421189+0000", 2024-08-19T14:13:17.120 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000160619 2024-08-19T14:13:17.120 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:13:17.120 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:13:17.120 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:13:17.120 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.120 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.120 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.24580.0:45095 2.6 e115/102 2:62e1878d:::benchmark_data_smithi045_46415_object45094:head v 115'2809, mlcod=115'2807)", 2024-08-19T14:13:17.120 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:13:05.398648+0000", 2024-08-19T14:13:17.120 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 11.699683805999999, 2024-08-19T14:13:17.120 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.019925136, 2024-08-19T14:13:17.121 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:13:17.121 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:13:17.121 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:13:17.121 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:13:17.121 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.121 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:13:17.121 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:05.398648+0000", 2024-08-19T14:13:17.121 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:13:17.121 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.121 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.121 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:13:17.121 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:05.398648+0000", 2024-08-19T14:13:17.121 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:13:17.121 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.122 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.122 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:13:17.122 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:05.398649+0000", 2024-08-19T14:13:17.122 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.8299999999999995e-07 2024-08-19T14:13:17.122 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.122 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.122 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:13:17.122 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:05.398674+0000", 2024-08-19T14:13:17.122 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.4468e-05 2024-08-19T14:13:17.122 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.122 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.122 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:13:17.122 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:05.398688+0000", 2024-08-19T14:13:17.122 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.438e-05 2024-08-19T14:13:17.123 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.123 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.123 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:13:17.123 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:05.398695+0000", 2024-08-19T14:13:17.123 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.1319999999999996e-06 2024-08-19T14:13:17.123 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.123 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.123 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:13:17.123 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:05.402095+0000", 2024-08-19T14:13:17.123 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0034002400000000001 2024-08-19T14:13:17.123 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.123 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.123 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:13:17.123 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:05.402145+0000", 2024-08-19T14:13:17.124 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.9815000000000003e-05 2024-08-19T14:13:17.124 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.124 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.124 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:13:17.124 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:05.418426+0000", 2024-08-19T14:13:17.124 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.016281137000000001 2024-08-19T14:13:17.124 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.124 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.124 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:13:17.124 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:05.418574+0000", 2024-08-19T14:13:17.124 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000147081 2024-08-19T14:13:17.124 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:13:17.124 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:13:17.124 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:13:17.125 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.125 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.125 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.24580.0:45094 2.e e115/102 2:747ce0cd:::benchmark_data_smithi045_46415_object45093:head v 115'2822, mlcod=115'2817)", 2024-08-19T14:13:17.125 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:13:05.400372+0000", 2024-08-19T14:13:17.125 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 11.697960532, 2024-08-19T14:13:17.125 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.020951278, 2024-08-19T14:13:17.125 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:13:17.125 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:13:17.125 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:13:17.125 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:13:17.125 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.125 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:13:17.125 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:05.400372+0000", 2024-08-19T14:13:17.125 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:13:17.126 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.126 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.126 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:13:17.126 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:05.400372+0000", 2024-08-19T14:13:17.126 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:13:17.126 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.126 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.126 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:13:17.126 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:05.400372+0000", 2024-08-19T14:13:17.126 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.6000000000000003e-07 2024-08-19T14:13:17.126 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.126 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.126 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:13:17.126 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:05.400397+0000", 2024-08-19T14:13:17.127 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.4895000000000001e-05 2024-08-19T14:13:17.127 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.127 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.127 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:13:17.127 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:05.400408+0000", 2024-08-19T14:13:17.127 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0550000000000001e-05 2024-08-19T14:13:17.127 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.127 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.127 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:13:17.127 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:05.400415+0000", 2024-08-19T14:13:17.127 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.8390000000000001e-06 2024-08-19T14:13:17.127 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.127 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.128 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:13:17.128 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:05.408345+0000", 2024-08-19T14:13:17.128 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0079305980000000005 2024-08-19T14:13:17.128 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.128 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.128 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:13:17.128 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:05.408367+0000", 2024-08-19T14:13:17.128 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.1373e-05 2024-08-19T14:13:17.128 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.128 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.128 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:13:17.128 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:05.421193+0000", 2024-08-19T14:13:17.128 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.012826641 2024-08-19T14:13:17.128 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.128 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.129 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:13:17.129 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:05.421323+0000", 2024-08-19T14:13:17.129 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000129622 2024-08-19T14:13:17.129 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:13:17.129 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:13:17.129 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:13:17.129 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.129 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.129 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "MOSDPGPush(2.7 124/119 [PushOp(2:e60ed909:::benchmark_data_smithi045_46415_object6717:head, version: 107'428, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:e60ed909:::benchmark_data_smithi045_46415_object6717:head@107'428, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-19T14:13:17.129 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:13:16.062408+0000", 2024-08-19T14:13:17.129 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 1.03592431, 2024-08-19T14:13:17.129 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.019885091000000001, 2024-08-19T14:13:17.129 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:13:17.130 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:13:17.130 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-08-19T14:13:17.130 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:13:17.130 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.130 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:13:17.130 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.062408+0000", 2024-08-19T14:13:17.130 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:13:17.130 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.130 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.130 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:13:17.130 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.062408+0000", 2024-08-19T14:13:17.130 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:13:17.130 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.130 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.131 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:13:17.131 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.062408+0000", 2024-08-19T14:13:17.131 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.5699999999999998e-07 2024-08-19T14:13:17.131 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.131 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.131 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:13:17.131 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.080891+0000", 2024-08-19T14:13:17.131 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.018482457000000001 2024-08-19T14:13:17.131 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.131 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.131 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:13:17.131 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.080912+0000", 2024-08-19T14:13:17.131 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.1401000000000001e-05 2024-08-19T14:13:17.131 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.132 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.132 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:13:17.132 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.080938+0000", 2024-08-19T14:13:17.132 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.5420999999999999e-05 2024-08-19T14:13:17.132 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.132 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.132 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:13:17.132 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.081042+0000", 2024-08-19T14:13:17.132 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00010440799999999999 2024-08-19T14:13:17.132 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.132 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.132 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:13:17.132 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.081054+0000", 2024-08-19T14:13:17.132 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2215000000000001e-05 2024-08-19T14:13:17.133 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.133 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.133 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:13:17.133 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.082293+0000", 2024-08-19T14:13:17.133 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.001238832 2024-08-19T14:13:17.133 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:13:17.133 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:13:17.133 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:13:17.133 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.133 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.133 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.24580.0:59130 2.7 e124/119 2:e71d1c4f:::benchmark_data_smithi045_46415_object59129:head v 124'3582, mlcod=107'427)", 2024-08-19T14:13:17.133 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:13:16.081152+0000", 2024-08-19T14:13:17.133 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 1.0171797300000001, 2024-08-19T14:13:17.133 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.054871600999999999, 2024-08-19T14:13:17.133 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:13:17.134 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:13:17.134 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:13:17.134 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:13:17.134 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.134 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:13:17.134 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.081152+0000", 2024-08-19T14:13:17.134 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:13:17.134 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.134 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.134 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:13:17.134 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.081152+0000", 2024-08-19T14:13:17.134 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:13:17.134 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.134 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.135 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:13:17.135 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.081153+0000", 2024-08-19T14:13:17.135 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.3799999999999998e-07 2024-08-19T14:13:17.135 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.135 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.135 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:13:17.135 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.100252+0000", 2024-08-19T14:13:17.135 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.019098909000000001 2024-08-19T14:13:17.135 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.135 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.135 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:13:17.135 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.100305+0000", 2024-08-19T14:13:17.135 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.2697e-05 2024-08-19T14:13:17.135 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.136 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.136 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:13:17.136 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.100319+0000", 2024-08-19T14:13:17.136 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4011e-05 2024-08-19T14:13:17.136 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.136 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.136 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:13:17.136 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.100703+0000", 2024-08-19T14:13:17.136 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00038468699999999998 2024-08-19T14:13:17.136 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.136 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.136 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:13:17.136 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.100729+0000", 2024-08-19T14:13:17.137 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.5479000000000001e-05 2024-08-19T14:13:17.137 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.137 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.137 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:13:17.137 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.135917+0000", 2024-08-19T14:13:17.137 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.035188812999999999 2024-08-19T14:13:17.137 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.137 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.137 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:13:17.137 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.136024+0000", 2024-08-19T14:13:17.137 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000106567 2024-08-19T14:13:17.137 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:13:17.137 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:13:17.137 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:13:17.137 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.138 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.138 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.24580.0:59143 2.7 e124/119 2:eb6376b2:::benchmark_data_smithi045_46415_object59142:head v 124'3583, mlcod=107'427)", 2024-08-19T14:13:17.138 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:13:16.100395+0000", 2024-08-19T14:13:17.138 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.99793763300000005, 2024-08-19T14:13:17.138 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.035677652999999997, 2024-08-19T14:13:17.138 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:13:17.138 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:13:17.138 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:13:17.138 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:13:17.138 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.138 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:13:17.138 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.100395+0000", 2024-08-19T14:13:17.138 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:13:17.139 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.139 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.139 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:13:17.139 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.100395+0000", 2024-08-19T14:13:17.139 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:13:17.139 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.139 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.139 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:13:17.139 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.100395+0000", 2024-08-19T14:13:17.139 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.2900000000000003e-07 2024-08-19T14:13:17.139 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.139 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.139 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:13:17.139 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.100446+0000", 2024-08-19T14:13:17.140 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1195e-05 2024-08-19T14:13:17.140 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.140 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.140 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:13:17.140 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.100485+0000", 2024-08-19T14:13:17.140 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.8263000000000003e-05 2024-08-19T14:13:17.140 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.140 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.140 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:13:17.140 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.100493+0000", 2024-08-19T14:13:17.140 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.9859999999999994e-06 2024-08-19T14:13:17.140 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.140 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.140 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:13:17.141 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.109916+0000", 2024-08-19T14:13:17.141 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0094235719999999999 2024-08-19T14:13:17.141 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.141 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.141 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:13:17.141 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.109938+0000", 2024-08-19T14:13:17.141 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.2056999999999999e-05 2024-08-19T14:13:17.141 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.141 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.141 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:13:17.141 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.136031+0000", 2024-08-19T14:13:17.141 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.026093057999999999 2024-08-19T14:13:17.141 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.141 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.141 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:13:17.142 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.136072+0000", 2024-08-19T14:13:17.142 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.0893000000000003e-05 2024-08-19T14:13:17.142 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:13:17.142 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:13:17.142 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:13:17.142 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.142 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.142 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "MOSDPGPush(2.7 124/119 [PushOp(2:e759fe6c:::benchmark_data_smithi045_46415_object6798:head, version: 107'429, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:e759fe6c:::benchmark_data_smithi045_46415_object6798:head@107'429, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-19T14:13:17.142 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:13:16.100508+0000", 2024-08-19T14:13:17.142 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.99782431900000002, 2024-08-19T14:13:17.142 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.035520154999999998, 2024-08-19T14:13:17.142 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:13:17.143 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:13:17.143 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-08-19T14:13:17.143 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:13:17.143 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.143 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:13:17.143 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.100508+0000", 2024-08-19T14:13:17.143 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:13:17.143 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.143 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.143 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:13:17.143 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.100508+0000", 2024-08-19T14:13:17.143 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:13:17.143 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.143 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.144 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:13:17.144 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.100508+0000", 2024-08-19T14:13:17.144 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.5699999999999998e-07 2024-08-19T14:13:17.144 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.144 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.144 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:13:17.144 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.100545+0000", 2024-08-19T14:13:17.144 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7061000000000003e-05 2024-08-19T14:13:17.144 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.144 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.144 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:13:17.144 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.100562+0000", 2024-08-19T14:13:17.144 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6915e-05 2024-08-19T14:13:17.144 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.145 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.145 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:13:17.145 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.100582+0000", 2024-08-19T14:13:17.145 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.9873000000000001e-05 2024-08-19T14:13:17.145 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.145 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.145 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:13:17.145 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.134380+0000", 2024-08-19T14:13:17.145 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.033797773000000003 2024-08-19T14:13:17.145 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.145 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.145 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:13:17.145 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.134393+0000", 2024-08-19T14:13:17.145 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2968e-05 2024-08-19T14:13:17.146 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.146 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.146 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:13:17.146 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.136028+0000", 2024-08-19T14:13:17.146 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.001635208 2024-08-19T14:13:17.146 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:13:17.146 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:13:17.146 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:13:17.146 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.146 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.146 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.24580.0:59175 2.7 e124/119 2:ec24474b:::benchmark_data_smithi045_46415_object59174:head v 124'3584, mlcod=107'428)", 2024-08-19T14:13:17.146 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:13:16.100599+0000", 2024-08-19T14:13:17.146 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.99773322600000003, 2024-08-19T14:13:17.146 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.035509361000000003, 2024-08-19T14:13:17.147 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:13:17.147 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:13:17.147 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:13:17.147 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:13:17.147 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.147 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:13:17.147 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.100599+0000", 2024-08-19T14:13:17.147 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:13:17.147 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.147 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.147 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:13:17.147 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.100599+0000", 2024-08-19T14:13:17.147 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:13:17.147 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.147 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.148 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:13:17.148 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.100599+0000", 2024-08-19T14:13:17.148 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.9700000000000003e-07 2024-08-19T14:13:17.148 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.148 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.148 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:13:17.148 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.100626+0000", 2024-08-19T14:13:17.148 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.6342e-05 2024-08-19T14:13:17.148 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.148 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.148 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:13:17.148 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.100636+0000", 2024-08-19T14:13:17.148 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0535e-05 2024-08-19T14:13:17.148 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.149 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.149 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:13:17.149 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.100641+0000", 2024-08-19T14:13:17.149 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7389999999999999e-06 2024-08-19T14:13:17.149 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.149 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.149 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:13:17.149 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.114316+0000", 2024-08-19T14:13:17.149 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.013674838 2024-08-19T14:13:17.149 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.149 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.149 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:13:17.149 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.114339+0000", 2024-08-19T14:13:17.150 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.3703e-05 2024-08-19T14:13:17.150 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.150 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.150 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:13:17.150 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.136075+0000", 2024-08-19T14:13:17.150 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.021735371 2024-08-19T14:13:17.150 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.150 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.150 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:13:17.150 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.136108+0000", 2024-08-19T14:13:17.150 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.3535999999999999e-05 2024-08-19T14:13:17.150 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:13:17.151 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:13:17.151 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:13:17.151 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.151 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.151 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.24580.0:59179 2.7 e124/119 2:e43474cb:::benchmark_data_smithi045_46415_object59178:head v 124'3585, mlcod=107'428)", 2024-08-19T14:13:17.151 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:13:16.103382+0000", 2024-08-19T14:13:17.151 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.994950102, 2024-08-19T14:13:17.151 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.032762377000000002, 2024-08-19T14:13:17.151 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:13:17.151 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:13:17.151 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:13:17.151 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:13:17.151 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.152 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:13:17.152 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.103382+0000", 2024-08-19T14:13:17.152 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:13:17.152 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.152 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.152 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:13:17.152 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.103382+0000", 2024-08-19T14:13:17.152 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:13:17.152 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.152 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.152 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:13:17.152 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.103383+0000", 2024-08-19T14:13:17.152 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.0500000000000002e-07 2024-08-19T14:13:17.152 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.153 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.153 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:13:17.153 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.103441+0000", 2024-08-19T14:13:17.153 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.8152999999999997e-05 2024-08-19T14:13:17.153 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.153 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.153 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:13:17.153 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.103452+0000", 2024-08-19T14:13:17.153 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1129e-05 2024-08-19T14:13:17.153 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.153 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.153 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:13:17.153 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.103460+0000", 2024-08-19T14:13:17.153 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.123e-06 2024-08-19T14:13:17.154 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.154 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.154 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:13:17.154 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.118094+0000", 2024-08-19T14:13:17.154 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.014633650999999999 2024-08-19T14:13:17.154 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.154 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.154 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:13:17.154 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.118127+0000", 2024-08-19T14:13:17.154 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.3142999999999998e-05 2024-08-19T14:13:17.154 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.154 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.154 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:13:17.154 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.136111+0000", 2024-08-19T14:13:17.155 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.017983971000000001 2024-08-19T14:13:17.155 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.155 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.155 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:13:17.155 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.136144+0000", 2024-08-19T14:13:17.155 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.3302000000000001e-05 2024-08-19T14:13:17.155 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:13:17.155 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:13:17.155 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:13:17.155 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.155 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.155 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.24580.0:59182 2.7 e124/119 2:e67e2a31:::benchmark_data_smithi045_46415_object59181:head v 124'3586, mlcod=107'428)", 2024-08-19T14:13:17.155 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:13:16.110955+0000", 2024-08-19T14:13:17.155 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.98737756499999996, 2024-08-19T14:13:17.155 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.025234295, 2024-08-19T14:13:17.156 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:13:17.156 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:13:17.156 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:13:17.156 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:13:17.156 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.156 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:13:17.156 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.110955+0000", 2024-08-19T14:13:17.156 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:13:17.156 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.156 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.157 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:13:17.157 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.110955+0000", 2024-08-19T14:13:17.158 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:13:17.158 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.158 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.158 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:13:17.158 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.110956+0000", 2024-08-19T14:13:17.158 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.33e-07 2024-08-19T14:13:17.158 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.158 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.158 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:13:17.158 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.111077+0000", 2024-08-19T14:13:17.158 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012131800000000001 2024-08-19T14:13:17.158 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.159 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.159 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:13:17.159 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.111092+0000", 2024-08-19T14:13:17.159 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5529999999999999e-05 2024-08-19T14:13:17.159 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.159 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.159 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:13:17.159 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.111172+0000", 2024-08-19T14:13:17.159 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.9635999999999994e-05 2024-08-19T14:13:17.159 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.159 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.159 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:13:17.159 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.123131+0000", 2024-08-19T14:13:17.159 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.011958662 2024-08-19T14:13:17.160 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.160 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.160 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:13:17.160 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.123155+0000", 2024-08-19T14:13:17.160 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.4769e-05 2024-08-19T14:13:17.160 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.160 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.160 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:13:17.160 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.136147+0000", 2024-08-19T14:13:17.160 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.012991137 2024-08-19T14:13:17.160 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.160 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.160 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:13:17.161 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.136189+0000", 2024-08-19T14:13:17.161 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.231e-05 2024-08-19T14:13:17.161 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:13:17.161 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:13:17.161 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:13:17.161 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.161 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.161 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.24580.0:59188 2.7 e124/119 2:e33325e6:::benchmark_data_smithi045_46415_object59187:head v 124'3587, mlcod=107'428)", 2024-08-19T14:13:17.161 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:13:16.115102+0000", 2024-08-19T14:13:17.161 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.983230088, 2024-08-19T14:13:17.161 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.021131457999999999, 2024-08-19T14:13:17.161 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:13:17.161 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:13:17.161 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:13:17.162 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:13:17.162 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.162 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:13:17.162 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.115102+0000", 2024-08-19T14:13:17.162 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:13:17.162 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.162 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.162 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:13:17.162 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.115102+0000", 2024-08-19T14:13:17.162 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:13:17.162 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.162 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.162 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:13:17.163 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.115103+0000", 2024-08-19T14:13:17.163 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.7499999999999999e-07 2024-08-19T14:13:17.163 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.163 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.163 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:13:17.163 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.115133+0000", 2024-08-19T14:13:17.163 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.9884000000000001e-05 2024-08-19T14:13:17.163 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.163 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.163 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:13:17.163 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.115149+0000", 2024-08-19T14:13:17.163 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6086000000000002e-05 2024-08-19T14:13:17.163 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.163 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.164 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:13:17.164 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.115157+0000", 2024-08-19T14:13:17.164 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.8099999999999998e-06 2024-08-19T14:13:17.164 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.164 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.164 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:13:17.164 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.126484+0000", 2024-08-19T14:13:17.164 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.011327152 2024-08-19T14:13:17.164 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.164 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.164 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:13:17.164 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.126512+0000", 2024-08-19T14:13:17.164 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.7849000000000001e-05 2024-08-19T14:13:17.165 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.165 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.165 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:13:17.165 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.136191+0000", 2024-08-19T14:13:17.165 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0096797710000000002 2024-08-19T14:13:17.165 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.165 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.165 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:13:17.165 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.136234+0000", 2024-08-19T14:13:17.165 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2131e-05 2024-08-19T14:13:17.165 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:13:17.165 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:13:17.165 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:13:17.165 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.166 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.166 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.24580.0:59622 2.7 e124/119 2:e4cb5c43:::benchmark_data_smithi045_46415_object59621:head v 124'3612, mlcod=107'447)", 2024-08-19T14:13:17.166 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:13:16.387125+0000", 2024-08-19T14:13:17.166 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.71120737099999998, 2024-08-19T14:13:17.166 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.025767545999999999, 2024-08-19T14:13:17.166 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:13:17.166 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:13:17.166 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:13:17.166 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:13:17.166 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.166 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:13:17.166 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.387125+0000", 2024-08-19T14:13:17.166 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:13:17.167 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.167 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.167 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:13:17.167 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.387125+0000", 2024-08-19T14:13:17.167 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:13:17.167 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.167 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.167 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:13:17.167 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.387126+0000", 2024-08-19T14:13:17.167 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.4900000000000005e-07 2024-08-19T14:13:17.167 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.167 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.167 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:13:17.167 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.387276+0000", 2024-08-19T14:13:17.168 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00015052500000000001 2024-08-19T14:13:17.168 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.168 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.168 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:13:17.168 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.387289+0000", 2024-08-19T14:13:17.168 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3291e-05 2024-08-19T14:13:17.168 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.168 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.168 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:13:17.168 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.387296+0000", 2024-08-19T14:13:17.168 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.6320000000000002e-06 2024-08-19T14:13:17.168 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.168 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.169 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:13:17.169 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.388314+0000", 2024-08-19T14:13:17.169 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.001017932 2024-08-19T14:13:17.169 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.169 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.169 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:13:17.169 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.388337+0000", 2024-08-19T14:13:17.169 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.3546000000000001e-05 2024-08-19T14:13:17.169 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.169 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.169 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:13:17.169 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.412843+0000", 2024-08-19T14:13:17.169 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.024505193000000002 2024-08-19T14:13:17.169 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.170 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.170 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:13:17.170 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.412892+0000", 2024-08-19T14:13:17.170 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.9678000000000001e-05 2024-08-19T14:13:17.170 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:13:17.170 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:13:17.170 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:13:17.170 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.170 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.170 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.24580.0:59628 2.7 e124/119 2:e2b2eb9b:::benchmark_data_smithi045_46415_object59627:head v 124'3613, mlcod=107'447)", 2024-08-19T14:13:17.170 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:13:16.392844+0000", 2024-08-19T14:13:17.170 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.70548854699999997, 2024-08-19T14:13:17.170 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.020079295, 2024-08-19T14:13:17.171 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:13:17.171 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:13:17.171 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:13:17.171 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:13:17.171 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.171 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:13:17.171 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.392844+0000", 2024-08-19T14:13:17.171 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:13:17.171 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.171 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.171 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:13:17.171 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.392844+0000", 2024-08-19T14:13:17.171 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:13:17.171 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.171 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.172 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:13:17.172 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.392844+0000", 2024-08-19T14:13:17.172 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.3200000000000004e-07 2024-08-19T14:13:17.172 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.172 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.172 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:13:17.172 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.392889+0000", 2024-08-19T14:13:17.172 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4579000000000001e-05 2024-08-19T14:13:17.172 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.172 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.172 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:13:17.172 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.392903+0000", 2024-08-19T14:13:17.172 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3647e-05 2024-08-19T14:13:17.173 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.173 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.173 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:13:17.173 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.392910+0000", 2024-08-19T14:13:17.173 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.0509999999999999e-06 2024-08-19T14:13:17.173 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.173 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.173 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:13:17.173 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.392957+0000", 2024-08-19T14:13:17.173 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.6919000000000003e-05 2024-08-19T14:13:17.173 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.173 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.173 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:13:17.173 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.392978+0000", 2024-08-19T14:13:17.174 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.1770999999999999e-05 2024-08-19T14:13:17.174 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.174 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.174 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:13:17.174 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.412896+0000", 2024-08-19T14:13:17.174 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.019917516 2024-08-19T14:13:17.174 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.174 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.174 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:13:17.174 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.412923+0000", 2024-08-19T14:13:17.174 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.7080000000000002e-05 2024-08-19T14:13:17.174 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:13:17.174 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:13:17.174 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:13:17.174 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.175 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.175 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.24580.0:59632 2.7 e124/119 2:e07d8d5d:::benchmark_data_smithi045_46415_object59631:head v 124'3614, mlcod=107'447)", 2024-08-19T14:13:17.175 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:13:16.392925+0000", 2024-08-19T14:13:17.175 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.70540672500000001, 2024-08-19T14:13:17.175 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.020025258000000001, 2024-08-19T14:13:17.175 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:13:17.175 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:13:17.175 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:13:17.175 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:13:17.175 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.175 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:13:17.175 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.392925+0000", 2024-08-19T14:13:17.175 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:13:17.175 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.175 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.176 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:13:17.176 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.392925+0000", 2024-08-19T14:13:17.176 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:13:17.176 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.176 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.176 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:13:17.176 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.392926+0000", 2024-08-19T14:13:17.176 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.0800000000000001e-07 2024-08-19T14:13:17.176 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.176 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.176 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:13:17.176 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.392959+0000", 2024-08-19T14:13:17.176 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.3225999999999997e-05 2024-08-19T14:13:17.176 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.177 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.177 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:13:17.177 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.392966+0000", 2024-08-19T14:13:17.177 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.9829999999999999e-06 2024-08-19T14:13:17.177 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.177 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.177 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:13:17.177 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.392971+0000", 2024-08-19T14:13:17.177 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.8829999999999997e-06 2024-08-19T14:13:17.177 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.177 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.177 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:13:17.177 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.396282+0000", 2024-08-19T14:13:17.177 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0033114110000000002 2024-08-19T14:13:17.177 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.178 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.178 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:13:17.178 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.396306+0000", 2024-08-19T14:13:17.178 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.4168e-05 2024-08-19T14:13:17.178 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.178 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.178 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:13:17.178 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.412925+0000", 2024-08-19T14:13:17.178 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.016618404 2024-08-19T14:13:17.178 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.178 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.178 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:13:17.178 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.412951+0000", 2024-08-19T14:13:17.178 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.5874999999999999e-05 2024-08-19T14:13:17.179 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:13:17.179 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:13:17.179 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:13:17.179 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.179 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.179 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.24580.0:60293 2.1 e125/102 2:83358ec1:::benchmark_data_smithi045_46415_object60292:head v 125'3794, mlcod=125'3791)", 2024-08-19T14:13:17.179 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:13:16.816353+0000", 2024-08-19T14:13:17.179 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.28197880800000003, 2024-08-19T14:13:17.179 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.020853468999999999, 2024-08-19T14:13:17.179 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:13:17.179 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:13:17.179 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:13:17.179 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:13:17.180 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.180 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:13:17.180 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.816353+0000", 2024-08-19T14:13:17.180 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:13:17.180 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.180 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.180 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:13:17.180 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.816353+0000", 2024-08-19T14:13:17.180 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:13:17.180 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.180 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.180 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:13:17.180 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.816354+0000", 2024-08-19T14:13:17.181 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.9500000000000002e-07 2024-08-19T14:13:17.181 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.181 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.181 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:13:17.181 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.816376+0000", 2024-08-19T14:13:17.181 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.1665999999999999e-05 2024-08-19T14:13:17.181 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.181 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.181 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:13:17.181 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.816386+0000", 2024-08-19T14:13:17.181 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0132000000000001e-05 2024-08-19T14:13:17.181 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.181 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.181 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:13:17.182 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.816392+0000", 2024-08-19T14:13:17.182 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.1020000000000002e-06 2024-08-19T14:13:17.182 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.182 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.182 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:13:17.182 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.821033+0000", 2024-08-19T14:13:17.182 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0046408370000000001 2024-08-19T14:13:17.182 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.182 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.182 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:13:17.182 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.821053+0000", 2024-08-19T14:13:17.182 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.9890000000000001e-05 2024-08-19T14:13:17.182 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.182 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.183 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:13:17.183 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.837020+0000", 2024-08-19T14:13:17.183 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.015967039999999998 2024-08-19T14:13:17.183 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.183 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.183 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:13:17.183 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.837207+0000", 2024-08-19T14:13:17.183 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000187107 2024-08-19T14:13:17.183 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:13:17.183 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:13:17.183 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:13:17.183 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.183 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.183 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.24580.0:60289 2.7 e125/119 2:e854ff3b:::benchmark_data_smithi045_46415_object60288:head v 125'3663, mlcod=107'476)", 2024-08-19T14:13:17.183 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:13:16.820717+0000", 2024-08-19T14:13:17.184 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.277615367, 2024-08-19T14:13:17.184 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.018698706999999998, 2024-08-19T14:13:17.184 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:13:17.184 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:13:17.184 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:13:17.184 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:13:17.184 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.184 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:13:17.184 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.820717+0000", 2024-08-19T14:13:17.184 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:13:17.184 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.184 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.184 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:13:17.184 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.820717+0000", 2024-08-19T14:13:17.185 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:13:17.185 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.185 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.185 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:13:17.185 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.820717+0000", 2024-08-19T14:13:17.185 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7e-07 2024-08-19T14:13:17.185 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.185 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.185 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:13:17.185 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.820750+0000", 2024-08-19T14:13:17.185 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.2534999999999999e-05 2024-08-19T14:13:17.185 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.185 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.185 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:13:17.185 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.820757+0000", 2024-08-19T14:13:17.185 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.7589999999999997e-06 2024-08-19T14:13:17.186 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.186 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.186 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:13:17.186 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.820762+0000", 2024-08-19T14:13:17.186 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.9280000000000001e-06 2024-08-19T14:13:17.186 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.186 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.186 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:13:17.186 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.821144+0000", 2024-08-19T14:13:17.186 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00038194800000000001 2024-08-19T14:13:17.186 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.186 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.187 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:13:17.187 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.821169+0000", 2024-08-19T14:13:17.187 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.4340000000000001e-05 2024-08-19T14:13:17.187 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.187 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.187 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:13:17.187 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.839261+0000", 2024-08-19T14:13:17.187 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.018092427000000001 2024-08-19T14:13:17.187 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.187 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.187 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:13:17.187 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.839416+0000", 2024-08-19T14:13:17.187 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00015440000000000001 2024-08-19T14:13:17.187 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:13:17.188 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:13:17.188 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:13:17.188 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.188 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.188 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.24580.0:60474 2.7 e125/119 2:e79cfb8f:::benchmark_data_smithi045_46415_object60473:head v 125'3673, mlcod=107'486)", 2024-08-19T14:13:17.188 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:13:16.936048+0000", 2024-08-19T14:13:17.188 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.16228378800000001, 2024-08-19T14:13:17.188 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.026006333, 2024-08-19T14:13:17.188 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:13:17.188 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:13:17.188 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:13:17.188 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:13:17.188 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.188 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:13:17.189 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.936048+0000", 2024-08-19T14:13:17.189 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:13:17.189 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.189 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.189 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:13:17.189 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.936048+0000", 2024-08-19T14:13:17.189 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:13:17.189 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.189 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.189 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:13:17.189 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.936049+0000", 2024-08-19T14:13:17.189 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.5199999999999996e-07 2024-08-19T14:13:17.189 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.189 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.190 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:13:17.190 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.936097+0000", 2024-08-19T14:13:17.190 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7902999999999999e-05 2024-08-19T14:13:17.190 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.190 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.190 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:13:17.190 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.936109+0000", 2024-08-19T14:13:17.190 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2126e-05 2024-08-19T14:13:17.190 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.190 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.190 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:13:17.190 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.936116+0000", 2024-08-19T14:13:17.190 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.9299999999999997e-06 2024-08-19T14:13:17.191 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.191 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.191 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:13:17.191 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.936200+0000", 2024-08-19T14:13:17.191 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.3801999999999995e-05 2024-08-19T14:13:17.191 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.191 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.191 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:13:17.191 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.936225+0000", 2024-08-19T14:13:17.191 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.4893999999999999e-05 2024-08-19T14:13:17.191 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.191 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.191 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:13:17.191 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.962013+0000", 2024-08-19T14:13:17.192 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.025787872 2024-08-19T14:13:17.192 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:13:17.192 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:13:17.192 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:13:17.192 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:16.962055+0000", 2024-08-19T14:13:17.192 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2154000000000003e-05 2024-08-19T14:13:17.192 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:13:17.192 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:13:17.192 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:13:17.192 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:13:17.192 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:13:17.192 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:13:17.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:17.194 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:17.194 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:13:17.296 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:13:17.334 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:17.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:17.335 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:13:17.436 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:13:17.463 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:17.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:17.464 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:13:17.565 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:13:17.590 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:17.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:17.591 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:13:17.692 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:13:17.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:17.723 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:17.723 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:13:17.824 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:13:17.832 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 28 16 61625 61609 137.508 81.6875 0.00485335 0.00726727 2024-08-19T14:13:17.853 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:17.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:17.854 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:13:17.955 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:13:17.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:17.985 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:17.985 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:13:18.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:17 smithi032 ceph-mon[20004]: pgmap v325: 27 pgs: 3 active+recovering+undersized+remapped, 24 active+clean; 3.4 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 83 MiB/s wr, 1.32k op/s; 8494/166959 objects misplaced (5.087%); 8.3 MiB/s, 132 objects/s recovering 2024-08-19T14:13:18.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:17 smithi032 ceph-mon[20004]: osdmap e125: 12 total, 12 up, 11 in 2024-08-19T14:13:18.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:17 smithi045 ceph-mon[20143]: pgmap v325: 27 pgs: 3 active+recovering+undersized+remapped, 24 active+clean; 3.4 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 83 MiB/s wr, 1.32k op/s; 8494/166959 objects misplaced (5.087%); 8.3 MiB/s, 132 objects/s recovering 2024-08-19T14:13:18.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:17 smithi045 ceph-mon[20143]: osdmap e125: 12 total, 12 up, 11 in 2024-08-19T14:13:18.086 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:13:18.115 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:18.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:17 smithi031 ceph-mon[16336]: pgmap v325: 27 pgs: 3 active+recovering+undersized+remapped, 24 active+clean; 3.4 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 83 MiB/s wr, 1.32k op/s; 8494/166959 objects misplaced (5.087%); 8.3 MiB/s, 132 objects/s recovering 2024-08-19T14:13:18.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:17 smithi031 ceph-mon[16336]: osdmap e125: 12 total, 12 up, 11 in 2024-08-19T14:13:18.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:17 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mon-a[16332]: 2024-08-19T14:13:17.807+0000 7f2ff503e640 -1 mon.a@0(leader).osd e125 definitely_dead 0 2024-08-19T14:13:18.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:18.117 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:13:18.218 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:13:18.250 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:18.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:18.251 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:13:18.352 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:13:18.379 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-08-19T14:13:18.399 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:18.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:18.400 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:13:18.501 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:13:18.527 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:18.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:18.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:13:18.629 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:13:18.658 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:18.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:18.659 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:13:18.715 INFO:teuthology.orchestra.run.smithi031.stderr:marked out osd.6. 2024-08-19T14:13:18.719 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:13:18.760 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:13:18.785 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:18.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:18.786 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:13:18.832 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 29 16 63032 63016 135.798 87.9375 0.00533931 0.00734132 2024-08-19T14:13:18.887 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:13:18.908 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:18.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:18.909 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:13:19.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:18 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1376931224' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-08-19T14:13:19.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:18 smithi032 ceph-mon[20004]: Client client.admin marked osd.6 out, while it was still marked up 2024-08-19T14:13:19.010 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:13:19.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:19.034 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:19.034 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:13:19.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:18 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1376931224' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-08-19T14:13:19.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:18 smithi045 ceph-mon[20143]: Client client.admin marked osd.6 out, while it was still marked up 2024-08-19T14:13:19.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:18 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1376931224' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-08-19T14:13:19.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:18 smithi031 ceph-mon[16336]: Client client.admin marked osd.6 out, while it was still marked up 2024-08-19T14:13:19.135 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:13:19.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:19.158 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:19.159 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:13:19.260 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:13:19.283 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:19.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:19.284 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:13:19.385 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:13:19.406 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:19.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:19.407 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:13:19.508 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:13:19.529 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:19.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:19.530 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:13:19.631 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:13:19.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:19.655 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:19.656 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:13:19.757 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:13:19.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:19 smithi032 ceph-mon[20004]: pgmap v327: 27 pgs: 3 active+recovering+undersized+remapped, 24 active+clean; 3.5 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 81 MiB/s wr, 1.30k op/s; 8314/173937 objects misplaced (4.780%); 8.9 MiB/s, 142 objects/s recovering 2024-08-19T14:13:19.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:19 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1376931224' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2024-08-19T14:13:19.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:19 smithi032 ceph-mon[20004]: osdmap e126: 12 total, 12 up, 10 in 2024-08-19T14:13:19.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:19.780 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:19.780 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:13:19.832 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 30 16 63032 63016 131.272 0 - 0.00734132 2024-08-19T14:13:19.881 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:13:19.899 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:13:19.900 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-08-19T14:13:19.900 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-08-19T14:13:19.900 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:13:19.905 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:19.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:19.906 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:13:20.007 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:13:20.031 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:20.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:20.032 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:13:20.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:19 smithi045 ceph-mon[20143]: pgmap v327: 27 pgs: 3 active+recovering+undersized+remapped, 24 active+clean; 3.5 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 81 MiB/s wr, 1.30k op/s; 8314/173937 objects misplaced (4.780%); 8.9 MiB/s, 142 objects/s recovering 2024-08-19T14:13:20.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:19 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1376931224' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2024-08-19T14:13:20.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:19 smithi045 ceph-mon[20143]: osdmap e126: 12 total, 12 up, 10 in 2024-08-19T14:13:20.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:19 smithi031 ceph-mon[16336]: pgmap v327: 27 pgs: 3 active+recovering+undersized+remapped, 24 active+clean; 3.5 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 81 MiB/s wr, 1.30k op/s; 8314/173937 objects misplaced (4.780%); 8.9 MiB/s, 142 objects/s recovering 2024-08-19T14:13:20.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:19 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1376931224' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2024-08-19T14:13:20.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:19 smithi031 ceph-mon[16336]: osdmap e126: 12 total, 12 up, 10 in 2024-08-19T14:13:20.132 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:13:20.153 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:20.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:20.154 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:13:20.255 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:13:20.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:20.278 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:20.278 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:13:20.379 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:13:20.400 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:20.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:20.401 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:13:20.502 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:13:20.523 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:20.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:20.525 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:13:20.609 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-08-19T14:13:20.625 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:13:20.646 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:20.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:20.647 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:13:20.748 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:13:20.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:20 smithi032 ceph-mon[20004]: osdmap e127: 12 total, 12 up, 10 in 2024-08-19T14:13:20.770 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:20.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:20.771 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:13:20.834 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 31 16 63037 63021 127.047 0.15625 0.00315598 0.00740735 2024-08-19T14:13:20.872 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:13:20.898 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:20.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:20.899 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:13:20.907 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:13:21.000 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:13:21.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:21.036 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:21.036 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:13:21.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:20 smithi045 ceph-mon[20143]: osdmap e127: 12 total, 12 up, 10 in 2024-08-19T14:13:21.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:20 smithi031 ceph-mon[16336]: osdmap e127: 12 total, 12 up, 10 in 2024-08-19T14:13:21.137 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:13:21.158 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:21.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:21.159 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:13:21.260 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:13:21.287 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:21.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:21.287 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:13:21.388 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:13:21.421 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:21.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:21.422 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:13:21.490 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:13:21.490 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-08-19T14:13:21.490 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-08-19T14:13:21.491 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-08-19T14:13:21.491 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:13:21.523 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:13:21.553 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:21.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:21.554 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:13:21.655 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:13:21.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:21.689 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:21.689 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:13:21.790 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:13:21.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:21.812 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:21.812 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:13:21.832 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 32 16 64193 64177 125.335 72.25 0.00459734 0.0079724 2024-08-19T14:13:21.913 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:13:21.948 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:21.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:21.950 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:13:22.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:21 smithi032 ceph-mon[20004]: pgmap v330: 27 pgs: 1 remapped+peering, 1 unknown, 2 active+recovering+undersized+remapped, 23 active+clean; 3.7 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 2.5 KiB/s rd, 98 MiB/s wr, 1.57k op/s; 5169/183117 objects misplaced (2.823%); 14 MiB/s, 228 objects/s recovering 2024-08-19T14:13:22.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:21 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:13:22.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:21 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:13:22.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:21 smithi032 ceph-mon[20004]: osdmap e128: 12 total, 12 up, 10 in 2024-08-19T14:13:22.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:21 smithi032 ceph-mon[20004]: osdmap e129: 12 total, 12 up, 10 in 2024-08-19T14:13:22.013 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:21 smithi031 ceph-mon[16336]: pgmap v330: 27 pgs: 1 remapped+peering, 1 unknown, 2 active+recovering+undersized+remapped, 23 active+clean; 3.7 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 2.5 KiB/s rd, 98 MiB/s wr, 1.57k op/s; 5169/183117 objects misplaced (2.823%); 14 MiB/s, 228 objects/s recovering 2024-08-19T14:13:22.013 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:21 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:13:22.013 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:21 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:13:22.013 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:21 smithi031 ceph-mon[16336]: osdmap e128: 12 total, 12 up, 10 in 2024-08-19T14:13:22.013 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:21 smithi031 ceph-mon[16336]: osdmap e129: 12 total, 12 up, 10 in 2024-08-19T14:13:22.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:21 smithi045 ceph-mon[20143]: pgmap v330: 27 pgs: 1 remapped+peering, 1 unknown, 2 active+recovering+undersized+remapped, 23 active+clean; 3.7 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 2.5 KiB/s rd, 98 MiB/s wr, 1.57k op/s; 5169/183117 objects misplaced (2.823%); 14 MiB/s, 228 objects/s recovering 2024-08-19T14:13:22.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:21 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:13:22.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:21 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:13:22.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:21 smithi045 ceph-mon[20143]: osdmap e128: 12 total, 12 up, 10 in 2024-08-19T14:13:22.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:21 smithi045 ceph-mon[20143]: osdmap e129: 12 total, 12 up, 10 in 2024-08-19T14:13:22.054 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:13:22.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:22.115 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:22.115 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:13:22.203 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-08-19T14:13:22.216 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:13:22.244 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:22.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:22.245 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:13:22.346 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:13:22.370 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:22.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:22.371 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:13:22.472 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:13:22.506 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:22.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:22.507 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:13:22.548 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:13:22.608 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:13:22.638 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:22.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:22.639 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:13:22.740 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:13:22.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:22.769 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:22.769 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:13:22.832 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 33 16 65471 65455 123.957 79.875 0.00514809 0.00806119 2024-08-19T14:13:22.870 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:13:22.900 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:22.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:22.901 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:13:23.002 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:13:23.030 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:23.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:23.031 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:13:23.080 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:13:23.080 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-08-19T14:13:23.080 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-08-19T14:13:23.080 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-08-19T14:13:23.080 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.080 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24580.0:62590 2.d e125/102 2:b1cd6c4c:::benchmark_data_smithi045_46415_object62589:head v 125'3988, mlcod=125'3985)", 2024-08-19T14:13:23.081 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:18.389311+0000", 2024-08-19T14:13:23.081 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 4.6879890450000001, 2024-08-19T14:13:23.081 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.025501861000000001, 2024-08-19T14:13:23.081 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:13:23.081 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:13:23.081 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:13:23.081 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:13:23.081 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.081 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:13:23.081 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.389311+0000", 2024-08-19T14:13:23.081 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:23.081 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.081 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.082 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:13:23.082 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.389311+0000", 2024-08-19T14:13:23.082 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:23.082 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.082 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.082 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:13:23.082 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.389311+0000", 2024-08-19T14:13:23.082 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8899999999999999e-07 2024-08-19T14:13:23.082 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.082 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.082 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:13:23.082 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.389426+0000", 2024-08-19T14:13:23.082 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000115003 2024-08-19T14:13:23.082 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.083 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.083 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:13:23.083 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.389437+0000", 2024-08-19T14:13:23.083 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0865e-05 2024-08-19T14:13:23.083 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.083 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.083 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:13:23.083 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.389444+0000", 2024-08-19T14:13:23.083 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2189999999999998e-06 2024-08-19T14:13:23.083 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.083 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.083 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:13:23.083 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.389490+0000", 2024-08-19T14:13:23.084 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5357e-05 2024-08-19T14:13:23.084 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.084 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.084 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:13:23.084 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.389504+0000", 2024-08-19T14:13:23.084 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4119e-05 2024-08-19T14:13:23.084 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.084 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.084 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:13:23.084 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.391313+0000", 2024-08-19T14:13:23.084 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0018095380000000001 2024-08-19T14:13:23.084 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.084 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.085 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:13:23.085 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.414812+0000", 2024-08-19T14:13:23.085 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.023499170999999999 2024-08-19T14:13:23.085 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:23.085 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:13:23.085 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:23.085 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.085 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.085 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24580.0:62594 2.5 e125/102 2:a15e85f0:::benchmark_data_smithi045_46415_object62593:head v 125'3968, mlcod=125'3966)", 2024-08-19T14:13:23.085 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:18.389960+0000", 2024-08-19T14:13:23.085 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 4.6873398740000001, 2024-08-19T14:13:23.085 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.027685131000000002, 2024-08-19T14:13:23.085 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:13:23.085 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:13:23.086 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:13:23.086 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:13:23.086 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.086 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:13:23.086 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.389960+0000", 2024-08-19T14:13:23.086 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:23.086 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.086 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.086 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:13:23.086 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.389960+0000", 2024-08-19T14:13:23.086 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:23.086 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.086 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.087 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:13:23.087 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.389960+0000", 2024-08-19T14:13:23.087 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8100000000000002e-07 2024-08-19T14:13:23.087 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.087 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.087 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:13:23.087 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.390013+0000", 2024-08-19T14:13:23.087 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2772000000000002e-05 2024-08-19T14:13:23.087 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.087 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.087 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:13:23.087 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.390027+0000", 2024-08-19T14:13:23.087 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3625000000000001e-05 2024-08-19T14:13:23.088 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.088 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.088 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:13:23.088 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.390033+0000", 2024-08-19T14:13:23.088 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3849999999999999e-06 2024-08-19T14:13:23.088 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.088 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.088 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:13:23.088 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.390089+0000", 2024-08-19T14:13:23.088 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5998000000000002e-05 2024-08-19T14:13:23.088 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.088 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.088 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:13:23.089 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.390109+0000", 2024-08-19T14:13:23.089 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9539e-05 2024-08-19T14:13:23.089 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.089 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.089 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:13:23.089 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.417596+0000", 2024-08-19T14:13:23.089 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.027487396000000001 2024-08-19T14:13:23.089 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.089 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.089 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:13:23.089 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.417645+0000", 2024-08-19T14:13:23.089 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8735e-05 2024-08-19T14:13:23.089 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:23.089 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:13:23.090 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:23.090 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.090 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.090 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24580.0:62609 2.f e125/102 2:fbece67b:::benchmark_data_smithi045_46415_object62608:head v 125'3942, mlcod=125'3940)", 2024-08-19T14:13:23.090 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:18.400668+0000", 2024-08-19T14:13:23.090 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 4.6766318450000002, 2024-08-19T14:13:23.090 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.017869350999999999, 2024-08-19T14:13:23.090 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:13:23.090 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:13:23.090 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:13:23.090 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:13:23.090 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.090 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:13:23.091 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.400668+0000", 2024-08-19T14:13:23.091 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:23.091 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.091 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.091 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:13:23.091 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.400668+0000", 2024-08-19T14:13:23.091 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:23.091 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.091 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.091 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:13:23.091 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.400669+0000", 2024-08-19T14:13:23.091 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.9699999999999994e-07 2024-08-19T14:13:23.091 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.091 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.091 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:13:23.092 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.400703+0000", 2024-08-19T14:13:23.092 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.4584e-05 2024-08-19T14:13:23.092 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.092 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.092 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:13:23.092 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.414645+0000", 2024-08-19T14:13:23.092 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.013941998000000001 2024-08-19T14:13:23.092 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.092 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.092 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:13:23.092 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.414656+0000", 2024-08-19T14:13:23.092 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0718e-05 2024-08-19T14:13:23.092 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.092 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.093 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:13:23.093 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.414725+0000", 2024-08-19T14:13:23.093 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8656000000000001e-05 2024-08-19T14:13:23.093 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.093 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.093 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:13:23.093 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.414748+0000", 2024-08-19T14:13:23.093 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3638e-05 2024-08-19T14:13:23.093 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.093 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.093 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:13:23.093 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.418369+0000", 2024-08-19T14:13:23.093 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0036201380000000002 2024-08-19T14:13:23.093 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.093 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.094 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:13:23.094 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.418537+0000", 2024-08-19T14:13:23.094 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000168622 2024-08-19T14:13:23.094 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:23.094 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:13:23.094 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:23.094 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.094 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.094 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24580.0:62615 2.d e125/102 2:b7204c12:::benchmark_data_smithi045_46415_object62614:head v 125'3989, mlcod=125'3986)", 2024-08-19T14:13:23.094 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:18.403214+0000", 2024-08-19T14:13:23.094 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 4.6740856520000005, 2024-08-19T14:13:23.094 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.016228660999999998, 2024-08-19T14:13:23.094 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:13:23.094 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:13:23.095 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:13:23.095 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:13:23.095 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.095 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:13:23.095 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.403214+0000", 2024-08-19T14:13:23.095 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:23.095 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.095 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.095 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:13:23.095 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.403214+0000", 2024-08-19T14:13:23.095 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:23.095 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.095 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.095 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:13:23.096 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.403215+0000", 2024-08-19T14:13:23.096 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0389999999999999e-06 2024-08-19T14:13:23.096 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.096 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.096 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:13:23.096 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.403280+0000", 2024-08-19T14:13:23.096 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4924000000000004e-05 2024-08-19T14:13:23.096 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.096 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.096 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:13:23.096 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.414224+0000", 2024-08-19T14:13:23.096 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010944469 2024-08-19T14:13:23.096 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.096 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.097 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:13:23.097 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.414242+0000", 2024-08-19T14:13:23.097 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7702000000000001e-05 2024-08-19T14:13:23.097 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.097 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.097 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:13:23.097 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.414840+0000", 2024-08-19T14:13:23.097 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00059743199999999995 2024-08-19T14:13:23.097 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.097 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.097 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:13:23.097 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.414859+0000", 2024-08-19T14:13:23.097 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9335e-05 2024-08-19T14:13:23.097 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.097 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.098 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:13:23.098 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.419398+0000", 2024-08-19T14:13:23.098 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0045388579999999998 2024-08-19T14:13:23.098 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.098 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.098 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:13:23.098 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.419443+0000", 2024-08-19T14:13:23.098 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4901999999999998e-05 2024-08-19T14:13:23.098 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:23.098 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:13:23.098 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:23.098 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.098 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.098 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24580.0:63067 2.7 e128/127 2:e2705906:::benchmark_data_smithi045_46415_object63066:head v 128'3850, mlcod=0'0)", 2024-08-19T14:13:23.099 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:20.938200+0000", 2024-08-19T14:13:23.099 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 2.1390991929999998, 2024-08-19T14:13:23.099 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.021572997999999999, 2024-08-19T14:13:23.099 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:13:23.099 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:13:23.099 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:13:23.099 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:13:23.099 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.099 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:13:23.099 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.938200+0000", 2024-08-19T14:13:23.099 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:23.099 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.099 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.099 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:13:23.100 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.938200+0000", 2024-08-19T14:13:23.100 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:23.100 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.100 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.100 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:13:23.100 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.938203+0000", 2024-08-19T14:13:23.100 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.638e-06 2024-08-19T14:13:23.100 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.100 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.100 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:13:23.100 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.938268+0000", 2024-08-19T14:13:23.100 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4980000000000005e-05 2024-08-19T14:13:23.100 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.100 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.101 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:13:23.101 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.938279+0000", 2024-08-19T14:13:23.101 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0658e-05 2024-08-19T14:13:23.101 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.101 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.101 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:13:23.101 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.938285+0000", 2024-08-19T14:13:23.101 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9410000000000004e-06 2024-08-19T14:13:23.101 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.101 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.101 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:13:23.101 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.940615+0000", 2024-08-19T14:13:23.101 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0023300819999999998 2024-08-19T14:13:23.101 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.102 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.102 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:13:23.102 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.940638+0000", 2024-08-19T14:13:23.102 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3691999999999999e-05 2024-08-19T14:13:23.102 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.102 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.102 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:13:23.102 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.959684+0000", 2024-08-19T14:13:23.102 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.019045973000000001 2024-08-19T14:13:23.102 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.102 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.102 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:13:23.102 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.959773+0000", 2024-08-19T14:13:23.102 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.9034000000000002e-05 2024-08-19T14:13:23.103 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:23.103 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:13:23.103 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:23.103 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.103 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.103 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24580.0:63086 2.7 e128/127 2:e280ae82:::benchmark_data_smithi045_46415_object63085:head v 128'3851, mlcod=0'0)", 2024-08-19T14:13:23.103 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:20.940402+0000", 2024-08-19T14:13:23.103 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 2.1368977760000001, 2024-08-19T14:13:23.103 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.022477548999999999, 2024-08-19T14:13:23.103 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:13:23.103 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:13:23.103 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:13:23.103 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:13:23.103 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.104 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:13:23.104 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.940402+0000", 2024-08-19T14:13:23.104 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:23.104 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.104 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.104 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:13:23.104 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.940402+0000", 2024-08-19T14:13:23.104 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:23.104 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.104 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.104 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:13:23.104 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.940402+0000", 2024-08-19T14:13:23.104 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0200000000000002e-07 2024-08-19T14:13:23.104 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.104 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.105 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:13:23.105 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.940470+0000", 2024-08-19T14:13:23.105 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7830999999999995e-05 2024-08-19T14:13:23.105 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.105 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.105 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:13:23.105 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.940479+0000", 2024-08-19T14:13:23.105 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.4100000000000008e-06 2024-08-19T14:13:23.105 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.105 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.105 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:13:23.105 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.940484+0000", 2024-08-19T14:13:23.105 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8679999999999998e-06 2024-08-19T14:13:23.105 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.106 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.106 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:13:23.106 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.944331+0000", 2024-08-19T14:13:23.106 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0038478200000000001 2024-08-19T14:13:23.106 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.106 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.106 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:13:23.106 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.944357+0000", 2024-08-19T14:13:23.106 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5836000000000001e-05 2024-08-19T14:13:23.106 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.106 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.106 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:13:23.106 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.962818+0000", 2024-08-19T14:13:23.106 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.018460977 2024-08-19T14:13:23.107 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.107 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.107 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:13:23.107 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.962879+0000", 2024-08-19T14:13:23.107 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1304999999999997e-05 2024-08-19T14:13:23.107 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:23.107 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:13:23.107 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:23.107 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.107 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.107 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24580.0:63088 2.7 e128/127 2:eb67bda3:::benchmark_data_smithi045_46415_object63087:head v 128'3852, mlcod=0'0)", 2024-08-19T14:13:23.107 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:20.944746+0000", 2024-08-19T14:13:23.107 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 2.1325538330000002, 2024-08-19T14:13:23.107 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.018166799000000001, 2024-08-19T14:13:23.108 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:13:23.108 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:13:23.108 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:13:23.108 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:13:23.108 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.108 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:13:23.108 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.944746+0000", 2024-08-19T14:13:23.108 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:23.108 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.109 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.109 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:13:23.109 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.944746+0000", 2024-08-19T14:13:23.109 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:23.109 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.109 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.109 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:13:23.109 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.944747+0000", 2024-08-19T14:13:23.109 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.218e-06 2024-08-19T14:13:23.110 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.110 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.110 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:13:23.110 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.944829+0000", 2024-08-19T14:13:23.110 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.2140999999999998e-05 2024-08-19T14:13:23.110 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.110 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.110 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:13:23.110 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.944842+0000", 2024-08-19T14:13:23.110 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.269e-05 2024-08-19T14:13:23.110 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.110 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.110 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:13:23.110 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.944849+0000", 2024-08-19T14:13:23.111 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.029e-06 2024-08-19T14:13:23.111 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.111 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.111 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:13:23.111 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.948292+0000", 2024-08-19T14:13:23.111 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0034428430000000001 2024-08-19T14:13:23.111 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.111 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.111 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:13:23.111 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.948319+0000", 2024-08-19T14:13:23.111 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7483000000000001e-05 2024-08-19T14:13:23.111 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.111 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.111 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:13:23.112 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.962883+0000", 2024-08-19T14:13:23.112 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.014563906999999999 2024-08-19T14:13:23.112 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.112 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.112 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:13:23.112 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.962913+0000", 2024-08-19T14:13:23.112 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9488e-05 2024-08-19T14:13:23.112 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:23.112 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:13:23.112 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:23.112 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.112 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.112 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24580.0:63097 2.7 e128/127 2:e33e4164:::benchmark_data_smithi045_46415_object63096:head v 128'3853, mlcod=0'0)", 2024-08-19T14:13:23.113 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:20.947520+0000", 2024-08-19T14:13:23.113 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 2.1297796010000001, 2024-08-19T14:13:23.113 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.015421218, 2024-08-19T14:13:23.113 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:13:23.113 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:13:23.113 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:13:23.113 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:13:23.113 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.113 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:13:23.113 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.947520+0000", 2024-08-19T14:13:23.113 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:23.113 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.113 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.113 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:13:23.114 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.947520+0000", 2024-08-19T14:13:23.114 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:23.114 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.114 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.114 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:13:23.114 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.947521+0000", 2024-08-19T14:13:23.114 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.0500000000000002e-07 2024-08-19T14:13:23.114 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.114 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.114 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:13:23.114 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.947559+0000", 2024-08-19T14:13:23.114 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8340999999999998e-05 2024-08-19T14:13:23.114 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.114 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.115 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:13:23.115 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.947573+0000", 2024-08-19T14:13:23.115 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3617e-05 2024-08-19T14:13:23.115 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.115 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.115 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:13:23.115 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.947580+0000", 2024-08-19T14:13:23.115 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2729999999999997e-06 2024-08-19T14:13:23.115 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.115 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.115 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:13:23.115 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.952281+0000", 2024-08-19T14:13:23.115 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0047006510000000001 2024-08-19T14:13:23.115 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.116 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.116 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:13:23.116 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.952309+0000", 2024-08-19T14:13:23.116 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8606e-05 2024-08-19T14:13:23.116 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.116 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.116 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:13:23.116 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.962914+0000", 2024-08-19T14:13:23.116 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010605033999999999 2024-08-19T14:13:23.116 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.116 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.116 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:13:23.116 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.962941+0000", 2024-08-19T14:13:23.116 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6791000000000001e-05 2024-08-19T14:13:23.117 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:23.117 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:13:23.117 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:23.117 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.117 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.117 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24580.0:63628 2.7 e128/127 2:e0b4181f:::benchmark_data_smithi045_46415_object63627:head v 128'3884, mlcod=0'0)", 2024-08-19T14:13:23.117 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:21.451919+0000", 2024-08-19T14:13:23.117 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 1.625380952, 2024-08-19T14:13:23.117 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.015290977000000001, 2024-08-19T14:13:23.117 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:13:23.117 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:13:23.117 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:13:23.117 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:13:23.117 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.118 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:13:23.118 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.451919+0000", 2024-08-19T14:13:23.118 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:23.118 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.118 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.118 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:13:23.118 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.451919+0000", 2024-08-19T14:13:23.118 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:23.118 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.118 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.118 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:13:23.118 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.451919+0000", 2024-08-19T14:13:23.118 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9100000000000003e-07 2024-08-19T14:13:23.118 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.119 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.119 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:13:23.119 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.451966+0000", 2024-08-19T14:13:23.119 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6996999999999997e-05 2024-08-19T14:13:23.119 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.119 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.119 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:13:23.119 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.451977+0000", 2024-08-19T14:13:23.119 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0426e-05 2024-08-19T14:13:23.119 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.119 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.119 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:13:23.119 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.451992+0000", 2024-08-19T14:13:23.119 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5437000000000001e-05 2024-08-19T14:13:23.119 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.120 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.120 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:13:23.120 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.452055+0000", 2024-08-19T14:13:23.120 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2959000000000005e-05 2024-08-19T14:13:23.120 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.120 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.120 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:13:23.120 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.452084+0000", 2024-08-19T14:13:23.120 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8411e-05 2024-08-19T14:13:23.120 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.120 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.120 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:13:23.120 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.467166+0000", 2024-08-19T14:13:23.120 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.015082342 2024-08-19T14:13:23.120 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.121 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.121 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:13:23.121 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.467210+0000", 2024-08-19T14:13:23.121 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3714000000000002e-05 2024-08-19T14:13:23.121 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:23.121 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:13:23.121 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:23.121 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.121 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.121 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24580.0:63687 2.7 e128/127 2:ea76d4f0:::benchmark_data_smithi045_46415_object63686:head v 128'3888, mlcod=0'0)", 2024-08-19T14:13:23.121 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:21.486322+0000", 2024-08-19T14:13:23.121 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 1.590977453, 2024-08-19T14:13:23.121 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.018889623000000001, 2024-08-19T14:13:23.121 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:13:23.121 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:13:23.122 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:13:23.122 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:13:23.122 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.122 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:13:23.122 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.486322+0000", 2024-08-19T14:13:23.122 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:23.122 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.122 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.122 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:13:23.122 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.486322+0000", 2024-08-19T14:13:23.122 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:23.122 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.122 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.122 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:13:23.123 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.486323+0000", 2024-08-19T14:13:23.123 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3799999999999996e-07 2024-08-19T14:13:23.123 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.123 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.123 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:13:23.123 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.486367+0000", 2024-08-19T14:13:23.123 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3430000000000003e-05 2024-08-19T14:13:23.123 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.123 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.123 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:13:23.123 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.486381+0000", 2024-08-19T14:13:23.123 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4579000000000001e-05 2024-08-19T14:13:23.123 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.123 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.124 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:13:23.124 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.486389+0000", 2024-08-19T14:13:23.124 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6899999999999992e-06 2024-08-19T14:13:23.124 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.124 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.124 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:13:23.124 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.486444+0000", 2024-08-19T14:13:23.124 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5359999999999999e-05 2024-08-19T14:13:23.124 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.124 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.124 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:13:23.124 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.486472+0000", 2024-08-19T14:13:23.124 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8019999999999999e-05 2024-08-19T14:13:23.124 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.125 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.125 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:13:23.125 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.505168+0000", 2024-08-19T14:13:23.125 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.01869618 2024-08-19T14:13:23.125 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.125 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.125 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:13:23.125 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.505212+0000", 2024-08-19T14:13:23.125 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3426000000000001e-05 2024-08-19T14:13:23.125 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:23.125 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:13:23.125 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:23.125 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.125 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.126 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24580.0:63689 2.7 e128/127 2:e3a5047d:::benchmark_data_smithi045_46415_object63688:head v 128'3889, mlcod=0'0)", 2024-08-19T14:13:23.126 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:21.488604+0000", 2024-08-19T14:13:23.126 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 1.5886952490000001, 2024-08-19T14:13:23.126 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.016639102999999999, 2024-08-19T14:13:23.126 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:13:23.126 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:13:23.126 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:13:23.126 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:13:23.126 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.126 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:13:23.126 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.488604+0000", 2024-08-19T14:13:23.126 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:23.126 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.126 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.127 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:13:23.127 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.488604+0000", 2024-08-19T14:13:23.127 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:23.127 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.127 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.127 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:13:23.127 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.488606+0000", 2024-08-19T14:13:23.127 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1620000000000001e-06 2024-08-19T14:13:23.127 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.127 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.127 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:13:23.127 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.488642+0000", 2024-08-19T14:13:23.127 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6835000000000002e-05 2024-08-19T14:13:23.127 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.128 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.128 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:13:23.128 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.488658+0000", 2024-08-19T14:13:23.128 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5160999999999999e-05 2024-08-19T14:13:23.128 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.128 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.128 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:13:23.128 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.488666+0000", 2024-08-19T14:13:23.128 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6000000000000007e-06 2024-08-19T14:13:23.128 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.128 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.128 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:13:23.128 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.490147+0000", 2024-08-19T14:13:23.128 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0014809860000000001 2024-08-19T14:13:23.129 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.129 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.129 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:13:23.129 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.490178+0000", 2024-08-19T14:13:23.129 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1041000000000003e-05 2024-08-19T14:13:23.129 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.129 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.129 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:13:23.129 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.505218+0000", 2024-08-19T14:13:23.129 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.015039947 2024-08-19T14:13:23.129 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.129 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.129 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:13:23.129 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.505244+0000", 2024-08-19T14:13:23.129 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5371000000000002e-05 2024-08-19T14:13:23.130 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:23.130 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:13:23.130 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:23.130 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.130 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.130 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24580.0:63715 2.7 e128/127 2:e8219184:::benchmark_data_smithi045_46415_object63714:head v 128'3893, mlcod=0'0)", 2024-08-19T14:13:23.130 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:21.497754+0000", 2024-08-19T14:13:23.130 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 1.579545671, 2024-08-19T14:13:23.130 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.015945902000000001, 2024-08-19T14:13:23.130 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:13:23.130 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:13:23.130 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:13:23.130 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:13:23.130 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.131 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:13:23.131 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.497754+0000", 2024-08-19T14:13:23.131 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:23.131 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.131 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.131 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:13:23.131 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.497754+0000", 2024-08-19T14:13:23.131 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:23.131 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.131 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.131 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:13:23.131 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.497755+0000", 2024-08-19T14:13:23.131 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.0400000000000005e-07 2024-08-19T14:13:23.131 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.132 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.132 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:13:23.132 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.497835+0000", 2024-08-19T14:13:23.132 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.9918000000000003e-05 2024-08-19T14:13:23.132 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.132 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.132 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:13:23.132 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.497853+0000", 2024-08-19T14:13:23.132 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7725000000000001e-05 2024-08-19T14:13:23.132 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.132 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.132 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:13:23.132 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.497861+0000", 2024-08-19T14:13:23.132 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.8400000000000001e-06 2024-08-19T14:13:23.132 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.133 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.133 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:13:23.133 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.505330+0000", 2024-08-19T14:13:23.133 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0074688929999999999 2024-08-19T14:13:23.133 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.133 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.133 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:13:23.133 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.505351+0000", 2024-08-19T14:13:23.133 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0333e-05 2024-08-19T14:13:23.133 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.133 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.133 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:13:23.133 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.513650+0000", 2024-08-19T14:13:23.133 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0082991030000000004 2024-08-19T14:13:23.134 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.134 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.134 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:13:23.134 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.513700+0000", 2024-08-19T14:13:23.134 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0185999999999997e-05 2024-08-19T14:13:23.134 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:23.134 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:13:23.134 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:23.134 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.134 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.134 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24580.0:64746 2.7 e129/127 2:e907466e:::benchmark_data_smithi045_46415_object64745:head v 129'3965, mlcod=0'0)", 2024-08-19T14:13:23.134 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:22.283936+0000", 2024-08-19T14:13:23.134 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.79336340800000005, 2024-08-19T14:13:23.134 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.015660540000000001, 2024-08-19T14:13:23.134 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:13:23.135 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:13:23.135 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:13:23.135 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:13:23.135 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.135 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:13:23.135 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.283936+0000", 2024-08-19T14:13:23.135 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:23.135 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.135 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.135 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:13:23.135 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.283936+0000", 2024-08-19T14:13:23.135 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:23.135 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.135 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.136 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:13:23.136 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.283937+0000", 2024-08-19T14:13:23.136 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6799999999999999e-07 2024-08-19T14:13:23.136 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.136 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.136 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:13:23.136 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.283978+0000", 2024-08-19T14:13:23.136 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0482000000000003e-05 2024-08-19T14:13:23.136 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.136 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.136 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:13:23.136 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.283999+0000", 2024-08-19T14:13:23.136 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1336999999999999e-05 2024-08-19T14:13:23.136 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.137 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.137 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:13:23.137 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.284006+0000", 2024-08-19T14:13:23.137 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4660000000000002e-06 2024-08-19T14:13:23.137 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.137 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.137 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:13:23.137 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.284069+0000", 2024-08-19T14:13:23.137 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2848000000000005e-05 2024-08-19T14:13:23.137 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.137 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.137 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:13:23.137 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.284097+0000", 2024-08-19T14:13:23.137 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7857000000000001e-05 2024-08-19T14:13:23.138 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.138 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.138 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:13:23.138 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.299549+0000", 2024-08-19T14:13:23.138 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.015451634000000001 2024-08-19T14:13:23.138 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.138 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.138 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:13:23.138 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.299597+0000", 2024-08-19T14:13:23.138 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8148e-05 2024-08-19T14:13:23.138 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:23.138 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:13:23.138 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:23.139 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.139 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.139 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24580.0:64923 2.7 e129/127 2:ef01f126:::benchmark_data_smithi045_46415_object64922:head v 129'3975, mlcod=0'0)", 2024-08-19T14:13:23.139 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:22.481822+0000", 2024-08-19T14:13:23.139 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.59547766899999999, 2024-08-19T14:13:23.139 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.015023626, 2024-08-19T14:13:23.139 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:13:23.139 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:13:23.139 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:13:23.139 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:13:23.139 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.139 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:13:23.139 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.481822+0000", 2024-08-19T14:13:23.139 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:23.139 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.140 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.140 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:13:23.140 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.481822+0000", 2024-08-19T14:13:23.140 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:23.140 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.140 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.140 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:13:23.140 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.481823+0000", 2024-08-19T14:13:23.140 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.37e-06 2024-08-19T14:13:23.140 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.140 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.140 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:13:23.140 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.481886+0000", 2024-08-19T14:13:23.140 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2605000000000001e-05 2024-08-19T14:13:23.141 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.141 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.141 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:13:23.141 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.481909+0000", 2024-08-19T14:13:23.141 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3213e-05 2024-08-19T14:13:23.141 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.141 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.141 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:13:23.141 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.481918+0000", 2024-08-19T14:13:23.141 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.0790000000000007e-06 2024-08-19T14:13:23.141 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.141 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.141 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:13:23.141 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.482012+0000", 2024-08-19T14:13:23.142 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3267e-05 2024-08-19T14:13:23.142 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.142 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.142 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:13:23.142 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.482043+0000", 2024-08-19T14:13:23.142 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1788000000000001e-05 2024-08-19T14:13:23.142 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.142 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.142 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:13:23.142 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.496800+0000", 2024-08-19T14:13:23.142 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.014756185 2024-08-19T14:13:23.142 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.142 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.143 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:13:23.143 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.496846+0000", 2024-08-19T14:13:23.143 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6118999999999997e-05 2024-08-19T14:13:23.143 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:23.143 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:13:23.143 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:23.143 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.143 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.143 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24580.0:64943 2.f e129/102 2:f4917538:::benchmark_data_smithi045_46415_object64942:head v 129'4101, mlcod=129'4099)", 2024-08-19T14:13:23.143 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:22.483598+0000", 2024-08-19T14:13:23.143 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.593701441, 2024-08-19T14:13:23.143 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.02030792, 2024-08-19T14:13:23.143 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:13:23.143 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:13:23.144 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:13:23.144 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:13:23.144 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.144 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:13:23.144 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.483598+0000", 2024-08-19T14:13:23.144 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:23.144 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.144 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.144 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:13:23.144 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.483598+0000", 2024-08-19T14:13:23.144 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:23.144 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.144 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.145 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:13:23.145 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.483599+0000", 2024-08-19T14:13:23.145 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0380000000000001e-06 2024-08-19T14:13:23.145 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.145 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.145 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:13:23.145 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.483667+0000", 2024-08-19T14:13:23.145 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7997999999999995e-05 2024-08-19T14:13:23.145 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.145 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.145 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:13:23.145 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.483682+0000", 2024-08-19T14:13:23.146 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.45e-05 2024-08-19T14:13:23.146 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.146 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.146 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:13:23.146 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.483688+0000", 2024-08-19T14:13:23.146 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6680000000000004e-06 2024-08-19T14:13:23.146 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.146 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.146 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:13:23.146 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.486328+0000", 2024-08-19T14:13:23.146 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0026395440000000002 2024-08-19T14:13:23.146 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.146 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.146 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:13:23.147 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.486351+0000", 2024-08-19T14:13:23.147 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2751e-05 2024-08-19T14:13:23.147 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.147 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.147 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:13:23.147 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.503857+0000", 2024-08-19T14:13:23.147 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.017505850999999999 2024-08-19T14:13:23.147 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.147 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.147 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:13:23.147 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.503906+0000", 2024-08-19T14:13:23.147 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9570000000000001e-05 2024-08-19T14:13:23.147 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:23.148 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:13:23.148 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:23.148 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.148 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.148 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24580.0:64926 2.7 e129/127 2:ee7d394c:::benchmark_data_smithi045_46415_object64925:head v 129'3976, mlcod=0'0)", 2024-08-19T14:13:23.148 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:22.485386+0000", 2024-08-19T14:13:23.148 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.59191363500000005, 2024-08-19T14:13:23.148 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.022392815999999999, 2024-08-19T14:13:23.148 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:13:23.148 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:13:23.148 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:13:23.148 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:13:23.148 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.148 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:13:23.149 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.485386+0000", 2024-08-19T14:13:23.149 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:23.149 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.149 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.149 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:13:23.149 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.485386+0000", 2024-08-19T14:13:23.149 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:23.149 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.149 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.149 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:13:23.149 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.485387+0000", 2024-08-19T14:13:23.149 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.9699999999999994e-07 2024-08-19T14:13:23.149 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.150 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.150 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:13:23.150 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.485434+0000", 2024-08-19T14:13:23.150 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7113000000000001e-05 2024-08-19T14:13:23.150 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.150 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.150 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:13:23.150 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.485447+0000", 2024-08-19T14:13:23.150 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2695e-05 2024-08-19T14:13:23.150 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.150 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.150 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:13:23.150 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.485455+0000", 2024-08-19T14:13:23.150 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.9799999999999998e-06 2024-08-19T14:13:23.151 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.151 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.151 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:13:23.152 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:13:23.152 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.487352+0000", 2024-08-19T14:13:23.152 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001896809 2024-08-19T14:13:23.152 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.153 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.153 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:13:23.153 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.487379+0000", 2024-08-19T14:13:23.153 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7248999999999999e-05 2024-08-19T14:13:23.153 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.153 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.153 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:13:23.153 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.507725+0000", 2024-08-19T14:13:23.153 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.020345797999999998 2024-08-19T14:13:23.153 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.153 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.153 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:13:23.153 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.507779+0000", 2024-08-19T14:13:23.153 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4175000000000002e-05 2024-08-19T14:13:23.154 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:23.154 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:13:23.154 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:23.154 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.154 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.154 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24580.0:64930 2.7 e129/127 2:e3128c9d:::benchmark_data_smithi045_46415_object64929:head v 129'3977, mlcod=0'0)", 2024-08-19T14:13:23.154 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:22.488188+0000", 2024-08-19T14:13:23.154 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.58911154899999996, 2024-08-19T14:13:23.154 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.019626971999999999, 2024-08-19T14:13:23.154 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:13:23.154 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:13:23.154 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:13:23.154 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:13:23.155 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.155 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:13:23.155 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.488188+0000", 2024-08-19T14:13:23.155 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:23.155 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.155 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.155 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:13:23.155 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.488188+0000", 2024-08-19T14:13:23.155 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:23.155 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.155 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.155 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:13:23.155 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.488189+0000", 2024-08-19T14:13:23.155 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6099999999999999e-07 2024-08-19T14:13:23.155 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.156 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.156 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:13:23.156 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.488230+0000", 2024-08-19T14:13:23.156 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1448e-05 2024-08-19T14:13:23.156 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.156 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.156 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:13:23.156 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.488242+0000", 2024-08-19T14:13:23.156 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1402e-05 2024-08-19T14:13:23.156 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.156 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.156 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:13:23.156 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.488249+0000", 2024-08-19T14:13:23.156 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7190000000000004e-06 2024-08-19T14:13:23.157 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.157 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.157 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:13:23.157 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.491525+0000", 2024-08-19T14:13:23.157 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0032765350000000001 2024-08-19T14:13:23.157 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.157 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.157 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:13:23.157 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.491554+0000", 2024-08-19T14:13:23.157 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9022000000000001e-05 2024-08-19T14:13:23.157 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.157 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.157 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:13:23.157 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.507783+0000", 2024-08-19T14:13:23.157 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.016228564000000001 2024-08-19T14:13:23.158 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.158 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.158 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:13:23.158 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.507815+0000", 2024-08-19T14:13:23.158 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2421e-05 2024-08-19T14:13:23.158 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:23.158 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:13:23.158 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:23.158 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.158 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.158 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24580.0:64948 2.f e129/102 2:fe97b461:::benchmark_data_smithi045_46415_object64947:head v 129'4102, mlcod=129'4099)", 2024-08-19T14:13:23.158 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:22.489275+0000", 2024-08-19T14:13:23.158 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.58802452800000005, 2024-08-19T14:13:23.158 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.018585981000000001, 2024-08-19T14:13:23.159 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:13:23.159 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:13:23.159 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:13:23.159 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:13:23.159 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.159 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:13:23.159 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.489275+0000", 2024-08-19T14:13:23.159 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:23.159 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.159 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.159 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:13:23.159 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.489275+0000", 2024-08-19T14:13:23.159 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:23.159 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.160 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.160 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:13:23.160 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.489276+0000", 2024-08-19T14:13:23.160 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.4200000000000004e-07 2024-08-19T14:13:23.160 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.160 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.160 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:13:23.160 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.489433+0000", 2024-08-19T14:13:23.160 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00015674299999999999 2024-08-19T14:13:23.160 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.160 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.160 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:13:23.160 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.489446+0000", 2024-08-19T14:13:23.160 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2738999999999999e-05 2024-08-19T14:13:23.161 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.161 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.161 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:13:23.161 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.489455+0000", 2024-08-19T14:13:23.161 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.1030000000000008e-06 2024-08-19T14:13:23.161 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.161 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.161 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:13:23.161 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.496819+0000", 2024-08-19T14:13:23.161 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0073646689999999999 2024-08-19T14:13:23.161 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.161 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.161 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:13:23.161 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.496838+0000", 2024-08-19T14:13:23.162 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8564999999999999e-05 2024-08-19T14:13:23.162 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.162 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.162 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:13:23.162 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.507820+0000", 2024-08-19T14:13:23.162 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010982122 2024-08-19T14:13:23.162 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.162 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.162 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:13:23.162 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.507861+0000", 2024-08-19T14:13:23.162 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1097999999999999e-05 2024-08-19T14:13:23.162 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:23.162 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:13:23.162 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:23.163 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.163 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.163 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24580.0:64932 2.7 e129/127 2:e500b327:::benchmark_data_smithi045_46415_object64931:head v 129'3978, mlcod=0'0)", 2024-08-19T14:13:23.163 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:22.490364+0000", 2024-08-19T14:13:23.163 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.58693609999999996, 2024-08-19T14:13:23.163 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.017530062999999999, 2024-08-19T14:13:23.163 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:13:23.163 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:13:23.163 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:13:23.163 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:13:23.163 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.163 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:13:23.163 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.490364+0000", 2024-08-19T14:13:23.163 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:23.163 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.164 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.164 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:13:23.164 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.490364+0000", 2024-08-19T14:13:23.164 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:23.164 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.164 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.164 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:13:23.164 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.490364+0000", 2024-08-19T14:13:23.164 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.8800000000000002e-07 2024-08-19T14:13:23.164 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.164 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.164 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:13:23.164 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.490411+0000", 2024-08-19T14:13:23.165 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6291000000000001e-05 2024-08-19T14:13:23.165 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.165 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.165 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:13:23.165 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.490426+0000", 2024-08-19T14:13:23.165 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.543e-05 2024-08-19T14:13:23.165 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.165 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.165 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:13:23.165 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.490433+0000", 2024-08-19T14:13:23.165 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3769999999999998e-06 2024-08-19T14:13:23.165 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.165 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.165 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:13:23.165 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.496875+0000", 2024-08-19T14:13:23.166 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0064419120000000002 2024-08-19T14:13:23.166 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.166 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.166 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:13:23.166 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.496900+0000", 2024-08-19T14:13:23.166 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4426999999999998e-05 2024-08-19T14:13:23.166 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.166 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.166 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:13:23.166 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.507864+0000", 2024-08-19T14:13:23.166 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010963914999999999 2024-08-19T14:13:23.166 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.166 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.167 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:13:23.167 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.507894+0000", 2024-08-19T14:13:23.167 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9923000000000001e-05 2024-08-19T14:13:23.167 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:23.167 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:13:23.167 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:23.167 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.167 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.167 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24580.0:65500 2.5 e130/102 2:aecb4856:::benchmark_data_smithi045_46415_object65499:head v 130'4132, mlcod=130'4130)", 2024-08-19T14:13:23.167 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:22.857484+0000", 2024-08-19T14:13:23.167 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.21981592699999999, 2024-08-19T14:13:23.167 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.014872279, 2024-08-19T14:13:23.167 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:13:23.167 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:13:23.167 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:13:23.168 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:13:23.168 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.168 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:13:23.168 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.857484+0000", 2024-08-19T14:13:23.168 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:23.168 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.168 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.168 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:13:23.168 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.857484+0000", 2024-08-19T14:13:23.168 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:23.168 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.168 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.168 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:13:23.168 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.857484+0000", 2024-08-19T14:13:23.168 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8400000000000004e-07 2024-08-19T14:13:23.169 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.169 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.169 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:13:23.169 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.869136+0000", 2024-08-19T14:13:23.169 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011651267 2024-08-19T14:13:23.169 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.169 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.169 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:13:23.169 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.869148+0000", 2024-08-19T14:13:23.169 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2462e-05 2024-08-19T14:13:23.169 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.169 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.169 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:13:23.169 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.869157+0000", 2024-08-19T14:13:23.170 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.4379999999999995e-06 2024-08-19T14:13:23.170 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.170 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.170 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:13:23.170 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.869227+0000", 2024-08-19T14:13:23.170 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0061999999999994e-05 2024-08-19T14:13:23.170 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.170 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.170 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:13:23.170 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.869251+0000", 2024-08-19T14:13:23.170 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3896999999999998e-05 2024-08-19T14:13:23.170 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.170 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.170 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:13:23.170 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.872318+0000", 2024-08-19T14:13:23.171 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0030679320000000002 2024-08-19T14:13:23.171 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:23.171 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:23.171 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:13:23.171 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.872356+0000", 2024-08-19T14:13:23.171 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7537000000000003e-05 2024-08-19T14:13:23.171 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:23.171 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:13:23.171 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:23.171 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:23.171 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:13:23.171 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:13:23.179 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:23.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:23.180 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:13:23.281 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:13:23.311 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:23.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:23.312 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:13:23.413 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:13:23.454 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:23.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:23.455 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:13:23.557 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:13:23.585 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:23.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:23.587 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:13:23.688 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:13:23.725 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:23.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:23.726 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:13:23.827 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:13:23.832 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 34 16 66813 66797 122.778 83.875 0.0371302 0.00813921 2024-08-19T14:13:23.860 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:23.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:23.861 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:13:23.962 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:13:24.001 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:24.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:24.002 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:13:24.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:23 smithi032 ceph-mon[20004]: pgmap v333: 27 pgs: 2 remapped+peering, 1 unknown, 3 active+recovering+undersized+remapped, 21 active+clean; 3.8 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 2.5 KiB/s rd, 72 MiB/s wr, 1.17k op/s; 9029/187845 objects misplaced (4.807%); 12 MiB/s, 196 objects/s recovering 2024-08-19T14:13:24.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:23 smithi032 ceph-mon[20004]: osdmap e130: 12 total, 12 up, 10 in 2024-08-19T14:13:24.016 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:13:24.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:23 smithi045 ceph-mon[20143]: pgmap v333: 27 pgs: 2 remapped+peering, 1 unknown, 3 active+recovering+undersized+remapped, 21 active+clean; 3.8 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 2.5 KiB/s rd, 72 MiB/s wr, 1.17k op/s; 9029/187845 objects misplaced (4.807%); 12 MiB/s, 196 objects/s recovering 2024-08-19T14:13:24.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:23 smithi045 ceph-mon[20143]: osdmap e130: 12 total, 12 up, 10 in 2024-08-19T14:13:24.065 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:23 smithi031 ceph-mon[16336]: pgmap v333: 27 pgs: 2 remapped+peering, 1 unknown, 3 active+recovering+undersized+remapped, 21 active+clean; 3.8 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 2.5 KiB/s rd, 72 MiB/s wr, 1.17k op/s; 9029/187845 objects misplaced (4.807%); 12 MiB/s, 196 objects/s recovering 2024-08-19T14:13:24.065 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:23 smithi031 ceph-mon[16336]: osdmap e130: 12 total, 12 up, 10 in 2024-08-19T14:13:24.103 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:13:24.136 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:24.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:24.137 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:13:24.238 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:13:24.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:24.290 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:24.290 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:13:24.391 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:13:24.413 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:24.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:24.414 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:13:24.515 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:13:24.540 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:24.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:24.541 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:13:24.642 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:13:24.670 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:24.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:24.671 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:13:24.772 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:13:24.797 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:24.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:24.798 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:13:24.832 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 35 16 67827 67811 121.08 63.375 0.0870756 0.00824503 2024-08-19T14:13:24.899 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:13:24.928 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:24.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:24.929 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:13:25.030 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:13:25.051 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:25.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:25.052 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:13:25.153 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:13:25.192 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:25.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:25.193 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:13:25.294 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:13:25.331 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:25.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:25.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:13:25.434 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:13:25.462 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:25.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:25.463 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:13:25.564 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:13:25.595 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:25.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:25.596 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:13:25.697 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:13:25.728 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:25.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:25.729 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:13:25.830 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:13:25.833 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 36 16 69025 69009 119.797 74.875 0.0103559 0.00834206 2024-08-19T14:13:25.860 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:25.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:25.861 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:13:25.962 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:13:25.994 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:25.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:25.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:13:26.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:25 smithi032 ceph-mon[20004]: pgmap v335: 27 pgs: 1 peering, 2 remapped+peering, 3 active+recovering+undersized+remapped, 21 active+clean; 3.9 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 47 MiB/s wr, 746 op/s; 7784/192138 objects misplaced (4.051%); 20 MiB/s, 319 objects/s recovering 2024-08-19T14:13:26.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:25 smithi045 ceph-mon[20143]: pgmap v335: 27 pgs: 1 peering, 2 remapped+peering, 3 active+recovering+undersized+remapped, 21 active+clean; 3.9 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 47 MiB/s wr, 746 op/s; 7784/192138 objects misplaced (4.051%); 20 MiB/s, 319 objects/s recovering 2024-08-19T14:13:26.096 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:13:26.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:25 smithi031 ceph-mon[16336]: pgmap v335: 27 pgs: 1 peering, 2 remapped+peering, 3 active+recovering+undersized+remapped, 21 active+clean; 3.9 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 47 MiB/s wr, 746 op/s; 7784/192138 objects misplaced (4.051%); 20 MiB/s, 319 objects/s recovering 2024-08-19T14:13:26.122 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:26.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:26.123 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:13:26.224 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:13:26.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:26.266 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:26.267 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:13:26.367 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:13:26.400 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-08-19T14:13:26.402 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:26.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:26.404 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:13:26.505 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:13:26.530 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:26.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:26.531 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:13:26.558 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:13:26.632 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:13:26.687 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:26.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:26.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:13:26.776 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:13:26.788 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:13:26.823 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:26.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:26.824 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:13:26.832 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 37 16 70394 70378 118.871 85.5625 0.0223326 0.00840596 2024-08-19T14:13:26.924 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:13:26.953 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:26.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:26.954 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:13:27.055 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:13:27.088 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:27.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:27.089 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:13:27.190 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:13:27.217 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:27.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:27.220 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:13:27.259 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:13:27.321 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:13:27.353 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:27.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:27.354 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:13:27.378 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:13:27.378 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-08-19T14:13:27.378 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:27.378 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:71107 2.2 2.45dbaa42 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-08-19T14:13:27.379 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:27.370129+0000", 2024-08-19T14:13:27.379 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.0061691009999999998, 2024-08-19T14:13:27.379 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0061959390000000001, 2024-08-19T14:13:27.379 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:13:27.379 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:13:27.379 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "queued for pg", 2024-08-19T14:13:27.379 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:13:27.379 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:13:27.379 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:13:27.379 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 71107 2024-08-19T14:13:27.379 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:27.379 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:13:27.379 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:27.380 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:13:27.380 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:27.370129+0000", 2024-08-19T14:13:27.380 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:27.380 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:27.380 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:27.380 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:13:27.380 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:27.370129+0000", 2024-08-19T14:13:27.380 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:27.380 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:27.380 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:27.380 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:13:27.380 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:27.370130+0000", 2024-08-19T14:13:27.380 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.266e-06 2024-08-19T14:13:27.380 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:27.381 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:27.381 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:13:27.381 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:27.370186+0000", 2024-08-19T14:13:27.381 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6079000000000003e-05 2024-08-19T14:13:27.381 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:27.381 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:27.381 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:13:27.381 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:27.370198+0000", 2024-08-19T14:13:27.381 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2539e-05 2024-08-19T14:13:27.381 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:27.381 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:27.381 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:13:27.381 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:27.370205+0000", 2024-08-19T14:13:27.381 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8929999999999999e-06 2024-08-19T14:13:27.382 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:27.382 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:13:27.382 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:27.382 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:27.382 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:27.382 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:71112 2.2 2.69ac5b72 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-08-19T14:13:27.382 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:27.374249+0000", 2024-08-19T14:13:27.382 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.0020490619999999999, 2024-08-19T14:13:27.382 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0021288330000000001, 2024-08-19T14:13:27.382 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:13:27.382 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:13:27.382 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "queued for pg", 2024-08-19T14:13:27.382 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:13:27.382 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:13:27.383 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:13:27.383 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 71112 2024-08-19T14:13:27.383 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:27.383 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:13:27.383 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:27.383 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:13:27.383 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:27.374249+0000", 2024-08-19T14:13:27.383 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:27.383 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:27.383 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:27.383 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:13:27.383 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:27.374249+0000", 2024-08-19T14:13:27.383 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:27.383 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:27.384 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:27.384 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:13:27.384 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:27.374253+0000", 2024-08-19T14:13:27.384 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.454e-06 2024-08-19T14:13:27.384 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:27.384 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:27.384 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:13:27.384 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:27.374293+0000", 2024-08-19T14:13:27.384 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0450000000000001e-05 2024-08-19T14:13:27.384 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:27.384 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:27.384 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:13:27.384 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:27.374310+0000", 2024-08-19T14:13:27.384 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6277999999999999e-05 2024-08-19T14:13:27.384 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:27.385 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:27.385 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:13:27.385 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:27.374318+0000", 2024-08-19T14:13:27.385 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6559999999999998e-06 2024-08-19T14:13:27.385 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:27.385 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:13:27.385 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:27.385 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:27.385 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-08-19T14:13:27.385 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 2 2024-08-19T14:13:27.385 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:13:27.455 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:13:27.482 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:27.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:27.483 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:13:27.584 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:13:27.611 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:27.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:27.612 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:13:27.713 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:13:27.734 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is set 2024-08-19T14:13:27.740 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:27.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:27.741 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:13:27.799 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:27 smithi031 ceph-mon[16336]: pgmap v336: 27 pgs: 1 peering, 4 active+recovering+undersized+remapped, 22 active+clean; 4.1 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 59 MiB/s wr, 944 op/s; 13813/200121 objects misplaced (6.902%); 14 MiB/s, 231 objects/s recovering 2024-08-19T14:13:27.799 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:27 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:13:27.799 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:27 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:13:27.799 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:27 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3063141748' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:13:27.833 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 38 16 71667 71651 117.837 79.5625 0.00524498 0.00848225 2024-08-19T14:13:27.842 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:13:27.873 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:13:27.874 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:13:27.883 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:27.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:27.884 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:13:27.909 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:13:27.910 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:13:27.942 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:13:27.942 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:13:27.954 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:27 smithi032 ceph-mon[20004]: pgmap v336: 27 pgs: 1 peering, 4 active+recovering+undersized+remapped, 22 active+clean; 4.1 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 59 MiB/s wr, 944 op/s; 13813/200121 objects misplaced (6.902%); 14 MiB/s, 231 objects/s recovering 2024-08-19T14:13:27.954 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:27 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:13:27.954 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:27 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:13:27.954 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:27 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3063141748' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:13:27.980 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:13:27.980 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:13:27.986 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:13:28.018 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:13:28.019 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:13:28.025 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:28.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:28.027 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:13:28.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:27 smithi045 ceph-mon[20143]: pgmap v336: 27 pgs: 1 peering, 4 active+recovering+undersized+remapped, 22 active+clean; 4.1 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 59 MiB/s wr, 944 op/s; 13813/200121 objects misplaced (6.902%); 14 MiB/s, 231 objects/s recovering 2024-08-19T14:13:28.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:27 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:13:28.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:27 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:13:28.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:27 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3063141748' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:13:28.051 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:13:28.051 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:13:28.091 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:13:28.091 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:13:28.125 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:13:28.125 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:13:28.127 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:13:28.138 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-08-19T14:13:28.159 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:13:28.159 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:13:28.168 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:28.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:28.169 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:13:28.195 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:13:28.195 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:13:28.227 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:13:28.227 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:13:28.260 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:13:28.260 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:13:28.269 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:13:28.305 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:28.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:28.307 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:13:28.408 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:13:28.438 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:28.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:28.439 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:13:28.535 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:13:28.539 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:13:28.561 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:28.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:28.562 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:13:28.663 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:13:28.691 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:28.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:28.692 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:13:28.793 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:13:28.826 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:28.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:28.827 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:13:28.833 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 39 16 72798 72782 116.628 70.6875 0.00423967 0.00856445 2024-08-19T14:13:28.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:28 smithi031 ceph-mon[16336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:13:28.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:28 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3063141748' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:13:28.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:28 smithi031 ceph-mon[16336]: osdmap e131: 12 total, 12 up, 10 in 2024-08-19T14:13:28.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:28 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2585725435' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:13:28.928 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:13:28.955 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:28.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:28.956 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:13:29.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:28 smithi032 ceph-mon[20004]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:13:29.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:28 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3063141748' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:13:29.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:28 smithi032 ceph-mon[20004]: osdmap e131: 12 total, 12 up, 10 in 2024-08-19T14:13:29.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:28 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2585725435' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:13:29.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:28 smithi045 ceph-mon[20143]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:13:29.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:28 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3063141748' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:13:29.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:28 smithi045 ceph-mon[20143]: osdmap e131: 12 total, 12 up, 10 in 2024-08-19T14:13:29.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:28 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2585725435' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:13:29.057 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:13:29.094 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:29.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:29.095 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:13:29.196 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:13:29.222 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:29.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:29.223 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:13:29.324 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:13:29.349 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:29.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:29.350 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:13:29.451 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:13:29.480 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:29.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:29.482 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:13:29.582 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:13:29.619 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:29.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:29.620 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:13:29.721 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:13:29.759 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:29.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:29.760 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:13:29.833 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:2024-08-19T14:13:29.833698+0000 min lat: 0.00106853 max lat: 2.17987 avg lat: 0.00864152 2024-08-19T14:13:29.833 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-19T14:13:29.833 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 40 16 74032 74016 115.64 77.125 0.059094 0.00864152 2024-08-19T14:13:29.861 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:13:29.889 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:29.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:29.890 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:13:29.991 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:13:30.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:29 smithi032 ceph-mon[20004]: pgmap v338: 27 pgs: 4 active+recovering+undersized+remapped, 23 active+clean; 4.2 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 62 MiB/s wr, 999 op/s; 13287/205911 objects misplaced (6.453%); 19 MiB/s, 311 objects/s recovering 2024-08-19T14:13:30.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:30.032 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:30.032 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:13:30.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:29 smithi045 ceph-mon[20143]: pgmap v338: 27 pgs: 4 active+recovering+undersized+remapped, 23 active+clean; 4.2 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 62 MiB/s wr, 999 op/s; 13287/205911 objects misplaced (6.453%); 19 MiB/s, 311 objects/s recovering 2024-08-19T14:13:30.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:29 smithi031 ceph-mon[16336]: pgmap v338: 27 pgs: 4 active+recovering+undersized+remapped, 23 active+clean; 4.2 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 62 MiB/s wr, 999 op/s; 13287/205911 objects misplaced (6.453%); 19 MiB/s, 311 objects/s recovering 2024-08-19T14:13:30.133 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:13:30.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:30.161 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:30.161 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:13:30.262 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:13:30.289 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:30.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:30.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:13:30.391 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:13:30.436 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:30.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:30.438 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:13:30.539 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:13:30.567 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:30.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:30.569 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:13:30.582 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-19T14:13:30.669 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:13:30.716 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:30.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:30.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:13:30.788 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:13:30.789 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:13:30.818 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:13:30.833 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 41 16 75164 75148 114.545 70.75 0.00298172 0.00870594 2024-08-19T14:13:30.842 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:30.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:30.843 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:13:30.944 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:13:30.948 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:13:30.970 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:30.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:30.971 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:13:31.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:30 smithi032 ceph-mon[20004]: pgmap v339: 27 pgs: 4 active+recovering+undersized+remapped, 23 active+clean; 4.3 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 65 MiB/s wr, 1.04k op/s; 12569/212955 objects misplaced (5.902%); 20 MiB/s, 324 objects/s recovering 2024-08-19T14:13:31.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:30 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:13:31.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:30 smithi045 ceph-mon[20143]: pgmap v339: 27 pgs: 4 active+recovering+undersized+remapped, 23 active+clean; 4.3 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 65 MiB/s wr, 1.04k op/s; 12569/212955 objects misplaced (5.902%); 20 MiB/s, 324 objects/s recovering 2024-08-19T14:13:31.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:30 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:13:31.072 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:13:31.098 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:30 smithi031 ceph-mon[16336]: pgmap v339: 27 pgs: 4 active+recovering+undersized+remapped, 23 active+clean; 4.3 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 65 MiB/s wr, 1.04k op/s; 12569/212955 objects misplaced (5.902%); 20 MiB/s, 324 objects/s recovering 2024-08-19T14:13:31.098 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:30 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:13:31.099 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:31.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:31.100 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:13:31.201 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:13:31.231 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:31.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:31.233 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:13:31.277 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:13:31.333 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:13:31.389 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:31.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:31.390 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:13:31.491 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:13:31.521 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:31.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:31.522 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:13:31.587 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 8, 9, 11] out_osds: [10, 6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-08-19T14:13:31.587 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-19T14:13:31.587 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_0 2024-08-19T14:13:31.587 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-19T14:13:31.623 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:13:31.680 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:31.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:31.681 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:13:31.782 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:13:31.808 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:31.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:31.809 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:13:31.833 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 42 16 76197 76181 113.355 64.5625 0.00424347 0.00881243 2024-08-19T14:13:31.910 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:13:31.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:31.933 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:31.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:13:32.034 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:13:32.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:32.057 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:32.057 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:13:32.158 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:13:32.182 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:32.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:32.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:13:32.284 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:13:32.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:32.312 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:32.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:13:32.413 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:13:32.437 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:32.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:32.438 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:13:32.539 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:13:32.561 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:32.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:32.562 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:13:32.663 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:13:32.690 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:32.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:32.691 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:13:32.792 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:13:32.829 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:32.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:32.830 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:13:32.833 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 43 16 76571 76555 111.262 23.375 0.017388 0.00898234 2024-08-19T14:13:32.931 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:13:32.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:32.963 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:32.964 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:13:33.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:32 smithi032 ceph-mon[20004]: pgmap v340: 27 pgs: 4 active+recovering+undersized+remapped, 23 active+clean; 4.5 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 71 MiB/s wr, 1.14k op/s; 11930/220947 objects misplaced (5.399%); 18 MiB/s, 289 objects/s recovering 2024-08-19T14:13:33.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:32 smithi032 ceph-mon[20004]: osdmap e132: 12 total, 12 up, 10 in 2024-08-19T14:13:33.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:32 smithi045 ceph-mon[20143]: pgmap v340: 27 pgs: 4 active+recovering+undersized+remapped, 23 active+clean; 4.5 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 71 MiB/s wr, 1.14k op/s; 11930/220947 objects misplaced (5.399%); 18 MiB/s, 289 objects/s recovering 2024-08-19T14:13:33.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:32 smithi045 ceph-mon[20143]: osdmap e132: 12 total, 12 up, 10 in 2024-08-19T14:13:33.064 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:13:33.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:33.113 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:33.113 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:13:33.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:32 smithi031 ceph-mon[16336]: pgmap v340: 27 pgs: 4 active+recovering+undersized+remapped, 23 active+clean; 4.5 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 71 MiB/s wr, 1.14k op/s; 11930/220947 objects misplaced (5.399%); 18 MiB/s, 289 objects/s recovering 2024-08-19T14:13:33.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:32 smithi031 ceph-mon[16336]: osdmap e132: 12 total, 12 up, 10 in 2024-08-19T14:13:33.214 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:13:33.236 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:33.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:33.237 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:13:33.338 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:13:33.363 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:33.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:33.364 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:13:33.465 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:13:33.498 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:33.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:33.499 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:13:33.600 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:13:33.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:33.639 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:33.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:13:33.740 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:13:33.770 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:33.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:33.771 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:13:33.833 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 44 16 77706 77690 110.346 70.9375 0.0353295 0.00905204 2024-08-19T14:13:33.863 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is unset 2024-08-19T14:13:33.871 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:13:33.896 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:33.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:33.897 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:13:33.998 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:13:34.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:33 smithi032 ceph-mon[20004]: osdmap e133: 12 total, 12 up, 10 in 2024-08-19T14:13:34.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:33 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:13:34.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:33 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:13:34.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:33 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:13:34.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:33 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:13:34.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:33 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:13:34.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:33 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2685138018' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:13:34.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:33 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:13:34.026 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:34.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:34.027 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:13:34.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:33 smithi045 ceph-mon[20143]: osdmap e133: 12 total, 12 up, 10 in 2024-08-19T14:13:34.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:33 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:13:34.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:33 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:13:34.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:33 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:13:34.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:33 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:13:34.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:33 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:13:34.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:33 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2685138018' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:13:34.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:33 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:13:34.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:33 smithi031 ceph-mon[16336]: osdmap e133: 12 total, 12 up, 10 in 2024-08-19T14:13:34.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:33 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:13:34.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:33 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:13:34.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:33 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:13:34.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:33 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:13:34.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:33 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:13:34.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:33 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2685138018' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:13:34.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:33 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:13:34.128 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:13:34.158 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:34.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:34.159 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:13:34.260 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:13:34.300 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:34.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:34.301 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:13:34.402 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:13:34.430 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:34.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:34.432 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:13:34.532 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:13:34.566 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:34.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:34.567 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:13:34.668 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:13:34.695 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:34.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:34.696 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:13:34.797 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:13:34.818 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:34.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:34.819 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:13:34.833 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 45 16 78875 78859 109.517 73.0625 0.00303811 0.00911823 2024-08-19T14:13:34.920 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:13:34.939 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-08-19T14:13:34.953 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:34.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:34.954 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:13:35.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:34 smithi032 ceph-mon[20004]: pgmap v343: 27 pgs: 3 active+recovering+undersized+remapped, 24 active+clean; 4.6 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 71 MiB/s wr, 1.14k op/s; 10767/226473 objects misplaced (4.754%); 25 MiB/s, 396 objects/s recovering 2024-08-19T14:13:35.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:34 smithi032 ceph-mon[20004]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:13:35.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:34 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:13:35.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:34 smithi032 ceph-mon[20004]: osdmap e134: 12 total, 12 up, 10 in 2024-08-19T14:13:35.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:34 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:13:35.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:34 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:13:35.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:34 smithi045 ceph-mon[20143]: pgmap v343: 27 pgs: 3 active+recovering+undersized+remapped, 24 active+clean; 4.6 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 71 MiB/s wr, 1.14k op/s; 10767/226473 objects misplaced (4.754%); 25 MiB/s, 396 objects/s recovering 2024-08-19T14:13:35.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:34 smithi045 ceph-mon[20143]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:13:35.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:34 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:13:35.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:34 smithi045 ceph-mon[20143]: osdmap e134: 12 total, 12 up, 10 in 2024-08-19T14:13:35.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:34 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:13:35.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:34 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:13:35.055 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:13:35.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:35.085 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:35.086 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:13:35.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:34 smithi031 ceph-mon[16336]: pgmap v343: 27 pgs: 3 active+recovering+undersized+remapped, 24 active+clean; 4.6 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 71 MiB/s wr, 1.14k op/s; 10767/226473 objects misplaced (4.754%); 25 MiB/s, 396 objects/s recovering 2024-08-19T14:13:35.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:34 smithi031 ceph-mon[16336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:13:35.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:34 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:13:35.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:34 smithi031 ceph-mon[16336]: osdmap e134: 12 total, 12 up, 10 in 2024-08-19T14:13:35.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:34 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:13:35.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:34 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:13:35.187 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:13:35.215 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:35.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:35.216 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:13:35.282 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:13:35.317 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:13:35.344 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:35.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:35.345 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:13:35.446 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:13:35.473 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:35.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:35.474 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:13:35.575 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:13:35.599 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:35.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:35.601 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:13:35.701 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:13:35.736 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:35.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:35.737 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:13:35.833 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 46 16 80235 80219 108.984 85 0.0627141 0.00916912 2024-08-19T14:13:35.838 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:13:35.866 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:35.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:35.867 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:13:35.932 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:13:35.932 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:13:35.933 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:35 smithi032 ceph-mon[20004]: osdmap e135: 12 total, 12 up, 10 in 2024-08-19T14:13:35.967 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:13:35.983 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:13:35.992 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:13:36.014 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:36.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:36.015 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:13:36.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:35 smithi045 ceph-mon[20143]: osdmap e135: 12 total, 12 up, 10 in 2024-08-19T14:13:36.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:35 smithi031 ceph-mon[16336]: osdmap e135: 12 total, 12 up, 10 in 2024-08-19T14:13:36.116 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:13:36.141 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:36.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:36.142 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:13:36.243 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:13:36.270 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:36.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:36.271 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:13:36.372 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:13:36.401 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:36.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:36.403 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:13:36.503 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:13:36.543 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:36.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:36.544 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:13:36.645 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:13:36.671 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:36.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:36.672 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:13:36.773 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:13:36.796 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:36.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:36.797 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:13:36.833 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 47 15 81329 81314 108.121 68.4375 0.0100729 0.00924289 2024-08-19T14:13:36.898 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:13:36.923 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:36.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:36.924 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:13:36.971 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:13:36.971 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:13:37.017 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:13:37.018 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:13:37.022 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:36 smithi032 ceph-mon[20004]: pgmap v346: 27 pgs: 3 active+recovering+undersized+remapped, 24 active+clean; 4.8 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 68 MiB/s wr, 1.08k op/s; 10453/233922 objects misplaced (4.469%); 21 MiB/s, 329 objects/s recovering 2024-08-19T14:13:37.025 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:13:37.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:36 smithi045 ceph-mon[20143]: pgmap v346: 27 pgs: 3 active+recovering+undersized+remapped, 24 active+clean; 4.8 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 68 MiB/s wr, 1.08k op/s; 10453/233922 objects misplaced (4.469%); 21 MiB/s, 329 objects/s recovering 2024-08-19T14:13:37.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:37.050 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:37.051 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:13:37.051 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:13:37.051 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:13:37.052 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:36 smithi031 ceph-mon[16336]: pgmap v346: 27 pgs: 3 active+recovering+undersized+remapped, 24 active+clean; 4.8 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 68 MiB/s wr, 1.08k op/s; 10453/233922 objects misplaced (4.469%); 21 MiB/s, 329 objects/s recovering 2024-08-19T14:13:37.059 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-08-19T14:13:37.086 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:13:37.086 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:13:37.094 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:13:37.094 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-08-19T14:13:37.125 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:13:37.125 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:13:37.151 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:13:37.168 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:13:37.168 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:13:37.178 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:37.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:37.179 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:13:37.204 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:13:37.205 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:13:37.238 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:13:37.239 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:13:37.272 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:13:37.272 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:13:37.280 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:13:37.310 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:13:37.310 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:13:37.318 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:37.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:37.322 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:13:37.346 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:13:37.346 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:13:37.381 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:13:37.382 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:13:37.390 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:13:37.423 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:13:37.449 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:37.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:37.450 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:13:37.551 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:13:37.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:37.583 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:37.584 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:13:37.685 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:13:37.721 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:37.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:37.722 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:13:37.823 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:13:37.833 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 48 15 82480 82465 107.367 71.9375 0.0155117 0.00930964 2024-08-19T14:13:37.853 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:37.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:37.855 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:13:37.881 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-19T14:13:37.956 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:13:37.963 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:13:37.964 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-08-19T14:13:37.964 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-08-19T14:13:37.964 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:13:37.985 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:37.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:37.986 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:13:38.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:37 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1496912287' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:13:38.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:37 smithi045 ceph-mon[20143]: from='client.24643 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:13:38.087 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:13:38.110 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:38.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:38.111 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:13:38.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:37 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1496912287' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:13:38.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:37 smithi031 ceph-mon[16336]: from='client.24643 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:13:38.211 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:13:38.239 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:38.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:38.240 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:13:38.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:37 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1496912287' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:13:38.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:37 smithi032 ceph-mon[20004]: from='client.24643 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:13:38.304 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":346,"stamp":"2024-08-19T14:13:35.744766+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5110712504,"num_objects":77974,"num_object_clones":0,"num_object_copies":233922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10453,"num_objects_unfound":0,"num_objects_dirty":77974,"num_whiteouts":0,"num_read":46,"num_read_kb":40,"num_write":78037,"num_write_kb":4991213,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6057,"num_bytes_recovered":397458758,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79500,"ondisk_log_size":79500,"up":81,"acting":78,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":81,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":14753336,"kb_used_data":14195844,"kb_used_omap":15,"kb_used_meta":557360,"kb_avail":922698184,"statfs":{"total":959950356480,"available":944842940416,"internally_reserved":0,"allocated":14536544256,"data_stored":14529588460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15948,"internal_metadata":570737076},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":283443200,"num_objects":4325,"num_object_clones":0,"num_object_copies":12975,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1477,"num_objects_unfound":0,"num_objects_dirty":4325,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4325,"num_write_kb":276800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1320,"num_bytes_recovered":86507520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000932"},"pg_stats":[{"pgid":"2.f","version":"134'4942","reported_seq":5016,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-08-19T14:13:34.618005+0000","last_change":"2024-08-19T14:12:43.418438+0000","last_active":"2024-08-19T14:13:34.618005+0000","last_peered":"2024-08-19T14:13:34.618005+0000","last_clean":"2024-08-19T14:13:34.618005+0000","last_became_active":"2024-08-19T14:12:43.417591+0000","last_became_peered":"2024-08-19T14:12:43.417591+0000","last_unstale":"2024-08-19T14:13:34.618005+0000","last_undegraded":"2024-08-19T14:13:34.618005+0000","last_fullsized":"2024-08-19T14:13:34.618005+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:12:41.713780+0000","objects_scrubbed":0,"log_size":4942,"log_dups_size":0,"ondisk_log_size":4942,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:14:03.830426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":323878912,"num_objects":4942,"num_object_clones":0,"num_object_copies":14826,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4942,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4942,"num_write_kb":316288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.e","version":"135'4974","reported_seq":5050,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-19T14:13:35.697482+0000","last_change":"2024-08-19T14:12:42.963558+0000","last_active":"2024-08-19T14:13:35.697482+0000","last_peered":"2024-08-19T14:13:35.697482+0000","last_clean":"2024-08-19T14:13:35.697482+0000","last_became_active":"2024-08-19T14:12:42.963208+0000","last_became_peered":"2024-08-19T14:12:42.963208+0000","last_unstale":"2024-08-19T14:13:35.697482+0000","last_undegraded":"2024-08-19T14:13:35.697482+0000","last_fullsized":"2024-08-19T14:13:35.697482+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:12:41.713780+0000","objects_scrubbed":0,"log_size":4974,"log_dups_size":0,"ondisk_log_size":4974,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:14:05.735203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":325976064,"num_objects":4974,"num_object_clones":0,"num_object_copies":14922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4974,"num_write_kb":318336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9],"acting":[2,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"133'4877","reported_seq":4949,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-08-19T14:13:33.122823+0000","last_change":"2024-08-19T14:12:43.417948+0000","last_active":"2024-08-19T14:13:33.122823+0000","last_peered":"2024-08-19T14:13:33.122823+0000","last_clean":"2024-08-19T14:13:33.122823+0000","last_became_active":"2024-08-19T14:12:43.417506+0000","last_became_peered":"2024-08-19T14:12:43.417506+0000","last_unstale":"2024-08-19T14:13:33.122823+0000","last_undegraded":"2024-08-19T14:13:33.122823+0000","last_fullsized":"2024-08-19T14:13:33.122823+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:12:41.713780+0000","objects_scrubbed":0,"log_size":4877,"log_dups_size":0,"ondisk_log_size":4877,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:13:53.364424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":319619072,"num_objects":4877,"num_object_clones":0,"num_object_copies":14631,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4877,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4877,"num_write_kb":312128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.c","version":"135'4977","reported_seq":5053,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-19T14:13:35.509064+0000","last_change":"2024-08-19T14:12:43.039428+0000","last_active":"2024-08-19T14:13:35.509064+0000","last_peered":"2024-08-19T14:13:35.509064+0000","last_clean":"2024-08-19T14:13:35.509064+0000","last_became_active":"2024-08-19T14:12:43.038912+0000","last_became_peered":"2024-08-19T14:12:43.038912+0000","last_unstale":"2024-08-19T14:13:35.509064+0000","last_undegraded":"2024-08-19T14:13:35.509064+0000","last_fullsized":"2024-08-19T14:13:35.509064+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:12:41.713780+0000","objects_scrubbed":0,"log_size":4977,"log_dups_size":0,"ondisk_log_size":4977,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:14:01.252952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":326172672,"num_objects":4977,"num_object_clones":0,"num_object_copies":14931,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4977,"num_write_kb":318528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,1],"acting":[11,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"70'139","reported_seq":449,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-08-19T14:13:32.747691+0000","last_change":"2024-08-19T14:13:22.729920+0000","last_active":"2024-08-19T14:13:32.747691+0000","last_peered":"2024-08-19T14:13:32.747691+0000","last_clean":"2024-08-19T14:13:32.747691+0000","last_became_active":"2024-08-19T14:13:22.729498+0000","last_became_peered":"2024-08-19T14:13:22.729498+0000","last_unstale":"2024-08-19T14:13:32.747691+0000","last_undegraded":"2024-08-19T14:13:32.747691+0000","last_fullsized":"2024-08-19T14:13:32.747691+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:13:01.659749+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:13:01.659749+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:14:31.517588+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,9],"acting":[7,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"131'4799","reported_seq":4867,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-08-19T14:13:31.389818+0000","last_change":"2024-08-19T14:12:42.729380+0000","last_active":"2024-08-19T14:13:31.389818+0000","last_peered":"2024-08-19T14:13:31.389818+0000","last_clean":"2024-08-19T14:13:31.389818+0000","last_became_active":"2024-08-19T14:12:42.728978+0000","last_became_peered":"2024-08-19T14:12:42.728978+0000","last_unstale":"2024-08-19T14:13:31.389818+0000","last_undegraded":"2024-08-19T14:13:31.389818+0000","last_fullsized":"2024-08-19T14:13:31.389818+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:12:41.713780+0000","objects_scrubbed":0,"log_size":4799,"log_dups_size":0,"ondisk_log_size":4799,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:14:01.040565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":314507264,"num_objects":4799,"num_object_clones":0,"num_object_copies":14397,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4799,"num_write_kb":307136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.3","version":"70'139","reported_seq":332,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-08-19T14:13:31.744689+0000","last_change":"2024-08-19T14:13:22.727606+0000","last_active":"2024-08-19T14:13:31.744689+0000","last_peered":"2024-08-19T14:13:31.744689+0000","last_clean":"2024-08-19T14:13:31.744689+0000","last_became_active":"2024-08-19T14:13:22.727234+0000","last_became_peered":"2024-08-19T14:13:22.727234+0000","last_unstale":"2024-08-19T14:13:31.744689+0000","last_undegraded":"2024-08-19T14:13:31.744689+0000","last_fullsized":"2024-08-19T14:13:31.744689+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:13:05.831711+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:13:05.831711+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:14:24.421871+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"133'4771","reported_seq":10902,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-08-19T14:13:32.931026+0000","last_change":"2024-08-19T14:13:32.755198+0000","last_active":"2024-08-19T14:13:32.931026+0000","last_peered":"2024-08-19T14:13:32.931026+0000","last_clean":"2024-08-19T14:13:32.931026+0000","last_became_active":"2024-08-19T14:13:32.749994+0000","last_became_peered":"2024-08-19T14:13:32.749994+0000","last_unstale":"2024-08-19T14:13:32.931026+0000","last_undegraded":"2024-08-19T14:13:32.931026+0000","last_fullsized":"2024-08-19T14:13:32.931026+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:12:41.713780+0000","objects_scrubbed":0,"log_size":4771,"log_dups_size":0,"ondisk_log_size":4771,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:13:45.602497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":312672256,"num_objects":4771,"num_object_clones":0,"num_object_copies":14313,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4771,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4771,"num_write_kb":305344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3024,"num_bytes_recovered":198180864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,11],"acting":[7,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"70'139","reported_seq":448,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-08-19T14:13:32.747597+0000","last_change":"2024-08-19T14:13:06.732015+0000","last_active":"2024-08-19T14:13:32.747597+0000","last_peered":"2024-08-19T14:13:32.747597+0000","last_clean":"2024-08-19T14:13:32.747597+0000","last_became_active":"2024-08-19T14:13:05.607783+0000","last_became_peered":"2024-08-19T14:13:05.607783+0000","last_unstale":"2024-08-19T14:13:32.747597+0000","last_undegraded":"2024-08-19T14:13:32.747597+0000","last_fullsized":"2024-08-19T14:13:32.747597+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:13:06.731936+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:13:06.731936+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:14:23.252293+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"135'5087","reported_seq":5163,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-19T14:13:35.723028+0000","last_change":"2024-08-19T14:12:43.039198+0000","last_active":"2024-08-19T14:13:35.723028+0000","last_peered":"2024-08-19T14:13:35.723028+0000","last_clean":"2024-08-19T14:13:35.723028+0000","last_became_active":"2024-08-19T14:12:43.038833+0000","last_became_peered":"2024-08-19T14:12:43.038833+0000","last_unstale":"2024-08-19T14:13:35.723028+0000","last_undegraded":"2024-08-19T14:13:35.723028+0000","last_fullsized":"2024-08-19T14:13:35.723028+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:12:41.713780+0000","objects_scrubbed":0,"log_size":5087,"log_dups_size":0,"ondisk_log_size":5087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:13:59.308042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":333381632,"num_objects":5087,"num_object_clones":0,"num_object_copies":15261,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5087,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5087,"num_write_kb":325568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"70'139","reported_seq":314,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-08-19T14:13:32.749326+0000","last_change":"2024-08-19T14:13:11.663591+0000","last_active":"2024-08-19T14:13:32.749326+0000","last_peered":"2024-08-19T14:13:32.749326+0000","last_clean":"2024-08-19T14:13:32.749326+0000","last_became_active":"2024-08-19T14:13:11.663185+0000","last_became_peered":"2024-08-19T14:13:11.663185+0000","last_unstale":"2024-08-19T14:13:32.749326+0000","last_undegraded":"2024-08-19T14:13:32.749326+0000","last_fullsized":"2024-08-19T14:13:32.749326+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:13:05.768246+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:13:05.768246+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:13:33.301461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":17,"num_write_kb":268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":192338,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2],"acting":[9,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.2","version":"131'4682","reported_seq":6234,"reported_epoch":131,"state":"active+recovering+undersized+remapped","last_fresh":"2024-08-19T14:13:31.452985+0000","last_change":"2024-08-19T14:13:20.833583+0000","last_active":"2024-08-19T14:13:31.452985+0000","last_peered":"2024-08-19T14:13:31.452985+0000","last_clean":"2024-08-19T14:13:18.714425+0000","last_became_active":"2024-08-19T14:13:20.811055+0000","last_became_peered":"2024-08-19T14:13:20.811055+0000","last_unstale":"2024-08-19T14:13:31.452985+0000","last_undegraded":"2024-08-19T14:13:31.452985+0000","last_fullsized":"2024-08-19T14:13:20.718295+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:12:41.713780+0000","objects_scrubbed":0,"log_size":4682,"log_dups_size":0,"ondisk_log_size":4682,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":306839552,"num_objects":4682,"num_object_clones":0,"num_object_copies":14046,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3162,"num_objects_unfound":0,"num_objects_dirty":4682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4682,"num_write_kb":299648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":740,"num_bytes_recovered":48496640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1],"avail_no_missing":["5","1","6"],"object_location_counts":[{"shards":"1,5","objects":1520},{"shards":"1,5,6","objects":3162}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"70'139","reported_seq":299,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-08-19T14:13:32.748507+0000","last_change":"2024-08-19T14:13:22.739488+0000","last_active":"2024-08-19T14:13:32.748507+0000","last_peered":"2024-08-19T14:13:32.748507+0000","last_clean":"2024-08-19T14:13:32.748507+0000","last_became_active":"2024-08-19T14:13:22.739150+0000","last_became_peered":"2024-08-19T14:13:22.739150+0000","last_unstale":"2024-08-19T14:13:32.748507+0000","last_undegraded":"2024-08-19T14:13:32.748507+0000","last_fullsized":"2024-08-19T14:13:32.748507+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:13:06.062715+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:13:06.062715+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:13:33.017828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":16,"num_write_kb":267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":192337,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"132'4816","reported_seq":7421,"reported_epoch":132,"state":"active+recovering+undersized+remapped","last_fresh":"2024-08-19T14:13:32.663119+0000","last_change":"2024-08-19T14:13:09.750288+0000","last_active":"2024-08-19T14:13:32.663119+0000","last_peered":"2024-08-19T14:13:32.663119+0000","last_clean":"2024-08-19T14:13:07.624729+0000","last_became_active":"2024-08-19T14:13:09.716419+0000","last_became_peered":"2024-08-19T14:13:09.716419+0000","last_unstale":"2024-08-19T14:13:32.663119+0000","last_undegraded":"2024-08-19T14:13:32.663119+0000","last_fullsized":"2024-08-19T14:13:09.628821+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:12:41.713780+0000","objects_scrubbed":0,"log_size":4816,"log_dups_size":0,"ondisk_log_size":4816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":315621376,"num_objects":4816,"num_object_clones":0,"num_object_copies":14448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1856,"num_objects_unfound":0,"num_objects_dirty":4816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4816,"num_write_kb":308224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1266,"num_bytes_recovered":82968576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0],"avail_no_missing":["1","0","10"],"object_location_counts":[{"shards":"0,1","objects":2960},{"shards":"0,1,10","objects":1856}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"70'139","reported_seq":314,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-08-19T14:13:32.749474+0000","last_change":"2024-08-19T14:13:11.647683+0000","last_active":"2024-08-19T14:13:32.749474+0000","last_peered":"2024-08-19T14:13:32.749474+0000","last_clean":"2024-08-19T14:13:32.749474+0000","last_became_active":"2024-08-19T14:13:11.647085+0000","last_became_peered":"2024-08-19T14:13:11.647085+0000","last_unstale":"2024-08-19T14:13:32.749474+0000","last_undegraded":"2024-08-19T14:13:32.749474+0000","last_fullsized":"2024-08-19T14:13:32.749474+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:13:06.776621+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:13:06.776621+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:13:33.301447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":16,"num_write_kb":267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":192337,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,4],"acting":[9,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.5","version":"134'4956","reported_seq":5030,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-08-19T14:13:34.621690+0000","last_change":"2024-08-19T14:12:43.418168+0000","last_active":"2024-08-19T14:13:34.621690+0000","last_peered":"2024-08-19T14:13:34.621690+0000","last_clean":"2024-08-19T14:13:34.621690+0000","last_became_active":"2024-08-19T14:12:43.417616+0000","last_became_peered":"2024-08-19T14:12:43.417616+0000","last_unstale":"2024-08-19T14:13:34.621690+0000","last_undegraded":"2024-08-19T14:13:34.621690+0000","last_fullsized":"2024-08-19T14:13:34.621690+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:12:41.713780+0000","objects_scrubbed":0,"log_size":4956,"log_dups_size":0,"ondisk_log_size":4956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:13:59.305980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":324796416,"num_objects":4956,"num_object_clones":0,"num_object_copies":14868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4956,"num_write_kb":317184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.5","version":"70'139","reported_seq":327,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-19T14:13:34.769998+0000","last_change":"2024-08-19T14:13:22.727452+0000","last_active":"2024-08-19T14:13:34.769998+0000","last_peered":"2024-08-19T14:13:34.769998+0000","last_clean":"2024-08-19T14:13:34.769998+0000","last_became_active":"2024-08-19T14:13:22.727098+0000","last_became_peered":"2024-08-19T14:13:22.727098+0000","last_unstale":"2024-08-19T14:13:34.769998+0000","last_undegraded":"2024-08-19T14:13:34.769998+0000","last_fullsized":"2024-08-19T14:13:34.769998+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:12:04.853650+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:12:04.853650+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:13:39.502830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":17,"num_write_kb":267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":192338,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"132'4702","reported_seq":4772,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-08-19T14:13:32.399834+0000","last_change":"2024-08-19T14:12:42.963821+0000","last_active":"2024-08-19T14:13:32.399834+0000","last_peered":"2024-08-19T14:13:32.399834+0000","last_clean":"2024-08-19T14:13:32.399834+0000","last_became_active":"2024-08-19T14:12:42.963203+0000","last_became_peered":"2024-08-19T14:12:42.963203+0000","last_unstale":"2024-08-19T14:13:32.399834+0000","last_undegraded":"2024-08-19T14:13:32.399834+0000","last_fullsized":"2024-08-19T14:13:32.399834+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:12:41.713780+0000","objects_scrubbed":0,"log_size":4702,"log_dups_size":0,"ondisk_log_size":4702,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:14:01.457016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":308150272,"num_objects":4702,"num_object_clones":0,"num_object_copies":14106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4702,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4702,"num_write_kb":300928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,3],"acting":[1,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"70'139","reported_seq":377,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-19T14:13:34.766331+0000","last_change":"2024-08-19T14:13:06.506271+0000","last_active":"2024-08-19T14:13:34.766331+0000","last_peered":"2024-08-19T14:13:34.766331+0000","last_clean":"2024-08-19T14:13:34.766331+0000","last_became_active":"2024-08-19T14:13:05.607447+0000","last_became_peered":"2024-08-19T14:13:05.607447+0000","last_unstale":"2024-08-19T14:13:34.766331+0000","last_undegraded":"2024-08-19T14:13:34.766331+0000","last_fullsized":"2024-08-19T14:13:34.766331+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:13:06.506192+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:13:06.506192+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:14:34.444601+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.7","version":"135'4918","reported_seq":7447,"reported_epoch":135,"state":"active+recovering+undersized+remapped","last_fresh":"2024-08-19T14:13:35.708639+0000","last_change":"2024-08-19T14:13:20.913688+0000","last_active":"2024-08-19T14:13:35.708639+0000","last_peered":"2024-08-19T14:13:35.708639+0000","last_clean":"2024-08-19T14:13:07.604184+0000","last_became_active":"2024-08-19T14:13:20.813460+0000","last_became_peered":"2024-08-19T14:13:20.813460+0000","last_unstale":"2024-08-19T14:13:35.708639+0000","last_undegraded":"2024-08-19T14:13:35.708639+0000","last_fullsized":"2024-08-19T14:13:20.722594+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:12:41.713780+0000","objects_scrubbed":0,"log_size":4918,"log_dups_size":0,"ondisk_log_size":4918,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":322306048,"num_objects":4918,"num_object_clones":0,"num_object_copies":14754,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5435,"num_objects_unfound":0,"num_objects_dirty":4918,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4918,"num_write_kb":314752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1023,"num_bytes_recovered":67043328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3],"avail_no_missing":["2","6","10"],"object_location_counts":[{"shards":"2,3","objects":1481},{"shards":"2,3,6","objects":816},{"shards":"2,3,6,10","objects":623},{"shards":"2,6,10","objects":1998}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.8","version":"133'4850","reported_seq":4922,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-08-19T14:13:32.920453+0000","last_change":"2024-08-19T14:12:43.039135+0000","last_active":"2024-08-19T14:13:32.920453+0000","last_peered":"2024-08-19T14:13:32.920453+0000","last_clean":"2024-08-19T14:13:32.920453+0000","last_became_active":"2024-08-19T14:12:43.038882+0000","last_became_peered":"2024-08-19T14:12:43.038882+0000","last_unstale":"2024-08-19T14:13:32.920453+0000","last_undegraded":"2024-08-19T14:13:32.920453+0000","last_fullsized":"2024-08-19T14:13:32.920453+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:12:41.713780+0000","objects_scrubbed":0,"log_size":4850,"log_dups_size":0,"ondisk_log_size":4850,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:13:55.854839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":317849600,"num_objects":4850,"num_object_clones":0,"num_object_copies":14550,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4850,"num_write_kb":310400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,0],"acting":[7,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"70'139","reported_seq":335,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-08-19T14:13:32.749406+0000","last_change":"2024-08-19T14:13:22.729315+0000","last_active":"2024-08-19T14:13:32.749406+0000","last_peered":"2024-08-19T14:13:32.749406+0000","last_clean":"2024-08-19T14:13:32.749406+0000","last_became_active":"2024-08-19T14:13:22.728779+0000","last_became_peered":"2024-08-19T14:13:22.728779+0000","last_unstale":"2024-08-19T14:13:32.749406+0000","last_undegraded":"2024-08-19T14:13:32.749406+0000","last_fullsized":"2024-08-19T14:13:32.749406+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:13:06.214307+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:13:06.214307+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:14:21.591348+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,0],"acting":[9,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"132'4731","reported_seq":4801,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-08-19T14:13:32.368411+0000","last_change":"2024-08-19T14:12:42.963504+0000","last_active":"2024-08-19T14:13:32.368411+0000","last_peered":"2024-08-19T14:13:32.368411+0000","last_clean":"2024-08-19T14:13:32.368411+0000","last_became_active":"2024-08-19T14:12:42.963114+0000","last_became_peered":"2024-08-19T14:12:42.963114+0000","last_unstale":"2024-08-19T14:13:32.368411+0000","last_undegraded":"2024-08-19T14:13:32.368411+0000","last_fullsized":"2024-08-19T14:13:32.368411+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:12:41.713780+0000","objects_scrubbed":0,"log_size":4731,"log_dups_size":0,"ondisk_log_size":4731,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:14:03.432018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":310050816,"num_objects":4731,"num_object_clones":0,"num_object_copies":14193,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4731,"num_write_kb":302784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"70'139","reported_seq":448,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-08-19T14:13:32.747625+0000","last_change":"2024-08-19T14:13:05.755102+0000","last_active":"2024-08-19T14:13:32.747625+0000","last_peered":"2024-08-19T14:13:32.747625+0000","last_clean":"2024-08-19T14:13:32.747625+0000","last_became_active":"2024-08-19T14:13:05.607605+0000","last_became_peered":"2024-08-19T14:13:05.607605+0000","last_unstale":"2024-08-19T14:13:32.747625+0000","last_undegraded":"2024-08-19T14:13:32.747625+0000","last_fullsized":"2024-08-19T14:13:32.747625+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:13:05.755019+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:13:05.755019+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:14:26.013388+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"135'4983","reported_seq":5059,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-19T14:13:35.504423+0000","last_change":"2024-08-19T14:12:42.963510+0000","last_active":"2024-08-19T14:13:35.504423+0000","last_peered":"2024-08-19T14:13:35.504423+0000","last_clean":"2024-08-19T14:13:35.504423+0000","last_became_active":"2024-08-19T14:12:42.963040+0000","last_became_peered":"2024-08-19T14:12:42.963040+0000","last_unstale":"2024-08-19T14:13:35.504423+0000","last_undegraded":"2024-08-19T14:13:35.504423+0000","last_fullsized":"2024-08-19T14:13:35.504423+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:12:41.713780+0000","objects_scrubbed":0,"log_size":4983,"log_dups_size":0,"ondisk_log_size":4983,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:14:08.585038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":326565888,"num_objects":4983,"num_object_clones":0,"num_object_copies":14949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4983,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4983,"num_write_kb":318912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,9],"acting":[11,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.8","version":"70'139","reported_seq":322,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-08-19T14:13:27.736226+0000","last_change":"2024-08-19T14:13:06.144886+0000","last_active":"2024-08-19T14:13:27.736226+0000","last_peered":"2024-08-19T14:13:27.736226+0000","last_clean":"2024-08-19T14:13:27.736226+0000","last_became_active":"2024-08-19T14:13:05.605694+0000","last_became_peered":"2024-08-19T14:13:05.605694+0000","last_unstale":"2024-08-19T14:13:27.736226+0000","last_undegraded":"2024-08-19T14:13:27.736226+0000","last_fullsized":"2024-08-19T14:13:27.736226+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:13:06.144774+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:13:06.144774+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:14:27.136488+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.b","version":"134'4906","reported_seq":4980,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-08-19T14:13:34.601339+0000","last_change":"2024-08-19T14:12:42.729556+0000","last_active":"2024-08-19T14:13:34.601339+0000","last_peered":"2024-08-19T14:13:34.601339+0000","last_clean":"2024-08-19T14:13:34.601339+0000","last_became_active":"2024-08-19T14:12:42.729014+0000","last_became_peered":"2024-08-19T14:12:42.729014+0000","last_unstale":"2024-08-19T14:13:34.601339+0000","last_undegraded":"2024-08-19T14:13:34.601339+0000","last_fullsized":"2024-08-19T14:13:34.601339+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:12:41.713780+0000","objects_scrubbed":0,"log_size":4906,"log_dups_size":0,"ondisk_log_size":4906,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:13:51.929231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":321519616,"num_objects":4906,"num_object_clones":0,"num_object_copies":14718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4906,"num_write_kb":313984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":5109907456,"num_objects":77971,"num_object_clones":0,"num_object_copies":233913,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10453,"num_objects_unfound":0,"num_objects_dirty":77971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":77971,"num_write_kb":4990144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6053,"num_bytes_recovered":396689408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15749873664,"data_stored":15749873664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77971,"ondisk_log_size":77971,"up":48,"acting":45,"num_store_stats":12},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":805048,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":46,"num_read_kb":40,"num_write":66,"num_write_kb":1069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":769350,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1529,"ondisk_log_size":1529,"up":33,"acting":33,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187483,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1304196,"kb_used_data":1259128,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":92440956,"statfs":{"total":95995035648,"available":94659538944,"internally_reserved":0,"allocated":1289347072,"data_stored":1288647916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":46135748},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":343597383711,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":74,"seq":317827579940,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2021352,"kb_used_data":1945760,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":91723800,"statfs":{"total":95995035648,"available":93925171200,"internally_reserved":0,"allocated":1992458240,"data_stored":1991767524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":77396417},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874282,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1304524,"kb_used_data":1246968,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":92440628,"statfs":{"total":95995035648,"available":94659203072,"internally_reserved":0,"allocated":1276895232,"data_stored":1276196076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":57,"seq":244813135916,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1575532,"kb_used_data":1530464,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":92169620,"statfs":{"total":95995035648,"available":94381690880,"internally_reserved":0,"allocated":1567195136,"data_stored":1566504420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":46135754},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":206158430256,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691892,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":639076,"kb_used_data":607900,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93106076,"statfs":{"total":95995035648,"available":95340621824,"internally_reserved":0,"allocated":622489600,"data_stored":621799552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986233,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1061288,"kb_used_data":1003744,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":92683864,"statfs":{"total":95995035648,"available":94908276736,"internally_reserved":0,"allocated":1027833856,"data_stored":1027143140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149759,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2086976,"kb_used_data":2011380,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":91658176,"statfs":{"total":95995035648,"available":93857972224,"internally_reserved":0,"allocated":2059653120,"data_stored":2058954611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77396423},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411396,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1551572,"kb_used_data":1488460,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":92193580,"statfs":{"total":95995035648,"available":94406225920,"internally_reserved":0,"allocated":1524183040,"data_stored":1523476133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574920,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1565544,"kb_used_data":1516320,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":92179608,"statfs":{"total":95995035648,"available":94391918592,"internally_reserved":0,"allocated":1552711680,"data_stored":1552020964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738445,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1643276,"kb_used_data":1585720,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":92101876,"statfs":{"total":95995035648,"available":94312321024,"internally_reserved":0,"allocated":1623777280,"data_stored":1623078124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1644888064,"data_stored":1644888064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1556217856,"data_stored":1556217856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1530134528,"data_stored":1530134528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2083389440,"data_stored":2083389440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1028063232,"data_stored":1028063232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":621346816,"data_stored":621346816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":507314176,"data_stored":507314176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1571946496,"data_stored":1571946496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1287127040,"data_stored":1287127040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":2013724672,"data_stored":2013724672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":600899584,"data_stored":600899584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1304821760,"data_stored":1304821760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T14:13:38.307 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd pool set unique_pool_0 pgp_num 16 2024-08-19T14:13:38.341 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:13:38.369 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:38.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:38.370 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:13:38.471 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:13:38.516 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:38.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:38.517 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:13:38.618 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:13:38.672 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:38.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:38.673 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:13:38.756 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:13:38.774 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:13:38.802 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:38.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:38.803 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:13:38.808 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:13:38.814 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:38 smithi032 ceph-mon[20004]: pgmap v347: 27 pgs: 3 active+recovering+undersized+remapped, 24 active+clean; 4.9 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 62 MiB/s wr, 998 op/s; 9723/238926 objects misplaced (4.069%); 21 MiB/s, 341 objects/s recovering 2024-08-19T14:13:38.819 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-08-19T14:13:38.834 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 49 16 83644 83628 106.659 72.6875 0.00331385 0.00936992 2024-08-19T14:13:38.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:38 smithi031 ceph-mon[16336]: pgmap v347: 27 pgs: 3 active+recovering+undersized+remapped, 24 active+clean; 4.9 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 62 MiB/s wr, 998 op/s; 9723/238926 objects misplaced (4.069%); 21 MiB/s, 341 objects/s recovering 2024-08-19T14:13:38.903 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:13:38.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:38.936 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:38.936 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:13:39.037 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:13:39.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:38 smithi045 ceph-mon[20143]: pgmap v347: 27 pgs: 3 active+recovering+undersized+remapped, 24 active+clean; 4.9 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 62 MiB/s wr, 998 op/s; 9723/238926 objects misplaced (4.069%); 21 MiB/s, 341 objects/s recovering 2024-08-19T14:13:39.065 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:39.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:39.066 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:13:39.167 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:13:39.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:39.188 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:39.188 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:13:39.190 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:13:39.289 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:13:39.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:39.317 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:39.317 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:13:39.418 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:13:39.441 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:39.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:39.443 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:13:39.543 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:13:39.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:39.591 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:39.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:13:39.692 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:13:39.738 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:39.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:39.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:13:39.834 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 50 15 84582 84567 105.7 58.6875 0.0362323 0.00945499 2024-08-19T14:13:39.836 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:13:39.836 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-08-19T14:13:39.837 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-08-19T14:13:39.837 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-08-19T14:13:39.837 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:13:39.840 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:13:39.865 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:39.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:39.866 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:13:39.967 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:13:40.012 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:40.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:40.013 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:13:40.114 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:13:40.139 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:40.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:40.140 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:13:40.241 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:13:40.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:40.270 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:40.270 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:13:40.371 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:13:40.405 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:40.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:40.406 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:13:40.507 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:13:40.547 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:40.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:40.548 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:13:40.616 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:13:40.649 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:13:40.675 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:40.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:40.676 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:13:40.775 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-08-19T14:13:40.777 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:13:40.801 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is unset 2024-08-19T14:13:40.805 INFO:teuthology.orchestra.run.smithi031.stderr:set pool 2 pgp_num to 16 2024-08-19T14:13:40.811 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:40.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:40.812 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:13:40.834 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 51 16 85834 85818 105.16 78.1875 0.0159025 0.00950237 2024-08-19T14:13:40.913 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:13:40.968 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:40.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:40.969 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:13:41.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:40 smithi032 ceph-mon[20004]: pgmap v348: 27 pgs: 3 active+recovering+undersized+remapped, 24 active+clean; 5.0 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 65 MiB/s wr, 1.04k op/s; 9723/245598 objects misplaced (3.959%); 15 MiB/s, 246 objects/s recovering 2024-08-19T14:13:41.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:40 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1514369424' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:13:41.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:40 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:13:41.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:40 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3105680242' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"}]: dispatch 2024-08-19T14:13:41.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:40 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"}]: dispatch 2024-08-19T14:13:41.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:40 smithi045 ceph-mon[20143]: pgmap v348: 27 pgs: 3 active+recovering+undersized+remapped, 24 active+clean; 5.0 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 65 MiB/s wr, 1.04k op/s; 9723/245598 objects misplaced (3.959%); 15 MiB/s, 246 objects/s recovering 2024-08-19T14:13:41.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:40 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1514369424' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:13:41.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:40 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:13:41.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:40 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3105680242' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"}]: dispatch 2024-08-19T14:13:41.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:40 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"}]: dispatch 2024-08-19T14:13:41.070 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:13:41.097 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:41.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:41.099 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:13:41.101 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:13:41.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:40 smithi031 ceph-mon[16336]: pgmap v348: 27 pgs: 3 active+recovering+undersized+remapped, 24 active+clean; 5.0 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 65 MiB/s wr, 1.04k op/s; 9723/245598 objects misplaced (3.959%); 15 MiB/s, 246 objects/s recovering 2024-08-19T14:13:41.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:40 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1514369424' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:13:41.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:40 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:13:41.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:40 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3105680242' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"}]: dispatch 2024-08-19T14:13:41.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:40 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"}]: dispatch 2024-08-19T14:13:41.199 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:13:41.221 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:41.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:41.222 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:13:41.323 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:13:41.359 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:41.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:41.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:13:41.461 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:13:41.482 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:41.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:41.483 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:13:41.584 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:13:41.611 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:41.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:41.612 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:13:41.713 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:13:41.743 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:41.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:41.744 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:13:41.756 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:13:41.756 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-08-19T14:13:41.756 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-08-19T14:13:41.756 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-08-19T14:13:41.756 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.756 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:8241 2.7 2:edf47d3e:::benchmark_data_smithi045_46415_object8240:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-08-19T14:13:41.756 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:12:51.684165+0000", 2024-08-19T14:13:41.756 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 50.068588841, 2024-08-19T14:13:41.756 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.204747761, 2024-08-19T14:13:41.757 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:13:41.757 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:13:41.757 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:13:41.757 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:13:41.757 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:13:41.757 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:13:41.757 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 8241 2024-08-19T14:13:41.757 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.757 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:13:41.757 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.757 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:13:41.757 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.684165+0000", 2024-08-19T14:13:41.757 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:41.757 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.758 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.758 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:13:41.758 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.684165+0000", 2024-08-19T14:13:41.758 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:41.758 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.758 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.758 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:13:41.758 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.684167+0000", 2024-08-19T14:13:41.758 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.313e-06 2024-08-19T14:13:41.758 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.758 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.758 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:13:41.758 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.684210+0000", 2024-08-19T14:13:41.759 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3049000000000001e-05 2024-08-19T14:13:41.759 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.759 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.759 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:13:41.759 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.684223+0000", 2024-08-19T14:13:41.759 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3019e-05 2024-08-19T14:13:41.759 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.759 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.759 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:13:41.759 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.684230+0000", 2024-08-19T14:13:41.759 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1969999999999999e-06 2024-08-19T14:13:41.759 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.759 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.760 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:13:41.760 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.684270+0000", 2024-08-19T14:13:41.760 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0649999999999999e-05 2024-08-19T14:13:41.760 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.760 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.760 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:13:41.760 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.684340+0000", 2024-08-19T14:13:41.760 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9979000000000002e-05 2024-08-19T14:13:41.760 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.760 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.760 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,10", 2024-08-19T14:13:41.760 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.684464+0000", 2024-08-19T14:13:41.760 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012321599999999999 2024-08-19T14:13:41.760 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.760 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.761 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:13:41.761 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.684888+0000", 2024-08-19T14:13:41.761 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00042467399999999999 2024-08-19T14:13:41.761 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.761 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.761 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:13:41.761 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.685794+0000", 2024-08-19T14:13:41.761 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00090609299999999996 2024-08-19T14:13:41.761 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.761 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.761 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:13:41.761 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.888736+0000", 2024-08-19T14:13:41.761 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20294189100000001 2024-08-19T14:13:41.761 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.762 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.762 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:13:41.762 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.888831+0000", 2024-08-19T14:13:41.762 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.5172000000000003e-05 2024-08-19T14:13:41.762 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.762 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.762 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:13:41.762 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.888913+0000", 2024-08-19T14:13:41.762 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1508000000000006e-05 2024-08-19T14:13:41.762 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:41.762 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:13:41.762 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:41.762 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.763 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.763 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:8259 2.7 2:e2e1720b:::benchmark_data_smithi045_46415_object8258:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-08-19T14:13:41.763 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:12:51.689217+0000", 2024-08-19T14:13:41.763 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 50.063536972000001, 2024-08-19T14:13:41.763 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.19987796899999999, 2024-08-19T14:13:41.763 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:13:41.763 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:13:41.763 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:13:41.763 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:13:41.763 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:13:41.763 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:13:41.763 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 8259 2024-08-19T14:13:41.764 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.764 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:13:41.764 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.764 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:13:41.764 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.689217+0000", 2024-08-19T14:13:41.764 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:41.764 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.764 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.764 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:13:41.764 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.689217+0000", 2024-08-19T14:13:41.764 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:41.764 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.764 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.765 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:13:41.765 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.689218+0000", 2024-08-19T14:13:41.765 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0240000000000001e-06 2024-08-19T14:13:41.765 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.765 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.765 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:13:41.765 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.689272+0000", 2024-08-19T14:13:41.765 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3974999999999997e-05 2024-08-19T14:13:41.765 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.765 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.765 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:13:41.765 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.689283+0000", 2024-08-19T14:13:41.765 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0647e-05 2024-08-19T14:13:41.766 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.766 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.766 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:13:41.766 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.689288+0000", 2024-08-19T14:13:41.766 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2390000000000002e-06 2024-08-19T14:13:41.766 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.766 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.766 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:13:41.766 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.689341+0000", 2024-08-19T14:13:41.766 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.342e-05 2024-08-19T14:13:41.766 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.766 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.766 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:13:41.767 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.689464+0000", 2024-08-19T14:13:41.767 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000122991 2024-08-19T14:13:41.767 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.767 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.767 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,10", 2024-08-19T14:13:41.767 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.689610+0000", 2024-08-19T14:13:41.767 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014557200000000001 2024-08-19T14:13:41.767 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.767 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.767 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:13:41.767 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.690297+0000", 2024-08-19T14:13:41.767 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00068681199999999997 2024-08-19T14:13:41.767 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.767 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.768 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:13:41.768 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.690649+0000", 2024-08-19T14:13:41.768 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00035261899999999998 2024-08-19T14:13:41.768 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.768 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.768 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:13:41.768 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.888991+0000", 2024-08-19T14:13:41.768 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.19834179800000001 2024-08-19T14:13:41.768 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.768 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.768 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:13:41.768 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.889045+0000", 2024-08-19T14:13:41.768 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3643000000000001e-05 2024-08-19T14:13:41.768 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.769 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.769 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:13:41.769 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.889095+0000", 2024-08-19T14:13:41.769 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0229000000000003e-05 2024-08-19T14:13:41.769 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:41.769 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:13:41.769 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:41.769 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.769 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.769 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:8272 2.7 2:ec6eef80:::benchmark_data_smithi045_46415_object8271:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-08-19T14:13:41.769 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:12:51.692591+0000", 2024-08-19T14:13:41.769 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 50.060163148999997, 2024-08-19T14:13:41.769 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.19659718500000001, 2024-08-19T14:13:41.770 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:13:41.770 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:13:41.770 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:13:41.770 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:13:41.771 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:13:41.771 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:13:41.771 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 8272 2024-08-19T14:13:41.771 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.771 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:13:41.771 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.771 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:13:41.771 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.692591+0000", 2024-08-19T14:13:41.771 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:41.771 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.771 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.771 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:13:41.771 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.692591+0000", 2024-08-19T14:13:41.772 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:41.772 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.772 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.772 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:13:41.772 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.692592+0000", 2024-08-19T14:13:41.772 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.2800000000000005e-07 2024-08-19T14:13:41.772 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.772 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.772 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:13:41.772 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.692661+0000", 2024-08-19T14:13:41.772 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8744000000000005e-05 2024-08-19T14:13:41.772 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.772 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.772 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:13:41.773 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.692675+0000", 2024-08-19T14:13:41.773 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3916e-05 2024-08-19T14:13:41.773 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.773 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.773 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:13:41.773 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.692682+0000", 2024-08-19T14:13:41.773 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5320000000000001e-06 2024-08-19T14:13:41.773 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.773 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.773 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:13:41.773 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.692731+0000", 2024-08-19T14:13:41.773 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8909999999999998e-05 2024-08-19T14:13:41.773 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.774 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.774 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:13:41.774 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.692805+0000", 2024-08-19T14:13:41.774 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4128000000000006e-05 2024-08-19T14:13:41.774 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.774 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.774 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,10", 2024-08-19T14:13:41.774 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.692885+0000", 2024-08-19T14:13:41.774 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.9655999999999998e-05 2024-08-19T14:13:41.774 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.774 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.774 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:13:41.774 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.693416+0000", 2024-08-19T14:13:41.774 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00053101099999999996 2024-08-19T14:13:41.775 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.775 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.775 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:13:41.775 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.694542+0000", 2024-08-19T14:13:41.775 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0011263810000000001 2024-08-19T14:13:41.775 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.775 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.775 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:13:41.775 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.889136+0000", 2024-08-19T14:13:41.775 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.194593922 2024-08-19T14:13:41.775 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.775 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.775 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:13:41.776 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.889164+0000", 2024-08-19T14:13:41.776 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8113999999999999e-05 2024-08-19T14:13:41.776 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.776 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.776 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:13:41.776 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.889188+0000", 2024-08-19T14:13:41.776 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3943000000000001e-05 2024-08-19T14:13:41.776 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:41.776 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:13:41.776 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:41.776 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.776 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.776 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:8281 2.7 2:edaf96ca:::benchmark_data_smithi045_46415_object8280:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-08-19T14:13:41.776 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:12:51.695230+0000", 2024-08-19T14:13:41.777 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 50.057524499000003, 2024-08-19T14:13:41.777 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.194082428, 2024-08-19T14:13:41.777 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:13:41.777 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:13:41.777 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:13:41.777 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:13:41.777 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:13:41.777 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:13:41.777 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 8281 2024-08-19T14:13:41.777 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.777 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:13:41.777 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.777 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:13:41.778 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.695230+0000", 2024-08-19T14:13:41.778 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:41.778 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.778 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.778 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:13:41.778 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.695230+0000", 2024-08-19T14:13:41.778 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:41.778 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.778 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.778 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:13:41.778 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.695231+0000", 2024-08-19T14:13:41.778 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0249999999999999e-06 2024-08-19T14:13:41.778 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.779 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.779 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:13:41.779 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.695295+0000", 2024-08-19T14:13:41.779 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4115999999999994e-05 2024-08-19T14:13:41.779 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.779 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.779 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:13:41.779 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.695309+0000", 2024-08-19T14:13:41.779 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.383e-05 2024-08-19T14:13:41.779 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.779 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.779 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:13:41.779 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.695317+0000", 2024-08-19T14:13:41.780 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.0490000000000004e-06 2024-08-19T14:13:41.780 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.780 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.780 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:13:41.780 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.695368+0000", 2024-08-19T14:13:41.780 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1819e-05 2024-08-19T14:13:41.780 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.780 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.780 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:13:41.780 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.695445+0000", 2024-08-19T14:13:41.780 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6705999999999994e-05 2024-08-19T14:13:41.780 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.780 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.780 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,10", 2024-08-19T14:13:41.781 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.695523+0000", 2024-08-19T14:13:41.781 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7989000000000002e-05 2024-08-19T14:13:41.781 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.781 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.781 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:13:41.781 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.696398+0000", 2024-08-19T14:13:41.781 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00087526300000000004 2024-08-19T14:13:41.781 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.781 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.781 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:13:41.781 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.696820+0000", 2024-08-19T14:13:41.781 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00042155799999999999 2024-08-19T14:13:41.781 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.782 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.782 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:13:41.782 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.889226+0000", 2024-08-19T14:13:41.782 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.19240576600000001 2024-08-19T14:13:41.782 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.782 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.782 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:13:41.782 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.889276+0000", 2024-08-19T14:13:41.782 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0389000000000002e-05 2024-08-19T14:13:41.782 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.782 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.782 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:13:41.782 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.889312+0000", 2024-08-19T14:13:41.783 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5917999999999997e-05 2024-08-19T14:13:41.783 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:41.783 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:13:41.783 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:41.783 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.783 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.783 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:8284 2.7 2:eea07cd5:::benchmark_data_smithi045_46415_object8283:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-08-19T14:13:41.783 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:12:51.695765+0000", 2024-08-19T14:13:41.783 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 50.056988978, 2024-08-19T14:13:41.783 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.19395319999999999, 2024-08-19T14:13:41.783 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:13:41.783 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:13:41.783 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:13:41.783 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:13:41.784 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:13:41.784 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:13:41.784 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 8284 2024-08-19T14:13:41.784 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.784 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:13:41.784 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.784 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:13:41.784 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.695765+0000", 2024-08-19T14:13:41.784 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:41.784 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.784 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.784 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:13:41.784 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.695765+0000", 2024-08-19T14:13:41.785 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:41.785 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.785 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.785 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:13:41.785 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.695766+0000", 2024-08-19T14:13:41.785 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2399999999999997e-07 2024-08-19T14:13:41.785 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.785 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.785 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:13:41.785 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.695825+0000", 2024-08-19T14:13:41.785 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8702000000000001e-05 2024-08-19T14:13:41.785 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.785 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.786 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:13:41.786 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.695845+0000", 2024-08-19T14:13:41.786 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0500999999999999e-05 2024-08-19T14:13:41.786 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.786 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.786 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:13:41.786 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.695850+0000", 2024-08-19T14:13:41.786 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3319999999999999e-06 2024-08-19T14:13:41.786 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.786 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.786 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:13:41.786 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.695932+0000", 2024-08-19T14:13:41.786 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.2089000000000006e-05 2024-08-19T14:13:41.787 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.787 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.787 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:13:41.787 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.696041+0000", 2024-08-19T14:13:41.787 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000108931 2024-08-19T14:13:41.787 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.787 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.787 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,10", 2024-08-19T14:13:41.787 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.696129+0000", 2024-08-19T14:13:41.787 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.7545000000000003e-05 2024-08-19T14:13:41.787 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.787 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.787 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:13:41.788 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.696793+0000", 2024-08-19T14:13:41.788 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000663946 2024-08-19T14:13:41.788 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.788 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.788 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:13:41.788 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.697250+0000", 2024-08-19T14:13:41.788 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000457541 2024-08-19T14:13:41.788 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.788 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.788 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:13:41.788 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.889620+0000", 2024-08-19T14:13:41.788 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.19236932600000001 2024-08-19T14:13:41.788 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.788 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.789 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:13:41.789 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.889680+0000", 2024-08-19T14:13:41.789 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0417000000000001e-05 2024-08-19T14:13:41.789 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.789 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.789 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:13:41.789 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.889718+0000", 2024-08-19T14:13:41.789 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8145999999999997e-05 2024-08-19T14:13:41.789 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:41.789 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:13:41.789 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:41.789 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.789 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.790 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:8285 2.7 2:e4063f99:::benchmark_data_smithi045_46415_object8284:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-08-19T14:13:41.790 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:12:51.695865+0000", 2024-08-19T14:13:41.790 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 50.056889112, 2024-08-19T14:13:41.790 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.19405726300000001, 2024-08-19T14:13:41.790 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:13:41.790 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:13:41.790 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:13:41.790 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:13:41.790 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:13:41.790 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:13:41.790 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 8285 2024-08-19T14:13:41.791 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.791 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:13:41.791 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.791 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:13:41.791 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.695865+0000", 2024-08-19T14:13:41.791 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:41.791 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.791 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.791 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:13:41.791 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.695865+0000", 2024-08-19T14:13:41.791 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:41.791 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.791 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.792 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:13:41.792 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.695866+0000", 2024-08-19T14:13:41.792 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8999999999999996e-07 2024-08-19T14:13:41.792 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.792 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.792 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:13:41.792 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.695901+0000", 2024-08-19T14:13:41.792 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5859999999999999e-05 2024-08-19T14:13:41.792 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.792 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.792 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:13:41.792 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.695913+0000", 2024-08-19T14:13:41.792 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1114000000000001e-05 2024-08-19T14:13:41.792 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.793 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.793 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:13:41.793 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.695920+0000", 2024-08-19T14:13:41.793 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1949999999999997e-06 2024-08-19T14:13:41.793 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.793 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.793 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:13:41.793 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.696419+0000", 2024-08-19T14:13:41.793 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00049909300000000004 2024-08-19T14:13:41.793 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.793 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.793 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:13:41.793 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.696489+0000", 2024-08-19T14:13:41.794 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9915999999999999e-05 2024-08-19T14:13:41.794 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.794 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.794 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,10", 2024-08-19T14:13:41.794 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.696578+0000", 2024-08-19T14:13:41.794 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.8825000000000004e-05 2024-08-19T14:13:41.794 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.794 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.794 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:13:41.794 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.696895+0000", 2024-08-19T14:13:41.794 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000317912 2024-08-19T14:13:41.794 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.794 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.795 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:13:41.795 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.697442+0000", 2024-08-19T14:13:41.795 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00054631000000000005 2024-08-19T14:13:41.795 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.795 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.795 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:13:41.795 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.889765+0000", 2024-08-19T14:13:41.795 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.192323409 2024-08-19T14:13:41.795 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.795 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.795 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:13:41.795 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.889882+0000", 2024-08-19T14:13:41.795 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00011684 2024-08-19T14:13:41.795 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.795 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.796 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:13:41.796 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.889922+0000", 2024-08-19T14:13:41.796 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0198999999999999e-05 2024-08-19T14:13:41.796 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:41.796 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:13:41.796 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:41.796 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.796 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.796 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:8318 2.7 2:e0a01dab:::benchmark_data_smithi045_46415_object8317:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-08-19T14:13:41.796 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:12:51.706539+0000", 2024-08-19T14:13:41.796 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 50.046215271000001, 2024-08-19T14:13:41.796 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.183777518, 2024-08-19T14:13:41.796 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:13:41.797 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:13:41.797 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:13:41.797 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:13:41.797 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:13:41.797 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:13:41.797 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 8318 2024-08-19T14:13:41.797 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.797 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:13:41.797 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.797 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:13:41.797 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.706539+0000", 2024-08-19T14:13:41.797 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:41.797 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.797 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.798 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:13:41.798 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.706539+0000", 2024-08-19T14:13:41.798 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:41.798 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.798 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.798 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:13:41.798 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.706541+0000", 2024-08-19T14:13:41.798 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8050000000000001e-06 2024-08-19T14:13:41.798 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.798 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.798 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:13:41.798 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.706582+0000", 2024-08-19T14:13:41.798 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1748000000000001e-05 2024-08-19T14:13:41.798 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.799 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.799 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:13:41.799 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.706605+0000", 2024-08-19T14:13:41.799 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2614000000000001e-05 2024-08-19T14:13:41.799 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.799 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.799 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:13:41.799 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.706615+0000", 2024-08-19T14:13:41.799 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.825e-06 2024-08-19T14:13:41.799 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.799 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.799 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:13:41.800 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.706681+0000", 2024-08-19T14:13:41.800 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.5729999999999996e-05 2024-08-19T14:13:41.800 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.800 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.800 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:13:41.800 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.706811+0000", 2024-08-19T14:13:41.800 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013086200000000001 2024-08-19T14:13:41.800 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.800 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.800 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,10", 2024-08-19T14:13:41.800 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.706948+0000", 2024-08-19T14:13:41.800 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013628800000000001 2024-08-19T14:13:41.800 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.800 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.801 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:13:41.801 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.707654+0000", 2024-08-19T14:13:41.801 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00070644600000000001 2024-08-19T14:13:41.801 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.801 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.801 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:13:41.801 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.707895+0000", 2024-08-19T14:13:41.801 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00024116599999999999 2024-08-19T14:13:41.801 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.801 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.801 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:13:41.801 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.890219+0000", 2024-08-19T14:13:41.801 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.18232368199999999 2024-08-19T14:13:41.802 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.802 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.802 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:13:41.802 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.890279+0000", 2024-08-19T14:13:41.802 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9941000000000001e-05 2024-08-19T14:13:41.802 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.802 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.802 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:13:41.802 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.890316+0000", 2024-08-19T14:13:41.802 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7410999999999998e-05 2024-08-19T14:13:41.802 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:41.802 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:13:41.802 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:41.803 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.803 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.803 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:48872 2.7 2:e2eb060c:::benchmark_data_smithi045_46415_object48871:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-08-19T14:13:41.803 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:07.265121+0000", 2024-08-19T14:13:41.803 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 34.487633064000001, 2024-08-19T14:13:41.803 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20642575599999999, 2024-08-19T14:13:41.803 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:13:41.803 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:13:41.803 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:13:41.803 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:13:41.803 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:13:41.803 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:13:41.803 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 48872 2024-08-19T14:13:41.803 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.804 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:13:41.804 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.804 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:13:41.804 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.265121+0000", 2024-08-19T14:13:41.804 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:41.804 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.804 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.804 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:13:41.804 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.265121+0000", 2024-08-19T14:13:41.804 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:41.804 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.804 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.804 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:13:41.804 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.265123+0000", 2024-08-19T14:13:41.805 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.494e-06 2024-08-19T14:13:41.805 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.805 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.805 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:13:41.805 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.265156+0000", 2024-08-19T14:13:41.805 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3651e-05 2024-08-19T14:13:41.805 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.805 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.805 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:13:41.805 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.265167+0000", 2024-08-19T14:13:41.805 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0669000000000001e-05 2024-08-19T14:13:41.805 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.805 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.806 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:13:41.806 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.265174+0000", 2024-08-19T14:13:41.806 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8750000000000002e-06 2024-08-19T14:13:41.806 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.806 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.806 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:13:41.806 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.265293+0000", 2024-08-19T14:13:41.806 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000119067 2024-08-19T14:13:41.806 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.806 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.806 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:13:41.806 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.265408+0000", 2024-08-19T14:13:41.807 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000115519 2024-08-19T14:13:41.807 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.807 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.807 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,10", 2024-08-19T14:13:41.807 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.265541+0000", 2024-08-19T14:13:41.807 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013279200000000001 2024-08-19T14:13:41.807 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.807 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.807 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:13:41.807 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.268514+0000", 2024-08-19T14:13:41.807 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0029729259999999999 2024-08-19T14:13:41.807 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.807 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.807 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:13:41.808 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.268576+0000", 2024-08-19T14:13:41.808 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2145999999999998e-05 2024-08-19T14:13:41.808 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.808 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.808 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:13:41.808 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.471473+0000", 2024-08-19T14:13:41.808 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.202897204 2024-08-19T14:13:41.808 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.808 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.808 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:13:41.808 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.471514+0000", 2024-08-19T14:13:41.808 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0484000000000001e-05 2024-08-19T14:13:41.808 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.809 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.809 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:13:41.809 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.471547+0000", 2024-08-19T14:13:41.809 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2929000000000003e-05 2024-08-19T14:13:41.809 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:41.809 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:13:41.809 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:41.809 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.809 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.809 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:48876 2.7 2:e761e839:::benchmark_data_smithi045_46415_object48875:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-08-19T14:13:41.809 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:07.265962+0000", 2024-08-19T14:13:41.809 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 34.486791744999998, 2024-08-19T14:13:41.809 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20685887, 2024-08-19T14:13:41.809 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:13:41.810 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:13:41.810 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:13:41.810 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:13:41.810 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:13:41.810 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:13:41.810 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 48876 2024-08-19T14:13:41.810 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.810 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:13:41.810 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.810 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:13:41.810 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.265962+0000", 2024-08-19T14:13:41.810 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:41.810 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.811 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.811 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:13:41.811 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.265962+0000", 2024-08-19T14:13:41.811 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:41.811 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.811 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.811 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:13:41.811 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.265963+0000", 2024-08-19T14:13:41.811 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0729999999999999e-06 2024-08-19T14:13:41.811 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.811 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.811 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:13:41.811 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.266042+0000", 2024-08-19T14:13:41.811 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.8845999999999997e-05 2024-08-19T14:13:41.812 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.812 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.812 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:13:41.812 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.266060+0000", 2024-08-19T14:13:41.812 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.783e-05 2024-08-19T14:13:41.812 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.812 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.812 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:13:41.812 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.266068+0000", 2024-08-19T14:13:41.812 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7009999999999996e-06 2024-08-19T14:13:41.812 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.812 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.812 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:13:41.812 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.266916+0000", 2024-08-19T14:13:41.813 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00084853200000000004 2024-08-19T14:13:41.813 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.813 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.813 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:13:41.813 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.267096+0000", 2024-08-19T14:13:41.814 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00017937699999999999 2024-08-19T14:13:41.814 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.814 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.814 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,10", 2024-08-19T14:13:41.814 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.267225+0000", 2024-08-19T14:13:41.815 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000129727 2024-08-19T14:13:41.815 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.815 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.815 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:13:41.815 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.268638+0000", 2024-08-19T14:13:41.815 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0014122640000000001 2024-08-19T14:13:41.815 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.815 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.815 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:13:41.815 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.269422+0000", 2024-08-19T14:13:41.815 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00078479600000000004 2024-08-19T14:13:41.815 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.815 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.816 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:13:41.816 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.472763+0000", 2024-08-19T14:13:41.816 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.203340627 2024-08-19T14:13:41.816 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.816 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.816 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:13:41.816 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.472797+0000", 2024-08-19T14:13:41.816 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.4060999999999998e-05 2024-08-19T14:13:41.816 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.816 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.816 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:13:41.816 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.472821+0000", 2024-08-19T14:13:41.816 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4036000000000002e-05 2024-08-19T14:13:41.816 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:41.816 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:13:41.817 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:41.817 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.817 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.817 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:48899 2.7 2:ed417148:::benchmark_data_smithi045_46415_object48898:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-08-19T14:13:41.817 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:07.275772+0000", 2024-08-19T14:13:41.817 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 34.476982006, 2024-08-19T14:13:41.817 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.19712979899999999, 2024-08-19T14:13:41.817 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:13:41.817 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:13:41.817 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:13:41.817 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:13:41.817 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:13:41.817 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:13:41.817 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 48899 2024-08-19T14:13:41.818 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.818 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:13:41.818 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.818 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:13:41.818 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.275772+0000", 2024-08-19T14:13:41.818 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:41.818 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.818 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.818 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:13:41.818 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.275772+0000", 2024-08-19T14:13:41.818 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:41.818 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.818 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.819 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:13:41.819 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.275774+0000", 2024-08-19T14:13:41.819 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5009999999999999e-06 2024-08-19T14:13:41.819 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.819 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.819 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:13:41.819 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.275826+0000", 2024-08-19T14:13:41.819 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2824000000000001e-05 2024-08-19T14:13:41.819 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.819 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.819 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:13:41.819 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.275841+0000", 2024-08-19T14:13:41.819 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4691e-05 2024-08-19T14:13:41.819 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.820 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.820 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:13:41.820 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.275849+0000", 2024-08-19T14:13:41.820 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1580000000000005e-06 2024-08-19T14:13:41.820 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.820 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.820 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:13:41.820 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.275913+0000", 2024-08-19T14:13:41.820 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3590999999999995e-05 2024-08-19T14:13:41.820 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.820 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.820 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:13:41.820 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.276130+0000", 2024-08-19T14:13:41.820 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000216901 2024-08-19T14:13:41.821 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.821 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.821 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,10", 2024-08-19T14:13:41.821 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.276273+0000", 2024-08-19T14:13:41.821 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014370100000000001 2024-08-19T14:13:41.821 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.821 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.821 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:13:41.821 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.278099+0000", 2024-08-19T14:13:41.821 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0018257849999999999 2024-08-19T14:13:41.821 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.821 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.821 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:13:41.822 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.281287+0000", 2024-08-19T14:13:41.822 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0031876769999999999 2024-08-19T14:13:41.822 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.822 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.822 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:13:41.822 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.472854+0000", 2024-08-19T14:13:41.822 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.19156674300000001 2024-08-19T14:13:41.822 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.822 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.822 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:13:41.822 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.472881+0000", 2024-08-19T14:13:41.822 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7521999999999998e-05 2024-08-19T14:13:41.822 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.823 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.823 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:13:41.823 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.472902+0000", 2024-08-19T14:13:41.823 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0704999999999999e-05 2024-08-19T14:13:41.823 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:41.823 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:13:41.823 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:41.823 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.823 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.823 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:48912 2.7 2:e6ffd652:::benchmark_data_smithi045_46415_object48911:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-08-19T14:13:41.823 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:07.279334+0000", 2024-08-19T14:13:41.823 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 34.473420505, 2024-08-19T14:13:41.823 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.195037181, 2024-08-19T14:13:41.824 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:13:41.824 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:13:41.824 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:13:41.824 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:13:41.824 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:13:41.824 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:13:41.824 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 48912 2024-08-19T14:13:41.824 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.824 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:13:41.824 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.824 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:13:41.824 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.279334+0000", 2024-08-19T14:13:41.824 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:41.824 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.825 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.825 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:13:41.825 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.279334+0000", 2024-08-19T14:13:41.825 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:41.825 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.825 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.825 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:13:41.825 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.279335+0000", 2024-08-19T14:13:41.825 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3599999999999999e-06 2024-08-19T14:13:41.825 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.825 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.825 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:13:41.825 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.279409+0000", 2024-08-19T14:13:41.825 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3819000000000006e-05 2024-08-19T14:13:41.826 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.826 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.826 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:13:41.826 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.279420+0000", 2024-08-19T14:13:41.826 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0892e-05 2024-08-19T14:13:41.826 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.826 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.826 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:13:41.826 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.279427+0000", 2024-08-19T14:13:41.826 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2790000000000001e-06 2024-08-19T14:13:41.826 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.826 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.826 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:13:41.827 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.279479+0000", 2024-08-19T14:13:41.827 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2547999999999999e-05 2024-08-19T14:13:41.827 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.827 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.827 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:13:41.827 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.279613+0000", 2024-08-19T14:13:41.827 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013337900000000001 2024-08-19T14:13:41.827 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.827 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.827 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,10", 2024-08-19T14:13:41.827 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.279754+0000", 2024-08-19T14:13:41.827 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000141359 2024-08-19T14:13:41.827 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.827 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.828 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:13:41.828 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.281352+0000", 2024-08-19T14:13:41.828 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0015973070000000001 2024-08-19T14:13:41.828 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.828 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.828 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:13:41.828 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.282744+0000", 2024-08-19T14:13:41.828 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0013921630000000001 2024-08-19T14:13:41.828 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.828 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.828 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:13:41.828 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.474309+0000", 2024-08-19T14:13:41.828 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.19156536800000001 2024-08-19T14:13:41.829 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.829 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.829 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:13:41.829 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.474347+0000", 2024-08-19T14:13:41.829 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.824e-05 2024-08-19T14:13:41.829 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.829 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.829 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:13:41.829 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.474371+0000", 2024-08-19T14:13:41.829 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3467000000000001e-05 2024-08-19T14:13:41.829 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:41.829 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:13:41.829 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:41.829 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.830 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.830 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:48921 2.7 2:e2434a00:::benchmark_data_smithi045_46415_object48920:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-08-19T14:13:41.830 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:07.283150+0000", 2024-08-19T14:13:41.830 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 34.469604271999998, 2024-08-19T14:13:41.830 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.19130543999999999, 2024-08-19T14:13:41.830 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:13:41.830 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:13:41.830 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:13:41.830 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:13:41.830 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:13:41.830 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:13:41.830 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 48921 2024-08-19T14:13:41.830 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.831 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:13:41.831 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.831 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:13:41.831 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.283150+0000", 2024-08-19T14:13:41.831 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:41.831 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.831 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.831 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:13:41.831 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.283150+0000", 2024-08-19T14:13:41.831 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:41.831 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.831 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.831 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:13:41.831 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.283151+0000", 2024-08-19T14:13:41.832 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6330000000000001e-06 2024-08-19T14:13:41.832 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.832 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.832 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:13:41.832 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.283177+0000", 2024-08-19T14:13:41.832 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5497e-05 2024-08-19T14:13:41.832 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.832 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.832 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:13:41.832 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.283190+0000", 2024-08-19T14:13:41.832 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3315e-05 2024-08-19T14:13:41.832 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.832 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.833 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:13:41.833 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.283197+0000", 2024-08-19T14:13:41.833 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7479999999999996e-06 2024-08-19T14:13:41.833 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.833 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.833 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:13:41.833 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.283262+0000", 2024-08-19T14:13:41.833 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4550999999999999e-05 2024-08-19T14:13:41.833 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.833 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.833 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:13:41.833 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.283434+0000", 2024-08-19T14:13:41.833 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00017221700000000001 2024-08-19T14:13:41.833 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.834 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.834 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,10", 2024-08-19T14:13:41.834 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.283553+0000", 2024-08-19T14:13:41.834 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0001197 2024-08-19T14:13:41.834 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.834 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.834 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:13:41.834 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.285150+0000", 2024-08-19T14:13:41.834 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00159615 2024-08-19T14:13:41.834 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.834 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.834 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:13:41.834 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.285949+0000", 2024-08-19T14:13:41.834 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00079935399999999995 2024-08-19T14:13:41.835 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.835 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.835 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:13:41.835 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.474403+0000", 2024-08-19T14:13:41.835 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.18845408199999999 2024-08-19T14:13:41.835 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.835 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.835 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:13:41.835 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.474433+0000", 2024-08-19T14:13:41.835 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9584999999999999e-05 2024-08-19T14:13:41.835 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.835 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.835 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:13:41.836 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.474455+0000", 2024-08-19T14:13:41.836 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2608000000000001e-05 2024-08-19T14:13:41.836 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:41.836 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:13:41.836 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:41.836 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.836 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.836 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:48971 2.7 2:ecf6a539:::benchmark_data_smithi045_46415_object48970:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-08-19T14:13:41.836 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:07.303688+0000", 2024-08-19T14:13:41.836 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 34.449066270000003, 2024-08-19T14:13:41.836 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.172252192, 2024-08-19T14:13:41.836 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:13:41.836 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:13:41.836 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:13:41.837 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:13:41.837 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:13:41.837 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:13:41.837 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 48971 2024-08-19T14:13:41.837 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.837 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:13:41.837 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.837 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:13:41.837 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.303688+0000", 2024-08-19T14:13:41.837 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:41.837 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.837 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.837 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:13:41.838 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.303688+0000", 2024-08-19T14:13:41.838 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:41.838 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.838 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.838 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:13:41.838 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.303689+0000", 2024-08-19T14:13:41.838 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.652e-06 2024-08-19T14:13:41.838 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.838 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.838 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:13:41.838 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.303729+0000", 2024-08-19T14:13:41.838 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9443000000000002e-05 2024-08-19T14:13:41.838 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.838 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.839 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:13:41.839 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.303744+0000", 2024-08-19T14:13:41.839 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5173e-05 2024-08-19T14:13:41.839 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.839 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.839 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:13:41.839 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.303751+0000", 2024-08-19T14:13:41.839 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.5910000000000001e-06 2024-08-19T14:13:41.839 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.839 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.839 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:13:41.839 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.303807+0000", 2024-08-19T14:13:41.839 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5918000000000002e-05 2024-08-19T14:13:41.839 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.840 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.840 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:13:41.840 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.303888+0000", 2024-08-19T14:13:41.840 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1302000000000002e-05 2024-08-19T14:13:41.840 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.840 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.840 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,10", 2024-08-19T14:13:41.840 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.303973+0000", 2024-08-19T14:13:41.840 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.5055000000000005e-05 2024-08-19T14:13:41.840 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.840 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.840 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:13:41.840 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.305453+0000", 2024-08-19T14:13:41.840 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001480182 2024-08-19T14:13:41.841 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.841 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.841 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:13:41.841 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.306105+0000", 2024-08-19T14:13:41.841 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00065189100000000004 2024-08-19T14:13:41.841 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.841 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.841 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:13:41.841 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.475889+0000", 2024-08-19T14:13:41.841 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.16978352599999999 2024-08-19T14:13:41.841 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.841 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.841 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:13:41.841 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.475919+0000", 2024-08-19T14:13:41.842 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0199e-05 2024-08-19T14:13:41.842 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.842 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.842 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:13:41.842 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.475940+0000", 2024-08-19T14:13:41.842 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.126e-05 2024-08-19T14:13:41.842 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:41.842 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:13:41.842 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:41.842 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.842 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.842 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:62930 2.7 2.377d1257 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-08-19T14:13:41.842 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:18.725228+0000", 2024-08-19T14:13:41.843 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 23.027525893, 2024-08-19T14:13:41.843 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.99330006599999998, 2024-08-19T14:13:41.843 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:13:41.843 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:13:41.843 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:13:41.843 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:13:41.843 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:13:41.843 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:13:41.843 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 62930 2024-08-19T14:13:41.843 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.843 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:13:41.843 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.843 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:13:41.843 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.725228+0000", 2024-08-19T14:13:41.844 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:41.844 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.844 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.844 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:13:41.844 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.725228+0000", 2024-08-19T14:13:41.844 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:41.844 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.844 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.844 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:13:41.844 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.725231+0000", 2024-08-19T14:13:41.844 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5950000000000001e-06 2024-08-19T14:13:41.844 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.844 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.844 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:13:41.845 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.725270+0000", 2024-08-19T14:13:41.845 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9162000000000002e-05 2024-08-19T14:13:41.845 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.845 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.845 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:13:41.845 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.725291+0000", 2024-08-19T14:13:41.845 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1220999999999999e-05 2024-08-19T14:13:41.845 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.845 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.845 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:13:41.845 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.725303+0000", 2024-08-19T14:13:41.845 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1788e-05 2024-08-19T14:13:41.845 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.846 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.846 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:13:41.846 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.725382+0000", 2024-08-19T14:13:41.846 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.9389999999999997e-05 2024-08-19T14:13:41.846 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.846 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.846 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:13:41.846 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.725395+0000", 2024-08-19T14:13:41.846 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.308e-05 2024-08-19T14:13:41.846 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.846 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.846 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:13:41.846 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.718510+0000", 2024-08-19T14:13:41.846 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.99311438900000004 2024-08-19T14:13:41.847 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.847 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.847 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:13:41.847 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.718528+0000", 2024-08-19T14:13:41.847 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8440999999999998e-05 2024-08-19T14:13:41.847 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:41.847 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:13:41.847 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:41.847 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.847 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.847 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:62932 2.7 2.84394fd7 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-08-19T14:13:41.847 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:18.727423+0000", 2024-08-19T14:13:41.847 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 23.025330732, 2024-08-19T14:13:41.847 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.99114474299999999, 2024-08-19T14:13:41.848 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:13:41.848 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:13:41.848 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:13:41.848 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:13:41.848 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:13:41.848 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:13:41.848 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 62932 2024-08-19T14:13:41.848 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.848 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:13:41.848 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.848 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:13:41.848 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.727423+0000", 2024-08-19T14:13:41.848 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:41.848 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.849 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.849 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:13:41.849 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.727423+0000", 2024-08-19T14:13:41.849 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:41.849 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.849 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.849 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:13:41.849 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.727425+0000", 2024-08-19T14:13:41.849 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0870000000000001e-06 2024-08-19T14:13:41.849 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.849 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.849 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:13:41.849 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.727471+0000", 2024-08-19T14:13:41.850 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5300999999999999e-05 2024-08-19T14:13:41.850 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.850 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.850 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:13:41.850 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.727489+0000", 2024-08-19T14:13:41.850 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7909000000000001e-05 2024-08-19T14:13:41.850 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.850 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.850 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:13:41.850 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.727497+0000", 2024-08-19T14:13:41.850 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.2150000000000001e-06 2024-08-19T14:13:41.850 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.850 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.850 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:13:41.851 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.727590+0000", 2024-08-19T14:13:41.851 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3193e-05 2024-08-19T14:13:41.851 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.851 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.851 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:13:41.851 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.727606+0000", 2024-08-19T14:13:41.851 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6345e-05 2024-08-19T14:13:41.851 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.851 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.851 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:13:41.851 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.718558+0000", 2024-08-19T14:13:41.851 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.99095196399999996 2024-08-19T14:13:41.851 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.852 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.852 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:13:41.852 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.718568+0000", 2024-08-19T14:13:41.852 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.7289999999999995e-06 2024-08-19T14:13:41.852 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:41.852 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:13:41.852 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:41.852 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.852 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.852 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:62948 2.7 2.63c33357 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-08-19T14:13:41.852 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:18.736424+0000", 2024-08-19T14:13:41.852 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 23.016330311000001, 2024-08-19T14:13:41.852 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.98218305500000003, 2024-08-19T14:13:41.852 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:13:41.853 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:13:41.853 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:13:41.853 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:13:41.853 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:13:41.853 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:13:41.853 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 62948 2024-08-19T14:13:41.853 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.853 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:13:41.853 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.853 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:13:41.853 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.736424+0000", 2024-08-19T14:13:41.853 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:41.853 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.853 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.854 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:13:41.854 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.736424+0000", 2024-08-19T14:13:41.854 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:41.854 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.854 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.854 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:13:41.854 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.736426+0000", 2024-08-19T14:13:41.854 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9980000000000002e-06 2024-08-19T14:13:41.854 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.854 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.854 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:13:41.854 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.736459+0000", 2024-08-19T14:13:41.854 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3155999999999999e-05 2024-08-19T14:13:41.854 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.855 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.855 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:13:41.855 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.736479+0000", 2024-08-19T14:13:41.855 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0052999999999999e-05 2024-08-19T14:13:41.855 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.855 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.855 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:13:41.855 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.736489+0000", 2024-08-19T14:13:41.855 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.7790000000000007e-06 2024-08-19T14:13:41.855 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.855 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.855 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:13:41.855 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.736567+0000", 2024-08-19T14:13:41.856 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.8240999999999998e-05 2024-08-19T14:13:41.856 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.856 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.856 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:13:41.856 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.736578+0000", 2024-08-19T14:13:41.856 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0813e-05 2024-08-19T14:13:41.856 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.856 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.856 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:13:41.856 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.718594+0000", 2024-08-19T14:13:41.856 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.98201632900000002 2024-08-19T14:13:41.856 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.856 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.857 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:13:41.857 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.718607+0000", 2024-08-19T14:13:41.857 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2686e-05 2024-08-19T14:13:41.857 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:41.857 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:13:41.857 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:41.857 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.857 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.857 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:62956 2.7 2.bf121107 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-08-19T14:13:41.857 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:18.741261+0000", 2024-08-19T14:13:41.857 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 23.011492881999999, 2024-08-19T14:13:41.857 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.97739631299999996, 2024-08-19T14:13:41.857 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:13:41.858 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:13:41.858 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:13:41.858 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:13:41.858 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:13:41.858 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:13:41.858 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 62956 2024-08-19T14:13:41.858 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.858 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:13:41.858 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.858 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:13:41.858 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.741261+0000", 2024-08-19T14:13:41.858 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:41.858 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.859 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.859 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:13:41.859 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.741261+0000", 2024-08-19T14:13:41.859 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:41.859 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.859 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.859 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:13:41.859 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.741263+0000", 2024-08-19T14:13:41.859 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.68e-06 2024-08-19T14:13:41.859 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.859 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.859 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:13:41.859 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.741301+0000", 2024-08-19T14:13:41.860 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8313999999999999e-05 2024-08-19T14:13:41.860 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.860 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.860 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:13:41.860 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.741320+0000", 2024-08-19T14:13:41.860 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.927e-05 2024-08-19T14:13:41.860 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.860 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.860 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:13:41.860 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.741328+0000", 2024-08-19T14:13:41.860 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3000000000000004e-06 2024-08-19T14:13:41.860 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.860 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.860 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:13:41.861 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.741408+0000", 2024-08-19T14:13:41.861 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.9839000000000006e-05 2024-08-19T14:13:41.861 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.861 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.861 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:13:41.861 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.741421+0000", 2024-08-19T14:13:41.861 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2914e-05 2024-08-19T14:13:41.861 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.861 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.861 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:13:41.861 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.718639+0000", 2024-08-19T14:13:41.861 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.97721861399999999 2024-08-19T14:13:41.861 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.861 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.862 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:13:41.862 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.718658+0000", 2024-08-19T14:13:41.862 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8382000000000001e-05 2024-08-19T14:13:41.862 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:41.862 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:13:41.862 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:41.862 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.862 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.862 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:62963 2.7 2.30acc547 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-08-19T14:13:41.862 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:18.745404+0000", 2024-08-19T14:13:41.862 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 23.007350338999998, 2024-08-19T14:13:41.862 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.97330407100000005, 2024-08-19T14:13:41.862 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:13:41.862 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:13:41.863 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:13:41.863 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:13:41.863 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:13:41.863 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:13:41.863 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 62963 2024-08-19T14:13:41.863 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.863 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:13:41.863 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.863 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:13:41.863 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.745404+0000", 2024-08-19T14:13:41.863 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:41.863 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.863 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.863 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:13:41.864 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.745404+0000", 2024-08-19T14:13:41.864 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:41.864 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.864 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.864 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:13:41.864 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.745406+0000", 2024-08-19T14:13:41.864 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.328e-06 2024-08-19T14:13:41.864 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.864 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.864 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:13:41.864 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.745440+0000", 2024-08-19T14:13:41.864 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3837000000000001e-05 2024-08-19T14:13:41.864 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.865 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.865 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:13:41.865 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.745474+0000", 2024-08-19T14:13:41.865 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.4196999999999998e-05 2024-08-19T14:13:41.865 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.865 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.865 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:13:41.865 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.745485+0000", 2024-08-19T14:13:41.865 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.095e-05 2024-08-19T14:13:41.865 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.865 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.865 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:13:41.865 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.745568+0000", 2024-08-19T14:13:41.865 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.2770999999999997e-05 2024-08-19T14:13:41.866 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.866 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.866 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:13:41.866 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.745577+0000", 2024-08-19T14:13:41.866 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.1269999999999992e-06 2024-08-19T14:13:41.866 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.866 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.866 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:13:41.866 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.718689+0000", 2024-08-19T14:13:41.866 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.97311253600000003 2024-08-19T14:13:41.866 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.866 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.866 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:13:41.866 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.718708+0000", 2024-08-19T14:13:41.867 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8325000000000002e-05 2024-08-19T14:13:41.867 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:41.867 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:13:41.867 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:41.867 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.867 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.867 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:62983 2.7 2.e3420e17 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-08-19T14:13:41.867 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:18.759553+0000", 2024-08-19T14:13:41.867 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 22.993200980000001, 2024-08-19T14:13:41.867 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.95919200800000004, 2024-08-19T14:13:41.867 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:13:41.867 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:13:41.867 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:13:41.867 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:13:41.868 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:13:41.868 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:13:41.868 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 62983 2024-08-19T14:13:41.868 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.868 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:13:41.868 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.868 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:13:41.868 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.759553+0000", 2024-08-19T14:13:41.868 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:41.868 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.868 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.868 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:13:41.868 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.759553+0000", 2024-08-19T14:13:41.868 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:41.869 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.869 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.869 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:13:41.869 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.759555+0000", 2024-08-19T14:13:41.869 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2630000000000002e-06 2024-08-19T14:13:41.869 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.869 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.869 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:13:41.869 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.759595+0000", 2024-08-19T14:13:41.869 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9490000000000003e-05 2024-08-19T14:13:41.869 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.869 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.869 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:13:41.869 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.759614+0000", 2024-08-19T14:13:41.870 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9446999999999999e-05 2024-08-19T14:13:41.870 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.870 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.870 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:13:41.870 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.759622+0000", 2024-08-19T14:13:41.870 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5519999999999999e-06 2024-08-19T14:13:41.870 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.870 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.870 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:13:41.870 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.759684+0000", 2024-08-19T14:13:41.870 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2168000000000005e-05 2024-08-19T14:13:41.870 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.870 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.870 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:13:41.871 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.759693+0000", 2024-08-19T14:13:41.871 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.8920000000000006e-06 2024-08-19T14:13:41.871 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.871 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.871 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:13:41.871 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.718735+0000", 2024-08-19T14:13:41.871 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.95904258099999995 2024-08-19T14:13:41.871 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.871 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.871 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:13:41.871 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.718745+0000", 2024-08-19T14:13:41.871 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.6150000000000003e-06 2024-08-19T14:13:41.871 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:41.871 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:13:41.872 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:41.872 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.872 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.872 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:63032 2.7 2.24f95377 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-08-19T14:13:41.872 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:18.809762+0000", 2024-08-19T14:13:41.872 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 22.942992297, 2024-08-19T14:13:41.872 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.909118655, 2024-08-19T14:13:41.872 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:13:41.872 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:13:41.872 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:13:41.872 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:13:41.872 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:13:41.872 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:13:41.873 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 63032 2024-08-19T14:13:41.873 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.873 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:13:41.873 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.873 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:13:41.873 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.809762+0000", 2024-08-19T14:13:41.873 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:41.873 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.873 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.873 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:13:41.873 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.809762+0000", 2024-08-19T14:13:41.873 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:41.873 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.873 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.873 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:13:41.874 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.809764+0000", 2024-08-19T14:13:41.874 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1490000000000001e-06 2024-08-19T14:13:41.874 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.874 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.874 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:13:41.874 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.809805+0000", 2024-08-19T14:13:41.874 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1241e-05 2024-08-19T14:13:41.874 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.874 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.874 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:13:41.874 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.809820+0000", 2024-08-19T14:13:41.874 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4355000000000001e-05 2024-08-19T14:13:41.874 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.874 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.875 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:13:41.875 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.809826+0000", 2024-08-19T14:13:41.875 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4829999999999996e-06 2024-08-19T14:13:41.875 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.875 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.875 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:13:41.875 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.809883+0000", 2024-08-19T14:13:41.875 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6885000000000002e-05 2024-08-19T14:13:41.875 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.875 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.875 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:13:41.875 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.809892+0000", 2024-08-19T14:13:41.875 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.8519999999999993e-06 2024-08-19T14:13:41.875 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.876 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.876 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:13:41.876 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.718868+0000", 2024-08-19T14:13:41.876 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.90897634900000002 2024-08-19T14:13:41.876 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:41.876 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:41.876 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:13:41.876 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.718880+0000", 2024-08-19T14:13:41.876 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2340999999999999e-05 2024-08-19T14:13:41.876 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:41.876 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:13:41.876 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:41.876 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:41.876 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:13:41.877 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:13:41.877 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:13:41.878 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 52 16 87049 87033 104.598 75.9375 0.00412663 0.00955577 2024-08-19T14:13:41.902 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:41.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:41.903 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:13:42.004 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:13:42.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:41 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:13:42.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:41 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:13:42.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:41 smithi032 ceph-mon[20004]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:13:42.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:41 smithi032 ceph-mon[20004]: Cluster is now healthy 2024-08-19T14:13:42.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:41 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:13:42.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:41 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"}]': finished 2024-08-19T14:13:42.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:41 smithi032 ceph-mon[20004]: osdmap e136: 12 total, 12 up, 10 in 2024-08-19T14:13:42.042 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:42.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:42.044 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:13:42.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:41 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:13:42.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:41 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:13:42.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:41 smithi045 ceph-mon[20143]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:13:42.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:41 smithi045 ceph-mon[20143]: Cluster is now healthy 2024-08-19T14:13:42.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:41 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:13:42.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:41 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"}]': finished 2024-08-19T14:13:42.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:41 smithi045 ceph-mon[20143]: osdmap e136: 12 total, 12 up, 10 in 2024-08-19T14:13:42.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:41 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:13:42.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:41 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:13:42.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:41 smithi031 ceph-mon[16336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:13:42.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:41 smithi031 ceph-mon[16336]: Cluster is now healthy 2024-08-19T14:13:42.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:41 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:13:42.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:41 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"}]': finished 2024-08-19T14:13:42.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:41 smithi031 ceph-mon[16336]: osdmap e136: 12 total, 12 up, 10 in 2024-08-19T14:13:42.145 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:13:42.169 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:42.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:42.170 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:13:42.271 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:13:42.298 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:42.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:42.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:13:42.400 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:13:42.424 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:42.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:42.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:13:42.526 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:13:42.552 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:42.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:42.553 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:13:42.654 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:13:42.682 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:42.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:42.684 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:13:42.785 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:13:42.811 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:42.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:42.812 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:13:42.834 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 53 16 88340 88324 104.147 80.6875 0.00334509 0.009596 2024-08-19T14:13:42.913 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:13:42.938 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:42.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:42.939 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:13:43.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:42 smithi032 ceph-mon[20004]: pgmap v350: 27 pgs: 3 active+recovering+undersized+remapped, 24 active+clean; 5.1 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 69 MiB/s wr, 1.10k op/s; 8967/252918 objects misplaced (3.545%); 12 MiB/s, 189 objects/s recovering 2024-08-19T14:13:43.039 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:13:43.065 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:43.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:43.066 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:13:43.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:42 smithi031 ceph-mon[16336]: pgmap v350: 27 pgs: 3 active+recovering+undersized+remapped, 24 active+clean; 5.1 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 69 MiB/s wr, 1.10k op/s; 8967/252918 objects misplaced (3.545%); 12 MiB/s, 189 objects/s recovering 2024-08-19T14:13:43.167 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:13:43.196 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:43.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:43.197 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:13:43.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:42 smithi045 ceph-mon[20143]: pgmap v350: 27 pgs: 3 active+recovering+undersized+remapped, 24 active+clean; 5.1 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 69 MiB/s wr, 1.10k op/s; 8967/252918 objects misplaced (3.545%); 12 MiB/s, 189 objects/s recovering 2024-08-19T14:13:43.297 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:13:43.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:43.327 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:43.327 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:13:43.428 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:13:43.454 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:43.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:43.455 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:13:43.556 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:13:43.581 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:43.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:43.582 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:13:43.594 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:13:43.683 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:13:43.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:43.725 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:43.725 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:13:43.826 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:13:43.834 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 54 16 89388 89372 103.431 65.5 0.00424883 0.00966299 2024-08-19T14:13:43.850 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:43.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:43.852 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:13:43.953 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:13:44.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:44.006 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:44.006 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:13:44.107 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:13:44.132 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:44.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:44.133 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:13:44.234 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:13:44.259 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:44.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:44.260 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:13:44.274 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:13:44.275 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:13:44.312 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:13:44.329 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:13:44.347 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:13:44.348 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:13:44.361 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:13:44.391 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:13:44.392 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:13:44.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:44.401 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:44.402 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:13:44.424 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:13:44.425 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:13:44.459 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:13:44.460 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:13:44.492 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:13:44.492 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:13:44.503 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:13:44.527 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:13:44.527 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:13:44.538 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:44.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:44.539 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:13:44.559 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:13:44.559 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:13:44.593 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:13:44.593 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:13:44.624 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:13:44.624 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:13:44.640 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:13:44.660 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:13:44.661 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:13:44.676 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:44.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:44.677 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:13:44.778 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:13:44.817 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:44.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:44.818 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:13:44.834 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 55 16 90465 90449 102.774 67.3125 0.0111919 0.00972482 2024-08-19T14:13:44.918 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:13:44.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:44 smithi031 ceph-mon[16336]: pgmap v351: 27 pgs: 3 active+recovering+undersized+remapped, 24 active+clean; 5.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 70 MiB/s wr, 1.13k op/s; 8722/259575 objects misplaced (3.360%); 12 MiB/s, 195 objects/s recovering 2024-08-19T14:13:44.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:44 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1092720772' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:13:44.944 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:44.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:44.945 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:13:45.046 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:13:45.075 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:45.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:45.076 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:13:45.177 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:13:45.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:45.205 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:45.205 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:13:45.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:44 smithi032 ceph-mon[20004]: pgmap v351: 27 pgs: 3 active+recovering+undersized+remapped, 24 active+clean; 5.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 70 MiB/s wr, 1.13k op/s; 8722/259575 objects misplaced (3.360%); 12 MiB/s, 195 objects/s recovering 2024-08-19T14:13:45.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:44 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1092720772' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:13:45.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:44 smithi045 ceph-mon[20143]: pgmap v351: 27 pgs: 3 active+recovering+undersized+remapped, 24 active+clean; 5.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 70 MiB/s wr, 1.13k op/s; 8722/259575 objects misplaced (3.360%); 12 MiB/s, 195 objects/s recovering 2024-08-19T14:13:45.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:44 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1092720772' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:13:45.305 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:13:45.342 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:45.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:45.343 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:13:45.444 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:13:45.470 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:45.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:45.471 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:13:45.481 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-19T14:13:45.572 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:13:45.602 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:45.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:45.603 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:13:45.704 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:13:45.731 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:45.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:45.732 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:13:45.817 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-08-19T14:13:45.832 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:13:45.834 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 56 16 91392 91376 101.973 57.9375 0.0074557 0.00980182 2024-08-19T14:13:45.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:45.891 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:45.891 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:13:45.923 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:13:45.992 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:13:46.039 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:46.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:46.040 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:13:46.074 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:45 smithi031 ceph-mon[16336]: 2.0 scrub starts 2024-08-19T14:13:46.141 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:13:46.156 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:13:46.172 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:46.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:46.173 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:13:46.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:45 smithi032 ceph-mon[20004]: 2.0 scrub starts 2024-08-19T14:13:46.274 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:13:46.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:45 smithi045 ceph-mon[20143]: 2.0 scrub starts 2024-08-19T14:13:46.298 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:46.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:46.300 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:13:46.401 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:13:46.425 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:46.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:46.426 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:13:46.527 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:13:46.562 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:46.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:46.563 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:13:46.664 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:13:46.689 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:46.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:46.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:13:46.793 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:13:46.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:46.821 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:46.821 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:13:46.834 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 57 16 92516 92500 101.417 70.25 0.0253474 0.00985541 2024-08-19T14:13:46.922 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:13:46.949 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:46.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:46.951 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:13:47.052 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:13:47.075 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:47.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:47.076 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:13:47.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:46 smithi031 ceph-mon[16336]: pgmap v352: 27 pgs: 3 active+recovering+undersized+remapped, 24 active+clean; 5.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 65 MiB/s wr, 1.04k op/s; 8722/265047 objects misplaced (3.291%); 10 MiB/s, 160 objects/s recovering 2024-08-19T14:13:47.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:46 smithi031 ceph-mon[16336]: 2.0 scrub ok 2024-08-19T14:13:47.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:46 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:13:47.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:46 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:13:47.177 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:13:47.203 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:47.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:47.204 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:13:47.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:46 smithi032 ceph-mon[20004]: pgmap v352: 27 pgs: 3 active+recovering+undersized+remapped, 24 active+clean; 5.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 65 MiB/s wr, 1.04k op/s; 8722/265047 objects misplaced (3.291%); 10 MiB/s, 160 objects/s recovering 2024-08-19T14:13:47.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:46 smithi032 ceph-mon[20004]: 2.0 scrub ok 2024-08-19T14:13:47.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:46 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:13:47.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:46 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:13:47.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:46 smithi045 ceph-mon[20143]: pgmap v352: 27 pgs: 3 active+recovering+undersized+remapped, 24 active+clean; 5.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 65 MiB/s wr, 1.04k op/s; 8722/265047 objects misplaced (3.291%); 10 MiB/s, 160 objects/s recovering 2024-08-19T14:13:47.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:46 smithi045 ceph-mon[20143]: 2.0 scrub ok 2024-08-19T14:13:47.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:46 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:13:47.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:46 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:13:47.305 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:13:47.337 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:47.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:47.338 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:13:47.367 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:13:47.440 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:13:47.482 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:47.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:47.483 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:13:47.584 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:13:47.608 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:47.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:47.609 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:13:47.711 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:13:47.762 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:47.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:47.763 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:13:47.833 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is set 2024-08-19T14:13:47.834 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 58 16 93655 93639 100.896 71.1875 0.0115156 0.00990502 2024-08-19T14:13:47.836 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:47 smithi031 ceph-mon[16336]: osdmap e137: 12 total, 12 up, 10 in 2024-08-19T14:13:47.836 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:47 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1735908225' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:13:47.842 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:13:47.864 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:13:47.871 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 8, 9, 11] out_osds: [10, 6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-08-19T14:13:47.871 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd reweight 0 1 2024-08-19T14:13:47.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:47.916 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:47.916 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:13:48.017 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:13:48.042 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:48.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:48.044 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:13:48.145 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:13:48.172 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:48.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:48.174 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:13:48.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:47 smithi032 ceph-mon[20004]: osdmap e137: 12 total, 12 up, 10 in 2024-08-19T14:13:48.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:47 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1735908225' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:13:48.275 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:13:48.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:47 smithi045 ceph-mon[20143]: osdmap e137: 12 total, 12 up, 10 in 2024-08-19T14:13:48.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:47 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1735908225' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:13:48.323 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:48.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:48.324 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:13:48.426 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:13:48.451 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:48.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:48.452 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:13:48.553 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:13:48.585 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:48.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:48.586 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:13:48.687 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:13:48.720 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:48.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:48.721 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:13:48.822 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:13:48.834 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 59 16 94794 94778 100.392 71.1875 0.00890962 0.00995482 2024-08-19T14:13:48.852 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:48.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:48.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:13:48.913 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:13:48.913 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:13:48.954 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:13:48.980 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:48.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:48.981 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:13:49.056 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:48 smithi031 ceph-mon[16336]: pgmap v354: 27 pgs: 3 active+recovering+undersized+remapped, 24 active+clean; 5.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 71 MiB/s wr, 1.14k op/s; 7587/273054 objects misplaced (2.779%); 15 MiB/s, 232 objects/s recovering 2024-08-19T14:13:49.056 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:48 smithi031 ceph-mon[16336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:13:49.056 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:48 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1735908225' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:13:49.056 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:48 smithi031 ceph-mon[16336]: osdmap e138: 12 total, 12 up, 10 in 2024-08-19T14:13:49.082 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:13:49.105 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:49.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:49.106 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:13:49.117 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:13:49.122 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:13:49.122 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:13:49.165 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:13:49.166 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:13:49.207 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:13:49.213 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:13:49.213 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:13:49.247 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:48 smithi032 ceph-mon[20004]: pgmap v354: 27 pgs: 3 active+recovering+undersized+remapped, 24 active+clean; 5.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 71 MiB/s wr, 1.14k op/s; 7587/273054 objects misplaced (2.779%); 15 MiB/s, 232 objects/s recovering 2024-08-19T14:13:49.247 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:48 smithi032 ceph-mon[20004]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:13:49.248 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:48 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1735908225' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:13:49.248 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:48 smithi032 ceph-mon[20004]: osdmap e138: 12 total, 12 up, 10 in 2024-08-19T14:13:49.255 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:13:49.255 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:13:49.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:49.262 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:49.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:13:49.289 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:13:49.290 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:13:49.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:48 smithi045 ceph-mon[20143]: pgmap v354: 27 pgs: 3 active+recovering+undersized+remapped, 24 active+clean; 5.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 71 MiB/s wr, 1.14k op/s; 7587/273054 objects misplaced (2.779%); 15 MiB/s, 232 objects/s recovering 2024-08-19T14:13:49.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:48 smithi045 ceph-mon[20143]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:13:49.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:48 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1735908225' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:13:49.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:48 smithi045 ceph-mon[20143]: osdmap e138: 12 total, 12 up, 10 in 2024-08-19T14:13:49.320 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:13:49.321 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:13:49.356 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:13:49.356 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:13:49.363 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:13:49.388 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:49.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:49.389 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:13:49.490 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:13:49.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:49.516 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:49.516 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:13:49.591 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:13:49.592 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:13:49.617 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:13:49.636 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:13:49.636 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:13:49.641 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:49.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:49.642 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:13:49.678 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:13:49.678 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:13:49.687 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-08-19T14:13:49.743 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:13:49.770 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:49.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:49.771 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:13:49.834 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:2024-08-19T14:13:49.835302+0000 min lat: 0.00106853 max lat: 2.17987 avg lat: 0.0100122 2024-08-19T14:13:49.835 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-19T14:13:49.835 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 60 16 95858 95842 99.8269 66.5 0.0278402 0.0100122 2024-08-19T14:13:49.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:49 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1879774959' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:13:49.872 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:13:49.906 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:49.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:49.907 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:13:49.916 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:13:49.916 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:13:49.957 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:13:49.957 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:13:50.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:49 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1879774959' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:13:50.008 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:13:50.012 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:13:50.047 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:50.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:50.048 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:13:50.149 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:13:50.180 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:50.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:50.182 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:13:50.283 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:13:50.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:49 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1879774959' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:13:50.308 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:50.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:50.309 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:13:50.410 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:13:50.436 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:50.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:50.437 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:13:50.537 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:13:50.539 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-19T14:13:50.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:50.577 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:50.577 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:13:50.679 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:13:50.707 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:50.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:50.708 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:13:50.809 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:13:50.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:50.836 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:50.837 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:13:50.837 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 61 16 96976 96960 99.3358 69.875 0.0167699 0.0100625 2024-08-19T14:13:50.856 INFO:teuthology.orchestra.run.smithi031.stderr:reweighted osd.0 to 1 (10000) 2024-08-19T14:13:50.938 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:13:50.975 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:50.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:50.976 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:13:51.077 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:13:51.078 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:50 smithi032 ceph-mon[20004]: pgmap v356: 27 pgs: 3 active+recovering+undersized+remapped, 24 active+clean; 5.7 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 68 MiB/s wr, 1.09k op/s; 7587/279192 objects misplaced (2.717%); 9.7 MiB/s, 154 objects/s recovering 2024-08-19T14:13:51.079 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:50 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3183719300' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-08-19T14:13:51.079 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:50 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-08-19T14:13:51.106 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:13:51.106 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-08-19T14:13:51.106 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-08-19T14:13:51.106 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-08-19T14:13:51.106 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:13:51.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:51.115 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:51.115 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:13:51.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:50 smithi031 ceph-mon[16336]: pgmap v356: 27 pgs: 3 active+recovering+undersized+remapped, 24 active+clean; 5.7 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 68 MiB/s wr, 1.09k op/s; 7587/279192 objects misplaced (2.717%); 9.7 MiB/s, 154 objects/s recovering 2024-08-19T14:13:51.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:50 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3183719300' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-08-19T14:13:51.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:50 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-08-19T14:13:51.216 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:13:51.239 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:51.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:51.240 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:13:51.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:50 smithi045 ceph-mon[20143]: pgmap v356: 27 pgs: 3 active+recovering+undersized+remapped, 24 active+clean; 5.7 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 68 MiB/s wr, 1.09k op/s; 7587/279192 objects misplaced (2.717%); 9.7 MiB/s, 154 objects/s recovering 2024-08-19T14:13:51.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:50 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3183719300' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-08-19T14:13:51.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:50 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-08-19T14:13:51.342 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:13:51.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:51.376 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:51.376 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:13:51.477 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:13:51.513 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:51.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:51.514 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:13:51.615 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:13:51.657 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:51.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:51.658 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:13:51.759 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:13:51.804 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:51.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:51.805 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:13:51.826 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd reweight 1 1 2024-08-19T14:13:51.834 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 62 16 98144 98128 98.9109 73 0.00583466 0.0101048 2024-08-19T14:13:51.852 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:13:51.906 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:13:51.947 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:51.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:51.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:13:52.049 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:13:52.076 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-08-19T14:13:52.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:52.094 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:52.094 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:13:52.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:51 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-08-19T14:13:52.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:51 smithi031 ceph-mon[16336]: osdmap e139: 12 total, 12 up, 10 in 2024-08-19T14:13:52.195 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:13:52.222 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:52.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:52.223 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:13:52.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:51 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-08-19T14:13:52.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:51 smithi032 ceph-mon[20004]: osdmap e139: 12 total, 12 up, 10 in 2024-08-19T14:13:52.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:51 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-08-19T14:13:52.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:51 smithi045 ceph-mon[20143]: osdmap e139: 12 total, 12 up, 10 in 2024-08-19T14:13:52.324 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:13:52.357 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:52.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:52.358 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:13:52.459 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:13:52.485 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:13:52.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:52.515 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:52.516 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:13:52.578 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:13:52.617 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:13:52.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:52.644 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:52.644 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:13:52.743 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:13:52.745 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:13:52.772 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:52.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:52.773 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:13:52.835 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 63 16 99146 99130 98.3349 62.625 0.0159617 0.0101635 2024-08-19T14:13:52.863 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is set 2024-08-19T14:13:52.874 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:13:52.898 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:52.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:52.899 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:13:52.988 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:52 smithi031 ceph-mon[16336]: pgmap v358: 27 pgs: 3 active+recovering+undersized+remapped, 24 active+clean; 5.8 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 75 MiB/s wr, 1.20k op/s; 6678/286716 objects misplaced (2.329%); 18 MiB/s, 294 objects/s recovering 2024-08-19T14:13:52.989 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:52 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/78279134' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:13:52.999 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:13:53.005 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:52 smithi032 ceph-mon[20004]: pgmap v358: 27 pgs: 3 active+recovering+undersized+remapped, 24 active+clean; 5.8 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 75 MiB/s wr, 1.20k op/s; 6678/286716 objects misplaced (2.329%); 18 MiB/s, 294 objects/s recovering 2024-08-19T14:13:53.005 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:52 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/78279134' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:13:53.019 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:13:53.020 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-08-19T14:13:53.020 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-08-19T14:13:53.020 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-08-19T14:13:53.020 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.020 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:63160 2.0 2:08542b0e:::benchmark_data_smithi045_46415_object63159:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-08-19T14:13:53.020 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:20.958791+0000", 2024-08-19T14:13:53.020 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 32.055662128000002, 2024-08-19T14:13:53.020 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.21296446799999999, 2024-08-19T14:13:53.020 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:13:53.021 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:13:53.021 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:13:53.021 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:13:53.021 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:13:53.021 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:13:53.021 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 63160 2024-08-19T14:13:53.021 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.021 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:13:53.021 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.021 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:13:53.021 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.958791+0000", 2024-08-19T14:13:53.021 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:53.021 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.021 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.022 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:13:53.022 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.958791+0000", 2024-08-19T14:13:53.022 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:53.022 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.022 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.022 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:13:53.022 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.958792+0000", 2024-08-19T14:13:53.022 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.514e-06 2024-08-19T14:13:53.022 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.022 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.022 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:13:53.022 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.958829+0000", 2024-08-19T14:13:53.022 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6594999999999998e-05 2024-08-19T14:13:53.023 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.023 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.023 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:13:53.023 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.958845+0000", 2024-08-19T14:13:53.023 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6571000000000001e-05 2024-08-19T14:13:53.023 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.023 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.023 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:13:53.023 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.958854+0000", 2024-08-19T14:13:53.023 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.4810000000000006e-06 2024-08-19T14:13:53.023 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.023 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.023 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:13:53.023 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.960591+0000", 2024-08-19T14:13:53.024 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0017371929999999999 2024-08-19T14:13:53.024 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.024 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.024 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:13:53.024 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.960793+0000", 2024-08-19T14:13:53.024 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00020158200000000001 2024-08-19T14:13:53.024 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.024 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.024 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 9,11", 2024-08-19T14:13:53.024 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.961006+0000", 2024-08-19T14:13:53.024 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000213169 2024-08-19T14:13:53.024 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.024 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.024 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:13:53.025 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.962896+0000", 2024-08-19T14:13:53.025 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0018897460000000001 2024-08-19T14:13:53.025 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.025 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.025 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:13:53.025 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.965630+0000", 2024-08-19T14:13:53.025 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.002733961 2024-08-19T14:13:53.025 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.025 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.025 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:13:53.025 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.170985+0000", 2024-08-19T14:13:53.025 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.205355648 2024-08-19T14:13:53.026 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.026 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.026 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:13:53.026 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.171055+0000", 2024-08-19T14:13:53.026 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9776000000000004e-05 2024-08-19T14:13:53.026 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.026 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.026 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:13:53.026 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.171755+0000", 2024-08-19T14:13:53.026 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00070023199999999996 2024-08-19T14:13:53.026 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:53.026 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:13:53.027 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:53.027 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.027 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.027 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:63163 2.8 2:1e3c608e:::benchmark_data_smithi045_46415_object63162:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-08-19T14:13:53.027 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:20.961739+0000", 2024-08-19T14:13:53.027 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 32.052713765999997, 2024-08-19T14:13:53.027 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.209352172, 2024-08-19T14:13:53.027 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:13:53.027 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:13:53.027 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:13:53.027 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:13:53.027 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:13:53.028 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:13:53.028 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 63163 2024-08-19T14:13:53.028 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.028 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:13:53.028 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.028 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:13:53.028 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.961739+0000", 2024-08-19T14:13:53.028 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:53.028 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.028 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.028 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:13:53.028 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.961739+0000", 2024-08-19T14:13:53.029 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:53.029 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.029 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.029 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:13:53.029 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.961740+0000", 2024-08-19T14:13:53.029 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2049999999999999e-06 2024-08-19T14:13:53.029 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.029 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.029 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:13:53.029 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.961818+0000", 2024-08-19T14:13:53.029 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7275999999999997e-05 2024-08-19T14:13:53.029 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.029 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.029 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:13:53.030 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.961828+0000", 2024-08-19T14:13:53.030 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0407999999999999e-05 2024-08-19T14:13:53.030 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.030 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.030 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:13:53.030 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.961835+0000", 2024-08-19T14:13:53.030 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5089999999999996e-06 2024-08-19T14:13:53.030 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.030 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.030 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:13:53.030 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.961891+0000", 2024-08-19T14:13:53.030 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5785000000000002e-05 2024-08-19T14:13:53.030 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.031 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.031 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:13:53.031 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.962014+0000", 2024-08-19T14:13:53.031 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012311200000000001 2024-08-19T14:13:53.031 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.031 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.031 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0,9", 2024-08-19T14:13:53.031 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.962141+0000", 2024-08-19T14:13:53.031 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012648 2024-08-19T14:13:53.031 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.031 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.031 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:13:53.031 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.966506+0000", 2024-08-19T14:13:53.031 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0043654719999999996 2024-08-19T14:13:53.032 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.032 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.032 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:13:53.032 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.968079+0000", 2024-08-19T14:13:53.032 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001572316 2024-08-19T14:13:53.032 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.032 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.032 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:13:53.032 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.170964+0000", 2024-08-19T14:13:53.032 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20288556799999999 2024-08-19T14:13:53.032 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.032 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.032 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:13:53.033 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.171048+0000", 2024-08-19T14:13:53.033 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.3412999999999996e-05 2024-08-19T14:13:53.033 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.033 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.033 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:13:53.033 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.171091+0000", 2024-08-19T14:13:53.033 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3628000000000003e-05 2024-08-19T14:13:53.033 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:53.033 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:13:53.033 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:53.033 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.033 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.033 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:63175 2.0 2:031c8466:::benchmark_data_smithi045_46415_object63174:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-08-19T14:13:53.034 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:20.969904+0000", 2024-08-19T14:13:53.034 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 32.044549285999999, 2024-08-19T14:13:53.034 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20268109100000001, 2024-08-19T14:13:53.034 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:13:53.034 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:13:53.034 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:13:53.034 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:13:53.034 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:13:53.034 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:13:53.034 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 63175 2024-08-19T14:13:53.034 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.034 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:13:53.035 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.035 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:13:53.035 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.969904+0000", 2024-08-19T14:13:53.035 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:53.035 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.035 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.035 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:13:53.035 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.969904+0000", 2024-08-19T14:13:53.035 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:53.035 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.035 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.035 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:13:53.035 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.969905+0000", 2024-08-19T14:13:53.035 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7969999999999999e-06 2024-08-19T14:13:53.036 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.036 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.036 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:13:53.036 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.969970+0000", 2024-08-19T14:13:53.036 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4850000000000004e-05 2024-08-19T14:13:53.036 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.036 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.036 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:13:53.036 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.969995+0000", 2024-08-19T14:13:53.036 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5168999999999999e-05 2024-08-19T14:13:53.036 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.036 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.036 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:13:53.036 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.970003+0000", 2024-08-19T14:13:53.037 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3939999999999998e-06 2024-08-19T14:13:53.037 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.037 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.037 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:13:53.037 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.971722+0000", 2024-08-19T14:13:53.037 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0017196220000000001 2024-08-19T14:13:53.037 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.037 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.037 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:13:53.037 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.971840+0000", 2024-08-19T14:13:53.037 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000117261 2024-08-19T14:13:53.037 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.037 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.038 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 9,11", 2024-08-19T14:13:53.038 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.972030+0000", 2024-08-19T14:13:53.038 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00018997799999999999 2024-08-19T14:13:53.038 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.038 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.038 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:13:53.038 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.976418+0000", 2024-08-19T14:13:53.038 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0043884140000000002 2024-08-19T14:13:53.038 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.038 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.038 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:13:53.038 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.976439+0000", 2024-08-19T14:13:53.038 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0979999999999999e-05 2024-08-19T14:13:53.039 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.039 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.039 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:13:53.039 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.171818+0000", 2024-08-19T14:13:53.039 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.19537907500000001 2024-08-19T14:13:53.039 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.039 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.039 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:13:53.039 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.171880+0000", 2024-08-19T14:13:53.039 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1791999999999994e-05 2024-08-19T14:13:53.039 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.039 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.039 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:13:53.040 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.172585+0000", 2024-08-19T14:13:53.040 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00070475900000000001 2024-08-19T14:13:53.040 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:53.040 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:13:53.040 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:53.040 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.040 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.040 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:63177 2.8 2:15a25ebc:::benchmark_data_smithi045_46415_object63176:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-08-19T14:13:53.040 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:20.970729+0000", 2024-08-19T14:13:53.040 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 32.043724257999997, 2024-08-19T14:13:53.040 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.200497493, 2024-08-19T14:13:53.040 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:13:53.041 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:13:53.041 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:13:53.041 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:13:53.041 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:13:53.041 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:13:53.041 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 63177 2024-08-19T14:13:53.041 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.041 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:13:53.041 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.041 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:13:53.041 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.970729+0000", 2024-08-19T14:13:53.041 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:53.041 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.042 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.042 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:13:53.042 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.970729+0000", 2024-08-19T14:13:53.042 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:53.042 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.042 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.042 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:13:53.042 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.970730+0000", 2024-08-19T14:13:53.042 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.02e-06 2024-08-19T14:13:53.042 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.042 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.042 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:13:53.042 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.970816+0000", 2024-08-19T14:13:53.043 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6040999999999998e-05 2024-08-19T14:13:53.043 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.043 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.043 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:13:53.043 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.970826+0000", 2024-08-19T14:13:53.043 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0361e-05 2024-08-19T14:13:53.043 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.043 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.043 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:13:53.043 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.970831+0000", 2024-08-19T14:13:53.043 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9459999999999997e-06 2024-08-19T14:13:53.043 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.043 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.044 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:13:53.044 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.971716+0000", 2024-08-19T14:13:53.044 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00088531199999999997 2024-08-19T14:13:53.044 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.044 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.044 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:13:53.044 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.971839+0000", 2024-08-19T14:13:53.044 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012251499999999999 2024-08-19T14:13:53.044 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.044 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.044 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0,9", 2024-08-19T14:13:53.044 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.972011+0000", 2024-08-19T14:13:53.044 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000172244 2024-08-19T14:13:53.045 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.045 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.045 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:13:53.045 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.975032+0000", 2024-08-19T14:13:53.045 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.003021195 2024-08-19T14:13:53.045 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.045 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.045 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:13:53.045 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.976447+0000", 2024-08-19T14:13:53.045 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001415152 2024-08-19T14:13:53.045 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.045 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.045 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:13:53.045 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.171136+0000", 2024-08-19T14:13:53.045 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.19468817199999999 2024-08-19T14:13:53.046 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.046 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.046 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:13:53.046 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.171189+0000", 2024-08-19T14:13:53.046 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3505999999999999e-05 2024-08-19T14:13:53.046 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.046 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.046 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:13:53.046 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.171226+0000", 2024-08-19T14:13:53.046 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7029000000000001e-05 2024-08-19T14:13:53.046 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:53.046 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:13:53.046 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:53.046 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.047 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.047 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:63179 2.0 2:034b6ab5:::benchmark_data_smithi045_46415_object63178:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-08-19T14:13:53.047 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:20.971796+0000", 2024-08-19T14:13:53.047 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 32.042657269000003, 2024-08-19T14:13:53.047 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20158538500000001, 2024-08-19T14:13:53.047 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:13:53.047 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:13:53.047 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:13:53.047 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:13:53.047 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:13:53.047 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:13:53.047 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 63179 2024-08-19T14:13:53.048 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.048 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:13:53.048 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.048 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:13:53.048 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.971796+0000", 2024-08-19T14:13:53.048 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:53.048 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.048 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.048 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:13:53.048 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.971796+0000", 2024-08-19T14:13:53.048 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:53.048 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.049 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.049 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:13:53.049 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.971796+0000", 2024-08-19T14:13:53.049 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.2399999999999996e-07 2024-08-19T14:13:53.049 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.049 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.049 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:13:53.050 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.971857+0000", 2024-08-19T14:13:53.050 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0863000000000003e-05 2024-08-19T14:13:53.050 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.050 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.050 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:13:53.050 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.971867+0000", 2024-08-19T14:13:53.050 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.5619999999999993e-06 2024-08-19T14:13:53.050 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.050 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.050 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:13:53.050 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.971874+0000", 2024-08-19T14:13:53.050 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.5989999999999998e-06 2024-08-19T14:13:53.050 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.050 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.051 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:13:53.051 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.973576+0000", 2024-08-19T14:13:53.051 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001702928 2024-08-19T14:13:53.051 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.051 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.051 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:13:53.051 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.973668+0000", 2024-08-19T14:13:53.051 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.1094000000000006e-05 2024-08-19T14:13:53.051 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.051 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.051 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 9,11", 2024-08-19T14:13:53.051 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.973776+0000", 2024-08-19T14:13:53.051 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000108453 2024-08-19T14:13:53.051 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.052 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.052 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:13:53.052 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.976463+0000", 2024-08-19T14:13:53.052 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0026873999999999999 2024-08-19T14:13:53.052 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.052 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.052 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:13:53.052 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.979780+0000", 2024-08-19T14:13:53.052 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0033170740000000002 2024-08-19T14:13:53.052 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.052 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.052 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:13:53.052 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.172645+0000", 2024-08-19T14:13:53.052 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.192864063 2024-08-19T14:13:53.052 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.053 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.053 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:13:53.053 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.172698+0000", 2024-08-19T14:13:53.053 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3647000000000003e-05 2024-08-19T14:13:53.053 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.053 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.053 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:13:53.053 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.173381+0000", 2024-08-19T14:13:53.053 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00068277799999999999 2024-08-19T14:13:53.053 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:53.053 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:13:53.053 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:53.053 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.054 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.054 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:63180 2.0 2:09d38ed3:::benchmark_data_smithi045_46415_object63179:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-08-19T14:13:53.054 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:20.971911+0000", 2024-08-19T14:13:53.054 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 32.042541601000003, 2024-08-19T14:13:53.054 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20210957399999999, 2024-08-19T14:13:53.054 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:13:53.054 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:13:53.054 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:13:53.054 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:13:53.054 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:13:53.054 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:13:53.054 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 63180 2024-08-19T14:13:53.054 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.054 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:13:53.055 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.055 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:13:53.055 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.971911+0000", 2024-08-19T14:13:53.055 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:53.055 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.055 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.055 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:13:53.055 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.971911+0000", 2024-08-19T14:13:53.055 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:53.055 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.056 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.056 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:13:53.056 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.971915+0000", 2024-08-19T14:13:53.056 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.07e-06 2024-08-19T14:13:53.056 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.056 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.056 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:13:53.056 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.971998+0000", 2024-08-19T14:13:53.056 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.2995000000000001e-05 2024-08-19T14:13:53.056 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.056 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.056 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:13:53.057 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.972011+0000", 2024-08-19T14:13:53.057 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2262999999999999e-05 2024-08-19T14:13:53.057 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.057 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.057 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:13:53.057 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.972017+0000", 2024-08-19T14:13:53.057 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4400000000000002e-06 2024-08-19T14:13:53.057 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.057 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.057 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:13:53.057 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.975045+0000", 2024-08-19T14:13:53.057 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0030280440000000001 2024-08-19T14:13:53.057 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.058 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.058 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:13:53.058 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.975155+0000", 2024-08-19T14:13:53.058 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00010984699999999999 2024-08-19T14:13:53.058 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.058 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.058 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 9,11", 2024-08-19T14:13:53.058 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.975247+0000", 2024-08-19T14:13:53.058 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.1947999999999994e-05 2024-08-19T14:13:53.058 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.058 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.058 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:13:53.059 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.979753+0000", 2024-08-19T14:13:53.059 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0045062119999999999 2024-08-19T14:13:53.059 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.059 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.059 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:13:53.059 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.979791+0000", 2024-08-19T14:13:53.059 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7511000000000001e-05 2024-08-19T14:13:53.059 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.059 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.059 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:13:53.060 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.173451+0000", 2024-08-19T14:13:53.060 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.19366074699999999 2024-08-19T14:13:53.060 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.060 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.060 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:13:53.060 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.173513+0000", 2024-08-19T14:13:53.060 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1480000000000001e-05 2024-08-19T14:13:53.060 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.060 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.060 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:13:53.060 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.174021+0000", 2024-08-19T14:13:53.060 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00050801700000000004 2024-08-19T14:13:53.061 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:53.061 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:13:53.061 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:53.061 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.061 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.061 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:63181 2.8 2:10ca0581:::benchmark_data_smithi045_46415_object63180:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-08-19T14:13:53.061 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:20.972411+0000", 2024-08-19T14:13:53.061 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 32.042042299999999, 2024-08-19T14:13:53.061 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20026014, 2024-08-19T14:13:53.061 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:13:53.061 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:13:53.061 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:13:53.062 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:13:53.062 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:13:53.062 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:13:53.062 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 63181 2024-08-19T14:13:53.062 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.062 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:13:53.062 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.062 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:13:53.062 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.972411+0000", 2024-08-19T14:13:53.062 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:53.062 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.062 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.062 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:13:53.063 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.972411+0000", 2024-08-19T14:13:53.063 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:53.063 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.063 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.063 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:13:53.063 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.972411+0000", 2024-08-19T14:13:53.063 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.3500000000000005e-07 2024-08-19T14:13:53.063 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.063 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.063 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:13:53.063 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.972473+0000", 2024-08-19T14:13:53.063 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1877000000000004e-05 2024-08-19T14:13:53.064 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.064 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.064 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:13:53.064 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.972485+0000", 2024-08-19T14:13:53.064 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1620999999999999e-05 2024-08-19T14:13:53.064 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.064 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.064 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:13:53.064 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.972492+0000", 2024-08-19T14:13:53.064 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9840000000000004e-06 2024-08-19T14:13:53.064 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.064 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.065 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:13:53.065 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.976487+0000", 2024-08-19T14:13:53.065 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0039953300000000001 2024-08-19T14:13:53.065 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.065 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.065 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:13:53.065 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.976598+0000", 2024-08-19T14:13:53.065 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000111099 2024-08-19T14:13:53.065 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.065 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.065 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0,9", 2024-08-19T14:13:53.065 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.976732+0000", 2024-08-19T14:13:53.065 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000134029 2024-08-19T14:13:53.065 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.065 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.066 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:13:53.066 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.979797+0000", 2024-08-19T14:13:53.066 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0030645109999999998 2024-08-19T14:13:53.066 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.066 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.066 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:13:53.066 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.979913+0000", 2024-08-19T14:13:53.066 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000116496 2024-08-19T14:13:53.066 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.066 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.066 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:13:53.066 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.172616+0000", 2024-08-19T14:13:53.066 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.19270316600000001 2024-08-19T14:13:53.066 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.067 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.067 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:13:53.067 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.172649+0000", 2024-08-19T14:13:53.067 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2589000000000003e-05 2024-08-19T14:13:53.067 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.067 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.067 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:13:53.067 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.172671+0000", 2024-08-19T14:13:53.067 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1603e-05 2024-08-19T14:13:53.067 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:53.067 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:13:53.067 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:53.067 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.067 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.067 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:63205 2.8 2:16572d7a:::benchmark_data_smithi045_46415_object63204:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-08-19T14:13:53.067 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:21.002885+0000", 2024-08-19T14:13:53.068 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 32.011568171999997, 2024-08-19T14:13:53.068 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.17128316199999999, 2024-08-19T14:13:53.068 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:13:53.068 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:13:53.068 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:13:53.068 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:13:53.068 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:13:53.068 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:13:53.068 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 63205 2024-08-19T14:13:53.068 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.068 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:13:53.068 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.068 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:13:53.068 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.002885+0000", 2024-08-19T14:13:53.069 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:53.069 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.069 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.069 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:13:53.069 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.002885+0000", 2024-08-19T14:13:53.069 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:53.069 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.069 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.069 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:13:53.069 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.002887+0000", 2024-08-19T14:13:53.069 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2139999999999999e-06 2024-08-19T14:13:53.069 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.069 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.069 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:13:53.070 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.002928+0000", 2024-08-19T14:13:53.070 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0703e-05 2024-08-19T14:13:53.070 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.070 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.070 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:13:53.070 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.002940+0000", 2024-08-19T14:13:53.070 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2831e-05 2024-08-19T14:13:53.070 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.070 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.070 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:13:53.070 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.002948+0000", 2024-08-19T14:13:53.070 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7740000000000004e-06 2024-08-19T14:13:53.070 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.071 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.071 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:13:53.071 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.003000+0000", 2024-08-19T14:13:53.071 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1483000000000002e-05 2024-08-19T14:13:53.071 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.071 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.071 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:13:53.071 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.003090+0000", 2024-08-19T14:13:53.071 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.0606000000000006e-05 2024-08-19T14:13:53.071 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.071 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.071 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0,9", 2024-08-19T14:13:53.071 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.003180+0000", 2024-08-19T14:13:53.072 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.9977999999999998e-05 2024-08-19T14:13:53.072 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.072 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.072 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:13:53.072 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.005559+0000", 2024-08-19T14:13:53.072 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0023788870000000001 2024-08-19T14:13:53.072 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.072 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.072 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:13:53.072 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.005623+0000", 2024-08-19T14:13:53.072 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4331000000000005e-05 2024-08-19T14:13:53.072 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.072 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.072 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:13:53.073 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.174071+0000", 2024-08-19T14:13:53.073 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.16844759200000001 2024-08-19T14:13:53.073 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.073 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.073 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:13:53.073 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.174125+0000", 2024-08-19T14:13:53.073 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3953000000000003e-05 2024-08-19T14:13:53.073 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.073 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.073 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:13:53.073 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.174168+0000", 2024-08-19T14:13:53.073 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2809999999999998e-05 2024-08-19T14:13:53.073 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:53.073 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:13:53.073 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:53.073 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.074 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.074 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:63230 2.0 2:025360f7:::benchmark_data_smithi045_46415_object63229:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-08-19T14:13:53.074 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:21.036099+0000", 2024-08-19T14:13:53.074 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 31.978353425999998, 2024-08-19T14:13:53.074 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.14053950100000001, 2024-08-19T14:13:53.074 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:13:53.074 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:13:53.074 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:13:53.074 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:13:53.074 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:13:53.074 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:13:53.074 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 63230 2024-08-19T14:13:53.074 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.074 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:13:53.074 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.075 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:13:53.075 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.036099+0000", 2024-08-19T14:13:53.075 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:53.075 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.075 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.075 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:13:53.075 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.036099+0000", 2024-08-19T14:13:53.075 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:53.075 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.075 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.075 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:13:53.075 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.036101+0000", 2024-08-19T14:13:53.075 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2279999999999999e-06 2024-08-19T14:13:53.075 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.075 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.076 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:13:53.076 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.036163+0000", 2024-08-19T14:13:53.076 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1940999999999995e-05 2024-08-19T14:13:53.076 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.076 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.076 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:13:53.076 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.036182+0000", 2024-08-19T14:13:53.076 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9686000000000001e-05 2024-08-19T14:13:53.076 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.076 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.076 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:13:53.076 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.036189+0000", 2024-08-19T14:13:53.076 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6640000000000001e-06 2024-08-19T14:13:53.076 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.076 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.077 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:13:53.077 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.037060+0000", 2024-08-19T14:13:53.077 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00087146399999999996 2024-08-19T14:13:53.077 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.077 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.077 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:13:53.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:53.078 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:53.078 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:13:53.079 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.037212+0000", 2024-08-19T14:13:53.079 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00015186400000000001 2024-08-19T14:13:53.079 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.080 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.080 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 9,11", 2024-08-19T14:13:53.080 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.037383+0000", 2024-08-19T14:13:53.080 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000170457 2024-08-19T14:13:53.080 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.080 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.080 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:13:53.080 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.039083+0000", 2024-08-19T14:13:53.080 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0016998 2024-08-19T14:13:53.080 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.080 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.080 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:13:53.080 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.041514+0000", 2024-08-19T14:13:53.080 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.002431519 2024-08-19T14:13:53.081 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.081 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.081 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:13:53.081 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.175973+0000", 2024-08-19T14:13:53.081 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.134458932 2024-08-19T14:13:53.081 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.081 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.081 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:13:53.081 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.176028+0000", 2024-08-19T14:13:53.081 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5036e-05 2024-08-19T14:13:53.082 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.082 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.082 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:13:53.082 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.176639+0000", 2024-08-19T14:13:53.082 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00061090999999999999 2024-08-19T14:13:53.082 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:53.082 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:13:53.082 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:53.082 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.082 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.082 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:63231 2.8 2:1da1a88e:::benchmark_data_smithi045_46415_object63230:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-08-19T14:13:53.082 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:21.036208+0000", 2024-08-19T14:13:53.082 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 31.978244397000001, 2024-08-19T14:13:53.083 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.13809454500000001, 2024-08-19T14:13:53.083 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:13:53.083 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:13:53.083 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:13:53.083 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:13:53.083 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:13:53.083 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:13:53.083 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 63231 2024-08-19T14:13:53.083 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.083 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:13:53.083 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.084 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:13:53.084 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.036208+0000", 2024-08-19T14:13:53.084 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:53.084 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.084 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.084 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:13:53.084 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.036208+0000", 2024-08-19T14:13:53.084 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:53.084 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.084 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.084 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:13:53.084 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.036209+0000", 2024-08-19T14:13:53.085 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4000000000000002e-07 2024-08-19T14:13:53.085 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.085 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.085 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:13:53.085 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.036233+0000", 2024-08-19T14:13:53.085 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4573999999999999e-05 2024-08-19T14:13:53.085 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.085 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.085 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:13:53.085 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.036240+0000", 2024-08-19T14:13:53.085 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4110000000000001e-06 2024-08-19T14:13:53.085 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.085 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.086 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:13:53.086 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.036244+0000", 2024-08-19T14:13:53.086 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9129999999999996e-06 2024-08-19T14:13:53.086 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.086 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.086 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:13:53.086 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.037035+0000", 2024-08-19T14:13:53.086 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00079092500000000005 2024-08-19T14:13:53.086 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.086 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.086 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:13:53.086 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.037123+0000", 2024-08-19T14:13:53.086 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.8616999999999995e-05 2024-08-19T14:13:53.087 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.087 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.087 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0,9", 2024-08-19T14:13:53.087 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.037210+0000", 2024-08-19T14:13:53.087 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6300999999999999e-05 2024-08-19T14:13:53.087 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.087 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.087 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:13:53.087 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.038951+0000", 2024-08-19T14:13:53.089 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001741151 2024-08-19T14:13:53.089 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.089 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.089 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:13:53.089 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.039784+0000", 2024-08-19T14:13:53.089 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00083278900000000001 2024-08-19T14:13:53.090 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.090 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.090 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:13:53.090 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.174204+0000", 2024-08-19T14:13:53.090 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.134420768 2024-08-19T14:13:53.090 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.090 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.090 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:13:53.090 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.174261+0000", 2024-08-19T14:13:53.090 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6947000000000002e-05 2024-08-19T14:13:53.090 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.090 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.091 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:13:53.091 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.174303+0000", 2024-08-19T14:13:53.091 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1709000000000003e-05 2024-08-19T14:13:53.091 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:53.091 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:13:53.091 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:53.091 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.091 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.091 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:63245 2.0 2:0a3678e6:::benchmark_data_smithi045_46415_object63244:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-08-19T14:13:53.091 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:21.051568+0000", 2024-08-19T14:13:53.091 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 31.962885108999998, 2024-08-19T14:13:53.091 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.12567051500000001, 2024-08-19T14:13:53.091 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:13:53.092 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:13:53.092 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:13:53.092 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:13:53.092 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:13:53.092 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:13:53.092 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 63245 2024-08-19T14:13:53.092 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.092 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:13:53.092 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.092 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:13:53.092 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.051568+0000", 2024-08-19T14:13:53.092 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:53.092 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.093 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.093 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:13:53.093 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.051568+0000", 2024-08-19T14:13:53.093 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:53.093 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.093 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.093 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:13:53.093 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.051569+0000", 2024-08-19T14:13:53.093 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2190000000000001e-06 2024-08-19T14:13:53.093 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.093 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.093 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:13:53.094 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.051592+0000", 2024-08-19T14:13:53.094 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3391e-05 2024-08-19T14:13:53.094 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.094 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.094 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:13:53.094 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.051603+0000", 2024-08-19T14:13:53.094 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0879e-05 2024-08-19T14:13:53.094 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.094 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.094 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:13:53.094 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.051608+0000", 2024-08-19T14:13:53.094 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.933e-06 2024-08-19T14:13:53.094 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.094 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.095 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:13:53.095 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.052675+0000", 2024-08-19T14:13:53.095 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001066621 2024-08-19T14:13:53.095 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.095 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.095 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:13:53.095 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.052841+0000", 2024-08-19T14:13:53.095 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00016639700000000001 2024-08-19T14:13:53.095 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.095 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.095 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 9,11", 2024-08-19T14:13:53.095 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.053037+0000", 2024-08-19T14:13:53.095 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00019577200000000001 2024-08-19T14:13:53.095 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.095 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.096 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:13:53.096 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.059090+0000", 2024-08-19T14:13:53.096 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0060532290000000002 2024-08-19T14:13:53.096 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.096 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.096 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:13:53.096 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.059156+0000", 2024-08-19T14:13:53.096 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6221000000000002e-05 2024-08-19T14:13:53.096 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.096 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.096 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:13:53.096 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.176685+0000", 2024-08-19T14:13:53.096 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.11752820899999999 2024-08-19T14:13:53.097 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.097 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.097 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:13:53.097 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.176716+0000", 2024-08-19T14:13:53.097 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1223000000000002e-05 2024-08-19T14:13:53.097 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.097 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.097 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:13:53.097 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.177238+0000", 2024-08-19T14:13:53.097 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00052242100000000002 2024-08-19T14:13:53.097 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:53.097 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:13:53.097 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:53.098 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.098 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.098 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:63247 2.0 2:0918bab6:::benchmark_data_smithi045_46415_object63246:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-08-19T14:13:53.098 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:21.052340+0000", 2024-08-19T14:13:53.098 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 31.962113223999999, 2024-08-19T14:13:53.098 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.12561350199999999, 2024-08-19T14:13:53.098 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:13:53.098 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:13:53.098 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:13:53.098 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:13:53.098 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:13:53.098 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:13:53.098 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 63247 2024-08-19T14:13:53.098 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.099 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:13:53.099 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.099 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:13:53.099 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.052340+0000", 2024-08-19T14:13:53.099 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:53.099 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.099 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.099 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:13:53.099 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.052340+0000", 2024-08-19T14:13:53.099 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:53.099 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.099 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.099 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:13:53.100 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.052340+0000", 2024-08-19T14:13:53.100 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5099999999999999e-07 2024-08-19T14:13:53.100 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.100 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.100 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:13:53.100 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.052371+0000", 2024-08-19T14:13:53.100 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1038000000000003e-05 2024-08-19T14:13:53.100 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.100 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.100 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:13:53.100 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.052382+0000", 2024-08-19T14:13:53.100 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0638999999999999e-05 2024-08-19T14:13:53.100 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.100 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.100 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:13:53.101 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.052387+0000", 2024-08-19T14:13:53.101 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.164e-06 2024-08-19T14:13:53.101 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.101 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.101 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:13:53.101 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.054907+0000", 2024-08-19T14:13:53.101 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0025194649999999998 2024-08-19T14:13:53.101 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.101 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.101 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:13:53.101 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.055057+0000", 2024-08-19T14:13:53.101 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000150607 2024-08-19T14:13:53.101 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.101 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.102 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 9,11", 2024-08-19T14:13:53.102 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.055161+0000", 2024-08-19T14:13:53.102 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000103644 2024-08-19T14:13:53.102 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.102 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.102 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:13:53.102 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.059147+0000", 2024-08-19T14:13:53.102 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0039857929999999996 2024-08-19T14:13:53.102 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.102 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.102 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:13:53.102 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.059160+0000", 2024-08-19T14:13:53.102 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3441000000000001e-05 2024-08-19T14:13:53.102 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.103 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.103 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:13:53.103 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.177310+0000", 2024-08-19T14:13:53.103 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.118149858 2024-08-19T14:13:53.103 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.103 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.103 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:13:53.103 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.177368+0000", 2024-08-19T14:13:53.103 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7633000000000002e-05 2024-08-19T14:13:53.103 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.103 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.103 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:13:53.103 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.177953+0000", 2024-08-19T14:13:53.103 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00058546900000000003 2024-08-19T14:13:53.104 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:53.104 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:13:53.104 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:53.104 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.104 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.104 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:63250 2.8 2:1918e081:::benchmark_data_smithi045_46415_object63249:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-08-19T14:13:53.104 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:21.056840+0000", 2024-08-19T14:13:53.104 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 31.957612869999998, 2024-08-19T14:13:53.104 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.119880845, 2024-08-19T14:13:53.104 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:13:53.104 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:13:53.104 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:13:53.104 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:13:53.104 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:13:53.105 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:13:53.105 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 63250 2024-08-19T14:13:53.105 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.105 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:13:53.105 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.105 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:13:53.105 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.056840+0000", 2024-08-19T14:13:53.105 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:53.105 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.105 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.106 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:13:53.106 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.056840+0000", 2024-08-19T14:13:53.106 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:53.106 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.106 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.106 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:13:53.106 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.056841+0000", 2024-08-19T14:13:53.106 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4360000000000001e-06 2024-08-19T14:13:53.106 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.106 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.106 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:13:53.106 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.056905+0000", 2024-08-19T14:13:53.106 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3677999999999996e-05 2024-08-19T14:13:53.107 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.107 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.107 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:13:53.107 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.056919+0000", 2024-08-19T14:13:53.107 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3556e-05 2024-08-19T14:13:53.107 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.107 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.107 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:13:53.107 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.056926+0000", 2024-08-19T14:13:53.107 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3579999999999997e-06 2024-08-19T14:13:53.107 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.107 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.107 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:13:53.107 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.059133+0000", 2024-08-19T14:13:53.108 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0022067789999999999 2024-08-19T14:13:53.108 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.108 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.108 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:13:53.108 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.059259+0000", 2024-08-19T14:13:53.108 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012647799999999999 2024-08-19T14:13:53.108 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.108 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.108 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0,9", 2024-08-19T14:13:53.108 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.059411+0000", 2024-08-19T14:13:53.108 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00015167800000000001 2024-08-19T14:13:53.108 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.108 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.108 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:13:53.109 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.061433+0000", 2024-08-19T14:13:53.109 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0020222510000000001 2024-08-19T14:13:53.109 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.109 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.109 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:13:53.109 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.062698+0000", 2024-08-19T14:13:53.109 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0012643559999999999 2024-08-19T14:13:53.109 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.109 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.109 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:13:53.109 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.176669+0000", 2024-08-19T14:13:53.109 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.11397160200000001 2024-08-19T14:13:53.109 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.109 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.110 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:13:53.110 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.176701+0000", 2024-08-19T14:13:53.110 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1863000000000003e-05 2024-08-19T14:13:53.110 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.110 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.110 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:13:53.110 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.176721+0000", 2024-08-19T14:13:53.110 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9809999999999998e-05 2024-08-19T14:13:53.110 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:53.110 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:13:53.110 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:53.110 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.110 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.111 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:63261 2.8 2:1032f80d:::benchmark_data_smithi045_46415_object63260:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-08-19T14:13:53.111 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:21.072158+0000", 2024-08-19T14:13:53.111 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 31.942295045000002, 2024-08-19T14:13:53.111 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.105196468, 2024-08-19T14:13:53.111 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:13:53.111 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:13:53.111 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:13:53.111 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:13:53.111 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:13:53.111 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:13:53.111 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 63261 2024-08-19T14:13:53.111 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.111 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:13:53.111 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.112 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:13:53.112 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.072158+0000", 2024-08-19T14:13:53.112 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:53.112 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.112 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.112 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:13:53.112 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.072158+0000", 2024-08-19T14:13:53.112 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:53.112 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.112 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.112 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:13:53.112 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.072159+0000", 2024-08-19T14:13:53.112 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.308e-06 2024-08-19T14:13:53.112 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.112 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.113 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:13:53.113 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.072228+0000", 2024-08-19T14:13:53.113 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9029999999999995e-05 2024-08-19T14:13:53.113 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.113 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.113 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:13:53.113 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.072240+0000", 2024-08-19T14:13:53.113 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2298e-05 2024-08-19T14:13:53.114 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.114 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.114 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:13:53.114 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.072247+0000", 2024-08-19T14:13:53.114 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6039999999999998e-06 2024-08-19T14:13:53.114 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.114 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.114 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:13:53.114 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.072303+0000", 2024-08-19T14:13:53.114 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6255999999999998e-05 2024-08-19T14:13:53.114 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.114 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.114 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:13:53.114 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.072403+0000", 2024-08-19T14:13:53.115 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000100207 2024-08-19T14:13:53.115 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.115 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.115 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0,9", 2024-08-19T14:13:53.115 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.072498+0000", 2024-08-19T14:13:53.115 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.4909999999999998e-05 2024-08-19T14:13:53.115 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.115 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.115 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:13:53.115 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.074415+0000", 2024-08-19T14:13:53.115 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001916765 2024-08-19T14:13:53.115 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.115 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.115 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:13:53.116 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.074532+0000", 2024-08-19T14:13:53.116 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000117135 2024-08-19T14:13:53.116 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.116 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.116 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:13:53.116 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.177287+0000", 2024-08-19T14:13:53.116 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.102755103 2024-08-19T14:13:53.116 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.116 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.116 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:13:53.116 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.177319+0000", 2024-08-19T14:13:53.116 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1219e-05 2024-08-19T14:13:53.116 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.116 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.117 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:13:53.117 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.177354+0000", 2024-08-19T14:13:53.117 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5633000000000003e-05 2024-08-19T14:13:53.117 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:53.117 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:13:53.117 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:53.117 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.117 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.117 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:63263 2.8 2:11991efe:::benchmark_data_smithi045_46415_object63262:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-08-19T14:13:53.118 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:21.076464+0000", 2024-08-19T14:13:53.118 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 31.937989103, 2024-08-19T14:13:53.118 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.101010905, 2024-08-19T14:13:53.118 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:13:53.118 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:13:53.118 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:13:53.118 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:13:53.118 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:13:53.118 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:13:53.118 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 63263 2024-08-19T14:13:53.118 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.118 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:13:53.118 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.119 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:13:53.119 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.076464+0000", 2024-08-19T14:13:53.119 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:53.119 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.119 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.119 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:13:53.119 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.076464+0000", 2024-08-19T14:13:53.119 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:53.119 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.119 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.119 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:13:53.119 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.076465+0000", 2024-08-19T14:13:53.119 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4610000000000001e-06 2024-08-19T14:13:53.119 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.120 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.120 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:13:53.120 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.076521+0000", 2024-08-19T14:13:53.120 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5457000000000001e-05 2024-08-19T14:13:53.120 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.120 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.120 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:13:53.120 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.076534+0000", 2024-08-19T14:13:53.120 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3787e-05 2024-08-19T14:13:53.120 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.120 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.120 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:13:53.120 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.076541+0000", 2024-08-19T14:13:53.120 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9099999999999999e-06 2024-08-19T14:13:53.121 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.121 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.121 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:13:53.121 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.076602+0000", 2024-08-19T14:13:53.121 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1074000000000005e-05 2024-08-19T14:13:53.121 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.121 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.121 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:13:53.121 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.076759+0000", 2024-08-19T14:13:53.121 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000157052 2024-08-19T14:13:53.121 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.121 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.121 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0,9", 2024-08-19T14:13:53.121 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.076885+0000", 2024-08-19T14:13:53.122 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012517100000000001 2024-08-19T14:13:53.122 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.122 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.122 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:13:53.122 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.078995+0000", 2024-08-19T14:13:53.122 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0021099880000000001 2024-08-19T14:13:53.122 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.122 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.122 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:13:53.122 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.080157+0000", 2024-08-19T14:13:53.122 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001162249 2024-08-19T14:13:53.122 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.123 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.123 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:13:53.123 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.177384+0000", 2024-08-19T14:13:53.123 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.097226707999999995 2024-08-19T14:13:53.123 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.123 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.123 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:13:53.123 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.177424+0000", 2024-08-19T14:13:53.123 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0599000000000002e-05 2024-08-19T14:13:53.123 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.123 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.123 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:13:53.123 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.177475+0000", 2024-08-19T14:13:53.123 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0448999999999998e-05 2024-08-19T14:13:53.123 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:53.124 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:13:53.124 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:53.124 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.124 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.124 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:63267 2.8 2:18ff9b6e:::benchmark_data_smithi045_46415_object63266:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-08-19T14:13:53.124 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:21.099988+0000", 2024-08-19T14:13:53.124 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 31.914464654, 2024-08-19T14:13:53.124 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.077616914999999995, 2024-08-19T14:13:53.124 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:13:53.124 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:13:53.124 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:13:53.124 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:13:53.124 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:13:53.124 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:13:53.125 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 63267 2024-08-19T14:13:53.125 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.125 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:13:53.125 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.125 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:13:53.125 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.099988+0000", 2024-08-19T14:13:53.125 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:53.125 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.125 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.125 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:13:53.125 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.099988+0000", 2024-08-19T14:13:53.125 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:53.125 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.125 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.126 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:13:53.126 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.099989+0000", 2024-08-19T14:13:53.126 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3060000000000001e-06 2024-08-19T14:13:53.126 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.126 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.126 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:13:53.126 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.100040+0000", 2024-08-19T14:13:53.126 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0207999999999998e-05 2024-08-19T14:13:53.126 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.126 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.126 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:13:53.126 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.100053+0000", 2024-08-19T14:13:53.126 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3171e-05 2024-08-19T14:13:53.127 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.127 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.127 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:13:53.127 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.100059+0000", 2024-08-19T14:13:53.127 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2149999999999998e-06 2024-08-19T14:13:53.127 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.127 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.127 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:13:53.127 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.100116+0000", 2024-08-19T14:13:53.127 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7142999999999998e-05 2024-08-19T14:13:53.127 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.127 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.127 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:13:53.127 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.100256+0000", 2024-08-19T14:13:53.128 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000139312 2024-08-19T14:13:53.128 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.128 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.128 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0,9", 2024-08-19T14:13:53.128 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.100400+0000", 2024-08-19T14:13:53.128 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000144566 2024-08-19T14:13:53.128 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.128 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.128 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:13:53.128 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.102529+0000", 2024-08-19T14:13:53.128 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0021289910000000002 2024-08-19T14:13:53.128 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.128 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.128 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:13:53.129 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.102561+0000", 2024-08-19T14:13:53.129 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2255000000000002e-05 2024-08-19T14:13:53.129 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.129 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.129 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:13:53.129 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.177515+0000", 2024-08-19T14:13:53.129 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.074953519999999996 2024-08-19T14:13:53.129 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.129 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.129 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:13:53.129 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.177570+0000", 2024-08-19T14:13:53.129 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5213999999999997e-05 2024-08-19T14:13:53.129 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.129 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.130 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:13:53.130 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.177605+0000", 2024-08-19T14:13:53.130 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5014000000000001e-05 2024-08-19T14:13:53.130 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:53.130 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:13:53.130 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:53.130 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.130 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.130 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:88725 2.8 2:164326b7:::benchmark_data_smithi045_46415_object88724:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-08-19T14:13:53.130 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:43.278957+0000", 2024-08-19T14:13:53.130 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 9.7354956280000007, 2024-08-19T14:13:53.130 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.055099461000000002, 2024-08-19T14:13:53.130 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:13:53.130 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:13:53.131 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:13:53.131 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:13:53.131 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:13:53.131 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:13:53.131 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 88725 2024-08-19T14:13:53.131 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.131 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:13:53.131 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.131 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:13:53.131 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:43.278957+0000", 2024-08-19T14:13:53.131 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:53.131 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.131 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.132 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:13:53.132 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:43.278957+0000", 2024-08-19T14:13:53.132 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:53.132 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.132 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.132 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:13:53.132 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:43.278959+0000", 2024-08-19T14:13:53.132 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5850000000000001e-06 2024-08-19T14:13:53.132 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.132 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.132 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:13:53.132 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:43.279015+0000", 2024-08-19T14:13:53.132 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6405e-05 2024-08-19T14:13:53.132 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.132 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.133 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:13:53.133 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:43.279028+0000", 2024-08-19T14:13:53.133 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2455e-05 2024-08-19T14:13:53.133 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.133 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.133 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:13:53.133 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:43.279035+0000", 2024-08-19T14:13:53.133 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1049999999999997e-06 2024-08-19T14:13:53.133 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.133 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.133 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:13:53.133 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:43.279113+0000", 2024-08-19T14:13:53.133 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7776999999999998e-05 2024-08-19T14:13:53.134 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.134 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.134 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:13:53.134 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:43.279218+0000", 2024-08-19T14:13:53.134 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00010556999999999999 2024-08-19T14:13:53.134 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.134 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.134 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0,9", 2024-08-19T14:13:53.134 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:43.279337+0000", 2024-08-19T14:13:53.134 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000118669 2024-08-19T14:13:53.134 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.134 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.134 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:13:53.134 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:43.283366+0000", 2024-08-19T14:13:53.135 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0040290159999999998 2024-08-19T14:13:53.135 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.135 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.135 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:13:53.135 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:43.285939+0000", 2024-08-19T14:13:53.135 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.002573667 2024-08-19T14:13:53.135 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.135 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.135 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:13:53.135 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:43.333941+0000", 2024-08-19T14:13:53.135 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.048001782 2024-08-19T14:13:53.135 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.135 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.135 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:13:53.135 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:43.334009+0000", 2024-08-19T14:13:53.136 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7484999999999995e-05 2024-08-19T14:13:53.136 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.136 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.136 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:13:53.136 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:43.334057+0000", 2024-08-19T14:13:53.136 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7945000000000003e-05 2024-08-19T14:13:53.136 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:53.136 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:13:53.136 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:53.136 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.136 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.136 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:88726 2.0 2:03ce9a49:::benchmark_data_smithi045_46415_object88725:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-08-19T14:13:53.136 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:43.279252+0000", 2024-08-19T14:13:53.136 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 9.7352012890000008, 2024-08-19T14:13:53.137 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.053784698999999998, 2024-08-19T14:13:53.137 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:13:53.137 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:13:53.137 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:13:53.137 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:13:53.137 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:13:53.137 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:13:53.137 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 88726 2024-08-19T14:13:53.137 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.137 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:13:53.137 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.137 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:13:53.137 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:43.279252+0000", 2024-08-19T14:13:53.137 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:53.137 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.137 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.138 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:13:53.138 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:43.279252+0000", 2024-08-19T14:13:53.138 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:53.138 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.138 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.138 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:13:53.138 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:43.279252+0000", 2024-08-19T14:13:53.138 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7100000000000002e-07 2024-08-19T14:13:53.138 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.138 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.138 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:13:53.138 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:43.279307+0000", 2024-08-19T14:13:53.138 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4783e-05 2024-08-19T14:13:53.139 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.139 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.139 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:13:53.139 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:43.279316+0000", 2024-08-19T14:13:53.139 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.7239999999999998e-06 2024-08-19T14:13:53.139 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.139 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.139 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:13:53.139 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:43.279320+0000", 2024-08-19T14:13:53.139 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3660000000000002e-06 2024-08-19T14:13:53.139 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.139 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.139 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:13:53.139 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:43.279362+0000", 2024-08-19T14:13:53.140 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1535999999999997e-05 2024-08-19T14:13:53.140 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.140 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.140 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:13:53.140 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:43.279494+0000", 2024-08-19T14:13:53.140 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013241500000000001 2024-08-19T14:13:53.140 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.140 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.140 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 9,11", 2024-08-19T14:13:53.140 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:43.279624+0000", 2024-08-19T14:13:53.140 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000129973 2024-08-19T14:13:53.140 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.140 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.140 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:13:53.141 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:43.283362+0000", 2024-08-19T14:13:53.141 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.003738154 2024-08-19T14:13:53.141 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.141 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.141 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:13:53.141 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:43.285953+0000", 2024-08-19T14:13:53.141 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0025910289999999999 2024-08-19T14:13:53.141 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.141 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.141 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:13:53.141 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:43.332906+0000", 2024-08-19T14:13:53.141 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.046952615000000003 2024-08-19T14:13:53.141 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.141 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.142 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:13:53.142 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:43.332974+0000", 2024-08-19T14:13:53.142 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8224000000000003e-05 2024-08-19T14:13:53.142 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.142 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.142 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:13:53.142 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:43.333036+0000", 2024-08-19T14:13:53.142 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2309000000000003e-05 2024-08-19T14:13:53.142 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:53.142 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:13:53.142 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:53.142 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.142 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.143 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:88727 2.0 2:009a1903:::benchmark_data_smithi045_46415_object88726:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-08-19T14:13:53.143 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:43.279448+0000", 2024-08-19T14:13:53.143 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 9.7350048850000004, 2024-08-19T14:13:53.143 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.054620123, 2024-08-19T14:13:53.143 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:13:53.143 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:13:53.143 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:13:53.143 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:13:53.143 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:13:53.143 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:13:53.143 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 88727 2024-08-19T14:13:53.143 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.143 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:13:53.143 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.144 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:13:53.144 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:43.279448+0000", 2024-08-19T14:13:53.144 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:53.144 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.144 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.144 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:13:53.144 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:43.279448+0000", 2024-08-19T14:13:53.144 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:53.144 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.144 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.144 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:13:53.144 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:43.279449+0000", 2024-08-19T14:13:53.144 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.8999999999999995e-07 2024-08-19T14:13:53.144 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.144 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.145 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:13:53.145 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:43.279494+0000", 2024-08-19T14:13:53.145 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5262999999999997e-05 2024-08-19T14:13:53.145 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.145 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.145 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:13:53.145 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:43.279503+0000", 2024-08-19T14:13:53.145 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.4639999999999995e-06 2024-08-19T14:13:53.145 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.145 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.145 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:13:53.145 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:43.279510+0000", 2024-08-19T14:13:53.145 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6930000000000001e-06 2024-08-19T14:13:53.145 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.146 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.146 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:13:53.146 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:43.281648+0000", 2024-08-19T14:13:53.146 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0021375729999999998 2024-08-19T14:13:53.146 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.146 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.146 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:13:53.146 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:43.281762+0000", 2024-08-19T14:13:53.146 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00011413600000000001 2024-08-19T14:13:53.146 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.146 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.146 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 9,11", 2024-08-19T14:13:53.146 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:43.281866+0000", 2024-08-19T14:13:53.146 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00010443899999999999 2024-08-19T14:13:53.147 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.147 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.147 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:13:53.147 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:43.285916+0000", 2024-08-19T14:13:53.147 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0040493200000000003 2024-08-19T14:13:53.147 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.147 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.147 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:13:53.147 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:43.330825+0000", 2024-08-19T14:13:53.147 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.044909141999999999 2024-08-19T14:13:53.147 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.147 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.147 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:13:53.147 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:43.333939+0000", 2024-08-19T14:13:53.148 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.003114052 2024-08-19T14:13:53.148 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.148 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.148 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:13:53.148 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:43.334006+0000", 2024-08-19T14:13:53.148 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6743999999999997e-05 2024-08-19T14:13:53.148 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.148 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.148 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:13:53.148 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:43.334068+0000", 2024-08-19T14:13:53.148 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2507000000000003e-05 2024-08-19T14:13:53.148 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:53.148 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:13:53.148 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:53.149 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.149 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.149 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:88739 2.8 2:11241d74:::benchmark_data_smithi045_46415_object88738:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-08-19T14:13:53.149 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:43.285721+0000", 2024-08-19T14:13:53.149 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 9.7287318250000006, 2024-08-19T14:13:53.149 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.054152862000000003, 2024-08-19T14:13:53.149 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:13:53.149 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:13:53.149 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:13:53.149 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:13:53.149 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:13:53.149 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:13:53.149 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 88739 2024-08-19T14:13:53.149 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.150 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:13:53.150 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.150 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:13:53.150 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:43.285721+0000", 2024-08-19T14:13:53.150 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:53.150 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.150 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.150 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:13:53.150 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:43.285721+0000", 2024-08-19T14:13:53.150 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:13:53.150 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.150 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.150 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:13:53.150 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:43.285723+0000", 2024-08-19T14:13:53.151 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.705e-06 2024-08-19T14:13:53.151 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.151 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.151 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:13:53.151 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:43.285822+0000", 2024-08-19T14:13:53.151 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.9084000000000002e-05 2024-08-19T14:13:53.151 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.151 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.151 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:13:53.151 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:43.285837+0000", 2024-08-19T14:13:53.151 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4975e-05 2024-08-19T14:13:53.151 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.151 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.151 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:13:53.151 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:43.285844+0000", 2024-08-19T14:13:53.152 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0700000000000001e-06 2024-08-19T14:13:53.152 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.152 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.152 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:13:53.152 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:43.285969+0000", 2024-08-19T14:13:53.152 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000125103 2024-08-19T14:13:53.152 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.152 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.152 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:13:53.152 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:43.286096+0000", 2024-08-19T14:13:53.152 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012718899999999999 2024-08-19T14:13:53.152 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.152 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.152 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0,9", 2024-08-19T14:13:53.153 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:43.331008+0000", 2024-08-19T14:13:53.153 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.044911735000000001 2024-08-19T14:13:53.153 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.153 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.153 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:13:53.153 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:43.337152+0000", 2024-08-19T14:13:53.153 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0061438860000000003 2024-08-19T14:13:53.153 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.153 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.153 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:13:53.153 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:43.337210+0000", 2024-08-19T14:13:53.153 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8525e-05 2024-08-19T14:13:53.153 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.153 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.154 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:13:53.154 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:43.339696+0000", 2024-08-19T14:13:53.154 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0024855099999999998 2024-08-19T14:13:53.154 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.154 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.154 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:13:53.154 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:43.339757+0000", 2024-08-19T14:13:53.154 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0749999999999999e-05 2024-08-19T14:13:53.154 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:13:53.154 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:13:53.154 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:13:53.154 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:43.339874+0000", 2024-08-19T14:13:53.154 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00011733 2024-08-19T14:13:53.154 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:53.154 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:13:53.155 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:53.155 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:13:53.155 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:13:53.155 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:13:53.179 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:13:53.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:53.234 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:53.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:13:53.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:52 smithi045 ceph-mon[20143]: pgmap v358: 27 pgs: 3 active+recovering+undersized+remapped, 24 active+clean; 5.8 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 75 MiB/s wr, 1.20k op/s; 6678/286716 objects misplaced (2.329%); 18 MiB/s, 294 objects/s recovering 2024-08-19T14:13:53.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:52 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/78279134' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:13:53.335 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:13:53.363 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:53.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:53.364 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:13:53.465 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:13:53.492 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:53.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:53.493 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:13:53.594 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:13:53.619 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:53.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:53.620 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:13:53.721 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:13:53.745 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:53.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:53.746 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:13:53.835 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 64 16 100230 100214 97.8569 67.75 0.00933365 0.0102144 2024-08-19T14:13:53.847 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:13:53.870 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:13:53.886 INFO:teuthology.orchestra.run.smithi031.stderr:reweighted osd.1 to 1 (10000) 2024-08-19T14:13:53.905 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:53.906 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:53 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:13:53.906 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:53 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:13:53.907 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:53 smithi031 ceph-mon[16336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:13:53.907 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:53 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/78279134' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:13:53.907 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:53 smithi031 ceph-mon[16336]: osdmap e140: 12 total, 12 up, 10 in 2024-08-19T14:13:53.907 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:53 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1679188511' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-08-19T14:13:53.907 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:53 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-08-19T14:13:53.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:53.908 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:13:54.009 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:13:54.052 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:54.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:54.053 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:13:54.154 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:13:54.162 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:53 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:13:54.162 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:53 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:13:54.162 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:53 smithi032 ceph-mon[20004]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:13:54.162 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:53 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/78279134' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:13:54.162 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:53 smithi032 ceph-mon[20004]: osdmap e140: 12 total, 12 up, 10 in 2024-08-19T14:13:54.163 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:53 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1679188511' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-08-19T14:13:54.163 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:53 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-08-19T14:13:54.164 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-08-19T14:13:54.189 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:53 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:13:54.189 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:53 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:13:54.189 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:53 smithi045 ceph-mon[20143]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:13:54.190 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:53 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/78279134' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:13:54.190 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:53 smithi045 ceph-mon[20143]: osdmap e140: 12 total, 12 up, 10 in 2024-08-19T14:13:54.190 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:53 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1679188511' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-08-19T14:13:54.190 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:53 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-08-19T14:13:54.207 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:54.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:54.207 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:13:54.309 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:13:54.333 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:54.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:54.334 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:13:54.435 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:13:54.459 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:54.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:54.460 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:13:54.515 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:13:54.561 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:13:54.587 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:54.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:54.588 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:13:54.689 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:13:54.716 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:54.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:54.717 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:13:54.766 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:13:54.767 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:13:54.814 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:13:54.814 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:13:54.818 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:13:54.835 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 65 16 101382 101366 97.459 72 0.00626216 0.010256 2024-08-19T14:13:54.855 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:54.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:54.856 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:13:54.856 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:13:54.857 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:13:54.895 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:13:54.895 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:13:54.933 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:13:54.933 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:13:54.957 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:13:54.965 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:13:54.965 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:13:54.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:54.993 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:54.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:13:54.998 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:13:54.999 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:13:55.037 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:13:55.037 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:13:55.074 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:13:55.074 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:13:55.094 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:13:55.111 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:54 smithi031 ceph-mon[16336]: pgmap v360: 27 pgs: 3 active+recovering+undersized+remapped, 24 active+clean; 6.0 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 68 MiB/s wr, 1.09k op/s; 6410/292674 objects misplaced (2.190%); 11 MiB/s, 173 objects/s recovering 2024-08-19T14:13:55.111 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:54 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-08-19T14:13:55.111 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:54 smithi031 ceph-mon[16336]: osdmap e141: 12 total, 12 up, 10 in 2024-08-19T14:13:55.111 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:54 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2145937790' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:13:55.111 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:13:55.112 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:13:55.118 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:55.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:55.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:13:55.144 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:13:55.144 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:13:55.177 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:13:55.177 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:13:55.220 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:13:55.227 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:54 smithi045 ceph-mon[20143]: pgmap v360: 27 pgs: 3 active+recovering+undersized+remapped, 24 active+clean; 6.0 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 68 MiB/s wr, 1.09k op/s; 6410/292674 objects misplaced (2.190%); 11 MiB/s, 173 objects/s recovering 2024-08-19T14:13:55.228 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:54 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-08-19T14:13:55.228 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:54 smithi045 ceph-mon[20143]: osdmap e141: 12 total, 12 up, 10 in 2024-08-19T14:13:55.228 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:54 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2145937790' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:13:55.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:54 smithi032 ceph-mon[20004]: pgmap v360: 27 pgs: 3 active+recovering+undersized+remapped, 24 active+clean; 6.0 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 68 MiB/s wr, 1.09k op/s; 6410/292674 objects misplaced (2.190%); 11 MiB/s, 173 objects/s recovering 2024-08-19T14:13:55.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:54 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-08-19T14:13:55.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:54 smithi032 ceph-mon[20004]: osdmap e141: 12 total, 12 up, 10 in 2024-08-19T14:13:55.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:54 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2145937790' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:13:55.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:55.275 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:55.275 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:13:55.276 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd reweight 2 1 2024-08-19T14:13:55.343 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:13:55.343 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:13:55.376 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:13:55.412 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:55.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:55.414 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:13:55.514 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:13:55.538 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:55.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:55.539 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:13:55.640 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:13:55.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:55.667 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:55.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:13:55.702 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-19T14:13:55.768 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:13:55.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:55.790 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:55.790 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:13:55.835 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 66 16 102413 102397 96.9586 64.4375 0.0159719 0.0103093 2024-08-19T14:13:55.891 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:13:55.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:55.936 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:55.936 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:13:56.037 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:13:56.062 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:56.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:56.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:13:56.147 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-08-19T14:13:56.164 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:13:56.214 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:56.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:56.215 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:13:56.316 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:13:56.342 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:56.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:56.343 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:13:56.444 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:13:56.470 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:56.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:56.471 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:13:56.488 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:13:56.573 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:13:56.609 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:56.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:56.610 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:13:56.711 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:13:56.742 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:56.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:56.743 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:13:56.835 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 67 16 103338 103322 96.3743 57.8125 0.0218141 0.0103692 2024-08-19T14:13:56.844 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:13:56.870 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:56.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:56.871 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:13:56.971 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:13:56.995 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:56.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:56.996 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:13:57.032 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:13:57.032 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:13:57.097 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:13:57.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:57.123 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:57.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:13:57.224 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:13:57.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:56 smithi032 ceph-mon[20004]: pgmap v362: 27 pgs: 3 active+recovering+undersized+remapped, 24 active+clean; 6.1 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 65 MiB/s wr, 1.04k op/s; 6410/297861 objects misplaced (2.152%); 11 MiB/s, 173 objects/s recovering 2024-08-19T14:13:57.267 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:57.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:57.268 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:13:57.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:56 smithi045 ceph-mon[20143]: pgmap v362: 27 pgs: 3 active+recovering+undersized+remapped, 24 active+clean; 6.1 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 65 MiB/s wr, 1.04k op/s; 6410/297861 objects misplaced (2.152%); 11 MiB/s, 173 objects/s recovering 2024-08-19T14:13:57.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:56 smithi031 ceph-mon[16336]: pgmap v362: 27 pgs: 3 active+recovering+undersized+remapped, 24 active+clean; 6.1 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 65 MiB/s wr, 1.04k op/s; 6410/297861 objects misplaced (2.152%); 11 MiB/s, 173 objects/s recovering 2024-08-19T14:13:57.369 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:13:57.404 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:57.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:57.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:13:57.507 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:13:57.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:57.538 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:57.538 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:13:57.640 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:13:57.662 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:57.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:57.663 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:13:57.677 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-08-19T14:13:57.765 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:13:57.790 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:57.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:57.792 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:13:57.835 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 68 16 104362 104346 95.8981 64 0.00368437 0.0104115 2024-08-19T14:13:57.893 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:13:57.916 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:57.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:57.917 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:13:57.955 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:13:58.018 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:13:58.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:58.039 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:58.039 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:13:58.140 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:13:58.162 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:58.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:58.163 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:13:58.264 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:13:58.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:58.286 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:58.286 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:13:58.387 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:13:58.410 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:58.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:58.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:13:58.424 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:13:58.425 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:13:58.512 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:13:58.532 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:58.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:58.533 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:13:58.634 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:13:58.664 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:58.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:58.665 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:13:58.766 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:13:58.792 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:58.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:58.793 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:13:58.835 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 69 16 104631 104615 94.7519 16.8125 0.0257685 0.0105491 2024-08-19T14:13:58.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:58 smithi031 ceph-mon[16336]: osdmap e142: 12 total, 12 up, 10 in 2024-08-19T14:13:58.894 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:13:58.923 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:58.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:58.924 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:13:59.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:58 smithi032 ceph-mon[20004]: osdmap e142: 12 total, 12 up, 10 in 2024-08-19T14:13:59.024 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:13:59.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:58 smithi045 ceph-mon[20143]: osdmap e142: 12 total, 12 up, 10 in 2024-08-19T14:13:59.058 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:59.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:59.060 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:13:59.071 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-08-19T14:13:59.161 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:13:59.186 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:59.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:59.187 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:13:59.288 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:13:59.310 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:13:59.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:59.311 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:13:59.320 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:13:59.333 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:13:59.412 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:13:59.448 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:59.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:59.449 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:13:59.453 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:13:59.550 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:13:59.581 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:59.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:59.582 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:13:59.683 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:13:59.722 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:13:59.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:59.723 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:13:59.760 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:59 smithi031 ceph-mon[16336]: pgmap v364: 27 pgs: 3 active+recovering+undersized+remapped, 24 active+clean; 6.2 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 67 MiB/s wr, 1.07k op/s; 4976/305952 objects misplaced (1.626%); 16 MiB/s, 262 objects/s recovering 2024-08-19T14:13:59.760 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:13:59 smithi031 ceph-mon[16336]: osdmap e143: 12 total, 12 up, 10 in 2024-08-19T14:13:59.824 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:13:59.835 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 70 16 105742 105726 94.3902 69.4375 0.0194721 0.0105895 2024-08-19T14:13:59.852 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:59.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:59.853 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:13:59.857 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:59 smithi045 ceph-mon[20143]: pgmap v364: 27 pgs: 3 active+recovering+undersized+remapped, 24 active+clean; 6.2 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 67 MiB/s wr, 1.07k op/s; 4976/305952 objects misplaced (1.626%); 16 MiB/s, 262 objects/s recovering 2024-08-19T14:13:59.857 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:13:59 smithi045 ceph-mon[20143]: osdmap e143: 12 total, 12 up, 10 in 2024-08-19T14:13:59.954 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:13:59.981 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:13:59.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:13:59.982 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:13:59.999 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:13:59.999 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:14:00.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:59 smithi032 ceph-mon[20004]: pgmap v364: 27 pgs: 3 active+recovering+undersized+remapped, 24 active+clean; 6.2 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 67 MiB/s wr, 1.07k op/s; 4976/305952 objects misplaced (1.626%); 16 MiB/s, 262 objects/s recovering 2024-08-19T14:14:00.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:13:59 smithi032 ceph-mon[20004]: osdmap e143: 12 total, 12 up, 10 in 2024-08-19T14:14:00.083 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:14:00.123 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:00.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:00.124 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:14:00.225 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:14:00.250 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:00.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:00.251 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:14:00.352 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:14:00.394 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:00.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:00.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:14:00.496 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:14:00.525 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:00.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:00.526 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:14:00.619 INFO:teuthology.orchestra.run.smithi031.stderr:reweighted osd.2 to 1 (10000) 2024-08-19T14:14:00.627 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:14:00.650 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:00.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:00.651 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:14:00.652 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-08-19T14:14:00.752 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:14:00.775 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:00.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:00.776 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:14:00.835 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 71 16 106637 106621 93.8486 55.9375 0.0108852 0.0106507 2024-08-19T14:14:00.877 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:14:00.911 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:00.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:00.912 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:14:00.978 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:14:00.978 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:00 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3017371813' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-08-19T14:14:00.978 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:00 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-08-19T14:14:01.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:00 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3017371813' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-08-19T14:14:01.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:00 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-08-19T14:14:01.012 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:14:01.044 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:01.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:01.045 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:14:01.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:00 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3017371813' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-08-19T14:14:01.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:00 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-08-19T14:14:01.147 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:14:01.174 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:01.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:01.175 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:14:01.276 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:14:01.284 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:14:01.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:01.307 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:01.308 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:14:01.360 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd reweight 3 1 2024-08-19T14:14:01.408 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:14:01.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:01.433 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:01.433 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:14:01.507 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:14:01.508 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:14:01.534 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:14:01.558 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:01.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:01.559 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:14:01.647 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is unset 2024-08-19T14:14:01.659 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:14:01.682 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:01.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:01.683 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:14:01.784 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:14:01.811 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:01.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:01.813 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:14:01.835 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 72 15 107723 107708 93.4886 67.9375 0.00571837 0.0106893 2024-08-19T14:14:01.913 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:14:01.936 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:01.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:01.937 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:14:02.000 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:01 smithi031 ceph-mon[16336]: pgmap v366: 27 pgs: 3 active+recovering+undersized+remapped, 24 active+clean; 6.3 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 64 MiB/s wr, 1.03k op/s; 4976/311169 objects misplaced (1.599%); 14 MiB/s, 217 objects/s recovering 2024-08-19T14:14:02.001 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:01 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-08-19T14:14:02.001 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:01 smithi031 ceph-mon[16336]: osdmap e144: 12 total, 12 up, 10 in 2024-08-19T14:14:02.001 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:01 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3202927884' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:14:02.001 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:01 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:14:02.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:01 smithi032 ceph-mon[20004]: pgmap v366: 27 pgs: 3 active+recovering+undersized+remapped, 24 active+clean; 6.3 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 64 MiB/s wr, 1.03k op/s; 4976/311169 objects misplaced (1.599%); 14 MiB/s, 217 objects/s recovering 2024-08-19T14:14:02.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:01 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-08-19T14:14:02.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:01 smithi032 ceph-mon[20004]: osdmap e144: 12 total, 12 up, 10 in 2024-08-19T14:14:02.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:01 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3202927884' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:14:02.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:01 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:14:02.038 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:14:02.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:01 smithi045 ceph-mon[20143]: pgmap v366: 27 pgs: 3 active+recovering+undersized+remapped, 24 active+clean; 6.3 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 64 MiB/s wr, 1.03k op/s; 4976/311169 objects misplaced (1.599%); 14 MiB/s, 217 objects/s recovering 2024-08-19T14:14:02.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:01 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-08-19T14:14:02.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:01 smithi045 ceph-mon[20143]: osdmap e144: 12 total, 12 up, 10 in 2024-08-19T14:14:02.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:01 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3202927884' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:14:02.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:01 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:14:02.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:02.081 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:02.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:14:02.182 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:14:02.204 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:02.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:02.205 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:14:02.241 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-08-19T14:14:02.306 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:14:02.353 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:02.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:02.354 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:14:02.374 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:14:02.438 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:14:02.454 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:14:02.480 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:02.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:02.481 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:14:02.582 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:14:02.607 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:02.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:02.608 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:14:02.634 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:14:02.708 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:14:02.758 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:02.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:02.759 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:14:02.828 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:02 smithi031 ceph-mon[16336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:14:02.828 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:02 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:14:02.828 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:02 smithi031 ceph-mon[16336]: osdmap e145: 12 total, 12 up, 10 in 2024-08-19T14:14:02.828 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:02 smithi031 ceph-mon[16336]: 2.3 scrub starts 2024-08-19T14:14:02.836 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 73 16 108781 108765 93.1128 66.0625 0.0130063 0.010734 2024-08-19T14:14:02.860 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:14:02.885 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:02.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:02.886 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:14:02.906 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:02 smithi045 ceph-mon[20143]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:14:02.906 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:02 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:14:02.906 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:02 smithi045 ceph-mon[20143]: osdmap e145: 12 total, 12 up, 10 in 2024-08-19T14:14:02.907 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:02 smithi045 ceph-mon[20143]: 2.3 scrub starts 2024-08-19T14:14:02.987 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:14:03.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:02 smithi032 ceph-mon[20004]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:14:03.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:02 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:14:03.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:02 smithi032 ceph-mon[20004]: osdmap e145: 12 total, 12 up, 10 in 2024-08-19T14:14:03.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:02 smithi032 ceph-mon[20004]: 2.3 scrub starts 2024-08-19T14:14:03.009 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:03.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:03.010 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:14:03.111 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:14:03.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:03.169 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:03.169 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:14:03.170 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:14:03.170 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:14:03.270 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:14:03.298 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:03.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:03.299 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:14:03.323 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:14:03.323 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:14:03.358 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:14:03.358 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:14:03.390 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:14:03.391 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:14:03.400 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:14:03.422 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:14:03.423 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:14:03.429 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:03.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:03.430 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:14:03.454 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:14:03.455 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:14:03.500 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:14:03.500 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:14:03.531 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:14:03.544 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:14:03.544 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:14:03.584 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:03.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:03.585 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:14:03.587 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:14:03.587 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:14:03.686 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:14:03.714 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:03.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:03.715 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:14:03.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:03 smithi045 ceph-mon[20143]: pgmap v369: 27 pgs: 2 active+recovering+undersized+remapped, 25 active+clean; 6.4 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 70 MiB/s wr, 1.12k op/s; 4490/316947 objects misplaced (1.417%); 8.3 MiB/s, 132 objects/s recovering 2024-08-19T14:14:03.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:03 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1551023852' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:14:03.816 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:14:03.824 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:14:03.824 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:14:03.835 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 74 16 109732 109716 92.6577 59.4375 0.00363394 0.0107798 2024-08-19T14:14:03.849 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-08-19T14:14:03.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:03.850 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:03.851 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:14:03.856 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:14:03.856 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:14:03.891 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:14:03.892 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:14:03.914 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:14:03.914 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:14:03.914 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:03 smithi031 ceph-mon[16336]: pgmap v369: 27 pgs: 2 active+recovering+undersized+remapped, 25 active+clean; 6.4 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 70 MiB/s wr, 1.12k op/s; 4490/316947 objects misplaced (1.417%); 8.3 MiB/s, 132 objects/s recovering 2024-08-19T14:14:03.914 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:03 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1551023852' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:14:03.952 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:14:03.973 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:03.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:03.974 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:14:04.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:03 smithi032 ceph-mon[20004]: pgmap v369: 27 pgs: 2 active+recovering+undersized+remapped, 25 active+clean; 6.4 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 70 MiB/s wr, 1.12k op/s; 4490/316947 objects misplaced (1.417%); 8.3 MiB/s, 132 objects/s recovering 2024-08-19T14:14:04.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:03 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1551023852' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:14:04.075 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:14:04.096 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:04.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:04.097 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:14:04.115 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:14:04.198 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:14:04.223 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:04.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:04.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:14:04.325 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:14:04.348 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:04.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:04.349 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:14:04.355 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-19T14:14:04.450 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:14:04.473 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:04.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:04.474 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:14:04.575 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:14:04.596 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:04.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:04.597 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:14:04.660 INFO:teuthology.orchestra.run.smithi031.stderr:reweighted osd.3 to 1 (10000) 2024-08-19T14:14:04.678 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:04 smithi031 ceph-mon[16336]: osdmap e146: 12 total, 12 up, 10 in 2024-08-19T14:14:04.678 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:04 smithi031 ceph-mon[16336]: 2.3 scrub ok 2024-08-19T14:14:04.678 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:04 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1590314209' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-08-19T14:14:04.688 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:14:04.688 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [], 2024-08-19T14:14:04.688 INFO:teuthology.orchestra.run.smithi045.stdout: "num_ops": 0 2024-08-19T14:14:04.689 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:14:04.698 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:14:04.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:04.737 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:04.738 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:14:04.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:04 smithi045 ceph-mon[20143]: osdmap e146: 12 total, 12 up, 10 in 2024-08-19T14:14:04.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:04 smithi045 ceph-mon[20143]: 2.3 scrub ok 2024-08-19T14:14:04.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:04 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1590314209' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-08-19T14:14:04.836 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 75 16 109921 109905 91.5798 11.8125 0.0337998 0.0109133 2024-08-19T14:14:04.838 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:14:04.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:04.876 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:04.876 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:14:04.977 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:14:05.003 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:05.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:05.004 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:14:05.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:04 smithi032 ceph-mon[20004]: osdmap e146: 12 total, 12 up, 10 in 2024-08-19T14:14:05.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:04 smithi032 ceph-mon[20004]: 2.3 scrub ok 2024-08-19T14:14:05.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:04 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1590314209' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-08-19T14:14:05.106 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:14:05.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:05.148 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:05.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:14:05.250 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:14:05.280 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:05.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:05.281 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:14:05.382 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:14:05.418 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:05.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:05.419 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:14:05.504 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-08-19T14:14:05.520 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:14:05.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:05.587 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:05.587 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:14:05.689 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:14:05.713 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:05.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:05.714 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:14:05.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:05 smithi045 ceph-mon[20143]: pgmap v371: 27 pgs: 2 active+recovering+undersized+remapped, 25 active+clean; 6.5 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 53 MiB/s wr, 848 op/s; 4020/321888 objects misplaced (1.249%); 9.2 MiB/s, 147 objects/s recovering 2024-08-19T14:14:05.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:05 smithi045 ceph-mon[20143]: 2.9 scrub starts 2024-08-19T14:14:05.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:05 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1590314209' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-08-19T14:14:05.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:05 smithi045 ceph-mon[20143]: osdmap e147: 12 total, 12 up, 10 in 2024-08-19T14:14:05.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:05 smithi045 ceph-mon[20143]: 2.b scrub starts 2024-08-19T14:14:05.815 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:14:05.836 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 76 16 110996 110980 91.2586 67.1875 0.00954354 0.0109531 2024-08-19T14:14:05.851 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:05.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:05.852 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:14:05.888 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:14:05.954 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:14:05.986 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:05.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:05.987 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:14:06.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:05 smithi032 ceph-mon[20004]: pgmap v371: 27 pgs: 2 active+recovering+undersized+remapped, 25 active+clean; 6.5 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 53 MiB/s wr, 848 op/s; 4020/321888 objects misplaced (1.249%); 9.2 MiB/s, 147 objects/s recovering 2024-08-19T14:14:06.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:05 smithi032 ceph-mon[20004]: 2.9 scrub starts 2024-08-19T14:14:06.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:05 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1590314209' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-08-19T14:14:06.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:05 smithi032 ceph-mon[20004]: osdmap e147: 12 total, 12 up, 10 in 2024-08-19T14:14:06.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:05 smithi032 ceph-mon[20004]: 2.b scrub starts 2024-08-19T14:14:06.087 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:14:06.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:05 smithi031 ceph-mon[16336]: pgmap v371: 27 pgs: 2 active+recovering+undersized+remapped, 25 active+clean; 6.5 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 53 MiB/s wr, 848 op/s; 4020/321888 objects misplaced (1.249%); 9.2 MiB/s, 147 objects/s recovering 2024-08-19T14:14:06.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:05 smithi031 ceph-mon[16336]: 2.9 scrub starts 2024-08-19T14:14:06.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:05 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1590314209' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-08-19T14:14:06.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:05 smithi031 ceph-mon[16336]: osdmap e147: 12 total, 12 up, 10 in 2024-08-19T14:14:06.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:05 smithi031 ceph-mon[16336]: 2.b scrub starts 2024-08-19T14:14:06.119 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:06.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:06.120 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:14:06.221 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:14:06.243 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:06.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:06.244 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:14:06.345 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:14:06.370 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:06.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:06.371 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:14:06.473 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:14:06.474 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:14:06.474 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [], 2024-08-19T14:14:06.475 INFO:teuthology.orchestra.run.smithi045.stdout: "complaint_time": 30, 2024-08-19T14:14:06.475 INFO:teuthology.orchestra.run.smithi045.stdout: "num_blocked_ops": 0 2024-08-19T14:14:06.475 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:14:06.496 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:06.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:06.497 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:14:06.598 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:14:06.627 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:06.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:06.628 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:14:06.729 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:14:06.751 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:06.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:06.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:14:06.836 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 77 16 112173 112157 91.0287 73.5625 0.00656935 0.0109814 2024-08-19T14:14:06.853 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:14:06.878 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:06.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:06.879 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:14:06.979 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:14:07.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:06 smithi032 ceph-mon[20004]: 2.9 scrub ok 2024-08-19T14:14:07.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:06 smithi032 ceph-mon[20004]: 1.6 scrub starts 2024-08-19T14:14:07.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:06 smithi032 ceph-mon[20004]: 1.6 scrub ok 2024-08-19T14:14:07.016 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:07.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:07.017 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:14:07.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:06 smithi045 ceph-mon[20143]: 2.9 scrub ok 2024-08-19T14:14:07.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:06 smithi045 ceph-mon[20143]: 1.6 scrub starts 2024-08-19T14:14:07.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:06 smithi045 ceph-mon[20143]: 1.6 scrub ok 2024-08-19T14:14:07.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:06 smithi031 ceph-mon[16336]: 2.9 scrub ok 2024-08-19T14:14:07.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:06 smithi031 ceph-mon[16336]: 1.6 scrub starts 2024-08-19T14:14:07.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:06 smithi031 ceph-mon[16336]: 1.6 scrub ok 2024-08-19T14:14:07.118 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:14:07.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:07.151 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:07.151 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:14:07.198 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-08-19T14:14:07.252 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:14:07.277 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:07.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:07.278 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:14:07.379 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:14:07.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:07.417 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:07.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:14:07.512 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:14:07.518 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:14:07.539 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:07.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:07.540 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:14:07.641 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:14:07.696 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:07.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:07.697 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:14:07.798 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:14:07.836 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:07.836 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 78 16 113175 113159 90.6644 62.625 0.00539351 0.0110251 2024-08-19T14:14:07.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:07.837 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:14:07.900 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:07 smithi045 ceph-mon[20143]: pgmap v373: 27 pgs: 2 active+recovering+undersized+remapped, 25 active+clean; 6.6 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 53 MiB/s wr, 849 op/s; 4020/325881 objects misplaced (1.234%); 9.2 MiB/s, 147 objects/s recovering 2024-08-19T14:14:07.900 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:07 smithi045 ceph-mon[20143]: 2.b scrub ok 2024-08-19T14:14:07.900 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:07 smithi045 ceph-mon[20143]: 1.1 scrub starts 2024-08-19T14:14:07.900 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:07 smithi045 ceph-mon[20143]: 1.1 scrub ok 2024-08-19T14:14:07.938 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:14:07.970 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:07.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:07.972 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:14:08.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:07 smithi032 ceph-mon[20004]: pgmap v373: 27 pgs: 2 active+recovering+undersized+remapped, 25 active+clean; 6.6 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 53 MiB/s wr, 849 op/s; 4020/325881 objects misplaced (1.234%); 9.2 MiB/s, 147 objects/s recovering 2024-08-19T14:14:08.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:07 smithi032 ceph-mon[20004]: 2.b scrub ok 2024-08-19T14:14:08.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:07 smithi032 ceph-mon[20004]: 1.1 scrub starts 2024-08-19T14:14:08.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:07 smithi032 ceph-mon[20004]: 1.1 scrub ok 2024-08-19T14:14:08.046 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:14:08.046 INFO:teuthology.orchestra.run.smithi045.stdout: "size": 20, 2024-08-19T14:14:08.046 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 600, 2024-08-19T14:14:08.046 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [ 2024-08-19T14:14:08.046 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.046 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:14 2.4 e106/102 2:291e9767:::benchmark_data_smithi045_46415_object13:head v 106'1, mlcod=0'0)", 2024-08-19T14:14:08.046 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:12:49.835194+0000", 2024-08-19T14:14:08.046 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 78.206521702000003, 2024-08-19T14:14:08.046 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.045178087999999998, 2024-08-19T14:14:08.046 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:14:08.047 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:14:08.047 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:14:08.047 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:14:08.047 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.047 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:14:08.047 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:49.835194+0000", 2024-08-19T14:14:08.047 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:14:08.047 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.047 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.047 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:14:08.047 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:49.835194+0000", 2024-08-19T14:14:08.047 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:14:08.047 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.047 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.048 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:14:08.048 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:49.835194+0000", 2024-08-19T14:14:08.048 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.7500000000000001e-07 2024-08-19T14:14:08.048 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.048 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.048 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:14:08.048 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:49.879644+0000", 2024-08-19T14:14:08.048 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.044449610000000001 2024-08-19T14:14:08.048 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.048 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.048 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:14:08.048 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:49.879665+0000", 2024-08-19T14:14:08.048 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1458999999999999e-05 2024-08-19T14:14:08.048 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.049 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.049 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:14:08.049 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:49.879706+0000", 2024-08-19T14:14:08.049 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.0748999999999999e-05 2024-08-19T14:14:08.049 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.049 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.049 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:14:08.049 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:49.879782+0000", 2024-08-19T14:14:08.049 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.6445000000000004e-05 2024-08-19T14:14:08.049 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.049 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.049 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:14:08.049 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:49.879807+0000", 2024-08-19T14:14:08.049 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.4765000000000001e-05 2024-08-19T14:14:08.050 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.050 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.050 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:14:08.050 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:49.880329+0000", 2024-08-19T14:14:08.050 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00052147599999999999 2024-08-19T14:14:08.050 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.050 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.050 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:14:08.050 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:49.880372+0000", 2024-08-19T14:14:08.050 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.3309000000000002e-05 2024-08-19T14:14:08.050 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:14:08.050 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:14:08.050 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:14:08.051 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.051 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.051 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:12156 2.4 e107/102 2:28b3607b:::benchmark_data_smithi045_46415_object12155:head v 107'793, mlcod=107'791)", 2024-08-19T14:14:08.051 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:12:53.026834+0000", 2024-08-19T14:14:08.051 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 75.014881236999997, 2024-08-19T14:14:08.051 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0078985359999999994, 2024-08-19T14:14:08.051 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:14:08.051 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:14:08.051 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:14:08.051 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:14:08.051 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.051 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:14:08.051 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:53.026834+0000", 2024-08-19T14:14:08.051 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:14:08.051 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.052 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.052 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:14:08.052 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:53.026834+0000", 2024-08-19T14:14:08.052 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:14:08.052 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.052 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.052 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:14:08.052 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:53.026835+0000", 2024-08-19T14:14:08.052 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.5199999999999996e-07 2024-08-19T14:14:08.052 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.052 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.052 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:14:08.052 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:53.034134+0000", 2024-08-19T14:14:08.053 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0072991369999999998 2024-08-19T14:14:08.053 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.053 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.053 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:14:08.053 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:53.034144+0000", 2024-08-19T14:14:08.053 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0370000000000001e-05 2024-08-19T14:14:08.053 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.053 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.053 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:14:08.053 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:53.034149+0000", 2024-08-19T14:14:08.053 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.8130000000000003e-06 2024-08-19T14:14:08.053 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.053 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.053 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:14:08.054 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:53.034196+0000", 2024-08-19T14:14:08.054 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.7159e-05 2024-08-19T14:14:08.054 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.054 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.054 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:14:08.054 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:53.034211+0000", 2024-08-19T14:14:08.054 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4429e-05 2024-08-19T14:14:08.054 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.054 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.054 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:14:08.054 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:53.034701+0000", 2024-08-19T14:14:08.054 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00049043000000000003 2024-08-19T14:14:08.054 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.054 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.055 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:14:08.055 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:53.034733+0000", 2024-08-19T14:14:08.055 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.1545999999999999e-05 2024-08-19T14:14:08.055 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:14:08.055 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:14:08.055 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:14:08.055 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.055 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.055 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:38741 2.4 e112/102 2:2ddcb16b:::benchmark_data_smithi045_46415_object38740:head v 112'2448, mlcod=112'2444)", 2024-08-19T14:14:08.055 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:02.435487+0000", 2024-08-19T14:14:08.055 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 65.606228810000005, 2024-08-19T14:14:08.055 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0060987159999999997, 2024-08-19T14:14:08.055 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:14:08.055 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:14:08.056 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:14:08.056 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:14:08.056 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.056 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:14:08.056 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.435487+0000", 2024-08-19T14:14:08.056 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:14:08.056 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.056 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.056 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:14:08.056 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.435487+0000", 2024-08-19T14:14:08.056 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:14:08.056 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.056 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.056 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:14:08.056 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.435487+0000", 2024-08-19T14:14:08.057 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2699999999999999e-07 2024-08-19T14:14:08.057 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.057 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.057 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:14:08.057 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.435514+0000", 2024-08-19T14:14:08.057 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.6457000000000001e-05 2024-08-19T14:14:08.057 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.057 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.057 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:14:08.057 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.435524+0000", 2024-08-19T14:14:08.057 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0299000000000001e-05 2024-08-19T14:14:08.057 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.057 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.057 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:14:08.058 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.435531+0000", 2024-08-19T14:14:08.058 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.6529999999999997e-06 2024-08-19T14:14:08.058 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.058 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.058 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:14:08.058 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.435603+0000", 2024-08-19T14:14:08.058 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.1963000000000002e-05 2024-08-19T14:14:08.058 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.058 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.058 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:14:08.058 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.435622+0000", 2024-08-19T14:14:08.058 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8926000000000001e-05 2024-08-19T14:14:08.058 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.058 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.059 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:14:08.059 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.441541+0000", 2024-08-19T14:14:08.059 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0059191479999999999 2024-08-19T14:14:08.059 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.059 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.059 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:14:08.059 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.441585+0000", 2024-08-19T14:14:08.059 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.4743000000000002e-05 2024-08-19T14:14:08.059 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:14:08.059 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:14:08.059 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:14:08.059 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.059 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.059 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:40033 2.0 e113/102 2:0512d183:::benchmark_data_smithi045_46415_object40032:head v 113'2436, mlcod=113'2434)", 2024-08-19T14:14:08.060 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:02.895855+0000", 2024-08-19T14:14:08.060 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 65.145860116999998, 2024-08-19T14:14:08.060 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0095757759999999994, 2024-08-19T14:14:08.060 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:14:08.060 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:14:08.060 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:14:08.060 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:14:08.060 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.060 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:14:08.060 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.895855+0000", 2024-08-19T14:14:08.060 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:14:08.060 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.060 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.061 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:14:08.061 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.895855+0000", 2024-08-19T14:14:08.061 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:14:08.061 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.061 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.061 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:14:08.061 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.895856+0000", 2024-08-19T14:14:08.061 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.2099999999999995e-07 2024-08-19T14:14:08.061 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.061 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.061 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:14:08.062 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.895922+0000", 2024-08-19T14:14:08.062 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.5188e-05 2024-08-19T14:14:08.062 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.062 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.062 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:14:08.062 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.895935+0000", 2024-08-19T14:14:08.062 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3601000000000001e-05 2024-08-19T14:14:08.062 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.062 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.062 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:14:08.062 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.895943+0000", 2024-08-19T14:14:08.062 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.9170000000000006e-06 2024-08-19T14:14:08.062 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.063 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.063 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:14:08.063 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.896012+0000", 2024-08-19T14:14:08.063 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.9053000000000004e-05 2024-08-19T14:14:08.063 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.063 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.063 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:14:08.063 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.896035+0000", 2024-08-19T14:14:08.063 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2943999999999999e-05 2024-08-19T14:14:08.063 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.063 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.063 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:14:08.063 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.905344+0000", 2024-08-19T14:14:08.063 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0093086450000000008 2024-08-19T14:14:08.064 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.064 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.064 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:14:08.064 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.905431+0000", 2024-08-19T14:14:08.064 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.7507000000000001e-05 2024-08-19T14:14:08.064 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:14:08.064 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:14:08.064 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:14:08.064 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.064 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.064 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:40035 2.0 e113/102 2:0960ca99:::benchmark_data_smithi045_46415_object40034:head v 113'2437, mlcod=113'2434)", 2024-08-19T14:14:08.064 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:02.897909+0000", 2024-08-19T14:14:08.064 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 65.143807038000006, 2024-08-19T14:14:08.064 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0076402179999999998, 2024-08-19T14:14:08.064 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:14:08.065 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:14:08.065 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:14:08.065 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:14:08.065 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.065 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:14:08.065 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.897909+0000", 2024-08-19T14:14:08.065 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:14:08.065 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.065 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.065 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:14:08.065 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.897909+0000", 2024-08-19T14:14:08.065 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:14:08.065 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.065 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.066 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:14:08.066 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.897909+0000", 2024-08-19T14:14:08.066 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.33e-07 2024-08-19T14:14:08.066 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.066 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.066 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:14:08.066 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.897950+0000", 2024-08-19T14:14:08.066 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.0683000000000003e-05 2024-08-19T14:14:08.066 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.066 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.066 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:14:08.066 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.897966+0000", 2024-08-19T14:14:08.066 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6209999999999999e-05 2024-08-19T14:14:08.066 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.067 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.067 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:14:08.067 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.897974+0000", 2024-08-19T14:14:08.067 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4569999999999999e-06 2024-08-19T14:14:08.067 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.067 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.067 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:14:08.067 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.898035+0000", 2024-08-19T14:14:08.067 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.1033000000000003e-05 2024-08-19T14:14:08.067 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.067 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.067 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:14:08.067 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.898068+0000", 2024-08-19T14:14:08.067 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.3593000000000002e-05 2024-08-19T14:14:08.068 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.068 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.068 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:14:08.068 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.905435+0000", 2024-08-19T14:14:08.068 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0073662830000000004 2024-08-19T14:14:08.068 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.068 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.068 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:14:08.068 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.905549+0000", 2024-08-19T14:14:08.068 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000114026 2024-08-19T14:14:08.068 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:14:08.068 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:14:08.068 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:14:08.068 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.068 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.069 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:40069 2.0 e113/102 2:03e60c7a:::benchmark_data_smithi045_46415_object40068:head v 113'2440, mlcod=113'2436)", 2024-08-19T14:14:08.069 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:02.908487+0000", 2024-08-19T14:14:08.069 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 65.133228265, 2024-08-19T14:14:08.069 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0062491439999999999, 2024-08-19T14:14:08.069 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:14:08.069 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:14:08.069 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:14:08.069 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:14:08.069 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.069 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:14:08.069 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.908487+0000", 2024-08-19T14:14:08.069 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:14:08.069 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.069 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.070 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:14:08.070 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.908487+0000", 2024-08-19T14:14:08.070 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:14:08.070 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.070 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.070 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:14:08.070 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.908488+0000", 2024-08-19T14:14:08.070 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.8999999999999995e-07 2024-08-19T14:14:08.070 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.070 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.070 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:14:08.070 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.908517+0000", 2024-08-19T14:14:08.070 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.8643999999999999e-05 2024-08-19T14:14:08.070 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.071 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.071 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:14:08.071 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.908533+0000", 2024-08-19T14:14:08.071 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5770999999999999e-05 2024-08-19T14:14:08.071 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.071 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.071 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:14:08.071 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.908540+0000", 2024-08-19T14:14:08.071 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4000000000000003e-06 2024-08-19T14:14:08.071 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.071 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.071 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:14:08.071 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.908593+0000", 2024-08-19T14:14:08.071 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.3146999999999998e-05 2024-08-19T14:14:08.071 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.072 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.072 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:14:08.072 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.908614+0000", 2024-08-19T14:14:08.072 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0900999999999999e-05 2024-08-19T14:14:08.072 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.072 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.072 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:14:08.072 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.914692+0000", 2024-08-19T14:14:08.072 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0060776399999999996 2024-08-19T14:14:08.072 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.072 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.072 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:14:08.072 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.914736+0000", 2024-08-19T14:14:08.072 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.4851000000000001e-05 2024-08-19T14:14:08.072 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:14:08.073 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:14:08.073 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:14:08.073 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.073 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.073 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:43484 2.4 e114/102 2:285d10cd:::benchmark_data_smithi045_46415_object43483:head v 114'2732, mlcod=114'2729)", 2024-08-19T14:14:08.073 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:04.598813+0000", 2024-08-19T14:14:08.073 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 63.44290238, 2024-08-19T14:14:08.073 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0062693649999999998, 2024-08-19T14:14:08.073 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:14:08.073 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:14:08.073 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:14:08.073 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:14:08.073 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.073 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:14:08.074 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:14:08.075 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:04.598813+0000", 2024-08-19T14:14:08.075 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:14:08.075 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.075 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.075 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:14:08.075 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:04.598813+0000", 2024-08-19T14:14:08.075 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:14:08.075 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.076 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.076 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:14:08.076 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:04.598814+0000", 2024-08-19T14:14:08.076 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.9699999999999994e-07 2024-08-19T14:14:08.076 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.076 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.076 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:14:08.076 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:04.598850+0000", 2024-08-19T14:14:08.076 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.5865000000000003e-05 2024-08-19T14:14:08.076 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.076 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.076 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:14:08.077 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:04.598861+0000", 2024-08-19T14:14:08.077 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1284000000000001e-05 2024-08-19T14:14:08.077 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.077 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.077 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:14:08.077 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:04.598870+0000", 2024-08-19T14:14:08.077 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.9479999999999997e-06 2024-08-19T14:14:08.077 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.077 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.077 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:14:08.077 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:04.598935+0000", 2024-08-19T14:14:08.077 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.4364999999999998e-05 2024-08-19T14:14:08.077 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.077 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.078 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:14:08.078 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:04.598955+0000", 2024-08-19T14:14:08.078 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0849999999999999e-05 2024-08-19T14:14:08.078 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.078 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.078 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:14:08.078 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:04.604957+0000", 2024-08-19T14:14:08.078 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0060020000000000004 2024-08-19T14:14:08.078 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.078 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.078 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:14:08.078 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:04.605083+0000", 2024-08-19T14:14:08.078 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012505599999999999 2024-08-19T14:14:08.079 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:14:08.079 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:14:08.079 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:14:08.079 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.079 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.079 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:46495 2.4 e116/102 2:27583565:::benchmark_data_smithi045_46415_object46494:head v 116'2919, mlcod=116'2914)", 2024-08-19T14:14:08.079 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:06.037606+0000", 2024-08-19T14:14:08.079 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 62.004109984000003, 2024-08-19T14:14:08.079 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0079993879999999996, 2024-08-19T14:14:08.079 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:14:08.079 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:14:08.079 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:14:08.079 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:14:08.079 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.080 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:14:08.080 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:06.037606+0000", 2024-08-19T14:14:08.080 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:14:08.080 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.080 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.080 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:14:08.080 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:06.037606+0000", 2024-08-19T14:14:08.080 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:14:08.080 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.080 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.080 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:14:08.080 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:06.037606+0000", 2024-08-19T14:14:08.080 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2300000000000001e-07 2024-08-19T14:14:08.080 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.081 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.081 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:14:08.081 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:06.043150+0000", 2024-08-19T14:14:08.081 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0055434869999999997 2024-08-19T14:14:08.081 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.081 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.081 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:14:08.081 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:06.043160+0000", 2024-08-19T14:14:08.081 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0088999999999999e-05 2024-08-19T14:14:08.081 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.081 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.081 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:14:08.081 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:06.043167+0000", 2024-08-19T14:14:08.081 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.3309999999999998e-06 2024-08-19T14:14:08.082 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.082 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.082 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:14:08.082 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:06.043225+0000", 2024-08-19T14:14:08.082 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.8062000000000001e-05 2024-08-19T14:14:08.082 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.082 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.082 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:14:08.082 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:06.043246+0000", 2024-08-19T14:14:08.082 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1316000000000001e-05 2024-08-19T14:14:08.082 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.082 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.082 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:14:08.082 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:06.045577+0000", 2024-08-19T14:14:08.083 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0023308890000000001 2024-08-19T14:14:08.083 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.083 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.083 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:14:08.083 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:06.045605+0000", 2024-08-19T14:14:08.083 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.7691e-05 2024-08-19T14:14:08.083 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:14:08.083 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:14:08.083 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:14:08.083 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.083 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.083 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:48912 2.7 e117/102 2:e6ffd652:::benchmark_data_smithi045_46415_object48911:head v 117'2997, mlcod=117'2992)", 2024-08-19T14:14:08.083 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:07.468103+0000", 2024-08-19T14:14:08.084 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 60.573613053000003, 2024-08-19T14:14:08.084 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0061113349999999999, 2024-08-19T14:14:08.084 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:14:08.084 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:14:08.084 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:14:08.084 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:14:08.084 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.084 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:14:08.084 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468103+0000", 2024-08-19T14:14:08.084 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:14:08.084 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.084 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.084 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:14:08.084 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468103+0000", 2024-08-19T14:14:08.085 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:14:08.085 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.085 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.085 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:14:08.085 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468103+0000", 2024-08-19T14:14:08.085 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.8000000000000002e-07 2024-08-19T14:14:08.085 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.085 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.085 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:14:08.085 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468136+0000", 2024-08-19T14:14:08.085 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.3411000000000002e-05 2024-08-19T14:14:08.085 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.085 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.085 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:14:08.086 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468143+0000", 2024-08-19T14:14:08.086 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.5869999999999998e-06 2024-08-19T14:14:08.086 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.086 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.086 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:14:08.086 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468146+0000", 2024-08-19T14:14:08.086 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.304e-06 2024-08-19T14:14:08.086 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.086 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.086 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:14:08.086 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.470630+0000", 2024-08-19T14:14:08.086 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0024838009999999999 2024-08-19T14:14:08.086 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.086 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.087 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:14:08.087 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.470643+0000", 2024-08-19T14:14:08.087 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2964e-05 2024-08-19T14:14:08.087 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.087 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.087 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:14:08.087 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.474173+0000", 2024-08-19T14:14:08.087 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.003530423 2024-08-19T14:14:08.087 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.087 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.087 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:14:08.087 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.474214+0000", 2024-08-19T14:14:08.087 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.0565000000000002e-05 2024-08-19T14:14:08.087 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:14:08.087 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:14:08.088 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:14:08.088 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.088 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.088 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:48971 2.7 e117/102 2:ecf6a539:::benchmark_data_smithi045_46415_object48970:head v 117'2999, mlcod=117'2992)", 2024-08-19T14:14:08.088 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:07.468206+0000", 2024-08-19T14:14:08.088 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 60.573509119000001, 2024-08-19T14:14:08.088 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0076013820000000003, 2024-08-19T14:14:08.088 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:14:08.088 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:14:08.088 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:14:08.088 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:14:08.088 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.088 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:14:08.089 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468206+0000", 2024-08-19T14:14:08.089 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:14:08.089 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.089 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.089 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:14:08.089 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468206+0000", 2024-08-19T14:14:08.089 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:14:08.089 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.089 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.089 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:14:08.089 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468207+0000", 2024-08-19T14:14:08.089 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6899999999999999e-07 2024-08-19T14:14:08.089 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.090 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.090 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:14:08.090 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468248+0000", 2024-08-19T14:14:08.090 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.0985999999999997e-05 2024-08-19T14:14:08.090 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.090 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.090 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:14:08.090 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468254+0000", 2024-08-19T14:14:08.090 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.9769999999999997e-06 2024-08-19T14:14:08.090 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.090 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.090 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:14:08.090 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468257+0000", 2024-08-19T14:14:08.090 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.0259999999999999e-06 2024-08-19T14:14:08.091 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.091 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.091 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:14:08.091 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.471956+0000", 2024-08-19T14:14:08.091 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0036996059999999998 2024-08-19T14:14:08.091 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.091 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.091 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:14:08.091 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.471974+0000", 2024-08-19T14:14:08.091 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8219999999999998e-05 2024-08-19T14:14:08.091 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.091 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.091 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:14:08.091 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.475774+0000", 2024-08-19T14:14:08.092 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0037994729999999998 2024-08-19T14:14:08.092 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.092 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.092 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:14:08.092 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.475808+0000", 2024-08-19T14:14:08.092 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.3924999999999998e-05 2024-08-19T14:14:08.092 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:14:08.092 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:14:08.092 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:14:08.092 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.092 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.092 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:48978 2.7 e117/102 2:e11d5fd6:::benchmark_data_smithi045_46415_object48977:head v 117'3000, mlcod=117'2992)", 2024-08-19T14:14:08.092 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:07.468262+0000", 2024-08-19T14:14:08.092 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 60.57345342, 2024-08-19T14:14:08.093 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0075744890000000002, 2024-08-19T14:14:08.093 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:14:08.093 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:14:08.093 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:14:08.093 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:14:08.093 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.093 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:14:08.093 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468262+0000", 2024-08-19T14:14:08.093 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:14:08.093 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.093 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.093 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:14:08.093 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468262+0000", 2024-08-19T14:14:08.094 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:14:08.094 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.094 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.094 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:14:08.094 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468262+0000", 2024-08-19T14:14:08.094 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.04e-07 2024-08-19T14:14:08.094 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.094 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.094 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:14:08.094 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468309+0000", 2024-08-19T14:14:08.094 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.6913999999999998e-05 2024-08-19T14:14:08.094 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.094 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.095 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:14:08.095 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468315+0000", 2024-08-19T14:14:08.095 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7139999999999998e-06 2024-08-19T14:14:08.095 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.095 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.095 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:14:08.095 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468318+0000", 2024-08-19T14:14:08.095 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.117e-06 2024-08-19T14:14:08.095 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.095 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.095 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:14:08.095 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.472643+0000", 2024-08-19T14:14:08.095 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0043251950000000004 2024-08-19T14:14:08.095 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.096 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.096 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:14:08.096 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.472657+0000", 2024-08-19T14:14:08.096 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3478e-05 2024-08-19T14:14:08.096 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.096 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.096 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:14:08.096 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.475812+0000", 2024-08-19T14:14:08.096 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0031556259999999999 2024-08-19T14:14:08.096 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.096 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.096 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:14:08.096 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.475837+0000", 2024-08-19T14:14:08.096 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.4241000000000001e-05 2024-08-19T14:14:08.097 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:14:08.097 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:14:08.097 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:14:08.097 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.097 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.097 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:48995 2.7 e117/102 2:e01c8116:::benchmark_data_smithi045_46415_object48994:head v 117'3001, mlcod=117'2992)", 2024-08-19T14:14:08.097 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:07.468324+0000", 2024-08-19T14:14:08.097 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 60.573391893, 2024-08-19T14:14:08.097 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0090529970000000001, 2024-08-19T14:14:08.097 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:14:08.097 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:14:08.097 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:14:08.097 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:14:08.098 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.098 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:14:08.098 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468324+0000", 2024-08-19T14:14:08.098 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:14:08.098 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.098 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.098 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:14:08.098 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468324+0000", 2024-08-19T14:14:08.098 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:14:08.098 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.098 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.098 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:14:08.098 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468324+0000", 2024-08-19T14:14:08.099 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.97e-07 2024-08-19T14:14:08.099 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.099 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.099 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:14:08.099 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468388+0000", 2024-08-19T14:14:08.099 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.4523999999999994e-05 2024-08-19T14:14:08.099 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.099 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.099 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:14:08.099 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468395+0000", 2024-08-19T14:14:08.099 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.579e-06 2024-08-19T14:14:08.099 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.099 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.099 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:14:08.100 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468398+0000", 2024-08-19T14:14:08.100 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.2780000000000002e-06 2024-08-19T14:14:08.100 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.100 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.100 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:14:08.100 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.473325+0000", 2024-08-19T14:14:08.100 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00492626 2024-08-19T14:14:08.100 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.100 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.100 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:14:08.100 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.473343+0000", 2024-08-19T14:14:08.100 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8788999999999999e-05 2024-08-19T14:14:08.100 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.100 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.101 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:14:08.101 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.477310+0000", 2024-08-19T14:14:08.101 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0039662819999999998 2024-08-19T14:14:08.101 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.101 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.101 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:14:08.101 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.477377+0000", 2024-08-19T14:14:08.101 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.7088000000000005e-05 2024-08-19T14:14:08.101 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:14:08.101 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:14:08.101 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:14:08.101 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.101 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.101 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:48996 2.7 e117/102 2:e8abf67b:::benchmark_data_smithi045_46415_object48995:head v 117'3002, mlcod=117'2992)", 2024-08-19T14:14:08.102 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:07.468406+0000", 2024-08-19T14:14:08.102 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 60.573309885, 2024-08-19T14:14:08.102 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0090294899999999994, 2024-08-19T14:14:08.102 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:14:08.102 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:14:08.102 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:14:08.102 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:14:08.102 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.102 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:14:08.102 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468406+0000", 2024-08-19T14:14:08.102 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:14:08.102 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.102 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.103 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:14:08.103 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468406+0000", 2024-08-19T14:14:08.103 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:14:08.103 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.103 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.103 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:14:08.103 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468406+0000", 2024-08-19T14:14:08.103 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0900000000000001e-07 2024-08-19T14:14:08.103 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.103 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.103 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:14:08.103 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468447+0000", 2024-08-19T14:14:08.103 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.0992000000000003e-05 2024-08-19T14:14:08.103 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.104 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.104 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:14:08.104 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468452+0000", 2024-08-19T14:14:08.104 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2179999999999998e-06 2024-08-19T14:14:08.104 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.104 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.104 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:14:08.104 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468455+0000", 2024-08-19T14:14:08.104 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.796e-06 2024-08-19T14:14:08.104 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.104 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.104 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:14:08.104 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.474288+0000", 2024-08-19T14:14:08.105 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0058327689999999998 2024-08-19T14:14:08.105 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.105 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.105 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:14:08.105 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.474307+0000", 2024-08-19T14:14:08.105 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9031000000000001e-05 2024-08-19T14:14:08.105 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.105 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.105 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:14:08.105 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.477383+0000", 2024-08-19T14:14:08.105 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0030758790000000001 2024-08-19T14:14:08.105 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.105 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.106 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:14:08.106 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.477435+0000", 2024-08-19T14:14:08.106 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2596000000000003e-05 2024-08-19T14:14:08.106 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:14:08.106 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:14:08.106 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:14:08.106 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.106 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.106 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:49005 2.7 e117/102 2:e58e73b6:::benchmark_data_smithi045_46415_object49004:head v 117'3003, mlcod=117'2992)", 2024-08-19T14:14:08.106 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:07.468460+0000", 2024-08-19T14:14:08.106 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 60.573255783999997, 2024-08-19T14:14:08.106 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0098532910000000001, 2024-08-19T14:14:08.106 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:14:08.107 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:14:08.107 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:14:08.107 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:14:08.107 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.107 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:14:08.107 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468460+0000", 2024-08-19T14:14:08.107 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:14:08.107 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.107 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.107 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:14:08.107 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468460+0000", 2024-08-19T14:14:08.107 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:14:08.107 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.107 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.108 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:14:08.108 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468460+0000", 2024-08-19T14:14:08.108 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5900000000000001e-07 2024-08-19T14:14:08.108 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.108 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.108 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:14:08.108 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468518+0000", 2024-08-19T14:14:08.108 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.8536000000000003e-05 2024-08-19T14:14:08.108 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.108 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.108 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:14:08.108 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468524+0000", 2024-08-19T14:14:08.108 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.8039999999999998e-06 2024-08-19T14:14:08.108 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.109 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.109 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:14:08.109 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468527+0000", 2024-08-19T14:14:08.109 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.004e-06 2024-08-19T14:14:08.109 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.109 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.109 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:14:08.109 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.475085+0000", 2024-08-19T14:14:08.109 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0065574090000000002 2024-08-19T14:14:08.109 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.109 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.109 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:14:08.109 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.475106+0000", 2024-08-19T14:14:08.110 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1056999999999998e-05 2024-08-19T14:14:08.110 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.110 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.110 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:14:08.110 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.478267+0000", 2024-08-19T14:14:08.110 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0031614049999999999 2024-08-19T14:14:08.110 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.110 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.110 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:14:08.110 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.478313+0000", 2024-08-19T14:14:08.110 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.5917000000000001e-05 2024-08-19T14:14:08.110 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:14:08.110 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:14:08.110 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:14:08.111 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.111 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.111 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:49015 2.7 e117/102 2:e4d8c3e1:::benchmark_data_smithi045_46415_object49014:head v 117'3004, mlcod=117'2992)", 2024-08-19T14:14:08.111 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:07.468533+0000", 2024-08-19T14:14:08.111 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 60.573182719999998, 2024-08-19T14:14:08.111 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0098176270000000006, 2024-08-19T14:14:08.111 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:14:08.111 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:14:08.111 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:14:08.111 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:14:08.111 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.111 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:14:08.111 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468533+0000", 2024-08-19T14:14:08.111 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:14:08.112 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.112 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.112 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:14:08.112 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468533+0000", 2024-08-19T14:14:08.112 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:14:08.112 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.112 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.112 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:14:08.112 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468533+0000", 2024-08-19T14:14:08.112 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2600000000000001e-07 2024-08-19T14:14:08.112 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.112 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.112 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:14:08.112 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468577+0000", 2024-08-19T14:14:08.113 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.3436000000000002e-05 2024-08-19T14:14:08.113 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.113 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.113 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:14:08.113 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468582+0000", 2024-08-19T14:14:08.113 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4399999999999996e-06 2024-08-19T14:14:08.113 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.113 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.113 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:14:08.113 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468585+0000", 2024-08-19T14:14:08.113 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.8619999999999999e-06 2024-08-19T14:14:08.113 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.113 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.113 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:14:08.114 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.475870+0000", 2024-08-19T14:14:08.114 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0072852919999999996 2024-08-19T14:14:08.114 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.114 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.114 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:14:08.114 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.475894+0000", 2024-08-19T14:14:08.114 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.3858000000000001e-05 2024-08-19T14:14:08.114 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.114 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.114 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:14:08.114 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.478319+0000", 2024-08-19T14:14:08.114 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0024247729999999999 2024-08-19T14:14:08.114 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.114 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.115 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:14:08.115 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.478350+0000", 2024-08-19T14:14:08.115 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.1739999999999998e-05 2024-08-19T14:14:08.115 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:14:08.115 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:14:08.115 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:14:08.115 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.115 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.115 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:49016 2.7 e117/102 2:e52959d5:::benchmark_data_smithi045_46415_object49015:head v 117'3005, mlcod=117'2992)", 2024-08-19T14:14:08.115 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:07.468590+0000", 2024-08-19T14:14:08.115 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 60.573125842000003, 2024-08-19T14:14:08.115 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.011903967999999999, 2024-08-19T14:14:08.115 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:14:08.116 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:14:08.116 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:14:08.116 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:14:08.116 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.116 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:14:08.116 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468590+0000", 2024-08-19T14:14:08.116 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:14:08.116 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.116 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.116 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:14:08.116 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468590+0000", 2024-08-19T14:14:08.116 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:14:08.116 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.116 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.117 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:14:08.117 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468590+0000", 2024-08-19T14:14:08.117 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.54e-07 2024-08-19T14:14:08.117 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.117 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.117 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:14:08.117 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468632+0000", 2024-08-19T14:14:08.117 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.2459999999999997e-05 2024-08-19T14:14:08.117 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.117 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.117 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:14:08.117 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468638+0000", 2024-08-19T14:14:08.117 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2410000000000003e-06 2024-08-19T14:14:08.117 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.119 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:07 smithi031 ceph-mon[16336]: pgmap v373: 27 pgs: 2 active+recovering+undersized+remapped, 25 active+clean; 6.6 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 53 MiB/s wr, 849 op/s; 4020/325881 objects misplaced (1.234%); 9.2 MiB/s, 147 objects/s recovering 2024-08-19T14:14:08.119 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:07 smithi031 ceph-mon[16336]: 2.b scrub ok 2024-08-19T14:14:08.119 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:07 smithi031 ceph-mon[16336]: 1.1 scrub starts 2024-08-19T14:14:08.119 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:07 smithi031 ceph-mon[16336]: 1.1 scrub ok 2024-08-19T14:14:08.119 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.119 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:14:08.119 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468641+0000", 2024-08-19T14:14:08.119 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.061e-06 2024-08-19T14:14:08.119 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.119 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.119 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:14:08.120 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.476571+0000", 2024-08-19T14:14:08.120 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0079302920000000002 2024-08-19T14:14:08.120 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.120 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.120 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:14:08.120 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.476593+0000", 2024-08-19T14:14:08.120 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2045e-05 2024-08-19T14:14:08.120 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.120 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.120 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:14:08.120 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.480447+0000", 2024-08-19T14:14:08.120 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.003853825 2024-08-19T14:14:08.120 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.121 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.121 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:14:08.121 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.480494+0000", 2024-08-19T14:14:08.121 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.689e-05 2024-08-19T14:14:08.121 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:14:08.121 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:14:08.121 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:14:08.121 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.121 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.121 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:49017 2.7 e117/102 2:eca28b17:::benchmark_data_smithi045_46415_object49016:head v 117'3006, mlcod=117'2992)", 2024-08-19T14:14:08.121 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:07.468646+0000", 2024-08-19T14:14:08.121 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 60.573069977999999, 2024-08-19T14:14:08.121 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.011882515999999999, 2024-08-19T14:14:08.122 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:14:08.122 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:14:08.122 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:14:08.122 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:14:08.122 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.122 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:14:08.122 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468646+0000", 2024-08-19T14:14:08.122 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:14:08.122 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.122 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.122 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:14:08.122 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468646+0000", 2024-08-19T14:14:08.122 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:14:08.122 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.123 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.123 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:14:08.123 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468646+0000", 2024-08-19T14:14:08.123 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5800000000000001e-07 2024-08-19T14:14:08.123 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.123 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.123 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:14:08.123 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468686+0000", 2024-08-19T14:14:08.123 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.0732999999999998e-05 2024-08-19T14:14:08.123 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.123 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.123 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:14:08.123 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468692+0000", 2024-08-19T14:14:08.123 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0860000000000001e-06 2024-08-19T14:14:08.124 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.124 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.124 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:14:08.124 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468694+0000", 2024-08-19T14:14:08.124 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.864e-06 2024-08-19T14:14:08.124 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.124 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.124 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:14:08.124 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.477463+0000", 2024-08-19T14:14:08.124 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0087685190000000007 2024-08-19T14:14:08.124 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.124 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.124 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:14:08.124 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.477483+0000", 2024-08-19T14:14:08.125 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0449999999999999e-05 2024-08-19T14:14:08.125 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.125 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.125 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:14:08.125 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.480501+0000", 2024-08-19T14:14:08.125 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.003017802 2024-08-19T14:14:08.125 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.125 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.125 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:14:08.125 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.480528+0000", 2024-08-19T14:14:08.125 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.6903999999999999e-05 2024-08-19T14:14:08.125 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:14:08.125 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:14:08.125 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:14:08.126 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.126 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.126 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:49027 2.7 e117/102 2:e96ce7ac:::benchmark_data_smithi045_46415_object49026:head v 117'3007, mlcod=117'2992)", 2024-08-19T14:14:08.126 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:07.468699+0000", 2024-08-19T14:14:08.126 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 60.573016566, 2024-08-19T14:14:08.126 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.011858521, 2024-08-19T14:14:08.126 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:14:08.126 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:14:08.126 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:14:08.126 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:14:08.126 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.126 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:14:08.126 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468699+0000", 2024-08-19T14:14:08.127 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:14:08.127 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.127 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.127 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:14:08.127 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468699+0000", 2024-08-19T14:14:08.127 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:14:08.127 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.127 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.127 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:14:08.127 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468699+0000", 2024-08-19T14:14:08.127 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5599999999999999e-07 2024-08-19T14:14:08.127 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.127 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.127 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:14:08.128 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468743+0000", 2024-08-19T14:14:08.128 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.3741e-05 2024-08-19T14:14:08.128 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.128 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.128 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:14:08.128 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468748+0000", 2024-08-19T14:14:08.128 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1000000000000003e-06 2024-08-19T14:14:08.128 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.128 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.128 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:14:08.128 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468751+0000", 2024-08-19T14:14:08.128 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.8260000000000001e-06 2024-08-19T14:14:08.128 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.128 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.129 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:14:08.129 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.478397+0000", 2024-08-19T14:14:08.129 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0096462240000000001 2024-08-19T14:14:08.129 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.129 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.129 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:14:08.129 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.478415+0000", 2024-08-19T14:14:08.129 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8450000000000001e-05 2024-08-19T14:14:08.129 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.129 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.129 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:14:08.129 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.480533+0000", 2024-08-19T14:14:08.129 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.002117196 2024-08-19T14:14:08.129 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.130 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.130 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:14:08.130 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.480558+0000", 2024-08-19T14:14:08.130 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.4828e-05 2024-08-19T14:14:08.130 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:14:08.130 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:14:08.130 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:14:08.130 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.130 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.130 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:49030 2.7 e117/102 2:e915209e:::benchmark_data_smithi045_46415_object49029:head v 117'3008, mlcod=117'2992)", 2024-08-19T14:14:08.130 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:07.468755+0000", 2024-08-19T14:14:08.130 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 60.572960279, 2024-08-19T14:14:08.130 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.015403441, 2024-08-19T14:14:08.130 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:14:08.131 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:14:08.131 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:14:08.131 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:14:08.131 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.131 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:14:08.131 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468755+0000", 2024-08-19T14:14:08.131 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:14:08.131 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.131 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.131 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:14:08.131 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468755+0000", 2024-08-19T14:14:08.131 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:14:08.131 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.131 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.132 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:14:08.132 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468755+0000", 2024-08-19T14:14:08.132 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5599999999999999e-07 2024-08-19T14:14:08.132 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.132 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.132 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:14:08.132 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468805+0000", 2024-08-19T14:14:08.132 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.9622e-05 2024-08-19T14:14:08.132 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.132 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.132 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:14:08.132 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468811+0000", 2024-08-19T14:14:08.132 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.6119999999999998e-06 2024-08-19T14:14:08.132 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.132 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.133 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:14:08.133 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468813+0000", 2024-08-19T14:14:08.133 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.7769999999999998e-06 2024-08-19T14:14:08.133 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.133 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.133 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:14:08.133 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.479393+0000", 2024-08-19T14:14:08.133 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.010579646999999999 2024-08-19T14:14:08.133 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.133 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.133 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:14:08.133 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.479412+0000", 2024-08-19T14:14:08.133 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8728000000000001e-05 2024-08-19T14:14:08.134 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.134 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.134 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:14:08.134 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.484094+0000", 2024-08-19T14:14:08.134 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0046820009999999999 2024-08-19T14:14:08.134 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.134 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.134 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:14:08.134 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.484159+0000", 2024-08-19T14:14:08.134 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.4898000000000001e-05 2024-08-19T14:14:08.134 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:14:08.134 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:14:08.134 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:14:08.134 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.135 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.135 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:49049 2.7 e117/102 2:eb29c0a1:::benchmark_data_smithi045_46415_object49048:head v 117'3009, mlcod=117'2992)", 2024-08-19T14:14:08.135 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:07.468819+0000", 2024-08-19T14:14:08.135 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 60.572897032, 2024-08-19T14:14:08.135 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.015383332, 2024-08-19T14:14:08.135 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:14:08.135 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:14:08.135 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:14:08.135 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:14:08.135 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.135 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:14:08.135 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468819+0000", 2024-08-19T14:14:08.135 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:14:08.135 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.136 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.136 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:14:08.136 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468819+0000", 2024-08-19T14:14:08.136 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:14:08.136 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.136 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.136 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:14:08.136 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468819+0000", 2024-08-19T14:14:08.136 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2000000000000001e-07 2024-08-19T14:14:08.136 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.136 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.136 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:14:08.136 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468854+0000", 2024-08-19T14:14:08.136 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.5079000000000001e-05 2024-08-19T14:14:08.137 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.137 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.137 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:14:08.137 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468859+0000", 2024-08-19T14:14:08.137 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1710000000000001e-06 2024-08-19T14:14:08.137 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.137 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.137 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:14:08.137 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468862+0000", 2024-08-19T14:14:08.137 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.0599999999999999e-06 2024-08-19T14:14:08.137 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.137 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.137 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:14:08.137 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.480596+0000", 2024-08-19T14:14:08.137 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.011734058 2024-08-19T14:14:08.138 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.138 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.138 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:14:08.138 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.480615+0000", 2024-08-19T14:14:08.138 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8919e-05 2024-08-19T14:14:08.138 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.138 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.138 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:14:08.138 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.484164+0000", 2024-08-19T14:14:08.138 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.003548486 2024-08-19T14:14:08.138 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:08.138 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:08.138 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:14:08.138 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.484202+0000", 2024-08-19T14:14:08.139 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.8339e-05 2024-08-19T14:14:08.139 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:14:08.139 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:14:08.139 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:14:08.139 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:14:08.139 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:14:08.139 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:14:08.144 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:08.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:08.145 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:14:08.246 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:14:08.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:08.273 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:08.273 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:14:08.374 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:14:08.397 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:08.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:08.398 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:14:08.499 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:14:08.542 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:08.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:08.543 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:14:08.644 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:14:08.676 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:08.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:08.677 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:14:08.778 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:14:08.805 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:08.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:08.807 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:14:08.836 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 79 16 114150 114134 90.2881 60.9375 0.00762661 0.0110711 2024-08-19T14:14:08.907 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:14:08.935 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:08.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:08.936 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:14:08.956 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:08 smithi045 ceph-mon[20143]: 2.4 scrub starts 2024-08-19T14:14:08.956 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:08 smithi045 ceph-mon[20143]: 2.8 scrub starts 2024-08-19T14:14:08.956 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:08 smithi045 ceph-mon[20143]: 2.c scrub starts 2024-08-19T14:14:09.003 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-08-19T14:14:09.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:08 smithi032 ceph-mon[20004]: 2.4 scrub starts 2024-08-19T14:14:09.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:08 smithi032 ceph-mon[20004]: 2.8 scrub starts 2024-08-19T14:14:09.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:08 smithi032 ceph-mon[20004]: 2.c scrub starts 2024-08-19T14:14:09.036 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:14:09.073 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:09.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:09.074 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:14:09.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:08 smithi031 ceph-mon[16336]: 2.4 scrub starts 2024-08-19T14:14:09.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:08 smithi031 ceph-mon[16336]: 2.8 scrub starts 2024-08-19T14:14:09.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:08 smithi031 ceph-mon[16336]: 2.c scrub starts 2024-08-19T14:14:09.175 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:14:09.212 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:09.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:09.214 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:14:09.315 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:14:09.353 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:09.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:09.355 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:14:09.452 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:14:09.455 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:14:09.477 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:09.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:09.478 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:14:09.579 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:14:09.617 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:09.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:09.618 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:14:09.719 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:14:09.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:09.741 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:09.742 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:14:09.837 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:2024-08-19T14:14:09.837121+0000 min lat: 0.00106853 max lat: 2.17987 avg lat: 0.0111253 2024-08-19T14:14:09.837 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-19T14:14:09.837 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 80 16 115003 114987 89.8258 53.3125 0.0127855 0.0111253 2024-08-19T14:14:09.842 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:14:09.879 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:09.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:09.880 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:14:09.981 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:14:10.006 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:10.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:09 smithi032 ceph-mon[20004]: pgmap v374: 27 pgs: 2 active+recovering+undersized+remapped, 25 active+clean; 6.7 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 44 MiB/s wr, 711 op/s; 3610/330000 objects misplaced (1.094%); 6.9 MiB/s, 110 objects/s recovering 2024-08-19T14:14:10.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:09 smithi032 ceph-mon[20004]: 2.a scrub starts 2024-08-19T14:14:10.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:09 smithi032 ceph-mon[20004]: 2.8 scrub ok 2024-08-19T14:14:10.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:09 smithi032 ceph-mon[20004]: 2.f scrub starts 2024-08-19T14:14:10.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:09 smithi032 ceph-mon[20004]: 2.d scrub starts 2024-08-19T14:14:10.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:10.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:14:10.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:09 smithi045 ceph-mon[20143]: pgmap v374: 27 pgs: 2 active+recovering+undersized+remapped, 25 active+clean; 6.7 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 44 MiB/s wr, 711 op/s; 3610/330000 objects misplaced (1.094%); 6.9 MiB/s, 110 objects/s recovering 2024-08-19T14:14:10.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:09 smithi045 ceph-mon[20143]: 2.a scrub starts 2024-08-19T14:14:10.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:09 smithi045 ceph-mon[20143]: 2.8 scrub ok 2024-08-19T14:14:10.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:09 smithi045 ceph-mon[20143]: 2.f scrub starts 2024-08-19T14:14:10.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:09 smithi045 ceph-mon[20143]: 2.d scrub starts 2024-08-19T14:14:10.110 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:14:10.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:09 smithi031 ceph-mon[16336]: pgmap v374: 27 pgs: 2 active+recovering+undersized+remapped, 25 active+clean; 6.7 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 44 MiB/s wr, 711 op/s; 3610/330000 objects misplaced (1.094%); 6.9 MiB/s, 110 objects/s recovering 2024-08-19T14:14:10.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:09 smithi031 ceph-mon[16336]: 2.a scrub starts 2024-08-19T14:14:10.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:09 smithi031 ceph-mon[16336]: 2.8 scrub ok 2024-08-19T14:14:10.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:09 smithi031 ceph-mon[16336]: 2.f scrub starts 2024-08-19T14:14:10.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:09 smithi031 ceph-mon[16336]: 2.d scrub starts 2024-08-19T14:14:10.141 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:10.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:10.142 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:14:10.243 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:14:10.267 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:10.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:10.268 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:14:10.369 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:14:10.407 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:10.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:10.409 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:14:10.509 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:14:10.545 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:10.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:10.546 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:14:10.646 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:14:10.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:10.683 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:10.683 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:14:10.784 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:14:10.805 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:10.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:10.806 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:14:10.836 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 81 16 115750 115734 89.2932 46.6875 0.0122978 0.011194 2024-08-19T14:14:10.897 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd reweight 4 1 2024-08-19T14:14:10.906 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:14:10.941 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:10.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:10.942 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:14:10.952 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:14:11.043 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:14:11.067 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:11.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:11.068 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:14:11.169 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:14:11.221 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:11.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:11.222 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:14:11.323 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:14:11.347 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:11.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:11.348 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:14:11.449 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:14:11.478 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:11.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:11.479 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:14:11.580 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:14:11.593 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:14:11.655 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:11.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:11.656 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:14:11.757 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:14:11.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:11.798 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:11.798 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:14:11.836 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 82 16 116590 116574 88.8444 52.5 0.00557894 0.011251 2024-08-19T14:14:11.899 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:14:11.921 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:11.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:11.922 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:14:11.938 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:11 smithi031 ceph-mon[16336]: pgmap v375: 27 pgs: 3 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 23 active+clean; 6.9 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 54 MiB/s wr, 862 op/s; 3557/337755 objects misplaced (1.053%); 5.6 MiB/s, 90 objects/s recovering 2024-08-19T14:14:11.938 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:11 smithi031 ceph-mon[16336]: 2.4 scrub ok 2024-08-19T14:14:11.938 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:11 smithi031 ceph-mon[16336]: 2.f scrub ok 2024-08-19T14:14:11.938 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:11 smithi031 ceph-mon[16336]: 2.c scrub ok 2024-08-19T14:14:12.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:11 smithi032 ceph-mon[20004]: pgmap v375: 27 pgs: 3 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 23 active+clean; 6.9 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 54 MiB/s wr, 862 op/s; 3557/337755 objects misplaced (1.053%); 5.6 MiB/s, 90 objects/s recovering 2024-08-19T14:14:12.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:11 smithi032 ceph-mon[20004]: 2.4 scrub ok 2024-08-19T14:14:12.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:11 smithi032 ceph-mon[20004]: 2.f scrub ok 2024-08-19T14:14:12.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:11 smithi032 ceph-mon[20004]: 2.c scrub ok 2024-08-19T14:14:12.023 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:14:12.043 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:12.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:12.044 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:14:12.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:11 smithi045 ceph-mon[20143]: pgmap v375: 27 pgs: 3 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 23 active+clean; 6.9 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 54 MiB/s wr, 862 op/s; 3557/337755 objects misplaced (1.053%); 5.6 MiB/s, 90 objects/s recovering 2024-08-19T14:14:12.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:11 smithi045 ceph-mon[20143]: 2.4 scrub ok 2024-08-19T14:14:12.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:11 smithi045 ceph-mon[20143]: 2.f scrub ok 2024-08-19T14:14:12.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:11 smithi045 ceph-mon[20143]: 2.c scrub ok 2024-08-19T14:14:12.145 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:14:12.169 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:12.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:12.170 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:14:12.271 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:14:12.297 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:12.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:12.298 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:14:12.400 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:14:12.425 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:12.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:12.426 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:14:12.527 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:14:12.552 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:12.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:12.554 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:14:12.654 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:14:12.682 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:12.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:12.683 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:14:12.693 INFO:teuthology.orchestra.run.smithi031.stderr:reweighted osd.4 to 1 (10000) 2024-08-19T14:14:12.723 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is unset 2024-08-19T14:14:12.769 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:14:12.783 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:14:12.812 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:12.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:12.813 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:14:12.836 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 83 16 117449 117433 88.4208 53.6875 0.021179 0.0113043 2024-08-19T14:14:12.914 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:14:12.938 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:12.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:12.939 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:14:13.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:12 smithi032 ceph-mon[20004]: 2.a scrub ok 2024-08-19T14:14:13.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:12 smithi032 ceph-mon[20004]: 2.2 scrub starts 2024-08-19T14:14:13.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:12 smithi032 ceph-mon[20004]: 2.d scrub ok 2024-08-19T14:14:13.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:12 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2430312181' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:14:13.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:12 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:14:13.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:12 smithi032 ceph-mon[20004]: 2.5 scrub starts 2024-08-19T14:14:13.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:12 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2303340433' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-08-19T14:14:13.040 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:14:13.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:12 smithi045 ceph-mon[20143]: 2.a scrub ok 2024-08-19T14:14:13.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:12 smithi045 ceph-mon[20143]: 2.2 scrub starts 2024-08-19T14:14:13.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:12 smithi045 ceph-mon[20143]: 2.d scrub ok 2024-08-19T14:14:13.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:12 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2430312181' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:14:13.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:12 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:14:13.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:12 smithi045 ceph-mon[20143]: 2.5 scrub starts 2024-08-19T14:14:13.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:12 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2303340433' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-08-19T14:14:13.066 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:13.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:13.068 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:14:13.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:12 smithi031 ceph-mon[16336]: 2.a scrub ok 2024-08-19T14:14:13.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:12 smithi031 ceph-mon[16336]: 2.2 scrub starts 2024-08-19T14:14:13.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:12 smithi031 ceph-mon[16336]: 2.d scrub ok 2024-08-19T14:14:13.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:12 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2430312181' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:14:13.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:12 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:14:13.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:12 smithi031 ceph-mon[16336]: 2.5 scrub starts 2024-08-19T14:14:13.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:12 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2303340433' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-08-19T14:14:13.168 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:14:13.196 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:13.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:13.197 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:14:13.298 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:14:13.329 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:13.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:13.330 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:14:13.431 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:14:13.459 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:13.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:13.460 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:14:13.561 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:14:13.584 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:13.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:13.585 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:14:13.686 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:14:13.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:13.709 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:13.709 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:14:13.810 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:14:13.836 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 84 16 118439 118423 88.1047 61.875 0.00909336 0.0113455 2024-08-19T14:14:13.838 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:13.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:13.839 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:14:13.927 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:14:13.927 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [ 2024-08-19T14:14:13.927 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:13.927 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24580.0:118282 2.a 2:50fc0aa1:::benchmark_data_smithi045_46415_object118281:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-08-19T14:14:13.928 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:14:13.660808+0000", 2024-08-19T14:14:13.928 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.033081278999999998, 2024-08-19T14:14:13.928 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.033115239999999997, 2024-08-19T14:14:13.928 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:14:13.928 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:14:13.928 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "waiting for sub ops", 2024-08-19T14:14:13.928 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:14:13.928 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24580", 2024-08-19T14:14:13.928 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:14:13.928 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 118282 2024-08-19T14:14:13.928 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:13.928 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:14:13.928 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:13.929 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:14:13.929 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:13.660808+0000", 2024-08-19T14:14:13.929 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:14:13.929 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:13.929 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:13.929 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:14:13.929 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:13.660808+0000", 2024-08-19T14:14:13.929 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:14:13.929 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:13.929 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:13.929 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:14:13.929 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:13.660809+0000", 2024-08-19T14:14:13.929 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.113e-06 2024-08-19T14:14:13.930 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:13.930 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:13.930 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:14:13.930 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:13.660829+0000", 2024-08-19T14:14:13.930 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9894e-05 2024-08-19T14:14:13.930 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:13.930 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:13.930 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:14:13.930 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:13.660837+0000", 2024-08-19T14:14:13.930 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.8979999999999996e-06 2024-08-19T14:14:13.930 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:13.930 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:13.931 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:14:13.931 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:13.660842+0000", 2024-08-19T14:14:13.931 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.6929999999999998e-06 2024-08-19T14:14:13.931 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:13.931 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:13.931 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:14:13.931 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:13.660886+0000", 2024-08-19T14:14:13.931 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.4063999999999997e-05 2024-08-19T14:14:13.931 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:13.931 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:13.932 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:14:13.932 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:13.660980+0000", 2024-08-19T14:14:13.932 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.4634000000000002e-05 2024-08-19T14:14:13.932 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:13.932 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:13.932 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 3,9", 2024-08-19T14:14:13.932 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:13.661067+0000", 2024-08-19T14:14:13.932 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.6448999999999999e-05 2024-08-19T14:14:13.932 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:13.932 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:13.932 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:14:13.932 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:13.663987+0000", 2024-08-19T14:14:13.933 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0029199460000000001 2024-08-19T14:14:13.933 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:13.933 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:13.933 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:14:13.933 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:13.672671+0000", 2024-08-19T14:14:13.933 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0086847539999999994 2024-08-19T14:14:13.933 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:14:13.933 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:14:13.933 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:14:13.933 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:13.933 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:13.933 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24580.0:118290 2.a 2:569c54de:::benchmark_data_smithi045_46415_object118289:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-08-19T14:14:13.933 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:14:13.665568+0000", 2024-08-19T14:14:13.934 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.028321591, 2024-08-19T14:14:13.934 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.028415883999999999, 2024-08-19T14:14:13.934 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:14:13.934 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:14:13.934 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "waiting for sub ops", 2024-08-19T14:14:13.934 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:14:13.934 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24580", 2024-08-19T14:14:13.934 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:14:13.934 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 118290 2024-08-19T14:14:13.935 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:13.935 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:14:13.935 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:13.935 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:14:13.935 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:13.665568+0000", 2024-08-19T14:14:13.935 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:14:13.935 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:13.935 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:13.935 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:14:13.935 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:13.665568+0000", 2024-08-19T14:14:13.936 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:14:13.936 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:13.936 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:13.936 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:14:13.936 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:13.665569+0000", 2024-08-19T14:14:13.936 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1069999999999999e-06 2024-08-19T14:14:13.936 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:13.936 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:13.936 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:14:13.936 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:13.665590+0000", 2024-08-19T14:14:13.936 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1302999999999999e-05 2024-08-19T14:14:13.936 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:13.936 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:13.936 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:14:13.937 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:13.665600+0000", 2024-08-19T14:14:13.937 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.5570000000000001e-06 2024-08-19T14:14:13.937 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:13.937 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:13.937 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:14:13.937 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:13.665605+0000", 2024-08-19T14:14:13.937 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.6200000000000004e-06 2024-08-19T14:14:13.937 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:13.937 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:13.938 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:14:13.938 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:13.665649+0000", 2024-08-19T14:14:13.938 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.4184000000000003e-05 2024-08-19T14:14:13.938 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:13.938 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:13.938 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:14:13.938 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:13.665748+0000", 2024-08-19T14:14:13.938 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.8635999999999995e-05 2024-08-19T14:14:13.938 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:13.938 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:13.938 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 3,9", 2024-08-19T14:14:13.938 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:13.665834+0000", 2024-08-19T14:14:13.938 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.5730999999999996e-05 2024-08-19T14:14:13.939 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:13.939 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:13.939 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:14:13.939 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:13.668542+0000", 2024-08-19T14:14:13.939 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0027084380000000001 2024-08-19T14:14:13.939 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:14:13.939 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:14:13.939 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:14:13.939 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:13.672893+0000", 2024-08-19T14:14:13.939 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0043511139999999997 2024-08-19T14:14:13.939 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:14:13.939 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:14:13.939 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:14:13.940 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:14:13.940 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-08-19T14:14:13.940 INFO:teuthology.orchestra.run.smithi045.stdout: "num_ops": 2 2024-08-19T14:14:13.940 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:14:13.940 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:14:13.941 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:13 smithi045 ceph-mon[20143]: pgmap v376: 27 pgs: 1 active+recovering+undersized+remapped, 6 active+clean+scrubbing, 20 active+clean; 7.0 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 57 MiB/s wr, 911 op/s; 3175/344127 objects misplaced (0.923%); 3.4 MiB/s, 55 objects/s recovering 2024-08-19T14:14:13.941 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:13 smithi045 ceph-mon[20143]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:14:13.941 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:13 smithi045 ceph-mon[20143]: Cluster is now healthy 2024-08-19T14:14:13.941 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:13 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:14:13.941 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:13 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2303340433' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-08-19T14:14:13.941 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:13 smithi045 ceph-mon[20143]: osdmap e148: 12 total, 12 up, 10 in 2024-08-19T14:14:13.941 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:13 smithi045 ceph-mon[20143]: 2.2 scrub ok 2024-08-19T14:14:13.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:13.971 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:13.971 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:14:14.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:13 smithi032 ceph-mon[20004]: pgmap v376: 27 pgs: 1 active+recovering+undersized+remapped, 6 active+clean+scrubbing, 20 active+clean; 7.0 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 57 MiB/s wr, 911 op/s; 3175/344127 objects misplaced (0.923%); 3.4 MiB/s, 55 objects/s recovering 2024-08-19T14:14:14.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:13 smithi032 ceph-mon[20004]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:14:14.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:13 smithi032 ceph-mon[20004]: Cluster is now healthy 2024-08-19T14:14:14.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:13 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:14:14.009 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:13 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2303340433' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-08-19T14:14:14.009 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:13 smithi032 ceph-mon[20004]: osdmap e148: 12 total, 12 up, 10 in 2024-08-19T14:14:14.009 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:13 smithi032 ceph-mon[20004]: 2.2 scrub ok 2024-08-19T14:14:14.011 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:13 smithi031 ceph-mon[16336]: pgmap v376: 27 pgs: 1 active+recovering+undersized+remapped, 6 active+clean+scrubbing, 20 active+clean; 7.0 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 57 MiB/s wr, 911 op/s; 3175/344127 objects misplaced (0.923%); 3.4 MiB/s, 55 objects/s recovering 2024-08-19T14:14:14.011 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:13 smithi031 ceph-mon[16336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:14:14.011 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:13 smithi031 ceph-mon[16336]: Cluster is now healthy 2024-08-19T14:14:14.011 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:13 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:14:14.012 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:13 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2303340433' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-08-19T14:14:14.012 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:13 smithi031 ceph-mon[16336]: osdmap e148: 12 total, 12 up, 10 in 2024-08-19T14:14:14.012 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:13 smithi031 ceph-mon[16336]: 2.2 scrub ok 2024-08-19T14:14:14.072 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:14:14.106 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:14.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:14.107 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:14:14.208 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:14:14.238 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:14.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:14.239 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:14:14.340 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:14:14.364 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:14.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:14.365 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:14:14.466 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:14:14.512 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:14.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:14.513 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:14:14.614 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:14:14.639 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:14.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:14.640 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:14:14.644 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-08-19T14:14:14.658 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd reweight 5 1 2024-08-19T14:14:14.741 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:14:14.765 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:14.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:14.766 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:14:14.828 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:14:14.837 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 85 16 119332 119316 87.7248 55.8125 0.0107396 0.0113886 2024-08-19T14:14:14.867 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:14:14.895 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:14.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:14.896 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:14:14.963 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:14:14.997 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:14:15.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:15.243 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:15.244 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:14:15.344 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:14:15.367 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:15.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:15.368 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:14:15.462 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:14:15.463 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:14:15.468 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:14:15.486 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:14:15.486 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [], 2024-08-19T14:14:15.486 INFO:teuthology.orchestra.run.smithi045.stdout: "complaint_time": 30, 2024-08-19T14:14:15.486 INFO:teuthology.orchestra.run.smithi045.stdout: "num_blocked_ops": 0 2024-08-19T14:14:15.486 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:14:15.502 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:14:15.503 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:14:15.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:15.526 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:15.527 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:14:15.535 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:14:15.541 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:14:15.541 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:14:15.579 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:14:15.579 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:14:15.613 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:14:15.613 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:14:15.628 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:14:15.642 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:14:15.643 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:14:15.654 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:15.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:15.655 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:14:15.687 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:14:15.688 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:14:15.722 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:14:15.722 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:14:15.755 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:14:15.759 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:14:15.760 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:14:15.777 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:15.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:15.779 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:14:15.795 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:14:15.795 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:14:15.838 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 86 16 120137 120121 87.2897 50.3125 0.0255222 0.0114498 2024-08-19T14:14:15.838 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:15 smithi031 ceph-mon[16336]: pgmap v378: 27 pgs: 1 active+recovering+undersized+remapped, 4 active+clean+scrubbing, 22 active+clean; 7.1 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 56 MiB/s wr, 898 op/s; 3175/348951 objects misplaced (0.910%); 3.1 MiB/s, 49 objects/s recovering 2024-08-19T14:14:15.838 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:15 smithi031 ceph-mon[16336]: 2.6 deep-scrub starts 2024-08-19T14:14:15.838 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:15 smithi031 ceph-mon[16336]: 2.5 scrub ok 2024-08-19T14:14:15.838 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:15 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2933652272' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:14:15.839 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:14:15.839 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:14:15.874 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:14:15.875 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:14:15.879 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:14:15.902 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:15.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:15.904 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:14:16.005 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:14:16.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:15 smithi032 ceph-mon[20004]: pgmap v378: 27 pgs: 1 active+recovering+undersized+remapped, 4 active+clean+scrubbing, 22 active+clean; 7.1 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 56 MiB/s wr, 898 op/s; 3175/348951 objects misplaced (0.910%); 3.1 MiB/s, 49 objects/s recovering 2024-08-19T14:14:16.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:15 smithi032 ceph-mon[20004]: 2.6 deep-scrub starts 2024-08-19T14:14:16.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:15 smithi032 ceph-mon[20004]: 2.5 scrub ok 2024-08-19T14:14:16.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:15 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2933652272' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:14:16.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:16.034 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:16.035 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:14:16.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:15 smithi045 ceph-mon[20143]: pgmap v378: 27 pgs: 1 active+recovering+undersized+remapped, 4 active+clean+scrubbing, 22 active+clean; 7.1 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 56 MiB/s wr, 898 op/s; 3175/348951 objects misplaced (0.910%); 3.1 MiB/s, 49 objects/s recovering 2024-08-19T14:14:16.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:15 smithi045 ceph-mon[20143]: 2.6 deep-scrub starts 2024-08-19T14:14:16.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:15 smithi045 ceph-mon[20143]: 2.5 scrub ok 2024-08-19T14:14:16.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:15 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2933652272' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:14:16.060 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-19T14:14:16.136 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:14:16.161 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:16.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:16.162 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:14:16.263 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:14:16.285 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:16.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:16.286 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:14:16.387 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:14:16.408 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:16.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:16.409 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:14:16.510 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:14:16.532 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:16.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:16.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:14:16.634 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:14:16.679 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:16.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:16.682 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:14:16.783 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:14:16.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:16.832 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:16.833 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:14:16.837 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 87 16 120823 120807 86.7792 42.875 0.00866539 0.0115149 2024-08-19T14:14:16.933 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:14:16.957 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:16.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:16.958 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:14:17.060 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:14:17.088 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:17.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:17.089 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:14:17.190 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:14:17.223 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:17.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:17.224 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:14:17.310 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:14:17.325 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:14:17.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:17.385 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:17.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:14:17.487 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:14:17.531 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:17.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:17.536 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:14:17.636 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:14:17.685 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:17.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:17.686 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:14:17.709 INFO:teuthology.orchestra.run.smithi031.stderr:reweighted osd.5 to 1 (10000) 2024-08-19T14:14:17.786 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:14:17.814 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:17.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:17.815 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:14:17.837 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 88 16 121640 121624 86.3732 51.0625 0.0112744 0.0115702 2024-08-19T14:14:17.916 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:14:17.931 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-08-19T14:14:17.961 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:17.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:17.963 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:14:17.999 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:17 smithi045 ceph-mon[20143]: pgmap v379: 27 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 25 active+clean; 7.2 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 56 MiB/s wr, 893 op/s; 3175/352797 objects misplaced (0.900%); 2.8 MiB/s, 44 objects/s recovering 2024-08-19T14:14:17.999 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:17 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3877452571' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-08-19T14:14:18.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:17 smithi032 ceph-mon[20004]: pgmap v379: 27 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 25 active+clean; 7.2 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 56 MiB/s wr, 893 op/s; 3175/352797 objects misplaced (0.900%); 2.8 MiB/s, 44 objects/s recovering 2024-08-19T14:14:18.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:17 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3877452571' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-08-19T14:14:18.031 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:17 smithi031 ceph-mon[16336]: pgmap v379: 27 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 25 active+clean; 7.2 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 56 MiB/s wr, 893 op/s; 3175/352797 objects misplaced (0.900%); 2.8 MiB/s, 44 objects/s recovering 2024-08-19T14:14:18.031 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:17 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3877452571' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-08-19T14:14:18.064 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:14:18.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:18.097 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:18.097 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:14:18.198 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:14:18.221 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:18.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:18.222 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:14:18.299 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:14:18.323 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:14:18.346 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:18.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:18.347 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:14:18.447 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:14:18.485 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:18.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:18.486 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:14:18.519 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd reweight 7 1 2024-08-19T14:14:18.586 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:14:18.648 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:18.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:18.650 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:14:18.728 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is set 2024-08-19T14:14:18.750 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:14:18.810 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:18.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:18.811 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:14:18.837 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 89 16 122454 122438 85.9743 50.875 0.0111143 0.011625 2024-08-19T14:14:18.858 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:14:18.859 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:14:18.911 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:14:18.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:18.961 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:18.962 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:14:19.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:18 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3877452571' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-08-19T14:14:19.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:18 smithi032 ceph-mon[20004]: osdmap e149: 12 total, 12 up, 10 in 2024-08-19T14:14:19.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:18 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:14:19.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:18 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:14:19.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:18 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2304439733' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:14:19.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:18 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3877452571' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-08-19T14:14:19.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:18 smithi045 ceph-mon[20143]: osdmap e149: 12 total, 12 up, 10 in 2024-08-19T14:14:19.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:18 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:14:19.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:18 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:14:19.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:18 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2304439733' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:14:19.062 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:14:19.094 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:19.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:19.095 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:14:19.106 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:18 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3877452571' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-08-19T14:14:19.106 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:18 smithi031 ceph-mon[16336]: osdmap e149: 12 total, 12 up, 10 in 2024-08-19T14:14:19.106 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:18 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:14:19.106 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:18 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:14:19.106 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:18 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2304439733' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:14:19.196 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:14:19.218 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:19.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:19.219 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:14:19.309 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:14:19.321 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:14:19.343 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:19.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:19.344 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:14:19.445 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:14:19.471 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:19.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:19.472 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:14:19.547 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:14:19.573 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:14:19.595 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:19.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:19.596 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:14:19.697 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:14:19.726 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:19.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:19.727 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:14:19.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:19 smithi045 ceph-mon[20143]: pgmap v381: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 7.3 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 50 MiB/s wr, 797 op/s; 2767/356892 objects misplaced (0.775%); 3.1 MiB/s, 49 objects/s recovering 2024-08-19T14:14:19.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:19 smithi045 ceph-mon[20143]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:14:19.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:19 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2304439733' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:14:19.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:19 smithi045 ceph-mon[20143]: osdmap e150: 12 total, 12 up, 10 in 2024-08-19T14:14:19.828 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:14:19.835 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-08-19T14:14:19.837 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 90 10 123278 123268 85.5954 51.875 0.0100579 0.0116773 2024-08-19T14:14:19.849 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:19.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:19.850 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:14:19.951 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:14:19.978 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:19.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:19.979 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:14:19.984 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:19 smithi031 ceph-mon[16336]: pgmap v381: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 7.3 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 50 MiB/s wr, 797 op/s; 2767/356892 objects misplaced (0.775%); 3.1 MiB/s, 49 objects/s recovering 2024-08-19T14:14:19.984 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:19 smithi031 ceph-mon[16336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:14:19.984 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:19 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2304439733' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:14:19.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:19 smithi031 ceph-mon[16336]: osdmap e150: 12 total, 12 up, 10 in 2024-08-19T14:14:20.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:19 smithi032 ceph-mon[20004]: pgmap v381: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 7.3 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 50 MiB/s wr, 797 op/s; 2767/356892 objects misplaced (0.775%); 3.1 MiB/s, 49 objects/s recovering 2024-08-19T14:14:20.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:19 smithi032 ceph-mon[20004]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:14:20.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:19 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2304439733' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:14:20.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:19 smithi032 ceph-mon[20004]: osdmap e150: 12 total, 12 up, 10 in 2024-08-19T14:14:20.080 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:14:20.105 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:20.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:20.106 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:14:20.207 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:14:20.228 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:20.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:20.229 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:14:20.330 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:14:20.352 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:20.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:20.353 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:14:20.454 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:14:20.477 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:20.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:20.478 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:14:20.534 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:14:20.578 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:14:20.599 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:20.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:20.600 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:14:20.701 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:14:20.725 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:20.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:20.726 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:14:20.737 INFO:teuthology.orchestra.run.smithi031.stderr:reweighted osd.7 to 1 (10000) 2024-08-19T14:14:20.752 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:20 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/4059887019' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-08-19T14:14:20.752 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:20 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-08-19T14:14:20.827 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:14:20.837 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Total time run: 90.0257 2024-08-19T14:14:20.837 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Total writes made: 123278 2024-08-19T14:14:20.838 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Write size: 65536 2024-08-19T14:14:20.838 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Object size: 65536 2024-08-19T14:14:20.838 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Bandwidth (MB/sec): 85.5853 2024-08-19T14:14:20.838 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Stddev Bandwidth: 52.4865 2024-08-19T14:14:20.838 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Max bandwidth (MB/sec): 288.062 2024-08-19T14:14:20.838 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Min bandwidth (MB/sec): 0 2024-08-19T14:14:20.838 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Average IOPS: 1369 2024-08-19T14:14:20.838 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Stddev IOPS: 839.802 2024-08-19T14:14:20.838 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Max IOPS: 4609 2024-08-19T14:14:20.838 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Min IOPS: 0 2024-08-19T14:14:20.838 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Average Latency(s): 0.0116795 2024-08-19T14:14:20.838 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Stddev Latency(s): 0.0407169 2024-08-19T14:14:20.838 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Max latency(s): 2.17987 2024-08-19T14:14:20.839 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Min latency(s): 0.00106853 2024-08-19T14:14:20.842 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Cleaning up (deleting benchmark objects) 2024-08-19T14:14:20.851 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:20.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:20.852 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:14:20.953 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:14:20.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:20.994 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:20.994 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:14:21.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:20 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/4059887019' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-08-19T14:14:21.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:20 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-08-19T14:14:21.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:20 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/4059887019' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-08-19T14:14:21.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:20 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-08-19T14:14:21.096 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:14:21.117 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:21.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:21.118 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:14:21.219 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:14:21.255 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:21.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:21.256 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:14:21.295 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:14:21.296 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:14:21.335 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:14:21.336 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:14:21.358 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:14:21.374 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:14:21.375 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:14:21.390 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:21.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:21.391 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:14:21.400 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:14:21.400 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:14:21.433 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:14:21.433 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:14:21.464 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-19T14:14:21.464 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:14:21.465 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:14:21.492 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:14:21.501 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:14:21.502 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:14:21.514 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:21.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:21.515 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:14:21.533 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:14:21.533 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:14:21.564 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:14:21.564 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:14:21.591 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:14:21.592 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:14:21.616 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:14:21.622 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:14:21.623 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:14:21.643 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:21.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:21.644 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:14:21.664 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:14:21.664 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:14:21.745 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:14:21.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:21.793 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:21.793 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:14:21.837 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd reweight 8 1 2024-08-19T14:14:21.894 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:14:21.916 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:21.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:21.916 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:14:22.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:21 smithi032 ceph-mon[20004]: pgmap v383: 27 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+remapped, 25 active+clean; 7.4 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 59 MiB/s wr, 936 op/s; 2767/363981 objects misplaced (0.760%); 1.8 MiB/s, 28 objects/s recovering 2024-08-19T14:14:22.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:21 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:14:22.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:21 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:14:22.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:21 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-08-19T14:14:22.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:21 smithi032 ceph-mon[20004]: osdmap e151: 12 total, 12 up, 10 in 2024-08-19T14:14:22.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:21 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/722523768' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:14:22.017 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:14:22.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:21 smithi045 ceph-mon[20143]: pgmap v383: 27 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+remapped, 25 active+clean; 7.4 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 59 MiB/s wr, 936 op/s; 2767/363981 objects misplaced (0.760%); 1.8 MiB/s, 28 objects/s recovering 2024-08-19T14:14:22.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:21 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:14:22.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:21 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:14:22.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:21 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-08-19T14:14:22.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:21 smithi045 ceph-mon[20143]: osdmap e151: 12 total, 12 up, 10 in 2024-08-19T14:14:22.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:21 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/722523768' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:14:22.066 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:14:22.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:22.083 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:22.084 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:14:22.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:21 smithi031 ceph-mon[16336]: pgmap v383: 27 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+remapped, 25 active+clean; 7.4 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 59 MiB/s wr, 936 op/s; 2767/363981 objects misplaced (0.760%); 1.8 MiB/s, 28 objects/s recovering 2024-08-19T14:14:22.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:21 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:14:22.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:21 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:14:22.118 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:21 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-08-19T14:14:22.118 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:21 smithi031 ceph-mon[16336]: osdmap e151: 12 total, 12 up, 10 in 2024-08-19T14:14:22.118 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:21 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/722523768' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:14:22.185 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:14:22.208 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:22.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:22.209 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:14:22.310 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:14:22.333 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:22.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:22.334 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:14:22.435 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:14:22.459 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:22.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:22.460 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:14:22.554 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:14:22.560 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:14:22.619 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:22.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:22.620 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:14:22.721 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:14:22.758 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:22.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:22.760 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:14:22.861 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:14:22.883 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:22.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:22.884 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:14:22.984 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:14:23.007 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:23.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:23.008 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:14:23.042 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:22 smithi031 ceph-mon[16336]: pgmap v385: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing+deep, 25 active+clean; 7.5 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 53 MiB/s wr, 864 op/s; 5/367809 objects degraded (0.001%); 2234/367809 objects misplaced (0.607%); 4.9 MiB/s, 77 objects/s recovering 2024-08-19T14:14:23.043 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:22 smithi031 ceph-mon[16336]: Health check failed: Degraded data redundancy: 5/367809 objects degraded (0.001%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-19T14:14:23.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:22 smithi045 ceph-mon[20143]: pgmap v385: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing+deep, 25 active+clean; 7.5 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 53 MiB/s wr, 864 op/s; 5/367809 objects degraded (0.001%); 2234/367809 objects misplaced (0.607%); 4.9 MiB/s, 77 objects/s recovering 2024-08-19T14:14:23.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:22 smithi045 ceph-mon[20143]: Health check failed: Degraded data redundancy: 5/367809 objects degraded (0.001%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-19T14:14:23.109 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:14:23.133 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:23.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:23.134 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:14:23.235 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:14:23.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:22 smithi032 ceph-mon[20004]: pgmap v385: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing+deep, 25 active+clean; 7.5 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 53 MiB/s wr, 864 op/s; 5/367809 objects degraded (0.001%); 2234/367809 objects misplaced (0.607%); 4.9 MiB/s, 77 objects/s recovering 2024-08-19T14:14:23.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:22 smithi032 ceph-mon[20004]: Health check failed: Degraded data redundancy: 5/367809 objects degraded (0.001%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-19T14:14:23.278 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:23.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:23.279 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:14:23.340 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:14:23.340 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [ 2024-08-19T14:14:23.340 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:23.340 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.24580.0:125320 2.5 e151/102 2:abb62c21:::benchmark_data_smithi045_46415_object2039:head v 151'7852, mlcod=151'7850)", 2024-08-19T14:14:23.340 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:14:23.328623+0000", 2024-08-19T14:14:23.340 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.010585848, 2024-08-19T14:14:23.340 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.010619511999999999, 2024-08-19T14:14:23.340 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:14:23.340 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:14:23.341 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:14:23.341 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:14:23.341 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:23.341 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:14:23.341 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:23.328623+0000", 2024-08-19T14:14:23.341 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:14:23.341 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:23.341 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:23.341 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:14:23.341 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:23.328623+0000", 2024-08-19T14:14:23.341 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:14:23.342 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:23.342 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:23.342 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:14:23.342 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:23.328624+0000", 2024-08-19T14:14:23.342 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.6499999999999998e-07 2024-08-19T14:14:23.342 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:23.342 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:23.342 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:14:23.342 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:23.328630+0000", 2024-08-19T14:14:23.342 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.0469999999999999e-06 2024-08-19T14:14:23.343 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:23.343 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:23.343 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:14:23.343 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:23.328644+0000", 2024-08-19T14:14:23.343 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3584000000000001e-05 2024-08-19T14:14:23.343 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:23.343 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:23.343 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:14:23.343 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:23.328651+0000", 2024-08-19T14:14:23.343 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.8390000000000001e-06 2024-08-19T14:14:23.343 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:23.343 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:23.343 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:14:23.344 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:23.328696+0000", 2024-08-19T14:14:23.344 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4934e-05 2024-08-19T14:14:23.344 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:23.344 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:23.344 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:14:23.344 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:23.328709+0000", 2024-08-19T14:14:23.344 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2884999999999999e-05 2024-08-19T14:14:23.344 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:23.344 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:23.344 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:14:23.344 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:23.339238+0000", 2024-08-19T14:14:23.344 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.010529948000000001 2024-08-19T14:14:23.344 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:14:23.344 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:14:23.345 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:14:23.345 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:23.345 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:23.345 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.24580.0:125326 2.5 e151/102 2:aa8c2940:::benchmark_data_smithi045_46415_object2045:head v 151'7853, mlcod=151'7850)", 2024-08-19T14:14:23.345 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:14:23.332075+0000", 2024-08-19T14:14:23.345 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.0071347479999999998, 2024-08-19T14:14:23.345 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0072367600000000001, 2024-08-19T14:14:23.345 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:14:23.345 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:14:23.345 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-08-19T14:14:23.345 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:14:23.346 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:23.346 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:14:23.346 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:23.332075+0000", 2024-08-19T14:14:23.346 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:14:23.346 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:23.346 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:23.346 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:14:23.346 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:23.332075+0000", 2024-08-19T14:14:23.346 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:14:23.346 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:23.346 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:23.346 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:14:23.346 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:23.332075+0000", 2024-08-19T14:14:23.347 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.68e-07 2024-08-19T14:14:23.347 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:23.347 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:23.347 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:14:23.347 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:23.332081+0000", 2024-08-19T14:14:23.347 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.9000000000000003e-06 2024-08-19T14:14:23.347 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:23.347 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:23.347 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:14:23.347 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:23.332091+0000", 2024-08-19T14:14:23.347 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.8269999999999993e-06 2024-08-19T14:14:23.347 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:23.347 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:23.348 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:14:23.348 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:23.332098+0000", 2024-08-19T14:14:23.348 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.1130000000000003e-06 2024-08-19T14:14:23.348 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:23.348 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:23.348 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:14:23.348 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:23.335668+0000", 2024-08-19T14:14:23.348 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0035702939999999999 2024-08-19T14:14:23.348 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:23.348 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:23.348 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:14:23.348 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:23.335696+0000", 2024-08-19T14:14:23.348 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.762e-05 2024-08-19T14:14:23.348 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:14:23.349 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:14:23.349 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:14:23.349 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:14:23.349 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-08-19T14:14:23.349 INFO:teuthology.orchestra.run.smithi031.stdout: "num_ops": 2 2024-08-19T14:14:23.349 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:14:23.380 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:14:23.403 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:23.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:23.404 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:14:23.504 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:14:23.543 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:23.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:23.544 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:14:23.645 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:14:23.671 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:23.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:23.672 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:14:23.773 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:14:23.797 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:23.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:23.798 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:14:23.899 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:14:23.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:23.934 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:23.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:14:24.036 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:14:24.069 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:24.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:24.070 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:14:24.171 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:14:24.194 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:24.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:24.195 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:14:24.296 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:14:24.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:24.325 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:24.325 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:14:24.403 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-08-19T14:14:24.425 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:14:24.448 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:24.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:24.449 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:14:24.523 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:14:24.548 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:14:24.550 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:14:24.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:24.622 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:24.623 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:14:24.724 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:14:24.748 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:24.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:24.749 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:14:24.806 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is set 2024-08-19T14:14:24.850 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:14:24.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:24.892 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:24.893 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:14:24.965 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:24 smithi031 ceph-mon[16336]: pgmap v386: 27 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+degraded+remapped, 25 active+clean; 7.5 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 169 B/s rd, 44 MiB/s wr, 832 op/s; 5/367506 objects degraded (0.001%); 2234/367506 objects misplaced (0.608%); 2.7 MiB/s, 43 objects/s recovering 2024-08-19T14:14:24.965 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:24 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3891423721' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:14:24.994 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:14:25.019 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:25.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:25.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:14:25.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:24 smithi045 ceph-mon[20143]: pgmap v386: 27 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+degraded+remapped, 25 active+clean; 7.5 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 169 B/s rd, 44 MiB/s wr, 832 op/s; 5/367506 objects degraded (0.001%); 2234/367506 objects misplaced (0.608%); 2.7 MiB/s, 43 objects/s recovering 2024-08-19T14:14:25.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:24 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3891423721' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:14:25.058 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:24 smithi032 ceph-mon[20004]: pgmap v386: 27 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+degraded+remapped, 25 active+clean; 7.5 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 169 B/s rd, 44 MiB/s wr, 832 op/s; 5/367506 objects degraded (0.001%); 2234/367506 objects misplaced (0.608%); 2.7 MiB/s, 43 objects/s recovering 2024-08-19T14:14:25.058 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:24 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3891423721' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:14:25.120 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:14:25.173 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:25.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:25.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:14:25.275 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:14:25.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:25.317 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:25.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:14:25.418 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:14:25.466 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:25.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:25.467 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:14:25.568 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:14:25.611 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:25.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:25.612 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:14:25.713 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:14:25.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:25.739 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:25.739 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:14:25.811 INFO:teuthology.orchestra.run.smithi031.stderr:reweighted osd.8 to 1 (10000) 2024-08-19T14:14:25.840 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:14:25.862 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:25.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:25.864 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:14:25.965 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:14:25.994 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:25.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:25.995 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:14:26.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:25 smithi045 ceph-mon[20143]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:14:26.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:25 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3891423721' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:14:26.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:25 smithi045 ceph-mon[20143]: osdmap e152: 12 total, 12 up, 10 in 2024-08-19T14:14:26.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:25 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1065407209' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-08-19T14:14:26.091 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:25 smithi031 ceph-mon[16336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:14:26.091 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:25 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3891423721' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:14:26.091 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:25 smithi031 ceph-mon[16336]: osdmap e152: 12 total, 12 up, 10 in 2024-08-19T14:14:26.091 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:25 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1065407209' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-08-19T14:14:26.096 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:14:26.130 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:26.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:26.131 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:14:26.232 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:14:26.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:25 smithi032 ceph-mon[20004]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:14:26.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:25 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3891423721' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:14:26.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:25 smithi032 ceph-mon[20004]: osdmap e152: 12 total, 12 up, 10 in 2024-08-19T14:14:26.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:25 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1065407209' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-08-19T14:14:26.263 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:26.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:26.264 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:14:26.365 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:14:26.388 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:26.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:26.390 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:14:26.491 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:14:26.521 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:26.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:26.522 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:14:26.623 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:14:26.651 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:26.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:26.652 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:14:26.753 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:14:26.779 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:26.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:26.780 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:14:26.833 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:14:26.861 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd reweight 9 1 2024-08-19T14:14:26.880 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:14:26.890 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:14:26.905 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:26.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:26.906 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:14:27.007 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:14:27.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:27.071 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:27.071 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:14:27.109 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:26 smithi031 ceph-mon[16336]: pgmap v388: 27 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+degraded+remapped, 25 active+clean; 7.4 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 145 B/s rd, 24 MiB/s wr, 646 op/s; 5/364311 objects degraded (0.001%); 2234/364311 objects misplaced (0.613%); 2.3 MiB/s, 37 objects/s recovering 2024-08-19T14:14:27.110 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:26 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1065407209' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-08-19T14:14:27.110 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:26 smithi031 ceph-mon[16336]: osdmap e153: 12 total, 12 up, 10 in 2024-08-19T14:14:27.172 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:14:27.195 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:27.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:27.196 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:14:27.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:26 smithi032 ceph-mon[20004]: pgmap v388: 27 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+degraded+remapped, 25 active+clean; 7.4 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 145 B/s rd, 24 MiB/s wr, 646 op/s; 5/364311 objects degraded (0.001%); 2234/364311 objects misplaced (0.613%); 2.3 MiB/s, 37 objects/s recovering 2024-08-19T14:14:27.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:26 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1065407209' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-08-19T14:14:27.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:26 smithi032 ceph-mon[20004]: osdmap e153: 12 total, 12 up, 10 in 2024-08-19T14:14:27.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:26 smithi045 ceph-mon[20143]: pgmap v388: 27 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+degraded+remapped, 25 active+clean; 7.4 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 145 B/s rd, 24 MiB/s wr, 646 op/s; 5/364311 objects degraded (0.001%); 2234/364311 objects misplaced (0.613%); 2.3 MiB/s, 37 objects/s recovering 2024-08-19T14:14:27.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:26 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1065407209' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-08-19T14:14:27.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:26 smithi045 ceph-mon[20143]: osdmap e153: 12 total, 12 up, 10 in 2024-08-19T14:14:27.297 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:14:27.321 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:27.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:27.322 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:14:27.423 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:14:27.448 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:27.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:27.449 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:14:27.551 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:14:27.574 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:27.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:27.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:14:27.676 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:14:27.699 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:27.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:27.700 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:14:27.718 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:14:27.719 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-08-19T14:14:27.719 INFO:teuthology.orchestra.run.smithi031.stdout: "complaint_time": 30, 2024-08-19T14:14:27.719 INFO:teuthology.orchestra.run.smithi031.stdout: "num_blocked_ops": 0 2024-08-19T14:14:27.719 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:14:27.801 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:14:27.827 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:27.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:27.828 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:14:27.929 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:14:27.963 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:27.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:27.964 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:14:27.967 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:14:27.968 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:14:27.968 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:27 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:14:27.968 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:27 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:14:27.968 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:27 smithi031 ceph-mon[16336]: osdmap e154: 12 total, 12 up, 10 in 2024-08-19T14:14:28.008 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:14:28.009 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:14:28.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:27 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:14:28.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:27 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:14:28.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:27 smithi045 ceph-mon[20143]: osdmap e154: 12 total, 12 up, 10 in 2024-08-19T14:14:28.049 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:14:28.050 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:14:28.065 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:14:28.089 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:28.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:28.090 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:14:28.094 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:14:28.095 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:14:28.129 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:14:28.129 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:14:28.137 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-19T14:14:28.162 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:14:28.162 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:14:28.191 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:14:28.194 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:14:28.195 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:14:28.213 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:28.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:28.214 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:14:28.229 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:14:28.229 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:14:28.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:27 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:14:28.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:27 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:14:28.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:27 smithi032 ceph-mon[20004]: osdmap e154: 12 total, 12 up, 10 in 2024-08-19T14:14:28.261 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:14:28.261 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:14:28.294 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:14:28.294 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:14:28.315 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:14:28.327 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:14:28.328 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:14:28.368 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:14:28.368 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:14:28.369 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:28.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:28.370 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:14:28.471 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:14:28.496 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:28.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:28.497 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:14:28.563 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-08-19T14:14:28.598 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:14:28.622 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:28.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:28.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:14:28.724 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:14:28.746 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:28.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:28.748 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:14:28.849 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:14:28.873 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:28.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:28.874 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:14:28.975 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:14:29.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:29.008 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:29.008 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:14:29.109 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:14:29.118 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:28 smithi031 ceph-mon[16336]: pgmap v391: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing+deep, 25 active+clean; 7.3 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 170 B/s rd, 6.1 MiB/s wr, 612 op/s; 277/360297 objects degraded (0.077%); 1550/360297 objects misplaced (0.430%); 2.1 MiB/s, 34 objects/s recovering 2024-08-19T14:14:29.118 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:28 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 277/360297 objects degraded (0.077%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-19T14:14:29.118 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:28 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/200843409' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:14:29.153 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:29.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:29.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:14:29.237 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:14:29.255 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:14:29.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:28 smithi032 ceph-mon[20004]: pgmap v391: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing+deep, 25 active+clean; 7.3 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 170 B/s rd, 6.1 MiB/s wr, 612 op/s; 277/360297 objects degraded (0.077%); 1550/360297 objects misplaced (0.430%); 2.1 MiB/s, 34 objects/s recovering 2024-08-19T14:14:29.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:28 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 277/360297 objects degraded (0.077%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-19T14:14:29.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:28 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/200843409' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:14:29.276 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:29.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:29.277 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:14:29.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:28 smithi045 ceph-mon[20143]: pgmap v391: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing+deep, 25 active+clean; 7.3 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 170 B/s rd, 6.1 MiB/s wr, 612 op/s; 277/360297 objects degraded (0.077%); 1550/360297 objects misplaced (0.430%); 2.1 MiB/s, 34 objects/s recovering 2024-08-19T14:14:29.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:28 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 277/360297 objects degraded (0.077%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-19T14:14:29.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:28 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/200843409' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:14:29.378 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:14:29.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:29.425 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:29.426 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:14:29.527 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:14:29.550 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:29.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:29.551 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:14:29.572 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:14:29.651 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:14:29.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:29.693 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:29.693 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:14:29.794 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:14:29.816 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:29.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:29.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:14:29.825 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:14:29.917 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:14:29.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:29.942 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:29.942 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:14:30.043 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:14:30.070 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:30.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:30.072 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:14:30.173 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:14:30.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:30.198 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:30.198 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:14:30.299 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:14:30.323 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:30.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:30.324 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:14:30.425 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:14:30.453 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:30.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:30.454 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:14:30.558 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:14:30.582 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:30.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:30.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:14:30.684 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:14:30.720 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:30.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:30.721 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:14:30.822 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:14:30.836 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is unset 2024-08-19T14:14:30.845 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:30.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:30.846 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:14:30.947 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:14:30.969 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:30.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:30.970 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:14:31.071 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:14:31.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:31.106 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:31.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:14:31.115 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:30 smithi031 ceph-mon[16336]: pgmap v392: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 7.2 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 352 KiB/s wr, 878 op/s; 277/351891 objects degraded (0.079%); 1550/351891 objects misplaced (0.440%); 2.1 MiB/s, 34 objects/s recovering 2024-08-19T14:14:31.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:30 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1118519519' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:14:31.198 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:14:31.208 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:14:31.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:30 smithi032 ceph-mon[20004]: pgmap v392: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 7.2 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 352 KiB/s wr, 878 op/s; 277/351891 objects degraded (0.079%); 1550/351891 objects misplaced (0.440%); 2.1 MiB/s, 34 objects/s recovering 2024-08-19T14:14:31.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:30 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1118519519' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:14:31.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:31.278 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:31.278 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:14:31.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:30 smithi045 ceph-mon[20143]: pgmap v392: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 7.2 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 352 KiB/s wr, 878 op/s; 277/351891 objects degraded (0.079%); 1550/351891 objects misplaced (0.440%); 2.1 MiB/s, 34 objects/s recovering 2024-08-19T14:14:31.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:30 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1118519519' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:14:31.382 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:14:31.407 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:31.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:31.408 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:14:31.509 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:14:31.572 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:31.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:31.574 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:14:31.674 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:14:31.700 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:31.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:31.701 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:14:31.802 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:14:31.823 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:31.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:31.824 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:14:31.925 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:14:31.946 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:31.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:31.947 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:14:32.048 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:14:32.078 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:32.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:32.079 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:14:32.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:31 smithi031 ceph-mon[16336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:14:32.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:31 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1118519519' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:14:32.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:31 smithi031 ceph-mon[16336]: osdmap e155: 12 total, 12 up, 10 in 2024-08-19T14:14:32.180 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:14:32.214 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:32.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:32.215 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:14:32.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:31 smithi032 ceph-mon[20004]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:14:32.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:31 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1118519519' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:14:32.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:31 smithi032 ceph-mon[20004]: osdmap e155: 12 total, 12 up, 10 in 2024-08-19T14:14:32.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:31 smithi045 ceph-mon[20143]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:14:32.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:31 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1118519519' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:14:32.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:31 smithi045 ceph-mon[20143]: osdmap e155: 12 total, 12 up, 10 in 2024-08-19T14:14:32.316 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:14:32.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:32.352 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:32.352 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:14:32.454 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:14:32.483 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:32.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:32.485 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:14:32.552 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:14:32.552 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:14:32.586 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:14:32.615 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:32.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:32.616 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:14:32.717 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:14:32.753 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:32.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:32.754 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:14:32.855 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:14:32.889 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:32.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:32.890 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:14:32.991 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:14:33.013 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:33.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:33.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:14:33.115 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:14:33.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:32 smithi031 ceph-mon[16336]: 1.2 scrub starts 2024-08-19T14:14:33.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:32 smithi031 ceph-mon[16336]: 1.2 scrub ok 2024-08-19T14:14:33.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:32 smithi031 ceph-mon[16336]: 1.a scrub starts 2024-08-19T14:14:33.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:32 smithi031 ceph-mon[16336]: 1.a scrub ok 2024-08-19T14:14:33.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:32 smithi031 ceph-mon[16336]: pgmap v394: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 7.0 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 1.06k op/s; 556/345261 objects degraded (0.161%); 851/345261 objects misplaced (0.246%); 4.4 MiB/s, 71 objects/s recovering 2024-08-19T14:14:33.138 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:33.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:33.140 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:14:33.240 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:14:33.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:32 smithi032 ceph-mon[20004]: 1.2 scrub starts 2024-08-19T14:14:33.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:32 smithi032 ceph-mon[20004]: 1.2 scrub ok 2024-08-19T14:14:33.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:32 smithi032 ceph-mon[20004]: 1.a scrub starts 2024-08-19T14:14:33.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:32 smithi032 ceph-mon[20004]: 1.a scrub ok 2024-08-19T14:14:33.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:32 smithi032 ceph-mon[20004]: pgmap v394: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 7.0 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 1.06k op/s; 556/345261 objects degraded (0.161%); 851/345261 objects misplaced (0.246%); 4.4 MiB/s, 71 objects/s recovering 2024-08-19T14:14:33.265 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:33.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:33.266 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:14:33.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:32 smithi045 ceph-mon[20143]: 1.2 scrub starts 2024-08-19T14:14:33.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:32 smithi045 ceph-mon[20143]: 1.2 scrub ok 2024-08-19T14:14:33.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:32 smithi045 ceph-mon[20143]: 1.a scrub starts 2024-08-19T14:14:33.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:32 smithi045 ceph-mon[20143]: 1.a scrub ok 2024-08-19T14:14:33.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:32 smithi045 ceph-mon[20143]: pgmap v394: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 7.0 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 1.06k op/s; 556/345261 objects degraded (0.161%); 851/345261 objects misplaced (0.246%); 4.4 MiB/s, 71 objects/s recovering 2024-08-19T14:14:33.367 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:14:33.393 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:33.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:33.394 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:14:33.495 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:14:33.523 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:33.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:33.525 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:14:33.626 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:14:33.659 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:33.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:33.660 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:14:33.761 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:14:33.794 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:33.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:33.795 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:14:33.896 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:14:33.921 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:33.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:33.922 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:14:34.023 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:14:34.048 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:34.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:34.049 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:14:34.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:33 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:14:34.149 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:14:34.177 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:34.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:34.178 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:14:34.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:33 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:14:34.280 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:14:34.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:33 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:14:34.306 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:34.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:34.307 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:14:34.408 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:14:34.447 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:34.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:34.448 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:14:34.549 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:14:34.573 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:34.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:34.575 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:14:34.675 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:14:34.699 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:34.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:34.700 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:14:34.801 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:14:34.832 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:34.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:34.833 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:14:34.934 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:14:34.957 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:34.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:34.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:14:35.059 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:14:35.091 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:35.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:35.092 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:14:35.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:34 smithi031 ceph-mon[16336]: pgmap v395: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 6.9 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 1.02k op/s; 556/339873 objects degraded (0.164%); 851/339873 objects misplaced (0.250%); 3.3 MiB/s, 54 objects/s recovering 2024-08-19T14:14:35.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:34 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 556/339873 objects degraded (0.164%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-19T14:14:35.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:34 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:14:35.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:34 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:14:35.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:34 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:14:35.193 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:14:35.218 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:35.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:35.219 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:14:35.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:34 smithi032 ceph-mon[20004]: pgmap v395: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 6.9 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 1.02k op/s; 556/339873 objects degraded (0.164%); 851/339873 objects misplaced (0.250%); 3.3 MiB/s, 54 objects/s recovering 2024-08-19T14:14:35.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:34 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 556/339873 objects degraded (0.164%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-19T14:14:35.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:34 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:14:35.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:34 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:14:35.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:34 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:14:35.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:34 smithi045 ceph-mon[20143]: pgmap v395: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 6.9 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 1.02k op/s; 556/339873 objects degraded (0.164%); 851/339873 objects misplaced (0.250%); 3.3 MiB/s, 54 objects/s recovering 2024-08-19T14:14:35.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:34 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 556/339873 objects degraded (0.164%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-19T14:14:35.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:34 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:14:35.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:34 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:14:35.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:34 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:14:35.320 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:14:35.347 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:35.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:35.348 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:14:35.397 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-08-19T14:14:35.449 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:14:35.473 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:35.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:35.474 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:14:35.575 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:14:35.626 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:35.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:35.627 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:14:35.728 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:14:35.750 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:35.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:35.751 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:14:35.852 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:14:35.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:35.894 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:35.895 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:14:35.995 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:14:36.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:36.019 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:36.020 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:14:36.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:35 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:14:36.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:35 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:14:36.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:35 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:14:36.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:35 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:14:36.120 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:14:36.144 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:36.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:36.145 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:14:36.247 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:14:36.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:35 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:14:36.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:35 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:14:36.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:35 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:14:36.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:35 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:14:36.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:36.294 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:36.295 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:14:36.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:35 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:14:36.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:35 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:14:36.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:35 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:14:36.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:35 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:14:36.396 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:14:36.424 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:36.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:36.425 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:14:36.526 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:14:36.552 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:36.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:36.553 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:14:36.644 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:14:36.654 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:14:36.676 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:36.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:36.678 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:14:36.709 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:14:36.778 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:14:36.804 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:36.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:36.805 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:14:36.873 INFO:teuthology.orchestra.run.smithi031.stderr:reweighted osd.9 to 1 (10000) 2024-08-19T14:14:36.906 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:14:36.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:36.950 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:36.950 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:14:37.052 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:14:37.088 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:37.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:37.089 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:14:37.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:36 smithi031 ceph-mon[16336]: 1.9 scrub starts 2024-08-19T14:14:37.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:36 smithi031 ceph-mon[16336]: 1.9 scrub ok 2024-08-19T14:14:37.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:36 smithi031 ceph-mon[16336]: 1.8 scrub starts 2024-08-19T14:14:37.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:36 smithi031 ceph-mon[16336]: 1.8 scrub ok 2024-08-19T14:14:37.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:36 smithi031 ceph-mon[16336]: pgmap v396: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 6.8 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 941 op/s; 556/335079 objects degraded (0.166%); 851/335079 objects misplaced (0.254%); 1.5 MiB/s, 25 objects/s recovering 2024-08-19T14:14:37.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:36 smithi031 ceph-mon[16336]: osdmap e156: 12 total, 12 up, 10 in 2024-08-19T14:14:37.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:36 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/652978828' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-08-19T14:14:37.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:36 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-08-19T14:14:37.190 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:14:37.215 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:37.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:37.216 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:14:37.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:36 smithi032 ceph-mon[20004]: 1.9 scrub starts 2024-08-19T14:14:37.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:36 smithi032 ceph-mon[20004]: 1.9 scrub ok 2024-08-19T14:14:37.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:36 smithi032 ceph-mon[20004]: 1.8 scrub starts 2024-08-19T14:14:37.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:36 smithi032 ceph-mon[20004]: 1.8 scrub ok 2024-08-19T14:14:37.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:36 smithi032 ceph-mon[20004]: pgmap v396: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 6.8 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 941 op/s; 556/335079 objects degraded (0.166%); 851/335079 objects misplaced (0.254%); 1.5 MiB/s, 25 objects/s recovering 2024-08-19T14:14:37.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:36 smithi032 ceph-mon[20004]: osdmap e156: 12 total, 12 up, 10 in 2024-08-19T14:14:37.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:36 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/652978828' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-08-19T14:14:37.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:36 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-08-19T14:14:37.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:36 smithi045 ceph-mon[20143]: 1.9 scrub starts 2024-08-19T14:14:37.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:36 smithi045 ceph-mon[20143]: 1.9 scrub ok 2024-08-19T14:14:37.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:36 smithi045 ceph-mon[20143]: 1.8 scrub starts 2024-08-19T14:14:37.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:36 smithi045 ceph-mon[20143]: 1.8 scrub ok 2024-08-19T14:14:37.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:36 smithi045 ceph-mon[20143]: pgmap v396: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 6.8 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 941 op/s; 556/335079 objects degraded (0.166%); 851/335079 objects misplaced (0.254%); 1.5 MiB/s, 25 objects/s recovering 2024-08-19T14:14:37.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:36 smithi045 ceph-mon[20143]: osdmap e156: 12 total, 12 up, 10 in 2024-08-19T14:14:37.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:36 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/652978828' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-08-19T14:14:37.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:36 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-08-19T14:14:37.317 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:14:37.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:37.343 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:37.343 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:14:37.444 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:14:37.480 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:37.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:37.481 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:14:37.582 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:14:37.600 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd reweight 11 1 2024-08-19T14:14:37.629 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:37.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:37.630 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:14:37.730 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:14:37.753 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:37.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:37.754 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:14:37.855 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:14:37.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:37.893 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:37.894 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:14:37.995 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:14:38.022 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:38.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:38.024 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:14:38.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:37 smithi045 ceph-mon[20143]: 1.0 scrub starts 2024-08-19T14:14:38.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:37 smithi045 ceph-mon[20143]: 1.0 scrub ok 2024-08-19T14:14:38.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:37 smithi045 ceph-mon[20143]: 1.4 scrub starts 2024-08-19T14:14:38.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:37 smithi045 ceph-mon[20143]: 1.4 scrub ok 2024-08-19T14:14:38.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:37 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-08-19T14:14:38.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:37 smithi045 ceph-mon[20143]: osdmap e157: 12 total, 12 up, 10 in 2024-08-19T14:14:38.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:37 smithi045 ceph-mon[20143]: 2.6 scrub starts 2024-08-19T14:14:38.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:37 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:14:38.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:37 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:14:38.125 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:14:38.146 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:38.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:38.147 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:14:38.195 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:37 smithi031 ceph-mon[16336]: 1.0 scrub starts 2024-08-19T14:14:38.195 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:37 smithi031 ceph-mon[16336]: 1.0 scrub ok 2024-08-19T14:14:38.195 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:37 smithi031 ceph-mon[16336]: 1.4 scrub starts 2024-08-19T14:14:38.195 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:37 smithi031 ceph-mon[16336]: 1.4 scrub ok 2024-08-19T14:14:38.195 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:37 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-08-19T14:14:38.195 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:37 smithi031 ceph-mon[16336]: osdmap e157: 12 total, 12 up, 10 in 2024-08-19T14:14:38.195 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:37 smithi031 ceph-mon[16336]: 2.6 scrub starts 2024-08-19T14:14:38.196 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:37 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:14:38.196 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:37 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:14:38.248 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:14:38.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:37 smithi032 ceph-mon[20004]: 1.0 scrub starts 2024-08-19T14:14:38.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:37 smithi032 ceph-mon[20004]: 1.0 scrub ok 2024-08-19T14:14:38.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:37 smithi032 ceph-mon[20004]: 1.4 scrub starts 2024-08-19T14:14:38.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:37 smithi032 ceph-mon[20004]: 1.4 scrub ok 2024-08-19T14:14:38.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:37 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-08-19T14:14:38.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:37 smithi032 ceph-mon[20004]: osdmap e157: 12 total, 12 up, 10 in 2024-08-19T14:14:38.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:37 smithi032 ceph-mon[20004]: 2.6 scrub starts 2024-08-19T14:14:38.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:37 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:14:38.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:37 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:14:38.270 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:38.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:38.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:14:38.372 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:14:38.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:38.397 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:38.397 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:14:38.499 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:14:38.525 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:38.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:38.526 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:14:38.597 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-19T14:14:38.626 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:14:38.679 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:38.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:38.680 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:14:38.781 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:14:38.805 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:38.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:38.806 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:14:38.903 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:14:38.904 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:14:38.906 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:14:38.931 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:38.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:38.932 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:14:39.033 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:14:39.056 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:39.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:39.057 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:14:39.158 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:14:39.181 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:39.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:39.182 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:14:39.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:38 smithi032 ceph-mon[20004]: pgmap v399: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 6.7 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 954 op/s; 574/330120 objects degraded (0.174%); 504/330120 objects misplaced (0.153%); 4.9 MiB/s, 79 objects/s recovering 2024-08-19T14:14:39.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:38 smithi032 ceph-mon[20004]: osdmap e158: 12 total, 12 up, 10 in 2024-08-19T14:14:39.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:38 smithi032 ceph-mon[20004]: 1.3 scrub starts 2024-08-19T14:14:39.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:38 smithi032 ceph-mon[20004]: 1.3 scrub ok 2024-08-19T14:14:39.283 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:14:39.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:38 smithi045 ceph-mon[20143]: pgmap v399: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 6.7 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 954 op/s; 574/330120 objects degraded (0.174%); 504/330120 objects misplaced (0.153%); 4.9 MiB/s, 79 objects/s recovering 2024-08-19T14:14:39.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:38 smithi045 ceph-mon[20143]: osdmap e158: 12 total, 12 up, 10 in 2024-08-19T14:14:39.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:38 smithi045 ceph-mon[20143]: 1.3 scrub starts 2024-08-19T14:14:39.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:38 smithi045 ceph-mon[20143]: 1.3 scrub ok 2024-08-19T14:14:39.306 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:39.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:39.307 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:14:39.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:38 smithi031 ceph-mon[16336]: pgmap v399: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 6.7 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 954 op/s; 574/330120 objects degraded (0.174%); 504/330120 objects misplaced (0.153%); 4.9 MiB/s, 79 objects/s recovering 2024-08-19T14:14:39.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:38 smithi031 ceph-mon[16336]: osdmap e158: 12 total, 12 up, 10 in 2024-08-19T14:14:39.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:38 smithi031 ceph-mon[16336]: 1.3 scrub starts 2024-08-19T14:14:39.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:38 smithi031 ceph-mon[16336]: 1.3 scrub ok 2024-08-19T14:14:39.408 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:14:39.434 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:39.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:39.435 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:14:39.536 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:14:39.557 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:39.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:39.558 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:14:39.620 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:14:39.620 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:14:39.658 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:14:39.659 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:14:39.659 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:14:39.687 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:39.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:39.688 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:14:39.690 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:14:39.690 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:14:39.735 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:14:39.735 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:14:39.770 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:14:39.770 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:14:39.789 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:14:39.803 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:14:39.804 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:14:39.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:39.823 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:39.824 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:14:39.836 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:14:39.837 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:14:39.869 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:14:39.870 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:14:39.904 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:14:39.904 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:14:39.925 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:14:39.943 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:14:39.943 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:14:39.970 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:39.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:39.971 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:14:39.980 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:14:39.981 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:14:40.013 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:14:40.013 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:14:40.014 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:39 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1980627192' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:14:40.072 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:14:40.096 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:40.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:40.097 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:14:40.120 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-08-19T14:14:40.197 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:14:40.221 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:40.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:40.222 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:14:40.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:39 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1980627192' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:14:40.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:39 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1980627192' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:14:40.323 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:14:40.370 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:40.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:40.371 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:14:40.472 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:14:40.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:40.519 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:40.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:14:40.621 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:14:40.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:40.658 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:40.658 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:14:40.759 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:14:40.785 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:40.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:40.786 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:14:40.843 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:14:40.888 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:14:40.911 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:40.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:40.912 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:14:40.920 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:14:41.002 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:40 smithi031 ceph-mon[16336]: pgmap v401: 27 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 25 active+clean; 6.5 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 1.01k op/s; 574/321705 objects degraded (0.178%); 504/321705 objects misplaced (0.157%); 3.4 MiB/s, 54 objects/s recovering 2024-08-19T14:14:41.002 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:40 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 574/321705 objects degraded (0.178%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-19T14:14:41.002 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:40 smithi031 ceph-mon[16336]: 2.6 scrub ok 2024-08-19T14:14:41.014 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:14:41.047 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:41.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:41.048 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:14:41.149 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:14:41.177 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:41.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:41.178 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:14:41.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:40 smithi032 ceph-mon[20004]: pgmap v401: 27 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 25 active+clean; 6.5 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 1.01k op/s; 574/321705 objects degraded (0.178%); 504/321705 objects misplaced (0.157%); 3.4 MiB/s, 54 objects/s recovering 2024-08-19T14:14:41.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:40 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 574/321705 objects degraded (0.178%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-19T14:14:41.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:40 smithi032 ceph-mon[20004]: 2.6 scrub ok 2024-08-19T14:14:41.279 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:14:41.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:40 smithi045 ceph-mon[20143]: pgmap v401: 27 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 25 active+clean; 6.5 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 1.01k op/s; 574/321705 objects degraded (0.178%); 504/321705 objects misplaced (0.157%); 3.4 MiB/s, 54 objects/s recovering 2024-08-19T14:14:41.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:40 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 574/321705 objects degraded (0.178%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-19T14:14:41.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:40 smithi045 ceph-mon[20143]: 2.6 scrub ok 2024-08-19T14:14:41.318 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:41.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:41.319 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:14:41.405 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:14:41.420 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:14:41.443 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:41.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:41.444 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:14:41.545 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:14:41.569 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:41.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:41.570 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:14:41.671 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:14:41.701 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:41.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:41.702 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:14:41.803 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:14:41.836 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:41.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:41.837 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:14:41.910 INFO:teuthology.orchestra.run.smithi031.stderr:reweighted osd.11 to 1 (10000) 2024-08-19T14:14:41.938 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:14:41.985 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:41.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:41.986 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:14:42.087 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:14:42.110 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:42.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:42.111 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:14:42.165 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:41 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1355942779' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-08-19T14:14:42.165 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:41 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-08-19T14:14:42.212 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:14:42.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:41 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1355942779' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-08-19T14:14:42.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:41 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-08-19T14:14:42.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:42.258 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:42.258 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:14:42.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:41 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1355942779' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-08-19T14:14:42.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:41 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-08-19T14:14:42.359 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:14:42.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:42.401 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:42.402 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:14:42.503 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:14:42.528 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:42.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:42.529 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:14:42.630 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:14:42.654 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:42.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:42.655 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:14:42.756 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:14:42.794 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:42.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:42.795 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:14:42.795 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:14:42.841 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-08-19T14:14:42.842 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-19T14:14:42.896 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:14:42.931 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is unset 2024-08-19T14:14:42.991 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:42.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:42.992 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:14:43.093 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:14:43.094 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:14:43.094 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-08-19T14:14:43.094 INFO:teuthology.orchestra.run.smithi031.stdout: "complaint_time": 30, 2024-08-19T14:14:43.094 INFO:teuthology.orchestra.run.smithi031.stdout: "num_blocked_ops": 0 2024-08-19T14:14:43.095 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:14:43.095 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:42 smithi031 ceph-mon[16336]: pgmap v402: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 25 active+clean; 6.4 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 1.09k op/s; 574/315471 objects degraded (0.182%); 158/315471 objects misplaced (0.050%); 7.0 MiB/s, 112 objects/s recovering 2024-08-19T14:14:43.095 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:42 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-08-19T14:14:43.095 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:42 smithi031 ceph-mon[16336]: osdmap e159: 12 total, 12 up, 10 in 2024-08-19T14:14:43.095 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:42 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/4285541216' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:14:43.095 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:42 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:14:43.118 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:43.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:43.119 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:14:43.220 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:14:43.248 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:43.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:43.249 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:14:43.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:42 smithi032 ceph-mon[20004]: pgmap v402: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 25 active+clean; 6.4 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 1.09k op/s; 574/315471 objects degraded (0.182%); 158/315471 objects misplaced (0.050%); 7.0 MiB/s, 112 objects/s recovering 2024-08-19T14:14:43.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:42 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-08-19T14:14:43.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:42 smithi032 ceph-mon[20004]: osdmap e159: 12 total, 12 up, 10 in 2024-08-19T14:14:43.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:42 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/4285541216' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:14:43.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:42 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:14:43.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:42 smithi045 ceph-mon[20143]: pgmap v402: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 25 active+clean; 6.4 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 1.09k op/s; 574/315471 objects degraded (0.182%); 158/315471 objects misplaced (0.050%); 7.0 MiB/s, 112 objects/s recovering 2024-08-19T14:14:43.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:42 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-08-19T14:14:43.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:42 smithi045 ceph-mon[20143]: osdmap e159: 12 total, 12 up, 10 in 2024-08-19T14:14:43.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:42 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/4285541216' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:14:43.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:42 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:14:43.350 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:14:43.372 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:43.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:43.373 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:14:43.474 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:14:43.499 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:43.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:43.501 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:14:43.602 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:14:43.634 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:43.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:43.635 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:14:43.735 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:14:43.761 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:43.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:43.762 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:14:43.863 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:14:43.900 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:43.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:43.901 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:14:44.003 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:14:44.029 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:44.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:44.030 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:14:44.131 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:14:44.154 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:44.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:44.155 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:14:44.256 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:14:44.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:43 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:14:44.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:43 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:14:44.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:43 smithi032 ceph-mon[20004]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:14:44.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:43 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:14:44.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:43 smithi032 ceph-mon[20004]: osdmap e160: 12 total, 12 up, 10 in 2024-08-19T14:14:44.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:43 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:14:44.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:43 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:14:44.284 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:44.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:44.285 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:14:44.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:43 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:14:44.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:43 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:14:44.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:43 smithi045 ceph-mon[20143]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:14:44.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:43 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:14:44.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:43 smithi045 ceph-mon[20143]: osdmap e160: 12 total, 12 up, 10 in 2024-08-19T14:14:44.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:43 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:14:44.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:43 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:14:44.355 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:43 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:14:44.356 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:43 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:14:44.356 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:43 smithi031 ceph-mon[16336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:14:44.356 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:43 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:14:44.356 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:43 smithi031 ceph-mon[16336]: osdmap e160: 12 total, 12 up, 10 in 2024-08-19T14:14:44.356 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:43 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:14:44.356 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:43 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:14:44.386 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:14:44.405 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-08-19T14:14:44.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:44.410 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:44.410 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:14:44.446 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:14:44.479 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:14:44.512 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:14:44.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:44.563 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:44.563 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:14:44.664 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:14:44.688 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:44.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:44.689 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:14:44.790 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:14:44.816 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:44.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:44.817 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:14:44.918 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:14:44.942 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:44.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:44.943 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:14:45.044 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:14:45.072 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:45.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:45.073 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:14:45.174 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:14:45.210 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:45.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:45.211 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:14:45.215 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:44 smithi031 ceph-mon[16336]: pgmap v405: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 6.3 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 1.11k op/s; 574/310026 objects degraded (0.185%); 158/310026 objects misplaced (0.051%); 3.6 MiB/s, 57 objects/s recovering 2024-08-19T14:14:45.215 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:44 smithi031 ceph-mon[16336]: osdmap e161: 12 total, 12 up, 10 in 2024-08-19T14:14:45.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:44 smithi032 ceph-mon[20004]: pgmap v405: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 6.3 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 1.11k op/s; 574/310026 objects degraded (0.185%); 158/310026 objects misplaced (0.051%); 3.6 MiB/s, 57 objects/s recovering 2024-08-19T14:14:45.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:44 smithi032 ceph-mon[20004]: osdmap e161: 12 total, 12 up, 10 in 2024-08-19T14:14:45.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:44 smithi045 ceph-mon[20143]: pgmap v405: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 6.3 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 1.11k op/s; 574/310026 objects degraded (0.185%); 158/310026 objects misplaced (0.051%); 3.6 MiB/s, 57 objects/s recovering 2024-08-19T14:14:45.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:44 smithi045 ceph-mon[20143]: osdmap e161: 12 total, 12 up, 10 in 2024-08-19T14:14:45.312 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:14:45.334 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:45.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:45.335 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:14:45.437 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:14:45.460 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:45.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:45.462 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:14:45.547 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:14:45.548 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-08-19T14:14:45.556 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:14:45.556 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:14:45.562 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:14:45.563 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:14:45.584 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:45.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:45.585 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:14:45.602 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:14:45.602 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:14:45.650 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:14:45.650 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:14:45.686 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:14:45.692 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:14:45.693 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:14:45.717 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-19T14:14:45.724 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:14:45.725 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:14:45.730 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:45.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:45.782 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:14:45.782 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:14:45.783 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:14:45.811 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:14:45.811 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:14:45.854 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:14:45.854 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:14:45.883 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:14:45.891 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:14:45.891 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:14:45.912 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:45.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:45.913 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:14:45.927 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:14:45.928 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:14:45.967 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:14:45.967 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:14:46.006 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:14:46.006 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:14:46.014 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:14:46.037 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:46.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:46.038 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:14:46.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:45 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:14:46.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:45 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:14:46.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:45 smithi031 ceph-mon[16336]: osdmap e162: 12 total, 12 up, 10 in 2024-08-19T14:14:46.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:45 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2489545142' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:14:46.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:45 smithi031 ceph-mon[16336]: from='client.24778 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:14:46.139 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:14:46.192 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:46.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:46.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:14:46.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:45 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:14:46.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:45 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:14:46.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:45 smithi032 ceph-mon[20004]: osdmap e162: 12 total, 12 up, 10 in 2024-08-19T14:14:46.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:45 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2489545142' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:14:46.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:45 smithi032 ceph-mon[20004]: from='client.24778 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:14:46.294 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:14:46.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:45 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:14:46.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:45 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:14:46.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:45 smithi045 ceph-mon[20143]: osdmap e162: 12 total, 12 up, 10 in 2024-08-19T14:14:46.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:45 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2489545142' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:14:46.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:45 smithi045 ceph-mon[20143]: from='client.24778 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:14:46.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:46.318 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:46.318 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:14:46.419 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:14:46.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:46.463 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:46.464 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:14:46.495 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":407,"stamp":"2024-08-19T14:14:44.943951+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6713156996,"num_objects":102427,"num_object_clones":0,"num_object_copies":307281,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":574,"num_objects_misplaced":158,"num_objects_unfound":0,"num_objects_dirty":102427,"num_whiteouts":0,"num_read":24,"num_read_kb":21,"num_write":144166,"num_write_kb":7890327,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13779,"num_bytes_recovered":902422179,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":146683,"ondisk_log_size":146683,"up":81,"acting":80,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":81,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":20564464,"kb_used_data":19767936,"kb_used_omap":15,"kb_used_meta":796144,"kb_avail":916887056,"statfs":{"total":959950356480,"available":938892345344,"internally_reserved":0,"allocated":20242366464,"data_stored":20234294867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15948,"internal_metadata":815251892},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,2,1,5,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-315097088,"num_objects":-4808,"num_object_clones":0,"num_object_copies":-14424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-346,"num_objects_unfound":0,"num_objects_dirty":-4808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4808,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":346,"num_bytes_recovered":22675456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.179723"},"pg_stats":[{"pgid":"2.f","version":"161'9211","reported_seq":9421,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-08-19T14:14:44.598628+0000","last_change":"2024-08-19T14:14:10.955308+0000","last_active":"2024-08-19T14:14:44.598628+0000","last_peered":"2024-08-19T14:14:44.598628+0000","last_clean":"2024-08-19T14:14:44.598628+0000","last_became_active":"2024-08-19T14:12:43.417591+0000","last_became_peered":"2024-08-19T14:12:43.417591+0000","last_unstale":"2024-08-19T14:14:44.598628+0000","last_undegraded":"2024-08-19T14:14:44.598628+0000","last_fullsized":"2024-08-19T14:14:44.598628+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'7296","last_scrub_stamp":"2024-08-19T14:14:10.955043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:14:10.955043+0000","objects_scrubbed":7248,"log_size":9211,"log_dups_size":0,"ondisk_log_size":9211,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:40.398760+0000","scrub_duration":1903,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":412549120,"num_objects":6295,"num_object_clones":0,"num_object_copies":18885,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9211,"num_write_kb":496192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.e","version":"158'8878","reported_seq":9001,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-08-19T14:14:41.020588+0000","last_change":"2024-08-19T14:12:42.963558+0000","last_active":"2024-08-19T14:14:41.020588+0000","last_peered":"2024-08-19T14:14:41.020588+0000","last_clean":"2024-08-19T14:14:41.020588+0000","last_became_active":"2024-08-19T14:12:42.963208+0000","last_became_peered":"2024-08-19T14:12:42.963208+0000","last_unstale":"2024-08-19T14:14:41.020588+0000","last_undegraded":"2024-08-19T14:14:41.020588+0000","last_fullsized":"2024-08-19T14:14:41.020588+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:12:41.713780+0000","objects_scrubbed":0,"log_size":8878,"log_dups_size":0,"ondisk_log_size":8878,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":424804352,"num_objects":6482,"num_object_clones":0,"num_object_copies":19446,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8878,"num_write_kb":491520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9],"acting":[2,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"160'9043","reported_seq":9251,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-08-19T14:14:43.107739+0000","last_change":"2024-08-19T14:14:11.678308+0000","last_active":"2024-08-19T14:14:43.107739+0000","last_peered":"2024-08-19T14:14:43.107739+0000","last_clean":"2024-08-19T14:14:43.107739+0000","last_became_active":"2024-08-19T14:12:43.417506+0000","last_became_peered":"2024-08-19T14:12:43.417506+0000","last_unstale":"2024-08-19T14:14:43.107739+0000","last_undegraded":"2024-08-19T14:14:43.107739+0000","last_fullsized":"2024-08-19T14:14:43.107739+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'7275","last_scrub_stamp":"2024-08-19T14:14:11.678069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:14:11.678069+0000","objects_scrubbed":7231,"log_size":9043,"log_dups_size":0,"ondisk_log_size":9043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:23.845013+0000","scrub_duration":2180,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":415498264,"num_objects":6341,"num_object_clones":0,"num_object_copies":19023,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6341,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9043,"num_write_kb":492225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.c","version":"158'8854","reported_seq":9062,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-08-19T14:14:40.461956+0000","last_change":"2024-08-19T14:14:11.208263+0000","last_active":"2024-08-19T14:14:40.461956+0000","last_peered":"2024-08-19T14:14:40.461956+0000","last_clean":"2024-08-19T14:14:40.461956+0000","last_became_active":"2024-08-19T14:12:43.038912+0000","last_became_peered":"2024-08-19T14:12:43.038912+0000","last_unstale":"2024-08-19T14:14:40.461956+0000","last_undegraded":"2024-08-19T14:14:40.461956+0000","last_fullsized":"2024-08-19T14:14:40.461956+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'7262","last_scrub_stamp":"2024-08-19T14:14:11.208131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:14:11.208131+0000","objects_scrubbed":7178,"log_size":8854,"log_dups_size":0,"ondisk_log_size":8854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:16.859224+0000","scrub_duration":3361,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":425197568,"num_objects":6488,"num_object_clones":0,"num_object_copies":19464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6488,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8854,"num_write_kb":490944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,1],"acting":[11,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"70'139","reported_seq":512,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-08-19T14:14:42.934099+0000","last_change":"2024-08-19T14:14:36.306513+0000","last_active":"2024-08-19T14:14:42.934099+0000","last_peered":"2024-08-19T14:14:42.934099+0000","last_clean":"2024-08-19T14:14:42.934099+0000","last_became_active":"2024-08-19T14:13:22.729498+0000","last_became_peered":"2024-08-19T14:13:22.729498+0000","last_unstale":"2024-08-19T14:14:42.934099+0000","last_undegraded":"2024-08-19T14:14:42.934099+0000","last_fullsized":"2024-08-19T14:14:42.934099+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:14:36.306409+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:14:36.306409+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:45.640981+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,9],"acting":[7,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"158'9000","reported_seq":9199,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-08-19T14:14:41.474922+0000","last_change":"2024-08-19T14:14:03.748630+0000","last_active":"2024-08-19T14:14:41.474922+0000","last_peered":"2024-08-19T14:14:41.474922+0000","last_clean":"2024-08-19T14:14:41.474922+0000","last_became_active":"2024-08-19T14:12:42.728978+0000","last_became_peered":"2024-08-19T14:12:42.728978+0000","last_unstale":"2024-08-19T14:14:41.474922+0000","last_undegraded":"2024-08-19T14:14:41.474922+0000","last_fullsized":"2024-08-19T14:14:41.474922+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"146'6847","last_scrub_stamp":"2024-08-19T14:14:03.748468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:14:03.748468+0000","objects_scrubbed":6806,"log_size":9000,"log_dups_size":0,"ondisk_log_size":9000,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:25.274768+0000","scrub_duration":1341,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":420085760,"num_objects":6410,"num_object_clones":0,"num_object_copies":19230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9000,"num_write_kb":493120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.3","version":"70'139","reported_seq":396,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-08-19T14:14:42.937894+0000","last_change":"2024-08-19T14:14:38.031170+0000","last_active":"2024-08-19T14:14:42.937894+0000","last_peered":"2024-08-19T14:14:42.937894+0000","last_clean":"2024-08-19T14:14:42.937894+0000","last_became_active":"2024-08-19T14:13:22.727234+0000","last_became_peered":"2024-08-19T14:13:22.727234+0000","last_unstale":"2024-08-19T14:14:42.937894+0000","last_undegraded":"2024-08-19T14:14:42.937894+0000","last_fullsized":"2024-08-19T14:14:42.937894+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:14:38.031056+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:14:38.031056+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:16:03.171231+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"159'8972","reported_seq":15225,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-08-19T14:14:42.934101+0000","last_change":"2024-08-19T14:13:45.917116+0000","last_active":"2024-08-19T14:14:42.934101+0000","last_peered":"2024-08-19T14:14:42.934101+0000","last_clean":"2024-08-19T14:14:42.934101+0000","last_became_active":"2024-08-19T14:13:32.749994+0000","last_became_peered":"2024-08-19T14:13:32.749994+0000","last_unstale":"2024-08-19T14:14:42.934101+0000","last_undegraded":"2024-08-19T14:14:42.934101+0000","last_fullsized":"2024-08-19T14:14:42.934101+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"136'5761","last_scrub_stamp":"2024-08-19T14:13:45.916860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:13:45.916860+0000","objects_scrubbed":5716,"log_size":8972,"log_dups_size":0,"ondisk_log_size":8972,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:09.768267+0000","scrub_duration":1249,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":423755776,"num_objects":6466,"num_object_clones":0,"num_object_copies":19398,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8972,"num_write_kb":494016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3024,"num_bytes_recovered":198180864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,11],"acting":[7,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"70'139","reported_seq":510,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-08-19T14:14:42.934067+0000","last_change":"2024-08-19T14:14:31.408564+0000","last_active":"2024-08-19T14:14:42.934067+0000","last_peered":"2024-08-19T14:14:42.934067+0000","last_clean":"2024-08-19T14:14:42.934067+0000","last_became_active":"2024-08-19T14:13:05.607783+0000","last_became_peered":"2024-08-19T14:13:05.607783+0000","last_unstale":"2024-08-19T14:14:42.934067+0000","last_undegraded":"2024-08-19T14:14:42.934067+0000","last_fullsized":"2024-08-19T14:14:42.934067+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:14:31.408492+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:14:31.408492+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:41.655086+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"158'8929","reported_seq":9052,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-08-19T14:14:40.981444+0000","last_change":"2024-08-19T14:12:43.039198+0000","last_active":"2024-08-19T14:14:40.981444+0000","last_peered":"2024-08-19T14:14:40.981444+0000","last_clean":"2024-08-19T14:14:40.981444+0000","last_became_active":"2024-08-19T14:12:43.038833+0000","last_became_peered":"2024-08-19T14:12:43.038833+0000","last_unstale":"2024-08-19T14:14:40.981444+0000","last_undegraded":"2024-08-19T14:14:40.981444+0000","last_fullsized":"2024-08-19T14:14:40.981444+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:12:41.713780+0000","objects_scrubbed":0,"log_size":8929,"log_dups_size":0,"ondisk_log_size":8929,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":431030272,"num_objects":6577,"num_object_clones":0,"num_object_copies":19731,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8929,"num_write_kb":496192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"70'139","reported_seq":376,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-08-19T14:14:42.931986+0000","last_change":"2024-08-19T14:14:07.456593+0000","last_active":"2024-08-19T14:14:42.931986+0000","last_peered":"2024-08-19T14:14:42.931986+0000","last_clean":"2024-08-19T14:14:42.931986+0000","last_became_active":"2024-08-19T14:13:11.663185+0000","last_became_peered":"2024-08-19T14:13:11.663185+0000","last_unstale":"2024-08-19T14:14:42.931986+0000","last_undegraded":"2024-08-19T14:14:42.931986+0000","last_fullsized":"2024-08-19T14:14:42.931986+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:14:07.456464+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:14:07.456464+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:09.321073+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2],"acting":[9,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.2","version":"158'8980","reported_seq":17000,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-08-19T14:14:41.449490+0000","last_change":"2024-08-19T14:14:13.196736+0000","last_active":"2024-08-19T14:14:41.449490+0000","last_peered":"2024-08-19T14:14:41.449490+0000","last_clean":"2024-08-19T14:14:41.449490+0000","last_became_active":"2024-08-19T14:13:58.623348+0000","last_became_peered":"2024-08-19T14:13:58.623348+0000","last_unstale":"2024-08-19T14:14:41.449490+0000","last_undegraded":"2024-08-19T14:14:41.449490+0000","last_fullsized":"2024-08-19T14:14:41.449490+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"148'7390","last_scrub_stamp":"2024-08-19T14:14:13.196497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:14:13.196497+0000","objects_scrubbed":7346,"log_size":8980,"log_dups_size":0,"ondisk_log_size":8980,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:26.733911+0000","scrub_duration":1724,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":423100416,"num_objects":6456,"num_object_clones":0,"num_object_copies":19368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6456,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8980,"num_write_kb":493952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3902,"num_bytes_recovered":255721472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"70'139","reported_seq":353,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-08-19T14:14:42.961511+0000","last_change":"2024-08-19T14:13:22.739488+0000","last_active":"2024-08-19T14:14:42.961511+0000","last_peered":"2024-08-19T14:14:42.961511+0000","last_clean":"2024-08-19T14:14:42.961511+0000","last_became_active":"2024-08-19T14:13:22.739150+0000","last_became_peered":"2024-08-19T14:13:22.739150+0000","last_unstale":"2024-08-19T14:14:42.961511+0000","last_undegraded":"2024-08-19T14:14:42.961511+0000","last_fullsized":"2024-08-19T14:14:42.961511+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:13:06.062715+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:13:06.062715+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":16,"num_write_kb":267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":192337,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"159'9071","reported_seq":15537,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-08-19T14:14:42.938189+0000","last_change":"2024-08-19T14:14:09.998977+0000","last_active":"2024-08-19T14:14:42.938189+0000","last_peered":"2024-08-19T14:14:42.938189+0000","last_clean":"2024-08-19T14:14:42.938189+0000","last_became_active":"2024-08-19T14:14:04.681118+0000","last_became_peered":"2024-08-19T14:14:04.681118+0000","last_unstale":"2024-08-19T14:14:42.938189+0000","last_undegraded":"2024-08-19T14:14:42.938189+0000","last_fullsized":"2024-08-19T14:14:42.938189+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'7194","last_scrub_stamp":"2024-08-19T14:14:09.998849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:14:09.998849+0000","objects_scrubbed":7135,"log_size":9071,"log_dups_size":0,"ondisk_log_size":9071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:22.243440+0000","scrub_duration":2842,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":417923072,"num_objects":6377,"num_object_clones":0,"num_object_copies":19131,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9071,"num_write_kb":494336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3123,"num_bytes_recovered":204668928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"70'139","reported_seq":376,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-08-19T14:14:42.932097+0000","last_change":"2024-08-19T14:14:06.461983+0000","last_active":"2024-08-19T14:14:42.932097+0000","last_peered":"2024-08-19T14:14:42.932097+0000","last_clean":"2024-08-19T14:14:42.932097+0000","last_became_active":"2024-08-19T14:13:11.647085+0000","last_became_peered":"2024-08-19T14:13:11.647085+0000","last_unstale":"2024-08-19T14:14:42.932097+0000","last_undegraded":"2024-08-19T14:14:42.932097+0000","last_fullsized":"2024-08-19T14:14:42.932097+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:14:06.461855+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:14:06.461855+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:12.802907+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,4],"acting":[9,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.5","version":"161'9167","reported_seq":9379,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-08-19T14:14:44.643758+0000","last_change":"2024-08-19T14:14:14.222353+0000","last_active":"2024-08-19T14:14:44.643758+0000","last_peered":"2024-08-19T14:14:44.643758+0000","last_clean":"2024-08-19T14:14:44.643758+0000","last_became_active":"2024-08-19T14:12:43.417616+0000","last_became_peered":"2024-08-19T14:12:43.417616+0000","last_unstale":"2024-08-19T14:14:44.643758+0000","last_undegraded":"2024-08-19T14:14:44.643758+0000","last_fullsized":"2024-08-19T14:14:44.643758+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"148'7426","last_scrub_stamp":"2024-08-19T14:14:14.222198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:14:14.222198+0000","objects_scrubbed":7364,"log_size":9167,"log_dups_size":0,"ondisk_log_size":9167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:35.064129+0000","scrub_duration":2125,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":411500544,"num_objects":6279,"num_object_clones":0,"num_object_copies":18837,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6279,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9167,"num_write_kb":494272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.5","version":"70'139","reported_seq":373,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-08-19T14:14:37.898848+0000","last_change":"2024-08-19T14:13:22.727452+0000","last_active":"2024-08-19T14:14:37.898848+0000","last_peered":"2024-08-19T14:14:37.898848+0000","last_clean":"2024-08-19T14:14:37.898848+0000","last_became_active":"2024-08-19T14:13:22.727098+0000","last_became_peered":"2024-08-19T14:13:22.727098+0000","last_unstale":"2024-08-19T14:14:37.898848+0000","last_undegraded":"2024-08-19T14:14:37.898848+0000","last_fullsized":"2024-08-19T14:14:37.898848+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:12:04.853650+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:12:04.853650+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:14:42.847671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":17,"num_write_kb":267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":192338,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"159'10210","reported_seq":10482,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-08-19T14:14:42.978546+0000","last_change":"2024-08-19T14:14:40.326683+0000","last_active":"2024-08-19T14:14:42.978546+0000","last_peered":"2024-08-19T14:14:42.978546+0000","last_clean":"2024-08-19T14:14:42.978546+0000","last_became_active":"2024-08-19T14:12:42.963203+0000","last_became_peered":"2024-08-19T14:12:42.963203+0000","last_unstale":"2024-08-19T14:14:42.978546+0000","last_undegraded":"2024-08-19T14:14:42.978546+0000","last_fullsized":"2024-08-19T14:14:42.978546+0000","mapping_epoch":102,"log_start":"106'200","ondisk_log_start":"106'200","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'10039","last_scrub_stamp":"2024-08-19T14:14:40.326519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:14:40.326519+0000","objects_scrubbed":6619,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:43.279737+0000","scrub_duration":3261,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":415694848,"num_objects":6343,"num_object_clones":0,"num_object_copies":19029,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6343,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8989,"num_write_kb":490624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,3],"acting":[1,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"70'139","reported_seq":431,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-08-19T14:14:37.897792+0000","last_change":"2024-08-19T14:14:36.831290+0000","last_active":"2024-08-19T14:14:37.897792+0000","last_peered":"2024-08-19T14:14:37.897792+0000","last_clean":"2024-08-19T14:14:37.897792+0000","last_became_active":"2024-08-19T14:13:05.607447+0000","last_became_peered":"2024-08-19T14:13:05.607447+0000","last_unstale":"2024-08-19T14:14:37.897792+0000","last_undegraded":"2024-08-19T14:14:37.897792+0000","last_fullsized":"2024-08-19T14:14:37.897792+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:14:36.831171+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:14:36.831171+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:53.437142+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.7","version":"158'8916","reported_seq":18894,"reported_epoch":158,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-19T14:14:41.031079+0000","last_change":"2024-08-19T14:14:20.865609+0000","last_active":"2024-08-19T14:14:41.031079+0000","last_peered":"2024-08-19T14:14:41.031079+0000","last_clean":"2024-08-19T14:13:07.604184+0000","last_became_active":"2024-08-19T14:13:20.813460+0000","last_became_peered":"2024-08-19T14:13:20.813460+0000","last_unstale":"2024-08-19T14:14:41.031079+0000","last_undegraded":"2024-08-19T14:14:20.858729+0000","last_fullsized":"2024-08-19T14:13:20.722594+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:12:41.713780+0000","objects_scrubbed":0,"log_size":8916,"log_dups_size":0,"ondisk_log_size":8916,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":426115072,"num_objects":6502,"num_object_clones":0,"num_object_copies":19506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":574,"num_objects_misplaced":158,"num_objects_unfound":0,"num_objects_dirty":6502,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8916,"num_write_kb":493376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3728,"num_bytes_recovered":243466240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3],"avail_no_missing":["2","3","6","10"],"object_location_counts":[{"shards":"2,3","objects":5770},{"shards":"2,3,6","objects":158}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.8","version":"160'9115","reported_seq":9322,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-08-19T14:14:42.934040+0000","last_change":"2024-08-19T14:14:08.982261+0000","last_active":"2024-08-19T14:14:42.934040+0000","last_peered":"2024-08-19T14:14:42.934040+0000","last_clean":"2024-08-19T14:14:42.934040+0000","last_became_active":"2024-08-19T14:12:43.038882+0000","last_became_peered":"2024-08-19T14:12:43.038882+0000","last_unstale":"2024-08-19T14:14:42.934040+0000","last_undegraded":"2024-08-19T14:14:42.934040+0000","last_fullsized":"2024-08-19T14:14:42.934040+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'7200","last_scrub_stamp":"2024-08-19T14:14:08.982137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:14:08.982137+0000","objects_scrubbed":7142,"log_size":9115,"log_dups_size":0,"ondisk_log_size":9115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:09.792768+0000","scrub_duration":1562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":418971648,"num_objects":6393,"num_object_clones":0,"num_object_copies":19179,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9115,"num_write_kb":496256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,0],"acting":[7,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"70'139","reported_seq":397,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-08-19T14:14:42.934366+0000","last_change":"2024-08-19T14:14:31.607909+0000","last_active":"2024-08-19T14:14:42.934366+0000","last_peered":"2024-08-19T14:14:42.934366+0000","last_clean":"2024-08-19T14:14:42.934366+0000","last_became_active":"2024-08-19T14:13:22.728779+0000","last_became_peered":"2024-08-19T14:13:22.728779+0000","last_unstale":"2024-08-19T14:14:42.934366+0000","last_undegraded":"2024-08-19T14:14:42.934366+0000","last_fullsized":"2024-08-19T14:14:42.934366+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:14:31.607795+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:14:31.607795+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:48.816381+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,0],"acting":[9,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"159'8941","reported_seq":9144,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-08-19T14:14:42.942533+0000","last_change":"2024-08-19T14:14:05.393154+0000","last_active":"2024-08-19T14:14:42.942533+0000","last_peered":"2024-08-19T14:14:42.942533+0000","last_clean":"2024-08-19T14:14:42.942533+0000","last_became_active":"2024-08-19T14:12:42.963114+0000","last_became_peered":"2024-08-19T14:12:42.963114+0000","last_unstale":"2024-08-19T14:14:42.942533+0000","last_undegraded":"2024-08-19T14:14:42.942533+0000","last_fullsized":"2024-08-19T14:14:42.942533+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'6812","last_scrub_stamp":"2024-08-19T14:14:05.392787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:14:05.392787+0000","objects_scrubbed":6777,"log_size":8941,"log_dups_size":0,"ondisk_log_size":8941,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:16.463202+0000","scrub_duration":1333,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":410583040,"num_objects":6265,"num_object_clones":0,"num_object_copies":18795,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8941,"num_write_kb":486592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"70'139","reported_seq":510,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-08-19T14:14:42.934017+0000","last_change":"2024-08-19T14:14:35.319452+0000","last_active":"2024-08-19T14:14:42.934017+0000","last_peered":"2024-08-19T14:14:42.934017+0000","last_clean":"2024-08-19T14:14:42.934017+0000","last_became_active":"2024-08-19T14:13:05.607605+0000","last_became_peered":"2024-08-19T14:13:05.607605+0000","last_unstale":"2024-08-19T14:14:42.934017+0000","last_undegraded":"2024-08-19T14:14:42.934017+0000","last_fullsized":"2024-08-19T14:14:42.934017+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:14:35.319333+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:14:35.319333+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:46.300019+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"158'8913","reported_seq":9117,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-08-19T14:14:40.533012+0000","last_change":"2024-08-19T14:14:11.437361+0000","last_active":"2024-08-19T14:14:40.533012+0000","last_peered":"2024-08-19T14:14:40.533012+0000","last_clean":"2024-08-19T14:14:40.533012+0000","last_became_active":"2024-08-19T14:12:42.963040+0000","last_became_peered":"2024-08-19T14:12:42.963040+0000","last_unstale":"2024-08-19T14:14:40.533012+0000","last_undegraded":"2024-08-19T14:14:40.533012+0000","last_fullsized":"2024-08-19T14:14:40.533012+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'7289","last_scrub_stamp":"2024-08-19T14:14:11.437152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:14:11.437152+0000","objects_scrubbed":7227,"log_size":8913,"log_dups_size":0,"ondisk_log_size":8913,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:32.239860+0000","scrub_duration":2564,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":427622400,"num_objects":6525,"num_object_clones":0,"num_object_copies":19575,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8913,"num_write_kb":494016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,9],"acting":[11,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.8","version":"70'139","reported_seq":384,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-08-19T14:14:37.891850+0000","last_change":"2024-08-19T14:14:35.594757+0000","last_active":"2024-08-19T14:14:37.891850+0000","last_peered":"2024-08-19T14:14:37.891850+0000","last_clean":"2024-08-19T14:14:37.891850+0000","last_became_active":"2024-08-19T14:13:05.605694+0000","last_became_peered":"2024-08-19T14:13:05.605694+0000","last_unstale":"2024-08-19T14:14:37.891850+0000","last_undegraded":"2024-08-19T14:14:37.891850+0000","last_fullsized":"2024-08-19T14:14:37.891850+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:14:35.594637+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:14:35.594637+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:43.962595+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.b","version":"161'9154","reported_seq":9360,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-08-19T14:14:44.641040+0000","last_change":"2024-08-19T14:14:06.266442+0000","last_active":"2024-08-19T14:14:44.641040+0000","last_peered":"2024-08-19T14:14:44.641040+0000","last_clean":"2024-08-19T14:14:44.641040+0000","last_became_active":"2024-08-19T14:12:42.729014+0000","last_became_peered":"2024-08-19T14:12:42.729014+0000","last_unstale":"2024-08-19T14:14:44.641040+0000","last_undegraded":"2024-08-19T14:14:44.641040+0000","last_fullsized":"2024-08-19T14:14:44.641040+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'6965","last_scrub_stamp":"2024-08-19T14:14:06.266214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:14:06.266214+0000","objects_scrubbed":6919,"log_size":9154,"log_dups_size":0,"ondisk_log_size":9154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:18.095310+0000","scrub_duration":1296,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":408027136,"num_objects":6226,"num_object_clones":0,"num_object_copies":18678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9154,"num_write_kb":492160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":6712459288,"num_objects":102425,"num_object_clones":0,"num_object_copies":307275,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":574,"num_objects_misplaced":158,"num_objects_unfound":0,"num_objects_dirty":102425,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":144133,"num_write_kb":7889793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13777,"num_bytes_recovered":902037504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20740517888,"data_stored":20740505672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":145154,"ondisk_log_size":145154,"up":48,"acting":47,"num_store_stats":12},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":697708,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":23,"num_read_kb":20,"num_write":33,"num_write_kb":534,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":384675,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1529,"ondisk_log_size":1529,"up":33,"acting":33,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187496,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1759656,"kb_used_data":1684056,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":91985496,"statfs":{"total":95995035648,"available":94193147904,"internally_reserved":0,"allocated":1724473344,"data_stored":1723672328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":77396420},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":343597383725,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":74,"seq":317827579954,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2988212,"kb_used_data":2876520,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":90756940,"statfs":{"total":95995035648,"available":92935106560,"internally_reserved":0,"allocated":2945556480,"data_stored":2944747093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":114358721},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874296,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1679176,"kb_used_data":1611900,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":92065976,"statfs":{"total":95995035648,"available":94275559424,"internally_reserved":0,"allocated":1650585600,"data_stored":1649771771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":57,"seq":244813135930,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2116784,"kb_used_data":2049508,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":91628368,"statfs":{"total":95995035648,"available":93827448832,"internally_reserved":0,"allocated":2098696192,"data_stored":2097890877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":68876746},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":206158430270,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691906,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":880424,"kb_used_data":831200,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":92864728,"statfs":{"total":95995035648,"available":95093481472,"internally_reserved":0,"allocated":851148800,"data_stored":850344232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986247,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1716980,"kb_used_data":1641256,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":92028172,"statfs":{"total":95995035648,"available":94236848128,"internally_reserved":0,"allocated":1680646144,"data_stored":1679836757,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149773,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2956168,"kb_used_data":2844476,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":90788984,"statfs":{"total":95995035648,"available":92967919616,"internally_reserved":0,"allocated":2912743424,"data_stored":2911926228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":114358727},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411409,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2196072,"kb_used_data":2084320,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":91549080,"statfs":{"total":95995035648,"available":93746257920,"internally_reserved":0,"allocated":2134343680,"data_stored":2133539112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574934,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2112176,"kb_used_data":2048996,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":91632976,"statfs":{"total":95995035648,"available":93832167424,"internally_reserved":0,"allocated":2098171904,"data_stored":2097366589,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,1,1,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738458,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2158816,"kb_used_data":2095704,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":91586336,"statfs":{"total":95995035648,"available":93784408064,"internally_reserved":0,"allocated":2146000896,"data_stored":2145199880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2140864512,"data_stored":2140864512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2077818880,"data_stored":2077818880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2127757312,"data_stored":2127757312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2897285120,"data_stored":2897281048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1675169792,"data_stored":1675165720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":843186176,"data_stored":843186176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":416940032,"data_stored":416940032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2083651584,"data_stored":2083651584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1641152512,"data_stored":1641152512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":2926972928,"data_stored":2926968856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":198115328,"data_stored":198115328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1711603712,"data_stored":1711603712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T14:14:46.497 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-19T14:14:46.564 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:14:46.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:46.588 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:46.588 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:14:46.689 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:14:46.711 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:46.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:46.712 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:14:46.813 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:14:46.836 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:46.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:46.837 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:14:46.938 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:14:46.988 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:46.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:46.989 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:14:47.090 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:14:47.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:46 smithi031 ceph-mon[16336]: pgmap v408: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 6.2 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 835 op/s; 574/305439 objects degraded (0.188%); 158/305439 objects misplaced (0.052%) 2024-08-19T14:14:47.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:46 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 574/305439 objects degraded (0.188%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-19T14:14:47.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:46 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:14:47.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:46 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:14:47.119 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:47.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:47.120 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:14:47.221 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:14:47.248 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:47.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:47.249 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:14:47.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:46 smithi032 ceph-mon[20004]: pgmap v408: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 6.2 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 835 op/s; 574/305439 objects degraded (0.188%); 158/305439 objects misplaced (0.052%) 2024-08-19T14:14:47.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:46 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 574/305439 objects degraded (0.188%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-19T14:14:47.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:46 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:14:47.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:46 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:14:47.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:46 smithi045 ceph-mon[20143]: pgmap v408: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 6.2 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 835 op/s; 574/305439 objects degraded (0.188%); 158/305439 objects misplaced (0.052%) 2024-08-19T14:14:47.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:46 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 574/305439 objects degraded (0.188%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-19T14:14:47.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:46 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:14:47.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:46 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:14:47.350 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:14:47.359 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:14:47.359 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:14:47.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:47.376 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:47.376 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:14:47.478 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:14:47.512 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:47.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:47.513 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:14:47.614 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:14:47.647 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:47.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:47.648 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:14:47.661 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:14:47.749 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:14:47.784 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:47.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:47.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:14:47.887 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:14:47.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:47.909 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:47.910 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:14:48.011 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:14:48.035 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:48.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:48.036 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:14:48.137 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:14:48.161 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-08-19T14:14:48.186 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:48.190 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:14:48.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:48.190 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:14:48.235 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:47 smithi031 ceph-mon[16336]: osdmap e163: 12 total, 12 up, 10 in 2024-08-19T14:14:48.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:47 smithi032 ceph-mon[20004]: osdmap e163: 12 total, 12 up, 10 in 2024-08-19T14:14:48.291 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:14:48.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:47 smithi045 ceph-mon[20143]: osdmap e163: 12 total, 12 up, 10 in 2024-08-19T14:14:48.313 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:48.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:48.314 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:14:48.415 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:14:48.438 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:48.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:48.439 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:14:48.540 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:14:48.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:48.579 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:48.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:14:48.680 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:14:48.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:48.707 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:48.708 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:14:48.808 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:14:48.832 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:48.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:48.833 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:14:48.935 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:14:48.959 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:48.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:48.960 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:14:49.046 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:48 smithi031 ceph-mon[16336]: pgmap v410: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 6.1 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 970 op/s; 268/301383 objects degraded (0.089%); 2.0 MiB/s, 95 objects/s recovering 2024-08-19T14:14:49.046 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:48 smithi031 ceph-mon[16336]: 1.7 scrub starts 2024-08-19T14:14:49.046 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:48 smithi031 ceph-mon[16336]: 1.7 scrub ok 2024-08-19T14:14:49.061 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:14:49.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:49.108 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:49.108 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:14:49.209 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:14:49.232 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:49.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:49.233 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:14:49.242 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:14:49.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:48 smithi032 ceph-mon[20004]: pgmap v410: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 6.1 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 970 op/s; 268/301383 objects degraded (0.089%); 2.0 MiB/s, 95 objects/s recovering 2024-08-19T14:14:49.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:48 smithi032 ceph-mon[20004]: 1.7 scrub starts 2024-08-19T14:14:49.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:48 smithi032 ceph-mon[20004]: 1.7 scrub ok 2024-08-19T14:14:49.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:48 smithi045 ceph-mon[20143]: pgmap v410: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 6.1 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 970 op/s; 268/301383 objects degraded (0.089%); 2.0 MiB/s, 95 objects/s recovering 2024-08-19T14:14:49.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:48 smithi045 ceph-mon[20143]: 1.7 scrub starts 2024-08-19T14:14:49.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:48 smithi045 ceph-mon[20143]: 1.7 scrub ok 2024-08-19T14:14:49.304 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:14:49.334 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:14:49.356 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:49.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:49.357 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:14:49.437 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:14:49.438 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-08-19T14:14:49.457 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:14:49.478 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:49.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:49.480 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:14:49.580 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:14:49.601 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:49.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:49.602 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:14:49.703 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:14:49.727 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:49.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:49.728 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:14:49.828 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:14:49.852 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:49.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:49.854 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:14:49.956 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:14:49.972 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:14:49.972 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:14:49.982 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:49.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:49.983 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:14:49.986 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is set 2024-08-19T14:14:50.084 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:14:50.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:50.117 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:50.118 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:14:50.218 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:14:50.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:50.255 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:50.255 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:14:50.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:49 smithi032 ceph-mon[20004]: osdmap e164: 12 total, 12 up, 10 in 2024-08-19T14:14:50.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:49 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1771527988' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:14:50.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:49 smithi032 ceph-mon[20004]: from='client.14970 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:14:50.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:49 smithi032 ceph-mon[20004]: 1.5 scrub starts 2024-08-19T14:14:50.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:49 smithi032 ceph-mon[20004]: 1.5 scrub ok 2024-08-19T14:14:50.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:49 smithi045 ceph-mon[20143]: osdmap e164: 12 total, 12 up, 10 in 2024-08-19T14:14:50.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:49 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1771527988' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:14:50.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:49 smithi045 ceph-mon[20143]: from='client.14970 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:14:50.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:49 smithi045 ceph-mon[20143]: 1.5 scrub starts 2024-08-19T14:14:50.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:49 smithi045 ceph-mon[20143]: 1.5 scrub ok 2024-08-19T14:14:50.356 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:14:50.359 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:49 smithi031 ceph-mon[16336]: osdmap e164: 12 total, 12 up, 10 in 2024-08-19T14:14:50.359 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:49 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1771527988' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:14:50.359 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:49 smithi031 ceph-mon[16336]: from='client.14970 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:14:50.359 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:49 smithi031 ceph-mon[16336]: 1.5 scrub starts 2024-08-19T14:14:50.360 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:49 smithi031 ceph-mon[16336]: 1.5 scrub ok 2024-08-19T14:14:50.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:50.395 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:50.395 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:14:50.497 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:14:50.519 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:50.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:50.520 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:14:50.621 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:14:50.644 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:50.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:50.644 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:14:50.651 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-08-19T14:14:50.745 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:14:50.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:50.771 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:50.771 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:14:50.873 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:14:50.895 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:50.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:50.896 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:14:50.997 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:14:51.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:51.047 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:51.048 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:14:51.150 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:14:51.173 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:51.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:51.174 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:14:51.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:50 smithi032 ceph-mon[20004]: pgmap v412: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 6.0 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 916 op/s; 268/293952 objects degraded (0.091%); 1.7 MiB/s, 79 objects/s recovering 2024-08-19T14:14:51.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:50 smithi032 ceph-mon[20004]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:14:51.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:50 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1771527988' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:14:51.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:50 smithi032 ceph-mon[20004]: osdmap e165: 12 total, 12 up, 10 in 2024-08-19T14:14:51.275 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:14:51.280 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:50 smithi031 ceph-mon[16336]: pgmap v412: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 6.0 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 916 op/s; 268/293952 objects degraded (0.091%); 1.7 MiB/s, 79 objects/s recovering 2024-08-19T14:14:51.280 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:50 smithi031 ceph-mon[16336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:14:51.281 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:50 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1771527988' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:14:51.281 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:50 smithi031 ceph-mon[16336]: osdmap e165: 12 total, 12 up, 10 in 2024-08-19T14:14:51.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:50 smithi045 ceph-mon[20143]: pgmap v412: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 6.0 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 916 op/s; 268/293952 objects degraded (0.091%); 1.7 MiB/s, 79 objects/s recovering 2024-08-19T14:14:51.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:50 smithi045 ceph-mon[20143]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:14:51.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:50 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1771527988' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:14:51.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:50 smithi045 ceph-mon[20143]: osdmap e165: 12 total, 12 up, 10 in 2024-08-19T14:14:51.334 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:51.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:51.335 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:14:51.436 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:14:51.467 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:51.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:51.468 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:14:51.569 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:14:51.595 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:51.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:51.596 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:14:51.697 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:14:51.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:51.719 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:51.719 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:14:51.729 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":411,"stamp":"2024-08-19T14:14:48.983142+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6473372638,"num_objects":98769,"num_object_clones":0,"num_object_copies":296307,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":268,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98769,"num_whiteouts":0,"num_read":13,"num_read_kb":11,"num_write":147808,"num_write_kb":7890060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14242,"num_bytes_recovered":912584530,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150141,"ondisk_log_size":150141,"up":81,"acting":80,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":81,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":19747976,"kb_used_data":18962584,"kb_used_omap":15,"kb_used_meta":785072,"kb_avail":917703544,"statfs":{"total":959950356480,"available":939728429056,"internally_reserved":0,"allocated":19417686016,"data_stored":19409469513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15948,"internal_metadata":803914164},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,6,2,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-299749798,"num_objects":-4573,"num_object_clones":0,"num_object_copies":-13719,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-306,"num_objects_misplaced":-158,"num_objects_unfound":0,"num_objects_dirty":-4573,"num_whiteouts":0,"num_read":-11,"num_read_kb":-10,"num_write":4557,"num_write_kb":-267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":463,"num_bytes_recovered":10162351,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.217976"},"pg_stats":[{"pgid":"2.f","version":"161'9211","reported_seq":9421,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-08-19T14:14:44.598628+0000","last_change":"2024-08-19T14:14:10.955308+0000","last_active":"2024-08-19T14:14:44.598628+0000","last_peered":"2024-08-19T14:14:44.598628+0000","last_clean":"2024-08-19T14:14:44.598628+0000","last_became_active":"2024-08-19T14:12:43.417591+0000","last_became_peered":"2024-08-19T14:12:43.417591+0000","last_unstale":"2024-08-19T14:14:44.598628+0000","last_undegraded":"2024-08-19T14:14:44.598628+0000","last_fullsized":"2024-08-19T14:14:44.598628+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'7296","last_scrub_stamp":"2024-08-19T14:14:10.955043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:14:10.955043+0000","objects_scrubbed":7248,"log_size":9211,"log_dups_size":0,"ondisk_log_size":9211,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:40.398760+0000","scrub_duration":1903,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":412549120,"num_objects":6295,"num_object_clones":0,"num_object_copies":18885,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9211,"num_write_kb":496192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.e","version":"162'9159","reported_seq":9290,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-08-19T14:14:46.003370+0000","last_change":"2024-08-19T14:12:42.963558+0000","last_active":"2024-08-19T14:14:46.003370+0000","last_peered":"2024-08-19T14:14:46.003370+0000","last_clean":"2024-08-19T14:14:46.003370+0000","last_became_active":"2024-08-19T14:12:42.963208+0000","last_became_peered":"2024-08-19T14:12:42.963208+0000","last_unstale":"2024-08-19T14:14:46.003370+0000","last_undegraded":"2024-08-19T14:14:46.003370+0000","last_fullsized":"2024-08-19T14:14:46.003370+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:12:41.713780+0000","objects_scrubbed":0,"log_size":9159,"log_dups_size":0,"ondisk_log_size":9159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":406388736,"num_objects":6201,"num_object_clones":0,"num_object_copies":18603,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6201,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9159,"num_write_kb":491520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9],"acting":[2,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"163'9336","reported_seq":9550,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-08-19T14:14:48.179585+0000","last_change":"2024-08-19T14:14:11.678308+0000","last_active":"2024-08-19T14:14:48.179585+0000","last_peered":"2024-08-19T14:14:48.179585+0000","last_clean":"2024-08-19T14:14:48.179585+0000","last_became_active":"2024-08-19T14:12:43.417506+0000","last_became_peered":"2024-08-19T14:12:43.417506+0000","last_unstale":"2024-08-19T14:14:48.179585+0000","last_undegraded":"2024-08-19T14:14:48.179585+0000","last_fullsized":"2024-08-19T14:14:48.179585+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'7275","last_scrub_stamp":"2024-08-19T14:14:11.678069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:14:11.678069+0000","objects_scrubbed":7231,"log_size":9336,"log_dups_size":0,"ondisk_log_size":9336,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:23.845013+0000","scrub_duration":2180,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":396296216,"num_objects":6048,"num_object_clones":0,"num_object_copies":18144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6048,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9336,"num_write_kb":492225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.c","version":"162'9159","reported_seq":9375,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-08-19T14:14:45.442020+0000","last_change":"2024-08-19T14:14:11.208263+0000","last_active":"2024-08-19T14:14:45.442020+0000","last_peered":"2024-08-19T14:14:45.442020+0000","last_clean":"2024-08-19T14:14:45.442020+0000","last_became_active":"2024-08-19T14:12:43.038912+0000","last_became_peered":"2024-08-19T14:12:43.038912+0000","last_unstale":"2024-08-19T14:14:45.442020+0000","last_undegraded":"2024-08-19T14:14:45.442020+0000","last_fullsized":"2024-08-19T14:14:45.442020+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'7262","last_scrub_stamp":"2024-08-19T14:14:11.208131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:14:11.208131+0000","objects_scrubbed":7178,"log_size":9159,"log_dups_size":0,"ondisk_log_size":9159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:16.859224+0000","scrub_duration":3361,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":405209088,"num_objects":6183,"num_object_clones":0,"num_object_copies":18549,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9159,"num_write_kb":490944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,1],"acting":[11,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"70'139","reported_seq":518,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-08-19T14:14:46.957763+0000","last_change":"2024-08-19T14:14:36.306513+0000","last_active":"2024-08-19T14:14:46.957763+0000","last_peered":"2024-08-19T14:14:46.957763+0000","last_clean":"2024-08-19T14:14:46.957763+0000","last_became_active":"2024-08-19T14:13:22.729498+0000","last_became_peered":"2024-08-19T14:13:22.729498+0000","last_unstale":"2024-08-19T14:14:46.957763+0000","last_undegraded":"2024-08-19T14:14:46.957763+0000","last_fullsized":"2024-08-19T14:14:46.957763+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:14:36.306409+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:14:36.306409+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:45.640981+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,9],"acting":[7,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"162'9255","reported_seq":9462,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-08-19T14:14:46.479873+0000","last_change":"2024-08-19T14:14:03.748630+0000","last_active":"2024-08-19T14:14:46.479873+0000","last_peered":"2024-08-19T14:14:46.479873+0000","last_clean":"2024-08-19T14:14:46.479873+0000","last_became_active":"2024-08-19T14:12:42.728978+0000","last_became_peered":"2024-08-19T14:12:42.728978+0000","last_unstale":"2024-08-19T14:14:46.479873+0000","last_undegraded":"2024-08-19T14:14:46.479873+0000","last_fullsized":"2024-08-19T14:14:46.479873+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"146'6847","last_scrub_stamp":"2024-08-19T14:14:03.748468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:14:03.748468+0000","objects_scrubbed":6806,"log_size":9255,"log_dups_size":0,"ondisk_log_size":9255,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:25.274768+0000","scrub_duration":1341,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403374080,"num_objects":6155,"num_object_clones":0,"num_object_copies":18465,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6155,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9255,"num_write_kb":493120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.3","version":"70'139","reported_seq":402,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-08-19T14:14:46.976414+0000","last_change":"2024-08-19T14:14:38.031170+0000","last_active":"2024-08-19T14:14:46.976414+0000","last_peered":"2024-08-19T14:14:46.976414+0000","last_clean":"2024-08-19T14:14:46.976414+0000","last_became_active":"2024-08-19T14:13:22.727234+0000","last_became_peered":"2024-08-19T14:13:22.727234+0000","last_unstale":"2024-08-19T14:14:46.976414+0000","last_undegraded":"2024-08-19T14:14:46.976414+0000","last_fullsized":"2024-08-19T14:14:46.976414+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:14:38.031056+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:14:38.031056+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:16:03.171231+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"163'9262","reported_seq":15521,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-08-19T14:14:47.870029+0000","last_change":"2024-08-19T14:13:45.917116+0000","last_active":"2024-08-19T14:14:47.870029+0000","last_peered":"2024-08-19T14:14:47.870029+0000","last_clean":"2024-08-19T14:14:47.870029+0000","last_became_active":"2024-08-19T14:13:32.749994+0000","last_became_peered":"2024-08-19T14:13:32.749994+0000","last_unstale":"2024-08-19T14:14:47.870029+0000","last_undegraded":"2024-08-19T14:14:47.870029+0000","last_fullsized":"2024-08-19T14:14:47.870029+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"136'5761","last_scrub_stamp":"2024-08-19T14:13:45.916860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:13:45.916860+0000","objects_scrubbed":5716,"log_size":9262,"log_dups_size":0,"ondisk_log_size":9262,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:09.768267+0000","scrub_duration":1249,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":404750336,"num_objects":6176,"num_object_clones":0,"num_object_copies":18528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6176,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9262,"num_write_kb":494016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3024,"num_bytes_recovered":198180864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,11],"acting":[7,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"70'139","reported_seq":516,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-08-19T14:14:46.957687+0000","last_change":"2024-08-19T14:14:31.408564+0000","last_active":"2024-08-19T14:14:46.957687+0000","last_peered":"2024-08-19T14:14:46.957687+0000","last_clean":"2024-08-19T14:14:46.957687+0000","last_became_active":"2024-08-19T14:13:05.607783+0000","last_became_peered":"2024-08-19T14:13:05.607783+0000","last_unstale":"2024-08-19T14:14:46.957687+0000","last_undegraded":"2024-08-19T14:14:46.957687+0000","last_fullsized":"2024-08-19T14:14:46.957687+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:14:31.408492+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:14:31.408492+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:41.655086+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"162'9228","reported_seq":9358,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-08-19T14:14:46.032540+0000","last_change":"2024-08-19T14:12:43.039198+0000","last_active":"2024-08-19T14:14:46.032540+0000","last_peered":"2024-08-19T14:14:46.032540+0000","last_clean":"2024-08-19T14:14:46.032540+0000","last_became_active":"2024-08-19T14:12:43.038833+0000","last_became_peered":"2024-08-19T14:12:43.038833+0000","last_unstale":"2024-08-19T14:14:46.032540+0000","last_undegraded":"2024-08-19T14:14:46.032540+0000","last_fullsized":"2024-08-19T14:14:46.032540+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:12:41.713780+0000","objects_scrubbed":0,"log_size":9228,"log_dups_size":0,"ondisk_log_size":9228,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":411435008,"num_objects":6278,"num_object_clones":0,"num_object_copies":18834,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9228,"num_write_kb":496192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"70'139","reported_seq":382,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-08-19T14:14:46.958836+0000","last_change":"2024-08-19T14:14:07.456593+0000","last_active":"2024-08-19T14:14:46.958836+0000","last_peered":"2024-08-19T14:14:46.958836+0000","last_clean":"2024-08-19T14:14:46.958836+0000","last_became_active":"2024-08-19T14:13:11.663185+0000","last_became_peered":"2024-08-19T14:13:11.663185+0000","last_unstale":"2024-08-19T14:14:46.958836+0000","last_undegraded":"2024-08-19T14:14:46.958836+0000","last_fullsized":"2024-08-19T14:14:46.958836+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:14:07.456464+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:14:07.456464+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:09.321073+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2],"acting":[9,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.2","version":"162'9243","reported_seq":17271,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-08-19T14:14:46.496971+0000","last_change":"2024-08-19T14:14:13.196736+0000","last_active":"2024-08-19T14:14:46.496971+0000","last_peered":"2024-08-19T14:14:46.496971+0000","last_clean":"2024-08-19T14:14:46.496971+0000","last_became_active":"2024-08-19T14:13:58.623348+0000","last_became_peered":"2024-08-19T14:13:58.623348+0000","last_unstale":"2024-08-19T14:14:46.496971+0000","last_undegraded":"2024-08-19T14:14:46.496971+0000","last_fullsized":"2024-08-19T14:14:46.496971+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"148'7390","last_scrub_stamp":"2024-08-19T14:14:13.196497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:14:13.196497+0000","objects_scrubbed":7346,"log_size":9243,"log_dups_size":0,"ondisk_log_size":9243,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:26.733911+0000","scrub_duration":1724,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":405864448,"num_objects":6193,"num_object_clones":0,"num_object_copies":18579,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9243,"num_write_kb":493952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3902,"num_bytes_recovered":255721472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"70'139","reported_seq":367,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-08-19T14:14:48.488445+0000","last_change":"2024-08-19T14:14:48.488485+0000","last_active":"2024-08-19T14:14:48.488445+0000","last_peered":"2024-08-19T14:14:48.488445+0000","last_clean":"2024-08-19T14:14:48.488445+0000","last_became_active":"2024-08-19T14:13:22.739150+0000","last_became_peered":"2024-08-19T14:13:22.739150+0000","last_unstale":"2024-08-19T14:14:48.488445+0000","last_undegraded":"2024-08-19T14:14:48.488445+0000","last_fullsized":"2024-08-19T14:14:48.488445+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:14:48.488367+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:14:48.488367+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:16:00.551219+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"163'9335","reported_seq":15807,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-08-19T14:14:47.876367+0000","last_change":"2024-08-19T14:14:09.998977+0000","last_active":"2024-08-19T14:14:47.876367+0000","last_peered":"2024-08-19T14:14:47.876367+0000","last_clean":"2024-08-19T14:14:47.876367+0000","last_became_active":"2024-08-19T14:14:04.681118+0000","last_became_peered":"2024-08-19T14:14:04.681118+0000","last_unstale":"2024-08-19T14:14:47.876367+0000","last_undegraded":"2024-08-19T14:14:47.876367+0000","last_fullsized":"2024-08-19T14:14:47.876367+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'7194","last_scrub_stamp":"2024-08-19T14:14:09.998849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:14:09.998849+0000","objects_scrubbed":7135,"log_size":9335,"log_dups_size":0,"ondisk_log_size":9335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:22.243440+0000","scrub_duration":2842,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400621568,"num_objects":6113,"num_object_clones":0,"num_object_copies":18339,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9335,"num_write_kb":494336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3123,"num_bytes_recovered":204668928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"70'139","reported_seq":382,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-08-19T14:14:46.958902+0000","last_change":"2024-08-19T14:14:06.461983+0000","last_active":"2024-08-19T14:14:46.958902+0000","last_peered":"2024-08-19T14:14:46.958902+0000","last_clean":"2024-08-19T14:14:46.958902+0000","last_became_active":"2024-08-19T14:13:11.647085+0000","last_became_peered":"2024-08-19T14:13:11.647085+0000","last_unstale":"2024-08-19T14:14:46.958902+0000","last_undegraded":"2024-08-19T14:14:46.958902+0000","last_fullsized":"2024-08-19T14:14:46.958902+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:14:06.461855+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:14:06.461855+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:12.802907+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,4],"acting":[9,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.5","version":"161'9167","reported_seq":9379,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-08-19T14:14:44.643758+0000","last_change":"2024-08-19T14:14:14.222353+0000","last_active":"2024-08-19T14:14:44.643758+0000","last_peered":"2024-08-19T14:14:44.643758+0000","last_clean":"2024-08-19T14:14:44.643758+0000","last_became_active":"2024-08-19T14:12:43.417616+0000","last_became_peered":"2024-08-19T14:12:43.417616+0000","last_unstale":"2024-08-19T14:14:44.643758+0000","last_undegraded":"2024-08-19T14:14:44.643758+0000","last_fullsized":"2024-08-19T14:14:44.643758+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"148'7426","last_scrub_stamp":"2024-08-19T14:14:14.222198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:14:14.222198+0000","objects_scrubbed":7364,"log_size":9167,"log_dups_size":0,"ondisk_log_size":9167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:35.064129+0000","scrub_duration":2125,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":411500544,"num_objects":6279,"num_object_clones":0,"num_object_copies":18837,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6279,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9167,"num_write_kb":494272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.5","version":"70'139","reported_seq":381,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-08-19T14:14:44.949910+0000","last_change":"2024-08-19T14:13:22.727452+0000","last_active":"2024-08-19T14:14:44.949910+0000","last_peered":"2024-08-19T14:14:44.949910+0000","last_clean":"2024-08-19T14:14:44.949910+0000","last_became_active":"2024-08-19T14:13:22.727098+0000","last_became_peered":"2024-08-19T14:13:22.727098+0000","last_unstale":"2024-08-19T14:14:44.949910+0000","last_undegraded":"2024-08-19T14:14:44.949910+0000","last_fullsized":"2024-08-19T14:14:44.949910+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:12:04.853650+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:12:04.853650+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:14:48.710473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":17,"num_write_kb":267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":192338,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"163'10486","reported_seq":10764,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-08-19T14:14:47.887332+0000","last_change":"2024-08-19T14:14:40.326683+0000","last_active":"2024-08-19T14:14:47.887332+0000","last_peered":"2024-08-19T14:14:47.887332+0000","last_clean":"2024-08-19T14:14:47.887332+0000","last_became_active":"2024-08-19T14:12:42.963203+0000","last_became_peered":"2024-08-19T14:12:42.963203+0000","last_unstale":"2024-08-19T14:14:47.887332+0000","last_undegraded":"2024-08-19T14:14:47.887332+0000","last_fullsized":"2024-08-19T14:14:47.887332+0000","mapping_epoch":102,"log_start":"107'400","ondisk_log_start":"107'400","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'10039","last_scrub_stamp":"2024-08-19T14:14:40.326519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:14:40.326519+0000","objects_scrubbed":6619,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:43.279737+0000","scrub_duration":3261,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":397606912,"num_objects":6067,"num_object_clones":0,"num_object_copies":18201,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6067,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9265,"num_write_kb":490624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,3],"acting":[1,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"70'139","reported_seq":439,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-08-19T14:14:44.948919+0000","last_change":"2024-08-19T14:14:36.831290+0000","last_active":"2024-08-19T14:14:44.948919+0000","last_peered":"2024-08-19T14:14:44.948919+0000","last_clean":"2024-08-19T14:14:44.948919+0000","last_became_active":"2024-08-19T14:13:05.607447+0000","last_became_peered":"2024-08-19T14:13:05.607447+0000","last_unstale":"2024-08-19T14:14:44.948919+0000","last_undegraded":"2024-08-19T14:14:44.948919+0000","last_fullsized":"2024-08-19T14:14:44.948919+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:14:36.831171+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:14:36.831171+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:53.437142+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.7","version":"162'9170","reported_seq":20087,"reported_epoch":162,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-19T14:14:46.031881+0000","last_change":"2024-08-19T14:14:20.865609+0000","last_active":"2024-08-19T14:14:46.031881+0000","last_peered":"2024-08-19T14:14:46.031881+0000","last_clean":"2024-08-19T14:13:07.604184+0000","last_became_active":"2024-08-19T14:13:20.813460+0000","last_became_peered":"2024-08-19T14:13:20.813460+0000","last_unstale":"2024-08-19T14:14:46.031881+0000","last_undegraded":"2024-08-19T14:14:20.858729+0000","last_fullsized":"2024-08-19T14:13:20.722594+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:12:41.713780+0000","objects_scrubbed":0,"log_size":9170,"log_dups_size":0,"ondisk_log_size":9170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":409468928,"num_objects":6248,"num_object_clones":0,"num_object_copies":18744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":268,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6248,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9170,"num_write_kb":493376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":253820928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3],"avail_no_missing":["2","3","6","10"],"object_location_counts":[{"shards":"2,3","objects":5980}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.8","version":"163'9397","reported_seq":9610,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-08-19T14:14:47.847436+0000","last_change":"2024-08-19T14:14:08.982261+0000","last_active":"2024-08-19T14:14:47.847436+0000","last_peered":"2024-08-19T14:14:47.847436+0000","last_clean":"2024-08-19T14:14:47.847436+0000","last_became_active":"2024-08-19T14:12:43.038882+0000","last_became_peered":"2024-08-19T14:12:43.038882+0000","last_unstale":"2024-08-19T14:14:47.847436+0000","last_undegraded":"2024-08-19T14:14:47.847436+0000","last_fullsized":"2024-08-19T14:14:47.847436+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'7200","last_scrub_stamp":"2024-08-19T14:14:08.982137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:14:08.982137+0000","objects_scrubbed":7142,"log_size":9397,"log_dups_size":0,"ondisk_log_size":9397,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:09.792768+0000","scrub_duration":1562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400490496,"num_objects":6111,"num_object_clones":0,"num_object_copies":18333,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6111,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9397,"num_write_kb":496256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,0],"acting":[7,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"70'139","reported_seq":403,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-08-19T14:14:46.959008+0000","last_change":"2024-08-19T14:14:31.607909+0000","last_active":"2024-08-19T14:14:46.959008+0000","last_peered":"2024-08-19T14:14:46.959008+0000","last_clean":"2024-08-19T14:14:46.959008+0000","last_became_active":"2024-08-19T14:13:22.728779+0000","last_became_peered":"2024-08-19T14:13:22.728779+0000","last_unstale":"2024-08-19T14:14:46.959008+0000","last_undegraded":"2024-08-19T14:14:46.959008+0000","last_fullsized":"2024-08-19T14:14:46.959008+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:14:31.607795+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:14:31.607795+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:48.816381+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,0],"acting":[9,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"163'9228","reported_seq":9437,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-08-19T14:14:47.867751+0000","last_change":"2024-08-19T14:14:05.393154+0000","last_active":"2024-08-19T14:14:47.867751+0000","last_peered":"2024-08-19T14:14:47.867751+0000","last_clean":"2024-08-19T14:14:47.867751+0000","last_became_active":"2024-08-19T14:12:42.963114+0000","last_became_peered":"2024-08-19T14:12:42.963114+0000","last_unstale":"2024-08-19T14:14:47.867751+0000","last_undegraded":"2024-08-19T14:14:47.867751+0000","last_fullsized":"2024-08-19T14:14:47.867751+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'6812","last_scrub_stamp":"2024-08-19T14:14:05.392787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:14:05.392787+0000","objects_scrubbed":6777,"log_size":9228,"log_dups_size":0,"ondisk_log_size":9228,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:16.463202+0000","scrub_duration":1333,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":391774208,"num_objects":5978,"num_object_clones":0,"num_object_copies":17934,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9228,"num_write_kb":486592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"70'139","reported_seq":516,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-08-19T14:14:46.957759+0000","last_change":"2024-08-19T14:14:35.319452+0000","last_active":"2024-08-19T14:14:46.957759+0000","last_peered":"2024-08-19T14:14:46.957759+0000","last_clean":"2024-08-19T14:14:46.957759+0000","last_became_active":"2024-08-19T14:13:05.607605+0000","last_became_peered":"2024-08-19T14:13:05.607605+0000","last_unstale":"2024-08-19T14:14:46.957759+0000","last_undegraded":"2024-08-19T14:14:46.957759+0000","last_fullsized":"2024-08-19T14:14:46.957759+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:14:35.319333+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:14:35.319333+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:46.300019+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"162'9222","reported_seq":9434,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-08-19T14:14:45.524193+0000","last_change":"2024-08-19T14:14:11.437361+0000","last_active":"2024-08-19T14:14:45.524193+0000","last_peered":"2024-08-19T14:14:45.524193+0000","last_clean":"2024-08-19T14:14:45.524193+0000","last_became_active":"2024-08-19T14:12:42.963040+0000","last_became_peered":"2024-08-19T14:12:42.963040+0000","last_unstale":"2024-08-19T14:14:45.524193+0000","last_undegraded":"2024-08-19T14:14:45.524193+0000","last_fullsized":"2024-08-19T14:14:45.524193+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'7289","last_scrub_stamp":"2024-08-19T14:14:11.437152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:14:11.437152+0000","objects_scrubbed":7227,"log_size":9222,"log_dups_size":0,"ondisk_log_size":9222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:32.239860+0000","scrub_duration":2564,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":407371776,"num_objects":6216,"num_object_clones":0,"num_object_copies":18648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9222,"num_write_kb":494016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,9],"acting":[11,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.8","version":"70'139","reported_seq":392,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-08-19T14:14:44.958105+0000","last_change":"2024-08-19T14:14:35.594757+0000","last_active":"2024-08-19T14:14:44.958105+0000","last_peered":"2024-08-19T14:14:44.958105+0000","last_clean":"2024-08-19T14:14:44.958105+0000","last_became_active":"2024-08-19T14:13:05.605694+0000","last_became_peered":"2024-08-19T14:13:05.605694+0000","last_unstale":"2024-08-19T14:14:44.958105+0000","last_undegraded":"2024-08-19T14:14:44.958105+0000","last_fullsized":"2024-08-19T14:14:44.958105+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:14:35.594637+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:14:35.594637+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:43.962595+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.b","version":"161'9154","reported_seq":9360,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-08-19T14:14:44.641040+0000","last_change":"2024-08-19T14:14:06.266442+0000","last_active":"2024-08-19T14:14:44.641040+0000","last_peered":"2024-08-19T14:14:44.641040+0000","last_clean":"2024-08-19T14:14:44.641040+0000","last_became_active":"2024-08-19T14:12:42.729014+0000","last_became_peered":"2024-08-19T14:12:42.729014+0000","last_unstale":"2024-08-19T14:14:44.641040+0000","last_undegraded":"2024-08-19T14:14:44.641040+0000","last_fullsized":"2024-08-19T14:14:44.641040+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'6965","last_scrub_stamp":"2024-08-19T14:14:06.266214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:14:06.266214+0000","objects_scrubbed":6919,"log_size":9154,"log_dups_size":0,"ondisk_log_size":9154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:18.095310+0000","scrub_duration":1296,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":408027136,"num_objects":6226,"num_object_clones":0,"num_object_copies":18678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9154,"num_write_kb":492160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":6472728600,"num_objects":98767,"num_object_clones":0,"num_object_copies":296301,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":268,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98767,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":147791,"num_write_kb":7889793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14241,"num_bytes_recovered":912392192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19929575424,"data_stored":19929563208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":148612,"ondisk_log_size":148612,"up":48,"acting":47,"num_store_stats":12},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":644038,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":17,"num_write_kb":267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":192338,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1529,"ondisk_log_size":1529,"up":33,"acting":33,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187497,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1666380,"kb_used_data":1603260,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":92078772,"statfs":{"total":95995035648,"available":94288662528,"internally_reserved":0,"allocated":1641738240,"data_stored":1640924411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":64616900},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":343597383726,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":74,"seq":317827579955,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2851168,"kb_used_data":2747800,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":90893984,"statfs":{"total":95995035648,"available":93075439616,"internally_reserved":0,"allocated":2813747200,"data_stored":2812920785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":105839041},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874296,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1679176,"kb_used_data":1611900,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":92065976,"statfs":{"total":95995035648,"available":94275559424,"internally_reserved":0,"allocated":1650585600,"data_stored":1649771771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":57,"seq":244813135931,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2029344,"kb_used_data":1957908,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":91715808,"statfs":{"total":95995035648,"available":93916987392,"internally_reserved":0,"allocated":2004897792,"data_stored":2004075449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73136586},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":206158430270,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691907,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":850656,"kb_used_data":797264,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":92894496,"statfs":{"total":95995035648,"available":95123963904,"internally_reserved":0,"allocated":816398336,"data_stored":815576698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986248,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1636960,"kb_used_data":1569688,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":92108192,"statfs":{"total":95995035648,"available":94318788608,"internally_reserved":0,"allocated":1607360512,"data_stored":1606534097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149774,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2820200,"kb_used_data":2716832,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":90924952,"statfs":{"total":95995035648,"available":93107150848,"internally_reserved":0,"allocated":2782035968,"data_stored":2781206001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":105839047},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411410,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2096856,"kb_used_data":1997520,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":91648296,"statfs":{"total":95995035648,"available":93847855104,"internally_reserved":0,"allocated":2045460480,"data_stored":2044638842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,2,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574935,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2044576,"kb_used_data":1959252,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":91700576,"statfs":{"total":95995035648,"available":93901389824,"internally_reserved":0,"allocated":2006274048,"data_stored":2005451705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738459,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2072660,"kb_used_data":2001160,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":91672492,"statfs":{"total":95995035648,"available":93872631808,"internally_reserved":0,"allocated":2049187840,"data_stored":2048369754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2043609088,"data_stored":2043609088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1988427776,"data_stored":1988427776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2038628352,"data_stored":2038628352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2770997248,"data_stored":2770993176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1601703936,"data_stored":1601699864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":808910848,"data_stored":808910848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":416940032,"data_stored":416940032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1992294400,"data_stored":1992294400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1641152512,"data_stored":1641152512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":2797867008,"data_stored":2797862936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":198115328,"data_stored":198115328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1630928896,"data_stored":1630928896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T14:14:51.731 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph status --format=json 2024-08-19T14:14:51.741 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:14:51.820 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:14:51.843 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:51.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:51.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:14:51.944 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:14:51.969 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:51.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:51.970 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:14:52.071 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:14:52.126 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:52.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:52.127 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:14:52.228 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:14:52.258 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:52 smithi031 ceph-mon[16336]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 268/293952 objects degraded (0.091%), 1 pg degraded, 1 pg undersized) 2024-08-19T14:14:52.267 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:52.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:52.271 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:14:52.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:52 smithi045 ceph-mon[20143]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 268/293952 objects degraded (0.091%), 1 pg degraded, 1 pg undersized) 2024-08-19T14:14:52.372 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:14:52.375 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:14:52.414 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:52.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:52.415 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:14:52.449 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:14:52.449 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:14:52.483 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:14:52.484 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:14:52.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:52 smithi032 ceph-mon[20004]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 268/293952 objects degraded (0.091%), 1 pg degraded, 1 pg undersized) 2024-08-19T14:14:52.516 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:14:52.526 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:14:52.527 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:14:52.540 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:52.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:52.541 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:14:52.565 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:14:52.565 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:14:52.595 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:14:52.596 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:14:52.630 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:14:52.630 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:14:52.641 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:14:52.662 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:14:52.662 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:14:52.664 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:52.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:52.665 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:14:52.697 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:14:52.697 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:14:52.729 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:14:52.730 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:14:52.765 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:14:52.767 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:14:52.767 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:14:52.789 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:52.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:52.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:14:52.795 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:14:52.796 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:14:52.825 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:14:52.826 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:14:52.890 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:14:52.918 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:52.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:52.919 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:14:53.021 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:14:53.030 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:14:53.030 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-08-19T14:14:53.030 INFO:teuthology.orchestra.run.smithi031.stdout: "complaint_time": 30, 2024-08-19T14:14:53.030 INFO:teuthology.orchestra.run.smithi031.stdout: "num_blocked_ops": 0 2024-08-19T14:14:53.030 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:14:53.046 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:53.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:53.047 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:14:53.148 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:14:53.176 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:53.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:53.177 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:14:53.278 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:14:53.308 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:53.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:53.309 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:14:53.327 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-19T14:14:53.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:53 smithi031 ceph-mon[16336]: pgmap v414: 27 pgs: 27 active+clean; 5.9 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 922 op/s; 1.6 MiB/s, 121 objects/s recovering 2024-08-19T14:14:53.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:53 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1338190813' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:14:53.409 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:14:53.436 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:53.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:53.437 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:14:53.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:53 smithi032 ceph-mon[20004]: pgmap v414: 27 pgs: 27 active+clean; 5.9 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 922 op/s; 1.6 MiB/s, 121 objects/s recovering 2024-08-19T14:14:53.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:53 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1338190813' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:14:53.538 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:14:53.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:53 smithi045 ceph-mon[20143]: pgmap v414: 27 pgs: 27 active+clean; 5.9 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 922 op/s; 1.6 MiB/s, 121 objects/s recovering 2024-08-19T14:14:53.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:53 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1338190813' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:14:53.559 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:53.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:53.560 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:14:53.661 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:14:53.684 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:53.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:53.685 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:14:53.785 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:14:53.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:53.849 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:53.849 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:14:53.950 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:14:53.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:53.984 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:53.984 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:14:54.085 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:14:54.125 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:54.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:54.126 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:14:54.228 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:14:54.240 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:54 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:14:54.240 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:54 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:14:54.251 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:54.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:54.252 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:14:54.286 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-08-19T14:14:54.317 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:14:54.353 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:14:54.401 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:54.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:54.402 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:14:54.503 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:14:54.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:54 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:14:54.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:54 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:14:54.526 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:54.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:54.527 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:14:54.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:54 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:14:54.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:54 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:14:54.627 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:14:54.643 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:14:54.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:54.670 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:54.670 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:14:54.771 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:14:54.795 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:54.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:54.796 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:14:54.897 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:14:54.941 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:54.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:54.942 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:14:55.043 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:14:55.067 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:55.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:55.069 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:14:55.170 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:14:55.193 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:55.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:55.194 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:14:55.295 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:14:55.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:55.323 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:55.323 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:14:55.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:55 smithi031 ceph-mon[16336]: pgmap v415: 27 pgs: 27 active+clean; 5.8 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 839 op/s; 0 B/s, 39 objects/s recovering 2024-08-19T14:14:55.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:55 smithi031 ceph-mon[16336]: osdmap e166: 12 total, 12 up, 10 in 2024-08-19T14:14:55.407 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:55 smithi032 ceph-mon[20004]: pgmap v415: 27 pgs: 27 active+clean; 5.8 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 839 op/s; 0 B/s, 39 objects/s recovering 2024-08-19T14:14:55.408 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:55 smithi032 ceph-mon[20004]: osdmap e166: 12 total, 12 up, 10 in 2024-08-19T14:14:55.424 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:14:55.446 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:55 smithi045 ceph-mon[20143]: pgmap v415: 27 pgs: 27 active+clean; 5.8 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 839 op/s; 0 B/s, 39 objects/s recovering 2024-08-19T14:14:55.447 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:55 smithi045 ceph-mon[20143]: osdmap e166: 12 total, 12 up, 10 in 2024-08-19T14:14:55.447 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:55.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:55.448 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:14:55.497 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:14:55.497 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"ed7d1c6e-5e33-11ef-bccf-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub flag(s) set","count":7},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":472,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":166,"num_osds":12,"num_up_osds":12,"osd_up_since":1724076699,"num_in_osds":10,"osd_in_since":1724076798,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":27}],"num_pgs":27,"num_pools":2,"num_objects":94706,"data_bytes":6207046200,"bytes_used":19576553472,"bytes_avail":940373803008,"bytes_total":959950356480,"recovering_objects_per_sec":39,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":266,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":839},"fsmap":{"epoch":1,"btime":"2024-08-19T14:05:22:841940+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.31:8443/"}},"servicemap":{"epoch":18,"modified":"2024-08-19T14:13:09.738190+0000","services":{}},"progress_events":{}} 2024-08-19T14:14:55.549 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:14:55.570 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:55.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:55.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:14:55.588 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:14:55.672 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:14:55.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:55.695 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:55.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:14:55.796 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:14:55.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:55.824 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:55.824 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:14:55.925 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:14:55.978 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:55.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:55.979 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:14:56.081 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:14:56.104 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:56.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:56.105 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:14:56.206 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:14:56.227 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:56.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:56.229 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:14:56.274 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:56 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1724688116' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-08-19T14:14:56.327 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-08-19T14:14:56.327 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-19T14:14:56.330 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:14:56.358 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:56.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:56.360 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:14:56.461 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:14:56.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:56.485 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:56.485 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:14:56.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:56 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1724688116' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-08-19T14:14:56.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:56 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1724688116' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-08-19T14:14:56.586 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:14:56.607 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:56.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:56.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:14:56.614 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:14:56.683 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:14:56.709 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:14:56.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:56.737 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:56.737 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:14:56.838 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:14:56.864 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:56.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:56.865 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:14:56.966 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:14:56.991 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:56.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:56.992 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:14:57.085 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is set 2024-08-19T14:14:57.093 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:14:57.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:57.121 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:57.122 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:14:57.223 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:14:57.246 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:57.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:57.247 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:14:57.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:57 smithi045 ceph-mon[20143]: pgmap v417: 27 pgs: 27 active+clean; 5.7 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 802 op/s; 0 B/s, 39 objects/s recovering 2024-08-19T14:14:57.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:57 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3992673811' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:14:57.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:57 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:14:57.347 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:14:57.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:57 smithi031 ceph-mon[16336]: pgmap v417: 27 pgs: 27 active+clean; 5.7 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 802 op/s; 0 B/s, 39 objects/s recovering 2024-08-19T14:14:57.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:57 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3992673811' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:14:57.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:57 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:14:57.370 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:57.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:57.371 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:14:57.472 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:14:57.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:57 smithi032 ceph-mon[20004]: pgmap v417: 27 pgs: 27 active+clean; 5.7 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 802 op/s; 0 B/s, 39 objects/s recovering 2024-08-19T14:14:57.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:57 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3992673811' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:14:57.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:57 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:14:57.521 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:57.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:57.523 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:14:57.531 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:14:57.531 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 20, 2024-08-19T14:14:57.531 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 600, 2024-08-19T14:14:57.531 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [ 2024-08-19T14:14:57.532 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.532 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:67321 2.7 2:e3a8d116:::benchmark_data_smithi045_46415_object67320:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-08-19T14:14:57.532 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:13:24.291635+0000", 2024-08-19T14:14:57.532 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 93.108459752000002, 2024-08-19T14:14:57.532 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.155385669, 2024-08-19T14:14:57.532 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:14:57.532 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:14:57.532 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:14:57.532 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:14:57.532 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:14:57.532 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:14:57.533 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 67321 2024-08-19T14:14:57.533 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.533 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:14:57.533 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.533 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:14:57.533 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:24.291635+0000", 2024-08-19T14:14:57.533 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:14:57.533 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.533 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.533 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:14:57.533 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:24.291635+0000", 2024-08-19T14:14:57.533 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:14:57.533 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.534 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.534 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:14:57.534 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:24.291636+0000", 2024-08-19T14:14:57.534 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1900000000000003e-07 2024-08-19T14:14:57.534 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.534 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.534 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:14:57.534 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:24.291668+0000", 2024-08-19T14:14:57.534 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.2385000000000002e-05 2024-08-19T14:14:57.534 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.534 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.534 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:14:57.534 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:24.291675+0000", 2024-08-19T14:14:57.535 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.8970000000000002e-06 2024-08-19T14:14:57.535 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.535 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.535 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:14:57.535 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:24.291681+0000", 2024-08-19T14:14:57.535 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.3129999999999998e-06 2024-08-19T14:14:57.535 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.535 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.535 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:14:57.535 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:24.299330+0000", 2024-08-19T14:14:57.535 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0076492920000000002 2024-08-19T14:14:57.535 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.536 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.536 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:14:57.536 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:24.299542+0000", 2024-08-19T14:14:57.536 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000212146 2024-08-19T14:14:57.536 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.536 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.536 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,4", 2024-08-19T14:14:57.536 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:24.299727+0000", 2024-08-19T14:14:57.536 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000184563 2024-08-19T14:14:57.536 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.536 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.537 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:14:57.537 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:24.308397+0000", 2024-08-19T14:14:57.537 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0086701539999999994 2024-08-19T14:14:57.537 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.537 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.537 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:14:57.537 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:24.310366+0000", 2024-08-19T14:14:57.537 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0019696179999999998 2024-08-19T14:14:57.537 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.537 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.538 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:14:57.538 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:24.444877+0000", 2024-08-19T14:14:57.538 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.13451043900000001 2024-08-19T14:14:57.538 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.538 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.538 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:14:57.538 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:24.444954+0000", 2024-08-19T14:14:57.538 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.6934000000000006e-05 2024-08-19T14:14:57.538 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.538 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.538 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:14:57.538 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:24.447021+0000", 2024-08-19T14:14:57.538 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0020674090000000001 2024-08-19T14:14:57.539 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:14:57.539 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:14:57.539 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:14:57.539 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.539 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.539 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "MOSDPGPushReply(2.7 130/127 [PushReplyOp(2:eaefdbca:::benchmark_data_smithi045_46415_object11414:head)])", 2024-08-19T14:14:57.539 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:13:24.292794+0000", 2024-08-19T14:14:57.539 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 93.107300875000007, 2024-08-19T14:14:57.539 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.177897579, 2024-08-19T14:14:57.539 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:14:57.539 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:14:57.539 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:14:57.539 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:14:57.539 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.540 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:14:57.540 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:24.292794+0000", 2024-08-19T14:14:57.540 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:14:57.540 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.540 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.540 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:14:57.540 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:24.292794+0000", 2024-08-19T14:14:57.540 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:14:57.540 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.540 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.540 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:14:57.540 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:24.292795+0000", 2024-08-19T14:14:57.540 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.55e-07 2024-08-19T14:14:57.541 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.541 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.541 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:14:57.541 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:24.292798+0000", 2024-08-19T14:14:57.541 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.3909999999999998e-06 2024-08-19T14:14:57.541 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.541 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.541 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:14:57.541 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:24.292804+0000", 2024-08-19T14:14:57.541 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.1190000000000002e-06 2024-08-19T14:14:57.541 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.541 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.542 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:14:57.542 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:24.292809+0000", 2024-08-19T14:14:57.542 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1410000000000004e-06 2024-08-19T14:14:57.542 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.542 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.542 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:14:57.542 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:24.454982+0000", 2024-08-19T14:14:57.542 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.162172709 2024-08-19T14:14:57.542 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.542 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.542 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:14:57.543 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:24.470692+0000", 2024-08-19T14:14:57.543 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.015710063999999999 2024-08-19T14:14:57.543 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:14:57.543 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:14:57.543 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:14:57.543 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.543 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.543 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:67338 2.7 2:e0d15a33:::benchmark_data_smithi045_46415_object67337:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-08-19T14:14:57.543 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:13:24.299525+0000", 2024-08-19T14:14:57.543 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 93.100569835000002, 2024-08-19T14:14:57.543 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.17372130099999999, 2024-08-19T14:14:57.543 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:14:57.543 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:14:57.544 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:14:57.544 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:14:57.544 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:14:57.544 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:14:57.544 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 67338 2024-08-19T14:14:57.544 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.544 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:14:57.544 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.544 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:14:57.544 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:24.299525+0000", 2024-08-19T14:14:57.544 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:14:57.544 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.544 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.545 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:14:57.545 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:24.299525+0000", 2024-08-19T14:14:57.545 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:14:57.545 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.545 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.545 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:14:57.545 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:24.299526+0000", 2024-08-19T14:14:57.545 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.8699999999999995e-07 2024-08-19T14:14:57.545 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.545 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.545 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:14:57.545 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:24.299553+0000", 2024-08-19T14:14:57.545 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.7203999999999999e-05 2024-08-19T14:14:57.546 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.546 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.546 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:14:57.546 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:24.299560+0000", 2024-08-19T14:14:57.546 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.0899999999999999e-06 2024-08-19T14:14:57.546 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.546 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.546 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:14:57.546 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:24.299565+0000", 2024-08-19T14:14:57.546 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.8629999999999999e-06 2024-08-19T14:14:57.546 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.546 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.546 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:14:57.546 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:24.449405+0000", 2024-08-19T14:14:57.547 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.149840321 2024-08-19T14:14:57.547 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.547 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.547 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:14:57.547 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:24.449644+0000", 2024-08-19T14:14:57.547 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000238216 2024-08-19T14:14:57.547 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.547 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.547 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,4", 2024-08-19T14:14:57.547 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:24.449809+0000", 2024-08-19T14:14:57.547 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000165527 2024-08-19T14:14:57.547 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.547 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.548 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:14:57.548 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:24.454939+0000", 2024-08-19T14:14:57.548 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0051303080000000001 2024-08-19T14:14:57.548 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.548 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.548 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:14:57.548 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:24.470689+0000", 2024-08-19T14:14:57.548 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.015749510000000001 2024-08-19T14:14:57.548 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.548 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.548 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:14:57.548 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:24.470803+0000", 2024-08-19T14:14:57.548 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00011425999999999999 2024-08-19T14:14:57.549 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.549 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.549 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:14:57.549 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:24.470961+0000", 2024-08-19T14:14:57.549 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00015803 2024-08-19T14:14:57.549 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.549 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.549 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:14:57.549 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:24.473247+0000", 2024-08-19T14:14:57.549 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0022854849999999999 2024-08-19T14:14:57.549 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:14:57.549 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:14:57.549 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:14:57.550 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.550 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.550 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:67353 2.7 2:e091265a:::benchmark_data_smithi045_46415_object67352:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-08-19T14:14:57.550 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:13:24.313914+0000", 2024-08-19T14:14:57.550 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 93.086181237999995, 2024-08-19T14:14:57.550 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.17686955900000001, 2024-08-19T14:14:57.550 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:14:57.550 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:14:57.550 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:14:57.550 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:14:57.550 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:14:57.550 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:14:57.550 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 67353 2024-08-19T14:14:57.550 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.551 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:14:57.551 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.551 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:14:57.551 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:24.313914+0000", 2024-08-19T14:14:57.551 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:14:57.551 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.551 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.551 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:14:57.551 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:24.313914+0000", 2024-08-19T14:14:57.551 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:14:57.551 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.551 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.551 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:14:57.552 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:24.313915+0000", 2024-08-19T14:14:57.552 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.364e-06 2024-08-19T14:14:57.552 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.552 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.552 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:14:57.552 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:24.313953+0000", 2024-08-19T14:14:57.552 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7546999999999998e-05 2024-08-19T14:14:57.552 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.552 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.552 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:14:57.552 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:24.313967+0000", 2024-08-19T14:14:57.552 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4263999999999999e-05 2024-08-19T14:14:57.552 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.553 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.553 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:14:57.553 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:24.313976+0000", 2024-08-19T14:14:57.553 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.3930000000000008e-06 2024-08-19T14:14:57.553 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.553 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.553 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:14:57.553 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:24.452041+0000", 2024-08-19T14:14:57.553 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.138065309 2024-08-19T14:14:57.553 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.553 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.553 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:14:57.553 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:24.452218+0000", 2024-08-19T14:14:57.554 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00017702500000000001 2024-08-19T14:14:57.554 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.554 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.554 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,4", 2024-08-19T14:14:57.554 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:24.452405+0000", 2024-08-19T14:14:57.554 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000186768 2024-08-19T14:14:57.554 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.554 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.554 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:14:57.554 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:24.470751+0000", 2024-08-19T14:14:57.554 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.018346446999999998 2024-08-19T14:14:57.555 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.555 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.555 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:14:57.555 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:24.473300+0000", 2024-08-19T14:14:57.555 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0025490249999999999 2024-08-19T14:14:57.555 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.555 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.555 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:14:57.555 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:24.488612+0000", 2024-08-19T14:14:57.555 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.015311666 2024-08-19T14:14:57.555 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.555 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.555 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:14:57.555 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:24.488669+0000", 2024-08-19T14:14:57.556 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.6987000000000002e-05 2024-08-19T14:14:57.556 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.556 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.556 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:14:57.556 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:24.490784+0000", 2024-08-19T14:14:57.556 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0021147639999999999 2024-08-19T14:14:57.556 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:14:57.556 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:14:57.556 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:14:57.556 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.556 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.556 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:81903 2.1 2:8a4e7877:::benchmark_data_smithi045_46415_object81902:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-08-19T14:14:57.557 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:13:37.340002+0000", 2024-08-19T14:14:57.557 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 80.060093319000003, 2024-08-19T14:14:57.557 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.20794894, 2024-08-19T14:14:57.557 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:14:57.557 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:14:57.557 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:14:57.557 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:14:57.557 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:14:57.557 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:14:57.557 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 81903 2024-08-19T14:14:57.557 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.557 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:14:57.557 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.558 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:14:57.558 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:37.340002+0000", 2024-08-19T14:14:57.558 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:14:57.558 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.558 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.558 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:14:57.558 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:37.340002+0000", 2024-08-19T14:14:57.558 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:14:57.558 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.558 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.558 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:14:57.558 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:37.340003+0000", 2024-08-19T14:14:57.558 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4610000000000001e-06 2024-08-19T14:14:57.559 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.559 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.559 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:14:57.559 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:37.340051+0000", 2024-08-19T14:14:57.559 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7822999999999999e-05 2024-08-19T14:14:57.559 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.559 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.559 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:14:57.559 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:37.340065+0000", 2024-08-19T14:14:57.559 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3429e-05 2024-08-19T14:14:57.559 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.559 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.559 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:14:57.559 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:37.340072+0000", 2024-08-19T14:14:57.560 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.1160000000000001e-06 2024-08-19T14:14:57.560 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.560 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.560 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:14:57.560 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:37.340126+0000", 2024-08-19T14:14:57.560 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.3789000000000003e-05 2024-08-19T14:14:57.560 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.560 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.560 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:14:57.560 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:37.340243+0000", 2024-08-19T14:14:57.560 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000117811 2024-08-19T14:14:57.560 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.560 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.561 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,3", 2024-08-19T14:14:57.561 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:37.340379+0000", 2024-08-19T14:14:57.561 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013583599999999999 2024-08-19T14:14:57.561 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.561 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.561 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:14:57.561 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:37.348452+0000", 2024-08-19T14:14:57.561 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0080726919999999994 2024-08-19T14:14:57.561 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.561 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.561 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:14:57.562 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:37.358813+0000", 2024-08-19T14:14:57.562 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.010360833999999999 2024-08-19T14:14:57.562 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.562 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.562 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:14:57.562 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:37.547831+0000", 2024-08-19T14:14:57.562 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.18901807800000001 2024-08-19T14:14:57.562 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.562 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.562 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:14:57.562 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:37.547905+0000", 2024-08-19T14:14:57.562 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.4488999999999998e-05 2024-08-19T14:14:57.562 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.563 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.563 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:14:57.563 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:37.547951+0000", 2024-08-19T14:14:57.563 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.5581999999999998e-05 2024-08-19T14:14:57.563 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:14:57.563 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:14:57.563 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:14:57.563 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.563 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.563 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:81907 2.1 2:81021fc5:::benchmark_data_smithi045_46415_object81906:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-08-19T14:14:57.563 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:13:37.343223+0000", 2024-08-19T14:14:57.563 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 80.056872011999999, 2024-08-19T14:14:57.564 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.21485626499999999, 2024-08-19T14:14:57.564 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:14:57.564 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:14:57.564 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:14:57.564 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:14:57.564 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:14:57.564 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:14:57.564 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 81907 2024-08-19T14:14:57.564 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.564 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:14:57.564 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.564 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:14:57.564 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:37.343223+0000", 2024-08-19T14:14:57.565 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:14:57.565 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.565 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.565 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:14:57.565 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:37.343223+0000", 2024-08-19T14:14:57.565 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:14:57.565 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.565 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.565 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:14:57.565 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:37.343225+0000", 2024-08-19T14:14:57.565 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3659999999999999e-06 2024-08-19T14:14:57.566 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.566 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.566 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:14:57.566 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:37.343276+0000", 2024-08-19T14:14:57.566 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1115000000000001e-05 2024-08-19T14:14:57.566 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.566 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.566 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:14:57.566 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:37.343291+0000", 2024-08-19T14:14:57.566 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5455e-05 2024-08-19T14:14:57.566 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.567 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.567 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:14:57.567 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:37.343299+0000", 2024-08-19T14:14:57.567 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.1189999999999997e-06 2024-08-19T14:14:57.567 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.567 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.567 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:14:57.567 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:37.343350+0000", 2024-08-19T14:14:57.567 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0943999999999999e-05 2024-08-19T14:14:57.567 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.567 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.567 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:14:57.567 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:37.343480+0000", 2024-08-19T14:14:57.567 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012985300000000001 2024-08-19T14:14:57.568 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.568 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.568 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,3", 2024-08-19T14:14:57.568 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:37.343708+0000", 2024-08-19T14:14:57.568 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00022757800000000001 2024-08-19T14:14:57.568 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.568 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.568 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:14:57.568 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:37.358799+0000", 2024-08-19T14:14:57.568 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.015091113 2024-08-19T14:14:57.568 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.568 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.568 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:14:57.569 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:37.358877+0000", 2024-08-19T14:14:57.569 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.8424000000000007e-05 2024-08-19T14:14:57.569 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.569 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.569 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:14:57.569 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:37.557957+0000", 2024-08-19T14:14:57.569 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.19907945399999999 2024-08-19T14:14:57.569 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.569 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.569 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:14:57.569 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:37.558030+0000", 2024-08-19T14:14:57.569 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.3678999999999997e-05 2024-08-19T14:14:57.569 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.569 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.570 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:14:57.570 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:37.558079+0000", 2024-08-19T14:14:57.570 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.9165000000000001e-05 2024-08-19T14:14:57.570 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:14:57.570 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:14:57.570 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:14:57.570 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.570 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.570 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:81909 2.1 2:85af555d:::benchmark_data_smithi045_46415_object81908:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-08-19T14:14:57.570 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:13:37.345361+0000", 2024-08-19T14:14:57.570 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 80.054733889000005, 2024-08-19T14:14:57.571 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.216364263, 2024-08-19T14:14:57.571 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:14:57.571 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:14:57.571 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:14:57.571 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:14:57.571 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:14:57.571 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:14:57.571 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 81909 2024-08-19T14:14:57.571 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.571 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:14:57.571 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.571 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:14:57.571 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:37.345361+0000", 2024-08-19T14:14:57.572 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:14:57.572 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.572 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.572 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:14:57.572 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:37.345361+0000", 2024-08-19T14:14:57.572 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:14:57.572 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.572 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.572 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:14:57.572 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:37.345363+0000", 2024-08-19T14:14:57.572 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3060000000000001e-06 2024-08-19T14:14:57.572 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.572 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.572 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:14:57.573 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:37.345394+0000", 2024-08-19T14:14:57.573 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.1256e-05 2024-08-19T14:14:57.573 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.573 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.573 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:14:57.573 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:37.345412+0000", 2024-08-19T14:14:57.573 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.7969999999999999e-05 2024-08-19T14:14:57.573 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.573 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.573 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:14:57.573 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:37.345419+0000", 2024-08-19T14:14:57.573 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.063e-06 2024-08-19T14:14:57.573 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.574 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.574 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:14:57.574 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:37.348534+0000", 2024-08-19T14:14:57.574 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0031148590000000002 2024-08-19T14:14:57.574 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.574 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.574 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:14:57.574 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:37.348646+0000", 2024-08-19T14:14:57.574 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000112543 2024-08-19T14:14:57.574 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.574 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.574 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,3", 2024-08-19T14:14:57.575 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:37.348793+0000", 2024-08-19T14:14:57.575 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00014693700000000001 2024-08-19T14:14:57.575 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.575 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.575 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:14:57.575 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:37.358850+0000", 2024-08-19T14:14:57.575 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.010056538 2024-08-19T14:14:57.575 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.575 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.575 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:14:57.575 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:37.358882+0000", 2024-08-19T14:14:57.575 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.2236000000000001e-05 2024-08-19T14:14:57.576 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.576 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.576 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:14:57.576 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:37.561514+0000", 2024-08-19T14:14:57.576 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.202631751 2024-08-19T14:14:57.576 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.576 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.576 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:14:57.576 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:37.561667+0000", 2024-08-19T14:14:57.576 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00015270600000000001 2024-08-19T14:14:57.576 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.576 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.577 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:14:57.577 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:37.561726+0000", 2024-08-19T14:14:57.577 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.9098000000000002e-05 2024-08-19T14:14:57.577 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:14:57.577 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:14:57.577 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:14:57.577 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.577 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.577 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:81914 2.1 2:87882a26:::benchmark_data_smithi045_46415_object81913:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-08-19T14:14:57.577 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:13:37.345901+0000", 2024-08-19T14:14:57.577 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 80.054194136000007, 2024-08-19T14:14:57.577 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.21599740000000001, 2024-08-19T14:14:57.578 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:14:57.578 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:14:57.578 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:14:57.578 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:14:57.578 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:14:57.578 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:14:57.578 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 81914 2024-08-19T14:14:57.578 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.578 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:14:57.578 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.578 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:14:57.579 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:37.345901+0000", 2024-08-19T14:14:57.579 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:14:57.579 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.579 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.579 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:14:57.579 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:37.345901+0000", 2024-08-19T14:14:57.579 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:14:57.579 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.579 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.579 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:14:57.579 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:37.345902+0000", 2024-08-19T14:14:57.579 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.8999999999999995e-07 2024-08-19T14:14:57.579 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.579 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.580 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:14:57.580 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:37.345934+0000", 2024-08-19T14:14:57.580 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.2063000000000001e-05 2024-08-19T14:14:57.580 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.580 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.580 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:14:57.580 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:37.345945+0000", 2024-08-19T14:14:57.580 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0587e-05 2024-08-19T14:14:57.580 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.580 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.580 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:14:57.580 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:37.345951+0000", 2024-08-19T14:14:57.580 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.4309999999999999e-06 2024-08-19T14:14:57.581 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.581 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.581 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:14:57.581 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:37.355714+0000", 2024-08-19T14:14:57.581 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0097627219999999997 2024-08-19T14:14:57.581 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.581 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.581 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:14:57.581 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:37.355857+0000", 2024-08-19T14:14:57.581 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00014296 2024-08-19T14:14:57.581 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.581 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.581 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,3", 2024-08-19T14:14:57.582 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:37.356019+0000", 2024-08-19T14:14:57.582 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00016237900000000001 2024-08-19T14:14:57.582 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.582 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.582 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:14:57.582 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:37.362240+0000", 2024-08-19T14:14:57.582 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0062210540000000002 2024-08-19T14:14:57.582 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.582 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.582 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:14:57.582 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:37.364242+0000", 2024-08-19T14:14:57.582 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0020021219999999998 2024-08-19T14:14:57.582 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.583 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.583 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:14:57.583 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:37.561766+0000", 2024-08-19T14:14:57.583 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.19752372300000001 2024-08-19T14:14:57.583 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.583 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.583 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:14:57.583 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:37.561852+0000", 2024-08-19T14:14:57.583 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.6087000000000004e-05 2024-08-19T14:14:57.583 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.583 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.583 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:14:57.584 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:37.561898+0000", 2024-08-19T14:14:57.584 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.6482e-05 2024-08-19T14:14:57.584 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:14:57.584 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:14:57.584 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:14:57.584 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.584 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.584 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:81917 2.1 2:8e23d9c5:::benchmark_data_smithi045_46415_object81916:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-08-19T14:14:57.584 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:13:37.352560+0000", 2024-08-19T14:14:57.584 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 80.047534967000004, 2024-08-19T14:14:57.584 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.20947323900000001, 2024-08-19T14:14:57.584 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:14:57.584 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:14:57.585 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:14:57.585 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:14:57.585 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:14:57.585 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:14:57.585 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 81917 2024-08-19T14:14:57.585 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.585 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:14:57.585 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.585 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:14:57.585 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:37.352560+0000", 2024-08-19T14:14:57.585 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:14:57.585 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.586 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.586 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:14:57.586 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:37.352560+0000", 2024-08-19T14:14:57.586 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:14:57.586 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.586 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.586 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:14:57.586 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:37.352561+0000", 2024-08-19T14:14:57.586 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.051e-06 2024-08-19T14:14:57.586 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.586 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.586 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:14:57.586 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:37.352599+0000", 2024-08-19T14:14:57.587 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7891e-05 2024-08-19T14:14:57.587 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.587 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.587 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:14:57.587 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:37.352611+0000", 2024-08-19T14:14:57.587 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2072000000000001e-05 2024-08-19T14:14:57.587 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.587 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.587 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:14:57.587 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:37.352619+0000", 2024-08-19T14:14:57.587 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.7400000000000004e-06 2024-08-19T14:14:57.587 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.587 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.587 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:14:57.588 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:37.358907+0000", 2024-08-19T14:14:57.588 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0062884530000000003 2024-08-19T14:14:57.588 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.588 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.588 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:14:57.588 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:37.359050+0000", 2024-08-19T14:14:57.588 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000142862 2024-08-19T14:14:57.588 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.588 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.588 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,3", 2024-08-19T14:14:57.588 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:37.359202+0000", 2024-08-19T14:14:57.588 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000151748 2024-08-19T14:14:57.588 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.589 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.589 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:14:57.589 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:37.364283+0000", 2024-08-19T14:14:57.589 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0050808509999999999 2024-08-19T14:14:57.589 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.589 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.589 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:14:57.589 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:37.364296+0000", 2024-08-19T14:14:57.589 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3521e-05 2024-08-19T14:14:57.589 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.589 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.589 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:14:57.590 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:37.561939+0000", 2024-08-19T14:14:57.590 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.197642919 2024-08-19T14:14:57.590 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.590 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.590 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:14:57.590 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:37.561991+0000", 2024-08-19T14:14:57.590 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1922000000000002e-05 2024-08-19T14:14:57.590 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.590 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.590 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:14:57.590 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:37.562033+0000", 2024-08-19T14:14:57.590 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2209000000000002e-05 2024-08-19T14:14:57.590 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:14:57.590 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:14:57.591 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:14:57.591 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.591 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.591 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:81976 2.1 2:84f3317c:::benchmark_data_smithi045_46415_object81975:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-08-19T14:14:57.591 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:13:37.401313+0000", 2024-08-19T14:14:57.591 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 79.998781773999994, 2024-08-19T14:14:57.591 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.16181509099999999, 2024-08-19T14:14:57.591 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:14:57.591 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:14:57.591 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:14:57.591 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:14:57.591 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:14:57.591 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:14:57.592 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 81976 2024-08-19T14:14:57.592 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.592 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:14:57.592 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.592 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:14:57.592 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:37.401313+0000", 2024-08-19T14:14:57.592 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:14:57.592 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.592 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.592 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:14:57.592 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:37.401313+0000", 2024-08-19T14:14:57.592 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:14:57.592 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.593 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.593 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:14:57.593 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:37.401317+0000", 2024-08-19T14:14:57.593 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.0219999999999998e-06 2024-08-19T14:14:57.593 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.593 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.593 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:14:57.593 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:37.401372+0000", 2024-08-19T14:14:57.593 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.4941999999999997e-05 2024-08-19T14:14:57.593 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.593 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.593 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:14:57.593 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:37.401389+0000", 2024-08-19T14:14:57.593 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6464999999999999e-05 2024-08-19T14:14:57.594 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.594 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.594 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:14:57.594 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:37.401397+0000", 2024-08-19T14:14:57.594 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.5590000000000006e-06 2024-08-19T14:14:57.594 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.594 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.594 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:14:57.594 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:37.401467+0000", 2024-08-19T14:14:57.594 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.9212000000000001e-05 2024-08-19T14:14:57.594 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.595 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.595 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:14:57.595 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:37.401626+0000", 2024-08-19T14:14:57.595 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00015887200000000001 2024-08-19T14:14:57.595 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.595 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.595 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,3", 2024-08-19T14:14:57.595 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:37.401788+0000", 2024-08-19T14:14:57.595 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000162182 2024-08-19T14:14:57.595 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.595 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.595 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:14:57.595 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:37.405343+0000", 2024-08-19T14:14:57.595 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0035553960000000002 2024-08-19T14:14:57.596 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.596 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.596 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:14:57.596 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:37.414083+0000", 2024-08-19T14:14:57.596 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0087400559999999995 2024-08-19T14:14:57.596 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.596 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.596 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:14:57.596 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:37.563030+0000", 2024-08-19T14:14:57.596 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.14894707099999999 2024-08-19T14:14:57.596 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.596 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.597 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:14:57.597 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:37.563087+0000", 2024-08-19T14:14:57.597 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.7076e-05 2024-08-19T14:14:57.597 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.597 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.597 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:14:57.597 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:37.563129+0000", 2024-08-19T14:14:57.597 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.1238000000000001e-05 2024-08-19T14:14:57.597 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:14:57.597 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:14:57.597 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:14:57.597 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.597 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.598 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:124013 2.1 2:8c401e52:::benchmark_data_smithi045_46415_object732:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-08-19T14:14:57.598 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:14:21.771687+0000", 2024-08-19T14:14:57.598 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 35.628407746000001, 2024-08-19T14:14:57.598 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.20968569100000001, 2024-08-19T14:14:57.598 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:14:57.598 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:14:57.598 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:14:57.598 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:14:57.599 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:14:57.599 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:14:57.599 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 124013 2024-08-19T14:14:57.599 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.599 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:14:57.599 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.599 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:14:57.599 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.771687+0000", 2024-08-19T14:14:57.599 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:14:57.599 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.599 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.599 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:14:57.600 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.771687+0000", 2024-08-19T14:14:57.600 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:14:57.600 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.600 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.601 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:14:57.602 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.771689+0000", 2024-08-19T14:14:57.602 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.654e-06 2024-08-19T14:14:57.602 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.602 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.602 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:14:57.602 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.771696+0000", 2024-08-19T14:14:57.602 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.5570000000000001e-06 2024-08-19T14:14:57.602 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.602 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.602 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:14:57.602 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.771718+0000", 2024-08-19T14:14:57.603 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.1852999999999999e-05 2024-08-19T14:14:57.603 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.603 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.603 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:14:57.603 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.771726+0000", 2024-08-19T14:14:57.603 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.6170000000000007e-06 2024-08-19T14:14:57.603 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.603 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.603 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:14:57.603 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.771793+0000", 2024-08-19T14:14:57.603 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.7082000000000006e-05 2024-08-19T14:14:57.603 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.603 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.604 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:14:57.604 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.771926+0000", 2024-08-19T14:14:57.604 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000133146 2024-08-19T14:14:57.604 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.604 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.604 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,3", 2024-08-19T14:14:57.604 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.772049+0000", 2024-08-19T14:14:57.604 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012255799999999999 2024-08-19T14:14:57.604 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.604 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.604 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:14:57.604 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.786120+0000", 2024-08-19T14:14:57.604 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.014070556999999999 2024-08-19T14:14:57.605 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.605 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.605 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:14:57.605 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.786309+0000", 2024-08-19T14:14:57.605 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00018948000000000001 2024-08-19T14:14:57.605 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.605 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.605 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:14:57.605 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.981217+0000", 2024-08-19T14:14:57.606 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.194907942 2024-08-19T14:14:57.606 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.606 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.606 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:14:57.606 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.981268+0000", 2024-08-19T14:14:57.606 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1035000000000002e-05 2024-08-19T14:14:57.606 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.606 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.606 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:14:57.607 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.981373+0000", 2024-08-19T14:14:57.607 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00010521 2024-08-19T14:14:57.607 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:14:57.607 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:14:57.607 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:14:57.607 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.607 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.607 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:124015 2.1 2:8654db16:::benchmark_data_smithi045_46415_object734:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-08-19T14:14:57.607 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:14:21.772721+0000", 2024-08-19T14:14:57.607 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 35.627373947999999, 2024-08-19T14:14:57.607 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.23538010400000001, 2024-08-19T14:14:57.607 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:14:57.607 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:14:57.608 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:14:57.608 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:14:57.608 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:14:57.608 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:14:57.608 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 124015 2024-08-19T14:14:57.608 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.608 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:14:57.608 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.608 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:14:57.608 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.772721+0000", 2024-08-19T14:14:57.608 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:14:57.608 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.608 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.609 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:14:57.609 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.772721+0000", 2024-08-19T14:14:57.609 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:14:57.609 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.609 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.609 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:14:57.609 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.772722+0000", 2024-08-19T14:14:57.609 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0070000000000001e-06 2024-08-19T14:14:57.609 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.609 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.609 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:14:57.609 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.772727+0000", 2024-08-19T14:14:57.609 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.8400000000000002e-06 2024-08-19T14:14:57.610 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.610 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.610 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:14:57.610 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.772739+0000", 2024-08-19T14:14:57.610 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1449e-05 2024-08-19T14:14:57.610 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.610 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.610 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:14:57.610 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.772746+0000", 2024-08-19T14:14:57.610 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.0500000000000003e-06 2024-08-19T14:14:57.610 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.610 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.611 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:14:57.611 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.775789+0000", 2024-08-19T14:14:57.611 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0030438639999999999 2024-08-19T14:14:57.611 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.611 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.611 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:14:57.611 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.775912+0000", 2024-08-19T14:14:57.611 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000122662 2024-08-19T14:14:57.611 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.611 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.611 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,3", 2024-08-19T14:14:57.611 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.776020+0000", 2024-08-19T14:14:57.611 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000107658 2024-08-19T14:14:57.612 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.612 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.612 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:14:57.612 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.786260+0000", 2024-08-19T14:14:57.612 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.010240285999999999 2024-08-19T14:14:57.612 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.612 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.612 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:14:57.612 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.786326+0000", 2024-08-19T14:14:57.612 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.5661999999999996e-05 2024-08-19T14:14:57.612 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.612 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.613 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:14:57.613 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:22.007996+0000", 2024-08-19T14:14:57.613 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.22166977600000001 2024-08-19T14:14:57.613 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.613 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.613 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:14:57.613 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:22.008043+0000", 2024-08-19T14:14:57.613 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7589000000000001e-05 2024-08-19T14:14:57.613 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.613 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.613 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:14:57.613 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:22.008101+0000", 2024-08-19T14:14:57.614 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.8260999999999997e-05 2024-08-19T14:14:57.614 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:14:57.614 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:14:57.614 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:14:57.614 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.614 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.614 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:124016 2.1 2:85a08517:::benchmark_data_smithi045_46415_object735:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-08-19T14:14:57.614 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:14:21.775314+0000", 2024-08-19T14:14:57.614 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 35.624781040000002, 2024-08-19T14:14:57.614 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.237750032, 2024-08-19T14:14:57.614 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:14:57.614 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:14:57.615 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:14:57.615 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:14:57.615 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:14:57.615 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:14:57.615 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 124016 2024-08-19T14:14:57.615 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.615 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:14:57.615 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.615 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:14:57.615 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.775314+0000", 2024-08-19T14:14:57.615 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:14:57.615 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.615 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.615 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:14:57.616 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.775314+0000", 2024-08-19T14:14:57.616 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:14:57.616 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.616 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.616 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:14:57.616 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.775316+0000", 2024-08-19T14:14:57.616 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.443e-06 2024-08-19T14:14:57.616 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.616 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.616 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:14:57.616 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.775320+0000", 2024-08-19T14:14:57.616 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4279999999999998e-06 2024-08-19T14:14:57.616 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.617 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.617 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:14:57.617 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.775341+0000", 2024-08-19T14:14:57.617 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.0627000000000001e-05 2024-08-19T14:14:57.617 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.617 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.617 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:14:57.617 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.775350+0000", 2024-08-19T14:14:57.617 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.2420000000000006e-06 2024-08-19T14:14:57.618 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.618 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.618 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:14:57.618 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.781652+0000", 2024-08-19T14:14:57.618 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0063019249999999999 2024-08-19T14:14:57.618 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.618 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.618 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:14:57.618 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.781788+0000", 2024-08-19T14:14:57.618 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013662900000000001 2024-08-19T14:14:57.618 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.619 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.619 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,3", 2024-08-19T14:14:57.619 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.782022+0000", 2024-08-19T14:14:57.619 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00023304399999999999 2024-08-19T14:14:57.619 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.620 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.620 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:14:57.620 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.786345+0000", 2024-08-19T14:14:57.620 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0043236519999999999 2024-08-19T14:14:57.620 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.620 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.620 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:14:57.620 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.803700+0000", 2024-08-19T14:14:57.620 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.017354973999999999 2024-08-19T14:14:57.620 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.621 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.621 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:14:57.621 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:22.012966+0000", 2024-08-19T14:14:57.621 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.20926541700000001 2024-08-19T14:14:57.621 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.621 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.621 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:14:57.621 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:22.013014+0000", 2024-08-19T14:14:57.621 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.8507000000000002e-05 2024-08-19T14:14:57.621 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.621 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.621 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:14:57.622 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:22.013064+0000", 2024-08-19T14:14:57.622 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0144e-05 2024-08-19T14:14:57.622 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:14:57.622 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:14:57.622 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:14:57.622 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.622 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.622 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:124025 2.1 2:85dd9c28:::benchmark_data_smithi045_46415_object744:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-08-19T14:14:57.622 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:14:21.791720+0000", 2024-08-19T14:14:57.622 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 35.608375183, 2024-08-19T14:14:57.622 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.22146496299999999, 2024-08-19T14:14:57.622 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:14:57.622 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:14:57.622 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:14:57.623 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:14:57.623 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:14:57.623 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:14:57.623 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 124025 2024-08-19T14:14:57.623 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.623 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:14:57.623 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.623 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:14:57.623 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.791720+0000", 2024-08-19T14:14:57.623 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:14:57.623 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.623 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.623 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:14:57.623 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.791720+0000", 2024-08-19T14:14:57.624 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:14:57.624 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.624 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.624 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:14:57.624 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.791722+0000", 2024-08-19T14:14:57.624 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.9719999999999999e-06 2024-08-19T14:14:57.624 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.624 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.624 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:14:57.624 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.791730+0000", 2024-08-19T14:14:57.624 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.9889999999999992e-06 2024-08-19T14:14:57.624 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.624 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.624 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:14:57.625 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.791753+0000", 2024-08-19T14:14:57.625 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.3343999999999999e-05 2024-08-19T14:14:57.625 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.625 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.625 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:14:57.625 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.791764+0000", 2024-08-19T14:14:57.625 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0482999999999999e-05 2024-08-19T14:14:57.625 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.625 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.625 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:14:57.625 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.793615+0000", 2024-08-19T14:14:57.625 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.001851304 2024-08-19T14:14:57.625 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.626 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.626 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:14:57.626 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.793957+0000", 2024-08-19T14:14:57.626 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00034150100000000002 2024-08-19T14:14:57.626 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.626 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.626 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,3", 2024-08-19T14:14:57.626 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.794373+0000", 2024-08-19T14:14:57.626 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000415957 2024-08-19T14:14:57.626 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.626 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.626 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:14:57.626 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.799218+0000", 2024-08-19T14:14:57.627 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0048456209999999996 2024-08-19T14:14:57.627 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.627 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.627 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:14:57.627 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.803739+0000", 2024-08-19T14:14:57.627 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0045205779999999999 2024-08-19T14:14:57.627 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.627 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.627 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:14:57.627 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:22.013107+0000", 2024-08-19T14:14:57.627 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.20936785599999999 2024-08-19T14:14:57.627 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.627 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.627 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:14:57.628 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:22.013147+0000", 2024-08-19T14:14:57.628 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.9913000000000002e-05 2024-08-19T14:14:57.628 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.628 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.628 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:14:57.628 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:22.013185+0000", 2024-08-19T14:14:57.628 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.8445000000000002e-05 2024-08-19T14:14:57.628 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:14:57.628 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:14:57.628 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:14:57.628 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.629 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.629 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:124038 2.1 2:8c2ecb16:::benchmark_data_smithi045_46415_object757:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-08-19T14:14:57.629 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:14:21.809768+0000", 2024-08-19T14:14:57.629 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 35.590326744999999, 2024-08-19T14:14:57.629 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.203525397, 2024-08-19T14:14:57.629 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:14:57.629 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:14:57.629 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:14:57.629 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:14:57.629 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:14:57.630 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:14:57.630 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 124038 2024-08-19T14:14:57.630 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.630 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:14:57.630 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.630 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:14:57.630 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.809768+0000", 2024-08-19T14:14:57.630 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:14:57.630 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.630 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.630 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:14:57.630 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.809768+0000", 2024-08-19T14:14:57.631 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:14:57.631 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.631 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.631 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:14:57.631 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.809770+0000", 2024-08-19T14:14:57.631 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.404e-06 2024-08-19T14:14:57.631 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.631 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.631 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:14:57.631 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.809776+0000", 2024-08-19T14:14:57.631 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.8080000000000001e-06 2024-08-19T14:14:57.631 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.631 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.631 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:14:57.632 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.809786+0000", 2024-08-19T14:14:57.632 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0376e-05 2024-08-19T14:14:57.632 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.632 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.632 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:14:57.632 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.809795+0000", 2024-08-19T14:14:57.632 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.5120000000000008e-06 2024-08-19T14:14:57.632 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.632 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.632 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:14:57.632 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.809855+0000", 2024-08-19T14:14:57.632 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.9951999999999998e-05 2024-08-19T14:14:57.632 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.633 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.633 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:14:57.633 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.809985+0000", 2024-08-19T14:14:57.633 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013072600000000001 2024-08-19T14:14:57.633 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.633 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.633 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,3", 2024-08-19T14:14:57.633 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.810180+0000", 2024-08-19T14:14:57.633 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00019461599999999999 2024-08-19T14:14:57.633 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.633 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.633 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:14:57.633 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.814441+0000", 2024-08-19T14:14:57.634 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.004261595 2024-08-19T14:14:57.634 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.634 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.634 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:14:57.634 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.818922+0000", 2024-08-19T14:14:57.634 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0044804190000000002 2024-08-19T14:14:57.634 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.634 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.634 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:14:57.634 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:22.013216+0000", 2024-08-19T14:14:57.634 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.194294101 2024-08-19T14:14:57.634 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.634 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.635 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:14:57.635 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:22.013256+0000", 2024-08-19T14:14:57.635 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.0117000000000002e-05 2024-08-19T14:14:57.635 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.635 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.635 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:14:57.635 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:22.013294+0000", 2024-08-19T14:14:57.635 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7771000000000001e-05 2024-08-19T14:14:57.635 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:14:57.635 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:14:57.635 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:14:57.635 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.635 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.636 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:124016 2.1 2:85a08517:::benchmark_data_smithi045_46415_object735:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e151)", 2024-08-19T14:14:57.636 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:14:21.860751+0000", 2024-08-19T14:14:57.636 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 35.539344333000003, 2024-08-19T14:14:57.636 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.15227008, 2024-08-19T14:14:57.636 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:14:57.636 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:14:57.636 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "waiting for ondisk", 2024-08-19T14:14:57.636 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:14:57.636 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:14:57.636 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:14:57.636 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 124016 2024-08-19T14:14:57.636 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.637 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:14:57.637 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.637 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:14:57.637 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.860751+0000", 2024-08-19T14:14:57.637 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:14:57.637 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.637 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.637 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:14:57.637 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.860751+0000", 2024-08-19T14:14:57.637 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:14:57.637 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.637 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.637 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:14:57.638 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.860752+0000", 2024-08-19T14:14:57.638 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.0900000000000001e-07 2024-08-19T14:14:57.638 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.638 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.638 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:14:57.638 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.860755+0000", 2024-08-19T14:14:57.638 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.5319999999999998e-06 2024-08-19T14:14:57.638 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.638 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.638 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:14:57.638 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.860764+0000", 2024-08-19T14:14:57.638 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.7239999999999998e-06 2024-08-19T14:14:57.638 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.638 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.639 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:14:57.639 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.860770+0000", 2024-08-19T14:14:57.639 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.5339999999999996e-06 2024-08-19T14:14:57.639 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.639 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.639 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:14:57.639 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.860890+0000", 2024-08-19T14:14:57.639 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000119579 2024-08-19T14:14:57.639 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.639 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.639 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for ondisk", 2024-08-19T14:14:57.639 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.860942+0000", 2024-08-19T14:14:57.639 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.2330999999999998e-05 2024-08-19T14:14:57.639 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.640 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.640 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:14:57.640 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:22.013021+0000", 2024-08-19T14:14:57.640 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.152078771 2024-08-19T14:14:57.640 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:14:57.640 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:14:57.640 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:14:57.640 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.640 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.640 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:124025 2.1 2:85dd9c28:::benchmark_data_smithi045_46415_object744:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e151)", 2024-08-19T14:14:57.640 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:14:21.860857+0000", 2024-08-19T14:14:57.640 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 35.539238701999999, 2024-08-19T14:14:57.640 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.152295653, 2024-08-19T14:14:57.641 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:14:57.641 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:14:57.641 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "waiting for ondisk", 2024-08-19T14:14:57.641 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:14:57.641 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:14:57.641 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:14:57.641 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 124025 2024-08-19T14:14:57.641 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.641 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:14:57.641 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.641 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:14:57.641 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.860857+0000", 2024-08-19T14:14:57.641 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:14:57.642 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.642 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.642 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:14:57.642 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.860857+0000", 2024-08-19T14:14:57.642 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:14:57.642 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.642 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.642 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:14:57.642 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.860857+0000", 2024-08-19T14:14:57.642 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.3399999999999999e-07 2024-08-19T14:14:57.642 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.642 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.642 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:14:57.642 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.860870+0000", 2024-08-19T14:14:57.643 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2795e-05 2024-08-19T14:14:57.643 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.643 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.643 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:14:57.643 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.860878+0000", 2024-08-19T14:14:57.643 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.5550000000000003e-06 2024-08-19T14:14:57.643 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.643 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.643 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:14:57.643 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.860884+0000", 2024-08-19T14:14:57.643 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.0789999999999997e-06 2024-08-19T14:14:57.643 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.643 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.644 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:14:57.644 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.860967+0000", 2024-08-19T14:14:57.644 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.2134999999999999e-05 2024-08-19T14:14:57.644 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.644 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.644 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for ondisk", 2024-08-19T14:14:57.644 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.861030+0000", 2024-08-19T14:14:57.644 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.3647999999999998e-05 2024-08-19T14:14:57.644 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.644 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.644 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:14:57.644 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:22.013152+0000", 2024-08-19T14:14:57.645 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.152121907 2024-08-19T14:14:57.645 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:14:57.645 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:14:57.645 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:14:57.645 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.645 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.645 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:124038 2.1 2:8c2ecb16:::benchmark_data_smithi045_46415_object757:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e151)", 2024-08-19T14:14:57.645 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:14:21.860923+0000", 2024-08-19T14:14:57.645 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 35.539171834999998, 2024-08-19T14:14:57.645 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.15233807699999999, 2024-08-19T14:14:57.645 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:14:57.645 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:14:57.645 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "waiting for ondisk", 2024-08-19T14:14:57.646 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:14:57.646 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:14:57.646 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:14:57.646 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 124038 2024-08-19T14:14:57.646 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.646 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:14:57.646 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.646 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:14:57.646 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.860923+0000", 2024-08-19T14:14:57.646 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:14:57.646 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.646 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.647 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:14:57.647 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.860923+0000", 2024-08-19T14:14:57.647 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:14:57.647 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.647 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.647 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:14:57.647 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.860924+0000", 2024-08-19T14:14:57.647 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2599999999999998e-07 2024-08-19T14:14:57.647 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.647 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.647 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:14:57.647 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.860926+0000", 2024-08-19T14:14:57.647 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.4109999999999999e-06 2024-08-19T14:14:57.647 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.648 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.648 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:14:57.648 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.860933+0000", 2024-08-19T14:14:57.648 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.0500000000000003e-06 2024-08-19T14:14:57.648 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.648 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.648 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:14:57.648 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.860939+0000", 2024-08-19T14:14:57.648 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.4079999999999997e-06 2024-08-19T14:14:57.648 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.648 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.648 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:14:57.648 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.861063+0000", 2024-08-19T14:14:57.649 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000124669 2024-08-19T14:14:57.649 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.649 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.649 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for ondisk", 2024-08-19T14:14:57.649 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.861116+0000", 2024-08-19T14:14:57.649 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.2723999999999999e-05 2024-08-19T14:14:57.649 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.649 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.649 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:14:57.649 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:22.013261+0000", 2024-08-19T14:14:57.649 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.15214538899999999 2024-08-19T14:14:57.649 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:14:57.649 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:14:57.650 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:14:57.650 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.650 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.650 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:127981 2.7 2:e437fe12:::benchmark_data_smithi045_46415_object4700:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-08-19T14:14:57.650 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:14:26.188669+0000", 2024-08-19T14:14:57.650 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 31.211425849000001, 2024-08-19T14:14:57.650 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.186748254, 2024-08-19T14:14:57.650 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:14:57.650 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:14:57.650 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:14:57.650 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:14:57.650 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:14:57.650 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:14:57.650 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 127981 2024-08-19T14:14:57.651 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.651 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:14:57.651 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.651 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:14:57.651 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:26.188669+0000", 2024-08-19T14:14:57.651 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:14:57.651 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.651 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.651 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:14:57.651 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:26.188669+0000", 2024-08-19T14:14:57.651 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:14:57.651 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.651 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.652 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:14:57.652 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:26.188671+0000", 2024-08-19T14:14:57.652 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.141e-06 2024-08-19T14:14:57.652 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.652 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.652 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:14:57.652 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:26.188675+0000", 2024-08-19T14:14:57.652 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.9690000000000002e-06 2024-08-19T14:14:57.652 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.652 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.652 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:14:57.652 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:26.188685+0000", 2024-08-19T14:14:57.652 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.7780000000000002e-06 2024-08-19T14:14:57.652 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.653 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.653 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:14:57.653 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:26.188692+0000", 2024-08-19T14:14:57.653 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.8600000000000004e-06 2024-08-19T14:14:57.653 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.653 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.653 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:14:57.653 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:26.190173+0000", 2024-08-19T14:14:57.653 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0014812429999999999 2024-08-19T14:14:57.653 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.653 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.653 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for missing object", 2024-08-19T14:14:57.653 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:26.190264+0000", 2024-08-19T14:14:57.653 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.0662999999999996e-05 2024-08-19T14:14:57.654 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.654 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.654 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:14:57.654 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:26.375404+0000", 2024-08-19T14:14:57.654 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.18513970799999999 2024-08-19T14:14:57.654 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.654 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.654 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:14:57.654 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:26.375418+0000", 2024-08-19T14:14:57.654 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3892e-05 2024-08-19T14:14:57.654 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:14:57.654 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:14:57.654 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:14:57.655 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.655 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.655 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:128197 2.7 2:ea40205b:::benchmark_data_smithi045_46415_object4916:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-08-19T14:14:57.655 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:14:26.359213+0000", 2024-08-19T14:14:57.655 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 31.040881836000001, 2024-08-19T14:14:57.655 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6391427649999999, 2024-08-19T14:14:57.655 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:14:57.655 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:14:57.655 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:14:57.655 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:14:57.655 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:14:57.655 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:14:57.655 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 128197 2024-08-19T14:14:57.655 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.656 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:14:57.656 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.656 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:14:57.656 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:26.359213+0000", 2024-08-19T14:14:57.656 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:14:57.656 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.656 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.656 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:14:57.656 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:26.359213+0000", 2024-08-19T14:14:57.656 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:14:57.656 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.656 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.656 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:14:57.656 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:26.359214+0000", 2024-08-19T14:14:57.657 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.9200000000000002e-07 2024-08-19T14:14:57.657 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.657 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.657 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:14:57.657 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:26.359215+0000", 2024-08-19T14:14:57.657 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.708e-06 2024-08-19T14:14:57.657 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.657 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.657 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:14:57.657 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:26.359221+0000", 2024-08-19T14:14:57.657 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.2530000000000004e-06 2024-08-19T14:14:57.657 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.657 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.657 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:14:57.658 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:26.359225+0000", 2024-08-19T14:14:57.658 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2610000000000004e-06 2024-08-19T14:14:57.658 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.658 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.658 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:14:57.658 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:26.362625+0000", 2024-08-19T14:14:57.658 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0034004370000000001 2024-08-19T14:14:57.658 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.658 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.658 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for missing object", 2024-08-19T14:14:57.658 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:26.362709+0000", 2024-08-19T14:14:57.658 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.3287000000000004e-05 2024-08-19T14:14:57.658 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.658 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.659 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:14:57.659 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:27.998347+0000", 2024-08-19T14:14:57.659 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6356388050000001 2024-08-19T14:14:57.659 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:14:57.659 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:14:57.659 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:14:57.659 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:27.998356+0000", 2024-08-19T14:14:57.659 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.7220000000000005e-06 2024-08-19T14:14:57.659 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:14:57.659 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:14:57.659 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:14:57.659 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:14:57.659 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:14:57.659 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:14:57.660 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:14:57.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:57.683 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:57.683 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:14:57.784 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:14:57.809 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:57.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:57.810 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:14:57.910 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:14:57.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:57.951 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:57.951 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:14:58.052 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:14:58.092 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:58.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:58.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:14:58.194 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:14:58.218 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:58.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:58.219 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:14:58.273 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:58 smithi031 ceph-mon[16336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:14:58.273 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:58 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:14:58.273 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:58 smithi031 ceph-mon[16336]: osdmap e167: 12 total, 12 up, 10 in 2024-08-19T14:14:58.319 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:14:58.323 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-08-19T14:14:58.342 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:58.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:58.343 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:14:58.444 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:14:58.468 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:58.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:58.469 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:14:58.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:58 smithi032 ceph-mon[20004]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:14:58.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:58 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:14:58.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:58 smithi032 ceph-mon[20004]: osdmap e167: 12 total, 12 up, 10 in 2024-08-19T14:14:58.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:58 smithi045 ceph-mon[20143]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:14:58.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:58 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:14:58.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:58 smithi045 ceph-mon[20143]: osdmap e167: 12 total, 12 up, 10 in 2024-08-19T14:14:58.571 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:14:58.615 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:58.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:58.616 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:14:58.717 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:14:58.740 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:58.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:58.741 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:14:58.842 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:14:58.866 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:58.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:58.867 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:14:58.968 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:14:58.994 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:58.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:58.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:14:59.096 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:14:59.124 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:59.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:59.125 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:14:59.226 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:14:59.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:59.250 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:59.250 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:14:59.351 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:14:59.368 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:14:59 smithi031 ceph-mon[16336]: pgmap v419: 27 pgs: 27 active+clean; 5.6 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 781 op/s; 0 B/s, 34 objects/s recovering 2024-08-19T14:14:59.374 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:59.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:59.375 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:14:59.429 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:14:59.452 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:14:59.452 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:14:59.459 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:14:59.476 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:14:59.489 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:14:59.489 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:14:59.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:59.503 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:59.503 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:14:59.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:14:59 smithi032 ceph-mon[20004]: pgmap v419: 27 pgs: 27 active+clean; 5.6 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 781 op/s; 0 B/s, 34 objects/s recovering 2024-08-19T14:14:59.526 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:14:59.526 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:14:59.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:14:59 smithi045 ceph-mon[20143]: pgmap v419: 27 pgs: 27 active+clean; 5.6 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 781 op/s; 0 B/s, 34 objects/s recovering 2024-08-19T14:14:59.554 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:14:59.554 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:14:59.583 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:14:59.584 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:14:59.604 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:14:59.608 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:14:59.608 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:14:59.639 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:14:59.639 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:14:59.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:59.648 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:14:59.648 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:14:59.673 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:14:59.673 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:14:59.704 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:14:59.704 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:14:59.742 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:14:59.743 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:14:59.749 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:14:59.775 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:14:59.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:59.777 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:14:59.877 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:14:59.902 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:14:59.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:14:59.903 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:15:00.004 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:15:00.021 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:15:00.022 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:15:00.028 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:00.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:00.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:15:00.065 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:15:00.066 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:15:00.130 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:15:00.160 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:00.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:00.161 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:15:00.262 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:15:00.301 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:00.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:00.303 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:15:00.361 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:00 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2912813102' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:15:00.386 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:15:00.387 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-08-19T14:15:00.404 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:15:00.428 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:00.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:00.429 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:15:00.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:00 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2912813102' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:15:00.529 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:15:00.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:00 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2912813102' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:15:00.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:00.565 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:00.566 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:15:00.667 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:15:00.690 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:00.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:00.691 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:15:00.792 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:15:00.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:00.825 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:00.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:15:00.926 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:15:00.959 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:00.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:00.960 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:15:01.061 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:15:01.087 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:01.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:01.088 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:15:01.176 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-19T14:15:01.189 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:15:01.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:01.212 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:01.212 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:15:01.313 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:15:01.352 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:01.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:01.353 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:15:01.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:01 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:15:01.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:01 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:15:01.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:01 smithi031 ceph-mon[16336]: pgmap v420: 27 pgs: 27 active+clean; 5.5 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 859 op/s 2024-08-19T14:15:01.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:01 smithi031 ceph-mon[16336]: osdmap e168: 12 total, 12 up, 10 in 2024-08-19T14:15:01.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:01 smithi031 ceph-mon[16336]: from='client.24842 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:15:01.454 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:15:01.478 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:01.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:01.480 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:15:01.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:01 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:15:01.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:01 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:15:01.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:01 smithi032 ceph-mon[20004]: pgmap v420: 27 pgs: 27 active+clean; 5.5 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 859 op/s 2024-08-19T14:15:01.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:01 smithi032 ceph-mon[20004]: osdmap e168: 12 total, 12 up, 10 in 2024-08-19T14:15:01.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:01 smithi032 ceph-mon[20004]: from='client.24842 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:15:01.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:01 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:15:01.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:01 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:15:01.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:01 smithi045 ceph-mon[20143]: pgmap v420: 27 pgs: 27 active+clean; 5.5 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 859 op/s 2024-08-19T14:15:01.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:01 smithi045 ceph-mon[20143]: osdmap e168: 12 total, 12 up, 10 in 2024-08-19T14:15:01.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:01 smithi045 ceph-mon[20143]: from='client.24842 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:15:01.581 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:15:01.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:01.629 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:01.630 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:15:01.731 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:15:01.758 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:01.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:01.760 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:15:01.861 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:15:01.891 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:01.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:01.892 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:15:01.992 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:15:02.018 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:02.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:02.019 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:15:02.121 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:15:02.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:02.158 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:02.158 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:15:02.259 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:15:02.296 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:02.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:02.298 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:15:02.399 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:15:02.433 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:02.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:02.434 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:15:02.536 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:15:02.559 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:02.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:02.560 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:15:02.661 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:15:02.695 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:02.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:02.696 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:15:02.797 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:15:02.820 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:02.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:02.821 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:15:02.922 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:15:02.953 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:02.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:02.954 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:15:03.055 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:15:03.081 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:03.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:03.082 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:15:03.183 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:15:03.214 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:03.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:03.215 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:15:03.316 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:15:03.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:03.352 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:03.353 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:15:03.440 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:03 smithi031 ceph-mon[16336]: pgmap v422: 27 pgs: 27 active+clean; 5.3 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 960 op/s 2024-08-19T14:15:03.454 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:15:03.476 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:03.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:03.478 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:15:03.485 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":421,"stamp":"2024-08-19T14:15:00.113145+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5856952888,"num_objects":89364,"num_object_clones":0,"num_object_copies":268092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89364,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":157196,"num_write_kb":7889793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14509,"num_bytes_recovered":912392192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158946,"ondisk_log_size":158946,"up":81,"acting":81,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":81,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":18053952,"kb_used_data":17256028,"kb_used_omap":15,"kb_used_meta":797552,"kb_avail":919397568,"statfs":{"total":959950356480,"available":941463109632,"internally_reserved":0,"allocated":17670172672,"data_stored":17661771144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15948,"internal_metadata":816693684},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,2,6,2,6,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-350093312,"num_objects":-5342,"num_object_clones":0,"num_object_copies":-16026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5342,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.345303"},"pg_stats":[{"pgid":"2.f","version":"167'10012","reported_seq":10234,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-08-19T14:14:59.583317+0000","last_change":"2024-08-19T14:14:10.955308+0000","last_active":"2024-08-19T14:14:59.583317+0000","last_peered":"2024-08-19T14:14:59.583317+0000","last_clean":"2024-08-19T14:14:59.583317+0000","last_became_active":"2024-08-19T14:12:43.417591+0000","last_became_peered":"2024-08-19T14:12:43.417591+0000","last_unstale":"2024-08-19T14:14:59.583317+0000","last_undegraded":"2024-08-19T14:14:59.583317+0000","last_fullsized":"2024-08-19T14:14:59.583317+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'7296","last_scrub_stamp":"2024-08-19T14:14:10.955043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:14:10.955043+0000","objects_scrubbed":7248,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:40.398760+0000","scrub_duration":1903,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":360054784,"num_objects":5494,"num_object_clones":0,"num_object_copies":16482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10012,"num_write_kb":496192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.e","version":"166'9684","reported_seq":9823,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-08-19T14:14:56.042782+0000","last_change":"2024-08-19T14:12:42.963558+0000","last_active":"2024-08-19T14:14:56.042782+0000","last_peered":"2024-08-19T14:14:56.042782+0000","last_clean":"2024-08-19T14:14:56.042782+0000","last_became_active":"2024-08-19T14:12:42.963208+0000","last_became_peered":"2024-08-19T14:12:42.963208+0000","last_unstale":"2024-08-19T14:14:56.042782+0000","last_undegraded":"2024-08-19T14:14:56.042782+0000","last_fullsized":"2024-08-19T14:14:56.042782+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:12:41.713780+0000","objects_scrubbed":0,"log_size":9684,"log_dups_size":0,"ondisk_log_size":9684,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":371982336,"num_objects":5676,"num_object_clones":0,"num_object_copies":17028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5676,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9684,"num_write_kb":491520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9],"acting":[2,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"167'9867","reported_seq":10088,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-08-19T14:14:58.202537+0000","last_change":"2024-08-19T14:14:11.678308+0000","last_active":"2024-08-19T14:14:58.202537+0000","last_peered":"2024-08-19T14:14:58.202537+0000","last_clean":"2024-08-19T14:14:58.202537+0000","last_became_active":"2024-08-19T14:12:43.417506+0000","last_became_peered":"2024-08-19T14:12:43.417506+0000","last_unstale":"2024-08-19T14:14:58.202537+0000","last_undegraded":"2024-08-19T14:14:58.202537+0000","last_fullsized":"2024-08-19T14:14:58.202537+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'7275","last_scrub_stamp":"2024-08-19T14:14:11.678069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:14:11.678069+0000","objects_scrubbed":7231,"log_size":9867,"log_dups_size":0,"ondisk_log_size":9867,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:23.845013+0000","scrub_duration":2180,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":361496600,"num_objects":5517,"num_object_clones":0,"num_object_copies":16551,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5517,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9867,"num_write_kb":492225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.c","version":"166'9670","reported_seq":9894,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-08-19T14:14:55.442087+0000","last_change":"2024-08-19T14:14:11.208263+0000","last_active":"2024-08-19T14:14:55.442087+0000","last_peered":"2024-08-19T14:14:55.442087+0000","last_clean":"2024-08-19T14:14:55.442087+0000","last_became_active":"2024-08-19T14:12:43.038912+0000","last_became_peered":"2024-08-19T14:12:43.038912+0000","last_unstale":"2024-08-19T14:14:55.442087+0000","last_undegraded":"2024-08-19T14:14:55.442087+0000","last_fullsized":"2024-08-19T14:14:55.442087+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'7262","last_scrub_stamp":"2024-08-19T14:14:11.208131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:14:11.208131+0000","objects_scrubbed":7178,"log_size":9670,"log_dups_size":0,"ondisk_log_size":9670,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:16.859224+0000","scrub_duration":3361,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":371720192,"num_objects":5672,"num_object_clones":0,"num_object_copies":17016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9670,"num_write_kb":490944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,1],"acting":[11,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"70'139","reported_seq":526,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-08-19T14:14:57.083530+0000","last_change":"2024-08-19T14:14:36.306513+0000","last_active":"2024-08-19T14:14:57.083530+0000","last_peered":"2024-08-19T14:14:57.083530+0000","last_clean":"2024-08-19T14:14:57.083530+0000","last_became_active":"2024-08-19T14:13:22.729498+0000","last_became_peered":"2024-08-19T14:13:22.729498+0000","last_unstale":"2024-08-19T14:14:57.083530+0000","last_undegraded":"2024-08-19T14:14:57.083530+0000","last_fullsized":"2024-08-19T14:14:57.083530+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:14:36.306409+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:14:36.306409+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:45.640981+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,9],"acting":[7,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"166'9845","reported_seq":10061,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-08-19T14:14:56.491290+0000","last_change":"2024-08-19T14:14:03.748630+0000","last_active":"2024-08-19T14:14:56.491290+0000","last_peered":"2024-08-19T14:14:56.491290+0000","last_clean":"2024-08-19T14:14:56.491290+0000","last_became_active":"2024-08-19T14:12:42.728978+0000","last_became_peered":"2024-08-19T14:12:42.728978+0000","last_unstale":"2024-08-19T14:14:56.491290+0000","last_undegraded":"2024-08-19T14:14:56.491290+0000","last_fullsized":"2024-08-19T14:14:56.491290+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"146'6847","last_scrub_stamp":"2024-08-19T14:14:03.748468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:14:03.748468+0000","objects_scrubbed":6806,"log_size":9845,"log_dups_size":0,"ondisk_log_size":9845,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:25.274768+0000","scrub_duration":1341,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":364707840,"num_objects":5565,"num_object_clones":0,"num_object_copies":16695,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5565,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9845,"num_write_kb":493120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.3","version":"70'139","reported_seq":410,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-08-19T14:14:57.096148+0000","last_change":"2024-08-19T14:14:38.031170+0000","last_active":"2024-08-19T14:14:57.096148+0000","last_peered":"2024-08-19T14:14:57.096148+0000","last_clean":"2024-08-19T14:14:57.096148+0000","last_became_active":"2024-08-19T14:13:22.727234+0000","last_became_peered":"2024-08-19T14:13:22.727234+0000","last_unstale":"2024-08-19T14:14:57.096148+0000","last_undegraded":"2024-08-19T14:14:57.096148+0000","last_fullsized":"2024-08-19T14:14:57.096148+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:14:38.031056+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:14:38.031056+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:16:03.171231+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"167'9786","reported_seq":16053,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-08-19T14:14:57.956891+0000","last_change":"2024-08-19T14:13:45.917116+0000","last_active":"2024-08-19T14:14:57.956891+0000","last_peered":"2024-08-19T14:14:57.956891+0000","last_clean":"2024-08-19T14:14:57.956891+0000","last_became_active":"2024-08-19T14:13:32.749994+0000","last_became_peered":"2024-08-19T14:13:32.749994+0000","last_unstale":"2024-08-19T14:14:57.956891+0000","last_undegraded":"2024-08-19T14:14:57.956891+0000","last_fullsized":"2024-08-19T14:14:57.956891+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"136'5761","last_scrub_stamp":"2024-08-19T14:13:45.916860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:13:45.916860+0000","objects_scrubbed":5716,"log_size":9786,"log_dups_size":0,"ondisk_log_size":9786,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:09.768267+0000","scrub_duration":1249,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370409472,"num_objects":5652,"num_object_clones":0,"num_object_copies":16956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5652,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9786,"num_write_kb":494016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3024,"num_bytes_recovered":198180864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,11],"acting":[7,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"70'139","reported_seq":524,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-08-19T14:14:57.083505+0000","last_change":"2024-08-19T14:14:31.408564+0000","last_active":"2024-08-19T14:14:57.083505+0000","last_peered":"2024-08-19T14:14:57.083505+0000","last_clean":"2024-08-19T14:14:57.083505+0000","last_became_active":"2024-08-19T14:13:05.607783+0000","last_became_peered":"2024-08-19T14:13:05.607783+0000","last_unstale":"2024-08-19T14:14:57.083505+0000","last_undegraded":"2024-08-19T14:14:57.083505+0000","last_fullsized":"2024-08-19T14:14:57.083505+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:14:31.408492+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:14:31.408492+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:41.655086+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"166'9766","reported_seq":9906,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-08-19T14:14:55.994417+0000","last_change":"2024-08-19T14:12:43.039198+0000","last_active":"2024-08-19T14:14:55.994417+0000","last_peered":"2024-08-19T14:14:55.994417+0000","last_clean":"2024-08-19T14:14:55.994417+0000","last_became_active":"2024-08-19T14:12:43.038833+0000","last_became_peered":"2024-08-19T14:12:43.038833+0000","last_unstale":"2024-08-19T14:14:55.994417+0000","last_undegraded":"2024-08-19T14:14:55.994417+0000","last_fullsized":"2024-08-19T14:14:55.994417+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:12:41.713780+0000","objects_scrubbed":0,"log_size":9766,"log_dups_size":0,"ondisk_log_size":9766,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":376176640,"num_objects":5740,"num_object_clones":0,"num_object_copies":17220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5740,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9766,"num_write_kb":496192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"70'139","reported_seq":390,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-08-19T14:14:57.085946+0000","last_change":"2024-08-19T14:14:07.456593+0000","last_active":"2024-08-19T14:14:57.085946+0000","last_peered":"2024-08-19T14:14:57.085946+0000","last_clean":"2024-08-19T14:14:57.085946+0000","last_became_active":"2024-08-19T14:13:11.663185+0000","last_became_peered":"2024-08-19T14:13:11.663185+0000","last_unstale":"2024-08-19T14:14:57.085946+0000","last_undegraded":"2024-08-19T14:14:57.085946+0000","last_fullsized":"2024-08-19T14:14:57.085946+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:14:07.456464+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:14:07.456464+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:09.321073+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2],"acting":[9,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.2","version":"166'9755","reported_seq":17791,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-08-19T14:14:56.461031+0000","last_change":"2024-08-19T14:14:13.196736+0000","last_active":"2024-08-19T14:14:56.461031+0000","last_peered":"2024-08-19T14:14:56.461031+0000","last_clean":"2024-08-19T14:14:56.461031+0000","last_became_active":"2024-08-19T14:13:58.623348+0000","last_became_peered":"2024-08-19T14:13:58.623348+0000","last_unstale":"2024-08-19T14:14:56.461031+0000","last_undegraded":"2024-08-19T14:14:56.461031+0000","last_fullsized":"2024-08-19T14:14:56.461031+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"148'7390","last_scrub_stamp":"2024-08-19T14:14:13.196497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:14:13.196497+0000","objects_scrubbed":7346,"log_size":9755,"log_dups_size":0,"ondisk_log_size":9755,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:26.733911+0000","scrub_duration":1724,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":372310016,"num_objects":5681,"num_object_clones":0,"num_object_copies":17043,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5681,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9755,"num_write_kb":493952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3902,"num_bytes_recovered":255721472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"70'139","reported_seq":375,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-08-19T14:14:57.111236+0000","last_change":"2024-08-19T14:14:48.488485+0000","last_active":"2024-08-19T14:14:57.111236+0000","last_peered":"2024-08-19T14:14:57.111236+0000","last_clean":"2024-08-19T14:14:57.111236+0000","last_became_active":"2024-08-19T14:13:22.739150+0000","last_became_peered":"2024-08-19T14:13:22.739150+0000","last_unstale":"2024-08-19T14:14:57.111236+0000","last_undegraded":"2024-08-19T14:14:57.111236+0000","last_fullsized":"2024-08-19T14:14:57.111236+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:14:48.488367+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:14:48.488367+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:16:00.551219+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"167'9916","reported_seq":16395,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-08-19T14:14:58.013089+0000","last_change":"2024-08-19T14:14:09.998977+0000","last_active":"2024-08-19T14:14:58.013089+0000","last_peered":"2024-08-19T14:14:58.013089+0000","last_clean":"2024-08-19T14:14:58.013089+0000","last_became_active":"2024-08-19T14:14:04.681118+0000","last_became_peered":"2024-08-19T14:14:04.681118+0000","last_unstale":"2024-08-19T14:14:58.013089+0000","last_undegraded":"2024-08-19T14:14:58.013089+0000","last_fullsized":"2024-08-19T14:14:58.013089+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'7194","last_scrub_stamp":"2024-08-19T14:14:09.998849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:14:09.998849+0000","objects_scrubbed":7135,"log_size":9916,"log_dups_size":0,"ondisk_log_size":9916,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:22.243440+0000","scrub_duration":2842,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362545152,"num_objects":5532,"num_object_clones":0,"num_object_copies":16596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5532,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9916,"num_write_kb":494336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3123,"num_bytes_recovered":204668928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"70'139","reported_seq":390,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-08-19T14:14:57.085994+0000","last_change":"2024-08-19T14:14:06.461983+0000","last_active":"2024-08-19T14:14:57.085994+0000","last_peered":"2024-08-19T14:14:57.085994+0000","last_clean":"2024-08-19T14:14:57.085994+0000","last_became_active":"2024-08-19T14:13:11.647085+0000","last_became_peered":"2024-08-19T14:13:11.647085+0000","last_unstale":"2024-08-19T14:14:57.085994+0000","last_undegraded":"2024-08-19T14:14:57.085994+0000","last_fullsized":"2024-08-19T14:14:57.085994+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:14:06.461855+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:14:06.461855+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:12.802907+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,4],"acting":[9,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.5","version":"167'9970","reported_seq":10195,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-08-19T14:14:59.551005+0000","last_change":"2024-08-19T14:14:14.222353+0000","last_active":"2024-08-19T14:14:59.551005+0000","last_peered":"2024-08-19T14:14:59.551005+0000","last_clean":"2024-08-19T14:14:59.551005+0000","last_became_active":"2024-08-19T14:12:43.417616+0000","last_became_peered":"2024-08-19T14:12:43.417616+0000","last_unstale":"2024-08-19T14:14:59.551005+0000","last_undegraded":"2024-08-19T14:14:59.551005+0000","last_fullsized":"2024-08-19T14:14:59.551005+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"148'7426","last_scrub_stamp":"2024-08-19T14:14:14.222198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:14:14.222198+0000","objects_scrubbed":7364,"log_size":9970,"log_dups_size":0,"ondisk_log_size":9970,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:35.064129+0000","scrub_duration":2125,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358875136,"num_objects":5476,"num_object_clones":0,"num_object_copies":16428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5476,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9970,"num_write_kb":494272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.5","version":"70'139","reported_seq":397,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-08-19T14:14:54.075762+0000","last_change":"2024-08-19T14:14:49.752154+0000","last_active":"2024-08-19T14:14:54.075762+0000","last_peered":"2024-08-19T14:14:54.075762+0000","last_clean":"2024-08-19T14:14:54.075762+0000","last_became_active":"2024-08-19T14:13:22.727098+0000","last_became_peered":"2024-08-19T14:13:22.727098+0000","last_unstale":"2024-08-19T14:14:54.075762+0000","last_undegraded":"2024-08-19T14:14:54.075762+0000","last_fullsized":"2024-08-19T14:14:54.075762+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:14:49.752072+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:14:49.752072+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:16:05.519885+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"167'11023","reported_seq":11309,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-08-19T14:14:58.009982+0000","last_change":"2024-08-19T14:14:40.326683+0000","last_active":"2024-08-19T14:14:58.009982+0000","last_peered":"2024-08-19T14:14:58.009982+0000","last_clean":"2024-08-19T14:14:58.009982+0000","last_became_active":"2024-08-19T14:12:42.963203+0000","last_became_peered":"2024-08-19T14:12:42.963203+0000","last_unstale":"2024-08-19T14:14:58.009982+0000","last_undegraded":"2024-08-19T14:14:58.009982+0000","last_fullsized":"2024-08-19T14:14:58.009982+0000","mapping_epoch":102,"log_start":"107'1000","ondisk_log_start":"107'1000","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'10039","last_scrub_stamp":"2024-08-19T14:14:40.326519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:14:40.326519+0000","objects_scrubbed":6619,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:43.279737+0000","scrub_duration":3261,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362414080,"num_objects":5530,"num_object_clones":0,"num_object_copies":16590,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5530,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9802,"num_write_kb":490624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,3],"acting":[1,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"70'139","reported_seq":447,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-08-19T14:14:54.065368+0000","last_change":"2024-08-19T14:14:36.831290+0000","last_active":"2024-08-19T14:14:54.065368+0000","last_peered":"2024-08-19T14:14:54.065368+0000","last_clean":"2024-08-19T14:14:54.065368+0000","last_became_active":"2024-08-19T14:13:05.607447+0000","last_became_peered":"2024-08-19T14:13:05.607447+0000","last_unstale":"2024-08-19T14:14:54.065368+0000","last_undegraded":"2024-08-19T14:14:54.065368+0000","last_fullsized":"2024-08-19T14:14:54.065368+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:14:36.831171+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:14:36.831171+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:53.437142+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.7","version":"166'9710","reported_seq":21181,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-08-19T14:14:56.049227+0000","last_change":"2024-08-19T14:14:50.008022+0000","last_active":"2024-08-19T14:14:56.049227+0000","last_peered":"2024-08-19T14:14:56.049227+0000","last_clean":"2024-08-19T14:14:56.049227+0000","last_became_active":"2024-08-19T14:14:49.991243+0000","last_became_peered":"2024-08-19T14:14:49.991243+0000","last_unstale":"2024-08-19T14:14:56.049227+0000","last_undegraded":"2024-08-19T14:14:56.049227+0000","last_fullsized":"2024-08-19T14:14:56.049227+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:12:41.713780+0000","objects_scrubbed":0,"log_size":9710,"log_dups_size":0,"ondisk_log_size":9710,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":374079488,"num_objects":5708,"num_object_clones":0,"num_object_copies":17124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5708,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9710,"num_write_kb":493376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4460,"num_bytes_recovered":253820928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.8","version":"167'9938","reported_seq":10159,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-08-19T14:14:57.958665+0000","last_change":"2024-08-19T14:14:08.982261+0000","last_active":"2024-08-19T14:14:57.958665+0000","last_peered":"2024-08-19T14:14:57.958665+0000","last_clean":"2024-08-19T14:14:57.958665+0000","last_became_active":"2024-08-19T14:12:43.038882+0000","last_became_peered":"2024-08-19T14:12:43.038882+0000","last_unstale":"2024-08-19T14:14:57.958665+0000","last_undegraded":"2024-08-19T14:14:57.958665+0000","last_fullsized":"2024-08-19T14:14:57.958665+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'7200","last_scrub_stamp":"2024-08-19T14:14:08.982137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:14:08.982137+0000","objects_scrubbed":7142,"log_size":9938,"log_dups_size":0,"ondisk_log_size":9938,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:09.792768+0000","scrub_duration":1562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":365035520,"num_objects":5570,"num_object_clones":0,"num_object_copies":16710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9938,"num_write_kb":496256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,0],"acting":[7,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"70'139","reported_seq":411,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-08-19T14:14:57.086040+0000","last_change":"2024-08-19T14:14:31.607909+0000","last_active":"2024-08-19T14:14:57.086040+0000","last_peered":"2024-08-19T14:14:57.086040+0000","last_clean":"2024-08-19T14:14:57.086040+0000","last_became_active":"2024-08-19T14:13:22.728779+0000","last_became_peered":"2024-08-19T14:13:22.728779+0000","last_unstale":"2024-08-19T14:14:57.086040+0000","last_undegraded":"2024-08-19T14:14:57.086040+0000","last_fullsized":"2024-08-19T14:14:57.086040+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:14:31.607795+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:14:31.607795+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:48.816381+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,0],"acting":[9,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"167'9751","reported_seq":9968,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-08-19T14:14:58.017387+0000","last_change":"2024-08-19T14:14:05.393154+0000","last_active":"2024-08-19T14:14:58.017387+0000","last_peered":"2024-08-19T14:14:58.017387+0000","last_clean":"2024-08-19T14:14:58.017387+0000","last_became_active":"2024-08-19T14:12:42.963114+0000","last_became_peered":"2024-08-19T14:12:42.963114+0000","last_unstale":"2024-08-19T14:14:58.017387+0000","last_undegraded":"2024-08-19T14:14:58.017387+0000","last_fullsized":"2024-08-19T14:14:58.017387+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'6812","last_scrub_stamp":"2024-08-19T14:14:05.392787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:14:05.392787+0000","objects_scrubbed":6777,"log_size":9751,"log_dups_size":0,"ondisk_log_size":9751,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:16.463202+0000","scrub_duration":1333,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":357498880,"num_objects":5455,"num_object_clones":0,"num_object_copies":16365,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5455,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9751,"num_write_kb":486592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"70'139","reported_seq":524,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-08-19T14:14:57.083465+0000","last_change":"2024-08-19T14:14:35.319452+0000","last_active":"2024-08-19T14:14:57.083465+0000","last_peered":"2024-08-19T14:14:57.083465+0000","last_clean":"2024-08-19T14:14:57.083465+0000","last_became_active":"2024-08-19T14:13:05.607605+0000","last_became_peered":"2024-08-19T14:13:05.607605+0000","last_unstale":"2024-08-19T14:14:57.083465+0000","last_undegraded":"2024-08-19T14:14:57.083465+0000","last_fullsized":"2024-08-19T14:14:57.083465+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:14:35.319333+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:14:35.319333+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:46.300019+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"166'9754","reported_seq":9974,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-08-19T14:14:55.473556+0000","last_change":"2024-08-19T14:14:11.437361+0000","last_active":"2024-08-19T14:14:55.473556+0000","last_peered":"2024-08-19T14:14:55.473556+0000","last_clean":"2024-08-19T14:14:55.473556+0000","last_became_active":"2024-08-19T14:12:42.963040+0000","last_became_peered":"2024-08-19T14:12:42.963040+0000","last_unstale":"2024-08-19T14:14:55.473556+0000","last_undegraded":"2024-08-19T14:14:55.473556+0000","last_fullsized":"2024-08-19T14:14:55.473556+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'7289","last_scrub_stamp":"2024-08-19T14:14:11.437152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:14:11.437152+0000","objects_scrubbed":7227,"log_size":9754,"log_dups_size":0,"ondisk_log_size":9754,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:32.239860+0000","scrub_duration":2564,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":372506624,"num_objects":5684,"num_object_clones":0,"num_object_copies":17052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9754,"num_write_kb":494016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,9],"acting":[11,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.8","version":"70'139","reported_seq":400,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-08-19T14:14:54.065298+0000","last_change":"2024-08-19T14:14:35.594757+0000","last_active":"2024-08-19T14:14:54.065298+0000","last_peered":"2024-08-19T14:14:54.065298+0000","last_clean":"2024-08-19T14:14:54.065298+0000","last_became_active":"2024-08-19T14:13:05.605694+0000","last_became_peered":"2024-08-19T14:13:05.605694+0000","last_unstale":"2024-08-19T14:14:54.065298+0000","last_undegraded":"2024-08-19T14:14:54.065298+0000","last_fullsized":"2024-08-19T14:14:54.065298+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:14:35.594637+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:14:35.594637+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:43.962595+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.b","version":"167'9970","reported_seq":10188,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-08-19T14:14:59.694284+0000","last_change":"2024-08-19T14:14:06.266442+0000","last_active":"2024-08-19T14:14:59.694284+0000","last_peered":"2024-08-19T14:14:59.694284+0000","last_clean":"2024-08-19T14:14:59.694284+0000","last_became_active":"2024-08-19T14:12:42.729014+0000","last_became_peered":"2024-08-19T14:12:42.729014+0000","last_unstale":"2024-08-19T14:14:59.694284+0000","last_undegraded":"2024-08-19T14:14:59.694284+0000","last_fullsized":"2024-08-19T14:14:59.694284+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'6965","last_scrub_stamp":"2024-08-19T14:14:06.266214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:14:06.266214+0000","objects_scrubbed":6919,"log_size":9970,"log_dups_size":0,"ondisk_log_size":9970,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:18.095310+0000","scrub_duration":1296,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":354549760,"num_objects":5410,"num_object_clones":0,"num_object_copies":16230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9970,"num_write_kb":492160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":5856362520,"num_objects":89362,"num_object_clones":0,"num_object_copies":268086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89362,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":157196,"num_write_kb":7889793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14509,"num_bytes_recovered":912392192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18120978432,"data_stored":18120966216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157417,"ondisk_log_size":157417,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1529,"ondisk_log_size":1529,"up":33,"acting":33,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187499,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1541120,"kb_used_data":1469688,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":92204032,"statfs":{"total":95995035648,"available":94416928768,"internally_reserved":0,"allocated":1504960512,"data_stored":1504125248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":73136580},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":343597383728,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":74,"seq":317827579957,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2614868,"kb_used_data":2515656,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":91130284,"statfs":{"total":95995035648,"available":93317410816,"internally_reserved":0,"allocated":2576031744,"data_stored":2575188072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":101579201},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874299,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1468556,"kb_used_data":1405444,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":92276596,"statfs":{"total":95995035648,"available":94491234304,"internally_reserved":0,"allocated":1439174656,"data_stored":1438335056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":57,"seq":244813135933,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1851344,"kb_used_data":1784068,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":91893808,"statfs":{"total":95995035648,"available":94099259392,"internally_reserved":0,"allocated":1826885632,"data_stored":1826046032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":68876746},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":206158430273,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691909,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":786824,"kb_used_data":729280,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":92958328,"statfs":{"total":95995035648,"available":95189327872,"internally_reserved":0,"allocated":746782720,"data_stored":745943904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986250,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1512916,"kb_used_data":1437320,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":92232236,"statfs":{"total":95995035648,"available":94445809664,"internally_reserved":0,"allocated":1471815680,"data_stored":1470972008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149776,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2573016,"kb_used_data":2473808,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":91172136,"statfs":{"total":95995035648,"available":93360267264,"internally_reserved":0,"allocated":2533179392,"data_stored":2532332168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":101579207},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411412,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1930248,"kb_used_data":1822720,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":91814904,"statfs":{"total":95995035648,"available":94018461696,"internally_reserved":0,"allocated":1866465280,"data_stored":1865626464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574937,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1882192,"kb_used_data":1788420,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":91862960,"statfs":{"total":95995035648,"available":94067671040,"internally_reserved":0,"allocated":1831342080,"data_stored":1830502480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":95877573},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,0,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738461,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1892868,"kb_used_data":1829624,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":91852284,"statfs":{"total":95995035648,"available":94056738816,"internally_reserved":0,"allocated":1873534976,"data_stored":1872699712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1869873152,"data_stored":1869873152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1811349504,"data_stored":1811349504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1859452928,"data_stored":1859452928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2522222592,"data_stored":2522218520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1464209408,"data_stored":1464205336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":737017856,"data_stored":737017856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":367853568,"data_stored":367853568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1812791296,"data_stored":1812791296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1429995520,"data_stored":1429995520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":2552631296,"data_stored":2552627224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":198115328,"data_stored":198115328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1495465984,"data_stored":1495465984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T14:15:03.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:03 smithi032 ceph-mon[20004]: pgmap v422: 27 pgs: 27 active+clean; 5.3 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 960 op/s 2024-08-19T14:15:03.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:03 smithi045 ceph-mon[20143]: pgmap v422: 27 pgs: 27 active+clean; 5.3 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 960 op/s 2024-08-19T14:15:03.578 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:15:03.604 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:03.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:03.605 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:15:03.682 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:15:03.682 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:15:03.706 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:15:03.729 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:03.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:03.730 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:15:03.831 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:15:03.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:03.856 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:03.856 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:15:03.957 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:15:04.000 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:04.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:04.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:15:04.103 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:15:04.126 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:04.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:04.128 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:15:04.228 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:15:04.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:04.264 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:04.264 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:15:04.365 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:15:04.375 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-08-19T14:15:04.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:04.409 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:04.409 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:15:04.511 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:15:04.570 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:04.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:04.571 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:15:04.672 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:15:04.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:04.698 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:04.699 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:15:04.800 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:15:04.828 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:04.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:04.829 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:15:04.930 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:15:04.956 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:04.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:04.956 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:15:05.039 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:15:05.057 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:15:05.080 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:05.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:05.081 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:15:05.182 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:15:05.227 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:05.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:05.228 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:15:05.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:05 smithi031 ceph-mon[16336]: pgmap v423: 27 pgs: 27 active+clean; 5.2 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 973 op/s 2024-08-19T14:15:05.329 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:15:05.354 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:05.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:05.356 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:15:05.457 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:15:05.502 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:05.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:05.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:15:05.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:05 smithi032 ceph-mon[20004]: pgmap v423: 27 pgs: 27 active+clean; 5.2 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 973 op/s 2024-08-19T14:15:05.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:05 smithi045 ceph-mon[20143]: pgmap v423: 27 pgs: 27 active+clean; 5.2 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 973 op/s 2024-08-19T14:15:05.603 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:15:05.631 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:05.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:05.632 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:15:05.733 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:15:05.756 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:05.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:05.757 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:15:05.858 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:15:05.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:05.894 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:05.894 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:15:05.995 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:15:06.023 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:06.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:06.025 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:15:06.125 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:15:06.149 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:06.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:06.150 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:15:06.251 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:15:06.288 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:06.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:06.289 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:15:06.390 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:15:06.414 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:06.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:06.415 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:15:06.488 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-19T14:15:06.516 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:15:06.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:06.540 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:06.540 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:15:06.600 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:15:06.641 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:15:06.664 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:06.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:06.665 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:15:06.766 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:15:06.789 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:06.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:06.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:15:06.891 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:15:06.939 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:06.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:06.940 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:15:07.041 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:15:07.054 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:15:07.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:07.084 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:07.084 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:15:07.134 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is unset 2024-08-19T14:15:07.185 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:15:07.207 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:07.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:07.208 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:15:07.225 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:15:07.225 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:15:07.309 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:15:07.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:07.348 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:07.348 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:15:07.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:07 smithi031 ceph-mon[16336]: pgmap v424: 27 pgs: 27 active+clean; 5.1 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 909 op/s 2024-08-19T14:15:07.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:07 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1953747049' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:15:07.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:07 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:15:07.449 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:15:07.473 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:07.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:07.474 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:15:07.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:07 smithi032 ceph-mon[20004]: pgmap v424: 27 pgs: 27 active+clean; 5.1 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 909 op/s 2024-08-19T14:15:07.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:07 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1953747049' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:15:07.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:07 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:15:07.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:07 smithi045 ceph-mon[20143]: pgmap v424: 27 pgs: 27 active+clean; 5.1 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 909 op/s 2024-08-19T14:15:07.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:07 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1953747049' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:15:07.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:07 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:15:07.576 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:15:07.601 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:07.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:07.602 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:15:07.703 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:15:07.730 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:07.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:07.731 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:15:07.832 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:15:07.857 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:07.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:07.858 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:15:07.959 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:15:07.981 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:07.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:07.982 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:15:08.083 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:15:08.115 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:08.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:08.116 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:15:08.217 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:15:08.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:08.253 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:08.253 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:15:08.254 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:08 smithi031 ceph-mon[16336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:15:08.254 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:08 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:15:08.254 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:08 smithi031 ceph-mon[16336]: osdmap e169: 12 total, 12 up, 10 in 2024-08-19T14:15:08.303 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-08-19T14:15:08.354 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:15:08.377 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:08.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:08.378 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:15:08.479 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:15:08.501 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:08.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:08.502 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:15:08.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:08 smithi032 ceph-mon[20004]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:15:08.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:08 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:15:08.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:08 smithi032 ceph-mon[20004]: osdmap e169: 12 total, 12 up, 10 in 2024-08-19T14:15:08.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:08 smithi045 ceph-mon[20143]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:15:08.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:08 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:15:08.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:08 smithi045 ceph-mon[20143]: osdmap e169: 12 total, 12 up, 10 in 2024-08-19T14:15:08.603 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:15:08.661 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:08.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:08.662 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:15:08.763 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:15:08.806 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:08.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:08.807 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:15:08.909 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:15:08.911 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:15:08.937 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:08.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:08.938 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:15:08.989 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:15:09.038 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:15:09.068 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:09.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:09.069 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:15:09.170 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:15:09.196 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:09.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:09.197 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:15:09.298 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:15:09.314 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:09 smithi031 ceph-mon[16336]: pgmap v426: 27 pgs: 27 active+clean; 5.0 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 865 op/s 2024-08-19T14:15:09.323 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:09.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:09.324 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:15:09.425 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:15:09.458 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:09.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:09.459 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:15:09.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:09 smithi032 ceph-mon[20004]: pgmap v426: 27 pgs: 27 active+clean; 5.0 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 865 op/s 2024-08-19T14:15:09.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:09 smithi045 ceph-mon[20143]: pgmap v426: 27 pgs: 27 active+clean; 5.0 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 865 op/s 2024-08-19T14:15:09.560 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:15:09.592 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:09.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:09.593 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:15:09.694 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:15:09.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:09.730 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:09.730 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:15:09.829 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:15:09.830 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:15:09.874 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:15:09.874 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:15:09.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:09.883 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:09.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:15:09.984 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:15:10.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:10.011 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:10.011 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:15:10.093 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:15:10.094 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-08-19T14:15:10.112 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:15:10.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:10.137 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:10.137 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:15:10.238 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:15:10.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:10.261 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:10.261 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:15:10.362 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:15:10.365 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:10 smithi031 ceph-mon[16336]: 1.1 scrub starts 2024-08-19T14:15:10.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:10 smithi031 ceph-mon[16336]: 1.1 scrub ok 2024-08-19T14:15:10.399 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:10.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:10.400 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:15:10.502 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:15:10.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:10 smithi032 ceph-mon[20004]: 1.1 scrub starts 2024-08-19T14:15:10.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:10 smithi032 ceph-mon[20004]: 1.1 scrub ok 2024-08-19T14:15:10.527 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:10.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:10.528 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:15:10.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:10 smithi045 ceph-mon[20143]: 1.1 scrub starts 2024-08-19T14:15:10.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:10 smithi045 ceph-mon[20143]: 1.1 scrub ok 2024-08-19T14:15:10.629 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:15:10.672 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:10.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:10.673 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:15:10.774 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:15:10.796 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:10.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:10.797 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:15:10.815 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-08-19T14:15:10.847 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-19T14:15:10.898 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:15:10.944 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:10.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:10.945 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:15:11.046 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:15:11.070 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:11.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:11.071 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:15:11.166 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:15:11.172 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:15:11.200 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:11.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:11.202 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:15:11.303 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:15:11.329 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:11.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:11.331 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:15:11.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:11 smithi031 ceph-mon[16336]: pgmap v427: 27 pgs: 27 active+clean; 4.9 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 990 op/s 2024-08-19T14:15:11.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:11 smithi031 ceph-mon[16336]: from='client.24854 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:15:11.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:11 smithi031 ceph-mon[16336]: 2.0 scrub starts 2024-08-19T14:15:11.432 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:15:11.452 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:11 smithi032 ceph-mon[20004]: pgmap v427: 27 pgs: 27 active+clean; 4.9 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 990 op/s 2024-08-19T14:15:11.452 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:11 smithi032 ceph-mon[20004]: from='client.24854 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:15:11.452 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:11 smithi032 ceph-mon[20004]: 2.0 scrub starts 2024-08-19T14:15:11.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:11.460 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:11.461 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:15:11.522 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":427,"stamp":"2024-08-19T14:15:09.771272+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5229969976,"num_objects":79797,"num_object_clones":0,"num_object_copies":239391,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79797,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":166763,"num_write_kb":7889793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14509,"num_bytes_recovered":912392192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":162313,"ondisk_log_size":162313,"up":81,"acting":81,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":81,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":16329136,"kb_used_data":15428556,"kb_used_omap":15,"kb_used_meta":900272,"kb_avail":921122384,"statfs":{"total":959950356480,"available":943229321216,"internally_reserved":0,"allocated":15798841344,"data_stored":15790353596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15948,"internal_metadata":921878964},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-626982912,"num_objects":-9567,"num_object_clones":0,"num_object_copies":-28701,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-9567,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9567,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.658127"},"pg_stats":[{"pgid":"2.f","version":"169'10610","reported_seq":10836,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-08-19T14:15:09.658075+0000","last_change":"2024-08-19T14:14:10.955308+0000","last_active":"2024-08-19T14:15:09.658075+0000","last_peered":"2024-08-19T14:15:09.658075+0000","last_clean":"2024-08-19T14:15:09.658075+0000","last_became_active":"2024-08-19T14:12:43.417591+0000","last_became_peered":"2024-08-19T14:12:43.417591+0000","last_unstale":"2024-08-19T14:15:09.658075+0000","last_undegraded":"2024-08-19T14:15:09.658075+0000","last_fullsized":"2024-08-19T14:15:09.658075+0000","mapping_epoch":102,"log_start":"107'600","ondisk_log_start":"107'600","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'7296","last_scrub_stamp":"2024-08-19T14:14:10.955043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:14:10.955043+0000","objects_scrubbed":7248,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:40.398760+0000","scrub_duration":1903,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320864256,"num_objects":4896,"num_object_clones":0,"num_object_copies":14688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10610,"num_write_kb":496192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.e","version":"168'10307","reported_seq":10451,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-08-19T14:15:06.038235+0000","last_change":"2024-08-19T14:12:42.963558+0000","last_active":"2024-08-19T14:15:06.038235+0000","last_peered":"2024-08-19T14:15:06.038235+0000","last_clean":"2024-08-19T14:15:06.038235+0000","last_became_active":"2024-08-19T14:12:42.963208+0000","last_became_peered":"2024-08-19T14:12:42.963208+0000","last_unstale":"2024-08-19T14:15:06.038235+0000","last_undegraded":"2024-08-19T14:15:06.038235+0000","last_fullsized":"2024-08-19T14:15:06.038235+0000","mapping_epoch":102,"log_start":"107'300","ondisk_log_start":"107'300","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:12:41.713780+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:09.852329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":331153408,"num_objects":5053,"num_object_clones":0,"num_object_copies":15159,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10307,"num_write_kb":491520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9],"acting":[2,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"169'10476","reported_seq":10700,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-08-19T14:15:08.231300+0000","last_change":"2024-08-19T14:14:11.678308+0000","last_active":"2024-08-19T14:15:08.231300+0000","last_peered":"2024-08-19T14:15:08.231300+0000","last_clean":"2024-08-19T14:15:08.231300+0000","last_became_active":"2024-08-19T14:12:43.417506+0000","last_became_peered":"2024-08-19T14:12:43.417506+0000","last_unstale":"2024-08-19T14:15:08.231300+0000","last_undegraded":"2024-08-19T14:15:08.231300+0000","last_fullsized":"2024-08-19T14:15:08.231300+0000","mapping_epoch":102,"log_start":"107'400","ondisk_log_start":"107'400","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'7275","last_scrub_stamp":"2024-08-19T14:14:11.678069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:14:11.678069+0000","objects_scrubbed":7231,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:23.845013+0000","scrub_duration":2180,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":321585176,"num_objects":4908,"num_object_clones":0,"num_object_copies":14724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4908,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10476,"num_write_kb":492225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.c","version":"168'10242","reported_seq":10465,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-08-19T14:15:05.529366+0000","last_change":"2024-08-19T14:14:11.208263+0000","last_active":"2024-08-19T14:15:05.529366+0000","last_peered":"2024-08-19T14:15:05.529366+0000","last_clean":"2024-08-19T14:15:05.529366+0000","last_became_active":"2024-08-19T14:12:43.038912+0000","last_became_peered":"2024-08-19T14:12:43.038912+0000","last_unstale":"2024-08-19T14:15:05.529366+0000","last_undegraded":"2024-08-19T14:15:05.529366+0000","last_fullsized":"2024-08-19T14:15:05.529366+0000","mapping_epoch":102,"log_start":"106'200","ondisk_log_start":"106'200","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'7262","last_scrub_stamp":"2024-08-19T14:14:11.208131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:14:11.208131+0000","objects_scrubbed":7178,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:16.859224+0000","scrub_duration":3361,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334233600,"num_objects":5100,"num_object_clones":0,"num_object_copies":15300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5100,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10242,"num_write_kb":490944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,1],"acting":[11,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"70'139","reported_seq":530,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-08-19T14:15:07.144128+0000","last_change":"2024-08-19T14:14:36.306513+0000","last_active":"2024-08-19T14:15:07.144128+0000","last_peered":"2024-08-19T14:15:07.144128+0000","last_clean":"2024-08-19T14:15:07.144128+0000","last_became_active":"2024-08-19T14:13:22.729498+0000","last_became_peered":"2024-08-19T14:13:22.729498+0000","last_unstale":"2024-08-19T14:15:07.144128+0000","last_undegraded":"2024-08-19T14:15:07.144128+0000","last_fullsized":"2024-08-19T14:15:07.144128+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:14:36.306409+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:14:36.306409+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:45.640981+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,9],"acting":[7,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"168'10453","reported_seq":10673,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-08-19T14:15:06.484725+0000","last_change":"2024-08-19T14:14:03.748630+0000","last_active":"2024-08-19T14:15:06.484725+0000","last_peered":"2024-08-19T14:15:06.484725+0000","last_clean":"2024-08-19T14:15:06.484725+0000","last_became_active":"2024-08-19T14:12:42.728978+0000","last_became_peered":"2024-08-19T14:12:42.728978+0000","last_unstale":"2024-08-19T14:15:06.484725+0000","last_undegraded":"2024-08-19T14:15:06.484725+0000","last_fullsized":"2024-08-19T14:15:06.484725+0000","mapping_epoch":102,"log_start":"107'400","ondisk_log_start":"107'400","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"146'6847","last_scrub_stamp":"2024-08-19T14:14:03.748468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:14:03.748468+0000","objects_scrubbed":6806,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:25.274768+0000","scrub_duration":1341,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":324861952,"num_objects":4957,"num_object_clones":0,"num_object_copies":14871,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10453,"num_write_kb":493120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.3","version":"70'139","reported_seq":414,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-08-19T14:15:07.142621+0000","last_change":"2024-08-19T14:14:38.031170+0000","last_active":"2024-08-19T14:15:07.142621+0000","last_peered":"2024-08-19T14:15:07.142621+0000","last_clean":"2024-08-19T14:15:07.142621+0000","last_became_active":"2024-08-19T14:13:22.727234+0000","last_became_peered":"2024-08-19T14:13:22.727234+0000","last_unstale":"2024-08-19T14:15:07.142621+0000","last_undegraded":"2024-08-19T14:15:07.142621+0000","last_fullsized":"2024-08-19T14:15:07.142621+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:14:38.031056+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:14:38.031056+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:16:03.171231+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"169'10396","reported_seq":16667,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-08-19T14:15:07.950341+0000","last_change":"2024-08-19T14:13:45.917116+0000","last_active":"2024-08-19T14:15:07.950341+0000","last_peered":"2024-08-19T14:15:07.950341+0000","last_clean":"2024-08-19T14:15:07.950341+0000","last_became_active":"2024-08-19T14:13:32.749994+0000","last_became_peered":"2024-08-19T14:13:32.749994+0000","last_unstale":"2024-08-19T14:15:07.950341+0000","last_undegraded":"2024-08-19T14:15:07.950341+0000","last_fullsized":"2024-08-19T14:15:07.950341+0000","mapping_epoch":132,"log_start":"107'300","ondisk_log_start":"107'300","created":102,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"136'5761","last_scrub_stamp":"2024-08-19T14:13:45.916860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:13:45.916860+0000","objects_scrubbed":5716,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:09.768267+0000","scrub_duration":1249,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330432512,"num_objects":5042,"num_object_clones":0,"num_object_copies":15126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5042,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10396,"num_write_kb":494016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3024,"num_bytes_recovered":198180864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,11],"acting":[7,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"70'139","reported_seq":529,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-08-19T14:15:07.144065+0000","last_change":"2024-08-19T14:14:31.408564+0000","last_active":"2024-08-19T14:15:07.144065+0000","last_peered":"2024-08-19T14:15:07.144065+0000","last_clean":"2024-08-19T14:15:07.144065+0000","last_became_active":"2024-08-19T14:13:05.607783+0000","last_became_peered":"2024-08-19T14:13:05.607783+0000","last_unstale":"2024-08-19T14:15:07.144065+0000","last_undegraded":"2024-08-19T14:15:07.144065+0000","last_fullsized":"2024-08-19T14:15:07.144065+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:14:31.408492+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:14:31.408492+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:41.655086+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"168'10409","reported_seq":10551,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-08-19T14:15:06.052480+0000","last_change":"2024-08-19T14:12:43.039198+0000","last_active":"2024-08-19T14:15:06.052480+0000","last_peered":"2024-08-19T14:15:06.052480+0000","last_clean":"2024-08-19T14:15:06.052480+0000","last_became_active":"2024-08-19T14:12:43.038833+0000","last_became_peered":"2024-08-19T14:12:43.038833+0000","last_unstale":"2024-08-19T14:15:06.052480+0000","last_undegraded":"2024-08-19T14:15:06.052480+0000","last_fullsized":"2024-08-19T14:15:06.052480+0000","mapping_epoch":102,"log_start":"107'400","ondisk_log_start":"107'400","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:12:41.713780+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:09.852427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334036992,"num_objects":5097,"num_object_clones":0,"num_object_copies":15291,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5097,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10409,"num_write_kb":496192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"70'139","reported_seq":394,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-08-19T14:15:07.132972+0000","last_change":"2024-08-19T14:14:07.456593+0000","last_active":"2024-08-19T14:15:07.132972+0000","last_peered":"2024-08-19T14:15:07.132972+0000","last_clean":"2024-08-19T14:15:07.132972+0000","last_became_active":"2024-08-19T14:13:11.663185+0000","last_became_peered":"2024-08-19T14:13:11.663185+0000","last_unstale":"2024-08-19T14:15:07.132972+0000","last_undegraded":"2024-08-19T14:15:07.132972+0000","last_fullsized":"2024-08-19T14:15:07.132972+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:14:07.456464+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:14:07.456464+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:09.321073+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2],"acting":[9,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.2","version":"168'10365","reported_seq":18405,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-08-19T14:15:06.455734+0000","last_change":"2024-08-19T14:14:13.196736+0000","last_active":"2024-08-19T14:15:06.455734+0000","last_peered":"2024-08-19T14:15:06.455734+0000","last_clean":"2024-08-19T14:15:06.455734+0000","last_became_active":"2024-08-19T14:13:58.623348+0000","last_became_peered":"2024-08-19T14:13:58.623348+0000","last_unstale":"2024-08-19T14:15:06.455734+0000","last_undegraded":"2024-08-19T14:15:06.455734+0000","last_fullsized":"2024-08-19T14:15:06.455734+0000","mapping_epoch":142,"log_start":"107'300","ondisk_log_start":"107'300","created":102,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"148'7390","last_scrub_stamp":"2024-08-19T14:14:13.196497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:14:13.196497+0000","objects_scrubbed":7346,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:26.733911+0000","scrub_duration":1724,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332333056,"num_objects":5071,"num_object_clones":0,"num_object_copies":15213,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5071,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10365,"num_write_kb":493952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3902,"num_bytes_recovered":255721472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"70'139","reported_seq":379,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-08-19T14:15:07.156545+0000","last_change":"2024-08-19T14:14:48.488485+0000","last_active":"2024-08-19T14:15:07.156545+0000","last_peered":"2024-08-19T14:15:07.156545+0000","last_clean":"2024-08-19T14:15:07.156545+0000","last_became_active":"2024-08-19T14:13:22.739150+0000","last_became_peered":"2024-08-19T14:13:22.739150+0000","last_unstale":"2024-08-19T14:15:07.156545+0000","last_undegraded":"2024-08-19T14:15:07.156545+0000","last_fullsized":"2024-08-19T14:15:07.156545+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:14:48.488367+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:14:48.488367+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:16:00.551219+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"169'10490","reported_seq":16974,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-08-19T14:15:08.034342+0000","last_change":"2024-08-19T14:14:09.998977+0000","last_active":"2024-08-19T14:15:08.034342+0000","last_peered":"2024-08-19T14:15:08.034342+0000","last_clean":"2024-08-19T14:15:08.034342+0000","last_became_active":"2024-08-19T14:14:04.681118+0000","last_became_peered":"2024-08-19T14:14:04.681118+0000","last_unstale":"2024-08-19T14:15:08.034342+0000","last_undegraded":"2024-08-19T14:15:08.034342+0000","last_fullsized":"2024-08-19T14:15:08.034342+0000","mapping_epoch":146,"log_start":"107'400","ondisk_log_start":"107'400","created":102,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'7194","last_scrub_stamp":"2024-08-19T14:14:09.998849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:14:09.998849+0000","objects_scrubbed":7135,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:22.243440+0000","scrub_duration":2842,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":324927488,"num_objects":4958,"num_object_clones":0,"num_object_copies":14874,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4958,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10490,"num_write_kb":494336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3123,"num_bytes_recovered":204668928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"70'139","reported_seq":394,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-08-19T14:15:07.132914+0000","last_change":"2024-08-19T14:14:06.461983+0000","last_active":"2024-08-19T14:15:07.132914+0000","last_peered":"2024-08-19T14:15:07.132914+0000","last_clean":"2024-08-19T14:15:07.132914+0000","last_became_active":"2024-08-19T14:13:11.647085+0000","last_became_peered":"2024-08-19T14:13:11.647085+0000","last_unstale":"2024-08-19T14:15:07.132914+0000","last_undegraded":"2024-08-19T14:15:07.132914+0000","last_fullsized":"2024-08-19T14:15:07.132914+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:14:06.461855+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:14:06.461855+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:12.802907+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,4],"acting":[9,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.5","version":"169'10570","reported_seq":10799,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-08-19T14:15:09.678650+0000","last_change":"2024-08-19T14:14:14.222353+0000","last_active":"2024-08-19T14:15:09.678650+0000","last_peered":"2024-08-19T14:15:09.678650+0000","last_clean":"2024-08-19T14:15:09.678650+0000","last_became_active":"2024-08-19T14:12:43.417616+0000","last_became_peered":"2024-08-19T14:12:43.417616+0000","last_unstale":"2024-08-19T14:15:09.678650+0000","last_undegraded":"2024-08-19T14:15:09.678650+0000","last_fullsized":"2024-08-19T14:15:09.678650+0000","mapping_epoch":102,"log_start":"107'500","ondisk_log_start":"107'500","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"148'7426","last_scrub_stamp":"2024-08-19T14:14:14.222198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:14:14.222198+0000","objects_scrubbed":7364,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:35.064129+0000","scrub_duration":2125,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":319553536,"num_objects":4876,"num_object_clones":0,"num_object_copies":14628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4876,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10570,"num_write_kb":494272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.5","version":"70'139","reported_seq":401,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-08-19T14:15:00.122443+0000","last_change":"2024-08-19T14:14:49.752154+0000","last_active":"2024-08-19T14:15:00.122443+0000","last_peered":"2024-08-19T14:15:00.122443+0000","last_clean":"2024-08-19T14:15:00.122443+0000","last_became_active":"2024-08-19T14:13:22.727098+0000","last_became_peered":"2024-08-19T14:13:22.727098+0000","last_unstale":"2024-08-19T14:15:00.122443+0000","last_undegraded":"2024-08-19T14:15:00.122443+0000","last_fullsized":"2024-08-19T14:15:00.122443+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:14:49.752072+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:14:49.752072+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:16:05.519885+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"169'11621","reported_seq":11911,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-08-19T14:15:07.894145+0000","last_change":"2024-08-19T14:14:40.326683+0000","last_active":"2024-08-19T14:15:07.894145+0000","last_peered":"2024-08-19T14:15:07.894145+0000","last_clean":"2024-08-19T14:15:07.894145+0000","last_became_active":"2024-08-19T14:12:42.963203+0000","last_became_peered":"2024-08-19T14:12:42.963203+0000","last_unstale":"2024-08-19T14:15:07.894145+0000","last_undegraded":"2024-08-19T14:15:07.894145+0000","last_fullsized":"2024-08-19T14:15:07.894145+0000","mapping_epoch":102,"log_start":"107'1600","ondisk_log_start":"107'1600","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'10039","last_scrub_stamp":"2024-08-19T14:14:40.326519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:14:40.326519+0000","objects_scrubbed":6619,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:43.279737+0000","scrub_duration":3261,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":323223552,"num_objects":4932,"num_object_clones":0,"num_object_copies":14796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4932,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10400,"num_write_kb":490624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,3],"acting":[1,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"70'139","reported_seq":451,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-08-19T14:15:00.122064+0000","last_change":"2024-08-19T14:14:36.831290+0000","last_active":"2024-08-19T14:15:00.122064+0000","last_peered":"2024-08-19T14:15:00.122064+0000","last_clean":"2024-08-19T14:15:00.122064+0000","last_became_active":"2024-08-19T14:13:05.607447+0000","last_became_peered":"2024-08-19T14:13:05.607447+0000","last_unstale":"2024-08-19T14:15:00.122064+0000","last_undegraded":"2024-08-19T14:15:00.122064+0000","last_fullsized":"2024-08-19T14:15:00.122064+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:14:36.831171+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:14:36.831171+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:53.437142+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.7","version":"168'10282","reported_seq":21757,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-08-19T14:15:06.036921+0000","last_change":"2024-08-19T14:14:50.008022+0000","last_active":"2024-08-19T14:15:06.036921+0000","last_peered":"2024-08-19T14:15:06.036921+0000","last_clean":"2024-08-19T14:15:06.036921+0000","last_became_active":"2024-08-19T14:14:49.991243+0000","last_became_peered":"2024-08-19T14:14:49.991243+0000","last_unstale":"2024-08-19T14:15:06.036921+0000","last_undegraded":"2024-08-19T14:15:06.036921+0000","last_fullsized":"2024-08-19T14:15:06.036921+0000","mapping_epoch":164,"log_start":"106'200","ondisk_log_start":"106'200","created":102,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:12:41.713780+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:09.852297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":336592896,"num_objects":5136,"num_object_clones":0,"num_object_copies":15408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10282,"num_write_kb":493376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4460,"num_bytes_recovered":253820928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.8","version":"169'10515","reported_seq":10740,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-08-19T14:15:07.920890+0000","last_change":"2024-08-19T14:14:08.982261+0000","last_active":"2024-08-19T14:15:07.920890+0000","last_peered":"2024-08-19T14:15:07.920890+0000","last_clean":"2024-08-19T14:15:07.920890+0000","last_became_active":"2024-08-19T14:12:43.038882+0000","last_became_peered":"2024-08-19T14:12:43.038882+0000","last_unstale":"2024-08-19T14:15:07.920890+0000","last_undegraded":"2024-08-19T14:15:07.920890+0000","last_fullsized":"2024-08-19T14:15:07.920890+0000","mapping_epoch":102,"log_start":"107'500","ondisk_log_start":"107'500","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'7200","last_scrub_stamp":"2024-08-19T14:14:08.982137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:14:08.982137+0000","objects_scrubbed":7142,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:09.792768+0000","scrub_duration":1562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327221248,"num_objects":4993,"num_object_clones":0,"num_object_copies":14979,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4993,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10515,"num_write_kb":496256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,0],"acting":[7,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"70'139","reported_seq":415,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-08-19T14:15:07.132933+0000","last_change":"2024-08-19T14:14:31.607909+0000","last_active":"2024-08-19T14:15:07.132933+0000","last_peered":"2024-08-19T14:15:07.132933+0000","last_clean":"2024-08-19T14:15:07.132933+0000","last_became_active":"2024-08-19T14:13:22.728779+0000","last_became_peered":"2024-08-19T14:13:22.728779+0000","last_unstale":"2024-08-19T14:15:07.132933+0000","last_undegraded":"2024-08-19T14:15:07.132933+0000","last_fullsized":"2024-08-19T14:15:07.132933+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:14:31.607795+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:14:31.607795+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:48.816381+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,0],"acting":[9,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"169'10317","reported_seq":10539,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-08-19T14:15:08.028758+0000","last_change":"2024-08-19T14:14:05.393154+0000","last_active":"2024-08-19T14:15:08.028758+0000","last_peered":"2024-08-19T14:15:08.028758+0000","last_clean":"2024-08-19T14:15:08.028758+0000","last_became_active":"2024-08-19T14:12:42.963114+0000","last_became_peered":"2024-08-19T14:12:42.963114+0000","last_unstale":"2024-08-19T14:15:08.028758+0000","last_undegraded":"2024-08-19T14:15:08.028758+0000","last_fullsized":"2024-08-19T14:15:08.028758+0000","mapping_epoch":102,"log_start":"107'300","ondisk_log_start":"107'300","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'6812","last_scrub_stamp":"2024-08-19T14:14:05.392787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:14:05.392787+0000","objects_scrubbed":6777,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:16.463202+0000","scrub_duration":1333,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320405504,"num_objects":4889,"num_object_clones":0,"num_object_copies":14667,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10317,"num_write_kb":486592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"70'139","reported_seq":528,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-08-19T14:15:07.144000+0000","last_change":"2024-08-19T14:14:35.319452+0000","last_active":"2024-08-19T14:15:07.144000+0000","last_peered":"2024-08-19T14:15:07.144000+0000","last_clean":"2024-08-19T14:15:07.144000+0000","last_became_active":"2024-08-19T14:13:05.607605+0000","last_became_peered":"2024-08-19T14:13:05.607605+0000","last_unstale":"2024-08-19T14:15:07.144000+0000","last_undegraded":"2024-08-19T14:15:07.144000+0000","last_fullsized":"2024-08-19T14:15:07.144000+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:14:35.319333+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:14:35.319333+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:46.300019+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"168'10342","reported_seq":10566,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-08-19T14:15:05.543501+0000","last_change":"2024-08-19T14:14:11.437361+0000","last_active":"2024-08-19T14:15:05.543501+0000","last_peered":"2024-08-19T14:15:05.543501+0000","last_clean":"2024-08-19T14:15:05.543501+0000","last_became_active":"2024-08-19T14:12:42.963040+0000","last_became_peered":"2024-08-19T14:12:42.963040+0000","last_unstale":"2024-08-19T14:15:05.543501+0000","last_undegraded":"2024-08-19T14:15:05.543501+0000","last_fullsized":"2024-08-19T14:15:05.543501+0000","mapping_epoch":102,"log_start":"107'300","ondisk_log_start":"107'300","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'7289","last_scrub_stamp":"2024-08-19T14:14:11.437152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:14:11.437152+0000","objects_scrubbed":7227,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:32.239860+0000","scrub_duration":2564,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":333971456,"num_objects":5096,"num_object_clones":0,"num_object_copies":15288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10342,"num_write_kb":494016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,9],"acting":[11,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.8","version":"70'139","reported_seq":404,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-08-19T14:15:00.113698+0000","last_change":"2024-08-19T14:14:35.594757+0000","last_active":"2024-08-19T14:15:00.113698+0000","last_peered":"2024-08-19T14:15:00.113698+0000","last_clean":"2024-08-19T14:15:00.113698+0000","last_became_active":"2024-08-19T14:13:05.605694+0000","last_became_peered":"2024-08-19T14:13:05.605694+0000","last_unstale":"2024-08-19T14:15:00.113698+0000","last_undegraded":"2024-08-19T14:15:00.113698+0000","last_fullsized":"2024-08-19T14:15:00.113698+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:14:35.594637+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:14:35.594637+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:43.962595+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.b","version":"169'10589","reported_seq":10812,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-08-19T14:15:09.676643+0000","last_change":"2024-08-19T14:14:06.266442+0000","last_active":"2024-08-19T14:15:09.676643+0000","last_peered":"2024-08-19T14:15:09.676643+0000","last_clean":"2024-08-19T14:15:09.676643+0000","last_became_active":"2024-08-19T14:12:42.729014+0000","last_became_peered":"2024-08-19T14:12:42.729014+0000","last_unstale":"2024-08-19T14:15:09.676643+0000","last_undegraded":"2024-08-19T14:15:09.676643+0000","last_fullsized":"2024-08-19T14:15:09.676643+0000","mapping_epoch":102,"log_start":"107'500","ondisk_log_start":"107'500","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'6965","last_scrub_stamp":"2024-08-19T14:14:06.266214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:14:06.266214+0000","objects_scrubbed":6919,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:18.095310+0000","scrub_duration":1296,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":313982976,"num_objects":4791,"num_object_clones":0,"num_object_copies":14373,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4791,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10589,"num_write_kb":492160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":5229379608,"num_objects":79795,"num_object_clones":0,"num_object_copies":239385,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79795,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":166763,"num_write_kb":7889793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14509,"num_bytes_recovered":912392192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16204640256,"data_stored":16204628040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160784,"ondisk_log_size":160784,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1529,"ondisk_log_size":1529,"up":33,"acting":33,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187501,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1387288,"kb_used_data":1320016,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":92357864,"statfs":{"total":95995035648,"available":94574452736,"internally_reserved":0,"allocated":1351696384,"data_stored":1350852498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":68876740},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":343597383730,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":74,"seq":317827579959,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2365932,"kb_used_data":2248672,"kb_used_omap":1,"kb_used_meta":117246,"kb_avail":91379220,"statfs":{"total":95995035648,"available":93572321280,"internally_reserved":0,"allocated":2302640128,"data_stored":2301787834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":120060353},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874301,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1326180,"kb_used_data":1254748,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":92418972,"statfs":{"total":95995035648,"available":94637027328,"internally_reserved":0,"allocated":1284861952,"data_stored":1284013730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":57,"seq":244813135935,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1689448,"kb_used_data":1591644,"kb_used_omap":1,"kb_used_meta":97790,"kb_avail":92055704,"statfs":{"total":95995035648,"available":94265040896,"internally_reserved":0,"allocated":1629843456,"data_stored":1628995234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":100137418},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":206158430275,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691911,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":696544,"kb_used_data":651480,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93048608,"statfs":{"total":95995035648,"available":95281774592,"internally_reserved":0,"allocated":667115520,"data_stored":666268082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986252,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1370988,"kb_used_data":1285664,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":92374164,"statfs":{"total":95995035648,"available":94591143936,"internally_reserved":0,"allocated":1316519936,"data_stored":1315667642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149778,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2334516,"kb_used_data":2204712,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":91410636,"statfs":{"total":95995035648,"available":93604491264,"internally_reserved":0,"allocated":2257625088,"data_stored":2256769242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":132839879},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411414,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1732064,"kb_used_data":1628696,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":92013088,"statfs":{"total":95995035648,"available":94221402112,"internally_reserved":0,"allocated":1667784704,"data_stored":1666937266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574939,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1689192,"kb_used_data":1599644,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":92055960,"statfs":{"total":95995035648,"available":94265303040,"internally_reserved":0,"allocated":1638035456,"data_stored":1637187234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91617733},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738463,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1736984,"kb_used_data":1643280,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":92008168,"statfs":{"total":95995035648,"available":94216364032,"internally_reserved":0,"allocated":1682718720,"data_stored":1681874834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":95877573},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1677262848,"data_stored":1677262848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1619722240,"data_stored":1619722240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1660813312,"data_stored":1660813312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2247692288,"data_stored":2247688216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1309020160,"data_stored":1309016088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":657063936,"data_stored":657063936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":328531968,"data_stored":328531968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1618608128,"data_stored":1618608128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1270939648,"data_stored":1270939648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":2277445632,"data_stored":2277441560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":198115328,"data_stored":198115328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1339424768,"data_stored":1339424768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T14:15:11.523 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-08-19T14:15:11.524 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-19T14:15:11.524 INFO:tasks.thrashosds.thrasher:Reweighting osd 3 to 0.6437291956265372 2024-08-19T14:15:11.524 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd reweight 3 0.6437291956265372 2024-08-19T14:15:11.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:11 smithi045 ceph-mon[20143]: pgmap v427: 27 pgs: 27 active+clean; 4.9 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 990 op/s 2024-08-19T14:15:11.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:11 smithi045 ceph-mon[20143]: from='client.24854 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:15:11.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:11 smithi045 ceph-mon[20143]: 2.0 scrub starts 2024-08-19T14:15:11.561 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:15:11.589 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:11.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:11.590 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:15:11.691 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:15:11.722 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:11.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:11.724 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:15:11.728 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:15:11.728 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:15:11.798 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:15:11.799 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:15:11.824 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:15:11.832 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:15:11.832 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:15:11.836 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:15:11.863 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:15:11.864 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:15:11.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:11.867 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:11.867 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:15:11.897 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:15:11.897 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:15:11.930 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:15:11.931 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:15:11.967 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:15:11.967 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:15:11.968 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:15:12.001 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:15:12.002 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:15:12.015 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:12.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:12.016 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:15:12.036 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:15:12.036 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:15:12.068 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:15:12.069 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:15:12.104 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:15:12.104 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:15:12.116 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:15:12.140 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:15:12.140 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:15:12.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:12.143 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:12.143 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:15:12.177 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:15:12.178 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:15:12.244 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:15:12.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:12 smithi032 ceph-mon[20004]: 2.0 scrub ok 2024-08-19T14:15:12.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:12 smithi032 ceph-mon[20004]: 2.8 scrub starts 2024-08-19T14:15:12.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:12 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/243036634' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:15:12.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:12 smithi032 ceph-mon[20004]: 2.7 scrub starts 2024-08-19T14:15:12.273 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:12.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:12.274 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:15:12.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:12 smithi031 ceph-mon[16336]: 2.0 scrub ok 2024-08-19T14:15:12.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:12 smithi031 ceph-mon[16336]: 2.8 scrub starts 2024-08-19T14:15:12.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:12 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/243036634' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:15:12.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:12 smithi031 ceph-mon[16336]: 2.7 scrub starts 2024-08-19T14:15:12.375 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:15:12.399 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:12.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:12.400 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:15:12.501 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:15:12.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:12.528 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:12.528 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:15:12.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:12 smithi045 ceph-mon[20143]: 2.0 scrub ok 2024-08-19T14:15:12.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:12 smithi045 ceph-mon[20143]: 2.8 scrub starts 2024-08-19T14:15:12.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:12 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/243036634' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:15:12.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:12 smithi045 ceph-mon[20143]: 2.7 scrub starts 2024-08-19T14:15:12.629 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:15:12.630 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:15:12.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:12.704 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:12.704 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:15:12.805 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:15:12.829 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:12.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:12.831 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:15:12.932 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:15:12.953 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:12.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:12.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:15:13.055 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:15:13.080 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:13.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:13.081 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:15:13.182 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:15:13.204 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:13.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:13.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:15:13.306 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:15:13.343 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:13.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:13.344 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:15:13.445 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:15:13.469 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:13.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:13.470 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:15:13.572 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:15:13.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:13.606 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:13.606 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:15:13.639 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is unset 2024-08-19T14:15:13.707 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:15:13.731 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:13.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:13.732 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:15:13.833 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:15:13.857 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:13.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:13.858 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:15:13.906 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:13 smithi031 ceph-mon[16336]: pgmap v428: 27 pgs: 27 active+clean; 4.7 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 952 op/s 2024-08-19T14:15:13.906 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:13 smithi031 ceph-mon[16336]: 2.e scrub starts 2024-08-19T14:15:13.906 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:13 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1677305877' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:15:13.906 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:13 smithi031 ceph-mon[16336]: 2.8 scrub ok 2024-08-19T14:15:13.906 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:13 smithi031 ceph-mon[16336]: 1.6 scrub starts 2024-08-19T14:15:13.906 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:13 smithi031 ceph-mon[16336]: 1.6 scrub ok 2024-08-19T14:15:13.909 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:13 smithi032 ceph-mon[20004]: pgmap v428: 27 pgs: 27 active+clean; 4.7 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 952 op/s 2024-08-19T14:15:13.909 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:13 smithi032 ceph-mon[20004]: 2.e scrub starts 2024-08-19T14:15:13.909 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:13 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1677305877' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:15:13.910 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:13 smithi032 ceph-mon[20004]: 2.8 scrub ok 2024-08-19T14:15:13.910 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:13 smithi032 ceph-mon[20004]: 1.6 scrub starts 2024-08-19T14:15:13.910 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:13 smithi032 ceph-mon[20004]: 1.6 scrub ok 2024-08-19T14:15:13.942 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-08-19T14:15:13.958 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:15:14.013 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:14.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:14.015 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:15:14.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:13 smithi045 ceph-mon[20143]: pgmap v428: 27 pgs: 27 active+clean; 4.7 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 952 op/s 2024-08-19T14:15:14.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:13 smithi045 ceph-mon[20143]: 2.e scrub starts 2024-08-19T14:15:14.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:13 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1677305877' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:15:14.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:13 smithi045 ceph-mon[20143]: 2.8 scrub ok 2024-08-19T14:15:14.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:13 smithi045 ceph-mon[20143]: 1.6 scrub starts 2024-08-19T14:15:14.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:13 smithi045 ceph-mon[20143]: 1.6 scrub ok 2024-08-19T14:15:14.116 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:15:14.143 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:14.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:14.144 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:15:14.245 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:15:14.262 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:15:14.271 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:14.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:14.272 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:15:14.373 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:15:14.416 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:14.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:14.417 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:15:14.518 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:15:14.544 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:14.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:14.545 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:15:14.646 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:15:14.649 INFO:teuthology.orchestra.run.smithi031.stderr:reweighted osd.3 to 0.643729 (a4cb) 2024-08-19T14:15:14.677 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:14.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:14.678 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:15:14.779 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:15:14.797 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:15:14.797 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-08-19T14:15:14.797 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-08-19T14:15:14.797 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-08-19T14:15:14.797 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:15:14.801 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:14.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:14.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:15:14.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:14 smithi031 ceph-mon[16336]: 2.7 scrub ok 2024-08-19T14:15:14.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:14 smithi031 ceph-mon[16336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:15:14.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:14 smithi031 ceph-mon[16336]: Cluster is now healthy 2024-08-19T14:15:14.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:14 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1677305877' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:15:14.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:14 smithi031 ceph-mon[16336]: osdmap e170: 12 total, 12 up, 10 in 2024-08-19T14:15:14.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:14 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1355411263' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 0.6437291956265372}]: dispatch 2024-08-19T14:15:14.902 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:15:14.923 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:14.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:14.924 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:15:15.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:14 smithi032 ceph-mon[20004]: 2.7 scrub ok 2024-08-19T14:15:15.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:14 smithi032 ceph-mon[20004]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:15:15.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:14 smithi032 ceph-mon[20004]: Cluster is now healthy 2024-08-19T14:15:15.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:14 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1677305877' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:15:15.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:14 smithi032 ceph-mon[20004]: osdmap e170: 12 total, 12 up, 10 in 2024-08-19T14:15:15.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:14 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1355411263' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 0.6437291956265372}]: dispatch 2024-08-19T14:15:15.025 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:15:15.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:14 smithi045 ceph-mon[20143]: 2.7 scrub ok 2024-08-19T14:15:15.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:14 smithi045 ceph-mon[20143]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:15:15.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:14 smithi045 ceph-mon[20143]: Cluster is now healthy 2024-08-19T14:15:15.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:14 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1677305877' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:15:15.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:14 smithi045 ceph-mon[20143]: osdmap e170: 12 total, 12 up, 10 in 2024-08-19T14:15:15.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:14 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1355411263' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 0.6437291956265372}]: dispatch 2024-08-19T14:15:15.047 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:15.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:15.048 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:15:15.149 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:15:15.170 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:15.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:15.171 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:15:15.272 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:15:15.293 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:15.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:15.294 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:15:15.352 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-08-19T14:15:15.394 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:15:15.417 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:15.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:15.418 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:15:15.518 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:15:15.544 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:15.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:15.545 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:15:15.605 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:15:15.646 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:15:15.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:15.669 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:15.669 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:15:15.770 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:15:15.791 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:15.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:15.792 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:15:15.893 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:15:15.915 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:15.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:15.917 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:15:15.953 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:15:15.979 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:15 smithi031 ceph-mon[16336]: pgmap v430: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 4.6 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 983 op/s 2024-08-19T14:15:15.979 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:15 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1355411263' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 0.6437291956265372}]': finished 2024-08-19T14:15:15.979 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:15 smithi031 ceph-mon[16336]: osdmap e171: 12 total, 12 up, 10 in 2024-08-19T14:15:15.979 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:15 smithi031 ceph-mon[16336]: osdmap e172: 12 total, 12 up, 10 in 2024-08-19T14:15:15.994 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:15 smithi032 ceph-mon[20004]: pgmap v430: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 4.6 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 983 op/s 2024-08-19T14:15:15.995 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:15 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1355411263' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 0.6437291956265372}]': finished 2024-08-19T14:15:15.995 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:15 smithi032 ceph-mon[20004]: osdmap e171: 12 total, 12 up, 10 in 2024-08-19T14:15:15.995 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:15 smithi032 ceph-mon[20004]: osdmap e172: 12 total, 12 up, 10 in 2024-08-19T14:15:16.017 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:15:16.038 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:16.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:16.039 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:15:16.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:15 smithi045 ceph-mon[20143]: pgmap v430: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 4.6 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 983 op/s 2024-08-19T14:15:16.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:15 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1355411263' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 0.6437291956265372}]': finished 2024-08-19T14:15:16.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:15 smithi045 ceph-mon[20143]: osdmap e171: 12 total, 12 up, 10 in 2024-08-19T14:15:16.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:15 smithi045 ceph-mon[20143]: osdmap e172: 12 total, 12 up, 10 in 2024-08-19T14:15:16.140 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:15:16.161 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:16.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:16.162 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:15:16.204 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:15:16.204 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-08-19T14:15:16.204 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-08-19T14:15:16.204 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-08-19T14:15:16.204 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.204 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24580.0:62590 2.d e125/102 2:b1cd6c4c:::benchmark_data_smithi045_46415_object62589:head v 125'3988, mlcod=125'3985)", 2024-08-19T14:15:16.204 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:18.389311+0000", 2024-08-19T14:15:16.205 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 117.706019785, 2024-08-19T14:15:16.205 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.025501861000000001, 2024-08-19T14:15:16.205 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:15:16.205 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:15:16.205 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:15:16.205 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:15:16.205 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.205 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:15:16.205 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.389311+0000", 2024-08-19T14:15:16.205 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:16.205 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.205 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.205 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:15:16.205 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.389311+0000", 2024-08-19T14:15:16.206 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:16.206 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.206 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.206 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:15:16.206 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.389311+0000", 2024-08-19T14:15:16.206 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8899999999999999e-07 2024-08-19T14:15:16.206 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.206 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.206 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:15:16.206 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.389426+0000", 2024-08-19T14:15:16.206 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000115003 2024-08-19T14:15:16.206 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.206 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.207 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:15:16.207 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.389437+0000", 2024-08-19T14:15:16.207 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0865e-05 2024-08-19T14:15:16.207 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.207 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.207 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:15:16.207 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.389444+0000", 2024-08-19T14:15:16.207 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2189999999999998e-06 2024-08-19T14:15:16.207 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.207 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.207 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:15:16.207 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.389490+0000", 2024-08-19T14:15:16.207 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5357e-05 2024-08-19T14:15:16.208 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.208 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.208 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:15:16.208 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.389504+0000", 2024-08-19T14:15:16.208 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4119e-05 2024-08-19T14:15:16.208 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.208 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.208 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:15:16.208 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.391313+0000", 2024-08-19T14:15:16.208 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0018095380000000001 2024-08-19T14:15:16.208 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.208 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.208 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:15:16.208 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.414812+0000", 2024-08-19T14:15:16.209 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.023499170999999999 2024-08-19T14:15:16.209 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:16.209 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:15:16.209 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:16.209 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.209 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.209 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24580.0:62594 2.5 e125/102 2:a15e85f0:::benchmark_data_smithi045_46415_object62593:head v 125'3968, mlcod=125'3966)", 2024-08-19T14:15:16.209 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:18.389960+0000", 2024-08-19T14:15:16.209 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 117.705370614, 2024-08-19T14:15:16.209 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.027685131000000002, 2024-08-19T14:15:16.209 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:15:16.209 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:15:16.209 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:15:16.209 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:15:16.210 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.210 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:15:16.210 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.389960+0000", 2024-08-19T14:15:16.210 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:16.210 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.210 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.210 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:15:16.210 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.389960+0000", 2024-08-19T14:15:16.210 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:16.210 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.210 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.210 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:15:16.211 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.389960+0000", 2024-08-19T14:15:16.211 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8100000000000002e-07 2024-08-19T14:15:16.211 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.211 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.211 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:15:16.211 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.390013+0000", 2024-08-19T14:15:16.211 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2772000000000002e-05 2024-08-19T14:15:16.211 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.211 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.211 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:15:16.211 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.390027+0000", 2024-08-19T14:15:16.211 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3625000000000001e-05 2024-08-19T14:15:16.212 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.212 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.212 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:15:16.212 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.390033+0000", 2024-08-19T14:15:16.212 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3849999999999999e-06 2024-08-19T14:15:16.212 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.212 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.212 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:15:16.212 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.390089+0000", 2024-08-19T14:15:16.212 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5998000000000002e-05 2024-08-19T14:15:16.212 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.212 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.213 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:15:16.213 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.390109+0000", 2024-08-19T14:15:16.213 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9539e-05 2024-08-19T14:15:16.213 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.213 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.213 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:15:16.213 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.417596+0000", 2024-08-19T14:15:16.213 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.027487396000000001 2024-08-19T14:15:16.213 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.213 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.213 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:15:16.213 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.417645+0000", 2024-08-19T14:15:16.213 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8735e-05 2024-08-19T14:15:16.213 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:16.214 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:15:16.214 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:16.214 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.214 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.214 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24580.0:63067 2.7 e128/127 2:e2705906:::benchmark_data_smithi045_46415_object63066:head v 128'3850, mlcod=0'0)", 2024-08-19T14:15:16.214 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:20.938200+0000", 2024-08-19T14:15:16.214 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 115.15712993299999, 2024-08-19T14:15:16.214 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.021572997999999999, 2024-08-19T14:15:16.214 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:15:16.214 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:15:16.215 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:15:16.215 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:15:16.215 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.215 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:15:16.215 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.938200+0000", 2024-08-19T14:15:16.215 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:16.215 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.215 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.215 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:15:16.215 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.938200+0000", 2024-08-19T14:15:16.215 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:16.215 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.215 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.216 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:15:16.216 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.938203+0000", 2024-08-19T14:15:16.216 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.638e-06 2024-08-19T14:15:16.216 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.216 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.216 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:15:16.216 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.938268+0000", 2024-08-19T14:15:16.216 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4980000000000005e-05 2024-08-19T14:15:16.216 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.216 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.216 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:15:16.216 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.938279+0000", 2024-08-19T14:15:16.216 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0658e-05 2024-08-19T14:15:16.216 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.216 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.217 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:15:16.217 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.938285+0000", 2024-08-19T14:15:16.217 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9410000000000004e-06 2024-08-19T14:15:16.217 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.217 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.217 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:15:16.217 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.940615+0000", 2024-08-19T14:15:16.217 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0023300819999999998 2024-08-19T14:15:16.217 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.217 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.217 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:15:16.217 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.940638+0000", 2024-08-19T14:15:16.217 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3691999999999999e-05 2024-08-19T14:15:16.217 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.218 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.218 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:15:16.218 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.959684+0000", 2024-08-19T14:15:16.218 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.019045973000000001 2024-08-19T14:15:16.218 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.218 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.218 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:15:16.218 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.959773+0000", 2024-08-19T14:15:16.218 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.9034000000000002e-05 2024-08-19T14:15:16.218 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:16.218 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:15:16.218 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:16.218 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.219 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.219 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24580.0:63086 2.7 e128/127 2:e280ae82:::benchmark_data_smithi045_46415_object63085:head v 128'3851, mlcod=0'0)", 2024-08-19T14:15:16.219 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:20.940402+0000", 2024-08-19T14:15:16.219 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 115.154928516, 2024-08-19T14:15:16.219 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.022477548999999999, 2024-08-19T14:15:16.219 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:15:16.219 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:15:16.219 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:15:16.219 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:15:16.219 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.219 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:15:16.219 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.940402+0000", 2024-08-19T14:15:16.220 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:16.220 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.220 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.220 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:15:16.220 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.940402+0000", 2024-08-19T14:15:16.220 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:16.220 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.220 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.220 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:15:16.220 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.940402+0000", 2024-08-19T14:15:16.220 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0200000000000002e-07 2024-08-19T14:15:16.221 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.221 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.221 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:15:16.221 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.940470+0000", 2024-08-19T14:15:16.221 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7830999999999995e-05 2024-08-19T14:15:16.221 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.221 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.221 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:15:16.221 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.940479+0000", 2024-08-19T14:15:16.221 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.4100000000000008e-06 2024-08-19T14:15:16.221 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.221 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.222 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:15:16.222 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.940484+0000", 2024-08-19T14:15:16.222 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8679999999999998e-06 2024-08-19T14:15:16.222 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.222 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.222 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:15:16.222 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.944331+0000", 2024-08-19T14:15:16.222 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0038478200000000001 2024-08-19T14:15:16.222 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.222 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.222 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:15:16.222 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.944357+0000", 2024-08-19T14:15:16.222 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5836000000000001e-05 2024-08-19T14:15:16.223 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.223 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.223 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:15:16.223 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.962818+0000", 2024-08-19T14:15:16.223 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.018460977 2024-08-19T14:15:16.223 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.223 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.223 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:15:16.223 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.962879+0000", 2024-08-19T14:15:16.223 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1304999999999997e-05 2024-08-19T14:15:16.223 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:16.223 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:15:16.223 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:16.224 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.224 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.224 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24580.0:63687 2.7 e128/127 2:ea76d4f0:::benchmark_data_smithi045_46415_object63686:head v 128'3888, mlcod=0'0)", 2024-08-19T14:15:16.224 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:21.486322+0000", 2024-08-19T14:15:16.224 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 114.60900819299999, 2024-08-19T14:15:16.224 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.018889623000000001, 2024-08-19T14:15:16.224 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:15:16.224 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:15:16.224 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:15:16.224 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:15:16.224 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.224 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:15:16.224 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.486322+0000", 2024-08-19T14:15:16.224 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:16.225 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.225 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.225 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:15:16.225 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.486322+0000", 2024-08-19T14:15:16.225 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:16.225 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.225 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.225 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:15:16.225 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.486323+0000", 2024-08-19T14:15:16.225 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3799999999999996e-07 2024-08-19T14:15:16.225 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.225 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.225 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:15:16.225 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.486367+0000", 2024-08-19T14:15:16.226 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3430000000000003e-05 2024-08-19T14:15:16.226 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.226 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.226 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:15:16.226 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.486381+0000", 2024-08-19T14:15:16.226 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4579000000000001e-05 2024-08-19T14:15:16.226 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.226 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.226 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:15:16.226 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.486389+0000", 2024-08-19T14:15:16.226 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6899999999999992e-06 2024-08-19T14:15:16.226 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.227 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.227 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:15:16.227 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.486444+0000", 2024-08-19T14:15:16.227 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5359999999999999e-05 2024-08-19T14:15:16.227 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.227 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.227 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:15:16.227 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.486472+0000", 2024-08-19T14:15:16.227 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8019999999999999e-05 2024-08-19T14:15:16.227 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.227 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.227 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:15:16.227 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.505168+0000", 2024-08-19T14:15:16.228 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.01869618 2024-08-19T14:15:16.228 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.228 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.228 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:15:16.228 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.505212+0000", 2024-08-19T14:15:16.228 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3426000000000001e-05 2024-08-19T14:15:16.228 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:16.228 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:15:16.228 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:16.228 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.228 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.228 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24580.0:64943 2.f e129/102 2:f4917538:::benchmark_data_smithi045_46415_object64942:head v 129'4101, mlcod=129'4099)", 2024-08-19T14:15:16.229 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:22.483598+0000", 2024-08-19T14:15:16.229 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 113.61173218099999, 2024-08-19T14:15:16.229 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.02030792, 2024-08-19T14:15:16.229 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:15:16.229 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:15:16.229 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:15:16.229 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:15:16.229 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.229 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:15:16.229 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.483598+0000", 2024-08-19T14:15:16.229 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:16.229 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.230 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.230 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:15:16.230 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.483598+0000", 2024-08-19T14:15:16.230 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:16.230 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.230 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.230 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:15:16.230 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.483599+0000", 2024-08-19T14:15:16.230 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0380000000000001e-06 2024-08-19T14:15:16.230 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.230 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.230 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:15:16.230 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.483667+0000", 2024-08-19T14:15:16.231 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7997999999999995e-05 2024-08-19T14:15:16.231 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.231 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.231 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:15:16.231 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.483682+0000", 2024-08-19T14:15:16.231 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.45e-05 2024-08-19T14:15:16.231 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.231 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.231 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:15:16.231 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.483688+0000", 2024-08-19T14:15:16.231 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6680000000000004e-06 2024-08-19T14:15:16.231 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.231 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.231 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:15:16.232 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.486328+0000", 2024-08-19T14:15:16.232 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0026395440000000002 2024-08-19T14:15:16.232 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.232 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.232 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:15:16.232 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.486351+0000", 2024-08-19T14:15:16.232 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2751e-05 2024-08-19T14:15:16.232 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.232 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.232 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:15:16.232 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.503857+0000", 2024-08-19T14:15:16.232 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.017505850999999999 2024-08-19T14:15:16.233 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.233 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.233 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:15:16.233 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.503906+0000", 2024-08-19T14:15:16.233 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9570000000000001e-05 2024-08-19T14:15:16.233 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:16.233 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:15:16.233 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:16.233 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.233 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.233 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24580.0:64926 2.7 e129/127 2:ee7d394c:::benchmark_data_smithi045_46415_object64925:head v 129'3976, mlcod=0'0)", 2024-08-19T14:15:16.233 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:22.485386+0000", 2024-08-19T14:15:16.233 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 113.609944375, 2024-08-19T14:15:16.234 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.022392815999999999, 2024-08-19T14:15:16.234 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:15:16.234 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:15:16.234 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:15:16.234 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:15:16.234 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.234 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:15:16.234 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.485386+0000", 2024-08-19T14:15:16.234 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:16.234 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.234 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.234 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:15:16.234 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.485386+0000", 2024-08-19T14:15:16.235 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:16.235 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.235 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.235 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:15:16.235 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.485387+0000", 2024-08-19T14:15:16.235 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.9699999999999994e-07 2024-08-19T14:15:16.235 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.235 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.235 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:15:16.235 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.485434+0000", 2024-08-19T14:15:16.235 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7113000000000001e-05 2024-08-19T14:15:16.235 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.235 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.236 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:15:16.236 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.485447+0000", 2024-08-19T14:15:16.236 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2695e-05 2024-08-19T14:15:16.236 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.236 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.236 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:15:16.236 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.485455+0000", 2024-08-19T14:15:16.236 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.9799999999999998e-06 2024-08-19T14:15:16.236 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.236 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.236 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:15:16.236 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.487352+0000", 2024-08-19T14:15:16.236 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001896809 2024-08-19T14:15:16.237 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.237 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.237 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:15:16.237 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.487379+0000", 2024-08-19T14:15:16.237 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7248999999999999e-05 2024-08-19T14:15:16.237 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.237 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.237 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:15:16.237 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.507725+0000", 2024-08-19T14:15:16.237 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.020345797999999998 2024-08-19T14:15:16.237 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.237 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.237 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:15:16.237 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.507779+0000", 2024-08-19T14:15:16.238 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4175000000000002e-05 2024-08-19T14:15:16.238 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:16.238 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:15:16.238 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:16.238 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.238 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.238 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24580.0:64930 2.7 e129/127 2:e3128c9d:::benchmark_data_smithi045_46415_object64929:head v 129'3977, mlcod=0'0)", 2024-08-19T14:15:16.238 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:22.488188+0000", 2024-08-19T14:15:16.238 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 113.607142289, 2024-08-19T14:15:16.238 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.019626971999999999, 2024-08-19T14:15:16.238 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:15:16.238 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:15:16.238 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:15:16.238 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:15:16.239 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.239 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:15:16.239 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.488188+0000", 2024-08-19T14:15:16.239 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:16.239 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.239 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.239 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:15:16.239 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.488188+0000", 2024-08-19T14:15:16.239 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:16.239 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.239 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.239 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:15:16.239 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.488189+0000", 2024-08-19T14:15:16.240 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6099999999999999e-07 2024-08-19T14:15:16.240 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.240 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.240 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:15:16.240 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.488230+0000", 2024-08-19T14:15:16.240 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1448e-05 2024-08-19T14:15:16.240 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.240 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.240 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:15:16.240 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.488242+0000", 2024-08-19T14:15:16.240 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1402e-05 2024-08-19T14:15:16.240 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.240 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.240 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:15:16.241 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.488249+0000", 2024-08-19T14:15:16.241 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7190000000000004e-06 2024-08-19T14:15:16.241 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.241 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.241 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:15:16.241 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.491525+0000", 2024-08-19T14:15:16.241 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0032765350000000001 2024-08-19T14:15:16.241 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.241 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.241 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:15:16.241 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.491554+0000", 2024-08-19T14:15:16.242 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9022000000000001e-05 2024-08-19T14:15:16.242 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.242 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.242 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:15:16.242 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.507783+0000", 2024-08-19T14:15:16.242 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.016228564000000001 2024-08-19T14:15:16.242 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.242 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.242 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:15:16.242 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.507815+0000", 2024-08-19T14:15:16.242 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2421e-05 2024-08-19T14:15:16.242 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:16.242 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:15:16.243 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:16.243 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.243 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.243 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24580.0:64948 2.f e129/102 2:fe97b461:::benchmark_data_smithi045_46415_object64947:head v 129'4102, mlcod=129'4099)", 2024-08-19T14:15:16.243 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:22.489275+0000", 2024-08-19T14:15:16.243 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 113.60605526800001, 2024-08-19T14:15:16.243 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.018585981000000001, 2024-08-19T14:15:16.243 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:15:16.243 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:15:16.243 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:15:16.243 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:15:16.243 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.243 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:15:16.244 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.489275+0000", 2024-08-19T14:15:16.244 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:16.244 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.244 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.244 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:15:16.244 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.489275+0000", 2024-08-19T14:15:16.244 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:16.244 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.244 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.244 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:15:16.244 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.489276+0000", 2024-08-19T14:15:16.244 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.4200000000000004e-07 2024-08-19T14:15:16.244 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.244 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.245 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:15:16.245 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.489433+0000", 2024-08-19T14:15:16.245 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00015674299999999999 2024-08-19T14:15:16.245 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.245 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.245 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:15:16.245 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.489446+0000", 2024-08-19T14:15:16.245 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2738999999999999e-05 2024-08-19T14:15:16.245 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.245 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.245 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:15:16.245 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.489455+0000", 2024-08-19T14:15:16.245 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.1030000000000008e-06 2024-08-19T14:15:16.246 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.246 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.246 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:15:16.246 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.496819+0000", 2024-08-19T14:15:16.246 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0073646689999999999 2024-08-19T14:15:16.246 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.246 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.246 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:15:16.246 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.496838+0000", 2024-08-19T14:15:16.246 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8564999999999999e-05 2024-08-19T14:15:16.246 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.246 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.246 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:15:16.247 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.507820+0000", 2024-08-19T14:15:16.247 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010982122 2024-08-19T14:15:16.247 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.247 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.247 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:15:16.247 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.507861+0000", 2024-08-19T14:15:16.247 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1097999999999999e-05 2024-08-19T14:15:16.247 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:16.247 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:15:16.247 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:16.247 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.247 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.247 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24580.0:70370 2.f e130/102 2:f673ee5b:::benchmark_data_smithi045_46415_object70369:head v 130'4431, mlcod=130'4429)", 2024-08-19T14:15:16.247 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:26.809945+0000", 2024-08-19T14:15:16.248 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 109.285385108, 2024-08-19T14:15:16.248 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.018445109000000001, 2024-08-19T14:15:16.248 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:15:16.248 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:15:16.248 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:15:16.248 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:15:16.248 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.248 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:15:16.248 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:26.809945+0000", 2024-08-19T14:15:16.248 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:16.248 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.248 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.248 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:15:16.248 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:26.809945+0000", 2024-08-19T14:15:16.249 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:16.249 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.249 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.249 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:15:16.249 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:26.809946+0000", 2024-08-19T14:15:16.249 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1000000000000001e-06 2024-08-19T14:15:16.249 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.249 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.249 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:15:16.249 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:26.810004+0000", 2024-08-19T14:15:16.249 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8019999999999997e-05 2024-08-19T14:15:16.249 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.249 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.249 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:15:16.250 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:26.810022+0000", 2024-08-19T14:15:16.250 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7130999999999999e-05 2024-08-19T14:15:16.250 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.250 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.250 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:15:16.250 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:26.810029+0000", 2024-08-19T14:15:16.250 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1779999999999997e-06 2024-08-19T14:15:16.250 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.250 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.250 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:15:16.250 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:26.817198+0000", 2024-08-19T14:15:16.250 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0071696609999999999 2024-08-19T14:15:16.250 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.251 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.251 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:15:16.251 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:26.817222+0000", 2024-08-19T14:15:16.251 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.338e-05 2024-08-19T14:15:16.251 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.251 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.251 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:15:16.251 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:26.828272+0000", 2024-08-19T14:15:16.251 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011050557000000001 2024-08-19T14:15:16.251 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.251 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.251 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:15:16.251 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:26.828390+0000", 2024-08-19T14:15:16.252 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000118082 2024-08-19T14:15:16.252 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:16.252 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:15:16.252 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:16.252 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.252 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.252 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24580.0:70754 2.d e130/102 2:b08378d7:::benchmark_data_smithi045_46415_object70753:head v 130'4497, mlcod=130'4493)", 2024-08-19T14:15:16.252 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:27.103284+0000", 2024-08-19T14:15:16.252 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 108.99204666200001, 2024-08-19T14:15:16.253 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.023299915000000001, 2024-08-19T14:15:16.253 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:15:16.253 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:15:16.253 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:15:16.253 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:15:16.253 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.253 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:15:16.253 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:27.103284+0000", 2024-08-19T14:15:16.253 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:16.253 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.253 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.253 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:15:16.253 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:27.103284+0000", 2024-08-19T14:15:16.254 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:16.254 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.254 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.254 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:15:16.254 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:27.103284+0000", 2024-08-19T14:15:16.254 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8000000000000001e-07 2024-08-19T14:15:16.254 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.254 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.254 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:15:16.254 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:27.122506+0000", 2024-08-19T14:15:16.254 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.019221723 2024-08-19T14:15:16.254 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.255 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.255 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:15:16.255 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:27.122521+0000", 2024-08-19T14:15:16.255 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5235000000000001e-05 2024-08-19T14:15:16.255 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.255 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.255 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:15:16.255 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:27.122529+0000", 2024-08-19T14:15:16.255 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5909999999999998e-06 2024-08-19T14:15:16.255 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.255 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.255 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:15:16.255 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:27.122589+0000", 2024-08-19T14:15:16.256 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0098e-05 2024-08-19T14:15:16.256 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.256 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.256 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:15:16.256 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:27.122608+0000", 2024-08-19T14:15:16.256 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9678999999999999e-05 2024-08-19T14:15:16.256 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.256 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.256 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:15:16.256 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:27.126540+0000", 2024-08-19T14:15:16.256 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0039318460000000001 2024-08-19T14:15:16.256 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.257 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.257 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:15:16.257 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:27.126584+0000", 2024-08-19T14:15:16.257 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3362999999999998e-05 2024-08-19T14:15:16.257 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:16.257 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:15:16.257 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:16.257 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.257 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.257 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24580.0:71004 2.f e130/102 2:f1f5b3f0:::benchmark_data_smithi045_46415_object71003:head v 130'4478, mlcod=130'4476)", 2024-08-19T14:15:16.257 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:27.290131+0000", 2024-08-19T14:15:16.257 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 108.805199237, 2024-08-19T14:15:16.257 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.018577585000000001, 2024-08-19T14:15:16.258 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:15:16.258 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:15:16.258 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:15:16.258 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:15:16.258 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.258 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:15:16.258 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:27.290131+0000", 2024-08-19T14:15:16.258 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:16.258 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.258 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.259 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:15:16.259 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:27.290131+0000", 2024-08-19T14:15:16.259 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:16.259 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.259 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.259 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:15:16.259 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:27.290133+0000", 2024-08-19T14:15:16.259 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0099999999999998e-06 2024-08-19T14:15:16.259 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.259 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.259 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:15:16.260 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:27.290163+0000", 2024-08-19T14:15:16.260 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9737e-05 2024-08-19T14:15:16.260 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.260 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.260 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:15:16.260 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:27.290170+0000", 2024-08-19T14:15:16.260 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3520000000000001e-06 2024-08-19T14:15:16.260 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.260 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.260 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:15:16.260 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:27.290178+0000", 2024-08-19T14:15:16.261 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7370000000000006e-06 2024-08-19T14:15:16.261 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.261 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.261 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:15:16.261 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:27.290229+0000", 2024-08-19T14:15:16.261 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0809000000000001e-05 2024-08-19T14:15:16.261 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.261 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.261 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:15:16.261 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:27.290247+0000", 2024-08-19T14:15:16.261 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8331000000000001e-05 2024-08-19T14:15:16.261 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.262 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.262 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:15:16.262 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:27.308659+0000", 2024-08-19T14:15:16.262 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.018411594999999999 2024-08-19T14:15:16.262 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.262 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.262 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:15:16.262 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:27.308709+0000", 2024-08-19T14:15:16.262 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0013999999999999e-05 2024-08-19T14:15:16.262 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:16.262 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:15:16.263 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:16.263 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.263 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.263 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24580.0:84047 2.7 e135/127 2:edfa0ea9:::benchmark_data_smithi045_46415_object84046:head v 135'5174, mlcod=0'0)", 2024-08-19T14:15:16.263 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:39.324634+0000", 2024-08-19T14:15:16.263 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 96.770696896000004, 2024-08-19T14:15:16.263 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.019448039, 2024-08-19T14:15:16.263 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:15:16.263 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:15:16.263 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:15:16.264 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:15:16.264 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.264 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:15:16.264 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:39.324634+0000", 2024-08-19T14:15:16.264 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:16.264 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.264 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.264 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:15:16.264 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:39.324634+0000", 2024-08-19T14:15:16.264 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:16.264 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.264 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.264 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:15:16.264 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:39.324634+0000", 2024-08-19T14:15:16.265 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5699999999999998e-07 2024-08-19T14:15:16.265 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.265 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.265 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:15:16.265 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:39.324702+0000", 2024-08-19T14:15:16.265 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8205000000000002e-05 2024-08-19T14:15:16.265 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.265 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.265 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:15:16.265 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:39.324713+0000", 2024-08-19T14:15:16.265 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1061e-05 2024-08-19T14:15:16.266 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.266 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.266 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:15:16.266 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:39.324720+0000", 2024-08-19T14:15:16.266 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.5699999999999998e-06 2024-08-19T14:15:16.266 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.266 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.266 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:15:16.266 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:39.329466+0000", 2024-08-19T14:15:16.266 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0047466649999999997 2024-08-19T14:15:16.267 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.267 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.267 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:15:16.267 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:39.329496+0000", 2024-08-19T14:15:16.267 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9323e-05 2024-08-19T14:15:16.267 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.267 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.267 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:15:16.267 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:39.344027+0000", 2024-08-19T14:15:16.267 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.014531376 2024-08-19T14:15:16.267 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.267 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.267 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:15:16.268 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:39.344082+0000", 2024-08-19T14:15:16.268 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4382000000000002e-05 2024-08-19T14:15:16.268 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:16.268 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:15:16.268 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:16.268 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.268 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.268 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24580.0:86085 2.7 e136/127 2:ed9d86de:::benchmark_data_smithi045_46415_object86084:head v 136'5307, mlcod=0'0)", 2024-08-19T14:15:16.268 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:41.049265+0000", 2024-08-19T14:15:16.268 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 95.046065892000001, 2024-08-19T14:15:16.268 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.019980502000000001, 2024-08-19T14:15:16.268 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:15:16.268 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:15:16.269 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:15:16.269 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:15:16.269 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.269 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:15:16.269 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:41.049265+0000", 2024-08-19T14:15:16.269 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:16.269 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.269 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.269 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:15:16.269 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:41.049265+0000", 2024-08-19T14:15:16.269 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:16.269 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.269 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.269 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:15:16.270 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:41.049265+0000", 2024-08-19T14:15:16.270 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6499999999999998e-07 2024-08-19T14:15:16.270 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.270 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.270 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:15:16.270 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:41.049324+0000", 2024-08-19T14:15:16.270 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8538000000000001e-05 2024-08-19T14:15:16.270 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.270 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.270 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:15:16.270 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:41.049340+0000", 2024-08-19T14:15:16.270 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6511000000000002e-05 2024-08-19T14:15:16.270 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.271 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.271 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:15:16.271 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:41.049348+0000", 2024-08-19T14:15:16.271 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2749999999999998e-06 2024-08-19T14:15:16.271 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.271 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.271 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:15:16.271 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:41.051328+0000", 2024-08-19T14:15:16.271 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0019801520000000002 2024-08-19T14:15:16.271 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.271 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.271 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:15:16.271 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:41.051355+0000", 2024-08-19T14:15:16.271 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6877999999999999e-05 2024-08-19T14:15:16.272 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.272 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.272 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:15:16.272 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:41.069193+0000", 2024-08-19T14:15:16.272 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.017838515999999999 2024-08-19T14:15:16.272 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.272 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.272 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:15:16.272 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:41.069245+0000", 2024-08-19T14:15:16.272 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1767000000000001e-05 2024-08-19T14:15:16.272 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:16.272 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:15:16.273 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:16.273 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.273 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.273 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24580.0:86499 2.5 e136/102 2:ac09c600:::benchmark_data_smithi045_46415_object86498:head v 136'5439, mlcod=136'5437)", 2024-08-19T14:15:16.273 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:41.382046+0000", 2024-08-19T14:15:16.273 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 94.713284615000006, 2024-08-19T14:15:16.273 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.020539521000000002, 2024-08-19T14:15:16.273 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:15:16.273 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:15:16.273 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:15:16.273 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:15:16.273 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.273 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:15:16.274 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:41.382046+0000", 2024-08-19T14:15:16.274 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:16.274 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.274 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.274 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:15:16.274 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:41.382046+0000", 2024-08-19T14:15:16.274 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:16.274 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.274 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.274 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:15:16.274 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:41.382047+0000", 2024-08-19T14:15:16.274 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.4300000000000002e-07 2024-08-19T14:15:16.275 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.275 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.275 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:15:16.275 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:41.400143+0000", 2024-08-19T14:15:16.275 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.018096395000000001 2024-08-19T14:15:16.275 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.275 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.275 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:15:16.275 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:41.400159+0000", 2024-08-19T14:15:16.275 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6001000000000001e-05 2024-08-19T14:15:16.275 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.275 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.275 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:15:16.275 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:41.400167+0000", 2024-08-19T14:15:16.276 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.8059999999999995e-06 2024-08-19T14:15:16.276 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.276 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.276 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:15:16.276 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:41.400240+0000", 2024-08-19T14:15:16.276 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3236000000000002e-05 2024-08-19T14:15:16.276 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.276 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.276 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:15:16.276 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:41.400263+0000", 2024-08-19T14:15:16.276 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2977e-05 2024-08-19T14:15:16.276 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.276 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.276 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:15:16.276 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:41.402545+0000", 2024-08-19T14:15:16.277 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0022824450000000001 2024-08-19T14:15:16.277 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.277 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.277 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:15:16.277 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:41.402585+0000", 2024-08-19T14:15:16.277 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9817999999999997e-05 2024-08-19T14:15:16.277 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:16.277 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:15:16.277 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:16.277 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.277 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.277 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24580.0:98618 2.7 e139/127 2:e41d73e1:::benchmark_data_smithi045_46415_object98617:head v 139'6106, mlcod=0'0)", 2024-08-19T14:15:16.277 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:52.307086+0000", 2024-08-19T14:15:16.278 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 83.788244504999994, 2024-08-19T14:15:16.278 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.019682899, 2024-08-19T14:15:16.278 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:15:16.278 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:15:16.278 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:15:16.278 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:15:16.278 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.278 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:15:16.278 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:52.307086+0000", 2024-08-19T14:15:16.278 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:16.278 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.278 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.278 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:15:16.279 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:52.307086+0000", 2024-08-19T14:15:16.279 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:16.279 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.279 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.279 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:15:16.279 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:52.307087+0000", 2024-08-19T14:15:16.279 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.3399999999999998e-07 2024-08-19T14:15:16.279 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.279 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.279 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:15:16.279 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:52.307165+0000", 2024-08-19T14:15:16.279 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7769999999999996e-05 2024-08-19T14:15:16.279 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.280 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.280 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:15:16.280 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:52.307181+0000", 2024-08-19T14:15:16.280 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6860000000000001e-05 2024-08-19T14:15:16.280 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.280 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.280 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:15:16.280 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:52.307190+0000", 2024-08-19T14:15:16.280 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.4570000000000005e-06 2024-08-19T14:15:16.280 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.280 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.280 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:15:16.280 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:52.307254+0000", 2024-08-19T14:15:16.280 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4103000000000006e-05 2024-08-19T14:15:16.280 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.281 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.281 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:15:16.281 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:52.307281+0000", 2024-08-19T14:15:16.281 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6758e-05 2024-08-19T14:15:16.281 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.281 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.281 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:15:16.281 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:52.326728+0000", 2024-08-19T14:15:16.281 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.019447762 2024-08-19T14:15:16.281 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.281 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.281 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:15:16.281 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:52.326769+0000", 2024-08-19T14:15:16.281 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0355000000000002e-05 2024-08-19T14:15:16.282 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:16.282 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:15:16.282 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:16.282 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.282 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.282 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24580.0:100982 2.7 e141/127 2:eeafa31f:::benchmark_data_smithi045_46415_object100981:head v 141'6273, mlcod=0'0)", 2024-08-19T14:15:16.282 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:54.502352+0000", 2024-08-19T14:15:16.282 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 81.592978801000001, 2024-08-19T14:15:16.282 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.019840074999999999, 2024-08-19T14:15:16.282 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:15:16.282 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:15:16.282 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:15:16.283 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:15:16.283 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.283 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:15:16.283 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:54.502352+0000", 2024-08-19T14:15:16.283 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:16.283 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.283 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.283 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:15:16.283 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:54.502352+0000", 2024-08-19T14:15:16.283 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:16.283 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.283 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.284 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:15:16.284 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:54.502352+0000", 2024-08-19T14:15:16.284 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7500000000000001e-07 2024-08-19T14:15:16.284 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.284 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.284 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:15:16.284 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:54.510583+0000", 2024-08-19T14:15:16.284 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0082310960000000002 2024-08-19T14:15:16.284 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.284 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.284 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:15:16.285 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:54.510594+0000", 2024-08-19T14:15:16.285 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1012000000000001e-05 2024-08-19T14:15:16.285 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.285 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.285 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:15:16.285 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:54.510601+0000", 2024-08-19T14:15:16.285 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7320000000000001e-06 2024-08-19T14:15:16.285 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.285 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.285 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:15:16.285 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:54.510647+0000", 2024-08-19T14:15:16.286 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6307999999999998e-05 2024-08-19T14:15:16.286 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.286 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.286 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:15:16.286 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:54.510666+0000", 2024-08-19T14:15:16.286 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8885e-05 2024-08-19T14:15:16.286 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.286 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.286 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:15:16.286 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:54.522152+0000", 2024-08-19T14:15:16.286 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011486478 2024-08-19T14:15:16.287 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.287 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.287 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:15:16.287 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:54.522192+0000", 2024-08-19T14:15:16.287 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9189e-05 2024-08-19T14:15:16.287 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:16.287 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:15:16.287 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:16.287 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.287 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.287 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24580.0:171212 2.f e169/102 2:f8cd8517:::benchmark_data_smithi045_46415_object47931:head v 169'10747, mlcod=169'10744)", 2024-08-19T14:15:16.287 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:15:12.433327+0000", 2024-08-19T14:15:16.287 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 3.6620031329999998, 2024-08-19T14:15:16.288 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.055094690000000002, 2024-08-19T14:15:16.288 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:15:16.288 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:15:16.288 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:15:16.288 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:15:16.288 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.288 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:15:16.288 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.433327+0000", 2024-08-19T14:15:16.288 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:16.288 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.288 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.288 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:15:16.288 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.433327+0000", 2024-08-19T14:15:16.288 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:16.289 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.289 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.289 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:15:16.289 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.433328+0000", 2024-08-19T14:15:16.289 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.7900000000000007e-07 2024-08-19T14:15:16.289 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.289 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.289 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:15:16.289 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.433334+0000", 2024-08-19T14:15:16.289 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1190000000000002e-06 2024-08-19T14:15:16.289 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.289 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.289 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:15:16.290 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.433342+0000", 2024-08-19T14:15:16.290 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4850000000000003e-06 2024-08-19T14:15:16.290 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.290 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.290 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:15:16.290 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.433346+0000", 2024-08-19T14:15:16.290 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.527e-06 2024-08-19T14:15:16.290 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.290 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.290 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:15:16.290 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.433388+0000", 2024-08-19T14:15:16.290 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1619999999999998e-05 2024-08-19T14:15:16.290 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.290 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.291 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:15:16.291 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.433404+0000", 2024-08-19T14:15:16.291 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5991e-05 2024-08-19T14:15:16.291 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.291 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.291 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:15:16.291 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.488372+0000", 2024-08-19T14:15:16.291 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.054968052000000003 2024-08-19T14:15:16.291 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.291 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.291 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:15:16.291 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.488422+0000", 2024-08-19T14:15:16.291 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9917000000000003e-05 2024-08-19T14:15:16.291 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:16.292 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:15:16.292 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:16.292 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.292 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.292 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "replica_scrub(pg: 2.7,from:0'0,to:169'10634,epoch:169/164,start:2:e4f5c3df:::benchmark_data_smithi045_46415_object101587:0,end:2:e553be1d:::benchmark_data_smithi045_46415_object51347:0,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-08-19T14:15:16.292 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:15:12.436695+0000", 2024-08-19T14:15:16.292 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 3.6586355450000001, 2024-08-19T14:15:16.292 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.039097697000000001, 2024-08-19T14:15:16.292 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:15:16.292 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:15:16.292 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-08-19T14:15:16.292 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:15:16.292 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.292 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:15:16.293 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.436695+0000", 2024-08-19T14:15:16.293 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:16.293 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.293 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.293 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:15:16.293 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.436695+0000", 2024-08-19T14:15:16.293 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:16.293 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.293 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.293 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:15:16.293 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.436696+0000", 2024-08-19T14:15:16.293 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8899999999999999e-07 2024-08-19T14:15:16.293 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.293 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.294 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:15:16.294 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.436702+0000", 2024-08-19T14:15:16.294 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9859999999999999e-06 2024-08-19T14:15:16.294 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.294 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.294 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:15:16.294 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.475491+0000", 2024-08-19T14:15:16.294 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.038789259999999999 2024-08-19T14:15:16.294 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.294 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.294 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:15:16.294 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.475509+0000", 2024-08-19T14:15:16.294 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8488e-05 2024-08-19T14:15:16.294 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.295 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.295 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:15:16.295 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.475746+0000", 2024-08-19T14:15:16.295 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00023667600000000001 2024-08-19T14:15:16.295 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.295 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.295 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:15:16.295 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.475761+0000", 2024-08-19T14:15:16.295 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4901e-05 2024-08-19T14:15:16.295 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.295 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.295 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:15:16.295 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.475793+0000", 2024-08-19T14:15:16.295 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1696999999999998e-05 2024-08-19T14:15:16.296 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:16.296 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:15:16.296 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:16.296 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.296 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.296 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24580.0:171221 2.d e169/102 2:b5ea1297:::benchmark_data_smithi045_46415_object47940:head v 169'10755, mlcod=169'10753)", 2024-08-19T14:15:16.296 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:15:12.444916+0000", 2024-08-19T14:15:16.296 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 3.6504145919999997, 2024-08-19T14:15:16.296 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.039377216999999999, 2024-08-19T14:15:16.296 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:15:16.296 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:15:16.296 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:15:16.296 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:15:16.296 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.297 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:15:16.297 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.444916+0000", 2024-08-19T14:15:16.297 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:16.297 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.297 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.297 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:15:16.297 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.444916+0000", 2024-08-19T14:15:16.297 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:16.297 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.297 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.297 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:15:16.297 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.444917+0000", 2024-08-19T14:15:16.297 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.018e-06 2024-08-19T14:15:16.297 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.298 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.298 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:15:16.298 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.444926+0000", 2024-08-19T14:15:16.298 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.8750000000000006e-06 2024-08-19T14:15:16.298 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.298 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.298 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:15:16.298 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.475484+0000", 2024-08-19T14:15:16.298 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.030558333 2024-08-19T14:15:16.298 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.298 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.298 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:15:16.298 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.475502+0000", 2024-08-19T14:15:16.299 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7948000000000002e-05 2024-08-19T14:15:16.299 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.299 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.299 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:15:16.299 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.475649+0000", 2024-08-19T14:15:16.299 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000147503 2024-08-19T14:15:16.299 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.299 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.299 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:15:16.299 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.475673+0000", 2024-08-19T14:15:16.299 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3181999999999999e-05 2024-08-19T14:15:16.299 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.299 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.299 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:15:16.299 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.484205+0000", 2024-08-19T14:15:16.300 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0085321779999999996 2024-08-19T14:15:16.300 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:16.300 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:16.300 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:15:16.300 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.484293+0000", 2024-08-19T14:15:16.300 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.8179999999999999e-05 2024-08-19T14:15:16.300 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:16.300 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:15:16.300 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:16.300 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:16.300 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:15:16.300 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:15:16.301 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:15:16.302 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:15:16.326 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:16.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:16.327 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:15:16.428 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:15:16.451 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:16.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:16.452 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:15:16.553 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:15:16.574 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:16.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:16.575 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:15:16.671 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:15:16.672 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:15:16.676 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:15:16.692 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:15:16.692 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:15:16.699 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:16.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:16.700 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:15:16.711 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:15:16.739 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:15:16.739 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:15:16.739 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:15:16.744 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:15:16.744 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:15:16.761 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:15:16.761 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:15:16.784 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:15:16.784 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:15:16.801 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:15:16.807 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:15:16.807 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:15:16.823 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:16.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:16.824 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:15:16.835 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:15:16.836 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:15:16.865 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:15:16.866 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:15:16.895 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:15:16.895 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:15:16.924 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:15:16.926 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:15:16.927 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:15:16.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:16.948 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:16.948 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:15:16.997 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-19T14:15:17.049 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:15:17.072 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:17.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:17.073 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:15:17.168 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-08-19T14:15:17.174 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:15:17.198 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:17.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:17.199 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:15:17.300 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:15:17.327 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:17.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:17.328 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:15:17.429 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:15:17.455 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:17.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:17.456 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:15:17.471 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:15:17.557 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:15:17.579 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:17.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:17.580 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:15:17.681 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:15:17.706 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:17.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:17.707 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:15:17.788 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:15:17.808 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:15:17.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:17 smithi031 ceph-mon[16336]: pgmap v433: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 4.6 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 818 op/s 2024-08-19T14:15:17.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:17 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/427756356' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:15:17.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:17 smithi031 ceph-mon[16336]: 2.e starting backfill to osd.7 from (0'0,0'0] MAX to 170'10790 2024-08-19T14:15:17.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:17 smithi031 ceph-mon[16336]: 2.1 starting backfill to osd.8 from (0'0,0'0] MAX to 170'10895 2024-08-19T14:15:17.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:17 smithi031 ceph-mon[16336]: osdmap e173: 12 total, 12 up, 10 in 2024-08-19T14:15:17.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:17 smithi031 ceph-mon[16336]: 2.c scrub starts 2024-08-19T14:15:17.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:17 smithi031 ceph-mon[16336]: osdmap e174: 12 total, 12 up, 10 in 2024-08-19T14:15:17.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:17.870 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:17.870 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:15:17.971 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:15:18.005 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:18.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:18.006 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:15:18.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:17 smithi032 ceph-mon[20004]: pgmap v433: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 4.6 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 818 op/s 2024-08-19T14:15:18.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:17 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/427756356' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:15:18.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:17 smithi032 ceph-mon[20004]: 2.e starting backfill to osd.7 from (0'0,0'0] MAX to 170'10790 2024-08-19T14:15:18.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:17 smithi032 ceph-mon[20004]: 2.1 starting backfill to osd.8 from (0'0,0'0] MAX to 170'10895 2024-08-19T14:15:18.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:17 smithi032 ceph-mon[20004]: osdmap e173: 12 total, 12 up, 10 in 2024-08-19T14:15:18.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:17 smithi032 ceph-mon[20004]: 2.c scrub starts 2024-08-19T14:15:18.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:17 smithi032 ceph-mon[20004]: osdmap e174: 12 total, 12 up, 10 in 2024-08-19T14:15:18.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:17 smithi045 ceph-mon[20143]: pgmap v433: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 4.6 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 818 op/s 2024-08-19T14:15:18.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:17 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/427756356' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:15:18.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:17 smithi045 ceph-mon[20143]: 2.e starting backfill to osd.7 from (0'0,0'0] MAX to 170'10790 2024-08-19T14:15:18.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:17 smithi045 ceph-mon[20143]: 2.1 starting backfill to osd.8 from (0'0,0'0] MAX to 170'10895 2024-08-19T14:15:18.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:17 smithi045 ceph-mon[20143]: osdmap e173: 12 total, 12 up, 10 in 2024-08-19T14:15:18.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:17 smithi045 ceph-mon[20143]: 2.c scrub starts 2024-08-19T14:15:18.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:17 smithi045 ceph-mon[20143]: osdmap e174: 12 total, 12 up, 10 in 2024-08-19T14:15:18.107 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:15:18.133 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:18.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:18.134 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:15:18.154 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:15:18.154 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:15:18.235 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:15:18.268 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:18.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:18.269 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:15:18.370 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:15:18.396 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:18.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:18.397 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:15:18.498 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:15:18.521 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:18.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:18.522 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:15:18.623 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:15:18.659 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:18.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:18.660 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:15:18.761 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:15:18.791 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:18.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:18.792 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:15:18.893 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:15:18.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:18.934 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:18.934 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:15:18.959 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-08-19T14:15:19.035 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:15:19.059 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:19.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:19.060 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:15:19.161 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:15:19.188 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:19.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:19.189 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:15:19.267 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:15:19.290 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:15:19.334 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:19.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:19.335 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:15:19.436 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:15:19.475 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:19.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:19.476 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:15:19.577 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:15:19.600 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:19.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:19.601 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:15:19.623 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:15:19.661 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is set 2024-08-19T14:15:19.682 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:19 smithi032 ceph-mon[20004]: pgmap v436: 27 pgs: 3 remapped+peering, 1 active+clean+scrubbing, 23 active+clean; 4.5 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 519 op/s 2024-08-19T14:15:19.682 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:19 smithi032 ceph-mon[20004]: 2.9 scrub starts 2024-08-19T14:15:19.683 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:19 smithi032 ceph-mon[20004]: osdmap e175: 12 total, 12 up, 10 in 2024-08-19T14:15:19.683 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:19 smithi032 ceph-mon[20004]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-08-19T14:15:19.683 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:19 smithi032 ceph-mon[20004]: 2.c scrub ok 2024-08-19T14:15:19.683 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:19 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3373228634' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:15:19.702 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:15:19.727 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:19.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:19.728 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:15:19.804 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:15:19.804 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:15:19.829 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:15:19.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:19.869 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:19.869 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:15:19.970 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:15:20.020 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:20.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:20.021 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:15:20.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:19 smithi045 ceph-mon[20143]: pgmap v436: 27 pgs: 3 remapped+peering, 1 active+clean+scrubbing, 23 active+clean; 4.5 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 519 op/s 2024-08-19T14:15:20.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:19 smithi045 ceph-mon[20143]: 2.9 scrub starts 2024-08-19T14:15:20.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:19 smithi045 ceph-mon[20143]: osdmap e175: 12 total, 12 up, 10 in 2024-08-19T14:15:20.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:19 smithi045 ceph-mon[20143]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-08-19T14:15:20.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:19 smithi045 ceph-mon[20143]: 2.c scrub ok 2024-08-19T14:15:20.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:19 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3373228634' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:15:20.075 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:19 smithi031 ceph-mon[16336]: pgmap v436: 27 pgs: 3 remapped+peering, 1 active+clean+scrubbing, 23 active+clean; 4.5 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 519 op/s 2024-08-19T14:15:20.075 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:19 smithi031 ceph-mon[16336]: 2.9 scrub starts 2024-08-19T14:15:20.076 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:19 smithi031 ceph-mon[16336]: osdmap e175: 12 total, 12 up, 10 in 2024-08-19T14:15:20.076 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:19 smithi031 ceph-mon[16336]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-08-19T14:15:20.076 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:19 smithi031 ceph-mon[16336]: 2.c scrub ok 2024-08-19T14:15:20.076 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:19 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3373228634' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:15:20.122 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:15:20.160 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:20.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:20.161 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:15:20.262 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:15:20.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:20.301 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:20.302 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:15:20.403 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:15:20.423 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 8, 9, 11] out_osds: [10, 6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-08-19T14:15:20.423 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-19T14:15:20.424 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-19T14:15:20.426 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:20.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:20.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:15:20.489 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:15:20.495 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-08-19T14:15:20.528 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:15:20.553 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:20.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:20.554 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:15:20.655 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:15:20.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:20.696 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:20.696 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:15:20.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:20 smithi032 ceph-mon[20004]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:15:20.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:20 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3373228634' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:15:20.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:20 smithi032 ceph-mon[20004]: osdmap e176: 12 total, 12 up, 10 in 2024-08-19T14:15:20.798 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:15:20.799 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:15:20.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:20.826 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:20.826 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:15:20.927 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:15:20.951 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:20.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:20.952 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:15:20.959 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:20 smithi031 ceph-mon[16336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:15:20.960 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:20 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3373228634' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:15:20.960 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:20 smithi031 ceph-mon[16336]: osdmap e176: 12 total, 12 up, 10 in 2024-08-19T14:15:21.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:20 smithi045 ceph-mon[20143]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:15:21.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:20 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3373228634' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:15:21.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:20 smithi045 ceph-mon[20143]: osdmap e176: 12 total, 12 up, 10 in 2024-08-19T14:15:21.053 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:15:21.076 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:21.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:21.077 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:15:21.135 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:15:21.158 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:15:21.159 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:15:21.178 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:15:21.197 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:15:21.197 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:15:21.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:21.232 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:21.232 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:15:21.239 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:15:21.239 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:15:21.276 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:15:21.276 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:15:21.314 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:15:21.315 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:15:21.333 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:15:21.349 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:15:21.349 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:15:21.358 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:21.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:21.359 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:15:21.386 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:15:21.387 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:15:21.425 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:15:21.425 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-08-19T14:15:21.425 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-08-19T14:15:21.425 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-08-19T14:15:21.425 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.425 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:62938 2.2 2.749ad082 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-08-19T14:15:21.426 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:18.730377+0000", 2024-08-19T14:15:21.426 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 122.692449966, 2024-08-19T14:15:21.426 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.98852832099999999, 2024-08-19T14:15:21.426 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:15:21.426 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:15:21.426 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:15:21.426 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:15:21.426 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:15:21.426 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:15:21.426 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 62938 2024-08-19T14:15:21.426 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.426 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:15:21.427 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.427 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:15:21.427 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.730377+0000", 2024-08-19T14:15:21.427 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:21.427 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.427 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.427 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:15:21.427 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.730377+0000", 2024-08-19T14:15:21.427 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:21.427 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.427 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.427 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:15:21.427 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.730380+0000", 2024-08-19T14:15:21.428 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5129999999999999e-06 2024-08-19T14:15:21.428 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.428 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.428 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:15:21.428 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.730416+0000", 2024-08-19T14:15:21.428 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5981e-05 2024-08-19T14:15:21.428 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.428 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.428 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:15:21.428 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.730441+0000", 2024-08-19T14:15:21.428 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4964e-05 2024-08-19T14:15:21.428 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.428 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.429 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:15:21.429 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.730450+0000", 2024-08-19T14:15:21.429 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.5850000000000006e-06 2024-08-19T14:15:21.429 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.429 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.429 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:15:21.429 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.730530+0000", 2024-08-19T14:15:21.429 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.9703999999999994e-05 2024-08-19T14:15:21.429 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.429 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.429 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:15:21.429 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.730544+0000", 2024-08-19T14:15:21.429 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4229e-05 2024-08-19T14:15:21.429 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.430 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.430 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:15:21.430 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.718832+0000", 2024-08-19T14:15:21.430 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.98828802699999996 2024-08-19T14:15:21.430 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.430 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.430 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:15:21.430 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.718905+0000", 2024-08-19T14:15:21.430 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3318000000000005e-05 2024-08-19T14:15:21.430 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:21.430 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:15:21.430 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:21.431 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.431 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.431 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:62942 2.2 2.8241cf42 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-08-19T14:15:21.431 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:18.731498+0000", 2024-08-19T14:15:21.431 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 122.691329293, 2024-08-19T14:15:21.431 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.98803843599999996, 2024-08-19T14:15:21.431 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:15:21.431 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:15:21.431 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:15:21.431 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:15:21.431 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:15:21.431 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:15:21.431 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 62942 2024-08-19T14:15:21.431 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.431 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:15:21.432 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.432 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:15:21.432 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.731498+0000", 2024-08-19T14:15:21.432 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:21.432 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.432 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.432 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:15:21.432 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.731498+0000", 2024-08-19T14:15:21.432 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:21.432 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.432 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.432 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:15:21.432 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.731499+0000", 2024-08-19T14:15:21.433 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.4600000000000003e-07 2024-08-19T14:15:21.433 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.433 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.433 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:15:21.433 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.731572+0000", 2024-08-19T14:15:21.433 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2946000000000003e-05 2024-08-19T14:15:21.433 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.433 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.433 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:15:21.433 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.731584+0000", 2024-08-19T14:15:21.433 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2272e-05 2024-08-19T14:15:21.433 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.434 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.434 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:15:21.434 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.731591+0000", 2024-08-19T14:15:21.434 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6290000000000001e-06 2024-08-19T14:15:21.434 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.434 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.434 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:15:21.434 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.731642+0000", 2024-08-19T14:15:21.434 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0930999999999997e-05 2024-08-19T14:15:21.434 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.434 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.434 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:15:21.434 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.731650+0000", 2024-08-19T14:15:21.435 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.554e-06 2024-08-19T14:15:21.435 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.435 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.435 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:15:21.435 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.719522+0000", 2024-08-19T14:15:21.435 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.98787217500000002 2024-08-19T14:15:21.435 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.435 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.435 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:15:21.435 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.719536+0000", 2024-08-19T14:15:21.435 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4083000000000001e-05 2024-08-19T14:15:21.435 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:21.435 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:15:21.435 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:21.435 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.436 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.436 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:62947 2.2 2.528f01a2 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-08-19T14:15:21.436 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:18.735148+0000", 2024-08-19T14:15:21.436 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 122.687678838, 2024-08-19T14:15:21.436 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.98444280500000003, 2024-08-19T14:15:21.436 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:15:21.436 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:15:21.436 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:15:21.436 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:15:21.436 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:15:21.436 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:15:21.436 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 62947 2024-08-19T14:15:21.436 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.436 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:15:21.437 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.437 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:15:21.437 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.735148+0000", 2024-08-19T14:15:21.437 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:21.437 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.437 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.437 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:15:21.437 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.735148+0000", 2024-08-19T14:15:21.437 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:21.437 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.437 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.437 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:15:21.437 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.735150+0000", 2024-08-19T14:15:21.437 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.903e-06 2024-08-19T14:15:21.438 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.438 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.438 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:15:21.438 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.735192+0000", 2024-08-19T14:15:21.438 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1780999999999999e-05 2024-08-19T14:15:21.438 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.438 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.438 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:15:21.439 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.735211+0000", 2024-08-19T14:15:21.439 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9021e-05 2024-08-19T14:15:21.439 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.439 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.439 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:15:21.439 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.735221+0000", 2024-08-19T14:15:21.439 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.9140000000000003e-06 2024-08-19T14:15:21.439 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.439 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.439 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:15:21.439 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.735291+0000", 2024-08-19T14:15:21.439 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0599999999999995e-05 2024-08-19T14:15:21.439 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.439 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.440 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:15:21.440 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.735304+0000", 2024-08-19T14:15:21.440 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2218999999999999e-05 2024-08-19T14:15:21.440 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.440 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.440 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:15:21.440 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.719582+0000", 2024-08-19T14:15:21.440 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.98427827999999995 2024-08-19T14:15:21.440 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.440 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.440 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:15:21.441 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:15:21.442 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:15:21.442 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.719591+0000", 2024-08-19T14:15:21.442 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.0869999999999996e-06 2024-08-19T14:15:21.442 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:21.442 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:15:21.442 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:21.442 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.442 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.442 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:62961 2.2 2.d8a35c52 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-08-19T14:15:21.442 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:18.743735+0000", 2024-08-19T14:15:21.442 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 122.679092095, 2024-08-19T14:15:21.442 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.97590511300000005, 2024-08-19T14:15:21.443 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:15:21.443 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:15:21.443 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:15:21.443 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:15:21.443 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:15:21.443 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:15:21.443 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 62961 2024-08-19T14:15:21.443 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.443 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:15:21.443 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.443 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:15:21.443 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.743735+0000", 2024-08-19T14:15:21.443 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:21.443 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.444 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.444 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:15:21.444 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.743735+0000", 2024-08-19T14:15:21.444 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:21.444 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.444 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.444 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:15:21.444 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.743737+0000", 2024-08-19T14:15:21.444 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1689999999999999e-06 2024-08-19T14:15:21.444 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.444 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.444 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:15:21.444 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.743772+0000", 2024-08-19T14:15:21.444 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5293000000000002e-05 2024-08-19T14:15:21.445 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.445 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.445 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:15:21.445 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.743795+0000", 2024-08-19T14:15:21.445 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2167e-05 2024-08-19T14:15:21.445 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.445 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.445 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:15:21.445 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.743804+0000", 2024-08-19T14:15:21.445 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3379999999999994e-06 2024-08-19T14:15:21.445 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.445 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.445 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:15:21.445 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.743875+0000", 2024-08-19T14:15:21.446 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1073000000000002e-05 2024-08-19T14:15:21.446 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.446 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.446 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:15:21.446 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.743888+0000", 2024-08-19T14:15:21.446 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2622999999999999e-05 2024-08-19T14:15:21.446 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.446 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.446 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:15:21.446 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.719630+0000", 2024-08-19T14:15:21.446 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.97574243299999996 2024-08-19T14:15:21.446 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.446 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.447 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:15:21.447 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.719640+0000", 2024-08-19T14:15:21.447 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0017000000000001e-05 2024-08-19T14:15:21.447 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:21.447 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:15:21.447 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:21.447 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.447 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.447 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:62991 2.2 2.365dd392 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-08-19T14:15:21.447 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:18.765407+0000", 2024-08-19T14:15:21.447 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 122.65741959499999, 2024-08-19T14:15:21.447 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.95428210599999996, 2024-08-19T14:15:21.447 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:15:21.447 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:15:21.447 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:15:21.448 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:15:21.448 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:15:21.448 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:15:21.448 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 62991 2024-08-19T14:15:21.448 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.448 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:15:21.448 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.448 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:15:21.448 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.765407+0000", 2024-08-19T14:15:21.448 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:21.448 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.448 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.448 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:15:21.449 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.765407+0000", 2024-08-19T14:15:21.449 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:21.449 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.449 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.449 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:15:21.449 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.765409+0000", 2024-08-19T14:15:21.449 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0950000000000001e-06 2024-08-19T14:15:21.449 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.449 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.449 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:15:21.449 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.765456+0000", 2024-08-19T14:15:21.449 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7061000000000002e-05 2024-08-19T14:15:21.449 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.449 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.450 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:15:21.450 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.765473+0000", 2024-08-19T14:15:21.450 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7377e-05 2024-08-19T14:15:21.450 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.450 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.450 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:15:21.450 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.765480+0000", 2024-08-19T14:15:21.450 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7889999999999997e-06 2024-08-19T14:15:21.450 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.450 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.450 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:15:21.450 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.765536+0000", 2024-08-19T14:15:21.450 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5897999999999999e-05 2024-08-19T14:15:21.451 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.451 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.451 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:15:21.451 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.765545+0000", 2024-08-19T14:15:21.451 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.4939999999999992e-06 2024-08-19T14:15:21.451 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.451 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.451 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:15:21.451 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.719682+0000", 2024-08-19T14:15:21.451 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.95413691499999997 2024-08-19T14:15:21.451 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.451 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.451 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:15:21.452 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.719690+0000", 2024-08-19T14:15:21.452 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4769999999999997e-06 2024-08-19T14:15:21.452 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:21.452 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:15:21.452 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:21.452 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.452 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.452 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:62995 2.2 2.d0be02c2 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-08-19T14:15:21.452 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:18.768402+0000", 2024-08-19T14:15:21.452 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 122.65442526, 2024-08-19T14:15:21.452 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.95133469699999995, 2024-08-19T14:15:21.452 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:15:21.452 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:15:21.452 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:15:21.453 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:15:21.453 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:15:21.453 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:15:21.453 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 62995 2024-08-19T14:15:21.453 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.453 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:15:21.453 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.453 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:15:21.453 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.768402+0000", 2024-08-19T14:15:21.453 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:21.453 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.453 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.453 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:15:21.453 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.768402+0000", 2024-08-19T14:15:21.454 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:21.454 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.454 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.454 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:15:21.454 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.768402+0000", 2024-08-19T14:15:21.454 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7000000000000004e-07 2024-08-19T14:15:21.454 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.454 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.454 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:15:21.454 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.768439+0000", 2024-08-19T14:15:21.454 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6641999999999999e-05 2024-08-19T14:15:21.454 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.454 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.454 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:15:21.455 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.768450+0000", 2024-08-19T14:15:21.455 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0635999999999999e-05 2024-08-19T14:15:21.455 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.455 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.455 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:15:21.455 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.768455+0000", 2024-08-19T14:15:21.455 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9840000000000001e-06 2024-08-19T14:15:21.455 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.455 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.455 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:15:21.455 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.768498+0000", 2024-08-19T14:15:21.455 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3350000000000003e-05 2024-08-19T14:15:21.455 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.455 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.456 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:15:21.456 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.768504+0000", 2024-08-19T14:15:21.456 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3380000000000001e-06 2024-08-19T14:15:21.456 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.456 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.456 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:15:21.456 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.719726+0000", 2024-08-19T14:15:21.456 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.95122155900000005 2024-08-19T14:15:21.456 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.456 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.456 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:15:21.456 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.719736+0000", 2024-08-19T14:15:21.456 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0518e-05 2024-08-19T14:15:21.457 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:21.457 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:15:21.457 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:21.457 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.457 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.457 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:63016 2.2 2.3ca449b2 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-08-19T14:15:21.457 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:18.789614+0000", 2024-08-19T14:15:21.457 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 122.633212774, 2024-08-19T14:15:21.457 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.93017577600000001, 2024-08-19T14:15:21.457 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:15:21.457 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:15:21.457 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:15:21.457 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:15:21.457 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:15:21.458 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:15:21.458 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 63016 2024-08-19T14:15:21.458 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.458 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:15:21.458 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.458 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:15:21.458 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.789614+0000", 2024-08-19T14:15:21.458 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:21.458 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.458 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.458 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:15:21.458 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.789614+0000", 2024-08-19T14:15:21.458 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:21.459 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.459 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.459 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:15:21.459 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.789616+0000", 2024-08-19T14:15:21.459 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.327e-06 2024-08-19T14:15:21.459 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.459 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.459 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:15:21.459 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.789658+0000", 2024-08-19T14:15:21.459 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1934000000000002e-05 2024-08-19T14:15:21.459 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.459 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.459 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:15:21.459 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.789667+0000", 2024-08-19T14:15:21.460 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.6549999999999999e-06 2024-08-19T14:15:21.460 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.460 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.460 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:15:21.460 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.789672+0000", 2024-08-19T14:15:21.460 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0080000000000002e-06 2024-08-19T14:15:21.460 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.460 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.460 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:15:21.460 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.789732+0000", 2024-08-19T14:15:21.462 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9568999999999998e-05 2024-08-19T14:15:21.462 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.462 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.462 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:15:21.462 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.789742+0000", 2024-08-19T14:15:21.462 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0740000000000001e-05 2024-08-19T14:15:21.462 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.462 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.462 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:15:21.463 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.719765+0000", 2024-08-19T14:15:21.463 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.930022723 2024-08-19T14:15:21.463 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.463 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.463 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:15:21.463 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.719790+0000", 2024-08-19T14:15:21.463 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.482e-05 2024-08-19T14:15:21.463 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:21.463 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:15:21.463 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:21.463 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.463 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.463 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:63026 2.2 2.519434f2 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-08-19T14:15:21.464 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:18.799709+0000", 2024-08-19T14:15:21.464 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 122.623117867, 2024-08-19T14:15:21.464 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.92011866099999995, 2024-08-19T14:15:21.464 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:15:21.464 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:15:21.464 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:15:21.464 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:15:21.464 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:15:21.464 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:15:21.464 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 63026 2024-08-19T14:15:21.464 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.464 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:15:21.464 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.464 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:15:21.465 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.799709+0000", 2024-08-19T14:15:21.465 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:21.465 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.465 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.465 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:15:21.465 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.799709+0000", 2024-08-19T14:15:21.465 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:21.465 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.465 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.465 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:15:21.465 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.799710+0000", 2024-08-19T14:15:21.465 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.333e-06 2024-08-19T14:15:21.465 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.465 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.466 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:15:21.466 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.799743+0000", 2024-08-19T14:15:21.466 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2845000000000001e-05 2024-08-19T14:15:21.466 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.466 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.466 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:15:21.466 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.799757+0000", 2024-08-19T14:15:21.466 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3655999999999999e-05 2024-08-19T14:15:21.466 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.466 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.466 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:15:21.466 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.799764+0000", 2024-08-19T14:15:21.466 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2760000000000003e-06 2024-08-19T14:15:21.466 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.467 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.467 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:15:21.467 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.799815+0000", 2024-08-19T14:15:21.467 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0269000000000003e-05 2024-08-19T14:15:21.467 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.467 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.467 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:15:21.467 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.799822+0000", 2024-08-19T14:15:21.467 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7840000000000004e-06 2024-08-19T14:15:21.467 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.467 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.467 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:15:21.467 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.719821+0000", 2024-08-19T14:15:21.467 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.91999866900000005 2024-08-19T14:15:21.468 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.468 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.468 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:15:21.468 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.719828+0000", 2024-08-19T14:15:21.468 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8290000000000002e-06 2024-08-19T14:15:21.468 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:21.468 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:15:21.468 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:21.468 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.468 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.468 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:63031 2.2 2.92065272 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-08-19T14:15:21.468 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:18.809226+0000", 2024-08-19T14:15:21.468 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 122.613600664, 2024-08-19T14:15:21.468 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.91063533200000002, 2024-08-19T14:15:21.468 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:15:21.469 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:15:21.469 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:15:21.469 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:15:21.469 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:15:21.469 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:15:21.469 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 63031 2024-08-19T14:15:21.469 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.469 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:15:21.469 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.469 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:15:21.469 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.809226+0000", 2024-08-19T14:15:21.469 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:21.469 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.469 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.470 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:15:21.470 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.809226+0000", 2024-08-19T14:15:21.470 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:21.470 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.470 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.470 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:15:21.470 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.809228+0000", 2024-08-19T14:15:21.470 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4670000000000001e-06 2024-08-19T14:15:21.470 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.470 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.470 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:15:21.470 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.809267+0000", 2024-08-19T14:15:21.470 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8881000000000003e-05 2024-08-19T14:15:21.471 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.471 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.471 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:15:21.471 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.809282+0000", 2024-08-19T14:15:21.471 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5333e-05 2024-08-19T14:15:21.471 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.471 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.471 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:15:21.471 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.809289+0000", 2024-08-19T14:15:21.471 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7299999999999999e-06 2024-08-19T14:15:21.471 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.471 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.471 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:15:21.471 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.809352+0000", 2024-08-19T14:15:21.471 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3639000000000005e-05 2024-08-19T14:15:21.472 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.472 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.472 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:15:21.472 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.809364+0000", 2024-08-19T14:15:21.472 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1995e-05 2024-08-19T14:15:21.472 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.472 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.472 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:15:21.472 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.719856+0000", 2024-08-19T14:15:21.472 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.91049139999999995 2024-08-19T14:15:21.472 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.472 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.472 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:15:21.473 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.719862+0000", 2024-08-19T14:15:21.473 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8869999999999997e-06 2024-08-19T14:15:21.473 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:21.473 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:15:21.473 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:21.473 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.473 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.473 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:67350 2.3 2:c4d0387d:::benchmark_data_smithi045_46415_object67349:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-08-19T14:15:21.473 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:24.307914+0000", 2024-08-19T14:15:21.473 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 117.114912942, 2024-08-19T14:15:21.473 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.13986817500000001, 2024-08-19T14:15:21.473 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:15:21.473 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:15:21.473 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:15:21.474 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:15:21.474 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:15:21.474 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:15:21.474 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 67350 2024-08-19T14:15:21.474 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.474 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:15:21.474 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.474 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:15:21.474 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:24.307914+0000", 2024-08-19T14:15:21.474 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:21.474 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.474 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.474 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:15:21.474 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:24.307914+0000", 2024-08-19T14:15:21.475 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:21.475 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.475 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.475 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:15:21.475 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:24.307915+0000", 2024-08-19T14:15:21.475 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3659999999999999e-06 2024-08-19T14:15:21.475 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.475 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.475 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:15:21.475 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:24.307971+0000", 2024-08-19T14:15:21.475 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5931999999999999e-05 2024-08-19T14:15:21.475 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.475 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.475 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:15:21.476 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:24.307991+0000", 2024-08-19T14:15:21.476 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0023000000000001e-05 2024-08-19T14:15:21.476 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.476 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.476 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:15:21.476 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:24.307998+0000", 2024-08-19T14:15:21.476 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3149999999999997e-06 2024-08-19T14:15:21.476 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.476 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.476 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:15:21.476 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:24.308053+0000", 2024-08-19T14:15:21.476 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5220000000000003e-05 2024-08-19T14:15:21.476 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.477 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.477 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:15:21.477 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:24.308188+0000", 2024-08-19T14:15:21.477 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013537999999999999 2024-08-19T14:15:21.477 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.477 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.477 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,7", 2024-08-19T14:15:21.477 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:24.308354+0000", 2024-08-19T14:15:21.477 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00016596300000000001 2024-08-19T14:15:21.477 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.477 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.477 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:15:21.477 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:24.310423+0000", 2024-08-19T14:15:21.477 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0020690349999999999 2024-08-19T14:15:21.478 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.478 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.478 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:21.478 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:24.310859+0000", 2024-08-19T14:15:21.478 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000435454 2024-08-19T14:15:21.478 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.478 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.478 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:21.478 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:24.447684+0000", 2024-08-19T14:15:21.478 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.13682524200000001 2024-08-19T14:15:21.478 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.478 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.478 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:15:21.479 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:24.447747+0000", 2024-08-19T14:15:21.479 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2657000000000007e-05 2024-08-19T14:15:21.479 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.479 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.479 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:15:21.479 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:24.447782+0000", 2024-08-19T14:15:21.479 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5587999999999999e-05 2024-08-19T14:15:21.479 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:21.479 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:15:21.479 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:21.479 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.479 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.479 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:97001 2.3 2:c1261b7a:::benchmark_data_smithi045_46415_object97000:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-08-19T14:15:21.480 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:50.862243+0000", 2024-08-19T14:15:21.480 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 90.560584315, 2024-08-19T14:15:21.480 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20703321999999999, 2024-08-19T14:15:21.480 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:15:21.480 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:15:21.480 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:15:21.480 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:15:21.480 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:15:21.480 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:15:21.480 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 97001 2024-08-19T14:15:21.480 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.480 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:15:21.480 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.480 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:15:21.481 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:50.862243+0000", 2024-08-19T14:15:21.481 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:21.481 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.481 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.481 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:15:21.481 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:50.862243+0000", 2024-08-19T14:15:21.481 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:21.481 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.481 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.481 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:15:21.481 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:50.862244+0000", 2024-08-19T14:15:21.481 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.4200000000000005e-07 2024-08-19T14:15:21.481 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.482 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.482 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:15:21.482 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:50.862301+0000", 2024-08-19T14:15:21.482 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7503000000000002e-05 2024-08-19T14:15:21.482 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.482 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.482 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:15:21.482 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:50.862312+0000", 2024-08-19T14:15:21.482 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1344e-05 2024-08-19T14:15:21.482 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.482 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.482 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:15:21.482 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:50.862320+0000", 2024-08-19T14:15:21.482 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2799999999999998e-06 2024-08-19T14:15:21.483 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.483 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.483 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:15:21.483 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:50.864128+0000", 2024-08-19T14:15:21.483 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001808507 2024-08-19T14:15:21.483 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.483 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.483 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:15:21.483 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:50.864261+0000", 2024-08-19T14:15:21.483 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000132936 2024-08-19T14:15:21.483 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.483 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.483 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,7", 2024-08-19T14:15:21.483 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:50.864414+0000", 2024-08-19T14:15:21.484 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00015251300000000001 2024-08-19T14:15:21.484 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.484 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.484 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:15:21.484 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:50.867763+0000", 2024-08-19T14:15:21.484 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0033497370000000002 2024-08-19T14:15:21.484 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.484 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.484 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:21.484 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:50.888271+0000", 2024-08-19T14:15:21.484 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.020507279999999999 2024-08-19T14:15:21.484 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.484 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.484 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:21.485 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:51.069158+0000", 2024-08-19T14:15:21.485 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.18088735 2024-08-19T14:15:21.485 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.485 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.485 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:15:21.485 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:51.069221+0000", 2024-08-19T14:15:21.485 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3505999999999998e-05 2024-08-19T14:15:21.485 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.485 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.485 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:15:21.485 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:51.069276+0000", 2024-08-19T14:15:21.485 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4422000000000002e-05 2024-08-19T14:15:21.485 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:21.485 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:15:21.485 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:21.486 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.486 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.486 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:15:21.487 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:97032 2.3 2:c018f995:::benchmark_data_smithi045_46415_object97031:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-08-19T14:15:21.487 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:50.910523+0000", 2024-08-19T14:15:21.487 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 90.512304286000003, 2024-08-19T14:15:21.488 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.17243983199999999, 2024-08-19T14:15:21.488 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:15:21.488 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:15:21.488 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:15:21.488 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:15:21.488 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:15:21.488 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:15:21.488 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 97032 2024-08-19T14:15:21.488 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.488 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:15:21.488 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.488 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:15:21.488 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:50.910523+0000", 2024-08-19T14:15:21.489 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:21.489 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.489 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.489 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:15:21.489 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:50.910523+0000", 2024-08-19T14:15:21.489 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:21.489 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.489 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.489 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:15:21.489 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:50.910524+0000", 2024-08-19T14:15:21.489 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4419999999999999e-06 2024-08-19T14:15:21.489 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.489 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.489 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:15:21.490 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:50.910586+0000", 2024-08-19T14:15:21.490 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1588999999999996e-05 2024-08-19T14:15:21.490 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.490 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.490 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:15:21.490 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:50.910597+0000", 2024-08-19T14:15:21.490 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1528999999999999e-05 2024-08-19T14:15:21.490 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.490 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.490 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:15:21.490 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:50.910620+0000", 2024-08-19T14:15:21.490 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2588000000000001e-05 2024-08-19T14:15:21.490 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.491 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.491 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:15:21.491 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:50.910666+0000", 2024-08-19T14:15:21.491 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5760999999999998e-05 2024-08-19T14:15:21.491 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.491 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.491 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:15:21.491 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:50.910754+0000", 2024-08-19T14:15:21.491 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.8374999999999993e-05 2024-08-19T14:15:21.491 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.491 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.491 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,7", 2024-08-19T14:15:21.491 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:50.910840+0000", 2024-08-19T14:15:21.491 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6407000000000001e-05 2024-08-19T14:15:21.492 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.492 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.492 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:15:21.492 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:50.913335+0000", 2024-08-19T14:15:21.492 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0024949960000000002 2024-08-19T14:15:21.492 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.492 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.492 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:21.492 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:50.916172+0000", 2024-08-19T14:15:21.492 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0028362000000000001 2024-08-19T14:15:21.492 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.492 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.492 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:21.492 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:51.082859+0000", 2024-08-19T14:15:21.492 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.16668744799999999 2024-08-19T14:15:21.493 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.493 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.493 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:15:21.493 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:51.082917+0000", 2024-08-19T14:15:21.493 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7689000000000003e-05 2024-08-19T14:15:21.493 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.493 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.493 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:15:21.493 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:51.082963+0000", 2024-08-19T14:15:21.493 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5807999999999999e-05 2024-08-19T14:15:21.493 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:21.493 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:15:21.493 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:21.493 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.494 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.494 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:97038 2.3 2:ca56678d:::benchmark_data_smithi045_46415_object97037:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-08-19T14:15:21.494 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:50.919445+0000", 2024-08-19T14:15:21.494 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 90.503381583999996, 2024-08-19T14:15:21.494 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.16627785, 2024-08-19T14:15:21.494 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:15:21.494 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:15:21.494 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:15:21.494 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:15:21.494 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:15:21.494 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:15:21.494 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 97038 2024-08-19T14:15:21.494 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.494 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:15:21.495 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.495 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:15:21.495 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:50.919445+0000", 2024-08-19T14:15:21.495 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:21.495 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.495 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.495 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:15:21.495 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:50.919445+0000", 2024-08-19T14:15:21.495 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:21.495 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.495 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.495 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:15:21.495 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:50.919447+0000", 2024-08-19T14:15:21.495 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.598e-06 2024-08-19T14:15:21.496 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.496 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.496 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:15:21.496 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:50.919587+0000", 2024-08-19T14:15:21.496 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000139791 2024-08-19T14:15:21.496 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.496 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.496 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:15:21.496 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:50.919603+0000", 2024-08-19T14:15:21.496 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5784999999999999e-05 2024-08-19T14:15:21.496 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.496 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.496 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:15:21.496 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:50.919610+0000", 2024-08-19T14:15:21.496 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0729999999999999e-06 2024-08-19T14:15:21.497 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.497 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.497 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:15:21.497 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:50.919665+0000", 2024-08-19T14:15:21.497 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4821999999999998e-05 2024-08-19T14:15:21.497 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.497 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.497 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:15:21.497 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:50.919806+0000", 2024-08-19T14:15:21.497 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000141643 2024-08-19T14:15:21.497 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.497 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.497 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,7", 2024-08-19T14:15:21.497 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:50.919966+0000", 2024-08-19T14:15:21.498 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00015960599999999999 2024-08-19T14:15:21.498 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.498 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.498 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:15:21.498 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:50.922407+0000", 2024-08-19T14:15:21.498 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0024416149999999998 2024-08-19T14:15:21.498 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.498 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.498 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:21.498 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:50.925173+0000", 2024-08-19T14:15:21.498 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.002766017 2024-08-19T14:15:21.498 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.498 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.498 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:21.499 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:51.085627+0000", 2024-08-19T14:15:21.499 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.16045364100000001 2024-08-19T14:15:21.499 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.499 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.499 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:15:21.499 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:51.085679+0000", 2024-08-19T14:15:21.499 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1533000000000003e-05 2024-08-19T14:15:21.499 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.499 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.499 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:15:21.499 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:51.085723+0000", 2024-08-19T14:15:21.499 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4725999999999998e-05 2024-08-19T14:15:21.499 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:21.499 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:15:21.500 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:21.500 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.500 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.500 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:97074 2.3 2:c018fd72:::benchmark_data_smithi045_46415_object97073:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-08-19T14:15:21.500 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:50.942481+0000", 2024-08-19T14:15:21.500 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 90.480346069999996, 2024-08-19T14:15:21.500 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.14415905500000001, 2024-08-19T14:15:21.500 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:15:21.500 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:15:21.500 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:15:21.500 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:15:21.500 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:15:21.500 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:15:21.501 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 97074 2024-08-19T14:15:21.501 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.501 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:15:21.501 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.501 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:15:21.501 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:50.942481+0000", 2024-08-19T14:15:21.501 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:21.501 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.501 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.501 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:15:21.501 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:50.942481+0000", 2024-08-19T14:15:21.501 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:21.501 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.501 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.502 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:15:21.502 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:50.942483+0000", 2024-08-19T14:15:21.502 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6700000000000001e-06 2024-08-19T14:15:21.502 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.502 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.502 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:15:21.502 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:50.942539+0000", 2024-08-19T14:15:21.502 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6833000000000003e-05 2024-08-19T14:15:21.502 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.502 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.502 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:15:21.502 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:50.942553+0000", 2024-08-19T14:15:21.502 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4027e-05 2024-08-19T14:15:21.502 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.503 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.503 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:15:21.503 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:50.942561+0000", 2024-08-19T14:15:21.503 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6939999999999995e-06 2024-08-19T14:15:21.503 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.503 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.503 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:15:21.503 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:50.942616+0000", 2024-08-19T14:15:21.503 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5216999999999997e-05 2024-08-19T14:15:21.503 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.503 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.503 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:15:21.503 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:50.942761+0000", 2024-08-19T14:15:21.503 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000144673 2024-08-19T14:15:21.504 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.504 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.504 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,7", 2024-08-19T14:15:21.504 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:50.942905+0000", 2024-08-19T14:15:21.504 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014373899999999999 2024-08-19T14:15:21.504 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.504 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.504 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:15:21.504 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:50.945551+0000", 2024-08-19T14:15:21.504 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.002646514 2024-08-19T14:15:21.504 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.504 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.504 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:21.504 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:50.950058+0000", 2024-08-19T14:15:21.505 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0045065820000000003 2024-08-19T14:15:21.505 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.505 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.505 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:21.505 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:51.086552+0000", 2024-08-19T14:15:21.505 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.13649413899999999 2024-08-19T14:15:21.505 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.505 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.505 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:15:21.505 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:51.086600+0000", 2024-08-19T14:15:21.505 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8301999999999999e-05 2024-08-19T14:15:21.505 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.505 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.505 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:15:21.505 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:51.086640+0000", 2024-08-19T14:15:21.506 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9665000000000001e-05 2024-08-19T14:15:21.506 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:21.506 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:15:21.506 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:21.506 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.506 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.506 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:171107 2.3 2:cdcc5df6:::benchmark_data_smithi045_46415_object47826:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e169)", 2024-08-19T14:15:21.506 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:15:12.315407+0000", 2024-08-19T14:15:21.506 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 9.1074199149999995, 2024-08-19T14:15:21.506 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.23111446799999999, 2024-08-19T14:15:21.506 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:15:21.506 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:15:21.506 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:15:21.507 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:15:21.507 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:15:21.507 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:15:21.507 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 171107 2024-08-19T14:15:21.507 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.507 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:15:21.507 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.507 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:15:21.507 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.315407+0000", 2024-08-19T14:15:21.507 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:21.507 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.507 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.507 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:15:21.507 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.315407+0000", 2024-08-19T14:15:21.508 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:21.508 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.508 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.508 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:15:21.508 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.315408+0000", 2024-08-19T14:15:21.508 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.094e-06 2024-08-19T14:15:21.508 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.508 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.508 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:15:21.508 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.315413+0000", 2024-08-19T14:15:21.508 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2070000000000003e-06 2024-08-19T14:15:21.508 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.508 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.508 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:15:21.509 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.315424+0000", 2024-08-19T14:15:21.509 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0873000000000001e-05 2024-08-19T14:15:21.509 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.509 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.509 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:15:21.509 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.315431+0000", 2024-08-19T14:15:21.509 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1149999999999996e-06 2024-08-19T14:15:21.509 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.509 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.509 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:15:21.509 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.315490+0000", 2024-08-19T14:15:21.509 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8118000000000002e-05 2024-08-19T14:15:21.509 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.509 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.509 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:15:21.510 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.315625+0000", 2024-08-19T14:15:21.510 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000135902 2024-08-19T14:15:21.510 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.510 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.510 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,7", 2024-08-19T14:15:21.510 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.315754+0000", 2024-08-19T14:15:21.510 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012891700000000001 2024-08-19T14:15:21.510 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.510 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.510 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:15:21.510 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.320475+0000", 2024-08-19T14:15:21.510 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0047206069999999999 2024-08-19T14:15:21.510 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.510 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.511 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:21.511 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.321696+0000", 2024-08-19T14:15:21.511 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00122135 2024-08-19T14:15:21.511 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.511 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.511 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:21.511 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.546437+0000", 2024-08-19T14:15:21.511 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.22474116199999999 2024-08-19T14:15:21.511 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.511 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.511 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:15:21.511 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.546477+0000", 2024-08-19T14:15:21.511 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9263e-05 2024-08-19T14:15:21.511 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.512 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.512 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:15:21.512 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.546522+0000", 2024-08-19T14:15:21.512 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4860000000000001e-05 2024-08-19T14:15:21.512 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:21.512 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:15:21.512 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:21.512 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.512 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.512 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:171113 2.3 2:cfe749bb:::benchmark_data_smithi045_46415_object47832:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e169)", 2024-08-19T14:15:21.512 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:15:12.322593+0000", 2024-08-19T14:15:21.512 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 9.1002342420000009, 2024-08-19T14:15:21.512 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.228733402, 2024-08-19T14:15:21.512 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:15:21.513 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:15:21.513 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:15:21.513 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:15:21.513 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:15:21.513 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:15:21.513 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 171113 2024-08-19T14:15:21.513 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.513 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:15:21.513 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.513 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:15:21.513 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.322593+0000", 2024-08-19T14:15:21.513 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:21.513 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.513 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.513 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:15:21.514 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.322593+0000", 2024-08-19T14:15:21.514 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:21.514 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.514 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.514 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:15:21.514 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.322594+0000", 2024-08-19T14:15:21.514 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2240000000000001e-06 2024-08-19T14:15:21.514 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.514 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.514 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:15:21.514 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.322603+0000", 2024-08-19T14:15:21.514 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6910000000000003e-06 2024-08-19T14:15:21.514 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.515 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.515 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:15:21.515 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.322615+0000", 2024-08-19T14:15:21.515 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2181000000000001e-05 2024-08-19T14:15:21.515 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.515 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.515 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:15:21.515 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.322620+0000", 2024-08-19T14:15:21.515 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3070000000000002e-06 2024-08-19T14:15:21.515 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.515 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.515 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:15:21.515 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.322677+0000", 2024-08-19T14:15:21.515 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6407999999999999e-05 2024-08-19T14:15:21.515 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.516 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.516 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:15:21.516 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.322808+0000", 2024-08-19T14:15:21.516 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013162499999999999 2024-08-19T14:15:21.516 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.516 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.516 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,7", 2024-08-19T14:15:21.516 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.322936+0000", 2024-08-19T14:15:21.516 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000127358 2024-08-19T14:15:21.516 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.516 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.516 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:15:21.516 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.332145+0000", 2024-08-19T14:15:21.516 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0092095149999999997 2024-08-19T14:15:21.517 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.517 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.517 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:21.517 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.333304+0000", 2024-08-19T14:15:21.517 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0011586330000000001 2024-08-19T14:15:21.517 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.517 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.517 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:21.517 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.551228+0000", 2024-08-19T14:15:21.517 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.2179239 2024-08-19T14:15:21.517 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.517 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.517 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:15:21.517 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.551282+0000", 2024-08-19T14:15:21.518 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4080999999999999e-05 2024-08-19T14:15:21.518 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.518 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.518 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:15:21.518 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.551326+0000", 2024-08-19T14:15:21.518 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4478999999999999e-05 2024-08-19T14:15:21.518 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:21.518 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:15:21.518 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:21.518 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.518 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.518 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:171114 2.3 2:cce6d5f2:::benchmark_data_smithi045_46415_object47833:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e169)", 2024-08-19T14:15:21.518 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:15:12.322676+0000", 2024-08-19T14:15:21.518 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 9.1001506410000008, 2024-08-19T14:15:21.519 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.228818197, 2024-08-19T14:15:21.519 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:15:21.519 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:15:21.519 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:15:21.519 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:15:21.519 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:15:21.519 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:15:21.519 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 171114 2024-08-19T14:15:21.519 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.519 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:15:21.519 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.519 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:15:21.519 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.322676+0000", 2024-08-19T14:15:21.520 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:21.520 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.520 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.520 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:15:21.520 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.322676+0000", 2024-08-19T14:15:21.520 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:21.520 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.520 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.520 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:15:21.520 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.322677+0000", 2024-08-19T14:15:21.520 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7999999999999996e-07 2024-08-19T14:15:21.520 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.520 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.520 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:15:21.521 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.322680+0000", 2024-08-19T14:15:21.521 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9519999999999999e-06 2024-08-19T14:15:21.521 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.521 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.521 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:15:21.521 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.322689+0000", 2024-08-19T14:15:21.521 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.6879999999999994e-06 2024-08-19T14:15:21.521 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.521 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.521 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:15:21.521 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.322693+0000", 2024-08-19T14:15:21.521 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9910000000000004e-06 2024-08-19T14:15:21.521 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.522 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.522 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:15:21.522 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.327524+0000", 2024-08-19T14:15:21.522 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0048305259999999999 2024-08-19T14:15:21.522 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.522 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.522 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:15:21.522 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.327692+0000", 2024-08-19T14:15:21.522 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000167956 2024-08-19T14:15:21.522 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.522 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.522 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,7", 2024-08-19T14:15:21.522 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.327829+0000", 2024-08-19T14:15:21.522 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013674999999999999 2024-08-19T14:15:21.523 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.523 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.523 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:15:21.523 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.332259+0000", 2024-08-19T14:15:21.523 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0044300859999999997 2024-08-19T14:15:21.523 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.523 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.523 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:21.523 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.333546+0000", 2024-08-19T14:15:21.523 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001287683 2024-08-19T14:15:21.523 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.523 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.523 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:21.524 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.551388+0000", 2024-08-19T14:15:21.524 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.21784162100000001 2024-08-19T14:15:21.524 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.524 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.524 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:15:21.524 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.551451+0000", 2024-08-19T14:15:21.524 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2556000000000002e-05 2024-08-19T14:15:21.524 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.524 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.524 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:15:21.524 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.551495+0000", 2024-08-19T14:15:21.524 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3908e-05 2024-08-19T14:15:21.524 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:21.524 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:15:21.525 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:21.525 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.525 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.525 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:171144 2.3 2:c9a546c7:::benchmark_data_smithi045_46415_object47863:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e169)", 2024-08-19T14:15:21.525 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:15:12.356410+0000", 2024-08-19T14:15:21.525 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 9.0664168249999992, 2024-08-19T14:15:21.525 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.19522335099999999, 2024-08-19T14:15:21.525 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:15:21.525 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:15:21.525 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:15:21.525 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:15:21.525 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:15:21.525 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:15:21.525 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 171144 2024-08-19T14:15:21.526 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.526 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:15:21.526 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.526 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:15:21.526 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.356410+0000", 2024-08-19T14:15:21.526 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:21.526 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.526 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.526 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:15:21.526 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.356410+0000", 2024-08-19T14:15:21.526 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:21.526 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.526 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.526 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:15:21.527 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.356411+0000", 2024-08-19T14:15:21.527 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1540000000000001e-06 2024-08-19T14:15:21.527 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.527 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.527 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:15:21.527 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.356416+0000", 2024-08-19T14:15:21.527 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5940000000000004e-06 2024-08-19T14:15:21.527 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.527 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.527 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:15:21.527 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.356429+0000", 2024-08-19T14:15:21.527 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3416e-05 2024-08-19T14:15:21.527 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.528 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.528 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:15:21.528 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.356437+0000", 2024-08-19T14:15:21.528 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.8699999999999992e-06 2024-08-19T14:15:21.528 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.528 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.528 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:15:21.528 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.356491+0000", 2024-08-19T14:15:21.528 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4150999999999997e-05 2024-08-19T14:15:21.528 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.528 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.528 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:15:21.528 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.356631+0000", 2024-08-19T14:15:21.528 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013998 2024-08-19T14:15:21.529 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.529 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.529 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,7", 2024-08-19T14:15:21.529 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.356774+0000", 2024-08-19T14:15:21.529 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000142786 2024-08-19T14:15:21.529 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.529 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.529 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:15:21.529 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.361685+0000", 2024-08-19T14:15:21.529 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0049104800000000001 2024-08-19T14:15:21.529 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.529 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.529 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:21.529 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.364217+0000", 2024-08-19T14:15:21.530 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0025319790000000002 2024-08-19T14:15:21.530 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.530 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.530 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:21.530 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.551537+0000", 2024-08-19T14:15:21.530 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.187320815 2024-08-19T14:15:21.530 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.530 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.530 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:15:21.530 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.551592+0000", 2024-08-19T14:15:21.530 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4775000000000003e-05 2024-08-19T14:15:21.530 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.530 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.530 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:15:21.531 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.551634+0000", 2024-08-19T14:15:21.531 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1350999999999998e-05 2024-08-19T14:15:21.531 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:21.531 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:15:21.531 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:21.531 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.531 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.531 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:171158 2.3 2:c268548f:::benchmark_data_smithi045_46415_object47877:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e169)", 2024-08-19T14:15:21.531 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:15:12.370127+0000", 2024-08-19T14:15:21.531 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 9.0527001140000003, 2024-08-19T14:15:21.531 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.186453074, 2024-08-19T14:15:21.531 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:15:21.531 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:15:21.531 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:15:21.531 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:15:21.532 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:15:21.532 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:15:21.532 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 171158 2024-08-19T14:15:21.532 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.532 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:15:21.532 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.532 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:15:21.532 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.370127+0000", 2024-08-19T14:15:21.532 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:21.532 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.532 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.532 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:15:21.532 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.370127+0000", 2024-08-19T14:15:21.532 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:21.533 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.533 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.533 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:15:21.533 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.370128+0000", 2024-08-19T14:15:21.533 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.345e-06 2024-08-19T14:15:21.533 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.533 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.533 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:15:21.533 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.370142+0000", 2024-08-19T14:15:21.533 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4248e-05 2024-08-19T14:15:21.533 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.533 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.533 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:15:21.533 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.370159+0000", 2024-08-19T14:15:21.534 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6654e-05 2024-08-19T14:15:21.534 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.534 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.534 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:15:21.534 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.370166+0000", 2024-08-19T14:15:21.534 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1380000000000001e-06 2024-08-19T14:15:21.534 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.534 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.534 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:15:21.534 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.370228+0000", 2024-08-19T14:15:21.534 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1693999999999996e-05 2024-08-19T14:15:21.534 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.534 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.534 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:15:21.535 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.370358+0000", 2024-08-19T14:15:21.535 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000130476 2024-08-19T14:15:21.535 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.535 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.535 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,7", 2024-08-19T14:15:21.535 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.370481+0000", 2024-08-19T14:15:21.535 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012303499999999999 2024-08-19T14:15:21.535 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.535 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.535 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:15:21.535 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.374580+0000", 2024-08-19T14:15:21.535 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0040981409999999996 2024-08-19T14:15:21.535 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.535 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.536 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:21.536 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.375959+0000", 2024-08-19T14:15:21.536 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0013793659999999999 2024-08-19T14:15:21.536 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.536 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.536 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:21.536 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.556472+0000", 2024-08-19T14:15:21.536 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.18051325900000001 2024-08-19T14:15:21.536 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.536 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.536 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:15:21.536 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.556532+0000", 2024-08-19T14:15:21.536 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9293000000000003e-05 2024-08-19T14:15:21.536 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.537 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.537 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:15:21.537 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.556580+0000", 2024-08-19T14:15:21.537 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8424999999999998e-05 2024-08-19T14:15:21.537 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:21.537 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:15:21.537 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:21.537 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.537 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.537 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:171193 2.3 2:c9dfbf61:::benchmark_data_smithi045_46415_object47912:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e169)", 2024-08-19T14:15:21.537 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:15:12.406652+0000", 2024-08-19T14:15:21.537 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 9.0161749530000002, 2024-08-19T14:15:21.537 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.17138025200000001, 2024-08-19T14:15:21.537 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:15:21.538 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:15:21.538 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:15:21.538 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:15:21.538 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:15:21.538 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:15:21.538 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 171193 2024-08-19T14:15:21.538 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.538 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:15:21.538 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.538 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:15:21.539 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.406652+0000", 2024-08-19T14:15:21.539 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:21.539 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.539 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.539 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:15:21.539 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.406652+0000", 2024-08-19T14:15:21.539 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:21.539 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.539 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.539 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:15:21.539 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.406654+0000", 2024-08-19T14:15:21.539 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5850000000000001e-06 2024-08-19T14:15:21.539 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.540 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.540 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:15:21.540 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.406661+0000", 2024-08-19T14:15:21.540 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9430000000000003e-06 2024-08-19T14:15:21.540 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.540 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.540 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:15:21.540 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.406679+0000", 2024-08-19T14:15:21.540 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8457999999999998e-05 2024-08-19T14:15:21.540 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.540 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.540 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:15:21.540 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.406686+0000", 2024-08-19T14:15:21.540 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3130000000000001e-06 2024-08-19T14:15:21.540 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.541 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.541 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:15:21.541 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.406751+0000", 2024-08-19T14:15:21.541 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4196999999999995e-05 2024-08-19T14:15:21.541 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.541 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.541 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:15:21.541 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.406894+0000", 2024-08-19T14:15:21.541 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014297800000000001 2024-08-19T14:15:21.541 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.541 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.541 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,7", 2024-08-19T14:15:21.541 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.407051+0000", 2024-08-19T14:15:21.541 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000157081 2024-08-19T14:15:21.542 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.542 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.542 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:15:21.542 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.411818+0000", 2024-08-19T14:15:21.542 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0047669169999999999 2024-08-19T14:15:21.542 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.542 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.542 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:21.542 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.429453+0000", 2024-08-19T14:15:21.542 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.017635432999999999 2024-08-19T14:15:21.542 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.542 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.542 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:21.543 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.577927+0000", 2024-08-19T14:15:21.543 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.148474044 2024-08-19T14:15:21.543 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.543 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.543 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:15:21.543 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.577985+0000", 2024-08-19T14:15:21.543 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8152000000000002e-05 2024-08-19T14:15:21.543 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:21.543 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:21.543 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:15:21.543 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.578032+0000", 2024-08-19T14:15:21.543 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7151000000000003e-05 2024-08-19T14:15:21.543 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:21.543 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:15:21.544 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:21.544 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:21.544 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:15:21.544 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:15:21.545 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:15:21.545 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:15:21.555 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:15:21.555 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:15:21.584 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:21.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:21.585 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:15:21.598 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:15:21.598 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:15:21.638 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:15:21.638 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:15:21.687 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:15:21.711 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:21.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:21.712 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:15:21.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:21 smithi032 ceph-mon[20004]: pgmap v439: 27 pgs: 3 remapped+peering, 24 active+clean; 4.4 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 535 op/s 2024-08-19T14:15:21.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:21 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1837409673' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:15:21.814 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:15:21.840 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:21.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:21.841 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:15:21.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:21 smithi031 ceph-mon[16336]: pgmap v439: 27 pgs: 3 remapped+peering, 24 active+clean; 4.4 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 535 op/s 2024-08-19T14:15:21.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:21 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1837409673' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:15:21.942 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:15:21.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:21.978 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:21.978 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:15:22.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:21 smithi045 ceph-mon[20143]: pgmap v439: 27 pgs: 3 remapped+peering, 24 active+clean; 4.4 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 535 op/s 2024-08-19T14:15:22.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:21 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1837409673' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:15:22.047 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:15:22.048 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-08-19T14:15:22.078 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:15:22.106 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:22.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:22.107 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:15:22.208 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:15:22.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:22.235 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:22.236 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:15:22.299 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-08-19T14:15:22.336 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:15:22.366 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:22.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:22.368 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:15:22.421 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-19T14:15:22.469 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:15:22.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:22.530 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:22.530 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:15:22.613 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:15:22.631 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:15:22.660 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:22.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:22.661 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:15:22.762 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:15:22.805 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:22.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:22.806 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:15:22.907 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:15:22.940 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:22.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:22.941 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:15:23.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:22 smithi032 ceph-mon[20004]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-08-19T14:15:23.042 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:15:23.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:22 smithi045 ceph-mon[20143]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-08-19T14:15:23.074 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:23.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:23.075 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:15:23.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:22 smithi031 ceph-mon[16336]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-08-19T14:15:23.164 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:15:23.164 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-08-19T14:15:23.164 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-08-19T14:15:23.164 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:15:23.176 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:15:23.204 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:23.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:23.205 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:15:23.232 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":440,"stamp":"2024-08-19T14:15:21.773882+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4669702712,"num_objects":71248,"num_object_clones":0,"num_object_copies":213744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7687,"num_objects_unfound":0,"num_objects_dirty":71248,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":175312,"num_write_kb":7889793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15570,"num_bytes_recovered":981925888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":162162,"ondisk_log_size":162162,"up":81,"acting":81,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":81,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":14834632,"kb_used_data":13861968,"kb_used_omap":15,"kb_used_meta":972464,"kb_avail":922616888,"statfs":{"total":959950356480,"available":944759693312,"internally_reserved":0,"allocated":14194655232,"data_stored":14185916744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15948,"internal_metadata":995803572},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-182190080,"num_objects":-2780,"num_object_clones":0,"num_object_copies":-8340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7687,"num_objects_unfound":0,"num_objects_dirty":-2780,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2780,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1061,"num_bytes_recovered":69533696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.121639"},"pg_stats":[{"pgid":"2.f","version":"175'11000","reported_seq":11240,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-08-19T14:15:19.659055+0000","last_change":"2024-08-19T14:14:10.955308+0000","last_active":"2024-08-19T14:15:19.659055+0000","last_peered":"2024-08-19T14:15:19.659055+0000","last_clean":"2024-08-19T14:15:19.659055+0000","last_became_active":"2024-08-19T14:12:43.417591+0000","last_became_peered":"2024-08-19T14:12:43.417591+0000","last_unstale":"2024-08-19T14:15:19.659055+0000","last_undegraded":"2024-08-19T14:15:19.659055+0000","last_fullsized":"2024-08-19T14:15:19.659055+0000","mapping_epoch":102,"log_start":"107'900","ondisk_log_start":"107'900","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'7296","last_scrub_stamp":"2024-08-19T14:14:10.955043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:14:10.955043+0000","objects_scrubbed":7248,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:40.398760+0000","scrub_duration":1903,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":295305216,"num_objects":4506,"num_object_clones":0,"num_object_copies":13518,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11000,"num_write_kb":496192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.e","version":"176'11026","reported_seq":12235,"reported_epoch":176,"state":"active+remapped+backfilling","last_fresh":"2024-08-19T14:15:21.119474+0000","last_change":"2024-08-19T14:15:16.957445+0000","last_active":"2024-08-19T14:15:21.119474+0000","last_peered":"2024-08-19T14:15:21.119474+0000","last_clean":"2024-08-19T14:15:14.653230+0000","last_became_active":"2024-08-19T14:15:16.902152+0000","last_became_peered":"2024-08-19T14:15:16.902152+0000","last_unstale":"2024-08-19T14:15:21.119474+0000","last_undegraded":"2024-08-19T14:15:21.119474+0000","last_fullsized":"2024-08-19T14:15:21.119474+0000","mapping_epoch":172,"log_start":"107'1000","ondisk_log_start":"107'1000","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:12:41.713780+0000","objects_scrubbed":3824,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284033024,"num_objects":4334,"num_object_clones":0,"num_object_copies":13002,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3839,"num_objects_unfound":0,"num_objects_dirty":4334,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11026,"num_write_kb":491520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":501,"num_bytes_recovered":32833536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,7],"acting":[2,9,3],"avail_no_missing":["2","3","9"],"object_location_counts":[{"shards":"2,3,9","objects":4334}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"174'10913","reported_seq":11150,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-08-19T14:15:18.215098+0000","last_change":"2024-08-19T14:14:11.678308+0000","last_active":"2024-08-19T14:15:18.215098+0000","last_peered":"2024-08-19T14:15:18.215098+0000","last_clean":"2024-08-19T14:15:18.215098+0000","last_became_active":"2024-08-19T14:12:43.417506+0000","last_became_peered":"2024-08-19T14:12:43.417506+0000","last_unstale":"2024-08-19T14:15:18.215098+0000","last_undegraded":"2024-08-19T14:15:18.215098+0000","last_fullsized":"2024-08-19T14:15:18.215098+0000","mapping_epoch":102,"log_start":"107'900","ondisk_log_start":"107'900","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'7275","last_scrub_stamp":"2024-08-19T14:14:11.678069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:14:11.678069+0000","objects_scrubbed":7231,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:23.845013+0000","scrub_duration":2180,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":292945944,"num_objects":4471,"num_object_clones":0,"num_object_copies":13413,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4471,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10913,"num_write_kb":492225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.c","version":"176'10948","reported_seq":11245,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-08-19T14:15:20.529397+0000","last_change":"2024-08-19T14:15:18.768595+0000","last_active":"2024-08-19T14:15:20.529397+0000","last_peered":"2024-08-19T14:15:20.529397+0000","last_clean":"2024-08-19T14:15:20.529397+0000","last_became_active":"2024-08-19T14:12:43.038912+0000","last_became_peered":"2024-08-19T14:12:43.038912+0000","last_unstale":"2024-08-19T14:15:20.529397+0000","last_undegraded":"2024-08-19T14:15:20.529397+0000","last_fullsized":"2024-08-19T14:15:20.529397+0000","mapping_epoch":102,"log_start":"107'900","ondisk_log_start":"107'900","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'10846","last_scrub_stamp":"2024-08-19T14:15:18.768342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:15:18.768342+0000","objects_scrubbed":4537,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:16:37.101539+0000","scrub_duration":1760,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":287965184,"num_objects":4394,"num_object_clones":0,"num_object_copies":13182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4394,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10948,"num_write_kb":490944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,1],"acting":[11,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"70'139","reported_seq":540,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-08-19T14:15:17.639665+0000","last_change":"2024-08-19T14:14:36.306513+0000","last_active":"2024-08-19T14:15:17.639665+0000","last_peered":"2024-08-19T14:15:17.639665+0000","last_clean":"2024-08-19T14:15:17.639665+0000","last_became_active":"2024-08-19T14:13:22.729498+0000","last_became_peered":"2024-08-19T14:13:22.729498+0000","last_unstale":"2024-08-19T14:15:17.639665+0000","last_undegraded":"2024-08-19T14:15:17.639665+0000","last_fullsized":"2024-08-19T14:15:17.639665+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:14:36.306409+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:14:36.306409+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:45.640981+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,9],"acting":[7,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"176'11117","reported_seq":11353,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-08-19T14:15:21.475929+0000","last_change":"2024-08-19T14:14:03.748630+0000","last_active":"2024-08-19T14:15:21.475929+0000","last_peered":"2024-08-19T14:15:21.475929+0000","last_clean":"2024-08-19T14:15:21.475929+0000","last_became_active":"2024-08-19T14:12:42.728978+0000","last_became_peered":"2024-08-19T14:12:42.728978+0000","last_unstale":"2024-08-19T14:15:21.475929+0000","last_undegraded":"2024-08-19T14:15:21.475929+0000","last_fullsized":"2024-08-19T14:15:21.475929+0000","mapping_epoch":102,"log_start":"107'1100","ondisk_log_start":"107'1100","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"146'6847","last_scrub_stamp":"2024-08-19T14:14:03.748468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:14:03.748468+0000","objects_scrubbed":6806,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:25.274768+0000","scrub_duration":1341,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281346048,"num_objects":4293,"num_object_clones":0,"num_object_copies":12879,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11117,"num_write_kb":493120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.3","version":"70'139","reported_seq":425,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-08-19T14:15:17.638712+0000","last_change":"2024-08-19T14:14:38.031170+0000","last_active":"2024-08-19T14:15:17.638712+0000","last_peered":"2024-08-19T14:15:17.638712+0000","last_clean":"2024-08-19T14:15:17.638712+0000","last_became_active":"2024-08-19T14:13:22.727234+0000","last_became_peered":"2024-08-19T14:13:22.727234+0000","last_unstale":"2024-08-19T14:15:17.638712+0000","last_undegraded":"2024-08-19T14:15:17.638712+0000","last_fullsized":"2024-08-19T14:15:17.638712+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:14:38.031056+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:14:38.031056+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:16:03.171231+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"174'10833","reported_seq":17171,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-08-19T14:15:18.011391+0000","last_change":"2024-08-19T14:15:11.392878+0000","last_active":"2024-08-19T14:15:18.011391+0000","last_peered":"2024-08-19T14:15:18.011391+0000","last_clean":"2024-08-19T14:15:18.011391+0000","last_became_active":"2024-08-19T14:13:32.749994+0000","last_became_peered":"2024-08-19T14:13:32.749994+0000","last_unstale":"2024-08-19T14:15:18.011391+0000","last_undegraded":"2024-08-19T14:15:18.011391+0000","last_fullsized":"2024-08-19T14:15:18.011391+0000","mapping_epoch":132,"log_start":"107'800","ondisk_log_start":"107'800","created":102,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"169'10608","last_scrub_stamp":"2024-08-19T14:15:11.392791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:15:11.392791+0000","objects_scrubbed":4851,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:16:30.589490+0000","scrub_duration":894,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301793280,"num_objects":4605,"num_object_clones":0,"num_object_copies":13815,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10833,"num_write_kb":494016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3024,"num_bytes_recovered":198180864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,11],"acting":[7,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"70'139","reported_seq":539,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-08-19T14:15:17.639629+0000","last_change":"2024-08-19T14:14:31.408564+0000","last_active":"2024-08-19T14:15:17.639629+0000","last_peered":"2024-08-19T14:15:17.639629+0000","last_clean":"2024-08-19T14:15:17.639629+0000","last_became_active":"2024-08-19T14:13:05.607783+0000","last_became_peered":"2024-08-19T14:13:05.607783+0000","last_unstale":"2024-08-19T14:15:17.639629+0000","last_undegraded":"2024-08-19T14:15:17.639629+0000","last_fullsized":"2024-08-19T14:15:17.639629+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:14:31.408492+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:14:31.408492+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:41.655086+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"176'11112","reported_seq":12398,"reported_epoch":176,"state":"active+remapped+backfilling","last_fresh":"2024-08-19T14:15:21.110502+0000","last_change":"2024-08-19T14:15:16.800434+0000","last_active":"2024-08-19T14:15:21.110502+0000","last_peered":"2024-08-19T14:15:21.110502+0000","last_clean":"2024-08-19T14:15:14.646022+0000","last_became_active":"2024-08-19T14:15:16.769586+0000","last_became_peered":"2024-08-19T14:15:16.769586+0000","last_unstale":"2024-08-19T14:15:21.110502+0000","last_undegraded":"2024-08-19T14:15:21.110502+0000","last_fullsized":"2024-08-19T14:15:21.110502+0000","mapping_epoch":172,"log_start":"107'1100","ondisk_log_start":"107'1100","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:12:41.713780+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":287965184,"num_objects":4394,"num_object_clones":0,"num_object_copies":13182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3848,"num_objects_unfound":0,"num_objects_dirty":4394,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11112,"num_write_kb":496192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":560,"num_bytes_recovered":36700160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,8],"acting":[2,0,3],"avail_no_missing":["2","0","3"],"object_location_counts":[{"shards":"0,2,3","objects":4394}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"70'139","reported_seq":413,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-08-19T14:15:17.637317+0000","last_change":"2024-08-19T14:15:09.350783+0000","last_active":"2024-08-19T14:15:17.637317+0000","last_peered":"2024-08-19T14:15:17.637317+0000","last_clean":"2024-08-19T14:15:17.637317+0000","last_became_active":"2024-08-19T14:13:11.663185+0000","last_became_peered":"2024-08-19T14:13:11.663185+0000","last_unstale":"2024-08-19T14:15:17.637317+0000","last_undegraded":"2024-08-19T14:15:17.637317+0000","last_fullsized":"2024-08-19T14:15:17.637317+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:15:09.350658+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:15:09.350658+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:16:12.339458+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2],"acting":[9,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.2","version":"176'11042","reported_seq":19098,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-08-19T14:15:21.502425+0000","last_change":"2024-08-19T14:14:13.196736+0000","last_active":"2024-08-19T14:15:21.502425+0000","last_peered":"2024-08-19T14:15:21.502425+0000","last_clean":"2024-08-19T14:15:21.502425+0000","last_became_active":"2024-08-19T14:13:58.623348+0000","last_became_peered":"2024-08-19T14:13:58.623348+0000","last_unstale":"2024-08-19T14:15:21.502425+0000","last_undegraded":"2024-08-19T14:15:21.502425+0000","last_fullsized":"2024-08-19T14:15:21.502425+0000","mapping_epoch":142,"log_start":"107'1000","ondisk_log_start":"107'1000","created":102,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"148'7390","last_scrub_stamp":"2024-08-19T14:14:13.196497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:14:13.196497+0000","objects_scrubbed":7346,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:26.733911+0000","scrub_duration":1724,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":287965184,"num_objects":4394,"num_object_clones":0,"num_object_copies":13182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4394,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11042,"num_write_kb":493952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3902,"num_bytes_recovered":255721472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"70'139","reported_seq":325,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-08-19T14:15:19.658904+0000","last_change":"2024-08-19T14:15:18.660553+0000","last_active":"2024-08-19T14:15:19.658904+0000","last_peered":"2024-08-19T14:15:19.658904+0000","last_clean":"2024-08-19T14:15:19.658904+0000","last_became_active":"2024-08-19T14:15:18.660091+0000","last_became_peered":"2024-08-19T14:15:18.660091+0000","last_unstale":"2024-08-19T14:15:19.658904+0000","last_undegraded":"2024-08-19T14:15:19.658904+0000","last_fullsized":"2024-08-19T14:15:19.658904+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:14:48.488367+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:14:48.488367+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:16:10.475499+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"174'10930","reported_seq":17424,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-08-19T14:15:18.053696+0000","last_change":"2024-08-19T14:14:09.998977+0000","last_active":"2024-08-19T14:15:18.053696+0000","last_peered":"2024-08-19T14:15:18.053696+0000","last_clean":"2024-08-19T14:15:18.053696+0000","last_became_active":"2024-08-19T14:14:04.681118+0000","last_became_peered":"2024-08-19T14:14:04.681118+0000","last_unstale":"2024-08-19T14:15:18.053696+0000","last_undegraded":"2024-08-19T14:15:18.053696+0000","last_fullsized":"2024-08-19T14:15:18.053696+0000","mapping_epoch":146,"log_start":"107'900","ondisk_log_start":"107'900","created":102,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'7194","last_scrub_stamp":"2024-08-19T14:14:09.998849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:14:09.998849+0000","objects_scrubbed":7135,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:22.243440+0000","scrub_duration":2842,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":296091648,"num_objects":4518,"num_object_clones":0,"num_object_copies":13554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10930,"num_write_kb":494336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3123,"num_bytes_recovered":204668928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"70'139","reported_seq":413,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-08-19T14:15:17.637390+0000","last_change":"2024-08-19T14:15:13.428744+0000","last_active":"2024-08-19T14:15:17.637390+0000","last_peered":"2024-08-19T14:15:17.637390+0000","last_clean":"2024-08-19T14:15:17.637390+0000","last_became_active":"2024-08-19T14:13:11.647085+0000","last_became_peered":"2024-08-19T14:13:11.647085+0000","last_unstale":"2024-08-19T14:15:17.637390+0000","last_undegraded":"2024-08-19T14:15:17.637390+0000","last_fullsized":"2024-08-19T14:15:17.637390+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:15:13.428627+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:15:13.428627+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:16:19.387490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,4],"acting":[9,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.5","version":"176'10990","reported_seq":11233,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-08-19T14:15:19.691790+0000","last_change":"2024-08-19T14:14:14.222353+0000","last_active":"2024-08-19T14:15:19.691790+0000","last_peered":"2024-08-19T14:15:19.691790+0000","last_clean":"2024-08-19T14:15:19.691790+0000","last_became_active":"2024-08-19T14:12:43.417616+0000","last_became_peered":"2024-08-19T14:12:43.417616+0000","last_unstale":"2024-08-19T14:15:19.691790+0000","last_undegraded":"2024-08-19T14:15:19.691790+0000","last_fullsized":"2024-08-19T14:15:19.691790+0000","mapping_epoch":102,"log_start":"107'900","ondisk_log_start":"107'900","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"148'7426","last_scrub_stamp":"2024-08-19T14:14:14.222198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:14:14.222198+0000","objects_scrubbed":7364,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:35.064129+0000","scrub_duration":2125,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":292028416,"num_objects":4456,"num_object_clones":0,"num_object_copies":13368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4456,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10990,"num_write_kb":494272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.5","version":"70'139","reported_seq":418,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-08-19T14:15:19.674409+0000","last_change":"2024-08-19T14:14:49.752154+0000","last_active":"2024-08-19T14:15:19.674409+0000","last_peered":"2024-08-19T14:15:19.674409+0000","last_clean":"2024-08-19T14:15:19.674409+0000","last_became_active":"2024-08-19T14:13:22.727098+0000","last_became_peered":"2024-08-19T14:13:22.727098+0000","last_unstale":"2024-08-19T14:15:19.674409+0000","last_undegraded":"2024-08-19T14:15:19.674409+0000","last_fullsized":"2024-08-19T14:15:19.674409+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:14:49.752072+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:14:49.752072+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:16:05.519885+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"174'12016","reported_seq":12315,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-08-19T14:15:17.979666+0000","last_change":"2024-08-19T14:14:40.326683+0000","last_active":"2024-08-19T14:15:17.979666+0000","last_peered":"2024-08-19T14:15:17.979666+0000","last_clean":"2024-08-19T14:15:17.979666+0000","last_became_active":"2024-08-19T14:12:42.963203+0000","last_became_peered":"2024-08-19T14:12:42.963203+0000","last_unstale":"2024-08-19T14:15:17.979666+0000","last_undegraded":"2024-08-19T14:15:17.979666+0000","last_fullsized":"2024-08-19T14:15:17.979666+0000","mapping_epoch":102,"log_start":"110'2000","ondisk_log_start":"110'2000","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'10039","last_scrub_stamp":"2024-08-19T14:14:40.326519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:14:40.326519+0000","objects_scrubbed":6619,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:43.279737+0000","scrub_duration":3261,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":297336832,"num_objects":4537,"num_object_clones":0,"num_object_copies":13611,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10795,"num_write_kb":490624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,3],"acting":[1,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"70'139","reported_seq":468,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-08-19T14:15:19.666609+0000","last_change":"2024-08-19T14:14:36.831290+0000","last_active":"2024-08-19T14:15:19.666609+0000","last_peered":"2024-08-19T14:15:19.666609+0000","last_clean":"2024-08-19T14:15:19.666609+0000","last_became_active":"2024-08-19T14:13:05.607447+0000","last_became_peered":"2024-08-19T14:13:05.607447+0000","last_unstale":"2024-08-19T14:15:19.666609+0000","last_undegraded":"2024-08-19T14:15:19.666609+0000","last_fullsized":"2024-08-19T14:15:19.666609+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:14:36.831171+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:14:36.831171+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:53.437142+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.7","version":"176'10950","reported_seq":22497,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-08-19T14:15:21.076295+0000","last_change":"2024-08-19T14:15:13.583652+0000","last_active":"2024-08-19T14:15:21.076295+0000","last_peered":"2024-08-19T14:15:21.076295+0000","last_clean":"2024-08-19T14:15:21.076295+0000","last_became_active":"2024-08-19T14:14:49.991243+0000","last_became_peered":"2024-08-19T14:14:49.991243+0000","last_unstale":"2024-08-19T14:15:21.076295+0000","last_undegraded":"2024-08-19T14:15:21.076295+0000","last_fullsized":"2024-08-19T14:15:21.076295+0000","mapping_epoch":164,"log_start":"107'900","ondisk_log_start":"107'900","created":102,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"169'10693","last_scrub_stamp":"2024-08-19T14:15:13.583522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:15:13.583522+0000","objects_scrubbed":4764,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:16:37.611856+0000","scrub_duration":1668,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":292814848,"num_objects":4468,"num_object_clones":0,"num_object_copies":13404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4468,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10950,"num_write_kb":493376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4460,"num_bytes_recovered":253820928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.8","version":"174'10940","reported_seq":11232,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-08-19T14:15:17.958437+0000","last_change":"2024-08-19T14:15:13.054824+0000","last_active":"2024-08-19T14:15:17.958437+0000","last_peered":"2024-08-19T14:15:17.958437+0000","last_clean":"2024-08-19T14:15:17.958437+0000","last_became_active":"2024-08-19T14:12:43.038882+0000","last_became_peered":"2024-08-19T14:12:43.038882+0000","last_unstale":"2024-08-19T14:15:17.958437+0000","last_undegraded":"2024-08-19T14:15:17.958437+0000","last_fullsized":"2024-08-19T14:15:17.958437+0000","mapping_epoch":102,"log_start":"107'900","ondisk_log_start":"107'900","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"169'10792","last_scrub_stamp":"2024-08-19T14:15:13.054698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:15:13.054698+0000","objects_scrubbed":4745,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:16:31.701146+0000","scrub_duration":1525,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299368448,"num_objects":4568,"num_object_clones":0,"num_object_copies":13704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4568,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10940,"num_write_kb":496256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,0],"acting":[7,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"70'139","reported_seq":426,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-08-19T14:15:17.637341+0000","last_change":"2024-08-19T14:14:31.607909+0000","last_active":"2024-08-19T14:15:17.637341+0000","last_peered":"2024-08-19T14:15:17.637341+0000","last_clean":"2024-08-19T14:15:17.637341+0000","last_became_active":"2024-08-19T14:13:22.728779+0000","last_became_peered":"2024-08-19T14:13:22.728779+0000","last_unstale":"2024-08-19T14:15:17.637341+0000","last_undegraded":"2024-08-19T14:15:17.637341+0000","last_fullsized":"2024-08-19T14:15:17.637341+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:14:31.607795+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:14:31.607795+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:48.816381+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,0],"acting":[9,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"174'10732","reported_seq":10964,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-08-19T14:15:18.039032+0000","last_change":"2024-08-19T14:14:05.393154+0000","last_active":"2024-08-19T14:15:18.039032+0000","last_peered":"2024-08-19T14:15:18.039032+0000","last_clean":"2024-08-19T14:15:18.039032+0000","last_became_active":"2024-08-19T14:12:42.963114+0000","last_became_peered":"2024-08-19T14:12:42.963114+0000","last_unstale":"2024-08-19T14:15:18.039032+0000","last_undegraded":"2024-08-19T14:15:18.039032+0000","last_fullsized":"2024-08-19T14:15:18.039032+0000","mapping_epoch":102,"log_start":"107'700","ondisk_log_start":"107'700","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'6812","last_scrub_stamp":"2024-08-19T14:14:05.392787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:14:05.392787+0000","objects_scrubbed":6777,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1333,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":293208064,"num_objects":4474,"num_object_clones":0,"num_object_copies":13422,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10732,"num_write_kb":486592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"70'139","reported_seq":539,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-08-19T14:15:17.639910+0000","last_change":"2024-08-19T14:14:35.319452+0000","last_active":"2024-08-19T14:15:17.639910+0000","last_peered":"2024-08-19T14:15:17.639910+0000","last_clean":"2024-08-19T14:15:17.639910+0000","last_became_active":"2024-08-19T14:13:05.607605+0000","last_became_peered":"2024-08-19T14:13:05.607605+0000","last_unstale":"2024-08-19T14:15:17.639910+0000","last_undegraded":"2024-08-19T14:15:17.639910+0000","last_fullsized":"2024-08-19T14:15:17.639910+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:14:35.319333+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:14:35.319333+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:46.300019+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"176'11013","reported_seq":11253,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-08-19T14:15:20.534346+0000","last_change":"2024-08-19T14:14:11.437361+0000","last_active":"2024-08-19T14:15:20.534346+0000","last_peered":"2024-08-19T14:15:20.534346+0000","last_clean":"2024-08-19T14:15:20.534346+0000","last_became_active":"2024-08-19T14:12:42.963040+0000","last_became_peered":"2024-08-19T14:12:42.963040+0000","last_unstale":"2024-08-19T14:15:20.534346+0000","last_undegraded":"2024-08-19T14:15:20.534346+0000","last_fullsized":"2024-08-19T14:15:20.534346+0000","mapping_epoch":102,"log_start":"107'1000","ondisk_log_start":"107'1000","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'7289","last_scrub_stamp":"2024-08-19T14:14:11.437152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:14:11.437152+0000","objects_scrubbed":7227,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:32.239860+0000","scrub_duration":2564,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":289996800,"num_objects":4425,"num_object_clones":0,"num_object_copies":13275,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11013,"num_write_kb":494016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,9],"acting":[11,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.8","version":"70'139","reported_seq":421,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-08-19T14:15:19.658870+0000","last_change":"2024-08-19T14:14:35.594757+0000","last_active":"2024-08-19T14:15:19.658870+0000","last_peered":"2024-08-19T14:15:19.658870+0000","last_clean":"2024-08-19T14:15:19.658870+0000","last_became_active":"2024-08-19T14:13:05.605694+0000","last_became_peered":"2024-08-19T14:13:05.605694+0000","last_unstale":"2024-08-19T14:15:19.658870+0000","last_undegraded":"2024-08-19T14:15:19.658870+0000","last_fullsized":"2024-08-19T14:15:19.658870+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:14:35.594637+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:14:35.594637+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:15:43.962595+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.b","version":"176'10971","reported_seq":11208,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-08-19T14:15:19.691946+0000","last_change":"2024-08-19T14:14:06.266442+0000","last_active":"2024-08-19T14:15:19.691946+0000","last_peered":"2024-08-19T14:15:19.691946+0000","last_clean":"2024-08-19T14:15:19.691946+0000","last_became_active":"2024-08-19T14:12:42.729014+0000","last_became_peered":"2024-08-19T14:12:42.729014+0000","last_unstale":"2024-08-19T14:15:19.691946+0000","last_undegraded":"2024-08-19T14:15:19.691946+0000","last_fullsized":"2024-08-19T14:15:19.691946+0000","mapping_epoch":102,"log_start":"107'900","ondisk_log_start":"107'900","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'6965","last_scrub_stamp":"2024-08-19T14:14:06.266214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:14:06.266214+0000","objects_scrubbed":6919,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1296,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":288948224,"num_objects":4409,"num_object_clones":0,"num_object_copies":13227,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10971,"num_write_kb":492160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":4669112344,"num_objects":71246,"num_object_clones":0,"num_object_copies":213738,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7687,"num_objects_unfound":0,"num_objects_dirty":71246,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":175312,"num_write_kb":7889793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15570,"num_bytes_recovered":981925888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14592323584,"data_stored":14592311368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160633,"ondisk_log_size":160633,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1529,"ondisk_log_size":1529,"up":33,"acting":33,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187504,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1234428,"kb_used_data":1140784,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":92510724,"statfs":{"total":95995035648,"available":94730981376,"internally_reserved":0,"allocated":1168162816,"data_stored":1167284865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":95877572},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":343597383732,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":74,"seq":317827579961,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2185368,"kb_used_data":2055632,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":91559784,"statfs":{"total":95995035648,"available":93757218816,"internally_reserved":0,"allocated":2104967168,"data_stored":2104097918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":132839873},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874303,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1230644,"kb_used_data":1167524,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":92514508,"statfs":{"total":95995035648,"available":94734856192,"internally_reserved":0,"allocated":1195544576,"data_stored":1194670895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":57,"seq":244813135937,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1566100,"kb_used_data":1458572,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":92179052,"statfs":{"total":95995035648,"available":94391349248,"internally_reserved":0,"allocated":1493577728,"data_stored":1492712550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":110098890},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":206158430277,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691914,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":617540,"kb_used_data":564152,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93127612,"statfs":{"total":95995035648,"available":95362674688,"internally_reserved":0,"allocated":577691648,"data_stored":576810145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986254,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1269400,"kb_used_data":1175760,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":92475752,"statfs":{"total":95995035648,"available":94695170048,"internally_reserved":0,"allocated":1203978240,"data_stored":1203108990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":95877573},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149780,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2164468,"kb_used_data":2020836,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":91580684,"statfs":{"total":95995035648,"available":93778620416,"internally_reserved":0,"allocated":2069336064,"data_stored":2068459021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":147061191},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411417,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1503812,"kb_used_data":1404600,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":92241340,"statfs":{"total":95995035648,"available":94455132160,"internally_reserved":0,"allocated":1438310400,"data_stored":1437428897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574941,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1562072,"kb_used_data":1462860,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":92183080,"statfs":{"total":95995035648,"available":94395473920,"internally_reserved":0,"allocated":1497968640,"data_stored":1497103462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738466,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1500800,"kb_used_data":1411248,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":92244352,"statfs":{"total":95995035648,"available":94458216448,"internally_reserved":0,"allocated":1445117952,"data_stored":1444240001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91617733},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1442578432,"data_stored":1442578432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1483866112,"data_stored":1483866112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1432158208,"data_stored":1432158208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2059145216,"data_stored":2059141144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1196625920,"data_stored":1196621848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":569114624,"data_stored":569114624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":289210368,"data_stored":289210368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1487536128,"data_stored":1487536128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1191510016,"data_stored":1191510016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":2084114432,"data_stored":2084110360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":198115328,"data_stored":198115328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1158348800,"data_stored":1158348800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T14:15:23.306 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:15:23.330 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:23.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:23.331 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:15:23.404 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:15:23.432 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:15:23.456 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:23.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:23.457 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:15:23.558 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:15:23.582 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:23.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:23.583 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:15:23.684 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:15:23.710 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:23.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:23.715 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:15:23.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:23 smithi032 ceph-mon[20004]: pgmap v440: 27 pgs: 2 active+remapped+backfilling, 25 active+clean; 4.3 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 542 op/s; 7687/213744 objects misplaced (3.596%); 13 MiB/s, 207 objects/s recovering 2024-08-19T14:15:23.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:23 smithi032 ceph-mon[20004]: from='client.24893 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:15:23.816 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:15:23.823 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:23 smithi031 ceph-mon[16336]: pgmap v440: 27 pgs: 2 active+remapped+backfilling, 25 active+clean; 4.3 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 542 op/s; 7687/213744 objects misplaced (3.596%); 13 MiB/s, 207 objects/s recovering 2024-08-19T14:15:23.823 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:23 smithi031 ceph-mon[16336]: from='client.24893 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:15:23.841 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:23.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:23.842 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:15:23.854 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-08-19T14:15:23.943 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:15:24.007 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:24.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:24.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:15:24.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:23 smithi045 ceph-mon[20143]: pgmap v440: 27 pgs: 2 active+remapped+backfilling, 25 active+clean; 4.3 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 542 op/s; 7687/213744 objects misplaced (3.596%); 13 MiB/s, 207 objects/s recovering 2024-08-19T14:15:24.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:23 smithi045 ceph-mon[20143]: from='client.24893 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:15:24.109 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:15:24.165 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:24.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:24.166 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:15:24.172 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:15:24.267 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:15:24.290 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:24.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:24.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:15:24.392 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:15:24.415 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:24.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:24.416 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:15:24.517 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:15:24.566 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:15:24.567 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:24.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:24.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:15:24.669 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:15:24.694 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:24.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:24.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:15:24.725 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is set 2024-08-19T14:15:24.795 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:15:24.822 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:24.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:24.823 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:15:24.830 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:15:24.830 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-08-19T14:15:24.830 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-08-19T14:15:24.830 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-08-19T14:15:24.830 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:15:24.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:24 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3586889202' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:15:24.925 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:15:24.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:24.977 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:24.977 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:15:25.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:24 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3586889202' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:15:25.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:24 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3586889202' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:15:25.079 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:15:25.106 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:25.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:25.107 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:15:25.208 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:15:25.245 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:25.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:25.246 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:15:25.350 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:15:25.376 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:25.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:25.377 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:15:25.477 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:15:25.530 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:15:25.543 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-08-19T14:15:25.544 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:25.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:25.545 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:15:25.645 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:15:25.672 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:25.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:25.673 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:15:25.774 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:15:25.817 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:25.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:25.818 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:15:25.906 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:15:25.919 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:15:25.948 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:25.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:25.949 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:15:25.978 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:25 smithi031 ceph-mon[16336]: pgmap v441: 27 pgs: 2 active+remapped+backfilling, 25 active+clean; 4.3 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 715 op/s; 7687/208908 objects misplaced (3.680%); 11 MiB/s, 172 objects/s recovering 2024-08-19T14:15:25.978 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:25 smithi031 ceph-mon[16336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:15:25.978 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:25 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3586889202' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:15:25.978 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:25 smithi031 ceph-mon[16336]: osdmap e177: 12 total, 12 up, 10 in 2024-08-19T14:15:26.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:25 smithi032 ceph-mon[20004]: pgmap v441: 27 pgs: 2 active+remapped+backfilling, 25 active+clean; 4.3 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 715 op/s; 7687/208908 objects misplaced (3.680%); 11 MiB/s, 172 objects/s recovering 2024-08-19T14:15:26.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:25 smithi032 ceph-mon[20004]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:15:26.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:25 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3586889202' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:15:26.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:25 smithi032 ceph-mon[20004]: osdmap e177: 12 total, 12 up, 10 in 2024-08-19T14:15:26.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:25 smithi045 ceph-mon[20143]: pgmap v441: 27 pgs: 2 active+remapped+backfilling, 25 active+clean; 4.3 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 715 op/s; 7687/208908 objects misplaced (3.680%); 11 MiB/s, 172 objects/s recovering 2024-08-19T14:15:26.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:25 smithi045 ceph-mon[20143]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:15:26.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:25 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3586889202' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:15:26.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:25 smithi045 ceph-mon[20143]: osdmap e177: 12 total, 12 up, 10 in 2024-08-19T14:15:26.051 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:15:26.083 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:26.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:26.084 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:15:26.160 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:15:26.160 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:15:26.184 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:15:26.195 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:15:26.196 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:15:26.211 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:26.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:26.212 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:15:26.227 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:15:26.227 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:15:26.266 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:15:26.266 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:15:26.295 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:15:26.296 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:15:26.313 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:15:26.331 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:15:26.331 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:15:26.341 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:26.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:26.342 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:15:26.364 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:15:26.364 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:15:26.404 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:15:26.405 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:15:26.436 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:15:26.436 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:15:26.443 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:15:26.468 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:26.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:26.469 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:15:26.472 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:15:26.472 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:15:26.513 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:15:26.513 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:15:26.545 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:15:26.545 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:15:26.563 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:15:26.563 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:15:26.570 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:15:26.599 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:26.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:26.600 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:15:26.701 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:15:26.732 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:26.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:26.735 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:15:26.836 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:15:26.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:26 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2919955115' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:15:26.876 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:26.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:26.878 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:15:26.982 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:15:27.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:26 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2919955115' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:15:27.020 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:27.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:27.021 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:15:27.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:26 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2919955115' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:15:27.122 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:15:27.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:27.151 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:27.151 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:15:27.252 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:15:27.289 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:27.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:27.290 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:15:27.314 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-08-19T14:15:27.390 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:15:27.434 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-19T14:15:27.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:27.436 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:27.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:15:27.537 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:15:27.567 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:27.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:27.568 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:15:27.656 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:15:27.670 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:15:27.703 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:27.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:27.704 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:15:27.806 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:15:27.830 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:27.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:27.831 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:15:27.884 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:15:27.932 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:15:27.956 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:27 smithi032 ceph-mon[20004]: pgmap v443: 27 pgs: 2 active+remapped+backfilling, 25 active+clean; 4.2 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 683 op/s; 7687/204813 objects misplaced (3.753%); 9.3 MiB/s, 148 objects/s recovering 2024-08-19T14:15:27.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:27.965 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:27.965 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:15:28.030 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:27 smithi031 ceph-mon[16336]: pgmap v443: 27 pgs: 2 active+remapped+backfilling, 25 active+clean; 4.2 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 683 op/s; 7687/204813 objects misplaced (3.753%); 9.3 MiB/s, 148 objects/s recovering 2024-08-19T14:15:28.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:27 smithi045 ceph-mon[20143]: pgmap v443: 27 pgs: 2 active+remapped+backfilling, 25 active+clean; 4.2 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 683 op/s; 7687/204813 objects misplaced (3.753%); 9.3 MiB/s, 148 objects/s recovering 2024-08-19T14:15:28.066 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:15:28.109 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:28.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:28.110 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:15:28.211 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:15:28.235 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 8, 9, 11] out_osds: [10, 6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-08-19T14:15:28.235 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-19T14:15:28.235 INFO:tasks.thrashosds.thrasher:inject_pause on osd.10 2024-08-19T14:15:28.235 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-08-19T14:15:28.235 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-08-19T14:15:28.235 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.10 config set filestore_inject_stall 3 2024-08-19T14:15:28.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:28.237 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:28.237 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:15:28.311 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:15:28.311 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:15:28.338 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:15:28.364 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:28.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:28.365 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:15:28.466 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:15:28.499 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:28.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:28.500 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:15:28.588 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:15:28.601 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:15:28.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:28.627 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:28.627 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:15:28.728 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:15:28.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:28.771 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:28.771 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:15:28.872 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:15:28.896 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:28.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:28.897 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:15:28.999 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:15:29.026 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-08-19T14:15:29.030 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:29.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:29.031 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:15:29.130 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:15:29.131 INFO:teuthology.orchestra.run.smithi045.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-08-19T14:15:29.131 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:15:29.132 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:15:29.159 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:29.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:29.160 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:15:29.261 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:15:29.293 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:29.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:29.294 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:15:29.326 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:15:29.360 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:15:29.395 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:15:29.422 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:29.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:29.423 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:15:29.524 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:15:29.552 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:29.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:29.553 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:15:29.655 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:15:29.714 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:29.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:29.715 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:15:29.750 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is unset 2024-08-19T14:15:29.816 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:15:29.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:29.842 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:29.842 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:15:29.943 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:15:29.973 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:29.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:29.974 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:15:30.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:29 smithi032 ceph-mon[20004]: pgmap v444: 27 pgs: 2 active+remapped+backfilling, 25 active+clean; 4.1 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 770 op/s; 5918/200694 objects misplaced (2.949%); 18 MiB/s, 294 objects/s recovering 2024-08-19T14:15:30.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:29 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2218725619' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:15:30.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:29 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:15:30.015 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:15:30.015 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:15:30.035 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:15:30.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:29 smithi045 ceph-mon[20143]: pgmap v444: 27 pgs: 2 active+remapped+backfilling, 25 active+clean; 4.1 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 770 op/s; 5918/200694 objects misplaced (2.949%); 18 MiB/s, 294 objects/s recovering 2024-08-19T14:15:30.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:29 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2218725619' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:15:30.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:29 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:15:30.075 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:15:30.099 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:30.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:30.101 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:15:30.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:29 smithi031 ceph-mon[16336]: pgmap v444: 27 pgs: 2 active+remapped+backfilling, 25 active+clean; 4.1 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 770 op/s; 5918/200694 objects misplaced (2.949%); 18 MiB/s, 294 objects/s recovering 2024-08-19T14:15:30.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:29 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2218725619' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:15:30.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:29 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:15:30.202 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:15:30.237 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:30.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:30.238 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:15:30.338 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:15:30.373 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:30.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:30.374 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:15:30.475 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:15:30.500 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:30.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:30.501 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:15:30.602 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:15:30.632 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:30.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:30.633 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:15:30.662 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-08-19T14:15:30.734 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:15:30.781 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:30.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:30.782 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:15:30.883 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:15:30.921 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:30.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:30.922 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:15:31.003 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:15:31.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:30 smithi032 ceph-mon[20004]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:15:31.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:30 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:15:31.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:30 smithi032 ceph-mon[20004]: osdmap e178: 12 total, 12 up, 10 in 2024-08-19T14:15:31.019 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:30 smithi031 ceph-mon[16336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:15:31.019 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:30 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:15:31.019 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:30 smithi031 ceph-mon[16336]: osdmap e178: 12 total, 12 up, 10 in 2024-08-19T14:15:31.024 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:15:31.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:30 smithi045 ceph-mon[20143]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:15:31.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:30 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:15:31.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:30 smithi045 ceph-mon[20143]: osdmap e178: 12 total, 12 up, 10 in 2024-08-19T14:15:31.052 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:31.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:31.053 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:15:31.080 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:15:31.081 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:15:31.121 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:15:31.122 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:15:31.154 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:15:31.178 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:31.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:31.179 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:15:31.194 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:15:31.194 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:15:31.228 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:15:31.228 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:15:31.259 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:15:31.259 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:15:31.280 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:15:31.290 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:15:31.290 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:15:31.316 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:31.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:31.317 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:15:31.327 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:15:31.327 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:15:31.362 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:15:31.363 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:15:31.396 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:15:31.396 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:15:31.419 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:15:31.428 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:15:31.429 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:15:31.443 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:31.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:31.444 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:15:31.459 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:15:31.459 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:15:31.494 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:15:31.494 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:15:31.546 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:15:31.571 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:31.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:31.572 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:15:31.673 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:15:31.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:31.704 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:31.704 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:15:31.744 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:15:31.744 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-08-19T14:15:31.745 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-08-19T14:15:31.745 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-08-19T14:15:31.745 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.745 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:63160 2.0 2:08542b0e:::benchmark_data_smithi045_46415_object63159:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-08-19T14:15:31.745 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:20.958791+0000", 2024-08-19T14:15:31.745 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 130.782373987, 2024-08-19T14:15:31.745 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.21296446799999999, 2024-08-19T14:15:31.745 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:15:31.745 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:15:31.745 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:15:31.745 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:15:31.745 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:15:31.745 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:15:31.746 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 63160 2024-08-19T14:15:31.746 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.746 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:15:31.746 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.746 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:15:31.746 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.958791+0000", 2024-08-19T14:15:31.746 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:31.746 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.746 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.746 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:15:31.747 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.958791+0000", 2024-08-19T14:15:31.747 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:31.747 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.747 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.747 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:15:31.747 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.958792+0000", 2024-08-19T14:15:31.747 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.514e-06 2024-08-19T14:15:31.747 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.747 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.747 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:15:31.747 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.958829+0000", 2024-08-19T14:15:31.747 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6594999999999998e-05 2024-08-19T14:15:31.748 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.748 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.748 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:15:31.748 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.958845+0000", 2024-08-19T14:15:31.748 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6571000000000001e-05 2024-08-19T14:15:31.748 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.748 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.748 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:15:31.748 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.958854+0000", 2024-08-19T14:15:31.748 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.4810000000000006e-06 2024-08-19T14:15:31.748 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.748 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.748 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:15:31.749 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.960591+0000", 2024-08-19T14:15:31.749 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0017371929999999999 2024-08-19T14:15:31.749 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.749 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.749 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:15:31.749 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.960793+0000", 2024-08-19T14:15:31.749 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00020158200000000001 2024-08-19T14:15:31.749 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.749 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.749 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 9,11", 2024-08-19T14:15:31.749 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.961006+0000", 2024-08-19T14:15:31.749 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000213169 2024-08-19T14:15:31.749 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.750 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.750 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:15:31.750 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.962896+0000", 2024-08-19T14:15:31.750 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0018897460000000001 2024-08-19T14:15:31.750 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.750 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.750 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:31.750 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.965630+0000", 2024-08-19T14:15:31.750 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.002733961 2024-08-19T14:15:31.750 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.750 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.751 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:31.751 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.170985+0000", 2024-08-19T14:15:31.751 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.205355648 2024-08-19T14:15:31.751 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.751 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.751 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:15:31.751 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.171055+0000", 2024-08-19T14:15:31.751 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9776000000000004e-05 2024-08-19T14:15:31.751 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.751 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.751 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:15:31.751 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.171755+0000", 2024-08-19T14:15:31.751 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00070023199999999996 2024-08-19T14:15:31.752 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:31.752 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:15:31.752 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:31.752 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.752 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.752 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:63163 2.8 2:1e3c608e:::benchmark_data_smithi045_46415_object63162:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-08-19T14:15:31.752 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:20.961739+0000", 2024-08-19T14:15:31.752 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 130.77942562499999, 2024-08-19T14:15:31.752 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.209352172, 2024-08-19T14:15:31.752 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:15:31.752 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:15:31.752 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:15:31.752 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:15:31.753 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:15:31.753 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:15:31.753 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 63163 2024-08-19T14:15:31.753 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.753 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:15:31.753 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.753 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:15:31.753 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.961739+0000", 2024-08-19T14:15:31.753 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:31.753 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.753 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.753 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:15:31.753 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.961739+0000", 2024-08-19T14:15:31.754 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:31.754 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.754 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.754 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:15:31.754 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.961740+0000", 2024-08-19T14:15:31.754 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2049999999999999e-06 2024-08-19T14:15:31.754 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.754 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.754 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:15:31.754 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.961818+0000", 2024-08-19T14:15:31.754 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7275999999999997e-05 2024-08-19T14:15:31.754 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.754 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.755 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:15:31.755 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.961828+0000", 2024-08-19T14:15:31.755 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0407999999999999e-05 2024-08-19T14:15:31.755 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.755 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.755 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:15:31.755 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.961835+0000", 2024-08-19T14:15:31.755 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5089999999999996e-06 2024-08-19T14:15:31.755 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.755 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.755 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:15:31.755 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.961891+0000", 2024-08-19T14:15:31.755 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5785000000000002e-05 2024-08-19T14:15:31.756 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.756 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.756 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:15:31.756 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.962014+0000", 2024-08-19T14:15:31.756 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012311200000000001 2024-08-19T14:15:31.756 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.756 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.756 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0,9", 2024-08-19T14:15:31.756 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.962141+0000", 2024-08-19T14:15:31.756 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012648 2024-08-19T14:15:31.756 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.756 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.756 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:31.757 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.966506+0000", 2024-08-19T14:15:31.757 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0043654719999999996 2024-08-19T14:15:31.757 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.757 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.757 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:15:31.757 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.968079+0000", 2024-08-19T14:15:31.757 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001572316 2024-08-19T14:15:31.757 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.757 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.757 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:31.757 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.170964+0000", 2024-08-19T14:15:31.757 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20288556799999999 2024-08-19T14:15:31.757 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.757 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.758 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:15:31.758 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.171048+0000", 2024-08-19T14:15:31.758 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.3412999999999996e-05 2024-08-19T14:15:31.758 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.758 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.758 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:15:31.758 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.171091+0000", 2024-08-19T14:15:31.758 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3628000000000003e-05 2024-08-19T14:15:31.758 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:31.758 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:15:31.758 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:31.758 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.758 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.759 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:63175 2.0 2:031c8466:::benchmark_data_smithi045_46415_object63174:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-08-19T14:15:31.759 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:20.969904+0000", 2024-08-19T14:15:31.759 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 130.77126114500001, 2024-08-19T14:15:31.759 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20268109100000001, 2024-08-19T14:15:31.759 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:15:31.759 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:15:31.759 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:15:31.759 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:15:31.759 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:15:31.759 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:15:31.759 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 63175 2024-08-19T14:15:31.759 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.759 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:15:31.760 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.760 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:15:31.760 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.969904+0000", 2024-08-19T14:15:31.760 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:31.760 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.760 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.760 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:15:31.760 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.969904+0000", 2024-08-19T14:15:31.760 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:31.760 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.760 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.761 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:15:31.761 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.969905+0000", 2024-08-19T14:15:31.761 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7969999999999999e-06 2024-08-19T14:15:31.761 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.761 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.761 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:15:31.761 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.969970+0000", 2024-08-19T14:15:31.761 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4850000000000004e-05 2024-08-19T14:15:31.761 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.762 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.762 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:15:31.762 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.969995+0000", 2024-08-19T14:15:31.762 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5168999999999999e-05 2024-08-19T14:15:31.762 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.762 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.762 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:15:31.762 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.970003+0000", 2024-08-19T14:15:31.762 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3939999999999998e-06 2024-08-19T14:15:31.762 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.762 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.762 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:15:31.762 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.971722+0000", 2024-08-19T14:15:31.763 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0017196220000000001 2024-08-19T14:15:31.763 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.763 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.763 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:15:31.763 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.971840+0000", 2024-08-19T14:15:31.763 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000117261 2024-08-19T14:15:31.763 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.763 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.763 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 9,11", 2024-08-19T14:15:31.763 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.972030+0000", 2024-08-19T14:15:31.763 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00018997799999999999 2024-08-19T14:15:31.763 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.763 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.763 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:31.764 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.976418+0000", 2024-08-19T14:15:31.764 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0043884140000000002 2024-08-19T14:15:31.764 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.764 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.764 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:15:31.764 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.976439+0000", 2024-08-19T14:15:31.764 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0979999999999999e-05 2024-08-19T14:15:31.764 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.764 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.764 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:31.764 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.171818+0000", 2024-08-19T14:15:31.764 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.19537907500000001 2024-08-19T14:15:31.764 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.765 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.765 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:15:31.765 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.171880+0000", 2024-08-19T14:15:31.765 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1791999999999994e-05 2024-08-19T14:15:31.765 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.765 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.766 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:15:31.766 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.172585+0000", 2024-08-19T14:15:31.766 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00070475900000000001 2024-08-19T14:15:31.766 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:31.766 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:15:31.766 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:31.766 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.767 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.767 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:63177 2.8 2:15a25ebc:::benchmark_data_smithi045_46415_object63176:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-08-19T14:15:31.767 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:20.970729+0000", 2024-08-19T14:15:31.767 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 130.770436117, 2024-08-19T14:15:31.767 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.200497493, 2024-08-19T14:15:31.767 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:15:31.767 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:15:31.767 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:15:31.767 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:15:31.767 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:15:31.767 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:15:31.767 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 63177 2024-08-19T14:15:31.767 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.768 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:15:31.768 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.768 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:15:31.768 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.970729+0000", 2024-08-19T14:15:31.768 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:31.768 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.768 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.768 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:15:31.768 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.970729+0000", 2024-08-19T14:15:31.769 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:31.769 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.769 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.769 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:15:31.769 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.970730+0000", 2024-08-19T14:15:31.769 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.02e-06 2024-08-19T14:15:31.769 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.769 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.769 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:15:31.770 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.970816+0000", 2024-08-19T14:15:31.770 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6040999999999998e-05 2024-08-19T14:15:31.770 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.770 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.770 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:15:31.770 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.970826+0000", 2024-08-19T14:15:31.770 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0361e-05 2024-08-19T14:15:31.770 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.770 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.770 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:15:31.770 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.970831+0000", 2024-08-19T14:15:31.771 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9459999999999997e-06 2024-08-19T14:15:31.771 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.771 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.771 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:15:31.771 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.971716+0000", 2024-08-19T14:15:31.771 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00088531199999999997 2024-08-19T14:15:31.771 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.771 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.771 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:15:31.771 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.971839+0000", 2024-08-19T14:15:31.771 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012251499999999999 2024-08-19T14:15:31.771 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.771 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.771 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0,9", 2024-08-19T14:15:31.772 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.972011+0000", 2024-08-19T14:15:31.772 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000172244 2024-08-19T14:15:31.772 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.772 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.772 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:31.772 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.975032+0000", 2024-08-19T14:15:31.772 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.003021195 2024-08-19T14:15:31.772 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.772 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.772 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:15:31.773 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.976447+0000", 2024-08-19T14:15:31.773 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001415152 2024-08-19T14:15:31.773 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.773 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.773 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:31.773 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.171136+0000", 2024-08-19T14:15:31.773 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.19468817199999999 2024-08-19T14:15:31.773 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.773 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.773 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:15:31.773 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.171189+0000", 2024-08-19T14:15:31.773 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3505999999999999e-05 2024-08-19T14:15:31.773 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.773 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.773 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:15:31.774 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.171226+0000", 2024-08-19T14:15:31.774 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7029000000000001e-05 2024-08-19T14:15:31.774 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:31.774 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:15:31.774 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:31.774 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.774 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.774 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:63179 2.0 2:034b6ab5:::benchmark_data_smithi045_46415_object63178:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-08-19T14:15:31.774 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:20.971796+0000", 2024-08-19T14:15:31.774 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 130.76936912799999, 2024-08-19T14:15:31.774 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20158538500000001, 2024-08-19T14:15:31.774 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:15:31.774 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:15:31.775 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:15:31.775 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:15:31.775 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:15:31.775 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:15:31.775 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 63179 2024-08-19T14:15:31.775 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.775 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:15:31.775 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.775 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:15:31.775 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.971796+0000", 2024-08-19T14:15:31.775 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:31.775 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.775 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.775 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:15:31.775 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.971796+0000", 2024-08-19T14:15:31.776 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:31.776 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.777 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.777 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:15:31.777 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.971796+0000", 2024-08-19T14:15:31.777 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.2399999999999996e-07 2024-08-19T14:15:31.777 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.777 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.777 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:15:31.777 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.971857+0000", 2024-08-19T14:15:31.777 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0863000000000003e-05 2024-08-19T14:15:31.777 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.778 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.778 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:15:31.778 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.971867+0000", 2024-08-19T14:15:31.778 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.5619999999999993e-06 2024-08-19T14:15:31.778 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.778 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.778 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:15:31.778 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.971874+0000", 2024-08-19T14:15:31.778 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.5989999999999998e-06 2024-08-19T14:15:31.779 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.779 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.779 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:15:31.779 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.973576+0000", 2024-08-19T14:15:31.779 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001702928 2024-08-19T14:15:31.779 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.779 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.779 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:15:31.779 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.973668+0000", 2024-08-19T14:15:31.779 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.1094000000000006e-05 2024-08-19T14:15:31.779 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.779 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.779 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 9,11", 2024-08-19T14:15:31.779 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.973776+0000", 2024-08-19T14:15:31.780 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000108453 2024-08-19T14:15:31.780 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.780 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.780 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:31.780 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.976463+0000", 2024-08-19T14:15:31.780 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0026873999999999999 2024-08-19T14:15:31.780 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.780 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.780 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:15:31.780 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.979780+0000", 2024-08-19T14:15:31.780 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0033170740000000002 2024-08-19T14:15:31.780 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.780 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.780 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:31.781 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.172645+0000", 2024-08-19T14:15:31.781 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.192864063 2024-08-19T14:15:31.781 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.781 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.781 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:15:31.781 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.172698+0000", 2024-08-19T14:15:31.781 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3647000000000003e-05 2024-08-19T14:15:31.781 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.781 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.781 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:15:31.781 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.173381+0000", 2024-08-19T14:15:31.781 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00068277799999999999 2024-08-19T14:15:31.781 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:31.781 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:15:31.782 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:31.782 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.782 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.782 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:63180 2.0 2:09d38ed3:::benchmark_data_smithi045_46415_object63179:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-08-19T14:15:31.782 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:20.971911+0000", 2024-08-19T14:15:31.782 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 130.76925345999999, 2024-08-19T14:15:31.782 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20210957399999999, 2024-08-19T14:15:31.782 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:15:31.782 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:15:31.782 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:15:31.782 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:15:31.782 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:15:31.782 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:15:31.783 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 63180 2024-08-19T14:15:31.783 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.783 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:15:31.783 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.783 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:15:31.783 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.971911+0000", 2024-08-19T14:15:31.783 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:31.783 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.783 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.783 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:15:31.783 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.971911+0000", 2024-08-19T14:15:31.783 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:31.783 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.783 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.784 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:15:31.784 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.971915+0000", 2024-08-19T14:15:31.784 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.07e-06 2024-08-19T14:15:31.784 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.784 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.784 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:15:31.784 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.971998+0000", 2024-08-19T14:15:31.784 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.2995000000000001e-05 2024-08-19T14:15:31.784 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.784 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.784 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:15:31.784 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.972011+0000", 2024-08-19T14:15:31.785 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2262999999999999e-05 2024-08-19T14:15:31.785 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.785 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.785 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:15:31.785 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.972017+0000", 2024-08-19T14:15:31.785 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4400000000000002e-06 2024-08-19T14:15:31.785 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.785 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.785 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:15:31.785 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.975045+0000", 2024-08-19T14:15:31.785 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0030280440000000001 2024-08-19T14:15:31.785 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.785 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.786 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:15:31.786 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.975155+0000", 2024-08-19T14:15:31.786 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00010984699999999999 2024-08-19T14:15:31.786 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.786 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.786 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 9,11", 2024-08-19T14:15:31.786 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.975247+0000", 2024-08-19T14:15:31.786 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.1947999999999994e-05 2024-08-19T14:15:31.786 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.786 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.786 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:31.786 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.979753+0000", 2024-08-19T14:15:31.786 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0045062119999999999 2024-08-19T14:15:31.786 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.787 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.787 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:15:31.787 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.979791+0000", 2024-08-19T14:15:31.787 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7511000000000001e-05 2024-08-19T14:15:31.787 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.787 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.787 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:31.787 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.173451+0000", 2024-08-19T14:15:31.787 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.19366074699999999 2024-08-19T14:15:31.787 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.787 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.787 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:15:31.787 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.173513+0000", 2024-08-19T14:15:31.787 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1480000000000001e-05 2024-08-19T14:15:31.788 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.788 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.788 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:15:31.788 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.174021+0000", 2024-08-19T14:15:31.788 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00050801700000000004 2024-08-19T14:15:31.788 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:31.788 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:15:31.788 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:31.788 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.788 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.788 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:63181 2.8 2:10ca0581:::benchmark_data_smithi045_46415_object63180:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-08-19T14:15:31.788 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:20.972411+0000", 2024-08-19T14:15:31.788 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 130.768754159, 2024-08-19T14:15:31.788 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20026014, 2024-08-19T14:15:31.788 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:15:31.789 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:15:31.789 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:15:31.789 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:15:31.789 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:15:31.789 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:15:31.789 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 63181 2024-08-19T14:15:31.789 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.789 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:15:31.789 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.789 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:15:31.789 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.972411+0000", 2024-08-19T14:15:31.789 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:31.789 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.790 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.790 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:15:31.790 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.972411+0000", 2024-08-19T14:15:31.790 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:31.790 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.790 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.790 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:15:31.790 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.972411+0000", 2024-08-19T14:15:31.790 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.3500000000000005e-07 2024-08-19T14:15:31.790 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.790 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.790 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:15:31.790 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.972473+0000", 2024-08-19T14:15:31.790 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1877000000000004e-05 2024-08-19T14:15:31.791 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.791 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.791 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:15:31.791 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.972485+0000", 2024-08-19T14:15:31.791 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1620999999999999e-05 2024-08-19T14:15:31.791 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.791 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.791 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:15:31.791 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.972492+0000", 2024-08-19T14:15:31.791 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9840000000000004e-06 2024-08-19T14:15:31.791 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.792 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.792 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:15:31.792 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.976487+0000", 2024-08-19T14:15:31.792 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0039953300000000001 2024-08-19T14:15:31.792 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.792 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.792 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:15:31.792 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.976598+0000", 2024-08-19T14:15:31.792 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000111099 2024-08-19T14:15:31.792 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.792 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.792 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0,9", 2024-08-19T14:15:31.793 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.976732+0000", 2024-08-19T14:15:31.793 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000134029 2024-08-19T14:15:31.793 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.793 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.793 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:15:31.793 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.979797+0000", 2024-08-19T14:15:31.793 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0030645109999999998 2024-08-19T14:15:31.793 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.793 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.793 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:31.793 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.979913+0000", 2024-08-19T14:15:31.793 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000116496 2024-08-19T14:15:31.793 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.793 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.794 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:31.794 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.172616+0000", 2024-08-19T14:15:31.794 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.19270316600000001 2024-08-19T14:15:31.794 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.794 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.794 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:15:31.794 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.172649+0000", 2024-08-19T14:15:31.794 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2589000000000003e-05 2024-08-19T14:15:31.794 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.794 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.794 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:15:31.794 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.172671+0000", 2024-08-19T14:15:31.794 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1603e-05 2024-08-19T14:15:31.795 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:31.795 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:15:31.795 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:31.795 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.795 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.795 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:63205 2.8 2:16572d7a:::benchmark_data_smithi045_46415_object63204:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-08-19T14:15:31.795 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:21.002885+0000", 2024-08-19T14:15:31.795 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 130.73828003099999, 2024-08-19T14:15:31.795 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.17128316199999999, 2024-08-19T14:15:31.795 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:15:31.795 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:15:31.795 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:15:31.795 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:15:31.795 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:15:31.795 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:15:31.796 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 63205 2024-08-19T14:15:31.796 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.796 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:15:31.796 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.796 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:15:31.796 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.002885+0000", 2024-08-19T14:15:31.796 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:31.796 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.796 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.796 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:15:31.796 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.002885+0000", 2024-08-19T14:15:31.796 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:31.796 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.796 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.797 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:15:31.797 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.002887+0000", 2024-08-19T14:15:31.797 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2139999999999999e-06 2024-08-19T14:15:31.797 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.797 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.797 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:15:31.797 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.002928+0000", 2024-08-19T14:15:31.797 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0703e-05 2024-08-19T14:15:31.797 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.797 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.797 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:15:31.797 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.002940+0000", 2024-08-19T14:15:31.797 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2831e-05 2024-08-19T14:15:31.797 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.798 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.798 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:15:31.798 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.002948+0000", 2024-08-19T14:15:31.798 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7740000000000004e-06 2024-08-19T14:15:31.798 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.798 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.798 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:15:31.798 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.003000+0000", 2024-08-19T14:15:31.799 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1483000000000002e-05 2024-08-19T14:15:31.799 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.799 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.799 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:15:31.799 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.003090+0000", 2024-08-19T14:15:31.800 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.0606000000000006e-05 2024-08-19T14:15:31.800 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.800 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.800 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0,9", 2024-08-19T14:15:31.800 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.003180+0000", 2024-08-19T14:15:31.801 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.9977999999999998e-05 2024-08-19T14:15:31.801 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.801 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.801 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:31.801 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.005559+0000", 2024-08-19T14:15:31.801 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0023788870000000001 2024-08-19T14:15:31.802 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.802 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.802 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:15:31.802 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.005623+0000", 2024-08-19T14:15:31.803 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4331000000000005e-05 2024-08-19T14:15:31.803 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.803 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.803 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:31.803 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.174071+0000", 2024-08-19T14:15:31.803 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.16844759200000001 2024-08-19T14:15:31.804 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.804 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.804 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:15:31.804 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.174125+0000", 2024-08-19T14:15:31.804 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3953000000000003e-05 2024-08-19T14:15:31.805 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.805 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.805 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:15:31.805 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.174168+0000", 2024-08-19T14:15:31.805 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2809999999999998e-05 2024-08-19T14:15:31.805 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:31.806 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:15:31.806 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:31.806 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.806 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.806 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:63230 2.0 2:025360f7:::benchmark_data_smithi045_46415_object63229:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-08-19T14:15:31.807 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:21.036099+0000", 2024-08-19T14:15:31.807 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 130.70506528499999, 2024-08-19T14:15:31.807 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.14053950100000001, 2024-08-19T14:15:31.807 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:15:31.807 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:15:31.808 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:15:31.808 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:15:31.808 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:15:31.808 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:15:31.808 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 63230 2024-08-19T14:15:31.809 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.809 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:15:31.809 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.809 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:15:31.809 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.036099+0000", 2024-08-19T14:15:31.809 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:31.810 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.810 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.810 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:15:31.810 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.036099+0000", 2024-08-19T14:15:31.810 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:31.811 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.811 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.811 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:15:31.811 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.036101+0000", 2024-08-19T14:15:31.811 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2279999999999999e-06 2024-08-19T14:15:31.812 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.812 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.812 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:15:31.812 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.036163+0000", 2024-08-19T14:15:31.812 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1940999999999995e-05 2024-08-19T14:15:31.813 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.813 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.813 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:15:31.813 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.036182+0000", 2024-08-19T14:15:31.813 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9686000000000001e-05 2024-08-19T14:15:31.813 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.814 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.814 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:15:31.814 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.036189+0000", 2024-08-19T14:15:31.814 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6640000000000001e-06 2024-08-19T14:15:31.814 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.815 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.815 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:15:31.815 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.037060+0000", 2024-08-19T14:15:31.815 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00087146399999999996 2024-08-19T14:15:31.815 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.816 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.816 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:15:31.816 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.037212+0000", 2024-08-19T14:15:31.816 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00015186400000000001 2024-08-19T14:15:31.816 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.817 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.817 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 9,11", 2024-08-19T14:15:31.817 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.037383+0000", 2024-08-19T14:15:31.817 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000170457 2024-08-19T14:15:31.817 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.818 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.818 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:15:31.818 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.039083+0000", 2024-08-19T14:15:31.818 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0016998 2024-08-19T14:15:31.818 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.818 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.819 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:31.819 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.041514+0000", 2024-08-19T14:15:31.819 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.002431519 2024-08-19T14:15:31.819 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.819 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.820 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:31.820 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.175973+0000", 2024-08-19T14:15:31.820 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.134458932 2024-08-19T14:15:31.820 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.820 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.820 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:15:31.821 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.176028+0000", 2024-08-19T14:15:31.821 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5036e-05 2024-08-19T14:15:31.821 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.821 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.822 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:15:31.822 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.176639+0000", 2024-08-19T14:15:31.822 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00061090999999999999 2024-08-19T14:15:31.822 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:31.822 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:15:31.823 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:31.823 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.823 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.823 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:63231 2.8 2:1da1a88e:::benchmark_data_smithi045_46415_object63230:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-08-19T14:15:31.823 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:21.036208+0000", 2024-08-19T14:15:31.824 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 130.704956256, 2024-08-19T14:15:31.824 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.13809454500000001, 2024-08-19T14:15:31.824 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:15:31.824 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:15:31.824 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:15:31.825 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:15:31.825 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:15:31.825 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:15:31.825 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 63231 2024-08-19T14:15:31.825 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.825 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:15:31.826 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.826 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:15:31.826 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.036208+0000", 2024-08-19T14:15:31.826 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:31.827 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.827 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.827 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:15:31.827 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.036208+0000", 2024-08-19T14:15:31.827 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:31.827 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.828 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.828 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:15:31.828 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.036209+0000", 2024-08-19T14:15:31.828 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4000000000000002e-07 2024-08-19T14:15:31.828 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.829 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.829 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:15:31.829 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.036233+0000", 2024-08-19T14:15:31.829 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4573999999999999e-05 2024-08-19T14:15:31.829 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.830 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.830 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:15:31.830 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.036240+0000", 2024-08-19T14:15:31.830 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4110000000000001e-06 2024-08-19T14:15:31.830 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.831 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.831 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:15:31.831 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.036244+0000", 2024-08-19T14:15:31.831 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9129999999999996e-06 2024-08-19T14:15:31.831 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.831 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.832 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:15:31.832 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.037035+0000", 2024-08-19T14:15:31.832 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00079092500000000005 2024-08-19T14:15:31.832 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.832 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.833 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:15:31.833 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.037123+0000", 2024-08-19T14:15:31.833 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.8616999999999995e-05 2024-08-19T14:15:31.833 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.833 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.834 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0,9", 2024-08-19T14:15:31.834 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.037210+0000", 2024-08-19T14:15:31.834 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6300999999999999e-05 2024-08-19T14:15:31.834 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.834 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.834 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:15:31.835 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.038951+0000", 2024-08-19T14:15:31.835 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001741151 2024-08-19T14:15:31.835 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.835 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.835 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:31.836 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.039784+0000", 2024-08-19T14:15:31.836 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00083278900000000001 2024-08-19T14:15:31.836 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.836 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.836 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:31.837 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.174204+0000", 2024-08-19T14:15:31.837 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.134420768 2024-08-19T14:15:31.837 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.837 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.837 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:15:31.837 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.174261+0000", 2024-08-19T14:15:31.838 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6947000000000002e-05 2024-08-19T14:15:31.838 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.838 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.838 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:15:31.838 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.174303+0000", 2024-08-19T14:15:31.839 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1709000000000003e-05 2024-08-19T14:15:31.839 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:31.839 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:15:31.839 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:31.839 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.840 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.840 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:63245 2.0 2:0a3678e6:::benchmark_data_smithi045_46415_object63244:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-08-19T14:15:31.840 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:21.051568+0000", 2024-08-19T14:15:31.840 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 130.68959696799999, 2024-08-19T14:15:31.840 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.12567051500000001, 2024-08-19T14:15:31.841 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:15:31.841 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:15:31.841 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:15:31.841 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:15:31.841 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:15:31.842 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:15:31.842 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 63245 2024-08-19T14:15:31.842 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.842 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:15:31.842 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.842 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:15:31.843 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.051568+0000", 2024-08-19T14:15:31.843 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:31.843 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.843 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.843 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:15:31.844 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.051568+0000", 2024-08-19T14:15:31.844 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:31.844 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.844 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.844 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:15:31.844 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.051569+0000", 2024-08-19T14:15:31.845 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2190000000000001e-06 2024-08-19T14:15:31.845 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.845 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.846 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:15:31.846 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.051592+0000", 2024-08-19T14:15:31.846 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3391e-05 2024-08-19T14:15:31.846 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.846 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.847 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:15:31.847 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.051603+0000", 2024-08-19T14:15:31.847 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0879e-05 2024-08-19T14:15:31.847 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.847 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.848 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:15:31.848 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.051608+0000", 2024-08-19T14:15:31.848 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.933e-06 2024-08-19T14:15:31.848 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.848 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.849 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:15:31.849 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.052675+0000", 2024-08-19T14:15:31.849 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001066621 2024-08-19T14:15:31.849 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.849 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.849 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:15:31.850 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.052841+0000", 2024-08-19T14:15:31.850 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00016639700000000001 2024-08-19T14:15:31.850 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.850 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.850 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 9,11", 2024-08-19T14:15:31.851 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.053037+0000", 2024-08-19T14:15:31.851 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00019577200000000001 2024-08-19T14:15:31.851 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.851 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.851 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:31.851 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.059090+0000", 2024-08-19T14:15:31.852 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0060532290000000002 2024-08-19T14:15:31.852 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.852 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.852 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:15:31.852 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.059156+0000", 2024-08-19T14:15:31.853 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6221000000000002e-05 2024-08-19T14:15:31.853 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.853 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.853 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:31.853 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.176685+0000", 2024-08-19T14:15:31.854 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.11752820899999999 2024-08-19T14:15:31.854 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.854 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.854 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:15:31.854 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.176716+0000", 2024-08-19T14:15:31.854 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1223000000000002e-05 2024-08-19T14:15:31.855 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.855 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.855 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:15:31.855 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.177238+0000", 2024-08-19T14:15:31.855 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00052242100000000002 2024-08-19T14:15:31.856 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:31.856 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:15:31.856 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:31.856 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.856 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.857 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:63247 2.0 2:0918bab6:::benchmark_data_smithi045_46415_object63246:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-08-19T14:15:31.857 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:21.052340+0000", 2024-08-19T14:15:31.857 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 130.68882508300001, 2024-08-19T14:15:31.857 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.12561350199999999, 2024-08-19T14:15:31.857 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:15:31.857 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:15:31.858 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:15:31.858 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:15:31.858 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:15:31.858 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:15:31.858 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 63247 2024-08-19T14:15:31.858 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.858 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:15:31.859 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.859 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:15:31.859 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.052340+0000", 2024-08-19T14:15:31.859 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:31.859 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.859 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.859 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:15:31.859 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.052340+0000", 2024-08-19T14:15:31.859 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:31.859 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.859 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.859 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:15:31.859 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.052340+0000", 2024-08-19T14:15:31.859 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5099999999999999e-07 2024-08-19T14:15:31.859 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.860 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.860 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:15:31.860 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.052371+0000", 2024-08-19T14:15:31.860 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1038000000000003e-05 2024-08-19T14:15:31.860 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.860 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.860 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:15:31.860 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.052382+0000", 2024-08-19T14:15:31.860 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0638999999999999e-05 2024-08-19T14:15:31.860 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.860 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.860 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:15:31.860 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.052387+0000", 2024-08-19T14:15:31.861 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.164e-06 2024-08-19T14:15:31.861 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.861 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.861 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:15:31.861 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.054907+0000", 2024-08-19T14:15:31.861 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0025194649999999998 2024-08-19T14:15:31.861 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.861 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.861 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:15:31.861 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.055057+0000", 2024-08-19T14:15:31.861 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000150607 2024-08-19T14:15:31.861 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.861 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.861 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 9,11", 2024-08-19T14:15:31.862 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.055161+0000", 2024-08-19T14:15:31.862 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000103644 2024-08-19T14:15:31.862 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.862 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.862 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:31.862 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.059147+0000", 2024-08-19T14:15:31.862 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0039857929999999996 2024-08-19T14:15:31.862 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.862 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.862 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:15:31.862 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.059160+0000", 2024-08-19T14:15:31.862 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3441000000000001e-05 2024-08-19T14:15:31.862 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.862 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.863 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:31.863 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.177310+0000", 2024-08-19T14:15:31.863 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.118149858 2024-08-19T14:15:31.863 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.863 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.863 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:15:31.863 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.177368+0000", 2024-08-19T14:15:31.863 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7633000000000002e-05 2024-08-19T14:15:31.863 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.863 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.863 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:15:31.863 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.177953+0000", 2024-08-19T14:15:31.863 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00058546900000000003 2024-08-19T14:15:31.864 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:31.864 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:15:31.864 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:31.864 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.864 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.864 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:63250 2.8 2:1918e081:::benchmark_data_smithi045_46415_object63249:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-08-19T14:15:31.864 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:21.056840+0000", 2024-08-19T14:15:31.864 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 130.684324729, 2024-08-19T14:15:31.864 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.119880845, 2024-08-19T14:15:31.864 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:15:31.864 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:15:31.864 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:15:31.864 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:15:31.864 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:15:31.865 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:15:31.865 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 63250 2024-08-19T14:15:31.865 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.865 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:15:31.865 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.865 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:15:31.865 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.056840+0000", 2024-08-19T14:15:31.865 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:31.865 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.865 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.865 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:15:31.865 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.056840+0000", 2024-08-19T14:15:31.865 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:31.865 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.866 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.866 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:15:31.866 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.056841+0000", 2024-08-19T14:15:31.866 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4360000000000001e-06 2024-08-19T14:15:31.866 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.866 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.867 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:15:31.867 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.056905+0000", 2024-08-19T14:15:31.867 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3677999999999996e-05 2024-08-19T14:15:31.867 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.867 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.867 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:15:31.867 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.056919+0000", 2024-08-19T14:15:31.867 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3556e-05 2024-08-19T14:15:31.867 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.867 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.867 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:15:31.867 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.056926+0000", 2024-08-19T14:15:31.867 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3579999999999997e-06 2024-08-19T14:15:31.867 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.868 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.868 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:15:31.868 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.059133+0000", 2024-08-19T14:15:31.868 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0022067789999999999 2024-08-19T14:15:31.868 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.868 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.868 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:15:31.868 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.059259+0000", 2024-08-19T14:15:31.868 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012647799999999999 2024-08-19T14:15:31.868 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.868 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.868 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0,9", 2024-08-19T14:15:31.868 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.059411+0000", 2024-08-19T14:15:31.868 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00015167800000000001 2024-08-19T14:15:31.869 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.869 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.869 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:31.869 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.061433+0000", 2024-08-19T14:15:31.869 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0020222510000000001 2024-08-19T14:15:31.869 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.869 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.869 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:15:31.869 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.062698+0000", 2024-08-19T14:15:31.869 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0012643559999999999 2024-08-19T14:15:31.869 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.869 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.869 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:31.869 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.176669+0000", 2024-08-19T14:15:31.869 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.11397160200000001 2024-08-19T14:15:31.870 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.870 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.870 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:15:31.870 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.176701+0000", 2024-08-19T14:15:31.870 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1863000000000003e-05 2024-08-19T14:15:31.870 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.870 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.870 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:15:31.870 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.176721+0000", 2024-08-19T14:15:31.870 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9809999999999998e-05 2024-08-19T14:15:31.870 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:31.870 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:15:31.870 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:31.870 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.871 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.871 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:115128 2.0 2:08e033d8:::benchmark_data_smithi045_46415_object115127:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-08-19T14:15:31.871 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:14:09.964413+0000", 2024-08-19T14:15:31.874 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 81.776752002999999, 2024-08-19T14:15:31.874 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.15347317799999999, 2024-08-19T14:15:31.874 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:15:31.875 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:15:31.875 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:15:31.875 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:15:31.875 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:15:31.875 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:15:31.875 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 115128 2024-08-19T14:15:31.875 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.875 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:15:31.875 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.875 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:15:31.875 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:09.964413+0000", 2024-08-19T14:15:31.875 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:31.875 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.876 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.876 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:15:31.876 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:09.964413+0000", 2024-08-19T14:15:31.876 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:31.876 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.876 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.876 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:15:31.876 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:09.964414+0000", 2024-08-19T14:15:31.876 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.689e-06 2024-08-19T14:15:31.876 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.876 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.876 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:15:31.876 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:09.964452+0000", 2024-08-19T14:15:31.876 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7565999999999999e-05 2024-08-19T14:15:31.877 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.877 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.877 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:15:31.877 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:09.964467+0000", 2024-08-19T14:15:31.877 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5160999999999999e-05 2024-08-19T14:15:31.877 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.877 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.877 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:15:31.877 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:09.964475+0000", 2024-08-19T14:15:31.877 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.276e-06 2024-08-19T14:15:31.877 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.877 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.877 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:15:31.877 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:09.964534+0000", 2024-08-19T14:15:31.878 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8727999999999997e-05 2024-08-19T14:15:31.878 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.878 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.878 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:15:31.878 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:09.964675+0000", 2024-08-19T14:15:31.878 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000140705 2024-08-19T14:15:31.878 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.878 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.878 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 9,11", 2024-08-19T14:15:31.878 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:09.964830+0000", 2024-08-19T14:15:31.878 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00015500900000000001 2024-08-19T14:15:31.878 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.878 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.878 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:15:31.879 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:09.967628+0000", 2024-08-19T14:15:31.879 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0027980909999999999 2024-08-19T14:15:31.879 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.879 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.879 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:31.879 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:09.968335+0000", 2024-08-19T14:15:31.879 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00070710399999999996 2024-08-19T14:15:31.879 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.879 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.879 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:31.879 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:10.117809+0000", 2024-08-19T14:15:31.879 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.149473572 2024-08-19T14:15:31.879 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.879 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.880 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:15:31.880 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:10.117849+0000", 2024-08-19T14:15:31.880 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0431e-05 2024-08-19T14:15:31.880 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.880 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.880 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:15:31.880 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:10.117886+0000", 2024-08-19T14:15:31.880 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6845999999999999e-05 2024-08-19T14:15:31.880 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:31.880 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:15:31.880 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:31.880 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.880 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.880 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:115136 2.0 2:02df7cb0:::benchmark_data_smithi045_46415_object115135:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-08-19T14:15:31.881 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:14:09.971356+0000", 2024-08-19T14:15:31.881 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 81.769808513000001, 2024-08-19T14:15:31.881 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.15309395200000001, 2024-08-19T14:15:31.881 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:15:31.881 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:15:31.881 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:15:31.881 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:15:31.881 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:15:31.881 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:15:31.881 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 115136 2024-08-19T14:15:31.881 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.881 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:15:31.881 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.881 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:15:31.882 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:09.971356+0000", 2024-08-19T14:15:31.882 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:31.882 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.882 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.882 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:15:31.882 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:09.971356+0000", 2024-08-19T14:15:31.882 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:31.882 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.882 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.882 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:15:31.882 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:09.971357+0000", 2024-08-19T14:15:31.882 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.172e-06 2024-08-19T14:15:31.882 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.882 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.883 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:15:31.883 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:09.971398+0000", 2024-08-19T14:15:31.883 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0955999999999999e-05 2024-08-19T14:15:31.883 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.883 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.883 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:15:31.883 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:09.971410+0000", 2024-08-19T14:15:31.883 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.135e-05 2024-08-19T14:15:31.883 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.883 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.883 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:15:31.883 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:09.971415+0000", 2024-08-19T14:15:31.883 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1819999999999997e-06 2024-08-19T14:15:31.883 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.884 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.884 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:15:31.884 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:09.971457+0000", 2024-08-19T14:15:31.884 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1972999999999999e-05 2024-08-19T14:15:31.884 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.884 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.884 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:15:31.884 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:09.971584+0000", 2024-08-19T14:15:31.884 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012695800000000001 2024-08-19T14:15:31.884 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.884 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.884 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 9,11", 2024-08-19T14:15:31.884 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:09.971773+0000", 2024-08-19T14:15:31.884 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000189101 2024-08-19T14:15:31.884 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.885 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.885 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:15:31.885 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:09.975215+0000", 2024-08-19T14:15:31.885 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.003441871 2024-08-19T14:15:31.885 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.885 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.885 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:31.885 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:10.123256+0000", 2024-08-19T14:15:31.885 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.14804097999999999 2024-08-19T14:15:31.885 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.885 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.885 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:31.885 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:10.124383+0000", 2024-08-19T14:15:31.885 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00112725 2024-08-19T14:15:31.886 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.886 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.886 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:15:31.886 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:10.124421+0000", 2024-08-19T14:15:31.886 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8031999999999998e-05 2024-08-19T14:15:31.886 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.886 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.886 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:15:31.886 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:10.124450+0000", 2024-08-19T14:15:31.886 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9127000000000001e-05 2024-08-19T14:15:31.886 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:31.886 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:15:31.886 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:31.886 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.886 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.887 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:116086 2.0 2:05ee7e1e:::benchmark_data_smithi045_46415_object116085:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-08-19T14:15:31.887 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:14:11.223235+0000", 2024-08-19T14:15:31.887 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 80.517929992000006, 2024-08-19T14:15:31.887 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.115283706, 2024-08-19T14:15:31.887 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:15:31.889 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:15:31.889 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:15:31.889 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:15:31.889 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:15:31.889 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:15:31.889 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:15:31.889 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 116086 2024-08-19T14:15:31.889 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.889 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:15:31.889 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.889 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:15:31.889 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:11.223235+0000", 2024-08-19T14:15:31.889 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:31.890 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.890 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.890 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:15:31.890 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:11.223235+0000", 2024-08-19T14:15:31.890 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:31.890 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.890 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.890 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:15:31.890 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:11.223235+0000", 2024-08-19T14:15:31.890 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1299999999999999e-07 2024-08-19T14:15:31.890 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.890 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.890 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:15:31.891 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:11.223313+0000", 2024-08-19T14:15:31.891 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7986999999999997e-05 2024-08-19T14:15:31.891 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.891 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.891 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:15:31.891 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:11.223323+0000", 2024-08-19T14:15:31.891 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.5510000000000005e-06 2024-08-19T14:15:31.891 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.891 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.891 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:15:31.891 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:11.223328+0000", 2024-08-19T14:15:31.891 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4360000000000001e-06 2024-08-19T14:15:31.891 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.891 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.892 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:15:31.892 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:11.223370+0000", 2024-08-19T14:15:31.892 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1551999999999998e-05 2024-08-19T14:15:31.892 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.892 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.892 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:15:31.892 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:11.223471+0000", 2024-08-19T14:15:31.892 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00010061 2024-08-19T14:15:31.892 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.892 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.892 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 9,11", 2024-08-19T14:15:31.892 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:11.223559+0000", 2024-08-19T14:15:31.892 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.8264999999999996e-05 2024-08-19T14:15:31.893 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.893 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.893 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:15:31.893 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:11.226453+0000", 2024-08-19T14:15:31.893 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0028945139999999999 2024-08-19T14:15:31.893 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.893 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.893 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:31.893 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:11.326208+0000", 2024-08-19T14:15:31.893 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.099755144000000004 2024-08-19T14:15:31.893 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.893 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.893 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:31.893 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:11.338458+0000", 2024-08-19T14:15:31.894 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.012249833 2024-08-19T14:15:31.894 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.894 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.894 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:15:31.894 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:11.338492+0000", 2024-08-19T14:15:31.894 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3806999999999997e-05 2024-08-19T14:15:31.894 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.894 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.894 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:15:31.894 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:11.338518+0000", 2024-08-19T14:15:31.894 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6293999999999999e-05 2024-08-19T14:15:31.894 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:31.894 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:15:31.895 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:31.895 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.895 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.895 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:171514 2.0 2:07116ad7:::benchmark_data_smithi045_46415_object48233:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e169)", 2024-08-19T14:15:31.895 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:15:12.896400+0000", 2024-08-19T14:15:31.895 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 18.844764297000001, 2024-08-19T14:15:31.895 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.112982306, 2024-08-19T14:15:31.895 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:15:31.895 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:15:31.895 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:15:31.895 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:15:31.895 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:15:31.895 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:15:31.895 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 171514 2024-08-19T14:15:31.896 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.896 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:15:31.896 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.896 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:15:31.896 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.896400+0000", 2024-08-19T14:15:31.896 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:31.896 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.896 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.896 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:15:31.896 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.896400+0000", 2024-08-19T14:15:31.896 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:31.896 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.896 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.896 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:15:31.897 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.896401+0000", 2024-08-19T14:15:31.897 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.3200000000000004e-07 2024-08-19T14:15:31.897 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.897 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.897 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:15:31.897 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.896405+0000", 2024-08-19T14:15:31.897 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0260000000000001e-06 2024-08-19T14:15:31.897 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.897 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.897 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:15:31.897 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.896416+0000", 2024-08-19T14:15:31.897 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0634e-05 2024-08-19T14:15:31.897 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.897 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.898 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:15:31.898 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.896423+0000", 2024-08-19T14:15:31.898 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3039999999999999e-06 2024-08-19T14:15:31.898 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.898 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.898 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:15:31.898 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.897983+0000", 2024-08-19T14:15:31.898 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0015601250000000001 2024-08-19T14:15:31.898 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.898 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.898 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:15:31.898 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.898096+0000", 2024-08-19T14:15:31.898 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00011288 2024-08-19T14:15:31.898 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.899 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.899 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 9,11", 2024-08-19T14:15:31.899 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.898211+0000", 2024-08-19T14:15:31.899 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000114633 2024-08-19T14:15:31.899 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.899 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.899 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:15:31.899 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:13.009151+0000", 2024-08-19T14:15:31.899 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.110940552 2024-08-19T14:15:31.899 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.899 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.899 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:31.899 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:13.009202+0000", 2024-08-19T14:15:31.900 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0250000000000002e-05 2024-08-19T14:15:31.900 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.900 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.900 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:31.900 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:13.009237+0000", 2024-08-19T14:15:31.900 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5660000000000001e-05 2024-08-19T14:15:31.900 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.900 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.900 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:15:31.900 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:13.009339+0000", 2024-08-19T14:15:31.900 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000101908 2024-08-19T14:15:31.900 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.900 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.900 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:15:31.900 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:13.009383+0000", 2024-08-19T14:15:31.901 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3501999999999998e-05 2024-08-19T14:15:31.901 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:31.901 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:15:31.901 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:31.901 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.901 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.901 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:171515 2.8 2:10779ea6:::benchmark_data_smithi045_46415_object48234:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e169)", 2024-08-19T14:15:31.901 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:15:12.902135+0000", 2024-08-19T14:15:31.901 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 18.839029642, 2024-08-19T14:15:31.901 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.131735885, 2024-08-19T14:15:31.901 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:15:31.901 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:15:31.901 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:15:31.901 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:15:31.902 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:15:31.902 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:15:31.902 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 171515 2024-08-19T14:15:31.902 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.902 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:15:31.902 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.902 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:15:31.902 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.902135+0000", 2024-08-19T14:15:31.902 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:31.902 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.902 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.902 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:15:31.902 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.902135+0000", 2024-08-19T14:15:31.902 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:31.903 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.903 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.903 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:15:31.903 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.902136+0000", 2024-08-19T14:15:31.903 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.9400000000000004e-07 2024-08-19T14:15:31.903 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.903 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.903 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:15:31.903 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.902140+0000", 2024-08-19T14:15:31.903 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9600000000000002e-06 2024-08-19T14:15:31.903 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.903 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.903 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:15:31.903 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.902148+0000", 2024-08-19T14:15:31.904 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.8290000000000008e-06 2024-08-19T14:15:31.904 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.904 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.904 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:15:31.904 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.902153+0000", 2024-08-19T14:15:31.904 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7990000000000001e-06 2024-08-19T14:15:31.904 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.904 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.904 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:15:31.904 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:13.009177+0000", 2024-08-19T14:15:31.904 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.107024059 2024-08-19T14:15:31.904 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.904 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.904 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:15:31.905 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:13.009300+0000", 2024-08-19T14:15:31.905 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012342399999999999 2024-08-19T14:15:31.905 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.905 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.905 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0,9", 2024-08-19T14:15:31.905 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:13.009424+0000", 2024-08-19T14:15:31.905 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012440500000000001 2024-08-19T14:15:31.905 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.905 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.905 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:31.905 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:13.023363+0000", 2024-08-19T14:15:31.905 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.013938623000000001 2024-08-19T14:15:31.905 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.905 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.906 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:15:31.906 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:13.023469+0000", 2024-08-19T14:15:31.906 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00010608 2024-08-19T14:15:31.906 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.906 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.906 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:31.906 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:13.033804+0000", 2024-08-19T14:15:31.906 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.01033537 2024-08-19T14:15:31.906 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.906 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.906 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:15:31.906 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:13.033837+0000", 2024-08-19T14:15:31.906 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2364999999999999e-05 2024-08-19T14:15:31.906 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.907 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.907 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:15:31.907 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:13.033871+0000", 2024-08-19T14:15:31.907 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.4076999999999999e-05 2024-08-19T14:15:31.907 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:31.907 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:15:31.907 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:31.907 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.907 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.907 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:171516 2.8 2:1157ba84:::benchmark_data_smithi045_46415_object48235:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e169)", 2024-08-19T14:15:31.907 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:15:12.902240+0000", 2024-08-19T14:15:31.907 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 18.838924454000001, 2024-08-19T14:15:31.907 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.13175336400000001, 2024-08-19T14:15:31.907 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:15:31.908 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:15:31.908 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:15:31.908 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:15:31.908 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:15:31.908 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:15:31.908 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 171516 2024-08-19T14:15:31.908 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.908 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:15:31.908 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.908 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:15:31.908 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.902240+0000", 2024-08-19T14:15:31.908 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:31.908 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.908 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.909 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:15:31.909 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.902240+0000", 2024-08-19T14:15:31.909 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:31.909 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.909 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.909 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:15:31.909 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.902241+0000", 2024-08-19T14:15:31.909 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9900000000000001e-07 2024-08-19T14:15:31.909 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.909 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.909 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:15:31.909 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.902244+0000", 2024-08-19T14:15:31.909 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.869e-06 2024-08-19T14:15:31.909 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.909 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.910 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:15:31.910 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.902249+0000", 2024-08-19T14:15:31.910 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8289999999999996e-06 2024-08-19T14:15:31.910 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.910 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.910 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:15:31.910 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.902253+0000", 2024-08-19T14:15:31.910 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8670000000000004e-06 2024-08-19T14:15:31.910 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.910 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.910 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:15:31.910 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:13.013300+0000", 2024-08-19T14:15:31.910 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.11104626400000001 2024-08-19T14:15:31.911 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.911 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.911 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:15:31.911 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:13.013422+0000", 2024-08-19T14:15:31.911 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012265099999999999 2024-08-19T14:15:31.911 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.911 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.911 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0,9", 2024-08-19T14:15:31.911 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:13.013552+0000", 2024-08-19T14:15:31.911 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000130123 2024-08-19T14:15:31.911 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.911 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.911 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:31.911 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:13.023455+0000", 2024-08-19T14:15:31.912 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0099022610000000007 2024-08-19T14:15:31.912 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.912 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.912 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:15:31.912 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:13.023476+0000", 2024-08-19T14:15:31.912 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1591e-05 2024-08-19T14:15:31.912 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.912 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.912 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:31.912 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:13.033911+0000", 2024-08-19T14:15:31.912 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010434824000000001 2024-08-19T14:15:31.912 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.912 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.912 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:15:31.913 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:13.033940+0000", 2024-08-19T14:15:31.913 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9033000000000001e-05 2024-08-19T14:15:31.913 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.913 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.913 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:15:31.913 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:13.033994+0000", 2024-08-19T14:15:31.913 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3553e-05 2024-08-19T14:15:31.913 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:31.913 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:15:31.913 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:31.913 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.913 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.913 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:171517 2.8 2:17a35dc4:::benchmark_data_smithi045_46415_object48236:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e169)", 2024-08-19T14:15:31.913 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:15:12.905580+0000", 2024-08-19T14:15:31.914 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 18.835585164000001, 2024-08-19T14:15:31.914 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.133369984, 2024-08-19T14:15:31.914 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:15:31.914 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:15:31.914 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:15:31.914 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:15:31.914 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:15:31.914 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:15:31.914 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 171517 2024-08-19T14:15:31.914 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.914 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:15:31.914 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.914 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:15:31.914 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.905580+0000", 2024-08-19T14:15:31.915 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:31.915 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.915 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.915 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:15:31.915 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.905580+0000", 2024-08-19T14:15:31.915 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:15:31.915 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.915 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.915 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:15:31.915 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.905580+0000", 2024-08-19T14:15:31.915 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6199999999999997e-07 2024-08-19T14:15:31.915 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.915 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.916 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:15:31.916 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.905583+0000", 2024-08-19T14:15:31.916 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2789999999999999e-06 2024-08-19T14:15:31.916 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.916 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.916 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:15:31.916 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:13.004022+0000", 2024-08-19T14:15:31.916 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.098438756000000002 2024-08-19T14:15:31.916 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.916 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.916 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:15:31.916 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:13.004038+0000", 2024-08-19T14:15:31.916 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6038000000000001e-05 2024-08-19T14:15:31.916 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.916 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.917 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:15:31.917 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:13.018509+0000", 2024-08-19T14:15:31.917 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.014470642000000001 2024-08-19T14:15:31.917 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.917 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.917 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:15:31.917 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:13.018652+0000", 2024-08-19T14:15:31.917 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000143048 2024-08-19T14:15:31.917 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.917 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.917 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0,9", 2024-08-19T14:15:31.917 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:13.018802+0000", 2024-08-19T14:15:31.917 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000149748 2024-08-19T14:15:31.917 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.918 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.918 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:31.918 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:13.023512+0000", 2024-08-19T14:15:31.918 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0047101729999999998 2024-08-19T14:15:31.918 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.918 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.918 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:15:31.918 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:13.028445+0000", 2024-08-19T14:15:31.918 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0049330479999999998 2024-08-19T14:15:31.918 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.918 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.918 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:31.918 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:13.038830+0000", 2024-08-19T14:15:31.918 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010384882999999999 2024-08-19T14:15:31.918 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.919 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.919 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:15:31.919 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:13.038884+0000", 2024-08-19T14:15:31.919 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4664999999999999e-05 2024-08-19T14:15:31.919 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:15:31.919 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:15:31.919 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:15:31.919 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:13.038950+0000", 2024-08-19T14:15:31.919 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.5042000000000005e-05 2024-08-19T14:15:31.919 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:31.919 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:15:31.919 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:31.919 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:15:31.919 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:15:31.920 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:15:31.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:31.921 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:31.921 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:15:32.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:31 smithi032 ceph-mon[20004]: pgmap v446: 27 pgs: 2 active+remapped+backfilling, 25 active+clean; 3.9 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 857 op/s; 5918/193152 objects misplaced (3.064%); 10 MiB/s, 165 objects/s recovering 2024-08-19T14:15:32.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:31 smithi032 ceph-mon[20004]: 2.2 scrub starts 2024-08-19T14:15:32.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:31 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1652542662' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:15:32.022 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:15:32.045 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:32.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:32.046 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:15:32.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:31 smithi045 ceph-mon[20143]: pgmap v446: 27 pgs: 2 active+remapped+backfilling, 25 active+clean; 3.9 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 857 op/s; 5918/193152 objects misplaced (3.064%); 10 MiB/s, 165 objects/s recovering 2024-08-19T14:15:32.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:31 smithi045 ceph-mon[20143]: 2.2 scrub starts 2024-08-19T14:15:32.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:31 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1652542662' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:15:32.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:31 smithi031 ceph-mon[16336]: pgmap v446: 27 pgs: 2 active+remapped+backfilling, 25 active+clean; 3.9 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 857 op/s; 5918/193152 objects misplaced (3.064%); 10 MiB/s, 165 objects/s recovering 2024-08-19T14:15:32.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:31 smithi031 ceph-mon[16336]: 2.2 scrub starts 2024-08-19T14:15:32.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:31 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1652542662' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:15:32.147 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:15:32.204 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:32.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:32.206 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:15:32.307 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:15:32.340 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:32.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:32.341 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:15:32.442 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:15:32.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:32.486 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:32.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:15:32.548 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-19T14:15:32.587 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:15:32.619 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:32.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:32.620 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:15:32.625 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-08-19T14:15:32.721 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:15:32.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:32.767 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:32.767 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:15:32.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:32 smithi031 ceph-mon[16336]: 2.9 scrub starts 2024-08-19T14:15:32.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:32 smithi031 ceph-mon[16336]: 2.2 scrub ok 2024-08-19T14:15:32.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:32 smithi031 ceph-mon[16336]: pgmap v447: 27 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfilling, 24 active+clean; 3.8 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 898 op/s; 4059/187347 objects misplaced (2.167%); 22 MiB/s, 355 objects/s recovering 2024-08-19T14:15:32.868 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:32 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:15:32.868 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:32 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:15:32.868 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:15:32.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:32.926 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:32.926 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:15:32.958 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:15:32.999 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:15:33.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:32 smithi032 ceph-mon[20004]: 2.9 scrub starts 2024-08-19T14:15:33.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:32 smithi032 ceph-mon[20004]: 2.2 scrub ok 2024-08-19T14:15:33.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:32 smithi032 ceph-mon[20004]: pgmap v447: 27 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfilling, 24 active+clean; 3.8 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 898 op/s; 4059/187347 objects misplaced (2.167%); 22 MiB/s, 355 objects/s recovering 2024-08-19T14:15:33.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:32 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:15:33.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:32 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:15:33.027 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:15:33.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:32 smithi045 ceph-mon[20143]: 2.9 scrub starts 2024-08-19T14:15:33.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:32 smithi045 ceph-mon[20143]: 2.2 scrub ok 2024-08-19T14:15:33.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:32 smithi045 ceph-mon[20143]: pgmap v447: 27 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfilling, 24 active+clean; 3.8 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 898 op/s; 4059/187347 objects misplaced (2.167%); 22 MiB/s, 355 objects/s recovering 2024-08-19T14:15:33.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:32 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:15:33.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:32 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:15:33.052 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:33.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:33.053 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:15:33.154 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:15:33.178 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:33.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:33.179 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:15:33.280 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:15:33.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:33.305 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:33.305 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:15:33.406 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:15:33.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:33.450 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:33.450 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:15:33.526 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:15:33.527 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [ 2024-08-19T14:15:33.527 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:33.527 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDPGPush(2.1 179/172 [PushOp(2:88956227:::benchmark_data_smithi045_46415_object109738:head, version: 146'6895, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:88956227:::benchmark_data_smithi045_46415_object109738:head@146'6895, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-19T14:15:33.527 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:15:33.522234+0000", 2024-08-19T14:15:33.527 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.000113484, 2024-08-19T14:15:33.527 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00016016199999999999, 2024-08-19T14:15:33.527 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:15:33.527 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:15:33.527 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-08-19T14:15:33.527 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:15:33.527 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:33.527 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:15:33.527 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:15:33.522234+0000", 2024-08-19T14:15:33.527 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:15:33.528 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:33.528 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:33.528 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:15:33.528 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:15:33.522234+0000", 2024-08-19T14:15:33.528 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:15:33.528 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:33.528 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:33.528 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:15:33.528 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:15:33.522234+0000", 2024-08-19T14:15:33.528 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.91e-07 2024-08-19T14:15:33.528 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:33.528 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:33.528 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:15:33.529 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:15:33.522283+0000", 2024-08-19T14:15:33.529 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.9567999999999997e-05 2024-08-19T14:15:33.529 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:33.529 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:33.529 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:15:33.529 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:15:33.522297+0000", 2024-08-19T14:15:33.529 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3597999999999999e-05 2024-08-19T14:15:33.529 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:33.529 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:33.529 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:15:33.529 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:15:33.522308+0000", 2024-08-19T14:15:33.529 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0983e-05 2024-08-19T14:15:33.529 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:33.529 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:33.529 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:15:33.530 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:15:33.522384+0000", 2024-08-19T14:15:33.530 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.5666000000000004e-05 2024-08-19T14:15:33.530 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:33.530 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:33.530 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:15:33.530 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:15:33.522397+0000", 2024-08-19T14:15:33.530 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3261999999999999e-05 2024-08-19T14:15:33.530 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:15:33.530 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:15:33.530 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:15:33.530 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:15:33.530 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-08-19T14:15:33.530 INFO:teuthology.orchestra.run.smithi045.stdout: "num_ops": 1 2024-08-19T14:15:33.531 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:15:33.551 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:15:33.579 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:33.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:33.580 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:15:33.681 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:15:33.705 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:33.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:33.706 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:15:33.807 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:15:33.849 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:33.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:33.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:15:33.951 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:15:33.973 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:33.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:33.975 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:15:34.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:33 smithi045 ceph-mon[20143]: 2.3 scrub starts 2024-08-19T14:15:34.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:33 smithi045 ceph-mon[20143]: osdmap e179: 12 total, 12 up, 10 in 2024-08-19T14:15:34.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:33 smithi045 ceph-mon[20143]: 2.3 scrub ok 2024-08-19T14:15:34.076 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:15:34.077 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:33 smithi031 ceph-mon[16336]: 2.3 scrub starts 2024-08-19T14:15:34.077 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:33 smithi031 ceph-mon[16336]: osdmap e179: 12 total, 12 up, 10 in 2024-08-19T14:15:34.077 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:33 smithi031 ceph-mon[16336]: 2.3 scrub ok 2024-08-19T14:15:34.103 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:34.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:34.104 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:15:34.205 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:15:34.235 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:15:34.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:34.237 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:34.237 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:15:34.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:33 smithi032 ceph-mon[20004]: 2.3 scrub starts 2024-08-19T14:15:34.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:33 smithi032 ceph-mon[20004]: osdmap e179: 12 total, 12 up, 10 in 2024-08-19T14:15:34.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:33 smithi032 ceph-mon[20004]: 2.3 scrub ok 2024-08-19T14:15:34.338 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:15:34.359 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-08-19T14:15:34.375 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:34.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:34.376 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:15:34.476 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:15:34.500 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:34.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:34.501 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:15:34.602 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:15:34.635 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:34.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:34.637 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:15:34.690 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:15:34.732 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:15:34.738 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:15:34.782 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:34.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:34.784 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:15:34.801 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is unset 2024-08-19T14:15:34.885 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:15:34.908 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 8, 9, 11] out_osds: [10, 6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-08-19T14:15:34.908 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-19T14:15:34.908 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-08-19T14:15:34.908 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd test-reweight-by-utilization 1000 0.05 2024-08-19T14:15:34.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:34.937 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:34.937 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:15:35.038 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:15:35.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:35.085 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:35.085 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:15:35.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:34 smithi031 ceph-mon[16336]: 2.9 scrub ok 2024-08-19T14:15:35.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:34 smithi031 ceph-mon[16336]: pgmap v449: 27 pgs: 2 active+clean+scrubbing, 2 active+remapped+backfilling, 23 active+clean; 3.7 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 931 op/s; 4059/182454 objects misplaced (2.225%); 22 MiB/s, 355 objects/s recovering 2024-08-19T14:15:35.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:34 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1184079066' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:15:35.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:34 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:15:35.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:34 smithi031 ceph-mon[16336]: 2.d scrub starts 2024-08-19T14:15:35.171 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:34 smithi045 ceph-mon[20143]: 2.9 scrub ok 2024-08-19T14:15:35.171 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:34 smithi045 ceph-mon[20143]: pgmap v449: 27 pgs: 2 active+clean+scrubbing, 2 active+remapped+backfilling, 23 active+clean; 3.7 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 931 op/s; 4059/182454 objects misplaced (2.225%); 22 MiB/s, 355 objects/s recovering 2024-08-19T14:15:35.171 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:34 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1184079066' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:15:35.171 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:34 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:15:35.171 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:34 smithi045 ceph-mon[20143]: 2.d scrub starts 2024-08-19T14:15:35.186 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:15:35.212 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:35.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:35.213 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:15:35.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:34 smithi032 ceph-mon[20004]: 2.9 scrub ok 2024-08-19T14:15:35.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:34 smithi032 ceph-mon[20004]: pgmap v449: 27 pgs: 2 active+clean+scrubbing, 2 active+remapped+backfilling, 23 active+clean; 3.7 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 931 op/s; 4059/182454 objects misplaced (2.225%); 22 MiB/s, 355 objects/s recovering 2024-08-19T14:15:35.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:34 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1184079066' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:15:35.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:34 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:15:35.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:34 smithi032 ceph-mon[20004]: 2.d scrub starts 2024-08-19T14:15:35.307 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:15:35.307 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [], 2024-08-19T14:15:35.307 INFO:teuthology.orchestra.run.smithi045.stdout: "complaint_time": 30, 2024-08-19T14:15:35.307 INFO:teuthology.orchestra.run.smithi045.stdout: "num_blocked_ops": 0 2024-08-19T14:15:35.307 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:15:35.314 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:15:35.338 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:35.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:35.339 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:15:35.440 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:15:35.464 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:35.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:35.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:15:35.566 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:15:35.590 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:35.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:35.591 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:15:35.692 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:15:35.739 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:35.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:35.740 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:15:35.841 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:15:35.888 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:35.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:35.889 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:15:35.990 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:15:35.999 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-08-19T14:15:36.025 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:36.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:36.027 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:15:36.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:35 smithi031 ceph-mon[16336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:15:36.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:35 smithi031 ceph-mon[16336]: Cluster is now healthy 2024-08-19T14:15:36.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:35 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:15:36.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:35 smithi031 ceph-mon[16336]: osdmap e180: 12 total, 12 up, 10 in 2024-08-19T14:15:36.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:35 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:15:36.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:35 smithi031 ceph-mon[16336]: 2.a scrub starts 2024-08-19T14:15:36.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:35 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:15:36.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:35 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:15:36.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:35 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:15:36.128 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:15:36.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:36.161 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:36.161 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:15:36.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:35 smithi032 ceph-mon[20004]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:15:36.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:35 smithi032 ceph-mon[20004]: Cluster is now healthy 2024-08-19T14:15:36.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:35 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:15:36.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:35 smithi032 ceph-mon[20004]: osdmap e180: 12 total, 12 up, 10 in 2024-08-19T14:15:36.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:35 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:15:36.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:35 smithi032 ceph-mon[20004]: 2.a scrub starts 2024-08-19T14:15:36.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:35 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:15:36.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:35 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:15:36.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:35 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:15:36.261 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:15:36.291 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:36.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:36.292 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:15:36.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:35 smithi045 ceph-mon[20143]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:15:36.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:35 smithi045 ceph-mon[20143]: Cluster is now healthy 2024-08-19T14:15:36.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:35 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:15:36.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:35 smithi045 ceph-mon[20143]: osdmap e180: 12 total, 12 up, 10 in 2024-08-19T14:15:36.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:35 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:15:36.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:35 smithi045 ceph-mon[20143]: 2.a scrub starts 2024-08-19T14:15:36.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:35 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:15:36.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:35 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:15:36.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:35 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:15:36.351 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:15:36.393 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:15:36.434 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:36.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:36.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:15:36.536 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:15:36.560 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:36.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:36.561 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:15:36.662 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:15:36.687 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:36.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:36.688 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:15:36.789 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:15:36.813 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:36.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:36.815 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:15:36.892 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:15:36.892 INFO:teuthology.orchestra.run.smithi045.stdout: "size": 20, 2024-08-19T14:15:36.892 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 600, 2024-08-19T14:15:36.893 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [ 2024-08-19T14:15:36.893 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.893 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24580.0:10284 2.5 2:a4caf01e:::benchmark_data_smithi045_46415_object10283:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-08-19T14:15:36.893 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:12:52.336926+0000", 2024-08-19T14:15:36.893 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 164.55337079099999, 2024-08-19T14:15:36.893 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.206277084, 2024-08-19T14:15:36.893 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:15:36.893 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:15:36.893 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:15:36.893 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:15:36.893 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24580", 2024-08-19T14:15:36.893 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:15:36.893 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 10284 2024-08-19T14:15:36.893 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.894 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:15:36.894 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.894 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:15:36.894 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.336926+0000", 2024-08-19T14:15:36.894 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:15:36.894 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.894 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.894 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:15:36.894 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.336926+0000", 2024-08-19T14:15:36.894 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:15:36.894 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.894 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.894 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:15:36.895 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.336927+0000", 2024-08-19T14:15:36.895 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3370000000000001e-06 2024-08-19T14:15:36.895 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.895 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.895 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:15:36.895 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.336949+0000", 2024-08-19T14:15:36.895 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1399e-05 2024-08-19T14:15:36.895 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.895 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.895 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:15:36.895 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.336961+0000", 2024-08-19T14:15:36.895 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.224e-05 2024-08-19T14:15:36.895 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.895 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.896 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:15:36.896 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.336968+0000", 2024-08-19T14:15:36.896 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.2710000000000003e-06 2024-08-19T14:15:36.896 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.896 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.896 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:15:36.896 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.337134+0000", 2024-08-19T14:15:36.896 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000165809 2024-08-19T14:15:36.896 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.896 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.896 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:15:36.896 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.337251+0000", 2024-08-19T14:15:36.896 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000116848 2024-08-19T14:15:36.897 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.897 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.897 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,4", 2024-08-19T14:15:36.897 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.337803+0000", 2024-08-19T14:15:36.897 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00055147699999999998 2024-08-19T14:15:36.897 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.897 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.897 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:15:36.897 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.339273+0000", 2024-08-19T14:15:36.897 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0014706750000000001 2024-08-19T14:15:36.897 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.897 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.897 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:36.898 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.339438+0000", 2024-08-19T14:15:36.898 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00016459299999999999 2024-08-19T14:15:36.898 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.898 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.898 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:36.898 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.543107+0000", 2024-08-19T14:15:36.898 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20366880100000001 2024-08-19T14:15:36.898 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.898 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.898 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:15:36.899 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.543162+0000", 2024-08-19T14:15:36.899 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.5075999999999999e-05 2024-08-19T14:15:36.899 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.899 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.899 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:15:36.899 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.543203+0000", 2024-08-19T14:15:36.899 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.1557999999999998e-05 2024-08-19T14:15:36.899 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:15:36.899 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:15:36.899 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:15:36.899 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.899 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.899 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24580.0:10301 2.f 2:f82f4cfa:::benchmark_data_smithi045_46415_object10300:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-08-19T14:15:36.900 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:12:52.341171+0000", 2024-08-19T14:15:36.900 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 164.549125946, 2024-08-19T14:15:36.900 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20191952899999999, 2024-08-19T14:15:36.900 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:15:36.900 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:15:36.900 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:15:36.900 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:15:36.900 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24580", 2024-08-19T14:15:36.900 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:15:36.900 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 10301 2024-08-19T14:15:36.900 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.900 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:15:36.900 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.901 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:15:36.901 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.341171+0000", 2024-08-19T14:15:36.901 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:15:36.901 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.901 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.901 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:15:36.901 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.341171+0000", 2024-08-19T14:15:36.901 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:15:36.901 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.901 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.901 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:15:36.901 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.341172+0000", 2024-08-19T14:15:36.901 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.093e-06 2024-08-19T14:15:36.901 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.902 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.902 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:15:36.902 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.341195+0000", 2024-08-19T14:15:36.902 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2435000000000001e-05 2024-08-19T14:15:36.902 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.902 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.902 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:15:36.902 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.341205+0000", 2024-08-19T14:15:36.902 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0567999999999999e-05 2024-08-19T14:15:36.902 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.902 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.902 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:15:36.902 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.341212+0000", 2024-08-19T14:15:36.903 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.686e-06 2024-08-19T14:15:36.903 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.903 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.903 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:15:36.903 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.341290+0000", 2024-08-19T14:15:36.903 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.8572999999999995e-05 2024-08-19T14:15:36.903 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.903 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.903 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:15:36.903 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.341427+0000", 2024-08-19T14:15:36.903 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000136754 2024-08-19T14:15:36.903 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.903 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.904 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 4,9", 2024-08-19T14:15:36.904 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.341748+0000", 2024-08-19T14:15:36.904 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00032119000000000001 2024-08-19T14:15:36.904 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.904 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.904 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:36.904 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.342661+0000", 2024-08-19T14:15:36.904 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000912761 2024-08-19T14:15:36.904 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.904 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.904 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:15:36.904 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.342681+0000", 2024-08-19T14:15:36.904 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0316999999999999e-05 2024-08-19T14:15:36.904 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.905 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.905 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:36.905 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.543009+0000", 2024-08-19T14:15:36.905 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.200327793 2024-08-19T14:15:36.905 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.905 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.905 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:15:36.905 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.543053+0000", 2024-08-19T14:15:36.905 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.4036000000000003e-05 2024-08-19T14:15:36.905 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.906 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.906 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:15:36.906 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.543091+0000", 2024-08-19T14:15:36.906 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.7323000000000002e-05 2024-08-19T14:15:36.906 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:15:36.906 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:15:36.906 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:15:36.906 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.906 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.906 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24580.0:10310 2.5 2:a1615b0f:::benchmark_data_smithi045_46415_object10309:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-08-19T14:15:36.906 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:12:52.342440+0000", 2024-08-19T14:15:36.906 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 164.54785714299999, 2024-08-19T14:15:36.906 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20090964, 2024-08-19T14:15:36.907 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:15:36.907 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:15:36.907 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:15:36.907 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:15:36.907 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24580", 2024-08-19T14:15:36.908 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:15:36.908 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 10310 2024-08-19T14:15:36.908 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.908 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:15:36.908 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.908 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:15:36.908 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.342440+0000", 2024-08-19T14:15:36.908 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:15:36.908 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.908 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.908 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:15:36.908 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.342440+0000", 2024-08-19T14:15:36.908 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:15:36.909 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.909 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.909 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:15:36.909 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.342441+0000", 2024-08-19T14:15:36.909 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.7499999999999998e-07 2024-08-19T14:15:36.909 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.909 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.909 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:15:36.909 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.342463+0000", 2024-08-19T14:15:36.909 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2303999999999999e-05 2024-08-19T14:15:36.909 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.909 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.909 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:15:36.909 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.342477+0000", 2024-08-19T14:15:36.910 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4099e-05 2024-08-19T14:15:36.910 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.910 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.910 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:15:36.910 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.342485+0000", 2024-08-19T14:15:36.910 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.6490000000000008e-06 2024-08-19T14:15:36.910 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.910 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.910 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:15:36.910 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.342571+0000", 2024-08-19T14:15:36.910 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.6238999999999999e-05 2024-08-19T14:15:36.910 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.910 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.911 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:15:36.911 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.343481+0000", 2024-08-19T14:15:36.911 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00091020900000000004 2024-08-19T14:15:36.911 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.911 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.911 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,4", 2024-08-19T14:15:36.911 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.343622+0000", 2024-08-19T14:15:36.911 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014069700000000001 2024-08-19T14:15:36.911 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.911 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.911 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:15:36.911 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.344447+0000", 2024-08-19T14:15:36.911 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00082507499999999998 2024-08-19T14:15:36.911 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.912 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.912 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:36.912 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.346610+0000", 2024-08-19T14:15:36.912 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0021627830000000002 2024-08-19T14:15:36.912 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.912 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.912 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:36.912 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.543255+0000", 2024-08-19T14:15:36.912 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.19664540699999999 2024-08-19T14:15:36.912 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.912 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.912 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:15:36.912 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.543309+0000", 2024-08-19T14:15:36.912 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4231999999999998e-05 2024-08-19T14:15:36.913 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.913 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.913 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:15:36.913 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.543349+0000", 2024-08-19T14:15:36.913 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.9971000000000001e-05 2024-08-19T14:15:36.913 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:15:36.913 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:15:36.913 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:15:36.913 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.913 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.913 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24580.0:10315 2.f 2:fe6f4b87:::benchmark_data_smithi045_46415_object10314:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-08-19T14:15:36.913 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:12:52.343698+0000", 2024-08-19T14:15:36.913 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 164.546599133, 2024-08-19T14:15:36.913 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.19949809700000001, 2024-08-19T14:15:36.914 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:15:36.914 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:15:36.914 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:15:36.914 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:15:36.914 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24580", 2024-08-19T14:15:36.914 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:15:36.914 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 10315 2024-08-19T14:15:36.914 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.914 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:15:36.914 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.914 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:15:36.914 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.343698+0000", 2024-08-19T14:15:36.914 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:15:36.914 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.915 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.915 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:15:36.915 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.343698+0000", 2024-08-19T14:15:36.915 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:15:36.915 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.915 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.915 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:15:36.915 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.343699+0000", 2024-08-19T14:15:36.915 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.238e-06 2024-08-19T14:15:36.915 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.915 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.915 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:15:36.915 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.343723+0000", 2024-08-19T14:15:36.916 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.4314000000000001e-05 2024-08-19T14:15:36.916 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.916 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.916 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:15:36.916 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.343735+0000", 2024-08-19T14:15:36.916 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2133999999999999e-05 2024-08-19T14:15:36.916 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.916 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.916 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:15:36.916 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.343743+0000", 2024-08-19T14:15:36.916 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.34e-06 2024-08-19T14:15:36.916 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.916 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.916 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:15:36.917 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.344026+0000", 2024-08-19T14:15:36.917 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00028303099999999999 2024-08-19T14:15:36.917 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.917 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.917 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:15:36.917 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.344278+0000", 2024-08-19T14:15:36.917 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00025181699999999999 2024-08-19T14:15:36.917 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.917 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.917 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 4,9", 2024-08-19T14:15:36.917 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.344520+0000", 2024-08-19T14:15:36.917 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000241864 2024-08-19T14:15:36.917 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.917 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.918 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:15:36.918 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.346213+0000", 2024-08-19T14:15:36.918 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001693398 2024-08-19T14:15:36.918 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.918 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.918 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:36.918 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.346420+0000", 2024-08-19T14:15:36.918 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00020658300000000001 2024-08-19T14:15:36.918 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.918 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.918 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:36.918 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.543130+0000", 2024-08-19T14:15:36.918 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.196710037 2024-08-19T14:15:36.918 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.919 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.919 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:15:36.919 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.543161+0000", 2024-08-19T14:15:36.919 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.1306999999999997e-05 2024-08-19T14:15:36.919 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.919 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.919 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:15:36.919 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.543196+0000", 2024-08-19T14:15:36.919 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.5033999999999997e-05 2024-08-19T14:15:36.919 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:15:36.919 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:15:36.919 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:15:36.919 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.919 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.920 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24580.0:10322 2.5 2:ad7f7050:::benchmark_data_smithi045_46415_object10321:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-08-19T14:15:36.920 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:12:52.345086+0000", 2024-08-19T14:15:36.920 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 164.54521104599999, 2024-08-19T14:15:36.920 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.19865560199999999, 2024-08-19T14:15:36.920 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:15:36.920 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:15:36.920 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:15:36.920 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:15:36.920 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24580", 2024-08-19T14:15:36.920 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:15:36.920 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 10322 2024-08-19T14:15:36.920 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.920 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:15:36.920 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.921 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:15:36.921 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.345086+0000", 2024-08-19T14:15:36.921 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:15:36.921 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.921 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.921 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:15:36.921 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.345086+0000", 2024-08-19T14:15:36.921 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:15:36.921 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.921 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.921 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:15:36.921 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.345087+0000", 2024-08-19T14:15:36.921 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.8299999999999996e-07 2024-08-19T14:15:36.922 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.922 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.922 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:15:36.922 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.345106+0000", 2024-08-19T14:15:36.922 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.91e-05 2024-08-19T14:15:36.922 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.922 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.922 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:15:36.922 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.345114+0000", 2024-08-19T14:15:36.922 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.6200000000000005e-06 2024-08-19T14:15:36.922 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.922 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.922 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:15:36.922 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.345122+0000", 2024-08-19T14:15:36.923 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.8310000000000001e-06 2024-08-19T14:15:36.923 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.923 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.923 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:15:36.923 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.345281+0000", 2024-08-19T14:15:36.923 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000158386 2024-08-19T14:15:36.923 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.923 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.923 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:15:36.923 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.345396+0000", 2024-08-19T14:15:36.923 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000115299 2024-08-19T14:15:36.923 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.923 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.924 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,4", 2024-08-19T14:15:36.924 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.345628+0000", 2024-08-19T14:15:36.924 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00023218499999999999 2024-08-19T14:15:36.924 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.924 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.924 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:15:36.924 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.347682+0000", 2024-08-19T14:15:36.924 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0020542189999999999 2024-08-19T14:15:36.924 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.924 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.924 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:36.924 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.348039+0000", 2024-08-19T14:15:36.924 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00035721800000000002 2024-08-19T14:15:36.924 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.925 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.925 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:36.925 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.543654+0000", 2024-08-19T14:15:36.925 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.19561498799999999 2024-08-19T14:15:36.925 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.925 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.925 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:15:36.925 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.543708+0000", 2024-08-19T14:15:36.925 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.3152999999999997e-05 2024-08-19T14:15:36.925 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.925 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.925 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:15:36.925 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.543741+0000", 2024-08-19T14:15:36.925 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.392e-05 2024-08-19T14:15:36.926 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:15:36.926 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:15:36.926 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:15:36.926 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.926 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.926 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24580.0:10329 2.5 2:a14cf373:::benchmark_data_smithi045_46415_object10328:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-08-19T14:15:36.926 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:12:52.346732+0000", 2024-08-19T14:15:36.926 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 164.54356470900001, 2024-08-19T14:15:36.926 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.197173301, 2024-08-19T14:15:36.926 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:15:36.926 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:15:36.926 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:15:36.926 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:15:36.927 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24580", 2024-08-19T14:15:36.927 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:15:36.927 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 10329 2024-08-19T14:15:36.927 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.927 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:15:36.927 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.927 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:15:36.927 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.346732+0000", 2024-08-19T14:15:36.927 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:15:36.927 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.927 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.927 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:15:36.927 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.346732+0000", 2024-08-19T14:15:36.927 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:15:36.928 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.928 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.928 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:15:36.928 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.346733+0000", 2024-08-19T14:15:36.928 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.4799999999999997e-07 2024-08-19T14:15:36.928 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.928 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.928 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:15:36.928 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.346755+0000", 2024-08-19T14:15:36.928 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1695000000000002e-05 2024-08-19T14:15:36.928 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.928 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.928 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:15:36.928 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.346768+0000", 2024-08-19T14:15:36.929 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2951e-05 2024-08-19T14:15:36.929 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.929 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.929 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:15:36.929 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.346773+0000", 2024-08-19T14:15:36.929 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4659999999999998e-06 2024-08-19T14:15:36.929 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.929 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.929 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:15:36.929 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.346886+0000", 2024-08-19T14:15:36.929 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000112957 2024-08-19T14:15:36.929 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.929 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.929 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:15:36.930 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.347003+0000", 2024-08-19T14:15:36.930 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000117138 2024-08-19T14:15:36.930 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.930 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.930 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,4", 2024-08-19T14:15:36.930 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.347219+0000", 2024-08-19T14:15:36.930 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00021526999999999999 2024-08-19T14:15:36.930 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.930 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.930 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:15:36.930 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.347921+0000", 2024-08-19T14:15:36.930 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00070217000000000001 2024-08-19T14:15:36.930 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.931 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.931 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:36.931 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.349966+0000", 2024-08-19T14:15:36.931 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0020451800000000002 2024-08-19T14:15:36.931 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.931 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.931 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:36.931 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.543791+0000", 2024-08-19T14:15:36.931 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.19382521 2024-08-19T14:15:36.931 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.931 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.931 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:15:36.931 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.543848+0000", 2024-08-19T14:15:36.931 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7153e-05 2024-08-19T14:15:36.932 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.932 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.932 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:15:36.932 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.543906+0000", 2024-08-19T14:15:36.932 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7163000000000001e-05 2024-08-19T14:15:36.932 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:15:36.932 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:15:36.932 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:15:36.932 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.932 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.932 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24580.0:10333 2.5 2:abe3c6e5:::benchmark_data_smithi045_46415_object10332:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-08-19T14:15:36.932 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:12:52.348039+0000", 2024-08-19T14:15:36.932 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 164.54225786399999, 2024-08-19T14:15:36.932 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.19636310100000001, 2024-08-19T14:15:36.933 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:15:36.933 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:15:36.933 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:15:36.933 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:15:36.933 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24580", 2024-08-19T14:15:36.933 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:15:36.933 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 10333 2024-08-19T14:15:36.933 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.933 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:15:36.933 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.933 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:15:36.933 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.348039+0000", 2024-08-19T14:15:36.933 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:15:36.934 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.934 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.934 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:15:36.934 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.348039+0000", 2024-08-19T14:15:36.934 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:15:36.934 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.934 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.934 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:15:36.934 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.348040+0000", 2024-08-19T14:15:36.934 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.0599999999999999e-07 2024-08-19T14:15:36.934 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.934 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.934 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:15:36.935 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.348064+0000", 2024-08-19T14:15:36.935 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.3768e-05 2024-08-19T14:15:36.935 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.935 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.935 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:15:36.935 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.348075+0000", 2024-08-19T14:15:36.935 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1083999999999999e-05 2024-08-19T14:15:36.935 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.935 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.935 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:15:36.935 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.348082+0000", 2024-08-19T14:15:36.935 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.3010000000000001e-06 2024-08-19T14:15:36.935 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.935 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.936 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:15:36.936 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.348125+0000", 2024-08-19T14:15:36.936 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.2559999999999999e-05 2024-08-19T14:15:36.936 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.936 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.936 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:15:36.936 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.348235+0000", 2024-08-19T14:15:36.936 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00011069300000000001 2024-08-19T14:15:36.936 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.936 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.936 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,4", 2024-08-19T14:15:36.936 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.348341+0000", 2024-08-19T14:15:36.936 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00010518099999999999 2024-08-19T14:15:36.937 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.937 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.937 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:15:36.937 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.349463+0000", 2024-08-19T14:15:36.937 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0011227909999999999 2024-08-19T14:15:36.937 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.937 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.937 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:36.937 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.350404+0000", 2024-08-19T14:15:36.937 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00094061000000000004 2024-08-19T14:15:36.937 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.937 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.937 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:36.937 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.544288+0000", 2024-08-19T14:15:36.938 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.19388386399999999 2024-08-19T14:15:36.938 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.938 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.938 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:15:36.938 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.544356+0000", 2024-08-19T14:15:36.938 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.8329000000000003e-05 2024-08-19T14:15:36.938 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.938 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.938 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:15:36.938 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.544402+0000", 2024-08-19T14:15:36.938 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.6013999999999997e-05 2024-08-19T14:15:36.938 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:15:36.938 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:15:36.939 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:15:36.939 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.939 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.939 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24580.0:10336 2.5 2:a6b0c637:::benchmark_data_smithi045_46415_object10335:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-08-19T14:15:36.939 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:12:52.348409+0000", 2024-08-19T14:15:36.939 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 164.54188746099999, 2024-08-19T14:15:36.939 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.19615854299999999, 2024-08-19T14:15:36.939 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:15:36.939 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:15:36.939 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:15:36.939 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:15:36.939 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24580", 2024-08-19T14:15:36.939 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:15:36.940 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 10336 2024-08-19T14:15:36.940 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.940 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:15:36.940 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.940 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:15:36.940 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.348409+0000", 2024-08-19T14:15:36.940 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:15:36.940 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.940 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.940 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:15:36.940 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.348409+0000", 2024-08-19T14:15:36.940 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:15:36.940 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.940 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.941 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:15:36.941 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.348410+0000", 2024-08-19T14:15:36.941 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4799999999999997e-07 2024-08-19T14:15:36.941 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.941 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.941 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:15:36.941 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.348431+0000", 2024-08-19T14:15:36.941 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0757000000000001e-05 2024-08-19T14:15:36.941 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.941 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.941 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:15:36.941 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.348442+0000", 2024-08-19T14:15:36.941 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0611000000000001e-05 2024-08-19T14:15:36.942 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.942 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.942 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:15:36.942 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.348449+0000", 2024-08-19T14:15:36.942 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.9949999999999999e-06 2024-08-19T14:15:36.942 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.942 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.942 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:15:36.942 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.348754+0000", 2024-08-19T14:15:36.942 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00030516199999999999 2024-08-19T14:15:36.942 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.942 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.942 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:15:36.942 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.348885+0000", 2024-08-19T14:15:36.943 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013173500000000001 2024-08-19T14:15:36.943 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.943 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.943 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,4", 2024-08-19T14:15:36.943 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.349038+0000", 2024-08-19T14:15:36.943 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015267899999999999 2024-08-19T14:15:36.943 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.943 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.943 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:15:36.943 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.349637+0000", 2024-08-19T14:15:36.943 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00059873499999999998 2024-08-19T14:15:36.943 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.943 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.944 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:36.944 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.350437+0000", 2024-08-19T14:15:36.944 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00079989500000000001 2024-08-19T14:15:36.944 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.944 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.944 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:36.944 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.544475+0000", 2024-08-19T14:15:36.944 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.19403847499999999 2024-08-19T14:15:36.944 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.944 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.944 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:15:36.944 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.544531+0000", 2024-08-19T14:15:36.944 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.5288000000000003e-05 2024-08-19T14:15:36.944 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.945 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.945 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:15:36.945 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.544568+0000", 2024-08-19T14:15:36.945 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.7462999999999997e-05 2024-08-19T14:15:36.945 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:15:36.945 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:15:36.945 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:15:36.945 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.945 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.945 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24580.0:10342 2.5 2:aaeb89b1:::benchmark_data_smithi045_46415_object10341:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-08-19T14:15:36.945 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:12:52.350928+0000", 2024-08-19T14:15:36.945 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 164.539368913, 2024-08-19T14:15:36.945 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.19439313599999999, 2024-08-19T14:15:36.946 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:15:36.946 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:15:36.946 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:15:36.946 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:15:36.946 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24580", 2024-08-19T14:15:36.946 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:15:36.946 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 10342 2024-08-19T14:15:36.946 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.946 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:15:36.946 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.946 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:15:36.946 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.350928+0000", 2024-08-19T14:15:36.946 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:15:36.947 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.947 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.947 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:15:36.947 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.350928+0000", 2024-08-19T14:15:36.947 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:15:36.947 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.947 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.947 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:15:36.947 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.350929+0000", 2024-08-19T14:15:36.947 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.3399999999999998e-07 2024-08-19T14:15:36.947 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.947 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.947 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:15:36.947 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.350951+0000", 2024-08-19T14:15:36.948 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2436999999999999e-05 2024-08-19T14:15:36.948 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.948 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.948 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:15:36.948 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.350961+0000", 2024-08-19T14:15:36.948 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.3039999999999994e-06 2024-08-19T14:15:36.948 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.948 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.948 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:15:36.948 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.350967+0000", 2024-08-19T14:15:36.948 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.8109999999999997e-06 2024-08-19T14:15:36.948 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.948 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.948 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:15:36.949 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.351045+0000", 2024-08-19T14:15:36.949 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.7655000000000001e-05 2024-08-19T14:15:36.949 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.949 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.949 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:15:36.949 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.351364+0000", 2024-08-19T14:15:36.949 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:15:36.951 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00031882500000000001 2024-08-19T14:15:36.951 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.951 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.951 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,4", 2024-08-19T14:15:36.951 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.351549+0000", 2024-08-19T14:15:36.951 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00018494900000000001 2024-08-19T14:15:36.951 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.951 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.952 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:36.952 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.352871+0000", 2024-08-19T14:15:36.952 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0013225579999999999 2024-08-19T14:15:36.952 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.952 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.952 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:15:36.952 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.352993+0000", 2024-08-19T14:15:36.952 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012199199999999999 2024-08-19T14:15:36.952 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.952 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.952 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:36.952 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.545110+0000", 2024-08-19T14:15:36.952 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.192116701 2024-08-19T14:15:36.953 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.953 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.953 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:15:36.953 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.545188+0000", 2024-08-19T14:15:36.953 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.8238000000000005e-05 2024-08-19T14:15:36.953 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.953 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.953 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:15:36.953 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.545321+0000", 2024-08-19T14:15:36.953 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013283199999999999 2024-08-19T14:15:36.953 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:15:36.953 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:15:36.953 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:15:36.953 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.954 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.954 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24580.0:10345 2.5 2:a57a0c16:::benchmark_data_smithi045_46415_object10344:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-08-19T14:15:36.954 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:12:52.351521+0000", 2024-08-19T14:15:36.954 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 164.53877617500001, 2024-08-19T14:15:36.954 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.193940009, 2024-08-19T14:15:36.954 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:15:36.954 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:15:36.954 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:15:36.954 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:15:36.954 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24580", 2024-08-19T14:15:36.954 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:15:36.954 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 10345 2024-08-19T14:15:36.954 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.955 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:15:36.955 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.955 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:15:36.955 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.351521+0000", 2024-08-19T14:15:36.955 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:15:36.955 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.955 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.955 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:15:36.955 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.351521+0000", 2024-08-19T14:15:36.955 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:15:36.955 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.955 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.955 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:15:36.955 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.351522+0000", 2024-08-19T14:15:36.956 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.09e-07 2024-08-19T14:15:36.956 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.956 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.956 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:15:36.956 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.351545+0000", 2024-08-19T14:15:36.956 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.3558e-05 2024-08-19T14:15:36.956 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.956 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.956 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:15:36.956 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.351557+0000", 2024-08-19T14:15:36.956 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2258e-05 2024-08-19T14:15:36.956 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.956 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.956 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:15:36.957 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.351565+0000", 2024-08-19T14:15:36.957 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.7600000000000002e-06 2024-08-19T14:15:36.957 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.957 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.957 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:15:36.957 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.352108+0000", 2024-08-19T14:15:36.957 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00054303399999999996 2024-08-19T14:15:36.957 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.957 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.957 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:15:36.957 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.352261+0000", 2024-08-19T14:15:36.957 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015242599999999999 2024-08-19T14:15:36.957 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.957 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.958 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,4", 2024-08-19T14:15:36.958 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.352406+0000", 2024-08-19T14:15:36.958 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014495099999999999 2024-08-19T14:15:36.958 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.958 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.958 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:15:36.958 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.353120+0000", 2024-08-19T14:15:36.958 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00071412699999999995 2024-08-19T14:15:36.958 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.958 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.958 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:36.958 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.353796+0000", 2024-08-19T14:15:36.958 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00067595899999999996 2024-08-19T14:15:36.958 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.959 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.959 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:36.959 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.545371+0000", 2024-08-19T14:15:36.959 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.19157498000000001 2024-08-19T14:15:36.959 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.959 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.959 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:15:36.959 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.545424+0000", 2024-08-19T14:15:36.959 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.3251000000000002e-05 2024-08-19T14:15:36.959 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.959 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.959 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:15:36.959 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.545461+0000", 2024-08-19T14:15:36.959 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.6795999999999998e-05 2024-08-19T14:15:36.960 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:15:36.960 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:15:36.960 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:15:36.960 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.960 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.960 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24580.0:10370 2.f 2:f05e8536:::benchmark_data_smithi045_46415_object10369:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-08-19T14:15:36.960 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:12:52.359955+0000", 2024-08-19T14:15:36.960 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 164.53034217999999, 2024-08-19T14:15:36.960 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.183329298, 2024-08-19T14:15:36.960 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:15:36.960 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:15:36.960 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:15:36.960 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:15:36.960 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24580", 2024-08-19T14:15:36.961 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:15:36.961 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 10370 2024-08-19T14:15:36.961 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.961 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:15:36.961 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.961 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:15:36.961 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.359955+0000", 2024-08-19T14:15:36.961 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:15:36.961 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.961 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.961 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:15:36.961 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.359955+0000", 2024-08-19T14:15:36.961 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:15:36.961 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.962 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.962 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:15:36.962 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.359956+0000", 2024-08-19T14:15:36.962 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.4e-07 2024-08-19T14:15:36.962 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.962 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.962 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:15:36.962 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.359977+0000", 2024-08-19T14:15:36.962 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1586e-05 2024-08-19T14:15:36.962 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.962 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.962 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:15:36.962 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.359985+0000", 2024-08-19T14:15:36.963 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.6199999999999999e-06 2024-08-19T14:15:36.963 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.963 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.963 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:15:36.963 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.359992+0000", 2024-08-19T14:15:36.963 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.289e-06 2024-08-19T14:15:36.963 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.963 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.963 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:15:36.963 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.360044+0000", 2024-08-19T14:15:36.963 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1924e-05 2024-08-19T14:15:36.963 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.963 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.963 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:15:36.963 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.360167+0000", 2024-08-19T14:15:36.964 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000123103 2024-08-19T14:15:36.964 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.964 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.964 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 4,9", 2024-08-19T14:15:36.964 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.360261+0000", 2024-08-19T14:15:36.964 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.4015e-05 2024-08-19T14:15:36.964 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.964 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.964 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:15:36.964 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.360806+0000", 2024-08-19T14:15:36.964 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00054517699999999999 2024-08-19T14:15:36.964 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.964 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.964 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:36.965 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.361184+0000", 2024-08-19T14:15:36.965 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00037772899999999999 2024-08-19T14:15:36.965 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.965 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.965 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:36.965 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.543236+0000", 2024-08-19T14:15:36.965 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.182052097 2024-08-19T14:15:36.965 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.965 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.965 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:15:36.965 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.543264+0000", 2024-08-19T14:15:36.965 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.8217000000000001e-05 2024-08-19T14:15:36.965 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.966 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.966 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:15:36.966 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.543284+0000", 2024-08-19T14:15:36.966 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9701e-05 2024-08-19T14:15:36.966 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:15:36.966 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:15:36.966 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:15:36.966 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.966 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.966 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24580.0:10385 2.f 2:fbb28454:::benchmark_data_smithi045_46415_object10384:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-08-19T14:15:36.966 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:12:52.363881+0000", 2024-08-19T14:15:36.966 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 164.52641578999999, 2024-08-19T14:15:36.966 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.179574126, 2024-08-19T14:15:36.966 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:15:36.967 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:15:36.967 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:15:36.967 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:15:36.967 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24580", 2024-08-19T14:15:36.967 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:15:36.967 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 10385 2024-08-19T14:15:36.967 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.967 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:15:36.967 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.967 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:15:36.967 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.363881+0000", 2024-08-19T14:15:36.967 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:15:36.967 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.967 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.968 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:15:36.968 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.363881+0000", 2024-08-19T14:15:36.968 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:15:36.968 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.968 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.968 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:15:36.968 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.363882+0000", 2024-08-19T14:15:36.968 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.1999999999999998e-07 2024-08-19T14:15:36.968 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.968 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.968 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:15:36.968 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.363899+0000", 2024-08-19T14:15:36.968 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7152000000000001e-05 2024-08-19T14:15:36.968 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.969 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.969 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:15:36.969 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.363910+0000", 2024-08-19T14:15:36.969 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0507e-05 2024-08-19T14:15:36.969 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.969 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.969 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:15:36.969 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.363915+0000", 2024-08-19T14:15:36.969 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.203e-06 2024-08-19T14:15:36.969 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.969 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.969 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:15:36.969 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.363956+0000", 2024-08-19T14:15:36.969 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.1127000000000001e-05 2024-08-19T14:15:36.970 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.970 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.970 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:15:36.970 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.364036+0000", 2024-08-19T14:15:36.970 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.0358999999999994e-05 2024-08-19T14:15:36.970 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.970 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.970 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 4,9", 2024-08-19T14:15:36.970 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.364118+0000", 2024-08-19T14:15:36.970 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.1799999999999996e-05 2024-08-19T14:15:36.970 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.970 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.970 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:15:36.971 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.364695+0000", 2024-08-19T14:15:36.971 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00057727699999999996 2024-08-19T14:15:36.971 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.971 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.971 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:36.971 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.364906+0000", 2024-08-19T14:15:36.971 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000210604 2024-08-19T14:15:36.971 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.971 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.971 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:36.971 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.543337+0000", 2024-08-19T14:15:36.971 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.17843088200000001 2024-08-19T14:15:36.971 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.971 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.972 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:15:36.972 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.543412+0000", 2024-08-19T14:15:36.972 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.5306e-05 2024-08-19T14:15:36.972 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.972 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.972 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:15:36.972 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.543455+0000", 2024-08-19T14:15:36.972 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.3089e-05 2024-08-19T14:15:36.972 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:15:36.972 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:15:36.972 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:15:36.972 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.972 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.973 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24580.0:10386 2.5 2:a9495656:::benchmark_data_smithi045_46415_object10385:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-08-19T14:15:36.973 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:12:52.365022+0000", 2024-08-19T14:15:36.973 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 164.52527479400001, 2024-08-19T14:15:36.973 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.18059128999999999, 2024-08-19T14:15:36.973 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:15:36.973 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:15:36.973 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:15:36.973 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:15:36.973 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24580", 2024-08-19T14:15:36.973 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:15:36.973 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 10386 2024-08-19T14:15:36.973 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.973 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:15:36.973 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.974 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:15:36.974 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.365022+0000", 2024-08-19T14:15:36.974 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:15:36.974 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.974 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.974 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:15:36.974 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.365022+0000", 2024-08-19T14:15:36.974 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:15:36.974 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.974 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.974 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:15:36.974 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.365023+0000", 2024-08-19T14:15:36.974 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.0900000000000001e-07 2024-08-19T14:15:36.975 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.975 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.975 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:15:36.975 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.365043+0000", 2024-08-19T14:15:36.975 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0115999999999998e-05 2024-08-19T14:15:36.975 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.975 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.975 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:15:36.975 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.365057+0000", 2024-08-19T14:15:36.975 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3665e-05 2024-08-19T14:15:36.975 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.975 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.975 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:15:36.975 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.365064+0000", 2024-08-19T14:15:36.976 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.7049999999999999e-06 2024-08-19T14:15:36.976 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.976 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.976 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:15:36.976 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.365107+0000", 2024-08-19T14:15:36.976 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.2422999999999997e-05 2024-08-19T14:15:36.976 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.976 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.976 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:15:36.976 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.365181+0000", 2024-08-19T14:15:36.976 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4196999999999995e-05 2024-08-19T14:15:36.976 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.976 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.976 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,4", 2024-08-19T14:15:36.977 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.365262+0000", 2024-08-19T14:15:36.977 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.1427999999999994e-05 2024-08-19T14:15:36.977 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.977 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.977 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:15:36.977 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.365792+0000", 2024-08-19T14:15:36.977 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00053006699999999995 2024-08-19T14:15:36.977 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.977 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.977 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:36.977 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.366086+0000", 2024-08-19T14:15:36.977 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00029387100000000002 2024-08-19T14:15:36.977 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.977 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.978 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:36.978 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.545532+0000", 2024-08-19T14:15:36.978 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.17944597700000001 2024-08-19T14:15:36.978 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.978 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.978 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:15:36.978 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.545577+0000", 2024-08-19T14:15:36.978 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.4459000000000002e-05 2024-08-19T14:15:36.978 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.978 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.978 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:15:36.978 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.545613+0000", 2024-08-19T14:15:36.978 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.6672999999999999e-05 2024-08-19T14:15:36.979 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:15:36.979 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:15:36.979 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:15:36.979 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.979 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.979 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24580.0:10391 2.5 2:af36409f:::benchmark_data_smithi045_46415_object10390:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-08-19T14:15:36.979 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:12:52.367100+0000", 2024-08-19T14:15:36.979 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 164.523197248, 2024-08-19T14:15:36.979 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.17861413200000001, 2024-08-19T14:15:36.979 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:15:36.979 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:15:36.979 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:15:36.979 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:15:36.980 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24580", 2024-08-19T14:15:36.980 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:15:36.980 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 10391 2024-08-19T14:15:36.980 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.980 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:15:36.980 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.980 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:15:36.980 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.367100+0000", 2024-08-19T14:15:36.980 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:15:36.980 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.980 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.980 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:15:36.980 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.367100+0000", 2024-08-19T14:15:36.980 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:15:36.981 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.981 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.981 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:15:36.981 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.367101+0000", 2024-08-19T14:15:36.981 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.1399999999999996e-07 2024-08-19T14:15:36.981 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.981 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.981 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:15:36.981 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.367119+0000", 2024-08-19T14:15:36.981 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8287e-05 2024-08-19T14:15:36.981 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.981 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.981 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:15:36.981 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.367130+0000", 2024-08-19T14:15:36.982 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.099e-05 2024-08-19T14:15:36.982 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.982 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.982 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:15:36.982 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.367135+0000", 2024-08-19T14:15:36.982 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.95e-06 2024-08-19T14:15:36.982 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.982 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.982 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:15:36.982 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.367172+0000", 2024-08-19T14:15:36.982 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.7387e-05 2024-08-19T14:15:36.982 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.982 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.983 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:15:36.983 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.367256+0000", 2024-08-19T14:15:36.983 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.4347999999999999e-05 2024-08-19T14:15:36.983 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.983 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.983 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,4", 2024-08-19T14:15:36.983 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.367354+0000", 2024-08-19T14:15:36.983 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.7762000000000004e-05 2024-08-19T14:15:36.983 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.983 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.983 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:15:36.983 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.367792+0000", 2024-08-19T14:15:36.983 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00043750400000000002 2024-08-19T14:15:36.983 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.984 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.984 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:36.984 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.368230+0000", 2024-08-19T14:15:36.984 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00043778399999999998 2024-08-19T14:15:36.984 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.984 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.984 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:36.984 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.545656+0000", 2024-08-19T14:15:36.984 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.177426475 2024-08-19T14:15:36.984 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.984 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.984 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:15:36.984 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.545690+0000", 2024-08-19T14:15:36.984 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.3952000000000003e-05 2024-08-19T14:15:36.984 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.985 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.985 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:15:36.985 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.545714+0000", 2024-08-19T14:15:36.985 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.3879e-05 2024-08-19T14:15:36.985 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:15:36.985 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:15:36.985 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:15:36.985 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.985 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.985 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24580.0:88314 2.f 2:fc0d2546:::benchmark_data_smithi045_46415_object88313:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-08-19T14:15:36.985 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:42.807395+0000", 2024-08-19T14:15:36.985 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 114.082901859, 2024-08-19T14:15:36.985 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.21007711900000001, 2024-08-19T14:15:36.986 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:15:36.986 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:15:36.986 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:15:36.986 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:15:36.986 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24580", 2024-08-19T14:15:36.986 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:15:36.986 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 88314 2024-08-19T14:15:36.986 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.986 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:15:36.986 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.986 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:15:36.986 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.807395+0000", 2024-08-19T14:15:36.986 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:15:36.986 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.987 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.987 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:15:36.987 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.807395+0000", 2024-08-19T14:15:36.987 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:15:36.987 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.987 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.987 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:15:36.987 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.807397+0000", 2024-08-19T14:15:36.987 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1239999999999999e-06 2024-08-19T14:15:36.987 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.987 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.987 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:15:36.987 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.807424+0000", 2024-08-19T14:15:36.987 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.6429000000000001e-05 2024-08-19T14:15:36.988 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.988 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.988 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:15:36.988 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.807445+0000", 2024-08-19T14:15:36.988 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1078e-05 2024-08-19T14:15:36.988 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.988 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.988 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:15:36.988 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.807455+0000", 2024-08-19T14:15:36.988 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0351e-05 2024-08-19T14:15:36.988 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.988 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.988 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:15:36.988 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.807524+0000", 2024-08-19T14:15:36.989 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.9122999999999995e-05 2024-08-19T14:15:36.989 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.989 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.989 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:15:36.989 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.807862+0000", 2024-08-19T14:15:36.989 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000338222 2024-08-19T14:15:36.989 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.989 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.989 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 4,9", 2024-08-19T14:15:36.989 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.808028+0000", 2024-08-19T14:15:36.989 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00016509700000000001 2024-08-19T14:15:36.989 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.989 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.989 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:15:36.990 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.811333+0000", 2024-08-19T14:15:36.990 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0033057300000000002 2024-08-19T14:15:36.990 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.990 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.990 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:36.990 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.814174+0000", 2024-08-19T14:15:36.990 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0028408959999999999 2024-08-19T14:15:36.990 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.990 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.990 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:36.990 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:43.017401+0000", 2024-08-19T14:15:36.990 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.203226459 2024-08-19T14:15:36.990 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.991 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.991 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:15:36.991 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:43.017443+0000", 2024-08-19T14:15:36.991 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.2438999999999998e-05 2024-08-19T14:15:36.991 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.991 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.991 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:15:36.991 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:43.017472+0000", 2024-08-19T14:15:36.991 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.9170999999999999e-05 2024-08-19T14:15:36.991 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:15:36.991 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:15:36.991 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:15:36.991 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.991 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.991 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24580.0:88316 2.5 2:ade27d21:::benchmark_data_smithi045_46415_object88315:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-08-19T14:15:36.992 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:42.807950+0000", 2024-08-19T14:15:36.992 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 114.08234690899999, 2024-08-19T14:15:36.992 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.21254900700000001, 2024-08-19T14:15:36.992 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:15:36.992 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:15:36.992 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:15:36.992 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:15:36.992 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24580", 2024-08-19T14:15:36.992 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:15:36.992 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 88316 2024-08-19T14:15:36.992 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.992 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:15:36.992 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.993 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:15:36.993 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.807950+0000", 2024-08-19T14:15:36.993 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:15:36.993 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.993 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.993 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:15:36.993 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.807950+0000", 2024-08-19T14:15:36.993 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:15:36.993 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.993 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.993 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:15:36.993 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.807951+0000", 2024-08-19T14:15:36.993 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.046e-06 2024-08-19T14:15:36.993 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.993 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.994 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:15:36.994 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.807985+0000", 2024-08-19T14:15:36.994 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.3565999999999997e-05 2024-08-19T14:15:36.994 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.994 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.994 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:15:36.994 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.807997+0000", 2024-08-19T14:15:36.994 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2690999999999999e-05 2024-08-19T14:15:36.994 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.994 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.994 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:15:36.994 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.808005+0000", 2024-08-19T14:15:36.994 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.8089999999999993e-06 2024-08-19T14:15:36.995 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.995 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.995 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:15:36.995 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.808063+0000", 2024-08-19T14:15:36.995 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.8108e-05 2024-08-19T14:15:36.995 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.995 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.995 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:15:36.995 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.808366+0000", 2024-08-19T14:15:36.995 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00030297899999999999 2024-08-19T14:15:36.995 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.995 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.995 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,4", 2024-08-19T14:15:36.995 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.808492+0000", 2024-08-19T14:15:36.996 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000125556 2024-08-19T14:15:36.996 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.996 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.996 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:15:36.996 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.811416+0000", 2024-08-19T14:15:36.996 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0029237540000000002 2024-08-19T14:15:36.996 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.996 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.996 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:36.996 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.818511+0000", 2024-08-19T14:15:36.996 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0070953980000000002 2024-08-19T14:15:36.996 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.996 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.996 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:36.997 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:43.020384+0000", 2024-08-19T14:15:36.997 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.201872945 2024-08-19T14:15:36.997 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.997 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.997 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:15:36.997 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:43.020458+0000", 2024-08-19T14:15:36.997 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.3699e-05 2024-08-19T14:15:36.997 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.997 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.997 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:15:36.997 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:43.020499+0000", 2024-08-19T14:15:36.997 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.1455999999999997e-05 2024-08-19T14:15:36.997 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:15:36.997 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:15:36.998 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:15:36.998 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.998 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.998 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24580.0:88321 2.5 2:a4cf6cb2:::benchmark_data_smithi045_46415_object88320:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-08-19T14:15:36.998 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:42.814619+0000", 2024-08-19T14:15:36.998 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 114.07567785800001, 2024-08-19T14:15:36.998 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20609303900000001, 2024-08-19T14:15:36.998 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:15:36.998 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:15:36.998 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:15:36.998 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:15:36.998 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24580", 2024-08-19T14:15:36.998 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:15:36.999 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 88321 2024-08-19T14:15:36.999 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.999 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:15:36.999 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.999 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:15:36.999 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.814619+0000", 2024-08-19T14:15:36.999 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:15:36.999 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.999 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:36.999 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:15:36.999 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.814619+0000", 2024-08-19T14:15:36.999 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:15:36.999 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:36.999 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:37.000 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:15:37.000 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.814621+0000", 2024-08-19T14:15:37.000 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.514e-06 2024-08-19T14:15:37.000 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:37.000 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:37.000 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:15:37.000 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.814641+0000", 2024-08-19T14:15:37.000 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0852e-05 2024-08-19T14:15:37.000 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:37.000 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:37.000 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:15:37.000 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.814653+0000", 2024-08-19T14:15:37.000 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1293e-05 2024-08-19T14:15:37.000 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:37.001 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:37.001 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:15:37.001 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.814660+0000", 2024-08-19T14:15:37.001 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.905e-06 2024-08-19T14:15:37.001 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:37.001 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:37.001 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:15:37.001 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.814711+0000", 2024-08-19T14:15:37.001 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1657999999999999e-05 2024-08-19T14:15:37.001 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:37.001 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:37.001 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:15:37.001 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.814825+0000", 2024-08-19T14:15:37.001 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000113597 2024-08-19T14:15:37.002 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:37.002 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:37.002 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,4", 2024-08-19T14:15:37.002 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.814949+0000", 2024-08-19T14:15:37.002 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012450800000000001 2024-08-19T14:15:37.002 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:37.002 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:37.002 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:15:37.002 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.817037+0000", 2024-08-19T14:15:37.002 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0020878089999999999 2024-08-19T14:15:37.002 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:37.002 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:37.002 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:37.002 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.819696+0000", 2024-08-19T14:15:37.003 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.002659004 2024-08-19T14:15:37.003 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:37.003 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:37.003 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:37.003 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:43.020593+0000", 2024-08-19T14:15:37.003 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20089647899999999 2024-08-19T14:15:37.003 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:37.003 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:37.003 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:15:37.003 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:43.020673+0000", 2024-08-19T14:15:37.003 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.9881000000000003e-05 2024-08-19T14:15:37.003 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:37.003 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:37.003 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:15:37.004 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:43.020712+0000", 2024-08-19T14:15:37.004 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.9539000000000002e-05 2024-08-19T14:15:37.004 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:15:37.004 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:15:37.004 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:15:37.004 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:37.004 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:37.004 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24580.0:88328 2.5 2:a21c7eb0:::benchmark_data_smithi045_46415_object88327:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-08-19T14:15:37.004 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:42.820640+0000", 2024-08-19T14:15:37.004 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 114.06965663299999, 2024-08-19T14:15:37.004 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20083906900000001, 2024-08-19T14:15:37.004 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:15:37.004 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:15:37.004 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:15:37.005 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:15:37.005 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24580", 2024-08-19T14:15:37.005 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:15:37.005 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 88328 2024-08-19T14:15:37.005 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:37.005 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:15:37.005 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:37.005 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:15:37.005 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.820640+0000", 2024-08-19T14:15:37.005 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:15:37.005 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:37.005 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:37.005 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:15:37.005 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.820640+0000", 2024-08-19T14:15:37.006 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:15:37.006 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:37.006 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:37.006 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:15:37.006 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.820641+0000", 2024-08-19T14:15:37.006 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.23e-07 2024-08-19T14:15:37.006 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:37.006 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:37.006 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:15:37.006 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.820660+0000", 2024-08-19T14:15:37.006 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8788e-05 2024-08-19T14:15:37.006 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:37.006 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:37.007 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:15:37.007 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.820670+0000", 2024-08-19T14:15:37.007 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.9089999999999995e-06 2024-08-19T14:15:37.007 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:37.007 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:37.007 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:15:37.007 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.820675+0000", 2024-08-19T14:15:37.007 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.8890000000000001e-06 2024-08-19T14:15:37.007 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:37.007 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:37.007 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:15:37.007 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.820718+0000", 2024-08-19T14:15:37.007 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.2997999999999997e-05 2024-08-19T14:15:37.007 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:37.008 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:37.008 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:15:37.008 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.820811+0000", 2024-08-19T14:15:37.008 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.3219000000000003e-05 2024-08-19T14:15:37.008 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:37.008 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:37.008 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,4", 2024-08-19T14:15:37.008 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.820907+0000", 2024-08-19T14:15:37.008 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.6267000000000005e-05 2024-08-19T14:15:37.008 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:37.008 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:37.008 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:15:37.008 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.823083+0000", 2024-08-19T14:15:37.008 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0021754299999999999 2024-08-19T14:15:37.009 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:37.009 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:37.009 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:37.009 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.824768+0000", 2024-08-19T14:15:37.009 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001685048 2024-08-19T14:15:37.009 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:37.009 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:37.009 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:37.009 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:43.021388+0000", 2024-08-19T14:15:37.009 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.196620295 2024-08-19T14:15:37.009 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:37.009 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:37.009 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:15:37.009 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:43.021436+0000", 2024-08-19T14:15:37.010 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.8195000000000002e-05 2024-08-19T14:15:37.010 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:37.010 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:37.010 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:15:37.010 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:43.021479+0000", 2024-08-19T14:15:37.010 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.3307999999999999e-05 2024-08-19T14:15:37.010 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:15:37.010 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:15:37.010 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:15:37.010 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:37.010 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:37.010 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24580.0:88341 2.f 2:f07bb047:::benchmark_data_smithi045_46415_object88340:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-08-19T14:15:37.010 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:42.835930+0000", 2024-08-19T14:15:37.011 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 114.05436722100001, 2024-08-19T14:15:37.011 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.18161846200000001, 2024-08-19T14:15:37.011 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:15:37.011 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:15:37.011 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:15:37.011 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:15:37.011 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24580", 2024-08-19T14:15:37.011 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:15:37.011 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 88341 2024-08-19T14:15:37.011 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:37.011 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:15:37.011 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:37.011 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:15:37.011 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.835930+0000", 2024-08-19T14:15:37.012 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:15:37.012 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:37.012 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:37.012 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:15:37.012 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.835930+0000", 2024-08-19T14:15:37.012 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:15:37.012 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:37.012 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:37.012 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:15:37.012 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.835931+0000", 2024-08-19T14:15:37.012 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.514e-06 2024-08-19T14:15:37.012 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:37.012 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:37.012 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:15:37.013 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.835953+0000", 2024-08-19T14:15:37.013 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2200000000000001e-05 2024-08-19T14:15:37.013 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:37.013 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:37.013 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:15:37.013 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.835970+0000", 2024-08-19T14:15:37.013 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6262000000000001e-05 2024-08-19T14:15:37.013 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:37.013 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:37.013 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:15:37.013 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.835978+0000", 2024-08-19T14:15:37.013 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.5990000000000002e-06 2024-08-19T14:15:37.013 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:37.013 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:37.014 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:15:37.014 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.836041+0000", 2024-08-19T14:15:37.014 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.2763999999999997e-05 2024-08-19T14:15:37.014 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:37.014 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:37.014 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:15:37.014 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.836178+0000", 2024-08-19T14:15:37.014 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013741299999999999 2024-08-19T14:15:37.014 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:37.014 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:37.014 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 4,9", 2024-08-19T14:15:37.014 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.836327+0000", 2024-08-19T14:15:37.014 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000148837 2024-08-19T14:15:37.015 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:37.015 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:37.015 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:37.015 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.838785+0000", 2024-08-19T14:15:37.015 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0024579440000000001 2024-08-19T14:15:37.015 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:37.015 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:37.015 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:15:37.015 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.838889+0000", 2024-08-19T14:15:37.015 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00010374600000000001 2024-08-19T14:15:37.015 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:37.015 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:37.015 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:37.015 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:43.017501+0000", 2024-08-19T14:15:37.015 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.17861226699999999 2024-08-19T14:15:37.016 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:37.016 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:37.016 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:15:37.016 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:43.017529+0000", 2024-08-19T14:15:37.016 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.8153999999999998e-05 2024-08-19T14:15:37.016 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:37.016 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:37.016 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:15:37.016 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:43.017548+0000", 2024-08-19T14:15:37.016 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8762000000000001e-05 2024-08-19T14:15:37.016 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:15:37.016 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:15:37.016 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:15:37.016 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:37.017 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:37.017 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24580.0:88346 2.5 2:a3b32ee4:::benchmark_data_smithi045_46415_object88345:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-08-19T14:15:37.017 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:42.840977+0000", 2024-08-19T14:15:37.017 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 114.049319852, 2024-08-19T14:15:37.017 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.18062758800000001, 2024-08-19T14:15:37.017 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:15:37.017 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:15:37.017 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:15:37.017 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:15:37.017 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24580", 2024-08-19T14:15:37.017 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:15:37.017 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 88346 2024-08-19T14:15:37.017 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:37.017 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:15:37.018 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:37.018 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:15:37.018 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.840977+0000", 2024-08-19T14:15:37.018 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:15:37.018 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:37.018 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:37.018 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:15:37.018 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.840977+0000", 2024-08-19T14:15:37.018 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:15:37.018 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:37.018 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:37.018 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:15:37.018 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.840978+0000", 2024-08-19T14:15:37.018 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.7899999999999998e-07 2024-08-19T14:15:37.019 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:37.019 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:37.019 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:15:37.019 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.840997+0000", 2024-08-19T14:15:37.019 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9460999999999999e-05 2024-08-19T14:15:37.019 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:37.019 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:37.019 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:15:37.019 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.841007+0000", 2024-08-19T14:15:37.019 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.3640000000000005e-06 2024-08-19T14:15:37.020 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:37.020 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:37.020 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:15:37.021 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.841013+0000", 2024-08-19T14:15:37.021 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.6980000000000001e-06 2024-08-19T14:15:37.021 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:37.021 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:37.021 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:15:37.021 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.841059+0000", 2024-08-19T14:15:37.021 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.6019000000000001e-05 2024-08-19T14:15:37.021 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:37.021 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:37.021 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:15:37.021 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.841136+0000", 2024-08-19T14:15:37.021 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.6654000000000002e-05 2024-08-19T14:15:37.021 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:37.021 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:37.022 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,4", 2024-08-19T14:15:37.022 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.841259+0000", 2024-08-19T14:15:37.022 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012317900000000001 2024-08-19T14:15:37.022 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:37.022 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:37.022 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:15:37.022 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.843356+0000", 2024-08-19T14:15:37.022 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0020965189999999998 2024-08-19T14:15:37.022 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:37.022 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:37.022 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:37.022 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.846892+0000", 2024-08-19T14:15:37.022 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0035363030000000002 2024-08-19T14:15:37.022 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:37.023 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:37.023 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:37.023 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:43.021518+0000", 2024-08-19T14:15:37.023 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.17462582600000001 2024-08-19T14:15:37.023 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:37.023 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:37.023 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:15:37.023 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:43.021568+0000", 2024-08-19T14:15:37.023 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.9755000000000001e-05 2024-08-19T14:15:37.023 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:37.023 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:37.023 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:15:37.023 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:43.021605+0000", 2024-08-19T14:15:37.023 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.7131000000000001e-05 2024-08-19T14:15:37.024 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:15:37.024 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:15:37.024 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:15:37.024 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:15:37.024 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:15:37.024 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:15:37.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:37.025 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:37.025 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:15:37.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:36 smithi045 ceph-mon[20143]: 2.4 scrub starts 2024-08-19T14:15:37.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:36 smithi045 ceph-mon[20143]: 2.d scrub ok 2024-08-19T14:15:37.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:36 smithi045 ceph-mon[20143]: pgmap v451: 27 pgs: 3 active+clean+scrubbing, 2 active+remapped+backfilling, 22 active+clean; 3.6 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 821 op/s; 4059/178296 objects misplaced (2.277%); 16 MiB/s, 251 objects/s recovering 2024-08-19T14:15:37.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:36 smithi045 ceph-mon[20143]: 2.a scrub ok 2024-08-19T14:15:37.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:36 smithi031 ceph-mon[16336]: 2.4 scrub starts 2024-08-19T14:15:37.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:36 smithi031 ceph-mon[16336]: 2.d scrub ok 2024-08-19T14:15:37.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:36 smithi031 ceph-mon[16336]: pgmap v451: 27 pgs: 3 active+clean+scrubbing, 2 active+remapped+backfilling, 22 active+clean; 3.6 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 821 op/s; 4059/178296 objects misplaced (2.277%); 16 MiB/s, 251 objects/s recovering 2024-08-19T14:15:37.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:36 smithi031 ceph-mon[16336]: 2.a scrub ok 2024-08-19T14:15:37.125 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:15:37.149 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:37.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:37.150 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:15:37.251 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:15:37.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:36 smithi032 ceph-mon[20004]: 2.4 scrub starts 2024-08-19T14:15:37.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:36 smithi032 ceph-mon[20004]: 2.d scrub ok 2024-08-19T14:15:37.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:36 smithi032 ceph-mon[20004]: pgmap v451: 27 pgs: 3 active+clean+scrubbing, 2 active+remapped+backfilling, 22 active+clean; 3.6 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 821 op/s; 4059/178296 objects misplaced (2.277%); 16 MiB/s, 251 objects/s recovering 2024-08-19T14:15:37.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:36 smithi032 ceph-mon[20004]: 2.a scrub ok 2024-08-19T14:15:37.280 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:37.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:37.281 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:15:37.382 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:15:37.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:37.416 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:37.416 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:15:37.517 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:15:37.539 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:37.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:37.540 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:15:37.641 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:15:37.693 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:37.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:37.694 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:15:37.795 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:15:37.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:37.824 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:37.824 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:15:37.925 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:15:37.948 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:37.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:37.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:15:37.994 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-08-19T14:15:38.050 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:15:38.072 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:38.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:38.073 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:15:38.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:37 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3482286314' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:15:38.174 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:15:38.206 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:38.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:38.207 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:15:38.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:37 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3482286314' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:15:38.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:37 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3482286314' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:15:38.307 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:15:38.333 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:15:38.343 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:38.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:38.344 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:15:38.434 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:15:38.434 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:15:38.434 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:15:38.434 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:15:38.434 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:15:38.434 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:15:38.434 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:15:38.434 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:15:38.435 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:15:38.435 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:15:38.435 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:15:38.435 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:15:38.435 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:15:38.435 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:15:38.435 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:15:38.435 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:15:38.435 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:15:38.436 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:15:38.436 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:15:38.436 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:15:38.436 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:15:38.436 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:15:38.436 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:15:38.436 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:15:38.445 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:15:38.479 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:38.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:38.480 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:15:38.581 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:15:38.605 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:38.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:38.606 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:15:38.707 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:15:38.732 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:38.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:38.733 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:15:38.834 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:15:38.856 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:38.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:38.857 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:15:38.902 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:15:38.902 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:15:38.957 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:15:38.990 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:38.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:38.991 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:15:39.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:38 smithi045 ceph-mon[20143]: 2.4 scrub ok 2024-08-19T14:15:39.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:38 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:15:39.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:38 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:15:39.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:38 smithi045 ceph-mon[20143]: pgmap v452: 27 pgs: 2 active+clean+scrubbing, 2 active+remapped+backfilling, 23 active+clean; 3.5 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 780 op/s; 2603/174405 objects misplaced (1.493%); 21 MiB/s, 343 objects/s recovering 2024-08-19T14:15:39.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:38 smithi045 ceph-mon[20143]: osdmap e181: 12 total, 12 up, 10 in 2024-08-19T14:15:39.092 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:15:39.113 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:39.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:39.114 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:15:39.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:38 smithi031 ceph-mon[16336]: 2.4 scrub ok 2024-08-19T14:15:39.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:38 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:15:39.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:38 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:15:39.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:38 smithi031 ceph-mon[16336]: pgmap v452: 27 pgs: 2 active+clean+scrubbing, 2 active+remapped+backfilling, 23 active+clean; 3.5 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 780 op/s; 2603/174405 objects misplaced (1.493%); 21 MiB/s, 343 objects/s recovering 2024-08-19T14:15:39.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:38 smithi031 ceph-mon[16336]: osdmap e181: 12 total, 12 up, 10 in 2024-08-19T14:15:39.215 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:15:39.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:39.242 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:39.242 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:15:39.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:38 smithi032 ceph-mon[20004]: 2.4 scrub ok 2024-08-19T14:15:39.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:38 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:15:39.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:38 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:15:39.259 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:38 smithi032 ceph-mon[20004]: pgmap v452: 27 pgs: 2 active+clean+scrubbing, 2 active+remapped+backfilling, 23 active+clean; 3.5 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 780 op/s; 2603/174405 objects misplaced (1.493%); 21 MiB/s, 343 objects/s recovering 2024-08-19T14:15:39.259 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:38 smithi032 ceph-mon[20004]: osdmap e181: 12 total, 12 up, 10 in 2024-08-19T14:15:39.343 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:15:39.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:39.375 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:39.375 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:15:39.475 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:15:39.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:39.513 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:39.513 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:15:39.614 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:15:39.638 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:39.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:39.640 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:15:39.679 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-08-19T14:15:39.740 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:15:39.771 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:39.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:39.772 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:15:39.873 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:15:39.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:39.908 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:39.908 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:15:40.009 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:15:40.047 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:15:40.051 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:40.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:40.052 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:15:40.153 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:15:40.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:40.190 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:40.191 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:15:40.292 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:15:40.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:40.318 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:40.319 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:15:40.420 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:15:40.467 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:40.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:40.468 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:15:40.568 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:15:40.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:40.603 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:40.603 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:15:40.704 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:15:40.729 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:40.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:40.730 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:15:40.831 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:15:40.856 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:40.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:40.857 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:15:40.865 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:40 smithi031 ceph-mon[16336]: pgmap v454: 27 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfilling, 24 active+clean; 3.4 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 927 op/s; 2603/167880 objects misplaced (1.551%); 11 MiB/s, 176 objects/s recovering 2024-08-19T14:15:40.957 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:15:40.963 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:15:40.963 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:15:40.992 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:40.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:40.993 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:15:41.094 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:15:41.120 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:41.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:41.121 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:15:41.222 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:15:41.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:41.248 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:41.249 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:15:41.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:40 smithi032 ceph-mon[20004]: pgmap v454: 27 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfilling, 24 active+clean; 3.4 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 927 op/s; 2603/167880 objects misplaced (1.551%); 11 MiB/s, 176 objects/s recovering 2024-08-19T14:15:41.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:40 smithi045 ceph-mon[20143]: pgmap v454: 27 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfilling, 24 active+clean; 3.4 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 927 op/s; 2603/167880 objects misplaced (1.551%); 11 MiB/s, 176 objects/s recovering 2024-08-19T14:15:41.350 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:15:41.393 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:41.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:41.394 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:15:41.406 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:15:41.496 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:15:41.522 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:41.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:41.523 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:15:41.624 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:15:41.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:41.669 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:41.669 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:15:41.766 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-08-19T14:15:41.769 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:15:41.795 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:41.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:41.796 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:15:41.897 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:15:41.921 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:41.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:41.922 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:15:42.023 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:15:42.046 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:42.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:42.047 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:15:42.137 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:15:42.149 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:15:42.197 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:42.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:42.198 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:15:42.299 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:15:42.300 INFO:teuthology.orchestra.run.smithi031.stdout:moved 0 / 81 (0%) 2024-08-19T14:15:42.300 INFO:teuthology.orchestra.run.smithi031.stdout:avg 8.1 2024-08-19T14:15:42.300 INFO:teuthology.orchestra.run.smithi031.stdout:stddev 2.16564 -> 2.16564 (expected baseline 2.7) 2024-08-19T14:15:42.300 INFO:teuthology.orchestra.run.smithi031.stdout:min osd.5 with 4 -> 4 pgs (0.493827 -> 0.493827 * mean) 2024-08-19T14:15:42.300 INFO:teuthology.orchestra.run.smithi031.stdout:max osd.9 with 12 -> 12 pgs (1.48148 -> 1.48148 * mean) 2024-08-19T14:15:42.300 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:15:42.300 INFO:teuthology.orchestra.run.smithi031.stdout:oload 1000 2024-08-19T14:15:42.300 INFO:teuthology.orchestra.run.smithi031.stdout:max_change 0.05 2024-08-19T14:15:42.300 INFO:teuthology.orchestra.run.smithi031.stdout:max_change_osds 4 2024-08-19T14:15:42.301 INFO:teuthology.orchestra.run.smithi031.stdout:average_utilization 0.0127 2024-08-19T14:15:42.301 INFO:teuthology.orchestra.run.smithi031.stdout:overload_utilization 0.1274 2024-08-19T14:15:42.301 INFO:teuthology.orchestra.run.smithi031.stderr:no change 2024-08-19T14:15:42.326 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:42.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:42.327 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:15:42.428 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:15:42.459 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:42.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:42.460 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:15:42.561 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:15:42.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:42.598 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:42.598 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:15:42.699 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:15:42.727 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:42.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:42.728 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:15:42.829 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:15:42.868 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:42.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:42.869 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:15:42.957 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-19T14:15:42.970 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:15:42.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:42 smithi031 ceph-mon[16336]: pgmap v455: 27 pgs: 2 active+remapped+backfilling, 25 active+clean; 3.3 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 826 op/s; 1162/162624 objects misplaced (0.715%); 20 MiB/s, 316 objects/s recovering 2024-08-19T14:15:42.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:42 smithi031 ceph-mon[16336]: from='client.24929 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T14:15:42.993 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:42.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:42.994 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:15:43.035 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-08-19T14:15:43.035 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd reweight-by-utilization 1000 3.0 2024-08-19T14:15:43.036 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:15:43.037 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:15:43.095 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:15:43.120 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:43.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:43.122 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:15:43.223 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:15:43.250 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:43.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:43.251 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:15:43.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:42 smithi032 ceph-mon[20004]: pgmap v455: 27 pgs: 2 active+remapped+backfilling, 25 active+clean; 3.3 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 826 op/s; 1162/162624 objects misplaced (0.715%); 20 MiB/s, 316 objects/s recovering 2024-08-19T14:15:43.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:42 smithi032 ceph-mon[20004]: from='client.24929 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T14:15:43.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:42 smithi045 ceph-mon[20143]: pgmap v455: 27 pgs: 2 active+remapped+backfilling, 25 active+clean; 3.3 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 826 op/s; 1162/162624 objects misplaced (0.715%); 20 MiB/s, 316 objects/s recovering 2024-08-19T14:15:43.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:42 smithi045 ceph-mon[20143]: from='client.24929 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T14:15:43.352 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:15:43.376 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:43.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:43.377 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:15:43.394 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:15:43.460 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:15:43.478 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:15:43.504 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:43.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:43.505 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:15:43.606 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:15:43.630 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:43.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:43.631 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:15:43.732 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:15:43.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:43.780 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:43.780 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:15:43.849 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-08-19T14:15:43.881 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:15:43.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:43.920 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:43.921 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:15:44.021 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:15:44.028 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:43 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:15:44.028 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:43 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:15:44.045 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:44.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:44.047 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:15:44.102 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:15:44.147 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:15:44.148 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:15:44.166 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:15:44.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:44.220 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:44.220 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:15:44.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:43 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:15:44.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:43 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:15:44.292 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:43 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:15:44.292 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:43 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:15:44.321 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:15:44.347 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:44.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:44.348 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:15:44.449 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:15:44.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:44.490 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:44.490 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:15:44.592 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:15:44.627 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:44.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:44.628 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:15:44.729 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:15:44.740 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:15:44.740 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:15:44.769 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:44.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:44.770 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:15:44.849 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is set 2024-08-19T14:15:44.872 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:15:44.894 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:44.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:44.895 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:15:44.996 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:15:45.022 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:45.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:45.023 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:15:45.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:44 smithi045 ceph-mon[20143]: pgmap v456: 27 pgs: 2 active+remapped+backfilling, 25 active+clean; 3.2 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 828 op/s; 1162/157563 objects misplaced (0.737%); 18 MiB/s, 281 objects/s recovering 2024-08-19T14:15:45.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:44 smithi045 ceph-mon[20143]: osdmap e182: 12 total, 12 up, 10 in 2024-08-19T14:15:45.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:44 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2080795930' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:15:45.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:44 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:15:45.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:44 smithi031 ceph-mon[16336]: pgmap v456: 27 pgs: 2 active+remapped+backfilling, 25 active+clean; 3.2 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 828 op/s; 1162/157563 objects misplaced (0.737%); 18 MiB/s, 281 objects/s recovering 2024-08-19T14:15:45.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:44 smithi031 ceph-mon[16336]: osdmap e182: 12 total, 12 up, 10 in 2024-08-19T14:15:45.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:44 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2080795930' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:15:45.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:44 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:15:45.125 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:15:45.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:45.147 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:45.147 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:15:45.248 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:15:45.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:44 smithi032 ceph-mon[20004]: pgmap v456: 27 pgs: 2 active+remapped+backfilling, 25 active+clean; 3.2 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 828 op/s; 1162/157563 objects misplaced (0.737%); 18 MiB/s, 281 objects/s recovering 2024-08-19T14:15:45.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:44 smithi032 ceph-mon[20004]: osdmap e182: 12 total, 12 up, 10 in 2024-08-19T14:15:45.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:44 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2080795930' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:15:45.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:44 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:15:45.272 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:45.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:45.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:15:45.373 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:15:45.398 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:45.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:45.399 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:15:45.491 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-08-19T14:15:45.499 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:15:45.521 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:45.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:45.522 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:15:45.623 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:15:45.645 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:45.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:45.646 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:15:45.747 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:15:45.770 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:45.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:45.771 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:15:45.780 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:15:45.872 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:15:45.893 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:45.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:45.894 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:15:45.995 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:15:46.023 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:46.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:46.024 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:15:46.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:45 smithi031 ceph-mon[16336]: 1.8 scrub starts 2024-08-19T14:15:46.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:45 smithi031 ceph-mon[16336]: 1.8 scrub ok 2024-08-19T14:15:46.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:45 smithi031 ceph-mon[16336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:15:46.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:45 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:15:46.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:45 smithi031 ceph-mon[16336]: osdmap e183: 12 total, 12 up, 10 in 2024-08-19T14:15:46.125 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:15:46.148 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:46.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:46.149 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:15:46.250 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:15:46.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:45 smithi032 ceph-mon[20004]: 1.8 scrub starts 2024-08-19T14:15:46.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:45 smithi032 ceph-mon[20004]: 1.8 scrub ok 2024-08-19T14:15:46.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:45 smithi032 ceph-mon[20004]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:15:46.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:45 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:15:46.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:45 smithi032 ceph-mon[20004]: osdmap e183: 12 total, 12 up, 10 in 2024-08-19T14:15:46.272 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:46.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:46.273 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:15:46.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:45 smithi045 ceph-mon[20143]: 1.8 scrub starts 2024-08-19T14:15:46.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:45 smithi045 ceph-mon[20143]: 1.8 scrub ok 2024-08-19T14:15:46.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:45 smithi045 ceph-mon[20143]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:15:46.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:45 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:15:46.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:45 smithi045 ceph-mon[20143]: osdmap e183: 12 total, 12 up, 10 in 2024-08-19T14:15:46.374 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:15:46.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:46.396 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:46.396 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:15:46.497 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:15:46.519 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:46.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:46.520 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:15:46.523 INFO:teuthology.orchestra.run.smithi031.stdout:moved 0 / 81 (0%) 2024-08-19T14:15:46.523 INFO:teuthology.orchestra.run.smithi031.stdout:avg 8.1 2024-08-19T14:15:46.523 INFO:teuthology.orchestra.run.smithi031.stdout:stddev 2.16564 -> 2.16564 (expected baseline 2.7) 2024-08-19T14:15:46.523 INFO:teuthology.orchestra.run.smithi031.stdout:min osd.5 with 4 -> 4 pgs (0.493827 -> 0.493827 * mean) 2024-08-19T14:15:46.524 INFO:teuthology.orchestra.run.smithi031.stdout:max osd.9 with 12 -> 12 pgs (1.48148 -> 1.48148 * mean) 2024-08-19T14:15:46.524 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:15:46.524 INFO:teuthology.orchestra.run.smithi031.stdout:oload 1000 2024-08-19T14:15:46.524 INFO:teuthology.orchestra.run.smithi031.stdout:max_change 3 2024-08-19T14:15:46.524 INFO:teuthology.orchestra.run.smithi031.stdout:max_change_osds 4 2024-08-19T14:15:46.524 INFO:teuthology.orchestra.run.smithi031.stdout:average_utilization 0.0120 2024-08-19T14:15:46.524 INFO:teuthology.orchestra.run.smithi031.stdout:overload_utilization 0.1201 2024-08-19T14:15:46.524 INFO:teuthology.orchestra.run.smithi031.stderr:no change 2024-08-19T14:15:46.620 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:15:46.643 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:46.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:46.644 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:15:46.745 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:15:46.768 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:46.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:46.769 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:15:46.870 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:15:46.896 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:46.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:46.898 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:15:46.998 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:15:47.027 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:47.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:47.029 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:15:47.058 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:46 smithi031 ceph-mon[16336]: pgmap v459: 27 pgs: 2 active+remapped+backfilling, 25 active+clean; 3.1 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 868 op/s; 1162/153657 objects misplaced (0.756%); 10 MiB/s, 162 objects/s recovering 2024-08-19T14:15:47.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:46 smithi031 ceph-mon[16336]: osdmap e184: 12 total, 12 up, 10 in 2024-08-19T14:15:47.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:46 smithi031 ceph-mon[16336]: from='client.15069 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T14:15:47.129 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:15:47.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:47.153 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:47.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:15:47.203 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:15:47.203 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:15:47.222 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:15:47.223 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:15:47.240 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:15:47.241 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:15:47.254 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:15:47.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:46 smithi032 ceph-mon[20004]: pgmap v459: 27 pgs: 2 active+remapped+backfilling, 25 active+clean; 3.1 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 868 op/s; 1162/153657 objects misplaced (0.756%); 10 MiB/s, 162 objects/s recovering 2024-08-19T14:15:47.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:46 smithi032 ceph-mon[20004]: osdmap e184: 12 total, 12 up, 10 in 2024-08-19T14:15:47.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:46 smithi032 ceph-mon[20004]: from='client.15069 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T14:15:47.274 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:15:47.274 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:15:47.285 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:47.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:47.286 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:15:47.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:46 smithi045 ceph-mon[20143]: pgmap v459: 27 pgs: 2 active+remapped+backfilling, 25 active+clean; 3.1 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 868 op/s; 1162/153657 objects misplaced (0.756%); 10 MiB/s, 162 objects/s recovering 2024-08-19T14:15:47.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:46 smithi045 ceph-mon[20143]: osdmap e184: 12 total, 12 up, 10 in 2024-08-19T14:15:47.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:46 smithi045 ceph-mon[20143]: from='client.15069 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T14:15:47.301 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:15:47.301 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:15:47.336 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:15:47.336 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:15:47.376 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:15:47.377 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:15:47.387 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:15:47.409 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:15:47.410 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:15:47.412 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:47.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:47.413 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:15:47.445 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:15:47.445 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:15:47.480 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:15:47.481 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:15:47.514 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:15:47.516 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:15:47.516 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:15:47.540 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:47.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:47.541 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:15:47.554 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:15:47.555 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:15:47.642 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:15:47.667 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:47.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:47.669 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:15:47.692 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-08-19T14:15:47.745 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd reweight-by-utilization 1000 1.0 2024-08-19T14:15:47.770 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:15:47.794 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:47.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:47.795 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:15:47.869 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:15:47.869 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [], 2024-08-19T14:15:47.869 INFO:teuthology.orchestra.run.smithi045.stdout: "complaint_time": 30, 2024-08-19T14:15:47.870 INFO:teuthology.orchestra.run.smithi045.stdout: "num_blocked_ops": 0 2024-08-19T14:15:47.870 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:15:47.895 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:15:47.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:47.939 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:47.939 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:15:48.040 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:15:48.082 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:48.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:48.083 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:15:48.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:47 smithi031 ceph-mon[16336]: osdmap e185: 12 total, 12 up, 10 in 2024-08-19T14:15:48.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:47 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:15:48.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:47 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:15:48.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:47 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/352648538' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:15:48.184 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:15:48.208 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:48.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:48.209 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:15:48.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:47 smithi032 ceph-mon[20004]: osdmap e185: 12 total, 12 up, 10 in 2024-08-19T14:15:48.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:47 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:15:48.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:47 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:15:48.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:47 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/352648538' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:15:48.292 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:47 smithi045 ceph-mon[20143]: osdmap e185: 12 total, 12 up, 10 in 2024-08-19T14:15:48.292 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:47 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:15:48.292 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:47 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:15:48.292 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:47 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/352648538' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:15:48.310 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:15:48.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:48.345 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:48.345 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:15:48.446 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:15:48.472 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:48.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:48.473 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:15:48.551 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:15:48.574 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:15:48.609 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:48.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:48.610 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:15:48.711 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:15:48.727 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-19T14:15:48.733 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:48.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:48.734 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:15:48.836 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:15:48.863 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:48.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:48.866 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:15:48.967 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:15:48.989 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:48.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:48.991 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:15:49.071 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:48 smithi031 ceph-mon[16336]: pgmap v462: 27 pgs: 1 peering, 26 active+clean; 3.1 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 585 op/s; 17 MiB/s, 278 objects/s recovering 2024-08-19T14:15:49.071 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:48 smithi031 ceph-mon[16336]: osdmap e186: 12 total, 12 up, 10 in 2024-08-19T14:15:49.092 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:15:49.118 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:49.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:49.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:15:49.220 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:15:49.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:48 smithi032 ceph-mon[20004]: pgmap v462: 27 pgs: 1 peering, 26 active+clean; 3.1 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 585 op/s; 17 MiB/s, 278 objects/s recovering 2024-08-19T14:15:49.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:48 smithi032 ceph-mon[20004]: osdmap e186: 12 total, 12 up, 10 in 2024-08-19T14:15:49.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:49.275 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:49.275 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:15:49.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:48 smithi045 ceph-mon[20143]: pgmap v462: 27 pgs: 1 peering, 26 active+clean; 3.1 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 585 op/s; 17 MiB/s, 278 objects/s recovering 2024-08-19T14:15:49.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:48 smithi045 ceph-mon[20143]: osdmap e186: 12 total, 12 up, 10 in 2024-08-19T14:15:49.350 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:15:49.375 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:15:49.407 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:49.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:49.408 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:15:49.509 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:15:49.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:49.533 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:49.533 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:15:49.553 INFO:teuthology.orchestra.run.smithi031.stdout:moved 0 / 81 (0%) 2024-08-19T14:15:49.553 INFO:teuthology.orchestra.run.smithi031.stdout:avg 8.1 2024-08-19T14:15:49.553 INFO:teuthology.orchestra.run.smithi031.stdout:stddev 2.16564 -> 2.16564 (expected baseline 2.7) 2024-08-19T14:15:49.553 INFO:teuthology.orchestra.run.smithi031.stdout:min osd.5 with 4 -> 4 pgs (0.493827 -> 0.493827 * mean) 2024-08-19T14:15:49.553 INFO:teuthology.orchestra.run.smithi031.stdout:max osd.9 with 12 -> 12 pgs (1.48148 -> 1.48148 * mean) 2024-08-19T14:15:49.554 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:15:49.554 INFO:teuthology.orchestra.run.smithi031.stdout:oload 1000 2024-08-19T14:15:49.554 INFO:teuthology.orchestra.run.smithi031.stdout:max_change 1 2024-08-19T14:15:49.554 INFO:teuthology.orchestra.run.smithi031.stdout:max_change_osds 4 2024-08-19T14:15:49.554 INFO:teuthology.orchestra.run.smithi031.stdout:average_utilization 0.0119 2024-08-19T14:15:49.554 INFO:teuthology.orchestra.run.smithi031.stdout:overload_utilization 0.1186 2024-08-19T14:15:49.554 INFO:teuthology.orchestra.run.smithi031.stderr:no change 2024-08-19T14:15:49.634 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:15:49.660 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:49.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:49.661 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:15:49.762 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:15:49.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:49.787 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:49.788 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:15:49.889 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:15:49.915 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:49.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:49.916 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:15:49.970 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-08-19T14:15:50.017 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:15:50.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:50.052 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:50.053 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:15:50.153 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:15:50.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:50.198 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:50.198 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:15:50.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:49 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:15:50.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:49 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:15:50.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:49 smithi032 ceph-mon[20004]: from='client.15081 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T14:15:50.261 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:49 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:15:50.261 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:49 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:15:50.261 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:49 smithi031 ceph-mon[16336]: from='client.15081 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T14:15:50.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:49 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:15:50.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:49 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:15:50.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:49 smithi045 ceph-mon[20143]: from='client.15081 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T14:15:50.299 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:15:50.306 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-08-19T14:15:50.306 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd test-reweight-by-utilization 1000 0.05 2024-08-19T14:15:50.339 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:50.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:50.340 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:15:50.358 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:15:50.442 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:15:50.463 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:50.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:50.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:15:50.468 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:15:50.565 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:15:50.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:50.623 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:50.623 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:15:50.725 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:15:50.747 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:50.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:50.748 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:15:50.791 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:15:50.849 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:15:50.895 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:15:50.895 INFO:teuthology.orchestra.run.smithi045.stdout: "size": 20, 2024-08-19T14:15:50.896 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 600, 2024-08-19T14:15:50.896 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [ 2024-08-19T14:15:50.896 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.896 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:14 2.4 e106/102 2:291e9767:::benchmark_data_smithi045_46415_object13:head v 106'1, mlcod=0'0)", 2024-08-19T14:15:50.896 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:12:49.835194+0000", 2024-08-19T14:15:50.896 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 181.05804774000001, 2024-08-19T14:15:50.896 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.045178087999999998, 2024-08-19T14:15:50.896 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:15:50.896 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:15:50.896 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:15:50.896 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:15:50.896 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.896 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:15:50.896 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:49.835194+0000", 2024-08-19T14:15:50.897 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:15:50.897 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.897 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.897 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:15:50.897 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:49.835194+0000", 2024-08-19T14:15:50.897 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:15:50.897 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.897 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.897 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:15:50.897 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:49.835194+0000", 2024-08-19T14:15:50.897 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.7500000000000001e-07 2024-08-19T14:15:50.897 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.897 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.898 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:15:50.898 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:49.879644+0000", 2024-08-19T14:15:50.898 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.044449610000000001 2024-08-19T14:15:50.898 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.898 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.898 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:15:50.898 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:49.879665+0000", 2024-08-19T14:15:50.898 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1458999999999999e-05 2024-08-19T14:15:50.898 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.898 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.898 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:15:50.898 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:49.879706+0000", 2024-08-19T14:15:50.898 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.0748999999999999e-05 2024-08-19T14:15:50.899 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.899 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.899 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:15:50.899 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:49.879782+0000", 2024-08-19T14:15:50.899 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.6445000000000004e-05 2024-08-19T14:15:50.899 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.899 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.899 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:15:50.899 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:49.879807+0000", 2024-08-19T14:15:50.899 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.4765000000000001e-05 2024-08-19T14:15:50.899 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.899 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.899 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:15:50.899 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:49.880329+0000", 2024-08-19T14:15:50.900 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00052147599999999999 2024-08-19T14:15:50.900 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.900 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.900 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:15:50.900 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:49.880372+0000", 2024-08-19T14:15:50.900 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.3309000000000002e-05 2024-08-19T14:15:50.900 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:15:50.900 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:15:50.900 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:15:50.900 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.900 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.900 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:12156 2.4 e107/102 2:28b3607b:::benchmark_data_smithi045_46415_object12155:head v 107'793, mlcod=107'791)", 2024-08-19T14:15:50.900 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:12:53.026834+0000", 2024-08-19T14:15:50.901 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 177.866407275, 2024-08-19T14:15:50.901 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0078985359999999994, 2024-08-19T14:15:50.901 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:15:50.901 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:15:50.901 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:15:50.901 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:15:50.901 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.901 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:15:50.901 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:53.026834+0000", 2024-08-19T14:15:50.901 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:15:50.901 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.901 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.901 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:15:50.902 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:53.026834+0000", 2024-08-19T14:15:50.902 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:15:50.902 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.902 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.902 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:15:50.902 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:53.026835+0000", 2024-08-19T14:15:50.902 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.5199999999999996e-07 2024-08-19T14:15:50.902 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.902 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.902 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:15:50.902 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:53.034134+0000", 2024-08-19T14:15:50.902 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0072991369999999998 2024-08-19T14:15:50.902 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.903 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.903 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:15:50.903 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:53.034144+0000", 2024-08-19T14:15:50.903 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0370000000000001e-05 2024-08-19T14:15:50.903 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.903 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.903 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:15:50.903 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:53.034149+0000", 2024-08-19T14:15:50.903 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.8130000000000003e-06 2024-08-19T14:15:50.903 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.903 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.904 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:15:50.904 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:53.034196+0000", 2024-08-19T14:15:50.904 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.7159e-05 2024-08-19T14:15:50.904 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.904 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.904 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:15:50.904 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:53.034211+0000", 2024-08-19T14:15:50.904 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4429e-05 2024-08-19T14:15:50.904 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.904 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.904 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:15:50.905 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:53.034701+0000", 2024-08-19T14:15:50.905 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00049043000000000003 2024-08-19T14:15:50.905 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.905 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.905 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:15:50.905 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:53.034733+0000", 2024-08-19T14:15:50.905 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.1545999999999999e-05 2024-08-19T14:15:50.905 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:15:50.905 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:15:50.905 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:15:50.905 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.905 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.906 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:38741 2.4 e112/102 2:2ddcb16b:::benchmark_data_smithi045_46415_object38740:head v 112'2448, mlcod=112'2444)", 2024-08-19T14:15:50.906 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:02.435487+0000", 2024-08-19T14:15:50.906 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 168.45775484800001, 2024-08-19T14:15:50.906 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0060987159999999997, 2024-08-19T14:15:50.906 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:15:50.906 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:15:50.909 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:15:50.909 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:15:50.909 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.910 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:15:50.910 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.435487+0000", 2024-08-19T14:15:50.910 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:15:50.910 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.910 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.910 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:15:50.910 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.435487+0000", 2024-08-19T14:15:50.910 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:15:50.910 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.910 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.911 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:15:50.911 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.435487+0000", 2024-08-19T14:15:50.911 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2699999999999999e-07 2024-08-19T14:15:50.911 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.911 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.911 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:15:50.911 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.435514+0000", 2024-08-19T14:15:50.911 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.6457000000000001e-05 2024-08-19T14:15:50.911 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.911 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.911 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:15:50.911 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.435524+0000", 2024-08-19T14:15:50.911 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0299000000000001e-05 2024-08-19T14:15:50.911 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.912 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.912 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:15:50.912 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.435531+0000", 2024-08-19T14:15:50.912 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.6529999999999997e-06 2024-08-19T14:15:50.912 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.912 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.912 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:15:50.912 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.435603+0000", 2024-08-19T14:15:50.912 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.1963000000000002e-05 2024-08-19T14:15:50.912 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.913 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.913 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:15:50.913 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.435622+0000", 2024-08-19T14:15:50.913 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8926000000000001e-05 2024-08-19T14:15:50.913 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.913 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.913 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:15:50.913 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.441541+0000", 2024-08-19T14:15:50.913 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0059191479999999999 2024-08-19T14:15:50.913 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.913 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.913 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:15:50.913 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.441585+0000", 2024-08-19T14:15:50.913 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.4743000000000002e-05 2024-08-19T14:15:50.914 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:15:50.914 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:15:50.914 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:15:50.914 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.914 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.914 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:40033 2.0 e113/102 2:0512d183:::benchmark_data_smithi045_46415_object40032:head v 113'2436, mlcod=113'2434)", 2024-08-19T14:15:50.914 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:02.895855+0000", 2024-08-19T14:15:50.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:50.916 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 167.99738615499999, 2024-08-19T14:15:50.916 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0095757759999999994, 2024-08-19T14:15:50.916 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:15:50.916 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:15:50.916 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:15:50.916 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:15:50.916 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.916 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:15:50.916 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.895855+0000", 2024-08-19T14:15:50.917 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:15:50.917 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.917 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.917 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:15:50.917 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.895855+0000", 2024-08-19T14:15:50.917 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:15:50.917 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.917 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.917 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:15:50.917 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.895856+0000", 2024-08-19T14:15:50.918 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.2099999999999995e-07 2024-08-19T14:15:50.918 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.918 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.918 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:15:50.918 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.895922+0000", 2024-08-19T14:15:50.918 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.5188e-05 2024-08-19T14:15:50.918 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.918 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.918 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:15:50.918 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.895935+0000", 2024-08-19T14:15:50.919 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3601000000000001e-05 2024-08-19T14:15:50.919 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.919 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.919 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:15:50.919 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.895943+0000", 2024-08-19T14:15:50.919 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.9170000000000006e-06 2024-08-19T14:15:50.919 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.919 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.919 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:15:50.919 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.896012+0000", 2024-08-19T14:15:50.920 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.9053000000000004e-05 2024-08-19T14:15:50.920 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.920 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.920 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:15:50.920 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.896035+0000", 2024-08-19T14:15:50.920 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2943999999999999e-05 2024-08-19T14:15:50.920 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.920 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.920 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:15:50.920 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.905344+0000", 2024-08-19T14:15:50.920 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0093086450000000008 2024-08-19T14:15:50.920 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.920 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.920 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:15:50.921 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.905431+0000", 2024-08-19T14:15:50.921 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.7507000000000001e-05 2024-08-19T14:15:50.921 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:15:50.921 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:15:50.921 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:15:50.921 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.921 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.921 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:40035 2.0 e113/102 2:0960ca99:::benchmark_data_smithi045_46415_object40034:head v 113'2437, mlcod=113'2434)", 2024-08-19T14:15:50.921 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:02.897909+0000", 2024-08-19T14:15:50.921 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 167.99533307600001, 2024-08-19T14:15:50.921 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0076402179999999998, 2024-08-19T14:15:50.921 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:15:50.921 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:15:50.921 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:15:50.922 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:15:50.922 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.922 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:15:50.922 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.897909+0000", 2024-08-19T14:15:50.922 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:15:50.922 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.922 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.922 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:15:50.922 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.897909+0000", 2024-08-19T14:15:50.922 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:15:50.922 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.922 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.922 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:15:50.923 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.897909+0000", 2024-08-19T14:15:50.923 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.33e-07 2024-08-19T14:15:50.923 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.923 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.923 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:15:50.923 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.897950+0000", 2024-08-19T14:15:50.923 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.0683000000000003e-05 2024-08-19T14:15:50.923 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.923 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.923 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:15:50.923 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.897966+0000", 2024-08-19T14:15:50.923 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6209999999999999e-05 2024-08-19T14:15:50.923 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.923 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.924 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:15:50.924 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.897974+0000", 2024-08-19T14:15:50.924 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4569999999999999e-06 2024-08-19T14:15:50.924 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.924 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.924 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:15:50.924 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.898035+0000", 2024-08-19T14:15:50.924 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.1033000000000003e-05 2024-08-19T14:15:50.924 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.924 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.924 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:15:50.924 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.898068+0000", 2024-08-19T14:15:50.924 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.3593000000000002e-05 2024-08-19T14:15:50.925 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.925 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.925 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:15:50.925 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.905435+0000", 2024-08-19T14:15:50.925 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0073662830000000004 2024-08-19T14:15:50.925 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.925 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.925 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:15:50.925 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.905549+0000", 2024-08-19T14:15:50.925 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000114026 2024-08-19T14:15:50.925 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:15:50.925 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:15:50.925 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:15:50.925 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.926 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.926 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:40069 2.0 e113/102 2:03e60c7a:::benchmark_data_smithi045_46415_object40068:head v 113'2440, mlcod=113'2436)", 2024-08-19T14:15:50.926 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:02.908487+0000", 2024-08-19T14:15:50.926 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 167.98475430299999, 2024-08-19T14:15:50.926 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0062491439999999999, 2024-08-19T14:15:50.926 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:15:50.926 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:15:50.926 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:15:50.926 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:15:50.926 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.926 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:15:50.926 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.908487+0000", 2024-08-19T14:15:50.926 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:15:50.926 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.927 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.927 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:15:50.927 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.908487+0000", 2024-08-19T14:15:50.927 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:15:50.927 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.927 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.927 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:15:50.927 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.908488+0000", 2024-08-19T14:15:50.927 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.8999999999999995e-07 2024-08-19T14:15:50.927 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.927 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.927 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:15:50.927 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.908517+0000", 2024-08-19T14:15:50.927 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.8643999999999999e-05 2024-08-19T14:15:50.928 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.928 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.928 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:15:50.928 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.908533+0000", 2024-08-19T14:15:50.928 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5770999999999999e-05 2024-08-19T14:15:50.928 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.928 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.928 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:15:50.928 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.908540+0000", 2024-08-19T14:15:50.928 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4000000000000003e-06 2024-08-19T14:15:50.928 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.928 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.928 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:15:50.928 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.908593+0000", 2024-08-19T14:15:50.929 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.3146999999999998e-05 2024-08-19T14:15:50.929 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.929 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.929 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:15:50.929 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.908614+0000", 2024-08-19T14:15:50.929 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0900999999999999e-05 2024-08-19T14:15:50.929 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.929 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.929 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:15:50.929 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.914692+0000", 2024-08-19T14:15:50.929 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0060776399999999996 2024-08-19T14:15:50.929 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.929 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.929 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:15:50.930 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.914736+0000", 2024-08-19T14:15:50.930 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.4851000000000001e-05 2024-08-19T14:15:50.930 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:15:50.930 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:15:50.930 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:15:50.930 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.930 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.930 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:43484 2.4 e114/102 2:285d10cd:::benchmark_data_smithi045_46415_object43483:head v 114'2732, mlcod=114'2729)", 2024-08-19T14:15:50.930 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:04.598813+0000", 2024-08-19T14:15:50.930 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 166.294428418, 2024-08-19T14:15:50.930 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0062693649999999998, 2024-08-19T14:15:50.930 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:15:50.930 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:15:50.930 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:15:50.931 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:15:50.931 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.931 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:15:50.931 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:04.598813+0000", 2024-08-19T14:15:50.931 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:15:50.931 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.931 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.931 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:15:50.931 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:04.598813+0000", 2024-08-19T14:15:50.931 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:15:50.931 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.931 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.931 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:15:50.932 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:04.598814+0000", 2024-08-19T14:15:50.932 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.9699999999999994e-07 2024-08-19T14:15:50.932 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.932 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.932 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:15:50.932 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:04.598850+0000", 2024-08-19T14:15:50.932 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.5865000000000003e-05 2024-08-19T14:15:50.932 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.932 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.932 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:15:50.932 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:04.598861+0000", 2024-08-19T14:15:50.932 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1284000000000001e-05 2024-08-19T14:15:50.932 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.932 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.933 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:15:50.933 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:04.598870+0000", 2024-08-19T14:15:50.933 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.9479999999999997e-06 2024-08-19T14:15:50.933 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.933 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.933 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:15:50.933 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:04.598935+0000", 2024-08-19T14:15:50.933 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.4364999999999998e-05 2024-08-19T14:15:50.933 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.933 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.933 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:15:50.933 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:04.598955+0000", 2024-08-19T14:15:50.933 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0849999999999999e-05 2024-08-19T14:15:50.933 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.934 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.934 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:15:50.934 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:04.604957+0000", 2024-08-19T14:15:50.934 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0060020000000000004 2024-08-19T14:15:50.934 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.934 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.934 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:15:50.934 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:04.605083+0000", 2024-08-19T14:15:50.934 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012505599999999999 2024-08-19T14:15:50.934 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:15:50.934 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:15:50.934 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:15:50.934 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.934 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.935 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:46495 2.4 e116/102 2:27583565:::benchmark_data_smithi045_46415_object46494:head v 116'2919, mlcod=116'2914)", 2024-08-19T14:15:50.935 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:06.037606+0000", 2024-08-19T14:15:50.935 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 164.855636022, 2024-08-19T14:15:50.935 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0079993879999999996, 2024-08-19T14:15:50.935 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:15:50.935 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:15:50.935 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:15:50.935 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:15:50.935 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.935 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:15:50.935 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:06.037606+0000", 2024-08-19T14:15:50.935 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:15:50.935 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.936 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.936 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:15:50.936 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:06.037606+0000", 2024-08-19T14:15:50.936 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:15:50.936 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.936 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.936 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:15:50.936 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:06.037606+0000", 2024-08-19T14:15:50.936 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2300000000000001e-07 2024-08-19T14:15:50.936 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.936 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.936 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:15:50.936 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:06.043150+0000", 2024-08-19T14:15:50.936 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0055434869999999997 2024-08-19T14:15:50.937 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.937 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.937 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:15:50.937 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:06.043160+0000", 2024-08-19T14:15:50.937 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0088999999999999e-05 2024-08-19T14:15:50.937 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.937 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.937 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:15:50.937 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:06.043167+0000", 2024-08-19T14:15:50.937 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.3309999999999998e-06 2024-08-19T14:15:50.937 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.937 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.937 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:15:50.937 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:06.043225+0000", 2024-08-19T14:15:50.938 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.8062000000000001e-05 2024-08-19T14:15:50.938 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.938 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.938 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:15:50.938 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:06.043246+0000", 2024-08-19T14:15:50.938 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1316000000000001e-05 2024-08-19T14:15:50.938 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.938 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.938 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:15:50.938 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:06.045577+0000", 2024-08-19T14:15:50.938 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0023308890000000001 2024-08-19T14:15:50.938 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.938 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.939 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:15:50.939 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:06.045605+0000", 2024-08-19T14:15:50.939 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.7691e-05 2024-08-19T14:15:50.939 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:15:50.939 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:15:50.939 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:15:50.939 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.939 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.939 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:48912 2.7 e117/102 2:e6ffd652:::benchmark_data_smithi045_46415_object48911:head v 117'2997, mlcod=117'2992)", 2024-08-19T14:15:50.939 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:07.468103+0000", 2024-08-19T14:15:50.939 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 163.42513909100001, 2024-08-19T14:15:50.939 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0061113349999999999, 2024-08-19T14:15:50.939 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:15:50.939 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:15:50.940 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:15:50.940 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:15:50.940 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.940 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:15:50.940 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468103+0000", 2024-08-19T14:15:50.940 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:15:50.940 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.940 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.940 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:15:50.940 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468103+0000", 2024-08-19T14:15:50.940 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:15:50.940 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.940 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.940 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:15:50.941 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468103+0000", 2024-08-19T14:15:50.941 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.8000000000000002e-07 2024-08-19T14:15:50.941 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.941 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.941 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:15:50.941 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468136+0000", 2024-08-19T14:15:50.941 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.3411000000000002e-05 2024-08-19T14:15:50.941 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.941 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.941 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:15:50.941 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468143+0000", 2024-08-19T14:15:50.941 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.5869999999999998e-06 2024-08-19T14:15:50.941 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.941 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.942 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:15:50.942 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468146+0000", 2024-08-19T14:15:50.942 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.304e-06 2024-08-19T14:15:50.942 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.942 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.942 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:15:50.942 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.470630+0000", 2024-08-19T14:15:50.942 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0024838009999999999 2024-08-19T14:15:50.942 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.942 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.942 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:15:50.942 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.470643+0000", 2024-08-19T14:15:50.942 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2964e-05 2024-08-19T14:15:50.942 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.943 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.943 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:15:50.943 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.474173+0000", 2024-08-19T14:15:50.943 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.003530423 2024-08-19T14:15:50.943 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.943 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.943 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:15:50.943 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.474214+0000", 2024-08-19T14:15:50.943 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.0565000000000002e-05 2024-08-19T14:15:50.943 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:15:50.943 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:15:50.943 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:15:50.943 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.943 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.944 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:48971 2.7 e117/102 2:ecf6a539:::benchmark_data_smithi045_46415_object48970:head v 117'2999, mlcod=117'2992)", 2024-08-19T14:15:50.944 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:07.468206+0000", 2024-08-19T14:15:50.944 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 163.425035157, 2024-08-19T14:15:50.944 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0076013820000000003, 2024-08-19T14:15:50.944 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:15:50.944 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:15:50.944 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:15:50.944 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:15:50.944 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.944 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:15:50.944 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468206+0000", 2024-08-19T14:15:50.944 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:15:50.944 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.944 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.945 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:15:50.945 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468206+0000", 2024-08-19T14:15:50.945 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:15:50.945 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.945 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:50 smithi045 ceph-mon[20143]: pgmap v464: 27 pgs: 1 peering, 26 active+clean; 3.0 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 635 op/s; 14 MiB/s, 226 objects/s recovering 2024-08-19T14:15:50.945 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:50 smithi045 ceph-mon[20143]: osdmap e187: 12 total, 12 up, 10 in 2024-08-19T14:15:50.946 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:50.946 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:15:50.947 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.947 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:15:50.947 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468207+0000", 2024-08-19T14:15:50.947 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6899999999999999e-07 2024-08-19T14:15:50.947 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.947 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.947 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:15:50.947 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468248+0000", 2024-08-19T14:15:50.947 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.0985999999999997e-05 2024-08-19T14:15:50.947 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.948 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.948 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:15:50.948 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468254+0000", 2024-08-19T14:15:50.948 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.9769999999999997e-06 2024-08-19T14:15:50.948 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.948 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.948 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:15:50.948 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468257+0000", 2024-08-19T14:15:50.948 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.0259999999999999e-06 2024-08-19T14:15:50.948 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.948 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.948 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:15:50.948 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.471956+0000", 2024-08-19T14:15:50.948 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0036996059999999998 2024-08-19T14:15:50.949 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.949 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.949 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:15:50.949 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.471974+0000", 2024-08-19T14:15:50.949 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8219999999999998e-05 2024-08-19T14:15:50.949 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.949 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.949 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:15:50.949 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.475774+0000", 2024-08-19T14:15:50.949 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0037994729999999998 2024-08-19T14:15:50.949 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.949 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.949 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:15:50.949 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.475808+0000", 2024-08-19T14:15:50.950 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.3924999999999998e-05 2024-08-19T14:15:50.950 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:15:50.950 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:15:50.950 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:15:50.950 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.950 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.950 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:48978 2.7 e117/102 2:e11d5fd6:::benchmark_data_smithi045_46415_object48977:head v 117'3000, mlcod=117'2992)", 2024-08-19T14:15:50.950 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:07.468262+0000", 2024-08-19T14:15:50.950 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 163.424979458, 2024-08-19T14:15:50.950 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0075744890000000002, 2024-08-19T14:15:50.950 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:15:50.950 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:15:50.950 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:15:50.951 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:15:50.951 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.951 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:15:50.951 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468262+0000", 2024-08-19T14:15:50.951 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:15:50.951 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.951 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.951 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:15:50.951 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468262+0000", 2024-08-19T14:15:50.951 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:15:50.951 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.951 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.951 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:15:50.951 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468262+0000", 2024-08-19T14:15:50.952 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.04e-07 2024-08-19T14:15:50.952 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.952 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.952 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:15:50.952 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468309+0000", 2024-08-19T14:15:50.952 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.6913999999999998e-05 2024-08-19T14:15:50.952 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.952 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.952 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:15:50.952 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468315+0000", 2024-08-19T14:15:50.952 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7139999999999998e-06 2024-08-19T14:15:50.952 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.952 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.952 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:15:50.953 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468318+0000", 2024-08-19T14:15:50.953 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.117e-06 2024-08-19T14:15:50.953 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.953 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.953 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:15:50.953 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.472643+0000", 2024-08-19T14:15:50.953 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0043251950000000004 2024-08-19T14:15:50.953 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.953 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.953 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:15:50.953 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.472657+0000", 2024-08-19T14:15:50.953 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3478e-05 2024-08-19T14:15:50.953 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.953 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.954 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:15:50.954 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.475812+0000", 2024-08-19T14:15:50.954 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0031556259999999999 2024-08-19T14:15:50.954 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.954 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.954 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:15:50.954 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.475837+0000", 2024-08-19T14:15:50.954 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.4241000000000001e-05 2024-08-19T14:15:50.954 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:15:50.954 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:15:50.954 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:15:50.954 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.954 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.954 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:48995 2.7 e117/102 2:e01c8116:::benchmark_data_smithi045_46415_object48994:head v 117'3001, mlcod=117'2992)", 2024-08-19T14:15:50.955 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:07.468324+0000", 2024-08-19T14:15:50.955 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 163.42491793100001, 2024-08-19T14:15:50.955 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0090529970000000001, 2024-08-19T14:15:50.955 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:15:50.955 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:15:50.955 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:15:50.955 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:15:50.955 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.955 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:15:50.955 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468324+0000", 2024-08-19T14:15:50.955 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:15:50.955 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.955 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.955 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:15:50.956 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468324+0000", 2024-08-19T14:15:50.956 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:15:50.956 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.956 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.956 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:15:50.956 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468324+0000", 2024-08-19T14:15:50.956 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.97e-07 2024-08-19T14:15:50.956 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.956 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.956 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:15:50.956 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468388+0000", 2024-08-19T14:15:50.956 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.4523999999999994e-05 2024-08-19T14:15:50.956 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.956 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.957 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:15:50.957 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468395+0000", 2024-08-19T14:15:50.957 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.579e-06 2024-08-19T14:15:50.957 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.957 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.957 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:15:50.957 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468398+0000", 2024-08-19T14:15:50.957 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.2780000000000002e-06 2024-08-19T14:15:50.957 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.957 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.957 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:15:50.957 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.473325+0000", 2024-08-19T14:15:50.957 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00492626 2024-08-19T14:15:50.957 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.958 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.958 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:15:50.958 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.473343+0000", 2024-08-19T14:15:50.958 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8788999999999999e-05 2024-08-19T14:15:50.958 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.958 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.958 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:15:50.958 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.477310+0000", 2024-08-19T14:15:50.958 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0039662819999999998 2024-08-19T14:15:50.958 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.958 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.958 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:15:50.958 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.477377+0000", 2024-08-19T14:15:50.958 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.7088000000000005e-05 2024-08-19T14:15:50.959 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:15:50.959 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:15:50.959 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:15:50.959 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.959 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.959 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:48996 2.7 e117/102 2:e8abf67b:::benchmark_data_smithi045_46415_object48995:head v 117'3002, mlcod=117'2992)", 2024-08-19T14:15:50.959 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:07.468406+0000", 2024-08-19T14:15:50.959 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 163.42483592299999, 2024-08-19T14:15:50.959 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0090294899999999994, 2024-08-19T14:15:50.959 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:15:50.959 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:15:50.959 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:15:50.959 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:15:50.959 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.960 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:15:50.960 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468406+0000", 2024-08-19T14:15:50.960 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:15:50.960 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.960 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.960 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:15:50.960 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468406+0000", 2024-08-19T14:15:50.960 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:15:50.960 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.960 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.960 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:15:50.960 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468406+0000", 2024-08-19T14:15:50.960 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0900000000000001e-07 2024-08-19T14:15:50.960 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.960 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.961 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:15:50.961 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468447+0000", 2024-08-19T14:15:50.961 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.0992000000000003e-05 2024-08-19T14:15:50.961 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.961 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.961 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:15:50.961 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468452+0000", 2024-08-19T14:15:50.961 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2179999999999998e-06 2024-08-19T14:15:50.961 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.961 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.961 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:15:50.961 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468455+0000", 2024-08-19T14:15:50.961 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.796e-06 2024-08-19T14:15:50.962 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.962 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.962 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:15:50.962 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.474288+0000", 2024-08-19T14:15:50.962 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0058327689999999998 2024-08-19T14:15:50.962 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.962 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.962 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:15:50.962 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.474307+0000", 2024-08-19T14:15:50.962 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9031000000000001e-05 2024-08-19T14:15:50.962 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.962 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.962 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:15:50.962 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.477383+0000", 2024-08-19T14:15:50.963 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0030758790000000001 2024-08-19T14:15:50.963 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.963 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.963 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:15:50.963 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.477435+0000", 2024-08-19T14:15:50.963 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2596000000000003e-05 2024-08-19T14:15:50.963 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:15:50.963 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:15:50.963 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:15:50.963 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.963 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.963 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:49005 2.7 e117/102 2:e58e73b6:::benchmark_data_smithi045_46415_object49004:head v 117'3003, mlcod=117'2992)", 2024-08-19T14:15:50.963 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:07.468460+0000", 2024-08-19T14:15:50.963 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 163.424781822, 2024-08-19T14:15:50.964 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0098532910000000001, 2024-08-19T14:15:50.964 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:15:50.964 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:15:50.964 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:15:50.964 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:15:50.964 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.964 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:15:50.964 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468460+0000", 2024-08-19T14:15:50.964 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:15:50.964 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.964 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.964 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:15:50.964 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468460+0000", 2024-08-19T14:15:50.965 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:15:50.965 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.965 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.965 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:15:50.965 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468460+0000", 2024-08-19T14:15:50.965 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5900000000000001e-07 2024-08-19T14:15:50.965 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.965 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.965 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:15:50.965 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468518+0000", 2024-08-19T14:15:50.965 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.8536000000000003e-05 2024-08-19T14:15:50.965 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.965 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.966 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:15:50.966 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468524+0000", 2024-08-19T14:15:50.966 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.8039999999999998e-06 2024-08-19T14:15:50.966 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.966 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.966 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:15:50.966 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468527+0000", 2024-08-19T14:15:50.966 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.004e-06 2024-08-19T14:15:50.966 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.967 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.967 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:15:50.967 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.475085+0000", 2024-08-19T14:15:50.967 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0065574090000000002 2024-08-19T14:15:50.967 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.967 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.967 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:15:50.967 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.475106+0000", 2024-08-19T14:15:50.967 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1056999999999998e-05 2024-08-19T14:15:50.967 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.967 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.967 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:15:50.967 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.478267+0000", 2024-08-19T14:15:50.968 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0031614049999999999 2024-08-19T14:15:50.968 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.968 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.968 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:15:50.968 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.478313+0000", 2024-08-19T14:15:50.968 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.5917000000000001e-05 2024-08-19T14:15:50.968 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:15:50.968 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:15:50.968 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:15:50.968 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.969 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.969 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:49015 2.7 e117/102 2:e4d8c3e1:::benchmark_data_smithi045_46415_object49014:head v 117'3004, mlcod=117'2992)", 2024-08-19T14:15:50.969 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:07.468533+0000", 2024-08-19T14:15:50.969 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 163.42470875800001, 2024-08-19T14:15:50.969 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0098176270000000006, 2024-08-19T14:15:50.969 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:15:50.969 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:15:50.969 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:15:50.969 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:15:50.969 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.969 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:15:50.969 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468533+0000", 2024-08-19T14:15:50.969 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:15:50.969 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.970 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.970 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:15:50.970 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468533+0000", 2024-08-19T14:15:50.970 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:15:50.970 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.970 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.970 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:15:50.970 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468533+0000", 2024-08-19T14:15:50.970 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2600000000000001e-07 2024-08-19T14:15:50.970 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.970 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.970 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:15:50.970 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468577+0000", 2024-08-19T14:15:50.971 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.3436000000000002e-05 2024-08-19T14:15:50.971 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.971 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.971 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:15:50.971 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468582+0000", 2024-08-19T14:15:50.971 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4399999999999996e-06 2024-08-19T14:15:50.971 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.971 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.971 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:15:50.971 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468585+0000", 2024-08-19T14:15:50.971 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.8619999999999999e-06 2024-08-19T14:15:50.971 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.971 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.972 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:15:50.972 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.475870+0000", 2024-08-19T14:15:50.972 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0072852919999999996 2024-08-19T14:15:50.972 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.972 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.972 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:15:50.972 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.475894+0000", 2024-08-19T14:15:50.972 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.3858000000000001e-05 2024-08-19T14:15:50.972 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.972 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.972 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:15:50.972 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.478319+0000", 2024-08-19T14:15:50.972 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0024247729999999999 2024-08-19T14:15:50.972 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.973 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.973 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:15:50.973 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.478350+0000", 2024-08-19T14:15:50.973 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.1739999999999998e-05 2024-08-19T14:15:50.973 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:15:50.973 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:15:50.973 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:15:50.973 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.973 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.973 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:49016 2.7 e117/102 2:e52959d5:::benchmark_data_smithi045_46415_object49015:head v 117'3005, mlcod=117'2992)", 2024-08-19T14:15:50.973 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:07.468590+0000", 2024-08-19T14:15:50.973 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 163.42465188, 2024-08-19T14:15:50.973 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.011903967999999999, 2024-08-19T14:15:50.973 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:15:50.974 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:15:50.974 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:15:50.974 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:15:50.974 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.974 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:15:50.974 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468590+0000", 2024-08-19T14:15:50.974 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:15:50.974 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.974 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.974 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:15:50.974 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468590+0000", 2024-08-19T14:15:50.974 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:15:50.974 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.974 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.975 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:15:50.975 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468590+0000", 2024-08-19T14:15:50.975 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.54e-07 2024-08-19T14:15:50.975 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.975 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.975 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:15:50.975 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468632+0000", 2024-08-19T14:15:50.975 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.2459999999999997e-05 2024-08-19T14:15:50.975 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.975 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.975 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:15:50.975 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468638+0000", 2024-08-19T14:15:50.975 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2410000000000003e-06 2024-08-19T14:15:50.975 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.976 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.976 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:15:50.976 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468641+0000", 2024-08-19T14:15:50.976 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.061e-06 2024-08-19T14:15:50.976 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.976 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.976 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:15:50.976 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.476571+0000", 2024-08-19T14:15:50.976 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0079302920000000002 2024-08-19T14:15:50.976 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.976 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.976 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:15:50.976 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.476593+0000", 2024-08-19T14:15:50.976 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2045e-05 2024-08-19T14:15:50.977 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.977 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.977 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:15:50.977 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.480447+0000", 2024-08-19T14:15:50.977 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.003853825 2024-08-19T14:15:50.977 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.977 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.977 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:15:50.977 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.480494+0000", 2024-08-19T14:15:50.977 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.689e-05 2024-08-19T14:15:50.977 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:15:50.977 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:15:50.977 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:15:50.977 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.978 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.978 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:49017 2.7 e117/102 2:eca28b17:::benchmark_data_smithi045_46415_object49016:head v 117'3006, mlcod=117'2992)", 2024-08-19T14:15:50.978 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:07.468646+0000", 2024-08-19T14:15:50.978 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 163.42459601600001, 2024-08-19T14:15:50.978 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.011882515999999999, 2024-08-19T14:15:50.978 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:15:50.978 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:15:50.978 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:15:50.978 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:15:50.978 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.978 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:15:50.978 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468646+0000", 2024-08-19T14:15:50.978 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:15:50.979 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.979 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.979 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:15:50.979 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468646+0000", 2024-08-19T14:15:50.979 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:15:50.979 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.979 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.979 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:15:50.979 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468646+0000", 2024-08-19T14:15:50.979 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5800000000000001e-07 2024-08-19T14:15:50.979 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.979 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.979 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:15:50.979 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468686+0000", 2024-08-19T14:15:50.980 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.0732999999999998e-05 2024-08-19T14:15:50.980 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.980 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.980 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:15:50.980 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468692+0000", 2024-08-19T14:15:50.980 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0860000000000001e-06 2024-08-19T14:15:50.980 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.980 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.980 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:15:50.980 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468694+0000", 2024-08-19T14:15:50.980 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.864e-06 2024-08-19T14:15:50.980 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.980 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.980 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:15:50.981 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.477463+0000", 2024-08-19T14:15:50.981 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0087685190000000007 2024-08-19T14:15:50.981 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.981 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.981 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:15:50.981 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.477483+0000", 2024-08-19T14:15:50.981 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0449999999999999e-05 2024-08-19T14:15:50.981 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.981 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.981 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:15:50.981 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.480501+0000", 2024-08-19T14:15:50.981 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.003017802 2024-08-19T14:15:50.981 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.981 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.982 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:15:50.982 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.480528+0000", 2024-08-19T14:15:50.982 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.6903999999999999e-05 2024-08-19T14:15:50.982 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:15:50.982 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:15:50.982 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:15:50.982 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.982 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.982 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:49027 2.7 e117/102 2:e96ce7ac:::benchmark_data_smithi045_46415_object49026:head v 117'3007, mlcod=117'2992)", 2024-08-19T14:15:50.982 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:07.468699+0000", 2024-08-19T14:15:50.982 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 163.42454260400001, 2024-08-19T14:15:50.982 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.011858521, 2024-08-19T14:15:50.982 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:15:50.982 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:15:50.983 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:15:50.983 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:15:50.983 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.983 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:15:50.983 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468699+0000", 2024-08-19T14:15:50.983 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:15:50.983 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.983 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.983 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:15:50.983 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468699+0000", 2024-08-19T14:15:50.983 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:15:50.983 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.983 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.984 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:15:50.984 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468699+0000", 2024-08-19T14:15:50.984 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5599999999999999e-07 2024-08-19T14:15:50.984 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.984 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.984 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:15:50.984 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468743+0000", 2024-08-19T14:15:50.984 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.3741e-05 2024-08-19T14:15:50.984 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.984 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.984 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:15:50.984 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468748+0000", 2024-08-19T14:15:50.984 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1000000000000003e-06 2024-08-19T14:15:50.984 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.985 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.985 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:15:50.985 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468751+0000", 2024-08-19T14:15:50.985 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.8260000000000001e-06 2024-08-19T14:15:50.985 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.985 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.985 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:15:50.985 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.478397+0000", 2024-08-19T14:15:50.985 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0096462240000000001 2024-08-19T14:15:50.985 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.985 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.985 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:15:50.985 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.478415+0000", 2024-08-19T14:15:50.985 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8450000000000001e-05 2024-08-19T14:15:50.986 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.986 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.986 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:15:50.986 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.480533+0000", 2024-08-19T14:15:50.986 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.002117196 2024-08-19T14:15:50.986 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.986 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.986 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:15:50.986 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.480558+0000", 2024-08-19T14:15:50.986 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.4828e-05 2024-08-19T14:15:50.986 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:15:50.986 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:15:50.986 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:15:50.987 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.987 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.987 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:49030 2.7 e117/102 2:e915209e:::benchmark_data_smithi045_46415_object49029:head v 117'3008, mlcod=117'2992)", 2024-08-19T14:15:50.987 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:07.468755+0000", 2024-08-19T14:15:50.987 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 163.424486317, 2024-08-19T14:15:50.987 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.015403441, 2024-08-19T14:15:50.987 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:15:50.987 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:15:50.987 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:15:50.987 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:15:50.987 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.987 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:15:50.987 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468755+0000", 2024-08-19T14:15:50.987 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:15:50.987 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.988 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.988 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:15:50.988 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468755+0000", 2024-08-19T14:15:50.988 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:15:50.988 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.988 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.988 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:15:50.988 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468755+0000", 2024-08-19T14:15:50.988 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5599999999999999e-07 2024-08-19T14:15:50.988 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.988 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.988 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:15:50.988 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468805+0000", 2024-08-19T14:15:50.988 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.9622e-05 2024-08-19T14:15:50.989 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.989 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.989 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:15:50.989 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468811+0000", 2024-08-19T14:15:50.989 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.6119999999999998e-06 2024-08-19T14:15:50.989 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.989 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.989 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:15:50.989 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468813+0000", 2024-08-19T14:15:50.989 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.7769999999999998e-06 2024-08-19T14:15:50.989 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.989 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.989 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:15:50.989 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.479393+0000", 2024-08-19T14:15:50.990 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.010579646999999999 2024-08-19T14:15:50.990 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.990 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.990 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:15:50.990 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.479412+0000", 2024-08-19T14:15:50.990 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8728000000000001e-05 2024-08-19T14:15:50.990 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.990 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.990 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:15:50.990 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.484094+0000", 2024-08-19T14:15:50.990 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0046820009999999999 2024-08-19T14:15:50.990 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.990 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.991 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:15:50.991 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.484159+0000", 2024-08-19T14:15:50.991 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.4898000000000001e-05 2024-08-19T14:15:50.991 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:15:50.991 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:15:50.991 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:15:50.991 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.991 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.991 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:49049 2.7 e117/102 2:eb29c0a1:::benchmark_data_smithi045_46415_object49048:head v 117'3009, mlcod=117'2992)", 2024-08-19T14:15:50.991 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:07.468819+0000", 2024-08-19T14:15:50.991 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 163.42442306999999, 2024-08-19T14:15:50.991 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.015383332, 2024-08-19T14:15:50.991 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:15:50.991 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:15:50.992 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:15:50.992 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:15:50.992 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.992 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:15:50.992 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468819+0000", 2024-08-19T14:15:50.992 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:15:50.992 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.992 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.992 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:15:50.992 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468819+0000", 2024-08-19T14:15:50.992 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:15:50.992 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.992 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.992 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:15:50.992 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468819+0000", 2024-08-19T14:15:50.993 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2000000000000001e-07 2024-08-19T14:15:50.993 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.993 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.993 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:15:50.993 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468854+0000", 2024-08-19T14:15:50.993 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.5079000000000001e-05 2024-08-19T14:15:50.993 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.993 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.993 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:15:50.993 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468859+0000", 2024-08-19T14:15:50.993 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1710000000000001e-06 2024-08-19T14:15:50.993 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.993 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.993 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:15:50.994 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468862+0000", 2024-08-19T14:15:50.994 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.0599999999999999e-06 2024-08-19T14:15:50.994 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.994 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.994 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:15:50.994 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.480596+0000", 2024-08-19T14:15:50.994 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.011734058 2024-08-19T14:15:50.994 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.994 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.994 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:15:50.994 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.480615+0000", 2024-08-19T14:15:50.994 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8919e-05 2024-08-19T14:15:50.994 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.994 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.995 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:15:50.995 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.484164+0000", 2024-08-19T14:15:50.995 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.003548486 2024-08-19T14:15:50.995 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:50.995 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:50.995 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:15:50.995 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.484202+0000", 2024-08-19T14:15:50.995 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.8339e-05 2024-08-19T14:15:50.995 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:15:50.995 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:15:50.995 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:15:50.995 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:15:50.995 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:15:50.995 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:15:51.047 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:15:51.070 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:51.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:51.071 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:15:51.172 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:15:51.199 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:51.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:51.201 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:15:51.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:50 smithi032 ceph-mon[20004]: pgmap v464: 27 pgs: 1 peering, 26 active+clean; 3.0 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 635 op/s; 14 MiB/s, 226 objects/s recovering 2024-08-19T14:15:51.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:50 smithi032 ceph-mon[20004]: osdmap e187: 12 total, 12 up, 10 in 2024-08-19T14:15:51.302 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:15:51.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:51.328 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:51.328 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:15:51.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:50 smithi031 ceph-mon[16336]: pgmap v464: 27 pgs: 1 peering, 26 active+clean; 3.0 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 635 op/s; 14 MiB/s, 226 objects/s recovering 2024-08-19T14:15:51.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:50 smithi031 ceph-mon[16336]: osdmap e187: 12 total, 12 up, 10 in 2024-08-19T14:15:51.429 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:15:51.450 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:51.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:51.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:15:51.552 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:15:51.589 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:51.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:51.590 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:15:51.691 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:15:51.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:51.727 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:51.727 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:15:51.820 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-08-19T14:15:51.827 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:15:51.884 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is set 2024-08-19T14:15:51.897 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:51.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:51.899 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:15:52.000 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:15:52.043 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:52.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:52.044 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:15:52.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:51 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2546068405' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:15:52.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:51 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:15:52.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:51 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2546068405' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:15:52.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:51 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:15:52.146 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:15:52.153 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:15:52.170 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:52.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:52.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:15:52.181 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:15:52.208 INFO:teuthology.orchestra.run.smithi031.stdout:moved 0 / 81 (0%) 2024-08-19T14:15:52.208 INFO:teuthology.orchestra.run.smithi031.stdout:avg 8.1 2024-08-19T14:15:52.208 INFO:teuthology.orchestra.run.smithi031.stdout:stddev 2.16564 -> 2.16564 (expected baseline 2.7) 2024-08-19T14:15:52.208 INFO:teuthology.orchestra.run.smithi031.stdout:min osd.5 with 4 -> 4 pgs (0.493827 -> 0.493827 * mean) 2024-08-19T14:15:52.209 INFO:teuthology.orchestra.run.smithi031.stdout:max osd.9 with 12 -> 12 pgs (1.48148 -> 1.48148 * mean) 2024-08-19T14:15:52.209 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:15:52.209 INFO:teuthology.orchestra.run.smithi031.stdout:oload 1000 2024-08-19T14:15:52.209 INFO:teuthology.orchestra.run.smithi031.stdout:max_change 0.05 2024-08-19T14:15:52.209 INFO:teuthology.orchestra.run.smithi031.stdout:max_change_osds 4 2024-08-19T14:15:52.209 INFO:teuthology.orchestra.run.smithi031.stdout:average_utilization 0.0109 2024-08-19T14:15:52.209 INFO:teuthology.orchestra.run.smithi031.stdout:overload_utilization 0.1092 2024-08-19T14:15:52.209 INFO:teuthology.orchestra.run.smithi031.stderr:no change 2024-08-19T14:15:52.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:51 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2546068405' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:15:52.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:51 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:15:52.273 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:15:52.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:52.315 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:52.316 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:15:52.417 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:15:52.440 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:52.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:52.441 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:15:52.543 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:15:52.569 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:52.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:52.571 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:15:52.671 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:15:52.697 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:52.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:52.698 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:15:52.800 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:15:52.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:52.835 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:52.835 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:15:52.928 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:15:52.928 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [ 2024-08-19T14:15:52.928 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:52.928 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24580.0:202649 2.a 2:53ca8704:::benchmark_data_smithi045_46415_object79368:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-08-19T14:15:52.928 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:15:52.918047+0000", 2024-08-19T14:15:52.929 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.0085358389999999999, 2024-08-19T14:15:52.929 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0085652720000000005, 2024-08-19T14:15:52.929 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:15:52.929 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:15:52.929 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "waiting for sub ops", 2024-08-19T14:15:52.929 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:15:52.929 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24580", 2024-08-19T14:15:52.929 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:15:52.929 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 202649 2024-08-19T14:15:52.929 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:52.929 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:15:52.929 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:52.929 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:15:52.929 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:15:52.918047+0000", 2024-08-19T14:15:52.930 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:15:52.930 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:52.930 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:52.930 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:15:52.930 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:15:52.918047+0000", 2024-08-19T14:15:52.930 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:15:52.930 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:52.930 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:52.930 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:15:52.930 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:15:52.918049+0000", 2024-08-19T14:15:52.930 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9099999999999999e-06 2024-08-19T14:15:52.930 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:52.930 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:52.931 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:15:52.931 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:15:52.918055+0000", 2024-08-19T14:15:52.931 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.677e-06 2024-08-19T14:15:52.931 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:52.931 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:52.931 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:15:52.931 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:15:52.918071+0000", 2024-08-19T14:15:52.931 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6872e-05 2024-08-19T14:15:52.931 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:52.931 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:52.931 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:15:52.931 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:15:52.918080+0000", 2024-08-19T14:15:52.931 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.208e-06 2024-08-19T14:15:52.931 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:52.932 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:52.932 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:15:52.932 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:15:52.918137+0000", 2024-08-19T14:15:52.932 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7632e-05 2024-08-19T14:15:52.932 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:52.932 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:52.932 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:15:52.932 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:15:52.918246+0000", 2024-08-19T14:15:52.932 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000108236 2024-08-19T14:15:52.932 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:52.932 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:52.932 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 3,9", 2024-08-19T14:15:52.932 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:15:52.918353+0000", 2024-08-19T14:15:52.933 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000107735 2024-08-19T14:15:52.933 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:52.933 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:52.933 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:15:52.933 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:15:52.924945+0000", 2024-08-19T14:15:52.933 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0065921620000000004 2024-08-19T14:15:52.933 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:52.933 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:52.933 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:15:52.933 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:15:52.925581+0000", 2024-08-19T14:15:52.933 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00063539200000000003 2024-08-19T14:15:52.933 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:15:52.934 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:15:52.934 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:15:52.934 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:52.934 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:52.934 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24580.0:202650 2.c 2:35c8be4b:::benchmark_data_smithi045_46415_object79369:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-08-19T14:15:52.934 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:15:52.920241+0000", 2024-08-19T14:15:52.934 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.0063422679999999999, 2024-08-19T14:15:52.934 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0064612849999999998, 2024-08-19T14:15:52.934 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:15:52.934 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:15:52.934 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "waiting for sub ops", 2024-08-19T14:15:52.934 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:15:52.934 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24580", 2024-08-19T14:15:52.935 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:15:52.935 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 202650 2024-08-19T14:15:52.935 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:52.935 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:15:52.935 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:52.935 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:15:52.935 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:15:52.920241+0000", 2024-08-19T14:15:52.935 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:15:52.935 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:52.935 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:52.936 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:15:52.936 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:15:52.920241+0000", 2024-08-19T14:15:52.936 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:15:52.936 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:52.936 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:52.936 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:15:52.936 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:15:52.920242+0000", 2024-08-19T14:15:52.936 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.384e-06 2024-08-19T14:15:52.936 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:52.936 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:52.936 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:15:52.937 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:15:52.920247+0000", 2024-08-19T14:15:52.937 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.181e-06 2024-08-19T14:15:52.937 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:52.937 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:52.937 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:15:52.937 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:15:52.920259+0000", 2024-08-19T14:15:52.937 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1875999999999999e-05 2024-08-19T14:15:52.937 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:52.937 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:52.937 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:15:52.937 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:15:52.920266+0000", 2024-08-19T14:15:52.937 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.1199999999999996e-06 2024-08-19T14:15:52.937 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:52.938 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:52.938 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:15:52.938 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:15:52.920508+0000", 2024-08-19T14:15:52.938 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000241721 2024-08-19T14:15:52.938 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:52.938 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:52.938 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:15:52.938 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:15:52.920613+0000", 2024-08-19T14:15:52.938 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00010475399999999999 2024-08-19T14:15:52.938 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:52.938 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:52.938 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,1", 2024-08-19T14:15:52.939 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:15:52.920719+0000", 2024-08-19T14:15:52.939 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000105969 2024-08-19T14:15:52.939 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:52.939 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:52.939 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:15:52.939 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:15:52.925017+0000", 2024-08-19T14:15:52.939 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0042988480000000001 2024-08-19T14:15:52.939 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:15:52.939 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:15:52.939 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:15:52.939 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:52.940 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:52.940 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:202658 2.0 e188/132 2:09881b7d:::benchmark_data_smithi045_46415_object79377:head v 188'12668, mlcod=188'12666)", 2024-08-19T14:15:52.940 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:15:52.926166+0000", 2024-08-19T14:15:52.940 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.00041700100000000001, 2024-08-19T14:15:52.940 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00061212400000000002, 2024-08-19T14:15:52.940 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:15:52.940 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:15:52.940 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-08-19T14:15:52.940 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:15:52.940 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:52.940 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:15:52.940 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:15:52.926166+0000", 2024-08-19T14:15:52.940 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:15:52.941 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:52.941 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:52.941 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:15:52.941 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:15:52.926166+0000", 2024-08-19T14:15:52.941 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:15:52.941 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:52.941 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:52.941 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:15:52.941 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:15:52.926167+0000", 2024-08-19T14:15:52.941 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.61e-07 2024-08-19T14:15:52.941 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:52.941 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:52.941 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:15:52.941 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:15:52.926173+0000", 2024-08-19T14:15:52.942 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.3579999999999999e-06 2024-08-19T14:15:52.942 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:52.942 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:52.942 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:15:52.942 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:15:52.926187+0000", 2024-08-19T14:15:52.942 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3900000000000001e-05 2024-08-19T14:15:52.942 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:52.942 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:52.942 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:15:52.942 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:15:52.926193+0000", 2024-08-19T14:15:52.942 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.4219999999999997e-06 2024-08-19T14:15:52.942 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:52.942 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:52.943 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:15:52.943 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:15:52.926252+0000", 2024-08-19T14:15:52.943 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.8650999999999998e-05 2024-08-19T14:15:52.943 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:52.943 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:52.943 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:15:52.943 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:15:52.926275+0000", 2024-08-19T14:15:52.943 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2935999999999999e-05 2024-08-19T14:15:52.943 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:15:52.943 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:15:52.943 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:15:52.943 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:52.943 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:52.944 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:52 smithi045 ceph-mon[20143]: pgmap v466: 27 pgs: 27 active+clean; 2.9 GiB data, 9.8 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 740 op/s; 12 MiB/s, 188 objects/s recovering 2024-08-19T14:15:52.944 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:52 smithi045 ceph-mon[20143]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:15:52.944 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:52 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:15:52.944 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:52 smithi045 ceph-mon[20143]: osdmap e188: 12 total, 12 up, 10 in 2024-08-19T14:15:52.944 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:52 smithi045 ceph-mon[20143]: from='client.24962 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T14:15:52.944 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:202659 2.b e188/102 2:db89b715:::benchmark_data_smithi045_46415_object79378:head v 188'12640, mlcod=188'12638)", 2024-08-19T14:15:52.944 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:15:52.926319+0000", 2024-08-19T14:15:52.944 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.000264071, 2024-08-19T14:15:52.945 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00051603200000000004, 2024-08-19T14:15:52.945 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:15:52.945 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:15:52.945 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-08-19T14:15:52.945 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:15:52.945 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:52.945 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:15:52.945 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:15:52.926319+0000", 2024-08-19T14:15:52.945 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:15:52.945 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:52.945 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:52.945 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:15:52.945 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:15:52.926319+0000", 2024-08-19T14:15:52.945 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:15:52.945 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:52.946 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:52.946 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:15:52.946 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:15:52.926319+0000", 2024-08-19T14:15:52.946 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.6000000000000004e-07 2024-08-19T14:15:52.946 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:52.946 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:52.946 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:15:52.946 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:15:52.926324+0000", 2024-08-19T14:15:52.946 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.7210000000000002e-06 2024-08-19T14:15:52.946 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:52.946 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:52.946 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:15:52.946 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:15:52.926333+0000", 2024-08-19T14:15:52.946 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.816e-06 2024-08-19T14:15:52.947 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:52.947 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:52.947 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:15:52.947 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:15:52.926338+0000", 2024-08-19T14:15:52.947 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4909999999999996e-06 2024-08-19T14:15:52.947 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:52.947 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:52.947 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:15:52.947 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:15:52.926404+0000", 2024-08-19T14:15:52.947 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.5196000000000004e-05 2024-08-19T14:15:52.947 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:15:52.947 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:15:52.947 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:15:52.947 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:15:52.926423+0000", 2024-08-19T14:15:52.948 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9332000000000001e-05 2024-08-19T14:15:52.948 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:15:52.948 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:15:52.948 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:15:52.948 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:15:52.948 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-08-19T14:15:52.948 INFO:teuthology.orchestra.run.smithi045.stdout: "num_ops": 4 2024-08-19T14:15:52.948 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:15:52.948 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:15:52.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:52.986 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:52.986 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:15:53.087 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:15:53.122 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:53.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:53.128 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:15:53.196 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:52 smithi031 ceph-mon[16336]: pgmap v466: 27 pgs: 27 active+clean; 2.9 GiB data, 9.8 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 740 op/s; 12 MiB/s, 188 objects/s recovering 2024-08-19T14:15:53.196 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:52 smithi031 ceph-mon[16336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:15:53.196 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:52 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:15:53.196 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:52 smithi031 ceph-mon[16336]: osdmap e188: 12 total, 12 up, 10 in 2024-08-19T14:15:53.196 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:52 smithi031 ceph-mon[16336]: from='client.24962 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T14:15:53.229 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:15:53.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:52 smithi032 ceph-mon[20004]: pgmap v466: 27 pgs: 27 active+clean; 2.9 GiB data, 9.8 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 740 op/s; 12 MiB/s, 188 objects/s recovering 2024-08-19T14:15:53.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:52 smithi032 ceph-mon[20004]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:15:53.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:52 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:15:53.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:52 smithi032 ceph-mon[20004]: osdmap e188: 12 total, 12 up, 10 in 2024-08-19T14:15:53.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:52 smithi032 ceph-mon[20004]: from='client.24962 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T14:15:53.265 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:53.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:53.267 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:15:53.368 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:15:53.399 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:53.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:53.400 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:15:53.501 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:15:53.535 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:53.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:53.536 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:15:53.637 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:15:53.662 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:53.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:53.663 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:15:53.764 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:15:53.769 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-08-19T14:15:53.769 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd reweight-by-utilization 110 0.05 2024-08-19T14:15:53.790 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:53.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:53.791 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:15:53.855 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-08-19T14:15:53.892 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:15:53.927 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:53.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:53.928 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:15:54.029 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:15:54.054 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:54.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:54.056 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:15:54.091 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:15:54.092 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:15:54.133 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:15:54.133 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:15:54.157 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:15:54.180 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:15:54.180 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:15:54.202 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:15:54.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:54.214 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:54.214 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:15:54.215 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:15:54.215 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:15:54.219 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:15:54.253 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:15:54.254 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:15:54.288 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:15:54.288 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:15:54.316 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:15:54.330 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:15:54.330 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:15:54.350 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:54.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:54.351 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:15:54.361 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:15:54.361 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:15:54.396 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:15:54.397 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:15:54.428 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:15:54.429 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:15:54.452 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:15:54.460 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:15:54.460 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:15:54.490 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:54.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:54.491 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:15:54.493 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:15:54.493 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:15:54.592 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:15:54.621 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:54.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:54.622 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:15:54.651 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-19T14:15:54.724 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:15:54.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:54.750 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:54.751 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:15:54.751 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:15:54.751 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:15:54.851 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:15:54.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:54.895 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:54.895 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:15:54.996 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:15:55.022 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:55.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:55.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:15:55.125 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:15:55.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:55.152 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:55.152 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:15:55.253 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:15:55.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:54 smithi032 ceph-mon[20004]: pgmap v468: 27 pgs: 27 active+clean; 2.8 GiB data, 9.4 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 843 op/s 2024-08-19T14:15:55.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:54 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3747934811' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:15:55.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:54 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:15:55.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:54 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:15:55.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:55.292 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:55.293 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:15:55.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:54 smithi045 ceph-mon[20143]: pgmap v468: 27 pgs: 27 active+clean; 2.8 GiB data, 9.4 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 843 op/s 2024-08-19T14:15:55.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:54 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3747934811' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:15:55.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:54 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:15:55.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:54 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:15:55.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:54 smithi031 ceph-mon[16336]: pgmap v468: 27 pgs: 27 active+clean; 2.8 GiB data, 9.4 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 843 op/s 2024-08-19T14:15:55.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:54 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3747934811' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:15:55.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:54 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:15:55.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:54 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:15:55.394 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:15:55.421 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:55.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:55.422 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:15:55.523 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:15:55.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:55.562 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:55.562 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:15:55.588 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:15:55.663 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:15:55.689 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:55.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:55.690 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:15:55.755 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-08-19T14:15:55.793 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:15:55.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:55.824 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:55.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:15:55.925 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:15:55.949 INFO:teuthology.orchestra.run.smithi031.stdout:moved 2 / 81 (2.46914%) 2024-08-19T14:15:55.949 INFO:teuthology.orchestra.run.smithi031.stdout:avg 8.1 2024-08-19T14:15:55.949 INFO:teuthology.orchestra.run.smithi031.stdout:stddev 2.16564 -> 1.92094 (expected baseline 2.7) 2024-08-19T14:15:55.949 INFO:teuthology.orchestra.run.smithi031.stdout:min osd.5 with 4 -> 4 pgs (0.493827 -> 0.493827 * mean) 2024-08-19T14:15:55.949 INFO:teuthology.orchestra.run.smithi031.stdout:max osd.9 with 12 -> 11 pgs (1.48148 -> 1.35802 * mean) 2024-08-19T14:15:55.949 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:15:55.950 INFO:teuthology.orchestra.run.smithi031.stdout:oload 110 2024-08-19T14:15:55.950 INFO:teuthology.orchestra.run.smithi031.stdout:max_change 0.05 2024-08-19T14:15:55.950 INFO:teuthology.orchestra.run.smithi031.stdout:max_change_osds 4 2024-08-19T14:15:55.950 INFO:teuthology.orchestra.run.smithi031.stdout:average_utilization 0.0103 2024-08-19T14:15:55.950 INFO:teuthology.orchestra.run.smithi031.stdout:overload_utilization 0.0113 2024-08-19T14:15:55.950 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9 weight 1.0000 -> 0.9500 2024-08-19T14:15:55.950 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7 weight 1.0000 -> 0.9500 2024-08-19T14:15:55.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:55.958 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:55.958 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:15:56.024 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:55 smithi031 ceph-mon[16336]: osdmap e189: 12 total, 12 up, 10 in 2024-08-19T14:15:56.024 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:55 smithi031 ceph-mon[16336]: from='client.24968 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T14:15:56.024 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:55 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'7':'62260','9':'62260'}"}]: dispatch 2024-08-19T14:15:56.024 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:55 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:15:56.025 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:55 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:15:56.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:55 smithi045 ceph-mon[20143]: osdmap e189: 12 total, 12 up, 10 in 2024-08-19T14:15:56.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:55 smithi045 ceph-mon[20143]: from='client.24968 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T14:15:56.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:55 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'7':'62260','9':'62260'}"}]: dispatch 2024-08-19T14:15:56.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:55 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:15:56.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:55 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:15:56.059 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:15:56.086 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:56.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:56.087 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:15:56.106 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:15:56.188 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:15:56.210 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:56.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:56.211 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:15:56.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:55 smithi032 ceph-mon[20004]: osdmap e189: 12 total, 12 up, 10 in 2024-08-19T14:15:56.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:55 smithi032 ceph-mon[20004]: from='client.24968 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-08-19T14:15:56.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:55 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'7':'62260','9':'62260'}"}]: dispatch 2024-08-19T14:15:56.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:55 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:15:56.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:55 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:15:56.312 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:15:56.336 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:56.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:56.337 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:15:56.439 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:15:56.460 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:56.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:56.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:15:56.563 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:15:56.564 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:15:56.564 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:15:56.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:56.587 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:56.587 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:15:56.688 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:15:56.710 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:56.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:56.711 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:15:56.811 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:15:56.833 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:56.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:56.834 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:15:56.935 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:15:56.955 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is unset 2024-08-19T14:15:56.958 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:56.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:56.959 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:15:57.060 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:15:57.081 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:57.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:57.082 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:15:57.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:56 smithi031 ceph-mon[16336]: pgmap v470: 27 pgs: 27 active+clean; 2.7 GiB data, 9.1 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 819 op/s 2024-08-19T14:15:57.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:56 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'7':'62260','9':'62260'}"}]': finished 2024-08-19T14:15:57.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:56 smithi031 ceph-mon[16336]: osdmap e190: 12 total, 12 up, 10 in 2024-08-19T14:15:57.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:56 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2434066151' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:15:57.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:56 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:15:57.183 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:15:57.194 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-08-19T14:15:57.203 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:57.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:57.204 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:15:57.246 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:56 smithi045 ceph-mon[20143]: pgmap v470: 27 pgs: 27 active+clean; 2.7 GiB data, 9.1 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 819 op/s 2024-08-19T14:15:57.246 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:56 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'7':'62260','9':'62260'}"}]': finished 2024-08-19T14:15:57.246 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:56 smithi045 ceph-mon[20143]: osdmap e190: 12 total, 12 up, 10 in 2024-08-19T14:15:57.246 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:56 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2434066151' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:15:57.246 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:56 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:15:57.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:56 smithi032 ceph-mon[20004]: pgmap v470: 27 pgs: 27 active+clean; 2.7 GiB data, 9.1 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 819 op/s 2024-08-19T14:15:57.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:56 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'7':'62260','9':'62260'}"}]': finished 2024-08-19T14:15:57.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:56 smithi032 ceph-mon[20004]: osdmap e190: 12 total, 12 up, 10 in 2024-08-19T14:15:57.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:56 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2434066151' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:15:57.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:56 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:15:57.305 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:15:57.327 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:57.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:57.328 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:15:57.429 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:15:57.449 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:57.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:57.450 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:15:57.551 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:15:57.564 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:15:57.573 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:57.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:57.574 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:15:57.647 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:15:57.674 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:15:57.697 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:57.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:57.698 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:15:57.799 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:15:57.822 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:57.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:57.823 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:15:57.924 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:15:57.946 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:57.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:57.947 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:15:58.048 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:15:58.071 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:58.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:58.072 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:15:58.146 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:15:58.146 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:15:58.146 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:57 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:15:58.146 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:57 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:15:58.147 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:57 smithi031 ceph-mon[16336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:15:58.147 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:57 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:15:58.147 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:57 smithi031 ceph-mon[16336]: osdmap e191: 12 total, 12 up, 10 in 2024-08-19T14:15:58.147 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:57 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:15:58.147 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:57 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:15:58.147 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:57 smithi031 ceph-mon[16336]: 2.6 scrub starts 2024-08-19T14:15:58.147 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:57 smithi031 ceph-mon[16336]: 2.6 scrub ok 2024-08-19T14:15:58.173 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:15:58.193 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:58.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:58.194 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:15:58.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:57 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:15:58.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:57 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:15:58.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:57 smithi032 ceph-mon[20004]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:15:58.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:57 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:15:58.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:57 smithi032 ceph-mon[20004]: osdmap e191: 12 total, 12 up, 10 in 2024-08-19T14:15:58.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:57 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:15:58.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:57 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:15:58.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:57 smithi032 ceph-mon[20004]: 2.6 scrub starts 2024-08-19T14:15:58.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:57 smithi032 ceph-mon[20004]: 2.6 scrub ok 2024-08-19T14:15:58.295 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:15:58.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:57 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:15:58.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:57 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:15:58.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:57 smithi045 ceph-mon[20143]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:15:58.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:57 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:15:58.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:57 smithi045 ceph-mon[20143]: osdmap e191: 12 total, 12 up, 10 in 2024-08-19T14:15:58.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:57 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:15:58.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:57 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:15:58.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:57 smithi045 ceph-mon[20143]: 2.6 scrub starts 2024-08-19T14:15:58.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:57 smithi045 ceph-mon[20143]: 2.6 scrub ok 2024-08-19T14:15:58.316 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:58.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:58.317 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:15:58.349 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:15:58.417 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:15:58.439 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:58.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:58.440 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:15:58.542 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:15:58.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:58.585 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:58.586 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:15:58.687 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:15:58.713 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:58.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:58.714 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:15:58.768 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-08-19T14:15:58.815 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:15:58.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:58.858 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:58.858 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:15:58.959 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:15:58.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:58.991 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:58.991 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:15:59.070 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:58 smithi031 ceph-mon[16336]: pgmap v473: 27 pgs: 1 peering, 26 active+clean; 2.6 GiB data, 8.9 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 787 op/s 2024-08-19T14:15:59.070 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:58 smithi031 ceph-mon[16336]: 2.e starting backfill to osd.11 from (0'0,0'0] MAX to 189'12791 2024-08-19T14:15:59.071 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:58 smithi031 ceph-mon[16336]: osdmap e192: 12 total, 12 up, 10 in 2024-08-19T14:15:59.094 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:15:59.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:59.154 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:15:59.155 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:15:59.256 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:15:59.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:58 smithi032 ceph-mon[20004]: pgmap v473: 27 pgs: 1 peering, 26 active+clean; 2.6 GiB data, 8.9 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 787 op/s 2024-08-19T14:15:59.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:58 smithi032 ceph-mon[20004]: 2.e starting backfill to osd.11 from (0'0,0'0] MAX to 189'12791 2024-08-19T14:15:59.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:58 smithi032 ceph-mon[20004]: osdmap e192: 12 total, 12 up, 10 in 2024-08-19T14:15:59.278 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:59.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:59.280 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:15:59.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:58 smithi045 ceph-mon[20143]: pgmap v473: 27 pgs: 1 peering, 26 active+clean; 2.6 GiB data, 8.9 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 787 op/s 2024-08-19T14:15:59.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:58 smithi045 ceph-mon[20143]: 2.e starting backfill to osd.11 from (0'0,0'0] MAX to 189'12791 2024-08-19T14:15:59.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:58 smithi045 ceph-mon[20143]: osdmap e192: 12 total, 12 up, 10 in 2024-08-19T14:15:59.381 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:15:59.405 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:59.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:59.406 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:15:59.418 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:15:59.441 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:15:59.442 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:15:59.481 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:15:59.481 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:15:59.507 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:15:59.522 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:15:59.522 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:15:59.534 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:59.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:59.535 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:15:59.555 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:15:59.556 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:15:59.591 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:15:59.591 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:15:59.623 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-19T14:15:59.627 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:15:59.627 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:15:59.636 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:15:59.660 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:15:59.661 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:15:59.664 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:59.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:59.665 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:15:59.699 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:15:59.699 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:15:59.733 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:15:59.733 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:15:59.765 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:15:59.772 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:15:59.772 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:15:59.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:59.791 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:15:59.791 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:15:59.827 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:15:59.828 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:15:59.870 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:15:59.871 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:15:59.892 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:15:59.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:15:59.919 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:15:59.919 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:16:00.020 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:16:00.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:00.068 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:00.068 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:16:00.169 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:16:00.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:00.204 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:00.204 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:16:00.222 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:16:00.223 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-08-19T14:16:00.223 INFO:teuthology.orchestra.run.smithi031.stdout: "complaint_time": 30, 2024-08-19T14:16:00.223 INFO:teuthology.orchestra.run.smithi031.stdout: "num_blocked_ops": 0 2024-08-19T14:16:00.223 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:16:00.223 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:59 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:16:00.223 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:59 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:16:00.223 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:59 smithi031 ceph-mon[16336]: osdmap e193: 12 total, 12 up, 10 in 2024-08-19T14:16:00.223 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:59 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1523405812' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:16:00.223 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:15:59 smithi031 ceph-mon[16336]: pgmap v476: 27 pgs: 1 peering, 26 active+clean; 2.5 GiB data, 8.5 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 745 op/s 2024-08-19T14:16:00.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:59 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:16:00.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:59 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:16:00.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:59 smithi032 ceph-mon[20004]: osdmap e193: 12 total, 12 up, 10 in 2024-08-19T14:16:00.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:59 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1523405812' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:16:00.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:15:59 smithi032 ceph-mon[20004]: pgmap v476: 27 pgs: 1 peering, 26 active+clean; 2.5 GiB data, 8.5 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 745 op/s 2024-08-19T14:16:00.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:59 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:16:00.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:59 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:16:00.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:59 smithi045 ceph-mon[20143]: osdmap e193: 12 total, 12 up, 10 in 2024-08-19T14:16:00.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:59 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1523405812' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:16:00.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:15:59 smithi045 ceph-mon[20143]: pgmap v476: 27 pgs: 1 peering, 26 active+clean; 2.5 GiB data, 8.5 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 745 op/s 2024-08-19T14:16:00.305 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:16:00.336 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:00.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:00.337 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:16:00.438 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:16:00.473 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:00.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:00.474 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:16:00.575 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:16:00.599 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:00.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:00.600 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:16:00.701 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:16:00.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:00.740 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:00.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:16:00.841 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:16:00.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:00.874 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:00.874 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:16:00.914 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-08-19T14:16:00.975 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:16:00.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:00.999 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:00.999 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:16:01.100 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:16:01.127 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:01.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:01.128 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:16:01.229 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:16:01.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:00 smithi032 ceph-mon[20004]: 1.0 scrub starts 2024-08-19T14:16:01.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:00 smithi032 ceph-mon[20004]: 1.0 scrub ok 2024-08-19T14:16:01.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:00 smithi032 ceph-mon[20004]: osdmap e194: 12 total, 12 up, 10 in 2024-08-19T14:16:01.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:00 smithi032 ceph-mon[20004]: 1.2 scrub starts 2024-08-19T14:16:01.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:00 smithi032 ceph-mon[20004]: 1.2 scrub ok 2024-08-19T14:16:01.284 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:00 smithi031 ceph-mon[16336]: 1.0 scrub starts 2024-08-19T14:16:01.284 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:00 smithi031 ceph-mon[16336]: 1.0 scrub ok 2024-08-19T14:16:01.284 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:00 smithi031 ceph-mon[16336]: osdmap e194: 12 total, 12 up, 10 in 2024-08-19T14:16:01.284 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:00 smithi031 ceph-mon[16336]: 1.2 scrub starts 2024-08-19T14:16:01.284 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:00 smithi031 ceph-mon[16336]: 1.2 scrub ok 2024-08-19T14:16:01.290 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:01.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:01.291 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:16:01.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:00 smithi045 ceph-mon[20143]: 1.0 scrub starts 2024-08-19T14:16:01.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:00 smithi045 ceph-mon[20143]: 1.0 scrub ok 2024-08-19T14:16:01.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:00 smithi045 ceph-mon[20143]: osdmap e194: 12 total, 12 up, 10 in 2024-08-19T14:16:01.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:00 smithi045 ceph-mon[20143]: 1.2 scrub starts 2024-08-19T14:16:01.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:00 smithi045 ceph-mon[20143]: 1.2 scrub ok 2024-08-19T14:16:01.392 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:16:01.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:01.434 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:01.434 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:16:01.535 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:16:01.563 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:01.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:01.564 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:16:01.665 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:16:01.669 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:16:01.705 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:01.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:01.706 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:16:01.744 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:16:01.807 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:16:01.814 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 8, 9, 11] out_osds: [10, 6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-08-19T14:16:01.814 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-19T14:16:01.814 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-08-19T14:16:01.814 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd dump -f json-pretty 2024-08-19T14:16:01.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:01.856 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:01.856 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:16:01.957 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:16:01.984 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:01.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:01.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:16:02.086 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:16:02.109 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:01 smithi031 ceph-mon[16336]: pgmap v478: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 2.4 GiB data, 8.4 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 559 op/s; 2032/118461 objects misplaced (1.715%); 4.6 MiB/s, 74 objects/s recovering 2024-08-19T14:16:02.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:02.115 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:02.115 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:16:02.216 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:16:02.242 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:02.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:02.245 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:16:02.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:01 smithi032 ceph-mon[20004]: pgmap v478: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 2.4 GiB data, 8.4 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 559 op/s; 2032/118461 objects misplaced (1.715%); 4.6 MiB/s, 74 objects/s recovering 2024-08-19T14:16:02.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:01 smithi045 ceph-mon[20143]: pgmap v478: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 2.4 GiB data, 8.4 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 559 op/s; 2032/118461 objects misplaced (1.715%); 4.6 MiB/s, 74 objects/s recovering 2024-08-19T14:16:02.346 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:16:02.386 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:02.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:02.387 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:16:02.488 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:16:02.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:02.551 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:02.552 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:16:02.653 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:16:02.677 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:02.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:02.678 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:16:02.757 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:16:02.757 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 20, 2024-08-19T14:16:02.757 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 600, 2024-08-19T14:16:02.757 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [ 2024-08-19T14:16:02.757 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.757 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.24580.0:112923 2.5 e147/102 2:a2655464:::benchmark_data_smithi045_46415_object112922:head v 147'7057, mlcod=147'7055)", 2024-08-19T14:16:02.757 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:14:07.457751+0000", 2024-08-19T14:16:02.757 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 115.296432013, 2024-08-19T14:16:02.757 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.101632213, 2024-08-19T14:16:02.758 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:16:02.758 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:16:02.758 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:16:02.758 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:16:02.758 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.758 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:16:02.758 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:07.457751+0000", 2024-08-19T14:16:02.758 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:02.758 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.758 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.758 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:16:02.759 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:07.457751+0000", 2024-08-19T14:16:02.759 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:02.759 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.759 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.759 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:16:02.759 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:07.457751+0000", 2024-08-19T14:16:02.759 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.2500000000000004e-07 2024-08-19T14:16:02.759 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.760 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.760 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:16:02.760 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:07.457901+0000", 2024-08-19T14:16:02.760 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00014971900000000001 2024-08-19T14:16:02.760 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.760 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.760 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:16:02.760 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:07.457912+0000", 2024-08-19T14:16:02.760 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0468999999999999e-05 2024-08-19T14:16:02.760 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.760 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.761 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:16:02.761 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:07.457918+0000", 2024-08-19T14:16:02.761 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.6649999999999997e-06 2024-08-19T14:16:02.761 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.761 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.761 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:16:02.761 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:07.457966+0000", 2024-08-19T14:16:02.761 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.8050000000000002e-05 2024-08-19T14:16:02.761 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.761 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.761 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:16:02.761 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:07.457983+0000", 2024-08-19T14:16:02.761 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6501e-05 2024-08-19T14:16:02.762 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.762 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.762 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:16:02.762 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:07.461251+0000", 2024-08-19T14:16:02.762 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0032679089999999998 2024-08-19T14:16:02.762 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.762 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.762 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:16:02.762 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:07.559383+0000", 2024-08-19T14:16:02.762 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.098131974999999996 2024-08-19T14:16:02.762 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:02.762 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:02.763 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:02.763 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.763 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.763 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.24580.0:112924 2.c e147/102 2:3df5b052:::benchmark_data_smithi045_46415_object112923:head v 147'7048, mlcod=147'7044)", 2024-08-19T14:16:02.763 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:14:07.460292+0000", 2024-08-19T14:16:02.763 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 115.29389007100001, 2024-08-19T14:16:02.763 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.099141834999999998, 2024-08-19T14:16:02.763 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:16:02.763 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:16:02.763 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:16:02.763 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:16:02.763 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.763 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:16:02.764 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:07.460292+0000", 2024-08-19T14:16:02.764 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:02.764 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.764 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.764 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:16:02.764 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:07.460292+0000", 2024-08-19T14:16:02.764 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:02.764 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.764 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.764 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:16:02.764 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:07.460293+0000", 2024-08-19T14:16:02.764 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.71e-07 2024-08-19T14:16:02.765 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.765 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.765 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:16:02.765 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:07.460362+0000", 2024-08-19T14:16:02.765 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.8928000000000001e-05 2024-08-19T14:16:02.765 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.765 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.765 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:16:02.765 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:07.460371+0000", 2024-08-19T14:16:02.765 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.7190000000000007e-06 2024-08-19T14:16:02.765 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.765 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.765 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:16:02.766 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:07.460376+0000", 2024-08-19T14:16:02.766 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.186e-06 2024-08-19T14:16:02.766 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.766 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.766 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:16:02.766 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:07.460423+0000", 2024-08-19T14:16:02.766 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.6913999999999998e-05 2024-08-19T14:16:02.766 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.766 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.766 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:16:02.766 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:07.460441+0000", 2024-08-19T14:16:02.766 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.8130000000000001e-05 2024-08-19T14:16:02.766 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.767 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.767 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:16:02.767 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:07.559401+0000", 2024-08-19T14:16:02.767 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.098959846000000004 2024-08-19T14:16:02.767 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.767 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.767 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:16:02.767 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:07.559434+0000", 2024-08-19T14:16:02.767 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.3241000000000002e-05 2024-08-19T14:16:02.767 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:02.767 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:02.767 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:02.767 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.768 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.768 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.24580.0:112925 2.5 e147/102 2:a57b6a75:::benchmark_data_smithi045_46415_object112924:head v 147'7058, mlcod=147'7055)", 2024-08-19T14:16:02.768 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:14:07.461210+0000", 2024-08-19T14:16:02.768 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 115.292972701, 2024-08-19T14:16:02.768 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.11225392300000001, 2024-08-19T14:16:02.768 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:16:02.768 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:16:02.768 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:16:02.768 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:16:02.768 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.768 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:16:02.768 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:07.461210+0000", 2024-08-19T14:16:02.768 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:02.768 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.769 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.769 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:16:02.769 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:07.461210+0000", 2024-08-19T14:16:02.769 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:02.769 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.769 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.769 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:16:02.769 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:07.461212+0000", 2024-08-19T14:16:02.769 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.3479999999999998e-06 2024-08-19T14:16:02.769 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.769 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.769 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:16:02.769 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:07.461275+0000", 2024-08-19T14:16:02.769 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.2607999999999994e-05 2024-08-19T14:16:02.769 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.770 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.770 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:16:02.770 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:07.461282+0000", 2024-08-19T14:16:02.770 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.5299999999999999e-06 2024-08-19T14:16:02.770 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.770 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.770 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:16:02.770 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:07.559119+0000", 2024-08-19T14:16:02.770 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.097836440999999996 2024-08-19T14:16:02.770 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.770 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.770 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:16:02.770 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:07.560139+0000", 2024-08-19T14:16:02.771 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0010199370000000001 2024-08-19T14:16:02.771 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.771 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.771 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:16:02.771 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:07.560159+0000", 2024-08-19T14:16:02.771 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.0302e-05 2024-08-19T14:16:02.771 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.771 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.771 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:16:02.771 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:07.573339+0000", 2024-08-19T14:16:02.771 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.013180434 2024-08-19T14:16:02.771 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.771 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.771 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:16:02.772 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:07.573464+0000", 2024-08-19T14:16:02.772 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000124323 2024-08-19T14:16:02.772 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:02.772 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:02.772 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:02.772 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.772 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.772 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "replica_scrub(pg: 2.8,from:0'0,to:147'7034,epoch:147/102,start:2:10bd9c76:::benchmark_data_smithi045_46415_object70758:0,end:2:10f379ad:::benchmark_data_smithi045_46415_object96090:0,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-08-19T14:16:02.773 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:14:07.468692+0000", 2024-08-19T14:16:02.773 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 115.28549084300001, 2024-08-19T14:16:02.773 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.090995744000000003, 2024-08-19T14:16:02.773 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:16:02.773 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:16:02.773 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-08-19T14:16:02.773 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:16:02.773 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.773 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:16:02.774 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:07.468692+0000", 2024-08-19T14:16:02.774 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:02.774 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.774 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.774 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:16:02.774 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:07.468692+0000", 2024-08-19T14:16:02.774 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:02.774 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.774 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.774 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:16:02.774 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:07.468701+0000", 2024-08-19T14:16:02.774 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.8729999999999996e-06 2024-08-19T14:16:02.774 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.774 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.775 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:16:02.775 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:07.468717+0000", 2024-08-19T14:16:02.775 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6690000000000001e-05 2024-08-19T14:16:02.775 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.775 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.775 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:16:02.775 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:07.559503+0000", 2024-08-19T14:16:02.775 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.090786121999999997 2024-08-19T14:16:02.775 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.775 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.775 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:16:02.775 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:07.559519+0000", 2024-08-19T14:16:02.775 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5801e-05 2024-08-19T14:16:02.776 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.776 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.776 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:16:02.776 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:07.559635+0000", 2024-08-19T14:16:02.776 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00011534 2024-08-19T14:16:02.776 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.776 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.776 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:16:02.776 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:07.559650+0000", 2024-08-19T14:16:02.776 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5637e-05 2024-08-19T14:16:02.776 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.776 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.776 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:16:02.776 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:07.559687+0000", 2024-08-19T14:16:02.777 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7280999999999998e-05 2024-08-19T14:16:02.777 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:02.777 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:02.777 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:02.777 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.777 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.777 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.24580.0:144572 2.c e160/102 2:3d1721cd:::benchmark_data_smithi045_46415_object21291:head v 160'8996, mlcod=159'8993)", 2024-08-19T14:16:02.777 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:14:42.950869+0000", 2024-08-19T14:16:02.777 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 79.803313084999999, 2024-08-19T14:16:02.777 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.046664573000000001, 2024-08-19T14:16:02.777 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:16:02.777 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:16:02.777 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:16:02.777 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:16:02.778 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.778 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:16:02.778 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:42.950869+0000", 2024-08-19T14:16:02.778 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:02.778 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.778 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.778 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:16:02.778 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:42.950869+0000", 2024-08-19T14:16:02.778 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:02.778 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.779 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.779 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:16:02.779 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:42.950871+0000", 2024-08-19T14:16:02.779 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.381e-06 2024-08-19T14:16:02.779 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.779 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.779 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:16:02.779 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:42.950881+0000", 2024-08-19T14:16:02.779 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0179e-05 2024-08-19T14:16:02.779 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.779 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.779 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:16:02.779 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:42.950902+0000", 2024-08-19T14:16:02.780 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.0766000000000001e-05 2024-08-19T14:16:02.780 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.780 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.780 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:16:02.780 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:42.950911+0000", 2024-08-19T14:16:02.780 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.0759999999999992e-06 2024-08-19T14:16:02.780 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.780 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.780 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:16:02.780 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:42.950994+0000", 2024-08-19T14:16:02.780 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.3483e-05 2024-08-19T14:16:02.780 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.780 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.780 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:16:02.781 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:42.951019+0000", 2024-08-19T14:16:02.781 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.4524000000000001e-05 2024-08-19T14:16:02.781 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.781 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.781 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:16:02.781 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:42.997345+0000", 2024-08-19T14:16:02.781 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.046326298000000002 2024-08-19T14:16:02.781 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.781 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.781 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:16:02.781 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:42.997534+0000", 2024-08-19T14:16:02.781 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00018886599999999999 2024-08-19T14:16:02.781 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:02.782 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:02.782 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:02.782 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.782 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.782 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.24580.0:144573 2.4 e160/146 2:2ced386c:::benchmark_data_smithi045_46415_object21292:head v 160'9072, mlcod=159'9070)", 2024-08-19T14:16:02.782 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:14:42.966356+0000", 2024-08-19T14:16:02.782 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 79.787826245000005, 2024-08-19T14:16:02.782 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.062465894000000001, 2024-08-19T14:16:02.782 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:16:02.782 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:16:02.782 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:16:02.782 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:16:02.782 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.782 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:16:02.783 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:42.966356+0000", 2024-08-19T14:16:02.783 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:02.783 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.783 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.783 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:16:02.783 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:42.966356+0000", 2024-08-19T14:16:02.783 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:02.783 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.783 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.783 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:16:02.783 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:42.966357+0000", 2024-08-19T14:16:02.783 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.8799999999999998e-07 2024-08-19T14:16:02.783 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.784 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.784 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:16:02.784 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:42.966364+0000", 2024-08-19T14:16:02.784 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.9120000000000001e-06 2024-08-19T14:16:02.784 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.784 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.784 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:16:02.784 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:42.966380+0000", 2024-08-19T14:16:02.784 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6688e-05 2024-08-19T14:16:02.784 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.784 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.784 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:16:02.784 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:42.966391+0000", 2024-08-19T14:16:02.784 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1246e-05 2024-08-19T14:16:02.784 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.785 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.785 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:16:02.785 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:42.969846+0000", 2024-08-19T14:16:02.785 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0034542399999999999 2024-08-19T14:16:02.785 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.785 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.785 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:16:02.785 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:42.969902+0000", 2024-08-19T14:16:02.785 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.6289000000000003e-05 2024-08-19T14:16:02.785 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.785 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.785 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:16:02.785 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:43.028616+0000", 2024-08-19T14:16:02.786 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.058713744999999998 2024-08-19T14:16:02.786 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.786 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.786 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:16:02.786 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:43.028822+0000", 2024-08-19T14:16:02.786 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000206486 2024-08-19T14:16:02.786 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:02.786 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:02.786 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:02.786 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.786 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.786 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.24580.0:144585 2.c e160/102 2:39d20804:::benchmark_data_smithi045_46415_object21304:head v 160'8997, mlcod=159'8994)", 2024-08-19T14:16:02.786 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:14:42.966414+0000", 2024-08-19T14:16:02.787 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 79.787768932000006, 2024-08-19T14:16:02.787 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.062656079000000003, 2024-08-19T14:16:02.787 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:16:02.787 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:16:02.787 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:16:02.787 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:16:02.787 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.787 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:16:02.787 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:42.966414+0000", 2024-08-19T14:16:02.787 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:02.787 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.787 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.787 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:16:02.787 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:42.966414+0000", 2024-08-19T14:16:02.788 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:02.788 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.788 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.788 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:16:02.788 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:42.966414+0000", 2024-08-19T14:16:02.788 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.0299999999999999e-07 2024-08-19T14:16:02.788 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.788 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.788 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:16:02.788 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:42.966422+0000", 2024-08-19T14:16:02.788 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.6820000000000003e-06 2024-08-19T14:16:02.788 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.788 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.788 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:16:02.789 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:42.966431+0000", 2024-08-19T14:16:02.789 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.1479999999999995e-06 2024-08-19T14:16:02.789 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.789 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.789 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:16:02.789 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:42.966438+0000", 2024-08-19T14:16:02.789 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.7889999999999997e-06 2024-08-19T14:16:02.789 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.789 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.789 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:16:02.789 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:42.989199+0000", 2024-08-19T14:16:02.789 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.022760944000000002 2024-08-19T14:16:02.789 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.789 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.790 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:16:02.790 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:42.989823+0000", 2024-08-19T14:16:02.790 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00062471 2024-08-19T14:16:02.790 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.790 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.790 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:16:02.790 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:43.028828+0000", 2024-08-19T14:16:02.790 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.039004124000000001 2024-08-19T14:16:02.790 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.790 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.790 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:16:02.790 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:43.029070+0000", 2024-08-19T14:16:02.790 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000242079 2024-08-19T14:16:02.791 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:02.791 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:02.791 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:02.791 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.791 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.791 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.24580.0:144588 2.5 e160/102 2:a0cc2746:::benchmark_data_smithi045_46415_object21307:head v 160'9063, mlcod=159'9060)", 2024-08-19T14:16:02.791 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:14:42.970049+0000", 2024-08-19T14:16:02.791 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 79.784133271000002, 2024-08-19T14:16:02.791 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.063447428, 2024-08-19T14:16:02.791 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:16:02.791 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:16:02.791 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:16:02.791 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:16:02.791 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.792 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:16:02.792 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:42.970049+0000", 2024-08-19T14:16:02.792 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:02.792 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.792 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.792 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:16:02.792 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:42.970049+0000", 2024-08-19T14:16:02.792 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:02.792 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.792 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.792 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:16:02.792 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:42.970050+0000", 2024-08-19T14:16:02.792 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.5700000000000002e-07 2024-08-19T14:16:02.792 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.793 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.793 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:16:02.793 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:42.970055+0000", 2024-08-19T14:16:02.793 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1869999999999996e-06 2024-08-19T14:16:02.793 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.793 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.793 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:16:02.793 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:42.970063+0000", 2024-08-19T14:16:02.793 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.5320000000000001e-06 2024-08-19T14:16:02.793 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.793 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.793 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:16:02.793 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:42.970069+0000", 2024-08-19T14:16:02.794 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.9599999999999997e-06 2024-08-19T14:16:02.794 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.794 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.794 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:16:02.794 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:43.010959+0000", 2024-08-19T14:16:02.794 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.040890735999999997 2024-08-19T14:16:02.794 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.794 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.794 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:16:02.794 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:43.011021+0000", 2024-08-19T14:16:02.794 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.1989000000000006e-05 2024-08-19T14:16:02.794 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.794 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.795 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:16:02.795 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:43.033336+0000", 2024-08-19T14:16:02.795 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.022315108 2024-08-19T14:16:02.795 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.795 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.795 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:16:02.795 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:43.033497+0000", 2024-08-19T14:16:02.795 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000160359 2024-08-19T14:16:02.795 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:02.795 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:02.795 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:02.795 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.795 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.796 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.24580.0:144576 2.4 e160/146 2:2f164cce:::benchmark_data_smithi045_46415_object21295:head v 160'9073, mlcod=159'9070)", 2024-08-19T14:16:02.796 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:14:42.974064+0000", 2024-08-19T14:16:02.796 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 79.780118467999998, 2024-08-19T14:16:02.796 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.059714088999999998, 2024-08-19T14:16:02.796 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:16:02.796 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:16:02.796 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:16:02.796 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:16:02.796 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.796 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:16:02.796 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:42.974064+0000", 2024-08-19T14:16:02.796 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:02.796 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.797 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.797 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:16:02.797 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:42.974064+0000", 2024-08-19T14:16:02.797 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:02.797 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.797 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.797 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:16:02.797 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:42.974066+0000", 2024-08-19T14:16:02.797 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.55e-06 2024-08-19T14:16:02.797 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.797 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.797 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:16:02.797 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:42.974074+0000", 2024-08-19T14:16:02.797 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.7409999999999998e-06 2024-08-19T14:16:02.798 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.798 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.798 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:16:02.798 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:42.974093+0000", 2024-08-19T14:16:02.798 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.8334000000000001e-05 2024-08-19T14:16:02.798 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.798 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.798 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:16:02.798 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:42.974102+0000", 2024-08-19T14:16:02.798 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.9889999999999998e-06 2024-08-19T14:16:02.798 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.798 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.798 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:16:02.798 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:42.990045+0000", 2024-08-19T14:16:02.799 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.015943138999999999 2024-08-19T14:16:02.799 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.799 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.799 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:16:02.799 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:42.990125+0000", 2024-08-19T14:16:02.799 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.0191000000000005e-05 2024-08-19T14:16:02.799 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.799 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.799 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:16:02.799 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:43.033553+0000", 2024-08-19T14:16:02.799 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.043428372 2024-08-19T14:16:02.799 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.799 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.799 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:16:02.800 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:43.033778+0000", 2024-08-19T14:16:02.800 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000224773 2024-08-19T14:16:02.800 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:02.800 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:02.800 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:02.800 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.800 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.800 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.24580.0:144577 2.4 e160/146 2:28d12d0f:::benchmark_data_smithi045_46415_object21296:head v 160'9074, mlcod=159'9070)", 2024-08-19T14:16:02.800 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:14:42.986496+0000", 2024-08-19T14:16:02.800 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 79.767686210999997, 2024-08-19T14:16:02.800 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.047900725999999998, 2024-08-19T14:16:02.800 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:16:02.800 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:16:02.800 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:16:02.801 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:16:02.801 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.801 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:16:02.801 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:42.986496+0000", 2024-08-19T14:16:02.801 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:02.801 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.801 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.801 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:16:02.801 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:42.986496+0000", 2024-08-19T14:16:02.801 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:02.801 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.801 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.801 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:16:02.801 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:42.986498+0000", 2024-08-19T14:16:02.802 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.821e-06 2024-08-19T14:16:02.802 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.802 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.802 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:16:02.802 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:16:02.803 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:42.986509+0000", 2024-08-19T14:16:02.803 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0645999999999999e-05 2024-08-19T14:16:02.804 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.804 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.804 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:16:02.804 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:42.986529+0000", 2024-08-19T14:16:02.804 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.0489999999999999e-05 2024-08-19T14:16:02.804 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.804 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.804 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:16:02.804 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:42.986539+0000", 2024-08-19T14:16:02.804 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.431e-06 2024-08-19T14:16:02.804 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.804 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.804 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:16:02.805 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:43.006068+0000", 2024-08-19T14:16:02.805 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.019529617999999999 2024-08-19T14:16:02.805 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.805 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.805 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:16:02.805 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:43.006089+0000", 2024-08-19T14:16:02.805 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.1080000000000001e-05 2024-08-19T14:16:02.805 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.805 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.805 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:16:02.805 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:43.033782+0000", 2024-08-19T14:16:02.805 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.027692432999999999 2024-08-19T14:16:02.805 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.806 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.806 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:16:02.806 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:43.034397+0000", 2024-08-19T14:16:02.806 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00061520699999999999 2024-08-19T14:16:02.806 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:02.806 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:02.806 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:02.806 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.806 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.806 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.24580.0:154732 2.4 e166/146 2:2eb608ea:::benchmark_data_smithi045_46415_object31451:head v 166'9720, mlcod=166'9717)", 2024-08-19T14:16:02.806 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:14:54.636956+0000", 2024-08-19T14:16:02.806 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 68.117226673000005, 2024-08-19T14:16:02.806 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.046964438999999997, 2024-08-19T14:16:02.806 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:16:02.807 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:16:02.807 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:16:02.807 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:16:02.807 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.807 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:16:02.807 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:54.636956+0000", 2024-08-19T14:16:02.807 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:02.807 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.807 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.807 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:16:02.807 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:54.636956+0000", 2024-08-19T14:16:02.807 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:02.807 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.807 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.808 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:16:02.808 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:54.636957+0000", 2024-08-19T14:16:02.808 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.35e-06 2024-08-19T14:16:02.808 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.808 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.808 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:16:02.808 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:54.636965+0000", 2024-08-19T14:16:02.808 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.8830000000000006e-06 2024-08-19T14:16:02.808 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.808 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.808 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:16:02.808 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:54.636978+0000", 2024-08-19T14:16:02.808 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2954e-05 2024-08-19T14:16:02.808 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.809 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.809 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:16:02.809 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:54.636986+0000", 2024-08-19T14:16:02.809 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.0539999999999995e-06 2024-08-19T14:16:02.809 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.809 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.809 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:16:02.809 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:54.657918+0000", 2024-08-19T14:16:02.809 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.020932261000000001 2024-08-19T14:16:02.809 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.809 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.809 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:16:02.809 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:54.657943+0000", 2024-08-19T14:16:02.809 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.4199e-05 2024-08-19T14:16:02.810 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.810 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.810 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:16:02.810 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:54.683799+0000", 2024-08-19T14:16:02.810 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.025856165 2024-08-19T14:16:02.810 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.810 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.810 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:16:02.810 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:54.683920+0000", 2024-08-19T14:16:02.810 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000121573 2024-08-19T14:16:02.810 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:02.810 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:02.810 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:02.811 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.811 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.811 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.24580.0:171479 2.4 e169/146 2:227aa082:::benchmark_data_smithi045_46415_object48198:head v 169'10776, mlcod=169'10772)", 2024-08-19T14:16:02.811 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:15:12.874075+0000", 2024-08-19T14:16:02.811 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 49.880107559999999, 2024-08-19T14:16:02.811 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.043841465000000003, 2024-08-19T14:16:02.811 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:16:02.811 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:16:02.811 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:16:02.811 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:16:02.811 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.811 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:16:02.812 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:15:12.874075+0000", 2024-08-19T14:16:02.812 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:02.812 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.812 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.812 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:16:02.812 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:15:12.874075+0000", 2024-08-19T14:16:02.812 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:02.812 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.812 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.812 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:16:02.812 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:15:12.874076+0000", 2024-08-19T14:16:02.812 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0100000000000001e-06 2024-08-19T14:16:02.812 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.813 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.813 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:16:02.813 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:15:12.874084+0000", 2024-08-19T14:16:02.813 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.7109999999999995e-06 2024-08-19T14:16:02.813 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.813 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.813 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:16:02.813 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:15:12.874099+0000", 2024-08-19T14:16:02.813 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.558e-05 2024-08-19T14:16:02.813 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.813 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.813 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:16:02.813 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:15:12.874108+0000", 2024-08-19T14:16:02.813 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.0289999999999994e-06 2024-08-19T14:16:02.814 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.814 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.814 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:16:02.814 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:15:12.880676+0000", 2024-08-19T14:16:02.814 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0065673809999999997 2024-08-19T14:16:02.814 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.814 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.814 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:16:02.814 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:15:12.880718+0000", 2024-08-19T14:16:02.814 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.1975999999999999e-05 2024-08-19T14:16:02.814 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.814 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.815 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:16:02.815 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:15:12.917821+0000", 2024-08-19T14:16:02.815 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.037103569000000003 2024-08-19T14:16:02.815 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.815 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.815 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:16:02.815 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:15:12.917916+0000", 2024-08-19T14:16:02.815 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.5209000000000003e-05 2024-08-19T14:16:02.815 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:02.815 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:02.815 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:02.815 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.816 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.816 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.24580.0:172752 2.8 e170/102 2:15fb1eba:::benchmark_data_smithi045_46415_object49471:head v 170'10858, mlcod=170'10851)", 2024-08-19T14:16:02.816 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:15:14.369892+0000", 2024-08-19T14:16:02.816 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 48.384290278000002, 2024-08-19T14:16:02.816 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.057376152999999999, 2024-08-19T14:16:02.816 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:16:02.816 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:16:02.816 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:16:02.816 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:16:02.816 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.816 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:16:02.816 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:15:14.369892+0000", 2024-08-19T14:16:02.817 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:02.817 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.817 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.817 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:16:02.817 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:15:14.369892+0000", 2024-08-19T14:16:02.817 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:02.817 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.817 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.817 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:16:02.817 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:15:14.369893+0000", 2024-08-19T14:16:02.817 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.8299999999999997e-07 2024-08-19T14:16:02.817 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.817 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.817 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:16:02.818 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:15:14.369898+0000", 2024-08-19T14:16:02.818 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.8980000000000004e-06 2024-08-19T14:16:02.818 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.818 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.818 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:16:02.818 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:15:14.369907+0000", 2024-08-19T14:16:02.818 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.1370000000000008e-06 2024-08-19T14:16:02.818 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.818 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.818 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:16:02.818 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:15:14.369913+0000", 2024-08-19T14:16:02.819 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.3820000000000001e-06 2024-08-19T14:16:02.819 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.819 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.819 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:16:02.819 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:15:14.381007+0000", 2024-08-19T14:16:02.819 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.011094015 2024-08-19T14:16:02.819 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.819 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.819 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:16:02.819 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:15:14.381055+0000", 2024-08-19T14:16:02.819 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7775999999999998e-05 2024-08-19T14:16:02.819 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.819 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.820 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:16:02.820 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:15:14.427191+0000", 2024-08-19T14:16:02.820 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.046135744999999999 2024-08-19T14:16:02.820 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.820 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.820 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:16:02.820 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:15:14.427268+0000", 2024-08-19T14:16:02.820 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.7616999999999999e-05 2024-08-19T14:16:02.820 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:02.820 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:02.820 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:02.820 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.820 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.820 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.24580.0:176912 2.c e176/102 2:3eae41ba:::benchmark_data_smithi045_46415_object53631:head v 176'11001, mlcod=176'10999)", 2024-08-19T14:16:02.820 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:15:21.597948+0000", 2024-08-19T14:16:02.821 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 41.156234613000002, 2024-08-19T14:16:02.821 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.054553732000000001, 2024-08-19T14:16:02.821 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:16:02.821 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:16:02.821 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:16:02.821 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:16:02.821 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.821 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:16:02.821 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:15:21.597948+0000", 2024-08-19T14:16:02.821 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:02.821 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.822 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.822 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:16:02.822 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:15:21.597948+0000", 2024-08-19T14:16:02.822 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:02.822 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.822 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.822 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:16:02.822 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:15:21.597948+0000", 2024-08-19T14:16:02.822 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.0499999999999999e-07 2024-08-19T14:16:02.822 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.822 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.822 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:16:02.823 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:15:21.597953+0000", 2024-08-19T14:16:02.823 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2309999999999998e-06 2024-08-19T14:16:02.823 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.823 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.823 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:16:02.823 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:15:21.597959+0000", 2024-08-19T14:16:02.823 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.8050000000000001e-06 2024-08-19T14:16:02.823 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.823 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.823 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:16:02.823 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:15:21.597964+0000", 2024-08-19T14:16:02.823 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.437e-06 2024-08-19T14:16:02.823 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.823 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.824 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:16:02.824 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:15:21.598142+0000", 2024-08-19T14:16:02.824 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000178157 2024-08-19T14:16:02.824 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.824 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.824 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:16:02.824 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:15:21.598189+0000", 2024-08-19T14:16:02.824 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.6689e-05 2024-08-19T14:16:02.824 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.824 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.824 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:16:02.824 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:15:21.652439+0000", 2024-08-19T14:16:02.824 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.054250376000000003 2024-08-19T14:16:02.825 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.825 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.825 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:16:02.825 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:15:21.652502+0000", 2024-08-19T14:16:02.825 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.2632000000000006e-05 2024-08-19T14:16:02.825 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:02.825 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:02.825 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:02.825 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.825 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.825 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.24580.0:187910 2.c e179/102 2:34ed00b5:::benchmark_data_smithi045_46415_object64629:head v 179'11704, mlcod=179'11702)", 2024-08-19T14:16:02.825 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:15:34.046827+0000", 2024-08-19T14:16:02.825 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 28.707355415999999, 2024-08-19T14:16:02.826 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.046353322000000002, 2024-08-19T14:16:02.826 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:16:02.826 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:16:02.826 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:16:02.826 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:16:02.826 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.826 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:16:02.826 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:15:34.046827+0000", 2024-08-19T14:16:02.826 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:02.826 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.827 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.827 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:16:02.827 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:15:34.046827+0000", 2024-08-19T14:16:02.827 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:02.827 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.827 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.827 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:16:02.827 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:15:34.046828+0000", 2024-08-19T14:16:02.827 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.8299999999999995e-07 2024-08-19T14:16:02.827 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.827 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.827 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:16:02.827 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:15:34.046836+0000", 2024-08-19T14:16:02.828 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.6190000000000002e-06 2024-08-19T14:16:02.828 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.828 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.828 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:16:02.828 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:15:34.046848+0000", 2024-08-19T14:16:02.828 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2508e-05 2024-08-19T14:16:02.828 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.828 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.828 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:16:02.828 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:15:34.046855+0000", 2024-08-19T14:16:02.828 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.8510000000000001e-06 2024-08-19T14:16:02.828 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.828 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.829 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:16:02.829 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:15:34.049977+0000", 2024-08-19T14:16:02.829 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.003122311 2024-08-19T14:16:02.829 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.829 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.829 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:16:02.829 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:15:34.050028+0000", 2024-08-19T14:16:02.829 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0847000000000003e-05 2024-08-19T14:16:02.829 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.829 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.829 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:16:02.829 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:15:34.093039+0000", 2024-08-19T14:16:02.829 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.043010925999999998 2024-08-19T14:16:02.830 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.830 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.830 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:16:02.830 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:15:34.093180+0000", 2024-08-19T14:16:02.830 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00014127700000000001 2024-08-19T14:16:02.830 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:02.830 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:02.830 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:02.830 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.830 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.830 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.24580.0:187903 2.4 e179/146 2:25cb6cc1:::benchmark_data_smithi045_46415_object64622:head v 179'11804, mlcod=179'11800)", 2024-08-19T14:16:02.830 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:15:34.046938+0000", 2024-08-19T14:16:02.830 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 28.707244678999999, 2024-08-19T14:16:02.831 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.046409312000000001, 2024-08-19T14:16:02.831 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:16:02.831 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:16:02.831 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:16:02.831 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:16:02.831 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.831 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:16:02.831 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:15:34.046938+0000", 2024-08-19T14:16:02.831 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:02.831 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.831 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.831 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:16:02.831 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:15:34.046938+0000", 2024-08-19T14:16:02.831 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:02.831 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.832 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.832 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:16:02.832 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:15:34.046938+0000", 2024-08-19T14:16:02.832 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.2500000000000001e-07 2024-08-19T14:16:02.832 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.832 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.832 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:16:02.832 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:15:34.046943+0000", 2024-08-19T14:16:02.832 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4959999999999998e-06 2024-08-19T14:16:02.832 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.832 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.832 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:16:02.832 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:15:34.046951+0000", 2024-08-19T14:16:02.833 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.9480000000000008e-06 2024-08-19T14:16:02.833 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.833 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.833 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:16:02.833 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:15:34.046956+0000", 2024-08-19T14:16:02.833 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.9350000000000002e-06 2024-08-19T14:16:02.833 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.833 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.833 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:16:02.834 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:15:34.052114+0000", 2024-08-19T14:16:02.835 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0051588529999999997 2024-08-19T14:16:02.835 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.835 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.835 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:16:02.835 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:15:34.052244+0000", 2024-08-19T14:16:02.835 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000130025 2024-08-19T14:16:02.835 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.835 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.835 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:16:02.835 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:15:34.093184+0000", 2024-08-19T14:16:02.835 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.040939215000000001 2024-08-19T14:16:02.835 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.836 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.836 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:16:02.836 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:15:34.093347+0000", 2024-08-19T14:16:02.836 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00016351500000000001 2024-08-19T14:16:02.836 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:02.836 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:02.836 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:02.836 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.836 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.836 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.24580.0:187914 2.4 e179/146 2:29f316da:::benchmark_data_smithi045_46415_object64633:head v 179'11805, mlcod=179'11801)", 2024-08-19T14:16:02.836 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:15:34.046964+0000", 2024-08-19T14:16:02.836 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 28.707218504, 2024-08-19T14:16:02.836 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.046680219000000002, 2024-08-19T14:16:02.836 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:16:02.836 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:16:02.837 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:16:02.837 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:16:02.837 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.837 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:16:02.837 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:15:34.046964+0000", 2024-08-19T14:16:02.837 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:02.837 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.837 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.837 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:16:02.837 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:15:34.046964+0000", 2024-08-19T14:16:02.837 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:02.837 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.837 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.838 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:16:02.838 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:15:34.046964+0000", 2024-08-19T14:16:02.838 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.1799999999999999e-07 2024-08-19T14:16:02.838 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.838 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.838 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:16:02.838 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:15:34.046969+0000", 2024-08-19T14:16:02.838 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2320000000000003e-06 2024-08-19T14:16:02.838 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.838 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.838 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:16:02.838 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:15:34.046974+0000", 2024-08-19T14:16:02.839 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.8969999999999996e-06 2024-08-19T14:16:02.839 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.839 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.839 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:16:02.839 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:15:34.046979+0000", 2024-08-19T14:16:02.839 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.5290000000000002e-06 2024-08-19T14:16:02.839 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.839 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.839 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:16:02.839 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:15:34.074210+0000", 2024-08-19T14:16:02.839 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.027230823000000001 2024-08-19T14:16:02.839 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.839 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.839 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:16:02.840 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:15:34.074230+0000", 2024-08-19T14:16:02.840 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.0628e-05 2024-08-19T14:16:02.840 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.840 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.840 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:16:02.840 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:15:34.093516+0000", 2024-08-19T14:16:02.840 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.019285277 2024-08-19T14:16:02.840 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.840 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.840 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:16:02.840 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:15:34.093644+0000", 2024-08-19T14:16:02.840 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000128615 2024-08-19T14:16:02.840 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:02.841 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:02.841 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:02.841 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.841 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.841 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.24580.0:200592 2.c e187/102 2:32739d16:::benchmark_data_smithi045_46415_object77311:head v 187'12495, mlcod=187'12493)", 2024-08-19T14:16:02.841 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:15:50.721940+0000", 2024-08-19T14:16:02.841 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 12.032242777, 2024-08-19T14:16:02.841 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.056288286, 2024-08-19T14:16:02.841 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:16:02.841 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:16:02.841 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:16:02.841 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:16:02.841 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.841 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:16:02.842 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:15:50.721940+0000", 2024-08-19T14:16:02.842 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:02.842 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.842 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.842 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:16:02.842 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:15:50.721940+0000", 2024-08-19T14:16:02.842 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:02.842 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.842 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.842 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:16:02.842 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:15:50.721941+0000", 2024-08-19T14:16:02.842 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.6299999999999993e-07 2024-08-19T14:16:02.842 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.843 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.843 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:16:02.843 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:15:50.721948+0000", 2024-08-19T14:16:02.843 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.1969999999999999e-06 2024-08-19T14:16:02.843 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.843 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.843 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:16:02.843 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:15:50.721963+0000", 2024-08-19T14:16:02.843 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5364999999999999e-05 2024-08-19T14:16:02.843 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.843 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.843 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:16:02.844 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:15:50.721971+0000", 2024-08-19T14:16:02.844 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.6850000000000001e-06 2024-08-19T14:16:02.844 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.844 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.844 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:16:02.844 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:15:50.733063+0000", 2024-08-19T14:16:02.844 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.011091905000000001 2024-08-19T14:16:02.844 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.844 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.844 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:16:02.844 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:15:50.733198+0000", 2024-08-19T14:16:02.844 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013551000000000001 2024-08-19T14:16:02.844 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.845 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.845 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:16:02.845 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:15:50.773151+0000", 2024-08-19T14:16:02.845 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.039952664999999998 2024-08-19T14:16:02.845 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.845 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.845 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:16:02.845 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:15:50.778228+0000", 2024-08-19T14:16:02.845 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0050769960000000003 2024-08-19T14:16:02.845 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:02.845 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:02.845 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:02.845 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.846 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.846 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.24580.0:200595 2.c e187/102 2:3085cf9f:::benchmark_data_smithi045_46415_object77314:head v 187'12496, mlcod=187'12493)", 2024-08-19T14:16:02.846 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:15:50.733477+0000", 2024-08-19T14:16:02.846 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 12.020705850000001, 2024-08-19T14:16:02.846 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.045131336000000001, 2024-08-19T14:16:02.846 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:16:02.846 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:16:02.846 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:16:02.846 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:16:02.846 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.846 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:16:02.846 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:15:50.733477+0000", 2024-08-19T14:16:02.846 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:02.846 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.847 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.847 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:16:02.847 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:15:50.733477+0000", 2024-08-19T14:16:02.847 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:02.847 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.847 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.847 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:16:02.847 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:15:50.733478+0000", 2024-08-19T14:16:02.847 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0389999999999999e-06 2024-08-19T14:16:02.847 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.847 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.847 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:16:02.847 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:15:50.733486+0000", 2024-08-19T14:16:02.847 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.7309999999999999e-06 2024-08-19T14:16:02.848 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.848 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.848 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:16:02.848 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:15:50.733502+0000", 2024-08-19T14:16:02.848 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5033e-05 2024-08-19T14:16:02.848 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.848 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.848 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:16:02.848 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:15:50.733510+0000", 2024-08-19T14:16:02.848 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.2460000000000003e-06 2024-08-19T14:16:02.848 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.848 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.848 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:16:02.848 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:15:50.738045+0000", 2024-08-19T14:16:02.849 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.004534822 2024-08-19T14:16:02.849 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.849 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.849 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:16:02.849 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:15:50.738067+0000", 2024-08-19T14:16:02.849 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.2453e-05 2024-08-19T14:16:02.849 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.849 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.849 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:16:02.849 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:15:50.778234+0000", 2024-08-19T14:16:02.849 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.040167142000000003 2024-08-19T14:16:02.849 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.849 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.849 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:16:02.850 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:15:50.778608+0000", 2024-08-19T14:16:02.850 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00037387 2024-08-19T14:16:02.850 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:02.850 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:02.850 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:02.850 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.850 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.850 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.24580.0:200601 2.4 e187/146 2:25ce11cb:::benchmark_data_smithi045_46415_object77320:head v 187'12586, mlcod=187'12584)", 2024-08-19T14:16:02.850 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:15:50.740505+0000", 2024-08-19T14:16:02.850 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 12.013677682999999, 2024-08-19T14:16:02.850 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.044414617000000003, 2024-08-19T14:16:02.850 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:16:02.850 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:16:02.851 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:16:02.851 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:16:02.851 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.851 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:16:02.851 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:15:50.740505+0000", 2024-08-19T14:16:02.851 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:02.851 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.851 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.851 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:16:02.851 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:15:50.740505+0000", 2024-08-19T14:16:02.851 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:02.851 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.851 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.851 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:16:02.851 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:15:50.740505+0000", 2024-08-19T14:16:02.852 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.1699999999999998e-07 2024-08-19T14:16:02.852 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.852 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.852 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:16:02.852 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:15:50.740511+0000", 2024-08-19T14:16:02.852 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1660000000000002e-06 2024-08-19T14:16:02.852 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.852 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.852 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:16:02.852 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:15:50.740519+0000", 2024-08-19T14:16:02.852 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.1370000000000002e-06 2024-08-19T14:16:02.852 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.852 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.853 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:16:02.853 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:15:50.740526+0000", 2024-08-19T14:16:02.853 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.0380000000000002e-06 2024-08-19T14:16:02.853 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.853 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.853 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:16:02.853 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:15:50.768176+0000", 2024-08-19T14:16:02.853 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.027650246 2024-08-19T14:16:02.853 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.853 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.853 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:16:02.853 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:15:50.768195+0000", 2024-08-19T14:16:02.853 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.8978999999999999e-05 2024-08-19T14:16:02.853 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.854 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.854 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:16:02.854 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:15:50.784674+0000", 2024-08-19T14:16:02.854 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.016479242000000002 2024-08-19T14:16:02.854 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:02.854 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:02.854 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:16:02.854 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:15:50.784919+0000", 2024-08-19T14:16:02.854 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00024519199999999998 2024-08-19T14:16:02.854 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:02.854 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:02.854 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:02.854 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:02.854 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:02.855 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:16:02.855 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:16:02.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:02.856 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:02.856 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:16:02.957 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:16:02.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:02.983 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:02.983 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:16:03.010 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is unset 2024-08-19T14:16:03.010 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:03 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/892908396' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:16:03.083 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:16:03.117 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:03.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:03.118 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:16:03.219 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:16:03.250 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:03.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:03.251 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:16:03.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:03 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/892908396' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:16:03.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:03 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/892908396' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:16:03.352 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:16:03.375 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:03.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:03.376 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:16:03.477 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:16:03.512 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:03.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:03.513 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:16:03.614 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:16:03.643 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:16:03.645 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:03.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:03.646 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:16:03.746 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:16:03.771 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:03.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:03.772 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:16:03.873 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:16:03.895 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:03.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:03.896 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:16:03.997 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:16:04.021 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:04.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:04.022 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:16:04.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:04 smithi031 ceph-mon[16336]: 1.a scrub starts 2024-08-19T14:16:04.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:04 smithi031 ceph-mon[16336]: 1.a scrub ok 2024-08-19T14:16:04.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:04 smithi031 ceph-mon[16336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:16:04.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:04 smithi031 ceph-mon[16336]: Cluster is now healthy 2024-08-19T14:16:04.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:04 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/892908396' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:16:04.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:04 smithi031 ceph-mon[16336]: osdmap e195: 12 total, 12 up, 10 in 2024-08-19T14:16:04.123 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:16:04.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:04.187 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:04.187 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:16:04.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:04 smithi032 ceph-mon[20004]: 1.a scrub starts 2024-08-19T14:16:04.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:04 smithi032 ceph-mon[20004]: 1.a scrub ok 2024-08-19T14:16:04.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:04 smithi032 ceph-mon[20004]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:16:04.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:04 smithi032 ceph-mon[20004]: Cluster is now healthy 2024-08-19T14:16:04.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:04 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/892908396' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:16:04.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:04 smithi032 ceph-mon[20004]: osdmap e195: 12 total, 12 up, 10 in 2024-08-19T14:16:04.288 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:16:04.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:04 smithi045 ceph-mon[20143]: 1.a scrub starts 2024-08-19T14:16:04.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:04 smithi045 ceph-mon[20143]: 1.a scrub ok 2024-08-19T14:16:04.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:04 smithi045 ceph-mon[20143]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:16:04.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:04 smithi045 ceph-mon[20143]: Cluster is now healthy 2024-08-19T14:16:04.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:04 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/892908396' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:16:04.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:04 smithi045 ceph-mon[20143]: osdmap e195: 12 total, 12 up, 10 in 2024-08-19T14:16:04.310 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:04.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:04.311 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:16:04.414 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:16:04.463 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:04.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:04.464 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:16:04.565 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:16:04.591 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:04.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:04.592 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:16:04.620 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-08-19T14:16:04.693 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:16:04.718 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:04.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:04.719 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:16:04.822 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:16:04.849 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:04.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:04.850 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:16:04.937 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:16:04.951 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:16:04.972 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:04.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:04.973 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:16:05.074 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:16:05.097 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:05 smithi031 ceph-mon[16336]: 2.b scrub starts 2024-08-19T14:16:05.097 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:05 smithi031 ceph-mon[16336]: 1.9 deep-scrub starts 2024-08-19T14:16:05.097 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:05 smithi031 ceph-mon[16336]: 1.9 deep-scrub ok 2024-08-19T14:16:05.097 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:05 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:16:05.098 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:05 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:16:05.098 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:05 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:16:05.098 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:05 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:16:05.098 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:05 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:16:05.098 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:05 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:16:05.098 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:05 smithi031 ceph-mon[16336]: osdmap e196: 12 total, 12 up, 10 in 2024-08-19T14:16:05.098 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:05 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:16:05.099 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:05 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:16:05.099 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:05 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:16:05.099 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:05 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:16:05.099 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:05 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:16:05.102 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:05.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:05.103 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:16:05.204 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:16:05.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:05.231 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:05.231 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:16:05.251 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:16:05.251 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:16:05.251 INFO:teuthology.orchestra.run.smithi031.stdout: "epoch": 196, 2024-08-19T14:16:05.252 INFO:teuthology.orchestra.run.smithi031.stdout: "fsid": "ed7d1c6e-5e33-11ef-bccf-c7b262605968", 2024-08-19T14:16:05.252 INFO:teuthology.orchestra.run.smithi031.stdout: "created": "2024-08-19T14:05:22.842169+0000", 2024-08-19T14:16:05.252 INFO:teuthology.orchestra.run.smithi031.stdout: "modified": "2024-08-19T14:16:04.019970+0000", 2024-08-19T14:16:05.252 INFO:teuthology.orchestra.run.smithi031.stdout: "last_up_change": "2024-08-19T14:11:39.491234+0000", 2024-08-19T14:16:05.252 INFO:teuthology.orchestra.run.smithi031.stdout: "last_in_change": "2024-08-19T14:13:18.699388+0000", 2024-08-19T14:16:05.252 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-08-19T14:16:05.252 INFO:teuthology.orchestra.run.smithi031.stdout: "flags_num": 5799936, 2024-08-19T14:16:05.252 INFO:teuthology.orchestra.run.smithi031.stdout: "flags_set": [ 2024-08-19T14:16:05.252 INFO:teuthology.orchestra.run.smithi031.stdout: "pglog_hardlimit", 2024-08-19T14:16:05.252 INFO:teuthology.orchestra.run.smithi031.stdout: "purged_snapdirs", 2024-08-19T14:16:05.252 INFO:teuthology.orchestra.run.smithi031.stdout: "recovery_deletes", 2024-08-19T14:16:05.252 INFO:teuthology.orchestra.run.smithi031.stdout: "sortbitwise" 2024-08-19T14:16:05.253 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-08-19T14:16:05.253 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_version": 27, 2024-08-19T14:16:05.253 INFO:teuthology.orchestra.run.smithi031.stdout: "full_ratio": 0.94999998807907104, 2024-08-19T14:16:05.253 INFO:teuthology.orchestra.run.smithi031.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-08-19T14:16:05.253 INFO:teuthology.orchestra.run.smithi031.stdout: "nearfull_ratio": 0.85000002384185791, 2024-08-19T14:16:05.253 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_snapshot": "", 2024-08-19T14:16:05.253 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_max": 2, 2024-08-19T14:16:05.253 INFO:teuthology.orchestra.run.smithi031.stdout: "max_osd": 12, 2024-08-19T14:16:05.253 INFO:teuthology.orchestra.run.smithi031.stdout: "require_min_compat_client": "luminous", 2024-08-19T14:16:05.253 INFO:teuthology.orchestra.run.smithi031.stdout: "min_compat_client": "jewel", 2024-08-19T14:16:05.253 INFO:teuthology.orchestra.run.smithi031.stdout: "require_osd_release": "squid", 2024-08-19T14:16:05.253 INFO:teuthology.orchestra.run.smithi031.stdout: "allow_crimson": false, 2024-08-19T14:16:05.253 INFO:teuthology.orchestra.run.smithi031.stdout: "pools": [ 2024-08-19T14:16:05.254 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.254 INFO:teuthology.orchestra.run.smithi031.stdout: "pool": 1, 2024-08-19T14:16:05.254 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_name": ".mgr", 2024-08-19T14:16:05.254 INFO:teuthology.orchestra.run.smithi031.stdout: "create_time": "2024-08-19T14:08:15.966221+0000", 2024-08-19T14:16:05.254 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-08-19T14:16:05.254 INFO:teuthology.orchestra.run.smithi031.stdout: "flags_names": "hashpspool", 2024-08-19T14:16:05.254 INFO:teuthology.orchestra.run.smithi031.stdout: "type": 1, 2024-08-19T14:16:05.254 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 3, 2024-08-19T14:16:05.254 INFO:teuthology.orchestra.run.smithi031.stdout: "min_size": 2, 2024-08-19T14:16:05.254 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_rule": 0, 2024-08-19T14:16:05.254 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_count": 0, 2024-08-19T14:16:05.254 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_target": 0, 2024-08-19T14:16:05.254 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_barrier": 0, 2024-08-19T14:16:05.255 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-08-19T14:16:05.255 INFO:teuthology.orchestra.run.smithi031.stdout: "object_hash": 2, 2024-08-19T14:16:05.255 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_autoscale_mode": "off", 2024-08-19T14:16:05.255 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num": 11, 2024-08-19T14:16:05.255 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_placement_num": 11, 2024-08-19T14:16:05.255 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_placement_num_target": 11, 2024-08-19T14:16:05.255 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num_target": 11, 2024-08-19T14:16:05.255 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num_pending": 11, 2024-08-19T14:16:05.255 INFO:teuthology.orchestra.run.smithi031.stdout: "last_pg_merge_meta": { 2024-08-19T14:16:05.256 INFO:teuthology.orchestra.run.smithi031.stdout: "source_pgid": "0.0", 2024-08-19T14:16:05.256 INFO:teuthology.orchestra.run.smithi031.stdout: "ready_epoch": 0, 2024-08-19T14:16:05.256 INFO:teuthology.orchestra.run.smithi031.stdout: "last_epoch_started": 0, 2024-08-19T14:16:05.256 INFO:teuthology.orchestra.run.smithi031.stdout: "last_epoch_clean": 0, 2024-08-19T14:16:05.256 INFO:teuthology.orchestra.run.smithi031.stdout: "source_version": "0'0", 2024-08-19T14:16:05.256 INFO:teuthology.orchestra.run.smithi031.stdout: "target_version": "0'0" 2024-08-19T14:16:05.256 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.256 INFO:teuthology.orchestra.run.smithi031.stdout: "last_change": "112", 2024-08-19T14:16:05.256 INFO:teuthology.orchestra.run.smithi031.stdout: "last_force_op_resend": "0", 2024-08-19T14:16:05.256 INFO:teuthology.orchestra.run.smithi031.stdout: "last_force_op_resend_prenautilus": "0", 2024-08-19T14:16:05.256 INFO:teuthology.orchestra.run.smithi031.stdout: "last_force_op_resend_preluminous": "110", 2024-08-19T14:16:05.256 INFO:teuthology.orchestra.run.smithi031.stdout: "auid": 0, 2024-08-19T14:16:05.256 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_mode": "selfmanaged", 2024-08-19T14:16:05.257 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_seq": 0, 2024-08-19T14:16:05.257 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_epoch": 0, 2024-08-19T14:16:05.257 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_snaps": [], 2024-08-19T14:16:05.257 INFO:teuthology.orchestra.run.smithi031.stdout: "removed_snaps": "[]", 2024-08-19T14:16:05.257 INFO:teuthology.orchestra.run.smithi031.stdout: "quota_max_bytes": 0, 2024-08-19T14:16:05.257 INFO:teuthology.orchestra.run.smithi031.stdout: "quota_max_objects": 0, 2024-08-19T14:16:05.257 INFO:teuthology.orchestra.run.smithi031.stdout: "tiers": [], 2024-08-19T14:16:05.257 INFO:teuthology.orchestra.run.smithi031.stdout: "tier_of": -1, 2024-08-19T14:16:05.257 INFO:teuthology.orchestra.run.smithi031.stdout: "read_tier": -1, 2024-08-19T14:16:05.257 INFO:teuthology.orchestra.run.smithi031.stdout: "write_tier": -1, 2024-08-19T14:16:05.257 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_mode": "none", 2024-08-19T14:16:05.257 INFO:teuthology.orchestra.run.smithi031.stdout: "target_max_bytes": 0, 2024-08-19T14:16:05.258 INFO:teuthology.orchestra.run.smithi031.stdout: "target_max_objects": 0, 2024-08-19T14:16:05.258 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-08-19T14:16:05.258 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-08-19T14:16:05.258 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_target_full_ratio_micro": 800000, 2024-08-19T14:16:05.258 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_min_flush_age": 0, 2024-08-19T14:16:05.258 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_min_evict_age": 0, 2024-08-19T14:16:05.258 INFO:teuthology.orchestra.run.smithi031.stdout: "erasure_code_profile": "", 2024-08-19T14:16:05.258 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_params": { 2024-08-19T14:16:05.258 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "none" 2024-08-19T14:16:05.258 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.258 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_period": 0, 2024-08-19T14:16:05.258 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_count": 0, 2024-08-19T14:16:05.258 INFO:teuthology.orchestra.run.smithi031.stdout: "use_gmt_hitset": true, 2024-08-19T14:16:05.259 INFO:teuthology.orchestra.run.smithi031.stdout: "min_read_recency_for_promote": 0, 2024-08-19T14:16:05.259 INFO:teuthology.orchestra.run.smithi031.stdout: "min_write_recency_for_promote": 0, 2024-08-19T14:16:05.259 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_grade_decay_rate": 0, 2024-08-19T14:16:05.259 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_search_last_n": 0, 2024-08-19T14:16:05.259 INFO:teuthology.orchestra.run.smithi031.stdout: "grade_table": [], 2024-08-19T14:16:05.259 INFO:teuthology.orchestra.run.smithi031.stdout: "stripe_width": 0, 2024-08-19T14:16:05.259 INFO:teuthology.orchestra.run.smithi031.stdout: "expected_num_objects": 0, 2024-08-19T14:16:05.259 INFO:teuthology.orchestra.run.smithi031.stdout: "fast_read": false, 2024-08-19T14:16:05.259 INFO:teuthology.orchestra.run.smithi031.stdout: "options": { 2024-08-19T14:16:05.259 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num_max": 32, 2024-08-19T14:16:05.259 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num_min": 1 2024-08-19T14:16:05.259 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.259 INFO:teuthology.orchestra.run.smithi031.stdout: "application_metadata": { 2024-08-19T14:16:05.259 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": {} 2024-08-19T14:16:05.260 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.260 INFO:teuthology.orchestra.run.smithi031.stdout: "read_balance": { 2024-08-19T14:16:05.260 INFO:teuthology.orchestra.run.smithi031.stdout: "score_type": "Fair distribution", 2024-08-19T14:16:05.260 INFO:teuthology.orchestra.run.smithi031.stdout: "score_acting": 3.2899999618530273, 2024-08-19T14:16:05.260 INFO:teuthology.orchestra.run.smithi031.stdout: "score_stable": 3.2899999618530273, 2024-08-19T14:16:05.260 INFO:teuthology.orchestra.run.smithi031.stdout: "optimal_score": 0.82999998331069946, 2024-08-19T14:16:05.260 INFO:teuthology.orchestra.run.smithi031.stdout: "raw_score_acting": 2.7300000190734863, 2024-08-19T14:16:05.260 INFO:teuthology.orchestra.run.smithi031.stdout: "raw_score_stable": 2.7300000190734863, 2024-08-19T14:16:05.260 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity_weighted": 0.99999988079071045, 2024-08-19T14:16:05.260 INFO:teuthology.orchestra.run.smithi031.stdout: "average_primary_affinity": 1, 2024-08-19T14:16:05.260 INFO:teuthology.orchestra.run.smithi031.stdout: "average_primary_affinity_weighted": 1 2024-08-19T14:16:05.260 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:05.260 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.261 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.261 INFO:teuthology.orchestra.run.smithi031.stdout: "pool": 2, 2024-08-19T14:16:05.261 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_name": "unique_pool_0", 2024-08-19T14:16:05.261 INFO:teuthology.orchestra.run.smithi031.stdout: "create_time": "2024-08-19T14:12:41.289434+0000", 2024-08-19T14:16:05.261 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-08-19T14:16:05.261 INFO:teuthology.orchestra.run.smithi031.stdout: "flags_names": "hashpspool", 2024-08-19T14:16:05.261 INFO:teuthology.orchestra.run.smithi031.stdout: "type": 1, 2024-08-19T14:16:05.261 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 3, 2024-08-19T14:16:05.261 INFO:teuthology.orchestra.run.smithi031.stdout: "min_size": 2, 2024-08-19T14:16:05.261 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_rule": 0, 2024-08-19T14:16:05.261 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_count": 0, 2024-08-19T14:16:05.261 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_target": 0, 2024-08-19T14:16:05.261 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_barrier": 0, 2024-08-19T14:16:05.262 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-08-19T14:16:05.262 INFO:teuthology.orchestra.run.smithi031.stdout: "object_hash": 2, 2024-08-19T14:16:05.262 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_autoscale_mode": "off", 2024-08-19T14:16:05.262 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num": 16, 2024-08-19T14:16:05.262 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_placement_num": 16, 2024-08-19T14:16:05.262 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_placement_num_target": 16, 2024-08-19T14:16:05.262 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num_target": 16, 2024-08-19T14:16:05.262 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num_pending": 16, 2024-08-19T14:16:05.262 INFO:teuthology.orchestra.run.smithi031.stdout: "last_pg_merge_meta": { 2024-08-19T14:16:05.262 INFO:teuthology.orchestra.run.smithi031.stdout: "source_pgid": "0.0", 2024-08-19T14:16:05.262 INFO:teuthology.orchestra.run.smithi031.stdout: "ready_epoch": 0, 2024-08-19T14:16:05.262 INFO:teuthology.orchestra.run.smithi031.stdout: "last_epoch_started": 0, 2024-08-19T14:16:05.262 INFO:teuthology.orchestra.run.smithi031.stdout: "last_epoch_clean": 0, 2024-08-19T14:16:05.263 INFO:teuthology.orchestra.run.smithi031.stdout: "source_version": "0'0", 2024-08-19T14:16:05.263 INFO:teuthology.orchestra.run.smithi031.stdout: "target_version": "0'0" 2024-08-19T14:16:05.263 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.263 INFO:teuthology.orchestra.run.smithi031.stdout: "last_change": "136", 2024-08-19T14:16:05.263 INFO:teuthology.orchestra.run.smithi031.stdout: "last_force_op_resend": "0", 2024-08-19T14:16:05.263 INFO:teuthology.orchestra.run.smithi031.stdout: "last_force_op_resend_prenautilus": "0", 2024-08-19T14:16:05.263 INFO:teuthology.orchestra.run.smithi031.stdout: "last_force_op_resend_preluminous": "0", 2024-08-19T14:16:05.264 INFO:teuthology.orchestra.run.smithi031.stdout: "auid": 0, 2024-08-19T14:16:05.264 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_mode": "selfmanaged", 2024-08-19T14:16:05.264 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_seq": 0, 2024-08-19T14:16:05.264 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_epoch": 0, 2024-08-19T14:16:05.264 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_snaps": [], 2024-08-19T14:16:05.264 INFO:teuthology.orchestra.run.smithi031.stdout: "removed_snaps": "[]", 2024-08-19T14:16:05.264 INFO:teuthology.orchestra.run.smithi031.stdout: "quota_max_bytes": 0, 2024-08-19T14:16:05.264 INFO:teuthology.orchestra.run.smithi031.stdout: "quota_max_objects": 0, 2024-08-19T14:16:05.265 INFO:teuthology.orchestra.run.smithi031.stdout: "tiers": [], 2024-08-19T14:16:05.265 INFO:teuthology.orchestra.run.smithi031.stdout: "tier_of": -1, 2024-08-19T14:16:05.265 INFO:teuthology.orchestra.run.smithi031.stdout: "read_tier": -1, 2024-08-19T14:16:05.265 INFO:teuthology.orchestra.run.smithi031.stdout: "write_tier": -1, 2024-08-19T14:16:05.265 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_mode": "none", 2024-08-19T14:16:05.265 INFO:teuthology.orchestra.run.smithi031.stdout: "target_max_bytes": 0, 2024-08-19T14:16:05.266 INFO:teuthology.orchestra.run.smithi031.stdout: "target_max_objects": 0, 2024-08-19T14:16:05.266 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-08-19T14:16:05.266 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-08-19T14:16:05.266 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_target_full_ratio_micro": 800000, 2024-08-19T14:16:05.266 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_min_flush_age": 0, 2024-08-19T14:16:05.266 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_min_evict_age": 0, 2024-08-19T14:16:05.266 INFO:teuthology.orchestra.run.smithi031.stdout: "erasure_code_profile": "", 2024-08-19T14:16:05.266 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_params": { 2024-08-19T14:16:05.266 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "none" 2024-08-19T14:16:05.266 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.266 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_period": 0, 2024-08-19T14:16:05.267 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_count": 0, 2024-08-19T14:16:05.267 INFO:teuthology.orchestra.run.smithi031.stdout: "use_gmt_hitset": true, 2024-08-19T14:16:05.267 INFO:teuthology.orchestra.run.smithi031.stdout: "min_read_recency_for_promote": 0, 2024-08-19T14:16:05.267 INFO:teuthology.orchestra.run.smithi031.stdout: "min_write_recency_for_promote": 0, 2024-08-19T14:16:05.267 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_grade_decay_rate": 0, 2024-08-19T14:16:05.267 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_search_last_n": 0, 2024-08-19T14:16:05.267 INFO:teuthology.orchestra.run.smithi031.stdout: "grade_table": [], 2024-08-19T14:16:05.267 INFO:teuthology.orchestra.run.smithi031.stdout: "stripe_width": 0, 2024-08-19T14:16:05.267 INFO:teuthology.orchestra.run.smithi031.stdout: "expected_num_objects": 0, 2024-08-19T14:16:05.267 INFO:teuthology.orchestra.run.smithi031.stdout: "fast_read": false, 2024-08-19T14:16:05.267 INFO:teuthology.orchestra.run.smithi031.stdout: "options": {}, 2024-08-19T14:16:05.267 INFO:teuthology.orchestra.run.smithi031.stdout: "application_metadata": { 2024-08-19T14:16:05.268 INFO:teuthology.orchestra.run.smithi031.stdout: "rados": {} 2024-08-19T14:16:05.268 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.268 INFO:teuthology.orchestra.run.smithi031.stdout: "read_balance": { 2024-08-19T14:16:05.268 INFO:teuthology.orchestra.run.smithi031.stdout: "score_type": "Fair distribution", 2024-08-19T14:16:05.268 INFO:teuthology.orchestra.run.smithi031.stdout: "score_acting": 2.2699999809265137, 2024-08-19T14:16:05.268 INFO:teuthology.orchestra.run.smithi031.stdout: "score_stable": 2.2699999809265137, 2024-08-19T14:16:05.268 INFO:teuthology.orchestra.run.smithi031.stdout: "optimal_score": 0.82999998331069946, 2024-08-19T14:16:05.268 INFO:teuthology.orchestra.run.smithi031.stdout: "raw_score_acting": 1.8799999952316284, 2024-08-19T14:16:05.268 INFO:teuthology.orchestra.run.smithi031.stdout: "raw_score_stable": 1.8799999952316284, 2024-08-19T14:16:05.268 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity_weighted": 0.99999988079071045, 2024-08-19T14:16:05.268 INFO:teuthology.orchestra.run.smithi031.stdout: "average_primary_affinity": 1, 2024-08-19T14:16:05.269 INFO:teuthology.orchestra.run.smithi031.stdout: "average_primary_affinity_weighted": 1 2024-08-19T14:16:05.269 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:05.269 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:05.269 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-08-19T14:16:05.269 INFO:teuthology.orchestra.run.smithi031.stdout: "osds": [ 2024-08-19T14:16:05.269 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.269 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 0, 2024-08-19T14:16:05.269 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "d7a0fff3-23c6-4ee3-a275-a844862aabfb", 2024-08-19T14:16:05.269 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-08-19T14:16:05.269 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-08-19T14:16:05.269 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-08-19T14:16:05.269 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-08-19T14:16:05.269 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-08-19T14:16:05.270 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-08-19T14:16:05.270 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 8, 2024-08-19T14:16:05.270 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 193, 2024-08-19T14:16:05.270 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-08-19T14:16:05.270 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-08-19T14:16:05.270 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-08-19T14:16:05.270 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:05.270 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.270 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:05.270 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6802", 2024-08-19T14:16:05.270 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1108945097 2024-08-19T14:16:05.271 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.271 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.271 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:05.271 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6803", 2024-08-19T14:16:05.271 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1108945097 2024-08-19T14:16:05.271 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:05.271 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:05.271 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.271 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-08-19T14:16:05.271 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:05.271 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.271 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:05.272 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6804", 2024-08-19T14:16:05.272 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1108945097 2024-08-19T14:16:05.272 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.272 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.272 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:05.273 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6805", 2024-08-19T14:16:05.273 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1108945097 2024-08-19T14:16:05.273 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:05.273 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:05.273 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.273 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-08-19T14:16:05.273 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:05.273 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.273 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:05.273 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6808", 2024-08-19T14:16:05.273 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1108945097 2024-08-19T14:16:05.274 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.274 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.274 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:05.274 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6809", 2024-08-19T14:16:05.274 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1108945097 2024-08-19T14:16:05.274 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:05.274 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:05.274 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.274 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-08-19T14:16:05.274 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:05.274 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.275 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:05.275 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6806", 2024-08-19T14:16:05.275 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1108945097 2024-08-19T14:16:05.275 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.275 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.275 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:05.275 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6807", 2024-08-19T14:16:05.275 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1108945097 2024-08-19T14:16:05.275 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:05.275 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:05.275 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.275 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.31:6803/1108945097", 2024-08-19T14:16:05.276 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.31:6805/1108945097", 2024-08-19T14:16:05.276 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.31:6809/1108945097", 2024-08-19T14:16:05.276 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.31:6807/1108945097", 2024-08-19T14:16:05.276 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-08-19T14:16:05.276 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-08-19T14:16:05.276 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-08-19T14:16:05.276 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:05.276 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.276 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.276 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 1, 2024-08-19T14:16:05.276 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "145d5bb4-74dd-440f-bcc9-7e9ebda9c5d6", 2024-08-19T14:16:05.277 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-08-19T14:16:05.277 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-08-19T14:16:05.277 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-08-19T14:16:05.277 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-08-19T14:16:05.277 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-08-19T14:16:05.277 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-08-19T14:16:05.277 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 13, 2024-08-19T14:16:05.277 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 146, 2024-08-19T14:16:05.277 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-08-19T14:16:05.277 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-08-19T14:16:05.277 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-08-19T14:16:05.278 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:05.278 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.278 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:05.278 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6810", 2024-08-19T14:16:05.278 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3236186806 2024-08-19T14:16:05.278 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.278 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.278 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:05.278 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6811", 2024-08-19T14:16:05.278 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3236186806 2024-08-19T14:16:05.278 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:05.279 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:05.279 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.279 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-08-19T14:16:05.279 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:05.279 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.279 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:05.279 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6812", 2024-08-19T14:16:05.279 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3236186806 2024-08-19T14:16:05.279 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.279 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.279 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:05.280 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6813", 2024-08-19T14:16:05.280 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3236186806 2024-08-19T14:16:05.280 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:05.280 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:05.280 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.280 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-08-19T14:16:05.280 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:05.280 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.280 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:05.280 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6816", 2024-08-19T14:16:05.280 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3236186806 2024-08-19T14:16:05.280 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.281 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.281 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:05.281 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6817", 2024-08-19T14:16:05.281 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3236186806 2024-08-19T14:16:05.281 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:05.281 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:05.281 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.281 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-08-19T14:16:05.281 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:05.281 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.281 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:05.281 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6814", 2024-08-19T14:16:05.282 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3236186806 2024-08-19T14:16:05.282 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.282 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.282 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:05.282 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6815", 2024-08-19T14:16:05.282 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3236186806 2024-08-19T14:16:05.282 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:05.282 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:05.282 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.282 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.31:6811/3236186806", 2024-08-19T14:16:05.282 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.31:6813/3236186806", 2024-08-19T14:16:05.283 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.31:6817/3236186806", 2024-08-19T14:16:05.283 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.31:6815/3236186806", 2024-08-19T14:16:05.283 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-08-19T14:16:05.283 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-08-19T14:16:05.283 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-08-19T14:16:05.283 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:05.283 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.283 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.283 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 2, 2024-08-19T14:16:05.283 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "376c18d0-4768-46a0-bb60-e79c88ca909c", 2024-08-19T14:16:05.283 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-08-19T14:16:05.283 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-08-19T14:16:05.284 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-08-19T14:16:05.284 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-08-19T14:16:05.284 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-08-19T14:16:05.284 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-08-19T14:16:05.284 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 18, 2024-08-19T14:16:05.284 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 191, 2024-08-19T14:16:05.284 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-08-19T14:16:05.284 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-08-19T14:16:05.284 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-08-19T14:16:05.284 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:05.285 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.285 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:05.285 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6818", 2024-08-19T14:16:05.285 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 4045553387 2024-08-19T14:16:05.285 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.285 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.285 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:05.285 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6819", 2024-08-19T14:16:05.285 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 4045553387 2024-08-19T14:16:05.286 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:05.286 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:05.286 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.286 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-08-19T14:16:05.286 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:05.286 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.286 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:05.286 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6820", 2024-08-19T14:16:05.286 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 4045553387 2024-08-19T14:16:05.286 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.286 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.287 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:05.287 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6821", 2024-08-19T14:16:05.287 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 4045553387 2024-08-19T14:16:05.287 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:05.287 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:05.287 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.287 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-08-19T14:16:05.287 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:05.288 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.288 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:05.288 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6824", 2024-08-19T14:16:05.288 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 4045553387 2024-08-19T14:16:05.288 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.288 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.288 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:05.288 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6825", 2024-08-19T14:16:05.288 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 4045553387 2024-08-19T14:16:05.288 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:05.288 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:05.289 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.289 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-08-19T14:16:05.289 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:05.289 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.289 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:05.289 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6822", 2024-08-19T14:16:05.289 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 4045553387 2024-08-19T14:16:05.289 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.289 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.290 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:05.290 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6823", 2024-08-19T14:16:05.290 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 4045553387 2024-08-19T14:16:05.290 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:05.290 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:05.290 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.290 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.31:6819/4045553387", 2024-08-19T14:16:05.290 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.31:6821/4045553387", 2024-08-19T14:16:05.290 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.31:6825/4045553387", 2024-08-19T14:16:05.290 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.31:6823/4045553387", 2024-08-19T14:16:05.291 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-08-19T14:16:05.291 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-08-19T14:16:05.291 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-08-19T14:16:05.291 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:05.291 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.291 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.291 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 3, 2024-08-19T14:16:05.291 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "a787ea9f-b9ba-450f-b388-614cfba90634", 2024-08-19T14:16:05.292 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-08-19T14:16:05.292 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-08-19T14:16:05.292 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 0.6437225341796875, 2024-08-19T14:16:05.292 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-08-19T14:16:05.292 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-08-19T14:16:05.292 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-08-19T14:16:05.292 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 26, 2024-08-19T14:16:05.292 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 129, 2024-08-19T14:16:05.292 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-08-19T14:16:05.293 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-08-19T14:16:05.293 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-08-19T14:16:05.293 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:05.293 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.293 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:05.293 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6826", 2024-08-19T14:16:05.293 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2288082716 2024-08-19T14:16:05.293 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.293 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.294 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:05.294 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6827", 2024-08-19T14:16:05.294 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2288082716 2024-08-19T14:16:05.294 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:05.294 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:05.294 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.294 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-08-19T14:16:05.294 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:05.294 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.295 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:05.295 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6828", 2024-08-19T14:16:05.295 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2288082716 2024-08-19T14:16:05.295 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.295 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.295 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:05.295 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6829", 2024-08-19T14:16:05.295 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2288082716 2024-08-19T14:16:05.296 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:05.296 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:05.296 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.296 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-08-19T14:16:05.296 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:05.296 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.296 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:05.297 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6832", 2024-08-19T14:16:05.297 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2288082716 2024-08-19T14:16:05.297 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.297 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:05 smithi045 ceph-mon[20143]: 2.b scrub starts 2024-08-19T14:16:05.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:05 smithi045 ceph-mon[20143]: 1.9 deep-scrub starts 2024-08-19T14:16:05.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:05 smithi045 ceph-mon[20143]: 1.9 deep-scrub ok 2024-08-19T14:16:05.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:05 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:16:05.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:05 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:16:05.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:05 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:16:05.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:05 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:16:05.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:05 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:16:05.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:05 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:16:05.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:05 smithi045 ceph-mon[20143]: osdmap e196: 12 total, 12 up, 10 in 2024-08-19T14:16:05.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:05 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:16:05.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:05 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:16:05.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:05 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:16:05.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:05 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:16:05.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:05 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:16:05.300 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:05.300 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6833", 2024-08-19T14:16:05.300 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2288082716 2024-08-19T14:16:05.300 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:05.300 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:05.300 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.300 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-08-19T14:16:05.300 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:05.300 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.300 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:05.300 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6830", 2024-08-19T14:16:05.301 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2288082716 2024-08-19T14:16:05.301 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.301 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.301 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:05.301 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6831", 2024-08-19T14:16:05.301 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2288082716 2024-08-19T14:16:05.301 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:05.301 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:05.301 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.301 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.31:6827/2288082716", 2024-08-19T14:16:05.301 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.31:6829/2288082716", 2024-08-19T14:16:05.301 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.31:6833/2288082716", 2024-08-19T14:16:05.302 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.31:6831/2288082716", 2024-08-19T14:16:05.302 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-08-19T14:16:05.302 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-08-19T14:16:05.302 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-08-19T14:16:05.302 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:05.302 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.302 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.302 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 4, 2024-08-19T14:16:05.302 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "7859751a-12f5-4a57-a332-435360479d53", 2024-08-19T14:16:05.302 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-08-19T14:16:05.302 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-08-19T14:16:05.302 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-08-19T14:16:05.303 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-08-19T14:16:05.303 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-08-19T14:16:05.303 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-08-19T14:16:05.303 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 31, 2024-08-19T14:16:05.303 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 0, 2024-08-19T14:16:05.303 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-08-19T14:16:05.303 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-08-19T14:16:05.303 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-08-19T14:16:05.303 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:05.303 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.303 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:05.304 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6800", 2024-08-19T14:16:05.304 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3934739984 2024-08-19T14:16:05.304 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.304 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.304 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:05.304 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6801", 2024-08-19T14:16:05.304 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3934739984 2024-08-19T14:16:05.304 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:05.304 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:05.304 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.304 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-08-19T14:16:05.304 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:05.305 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.305 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:05.305 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6802", 2024-08-19T14:16:05.305 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3934739984 2024-08-19T14:16:05.305 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.305 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.305 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:05.305 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6803", 2024-08-19T14:16:05.305 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3934739984 2024-08-19T14:16:05.305 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:05.305 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:05.305 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.306 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-08-19T14:16:05.306 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:05.306 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.306 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:05.306 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6806", 2024-08-19T14:16:05.306 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3934739984 2024-08-19T14:16:05.306 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.306 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.306 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:05.306 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6807", 2024-08-19T14:16:05.306 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3934739984 2024-08-19T14:16:05.306 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:05.307 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:05.307 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.307 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-08-19T14:16:05.307 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:05.307 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.307 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:05.307 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6804", 2024-08-19T14:16:05.307 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3934739984 2024-08-19T14:16:05.307 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.307 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.307 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:05.307 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6805", 2024-08-19T14:16:05.308 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3934739984 2024-08-19T14:16:05.308 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:05.308 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:05.308 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.308 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.32:6801/3934739984", 2024-08-19T14:16:05.308 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.32:6803/3934739984", 2024-08-19T14:16:05.308 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.32:6807/3934739984", 2024-08-19T14:16:05.308 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.32:6805/3934739984", 2024-08-19T14:16:05.308 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-08-19T14:16:05.308 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-08-19T14:16:05.308 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-08-19T14:16:05.309 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:05.309 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.309 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.309 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 5, 2024-08-19T14:16:05.309 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "017f3eef-ac60-4b82-8350-919017cb4ccd", 2024-08-19T14:16:05.309 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-08-19T14:16:05.309 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-08-19T14:16:05.309 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-08-19T14:16:05.309 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-08-19T14:16:05.309 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-08-19T14:16:05.309 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-08-19T14:16:05.310 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 40, 2024-08-19T14:16:05.310 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 174, 2024-08-19T14:16:05.310 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-08-19T14:16:05.310 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-08-19T14:16:05.310 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-08-19T14:16:05.310 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:05.310 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.310 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:05.310 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6808", 2024-08-19T14:16:05.310 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 526388985 2024-08-19T14:16:05.310 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.310 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.311 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:05.311 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6809", 2024-08-19T14:16:05.311 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 526388985 2024-08-19T14:16:05.311 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:05.311 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:05.311 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.311 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-08-19T14:16:05.311 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:05.311 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.311 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:05.311 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6810", 2024-08-19T14:16:05.312 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 526388985 2024-08-19T14:16:05.312 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.312 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.312 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:05.312 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6811", 2024-08-19T14:16:05.312 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 526388985 2024-08-19T14:16:05.312 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:05.312 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:05.312 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.312 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-08-19T14:16:05.312 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:05.313 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.313 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:05.313 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6814", 2024-08-19T14:16:05.313 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 526388985 2024-08-19T14:16:05.313 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.313 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.313 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:05.313 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6815", 2024-08-19T14:16:05.313 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 526388985 2024-08-19T14:16:05.313 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:05.313 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:05.313 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.314 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-08-19T14:16:05.314 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:05.314 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.314 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:05.314 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6812", 2024-08-19T14:16:05.314 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 526388985 2024-08-19T14:16:05.314 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.314 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.314 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:05.314 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6813", 2024-08-19T14:16:05.314 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 526388985 2024-08-19T14:16:05.315 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:05.315 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:05.315 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.315 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.32:6809/526388985", 2024-08-19T14:16:05.315 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.32:6811/526388985", 2024-08-19T14:16:05.315 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.32:6815/526388985", 2024-08-19T14:16:05.315 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.32:6813/526388985", 2024-08-19T14:16:05.315 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-08-19T14:16:05.315 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-08-19T14:16:05.315 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-08-19T14:16:05.315 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:05.315 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.315 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.316 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 6, 2024-08-19T14:16:05.316 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "44aa8229-0268-49d7-bd57-3bf1f384e5e6", 2024-08-19T14:16:05.316 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-08-19T14:16:05.316 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 0, 2024-08-19T14:16:05.316 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 0, 2024-08-19T14:16:05.316 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-08-19T14:16:05.316 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-08-19T14:16:05.316 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-08-19T14:16:05.316 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 48, 2024-08-19T14:16:05.316 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 126, 2024-08-19T14:16:05.317 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-08-19T14:16:05.317 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-08-19T14:16:05.317 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-08-19T14:16:05.317 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:05.317 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.317 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:05.317 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6816", 2024-08-19T14:16:05.317 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1131035844 2024-08-19T14:16:05.317 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.317 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.317 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:05.317 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6817", 2024-08-19T14:16:05.318 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1131035844 2024-08-19T14:16:05.318 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:05.318 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:05.319 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.319 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-08-19T14:16:05.319 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:05.319 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.319 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:05.319 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6818", 2024-08-19T14:16:05.319 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1131035844 2024-08-19T14:16:05.319 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.319 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.319 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:05.319 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6819", 2024-08-19T14:16:05.320 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1131035844 2024-08-19T14:16:05.320 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:05.320 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:05.320 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.320 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-08-19T14:16:05.320 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:05.320 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.320 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:05.320 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6822", 2024-08-19T14:16:05.320 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1131035844 2024-08-19T14:16:05.320 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.320 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.321 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:05.321 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6823", 2024-08-19T14:16:05.321 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1131035844 2024-08-19T14:16:05.321 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:05.321 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:05.321 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.321 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-08-19T14:16:05.321 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:05.321 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.321 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:05.321 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6820", 2024-08-19T14:16:05.321 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1131035844 2024-08-19T14:16:05.322 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.322 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.322 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:05.322 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6821", 2024-08-19T14:16:05.322 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1131035844 2024-08-19T14:16:05.322 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:05.322 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:05.322 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.322 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.32:6817/1131035844", 2024-08-19T14:16:05.322 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.32:6819/1131035844", 2024-08-19T14:16:05.322 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.32:6823/1131035844", 2024-08-19T14:16:05.323 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.32:6821/1131035844", 2024-08-19T14:16:05.323 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-08-19T14:16:05.323 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-08-19T14:16:05.323 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-08-19T14:16:05.323 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:05.323 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.323 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.323 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 7, 2024-08-19T14:16:05.323 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "263433e5-70b6-4192-86ff-dc2ad95927eb", 2024-08-19T14:16:05.323 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-08-19T14:16:05.323 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-08-19T14:16:05.323 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 0.95001220703125, 2024-08-19T14:16:05.324 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-08-19T14:16:05.324 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-08-19T14:16:05.324 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-08-19T14:16:05.324 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 57, 2024-08-19T14:16:05.324 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 132, 2024-08-19T14:16:05.324 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-08-19T14:16:05.324 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-08-19T14:16:05.324 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-08-19T14:16:05.324 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:05.324 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.324 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:05.325 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6824", 2024-08-19T14:16:05.325 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2014939947 2024-08-19T14:16:05.325 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.325 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.325 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:05.325 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6825", 2024-08-19T14:16:05.325 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2014939947 2024-08-19T14:16:05.325 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:05.325 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:05.325 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.325 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-08-19T14:16:05.325 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:05.326 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.326 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:05.326 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6826", 2024-08-19T14:16:05.326 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2014939947 2024-08-19T14:16:05.326 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.326 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.326 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:05.326 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6827", 2024-08-19T14:16:05.326 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2014939947 2024-08-19T14:16:05.326 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:05.326 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:05.326 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.327 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-08-19T14:16:05.327 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:05.327 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.327 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:05.327 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6830", 2024-08-19T14:16:05.327 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2014939947 2024-08-19T14:16:05.327 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.327 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.327 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:05.327 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6831", 2024-08-19T14:16:05.327 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2014939947 2024-08-19T14:16:05.327 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:05.328 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:05.328 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.328 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-08-19T14:16:05.328 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:05.328 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.328 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:05.328 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6828", 2024-08-19T14:16:05.328 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2014939947 2024-08-19T14:16:05.328 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.328 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.328 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:05.329 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6829", 2024-08-19T14:16:05.329 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2014939947 2024-08-19T14:16:05.329 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:05.329 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:05.329 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.329 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.32:6825/2014939947", 2024-08-19T14:16:05.329 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.32:6827/2014939947", 2024-08-19T14:16:05.329 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.32:6831/2014939947", 2024-08-19T14:16:05.329 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.32:6829/2014939947", 2024-08-19T14:16:05.329 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-08-19T14:16:05.329 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-08-19T14:16:05.329 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-08-19T14:16:05.330 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:05.330 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.330 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.330 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 8, 2024-08-19T14:16:05.330 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "2bbcad36-0c74-4528-9fb9-a8812afa1662", 2024-08-19T14:16:05.330 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-08-19T14:16:05.330 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-08-19T14:16:05.330 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-08-19T14:16:05.330 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-08-19T14:16:05.330 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-08-19T14:16:05.330 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-08-19T14:16:05.331 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 65, 2024-08-19T14:16:05.331 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 102, 2024-08-19T14:16:05.331 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-08-19T14:16:05.331 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-08-19T14:16:05.331 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-08-19T14:16:05.331 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:05.331 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.331 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:05.331 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6800", 2024-08-19T14:16:05.331 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1445140892 2024-08-19T14:16:05.331 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.331 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.332 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:05.332 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6801", 2024-08-19T14:16:05.332 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1445140892 2024-08-19T14:16:05.332 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:05.332 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:05.332 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.332 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-08-19T14:16:05.332 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:05.332 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.332 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:05.332 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6802", 2024-08-19T14:16:05.332 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1445140892 2024-08-19T14:16:05.333 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.333 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.333 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:05.333 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6803", 2024-08-19T14:16:05.333 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1445140892 2024-08-19T14:16:05.333 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:05.333 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:05.333 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.333 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-08-19T14:16:05.333 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:05.333 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.333 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:05.334 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6806", 2024-08-19T14:16:05.334 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1445140892 2024-08-19T14:16:05.334 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.334 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.334 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:05.334 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6807", 2024-08-19T14:16:05.334 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1445140892 2024-08-19T14:16:05.334 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:05.334 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:05.334 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.334 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-08-19T14:16:05.335 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:05.335 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.335 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:05.335 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6804", 2024-08-19T14:16:05.335 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1445140892 2024-08-19T14:16:05.335 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.335 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.335 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:05.335 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6805", 2024-08-19T14:16:05.335 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1445140892 2024-08-19T14:16:05.335 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:05.335 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:05.336 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.336 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.45:6801/1445140892", 2024-08-19T14:16:05.336 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.45:6803/1445140892", 2024-08-19T14:16:05.336 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.45:6807/1445140892", 2024-08-19T14:16:05.336 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.45:6805/1445140892", 2024-08-19T14:16:05.336 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-08-19T14:16:05.336 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-08-19T14:16:05.336 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-08-19T14:16:05.336 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:05.336 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.336 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.336 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 9, 2024-08-19T14:16:05.337 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "c8ee4277-3bf9-46bf-9cc0-152c1c4b9508", 2024-08-19T14:16:05.337 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-08-19T14:16:05.337 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-08-19T14:16:05.337 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 0.95001220703125, 2024-08-19T14:16:05.337 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-08-19T14:16:05.337 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-08-19T14:16:05.337 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-08-19T14:16:05.337 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 74, 2024-08-19T14:16:05.337 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 129, 2024-08-19T14:16:05.337 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-08-19T14:16:05.337 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-08-19T14:16:05.337 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-08-19T14:16:05.338 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:05.338 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.338 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:05.338 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6808", 2024-08-19T14:16:05.338 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1213787169 2024-08-19T14:16:05.338 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.338 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.338 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:05.338 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6809", 2024-08-19T14:16:05.338 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1213787169 2024-08-19T14:16:05.338 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:05.339 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:05.339 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.339 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-08-19T14:16:05.339 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:05.339 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.339 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:05.339 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6810", 2024-08-19T14:16:05.339 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1213787169 2024-08-19T14:16:05.339 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.339 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.339 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:05.340 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6811", 2024-08-19T14:16:05.340 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1213787169 2024-08-19T14:16:05.340 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:05.340 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:05.340 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.340 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-08-19T14:16:05.340 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:05.340 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.340 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:05.340 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6814", 2024-08-19T14:16:05.341 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1213787169 2024-08-19T14:16:05.341 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.341 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.341 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:05.341 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6815", 2024-08-19T14:16:05.341 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1213787169 2024-08-19T14:16:05.341 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:05.341 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:05.341 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.341 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-08-19T14:16:05.341 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:05.342 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.342 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:05.342 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6812", 2024-08-19T14:16:05.342 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1213787169 2024-08-19T14:16:05.342 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.342 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.342 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:05.342 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6813", 2024-08-19T14:16:05.342 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1213787169 2024-08-19T14:16:05.342 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:05.342 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:05.342 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.343 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.45:6809/1213787169", 2024-08-19T14:16:05.343 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.45:6811/1213787169", 2024-08-19T14:16:05.343 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.45:6815/1213787169", 2024-08-19T14:16:05.343 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.45:6813/1213787169", 2024-08-19T14:16:05.343 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-08-19T14:16:05.343 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-08-19T14:16:05.343 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-08-19T14:16:05.343 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:05.343 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.343 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.343 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 10, 2024-08-19T14:16:05.344 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "29031395-5f5c-40f8-ac90-9f6ac48e5c9f", 2024-08-19T14:16:05.344 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-08-19T14:16:05.344 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:16:05.345 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 0, 2024-08-19T14:16:05.345 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 0, 2024-08-19T14:16:05.345 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-08-19T14:16:05.345 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-08-19T14:16:05.345 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-08-19T14:16:05.345 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 80, 2024-08-19T14:16:05.346 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 115, 2024-08-19T14:16:05.346 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-08-19T14:16:05.346 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-08-19T14:16:05.346 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-08-19T14:16:05.346 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:05.346 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.346 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:05.346 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6816", 2024-08-19T14:16:05.346 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2461824621 2024-08-19T14:16:05.346 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.346 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.347 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:05.347 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6817", 2024-08-19T14:16:05.347 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2461824621 2024-08-19T14:16:05.347 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:05.347 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:05.347 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.347 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-08-19T14:16:05.347 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:05.347 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.347 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:05.347 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6818", 2024-08-19T14:16:05.348 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2461824621 2024-08-19T14:16:05.348 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.348 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.348 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:05.348 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6819", 2024-08-19T14:16:05.348 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2461824621 2024-08-19T14:16:05.348 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:05.348 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:05.348 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.348 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-08-19T14:16:05.348 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:05.349 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.349 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:05.349 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6822", 2024-08-19T14:16:05.349 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2461824621 2024-08-19T14:16:05.349 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.349 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.349 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:05.349 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6823", 2024-08-19T14:16:05.349 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2461824621 2024-08-19T14:16:05.349 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:05.349 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:05.350 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.350 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-08-19T14:16:05.350 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:05.350 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.350 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:05.350 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6820", 2024-08-19T14:16:05.350 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2461824621 2024-08-19T14:16:05.350 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.350 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.350 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:05.350 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6821", 2024-08-19T14:16:05.350 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2461824621 2024-08-19T14:16:05.351 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:05.351 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:05.351 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.351 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.45:6817/2461824621", 2024-08-19T14:16:05.351 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.45:6819/2461824621", 2024-08-19T14:16:05.351 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.45:6823/2461824621", 2024-08-19T14:16:05.351 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.45:6821/2461824621", 2024-08-19T14:16:05.351 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-08-19T14:16:05.351 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-08-19T14:16:05.351 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-08-19T14:16:05.351 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:05.351 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.352 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.352 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 11, 2024-08-19T14:16:05.352 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "5ef0db4b-0fef-4ea0-a19b-d3677e082c29", 2024-08-19T14:16:05.352 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-08-19T14:16:05.352 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-08-19T14:16:05.352 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-08-19T14:16:05.352 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-08-19T14:16:05.352 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-08-19T14:16:05.352 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-08-19T14:16:05.352 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 86, 2024-08-19T14:16:05.352 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 115, 2024-08-19T14:16:05.352 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-08-19T14:16:05.353 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-08-19T14:16:05.353 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-08-19T14:16:05.353 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:05.353 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.353 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:05.353 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6824", 2024-08-19T14:16:05.353 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2438396298 2024-08-19T14:16:05.353 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.353 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.354 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:05.354 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6825", 2024-08-19T14:16:05.354 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2438396298 2024-08-19T14:16:05.354 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:05.354 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:05.354 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.354 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-08-19T14:16:05.354 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:05.354 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.354 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:05.355 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6826", 2024-08-19T14:16:05.355 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2438396298 2024-08-19T14:16:05.355 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.355 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.355 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:05.355 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6827", 2024-08-19T14:16:05.355 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2438396298 2024-08-19T14:16:05.355 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:05.355 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:05.355 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.355 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-08-19T14:16:05.356 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:05.356 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.356 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:05.356 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6830", 2024-08-19T14:16:05.356 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2438396298 2024-08-19T14:16:05.356 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.356 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.356 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:05.356 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6831", 2024-08-19T14:16:05.356 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2438396298 2024-08-19T14:16:05.356 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:05.356 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:05.357 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.357 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-08-19T14:16:05.357 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:05.357 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.357 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:05.357 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6828", 2024-08-19T14:16:05.357 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2438396298 2024-08-19T14:16:05.357 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.357 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.357 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:05.357 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6829", 2024-08-19T14:16:05.358 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2438396298 2024-08-19T14:16:05.358 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:05.358 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:05.358 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.358 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.45:6825/2438396298", 2024-08-19T14:16:05.358 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.45:6827/2438396298", 2024-08-19T14:16:05.358 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.45:6831/2438396298", 2024-08-19T14:16:05.358 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.45:6829/2438396298", 2024-08-19T14:16:05.358 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-08-19T14:16:05.358 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-08-19T14:16:05.358 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-08-19T14:16:05.358 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:05.359 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:05.359 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-08-19T14:16:05.359 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_xinfo": [ 2024-08-19T14:16:05.359 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.359 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 0, 2024-08-19T14:16:05.359 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-08-19T14:16:05.359 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-08-19T14:16:05.359 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-08-19T14:16:05.359 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-08-19T14:16:05.359 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-08-19T14:16:05.359 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-08-19T14:15:56.929307+0000", 2024-08-19T14:16:05.359 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-08-19T14:16:05.360 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.360 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.360 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 1, 2024-08-19T14:16:05.360 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-08-19T14:16:05.360 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-08-19T14:16:05.360 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-08-19T14:16:05.360 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-08-19T14:16:05.360 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-08-19T14:16:05.360 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-08-19T14:15:55.638039+0000", 2024-08-19T14:16:05.360 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-08-19T14:16:05.360 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.360 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.361 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 2, 2024-08-19T14:16:05.361 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-08-19T14:16:05.361 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-08-19T14:16:05.361 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-08-19T14:16:05.361 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-08-19T14:16:05.361 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-08-19T14:16:05.361 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-08-19T14:15:58.904876+0000", 2024-08-19T14:16:05.361 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-08-19T14:16:05.361 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.361 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.361 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 3, 2024-08-19T14:16:05.361 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-08-19T14:16:05.362 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-08-19T14:16:05.362 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-08-19T14:16:05.362 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-08-19T14:16:05.362 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-08-19T14:16:05.362 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-08-19T14:14:44.877317+0000", 2024-08-19T14:16:05.362 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-08-19T14:16:05.362 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.362 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.362 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 4, 2024-08-19T14:16:05.362 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-08-19T14:16:05.362 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-08-19T14:16:05.363 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-08-19T14:16:05.363 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-08-19T14:16:05.363 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-08-19T14:16:05.363 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-08-19T14:16:03.945511+0000", 2024-08-19T14:16:05.363 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-08-19T14:16:05.363 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.363 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.363 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 5, 2024-08-19T14:16:05.363 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-08-19T14:16:05.363 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-08-19T14:16:05.364 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-08-19T14:16:05.364 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-08-19T14:16:05.364 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-08-19T14:16:05.364 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-08-19T14:15:49.417209+0000", 2024-08-19T14:16:05.364 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-08-19T14:16:05.364 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.364 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.364 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 6, 2024-08-19T14:16:05.364 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-08-19T14:16:05.364 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-08-19T14:16:05.364 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-08-19T14:16:05.365 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-08-19T14:16:05.365 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 65536, 2024-08-19T14:16:05.365 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-08-19T14:15:37.737868+0000", 2024-08-19T14:16:05.365 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-08-19T14:16:05.365 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.365 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.365 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 7, 2024-08-19T14:16:05.365 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-08-19T14:16:05.365 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-08-19T14:16:05.365 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-08-19T14:16:05.365 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-08-19T14:16:05.366 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-08-19T14:16:05.366 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-08-19T14:15:46.985559+0000", 2024-08-19T14:16:05.366 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-08-19T14:16:05.366 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.366 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.366 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 8, 2024-08-19T14:16:05.366 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-08-19T14:16:05.366 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-08-19T14:16:05.366 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-08-19T14:16:05.367 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-08-19T14:16:05.367 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-08-19T14:16:05.367 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-08-19T14:15:31.869033+0000", 2024-08-19T14:16:05.367 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-08-19T14:16:05.367 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.367 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.367 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 9, 2024-08-19T14:16:05.367 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-08-19T14:16:05.367 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-08-19T14:16:05.368 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-08-19T14:16:05.368 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-08-19T14:16:05.368 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-08-19T14:16:05.368 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-08-19T14:15:54.288635+0000", 2024-08-19T14:16:05.368 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-08-19T14:16:05.368 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.368 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.368 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 10, 2024-08-19T14:16:05.368 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-08-19T14:16:05.368 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-08-19T14:16:05.368 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-08-19T14:16:05.369 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-08-19T14:16:05.369 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 65536, 2024-08-19T14:16:05.369 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-08-19T14:15:57.551806+0000", 2024-08-19T14:16:05.369 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-08-19T14:16:05.369 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.369 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.369 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 11, 2024-08-19T14:16:05.369 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-08-19T14:16:05.369 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-08-19T14:16:05.369 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-08-19T14:16:05.369 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-08-19T14:16:05.370 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-08-19T14:16:05.370 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-08-19T14:15:42.946576+0000", 2024-08-19T14:16:05.370 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-08-19T14:16:05.370 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:05.370 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-08-19T14:16:05.370 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_upmap": [], 2024-08-19T14:16:05.370 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_upmap_items": [], 2024-08-19T14:16:05.370 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_upmap_primaries": [], 2024-08-19T14:16:05.370 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_temp": [ 2024-08-19T14:16:05.370 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:05.370 INFO:teuthology.orchestra.run.smithi031.stdout: "pgid": "2.e", 2024-08-19T14:16:05.370 INFO:teuthology.orchestra.run.smithi031.stdout: "osds": [ 2024-08-19T14:16:05.371 INFO:teuthology.orchestra.run.smithi031.stdout: 2, 2024-08-19T14:16:05.371 INFO:teuthology.orchestra.run.smithi031.stdout: 7, 2024-08-19T14:16:05.371 INFO:teuthology.orchestra.run.smithi031.stdout: 9 2024-08-19T14:16:05.371 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:05.371 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:05.371 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-08-19T14:16:05.371 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_temp": [], 2024-08-19T14:16:05.371 INFO:teuthology.orchestra.run.smithi031.stdout: "blocklist": { 2024-08-19T14:16:05.371 INFO:teuthology.orchestra.run.smithi031.stdout: "172.21.15.31:0/1451745969": "2024-08-20T14:06:13.660657+0000", 2024-08-19T14:16:05.371 INFO:teuthology.orchestra.run.smithi031.stdout: "172.21.15.31:0/1515698918": "2024-08-20T14:06:13.660657+0000", 2024-08-19T14:16:05.371 INFO:teuthology.orchestra.run.smithi031.stdout: "172.21.15.31:6801/3297539242": "2024-08-20T14:06:13.660657+0000", 2024-08-19T14:16:05.372 INFO:teuthology.orchestra.run.smithi031.stdout: "172.21.15.31:6800/3297539242": "2024-08-20T14:06:13.660657+0000", 2024-08-19T14:16:05.372 INFO:teuthology.orchestra.run.smithi031.stdout: "172.21.15.31:0/2217423818": "2024-08-20T14:05:48.827576+0000", 2024-08-19T14:16:05.372 INFO:teuthology.orchestra.run.smithi031.stdout: "172.21.15.31:0/3353498281": "2024-08-20T14:06:13.660657+0000", 2024-08-19T14:16:05.372 INFO:teuthology.orchestra.run.smithi031.stdout: "172.21.15.31:0/4013594220": "2024-08-20T14:05:48.827576+0000", 2024-08-19T14:16:05.372 INFO:teuthology.orchestra.run.smithi031.stdout: "172.21.15.31:0/1654518065": "2024-08-20T14:05:48.827576+0000", 2024-08-19T14:16:05.372 INFO:teuthology.orchestra.run.smithi031.stdout: "172.21.15.31:6801/2368821258": "2024-08-20T14:05:48.827576+0000", 2024-08-19T14:16:05.372 INFO:teuthology.orchestra.run.smithi031.stdout: "172.21.15.31:6800/2368821258": "2024-08-20T14:05:48.827576+0000" 2024-08-19T14:16:05.372 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.372 INFO:teuthology.orchestra.run.smithi031.stdout: "range_blocklist": {}, 2024-08-19T14:16:05.372 INFO:teuthology.orchestra.run.smithi031.stdout: "erasure_code_profiles": { 2024-08-19T14:16:05.372 INFO:teuthology.orchestra.run.smithi031.stdout: "default": { 2024-08-19T14:16:05.373 INFO:teuthology.orchestra.run.smithi031.stdout: "crush-failure-domain": "osd", 2024-08-19T14:16:05.373 INFO:teuthology.orchestra.run.smithi031.stdout: "k": "2", 2024-08-19T14:16:05.373 INFO:teuthology.orchestra.run.smithi031.stdout: "m": "1", 2024-08-19T14:16:05.373 INFO:teuthology.orchestra.run.smithi031.stdout: "plugin": "jerasure", 2024-08-19T14:16:05.373 INFO:teuthology.orchestra.run.smithi031.stdout: "technique": "reed_sol_van" 2024-08-19T14:16:05.373 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:05.373 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:05.373 INFO:teuthology.orchestra.run.smithi031.stdout: "removed_snaps_queue": [], 2024-08-19T14:16:05.373 INFO:teuthology.orchestra.run.smithi031.stdout: "new_removed_snaps": [], 2024-08-19T14:16:05.373 INFO:teuthology.orchestra.run.smithi031.stdout: "new_purged_snaps": [], 2024-08-19T14:16:05.373 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_node_flags": {}, 2024-08-19T14:16:05.373 INFO:teuthology.orchestra.run.smithi031.stdout: "device_class_flags": {}, 2024-08-19T14:16:05.374 INFO:teuthology.orchestra.run.smithi031.stdout: "stretch_mode": { 2024-08-19T14:16:05.374 INFO:teuthology.orchestra.run.smithi031.stdout: "stretch_mode_enabled": false, 2024-08-19T14:16:05.374 INFO:teuthology.orchestra.run.smithi031.stdout: "stretch_bucket_count": 0, 2024-08-19T14:16:05.374 INFO:teuthology.orchestra.run.smithi031.stdout: "degraded_stretch_mode": 0, 2024-08-19T14:16:05.374 INFO:teuthology.orchestra.run.smithi031.stdout: "recovering_stretch_mode": 0, 2024-08-19T14:16:05.374 INFO:teuthology.orchestra.run.smithi031.stdout: "stretch_mode_bucket": 0 2024-08-19T14:16:05.374 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:05.374 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:16:05.396 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:05.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:05.397 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:16:05.498 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:16:05.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:05 smithi032 ceph-mon[20004]: 2.b scrub starts 2024-08-19T14:16:05.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:05 smithi032 ceph-mon[20004]: 1.9 deep-scrub starts 2024-08-19T14:16:05.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:05 smithi032 ceph-mon[20004]: 1.9 deep-scrub ok 2024-08-19T14:16:05.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:05 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:16:05.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:05 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:16:05.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:05 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:16:05.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:05 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:16:05.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:05 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:16:05.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:05 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:16:05.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:05 smithi032 ceph-mon[20004]: osdmap e196: 12 total, 12 up, 10 in 2024-08-19T14:16:05.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:05 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:16:05.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:05 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:16:05.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:05 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:16:05.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:05 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:16:05.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:05 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:16:05.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:05.528 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:05.528 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:16:05.629 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:16:05.659 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:05.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:05.660 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:16:05.761 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:16:05.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:05.793 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:05.794 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:16:05.890 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-19T14:16:05.895 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:16:05.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:05.973 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:05.973 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:16:06.074 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:16:06.112 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:06.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:06.114 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:16:06.215 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:16:06.237 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:06.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:06.238 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:16:06.240 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:06 smithi031 ceph-mon[16336]: 2.b scrub ok 2024-08-19T14:16:06.240 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:06 smithi031 ceph-mon[16336]: 2.5 scrub starts 2024-08-19T14:16:06.240 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:06 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:16:06.240 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:06 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:16:06.240 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:06 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2556243735' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-08-19T14:16:06.240 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:06 smithi031 ceph-mon[16336]: osdmap e197: 12 total, 12 up, 10 in 2024-08-19T14:16:06.284 INFO:tasks.thrashosds.thrasher:j is {'epoch': 196, 'fsid': 'ed7d1c6e-5e33-11ef-bccf-c7b262605968', 'created': '2024-08-19T14:05:22.842169+0000', 'modified': '2024-08-19T14:16:04.019970+0000', 'last_up_change': '2024-08-19T14:11:39.491234+0000', 'last_in_change': '2024-08-19T14:13:18.699388+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 27, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 2, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-08-19T14:08:15.966221+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '112', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '110', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3.2899999618530273, 'score_stable': 3.2899999618530273, 'optimal_score': 0.8299999833106995, 'raw_score_acting': 2.7300000190734863, 'raw_score_stable': 2.7300000190734863, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'unique_pool_0', 'create_time': '2024-08-19T14:12:41.289434+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '136', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.2699999809265137, 'score_stable': 2.2699999809265137, 'optimal_score': 0.8299999833106995, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': 'd7a0fff3-23c6-4ee3-a275-a844862aabfb', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 193, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6802', 'nonce': 1108945097}, {'type': 'v1', 'addr': '172.21.15.31:6803', 'nonce': 1108945097}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6804', 'nonce': 1108945097}, {'type': 'v1', 'addr': '172.21.15.31:6805', 'nonce': 1108945097}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6808', 'nonce': 1108945097}, {'type': 'v1', 'addr': '172.21.15.31:6809', 'nonce': 1108945097}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6806', 'nonce': 1108945097}, {'type': 'v1', 'addr': '172.21.15.31:6807', 'nonce': 1108945097}]}, 'public_addr': '172.21.15.31:6803/1108945097', 'cluster_addr': '172.21.15.31:6805/1108945097', 'heartbeat_back_addr': '172.21.15.31:6809/1108945097', 'heartbeat_front_addr': '172.21.15.31:6807/1108945097', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '145d5bb4-74dd-440f-bcc9-7e9ebda9c5d6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 146, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6810', 'nonce': 3236186806}, {'type': 'v1', 'addr': '172.21.15.31:6811', 'nonce': 3236186806}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6812', 'nonce': 3236186806}, {'type': 'v1', 'addr': '172.21.15.31:6813', 'nonce': 3236186806}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6816', 'nonce': 3236186806}, {'type': 'v1', 'addr': '172.21.15.31:6817', 'nonce': 3236186806}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6814', 'nonce': 3236186806}, {'type': 'v1', 'addr': '172.21.15.31:6815', 'nonce': 3236186806}]}, 'public_addr': '172.21.15.31:6811/3236186806', 'cluster_addr': '172.21.15.31:6813/3236186806', 'heartbeat_back_addr': '172.21.15.31:6817/3236186806', 'heartbeat_front_addr': '172.21.15.31:6815/3236186806', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '376c18d0-4768-46a0-bb60-e79c88ca909c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 191, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6818', 'nonce': 4045553387}, {'type': 'v1', 'addr': '172.21.15.31:6819', 'nonce': 4045553387}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6820', 'nonce': 4045553387}, {'type': 'v1', 'addr': '172.21.15.31:6821', 'nonce': 4045553387}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6824', 'nonce': 4045553387}, {'type': 'v1', 'addr': '172.21.15.31:6825', 'nonce': 4045553387}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6822', 'nonce': 4045553387}, {'type': 'v1', 'addr': '172.21.15.31:6823', 'nonce': 4045553387}]}, 'public_addr': '172.21.15.31:6819/4045553387', 'cluster_addr': '172.21.15.31:6821/4045553387', 'heartbeat_back_addr': '172.21.15.31:6825/4045553387', 'heartbeat_front_addr': '172.21.15.31:6823/4045553387', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'a787ea9f-b9ba-450f-b388-614cfba90634', 'up': 1, 'in': 1, 'weight': 0.6437225341796875, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 129, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6826', 'nonce': 2288082716}, {'type': 'v1', 'addr': '172.21.15.31:6827', 'nonce': 2288082716}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6828', 'nonce': 2288082716}, {'type': 'v1', 'addr': '172.21.15.31:6829', 'nonce': 2288082716}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6832', 'nonce': 2288082716}, {'type': 'v1', 'addr': '172.21.15.31:6833', 'nonce': 2288082716}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6830', 'nonce': 2288082716}, {'type': 'v1', 'addr': '172.21.15.31:6831', 'nonce': 2288082716}]}, 'public_addr': '172.21.15.31:6827/2288082716', 'cluster_addr': '172.21.15.31:6829/2288082716', 'heartbeat_back_addr': '172.21.15.31:6833/2288082716', 'heartbeat_front_addr': '172.21.15.31:6831/2288082716', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '7859751a-12f5-4a57-a332-435360479d53', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6800', 'nonce': 3934739984}, {'type': 'v1', 'addr': '172.21.15.32:6801', 'nonce': 3934739984}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6802', 'nonce': 3934739984}, {'type': 'v1', 'addr': '172.21.15.32:6803', 'nonce': 3934739984}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6806', 'nonce': 3934739984}, {'type': 'v1', 'addr': '172.21.15.32:6807', 'nonce': 3934739984}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6804', 'nonce': 3934739984}, {'type': 'v1', 'addr': '172.21.15.32:6805', 'nonce': 3934739984}]}, 'public_addr': '172.21.15.32:6801/3934739984', 'cluster_addr': '172.21.15.32:6803/3934739984', 'heartbeat_back_addr': '172.21.15.32:6807/3934739984', 'heartbeat_front_addr': '172.21.15.32:6805/3934739984', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '017f3eef-ac60-4b82-8350-919017cb4ccd', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 40, 'up_thru': 174, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6808', 'nonce': 526388985}, {'type': 'v1', 'addr': '172.21.15.32:6809', 'nonce': 526388985}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6810', 'nonce': 526388985}, {'type': 'v1', 'addr': '172.21.15.32:6811', 'nonce': 526388985}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6814', 'nonce': 526388985}, {'type': 'v1', 'addr': '172.21.15.32:6815', 'nonce': 526388985}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6812', 'nonce': 526388985}, {'type': 'v1', 'addr': '172.21.15.32:6813', 'nonce': 526388985}]}, 'public_addr': '172.21.15.32:6809/526388985', 'cluster_addr': '172.21.15.32:6811/526388985', 'heartbeat_back_addr': '172.21.15.32:6815/526388985', 'heartbeat_front_addr': '172.21.15.32:6813/526388985', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '44aa8229-0268-49d7-bd57-3bf1f384e5e6', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 48, 'up_thru': 126, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6816', 'nonce': 1131035844}, {'type': 'v1', 'addr': '172.21.15.32:6817', 'nonce': 1131035844}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6818', 'nonce': 1131035844}, {'type': 'v1', 'addr': '172.21.15.32:6819', 'nonce': 1131035844}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6822', 'nonce': 1131035844}, {'type': 'v1', 'addr': '172.21.15.32:6823', 'nonce': 1131035844}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6820', 'nonce': 1131035844}, {'type': 'v1', 'addr': '172.21.15.32:6821', 'nonce': 1131035844}]}, 'public_addr': '172.21.15.32:6817/1131035844', 'cluster_addr': '172.21.15.32:6819/1131035844', 'heartbeat_back_addr': '172.21.15.32:6823/1131035844', 'heartbeat_front_addr': '172.21.15.32:6821/1131035844', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '263433e5-70b6-4192-86ff-dc2ad95927eb', 'up': 1, 'in': 1, 'weight': 0.95001220703125, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 57, 'up_thru': 132, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6824', 'nonce': 2014939947}, {'type': 'v1', 'addr': '172.21.15.32:6825', 'nonce': 2014939947}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6826', 'nonce': 2014939947}, {'type': 'v1', 'addr': '172.21.15.32:6827', 'nonce': 2014939947}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6830', 'nonce': 2014939947}, {'type': 'v1', 'addr': '172.21.15.32:6831', 'nonce': 2014939947}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6828', 'nonce': 2014939947}, {'type': 'v1', 'addr': '172.21.15.32:6829', 'nonce': 2014939947}]}, 'public_addr': '172.21.15.32:6825/2014939947', 'cluster_addr': '172.21.15.32:6827/2014939947', 'heartbeat_back_addr': '172.21.15.32:6831/2014939947', 'heartbeat_front_addr': '172.21.15.32:6829/2014939947', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': '2bbcad36-0c74-4528-9fb9-a8812afa1662', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 65, 'up_thru': 102, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6800', 'nonce': 1445140892}, {'type': 'v1', 'addr': '172.21.15.45:6801', 'nonce': 1445140892}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6802', 'nonce': 1445140892}, {'type': 'v1', 'addr': '172.21.15.45:6803', 'nonce': 1445140892}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6806', 'nonce': 1445140892}, {'type': 'v1', 'addr': '172.21.15.45:6807', 'nonce': 1445140892}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6804', 'nonce': 1445140892}, {'type': 'v1', 'addr': '172.21.15.45:6805', 'nonce': 1445140892}]}, 'public_addr': '172.21.15.45:6801/1445140892', 'cluster_addr': '172.21.15.45:6803/1445140892', 'heartbeat_back_addr': '172.21.15.45:6807/1445140892', 'heartbeat_front_addr': '172.21.15.45:6805/1445140892', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': 'c8ee4277-3bf9-46bf-9cc0-152c1c4b9508', 'up': 1, 'in': 1, 'weight': 0.95001220703125, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 74, 'up_thru': 129, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6808', 'nonce': 1213787169}, {'type': 'v1', 'addr': '172.21.15.45:6809', 'nonce': 1213787169}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6810', 'nonce': 1213787169}, {'type': 'v1', 'addr': '172.21.15.45:6811', 'nonce': 1213787169}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6814', 'nonce': 1213787169}, {'type': 'v1', 'addr': '172.21.15.45:6815', 'nonce': 1213787169}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6812', 'nonce': 1213787169}, {'type': 'v1', 'addr': '172.21.15.45:6813', 'nonce': 1213787169}]}, 'public_addr': '172.21.15.45:6809/1213787169', 'cluster_addr': '172.21.15.45:6811/1213787169', 'heartbeat_back_addr': '172.21.15.45:6815/1213787169', 'heartbeat_front_addr': '172.21.15.45:6813/1213787169', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '29031395-5f5c-40f8-ac90-9f6ac48e5c9f', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 80, 'up_thru': 115, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6816', 'nonce': 2461824621}, {'type': 'v1', 'addr': '172.21.15.45:6817', 'nonce': 2461824621}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6818', 'nonce': 2461824621}, {'type': 'v1', 'addr': '172.21.15.45:6819', 'nonce': 2461824621}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6822', 'nonce': 2461824621}, {'type': 'v1', 'addr': '172.21.15.45:6823', 'nonce': 2461824621}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6820', 'nonce': 2461824621}, {'type': 'v1', 'addr': '172.21.15.45:6821', 'nonce': 2461824621}]}, 'public_addr': '172.21.15.45:6817/2461824621', 'cluster_addr': '172.21.15.45:6819/2461824621', 'heartbeat_back_addr': '172.21.15.45:6823/2461824621', 'heartbeat_front_addr': '172.21.15.45:6821/2461824621', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': '5ef0db4b-0fef-4ea0-a19b-d3677e082c29', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 86, 'up_thru': 115, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6824', 'nonce': 2438396298}, {'type': 'v1', 'addr': '172.21.15.45:6825', 'nonce': 2438396298}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6826', 'nonce': 2438396298}, {'type': 'v1', 'addr': '172.21.15.45:6827', 'nonce': 2438396298}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6830', 'nonce': 2438396298}, {'type': 'v1', 'addr': '172.21.15.45:6831', 'nonce': 2438396298}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6828', 'nonce': 2438396298}, {'type': 'v1', 'addr': '172.21.15.45:6829', 'nonce': 2438396298}]}, 'public_addr': '172.21.15.45:6825/2438396298', 'cluster_addr': '172.21.15.45:6827/2438396298', 'heartbeat_back_addr': '172.21.15.45:6831/2438396298', 'heartbeat_front_addr': '172.21.15.45:6829/2438396298', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-19T14:15:56.929307+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-19T14:15:55.638039+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-19T14:15:58.904876+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-19T14:14:44.877317+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-19T14:16:03.945511+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-19T14:15:49.417209+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-08-19T14:15:37.737868+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-19T14:15:46.985559+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-19T14:15:31.869033+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-19T14:15:54.288635+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-08-19T14:15:57.551806+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-19T14:15:42.946576+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '2.e', 'osds': [2, 7, 9]}], 'primary_temp': [], 'blocklist': {'172.21.15.31:0/1451745969': '2024-08-20T14:06:13.660657+0000', '172.21.15.31:0/1515698918': '2024-08-20T14:06:13.660657+0000', '172.21.15.31:6801/3297539242': '2024-08-20T14:06:13.660657+0000', '172.21.15.31:6800/3297539242': '2024-08-20T14:06:13.660657+0000', '172.21.15.31:0/2217423818': '2024-08-20T14:05:48.827576+0000', '172.21.15.31:0/3353498281': '2024-08-20T14:06:13.660657+0000', '172.21.15.31:0/4013594220': '2024-08-20T14:05:48.827576+0000', '172.21.15.31:0/1654518065': '2024-08-20T14:05:48.827576+0000', '172.21.15.31:6801/2368821258': '2024-08-20T14:05:48.827576+0000', '172.21.15.31:6800/2368821258': '2024-08-20T14:05:48.827576+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-08-19T14:16:06.285 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-19T14:16:06.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:06 smithi045 ceph-mon[20143]: 2.b scrub ok 2024-08-19T14:16:06.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:06 smithi045 ceph-mon[20143]: 2.5 scrub starts 2024-08-19T14:16:06.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:06 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:16:06.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:06 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:16:06.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:06 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2556243735' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-08-19T14:16:06.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:06 smithi045 ceph-mon[20143]: osdmap e197: 12 total, 12 up, 10 in 2024-08-19T14:16:06.339 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:16:06.362 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:06.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:06.364 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:16:06.433 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:06 smithi032 ceph-mon[20004]: 2.b scrub ok 2024-08-19T14:16:06.433 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:06 smithi032 ceph-mon[20004]: 2.5 scrub starts 2024-08-19T14:16:06.433 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:06 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:16:06.433 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:06 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:16:06.433 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:06 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2556243735' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-08-19T14:16:06.433 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:06 smithi032 ceph-mon[20004]: osdmap e197: 12 total, 12 up, 10 in 2024-08-19T14:16:06.465 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:16:06.487 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:06.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:06.489 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:16:06.526 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:16:06.532 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:16:06.533 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:16:06.583 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:16:06.583 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:16:06.590 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:16:06.632 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:16:06.632 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:16:06.644 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:06.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:06.645 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:16:06.674 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:16:06.675 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:16:06.707 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:16:06.741 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:16:06.741 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:16:06.747 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:16:06.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:06.771 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:06.772 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:16:06.776 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:16:06.776 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:16:06.810 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:16:06.810 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:16:06.845 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:16:06.845 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:16:06.874 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:16:06.880 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:16:06.880 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:16:06.903 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:06.904 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:16:06.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:06.905 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:16:06.905 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:16:06.935 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:16:06.935 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:16:07.006 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:16:07.038 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:07.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:07.039 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:16:07.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:07 smithi031 ceph-mon[16336]: 2.5 scrub ok 2024-08-19T14:16:07.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:07 smithi031 ceph-mon[16336]: 2.f scrub starts 2024-08-19T14:16:07.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:07 smithi031 ceph-mon[16336]: pgmap v483: 27 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfilling, 25 active+clean; 2.2 GiB data, 7.8 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 742 op/s; 2032/109038 objects misplaced (1.864%); 3.9 MiB/s, 61 objects/s recovering 2024-08-19T14:16:07.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:07 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/757258114' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:16:07.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:07 smithi031 ceph-mon[16336]: 1.3 scrub starts 2024-08-19T14:16:07.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:07 smithi031 ceph-mon[16336]: 1.3 scrub ok 2024-08-19T14:16:07.140 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:16:07.174 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:07.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:07.175 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:16:07.276 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:16:07.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:07 smithi045 ceph-mon[20143]: 2.5 scrub ok 2024-08-19T14:16:07.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:07 smithi045 ceph-mon[20143]: 2.f scrub starts 2024-08-19T14:16:07.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:07 smithi045 ceph-mon[20143]: pgmap v483: 27 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfilling, 25 active+clean; 2.2 GiB data, 7.8 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 742 op/s; 2032/109038 objects misplaced (1.864%); 3.9 MiB/s, 61 objects/s recovering 2024-08-19T14:16:07.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:07 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/757258114' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:16:07.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:07 smithi045 ceph-mon[20143]: 1.3 scrub starts 2024-08-19T14:16:07.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:07 smithi045 ceph-mon[20143]: 1.3 scrub ok 2024-08-19T14:16:07.301 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:07.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:07.303 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:16:07.404 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:16:07.440 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:07.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:07.441 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:16:07.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:07 smithi032 ceph-mon[20004]: 2.5 scrub ok 2024-08-19T14:16:07.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:07 smithi032 ceph-mon[20004]: 2.f scrub starts 2024-08-19T14:16:07.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:07 smithi032 ceph-mon[20004]: pgmap v483: 27 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfilling, 25 active+clean; 2.2 GiB data, 7.8 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 742 op/s; 2032/109038 objects misplaced (1.864%); 3.9 MiB/s, 61 objects/s recovering 2024-08-19T14:16:07.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:07 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/757258114' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:16:07.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:07 smithi032 ceph-mon[20004]: 1.3 scrub starts 2024-08-19T14:16:07.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:07 smithi032 ceph-mon[20004]: 1.3 scrub ok 2024-08-19T14:16:07.542 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:16:07.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:07.571 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:07.572 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:16:07.673 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:16:07.704 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:07.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:07.706 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:16:07.807 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:16:07.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:07.836 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:07.836 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:16:07.937 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:16:07.960 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:07.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:07.961 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:16:08.062 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:16:08.087 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:08.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:08.089 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:16:08.190 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:16:08.213 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:08.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:08.214 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:16:08.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:08 smithi045 ceph-mon[20143]: 2.f scrub ok 2024-08-19T14:16:08.315 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:16:08.330 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:08 smithi031 ceph-mon[16336]: 2.f scrub ok 2024-08-19T14:16:08.341 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:08.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:08.342 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:16:08.443 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:16:08.479 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:08.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:08.480 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:16:08.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:08 smithi032 ceph-mon[20004]: 2.f scrub ok 2024-08-19T14:16:08.581 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:16:08.611 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:08.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:08.612 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:16:08.713 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:16:08.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:08.738 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:08.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:16:08.838 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:16:08.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:08.878 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:08.878 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:16:08.979 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:16:09.015 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:09.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:09.016 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:16:09.028 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:09 smithi031 ceph-mon[16336]: pgmap v484: 27 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 25 active+clean; 2.1 GiB data, 7.7 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 750 op/s; 1209/104934 objects misplaced (1.152%); 6.5 MiB/s, 103 objects/s recovering 2024-08-19T14:16:09.030 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:16:09.066 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:16:09.066 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:16:09.116 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:16:09.144 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:09.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:09.145 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:16:09.246 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:16:09.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:09.283 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:09.283 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:16:09.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:09 smithi045 ceph-mon[20143]: pgmap v484: 27 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 25 active+clean; 2.1 GiB data, 7.7 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 750 op/s; 1209/104934 objects misplaced (1.152%); 6.5 MiB/s, 103 objects/s recovering 2024-08-19T14:16:09.384 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:16:09.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:09.415 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:09.415 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:16:09.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:09 smithi032 ceph-mon[20004]: pgmap v484: 27 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 25 active+clean; 2.1 GiB data, 7.7 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 750 op/s; 1209/104934 objects misplaced (1.152%); 6.5 MiB/s, 103 objects/s recovering 2024-08-19T14:16:09.516 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:16:09.550 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:09.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:09.552 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:16:09.653 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:16:09.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:09.694 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:09.694 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:16:09.795 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:16:09.833 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:09.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:09.834 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:16:09.935 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:16:09.964 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:09.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:09.965 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:16:10.066 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:16:10.089 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:10.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:10.090 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:16:10.104 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-08-19T14:16:10.191 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:16:10.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:10.222 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:10.222 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:16:10.323 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:16:10.347 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:10.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:10.348 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:16:10.359 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:16:10.448 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:16:10.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:10.484 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:10.485 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:16:10.586 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:16:10.623 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:10.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:10.624 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:16:10.725 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:16:10.751 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:10.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:10.752 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:16:10.820 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:16:10.853 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:16:10.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:10.906 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:10.906 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:16:10.970 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:16:11.006 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:16:11.029 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:11.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:11.030 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:16:11.039 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is set 2024-08-19T14:16:11.131 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:16:11.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:11.156 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:11.156 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:16:11.256 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:16:11.282 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:11.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:11.283 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:16:11.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:11 smithi045 ceph-mon[20143]: pgmap v485: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 2.0 GiB data, 7.2 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 1.04k op/s; 1209/97353 objects misplaced (1.242%); 5.7 MiB/s, 91 objects/s recovering 2024-08-19T14:16:11.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:11 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/4049278991' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:16:11.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:11 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:16:11.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:11 smithi031 ceph-mon[16336]: pgmap v485: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 2.0 GiB data, 7.2 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 1.04k op/s; 1209/97353 objects misplaced (1.242%); 5.7 MiB/s, 91 objects/s recovering 2024-08-19T14:16:11.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:11 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/4049278991' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:16:11.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:11 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:16:11.321 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:16:11.322 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-08-19T14:16:11.386 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:16:11.431 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:11.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:11.432 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:16:11.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:11 smithi032 ceph-mon[20004]: pgmap v485: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 2.0 GiB data, 7.2 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 1.04k op/s; 1209/97353 objects misplaced (1.242%); 5.7 MiB/s, 91 objects/s recovering 2024-08-19T14:16:11.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:11 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/4049278991' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:16:11.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:11 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:16:11.533 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:16:11.556 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:11.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:11.557 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:16:11.658 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:16:11.682 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:11.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:11.683 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:16:11.784 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:16:11.824 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:11.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:11.825 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:16:11.926 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:16:11.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:11.953 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:11.953 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:16:12.046 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:16:12.054 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:16:12.055 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:16:12.055 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:16:12.123 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:12.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:12.124 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:16:12.217 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:12 smithi031 ceph-mon[16336]: 1.7 deep-scrub starts 2024-08-19T14:16:12.217 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:12 smithi031 ceph-mon[16336]: 1.7 deep-scrub ok 2024-08-19T14:16:12.217 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:12 smithi031 ceph-mon[16336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:16:12.217 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:12 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:16:12.217 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:12 smithi031 ceph-mon[16336]: osdmap e198: 12 total, 12 up, 10 in 2024-08-19T14:16:12.217 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:12 smithi031 ceph-mon[16336]: from='client.25001 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:16:12.225 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:16:12.253 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:12.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:12.254 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:16:12.355 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:16:12.378 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:12.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:12.380 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:16:12.481 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:16:12.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:12 smithi032 ceph-mon[20004]: 1.7 deep-scrub starts 2024-08-19T14:16:12.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:12 smithi032 ceph-mon[20004]: 1.7 deep-scrub ok 2024-08-19T14:16:12.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:12 smithi032 ceph-mon[20004]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:16:12.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:12 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:16:12.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:12 smithi032 ceph-mon[20004]: osdmap e198: 12 total, 12 up, 10 in 2024-08-19T14:16:12.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:12 smithi032 ceph-mon[20004]: from='client.25001 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:16:12.518 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:12.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:12.519 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:16:12.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:12 smithi045 ceph-mon[20143]: 1.7 deep-scrub starts 2024-08-19T14:16:12.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:12 smithi045 ceph-mon[20143]: 1.7 deep-scrub ok 2024-08-19T14:16:12.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:12 smithi045 ceph-mon[20143]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:16:12.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:12 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:16:12.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:12 smithi045 ceph-mon[20143]: osdmap e198: 12 total, 12 up, 10 in 2024-08-19T14:16:12.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:12 smithi045 ceph-mon[20143]: from='client.25001 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:16:12.619 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:16:12.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:12.652 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:12.653 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:16:12.754 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:16:12.816 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":486,"stamp":"2024-08-19T14:16:11.040008+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2085618232,"num_objects":31818,"num_object_clones":0,"num_object_copies":95454,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1209,"num_objects_unfound":0,"num_objects_dirty":31818,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":214742,"num_write_kb":7889793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23037,"num_bytes_recovered":1471283200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":162392,"ondisk_log_size":162392,"up":81,"acting":81,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":7360436,"kb_used_data":6247376,"kb_used_omap":15,"kb_used_meta":1112688,"kb_avail":930091084,"statfs":{"total":959950356480,"available":952413270016,"internally_reserved":0,"allocated":6397313024,"data_stored":6387620065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15948,"internal_metadata":1139392948},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,3,3,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-391184384,"num_objects":-5969,"num_object_clones":0,"num_object_copies":-17907,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-823,"num_objects_unfound":0,"num_objects_dirty":-5969,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5969,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":620,"num_bytes_recovered":40632320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.252117"},"pg_stats":[{"pgid":"2.f","version":"197'13596","reported_seq":13907,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-08-19T14:16:09.664838+0000","last_change":"2024-08-19T14:16:06.289451+0000","last_active":"2024-08-19T14:16:09.664838+0000","last_peered":"2024-08-19T14:16:09.664838+0000","last_clean":"2024-08-19T14:16:09.664838+0000","last_became_active":"2024-08-19T14:12:43.417591+0000","last_became_peered":"2024-08-19T14:12:43.417591+0000","last_unstale":"2024-08-19T14:16:09.664838+0000","last_undegraded":"2024-08-19T14:16:09.664838+0000","last_fullsized":"2024-08-19T14:16:09.664838+0000","mapping_epoch":102,"log_start":"123'3500","ondisk_log_start":"123'3500","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"197'13388","last_scrub_stamp":"2024-08-19T14:16:06.289300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:16:06.289300+0000","objects_scrubbed":2129,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:17:08.304647+0000","scrub_duration":507,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125173760,"num_objects":1910,"num_object_clones":0,"num_object_copies":5730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13596,"num_write_kb":496192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.e","version":"197'13265","reported_seq":22999,"reported_epoch":197,"state":"active+remapped+backfilling","last_fresh":"2024-08-19T14:16:06.293305+0000","last_change":"2024-08-19T14:15:58.435193+0000","last_active":"2024-08-19T14:16:06.293305+0000","last_peered":"2024-08-19T14:16:06.293305+0000","last_clean":"2024-08-19T14:15:55.934689+0000","last_became_active":"2024-08-19T14:15:58.393026+0000","last_became_peered":"2024-08-19T14:15:58.393026+0000","last_unstale":"2024-08-19T14:16:06.293305+0000","last_undegraded":"2024-08-19T14:16:06.293305+0000","last_fullsized":"2024-08-19T14:16:06.293305+0000","mapping_epoch":191,"log_start":"121'3200","ondisk_log_start":"121'3200","created":102,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:12:41.713780+0000","objects_scrubbed":3824,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":137297920,"num_objects":2095,"num_object_clones":0,"num_object_copies":6285,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1209,"num_objects_unfound":0,"num_objects_dirty":2095,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13265,"num_write_kb":491520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4736,"num_bytes_recovered":310378496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,11],"acting":[2,7,9],"avail_no_missing":["2","7","9"],"object_location_counts":[{"shards":"2,7,9","objects":2095}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"197'13467","reported_seq":13794,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-08-19T14:16:08.251305+0000","last_change":"2024-08-19T14:15:35.692830+0000","last_active":"2024-08-19T14:16:08.251305+0000","last_peered":"2024-08-19T14:16:08.251305+0000","last_clean":"2024-08-19T14:16:08.251305+0000","last_became_active":"2024-08-19T14:12:43.417506+0000","last_became_peered":"2024-08-19T14:12:43.417506+0000","last_unstale":"2024-08-19T14:16:08.251305+0000","last_undegraded":"2024-08-19T14:16:08.251305+0000","last_fullsized":"2024-08-19T14:16:08.251305+0000","mapping_epoch":102,"log_start":"122'3400","ondisk_log_start":"122'3400","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"180'11879","last_scrub_stamp":"2024-08-19T14:15:35.692699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:15:35.692699+0000","objects_scrubbed":3525,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:16:57.995467+0000","scrub_duration":1207,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125567000,"num_objects":1917,"num_object_clones":0,"num_object_copies":5751,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1917,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13467,"num_write_kb":492225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.c","version":"197'13523","reported_seq":13862,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-08-19T14:16:10.509412+0000","last_change":"2024-08-19T14:15:18.768595+0000","last_active":"2024-08-19T14:16:10.509412+0000","last_peered":"2024-08-19T14:16:10.509412+0000","last_clean":"2024-08-19T14:16:10.509412+0000","last_became_active":"2024-08-19T14:12:43.038912+0000","last_became_peered":"2024-08-19T14:12:43.038912+0000","last_unstale":"2024-08-19T14:16:10.509412+0000","last_undegraded":"2024-08-19T14:16:10.509412+0000","last_fullsized":"2024-08-19T14:16:10.509412+0000","mapping_epoch":102,"log_start":"123'3500","ondisk_log_start":"123'3500","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'10846","last_scrub_stamp":"2024-08-19T14:15:18.768342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:15:18.768342+0000","objects_scrubbed":4537,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:16:37.101539+0000","scrub_duration":1760,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":119209984,"num_objects":1819,"num_object_clones":0,"num_object_copies":5457,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13523,"num_write_kb":490944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,1],"acting":[11,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"70'139","reported_seq":594,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-08-19T14:16:05.744934+0000","last_change":"2024-08-19T14:15:59.869341+0000","last_active":"2024-08-19T14:16:05.744934+0000","last_peered":"2024-08-19T14:16:05.744934+0000","last_clean":"2024-08-19T14:16:05.744934+0000","last_became_active":"2024-08-19T14:13:22.729498+0000","last_became_peered":"2024-08-19T14:13:22.729498+0000","last_unstale":"2024-08-19T14:16:05.744934+0000","last_undegraded":"2024-08-19T14:16:05.744934+0000","last_fullsized":"2024-08-19T14:16:05.744934+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:15:59.869266+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:15:59.869266+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:17:29.463231+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,9],"acting":[7,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"197'13337","reported_seq":13661,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-08-19T14:16:06.448748+0000","last_change":"2024-08-19T14:15:33.408438+0000","last_active":"2024-08-19T14:16:06.448748+0000","last_peered":"2024-08-19T14:16:06.448748+0000","last_clean":"2024-08-19T14:16:06.448748+0000","last_became_active":"2024-08-19T14:12:42.728978+0000","last_became_peered":"2024-08-19T14:12:42.728978+0000","last_unstale":"2024-08-19T14:16:06.448748+0000","last_undegraded":"2024-08-19T14:16:06.448748+0000","last_fullsized":"2024-08-19T14:16:06.448748+0000","mapping_epoch":102,"log_start":"121'3300","ondisk_log_start":"121'3300","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"179'11759","last_scrub_stamp":"2024-08-19T14:15:33.408298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:15:33.408298+0000","objects_scrubbed":3675,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:16:50.212363+0000","scrub_duration":864,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":135856128,"num_objects":2073,"num_object_clones":0,"num_object_copies":6219,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2073,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13337,"num_write_kb":493120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.3","version":"70'139","reported_seq":479,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-08-19T14:16:06.679161+0000","last_change":"2024-08-19T14:16:06.679203+0000","last_active":"2024-08-19T14:16:06.679161+0000","last_peered":"2024-08-19T14:16:06.679161+0000","last_clean":"2024-08-19T14:16:06.679161+0000","last_became_active":"2024-08-19T14:13:22.727234+0000","last_became_peered":"2024-08-19T14:13:22.727234+0000","last_unstale":"2024-08-19T14:16:06.679161+0000","last_undegraded":"2024-08-19T14:16:06.679161+0000","last_fullsized":"2024-08-19T14:16:06.679161+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:16:06.679075+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:16:06.679075+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:17:31.357116+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"197'13461","reported_seq":19845,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-08-19T14:16:08.074564+0000","last_change":"2024-08-19T14:15:11.392878+0000","last_active":"2024-08-19T14:16:08.074564+0000","last_peered":"2024-08-19T14:16:08.074564+0000","last_clean":"2024-08-19T14:16:08.074564+0000","last_became_active":"2024-08-19T14:13:32.749994+0000","last_became_peered":"2024-08-19T14:13:32.749994+0000","last_unstale":"2024-08-19T14:16:08.074564+0000","last_undegraded":"2024-08-19T14:16:08.074564+0000","last_fullsized":"2024-08-19T14:16:08.074564+0000","mapping_epoch":132,"log_start":"122'3400","ondisk_log_start":"122'3400","created":102,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"169'10608","last_scrub_stamp":"2024-08-19T14:15:11.392791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:15:11.392791+0000","objects_scrubbed":4851,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:16:30.589490+0000","scrub_duration":894,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":129564672,"num_objects":1977,"num_object_clones":0,"num_object_copies":5931,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13461,"num_write_kb":494016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3024,"num_bytes_recovered":198180864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,11],"acting":[7,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"70'139","reported_seq":424,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-08-19T14:16:05.743553+0000","last_change":"2024-08-19T14:16:00.592808+0000","last_active":"2024-08-19T14:16:05.743553+0000","last_peered":"2024-08-19T14:16:05.743553+0000","last_clean":"2024-08-19T14:16:05.743553+0000","last_became_active":"2024-08-19T14:15:59.993566+0000","last_became_peered":"2024-08-19T14:15:59.993566+0000","last_unstale":"2024-08-19T14:16:05.743553+0000","last_undegraded":"2024-08-19T14:16:05.743553+0000","last_fullsized":"2024-08-19T14:16:05.743553+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:16:00.592678+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:16:00.592678+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:17:07.175192+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,8],"acting":[0,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"197'13401","reported_seq":21202,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-08-19T14:16:06.306370+0000","last_change":"2024-08-19T14:15:46.885929+0000","last_active":"2024-08-19T14:16:06.306370+0000","last_peered":"2024-08-19T14:16:06.306370+0000","last_clean":"2024-08-19T14:16:06.306370+0000","last_became_active":"2024-08-19T14:15:46.858480+0000","last_became_peered":"2024-08-19T14:15:46.858480+0000","last_unstale":"2024-08-19T14:16:06.306370+0000","last_undegraded":"2024-08-19T14:16:06.306370+0000","last_fullsized":"2024-08-19T14:16:06.306370+0000","mapping_epoch":184,"log_start":"122'3400","ondisk_log_start":"122'3400","created":102,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:12:41.713780+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":137953280,"num_objects":2105,"num_object_clones":0,"num_object_copies":6315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2105,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13401,"num_write_kb":496192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3792,"num_bytes_recovered":248512512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,8],"acting":[2,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"70'139","reported_seq":459,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-08-19T14:16:05.736154+0000","last_change":"2024-08-19T14:15:09.350783+0000","last_active":"2024-08-19T14:16:05.736154+0000","last_peered":"2024-08-19T14:16:05.736154+0000","last_clean":"2024-08-19T14:16:05.736154+0000","last_became_active":"2024-08-19T14:13:11.663185+0000","last_became_peered":"2024-08-19T14:13:11.663185+0000","last_unstale":"2024-08-19T14:16:05.736154+0000","last_undegraded":"2024-08-19T14:16:05.736154+0000","last_fullsized":"2024-08-19T14:16:05.736154+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:15:09.350658+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:15:09.350658+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:16:12.339458+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2],"acting":[9,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.2","version":"197'13286","reported_seq":21432,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-08-19T14:16:06.517520+0000","last_change":"2024-08-19T14:15:31.539950+0000","last_active":"2024-08-19T14:16:06.517520+0000","last_peered":"2024-08-19T14:16:06.517520+0000","last_clean":"2024-08-19T14:16:06.517520+0000","last_became_active":"2024-08-19T14:13:58.623348+0000","last_became_peered":"2024-08-19T14:13:58.623348+0000","last_unstale":"2024-08-19T14:16:06.517520+0000","last_undegraded":"2024-08-19T14:16:06.517520+0000","last_fullsized":"2024-08-19T14:16:06.517520+0000","mapping_epoch":142,"log_start":"121'3200","ondisk_log_start":"121'3200","created":102,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"178'11591","last_scrub_stamp":"2024-08-19T14:15:31.539835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:15:31.539835+0000","objects_scrubbed":3877,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:17:01.072608+0000","scrub_duration":986,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":140902400,"num_objects":2150,"num_object_clones":0,"num_object_copies":6450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2150,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13286,"num_write_kb":493952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3902,"num_bytes_recovered":255721472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"70'139","reported_seq":367,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-08-19T14:16:05.740221+0000","last_change":"2024-08-19T14:15:18.660553+0000","last_active":"2024-08-19T14:16:05.740221+0000","last_peered":"2024-08-19T14:16:05.740221+0000","last_clean":"2024-08-19T14:16:05.740221+0000","last_became_active":"2024-08-19T14:15:18.660091+0000","last_became_peered":"2024-08-19T14:15:18.660091+0000","last_unstale":"2024-08-19T14:16:05.740221+0000","last_undegraded":"2024-08-19T14:16:05.740221+0000","last_fullsized":"2024-08-19T14:16:05.740221+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:14:48.488367+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:14:48.488367+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:16:10.475499+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"197'13464","reported_seq":20048,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-08-19T14:16:08.219594+0000","last_change":"2024-08-19T14:15:37.502240+0000","last_active":"2024-08-19T14:16:08.219594+0000","last_peered":"2024-08-19T14:16:08.219594+0000","last_clean":"2024-08-19T14:16:08.219594+0000","last_became_active":"2024-08-19T14:14:04.681118+0000","last_became_peered":"2024-08-19T14:14:04.681118+0000","last_unstale":"2024-08-19T14:16:08.219594+0000","last_undegraded":"2024-08-19T14:16:08.219594+0000","last_fullsized":"2024-08-19T14:16:08.219594+0000","mapping_epoch":146,"log_start":"122'3400","ondisk_log_start":"122'3400","created":102,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"180'11953","last_scrub_stamp":"2024-08-19T14:15:37.501902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:15:37.501902+0000","objects_scrubbed":3536,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:16:44.088914+0000","scrub_duration":1934,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130023424,"num_objects":1984,"num_object_clones":0,"num_object_copies":5952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1984,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13464,"num_write_kb":494336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3123,"num_bytes_recovered":204668928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"70'139","reported_seq":459,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-08-19T14:16:05.736134+0000","last_change":"2024-08-19T14:15:13.428744+0000","last_active":"2024-08-19T14:16:05.736134+0000","last_peered":"2024-08-19T14:16:05.736134+0000","last_clean":"2024-08-19T14:16:05.736134+0000","last_became_active":"2024-08-19T14:13:11.647085+0000","last_became_peered":"2024-08-19T14:13:11.647085+0000","last_unstale":"2024-08-19T14:16:05.736134+0000","last_undegraded":"2024-08-19T14:16:05.736134+0000","last_fullsized":"2024-08-19T14:16:05.736134+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:15:13.428627+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:15:13.428627+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:16:19.387490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,4],"acting":[9,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.5","version":"197'13560","reported_seq":13875,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-08-19T14:16:09.765640+0000","last_change":"2024-08-19T14:16:05.351767+0000","last_active":"2024-08-19T14:16:09.765640+0000","last_peered":"2024-08-19T14:16:09.765640+0000","last_clean":"2024-08-19T14:16:09.765640+0000","last_became_active":"2024-08-19T14:12:43.417616+0000","last_became_peered":"2024-08-19T14:12:43.417616+0000","last_unstale":"2024-08-19T14:16:09.765640+0000","last_undegraded":"2024-08-19T14:16:09.765640+0000","last_fullsized":"2024-08-19T14:16:09.765640+0000","mapping_epoch":102,"log_start":"123'3500","ondisk_log_start":"123'3500","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"196'13313","last_scrub_stamp":"2024-08-19T14:16:05.351209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:16:05.351209+0000","objects_scrubbed":2149,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:17:30.130226+0000","scrub_duration":592,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123600896,"num_objects":1886,"num_object_clones":0,"num_object_copies":5658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13560,"num_write_kb":494272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.5","version":"70'139","reported_seq":460,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-08-19T14:16:05.736781+0000","last_change":"2024-08-19T14:14:49.752154+0000","last_active":"2024-08-19T14:16:05.736781+0000","last_peered":"2024-08-19T14:16:05.736781+0000","last_clean":"2024-08-19T14:16:05.736781+0000","last_became_active":"2024-08-19T14:13:22.727098+0000","last_became_peered":"2024-08-19T14:13:22.727098+0000","last_unstale":"2024-08-19T14:16:05.736781+0000","last_undegraded":"2024-08-19T14:16:05.736781+0000","last_fullsized":"2024-08-19T14:16:05.736781+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:14:49.752072+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:14:49.752072+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"197'14522","reported_seq":14899,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-08-19T14:16:08.192998+0000","last_change":"2024-08-19T14:15:57.834926+0000","last_active":"2024-08-19T14:16:08.192998+0000","last_peered":"2024-08-19T14:16:08.192998+0000","last_clean":"2024-08-19T14:16:08.192998+0000","last_became_active":"2024-08-19T14:12:42.963203+0000","last_became_peered":"2024-08-19T14:12:42.963203+0000","last_unstale":"2024-08-19T14:16:08.192998+0000","last_undegraded":"2024-08-19T14:16:08.192998+0000","last_fullsized":"2024-08-19T14:16:08.192998+0000","mapping_epoch":102,"log_start":"131'4500","ondisk_log_start":"131'4500","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'13968","last_scrub_stamp":"2024-08-19T14:15:57.834750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:15:57.834750+0000","objects_scrubbed":2585,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:17:23.845680+0000","scrub_duration":281,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":133103616,"num_objects":2031,"num_object_clones":0,"num_object_copies":6093,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13301,"num_write_kb":490624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,3],"acting":[1,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"70'139","reported_seq":510,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-08-19T14:16:05.737815+0000","last_change":"2024-08-19T14:14:36.831290+0000","last_active":"2024-08-19T14:16:05.737815+0000","last_peered":"2024-08-19T14:16:05.737815+0000","last_clean":"2024-08-19T14:16:05.737815+0000","last_became_active":"2024-08-19T14:13:05.607447+0000","last_became_peered":"2024-08-19T14:13:05.607447+0000","last_unstale":"2024-08-19T14:16:05.737815+0000","last_undegraded":"2024-08-19T14:16:05.737815+0000","last_fullsized":"2024-08-19T14:16:05.737815+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:14:36.831171+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:14:36.831171+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.7","version":"197'13239","reported_seq":24829,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-08-19T14:16:06.258385+0000","last_change":"2024-08-19T14:15:13.583652+0000","last_active":"2024-08-19T14:16:06.258385+0000","last_peered":"2024-08-19T14:16:06.258385+0000","last_clean":"2024-08-19T14:16:06.258385+0000","last_became_active":"2024-08-19T14:14:49.991243+0000","last_became_peered":"2024-08-19T14:14:49.991243+0000","last_unstale":"2024-08-19T14:16:06.258385+0000","last_undegraded":"2024-08-19T14:16:06.258385+0000","last_fullsized":"2024-08-19T14:16:06.258385+0000","mapping_epoch":164,"log_start":"122'3200","ondisk_log_start":"122'3200","created":102,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"169'10693","last_scrub_stamp":"2024-08-19T14:15:13.583522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:15:13.583522+0000","objects_scrubbed":4764,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:16:37.611856+0000","scrub_duration":1668,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":142802944,"num_objects":2179,"num_object_clones":0,"num_object_copies":6537,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13239,"num_write_kb":493376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4460,"num_bytes_recovered":253820928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.8","version":"197'13531","reported_seq":13869,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-08-19T14:16:08.069493+0000","last_change":"2024-08-19T14:15:13.054824+0000","last_active":"2024-08-19T14:16:08.069493+0000","last_peered":"2024-08-19T14:16:08.069493+0000","last_clean":"2024-08-19T14:16:08.069493+0000","last_became_active":"2024-08-19T14:12:43.038882+0000","last_became_peered":"2024-08-19T14:12:43.038882+0000","last_unstale":"2024-08-19T14:16:08.069493+0000","last_undegraded":"2024-08-19T14:16:08.069493+0000","last_fullsized":"2024-08-19T14:16:08.069493+0000","mapping_epoch":102,"log_start":"122'3500","ondisk_log_start":"122'3500","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"169'10792","last_scrub_stamp":"2024-08-19T14:15:13.054698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:15:13.054698+0000","objects_scrubbed":4745,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:16:31.701146+0000","scrub_duration":1525,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":129564672,"num_objects":1977,"num_object_clones":0,"num_object_copies":5931,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13531,"num_write_kb":496256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,0],"acting":[7,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"70'139","reported_seq":480,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-08-19T14:16:05.736229+0000","last_change":"2024-08-19T14:16:02.818848+0000","last_active":"2024-08-19T14:16:05.736229+0000","last_peered":"2024-08-19T14:16:05.736229+0000","last_clean":"2024-08-19T14:16:05.736229+0000","last_became_active":"2024-08-19T14:13:22.728779+0000","last_became_peered":"2024-08-19T14:13:22.728779+0000","last_unstale":"2024-08-19T14:16:05.736229+0000","last_undegraded":"2024-08-19T14:16:05.736229+0000","last_fullsized":"2024-08-19T14:16:05.736229+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:16:02.818774+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:16:02.818774+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:17:09.255652+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,0],"acting":[9,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"197'13228","reported_seq":13584,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-08-19T14:16:08.176017+0000","last_change":"2024-08-19T14:15:33.182944+0000","last_active":"2024-08-19T14:16:08.176017+0000","last_peered":"2024-08-19T14:16:08.176017+0000","last_clean":"2024-08-19T14:16:08.176017+0000","last_became_active":"2024-08-19T14:12:42.963114+0000","last_became_peered":"2024-08-19T14:12:42.963114+0000","last_unstale":"2024-08-19T14:16:08.176017+0000","last_undegraded":"2024-08-19T14:16:08.176017+0000","last_fullsized":"2024-08-19T14:16:08.176017+0000","mapping_epoch":102,"log_start":"121'3200","ondisk_log_start":"121'3200","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"179'11533","last_scrub_stamp":"2024-08-19T14:15:33.182797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:15:33.182797+0000","objects_scrubbed":3720,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:17:01.807642+0000","scrub_duration":1671,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":129630208,"num_objects":1978,"num_object_clones":0,"num_object_copies":5934,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13228,"num_write_kb":486592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"70'139","reported_seq":593,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-08-19T14:16:05.744826+0000","last_change":"2024-08-19T14:16:03.918123+0000","last_active":"2024-08-19T14:16:05.744826+0000","last_peered":"2024-08-19T14:16:05.744826+0000","last_clean":"2024-08-19T14:16:05.744826+0000","last_became_active":"2024-08-19T14:13:05.607605+0000","last_became_peered":"2024-08-19T14:13:05.607605+0000","last_unstale":"2024-08-19T14:16:05.744826+0000","last_undegraded":"2024-08-19T14:16:05.744826+0000","last_fullsized":"2024-08-19T14:16:05.744826+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:16:03.918046+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:17:20.143217+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"197'13593","reported_seq":13920,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-08-19T14:16:10.562921+0000","last_change":"2024-08-19T14:15:36.099307+0000","last_active":"2024-08-19T14:16:10.562921+0000","last_peered":"2024-08-19T14:16:10.562921+0000","last_clean":"2024-08-19T14:16:10.562921+0000","last_became_active":"2024-08-19T14:12:42.963040+0000","last_became_peered":"2024-08-19T14:12:42.963040+0000","last_unstale":"2024-08-19T14:16:10.562921+0000","last_undegraded":"2024-08-19T14:16:10.562921+0000","last_fullsized":"2024-08-19T14:16:10.562921+0000","mapping_epoch":102,"log_start":"123'3500","ondisk_log_start":"123'3500","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"180'11846","last_scrub_stamp":"2024-08-19T14:15:36.099059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:15:36.099059+0000","objects_scrubbed":3612,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:16:36.324214+0000","scrub_duration":1034,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":120913920,"num_objects":1845,"num_object_clones":0,"num_object_copies":5535,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1845,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13593,"num_write_kb":494016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,9],"acting":[11,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.8","version":"70'139","reported_seq":471,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-08-19T14:16:05.740177+0000","last_change":"2024-08-19T14:15:44.541566+0000","last_active":"2024-08-19T14:16:05.740177+0000","last_peered":"2024-08-19T14:16:05.740177+0000","last_clean":"2024-08-19T14:16:05.740177+0000","last_became_active":"2024-08-19T14:13:05.605694+0000","last_became_peered":"2024-08-19T14:13:05.605694+0000","last_unstale":"2024-08-19T14:16:05.740177+0000","last_undegraded":"2024-08-19T14:16:05.740177+0000","last_fullsized":"2024-08-19T14:16:05.740177+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:15:44.541487+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:15:44.541487+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:16:49.105540+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.b","version":"197'13490","reported_seq":13799,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-08-19T14:16:09.729074+0000","last_change":"2024-08-19T14:16:04.247292+0000","last_active":"2024-08-19T14:16:09.729074+0000","last_peered":"2024-08-19T14:16:09.729074+0000","last_clean":"2024-08-19T14:16:09.729074+0000","last_became_active":"2024-08-19T14:12:42.729014+0000","last_became_peered":"2024-08-19T14:12:42.729014+0000","last_unstale":"2024-08-19T14:16:09.729074+0000","last_undegraded":"2024-08-19T14:16:09.729074+0000","last_fullsized":"2024-08-19T14:16:09.729074+0000","mapping_epoch":102,"log_start":"122'3400","ondisk_log_start":"122'3400","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"196'13194","last_scrub_stamp":"2024-08-19T14:16:04.247138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:16:04.247138+0000","objects_scrubbed":2194,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:17:05.520387+0000","scrub_duration":449,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123863040,"num_objects":1890,"num_object_clones":0,"num_object_copies":5670,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13490,"num_write_kb":492160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":2085027864,"num_objects":31816,"num_object_clones":0,"num_object_copies":95448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1209,"num_objects_unfound":0,"num_objects_dirty":31816,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":214742,"num_write_kb":7889793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23037,"num_bytes_recovered":1471283200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6563966976,"data_stored":6563954760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160863,"ondisk_log_size":160863,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1529,"ondisk_log_size":1529,"up":33,"acting":33,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187514,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":656052,"kb_used_data":552684,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93089100,"statfs":{"total":95995035648,"available":95323238400,"internally_reserved":0,"allocated":565948416,"data_stored":564980982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":105839044},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":343597383742,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":74,"seq":317827579971,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1036988,"kb_used_data":901680,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":92708164,"statfs":{"total":95995035648,"available":94933159936,"internally_reserved":0,"allocated":923320320,"data_stored":922348814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":138541505},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874313,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":744116,"kb_used_data":632364,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93001036,"statfs":{"total":95995035648,"available":95233060864,"internally_reserved":0,"allocated":647540736,"data_stored":646573302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":57,"seq":244813135947,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":917240,"kb_used_data":769452,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":92827912,"statfs":{"total":95995035648,"available":95055781888,"internally_reserved":0,"allocated":787918848,"data_stored":786951414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":151321034},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":206158430287,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691923,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":332336,"kb_used_data":278952,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93412816,"statfs":{"total":95995035648,"available":95654723584,"internally_reserved":0,"allocated":285646848,"data_stored":284680115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986264,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":630908,"kb_used_data":527536,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93114244,"statfs":{"total":95995035648,"available":95348985856,"internally_reserved":0,"allocated":540196864,"data_stored":539225358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149790,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":775936,"kb_used_data":640568,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":92969216,"statfs":{"total":95995035648,"available":95200477184,"internally_reserved":0,"allocated":655941632,"data_stored":654966574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":138541511},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411426,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":790080,"kb_used_data":682484,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":92955072,"statfs":{"total":95995035648,"available":95185993728,"internally_reserved":0,"allocated":698863616,"data_stored":697892630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574951,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":764664,"kb_used_data":652908,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":92980488,"statfs":{"total":95995035648,"available":95212019712,"internally_reserved":0,"allocated":668577792,"data_stored":667610358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738476,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":712116,"kb_used_data":608748,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93033036,"statfs":{"total":95995035648,"available":95265828864,"internally_reserved":0,"allocated":623357952,"data_stored":622390518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":602669056,"data_stored":602669056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":656277504,"data_stored":656277504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":690487296,"data_stored":690487296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":638783488,"data_stored":638779416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":527831040,"data_stored":527826968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":276692992,"data_stored":276692992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":94371840,"data_stored":94371840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":781713408,"data_stored":781713408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":624951296,"data_stored":624951296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":911216640,"data_stored":911212568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":198115328,"data_stored":198115328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":560857088,"data_stored":560857088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T14:16:12.818 INFO:tasks.thrashosds.thrasher:Setting 1.0 to [2, 9, 0] 2024-08-19T14:16:12.818 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '1.0', '2', '9', '0'] 2024-08-19T14:16:12.819 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd pg-upmap 1.0 2 9 0 2024-08-19T14:16:12.822 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:12.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:12.823 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:16:12.924 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:16:12.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:12.948 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:12.948 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:16:13.049 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:16:13.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:13.090 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:13.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:16:13.192 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:16:13.233 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:13.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:13.234 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:16:13.335 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:16:13.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:13 smithi031 ceph-mon[16336]: pgmap v487: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 1.9 GiB data, 6.9 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 951 op/s; 482/91224 objects misplaced (0.528%); 9.8 MiB/s, 157 objects/s recovering 2024-08-19T14:16:13.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:13.373 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:13.373 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:16:13.474 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:16:13.506 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:13.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:13 smithi032 ceph-mon[20004]: pgmap v487: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 1.9 GiB data, 6.9 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 951 op/s; 482/91224 objects misplaced (0.528%); 9.8 MiB/s, 157 objects/s recovering 2024-08-19T14:16:13.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:13.508 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:16:13.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:13 smithi045 ceph-mon[20143]: pgmap v487: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 1.9 GiB data, 6.9 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 951 op/s; 482/91224 objects misplaced (0.528%); 9.8 MiB/s, 157 objects/s recovering 2024-08-19T14:16:13.609 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:16:13.634 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:13.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:13.635 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:16:13.736 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:16:13.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:13.763 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:13.764 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:16:13.864 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:16:13.891 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:13.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:13.892 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:16:13.993 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:16:14.019 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:14.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:14.020 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:16:14.121 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:16:14.145 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:14.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:14.146 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:16:14.247 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:16:14.273 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:14.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:14.274 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:16:14.375 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:16:14.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:14.402 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:14.402 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:16:14.503 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:16:14.535 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:14.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:14.536 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:16:14.638 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:16:14.666 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:14.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:14.667 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:16:14.768 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:16:14.797 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:14.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:14.798 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:16:14.898 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:16:14.920 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:14.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:14.921 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:16:15.022 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:16:15.053 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:15.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:15.054 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:16:15.155 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:16:15.181 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:15.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:15.182 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:16:15.283 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:16:15.311 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:15.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:15.313 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:16:15.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:15 smithi031 ceph-mon[16336]: pgmap v488: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 1.7 GiB data, 6.5 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 958 op/s; 482/85866 objects misplaced (0.561%); 9.5 MiB/s, 151 objects/s recovering 2024-08-19T14:16:15.413 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:16:15.435 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:15.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:15.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:16:15.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:15 smithi032 ceph-mon[20004]: pgmap v488: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 1.7 GiB data, 6.5 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 958 op/s; 482/85866 objects misplaced (0.561%); 9.5 MiB/s, 151 objects/s recovering 2024-08-19T14:16:15.537 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:16:15.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:15 smithi045 ceph-mon[20143]: pgmap v488: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 1.7 GiB data, 6.5 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 958 op/s; 482/85866 objects misplaced (0.561%); 9.5 MiB/s, 151 objects/s recovering 2024-08-19T14:16:15.558 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:15.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:15.559 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:16:15.660 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:16:15.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:15.685 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:15.685 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:16:15.785 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:16:15.807 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:15.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:15.808 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:16:15.895 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-08-19T14:16:15.909 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:16:15.927 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:16:15.930 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:15.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:15.931 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:16:16.032 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:16:16.056 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:16.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:16.057 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:16:16.158 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:16:16.185 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:16.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:16.186 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:16:16.287 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:16:16.314 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:16.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:16.315 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:16:16.316 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:16 smithi031 ceph-mon[16336]: osdmap e199: 12 total, 12 up, 10 in 2024-08-19T14:16:16.416 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:16:16.438 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:16.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:16.439 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:16:16.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:16 smithi032 ceph-mon[20004]: osdmap e199: 12 total, 12 up, 10 in 2024-08-19T14:16:16.541 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:16:16.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:16 smithi045 ceph-mon[20143]: osdmap e199: 12 total, 12 up, 10 in 2024-08-19T14:16:16.565 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:16.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:16.566 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:16:16.667 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:16:16.692 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:16.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:16.693 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:16:16.706 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:16:16.707 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:16:16.740 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:16:16.740 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:16:16.784 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:16:16.784 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:16:16.793 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:16:16.815 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:16.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:16.816 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:16:16.834 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:16:16.835 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:16:16.881 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:16:16.882 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:16:16.916 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:16:16.938 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:16:16.939 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:16:16.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:16.940 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:16.940 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:16:16.955 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:16:16.970 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:16:16.970 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:16:17.007 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:16:17.007 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:16:17.040 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:16:17.052 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:16:17.053 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:16:17.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:17.096 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:17.096 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:16:17.102 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:16:17.102 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:16:17.138 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:16:17.138 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:16:17.179 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:17 smithi031 ceph-mon[16336]: pgmap v490: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 1.7 GiB data, 6.2 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 968 op/s; 482/81687 objects misplaced (0.590%); 4.7 MiB/s, 74 objects/s recovering 2024-08-19T14:16:17.179 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:17 smithi031 ceph-mon[16336]: osdmap e200: 12 total, 12 up, 10 in 2024-08-19T14:16:17.179 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:17 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/4167314480' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:16:17.179 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:16:17.179 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:16:17.198 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:16:17.237 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:17.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:17.239 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:16:17.340 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:16:17.366 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:17.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:17.367 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:16:17.468 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:16:17.491 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:17.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:17.493 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:16:17.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:17 smithi032 ceph-mon[20004]: pgmap v490: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 1.7 GiB data, 6.2 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 968 op/s; 482/81687 objects misplaced (0.590%); 4.7 MiB/s, 74 objects/s recovering 2024-08-19T14:16:17.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:17 smithi032 ceph-mon[20004]: osdmap e200: 12 total, 12 up, 10 in 2024-08-19T14:16:17.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:17 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/4167314480' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:16:17.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:17 smithi045 ceph-mon[20143]: pgmap v490: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 1.7 GiB data, 6.2 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 968 op/s; 482/81687 objects misplaced (0.590%); 4.7 MiB/s, 74 objects/s recovering 2024-08-19T14:16:17.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:17 smithi045 ceph-mon[20143]: osdmap e200: 12 total, 12 up, 10 in 2024-08-19T14:16:17.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:17 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/4167314480' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:16:17.593 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:16:17.630 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:17.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:17.631 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:16:17.681 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:16:17.681 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 20, 2024-08-19T14:16:17.681 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 600, 2024-08-19T14:16:17.681 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [ 2024-08-19T14:16:17.681 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.681 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.14156.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-08-19T14:16:17.681 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:08:17.668325+0000", 2024-08-19T14:16:17.681 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 480.00792099199998, 2024-08-19T14:16:17.681 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.19289305400000001, 2024-08-19T14:16:17.682 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:16:17.682 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:16:17.682 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:16:17.682 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:16:17.682 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.14156", 2024-08-19T14:16:17.682 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.31:0/611436414", 2024-08-19T14:16:17.682 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 1 2024-08-19T14:16:17.682 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.682 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:16:17.682 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.682 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:16:17.682 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:08:17.668325+0000", 2024-08-19T14:16:17.682 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:17.683 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.683 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.683 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:16:17.683 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:08:17.668325+0000", 2024-08-19T14:16:17.683 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:17.683 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.683 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.683 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:16:17.683 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:08:17.668326+0000", 2024-08-19T14:16:17.683 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.015e-06 2024-08-19T14:16:17.683 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.683 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.683 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:16:17.683 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:08:17.668338+0000", 2024-08-19T14:16:17.684 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.181e-05 2024-08-19T14:16:17.684 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.684 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.684 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:16:17.684 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:08:17.668346+0000", 2024-08-19T14:16:17.684 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.4309999999999994e-06 2024-08-19T14:16:17.684 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.684 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.684 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:16:17.684 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:08:17.668353+0000", 2024-08-19T14:16:17.684 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.1110000000000002e-06 2024-08-19T14:16:17.684 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.684 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.684 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:16:17.685 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:08:17.668398+0000", 2024-08-19T14:16:17.685 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4821000000000003e-05 2024-08-19T14:16:17.685 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.685 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.685 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:16:17.686 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:08:17.668405+0000", 2024-08-19T14:16:17.686 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.2250000000000003e-06 2024-08-19T14:16:17.686 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.686 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.686 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:16:17.686 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:08:17.860190+0000", 2024-08-19T14:16:17.686 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.19178468600000001 2024-08-19T14:16:17.686 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.686 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.686 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:16:17.687 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:08:17.860320+0000", 2024-08-19T14:16:17.687 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000129398 2024-08-19T14:16:17.687 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.687 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.687 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,2", 2024-08-19T14:16:17.687 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:08:17.860454+0000", 2024-08-19T14:16:17.687 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013426399999999999 2024-08-19T14:16:17.687 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.687 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.687 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:16:17.687 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:08:17.860972+0000", 2024-08-19T14:16:17.687 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00051784499999999996 2024-08-19T14:16:17.687 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.688 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.688 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:17.688 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:08:17.861120+0000", 2024-08-19T14:16:17.688 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000148262 2024-08-19T14:16:17.688 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.688 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.688 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:17.688 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:08:17.861160+0000", 2024-08-19T14:16:17.688 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.0058000000000002e-05 2024-08-19T14:16:17.688 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.688 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.688 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:16:17.688 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:08:17.861196+0000", 2024-08-19T14:16:17.689 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.5799000000000001e-05 2024-08-19T14:16:17.689 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.689 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.689 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:16:17.689 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:08:17.861218+0000", 2024-08-19T14:16:17.689 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.2328999999999999e-05 2024-08-19T14:16:17.689 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:17.689 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:17.689 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:17.689 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.689 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.690 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:40314 2.6 2:65a7b859:::benchmark_data_smithi045_46415_object40313:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-08-19T14:16:17.690 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:13:02.993308+0000", 2024-08-19T14:16:17.690 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 194.682937815, 2024-08-19T14:16:17.690 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.20540061800000001, 2024-08-19T14:16:17.690 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:16:17.690 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:16:17.690 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:16:17.690 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:16:17.690 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:16:17.690 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:16:17.690 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 40314 2024-08-19T14:16:17.690 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.690 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:16:17.690 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.691 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:16:17.691 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.993308+0000", 2024-08-19T14:16:17.691 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:17.691 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.691 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.691 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:16:17.691 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.993308+0000", 2024-08-19T14:16:17.691 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:17.691 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.691 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.691 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:16:17.691 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.993309+0000", 2024-08-19T14:16:17.691 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1200000000000001e-06 2024-08-19T14:16:17.691 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.691 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.692 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:16:17.692 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.993470+0000", 2024-08-19T14:16:17.692 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00016027899999999999 2024-08-19T14:16:17.692 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.692 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.692 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:16:17.692 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.993482+0000", 2024-08-19T14:16:17.692 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2276000000000001e-05 2024-08-19T14:16:17.692 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.692 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.692 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:16:17.692 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.993487+0000", 2024-08-19T14:16:17.692 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.4199999999999998e-06 2024-08-19T14:16:17.692 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.692 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.693 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:16:17.693 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.993536+0000", 2024-08-19T14:16:17.693 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.9051999999999997e-05 2024-08-19T14:16:17.693 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.693 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.693 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:16:17.693 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.993628+0000", 2024-08-19T14:16:17.693 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.1546999999999996e-05 2024-08-19T14:16:17.693 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.693 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.693 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,8", 2024-08-19T14:16:17.693 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.993732+0000", 2024-08-19T14:16:17.693 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000104467 2024-08-19T14:16:17.694 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.694 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.694 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:17.694 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.997603+0000", 2024-08-19T14:16:17.694 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0038706529999999999 2024-08-19T14:16:17.694 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.694 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.694 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:16:17.694 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.997654+0000", 2024-08-19T14:16:17.694 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0628999999999999e-05 2024-08-19T14:16:17.694 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.694 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.694 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:17.694 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.198613+0000", 2024-08-19T14:16:17.694 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.20095934500000001 2024-08-19T14:16:17.695 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.695 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.695 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:16:17.695 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.198666+0000", 2024-08-19T14:16:17.695 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.3340000000000001e-05 2024-08-19T14:16:17.695 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.695 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.695 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:16:17.695 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.198709+0000", 2024-08-19T14:16:17.695 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2490000000000001e-05 2024-08-19T14:16:17.695 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:17.695 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:17.695 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:17.695 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.696 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.696 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:40315 2.9 2:95c123d2:::benchmark_data_smithi045_46415_object40314:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-08-19T14:16:17.696 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:13:02.993698+0000", 2024-08-19T14:16:17.696 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 194.682548148, 2024-08-19T14:16:17.696 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.20320638299999999, 2024-08-19T14:16:17.696 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:16:17.696 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:16:17.696 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:16:17.696 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:16:17.697 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:16:17.697 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:16:17.697 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 40315 2024-08-19T14:16:17.697 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.697 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:16:17.697 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.697 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:16:17.697 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.993698+0000", 2024-08-19T14:16:17.697 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:17.697 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.698 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.698 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:16:17.698 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.993698+0000", 2024-08-19T14:16:17.698 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:17.698 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.698 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.698 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:16:17.698 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.993699+0000", 2024-08-19T14:16:17.698 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.71e-07 2024-08-19T14:16:17.698 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.698 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.698 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:16:17.698 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.993770+0000", 2024-08-19T14:16:17.698 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.1186000000000006e-05 2024-08-19T14:16:17.699 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.699 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.699 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:16:17.699 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.993782+0000", 2024-08-19T14:16:17.699 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1822e-05 2024-08-19T14:16:17.699 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.699 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.699 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:16:17.699 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.993789+0000", 2024-08-19T14:16:17.699 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.3640000000000001e-06 2024-08-19T14:16:17.699 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.699 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.699 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:16:17.700 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.993844+0000", 2024-08-19T14:16:17.700 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.4688999999999998e-05 2024-08-19T14:16:17.700 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.700 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.700 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:16:17.700 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.993997+0000", 2024-08-19T14:16:17.700 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000153709 2024-08-19T14:16:17.700 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.701 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.701 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,7", 2024-08-19T14:16:17.701 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.994134+0000", 2024-08-19T14:16:17.701 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013627600000000001 2024-08-19T14:16:17.701 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.701 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.701 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:16:17.701 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.995858+0000", 2024-08-19T14:16:17.701 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00172466 2024-08-19T14:16:17.701 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.701 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.701 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:17.701 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.996156+0000", 2024-08-19T14:16:17.701 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000297764 2024-08-19T14:16:17.702 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.702 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.702 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:17.702 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.196808+0000", 2024-08-19T14:16:17.702 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.20065148399999999 2024-08-19T14:16:17.702 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.702 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.702 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:16:17.702 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.196850+0000", 2024-08-19T14:16:17.702 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.1934000000000002e-05 2024-08-19T14:16:17.702 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.702 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.702 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:16:17.702 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.196904+0000", 2024-08-19T14:16:17.703 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.4623999999999997e-05 2024-08-19T14:16:17.703 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:17.703 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:17.703 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:17.703 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.703 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.703 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:40316 2.6 2:6b76d263:::benchmark_data_smithi045_46415_object40315:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-08-19T14:16:17.703 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:13:02.994308+0000", 2024-08-19T14:16:17.703 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 194.68193753899999, 2024-08-19T14:16:17.703 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.20450618700000001, 2024-08-19T14:16:17.703 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:16:17.703 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:16:17.703 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:16:17.703 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:16:17.704 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:16:17.704 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:16:17.704 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 40316 2024-08-19T14:16:17.704 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.704 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:16:17.704 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.704 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:16:17.704 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.994308+0000", 2024-08-19T14:16:17.704 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:17.704 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.704 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.704 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:16:17.704 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.994308+0000", 2024-08-19T14:16:17.704 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:17.705 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.705 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.705 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:16:17.705 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.994309+0000", 2024-08-19T14:16:17.705 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.7899999999999997e-07 2024-08-19T14:16:17.705 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.705 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.705 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:16:17.705 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.994385+0000", 2024-08-19T14:16:17.705 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.6071999999999999e-05 2024-08-19T14:16:17.705 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.705 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.705 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:16:17.705 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.994398+0000", 2024-08-19T14:16:17.705 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2228e-05 2024-08-19T14:16:17.706 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.706 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.706 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:16:17.706 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.994405+0000", 2024-08-19T14:16:17.706 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.3189999999999997e-06 2024-08-19T14:16:17.706 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.706 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.706 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:16:17.706 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.995083+0000", 2024-08-19T14:16:17.706 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000678117 2024-08-19T14:16:17.707 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.707 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.707 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:16:17.707 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.995204+0000", 2024-08-19T14:16:17.707 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012103399999999999 2024-08-19T14:16:17.707 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.707 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.707 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,8", 2024-08-19T14:16:17.707 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.995352+0000", 2024-08-19T14:16:17.707 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00014803400000000001 2024-08-19T14:16:17.707 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.707 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.707 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:17.707 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.997636+0000", 2024-08-19T14:16:17.708 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0022837830000000002 2024-08-19T14:16:17.708 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.708 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.708 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:16:17.708 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.997659+0000", 2024-08-19T14:16:17.708 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.3473e-05 2024-08-19T14:16:17.708 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.708 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.708 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:17.708 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.198750+0000", 2024-08-19T14:16:17.708 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.201090301 2024-08-19T14:16:17.708 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.708 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.708 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:16:17.708 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.198780+0000", 2024-08-19T14:16:17.709 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.0688999999999997e-05 2024-08-19T14:16:17.709 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.709 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.709 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:16:17.709 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.198815+0000", 2024-08-19T14:16:17.709 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.4258000000000003e-05 2024-08-19T14:16:17.709 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:17.709 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:17.709 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:17.709 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.709 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.709 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:40320 2.6 2:6546d5b0:::benchmark_data_smithi045_46415_object40319:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-08-19T14:16:17.709 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:13:02.995448+0000", 2024-08-19T14:16:17.709 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 194.68079764300001, 2024-08-19T14:16:17.709 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.20356038500000001, 2024-08-19T14:16:17.710 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:16:17.710 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:16:17.711 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:16:17.711 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:16:17.711 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:16:17.711 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:16:17.711 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 40320 2024-08-19T14:16:17.711 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.711 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:16:17.711 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.711 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:16:17.712 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.995448+0000", 2024-08-19T14:16:17.712 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:17.712 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.712 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.712 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:16:17.712 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.995448+0000", 2024-08-19T14:16:17.712 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:17.712 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.712 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.712 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:16:17.712 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.995449+0000", 2024-08-19T14:16:17.712 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.7000000000000004e-07 2024-08-19T14:16:17.712 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.712 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.713 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:16:17.713 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.995511+0000", 2024-08-19T14:16:17.713 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.2249000000000007e-05 2024-08-19T14:16:17.713 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.713 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.713 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:16:17.713 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.995520+0000", 2024-08-19T14:16:17.713 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.8799999999999997e-06 2024-08-19T14:16:17.713 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.713 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.713 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:16:17.713 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.995525+0000", 2024-08-19T14:16:17.713 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.6659999999999999e-06 2024-08-19T14:16:17.714 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.714 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.714 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:16:17.714 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.996261+0000", 2024-08-19T14:16:17.714 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000736351 2024-08-19T14:16:17.714 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.714 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.714 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:16:17.714 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.996382+0000", 2024-08-19T14:16:17.714 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000120983 2024-08-19T14:16:17.714 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.714 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.714 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,8", 2024-08-19T14:16:17.714 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.996517+0000", 2024-08-19T14:16:17.715 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013479900000000001 2024-08-19T14:16:17.715 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.715 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.715 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:16:17.715 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.997767+0000", 2024-08-19T14:16:17.715 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0012497739999999999 2024-08-19T14:16:17.715 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.715 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.715 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:17.715 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.999297+0000", 2024-08-19T14:16:17.715 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0015301150000000001 2024-08-19T14:16:17.715 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.715 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.716 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:17.716 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.198909+0000", 2024-08-19T14:16:17.716 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.199612074 2024-08-19T14:16:17.716 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.716 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.716 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:16:17.716 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.198968+0000", 2024-08-19T14:16:17.716 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.8801999999999997e-05 2024-08-19T14:16:17.716 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.716 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.716 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:16:17.716 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.199009+0000", 2024-08-19T14:16:17.716 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.0921999999999999e-05 2024-08-19T14:16:17.716 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:17.717 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:17.717 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:17.717 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.717 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.717 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:40322 2.9 2:99a1eedb:::benchmark_data_smithi045_46415_object40321:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-08-19T14:16:17.717 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:13:02.996325+0000", 2024-08-19T14:16:17.717 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 194.679920947, 2024-08-19T14:16:17.717 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.20255289300000001, 2024-08-19T14:16:17.717 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:16:17.717 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:16:17.717 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:16:17.717 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:16:17.717 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:16:17.717 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:16:17.718 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 40322 2024-08-19T14:16:17.718 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.718 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:16:17.718 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.718 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:16:17.718 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.996325+0000", 2024-08-19T14:16:17.718 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:17.718 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.718 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.718 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:16:17.718 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.996325+0000", 2024-08-19T14:16:17.718 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:17.718 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.718 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.719 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:16:17.719 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.996326+0000", 2024-08-19T14:16:17.719 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0470000000000001e-06 2024-08-19T14:16:17.719 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.719 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.719 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:16:17.719 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.996355+0000", 2024-08-19T14:16:17.719 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.9167999999999999e-05 2024-08-19T14:16:17.719 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.719 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.719 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:16:17.719 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.996366+0000", 2024-08-19T14:16:17.719 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0679999999999999e-05 2024-08-19T14:16:17.719 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.719 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.720 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:16:17.720 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.996373+0000", 2024-08-19T14:16:17.720 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.5229999999999998e-06 2024-08-19T14:16:17.720 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.720 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.720 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:16:17.721 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.996428+0000", 2024-08-19T14:16:17.721 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.4580000000000003e-05 2024-08-19T14:16:17.721 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.721 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.721 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:16:17.721 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.996560+0000", 2024-08-19T14:16:17.721 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013169399999999999 2024-08-19T14:16:17.721 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.721 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.721 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,7", 2024-08-19T14:16:17.721 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.996700+0000", 2024-08-19T14:16:17.721 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000139973 2024-08-19T14:16:17.721 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.722 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.722 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:16:17.722 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.998183+0000", 2024-08-19T14:16:17.722 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0014834609999999999 2024-08-19T14:16:17.722 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.722 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.722 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:17.722 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.998340+0000", 2024-08-19T14:16:17.722 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000156658 2024-08-19T14:16:17.722 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.723 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.724 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:17.724 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.198775+0000", 2024-08-19T14:16:17.724 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.20043515100000001 2024-08-19T14:16:17.724 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.724 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.724 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:16:17.724 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.198828+0000", 2024-08-19T14:16:17.726 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.2556999999999998e-05 2024-08-19T14:16:17.726 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.726 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.726 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:16:17.726 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.198878+0000", 2024-08-19T14:16:17.726 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0401000000000001e-05 2024-08-19T14:16:17.726 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:17.726 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:17.727 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:17.727 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.727 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.727 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:40331 2.9 2:9a78e0ca:::benchmark_data_smithi045_46415_object40330:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-08-19T14:16:17.727 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:13:03.000136+0000", 2024-08-19T14:16:17.727 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 194.676110204, 2024-08-19T14:16:17.727 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.198840613, 2024-08-19T14:16:17.727 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:16:17.727 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:16:17.727 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:16:17.727 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:16:17.727 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:16:17.727 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:16:17.727 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 40331 2024-08-19T14:16:17.728 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.728 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:16:17.728 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.728 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:16:17.728 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.000136+0000", 2024-08-19T14:16:17.728 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:17.728 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.728 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.728 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:16:17.728 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.000136+0000", 2024-08-19T14:16:17.728 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:17.728 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.728 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.729 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:16:17.729 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.000137+0000", 2024-08-19T14:16:17.729 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.61e-06 2024-08-19T14:16:17.729 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.729 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.729 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:16:17.729 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.000236+0000", 2024-08-19T14:16:17.729 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.8456999999999996e-05 2024-08-19T14:16:17.729 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.729 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.729 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:16:17.729 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.000250+0000", 2024-08-19T14:16:17.729 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4147e-05 2024-08-19T14:16:17.729 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.730 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.730 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:16:17.730 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.000258+0000", 2024-08-19T14:16:17.730 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.0730000000000005e-06 2024-08-19T14:16:17.730 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.730 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.730 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:16:17.730 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.000317+0000", 2024-08-19T14:16:17.730 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.8529000000000002e-05 2024-08-19T14:16:17.730 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.730 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.730 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:16:17.730 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.000446+0000", 2024-08-19T14:16:17.731 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012958499999999999 2024-08-19T14:16:17.731 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.731 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.731 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,7", 2024-08-19T14:16:17.731 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.000591+0000", 2024-08-19T14:16:17.731 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00014517600000000001 2024-08-19T14:16:17.731 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.731 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.731 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:17.731 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.003719+0000", 2024-08-19T14:16:17.731 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0031274900000000001 2024-08-19T14:16:17.731 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.731 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.731 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:16:17.732 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.003735+0000", 2024-08-19T14:16:17.732 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6030000000000001e-05 2024-08-19T14:16:17.732 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.732 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.732 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:17.732 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.198911+0000", 2024-08-19T14:16:17.732 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.195175671 2024-08-19T14:16:17.732 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.732 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.732 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:16:17.732 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.198942+0000", 2024-08-19T14:16:17.732 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.1293e-05 2024-08-19T14:16:17.732 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.732 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.733 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:16:17.733 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.198976+0000", 2024-08-19T14:16:17.733 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.4551999999999997e-05 2024-08-19T14:16:17.733 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:17.733 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:17.733 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:17.733 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.733 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.733 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:40334 2.6 2:65da395f:::benchmark_data_smithi045_46415_object40333:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-08-19T14:16:17.733 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:13:03.000682+0000", 2024-08-19T14:16:17.733 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 194.675563665, 2024-08-19T14:16:17.733 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.198468745, 2024-08-19T14:16:17.733 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:16:17.733 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:16:17.734 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:16:17.734 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:16:17.734 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:16:17.734 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:16:17.734 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 40334 2024-08-19T14:16:17.734 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.734 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:16:17.734 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.734 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:16:17.734 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.000682+0000", 2024-08-19T14:16:17.734 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:17.734 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.734 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.735 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:16:17.735 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.000682+0000", 2024-08-19T14:16:17.735 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:17.735 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.735 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.735 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:16:17.735 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.000683+0000", 2024-08-19T14:16:17.735 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.6199999999999996e-07 2024-08-19T14:16:17.735 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.735 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.735 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:16:17.735 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.000744+0000", 2024-08-19T14:16:17.735 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.0466e-05 2024-08-19T14:16:17.735 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.736 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.736 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:16:17.736 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.000765+0000", 2024-08-19T14:16:17.736 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.1691e-05 2024-08-19T14:16:17.736 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.736 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.736 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:16:17.736 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.000773+0000", 2024-08-19T14:16:17.736 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.1429999999999998e-06 2024-08-19T14:16:17.736 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.736 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.736 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:16:17.736 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.000823+0000", 2024-08-19T14:16:17.737 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.9149e-05 2024-08-19T14:16:17.737 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.737 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.737 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:16:17.737 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.000944+0000", 2024-08-19T14:16:17.737 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012130599999999999 2024-08-19T14:16:17.737 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.737 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.737 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,8", 2024-08-19T14:16:17.737 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.001068+0000", 2024-08-19T14:16:17.737 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012416900000000001 2024-08-19T14:16:17.737 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.737 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.737 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:16:17.738 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.003517+0000", 2024-08-19T14:16:17.738 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0024484210000000001 2024-08-19T14:16:17.738 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.738 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.738 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:17.738 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.003557+0000", 2024-08-19T14:16:17.738 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.0033000000000001e-05 2024-08-19T14:16:17.738 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.738 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.738 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:17.738 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.199070+0000", 2024-08-19T14:16:17.738 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.19551357799999999 2024-08-19T14:16:17.738 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.738 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.739 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:16:17.739 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.199117+0000", 2024-08-19T14:16:17.739 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.6723e-05 2024-08-19T14:16:17.739 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.739 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.739 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:16:17.739 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.199151+0000", 2024-08-19T14:16:17.739 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.4204e-05 2024-08-19T14:16:17.739 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:17.739 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:17.739 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:17.739 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.739 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.739 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:40335 2.9 2:91a68a3e:::benchmark_data_smithi045_46415_object40334:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-08-19T14:16:17.740 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:13:03.000790+0000", 2024-08-19T14:16:17.740 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 194.67545569699999, 2024-08-19T14:16:17.740 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.19867384900000001, 2024-08-19T14:16:17.740 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:16:17.740 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:16:17.740 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:16:17.740 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:16:17.740 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:16:17.740 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:16:17.740 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 40335 2024-08-19T14:16:17.740 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.740 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:16:17.740 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.740 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:16:17.741 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.000790+0000", 2024-08-19T14:16:17.741 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:17.741 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.741 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.741 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:16:17.741 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.000790+0000", 2024-08-19T14:16:17.741 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:17.741 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.741 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.741 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:16:17.741 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.000791+0000", 2024-08-19T14:16:17.741 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.3799999999999997e-07 2024-08-19T14:16:17.741 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.741 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.742 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:16:17.742 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.000829+0000", 2024-08-19T14:16:17.742 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.8578999999999998e-05 2024-08-19T14:16:17.742 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.742 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.742 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:16:17.742 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.000837+0000", 2024-08-19T14:16:17.742 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.9200000000000004e-06 2024-08-19T14:16:17.742 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.742 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.743 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:16:17.743 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.000855+0000", 2024-08-19T14:16:17.743 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.7261e-05 2024-08-19T14:16:17.743 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.743 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.743 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:16:17.743 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.002117+0000", 2024-08-19T14:16:17.743 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0012625049999999999 2024-08-19T14:16:17.743 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.743 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.743 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:16:17.743 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.002255+0000", 2024-08-19T14:16:17.743 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013797700000000001 2024-08-19T14:16:17.744 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.744 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.744 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,7", 2024-08-19T14:16:17.744 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.002397+0000", 2024-08-19T14:16:17.744 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00014145999999999999 2024-08-19T14:16:17.744 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.744 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.744 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:16:17.744 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.003775+0000", 2024-08-19T14:16:17.744 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0013782759999999999 2024-08-19T14:16:17.744 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.744 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.744 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:17.744 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.005174+0000", 2024-08-19T14:16:17.745 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.001399057 2024-08-19T14:16:17.745 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.745 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.745 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:17.745 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.199375+0000", 2024-08-19T14:16:17.745 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.19420136099999999 2024-08-19T14:16:17.745 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.745 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.745 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:16:17.745 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.199423+0000", 2024-08-19T14:16:17.745 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.8154e-05 2024-08-19T14:16:17.745 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.745 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.745 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:16:17.746 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.199464+0000", 2024-08-19T14:16:17.746 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.0760999999999998e-05 2024-08-19T14:16:17.746 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:17.746 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:17.746 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:17.746 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.746 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.746 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:40336 2.6 2:6d681dd4:::benchmark_data_smithi045_46415_object40335:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-08-19T14:16:17.746 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:13:03.000884+0000", 2024-08-19T14:16:17.746 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 194.67536244600001, 2024-08-19T14:16:17.746 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.19840814800000001, 2024-08-19T14:16:17.746 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:16:17.746 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:16:17.747 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:16:17.747 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:16:17.747 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:16:17.747 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:16:17.747 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 40336 2024-08-19T14:16:17.747 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.747 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:16:17.747 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.747 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:16:17.747 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.000884+0000", 2024-08-19T14:16:17.747 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:17.747 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.747 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.748 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:16:17.748 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.000884+0000", 2024-08-19T14:16:17.748 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:17.748 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.748 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.748 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:16:17.748 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.000884+0000", 2024-08-19T14:16:17.748 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.9699999999999996e-07 2024-08-19T14:16:17.748 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.748 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.748 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:16:17.748 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.000926+0000", 2024-08-19T14:16:17.748 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.1542999999999998e-05 2024-08-19T14:16:17.749 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.749 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.749 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:16:17.749 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.000934+0000", 2024-08-19T14:16:17.749 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.8610000000000004e-06 2024-08-19T14:16:17.749 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.749 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.749 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:16:17.749 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.000940+0000", 2024-08-19T14:16:17.749 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.7289999999999997e-06 2024-08-19T14:16:17.749 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.749 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.749 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:16:17.749 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.002371+0000", 2024-08-19T14:16:17.750 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0014310169999999999 2024-08-19T14:16:17.750 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.750 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.750 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:16:17.750 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.002486+0000", 2024-08-19T14:16:17.750 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000114317 2024-08-19T14:16:17.750 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.750 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.750 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,8", 2024-08-19T14:16:17.750 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.002611+0000", 2024-08-19T14:16:17.750 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012566899999999999 2024-08-19T14:16:17.750 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.750 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.751 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:16:17.751 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.003645+0000", 2024-08-19T14:16:17.751 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.001034135 2024-08-19T14:16:17.751 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.751 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.751 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:17.751 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.004552+0000", 2024-08-19T14:16:17.751 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00090656799999999996 2024-08-19T14:16:17.751 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.751 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.751 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:17.751 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.199200+0000", 2024-08-19T14:16:17.751 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.194647869 2024-08-19T14:16:17.751 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.751 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.752 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:16:17.752 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.199256+0000", 2024-08-19T14:16:17.752 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.5781e-05 2024-08-19T14:16:17.752 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.752 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.752 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:16:17.752 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.199292+0000", 2024-08-19T14:16:17.752 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.6161999999999997e-05 2024-08-19T14:16:17.752 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:17.752 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:17.752 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:17.752 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.752 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.752 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:40337 2.9 2:9e869513:::benchmark_data_smithi045_46415_object40336:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-08-19T14:16:17.752 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:13:03.000952+0000", 2024-08-19T14:16:17.753 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 194.67529447800001, 2024-08-19T14:16:17.753 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.198664388, 2024-08-19T14:16:17.753 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:16:17.753 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:16:17.753 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:16:17.753 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:16:17.753 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:16:17.753 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:16:17.753 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 40337 2024-08-19T14:16:17.753 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.753 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:16:17.753 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.753 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:16:17.753 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.000952+0000", 2024-08-19T14:16:17.754 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:17.754 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.754 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.754 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:16:17.754 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.000952+0000", 2024-08-19T14:16:17.754 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:17.754 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.754 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.754 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:16:17.754 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.000952+0000", 2024-08-19T14:16:17.754 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.3000000000000001e-07 2024-08-19T14:16:17.754 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.754 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.754 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:16:17.755 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.000997+0000", 2024-08-19T14:16:17.755 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4558999999999998e-05 2024-08-19T14:16:17.755 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.755 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.755 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:16:17.755 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.001004+0000", 2024-08-19T14:16:17.755 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.8420000000000005e-06 2024-08-19T14:16:17.755 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.755 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.755 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:16:17.755 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.001011+0000", 2024-08-19T14:16:17.755 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.3280000000000002e-06 2024-08-19T14:16:17.755 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.756 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.756 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:16:17.756 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.003899+0000", 2024-08-19T14:16:17.756 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.002888757 2024-08-19T14:16:17.756 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.756 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.756 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:16:17.756 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.004004+0000", 2024-08-19T14:16:17.756 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:16:17.757 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-19T14:16:17.758 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000104138 2024-08-19T14:16:17.759 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.759 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.759 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,7", 2024-08-19T14:16:17.759 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.004135+0000", 2024-08-19T14:16:17.759 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013134900000000001 2024-08-19T14:16:17.759 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.759 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.759 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:16:17.759 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.005347+0000", 2024-08-19T14:16:17.759 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.001212409 2024-08-19T14:16:17.759 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.759 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.760 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:17.760 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.005771+0000", 2024-08-19T14:16:17.760 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00042346199999999999 2024-08-19T14:16:17.760 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.760 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.760 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:17.760 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.199507+0000", 2024-08-19T14:16:17.760 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.19373660500000001 2024-08-19T14:16:17.760 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.760 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.760 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:16:17.760 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.199578+0000", 2024-08-19T14:16:17.760 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.0559e-05 2024-08-19T14:16:17.760 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.761 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.761 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:16:17.761 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.199616+0000", 2024-08-19T14:16:17.761 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7950000000000001e-05 2024-08-19T14:16:17.761 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:17.761 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:17.761 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:17.761 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.761 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.761 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:40346 2.6 2:6807eaa8:::benchmark_data_smithi045_46415_object40345:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-08-19T14:16:17.761 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:13:03.005519+0000", 2024-08-19T14:16:17.761 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 194.67072745300001, 2024-08-19T14:16:17.761 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.19392162499999999, 2024-08-19T14:16:17.761 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:16:17.762 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:16:17.762 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:16:17.762 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:16:17.762 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:16:17.762 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:16:17.762 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 40346 2024-08-19T14:16:17.762 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.762 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:16:17.762 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.762 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:16:17.762 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.005519+0000", 2024-08-19T14:16:17.762 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:17.762 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.763 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.763 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:16:17.763 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.005519+0000", 2024-08-19T14:16:17.763 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:17.763 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.763 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.763 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:16:17.763 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.005520+0000", 2024-08-19T14:16:17.763 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.083e-06 2024-08-19T14:16:17.763 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.763 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.763 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:16:17.763 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.005575+0000", 2024-08-19T14:16:17.763 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.5426000000000001e-05 2024-08-19T14:16:17.764 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.764 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.764 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:16:17.764 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.005585+0000", 2024-08-19T14:16:17.764 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.9960000000000005e-06 2024-08-19T14:16:17.764 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.764 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.764 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:16:17.764 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.005590+0000", 2024-08-19T14:16:17.764 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1139999999999997e-06 2024-08-19T14:16:17.764 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.764 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.764 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:16:17.764 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.005631+0000", 2024-08-19T14:16:17.765 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.0886999999999997e-05 2024-08-19T14:16:17.765 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.765 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.765 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:16:17.765 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.005726+0000", 2024-08-19T14:16:17.765 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.5130000000000006e-05 2024-08-19T14:16:17.765 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.765 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.765 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,8", 2024-08-19T14:16:17.765 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.005868+0000", 2024-08-19T14:16:17.765 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00014200399999999999 2024-08-19T14:16:17.765 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.765 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.765 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:16:17.766 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.006975+0000", 2024-08-19T14:16:17.766 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0011071239999999999 2024-08-19T14:16:17.766 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.766 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.766 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:17.766 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.007651+0000", 2024-08-19T14:16:17.766 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00067605300000000003 2024-08-19T14:16:17.766 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.766 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.766 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:17.766 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.199348+0000", 2024-08-19T14:16:17.766 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.191696226 2024-08-19T14:16:17.766 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.766 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.767 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:16:17.767 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.199400+0000", 2024-08-19T14:16:17.767 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.2170999999999999e-05 2024-08-19T14:16:17.767 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.767 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.767 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:16:17.767 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.199440+0000", 2024-08-19T14:16:17.767 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.0411000000000003e-05 2024-08-19T14:16:17.767 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:17.767 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:17.767 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:17.767 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.767 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.767 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:40357 2.6 2:668471d1:::benchmark_data_smithi045_46415_object40356:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-08-19T14:16:17.768 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:13:03.011411+0000", 2024-08-19T14:16:17.768 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 194.66483494400001, 2024-08-19T14:16:17.768 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.188200109, 2024-08-19T14:16:17.768 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:16:17.768 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:16:17.768 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:16:17.768 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:16:17.768 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:16:17.768 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:16:17.768 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 40357 2024-08-19T14:16:17.768 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.768 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:16:17.768 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.768 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:16:17.769 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.011411+0000", 2024-08-19T14:16:17.769 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:17.769 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.769 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.769 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:16:17.769 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.011411+0000", 2024-08-19T14:16:17.769 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:17.769 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.769 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.769 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:16:17.769 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.011412+0000", 2024-08-19T14:16:17.769 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.376e-06 2024-08-19T14:16:17.769 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.770 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.770 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:16:17.770 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.011467+0000", 2024-08-19T14:16:17.770 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.4435999999999999e-05 2024-08-19T14:16:17.770 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.770 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.770 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:16:17.770 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.011477+0000", 2024-08-19T14:16:17.770 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0132999999999999e-05 2024-08-19T14:16:17.770 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.770 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.770 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:16:17.770 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.011484+0000", 2024-08-19T14:16:17.770 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.0639999999999996e-06 2024-08-19T14:16:17.771 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.771 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.771 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:16:17.771 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.011529+0000", 2024-08-19T14:16:17.771 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4506999999999999e-05 2024-08-19T14:16:17.771 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.771 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.771 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:16:17.771 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.011615+0000", 2024-08-19T14:16:17.771 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.6231999999999997e-05 2024-08-19T14:16:17.771 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.771 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.771 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,8", 2024-08-19T14:16:17.771 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.011703+0000", 2024-08-19T14:16:17.772 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.8057000000000001e-05 2024-08-19T14:16:17.772 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.772 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.772 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:16:17.772 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.012779+0000", 2024-08-19T14:16:17.772 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0010766370000000001 2024-08-19T14:16:17.772 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.772 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.772 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:17.772 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.013669+0000", 2024-08-19T14:16:17.772 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00088930699999999999 2024-08-19T14:16:17.772 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.772 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.773 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:17.773 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.199503+0000", 2024-08-19T14:16:17.773 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.185834265 2024-08-19T14:16:17.773 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.773 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.773 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:16:17.773 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.199570+0000", 2024-08-19T14:16:17.773 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.6694999999999998e-05 2024-08-19T14:16:17.773 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.773 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.773 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:16:17.774 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.199611+0000", 2024-08-19T14:16:17.774 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.1399999999999997e-05 2024-08-19T14:16:17.774 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:17.774 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:17.774 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:17.774 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.774 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.774 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:40364 2.6 2:6bd48979:::benchmark_data_smithi045_46415_object40363:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-08-19T14:16:17.774 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:13:03.016699+0000", 2024-08-19T14:16:17.774 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 194.659546535, 2024-08-19T14:16:17.774 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.18891474, 2024-08-19T14:16:17.774 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:16:17.774 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:16:17.774 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:16:17.775 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:16:17.775 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:16:17.775 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:16:17.775 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 40364 2024-08-19T14:16:17.775 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.775 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:16:17.775 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.775 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:16:17.775 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.016699+0000", 2024-08-19T14:16:17.775 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:17.775 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.775 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.775 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:16:17.776 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.016699+0000", 2024-08-19T14:16:17.776 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:17.776 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.776 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.776 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:16:17.776 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.016701+0000", 2024-08-19T14:16:17.776 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.226e-06 2024-08-19T14:16:17.776 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.776 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.776 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:16:17.776 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.016753+0000", 2024-08-19T14:16:17.776 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.2482999999999999e-05 2024-08-19T14:16:17.776 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.776 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.777 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:16:17.777 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.016765+0000", 2024-08-19T14:16:17.777 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1453e-05 2024-08-19T14:16:17.777 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.777 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.777 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:16:17.777 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.016772+0000", 2024-08-19T14:16:17.777 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.187e-06 2024-08-19T14:16:17.777 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.777 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.777 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:16:17.777 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.016814+0000", 2024-08-19T14:16:17.777 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.197e-05 2024-08-19T14:16:17.778 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.778 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.778 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:16:17.778 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.016910+0000", 2024-08-19T14:16:17.778 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.6312999999999998e-05 2024-08-19T14:16:17.778 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.778 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.778 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,8", 2024-08-19T14:16:17.778 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.017000+0000", 2024-08-19T14:16:17.778 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.0185000000000005e-05 2024-08-19T14:16:17.778 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.778 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.778 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:16:17.778 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.017957+0000", 2024-08-19T14:16:17.779 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00095644999999999997 2024-08-19T14:16:17.779 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.779 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.779 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:17.779 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.018839+0000", 2024-08-19T14:16:17.779 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00088258100000000001 2024-08-19T14:16:17.779 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.779 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.779 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:17.779 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.205522+0000", 2024-08-19T14:16:17.779 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.186682292 2024-08-19T14:16:17.779 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.779 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.780 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:16:17.780 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.205577+0000", 2024-08-19T14:16:17.780 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.5244999999999997e-05 2024-08-19T14:16:17.780 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.780 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.780 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:16:17.780 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.205614+0000", 2024-08-19T14:16:17.780 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7354999999999997e-05 2024-08-19T14:16:17.780 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:17.780 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:17.780 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:17.780 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.780 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.780 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:40369 2.9 2:9728ae15:::benchmark_data_smithi045_46415_object40368:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-08-19T14:16:17.781 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:13:03.019852+0000", 2024-08-19T14:16:17.781 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 194.65639384100001, 2024-08-19T14:16:17.781 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.18066115699999999, 2024-08-19T14:16:17.781 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:16:17.781 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:16:17.781 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:16:17.781 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:16:17.781 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:16:17.781 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:16:17.781 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 40369 2024-08-19T14:16:17.781 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.781 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:16:17.781 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.781 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:16:17.782 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.019852+0000", 2024-08-19T14:16:17.782 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:17.782 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.782 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.782 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:16:17.782 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.019852+0000", 2024-08-19T14:16:17.782 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:17.782 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.782 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.782 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:16:17.782 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.019853+0000", 2024-08-19T14:16:17.782 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.1500000000000003e-07 2024-08-19T14:16:17.782 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.783 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.783 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:16:17.783 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.019959+0000", 2024-08-19T14:16:17.783 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000105604 2024-08-19T14:16:17.783 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.783 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.783 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:16:17.783 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.019972+0000", 2024-08-19T14:16:17.783 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.378e-05 2024-08-19T14:16:17.783 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.783 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.783 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:16:17.783 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.019979+0000", 2024-08-19T14:16:17.783 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.9070000000000001e-06 2024-08-19T14:16:17.784 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.784 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.784 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:16:17.784 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.020034+0000", 2024-08-19T14:16:17.784 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.4540000000000003e-05 2024-08-19T14:16:17.784 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.784 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.784 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:16:17.784 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.020158+0000", 2024-08-19T14:16:17.784 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000124478 2024-08-19T14:16:17.784 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.784 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.784 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,7", 2024-08-19T14:16:17.784 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.020301+0000", 2024-08-19T14:16:17.785 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00014256800000000001 2024-08-19T14:16:17.785 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.785 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.785 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:16:17.785 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.021496+0000", 2024-08-19T14:16:17.785 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.001195162 2024-08-19T14:16:17.785 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.785 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.785 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:17.785 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.021809+0000", 2024-08-19T14:16:17.785 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000313181 2024-08-19T14:16:17.785 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.785 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.785 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:17.786 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.200411+0000", 2024-08-19T14:16:17.786 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.17860152000000001 2024-08-19T14:16:17.786 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.786 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.786 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:16:17.786 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.200472+0000", 2024-08-19T14:16:17.786 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.1200999999999999e-05 2024-08-19T14:16:17.786 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.786 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.786 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:16:17.786 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.200513+0000", 2024-08-19T14:16:17.786 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.1301000000000003e-05 2024-08-19T14:16:17.786 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:17.787 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:17.787 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:17.787 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.787 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.787 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:40373 2.9 2:96826734:::benchmark_data_smithi045_46415_object40372:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-08-19T14:16:17.787 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:13:03.023452+0000", 2024-08-19T14:16:17.787 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 194.65279412500001, 2024-08-19T14:16:17.787 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.17720892399999999, 2024-08-19T14:16:17.787 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:16:17.787 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:16:17.787 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:16:17.787 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:16:17.787 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:16:17.787 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:16:17.788 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 40373 2024-08-19T14:16:17.788 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.788 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:16:17.788 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.788 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:16:17.788 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.023452+0000", 2024-08-19T14:16:17.788 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:17.788 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.788 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.788 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:16:17.788 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.023452+0000", 2024-08-19T14:16:17.788 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:17.788 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.788 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.788 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:16:17.789 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.023453+0000", 2024-08-19T14:16:17.789 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.95e-07 2024-08-19T14:16:17.789 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.789 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.789 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:16:17.789 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.023498+0000", 2024-08-19T14:16:17.789 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4656000000000001e-05 2024-08-19T14:16:17.789 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.789 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.789 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:16:17.789 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.023508+0000", 2024-08-19T14:16:17.789 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0943e-05 2024-08-19T14:16:17.789 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.789 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.790 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:16:17.790 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.023514+0000", 2024-08-19T14:16:17.790 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.2259999999999996e-06 2024-08-19T14:16:17.790 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.790 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.790 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:16:17.790 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.023554+0000", 2024-08-19T14:16:17.790 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.0763999999999998e-05 2024-08-19T14:16:17.790 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.790 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.790 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:16:17.790 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.023637+0000", 2024-08-19T14:16:17.790 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.2595000000000005e-05 2024-08-19T14:16:17.791 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.791 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.791 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,7", 2024-08-19T14:16:17.791 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.023727+0000", 2024-08-19T14:16:17.791 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.9566000000000003e-05 2024-08-19T14:16:17.791 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.791 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.791 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:16:17.791 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.024768+0000", 2024-08-19T14:16:17.791 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0010417390000000001 2024-08-19T14:16:17.791 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.791 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.791 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:17.791 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.025015+0000", 2024-08-19T14:16:17.791 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00024697400000000001 2024-08-19T14:16:17.792 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.792 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.792 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:17.792 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.200563+0000", 2024-08-19T14:16:17.792 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.175547283 2024-08-19T14:16:17.792 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.792 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.792 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:16:17.792 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.200619+0000", 2024-08-19T14:16:17.792 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.6656000000000001e-05 2024-08-19T14:16:17.792 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.792 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.792 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:16:17.792 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.200661+0000", 2024-08-19T14:16:17.793 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.1526999999999997e-05 2024-08-19T14:16:17.793 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:17.793 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:17.793 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:17.793 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.793 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.793 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:119222 2.6 2:68d6b19d:::benchmark_data_smithi045_46415_object119221:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-08-19T14:16:17.793 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:14:14.682744+0000", 2024-08-19T14:16:17.793 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 122.993502444, 2024-08-19T14:16:17.793 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.163744799, 2024-08-19T14:16:17.793 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:16:17.793 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:16:17.794 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:16:17.794 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:16:17.794 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:16:17.794 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:16:17.794 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 119222 2024-08-19T14:16:17.794 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.794 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:16:17.794 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.794 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:16:17.794 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:14.682744+0000", 2024-08-19T14:16:17.794 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:17.794 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.794 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.795 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:16:17.795 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:14.682744+0000", 2024-08-19T14:16:17.795 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:17.795 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.795 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.795 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:16:17.795 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:14.682745+0000", 2024-08-19T14:16:17.795 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.8679999999999999e-06 2024-08-19T14:16:17.795 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.795 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.795 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:16:17.795 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:14.682781+0000", 2024-08-19T14:16:17.795 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.5472999999999997e-05 2024-08-19T14:16:17.796 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.796 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.796 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:16:17.796 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:14.682795+0000", 2024-08-19T14:16:17.796 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4095e-05 2024-08-19T14:16:17.796 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.796 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.796 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:16:17.796 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:14.682804+0000", 2024-08-19T14:16:17.796 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.1889999999999996e-06 2024-08-19T14:16:17.796 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.796 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.796 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:16:17.797 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:14.691492+0000", 2024-08-19T14:16:17.797 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0086881460000000008 2024-08-19T14:16:17.797 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.797 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.797 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:16:17.797 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:14.691748+0000", 2024-08-19T14:16:17.797 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00025553400000000002 2024-08-19T14:16:17.797 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.797 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.797 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,8", 2024-08-19T14:16:17.797 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:14.692560+0000", 2024-08-19T14:16:17.797 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00081248699999999997 2024-08-19T14:16:17.797 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.797 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.798 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:17.798 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:14.706454+0000", 2024-08-19T14:16:17.798 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.013894012000000001 2024-08-19T14:16:17.798 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.798 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.798 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:16:17.798 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:14.715404+0000", 2024-08-19T14:16:17.798 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0089496260000000005 2024-08-19T14:16:17.798 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.798 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.798 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:17.798 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:14.846181+0000", 2024-08-19T14:16:17.798 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.13077704300000001 2024-08-19T14:16:17.798 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.799 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.799 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:16:17.799 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:14.846300+0000", 2024-08-19T14:16:17.799 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000119082 2024-08-19T14:16:17.799 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.799 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.799 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:16:17.799 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:14.846488+0000", 2024-08-19T14:16:17.799 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00018824400000000001 2024-08-19T14:16:17.799 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:17.799 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:17.799 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:17.799 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.799 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.800 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:123876 2.6 2:60a07204:::benchmark_data_smithi045_46415_object595:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-08-19T14:16:17.800 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:14:21.576527+0000", 2024-08-19T14:16:17.800 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 116.099718923, 2024-08-19T14:16:17.800 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.17839765899999999, 2024-08-19T14:16:17.800 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:16:17.800 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:16:17.800 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:16:17.800 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:16:17.800 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:16:17.800 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:16:17.800 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 123876 2024-08-19T14:16:17.800 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.800 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:16:17.800 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.801 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:16:17.801 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.576527+0000", 2024-08-19T14:16:17.801 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:17.801 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.801 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.801 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:16:17.801 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.576527+0000", 2024-08-19T14:16:17.801 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:17.801 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.801 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.801 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:16:17.801 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.576528+0000", 2024-08-19T14:16:17.801 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1200000000000001e-06 2024-08-19T14:16:17.801 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.802 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.802 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:16:17.802 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.576533+0000", 2024-08-19T14:16:17.802 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.938e-06 2024-08-19T14:16:17.802 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.802 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.802 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:16:17.802 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.576543+0000", 2024-08-19T14:16:17.802 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.8009999999999998e-06 2024-08-19T14:16:17.802 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.802 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.802 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:16:17.802 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.576550+0000", 2024-08-19T14:16:17.803 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.046e-06 2024-08-19T14:16:17.803 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.803 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.803 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:16:17.803 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.583472+0000", 2024-08-19T14:16:17.803 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0069216119999999997 2024-08-19T14:16:17.803 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.803 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.803 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:16:17.803 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.583614+0000", 2024-08-19T14:16:17.803 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00014287800000000001 2024-08-19T14:16:17.803 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.803 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.803 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,8", 2024-08-19T14:16:17.804 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.583755+0000", 2024-08-19T14:16:17.804 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00014006199999999999 2024-08-19T14:16:17.804 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.804 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.804 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:17.804 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.588281+0000", 2024-08-19T14:16:17.804 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0045268779999999998 2024-08-19T14:16:17.804 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.804 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.804 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:16:17.804 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.588476+0000", 2024-08-19T14:16:17.804 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00019452599999999999 2024-08-19T14:16:17.804 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.804 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.805 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:17.805 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.754800+0000", 2024-08-19T14:16:17.805 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.166323729 2024-08-19T14:16:17.805 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.805 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.805 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:16:17.805 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.754855+0000", 2024-08-19T14:16:17.805 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.5670000000000001e-05 2024-08-19T14:16:17.805 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.805 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.805 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:16:17.805 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.754925+0000", 2024-08-19T14:16:17.805 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.9399000000000004e-05 2024-08-19T14:16:17.805 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:17.806 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:17.806 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:17.806 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.806 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.806 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:124120 2.6 2:6f1e6106:::benchmark_data_smithi045_46415_object839:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-08-19T14:16:17.806 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:14:21.953686+0000", 2024-08-19T14:16:17.806 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 115.72256034900001, 2024-08-19T14:16:17.806 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.17718766, 2024-08-19T14:16:17.806 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:16:17.806 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:16:17.806 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:16:17.806 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:16:17.806 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:16:17.807 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:16:17.807 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 124120 2024-08-19T14:16:17.807 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.807 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:16:17.807 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.807 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:16:17.807 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.953686+0000", 2024-08-19T14:16:17.807 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:17.807 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.807 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.807 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:16:17.807 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.953686+0000", 2024-08-19T14:16:17.807 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:17.807 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.808 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.808 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:16:17.808 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.953687+0000", 2024-08-19T14:16:17.808 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5090000000000001e-06 2024-08-19T14:16:17.808 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.808 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.808 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:16:17.808 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.953703+0000", 2024-08-19T14:16:17.808 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6121e-05 2024-08-19T14:16:17.808 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.808 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.808 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:16:17.808 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.953715+0000", 2024-08-19T14:16:17.808 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2072000000000001e-05 2024-08-19T14:16:17.809 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.809 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.809 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:16:17.809 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:21.953723+0000", 2024-08-19T14:16:17.809 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.3529999999999997e-06 2024-08-19T14:16:17.809 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.809 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.809 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:16:17.809 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:22.052557+0000", 2024-08-19T14:16:17.809 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.098834648999999997 2024-08-19T14:16:17.809 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.809 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.809 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:16:17.809 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:22.052674+0000", 2024-08-19T14:16:17.809 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000116774 2024-08-19T14:16:17.810 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.810 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.810 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,8", 2024-08-19T14:16:17.810 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:22.052793+0000", 2024-08-19T14:16:17.810 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000119252 2024-08-19T14:16:17.810 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.810 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.810 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:17.810 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:22.059532+0000", 2024-08-19T14:16:17.810 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0067388470000000001 2024-08-19T14:16:17.810 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.810 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.810 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:16:17.810 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:22.075983+0000", 2024-08-19T14:16:17.811 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.016450854000000001 2024-08-19T14:16:17.811 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.811 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.811 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:17.811 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:22.130704+0000", 2024-08-19T14:16:17.811 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.054720717000000002 2024-08-19T14:16:17.811 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.811 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.811 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:16:17.811 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:22.130801+0000", 2024-08-19T14:16:17.811 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.6916999999999994e-05 2024-08-19T14:16:17.811 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.811 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.811 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:16:17.812 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:22.130873+0000", 2024-08-19T14:16:17.812 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.2595000000000005e-05 2024-08-19T14:16:17.812 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:17.812 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:17.812 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:17.812 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.812 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.812 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:124633 2.6 2:61c364ff:::benchmark_data_smithi045_46415_object1352:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-08-19T14:16:17.812 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:14:22.535924+0000", 2024-08-19T14:16:17.812 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 115.140322461, 2024-08-19T14:16:17.812 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.163485724, 2024-08-19T14:16:17.812 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:16:17.812 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:16:17.812 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:16:17.812 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:16:17.813 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:16:17.813 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:16:17.813 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 124633 2024-08-19T14:16:17.813 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.813 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:16:17.813 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.813 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:16:17.813 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:22.535924+0000", 2024-08-19T14:16:17.813 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:17.813 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.813 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.813 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:16:17.813 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:22.535924+0000", 2024-08-19T14:16:17.813 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:17.814 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.814 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.814 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:16:17.815 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:22.535925+0000", 2024-08-19T14:16:17.816 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4810000000000001e-06 2024-08-19T14:16:17.816 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.816 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.816 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:16:17.816 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:22.535930+0000", 2024-08-19T14:16:17.816 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.9289999999999997e-06 2024-08-19T14:16:17.816 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.816 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.816 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:16:17.816 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:22.535942+0000", 2024-08-19T14:16:17.816 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.216e-05 2024-08-19T14:16:17.816 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.816 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.817 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:16:17.817 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:22.535950+0000", 2024-08-19T14:16:17.817 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.4390000000000003e-06 2024-08-19T14:16:17.817 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.817 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.817 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:16:17.817 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:22.562423+0000", 2024-08-19T14:16:17.817 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.026473647999999999 2024-08-19T14:16:17.817 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.817 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.817 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:16:17.817 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:22.562547+0000", 2024-08-19T14:16:17.817 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012408800000000001 2024-08-19T14:16:17.817 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.817 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.818 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,8", 2024-08-19T14:16:17.818 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:22.562671+0000", 2024-08-19T14:16:17.818 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012323400000000001 2024-08-19T14:16:17.818 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.818 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.818 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:17.818 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:22.568451+0000", 2024-08-19T14:16:17.818 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0057805599999999997 2024-08-19T14:16:17.818 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.818 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.818 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:16:17.818 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:22.572201+0000", 2024-08-19T14:16:17.818 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0037498190000000002 2024-08-19T14:16:17.819 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.819 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.819 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:17.819 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:22.699289+0000", 2024-08-19T14:16:17.819 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.127088338 2024-08-19T14:16:17.819 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.819 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.819 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:16:17.819 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:22.699351+0000", 2024-08-19T14:16:17.819 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.1674999999999995e-05 2024-08-19T14:16:17.819 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:17.819 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:17.819 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:16:17.819 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:14:22.699409+0000", 2024-08-19T14:16:17.820 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.8353000000000002e-05 2024-08-19T14:16:17.820 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:17.820 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:17.820 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:17.820 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:17.820 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:17.820 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:16:17.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:17.821 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:17.821 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:16:17.922 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:16:17.946 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:17.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:17.947 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:16:18.048 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:16:18.071 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:18.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:18.072 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:16:18.091 INFO:teuthology.orchestra.run.smithi031.stderr:set 1.0 pg_upmap mapping to [2,9,0] 2024-08-19T14:16:18.173 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:16:18.198 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:18.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:18.199 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:16:18.300 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:16:18.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:18.340 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:18.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:16:18.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:18 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1593176060' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "1.0", "id": [2, 9, 0]}]: dispatch 2024-08-19T14:16:18.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:18 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "1.0", "id": [2, 9, 0]}]: dispatch 2024-08-19T14:16:18.441 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:16:18.480 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:18.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:18.481 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:16:18.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:18 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1593176060' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "1.0", "id": [2, 9, 0]}]: dispatch 2024-08-19T14:16:18.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:18 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "1.0", "id": [2, 9, 0]}]: dispatch 2024-08-19T14:16:18.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:18 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1593176060' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "1.0", "id": [2, 9, 0]}]: dispatch 2024-08-19T14:16:18.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:18 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "1.0", "id": [2, 9, 0]}]: dispatch 2024-08-19T14:16:18.581 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:16:18.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:18.614 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:18.615 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:16:18.715 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:16:18.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:18.743 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:18.743 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:16:18.845 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:16:18.874 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:18.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:18.875 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:16:18.977 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:16:19.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:19.157 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:19.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:16:19.258 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:16:19.287 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:19.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:19.288 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:16:19.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:19 smithi031 ceph-mon[16336]: pgmap v492: 27 pgs: 27 active+clean; 1.6 GiB data, 6.1 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 870 op/s; 9.7 MiB/s, 155 objects/s recovering 2024-08-19T14:16:19.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:19 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "1.0", "id": [2, 9, 0]}]': finished 2024-08-19T14:16:19.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:19 smithi031 ceph-mon[16336]: osdmap e201: 12 total, 12 up, 10 in 2024-08-19T14:16:19.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:19 smithi031 ceph-mon[16336]: osdmap e202: 12 total, 12 up, 10 in 2024-08-19T14:16:19.389 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:16:19.426 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:19.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:19.427 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:16:19.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:19 smithi032 ceph-mon[20004]: pgmap v492: 27 pgs: 27 active+clean; 1.6 GiB data, 6.1 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 870 op/s; 9.7 MiB/s, 155 objects/s recovering 2024-08-19T14:16:19.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:19 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "1.0", "id": [2, 9, 0]}]': finished 2024-08-19T14:16:19.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:19 smithi032 ceph-mon[20004]: osdmap e201: 12 total, 12 up, 10 in 2024-08-19T14:16:19.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:19 smithi032 ceph-mon[20004]: osdmap e202: 12 total, 12 up, 10 in 2024-08-19T14:16:19.528 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:16:19.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:19 smithi045 ceph-mon[20143]: pgmap v492: 27 pgs: 27 active+clean; 1.6 GiB data, 6.1 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 870 op/s; 9.7 MiB/s, 155 objects/s recovering 2024-08-19T14:16:19.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:19 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "1.0", "id": [2, 9, 0]}]': finished 2024-08-19T14:16:19.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:19 smithi045 ceph-mon[20143]: osdmap e201: 12 total, 12 up, 10 in 2024-08-19T14:16:19.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:19 smithi045 ceph-mon[20143]: osdmap e202: 12 total, 12 up, 10 in 2024-08-19T14:16:19.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:19.580 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:19.580 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:16:19.681 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:16:19.709 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:19.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:19.710 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:16:19.811 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:16:19.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:19.848 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:19.848 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:16:19.949 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:16:19.976 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:19.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:19.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:16:20.078 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:16:20.109 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:20.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:20.110 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:16:20.211 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:16:20.234 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:20.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:20.235 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:16:20.308 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-08-19T14:16:20.320 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:16:20.336 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:16:20.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:20.392 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:20.392 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:16:20.493 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:16:20.518 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:20.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:20.519 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:16:20.620 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:16:20.646 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:20.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:20.647 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:16:20.748 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:16:20.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:20.775 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:20.775 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:16:20.817 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:16:20.875 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:16:20.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:20.944 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:20.944 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:16:21.023 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:16:21.045 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:16:21.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:21.095 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:21.096 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:16:21.196 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:16:21.226 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:21.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:21.227 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:16:21.328 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:16:21.357 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:21 smithi031 ceph-mon[16336]: pgmap v495: 27 pgs: 27 active+clean; 1.5 GiB data, 5.7 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 826 op/s; 6.0 MiB/s, 95 objects/s recovering 2024-08-19T14:16:21.357 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:21 smithi031 ceph-mon[16336]: osdmap e203: 12 total, 12 up, 10 in 2024-08-19T14:16:21.360 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:21.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:21.361 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:16:21.462 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:16:21.499 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:21.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:21.500 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:16:21.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:21 smithi032 ceph-mon[20004]: pgmap v495: 27 pgs: 27 active+clean; 1.5 GiB data, 5.7 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 826 op/s; 6.0 MiB/s, 95 objects/s recovering 2024-08-19T14:16:21.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:21 smithi032 ceph-mon[20004]: osdmap e203: 12 total, 12 up, 10 in 2024-08-19T14:16:21.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:21 smithi045 ceph-mon[20143]: pgmap v495: 27 pgs: 27 active+clean; 1.5 GiB data, 5.7 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 826 op/s; 6.0 MiB/s, 95 objects/s recovering 2024-08-19T14:16:21.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:21 smithi045 ceph-mon[20143]: osdmap e203: 12 total, 12 up, 10 in 2024-08-19T14:16:21.601 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:16:21.627 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:21.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:21.628 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:16:21.666 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:16:21.667 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [ 2024-08-19T14:16:21.667 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:21.667 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:226421 2.e 2:768c65f7:::benchmark_data_smithi045_46415_object103140:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-08-19T14:16:21.667 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:16:21.627355+0000", 2024-08-19T14:16:21.667 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.034562201000000001, 2024-08-19T14:16:21.667 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.034594311000000003, 2024-08-19T14:16:21.667 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:16:21.667 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:16:21.667 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "waiting for sub ops", 2024-08-19T14:16:21.667 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:16:21.667 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:16:21.667 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:16:21.668 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 226421 2024-08-19T14:16:21.668 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:21.668 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:16:21.668 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:21.668 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:16:21.668 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:21.627355+0000", 2024-08-19T14:16:21.668 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:21.668 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:21.668 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:21.668 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:16:21.668 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:21.627355+0000", 2024-08-19T14:16:21.668 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:21.668 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:21.668 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:21.669 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:16:21.669 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:21.627357+0000", 2024-08-19T14:16:21.669 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.939e-06 2024-08-19T14:16:21.669 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:21.669 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:21.669 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:16:21.669 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:21.627364+0000", 2024-08-19T14:16:21.669 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.9110000000000004e-06 2024-08-19T14:16:21.669 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:21.669 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:21.669 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:16:21.669 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:21.627379+0000", 2024-08-19T14:16:21.669 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4837e-05 2024-08-19T14:16:21.669 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:21.670 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:21.670 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:16:21.670 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:21.627387+0000", 2024-08-19T14:16:21.670 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.2090000000000005e-06 2024-08-19T14:16:21.670 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:21.670 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:21.670 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:16:21.670 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:21.643990+0000", 2024-08-19T14:16:21.670 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.016603693999999999 2024-08-19T14:16:21.670 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:21.670 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:21.670 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:16:21.670 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:21.644264+0000", 2024-08-19T14:16:21.671 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00027372200000000001 2024-08-19T14:16:21.671 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:21.671 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:21.671 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 7,11", 2024-08-19T14:16:21.671 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:21.644644+0000", 2024-08-19T14:16:21.671 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00037994899999999998 2024-08-19T14:16:21.671 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:21.671 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:21.671 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:21.671 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:21.656569+0000", 2024-08-19T14:16:21.671 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.011924778 2024-08-19T14:16:21.671 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:21.671 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:21.672 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:21.672 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:21.656608+0000", 2024-08-19T14:16:21.672 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.8962999999999999e-05 2024-08-19T14:16:21.672 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:21.672 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:21.672 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:21.672 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:21.672 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:21.672 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.24580.0:226431 2.4 e204/146 2:29ad6425:::benchmark_data_smithi045_46415_object103150:head v 204'14181, mlcod=204'14179)", 2024-08-19T14:16:21.672 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:16:21.644989+0000", 2024-08-19T14:16:21.672 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.016928529000000001, 2024-08-19T14:16:21.672 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.017050704999999999, 2024-08-19T14:16:21.672 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:16:21.673 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:16:21.673 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-08-19T14:16:21.673 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:16:21.673 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:21.673 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:16:21.673 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:21.644989+0000", 2024-08-19T14:16:21.673 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:21.673 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:21.673 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:21.673 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:16:21.673 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:21.644989+0000", 2024-08-19T14:16:21.673 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:21.673 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:21.674 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:21.674 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:16:21.674 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:21.644989+0000", 2024-08-19T14:16:21.674 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.6899999999999996e-07 2024-08-19T14:16:21.674 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:21.674 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:21.674 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:16:21.674 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:21.644995+0000", 2024-08-19T14:16:21.674 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.7699999999999998e-06 2024-08-19T14:16:21.674 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:21.674 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:21.674 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:16:21.674 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:21.645005+0000", 2024-08-19T14:16:21.674 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0200000000000001e-05 2024-08-19T14:16:21.675 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:21.675 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:21.675 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:16:21.675 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:21.645017+0000", 2024-08-19T14:16:21.675 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1396000000000001e-05 2024-08-19T14:16:21.675 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:21.675 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:21.675 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:16:21.675 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:21.648563+0000", 2024-08-19T14:16:21.675 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0035458270000000001 2024-08-19T14:16:21.675 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:21.675 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:21.675 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:16:21.676 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:21.648610+0000", 2024-08-19T14:16:21.676 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7207999999999999e-05 2024-08-19T14:16:21.676 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:21.676 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:21.676 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:21.676 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:21.676 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:21.676 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.24580.0:226437 2.3 e204/102 2:c26bca2a:::benchmark_data_smithi045_46415_object103156:head v 204'14135, mlcod=204'14133)", 2024-08-19T14:16:21.676 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:16:21.645092+0000", 2024-08-19T14:16:21.676 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.016825008999999998, 2024-08-19T14:16:21.676 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.017005861000000001, 2024-08-19T14:16:21.676 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:16:21.676 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:16:21.677 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-08-19T14:16:21.677 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:16:21.677 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:21.677 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:16:21.677 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:21.645092+0000", 2024-08-19T14:16:21.677 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:21.677 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:21.677 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:21.677 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:16:21.677 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:21.645092+0000", 2024-08-19T14:16:21.677 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:21.677 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:21.677 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:21.678 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:16:21.678 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:21.645092+0000", 2024-08-19T14:16:21.678 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.53e-07 2024-08-19T14:16:21.678 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:21.678 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:21.678 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:16:21.678 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:21.645099+0000", 2024-08-19T14:16:21.678 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.6769999999999998e-06 2024-08-19T14:16:21.678 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:21.678 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:21.678 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:16:21.678 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:21.645106+0000", 2024-08-19T14:16:21.678 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.8430000000000004e-06 2024-08-19T14:16:21.679 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:21.679 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:21.679 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:16:21.679 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:21.645110+0000", 2024-08-19T14:16:21.679 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4580000000000003e-06 2024-08-19T14:16:21.679 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:21.679 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:21.679 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:16:21.679 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:21.649060+0000", 2024-08-19T14:16:21.679 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0039500730000000001 2024-08-19T14:16:21.679 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:21.679 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:21.679 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:16:21.680 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:21.649079+0000", 2024-08-19T14:16:21.680 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.8468999999999999e-05 2024-08-19T14:16:21.680 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:21.680 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:21.680 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:21.680 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:21.680 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:21.680 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:226441 2.7 2:ef23e4c0:::benchmark_data_smithi045_46415_object103160:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-08-19T14:16:21.680 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:16:21.650342+0000", 2024-08-19T14:16:21.680 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.011575454000000001, 2024-08-19T14:16:21.680 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.011813888999999999, 2024-08-19T14:16:21.680 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:16:21.680 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:16:21.681 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "waiting for sub ops", 2024-08-19T14:16:21.681 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:16:21.681 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:16:21.681 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:16:21.681 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 226441 2024-08-19T14:16:21.681 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:21.681 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:16:21.681 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:21.681 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:16:21.681 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:21.650342+0000", 2024-08-19T14:16:21.681 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:21.681 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:21.681 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:21.682 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:16:21.682 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:21.650342+0000", 2024-08-19T14:16:21.682 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:21.682 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:21.682 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:21.682 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:16:21.682 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:21.650343+0000", 2024-08-19T14:16:21.682 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.519e-06 2024-08-19T14:16:21.682 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:21.682 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:21.683 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:16:21.683 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:21.650349+0000", 2024-08-19T14:16:21.683 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.054e-06 2024-08-19T14:16:21.683 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:21.683 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:21.683 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:16:21.683 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:21.650362+0000", 2024-08-19T14:16:21.683 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2646000000000001e-05 2024-08-19T14:16:21.683 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:21.683 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:21.683 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:16:21.684 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:21.650370+0000", 2024-08-19T14:16:21.684 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.2689999999999999e-06 2024-08-19T14:16:21.684 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:21.684 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:21.684 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:16:21.684 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:21.650441+0000", 2024-08-19T14:16:21.684 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.1027999999999998e-05 2024-08-19T14:16:21.684 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:21.684 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:21.684 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:16:21.684 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:21.650556+0000", 2024-08-19T14:16:21.684 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000114616 2024-08-19T14:16:21.685 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:21.685 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:21.685 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,4", 2024-08-19T14:16:21.685 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:21.650667+0000", 2024-08-19T14:16:21.685 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000111576 2024-08-19T14:16:21.685 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:21.685 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:21.685 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:21.685 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:21.685 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:21.685 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:226448 2.1 2:8524c1da:::benchmark_data_smithi045_46415_object103167:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-08-19T14:16:21.685 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:16:21.658400+0000", 2024-08-19T14:16:21.685 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.0035169160000000001, 2024-08-19T14:16:21.686 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0038242889999999998, 2024-08-19T14:16:21.686 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:16:21.686 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:16:21.686 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "waiting for sub ops", 2024-08-19T14:16:21.686 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:16:21.686 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:16:21.686 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:16:21.686 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 226448 2024-08-19T14:16:21.686 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:21.686 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:16:21.686 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:21.686 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:16:21.687 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:21.658400+0000", 2024-08-19T14:16:21.687 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:21.687 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:21.687 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:21.687 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:16:21.687 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:21.658400+0000", 2024-08-19T14:16:21.687 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:21.687 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:21.687 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:21.687 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:16:21.687 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:21.658402+0000", 2024-08-19T14:16:21.687 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5650000000000001e-06 2024-08-19T14:16:21.687 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:21.687 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:21.688 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:16:21.688 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:21.658407+0000", 2024-08-19T14:16:21.688 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0900000000000004e-06 2024-08-19T14:16:21.688 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:21.688 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:21.688 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:16:21.688 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:21.658421+0000", 2024-08-19T14:16:21.688 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3813e-05 2024-08-19T14:16:21.688 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:21.688 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:21.688 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:16:21.688 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:21.658429+0000", 2024-08-19T14:16:21.688 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.3280000000000006e-06 2024-08-19T14:16:21.689 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:21.689 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:21.689 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:16:21.689 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:21.658619+0000", 2024-08-19T14:16:21.689 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00019014499999999999 2024-08-19T14:16:21.689 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:21.689 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:21.689 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:16:21.689 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:21.661377+0000", 2024-08-19T14:16:21.689 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0027578730000000001 2024-08-19T14:16:21.689 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:21.689 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:21.689 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,8", 2024-08-19T14:16:21.689 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:21.661544+0000", 2024-08-19T14:16:21.690 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000167453 2024-08-19T14:16:21.690 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:21.690 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:21.690 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:21.690 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:21.690 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:21.690 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop_reply(client.24580.0:226441 2.7 e204/164)", 2024-08-19T14:16:21.690 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:16:21.658886+0000", 2024-08-19T14:16:21.690 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.0030308380000000001, 2024-08-19T14:16:21.690 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0034016229999999999, 2024-08-19T14:16:21.690 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:16:21.690 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:16:21.690 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "queued for pg", 2024-08-19T14:16:21.691 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:16:21.691 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:21.691 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:16:21.691 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:21.658886+0000", 2024-08-19T14:16:21.691 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:21.691 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:21.691 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:21.691 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:16:21.691 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:21.658886+0000", 2024-08-19T14:16:21.691 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:21.691 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:21.691 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:21.691 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:16:21.692 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:21.658887+0000", 2024-08-19T14:16:21.692 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.7199999999999997e-07 2024-08-19T14:16:21.692 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:21.692 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:21.692 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:16:21.692 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:21.658893+0000", 2024-08-19T14:16:21.692 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.071e-06 2024-08-19T14:16:21.692 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:21.692 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:21.693 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:16:21.693 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:21.658911+0000", 2024-08-19T14:16:21.693 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.7581999999999999e-05 2024-08-19T14:16:21.693 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:21.693 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:21.693 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:16:21.693 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:21.658918+0000", 2024-08-19T14:16:21.693 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.8129999999999998e-06 2024-08-19T14:16:21.693 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:21.693 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:21.694 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:21.694 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:21.694 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:21.694 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.24580.0:226445 2.3 e204/102 2:c05dc6da:::benchmark_data_smithi045_46415_object103164:head v 204'14136, mlcod=204'14133)", 2024-08-19T14:16:21.694 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:16:21.658947+0000", 2024-08-19T14:16:21.694 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.00296956, 2024-08-19T14:16:21.694 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0033863640000000002, 2024-08-19T14:16:21.694 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:16:21.694 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:16:21.694 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-08-19T14:16:21.694 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:16:21.694 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:21.694 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:16:21.695 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:21.658947+0000", 2024-08-19T14:16:21.695 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:21.695 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:21.695 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:21.695 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:16:21.695 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:21.658947+0000", 2024-08-19T14:16:21.695 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:21.695 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:21.695 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:21.695 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:16:21.695 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:21.658948+0000", 2024-08-19T14:16:21.695 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.5400000000000002e-07 2024-08-19T14:16:21.695 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:21.695 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:21.696 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:16:21.696 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:21.658953+0000", 2024-08-19T14:16:21.696 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.4419999999999997e-06 2024-08-19T14:16:21.696 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:21.696 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:21.696 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:16:21.696 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:21.658961+0000", 2024-08-19T14:16:21.696 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.4440000000000002e-06 2024-08-19T14:16:21.696 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:21.696 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:21.696 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:16:21.696 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:21.658966+0000", 2024-08-19T14:16:21.696 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0529999999999997e-06 2024-08-19T14:16:21.696 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:21.697 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:21.697 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:16:21.697 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:21.659114+0000", 2024-08-19T14:16:21.697 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000148305 2024-08-19T14:16:21.697 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:21.697 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:21.697 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:16:21.697 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:21.659133+0000", 2024-08-19T14:16:21.697 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.8938999999999999e-05 2024-08-19T14:16:21.697 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:21.697 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:21.697 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:21.697 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:21.697 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:21.698 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.24580.0:226444 2.4 e204/146 2:218c2b1c:::benchmark_data_smithi045_46415_object103163:head v 204'14182, mlcod=204'14179)", 2024-08-19T14:16:21.698 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:16:21.658987+0000", 2024-08-19T14:16:21.698 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.0029304700000000001, 2024-08-19T14:16:21.698 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0034038390000000001, 2024-08-19T14:16:21.698 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:16:21.698 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:16:21.698 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-08-19T14:16:21.698 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:16:21.698 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:21.698 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:16:21.698 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:21.658987+0000", 2024-08-19T14:16:21.698 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:21.698 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:21.698 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:21.699 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:16:21.699 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:21.658987+0000", 2024-08-19T14:16:21.699 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:21.699 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:21.699 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:21.699 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:16:21.699 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:21.658987+0000", 2024-08-19T14:16:21.699 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.03e-07 2024-08-19T14:16:21.699 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:21.699 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:21.699 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:16:21.699 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:21.658991+0000", 2024-08-19T14:16:21.699 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.0470000000000004e-06 2024-08-19T14:16:21.699 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:21.699 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:21.700 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:16:21.700 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:21.658997+0000", 2024-08-19T14:16:21.700 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.3450000000000002e-06 2024-08-19T14:16:21.700 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:21.700 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:21.700 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:16:21.700 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:21.659001+0000", 2024-08-19T14:16:21.700 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.1559999999999997e-06 2024-08-19T14:16:21.700 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:21.700 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:21.700 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:16:21.700 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:21.662003+0000", 2024-08-19T14:16:21.700 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0030011249999999999 2024-08-19T14:16:21.700 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:21.701 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:21.701 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:16:21.701 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:21.662022+0000", 2024-08-19T14:16:21.701 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.9548e-05 2024-08-19T14:16:21.701 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:21.701 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:21.701 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:21.701 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:21.701 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-08-19T14:16:21.701 INFO:teuthology.orchestra.run.smithi031.stdout: "num_ops": 8 2024-08-19T14:16:21.701 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:16:21.729 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:16:21.750 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:21.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:21.752 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:16:21.852 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:16:21.874 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:21.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:21.875 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:16:21.976 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:16:22.002 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:22.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:22.004 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:16:22.105 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:16:22.111 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is set 2024-08-19T14:16:22.128 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:22.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:22.129 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:16:22.230 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:16:22.256 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:22.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:22.257 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:16:22.325 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:22 smithi031 ceph-mon[16336]: osdmap e204: 12 total, 12 up, 10 in 2024-08-19T14:16:22.325 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:22 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2027302048' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:16:22.325 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:22 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:16:22.358 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:16:22.373 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-08-19T14:16:22.402 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:22.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:22.403 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:16:22.504 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:16:22.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:22 smithi032 ceph-mon[20004]: osdmap e204: 12 total, 12 up, 10 in 2024-08-19T14:16:22.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:22 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2027302048' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:16:22.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:22 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:16:22.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:22.529 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:22.529 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:16:22.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:22 smithi045 ceph-mon[20143]: osdmap e204: 12 total, 12 up, 10 in 2024-08-19T14:16:22.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:22 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2027302048' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:16:22.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:22 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:16:22.630 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:16:22.687 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:22.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:22.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:16:22.789 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:16:22.814 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:22.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:22.815 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:16:22.916 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:16:22.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:22.943 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:22.943 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:16:23.047 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:16:23.074 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:23.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:23.075 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:16:23.119 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:16:23.176 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:16:23.180 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:16:23.222 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:23.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:23.223 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:16:23.259 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:23 smithi031 ceph-mon[16336]: pgmap v498: 27 pgs: 1 peering, 26 active+clean; 1.3 GiB data, 5.4 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 1.01k op/s 2024-08-19T14:16:23.259 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:23 smithi031 ceph-mon[16336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:16:23.259 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:23 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:16:23.259 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:23 smithi031 ceph-mon[16336]: osdmap e205: 12 total, 12 up, 10 in 2024-08-19T14:16:23.324 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:16:23.352 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:23.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:23.353 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:16:23.454 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:16:23.503 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:23.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:23.504 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:16:23.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:23 smithi032 ceph-mon[20004]: pgmap v498: 27 pgs: 1 peering, 26 active+clean; 1.3 GiB data, 5.4 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 1.01k op/s 2024-08-19T14:16:23.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:23 smithi032 ceph-mon[20004]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:16:23.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:23 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:16:23.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:23 smithi032 ceph-mon[20004]: osdmap e205: 12 total, 12 up, 10 in 2024-08-19T14:16:23.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:23 smithi045 ceph-mon[20143]: pgmap v498: 27 pgs: 1 peering, 26 active+clean; 1.3 GiB data, 5.4 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 1.01k op/s 2024-08-19T14:16:23.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:23 smithi045 ceph-mon[20143]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:16:23.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:23 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:16:23.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:23 smithi045 ceph-mon[20143]: osdmap e205: 12 total, 12 up, 10 in 2024-08-19T14:16:23.605 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:16:23.637 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:23.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:23.638 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:16:23.739 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:16:23.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:23.778 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:23.778 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:16:23.879 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:16:23.902 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:23.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:23.903 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:16:23.962 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:16:23.963 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:16:23.986 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:16:23.987 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:16:24.004 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:16:24.037 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:16:24.037 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:16:24.066 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:24.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:24.067 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:16:24.080 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:16:24.080 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:16:24.097 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:16:24.098 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:16:24.140 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:16:24.141 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:16:24.168 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:16:24.170 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:16:24.171 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:16:24.194 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:24.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:24.195 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:16:24.202 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:16:24.203 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:16:24.234 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:16:24.234 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:16:24.270 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:16:24.270 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:16:24.295 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:16:24.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:24 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3936821357' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:16:24.307 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:16:24.307 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:16:24.319 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:24.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:24.320 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:16:24.337 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:16:24.338 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:16:24.366 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:16:24.367 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:16:24.421 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:16:24.445 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:24.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:24.446 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:16:24.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:24 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3936821357' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:16:24.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:24 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3936821357' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:16:24.547 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:16:24.580 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:24.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:24.581 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:16:24.682 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:16:24.690 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 8, 9, 11] out_osds: [10, 6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-08-19T14:16:24.690 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-19T14:16:24.690 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-08-19T14:16:24.691 INFO:tasks.thrashosds.thrasher:Setting osd 2 primary_affinity to 0.073801 2024-08-19T14:16:24.691 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd primary-affinity 2 0.07380103028877427 2024-08-19T14:16:24.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:24.716 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:24.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:16:24.779 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-08-19T14:16:24.818 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:16:24.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:24.846 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:24.846 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:16:24.947 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:16:24.970 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:24.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:24.971 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:16:25.072 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:16:25.072 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-19T14:16:25.110 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:25.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:25.112 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:16:25.213 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:16:25.235 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:25.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:25.237 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:16:25.337 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:16:25.362 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:25.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:25.363 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:16:25.407 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:25 smithi032 ceph-mon[20004]: pgmap v500: 27 pgs: 1 peering, 26 active+clean; 1.2 GiB data, 5.0 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 948 op/s 2024-08-19T14:16:25.447 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:25 smithi045 ceph-mon[20143]: pgmap v500: 27 pgs: 1 peering, 26 active+clean; 1.2 GiB data, 5.0 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 948 op/s 2024-08-19T14:16:25.464 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:16:25.465 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:25 smithi031 ceph-mon[16336]: pgmap v500: 27 pgs: 1 peering, 26 active+clean; 1.2 GiB data, 5.0 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 948 op/s 2024-08-19T14:16:25.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:25.498 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:25.498 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:16:25.599 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:16:25.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:25.625 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:25.625 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:16:25.642 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:16:25.652 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:16:25.680 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:16:25.726 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:16:25.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:25.758 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:25.758 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:16:25.859 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:16:25.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:25.893 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:25.893 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:16:25.994 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:16:26.024 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:26.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:26.025 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:16:26.126 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:16:26.149 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:26.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:26.150 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:16:26.251 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:16:26.285 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:26.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:26.286 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:16:26.387 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:16:26.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:26.415 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:26.415 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:16:26.516 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:16:26.553 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:26.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:26.554 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:16:26.655 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:16:26.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:26.690 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:26.690 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:16:26.791 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:16:26.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:26.818 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:26.818 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:16:26.919 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:16:26.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:26.955 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:26.956 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:16:27.057 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:16:27.084 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:27.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:27.085 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:16:27.186 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:16:27.191 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:27 smithi031 ceph-mon[16336]: pgmap v501: 27 pgs: 1 peering, 26 active+clean; 1.1 GiB data, 4.7 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 844 op/s 2024-08-19T14:16:27.207 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:27.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:27.208 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:16:27.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:27 smithi032 ceph-mon[20004]: pgmap v501: 27 pgs: 1 peering, 26 active+clean; 1.1 GiB data, 4.7 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 844 op/s 2024-08-19T14:16:27.309 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:16:27.340 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:27.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:27.341 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:16:27.442 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:16:27.466 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:27.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:27.467 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:16:27.519 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:16:27.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:27 smithi045 ceph-mon[20143]: pgmap v501: 27 pgs: 1 peering, 26 active+clean; 1.1 GiB data, 4.7 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 844 op/s 2024-08-19T14:16:27.546 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:16:27.546 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:16:27.568 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:16:27.590 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:27.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:27.591 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:16:27.692 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:16:27.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:27.731 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:27.732 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:16:27.833 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:16:27.862 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:27.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:27.863 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:16:27.965 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:16:27.989 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:27.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:27.990 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:16:28.091 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:16:28.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:28.126 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:28.126 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:16:28.140 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is unset 2024-08-19T14:16:28.227 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:16:28.253 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:28.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:28.255 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:16:28.356 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:16:28.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:28 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3881917992' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:16:28.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:28 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:16:28.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:28 smithi031 ceph-mon[16336]: pgmap v502: 27 pgs: 27 active+clean; 1.1 GiB data, 4.5 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 842 op/s 2024-08-19T14:16:28.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:28.393 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:28.393 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:16:28.473 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-08-19T14:16:28.493 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:16:28.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:28 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3881917992' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:16:28.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:28 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:16:28.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:28 smithi032 ceph-mon[20004]: pgmap v502: 27 pgs: 27 active+clean; 1.1 GiB data, 4.5 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 842 op/s 2024-08-19T14:16:28.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:28.520 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:28.520 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:16:28.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:28 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3881917992' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:16:28.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:28 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:16:28.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:28 smithi045 ceph-mon[20143]: pgmap v502: 27 pgs: 27 active+clean; 1.1 GiB data, 4.5 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 842 op/s 2024-08-19T14:16:28.621 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:16:28.647 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:28.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:28.648 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:16:28.749 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:16:28.778 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:28.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:28.779 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:16:28.880 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:16:28.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:28.915 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:28.915 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:16:29.017 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:16:29.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:29.047 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:29.047 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:16:29.117 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:16:29.147 INFO:teuthology.orchestra.run.smithi031.stderr:set osd.2 primary-affinity to 0.073801 (848362) 2024-08-19T14:16:29.149 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:16:29.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:29.203 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:29.203 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:16:29.304 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:16:29.325 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:29.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:29.326 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:16:29.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:29 smithi031 ceph-mon[16336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:16:29.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:29 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:16:29.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:29 smithi031 ceph-mon[16336]: osdmap e206: 12 total, 12 up, 10 in 2024-08-19T14:16:29.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:29 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/700244456' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.07380103028877427}]: dispatch 2024-08-19T14:16:29.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:29 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.07380103028877427}]: dispatch 2024-08-19T14:16:29.427 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:16:29.448 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:29.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:29.449 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:16:29.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:29 smithi032 ceph-mon[20004]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:16:29.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:29 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:16:29.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:29 smithi032 ceph-mon[20004]: osdmap e206: 12 total, 12 up, 10 in 2024-08-19T14:16:29.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:29 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/700244456' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.07380103028877427}]: dispatch 2024-08-19T14:16:29.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:29 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.07380103028877427}]: dispatch 2024-08-19T14:16:29.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:29 smithi045 ceph-mon[20143]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:16:29.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:29 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:16:29.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:29 smithi045 ceph-mon[20143]: osdmap e206: 12 total, 12 up, 10 in 2024-08-19T14:16:29.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:29 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/700244456' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.07380103028877427}]: dispatch 2024-08-19T14:16:29.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:29 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.07380103028877427}]: dispatch 2024-08-19T14:16:29.550 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:16:29.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:29.572 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:29.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:16:29.673 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:16:29.694 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:29.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:29.695 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:16:29.796 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:16:29.817 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:29.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:29.818 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:16:29.920 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:16:29.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:29.942 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:29.943 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:16:30.043 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:16:30.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:30.069 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:30.069 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:16:30.153 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:16:30.170 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:16:30.181 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:16:30.182 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:16:30.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:30.202 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:30.202 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:16:30.212 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:16:30.212 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:16:30.236 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:16:30.237 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:16:30.259 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:16:30.259 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:16:30.291 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:16:30.291 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:16:30.304 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:16:30.326 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:16:30.327 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:16:30.327 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:30 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.07380103028877427}]': finished 2024-08-19T14:16:30.327 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:30 smithi031 ceph-mon[16336]: osdmap e207: 12 total, 12 up, 10 in 2024-08-19T14:16:30.327 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:30 smithi031 ceph-mon[16336]: 1.4 scrub starts 2024-08-19T14:16:30.327 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:30 smithi031 ceph-mon[16336]: 1.4 scrub ok 2024-08-19T14:16:30.327 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:30 smithi031 ceph-mon[16336]: pgmap v505: 27 pgs: 27 active+clean; 927 MiB data, 4.0 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 917 op/s 2024-08-19T14:16:30.328 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:30 smithi031 ceph-mon[16336]: osdmap e208: 12 total, 12 up, 10 in 2024-08-19T14:16:30.328 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:30 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/507053191' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:16:30.329 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:30.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:30.330 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:16:30.362 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:16:30.362 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:16:30.399 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:16:30.400 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:16:30.430 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:16:30.432 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:16:30.432 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:16:30.465 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:30.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:30.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:16:30.466 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:16:30.466 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:16:30.505 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:16:30.506 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:16:30.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:30 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.07380103028877427}]': finished 2024-08-19T14:16:30.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:30 smithi032 ceph-mon[20004]: osdmap e207: 12 total, 12 up, 10 in 2024-08-19T14:16:30.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:30 smithi032 ceph-mon[20004]: 1.4 scrub starts 2024-08-19T14:16:30.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:30 smithi032 ceph-mon[20004]: 1.4 scrub ok 2024-08-19T14:16:30.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:30 smithi032 ceph-mon[20004]: pgmap v505: 27 pgs: 27 active+clean; 927 MiB data, 4.0 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 917 op/s 2024-08-19T14:16:30.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:30 smithi032 ceph-mon[20004]: osdmap e208: 12 total, 12 up, 10 in 2024-08-19T14:16:30.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:30 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/507053191' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:16:30.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:30 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.07380103028877427}]': finished 2024-08-19T14:16:30.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:30 smithi045 ceph-mon[20143]: osdmap e207: 12 total, 12 up, 10 in 2024-08-19T14:16:30.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:30 smithi045 ceph-mon[20143]: 1.4 scrub starts 2024-08-19T14:16:30.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:30 smithi045 ceph-mon[20143]: 1.4 scrub ok 2024-08-19T14:16:30.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:30 smithi045 ceph-mon[20143]: pgmap v505: 27 pgs: 27 active+clean; 927 MiB data, 4.0 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 917 op/s 2024-08-19T14:16:30.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:30 smithi045 ceph-mon[20143]: osdmap e208: 12 total, 12 up, 10 in 2024-08-19T14:16:30.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:30 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/507053191' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:16:30.552 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:16:30.552 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:16:30.567 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:16:30.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:30.606 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:30.606 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:16:30.708 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:16:30.747 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:30.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:30.748 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:16:30.797 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:16:30.797 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [ 2024-08-19T14:16:30.797 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:30.797 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.24580.0:233869 2.a e208/102 2:55da1ffe:::benchmark_data_smithi045_46415_object110588:head v 208'14661, mlcod=208'14658)", 2024-08-19T14:16:30.798 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:16:30.783903+0000", 2024-08-19T14:16:30.798 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.0083495640000000003, 2024-08-19T14:16:30.798 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0083792740000000008, 2024-08-19T14:16:30.798 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:16:30.798 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:16:30.798 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-08-19T14:16:30.798 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:16:30.798 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:30.798 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:16:30.798 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:30.783903+0000", 2024-08-19T14:16:30.798 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:30.798 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:30.799 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:30.799 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:16:30.799 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:30.783903+0000", 2024-08-19T14:16:30.799 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:30.799 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:30.799 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:30.799 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:16:30.799 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:30.783904+0000", 2024-08-19T14:16:30.799 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.6800000000000009e-07 2024-08-19T14:16:30.799 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:30.799 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:30.800 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:16:30.800 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:30.783912+0000", 2024-08-19T14:16:30.800 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.1319999999999994e-06 2024-08-19T14:16:30.800 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:30.800 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:30.800 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:16:30.800 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:30.783925+0000", 2024-08-19T14:16:30.800 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3308e-05 2024-08-19T14:16:30.800 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:30.800 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:30.800 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:16:30.800 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:30.783933+0000", 2024-08-19T14:16:30.801 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.5390000000000002e-06 2024-08-19T14:16:30.801 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:30.801 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:30.801 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:16:30.801 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:30.784721+0000", 2024-08-19T14:16:30.801 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00078837000000000004 2024-08-19T14:16:30.801 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:30.801 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:30.801 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:16:30.801 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:30.784740+0000", 2024-08-19T14:16:30.801 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.8422000000000001e-05 2024-08-19T14:16:30.801 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:30.802 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:30.802 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:30.802 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:30.802 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:30.802 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.24580.0:233870 2.6 e208/102 2:64f17d01:::benchmark_data_smithi045_46415_object110589:head v 208'15729, mlcod=208'15727)", 2024-08-19T14:16:30.802 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:16:30.784678+0000", 2024-08-19T14:16:30.802 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.0075746160000000002, 2024-08-19T14:16:30.802 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0076720790000000001, 2024-08-19T14:16:30.802 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:16:30.802 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:16:30.802 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-08-19T14:16:30.803 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:16:30.803 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:30.803 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:16:30.803 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:30.784678+0000", 2024-08-19T14:16:30.803 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:30.803 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:30.803 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:30.803 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:16:30.803 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:30.784678+0000", 2024-08-19T14:16:30.803 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:30.803 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:30.803 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:30.804 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:16:30.804 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:30.784678+0000", 2024-08-19T14:16:30.804 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.8800000000000003e-07 2024-08-19T14:16:30.804 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:30.804 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:30.804 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:16:30.804 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:30.784684+0000", 2024-08-19T14:16:30.804 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.8459999999999996e-06 2024-08-19T14:16:30.804 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:30.804 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:30.804 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:16:30.805 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:30.784698+0000", 2024-08-19T14:16:30.805 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3404000000000001e-05 2024-08-19T14:16:30.805 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:30.805 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:30.805 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:16:30.805 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:30.784706+0000", 2024-08-19T14:16:30.805 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.1459999999999996e-06 2024-08-19T14:16:30.805 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:30.805 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:30.805 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:16:30.805 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:30.785003+0000", 2024-08-19T14:16:30.805 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00029687600000000001 2024-08-19T14:16:30.805 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:30.806 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:30.806 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:16:30.806 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:30.785061+0000", 2024-08-19T14:16:30.806 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.8656000000000002e-05 2024-08-19T14:16:30.806 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:30.806 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:30.806 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:30.806 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:30.806 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:30.806 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.24580.0:233881 2.d e208/102 2:ba97b142:::benchmark_data_smithi045_46415_object110600:head v 208'14609, mlcod=208'14607)", 2024-08-19T14:16:30.806 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:16:30.788951+0000", 2024-08-19T14:16:30.807 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.0033014379999999999, 2024-08-19T14:16:30.807 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0034709200000000002, 2024-08-19T14:16:30.807 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:16:30.807 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:16:30.807 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-08-19T14:16:30.807 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:16:30.807 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:30.807 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:16:30.807 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:30.788951+0000", 2024-08-19T14:16:30.807 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:30.807 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:30.808 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:30.808 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:16:30.808 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:30.788951+0000", 2024-08-19T14:16:30.808 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:30.808 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:30.808 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:30.808 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:16:30.808 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:30.788952+0000", 2024-08-19T14:16:30.808 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2330000000000001e-06 2024-08-19T14:16:30.808 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:30.808 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:30.808 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:16:30.808 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:30.788961+0000", 2024-08-19T14:16:30.809 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.2949999999999994e-06 2024-08-19T14:16:30.809 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:30.809 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:30.809 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:16:30.809 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:30.788980+0000", 2024-08-19T14:16:30.809 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.8992e-05 2024-08-19T14:16:30.809 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:30.809 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:30.809 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:16:30.809 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:30.788989+0000", 2024-08-19T14:16:30.809 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.1009999999999998e-06 2024-08-19T14:16:30.809 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:30.810 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:30.810 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:16:30.810 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:30.790577+0000", 2024-08-19T14:16:30.810 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.001588302 2024-08-19T14:16:30.810 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:30.810 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:30.810 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:16:30.810 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:30.790627+0000", 2024-08-19T14:16:30.810 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0382e-05 2024-08-19T14:16:30.810 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:30.810 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:30.810 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:30.811 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:30.811 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:30.811 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.24580.0:233872 2.6 e208/102 2:66ecd168:::benchmark_data_smithi045_46415_object110591:head v 208'15730, mlcod=208'15727)", 2024-08-19T14:16:30.811 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:16:30.790231+0000", 2024-08-19T14:16:30.811 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.0020214880000000001, 2024-08-19T14:16:30.811 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.002267322, 2024-08-19T14:16:30.811 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:16:30.811 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:16:30.811 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-08-19T14:16:30.811 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:16:30.811 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:30.811 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:16:30.811 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:30.790231+0000", 2024-08-19T14:16:30.811 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:30.811 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:30.812 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:30.812 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:16:30.812 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:30.790231+0000", 2024-08-19T14:16:30.812 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:30.812 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:30.812 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:30.812 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:16:30.812 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:30.790232+0000", 2024-08-19T14:16:30.812 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.9999999999999997e-07 2024-08-19T14:16:30.812 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:30.812 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:30.812 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:16:30.812 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:30.790238+0000", 2024-08-19T14:16:30.812 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.1569999999999997e-06 2024-08-19T14:16:30.812 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:30.813 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:30.813 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:16:30.813 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:30.790256+0000", 2024-08-19T14:16:30.813 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.8298999999999999e-05 2024-08-19T14:16:30.813 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:30.813 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:30.813 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:16:30.813 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:30.790263+0000", 2024-08-19T14:16:30.813 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.1620000000000002e-06 2024-08-19T14:16:30.813 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:30.813 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:30.813 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:16:30.813 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:30.790365+0000", 2024-08-19T14:16:30.813 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000101296 2024-08-19T14:16:30.814 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:30.814 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:30.814 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:16:30.814 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:30.790391+0000", 2024-08-19T14:16:30.814 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.6774000000000001e-05 2024-08-19T14:16:30.814 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:30.814 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:30.814 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:30.814 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:30.814 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:30.814 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:233882 2.7 2:e95c4edf:::benchmark_data_smithi045_46415_object110601:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e208)", 2024-08-19T14:16:30.817 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:16:30.792190+0000", 2024-08-19T14:16:30.818 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 6.2477000000000005e-05, 2024-08-19T14:16:30.818 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00037535299999999998, 2024-08-19T14:16:30.818 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:16:30.818 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:16:30.818 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-08-19T14:16:30.818 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:16:30.818 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:16:30.819 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:16:30.819 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 233882 2024-08-19T14:16:30.819 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:30.819 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:16:30.819 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:30.819 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:16:30.819 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:30.792190+0000", 2024-08-19T14:16:30.819 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:30.819 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:30.819 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:30.819 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:16:30.819 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:30.792190+0000", 2024-08-19T14:16:30.819 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:16:30.820 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:30.820 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:30.820 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:16:30.820 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:30.792191+0000", 2024-08-19T14:16:30.820 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.491e-06 2024-08-19T14:16:30.820 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:30.820 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:30.820 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:16:30.821 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:30.792198+0000", 2024-08-19T14:16:30.821 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.2559999999999999e-06 2024-08-19T14:16:30.821 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:30.821 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:30.821 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:16:30.821 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:30.792215+0000", 2024-08-19T14:16:30.821 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.7291000000000001e-05 2024-08-19T14:16:30.821 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:30.821 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:30.821 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:16:30.821 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:30.792223+0000", 2024-08-19T14:16:30.821 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.2470000000000008e-06 2024-08-19T14:16:30.822 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:30.822 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:30.822 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:16:30.822 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:30.792296+0000", 2024-08-19T14:16:30.822 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.2769999999999996e-05 2024-08-19T14:16:30.822 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:30.822 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:30.822 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:16:30.822 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:30.792439+0000", 2024-08-19T14:16:30.822 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00014325199999999999 2024-08-19T14:16:30.822 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:30.822 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:30.823 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 2,4", 2024-08-19T14:16:30.823 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:30.792576+0000", 2024-08-19T14:16:30.823 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013624399999999999 2024-08-19T14:16:30.823 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:30.823 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:30.823 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:30.823 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:30.823 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-08-19T14:16:30.823 INFO:teuthology.orchestra.run.smithi031.stdout: "num_ops": 5 2024-08-19T14:16:30.823 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:16:30.849 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:16:30.891 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:30.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:30.892 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:16:30.993 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:16:31.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:31.033 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:31.034 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:16:31.135 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:16:31.164 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:31.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:31.165 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:16:31.172 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-19T14:16:31.265 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:16:31.287 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:31.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:31.288 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:16:31.389 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:16:31.415 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:31.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:31.416 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:16:31.430 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:31 smithi031 ceph-mon[16336]: 1.1 scrub starts 2024-08-19T14:16:31.430 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:31 smithi031 ceph-mon[16336]: 1.1 scrub ok 2024-08-19T14:16:31.430 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:31 smithi031 ceph-mon[16336]: 1.5 scrub starts 2024-08-19T14:16:31.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:31 smithi031 ceph-mon[16336]: 1.5 scrub ok 2024-08-19T14:16:31.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:31 smithi032 ceph-mon[20004]: 1.1 scrub starts 2024-08-19T14:16:31.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:31 smithi032 ceph-mon[20004]: 1.1 scrub ok 2024-08-19T14:16:31.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:31 smithi032 ceph-mon[20004]: 1.5 scrub starts 2024-08-19T14:16:31.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:31 smithi032 ceph-mon[20004]: 1.5 scrub ok 2024-08-19T14:16:31.517 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:16:31.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:31 smithi045 ceph-mon[20143]: 1.1 scrub starts 2024-08-19T14:16:31.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:31 smithi045 ceph-mon[20143]: 1.1 scrub ok 2024-08-19T14:16:31.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:31 smithi045 ceph-mon[20143]: 1.5 scrub starts 2024-08-19T14:16:31.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:31 smithi045 ceph-mon[20143]: 1.5 scrub ok 2024-08-19T14:16:31.550 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:31.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:31.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:16:31.652 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:16:31.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:31.692 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:31.692 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:16:31.793 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:16:31.817 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:31.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:31.818 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:16:31.919 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:16:31.942 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:31.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:31.943 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:16:32.044 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:16:32.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:32.075 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:32.075 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:16:32.176 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:16:32.202 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:32.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:32.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:16:32.304 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:16:32.327 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:32 smithi031 ceph-mon[16336]: 2.7 scrub starts 2024-08-19T14:16:32.327 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:32 smithi031 ceph-mon[16336]: pgmap v507: 27 pgs: 27 active+clean; 856 MiB data, 3.8 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 854 op/s 2024-08-19T14:16:32.327 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:32 smithi031 ceph-mon[16336]: 2.7 scrub ok 2024-08-19T14:16:32.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:32.340 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:32.340 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:16:32.378 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-08-19T14:16:32.391 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:16:32.441 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:16:32.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:32.468 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:32.469 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:16:32.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:32 smithi032 ceph-mon[20004]: 2.7 scrub starts 2024-08-19T14:16:32.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:32 smithi032 ceph-mon[20004]: pgmap v507: 27 pgs: 27 active+clean; 856 MiB data, 3.8 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 854 op/s 2024-08-19T14:16:32.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:32 smithi032 ceph-mon[20004]: 2.7 scrub ok 2024-08-19T14:16:32.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:32 smithi045 ceph-mon[20143]: 2.7 scrub starts 2024-08-19T14:16:32.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:32 smithi045 ceph-mon[20143]: pgmap v507: 27 pgs: 27 active+clean; 856 MiB data, 3.8 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 854 op/s 2024-08-19T14:16:32.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:32 smithi045 ceph-mon[20143]: 2.7 scrub ok 2024-08-19T14:16:32.569 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:16:32.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:32.619 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:32.620 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:16:32.721 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:16:32.758 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:32.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:32.759 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:16:32.861 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:16:32.903 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:32.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:32.904 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:16:33.005 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:16:33.024 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:16:33.029 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:33.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:33.030 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:16:33.130 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:16:33.155 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:33.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:33.156 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:16:33.257 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:16:33.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:33.282 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:33.282 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:16:33.383 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:16:33.410 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:33.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:33.411 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:16:33.465 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:33 smithi031 ceph-mon[16336]: 2.e scrub starts 2024-08-19T14:16:33.480 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:16:33.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:33 smithi032 ceph-mon[20004]: 2.e scrub starts 2024-08-19T14:16:33.512 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:16:33.532 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:33.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:33.534 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:16:33.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:33 smithi045 ceph-mon[20143]: 2.e scrub starts 2024-08-19T14:16:33.629 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:16:33.629 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-08-19T14:16:33.630 INFO:teuthology.orchestra.run.smithi031.stdout: "complaint_time": 30, 2024-08-19T14:16:33.630 INFO:teuthology.orchestra.run.smithi031.stdout: "num_blocked_ops": 0 2024-08-19T14:16:33.630 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:16:33.635 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:16:33.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:33.663 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:33.664 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:16:33.765 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:16:33.790 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:33.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:33.791 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:16:33.892 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:16:33.921 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:33.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:33.923 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:16:34.023 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:16:34.048 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:34.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:34.049 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:16:34.150 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:16:34.175 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:34.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:34.176 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:16:34.184 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is unset 2024-08-19T14:16:34.277 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:16:34.308 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:34.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:34.309 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:16:34.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:34 smithi031 ceph-mon[16336]: 2.1 scrub starts 2024-08-19T14:16:34.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:34 smithi031 ceph-mon[16336]: 2.1 scrub ok 2024-08-19T14:16:34.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:34 smithi031 ceph-mon[16336]: 2.e scrub ok 2024-08-19T14:16:34.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:34 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3022446989' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:16:34.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:34 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:16:34.368 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:34 smithi031 ceph-mon[16336]: pgmap v508: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 745 MiB data, 3.4 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 913 op/s; 0 B/s, 0 objects/s recovering 2024-08-19T14:16:34.411 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:16:34.436 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:34.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:34.437 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:16:34.511 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:34 smithi032 ceph-mon[20004]: 2.1 scrub starts 2024-08-19T14:16:34.511 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:34 smithi032 ceph-mon[20004]: 2.1 scrub ok 2024-08-19T14:16:34.511 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:34 smithi032 ceph-mon[20004]: 2.e scrub ok 2024-08-19T14:16:34.511 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:34 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3022446989' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:16:34.511 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:34 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:16:34.511 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:34 smithi032 ceph-mon[20004]: pgmap v508: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 745 MiB data, 3.4 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 913 op/s; 0 B/s, 0 objects/s recovering 2024-08-19T14:16:34.538 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:16:34.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:34 smithi045 ceph-mon[20143]: 2.1 scrub starts 2024-08-19T14:16:34.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:34 smithi045 ceph-mon[20143]: 2.1 scrub ok 2024-08-19T14:16:34.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:34 smithi045 ceph-mon[20143]: 2.e scrub ok 2024-08-19T14:16:34.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:34 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3022446989' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:16:34.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:34 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:16:34.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:34 smithi045 ceph-mon[20143]: pgmap v508: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 745 MiB data, 3.4 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 913 op/s; 0 B/s, 0 objects/s recovering 2024-08-19T14:16:34.575 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:34.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:34.576 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:16:34.677 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:16:34.701 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:34.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:34.702 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:16:34.736 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 8, 9, 11] out_osds: [10, 6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-08-19T14:16:34.736 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-19T14:16:34.736 INFO:tasks.thrashosds.thrasher:Removing osd 2, in_osds are: [0, 1, 2, 3, 4, 5, 7, 8, 9, 11] 2024-08-19T14:16:34.737 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd out 2 2024-08-19T14:16:34.803 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:16:34.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:34.837 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:34.838 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:16:34.939 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:16:34.966 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:34.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:34.967 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:16:35.068 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:16:35.099 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:35.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:35.100 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:16:35.176 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-08-19T14:16:35.200 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:16:35.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:35.268 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:35.268 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:16:35.369 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:16:35.398 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:35.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:35.399 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:16:35.500 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:16:35.504 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:35 smithi031 ceph-mon[16336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:16:35.504 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:35 smithi031 ceph-mon[16336]: Cluster is now healthy 2024-08-19T14:16:35.504 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:35 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:16:35.504 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:35 smithi031 ceph-mon[16336]: osdmap e209: 12 total, 12 up, 10 in 2024-08-19T14:16:35.504 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:35 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:16:35.504 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:35 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:16:35.504 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:35 smithi031 ceph-mon[16336]: 2.0 scrub starts 2024-08-19T14:16:35.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:35 smithi032 ceph-mon[20004]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:16:35.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:35 smithi032 ceph-mon[20004]: Cluster is now healthy 2024-08-19T14:16:35.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:35 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:16:35.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:35 smithi032 ceph-mon[20004]: osdmap e209: 12 total, 12 up, 10 in 2024-08-19T14:16:35.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:35 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:16:35.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:35 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:16:35.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:35 smithi032 ceph-mon[20004]: 2.0 scrub starts 2024-08-19T14:16:35.522 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:35.522 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:16:35.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:35.523 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:16:35.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:35 smithi045 ceph-mon[20143]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:16:35.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:35 smithi045 ceph-mon[20143]: Cluster is now healthy 2024-08-19T14:16:35.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:35 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:16:35.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:35 smithi045 ceph-mon[20143]: osdmap e209: 12 total, 12 up, 10 in 2024-08-19T14:16:35.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:35 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:16:35.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:35 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:16:35.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:35 smithi045 ceph-mon[20143]: 2.0 scrub starts 2024-08-19T14:16:35.624 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:16:35.649 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:35.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:35.650 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:16:35.752 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:16:35.803 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:35.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:35.805 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:16:35.906 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:16:35.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:35.940 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:35.940 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:16:36.041 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:16:36.067 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:36.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:36.068 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:16:36.110 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:16:36.152 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:16:36.155 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:16:36.155 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:16:36.168 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:16:36.191 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:36.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:36.193 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:16:36.194 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:16:36.195 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:16:36.225 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:16:36.226 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:16:36.265 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:16:36.266 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:16:36.293 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:16:36.300 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:16:36.300 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:16:36.318 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:36 smithi031 ceph-mon[16336]: 2.0 scrub ok 2024-08-19T14:16:36.318 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:36 smithi031 ceph-mon[16336]: osdmap e210: 12 total, 12 up, 10 in 2024-08-19T14:16:36.318 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:36 smithi031 ceph-mon[16336]: pgmap v511: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 608 MiB data, 3.1 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 852 op/s; 0 B/s, 0 objects/s recovering 2024-08-19T14:16:36.318 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:36 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1562384248' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:16:36.338 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:16:36.338 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:16:36.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:36.340 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:36.340 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:16:36.371 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:16:36.371 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:16:36.400 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:16:36.400 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:16:36.438 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:16:36.439 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:16:36.441 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:16:36.464 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:36.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:36.465 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:16:36.477 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:16:36.477 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:16:36.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:36 smithi032 ceph-mon[20004]: 2.0 scrub ok 2024-08-19T14:16:36.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:36 smithi032 ceph-mon[20004]: osdmap e210: 12 total, 12 up, 10 in 2024-08-19T14:16:36.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:36 smithi032 ceph-mon[20004]: pgmap v511: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 608 MiB data, 3.1 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 852 op/s; 0 B/s, 0 objects/s recovering 2024-08-19T14:16:36.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:36 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1562384248' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:16:36.514 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:16:36.514 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:16:36.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:36 smithi045 ceph-mon[20143]: 2.0 scrub ok 2024-08-19T14:16:36.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:36 smithi045 ceph-mon[20143]: osdmap e210: 12 total, 12 up, 10 in 2024-08-19T14:16:36.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:36 smithi045 ceph-mon[20143]: pgmap v511: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 608 MiB data, 3.1 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 852 op/s; 0 B/s, 0 objects/s recovering 2024-08-19T14:16:36.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:36 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1562384248' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:16:36.548 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:16:36.549 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:16:36.566 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:16:36.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:36.607 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:36.607 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:16:36.708 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:16:36.737 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:36.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:36.738 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:16:36.838 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:16:36.861 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:36.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:36.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:16:36.963 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:16:36.992 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:36.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:36.993 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:16:37.094 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:16:37.118 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:37.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:37.119 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:16:37.220 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:16:37.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:37 smithi032 ceph-mon[20004]: 2.8 scrub starts 2024-08-19T14:16:37.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:37 smithi032 ceph-mon[20004]: 2.8 scrub ok 2024-08-19T14:16:37.259 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:37.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:37.260 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:16:37.361 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:16:37.387 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:37.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:37.388 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:16:37.445 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:37 smithi031 ceph-mon[16336]: 2.8 scrub starts 2024-08-19T14:16:37.445 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:37 smithi031 ceph-mon[16336]: 2.8 scrub ok 2024-08-19T14:16:37.489 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:16:37.520 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:37.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:37.521 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:16:37.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:37 smithi045 ceph-mon[20143]: 2.8 scrub starts 2024-08-19T14:16:37.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:37 smithi045 ceph-mon[20143]: 2.8 scrub ok 2024-08-19T14:16:37.565 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-19T14:16:37.612 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:16:37.613 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:16:37.622 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:16:37.679 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:37.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:37.680 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:16:37.781 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:16:37.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:37.829 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:37.829 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:16:37.930 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:16:37.975 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:37.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:37.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:16:38.078 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:16:38.105 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:38.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:38.106 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:16:38.208 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:16:38.213 INFO:teuthology.orchestra.run.smithi031.stderr:marked out osd.2. 2024-08-19T14:16:38.233 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:38.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:38.234 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:16:38.235 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:38 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mon-a[16332]: 2024-08-19T14:16:38.170+0000 7f2ff503e640 -1 mon.a@0(leader).osd e210 definitely_dead 0 2024-08-19T14:16:38.235 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:38 smithi031 ceph-mon[16336]: 1.6 scrub starts 2024-08-19T14:16:38.235 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:38 smithi031 ceph-mon[16336]: 1.6 scrub ok 2024-08-19T14:16:38.235 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:38 smithi031 ceph-mon[16336]: 2.a scrub starts 2024-08-19T14:16:38.235 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:38 smithi031 ceph-mon[16336]: 2.a scrub ok 2024-08-19T14:16:38.235 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:38 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3417983934' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-08-19T14:16:38.235 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:38 smithi031 ceph-mon[16336]: Client client.admin marked osd.2 out, while it was still marked up 2024-08-19T14:16:38.335 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:16:38.356 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:38.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:38.357 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:16:38.458 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:16:38.472 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-08-19T14:16:38.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:38.480 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:38.481 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:16:38.500 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:38 smithi032 ceph-mon[20004]: 1.6 scrub starts 2024-08-19T14:16:38.500 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:38 smithi032 ceph-mon[20004]: 1.6 scrub ok 2024-08-19T14:16:38.500 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:38 smithi032 ceph-mon[20004]: 2.a scrub starts 2024-08-19T14:16:38.500 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:38 smithi032 ceph-mon[20004]: 2.a scrub ok 2024-08-19T14:16:38.500 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:38 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3417983934' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-08-19T14:16:38.500 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:38 smithi032 ceph-mon[20004]: Client client.admin marked osd.2 out, while it was still marked up 2024-08-19T14:16:38.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:38 smithi045 ceph-mon[20143]: 1.6 scrub starts 2024-08-19T14:16:38.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:38 smithi045 ceph-mon[20143]: 1.6 scrub ok 2024-08-19T14:16:38.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:38 smithi045 ceph-mon[20143]: 2.a scrub starts 2024-08-19T14:16:38.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:38 smithi045 ceph-mon[20143]: 2.a scrub ok 2024-08-19T14:16:38.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:38 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3417983934' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-08-19T14:16:38.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:38 smithi045 ceph-mon[20143]: Client client.admin marked osd.2 out, while it was still marked up 2024-08-19T14:16:38.581 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:16:38.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:38.604 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:38.604 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:16:38.687 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:16:38.706 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:16:38.728 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:38.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:38.729 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:16:38.732 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:16:38.830 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:16:38.852 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:38.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:38.854 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:16:38.954 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:16:38.976 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:38.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:38.977 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:16:39.077 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:16:39.098 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:39.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:39.099 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:16:39.200 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:16:39.226 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:39.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:39.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:16:39.233 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:16:39.307 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:16:39.307 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:16:39.328 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:16:39.350 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:39.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:39.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:16:39.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:39 smithi031 ceph-mon[16336]: pgmap v512: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 569 MiB data, 3.0 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 749 op/s; 0 B/s, 0 objects/s recovering 2024-08-19T14:16:39.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:39 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3417983934' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2024-08-19T14:16:39.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:39 smithi031 ceph-mon[16336]: osdmap e211: 12 total, 12 up, 9 in 2024-08-19T14:16:39.452 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:16:39.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:39.478 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:39.478 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:16:39.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:39 smithi032 ceph-mon[20004]: pgmap v512: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 569 MiB data, 3.0 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 749 op/s; 0 B/s, 0 objects/s recovering 2024-08-19T14:16:39.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:39 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3417983934' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2024-08-19T14:16:39.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:39 smithi032 ceph-mon[20004]: osdmap e211: 12 total, 12 up, 9 in 2024-08-19T14:16:39.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:39 smithi045 ceph-mon[20143]: pgmap v512: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 569 MiB data, 3.0 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 749 op/s; 0 B/s, 0 objects/s recovering 2024-08-19T14:16:39.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:39 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3417983934' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2024-08-19T14:16:39.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:39 smithi045 ceph-mon[20143]: osdmap e211: 12 total, 12 up, 9 in 2024-08-19T14:16:39.579 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:16:39.600 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:39.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:39.601 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:16:39.702 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:16:39.724 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:39.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:39.725 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:16:39.826 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:16:39.849 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:39.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:39.850 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:16:39.876 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:16:39.933 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-08-19T14:16:39.950 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:16:39.971 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:39.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:39.972 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:16:40.073 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:16:40.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:40.096 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:40.096 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:16:40.191 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:16:40.197 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:16:40.223 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:40.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:40.224 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:16:40.225 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is set 2024-08-19T14:16:40.264 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:40 smithi031 ceph-mon[16336]: osdmap e212: 12 total, 12 up, 9 in 2024-08-19T14:16:40.264 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:40 smithi031 ceph-mon[16336]: 2.c scrub starts 2024-08-19T14:16:40.264 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:40 smithi031 ceph-mon[16336]: 2.c scrub ok 2024-08-19T14:16:40.264 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:40 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1285027601' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:16:40.325 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:16:40.347 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:40.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:40.348 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:16:40.449 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:16:40.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:40.492 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:40.492 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:16:40.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:40 smithi032 ceph-mon[20004]: osdmap e212: 12 total, 12 up, 9 in 2024-08-19T14:16:40.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:40 smithi032 ceph-mon[20004]: 2.c scrub starts 2024-08-19T14:16:40.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:40 smithi032 ceph-mon[20004]: 2.c scrub ok 2024-08-19T14:16:40.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:40 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1285027601' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:16:40.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:40 smithi045 ceph-mon[20143]: osdmap e212: 12 total, 12 up, 9 in 2024-08-19T14:16:40.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:40 smithi045 ceph-mon[20143]: 2.c scrub starts 2024-08-19T14:16:40.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:40 smithi045 ceph-mon[20143]: 2.c scrub ok 2024-08-19T14:16:40.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:40 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1285027601' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:16:40.593 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:16:40.636 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:40.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:40.637 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:16:40.738 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:16:40.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:40.770 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:40.770 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:16:40.871 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:16:40.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:40.901 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:40.901 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:16:41.002 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:16:41.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:41.025 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:41.026 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:16:41.126 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:16:41.158 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:41.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:41.160 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:16:41.162 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:16:41.162 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-08-19T14:16:41.162 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-08-19T14:16:41.162 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-08-19T14:16:41.162 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:16:41.260 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:16:41.288 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:41.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:41.289 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:16:41.390 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:16:41.420 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:41.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:41.421 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:16:41.503 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:41 smithi031 ceph-mon[16336]: pgmap v515: 27 pgs: 3 peering, 24 active+clean; 425 MiB data, 2.2 GiB used, 802 GiB / 805 GiB avail; 0 B/s wr, 832 op/s 2024-08-19T14:16:41.504 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:41 smithi031 ceph-mon[16336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:16:41.504 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:41 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1285027601' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:16:41.504 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:41 smithi031 ceph-mon[16336]: 2.4 starting backfill to osd.8 from (0'0,0'0] MAX to 210'15104 2024-08-19T14:16:41.504 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:41 smithi031 ceph-mon[16336]: 2.3 starting backfill to osd.9 from (0'0,0'0] MAX to 210'15052 2024-08-19T14:16:41.504 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:41 smithi031 ceph-mon[16336]: osdmap e213: 12 total, 12 up, 9 in 2024-08-19T14:16:41.504 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:41 smithi031 ceph-mon[16336]: 2.e starting backfill to osd.5 from (0'0,0'0] MAX to 210'15011 2024-08-19T14:16:41.504 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:41 smithi031 ceph-mon[16336]: 2.1 starting backfill to osd.1 from (0'0,0'0] MAX to 210'15173 2024-08-19T14:16:41.504 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:41 smithi031 ceph-mon[16336]: 2.7 starting backfill to osd.5 from (0'0,0'0] MAX to 210'15066 2024-08-19T14:16:41.504 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:41 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2856363142' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:16:41.504 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:16:41.505 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:16:41.505 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:16:41.505 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:16:41.505 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:16:41.505 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:16:41.505 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:16:41.505 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:16:41.505 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:16:41.505 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:16:41.505 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:16:41.505 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:16:41.505 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:16:41.506 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:16:41.506 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:16:41.506 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:16:41.506 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:16:41.506 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:16:41.506 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:16:41.506 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:16:41.506 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:16:41.506 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:16:41.506 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:16:41.506 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:16:41.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:41 smithi032 ceph-mon[20004]: pgmap v515: 27 pgs: 3 peering, 24 active+clean; 425 MiB data, 2.2 GiB used, 802 GiB / 805 GiB avail; 0 B/s wr, 832 op/s 2024-08-19T14:16:41.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:41 smithi032 ceph-mon[20004]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:16:41.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:41 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1285027601' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:16:41.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:41 smithi032 ceph-mon[20004]: 2.4 starting backfill to osd.8 from (0'0,0'0] MAX to 210'15104 2024-08-19T14:16:41.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:41 smithi032 ceph-mon[20004]: 2.3 starting backfill to osd.9 from (0'0,0'0] MAX to 210'15052 2024-08-19T14:16:41.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:41 smithi032 ceph-mon[20004]: osdmap e213: 12 total, 12 up, 9 in 2024-08-19T14:16:41.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:41 smithi032 ceph-mon[20004]: 2.e starting backfill to osd.5 from (0'0,0'0] MAX to 210'15011 2024-08-19T14:16:41.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:41 smithi032 ceph-mon[20004]: 2.1 starting backfill to osd.1 from (0'0,0'0] MAX to 210'15173 2024-08-19T14:16:41.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:41 smithi032 ceph-mon[20004]: 2.7 starting backfill to osd.5 from (0'0,0'0] MAX to 210'15066 2024-08-19T14:16:41.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:41 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2856363142' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:16:41.522 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:16:41.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:41 smithi045 ceph-mon[20143]: pgmap v515: 27 pgs: 3 peering, 24 active+clean; 425 MiB data, 2.2 GiB used, 802 GiB / 805 GiB avail; 0 B/s wr, 832 op/s 2024-08-19T14:16:41.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:41 smithi045 ceph-mon[20143]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:16:41.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:41 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1285027601' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:16:41.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:41 smithi045 ceph-mon[20143]: 2.4 starting backfill to osd.8 from (0'0,0'0] MAX to 210'15104 2024-08-19T14:16:41.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:41 smithi045 ceph-mon[20143]: 2.3 starting backfill to osd.9 from (0'0,0'0] MAX to 210'15052 2024-08-19T14:16:41.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:41 smithi045 ceph-mon[20143]: osdmap e213: 12 total, 12 up, 9 in 2024-08-19T14:16:41.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:41 smithi045 ceph-mon[20143]: 2.e starting backfill to osd.5 from (0'0,0'0] MAX to 210'15011 2024-08-19T14:16:41.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:41 smithi045 ceph-mon[20143]: 2.1 starting backfill to osd.1 from (0'0,0'0] MAX to 210'15173 2024-08-19T14:16:41.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:41 smithi045 ceph-mon[20143]: 2.7 starting backfill to osd.5 from (0'0,0'0] MAX to 210'15066 2024-08-19T14:16:41.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:41 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2856363142' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:16:41.550 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:41.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:41.551 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:16:41.652 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:16:41.675 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:41.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:41.676 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:16:41.777 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:16:41.798 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:41.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:41.799 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:16:41.900 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:16:41.921 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:41.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:41.922 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:16:42.013 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-08-19T14:16:42.023 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:16:42.044 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:42.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:42.045 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:16:42.146 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:16:42.167 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:42.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:42.168 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:16:42.268 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:16:42.269 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:16:42.290 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:42.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:42.291 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:16:42.392 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:16:42.414 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:42.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:42.415 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:16:42.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:42 smithi032 ceph-mon[20004]: osdmap e214: 12 total, 12 up, 9 in 2024-08-19T14:16:42.516 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:16:42.536 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:42.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:42.537 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:16:42.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:42 smithi045 ceph-mon[20143]: osdmap e214: 12 total, 12 up, 9 in 2024-08-19T14:16:42.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:42 smithi031 ceph-mon[16336]: osdmap e214: 12 total, 12 up, 9 in 2024-08-19T14:16:42.638 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:16:42.659 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:42.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:42.660 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:16:42.761 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:16:42.784 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:42.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:42.785 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:16:42.886 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:16:42.886 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-08-19T14:16:42.887 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-08-19T14:16:42.887 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-08-19T14:16:42.887 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.887 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24580.0:62590 2.d e125/102 2:b1cd6c4c:::benchmark_data_smithi045_46415_object62589:head v 125'3988, mlcod=125'3985)", 2024-08-19T14:16:42.887 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:18.389311+0000", 2024-08-19T14:16:42.887 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 204.492893434, 2024-08-19T14:16:42.887 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.025501861000000001, 2024-08-19T14:16:42.887 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:16:42.887 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:16:42.888 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:16:42.888 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:16:42.888 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.888 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:16:42.888 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.389311+0000", 2024-08-19T14:16:42.888 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:42.888 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.888 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.888 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:16:42.889 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.389311+0000", 2024-08-19T14:16:42.889 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:42.889 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.889 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.889 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:16:42.889 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.389311+0000", 2024-08-19T14:16:42.889 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8899999999999999e-07 2024-08-19T14:16:42.889 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.889 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.889 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:16:42.890 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.389426+0000", 2024-08-19T14:16:42.890 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000115003 2024-08-19T14:16:42.890 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.890 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.890 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:16:42.890 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.389437+0000", 2024-08-19T14:16:42.891 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0865e-05 2024-08-19T14:16:42.891 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.891 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.891 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:16:42.891 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.389444+0000", 2024-08-19T14:16:42.891 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2189999999999998e-06 2024-08-19T14:16:42.892 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.892 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.892 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:16:42.892 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.389490+0000", 2024-08-19T14:16:42.892 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5357e-05 2024-08-19T14:16:42.892 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.893 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.893 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:16:42.893 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.389504+0000", 2024-08-19T14:16:42.893 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4119e-05 2024-08-19T14:16:42.893 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.894 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.894 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:16:42.894 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.391313+0000", 2024-08-19T14:16:42.894 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0018095380000000001 2024-08-19T14:16:42.894 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.894 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.894 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:16:42.894 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.414812+0000", 2024-08-19T14:16:42.894 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.023499170999999999 2024-08-19T14:16:42.894 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:42.894 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:16:42.895 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:42.895 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.895 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.895 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24580.0:62594 2.5 e125/102 2:a15e85f0:::benchmark_data_smithi045_46415_object62593:head v 125'3968, mlcod=125'3966)", 2024-08-19T14:16:42.895 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:18.389960+0000", 2024-08-19T14:16:42.895 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 204.492244263, 2024-08-19T14:16:42.895 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.027685131000000002, 2024-08-19T14:16:42.895 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:16:42.895 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:16:42.895 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:16:42.895 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:16:42.895 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.895 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:16:42.895 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.389960+0000", 2024-08-19T14:16:42.896 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:42.896 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.896 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.896 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:16:42.896 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.389960+0000", 2024-08-19T14:16:42.896 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:42.896 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.896 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.896 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:16:42.896 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.389960+0000", 2024-08-19T14:16:42.897 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8100000000000002e-07 2024-08-19T14:16:42.897 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.897 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.897 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:16:42.897 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.390013+0000", 2024-08-19T14:16:42.897 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2772000000000002e-05 2024-08-19T14:16:42.897 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.897 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.897 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:16:42.897 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.390027+0000", 2024-08-19T14:16:42.897 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3625000000000001e-05 2024-08-19T14:16:42.897 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.897 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.897 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:16:42.898 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.390033+0000", 2024-08-19T14:16:42.898 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3849999999999999e-06 2024-08-19T14:16:42.898 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.898 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.898 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:16:42.898 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.390089+0000", 2024-08-19T14:16:42.898 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5998000000000002e-05 2024-08-19T14:16:42.898 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.898 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.898 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:16:42.898 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.390109+0000", 2024-08-19T14:16:42.898 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9539e-05 2024-08-19T14:16:42.898 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.899 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.899 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:16:42.899 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.417596+0000", 2024-08-19T14:16:42.899 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.027487396000000001 2024-08-19T14:16:42.899 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.899 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.899 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:16:42.899 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.417645+0000", 2024-08-19T14:16:42.899 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8735e-05 2024-08-19T14:16:42.899 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:42.899 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:16:42.899 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:42.899 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.899 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.899 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24580.0:63067 2.7 e128/127 2:e2705906:::benchmark_data_smithi045_46415_object63066:head v 128'3850, mlcod=0'0)", 2024-08-19T14:16:42.900 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:20.938200+0000", 2024-08-19T14:16:42.900 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 201.94400358199999, 2024-08-19T14:16:42.900 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.021572997999999999, 2024-08-19T14:16:42.900 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:16:42.900 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:16:42.900 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:16:42.900 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:16:42.900 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.900 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:16:42.900 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.938200+0000", 2024-08-19T14:16:42.900 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:42.900 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.900 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.901 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:16:42.901 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.938200+0000", 2024-08-19T14:16:42.901 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:42.901 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.901 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.901 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:16:42.901 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.938203+0000", 2024-08-19T14:16:42.901 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.638e-06 2024-08-19T14:16:42.901 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.901 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.901 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:16:42.901 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.938268+0000", 2024-08-19T14:16:42.901 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4980000000000005e-05 2024-08-19T14:16:42.901 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.902 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.902 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:16:42.902 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.938279+0000", 2024-08-19T14:16:42.902 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0658e-05 2024-08-19T14:16:42.902 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.902 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.902 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:16:42.902 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.938285+0000", 2024-08-19T14:16:42.902 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9410000000000004e-06 2024-08-19T14:16:42.902 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.902 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.902 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:16:42.902 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.940615+0000", 2024-08-19T14:16:42.903 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0023300819999999998 2024-08-19T14:16:42.903 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.903 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.903 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:16:42.903 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.940638+0000", 2024-08-19T14:16:42.903 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3691999999999999e-05 2024-08-19T14:16:42.903 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.903 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.903 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:16:42.903 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.959684+0000", 2024-08-19T14:16:42.903 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.019045973000000001 2024-08-19T14:16:42.903 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.903 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.903 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:16:42.904 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.959773+0000", 2024-08-19T14:16:42.904 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.9034000000000002e-05 2024-08-19T14:16:42.904 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:42.904 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:16:42.904 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:42.904 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.904 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.904 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24580.0:63086 2.7 e128/127 2:e280ae82:::benchmark_data_smithi045_46415_object63085:head v 128'3851, mlcod=0'0)", 2024-08-19T14:16:42.904 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:20.940402+0000", 2024-08-19T14:16:42.904 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 201.94180216500001, 2024-08-19T14:16:42.904 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.022477548999999999, 2024-08-19T14:16:42.904 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:16:42.904 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:16:42.904 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:16:42.905 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:16:42.905 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.905 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:16:42.905 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.940402+0000", 2024-08-19T14:16:42.905 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:42.905 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.905 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.905 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:16:42.905 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.940402+0000", 2024-08-19T14:16:42.905 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:42.905 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.905 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.905 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:16:42.906 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.940402+0000", 2024-08-19T14:16:42.906 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0200000000000002e-07 2024-08-19T14:16:42.906 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.906 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.906 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:16:42.906 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.940470+0000", 2024-08-19T14:16:42.906 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7830999999999995e-05 2024-08-19T14:16:42.906 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.906 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.906 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:16:42.906 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.940479+0000", 2024-08-19T14:16:42.906 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.4100000000000008e-06 2024-08-19T14:16:42.906 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.906 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.907 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:16:42.907 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.940484+0000", 2024-08-19T14:16:42.907 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8679999999999998e-06 2024-08-19T14:16:42.907 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.907 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.907 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:16:42.907 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.944331+0000", 2024-08-19T14:16:42.907 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0038478200000000001 2024-08-19T14:16:42.907 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.907 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.907 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:16:42.907 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.944357+0000", 2024-08-19T14:16:42.907 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5836000000000001e-05 2024-08-19T14:16:42.908 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.908 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.908 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:16:42.908 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.962818+0000", 2024-08-19T14:16:42.908 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.018460977 2024-08-19T14:16:42.908 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.908 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.908 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:16:42.908 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.962879+0000", 2024-08-19T14:16:42.908 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1304999999999997e-05 2024-08-19T14:16:42.908 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:42.908 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:16:42.908 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:42.908 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.909 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.909 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24580.0:63687 2.7 e128/127 2:ea76d4f0:::benchmark_data_smithi045_46415_object63686:head v 128'3888, mlcod=0'0)", 2024-08-19T14:16:42.909 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:21.486322+0000", 2024-08-19T14:16:42.909 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 201.39588184199999, 2024-08-19T14:16:42.909 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.018889623000000001, 2024-08-19T14:16:42.909 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:16:42.909 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:16:42.909 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:16:42.909 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:16:42.909 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.909 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:16:42.909 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.486322+0000", 2024-08-19T14:16:42.909 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:42.910 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.910 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.910 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:16:42.910 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.486322+0000", 2024-08-19T14:16:42.910 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:42.910 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.910 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.910 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:16:42.910 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.486323+0000", 2024-08-19T14:16:42.910 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3799999999999996e-07 2024-08-19T14:16:42.910 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.910 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.910 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:16:42.910 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.486367+0000", 2024-08-19T14:16:42.911 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3430000000000003e-05 2024-08-19T14:16:42.911 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.911 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.911 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:16:42.911 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.486381+0000", 2024-08-19T14:16:42.911 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4579000000000001e-05 2024-08-19T14:16:42.911 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.911 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.911 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:16:42.911 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.486389+0000", 2024-08-19T14:16:42.911 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6899999999999992e-06 2024-08-19T14:16:42.911 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.911 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.911 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:16:42.912 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.486444+0000", 2024-08-19T14:16:42.912 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5359999999999999e-05 2024-08-19T14:16:42.912 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.912 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.912 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:16:42.912 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.486472+0000", 2024-08-19T14:16:42.912 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8019999999999999e-05 2024-08-19T14:16:42.912 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.912 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.912 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:16:42.912 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.505168+0000", 2024-08-19T14:16:42.912 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.01869618 2024-08-19T14:16:42.912 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.913 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.913 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:16:42.913 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.505212+0000", 2024-08-19T14:16:42.913 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3426000000000001e-05 2024-08-19T14:16:42.913 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:42.913 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:16:42.913 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:42.913 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.913 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.913 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24580.0:64943 2.f e129/102 2:f4917538:::benchmark_data_smithi045_46415_object64942:head v 129'4101, mlcod=129'4099)", 2024-08-19T14:16:42.913 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:22.483598+0000", 2024-08-19T14:16:42.913 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 200.39860583000001, 2024-08-19T14:16:42.913 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.02030792, 2024-08-19T14:16:42.913 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:16:42.914 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:16:42.914 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:16:42.914 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:16:42.914 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.914 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:16:42.914 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.483598+0000", 2024-08-19T14:16:42.914 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:42.914 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.914 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.914 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:16:42.914 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.483598+0000", 2024-08-19T14:16:42.914 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:42.914 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.915 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.915 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:16:42.915 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.483599+0000", 2024-08-19T14:16:42.915 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0380000000000001e-06 2024-08-19T14:16:42.915 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.915 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.915 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:16:42.915 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.483667+0000", 2024-08-19T14:16:42.915 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7997999999999995e-05 2024-08-19T14:16:42.915 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.915 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.915 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:16:42.915 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.483682+0000", 2024-08-19T14:16:42.915 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.45e-05 2024-08-19T14:16:42.915 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.916 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.916 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:16:42.916 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.483688+0000", 2024-08-19T14:16:42.916 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6680000000000004e-06 2024-08-19T14:16:42.916 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.916 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.916 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:16:42.916 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.486328+0000", 2024-08-19T14:16:42.916 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0026395440000000002 2024-08-19T14:16:42.916 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.916 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.916 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:16:42.916 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.486351+0000", 2024-08-19T14:16:42.916 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2751e-05 2024-08-19T14:16:42.917 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.917 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.917 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:16:42.917 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.503857+0000", 2024-08-19T14:16:42.917 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.017505850999999999 2024-08-19T14:16:42.917 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.917 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.917 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:16:42.917 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.503906+0000", 2024-08-19T14:16:42.917 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9570000000000001e-05 2024-08-19T14:16:42.917 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:42.917 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:16:42.917 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:42.918 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.918 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.918 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24580.0:64926 2.7 e129/127 2:ee7d394c:::benchmark_data_smithi045_46415_object64925:head v 129'3976, mlcod=0'0)", 2024-08-19T14:16:42.918 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:22.485386+0000", 2024-08-19T14:16:42.918 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 200.396818024, 2024-08-19T14:16:42.918 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.022392815999999999, 2024-08-19T14:16:42.918 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:16:42.918 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:16:42.918 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:16:42.918 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:16:42.919 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.919 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:16:42.919 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.485386+0000", 2024-08-19T14:16:42.919 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:42.919 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.919 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.919 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:16:42.919 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.485386+0000", 2024-08-19T14:16:42.919 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:42.919 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.919 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.919 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:16:42.919 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.485387+0000", 2024-08-19T14:16:42.920 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.9699999999999994e-07 2024-08-19T14:16:42.920 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.920 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.920 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:16:42.920 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.485434+0000", 2024-08-19T14:16:42.920 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7113000000000001e-05 2024-08-19T14:16:42.920 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.920 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.920 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:16:42.920 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.485447+0000", 2024-08-19T14:16:42.920 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2695e-05 2024-08-19T14:16:42.920 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.920 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.921 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:16:42.921 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.485455+0000", 2024-08-19T14:16:42.921 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.9799999999999998e-06 2024-08-19T14:16:42.921 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.921 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.921 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:16:42.921 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.487352+0000", 2024-08-19T14:16:42.921 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001896809 2024-08-19T14:16:42.921 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.921 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.921 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:16:42.921 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.487379+0000", 2024-08-19T14:16:42.922 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7248999999999999e-05 2024-08-19T14:16:42.922 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.922 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.922 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:16:42.922 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.507725+0000", 2024-08-19T14:16:42.922 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.020345797999999998 2024-08-19T14:16:42.922 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.922 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.922 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:16:42.922 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.507779+0000", 2024-08-19T14:16:42.922 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4175000000000002e-05 2024-08-19T14:16:42.922 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:42.922 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:16:42.923 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:42.923 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.923 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.923 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24580.0:64930 2.7 e129/127 2:e3128c9d:::benchmark_data_smithi045_46415_object64929:head v 129'3977, mlcod=0'0)", 2024-08-19T14:16:42.923 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:22.488188+0000", 2024-08-19T14:16:42.923 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 200.394015938, 2024-08-19T14:16:42.923 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.019626971999999999, 2024-08-19T14:16:42.923 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:16:42.923 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:16:42.923 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:16:42.923 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:16:42.923 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.923 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:16:42.924 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.488188+0000", 2024-08-19T14:16:42.924 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:42.924 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.924 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.924 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:16:42.924 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.488188+0000", 2024-08-19T14:16:42.924 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:42.924 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.924 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.924 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:16:42.924 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.488189+0000", 2024-08-19T14:16:42.924 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6099999999999999e-07 2024-08-19T14:16:42.924 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.925 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.925 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:16:42.925 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.488230+0000", 2024-08-19T14:16:42.925 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1448e-05 2024-08-19T14:16:42.926 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.926 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.926 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:16:42.926 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.488242+0000", 2024-08-19T14:16:42.926 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1402e-05 2024-08-19T14:16:42.926 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.926 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.927 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:16:42.927 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.488249+0000", 2024-08-19T14:16:42.927 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7190000000000004e-06 2024-08-19T14:16:42.927 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.927 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.927 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:16:42.927 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.491525+0000", 2024-08-19T14:16:42.927 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0032765350000000001 2024-08-19T14:16:42.927 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.927 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.927 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:16:42.928 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.491554+0000", 2024-08-19T14:16:42.928 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9022000000000001e-05 2024-08-19T14:16:42.928 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.928 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.928 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:16:42.928 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.507783+0000", 2024-08-19T14:16:42.928 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.016228564000000001 2024-08-19T14:16:42.928 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.928 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.928 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:16:42.928 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.507815+0000", 2024-08-19T14:16:42.928 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2421e-05 2024-08-19T14:16:42.928 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:42.928 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:16:42.928 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:42.929 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.929 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.929 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24580.0:64948 2.f e129/102 2:fe97b461:::benchmark_data_smithi045_46415_object64947:head v 129'4102, mlcod=129'4099)", 2024-08-19T14:16:42.929 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:22.489275+0000", 2024-08-19T14:16:42.929 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 200.39292891700001, 2024-08-19T14:16:42.929 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.018585981000000001, 2024-08-19T14:16:42.929 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:16:42.929 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:16:42.929 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:16:42.929 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:16:42.929 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.929 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:16:42.929 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.489275+0000", 2024-08-19T14:16:42.929 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:42.930 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.930 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.930 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:16:42.930 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.489275+0000", 2024-08-19T14:16:42.930 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:42.930 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.930 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.930 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:16:42.930 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.489276+0000", 2024-08-19T14:16:42.930 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.4200000000000004e-07 2024-08-19T14:16:42.930 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.930 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.930 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:16:42.931 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.489433+0000", 2024-08-19T14:16:42.931 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00015674299999999999 2024-08-19T14:16:42.931 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.931 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.931 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:16:42.931 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.489446+0000", 2024-08-19T14:16:42.931 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2738999999999999e-05 2024-08-19T14:16:42.931 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.931 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.931 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:16:42.931 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.489455+0000", 2024-08-19T14:16:42.931 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.1030000000000008e-06 2024-08-19T14:16:42.931 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.931 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.931 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:16:42.932 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.496819+0000", 2024-08-19T14:16:42.932 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0073646689999999999 2024-08-19T14:16:42.932 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.932 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.932 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:16:42.932 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.496838+0000", 2024-08-19T14:16:42.932 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8564999999999999e-05 2024-08-19T14:16:42.932 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.932 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.932 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:16:42.932 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.507820+0000", 2024-08-19T14:16:42.932 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010982122 2024-08-19T14:16:42.932 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.932 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.933 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:16:42.933 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:22.507861+0000", 2024-08-19T14:16:42.933 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1097999999999999e-05 2024-08-19T14:16:42.933 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:42.933 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:16:42.933 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:42.933 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.933 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.933 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24580.0:70754 2.d e130/102 2:b08378d7:::benchmark_data_smithi045_46415_object70753:head v 130'4497, mlcod=130'4493)", 2024-08-19T14:16:42.933 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:27.103284+0000", 2024-08-19T14:16:42.933 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 195.77892031100001, 2024-08-19T14:16:42.933 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.023299915000000001, 2024-08-19T14:16:42.933 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:16:42.933 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:16:42.933 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:16:42.934 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:16:42.934 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.934 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:16:42.934 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:27.103284+0000", 2024-08-19T14:16:42.934 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:42.934 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.934 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.934 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:16:42.934 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:27.103284+0000", 2024-08-19T14:16:42.934 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:42.934 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.934 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.934 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:16:42.934 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:27.103284+0000", 2024-08-19T14:16:42.935 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:16:42.936 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8000000000000001e-07 2024-08-19T14:16:42.936 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.936 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.936 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:16:42.936 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:27.122506+0000", 2024-08-19T14:16:42.936 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.019221723 2024-08-19T14:16:42.936 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.936 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.937 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:16:42.937 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:27.122521+0000", 2024-08-19T14:16:42.937 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5235000000000001e-05 2024-08-19T14:16:42.937 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.937 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.937 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:16:42.937 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:27.122529+0000", 2024-08-19T14:16:42.937 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5909999999999998e-06 2024-08-19T14:16:42.937 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.937 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.937 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:16:42.937 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:27.122589+0000", 2024-08-19T14:16:42.937 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0098e-05 2024-08-19T14:16:42.937 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.937 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.938 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:16:42.938 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:27.122608+0000", 2024-08-19T14:16:42.938 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9678999999999999e-05 2024-08-19T14:16:42.938 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.938 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.938 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:16:42.938 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:27.126540+0000", 2024-08-19T14:16:42.938 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0039318460000000001 2024-08-19T14:16:42.938 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.938 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.938 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:16:42.938 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:27.126584+0000", 2024-08-19T14:16:42.938 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3362999999999998e-05 2024-08-19T14:16:42.938 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:42.939 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:16:42.939 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:42.939 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.939 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.939 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24580.0:71004 2.f e130/102 2:f1f5b3f0:::benchmark_data_smithi045_46415_object71003:head v 130'4478, mlcod=130'4476)", 2024-08-19T14:16:42.939 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:27.290131+0000", 2024-08-19T14:16:42.939 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 195.59207288600001, 2024-08-19T14:16:42.939 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.018577585000000001, 2024-08-19T14:16:42.939 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:16:42.939 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:16:42.939 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:16:42.939 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:16:42.939 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.939 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:16:42.940 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:27.290131+0000", 2024-08-19T14:16:42.940 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:42.940 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.940 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.940 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:16:42.940 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:27.290131+0000", 2024-08-19T14:16:42.940 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:42.940 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.940 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.940 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:16:42.940 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:27.290133+0000", 2024-08-19T14:16:42.940 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0099999999999998e-06 2024-08-19T14:16:42.940 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.940 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.940 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:16:42.941 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:27.290163+0000", 2024-08-19T14:16:42.941 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9737e-05 2024-08-19T14:16:42.941 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.941 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.941 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:16:42.941 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:27.290170+0000", 2024-08-19T14:16:42.941 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3520000000000001e-06 2024-08-19T14:16:42.941 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.941 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.941 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:16:42.941 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:27.290178+0000", 2024-08-19T14:16:42.941 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7370000000000006e-06 2024-08-19T14:16:42.941 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.941 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.942 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:16:42.942 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:27.290229+0000", 2024-08-19T14:16:42.942 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0809000000000001e-05 2024-08-19T14:16:42.942 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.942 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.942 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:16:42.942 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:27.290247+0000", 2024-08-19T14:16:42.942 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8331000000000001e-05 2024-08-19T14:16:42.942 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.942 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.942 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:16:42.942 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:27.308659+0000", 2024-08-19T14:16:42.942 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.018411594999999999 2024-08-19T14:16:42.942 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.943 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.943 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:16:42.943 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:27.308709+0000", 2024-08-19T14:16:42.943 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0013999999999999e-05 2024-08-19T14:16:42.943 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:42.943 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:16:42.943 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:42.943 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.943 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.943 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24580.0:84047 2.7 e135/127 2:edfa0ea9:::benchmark_data_smithi045_46415_object84046:head v 135'5174, mlcod=0'0)", 2024-08-19T14:16:42.943 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:39.324634+0000", 2024-08-19T14:16:42.943 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 183.557570545, 2024-08-19T14:16:42.943 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.019448039, 2024-08-19T14:16:42.943 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:16:42.944 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:16:42.944 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:16:42.944 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:16:42.944 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.944 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:16:42.944 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:39.324634+0000", 2024-08-19T14:16:42.944 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:42.944 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.944 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.944 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:16:42.944 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:39.324634+0000", 2024-08-19T14:16:42.944 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:42.944 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.944 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.944 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:16:42.945 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:39.324634+0000", 2024-08-19T14:16:42.945 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5699999999999998e-07 2024-08-19T14:16:42.945 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.945 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.945 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:16:42.945 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:39.324702+0000", 2024-08-19T14:16:42.945 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8205000000000002e-05 2024-08-19T14:16:42.945 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.945 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.945 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:16:42.945 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:39.324713+0000", 2024-08-19T14:16:42.945 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1061e-05 2024-08-19T14:16:42.945 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.945 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.946 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:16:42.946 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:39.324720+0000", 2024-08-19T14:16:42.946 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.5699999999999998e-06 2024-08-19T14:16:42.946 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.946 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.946 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:16:42.946 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:39.329466+0000", 2024-08-19T14:16:42.946 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0047466649999999997 2024-08-19T14:16:42.946 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.946 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.946 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:16:42.946 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:39.329496+0000", 2024-08-19T14:16:42.946 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9323e-05 2024-08-19T14:16:42.946 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.947 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.947 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:16:42.947 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:39.344027+0000", 2024-08-19T14:16:42.947 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.014531376 2024-08-19T14:16:42.947 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.947 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.947 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:16:42.947 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:39.344082+0000", 2024-08-19T14:16:42.947 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4382000000000002e-05 2024-08-19T14:16:42.947 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:42.947 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:16:42.947 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:42.947 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.947 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.948 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24580.0:86085 2.7 e136/127 2:ed9d86de:::benchmark_data_smithi045_46415_object86084:head v 136'5307, mlcod=0'0)", 2024-08-19T14:16:42.948 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:41.049265+0000", 2024-08-19T14:16:42.948 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 181.832939541, 2024-08-19T14:16:42.948 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.019980502000000001, 2024-08-19T14:16:42.948 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:16:42.948 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:16:42.948 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:16:42.948 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:16:42.948 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.948 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:16:42.948 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:41.049265+0000", 2024-08-19T14:16:42.948 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:42.948 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.949 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.949 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:16:42.949 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:41.049265+0000", 2024-08-19T14:16:42.949 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:42.949 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.949 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.949 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:16:42.949 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:41.049265+0000", 2024-08-19T14:16:42.949 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6499999999999998e-07 2024-08-19T14:16:42.949 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.949 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.949 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:16:42.949 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:41.049324+0000", 2024-08-19T14:16:42.949 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8538000000000001e-05 2024-08-19T14:16:42.949 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.950 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.950 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:16:42.950 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:41.049340+0000", 2024-08-19T14:16:42.950 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6511000000000002e-05 2024-08-19T14:16:42.950 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.950 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.950 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:16:42.950 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:41.049348+0000", 2024-08-19T14:16:42.950 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2749999999999998e-06 2024-08-19T14:16:42.950 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.950 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.950 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:16:42.950 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:41.051328+0000", 2024-08-19T14:16:42.951 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0019801520000000002 2024-08-19T14:16:42.951 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.951 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.951 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:16:42.951 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:41.051355+0000", 2024-08-19T14:16:42.951 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6877999999999999e-05 2024-08-19T14:16:42.951 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.951 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.951 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:16:42.951 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:41.069193+0000", 2024-08-19T14:16:42.951 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.017838515999999999 2024-08-19T14:16:42.951 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.951 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.951 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:16:42.952 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:41.069245+0000", 2024-08-19T14:16:42.952 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1767000000000001e-05 2024-08-19T14:16:42.952 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:42.952 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:16:42.952 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:42.952 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.952 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.952 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24580.0:86499 2.5 e136/102 2:ac09c600:::benchmark_data_smithi045_46415_object86498:head v 136'5439, mlcod=136'5437)", 2024-08-19T14:16:42.953 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:41.382046+0000", 2024-08-19T14:16:42.953 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 181.50015826399999, 2024-08-19T14:16:42.953 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.020539521000000002, 2024-08-19T14:16:42.953 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:16:42.953 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:16:42.953 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:16:42.953 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:16:42.953 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.953 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:16:42.953 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:41.382046+0000", 2024-08-19T14:16:42.953 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:42.953 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.953 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.953 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:16:42.954 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:41.382046+0000", 2024-08-19T14:16:42.954 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:42.954 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.954 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.954 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:16:42.954 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:41.382047+0000", 2024-08-19T14:16:42.954 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.4300000000000002e-07 2024-08-19T14:16:42.954 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.954 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.954 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:16:42.954 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:41.400143+0000", 2024-08-19T14:16:42.954 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.018096395000000001 2024-08-19T14:16:42.954 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.955 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.955 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:16:42.955 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:41.400159+0000", 2024-08-19T14:16:42.955 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6001000000000001e-05 2024-08-19T14:16:42.955 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.955 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.955 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:16:42.955 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:41.400167+0000", 2024-08-19T14:16:42.955 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.8059999999999995e-06 2024-08-19T14:16:42.955 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.955 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.955 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:16:42.955 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:41.400240+0000", 2024-08-19T14:16:42.955 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3236000000000002e-05 2024-08-19T14:16:42.956 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.956 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.956 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:16:42.956 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:41.400263+0000", 2024-08-19T14:16:42.956 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2977e-05 2024-08-19T14:16:42.956 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.956 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.956 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:16:42.956 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:41.402545+0000", 2024-08-19T14:16:42.956 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0022824450000000001 2024-08-19T14:16:42.956 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.956 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.956 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:16:42.956 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:41.402585+0000", 2024-08-19T14:16:42.957 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9817999999999997e-05 2024-08-19T14:16:42.957 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:42.957 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:16:42.957 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:42.957 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.957 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.957 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24580.0:98618 2.7 e139/127 2:e41d73e1:::benchmark_data_smithi045_46415_object98617:head v 139'6106, mlcod=0'0)", 2024-08-19T14:16:42.957 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:52.307086+0000", 2024-08-19T14:16:42.957 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 170.57511815399999, 2024-08-19T14:16:42.957 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.019682899, 2024-08-19T14:16:42.957 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:16:42.957 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:16:42.957 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:16:42.957 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:16:42.958 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.958 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:16:42.958 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:52.307086+0000", 2024-08-19T14:16:42.958 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:42.958 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.958 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.958 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:16:42.958 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:52.307086+0000", 2024-08-19T14:16:42.958 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:42.958 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.958 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.958 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:16:42.958 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:52.307087+0000", 2024-08-19T14:16:42.958 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.3399999999999998e-07 2024-08-19T14:16:42.959 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.959 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.959 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:16:42.959 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:52.307165+0000", 2024-08-19T14:16:42.959 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7769999999999996e-05 2024-08-19T14:16:42.959 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.959 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.959 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:16:42.959 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:52.307181+0000", 2024-08-19T14:16:42.959 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6860000000000001e-05 2024-08-19T14:16:42.959 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.959 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.959 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:16:42.959 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:52.307190+0000", 2024-08-19T14:16:42.960 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.4570000000000005e-06 2024-08-19T14:16:42.960 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.960 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.960 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:16:42.960 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:52.307254+0000", 2024-08-19T14:16:42.960 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4103000000000006e-05 2024-08-19T14:16:42.960 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.960 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.960 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:16:42.960 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:52.307281+0000", 2024-08-19T14:16:42.960 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6758e-05 2024-08-19T14:16:42.960 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.960 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.960 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:16:42.961 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:52.326728+0000", 2024-08-19T14:16:42.961 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.019447762 2024-08-19T14:16:42.961 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.961 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.961 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:16:42.961 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:52.326769+0000", 2024-08-19T14:16:42.961 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0355000000000002e-05 2024-08-19T14:16:42.961 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:42.961 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:16:42.961 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:42.961 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.961 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.961 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24580.0:100982 2.7 e141/127 2:eeafa31f:::benchmark_data_smithi045_46415_object100981:head v 141'6273, mlcod=0'0)", 2024-08-19T14:16:42.962 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:54.502352+0000", 2024-08-19T14:16:42.962 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 168.37985244999999, 2024-08-19T14:16:42.962 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.019840074999999999, 2024-08-19T14:16:42.962 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:16:42.962 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:16:42.962 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:16:42.962 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:16:42.962 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.962 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:16:42.962 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:54.502352+0000", 2024-08-19T14:16:42.962 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:42.962 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.962 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.962 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:16:42.963 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:54.502352+0000", 2024-08-19T14:16:42.963 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:42.963 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.963 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.963 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:16:42.963 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:54.502352+0000", 2024-08-19T14:16:42.963 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7500000000000001e-07 2024-08-19T14:16:42.963 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.963 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.963 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:16:42.963 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:54.510583+0000", 2024-08-19T14:16:42.963 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0082310960000000002 2024-08-19T14:16:42.963 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.963 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.964 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:16:42.964 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:54.510594+0000", 2024-08-19T14:16:42.964 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1012000000000001e-05 2024-08-19T14:16:42.964 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.964 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.964 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:16:42.964 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:54.510601+0000", 2024-08-19T14:16:42.964 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7320000000000001e-06 2024-08-19T14:16:42.964 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.964 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.964 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:16:42.964 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:54.510647+0000", 2024-08-19T14:16:42.964 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6307999999999998e-05 2024-08-19T14:16:42.964 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.965 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.965 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:16:42.965 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:54.510666+0000", 2024-08-19T14:16:42.965 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8885e-05 2024-08-19T14:16:42.965 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.965 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.965 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:16:42.965 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:54.522152+0000", 2024-08-19T14:16:42.965 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011486478 2024-08-19T14:16:42.965 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.965 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.965 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:16:42.965 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:54.522192+0000", 2024-08-19T14:16:42.966 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9189e-05 2024-08-19T14:16:42.966 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:42.966 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:16:42.966 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:42.966 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.966 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.966 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24580.0:171212 2.f e169/102 2:f8cd8517:::benchmark_data_smithi045_46415_object47931:head v 169'10747, mlcod=169'10744)", 2024-08-19T14:16:42.966 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:15:12.433327+0000", 2024-08-19T14:16:42.966 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 90.448876781999999, 2024-08-19T14:16:42.966 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.055094690000000002, 2024-08-19T14:16:42.966 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:16:42.966 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:16:42.967 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:16:42.967 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:16:42.967 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.967 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:16:42.967 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.433327+0000", 2024-08-19T14:16:42.967 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:42.967 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.967 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.967 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:16:42.967 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.433327+0000", 2024-08-19T14:16:42.967 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:42.967 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.967 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.967 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:16:42.968 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.433328+0000", 2024-08-19T14:16:42.968 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.7900000000000007e-07 2024-08-19T14:16:42.968 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.968 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.968 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:16:42.968 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.433334+0000", 2024-08-19T14:16:42.968 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1190000000000002e-06 2024-08-19T14:16:42.968 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.968 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.968 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:16:42.968 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.433342+0000", 2024-08-19T14:16:42.968 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4850000000000003e-06 2024-08-19T14:16:42.968 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.968 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.969 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:16:42.969 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.433346+0000", 2024-08-19T14:16:42.969 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.527e-06 2024-08-19T14:16:42.969 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.969 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.969 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:16:42.969 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.433388+0000", 2024-08-19T14:16:42.969 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1619999999999998e-05 2024-08-19T14:16:42.969 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.969 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.969 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:16:42.969 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.433404+0000", 2024-08-19T14:16:42.969 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5991e-05 2024-08-19T14:16:42.969 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.970 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.970 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:16:42.970 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.488372+0000", 2024-08-19T14:16:42.970 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.054968052000000003 2024-08-19T14:16:42.970 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.970 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.970 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:16:42.970 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.488422+0000", 2024-08-19T14:16:42.970 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9917000000000003e-05 2024-08-19T14:16:42.970 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:42.970 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:16:42.970 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:42.970 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.970 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.971 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "replica_scrub(pg: 2.7,from:0'0,to:169'10634,epoch:169/164,start:2:e4f5c3df:::benchmark_data_smithi045_46415_object101587:0,end:2:e553be1d:::benchmark_data_smithi045_46415_object51347:0,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-08-19T14:16:42.971 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:15:12.436695+0000", 2024-08-19T14:16:42.971 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 90.445509193999996, 2024-08-19T14:16:42.971 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.039097697000000001, 2024-08-19T14:16:42.971 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:16:42.971 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:16:42.971 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-08-19T14:16:42.971 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:16:42.971 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.971 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:16:42.971 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.436695+0000", 2024-08-19T14:16:42.971 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:42.971 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.971 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.972 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:16:42.972 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.436695+0000", 2024-08-19T14:16:42.972 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:42.972 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.972 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.972 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:16:42.972 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.436696+0000", 2024-08-19T14:16:42.972 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8899999999999999e-07 2024-08-19T14:16:42.972 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.972 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.972 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:16:42.972 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.436702+0000", 2024-08-19T14:16:42.972 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9859999999999999e-06 2024-08-19T14:16:42.972 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.973 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.973 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:16:42.973 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.475491+0000", 2024-08-19T14:16:42.973 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.038789259999999999 2024-08-19T14:16:42.973 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.973 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.973 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:16:42.973 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.475509+0000", 2024-08-19T14:16:42.973 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8488e-05 2024-08-19T14:16:42.973 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.973 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.973 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:16:42.973 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.475746+0000", 2024-08-19T14:16:42.973 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00023667600000000001 2024-08-19T14:16:42.974 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.974 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.974 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:16:42.974 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.475761+0000", 2024-08-19T14:16:42.974 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4901e-05 2024-08-19T14:16:42.974 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.974 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.974 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:16:42.974 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.475793+0000", 2024-08-19T14:16:42.974 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1696999999999998e-05 2024-08-19T14:16:42.974 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:42.974 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:16:42.974 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:42.975 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.975 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.975 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24580.0:171221 2.d e169/102 2:b5ea1297:::benchmark_data_smithi045_46415_object47940:head v 169'10755, mlcod=169'10753)", 2024-08-19T14:16:42.975 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:15:12.444916+0000", 2024-08-19T14:16:42.975 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 90.437288241000005, 2024-08-19T14:16:42.975 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.039377216999999999, 2024-08-19T14:16:42.975 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:16:42.975 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:16:42.975 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:16:42.975 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:16:42.975 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.975 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:16:42.975 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.444916+0000", 2024-08-19T14:16:42.976 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:42.976 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.976 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.976 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:16:42.976 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.444916+0000", 2024-08-19T14:16:42.976 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:42.976 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.976 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.976 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:16:42.976 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.444917+0000", 2024-08-19T14:16:42.976 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.018e-06 2024-08-19T14:16:42.976 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.976 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.976 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:16:42.976 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.444926+0000", 2024-08-19T14:16:42.977 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.8750000000000006e-06 2024-08-19T14:16:42.977 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.977 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.977 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:16:42.977 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.475484+0000", 2024-08-19T14:16:42.977 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.030558333 2024-08-19T14:16:42.977 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.977 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.977 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:16:42.977 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.475502+0000", 2024-08-19T14:16:42.977 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7948000000000002e-05 2024-08-19T14:16:42.977 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.977 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.977 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:16:42.978 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.475649+0000", 2024-08-19T14:16:42.978 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000147503 2024-08-19T14:16:42.978 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.978 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.978 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:16:42.978 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.475673+0000", 2024-08-19T14:16:42.978 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3181999999999999e-05 2024-08-19T14:16:42.978 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.978 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.978 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:16:42.978 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.484205+0000", 2024-08-19T14:16:42.978 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0085321779999999996 2024-08-19T14:16:42.978 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.978 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.979 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:16:42.979 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.484293+0000", 2024-08-19T14:16:42.979 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.8179999999999999e-05 2024-08-19T14:16:42.979 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:42.979 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:16:42.979 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:42.979 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.979 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.979 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24580.0:178167 2.5 e176/102 2:aaf1e331:::benchmark_data_smithi045_46415_object54886:head v 176'11164, mlcod=176'11161)", 2024-08-19T14:16:42.979 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:15:23.053948+0000", 2024-08-19T14:16:42.979 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 79.828255823999996, 2024-08-19T14:16:42.979 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.019757885999999999, 2024-08-19T14:16:42.979 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:16:42.979 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:16:42.979 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:16:42.980 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:16:42.980 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.980 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:16:42.980 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:23.053948+0000", 2024-08-19T14:16:42.980 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:42.980 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.980 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.980 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:16:42.980 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:23.053948+0000", 2024-08-19T14:16:42.980 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:42.980 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.980 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.980 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:16:42.980 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:23.053950+0000", 2024-08-19T14:16:42.981 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.466e-06 2024-08-19T14:16:42.981 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.981 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.981 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:16:42.981 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:23.053956+0000", 2024-08-19T14:16:42.981 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3339999999999998e-06 2024-08-19T14:16:42.981 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.981 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.981 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:16:42.981 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:23.053967+0000", 2024-08-19T14:16:42.981 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0721e-05 2024-08-19T14:16:42.981 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.981 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.981 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:16:42.981 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:23.053974+0000", 2024-08-19T14:16:42.982 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1509999999999998e-06 2024-08-19T14:16:42.982 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.982 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.982 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:16:42.982 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:23.054035+0000", 2024-08-19T14:16:42.982 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1328000000000006e-05 2024-08-19T14:16:42.982 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.982 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.982 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:16:42.982 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:23.054057+0000", 2024-08-19T14:16:42.982 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1437000000000002e-05 2024-08-19T14:16:42.982 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.982 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.982 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:16:42.983 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:23.073651+0000", 2024-08-19T14:16:42.983 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.019594295000000001 2024-08-19T14:16:42.983 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:42.983 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:42.983 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:16:42.983 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:23.073706+0000", 2024-08-19T14:16:42.983 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5154000000000001e-05 2024-08-19T14:16:42.983 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:42.983 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:16:42.983 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:42.983 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:42.983 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:16:42.983 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:16:43.008 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:43.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:43.009 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:16:43.110 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:16:43.135 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:43.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:43.136 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:16:43.237 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:16:43.263 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:43.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:43.264 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:16:43.365 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:16:43.394 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:43.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:43.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:16:43.496 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:16:43.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:43 smithi032 ceph-mon[20004]: pgmap v518: 27 pgs: 1 active+clean+remapped, 1 active+remapped+backfilling, 2 active+undersized+remapped, 5 peering, 18 active+clean; 341 MiB data, 2.1 GiB used, 803 GiB / 805 GiB avail; 0 B/s wr, 909 op/s; 304/16359 objects misplaced (1.858%); 5.6 MiB/s, 89 objects/s recovering 2024-08-19T14:16:43.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:43 smithi032 ceph-mon[20004]: osdmap e215: 12 total, 12 up, 9 in 2024-08-19T14:16:43.531 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:43.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:43.532 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:16:43.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:43 smithi045 ceph-mon[20143]: pgmap v518: 27 pgs: 1 active+clean+remapped, 1 active+remapped+backfilling, 2 active+undersized+remapped, 5 peering, 18 active+clean; 341 MiB data, 2.1 GiB used, 803 GiB / 805 GiB avail; 0 B/s wr, 909 op/s; 304/16359 objects misplaced (1.858%); 5.6 MiB/s, 89 objects/s recovering 2024-08-19T14:16:43.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:43 smithi045 ceph-mon[20143]: osdmap e215: 12 total, 12 up, 9 in 2024-08-19T14:16:43.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:43 smithi031 ceph-mon[16336]: pgmap v518: 27 pgs: 1 active+clean+remapped, 1 active+remapped+backfilling, 2 active+undersized+remapped, 5 peering, 18 active+clean; 341 MiB data, 2.1 GiB used, 803 GiB / 805 GiB avail; 0 B/s wr, 909 op/s; 304/16359 objects misplaced (1.858%); 5.6 MiB/s, 89 objects/s recovering 2024-08-19T14:16:43.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:43 smithi031 ceph-mon[16336]: osdmap e215: 12 total, 12 up, 9 in 2024-08-19T14:16:43.633 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:16:43.656 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:43.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:43.657 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:16:43.663 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-08-19T14:16:43.758 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:16:43.782 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:43.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:43.784 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:16:43.885 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:16:43.917 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:43.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:43.918 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:16:44.019 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:16:44.049 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:44.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:44.050 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:16:44.061 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 7, 8, 9, 11] out_osds: [10, 6, 2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-08-19T14:16:44.061 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-19T14:16:44.061 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_0 2024-08-19T14:16:44.062 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-19T14:16:44.070 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:16:44.152 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:16:44.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:44.198 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:44.198 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:16:44.243 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:16:44.299 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:16:44.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:44.357 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:44.358 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:16:44.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:44 smithi031 ceph-mon[16336]: osdmap e216: 12 total, 12 up, 9 in 2024-08-19T14:16:44.459 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:16:44.486 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:16:44.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:44.489 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:44.489 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:16:44.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:44 smithi032 ceph-mon[20004]: osdmap e216: 12 total, 12 up, 9 in 2024-08-19T14:16:44.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:44 smithi045 ceph-mon[20143]: osdmap e216: 12 total, 12 up, 9 in 2024-08-19T14:16:44.591 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:16:44.624 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:44.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:44.625 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:16:44.726 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:16:44.738 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-19T14:16:44.758 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:44.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:44.759 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:16:44.861 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:16:44.898 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:44.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:44.899 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:16:44.903 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:16:44.904 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-08-19T14:16:44.904 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:44.904 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24580.0:243474 2.e e216/214 2:7660ccaa:::benchmark_data_smithi045_46415_object120193:head v 216'15165, mlcod=216'15163)", 2024-08-19T14:16:44.904 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:16:44.900769+0000", 2024-08-19T14:16:44.904 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.002122203, 2024-08-19T14:16:44.904 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0021520860000000001, 2024-08-19T14:16:44.904 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:16:44.904 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:16:44.904 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-08-19T14:16:44.904 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:16:44.904 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:44.904 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:16:44.905 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:16:44.900769+0000", 2024-08-19T14:16:44.905 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:44.905 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:44.905 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:44.905 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:16:44.905 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:16:44.900769+0000", 2024-08-19T14:16:44.905 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:44.905 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:44.905 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:44.905 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:16:44.905 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:16:44.900770+0000", 2024-08-19T14:16:44.905 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.8100000000000001e-07 2024-08-19T14:16:44.905 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:44.905 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:44.906 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:16:44.906 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:16:44.900778+0000", 2024-08-19T14:16:44.906 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6720000000000004e-06 2024-08-19T14:16:44.906 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:44.906 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:44.906 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:16:44.906 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:16:44.900795+0000", 2024-08-19T14:16:44.906 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7419e-05 2024-08-19T14:16:44.906 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:44.906 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:44.906 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:16:44.906 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:16:44.900802+0000", 2024-08-19T14:16:44.906 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1690000000000003e-06 2024-08-19T14:16:44.906 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:44.907 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:44.907 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:16:44.907 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:16:44.900889+0000", 2024-08-19T14:16:44.907 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6769999999999998e-05 2024-08-19T14:16:44.907 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:44.907 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:44.907 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:16:44.907 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:16:44.900909+0000", 2024-08-19T14:16:44.907 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9873000000000001e-05 2024-08-19T14:16:44.907 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:44.907 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:16:44.907 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:44.907 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:44.907 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-08-19T14:16:44.908 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 1 2024-08-19T14:16:44.908 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:16:44.999 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:16:45.033 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:45.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:45.034 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:16:45.104 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:16:45.135 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:16:45.191 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:45.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:45.192 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:16:45.293 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:16:45.320 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:45.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:45.321 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:16:45.423 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:16:45.428 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:16:45.429 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-08-19T14:16:45.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:45.455 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:45.455 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:16:45.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:45 smithi045 ceph-mon[20143]: pgmap v521: 27 pgs: 1 active+clean+remapped, 1 active+remapped+backfilling, 1 active+undersized+remapped, 3 peering, 21 active+clean; 316 MiB data, 2.0 GiB used, 803 GiB / 805 GiB avail; 0 B/s wr, 437 op/s; 304/15147 objects misplaced (2.007%); 15 MiB/s, 247 objects/s recovering 2024-08-19T14:16:45.555 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:16:45.584 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:45.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:45.585 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:16:45.589 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:45 smithi032 ceph-mon[20004]: pgmap v521: 27 pgs: 1 active+clean+remapped, 1 active+remapped+backfilling, 1 active+undersized+remapped, 3 peering, 21 active+clean; 316 MiB data, 2.0 GiB used, 803 GiB / 805 GiB avail; 0 B/s wr, 437 op/s; 304/15147 objects misplaced (2.007%); 15 MiB/s, 247 objects/s recovering 2024-08-19T14:16:45.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:45 smithi031 ceph-mon[16336]: pgmap v521: 27 pgs: 1 active+clean+remapped, 1 active+remapped+backfilling, 1 active+undersized+remapped, 3 peering, 21 active+clean; 316 MiB data, 2.0 GiB used, 803 GiB / 805 GiB avail; 0 B/s wr, 437 op/s; 304/15147 objects misplaced (2.007%); 15 MiB/s, 247 objects/s recovering 2024-08-19T14:16:45.686 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:16:45.715 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:45.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:45.716 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:16:45.817 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:16:45.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:45.848 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:45.848 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:16:45.916 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:16:45.919 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-08-19T14:16:45.949 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:16:45.955 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:16:45.956 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:16:45.980 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:45.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:45.982 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:16:45.995 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:16:45.995 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:16:46.023 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:16:46.024 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:16:46.051 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:16:46.052 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:16:46.080 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:16:46.080 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:16:46.083 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:16:46.109 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:16:46.109 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:16:46.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:46.119 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:46.120 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:16:46.141 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:16:46.141 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:16:46.177 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:16:46.177 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:16:46.212 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:16:46.212 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:16:46.220 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:16:46.238 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:16:46.238 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:16:46.251 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:46.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:46.252 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:16:46.276 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:16:46.277 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:16:46.305 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:16:46.305 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:16:46.317 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:16:46.353 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:16:46.389 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:46.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:46.390 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:16:46.491 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:16:46.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:46.524 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:46.524 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:16:46.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:46 smithi045 ceph-mon[20143]: from='client.25058 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:16:46.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:46 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2006112960' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:16:46.553 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:46 smithi031 ceph-mon[16336]: from='client.25058 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:16:46.553 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:46 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2006112960' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:16:46.594 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:46 smithi032 ceph-mon[20004]: from='client.25058 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:16:46.594 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:46 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2006112960' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:16:46.601 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":521,"stamp":"2024-08-19T14:16:43.798980+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":331285048,"num_objects":5049,"num_object_clones":0,"num_object_copies":15147,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":304,"num_objects_unfound":0,"num_objects_dirty":5049,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":241511,"num_write_kb":7889793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25081,"num_bytes_recovered":1605308960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":162361,"ondisk_log_size":162361,"up":81,"acting":80,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":77,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":2122708,"kb_used_data":1028568,"kb_used_omap":14,"kb_used_meta":1094001,"kb_avail":841583660,"statfs":{"total":863955320832,"available":861781667840,"internally_reserved":0,"allocated":1053253632,"data_stored":1043908814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14353,"internal_metadata":1120258031},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-114819072,"num_objects":-1752,"num_object_clones":0,"num_object_copies":-5256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":304,"num_objects_unfound":0,"num_objects_dirty":-1752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1752,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":990,"num_bytes_recovered":65016352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000670"},"pg_stats":[{"pgid":"2.f","version":"211'15154","reported_seq":15496,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-08-19T14:16:39.216195+0000","last_change":"2024-08-19T14:16:06.289451+0000","last_active":"2024-08-19T14:16:39.216195+0000","last_peered":"2024-08-19T14:16:39.216195+0000","last_clean":"2024-08-19T14:16:39.216195+0000","last_became_active":"2024-08-19T14:12:43.417591+0000","last_became_peered":"2024-08-19T14:12:43.417591+0000","last_unstale":"2024-08-19T14:16:39.216195+0000","last_undegraded":"2024-08-19T14:16:39.216195+0000","last_fullsized":"2024-08-19T14:16:39.216195+0000","mapping_epoch":102,"log_start":"135'5100","ondisk_log_start":"135'5100","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"197'13388","last_scrub_stamp":"2024-08-19T14:16:06.289300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:16:06.289300+0000","objects_scrubbed":2129,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:17:08.304647+0000","scrub_duration":507,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23068672,"num_objects":352,"num_object_clones":0,"num_object_copies":1056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15154,"num_write_kb":496192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.e","version":"215'15060","reported_seq":27646,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-08-19T14:16:42.350243+0000","last_change":"2024-08-19T14:16:42.301236+0000","last_active":"2024-08-19T14:16:42.350243+0000","last_peered":"2024-08-19T14:16:42.350243+0000","last_clean":"2024-08-19T14:16:42.350243+0000","last_became_active":"2024-08-19T14:16:42.252189+0000","last_became_peered":"2024-08-19T14:16:42.252189+0000","last_unstale":"2024-08-19T14:16:42.350243+0000","last_undegraded":"2024-08-19T14:16:42.350243+0000","last_fullsized":"2024-08-19T14:16:42.350243+0000","mapping_epoch":214,"log_start":"135'5000","ondisk_log_start":"135'5000","created":102,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'14730","last_scrub_stamp":"2024-08-19T14:16:33.170932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:16:33.170932+0000","objects_scrubbed":635,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:17:57.187104+0000","scrub_duration":134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19660800,"num_objects":300,"num_object_clones":0,"num_object_copies":900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15060,"num_write_kb":491520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6120,"num_bytes_recovered":401014784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,5],"acting":[7,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"215'15089","reported_seq":15454,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-08-19T14:16:43.252550+0000","last_change":"2024-08-19T14:15:35.692830+0000","last_active":"2024-08-19T14:16:43.252550+0000","last_peered":"2024-08-19T14:16:43.252550+0000","last_clean":"2024-08-19T14:16:43.252550+0000","last_became_active":"2024-08-19T14:12:43.417506+0000","last_became_peered":"2024-08-19T14:12:43.417506+0000","last_unstale":"2024-08-19T14:16:43.252550+0000","last_undegraded":"2024-08-19T14:16:43.252550+0000","last_fullsized":"2024-08-19T14:16:43.252550+0000","mapping_epoch":102,"log_start":"134'5000","ondisk_log_start":"134'5000","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"180'11879","last_scrub_stamp":"2024-08-19T14:15:35.692699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:15:35.692699+0000","objects_scrubbed":3525,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:16:57.995467+0000","scrub_duration":1207,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19267608,"num_objects":295,"num_object_clones":0,"num_object_copies":885,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":295,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15089,"num_write_kb":492225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.c","version":"213'15026","reported_seq":15409,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-08-19T14:16:40.531738+0000","last_change":"2024-08-19T14:16:39.378051+0000","last_active":"2024-08-19T14:16:40.531738+0000","last_peered":"2024-08-19T14:16:40.531738+0000","last_clean":"2024-08-19T14:16:40.531738+0000","last_became_active":"2024-08-19T14:12:43.038912+0000","last_became_peered":"2024-08-19T14:12:43.038912+0000","last_unstale":"2024-08-19T14:16:40.531738+0000","last_undegraded":"2024-08-19T14:16:40.531738+0000","last_fullsized":"2024-08-19T14:16:40.531738+0000","mapping_epoch":102,"log_start":"135'5000","ondisk_log_start":"135'5000","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"211'15018","last_scrub_stamp":"2024-08-19T14:16:39.377932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:16:39.377932+0000","objects_scrubbed":324,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:17:49.123245+0000","scrub_duration":77,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20709376,"num_objects":316,"num_object_clones":0,"num_object_copies":948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15026,"num_write_kb":490944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,1],"acting":[11,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"70'139","reported_seq":617,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-08-19T14:16:42.251537+0000","last_change":"2024-08-19T14:16:42.251163+0000","last_active":"2024-08-19T14:16:42.251537+0000","last_peered":"2024-08-19T14:16:42.251537+0000","last_clean":"2024-08-19T14:16:42.251537+0000","last_became_active":"2024-08-19T14:16:42.250702+0000","last_became_peered":"2024-08-19T14:16:42.250702+0000","last_unstale":"2024-08-19T14:16:42.251537+0000","last_undegraded":"2024-08-19T14:16:42.251537+0000","last_fullsized":"2024-08-19T14:16:42.251537+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:15:59.869266+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:15:59.869266+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:17:24.361856+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,9],"acting":[7,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"214'15102","reported_seq":16143,"reported_epoch":214,"state":"active+clean+remapped","last_fresh":"2024-08-19T14:16:41.525379+0000","last_change":"2024-08-19T14:16:41.278162+0000","last_active":"2024-08-19T14:16:41.525379+0000","last_peered":"2024-08-19T14:16:41.525379+0000","last_clean":"2024-08-19T14:16:41.525379+0000","last_became_active":"2024-08-19T14:16:40.360387+0000","last_became_peered":"2024-08-19T14:16:40.360387+0000","last_unstale":"2024-08-19T14:16:41.525379+0000","last_undegraded":"2024-08-19T14:16:41.525379+0000","last_fullsized":"2024-08-19T14:16:41.525379+0000","mapping_epoch":212,"log_start":"135'5100","ondisk_log_start":"135'5100","created":102,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"179'11759","last_scrub_stamp":"2024-08-19T14:15:33.408298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:15:33.408298+0000","objects_scrubbed":3675,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":864,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20185088,"num_objects":308,"num_object_clones":0,"num_object_copies":924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15102,"num_write_kb":493120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":22151168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,9],"acting":[5,7,2],"avail_no_missing":["5","2","7","9"],"object_location_counts":[{"shards":"2,5,7","objects":308}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.3","version":"70'139","reported_seq":517,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-08-19T14:16:43.251343+0000","last_change":"2024-08-19T14:16:06.679203+0000","last_active":"2024-08-19T14:16:43.251343+0000","last_peered":"2024-08-19T14:16:43.251343+0000","last_clean":"2024-08-19T14:16:43.251343+0000","last_became_active":"2024-08-19T14:13:22.727234+0000","last_became_peered":"2024-08-19T14:13:22.727234+0000","last_unstale":"2024-08-19T14:16:43.251343+0000","last_undegraded":"2024-08-19T14:16:43.251343+0000","last_fullsized":"2024-08-19T14:16:43.251343+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:16:06.679075+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:16:06.679075+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:17:31.357116+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"215'15154","reported_seq":21588,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-08-19T14:16:42.318320+0000","last_change":"2024-08-19T14:16:35.161651+0000","last_active":"2024-08-19T14:16:42.318320+0000","last_peered":"2024-08-19T14:16:42.318320+0000","last_clean":"2024-08-19T14:16:42.318320+0000","last_became_active":"2024-08-19T14:13:32.749994+0000","last_became_peered":"2024-08-19T14:13:32.749994+0000","last_unstale":"2024-08-19T14:16:42.318320+0000","last_undegraded":"2024-08-19T14:16:42.318320+0000","last_fullsized":"2024-08-19T14:16:42.318320+0000","mapping_epoch":132,"log_start":"135'5100","ondisk_log_start":"135'5100","created":102,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"209'14927","last_scrub_stamp":"2024-08-19T14:16:35.161496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:16:35.161496+0000","objects_scrubbed":512,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:17:47.799709+0000","scrub_duration":111,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18612224,"num_objects":284,"num_object_clones":0,"num_object_copies":852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":284,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15154,"num_write_kb":494016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3024,"num_bytes_recovered":198180864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,11],"acting":[7,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"70'139","reported_seq":456,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-08-19T14:16:40.231231+0000","last_change":"2024-08-19T14:16:00.592808+0000","last_active":"2024-08-19T14:16:40.231231+0000","last_peered":"2024-08-19T14:16:40.231231+0000","last_clean":"2024-08-19T14:16:40.231231+0000","last_became_active":"2024-08-19T14:15:59.993566+0000","last_became_peered":"2024-08-19T14:15:59.993566+0000","last_unstale":"2024-08-19T14:16:40.231231+0000","last_undegraded":"2024-08-19T14:16:40.231231+0000","last_fullsized":"2024-08-19T14:16:40.231231+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:16:00.592678+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:16:00.592678+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:17:07.175192+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,8],"acting":[0,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"213'15185","reported_seq":23080,"reported_epoch":213,"state":"active+remapped+backfilling","last_fresh":"2024-08-19T14:16:40.648429+0000","last_change":"2024-08-19T14:16:40.406631+0000","last_active":"2024-08-19T14:16:40.648429+0000","last_peered":"2024-08-19T14:16:40.648429+0000","last_clean":"2024-08-19T14:16:38.215326+0000","last_became_active":"2024-08-19T14:16:40.358545+0000","last_became_peered":"2024-08-19T14:16:40.358545+0000","last_unstale":"2024-08-19T14:16:40.648429+0000","last_undegraded":"2024-08-19T14:16:40.648429+0000","last_fullsized":"2024-08-19T14:16:40.648429+0000","mapping_epoch":212,"log_start":"135'5100","ondisk_log_start":"135'5100","created":102,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'14862","last_scrub_stamp":"2024-08-19T14:16:33.133135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:16:33.133135+0000","objects_scrubbed":652,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":398,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21037056,"num_objects":321,"num_object_clones":0,"num_object_copies":963,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":304,"num_objects_unfound":0,"num_objects_dirty":321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15185,"num_write_kb":496192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3810,"num_bytes_recovered":249692160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,1],"acting":[0,8,2],"avail_no_missing":["0","2","8"],"object_location_counts":[{"shards":"0,2,8","objects":321}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"70'139","reported_seq":514,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-08-19T14:16:43.252267+0000","last_change":"2024-08-19T14:16:42.252108+0000","last_active":"2024-08-19T14:16:43.252267+0000","last_peered":"2024-08-19T14:16:43.252267+0000","last_clean":"2024-08-19T14:16:43.252267+0000","last_became_active":"2024-08-19T14:16:42.251603+0000","last_became_peered":"2024-08-19T14:16:42.251603+0000","last_unstale":"2024-08-19T14:16:43.252267+0000","last_undegraded":"2024-08-19T14:16:43.252267+0000","last_fullsized":"2024-08-19T14:16:43.252267+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:16:30.043526+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:16:30.043526+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:17:54.530868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,11],"acting":[9,0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.2","version":"214'15148","reported_seq":23328,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-08-19T14:16:41.478791+0000","last_change":"2024-08-19T14:15:31.539950+0000","last_active":"2024-08-19T14:16:41.478791+0000","last_peered":"2024-08-19T14:16:41.478791+0000","last_clean":"2024-08-19T14:16:41.478791+0000","last_became_active":"2024-08-19T14:13:58.623348+0000","last_became_peered":"2024-08-19T14:13:58.623348+0000","last_unstale":"2024-08-19T14:16:41.478791+0000","last_undegraded":"2024-08-19T14:16:41.478791+0000","last_fullsized":"2024-08-19T14:16:41.478791+0000","mapping_epoch":142,"log_start":"135'5100","ondisk_log_start":"135'5100","created":102,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"178'11591","last_scrub_stamp":"2024-08-19T14:15:31.539835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:15:31.539835+0000","objects_scrubbed":3877,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:17:01.072608+0000","scrub_duration":986,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18874368,"num_objects":288,"num_object_clones":0,"num_object_copies":864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15148,"num_write_kb":493952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3902,"num_bytes_recovered":255721472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"70'139","reported_seq":411,"reported_epoch":213,"state":"peering","last_fresh":"2024-08-19T14:16:40.232894+0000","last_change":"2024-08-19T14:16:41.243111+0000","last_active":"2024-08-19T14:16:41.243111+0000","last_peered":"2024-08-19T14:16:40.232894+0000","last_clean":"2024-08-19T14:16:35.203369+0000","last_became_active":"2024-08-19T14:16:40.232629+0000","last_became_peered":"2024-08-19T14:16:40.232629+0000","last_unstale":"2024-08-19T14:16:40.232894+0000","last_undegraded":"2024-08-19T14:16:40.232894+0000","last_fullsized":"2024-08-19T14:16:40.227170+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:16:10.672834+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"216'15168","reported_seq":22413,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-08-19T14:16:43.329865+0000","last_change":"2024-08-19T14:16:43.316934+0000","last_active":"2024-08-19T14:16:43.329865+0000","last_peered":"2024-08-19T14:16:43.329865+0000","last_clean":"2024-08-19T14:16:43.329865+0000","last_became_active":"2024-08-19T14:16:43.254813+0000","last_became_peered":"2024-08-19T14:16:43.254813+0000","last_unstale":"2024-08-19T14:16:43.329865+0000","last_undegraded":"2024-08-19T14:16:43.329865+0000","last_fullsized":"2024-08-19T14:16:43.329865+0000","mapping_epoch":215,"log_start":"135'5100","ondisk_log_start":"135'5100","created":102,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"180'11953","last_scrub_stamp":"2024-08-19T14:15:37.501902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:15:37.501902+0000","objects_scrubbed":3536,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:17:00.198024+0000","scrub_duration":1934,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18350080,"num_objects":280,"num_object_clones":0,"num_object_copies":840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15168,"num_write_kb":494336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3426,"num_bytes_recovered":224526336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,8],"acting":[1,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"70'139","reported_seq":505,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-08-19T14:16:43.252581+0000","last_change":"2024-08-19T14:16:36.969224+0000","last_active":"2024-08-19T14:16:43.252581+0000","last_peered":"2024-08-19T14:16:43.252581+0000","last_clean":"2024-08-19T14:16:43.252581+0000","last_became_active":"2024-08-19T14:13:11.647085+0000","last_became_peered":"2024-08-19T14:13:11.647085+0000","last_unstale":"2024-08-19T14:16:43.252581+0000","last_undegraded":"2024-08-19T14:16:43.252581+0000","last_fullsized":"2024-08-19T14:16:43.252581+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:16:36.969114+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:16:36.969114+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:17:44.848868+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,4],"acting":[9,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.5","version":"211'15085","reported_seq":15430,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-08-19T14:16:39.216187+0000","last_change":"2024-08-19T14:16:05.351767+0000","last_active":"2024-08-19T14:16:39.216187+0000","last_peered":"2024-08-19T14:16:39.216187+0000","last_clean":"2024-08-19T14:16:39.216187+0000","last_became_active":"2024-08-19T14:12:43.417616+0000","last_became_peered":"2024-08-19T14:12:43.417616+0000","last_unstale":"2024-08-19T14:16:39.216187+0000","last_undegraded":"2024-08-19T14:16:39.216187+0000","last_fullsized":"2024-08-19T14:16:39.216187+0000","mapping_epoch":102,"log_start":"135'5000","ondisk_log_start":"135'5000","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"196'13313","last_scrub_stamp":"2024-08-19T14:16:05.351209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:16:05.351209+0000","objects_scrubbed":2149,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:17:30.130226+0000","scrub_duration":592,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23658496,"num_objects":361,"num_object_clones":0,"num_object_copies":1083,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":361,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15085,"num_write_kb":494272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.5","version":"70'139","reported_seq":307,"reported_epoch":213,"state":"active+undersized+remapped","last_fresh":"2024-08-19T14:16:40.232725+0000","last_change":"2024-08-19T14:16:40.232656+0000","last_active":"2024-08-19T14:16:40.232725+0000","last_peered":"2024-08-19T14:16:40.232725+0000","last_clean":"2024-08-19T14:16:35.204390+0000","last_became_active":"2024-08-19T14:16:40.232656+0000","last_became_peered":"2024-08-19T14:16:40.232656+0000","last_unstale":"2024-08-19T14:16:40.232725+0000","last_undegraded":"2024-08-19T14:16:40.232725+0000","last_fullsized":"2024-08-19T14:16:40.230602+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:16:30.232687+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:16:30.232687+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,4],"acting":[11,0],"avail_no_missing":["11","0","2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.6","version":"216'16250","reported_seq":16667,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-08-19T14:16:43.323942+0000","last_change":"2024-08-19T14:15:57.834926+0000","last_active":"2024-08-19T14:16:43.323942+0000","last_peered":"2024-08-19T14:16:43.323942+0000","last_clean":"2024-08-19T14:16:43.323942+0000","last_became_active":"2024-08-19T14:12:42.963203+0000","last_became_peered":"2024-08-19T14:12:42.963203+0000","last_unstale":"2024-08-19T14:16:43.323942+0000","last_undegraded":"2024-08-19T14:16:43.323942+0000","last_fullsized":"2024-08-19T14:16:43.323942+0000","mapping_epoch":102,"log_start":"141'6200","ondisk_log_start":"141'6200","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'13968","last_scrub_stamp":"2024-08-19T14:15:57.834750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:15:57.834750+0000","objects_scrubbed":2585,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:17:23.845680+0000","scrub_duration":281,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19857408,"num_objects":303,"num_object_clones":0,"num_object_copies":909,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":303,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15029,"num_write_kb":490624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,3],"acting":[1,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"70'139","reported_seq":550,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-08-19T14:16:40.230680+0000","last_change":"2024-08-19T14:16:29.257033+0000","last_active":"2024-08-19T14:16:40.230680+0000","last_peered":"2024-08-19T14:16:40.230680+0000","last_clean":"2024-08-19T14:16:40.230680+0000","last_became_active":"2024-08-19T14:13:05.607447+0000","last_became_peered":"2024-08-19T14:13:05.607447+0000","last_unstale":"2024-08-19T14:16:40.230680+0000","last_undegraded":"2024-08-19T14:16:40.230680+0000","last_fullsized":"2024-08-19T14:16:40.230680+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:16:29.256959+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:16:29.256959+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:17:40.848859+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.7","version":"210'15066","reported_seq":26702,"reported_epoch":210,"state":"peering","last_fresh":"2024-08-19T14:16:38.226634+0000","last_change":"2024-08-19T14:16:38.227770+0000","last_active":"2024-08-19T14:16:38.227770+0000","last_peered":"2024-08-19T14:16:38.226634+0000","last_clean":"2024-08-19T14:16:38.226634+0000","last_became_active":"2024-08-19T14:16:30.145835+0000","last_became_peered":"2024-08-19T14:16:30.145835+0000","last_unstale":"2024-08-19T14:16:38.226634+0000","last_undegraded":"2024-08-19T14:16:38.226634+0000","last_fullsized":"2024-08-19T14:16:38.226634+0000","mapping_epoch":211,"log_start":"135'5000","ondisk_log_start":"135'5000","created":102,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'14669","last_scrub_stamp":"2024-08-19T14:16:31.927994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:16:31.927994+0000","objects_scrubbed":756,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":325,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23068672,"num_objects":352,"num_object_clones":0,"num_object_copies":1056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15066,"num_write_kb":493376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4460,"num_bytes_recovered":253820928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,4,5],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.8","version":"215'15216","reported_seq":15602,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-08-19T14:16:42.249626+0000","last_change":"2024-08-19T14:16:37.131998+0000","last_active":"2024-08-19T14:16:42.249626+0000","last_peered":"2024-08-19T14:16:42.249626+0000","last_clean":"2024-08-19T14:16:42.249626+0000","last_became_active":"2024-08-19T14:12:43.038882+0000","last_became_peered":"2024-08-19T14:12:43.038882+0000","last_unstale":"2024-08-19T14:16:42.249626+0000","last_undegraded":"2024-08-19T14:16:42.249626+0000","last_fullsized":"2024-08-19T14:16:42.249626+0000","mapping_epoch":102,"log_start":"135'5200","ondisk_log_start":"135'5200","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"210'15097","last_scrub_stamp":"2024-08-19T14:16:37.131782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:16:37.131782+0000","objects_scrubbed":412,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:17:58.673898+0000","scrub_duration":110,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19136512,"num_objects":292,"num_object_clones":0,"num_object_copies":876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15216,"num_write_kb":496256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,0],"acting":[7,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"70'139","reported_seq":518,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-08-19T14:16:43.252645+0000","last_change":"2024-08-19T14:16:02.818848+0000","last_active":"2024-08-19T14:16:43.252645+0000","last_peered":"2024-08-19T14:16:43.252645+0000","last_clean":"2024-08-19T14:16:43.252645+0000","last_became_active":"2024-08-19T14:13:22.728779+0000","last_became_peered":"2024-08-19T14:13:22.728779+0000","last_unstale":"2024-08-19T14:16:43.252645+0000","last_undegraded":"2024-08-19T14:16:43.252645+0000","last_fullsized":"2024-08-19T14:16:43.252645+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:16:02.818774+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:16:02.818774+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:17:09.255652+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,0],"acting":[9,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"216'14911","reported_seq":15305,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-08-19T14:16:43.251547+0000","last_change":"2024-08-19T14:15:33.182944+0000","last_active":"2024-08-19T14:16:43.251547+0000","last_peered":"2024-08-19T14:16:43.251547+0000","last_clean":"2024-08-19T14:16:43.251547+0000","last_became_active":"2024-08-19T14:12:42.963114+0000","last_became_peered":"2024-08-19T14:12:42.963114+0000","last_unstale":"2024-08-19T14:16:43.251547+0000","last_undegraded":"2024-08-19T14:16:43.251547+0000","last_fullsized":"2024-08-19T14:16:43.251547+0000","mapping_epoch":102,"log_start":"135'4900","ondisk_log_start":"135'4900","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"179'11533","last_scrub_stamp":"2024-08-19T14:15:33.182797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:15:33.182797+0000","objects_scrubbed":3720,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:17:01.807642+0000","scrub_duration":1671,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19333120,"num_objects":295,"num_object_clones":0,"num_object_copies":885,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14911,"num_write_kb":486592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"70'139","reported_seq":638,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-08-19T14:16:42.252897+0000","last_change":"2024-08-19T14:16:42.252897+0000","last_active":"2024-08-19T14:16:42.252897+0000","last_peered":"2024-08-19T14:16:42.252897+0000","last_clean":"2024-08-19T14:16:42.252897+0000","last_became_active":"2024-08-19T14:16:42.252391+0000","last_became_peered":"2024-08-19T14:16:42.252391+0000","last_unstale":"2024-08-19T14:16:42.252897+0000","last_undegraded":"2024-08-19T14:16:42.252897+0000","last_fullsized":"2024-08-19T14:16:42.252897+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:16:03.918046+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:17:27.707808+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"213'15092","reported_seq":15463,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-08-19T14:16:40.588143+0000","last_change":"2024-08-19T14:16:37.437572+0000","last_active":"2024-08-19T14:16:40.588143+0000","last_peered":"2024-08-19T14:16:40.588143+0000","last_clean":"2024-08-19T14:16:40.588143+0000","last_became_active":"2024-08-19T14:12:42.963040+0000","last_became_peered":"2024-08-19T14:12:42.963040+0000","last_unstale":"2024-08-19T14:16:40.588143+0000","last_undegraded":"2024-08-19T14:16:40.588143+0000","last_fullsized":"2024-08-19T14:16:40.588143+0000","mapping_epoch":102,"log_start":"135'5000","ondisk_log_start":"135'5000","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"210'15050","last_scrub_stamp":"2024-08-19T14:16:37.437401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:16:37.437401+0000","objects_scrubbed":388,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:17:47.761775+0000","scrub_duration":127,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22675456,"num_objects":346,"num_object_clones":0,"num_object_copies":1038,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15092,"num_write_kb":494016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,9],"acting":[11,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.8","version":"70'139","reported_seq":511,"reported_epoch":213,"state":"peering","last_fresh":"2024-08-19T14:16:40.274467+0000","last_change":"2024-08-19T14:16:41.243246+0000","last_active":"2024-08-19T14:16:41.243246+0000","last_peered":"2024-08-19T14:16:40.274467+0000","last_clean":"2024-08-19T14:16:35.203353+0000","last_became_active":"2024-08-19T14:16:40.259494+0000","last_became_peered":"2024-08-19T14:16:40.259494+0000","last_unstale":"2024-08-19T14:16:40.274467+0000","last_undegraded":"2024-08-19T14:16:40.274467+0000","last_fullsized":"2024-08-19T14:16:40.227190+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:15:44.541487+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:15:44.541487+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,8],"acting":[5,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3,8],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.b","version":"211'15026","reported_seq":15365,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-08-19T14:16:39.216099+0000","last_change":"2024-08-19T14:16:04.247292+0000","last_active":"2024-08-19T14:16:39.216099+0000","last_peered":"2024-08-19T14:16:39.216099+0000","last_clean":"2024-08-19T14:16:39.216099+0000","last_became_active":"2024-08-19T14:12:42.729014+0000","last_became_peered":"2024-08-19T14:12:42.729014+0000","last_unstale":"2024-08-19T14:16:39.216099+0000","last_undegraded":"2024-08-19T14:16:39.216099+0000","last_fullsized":"2024-08-19T14:16:39.216099+0000","mapping_epoch":102,"log_start":"135'5000","ondisk_log_start":"135'5000","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"196'13194","last_scrub_stamp":"2024-08-19T14:16:04.247138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:12:41.713780+0000","last_clean_scrub_stamp":"2024-08-19T14:16:04.247138+0000","objects_scrubbed":2194,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:17:05.520387+0000","scrub_duration":449,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23199744,"num_objects":354,"num_object_clones":0,"num_object_copies":1062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":354,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15026,"num_write_kb":492160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":330694680,"num_objects":5047,"num_object_clones":0,"num_object_copies":15141,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":304,"num_objects_unfound":0,"num_objects_dirty":5047,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":241511,"num_write_kb":7889793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25080,"num_bytes_recovered":1605107712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1359228928,"data_stored":1359216712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160832,"ondisk_log_size":160832,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1529,"ondisk_log_size":1529,"up":33,"acting":32,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187520,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":216448,"kb_used_data":113068,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93528704,"statfs":{"total":95995035648,"available":95773392896,"internally_reserved":0,"allocated":115781632,"data_stored":114747346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":105839044},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":343597383749,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":74,"seq":317827579978,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":355092,"kb_used_data":189256,"kb_used_omap":1,"kb_used_meta":165822,"kb_avail":93390060,"statfs":{"total":95995035648,"available":95631421440,"internally_reserved":0,"allocated":193798144,"data_stored":192751735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":169802177},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874319,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":222448,"kb_used_data":114912,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93522704,"statfs":{"total":95995035648,"available":95767248896,"internally_reserved":0,"allocated":117669888,"data_stored":116639462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":57,"seq":244813135954,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":259732,"kb_used_data":116100,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":93485420,"statfs":{"total":95995035648,"available":95729070080,"internally_reserved":0,"allocated":118886400,"data_stored":117844063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":147061194},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":206158430293,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691930,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":150988,"kb_used_data":75392,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93594164,"statfs":{"total":95995035648,"available":95840423936,"internally_reserved":0,"allocated":77201408,"data_stored":76159634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986271,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":179524,"kb_used_data":80316,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":93565628,"statfs":{"total":95995035648,"available":95811203072,"internally_reserved":0,"allocated":82243584,"data_stored":81201290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149796,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":280044,"kb_used_data":114208,"kb_used_omap":1,"kb_used_meta":165822,"kb_avail":93465108,"statfs":{"total":95995035648,"available":95708270592,"internally_reserved":0,"allocated":116948992,"data_stored":115914463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":169802183},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411433,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574958,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":243280,"kb_used_data":113540,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":93501872,"statfs":{"total":95995035648,"available":95745916928,"internally_reserved":0,"allocated":116264960,"data_stored":115222623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":132839877},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738482,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":215152,"kb_used_data":111776,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93530000,"statfs":{"total":95995035648,"available":95774720000,"internally_reserved":0,"allocated":114458624,"data_stored":113428198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":107413504,"data_stored":107413504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":112852992,"data_stored":112852992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":100532224,"data_stored":100532224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":114298880,"data_stored":114294808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":79237120,"data_stored":79233048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":75694080,"data_stored":75694080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":39387136,"data_stored":39387136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":116064256,"data_stored":116064256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":114884608,"data_stored":114884608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":191107072,"data_stored":191103000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":198115328,"data_stored":198115328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":109641728,"data_stored":109641728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T14:16:46.603 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-08-19T14:16:46.603 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd pool set unique_pool_0 pg_num 26 2024-08-19T14:16:46.625 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:16:46.661 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:46.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:46.663 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:16:46.763 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:16:46.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:46.801 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:46.802 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:16:46.883 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:16:46.883 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:16:46.902 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:16:46.943 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:46.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:46.944 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:16:47.045 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:16:47.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:47.080 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:47.080 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:16:47.181 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:16:47.215 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:47.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:47.216 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:16:47.317 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:16:47.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:47.344 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:47.344 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:16:47.445 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:16:47.476 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:47.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:47.477 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:16:47.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:47 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:16:47.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:47 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:16:47.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:47 smithi045 ceph-mon[20143]: pgmap v522: 27 pgs: 1 active+clean+remapped, 2 peering, 24 active+clean; 248 MiB data, 1.9 GiB used, 803 GiB / 805 GiB avail; 0 B/s wr, 402 op/s; 18 MiB/s, 284 objects/s recovering 2024-08-19T14:16:47.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:47 smithi045 ceph-mon[20143]: osdmap e217: 12 total, 12 up, 9 in 2024-08-19T14:16:47.578 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:16:47.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:47 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:16:47.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:47 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:16:47.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:47 smithi031 ceph-mon[16336]: pgmap v522: 27 pgs: 1 active+clean+remapped, 2 peering, 24 active+clean; 248 MiB data, 1.9 GiB used, 803 GiB / 805 GiB avail; 0 B/s wr, 402 op/s; 18 MiB/s, 284 objects/s recovering 2024-08-19T14:16:47.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:47 smithi031 ceph-mon[16336]: osdmap e217: 12 total, 12 up, 9 in 2024-08-19T14:16:47.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:47.622 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:47.623 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:16:47.715 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:47 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:16:47.716 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:47 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:16:47.716 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:47 smithi032 ceph-mon[20004]: pgmap v522: 27 pgs: 1 active+clean+remapped, 2 peering, 24 active+clean; 248 MiB data, 1.9 GiB used, 803 GiB / 805 GiB avail; 0 B/s wr, 402 op/s; 18 MiB/s, 284 objects/s recovering 2024-08-19T14:16:47.716 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:47 smithi032 ceph-mon[20004]: osdmap e217: 12 total, 12 up, 9 in 2024-08-19T14:16:47.716 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Removed 123278 objects 2024-08-19T14:16:47.717 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Clean up completed and total clean up time :146.874 2024-08-19T14:16:47.724 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:16:47.743 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-08-19T14:16:47.745 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:47.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:47.746 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:16:47.847 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:16:47.867 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:47.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:47.868 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:16:47.969 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:16:47.991 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:47.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:47.992 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:16:47.994 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:16:48.093 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:16:48.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:48.116 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:48.116 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:16:48.174 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:16:48.217 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:16:48.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:48.242 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:48.242 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:16:48.343 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:16:48.365 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:48.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:48.366 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:16:48.460 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:16:48.460 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:16:48.467 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:16:48.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:48.489 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:48.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:16:48.590 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:16:48.617 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:48.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:48.618 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:16:48.719 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:16:48.741 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:48.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:48.742 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:16:48.843 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:16:48.864 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:48.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:48.865 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:16:48.967 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:16:48.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:48.989 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:48.989 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:16:49.091 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:16:49.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:49.113 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:49.113 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:16:49.176 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-08-19T14:16:49.214 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:16:49.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:49.244 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:49.244 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:16:49.282 INFO:teuthology.orchestra.run.smithi031.stderr:set pool 2 pg_num to 26 2024-08-19T14:16:49.283 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is set 2024-08-19T14:16:49.345 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:16:49.368 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:49.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:49.369 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:16:49.429 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:16:49.469 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:16:49.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:49.492 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:49.492 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:16:49.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:49 smithi032 ceph-mon[20004]: pgmap v524: 27 pgs: 27 active+clean; 221 MiB data, 1.8 GiB used, 803 GiB / 805 GiB avail; 0 B/s wr, 319 op/s; 13 MiB/s, 204 objects/s recovering 2024-08-19T14:16:49.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:49 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1237350617' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:16:49.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:49 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2374775219' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]: dispatch 2024-08-19T14:16:49.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:49 smithi045 ceph-mon[20143]: pgmap v524: 27 pgs: 27 active+clean; 221 MiB data, 1.8 GiB used, 803 GiB / 805 GiB avail; 0 B/s wr, 319 op/s; 13 MiB/s, 204 objects/s recovering 2024-08-19T14:16:49.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:49 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1237350617' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:16:49.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:49 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2374775219' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]: dispatch 2024-08-19T14:16:49.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:49 smithi031 ceph-mon[16336]: pgmap v524: 27 pgs: 27 active+clean; 221 MiB data, 1.8 GiB used, 803 GiB / 805 GiB avail; 0 B/s wr, 319 op/s; 13 MiB/s, 204 objects/s recovering 2024-08-19T14:16:49.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:49 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1237350617' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:16:49.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:49 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2374775219' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]: dispatch 2024-08-19T14:16:49.593 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:16:49.619 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:49.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:49.620 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:16:49.690 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:16:49.721 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:16:49.746 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:49.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:49.748 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:16:49.848 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:16:49.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:49.875 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:49.875 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:16:49.888 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_0 2024-08-19T14:16:49.984 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-08-19T14:16:49.985 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:16:49.986 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:16:49.986 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:16:50.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:50.010 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:50.010 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:16:50.111 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:16:50.133 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:50.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:50.134 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:16:50.235 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:16:50.262 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:50.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:50.263 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:16:50.364 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:16:50.389 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:50.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:50.390 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:16:50.479 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:50 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:16:50.479 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:50 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:16:50.480 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:50 smithi031 ceph-mon[16336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:16:50.480 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:50 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:16:50.480 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:50 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:16:50.480 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:50 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1237350617' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:16:50.480 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:50 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2374775219' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]': finished 2024-08-19T14:16:50.480 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:50 smithi031 ceph-mon[16336]: osdmap e218: 12 total, 12 up, 9 in 2024-08-19T14:16:50.480 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:50 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-08-19T14:16:50.480 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:50 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-08-19T14:16:50.492 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:16:50.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:50.531 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:50.532 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:16:50.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:50 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:16:50.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:50 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:16:50.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:50 smithi045 ceph-mon[20143]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:16:50.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:50 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:16:50.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:50 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:16:50.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:50 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1237350617' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:16:50.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:50 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2374775219' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]': finished 2024-08-19T14:16:50.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:50 smithi045 ceph-mon[20143]: osdmap e218: 12 total, 12 up, 9 in 2024-08-19T14:16:50.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:50 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-08-19T14:16:50.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:50 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-08-19T14:16:50.577 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:16:50.632 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:16:50.651 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:16:50.691 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:50.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:50.692 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:16:50.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:50 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:16:50.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:50 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:16:50.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:50 smithi032 ceph-mon[20004]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:16:50.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:50 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:16:50.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:50 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:16:50.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:50 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1237350617' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:16:50.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:50 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2374775219' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]': finished 2024-08-19T14:16:50.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:50 smithi032 ceph-mon[20004]: osdmap e218: 12 total, 12 up, 9 in 2024-08-19T14:16:50.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:50 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-08-19T14:16:50.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:50 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-08-19T14:16:50.793 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:16:50.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:50.823 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:50.824 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:16:50.925 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:16:50.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:50.947 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:50.947 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:16:51.048 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:16:51.070 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:51.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:51.071 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:16:51.173 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:16:51.195 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:51.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:51.196 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:16:51.202 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:16:51.203 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:16:51.219 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:16:51.219 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:16:51.237 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:16:51.237 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:16:51.253 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:16:51.254 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:16:51.270 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:16:51.270 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:16:51.287 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:16:51.287 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:16:51.297 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:16:51.304 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:16:51.304 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:16:51.318 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:51.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:51.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:16:51.320 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:16:51.321 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:16:51.339 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:16:51.339 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:16:51.356 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:16:51.356 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:16:51.397 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:16:51.397 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:16:51.407 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:51 smithi032 ceph-mon[20004]: pgmap v526: 27 pgs: 27 active+clean; 77 MiB data, 1.3 GiB used, 803 GiB / 805 GiB avail; 0 B/s wr, 614 op/s; 8.5 MiB/s, 136 objects/s recovering 2024-08-19T14:16:51.407 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:51 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]': finished 2024-08-19T14:16:51.407 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:51 smithi032 ceph-mon[20004]: osdmap e219: 12 total, 12 up, 9 in 2024-08-19T14:16:51.407 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:51 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1700423759' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:16:51.419 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:16:51.436 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:16:51.437 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:16:51.441 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:51.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:51.442 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:16:51.445 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-08-19T14:16:51.543 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:16:51.564 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:51.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:51.565 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:16:51.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:51 smithi031 ceph-mon[16336]: pgmap v526: 27 pgs: 27 active+clean; 77 MiB data, 1.3 GiB used, 803 GiB / 805 GiB avail; 0 B/s wr, 614 op/s; 8.5 MiB/s, 136 objects/s recovering 2024-08-19T14:16:51.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:51 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]': finished 2024-08-19T14:16:51.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:51 smithi031 ceph-mon[16336]: osdmap e219: 12 total, 12 up, 9 in 2024-08-19T14:16:51.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:51 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1700423759' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:16:51.666 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:16:51.689 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:51.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:51.690 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:16:51.698 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:16:51.790 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:16:51.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:51 smithi045 ceph-mon[20143]: pgmap v526: 27 pgs: 27 active+clean; 77 MiB data, 1.3 GiB used, 803 GiB / 805 GiB avail; 0 B/s wr, 614 op/s; 8.5 MiB/s, 136 objects/s recovering 2024-08-19T14:16:51.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:51 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]': finished 2024-08-19T14:16:51.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:51 smithi045 ceph-mon[20143]: osdmap e219: 12 total, 12 up, 9 in 2024-08-19T14:16:51.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:51 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1700423759' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:16:51.811 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:51.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:51.812 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:16:51.913 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:16:51.936 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:51.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:51.937 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:16:52.038 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:16:52.063 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:52.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:52.064 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:16:52.165 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:16:52.176 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:16:52.176 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-08-19T14:16:52.176 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-08-19T14:16:52.176 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-08-19T14:16:52.176 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:16:52.191 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:52.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:52.192 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:16:52.293 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:16:52.319 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:52.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:52.320 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:16:52.368 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'unique_pool_0' removed 2024-08-19T14:16:52.421 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:16:52.443 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:52.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:52.446 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:16:52.505 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:52 smithi032 ceph-mon[20004]: osdmap e220: 12 total, 12 up, 9 in 2024-08-19T14:16:52.505 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:52 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3281440602' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-08-19T14:16:52.505 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:52 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-08-19T14:16:52.505 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:52 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-08-19T14:16:52.547 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:16:52.549 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-19T14:16:52.571 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:52.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:52.572 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:16:52.674 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:16:52.694 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:52.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:52.695 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:16:52.774 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-08-19T14:16:52.795 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:16:52.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:52 smithi045 ceph-mon[20143]: osdmap e220: 12 total, 12 up, 9 in 2024-08-19T14:16:52.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:52 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3281440602' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-08-19T14:16:52.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:52 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-08-19T14:16:52.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:52 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-08-19T14:16:52.837 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:52.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:52.838 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:16:52.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:52 smithi031 ceph-mon[16336]: osdmap e220: 12 total, 12 up, 9 in 2024-08-19T14:16:52.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:52 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3281440602' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-08-19T14:16:52.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:52 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-08-19T14:16:52.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:52 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-08-19T14:16:52.939 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:16:52.967 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:52.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:52.968 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:16:53.024 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:16:53.069 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:16:53.090 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:53.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:53.091 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:16:53.192 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:16:53.214 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:53.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:53.215 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:16:53.227 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:16:53.315 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:16:53.336 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:53.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:53.337 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:16:53.438 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:16:53.459 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:53 smithi032 ceph-mon[20004]: pgmap v529: 37 pgs: 1 peering, 8 unknown, 28 active+clean; 577 KiB data, 1.2 GiB used, 803 GiB / 805 GiB avail; 0 B/s wr, 717 op/s 2024-08-19T14:16:53.459 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:53 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3281440602' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-08-19T14:16:53.459 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:53 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]': finished 2024-08-19T14:16:53.459 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:53 smithi032 ceph-mon[20004]: osdmap e221: 12 total, 12 up, 9 in 2024-08-19T14:16:53.460 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:53.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:53.461 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:16:53.562 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:16:53.583 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:53.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:53.584 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:16:53.585 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:16:53.585 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-08-19T14:16:53.585 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-08-19T14:16:53.585 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-08-19T14:16:53.585 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.585 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:8241 2.7 2:edf47d3e:::benchmark_data_smithi045_46415_object8240:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-08-19T14:16:53.585 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:12:51.684165+0000", 2024-08-19T14:16:53.585 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 241.875506487, 2024-08-19T14:16:53.586 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.204747761, 2024-08-19T14:16:53.586 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:16:53.586 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:16:53.586 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:16:53.586 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:16:53.586 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:16:53.586 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:16:53.586 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 8241 2024-08-19T14:16:53.586 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.586 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:16:53.586 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.586 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:16:53.586 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.684165+0000", 2024-08-19T14:16:53.587 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:53.587 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.587 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.587 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:16:53.587 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.684165+0000", 2024-08-19T14:16:53.587 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:53.587 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.587 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.587 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:16:53.587 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.684167+0000", 2024-08-19T14:16:53.587 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.313e-06 2024-08-19T14:16:53.587 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.587 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.587 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:16:53.588 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.684210+0000", 2024-08-19T14:16:53.588 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3049000000000001e-05 2024-08-19T14:16:53.588 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.588 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.588 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:16:53.588 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.684223+0000", 2024-08-19T14:16:53.588 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3019e-05 2024-08-19T14:16:53.588 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.588 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.588 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:16:53.588 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.684230+0000", 2024-08-19T14:16:53.588 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1969999999999999e-06 2024-08-19T14:16:53.588 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.588 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.589 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:16:53.589 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.684270+0000", 2024-08-19T14:16:53.589 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0649999999999999e-05 2024-08-19T14:16:53.589 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.589 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.589 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:16:53.589 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.684340+0000", 2024-08-19T14:16:53.589 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9979000000000002e-05 2024-08-19T14:16:53.589 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.589 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.589 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,10", 2024-08-19T14:16:53.589 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.684464+0000", 2024-08-19T14:16:53.589 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012321599999999999 2024-08-19T14:16:53.589 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.590 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.590 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:16:53.590 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.684888+0000", 2024-08-19T14:16:53.590 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00042467399999999999 2024-08-19T14:16:53.590 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.590 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.590 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:53.590 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.685794+0000", 2024-08-19T14:16:53.590 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00090609299999999996 2024-08-19T14:16:53.591 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.591 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.591 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:53.591 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.888736+0000", 2024-08-19T14:16:53.591 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20294189100000001 2024-08-19T14:16:53.591 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.591 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.591 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:16:53.591 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.888831+0000", 2024-08-19T14:16:53.591 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.5172000000000003e-05 2024-08-19T14:16:53.592 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.592 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.592 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:16:53.592 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.888913+0000", 2024-08-19T14:16:53.592 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1508000000000006e-05 2024-08-19T14:16:53.592 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:53.592 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:16:53.592 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:53.592 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.592 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.592 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:8259 2.7 2:e2e1720b:::benchmark_data_smithi045_46415_object8258:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-08-19T14:16:53.593 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:12:51.689217+0000", 2024-08-19T14:16:53.593 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 241.870454618, 2024-08-19T14:16:53.593 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.19987796899999999, 2024-08-19T14:16:53.593 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:16:53.593 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:16:53.593 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:16:53.593 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:16:53.593 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:16:53.593 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:16:53.593 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 8259 2024-08-19T14:16:53.593 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.593 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:16:53.593 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.593 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:16:53.594 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.689217+0000", 2024-08-19T14:16:53.594 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:53.594 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.594 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.594 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:16:53.594 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.689217+0000", 2024-08-19T14:16:53.594 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:53.594 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.594 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.594 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:16:53.594 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.689218+0000", 2024-08-19T14:16:53.594 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0240000000000001e-06 2024-08-19T14:16:53.594 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.595 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.595 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:16:53.595 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.689272+0000", 2024-08-19T14:16:53.595 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3974999999999997e-05 2024-08-19T14:16:53.595 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.595 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.595 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:16:53.595 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.689283+0000", 2024-08-19T14:16:53.595 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0647e-05 2024-08-19T14:16:53.595 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.595 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.595 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:16:53.595 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.689288+0000", 2024-08-19T14:16:53.595 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2390000000000002e-06 2024-08-19T14:16:53.596 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.596 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.596 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:16:53.596 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.689341+0000", 2024-08-19T14:16:53.596 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.342e-05 2024-08-19T14:16:53.596 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.596 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.596 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:16:53.596 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.689464+0000", 2024-08-19T14:16:53.596 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000122991 2024-08-19T14:16:53.596 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.596 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.596 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,10", 2024-08-19T14:16:53.596 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.689610+0000", 2024-08-19T14:16:53.597 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014557200000000001 2024-08-19T14:16:53.597 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.597 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.597 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:16:53.597 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.690297+0000", 2024-08-19T14:16:53.597 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00068681199999999997 2024-08-19T14:16:53.597 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.597 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.597 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:53.597 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.690649+0000", 2024-08-19T14:16:53.597 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00035261899999999998 2024-08-19T14:16:53.597 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.597 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.597 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:53.598 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.888991+0000", 2024-08-19T14:16:53.598 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.19834179800000001 2024-08-19T14:16:53.598 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.598 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.598 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:16:53.598 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.889045+0000", 2024-08-19T14:16:53.598 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3643000000000001e-05 2024-08-19T14:16:53.598 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.598 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.598 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:16:53.598 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.889095+0000", 2024-08-19T14:16:53.598 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0229000000000003e-05 2024-08-19T14:16:53.598 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:53.598 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:16:53.599 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:53.599 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.599 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.599 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:8272 2.7 2:ec6eef80:::benchmark_data_smithi045_46415_object8271:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-08-19T14:16:53.599 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:12:51.692591+0000", 2024-08-19T14:16:53.599 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 241.86708079499999, 2024-08-19T14:16:53.599 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.19659718500000001, 2024-08-19T14:16:53.599 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:16:53.599 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:16:53.599 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:16:53.599 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:16:53.599 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:16:53.599 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:16:53.600 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 8272 2024-08-19T14:16:53.600 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.600 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:16:53.600 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.600 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:16:53.600 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.692591+0000", 2024-08-19T14:16:53.600 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:53.600 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.600 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.600 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:16:53.600 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.692591+0000", 2024-08-19T14:16:53.600 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:53.600 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.600 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.601 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:16:53.601 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.692592+0000", 2024-08-19T14:16:53.601 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.2800000000000005e-07 2024-08-19T14:16:53.601 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.601 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.601 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:16:53.601 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.692661+0000", 2024-08-19T14:16:53.601 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8744000000000005e-05 2024-08-19T14:16:53.601 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.601 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.601 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:16:53.601 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.692675+0000", 2024-08-19T14:16:53.601 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3916e-05 2024-08-19T14:16:53.601 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.602 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.602 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:16:53.602 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.692682+0000", 2024-08-19T14:16:53.602 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5320000000000001e-06 2024-08-19T14:16:53.602 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.602 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.602 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:16:53.602 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.692731+0000", 2024-08-19T14:16:53.603 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8909999999999998e-05 2024-08-19T14:16:53.603 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.603 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.603 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:16:53.603 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.692805+0000", 2024-08-19T14:16:53.603 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4128000000000006e-05 2024-08-19T14:16:53.603 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.603 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.603 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,10", 2024-08-19T14:16:53.603 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.692885+0000", 2024-08-19T14:16:53.603 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.9655999999999998e-05 2024-08-19T14:16:53.603 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.603 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.603 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:16:53.604 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.693416+0000", 2024-08-19T14:16:53.604 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00053101099999999996 2024-08-19T14:16:53.604 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.604 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.604 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:53.604 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.694542+0000", 2024-08-19T14:16:53.604 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0011263810000000001 2024-08-19T14:16:53.604 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.604 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.604 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:53.604 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.889136+0000", 2024-08-19T14:16:53.604 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.194593922 2024-08-19T14:16:53.604 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.604 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.605 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:16:53.605 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.889164+0000", 2024-08-19T14:16:53.605 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8113999999999999e-05 2024-08-19T14:16:53.605 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.605 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.605 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:16:53.605 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.889188+0000", 2024-08-19T14:16:53.605 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3943000000000001e-05 2024-08-19T14:16:53.605 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:53.605 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:16:53.605 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:53.605 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.605 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.605 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:8281 2.7 2:edaf96ca:::benchmark_data_smithi045_46415_object8280:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-08-19T14:16:53.606 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:12:51.695230+0000", 2024-08-19T14:16:53.606 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 241.864442145, 2024-08-19T14:16:53.606 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.194082428, 2024-08-19T14:16:53.606 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:16:53.606 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:16:53.606 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:16:53.606 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:16:53.606 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:16:53.606 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:16:53.606 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 8281 2024-08-19T14:16:53.606 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.606 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:16:53.606 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.606 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:16:53.607 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.695230+0000", 2024-08-19T14:16:53.607 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:53.607 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.607 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.607 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:16:53.607 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.695230+0000", 2024-08-19T14:16:53.607 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:53.607 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.607 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.607 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:16:53.607 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.695231+0000", 2024-08-19T14:16:53.607 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0249999999999999e-06 2024-08-19T14:16:53.607 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.608 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.608 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:16:53.608 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.695295+0000", 2024-08-19T14:16:53.608 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4115999999999994e-05 2024-08-19T14:16:53.608 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.608 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.608 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:16:53.609 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.695309+0000", 2024-08-19T14:16:53.609 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.383e-05 2024-08-19T14:16:53.609 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.609 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.609 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:16:53.609 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.695317+0000", 2024-08-19T14:16:53.609 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.0490000000000004e-06 2024-08-19T14:16:53.609 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.609 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.610 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:16:53.610 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.695368+0000", 2024-08-19T14:16:53.610 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1819e-05 2024-08-19T14:16:53.610 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.610 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.610 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:16:53.610 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.695445+0000", 2024-08-19T14:16:53.610 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6705999999999994e-05 2024-08-19T14:16:53.610 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.610 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.611 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,10", 2024-08-19T14:16:53.611 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.695523+0000", 2024-08-19T14:16:53.611 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7989000000000002e-05 2024-08-19T14:16:53.611 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.611 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.611 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:16:53.611 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.696398+0000", 2024-08-19T14:16:53.611 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00087526300000000004 2024-08-19T14:16:53.611 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.611 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.611 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:53.611 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.696820+0000", 2024-08-19T14:16:53.611 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00042155799999999999 2024-08-19T14:16:53.612 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.612 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.612 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:53.612 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.889226+0000", 2024-08-19T14:16:53.612 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.19240576600000001 2024-08-19T14:16:53.612 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.612 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.612 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:16:53.612 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.889276+0000", 2024-08-19T14:16:53.613 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0389000000000002e-05 2024-08-19T14:16:53.613 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.613 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.613 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:16:53.613 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.889312+0000", 2024-08-19T14:16:53.613 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5917999999999997e-05 2024-08-19T14:16:53.613 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:53.613 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:16:53.613 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:53.613 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.613 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.613 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:8284 2.7 2:eea07cd5:::benchmark_data_smithi045_46415_object8283:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-08-19T14:16:53.613 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:12:51.695765+0000", 2024-08-19T14:16:53.613 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 241.86390662400001, 2024-08-19T14:16:53.614 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.19395319999999999, 2024-08-19T14:16:53.614 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:16:53.614 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:16:53.614 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:16:53.614 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:16:53.614 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:16:53.614 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:16:53.614 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 8284 2024-08-19T14:16:53.614 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.614 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:16:53.614 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.614 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:16:53.614 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.695765+0000", 2024-08-19T14:16:53.614 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:53.615 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.615 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.615 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:16:53.615 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.695765+0000", 2024-08-19T14:16:53.615 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:53.615 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.615 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.615 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:16:53.615 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.695766+0000", 2024-08-19T14:16:53.615 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2399999999999997e-07 2024-08-19T14:16:53.615 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.615 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.615 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:16:53.615 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.695825+0000", 2024-08-19T14:16:53.616 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8702000000000001e-05 2024-08-19T14:16:53.616 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.616 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.616 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:16:53.616 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.695845+0000", 2024-08-19T14:16:53.616 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0500999999999999e-05 2024-08-19T14:16:53.616 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.616 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.616 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:16:53.616 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.695850+0000", 2024-08-19T14:16:53.616 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3319999999999999e-06 2024-08-19T14:16:53.616 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.616 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.616 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:16:53.617 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.695932+0000", 2024-08-19T14:16:53.617 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.2089000000000006e-05 2024-08-19T14:16:53.617 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.617 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.617 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:16:53.617 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.696041+0000", 2024-08-19T14:16:53.617 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000108931 2024-08-19T14:16:53.617 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.617 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.617 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,10", 2024-08-19T14:16:53.617 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.696129+0000", 2024-08-19T14:16:53.617 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.7545000000000003e-05 2024-08-19T14:16:53.617 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.618 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.618 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:16:53.618 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.696793+0000", 2024-08-19T14:16:53.618 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000663946 2024-08-19T14:16:53.618 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.618 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.618 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:53.618 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.697250+0000", 2024-08-19T14:16:53.618 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000457541 2024-08-19T14:16:53.618 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.618 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.618 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:53.618 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.889620+0000", 2024-08-19T14:16:53.618 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.19236932600000001 2024-08-19T14:16:53.619 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.619 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.619 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:16:53.619 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.889680+0000", 2024-08-19T14:16:53.619 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0417000000000001e-05 2024-08-19T14:16:53.619 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.619 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.619 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:16:53.619 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.889718+0000", 2024-08-19T14:16:53.620 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8145999999999997e-05 2024-08-19T14:16:53.620 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:53.620 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:16:53.620 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:53.620 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.620 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.620 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:8285 2.7 2:e4063f99:::benchmark_data_smithi045_46415_object8284:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-08-19T14:16:53.620 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:12:51.695865+0000", 2024-08-19T14:16:53.620 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 241.86380675800001, 2024-08-19T14:16:53.620 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.19405726300000001, 2024-08-19T14:16:53.620 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:16:53.620 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:16:53.620 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:16:53.620 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:16:53.621 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:16:53.621 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:16:53.621 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 8285 2024-08-19T14:16:53.621 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.621 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:16:53.621 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.621 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:16:53.621 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.695865+0000", 2024-08-19T14:16:53.621 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:53.621 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.621 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.621 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:16:53.621 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.695865+0000", 2024-08-19T14:16:53.622 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:53.622 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.622 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.622 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:16:53.622 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.695866+0000", 2024-08-19T14:16:53.622 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8999999999999996e-07 2024-08-19T14:16:53.622 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.622 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.622 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:16:53.622 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.695901+0000", 2024-08-19T14:16:53.622 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5859999999999999e-05 2024-08-19T14:16:53.622 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.622 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.622 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:16:53.623 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.695913+0000", 2024-08-19T14:16:53.623 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1114000000000001e-05 2024-08-19T14:16:53.623 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.623 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.623 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:16:53.623 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.695920+0000", 2024-08-19T14:16:53.623 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1949999999999997e-06 2024-08-19T14:16:53.623 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.623 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.623 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:16:53.623 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.696419+0000", 2024-08-19T14:16:53.623 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00049909300000000004 2024-08-19T14:16:53.623 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.623 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.624 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:16:53.624 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.696489+0000", 2024-08-19T14:16:53.624 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9915999999999999e-05 2024-08-19T14:16:53.624 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.624 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.624 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,10", 2024-08-19T14:16:53.624 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.696578+0000", 2024-08-19T14:16:53.624 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.8825000000000004e-05 2024-08-19T14:16:53.624 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.624 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.624 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:16:53.624 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.696895+0000", 2024-08-19T14:16:53.624 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000317912 2024-08-19T14:16:53.624 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.625 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.625 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:53.625 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.697442+0000", 2024-08-19T14:16:53.625 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00054631000000000005 2024-08-19T14:16:53.625 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.625 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.625 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:53.625 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.889765+0000", 2024-08-19T14:16:53.625 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.192323409 2024-08-19T14:16:53.625 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.625 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.625 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:16:53.625 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.889882+0000", 2024-08-19T14:16:53.625 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00011684 2024-08-19T14:16:53.626 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.626 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.626 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:16:53.626 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.889922+0000", 2024-08-19T14:16:53.626 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0198999999999999e-05 2024-08-19T14:16:53.626 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:53.626 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:16:53.626 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:53.626 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.626 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.626 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:8318 2.7 2:e0a01dab:::benchmark_data_smithi045_46415_object8317:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-08-19T14:16:53.626 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:12:51.706539+0000", 2024-08-19T14:16:53.626 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 241.85313291700001, 2024-08-19T14:16:53.626 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.183777518, 2024-08-19T14:16:53.627 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:16:53.627 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:16:53.627 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:16:53.627 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:16:53.627 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:16:53.627 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:16:53.627 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 8318 2024-08-19T14:16:53.627 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.627 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:16:53.627 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.627 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:16:53.627 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.706539+0000", 2024-08-19T14:16:53.627 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:53.627 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.628 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.628 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:16:53.628 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.706539+0000", 2024-08-19T14:16:53.628 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:53.628 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.628 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.628 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:16:53.628 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.706541+0000", 2024-08-19T14:16:53.628 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8050000000000001e-06 2024-08-19T14:16:53.628 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.628 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.628 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:16:53.628 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.706582+0000", 2024-08-19T14:16:53.628 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1748000000000001e-05 2024-08-19T14:16:53.629 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.629 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.629 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:16:53.629 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.706605+0000", 2024-08-19T14:16:53.629 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2614000000000001e-05 2024-08-19T14:16:53.629 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.629 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.629 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:16:53.629 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.706615+0000", 2024-08-19T14:16:53.629 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.825e-06 2024-08-19T14:16:53.629 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.629 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.629 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:16:53.630 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.706681+0000", 2024-08-19T14:16:53.630 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.5729999999999996e-05 2024-08-19T14:16:53.630 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.630 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.630 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:16:53.630 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.706811+0000", 2024-08-19T14:16:53.630 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013086200000000001 2024-08-19T14:16:53.630 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.630 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.630 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,10", 2024-08-19T14:16:53.630 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.706948+0000", 2024-08-19T14:16:53.630 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013628800000000001 2024-08-19T14:16:53.630 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.630 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.631 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:16:53.631 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.707654+0000", 2024-08-19T14:16:53.631 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00070644600000000001 2024-08-19T14:16:53.631 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.631 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.631 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:53.631 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.707895+0000", 2024-08-19T14:16:53.631 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00024116599999999999 2024-08-19T14:16:53.631 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.631 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.631 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:53.631 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.890219+0000", 2024-08-19T14:16:53.631 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.18232368199999999 2024-08-19T14:16:53.631 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.632 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.632 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:16:53.632 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.890279+0000", 2024-08-19T14:16:53.632 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9941000000000001e-05 2024-08-19T14:16:53.632 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.632 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.632 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:16:53.632 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.890316+0000", 2024-08-19T14:16:53.632 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7410999999999998e-05 2024-08-19T14:16:53.632 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:53.632 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:16:53.632 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:53.632 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.632 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.633 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:48872 2.7 2:e2eb060c:::benchmark_data_smithi045_46415_object48871:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-08-19T14:16:53.633 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:07.265121+0000", 2024-08-19T14:16:53.633 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 226.29455071000001, 2024-08-19T14:16:53.633 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20642575599999999, 2024-08-19T14:16:53.633 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:16:53.633 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:16:53.633 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:16:53.633 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:16:53.633 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:16:53.633 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:16:53.633 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 48872 2024-08-19T14:16:53.633 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.633 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:16:53.633 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.634 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:16:53.634 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.265121+0000", 2024-08-19T14:16:53.634 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:53.634 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.634 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.634 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:16:53.634 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.265121+0000", 2024-08-19T14:16:53.634 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:53.634 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.634 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.634 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:16:53.634 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.265123+0000", 2024-08-19T14:16:53.634 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.494e-06 2024-08-19T14:16:53.634 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.635 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.635 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:16:53.635 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.265156+0000", 2024-08-19T14:16:53.635 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3651e-05 2024-08-19T14:16:53.635 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.635 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.635 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:16:53.635 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.265167+0000", 2024-08-19T14:16:53.635 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0669000000000001e-05 2024-08-19T14:16:53.635 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.635 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.635 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:16:53.635 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.265174+0000", 2024-08-19T14:16:53.635 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8750000000000002e-06 2024-08-19T14:16:53.636 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.636 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.636 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:16:53.636 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.265293+0000", 2024-08-19T14:16:53.636 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000119067 2024-08-19T14:16:53.636 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.636 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.636 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:16:53.636 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.265408+0000", 2024-08-19T14:16:53.636 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000115519 2024-08-19T14:16:53.636 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.636 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.636 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,10", 2024-08-19T14:16:53.636 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.265541+0000", 2024-08-19T14:16:53.637 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013279200000000001 2024-08-19T14:16:53.637 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.637 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.637 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:16:53.637 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.268514+0000", 2024-08-19T14:16:53.637 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0029729259999999999 2024-08-19T14:16:53.637 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.637 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.637 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:53.637 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.268576+0000", 2024-08-19T14:16:53.637 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2145999999999998e-05 2024-08-19T14:16:53.637 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.637 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.637 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:53.638 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.471473+0000", 2024-08-19T14:16:53.638 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.202897204 2024-08-19T14:16:53.638 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.638 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.638 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:16:53.638 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.471514+0000", 2024-08-19T14:16:53.638 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0484000000000001e-05 2024-08-19T14:16:53.638 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.638 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.638 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:16:53.638 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.471547+0000", 2024-08-19T14:16:53.638 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2929000000000003e-05 2024-08-19T14:16:53.638 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:53.639 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:16:53.639 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:53.639 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.639 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.639 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:48876 2.7 2:e761e839:::benchmark_data_smithi045_46415_object48875:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-08-19T14:16:53.639 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:07.265962+0000", 2024-08-19T14:16:53.639 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 226.29370939099999, 2024-08-19T14:16:53.639 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20685887, 2024-08-19T14:16:53.639 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:16:53.639 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:16:53.639 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:16:53.639 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:16:53.639 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:16:53.639 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:16:53.639 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 48876 2024-08-19T14:16:53.640 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.640 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:16:53.640 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.640 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:16:53.640 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.265962+0000", 2024-08-19T14:16:53.640 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:53.640 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.640 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.640 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:16:53.640 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.265962+0000", 2024-08-19T14:16:53.640 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:53.640 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.640 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.640 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:16:53.641 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.265963+0000", 2024-08-19T14:16:53.641 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0729999999999999e-06 2024-08-19T14:16:53.641 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.641 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.641 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:16:53.641 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.266042+0000", 2024-08-19T14:16:53.641 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.8845999999999997e-05 2024-08-19T14:16:53.641 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.641 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.641 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:16:53.641 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.266060+0000", 2024-08-19T14:16:53.641 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.783e-05 2024-08-19T14:16:53.641 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.641 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.642 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:16:53.642 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.266068+0000", 2024-08-19T14:16:53.642 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7009999999999996e-06 2024-08-19T14:16:53.642 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.642 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.642 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:16:53.642 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.266916+0000", 2024-08-19T14:16:53.642 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00084853200000000004 2024-08-19T14:16:53.642 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.642 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.642 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:16:53.642 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.267096+0000", 2024-08-19T14:16:53.642 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00017937699999999999 2024-08-19T14:16:53.642 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.643 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.643 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,10", 2024-08-19T14:16:53.643 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.267225+0000", 2024-08-19T14:16:53.643 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000129727 2024-08-19T14:16:53.643 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.643 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.643 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:16:53.643 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.268638+0000", 2024-08-19T14:16:53.643 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0014122640000000001 2024-08-19T14:16:53.643 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.643 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.643 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:53.643 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.269422+0000", 2024-08-19T14:16:53.644 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00078479600000000004 2024-08-19T14:16:53.644 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.644 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.644 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:53.644 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.472763+0000", 2024-08-19T14:16:53.644 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.203340627 2024-08-19T14:16:53.644 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.644 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.644 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:16:53.644 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.472797+0000", 2024-08-19T14:16:53.644 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.4060999999999998e-05 2024-08-19T14:16:53.644 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.644 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.644 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:16:53.645 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.472821+0000", 2024-08-19T14:16:53.645 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4036000000000002e-05 2024-08-19T14:16:53.645 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:53.645 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:16:53.645 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:53.645 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.645 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.645 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:48899 2.7 2:ed417148:::benchmark_data_smithi045_46415_object48898:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-08-19T14:16:53.645 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:07.275772+0000", 2024-08-19T14:16:53.645 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 226.283899652, 2024-08-19T14:16:53.645 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.19712979899999999, 2024-08-19T14:16:53.645 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:16:53.645 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:16:53.645 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:16:53.646 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:16:53.646 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:16:53.646 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:16:53.646 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 48899 2024-08-19T14:16:53.646 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.646 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:16:53.646 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.646 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:16:53.646 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.275772+0000", 2024-08-19T14:16:53.646 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:53.646 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.646 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.646 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:16:53.646 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.275772+0000", 2024-08-19T14:16:53.647 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:53.647 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.647 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.647 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:16:53.647 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.275774+0000", 2024-08-19T14:16:53.647 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5009999999999999e-06 2024-08-19T14:16:53.647 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.647 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.647 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:16:53.647 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.275826+0000", 2024-08-19T14:16:53.647 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2824000000000001e-05 2024-08-19T14:16:53.647 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.647 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.647 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:16:53.648 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.275841+0000", 2024-08-19T14:16:53.648 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4691e-05 2024-08-19T14:16:53.648 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.648 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.648 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:16:53.648 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.275849+0000", 2024-08-19T14:16:53.648 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1580000000000005e-06 2024-08-19T14:16:53.648 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.648 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.648 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:16:53.648 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.275913+0000", 2024-08-19T14:16:53.648 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3590999999999995e-05 2024-08-19T14:16:53.649 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.649 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.649 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:16:53.649 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.276130+0000", 2024-08-19T14:16:53.649 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000216901 2024-08-19T14:16:53.649 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.649 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.649 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,10", 2024-08-19T14:16:53.649 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.276273+0000", 2024-08-19T14:16:53.649 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014370100000000001 2024-08-19T14:16:53.649 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.650 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.650 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:16:53.650 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.278099+0000", 2024-08-19T14:16:53.650 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0018257849999999999 2024-08-19T14:16:53.650 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.650 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.650 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:53.650 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.281287+0000", 2024-08-19T14:16:53.650 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0031876769999999999 2024-08-19T14:16:53.650 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.650 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.650 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:53.650 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.472854+0000", 2024-08-19T14:16:53.650 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.19156674300000001 2024-08-19T14:16:53.651 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.651 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.651 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:16:53.651 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.472881+0000", 2024-08-19T14:16:53.651 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7521999999999998e-05 2024-08-19T14:16:53.651 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.651 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.651 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:16:53.651 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.472902+0000", 2024-08-19T14:16:53.651 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0704999999999999e-05 2024-08-19T14:16:53.651 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:53.651 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:16:53.651 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:53.651 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.652 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.652 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:48912 2.7 2:e6ffd652:::benchmark_data_smithi045_46415_object48911:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-08-19T14:16:53.652 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:07.279334+0000", 2024-08-19T14:16:53.652 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 226.280338151, 2024-08-19T14:16:53.652 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.195037181, 2024-08-19T14:16:53.652 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:16:53.652 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:16:53.652 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:16:53.652 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:16:53.652 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:16:53.652 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:16:53.652 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 48912 2024-08-19T14:16:53.652 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.652 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:16:53.653 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.653 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:16:53.653 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.279334+0000", 2024-08-19T14:16:53.653 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:53.653 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.653 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.653 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:16:53.653 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.279334+0000", 2024-08-19T14:16:53.653 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:53.653 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.653 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.653 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:16:53.653 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.279335+0000", 2024-08-19T14:16:53.653 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3599999999999999e-06 2024-08-19T14:16:53.654 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.654 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.654 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:16:53.654 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.279409+0000", 2024-08-19T14:16:53.654 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3819000000000006e-05 2024-08-19T14:16:53.654 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.654 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.654 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:16:53.654 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.279420+0000", 2024-08-19T14:16:53.654 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0892e-05 2024-08-19T14:16:53.654 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.654 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.654 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:16:53.654 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.279427+0000", 2024-08-19T14:16:53.655 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2790000000000001e-06 2024-08-19T14:16:53.655 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.655 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.655 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:16:53.655 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.279479+0000", 2024-08-19T14:16:53.655 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2547999999999999e-05 2024-08-19T14:16:53.655 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.655 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.655 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:16:53.655 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.279613+0000", 2024-08-19T14:16:53.655 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013337900000000001 2024-08-19T14:16:53.655 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.655 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.655 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,10", 2024-08-19T14:16:53.656 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.279754+0000", 2024-08-19T14:16:53.656 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000141359 2024-08-19T14:16:53.656 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.656 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.656 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:16:53.656 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.281352+0000", 2024-08-19T14:16:53.656 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0015973070000000001 2024-08-19T14:16:53.656 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.656 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.656 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:53.656 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.282744+0000", 2024-08-19T14:16:53.656 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0013921630000000001 2024-08-19T14:16:53.656 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.656 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.657 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:53.657 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.474309+0000", 2024-08-19T14:16:53.657 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.19156536800000001 2024-08-19T14:16:53.657 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.657 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.657 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:16:53.657 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.474347+0000", 2024-08-19T14:16:53.657 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.824e-05 2024-08-19T14:16:53.657 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.657 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.657 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:16:53.657 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.474371+0000", 2024-08-19T14:16:53.657 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3467000000000001e-05 2024-08-19T14:16:53.657 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:53.658 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:16:53.658 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:53.658 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.658 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.658 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:48921 2.7 2:e2434a00:::benchmark_data_smithi045_46415_object48920:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-08-19T14:16:53.658 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:07.283150+0000", 2024-08-19T14:16:53.658 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 226.27652191799999, 2024-08-19T14:16:53.658 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.19130543999999999, 2024-08-19T14:16:53.658 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:16:53.658 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:16:53.658 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:16:53.658 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:16:53.658 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:16:53.658 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:16:53.659 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 48921 2024-08-19T14:16:53.659 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.659 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:16:53.659 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.659 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:16:53.659 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.283150+0000", 2024-08-19T14:16:53.659 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:53.659 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.659 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.659 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:16:53.659 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.283150+0000", 2024-08-19T14:16:53.659 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:53.659 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.659 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.660 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:16:53.660 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.283151+0000", 2024-08-19T14:16:53.660 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6330000000000001e-06 2024-08-19T14:16:53.660 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.660 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.660 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:16:53.660 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.283177+0000", 2024-08-19T14:16:53.660 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5497e-05 2024-08-19T14:16:53.660 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.660 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.661 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:16:53.661 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.283190+0000", 2024-08-19T14:16:53.661 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3315e-05 2024-08-19T14:16:53.661 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.661 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.661 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:16:53.661 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.283197+0000", 2024-08-19T14:16:53.661 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7479999999999996e-06 2024-08-19T14:16:53.661 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.661 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.661 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:16:53.662 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.283262+0000", 2024-08-19T14:16:53.662 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4550999999999999e-05 2024-08-19T14:16:53.662 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.662 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.662 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:16:53.662 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.283434+0000", 2024-08-19T14:16:53.663 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00017221700000000001 2024-08-19T14:16:53.663 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.663 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.663 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,10", 2024-08-19T14:16:53.663 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.283553+0000", 2024-08-19T14:16:53.663 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0001197 2024-08-19T14:16:53.663 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.663 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.663 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:16:53.663 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.285150+0000", 2024-08-19T14:16:53.663 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00159615 2024-08-19T14:16:53.663 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.663 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.663 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:53.664 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.285949+0000", 2024-08-19T14:16:53.664 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00079935399999999995 2024-08-19T14:16:53.664 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.664 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.664 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:53.664 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.474403+0000", 2024-08-19T14:16:53.664 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.18845408199999999 2024-08-19T14:16:53.664 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.664 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.665 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:16:53.665 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.474433+0000", 2024-08-19T14:16:53.665 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9584999999999999e-05 2024-08-19T14:16:53.665 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.665 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.665 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:16:53.665 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.474455+0000", 2024-08-19T14:16:53.665 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2608000000000001e-05 2024-08-19T14:16:53.665 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:53.665 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:16:53.665 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:53.665 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.665 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.665 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:48971 2.7 2:ecf6a539:::benchmark_data_smithi045_46415_object48970:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-08-19T14:16:53.666 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:07.303688+0000", 2024-08-19T14:16:53.666 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 226.25598391599999, 2024-08-19T14:16:53.666 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.172252192, 2024-08-19T14:16:53.666 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:16:53.666 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:16:53.666 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:16:53.666 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:16:53.666 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:16:53.666 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:16:53.666 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 48971 2024-08-19T14:16:53.666 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.666 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:16:53.666 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.667 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:16:53.667 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.303688+0000", 2024-08-19T14:16:53.667 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:53.667 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.667 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.667 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:16:53.667 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.303688+0000", 2024-08-19T14:16:53.667 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:53.667 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.667 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.667 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:16:53.667 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.303689+0000", 2024-08-19T14:16:53.667 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.652e-06 2024-08-19T14:16:53.667 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.667 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.668 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:16:53.668 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.303729+0000", 2024-08-19T14:16:53.668 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9443000000000002e-05 2024-08-19T14:16:53.668 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.668 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.668 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:16:53.668 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.303744+0000", 2024-08-19T14:16:53.668 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5173e-05 2024-08-19T14:16:53.668 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.668 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.668 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:16:53.668 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.303751+0000", 2024-08-19T14:16:53.668 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.5910000000000001e-06 2024-08-19T14:16:53.669 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.669 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.669 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:16:53.669 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.303807+0000", 2024-08-19T14:16:53.669 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5918000000000002e-05 2024-08-19T14:16:53.669 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.669 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.669 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:16:53.669 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.303888+0000", 2024-08-19T14:16:53.669 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1302000000000002e-05 2024-08-19T14:16:53.669 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.669 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.669 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,10", 2024-08-19T14:16:53.669 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.303973+0000", 2024-08-19T14:16:53.670 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.5055000000000005e-05 2024-08-19T14:16:53.670 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.670 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.670 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:16:53.670 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.305453+0000", 2024-08-19T14:16:53.670 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001480182 2024-08-19T14:16:53.670 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.670 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.670 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:53.670 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.306105+0000", 2024-08-19T14:16:53.670 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00065189100000000004 2024-08-19T14:16:53.670 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.670 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.670 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:53.671 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.475889+0000", 2024-08-19T14:16:53.671 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.16978352599999999 2024-08-19T14:16:53.671 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.671 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.671 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:16:53.671 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.475919+0000", 2024-08-19T14:16:53.671 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0199e-05 2024-08-19T14:16:53.671 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.671 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.671 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:16:53.672 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:53 smithi031 ceph-mon[16336]: pgmap v529: 37 pgs: 1 peering, 8 unknown, 28 active+clean; 577 KiB data, 1.2 GiB used, 803 GiB / 805 GiB avail; 0 B/s wr, 717 op/s 2024-08-19T14:16:53.672 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:53 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3281440602' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-08-19T14:16:53.672 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:53 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]': finished 2024-08-19T14:16:53.672 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:53 smithi031 ceph-mon[16336]: osdmap e221: 12 total, 12 up, 9 in 2024-08-19T14:16:53.673 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.475940+0000", 2024-08-19T14:16:53.673 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.126e-05 2024-08-19T14:16:53.673 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:53.673 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:16:53.673 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:53.673 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.673 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.673 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:62930 2.7 2.377d1257 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-08-19T14:16:53.673 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:18.725228+0000", 2024-08-19T14:16:53.673 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 214.83444353900001, 2024-08-19T14:16:53.673 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.99330006599999998, 2024-08-19T14:16:53.673 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:16:53.674 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:16:53.674 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:16:53.674 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:16:53.674 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:16:53.674 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:16:53.674 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 62930 2024-08-19T14:16:53.674 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.674 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:16:53.674 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.674 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:16:53.674 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.725228+0000", 2024-08-19T14:16:53.674 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:53.674 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.674 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.675 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:16:53.675 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.725228+0000", 2024-08-19T14:16:53.675 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:53.675 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.675 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.675 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:16:53.675 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.725231+0000", 2024-08-19T14:16:53.675 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5950000000000001e-06 2024-08-19T14:16:53.675 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.675 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.675 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:16:53.675 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.725270+0000", 2024-08-19T14:16:53.675 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9162000000000002e-05 2024-08-19T14:16:53.675 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.676 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.676 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:16:53.676 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.725291+0000", 2024-08-19T14:16:53.676 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1220999999999999e-05 2024-08-19T14:16:53.676 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.676 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.676 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:16:53.676 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.725303+0000", 2024-08-19T14:16:53.676 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1788e-05 2024-08-19T14:16:53.676 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.676 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.676 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:16:53.676 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.725382+0000", 2024-08-19T14:16:53.677 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.9389999999999997e-05 2024-08-19T14:16:53.677 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.677 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.677 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:16:53.677 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.725395+0000", 2024-08-19T14:16:53.677 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.308e-05 2024-08-19T14:16:53.677 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.677 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.677 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:16:53.677 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.718510+0000", 2024-08-19T14:16:53.677 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.99311438900000004 2024-08-19T14:16:53.677 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.677 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.677 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:16:53.678 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.718528+0000", 2024-08-19T14:16:53.678 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8440999999999998e-05 2024-08-19T14:16:53.678 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:53.678 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:16:53.678 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:53.678 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.678 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.678 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:62932 2.7 2.84394fd7 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-08-19T14:16:53.678 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:18.727423+0000", 2024-08-19T14:16:53.678 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 214.832248378, 2024-08-19T14:16:53.678 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.99114474299999999, 2024-08-19T14:16:53.678 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:16:53.678 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:16:53.678 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:16:53.679 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:16:53.679 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:16:53.679 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:16:53.679 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 62932 2024-08-19T14:16:53.679 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.679 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:16:53.679 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.679 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:16:53.679 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.727423+0000", 2024-08-19T14:16:53.679 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:53.679 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.679 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.679 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:16:53.679 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.727423+0000", 2024-08-19T14:16:53.680 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:53.680 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.680 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.680 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:16:53.680 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.727425+0000", 2024-08-19T14:16:53.680 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0870000000000001e-06 2024-08-19T14:16:53.680 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.680 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.680 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:16:53.680 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.727471+0000", 2024-08-19T14:16:53.680 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5300999999999999e-05 2024-08-19T14:16:53.680 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.680 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.680 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:16:53.681 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.727489+0000", 2024-08-19T14:16:53.681 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7909000000000001e-05 2024-08-19T14:16:53.681 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.681 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.681 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:16:53.681 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.727497+0000", 2024-08-19T14:16:53.681 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.2150000000000001e-06 2024-08-19T14:16:53.681 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.681 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.681 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:16:53.681 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.727590+0000", 2024-08-19T14:16:53.681 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3193e-05 2024-08-19T14:16:53.681 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.681 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.682 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:16:53.682 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.727606+0000", 2024-08-19T14:16:53.682 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6345e-05 2024-08-19T14:16:53.682 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.682 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.682 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:16:53.682 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.718558+0000", 2024-08-19T14:16:53.682 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.99095196399999996 2024-08-19T14:16:53.682 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.682 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.682 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:16:53.682 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.718568+0000", 2024-08-19T14:16:53.682 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.7289999999999995e-06 2024-08-19T14:16:53.682 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:53.683 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:16:53.683 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:53.683 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.683 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.683 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:62948 2.7 2.63c33357 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-08-19T14:16:53.683 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:18.736424+0000", 2024-08-19T14:16:53.683 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 214.82324795700001, 2024-08-19T14:16:53.683 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.98218305500000003, 2024-08-19T14:16:53.683 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:16:53.683 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:16:53.683 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:16:53.683 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:16:53.683 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:16:53.683 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:16:53.684 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 62948 2024-08-19T14:16:53.684 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.684 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:16:53.684 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.684 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:16:53.684 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.736424+0000", 2024-08-19T14:16:53.684 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:53.684 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.684 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.684 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:16:53.684 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.736424+0000", 2024-08-19T14:16:53.684 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:53.684 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.684 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.685 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:16:53.685 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.736426+0000", 2024-08-19T14:16:53.685 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9980000000000002e-06 2024-08-19T14:16:53.685 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.685 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.685 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:16:53.685 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.736459+0000", 2024-08-19T14:16:53.685 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3155999999999999e-05 2024-08-19T14:16:53.685 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.685 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.685 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:16:53.685 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.736479+0000", 2024-08-19T14:16:53.685 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0052999999999999e-05 2024-08-19T14:16:53.685 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.686 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.686 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:16:53.686 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.736489+0000", 2024-08-19T14:16:53.686 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.7790000000000007e-06 2024-08-19T14:16:53.686 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.686 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.686 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:16:53.686 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.736567+0000", 2024-08-19T14:16:53.686 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.8240999999999998e-05 2024-08-19T14:16:53.686 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.686 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.686 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:16:53.686 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.736578+0000", 2024-08-19T14:16:53.686 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0813e-05 2024-08-19T14:16:53.687 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.687 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.687 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:16:53.687 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.718594+0000", 2024-08-19T14:16:53.687 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.98201632900000002 2024-08-19T14:16:53.687 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.687 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.687 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:16:53.687 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.718607+0000", 2024-08-19T14:16:53.687 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2686e-05 2024-08-19T14:16:53.687 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:53.687 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:16:53.687 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:53.687 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.688 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.688 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:62956 2.7 2.bf121107 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-08-19T14:16:53.688 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:18.741261+0000", 2024-08-19T14:16:53.688 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 214.81841052799999, 2024-08-19T14:16:53.688 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.97739631299999996, 2024-08-19T14:16:53.688 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:16:53.688 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:16:53.688 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:16:53.688 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:16:53.688 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:16:53.688 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:16:53.688 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 62956 2024-08-19T14:16:53.688 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.688 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:16:53.689 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.689 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:16:53.689 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.741261+0000", 2024-08-19T14:16:53.689 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:53.689 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.689 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.689 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:16:53.689 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.741261+0000", 2024-08-19T14:16:53.689 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:53.689 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.689 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.689 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:16:53.689 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.741263+0000", 2024-08-19T14:16:53.689 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.68e-06 2024-08-19T14:16:53.690 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.690 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.690 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:16:53.690 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.741301+0000", 2024-08-19T14:16:53.690 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8313999999999999e-05 2024-08-19T14:16:53.690 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.690 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.690 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:16:53.690 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.741320+0000", 2024-08-19T14:16:53.690 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.927e-05 2024-08-19T14:16:53.691 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.691 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.691 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:16:53.691 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.741328+0000", 2024-08-19T14:16:53.691 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3000000000000004e-06 2024-08-19T14:16:53.691 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.691 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.691 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:16:53.691 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.741408+0000", 2024-08-19T14:16:53.691 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.9839000000000006e-05 2024-08-19T14:16:53.691 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.691 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.691 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:16:53.692 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.741421+0000", 2024-08-19T14:16:53.692 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2914e-05 2024-08-19T14:16:53.692 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.692 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.692 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:16:53.692 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.718639+0000", 2024-08-19T14:16:53.692 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.97721861399999999 2024-08-19T14:16:53.692 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.692 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.692 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:16:53.692 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.718658+0000", 2024-08-19T14:16:53.692 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8382000000000001e-05 2024-08-19T14:16:53.692 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:53.692 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:16:53.693 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:53.693 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.693 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.693 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:62963 2.7 2.30acc547 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-08-19T14:16:53.693 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:18.745404+0000", 2024-08-19T14:16:53.693 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 214.81426798499999, 2024-08-19T14:16:53.693 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.97330407100000005, 2024-08-19T14:16:53.693 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:16:53.693 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:16:53.693 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:16:53.693 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:16:53.693 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:16:53.693 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:16:53.693 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 62963 2024-08-19T14:16:53.694 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.694 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:16:53.694 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.694 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:16:53.694 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.745404+0000", 2024-08-19T14:16:53.694 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:53.694 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.694 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.694 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:16:53.694 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.745404+0000", 2024-08-19T14:16:53.694 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:53.694 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.694 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.695 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:16:53.695 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.745406+0000", 2024-08-19T14:16:53.695 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.328e-06 2024-08-19T14:16:53.695 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.695 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.695 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:16:53.695 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.745440+0000", 2024-08-19T14:16:53.695 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3837000000000001e-05 2024-08-19T14:16:53.695 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.695 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.695 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:16:53.695 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.745474+0000", 2024-08-19T14:16:53.695 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.4196999999999998e-05 2024-08-19T14:16:53.695 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.696 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.696 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:16:53.696 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.745485+0000", 2024-08-19T14:16:53.696 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.095e-05 2024-08-19T14:16:53.696 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.696 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.696 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:16:53.696 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.745568+0000", 2024-08-19T14:16:53.696 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.2770999999999997e-05 2024-08-19T14:16:53.696 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.696 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.696 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:16:53.696 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.745577+0000", 2024-08-19T14:16:53.696 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.1269999999999992e-06 2024-08-19T14:16:53.697 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.697 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.697 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:16:53.697 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.718689+0000", 2024-08-19T14:16:53.697 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.97311253600000003 2024-08-19T14:16:53.697 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.697 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.697 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:16:53.697 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.718708+0000", 2024-08-19T14:16:53.697 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8325000000000002e-05 2024-08-19T14:16:53.697 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:53.697 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:16:53.697 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:53.697 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.698 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.698 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:62983 2.7 2.e3420e17 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-08-19T14:16:53.698 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:18.759553+0000", 2024-08-19T14:16:53.698 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 214.800118626, 2024-08-19T14:16:53.698 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.95919200800000004, 2024-08-19T14:16:53.698 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:16:53.698 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:16:53.698 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:16:53.698 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:16:53.698 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:16:53.698 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:16:53.698 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 62983 2024-08-19T14:16:53.698 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.698 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:16:53.699 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.699 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:16:53.699 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.759553+0000", 2024-08-19T14:16:53.699 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:53.699 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.699 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.699 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:16:53.699 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.759553+0000", 2024-08-19T14:16:53.699 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:53.699 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.699 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.699 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:16:53.699 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.759555+0000", 2024-08-19T14:16:53.699 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2630000000000002e-06 2024-08-19T14:16:53.700 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.700 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.700 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:16:53.700 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.759595+0000", 2024-08-19T14:16:53.700 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9490000000000003e-05 2024-08-19T14:16:53.700 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.700 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.700 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:16:53.700 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.759614+0000", 2024-08-19T14:16:53.700 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9446999999999999e-05 2024-08-19T14:16:53.700 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.700 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.700 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:16:53.700 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.759622+0000", 2024-08-19T14:16:53.701 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5519999999999999e-06 2024-08-19T14:16:53.701 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.701 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.701 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:16:53.701 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.759684+0000", 2024-08-19T14:16:53.701 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2168000000000005e-05 2024-08-19T14:16:53.701 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.701 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.701 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:16:53.701 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.759693+0000", 2024-08-19T14:16:53.701 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.8920000000000006e-06 2024-08-19T14:16:53.701 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.701 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.701 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:16:53.702 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.718735+0000", 2024-08-19T14:16:53.702 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.95904258099999995 2024-08-19T14:16:53.702 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.702 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.702 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:16:53.702 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.718745+0000", 2024-08-19T14:16:53.702 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.6150000000000003e-06 2024-08-19T14:16:53.702 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:53.702 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:16:53.702 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:53.702 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.702 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.702 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:63032 2.7 2.24f95377 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-08-19T14:16:53.702 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:18.809762+0000", 2024-08-19T14:16:53.703 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 214.74990994300001, 2024-08-19T14:16:53.703 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.909118655, 2024-08-19T14:16:53.703 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:16:53.703 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:16:53.703 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:16:53.703 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:16:53.703 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:16:53.703 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:16:53.703 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 63032 2024-08-19T14:16:53.703 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.703 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:16:53.703 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.703 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:16:53.703 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.809762+0000", 2024-08-19T14:16:53.704 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:53.704 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.704 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.704 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:16:53.704 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.809762+0000", 2024-08-19T14:16:53.704 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:53.704 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.704 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.704 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:16:53.704 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.809764+0000", 2024-08-19T14:16:53.704 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1490000000000001e-06 2024-08-19T14:16:53.704 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.704 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.704 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:16:53.704 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.809805+0000", 2024-08-19T14:16:53.705 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1241e-05 2024-08-19T14:16:53.705 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.705 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.705 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:16:53.705 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.809820+0000", 2024-08-19T14:16:53.705 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4355000000000001e-05 2024-08-19T14:16:53.705 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.705 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.705 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:16:53.705 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.809826+0000", 2024-08-19T14:16:53.705 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4829999999999996e-06 2024-08-19T14:16:53.705 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.705 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.705 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:16:53.706 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.809883+0000", 2024-08-19T14:16:53.706 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6885000000000002e-05 2024-08-19T14:16:53.706 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.706 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.706 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:16:53.706 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.809892+0000", 2024-08-19T14:16:53.706 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.8519999999999993e-06 2024-08-19T14:16:53.706 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.706 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.706 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:16:53.706 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.718868+0000", 2024-08-19T14:16:53.706 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.90897634900000002 2024-08-19T14:16:53.706 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:53.706 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:53.707 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:16:53.707 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.718880+0000", 2024-08-19T14:16:53.707 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2340999999999999e-05 2024-08-19T14:16:53.707 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:53.707 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:16:53.707 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:53.707 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:53.707 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:16:53.707 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:16:53.707 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:16:53.733 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:53.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:53.735 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:16:53.758 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-08-19T14:16:53.759 INFO:tasks.radosbench:Beginning radosbench... 2024-08-19T14:16:53.760 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_1 2024-08-19T14:16:53.760 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd pool create unique_pool_1 16 2024-08-19T14:16:53.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:53 smithi045 ceph-mon[20143]: pgmap v529: 37 pgs: 1 peering, 8 unknown, 28 active+clean; 577 KiB data, 1.2 GiB used, 803 GiB / 805 GiB avail; 0 B/s wr, 717 op/s 2024-08-19T14:16:53.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:53 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3281440602' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-08-19T14:16:53.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:53 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]': finished 2024-08-19T14:16:53.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:53 smithi045 ceph-mon[20143]: osdmap e221: 12 total, 12 up, 9 in 2024-08-19T14:16:53.836 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:16:53.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:53.861 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:53.861 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:16:53.962 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:16:53.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:53.987 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:53.987 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:16:54.088 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:16:54.109 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:54.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:54.110 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:16:54.211 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:16:54.237 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:54.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:54.238 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:16:54.339 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:16:54.353 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-08-19T14:16:54.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:54.366 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:54.366 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:16:54.450 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:16:54.467 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:16:54.493 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:54.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:54.493 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:16:54.594 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:16:54.603 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:16:54.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:54.616 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:54.617 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:16:54.718 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:16:54.740 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:54.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:54.741 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:16:54.842 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:16:54.867 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:54.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:54.868 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:16:54.888 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 7, 8, 9, 11] out_osds: [10, 6, 2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-08-19T14:16:54.888 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-19T14:16:54.888 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-08-19T14:16:54.889 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd dump -f json-pretty 2024-08-19T14:16:54.969 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:16:54.987 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:16:54.991 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:54.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:54.992 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:16:55.093 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:16:55.094 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:16:55.094 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-08-19T14:16:55.094 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-08-19T14:16:55.094 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:16:55.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:55.119 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:55.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:16:55.176 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:16:55.221 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:16:55.241 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:55.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:55.242 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:16:55.343 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:16:55.366 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:55.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:55.367 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:16:55.401 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'unique_pool_1' created 2024-08-19T14:16:55.401 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is unset 2024-08-19T14:16:55.468 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:16:55.490 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:55.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:55.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:16:55.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:55 smithi031 ceph-mon[16336]: pgmap v531: 11 pgs: 11 active+clean; 577 KiB data, 1.2 GiB used, 803 GiB / 805 GiB avail 2024-08-19T14:16:55.562 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:55 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/4174012472' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:16:55.591 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:16:55.612 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:55.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:55.613 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:16:55.672 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:55 smithi032 ceph-mon[20004]: pgmap v531: 11 pgs: 11 active+clean; 577 KiB data, 1.2 GiB used, 803 GiB / 805 GiB avail 2024-08-19T14:16:55.672 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:55 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/4174012472' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:16:55.700 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-08-19T14:16:55.714 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:16:55.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:55.767 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:55.767 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:16:55.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:55 smithi045 ceph-mon[20143]: pgmap v531: 11 pgs: 11 active+clean; 577 KiB data, 1.2 GiB used, 803 GiB / 805 GiB avail 2024-08-19T14:16:55.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:55 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/4174012472' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:16:55.868 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:16:55.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:55.891 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:55.891 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:16:55.959 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:16:55.992 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:16:56.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:56.015 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:56.016 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:16:56.117 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:16:56.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:56.139 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:56.139 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:16:56.240 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:16:56.262 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:56.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:56.263 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:16:56.364 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:16:56.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:56.387 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:56.388 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:16:56.474 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:16:56.474 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-08-19T14:16:56.474 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-08-19T14:16:56.474 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-08-19T14:16:56.475 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:16:56.484 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:56 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/545032181' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-08-19T14:16:56.485 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:56 smithi031 ceph-mon[16336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:16:56.485 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:56 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/4174012472' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:16:56.485 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:56 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/545032181' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-08-19T14:16:56.485 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:56 smithi031 ceph-mon[16336]: osdmap e222: 12 total, 12 up, 9 in 2024-08-19T14:16:56.489 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:16:56.511 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:56.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:56.512 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:16:56.613 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:16:56.637 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:56.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:56.638 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:16:56.739 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:16:56.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:56 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/545032181' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-08-19T14:16:56.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:56 smithi032 ceph-mon[20004]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:16:56.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:56 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/4174012472' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:16:56.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:56 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/545032181' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-08-19T14:16:56.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:56 smithi032 ceph-mon[20004]: osdmap e222: 12 total, 12 up, 9 in 2024-08-19T14:16:56.762 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:56.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:56.763 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:16:56.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:56 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/545032181' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-08-19T14:16:56.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:56 smithi045 ceph-mon[20143]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:16:56.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:56 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/4174012472' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:16:56.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:56 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/545032181' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-08-19T14:16:56.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:56 smithi045 ceph-mon[20143]: osdmap e222: 12 total, 12 up, 9 in 2024-08-19T14:16:56.840 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd pool application enable unique_pool_1 rados --yes-i-really-mean-it || true 2024-08-19T14:16:56.864 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:16:56.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:56.887 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:56.887 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:16:56.988 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:16:57.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:57.011 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:57.012 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:16:57.024 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:16:57.090 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-08-19T14:16:57.112 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:16:57.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:57.154 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:57.154 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:16:57.164 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:16:57.164 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:16:57.164 INFO:teuthology.orchestra.run.smithi031.stdout: "epoch": 223, 2024-08-19T14:16:57.164 INFO:teuthology.orchestra.run.smithi031.stdout: "fsid": "ed7d1c6e-5e33-11ef-bccf-c7b262605968", 2024-08-19T14:16:57.165 INFO:teuthology.orchestra.run.smithi031.stdout: "created": "2024-08-19T14:05:22.842169+0000", 2024-08-19T14:16:57.165 INFO:teuthology.orchestra.run.smithi031.stdout: "modified": "2024-08-19T14:16:56.401926+0000", 2024-08-19T14:16:57.165 INFO:teuthology.orchestra.run.smithi031.stdout: "last_up_change": "2024-08-19T14:11:39.491234+0000", 2024-08-19T14:16:57.165 INFO:teuthology.orchestra.run.smithi031.stdout: "last_in_change": "2024-08-19T14:16:38.207673+0000", 2024-08-19T14:16:57.165 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-08-19T14:16:57.165 INFO:teuthology.orchestra.run.smithi031.stdout: "flags_num": 5804032, 2024-08-19T14:16:57.165 INFO:teuthology.orchestra.run.smithi031.stdout: "flags_set": [ 2024-08-19T14:16:57.165 INFO:teuthology.orchestra.run.smithi031.stdout: "nodeep-scrub", 2024-08-19T14:16:57.165 INFO:teuthology.orchestra.run.smithi031.stdout: "pglog_hardlimit", 2024-08-19T14:16:57.165 INFO:teuthology.orchestra.run.smithi031.stdout: "purged_snapdirs", 2024-08-19T14:16:57.165 INFO:teuthology.orchestra.run.smithi031.stdout: "recovery_deletes", 2024-08-19T14:16:57.165 INFO:teuthology.orchestra.run.smithi031.stdout: "sortbitwise" 2024-08-19T14:16:57.165 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-08-19T14:16:57.166 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_version": 27, 2024-08-19T14:16:57.166 INFO:teuthology.orchestra.run.smithi031.stdout: "full_ratio": 0.94999998807907104, 2024-08-19T14:16:57.166 INFO:teuthology.orchestra.run.smithi031.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-08-19T14:16:57.166 INFO:teuthology.orchestra.run.smithi031.stdout: "nearfull_ratio": 0.85000002384185791, 2024-08-19T14:16:57.166 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_snapshot": "", 2024-08-19T14:16:57.166 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_max": 3, 2024-08-19T14:16:57.166 INFO:teuthology.orchestra.run.smithi031.stdout: "max_osd": 12, 2024-08-19T14:16:57.166 INFO:teuthology.orchestra.run.smithi031.stdout: "require_min_compat_client": "luminous", 2024-08-19T14:16:57.166 INFO:teuthology.orchestra.run.smithi031.stdout: "min_compat_client": "luminous", 2024-08-19T14:16:57.166 INFO:teuthology.orchestra.run.smithi031.stdout: "require_osd_release": "squid", 2024-08-19T14:16:57.166 INFO:teuthology.orchestra.run.smithi031.stdout: "allow_crimson": false, 2024-08-19T14:16:57.166 INFO:teuthology.orchestra.run.smithi031.stdout: "pools": [ 2024-08-19T14:16:57.166 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.166 INFO:teuthology.orchestra.run.smithi031.stdout: "pool": 1, 2024-08-19T14:16:57.167 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_name": ".mgr", 2024-08-19T14:16:57.167 INFO:teuthology.orchestra.run.smithi031.stdout: "create_time": "2024-08-19T14:08:15.966221+0000", 2024-08-19T14:16:57.167 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-08-19T14:16:57.167 INFO:teuthology.orchestra.run.smithi031.stdout: "flags_names": "hashpspool", 2024-08-19T14:16:57.167 INFO:teuthology.orchestra.run.smithi031.stdout: "type": 1, 2024-08-19T14:16:57.167 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 3, 2024-08-19T14:16:57.167 INFO:teuthology.orchestra.run.smithi031.stdout: "min_size": 2, 2024-08-19T14:16:57.167 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_rule": 0, 2024-08-19T14:16:57.167 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_count": 0, 2024-08-19T14:16:57.168 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_target": 0, 2024-08-19T14:16:57.168 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_barrier": 0, 2024-08-19T14:16:57.168 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-08-19T14:16:57.168 INFO:teuthology.orchestra.run.smithi031.stdout: "object_hash": 2, 2024-08-19T14:16:57.168 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_autoscale_mode": "off", 2024-08-19T14:16:57.168 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num": 11, 2024-08-19T14:16:57.168 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_placement_num": 11, 2024-08-19T14:16:57.168 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_placement_num_target": 11, 2024-08-19T14:16:57.169 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num_target": 11, 2024-08-19T14:16:57.169 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num_pending": 11, 2024-08-19T14:16:57.169 INFO:teuthology.orchestra.run.smithi031.stdout: "last_pg_merge_meta": { 2024-08-19T14:16:57.169 INFO:teuthology.orchestra.run.smithi031.stdout: "source_pgid": "0.0", 2024-08-19T14:16:57.169 INFO:teuthology.orchestra.run.smithi031.stdout: "ready_epoch": 0, 2024-08-19T14:16:57.169 INFO:teuthology.orchestra.run.smithi031.stdout: "last_epoch_started": 0, 2024-08-19T14:16:57.169 INFO:teuthology.orchestra.run.smithi031.stdout: "last_epoch_clean": 0, 2024-08-19T14:16:57.169 INFO:teuthology.orchestra.run.smithi031.stdout: "source_version": "0'0", 2024-08-19T14:16:57.170 INFO:teuthology.orchestra.run.smithi031.stdout: "target_version": "0'0" 2024-08-19T14:16:57.170 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.170 INFO:teuthology.orchestra.run.smithi031.stdout: "last_change": "112", 2024-08-19T14:16:57.170 INFO:teuthology.orchestra.run.smithi031.stdout: "last_force_op_resend": "0", 2024-08-19T14:16:57.170 INFO:teuthology.orchestra.run.smithi031.stdout: "last_force_op_resend_prenautilus": "0", 2024-08-19T14:16:57.170 INFO:teuthology.orchestra.run.smithi031.stdout: "last_force_op_resend_preluminous": "110", 2024-08-19T14:16:57.170 INFO:teuthology.orchestra.run.smithi031.stdout: "auid": 0, 2024-08-19T14:16:57.170 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_mode": "selfmanaged", 2024-08-19T14:16:57.171 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_seq": 0, 2024-08-19T14:16:57.171 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_epoch": 0, 2024-08-19T14:16:57.171 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_snaps": [], 2024-08-19T14:16:57.171 INFO:teuthology.orchestra.run.smithi031.stdout: "removed_snaps": "[]", 2024-08-19T14:16:57.171 INFO:teuthology.orchestra.run.smithi031.stdout: "quota_max_bytes": 0, 2024-08-19T14:16:57.171 INFO:teuthology.orchestra.run.smithi031.stdout: "quota_max_objects": 0, 2024-08-19T14:16:57.171 INFO:teuthology.orchestra.run.smithi031.stdout: "tiers": [], 2024-08-19T14:16:57.171 INFO:teuthology.orchestra.run.smithi031.stdout: "tier_of": -1, 2024-08-19T14:16:57.171 INFO:teuthology.orchestra.run.smithi031.stdout: "read_tier": -1, 2024-08-19T14:16:57.172 INFO:teuthology.orchestra.run.smithi031.stdout: "write_tier": -1, 2024-08-19T14:16:57.172 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_mode": "none", 2024-08-19T14:16:57.172 INFO:teuthology.orchestra.run.smithi031.stdout: "target_max_bytes": 0, 2024-08-19T14:16:57.172 INFO:teuthology.orchestra.run.smithi031.stdout: "target_max_objects": 0, 2024-08-19T14:16:57.172 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-08-19T14:16:57.172 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-08-19T14:16:57.172 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_target_full_ratio_micro": 800000, 2024-08-19T14:16:57.172 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_min_flush_age": 0, 2024-08-19T14:16:57.173 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_min_evict_age": 0, 2024-08-19T14:16:57.173 INFO:teuthology.orchestra.run.smithi031.stdout: "erasure_code_profile": "", 2024-08-19T14:16:57.173 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_params": { 2024-08-19T14:16:57.173 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "none" 2024-08-19T14:16:57.173 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.173 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_period": 0, 2024-08-19T14:16:57.173 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_count": 0, 2024-08-19T14:16:57.173 INFO:teuthology.orchestra.run.smithi031.stdout: "use_gmt_hitset": true, 2024-08-19T14:16:57.173 INFO:teuthology.orchestra.run.smithi031.stdout: "min_read_recency_for_promote": 0, 2024-08-19T14:16:57.173 INFO:teuthology.orchestra.run.smithi031.stdout: "min_write_recency_for_promote": 0, 2024-08-19T14:16:57.173 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_grade_decay_rate": 0, 2024-08-19T14:16:57.173 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_search_last_n": 0, 2024-08-19T14:16:57.173 INFO:teuthology.orchestra.run.smithi031.stdout: "grade_table": [], 2024-08-19T14:16:57.174 INFO:teuthology.orchestra.run.smithi031.stdout: "stripe_width": 0, 2024-08-19T14:16:57.174 INFO:teuthology.orchestra.run.smithi031.stdout: "expected_num_objects": 0, 2024-08-19T14:16:57.174 INFO:teuthology.orchestra.run.smithi031.stdout: "fast_read": false, 2024-08-19T14:16:57.174 INFO:teuthology.orchestra.run.smithi031.stdout: "options": { 2024-08-19T14:16:57.174 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num_max": 32, 2024-08-19T14:16:57.174 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num_min": 1 2024-08-19T14:16:57.174 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.174 INFO:teuthology.orchestra.run.smithi031.stdout: "application_metadata": { 2024-08-19T14:16:57.174 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": {} 2024-08-19T14:16:57.174 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.174 INFO:teuthology.orchestra.run.smithi031.stdout: "read_balance": { 2024-08-19T14:16:57.174 INFO:teuthology.orchestra.run.smithi031.stdout: "score_type": "Fair distribution", 2024-08-19T14:16:57.174 INFO:teuthology.orchestra.run.smithi031.stdout: "score_acting": 3.2699999809265137, 2024-08-19T14:16:57.174 INFO:teuthology.orchestra.run.smithi031.stdout: "score_stable": 3.2699999809265137, 2024-08-19T14:16:57.175 INFO:teuthology.orchestra.run.smithi031.stdout: "optimal_score": 0.75, 2024-08-19T14:16:57.175 INFO:teuthology.orchestra.run.smithi031.stdout: "raw_score_acting": 2.4500000476837158, 2024-08-19T14:16:57.175 INFO:teuthology.orchestra.run.smithi031.stdout: "raw_score_stable": 2.4500000476837158, 2024-08-19T14:16:57.175 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity_weighted": 0.92281585931777954, 2024-08-19T14:16:57.175 INFO:teuthology.orchestra.run.smithi031.stdout: "average_primary_affinity": 0.92000001668930054, 2024-08-19T14:16:57.175 INFO:teuthology.orchestra.run.smithi031.stdout: "average_primary_affinity_weighted": 0.92000001668930054 2024-08-19T14:16:57.175 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:57.175 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.175 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.175 INFO:teuthology.orchestra.run.smithi031.stdout: "pool": 3, 2024-08-19T14:16:57.175 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_name": "unique_pool_1", 2024-08-19T14:16:57.175 INFO:teuthology.orchestra.run.smithi031.stdout: "create_time": "2024-08-19T14:16:55.393520+0000", 2024-08-19T14:16:57.175 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 32769, 2024-08-19T14:16:57.176 INFO:teuthology.orchestra.run.smithi031.stdout: "flags_names": "hashpspool,creating", 2024-08-19T14:16:57.176 INFO:teuthology.orchestra.run.smithi031.stdout: "type": 1, 2024-08-19T14:16:57.176 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 3, 2024-08-19T14:16:57.176 INFO:teuthology.orchestra.run.smithi031.stdout: "min_size": 2, 2024-08-19T14:16:57.176 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_rule": 0, 2024-08-19T14:16:57.176 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_count": 0, 2024-08-19T14:16:57.176 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_target": 0, 2024-08-19T14:16:57.176 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_barrier": 0, 2024-08-19T14:16:57.176 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-08-19T14:16:57.176 INFO:teuthology.orchestra.run.smithi031.stdout: "object_hash": 2, 2024-08-19T14:16:57.176 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_autoscale_mode": "off", 2024-08-19T14:16:57.176 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num": 16, 2024-08-19T14:16:57.176 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_placement_num": 16, 2024-08-19T14:16:57.176 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_placement_num_target": 16, 2024-08-19T14:16:57.177 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num_target": 16, 2024-08-19T14:16:57.177 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num_pending": 16, 2024-08-19T14:16:57.177 INFO:teuthology.orchestra.run.smithi031.stdout: "last_pg_merge_meta": { 2024-08-19T14:16:57.177 INFO:teuthology.orchestra.run.smithi031.stdout: "source_pgid": "0.0", 2024-08-19T14:16:57.177 INFO:teuthology.orchestra.run.smithi031.stdout: "ready_epoch": 0, 2024-08-19T14:16:57.177 INFO:teuthology.orchestra.run.smithi031.stdout: "last_epoch_started": 0, 2024-08-19T14:16:57.177 INFO:teuthology.orchestra.run.smithi031.stdout: "last_epoch_clean": 0, 2024-08-19T14:16:57.177 INFO:teuthology.orchestra.run.smithi031.stdout: "source_version": "0'0", 2024-08-19T14:16:57.177 INFO:teuthology.orchestra.run.smithi031.stdout: "target_version": "0'0" 2024-08-19T14:16:57.177 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.177 INFO:teuthology.orchestra.run.smithi031.stdout: "last_change": "222", 2024-08-19T14:16:57.177 INFO:teuthology.orchestra.run.smithi031.stdout: "last_force_op_resend": "0", 2024-08-19T14:16:57.177 INFO:teuthology.orchestra.run.smithi031.stdout: "last_force_op_resend_prenautilus": "0", 2024-08-19T14:16:57.177 INFO:teuthology.orchestra.run.smithi031.stdout: "last_force_op_resend_preluminous": "0", 2024-08-19T14:16:57.178 INFO:teuthology.orchestra.run.smithi031.stdout: "auid": 0, 2024-08-19T14:16:57.178 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_mode": "selfmanaged", 2024-08-19T14:16:57.178 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_seq": 0, 2024-08-19T14:16:57.178 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_epoch": 0, 2024-08-19T14:16:57.178 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_snaps": [], 2024-08-19T14:16:57.178 INFO:teuthology.orchestra.run.smithi031.stdout: "removed_snaps": "[]", 2024-08-19T14:16:57.178 INFO:teuthology.orchestra.run.smithi031.stdout: "quota_max_bytes": 0, 2024-08-19T14:16:57.178 INFO:teuthology.orchestra.run.smithi031.stdout: "quota_max_objects": 0, 2024-08-19T14:16:57.178 INFO:teuthology.orchestra.run.smithi031.stdout: "tiers": [], 2024-08-19T14:16:57.178 INFO:teuthology.orchestra.run.smithi031.stdout: "tier_of": -1, 2024-08-19T14:16:57.178 INFO:teuthology.orchestra.run.smithi031.stdout: "read_tier": -1, 2024-08-19T14:16:57.178 INFO:teuthology.orchestra.run.smithi031.stdout: "write_tier": -1, 2024-08-19T14:16:57.178 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_mode": "none", 2024-08-19T14:16:57.178 INFO:teuthology.orchestra.run.smithi031.stdout: "target_max_bytes": 0, 2024-08-19T14:16:57.179 INFO:teuthology.orchestra.run.smithi031.stdout: "target_max_objects": 0, 2024-08-19T14:16:57.179 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-08-19T14:16:57.179 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-08-19T14:16:57.179 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_target_full_ratio_micro": 800000, 2024-08-19T14:16:57.179 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_min_flush_age": 0, 2024-08-19T14:16:57.179 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_min_evict_age": 0, 2024-08-19T14:16:57.179 INFO:teuthology.orchestra.run.smithi031.stdout: "erasure_code_profile": "", 2024-08-19T14:16:57.179 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_params": { 2024-08-19T14:16:57.179 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "none" 2024-08-19T14:16:57.179 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.179 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_period": 0, 2024-08-19T14:16:57.179 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_count": 0, 2024-08-19T14:16:57.179 INFO:teuthology.orchestra.run.smithi031.stdout: "use_gmt_hitset": true, 2024-08-19T14:16:57.179 INFO:teuthology.orchestra.run.smithi031.stdout: "min_read_recency_for_promote": 0, 2024-08-19T14:16:57.180 INFO:teuthology.orchestra.run.smithi031.stdout: "min_write_recency_for_promote": 0, 2024-08-19T14:16:57.180 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_grade_decay_rate": 0, 2024-08-19T14:16:57.180 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_search_last_n": 0, 2024-08-19T14:16:57.180 INFO:teuthology.orchestra.run.smithi031.stdout: "grade_table": [], 2024-08-19T14:16:57.180 INFO:teuthology.orchestra.run.smithi031.stdout: "stripe_width": 0, 2024-08-19T14:16:57.180 INFO:teuthology.orchestra.run.smithi031.stdout: "expected_num_objects": 0, 2024-08-19T14:16:57.180 INFO:teuthology.orchestra.run.smithi031.stdout: "fast_read": false, 2024-08-19T14:16:57.180 INFO:teuthology.orchestra.run.smithi031.stdout: "options": {}, 2024-08-19T14:16:57.180 INFO:teuthology.orchestra.run.smithi031.stdout: "application_metadata": {}, 2024-08-19T14:16:57.180 INFO:teuthology.orchestra.run.smithi031.stdout: "read_balance": { 2024-08-19T14:16:57.180 INFO:teuthology.orchestra.run.smithi031.stdout: "score_type": "Fair distribution", 2024-08-19T14:16:57.180 INFO:teuthology.orchestra.run.smithi031.stdout: "score_acting": 2.25, 2024-08-19T14:16:57.180 INFO:teuthology.orchestra.run.smithi031.stdout: "score_stable": 2.25, 2024-08-19T14:16:57.180 INFO:teuthology.orchestra.run.smithi031.stdout: "optimal_score": 0.75, 2024-08-19T14:16:57.181 INFO:teuthology.orchestra.run.smithi031.stdout: "raw_score_acting": 1.690000057220459, 2024-08-19T14:16:57.181 INFO:teuthology.orchestra.run.smithi031.stdout: "raw_score_stable": 1.690000057220459, 2024-08-19T14:16:57.181 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity_weighted": 0.92281585931777954, 2024-08-19T14:16:57.181 INFO:teuthology.orchestra.run.smithi031.stdout: "average_primary_affinity": 0.92000001668930054, 2024-08-19T14:16:57.181 INFO:teuthology.orchestra.run.smithi031.stdout: "average_primary_affinity_weighted": 0.92000001668930054 2024-08-19T14:16:57.181 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:57.181 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:57.181 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-08-19T14:16:57.181 INFO:teuthology.orchestra.run.smithi031.stdout: "osds": [ 2024-08-19T14:16:57.181 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.181 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 0, 2024-08-19T14:16:57.181 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "d7a0fff3-23c6-4ee3-a275-a844862aabfb", 2024-08-19T14:16:57.181 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-08-19T14:16:57.182 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-08-19T14:16:57.182 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-08-19T14:16:57.182 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-08-19T14:16:57.182 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-08-19T14:16:57.182 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-08-19T14:16:57.182 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 8, 2024-08-19T14:16:57.182 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 222, 2024-08-19T14:16:57.182 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-08-19T14:16:57.182 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-08-19T14:16:57.182 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-08-19T14:16:57.182 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:57.182 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.182 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:57.182 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6802", 2024-08-19T14:16:57.183 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1108945097 2024-08-19T14:16:57.183 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.183 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.183 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:57.183 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6803", 2024-08-19T14:16:57.183 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1108945097 2024-08-19T14:16:57.183 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:57.183 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:57.183 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.183 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-08-19T14:16:57.183 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:57.183 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.183 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:57.184 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6804", 2024-08-19T14:16:57.184 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1108945097 2024-08-19T14:16:57.184 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.184 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.185 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:57.185 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6805", 2024-08-19T14:16:57.185 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1108945097 2024-08-19T14:16:57.185 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:57.185 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:57.185 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.185 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-08-19T14:16:57.185 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:57.185 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.185 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:57.185 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6808", 2024-08-19T14:16:57.185 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1108945097 2024-08-19T14:16:57.185 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.186 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.186 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:57.186 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6809", 2024-08-19T14:16:57.186 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1108945097 2024-08-19T14:16:57.186 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:57.186 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:57.186 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.186 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-08-19T14:16:57.186 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:57.186 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.187 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:57.187 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6806", 2024-08-19T14:16:57.187 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1108945097 2024-08-19T14:16:57.188 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.188 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.188 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:57.188 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6807", 2024-08-19T14:16:57.188 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1108945097 2024-08-19T14:16:57.188 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:57.188 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:57.188 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.188 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.31:6803/1108945097", 2024-08-19T14:16:57.188 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.31:6805/1108945097", 2024-08-19T14:16:57.188 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.31:6809/1108945097", 2024-08-19T14:16:57.189 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.31:6807/1108945097", 2024-08-19T14:16:57.189 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-08-19T14:16:57.189 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-08-19T14:16:57.189 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-08-19T14:16:57.189 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:57.189 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.189 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.189 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 1, 2024-08-19T14:16:57.189 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "145d5bb4-74dd-440f-bcc9-7e9ebda9c5d6", 2024-08-19T14:16:57.189 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-08-19T14:16:57.189 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-08-19T14:16:57.189 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-08-19T14:16:57.189 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-08-19T14:16:57.189 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-08-19T14:16:57.190 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-08-19T14:16:57.190 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 13, 2024-08-19T14:16:57.190 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 222, 2024-08-19T14:16:57.190 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-08-19T14:16:57.190 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-08-19T14:16:57.190 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-08-19T14:16:57.190 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:57.190 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.190 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:57.190 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6810", 2024-08-19T14:16:57.190 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3236186806 2024-08-19T14:16:57.190 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.191 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.191 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:57.191 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6811", 2024-08-19T14:16:57.191 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3236186806 2024-08-19T14:16:57.191 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:57.191 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:57.191 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.191 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-08-19T14:16:57.191 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:57.191 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.191 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:57.191 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6812", 2024-08-19T14:16:57.191 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3236186806 2024-08-19T14:16:57.191 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.192 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.192 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:57.192 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6813", 2024-08-19T14:16:57.192 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3236186806 2024-08-19T14:16:57.192 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:57.192 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:57.192 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.192 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-08-19T14:16:57.192 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:57.192 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.192 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:57.192 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6816", 2024-08-19T14:16:57.192 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3236186806 2024-08-19T14:16:57.192 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.193 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.193 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:57.193 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6817", 2024-08-19T14:16:57.193 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3236186806 2024-08-19T14:16:57.193 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:57.193 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:57.193 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.193 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-08-19T14:16:57.193 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:57.193 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.193 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:57.193 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6814", 2024-08-19T14:16:57.193 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3236186806 2024-08-19T14:16:57.193 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.194 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.194 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:57.194 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6815", 2024-08-19T14:16:57.194 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3236186806 2024-08-19T14:16:57.194 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:57.194 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:57.194 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.194 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.31:6811/3236186806", 2024-08-19T14:16:57.194 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.31:6813/3236186806", 2024-08-19T14:16:57.194 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.31:6817/3236186806", 2024-08-19T14:16:57.194 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.31:6815/3236186806", 2024-08-19T14:16:57.194 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-08-19T14:16:57.194 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-08-19T14:16:57.195 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-08-19T14:16:57.195 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:57.195 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.195 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.195 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 2, 2024-08-19T14:16:57.195 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "376c18d0-4768-46a0-bb60-e79c88ca909c", 2024-08-19T14:16:57.195 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-08-19T14:16:57.195 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 0, 2024-08-19T14:16:57.195 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 0, 2024-08-19T14:16:57.195 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 0.07379150390625, 2024-08-19T14:16:57.195 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-08-19T14:16:57.195 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-08-19T14:16:57.195 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 18, 2024-08-19T14:16:57.196 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 204, 2024-08-19T14:16:57.196 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-08-19T14:16:57.196 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-08-19T14:16:57.196 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-08-19T14:16:57.196 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:57.196 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.196 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:57.196 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6818", 2024-08-19T14:16:57.196 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 4045553387 2024-08-19T14:16:57.196 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.196 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.196 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:57.196 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6819", 2024-08-19T14:16:57.197 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 4045553387 2024-08-19T14:16:57.197 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:57.197 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:57.197 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.197 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-08-19T14:16:57.197 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:57.197 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.197 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:57.197 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6820", 2024-08-19T14:16:57.197 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 4045553387 2024-08-19T14:16:57.197 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.197 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.197 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:57.197 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6821", 2024-08-19T14:16:57.198 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 4045553387 2024-08-19T14:16:57.198 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:57.198 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:57.198 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.198 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-08-19T14:16:57.198 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:57.198 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.198 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:57.198 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6824", 2024-08-19T14:16:57.198 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 4045553387 2024-08-19T14:16:57.198 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.198 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.198 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:57.199 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6825", 2024-08-19T14:16:57.199 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 4045553387 2024-08-19T14:16:57.199 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:57.199 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:57.199 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.199 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-08-19T14:16:57.199 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:57.199 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.199 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:57.199 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6822", 2024-08-19T14:16:57.199 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 4045553387 2024-08-19T14:16:57.199 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.199 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.199 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:57.199 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6823", 2024-08-19T14:16:57.200 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 4045553387 2024-08-19T14:16:57.200 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:57.200 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:57.200 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.200 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.31:6819/4045553387", 2024-08-19T14:16:57.200 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.31:6821/4045553387", 2024-08-19T14:16:57.200 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.31:6825/4045553387", 2024-08-19T14:16:57.200 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.31:6823/4045553387", 2024-08-19T14:16:57.200 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-08-19T14:16:57.200 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-08-19T14:16:57.200 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-08-19T14:16:57.200 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:57.200 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.200 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.201 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 3, 2024-08-19T14:16:57.201 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "a787ea9f-b9ba-450f-b388-614cfba90634", 2024-08-19T14:16:57.201 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-08-19T14:16:57.201 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-08-19T14:16:57.201 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 0.6437225341796875, 2024-08-19T14:16:57.201 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-08-19T14:16:57.201 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-08-19T14:16:57.201 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-08-19T14:16:57.201 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 26, 2024-08-19T14:16:57.201 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 222, 2024-08-19T14:16:57.201 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-08-19T14:16:57.201 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-08-19T14:16:57.201 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-08-19T14:16:57.201 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:57.202 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.202 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:57.202 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6826", 2024-08-19T14:16:57.202 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2288082716 2024-08-19T14:16:57.202 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.202 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.202 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:57.202 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6827", 2024-08-19T14:16:57.203 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2288082716 2024-08-19T14:16:57.203 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:57.203 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:57.203 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.203 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-08-19T14:16:57.203 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:57.203 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.203 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:57.203 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6828", 2024-08-19T14:16:57.204 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2288082716 2024-08-19T14:16:57.204 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.204 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.204 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:57.204 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6829", 2024-08-19T14:16:57.204 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2288082716 2024-08-19T14:16:57.204 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:57.204 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:57.204 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.204 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-08-19T14:16:57.205 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:57.205 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.205 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:57.205 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6832", 2024-08-19T14:16:57.205 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2288082716 2024-08-19T14:16:57.205 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.205 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.205 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:57.205 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6833", 2024-08-19T14:16:57.205 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2288082716 2024-08-19T14:16:57.205 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:57.205 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:57.205 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.206 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-08-19T14:16:57.206 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:57.206 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.206 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:57.206 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6830", 2024-08-19T14:16:57.206 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2288082716 2024-08-19T14:16:57.206 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.206 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.206 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:57.206 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6831", 2024-08-19T14:16:57.206 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2288082716 2024-08-19T14:16:57.206 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:57.206 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:57.207 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.207 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.31:6827/2288082716", 2024-08-19T14:16:57.207 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.31:6829/2288082716", 2024-08-19T14:16:57.207 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.31:6833/2288082716", 2024-08-19T14:16:57.207 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.31:6831/2288082716", 2024-08-19T14:16:57.207 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-08-19T14:16:57.207 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-08-19T14:16:57.207 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-08-19T14:16:57.207 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:57.207 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.207 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.207 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 4, 2024-08-19T14:16:57.207 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "7859751a-12f5-4a57-a332-435360479d53", 2024-08-19T14:16:57.207 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-08-19T14:16:57.208 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-08-19T14:16:57.208 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-08-19T14:16:57.208 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-08-19T14:16:57.208 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-08-19T14:16:57.208 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-08-19T14:16:57.208 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 31, 2024-08-19T14:16:57.208 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 222, 2024-08-19T14:16:57.208 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-08-19T14:16:57.208 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-08-19T14:16:57.208 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-08-19T14:16:57.208 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:57.208 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.208 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:57.208 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6800", 2024-08-19T14:16:57.209 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3934739984 2024-08-19T14:16:57.209 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.209 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.209 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:57.209 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6801", 2024-08-19T14:16:57.209 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3934739984 2024-08-19T14:16:57.209 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:57.209 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:57.209 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.209 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-08-19T14:16:57.210 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:57.210 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.210 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:57.210 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6802", 2024-08-19T14:16:57.210 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3934739984 2024-08-19T14:16:57.210 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.210 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.210 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:57.210 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6803", 2024-08-19T14:16:57.211 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3934739984 2024-08-19T14:16:57.211 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:57.211 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:57.211 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.211 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-08-19T14:16:57.211 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:57.211 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.211 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:57.211 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6806", 2024-08-19T14:16:57.211 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3934739984 2024-08-19T14:16:57.211 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.211 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.212 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:57.212 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6807", 2024-08-19T14:16:57.212 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3934739984 2024-08-19T14:16:57.212 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:57.212 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:57.212 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.212 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-08-19T14:16:57.212 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:57.212 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.212 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:57.212 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6804", 2024-08-19T14:16:57.213 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3934739984 2024-08-19T14:16:57.213 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.213 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.213 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:57.213 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6805", 2024-08-19T14:16:57.213 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3934739984 2024-08-19T14:16:57.213 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:57.213 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:57.213 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.213 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.32:6801/3934739984", 2024-08-19T14:16:57.213 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.32:6803/3934739984", 2024-08-19T14:16:57.213 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.32:6807/3934739984", 2024-08-19T14:16:57.213 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.32:6805/3934739984", 2024-08-19T14:16:57.213 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-08-19T14:16:57.214 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-08-19T14:16:57.214 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-08-19T14:16:57.214 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:57.214 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.214 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.214 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 5, 2024-08-19T14:16:57.214 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "017f3eef-ac60-4b82-8350-919017cb4ccd", 2024-08-19T14:16:57.214 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-08-19T14:16:57.214 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-08-19T14:16:57.215 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-08-19T14:16:57.215 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-08-19T14:16:57.215 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-08-19T14:16:57.215 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-08-19T14:16:57.215 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 40, 2024-08-19T14:16:57.215 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 222, 2024-08-19T14:16:57.215 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-08-19T14:16:57.215 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-08-19T14:16:57.215 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-08-19T14:16:57.215 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:57.215 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.216 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:57.216 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6808", 2024-08-19T14:16:57.216 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 526388985 2024-08-19T14:16:57.216 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.216 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.216 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:57.216 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6809", 2024-08-19T14:16:57.216 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 526388985 2024-08-19T14:16:57.216 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:57.216 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:57.216 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.216 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-08-19T14:16:57.216 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:57.217 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.217 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:57.217 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6810", 2024-08-19T14:16:57.217 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 526388985 2024-08-19T14:16:57.217 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.217 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.217 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:57.217 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6811", 2024-08-19T14:16:57.217 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 526388985 2024-08-19T14:16:57.217 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:57.217 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:57.218 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.218 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-08-19T14:16:57.218 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:57.218 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.218 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:57.218 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6814", 2024-08-19T14:16:57.218 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 526388985 2024-08-19T14:16:57.218 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.218 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.218 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:57.218 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6815", 2024-08-19T14:16:57.218 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 526388985 2024-08-19T14:16:57.218 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:57.219 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:57.219 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.219 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-08-19T14:16:57.219 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:57.219 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.219 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:57.219 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6812", 2024-08-19T14:16:57.219 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 526388985 2024-08-19T14:16:57.219 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.219 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.219 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:57.219 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6813", 2024-08-19T14:16:57.219 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 526388985 2024-08-19T14:16:57.220 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:57.220 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:57.220 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.220 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.32:6809/526388985", 2024-08-19T14:16:57.220 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.32:6811/526388985", 2024-08-19T14:16:57.220 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.32:6815/526388985", 2024-08-19T14:16:57.220 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.32:6813/526388985", 2024-08-19T14:16:57.220 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-08-19T14:16:57.220 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-08-19T14:16:57.220 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-08-19T14:16:57.221 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:57.221 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.221 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.221 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 6, 2024-08-19T14:16:57.221 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "44aa8229-0268-49d7-bd57-3bf1f384e5e6", 2024-08-19T14:16:57.221 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-08-19T14:16:57.221 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 0, 2024-08-19T14:16:57.221 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 0, 2024-08-19T14:16:57.221 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-08-19T14:16:57.221 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-08-19T14:16:57.221 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-08-19T14:16:57.221 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 48, 2024-08-19T14:16:57.221 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 126, 2024-08-19T14:16:57.221 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-08-19T14:16:57.222 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-08-19T14:16:57.222 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-08-19T14:16:57.222 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:57.222 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.222 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:57.222 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6816", 2024-08-19T14:16:57.222 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1131035844 2024-08-19T14:16:57.222 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.222 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.223 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:57.223 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6817", 2024-08-19T14:16:57.223 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1131035844 2024-08-19T14:16:57.223 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:57.223 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:57.223 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.223 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-08-19T14:16:57.223 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:57.223 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.223 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:57.223 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6818", 2024-08-19T14:16:57.223 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1131035844 2024-08-19T14:16:57.223 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.224 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.224 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:57.224 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6819", 2024-08-19T14:16:57.224 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1131035844 2024-08-19T14:16:57.224 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:57.224 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:57.224 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.224 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-08-19T14:16:57.224 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:57.224 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.224 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:57.224 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6822", 2024-08-19T14:16:57.225 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1131035844 2024-08-19T14:16:57.225 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.225 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.225 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:57.225 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6823", 2024-08-19T14:16:57.225 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1131035844 2024-08-19T14:16:57.225 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:57.225 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:57.225 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.225 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-08-19T14:16:57.225 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:57.225 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.225 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:57.225 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6820", 2024-08-19T14:16:57.226 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1131035844 2024-08-19T14:16:57.226 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.226 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.226 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:57.226 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6821", 2024-08-19T14:16:57.226 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1131035844 2024-08-19T14:16:57.226 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:57.226 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:57.226 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.226 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.32:6817/1131035844", 2024-08-19T14:16:57.226 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.32:6819/1131035844", 2024-08-19T14:16:57.226 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.32:6823/1131035844", 2024-08-19T14:16:57.226 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.32:6821/1131035844", 2024-08-19T14:16:57.227 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-08-19T14:16:57.227 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-08-19T14:16:57.227 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-08-19T14:16:57.227 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:57.227 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.227 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.227 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 7, 2024-08-19T14:16:57.227 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "263433e5-70b6-4192-86ff-dc2ad95927eb", 2024-08-19T14:16:57.227 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-08-19T14:16:57.227 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-08-19T14:16:57.227 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 0.95001220703125, 2024-08-19T14:16:57.227 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-08-19T14:16:57.227 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-08-19T14:16:57.228 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-08-19T14:16:57.228 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 57, 2024-08-19T14:16:57.228 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 222, 2024-08-19T14:16:57.228 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-08-19T14:16:57.228 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-08-19T14:16:57.228 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-08-19T14:16:57.228 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:57.228 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.228 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:57.228 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6824", 2024-08-19T14:16:57.228 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2014939947 2024-08-19T14:16:57.229 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.229 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.229 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:57.229 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6825", 2024-08-19T14:16:57.229 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2014939947 2024-08-19T14:16:57.229 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:57.229 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:57.230 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.230 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-08-19T14:16:57.230 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:57.230 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.230 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:57.230 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6826", 2024-08-19T14:16:57.230 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2014939947 2024-08-19T14:16:57.231 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.231 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.231 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:57.231 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6827", 2024-08-19T14:16:57.231 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2014939947 2024-08-19T14:16:57.231 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:57.232 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:57.232 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.232 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-08-19T14:16:57.232 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:57.232 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.232 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:57.233 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6830", 2024-08-19T14:16:57.233 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2014939947 2024-08-19T14:16:57.233 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.233 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.233 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:57.233 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6831", 2024-08-19T14:16:57.234 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2014939947 2024-08-19T14:16:57.234 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:57.234 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:57.234 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.234 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-08-19T14:16:57.234 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:57.235 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.235 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:57.235 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6828", 2024-08-19T14:16:57.235 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2014939947 2024-08-19T14:16:57.235 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.235 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.236 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:57.236 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6829", 2024-08-19T14:16:57.236 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2014939947 2024-08-19T14:16:57.236 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:57.236 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:57.236 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.237 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.32:6825/2014939947", 2024-08-19T14:16:57.237 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.32:6827/2014939947", 2024-08-19T14:16:57.237 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.32:6831/2014939947", 2024-08-19T14:16:57.237 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.32:6829/2014939947", 2024-08-19T14:16:57.237 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-08-19T14:16:57.238 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-08-19T14:16:57.238 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-08-19T14:16:57.238 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:57.238 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.238 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.238 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 8, 2024-08-19T14:16:57.239 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "2bbcad36-0c74-4528-9fb9-a8812afa1662", 2024-08-19T14:16:57.239 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-08-19T14:16:57.239 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-08-19T14:16:57.239 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-08-19T14:16:57.239 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-08-19T14:16:57.240 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-08-19T14:16:57.240 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-08-19T14:16:57.240 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 65, 2024-08-19T14:16:57.240 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 222, 2024-08-19T14:16:57.240 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-08-19T14:16:57.240 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-08-19T14:16:57.241 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-08-19T14:16:57.241 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:57.241 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.241 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:57.241 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6800", 2024-08-19T14:16:57.241 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1445140892 2024-08-19T14:16:57.242 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.242 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.242 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:57.242 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6801", 2024-08-19T14:16:57.242 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1445140892 2024-08-19T14:16:57.243 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:57.243 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:57.243 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.243 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-08-19T14:16:57.243 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:57.244 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.244 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:57.244 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6802", 2024-08-19T14:16:57.244 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1445140892 2024-08-19T14:16:57.244 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.245 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.245 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:57.245 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6803", 2024-08-19T14:16:57.245 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1445140892 2024-08-19T14:16:57.245 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:57.246 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:57.246 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.246 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-08-19T14:16:57.246 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:57.246 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.246 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:57.247 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6806", 2024-08-19T14:16:57.247 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1445140892 2024-08-19T14:16:57.247 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.247 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.247 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:57.247 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6807", 2024-08-19T14:16:57.248 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1445140892 2024-08-19T14:16:57.248 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:57.248 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:57.248 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.248 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-08-19T14:16:57.249 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:57.249 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.249 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:57.249 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6804", 2024-08-19T14:16:57.249 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1445140892 2024-08-19T14:16:57.249 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.250 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.250 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:57.250 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6805", 2024-08-19T14:16:57.250 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1445140892 2024-08-19T14:16:57.250 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:57.250 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:57.251 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.251 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.45:6801/1445140892", 2024-08-19T14:16:57.251 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.45:6803/1445140892", 2024-08-19T14:16:57.251 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.45:6807/1445140892", 2024-08-19T14:16:57.251 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.45:6805/1445140892", 2024-08-19T14:16:57.251 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-08-19T14:16:57.252 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-08-19T14:16:57.252 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-08-19T14:16:57.252 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:57.252 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.252 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.252 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 9, 2024-08-19T14:16:57.253 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "c8ee4277-3bf9-46bf-9cc0-152c1c4b9508", 2024-08-19T14:16:57.253 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-08-19T14:16:57.253 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-08-19T14:16:57.253 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 0.95001220703125, 2024-08-19T14:16:57.253 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-08-19T14:16:57.254 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-08-19T14:16:57.254 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-08-19T14:16:57.254 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 74, 2024-08-19T14:16:57.254 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 222, 2024-08-19T14:16:57.254 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-08-19T14:16:57.254 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-08-19T14:16:57.255 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-08-19T14:16:57.255 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:57.255 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.255 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:57.255 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6808", 2024-08-19T14:16:57.256 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1213787169 2024-08-19T14:16:57.256 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.256 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.256 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:57.256 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6809", 2024-08-19T14:16:57.256 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1213787169 2024-08-19T14:16:57.256 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:57.257 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:57.257 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.257 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-08-19T14:16:57.257 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:57.257 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.257 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:57.257 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6810", 2024-08-19T14:16:57.258 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1213787169 2024-08-19T14:16:57.258 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.258 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.258 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:57.259 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6811", 2024-08-19T14:16:57.259 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1213787169 2024-08-19T14:16:57.259 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:57.259 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:57.259 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.259 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-08-19T14:16:57.259 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:57.259 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.259 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:57.260 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6814", 2024-08-19T14:16:57.260 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1213787169 2024-08-19T14:16:57.260 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.260 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.260 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:57.260 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6815", 2024-08-19T14:16:57.260 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1213787169 2024-08-19T14:16:57.261 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:57.261 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:57.261 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.261 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-08-19T14:16:57.261 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:57.261 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.261 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:57.261 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6812", 2024-08-19T14:16:57.261 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1213787169 2024-08-19T14:16:57.261 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.262 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.262 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:57.262 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6813", 2024-08-19T14:16:57.262 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1213787169 2024-08-19T14:16:57.262 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:57.262 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:57.262 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.262 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.45:6809/1213787169", 2024-08-19T14:16:57.263 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.45:6811/1213787169", 2024-08-19T14:16:57.263 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.45:6815/1213787169", 2024-08-19T14:16:57.263 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.45:6813/1213787169", 2024-08-19T14:16:57.263 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-08-19T14:16:57.263 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-08-19T14:16:57.263 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-08-19T14:16:57.263 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:57.263 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.263 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.263 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 10, 2024-08-19T14:16:57.263 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:16:57.265 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "29031395-5f5c-40f8-ac90-9f6ac48e5c9f", 2024-08-19T14:16:57.265 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-08-19T14:16:57.265 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 0, 2024-08-19T14:16:57.265 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 0, 2024-08-19T14:16:57.265 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-08-19T14:16:57.265 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-08-19T14:16:57.265 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-08-19T14:16:57.265 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 80, 2024-08-19T14:16:57.265 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 115, 2024-08-19T14:16:57.265 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-08-19T14:16:57.266 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-08-19T14:16:57.266 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-08-19T14:16:57.266 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:57.266 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.266 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:57.266 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6816", 2024-08-19T14:16:57.266 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2461824621 2024-08-19T14:16:57.266 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.266 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.266 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:57.266 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6817", 2024-08-19T14:16:57.266 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2461824621 2024-08-19T14:16:57.267 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:57.267 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:57.267 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.267 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-08-19T14:16:57.267 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:57.267 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.267 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:57.267 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6818", 2024-08-19T14:16:57.267 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2461824621 2024-08-19T14:16:57.267 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.267 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.267 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:57.267 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6819", 2024-08-19T14:16:57.267 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2461824621 2024-08-19T14:16:57.268 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:57.268 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:57.268 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.268 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-08-19T14:16:57.268 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:57.268 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.268 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:57.268 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6822", 2024-08-19T14:16:57.268 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2461824621 2024-08-19T14:16:57.268 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.268 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.268 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:57.268 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6823", 2024-08-19T14:16:57.268 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2461824621 2024-08-19T14:16:57.269 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:57.269 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:57.269 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.269 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-08-19T14:16:57.269 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:57.269 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.269 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:57.269 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6820", 2024-08-19T14:16:57.269 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2461824621 2024-08-19T14:16:57.269 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.269 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.270 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:57.270 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6821", 2024-08-19T14:16:57.270 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2461824621 2024-08-19T14:16:57.270 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:57.270 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:57.270 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.270 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.45:6817/2461824621", 2024-08-19T14:16:57.270 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.45:6819/2461824621", 2024-08-19T14:16:57.270 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.45:6823/2461824621", 2024-08-19T14:16:57.270 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.45:6821/2461824621", 2024-08-19T14:16:57.270 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-08-19T14:16:57.270 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-08-19T14:16:57.270 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-08-19T14:16:57.271 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:57.271 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.271 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.271 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 11, 2024-08-19T14:16:57.271 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "5ef0db4b-0fef-4ea0-a19b-d3677e082c29", 2024-08-19T14:16:57.271 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-08-19T14:16:57.271 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-08-19T14:16:57.271 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-08-19T14:16:57.271 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-08-19T14:16:57.271 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-08-19T14:16:57.271 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-08-19T14:16:57.271 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 86, 2024-08-19T14:16:57.271 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 222, 2024-08-19T14:16:57.272 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-08-19T14:16:57.272 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-08-19T14:16:57.272 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-08-19T14:16:57.272 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:57.272 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.272 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:57.272 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6824", 2024-08-19T14:16:57.272 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2438396298 2024-08-19T14:16:57.272 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.272 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.272 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:57.272 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6825", 2024-08-19T14:16:57.272 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2438396298 2024-08-19T14:16:57.272 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:57.273 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:57.273 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.273 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-08-19T14:16:57.273 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:57.273 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.273 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:57.273 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6826", 2024-08-19T14:16:57.273 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2438396298 2024-08-19T14:16:57.273 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.273 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.273 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:57.273 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6827", 2024-08-19T14:16:57.273 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2438396298 2024-08-19T14:16:57.273 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:57.274 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:57.274 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.274 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-08-19T14:16:57.274 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:57.274 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.274 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:57.274 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6830", 2024-08-19T14:16:57.274 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2438396298 2024-08-19T14:16:57.274 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.274 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.274 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:57.274 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6831", 2024-08-19T14:16:57.275 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2438396298 2024-08-19T14:16:57.275 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:57.275 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:57.275 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.275 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-08-19T14:16:57.275 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:16:57.275 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.275 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:16:57.275 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6828", 2024-08-19T14:16:57.275 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2438396298 2024-08-19T14:16:57.276 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.276 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.276 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:16:57.276 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6829", 2024-08-19T14:16:57.276 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2438396298 2024-08-19T14:16:57.276 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:57.276 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:57.276 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.276 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.45:6825/2438396298", 2024-08-19T14:16:57.276 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.45:6827/2438396298", 2024-08-19T14:16:57.276 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.45:6831/2438396298", 2024-08-19T14:16:57.276 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.45:6829/2438396298", 2024-08-19T14:16:57.276 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-08-19T14:16:57.277 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-08-19T14:16:57.277 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-08-19T14:16:57.277 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:57.277 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:57.277 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-08-19T14:16:57.277 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_xinfo": [ 2024-08-19T14:16:57.277 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.277 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 0, 2024-08-19T14:16:57.277 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-08-19T14:16:57.277 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-08-19T14:16:57.277 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-08-19T14:16:57.278 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-08-19T14:16:57.278 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-08-19T14:16:57.278 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-08-19T14:15:56.929307+0000", 2024-08-19T14:16:57.278 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-08-19T14:16:57.278 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.278 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.278 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 1, 2024-08-19T14:16:57.278 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-08-19T14:16:57.278 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-08-19T14:16:57.278 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-08-19T14:16:57.278 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-08-19T14:16:57.279 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-08-19T14:16:57.279 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-08-19T14:15:55.638039+0000", 2024-08-19T14:16:57.279 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-08-19T14:16:57.279 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.279 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.280 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 2, 2024-08-19T14:16:57.280 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-08-19T14:16:57.280 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-08-19T14:16:57.280 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-08-19T14:16:57.280 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-08-19T14:16:57.280 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 65536, 2024-08-19T14:16:57.280 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-08-19T14:15:58.904876+0000", 2024-08-19T14:16:57.280 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-08-19T14:16:57.280 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.280 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.280 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 3, 2024-08-19T14:16:57.280 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-08-19T14:16:57.281 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-08-19T14:16:57.281 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-08-19T14:16:57.281 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-08-19T14:16:57.281 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-08-19T14:16:57.281 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-08-19T14:16:04.911361+0000", 2024-08-19T14:16:57.281 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-08-19T14:16:57.281 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.281 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.281 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 4, 2024-08-19T14:16:57.281 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-08-19T14:16:57.281 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-08-19T14:16:57.281 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-08-19T14:16:57.281 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-08-19T14:16:57.281 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-08-19T14:16:57.282 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-08-19T14:16:03.945511+0000", 2024-08-19T14:16:57.282 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-08-19T14:16:57.282 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.282 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.282 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 5, 2024-08-19T14:16:57.282 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-08-19T14:16:57.282 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-08-19T14:16:57.282 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-08-19T14:16:57.282 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-08-19T14:16:57.282 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-08-19T14:16:57.282 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-08-19T14:15:49.417209+0000", 2024-08-19T14:16:57.282 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-08-19T14:16:57.282 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.283 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.283 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 6, 2024-08-19T14:16:57.283 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-08-19T14:16:57.283 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-08-19T14:16:57.283 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-08-19T14:16:57.283 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-08-19T14:16:57.283 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 65536, 2024-08-19T14:16:57.283 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-08-19T14:16:48.606221+0000", 2024-08-19T14:16:57.283 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-08-19T14:16:57.283 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.283 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.283 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 7, 2024-08-19T14:16:57.283 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-08-19T14:16:57.283 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-08-19T14:16:57.284 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-08-19T14:16:57.284 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-08-19T14:16:57.284 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-08-19T14:16:57.284 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-08-19T14:16:49.277037+0000", 2024-08-19T14:16:57.284 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-08-19T14:16:57.284 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.284 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.284 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 8, 2024-08-19T14:16:57.284 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-08-19T14:16:57.285 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-08-19T14:16:57.285 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-08-19T14:16:57.285 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-08-19T14:16:57.285 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-08-19T14:16:57.285 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-08-19T14:16:35.019990+0000", 2024-08-19T14:16:57.285 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-08-19T14:16:57.286 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.286 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.286 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 9, 2024-08-19T14:16:57.286 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-08-19T14:16:57.286 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-08-19T14:16:57.286 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-08-19T14:16:57.286 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-08-19T14:16:57.286 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-08-19T14:16:57.286 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-08-19T14:15:54.288635+0000", 2024-08-19T14:16:57.286 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-08-19T14:16:57.286 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.286 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.287 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 10, 2024-08-19T14:16:57.287 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-08-19T14:16:57.287 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-08-19T14:16:57.287 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-08-19T14:16:57.287 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-08-19T14:16:57.287 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 65536, 2024-08-19T14:16:57.287 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-08-19T14:15:57.551806+0000", 2024-08-19T14:16:57.287 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-08-19T14:16:57.287 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.287 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.287 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 11, 2024-08-19T14:16:57.288 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-08-19T14:16:57.288 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-08-19T14:16:57.288 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-08-19T14:16:57.288 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-08-19T14:16:57.288 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-08-19T14:16:57.288 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-08-19T14:16:45.469890+0000", 2024-08-19T14:16:57.288 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-08-19T14:16:57.288 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:57.288 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-08-19T14:16:57.288 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_upmap": [ 2024-08-19T14:16:57.288 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:16:57.288 INFO:teuthology.orchestra.run.smithi031.stdout: "pgid": "1.0", 2024-08-19T14:16:57.288 INFO:teuthology.orchestra.run.smithi031.stdout: "osds": [ 2024-08-19T14:16:57.288 INFO:teuthology.orchestra.run.smithi031.stdout: 2, 2024-08-19T14:16:57.289 INFO:teuthology.orchestra.run.smithi031.stdout: 9, 2024-08-19T14:16:57.289 INFO:teuthology.orchestra.run.smithi031.stdout: 0 2024-08-19T14:16:57.289 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:16:57.289 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:57.289 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-08-19T14:16:57.289 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_upmap_items": [], 2024-08-19T14:16:57.289 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_upmap_primaries": [], 2024-08-19T14:16:57.289 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_temp": [], 2024-08-19T14:16:57.289 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_temp": [], 2024-08-19T14:16:57.289 INFO:teuthology.orchestra.run.smithi031.stdout: "blocklist": { 2024-08-19T14:16:57.289 INFO:teuthology.orchestra.run.smithi031.stdout: "172.21.15.31:0/1451745969": "2024-08-20T14:06:13.660657+0000", 2024-08-19T14:16:57.289 INFO:teuthology.orchestra.run.smithi031.stdout: "172.21.15.31:0/1515698918": "2024-08-20T14:06:13.660657+0000", 2024-08-19T14:16:57.289 INFO:teuthology.orchestra.run.smithi031.stdout: "172.21.15.31:6801/3297539242": "2024-08-20T14:06:13.660657+0000", 2024-08-19T14:16:57.290 INFO:teuthology.orchestra.run.smithi031.stdout: "172.21.15.31:6800/3297539242": "2024-08-20T14:06:13.660657+0000", 2024-08-19T14:16:57.290 INFO:teuthology.orchestra.run.smithi031.stdout: "172.21.15.31:0/2217423818": "2024-08-20T14:05:48.827576+0000", 2024-08-19T14:16:57.290 INFO:teuthology.orchestra.run.smithi031.stdout: "172.21.15.31:0/3353498281": "2024-08-20T14:06:13.660657+0000", 2024-08-19T14:16:57.290 INFO:teuthology.orchestra.run.smithi031.stdout: "172.21.15.31:0/4013594220": "2024-08-20T14:05:48.827576+0000", 2024-08-19T14:16:57.290 INFO:teuthology.orchestra.run.smithi031.stdout: "172.21.15.31:0/1654518065": "2024-08-20T14:05:48.827576+0000", 2024-08-19T14:16:57.290 INFO:teuthology.orchestra.run.smithi031.stdout: "172.21.15.31:6801/2368821258": "2024-08-20T14:05:48.827576+0000", 2024-08-19T14:16:57.290 INFO:teuthology.orchestra.run.smithi031.stdout: "172.21.15.31:6800/2368821258": "2024-08-20T14:05:48.827576+0000" 2024-08-19T14:16:57.290 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.290 INFO:teuthology.orchestra.run.smithi031.stdout: "range_blocklist": {}, 2024-08-19T14:16:57.290 INFO:teuthology.orchestra.run.smithi031.stdout: "erasure_code_profiles": { 2024-08-19T14:16:57.290 INFO:teuthology.orchestra.run.smithi031.stdout: "default": { 2024-08-19T14:16:57.290 INFO:teuthology.orchestra.run.smithi031.stdout: "crush-failure-domain": "osd", 2024-08-19T14:16:57.291 INFO:teuthology.orchestra.run.smithi031.stdout: "k": "2", 2024-08-19T14:16:57.291 INFO:teuthology.orchestra.run.smithi031.stdout: "m": "1", 2024-08-19T14:16:57.291 INFO:teuthology.orchestra.run.smithi031.stdout: "plugin": "jerasure", 2024-08-19T14:16:57.291 INFO:teuthology.orchestra.run.smithi031.stdout: "technique": "reed_sol_van" 2024-08-19T14:16:57.291 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:57.291 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:16:57.291 INFO:teuthology.orchestra.run.smithi031.stdout: "removed_snaps_queue": [], 2024-08-19T14:16:57.291 INFO:teuthology.orchestra.run.smithi031.stdout: "new_removed_snaps": [], 2024-08-19T14:16:57.291 INFO:teuthology.orchestra.run.smithi031.stdout: "new_purged_snaps": [], 2024-08-19T14:16:57.291 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_node_flags": {}, 2024-08-19T14:16:57.291 INFO:teuthology.orchestra.run.smithi031.stdout: "device_class_flags": {}, 2024-08-19T14:16:57.291 INFO:teuthology.orchestra.run.smithi031.stdout: "stretch_mode": { 2024-08-19T14:16:57.291 INFO:teuthology.orchestra.run.smithi031.stdout: "stretch_mode_enabled": false, 2024-08-19T14:16:57.291 INFO:teuthology.orchestra.run.smithi031.stdout: "stretch_bucket_count": 0, 2024-08-19T14:16:57.292 INFO:teuthology.orchestra.run.smithi031.stdout: "degraded_stretch_mode": 0, 2024-08-19T14:16:57.292 INFO:teuthology.orchestra.run.smithi031.stdout: "recovering_stretch_mode": 0, 2024-08-19T14:16:57.292 INFO:teuthology.orchestra.run.smithi031.stdout: "stretch_mode_bucket": 0 2024-08-19T14:16:57.292 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:16:57.292 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:16:57.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:57.293 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:57.294 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:16:57.345 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:16:57.394 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:16:57.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:57.420 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:57.421 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:16:57.522 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:16:57.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:57.548 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:57.548 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:16:57.619 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:57 smithi031 ceph-mon[16336]: pgmap v533: 27 pgs: 3 creating+peering, 13 unknown, 11 active+clean; 577 KiB data, 1.2 GiB used, 803 GiB / 805 GiB avail 2024-08-19T14:16:57.620 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:57 smithi031 ceph-mon[16336]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-19T14:16:57.620 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:57 smithi031 ceph-mon[16336]: osdmap e223: 12 total, 12 up, 9 in 2024-08-19T14:16:57.620 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:57 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3714855523' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-08-19T14:16:57.649 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:16:57.673 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:57.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:57.674 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:16:57.733 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:57 smithi032 ceph-mon[20004]: pgmap v533: 27 pgs: 3 creating+peering, 13 unknown, 11 active+clean; 577 KiB data, 1.2 GiB used, 803 GiB / 805 GiB avail 2024-08-19T14:16:57.733 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:57 smithi032 ceph-mon[20004]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-19T14:16:57.733 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:57 smithi032 ceph-mon[20004]: osdmap e223: 12 total, 12 up, 9 in 2024-08-19T14:16:57.733 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:57 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3714855523' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-08-19T14:16:57.775 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:16:57.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:57 smithi045 ceph-mon[20143]: pgmap v533: 27 pgs: 3 creating+peering, 13 unknown, 11 active+clean; 577 KiB data, 1.2 GiB used, 803 GiB / 805 GiB avail 2024-08-19T14:16:57.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:57 smithi045 ceph-mon[20143]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-19T14:16:57.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:57 smithi045 ceph-mon[20143]: osdmap e223: 12 total, 12 up, 9 in 2024-08-19T14:16:57.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:57 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3714855523' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-08-19T14:16:57.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:57.799 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:57.799 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:16:57.836 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:16:57.837 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-08-19T14:16:57.837 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-08-19T14:16:57.837 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-08-19T14:16:57.837 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.837 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:63160 2.0 2:08542b0e:::benchmark_data_smithi045_46415_object63159:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-08-19T14:16:57.837 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:20.958791+0000", 2024-08-19T14:16:57.838 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 216.87593838999999, 2024-08-19T14:16:57.838 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.21296446799999999, 2024-08-19T14:16:57.838 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:16:57.838 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:16:57.838 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:16:57.838 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:16:57.839 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:16:57.839 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:16:57.839 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 63160 2024-08-19T14:16:57.839 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.839 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:16:57.840 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.840 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:16:57.840 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.958791+0000", 2024-08-19T14:16:57.840 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:57.840 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.840 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.841 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:16:57.841 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.958791+0000", 2024-08-19T14:16:57.841 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:57.841 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.841 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.841 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:16:57.842 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.958792+0000", 2024-08-19T14:16:57.842 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.514e-06 2024-08-19T14:16:57.842 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.842 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.842 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:16:57.842 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.958829+0000", 2024-08-19T14:16:57.843 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6594999999999998e-05 2024-08-19T14:16:57.843 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.843 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.843 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:16:57.843 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.958845+0000", 2024-08-19T14:16:57.843 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6571000000000001e-05 2024-08-19T14:16:57.844 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.844 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.844 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:16:57.844 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.958854+0000", 2024-08-19T14:16:57.844 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.4810000000000006e-06 2024-08-19T14:16:57.844 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.845 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.845 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:16:57.846 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.960591+0000", 2024-08-19T14:16:57.846 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0017371929999999999 2024-08-19T14:16:57.846 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.846 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.846 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:16:57.847 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.960793+0000", 2024-08-19T14:16:57.847 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00020158200000000001 2024-08-19T14:16:57.847 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.847 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.847 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 9,11", 2024-08-19T14:16:57.847 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.961006+0000", 2024-08-19T14:16:57.848 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000213169 2024-08-19T14:16:57.848 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.848 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.848 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:16:57.848 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.962896+0000", 2024-08-19T14:16:57.848 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0018897460000000001 2024-08-19T14:16:57.848 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.848 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.848 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:57.849 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.965630+0000", 2024-08-19T14:16:57.849 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.002733961 2024-08-19T14:16:57.849 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.849 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.849 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:57.849 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.170985+0000", 2024-08-19T14:16:57.849 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.205355648 2024-08-19T14:16:57.849 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.849 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.850 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:16:57.850 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.171055+0000", 2024-08-19T14:16:57.850 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9776000000000004e-05 2024-08-19T14:16:57.850 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.850 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.850 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:16:57.850 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.171755+0000", 2024-08-19T14:16:57.850 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00070023199999999996 2024-08-19T14:16:57.850 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:57.850 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:16:57.850 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:57.850 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.850 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.851 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:63163 2.8 2:1e3c608e:::benchmark_data_smithi045_46415_object63162:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-08-19T14:16:57.851 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:20.961739+0000", 2024-08-19T14:16:57.851 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 216.872990028, 2024-08-19T14:16:57.851 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.209352172, 2024-08-19T14:16:57.851 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:16:57.851 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:16:57.851 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:16:57.851 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:16:57.852 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:16:57.852 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:16:57.852 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 63163 2024-08-19T14:16:57.852 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.852 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:16:57.852 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.852 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:16:57.852 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.961739+0000", 2024-08-19T14:16:57.852 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:57.852 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.852 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.852 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:16:57.852 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.961739+0000", 2024-08-19T14:16:57.852 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:57.853 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.853 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.853 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:16:57.853 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.961740+0000", 2024-08-19T14:16:57.853 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2049999999999999e-06 2024-08-19T14:16:57.853 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.853 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.853 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:16:57.853 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.961818+0000", 2024-08-19T14:16:57.853 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7275999999999997e-05 2024-08-19T14:16:57.853 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.853 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.854 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:16:57.854 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.961828+0000", 2024-08-19T14:16:57.854 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0407999999999999e-05 2024-08-19T14:16:57.854 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.854 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.854 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:16:57.854 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.961835+0000", 2024-08-19T14:16:57.854 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5089999999999996e-06 2024-08-19T14:16:57.854 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.854 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.855 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:16:57.855 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.961891+0000", 2024-08-19T14:16:57.855 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5785000000000002e-05 2024-08-19T14:16:57.855 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.855 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.855 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:16:57.855 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.962014+0000", 2024-08-19T14:16:57.855 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012311200000000001 2024-08-19T14:16:57.855 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.855 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.855 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0,9", 2024-08-19T14:16:57.856 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.962141+0000", 2024-08-19T14:16:57.856 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012648 2024-08-19T14:16:57.856 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.856 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.856 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:57.856 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.966506+0000", 2024-08-19T14:16:57.857 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0043654719999999996 2024-08-19T14:16:57.857 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.857 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.857 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:16:57.857 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.968079+0000", 2024-08-19T14:16:57.857 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001572316 2024-08-19T14:16:57.857 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.857 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.858 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:57.858 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.170964+0000", 2024-08-19T14:16:57.858 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20288556799999999 2024-08-19T14:16:57.858 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.858 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.858 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:16:57.858 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.171048+0000", 2024-08-19T14:16:57.858 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.3412999999999996e-05 2024-08-19T14:16:57.858 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.858 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.859 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:16:57.859 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.171091+0000", 2024-08-19T14:16:57.859 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3628000000000003e-05 2024-08-19T14:16:57.859 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:57.859 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:16:57.859 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:57.859 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.859 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.859 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:63175 2.0 2:031c8466:::benchmark_data_smithi045_46415_object63174:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-08-19T14:16:57.859 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:20.969904+0000", 2024-08-19T14:16:57.859 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 216.864825548, 2024-08-19T14:16:57.859 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20268109100000001, 2024-08-19T14:16:57.860 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:16:57.860 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:16:57.860 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:16:57.861 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:16:57.861 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:16:57.861 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:16:57.861 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 63175 2024-08-19T14:16:57.861 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.861 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:16:57.861 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.861 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:16:57.861 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.969904+0000", 2024-08-19T14:16:57.862 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:57.862 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.862 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.862 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:16:57.862 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.969904+0000", 2024-08-19T14:16:57.862 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:57.862 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.862 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.863 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:16:57.863 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.969905+0000", 2024-08-19T14:16:57.863 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7969999999999999e-06 2024-08-19T14:16:57.863 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.863 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.863 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:16:57.863 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.969970+0000", 2024-08-19T14:16:57.863 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4850000000000004e-05 2024-08-19T14:16:57.863 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.863 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.863 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:16:57.863 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.969995+0000", 2024-08-19T14:16:57.863 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5168999999999999e-05 2024-08-19T14:16:57.864 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.864 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.864 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:16:57.864 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.970003+0000", 2024-08-19T14:16:57.864 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3939999999999998e-06 2024-08-19T14:16:57.864 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.864 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.864 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:16:57.864 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.971722+0000", 2024-08-19T14:16:57.864 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0017196220000000001 2024-08-19T14:16:57.864 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.864 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.864 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:16:57.864 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.971840+0000", 2024-08-19T14:16:57.865 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000117261 2024-08-19T14:16:57.865 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.865 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.865 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 9,11", 2024-08-19T14:16:57.865 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.972030+0000", 2024-08-19T14:16:57.865 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00018997799999999999 2024-08-19T14:16:57.865 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.865 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.865 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:57.865 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.976418+0000", 2024-08-19T14:16:57.865 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0043884140000000002 2024-08-19T14:16:57.865 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.865 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.865 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:16:57.866 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.976439+0000", 2024-08-19T14:16:57.866 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0979999999999999e-05 2024-08-19T14:16:57.866 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.866 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.866 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:57.867 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.171818+0000", 2024-08-19T14:16:57.867 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.19537907500000001 2024-08-19T14:16:57.867 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.867 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.867 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:16:57.867 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.171880+0000", 2024-08-19T14:16:57.867 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1791999999999994e-05 2024-08-19T14:16:57.867 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.867 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.867 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:16:57.868 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.172585+0000", 2024-08-19T14:16:57.868 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00070475900000000001 2024-08-19T14:16:57.868 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:57.868 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:16:57.868 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:57.868 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.868 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.868 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:63177 2.8 2:15a25ebc:::benchmark_data_smithi045_46415_object63176:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-08-19T14:16:57.868 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:20.970729+0000", 2024-08-19T14:16:57.868 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 216.86400051999999, 2024-08-19T14:16:57.868 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.200497493, 2024-08-19T14:16:57.868 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:16:57.868 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:16:57.869 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:16:57.869 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:16:57.869 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:16:57.869 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:16:57.869 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 63177 2024-08-19T14:16:57.869 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.869 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:16:57.869 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.869 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:16:57.869 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.970729+0000", 2024-08-19T14:16:57.869 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:57.869 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.869 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.869 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:16:57.869 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.970729+0000", 2024-08-19T14:16:57.870 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:57.870 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.870 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.870 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:16:57.870 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.970730+0000", 2024-08-19T14:16:57.870 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.02e-06 2024-08-19T14:16:57.870 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.870 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.870 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:16:57.870 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.970816+0000", 2024-08-19T14:16:57.870 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6040999999999998e-05 2024-08-19T14:16:57.870 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.870 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.871 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:16:57.871 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.970826+0000", 2024-08-19T14:16:57.871 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0361e-05 2024-08-19T14:16:57.871 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.871 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.871 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:16:57.871 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.970831+0000", 2024-08-19T14:16:57.871 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9459999999999997e-06 2024-08-19T14:16:57.871 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.872 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.872 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:16:57.872 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.971716+0000", 2024-08-19T14:16:57.872 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00088531199999999997 2024-08-19T14:16:57.872 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.872 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.873 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:16:57.873 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.971839+0000", 2024-08-19T14:16:57.873 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012251499999999999 2024-08-19T14:16:57.873 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.873 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.874 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0,9", 2024-08-19T14:16:57.874 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.972011+0000", 2024-08-19T14:16:57.874 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000172244 2024-08-19T14:16:57.875 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.875 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.875 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:57.875 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.975032+0000", 2024-08-19T14:16:57.875 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.003021195 2024-08-19T14:16:57.875 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.876 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.876 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:16:57.876 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.976447+0000", 2024-08-19T14:16:57.876 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001415152 2024-08-19T14:16:57.876 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.876 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.876 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:57.876 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.171136+0000", 2024-08-19T14:16:57.877 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.19468817199999999 2024-08-19T14:16:57.877 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.877 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.877 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:16:57.877 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.171189+0000", 2024-08-19T14:16:57.877 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3505999999999999e-05 2024-08-19T14:16:57.878 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.878 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.878 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:16:57.878 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.171226+0000", 2024-08-19T14:16:57.878 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7029000000000001e-05 2024-08-19T14:16:57.878 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:57.879 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:16:57.879 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:57.879 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.879 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.879 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:63179 2.0 2:034b6ab5:::benchmark_data_smithi045_46415_object63178:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-08-19T14:16:57.879 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:20.971796+0000", 2024-08-19T14:16:57.879 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 216.86293353100001, 2024-08-19T14:16:57.880 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20158538500000001, 2024-08-19T14:16:57.880 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:16:57.880 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:16:57.880 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:16:57.880 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:16:57.880 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:16:57.880 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:16:57.881 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 63179 2024-08-19T14:16:57.881 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.881 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:16:57.881 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.881 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:16:57.881 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.971796+0000", 2024-08-19T14:16:57.881 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:57.881 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.881 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.881 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:16:57.881 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.971796+0000", 2024-08-19T14:16:57.881 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:57.881 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.881 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.882 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:16:57.882 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.971796+0000", 2024-08-19T14:16:57.883 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.2399999999999996e-07 2024-08-19T14:16:57.883 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.883 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.883 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:16:57.883 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.971857+0000", 2024-08-19T14:16:57.883 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0863000000000003e-05 2024-08-19T14:16:57.884 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.884 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.884 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:16:57.884 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.971867+0000", 2024-08-19T14:16:57.884 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.5619999999999993e-06 2024-08-19T14:16:57.885 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.885 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.885 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:16:57.885 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.971874+0000", 2024-08-19T14:16:57.885 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.5989999999999998e-06 2024-08-19T14:16:57.885 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.885 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.885 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:16:57.885 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.973576+0000", 2024-08-19T14:16:57.885 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001702928 2024-08-19T14:16:57.886 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.886 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.886 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:16:57.886 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.973668+0000", 2024-08-19T14:16:57.886 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.1094000000000006e-05 2024-08-19T14:16:57.886 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.886 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.886 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 9,11", 2024-08-19T14:16:57.886 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.973776+0000", 2024-08-19T14:16:57.887 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000108453 2024-08-19T14:16:57.887 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.887 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.887 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:57.887 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.976463+0000", 2024-08-19T14:16:57.887 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0026873999999999999 2024-08-19T14:16:57.887 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.887 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.888 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:16:57.888 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.979780+0000", 2024-08-19T14:16:57.888 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0033170740000000002 2024-08-19T14:16:57.888 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.888 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.888 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:57.888 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.172645+0000", 2024-08-19T14:16:57.888 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.192864063 2024-08-19T14:16:57.889 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.889 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.889 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:16:57.889 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.172698+0000", 2024-08-19T14:16:57.889 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3647000000000003e-05 2024-08-19T14:16:57.889 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.889 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.889 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:16:57.889 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.173381+0000", 2024-08-19T14:16:57.889 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00068277799999999999 2024-08-19T14:16:57.890 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:57.890 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:16:57.890 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:57.890 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.890 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.890 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:63180 2.0 2:09d38ed3:::benchmark_data_smithi045_46415_object63179:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-08-19T14:16:57.890 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:20.971911+0000", 2024-08-19T14:16:57.891 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 216.862817863, 2024-08-19T14:16:57.891 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20210957399999999, 2024-08-19T14:16:57.891 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:16:57.891 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:16:57.891 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:16:57.891 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:16:57.892 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:16:57.892 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:16:57.892 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 63180 2024-08-19T14:16:57.892 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.892 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:16:57.892 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.892 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:16:57.892 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.971911+0000", 2024-08-19T14:16:57.893 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:57.893 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.893 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.893 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:16:57.893 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.971911+0000", 2024-08-19T14:16:57.893 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:57.893 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.893 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.893 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:16:57.893 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.971915+0000", 2024-08-19T14:16:57.893 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.07e-06 2024-08-19T14:16:57.893 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.893 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.893 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:16:57.894 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.971998+0000", 2024-08-19T14:16:57.894 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.2995000000000001e-05 2024-08-19T14:16:57.894 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.894 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.894 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:16:57.894 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.972011+0000", 2024-08-19T14:16:57.894 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2262999999999999e-05 2024-08-19T14:16:57.894 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.894 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.894 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:16:57.894 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.972017+0000", 2024-08-19T14:16:57.894 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4400000000000002e-06 2024-08-19T14:16:57.894 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.895 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.895 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:16:57.895 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.975045+0000", 2024-08-19T14:16:57.895 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0030280440000000001 2024-08-19T14:16:57.895 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.895 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.895 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:16:57.895 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.975155+0000", 2024-08-19T14:16:57.895 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00010984699999999999 2024-08-19T14:16:57.895 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.895 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.895 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 9,11", 2024-08-19T14:16:57.895 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.975247+0000", 2024-08-19T14:16:57.896 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.1947999999999994e-05 2024-08-19T14:16:57.896 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.896 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.896 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:57.896 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.979753+0000", 2024-08-19T14:16:57.896 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0045062119999999999 2024-08-19T14:16:57.896 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.896 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.896 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:16:57.896 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.979791+0000", 2024-08-19T14:16:57.896 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7511000000000001e-05 2024-08-19T14:16:57.896 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.896 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.896 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:57.896 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.173451+0000", 2024-08-19T14:16:57.897 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.19366074699999999 2024-08-19T14:16:57.897 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.897 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.897 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:16:57.897 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.173513+0000", 2024-08-19T14:16:57.897 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1480000000000001e-05 2024-08-19T14:16:57.897 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.897 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.897 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:16:57.898 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.174021+0000", 2024-08-19T14:16:57.898 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00050801700000000004 2024-08-19T14:16:57.898 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:57.898 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:16:57.898 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:57.898 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.898 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.898 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:63181 2.8 2:10ca0581:::benchmark_data_smithi045_46415_object63180:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-08-19T14:16:57.898 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:20.972411+0000", 2024-08-19T14:16:57.898 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 216.86231856200001, 2024-08-19T14:16:57.898 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20026014, 2024-08-19T14:16:57.898 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:16:57.898 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:16:57.899 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:16:57.899 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:16:57.899 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:16:57.899 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:16:57.899 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 63181 2024-08-19T14:16:57.899 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.899 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:16:57.899 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.899 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:16:57.899 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.972411+0000", 2024-08-19T14:16:57.899 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:57.900 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.900 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.900 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:16:57.900 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.972411+0000", 2024-08-19T14:16:57.900 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:57.900 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.901 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.901 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:16:57.901 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.972411+0000", 2024-08-19T14:16:57.901 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.3500000000000005e-07 2024-08-19T14:16:57.901 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.901 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.902 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:16:57.902 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.972473+0000", 2024-08-19T14:16:57.902 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1877000000000004e-05 2024-08-19T14:16:57.902 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.902 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.903 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:16:57.903 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.972485+0000", 2024-08-19T14:16:57.903 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1620999999999999e-05 2024-08-19T14:16:57.903 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.903 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.903 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:16:57.904 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.972492+0000", 2024-08-19T14:16:57.904 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9840000000000004e-06 2024-08-19T14:16:57.904 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.904 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.904 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:16:57.905 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.976487+0000", 2024-08-19T14:16:57.905 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0039953300000000001 2024-08-19T14:16:57.905 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.905 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.905 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:16:57.905 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.976598+0000", 2024-08-19T14:16:57.906 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000111099 2024-08-19T14:16:57.906 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.906 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.906 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0,9", 2024-08-19T14:16:57.906 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.976732+0000", 2024-08-19T14:16:57.906 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000134029 2024-08-19T14:16:57.907 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.907 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.907 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:16:57.907 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.979797+0000", 2024-08-19T14:16:57.907 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0030645109999999998 2024-08-19T14:16:57.907 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.908 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.908 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:57.908 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:20.979913+0000", 2024-08-19T14:16:57.908 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000116496 2024-08-19T14:16:57.909 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.909 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.909 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:57.909 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.172616+0000", 2024-08-19T14:16:57.909 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.19270316600000001 2024-08-19T14:16:57.909 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.910 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.910 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:16:57.910 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.172649+0000", 2024-08-19T14:16:57.910 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2589000000000003e-05 2024-08-19T14:16:57.910 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.910 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.911 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:16:57.911 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.172671+0000", 2024-08-19T14:16:57.911 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1603e-05 2024-08-19T14:16:57.911 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:57.911 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:16:57.912 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:57.912 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.912 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.912 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:63205 2.8 2:16572d7a:::benchmark_data_smithi045_46415_object63204:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-08-19T14:16:57.912 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:21.002885+0000", 2024-08-19T14:16:57.912 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 216.831844434, 2024-08-19T14:16:57.913 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.17128316199999999, 2024-08-19T14:16:57.913 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:16:57.913 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:16:57.913 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:16:57.913 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:16:57.913 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:16:57.913 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:16:57.913 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 63205 2024-08-19T14:16:57.913 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.913 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:16:57.913 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.913 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:16:57.914 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.002885+0000", 2024-08-19T14:16:57.914 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:57.914 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.914 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.915 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:16:57.915 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.002885+0000", 2024-08-19T14:16:57.915 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:57.915 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.915 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.915 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:16:57.915 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.002887+0000", 2024-08-19T14:16:57.915 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2139999999999999e-06 2024-08-19T14:16:57.915 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.915 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.915 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:16:57.915 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.002928+0000", 2024-08-19T14:16:57.916 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0703e-05 2024-08-19T14:16:57.916 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.916 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.916 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:16:57.916 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.002940+0000", 2024-08-19T14:16:57.916 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2831e-05 2024-08-19T14:16:57.916 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.916 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.916 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:16:57.916 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.002948+0000", 2024-08-19T14:16:57.916 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7740000000000004e-06 2024-08-19T14:16:57.916 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.916 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.916 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:16:57.917 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.003000+0000", 2024-08-19T14:16:57.917 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1483000000000002e-05 2024-08-19T14:16:57.917 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.917 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.917 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:16:57.917 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.003090+0000", 2024-08-19T14:16:57.918 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.0606000000000006e-05 2024-08-19T14:16:57.918 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.918 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.918 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0,9", 2024-08-19T14:16:57.919 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.003180+0000", 2024-08-19T14:16:57.919 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.9977999999999998e-05 2024-08-19T14:16:57.919 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.919 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.919 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:57.919 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.005559+0000", 2024-08-19T14:16:57.919 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0023788870000000001 2024-08-19T14:16:57.919 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.919 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.919 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:16:57.919 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.005623+0000", 2024-08-19T14:16:57.919 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4331000000000005e-05 2024-08-19T14:16:57.919 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.919 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.920 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:57.920 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.174071+0000", 2024-08-19T14:16:57.920 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.16844759200000001 2024-08-19T14:16:57.920 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.920 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.920 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:16:57.920 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.174125+0000", 2024-08-19T14:16:57.920 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3953000000000003e-05 2024-08-19T14:16:57.920 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.920 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.920 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:16:57.920 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.174168+0000", 2024-08-19T14:16:57.920 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2809999999999998e-05 2024-08-19T14:16:57.920 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:57.921 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:16:57.921 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:57.921 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.921 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.921 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:63230 2.0 2:025360f7:::benchmark_data_smithi045_46415_object63229:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-08-19T14:16:57.921 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:21.036099+0000", 2024-08-19T14:16:57.921 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 216.79862968800001, 2024-08-19T14:16:57.921 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.14053950100000001, 2024-08-19T14:16:57.921 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:16:57.921 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:16:57.921 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:16:57.921 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:16:57.922 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:16:57.922 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:16:57.922 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 63230 2024-08-19T14:16:57.922 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.922 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:16:57.923 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.923 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:16:57.923 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.036099+0000", 2024-08-19T14:16:57.923 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:57.923 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.923 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.923 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:16:57.923 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.036099+0000", 2024-08-19T14:16:57.923 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:57.923 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.924 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.924 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:16:57.924 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.036101+0000", 2024-08-19T14:16:57.924 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2279999999999999e-06 2024-08-19T14:16:57.924 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.924 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.924 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:16:57.924 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.036163+0000", 2024-08-19T14:16:57.924 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1940999999999995e-05 2024-08-19T14:16:57.924 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.924 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.924 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:16:57.924 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.036182+0000", 2024-08-19T14:16:57.925 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9686000000000001e-05 2024-08-19T14:16:57.925 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.925 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.925 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:16:57.925 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.036189+0000", 2024-08-19T14:16:57.925 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6640000000000001e-06 2024-08-19T14:16:57.925 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.925 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.925 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:16:57.925 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.037060+0000", 2024-08-19T14:16:57.925 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00087146399999999996 2024-08-19T14:16:57.925 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.925 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.925 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:16:57.926 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.037212+0000", 2024-08-19T14:16:57.926 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00015186400000000001 2024-08-19T14:16:57.926 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.926 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.926 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 9,11", 2024-08-19T14:16:57.926 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.037383+0000", 2024-08-19T14:16:57.926 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000170457 2024-08-19T14:16:57.926 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.926 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.926 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:16:57.926 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.039083+0000", 2024-08-19T14:16:57.926 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0016998 2024-08-19T14:16:57.926 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.927 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.927 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:57.927 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.041514+0000", 2024-08-19T14:16:57.927 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.002431519 2024-08-19T14:16:57.927 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.927 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.927 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:57.927 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.175973+0000", 2024-08-19T14:16:57.927 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.134458932 2024-08-19T14:16:57.927 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.927 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.927 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:16:57.927 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.176028+0000", 2024-08-19T14:16:57.928 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5036e-05 2024-08-19T14:16:57.928 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.928 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.928 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:16:57.928 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.176639+0000", 2024-08-19T14:16:57.928 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00061090999999999999 2024-08-19T14:16:57.928 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:57.928 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:16:57.928 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:57.928 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.928 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.928 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:63231 2.8 2:1da1a88e:::benchmark_data_smithi045_46415_object63230:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-08-19T14:16:57.928 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:21.036208+0000", 2024-08-19T14:16:57.928 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 216.79852065899999, 2024-08-19T14:16:57.929 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.13809454500000001, 2024-08-19T14:16:57.929 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:16:57.929 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:16:57.929 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:16:57.929 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:16:57.929 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:16:57.929 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:16:57.929 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 63231 2024-08-19T14:16:57.929 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.929 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:16:57.929 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.929 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:16:57.929 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.036208+0000", 2024-08-19T14:16:57.929 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:57.930 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.930 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.930 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:16:57.930 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.036208+0000", 2024-08-19T14:16:57.930 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:57.930 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.930 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.930 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:16:57.930 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.036209+0000", 2024-08-19T14:16:57.930 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4000000000000002e-07 2024-08-19T14:16:57.930 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.930 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.930 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:16:57.931 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.036233+0000", 2024-08-19T14:16:57.931 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4573999999999999e-05 2024-08-19T14:16:57.931 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.931 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.931 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:16:57.931 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.036240+0000", 2024-08-19T14:16:57.931 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4110000000000001e-06 2024-08-19T14:16:57.931 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.931 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.931 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:16:57.931 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.036244+0000", 2024-08-19T14:16:57.931 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9129999999999996e-06 2024-08-19T14:16:57.931 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.931 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.932 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:16:57.932 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.037035+0000", 2024-08-19T14:16:57.932 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00079092500000000005 2024-08-19T14:16:57.932 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.932 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.932 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:16:57.932 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.037123+0000", 2024-08-19T14:16:57.932 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.8616999999999995e-05 2024-08-19T14:16:57.932 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.932 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.932 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0,9", 2024-08-19T14:16:57.932 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.037210+0000", 2024-08-19T14:16:57.932 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6300999999999999e-05 2024-08-19T14:16:57.932 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.933 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.933 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:16:57.933 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.038951+0000", 2024-08-19T14:16:57.933 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001741151 2024-08-19T14:16:57.933 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.933 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.933 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:57.933 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.039784+0000", 2024-08-19T14:16:57.933 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00083278900000000001 2024-08-19T14:16:57.933 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.933 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.933 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:57.933 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.174204+0000", 2024-08-19T14:16:57.933 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.134420768 2024-08-19T14:16:57.934 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.934 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.934 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:16:57.934 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.174261+0000", 2024-08-19T14:16:57.934 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6947000000000002e-05 2024-08-19T14:16:57.934 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.934 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.934 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:16:57.934 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.174303+0000", 2024-08-19T14:16:57.934 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1709000000000003e-05 2024-08-19T14:16:57.934 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:57.935 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:16:57.935 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:57.935 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.935 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.935 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:63245 2.0 2:0a3678e6:::benchmark_data_smithi045_46415_object63244:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-08-19T14:16:57.935 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:21.051568+0000", 2024-08-19T14:16:57.935 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 216.78316137100001, 2024-08-19T14:16:57.935 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.12567051500000001, 2024-08-19T14:16:57.935 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:16:57.935 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:16:57.935 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:16:57.936 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:16:57.936 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:16:57.936 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:16:57.936 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 63245 2024-08-19T14:16:57.936 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.936 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:16:57.936 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.936 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:16:57.936 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.051568+0000", 2024-08-19T14:16:57.936 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:57.936 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.936 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.936 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:16:57.936 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.051568+0000", 2024-08-19T14:16:57.937 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:57.937 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.937 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.937 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:16:57.937 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.051569+0000", 2024-08-19T14:16:57.937 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2190000000000001e-06 2024-08-19T14:16:57.937 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.937 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.937 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:16:57.937 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.051592+0000", 2024-08-19T14:16:57.937 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3391e-05 2024-08-19T14:16:57.937 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.937 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.937 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:16:57.937 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.051603+0000", 2024-08-19T14:16:57.938 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0879e-05 2024-08-19T14:16:57.938 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.938 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.938 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:16:57.938 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.051608+0000", 2024-08-19T14:16:57.938 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.933e-06 2024-08-19T14:16:57.938 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.938 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.938 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:16:57.938 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.052675+0000", 2024-08-19T14:16:57.938 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001066621 2024-08-19T14:16:57.938 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.938 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.939 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:16:57.939 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.052841+0000", 2024-08-19T14:16:57.939 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00016639700000000001 2024-08-19T14:16:57.939 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.939 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.939 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 9,11", 2024-08-19T14:16:57.939 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.053037+0000", 2024-08-19T14:16:57.939 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00019577200000000001 2024-08-19T14:16:57.939 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.939 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.939 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:57.939 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.059090+0000", 2024-08-19T14:16:57.939 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0060532290000000002 2024-08-19T14:16:57.939 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.940 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.940 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:16:57.940 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.059156+0000", 2024-08-19T14:16:57.940 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6221000000000002e-05 2024-08-19T14:16:57.940 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.940 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.940 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:57.940 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.176685+0000", 2024-08-19T14:16:57.940 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.11752820899999999 2024-08-19T14:16:57.940 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.940 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.940 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:16:57.941 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.176716+0000", 2024-08-19T14:16:57.941 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1223000000000002e-05 2024-08-19T14:16:57.941 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.941 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.941 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:16:57.941 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.177238+0000", 2024-08-19T14:16:57.941 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00052242100000000002 2024-08-19T14:16:57.941 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:57.941 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:16:57.941 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:57.941 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.941 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.941 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:63247 2.0 2:0918bab6:::benchmark_data_smithi045_46415_object63246:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-08-19T14:16:57.942 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:21.052340+0000", 2024-08-19T14:16:57.942 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 216.782389486, 2024-08-19T14:16:57.942 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.12561350199999999, 2024-08-19T14:16:57.942 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:16:57.942 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:16:57.942 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:16:57.942 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:16:57.942 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:16:57.942 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:16:57.942 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 63247 2024-08-19T14:16:57.942 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.942 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:16:57.942 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.942 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:16:57.943 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.052340+0000", 2024-08-19T14:16:57.943 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:57.943 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.943 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.943 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:16:57.943 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.052340+0000", 2024-08-19T14:16:57.943 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:57.943 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.943 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.943 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:16:57.943 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.052340+0000", 2024-08-19T14:16:57.943 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5099999999999999e-07 2024-08-19T14:16:57.943 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.944 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.944 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:16:57.944 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.052371+0000", 2024-08-19T14:16:57.944 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1038000000000003e-05 2024-08-19T14:16:57.944 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.944 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.944 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:16:57.944 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.052382+0000", 2024-08-19T14:16:57.944 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0638999999999999e-05 2024-08-19T14:16:57.944 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.944 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.944 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:16:57.944 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.052387+0000", 2024-08-19T14:16:57.944 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.164e-06 2024-08-19T14:16:57.945 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.945 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.945 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:16:57.945 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.054907+0000", 2024-08-19T14:16:57.945 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0025194649999999998 2024-08-19T14:16:57.945 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.945 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.945 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:16:57.945 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.055057+0000", 2024-08-19T14:16:57.945 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000150607 2024-08-19T14:16:57.945 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.946 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.946 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 9,11", 2024-08-19T14:16:57.946 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.055161+0000", 2024-08-19T14:16:57.946 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000103644 2024-08-19T14:16:57.946 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.946 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.946 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:57.946 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.059147+0000", 2024-08-19T14:16:57.946 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0039857929999999996 2024-08-19T14:16:57.946 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.946 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.947 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:16:57.947 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.059160+0000", 2024-08-19T14:16:57.947 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3441000000000001e-05 2024-08-19T14:16:57.947 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.947 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.947 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:57.947 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.177310+0000", 2024-08-19T14:16:57.947 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.118149858 2024-08-19T14:16:57.947 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.947 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.947 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:16:57.947 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.177368+0000", 2024-08-19T14:16:57.947 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7633000000000002e-05 2024-08-19T14:16:57.947 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.948 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.948 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:16:57.948 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.177953+0000", 2024-08-19T14:16:57.948 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00058546900000000003 2024-08-19T14:16:57.948 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:57.948 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:16:57.948 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:57.948 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.948 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.948 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:63250 2.8 2:1918e081:::benchmark_data_smithi045_46415_object63249:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-08-19T14:16:57.948 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:21.056840+0000", 2024-08-19T14:16:57.948 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 216.77788913200001, 2024-08-19T14:16:57.948 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.119880845, 2024-08-19T14:16:57.949 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:16:57.949 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:16:57.949 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:16:57.949 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:16:57.949 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:16:57.949 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:16:57.949 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 63250 2024-08-19T14:16:57.949 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.949 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:16:57.949 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.949 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:16:57.949 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.056840+0000", 2024-08-19T14:16:57.949 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:57.949 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.950 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.950 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:16:57.950 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.056840+0000", 2024-08-19T14:16:57.950 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:57.950 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.950 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.950 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:16:57.950 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.056841+0000", 2024-08-19T14:16:57.950 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4360000000000001e-06 2024-08-19T14:16:57.950 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.950 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.950 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:16:57.950 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.056905+0000", 2024-08-19T14:16:57.951 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3677999999999996e-05 2024-08-19T14:16:57.951 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.951 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.951 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:16:57.951 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.056919+0000", 2024-08-19T14:16:57.951 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3556e-05 2024-08-19T14:16:57.951 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.951 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.951 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:16:57.951 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.056926+0000", 2024-08-19T14:16:57.951 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3579999999999997e-06 2024-08-19T14:16:57.951 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.951 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.951 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:16:57.952 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.059133+0000", 2024-08-19T14:16:57.952 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0022067789999999999 2024-08-19T14:16:57.952 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.952 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.952 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:16:57.952 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.059259+0000", 2024-08-19T14:16:57.952 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012647799999999999 2024-08-19T14:16:57.952 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.952 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.953 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0,9", 2024-08-19T14:16:57.953 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.059411+0000", 2024-08-19T14:16:57.953 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00015167800000000001 2024-08-19T14:16:57.953 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.953 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.953 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:57.953 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.061433+0000", 2024-08-19T14:16:57.953 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0020222510000000001 2024-08-19T14:16:57.953 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.953 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.953 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:16:57.953 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.062698+0000", 2024-08-19T14:16:57.953 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0012643559999999999 2024-08-19T14:16:57.953 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.954 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.954 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:57.954 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.176669+0000", 2024-08-19T14:16:57.954 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.11397160200000001 2024-08-19T14:16:57.954 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.954 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.954 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:16:57.954 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.176701+0000", 2024-08-19T14:16:57.954 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1863000000000003e-05 2024-08-19T14:16:57.954 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.954 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.955 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:16:57.956 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:16:57.956 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:21.176721+0000", 2024-08-19T14:16:57.956 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9809999999999998e-05 2024-08-19T14:16:57.957 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:57.957 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:16:57.957 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:57.957 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.957 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.957 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:115128 2.0 2:08e033d8:::benchmark_data_smithi045_46415_object115127:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-08-19T14:16:57.957 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:14:09.964413+0000", 2024-08-19T14:16:57.957 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 167.870316406, 2024-08-19T14:16:57.957 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.15347317799999999, 2024-08-19T14:16:57.957 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:16:57.957 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:16:57.957 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:16:57.957 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:16:57.958 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:16:57.958 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:16:57.958 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 115128 2024-08-19T14:16:57.958 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.958 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:16:57.958 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.958 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:16:57.958 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:09.964413+0000", 2024-08-19T14:16:57.958 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:57.958 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.958 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.958 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:16:57.958 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:09.964413+0000", 2024-08-19T14:16:57.959 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:57.959 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.959 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.959 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:16:57.959 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:09.964414+0000", 2024-08-19T14:16:57.959 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.689e-06 2024-08-19T14:16:57.959 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.959 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.959 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:16:57.959 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:09.964452+0000", 2024-08-19T14:16:57.959 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7565999999999999e-05 2024-08-19T14:16:57.959 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.959 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.959 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:16:57.960 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:09.964467+0000", 2024-08-19T14:16:57.960 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5160999999999999e-05 2024-08-19T14:16:57.960 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.960 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.960 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:16:57.960 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:09.964475+0000", 2024-08-19T14:16:57.960 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.276e-06 2024-08-19T14:16:57.960 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.960 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.960 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:16:57.960 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:09.964534+0000", 2024-08-19T14:16:57.960 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8727999999999997e-05 2024-08-19T14:16:57.960 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.960 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.961 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:16:57.961 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:09.964675+0000", 2024-08-19T14:16:57.961 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000140705 2024-08-19T14:16:57.961 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.961 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.961 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 9,11", 2024-08-19T14:16:57.961 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:09.964830+0000", 2024-08-19T14:16:57.961 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00015500900000000001 2024-08-19T14:16:57.962 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.962 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.962 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:16:57.962 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:09.967628+0000", 2024-08-19T14:16:57.962 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0027980909999999999 2024-08-19T14:16:57.962 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.962 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.962 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:57.962 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:09.968335+0000", 2024-08-19T14:16:57.962 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00070710399999999996 2024-08-19T14:16:57.962 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.962 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.962 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:57.963 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:10.117809+0000", 2024-08-19T14:16:57.963 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.149473572 2024-08-19T14:16:57.963 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.963 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.963 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:16:57.963 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:10.117849+0000", 2024-08-19T14:16:57.963 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0431e-05 2024-08-19T14:16:57.963 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.964 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.964 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:16:57.964 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:10.117886+0000", 2024-08-19T14:16:57.964 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6845999999999999e-05 2024-08-19T14:16:57.964 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:57.964 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:16:57.964 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:57.964 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.965 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.965 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:115136 2.0 2:02df7cb0:::benchmark_data_smithi045_46415_object115135:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-08-19T14:16:57.965 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:14:09.971356+0000", 2024-08-19T14:16:57.965 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 167.863372916, 2024-08-19T14:16:57.965 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.15309395200000001, 2024-08-19T14:16:57.965 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:16:57.965 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:16:57.965 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:16:57.965 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:16:57.965 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:16:57.965 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:16:57.965 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 115136 2024-08-19T14:16:57.966 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.966 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:16:57.966 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.966 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:16:57.966 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:09.971356+0000", 2024-08-19T14:16:57.966 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:57.966 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.966 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.966 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:16:57.966 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:09.971356+0000", 2024-08-19T14:16:57.966 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:57.966 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.966 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.967 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:16:57.967 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:09.971357+0000", 2024-08-19T14:16:57.967 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.172e-06 2024-08-19T14:16:57.967 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.967 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.967 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:16:57.967 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:09.971398+0000", 2024-08-19T14:16:57.967 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0955999999999999e-05 2024-08-19T14:16:57.967 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.967 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.967 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:16:57.967 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:09.971410+0000", 2024-08-19T14:16:57.968 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.135e-05 2024-08-19T14:16:57.968 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.968 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.968 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:16:57.968 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:09.971415+0000", 2024-08-19T14:16:57.968 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1819999999999997e-06 2024-08-19T14:16:57.968 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.968 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.968 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:16:57.968 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:09.971457+0000", 2024-08-19T14:16:57.968 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1972999999999999e-05 2024-08-19T14:16:57.968 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.968 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.969 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:16:57.969 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:09.971584+0000", 2024-08-19T14:16:57.969 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012695800000000001 2024-08-19T14:16:57.969 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.969 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.969 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 9,11", 2024-08-19T14:16:57.969 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:09.971773+0000", 2024-08-19T14:16:57.969 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000189101 2024-08-19T14:16:57.969 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.969 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.969 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:16:57.969 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:09.975215+0000", 2024-08-19T14:16:57.969 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.003441871 2024-08-19T14:16:57.969 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.970 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.970 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:57.970 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:10.123256+0000", 2024-08-19T14:16:57.970 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.14804097999999999 2024-08-19T14:16:57.970 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.970 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.970 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:57.970 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:10.124383+0000", 2024-08-19T14:16:57.970 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00112725 2024-08-19T14:16:57.970 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.970 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.970 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:16:57.970 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:10.124421+0000", 2024-08-19T14:16:57.971 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8031999999999998e-05 2024-08-19T14:16:57.971 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.971 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.971 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:16:57.971 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:10.124450+0000", 2024-08-19T14:16:57.971 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9127000000000001e-05 2024-08-19T14:16:57.971 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:57.971 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:16:57.971 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:57.971 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.971 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.971 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:116086 2.0 2:05ee7e1e:::benchmark_data_smithi045_46415_object116085:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-08-19T14:16:57.972 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:14:11.223235+0000", 2024-08-19T14:16:57.972 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 166.61149439499999, 2024-08-19T14:16:57.972 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.115283706, 2024-08-19T14:16:57.972 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:16:57.972 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:16:57.972 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:16:57.972 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:16:57.972 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:16:57.972 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:16:57.972 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 116086 2024-08-19T14:16:57.972 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.972 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:16:57.972 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.973 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:16:57.973 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:11.223235+0000", 2024-08-19T14:16:57.973 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:57.973 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.973 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.973 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:16:57.973 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:11.223235+0000", 2024-08-19T14:16:57.973 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:57.973 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.973 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.973 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:16:57.973 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:11.223235+0000", 2024-08-19T14:16:57.973 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1299999999999999e-07 2024-08-19T14:16:57.973 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.974 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.974 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:16:57.974 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:11.223313+0000", 2024-08-19T14:16:57.974 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7986999999999997e-05 2024-08-19T14:16:57.974 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.974 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.974 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:16:57.974 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:11.223323+0000", 2024-08-19T14:16:57.974 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.5510000000000005e-06 2024-08-19T14:16:57.974 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.974 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.974 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:16:57.974 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:11.223328+0000", 2024-08-19T14:16:57.975 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4360000000000001e-06 2024-08-19T14:16:57.975 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.975 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.975 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:16:57.975 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:11.223370+0000", 2024-08-19T14:16:57.975 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1551999999999998e-05 2024-08-19T14:16:57.975 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.975 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.975 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:16:57.975 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:11.223471+0000", 2024-08-19T14:16:57.975 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00010061 2024-08-19T14:16:57.975 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.975 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.976 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 9,11", 2024-08-19T14:16:57.976 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:11.223559+0000", 2024-08-19T14:16:57.976 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.8264999999999996e-05 2024-08-19T14:16:57.976 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.976 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.976 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:16:57.976 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:11.226453+0000", 2024-08-19T14:16:57.976 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0028945139999999999 2024-08-19T14:16:57.976 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.976 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.976 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:57.976 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:11.326208+0000", 2024-08-19T14:16:57.976 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.099755144000000004 2024-08-19T14:16:57.977 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.977 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.977 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:57.977 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:11.338458+0000", 2024-08-19T14:16:57.977 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.012249833 2024-08-19T14:16:57.977 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.977 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.977 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:16:57.977 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:11.338492+0000", 2024-08-19T14:16:57.977 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3806999999999997e-05 2024-08-19T14:16:57.977 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.977 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.977 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:16:57.977 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:11.338518+0000", 2024-08-19T14:16:57.978 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6293999999999999e-05 2024-08-19T14:16:57.978 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:57.978 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:16:57.978 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:57.978 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.978 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.978 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:171514 2.0 2:07116ad7:::benchmark_data_smithi045_46415_object48233:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e169)", 2024-08-19T14:16:57.978 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:15:12.896400+0000", 2024-08-19T14:16:57.978 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 104.9383287, 2024-08-19T14:16:57.978 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.112982306, 2024-08-19T14:16:57.978 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:16:57.978 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:16:57.978 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:16:57.978 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:16:57.979 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:16:57.979 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:16:57.979 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 171514 2024-08-19T14:16:57.979 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.979 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:16:57.979 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.979 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:16:57.979 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.896400+0000", 2024-08-19T14:16:57.979 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:57.979 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.979 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.979 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:16:57.979 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.896400+0000", 2024-08-19T14:16:57.980 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:57.980 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.980 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.980 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:16:57.980 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.896401+0000", 2024-08-19T14:16:57.980 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.3200000000000004e-07 2024-08-19T14:16:57.980 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.980 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.980 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:16:57.980 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.896405+0000", 2024-08-19T14:16:57.980 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0260000000000001e-06 2024-08-19T14:16:57.980 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.980 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.980 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:16:57.981 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.896416+0000", 2024-08-19T14:16:57.981 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0634e-05 2024-08-19T14:16:57.981 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.981 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.981 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:16:57.981 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.896423+0000", 2024-08-19T14:16:57.981 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3039999999999999e-06 2024-08-19T14:16:57.981 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.981 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.981 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:16:57.981 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.897983+0000", 2024-08-19T14:16:57.981 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0015601250000000001 2024-08-19T14:16:57.981 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.981 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.982 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:16:57.982 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.898096+0000", 2024-08-19T14:16:57.982 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00011288 2024-08-19T14:16:57.982 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.982 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.982 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 9,11", 2024-08-19T14:16:57.982 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.898211+0000", 2024-08-19T14:16:57.982 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000114633 2024-08-19T14:16:57.982 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.982 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.982 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:16:57.982 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:13.009151+0000", 2024-08-19T14:16:57.982 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.110940552 2024-08-19T14:16:57.983 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.983 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.983 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:57.983 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:13.009202+0000", 2024-08-19T14:16:57.983 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0250000000000002e-05 2024-08-19T14:16:57.983 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.983 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.983 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:57.983 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:13.009237+0000", 2024-08-19T14:16:57.983 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5660000000000001e-05 2024-08-19T14:16:57.983 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.983 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.983 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:16:57.983 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:13.009339+0000", 2024-08-19T14:16:57.984 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000101908 2024-08-19T14:16:57.984 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.984 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.984 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:16:57.984 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:13.009383+0000", 2024-08-19T14:16:57.984 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3501999999999998e-05 2024-08-19T14:16:57.984 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:57.984 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:16:57.984 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:57.984 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.984 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.984 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:171515 2.8 2:10779ea6:::benchmark_data_smithi045_46415_object48234:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e169)", 2024-08-19T14:16:57.984 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:15:12.902135+0000", 2024-08-19T14:16:57.984 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 104.932594045, 2024-08-19T14:16:57.985 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.131735885, 2024-08-19T14:16:57.985 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:16:57.985 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:16:57.985 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:16:57.985 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:16:57.985 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:16:57.985 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:16:57.985 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 171515 2024-08-19T14:16:57.985 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.985 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:16:57.985 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.985 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:16:57.985 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.902135+0000", 2024-08-19T14:16:57.986 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:57.986 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.986 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.986 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:16:57.986 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.902135+0000", 2024-08-19T14:16:57.986 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:57.986 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.986 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.986 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:16:57.986 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.902136+0000", 2024-08-19T14:16:57.986 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.9400000000000004e-07 2024-08-19T14:16:57.986 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.986 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.987 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:16:57.987 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.902140+0000", 2024-08-19T14:16:57.987 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9600000000000002e-06 2024-08-19T14:16:57.987 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.987 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.987 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:16:57.987 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.902148+0000", 2024-08-19T14:16:57.987 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.8290000000000008e-06 2024-08-19T14:16:57.987 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.987 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.987 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:16:57.987 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.902153+0000", 2024-08-19T14:16:57.987 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7990000000000001e-06 2024-08-19T14:16:57.987 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.988 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.988 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:16:57.988 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:13.009177+0000", 2024-08-19T14:16:57.988 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.107024059 2024-08-19T14:16:57.988 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.988 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.988 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:16:57.988 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:13.009300+0000", 2024-08-19T14:16:57.988 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012342399999999999 2024-08-19T14:16:57.988 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.988 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.988 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0,9", 2024-08-19T14:16:57.988 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:13.009424+0000", 2024-08-19T14:16:57.989 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012440500000000001 2024-08-19T14:16:57.989 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.989 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.989 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:57.989 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:13.023363+0000", 2024-08-19T14:16:57.989 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.013938623000000001 2024-08-19T14:16:57.989 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.989 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.989 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:16:57.989 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:13.023469+0000", 2024-08-19T14:16:57.989 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00010608 2024-08-19T14:16:57.990 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.990 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.990 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:57.990 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:13.033804+0000", 2024-08-19T14:16:57.990 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.01033537 2024-08-19T14:16:57.990 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.990 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.990 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:16:57.990 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:13.033837+0000", 2024-08-19T14:16:57.990 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2364999999999999e-05 2024-08-19T14:16:57.990 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.990 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.991 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:16:57.991 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:13.033871+0000", 2024-08-19T14:16:57.991 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.4076999999999999e-05 2024-08-19T14:16:57.991 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:57.991 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:16:57.991 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:57.991 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.991 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.991 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:171516 2.8 2:1157ba84:::benchmark_data_smithi045_46415_object48235:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e169)", 2024-08-19T14:16:57.991 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:15:12.902240+0000", 2024-08-19T14:16:57.991 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 104.932488857, 2024-08-19T14:16:57.991 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.13175336400000001, 2024-08-19T14:16:57.991 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:16:57.991 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:16:57.992 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:16:57.992 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:16:57.992 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:16:57.992 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:16:57.992 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 171516 2024-08-19T14:16:57.992 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.992 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:16:57.992 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.992 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:16:57.992 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.902240+0000", 2024-08-19T14:16:57.992 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:57.992 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.992 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.992 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:16:57.993 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.902240+0000", 2024-08-19T14:16:57.993 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:57.993 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.993 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.993 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:16:57.993 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.902241+0000", 2024-08-19T14:16:57.993 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9900000000000001e-07 2024-08-19T14:16:57.993 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.993 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.993 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:16:57.993 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.902244+0000", 2024-08-19T14:16:57.993 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.869e-06 2024-08-19T14:16:57.993 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.993 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.994 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:16:57.994 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.902249+0000", 2024-08-19T14:16:57.994 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8289999999999996e-06 2024-08-19T14:16:57.994 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.994 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.994 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:16:57.994 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.902253+0000", 2024-08-19T14:16:57.994 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8670000000000004e-06 2024-08-19T14:16:57.994 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.994 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.994 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:16:57.994 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:13.013300+0000", 2024-08-19T14:16:57.994 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.11104626400000001 2024-08-19T14:16:57.995 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.995 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.995 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:16:57.995 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:13.013422+0000", 2024-08-19T14:16:57.995 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012265099999999999 2024-08-19T14:16:57.995 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.995 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.995 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0,9", 2024-08-19T14:16:57.995 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:13.013552+0000", 2024-08-19T14:16:57.995 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000130123 2024-08-19T14:16:57.995 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.995 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.995 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:57.996 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:13.023455+0000", 2024-08-19T14:16:57.996 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0099022610000000007 2024-08-19T14:16:57.996 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.996 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.996 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:16:57.996 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:13.023476+0000", 2024-08-19T14:16:57.996 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1591e-05 2024-08-19T14:16:57.996 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.996 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.996 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:57.996 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:13.033911+0000", 2024-08-19T14:16:57.997 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010434824000000001 2024-08-19T14:16:57.997 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.997 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.997 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:16:57.997 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:13.033940+0000", 2024-08-19T14:16:57.997 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9033000000000001e-05 2024-08-19T14:16:57.997 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.997 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.997 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:16:57.997 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:13.033994+0000", 2024-08-19T14:16:57.997 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3553e-05 2024-08-19T14:16:57.997 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:57.997 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:16:57.997 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:57.998 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.998 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.998 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:171517 2.8 2:17a35dc4:::benchmark_data_smithi045_46415_object48236:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e169)", 2024-08-19T14:16:57.998 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:15:12.905580+0000", 2024-08-19T14:16:57.998 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 104.929149567, 2024-08-19T14:16:57.998 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.133369984, 2024-08-19T14:16:57.998 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:16:57.998 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:16:57.998 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:16:57.998 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:16:57.998 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:16:57.998 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:16:57.998 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 171517 2024-08-19T14:16:57.999 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.999 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:16:57.999 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.999 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:16:57.999 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.905580+0000", 2024-08-19T14:16:57.999 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:57.999 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:57.999 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:57.999 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:16:57.999 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.905580+0000", 2024-08-19T14:16:57.999 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:16:57.999 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:58.000 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:58.000 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:16:58.000 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.905580+0000", 2024-08-19T14:16:58.000 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6199999999999997e-07 2024-08-19T14:16:58.000 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:58.000 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:58.000 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:16:58.000 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.905583+0000", 2024-08-19T14:16:58.000 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2789999999999999e-06 2024-08-19T14:16:58.000 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:58.000 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:58.000 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:16:58.000 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:13.004022+0000", 2024-08-19T14:16:58.000 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.098438756000000002 2024-08-19T14:16:58.001 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:58.001 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:58.001 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:16:58.001 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:13.004038+0000", 2024-08-19T14:16:58.001 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6038000000000001e-05 2024-08-19T14:16:58.001 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:58.001 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:58.001 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:16:58.001 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:13.018509+0000", 2024-08-19T14:16:58.001 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.014470642000000001 2024-08-19T14:16:58.001 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:58.001 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:58.001 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:16:58.002 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:13.018652+0000", 2024-08-19T14:16:58.002 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000143048 2024-08-19T14:16:58.002 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:58.002 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:58.002 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0,9", 2024-08-19T14:16:58.002 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:13.018802+0000", 2024-08-19T14:16:58.002 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000149748 2024-08-19T14:16:58.002 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:58.002 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:58.002 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:58.002 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:13.023512+0000", 2024-08-19T14:16:58.002 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0047101729999999998 2024-08-19T14:16:58.002 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:58.002 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:58.003 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:16:58.003 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:13.028445+0000", 2024-08-19T14:16:58.003 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0049330479999999998 2024-08-19T14:16:58.003 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:58.003 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:58.003 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:16:58.003 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:13.038830+0000", 2024-08-19T14:16:58.003 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010384882999999999 2024-08-19T14:16:58.003 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:58.003 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:58.003 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:16:58.003 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:13.038884+0000", 2024-08-19T14:16:58.003 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4664999999999999e-05 2024-08-19T14:16:58.003 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:16:58.004 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:16:58.004 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:16:58.004 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:13.038950+0000", 2024-08-19T14:16:58.004 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.5042000000000005e-05 2024-08-19T14:16:58.004 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:58.004 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:16:58.004 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:58.004 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:16:58.004 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:16:58.004 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:16:58.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:58.006 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:58.006 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:16:58.107 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:16:58.130 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:58.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:58.131 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:16:58.203 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-19T14:16:58.232 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:16:58.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:58.262 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:58.262 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:16:58.362 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:16:58.386 INFO:tasks.thrashosds.thrasher:j is {'epoch': 223, 'fsid': 'ed7d1c6e-5e33-11ef-bccf-c7b262605968', 'created': '2024-08-19T14:05:22.842169+0000', 'modified': '2024-08-19T14:16:56.401926+0000', 'last_up_change': '2024-08-19T14:11:39.491234+0000', 'last_in_change': '2024-08-19T14:16:38.207673+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 27, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-08-19T14:08:15.966221+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '112', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '110', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3.2699999809265137, 'score_stable': 3.2699999809265137, 'optimal_score': 0.75, 'raw_score_acting': 2.450000047683716, 'raw_score_stable': 2.450000047683716, 'primary_affinity_weighted': 0.9228158593177795, 'average_primary_affinity': 0.9200000166893005, 'average_primary_affinity_weighted': 0.9200000166893005}}, {'pool': 3, 'pool_name': 'unique_pool_1', 'create_time': '2024-08-19T14:16:55.393520+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '222', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 0.75, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 0.9228158593177795, 'average_primary_affinity': 0.9200000166893005, 'average_primary_affinity_weighted': 0.9200000166893005}}], 'osds': [{'osd': 0, 'uuid': 'd7a0fff3-23c6-4ee3-a275-a844862aabfb', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 222, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6802', 'nonce': 1108945097}, {'type': 'v1', 'addr': '172.21.15.31:6803', 'nonce': 1108945097}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6804', 'nonce': 1108945097}, {'type': 'v1', 'addr': '172.21.15.31:6805', 'nonce': 1108945097}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6808', 'nonce': 1108945097}, {'type': 'v1', 'addr': '172.21.15.31:6809', 'nonce': 1108945097}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6806', 'nonce': 1108945097}, {'type': 'v1', 'addr': '172.21.15.31:6807', 'nonce': 1108945097}]}, 'public_addr': '172.21.15.31:6803/1108945097', 'cluster_addr': '172.21.15.31:6805/1108945097', 'heartbeat_back_addr': '172.21.15.31:6809/1108945097', 'heartbeat_front_addr': '172.21.15.31:6807/1108945097', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '145d5bb4-74dd-440f-bcc9-7e9ebda9c5d6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 222, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6810', 'nonce': 3236186806}, {'type': 'v1', 'addr': '172.21.15.31:6811', 'nonce': 3236186806}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6812', 'nonce': 3236186806}, {'type': 'v1', 'addr': '172.21.15.31:6813', 'nonce': 3236186806}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6816', 'nonce': 3236186806}, {'type': 'v1', 'addr': '172.21.15.31:6817', 'nonce': 3236186806}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6814', 'nonce': 3236186806}, {'type': 'v1', 'addr': '172.21.15.31:6815', 'nonce': 3236186806}]}, 'public_addr': '172.21.15.31:6811/3236186806', 'cluster_addr': '172.21.15.31:6813/3236186806', 'heartbeat_back_addr': '172.21.15.31:6817/3236186806', 'heartbeat_front_addr': '172.21.15.31:6815/3236186806', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '376c18d0-4768-46a0-bb60-e79c88ca909c', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 0.07379150390625, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 204, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6818', 'nonce': 4045553387}, {'type': 'v1', 'addr': '172.21.15.31:6819', 'nonce': 4045553387}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6820', 'nonce': 4045553387}, {'type': 'v1', 'addr': '172.21.15.31:6821', 'nonce': 4045553387}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6824', 'nonce': 4045553387}, {'type': 'v1', 'addr': '172.21.15.31:6825', 'nonce': 4045553387}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6822', 'nonce': 4045553387}, {'type': 'v1', 'addr': '172.21.15.31:6823', 'nonce': 4045553387}]}, 'public_addr': '172.21.15.31:6819/4045553387', 'cluster_addr': '172.21.15.31:6821/4045553387', 'heartbeat_back_addr': '172.21.15.31:6825/4045553387', 'heartbeat_front_addr': '172.21.15.31:6823/4045553387', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'a787ea9f-b9ba-450f-b388-614cfba90634', 'up': 1, 'in': 1, 'weight': 0.6437225341796875, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 222, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6826', 'nonce': 2288082716}, {'type': 'v1', 'addr': '172.21.15.31:6827', 'nonce': 2288082716}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6828', 'nonce': 2288082716}, {'type': 'v1', 'addr': '172.21.15.31:6829', 'nonce': 2288082716}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6832', 'nonce': 2288082716}, {'type': 'v1', 'addr': '172.21.15.31:6833', 'nonce': 2288082716}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6830', 'nonce': 2288082716}, {'type': 'v1', 'addr': '172.21.15.31:6831', 'nonce': 2288082716}]}, 'public_addr': '172.21.15.31:6827/2288082716', 'cluster_addr': '172.21.15.31:6829/2288082716', 'heartbeat_back_addr': '172.21.15.31:6833/2288082716', 'heartbeat_front_addr': '172.21.15.31:6831/2288082716', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '7859751a-12f5-4a57-a332-435360479d53', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 222, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6800', 'nonce': 3934739984}, {'type': 'v1', 'addr': '172.21.15.32:6801', 'nonce': 3934739984}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6802', 'nonce': 3934739984}, {'type': 'v1', 'addr': '172.21.15.32:6803', 'nonce': 3934739984}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6806', 'nonce': 3934739984}, {'type': 'v1', 'addr': '172.21.15.32:6807', 'nonce': 3934739984}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6804', 'nonce': 3934739984}, {'type': 'v1', 'addr': '172.21.15.32:6805', 'nonce': 3934739984}]}, 'public_addr': '172.21.15.32:6801/3934739984', 'cluster_addr': '172.21.15.32:6803/3934739984', 'heartbeat_back_addr': '172.21.15.32:6807/3934739984', 'heartbeat_front_addr': '172.21.15.32:6805/3934739984', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '017f3eef-ac60-4b82-8350-919017cb4ccd', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 40, 'up_thru': 222, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6808', 'nonce': 526388985}, {'type': 'v1', 'addr': '172.21.15.32:6809', 'nonce': 526388985}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6810', 'nonce': 526388985}, {'type': 'v1', 'addr': '172.21.15.32:6811', 'nonce': 526388985}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6814', 'nonce': 526388985}, {'type': 'v1', 'addr': '172.21.15.32:6815', 'nonce': 526388985}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6812', 'nonce': 526388985}, {'type': 'v1', 'addr': '172.21.15.32:6813', 'nonce': 526388985}]}, 'public_addr': '172.21.15.32:6809/526388985', 'cluster_addr': '172.21.15.32:6811/526388985', 'heartbeat_back_addr': '172.21.15.32:6815/526388985', 'heartbeat_front_addr': '172.21.15.32:6813/526388985', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '44aa8229-0268-49d7-bd57-3bf1f384e5e6', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 48, 'up_thru': 126, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6816', 'nonce': 1131035844}, {'type': 'v1', 'addr': '172.21.15.32:6817', 'nonce': 1131035844}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6818', 'nonce': 1131035844}, {'type': 'v1', 'addr': '172.21.15.32:6819', 'nonce': 1131035844}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6822', 'nonce': 1131035844}, {'type': 'v1', 'addr': '172.21.15.32:6823', 'nonce': 1131035844}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6820', 'nonce': 1131035844}, {'type': 'v1', 'addr': '172.21.15.32:6821', 'nonce': 1131035844}]}, 'public_addr': '172.21.15.32:6817/1131035844', 'cluster_addr': '172.21.15.32:6819/1131035844', 'heartbeat_back_addr': '172.21.15.32:6823/1131035844', 'heartbeat_front_addr': '172.21.15.32:6821/1131035844', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '263433e5-70b6-4192-86ff-dc2ad95927eb', 'up': 1, 'in': 1, 'weight': 0.95001220703125, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 57, 'up_thru': 222, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6824', 'nonce': 2014939947}, {'type': 'v1', 'addr': '172.21.15.32:6825', 'nonce': 2014939947}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6826', 'nonce': 2014939947}, {'type': 'v1', 'addr': '172.21.15.32:6827', 'nonce': 2014939947}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6830', 'nonce': 2014939947}, {'type': 'v1', 'addr': '172.21.15.32:6831', 'nonce': 2014939947}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6828', 'nonce': 2014939947}, {'type': 'v1', 'addr': '172.21.15.32:6829', 'nonce': 2014939947}]}, 'public_addr': '172.21.15.32:6825/2014939947', 'cluster_addr': '172.21.15.32:6827/2014939947', 'heartbeat_back_addr': '172.21.15.32:6831/2014939947', 'heartbeat_front_addr': '172.21.15.32:6829/2014939947', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': '2bbcad36-0c74-4528-9fb9-a8812afa1662', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 65, 'up_thru': 222, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6800', 'nonce': 1445140892}, {'type': 'v1', 'addr': '172.21.15.45:6801', 'nonce': 1445140892}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6802', 'nonce': 1445140892}, {'type': 'v1', 'addr': '172.21.15.45:6803', 'nonce': 1445140892}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6806', 'nonce': 1445140892}, {'type': 'v1', 'addr': '172.21.15.45:6807', 'nonce': 1445140892}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6804', 'nonce': 1445140892}, {'type': 'v1', 'addr': '172.21.15.45:6805', 'nonce': 1445140892}]}, 'public_addr': '172.21.15.45:6801/1445140892', 'cluster_addr': '172.21.15.45:6803/1445140892', 'heartbeat_back_addr': '172.21.15.45:6807/1445140892', 'heartbeat_front_addr': '172.21.15.45:6805/1445140892', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': 'c8ee4277-3bf9-46bf-9cc0-152c1c4b9508', 'up': 1, 'in': 1, 'weight': 0.95001220703125, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 74, 'up_thru': 222, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6808', 'nonce': 1213787169}, {'type': 'v1', 'addr': '172.21.15.45:6809', 'nonce': 1213787169}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6810', 'nonce': 1213787169}, {'type': 'v1', 'addr': '172.21.15.45:6811', 'nonce': 1213787169}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6814', 'nonce': 1213787169}, {'type': 'v1', 'addr': '172.21.15.45:6815', 'nonce': 1213787169}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6812', 'nonce': 1213787169}, {'type': 'v1', 'addr': '172.21.15.45:6813', 'nonce': 1213787169}]}, 'public_addr': '172.21.15.45:6809/1213787169', 'cluster_addr': '172.21.15.45:6811/1213787169', 'heartbeat_back_addr': '172.21.15.45:6815/1213787169', 'heartbeat_front_addr': '172.21.15.45:6813/1213787169', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '29031395-5f5c-40f8-ac90-9f6ac48e5c9f', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 80, 'up_thru': 115, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6816', 'nonce': 2461824621}, {'type': 'v1', 'addr': '172.21.15.45:6817', 'nonce': 2461824621}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6818', 'nonce': 2461824621}, {'type': 'v1', 'addr': '172.21.15.45:6819', 'nonce': 2461824621}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6822', 'nonce': 2461824621}, {'type': 'v1', 'addr': '172.21.15.45:6823', 'nonce': 2461824621}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6820', 'nonce': 2461824621}, {'type': 'v1', 'addr': '172.21.15.45:6821', 'nonce': 2461824621}]}, 'public_addr': '172.21.15.45:6817/2461824621', 'cluster_addr': '172.21.15.45:6819/2461824621', 'heartbeat_back_addr': '172.21.15.45:6823/2461824621', 'heartbeat_front_addr': '172.21.15.45:6821/2461824621', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': '5ef0db4b-0fef-4ea0-a19b-d3677e082c29', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 86, 'up_thru': 222, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6824', 'nonce': 2438396298}, {'type': 'v1', 'addr': '172.21.15.45:6825', 'nonce': 2438396298}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6826', 'nonce': 2438396298}, {'type': 'v1', 'addr': '172.21.15.45:6827', 'nonce': 2438396298}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6830', 'nonce': 2438396298}, {'type': 'v1', 'addr': '172.21.15.45:6831', 'nonce': 2438396298}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6828', 'nonce': 2438396298}, {'type': 'v1', 'addr': '172.21.15.45:6829', 'nonce': 2438396298}]}, 'public_addr': '172.21.15.45:6825/2438396298', 'cluster_addr': '172.21.15.45:6827/2438396298', 'heartbeat_back_addr': '172.21.15.45:6831/2438396298', 'heartbeat_front_addr': '172.21.15.45:6829/2438396298', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-19T14:15:56.929307+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-19T14:15:55.638039+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-08-19T14:15:58.904876+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-19T14:16:04.911361+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-19T14:16:03.945511+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-19T14:15:49.417209+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-08-19T14:16:48.606221+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-19T14:16:49.277037+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-19T14:16:35.019990+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-19T14:15:54.288635+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-08-19T14:15:57.551806+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-19T14:16:45.469890+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '1.0', 'osds': [2, 9, 0]}], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.31:0/1451745969': '2024-08-20T14:06:13.660657+0000', '172.21.15.31:0/1515698918': '2024-08-20T14:06:13.660657+0000', '172.21.15.31:6801/3297539242': '2024-08-20T14:06:13.660657+0000', '172.21.15.31:6800/3297539242': '2024-08-20T14:06:13.660657+0000', '172.21.15.31:0/2217423818': '2024-08-20T14:05:48.827576+0000', '172.21.15.31:0/3353498281': '2024-08-20T14:06:13.660657+0000', '172.21.15.31:0/4013594220': '2024-08-20T14:05:48.827576+0000', '172.21.15.31:0/1654518065': '2024-08-20T14:05:48.827576+0000', '172.21.15.31:6801/2368821258': '2024-08-20T14:05:48.827576+0000', '172.21.15.31:6800/2368821258': '2024-08-20T14:05:48.827576+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-08-19T14:16:58.386 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-19T14:16:58.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:58.388 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:58.389 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:16:58.485 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:58 smithi031 ceph-mon[16336]: osdmap e224: 12 total, 12 up, 9 in 2024-08-19T14:16:58.485 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:58 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2426120598' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:16:58.489 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:16:58.496 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:16:58.496 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:16:58.496 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:16:58.496 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:16:58.496 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:16:58.496 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:16:58.496 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:16:58.496 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:16:58.496 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:16:58.497 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:16:58.497 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:16:58.497 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:16:58.497 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:16:58.497 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:16:58.497 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:16:58.497 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:16:58.497 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:16:58.498 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:16:58.499 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:16:58.499 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:16:58.499 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:16:58.499 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:16:58.499 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:16:58.499 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:16:58.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:58.513 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:58.513 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:16:58.614 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:16:58.635 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:58.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:58.640 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:16:58.686 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:58 smithi032 ceph-mon[20004]: osdmap e224: 12 total, 12 up, 9 in 2024-08-19T14:16:58.686 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:58 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2426120598' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:16:58.687 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-08-19T14:16:58.714 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:58 smithi045 ceph-mon[20143]: osdmap e224: 12 total, 12 up, 9 in 2024-08-19T14:16:58.714 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:58 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2426120598' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:16:58.741 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:16:58.741 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:16:58.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:58.763 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:58.763 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:16:58.776 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:16:58.864 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:16:58.886 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:58.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:58.887 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:16:58.956 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:16:58.988 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:16:59.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:59.011 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:16:59.011 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:16:59.114 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:16:59.137 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:59.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:59.138 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:16:59.240 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:16:59.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:59.265 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:59.265 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:16:59.367 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:16:59.388 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:59.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:59.389 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:16:59.455 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:16:59.455 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:16:59.490 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:16:59.511 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:16:59.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:59.512 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:16:59.534 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:59 smithi031 ceph-mon[16336]: pgmap v536: 27 pgs: 5 creating+peering, 9 unknown, 13 active+clean; 577 KiB data, 1.2 GiB used, 803 GiB / 805 GiB avail 2024-08-19T14:16:59.534 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:59 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:16:59.534 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:59 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:16:59.534 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:16:59 smithi031 ceph-mon[16336]: osdmap e225: 12 total, 12 up, 9 in 2024-08-19T14:16:59.612 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:16:59.638 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:59.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:59.639 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:16:59.740 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:16:59.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:59 smithi032 ceph-mon[20004]: pgmap v536: 27 pgs: 5 creating+peering, 9 unknown, 13 active+clean; 577 KiB data, 1.2 GiB used, 803 GiB / 805 GiB avail 2024-08-19T14:16:59.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:59 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:16:59.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:59 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:16:59.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:16:59 smithi032 ceph-mon[20004]: osdmap e225: 12 total, 12 up, 9 in 2024-08-19T14:16:59.765 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:59.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:59.766 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:16:59.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:59 smithi045 ceph-mon[20143]: pgmap v536: 27 pgs: 5 creating+peering, 9 unknown, 13 active+clean; 577 KiB data, 1.2 GiB used, 803 GiB / 805 GiB avail 2024-08-19T14:16:59.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:59 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:16:59.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:59 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:16:59.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:16:59 smithi045 ceph-mon[20143]: osdmap e225: 12 total, 12 up, 9 in 2024-08-19T14:16:59.866 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:16:59.892 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:16:59.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:16:59.893 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:16:59.994 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:17:00.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:00.022 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:00.022 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:17:00.024 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:17:00.123 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:17:00.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:00.145 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:00.145 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:17:00.171 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-08-19T14:17:00.219 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:17:00.220 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-08-19T14:17:00.247 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:17:00.268 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:00.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:00.269 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:17:00.370 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:17:00.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:00.393 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:00.393 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:17:00.422 INFO:teuthology.orchestra.run.smithi031.stderr:enabled application 'rados' on pool 'unique_pool_1' 2024-08-19T14:17:00.425 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:17:00.494 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:17:00.520 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:00.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:00.521 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:17:00.622 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:17:00.647 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:00.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:00.648 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:17:00.669 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:00 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/462933345' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-08-19T14:17:00.749 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:17:00.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:00 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/462933345' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-08-19T14:17:00.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:00.775 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:00.775 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:17:00.861 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:17:00.861 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:17:00.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:00 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/462933345' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-08-19T14:17:00.875 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:17:00.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:00.901 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:00.902 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:17:01.003 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:17:01.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:01.026 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:01.026 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:17:01.127 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:17:01.152 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:01.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:01.153 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:17:01.254 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:17:01.277 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:01.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:01.278 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:17:01.379 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:17:01.405 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:01.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:01.406 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:17:01.456 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:01 smithi045 ceph-mon[20143]: pgmap v538: 27 pgs: 5 creating+peering, 1 unknown, 21 active+clean; 577 KiB data, 1.3 GiB used, 803 GiB / 805 GiB avail 2024-08-19T14:17:01.457 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:01 smithi045 ceph-mon[20143]: from='client.15261 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:17:01.457 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:01 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/462933345' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-08-19T14:17:01.457 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:01 smithi045 ceph-mon[20143]: osdmap e226: 12 total, 12 up, 9 in 2024-08-19T14:17:01.485 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:01 smithi031 ceph-mon[16336]: pgmap v538: 27 pgs: 5 creating+peering, 1 unknown, 21 active+clean; 577 KiB data, 1.3 GiB used, 803 GiB / 805 GiB avail 2024-08-19T14:17:01.485 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:01 smithi031 ceph-mon[16336]: from='client.15261 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:17:01.486 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:01 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/462933345' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-08-19T14:17:01.486 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:01 smithi031 ceph-mon[16336]: osdmap e226: 12 total, 12 up, 9 in 2024-08-19T14:17:01.487 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-08-19T14:17:01.507 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:17:01.533 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:01.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:01.534 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:17:01.635 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:17:01.655 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:01.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:01.656 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:17:01.738 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:17:01.757 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:17:01.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:01 smithi032 ceph-mon[20004]: pgmap v538: 27 pgs: 5 creating+peering, 1 unknown, 21 active+clean; 577 KiB data, 1.3 GiB used, 803 GiB / 805 GiB avail 2024-08-19T14:17:01.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:01 smithi032 ceph-mon[20004]: from='client.15261 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:17:01.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:01 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/462933345' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-08-19T14:17:01.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:01 smithi032 ceph-mon[20004]: osdmap e226: 12 total, 12 up, 9 in 2024-08-19T14:17:01.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:01.779 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:01.780 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:17:01.791 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":538,"stamp":"2024-08-19T14:16:59.804439+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1529,"ondisk_log_size":1529,"up":78,"acting":78,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":65,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":1318520,"kb_used_data":50816,"kb_used_omap":14,"kb_used_meta":1267569,"kb_avail":842387848,"statfs":{"total":863955320832,"available":862605156352,"internally_reserved":0,"allocated":52035584,"data_stored":42316547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14353,"internal_metadata":1297991663},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.402814"},"pg_stats":[{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-08-19T14:16:57.410848+0000","last_change":"2024-08-19T14:16:56.416145+0000","last_active":"2024-08-19T14:16:57.410848+0000","last_peered":"2024-08-19T14:16:57.410848+0000","last_clean":"2024-08-19T14:16:57.410848+0000","last_became_active":"2024-08-19T14:16:56.415674+0000","last_became_peered":"2024-08-19T14:16:56.415674+0000","last_unstale":"2024-08-19T14:16:57.410848+0000","last_undegraded":"2024-08-19T14:16:57.410848+0000","last_fullsized":"2024-08-19T14:16:57.410848+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:25.076351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":1,"reported_epoch":222,"state":"creating+peering","last_fresh":"2024-08-19T14:16:55.405344+0000","last_change":"2024-08-19T14:16:55.405344+0000","last_active":"2024-08-19T14:16:55.393990+0000","last_peered":"2024-08-19T14:16:55.393990+0000","last_clean":"2024-08-19T14:16:55.393990+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-19T14:16:55.405344+0000","last_undegraded":"2024-08-19T14:16:55.405344+0000","last_fullsized":"2024-08-19T14:16:55.405344+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,0],"acting":[11,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-08-19T14:16:57.410848+0000","last_change":"2024-08-19T14:16:56.415868+0000","last_active":"2024-08-19T14:16:57.410848+0000","last_peered":"2024-08-19T14:16:57.410848+0000","last_clean":"2024-08-19T14:16:57.410848+0000","last_became_active":"2024-08-19T14:16:56.414976+0000","last_became_peered":"2024-08-19T14:16:56.414976+0000","last_unstale":"2024-08-19T14:16:57.410848+0000","last_undegraded":"2024-08-19T14:16:57.410848+0000","last_fullsized":"2024-08-19T14:16:57.410848+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:19.697449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,9],"acting":[7,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-08-19T14:16:56.416709+0000","last_change":"2024-08-19T14:16:56.415146+0000","last_active":"2024-08-19T14:16:56.416709+0000","last_peered":"2024-08-19T14:16:56.416709+0000","last_clean":"2024-08-19T14:16:56.416709+0000","last_became_active":"2024-08-19T14:16:56.414671+0000","last_became_peered":"2024-08-19T14:16:56.414671+0000","last_unstale":"2024-08-19T14:16:56.416709+0000","last_undegraded":"2024-08-19T14:16:56.416709+0000","last_fullsized":"2024-08-19T14:16:56.416709+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:20.691991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,9],"acting":[5,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-19T14:16:55.393990+0000","last_change":"2024-08-19T14:16:55.393990+0000","last_active":"2024-08-19T14:16:55.393990+0000","last_peered":"2024-08-19T14:16:55.393990+0000","last_clean":"2024-08-19T14:16:55.393990+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-19T14:16:55.393990+0000","last_undegraded":"2024-08-19T14:16:55.393990+0000","last_fullsized":"2024-08-19T14:16:55.393990+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-08-19T14:16:57.635663+0000","last_change":"2024-08-19T14:16:56.415768+0000","last_active":"2024-08-19T14:16:57.635663+0000","last_peered":"2024-08-19T14:16:57.635663+0000","last_clean":"2024-08-19T14:16:57.635663+0000","last_became_active":"2024-08-19T14:16:56.415495+0000","last_became_peered":"2024-08-19T14:16:56.415495+0000","last_unstale":"2024-08-19T14:16:57.635663+0000","last_undegraded":"2024-08-19T14:16:57.635663+0000","last_fullsized":"2024-08-19T14:16:57.635663+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:07.013913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,8],"acting":[4,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"70'139","reported_seq":536,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-08-19T14:16:56.413260+0000","last_change":"2024-08-19T14:16:42.249971+0000","last_active":"2024-08-19T14:16:56.413260+0000","last_peered":"2024-08-19T14:16:56.413260+0000","last_clean":"2024-08-19T14:16:56.413260+0000","last_became_active":"2024-08-19T14:16:42.249518+0000","last_became_peered":"2024-08-19T14:16:42.249518+0000","last_unstale":"2024-08-19T14:16:56.413260+0000","last_undegraded":"2024-08-19T14:16:56.413260+0000","last_fullsized":"2024-08-19T14:16:56.413260+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:15:44.541487+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:15:44.541487+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:17:01.830360+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,8],"acting":[5,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":15,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-08-19T14:16:58.683480+0000","last_change":"2024-08-19T14:16:56.414110+0000","last_active":"2024-08-19T14:16:58.683480+0000","last_peered":"2024-08-19T14:16:58.683480+0000","last_clean":"2024-08-19T14:16:58.683480+0000","last_became_active":"2024-08-19T14:16:56.413751+0000","last_became_peered":"2024-08-19T14:16:56.413751+0000","last_unstale":"2024-08-19T14:16:58.683480+0000","last_undegraded":"2024-08-19T14:16:58.683480+0000","last_fullsized":"2024-08-19T14:16:58.683480+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:17:56.364651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"70'139","reported_seq":656,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-08-19T14:16:57.410729+0000","last_change":"2024-08-19T14:16:42.252897+0000","last_active":"2024-08-19T14:16:57.410729+0000","last_peered":"2024-08-19T14:16:57.410729+0000","last_clean":"2024-08-19T14:16:57.410729+0000","last_became_active":"2024-08-19T14:16:42.252391+0000","last_became_peered":"2024-08-19T14:16:42.252391+0000","last_unstale":"2024-08-19T14:16:57.410729+0000","last_undegraded":"2024-08-19T14:16:57.410729+0000","last_fullsized":"2024-08-19T14:16:57.410729+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:16:03.918046+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:17:27.707808+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1,"reported_epoch":222,"state":"creating+peering","last_fresh":"2024-08-19T14:16:55.405808+0000","last_change":"2024-08-19T14:16:55.405808+0000","last_active":"2024-08-19T14:16:55.393990+0000","last_peered":"2024-08-19T14:16:55.393990+0000","last_clean":"2024-08-19T14:16:55.393990+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-19T14:16:55.405808+0000","last_undegraded":"2024-08-19T14:16:55.405808+0000","last_fullsized":"2024-08-19T14:16:55.405808+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,4],"acting":[11,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.a","version":"70'139","reported_seq":534,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-08-19T14:16:57.635944+0000","last_change":"2024-08-19T14:16:02.818848+0000","last_active":"2024-08-19T14:16:57.635944+0000","last_peered":"2024-08-19T14:16:57.635944+0000","last_clean":"2024-08-19T14:16:57.635944+0000","last_became_active":"2024-08-19T14:13:22.728779+0000","last_became_peered":"2024-08-19T14:13:22.728779+0000","last_unstale":"2024-08-19T14:16:57.635944+0000","last_undegraded":"2024-08-19T14:16:57.635944+0000","last_fullsized":"2024-08-19T14:16:57.635944+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:16:02.818774+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:16:02.818774+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:17:09.255652+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,0],"acting":[9,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":1,"reported_epoch":222,"state":"creating+peering","last_fresh":"2024-08-19T14:16:55.407163+0000","last_change":"2024-08-19T14:16:55.407163+0000","last_active":"2024-08-19T14:16:55.393990+0000","last_peered":"2024-08-19T14:16:55.393990+0000","last_clean":"2024-08-19T14:16:55.393990+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-19T14:16:55.407163+0000","last_undegraded":"2024-08-19T14:16:55.407163+0000","last_fullsized":"2024-08-19T14:16:55.407163+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"70'139","reported_seq":568,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-08-19T14:16:55.406182+0000","last_change":"2024-08-19T14:16:29.257033+0000","last_active":"2024-08-19T14:16:55.406182+0000","last_peered":"2024-08-19T14:16:55.406182+0000","last_clean":"2024-08-19T14:16:55.406182+0000","last_became_active":"2024-08-19T14:13:05.607447+0000","last_became_peered":"2024-08-19T14:13:05.607447+0000","last_unstale":"2024-08-19T14:16:55.406182+0000","last_undegraded":"2024-08-19T14:16:55.406182+0000","last_fullsized":"2024-08-19T14:16:55.406182+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:16:29.256959+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:16:29.256959+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:17:40.848859+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":15,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-08-19T14:16:58.683488+0000","last_change":"2024-08-19T14:16:56.415838+0000","last_active":"2024-08-19T14:16:58.683488+0000","last_peered":"2024-08-19T14:16:58.683488+0000","last_clean":"2024-08-19T14:16:58.683488+0000","last_became_active":"2024-08-19T14:16:56.415384+0000","last_became_peered":"2024-08-19T14:16:56.415384+0000","last_unstale":"2024-08-19T14:16:58.683488+0000","last_undegraded":"2024-08-19T14:16:58.683488+0000","last_fullsized":"2024-08-19T14:16:58.683488+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:01.112132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"70'139","reported_seq":331,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-08-19T14:16:55.406183+0000","last_change":"2024-08-19T14:16:42.252111+0000","last_active":"2024-08-19T14:16:55.406183+0000","last_peered":"2024-08-19T14:16:55.406183+0000","last_clean":"2024-08-19T14:16:55.406183+0000","last_became_active":"2024-08-19T14:16:42.251643+0000","last_became_peered":"2024-08-19T14:16:42.251643+0000","last_unstale":"2024-08-19T14:16:55.406183+0000","last_undegraded":"2024-08-19T14:16:55.406183+0000","last_fullsized":"2024-08-19T14:16:55.406183+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:16:30.232687+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:16:30.232687+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:17:59.854133+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,4],"acting":[11,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-08-19T14:16:57.635390+0000","last_change":"2024-08-19T14:16:56.417053+0000","last_active":"2024-08-19T14:16:57.635390+0000","last_peered":"2024-08-19T14:16:57.635390+0000","last_clean":"2024-08-19T14:16:57.635390+0000","last_became_active":"2024-08-19T14:16:56.416751+0000","last_became_peered":"2024-08-19T14:16:56.416751+0000","last_unstale":"2024-08-19T14:16:57.635390+0000","last_undegraded":"2024-08-19T14:16:57.635390+0000","last_fullsized":"2024-08-19T14:16:57.635390+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:19.439377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,5],"acting":[1,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"70'139","reported_seq":521,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-08-19T14:16:57.635941+0000","last_change":"2024-08-19T14:16:36.969224+0000","last_active":"2024-08-19T14:16:57.635941+0000","last_peered":"2024-08-19T14:16:57.635941+0000","last_clean":"2024-08-19T14:16:57.635941+0000","last_became_active":"2024-08-19T14:13:11.647085+0000","last_became_peered":"2024-08-19T14:13:11.647085+0000","last_unstale":"2024-08-19T14:16:57.635941+0000","last_undegraded":"2024-08-19T14:16:57.635941+0000","last_fullsized":"2024-08-19T14:16:57.635941+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:16:36.969114+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:16:36.969114+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:17:44.848868+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,4],"acting":[9,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-08-19T14:16:56.416725+0000","last_change":"2024-08-19T14:16:56.416036+0000","last_active":"2024-08-19T14:16:56.416725+0000","last_peered":"2024-08-19T14:16:56.416725+0000","last_clean":"2024-08-19T14:16:56.416725+0000","last_became_active":"2024-08-19T14:16:56.415655+0000","last_became_peered":"2024-08-19T14:16:56.415655+0000","last_unstale":"2024-08-19T14:16:56.416725+0000","last_undegraded":"2024-08-19T14:16:56.416725+0000","last_fullsized":"2024-08-19T14:16:56.416725+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:17:56.536100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"70'139","reported_seq":436,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-08-19T14:16:56.413440+0000","last_change":"2024-08-19T14:16:42.252166+0000","last_active":"2024-08-19T14:16:56.413440+0000","last_peered":"2024-08-19T14:16:56.413440+0000","last_clean":"2024-08-19T14:16:56.413440+0000","last_became_active":"2024-08-19T14:16:42.251741+0000","last_became_peered":"2024-08-19T14:16:42.251741+0000","last_unstale":"2024-08-19T14:16:56.413440+0000","last_undegraded":"2024-08-19T14:16:56.413440+0000","last_fullsized":"2024-08-19T14:16:56.413440+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:16:10.672834+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:17:33.011576+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-08-19T14:16:57.635797+0000","last_change":"2024-08-19T14:16:56.415826+0000","last_active":"2024-08-19T14:16:57.635797+0000","last_peered":"2024-08-19T14:16:57.635797+0000","last_clean":"2024-08-19T14:16:57.635797+0000","last_became_active":"2024-08-19T14:16:56.415523+0000","last_became_peered":"2024-08-19T14:16:56.415523+0000","last_unstale":"2024-08-19T14:16:57.635797+0000","last_undegraded":"2024-08-19T14:16:57.635797+0000","last_fullsized":"2024-08-19T14:16:57.635797+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:05.782053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,11],"acting":[4,0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"70'139","reported_seq":530,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-08-19T14:16:57.635788+0000","last_change":"2024-08-19T14:16:42.252108+0000","last_active":"2024-08-19T14:16:57.635788+0000","last_peered":"2024-08-19T14:16:57.635788+0000","last_clean":"2024-08-19T14:16:57.635788+0000","last_became_active":"2024-08-19T14:16:42.251603+0000","last_became_peered":"2024-08-19T14:16:42.251603+0000","last_unstale":"2024-08-19T14:16:57.635788+0000","last_undegraded":"2024-08-19T14:16:57.635788+0000","last_fullsized":"2024-08-19T14:16:57.635788+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:16:30.043526+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:16:30.043526+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:17:54.530868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,11],"acting":[9,0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":1,"reported_epoch":222,"state":"creating+peering","last_fresh":"2024-08-19T14:16:55.405258+0000","last_change":"2024-08-19T14:16:55.405258+0000","last_active":"2024-08-19T14:16:55.393990+0000","last_peered":"2024-08-19T14:16:55.393990+0000","last_clean":"2024-08-19T14:16:55.393990+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-19T14:16:55.405258+0000","last_undegraded":"2024-08-19T14:16:55.405258+0000","last_fullsized":"2024-08-19T14:16:55.405258+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,1],"acting":[11,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.2","version":"70'139","reported_seq":474,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-08-19T14:16:55.406266+0000","last_change":"2024-08-19T14:16:00.592808+0000","last_active":"2024-08-19T14:16:55.406266+0000","last_peered":"2024-08-19T14:16:55.406266+0000","last_clean":"2024-08-19T14:16:55.406266+0000","last_became_active":"2024-08-19T14:15:59.993566+0000","last_became_peered":"2024-08-19T14:15:59.993566+0000","last_unstale":"2024-08-19T14:16:55.406266+0000","last_undegraded":"2024-08-19T14:16:55.406266+0000","last_fullsized":"2024-08-19T14:16:55.406266+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:16:00.592678+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:16:00.592678+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:17:07.175192+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,8],"acting":[0,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1,"reported_epoch":222,"state":"creating+peering","last_fresh":"2024-08-19T14:16:55.407160+0000","last_change":"2024-08-19T14:16:55.407160+0000","last_active":"2024-08-19T14:16:55.393990+0000","last_peered":"2024-08-19T14:16:55.393990+0000","last_clean":"2024-08-19T14:16:55.393990+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-19T14:16:55.407160+0000","last_undegraded":"2024-08-19T14:16:55.407160+0000","last_fullsized":"2024-08-19T14:16:55.407160+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,11],"acting":[0,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"70'139","reported_seq":533,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-08-19T14:16:57.635247+0000","last_change":"2024-08-19T14:16:06.679203+0000","last_active":"2024-08-19T14:16:57.635247+0000","last_peered":"2024-08-19T14:16:57.635247+0000","last_clean":"2024-08-19T14:16:57.635247+0000","last_became_active":"2024-08-19T14:13:22.727234+0000","last_became_peered":"2024-08-19T14:13:22.727234+0000","last_unstale":"2024-08-19T14:16:57.635247+0000","last_undegraded":"2024-08-19T14:16:57.635247+0000","last_fullsized":"2024-08-19T14:16:57.635247+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:16:06.679075+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:16:06.679075+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:17:31.357116+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-08-19T14:16:57.636063+0000","last_change":"2024-08-19T14:16:56.414945+0000","last_active":"2024-08-19T14:16:57.636063+0000","last_peered":"2024-08-19T14:16:57.636063+0000","last_clean":"2024-08-19T14:16:57.636063+0000","last_became_active":"2024-08-19T14:16:56.414587+0000","last_became_peered":"2024-08-19T14:16:56.414587+0000","last_unstale":"2024-08-19T14:16:57.636063+0000","last_undegraded":"2024-08-19T14:16:57.636063+0000","last_fullsized":"2024-08-19T14:16:57.636063+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:16.146711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,5],"acting":[9,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.0","version":"70'139","reported_seq":635,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-08-19T14:16:57.410638+0000","last_change":"2024-08-19T14:16:42.251163+0000","last_active":"2024-08-19T14:16:57.410638+0000","last_peered":"2024-08-19T14:16:57.410638+0000","last_clean":"2024-08-19T14:16:57.410638+0000","last_became_active":"2024-08-19T14:16:42.250702+0000","last_became_peered":"2024-08-19T14:16:42.250702+0000","last_unstale":"2024-08-19T14:16:57.410638+0000","last_undegraded":"2024-08-19T14:16:57.410638+0000","last_fullsized":"2024-08-19T14:16:57.410638+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:15:59.869266+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:15:59.869266+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:17:24.361856+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,9],"acting":[7,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":45,"acting":45,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1802240,"data_stored":1788032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1529,"ondisk_log_size":1529,"up":33,"acting":33,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187523,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":128424,"kb_used_data":2844,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":93616728,"statfs":{"total":95995035648,"available":95863529472,"internally_reserved":0,"allocated":2912256,"data_stored":1837663,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":128580036},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":10,"up_from":0,"seq":343597383752,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":74,"seq":317827579981,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":204544,"kb_used_data":28980,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":93540608,"statfs":{"total":95995035648,"available":95785582592,"internally_reserved":0,"allocated":29675520,"data_stored":28592790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":179763649},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874322,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":160548,"kb_used_data":3032,"kb_used_omap":1,"kb_used_meta":157502,"kb_avail":93584604,"statfs":{"total":95995035648,"available":95830634496,"internally_reserved":0,"allocated":3104768,"data_stored":2030519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":161282501},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":7,"up_from":57,"seq":244813135957,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":168708,"kb_used_data":2868,"kb_used_omap":1,"kb_used_meta":165822,"kb_avail":93576444,"statfs":{"total":95995035648,"available":95822278656,"internally_reserved":0,"allocated":2936832,"data_stored":1854102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":169802186},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":0,"seq":206158430297,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691933,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":83900,"kb_used_data":2736,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93661252,"statfs":{"total":95995035648,"available":95909122048,"internally_reserved":0,"allocated":2801664,"data_stored":1719323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":83098053},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986274,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":123960,"kb_used_data":2536,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":93621192,"statfs":{"total":95995035648,"available":95868100608,"internally_reserved":0,"allocated":2596864,"data_stored":1518075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149800,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":178312,"kb_used_data":2748,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":93566840,"statfs":{"total":95995035648,"available":95812444160,"internally_reserved":0,"allocated":2813952,"data_stored":1727670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":179763655},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411436,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574961,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":142016,"kb_used_data":2548,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":93603136,"statfs":{"total":95995035648,"available":95849611264,"internally_reserved":0,"allocated":2609152,"data_stored":1526422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":142801349},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738485,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":128108,"kb_used_data":2524,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":93617044,"statfs":{"total":95995035648,"available":95863853056,"internally_reserved":0,"allocated":2584576,"data_stored":1509983,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":128580037},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T14:17:01.793 INFO:tasks.thrashosds.thrasher:Setting 3.a to [6, 1, 4] 2024-08-19T14:17:01.793 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '3.a', '6', '1', '4'] 2024-08-19T14:17:01.793 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd pg-upmap 3.a 6 1 4 2024-08-19T14:17:01.854 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:17:01.880 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:17:01.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:01.908 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:01.908 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:17:02.008 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:17:02.034 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:02.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:02.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:17:02.060 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:17:02.136 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:17:02.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:02.159 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:02.159 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:17:02.243 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:17:02.244 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:17:02.260 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:17:02.261 DEBUG:teuthology.orchestra.run.smithi045:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_1 bench 90 write' 2024-08-19T14:17:02.264 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-08-19T14:17:02.264 INFO:teuthology.orchestra.run:waiting for 3000 2024-08-19T14:17:02.287 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:02.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:02.288 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:17:02.308 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:hints = 1 2024-08-19T14:17:02.309 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-08-19T14:17:02.309 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Object prefix: benchmark_data_smithi045_59973 2024-08-19T14:17:02.309 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-19T14:17:02.309 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 0 0 0 0 0 0 - 0 2024-08-19T14:17:02.389 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:17:02.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:02.429 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:02.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:17:02.431 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is unset 2024-08-19T14:17:02.446 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:02 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2352329776' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:17:02.530 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:17:02.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:02.567 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:02.567 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:17:02.668 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:17:02.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:02.703 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:02.703 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:17:02.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:02 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2352329776' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:17:02.759 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:17:02.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:02 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2352329776' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:17:02.804 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:17:02.850 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:02.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:02.852 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:17:02.951 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-08-19T14:17:02.952 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:17:02.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:02.989 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:02.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:17:03.091 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:17:03.127 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:03.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:03.128 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:17:03.229 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:17:03.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:03.262 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:03.262 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:17:03.309 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 1 16 4363 4347 271.423 271.688 0.00442837 0.00366937 2024-08-19T14:17:03.363 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:17:03.388 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:17:03.394 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:03.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:03.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:17:03.414 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:17:03.415 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:17:03.439 INFO:teuthology.orchestra.run.smithi031.stderr:set 3.a pg_upmap mapping to [6,1,4] 2024-08-19T14:17:03.443 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:17:03.444 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:17:03.475 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:17:03.476 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:17:03.496 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:17:03.506 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:17:03.506 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:17:03.507 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:03 smithi031 ceph-mon[16336]: pgmap v540: 27 pgs: 27 active+clean; 577 KiB data, 1.3 GiB used, 803 GiB / 805 GiB avail 2024-08-19T14:17:03.507 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:03 smithi031 ceph-mon[16336]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-19T14:17:03.507 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:03 smithi031 ceph-mon[16336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:17:03.507 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:03 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2352329776' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:17:03.508 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:03 smithi031 ceph-mon[16336]: osdmap e227: 12 total, 12 up, 9 in 2024-08-19T14:17:03.508 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {auth=26,config=2,health=12,kv=60} crc {auth=1738934069,config=2267314355,health=3897446751,kv=1392146778}) 2024-08-19T14:17:03.508 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=2622277944}) 2024-08-19T14:17:03.508 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=3117381722}) 2024-08-19T14:17:03.508 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {kv=5,logm=95} crc {kv=4225105953,logm=1768374466}) 2024-08-19T14:17:03.508 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1122785131}) 2024-08-19T14:17:03.508 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2976947255}) 2024-08-19T14:17:03.508 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1933929334}) 2024-08-19T14:17:03.508 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2745507459}) 2024-08-19T14:17:03.508 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3453536671}) 2024-08-19T14:17:03.508 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1840838618}) 2024-08-19T14:17:03.508 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4193173560}) 2024-08-19T14:17:03.508 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1314578655}) 2024-08-19T14:17:03.509 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=333399309}) 2024-08-19T14:17:03.509 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1595720842}) 2024-08-19T14:17:03.509 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=321683601}) 2024-08-19T14:17:03.509 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3490481030}) 2024-08-19T14:17:03.509 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3845395639}) 2024-08-19T14:17:03.509 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1421137131}) 2024-08-19T14:17:03.509 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2850101099}) 2024-08-19T14:17:03.509 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3767602861}) 2024-08-19T14:17:03.509 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1321721630}) 2024-08-19T14:17:03.509 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=448051810}) 2024-08-19T14:17:03.509 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3318402347}) 2024-08-19T14:17:03.509 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2762954482}) 2024-08-19T14:17:03.509 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2543995958}) 2024-08-19T14:17:03.509 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2358263533}) 2024-08-19T14:17:03.510 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3188054}) 2024-08-19T14:17:03.510 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3744189267}) 2024-08-19T14:17:03.510 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=778501199}) 2024-08-19T14:17:03.510 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1909465173}) 2024-08-19T14:17:03.510 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=84,mdsmap=3,mgr=13} crc {logm=2101507062,mdsmap=2474562478,mgr=2390531971}) 2024-08-19T14:17:03.510 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {mgr=5,mgr_command_descs=1,mgr_metadata=2,mgrstat=63,mon_config_key=29} crc {mgr=491192212,mgr_command_descs=1300230762,mgr_metadata=482233419,mgrstat=1928248053,mon_config_key=1808505436}) 2024-08-19T14:17:03.510 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2182784630}) 2024-08-19T14:17:03.510 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2038026944}) 2024-08-19T14:17:03.510 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=47,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=35} crc {mon_config_key=4007349174,monmap=3193798142,osd_metadata=3369745613,osd_pg_creating=2085363428,osdmap=4232502758}) 2024-08-19T14:17:03.510 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2990260415}) 2024-08-19T14:17:03.510 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2285297135}) 2024-08-19T14:17:03.510 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3945684529}) 2024-08-19T14:17:03.510 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=665336460}) 2024-08-19T14:17:03.511 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=22} crc {osdmap=2845182064}) 2024-08-19T14:17:03.511 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:03 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3191638218' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "3.a", "id": [6, 1, 4]}]: dispatch 2024-08-19T14:17:03.511 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:03 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2954180303' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:17:03.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:03.532 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:03.532 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:17:03.540 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:17:03.541 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:17:03.571 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:17:03.571 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:17:03.603 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:17:03.603 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:17:03.634 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:17:03.635 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:17:03.635 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:17:03.664 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:17:03.664 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:17:03.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:03.667 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:03.668 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:17:03.683 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:03 smithi045 ceph-mon[20143]: pgmap v540: 27 pgs: 27 active+clean; 577 KiB data, 1.3 GiB used, 803 GiB / 805 GiB avail 2024-08-19T14:17:03.684 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:03 smithi045 ceph-mon[20143]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-19T14:17:03.684 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:03 smithi045 ceph-mon[20143]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:17:03.684 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:03 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2352329776' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:17:03.684 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:03 smithi045 ceph-mon[20143]: osdmap e227: 12 total, 12 up, 9 in 2024-08-19T14:17:03.684 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {auth=26,config=2,health=12,kv=60} crc {auth=1738934069,config=2267314355,health=3897446751,kv=1392146778}) 2024-08-19T14:17:03.684 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=2622277944}) 2024-08-19T14:17:03.684 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=3117381722}) 2024-08-19T14:17:03.684 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {kv=5,logm=95} crc {kv=4225105953,logm=1768374466}) 2024-08-19T14:17:03.684 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1122785131}) 2024-08-19T14:17:03.684 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2976947255}) 2024-08-19T14:17:03.684 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1933929334}) 2024-08-19T14:17:03.684 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2745507459}) 2024-08-19T14:17:03.685 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3453536671}) 2024-08-19T14:17:03.685 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1840838618}) 2024-08-19T14:17:03.685 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4193173560}) 2024-08-19T14:17:03.685 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1314578655}) 2024-08-19T14:17:03.685 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=333399309}) 2024-08-19T14:17:03.685 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1595720842}) 2024-08-19T14:17:03.685 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=321683601}) 2024-08-19T14:17:03.685 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3490481030}) 2024-08-19T14:17:03.685 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3845395639}) 2024-08-19T14:17:03.685 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1421137131}) 2024-08-19T14:17:03.685 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2850101099}) 2024-08-19T14:17:03.685 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3767602861}) 2024-08-19T14:17:03.685 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1321721630}) 2024-08-19T14:17:03.685 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=448051810}) 2024-08-19T14:17:03.686 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3318402347}) 2024-08-19T14:17:03.686 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2762954482}) 2024-08-19T14:17:03.686 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2543995958}) 2024-08-19T14:17:03.686 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2358263533}) 2024-08-19T14:17:03.686 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3188054}) 2024-08-19T14:17:03.686 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3744189267}) 2024-08-19T14:17:03.686 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=778501199}) 2024-08-19T14:17:03.686 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1909465173}) 2024-08-19T14:17:03.686 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=84,mdsmap=3,mgr=13} crc {logm=2101507062,mdsmap=2474562478,mgr=2390531971}) 2024-08-19T14:17:03.686 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {mgr=5,mgr_command_descs=1,mgr_metadata=2,mgrstat=63,mon_config_key=29} crc {mgr=491192212,mgr_command_descs=1300230762,mgr_metadata=482233419,mgrstat=1928248053,mon_config_key=1808505436}) 2024-08-19T14:17:03.686 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2182784630}) 2024-08-19T14:17:03.686 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2038026944}) 2024-08-19T14:17:03.686 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=47,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=35} crc {mon_config_key=4007349174,monmap=3193798142,osd_metadata=3369745613,osd_pg_creating=2085363428,osdmap=4232502758}) 2024-08-19T14:17:03.686 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2990260415}) 2024-08-19T14:17:03.687 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2285297135}) 2024-08-19T14:17:03.687 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3945684529}) 2024-08-19T14:17:03.687 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=665336460}) 2024-08-19T14:17:03.687 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=22} crc {osdmap=2845182064}) 2024-08-19T14:17:03.687 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:03 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3191638218' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "3.a", "id": [6, 1, 4]}]: dispatch 2024-08-19T14:17:03.687 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:03 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2954180303' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:17:03.691 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:17:03.691 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:17:03.720 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:17:03.720 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:17:03.755 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:17:03.755 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:17:03.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:03 smithi032 ceph-mon[20004]: pgmap v540: 27 pgs: 27 active+clean; 577 KiB data, 1.3 GiB used, 803 GiB / 805 GiB avail 2024-08-19T14:17:03.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:03 smithi032 ceph-mon[20004]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-19T14:17:03.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:03 smithi032 ceph-mon[20004]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:17:03.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:03 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2352329776' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:17:03.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:03 smithi032 ceph-mon[20004]: osdmap e227: 12 total, 12 up, 9 in 2024-08-19T14:17:03.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {auth=26,config=2,health=12,kv=60} crc {auth=1738934069,config=2267314355,health=3897446751,kv=1392146778}) 2024-08-19T14:17:03.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=2622277944}) 2024-08-19T14:17:03.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=3117381722}) 2024-08-19T14:17:03.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {kv=5,logm=95} crc {kv=4225105953,logm=1768374466}) 2024-08-19T14:17:03.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1122785131}) 2024-08-19T14:17:03.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2976947255}) 2024-08-19T14:17:03.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1933929334}) 2024-08-19T14:17:03.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2745507459}) 2024-08-19T14:17:03.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3453536671}) 2024-08-19T14:17:03.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1840838618}) 2024-08-19T14:17:03.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4193173560}) 2024-08-19T14:17:03.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1314578655}) 2024-08-19T14:17:03.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=333399309}) 2024-08-19T14:17:03.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1595720842}) 2024-08-19T14:17:03.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=321683601}) 2024-08-19T14:17:03.760 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3490481030}) 2024-08-19T14:17:03.760 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3845395639}) 2024-08-19T14:17:03.760 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1421137131}) 2024-08-19T14:17:03.760 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2850101099}) 2024-08-19T14:17:03.760 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3767602861}) 2024-08-19T14:17:03.760 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1321721630}) 2024-08-19T14:17:03.760 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=448051810}) 2024-08-19T14:17:03.760 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3318402347}) 2024-08-19T14:17:03.760 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2762954482}) 2024-08-19T14:17:03.760 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2543995958}) 2024-08-19T14:17:03.760 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2358263533}) 2024-08-19T14:17:03.760 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3188054}) 2024-08-19T14:17:03.760 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3744189267}) 2024-08-19T14:17:03.761 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=778501199}) 2024-08-19T14:17:03.761 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1909465173}) 2024-08-19T14:17:03.761 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=84,mdsmap=3,mgr=13} crc {logm=2101507062,mdsmap=2474562478,mgr=2390531971}) 2024-08-19T14:17:03.761 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {mgr=5,mgr_command_descs=1,mgr_metadata=2,mgrstat=63,mon_config_key=29} crc {mgr=491192212,mgr_command_descs=1300230762,mgr_metadata=482233419,mgrstat=1928248053,mon_config_key=1808505436}) 2024-08-19T14:17:03.761 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2182784630}) 2024-08-19T14:17:03.761 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2038026944}) 2024-08-19T14:17:03.761 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=47,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=35} crc {mon_config_key=4007349174,monmap=3193798142,osd_metadata=3369745613,osd_pg_creating=2085363428,osdmap=4232502758}) 2024-08-19T14:17:03.761 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2990260415}) 2024-08-19T14:17:03.761 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2285297135}) 2024-08-19T14:17:03.761 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3945684529}) 2024-08-19T14:17:03.761 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=665336460}) 2024-08-19T14:17:03.761 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=22} crc {osdmap=2845182064}) 2024-08-19T14:17:03.761 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:03 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3191638218' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "3.a", "id": [6, 1, 4]}]: dispatch 2024-08-19T14:17:03.761 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:03 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2954180303' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:17:03.769 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:17:03.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:03.802 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:03.802 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:17:03.903 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:17:03.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:03.947 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:03.947 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:17:03.992 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:17:03.992 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [ 2024-08-19T14:17:03.992 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:03.992 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.25057.0:7287 3.d e228/222 3:b6dabb26:::benchmark_data_smithi045_59973_object7286:head v 228'456, mlcod=228'453)", 2024-08-19T14:17:03.992 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:17:03.990170+0000", 2024-08-19T14:17:03.992 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.00065472200000000005, 2024-08-19T14:17:03.992 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00069968199999999997, 2024-08-19T14:17:03.993 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:17:03.993 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:17:03.993 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:17:03.993 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:17:03.993 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:03.993 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:17:03.993 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:03.990170+0000", 2024-08-19T14:17:03.993 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:03.993 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:03.994 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:03.994 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:17:03.994 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:03.990170+0000", 2024-08-19T14:17:03.994 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:03.994 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:03.994 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:03.994 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:17:03.994 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:03.990171+0000", 2024-08-19T14:17:03.994 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.5099999999999999e-07 2024-08-19T14:17:03.994 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:03.994 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:03.994 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:17:03.994 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:03.990205+0000", 2024-08-19T14:17:03.995 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.3732000000000002e-05 2024-08-19T14:17:03.995 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:03.995 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:03.995 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:17:03.995 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:03.990220+0000", 2024-08-19T14:17:03.995 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5238e-05 2024-08-19T14:17:03.995 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:03.995 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:03.995 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:17:03.995 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:03.990228+0000", 2024-08-19T14:17:03.995 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.5939999999999996e-06 2024-08-19T14:17:03.995 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:03.995 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:03.995 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:17:03.995 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:03.990309+0000", 2024-08-19T14:17:03.996 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.1409999999999995e-05 2024-08-19T14:17:03.996 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:03.996 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:03.996 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:17:03.996 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:03.990326+0000", 2024-08-19T14:17:03.996 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7311000000000001e-05 2024-08-19T14:17:03.996 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:03.996 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:03.996 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:17:03.996 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:03.990839+0000", 2024-08-19T14:17:03.996 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00051291599999999998 2024-08-19T14:17:03.996 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:03.996 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:03.996 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:17:03.997 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:03.990888+0000", 2024-08-19T14:17:03.997 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.8436000000000002e-05 2024-08-19T14:17:03.997 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:03.997 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:17:03.997 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:03.997 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:03.997 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-08-19T14:17:03.997 INFO:teuthology.orchestra.run.smithi045.stdout: "num_ops": 1 2024-08-19T14:17:03.997 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:17:04.047 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:17:04.082 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:04.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:04.083 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:17:04.183 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:17:04.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:04.218 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:04.218 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:17:04.309 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 2 16 8686 8670 270.793 270.188 0.00348247 0.00368506 2024-08-19T14:17:04.320 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:17:04.356 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:04.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:04.357 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:17:04.458 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:17:04.489 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:04.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:04.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:17:04.590 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:17:04.626 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:04.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:04.627 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:17:04.698 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-08-19T14:17:04.718 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:04 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3191638218' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "3.a", "id": [6, 1, 4]}]': finished 2024-08-19T14:17:04.719 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:04 smithi045 ceph-mon[20143]: osdmap e228: 12 total, 12 up, 9 in 2024-08-19T14:17:04.728 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:17:04.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:04.750 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:04.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:17:04.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:04 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3191638218' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "3.a", "id": [6, 1, 4]}]': finished 2024-08-19T14:17:04.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:04 smithi032 ceph-mon[20004]: osdmap e228: 12 total, 12 up, 9 in 2024-08-19T14:17:04.852 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:17:04.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:04 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3191638218' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "3.a", "id": [6, 1, 4]}]': finished 2024-08-19T14:17:04.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:04 smithi031 ceph-mon[16336]: osdmap e228: 12 total, 12 up, 9 in 2024-08-19T14:17:04.886 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:04.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:04.887 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:17:04.988 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:17:05.030 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:05.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:05.033 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:17:05.033 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:17:05.133 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:17:05.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:05.173 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:05.174 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:17:05.275 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:17:05.309 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 3 16 12017 12001 249.924 208.188 0.00706674 0.00399503 2024-08-19T14:17:05.311 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:05.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:05.312 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:17:05.414 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:17:05.450 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:05.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:05.451 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:17:05.552 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:17:05.576 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:05.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:05.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:17:05.678 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:17:05.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:05.714 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:05.715 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:17:05.715 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:05 smithi032 ceph-mon[20004]: pgmap v543: 27 pgs: 27 active+clean; 85 MiB data, 1.4 GiB used, 803 GiB / 805 GiB avail; 16 MiB/s wr, 249 op/s 2024-08-19T14:17:05.715 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:05 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:17:05.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:05 smithi045 ceph-mon[20143]: pgmap v543: 27 pgs: 27 active+clean; 85 MiB data, 1.4 GiB used, 803 GiB / 805 GiB avail; 16 MiB/s wr, 249 op/s 2024-08-19T14:17:05.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:05 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:17:05.815 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:17:05.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:05.851 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:05.851 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:17:05.852 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:17:05.852 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [], 2024-08-19T14:17:05.852 INFO:teuthology.orchestra.run.smithi045.stdout: "complaint_time": 30, 2024-08-19T14:17:05.852 INFO:teuthology.orchestra.run.smithi045.stdout: "num_blocked_ops": 0 2024-08-19T14:17:05.852 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:17:05.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:05 smithi031 ceph-mon[16336]: pgmap v543: 27 pgs: 27 active+clean; 85 MiB data, 1.4 GiB used, 803 GiB / 805 GiB avail; 16 MiB/s wr, 249 op/s 2024-08-19T14:17:05.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:05 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:17:05.952 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:17:05.989 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:05.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:05.990 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:17:06.091 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:17:06.130 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:06.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:06.131 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:17:06.178 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-19T14:17:06.231 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:17:06.263 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:06.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:06.264 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:17:06.309 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 4 16 15476 15460 241.487 216.188 0.00352158 0.00413393 2024-08-19T14:17:06.364 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:17:06.399 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:06.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:06.400 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:17:06.501 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:17:06.539 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:06.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:06.540 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:17:06.559 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-08-19T14:17:06.577 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:17:06.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:06 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:17:06.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:06 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:17:06.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:06 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:17:06.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:06 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:17:06.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:06 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:17:06.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:06 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:17:06.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:06 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:17:06.641 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:17:06.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:06.672 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:06.672 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:17:06.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:06 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:17:06.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:06 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:17:06.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:06 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:17:06.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:06 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:17:06.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:06 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:17:06.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:06 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:17:06.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:06 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:17:06.773 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:17:06.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:06 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:17:06.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:06 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:17:06.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:06 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:17:06.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:06 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:17:06.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:06 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:17:06.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:06 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:17:06.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:06 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:17:06.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:06.810 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:06.810 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:17:06.912 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:17:06.944 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:06.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:06.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:17:07.030 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:17:07.046 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:17:07.070 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:17:07.079 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:07.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:07.080 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:17:07.181 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:17:07.218 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:07.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:07.219 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:17:07.310 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 5 16 19060 19044 237.986 224 0.00342146 0.00419744 2024-08-19T14:17:07.320 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:17:07.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:07.357 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:07.357 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:17:07.458 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:17:07.489 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:07.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:07.490 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:17:07.494 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:17:07.591 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:17:07.619 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:07.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:07.620 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:17:07.631 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:17:07.631 INFO:teuthology.orchestra.run.smithi045.stdout: "size": 20, 2024-08-19T14:17:07.631 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 600, 2024-08-19T14:17:07.631 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [ 2024-08-19T14:17:07.631 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.631 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24580.0:114944 2.d 2:beefb863:::benchmark_data_smithi045_46415_object114943:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-08-19T14:17:07.631 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:14:09.695253+0000", 2024-08-19T14:17:07.631 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 177.931357741, 2024-08-19T14:17:07.632 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.090495224999999999, 2024-08-19T14:17:07.632 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:17:07.632 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:17:07.632 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:17:07.632 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:17:07.632 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24580", 2024-08-19T14:17:07.632 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:17:07.632 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 114944 2024-08-19T14:17:07.632 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.632 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:17:07.632 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.632 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:17:07.632 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.695253+0000", 2024-08-19T14:17:07.633 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:07.633 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.633 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.633 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:17:07.633 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.695253+0000", 2024-08-19T14:17:07.633 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:07.633 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.633 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.633 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:17:07.633 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.695255+0000", 2024-08-19T14:17:07.633 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.418e-06 2024-08-19T14:17:07.633 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.633 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.634 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:17:07.634 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.695276+0000", 2024-08-19T14:17:07.634 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1501e-05 2024-08-19T14:17:07.634 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.634 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.634 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:17:07.634 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.695285+0000", 2024-08-19T14:17:07.634 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.0010000000000007e-06 2024-08-19T14:17:07.634 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.634 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.634 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:17:07.634 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.695292+0000", 2024-08-19T14:17:07.635 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.4470000000000003e-06 2024-08-19T14:17:07.635 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.635 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.635 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:17:07.635 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.695381+0000", 2024-08-19T14:17:07.635 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.9339999999999995e-05 2024-08-19T14:17:07.635 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.635 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.635 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:17:07.635 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.695523+0000", 2024-08-19T14:17:07.635 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014177100000000001 2024-08-19T14:17:07.635 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.635 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.636 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 3,4", 2024-08-19T14:17:07.636 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.695660+0000", 2024-08-19T14:17:07.636 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013723300000000001 2024-08-19T14:17:07.636 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.636 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.637 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:17:07.637 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.699148+0000", 2024-08-19T14:17:07.637 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0034881130000000002 2024-08-19T14:17:07.637 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.637 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.637 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:17:07.637 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.785564+0000", 2024-08-19T14:17:07.637 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.086416193000000002 2024-08-19T14:17:07.637 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.637 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.637 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:17:07.637 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.785623+0000", 2024-08-19T14:17:07.637 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.8495999999999997e-05 2024-08-19T14:17:07.638 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.638 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.638 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:17:07.638 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.785684+0000", 2024-08-19T14:17:07.638 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.1251e-05 2024-08-19T14:17:07.638 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.638 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.638 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:17:07.638 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.785748+0000", 2024-08-19T14:17:07.638 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.4461000000000005e-05 2024-08-19T14:17:07.638 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:07.638 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:17:07.638 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:07.639 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.639 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.639 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:115112 2.a e147/102 2:56fe95bd:::benchmark_data_smithi045_46415_object115111:head v 147'7223, mlcod=147'7220)", 2024-08-19T14:17:07.639 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:14:09.954245+0000", 2024-08-19T14:17:07.639 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 177.672365552, 2024-08-19T14:17:07.639 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.17428000299999999, 2024-08-19T14:17:07.639 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:17:07.639 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:17:07.639 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:17:07.639 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:17:07.639 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.639 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:17:07.639 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.954245+0000", 2024-08-19T14:17:07.639 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:07.640 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.640 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.640 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:17:07.640 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.954245+0000", 2024-08-19T14:17:07.640 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:07.640 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.640 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.640 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:17:07.640 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.954246+0000", 2024-08-19T14:17:07.640 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.1100000000000005e-07 2024-08-19T14:17:07.640 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.640 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.641 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:17:07.641 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.954272+0000", 2024-08-19T14:17:07.641 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.5551e-05 2024-08-19T14:17:07.641 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.641 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.641 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:17:07.641 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.954284+0000", 2024-08-19T14:17:07.641 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1717e-05 2024-08-19T14:17:07.641 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.641 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.641 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:17:07.641 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.954291+0000", 2024-08-19T14:17:07.641 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.5279999999999998e-06 2024-08-19T14:17:07.641 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.642 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.642 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:17:07.642 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.955487+0000", 2024-08-19T14:17:07.642 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001196428 2024-08-19T14:17:07.642 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.642 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.642 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:17:07.642 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.955510+0000", 2024-08-19T14:17:07.642 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2396e-05 2024-08-19T14:17:07.642 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.642 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.642 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:17:07.642 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:10.128404+0000", 2024-08-19T14:17:07.642 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.17289379499999999 2024-08-19T14:17:07.643 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.643 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.646 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:17:07.647 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:10.128525+0000", 2024-08-19T14:17:07.647 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000121777 2024-08-19T14:17:07.647 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:07.647 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:17:07.647 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:07.647 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.647 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.647 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:115113 2.a e147/102 2:57a3b2e5:::benchmark_data_smithi045_46415_object115112:head v 147'7224, mlcod=147'7220)", 2024-08-19T14:17:07.647 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:14:09.957896+0000", 2024-08-19T14:17:07.647 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 177.66871469200001, 2024-08-19T14:17:07.648 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.17072668099999999, 2024-08-19T14:17:07.648 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:17:07.648 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:17:07.648 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:17:07.648 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:17:07.648 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.648 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:17:07.648 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.957896+0000", 2024-08-19T14:17:07.648 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:07.648 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.648 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.648 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:17:07.649 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.957896+0000", 2024-08-19T14:17:07.649 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:07.649 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.649 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.649 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:17:07.649 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.957897+0000", 2024-08-19T14:17:07.649 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.8999999999999995e-07 2024-08-19T14:17:07.649 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.649 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.649 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:17:07.649 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.957925+0000", 2024-08-19T14:17:07.649 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.7433e-05 2024-08-19T14:17:07.649 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.649 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.650 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:17:07.650 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.957936+0000", 2024-08-19T14:17:07.650 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1337e-05 2024-08-19T14:17:07.650 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.650 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.650 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:17:07.650 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.957942+0000", 2024-08-19T14:17:07.650 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.6170000000000003e-06 2024-08-19T14:17:07.650 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.650 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.650 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:17:07.650 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.959198+0000", 2024-08-19T14:17:07.651 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0012551490000000001 2024-08-19T14:17:07.651 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.651 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.651 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:17:07.651 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.959218+0000", 2024-08-19T14:17:07.651 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0621000000000001e-05 2024-08-19T14:17:07.651 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.651 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.651 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:17:07.651 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:10.128532+0000", 2024-08-19T14:17:07.651 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.169313305 2024-08-19T14:17:07.651 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.652 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.652 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:17:07.652 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:10.128623+0000", 2024-08-19T14:17:07.652 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.1428999999999995e-05 2024-08-19T14:17:07.652 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:07.652 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:17:07.652 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:07.652 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.652 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.652 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:115120 2.f e147/102 2:f135df33:::benchmark_data_smithi045_46415_object115119:head v 147'7250, mlcod=147'7248)", 2024-08-19T14:17:07.652 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:14:09.960501+0000", 2024-08-19T14:17:07.652 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 177.66610999599999, 2024-08-19T14:17:07.653 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.15682758499999999, 2024-08-19T14:17:07.653 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:17:07.653 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:17:07.653 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:17:07.653 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:17:07.653 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.653 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:17:07.653 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.960501+0000", 2024-08-19T14:17:07.653 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:07.653 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.653 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.654 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:17:07.654 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.960501+0000", 2024-08-19T14:17:07.654 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:07.654 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.654 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.654 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:17:07.654 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.960502+0000", 2024-08-19T14:17:07.654 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.5199999999999995e-07 2024-08-19T14:17:07.654 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.654 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.654 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:17:07.654 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.960531+0000", 2024-08-19T14:17:07.654 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.9034e-05 2024-08-19T14:17:07.655 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.655 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.655 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:17:07.655 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.960541+0000", 2024-08-19T14:17:07.655 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0596e-05 2024-08-19T14:17:07.655 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.655 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.655 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:17:07.655 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.960548+0000", 2024-08-19T14:17:07.655 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.99e-06 2024-08-19T14:17:07.655 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.655 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.655 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:17:07.656 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.960622+0000", 2024-08-19T14:17:07.656 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.3458e-05 2024-08-19T14:17:07.656 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.656 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.656 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:17:07.656 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.960641+0000", 2024-08-19T14:17:07.656 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9032e-05 2024-08-19T14:17:07.656 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.656 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.656 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:17:07.656 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:10.117268+0000", 2024-08-19T14:17:07.657 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.156626723 2024-08-19T14:17:07.657 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.657 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.657 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:17:07.657 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:10.117329+0000", 2024-08-19T14:17:07.657 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.0900000000000003e-05 2024-08-19T14:17:07.657 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:07.657 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:17:07.657 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:07.657 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.657 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.657 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24580.0:115126 2.d 2:b84de1a6:::benchmark_data_smithi045_46415_object115125:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-08-19T14:17:07.657 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:14:09.961680+0000", 2024-08-19T14:17:07.658 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 177.66493076699999, 2024-08-19T14:17:07.658 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.17432350099999999, 2024-08-19T14:17:07.658 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:17:07.658 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:17:07.658 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:17:07.658 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:17:07.658 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24580", 2024-08-19T14:17:07.658 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:17:07.658 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 115126 2024-08-19T14:17:07.658 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.658 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:17:07.658 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.658 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:17:07.659 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.961680+0000", 2024-08-19T14:17:07.659 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:07.659 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.659 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.659 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:17:07.659 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.961680+0000", 2024-08-19T14:17:07.659 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:07.659 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.659 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.659 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:17:07.659 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.961682+0000", 2024-08-19T14:17:07.659 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6959999999999999e-06 2024-08-19T14:17:07.659 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.659 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.660 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:17:07.660 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.961705+0000", 2024-08-19T14:17:07.660 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.3292e-05 2024-08-19T14:17:07.660 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.660 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.660 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:17:07.660 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.961717+0000", 2024-08-19T14:17:07.660 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1976e-05 2024-08-19T14:17:07.660 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.660 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.660 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:17:07.660 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.961725+0000", 2024-08-19T14:17:07.660 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.7519999999999996e-06 2024-08-19T14:17:07.660 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.660 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.661 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:17:07.661 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:10.119167+0000", 2024-08-19T14:17:07.661 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.157442057 2024-08-19T14:17:07.661 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.661 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.661 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:17:07.661 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:10.119308+0000", 2024-08-19T14:17:07.661 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000140798 2024-08-19T14:17:07.661 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.661 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.661 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 3,4", 2024-08-19T14:17:07.661 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:10.119454+0000", 2024-08-19T14:17:07.661 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014580699999999999 2024-08-19T14:17:07.662 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.662 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.663 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:17:07.663 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:10.131417+0000", 2024-08-19T14:17:07.663 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.011963055 2024-08-19T14:17:07.663 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.663 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.663 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:17:07.663 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:10.131458+0000", 2024-08-19T14:17:07.663 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.1291000000000002e-05 2024-08-19T14:17:07.663 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.663 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.664 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:17:07.664 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:10.135887+0000", 2024-08-19T14:17:07.664 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0044292050000000003 2024-08-19T14:17:07.664 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.664 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.664 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:17:07.664 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:10.135942+0000", 2024-08-19T14:17:07.664 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4707999999999999e-05 2024-08-19T14:17:07.664 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.665 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.665 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:17:07.665 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:10.136004+0000", 2024-08-19T14:17:07.665 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.1864000000000003e-05 2024-08-19T14:17:07.665 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:07.665 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:17:07.665 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:07.665 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.665 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.665 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:115124 2.f e147/102 2:f4df7718:::benchmark_data_smithi045_46415_object115123:head v 147'7251, mlcod=147'7248)", 2024-08-19T14:17:07.666 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:14:09.964066+0000", 2024-08-19T14:17:07.666 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 177.66254450599999, 2024-08-19T14:17:07.666 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.15330069800000001, 2024-08-19T14:17:07.666 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:17:07.666 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:17:07.666 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:17:07.666 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:17:07.666 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.666 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:17:07.666 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.964066+0000", 2024-08-19T14:17:07.666 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:07.667 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.667 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.667 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:17:07.667 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.964066+0000", 2024-08-19T14:17:07.667 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:07.667 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.667 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.667 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:17:07.668 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is set 2024-08-19T14:17:07.669 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.964067+0000", 2024-08-19T14:17:07.670 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4000000000000001e-07 2024-08-19T14:17:07.670 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.670 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.670 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:17:07.671 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.964100+0000", 2024-08-19T14:17:07.671 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.2298000000000001e-05 2024-08-19T14:17:07.671 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.671 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.671 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:17:07.671 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.964110+0000", 2024-08-19T14:17:07.671 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0193e-05 2024-08-19T14:17:07.671 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.671 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.672 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:17:07.672 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.964117+0000", 2024-08-19T14:17:07.672 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.827e-06 2024-08-19T14:17:07.672 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.672 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.672 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:17:07.673 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.964209+0000", 2024-08-19T14:17:07.673 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.2227999999999999e-05 2024-08-19T14:17:07.673 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.673 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.673 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:17:07.674 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.964228+0000", 2024-08-19T14:17:07.674 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9381999999999998e-05 2024-08-19T14:17:07.674 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.674 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.674 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:17:07.674 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:10.117338+0000", 2024-08-19T14:17:07.674 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.15311018000000001 2024-08-19T14:17:07.674 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.674 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.675 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:17:07.675 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:10.117367+0000", 2024-08-19T14:17:07.675 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.885e-05 2024-08-19T14:17:07.675 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:07.675 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:17:07.675 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:07.675 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.675 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.675 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24580.0:115129 2.d 2:b66d07ae:::benchmark_data_smithi045_46415_object115128:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-08-19T14:17:07.675 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:14:09.965238+0000", 2024-08-19T14:17:07.675 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 177.661373475, 2024-08-19T14:17:07.675 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.17130973899999999, 2024-08-19T14:17:07.676 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:17:07.676 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:17:07.676 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:17:07.676 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:17:07.676 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24580", 2024-08-19T14:17:07.676 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:17:07.676 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 115129 2024-08-19T14:17:07.677 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.677 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:17:07.677 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.678 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:17:07.678 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.965238+0000", 2024-08-19T14:17:07.678 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:07.678 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.678 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.678 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:17:07.678 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.965238+0000", 2024-08-19T14:17:07.679 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:07.679 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.679 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.679 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:17:07.679 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.965239+0000", 2024-08-19T14:17:07.679 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8160000000000001e-06 2024-08-19T14:17:07.679 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.679 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.679 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:17:07.679 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.965263+0000", 2024-08-19T14:17:07.680 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.3937000000000001e-05 2024-08-19T14:17:07.680 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.680 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.680 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:17:07.680 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.965275+0000", 2024-08-19T14:17:07.680 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2009e-05 2024-08-19T14:17:07.680 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.680 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.680 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:17:07.680 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.965283+0000", 2024-08-19T14:17:07.681 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4189999999999996e-06 2024-08-19T14:17:07.681 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.681 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.682 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:17:07.682 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:10.122985+0000", 2024-08-19T14:17:07.682 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.157702077 2024-08-19T14:17:07.682 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.682 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.682 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:17:07.682 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:10.123131+0000", 2024-08-19T14:17:07.682 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014671500000000001 2024-08-19T14:17:07.683 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.683 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.683 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 3,4", 2024-08-19T14:17:07.683 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:10.123284+0000", 2024-08-19T14:17:07.683 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015256199999999999 2024-08-19T14:17:07.683 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.683 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.683 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:17:07.683 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:10.131497+0000", 2024-08-19T14:17:07.684 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0082132809999999994 2024-08-19T14:17:07.684 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.684 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.684 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:17:07.684 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:10.135830+0000", 2024-08-19T14:17:07.684 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0043327130000000002 2024-08-19T14:17:07.684 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.684 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.684 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:17:07.684 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:10.136011+0000", 2024-08-19T14:17:07.684 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00018142399999999999 2024-08-19T14:17:07.684 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.684 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.684 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:17:07.685 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:10.136064+0000", 2024-08-19T14:17:07.685 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2139999999999999e-05 2024-08-19T14:17:07.685 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.685 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.685 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:17:07.685 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:10.136547+0000", 2024-08-19T14:17:07.685 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00048364600000000001 2024-08-19T14:17:07.685 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:07.685 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:17:07.685 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:07.685 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.685 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.685 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:115128 2.0 e147/132 2:08e033d8:::benchmark_data_smithi045_46415_object115127:head v 147'7222, mlcod=147'7220)", 2024-08-19T14:17:07.686 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:14:09.965305+0000", 2024-08-19T14:17:07.686 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 177.661306346, 2024-08-19T14:17:07.686 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.152300509, 2024-08-19T14:17:07.686 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:17:07.686 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:17:07.686 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:17:07.686 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:17:07.686 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.686 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:17:07.686 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.965305+0000", 2024-08-19T14:17:07.686 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:07.686 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.686 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.686 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:17:07.687 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.965305+0000", 2024-08-19T14:17:07.687 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:07.687 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.687 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.687 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:17:07.687 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.965305+0000", 2024-08-19T14:17:07.687 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2300000000000001e-07 2024-08-19T14:17:07.687 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.687 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.687 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:17:07.687 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.965331+0000", 2024-08-19T14:17:07.687 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.5709999999999999e-05 2024-08-19T14:17:07.688 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.688 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.688 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:17:07.688 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.965339+0000", 2024-08-19T14:17:07.688 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.8199999999999997e-06 2024-08-19T14:17:07.688 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.688 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.688 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:17:07.688 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.965344+0000", 2024-08-19T14:17:07.688 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1449999999999999e-06 2024-08-19T14:17:07.688 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.688 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.688 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:17:07.688 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.965409+0000", 2024-08-19T14:17:07.689 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.5573000000000004e-05 2024-08-19T14:17:07.689 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.689 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.689 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:17:07.689 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.965428+0000", 2024-08-19T14:17:07.689 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9038e-05 2024-08-19T14:17:07.689 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.689 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.689 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:17:07.689 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:10.117559+0000", 2024-08-19T14:17:07.689 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.15213016200000001 2024-08-19T14:17:07.689 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.689 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.689 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:17:07.690 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:10.117605+0000", 2024-08-19T14:17:07.690 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.6538000000000001e-05 2024-08-19T14:17:07.690 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:07.690 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:17:07.690 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:07.690 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.690 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.690 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:115127 2.f e147/102 2:fb596b92:::benchmark_data_smithi045_46415_object115126:head v 147'7252, mlcod=147'7248)", 2024-08-19T14:17:07.690 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:14:09.967625+0000", 2024-08-19T14:17:07.690 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 177.65898643599999, 2024-08-19T14:17:07.690 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.16869712100000001, 2024-08-19T14:17:07.690 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:17:07.690 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:17:07.691 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:17:07.691 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:17:07.691 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.691 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:17:07.691 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.967625+0000", 2024-08-19T14:17:07.691 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:07.691 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.691 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.692 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:17:07.692 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.967625+0000", 2024-08-19T14:17:07.692 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:07.692 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.692 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.692 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:17:07.692 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.967625+0000", 2024-08-19T14:17:07.692 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.5799999999999998e-07 2024-08-19T14:17:07.692 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.692 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.692 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:17:07.692 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.967650+0000", 2024-08-19T14:17:07.693 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.4702000000000002e-05 2024-08-19T14:17:07.693 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.693 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.693 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:17:07.693 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:10.117223+0000", 2024-08-19T14:17:07.693 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.149573331 2024-08-19T14:17:07.693 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.693 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.693 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:17:07.694 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:10.117232+0000", 2024-08-19T14:17:07.694 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.901e-06 2024-08-19T14:17:07.694 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.694 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.694 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:17:07.694 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:10.117397+0000", 2024-08-19T14:17:07.694 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00016443 2024-08-19T14:17:07.694 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.694 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.694 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:17:07.694 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:10.117417+0000", 2024-08-19T14:17:07.694 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0259e-05 2024-08-19T14:17:07.694 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.695 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.695 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:17:07.695 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:10.136126+0000", 2024-08-19T14:17:07.695 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.018709078000000001 2024-08-19T14:17:07.695 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.695 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.695 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:17:07.695 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:10.136322+0000", 2024-08-19T14:17:07.696 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00019566200000000001 2024-08-19T14:17:07.696 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:07.696 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:17:07.696 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:07.696 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.696 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.696 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:115136 2.0 e147/132 2:02df7cb0:::benchmark_data_smithi045_46415_object115135:head v 147'7223, mlcod=147'7220)", 2024-08-19T14:17:07.696 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:14:09.971911+0000", 2024-08-19T14:17:07.696 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 177.654700437, 2024-08-19T14:17:07.696 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.152436297, 2024-08-19T14:17:07.697 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:17:07.697 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:17:07.697 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:17:07.697 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:17:07.697 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.697 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:17:07.697 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.971911+0000", 2024-08-19T14:17:07.697 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:07.697 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.697 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.697 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:17:07.698 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.971911+0000", 2024-08-19T14:17:07.698 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:07.698 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.698 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.698 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:17:07.698 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.971911+0000", 2024-08-19T14:17:07.698 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.4600000000000003e-07 2024-08-19T14:17:07.698 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.698 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.698 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:17:07.698 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.972026+0000", 2024-08-19T14:17:07.699 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000114629 2024-08-19T14:17:07.699 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.699 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.699 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:17:07.699 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:10.117028+0000", 2024-08-19T14:17:07.699 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.145002193 2024-08-19T14:17:07.699 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.699 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.699 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:17:07.699 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:10.117039+0000", 2024-08-19T14:17:07.699 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0618000000000001e-05 2024-08-19T14:17:07.699 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.699 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.700 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:17:07.700 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:10.117632+0000", 2024-08-19T14:17:07.700 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00059258399999999995 2024-08-19T14:17:07.700 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.700 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.700 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:17:07.700 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:10.117652+0000", 2024-08-19T14:17:07.700 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0975000000000002e-05 2024-08-19T14:17:07.700 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.700 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.700 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:17:07.700 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:10.124240+0000", 2024-08-19T14:17:07.700 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0065879629999999996 2024-08-19T14:17:07.700 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.701 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.701 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:17:07.701 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:10.124347+0000", 2024-08-19T14:17:07.701 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000106389 2024-08-19T14:17:07.701 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:07.701 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:17:07.701 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:07.701 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.701 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.702 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24580.0:116074 2.d 2:b7e42e23:::benchmark_data_smithi045_46415_object116073:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-08-19T14:17:07.702 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:14:11.206338+0000", 2024-08-19T14:17:07.702 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 176.42027275800001, 2024-08-19T14:17:07.702 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.115659096, 2024-08-19T14:17:07.702 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:17:07.702 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:17:07.702 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:17:07.702 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:17:07.702 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24580", 2024-08-19T14:17:07.702 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:17:07.702 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 116074 2024-08-19T14:17:07.702 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.703 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:17:07.703 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.703 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:17:07.703 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:11.206338+0000", 2024-08-19T14:17:07.703 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:07.703 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.703 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.703 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:17:07.703 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:11.206338+0000", 2024-08-19T14:17:07.703 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:07.703 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.703 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.703 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:17:07.703 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:11.206340+0000", 2024-08-19T14:17:07.704 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.672e-06 2024-08-19T14:17:07.704 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.704 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.704 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:17:07.704 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:11.206363+0000", 2024-08-19T14:17:07.704 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2929e-05 2024-08-19T14:17:07.704 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.704 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.704 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:17:07.704 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:11.206376+0000", 2024-08-19T14:17:07.704 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3044000000000001e-05 2024-08-19T14:17:07.704 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.704 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.704 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:17:07.705 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:11.206384+0000", 2024-08-19T14:17:07.705 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.8089999999999993e-06 2024-08-19T14:17:07.705 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.705 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.705 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:17:07.705 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:11.314413+0000", 2024-08-19T14:17:07.705 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.108029721 2024-08-19T14:17:07.705 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.705 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.705 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:17:07.705 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:11.314553+0000", 2024-08-19T14:17:07.705 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013989800000000001 2024-08-19T14:17:07.705 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.705 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.705 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 3,4", 2024-08-19T14:17:07.706 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:11.314711+0000", 2024-08-19T14:17:07.706 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015746099999999999 2024-08-19T14:17:07.706 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.706 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.706 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:17:07.706 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:11.321791+0000", 2024-08-19T14:17:07.706 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0070803680000000001 2024-08-19T14:17:07.706 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.706 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.706 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:17:07.706 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:11.321804+0000", 2024-08-19T14:17:07.706 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2408999999999999e-05 2024-08-19T14:17:07.706 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.707 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.707 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:17:07.707 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:11.321866+0000", 2024-08-19T14:17:07.707 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.2206999999999996e-05 2024-08-19T14:17:07.707 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.707 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.707 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:17:07.707 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:11.321937+0000", 2024-08-19T14:17:07.708 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.1352000000000004e-05 2024-08-19T14:17:07.708 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.708 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.708 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:17:07.708 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:11.321997+0000", 2024-08-19T14:17:07.708 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.0226000000000002e-05 2024-08-19T14:17:07.708 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:07.708 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:17:07.708 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:07.708 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.708 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.708 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:116084 2.f e147/102 2:f80ab7f0:::benchmark_data_smithi045_46415_object116083:head v 147'7308, mlcod=147'7306)", 2024-08-19T14:17:07.708 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:14:11.221797+0000", 2024-08-19T14:17:07.709 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 176.40481422900001, 2024-08-19T14:17:07.709 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.096186603999999995, 2024-08-19T14:17:07.709 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:17:07.709 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:17:07.709 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:17:07.709 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:17:07.709 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.709 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:17:07.709 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:11.221797+0000", 2024-08-19T14:17:07.709 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:07.709 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.709 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.709 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:17:07.709 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:11.221797+0000", 2024-08-19T14:17:07.710 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:07.710 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.710 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.710 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:17:07.710 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:11.221798+0000", 2024-08-19T14:17:07.710 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.8700000000000004e-07 2024-08-19T14:17:07.710 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.710 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.710 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:17:07.710 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:11.221830+0000", 2024-08-19T14:17:07.710 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.2449000000000001e-05 2024-08-19T14:17:07.710 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.710 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.711 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:17:07.711 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:11.312366+0000", 2024-08-19T14:17:07.711 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.090535632000000005 2024-08-19T14:17:07.711 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.711 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.711 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:17:07.711 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:11.312381+0000", 2024-08-19T14:17:07.711 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5251e-05 2024-08-19T14:17:07.711 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.711 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.712 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:17:07.712 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:11.312494+0000", 2024-08-19T14:17:07.712 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00011322799999999999 2024-08-19T14:17:07.712 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.712 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.712 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:17:07.712 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:11.312515+0000", 2024-08-19T14:17:07.712 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0729000000000001e-05 2024-08-19T14:17:07.712 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.712 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.712 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:17:07.712 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:11.317939+0000", 2024-08-19T14:17:07.713 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0054243800000000003 2024-08-19T14:17:07.713 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.713 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.713 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:17:07.713 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:11.317983+0000", 2024-08-19T14:17:07.713 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.4048000000000002e-05 2024-08-19T14:17:07.713 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:07.713 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:17:07.713 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:07.713 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.713 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.713 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:116085 2.8 e147/102 2:1d66bede:::benchmark_data_smithi045_46415_object116084:head v 147'7307, mlcod=147'7305)", 2024-08-19T14:17:07.713 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:14:11.222411+0000", 2024-08-19T14:17:07.714 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 176.40420002100001, 2024-08-19T14:17:07.714 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.101047811, 2024-08-19T14:17:07.714 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:17:07.714 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:17:07.714 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:17:07.714 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:17:07.714 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.714 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:17:07.714 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:11.222411+0000", 2024-08-19T14:17:07.714 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:07.714 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.714 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.714 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:17:07.714 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:11.222411+0000", 2024-08-19T14:17:07.715 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:07.715 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.715 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.715 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:17:07.715 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:11.222412+0000", 2024-08-19T14:17:07.715 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.7400000000000002e-07 2024-08-19T14:17:07.715 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.716 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.716 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:17:07.716 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:11.222449+0000", 2024-08-19T14:17:07.716 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.7537000000000003e-05 2024-08-19T14:17:07.716 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.716 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.716 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:17:07.716 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:11.313595+0000", 2024-08-19T14:17:07.716 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.091146164000000002 2024-08-19T14:17:07.716 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.716 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.716 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:17:07.716 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:11.313605+0000", 2024-08-19T14:17:07.717 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.9639999999999998e-06 2024-08-19T14:17:07.717 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.717 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.717 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:17:07.717 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:11.314857+0000", 2024-08-19T14:17:07.717 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001251928 2024-08-19T14:17:07.717 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.717 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.717 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:17:07.717 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:11.314875+0000", 2024-08-19T14:17:07.717 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8148e-05 2024-08-19T14:17:07.717 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.717 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.717 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:17:07.718 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:11.323421+0000", 2024-08-19T14:17:07.718 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0085454009999999993 2024-08-19T14:17:07.718 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.718 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.718 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:17:07.718 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:11.323459+0000", 2024-08-19T14:17:07.718 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.7895000000000002e-05 2024-08-19T14:17:07.718 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:07.718 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:17:07.718 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:07.718 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.718 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.718 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:116087 2.e e147/102 2:77b89bfd:::benchmark_data_smithi045_46415_object116086:head v 147'7225, mlcod=147'7223)", 2024-08-19T14:17:07.719 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:14:11.226986+0000", 2024-08-19T14:17:07.719 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 176.399624918, 2024-08-19T14:17:07.719 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.091217643000000001, 2024-08-19T14:17:07.719 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:17:07.719 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:17:07.719 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:17:07.719 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:17:07.719 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.719 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:17:07.719 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:11.226986+0000", 2024-08-19T14:17:07.719 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:07.719 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.719 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.719 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:17:07.720 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:11.226986+0000", 2024-08-19T14:17:07.720 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:07.720 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.720 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.720 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:17:07.720 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:11.226987+0000", 2024-08-19T14:17:07.720 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.8999999999999996e-07 2024-08-19T14:17:07.720 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.720 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.720 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:17:07.720 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:11.227037+0000", 2024-08-19T14:17:07.720 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.9908999999999999e-05 2024-08-19T14:17:07.720 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.720 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.721 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:17:07.721 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:11.312758+0000", 2024-08-19T14:17:07.721 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.085721669 2024-08-19T14:17:07.721 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.721 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.721 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:17:07.721 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:11.312769+0000", 2024-08-19T14:17:07.721 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0593e-05 2024-08-19T14:17:07.721 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.721 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.721 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:17:07.721 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:11.312855+0000", 2024-08-19T14:17:07.721 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.6515999999999996e-05 2024-08-19T14:17:07.721 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.722 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.722 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:17:07.722 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:11.312876+0000", 2024-08-19T14:17:07.722 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0075999999999999e-05 2024-08-19T14:17:07.722 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.722 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.722 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:17:07.722 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:11.318132+0000", 2024-08-19T14:17:07.723 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:17:07.724 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.005256167 2024-08-19T14:17:07.724 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.724 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.724 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:17:07.724 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:11.318204+0000", 2024-08-19T14:17:07.724 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.2022999999999998e-05 2024-08-19T14:17:07.724 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:07.724 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:17:07.725 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:07.725 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.725 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.725 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24580.0:116730 2.d 2:b8d5bc84:::benchmark_data_smithi045_46415_object116729:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-08-19T14:17:07.725 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:14:12.005889+0000", 2024-08-19T14:17:07.725 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 175.62072192900001, 2024-08-19T14:17:07.725 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.090255616999999996, 2024-08-19T14:17:07.725 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:17:07.725 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:17:07.725 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:17:07.725 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:17:07.725 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24580", 2024-08-19T14:17:07.725 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:17:07.726 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 116730 2024-08-19T14:17:07.726 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.726 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:17:07.726 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.726 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:17:07.726 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:12.005889+0000", 2024-08-19T14:17:07.726 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:07.726 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.726 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.726 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:17:07.726 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:12.005889+0000", 2024-08-19T14:17:07.726 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:07.726 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.726 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.727 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:17:07.727 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:12.005891+0000", 2024-08-19T14:17:07.727 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5340000000000001e-06 2024-08-19T14:17:07.727 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.727 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.727 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:17:07.727 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:12.005911+0000", 2024-08-19T14:17:07.727 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0069e-05 2024-08-19T14:17:07.727 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.727 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.727 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:17:07.727 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:12.005924+0000", 2024-08-19T14:17:07.727 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3040000000000001e-05 2024-08-19T14:17:07.727 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.728 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.728 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:17:07.728 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:12.005931+0000", 2024-08-19T14:17:07.728 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.0199999999999997e-06 2024-08-19T14:17:07.728 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.728 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.728 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:17:07.728 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:12.005973+0000", 2024-08-19T14:17:07.728 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.2058000000000003e-05 2024-08-19T14:17:07.728 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.728 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.728 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:17:07.728 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:12.006047+0000", 2024-08-19T14:17:07.729 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.3918999999999995e-05 2024-08-19T14:17:07.729 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.729 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.729 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 3,4", 2024-08-19T14:17:07.729 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:12.006125+0000", 2024-08-19T14:17:07.729 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.8345999999999998e-05 2024-08-19T14:17:07.729 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.729 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.729 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:17:07.729 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:12.095982+0000", 2024-08-19T14:17:07.729 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.089856806999999997 2024-08-19T14:17:07.729 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.729 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.729 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:17:07.730 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:12.096006+0000", 2024-08-19T14:17:07.730 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.3818000000000001e-05 2024-08-19T14:17:07.730 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.730 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.730 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:17:07.730 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:12.096044+0000", 2024-08-19T14:17:07.730 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.8557999999999999e-05 2024-08-19T14:17:07.730 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.730 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.730 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:17:07.730 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:12.096094+0000", 2024-08-19T14:17:07.730 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0271e-05 2024-08-19T14:17:07.730 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.730 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.731 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:17:07.731 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:12.096145+0000", 2024-08-19T14:17:07.731 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0176999999999997e-05 2024-08-19T14:17:07.731 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:07.731 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:17:07.731 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:07.731 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.731 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.731 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:116728 2.a e147/102 2:5713d570:::benchmark_data_smithi045_46415_object116727:head v 147'7318, mlcod=147'7316)", 2024-08-19T14:17:07.731 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:14:12.006222+0000", 2024-08-19T14:17:07.731 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 175.62038867300001, 2024-08-19T14:17:07.731 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.091060481999999998, 2024-08-19T14:17:07.731 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:17:07.731 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:17:07.732 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:17:07.732 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:17:07.732 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.732 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:17:07.732 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:12.006222+0000", 2024-08-19T14:17:07.732 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:07.732 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.732 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.732 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:17:07.733 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:12.006222+0000", 2024-08-19T14:17:07.733 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:07.733 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.733 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.733 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:17:07.733 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:12.006224+0000", 2024-08-19T14:17:07.733 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.978e-06 2024-08-19T14:17:07.733 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.733 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.733 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:17:07.733 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:12.006254+0000", 2024-08-19T14:17:07.733 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.9734999999999999e-05 2024-08-19T14:17:07.733 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.733 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.734 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:17:07.734 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:12.006265+0000", 2024-08-19T14:17:07.734 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1127e-05 2024-08-19T14:17:07.734 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.734 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.734 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:17:07.734 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:12.006271+0000", 2024-08-19T14:17:07.734 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.2319999999999998e-06 2024-08-19T14:17:07.734 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.734 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.734 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:17:07.734 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:12.006320+0000", 2024-08-19T14:17:07.734 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.8770000000000002e-05 2024-08-19T14:17:07.735 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.735 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.735 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:17:07.735 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:12.006338+0000", 2024-08-19T14:17:07.735 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7948000000000002e-05 2024-08-19T14:17:07.735 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.735 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.735 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:17:07.735 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:12.097248+0000", 2024-08-19T14:17:07.735 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.090909918000000006 2024-08-19T14:17:07.735 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.735 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.735 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:17:07.735 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:12.097283+0000", 2024-08-19T14:17:07.736 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.4774000000000003e-05 2024-08-19T14:17:07.736 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:07.736 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:17:07.736 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:07.736 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.736 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.736 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24580.0:116732 2.d 2:bd0455f5:::benchmark_data_smithi045_46415_object116731:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-08-19T14:17:07.736 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:14:12.006637+0000", 2024-08-19T14:17:07.736 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 175.619973781, 2024-08-19T14:17:07.736 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.10503797400000001, 2024-08-19T14:17:07.736 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:17:07.736 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:17:07.736 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:17:07.736 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:17:07.737 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24580", 2024-08-19T14:17:07.737 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:17:07.737 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 116732 2024-08-19T14:17:07.737 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.737 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:17:07.737 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.737 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:17:07.737 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:12.006637+0000", 2024-08-19T14:17:07.737 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:07.737 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.737 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.737 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:17:07.737 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:12.006637+0000", 2024-08-19T14:17:07.737 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:07.738 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.738 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.738 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:17:07.738 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:12.006638+0000", 2024-08-19T14:17:07.738 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.8100000000000001e-07 2024-08-19T14:17:07.738 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.738 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.738 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:17:07.738 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:12.006659+0000", 2024-08-19T14:17:07.738 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1398000000000001e-05 2024-08-19T14:17:07.739 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.739 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.739 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:17:07.739 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:12.006670+0000", 2024-08-19T14:17:07.739 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0903999999999999e-05 2024-08-19T14:17:07.739 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.739 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.739 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:17:07.739 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:12.006677+0000", 2024-08-19T14:17:07.739 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.8210000000000004e-06 2024-08-19T14:17:07.739 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.739 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.739 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:17:07.739 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:12.091573+0000", 2024-08-19T14:17:07.740 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.084895913000000003 2024-08-19T14:17:07.740 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.740 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.740 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:17:07.740 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:12.091715+0000", 2024-08-19T14:17:07.740 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000141416 2024-08-19T14:17:07.740 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.740 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.740 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 3,4", 2024-08-19T14:17:07.740 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:12.091964+0000", 2024-08-19T14:17:07.740 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00024988000000000001 2024-08-19T14:17:07.740 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.740 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.740 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:17:07.741 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:12.100726+0000", 2024-08-19T14:17:07.741 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.008761507 2024-08-19T14:17:07.741 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.741 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.741 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:17:07.741 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:12.103900+0000", 2024-08-19T14:17:07.741 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.003173978 2024-08-19T14:17:07.741 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.741 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.741 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:17:07.741 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:12.111566+0000", 2024-08-19T14:17:07.741 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0076658960000000002 2024-08-19T14:17:07.741 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.742 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.742 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:17:07.742 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:12.111627+0000", 2024-08-19T14:17:07.742 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.0887999999999997e-05 2024-08-19T14:17:07.742 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.742 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.742 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:17:07.742 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:12.111675+0000", 2024-08-19T14:17:07.742 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.8492000000000003e-05 2024-08-19T14:17:07.743 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:07.743 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:17:07.743 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:07.743 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.743 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.743 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24580.0:116735 2.d 2:b15cff4c:::benchmark_data_smithi045_46415_object116734:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-08-19T14:17:07.743 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:14:12.007507+0000", 2024-08-19T14:17:07.743 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 175.61910354899999, 2024-08-19T14:17:07.743 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.104333886, 2024-08-19T14:17:07.743 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:17:07.743 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:17:07.743 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:17:07.743 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:17:07.744 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24580", 2024-08-19T14:17:07.744 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:17:07.744 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 116735 2024-08-19T14:17:07.744 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.744 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:17:07.744 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.744 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:17:07.744 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:12.007507+0000", 2024-08-19T14:17:07.744 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:07.744 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.744 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.744 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:17:07.744 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:12.007507+0000", 2024-08-19T14:17:07.744 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:07.745 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.745 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.745 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:17:07.745 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:12.007508+0000", 2024-08-19T14:17:07.745 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.1100000000000005e-07 2024-08-19T14:17:07.745 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.745 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.745 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:17:07.745 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:12.007528+0000", 2024-08-19T14:17:07.745 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9439000000000001e-05 2024-08-19T14:17:07.745 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.745 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.745 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:17:07.746 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:12.007538+0000", 2024-08-19T14:17:07.746 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0207000000000001e-05 2024-08-19T14:17:07.746 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.746 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.746 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:17:07.746 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:12.007544+0000", 2024-08-19T14:17:07.746 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.5200000000000003e-06 2024-08-19T14:17:07.746 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.746 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.746 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:17:07.747 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:12.096182+0000", 2024-08-19T14:17:07.747 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.088637860999999998 2024-08-19T14:17:07.747 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.747 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.747 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:17:07.747 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:12.096293+0000", 2024-08-19T14:17:07.747 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000110678 2024-08-19T14:17:07.747 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.747 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.747 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 3,4", 2024-08-19T14:17:07.747 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:12.096443+0000", 2024-08-19T14:17:07.747 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0001504 2024-08-19T14:17:07.747 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.748 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.748 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:17:07.748 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:12.103838+0000", 2024-08-19T14:17:07.748 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0073946009999999998 2024-08-19T14:17:07.748 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.748 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.748 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:17:07.748 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:12.103947+0000", 2024-08-19T14:17:07.748 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000109437 2024-08-19T14:17:07.748 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.748 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.748 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:17:07.748 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:12.111734+0000", 2024-08-19T14:17:07.748 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0077861370000000003 2024-08-19T14:17:07.749 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.749 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.749 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:17:07.749 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:12.111797+0000", 2024-08-19T14:17:07.749 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.3888999999999998e-05 2024-08-19T14:17:07.749 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.749 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.749 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:17:07.749 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:12.111841+0000", 2024-08-19T14:17:07.749 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.3906000000000003e-05 2024-08-19T14:17:07.749 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:07.749 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:17:07.749 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:07.749 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.750 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.750 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop_reply(client.24580.0:116730 2.d e147/102 ondisk, result = 0)", 2024-08-19T14:17:07.750 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:14:12.009790+0000", 2024-08-19T14:17:07.750 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 175.616821402, 2024-08-19T14:17:07.750 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.086216922000000001, 2024-08-19T14:17:07.750 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:17:07.750 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:17:07.750 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-08-19T14:17:07.750 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:17:07.750 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.750 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:17:07.750 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:12.009790+0000", 2024-08-19T14:17:07.750 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:07.751 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.751 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.751 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:17:07.751 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:12.009790+0000", 2024-08-19T14:17:07.751 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:07.751 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.751 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.751 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:17:07.751 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:12.009790+0000", 2024-08-19T14:17:07.751 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.68e-07 2024-08-19T14:17:07.751 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.751 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.751 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:17:07.752 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:12.009797+0000", 2024-08-19T14:17:07.752 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.7120000000000003e-06 2024-08-19T14:17:07.752 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.752 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.752 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:17:07.752 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:12.091442+0000", 2024-08-19T14:17:07.752 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.081645439 2024-08-19T14:17:07.752 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.752 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.752 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:17:07.752 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:12.091460+0000", 2024-08-19T14:17:07.752 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7575999999999999e-05 2024-08-19T14:17:07.752 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.752 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.753 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:17:07.753 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:12.095968+0000", 2024-08-19T14:17:07.753 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0045081590000000003 2024-08-19T14:17:07.753 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.753 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.754 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:17:07.754 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:12.095976+0000", 2024-08-19T14:17:07.754 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.8359999999999992e-06 2024-08-19T14:17:07.754 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.754 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.754 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:17:07.754 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:12.096006+0000", 2024-08-19T14:17:07.754 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.0531999999999998e-05 2024-08-19T14:17:07.754 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:07.754 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:17:07.754 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:07.754 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.755 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.755 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24580.0:116747 2.d 2:b9270f19:::benchmark_data_smithi045_46415_object116746:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-08-19T14:17:07.755 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:14:12.025480+0000", 2024-08-19T14:17:07.755 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 175.601130789, 2024-08-19T14:17:07.755 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.086516404000000005, 2024-08-19T14:17:07.755 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:17:07.755 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:17:07.755 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:17:07.755 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:17:07.755 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24580", 2024-08-19T14:17:07.755 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:17:07.755 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 116747 2024-08-19T14:17:07.755 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.755 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:17:07.756 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.756 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:17:07.756 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:12.025480+0000", 2024-08-19T14:17:07.756 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:07.756 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.756 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.756 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:17:07.756 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:12.025480+0000", 2024-08-19T14:17:07.756 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:07.756 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.756 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.757 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:17:07.757 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:12.025481+0000", 2024-08-19T14:17:07.757 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.9500000000000001e-07 2024-08-19T14:17:07.757 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.757 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.757 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:17:07.757 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:12.025500+0000", 2024-08-19T14:17:07.757 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9194999999999998e-05 2024-08-19T14:17:07.757 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.757 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.757 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:17:07.757 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:12.095336+0000", 2024-08-19T14:17:07.757 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.069835805000000001 2024-08-19T14:17:07.757 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.758 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.758 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:17:07.758 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:12.095357+0000", 2024-08-19T14:17:07.758 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1294999999999999e-05 2024-08-19T14:17:07.758 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.758 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.758 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:17:07.758 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:12.100750+0000", 2024-08-19T14:17:07.758 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0053924230000000004 2024-08-19T14:17:07.759 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.759 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.759 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:17:07.759 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:12.100833+0000", 2024-08-19T14:17:07.759 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.3027000000000003e-05 2024-08-19T14:17:07.759 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.759 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.759 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 3,4", 2024-08-19T14:17:07.759 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:12.100935+0000", 2024-08-19T14:17:07.759 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00010226 2024-08-19T14:17:07.759 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.759 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.759 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:17:07.760 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:12.107550+0000", 2024-08-19T14:17:07.760 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0066151329999999996 2024-08-19T14:17:07.760 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.760 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.760 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:17:07.760 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:12.108052+0000", 2024-08-19T14:17:07.760 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00050234699999999999 2024-08-19T14:17:07.760 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.760 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.760 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:17:07.760 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:12.111902+0000", 2024-08-19T14:17:07.761 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0038495420000000001 2024-08-19T14:17:07.761 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.761 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.761 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:17:07.761 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:12.111955+0000", 2024-08-19T14:17:07.761 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.3044999999999998e-05 2024-08-19T14:17:07.761 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:07.761 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:07.761 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:17:07.761 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:12.111997+0000", 2024-08-19T14:17:07.761 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.1536999999999999e-05 2024-08-19T14:17:07.761 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:07.761 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:17:07.762 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:07.762 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:07.762 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:17:07.762 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:17:07.762 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:07 smithi045 ceph-mon[20143]: pgmap v544: 27 pgs: 27 active+clean; 339 MiB data, 2.0 GiB used, 803 GiB / 805 GiB avail; 56 MiB/s wr, 903 op/s 2024-08-19T14:17:07.762 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:07 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/4210736122' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:17:07.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:07.798 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:07.798 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:17:07.899 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:17:07.905 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:07 smithi031 ceph-mon[16336]: pgmap v544: 27 pgs: 27 active+clean; 339 MiB data, 2.0 GiB used, 803 GiB / 805 GiB avail; 56 MiB/s wr, 903 op/s 2024-08-19T14:17:07.906 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:07 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/4210736122' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:17:07.933 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:07.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:07.934 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:17:08.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:07 smithi032 ceph-mon[20004]: pgmap v544: 27 pgs: 27 active+clean; 339 MiB data, 2.0 GiB used, 803 GiB / 805 GiB avail; 56 MiB/s wr, 903 op/s 2024-08-19T14:17:08.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:07 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/4210736122' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:17:08.035 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:17:08.058 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:08.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:08.059 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:17:08.160 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:17:08.191 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:08.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:08.192 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:17:08.293 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:17:08.309 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 6 16 22056 22040 229.528 187.25 0.00588707 0.00435143 2024-08-19T14:17:08.328 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:08.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:08.329 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:17:08.332 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:17:08.332 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:17:08.362 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:17:08.362 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:17:08.392 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:17:08.392 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:17:08.420 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:17:08.420 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:17:08.429 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:17:08.452 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:17:08.453 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:17:08.463 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:08.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:08.464 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:17:08.484 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:17:08.485 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:17:08.516 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:17:08.516 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:17:08.546 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:17:08.546 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:17:08.566 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:17:08.573 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:17:08.573 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:17:08.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:08.591 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:08.591 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:17:08.595 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:17:08.596 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:17:08.613 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:17:08.613 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:17:08.629 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:17:08.630 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:17:08.692 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:17:08.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:08.714 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:08.714 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:17:08.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:08 smithi045 ceph-mon[20143]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:17:08.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:08 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/4210736122' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:17:08.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:08 smithi045 ceph-mon[20143]: osdmap e229: 12 total, 12 up, 9 in 2024-08-19T14:17:08.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:08 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2234530342' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:17:08.815 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:17:08.822 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:08 smithi031 ceph-mon[16336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:17:08.822 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:08 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/4210736122' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:17:08.822 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:08 smithi031 ceph-mon[16336]: osdmap e229: 12 total, 12 up, 9 in 2024-08-19T14:17:08.822 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:08 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2234530342' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:17:08.849 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:08.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:08.850 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:17:08.950 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:17:08.982 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:08.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:08.983 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:17:09.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:08 smithi032 ceph-mon[20004]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:17:09.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:08 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/4210736122' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:17:09.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:08 smithi032 ceph-mon[20004]: osdmap e229: 12 total, 12 up, 9 in 2024-08-19T14:17:09.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:08 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2234530342' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:17:09.064 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-08-19T14:17:09.083 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:17:09.117 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:09.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:09.118 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:17:09.219 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:17:09.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:09.258 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:09.258 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:17:09.310 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 7 16 24770 24754 220.97 169.625 0.00188291 0.0044985 2024-08-19T14:17:09.359 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:17:09.380 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:09.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:09.381 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:17:09.410 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:17:09.482 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:17:09.511 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:09.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:09.512 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:17:09.613 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:17:09.635 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:09.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:09.636 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:17:09.737 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:17:09.765 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:09.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:09.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:17:09.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:09 smithi045 ceph-mon[20143]: pgmap v546: 27 pgs: 27 active+clean; 572 MiB data, 2.6 GiB used, 802 GiB / 805 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-08-19T14:17:09.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:09 smithi031 ceph-mon[16336]: pgmap v546: 27 pgs: 27 active+clean; 572 MiB data, 2.6 GiB used, 802 GiB / 805 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-08-19T14:17:09.867 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:17:09.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:09.903 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:09.903 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:17:10.005 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:17:10.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:09 smithi032 ceph-mon[20004]: pgmap v546: 27 pgs: 27 active+clean; 572 MiB data, 2.6 GiB used, 802 GiB / 805 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-08-19T14:17:10.045 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:10.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:10.046 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:17:10.143 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:17:10.143 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:17:10.146 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:17:10.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:10.179 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:10.179 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:17:10.280 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:17:10.312 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 8 16 27089 27073 211.466 144.938 0.00221606 0.00472358 2024-08-19T14:17:10.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:10.313 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:10.313 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:17:10.414 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:17:10.450 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:10.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:10.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:17:10.552 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:17:10.586 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:10.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:10.587 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:17:10.684 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 7, 8, 9, 11] out_osds: [10, 6, 2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-08-19T14:17:10.684 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-19T14:17:10.684 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_1 2024-08-19T14:17:10.685 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-19T14:17:10.687 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:17:10.720 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:10.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:10.721 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:17:10.822 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:17:10.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:10.857 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:10.857 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:17:10.913 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-19T14:17:10.958 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:17:10.981 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:10.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:10.982 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:17:11.064 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:17:11.083 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:17:11.118 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:11.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:11.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:17:11.220 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:17:11.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:11.257 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:11.257 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:17:11.310 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 9 16 30128 30112 209.072 189.938 0.00460911 0.00477841 2024-08-19T14:17:11.359 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:17:11.387 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:11.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:11.388 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:17:11.489 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:17:11.512 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:11.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:11.513 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:17:11.537 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:17:11.614 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:17:11.637 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-08-19T14:17:11.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:11.647 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:11.647 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:17:11.748 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:17:11.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:11.780 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:11.780 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:17:11.786 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:17:11.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:11 smithi031 ceph-mon[16336]: pgmap v547: 27 pgs: 27 active+clean; 1.1 GiB data, 4.4 GiB used, 800 GiB / 805 GiB avail; 152 MiB/s wr, 2.44k op/s 2024-08-19T14:17:11.882 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:17:11.918 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:11.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:11.919 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:17:12.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:11 smithi032 ceph-mon[20004]: pgmap v547: 27 pgs: 27 active+clean; 1.1 GiB data, 4.4 GiB used, 800 GiB / 805 GiB avail; 152 MiB/s wr, 2.44k op/s 2024-08-19T14:17:12.020 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:17:12.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:11 smithi045 ceph-mon[20143]: pgmap v547: 27 pgs: 27 active+clean; 1.1 GiB data, 4.4 GiB used, 800 GiB / 805 GiB avail; 152 MiB/s wr, 2.44k op/s 2024-08-19T14:17:12.052 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:17:12.054 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:12.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:12.055 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:17:12.097 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:17:12.098 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-08-19T14:17:12.158 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:17:12.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:12.197 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:12.197 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:17:12.299 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:17:12.310 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 10 16 32579 32563 203.479 153.188 0.00318089 0.00490983 2024-08-19T14:17:12.334 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:12.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:12.335 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:17:12.436 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:17:12.468 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:12.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:12.469 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:17:12.570 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:17:12.602 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:12.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:12.604 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:17:12.683 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:17:12.683 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:17:12.704 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:17:12.739 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:12.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:12.841 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:17:12.841 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:12 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:17:12.841 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:12 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:17:12.941 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:17:12.972 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:12.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:12.973 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:17:13.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:12 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:17:13.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:12 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:17:13.074 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:17:13.103 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:13.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:13.104 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:17:13.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:12 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:17:13.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:12 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:17:13.205 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:17:13.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:13.240 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:13.241 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:17:13.310 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 11 16 35431 35415 201.184 178.25 0.00559475 0.00496605 2024-08-19T14:17:13.341 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:17:13.373 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:13.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:13.374 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:17:13.475 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:17:13.506 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:13.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:13.507 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:17:13.608 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:17:13.642 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:13.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:13.643 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:17:13.680 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is set 2024-08-19T14:17:13.744 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:17:13.774 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:13.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:13.776 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:17:13.877 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:17:13.898 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:13.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:13.899 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:17:14.000 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:17:14.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:13 smithi032 ceph-mon[20004]: pgmap v548: 27 pgs: 27 active+clean; 1.5 GiB data, 5.7 GiB used, 799 GiB / 805 GiB avail; 179 MiB/s wr, 2.87k op/s 2024-08-19T14:17:14.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:13 smithi032 ceph-mon[20004]: from='client.25084 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:17:14.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:13 smithi032 ceph-mon[20004]: osdmap e230: 12 total, 12 up, 9 in 2024-08-19T14:17:14.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:13 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3024717407' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:17:14.021 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:14.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:14.022 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:17:14.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:13 smithi045 ceph-mon[20143]: pgmap v548: 27 pgs: 27 active+clean; 1.5 GiB data, 5.7 GiB used, 799 GiB / 805 GiB avail; 179 MiB/s wr, 2.87k op/s 2024-08-19T14:17:14.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:13 smithi045 ceph-mon[20143]: from='client.25084 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:17:14.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:13 smithi045 ceph-mon[20143]: osdmap e230: 12 total, 12 up, 9 in 2024-08-19T14:17:14.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:13 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3024717407' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:17:14.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:13 smithi031 ceph-mon[16336]: pgmap v548: 27 pgs: 27 active+clean; 1.5 GiB data, 5.7 GiB used, 799 GiB / 805 GiB avail; 179 MiB/s wr, 2.87k op/s 2024-08-19T14:17:14.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:13 smithi031 ceph-mon[16336]: from='client.25084 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:17:14.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:13 smithi031 ceph-mon[16336]: osdmap e230: 12 total, 12 up, 9 in 2024-08-19T14:17:14.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:13 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3024717407' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:17:14.123 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:17:14.152 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:14.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:14.153 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:17:14.254 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:17:14.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:14.276 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:14.276 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:17:14.310 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 12 16 37365 37349 194.492 120.875 0.00277358 0.00511192 2024-08-19T14:17:14.377 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:17:14.398 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:14.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:14.399 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:17:14.499 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:17:14.529 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:14.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:14.530 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:17:14.591 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-08-19T14:17:14.631 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:17:14.659 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:14.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:14.660 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:17:14.761 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:17:14.793 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:14.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:14.794 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:17:14.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:14 smithi045 ceph-mon[20143]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:17:14.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:14 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3024717407' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:17:14.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:14 smithi045 ceph-mon[20143]: osdmap e231: 12 total, 12 up, 9 in 2024-08-19T14:17:14.820 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":548,"stamp":"2024-08-19T14:17:11.808467+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1663238688,"num_objects":25372,"num_object_clones":0,"num_object_copies":76116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25372,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25370,"num_write_kb":1623680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":26899,"ondisk_log_size":26899,"up":81,"acting":81,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":5928596,"kb_used_data":4608188,"kb_used_omap":14,"kb_used_meta":1320305,"kb_avail":837777772,"statfs":{"total":863955320832,"available":857884438528,"internally_reserved":0,"allocated":4718784512,"data_stored":4708846317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14353,"internal_metadata":1351993327},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[13,1,5,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1574633472,"num_objects":24027,"num_object_clones":0,"num_object_copies":72081,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24027,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24027,"num_write_kb":1537728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.367078"},"pg_stats":[{"pgid":"3.f","version":"229'1397","reported_seq":1421,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-08-19T14:17:08.074646+0000","last_change":"2024-08-19T14:16:56.416145+0000","last_active":"2024-08-19T14:17:08.074646+0000","last_peered":"2024-08-19T14:17:08.074646+0000","last_clean":"2024-08-19T14:17:08.074646+0000","last_became_active":"2024-08-19T14:16:56.415674+0000","last_became_peered":"2024-08-19T14:16:56.415674+0000","last_unstale":"2024-08-19T14:17:08.074646+0000","last_undegraded":"2024-08-19T14:17:08.074646+0000","last_fullsized":"2024-08-19T14:17:08.074646+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":1397,"log_dups_size":0,"ondisk_log_size":1397,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:17:57.127950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":91553792,"num_objects":1397,"num_object_clones":0,"num_object_copies":4191,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1397,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1397,"num_write_kb":89408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"229'1800","reported_seq":1824,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-08-19T14:17:10.686779+0000","last_change":"2024-08-19T14:16:56.417716+0000","last_active":"2024-08-19T14:17:10.686779+0000","last_peered":"2024-08-19T14:17:10.686779+0000","last_clean":"2024-08-19T14:17:10.686779+0000","last_became_active":"2024-08-19T14:16:56.417440+0000","last_became_peered":"2024-08-19T14:16:56.417440+0000","last_unstale":"2024-08-19T14:17:10.686779+0000","last_undegraded":"2024-08-19T14:17:10.686779+0000","last_fullsized":"2024-08-19T14:17:10.686779+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":1800,"log_dups_size":0,"ondisk_log_size":1800,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:17:55.463422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117964800,"num_objects":1800,"num_object_clones":0,"num_object_copies":5400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1800,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1800,"num_write_kb":115200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,0],"acting":[11,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.d","version":"229'1342","reported_seq":1366,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-08-19T14:17:08.081376+0000","last_change":"2024-08-19T14:16:56.415868+0000","last_active":"2024-08-19T14:17:08.081376+0000","last_peered":"2024-08-19T14:17:08.081376+0000","last_clean":"2024-08-19T14:17:08.081376+0000","last_became_active":"2024-08-19T14:16:56.414976+0000","last_became_peered":"2024-08-19T14:16:56.414976+0000","last_unstale":"2024-08-19T14:17:08.081376+0000","last_undegraded":"2024-08-19T14:17:08.081376+0000","last_fullsized":"2024-08-19T14:17:08.081376+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":1342,"log_dups_size":0,"ondisk_log_size":1342,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:14.378055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87949312,"num_objects":1342,"num_object_clones":0,"num_object_copies":4026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1342,"num_write_kb":85888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,9],"acting":[7,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"229'1905","reported_seq":1929,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-08-19T14:17:11.434787+0000","last_change":"2024-08-19T14:16:56.415146+0000","last_active":"2024-08-19T14:17:11.434787+0000","last_peered":"2024-08-19T14:17:11.434787+0000","last_clean":"2024-08-19T14:17:11.434787+0000","last_became_active":"2024-08-19T14:16:56.414671+0000","last_became_peered":"2024-08-19T14:16:56.414671+0000","last_unstale":"2024-08-19T14:17:11.434787+0000","last_undegraded":"2024-08-19T14:17:11.434787+0000","last_fullsized":"2024-08-19T14:17:11.434787+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":1905,"log_dups_size":0,"ondisk_log_size":1905,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:05.042513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":124846080,"num_objects":1905,"num_object_clones":0,"num_object_copies":5715,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1905,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1905,"num_write_kb":121920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,9],"acting":[5,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"229'1574","reported_seq":1598,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-08-19T14:17:09.800447+0000","last_change":"2024-08-19T14:16:56.413983+0000","last_active":"2024-08-19T14:17:09.800447+0000","last_peered":"2024-08-19T14:17:09.800447+0000","last_clean":"2024-08-19T14:17:09.800447+0000","last_became_active":"2024-08-19T14:16:56.413659+0000","last_became_peered":"2024-08-19T14:16:56.413659+0000","last_unstale":"2024-08-19T14:17:09.800447+0000","last_undegraded":"2024-08-19T14:17:09.800447+0000","last_fullsized":"2024-08-19T14:17:09.800447+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":1574,"log_dups_size":0,"ondisk_log_size":1574,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:20.881300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103153664,"num_objects":1574,"num_object_clones":0,"num_object_copies":4722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1574,"num_write_kb":100736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"229'1342","reported_seq":1366,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-08-19T14:17:07.843013+0000","last_change":"2024-08-19T14:16:56.415768+0000","last_active":"2024-08-19T14:17:07.843013+0000","last_peered":"2024-08-19T14:17:07.843013+0000","last_clean":"2024-08-19T14:17:07.843013+0000","last_became_active":"2024-08-19T14:16:56.415495+0000","last_became_peered":"2024-08-19T14:16:56.415495+0000","last_unstale":"2024-08-19T14:17:07.843013+0000","last_undegraded":"2024-08-19T14:17:07.843013+0000","last_fullsized":"2024-08-19T14:17:07.843013+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":1342,"log_dups_size":0,"ondisk_log_size":1342,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:17:56.248546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87949312,"num_objects":1342,"num_object_clones":0,"num_object_copies":4026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1342,"num_write_kb":85888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,8],"acting":[4,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"70'139","reported_seq":548,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-08-19T14:17:07.661762+0000","last_change":"2024-08-19T14:16:42.249971+0000","last_active":"2024-08-19T14:17:07.661762+0000","last_peered":"2024-08-19T14:17:07.661762+0000","last_clean":"2024-08-19T14:17:07.661762+0000","last_became_active":"2024-08-19T14:16:42.249518+0000","last_became_peered":"2024-08-19T14:16:42.249518+0000","last_unstale":"2024-08-19T14:17:07.661762+0000","last_undegraded":"2024-08-19T14:17:07.661762+0000","last_fullsized":"2024-08-19T14:17:07.661762+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:15:44.541487+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:15:44.541487+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,8],"acting":[5,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.b","version":"229'1486","reported_seq":1508,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-08-19T14:17:09.110889+0000","last_change":"2024-08-19T14:16:56.414110+0000","last_active":"2024-08-19T14:17:09.110889+0000","last_peered":"2024-08-19T14:17:09.110889+0000","last_clean":"2024-08-19T14:17:09.110889+0000","last_became_active":"2024-08-19T14:16:56.413751+0000","last_became_peered":"2024-08-19T14:16:56.413751+0000","last_unstale":"2024-08-19T14:17:09.110889+0000","last_undegraded":"2024-08-19T14:17:09.110889+0000","last_fullsized":"2024-08-19T14:17:09.110889+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":1486,"log_dups_size":0,"ondisk_log_size":1486,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:04.197986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":97386496,"num_objects":1486,"num_object_clones":0,"num_object_copies":4458,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1486,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1486,"num_write_kb":95104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"70'139","reported_seq":666,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-08-19T14:17:07.660727+0000","last_change":"2024-08-19T14:16:42.252897+0000","last_active":"2024-08-19T14:17:07.660727+0000","last_peered":"2024-08-19T14:17:07.660727+0000","last_clean":"2024-08-19T14:17:07.660727+0000","last_became_active":"2024-08-19T14:16:42.252391+0000","last_became_peered":"2024-08-19T14:16:42.252391+0000","last_unstale":"2024-08-19T14:17:07.660727+0000","last_undegraded":"2024-08-19T14:17:07.660727+0000","last_fullsized":"2024-08-19T14:17:07.660727+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:16:03.918046+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:17:27.707808+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"229'1820","reported_seq":1844,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-08-19T14:17:10.687003+0000","last_change":"2024-08-19T14:16:56.416812+0000","last_active":"2024-08-19T14:17:10.687003+0000","last_peered":"2024-08-19T14:17:10.687003+0000","last_clean":"2024-08-19T14:17:10.687003+0000","last_became_active":"2024-08-19T14:16:56.416405+0000","last_became_peered":"2024-08-19T14:16:56.416405+0000","last_unstale":"2024-08-19T14:17:10.687003+0000","last_undegraded":"2024-08-19T14:17:10.687003+0000","last_fullsized":"2024-08-19T14:17:10.687003+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":1820,"log_dups_size":0,"ondisk_log_size":1820,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:12.298581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":119275520,"num_objects":1820,"num_object_clones":0,"num_object_copies":5460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1820,"num_write_kb":116480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,4],"acting":[11,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.a","version":"70'139","reported_seq":544,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-08-19T14:17:07.664598+0000","last_change":"2024-08-19T14:16:02.818848+0000","last_active":"2024-08-19T14:17:07.664598+0000","last_peered":"2024-08-19T14:17:07.664598+0000","last_clean":"2024-08-19T14:17:07.664598+0000","last_became_active":"2024-08-19T14:13:22.728779+0000","last_became_peered":"2024-08-19T14:13:22.728779+0000","last_unstale":"2024-08-19T14:17:07.664598+0000","last_undegraded":"2024-08-19T14:17:07.664598+0000","last_fullsized":"2024-08-19T14:17:07.664598+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:16:02.818774+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:16:02.818774+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:17:09.255652+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,0],"acting":[9,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.6","version":"229'1782","reported_seq":1806,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-08-19T14:17:10.797292+0000","last_change":"2024-08-19T14:16:56.414514+0000","last_active":"2024-08-19T14:17:10.797292+0000","last_peered":"2024-08-19T14:17:10.797292+0000","last_clean":"2024-08-19T14:17:10.797292+0000","last_became_active":"2024-08-19T14:16:56.414199+0000","last_became_peered":"2024-08-19T14:16:56.414199+0000","last_unstale":"2024-08-19T14:17:10.797292+0000","last_undegraded":"2024-08-19T14:17:10.797292+0000","last_fullsized":"2024-08-19T14:17:10.797292+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":1782,"log_dups_size":0,"ondisk_log_size":1782,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:05.443237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":116785152,"num_objects":1782,"num_object_clones":0,"num_object_copies":5346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1782,"num_write_kb":114048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"70'139","reported_seq":582,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-08-19T14:17:07.661233+0000","last_change":"2024-08-19T14:16:29.257033+0000","last_active":"2024-08-19T14:17:07.661233+0000","last_peered":"2024-08-19T14:17:07.661233+0000","last_clean":"2024-08-19T14:17:07.661233+0000","last_became_active":"2024-08-19T14:13:05.607447+0000","last_became_peered":"2024-08-19T14:13:05.607447+0000","last_unstale":"2024-08-19T14:17:07.661233+0000","last_undegraded":"2024-08-19T14:17:07.661233+0000","last_fullsized":"2024-08-19T14:17:07.661233+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:16:29.256959+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:16:29.256959+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:17:40.848859+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.7","version":"229'1467","reported_seq":1491,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-08-19T14:17:09.099631+0000","last_change":"2024-08-19T14:16:56.415838+0000","last_active":"2024-08-19T14:17:09.099631+0000","last_peered":"2024-08-19T14:17:09.099631+0000","last_clean":"2024-08-19T14:17:09.099631+0000","last_became_active":"2024-08-19T14:16:56.415384+0000","last_became_peered":"2024-08-19T14:16:56.415384+0000","last_unstale":"2024-08-19T14:17:09.099631+0000","last_undegraded":"2024-08-19T14:17:09.099631+0000","last_fullsized":"2024-08-19T14:17:09.099631+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":1467,"log_dups_size":0,"ondisk_log_size":1467,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:16.663918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96141312,"num_objects":1467,"num_object_clones":0,"num_object_copies":4401,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1467,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1467,"num_write_kb":93888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"70'139","reported_seq":345,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-08-19T14:17:07.661167+0000","last_change":"2024-08-19T14:16:42.252111+0000","last_active":"2024-08-19T14:17:07.661167+0000","last_peered":"2024-08-19T14:17:07.661167+0000","last_clean":"2024-08-19T14:17:07.661167+0000","last_became_active":"2024-08-19T14:16:42.251643+0000","last_became_peered":"2024-08-19T14:16:42.251643+0000","last_unstale":"2024-08-19T14:17:07.661167+0000","last_undegraded":"2024-08-19T14:17:07.661167+0000","last_fullsized":"2024-08-19T14:17:07.661167+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:16:30.232687+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:16:30.232687+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:17:59.854133+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,4],"acting":[11,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"229'1416","reported_seq":1440,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-08-19T14:17:08.336253+0000","last_change":"2024-08-19T14:16:56.417053+0000","last_active":"2024-08-19T14:17:08.336253+0000","last_peered":"2024-08-19T14:17:08.336253+0000","last_clean":"2024-08-19T14:17:08.336253+0000","last_became_active":"2024-08-19T14:16:56.416751+0000","last_became_peered":"2024-08-19T14:16:56.416751+0000","last_unstale":"2024-08-19T14:17:08.336253+0000","last_undegraded":"2024-08-19T14:17:08.336253+0000","last_fullsized":"2024-08-19T14:17:08.336253+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":1416,"log_dups_size":0,"ondisk_log_size":1416,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:23.166649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92798976,"num_objects":1416,"num_object_clones":0,"num_object_copies":4248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1416,"num_write_kb":90624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,5],"acting":[1,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"70'139","reported_seq":531,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-08-19T14:17:07.664159+0000","last_change":"2024-08-19T14:16:36.969224+0000","last_active":"2024-08-19T14:17:07.664159+0000","last_peered":"2024-08-19T14:17:07.664159+0000","last_clean":"2024-08-19T14:17:07.664159+0000","last_became_active":"2024-08-19T14:13:11.647085+0000","last_became_peered":"2024-08-19T14:13:11.647085+0000","last_unstale":"2024-08-19T14:17:07.664159+0000","last_undegraded":"2024-08-19T14:17:07.664159+0000","last_fullsized":"2024-08-19T14:17:07.664159+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:16:36.969114+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:16:36.969114+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:17:44.848868+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,4],"acting":[9,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"229'1891","reported_seq":1915,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-08-19T14:17:11.430788+0000","last_change":"2024-08-19T14:16:56.416036+0000","last_active":"2024-08-19T14:17:11.430788+0000","last_peered":"2024-08-19T14:17:11.430788+0000","last_clean":"2024-08-19T14:17:11.430788+0000","last_became_active":"2024-08-19T14:16:56.415655+0000","last_became_peered":"2024-08-19T14:16:56.415655+0000","last_unstale":"2024-08-19T14:17:11.430788+0000","last_undegraded":"2024-08-19T14:17:11.430788+0000","last_fullsized":"2024-08-19T14:17:11.430788+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":1891,"log_dups_size":0,"ondisk_log_size":1891,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:17:57.734240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123928576,"num_objects":1891,"num_object_clones":0,"num_object_copies":5673,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1891,"num_write_kb":121024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"70'139","reported_seq":448,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-08-19T14:17:07.661821+0000","last_change":"2024-08-19T14:16:42.252166+0000","last_active":"2024-08-19T14:17:07.661821+0000","last_peered":"2024-08-19T14:17:07.661821+0000","last_clean":"2024-08-19T14:17:07.661821+0000","last_became_active":"2024-08-19T14:16:42.251741+0000","last_became_peered":"2024-08-19T14:16:42.251741+0000","last_unstale":"2024-08-19T14:17:07.661821+0000","last_undegraded":"2024-08-19T14:17:07.661821+0000","last_fullsized":"2024-08-19T14:17:07.661821+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:16:10.672834+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:17:33.011576+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"229'1288","reported_seq":1312,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-08-19T14:17:07.833226+0000","last_change":"2024-08-19T14:16:56.415826+0000","last_active":"2024-08-19T14:17:07.833226+0000","last_peered":"2024-08-19T14:17:07.833226+0000","last_clean":"2024-08-19T14:17:07.833226+0000","last_became_active":"2024-08-19T14:16:56.415523+0000","last_became_peered":"2024-08-19T14:16:56.415523+0000","last_unstale":"2024-08-19T14:17:07.833226+0000","last_undegraded":"2024-08-19T14:17:07.833226+0000","last_fullsized":"2024-08-19T14:17:07.833226+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":1288,"log_dups_size":0,"ondisk_log_size":1288,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:07.688699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84410368,"num_objects":1288,"num_object_clones":0,"num_object_copies":3864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1288,"num_write_kb":82432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,11],"acting":[4,0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"70'139","reported_seq":540,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-08-19T14:17:07.664021+0000","last_change":"2024-08-19T14:16:42.252108+0000","last_active":"2024-08-19T14:17:07.664021+0000","last_peered":"2024-08-19T14:17:07.664021+0000","last_clean":"2024-08-19T14:17:07.664021+0000","last_became_active":"2024-08-19T14:16:42.251603+0000","last_became_peered":"2024-08-19T14:16:42.251603+0000","last_unstale":"2024-08-19T14:17:07.664021+0000","last_undegraded":"2024-08-19T14:17:07.664021+0000","last_fullsized":"2024-08-19T14:17:07.664021+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:16:30.043526+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:16:30.043526+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:17:54.530868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,11],"acting":[9,0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.0","version":"229'1708","reported_seq":1732,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-08-19T14:17:10.682937+0000","last_change":"2024-08-19T14:16:56.417098+0000","last_active":"2024-08-19T14:17:10.682937+0000","last_peered":"2024-08-19T14:17:10.682937+0000","last_clean":"2024-08-19T14:17:10.682937+0000","last_became_active":"2024-08-19T14:16:56.416567+0000","last_became_peered":"2024-08-19T14:16:56.416567+0000","last_unstale":"2024-08-19T14:17:10.682937+0000","last_undegraded":"2024-08-19T14:17:10.682937+0000","last_fullsized":"2024-08-19T14:17:10.682937+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":1708,"log_dups_size":0,"ondisk_log_size":1708,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:09.765431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":111935488,"num_objects":1708,"num_object_clones":0,"num_object_copies":5124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1708,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1708,"num_write_kb":109312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,1],"acting":[11,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.2","version":"70'139","reported_seq":488,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-08-19T14:17:07.658704+0000","last_change":"2024-08-19T14:16:00.592808+0000","last_active":"2024-08-19T14:17:07.658704+0000","last_peered":"2024-08-19T14:17:07.658704+0000","last_clean":"2024-08-19T14:17:07.658704+0000","last_became_active":"2024-08-19T14:15:59.993566+0000","last_became_peered":"2024-08-19T14:15:59.993566+0000","last_unstale":"2024-08-19T14:17:07.658704+0000","last_undegraded":"2024-08-19T14:17:07.658704+0000","last_fullsized":"2024-08-19T14:17:07.658704+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:16:00.592678+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:16:00.592678+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,8],"acting":[0,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"229'1785","reported_seq":1809,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-08-19T14:17:10.813907+0000","last_change":"2024-08-19T14:16:56.415773+0000","last_active":"2024-08-19T14:17:10.813907+0000","last_peered":"2024-08-19T14:17:10.813907+0000","last_clean":"2024-08-19T14:17:10.813907+0000","last_became_active":"2024-08-19T14:16:56.415446+0000","last_became_peered":"2024-08-19T14:16:56.415446+0000","last_unstale":"2024-08-19T14:17:10.813907+0000","last_undegraded":"2024-08-19T14:17:10.813907+0000","last_fullsized":"2024-08-19T14:17:10.813907+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":1785,"log_dups_size":0,"ondisk_log_size":1785,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:08.064764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":116981760,"num_objects":1785,"num_object_clones":0,"num_object_copies":5355,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1785,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1785,"num_write_kb":114240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,11],"acting":[0,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"70'139","reported_seq":543,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-08-19T14:17:07.660225+0000","last_change":"2024-08-19T14:16:06.679203+0000","last_active":"2024-08-19T14:17:07.660225+0000","last_peered":"2024-08-19T14:17:07.660225+0000","last_clean":"2024-08-19T14:17:07.660225+0000","last_became_active":"2024-08-19T14:13:22.727234+0000","last_became_peered":"2024-08-19T14:13:22.727234+0000","last_unstale":"2024-08-19T14:17:07.660225+0000","last_undegraded":"2024-08-19T14:17:07.660225+0000","last_fullsized":"2024-08-19T14:17:07.660225+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:16:06.679075+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:16:06.679075+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:17:31.357116+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"229'1367","reported_seq":1391,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-08-19T14:17:08.294584+0000","last_change":"2024-08-19T14:16:56.414945+0000","last_active":"2024-08-19T14:17:08.294584+0000","last_peered":"2024-08-19T14:17:08.294584+0000","last_clean":"2024-08-19T14:17:08.294584+0000","last_became_active":"2024-08-19T14:16:56.414587+0000","last_became_peered":"2024-08-19T14:16:56.414587+0000","last_unstale":"2024-08-19T14:17:08.294584+0000","last_undegraded":"2024-08-19T14:17:08.294584+0000","last_fullsized":"2024-08-19T14:17:08.294584+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":1367,"log_dups_size":0,"ondisk_log_size":1367,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:17:55.980838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":89587712,"num_objects":1367,"num_object_clones":0,"num_object_copies":4101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1367,"num_write_kb":87488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,5],"acting":[9,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.0","version":"70'139","reported_seq":645,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-08-19T14:17:07.660727+0000","last_change":"2024-08-19T14:16:42.251163+0000","last_active":"2024-08-19T14:17:07.660727+0000","last_peered":"2024-08-19T14:17:07.660727+0000","last_clean":"2024-08-19T14:17:07.660727+0000","last_became_active":"2024-08-19T14:16:42.250702+0000","last_became_peered":"2024-08-19T14:16:42.250702+0000","last_unstale":"2024-08-19T14:17:07.660727+0000","last_undegraded":"2024-08-19T14:17:07.660727+0000","last_fullsized":"2024-08-19T14:17:07.660727+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:15:59.869266+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:15:59.869266+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:17:24.361856+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,9],"acting":[7,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":1662648320,"num_objects":25370,"num_object_clones":0,"num_object_copies":76110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25370,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25370,"num_write_kb":1623680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4897832960,"data_stored":4897832960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":25370,"ondisk_log_size":25370,"up":48,"acting":48,"num_store_stats":9},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1802240,"data_stored":1788032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1529,"ondisk_log_size":1529,"up":33,"acting":33,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187526,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1142840,"kb_used_data":999216,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":92602312,"statfs":{"total":95995035648,"available":94824767488,"internally_reserved":0,"allocated":1023197184,"data_stored":1022093655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":147061188},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":343597383754,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":74,"seq":317827579983,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":521532,"kb_used_data":341808,"kb_used_omap":1,"kb_used_meta":179710,"kb_avail":93223620,"statfs":{"total":95995035648,"available":95460986880,"internally_reserved":0,"allocated":350011392,"data_stored":348907863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":184023489},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874325,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":445376,"kb_used_data":287864,"kb_used_omap":1,"kb_used_meta":157502,"kb_avail":93299776,"statfs":{"total":95995035648,"available":95538970624,"internally_reserved":0,"allocated":294772736,"data_stored":293665655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":161282501},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":57,"seq":244813135959,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":594876,"kb_used_data":433200,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":93150276,"statfs":{"total":95995035648,"available":95385882624,"internally_reserved":0,"allocated":443596800,"data_stored":442493271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":165542346},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":206158430299,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691936,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":701504,"kb_used_data":616184,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":93043648,"statfs":{"total":95995035648,"available":95276695552,"internally_reserved":0,"allocated":630972416,"data_stored":629865335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986276,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":698924,"kb_used_data":577508,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":93046228,"statfs":{"total":95995035648,"available":95279337472,"internally_reserved":0,"allocated":591368192,"data_stored":590268885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149802,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":543684,"kb_used_data":363960,"kb_used_omap":1,"kb_used_meta":179710,"kb_avail":93201468,"statfs":{"total":95995035648,"available":95438303232,"internally_reserved":0,"allocated":372695040,"data_stored":371587959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":184023495},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411439,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574963,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":582972,"kb_used_data":435184,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":93162180,"statfs":{"total":95995035648,"available":95398072320,"internally_reserved":0,"allocated":445628416,"data_stored":444524887,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151321029},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738488,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":696888,"kb_used_data":553264,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":93048264,"statfs":{"total":95995035648,"available":95281422336,"internally_reserved":0,"allocated":566542336,"data_stored":565438807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":147061189},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":584646656,"data_stored":584646656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":456261632,"data_stored":456261632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":391774208,"data_stored":391774208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":684326912,"data_stored":684326912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":628097024,"data_stored":628097024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":441974784,"data_stored":441974784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":313786368,"data_stored":313786368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":355860480,"data_stored":355860480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1041104896,"data_stored":1041104896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T14:17:14.822 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-08-19T14:17:14.822 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd pool set unique_pool_1 pg_num 26 2024-08-19T14:17:14.894 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:17:14.926 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:14.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:14.927 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:17:14.946 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:17:15.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:14 smithi032 ceph-mon[20004]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:17:15.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:14 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3024717407' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:17:15.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:14 smithi032 ceph-mon[20004]: osdmap e231: 12 total, 12 up, 9 in 2024-08-19T14:17:15.029 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:17:15.066 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:15.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:15.067 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:17:15.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:14 smithi031 ceph-mon[16336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:17:15.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:14 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3024717407' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:17:15.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:14 smithi031 ceph-mon[16336]: osdmap e231: 12 total, 12 up, 9 in 2024-08-19T14:17:15.168 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:17:15.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:15.199 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:15.199 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:17:15.300 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:17:15.310 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 13 16 39765 39749 191.069 150 0.00364257 0.00522956 2024-08-19T14:17:15.328 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:15.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:15.347 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:17:15.448 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:17:15.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:15.481 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:15.482 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:17:15.583 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:17:15.614 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:15.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:15.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:17:15.616 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:17:15.669 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:17:15.715 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:17:15.747 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:15.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:15.748 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:17:15.850 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:17:15.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:15 smithi031 ceph-mon[16336]: pgmap v551: 27 pgs: 27 active+clean; 1.9 GiB data, 6.8 GiB used, 798 GiB / 805 GiB avail; 216 MiB/s wr, 3.45k op/s 2024-08-19T14:17:15.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:15 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:17:15.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:15 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:17:15.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:15 smithi031 ceph-mon[16336]: osdmap e232: 12 total, 12 up, 9 in 2024-08-19T14:17:15.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:15.886 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:15.886 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:17:15.987 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:17:16.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:15 smithi032 ceph-mon[20004]: pgmap v551: 27 pgs: 27 active+clean; 1.9 GiB data, 6.8 GiB used, 798 GiB / 805 GiB avail; 216 MiB/s wr, 3.45k op/s 2024-08-19T14:17:16.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:15 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:17:16.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:15 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:17:16.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:15 smithi032 ceph-mon[20004]: osdmap e232: 12 total, 12 up, 9 in 2024-08-19T14:17:16.018 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:16.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:16.020 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:17:16.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:15 smithi045 ceph-mon[20143]: pgmap v551: 27 pgs: 27 active+clean; 1.9 GiB data, 6.8 GiB used, 798 GiB / 805 GiB avail; 216 MiB/s wr, 3.45k op/s 2024-08-19T14:17:16.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:15 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:17:16.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:15 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:17:16.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:15 smithi045 ceph-mon[20143]: osdmap e232: 12 total, 12 up, 9 in 2024-08-19T14:17:16.120 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:17:16.143 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:16.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:16.145 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:17:16.245 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:17:16.280 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:16.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:16.281 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:17:16.311 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 14 16 42094 42078 187.817 145.562 0.00219999 0.00531132 2024-08-19T14:17:16.381 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:17:16.414 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:16.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:16.415 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:17:16.437 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:17:16.437 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:17:16.438 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:17:16.438 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:17:16.438 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:17:16.438 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:17:16.438 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:17:16.438 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:17:16.438 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:17:16.438 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:17:16.438 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:17:16.438 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:17:16.438 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:17:16.439 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:17:16.439 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:17:16.439 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:17:16.439 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:17:16.439 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:17:16.439 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:17:16.439 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:17:16.442 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:17:16.442 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:17:16.474 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:17:16.474 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:17:16.516 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:17:16.549 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:16.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:16.551 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:17:16.651 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:17:16.683 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:16.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:16.685 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:17:16.786 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:17:16.816 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:16.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:16.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:17:16.898 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:16 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/949712500' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:17:16.918 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:17:16.939 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:16.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:16.940 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:17:17.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:16 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/949712500' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:17:17.041 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:17:17.074 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:17.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:17.075 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:17:17.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:16 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/949712500' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:17:17.128 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:17:17.128 INFO:teuthology.orchestra.run.smithi045.stdout: "size": 20, 2024-08-19T14:17:17.129 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 600, 2024-08-19T14:17:17.129 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [ 2024-08-19T14:17:17.129 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.129 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:14 2.4 e106/102 2:291e9767:::benchmark_data_smithi045_46415_object13:head v 106'1, mlcod=0'0)", 2024-08-19T14:17:17.129 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:12:49.835194+0000", 2024-08-19T14:17:17.129 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 267.06181837899999, 2024-08-19T14:17:17.129 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.045178087999999998, 2024-08-19T14:17:17.129 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:17:17.129 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:17:17.129 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:17:17.129 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:17:17.129 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.129 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:17:17.130 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:49.835194+0000", 2024-08-19T14:17:17.130 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:17.130 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.130 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.130 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:17:17.130 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:49.835194+0000", 2024-08-19T14:17:17.130 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:17.130 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.130 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.130 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:17:17.130 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:49.835194+0000", 2024-08-19T14:17:17.130 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.7500000000000001e-07 2024-08-19T14:17:17.130 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.131 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.131 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:17:17.131 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:49.879644+0000", 2024-08-19T14:17:17.131 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.044449610000000001 2024-08-19T14:17:17.131 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.131 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.131 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:17:17.131 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:49.879665+0000", 2024-08-19T14:17:17.131 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1458999999999999e-05 2024-08-19T14:17:17.131 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.131 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.131 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:17:17.131 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:49.879706+0000", 2024-08-19T14:17:17.131 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.0748999999999999e-05 2024-08-19T14:17:17.132 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.132 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.132 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:17:17.132 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:49.879782+0000", 2024-08-19T14:17:17.132 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.6445000000000004e-05 2024-08-19T14:17:17.132 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.132 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.132 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:17:17.132 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:49.879807+0000", 2024-08-19T14:17:17.132 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.4765000000000001e-05 2024-08-19T14:17:17.132 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.132 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.132 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:17:17.132 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:49.880329+0000", 2024-08-19T14:17:17.133 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00052147599999999999 2024-08-19T14:17:17.133 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.133 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.133 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:17:17.133 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:49.880372+0000", 2024-08-19T14:17:17.133 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.3309000000000002e-05 2024-08-19T14:17:17.133 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:17.133 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:17:17.133 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:17.133 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.133 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.133 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:12156 2.4 e107/102 2:28b3607b:::benchmark_data_smithi045_46415_object12155:head v 107'793, mlcod=107'791)", 2024-08-19T14:17:17.133 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:12:53.026834+0000", 2024-08-19T14:17:17.134 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 263.87017791400001, 2024-08-19T14:17:17.134 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0078985359999999994, 2024-08-19T14:17:17.134 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:17:17.134 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:17:17.134 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:17:17.134 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:17:17.134 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.134 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:17:17.134 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:53.026834+0000", 2024-08-19T14:17:17.134 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:17.134 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.134 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.134 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:17:17.134 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:53.026834+0000", 2024-08-19T14:17:17.135 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:17.135 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.135 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.135 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:17:17.135 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:53.026835+0000", 2024-08-19T14:17:17.135 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.5199999999999996e-07 2024-08-19T14:17:17.135 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.135 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.135 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:17:17.135 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:53.034134+0000", 2024-08-19T14:17:17.135 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0072991369999999998 2024-08-19T14:17:17.135 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.135 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.135 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:17:17.136 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:53.034144+0000", 2024-08-19T14:17:17.136 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0370000000000001e-05 2024-08-19T14:17:17.136 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.136 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.136 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:17:17.136 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:53.034149+0000", 2024-08-19T14:17:17.136 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.8130000000000003e-06 2024-08-19T14:17:17.136 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.136 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.136 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:17:17.136 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:53.034196+0000", 2024-08-19T14:17:17.136 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.7159e-05 2024-08-19T14:17:17.136 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.137 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.137 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:17:17.137 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:53.034211+0000", 2024-08-19T14:17:17.137 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4429e-05 2024-08-19T14:17:17.137 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.137 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.137 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:17:17.137 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:53.034701+0000", 2024-08-19T14:17:17.137 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00049043000000000003 2024-08-19T14:17:17.137 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.137 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.137 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:17:17.137 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:53.034733+0000", 2024-08-19T14:17:17.137 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.1545999999999999e-05 2024-08-19T14:17:17.138 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:17.138 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:17:17.138 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:17.138 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.138 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.138 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:38741 2.4 e112/102 2:2ddcb16b:::benchmark_data_smithi045_46415_object38740:head v 112'2448, mlcod=112'2444)", 2024-08-19T14:17:17.138 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:02.435487+0000", 2024-08-19T14:17:17.138 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 254.46152548699999, 2024-08-19T14:17:17.138 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0060987159999999997, 2024-08-19T14:17:17.138 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:17:17.138 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:17:17.138 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:17:17.138 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:17:17.138 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.139 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:17:17.139 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.435487+0000", 2024-08-19T14:17:17.139 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:17.139 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.139 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.139 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:17:17.139 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.435487+0000", 2024-08-19T14:17:17.139 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:17.139 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.139 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.139 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:17:17.139 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.435487+0000", 2024-08-19T14:17:17.139 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2699999999999999e-07 2024-08-19T14:17:17.140 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.140 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.140 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:17:17.140 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.435514+0000", 2024-08-19T14:17:17.140 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.6457000000000001e-05 2024-08-19T14:17:17.140 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.140 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.140 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:17:17.140 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.435524+0000", 2024-08-19T14:17:17.140 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0299000000000001e-05 2024-08-19T14:17:17.140 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.140 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.140 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:17:17.140 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.435531+0000", 2024-08-19T14:17:17.140 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.6529999999999997e-06 2024-08-19T14:17:17.141 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.141 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.141 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:17:17.141 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.435603+0000", 2024-08-19T14:17:17.141 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.1963000000000002e-05 2024-08-19T14:17:17.141 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.141 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.141 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:17:17.141 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.435622+0000", 2024-08-19T14:17:17.141 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8926000000000001e-05 2024-08-19T14:17:17.141 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.141 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.141 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:17:17.141 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.441541+0000", 2024-08-19T14:17:17.141 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0059191479999999999 2024-08-19T14:17:17.142 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.142 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.142 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:17:17.142 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.441585+0000", 2024-08-19T14:17:17.142 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.4743000000000002e-05 2024-08-19T14:17:17.142 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:17.142 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:17:17.142 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:17.142 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.142 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.142 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:40033 2.0 e113/102 2:0512d183:::benchmark_data_smithi045_46415_object40032:head v 113'2436, mlcod=113'2434)", 2024-08-19T14:17:17.142 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:02.895855+0000", 2024-08-19T14:17:17.142 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 254.001156794, 2024-08-19T14:17:17.142 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0095757759999999994, 2024-08-19T14:17:17.143 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:17:17.143 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:17:17.143 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:17:17.143 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:17:17.143 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.143 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:17:17.143 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.895855+0000", 2024-08-19T14:17:17.143 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:17.143 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.143 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.143 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:17:17.143 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.895855+0000", 2024-08-19T14:17:17.143 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:17.143 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.144 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.144 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:17:17.144 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.895856+0000", 2024-08-19T14:17:17.144 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.2099999999999995e-07 2024-08-19T14:17:17.144 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.144 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.144 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:17:17.144 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.895922+0000", 2024-08-19T14:17:17.144 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.5188e-05 2024-08-19T14:17:17.144 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.144 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.144 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:17:17.144 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.895935+0000", 2024-08-19T14:17:17.144 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3601000000000001e-05 2024-08-19T14:17:17.144 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.145 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.145 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:17:17.145 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.895943+0000", 2024-08-19T14:17:17.145 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.9170000000000006e-06 2024-08-19T14:17:17.145 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.145 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.145 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:17:17.145 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.896012+0000", 2024-08-19T14:17:17.145 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.9053000000000004e-05 2024-08-19T14:17:17.145 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.145 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.145 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:17:17.145 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.896035+0000", 2024-08-19T14:17:17.145 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2943999999999999e-05 2024-08-19T14:17:17.146 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.146 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.146 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:17:17.146 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.905344+0000", 2024-08-19T14:17:17.146 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0093086450000000008 2024-08-19T14:17:17.146 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.146 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.146 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:17:17.146 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.905431+0000", 2024-08-19T14:17:17.146 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.7507000000000001e-05 2024-08-19T14:17:17.146 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:17.146 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:17:17.146 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:17.147 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.147 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.147 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:40035 2.0 e113/102 2:0960ca99:::benchmark_data_smithi045_46415_object40034:head v 113'2437, mlcod=113'2434)", 2024-08-19T14:17:17.147 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:02.897909+0000", 2024-08-19T14:17:17.147 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 253.99910371499999, 2024-08-19T14:17:17.147 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0076402179999999998, 2024-08-19T14:17:17.147 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:17:17.147 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:17:17.147 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:17:17.147 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:17:17.148 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.148 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:17:17.148 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.897909+0000", 2024-08-19T14:17:17.148 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:17.148 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.148 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.148 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:17:17.148 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.897909+0000", 2024-08-19T14:17:17.148 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:17.148 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.148 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.148 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:17:17.148 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.897909+0000", 2024-08-19T14:17:17.148 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.33e-07 2024-08-19T14:17:17.149 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.149 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.149 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:17:17.149 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.897950+0000", 2024-08-19T14:17:17.149 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.0683000000000003e-05 2024-08-19T14:17:17.149 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.149 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.149 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:17:17.149 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.897966+0000", 2024-08-19T14:17:17.149 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6209999999999999e-05 2024-08-19T14:17:17.149 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.149 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.149 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:17:17.149 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.897974+0000", 2024-08-19T14:17:17.149 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4569999999999999e-06 2024-08-19T14:17:17.150 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.150 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.150 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:17:17.150 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.898035+0000", 2024-08-19T14:17:17.150 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.1033000000000003e-05 2024-08-19T14:17:17.150 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.150 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.150 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:17:17.150 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.898068+0000", 2024-08-19T14:17:17.150 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.3593000000000002e-05 2024-08-19T14:17:17.150 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.150 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.151 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:17:17.151 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.905435+0000", 2024-08-19T14:17:17.151 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0073662830000000004 2024-08-19T14:17:17.151 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.151 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.151 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:17:17.151 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.905549+0000", 2024-08-19T14:17:17.151 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000114026 2024-08-19T14:17:17.151 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:17.151 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:17:17.151 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:17.151 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.151 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.151 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:40069 2.0 e113/102 2:03e60c7a:::benchmark_data_smithi045_46415_object40068:head v 113'2440, mlcod=113'2436)", 2024-08-19T14:17:17.152 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:02.908487+0000", 2024-08-19T14:17:17.152 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 253.988524942, 2024-08-19T14:17:17.152 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0062491439999999999, 2024-08-19T14:17:17.152 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:17:17.152 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:17:17.152 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:17:17.152 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:17:17.152 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.152 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:17:17.152 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.908487+0000", 2024-08-19T14:17:17.152 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:17.152 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.152 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.152 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:17:17.153 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.908487+0000", 2024-08-19T14:17:17.153 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:17.153 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.153 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.153 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:17:17.153 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.908488+0000", 2024-08-19T14:17:17.153 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.8999999999999995e-07 2024-08-19T14:17:17.153 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.153 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.153 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:17:17.153 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.908517+0000", 2024-08-19T14:17:17.153 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.8643999999999999e-05 2024-08-19T14:17:17.153 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.153 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.153 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:17:17.154 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.908533+0000", 2024-08-19T14:17:17.154 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5770999999999999e-05 2024-08-19T14:17:17.154 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.154 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.154 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:17:17.154 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.908540+0000", 2024-08-19T14:17:17.154 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4000000000000003e-06 2024-08-19T14:17:17.154 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.154 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.154 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:17:17.154 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.908593+0000", 2024-08-19T14:17:17.154 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.3146999999999998e-05 2024-08-19T14:17:17.154 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.155 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.155 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:17:17.155 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.908614+0000", 2024-08-19T14:17:17.155 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0900999999999999e-05 2024-08-19T14:17:17.155 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.155 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.155 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:17:17.155 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.914692+0000", 2024-08-19T14:17:17.155 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0060776399999999996 2024-08-19T14:17:17.155 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.155 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.155 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:17:17.156 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.914736+0000", 2024-08-19T14:17:17.156 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.4851000000000001e-05 2024-08-19T14:17:17.156 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:17.156 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:17:17.156 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:17.156 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.156 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.156 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:43484 2.4 e114/102 2:285d10cd:::benchmark_data_smithi045_46415_object43483:head v 114'2732, mlcod=114'2729)", 2024-08-19T14:17:17.156 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:04.598813+0000", 2024-08-19T14:17:17.156 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 252.29819905700001, 2024-08-19T14:17:17.156 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0062693649999999998, 2024-08-19T14:17:17.156 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:17:17.156 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:17:17.156 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:17:17.157 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:17:17.157 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.157 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:17:17.157 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:04.598813+0000", 2024-08-19T14:17:17.157 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:17.157 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.157 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.157 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:17:17.157 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:04.598813+0000", 2024-08-19T14:17:17.157 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:17.157 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.157 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.157 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:17:17.157 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:04.598814+0000", 2024-08-19T14:17:17.158 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.9699999999999994e-07 2024-08-19T14:17:17.158 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.158 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.158 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:17:17.158 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:04.598850+0000", 2024-08-19T14:17:17.158 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.5865000000000003e-05 2024-08-19T14:17:17.158 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.158 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.158 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:17:17.158 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:04.598861+0000", 2024-08-19T14:17:17.158 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1284000000000001e-05 2024-08-19T14:17:17.158 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.158 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.159 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:17:17.159 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:04.598870+0000", 2024-08-19T14:17:17.159 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.9479999999999997e-06 2024-08-19T14:17:17.159 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.159 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.159 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:17:17.159 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:04.598935+0000", 2024-08-19T14:17:17.159 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.4364999999999998e-05 2024-08-19T14:17:17.159 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.159 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.159 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:17:17.159 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:04.598955+0000", 2024-08-19T14:17:17.159 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0849999999999999e-05 2024-08-19T14:17:17.159 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.159 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.160 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:17:17.160 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:04.604957+0000", 2024-08-19T14:17:17.160 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0060020000000000004 2024-08-19T14:17:17.160 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.160 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.160 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:17:17.160 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:04.605083+0000", 2024-08-19T14:17:17.160 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012505599999999999 2024-08-19T14:17:17.160 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:17.160 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:17:17.160 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:17.160 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.160 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.160 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:46495 2.4 e116/102 2:27583565:::benchmark_data_smithi045_46415_object46494:head v 116'2919, mlcod=116'2914)", 2024-08-19T14:17:17.161 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:06.037606+0000", 2024-08-19T14:17:17.161 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 250.85940666100001, 2024-08-19T14:17:17.161 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0079993879999999996, 2024-08-19T14:17:17.161 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:17:17.161 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:17:17.161 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:17:17.161 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:17:17.161 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.161 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:17:17.161 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:06.037606+0000", 2024-08-19T14:17:17.161 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:17.161 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.161 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.161 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:17:17.162 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:06.037606+0000", 2024-08-19T14:17:17.162 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:17.162 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.162 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.162 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:17:17.162 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:06.037606+0000", 2024-08-19T14:17:17.162 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2300000000000001e-07 2024-08-19T14:17:17.162 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.162 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.162 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:17:17.162 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:06.043150+0000", 2024-08-19T14:17:17.162 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0055434869999999997 2024-08-19T14:17:17.162 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.163 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.163 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:17:17.163 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:06.043160+0000", 2024-08-19T14:17:17.163 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0088999999999999e-05 2024-08-19T14:17:17.163 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.163 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.163 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:17:17.163 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:06.043167+0000", 2024-08-19T14:17:17.163 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.3309999999999998e-06 2024-08-19T14:17:17.163 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.163 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.163 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:17:17.164 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:06.043225+0000", 2024-08-19T14:17:17.164 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.8062000000000001e-05 2024-08-19T14:17:17.164 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.164 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.164 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:17:17.164 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:06.043246+0000", 2024-08-19T14:17:17.164 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1316000000000001e-05 2024-08-19T14:17:17.164 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.164 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.164 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:17:17.164 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:06.045577+0000", 2024-08-19T14:17:17.164 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0023308890000000001 2024-08-19T14:17:17.165 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.165 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.165 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:17:17.165 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:06.045605+0000", 2024-08-19T14:17:17.165 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.7691e-05 2024-08-19T14:17:17.165 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:17.165 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:17:17.165 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:17.165 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.165 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.165 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:48912 2.7 e117/102 2:e6ffd652:::benchmark_data_smithi045_46415_object48911:head v 117'2997, mlcod=117'2992)", 2024-08-19T14:17:17.165 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:07.468103+0000", 2024-08-19T14:17:17.165 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 249.42890972999999, 2024-08-19T14:17:17.166 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0061113349999999999, 2024-08-19T14:17:17.166 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:17:17.166 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:17:17.166 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:17:17.166 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:17:17.166 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.166 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:17:17.166 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468103+0000", 2024-08-19T14:17:17.166 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:17.166 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.166 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.166 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:17:17.166 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468103+0000", 2024-08-19T14:17:17.166 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:17.167 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.167 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.167 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:17:17.167 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468103+0000", 2024-08-19T14:17:17.167 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.8000000000000002e-07 2024-08-19T14:17:17.167 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.167 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.167 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:17:17.167 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468136+0000", 2024-08-19T14:17:17.167 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.3411000000000002e-05 2024-08-19T14:17:17.167 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.167 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.167 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:17:17.167 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468143+0000", 2024-08-19T14:17:17.168 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.5869999999999998e-06 2024-08-19T14:17:17.168 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.168 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.168 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:17:17.168 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468146+0000", 2024-08-19T14:17:17.168 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.304e-06 2024-08-19T14:17:17.168 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.168 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.168 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:17:17.168 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.470630+0000", 2024-08-19T14:17:17.168 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0024838009999999999 2024-08-19T14:17:17.168 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.168 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.168 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:17:17.169 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.470643+0000", 2024-08-19T14:17:17.169 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2964e-05 2024-08-19T14:17:17.169 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.169 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.169 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:17:17.169 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.474173+0000", 2024-08-19T14:17:17.169 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.003530423 2024-08-19T14:17:17.169 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.169 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.169 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:17:17.169 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.474214+0000", 2024-08-19T14:17:17.169 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.0565000000000002e-05 2024-08-19T14:17:17.169 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:17.169 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:17:17.170 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:17.170 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.170 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.170 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:48971 2.7 e117/102 2:ecf6a539:::benchmark_data_smithi045_46415_object48970:head v 117'2999, mlcod=117'2992)", 2024-08-19T14:17:17.170 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:07.468206+0000", 2024-08-19T14:17:17.170 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 249.42880579600001, 2024-08-19T14:17:17.170 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0076013820000000003, 2024-08-19T14:17:17.170 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:17:17.170 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:17:17.170 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:17:17.170 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:17:17.170 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.170 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:17:17.170 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468206+0000", 2024-08-19T14:17:17.171 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:17.171 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.171 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.171 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:17:17.171 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468206+0000", 2024-08-19T14:17:17.171 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:17.171 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.171 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.171 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:17:17.171 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468207+0000", 2024-08-19T14:17:17.171 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6899999999999999e-07 2024-08-19T14:17:17.171 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.171 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.171 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:17:17.172 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468248+0000", 2024-08-19T14:17:17.172 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.0985999999999997e-05 2024-08-19T14:17:17.172 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.172 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.172 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:17:17.172 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468254+0000", 2024-08-19T14:17:17.172 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.9769999999999997e-06 2024-08-19T14:17:17.172 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.172 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.172 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:17:17.172 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468257+0000", 2024-08-19T14:17:17.172 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.0259999999999999e-06 2024-08-19T14:17:17.172 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.172 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.173 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:17:17.173 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.471956+0000", 2024-08-19T14:17:17.173 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0036996059999999998 2024-08-19T14:17:17.173 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.173 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.173 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:17:17.173 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.471974+0000", 2024-08-19T14:17:17.173 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8219999999999998e-05 2024-08-19T14:17:17.173 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.173 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.173 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:17:17.173 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.475774+0000", 2024-08-19T14:17:17.173 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0037994729999999998 2024-08-19T14:17:17.173 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.174 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.174 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:17:17.174 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.475808+0000", 2024-08-19T14:17:17.174 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.3924999999999998e-05 2024-08-19T14:17:17.174 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:17.174 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:17:17.174 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:17.174 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.174 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.174 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:48978 2.7 e117/102 2:e11d5fd6:::benchmark_data_smithi045_46415_object48977:head v 117'3000, mlcod=117'2992)", 2024-08-19T14:17:17.174 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:07.468262+0000", 2024-08-19T14:17:17.174 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 249.42875009700001, 2024-08-19T14:17:17.174 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0075744890000000002, 2024-08-19T14:17:17.175 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:17:17.175 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:17:17.175 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:17:17.175 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:17:17.175 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.175 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:17:17.175 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468262+0000", 2024-08-19T14:17:17.175 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:17.175 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.175 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.175 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:17:17.175 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468262+0000", 2024-08-19T14:17:17.175 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:17.175 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.175 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.176 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:17:17.176 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468262+0000", 2024-08-19T14:17:17.176 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.04e-07 2024-08-19T14:17:17.176 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.176 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.176 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:17:17.176 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468309+0000", 2024-08-19T14:17:17.176 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.6913999999999998e-05 2024-08-19T14:17:17.176 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.176 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.176 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:17:17.176 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468315+0000", 2024-08-19T14:17:17.176 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7139999999999998e-06 2024-08-19T14:17:17.176 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.176 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.177 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:17:17.177 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468318+0000", 2024-08-19T14:17:17.177 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.117e-06 2024-08-19T14:17:17.177 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.177 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.177 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:17:17.177 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.472643+0000", 2024-08-19T14:17:17.177 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0043251950000000004 2024-08-19T14:17:17.177 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.177 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.177 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:17:17.178 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.472657+0000", 2024-08-19T14:17:17.178 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3478e-05 2024-08-19T14:17:17.178 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.178 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.178 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:17:17.178 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.475812+0000", 2024-08-19T14:17:17.178 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0031556259999999999 2024-08-19T14:17:17.178 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.178 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.178 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:17:17.178 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.475837+0000", 2024-08-19T14:17:17.178 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.4241000000000001e-05 2024-08-19T14:17:17.178 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:17.178 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:17:17.179 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:17.179 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.179 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.179 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:48995 2.7 e117/102 2:e01c8116:::benchmark_data_smithi045_46415_object48994:head v 117'3001, mlcod=117'2992)", 2024-08-19T14:17:17.179 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:07.468324+0000", 2024-08-19T14:17:17.179 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 249.42868856999999, 2024-08-19T14:17:17.179 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0090529970000000001, 2024-08-19T14:17:17.179 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:17:17.179 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:17:17.179 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:17:17.179 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:17:17.179 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.179 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:17:17.179 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468324+0000", 2024-08-19T14:17:17.179 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:17.180 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.180 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.180 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:17:17.180 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468324+0000", 2024-08-19T14:17:17.180 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:17.180 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.180 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.180 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:17:17.180 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468324+0000", 2024-08-19T14:17:17.180 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.97e-07 2024-08-19T14:17:17.180 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.180 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.180 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:17:17.181 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468388+0000", 2024-08-19T14:17:17.181 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.4523999999999994e-05 2024-08-19T14:17:17.181 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.181 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.181 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:17:17.181 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468395+0000", 2024-08-19T14:17:17.181 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.579e-06 2024-08-19T14:17:17.181 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.181 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.181 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:17:17.181 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468398+0000", 2024-08-19T14:17:17.181 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.2780000000000002e-06 2024-08-19T14:17:17.181 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.181 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.182 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:17:17.182 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.473325+0000", 2024-08-19T14:17:17.182 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00492626 2024-08-19T14:17:17.182 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.182 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.182 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:17:17.182 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.473343+0000", 2024-08-19T14:17:17.182 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8788999999999999e-05 2024-08-19T14:17:17.182 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.182 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.182 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:17:17.182 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.477310+0000", 2024-08-19T14:17:17.182 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0039662819999999998 2024-08-19T14:17:17.182 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.183 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.183 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:17:17.183 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.477377+0000", 2024-08-19T14:17:17.183 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.7088000000000005e-05 2024-08-19T14:17:17.183 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:17.183 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:17:17.183 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:17.183 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.183 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.183 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:48996 2.7 e117/102 2:e8abf67b:::benchmark_data_smithi045_46415_object48995:head v 117'3002, mlcod=117'2992)", 2024-08-19T14:17:17.183 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:07.468406+0000", 2024-08-19T14:17:17.183 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 249.428606562, 2024-08-19T14:17:17.183 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0090294899999999994, 2024-08-19T14:17:17.183 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:17:17.184 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:17:17.184 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:17:17.184 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:17:17.184 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.184 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:17:17.184 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468406+0000", 2024-08-19T14:17:17.184 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:17.184 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.184 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.184 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:17:17.184 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468406+0000", 2024-08-19T14:17:17.184 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:17.185 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.185 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.185 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:17:17.185 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468406+0000", 2024-08-19T14:17:17.185 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0900000000000001e-07 2024-08-19T14:17:17.185 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.185 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.185 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:17:17.185 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468447+0000", 2024-08-19T14:17:17.185 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.0992000000000003e-05 2024-08-19T14:17:17.185 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.185 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.185 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:17:17.185 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468452+0000", 2024-08-19T14:17:17.186 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2179999999999998e-06 2024-08-19T14:17:17.186 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.186 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.186 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:17:17.186 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468455+0000", 2024-08-19T14:17:17.186 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.796e-06 2024-08-19T14:17:17.186 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.186 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.186 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:17:17.186 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.474288+0000", 2024-08-19T14:17:17.186 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0058327689999999998 2024-08-19T14:17:17.186 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.186 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.187 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:17:17.187 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.474307+0000", 2024-08-19T14:17:17.187 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9031000000000001e-05 2024-08-19T14:17:17.187 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.187 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.187 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:17:17.187 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.477383+0000", 2024-08-19T14:17:17.187 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0030758790000000001 2024-08-19T14:17:17.187 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.187 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.187 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:17:17.187 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.477435+0000", 2024-08-19T14:17:17.187 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2596000000000003e-05 2024-08-19T14:17:17.187 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:17.187 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:17:17.187 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:17.188 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.188 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.188 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:49005 2.7 e117/102 2:e58e73b6:::benchmark_data_smithi045_46415_object49004:head v 117'3003, mlcod=117'2992)", 2024-08-19T14:17:17.188 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:07.468460+0000", 2024-08-19T14:17:17.188 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 249.42855246100001, 2024-08-19T14:17:17.188 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0098532910000000001, 2024-08-19T14:17:17.188 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:17:17.188 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:17:17.188 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:17:17.188 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:17:17.188 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.188 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:17:17.188 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468460+0000", 2024-08-19T14:17:17.188 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:17.189 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.189 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.189 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:17:17.189 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468460+0000", 2024-08-19T14:17:17.189 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:17.189 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.189 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.189 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:17:17.189 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468460+0000", 2024-08-19T14:17:17.189 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5900000000000001e-07 2024-08-19T14:17:17.189 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.189 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.189 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:17:17.189 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468518+0000", 2024-08-19T14:17:17.190 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.8536000000000003e-05 2024-08-19T14:17:17.190 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.190 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.190 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:17:17.190 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468524+0000", 2024-08-19T14:17:17.190 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.8039999999999998e-06 2024-08-19T14:17:17.190 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.190 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.190 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:17:17.190 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468527+0000", 2024-08-19T14:17:17.190 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.004e-06 2024-08-19T14:17:17.190 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.190 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.190 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:17:17.191 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.475085+0000", 2024-08-19T14:17:17.191 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0065574090000000002 2024-08-19T14:17:17.191 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.191 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.191 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:17:17.191 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.475106+0000", 2024-08-19T14:17:17.191 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1056999999999998e-05 2024-08-19T14:17:17.191 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.191 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.191 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:17:17.191 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.478267+0000", 2024-08-19T14:17:17.191 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0031614049999999999 2024-08-19T14:17:17.191 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.191 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.191 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:17:17.192 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.478313+0000", 2024-08-19T14:17:17.192 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.5917000000000001e-05 2024-08-19T14:17:17.192 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:17.192 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:17:17.192 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:17.192 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.192 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.192 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:49015 2.7 e117/102 2:e4d8c3e1:::benchmark_data_smithi045_46415_object49014:head v 117'3004, mlcod=117'2992)", 2024-08-19T14:17:17.192 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:07.468533+0000", 2024-08-19T14:17:17.192 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 249.42847939699999, 2024-08-19T14:17:17.192 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0098176270000000006, 2024-08-19T14:17:17.192 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:17:17.192 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:17:17.192 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:17:17.192 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:17:17.193 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.193 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:17:17.193 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468533+0000", 2024-08-19T14:17:17.193 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:17.193 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.193 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.193 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:17:17.193 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468533+0000", 2024-08-19T14:17:17.193 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:17.193 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.193 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.193 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:17:17.193 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468533+0000", 2024-08-19T14:17:17.193 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2600000000000001e-07 2024-08-19T14:17:17.193 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.194 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.194 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:17:17.194 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468577+0000", 2024-08-19T14:17:17.194 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.3436000000000002e-05 2024-08-19T14:17:17.194 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.194 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.194 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:17:17.194 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468582+0000", 2024-08-19T14:17:17.194 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4399999999999996e-06 2024-08-19T14:17:17.194 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.194 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.194 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:17:17.195 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468585+0000", 2024-08-19T14:17:17.195 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.8619999999999999e-06 2024-08-19T14:17:17.195 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.195 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.195 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:17:17.195 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.475870+0000", 2024-08-19T14:17:17.195 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0072852919999999996 2024-08-19T14:17:17.195 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.195 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.195 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:17:17.195 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.475894+0000", 2024-08-19T14:17:17.195 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.3858000000000001e-05 2024-08-19T14:17:17.195 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.195 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.196 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:17:17.196 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.478319+0000", 2024-08-19T14:17:17.196 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0024247729999999999 2024-08-19T14:17:17.196 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.196 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.196 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:17:17.196 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.478350+0000", 2024-08-19T14:17:17.196 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.1739999999999998e-05 2024-08-19T14:17:17.196 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:17.196 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:17:17.196 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:17.196 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.196 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.196 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:49016 2.7 e117/102 2:e52959d5:::benchmark_data_smithi045_46415_object49015:head v 117'3005, mlcod=117'2992)", 2024-08-19T14:17:17.197 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:07.468590+0000", 2024-08-19T14:17:17.197 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 249.42842251900001, 2024-08-19T14:17:17.197 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.011903967999999999, 2024-08-19T14:17:17.197 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:17:17.197 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:17:17.197 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:17:17.197 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:17:17.197 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.197 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:17:17.197 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468590+0000", 2024-08-19T14:17:17.197 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:17.197 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.197 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.198 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:17:17.198 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468590+0000", 2024-08-19T14:17:17.198 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:17.198 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.198 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.198 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:17:17.198 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468590+0000", 2024-08-19T14:17:17.198 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.54e-07 2024-08-19T14:17:17.198 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.198 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.198 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:17:17.198 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468632+0000", 2024-08-19T14:17:17.198 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.2459999999999997e-05 2024-08-19T14:17:17.199 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.199 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.199 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:17:17.199 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468638+0000", 2024-08-19T14:17:17.199 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2410000000000003e-06 2024-08-19T14:17:17.199 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.199 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.199 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:17:17.199 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468641+0000", 2024-08-19T14:17:17.199 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.061e-06 2024-08-19T14:17:17.199 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.199 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.199 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:17:17.199 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.476571+0000", 2024-08-19T14:17:17.200 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0079302920000000002 2024-08-19T14:17:17.200 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.200 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.200 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:17:17.200 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.476593+0000", 2024-08-19T14:17:17.200 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2045e-05 2024-08-19T14:17:17.200 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.200 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.200 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:17:17.200 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.480447+0000", 2024-08-19T14:17:17.201 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.003853825 2024-08-19T14:17:17.201 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.201 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.201 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:17:17.201 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.480494+0000", 2024-08-19T14:17:17.201 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.689e-05 2024-08-19T14:17:17.201 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:17.201 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:17:17.201 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:17.201 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.201 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.201 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:49017 2.7 e117/102 2:eca28b17:::benchmark_data_smithi045_46415_object49016:head v 117'3006, mlcod=117'2992)", 2024-08-19T14:17:17.201 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:07.468646+0000", 2024-08-19T14:17:17.201 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 249.42836665499999, 2024-08-19T14:17:17.201 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.011882515999999999, 2024-08-19T14:17:17.202 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:17:17.202 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:17:17.202 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:17:17.202 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:17:17.202 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.202 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:17:17.202 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468646+0000", 2024-08-19T14:17:17.202 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:17.202 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.202 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.202 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:17:17.202 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468646+0000", 2024-08-19T14:17:17.202 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:17.203 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.203 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.203 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:17:17.203 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468646+0000", 2024-08-19T14:17:17.203 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5800000000000001e-07 2024-08-19T14:17:17.203 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.203 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.203 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:17:17.203 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468686+0000", 2024-08-19T14:17:17.203 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.0732999999999998e-05 2024-08-19T14:17:17.203 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.203 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.203 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:17:17.203 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468692+0000", 2024-08-19T14:17:17.204 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0860000000000001e-06 2024-08-19T14:17:17.204 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.204 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.204 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:17:17.204 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468694+0000", 2024-08-19T14:17:17.204 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.864e-06 2024-08-19T14:17:17.204 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.204 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.204 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:17:17.204 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.477463+0000", 2024-08-19T14:17:17.204 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0087685190000000007 2024-08-19T14:17:17.204 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.204 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.204 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:17:17.205 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.477483+0000", 2024-08-19T14:17:17.205 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0449999999999999e-05 2024-08-19T14:17:17.205 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.205 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.205 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:17:17.205 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.480501+0000", 2024-08-19T14:17:17.205 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.003017802 2024-08-19T14:17:17.205 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.205 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.205 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:17:17.205 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.480528+0000", 2024-08-19T14:17:17.205 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.6903999999999999e-05 2024-08-19T14:17:17.205 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:17.205 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:17:17.206 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:17.206 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.206 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.206 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:49027 2.7 e117/102 2:e96ce7ac:::benchmark_data_smithi045_46415_object49026:head v 117'3007, mlcod=117'2992)", 2024-08-19T14:17:17.206 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:07.468699+0000", 2024-08-19T14:17:17.206 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 249.42831324299999, 2024-08-19T14:17:17.206 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.011858521, 2024-08-19T14:17:17.206 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:17:17.206 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:17:17.206 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:17:17.206 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:17:17.206 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.206 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:17:17.207 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468699+0000", 2024-08-19T14:17:17.207 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:17.207 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.207 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.207 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:17:17.207 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468699+0000", 2024-08-19T14:17:17.207 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:17.207 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.207 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.207 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:17:17.207 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468699+0000", 2024-08-19T14:17:17.207 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5599999999999999e-07 2024-08-19T14:17:17.207 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.207 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.208 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:17:17.208 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468743+0000", 2024-08-19T14:17:17.208 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.3741e-05 2024-08-19T14:17:17.208 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.208 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.208 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:17:17.208 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468748+0000", 2024-08-19T14:17:17.208 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1000000000000003e-06 2024-08-19T14:17:17.208 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.208 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.208 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:17:17.208 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468751+0000", 2024-08-19T14:17:17.208 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.8260000000000001e-06 2024-08-19T14:17:17.208 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.209 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.209 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:17:17.209 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.478397+0000", 2024-08-19T14:17:17.209 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0096462240000000001 2024-08-19T14:17:17.209 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.209 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.209 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:17:17.209 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.478415+0000", 2024-08-19T14:17:17.209 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8450000000000001e-05 2024-08-19T14:17:17.209 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.209 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.209 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:17:17.209 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.480533+0000", 2024-08-19T14:17:17.209 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.002117196 2024-08-19T14:17:17.210 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.210 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.210 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:17:17.210 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.480558+0000", 2024-08-19T14:17:17.210 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.4828e-05 2024-08-19T14:17:17.210 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:17.210 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:17:17.210 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:17.210 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.210 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.210 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:49030 2.7 e117/102 2:e915209e:::benchmark_data_smithi045_46415_object49029:head v 117'3008, mlcod=117'2992)", 2024-08-19T14:17:17.210 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:07.468755+0000", 2024-08-19T14:17:17.210 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 249.42825695600001, 2024-08-19T14:17:17.210 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.015403441, 2024-08-19T14:17:17.211 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:17:17.211 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:17:17.211 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:17:17.211 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:17:17.211 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.211 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:17:17.211 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468755+0000", 2024-08-19T14:17:17.211 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:17.211 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.211 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.211 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:17:17.211 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468755+0000", 2024-08-19T14:17:17.211 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:17.211 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.212 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.212 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:17:17.212 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468755+0000", 2024-08-19T14:17:17.212 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5599999999999999e-07 2024-08-19T14:17:17.212 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.212 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.212 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:17:17.212 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468805+0000", 2024-08-19T14:17:17.212 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.9622e-05 2024-08-19T14:17:17.212 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.212 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.212 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:17:17.212 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468811+0000", 2024-08-19T14:17:17.212 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.6119999999999998e-06 2024-08-19T14:17:17.212 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.213 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.213 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:17:17.213 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468813+0000", 2024-08-19T14:17:17.213 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.7769999999999998e-06 2024-08-19T14:17:17.213 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.213 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.213 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:17:17.213 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.479393+0000", 2024-08-19T14:17:17.213 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.010579646999999999 2024-08-19T14:17:17.213 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.213 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.213 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:17:17.213 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.479412+0000", 2024-08-19T14:17:17.213 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8728000000000001e-05 2024-08-19T14:17:17.214 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.214 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.214 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:17:17.214 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.484094+0000", 2024-08-19T14:17:17.214 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0046820009999999999 2024-08-19T14:17:17.214 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.214 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.214 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:17:17.214 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.484159+0000", 2024-08-19T14:17:17.214 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.4898000000000001e-05 2024-08-19T14:17:17.214 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:17.214 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:17:17.215 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:17.215 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.215 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.215 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:49049 2.7 e117/102 2:eb29c0a1:::benchmark_data_smithi045_46415_object49048:head v 117'3009, mlcod=117'2992)", 2024-08-19T14:17:17.215 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:07.468819+0000", 2024-08-19T14:17:17.215 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 249.428193709, 2024-08-19T14:17:17.215 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.015383332, 2024-08-19T14:17:17.215 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:17:17.215 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:17:17.215 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:17:17.215 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:17:17.215 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.215 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:17:17.215 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468819+0000", 2024-08-19T14:17:17.215 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:17.215 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.216 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.216 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:17:17.216 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468819+0000", 2024-08-19T14:17:17.216 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:17.216 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.216 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.216 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:17:17.216 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468819+0000", 2024-08-19T14:17:17.216 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2000000000000001e-07 2024-08-19T14:17:17.216 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.216 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.216 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:17:17.217 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468854+0000", 2024-08-19T14:17:17.217 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.5079000000000001e-05 2024-08-19T14:17:17.217 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.217 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.217 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:17:17.217 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468859+0000", 2024-08-19T14:17:17.217 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1710000000000001e-06 2024-08-19T14:17:17.217 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.217 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.217 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:17:17.217 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468862+0000", 2024-08-19T14:17:17.217 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.0599999999999999e-06 2024-08-19T14:17:17.217 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.218 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.218 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:17:17.218 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.480596+0000", 2024-08-19T14:17:17.218 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.011734058 2024-08-19T14:17:17.218 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.218 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.218 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:17:17.218 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.480615+0000", 2024-08-19T14:17:17.219 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8919e-05 2024-08-19T14:17:17.219 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.219 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.219 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:17:17.219 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.484164+0000", 2024-08-19T14:17:17.219 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.003548486 2024-08-19T14:17:17.219 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:17.219 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:17.219 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:17:17.219 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.484202+0000", 2024-08-19T14:17:17.219 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.8339e-05 2024-08-19T14:17:17.219 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:17.219 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:17:17.219 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:17.219 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:17.220 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:17:17.220 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:17:17.220 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:17:17.254 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:17.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:17.255 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:17:17.311 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 15 16 44400 44384 184.903 144.125 0.00406116 0.00540281 2024-08-19T14:17:17.355 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:17:17.385 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:17.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:17.386 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:17:17.487 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:17:17.519 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:17.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:17.521 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:17:17.622 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:17:17.634 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-19T14:17:17.655 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:17.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:17.656 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:17:17.708 INFO:teuthology.orchestra.run.smithi031.stderr:set pool 3 pg_num to 26 2024-08-19T14:17:17.757 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:17:17.788 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:17.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:17.789 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:17:17.891 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:17:17.917 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:17 smithi045 ceph-mon[20143]: pgmap v553: 27 pgs: 27 active+clean; 2.1 GiB data, 7.6 GiB used, 797 GiB / 805 GiB avail; 178 MiB/s wr, 2.85k op/s 2024-08-19T14:17:17.917 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:17 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2540320774' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "26"}]: dispatch 2024-08-19T14:17:17.925 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:17.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:17.926 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:17:17.952 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-08-19T14:17:18.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:17 smithi032 ceph-mon[20004]: pgmap v553: 27 pgs: 27 active+clean; 2.1 GiB data, 7.6 GiB used, 797 GiB / 805 GiB avail; 178 MiB/s wr, 2.85k op/s 2024-08-19T14:17:18.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:17 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2540320774' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "26"}]: dispatch 2024-08-19T14:17:18.027 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:17:18.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:18.066 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:18.067 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:17:18.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:17 smithi031 ceph-mon[16336]: pgmap v553: 27 pgs: 27 active+clean; 2.1 GiB data, 7.6 GiB used, 797 GiB / 805 GiB avail; 178 MiB/s wr, 2.85k op/s 2024-08-19T14:17:18.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:17 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2540320774' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "26"}]: dispatch 2024-08-19T14:17:18.168 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:17:18.201 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:18.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:18.203 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:17:18.304 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:17:18.312 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 16 16 46889 46873 183.052 155.562 0.00676732 0.00545779 2024-08-19T14:17:18.346 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:18.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:18.347 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:17:18.379 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:17:18.448 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:17:18.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:18.480 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:18.480 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:17:18.581 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:17:18.616 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:18.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:18.617 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:17:18.718 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:17:18.741 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:18.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:18.742 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:17:18.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:18 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2540320774' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "26"}]': finished 2024-08-19T14:17:18.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:18 smithi045 ceph-mon[20143]: osdmap e233: 12 total, 12 up, 9 in 2024-08-19T14:17:18.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:18 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-08-19T14:17:18.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:18 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-08-19T14:17:18.843 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:17:18.864 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:18.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:18.865 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:17:18.926 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:17:18.927 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [], 2024-08-19T14:17:18.927 INFO:teuthology.orchestra.run.smithi045.stdout: "num_ops": 0 2024-08-19T14:17:18.927 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:17:18.965 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:17:18.987 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:18.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:18.988 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:17:19.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:18 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2540320774' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "26"}]': finished 2024-08-19T14:17:19.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:18 smithi032 ceph-mon[20004]: osdmap e233: 12 total, 12 up, 9 in 2024-08-19T14:17:19.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:18 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-08-19T14:17:19.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:18 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-08-19T14:17:19.077 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:18 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2540320774' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "26"}]': finished 2024-08-19T14:17:19.077 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:18 smithi031 ceph-mon[16336]: osdmap e233: 12 total, 12 up, 9 in 2024-08-19T14:17:19.077 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:18 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-08-19T14:17:19.077 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:18 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-08-19T14:17:19.089 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:17:19.109 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:19.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:19.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:17:19.212 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:17:19.234 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:19.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:19.235 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:17:19.312 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 17 16 47846 47830 175.803 59.8125 0.028416 0.00549305 2024-08-19T14:17:19.336 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:17:19.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:19.360 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:19.360 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:17:19.461 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:17:19.486 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:19.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:19.487 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:17:19.528 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-08-19T14:17:19.588 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:17:19.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:19.614 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:19.614 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:17:19.715 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:17:19.746 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:19.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:19.748 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:17:19.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:19 smithi045 ceph-mon[20143]: pgmap v555: 27 pgs: 27 active+clean; 2.3 GiB data, 8.1 GiB used, 797 GiB / 805 GiB avail; 155 MiB/s wr, 2.48k op/s 2024-08-19T14:17:19.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:19 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "26"}]': finished 2024-08-19T14:17:19.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:19 smithi045 ceph-mon[20143]: osdmap e234: 12 total, 12 up, 9 in 2024-08-19T14:17:19.814 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:17:19.848 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:17:19.882 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:19.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:19.883 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:17:19.984 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:17:20.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:19 smithi032 ceph-mon[20004]: pgmap v555: 27 pgs: 27 active+clean; 2.3 GiB data, 8.1 GiB used, 797 GiB / 805 GiB avail; 155 MiB/s wr, 2.48k op/s 2024-08-19T14:17:20.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:19 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "26"}]': finished 2024-08-19T14:17:20.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:19 smithi032 ceph-mon[20004]: osdmap e234: 12 total, 12 up, 9 in 2024-08-19T14:17:20.019 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:20.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:20.020 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:17:20.081 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:19 smithi031 ceph-mon[16336]: pgmap v555: 27 pgs: 27 active+clean; 2.3 GiB data, 8.1 GiB used, 797 GiB / 805 GiB avail; 155 MiB/s wr, 2.48k op/s 2024-08-19T14:17:20.082 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:19 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "26"}]': finished 2024-08-19T14:17:20.082 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:19 smithi031 ceph-mon[16336]: osdmap e234: 12 total, 12 up, 9 in 2024-08-19T14:17:20.120 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:17:20.141 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:20.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:20.142 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:17:20.243 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:17:20.279 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:20.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:20.280 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:17:20.312 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 18 16 48984 48968 169.988 71.125 0.00528601 0.00587819 2024-08-19T14:17:20.318 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:17:20.318 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [], 2024-08-19T14:17:20.318 INFO:teuthology.orchestra.run.smithi045.stdout: "complaint_time": 30, 2024-08-19T14:17:20.318 INFO:teuthology.orchestra.run.smithi045.stdout: "num_blocked_ops": 0 2024-08-19T14:17:20.319 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:17:20.381 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:17:20.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:20.420 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:20.420 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:17:20.522 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:17:20.524 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:17:20.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:20.572 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:20.572 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:17:20.674 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:17:20.723 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:20.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:20.724 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:17:20.825 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:17:20.851 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:20.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:20.852 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:17:20.933 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:20 smithi045 ceph-mon[20143]: osdmap e235: 12 total, 12 up, 9 in 2024-08-19T14:17:20.933 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:20 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-08-19T14:17:20.933 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:20 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-08-19T14:17:20.935 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:20 smithi031 ceph-mon[16336]: osdmap e235: 12 total, 12 up, 9 in 2024-08-19T14:17:20.935 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:20 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-08-19T14:17:20.936 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:20 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-08-19T14:17:20.952 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:17:20.958 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-08-19T14:17:20.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:20.975 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:20.975 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:17:21.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:20 smithi032 ceph-mon[20004]: osdmap e235: 12 total, 12 up, 9 in 2024-08-19T14:17:21.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:20 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-08-19T14:17:21.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:20 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-08-19T14:17:21.076 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:17:21.097 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:21.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:21.098 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:17:21.199 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:17:21.221 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:21.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:21.222 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:17:21.263 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:17:21.313 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 19 16 50196 50180 165.028 75.75 0.00169419 0.00587622 2024-08-19T14:17:21.322 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:17:21.345 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:21.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:21.346 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:17:21.447 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:17:21.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:21.470 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:21.470 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:17:21.571 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:17:21.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:21.594 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:21.594 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:17:21.695 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:17:21.696 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:17:21.696 INFO:teuthology.orchestra.run.smithi045.stdout: "size": 20, 2024-08-19T14:17:21.696 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 600, 2024-08-19T14:17:21.697 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [ 2024-08-19T14:17:21.697 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.697 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24580.0:88312 2.a 2:5ace1bf2:::benchmark_data_smithi045_46415_object88311:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-08-19T14:17:21.697 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:42.807146+0000", 2024-08-19T14:17:21.697 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 218.88741114300001, 2024-08-19T14:17:21.697 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20714613400000001, 2024-08-19T14:17:21.697 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:17:21.697 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:17:21.697 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:17:21.697 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:17:21.697 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24580", 2024-08-19T14:17:21.697 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:17:21.697 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 88312 2024-08-19T14:17:21.698 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.698 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:17:21.698 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.698 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:17:21.698 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.807146+0000", 2024-08-19T14:17:21.698 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:21.698 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.698 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.698 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:17:21.698 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.807146+0000", 2024-08-19T14:17:21.698 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:21.698 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.698 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.699 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:17:21.699 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.807147+0000", 2024-08-19T14:17:21.699 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2240000000000001e-06 2024-08-19T14:17:21.699 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.699 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.699 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:17:21.699 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.807170+0000", 2024-08-19T14:17:21.699 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.3278e-05 2024-08-19T14:17:21.699 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.699 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.699 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:17:21.699 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.807181+0000", 2024-08-19T14:17:21.699 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1049e-05 2024-08-19T14:17:21.699 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.700 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.700 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:17:21.700 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.807188+0000", 2024-08-19T14:17:21.700 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.8730000000000001e-06 2024-08-19T14:17:21.700 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.700 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.700 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:17:21.700 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.807263+0000", 2024-08-19T14:17:21.700 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.5128000000000003e-05 2024-08-19T14:17:21.700 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.700 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.700 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:17:21.700 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.807423+0000", 2024-08-19T14:17:21.700 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000160058 2024-08-19T14:17:21.701 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.701 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.701 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 3,9", 2024-08-19T14:17:21.701 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.807606+0000", 2024-08-19T14:17:21.701 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000182354 2024-08-19T14:17:21.701 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.701 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.701 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:17:21.701 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.810474+0000", 2024-08-19T14:17:21.701 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0028687500000000002 2024-08-19T14:17:21.701 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.701 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.701 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:17:21.702 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.811576+0000", 2024-08-19T14:17:21.702 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0011014900000000001 2024-08-19T14:17:21.702 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.702 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.702 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:17:21.702 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:43.014220+0000", 2024-08-19T14:17:21.702 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20264374800000001 2024-08-19T14:17:21.702 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.702 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.702 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:17:21.702 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:43.014258+0000", 2024-08-19T14:17:21.702 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.8318999999999997e-05 2024-08-19T14:17:21.702 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.702 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.703 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:17:21.703 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:43.014292+0000", 2024-08-19T14:17:21.703 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.3862999999999997e-05 2024-08-19T14:17:21.703 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:21.703 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:17:21.703 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:21.703 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.703 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.703 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24580.0:88332 2.a 2:5d5c27b9:::benchmark_data_smithi045_46415_object88331:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-08-19T14:17:21.703 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:42.823149+0000", 2024-08-19T14:17:21.703 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 218.87140765500001, 2024-08-19T14:17:21.703 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.19130311899999999, 2024-08-19T14:17:21.703 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:17:21.704 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:17:21.704 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:17:21.704 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:17:21.704 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24580", 2024-08-19T14:17:21.704 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:17:21.704 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 88332 2024-08-19T14:17:21.704 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.704 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:17:21.704 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.704 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:17:21.704 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.823149+0000", 2024-08-19T14:17:21.704 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:21.704 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.704 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.705 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:17:21.705 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.823149+0000", 2024-08-19T14:17:21.705 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:21.705 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.705 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.705 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:17:21.705 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.823150+0000", 2024-08-19T14:17:21.705 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.268e-06 2024-08-19T14:17:21.705 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.705 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.705 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:17:21.705 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.823169+0000", 2024-08-19T14:17:21.706 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8111999999999999e-05 2024-08-19T14:17:21.706 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.706 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.706 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:17:21.706 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.823179+0000", 2024-08-19T14:17:21.706 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0389e-05 2024-08-19T14:17:21.706 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.706 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.706 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:17:21.706 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.823185+0000", 2024-08-19T14:17:21.706 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.5470000000000004e-06 2024-08-19T14:17:21.707 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.707 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.707 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:17:21.707 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.823236+0000", 2024-08-19T14:17:21.707 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1891000000000002e-05 2024-08-19T14:17:21.707 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.707 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.707 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:17:21.707 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.823365+0000", 2024-08-19T14:17:21.707 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012814900000000001 2024-08-19T14:17:21.708 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.708 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.708 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 3,9", 2024-08-19T14:17:21.708 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.823514+0000", 2024-08-19T14:17:21.708 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014974899999999999 2024-08-19T14:17:21.708 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.708 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.708 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:17:21.708 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.825643+0000", 2024-08-19T14:17:21.708 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0021290580000000001 2024-08-19T14:17:21.708 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.708 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.708 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:17:21.709 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.830104+0000", 2024-08-19T14:17:21.709 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0044611219999999997 2024-08-19T14:17:21.709 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.709 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.709 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:17:21.709 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:43.014346+0000", 2024-08-19T14:17:21.709 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.184241868 2024-08-19T14:17:21.709 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.709 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.709 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:17:21.709 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:43.014408+0000", 2024-08-19T14:17:21.709 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.1571999999999999e-05 2024-08-19T14:17:21.710 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.710 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.710 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:17:21.710 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:43.014452+0000", 2024-08-19T14:17:21.710 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.4394000000000002e-05 2024-08-19T14:17:21.710 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:21.710 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:17:21.710 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:21.710 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.710 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.710 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24580.0:88340 2.a 2:54ce74a4:::benchmark_data_smithi045_46415_object88339:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-08-19T14:17:21.710 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:42.833990+0000", 2024-08-19T14:17:21.710 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 218.86056725700001, 2024-08-19T14:17:21.711 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.18138072099999999, 2024-08-19T14:17:21.711 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:17:21.711 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:17:21.711 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:17:21.712 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:17:21.712 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24580", 2024-08-19T14:17:21.712 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:17:21.712 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 88340 2024-08-19T14:17:21.712 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.712 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:17:21.712 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.713 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:17:21.713 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.833990+0000", 2024-08-19T14:17:21.713 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:21.713 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.713 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.713 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:17:21.713 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.833990+0000", 2024-08-19T14:17:21.713 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:21.713 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.713 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.713 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:17:21.713 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.833992+0000", 2024-08-19T14:17:21.713 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9690000000000001e-06 2024-08-19T14:17:21.714 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.714 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.714 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:17:21.714 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.834015+0000", 2024-08-19T14:17:21.714 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.3815000000000002e-05 2024-08-19T14:17:21.714 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.714 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.714 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:17:21.714 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.834027+0000", 2024-08-19T14:17:21.714 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2058999999999999e-05 2024-08-19T14:17:21.714 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.714 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.714 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:17:21.714 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.834033+0000", 2024-08-19T14:17:21.715 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.9170000000000003e-06 2024-08-19T14:17:21.715 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.715 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.715 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:17:21.715 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.834086+0000", 2024-08-19T14:17:21.715 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2416000000000001e-05 2024-08-19T14:17:21.715 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.715 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.715 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:17:21.715 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.834181+0000", 2024-08-19T14:17:21.715 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.4864999999999994e-05 2024-08-19T14:17:21.715 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.715 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.716 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 3,9", 2024-08-19T14:17:21.716 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.834296+0000", 2024-08-19T14:17:21.716 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00011565600000000001 2024-08-19T14:17:21.716 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.716 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.716 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:17:21.716 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.836351+0000", 2024-08-19T14:17:21.716 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.002054223 2024-08-19T14:17:21.716 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.717 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.717 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:17:21.717 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.838773+0000", 2024-08-19T14:17:21.717 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0024221609999999999 2024-08-19T14:17:21.717 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.717 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.717 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:17:21.717 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:43.015293+0000", 2024-08-19T14:17:21.717 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.17652014899999999 2024-08-19T14:17:21.717 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.717 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.717 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:17:21.717 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:43.015341+0000", 2024-08-19T14:17:21.717 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.8319999999999998e-05 2024-08-19T14:17:21.718 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.718 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.718 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:17:21.718 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:43.015370+0000", 2024-08-19T14:17:21.718 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.9170999999999999e-05 2024-08-19T14:17:21.718 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:21.718 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:17:21.718 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:21.718 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.718 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.718 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24580.0:115112 2.a 2:56fe95bd:::benchmark_data_smithi045_46415_object115111:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-08-19T14:17:21.719 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:14:09.947566+0000", 2024-08-19T14:17:21.719 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 191.746991084, 2024-08-19T14:17:21.719 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.18974571500000001, 2024-08-19T14:17:21.719 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:17:21.719 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:17:21.719 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:17:21.719 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:17:21.719 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24580", 2024-08-19T14:17:21.719 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:17:21.719 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 115112 2024-08-19T14:17:21.719 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.719 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:17:21.719 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.719 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:17:21.720 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.947566+0000", 2024-08-19T14:17:21.720 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:21.720 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.720 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.720 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:17:21.720 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.947566+0000", 2024-08-19T14:17:21.720 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:21.720 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.720 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.720 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:17:21.720 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.947566+0000", 2024-08-19T14:17:21.720 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1900000000000003e-07 2024-08-19T14:17:21.720 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.720 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.721 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:17:21.721 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.947586+0000", 2024-08-19T14:17:21.721 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9678999999999999e-05 2024-08-19T14:17:21.721 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.721 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.721 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:17:21.721 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.947594+0000", 2024-08-19T14:17:21.721 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.3010000000000007e-06 2024-08-19T14:17:21.721 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.721 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.721 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:17:21.721 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.947601+0000", 2024-08-19T14:17:21.721 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.9290000000000001e-06 2024-08-19T14:17:21.721 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.722 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.722 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:17:21.722 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.951111+0000", 2024-08-19T14:17:21.722 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0035093189999999999 2024-08-19T14:17:21.722 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.722 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.722 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:17:21.722 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.951253+0000", 2024-08-19T14:17:21.722 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000142006 2024-08-19T14:17:21.722 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.722 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.722 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 3,9", 2024-08-19T14:17:21.722 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.951383+0000", 2024-08-19T14:17:21.723 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000130023 2024-08-19T14:17:21.723 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.723 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.723 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:17:21.723 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.961965+0000", 2024-08-19T14:17:21.723 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.010582794 2024-08-19T14:17:21.723 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.723 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.723 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:17:21.723 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.961998+0000", 2024-08-19T14:17:21.723 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.2607999999999997e-05 2024-08-19T14:17:21.723 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.723 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.723 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:17:21.723 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:10.137205+0000", 2024-08-19T14:17:21.724 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.17520725100000001 2024-08-19T14:17:21.724 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.724 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.724 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:17:21.724 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:10.137261+0000", 2024-08-19T14:17:21.724 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.5779000000000002e-05 2024-08-19T14:17:21.724 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.724 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.724 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:17:21.724 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:10.137311+0000", 2024-08-19T14:17:21.724 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0507000000000003e-05 2024-08-19T14:17:21.724 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:21.724 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:17:21.724 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:21.725 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.725 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.725 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24580.0:115113 2.a 2:57a3b2e5:::benchmark_data_smithi045_46415_object115112:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-08-19T14:17:21.725 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:14:09.947611+0000", 2024-08-19T14:17:21.725 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 191.746946076, 2024-08-19T14:17:21.725 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.18985379399999999, 2024-08-19T14:17:21.725 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:17:21.725 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:17:21.725 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:17:21.725 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:17:21.725 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24580", 2024-08-19T14:17:21.725 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:17:21.725 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 115113 2024-08-19T14:17:21.725 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.726 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:17:21.726 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.726 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:17:21.726 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.947611+0000", 2024-08-19T14:17:21.726 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:21.726 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.726 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.726 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:17:21.726 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.947611+0000", 2024-08-19T14:17:21.726 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:21.726 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.726 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.726 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:17:21.727 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.947611+0000", 2024-08-19T14:17:21.727 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7199999999999999e-07 2024-08-19T14:17:21.727 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.727 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.727 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:17:21.727 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.947631+0000", 2024-08-19T14:17:21.727 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9986000000000001e-05 2024-08-19T14:17:21.727 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.727 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.727 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:17:21.727 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.947640+0000", 2024-08-19T14:17:21.727 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.6139999999999992e-06 2024-08-19T14:17:21.727 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.728 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.728 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:17:21.728 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.947646+0000", 2024-08-19T14:17:21.728 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.0809999999999999e-06 2024-08-19T14:17:21.728 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.728 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.728 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:17:21.728 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.954599+0000", 2024-08-19T14:17:21.728 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0069529650000000002 2024-08-19T14:17:21.728 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.728 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.728 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:17:21.728 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.954718+0000", 2024-08-19T14:17:21.728 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000119492 2024-08-19T14:17:21.729 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.729 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.729 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 3,9", 2024-08-19T14:17:21.729 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.954856+0000", 2024-08-19T14:17:21.729 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013726600000000001 2024-08-19T14:17:21.729 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.729 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.729 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:17:21.729 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.962009+0000", 2024-08-19T14:17:21.729 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0071530150000000004 2024-08-19T14:17:21.729 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.729 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.729 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:17:21.730 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.962036+0000", 2024-08-19T14:17:21.730 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.7336999999999999e-05 2024-08-19T14:17:21.730 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.730 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.730 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:17:21.730 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:10.137351+0000", 2024-08-19T14:17:21.730 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.17531533699999999 2024-08-19T14:17:21.730 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.730 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.730 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:17:21.730 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:10.137419+0000", 2024-08-19T14:17:21.730 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.7792000000000004e-05 2024-08-19T14:17:21.730 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.730 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.731 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:17:21.731 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:10.137465+0000", 2024-08-19T14:17:21.731 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.5337000000000003e-05 2024-08-19T14:17:21.731 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:21.731 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:17:21.731 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:21.731 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.731 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.731 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24580.0:115133 2.a 2:5974b323:::benchmark_data_smithi045_46415_object115132:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-08-19T14:17:21.731 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:14:09.968449+0000", 2024-08-19T14:17:21.731 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 191.72610759400001, 2024-08-19T14:17:21.731 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.16914753099999999, 2024-08-19T14:17:21.731 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:17:21.731 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:17:21.732 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:17:21.732 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:17:21.732 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24580", 2024-08-19T14:17:21.732 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:17:21.732 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 115133 2024-08-19T14:17:21.732 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.732 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:17:21.732 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.732 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:17:21.732 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.968449+0000", 2024-08-19T14:17:21.732 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:21.732 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.732 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.732 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:17:21.733 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.968449+0000", 2024-08-19T14:17:21.733 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:21.733 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.733 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.733 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:17:21.733 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.968451+0000", 2024-08-19T14:17:21.733 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.446e-06 2024-08-19T14:17:21.733 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.733 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.733 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:17:21.733 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.968471+0000", 2024-08-19T14:17:21.733 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9967999999999999e-05 2024-08-19T14:17:21.733 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.733 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.734 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:17:21.734 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.968483+0000", 2024-08-19T14:17:21.734 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1997999999999999e-05 2024-08-19T14:17:21.734 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.734 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.734 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:17:21.734 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.968490+0000", 2024-08-19T14:17:21.734 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.1189999999999999e-06 2024-08-19T14:17:21.734 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.734 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.734 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:17:21.734 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.968547+0000", 2024-08-19T14:17:21.735 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.6997000000000003e-05 2024-08-19T14:17:21.735 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.735 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.735 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:17:21.735 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.968661+0000", 2024-08-19T14:17:21.735 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00011401299999999999 2024-08-19T14:17:21.735 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.735 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.735 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 3,9", 2024-08-19T14:17:21.735 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.968797+0000", 2024-08-19T14:17:21.735 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013640599999999999 2024-08-19T14:17:21.735 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.735 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.735 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:17:21.736 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:10.117661+0000", 2024-08-19T14:17:21.736 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.14886405699999999 2024-08-19T14:17:21.736 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.736 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.736 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:17:21.736 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:10.117674+0000", 2024-08-19T14:17:21.736 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2323e-05 2024-08-19T14:17:21.736 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.736 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.736 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:17:21.736 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:10.137506+0000", 2024-08-19T14:17:21.736 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.019832809 2024-08-19T14:17:21.736 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.736 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.737 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:17:21.737 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:10.137555+0000", 2024-08-19T14:17:21.737 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.8965999999999998e-05 2024-08-19T14:17:21.737 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.737 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.737 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:17:21.737 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:10.137597+0000", 2024-08-19T14:17:21.737 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.1428999999999999e-05 2024-08-19T14:17:21.737 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:21.737 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:17:21.737 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:21.737 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.737 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.737 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24580.0:159035 2.c 2:32456cd8:::benchmark_data_smithi045_46415_object35754:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e167)", 2024-08-19T14:17:21.738 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:14:59.419140+0000", 2024-08-19T14:17:21.738 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 142.275416871, 2024-08-19T14:17:21.738 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.21467188200000001, 2024-08-19T14:17:21.738 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:17:21.738 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:17:21.738 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:17:21.738 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:17:21.738 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24580", 2024-08-19T14:17:21.738 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:17:21.738 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 159035 2024-08-19T14:17:21.738 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.738 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:17:21.738 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.739 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:17:21.739 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.419140+0000", 2024-08-19T14:17:21.739 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:21.739 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.739 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.739 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:17:21.739 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.419140+0000", 2024-08-19T14:17:21.739 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:21.739 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.739 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.739 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:17:21.739 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.419141+0000", 2024-08-19T14:17:21.739 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4079999999999999e-06 2024-08-19T14:17:21.739 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.739 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.740 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:17:21.740 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.419146+0000", 2024-08-19T14:17:21.740 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0610000000000003e-06 2024-08-19T14:17:21.740 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.740 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.740 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:17:21.740 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.419163+0000", 2024-08-19T14:17:21.740 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6770999999999999e-05 2024-08-19T14:17:21.740 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.740 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.740 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:17:21.740 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.419172+0000", 2024-08-19T14:17:21.740 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.276e-06 2024-08-19T14:17:21.741 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.741 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.741 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:17:21.741 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.419223+0000", 2024-08-19T14:17:21.741 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1588999999999997e-05 2024-08-19T14:17:21.741 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.741 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.741 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:17:21.741 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.419320+0000", 2024-08-19T14:17:21.741 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.7237000000000004e-05 2024-08-19T14:17:21.741 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.741 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.741 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,1", 2024-08-19T14:17:21.741 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.419386+0000", 2024-08-19T14:17:21.742 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.6093999999999994e-05 2024-08-19T14:17:21.742 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.742 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.742 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:17:21.742 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.422728+0000", 2024-08-19T14:17:21.742 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0033416489999999999 2024-08-19T14:17:21.742 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.742 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.742 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:17:21.742 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.439269+0000", 2024-08-19T14:17:21.742 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.016540764999999999 2024-08-19T14:17:21.742 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.742 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.743 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:17:21.743 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.633710+0000", 2024-08-19T14:17:21.743 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.194440742 2024-08-19T14:17:21.743 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.743 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.743 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:17:21.743 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.633767+0000", 2024-08-19T14:17:21.743 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7210999999999998e-05 2024-08-19T14:17:21.743 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.743 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.743 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:17:21.743 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.633812+0000", 2024-08-19T14:17:21.743 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.5079e-05 2024-08-19T14:17:21.743 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:21.744 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:17:21.744 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:21.744 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.744 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.744 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24580.0:159041 2.c 2:326ec62a:::benchmark_data_smithi045_46415_object35760:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e167)", 2024-08-19T14:17:21.744 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:14:59.424792+0000", 2024-08-19T14:17:21.744 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 142.26976436699999, 2024-08-19T14:17:21.744 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.25135169800000001, 2024-08-19T14:17:21.744 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:17:21.744 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:17:21.744 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:17:21.744 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:17:21.744 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24580", 2024-08-19T14:17:21.745 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:17:21.745 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 159041 2024-08-19T14:17:21.745 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.745 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:17:21.745 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.745 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:17:21.745 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.424792+0000", 2024-08-19T14:17:21.745 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:21.745 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.745 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.745 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:17:21.745 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.424792+0000", 2024-08-19T14:17:21.745 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:21.746 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.746 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.746 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:17:21.746 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.424794+0000", 2024-08-19T14:17:21.746 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.883e-06 2024-08-19T14:17:21.746 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.746 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.746 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:17:21.746 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.424800+0000", 2024-08-19T14:17:21.746 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4349999999999996e-06 2024-08-19T14:17:21.746 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.746 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.746 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:17:21.746 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.424813+0000", 2024-08-19T14:17:21.747 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3619e-05 2024-08-19T14:17:21.747 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.747 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.747 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:17:21.747 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.424821+0000", 2024-08-19T14:17:21.747 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4649999999999997e-06 2024-08-19T14:17:21.747 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.747 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.747 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:17:21.747 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.424864+0000", 2024-08-19T14:17:21.747 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.2786e-05 2024-08-19T14:17:21.747 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.747 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.747 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:17:21.748 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.424944+0000", 2024-08-19T14:17:21.748 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.0628000000000001e-05 2024-08-19T14:17:21.748 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.748 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.748 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,1", 2024-08-19T14:17:21.748 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.425008+0000", 2024-08-19T14:17:21.748 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.3223000000000001e-05 2024-08-19T14:17:21.748 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.748 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.748 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:17:21.748 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.429480+0000", 2024-08-19T14:17:21.748 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0044724830000000002 2024-08-19T14:17:21.748 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.749 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.749 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:17:21.749 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.452535+0000", 2024-08-19T14:17:21.749 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.023055386000000001 2024-08-19T14:17:21.749 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.749 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.749 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:17:21.749 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.676047+0000", 2024-08-19T14:17:21.749 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.22351151399999999 2024-08-19T14:17:21.749 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.749 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.749 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:17:21.749 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.676101+0000", 2024-08-19T14:17:21.749 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4237000000000003e-05 2024-08-19T14:17:21.750 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.750 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.750 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:17:21.750 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.676144+0000", 2024-08-19T14:17:21.750 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.3038999999999999e-05 2024-08-19T14:17:21.750 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:21.750 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:17:21.750 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:21.750 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.750 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.750 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24580.0:159045 2.c 2:30c9429c:::benchmark_data_smithi045_46415_object35764:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e167)", 2024-08-19T14:17:21.750 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:14:59.429858+0000", 2024-08-19T14:17:21.750 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 142.26469841400001, 2024-08-19T14:17:21.750 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.24643119899999999, 2024-08-19T14:17:21.751 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:17:21.751 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:17:21.751 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:17:21.751 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:17:21.751 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24580", 2024-08-19T14:17:21.751 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:17:21.751 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 159045 2024-08-19T14:17:21.751 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.751 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:17:21.751 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.751 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:17:21.751 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.429858+0000", 2024-08-19T14:17:21.751 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:21.751 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.752 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.752 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:17:21.752 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.429858+0000", 2024-08-19T14:17:21.752 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:21.752 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.752 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.752 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:17:21.752 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.429860+0000", 2024-08-19T14:17:21.752 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.531e-06 2024-08-19T14:17:21.752 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.752 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.752 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:17:21.752 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.429865+0000", 2024-08-19T14:17:21.752 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0989999999999998e-06 2024-08-19T14:17:21.753 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.753 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.753 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:17:21.753 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.429875+0000", 2024-08-19T14:17:21.753 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0043e-05 2024-08-19T14:17:21.753 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.753 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.753 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:17:21.753 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.429891+0000", 2024-08-19T14:17:21.753 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5539000000000001e-05 2024-08-19T14:17:21.753 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.753 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.753 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:17:21.753 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.429951+0000", 2024-08-19T14:17:21.754 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.0528000000000001e-05 2024-08-19T14:17:21.754 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.754 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.754 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:17:21.754 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.430061+0000", 2024-08-19T14:17:21.754 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000110203 2024-08-19T14:17:21.754 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.754 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.754 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,1", 2024-08-19T14:17:21.754 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.430129+0000", 2024-08-19T14:17:21.754 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.7970000000000001e-05 2024-08-19T14:17:21.754 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.754 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.755 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:17:21.755 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.435692+0000", 2024-08-19T14:17:21.755 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0055628550000000002 2024-08-19T14:17:21.755 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.755 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.755 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:17:21.755 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.459086+0000", 2024-08-19T14:17:21.755 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.023393942000000001 2024-08-19T14:17:21.755 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.755 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.755 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:17:21.755 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.676201+0000", 2024-08-19T14:17:21.755 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.21711492499999999 2024-08-19T14:17:21.755 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.756 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.756 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:17:21.756 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.676252+0000", 2024-08-19T14:17:21.756 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1334e-05 2024-08-19T14:17:21.756 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.756 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.756 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:17:21.756 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.676290+0000", 2024-08-19T14:17:21.756 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.7230000000000001e-05 2024-08-19T14:17:21.756 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:21.756 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:17:21.756 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:21.756 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.756 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.757 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24580.0:159089 2.c 2:36327fd7:::benchmark_data_smithi045_46415_object35808:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e167)", 2024-08-19T14:17:21.757 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:14:59.486789+0000", 2024-08-19T14:17:21.757 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 142.20776795899999, 2024-08-19T14:17:21.757 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.18962784899999999, 2024-08-19T14:17:21.757 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:17:21.757 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:17:21.758 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:17:21.758 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:17:21.758 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24580", 2024-08-19T14:17:21.758 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:17:21.759 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 159089 2024-08-19T14:17:21.759 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.759 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:17:21.759 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.759 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:17:21.759 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.486789+0000", 2024-08-19T14:17:21.759 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:21.759 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.759 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.759 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:17:21.759 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.486789+0000", 2024-08-19T14:17:21.759 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:21.759 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.760 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.760 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:17:21.760 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.486790+0000", 2024-08-19T14:17:21.760 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.578e-06 2024-08-19T14:17:21.760 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.760 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.760 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:17:21.760 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.486795+0000", 2024-08-19T14:17:21.760 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.6020000000000002e-06 2024-08-19T14:17:21.760 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.760 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.760 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:17:21.760 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.486808+0000", 2024-08-19T14:17:21.761 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2544999999999999e-05 2024-08-19T14:17:21.761 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.761 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.761 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:17:21.761 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.486815+0000", 2024-08-19T14:17:21.761 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4499999999999998e-06 2024-08-19T14:17:21.761 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.761 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.761 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:17:21.761 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.486883+0000", 2024-08-19T14:17:21.761 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.7889e-05 2024-08-19T14:17:21.761 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.761 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.761 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:17:21.762 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.487009+0000", 2024-08-19T14:17:21.762 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012568099999999999 2024-08-19T14:17:21.762 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.762 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.762 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,1", 2024-08-19T14:17:21.762 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.487121+0000", 2024-08-19T14:17:21.762 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00011189800000000001 2024-08-19T14:17:21.762 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.762 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.762 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:17:21.762 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.491681+0000", 2024-08-19T14:17:21.762 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0045600909999999996 2024-08-19T14:17:21.762 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.763 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.763 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:17:21.763 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.494160+0000", 2024-08-19T14:17:21.763 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0024789030000000002 2024-08-19T14:17:21.763 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.763 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.763 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:17:21.763 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.676345+0000", 2024-08-19T14:17:21.763 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.18218579600000001 2024-08-19T14:17:21.763 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.763 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.763 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:17:21.763 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.676383+0000", 2024-08-19T14:17:21.764 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.8105000000000002e-05 2024-08-19T14:17:21.764 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.764 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.764 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:17:21.764 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.676417+0000", 2024-08-19T14:17:21.764 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.3311e-05 2024-08-19T14:17:21.764 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:21.764 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:17:21.764 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:21.764 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.764 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.764 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24580.0:159109 2.c 2:34b94e13:::benchmark_data_smithi045_46415_object35828:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e167)", 2024-08-19T14:17:21.764 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:14:59.507153+0000", 2024-08-19T14:17:21.764 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 142.18740425499999, 2024-08-19T14:17:21.765 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.16938667499999999, 2024-08-19T14:17:21.765 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:17:21.765 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:17:21.765 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:17:21.765 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:17:21.765 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24580", 2024-08-19T14:17:21.765 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:17:21.765 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 159109 2024-08-19T14:17:21.765 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.765 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:17:21.765 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.765 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:17:21.765 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.507153+0000", 2024-08-19T14:17:21.765 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:21.766 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.766 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.766 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:17:21.766 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.507153+0000", 2024-08-19T14:17:21.766 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:21.766 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.766 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.766 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:17:21.766 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.507154+0000", 2024-08-19T14:17:21.766 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5030000000000001e-06 2024-08-19T14:17:21.766 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.766 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.766 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:17:21.767 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.507159+0000", 2024-08-19T14:17:21.767 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.3430000000000003e-06 2024-08-19T14:17:21.767 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.767 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.767 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:17:21.767 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.507173+0000", 2024-08-19T14:17:21.767 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3618000000000001e-05 2024-08-19T14:17:21.767 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.767 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.767 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:17:21.767 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.507180+0000", 2024-08-19T14:17:21.767 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.1810000000000004e-06 2024-08-19T14:17:21.767 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.767 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.768 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:17:21.768 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.507239+0000", 2024-08-19T14:17:21.768 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.8919999999999998e-05 2024-08-19T14:17:21.768 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.768 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.768 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:17:21.768 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.507351+0000", 2024-08-19T14:17:21.768 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000111777 2024-08-19T14:17:21.768 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.768 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.768 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,1", 2024-08-19T14:17:21.768 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.507460+0000", 2024-08-19T14:17:21.768 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00010922800000000001 2024-08-19T14:17:21.768 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.769 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.769 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:17:21.769 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.517158+0000", 2024-08-19T14:17:21.769 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0096979530000000005 2024-08-19T14:17:21.769 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.769 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.769 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:17:21.769 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.517190+0000", 2024-08-19T14:17:21.769 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.1404e-05 2024-08-19T14:17:21.769 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.769 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.769 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:17:21.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:21.772 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.676467+0000", 2024-08-19T14:17:21.772 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.15927707899999999 2024-08-19T14:17:21.772 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.773 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.773 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:17:21.773 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.676513+0000", 2024-08-19T14:17:21.773 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.6678999999999998e-05 2024-08-19T14:17:21.773 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.774 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.774 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:17:21.774 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.676539+0000", 2024-08-19T14:17:21.774 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.599e-05 2024-08-19T14:17:21.774 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:21.774 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:17:21.774 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:21.774 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.774 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.774 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24580.0:159114 2.c 2:3d95e498:::benchmark_data_smithi045_46415_object35833:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e167)", 2024-08-19T14:17:21.774 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:14:59.512914+0000", 2024-08-19T14:17:21.774 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 142.18164305799999, 2024-08-19T14:17:21.774 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.172700303, 2024-08-19T14:17:21.775 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:17:21.775 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:17:21.775 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:17:21.775 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:17:21.775 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24580", 2024-08-19T14:17:21.775 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:17:21.775 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 159114 2024-08-19T14:17:21.775 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.775 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:17:21.775 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.775 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:17:21.775 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.512914+0000", 2024-08-19T14:17:21.775 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:21.776 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.776 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.776 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:17:21.776 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.512914+0000", 2024-08-19T14:17:21.776 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:21.776 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.776 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.776 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:17:21.776 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.512915+0000", 2024-08-19T14:17:21.776 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5719999999999999e-06 2024-08-19T14:17:21.776 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.776 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.776 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:17:21.776 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.512921+0000", 2024-08-19T14:17:21.777 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4670000000000003e-06 2024-08-19T14:17:21.777 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.777 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.777 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:17:21.777 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.512937+0000", 2024-08-19T14:17:21.777 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6413999999999999e-05 2024-08-19T14:17:21.777 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.777 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.777 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:17:21.777 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.512946+0000", 2024-08-19T14:17:21.777 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.4910000000000005e-06 2024-08-19T14:17:21.777 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.777 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.777 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:17:21.778 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.513003+0000", 2024-08-19T14:17:21.778 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7148000000000002e-05 2024-08-19T14:17:21.778 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.778 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.778 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:17:21.778 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.513124+0000", 2024-08-19T14:17:21.778 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012062599999999999 2024-08-19T14:17:21.778 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.778 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.778 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,1", 2024-08-19T14:17:21.778 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.513248+0000", 2024-08-19T14:17:21.778 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012398699999999999 2024-08-19T14:17:21.778 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.779 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.779 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:17:21.779 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.517252+0000", 2024-08-19T14:17:21.779 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0040042929999999999 2024-08-19T14:17:21.779 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.779 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.779 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:17:21.779 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.522861+0000", 2024-08-19T14:17:21.779 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0056093050000000002 2024-08-19T14:17:21.779 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.779 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.779 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:17:21.779 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.685394+0000", 2024-08-19T14:17:21.779 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.16253331400000001 2024-08-19T14:17:21.780 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.780 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.780 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:17:21.780 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.685564+0000", 2024-08-19T14:17:21.780 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000169572 2024-08-19T14:17:21.780 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.780 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.780 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:17:21.780 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.685614+0000", 2024-08-19T14:17:21.780 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0114000000000002e-05 2024-08-19T14:17:21.780 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:21.780 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:17:21.780 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:21.781 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.781 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.781 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24580.0:159121 2.c 2:308dfb60:::benchmark_data_smithi045_46415_object35840:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e167)", 2024-08-19T14:17:21.781 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:14:59.518143+0000", 2024-08-19T14:17:21.781 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 142.17641381499999, 2024-08-19T14:17:21.781 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.16761131500000001, 2024-08-19T14:17:21.781 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:17:21.781 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:17:21.781 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:17:21.781 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:17:21.781 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24580", 2024-08-19T14:17:21.781 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:17:21.781 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 159121 2024-08-19T14:17:21.781 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.782 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:17:21.782 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.782 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:17:21.782 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.518143+0000", 2024-08-19T14:17:21.782 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:21.782 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.782 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.782 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:17:21.782 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.518143+0000", 2024-08-19T14:17:21.782 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:21.782 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.782 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.782 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:17:21.783 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.518144+0000", 2024-08-19T14:17:21.783 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.42e-06 2024-08-19T14:17:21.783 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.783 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.783 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:17:21.783 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.518149+0000", 2024-08-19T14:17:21.783 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0119999999999996e-06 2024-08-19T14:17:21.783 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.783 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.783 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:17:21.783 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.518163+0000", 2024-08-19T14:17:21.783 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3951000000000001e-05 2024-08-19T14:17:21.783 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.783 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.784 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:17:21.784 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.518171+0000", 2024-08-19T14:17:21.784 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.1389999999999998e-06 2024-08-19T14:17:21.784 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.784 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.784 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:17:21.784 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.518229+0000", 2024-08-19T14:17:21.784 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.8878000000000001e-05 2024-08-19T14:17:21.784 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.784 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.784 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:17:21.784 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.518352+0000", 2024-08-19T14:17:21.784 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000122268 2024-08-19T14:17:21.784 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.785 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.785 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,1", 2024-08-19T14:17:21.785 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.518459+0000", 2024-08-19T14:17:21.785 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000107636 2024-08-19T14:17:21.785 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.785 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.785 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:17:21.785 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.522946+0000", 2024-08-19T14:17:21.785 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0044869970000000004 2024-08-19T14:17:21.785 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.785 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.785 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:17:21.785 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.524276+0000", 2024-08-19T14:17:21.786 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0013293389999999999 2024-08-19T14:17:21.786 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.786 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.786 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:17:21.786 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.685658+0000", 2024-08-19T14:17:21.786 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.16138230000000001 2024-08-19T14:17:21.786 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.786 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.786 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:17:21.786 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.685713+0000", 2024-08-19T14:17:21.786 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.5451000000000002e-05 2024-08-19T14:17:21.786 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.786 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.786 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:17:21.787 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.685754+0000", 2024-08-19T14:17:21.787 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.0924000000000003e-05 2024-08-19T14:17:21.787 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:21.787 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:17:21.787 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:21.787 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.787 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.787 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25057.0:10176 3.e 3:7b716bad:::benchmark_data_smithi045_59973_object10175:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-08-19T14:17:21.787 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:17:04.641386+0000", 2024-08-19T14:17:21.787 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 17.053170806000001, 2024-08-19T14:17:21.787 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.203336451, 2024-08-19T14:17:21.787 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:17:21.787 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:17:21.788 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:17:21.788 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:17:21.788 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25057", 2024-08-19T14:17:21.788 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:17:21.788 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 10176 2024-08-19T14:17:21.788 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.788 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:17:21.788 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.788 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:17:21.788 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.641386+0000", 2024-08-19T14:17:21.788 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:21.788 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.788 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.788 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:17:21.789 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.641386+0000", 2024-08-19T14:17:21.789 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:21.789 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.789 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.789 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:17:21.789 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.641387+0000", 2024-08-19T14:17:21.789 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.6699999999999999e-07 2024-08-19T14:17:21.789 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.789 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.789 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:17:21.789 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.641402+0000", 2024-08-19T14:17:21.789 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5415999999999999e-05 2024-08-19T14:17:21.789 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.790 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.790 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:17:21.790 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.641407+0000", 2024-08-19T14:17:21.790 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1569999999999999e-06 2024-08-19T14:17:21.790 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.790 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.790 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:17:21.790 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.641411+0000", 2024-08-19T14:17:21.790 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.427e-06 2024-08-19T14:17:21.790 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.790 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.790 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:17:21.790 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.641463+0000", 2024-08-19T14:17:21.790 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2333999999999997e-05 2024-08-19T14:17:21.791 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.791 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.791 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:17:21.791 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.641829+0000", 2024-08-19T14:17:21.791 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00036651300000000001 2024-08-19T14:17:21.791 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.791 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.791 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,4", 2024-08-19T14:17:21.791 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.641975+0000", 2024-08-19T14:17:21.791 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000145647 2024-08-19T14:17:21.791 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.791 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.791 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:17:21.791 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.644044+0000", 2024-08-19T14:17:21.792 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0020694089999999999 2024-08-19T14:17:21.792 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.792 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.792 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:17:21.792 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.644092+0000", 2024-08-19T14:17:21.792 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.7811e-05 2024-08-19T14:17:21.792 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.792 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.792 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:17:21.792 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.844651+0000", 2024-08-19T14:17:21.792 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20055835 2024-08-19T14:17:21.792 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.792 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.793 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:17:21.793 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.844691+0000", 2024-08-19T14:17:21.793 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.0692999999999998e-05 2024-08-19T14:17:21.793 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.793 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.793 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:17:21.793 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.844723+0000", 2024-08-19T14:17:21.793 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.1226999999999998e-05 2024-08-19T14:17:21.793 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:21.793 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:17:21.793 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:21.793 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.793 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.793 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25057.0:10177 3.e 3:7db1ed9f:::benchmark_data_smithi045_59973_object10176:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-08-19T14:17:21.794 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:17:04.641421+0000", 2024-08-19T14:17:21.794 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 17.053136277, 2024-08-19T14:17:21.794 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20339621299999999, 2024-08-19T14:17:21.794 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:17:21.794 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:17:21.794 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:17:21.794 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:17:21.794 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25057", 2024-08-19T14:17:21.794 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:17:21.794 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 10177 2024-08-19T14:17:21.794 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.794 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:17:21.794 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.795 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:17:21.795 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.641421+0000", 2024-08-19T14:17:21.795 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:21.795 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.795 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.795 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:17:21.795 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.641421+0000", 2024-08-19T14:17:21.795 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:21.795 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.795 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.795 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:17:21.795 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.641421+0000", 2024-08-19T14:17:21.795 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.5999999999999999e-07 2024-08-19T14:17:21.795 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.796 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.796 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:17:21.796 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.641441+0000", 2024-08-19T14:17:21.796 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0414000000000001e-05 2024-08-19T14:17:21.796 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.796 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.796 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:17:21.796 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.641449+0000", 2024-08-19T14:17:21.796 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.2490000000000004e-06 2024-08-19T14:17:21.796 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.796 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.796 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:17:21.796 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.641454+0000", 2024-08-19T14:17:21.796 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.3059999999999997e-06 2024-08-19T14:17:21.797 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.797 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.797 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:17:21.797 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.642934+0000", 2024-08-19T14:17:21.797 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0014798719999999999 2024-08-19T14:17:21.797 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.797 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.797 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:17:21.797 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.643012+0000", 2024-08-19T14:17:21.797 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.8163000000000004e-05 2024-08-19T14:17:21.797 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.797 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.797 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,4", 2024-08-19T14:17:21.797 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.643577+0000", 2024-08-19T14:17:21.798 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00056543800000000003 2024-08-19T14:17:21.798 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.798 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.798 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:17:21.798 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.644183+0000", 2024-08-19T14:17:21.798 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00060506100000000005 2024-08-19T14:17:21.798 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.798 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.798 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:17:21.798 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.647907+0000", 2024-08-19T14:17:21.798 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.003724336 2024-08-19T14:17:21.798 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.798 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.798 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:17:21.799 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.844754+0000", 2024-08-19T14:17:21.799 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.19684715799999999 2024-08-19T14:17:21.799 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.799 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.799 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:17:21.799 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.844783+0000", 2024-08-19T14:17:21.799 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.9090000000000001e-05 2024-08-19T14:17:21.799 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.799 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.799 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:17:21.799 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.844817+0000", 2024-08-19T14:17:21.799 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.3665999999999999e-05 2024-08-19T14:17:21.799 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:21.800 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:17:21.800 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:21.800 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.800 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.800 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25057.0:10198 3.e 3:78135d2f:::benchmark_data_smithi045_59973_object10197:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-08-19T14:17:21.800 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:17:04.646752+0000", 2024-08-19T14:17:21.800 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 17.047804785, 2024-08-19T14:17:21.800 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.19900073800000001, 2024-08-19T14:17:21.800 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:17:21.800 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:17:21.800 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:17:21.800 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:17:21.800 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25057", 2024-08-19T14:17:21.801 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:17:21.801 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 10198 2024-08-19T14:17:21.801 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.801 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:17:21.801 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.801 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:17:21.801 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.646752+0000", 2024-08-19T14:17:21.801 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:21.801 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.801 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.801 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:17:21.801 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.646752+0000", 2024-08-19T14:17:21.801 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:21.801 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.802 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.802 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:17:21.802 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.646753+0000", 2024-08-19T14:17:21.802 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.540000000000001e-07 2024-08-19T14:17:21.802 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.802 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.802 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:17:21.802 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.646775+0000", 2024-08-19T14:17:21.802 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1586999999999998e-05 2024-08-19T14:17:21.802 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.802 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.802 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:17:21.802 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.646785+0000", 2024-08-19T14:17:21.803 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0679999999999999e-05 2024-08-19T14:17:21.803 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.803 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.803 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:17:21.803 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.646793+0000", 2024-08-19T14:17:21.803 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.6389999999999992e-06 2024-08-19T14:17:21.803 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.803 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.803 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:17:21.803 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.646857+0000", 2024-08-19T14:17:21.803 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.4139000000000004e-05 2024-08-19T14:17:21.803 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.804 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.804 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:17:21.804 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.646966+0000", 2024-08-19T14:17:21.804 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000109355 2024-08-19T14:17:21.804 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.804 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.804 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,4", 2024-08-19T14:17:21.804 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.647193+0000", 2024-08-19T14:17:21.804 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000226851 2024-08-19T14:17:21.804 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.804 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.804 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:17:21.804 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.649451+0000", 2024-08-19T14:17:21.804 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0022581950000000002 2024-08-19T14:17:21.805 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.805 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.805 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:17:21.805 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.649463+0000", 2024-08-19T14:17:21.805 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1749999999999999e-05 2024-08-19T14:17:21.805 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.805 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.805 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:17:21.805 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.845696+0000", 2024-08-19T14:17:21.805 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.19623284699999999 2024-08-19T14:17:21.805 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.805 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.805 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:17:21.806 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.845728+0000", 2024-08-19T14:17:21.806 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.2291e-05 2024-08-19T14:17:21.806 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.806 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.806 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:17:21.806 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.845753+0000", 2024-08-19T14:17:21.806 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.4450000000000001e-05 2024-08-19T14:17:21.806 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:21.806 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:17:21.806 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:21.806 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.806 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.806 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25057.0:10219 3.e 3:703e1b54:::benchmark_data_smithi045_59973_object10218:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-08-19T14:17:21.807 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:17:04.652012+0000", 2024-08-19T14:17:21.807 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 17.042544877000001, 2024-08-19T14:17:21.807 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.19383710500000001, 2024-08-19T14:17:21.807 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:17:21.807 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:17:21.807 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:17:21.807 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:17:21.807 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25057", 2024-08-19T14:17:21.807 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:17:21.807 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 10219 2024-08-19T14:17:21.807 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.807 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:17:21.807 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.807 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:17:21.808 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.652012+0000", 2024-08-19T14:17:21.808 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:21.808 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.808 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.808 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:17:21.808 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.652012+0000", 2024-08-19T14:17:21.808 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:21.808 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.808 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.808 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:17:21.808 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.652013+0000", 2024-08-19T14:17:21.808 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.218e-06 2024-08-19T14:17:21.808 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.809 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.809 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:17:21.809 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.652032+0000", 2024-08-19T14:17:21.809 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8740999999999999e-05 2024-08-19T14:17:21.809 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.809 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.809 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:17:21.809 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.652042+0000", 2024-08-19T14:17:21.809 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0294e-05 2024-08-19T14:17:21.809 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.809 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.809 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:17:21.809 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.652169+0000", 2024-08-19T14:17:21.809 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000127169 2024-08-19T14:17:21.810 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.810 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.810 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:17:21.810 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.652253+0000", 2024-08-19T14:17:21.810 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.3961000000000005e-05 2024-08-19T14:17:21.810 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.810 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.810 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:17:21.810 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.652362+0000", 2024-08-19T14:17:21.810 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000108378 2024-08-19T14:17:21.810 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.810 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.810 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,4", 2024-08-19T14:17:21.811 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.652501+0000", 2024-08-19T14:17:21.811 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013897700000000001 2024-08-19T14:17:21.811 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.811 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.811 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:17:21.811 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.653164+0000", 2024-08-19T14:17:21.811 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00066358900000000002 2024-08-19T14:17:21.811 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.811 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.811 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:17:21.811 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.653648+0000", 2024-08-19T14:17:21.811 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00048354700000000002 2024-08-19T14:17:21.811 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.811 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.812 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:17:21.812 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.845793+0000", 2024-08-19T14:17:21.812 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.19214476799999999 2024-08-19T14:17:21.812 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.812 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.812 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:17:21.812 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.845827+0000", 2024-08-19T14:17:21.812 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.4758999999999997e-05 2024-08-19T14:17:21.812 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.812 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.812 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:17:21.812 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.845849+0000", 2024-08-19T14:17:21.812 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1704000000000001e-05 2024-08-19T14:17:21.813 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:21.813 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:17:21.813 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:21.813 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.813 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.813 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25057.0:10233 3.e 3:72ef98e3:::benchmark_data_smithi045_59973_object10232:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-08-19T14:17:21.813 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:17:04.655729+0000", 2024-08-19T14:17:21.813 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 17.038828144, 2024-08-19T14:17:21.813 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.19024387100000001, 2024-08-19T14:17:21.813 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:17:21.813 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:17:21.813 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:17:21.813 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:17:21.813 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25057", 2024-08-19T14:17:21.814 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:17:21.814 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 10233 2024-08-19T14:17:21.814 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.814 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:17:21.814 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.814 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:17:21.814 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.655729+0000", 2024-08-19T14:17:21.814 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:21.814 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.814 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.814 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:17:21.814 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.655729+0000", 2024-08-19T14:17:21.814 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:21.815 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.815 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.815 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:17:21.815 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.655729+0000", 2024-08-19T14:17:21.815 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.5000000000000002e-07 2024-08-19T14:17:21.815 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.815 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.815 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:17:21.815 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.655754+0000", 2024-08-19T14:17:21.815 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.4796000000000001e-05 2024-08-19T14:17:21.815 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.815 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.815 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:17:21.816 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.655765+0000", 2024-08-19T14:17:21.816 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.048e-05 2024-08-19T14:17:21.816 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.816 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.816 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:17:21.816 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.655772+0000", 2024-08-19T14:17:21.816 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.0430000000000002e-06 2024-08-19T14:17:21.816 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.816 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.816 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:17:21.816 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.655822+0000", 2024-08-19T14:17:21.816 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0228000000000001e-05 2024-08-19T14:17:21.816 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.817 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.817 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:17:21.817 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.655994+0000", 2024-08-19T14:17:21.817 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00017224499999999999 2024-08-19T14:17:21.817 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.817 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.817 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,4", 2024-08-19T14:17:21.817 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.656766+0000", 2024-08-19T14:17:21.817 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00077197400000000003 2024-08-19T14:17:21.817 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.817 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.817 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:17:21.817 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.657537+0000", 2024-08-19T14:17:21.817 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00077106699999999995 2024-08-19T14:17:21.818 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.818 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.818 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:17:21.818 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.658336+0000", 2024-08-19T14:17:21.818 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00079911200000000004 2024-08-19T14:17:21.818 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.818 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.818 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:17:21.818 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.845900+0000", 2024-08-19T14:17:21.818 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.187563549 2024-08-19T14:17:21.818 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.818 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.818 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:17:21.819 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.845946+0000", 2024-08-19T14:17:21.819 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.6529000000000001e-05 2024-08-19T14:17:21.819 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.819 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.819 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:17:21.819 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.845973+0000", 2024-08-19T14:17:21.819 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.6098e-05 2024-08-19T14:17:21.819 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:21.819 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:17:21.819 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:21.819 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.819 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.819 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25057.0:10236 3.e 3:73c8d5b3:::benchmark_data_smithi045_59973_object10235:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-08-19T14:17:21.820 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:17:04.657575+0000", 2024-08-19T14:17:21.820 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 17.036981872999998, 2024-08-19T14:17:21.820 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.18886002299999999, 2024-08-19T14:17:21.820 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:17:21.820 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:17:21.820 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:17:21.820 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:17:21.820 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25057", 2024-08-19T14:17:21.820 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:17:21.820 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 10236 2024-08-19T14:17:21.820 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.820 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:17:21.820 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.820 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:17:21.821 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.657575+0000", 2024-08-19T14:17:21.821 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:21.821 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.821 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.821 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:17:21.821 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.657575+0000", 2024-08-19T14:17:21.821 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:21.821 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.821 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.821 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:17:21.821 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.657576+0000", 2024-08-19T14:17:21.821 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1340000000000001e-06 2024-08-19T14:17:21.821 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.822 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.822 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:17:21.822 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.657610+0000", 2024-08-19T14:17:21.822 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.3584999999999998e-05 2024-08-19T14:17:21.822 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.822 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.822 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:17:21.822 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.657624+0000", 2024-08-19T14:17:21.822 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4528e-05 2024-08-19T14:17:21.822 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.822 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.822 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:17:21.822 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.657632+0000", 2024-08-19T14:17:21.823 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.8620000000000003e-06 2024-08-19T14:17:21.823 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.823 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.823 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:17:21.823 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.657691+0000", 2024-08-19T14:17:21.823 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.8783999999999998e-05 2024-08-19T14:17:21.823 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.823 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.823 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:17:21.823 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.657808+0000", 2024-08-19T14:17:21.823 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000117142 2024-08-19T14:17:21.823 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.823 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.824 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,4", 2024-08-19T14:17:21.824 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.657949+0000", 2024-08-19T14:17:21.824 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014060200000000001 2024-08-19T14:17:21.824 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.824 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.824 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:17:21.824 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.658676+0000", 2024-08-19T14:17:21.824 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00072753900000000003 2024-08-19T14:17:21.824 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.824 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.824 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:17:21.824 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.658814+0000", 2024-08-19T14:17:21.824 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000137668 2024-08-19T14:17:21.825 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.825 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.825 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:17:21.825 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.846125+0000", 2024-08-19T14:17:21.825 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.18731145900000001 2024-08-19T14:17:21.825 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.825 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.825 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:17:21.825 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.846376+0000", 2024-08-19T14:17:21.825 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00025098200000000002 2024-08-19T14:17:21.825 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.825 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.825 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:17:21.825 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.846435+0000", 2024-08-19T14:17:21.825 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.8737999999999999e-05 2024-08-19T14:17:21.826 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:21.826 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:17:21.826 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:21.826 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.826 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.826 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25057.0:10244 3.e 3:7a99c205:::benchmark_data_smithi045_59973_object10243:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-08-19T14:17:21.826 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:17:04.660635+0000", 2024-08-19T14:17:21.826 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 17.033922314000002, 2024-08-19T14:17:21.826 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.18595352400000001, 2024-08-19T14:17:21.826 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:17:21.826 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:17:21.826 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:17:21.826 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:17:21.827 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25057", 2024-08-19T14:17:21.827 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:17:21.827 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 10244 2024-08-19T14:17:21.827 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.827 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:17:21.827 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.827 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:17:21.827 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.660635+0000", 2024-08-19T14:17:21.827 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:21.827 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.827 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.827 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:17:21.827 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.660635+0000", 2024-08-19T14:17:21.827 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:17:21.828 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.828 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.828 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:17:21.828 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.660636+0000", 2024-08-19T14:17:21.828 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3209999999999999e-06 2024-08-19T14:17:21.828 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.828 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.828 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:17:21.828 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.660654+0000", 2024-08-19T14:17:21.828 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8508e-05 2024-08-19T14:17:21.828 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.828 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.828 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:17:21.829 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.660667+0000", 2024-08-19T14:17:21.829 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2369e-05 2024-08-19T14:17:21.829 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.829 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.829 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:17:21.829 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.660672+0000", 2024-08-19T14:17:21.829 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0699999999999997e-06 2024-08-19T14:17:21.829 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.829 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.829 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:17:21.829 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.660720+0000", 2024-08-19T14:17:21.829 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.7698000000000003e-05 2024-08-19T14:17:21.829 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.830 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.830 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:17:21.830 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.660795+0000", 2024-08-19T14:17:21.830 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4998000000000003e-05 2024-08-19T14:17:21.830 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.830 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.830 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,4", 2024-08-19T14:17:21.830 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.660899+0000", 2024-08-19T14:17:21.830 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00010465 2024-08-19T14:17:21.830 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.830 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.830 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:17:21.830 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.661333+0000", 2024-08-19T14:17:21.830 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00043340300000000001 2024-08-19T14:17:21.831 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.831 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.831 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:17:21.831 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.661673+0000", 2024-08-19T14:17:21.831 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00034083399999999998 2024-08-19T14:17:21.831 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.831 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.831 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:17:21.831 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.846488+0000", 2024-08-19T14:17:21.831 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.184814757 2024-08-19T14:17:21.831 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.831 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.831 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:17:21.831 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.846550+0000", 2024-08-19T14:17:21.832 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.1364000000000004e-05 2024-08-19T14:17:21.832 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:17:21.832 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:17:21.832 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:17:21.832 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:04.846588+0000", 2024-08-19T14:17:21.832 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.8552e-05 2024-08-19T14:17:21.832 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:21.832 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:17:21.832 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:21.832 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:17:21.832 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:17:21.832 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:17:21.833 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:21.833 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:17:21.833 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is unset 2024-08-19T14:17:21.834 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:17:21.934 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:17:21.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:21.957 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:21.958 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:17:22.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:21 smithi032 ceph-mon[20004]: pgmap v558: 37 pgs: 1 peering, 10 unknown, 26 active+clean; 2.7 GiB data, 9.2 GiB used, 795 GiB / 805 GiB avail; 145 MiB/s wr, 2.32k op/s 2024-08-19T14:17:22.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:21 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "26"}]': finished 2024-08-19T14:17:22.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:21 smithi032 ceph-mon[20004]: osdmap e236: 12 total, 12 up, 9 in 2024-08-19T14:17:22.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:21 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2273916811' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:17:22.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:21 smithi045 ceph-mon[20143]: pgmap v558: 37 pgs: 1 peering, 10 unknown, 26 active+clean; 2.7 GiB data, 9.2 GiB used, 795 GiB / 805 GiB avail; 145 MiB/s wr, 2.32k op/s 2024-08-19T14:17:22.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:21 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "26"}]': finished 2024-08-19T14:17:22.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:21 smithi045 ceph-mon[20143]: osdmap e236: 12 total, 12 up, 9 in 2024-08-19T14:17:22.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:21 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2273916811' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:17:22.058 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:17:22.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:22.085 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:22.086 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:17:22.097 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:21 smithi031 ceph-mon[16336]: pgmap v558: 37 pgs: 1 peering, 10 unknown, 26 active+clean; 2.7 GiB data, 9.2 GiB used, 795 GiB / 805 GiB avail; 145 MiB/s wr, 2.32k op/s 2024-08-19T14:17:22.097 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:21 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "26"}]': finished 2024-08-19T14:17:22.097 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:21 smithi031 ceph-mon[16336]: osdmap e236: 12 total, 12 up, 9 in 2024-08-19T14:17:22.098 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:21 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2273916811' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:17:22.186 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:17:22.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:22.209 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:22.209 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:17:22.310 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:17:22.312 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:2024-08-19T14:17:22.313388+0000 min lat: 0.00117714 max lat: 1.07665 avg lat: 0.00587622 2024-08-19T14:17:22.313 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-19T14:17:22.313 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 20 16 50196 50180 156.778 0 - 0.00587622 2024-08-19T14:17:22.336 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:22.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:22.337 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:17:22.438 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:17:22.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:22.461 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:22.461 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:17:22.562 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:17:22.588 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:22.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:22.589 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:17:22.659 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:17:22.690 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:17:22.703 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-08-19T14:17:22.710 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:22.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:22.712 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:17:22.791 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:22 smithi031 ceph-mon[16336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:17:22.791 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:22 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2273916811' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:17:22.792 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:22 smithi031 ceph-mon[16336]: osdmap e237: 12 total, 12 up, 9 in 2024-08-19T14:17:22.792 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:22 smithi031 ceph-mon[16336]: osdmap e238: 12 total, 12 up, 9 in 2024-08-19T14:17:22.813 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:17:22.836 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:22.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:22.837 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:17:22.938 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:17:22.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:22.999 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:22.999 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:17:23.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:22 smithi045 ceph-mon[20143]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:17:23.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:22 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2273916811' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:17:23.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:22 smithi045 ceph-mon[20143]: osdmap e237: 12 total, 12 up, 9 in 2024-08-19T14:17:23.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:22 smithi045 ceph-mon[20143]: osdmap e238: 12 total, 12 up, 9 in 2024-08-19T14:17:23.100 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:17:23.125 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:23.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:23.126 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:17:23.227 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:17:23.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:22 smithi032 ceph-mon[20004]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:17:23.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:22 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2273916811' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:17:23.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:22 smithi032 ceph-mon[20004]: osdmap e237: 12 total, 12 up, 9 in 2024-08-19T14:17:23.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:22 smithi032 ceph-mon[20004]: osdmap e238: 12 total, 12 up, 9 in 2024-08-19T14:17:23.258 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:23.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:23.259 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:17:23.307 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:17:23.310 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:17:23.311 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:17:23.313 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 21 16 50592 50576 150.491 12.375 0.00789614 0.00663956 2024-08-19T14:17:23.359 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:17:23.360 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:17:23.360 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:17:23.396 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:17:23.397 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:17:23.400 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:23.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:23.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:17:23.434 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:17:23.434 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:17:23.468 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:17:23.468 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:17:23.502 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:17:23.503 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:17:23.503 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:17:23.529 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:23.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:23.530 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:17:23.536 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:17:23.536 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:17:23.577 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:17:23.578 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:17:23.615 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:17:23.616 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:17:23.630 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:17:23.665 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:17:23.665 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:17:23.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:23.688 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:23.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:17:23.703 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:17:23.703 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:17:23.742 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:17:23.742 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:17:23.789 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:17:23.814 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:23.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:23.815 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:17:23.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:23 smithi031 ceph-mon[16336]: pgmap v561: 37 pgs: 1 peering, 7 unknown, 29 active+clean; 2.6 GiB data, 10 GiB used, 795 GiB / 805 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-08-19T14:17:23.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:23 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2986790459' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:17:23.916 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:17:23.941 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:23.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:23.942 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:17:24.043 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:17:24.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:23 smithi045 ceph-mon[20143]: pgmap v561: 37 pgs: 1 peering, 7 unknown, 29 active+clean; 2.6 GiB data, 10 GiB used, 795 GiB / 805 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-08-19T14:17:24.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:23 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2986790459' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:17:24.071 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:24.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:24.073 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:17:24.173 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:17:24.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:24.208 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:24.209 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:17:24.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:23 smithi032 ceph-mon[20004]: pgmap v561: 37 pgs: 1 peering, 7 unknown, 29 active+clean; 2.6 GiB data, 10 GiB used, 795 GiB / 805 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-08-19T14:17:24.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:23 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2986790459' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:17:24.309 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:17:24.313 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 22 16 51807 51791 147.103 75.9375 0.0195408 0.00679178 2024-08-19T14:17:24.350 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:24.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:24.351 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:17:24.452 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:17:24.491 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:24.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:24.492 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:17:24.559 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-19T14:17:24.593 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:17:24.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:24.629 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:24.629 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:17:24.653 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:17:24.653 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:17:24.730 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:17:24.760 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:24.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:24.761 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:17:24.862 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:17:24.889 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:24.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:24.889 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:17:24.990 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:17:25.031 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:25.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:25.034 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:17:25.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:24 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:17:25.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:24 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:17:25.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:24 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:17:25.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:24 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:17:25.135 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:17:25.158 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:24 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:17:25.158 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:24 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:17:25.166 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:25.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:25.167 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:17:25.267 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:17:25.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:25.304 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:25.305 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:17:25.313 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 23 15 53092 53077 144.202 80.375 0.0135134 0.00692893 2024-08-19T14:17:25.406 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:17:25.407 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 7, 8, 9, 11] out_osds: [10, 6, 2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-08-19T14:17:25.407 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-19T14:17:25.407 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_1 2024-08-19T14:17:25.407 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-19T14:17:25.442 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-08-19T14:17:25.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:25.443 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:25.443 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:17:25.544 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:17:25.570 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:25.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:25.571 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:17:25.596 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:17:25.672 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:17:25.705 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:25.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:25.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:17:25.807 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:17:25.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:25.862 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:25.862 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:17:25.963 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:17:25.989 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:25.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:25.991 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:17:26.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:25 smithi032 ceph-mon[20004]: pgmap v563: 37 pgs: 2 activating+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 1 peering, 4 unknown, 26 active+clean; 2.7 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 4778/131115 objects degraded (3.644%); 15646/131115 objects misplaced (11.933%); 1020 KiB/s, 15 objects/s recovering 2024-08-19T14:17:26.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:25 smithi032 ceph-mon[20004]: Health check failed: Degraded data redundancy: 4778/131115 objects degraded (3.644%), 2 pgs degraded (PG_DEGRADED) 2024-08-19T14:17:26.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:25 smithi032 ceph-mon[20004]: osdmap e239: 12 total, 12 up, 9 in 2024-08-19T14:17:26.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:25 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:17:26.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:25 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:17:26.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:25 smithi045 ceph-mon[20143]: pgmap v563: 37 pgs: 2 activating+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 1 peering, 4 unknown, 26 active+clean; 2.7 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 4778/131115 objects degraded (3.644%); 15646/131115 objects misplaced (11.933%); 1020 KiB/s, 15 objects/s recovering 2024-08-19T14:17:26.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:25 smithi045 ceph-mon[20143]: Health check failed: Degraded data redundancy: 4778/131115 objects degraded (3.644%), 2 pgs degraded (PG_DEGRADED) 2024-08-19T14:17:26.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:25 smithi045 ceph-mon[20143]: osdmap e239: 12 total, 12 up, 9 in 2024-08-19T14:17:26.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:25 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:17:26.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:25 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:17:26.086 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:25 smithi031 ceph-mon[16336]: pgmap v563: 37 pgs: 2 activating+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 1 peering, 4 unknown, 26 active+clean; 2.7 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 4778/131115 objects degraded (3.644%); 15646/131115 objects misplaced (11.933%); 1020 KiB/s, 15 objects/s recovering 2024-08-19T14:17:26.086 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:25 smithi031 ceph-mon[16336]: Health check failed: Degraded data redundancy: 4778/131115 objects degraded (3.644%), 2 pgs degraded (PG_DEGRADED) 2024-08-19T14:17:26.086 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:25 smithi031 ceph-mon[16336]: osdmap e239: 12 total, 12 up, 9 in 2024-08-19T14:17:26.086 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:25 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:17:26.087 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:25 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:17:26.092 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:17:26.123 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:26.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:26.124 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:17:26.225 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:17:26.250 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:26.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:26.252 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:17:26.287 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:17:26.313 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 24 16 54055 54039 140.699 60.125 0.0292729 0.00710171 2024-08-19T14:17:26.334 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:17:26.353 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:17:26.418 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:26.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:26.419 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:17:26.520 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:17:26.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:26.559 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:26.560 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:17:26.578 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:17:26.660 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:17:26.689 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:26.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:26.690 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:17:26.791 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:17:26.793 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is unset 2024-08-19T14:17:26.818 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:26.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:26.819 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:17:26.920 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:17:26.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:26.976 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:26.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:17:27.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:26 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:17:27.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:26 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:17:27.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:26 smithi045 ceph-mon[20143]: osdmap e240: 12 total, 12 up, 9 in 2024-08-19T14:17:27.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:26 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2822037273' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:17:27.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:26 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:17:27.077 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:17:27.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:26 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:17:27.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:26 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:17:27.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:26 smithi031 ceph-mon[16336]: osdmap e240: 12 total, 12 up, 9 in 2024-08-19T14:17:27.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:26 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2822037273' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:17:27.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:26 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:17:27.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:27.137 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:27.137 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:17:27.238 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:17:27.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:26 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:17:27.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:26 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:17:27.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:26 smithi032 ceph-mon[20004]: osdmap e240: 12 total, 12 up, 9 in 2024-08-19T14:17:27.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:26 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2822037273' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:17:27.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:26 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:17:27.264 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:27.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:27.265 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:17:27.313 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 25 16 55155 55139 137.821 68.75 0.0191675 0.00724976 2024-08-19T14:17:27.367 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:17:27.410 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:27.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:27.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:17:27.512 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:17:27.557 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:17:27.557 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:17:27.558 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:27.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:27.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:17:27.660 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:17:27.703 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:27.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:27.704 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:17:27.759 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:17:27.760 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-08-19T14:17:27.805 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:17:27.831 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:27.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:27.832 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:17:27.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:27 smithi031 ceph-mon[16336]: pgmap v566: 37 pgs: 2 activating+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 2 unknown, 27 active+clean; 2.9 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 86 MiB/s wr, 1.38k op/s; 4778/143289 objects degraded (3.335%); 23093/143289 objects misplaced (16.116%); 2.5 MiB/s, 40 objects/s recovering 2024-08-19T14:17:27.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:27 smithi031 ceph-mon[16336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:17:27.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:27 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:17:27.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:27 smithi031 ceph-mon[16336]: osdmap e241: 12 total, 12 up, 9 in 2024-08-19T14:17:27.934 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:17:27.961 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:27.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:27.962 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:17:28.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:27 smithi045 ceph-mon[20143]: pgmap v566: 37 pgs: 2 activating+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 2 unknown, 27 active+clean; 2.9 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 86 MiB/s wr, 1.38k op/s; 4778/143289 objects degraded (3.335%); 23093/143289 objects misplaced (16.116%); 2.5 MiB/s, 40 objects/s recovering 2024-08-19T14:17:28.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:27 smithi045 ceph-mon[20143]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:17:28.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:27 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:17:28.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:27 smithi045 ceph-mon[20143]: osdmap e241: 12 total, 12 up, 9 in 2024-08-19T14:17:28.063 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:17:28.104 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:28.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:28.105 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:17:28.206 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:17:28.243 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:28.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:28.244 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:17:28.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:27 smithi032 ceph-mon[20004]: pgmap v566: 37 pgs: 2 activating+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 2 unknown, 27 active+clean; 2.9 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 86 MiB/s wr, 1.38k op/s; 4778/143289 objects degraded (3.335%); 23093/143289 objects misplaced (16.116%); 2.5 MiB/s, 40 objects/s recovering 2024-08-19T14:17:28.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:27 smithi032 ceph-mon[20004]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:17:28.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:27 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:17:28.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:27 smithi032 ceph-mon[20004]: osdmap e241: 12 total, 12 up, 9 in 2024-08-19T14:17:28.314 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 26 16 56408 56392 135.532 78.3125 0.0178495 0.00737284 2024-08-19T14:17:28.345 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:17:28.373 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:28.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:28.374 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:17:28.475 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:17:28.501 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:28.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:28.502 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:17:28.541 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":567,"stamp":"2024-08-19T14:17:26.788001+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3395420704,"num_objects":51803,"num_object_clones":0,"num_object_copies":155409,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4778,"num_objects_misplaced":28821,"num_objects_unfound":0,"num_objects_dirty":51803,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":51801,"num_write_kb":3315264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":436,"num_bytes_recovered":28709408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83227,"ondisk_log_size":83227,"up":111,"acting":101,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":120,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":11546944,"kb_used_data":10014072,"kb_used_omap":14,"kb_used_meta":1532721,"kb_avail":832159424,"statfs":{"total":863955320832,"available":852131250176,"internally_reserved":0,"allocated":10254409728,"data_stored":10244133301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14353,"internal_metadata":1569507311},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,1,0,0,0,0,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":632127488,"num_objects":9647,"num_object_clones":0,"num_object_copies":28941,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4778,"num_objects_misplaced":28821,"num_objects_unfound":0,"num_objects_dirty":9647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9647,"num_write_kb":617312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":435,"num_bytes_recovered":28508160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.977363"},"pg_stats":[{"pgid":"3.19","version":"238'3098","reported_seq":3180,"reported_epoch":238,"state":"active+recovering+undersized+remapped","last_fresh":"2024-08-19T14:17:23.115837+0000","last_change":"2024-08-19T14:17:23.012769+0000","last_active":"2024-08-19T14:17:23.115837+0000","last_peered":"2024-08-19T14:17:23.115837+0000","last_clean":"2024-08-19T14:17:20.708705+0000","last_became_active":"2024-08-19T14:17:22.981549+0000","last_became_peered":"2024-08-19T14:17:22.981549+0000","last_unstale":"2024-08-19T14:17:23.115837+0000","last_undegraded":"2024-08-19T14:17:23.115837+0000","last_fullsized":"2024-08-19T14:17:22.744477+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":3098,"log_dups_size":0,"ondisk_log_size":3098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104497152,"num_objects":1595,"num_object_clones":0,"num_object_copies":4785,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4752,"num_objects_unfound":0,"num_objects_dirty":1595,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1595,"num_write_kb":102048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":458752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[7,1],"avail_no_missing":["7","8","11"],"object_location_counts":[{"shards":"1,7","objects":11},{"shards":"7,8,11","objects":1584}],"blocked_by":[],"up_primary":1,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.18","version":"238'3190","reported_seq":3285,"reported_epoch":238,"state":"active+recovering+undersized+remapped","last_fresh":"2024-08-19T14:17:23.341483+0000","last_change":"2024-08-19T14:17:22.929098+0000","last_active":"2024-08-19T14:17:23.341483+0000","last_peered":"2024-08-19T14:17:23.341483+0000","last_clean":"2024-08-19T14:17:20.732716+0000","last_became_active":"2024-08-19T14:17:22.913380+0000","last_became_peered":"2024-08-19T14:17:22.913380+0000","last_unstale":"2024-08-19T14:17:23.341483+0000","last_undegraded":"2024-08-19T14:17:23.341483+0000","last_fullsized":"2024-08-19T14:17:22.745008+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":3190,"log_dups_size":0,"ondisk_log_size":3190,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107577344,"num_objects":1642,"num_object_clones":0,"num_object_copies":4926,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3116,"num_objects_unfound":0,"num_objects_dirty":1642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1642,"num_write_kb":105056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":917504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,1],"acting":[1,3],"avail_no_missing":["1","4","11"],"object_location_counts":[{"shards":"1,3","objects":84},{"shards":"1,4,11","objects":1558}],"blocked_by":[],"up_primary":3,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"238'3057","reported_seq":3275,"reported_epoch":238,"state":"active+recovering+undersized+remapped","last_fresh":"2024-08-19T14:17:24.132436+0000","last_change":"2024-08-19T14:17:23.035784+0000","last_active":"2024-08-19T14:17:24.132436+0000","last_peered":"2024-08-19T14:17:24.132436+0000","last_clean":"2024-08-19T14:17:20.743173+0000","last_became_active":"2024-08-19T14:17:22.948070+0000","last_became_peered":"2024-08-19T14:17:22.948070+0000","last_unstale":"2024-08-19T14:17:24.132436+0000","last_undegraded":"2024-08-19T14:17:24.132436+0000","last_fullsized":"2024-08-19T14:17:22.744386+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":3057,"log_dups_size":0,"ondisk_log_size":3057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103809024,"num_objects":1584,"num_object_clones":0,"num_object_copies":4752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4479,"num_objects_unfound":0,"num_objects_dirty":1584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1584,"num_write_kb":101376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":2686976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,9],"acting":[3,0],"avail_no_missing":["3","7","11"],"object_location_counts":[{"shards":"0,3","objects":91},{"shards":"3,7,11","objects":1493}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"239'3253","reported_seq":3695,"reported_epoch":240,"state":"active+recovering+undersized+remapped","last_fresh":"2024-08-19T14:17:25.822896+0000","last_change":"2024-08-19T14:17:22.960911+0000","last_active":"2024-08-19T14:17:25.822896+0000","last_peered":"2024-08-19T14:17:25.822896+0000","last_clean":"2024-08-19T14:17:20.736413+0000","last_became_active":"2024-08-19T14:17:22.917149+0000","last_became_peered":"2024-08-19T14:17:22.917149+0000","last_unstale":"2024-08-19T14:17:25.822896+0000","last_undegraded":"2024-08-19T14:17:25.822896+0000","last_fullsized":"2024-08-19T14:17:22.744893+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":3253,"log_dups_size":0,"ondisk_log_size":3253,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113147904,"num_objects":1727,"num_object_clones":0,"num_object_copies":5181,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2938,"num_objects_unfound":0,"num_objects_dirty":1727,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1727,"num_write_kb":110496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":128,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[0,1],"avail_no_missing":["0","1","4"],"object_location_counts":[{"shards":"0,1","objects":258},{"shards":"0,1,4","objects":1469}],"blocked_by":[],"up_primary":5,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"238'3068","reported_seq":3301,"reported_epoch":238,"state":"active+recovering+undersized+remapped","last_fresh":"2024-08-19T14:17:24.125417+0000","last_change":"2024-08-19T14:17:22.967109+0000","last_active":"2024-08-19T14:17:24.125417+0000","last_peered":"2024-08-19T14:17:24.125417+0000","last_clean":"2024-08-19T14:17:20.738019+0000","last_became_active":"2024-08-19T14:17:22.933773+0000","last_became_peered":"2024-08-19T14:17:22.933773+0000","last_unstale":"2024-08-19T14:17:24.125417+0000","last_undegraded":"2024-08-19T14:17:24.125417+0000","last_fullsized":"2024-08-19T14:17:22.744381+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":3068,"log_dups_size":0,"ondisk_log_size":3068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104103936,"num_objects":1589,"num_object_clones":0,"num_object_copies":4767,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2968,"num_objects_unfound":0,"num_objects_dirty":1589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1589,"num_write_kb":101664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":3932160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[3,7],"avail_no_missing":["3","5","7"],"object_location_counts":[{"shards":"3,5,7","objects":1484},{"shards":"3,7","objects":105}],"blocked_by":[],"up_primary":8,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.14","version":"238'3091","reported_seq":3190,"reported_epoch":238,"state":"active+recovering+undersized+remapped","last_fresh":"2024-08-19T14:17:23.342116+0000","last_change":"2024-08-19T14:17:22.945483+0000","last_active":"2024-08-19T14:17:23.342116+0000","last_peered":"2024-08-19T14:17:23.342116+0000","last_clean":"2024-08-19T14:17:20.729919+0000","last_became_active":"2024-08-19T14:17:22.891171+0000","last_became_peered":"2024-08-19T14:17:22.891171+0000","last_unstale":"2024-08-19T14:17:23.342116+0000","last_undegraded":"2024-08-19T14:17:23.342116+0000","last_fullsized":"2024-08-19T14:17:22.746029+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":3091,"log_dups_size":0,"ondisk_log_size":3091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104300544,"num_objects":1592,"num_object_clones":0,"num_object_copies":4776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3122,"num_objects_unfound":0,"num_objects_dirty":1592,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1592,"num_write_kb":101856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":983040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,11],"acting":[1,11],"avail_no_missing":["1","5","11"],"object_location_counts":[{"shards":"1,5,11","objects":1561},{"shards":"1,11","objects":31}],"blocked_by":[],"up_primary":4,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.13","version":"239'3158","reported_seq":3638,"reported_epoch":240,"state":"active+recovering+undersized+remapped","last_fresh":"2024-08-19T14:17:25.820615+0000","last_change":"2024-08-19T14:17:22.923581+0000","last_active":"2024-08-19T14:17:25.820615+0000","last_peered":"2024-08-19T14:17:25.820615+0000","last_clean":"2024-08-19T14:17:20.721339+0000","last_became_active":"2024-08-19T14:17:22.894421+0000","last_became_peered":"2024-08-19T14:17:22.894421+0000","last_unstale":"2024-08-19T14:17:25.820615+0000","last_undegraded":"2024-08-19T14:17:25.820615+0000","last_fullsized":"2024-08-19T14:17:22.743946+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":3158,"log_dups_size":0,"ondisk_log_size":3158,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109936640,"num_objects":1678,"num_object_clones":0,"num_object_copies":5034,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2790,"num_objects_unfound":0,"num_objects_dirty":1678,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1678,"num_write_kb":107360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":141,"num_bytes_recovered":9240576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,9],"acting":[0,4],"avail_no_missing":["0","4","11"],"object_location_counts":[{"shards":"0,4","objects":283},{"shards":"0,4,11","objects":1395}],"blocked_by":[],"up_primary":7,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"235'3101","reported_seq":3147,"reported_epoch":238,"state":"activating+undersized+degraded+remapped","last_fresh":"2024-08-19T14:17:22.859080+0000","last_change":"2024-08-19T14:17:22.859080+0000","last_active":"2024-08-19T14:17:21.742666+0000","last_peered":"2024-08-19T14:17:20.726018+0000","last_clean":"2024-08-19T14:17:20.726018+0000","last_became_active":"2024-08-19T14:17:19.753357+0000","last_became_peered":"2024-08-19T14:17:19.753357+0000","last_unstale":"2024-08-19T14:17:22.859080+0000","last_undegraded":"2024-08-19T14:17:22.820013+0000","last_fullsized":"2024-08-19T14:17:22.743872+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":3101,"log_dups_size":0,"ondisk_log_size":3101,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103809024,"num_objects":1584,"num_object_clones":0,"num_object_copies":4752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3226,"num_objects_misplaced":1613,"num_objects_unfound":0,"num_objects_dirty":1584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1584,"num_write_kb":101376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[4,0],"avail_no_missing":["4","5","9"],"object_location_counts":[{"shards":"0,4","objects":-29},{"shards":"4,5,9","objects":1613}],"blocked_by":[],"up_primary":0,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"235'3011","reported_seq":3049,"reported_epoch":238,"state":"activating+undersized+degraded+remapped","last_fresh":"2024-08-19T14:17:22.785385+0000","last_change":"2024-08-19T14:17:22.785385+0000","last_active":"2024-08-19T14:17:20.743276+0000","last_peered":"2024-08-19T14:17:20.703417+0000","last_clean":"2024-08-19T14:17:20.703417+0000","last_became_active":"2024-08-19T14:17:19.756447+0000","last_became_peered":"2024-08-19T14:17:19.756447+0000","last_unstale":"2024-08-19T14:17:22.785385+0000","last_undegraded":"2024-08-19T14:17:22.755483+0000","last_fullsized":"2024-08-19T14:17:22.743857+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":3011,"log_dups_size":0,"ondisk_log_size":3011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100892672,"num_objects":1540,"num_object_clones":0,"num_object_copies":4620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1540,"num_write_kb":98528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0],"avail_no_missing":["4","0","11"],"object_location_counts":[{"shards":"0,4","objects":-12},{"shards":"0,4,11","objects":1552}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"238'2959","reported_seq":3099,"reported_epoch":238,"state":"active+recovering+undersized+remapped","last_fresh":"2024-08-19T14:17:23.299075+0000","last_change":"2024-08-19T14:17:22.932233+0000","last_active":"2024-08-19T14:17:23.299075+0000","last_peered":"2024-08-19T14:17:23.299075+0000","last_clean":"2024-08-19T14:17:20.697485+0000","last_became_active":"2024-08-19T14:17:22.913401+0000","last_became_peered":"2024-08-19T14:17:22.913401+0000","last_unstale":"2024-08-19T14:17:23.299075+0000","last_undegraded":"2024-08-19T14:17:23.299075+0000","last_fullsized":"2024-08-19T14:17:22.746763+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":2959,"log_dups_size":0,"ondisk_log_size":2959,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98959360,"num_objects":1510,"num_object_clones":0,"num_object_copies":4530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3043,"num_objects_unfound":0,"num_objects_dirty":1510,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1510,"num_write_kb":96640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":1900544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,3],"acting":[9,3],"avail_no_missing":["9","1","11"],"object_location_counts":[{"shards":"1,5,9,11","objects":1},{"shards":"1,9,11","objects":1521},{"shards":"3,9","objects":-12}],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.f","version":"238'3154","reported_seq":3199,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-08-19T14:17:23.107235+0000","last_change":"2024-08-19T14:16:56.416145+0000","last_active":"2024-08-19T14:17:23.107235+0000","last_peered":"2024-08-19T14:17:23.107235+0000","last_clean":"2024-08-19T14:17:23.107235+0000","last_became_active":"2024-08-19T14:16:56.415674+0000","last_became_peered":"2024-08-19T14:16:56.415674+0000","last_unstale":"2024-08-19T14:17:23.107235+0000","last_undegraded":"2024-08-19T14:17:23.107235+0000","last_fullsized":"2024-08-19T14:17:23.107235+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":3154,"log_dups_size":0,"ondisk_log_size":3154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:14.958699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":206700544,"num_objects":3154,"num_object_clones":0,"num_object_copies":9462,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3154,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3154,"num_write_kb":201856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"239'3321","reported_seq":3365,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-08-19T14:17:25.630008+0000","last_change":"2024-08-19T14:16:56.417716+0000","last_active":"2024-08-19T14:17:25.630008+0000","last_peered":"2024-08-19T14:17:25.630008+0000","last_clean":"2024-08-19T14:17:25.630008+0000","last_became_active":"2024-08-19T14:16:56.417440+0000","last_became_peered":"2024-08-19T14:16:56.417440+0000","last_unstale":"2024-08-19T14:17:25.630008+0000","last_undegraded":"2024-08-19T14:17:25.630008+0000","last_fullsized":"2024-08-19T14:17:25.630008+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":3321,"log_dups_size":0,"ondisk_log_size":3321,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:21.555611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217645056,"num_objects":3321,"num_object_clones":0,"num_object_copies":9963,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3321,"num_write_kb":212544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,0],"acting":[11,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.d","version":"238'3237","reported_seq":3282,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-08-19T14:17:23.069727+0000","last_change":"2024-08-19T14:16:56.415868+0000","last_active":"2024-08-19T14:17:23.069727+0000","last_peered":"2024-08-19T14:17:23.069727+0000","last_clean":"2024-08-19T14:17:23.069727+0000","last_became_active":"2024-08-19T14:16:56.414976+0000","last_became_peered":"2024-08-19T14:16:56.414976+0000","last_unstale":"2024-08-19T14:17:23.069727+0000","last_undegraded":"2024-08-19T14:17:23.069727+0000","last_fullsized":"2024-08-19T14:17:23.069727+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":3237,"log_dups_size":0,"ondisk_log_size":3237,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:09.977685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":212140032,"num_objects":3237,"num_object_clones":0,"num_object_copies":9711,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3237,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3237,"num_write_kb":207168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,9],"acting":[7,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"240'3357","reported_seq":3406,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-08-19T14:17:26.557307+0000","last_change":"2024-08-19T14:16:56.415146+0000","last_active":"2024-08-19T14:17:26.557307+0000","last_peered":"2024-08-19T14:17:26.557307+0000","last_clean":"2024-08-19T14:17:26.557307+0000","last_became_active":"2024-08-19T14:16:56.414671+0000","last_became_peered":"2024-08-19T14:16:56.414671+0000","last_unstale":"2024-08-19T14:17:26.557307+0000","last_undegraded":"2024-08-19T14:17:26.557307+0000","last_fullsized":"2024-08-19T14:17:26.557307+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":3357,"log_dups_size":0,"ondisk_log_size":3357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:05.145613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220004352,"num_objects":3357,"num_object_clones":0,"num_object_copies":10071,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3357,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3357,"num_write_kb":214848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,9],"acting":[5,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"238'3146","reported_seq":3199,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-08-19T14:17:24.778126+0000","last_change":"2024-08-19T14:17:19.745357+0000","last_active":"2024-08-19T14:17:24.778126+0000","last_peered":"2024-08-19T14:17:24.778126+0000","last_clean":"2024-08-19T14:17:24.778126+0000","last_became_active":"2024-08-19T14:17:19.743605+0000","last_became_peered":"2024-08-19T14:17:19.743605+0000","last_unstale":"2024-08-19T14:17:24.778126+0000","last_undegraded":"2024-08-19T14:17:24.778126+0000","last_fullsized":"2024-08-19T14:17:24.778126+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":3146,"log_dups_size":0,"ondisk_log_size":3146,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:17:25.980075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107642880,"num_objects":1642,"num_object_clones":0,"num_object_copies":4926,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1642,"num_write_kb":105120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"236'3131","reported_seq":3176,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-08-19T14:17:22.744128+0000","last_change":"2024-08-19T14:16:56.415768+0000","last_active":"2024-08-19T14:17:22.744128+0000","last_peered":"2024-08-19T14:17:22.744128+0000","last_clean":"2024-08-19T14:17:22.744128+0000","last_became_active":"2024-08-19T14:16:56.415495+0000","last_became_peered":"2024-08-19T14:16:56.415495+0000","last_unstale":"2024-08-19T14:17:22.744128+0000","last_undegraded":"2024-08-19T14:17:22.744128+0000","last_fullsized":"2024-08-19T14:17:22.744128+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":3131,"log_dups_size":0,"ondisk_log_size":3131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:00.319593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205193216,"num_objects":3131,"num_object_clones":0,"num_object_copies":9393,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3131,"num_write_kb":200384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,8],"acting":[4,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"70'139","reported_seq":571,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-08-19T14:17:25.786461+0000","last_change":"2024-08-19T14:16:42.249971+0000","last_active":"2024-08-19T14:17:25.786461+0000","last_peered":"2024-08-19T14:17:25.786461+0000","last_clean":"2024-08-19T14:17:25.786461+0000","last_became_active":"2024-08-19T14:16:42.249518+0000","last_became_peered":"2024-08-19T14:16:42.249518+0000","last_unstale":"2024-08-19T14:17:25.786461+0000","last_undegraded":"2024-08-19T14:17:25.786461+0000","last_fullsized":"2024-08-19T14:17:25.786461+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:15:44.541487+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:15:44.541487+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,8],"acting":[5,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.b","version":"238'3197","reported_seq":3242,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-08-19T14:17:24.086313+0000","last_change":"2024-08-19T14:16:56.414110+0000","last_active":"2024-08-19T14:17:24.086313+0000","last_peered":"2024-08-19T14:17:24.086313+0000","last_clean":"2024-08-19T14:17:24.086313+0000","last_became_active":"2024-08-19T14:16:56.413751+0000","last_became_peered":"2024-08-19T14:16:56.413751+0000","last_unstale":"2024-08-19T14:17:24.086313+0000","last_undegraded":"2024-08-19T14:17:24.086313+0000","last_fullsized":"2024-08-19T14:17:24.086313+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":3197,"log_dups_size":0,"ondisk_log_size":3197,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:17:58.823513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209518592,"num_objects":3197,"num_object_clones":0,"num_object_copies":9591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3197,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3197,"num_write_kb":204608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"70'139","reported_seq":685,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-08-19T14:17:22.822023+0000","last_change":"2024-08-19T14:16:42.252897+0000","last_active":"2024-08-19T14:17:22.822023+0000","last_peered":"2024-08-19T14:17:22.822023+0000","last_clean":"2024-08-19T14:17:22.822023+0000","last_became_active":"2024-08-19T14:16:42.252391+0000","last_became_peered":"2024-08-19T14:16:42.252391+0000","last_unstale":"2024-08-19T14:17:22.822023+0000","last_undegraded":"2024-08-19T14:17:22.822023+0000","last_fullsized":"2024-08-19T14:17:22.822023+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:16:03.918046+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:17:27.707808+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"239'3264","reported_seq":3317,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-08-19T14:17:25.580976+0000","last_change":"2024-08-19T14:17:19.755178+0000","last_active":"2024-08-19T14:17:25.580976+0000","last_peered":"2024-08-19T14:17:25.580976+0000","last_clean":"2024-08-19T14:17:25.580976+0000","last_became_active":"2024-08-19T14:17:19.753063+0000","last_became_peered":"2024-08-19T14:17:19.753063+0000","last_unstale":"2024-08-19T14:17:25.580976+0000","last_undegraded":"2024-08-19T14:17:25.580976+0000","last_fullsized":"2024-08-19T14:17:25.580976+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":3264,"log_dups_size":0,"ondisk_log_size":3264,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:17:26.435959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112427008,"num_objects":1715,"num_object_clones":0,"num_object_copies":5145,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1715,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1715,"num_write_kb":109792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,4],"acting":[11,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.a","version":"70'139","reported_seq":562,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-08-19T14:17:22.746750+0000","last_change":"2024-08-19T14:16:02.818848+0000","last_active":"2024-08-19T14:17:22.746750+0000","last_peered":"2024-08-19T14:17:22.746750+0000","last_clean":"2024-08-19T14:17:22.746750+0000","last_became_active":"2024-08-19T14:13:22.728779+0000","last_became_peered":"2024-08-19T14:13:22.728779+0000","last_unstale":"2024-08-19T14:17:22.746750+0000","last_undegraded":"2024-08-19T14:17:22.746750+0000","last_fullsized":"2024-08-19T14:17:22.746750+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:16:02.818774+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:16:02.818774+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,0],"acting":[9,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.6","version":"239'3241","reported_seq":3296,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-08-19T14:17:25.782402+0000","last_change":"2024-08-19T14:17:19.750488+0000","last_active":"2024-08-19T14:17:25.782402+0000","last_peered":"2024-08-19T14:17:25.782402+0000","last_clean":"2024-08-19T14:17:25.782402+0000","last_became_active":"2024-08-19T14:17:19.748180+0000","last_became_peered":"2024-08-19T14:17:19.748180+0000","last_unstale":"2024-08-19T14:17:25.782402+0000","last_undegraded":"2024-08-19T14:17:25.782402+0000","last_fullsized":"2024-08-19T14:17:25.782402+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":3241,"log_dups_size":0,"ondisk_log_size":3241,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:17:26.589045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112361472,"num_objects":1714,"num_object_clones":0,"num_object_copies":5142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1714,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1714,"num_write_kb":109728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"70'139","reported_seq":603,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-08-19T14:17:24.769576+0000","last_change":"2024-08-19T14:16:29.257033+0000","last_active":"2024-08-19T14:17:24.769576+0000","last_peered":"2024-08-19T14:17:24.769576+0000","last_clean":"2024-08-19T14:17:24.769576+0000","last_became_active":"2024-08-19T14:13:05.607447+0000","last_became_peered":"2024-08-19T14:13:05.607447+0000","last_unstale":"2024-08-19T14:17:24.769576+0000","last_undegraded":"2024-08-19T14:17:24.769576+0000","last_fullsized":"2024-08-19T14:17:24.769576+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:16:29.256959+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:16:29.256959+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:17:40.848859+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.7","version":"238'3062","reported_seq":3113,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-08-19T14:17:24.055181+0000","last_change":"2024-08-19T14:17:19.746696+0000","last_active":"2024-08-19T14:17:24.055181+0000","last_peered":"2024-08-19T14:17:24.055181+0000","last_clean":"2024-08-19T14:17:24.055181+0000","last_became_active":"2024-08-19T14:17:19.742604+0000","last_became_peered":"2024-08-19T14:17:19.742604+0000","last_unstale":"2024-08-19T14:17:24.055181+0000","last_undegraded":"2024-08-19T14:17:24.055181+0000","last_fullsized":"2024-08-19T14:17:24.055181+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":3062,"log_dups_size":0,"ondisk_log_size":3062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:17:26.449575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104136704,"num_objects":1589,"num_object_clones":0,"num_object_copies":4767,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1589,"num_write_kb":101696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"70'139","reported_seq":365,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-08-19T14:17:24.769505+0000","last_change":"2024-08-19T14:16:42.252111+0000","last_active":"2024-08-19T14:17:24.769505+0000","last_peered":"2024-08-19T14:17:24.769505+0000","last_clean":"2024-08-19T14:17:24.769505+0000","last_became_active":"2024-08-19T14:16:42.251643+0000","last_became_peered":"2024-08-19T14:16:42.251643+0000","last_unstale":"2024-08-19T14:17:24.769505+0000","last_undegraded":"2024-08-19T14:17:24.769505+0000","last_fullsized":"2024-08-19T14:17:24.769505+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:16:30.232687+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:16:30.232687+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:17:59.854133+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,4],"acting":[11,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"238'3072","reported_seq":3123,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-08-19T14:17:23.282141+0000","last_change":"2024-08-19T14:17:19.744537+0000","last_active":"2024-08-19T14:17:23.282141+0000","last_peered":"2024-08-19T14:17:23.282141+0000","last_clean":"2024-08-19T14:17:23.282141+0000","last_became_active":"2024-08-19T14:17:19.741977+0000","last_became_peered":"2024-08-19T14:17:19.741977+0000","last_unstale":"2024-08-19T14:17:23.282141+0000","last_undegraded":"2024-08-19T14:17:23.282141+0000","last_fullsized":"2024-08-19T14:17:23.282141+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":3072,"log_dups_size":0,"ondisk_log_size":3072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:17:26.177316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103055360,"num_objects":1572,"num_object_clones":0,"num_object_copies":4716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1572,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1572,"num_write_kb":100640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,5],"acting":[1,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"70'139","reported_seq":549,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-08-19T14:17:22.746623+0000","last_change":"2024-08-19T14:16:36.969224+0000","last_active":"2024-08-19T14:17:22.746623+0000","last_peered":"2024-08-19T14:17:22.746623+0000","last_clean":"2024-08-19T14:17:22.746623+0000","last_became_active":"2024-08-19T14:13:11.647085+0000","last_became_peered":"2024-08-19T14:13:11.647085+0000","last_unstale":"2024-08-19T14:17:22.746623+0000","last_undegraded":"2024-08-19T14:17:22.746623+0000","last_fullsized":"2024-08-19T14:17:22.746623+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:16:36.969114+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:16:36.969114+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:17:44.848868+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,4],"acting":[9,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"240'3149","reported_seq":3204,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-08-19T14:17:26.490188+0000","last_change":"2024-08-19T14:17:19.749394+0000","last_active":"2024-08-19T14:17:26.490188+0000","last_peered":"2024-08-19T14:17:26.490188+0000","last_clean":"2024-08-19T14:17:26.490188+0000","last_became_active":"2024-08-19T14:17:19.745911+0000","last_became_peered":"2024-08-19T14:17:19.745911+0000","last_unstale":"2024-08-19T14:17:26.490188+0000","last_undegraded":"2024-08-19T14:17:26.490188+0000","last_fullsized":"2024-08-19T14:17:26.490188+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":3149,"log_dups_size":0,"ondisk_log_size":3149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:17:31.404668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109412352,"num_objects":1669,"num_object_clones":0,"num_object_copies":5007,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1669,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1669,"num_write_kb":106848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"70'139","reported_seq":471,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-08-19T14:17:25.786710+0000","last_change":"2024-08-19T14:16:42.252166+0000","last_active":"2024-08-19T14:17:25.786710+0000","last_peered":"2024-08-19T14:17:25.786710+0000","last_clean":"2024-08-19T14:17:25.786710+0000","last_became_active":"2024-08-19T14:16:42.251741+0000","last_became_peered":"2024-08-19T14:16:42.251741+0000","last_unstale":"2024-08-19T14:17:25.786710+0000","last_undegraded":"2024-08-19T14:17:25.786710+0000","last_fullsized":"2024-08-19T14:17:25.786710+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:16:10.672834+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:17:33.011576+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"236'3041","reported_seq":3092,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-08-19T14:17:22.744597+0000","last_change":"2024-08-19T14:17:19.753730+0000","last_active":"2024-08-19T14:17:22.744597+0000","last_peered":"2024-08-19T14:17:22.744597+0000","last_clean":"2024-08-19T14:17:22.744597+0000","last_became_active":"2024-08-19T14:17:19.751952+0000","last_became_peered":"2024-08-19T14:17:19.751952+0000","last_unstale":"2024-08-19T14:17:22.744597+0000","last_undegraded":"2024-08-19T14:17:22.744597+0000","last_fullsized":"2024-08-19T14:17:22.744597+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":3041,"log_dups_size":0,"ondisk_log_size":3041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:17:26.016293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102268928,"num_objects":1560,"num_object_clones":0,"num_object_copies":4680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1560,"num_write_kb":99872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,11],"acting":[4,0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"70'139","reported_seq":559,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-08-19T14:17:22.746600+0000","last_change":"2024-08-19T14:16:42.252108+0000","last_active":"2024-08-19T14:17:22.746600+0000","last_peered":"2024-08-19T14:17:22.746600+0000","last_clean":"2024-08-19T14:17:22.746600+0000","last_became_active":"2024-08-19T14:16:42.251603+0000","last_became_peered":"2024-08-19T14:16:42.251603+0000","last_unstale":"2024-08-19T14:17:22.746600+0000","last_undegraded":"2024-08-19T14:17:22.746600+0000","last_fullsized":"2024-08-19T14:17:22.746600+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:16:30.043526+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:16:30.043526+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:17:54.530868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,11],"acting":[9,0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.0","version":"239'3085","reported_seq":3138,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-08-19T14:17:25.525411+0000","last_change":"2024-08-19T14:17:19.755432+0000","last_active":"2024-08-19T14:17:25.525411+0000","last_peered":"2024-08-19T14:17:25.525411+0000","last_clean":"2024-08-19T14:17:25.525411+0000","last_became_active":"2024-08-19T14:17:19.753833+0000","last_became_peered":"2024-08-19T14:17:19.753833+0000","last_unstale":"2024-08-19T14:17:25.525411+0000","last_undegraded":"2024-08-19T14:17:25.525411+0000","last_fullsized":"2024-08-19T14:17:25.525411+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":3085,"log_dups_size":0,"ondisk_log_size":3085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:17:26.435908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107216896,"num_objects":1636,"num_object_clones":0,"num_object_copies":4908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1636,"num_write_kb":104704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,1],"acting":[11,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.2","version":"70'139","reported_seq":510,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-08-19T14:17:25.781341+0000","last_change":"2024-08-19T14:16:00.592808+0000","last_active":"2024-08-19T14:17:25.781341+0000","last_peered":"2024-08-19T14:17:25.781341+0000","last_clean":"2024-08-19T14:17:25.781341+0000","last_became_active":"2024-08-19T14:15:59.993566+0000","last_became_peered":"2024-08-19T14:15:59.993566+0000","last_unstale":"2024-08-19T14:17:25.781341+0000","last_undegraded":"2024-08-19T14:17:25.781341+0000","last_fullsized":"2024-08-19T14:17:25.781341+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:16:00.592678+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:16:00.592678+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:17:26.589063+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,8],"acting":[0,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"240'3130","reported_seq":3176,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-08-19T14:17:25.780157+0000","last_change":"2024-08-19T14:17:19.755555+0000","last_active":"2024-08-19T14:17:25.780157+0000","last_peered":"2024-08-19T14:17:25.780157+0000","last_clean":"2024-08-19T14:17:25.780157+0000","last_became_active":"2024-08-19T14:17:19.753198+0000","last_became_peered":"2024-08-19T14:17:19.753198+0000","last_unstale":"2024-08-19T14:17:25.780157+0000","last_undegraded":"2024-08-19T14:17:25.780157+0000","last_fullsized":"2024-08-19T14:17:25.780157+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":3130,"log_dups_size":0,"ondisk_log_size":3130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:17:26.589023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108691456,"num_objects":1658,"num_object_clones":0,"num_object_copies":4974,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1658,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1658,"num_write_kb":106144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,11],"acting":[0,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"70'139","reported_seq":562,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-08-19T14:17:22.745252+0000","last_change":"2024-08-19T14:16:06.679203+0000","last_active":"2024-08-19T14:17:22.745252+0000","last_peered":"2024-08-19T14:17:22.745252+0000","last_clean":"2024-08-19T14:17:22.745252+0000","last_became_active":"2024-08-19T14:13:22.727234+0000","last_became_peered":"2024-08-19T14:13:22.727234+0000","last_unstale":"2024-08-19T14:17:22.745252+0000","last_undegraded":"2024-08-19T14:17:22.745252+0000","last_fullsized":"2024-08-19T14:17:22.745252+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:16:06.679075+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:16:06.679075+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:17:31.357116+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"238'3125","reported_seq":3176,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-08-19T14:17:23.234582+0000","last_change":"2024-08-19T14:17:19.741597+0000","last_active":"2024-08-19T14:17:23.234582+0000","last_peered":"2024-08-19T14:17:23.234582+0000","last_clean":"2024-08-19T14:17:23.234582+0000","last_became_active":"2024-08-19T14:17:19.739846+0000","last_became_peered":"2024-08-19T14:17:19.739846+0000","last_unstale":"2024-08-19T14:17:23.234582+0000","last_undegraded":"2024-08-19T14:17:23.234582+0000","last_fullsized":"2024-08-19T14:17:23.234582+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":3125,"log_dups_size":0,"ondisk_log_size":3125,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:17:26.048989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":105381888,"num_objects":1608,"num_object_clones":0,"num_object_copies":4824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1608,"num_write_kb":102912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,5],"acting":[9,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.0","version":"70'139","reported_seq":663,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-08-19T14:17:22.744580+0000","last_change":"2024-08-19T14:16:42.251163+0000","last_active":"2024-08-19T14:17:22.744580+0000","last_peered":"2024-08-19T14:17:22.744580+0000","last_clean":"2024-08-19T14:17:22.744580+0000","last_became_active":"2024-08-19T14:16:42.250702+0000","last_became_peered":"2024-08-19T14:16:42.250702+0000","last_unstale":"2024-08-19T14:17:22.744580+0000","last_undegraded":"2024-08-19T14:17:22.744580+0000","last_fullsized":"2024-08-19T14:17:22.744580+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:15:59.869266+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:15:59.869266+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:17:24.361856+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,9],"acting":[7,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":26,"stat_sum":{"num_bytes":3394830336,"num_objects":51801,"num_object_clones":0,"num_object_copies":155403,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4778,"num_objects_misplaced":28821,"num_objects_unfound":0,"num_objects_dirty":51801,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":51801,"num_write_kb":3315264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":435,"num_bytes_recovered":28508160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10312417280,"data_stored":10312417280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81698,"ondisk_log_size":81698,"up":78,"acting":68,"num_store_stats":9},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1802240,"data_stored":1788032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1529,"ondisk_log_size":1529,"up":33,"acting":33,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187529,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2072696,"kb_used_data":1892968,"kb_used_omap":1,"kb_used_meta":179710,"kb_avail":91672456,"statfs":{"total":95995035648,"available":93872594944,"internally_reserved":0,"allocated":1938399232,"data_stored":1937255997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":184023492},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":343597383757,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":74,"seq":317827579986,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1017900,"kb_used_data":806236,"kb_used_omap":1,"kb_used_meta":211646,"kb_avail":92727252,"statfs":{"total":95995035648,"available":94952706048,"internally_reserved":0,"allocated":825585664,"data_stored":824446311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":216725953},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874328,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":796268,"kb_used_data":630436,"kb_used_omap":1,"kb_used_meta":165822,"kb_avail":92948884,"statfs":{"total":95995035648,"available":95179657216,"internally_reserved":0,"allocated":645566464,"data_stored":644423559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":169802181},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":57,"seq":244813135962,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1208944,"kb_used_data":1011164,"kb_used_omap":1,"kb_used_meta":197758,"kb_avail":92536208,"statfs":{"total":95995035648,"available":94757076992,"internally_reserved":0,"allocated":1035431936,"data_stored":1034292583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":202504650},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":206158430302,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691939,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1246920,"kb_used_data":1125500,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":92498232,"statfs":{"total":95995035648,"available":94718189568,"internally_reserved":0,"allocated":1152512000,"data_stored":1151361381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986279,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1781536,"kb_used_data":1615696,"kb_used_omap":1,"kb_used_meta":165822,"kb_avail":91963616,"statfs":{"total":95995035648,"available":94170742784,"internally_reserved":0,"allocated":1654472704,"data_stored":1653337235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":169802181},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149805,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":991672,"kb_used_data":816100,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":92753480,"statfs":{"total":95995035648,"available":94979563520,"internally_reserved":0,"allocated":835686400,"data_stored":834543495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":179763655},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411442,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574966,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1190828,"kb_used_data":1019420,"kb_used_omap":1,"kb_used_meta":171390,"kb_avail":92554324,"statfs":{"total":95995035648,"available":94775627776,"internally_reserved":0,"allocated":1043886080,"data_stored":1042746727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":175503813},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738491,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1240180,"kb_used_data":1096552,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":92504972,"statfs":{"total":95995035648,"available":94725091328,"internally_reserved":0,"allocated":1122869248,"data_stored":1121726013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":147061189},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1127874560,"data_stored":1127874560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1045102592,"data_stored":1045102592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":863174656,"data_stored":863174656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1651703808,"data_stored":1651703808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1154809856,"data_stored":1154809856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1037959168,"data_stored":1037959168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":661389312,"data_stored":661389312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":829816832,"data_stored":829816832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1940586496,"data_stored":1940586496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T14:17:28.543 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-08-19T14:17:28.543 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd pool set unique_pool_1 pg_num 36 2024-08-19T14:17:28.603 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:17:28.636 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:28.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:28.638 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:17:28.739 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:17:28.768 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:28.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:28.770 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:17:28.871 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:17:28.899 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:28.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:28.901 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:17:29.002 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:17:29.024 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:29.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:29.026 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:17:29.091 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:28 smithi031 ceph-mon[16336]: from='client.25123 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:17:29.127 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:17:29.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:29.169 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:29.169 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:17:29.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:28 smithi032 ceph-mon[20004]: from='client.25123 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:17:29.271 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:17:29.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:28 smithi045 ceph-mon[20143]: from='client.25123 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:17:29.297 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:29.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:29.299 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:17:29.313 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 27 16 57659 57643 133.408 78.1875 0.00360297 0.00749024 2024-08-19T14:17:29.400 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:17:29.436 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:29.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:29.437 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:17:29.512 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-08-19T14:17:29.537 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:17:29.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:29.563 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:29.564 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:17:29.571 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:17:29.575 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:17:29.664 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:17:29.690 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:29.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:29.691 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:17:29.793 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:17:29.839 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:29.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:29.840 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:17:29.942 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:17:29.967 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:29.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:29.968 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:17:30.069 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:17:30.085 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:29 smithi031 ceph-mon[16336]: pgmap v568: 37 pgs: 2 activating+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 27 active+clean; 3.2 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 119 MiB/s wr, 1.90k op/s; 4778/155409 objects degraded (3.074%); 28821/155409 objects misplaced (18.545%); 5.4 MiB/s, 85 objects/s recovering 2024-08-19T14:17:30.097 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:30.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:30.099 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:17:30.200 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:17:30.232 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:30.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:30.233 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:17:30.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:29 smithi032 ceph-mon[20004]: pgmap v568: 37 pgs: 2 activating+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 27 active+clean; 3.2 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 119 MiB/s wr, 1.90k op/s; 4778/155409 objects degraded (3.074%); 28821/155409 objects misplaced (18.545%); 5.4 MiB/s, 85 objects/s recovering 2024-08-19T14:17:30.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:29 smithi045 ceph-mon[20143]: pgmap v568: 37 pgs: 2 activating+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 27 active+clean; 3.2 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 119 MiB/s wr, 1.90k op/s; 4778/155409 objects degraded (3.074%); 28821/155409 objects misplaced (18.545%); 5.4 MiB/s, 85 objects/s recovering 2024-08-19T14:17:30.313 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 28 16 58922 58906 131.463 78.9375 0.0108951 0.00760195 2024-08-19T14:17:30.334 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:17:30.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:30.388 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:30.388 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:17:30.490 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:17:30.533 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:30.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:30.534 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:17:30.635 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:17:30.679 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:30.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:30.680 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:17:30.781 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:17:30.812 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:30.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:30.813 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:17:30.891 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:17:30.914 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:17:30.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:30.938 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:30.938 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:17:30.951 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:30 smithi031 ceph-mon[16336]: pgmap v569: 37 pgs: 10 active+recovering+undersized+remapped, 27 active+clean; 3.4 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 122 MiB/s wr, 1.95k op/s; 28649/166152 objects misplaced (17.243%); 28 MiB/s, 443 objects/s recovering 2024-08-19T14:17:30.951 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:30 smithi031 ceph-mon[16336]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 4778/155409 objects degraded (3.074%), 2 pgs degraded) 2024-08-19T14:17:30.951 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:30 smithi031 ceph-mon[16336]: Cluster is now healthy 2024-08-19T14:17:30.952 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:17:30.952 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:17:30.997 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:17:30.997 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:17:31.023 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-19T14:17:31.038 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:17:31.039 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:17:31.039 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:17:31.064 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:31.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:31.065 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:17:31.077 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:17:31.077 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:17:31.121 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:17:31.121 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:17:31.162 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:17:31.162 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:17:31.166 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:17:31.195 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:31.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:31.196 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:17:31.201 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:17:31.201 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:17:31.245 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:17:31.245 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:17:31.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:30 smithi032 ceph-mon[20004]: pgmap v569: 37 pgs: 10 active+recovering+undersized+remapped, 27 active+clean; 3.4 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 122 MiB/s wr, 1.95k op/s; 28649/166152 objects misplaced (17.243%); 28 MiB/s, 443 objects/s recovering 2024-08-19T14:17:31.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:30 smithi032 ceph-mon[20004]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 4778/155409 objects degraded (3.074%), 2 pgs degraded) 2024-08-19T14:17:31.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:30 smithi032 ceph-mon[20004]: Cluster is now healthy 2024-08-19T14:17:31.287 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:17:31.287 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:17:31.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:30 smithi045 ceph-mon[20143]: pgmap v569: 37 pgs: 10 active+recovering+undersized+remapped, 27 active+clean; 3.4 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 122 MiB/s wr, 1.95k op/s; 28649/166152 objects misplaced (17.243%); 28 MiB/s, 443 objects/s recovering 2024-08-19T14:17:31.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:30 smithi045 ceph-mon[20143]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 4778/155409 objects degraded (3.074%), 2 pgs degraded) 2024-08-19T14:17:31.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:30 smithi045 ceph-mon[20143]: Cluster is now healthy 2024-08-19T14:17:31.297 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:17:31.314 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 29 16 59989 59973 129.229 66.6875 0.00512177 0.00773238 2024-08-19T14:17:31.327 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:31.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:31.328 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:17:31.330 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:17:31.330 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:17:31.371 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:17:31.372 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:17:31.409 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:17:31.409 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:17:31.429 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:17:31.472 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:31.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:31.473 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:17:31.537 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:17:31.538 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 20, 2024-08-19T14:17:31.538 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 600, 2024-08-19T14:17:31.538 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [ 2024-08-19T14:17:31.538 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.538 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:241314 2.1 2:8e6fec04:::benchmark_data_smithi045_46415_object118033:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-08-19T14:17:31.538 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:16:40.787553+0000", 2024-08-19T14:17:31.539 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 50.738385977999997, 2024-08-19T14:17:31.539 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.29700766899999997, 2024-08-19T14:17:31.539 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:17:31.539 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:17:31.539 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:17:31.539 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:17:31.539 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:17:31.539 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:17:31.539 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 241314 2024-08-19T14:17:31.539 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.539 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:17:31.539 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.539 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:17:31.539 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.787553+0000", 2024-08-19T14:17:31.540 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:17:31.540 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.540 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.540 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:17:31.540 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.787553+0000", 2024-08-19T14:17:31.540 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:17:31.540 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.540 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.540 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:17:31.540 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.787554+0000", 2024-08-19T14:17:31.540 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5770000000000001e-06 2024-08-19T14:17:31.540 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.540 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.541 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:17:31.541 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.787560+0000", 2024-08-19T14:17:31.541 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.6930000000000004e-06 2024-08-19T14:17:31.541 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.541 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.541 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:17:31.541 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.787574+0000", 2024-08-19T14:17:31.541 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3828e-05 2024-08-19T14:17:31.541 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.541 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.541 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:17:31.541 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.787582+0000", 2024-08-19T14:17:31.542 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.225e-06 2024-08-19T14:17:31.542 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.542 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.542 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:17:31.542 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.788601+0000", 2024-08-19T14:17:31.542 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.001018833 2024-08-19T14:17:31.542 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.542 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.542 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:17:31.542 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.788734+0000", 2024-08-19T14:17:31.542 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013335699999999999 2024-08-19T14:17:31.542 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.542 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.543 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 1,2,8", 2024-08-19T14:17:31.543 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.788881+0000", 2024-08-19T14:17:31.543 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000146644 2024-08-19T14:17:31.543 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.543 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.543 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:17:31.543 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.793613+0000", 2024-08-19T14:17:31.543 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0047319600000000003 2024-08-19T14:17:31.543 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.543 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.543 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:17:31.543 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.795531+0000", 2024-08-19T14:17:31.543 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.001918335 2024-08-19T14:17:31.544 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.544 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.544 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:17:31.544 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.796093+0000", 2024-08-19T14:17:31.544 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00056226599999999996 2024-08-19T14:17:31.544 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.544 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.544 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:17:31.544 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.084346+0000", 2024-08-19T14:17:31.544 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.28825296 2024-08-19T14:17:31.544 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.544 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.544 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:17:31.544 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.084417+0000", 2024-08-19T14:17:31.545 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.0599999999999995e-05 2024-08-19T14:17:31.545 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.545 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.545 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:17:31.545 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.084560+0000", 2024-08-19T14:17:31.545 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00014339100000000001 2024-08-19T14:17:31.545 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:17:31.545 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:17:31.545 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:17:31.545 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.545 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.545 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:241325 2.1 2:8f07de09:::benchmark_data_smithi045_46415_object118044:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-08-19T14:17:31.545 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:16:40.798902+0000", 2024-08-19T14:17:31.546 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 50.727036927999997, 2024-08-19T14:17:31.546 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.29144186999999999, 2024-08-19T14:17:31.546 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:17:31.546 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:17:31.546 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:17:31.546 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:17:31.546 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:17:31.546 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:17:31.546 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 241325 2024-08-19T14:17:31.546 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.546 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:17:31.546 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.546 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:17:31.547 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.798902+0000", 2024-08-19T14:17:31.547 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:17:31.547 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.547 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.547 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:17:31.547 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.798902+0000", 2024-08-19T14:17:31.547 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:17:31.547 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.547 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.547 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:17:31.547 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.798904+0000", 2024-08-19T14:17:31.547 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.1780000000000002e-06 2024-08-19T14:17:31.547 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.547 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.548 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:17:31.548 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.798914+0000", 2024-08-19T14:17:31.548 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.7480000000000005e-06 2024-08-19T14:17:31.548 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.548 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.548 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:17:31.548 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.798933+0000", 2024-08-19T14:17:31.548 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.9026e-05 2024-08-19T14:17:31.548 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.548 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.548 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:17:31.548 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.798942+0000", 2024-08-19T14:17:31.548 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.3200000000000006e-06 2024-08-19T14:17:31.549 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.549 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.549 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:17:31.549 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.799163+0000", 2024-08-19T14:17:31.549 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000221337 2024-08-19T14:17:31.549 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.549 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.549 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:17:31.549 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.799310+0000", 2024-08-19T14:17:31.549 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00014677399999999999 2024-08-19T14:17:31.549 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.549 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.549 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 1,2,8", 2024-08-19T14:17:31.549 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.799452+0000", 2024-08-19T14:17:31.550 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00014219499999999999 2024-08-19T14:17:31.550 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.550 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.550 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:17:31.550 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.804505+0000", 2024-08-19T14:17:31.550 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.005052384 2024-08-19T14:17:31.550 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.550 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.550 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:17:31.550 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.804768+0000", 2024-08-19T14:17:31.550 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00026380700000000002 2024-08-19T14:17:31.550 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.550 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.551 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:17:31.551 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.805302+0000", 2024-08-19T14:17:31.551 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00053321300000000005 2024-08-19T14:17:31.551 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.551 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.551 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:17:31.551 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.090219+0000", 2024-08-19T14:17:31.551 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.284917107 2024-08-19T14:17:31.551 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.551 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.551 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:17:31.551 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.090271+0000", 2024-08-19T14:17:31.551 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.2289e-05 2024-08-19T14:17:31.551 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.552 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.552 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:17:31.552 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.090343+0000", 2024-08-19T14:17:31.552 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.2491999999999996e-05 2024-08-19T14:17:31.552 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:17:31.552 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:17:31.552 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:17:31.552 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.552 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.552 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:241358 2.1 2:8659030e:::benchmark_data_smithi045_46415_object118077:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-08-19T14:17:31.552 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:16:40.843360+0000", 2024-08-19T14:17:31.552 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 50.682578866999997, 2024-08-19T14:17:31.552 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.24721465500000001, 2024-08-19T14:17:31.553 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:17:31.553 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:17:31.553 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:17:31.553 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:17:31.553 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:17:31.553 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:17:31.553 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 241358 2024-08-19T14:17:31.553 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.553 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:17:31.553 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.553 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:17:31.553 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.843360+0000", 2024-08-19T14:17:31.553 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:17:31.554 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.554 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.554 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:17:31.554 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.843360+0000", 2024-08-19T14:17:31.554 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:17:31.554 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.554 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.554 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:17:31.554 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.843362+0000", 2024-08-19T14:17:31.554 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.0820000000000001e-06 2024-08-19T14:17:31.554 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.554 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.554 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:17:31.555 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.843373+0000", 2024-08-19T14:17:31.555 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1600000000000001e-05 2024-08-19T14:17:31.555 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.555 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.555 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:17:31.555 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.843395+0000", 2024-08-19T14:17:31.555 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.1670000000000001e-05 2024-08-19T14:17:31.555 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.555 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.555 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:17:31.555 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.843405+0000", 2024-08-19T14:17:31.555 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.5030000000000003e-06 2024-08-19T14:17:31.555 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.556 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.556 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:17:31.556 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.843474+0000", 2024-08-19T14:17:31.556 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.9620000000000001e-05 2024-08-19T14:17:31.556 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.556 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.556 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:17:31.556 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.843613+0000", 2024-08-19T14:17:31.556 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013863699999999999 2024-08-19T14:17:31.556 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.556 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.556 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 1,2,8", 2024-08-19T14:17:31.556 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.843771+0000", 2024-08-19T14:17:31.557 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000158691 2024-08-19T14:17:31.557 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.557 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.557 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:17:31.557 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.847962+0000", 2024-08-19T14:17:31.557 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0041908989999999997 2024-08-19T14:17:31.557 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.557 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.557 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:17:31.557 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.848335+0000", 2024-08-19T14:17:31.557 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00037213900000000001 2024-08-19T14:17:31.557 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.557 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.558 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:17:31.558 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.849083+0000", 2024-08-19T14:17:31.558 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00074856300000000005 2024-08-19T14:17:31.558 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.558 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.558 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:17:31.558 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.090392+0000", 2024-08-19T14:17:31.558 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.24130854600000001 2024-08-19T14:17:31.558 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.558 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.558 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:17:31.558 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.090453+0000", 2024-08-19T14:17:31.559 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.1635000000000002e-05 2024-08-19T14:17:31.559 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.559 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.559 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:17:31.559 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.090574+0000", 2024-08-19T14:17:31.559 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012107 2024-08-19T14:17:31.559 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:17:31.559 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:17:31.559 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:17:31.559 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.559 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.559 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:241367 2.1 2:870a5187:::benchmark_data_smithi045_46415_object118086:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-08-19T14:17:31.559 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:16:40.854128+0000", 2024-08-19T14:17:31.560 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 50.671810342000001, 2024-08-19T14:17:31.560 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.239748353, 2024-08-19T14:17:31.560 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:17:31.560 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:17:31.560 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:17:31.560 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:17:31.560 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:17:31.560 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:17:31.560 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 241367 2024-08-19T14:17:31.560 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.560 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:17:31.560 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.561 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:17:31.561 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.854128+0000", 2024-08-19T14:17:31.561 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:17:31.561 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.561 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.561 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:17:31.561 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.854128+0000", 2024-08-19T14:17:31.561 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:17:31.561 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.561 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.561 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:17:31.561 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.854130+0000", 2024-08-19T14:17:31.561 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6139999999999999e-06 2024-08-19T14:17:31.562 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.562 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.562 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:17:31.562 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.854138+0000", 2024-08-19T14:17:31.562 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.9370000000000004e-06 2024-08-19T14:17:31.562 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.562 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.562 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:17:31.562 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.854151+0000", 2024-08-19T14:17:31.562 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3338e-05 2024-08-19T14:17:31.562 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.562 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.562 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:17:31.563 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.854158+0000", 2024-08-19T14:17:31.563 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.3350000000000001e-06 2024-08-19T14:17:31.563 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.563 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.563 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:17:31.563 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.854208+0000", 2024-08-19T14:17:31.563 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.9373000000000003e-05 2024-08-19T14:17:31.563 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.563 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.563 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:17:31.563 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.854301+0000", 2024-08-19T14:17:31.563 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.2990000000000002e-05 2024-08-19T14:17:31.563 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.563 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.564 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 1,2,8", 2024-08-19T14:17:31.564 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.854388+0000", 2024-08-19T14:17:31.564 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.7321e-05 2024-08-19T14:17:31.564 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.564 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.564 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:17:31.564 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.860101+0000", 2024-08-19T14:17:31.564 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0057132839999999999 2024-08-19T14:17:31.564 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.564 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.564 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:17:31.564 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.860163+0000", 2024-08-19T14:17:31.564 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.1896000000000005e-05 2024-08-19T14:17:31.565 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.565 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.565 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:17:31.565 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.860183+0000", 2024-08-19T14:17:31.565 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.0041e-05 2024-08-19T14:17:31.565 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.565 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.565 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:17:31.565 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.093641+0000", 2024-08-19T14:17:31.565 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.23345814300000001 2024-08-19T14:17:31.565 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.565 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.565 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:17:31.566 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.093706+0000", 2024-08-19T14:17:31.566 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.4453000000000001e-05 2024-08-19T14:17:31.566 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.566 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.566 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:17:31.566 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.093877+0000", 2024-08-19T14:17:31.566 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00017062800000000001 2024-08-19T14:17:31.566 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:17:31.566 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:17:31.566 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:17:31.566 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.566 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.566 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:241375 2.1 2:8e818ec7:::benchmark_data_smithi045_46415_object118094:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-08-19T14:17:31.567 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:16:40.863736+0000", 2024-08-19T14:17:31.567 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 50.662202137000001, 2024-08-19T14:17:31.567 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.23047135799999999, 2024-08-19T14:17:31.567 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:17:31.567 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:17:31.567 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:17:31.567 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:17:31.567 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:17:31.567 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:17:31.567 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 241375 2024-08-19T14:17:31.567 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.567 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:17:31.567 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.568 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:17:31.568 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.863736+0000", 2024-08-19T14:17:31.568 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:17:31.568 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.568 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.568 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:17:31.568 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.863736+0000", 2024-08-19T14:17:31.568 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:17:31.568 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.568 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.568 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:17:31.568 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.863738+0000", 2024-08-19T14:17:31.568 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5820000000000001e-06 2024-08-19T14:17:31.569 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.569 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.569 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:17:31.569 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.863744+0000", 2024-08-19T14:17:31.569 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.0109999999999997e-06 2024-08-19T14:17:31.569 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.569 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.569 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:17:31.569 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.863759+0000", 2024-08-19T14:17:31.569 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5e-05 2024-08-19T14:17:31.569 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.569 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.569 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:17:31.570 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.863766+0000", 2024-08-19T14:17:31.570 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.311e-06 2024-08-19T14:17:31.570 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.570 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.570 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:17:31.570 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.863821+0000", 2024-08-19T14:17:31.570 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.4954999999999998e-05 2024-08-19T14:17:31.570 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.570 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.570 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:17:31.570 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.863953+0000", 2024-08-19T14:17:31.570 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013166600000000001 2024-08-19T14:17:31.570 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.571 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.571 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 1,2,8", 2024-08-19T14:17:31.571 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.864114+0000", 2024-08-19T14:17:31.571 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000161123 2024-08-19T14:17:31.571 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.571 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.571 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:17:31.571 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.868901+0000", 2024-08-19T14:17:31.571 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0047872490000000004 2024-08-19T14:17:31.572 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.572 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.572 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:17:31.572 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.869524+0000", 2024-08-19T14:17:31.572 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00062289799999999998 2024-08-19T14:17:31.572 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.572 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.572 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:17:31.572 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.870384+0000", 2024-08-19T14:17:31.572 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00085993899999999999 2024-08-19T14:17:31.572 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.572 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.572 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:17:31.573 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.093973+0000", 2024-08-19T14:17:31.573 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.22358860999999999 2024-08-19T14:17:31.573 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.573 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.573 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:17:31.573 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.094038+0000", 2024-08-19T14:17:31.573 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.5326000000000004e-05 2024-08-19T14:17:31.573 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.573 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.573 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:17:31.573 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.094208+0000", 2024-08-19T14:17:31.573 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00016968800000000001 2024-08-19T14:17:31.573 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:17:31.573 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:17:31.574 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:17:31.574 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.574 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.574 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:241390 2.1 2:87dd3bf7:::benchmark_data_smithi045_46415_object118109:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-08-19T14:17:31.574 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:16:40.906380+0000", 2024-08-19T14:17:31.574 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 50.619558187999999, 2024-08-19T14:17:31.574 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.18873680500000001, 2024-08-19T14:17:31.574 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:17:31.574 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:17:31.574 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:17:31.574 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:17:31.574 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:17:31.574 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:17:31.575 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 241390 2024-08-19T14:17:31.575 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.575 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:17:31.575 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.575 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:17:31.575 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.906380+0000", 2024-08-19T14:17:31.575 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:17:31.575 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.575 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.575 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:17:31.575 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.906380+0000", 2024-08-19T14:17:31.575 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:17:31.575 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.576 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.576 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:17:31.576 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.906382+0000", 2024-08-19T14:17:31.576 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.269e-06 2024-08-19T14:17:31.576 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.576 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.576 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:17:31.576 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.906386+0000", 2024-08-19T14:17:31.576 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.5589999999999999e-06 2024-08-19T14:17:31.576 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.576 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.576 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:17:31.577 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.906397+0000", 2024-08-19T14:17:31.577 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1083e-05 2024-08-19T14:17:31.577 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.577 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.577 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:17:31.577 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.906403+0000", 2024-08-19T14:17:31.577 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.4689999999999996e-06 2024-08-19T14:17:31.577 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.577 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.577 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:17:31.577 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.906450+0000", 2024-08-19T14:17:31.577 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.6981000000000003e-05 2024-08-19T14:17:31.577 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.577 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.578 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:17:31.578 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.906534+0000", 2024-08-19T14:17:31.578 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.4714000000000002e-05 2024-08-19T14:17:31.578 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.578 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.578 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 1,2,8", 2024-08-19T14:17:31.578 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.906630+0000", 2024-08-19T14:17:31.578 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.5888000000000001e-05 2024-08-19T14:17:31.578 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.578 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.578 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:17:31.578 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.910150+0000", 2024-08-19T14:17:31.578 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0035197689999999998 2024-08-19T14:17:31.579 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.579 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.579 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:17:31.579 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.910337+0000", 2024-08-19T14:17:31.579 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00018682700000000001 2024-08-19T14:17:31.579 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.579 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.579 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:17:31.579 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.910696+0000", 2024-08-19T14:17:31.579 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000358938 2024-08-19T14:17:31.580 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.580 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.580 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:17:31.580 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.094914+0000", 2024-08-19T14:17:31.580 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.18421832099999999 2024-08-19T14:17:31.580 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.580 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.580 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:17:31.580 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.094966+0000", 2024-08-19T14:17:31.580 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1459999999999999e-05 2024-08-19T14:17:31.580 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.580 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.581 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:17:31.581 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.095117+0000", 2024-08-19T14:17:31.581 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00015152699999999999 2024-08-19T14:17:31.581 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:17:31.581 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:17:31.581 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:17:31.581 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.581 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.581 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25057.0:10160 3.1 3:8327ff00:::benchmark_data_smithi045_59973_object10159:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-08-19T14:17:31.581 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:17:04.638086+0000", 2024-08-19T14:17:31.581 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 26.887853, 2024-08-19T14:17:31.581 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.204737321, 2024-08-19T14:17:31.581 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:17:31.581 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:17:31.581 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:17:31.582 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:17:31.582 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25057", 2024-08-19T14:17:31.582 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:17:31.582 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 10160 2024-08-19T14:17:31.582 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.582 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:17:31.582 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.582 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:17:31.583 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:17:31.583 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.638086+0000", 2024-08-19T14:17:31.584 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:17:31.584 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.584 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.584 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:17:31.584 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.638086+0000", 2024-08-19T14:17:31.584 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:17:31.584 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.584 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.584 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:17:31.584 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.638086+0000", 2024-08-19T14:17:31.584 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.5700000000000001e-07 2024-08-19T14:17:31.584 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.584 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.584 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:17:31.585 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.638129+0000", 2024-08-19T14:17:31.585 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2620000000000002e-05 2024-08-19T14:17:31.585 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.585 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.585 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:17:31.585 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.638140+0000", 2024-08-19T14:17:31.585 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0634e-05 2024-08-19T14:17:31.585 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.585 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.585 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:17:31.585 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.638146+0000", 2024-08-19T14:17:31.585 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.7649999999999996e-06 2024-08-19T14:17:31.585 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.586 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.586 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:17:31.586 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.638197+0000", 2024-08-19T14:17:31.586 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0980999999999999e-05 2024-08-19T14:17:31.586 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.586 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.586 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:17:31.586 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.638528+0000", 2024-08-19T14:17:31.586 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00033104199999999998 2024-08-19T14:17:31.586 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.586 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.586 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 4,11", 2024-08-19T14:17:31.587 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.638676+0000", 2024-08-19T14:17:31.587 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00014772900000000001 2024-08-19T14:17:31.587 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.587 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.587 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:17:31.587 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.639593+0000", 2024-08-19T14:17:31.587 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00091684799999999995 2024-08-19T14:17:31.587 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.587 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.587 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:17:31.587 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.640239+0000", 2024-08-19T14:17:31.587 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00064556699999999999 2024-08-19T14:17:31.587 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.587 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.588 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:17:31.588 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.842749+0000", 2024-08-19T14:17:31.588 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.20251054499999999 2024-08-19T14:17:31.588 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.588 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.588 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:17:31.588 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.842790+0000", 2024-08-19T14:17:31.588 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.0621000000000003e-05 2024-08-19T14:17:31.588 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.588 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.588 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:17:31.588 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.842823+0000", 2024-08-19T14:17:31.588 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.3111999999999997e-05 2024-08-19T14:17:31.589 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:17:31.589 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:17:31.589 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:17:31.589 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.589 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.589 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25057.0:10189 3.1 3:8425fb22:::benchmark_data_smithi045_59973_object10188:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-08-19T14:17:31.589 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:17:04.644533+0000", 2024-08-19T14:17:31.589 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 26.881405497999999, 2024-08-19T14:17:31.589 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.20055124199999999, 2024-08-19T14:17:31.589 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:17:31.589 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:17:31.589 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:17:31.589 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:17:31.589 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25057", 2024-08-19T14:17:31.590 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:17:31.590 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 10189 2024-08-19T14:17:31.590 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.590 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:17:31.590 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.590 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:17:31.590 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.644533+0000", 2024-08-19T14:17:31.590 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:17:31.590 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.590 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.590 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:17:31.590 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.644533+0000", 2024-08-19T14:17:31.590 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:17:31.590 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.591 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.591 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:17:31.591 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.644537+0000", 2024-08-19T14:17:31.591 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.596e-06 2024-08-19T14:17:31.591 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.591 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.591 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:17:31.591 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.644603+0000", 2024-08-19T14:17:31.591 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.6211999999999996e-05 2024-08-19T14:17:31.591 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.591 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.591 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:17:31.591 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.644614+0000", 2024-08-19T14:17:31.591 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1048e-05 2024-08-19T14:17:31.592 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.592 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.592 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:17:31.592 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.644619+0000", 2024-08-19T14:17:31.592 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.9960000000000001e-06 2024-08-19T14:17:31.592 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.592 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.592 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:17:31.592 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.644670+0000", 2024-08-19T14:17:31.592 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1533000000000003e-05 2024-08-19T14:17:31.592 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.592 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.592 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:17:31.592 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.644802+0000", 2024-08-19T14:17:31.593 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013154799999999999 2024-08-19T14:17:31.593 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.593 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.593 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 4,11", 2024-08-19T14:17:31.593 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.644967+0000", 2024-08-19T14:17:31.593 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00016497300000000001 2024-08-19T14:17:31.593 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.593 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.593 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:17:31.593 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.645646+0000", 2024-08-19T14:17:31.593 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00067895300000000004 2024-08-19T14:17:31.593 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.593 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.594 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:17:31.594 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.646472+0000", 2024-08-19T14:17:31.594 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00082639300000000003 2024-08-19T14:17:31.594 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.594 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.594 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:17:31.594 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.844988+0000", 2024-08-19T14:17:31.594 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.19851542999999999 2024-08-19T14:17:31.594 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.594 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.594 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:17:31.594 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.845043+0000", 2024-08-19T14:17:31.594 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.5642e-05 2024-08-19T14:17:31.595 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.595 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.595 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:17:31.595 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.845084+0000", 2024-08-19T14:17:31.595 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.0917999999999997e-05 2024-08-19T14:17:31.595 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:17:31.595 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:17:31.595 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:17:31.595 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.595 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.595 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25057.0:10212 3.1 3:84b15d00:::benchmark_data_smithi045_59973_object10211:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-08-19T14:17:31.595 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:17:04.650137+0000", 2024-08-19T14:17:31.595 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 26.875801393, 2024-08-19T14:17:31.595 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.19513108600000001, 2024-08-19T14:17:31.596 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:17:31.596 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:17:31.596 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:17:31.596 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:17:31.596 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25057", 2024-08-19T14:17:31.596 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:17:31.596 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 10212 2024-08-19T14:17:31.596 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.596 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:17:31.596 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.596 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:17:31.596 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.650137+0000", 2024-08-19T14:17:31.596 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:17:31.596 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.597 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.597 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:17:31.597 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.650137+0000", 2024-08-19T14:17:31.597 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:17:31.597 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.597 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.597 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:17:31.597 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.650138+0000", 2024-08-19T14:17:31.597 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.5300000000000003e-07 2024-08-19T14:17:31.597 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.597 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.597 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:17:31.598 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.650209+0000", 2024-08-19T14:17:31.598 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.1126999999999998e-05 2024-08-19T14:17:31.598 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.598 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.598 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:17:31.598 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.650222+0000", 2024-08-19T14:17:31.598 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.33e-05 2024-08-19T14:17:31.598 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.598 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.598 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:17:31.598 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.650228+0000", 2024-08-19T14:17:31.598 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.3159999999999996e-06 2024-08-19T14:17:31.598 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.598 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.599 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:17:31.599 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.650294+0000", 2024-08-19T14:17:31.599 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.6221000000000002e-05 2024-08-19T14:17:31.599 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.599 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.599 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:17:31.599 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.650410+0000", 2024-08-19T14:17:31.599 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000115877 2024-08-19T14:17:31.599 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.599 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.599 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 4,11", 2024-08-19T14:17:31.599 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.650508+0000", 2024-08-19T14:17:31.599 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.8084000000000005e-05 2024-08-19T14:17:31.600 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.600 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.600 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:17:31.600 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.651008+0000", 2024-08-19T14:17:31.600 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00049981199999999998 2024-08-19T14:17:31.600 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.600 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.600 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:17:31.600 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.651849+0000", 2024-08-19T14:17:31.600 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00084125599999999997 2024-08-19T14:17:31.600 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.600 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.600 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:17:31.601 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.845139+0000", 2024-08-19T14:17:31.601 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.193290502 2024-08-19T14:17:31.601 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.601 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.601 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:17:31.601 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.845217+0000", 2024-08-19T14:17:31.601 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.7763999999999996e-05 2024-08-19T14:17:31.601 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.601 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.601 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:17:31.601 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.845268+0000", 2024-08-19T14:17:31.601 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0973999999999997e-05 2024-08-19T14:17:31.601 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:17:31.601 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:17:31.602 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:17:31.602 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.602 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.602 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25057.0:10221 3.1 3:8da3ca82:::benchmark_data_smithi045_59973_object10220:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-08-19T14:17:31.602 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:17:04.652160+0000", 2024-08-19T14:17:31.602 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 26.873778532999999, 2024-08-19T14:17:31.602 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.19366444299999999, 2024-08-19T14:17:31.602 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:17:31.602 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:17:31.602 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:17:31.602 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:17:31.602 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25057", 2024-08-19T14:17:31.603 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:17:31.603 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 10221 2024-08-19T14:17:31.603 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.603 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:17:31.603 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.603 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:17:31.603 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.652160+0000", 2024-08-19T14:17:31.603 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:17:31.603 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.603 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.603 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:17:31.603 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.652160+0000", 2024-08-19T14:17:31.603 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:17:31.604 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.604 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.604 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:17:31.604 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.652161+0000", 2024-08-19T14:17:31.604 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.71e-07 2024-08-19T14:17:31.604 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.604 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.604 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:17:31.604 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.652196+0000", 2024-08-19T14:17:31.604 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.4910000000000003e-05 2024-08-19T14:17:31.604 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.604 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.604 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:17:31.604 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.652208+0000", 2024-08-19T14:17:31.605 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1772e-05 2024-08-19T14:17:31.605 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.605 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.605 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:17:31.605 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.652213+0000", 2024-08-19T14:17:31.605 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.2299999999999999e-06 2024-08-19T14:17:31.605 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.605 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.605 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:17:31.606 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.652264+0000", 2024-08-19T14:17:31.606 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1017000000000003e-05 2024-08-19T14:17:31.606 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.606 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.606 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:17:31.606 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.652424+0000", 2024-08-19T14:17:31.606 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00015976799999999999 2024-08-19T14:17:31.606 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.606 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.606 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 4,11", 2024-08-19T14:17:31.606 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.652564+0000", 2024-08-19T14:17:31.606 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00014018799999999999 2024-08-19T14:17:31.606 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.607 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.607 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:17:31.607 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.653656+0000", 2024-08-19T14:17:31.607 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0010920389999999999 2024-08-19T14:17:31.607 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.607 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.607 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:17:31.607 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.653678+0000", 2024-08-19T14:17:31.607 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.2351999999999999e-05 2024-08-19T14:17:31.607 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.607 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.607 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:17:31.607 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.845727+0000", 2024-08-19T14:17:31.608 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.19204908700000001 2024-08-19T14:17:31.608 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.608 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.608 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:17:31.608 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.845778+0000", 2024-08-19T14:17:31.608 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0445000000000002e-05 2024-08-19T14:17:31.608 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.608 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.608 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:17:31.608 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.845824+0000", 2024-08-19T14:17:31.608 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.6764000000000002e-05 2024-08-19T14:17:31.608 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:17:31.608 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:17:31.608 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:17:31.609 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.609 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.609 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25057.0:10222 3.1 3:8a9228ce:::benchmark_data_smithi045_59973_object10221:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-08-19T14:17:31.609 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:17:04.652316+0000", 2024-08-19T14:17:31.609 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 26.873622701999999, 2024-08-19T14:17:31.609 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.193655784, 2024-08-19T14:17:31.609 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:17:31.609 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:17:31.609 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:17:31.609 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:17:31.609 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25057", 2024-08-19T14:17:31.610 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:17:31.610 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 10222 2024-08-19T14:17:31.610 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.610 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:17:31.610 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.610 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:17:31.610 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.652316+0000", 2024-08-19T14:17:31.610 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:17:31.610 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.610 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.610 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:17:31.610 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.652316+0000", 2024-08-19T14:17:31.610 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:17:31.611 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.611 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.611 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:17:31.611 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.652316+0000", 2024-08-19T14:17:31.611 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.1600000000000001e-07 2024-08-19T14:17:31.611 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.611 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.611 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:17:31.611 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.652373+0000", 2024-08-19T14:17:31.611 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.6818999999999999e-05 2024-08-19T14:17:31.611 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.611 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.612 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:17:31.612 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.652385+0000", 2024-08-19T14:17:31.612 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.13e-05 2024-08-19T14:17:31.612 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.612 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.612 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:17:31.612 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.652391+0000", 2024-08-19T14:17:31.612 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.8279999999999997e-06 2024-08-19T14:17:31.612 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.612 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.612 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:17:31.612 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.653047+0000", 2024-08-19T14:17:31.612 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00065595200000000001 2024-08-19T14:17:31.612 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.612 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.613 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:17:31.613 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.653155+0000", 2024-08-19T14:17:31.613 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000107642 2024-08-19T14:17:31.613 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.613 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.613 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 4,11", 2024-08-19T14:17:31.613 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.653295+0000", 2024-08-19T14:17:31.613 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000139968 2024-08-19T14:17:31.613 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.613 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.613 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:17:31.613 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.653783+0000", 2024-08-19T14:17:31.613 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000487601 2024-08-19T14:17:31.614 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.614 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.614 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:17:31.614 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.654258+0000", 2024-08-19T14:17:31.614 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00047582100000000002 2024-08-19T14:17:31.614 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.614 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.614 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:17:31.614 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.845872+0000", 2024-08-19T14:17:31.614 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.19161402 2024-08-19T14:17:31.614 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.614 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.614 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:17:31.615 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.845930+0000", 2024-08-19T14:17:31.615 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.8069000000000003e-05 2024-08-19T14:17:31.615 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.615 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.615 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:17:31.615 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.845972+0000", 2024-08-19T14:17:31.615 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.1148e-05 2024-08-19T14:17:31.615 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:17:31.615 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:17:31.615 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:17:31.615 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.615 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.616 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25057.0:10230 3.1 3:8b2bcb46:::benchmark_data_smithi045_59973_object10229:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-08-19T14:17:31.616 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:17:04.655319+0000", 2024-08-19T14:17:31.616 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 26.870619871999999, 2024-08-19T14:17:31.616 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.19081735699999999, 2024-08-19T14:17:31.616 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:17:31.616 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:17:31.616 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:17:31.616 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:17:31.616 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25057", 2024-08-19T14:17:31.616 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:17:31.616 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 10230 2024-08-19T14:17:31.616 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.617 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:17:31.617 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.617 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:17:31.617 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.655319+0000", 2024-08-19T14:17:31.617 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:17:31.617 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.617 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.617 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:17:31.617 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.655319+0000", 2024-08-19T14:17:31.617 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:17:31.617 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.617 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.618 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:17:31.618 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.655320+0000", 2024-08-19T14:17:31.618 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.5499999999999996e-07 2024-08-19T14:17:31.618 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.618 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.618 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:17:31.618 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.655361+0000", 2024-08-19T14:17:31.618 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.0976000000000002e-05 2024-08-19T14:17:31.618 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.618 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.618 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:17:31.618 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.655371+0000", 2024-08-19T14:17:31.618 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0638e-05 2024-08-19T14:17:31.619 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.619 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.619 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:17:31.619 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.655376+0000", 2024-08-19T14:17:31.619 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.2599999999999996e-06 2024-08-19T14:17:31.619 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.619 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.619 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:17:31.619 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.655413+0000", 2024-08-19T14:17:31.619 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.6693000000000003e-05 2024-08-19T14:17:31.619 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.619 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.619 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:17:31.619 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.655488+0000", 2024-08-19T14:17:31.620 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.4605999999999997e-05 2024-08-19T14:17:31.620 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.620 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.620 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 4,11", 2024-08-19T14:17:31.620 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.655563+0000", 2024-08-19T14:17:31.620 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.5087999999999997e-05 2024-08-19T14:17:31.620 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.620 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.620 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:17:31.620 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.655991+0000", 2024-08-19T14:17:31.620 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00042848700000000002 2024-08-19T14:17:31.620 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.620 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.620 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:17:31.621 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.656601+0000", 2024-08-19T14:17:31.621 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00060932800000000002 2024-08-19T14:17:31.621 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.621 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.621 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:17:31.621 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.846037+0000", 2024-08-19T14:17:31.621 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.18943663199999999 2024-08-19T14:17:31.621 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.621 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.621 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:17:31.621 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.846095+0000", 2024-08-19T14:17:31.621 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.7615999999999999e-05 2024-08-19T14:17:31.621 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.622 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.622 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:17:31.622 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.846136+0000", 2024-08-19T14:17:31.622 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.1078000000000002e-05 2024-08-19T14:17:31.622 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:17:31.622 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:17:31.622 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:17:31.622 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.622 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.622 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25057.0:10242 3.1 3:86d5ceb1:::benchmark_data_smithi045_59973_object10241:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-08-19T14:17:31.622 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:17:04.659944+0000", 2024-08-19T14:17:31.622 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 26.865994835999999, 2024-08-19T14:17:31.622 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.186369811, 2024-08-19T14:17:31.623 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:17:31.623 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:17:31.623 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:17:31.623 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:17:31.623 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25057", 2024-08-19T14:17:31.623 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:17:31.623 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 10242 2024-08-19T14:17:31.623 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.623 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:17:31.623 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.623 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:17:31.623 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.659944+0000", 2024-08-19T14:17:31.623 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:17:31.624 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.624 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.624 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:17:31.624 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.659944+0000", 2024-08-19T14:17:31.624 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:17:31.624 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.624 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.624 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:17:31.624 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.659944+0000", 2024-08-19T14:17:31.624 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.9400000000000005e-07 2024-08-19T14:17:31.624 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.624 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.624 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:17:31.624 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.659981+0000", 2024-08-19T14:17:31.625 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7098000000000003e-05 2024-08-19T14:17:31.625 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.625 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.625 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:17:31.625 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.659993+0000", 2024-08-19T14:17:31.625 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1484999999999999e-05 2024-08-19T14:17:31.625 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.625 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.625 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:17:31.625 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.659998+0000", 2024-08-19T14:17:31.625 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.5650000000000003e-06 2024-08-19T14:17:31.625 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.625 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.625 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:17:31.626 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.660039+0000", 2024-08-19T14:17:31.626 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.1556e-05 2024-08-19T14:17:31.626 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.626 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.626 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:17:31.626 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.660133+0000", 2024-08-19T14:17:31.626 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.4247999999999996e-05 2024-08-19T14:17:31.626 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.626 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.626 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 4,11", 2024-08-19T14:17:31.628 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.660224+0000", 2024-08-19T14:17:31.628 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.0552999999999999e-05 2024-08-19T14:17:31.628 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.628 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.628 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:17:31.628 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.660771+0000", 2024-08-19T14:17:31.628 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00054699600000000001 2024-08-19T14:17:31.628 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.628 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.628 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:17:31.628 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.661221+0000", 2024-08-19T14:17:31.628 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00045049599999999999 2024-08-19T14:17:31.629 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.629 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.629 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:17:31.629 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.846189+0000", 2024-08-19T14:17:31.629 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.18496760900000001 2024-08-19T14:17:31.629 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.629 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.629 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:17:31.629 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.846250+0000", 2024-08-19T14:17:31.629 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.1428999999999997e-05 2024-08-19T14:17:31.629 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.629 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.630 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:17:31.630 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.846314+0000", 2024-08-19T14:17:31.630 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.3082000000000004e-05 2024-08-19T14:17:31.630 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:17:31.630 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:17:31.630 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:17:31.630 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.630 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.630 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25057.0:10258 3.1 3:8a683978:::benchmark_data_smithi045_59973_object10257:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-08-19T14:17:31.630 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:17:04.670765+0000", 2024-08-19T14:17:31.630 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 26.855173689000001, 2024-08-19T14:17:31.630 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.17568293600000001, 2024-08-19T14:17:31.630 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:17:31.631 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:17:31.631 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:17:31.631 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:17:31.631 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25057", 2024-08-19T14:17:31.631 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:17:31.631 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 10258 2024-08-19T14:17:31.631 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.631 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:17:31.631 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.631 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:17:31.631 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.670765+0000", 2024-08-19T14:17:31.631 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:17:31.631 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.632 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.632 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:17:31.632 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.670765+0000", 2024-08-19T14:17:31.632 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:17:31.632 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.632 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.632 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:17:31.632 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.670766+0000", 2024-08-19T14:17:31.632 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0890000000000001e-06 2024-08-19T14:17:31.632 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.632 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.632 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:17:31.632 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.670795+0000", 2024-08-19T14:17:31.633 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.8765999999999999e-05 2024-08-19T14:17:31.633 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.633 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.633 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:17:31.633 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.670806+0000", 2024-08-19T14:17:31.633 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0865e-05 2024-08-19T14:17:31.633 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.633 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.633 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:17:31.633 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.670812+0000", 2024-08-19T14:17:31.633 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.0260000000000004e-06 2024-08-19T14:17:31.633 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.633 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.634 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:17:31.634 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.670866+0000", 2024-08-19T14:17:31.634 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.4168e-05 2024-08-19T14:17:31.634 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.634 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.634 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:17:31.634 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.671022+0000", 2024-08-19T14:17:31.634 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00015581899999999999 2024-08-19T14:17:31.634 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.634 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.634 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 4,11", 2024-08-19T14:17:31.634 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.671177+0000", 2024-08-19T14:17:31.634 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000154921 2024-08-19T14:17:31.635 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.635 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.635 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:17:31.635 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.671706+0000", 2024-08-19T14:17:31.635 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00052993599999999995 2024-08-19T14:17:31.635 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.635 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.635 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:17:31.635 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.671994+0000", 2024-08-19T14:17:31.635 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00028720800000000001 2024-08-19T14:17:31.635 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.635 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.635 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:17:31.636 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.846381+0000", 2024-08-19T14:17:31.636 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.174387396 2024-08-19T14:17:31.636 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.636 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.636 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:17:31.636 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.846413+0000", 2024-08-19T14:17:31.636 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.1656000000000003e-05 2024-08-19T14:17:31.636 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.636 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.636 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:17:31.636 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:04.846448+0000", 2024-08-19T14:17:31.636 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.5086000000000003e-05 2024-08-19T14:17:31.636 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:17:31.637 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:17:31.637 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:17:31.637 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.637 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.637 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25057.0:47774 3.11 3:8e95e964:::benchmark_data_smithi045_59973_object47773:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e234)", 2024-08-19T14:17:31.637 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:17:18.724217+0000", 2024-08-19T14:17:31.637 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 12.801721311, 2024-08-19T14:17:31.637 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0342696659999999, 2024-08-19T14:17:31.637 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:17:31.637 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:17:31.637 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:17:31.637 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:17:31.637 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25057", 2024-08-19T14:17:31.638 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:17:31.638 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 47774 2024-08-19T14:17:31.638 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.638 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:17:31.638 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.638 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:17:31.638 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724217+0000", 2024-08-19T14:17:31.638 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:17:31.638 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.638 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.638 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:17:31.638 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724217+0000", 2024-08-19T14:17:31.639 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:17:31.639 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.639 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.639 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:17:31.639 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724218+0000", 2024-08-19T14:17:31.639 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.4499999999999997e-07 2024-08-19T14:17:31.639 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.639 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.639 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:17:31.639 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724254+0000", 2024-08-19T14:17:31.639 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.5914000000000002e-05 2024-08-19T14:17:31.639 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.639 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.640 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:17:31.640 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724263+0000", 2024-08-19T14:17:31.640 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.4290000000000007e-06 2024-08-19T14:17:31.640 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.640 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.640 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:17:31.640 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724270+0000", 2024-08-19T14:17:31.640 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.7959999999999998e-06 2024-08-19T14:17:31.640 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.640 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.640 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:17:31.640 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.774143+0000", 2024-08-19T14:17:31.640 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.049872861999999997 2024-08-19T14:17:31.640 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.640 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.641 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:17:31.641 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.774151+0000", 2024-08-19T14:17:31.641 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.6489999999999997e-06 2024-08-19T14:17:31.641 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.641 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.641 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:17:31.641 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.758417+0000", 2024-08-19T14:17:31.641 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.98426575800000005 2024-08-19T14:17:31.641 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.641 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.641 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:17:31.641 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.758487+0000", 2024-08-19T14:17:31.641 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.9713000000000002e-05 2024-08-19T14:17:31.642 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:17:31.642 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:17:31.642 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:17:31.642 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.642 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.642 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25057.0:47788 3.11 3:8e23edc0:::benchmark_data_smithi045_59973_object47787:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e234)", 2024-08-19T14:17:31.642 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:17:18.724282+0000", 2024-08-19T14:17:31.642 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 12.801656466000001, 2024-08-19T14:17:31.642 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0343071210000001, 2024-08-19T14:17:31.642 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:17:31.642 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:17:31.642 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:17:31.642 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:17:31.643 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25057", 2024-08-19T14:17:31.643 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:17:31.643 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 47788 2024-08-19T14:17:31.643 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.643 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:17:31.643 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.643 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:17:31.643 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724282+0000", 2024-08-19T14:17:31.643 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:17:31.643 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.643 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.643 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:17:31.643 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724282+0000", 2024-08-19T14:17:31.644 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:17:31.644 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.644 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.644 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:17:31.644 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724283+0000", 2024-08-19T14:17:31.644 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.6100000000000001e-07 2024-08-19T14:17:31.644 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.644 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.644 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:17:31.644 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724318+0000", 2024-08-19T14:17:31.644 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.578e-05 2024-08-19T14:17:31.644 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.644 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.644 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:17:31.645 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724326+0000", 2024-08-19T14:17:31.645 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.4170000000000003e-06 2024-08-19T14:17:31.645 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.645 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.645 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:17:31.645 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724331+0000", 2024-08-19T14:17:31.645 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.5820000000000001e-06 2024-08-19T14:17:31.645 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.645 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.645 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:17:31.645 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.774172+0000", 2024-08-19T14:17:31.645 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.049840256999999999 2024-08-19T14:17:31.645 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.645 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.646 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:17:31.646 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.774177+0000", 2024-08-19T14:17:31.646 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.5160000000000002e-06 2024-08-19T14:17:31.646 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.646 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.646 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:17:31.646 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.758533+0000", 2024-08-19T14:17:31.646 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.98435635099999996 2024-08-19T14:17:31.646 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.646 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.646 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:17:31.646 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.758589+0000", 2024-08-19T14:17:31.646 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.5757000000000002e-05 2024-08-19T14:17:31.647 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:17:31.647 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:17:31.647 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:17:31.647 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.647 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.647 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25057.0:47814 3.11 3:8918a6df:::benchmark_data_smithi045_59973_object47813:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e234)", 2024-08-19T14:17:31.647 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:17:18.724406+0000", 2024-08-19T14:17:31.647 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 12.801533027, 2024-08-19T14:17:31.647 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0342602160000001, 2024-08-19T14:17:31.647 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:17:31.647 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:17:31.647 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:17:31.647 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:17:31.647 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25057", 2024-08-19T14:17:31.648 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:17:31.648 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 47814 2024-08-19T14:17:31.648 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.648 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:17:31.648 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.648 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:17:31.648 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724406+0000", 2024-08-19T14:17:31.648 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:17:31.648 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.648 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.648 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:17:31.648 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724406+0000", 2024-08-19T14:17:31.648 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:17:31.649 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.649 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.649 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:17:31.649 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724406+0000", 2024-08-19T14:17:31.649 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.0200000000000003e-07 2024-08-19T14:17:31.649 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.649 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.649 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:17:31.649 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724440+0000", 2024-08-19T14:17:31.649 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.3689000000000002e-05 2024-08-19T14:17:31.649 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.649 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.649 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:17:31.650 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724449+0000", 2024-08-19T14:17:31.650 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.0469999999999999e-06 2024-08-19T14:17:31.650 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.650 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.650 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:17:31.650 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724455+0000", 2024-08-19T14:17:31.650 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.1190000000000002e-06 2024-08-19T14:17:31.650 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.650 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.650 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:17:31.650 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.774196+0000", 2024-08-19T14:17:31.650 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.049740924999999998 2024-08-19T14:17:31.651 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.651 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.651 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:17:31.651 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.774201+0000", 2024-08-19T14:17:31.651 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.3090000000000003e-06 2024-08-19T14:17:31.651 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.651 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.651 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:17:31.651 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.758615+0000", 2024-08-19T14:17:31.651 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.98441362200000004 2024-08-19T14:17:31.651 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.651 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.651 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:17:31.652 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.758666+0000", 2024-08-19T14:17:31.652 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1103000000000002e-05 2024-08-19T14:17:31.652 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:17:31.652 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:17:31.652 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:17:31.652 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.652 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.652 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25057.0:47832 3.11 3:8d79f59c:::benchmark_data_smithi045_59973_object47831:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e234)", 2024-08-19T14:17:31.652 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:17:18.724649+0000", 2024-08-19T14:17:31.652 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 12.801289812, 2024-08-19T14:17:31.652 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0340940759999999, 2024-08-19T14:17:31.652 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:17:31.652 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:17:31.653 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:17:31.653 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:17:31.653 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25057", 2024-08-19T14:17:31.653 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:17:31.653 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 47832 2024-08-19T14:17:31.653 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.653 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:17:31.653 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.653 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:17:31.653 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724649+0000", 2024-08-19T14:17:31.653 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:17:31.653 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.653 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.653 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:17:31.654 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724649+0000", 2024-08-19T14:17:31.654 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:17:31.654 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.654 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.654 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:17:31.654 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724649+0000", 2024-08-19T14:17:31.654 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0399999999999996e-07 2024-08-19T14:17:31.654 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.654 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.654 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:17:31.654 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724680+0000", 2024-08-19T14:17:31.654 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.0477e-05 2024-08-19T14:17:31.654 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.655 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.655 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:17:31.655 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724688+0000", 2024-08-19T14:17:31.655 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.7509999999999997e-06 2024-08-19T14:17:31.655 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.655 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.655 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:17:31.655 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724694+0000", 2024-08-19T14:17:31.655 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.7679999999999997e-06 2024-08-19T14:17:31.655 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.655 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.655 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:17:31.655 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.774220+0000", 2024-08-19T14:17:31.656 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.049526175999999998 2024-08-19T14:17:31.656 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.656 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.656 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:17:31.656 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.774226+0000", 2024-08-19T14:17:31.656 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.2820000000000004e-06 2024-08-19T14:17:31.656 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.656 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.656 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:17:31.656 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.758693+0000", 2024-08-19T14:17:31.656 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.98446715699999998 2024-08-19T14:17:31.656 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.656 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.656 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:17:31.657 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.758743+0000", 2024-08-19T14:17:31.657 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.9960999999999998e-05 2024-08-19T14:17:31.657 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:17:31.657 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:17:31.657 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:17:31.657 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.657 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.657 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25057.0:47838 3.16 3:68f5225c:::benchmark_data_smithi045_59973_object47837:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e234)", 2024-08-19T14:17:31.657 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:17:18.777436+0000", 2024-08-19T14:17:31.657 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 12.748502243000001, 2024-08-19T14:17:31.657 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.98807081600000002, 2024-08-19T14:17:31.657 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:17:31.657 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:17:31.657 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:17:31.658 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:17:31.658 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25057", 2024-08-19T14:17:31.658 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:17:31.658 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 47838 2024-08-19T14:17:31.658 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.658 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:17:31.658 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.658 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:17:31.658 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.777436+0000", 2024-08-19T14:17:31.658 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:17:31.658 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.658 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.659 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:17:31.659 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.777436+0000", 2024-08-19T14:17:31.659 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:17:31.659 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.659 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.659 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:17:31.659 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.777438+0000", 2024-08-19T14:17:31.659 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.7489999999999999e-06 2024-08-19T14:17:31.659 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.659 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.659 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:17:31.659 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.777473+0000", 2024-08-19T14:17:31.659 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.5284999999999998e-05 2024-08-19T14:17:31.659 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.660 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.660 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:17:31.660 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.777492+0000", 2024-08-19T14:17:31.660 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.8853999999999999e-05 2024-08-19T14:17:31.660 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.660 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.660 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:17:31.660 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.777500+0000", 2024-08-19T14:17:31.660 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.9759999999999995e-06 2024-08-19T14:17:31.660 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.660 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.660 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:17:31.660 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.778853+0000", 2024-08-19T14:17:31.661 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0013526460000000001 2024-08-19T14:17:31.661 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.661 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.661 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:17:31.661 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.778867+0000", 2024-08-19T14:17:31.661 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4448e-05 2024-08-19T14:17:31.661 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.661 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.661 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:17:31.661 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.759774+0000", 2024-08-19T14:17:31.661 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.98090639700000004 2024-08-19T14:17:31.661 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.661 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.661 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:17:31.662 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.759902+0000", 2024-08-19T14:17:31.662 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000128128 2024-08-19T14:17:31.662 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.662 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.662 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 1,4", 2024-08-19T14:17:31.662 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.760062+0000", 2024-08-19T14:17:31.662 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000159956 2024-08-19T14:17:31.662 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.662 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.662 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:17:31.662 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.762325+0000", 2024-08-19T14:17:31.662 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0022629880000000001 2024-08-19T14:17:31.662 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.663 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.663 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:17:31.663 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.763087+0000", 2024-08-19T14:17:31.663 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00076208899999999997 2024-08-19T14:17:31.663 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.663 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.663 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:17:31.663 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.765400+0000", 2024-08-19T14:17:31.663 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0023133120000000001 2024-08-19T14:17:31.663 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.663 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.663 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:17:31.663 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.765462+0000", 2024-08-19T14:17:31.664 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.2048e-05 2024-08-19T14:17:31.664 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.664 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.664 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:17:31.664 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.765507+0000", 2024-08-19T14:17:31.664 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.494e-05 2024-08-19T14:17:31.664 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:17:31.664 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:17:31.664 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:17:31.664 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.664 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.664 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25057.0:47845 3.16 3:6c285014:::benchmark_data_smithi045_59973_object47844:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e234)", 2024-08-19T14:17:31.664 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:17:18.799489+0000", 2024-08-19T14:17:31.664 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 12.726449939, 2024-08-19T14:17:31.665 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.96674353099999999, 2024-08-19T14:17:31.665 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:17:31.665 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:17:31.665 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:17:31.665 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:17:31.665 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25057", 2024-08-19T14:17:31.665 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:17:31.665 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 47845 2024-08-19T14:17:31.665 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.665 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:17:31.665 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.665 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:17:31.665 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.799489+0000", 2024-08-19T14:17:31.665 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:17:31.666 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.666 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.666 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:17:31.666 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.799489+0000", 2024-08-19T14:17:31.666 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:17:31.666 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.666 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.666 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:17:31.666 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.799490+0000", 2024-08-19T14:17:31.666 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.779e-06 2024-08-19T14:17:31.666 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.666 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.666 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:17:31.666 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.799530+0000", 2024-08-19T14:17:31.667 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.9555000000000003e-05 2024-08-19T14:17:31.667 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.667 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.667 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:17:31.667 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.799544+0000", 2024-08-19T14:17:31.667 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4336e-05 2024-08-19T14:17:31.667 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.667 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.667 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:17:31.667 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.799554+0000", 2024-08-19T14:17:31.667 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.5580000000000006e-06 2024-08-19T14:17:31.667 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.667 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.667 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:17:31.668 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.799613+0000", 2024-08-19T14:17:31.668 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.8995e-05 2024-08-19T14:17:31.668 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.668 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.668 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:17:31.668 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.799625+0000", 2024-08-19T14:17:31.668 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2088e-05 2024-08-19T14:17:31.668 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.668 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.668 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:17:31.668 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.760956+0000", 2024-08-19T14:17:31.668 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.96133111299999996 2024-08-19T14:17:31.668 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.668 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.669 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:17:31.669 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.761072+0000", 2024-08-19T14:17:31.669 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000116173 2024-08-19T14:17:31.669 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.669 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.669 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 1,4", 2024-08-19T14:17:31.669 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.761217+0000", 2024-08-19T14:17:31.669 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000144308 2024-08-19T14:17:31.669 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.669 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.669 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:17:31.669 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.763046+0000", 2024-08-19T14:17:31.669 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00182978 2024-08-19T14:17:31.669 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.669 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.670 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:17:31.670 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.763278+0000", 2024-08-19T14:17:31.670 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00023194700000000001 2024-08-19T14:17:31.670 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.670 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.670 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:17:31.670 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.766122+0000", 2024-08-19T14:17:31.670 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0028441439999999998 2024-08-19T14:17:31.670 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.670 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.670 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:17:31.670 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.766190+0000", 2024-08-19T14:17:31.670 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.7595999999999995e-05 2024-08-19T14:17:31.670 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:17:31.671 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:17:31.671 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:17:31.671 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.766232+0000", 2024-08-19T14:17:31.671 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2159000000000001e-05 2024-08-19T14:17:31.671 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:17:31.671 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:17:31.671 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:17:31.671 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:17:31.671 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:17:31.671 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:17:31.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:31.673 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:31.673 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:17:31.774 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:17:31.801 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:31.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:31.802 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:17:31.876 INFO:teuthology.orchestra.run.smithi031.stderr:set pool 3 pg_num to 36 2024-08-19T14:17:31.903 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:17:31.934 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:31.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:31.935 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:17:32.026 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:31 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2783244056' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:17:32.026 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:31 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/183862380' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "36"}]: dispatch 2024-08-19T14:17:32.026 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:31 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "36"}]: dispatch 2024-08-19T14:17:32.026 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:31 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:17:32.026 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:31 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:17:32.036 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:17:32.073 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:32.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:32.074 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:17:32.175 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:17:32.205 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:32.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:32.206 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:17:32.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:31 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2783244056' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:17:32.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:31 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/183862380' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "36"}]: dispatch 2024-08-19T14:17:32.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:31 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "36"}]: dispatch 2024-08-19T14:17:32.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:31 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:17:32.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:31 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:17:32.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:31 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2783244056' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:17:32.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:31 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/183862380' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "36"}]: dispatch 2024-08-19T14:17:32.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:31 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "36"}]: dispatch 2024-08-19T14:17:32.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:31 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:17:32.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:31 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:17:32.307 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:17:32.313 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 30 16 61221 61205 127.488 77 0.00751267 0.00783896 2024-08-19T14:17:32.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:32.341 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:32.341 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:17:32.443 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:17:32.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:32.488 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:32.488 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:17:32.589 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:17:32.631 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:32.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:32.633 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:17:32.734 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:17:32.773 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:32.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:32.774 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:17:32.875 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:17:32.902 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:32.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:32.903 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:17:33.004 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:17:33.032 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:33.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:33.033 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:17:33.134 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:17:33.162 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:33.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:33.164 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:17:33.259 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:32 smithi032 ceph-mon[20004]: pgmap v570: 37 pgs: 10 active+recovering+undersized+remapped, 27 active+clean; 3.5 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 92 MiB/s wr, 1.48k op/s; 27649/172824 objects misplaced (15.998%); 27 MiB/s, 434 objects/s recovering 2024-08-19T14:17:33.260 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:32 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "36"}]': finished 2024-08-19T14:17:33.260 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:32 smithi032 ceph-mon[20004]: osdmap e242: 12 total, 12 up, 9 in 2024-08-19T14:17:33.265 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:17:33.294 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:33.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:33.295 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:17:33.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:32 smithi045 ceph-mon[20143]: pgmap v570: 37 pgs: 10 active+recovering+undersized+remapped, 27 active+clean; 3.5 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 92 MiB/s wr, 1.48k op/s; 27649/172824 objects misplaced (15.998%); 27 MiB/s, 434 objects/s recovering 2024-08-19T14:17:33.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:32 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "36"}]': finished 2024-08-19T14:17:33.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:32 smithi045 ceph-mon[20143]: osdmap e242: 12 total, 12 up, 9 in 2024-08-19T14:17:33.313 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 31 15 62606 62591 126.17 86.625 0.00501671 0.00792098 2024-08-19T14:17:33.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:32 smithi031 ceph-mon[16336]: pgmap v570: 37 pgs: 10 active+recovering+undersized+remapped, 27 active+clean; 3.5 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 92 MiB/s wr, 1.48k op/s; 27649/172824 objects misplaced (15.998%); 27 MiB/s, 434 objects/s recovering 2024-08-19T14:17:33.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:32 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "36"}]': finished 2024-08-19T14:17:33.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:32 smithi031 ceph-mon[16336]: osdmap e242: 12 total, 12 up, 9 in 2024-08-19T14:17:33.397 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:17:33.438 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:33.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:33.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:17:33.495 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-08-19T14:17:33.540 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:17:33.570 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:33.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:33.571 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:17:33.672 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:17:33.700 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:33.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:33.701 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:17:33.802 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:17:33.827 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:33.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:33.828 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:17:33.929 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:17:33.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:33.978 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:33.978 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:17:34.079 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:17:34.107 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:34.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:34.108 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:17:34.209 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:17:34.236 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:34.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:34.237 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:17:34.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:33 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:17:34.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:33 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:17:34.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:33 smithi032 ceph-mon[20004]: osdmap e243: 12 total, 12 up, 9 in 2024-08-19T14:17:34.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:33 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "36"}]: dispatch 2024-08-19T14:17:34.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:33 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "36"}]: dispatch 2024-08-19T14:17:34.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:33 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:17:34.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:33 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:17:34.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:33 smithi045 ceph-mon[20143]: osdmap e243: 12 total, 12 up, 9 in 2024-08-19T14:17:34.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:33 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "36"}]: dispatch 2024-08-19T14:17:34.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:33 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "36"}]: dispatch 2024-08-19T14:17:34.314 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 32 16 63341 63325 123.661 45.875 0.0393633 0.00798117 2024-08-19T14:17:34.338 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:17:34.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:34.366 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:34.366 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:17:34.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:33 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:17:34.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:33 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:17:34.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:33 smithi031 ceph-mon[16336]: osdmap e243: 12 total, 12 up, 9 in 2024-08-19T14:17:34.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:33 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "36"}]: dispatch 2024-08-19T14:17:34.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:33 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "36"}]: dispatch 2024-08-19T14:17:34.467 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:17:34.499 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:34.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:34.500 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:17:34.601 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:17:34.629 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:34.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:34.630 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:17:34.731 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:17:34.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:34.765 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:34.765 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:17:34.866 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:17:34.887 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:34.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:34.888 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:17:34.989 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:17:34.996 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:17:35.024 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:35.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:35.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:17:35.127 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:17:35.161 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:35.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:35.162 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:17:35.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:34 smithi032 ceph-mon[20004]: pgmap v573: 37 pgs: 10 active+recovering+undersized+remapped, 27 active+clean; 3.7 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 78 MiB/s wr, 1.24k op/s; 23460/181635 objects misplaced (12.916%); 43 MiB/s, 685 objects/s recovering 2024-08-19T14:17:35.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:34 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "36"}]': finished 2024-08-19T14:17:35.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:34 smithi032 ceph-mon[20004]: osdmap e244: 12 total, 12 up, 9 in 2024-08-19T14:17:35.264 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:17:35.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:34 smithi045 ceph-mon[20143]: pgmap v573: 37 pgs: 10 active+recovering+undersized+remapped, 27 active+clean; 3.7 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 78 MiB/s wr, 1.24k op/s; 23460/181635 objects misplaced (12.916%); 43 MiB/s, 685 objects/s recovering 2024-08-19T14:17:35.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:34 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "36"}]': finished 2024-08-19T14:17:35.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:34 smithi045 ceph-mon[20143]: osdmap e244: 12 total, 12 up, 9 in 2024-08-19T14:17:35.297 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:35.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:35.298 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:17:35.314 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 33 16 63731 63715 120.652 24.375 0.0150565 0.00828325 2024-08-19T14:17:35.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:34 smithi031 ceph-mon[16336]: pgmap v573: 37 pgs: 10 active+recovering+undersized+remapped, 27 active+clean; 3.7 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 78 MiB/s wr, 1.24k op/s; 23460/181635 objects misplaced (12.916%); 43 MiB/s, 685 objects/s recovering 2024-08-19T14:17:35.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:34 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "36"}]': finished 2024-08-19T14:17:35.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:34 smithi031 ceph-mon[16336]: osdmap e244: 12 total, 12 up, 9 in 2024-08-19T14:17:35.399 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:17:35.434 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:35.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:35.435 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:17:35.536 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:17:35.580 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:35.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:35.581 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:17:35.682 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:17:35.717 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:35.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:35.718 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:17:35.819 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:17:35.851 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:35.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:35.852 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:17:35.953 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:17:35.987 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:35.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:35.988 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:17:36.089 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:17:36.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:36.112 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:36.113 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:17:36.214 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:17:36.251 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:36.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:36.252 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:17:36.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:35 smithi032 ceph-mon[20004]: osdmap e245: 12 total, 12 up, 9 in 2024-08-19T14:17:36.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:35 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "36"}]: dispatch 2024-08-19T14:17:36.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:35 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "36"}]: dispatch 2024-08-19T14:17:36.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:35 smithi045 ceph-mon[20143]: osdmap e245: 12 total, 12 up, 9 in 2024-08-19T14:17:36.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:35 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "36"}]: dispatch 2024-08-19T14:17:36.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:35 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "36"}]: dispatch 2024-08-19T14:17:36.314 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 34 16 64656 64640 118.804 57.8125 0.0100523 0.00834613 2024-08-19T14:17:36.353 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:17:36.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:35 smithi031 ceph-mon[16336]: osdmap e245: 12 total, 12 up, 9 in 2024-08-19T14:17:36.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:35 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "36"}]: dispatch 2024-08-19T14:17:36.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:35 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "36"}]: dispatch 2024-08-19T14:17:36.375 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:36.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:36.376 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:17:36.476 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:17:36.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:36.499 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:36.499 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:17:36.600 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:17:36.628 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:36.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:36.630 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:17:36.731 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:17:36.758 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:36.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:36.759 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:17:36.860 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:17:36.888 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:36.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:36.889 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:17:36.990 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:17:37.026 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:37.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:37.027 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:17:37.128 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:17:37.156 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:37.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:37.158 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:17:37.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:36 smithi032 ceph-mon[20004]: pgmap v576: 47 pgs: 1 peering, 8 unknown, 10 active+recovering+undersized+remapped, 28 active+clean; 3.8 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 73 MiB/s wr, 1.16k op/s; 22118/186795 objects misplaced (11.841%); 40 MiB/s, 646 objects/s recovering 2024-08-19T14:17:37.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:36 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "36"}]': finished 2024-08-19T14:17:37.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:36 smithi032 ceph-mon[20004]: osdmap e246: 12 total, 12 up, 9 in 2024-08-19T14:17:37.259 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:17:37.281 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:37.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:37.282 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:17:37.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:36 smithi045 ceph-mon[20143]: pgmap v576: 47 pgs: 1 peering, 8 unknown, 10 active+recovering+undersized+remapped, 28 active+clean; 3.8 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 73 MiB/s wr, 1.16k op/s; 22118/186795 objects misplaced (11.841%); 40 MiB/s, 646 objects/s recovering 2024-08-19T14:17:37.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:36 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "36"}]': finished 2024-08-19T14:17:37.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:36 smithi045 ceph-mon[20143]: osdmap e246: 12 total, 12 up, 9 in 2024-08-19T14:17:37.314 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 35 16 64656 64640 115.41 0 - 0.00834613 2024-08-19T14:17:37.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:36 smithi031 ceph-mon[16336]: pgmap v576: 47 pgs: 1 peering, 8 unknown, 10 active+recovering+undersized+remapped, 28 active+clean; 3.8 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 73 MiB/s wr, 1.16k op/s; 22118/186795 objects misplaced (11.841%); 40 MiB/s, 646 objects/s recovering 2024-08-19T14:17:37.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:36 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "36"}]': finished 2024-08-19T14:17:37.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:36 smithi031 ceph-mon[16336]: osdmap e246: 12 total, 12 up, 9 in 2024-08-19T14:17:37.382 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:17:37.410 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:37.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:37.411 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:17:37.512 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:17:37.540 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:37.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:37.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:17:37.641 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:17:37.662 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:37.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:37.663 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:17:37.764 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:17:37.797 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:37.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:37.798 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:17:37.899 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:17:37.932 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:37.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:37.933 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:17:38.034 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:17:38.057 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:38.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:38.058 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:17:38.159 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:17:38.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:38.206 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:38.206 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:17:38.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:37 smithi032 ceph-mon[20004]: osdmap e247: 12 total, 12 up, 9 in 2024-08-19T14:17:38.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:37 smithi032 ceph-mon[20004]: osdmap e248: 12 total, 12 up, 9 in 2024-08-19T14:17:38.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:37 smithi045 ceph-mon[20143]: osdmap e247: 12 total, 12 up, 9 in 2024-08-19T14:17:38.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:37 smithi045 ceph-mon[20143]: osdmap e248: 12 total, 12 up, 9 in 2024-08-19T14:17:38.307 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:17:38.314 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 36 16 64805 64789 112.463 4.65625 0.0112621 0.00883475 2024-08-19T14:17:38.348 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:38.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:38.350 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:17:38.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:37 smithi031 ceph-mon[16336]: osdmap e247: 12 total, 12 up, 9 in 2024-08-19T14:17:38.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:38 smithi031 ceph-mon[16336]: osdmap e248: 12 total, 12 up, 9 in 2024-08-19T14:17:38.451 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:17:38.476 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:38.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:38.477 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:17:38.578 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:17:38.605 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:38.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:38.606 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:17:38.707 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:17:38.739 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:38.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:38.741 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:17:38.800 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:17:38.834 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:17:38.838 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:17:38.842 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:17:38.876 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:38.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:38.877 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:17:38.978 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:17:39.026 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:39.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:39.027 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:17:39.128 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:17:39.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:39.176 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:39.176 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:17:39.277 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:17:39.300 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:39.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:39.301 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:17:39.314 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 37 16 65342 65326 110.33 33.5625 0.00857583 0.00905156 2024-08-19T14:17:39.402 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:17:39.445 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:39.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:39.447 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:17:39.548 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:17:39.591 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:39.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:39.592 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:17:39.693 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:17:39.727 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:39.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:39.728 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:17:39.829 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:17:39.875 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:39.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:39.877 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:17:39.978 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:17:40.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:39 smithi032 ceph-mon[20004]: pgmap v580: 47 pgs: 1 peering, 7 unknown, 10 active+recovering+undersized+remapped, 29 active+clean; 3.9 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 43 MiB/s wr, 691 op/s; 20890/189780 objects misplaced (11.007%); 19 MiB/s, 300 objects/s recovering 2024-08-19T14:17:40.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:39 smithi032 ceph-mon[20004]: osdmap e249: 12 total, 12 up, 9 in 2024-08-19T14:17:40.014 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:40.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:40.015 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:17:40.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:39 smithi045 ceph-mon[20143]: pgmap v580: 47 pgs: 1 peering, 7 unknown, 10 active+recovering+undersized+remapped, 29 active+clean; 3.9 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 43 MiB/s wr, 691 op/s; 20890/189780 objects misplaced (11.007%); 19 MiB/s, 300 objects/s recovering 2024-08-19T14:17:40.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:39 smithi045 ceph-mon[20143]: osdmap e249: 12 total, 12 up, 9 in 2024-08-19T14:17:40.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:39 smithi031 ceph-mon[16336]: pgmap v580: 47 pgs: 1 peering, 7 unknown, 10 active+recovering+undersized+remapped, 29 active+clean; 3.9 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 43 MiB/s wr, 691 op/s; 20890/189780 objects misplaced (11.007%); 19 MiB/s, 300 objects/s recovering 2024-08-19T14:17:40.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:39 smithi031 ceph-mon[16336]: osdmap e249: 12 total, 12 up, 9 in 2024-08-19T14:17:40.116 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:17:40.153 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:40.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:40.154 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:17:40.255 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:17:40.290 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:40.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:40.291 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:17:40.314 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 38 16 66331 66315 109.054 61.8125 0.00761768 0.00916339 2024-08-19T14:17:40.392 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:17:40.421 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:40.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:40.422 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:17:40.523 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:17:40.551 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:40.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:40.553 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:17:40.654 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:17:40.691 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:40.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:40.692 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:17:40.793 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:17:40.834 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:40.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:40.835 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:17:40.936 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:17:40.967 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:40.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:40.968 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:17:41.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:40 smithi032 ceph-mon[20004]: Health check failed: Degraded data redundancy: 4054/178794 objects degraded (2.267%), 1 pg degraded (PG_DEGRADED) 2024-08-19T14:17:41.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:40 smithi045 ceph-mon[20143]: Health check failed: Degraded data redundancy: 4054/178794 objects degraded (2.267%), 1 pg degraded (PG_DEGRADED) 2024-08-19T14:17:41.069 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:17:41.093 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:41.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:41.094 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:17:41.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:40 smithi031 ceph-mon[16336]: Health check failed: Degraded data redundancy: 4054/178794 objects degraded (2.267%), 1 pg degraded (PG_DEGRADED) 2024-08-19T14:17:41.194 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:17:41.227 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:41.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:41.228 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:17:41.314 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 39 16 67131 67115 107.539 50 0.0102348 0.00929442 2024-08-19T14:17:41.329 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:17:41.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:41.373 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:41.373 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:17:41.474 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:17:41.524 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:41.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:41.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:17:41.626 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:17:41.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:41.659 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:41.659 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:17:41.761 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:17:41.789 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:41.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:41.790 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:17:41.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:41 smithi031 ceph-mon[16336]: pgmap v582: 47 pgs: 1 active+undersized+degraded+remapped, 2 peering, 4 unknown, 12 active+recovering+undersized+remapped, 28 active+clean; 3.6 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 4054/178794 objects degraded (2.267%); 27357/178794 objects misplaced (15.301%); 75 MiB/s, 1.19k objects/s recovering 2024-08-19T14:17:41.891 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:17:41.919 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:41.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:41.920 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:17:42.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:41 smithi032 ceph-mon[20004]: pgmap v582: 47 pgs: 1 active+undersized+degraded+remapped, 2 peering, 4 unknown, 12 active+recovering+undersized+remapped, 28 active+clean; 3.6 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 4054/178794 objects degraded (2.267%); 27357/178794 objects misplaced (15.301%); 75 MiB/s, 1.19k objects/s recovering 2024-08-19T14:17:42.021 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:17:42.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:41 smithi045 ceph-mon[20143]: pgmap v582: 47 pgs: 1 active+undersized+degraded+remapped, 2 peering, 4 unknown, 12 active+recovering+undersized+remapped, 28 active+clean; 3.6 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 4054/178794 objects degraded (2.267%); 27357/178794 objects misplaced (15.301%); 75 MiB/s, 1.19k objects/s recovering 2024-08-19T14:17:42.063 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:42.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:42.065 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:17:42.165 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:17:42.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:42.201 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:42.201 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:17:42.302 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:17:42.314 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:2024-08-19T14:17:42.315054+0000 min lat: 0.00117714 max lat: 2.24324 avg lat: 0.00936546 2024-08-19T14:17:42.314 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-19T14:17:42.314 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 40 16 67894 67878 106.043 47.6875 0.0150265 0.00936546 2024-08-19T14:17:42.337 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:42.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:42.338 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:17:42.439 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:17:42.463 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:42.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:42.464 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:17:42.565 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:17:42.605 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:42.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:42.607 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:17:42.708 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:17:42.736 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:42.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:42.737 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:17:42.838 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:17:42.875 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:42.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:42.876 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:17:42.977 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:17:43.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:42 smithi032 ceph-mon[20004]: osdmap e250: 12 total, 12 up, 9 in 2024-08-19T14:17:43.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:42 smithi032 ceph-mon[20004]: osdmap e251: 12 total, 12 up, 9 in 2024-08-19T14:17:43.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:43.018 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:43.018 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:17:43.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:42 smithi045 ceph-mon[20143]: osdmap e250: 12 total, 12 up, 9 in 2024-08-19T14:17:43.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:42 smithi045 ceph-mon[20143]: osdmap e251: 12 total, 12 up, 9 in 2024-08-19T14:17:43.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:42 smithi031 ceph-mon[16336]: osdmap e250: 12 total, 12 up, 9 in 2024-08-19T14:17:43.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:42 smithi031 ceph-mon[16336]: osdmap e251: 12 total, 12 up, 9 in 2024-08-19T14:17:43.119 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:17:43.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:43.160 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:43.160 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:17:43.261 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:17:43.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:43.288 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:43.288 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:17:43.315 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 41 16 68515 68499 104.403 38.8125 0.0164945 0.00954567 2024-08-19T14:17:43.389 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:17:43.419 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:43.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:43.420 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:17:43.522 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:17:43.555 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:43.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:43.557 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:17:43.658 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:17:43.704 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:43.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:43.705 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:17:43.755 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is set 2024-08-19T14:17:43.769 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 7, 8, 9, 11] out_osds: [10, 6, 2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-08-19T14:17:43.770 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-19T14:17:43.770 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-08-19T14:17:43.770 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-19T14:17:43.806 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:17:43.854 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:43.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:43.855 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:17:43.956 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:17:43.995 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:43.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:43.996 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:17:44.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:43 smithi032 ceph-mon[20004]: pgmap v584: 47 pgs: 1 active+undersized+degraded+remapped, 2 peering, 17 active+recovering+undersized+remapped, 27 active+clean; 4.0 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 31 MiB/s wr, 492 op/s; 4054/196959 objects degraded (2.058%); 37728/196959 objects misplaced (19.155%); 79 MiB/s, 1.26k objects/s recovering 2024-08-19T14:17:44.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:43 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/266789086' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:17:44.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:43 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:17:44.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:43 smithi032 ceph-mon[20004]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:17:44.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:43 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:17:44.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:43 smithi032 ceph-mon[20004]: osdmap e252: 12 total, 12 up, 9 in 2024-08-19T14:17:44.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:43 smithi045 ceph-mon[20143]: pgmap v584: 47 pgs: 1 active+undersized+degraded+remapped, 2 peering, 17 active+recovering+undersized+remapped, 27 active+clean; 4.0 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 31 MiB/s wr, 492 op/s; 4054/196959 objects degraded (2.058%); 37728/196959 objects misplaced (19.155%); 79 MiB/s, 1.26k objects/s recovering 2024-08-19T14:17:44.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:43 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/266789086' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:17:44.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:43 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:17:44.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:43 smithi045 ceph-mon[20143]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:17:44.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:43 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:17:44.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:43 smithi045 ceph-mon[20143]: osdmap e252: 12 total, 12 up, 9 in 2024-08-19T14:17:44.097 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:17:44.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:43 smithi031 ceph-mon[16336]: pgmap v584: 47 pgs: 1 active+undersized+degraded+remapped, 2 peering, 17 active+recovering+undersized+remapped, 27 active+clean; 4.0 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 31 MiB/s wr, 492 op/s; 4054/196959 objects degraded (2.058%); 37728/196959 objects misplaced (19.155%); 79 MiB/s, 1.26k objects/s recovering 2024-08-19T14:17:44.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:43 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/266789086' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:17:44.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:43 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:17:44.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:43 smithi031 ceph-mon[16336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:17:44.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:43 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:17:44.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:43 smithi031 ceph-mon[16336]: osdmap e252: 12 total, 12 up, 9 in 2024-08-19T14:17:44.123 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:44.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:44.124 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:17:44.225 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:17:44.252 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:44.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:44.253 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:17:44.314 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 42 16 69310 69294 103.101 49.6875 0.00760853 0.00965827 2024-08-19T14:17:44.354 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:17:44.392 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:44.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:44.393 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:17:44.494 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:17:44.533 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:44.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:44.534 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:17:44.636 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:17:44.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:44.667 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:44.667 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:17:44.769 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:17:44.813 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:44.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:44.814 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:17:44.915 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:17:44.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:44.962 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:44.962 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:17:45.063 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:17:45.085 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:45.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:45.086 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:17:45.187 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:17:45.212 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:45.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:45.214 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:17:45.314 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:17:45.315 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 43 16 69728 69712 101.311 26.125 0.00299327 0.00978493 2024-08-19T14:17:45.341 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:45.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:45.342 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:17:45.444 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:17:45.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:45.472 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:45.473 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:17:45.574 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:17:45.603 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:45.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:45.604 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:17:45.705 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:17:45.735 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:45.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:45.736 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:17:45.837 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:17:45.862 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:45.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:45.863 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:17:45.964 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:17:45.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:45.998 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:45.999 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:17:46.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:45 smithi032 ceph-mon[20004]: pgmap v587: 47 pgs: 1 peering, 17 active+recovering+undersized+remapped, 29 active+clean; 4.1 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 96 MiB/s wr, 1.54k op/s; 34997/201870 objects misplaced (17.336%); 76 MiB/s, 1.21k objects/s recovering 2024-08-19T14:17:46.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:45 smithi032 ceph-mon[20004]: osdmap e253: 12 total, 12 up, 9 in 2024-08-19T14:17:46.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:45 smithi032 ceph-mon[20004]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 4054/196959 objects degraded (2.058%), 1 pg degraded) 2024-08-19T14:17:46.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:45 smithi045 ceph-mon[20143]: pgmap v587: 47 pgs: 1 peering, 17 active+recovering+undersized+remapped, 29 active+clean; 4.1 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 96 MiB/s wr, 1.54k op/s; 34997/201870 objects misplaced (17.336%); 76 MiB/s, 1.21k objects/s recovering 2024-08-19T14:17:46.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:45 smithi045 ceph-mon[20143]: osdmap e253: 12 total, 12 up, 9 in 2024-08-19T14:17:46.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:45 smithi045 ceph-mon[20143]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 4054/196959 objects degraded (2.058%), 1 pg degraded) 2024-08-19T14:17:46.100 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:17:46.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:45 smithi031 ceph-mon[16336]: pgmap v587: 47 pgs: 1 peering, 17 active+recovering+undersized+remapped, 29 active+clean; 4.1 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 96 MiB/s wr, 1.54k op/s; 34997/201870 objects misplaced (17.336%); 76 MiB/s, 1.21k objects/s recovering 2024-08-19T14:17:46.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:45 smithi031 ceph-mon[16336]: osdmap e253: 12 total, 12 up, 9 in 2024-08-19T14:17:46.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:45 smithi031 ceph-mon[16336]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 4054/196959 objects degraded (2.058%), 1 pg degraded) 2024-08-19T14:17:46.133 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:46.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:46.134 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:17:46.235 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:17:46.258 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:46.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:46.259 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:17:46.316 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 44 16 69915 69899 99.2738 11.6875 0.0129188 0.00996301 2024-08-19T14:17:46.360 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:17:46.386 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:46.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:46.387 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:17:46.488 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:17:46.562 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:46.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:46.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:17:46.664 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:17:46.687 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:46.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:46.688 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:17:46.789 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:17:46.818 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:46.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:46.819 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:17:46.920 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:17:46.958 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:46.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:46.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:17:47.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:46 smithi032 ceph-mon[20004]: osdmap e254: 12 total, 12 up, 9 in 2024-08-19T14:17:47.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:46 smithi045 ceph-mon[20143]: osdmap e254: 12 total, 12 up, 9 in 2024-08-19T14:17:47.060 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:17:47.094 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:47.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:47.095 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:17:47.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:46 smithi031 ceph-mon[16336]: osdmap e254: 12 total, 12 up, 9 in 2024-08-19T14:17:47.196 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:17:47.218 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:47.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:47.219 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:17:47.315 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 45 16 70141 70125 97.3817 14.125 0.00287757 0.0101742 2024-08-19T14:17:47.319 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:17:47.342 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:47.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:47.343 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:17:47.444 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:17:47.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:47.478 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:47.478 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:17:47.579 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:17:47.603 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:47.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:47.604 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:17:47.705 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:17:47.740 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:47.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:47.741 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:17:47.841 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:17:47.868 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:47.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:47.870 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:17:47.971 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:17:47.996 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:47.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:47.997 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:17:48.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:47 smithi032 ceph-mon[20004]: pgmap v590: 47 pgs: 1 peering, 16 active+recovering+undersized+remapped, 30 active+clean; 4.2 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 42 MiB/s wr, 668 op/s; 32190/204978 objects misplaced (15.704%); 82 MiB/s, 1.31k objects/s recovering 2024-08-19T14:17:48.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:47 smithi032 ceph-mon[20004]: osdmap e255: 12 total, 12 up, 9 in 2024-08-19T14:17:48.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:47 smithi032 ceph-mon[20004]: osdmap e256: 12 total, 12 up, 9 in 2024-08-19T14:17:48.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:47 smithi045 ceph-mon[20143]: pgmap v590: 47 pgs: 1 peering, 16 active+recovering+undersized+remapped, 30 active+clean; 4.2 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 42 MiB/s wr, 668 op/s; 32190/204978 objects misplaced (15.704%); 82 MiB/s, 1.31k objects/s recovering 2024-08-19T14:17:48.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:47 smithi045 ceph-mon[20143]: osdmap e255: 12 total, 12 up, 9 in 2024-08-19T14:17:48.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:47 smithi045 ceph-mon[20143]: osdmap e256: 12 total, 12 up, 9 in 2024-08-19T14:17:48.099 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:17:48.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:47 smithi031 ceph-mon[16336]: pgmap v590: 47 pgs: 1 peering, 16 active+recovering+undersized+remapped, 30 active+clean; 4.2 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 42 MiB/s wr, 668 op/s; 32190/204978 objects misplaced (15.704%); 82 MiB/s, 1.31k objects/s recovering 2024-08-19T14:17:48.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:47 smithi031 ceph-mon[16336]: osdmap e255: 12 total, 12 up, 9 in 2024-08-19T14:17:48.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:47 smithi031 ceph-mon[16336]: osdmap e256: 12 total, 12 up, 9 in 2024-08-19T14:17:48.121 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:48.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:48.122 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:17:48.223 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:17:48.258 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:48.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:48.259 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:17:48.315 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 46 16 70792 70776 96.1492 40.6875 0.00282516 0.0103325 2024-08-19T14:17:48.362 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:17:48.396 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:48.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:48.397 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:17:48.498 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:17:48.522 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:48.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:48.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:17:48.624 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:17:48.657 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:48.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:48.658 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:17:48.759 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:17:48.785 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:48.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:48.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:17:48.887 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:17:48.911 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:48.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:48.912 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:17:49.012 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:17:49.035 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:49.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:49.036 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:17:49.137 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:17:49.159 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:49.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:49.160 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:17:49.261 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:17:49.282 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:49.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:49.283 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:17:49.315 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 47 16 71266 71250 94.7338 29.625 0.0015394 0.0104484 2024-08-19T14:17:49.386 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:17:49.407 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:49.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:49.408 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:17:49.509 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:17:49.531 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:49.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:49.532 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:17:49.633 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:17:49.654 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:49.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:49.655 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:17:49.756 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:17:49.780 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:49.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:49.781 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:17:49.882 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:17:49.905 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:49.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:49.906 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:17:49.938 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:49 smithi031 ceph-mon[16336]: pgmap v593: 47 pgs: 2 peering, 15 active+recovering+undersized+remapped, 30 active+clean; 4.2 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 26 MiB/s wr, 411 op/s; 28003/206808 objects misplaced (13.541%); 65 MiB/s, 1.03k objects/s recovering 2024-08-19T14:17:49.939 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:49 smithi031 ceph-mon[16336]: osdmap e257: 12 total, 12 up, 9 in 2024-08-19T14:17:50.007 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:17:50.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:49 smithi032 ceph-mon[20004]: pgmap v593: 47 pgs: 2 peering, 15 active+recovering+undersized+remapped, 30 active+clean; 4.2 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 26 MiB/s wr, 411 op/s; 28003/206808 objects misplaced (13.541%); 65 MiB/s, 1.03k objects/s recovering 2024-08-19T14:17:50.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:49 smithi032 ceph-mon[20004]: osdmap e257: 12 total, 12 up, 9 in 2024-08-19T14:17:50.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:50.034 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:50.034 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:17:50.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:49 smithi045 ceph-mon[20143]: pgmap v593: 47 pgs: 2 peering, 15 active+recovering+undersized+remapped, 30 active+clean; 4.2 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 26 MiB/s wr, 411 op/s; 28003/206808 objects misplaced (13.541%); 65 MiB/s, 1.03k objects/s recovering 2024-08-19T14:17:50.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:49 smithi045 ceph-mon[20143]: osdmap e257: 12 total, 12 up, 9 in 2024-08-19T14:17:50.136 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:17:50.157 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:50.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:50.158 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:17:50.259 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:17:50.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:50.304 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:50.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:17:50.315 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 48 16 72102 72086 93.8486 52.25 0.0100219 0.0106513 2024-08-19T14:17:50.405 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:17:50.435 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:50.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:50.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:17:50.538 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:17:50.566 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:50.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:50.567 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:17:50.668 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:17:50.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:50.705 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:50.705 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:17:50.806 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:17:50.841 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:50.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:50.842 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:17:50.943 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:17:50.970 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:50.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:50.972 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:17:51.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:50 smithi032 ceph-mon[20004]: osdmap e258: 12 total, 12 up, 9 in 2024-08-19T14:17:51.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:50 smithi045 ceph-mon[20143]: osdmap e258: 12 total, 12 up, 9 in 2024-08-19T14:17:51.072 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:17:51.100 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:50 smithi031 ceph-mon[16336]: osdmap e258: 12 total, 12 up, 9 in 2024-08-19T14:17:51.100 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:51.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:51.102 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:17:51.203 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:17:51.247 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:51.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:51.248 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:17:51.315 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 49 16 73838 73822 94.1474 108.5 0.00285308 0.0106165 2024-08-19T14:17:51.348 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:17:51.374 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:51.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:51.375 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:17:51.477 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:17:51.511 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:51.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:51.512 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:17:51.614 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:17:51.644 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:51.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:51.645 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:17:51.746 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:17:51.786 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:51.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:51.787 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:17:51.888 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:17:51.909 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:51.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:51.910 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:17:52.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:51 smithi032 ceph-mon[20004]: pgmap v596: 47 pgs: 1 creating+peering, 1 peering, 10 active+recovering+undersized+remapped, 35 active+clean; 4.3 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 29 MiB/s wr, 468 op/s; 22664/210597 objects misplaced (10.762%); 71 MiB/s, 1.14k objects/s recovering 2024-08-19T14:17:52.011 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:17:52.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:52.042 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:52.043 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:17:52.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:51 smithi045 ceph-mon[20143]: pgmap v596: 47 pgs: 1 creating+peering, 1 peering, 10 active+recovering+undersized+remapped, 35 active+clean; 4.3 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 29 MiB/s wr, 468 op/s; 22664/210597 objects misplaced (10.762%); 71 MiB/s, 1.14k objects/s recovering 2024-08-19T14:17:52.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:51 smithi031 ceph-mon[16336]: pgmap v596: 47 pgs: 1 creating+peering, 1 peering, 10 active+recovering+undersized+remapped, 35 active+clean; 4.3 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 29 MiB/s wr, 468 op/s; 22664/210597 objects misplaced (10.762%); 71 MiB/s, 1.14k objects/s recovering 2024-08-19T14:17:52.143 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:17:52.168 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:52.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:52.169 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:17:52.269 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:17:52.296 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:52.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:52.297 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:17:52.315 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 50 16 75246 75230 94.0244 88 0.00832288 0.0106313 2024-08-19T14:17:52.398 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:17:52.425 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:52.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:52.426 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:17:52.527 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:17:52.555 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:52.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:52.556 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:17:52.657 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:17:52.684 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:52.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:52.685 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:17:52.757 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:52 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3013703163' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:17:52.785 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:17:52.806 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:52.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:52.807 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:17:52.908 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:17:52.937 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:52.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:52.938 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:17:53.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:52 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3013703163' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:17:53.039 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:17:53.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:52 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3013703163' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:17:53.066 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:53.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:53.068 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:17:53.169 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:17:53.195 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:53.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:53.196 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:17:53.297 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:17:53.315 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 51 16 76703 76687 93.9662 91.0625 0.00611464 0.0106376 2024-08-19T14:17:53.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:53.344 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:53.344 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:17:53.446 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:17:53.485 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:53.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:53.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:17:53.588 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:17:53.615 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:53.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:53.616 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:17:53.717 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:17:53.742 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:53.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:53.743 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:17:53.844 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:17:53.884 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:53.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:53.885 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:17:53.891 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:17:53.891 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:17:53.891 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:17:53.891 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:17:53.891 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:17:53.891 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:17:53.891 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:17:53.891 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:17:53.892 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:17:53.892 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:17:53.892 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:17:53.892 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:17:53.892 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:17:53.892 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:17:53.892 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:17:53.892 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:17:53.892 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:17:53.893 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:17:53.893 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:17:53.893 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:17:53.893 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:17:53.893 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:17:53.893 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:17:53.893 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:17:53.986 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:17:54.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:53 smithi032 ceph-mon[20004]: pgmap v597: 47 pgs: 1 creating+peering, 5 active+recovering+undersized+remapped, 41 active+clean; 4.4 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 35 MiB/s wr, 564 op/s; 16723/215430 objects misplaced (7.763%); 60 MiB/s, 955 objects/s recovering 2024-08-19T14:17:54.007 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:54.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:54.008 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:17:54.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:53 smithi045 ceph-mon[20143]: pgmap v597: 47 pgs: 1 creating+peering, 5 active+recovering+undersized+remapped, 41 active+clean; 4.4 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 35 MiB/s wr, 564 op/s; 16723/215430 objects misplaced (7.763%); 60 MiB/s, 955 objects/s recovering 2024-08-19T14:17:54.109 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:17:54.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:53 smithi031 ceph-mon[16336]: pgmap v597: 47 pgs: 1 creating+peering, 5 active+recovering+undersized+remapped, 41 active+clean; 4.4 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 35 MiB/s wr, 564 op/s; 16723/215430 objects misplaced (7.763%); 60 MiB/s, 955 objects/s recovering 2024-08-19T14:17:54.147 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:54.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:54.148 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:17:54.249 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:17:54.271 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:54.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:54.272 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:17:54.315 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 52 16 78156 78140 93.9054 90.8125 0.0050094 0.0106295 2024-08-19T14:17:54.372 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:17:54.397 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:54.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:54.398 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:17:54.499 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:17:54.530 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:54.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:54.531 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:17:54.632 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:17:54.659 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:54.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:54.660 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:17:54.761 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:17:54.787 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:54.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:54.788 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:17:54.889 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:17:54.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:54.913 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:54.913 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:17:55.013 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:17:55.041 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:55.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:55.042 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:17:55.143 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:17:55.167 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:55.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:55.168 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:17:55.269 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:17:55.297 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:55.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:55.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:17:55.316 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 53 16 79508 79492 93.7278 84.5 0.00275398 0.0106628 2024-08-19T14:17:55.399 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:17:55.421 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:55.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:55.422 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:17:55.523 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:17:55.548 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:55.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:55.549 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:17:55.650 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:17:55.678 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:55.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:55.680 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:17:55.781 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:17:55.822 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:55.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:55.823 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:17:55.924 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:17:55.957 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:55.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:55.958 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:17:56.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:55 smithi032 ceph-mon[20004]: pgmap v598: 47 pgs: 1 creating+peering, 5 active+recovering+undersized+remapped, 41 active+clean; 4.5 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 54 MiB/s wr, 865 op/s; 14537/222759 objects misplaced (6.526%); 58 MiB/s, 929 objects/s recovering 2024-08-19T14:17:56.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:55 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:17:56.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:55 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:17:56.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:55 smithi045 ceph-mon[20143]: pgmap v598: 47 pgs: 1 creating+peering, 5 active+recovering+undersized+remapped, 41 active+clean; 4.5 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 54 MiB/s wr, 865 op/s; 14537/222759 objects misplaced (6.526%); 58 MiB/s, 929 objects/s recovering 2024-08-19T14:17:56.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:55 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:17:56.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:55 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:17:56.060 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:17:56.089 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:56.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:56.090 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:17:56.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:55 smithi031 ceph-mon[16336]: pgmap v598: 47 pgs: 1 creating+peering, 5 active+recovering+undersized+remapped, 41 active+clean; 4.5 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 54 MiB/s wr, 865 op/s; 14537/222759 objects misplaced (6.526%); 58 MiB/s, 929 objects/s recovering 2024-08-19T14:17:56.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:55 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:17:56.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:55 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:17:56.191 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:17:56.217 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:56.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:56.218 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:17:56.315 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 54 16 80734 80718 93.411 76.625 0.00305234 0.0106934 2024-08-19T14:17:56.318 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:17:56.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:56.356 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:56.356 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:17:56.457 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:17:56.478 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:56.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:56.480 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:17:56.580 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:17:56.607 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:56.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:56.608 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:17:56.709 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:17:56.750 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:56.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:56.751 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:17:56.852 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:17:56.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:56.889 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:56.889 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:17:56.990 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:17:57.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:56 smithi032 ceph-mon[20004]: osdmap e259: 12 total, 12 up, 9 in 2024-08-19T14:17:57.028 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:57.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:57.030 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:17:57.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:56 smithi045 ceph-mon[20143]: osdmap e259: 12 total, 12 up, 9 in 2024-08-19T14:17:57.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:56 smithi031 ceph-mon[16336]: osdmap e259: 12 total, 12 up, 9 in 2024-08-19T14:17:57.130 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:17:57.162 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:57.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:57.163 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:17:57.265 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:17:57.286 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:57.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:57.287 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:17:57.316 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 55 16 82232 82216 93.4147 93.625 0.0100645 0.0106808 2024-08-19T14:17:57.388 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:17:57.417 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:57.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:57.418 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:17:57.519 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:17:57.548 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:57.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:57.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:17:57.650 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:17:57.678 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:57.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:57.679 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:17:57.780 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:17:57.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:57 smithi045 ceph-mon[20143]: pgmap v600: 47 pgs: 5 active+recovering+undersized+remapped, 42 active+clean; 4.7 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 56 MiB/s wr, 903 op/s; 14537/229299 objects misplaced (6.340%); 33 MiB/s, 523 objects/s recovering 2024-08-19T14:17:57.803 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:57.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:57.804 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:17:57.905 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:17:57.935 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:57.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:57.936 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:17:58.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:57 smithi032 ceph-mon[20004]: pgmap v600: 47 pgs: 5 active+recovering+undersized+remapped, 42 active+clean; 4.7 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 56 MiB/s wr, 903 op/s; 14537/229299 objects misplaced (6.340%); 33 MiB/s, 523 objects/s recovering 2024-08-19T14:17:58.037 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:17:58.067 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:58.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:58.068 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:17:58.091 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:57 smithi031 ceph-mon[16336]: pgmap v600: 47 pgs: 5 active+recovering+undersized+remapped, 42 active+clean; 4.7 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 56 MiB/s wr, 903 op/s; 14537/229299 objects misplaced (6.340%); 33 MiB/s, 523 objects/s recovering 2024-08-19T14:17:58.169 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:17:58.199 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:58.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:58.200 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:17:58.301 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:17:58.316 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 56 15 83803 83788 93.5009 98.25 0.0108854 0.0106907 2024-08-19T14:17:58.333 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:58.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:58.334 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:17:58.436 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:17:58.456 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:58.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:58.458 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:17:58.559 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:17:58.587 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:58.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:58.588 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:17:58.689 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:17:58.716 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:58.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:58.717 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:17:58.819 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:17:58.843 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:58.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:58.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:17:58.945 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:17:58.981 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:58.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:58.982 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:17:59.083 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:17:59.108 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:59.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:59.109 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:17:59.210 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:17:59.234 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:59.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:59.235 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:17:59.316 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 57 16 85154 85138 93.3407 84.375 0.00304714 0.010676 2024-08-19T14:17:59.336 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:17:59.369 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:59.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:59.370 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:17:59.472 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:17:59.493 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:59.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:59.494 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:17:59.595 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:17:59.619 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:17:59.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:59.621 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:17:59.721 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:17:59.744 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:17:59.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:59.745 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:17:59.846 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:17:59.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:17:59.885 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:17:59.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:17:59.987 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:18:00.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:59 smithi032 ceph-mon[20004]: pgmap v601: 47 pgs: 5 active+recovering+undersized+remapped, 42 active+clean; 4.8 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 63 MiB/s wr, 1.00k op/s; 12685/235077 objects misplaced (5.396%); 33 MiB/s, 532 objects/s recovering 2024-08-19T14:18:00.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:17:59 smithi032 ceph-mon[20004]: osdmap e260: 12 total, 12 up, 9 in 2024-08-19T14:18:00.009 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:00.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:00.010 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:18:00.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:59 smithi045 ceph-mon[20143]: pgmap v601: 47 pgs: 5 active+recovering+undersized+remapped, 42 active+clean; 4.8 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 63 MiB/s wr, 1.00k op/s; 12685/235077 objects misplaced (5.396%); 33 MiB/s, 532 objects/s recovering 2024-08-19T14:18:00.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:17:59 smithi045 ceph-mon[20143]: osdmap e260: 12 total, 12 up, 9 in 2024-08-19T14:18:00.111 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:18:00.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:59 smithi031 ceph-mon[16336]: pgmap v601: 47 pgs: 5 active+recovering+undersized+remapped, 42 active+clean; 4.8 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 63 MiB/s wr, 1.00k op/s; 12685/235077 objects misplaced (5.396%); 33 MiB/s, 532 objects/s recovering 2024-08-19T14:18:00.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:17:59 smithi031 ceph-mon[16336]: osdmap e260: 12 total, 12 up, 9 in 2024-08-19T14:18:00.147 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:00.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:00.148 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:18:00.249 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:18:00.279 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:00.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:00.280 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:18:00.316 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 58 16 86013 85997 92.657 53.6875 0.166216 0.010783 2024-08-19T14:18:00.381 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:18:00.410 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:00.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:00.414 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:18:00.515 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:18:00.536 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:00.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:00.537 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:18:00.638 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:18:00.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:00.676 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:00.676 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:18:00.778 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:18:00.814 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:00.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:00.815 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:18:00.916 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:18:00.950 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:00.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:00.951 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:18:01.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:00 smithi032 ceph-mon[20004]: osdmap e261: 12 total, 12 up, 9 in 2024-08-19T14:18:01.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:00 smithi045 ceph-mon[20143]: osdmap e261: 12 total, 12 up, 9 in 2024-08-19T14:18:01.051 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:18:01.078 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:01.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:01.079 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:18:01.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:00 smithi031 ceph-mon[16336]: osdmap e261: 12 total, 12 up, 9 in 2024-08-19T14:18:01.180 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:18:01.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:01.218 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:01.218 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:18:01.316 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 59 16 87742 87726 92.9179 108.062 0.00425885 0.0107564 2024-08-19T14:18:01.319 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:18:01.350 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:01.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:01.351 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:18:01.452 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:18:01.492 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:01.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:01.493 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:18:01.594 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:18:01.625 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:01.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:01.626 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:18:01.727 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:18:01.764 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:01.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:01.765 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:18:01.866 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:18:01.897 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:01.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:01.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:18:02.000 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:18:02.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:01 smithi032 ceph-mon[20004]: pgmap v604: 47 pgs: 5 active+recovering+undersized+remapped, 42 active+clean; 5.0 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 82 MiB/s wr, 1.31k op/s; 10689/246450 objects misplaced (4.337%); 18 MiB/s, 283 objects/s recovering 2024-08-19T14:18:02.026 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:02.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:02.027 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:18:02.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:01 smithi045 ceph-mon[20143]: pgmap v604: 47 pgs: 5 active+recovering+undersized+remapped, 42 active+clean; 5.0 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 82 MiB/s wr, 1.31k op/s; 10689/246450 objects misplaced (4.337%); 18 MiB/s, 283 objects/s recovering 2024-08-19T14:18:02.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:01 smithi031 ceph-mon[16336]: pgmap v604: 47 pgs: 5 active+recovering+undersized+remapped, 42 active+clean; 5.0 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 82 MiB/s wr, 1.31k op/s; 10689/246450 objects misplaced (4.337%); 18 MiB/s, 283 objects/s recovering 2024-08-19T14:18:02.127 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:18:02.164 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:02.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:02.165 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:18:02.266 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:18:02.297 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:02.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:02.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:18:02.316 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:2024-08-19T14:18:02.316713+0000 min lat: 0.00117714 max lat: 2.24324 avg lat: 0.0107395 2024-08-19T14:18:02.316 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-19T14:18:02.316 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 60 15 89381 89366 93.0775 102.5 0.00764376 0.0107395 2024-08-19T14:18:02.399 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:18:02.425 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:02.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:02.426 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:18:02.527 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:18:02.568 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:02.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:02.569 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:18:02.670 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:18:02.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:02.700 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:02.701 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:18:02.802 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:18:02.833 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:02.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:02.834 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:18:02.935 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:18:02.960 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:02.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:02.961 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:18:03.062 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:18:03.099 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:03.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:03.100 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:18:03.202 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:18:03.229 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:03.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:03.230 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:18:03.316 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 61 16 91267 91251 93.4828 117.812 0.00714933 0.010693 2024-08-19T14:18:03.332 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:18:03.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:03.371 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:03.372 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:18:03.472 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:18:03.503 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:03.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:03.504 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:18:03.605 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:18:03.635 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:03.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:03.636 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:18:03.737 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:18:03.775 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:03.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:03.776 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:18:03.858 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:18:03.858 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:18:03.876 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:18:03.899 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:03.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:03.900 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:18:04.001 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:18:04.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:03 smithi032 ceph-mon[20004]: pgmap v605: 47 pgs: 4 active+recovering+undersized+remapped, 43 active+clean; 5.2 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 91 MiB/s wr, 1.45k op/s; 9289/255858 objects misplaced (3.631%); 25 MiB/s, 392 objects/s recovering 2024-08-19T14:18:04.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:04.040 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:04.041 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:18:04.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:03 smithi045 ceph-mon[20143]: pgmap v605: 47 pgs: 4 active+recovering+undersized+remapped, 43 active+clean; 5.2 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 91 MiB/s wr, 1.45k op/s; 9289/255858 objects misplaced (3.631%); 25 MiB/s, 392 objects/s recovering 2024-08-19T14:18:04.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:03 smithi031 ceph-mon[16336]: pgmap v605: 47 pgs: 4 active+recovering+undersized+remapped, 43 active+clean; 5.2 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 91 MiB/s wr, 1.45k op/s; 9289/255858 objects misplaced (3.631%); 25 MiB/s, 392 objects/s recovering 2024-08-19T14:18:04.141 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:18:04.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:04.175 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:04.175 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:18:04.276 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:18:04.305 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:04.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:04.306 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:18:04.316 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 62 15 93131 93116 93.8549 116.562 0.00968729 0.0106503 2024-08-19T14:18:04.407 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:18:04.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:04.440 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:04.440 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:18:04.540 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:18:04.569 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:04.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:04.570 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:18:04.671 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:18:04.698 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:04.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:04.699 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:18:04.800 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:18:04.833 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:04.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:04.834 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:18:04.935 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:18:04.962 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:04.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:04.963 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:18:05.064 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:18:05.086 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:05.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:05.091 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:18:05.192 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:18:05.214 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:05.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:05.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:18:05.316 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:18:05.317 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 63 16 94222 94206 93.4464 68.125 0.00226455 0.0106268 2024-08-19T14:18:05.340 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:05.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:05.342 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:18:05.443 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:18:05.466 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:05.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:05.467 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:18:05.568 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:18:05.589 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:05.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:05.590 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:18:05.691 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:18:05.712 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:05.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:05.713 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:18:05.814 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:18:05.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:05.839 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:05.839 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:18:05.940 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:18:05.969 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:05.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:05.971 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:18:06.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:05 smithi032 ceph-mon[20004]: pgmap v606: 47 pgs: 4 active+recovering+undersized+remapped, 43 active+clean; 5.4 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 93 MiB/s wr, 1.49k op/s; 6626/265095 objects misplaced (2.499%); 27 MiB/s, 430 objects/s recovering 2024-08-19T14:18:06.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:05 smithi032 ceph-mon[20004]: osdmap e262: 12 total, 12 up, 9 in 2024-08-19T14:18:06.015 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:05 smithi031 ceph-mon[16336]: pgmap v606: 47 pgs: 4 active+recovering+undersized+remapped, 43 active+clean; 5.4 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 93 MiB/s wr, 1.49k op/s; 6626/265095 objects misplaced (2.499%); 27 MiB/s, 430 objects/s recovering 2024-08-19T14:18:06.015 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:05 smithi031 ceph-mon[16336]: osdmap e262: 12 total, 12 up, 9 in 2024-08-19T14:18:06.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:05 smithi045 ceph-mon[20143]: pgmap v606: 47 pgs: 4 active+recovering+undersized+remapped, 43 active+clean; 5.4 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 93 MiB/s wr, 1.49k op/s; 6626/265095 objects misplaced (2.499%); 27 MiB/s, 430 objects/s recovering 2024-08-19T14:18:06.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:05 smithi045 ceph-mon[20143]: osdmap e262: 12 total, 12 up, 9 in 2024-08-19T14:18:06.072 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:18:06.095 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:06.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:06.096 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:18:06.196 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:18:06.218 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:06.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:06.220 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:18:06.316 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 64 16 94743 94727 92.4951 32.5625 0.00254733 0.0107535 2024-08-19T14:18:06.321 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:18:06.345 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:06.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:06.346 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:18:06.448 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:18:06.472 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:06.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:06.473 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:18:06.574 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:18:06.597 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:06.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:06.598 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:18:06.639 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-08-19T14:18:06.699 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:18:06.711 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:18:06.725 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:06.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:06.726 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:18:06.827 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:18:06.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:06 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:18:06.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:06 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:18:06.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:06 smithi031 ceph-mon[16336]: osdmap e263: 12 total, 12 up, 9 in 2024-08-19T14:18:06.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:06 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:18:06.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:06 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:18:06.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:06 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:18:06.870 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:06.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:06.871 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:18:06.971 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:18:07.027 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:07.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:07.028 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:18:07.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:06 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:18:07.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:06 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:18:07.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:06 smithi045 ceph-mon[20143]: osdmap e263: 12 total, 12 up, 9 in 2024-08-19T14:18:07.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:06 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:18:07.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:06 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:18:07.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:06 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:18:07.129 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:18:07.163 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:07.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:07.164 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:18:07.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:06 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:18:07.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:06 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:18:07.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:06 smithi032 ceph-mon[20004]: osdmap e263: 12 total, 12 up, 9 in 2024-08-19T14:18:07.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:06 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:18:07.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:06 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:18:07.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:06 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:18:07.265 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:18:07.301 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:07.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:07.303 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:18:07.316 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 65 16 95816 95800 92.1038 67.0625 0.00980453 0.0108532 2024-08-19T14:18:07.328 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:18:07.404 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:18:07.439 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:07.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:07.440 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:18:07.541 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:18:07.576 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:07.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:07.577 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:18:07.612 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:18:07.613 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-08-19T14:18:07.678 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:18:07.720 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:07.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:07.721 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:18:07.822 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:18:07.856 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:07.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:07.857 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:18:07.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:07 smithi031 ceph-mon[16336]: pgmap v609: 47 pgs: 4 active+recovering+undersized+remapped, 43 active+clean; 5.6 GiB data, 20 GiB used, 784 GiB / 805 GiB avail; 91 MiB/s wr, 1.45k op/s; 6626/273021 objects misplaced (2.427%); 18 MiB/s, 286 objects/s recovering 2024-08-19T14:18:07.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:07 smithi031 ceph-mon[16336]: osdmap e264: 12 total, 12 up, 9 in 2024-08-19T14:18:07.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:07 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:18:07.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:07 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:18:07.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:07 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:18:07.958 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:18:07.988 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:07.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:07.989 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:18:08.017 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-19T14:18:08.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:07 smithi045 ceph-mon[20143]: pgmap v609: 47 pgs: 4 active+recovering+undersized+remapped, 43 active+clean; 5.6 GiB data, 20 GiB used, 784 GiB / 805 GiB avail; 91 MiB/s wr, 1.45k op/s; 6626/273021 objects misplaced (2.427%); 18 MiB/s, 286 objects/s recovering 2024-08-19T14:18:08.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:07 smithi045 ceph-mon[20143]: osdmap e264: 12 total, 12 up, 9 in 2024-08-19T14:18:08.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:07 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:18:08.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:07 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:18:08.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:07 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:18:08.090 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:18:08.131 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:08.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:08.133 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:18:08.175 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:18:08.234 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:18:08.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:07 smithi032 ceph-mon[20004]: pgmap v609: 47 pgs: 4 active+recovering+undersized+remapped, 43 active+clean; 5.6 GiB data, 20 GiB used, 784 GiB / 805 GiB avail; 91 MiB/s wr, 1.45k op/s; 6626/273021 objects misplaced (2.427%); 18 MiB/s, 286 objects/s recovering 2024-08-19T14:18:08.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:07 smithi032 ceph-mon[20004]: osdmap e264: 12 total, 12 up, 9 in 2024-08-19T14:18:08.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:07 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:18:08.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:07 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:18:08.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:07 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:18:08.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:08.271 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:08.272 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:18:08.317 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 66 16 98089 98073 92.8605 142.062 0.00360001 0.0107645 2024-08-19T14:18:08.372 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:18:08.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:08.415 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:08.416 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:18:08.470 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":610,"stamp":"2024-08-19T14:18:06.756165+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6076170784,"num_objects":92708,"num_object_clones":0,"num_object_copies":278124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5956,"num_objects_unfound":0,"num_objects_dirty":92708,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":92706,"num_write_kb":5933184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29565,"num_bytes_recovered":1937707552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161693,"ondisk_log_size":161693,"up":141,"acting":138,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":163,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":21682632,"kb_used_data":19545908,"kb_used_omap":14,"kb_used_meta":2136625,"kb_avail":822023736,"statfs":{"total":863955320832,"available":841752305664,"internally_reserved":0,"allocated":20015009792,"data_stored":20003681405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14353,"internal_metadata":2187905007},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":691929088,"num_objects":10558,"num_object_clones":0,"num_object_copies":31674,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-4733,"num_objects_unfound":0,"num_objects_dirty":10558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10558,"num_write_kb":675712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2083,"num_bytes_recovered":136511488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.935240"},"pg_stats":[{"pgid":"3.23","version":"263'3943","reported_seq":6123,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-08-19T14:18:05.901146+0000","last_change":"2024-08-19T14:17:46.799697+0000","last_active":"2024-08-19T14:18:05.901146+0000","last_peered":"2024-08-19T14:18:05.901146+0000","last_clean":"2024-08-19T14:18:05.901146+0000","last_became_active":"2024-08-19T14:17:46.797296+0000","last_became_peered":"2024-08-19T14:17:46.797296+0000","last_unstale":"2024-08-19T14:18:05.901146+0000","last_undegraded":"2024-08-19T14:18:05.901146+0000","last_fullsized":"2024-08-19T14:18:05.901146+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":3943,"log_dups_size":0,"ondisk_log_size":3943,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:07.992413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96944128,"num_objects":1479,"num_object_clones":0,"num_object_copies":4437,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1479,"num_write_kb":94672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1031,"num_bytes_recovered":67567616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"261'4032","reported_seq":7145,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-08-19T14:18:04.152074+0000","last_change":"2024-08-19T14:17:49.689584+0000","last_active":"2024-08-19T14:18:04.152074+0000","last_peered":"2024-08-19T14:18:04.152074+0000","last_clean":"2024-08-19T14:18:04.152074+0000","last_became_active":"2024-08-19T14:17:49.687020+0000","last_became_peered":"2024-08-19T14:17:49.687020+0000","last_unstale":"2024-08-19T14:18:04.152074+0000","last_undegraded":"2024-08-19T14:18:04.152074+0000","last_fullsized":"2024-08-19T14:18:04.152074+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":4032,"log_dups_size":0,"ondisk_log_size":4032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:06.535514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":99123200,"num_objects":1513,"num_object_clones":0,"num_object_copies":4539,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1513,"num_write_kb":96800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1037,"num_bytes_recovered":67960832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,9],"acting":[3,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"262'3894","reported_seq":7917,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-08-19T14:18:04.754956+0000","last_change":"2024-08-19T14:17:49.682222+0000","last_active":"2024-08-19T14:18:04.754956+0000","last_peered":"2024-08-19T14:18:04.754956+0000","last_clean":"2024-08-19T14:18:04.754956+0000","last_became_active":"2024-08-19T14:17:49.680746+0000","last_became_peered":"2024-08-19T14:17:49.680746+0000","last_unstale":"2024-08-19T14:18:04.754956+0000","last_undegraded":"2024-08-19T14:18:04.754956+0000","last_fullsized":"2024-08-19T14:18:04.754956+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":3894,"log_dups_size":0,"ondisk_log_size":3894,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:06.267397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":94748672,"num_objects":1446,"num_object_clones":0,"num_object_copies":4338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1446,"num_write_kb":92528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":978,"num_bytes_recovered":64094208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,9],"acting":[8,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.20","version":"261'3831","reported_seq":5911,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-08-19T14:18:03.263246+0000","last_change":"2024-08-19T14:17:43.688336+0000","last_active":"2024-08-19T14:18:03.263246+0000","last_peered":"2024-08-19T14:18:03.263246+0000","last_clean":"2024-08-19T14:18:03.263246+0000","last_became_active":"2024-08-19T14:17:43.686357+0000","last_became_peered":"2024-08-19T14:17:43.686357+0000","last_unstale":"2024-08-19T14:18:03.263246+0000","last_undegraded":"2024-08-19T14:18:03.263246+0000","last_fullsized":"2024-08-19T14:18:03.263246+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":3831,"log_dups_size":0,"ondisk_log_size":3831,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:05.877965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92012544,"num_objects":1404,"num_object_clones":0,"num_object_copies":4212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1404,"num_write_kb":89856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":982,"num_bytes_recovered":64356352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,11],"acting":[9,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1f","version":"261'4780","reported_seq":9248,"reported_epoch":261,"state":"active+recovering+undersized+remapped","last_fresh":"2024-08-19T14:18:03.181049+0000","last_change":"2024-08-19T14:17:37.937497+0000","last_active":"2024-08-19T14:18:03.181049+0000","last_peered":"2024-08-19T14:18:03.181049+0000","last_clean":"2024-08-19T14:17:35.957523+0000","last_became_active":"2024-08-19T14:17:37.872209+0000","last_became_peered":"2024-08-19T14:17:37.872209+0000","last_unstale":"2024-08-19T14:18:03.181049+0000","last_undegraded":"2024-08-19T14:18:03.181049+0000","last_fullsized":"2024-08-19T14:17:37.674519+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":4780,"log_dups_size":0,"ondisk_log_size":4780,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":183402496,"num_objects":2799,"num_object_clones":0,"num_object_copies":8397,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2852,"num_objects_unfound":0,"num_objects_dirty":2799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2799,"num_write_kb":179104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1091,"num_bytes_recovered":71499776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[7,0],"avail_no_missing":["7","8","11"],"object_location_counts":[{"shards":"0,7","objects":1848},{"shards":"0,7,8,11","objects":1},{"shards":"7,8,11","objects":950}],"blocked_by":[],"up_primary":0,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1e","version":"261'4760","reported_seq":9632,"reported_epoch":261,"state":"active+recovering+undersized+remapped","last_fresh":"2024-08-19T14:18:02.985561+0000","last_change":"2024-08-19T14:17:38.098204+0000","last_active":"2024-08-19T14:18:02.985561+0000","last_peered":"2024-08-19T14:18:02.985561+0000","last_clean":"2024-08-19T14:17:35.902763+0000","last_became_active":"2024-08-19T14:17:37.903239+0000","last_became_peered":"2024-08-19T14:17:37.903239+0000","last_unstale":"2024-08-19T14:18:02.985561+0000","last_undegraded":"2024-08-19T14:18:02.985561+0000","last_fullsized":"2024-08-19T14:17:37.672761+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":4760,"log_dups_size":0,"ondisk_log_size":4760,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":182386688,"num_objects":2783,"num_object_clones":0,"num_object_copies":8349,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2502,"num_objects_unfound":0,"num_objects_dirty":2783,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2783,"num_write_kb":178112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1193,"num_bytes_recovered":78184448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,8],"acting":[4,3],"avail_no_missing":["4","0","11"],"object_location_counts":[{"shards":"0,4,11","objects":834},{"shards":"3,4","objects":1949}],"blocked_by":[],"up_primary":3,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"262'4973","reported_seq":11117,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-08-19T14:18:04.806013+0000","last_change":"2024-08-19T14:17:59.738739+0000","last_active":"2024-08-19T14:18:04.806013+0000","last_peered":"2024-08-19T14:18:04.806013+0000","last_clean":"2024-08-19T14:18:04.806013+0000","last_became_active":"2024-08-19T14:17:59.735912+0000","last_became_peered":"2024-08-19T14:17:59.735912+0000","last_unstale":"2024-08-19T14:18:04.806013+0000","last_undegraded":"2024-08-19T14:18:04.806013+0000","last_fullsized":"2024-08-19T14:18:04.806013+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":4973,"log_dups_size":0,"ondisk_log_size":4973,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:05.248586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":193822720,"num_objects":2958,"num_object_clones":0,"num_object_copies":8874,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2958,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2958,"num_write_kb":189280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2011,"num_bytes_recovered":131792896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,5],"acting":[8,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"263'4877","reported_seq":10769,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-08-19T14:18:06.083288+0000","last_change":"2024-08-19T14:18:05.761531+0000","last_active":"2024-08-19T14:18:06.083288+0000","last_peered":"2024-08-19T14:18:06.083288+0000","last_clean":"2024-08-19T14:18:06.083288+0000","last_became_active":"2024-08-19T14:18:05.757137+0000","last_became_peered":"2024-08-19T14:18:05.757137+0000","last_unstale":"2024-08-19T14:18:06.083288+0000","last_undegraded":"2024-08-19T14:18:06.083288+0000","last_fullsized":"2024-08-19T14:18:06.083288+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":4877,"log_dups_size":0,"ondisk_log_size":4877,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:11.106527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":191791104,"num_objects":2927,"num_object_clones":0,"num_object_copies":8781,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2927,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2927,"num_write_kb":187296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1926,"num_bytes_recovered":126222336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"261'4800","reported_seq":9974,"reported_epoch":261,"state":"active+recovering+undersized+remapped","last_fresh":"2024-08-19T14:18:02.981012+0000","last_change":"2024-08-19T14:17:37.899646+0000","last_active":"2024-08-19T14:18:02.981012+0000","last_peered":"2024-08-19T14:18:02.981012+0000","last_clean":"2024-08-19T14:17:35.902120+0000","last_became_active":"2024-08-19T14:17:37.823149+0000","last_became_peered":"2024-08-19T14:17:37.823149+0000","last_unstale":"2024-08-19T14:18:02.981012+0000","last_undegraded":"2024-08-19T14:18:02.981012+0000","last_fullsized":"2024-08-19T14:17:37.672345+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":4800,"log_dups_size":0,"ondisk_log_size":4800,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184516608,"num_objects":2816,"num_object_clones":0,"num_object_copies":8448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":602,"num_objects_unfound":0,"num_objects_dirty":2816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2816,"num_write_kb":180192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1691,"num_bytes_recovered":110821376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,1],"acting":[4,1],"avail_no_missing":["4","0","3"],"object_location_counts":[{"shards":"0,3,4","objects":301},{"shards":"1,4","objects":2515}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"262'4940","reported_seq":9065,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-08-19T14:18:04.817743+0000","last_change":"2024-08-19T14:17:46.733798+0000","last_active":"2024-08-19T14:18:04.817743+0000","last_peered":"2024-08-19T14:18:04.817743+0000","last_clean":"2024-08-19T14:18:04.817743+0000","last_became_active":"2024-08-19T14:17:46.731170+0000","last_became_peered":"2024-08-19T14:17:46.731170+0000","last_unstale":"2024-08-19T14:18:04.817743+0000","last_undegraded":"2024-08-19T14:18:04.817743+0000","last_fullsized":"2024-08-19T14:18:04.817743+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":4940,"log_dups_size":0,"ondisk_log_size":4940,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:08.286014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":194510848,"num_objects":2968,"num_object_clones":0,"num_object_copies":8904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2968,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2968,"num_write_kb":189952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2009,"num_bytes_recovered":131661824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,9],"acting":[8,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.19","version":"261'4404","reported_seq":10789,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-08-19T14:18:03.362559+0000","last_change":"2024-08-19T14:17:46.800856+0000","last_active":"2024-08-19T14:18:03.362559+0000","last_peered":"2024-08-19T14:18:03.362559+0000","last_clean":"2024-08-19T14:18:03.362559+0000","last_became_active":"2024-08-19T14:17:46.796533+0000","last_became_peered":"2024-08-19T14:17:46.796533+0000","last_unstale":"2024-08-19T14:18:03.362559+0000","last_undegraded":"2024-08-19T14:18:03.362559+0000","last_fullsized":"2024-08-19T14:18:03.362559+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":4404,"log_dups_size":0,"ondisk_log_size":4404,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:03.538617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":190087168,"num_objects":2901,"num_object_clones":0,"num_object_copies":8703,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2901,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2901,"num_write_kb":185632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1569,"num_bytes_recovered":102825984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"261'4510","reported_seq":9335,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-08-19T14:18:04.218469+0000","last_change":"2024-08-19T14:17:45.732581+0000","last_active":"2024-08-19T14:18:04.218469+0000","last_peered":"2024-08-19T14:18:04.218469+0000","last_clean":"2024-08-19T14:18:04.218469+0000","last_became_active":"2024-08-19T14:17:45.729134+0000","last_became_peered":"2024-08-19T14:17:45.729134+0000","last_unstale":"2024-08-19T14:18:04.218469+0000","last_undegraded":"2024-08-19T14:18:04.218469+0000","last_fullsized":"2024-08-19T14:18:04.218469+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":4510,"log_dups_size":0,"ondisk_log_size":4510,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:08.553916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":194084864,"num_objects":2962,"num_object_clones":0,"num_object_copies":8886,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2962,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2962,"num_write_kb":189536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1572,"num_bytes_recovered":103022592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,1],"acting":[3,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"263'4330","reported_seq":10558,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-08-19T14:18:05.832039+0000","last_change":"2024-08-19T14:17:48.685189+0000","last_active":"2024-08-19T14:18:05.832039+0000","last_peered":"2024-08-19T14:18:05.832039+0000","last_clean":"2024-08-19T14:18:05.832039+0000","last_became_active":"2024-08-19T14:17:48.682321+0000","last_became_peered":"2024-08-19T14:17:48.682321+0000","last_unstale":"2024-08-19T14:18:05.832039+0000","last_undegraded":"2024-08-19T14:18:05.832039+0000","last_fullsized":"2024-08-19T14:18:05.832039+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":4330,"log_dups_size":0,"ondisk_log_size":4330,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:09.030062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":187236352,"num_objects":2857,"num_object_clones":0,"num_object_copies":8571,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2857,"num_write_kb":182848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1529,"num_bytes_recovered":100204544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,9],"acting":[0,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"263'4545","reported_seq":9250,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-08-19T14:18:06.012650+0000","last_change":"2024-08-19T14:17:47.684929+0000","last_active":"2024-08-19T14:18:06.012650+0000","last_peered":"2024-08-19T14:18:06.012650+0000","last_clean":"2024-08-19T14:18:06.012650+0000","last_became_active":"2024-08-19T14:17:47.681397+0000","last_became_peered":"2024-08-19T14:17:47.681397+0000","last_unstale":"2024-08-19T14:18:06.012650+0000","last_undegraded":"2024-08-19T14:18:06.012650+0000","last_fullsized":"2024-08-19T14:18:06.012650+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":4545,"log_dups_size":0,"ondisk_log_size":4545,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:10.148534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197820416,"num_objects":3019,"num_object_clones":0,"num_object_copies":9057,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3019,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3019,"num_write_kb":193184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1597,"num_bytes_recovered":104660992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"262'4396","reported_seq":9141,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-08-19T14:18:04.809932+0000","last_change":"2024-08-19T14:17:42.700414+0000","last_active":"2024-08-19T14:18:04.809932+0000","last_peered":"2024-08-19T14:18:04.809932+0000","last_clean":"2024-08-19T14:18:04.809932+0000","last_became_active":"2024-08-19T14:17:42.696659+0000","last_became_peered":"2024-08-19T14:17:42.696659+0000","last_unstale":"2024-08-19T14:18:04.809932+0000","last_undegraded":"2024-08-19T14:18:04.809932+0000","last_fullsized":"2024-08-19T14:18:04.809932+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":4396,"log_dups_size":0,"ondisk_log_size":4396,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:09.299192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":191135744,"num_objects":2917,"num_object_clones":0,"num_object_copies":8751,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2917,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2917,"num_write_kb":186656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1544,"num_bytes_recovered":101187584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.14","version":"261'4373","reported_seq":9213,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-08-19T14:18:02.896766+0000","last_change":"2024-08-19T14:17:44.694783+0000","last_active":"2024-08-19T14:18:02.896766+0000","last_peered":"2024-08-19T14:18:02.896766+0000","last_clean":"2024-08-19T14:18:02.896766+0000","last_became_active":"2024-08-19T14:17:44.691159+0000","last_became_peered":"2024-08-19T14:17:44.691159+0000","last_unstale":"2024-08-19T14:18:02.896766+0000","last_undegraded":"2024-08-19T14:18:02.896766+0000","last_fullsized":"2024-08-19T14:18:02.896766+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":4373,"log_dups_size":0,"ondisk_log_size":4373,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:06.829429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188317696,"num_objects":2874,"num_object_clones":0,"num_object_copies":8622,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2874,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2874,"num_write_kb":183904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1576,"num_bytes_recovered":103284736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,11],"acting":[4,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.13","version":"261'4300","reported_seq":9019,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-08-19T14:18:03.159003+0000","last_change":"2024-08-19T14:17:45.734788+0000","last_active":"2024-08-19T14:18:03.159003+0000","last_peered":"2024-08-19T14:18:03.159003+0000","last_clean":"2024-08-19T14:18:03.159003+0000","last_became_active":"2024-08-19T14:17:45.731114+0000","last_became_peered":"2024-08-19T14:17:45.731114+0000","last_unstale":"2024-08-19T14:18:03.159003+0000","last_undegraded":"2024-08-19T14:18:03.159003+0000","last_fullsized":"2024-08-19T14:18:03.159003+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":4300,"log_dups_size":0,"ondisk_log_size":4300,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:06.800180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184778752,"num_objects":2820,"num_object_clones":0,"num_object_copies":8460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2820,"num_write_kb":180448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1536,"num_bytes_recovered":100663296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,9],"acting":[7,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"263'4428","reported_seq":10895,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-08-19T14:18:05.839050+0000","last_change":"2024-08-19T14:17:47.707364+0000","last_active":"2024-08-19T14:18:05.839050+0000","last_peered":"2024-08-19T14:18:05.839050+0000","last_clean":"2024-08-19T14:18:05.839050+0000","last_became_active":"2024-08-19T14:17:47.703601+0000","last_became_peered":"2024-08-19T14:17:47.703601+0000","last_unstale":"2024-08-19T14:18:05.839050+0000","last_undegraded":"2024-08-19T14:18:05.839050+0000","last_fullsized":"2024-08-19T14:18:05.839050+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":4428,"log_dups_size":0,"ondisk_log_size":4428,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:07.992426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":190775296,"num_objects":2911,"num_object_clones":0,"num_object_copies":8733,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2911,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2911,"num_write_kb":186304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1589,"num_bytes_recovered":104136704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"261'4238","reported_seq":7441,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-08-19T14:18:02.956939+0000","last_change":"2024-08-19T14:17:38.697673+0000","last_active":"2024-08-19T14:18:02.956939+0000","last_peered":"2024-08-19T14:18:02.956939+0000","last_clean":"2024-08-19T14:18:02.956939+0000","last_became_active":"2024-08-19T14:17:38.695549+0000","last_became_peered":"2024-08-19T14:17:38.695549+0000","last_unstale":"2024-08-19T14:18:02.956939+0000","last_undegraded":"2024-08-19T14:18:02.956939+0000","last_fullsized":"2024-08-19T14:18:02.956939+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":4238,"log_dups_size":0,"ondisk_log_size":4238,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:06.829405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":181305344,"num_objects":2767,"num_object_clones":0,"num_object_copies":8301,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2767,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2767,"num_write_kb":177056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1552,"num_bytes_recovered":101711872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"261'4274","reported_seq":9036,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-08-19T14:18:03.307451+0000","last_change":"2024-08-19T14:17:38.687862+0000","last_active":"2024-08-19T14:18:03.307451+0000","last_peered":"2024-08-19T14:18:03.307451+0000","last_clean":"2024-08-19T14:18:03.307451+0000","last_became_active":"2024-08-19T14:17:38.684742+0000","last_became_peered":"2024-08-19T14:17:38.684742+0000","last_unstale":"2024-08-19T14:18:03.307451+0000","last_undegraded":"2024-08-19T14:18:03.307451+0000","last_fullsized":"2024-08-19T14:18:03.307451+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":4274,"log_dups_size":0,"ondisk_log_size":4274,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:06.899162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":185139200,"num_objects":2825,"num_object_clones":0,"num_object_copies":8475,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2825,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2825,"num_write_kb":180800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1551,"num_bytes_recovered":101646336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,3],"acting":[9,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.f","version":"261'4824","reported_seq":4924,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-08-19T14:18:03.076397+0000","last_change":"2024-08-19T14:17:34.958285+0000","last_active":"2024-08-19T14:18:03.076397+0000","last_peered":"2024-08-19T14:18:03.076397+0000","last_clean":"2024-08-19T14:18:03.076397+0000","last_became_active":"2024-08-19T14:17:34.954276+0000","last_became_peered":"2024-08-19T14:17:34.954276+0000","last_unstale":"2024-08-19T14:18:03.076397+0000","last_undegraded":"2024-08-19T14:18:03.076397+0000","last_fullsized":"2024-08-19T14:18:03.076397+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":4824,"log_dups_size":0,"ondisk_log_size":4824,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":186286080,"num_objects":2842,"num_object_clones":0,"num_object_copies":8526,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2842,"num_write_kb":181920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"262'4938","reported_seq":5039,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-08-19T14:18:05.030568+0000","last_change":"2024-08-19T14:17:34.973808+0000","last_active":"2024-08-19T14:18:05.030568+0000","last_peered":"2024-08-19T14:18:05.030568+0000","last_clean":"2024-08-19T14:18:05.030568+0000","last_became_active":"2024-08-19T14:17:34.971738+0000","last_became_peered":"2024-08-19T14:17:34.971738+0000","last_unstale":"2024-08-19T14:18:05.030568+0000","last_undegraded":"2024-08-19T14:18:05.030568+0000","last_fullsized":"2024-08-19T14:18:05.030568+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":4938,"log_dups_size":0,"ondisk_log_size":4938,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":194052096,"num_objects":2961,"num_object_clones":0,"num_object_copies":8883,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2961,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2961,"num_write_kb":189504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,0],"acting":[11,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.d","version":"261'4906","reported_seq":5006,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-08-19T14:18:03.168295+0000","last_change":"2024-08-19T14:17:34.971437+0000","last_active":"2024-08-19T14:18:03.168295+0000","last_peered":"2024-08-19T14:18:03.168295+0000","last_clean":"2024-08-19T14:18:03.168295+0000","last_became_active":"2024-08-19T14:17:34.969333+0000","last_became_peered":"2024-08-19T14:17:34.969333+0000","last_unstale":"2024-08-19T14:18:03.168295+0000","last_undegraded":"2024-08-19T14:18:03.168295+0000","last_fullsized":"2024-08-19T14:18:03.168295+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":4906,"log_dups_size":0,"ondisk_log_size":4906,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189431808,"num_objects":2890,"num_object_clones":0,"num_object_copies":8670,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2890,"num_write_kb":184992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,9],"acting":[7,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"263'4879","reported_seq":4983,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-08-19T14:18:06.015486+0000","last_change":"2024-08-19T14:17:34.976154+0000","last_active":"2024-08-19T14:18:06.015486+0000","last_peered":"2024-08-19T14:18:06.015486+0000","last_clean":"2024-08-19T14:18:06.015486+0000","last_became_active":"2024-08-19T14:17:34.973857+0000","last_became_peered":"2024-08-19T14:17:34.973857+0000","last_unstale":"2024-08-19T14:18:06.015486+0000","last_undegraded":"2024-08-19T14:18:06.015486+0000","last_fullsized":"2024-08-19T14:18:06.015486+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":4879,"log_dups_size":0,"ondisk_log_size":4879,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:09.150181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":191922176,"num_objects":2928,"num_object_clones":0,"num_object_copies":8784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2928,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2928,"num_write_kb":187424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,9],"acting":[5,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"262'4380","reported_seq":4482,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-08-19T14:18:04.755264+0000","last_change":"2024-08-19T14:17:19.745357+0000","last_active":"2024-08-19T14:18:04.755264+0000","last_peered":"2024-08-19T14:18:04.755264+0000","last_clean":"2024-08-19T14:18:04.755264+0000","last_became_active":"2024-08-19T14:17:19.743605+0000","last_became_peered":"2024-08-19T14:17:19.743605+0000","last_unstale":"2024-08-19T14:18:04.755264+0000","last_undegraded":"2024-08-19T14:18:04.755264+0000","last_fullsized":"2024-08-19T14:18:04.755264+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":4380,"log_dups_size":0,"ondisk_log_size":4380,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:07.289156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188514304,"num_objects":2876,"num_object_clones":0,"num_object_copies":8628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2876,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2876,"num_write_kb":184096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"261'4786","reported_seq":4882,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-08-19T14:18:02.956592+0000","last_change":"2024-08-19T14:17:34.964717+0000","last_active":"2024-08-19T14:18:02.956592+0000","last_peered":"2024-08-19T14:18:02.956592+0000","last_clean":"2024-08-19T14:18:02.956592+0000","last_became_active":"2024-08-19T14:17:34.962596+0000","last_became_peered":"2024-08-19T14:17:34.962596+0000","last_unstale":"2024-08-19T14:18:02.956592+0000","last_undegraded":"2024-08-19T14:18:02.956592+0000","last_fullsized":"2024-08-19T14:18:02.956592+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":4786,"log_dups_size":0,"ondisk_log_size":4786,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:03.922101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184418304,"num_objects":2814,"num_object_clones":0,"num_object_copies":8442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2814,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2814,"num_write_kb":180096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,8],"acting":[4,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"70'139","reported_seq":617,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-08-19T14:18:05.754173+0000","last_change":"2024-08-19T14:16:42.249971+0000","last_active":"2024-08-19T14:18:05.754173+0000","last_peered":"2024-08-19T14:18:05.754173+0000","last_clean":"2024-08-19T14:18:05.754173+0000","last_became_active":"2024-08-19T14:16:42.249518+0000","last_became_peered":"2024-08-19T14:16:42.249518+0000","last_unstale":"2024-08-19T14:18:05.754173+0000","last_undegraded":"2024-08-19T14:18:05.754173+0000","last_fullsized":"2024-08-19T14:18:05.754173+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:15:44.541487+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:15:44.541487+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,8],"acting":[5,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.b","version":"261'4936","reported_seq":5036,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-08-19T14:18:04.203143+0000","last_change":"2024-08-19T14:17:34.975300+0000","last_active":"2024-08-19T14:18:04.203143+0000","last_peered":"2024-08-19T14:18:04.203143+0000","last_clean":"2024-08-19T14:18:04.203143+0000","last_became_active":"2024-08-19T14:17:34.972934+0000","last_became_peered":"2024-08-19T14:17:34.972934+0000","last_unstale":"2024-08-19T14:18:04.203143+0000","last_undegraded":"2024-08-19T14:18:04.203143+0000","last_fullsized":"2024-08-19T14:18:04.203143+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":4936,"log_dups_size":0,"ondisk_log_size":4936,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":193429504,"num_objects":2951,"num_object_clones":0,"num_object_copies":8853,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2951,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2951,"num_write_kb":188896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"70'139","reported_seq":731,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-08-19T14:17:59.736765+0000","last_change":"2024-08-19T14:16:42.252897+0000","last_active":"2024-08-19T14:17:59.736765+0000","last_peered":"2024-08-19T14:17:59.736765+0000","last_clean":"2024-08-19T14:17:59.736765+0000","last_became_active":"2024-08-19T14:16:42.252391+0000","last_became_peered":"2024-08-19T14:16:42.252391+0000","last_unstale":"2024-08-19T14:17:59.736765+0000","last_undegraded":"2024-08-19T14:17:59.736765+0000","last_fullsized":"2024-08-19T14:17:59.736765+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:16:03.918046+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"262'4579","reported_seq":4681,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-08-19T14:18:04.987044+0000","last_change":"2024-08-19T14:17:19.755178+0000","last_active":"2024-08-19T14:18:04.987044+0000","last_peered":"2024-08-19T14:18:04.987044+0000","last_clean":"2024-08-19T14:18:04.987044+0000","last_became_active":"2024-08-19T14:17:19.753063+0000","last_became_peered":"2024-08-19T14:17:19.753063+0000","last_unstale":"2024-08-19T14:18:04.987044+0000","last_undegraded":"2024-08-19T14:18:04.987044+0000","last_fullsized":"2024-08-19T14:18:04.987044+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":4579,"log_dups_size":0,"ondisk_log_size":4579,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198606848,"num_objects":3030,"num_object_clones":0,"num_object_copies":9090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3030,"num_write_kb":193952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,4],"acting":[11,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.a","version":"70'139","reported_seq":609,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-08-19T14:17:59.745078+0000","last_change":"2024-08-19T14:16:02.818848+0000","last_active":"2024-08-19T14:17:59.745078+0000","last_peered":"2024-08-19T14:17:59.745078+0000","last_clean":"2024-08-19T14:17:59.745078+0000","last_became_active":"2024-08-19T14:13:22.728779+0000","last_became_peered":"2024-08-19T14:13:22.728779+0000","last_unstale":"2024-08-19T14:17:59.745078+0000","last_undegraded":"2024-08-19T14:17:59.745078+0000","last_fullsized":"2024-08-19T14:17:59.745078+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:16:02.818774+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:16:02.818774+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,0],"acting":[9,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.6","version":"263'4574","reported_seq":4678,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-08-19T14:18:05.816580+0000","last_change":"2024-08-19T14:17:19.750488+0000","last_active":"2024-08-19T14:18:05.816580+0000","last_peered":"2024-08-19T14:18:05.816580+0000","last_clean":"2024-08-19T14:18:05.816580+0000","last_became_active":"2024-08-19T14:17:19.748180+0000","last_became_peered":"2024-08-19T14:17:19.748180+0000","last_unstale":"2024-08-19T14:18:05.816580+0000","last_undegraded":"2024-08-19T14:18:05.816580+0000","last_fullsized":"2024-08-19T14:18:05.816580+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":4574,"log_dups_size":0,"ondisk_log_size":4574,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:07.027447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":199720960,"num_objects":3047,"num_object_clones":0,"num_object_copies":9141,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3047,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3047,"num_write_kb":195040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"70'139","reported_seq":649,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-08-19T14:18:04.754192+0000","last_change":"2024-08-19T14:16:29.257033+0000","last_active":"2024-08-19T14:18:04.754192+0000","last_peered":"2024-08-19T14:18:04.754192+0000","last_clean":"2024-08-19T14:18:04.754192+0000","last_became_active":"2024-08-19T14:13:05.607447+0000","last_became_peered":"2024-08-19T14:13:05.607447+0000","last_unstale":"2024-08-19T14:18:04.754192+0000","last_undegraded":"2024-08-19T14:18:04.754192+0000","last_fullsized":"2024-08-19T14:18:04.754192+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:16:29.256959+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:16:29.256959+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.7","version":"261'4358","reported_seq":4458,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-08-19T14:18:04.194129+0000","last_change":"2024-08-19T14:17:19.746696+0000","last_active":"2024-08-19T14:18:04.194129+0000","last_peered":"2024-08-19T14:18:04.194129+0000","last_clean":"2024-08-19T14:18:04.194129+0000","last_became_active":"2024-08-19T14:17:19.742604+0000","last_became_peered":"2024-08-19T14:17:19.742604+0000","last_unstale":"2024-08-19T14:18:04.194129+0000","last_undegraded":"2024-08-19T14:18:04.194129+0000","last_fullsized":"2024-08-19T14:18:04.194129+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":4358,"log_dups_size":0,"ondisk_log_size":4358,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189071360,"num_objects":2885,"num_object_clones":0,"num_object_copies":8655,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2885,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2885,"num_write_kb":184640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"70'139","reported_seq":411,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-08-19T14:18:04.752819+0000","last_change":"2024-08-19T14:16:42.252111+0000","last_active":"2024-08-19T14:18:04.752819+0000","last_peered":"2024-08-19T14:18:04.752819+0000","last_clean":"2024-08-19T14:18:04.752819+0000","last_became_active":"2024-08-19T14:16:42.251643+0000","last_became_peered":"2024-08-19T14:16:42.251643+0000","last_unstale":"2024-08-19T14:18:04.752819+0000","last_undegraded":"2024-08-19T14:18:04.752819+0000","last_fullsized":"2024-08-19T14:18:04.752819+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:16:30.232687+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:16:30.232687+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,4],"acting":[11,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"261'4364","reported_seq":4463,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-08-19T14:18:03.375256+0000","last_change":"2024-08-19T14:17:19.744537+0000","last_active":"2024-08-19T14:18:03.375256+0000","last_peered":"2024-08-19T14:18:03.375256+0000","last_clean":"2024-08-19T14:18:03.375256+0000","last_became_active":"2024-08-19T14:17:19.741977+0000","last_became_peered":"2024-08-19T14:17:19.741977+0000","last_unstale":"2024-08-19T14:18:03.375256+0000","last_undegraded":"2024-08-19T14:18:03.375256+0000","last_fullsized":"2024-08-19T14:18:03.375256+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":4364,"log_dups_size":0,"ondisk_log_size":4364,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:07.612683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":187727872,"num_objects":2864,"num_object_clones":0,"num_object_copies":8592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2864,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2864,"num_write_kb":183328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,5],"acting":[1,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"70'139","reported_seq":596,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-08-19T14:17:59.744783+0000","last_change":"2024-08-19T14:16:36.969224+0000","last_active":"2024-08-19T14:17:59.744783+0000","last_peered":"2024-08-19T14:17:59.744783+0000","last_clean":"2024-08-19T14:17:59.744783+0000","last_became_active":"2024-08-19T14:13:11.647085+0000","last_became_peered":"2024-08-19T14:13:11.647085+0000","last_unstale":"2024-08-19T14:17:59.744783+0000","last_undegraded":"2024-08-19T14:17:59.744783+0000","last_fullsized":"2024-08-19T14:17:59.744783+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:16:36.969114+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:16:36.969114+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,4],"acting":[9,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"263'4419","reported_seq":4523,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-08-19T14:18:06.072023+0000","last_change":"2024-08-19T14:17:19.749394+0000","last_active":"2024-08-19T14:18:06.072023+0000","last_peered":"2024-08-19T14:18:06.072023+0000","last_clean":"2024-08-19T14:18:06.072023+0000","last_became_active":"2024-08-19T14:17:19.745911+0000","last_became_peered":"2024-08-19T14:17:19.745911+0000","last_unstale":"2024-08-19T14:18:06.072023+0000","last_undegraded":"2024-08-19T14:18:06.072023+0000","last_fullsized":"2024-08-19T14:18:06.072023+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":4419,"log_dups_size":0,"ondisk_log_size":4419,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:09.145814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":192643072,"num_objects":2939,"num_object_clones":0,"num_object_copies":8817,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2939,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2939,"num_write_kb":188128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"70'139","reported_seq":517,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-08-19T14:18:05.754427+0000","last_change":"2024-08-19T14:16:42.252166+0000","last_active":"2024-08-19T14:18:05.754427+0000","last_peered":"2024-08-19T14:18:05.754427+0000","last_clean":"2024-08-19T14:18:05.754427+0000","last_became_active":"2024-08-19T14:16:42.251741+0000","last_became_peered":"2024-08-19T14:16:42.251741+0000","last_unstale":"2024-08-19T14:18:05.754427+0000","last_undegraded":"2024-08-19T14:18:05.754427+0000","last_fullsized":"2024-08-19T14:18:05.754427+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:16:10.672834+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"261'3876","reported_seq":3981,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-08-19T14:18:02.893412+0000","last_change":"2024-08-19T14:17:34.971150+0000","last_active":"2024-08-19T14:18:02.893412+0000","last_peered":"2024-08-19T14:18:02.893412+0000","last_clean":"2024-08-19T14:18:02.893412+0000","last_became_active":"2024-08-19T14:17:34.968430+0000","last_became_peered":"2024-08-19T14:17:34.968430+0000","last_unstale":"2024-08-19T14:18:02.893412+0000","last_undegraded":"2024-08-19T14:18:02.893412+0000","last_fullsized":"2024-08-19T14:18:02.893412+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":3876,"log_dups_size":0,"ondisk_log_size":3876,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:03.922075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92553216,"num_objects":1412,"num_object_clones":0,"num_object_copies":4236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1412,"num_write_kb":90384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,11],"acting":[4,0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"70'139","reported_seq":605,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-08-19T14:17:59.744606+0000","last_change":"2024-08-19T14:16:42.252108+0000","last_active":"2024-08-19T14:17:59.744606+0000","last_peered":"2024-08-19T14:17:59.744606+0000","last_clean":"2024-08-19T14:17:59.744606+0000","last_became_active":"2024-08-19T14:16:42.251603+0000","last_became_peered":"2024-08-19T14:16:42.251603+0000","last_unstale":"2024-08-19T14:17:59.744606+0000","last_undegraded":"2024-08-19T14:17:59.744606+0000","last_fullsized":"2024-08-19T14:17:59.744606+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:16:30.043526+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:16:30.043526+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,11],"acting":[9,0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.0","version":"262'3878","reported_seq":3986,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-08-19T14:18:04.904227+0000","last_change":"2024-08-19T14:17:34.964487+0000","last_active":"2024-08-19T14:18:04.904227+0000","last_peered":"2024-08-19T14:18:04.904227+0000","last_clean":"2024-08-19T14:18:04.904227+0000","last_became_active":"2024-08-19T14:17:34.963390+0000","last_became_peered":"2024-08-19T14:17:34.963390+0000","last_unstale":"2024-08-19T14:18:04.904227+0000","last_undegraded":"2024-08-19T14:18:04.904227+0000","last_fullsized":"2024-08-19T14:18:04.904227+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":3878,"log_dups_size":0,"ondisk_log_size":3878,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":95092736,"num_objects":1451,"num_object_clones":0,"num_object_copies":4353,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1451,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1451,"num_write_kb":92864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,1],"acting":[11,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.2","version":"70'139","reported_seq":557,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-08-19T14:18:05.757562+0000","last_change":"2024-08-19T14:16:00.592808+0000","last_active":"2024-08-19T14:18:05.757562+0000","last_peered":"2024-08-19T14:18:05.757562+0000","last_clean":"2024-08-19T14:18:05.757562+0000","last_became_active":"2024-08-19T14:15:59.993566+0000","last_became_peered":"2024-08-19T14:15:59.993566+0000","last_unstale":"2024-08-19T14:18:05.757562+0000","last_undegraded":"2024-08-19T14:18:05.757562+0000","last_fullsized":"2024-08-19T14:18:05.757562+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:16:00.592678+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:16:00.592678+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,8],"acting":[0,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"263'3881","reported_seq":3982,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-08-19T14:18:05.912800+0000","last_change":"2024-08-19T14:17:34.964957+0000","last_active":"2024-08-19T14:18:05.912800+0000","last_peered":"2024-08-19T14:18:05.912800+0000","last_clean":"2024-08-19T14:18:05.912800+0000","last_became_active":"2024-08-19T14:17:34.963892+0000","last_became_peered":"2024-08-19T14:17:34.963892+0000","last_unstale":"2024-08-19T14:18:05.912800+0000","last_undegraded":"2024-08-19T14:18:05.912800+0000","last_fullsized":"2024-08-19T14:18:05.912800+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":3881,"log_dups_size":0,"ondisk_log_size":3881,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:07.027433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":93896704,"num_objects":1432,"num_object_clones":0,"num_object_copies":4296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1432,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1432,"num_write_kb":91696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,11],"acting":[0,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"70'139","reported_seq":608,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-08-19T14:17:59.736611+0000","last_change":"2024-08-19T14:16:06.679203+0000","last_active":"2024-08-19T14:17:59.736611+0000","last_peered":"2024-08-19T14:17:59.736611+0000","last_clean":"2024-08-19T14:17:59.736611+0000","last_became_active":"2024-08-19T14:13:22.727234+0000","last_became_peered":"2024-08-19T14:13:22.727234+0000","last_unstale":"2024-08-19T14:17:59.736611+0000","last_undegraded":"2024-08-19T14:17:59.736611+0000","last_fullsized":"2024-08-19T14:17:59.736611+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:16:06.679075+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:16:06.679075+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"261'3958","reported_seq":4064,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-08-19T14:18:03.211902+0000","last_change":"2024-08-19T14:17:34.969785+0000","last_active":"2024-08-19T14:18:03.211902+0000","last_peered":"2024-08-19T14:18:03.211902+0000","last_clean":"2024-08-19T14:18:03.211902+0000","last_became_active":"2024-08-19T14:17:34.968554+0000","last_became_peered":"2024-08-19T14:17:34.968554+0000","last_unstale":"2024-08-19T14:18:03.211902+0000","last_undegraded":"2024-08-19T14:18:03.211902+0000","last_fullsized":"2024-08-19T14:18:03.211902+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":3958,"log_dups_size":0,"ondisk_log_size":3958,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:03.879179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":94273536,"num_objects":1438,"num_object_clones":0,"num_object_copies":4314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1438,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1438,"num_write_kb":92064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,5],"acting":[9,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.0","version":"70'139","reported_seq":709,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-08-19T14:17:59.736119+0000","last_change":"2024-08-19T14:16:42.251163+0000","last_active":"2024-08-19T14:17:59.736119+0000","last_peered":"2024-08-19T14:17:59.736119+0000","last_clean":"2024-08-19T14:17:59.736119+0000","last_became_active":"2024-08-19T14:16:42.250702+0000","last_became_peered":"2024-08-19T14:16:42.250702+0000","last_unstale":"2024-08-19T14:17:59.736119+0000","last_undegraded":"2024-08-19T14:17:59.736119+0000","last_fullsized":"2024-08-19T14:17:59.736119+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:15:59.869266+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:15:59.869266+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,9],"acting":[7,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":36,"stat_sum":{"num_bytes":6075580416,"num_objects":92706,"num_object_clones":0,"num_object_copies":278118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5956,"num_objects_unfound":0,"num_objects_dirty":92706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":92706,"num_write_kb":5933184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29564,"num_bytes_recovered":1937506304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20140195840,"data_stored":20140195840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160164,"ondisk_log_size":160164,"up":108,"acting":105,"num_store_stats":9},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1802240,"data_stored":1788032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1529,"ondisk_log_size":1529,"up":33,"acting":33,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187537,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2359908,"kb_used_data":2148248,"kb_used_omap":1,"kb_used_meta":211646,"kb_avail":91385244,"statfs":{"total":95995035648,"available":93578489856,"internally_reserved":0,"allocated":2199805952,"data_stored":2198546142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":216725956},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":343597383766,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":74,"seq":317827579994,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2466452,"kb_used_data":2160396,"kb_used_omap":1,"kb_used_meta":306046,"kb_avail":91278700,"statfs":{"total":95995035648,"available":93469388800,"internally_reserved":0,"allocated":2212245504,"data_stored":2210989800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":313391553},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874336,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2063516,"kb_used_data":1829652,"kb_used_omap":1,"kb_used_meta":233854,"kb_avail":91681636,"statfs":{"total":95995035648,"available":93881995264,"internally_reserved":0,"allocated":1873563648,"data_stored":1872304392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":239466949},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":57,"seq":244813135970,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2150612,"kb_used_data":1894540,"kb_used_omap":1,"kb_used_meta":256062,"kb_avail":91594540,"statfs":{"total":95995035648,"available":93792808960,"internally_reserved":0,"allocated":1940008960,"data_stored":1938753256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":262207946},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":0,"seq":206158430310,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691947,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2430264,"kb_used_data":2250540,"kb_used_omap":1,"kb_used_meta":179710,"kb_avail":91314888,"statfs":{"total":95995035648,"available":93506445312,"internally_reserved":0,"allocated":2304552960,"data_stored":2303285521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":184023493},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986287,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3489556,"kb_used_data":3255692,"kb_used_omap":1,"kb_used_meta":233854,"kb_avail":90255596,"statfs":{"total":95995035648,"available":92421730304,"internally_reserved":0,"allocated":3333828608,"data_stored":3332572904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":239466949},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149813,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2331804,"kb_used_data":2047956,"kb_used_omap":1,"kb_used_meta":283838,"kb_avail":91413348,"statfs":{"total":95995035648,"available":93607268352,"internally_reserved":0,"allocated":2097106944,"data_stored":2095847688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":290650567},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411450,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574974,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2266964,"kb_used_data":2051148,"kb_used_omap":1,"kb_used_meta":215806,"kb_avail":91478188,"statfs":{"total":95995035648,"available":93673664512,"internally_reserved":0,"allocated":2100375552,"data_stored":2099119848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":220985797},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738499,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2123556,"kb_used_data":1907736,"kb_used_omap":1,"kb_used_meta":215806,"kb_avail":91621596,"statfs":{"total":95995035648,"available":93820514304,"internally_reserved":0,"allocated":1953521664,"data_stored":1952261854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":220985797},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1963786240,"data_stored":1963786240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2140471296,"data_stored":2140471296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2120548352,"data_stored":2120548352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":3339845632,"data_stored":3339845632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2308898816,"data_stored":2308898816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1950220288,"data_stored":1950220288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1893662720,"data_stored":1893662720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":2226388992,"data_stored":2226388992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":2196373504,"data_stored":2196373504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T14:18:08.473 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-08-19T14:18:08.473 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd pool set .mgr pg_num 21 2024-08-19T14:18:08.517 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:18:08.556 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:08.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:08.557 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:18:08.658 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:18:08.699 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:08.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:08.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:18:08.807 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:18:08.859 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:08.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:08.860 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:18:08.865 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:18:08.865 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:18:08.961 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:18:08.993 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:08.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:08.994 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:18:09.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:08 smithi045 ceph-mon[20143]: from='client.25205 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:18:09.095 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:18:09.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:08 smithi031 ceph-mon[16336]: from='client.25205 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:18:09.132 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:09.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:09.133 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:18:09.135 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:18:09.234 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:18:09.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:08 smithi032 ceph-mon[20004]: from='client.25205 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:18:09.263 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:09.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:09.264 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:18:09.317 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 67 16 100328 100312 93.563 139.938 0.00676697 0.0106842 2024-08-19T14:18:09.365 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:18:09.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:09.402 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:09.403 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:18:09.503 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:18:09.530 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:09.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:09.531 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:18:09.634 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:18:09.638 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:18:09.650 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:18:09.657 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-08-19T14:18:09.667 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:09.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:09.668 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:18:09.769 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:18:09.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:09.809 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:09.809 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:18:09.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:09 smithi031 ceph-mon[16336]: pgmap v611: 47 pgs: 3 active+recovering+undersized+remapped, 44 active+clean; 5.7 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 77 MiB/s wr, 1.24k op/s; 5956/278124 objects misplaced (2.141%); 14 MiB/s, 230 objects/s recovering 2024-08-19T14:18:09.910 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:18:09.937 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:09.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:09.939 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:18:10.041 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:18:10.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:09 smithi045 ceph-mon[20143]: pgmap v611: 47 pgs: 3 active+recovering+undersized+remapped, 44 active+clean; 5.7 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 77 MiB/s wr, 1.24k op/s; 5956/278124 objects misplaced (2.141%); 14 MiB/s, 230 objects/s recovering 2024-08-19T14:18:10.075 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:10.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:10.076 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:18:10.176 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:18:10.212 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:10.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:10.213 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:18:10.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:09 smithi032 ceph-mon[20004]: pgmap v611: 47 pgs: 3 active+recovering+undersized+remapped, 44 active+clean; 5.7 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 77 MiB/s wr, 1.24k op/s; 5956/278124 objects misplaced (2.141%); 14 MiB/s, 230 objects/s recovering 2024-08-19T14:18:10.314 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:18:10.317 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 68 16 102538 102522 94.2181 138.125 0.0052636 0.0106095 2024-08-19T14:18:10.351 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:10.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:10.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:18:10.453 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:18:10.486 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:10.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:10.487 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:18:10.588 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:18:10.622 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:10.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:10.623 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:18:10.724 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:18:10.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:10.758 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:10.758 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:18:10.783 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is set 2024-08-19T14:18:10.859 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:18:10.890 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:10.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:10.891 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:18:10.991 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:18:11.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:11.017 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:11.017 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:18:11.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:10 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:18:11.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:10 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:18:11.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:10 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/216833409' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:18:11.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:10 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:18:11.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:10 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:18:11.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:10 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/216833409' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:18:11.118 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:18:11.139 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:11.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:11.140 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:18:11.240 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:18:11.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:10 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:18:11.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:10 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:18:11.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:10 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/216833409' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:18:11.261 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:11.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:11.262 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:18:11.317 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 69 16 103849 103833 94.0399 81.9375 0.00254619 0.0105728 2024-08-19T14:18:11.362 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:18:11.364 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:18:11.366 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:18:11.367 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:18:11.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:11.385 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:11.385 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:18:11.391 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:18:11.391 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:18:11.407 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:18:11.408 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:18:11.423 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:18:11.424 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:18:11.441 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:18:11.441 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:18:11.457 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:18:11.458 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:18:11.474 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:18:11.474 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:18:11.486 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:18:11.490 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:18:11.490 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:18:11.507 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:18:11.508 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:18:11.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:11.509 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:11.509 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:18:11.524 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:18:11.524 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:18:11.541 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:18:11.542 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:18:11.558 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:18:11.558 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:18:11.610 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:18:11.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:11.633 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:11.633 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:18:11.734 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:18:11.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:11.757 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:11.757 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:18:11.786 INFO:teuthology.orchestra.run.smithi031.stderr:set pool 1 pg_num to 21 2024-08-19T14:18:11.859 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:18:11.931 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:11.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:11.932 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:18:12.033 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:18:12.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:11 smithi045 ceph-mon[20143]: pgmap v612: 47 pgs: 2 active+recovering+undersized+remapped, 45 active+clean; 5.9 GiB data, 21 GiB used, 783 GiB / 805 GiB avail; 83 MiB/s wr, 1.33k op/s; 2049/289080 objects misplaced (0.709%); 18 MiB/s, 289 objects/s recovering 2024-08-19T14:18:12.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:11 smithi045 ceph-mon[20143]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:18:12.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:11 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/216833409' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:18:12.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:11 smithi045 ceph-mon[20143]: osdmap e265: 12 total, 12 up, 9 in 2024-08-19T14:18:12.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:11 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/231148189' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-08-19T14:18:12.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:11 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2410691289' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:18:12.057 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:12.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:12.058 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:18:12.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:11 smithi031 ceph-mon[16336]: pgmap v612: 47 pgs: 2 active+recovering+undersized+remapped, 45 active+clean; 5.9 GiB data, 21 GiB used, 783 GiB / 805 GiB avail; 83 MiB/s wr, 1.33k op/s; 2049/289080 objects misplaced (0.709%); 18 MiB/s, 289 objects/s recovering 2024-08-19T14:18:12.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:11 smithi031 ceph-mon[16336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:18:12.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:11 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/216833409' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:18:12.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:11 smithi031 ceph-mon[16336]: osdmap e265: 12 total, 12 up, 9 in 2024-08-19T14:18:12.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:11 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/231148189' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-08-19T14:18:12.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:11 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2410691289' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:18:12.159 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:18:12.180 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:12.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:12.181 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:18:12.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:11 smithi032 ceph-mon[20004]: pgmap v612: 47 pgs: 2 active+recovering+undersized+remapped, 45 active+clean; 5.9 GiB data, 21 GiB used, 783 GiB / 805 GiB avail; 83 MiB/s wr, 1.33k op/s; 2049/289080 objects misplaced (0.709%); 18 MiB/s, 289 objects/s recovering 2024-08-19T14:18:12.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:11 smithi032 ceph-mon[20004]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:18:12.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:11 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/216833409' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:18:12.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:11 smithi032 ceph-mon[20004]: osdmap e265: 12 total, 12 up, 9 in 2024-08-19T14:18:12.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:11 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/231148189' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-08-19T14:18:12.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:11 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2410691289' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:18:12.282 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:18:12.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:12.305 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:12.305 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:18:12.317 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 70 16 104331 104315 93.1268 30.125 0.00231664 0.0106768 2024-08-19T14:18:12.406 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:18:12.427 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:12.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:12.428 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:18:12.529 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:18:12.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:12.555 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:12.555 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:18:12.656 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:18:12.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:12.681 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:12.681 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:18:12.783 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:18:12.811 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:12.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:12.813 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:18:12.915 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:18:12.944 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:12.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:12.945 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:18:13.007 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:12 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/231148189' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-08-19T14:18:13.008 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:12 smithi031 ceph-mon[16336]: osdmap e266: 12 total, 12 up, 9 in 2024-08-19T14:18:13.008 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:12 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-08-19T14:18:13.008 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:12 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-08-19T14:18:13.046 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:18:13.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:12 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/231148189' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-08-19T14:18:13.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:12 smithi045 ceph-mon[20143]: osdmap e266: 12 total, 12 up, 9 in 2024-08-19T14:18:13.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:12 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-08-19T14:18:13.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:12 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-08-19T14:18:13.075 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:13.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:13.076 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:18:13.177 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:18:13.206 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:13.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:13.207 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:18:13.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:12 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/231148189' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-08-19T14:18:13.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:12 smithi032 ceph-mon[20004]: osdmap e266: 12 total, 12 up, 9 in 2024-08-19T14:18:13.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:12 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-08-19T14:18:13.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:12 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-08-19T14:18:13.308 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:18:13.317 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 71 16 105509 105493 92.8521 73.625 0.00550829 0.0107656 2024-08-19T14:18:13.339 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:13.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:13.340 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:18:13.441 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:18:13.474 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:13.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:13.475 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:18:13.554 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:18:13.554 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 20, 2024-08-19T14:18:13.554 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 600, 2024-08-19T14:18:13.554 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [ 2024-08-19T14:18:13.554 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.554 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:40314 2.6 2:65a7b859:::benchmark_data_smithi045_46415_object40313:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-08-19T14:18:13.554 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:13:02.993308+0000", 2024-08-19T14:18:13.554 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 310.55726978899997, 2024-08-19T14:18:13.554 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.20540061800000001, 2024-08-19T14:18:13.554 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:18:13.555 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:18:13.555 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:18:13.555 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:18:13.555 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:18:13.555 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:18:13.555 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 40314 2024-08-19T14:18:13.555 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.555 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:18:13.555 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.555 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:18:13.555 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.993308+0000", 2024-08-19T14:18:13.555 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:18:13.556 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.556 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.556 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:18:13.556 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.993308+0000", 2024-08-19T14:18:13.556 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:18:13.556 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.556 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.556 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:18:13.556 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.993309+0000", 2024-08-19T14:18:13.556 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1200000000000001e-06 2024-08-19T14:18:13.556 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.556 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.556 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:18:13.557 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.993470+0000", 2024-08-19T14:18:13.557 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00016027899999999999 2024-08-19T14:18:13.557 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.557 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.557 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:18:13.557 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.993482+0000", 2024-08-19T14:18:13.557 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2276000000000001e-05 2024-08-19T14:18:13.557 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.557 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.557 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:18:13.557 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.993487+0000", 2024-08-19T14:18:13.557 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.4199999999999998e-06 2024-08-19T14:18:13.557 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.557 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.558 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:18:13.558 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.993536+0000", 2024-08-19T14:18:13.558 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.9051999999999997e-05 2024-08-19T14:18:13.558 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.558 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.558 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:18:13.558 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.993628+0000", 2024-08-19T14:18:13.558 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.1546999999999996e-05 2024-08-19T14:18:13.558 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.558 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.558 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,8", 2024-08-19T14:18:13.558 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.993732+0000", 2024-08-19T14:18:13.559 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000104467 2024-08-19T14:18:13.559 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.559 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.559 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:18:13.559 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.997603+0000", 2024-08-19T14:18:13.559 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0038706529999999999 2024-08-19T14:18:13.559 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.559 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.559 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:18:13.559 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.997654+0000", 2024-08-19T14:18:13.559 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0628999999999999e-05 2024-08-19T14:18:13.559 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.559 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.560 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:18:13.560 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.198613+0000", 2024-08-19T14:18:13.560 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.20095934500000001 2024-08-19T14:18:13.560 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.560 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.560 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:18:13.560 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.198666+0000", 2024-08-19T14:18:13.560 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.3340000000000001e-05 2024-08-19T14:18:13.560 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.560 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.560 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:18:13.560 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.198709+0000", 2024-08-19T14:18:13.560 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2490000000000001e-05 2024-08-19T14:18:13.561 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:13.561 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:18:13.561 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:13.561 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.561 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.561 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:40315 2.9 2:95c123d2:::benchmark_data_smithi045_46415_object40314:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-08-19T14:18:13.561 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:13:02.993698+0000", 2024-08-19T14:18:13.561 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 310.556880122, 2024-08-19T14:18:13.561 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.20320638299999999, 2024-08-19T14:18:13.561 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:18:13.561 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:18:13.561 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:18:13.561 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:18:13.562 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:18:13.562 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:18:13.562 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 40315 2024-08-19T14:18:13.562 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.562 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:18:13.562 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.562 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:18:13.562 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.993698+0000", 2024-08-19T14:18:13.562 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:18:13.562 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.562 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.563 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:18:13.563 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.993698+0000", 2024-08-19T14:18:13.563 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:18:13.563 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.563 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.563 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:18:13.563 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.993699+0000", 2024-08-19T14:18:13.563 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.71e-07 2024-08-19T14:18:13.563 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.563 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.563 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:18:13.563 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.993770+0000", 2024-08-19T14:18:13.563 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.1186000000000006e-05 2024-08-19T14:18:13.563 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.564 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.564 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:18:13.564 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.993782+0000", 2024-08-19T14:18:13.564 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1822e-05 2024-08-19T14:18:13.564 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.564 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.564 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:18:13.564 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.993789+0000", 2024-08-19T14:18:13.564 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.3640000000000001e-06 2024-08-19T14:18:13.564 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.564 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.565 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:18:13.565 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.993844+0000", 2024-08-19T14:18:13.565 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.4688999999999998e-05 2024-08-19T14:18:13.565 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.565 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.565 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:18:13.565 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.993997+0000", 2024-08-19T14:18:13.565 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000153709 2024-08-19T14:18:13.565 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.565 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.565 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,7", 2024-08-19T14:18:13.565 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.994134+0000", 2024-08-19T14:18:13.565 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013627600000000001 2024-08-19T14:18:13.565 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.566 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.566 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:18:13.566 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.995858+0000", 2024-08-19T14:18:13.566 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00172466 2024-08-19T14:18:13.566 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.566 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.566 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:18:13.566 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.996156+0000", 2024-08-19T14:18:13.566 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000297764 2024-08-19T14:18:13.566 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.566 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.566 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:18:13.566 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.196808+0000", 2024-08-19T14:18:13.566 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.20065148399999999 2024-08-19T14:18:13.566 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.567 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.567 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:18:13.567 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.196850+0000", 2024-08-19T14:18:13.567 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.1934000000000002e-05 2024-08-19T14:18:13.567 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.567 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.567 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:18:13.567 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.196904+0000", 2024-08-19T14:18:13.567 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.4623999999999997e-05 2024-08-19T14:18:13.567 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:13.567 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:18:13.567 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:13.567 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.567 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.568 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:40316 2.6 2:6b76d263:::benchmark_data_smithi045_46415_object40315:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-08-19T14:18:13.568 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:13:02.994308+0000", 2024-08-19T14:18:13.568 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 310.55626951300002, 2024-08-19T14:18:13.568 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.20450618700000001, 2024-08-19T14:18:13.568 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:18:13.568 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:18:13.568 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:18:13.569 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:18:13.569 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:18:13.569 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:18:13.569 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 40316 2024-08-19T14:18:13.569 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.569 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:18:13.569 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.569 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:18:13.569 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.994308+0000", 2024-08-19T14:18:13.569 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:18:13.569 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.569 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.570 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:18:13.570 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.994308+0000", 2024-08-19T14:18:13.570 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:18:13.570 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.570 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.570 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:18:13.570 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.994309+0000", 2024-08-19T14:18:13.570 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.7899999999999997e-07 2024-08-19T14:18:13.570 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.570 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.570 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:18:13.570 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.994385+0000", 2024-08-19T14:18:13.570 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.6071999999999999e-05 2024-08-19T14:18:13.570 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.571 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.571 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:18:13.571 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.994398+0000", 2024-08-19T14:18:13.571 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2228e-05 2024-08-19T14:18:13.571 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.571 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.571 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:18:13.571 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.994405+0000", 2024-08-19T14:18:13.571 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.3189999999999997e-06 2024-08-19T14:18:13.571 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.571 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.571 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:18:13.571 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.995083+0000", 2024-08-19T14:18:13.571 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000678117 2024-08-19T14:18:13.572 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.572 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.572 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:18:13.572 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.995204+0000", 2024-08-19T14:18:13.572 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012103399999999999 2024-08-19T14:18:13.572 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.572 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.572 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,8", 2024-08-19T14:18:13.572 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.995352+0000", 2024-08-19T14:18:13.572 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00014803400000000001 2024-08-19T14:18:13.572 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.572 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.572 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:18:13.572 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.997636+0000", 2024-08-19T14:18:13.573 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0022837830000000002 2024-08-19T14:18:13.573 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.573 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.573 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:18:13.573 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.997659+0000", 2024-08-19T14:18:13.573 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.3473e-05 2024-08-19T14:18:13.573 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.573 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.573 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:18:13.573 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.198750+0000", 2024-08-19T14:18:13.573 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.201090301 2024-08-19T14:18:13.573 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.573 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.573 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:18:13.574 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.198780+0000", 2024-08-19T14:18:13.574 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.0688999999999997e-05 2024-08-19T14:18:13.574 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.574 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.574 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:18:13.574 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.198815+0000", 2024-08-19T14:18:13.574 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.4258000000000003e-05 2024-08-19T14:18:13.574 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:13.574 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:18:13.574 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:13.574 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.574 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.574 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:40320 2.6 2:6546d5b0:::benchmark_data_smithi045_46415_object40319:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-08-19T14:18:13.575 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:13:02.995448+0000", 2024-08-19T14:18:13.575 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 310.55512961699998, 2024-08-19T14:18:13.575 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.20356038500000001, 2024-08-19T14:18:13.575 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:18:13.575 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:18:13.575 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:18:13.575 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:18:13.575 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:18:13.575 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:18:13.575 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 40320 2024-08-19T14:18:13.575 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.575 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:18:13.575 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.575 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:18:13.576 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.995448+0000", 2024-08-19T14:18:13.576 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:18:13.576 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.576 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.576 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:18:13.576 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.995448+0000", 2024-08-19T14:18:13.576 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:18:13.576 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.576 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.576 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:18:13.576 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.995449+0000", 2024-08-19T14:18:13.576 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.7000000000000004e-07 2024-08-19T14:18:13.576 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.576 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.577 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:18:13.577 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.995511+0000", 2024-08-19T14:18:13.577 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.2249000000000007e-05 2024-08-19T14:18:13.577 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.577 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.577 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:18:13.577 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.995520+0000", 2024-08-19T14:18:13.577 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.8799999999999997e-06 2024-08-19T14:18:13.577 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.577 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.577 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:18:13.577 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.995525+0000", 2024-08-19T14:18:13.577 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.6659999999999999e-06 2024-08-19T14:18:13.577 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.578 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.578 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:18:13.578 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.996261+0000", 2024-08-19T14:18:13.578 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000736351 2024-08-19T14:18:13.578 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.578 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.578 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:18:13.578 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.996382+0000", 2024-08-19T14:18:13.578 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000120983 2024-08-19T14:18:13.578 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.578 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.578 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,8", 2024-08-19T14:18:13.578 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.996517+0000", 2024-08-19T14:18:13.578 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013479900000000001 2024-08-19T14:18:13.579 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.579 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.579 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:18:13.579 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.997767+0000", 2024-08-19T14:18:13.579 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0012497739999999999 2024-08-19T14:18:13.579 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.579 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.579 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:18:13.579 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.999297+0000", 2024-08-19T14:18:13.579 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0015301150000000001 2024-08-19T14:18:13.579 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.579 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.579 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:18:13.580 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.198909+0000", 2024-08-19T14:18:13.580 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.199612074 2024-08-19T14:18:13.580 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.580 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.580 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:18:13.580 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.198968+0000", 2024-08-19T14:18:13.580 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.8801999999999997e-05 2024-08-19T14:18:13.580 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.580 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.580 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:18:13.580 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.199009+0000", 2024-08-19T14:18:13.580 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.0921999999999999e-05 2024-08-19T14:18:13.580 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:13.580 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:18:13.580 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:13.581 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.581 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.581 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:40322 2.9 2:99a1eedb:::benchmark_data_smithi045_46415_object40321:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-08-19T14:18:13.581 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:13:02.996325+0000", 2024-08-19T14:18:13.581 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 310.554252921, 2024-08-19T14:18:13.581 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.20255289300000001, 2024-08-19T14:18:13.581 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:18:13.581 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:18:13.581 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:18:13.582 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:18:13.582 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:18:13.582 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:18:13.582 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 40322 2024-08-19T14:18:13.582 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.582 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:18:13.582 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.582 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:18:13.582 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.996325+0000", 2024-08-19T14:18:13.582 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:18:13.582 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.582 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.582 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:18:13.583 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.996325+0000", 2024-08-19T14:18:13.583 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:18:13.583 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.583 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.583 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:18:13.583 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.996326+0000", 2024-08-19T14:18:13.583 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0470000000000001e-06 2024-08-19T14:18:13.583 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.583 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.583 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:18:13.583 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.996355+0000", 2024-08-19T14:18:13.583 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.9167999999999999e-05 2024-08-19T14:18:13.583 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.584 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.584 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:18:13.584 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.996366+0000", 2024-08-19T14:18:13.584 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0679999999999999e-05 2024-08-19T14:18:13.584 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.584 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.584 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:18:13.584 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.996373+0000", 2024-08-19T14:18:13.584 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.5229999999999998e-06 2024-08-19T14:18:13.584 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.584 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.584 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:18:13.584 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.996428+0000", 2024-08-19T14:18:13.584 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.4580000000000003e-05 2024-08-19T14:18:13.585 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.585 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.585 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:18:13.585 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.996560+0000", 2024-08-19T14:18:13.585 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013169399999999999 2024-08-19T14:18:13.585 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.585 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.585 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,7", 2024-08-19T14:18:13.585 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.996700+0000", 2024-08-19T14:18:13.585 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000139973 2024-08-19T14:18:13.585 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.585 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.585 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:18:13.585 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.998183+0000", 2024-08-19T14:18:13.586 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0014834609999999999 2024-08-19T14:18:13.586 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.586 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.586 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:18:13.586 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.998340+0000", 2024-08-19T14:18:13.586 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000156658 2024-08-19T14:18:13.586 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.586 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.586 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:18:13.586 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.198775+0000", 2024-08-19T14:18:13.586 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.20043515100000001 2024-08-19T14:18:13.586 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.586 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.586 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:18:13.587 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.198828+0000", 2024-08-19T14:18:13.587 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.2556999999999998e-05 2024-08-19T14:18:13.587 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.587 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.587 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:18:13.587 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.198878+0000", 2024-08-19T14:18:13.587 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0401000000000001e-05 2024-08-19T14:18:13.587 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:13.587 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:18:13.587 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:13.587 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.587 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.587 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:40331 2.9 2:9a78e0ca:::benchmark_data_smithi045_46415_object40330:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-08-19T14:18:13.587 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:13:03.000136+0000", 2024-08-19T14:18:13.588 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 310.55044217800003, 2024-08-19T14:18:13.588 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.198840613, 2024-08-19T14:18:13.588 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:18:13.588 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:18:13.588 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:18:13.588 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:18:13.588 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:18:13.588 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:18:13.588 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 40331 2024-08-19T14:18:13.588 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.588 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:18:13.588 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.588 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:18:13.588 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.000136+0000", 2024-08-19T14:18:13.589 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:18:13.589 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.589 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.589 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:18:13.589 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.000136+0000", 2024-08-19T14:18:13.589 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:18:13.589 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.589 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.589 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:18:13.589 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.000137+0000", 2024-08-19T14:18:13.589 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.61e-06 2024-08-19T14:18:13.589 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.589 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.590 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:18:13.590 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.000236+0000", 2024-08-19T14:18:13.590 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.8456999999999996e-05 2024-08-19T14:18:13.590 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.590 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.590 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:18:13.590 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.000250+0000", 2024-08-19T14:18:13.590 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4147e-05 2024-08-19T14:18:13.590 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.590 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.590 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:18:13.590 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.000258+0000", 2024-08-19T14:18:13.590 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.0730000000000005e-06 2024-08-19T14:18:13.591 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.591 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.591 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:18:13.591 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.000317+0000", 2024-08-19T14:18:13.591 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.8529000000000002e-05 2024-08-19T14:18:13.591 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.591 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.591 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:18:13.591 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.000446+0000", 2024-08-19T14:18:13.591 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012958499999999999 2024-08-19T14:18:13.591 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.591 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.591 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,7", 2024-08-19T14:18:13.591 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.000591+0000", 2024-08-19T14:18:13.592 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00014517600000000001 2024-08-19T14:18:13.592 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.592 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.592 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:18:13.592 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.003719+0000", 2024-08-19T14:18:13.592 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0031274900000000001 2024-08-19T14:18:13.592 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.592 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.592 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:18:13.592 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.003735+0000", 2024-08-19T14:18:13.592 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6030000000000001e-05 2024-08-19T14:18:13.592 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.592 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.592 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:18:13.593 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.198911+0000", 2024-08-19T14:18:13.593 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.195175671 2024-08-19T14:18:13.593 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.593 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.593 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:18:13.593 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.198942+0000", 2024-08-19T14:18:13.593 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.1293e-05 2024-08-19T14:18:13.593 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.593 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.593 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:18:13.593 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.198976+0000", 2024-08-19T14:18:13.593 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.4551999999999997e-05 2024-08-19T14:18:13.594 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:13.594 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:18:13.594 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:13.594 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.594 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.594 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:40334 2.6 2:65da395f:::benchmark_data_smithi045_46415_object40333:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-08-19T14:18:13.594 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:13:03.000682+0000", 2024-08-19T14:18:13.594 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 310.549895639, 2024-08-19T14:18:13.594 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.198468745, 2024-08-19T14:18:13.594 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:18:13.594 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:18:13.594 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:18:13.594 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:18:13.595 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:18:13.595 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:18:13.595 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 40334 2024-08-19T14:18:13.595 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.595 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:18:13.595 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.595 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:18:13.595 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.000682+0000", 2024-08-19T14:18:13.595 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:18:13.595 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.595 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.595 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:18:13.595 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.000682+0000", 2024-08-19T14:18:13.595 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:18:13.595 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.596 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.596 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:18:13.596 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.000683+0000", 2024-08-19T14:18:13.596 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.6199999999999996e-07 2024-08-19T14:18:13.596 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.596 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.596 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:18:13.596 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.000744+0000", 2024-08-19T14:18:13.596 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.0466e-05 2024-08-19T14:18:13.596 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.596 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.596 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:18:13.596 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.000765+0000", 2024-08-19T14:18:13.596 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.1691e-05 2024-08-19T14:18:13.597 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.597 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:18:13.598 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.598 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:18:13.598 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.000773+0000", 2024-08-19T14:18:13.598 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.1429999999999998e-06 2024-08-19T14:18:13.598 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.598 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.598 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:18:13.598 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.000823+0000", 2024-08-19T14:18:13.599 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.9149e-05 2024-08-19T14:18:13.599 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.599 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.599 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:18:13.599 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.000944+0000", 2024-08-19T14:18:13.599 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012130599999999999 2024-08-19T14:18:13.599 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.599 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.599 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,8", 2024-08-19T14:18:13.599 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.001068+0000", 2024-08-19T14:18:13.599 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012416900000000001 2024-08-19T14:18:13.599 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.599 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.599 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:18:13.600 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.003517+0000", 2024-08-19T14:18:13.600 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0024484210000000001 2024-08-19T14:18:13.600 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.600 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.600 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:18:13.600 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.003557+0000", 2024-08-19T14:18:13.600 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.0033000000000001e-05 2024-08-19T14:18:13.600 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.600 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.600 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:18:13.600 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.199070+0000", 2024-08-19T14:18:13.600 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.19551357799999999 2024-08-19T14:18:13.600 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.600 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.601 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:18:13.601 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.199117+0000", 2024-08-19T14:18:13.601 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.6723e-05 2024-08-19T14:18:13.601 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.601 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.601 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:18:13.601 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.199151+0000", 2024-08-19T14:18:13.601 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.4204e-05 2024-08-19T14:18:13.601 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:13.601 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:18:13.601 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:13.601 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.601 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.601 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:40335 2.9 2:91a68a3e:::benchmark_data_smithi045_46415_object40334:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-08-19T14:18:13.602 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:13:03.000790+0000", 2024-08-19T14:18:13.602 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 310.54978767099999, 2024-08-19T14:18:13.602 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.19867384900000001, 2024-08-19T14:18:13.602 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:18:13.602 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:18:13.602 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:18:13.602 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:18:13.602 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:18:13.602 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:18:13.602 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 40335 2024-08-19T14:18:13.602 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.602 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:18:13.602 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.602 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:18:13.603 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.000790+0000", 2024-08-19T14:18:13.603 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:18:13.603 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.603 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.603 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:18:13.603 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.000790+0000", 2024-08-19T14:18:13.603 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:18:13.603 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.603 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.603 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:18:13.603 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.000791+0000", 2024-08-19T14:18:13.603 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.3799999999999997e-07 2024-08-19T14:18:13.603 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.603 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.604 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:18:13.604 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.000829+0000", 2024-08-19T14:18:13.604 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.8578999999999998e-05 2024-08-19T14:18:13.604 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.604 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.604 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:18:13.604 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.000837+0000", 2024-08-19T14:18:13.604 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.9200000000000004e-06 2024-08-19T14:18:13.604 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.604 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.604 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:18:13.604 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.000855+0000", 2024-08-19T14:18:13.604 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.7261e-05 2024-08-19T14:18:13.605 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.605 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.605 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:18:13.605 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.002117+0000", 2024-08-19T14:18:13.605 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0012625049999999999 2024-08-19T14:18:13.605 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.605 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.605 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:18:13.605 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.002255+0000", 2024-08-19T14:18:13.605 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013797700000000001 2024-08-19T14:18:13.605 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.605 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.605 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,7", 2024-08-19T14:18:13.605 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.002397+0000", 2024-08-19T14:18:13.605 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00014145999999999999 2024-08-19T14:18:13.606 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.606 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.606 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:18:13.606 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.003775+0000", 2024-08-19T14:18:13.606 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0013782759999999999 2024-08-19T14:18:13.606 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.606 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.606 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:18:13.606 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.005174+0000", 2024-08-19T14:18:13.606 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.001399057 2024-08-19T14:18:13.606 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.606 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.606 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:18:13.607 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.199375+0000", 2024-08-19T14:18:13.607 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.19420136099999999 2024-08-19T14:18:13.607 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.607 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.607 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:18:13.607 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.199423+0000", 2024-08-19T14:18:13.607 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.8154e-05 2024-08-19T14:18:13.607 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.607 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.607 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:18:13.607 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.199464+0000", 2024-08-19T14:18:13.607 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.0760999999999998e-05 2024-08-19T14:18:13.607 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:13.607 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:18:13.608 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:13.608 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.608 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.608 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:40336 2.6 2:6d681dd4:::benchmark_data_smithi045_46415_object40335:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-08-19T14:18:13.608 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:13:03.000884+0000", 2024-08-19T14:18:13.608 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 310.54969441999998, 2024-08-19T14:18:13.608 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.19840814800000001, 2024-08-19T14:18:13.608 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:18:13.608 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:18:13.608 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:18:13.608 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:18:13.608 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:18:13.608 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:18:13.608 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 40336 2024-08-19T14:18:13.609 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.609 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:18:13.609 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.609 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:18:13.609 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.000884+0000", 2024-08-19T14:18:13.609 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:18:13.609 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.609 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.609 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:18:13.609 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.000884+0000", 2024-08-19T14:18:13.609 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:18:13.609 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.609 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.609 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:18:13.609 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.000884+0000", 2024-08-19T14:18:13.610 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.9699999999999996e-07 2024-08-19T14:18:13.610 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.610 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.610 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:18:13.610 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.000926+0000", 2024-08-19T14:18:13.610 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.1542999999999998e-05 2024-08-19T14:18:13.610 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.610 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.610 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:18:13.610 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.000934+0000", 2024-08-19T14:18:13.610 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.8610000000000004e-06 2024-08-19T14:18:13.610 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.610 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.611 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:18:13.611 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.000940+0000", 2024-08-19T14:18:13.611 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.7289999999999997e-06 2024-08-19T14:18:13.611 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.611 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.611 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:18:13.611 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.002371+0000", 2024-08-19T14:18:13.611 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0014310169999999999 2024-08-19T14:18:13.611 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.611 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.611 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:18:13.611 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.002486+0000", 2024-08-19T14:18:13.611 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000114317 2024-08-19T14:18:13.611 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.612 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.612 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,8", 2024-08-19T14:18:13.612 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.002611+0000", 2024-08-19T14:18:13.612 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012566899999999999 2024-08-19T14:18:13.612 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.612 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.612 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:18:13.612 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.003645+0000", 2024-08-19T14:18:13.612 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.001034135 2024-08-19T14:18:13.612 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.612 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.613 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:18:13.613 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.004552+0000", 2024-08-19T14:18:13.613 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00090656799999999996 2024-08-19T14:18:13.613 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.613 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.613 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:18:13.613 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.199200+0000", 2024-08-19T14:18:13.613 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.194647869 2024-08-19T14:18:13.613 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.613 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.613 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:18:13.613 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.199256+0000", 2024-08-19T14:18:13.613 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.5781e-05 2024-08-19T14:18:13.613 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.614 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.614 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:18:13.614 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.199292+0000", 2024-08-19T14:18:13.614 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.6161999999999997e-05 2024-08-19T14:18:13.614 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:13.614 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:18:13.614 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:13.614 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.614 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.614 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:40337 2.9 2:9e869513:::benchmark_data_smithi045_46415_object40336:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-08-19T14:18:13.614 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:13:03.000952+0000", 2024-08-19T14:18:13.615 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 310.54962645199998, 2024-08-19T14:18:13.615 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.198664388, 2024-08-19T14:18:13.615 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:18:13.615 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:18:13.615 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:18:13.615 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:18:13.615 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:18:13.615 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:18:13.615 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 40337 2024-08-19T14:18:13.615 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.615 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:18:13.615 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.616 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:18:13.616 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.000952+0000", 2024-08-19T14:18:13.616 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:18:13.616 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.616 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.616 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:18:13.616 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.000952+0000", 2024-08-19T14:18:13.616 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:18:13.616 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.616 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.616 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:18:13.617 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.000952+0000", 2024-08-19T14:18:13.617 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.3000000000000001e-07 2024-08-19T14:18:13.617 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.617 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.617 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:18:13.617 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.000997+0000", 2024-08-19T14:18:13.617 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4558999999999998e-05 2024-08-19T14:18:13.617 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.617 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.617 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:18:13.617 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.001004+0000", 2024-08-19T14:18:13.617 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.8420000000000005e-06 2024-08-19T14:18:13.617 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.618 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.618 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:18:13.618 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.001011+0000", 2024-08-19T14:18:13.618 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.3280000000000002e-06 2024-08-19T14:18:13.618 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.618 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.618 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:18:13.618 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.003899+0000", 2024-08-19T14:18:13.618 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.002888757 2024-08-19T14:18:13.618 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.618 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.618 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:18:13.618 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.004004+0000", 2024-08-19T14:18:13.619 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000104138 2024-08-19T14:18:13.619 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.619 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.619 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,7", 2024-08-19T14:18:13.619 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.004135+0000", 2024-08-19T14:18:13.619 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013134900000000001 2024-08-19T14:18:13.619 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.619 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.619 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:18:13.619 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.005347+0000", 2024-08-19T14:18:13.619 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.001212409 2024-08-19T14:18:13.619 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.619 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.619 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:18:13.620 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.005771+0000", 2024-08-19T14:18:13.620 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00042346199999999999 2024-08-19T14:18:13.620 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.620 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.620 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:18:13.620 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.199507+0000", 2024-08-19T14:18:13.620 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.19373660500000001 2024-08-19T14:18:13.620 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.620 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.620 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:18:13.620 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.199578+0000", 2024-08-19T14:18:13.620 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.0559e-05 2024-08-19T14:18:13.620 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.620 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.621 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:18:13.621 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.199616+0000", 2024-08-19T14:18:13.621 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7950000000000001e-05 2024-08-19T14:18:13.621 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:13.621 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:18:13.621 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:13.621 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.621 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.621 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:241301 2.4 2:26065fe2:::benchmark_data_smithi045_46415_object118020:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-08-19T14:18:13.621 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:16:40.774899+0000", 2024-08-19T14:18:13.621 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 92.775679350000004, 2024-08-19T14:18:13.621 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.29898516600000002, 2024-08-19T14:18:13.621 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:18:13.621 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:18:13.622 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:18:13.622 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:18:13.622 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:18:13.622 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:18:13.622 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 241301 2024-08-19T14:18:13.622 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.622 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:18:13.622 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.622 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:18:13.622 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.774899+0000", 2024-08-19T14:18:13.622 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:18:13.622 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.622 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.622 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:18:13.623 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.774899+0000", 2024-08-19T14:18:13.623 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:18:13.623 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.623 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.623 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:18:13.623 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.774900+0000", 2024-08-19T14:18:13.623 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6190000000000001e-06 2024-08-19T14:18:13.623 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.623 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.623 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:18:13.623 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.774907+0000", 2024-08-19T14:18:13.623 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.6540000000000002e-06 2024-08-19T14:18:13.623 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.624 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.624 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:18:13.624 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.774921+0000", 2024-08-19T14:18:13.624 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4355000000000001e-05 2024-08-19T14:18:13.624 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.624 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.624 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:18:13.624 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.774928+0000", 2024-08-19T14:18:13.624 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.2339999999999997e-06 2024-08-19T14:18:13.624 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.625 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.625 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:18:13.625 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.780109+0000", 2024-08-19T14:18:13.625 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0051809409999999997 2024-08-19T14:18:13.625 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.625 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.625 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:18:13.625 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.780240+0000", 2024-08-19T14:18:13.625 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000131006 2024-08-19T14:18:13.625 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.625 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.625 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,2,8", 2024-08-19T14:18:13.625 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.780378+0000", 2024-08-19T14:18:13.626 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013800999999999999 2024-08-19T14:18:13.626 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.626 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.626 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:18:13.626 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.794340+0000", 2024-08-19T14:18:13.626 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.013961559 2024-08-19T14:18:13.626 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.626 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.626 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:18:13.627 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.802616+0000", 2024-08-19T14:18:13.627 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0082755490000000001 2024-08-19T14:18:13.627 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.627 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.627 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:18:13.627 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.809365+0000", 2024-08-19T14:18:13.627 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0067497039999999996 2024-08-19T14:18:13.627 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.627 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.627 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:18:13.627 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.073684+0000", 2024-08-19T14:18:13.627 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.26431834199999998 2024-08-19T14:18:13.627 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.628 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.628 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:18:13.628 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.073749+0000", 2024-08-19T14:18:13.628 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.5343000000000001e-05 2024-08-19T14:18:13.628 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.628 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.628 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:18:13.628 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.073884+0000", 2024-08-19T14:18:13.628 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013485 2024-08-19T14:18:13.628 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:13.628 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:18:13.628 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:13.628 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.628 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.628 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:241310 2.4 2:2df264d1:::benchmark_data_smithi045_46415_object118029:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-08-19T14:18:13.629 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:16:40.777745+0000", 2024-08-19T14:18:13.629 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 92.772832860999998, 2024-08-19T14:18:13.629 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.34201599500000002, 2024-08-19T14:18:13.629 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:18:13.629 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:18:13.629 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:18:13.629 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:18:13.629 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:18:13.629 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:18:13.629 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 241310 2024-08-19T14:18:13.629 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.629 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:18:13.629 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.629 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:18:13.630 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.777745+0000", 2024-08-19T14:18:13.630 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:18:13.630 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.630 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.630 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:18:13.630 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.777745+0000", 2024-08-19T14:18:13.630 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:18:13.630 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.630 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.630 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:18:13.630 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.777746+0000", 2024-08-19T14:18:13.630 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.5999999999999991e-07 2024-08-19T14:18:13.630 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.631 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.631 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:18:13.631 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.777751+0000", 2024-08-19T14:18:13.631 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.6369999999999998e-06 2024-08-19T14:18:13.631 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.631 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.631 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:18:13.631 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.777763+0000", 2024-08-19T14:18:13.631 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2279000000000001e-05 2024-08-19T14:18:13.631 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.631 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.631 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:18:13.631 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.777771+0000", 2024-08-19T14:18:13.631 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.6109999999999996e-06 2024-08-19T14:18:13.632 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.632 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.632 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:18:13.632 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.788985+0000", 2024-08-19T14:18:13.632 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.011214636 2024-08-19T14:18:13.632 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.632 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.632 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:18:13.632 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.789090+0000", 2024-08-19T14:18:13.632 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000104583 2024-08-19T14:18:13.632 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.632 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.632 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,2,8", 2024-08-19T14:18:13.632 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.789223+0000", 2024-08-19T14:18:13.632 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013313299999999999 2024-08-19T14:18:13.633 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.633 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.633 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:18:13.633 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.802660+0000", 2024-08-19T14:18:13.633 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.013437065999999999 2024-08-19T14:18:13.633 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.633 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.633 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:18:13.633 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.808077+0000", 2024-08-19T14:18:13.633 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0054167759999999999 2024-08-19T14:18:13.633 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.633 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.633 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:18:13.633 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.812929+0000", 2024-08-19T14:18:13.634 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0048518399999999996 2024-08-19T14:18:13.634 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.634 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.634 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:18:13.634 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.119563+0000", 2024-08-19T14:18:13.634 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.30663414300000003 2024-08-19T14:18:13.634 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.634 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.634 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:18:13.634 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.119627+0000", 2024-08-19T14:18:13.634 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.4622999999999994e-05 2024-08-19T14:18:13.634 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.634 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.635 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:18:13.635 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.119761+0000", 2024-08-19T14:18:13.635 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013370799999999999 2024-08-19T14:18:13.635 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:13.635 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:18:13.635 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:13.635 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.635 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.635 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:241313 2.4 2:22ac58a2:::benchmark_data_smithi045_46415_object118032:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-08-19T14:18:13.635 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:16:40.785407+0000", 2024-08-19T14:18:13.635 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 92.765171018000004, 2024-08-19T14:18:13.635 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.33459377200000001, 2024-08-19T14:18:13.635 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:18:13.635 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:18:13.636 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:18:13.636 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:18:13.636 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:18:13.636 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:18:13.636 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 241313 2024-08-19T14:18:13.636 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.636 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:18:13.636 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.636 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:18:13.636 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.785407+0000", 2024-08-19T14:18:13.636 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:18:13.636 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.636 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.636 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:18:13.637 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.785407+0000", 2024-08-19T14:18:13.637 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:18:13.637 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.637 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.637 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:18:13.637 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.785409+0000", 2024-08-19T14:18:13.637 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6109999999999999e-06 2024-08-19T14:18:13.637 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.637 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.637 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:18:13.637 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.785414+0000", 2024-08-19T14:18:13.637 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1919999999999996e-06 2024-08-19T14:18:13.638 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.638 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.638 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:18:13.638 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.785424+0000", 2024-08-19T14:18:13.638 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0719e-05 2024-08-19T14:18:13.638 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.638 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.638 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:18:13.638 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.785430+0000", 2024-08-19T14:18:13.638 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.8019999999999997e-06 2024-08-19T14:18:13.638 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.638 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.638 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:18:13.639 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.794371+0000", 2024-08-19T14:18:13.639 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0089406929999999996 2024-08-19T14:18:13.639 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.639 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.639 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:18:13.639 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.794483+0000", 2024-08-19T14:18:13.639 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000111539 2024-08-19T14:18:13.639 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.639 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.639 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,2,8", 2024-08-19T14:18:13.639 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.794630+0000", 2024-08-19T14:18:13.639 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00014782700000000001 2024-08-19T14:18:13.639 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.639 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.640 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:18:13.640 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.802700+0000", 2024-08-19T14:18:13.640 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0080701290000000005 2024-08-19T14:18:13.640 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.640 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.640 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:18:13.640 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.808117+0000", 2024-08-19T14:18:13.640 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0054165469999999999 2024-08-19T14:18:13.640 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.640 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.640 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:18:13.640 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.812933+0000", 2024-08-19T14:18:13.640 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0048158460000000004 2024-08-19T14:18:13.640 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.642 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.642 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:18:13.642 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.119824+0000", 2024-08-19T14:18:13.642 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.30689145899999998 2024-08-19T14:18:13.642 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.642 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.642 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:18:13.642 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.119884+0000", 2024-08-19T14:18:13.642 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.9642000000000003e-05 2024-08-19T14:18:13.643 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.643 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.643 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:18:13.643 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.120001+0000", 2024-08-19T14:18:13.643 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000116766 2024-08-19T14:18:13.643 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:13.643 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:18:13.643 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:13.643 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.643 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.643 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:241315 2.4 2:24bb1245:::benchmark_data_smithi045_46415_object118034:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-08-19T14:18:13.643 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:16:40.789772+0000", 2024-08-19T14:18:13.643 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 92.760805571999995, 2024-08-19T14:18:13.643 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.33090881500000002, 2024-08-19T14:18:13.644 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:18:13.644 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:18:13.644 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:18:13.644 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:18:13.644 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:18:13.644 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:18:13.644 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 241315 2024-08-19T14:18:13.644 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.644 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:18:13.644 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.644 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:18:13.644 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.789772+0000", 2024-08-19T14:18:13.644 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:18:13.644 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.645 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.645 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:18:13.645 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.789772+0000", 2024-08-19T14:18:13.645 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:18:13.645 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.645 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.645 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:18:13.645 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.789774+0000", 2024-08-19T14:18:13.645 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.483e-06 2024-08-19T14:18:13.645 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.645 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.645 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:18:13.645 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.789780+0000", 2024-08-19T14:18:13.646 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.9390000000000002e-06 2024-08-19T14:18:13.646 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.646 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.646 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:18:13.646 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.789796+0000", 2024-08-19T14:18:13.646 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5889999999999999e-05 2024-08-19T14:18:13.646 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.646 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.646 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:18:13.646 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.789803+0000", 2024-08-19T14:18:13.646 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.3900000000000004e-06 2024-08-19T14:18:13.646 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.646 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.646 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:18:13.647 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.802735+0000", 2024-08-19T14:18:13.647 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.012932332 2024-08-19T14:18:13.647 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.647 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.647 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:18:13.647 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.802891+0000", 2024-08-19T14:18:13.647 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000155301 2024-08-19T14:18:13.647 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.647 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.647 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,2,8", 2024-08-19T14:18:13.647 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.803031+0000", 2024-08-19T14:18:13.647 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000140414 2024-08-19T14:18:13.647 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.647 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.648 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:18:13.648 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.809352+0000", 2024-08-19T14:18:13.648 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0063210699999999998 2024-08-19T14:18:13.648 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.648 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.648 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:18:13.648 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.812937+0000", 2024-08-19T14:18:13.648 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0035844050000000001 2024-08-19T14:18:13.649 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.649 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.649 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:18:13.649 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.812965+0000", 2024-08-19T14:18:13.649 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.8099999999999999e-05 2024-08-19T14:18:13.649 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.649 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.649 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:18:13.649 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.120323+0000", 2024-08-19T14:18:13.649 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.30735870700000001 2024-08-19T14:18:13.649 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.649 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.649 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:18:13.649 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.120566+0000", 2024-08-19T14:18:13.650 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00024232400000000001 2024-08-19T14:18:13.650 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.650 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.650 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:18:13.650 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.120681+0000", 2024-08-19T14:18:13.650 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00011546 2024-08-19T14:18:13.650 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:13.650 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:18:13.650 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:13.650 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.650 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.650 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:241316 2.4 2:2a938701:::benchmark_data_smithi045_46415_object118035:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-08-19T14:18:13.650 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:16:40.789816+0000", 2024-08-19T14:18:13.651 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 92.760761500000001, 2024-08-19T14:18:13.651 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.33160296900000003, 2024-08-19T14:18:13.651 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:18:13.651 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:18:13.651 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:18:13.651 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:18:13.651 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:18:13.651 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:18:13.651 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 241316 2024-08-19T14:18:13.651 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.651 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:18:13.651 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.651 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:18:13.652 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.789816+0000", 2024-08-19T14:18:13.652 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:18:13.652 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.652 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.652 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:18:13.652 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.789816+0000", 2024-08-19T14:18:13.652 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:18:13.652 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.652 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.652 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:18:13.652 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.789817+0000", 2024-08-19T14:18:13.653 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.0699999999999998e-07 2024-08-19T14:18:13.653 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.653 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.653 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:18:13.653 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.789819+0000", 2024-08-19T14:18:13.653 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.4380000000000002e-06 2024-08-19T14:18:13.653 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.653 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.653 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:18:13.653 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.789826+0000", 2024-08-19T14:18:13.653 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.4060000000000002e-06 2024-08-19T14:18:13.653 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.653 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.653 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:18:13.654 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.789831+0000", 2024-08-19T14:18:13.654 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.8960000000000002e-06 2024-08-19T14:18:13.654 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.654 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.654 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:18:13.654 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.812990+0000", 2024-08-19T14:18:13.654 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.023159863999999999 2024-08-19T14:18:13.654 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.654 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.654 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:18:13.654 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.813089+0000", 2024-08-19T14:18:13.654 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.8478000000000001e-05 2024-08-19T14:18:13.654 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.654 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.655 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,2,8", 2024-08-19T14:18:13.655 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.813175+0000", 2024-08-19T14:18:13.655 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.5834000000000005e-05 2024-08-19T14:18:13.655 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.655 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.655 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:18:13.655 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.818905+0000", 2024-08-19T14:18:13.655 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0057298870000000003 2024-08-19T14:18:13.655 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.655 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.655 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:18:13.655 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.820238+0000", 2024-08-19T14:18:13.655 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.001333603 2024-08-19T14:18:13.655 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.656 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.656 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:18:13.656 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.831561+0000", 2024-08-19T14:18:13.656 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.011322334 2024-08-19T14:18:13.656 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.656 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.656 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:18:13.656 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.120731+0000", 2024-08-19T14:18:13.656 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.289170123 2024-08-19T14:18:13.656 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.656 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.656 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:18:13.656 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.121237+0000", 2024-08-19T14:18:13.656 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00050635100000000004 2024-08-19T14:18:13.657 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.657 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.657 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:18:13.657 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.121419+0000", 2024-08-19T14:18:13.657 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00018234800000000001 2024-08-19T14:18:13.657 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:13.657 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:18:13.657 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:13.657 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.657 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.657 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:241344 2.4 2:2b291422:::benchmark_data_smithi045_46415_object118063:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-08-19T14:18:13.657 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:16:40.823839+0000", 2024-08-19T14:18:13.657 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 92.726738682999994, 2024-08-19T14:18:13.657 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.29983483399999999, 2024-08-19T14:18:13.658 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:18:13.658 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:18:13.658 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:18:13.658 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:18:13.658 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:18:13.658 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:18:13.658 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 241344 2024-08-19T14:18:13.658 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.658 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:18:13.658 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.658 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:18:13.658 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.823839+0000", 2024-08-19T14:18:13.658 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:18:13.658 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.659 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.659 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:18:13.659 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.823839+0000", 2024-08-19T14:18:13.659 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:18:13.659 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.659 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.659 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:18:13.659 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.823841+0000", 2024-08-19T14:18:13.659 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.607e-06 2024-08-19T14:18:13.659 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.659 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.659 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:18:13.659 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.823847+0000", 2024-08-19T14:18:13.659 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.7620000000000001e-06 2024-08-19T14:18:13.660 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.660 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.660 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:18:13.660 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.823865+0000", 2024-08-19T14:18:13.660 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.8134999999999998e-05 2024-08-19T14:18:13.660 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.660 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.660 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:18:13.660 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.823872+0000", 2024-08-19T14:18:13.660 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.7039999999999994e-06 2024-08-19T14:18:13.660 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.660 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.660 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:18:13.661 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.824022+0000", 2024-08-19T14:18:13.661 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000149405 2024-08-19T14:18:13.661 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.661 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.661 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:18:13.661 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.824180+0000", 2024-08-19T14:18:13.661 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00015828199999999999 2024-08-19T14:18:13.661 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.661 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.661 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,2,8", 2024-08-19T14:18:13.661 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.824332+0000", 2024-08-19T14:18:13.661 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00015196799999999999 2024-08-19T14:18:13.661 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.661 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.662 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:18:13.662 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.831601+0000", 2024-08-19T14:18:13.662 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0072684150000000003 2024-08-19T14:18:13.662 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.662 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.662 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:18:13.662 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.832795+0000", 2024-08-19T14:18:13.662 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0011945739999999999 2024-08-19T14:18:13.662 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.662 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.662 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:18:13.662 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.843115+0000", 2024-08-19T14:18:13.662 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.010319488 2024-08-19T14:18:13.662 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.663 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.663 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:18:13.663 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.121468+0000", 2024-08-19T14:18:13.663 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.27835382199999997 2024-08-19T14:18:13.663 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.663 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.663 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:18:13.663 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.123536+0000", 2024-08-19T14:18:13.663 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.002067892 2024-08-19T14:18:13.663 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.663 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.663 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:18:13.663 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.123674+0000", 2024-08-19T14:18:13.663 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013778 2024-08-19T14:18:13.664 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:13.664 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:18:13.664 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:13.664 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.664 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.664 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:241352 2.4 2:258e31fc:::benchmark_data_smithi045_46415_object118071:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-08-19T14:18:13.664 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:16:40.833613+0000", 2024-08-19T14:18:13.664 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 92.716964774000004, 2024-08-19T14:18:13.664 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.290304479, 2024-08-19T14:18:13.664 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:18:13.664 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:18:13.665 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:18:13.665 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:18:13.665 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:18:13.665 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:18:13.665 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 241352 2024-08-19T14:18:13.665 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.665 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:18:13.665 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.665 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:18:13.665 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.833613+0000", 2024-08-19T14:18:13.665 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:18:13.665 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.665 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.666 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:18:13.666 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.833613+0000", 2024-08-19T14:18:13.666 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:18:13.666 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.666 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.666 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:18:13.666 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.833615+0000", 2024-08-19T14:18:13.666 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4839999999999999e-06 2024-08-19T14:18:13.666 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.666 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.666 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:18:13.666 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.833620+0000", 2024-08-19T14:18:13.666 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.5740000000000003e-06 2024-08-19T14:18:13.667 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.667 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.667 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:18:13.667 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.833635+0000", 2024-08-19T14:18:13.667 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4893e-05 2024-08-19T14:18:13.667 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.667 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.667 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:18:13.667 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.833642+0000", 2024-08-19T14:18:13.667 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.2620000000000001e-06 2024-08-19T14:18:13.667 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.667 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.667 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:18:13.667 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.836631+0000", 2024-08-19T14:18:13.668 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0029890400000000001 2024-08-19T14:18:13.668 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.668 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.668 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:18:13.668 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.836733+0000", 2024-08-19T14:18:13.668 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000101532 2024-08-19T14:18:13.668 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.668 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.668 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,2,8", 2024-08-19T14:18:13.668 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.836833+0000", 2024-08-19T14:18:13.668 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.9652000000000001e-05 2024-08-19T14:18:13.668 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.668 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.669 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:18:13.669 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.841532+0000", 2024-08-19T14:18:13.669 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0046991769999999997 2024-08-19T14:18:13.669 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.669 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.669 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:18:13.669 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.843144+0000", 2024-08-19T14:18:13.669 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0016120139999999999 2024-08-19T14:18:13.669 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.669 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.669 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:18:13.669 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.843187+0000", 2024-08-19T14:18:13.669 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.3198000000000002e-05 2024-08-19T14:18:13.669 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.670 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.670 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:18:13.670 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.123725+0000", 2024-08-19T14:18:13.670 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.28053793599999999 2024-08-19T14:18:13.670 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.670 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.670 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:18:13.670 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.123784+0000", 2024-08-19T14:18:13.670 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.8931000000000002e-05 2024-08-19T14:18:13.670 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.670 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.670 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:18:13.670 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.123918+0000", 2024-08-19T14:18:13.670 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000133786 2024-08-19T14:18:13.671 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:13.671 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:18:13.671 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:13.671 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.671 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.671 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:241376 2.4 2:259a9829:::benchmark_data_smithi045_46415_object118095:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-08-19T14:18:13.671 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:16:40.865722+0000", 2024-08-19T14:18:13.671 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 92.684855825, 2024-08-19T14:18:13.671 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.25843650499999998, 2024-08-19T14:18:13.671 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:18:13.671 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:18:13.671 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:18:13.671 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:18:13.672 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:18:13.672 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:18:13.672 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 241376 2024-08-19T14:18:13.672 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.672 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:18:13.672 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.672 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:18:13.672 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.865722+0000", 2024-08-19T14:18:13.672 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:18:13.672 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.672 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.672 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:18:13.672 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.865722+0000", 2024-08-19T14:18:13.672 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:18:13.673 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.673 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.673 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:18:13.673 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.865724+0000", 2024-08-19T14:18:13.673 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6619999999999999e-06 2024-08-19T14:18:13.673 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.673 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.673 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:18:13.673 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.865729+0000", 2024-08-19T14:18:13.673 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7040000000000002e-06 2024-08-19T14:18:13.673 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.673 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.673 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:18:13.673 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.865739+0000", 2024-08-19T14:18:13.674 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0541e-05 2024-08-19T14:18:13.674 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.674 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.674 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:18:13.674 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.865745+0000", 2024-08-19T14:18:13.674 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.6609999999999997e-06 2024-08-19T14:18:13.674 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.674 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.674 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:18:13.674 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.865926+0000", 2024-08-19T14:18:13.674 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00018153000000000001 2024-08-19T14:18:13.674 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.674 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.675 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:18:13.675 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.866094+0000", 2024-08-19T14:18:13.675 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000167844 2024-08-19T14:18:13.675 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.675 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.675 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,2,8", 2024-08-19T14:18:13.675 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.866236+0000", 2024-08-19T14:18:13.675 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00014171600000000001 2024-08-19T14:18:13.675 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.675 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.675 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:18:13.675 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.871503+0000", 2024-08-19T14:18:13.675 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0052669960000000004 2024-08-19T14:18:13.675 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.676 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.676 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:18:13.676 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.871600+0000", 2024-08-19T14:18:13.676 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.7027000000000005e-05 2024-08-19T14:18:13.676 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.676 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.676 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:18:13.676 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.872043+0000", 2024-08-19T14:18:13.676 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00044304300000000001 2024-08-19T14:18:13.676 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.676 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.676 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:18:13.676 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.123962+0000", 2024-08-19T14:18:13.676 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.25191884199999998 2024-08-19T14:18:13.677 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.677 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.677 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:18:13.677 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.124022+0000", 2024-08-19T14:18:13.677 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.0331999999999998e-05 2024-08-19T14:18:13.677 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.677 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.677 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:18:13.677 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.124159+0000", 2024-08-19T14:18:13.677 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013660699999999999 2024-08-19T14:18:13.677 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:13.677 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:18:13.677 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:13.677 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.678 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.678 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:241392 2.4 2:29469d82:::benchmark_data_smithi045_46415_object118111:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-08-19T14:18:13.678 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:16:40.925642+0000", 2024-08-19T14:18:13.678 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 92.624936000000005, 2024-08-19T14:18:13.678 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.198757618, 2024-08-19T14:18:13.678 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:18:13.678 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:18:13.678 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:18:13.678 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:18:13.678 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:18:13.678 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:18:13.678 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 241392 2024-08-19T14:18:13.678 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.678 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:18:13.679 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.679 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:18:13.679 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.925642+0000", 2024-08-19T14:18:13.679 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:18:13.679 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.679 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.679 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:18:13.679 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.925642+0000", 2024-08-19T14:18:13.679 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:18:13.679 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.679 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.679 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:18:13.679 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.925644+0000", 2024-08-19T14:18:13.679 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.7680000000000001e-06 2024-08-19T14:18:13.680 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.680 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.680 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:18:13.680 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.925650+0000", 2024-08-19T14:18:13.680 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.7000000000000002e-06 2024-08-19T14:18:13.680 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.680 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.680 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:18:13.680 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.966573+0000", 2024-08-19T14:18:13.680 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.040922839000000003 2024-08-19T14:18:13.680 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.680 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.680 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:18:13.680 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.966586+0000", 2024-08-19T14:18:13.681 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2417e-05 2024-08-19T14:18:13.681 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.681 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.681 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:18:13.681 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.966819+0000", 2024-08-19T14:18:13.681 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00023348800000000001 2024-08-19T14:18:13.681 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.681 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.681 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:18:13.681 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.966979+0000", 2024-08-19T14:18:13.681 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00015934900000000001 2024-08-19T14:18:13.681 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.681 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.682 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,2,8", 2024-08-19T14:18:13.682 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.967132+0000", 2024-08-19T14:18:13.682 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00015313800000000001 2024-08-19T14:18:13.682 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.682 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.682 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:18:13.682 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.972222+0000", 2024-08-19T14:18:13.682 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0050908170000000001 2024-08-19T14:18:13.682 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.682 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.682 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:18:13.682 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.972235+0000", 2024-08-19T14:18:13.682 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2211e-05 2024-08-19T14:18:13.683 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.683 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.683 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:18:13.683 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.972265+0000", 2024-08-19T14:18:13.683 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.9802999999999999e-05 2024-08-19T14:18:13.683 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.683 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.683 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:18:13.683 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.124204+0000", 2024-08-19T14:18:13.683 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.15193910799999999 2024-08-19T14:18:13.683 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.683 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.683 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:18:13.683 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.124263+0000", 2024-08-19T14:18:13.683 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.9719000000000002e-05 2024-08-19T14:18:13.684 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.684 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.684 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:18:13.684 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.124400+0000", 2024-08-19T14:18:13.684 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013626099999999999 2024-08-19T14:18:13.684 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:13.684 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:18:13.684 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:13.684 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.684 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.684 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25057.0:47840 3.14 3:2da72587:::benchmark_data_smithi045_59973_object47839:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e234)", 2024-08-19T14:18:13.684 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:17:18.780442+0000", 2024-08-19T14:18:13.684 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 54.770135826999997, 2024-08-19T14:18:13.685 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.98449229999999999, 2024-08-19T14:18:13.685 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:18:13.685 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:18:13.685 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:18:13.685 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:18:13.685 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25057", 2024-08-19T14:18:13.685 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:18:13.685 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 47840 2024-08-19T14:18:13.685 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.685 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:18:13.685 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.685 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:18:13.685 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.780442+0000", 2024-08-19T14:18:13.685 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:18:13.686 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.686 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.686 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:18:13.686 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.780442+0000", 2024-08-19T14:18:13.686 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:18:13.686 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.686 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.686 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:18:13.686 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.780444+0000", 2024-08-19T14:18:13.686 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.216e-06 2024-08-19T14:18:13.686 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.686 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.686 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:18:13.686 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.780527+0000", 2024-08-19T14:18:13.687 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.3096000000000005e-05 2024-08-19T14:18:13.687 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.687 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.687 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:18:13.687 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.780548+0000", 2024-08-19T14:18:13.687 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.0424999999999998e-05 2024-08-19T14:18:13.687 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.687 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.687 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:18:13.687 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.780559+0000", 2024-08-19T14:18:13.687 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1579999999999999e-05 2024-08-19T14:18:13.687 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.687 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.687 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:18:13.688 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.815099+0000", 2024-08-19T14:18:13.688 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.034539489999999999 2024-08-19T14:18:13.688 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.688 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.688 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:18:13.688 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.815109+0000", 2024-08-19T14:18:13.688 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0036e-05 2024-08-19T14:18:13.688 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.688 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.688 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:18:13.688 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.756630+0000", 2024-08-19T14:18:13.688 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.94152143899999996 2024-08-19T14:18:13.688 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.689 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.689 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:18:13.689 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.756793+0000", 2024-08-19T14:18:13.689 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00016217400000000001 2024-08-19T14:18:13.689 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.689 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.689 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 5,11", 2024-08-19T14:18:13.689 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.756983+0000", 2024-08-19T14:18:13.689 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00019000000000000001 2024-08-19T14:18:13.689 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.689 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.689 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:18:13.689 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.759245+0000", 2024-08-19T14:18:13.689 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0022623140000000001 2024-08-19T14:18:13.690 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.690 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.690 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:18:13.690 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.762237+0000", 2024-08-19T14:18:13.690 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0029923850000000002 2024-08-19T14:18:13.690 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.690 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.690 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:18:13.690 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.764795+0000", 2024-08-19T14:18:13.690 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0025578240000000002 2024-08-19T14:18:13.690 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.690 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.690 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:18:13.690 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.764889+0000", 2024-08-19T14:18:13.691 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.3927999999999999e-05 2024-08-19T14:18:13.691 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:13.691 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:13.691 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:18:13.691 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.764934+0000", 2024-08-19T14:18:13.691 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.5392999999999997e-05 2024-08-19T14:18:13.691 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:13.691 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:18:13.691 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:13.691 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:13.691 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:18:13.691 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:18:13.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:13.693 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:13.693 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:18:13.794 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:18:13.833 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:13.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:13.834 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:18:13.861 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-19T14:18:13.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:13 smithi031 ceph-mon[16336]: pgmap v615: 47 pgs: 2 active+recovering+undersized+remapped, 45 active+clean; 6.1 GiB data, 22 GiB used, 783 GiB / 805 GiB avail; 98 MiB/s wr, 1.56k op/s; 2049/301209 objects misplaced (0.680%); 18 MiB/s, 289 objects/s recovering 2024-08-19T14:18:13.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:13 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-08-19T14:18:13.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:13 smithi031 ceph-mon[16336]: osdmap e267: 12 total, 12 up, 9 in 2024-08-19T14:18:13.935 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:18:13.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:13.973 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:13.973 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:18:14.074 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:18:14.106 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:14.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:14.107 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:18:14.208 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:18:14.243 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:14.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:14.244 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:18:14.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:13 smithi032 ceph-mon[20004]: pgmap v615: 47 pgs: 2 active+recovering+undersized+remapped, 45 active+clean; 6.1 GiB data, 22 GiB used, 783 GiB / 805 GiB avail; 98 MiB/s wr, 1.56k op/s; 2049/301209 objects misplaced (0.680%); 18 MiB/s, 289 objects/s recovering 2024-08-19T14:18:14.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:13 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-08-19T14:18:14.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:13 smithi032 ceph-mon[20004]: osdmap e267: 12 total, 12 up, 9 in 2024-08-19T14:18:14.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:13 smithi045 ceph-mon[20143]: pgmap v615: 47 pgs: 2 active+recovering+undersized+remapped, 45 active+clean; 6.1 GiB data, 22 GiB used, 783 GiB / 805 GiB avail; 98 MiB/s wr, 1.56k op/s; 2049/301209 objects misplaced (0.680%); 18 MiB/s, 289 objects/s recovering 2024-08-19T14:18:14.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:13 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-08-19T14:18:14.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:13 smithi045 ceph-mon[20143]: osdmap e267: 12 total, 12 up, 9 in 2024-08-19T14:18:14.317 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 72 16 108064 108048 93.7801 159.688 0.00461122 0.010659 2024-08-19T14:18:14.345 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:18:14.378 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:14.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:14.379 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:18:14.480 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:18:14.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:14.518 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:14.519 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:18:14.619 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:18:14.639 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-08-19T14:18:14.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:14.675 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:14.675 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:18:14.716 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:18:14.776 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:18:14.811 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:14.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:14.812 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:18:14.855 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:14 smithi031 ceph-mon[16336]: osdmap e268: 12 total, 12 up, 9 in 2024-08-19T14:18:14.855 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:14 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-08-19T14:18:14.855 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:14 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-08-19T14:18:14.913 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:18:14.955 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:14.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:14.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:18:15.056 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:18:15.057 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:18:15.084 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:15.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:15.085 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:18:15.186 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:18:15.219 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:15.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:15.220 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:18:15.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:14 smithi032 ceph-mon[20004]: osdmap e268: 12 total, 12 up, 9 in 2024-08-19T14:18:15.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:14 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-08-19T14:18:15.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:14 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-08-19T14:18:15.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:14 smithi045 ceph-mon[20143]: osdmap e268: 12 total, 12 up, 9 in 2024-08-19T14:18:15.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:14 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-08-19T14:18:15.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:14 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-08-19T14:18:15.317 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 73 16 110441 110425 94.5304 148.562 0.00910915 0.0105744 2024-08-19T14:18:15.321 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:18:15.370 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:18:15.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:15.376 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:15.376 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:18:15.477 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:18:15.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:15.519 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:15.519 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:18:15.619 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:18:15.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:15.649 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:15.650 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:18:15.751 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:18:15.780 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:15.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:15.782 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:18:15.808 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is unset 2024-08-19T14:18:15.883 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:18:15.904 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:15.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:15.905 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:18:15.918 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:18:15.918 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-08-19T14:18:15.918 INFO:teuthology.orchestra.run.smithi031.stdout: "num_ops": 0 2024-08-19T14:18:15.919 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:18:16.007 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:18:16.040 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:16.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:16.041 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:18:16.106 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:15 smithi031 ceph-mon[16336]: pgmap v618: 57 pgs: 4 peering, 10 unknown, 2 active+recovering+undersized+remapped, 41 active+clean; 6.3 GiB data, 23 GiB used, 782 GiB / 805 GiB avail; 105 MiB/s wr, 1.68k op/s; 2049/309294 objects misplaced (0.662%) 2024-08-19T14:18:16.106 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:15 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-08-19T14:18:16.106 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:15 smithi031 ceph-mon[16336]: osdmap e269: 12 total, 12 up, 9 in 2024-08-19T14:18:16.107 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:15 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3093586649' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:18:16.107 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:15 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:18:16.107 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:15 smithi031 ceph-mon[16336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:18:16.107 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:15 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:18:16.107 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:15 smithi031 ceph-mon[16336]: osdmap e270: 12 total, 12 up, 9 in 2024-08-19T14:18:16.143 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:18:16.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:16.175 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:16.175 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:18:16.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:15 smithi032 ceph-mon[20004]: pgmap v618: 57 pgs: 4 peering, 10 unknown, 2 active+recovering+undersized+remapped, 41 active+clean; 6.3 GiB data, 23 GiB used, 782 GiB / 805 GiB avail; 105 MiB/s wr, 1.68k op/s; 2049/309294 objects misplaced (0.662%) 2024-08-19T14:18:16.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:15 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-08-19T14:18:16.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:15 smithi032 ceph-mon[20004]: osdmap e269: 12 total, 12 up, 9 in 2024-08-19T14:18:16.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:15 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3093586649' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:18:16.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:15 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:18:16.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:15 smithi032 ceph-mon[20004]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:18:16.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:15 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:18:16.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:15 smithi032 ceph-mon[20004]: osdmap e270: 12 total, 12 up, 9 in 2024-08-19T14:18:16.277 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:18:16.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:15 smithi045 ceph-mon[20143]: pgmap v618: 57 pgs: 4 peering, 10 unknown, 2 active+recovering+undersized+remapped, 41 active+clean; 6.3 GiB data, 23 GiB used, 782 GiB / 805 GiB avail; 105 MiB/s wr, 1.68k op/s; 2049/309294 objects misplaced (0.662%) 2024-08-19T14:18:16.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:15 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-08-19T14:18:16.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:15 smithi045 ceph-mon[20143]: osdmap e269: 12 total, 12 up, 9 in 2024-08-19T14:18:16.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:15 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3093586649' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:18:16.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:15 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:18:16.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:15 smithi045 ceph-mon[20143]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:18:16.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:15 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:18:16.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:15 smithi045 ceph-mon[20143]: osdmap e270: 12 total, 12 up, 9 in 2024-08-19T14:18:16.315 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:16.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:16.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:18:16.317 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 74 16 112285 112269 94.8102 115.25 0.00414532 0.0105429 2024-08-19T14:18:16.417 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:18:16.442 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:16.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:16.443 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:18:16.543 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:18:16.576 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:16.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:16.577 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:18:16.678 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:18:16.713 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:16.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:16.714 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:18:16.815 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:18:16.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:16.850 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:16.851 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:18:16.952 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:18:16.987 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:16.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:16.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:18:17.089 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:18:17.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:16 smithi031 ceph-mon[16336]: 3.5 scrub starts 2024-08-19T14:18:17.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:16 smithi031 ceph-mon[16336]: 3.8 scrub starts 2024-08-19T14:18:17.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:16 smithi031 ceph-mon[16336]: osdmap e271: 12 total, 12 up, 9 in 2024-08-19T14:18:17.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:17.124 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:17.124 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:18:17.226 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:18:17.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:16 smithi032 ceph-mon[20004]: 3.5 scrub starts 2024-08-19T14:18:17.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:16 smithi032 ceph-mon[20004]: 3.8 scrub starts 2024-08-19T14:18:17.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:16 smithi032 ceph-mon[20004]: osdmap e271: 12 total, 12 up, 9 in 2024-08-19T14:18:17.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:17.288 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:17.289 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:18:17.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:16 smithi045 ceph-mon[20143]: 3.5 scrub starts 2024-08-19T14:18:17.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:16 smithi045 ceph-mon[20143]: 3.8 scrub starts 2024-08-19T14:18:17.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:16 smithi045 ceph-mon[20143]: osdmap e271: 12 total, 12 up, 9 in 2024-08-19T14:18:17.317 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 75 16 114103 114087 95.0609 113.625 0.00225622 0.0105106 2024-08-19T14:18:17.356 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:18:17.389 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:18:17.401 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-08-19T14:18:17.403 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 7, 8, 9, 11] out_osds: [10, 6, 2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-08-19T14:18:17.403 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-19T14:18:17.403 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-08-19T14:18:17.404 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-19T14:18:17.419 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:17.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:17.420 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:18:17.521 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:18:17.570 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:17.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:17.571 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:18:17.672 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:18:17.705 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:17.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:17.707 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:18:17.808 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:18:17.851 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:17.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:17.852 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:18:17.935 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:17 smithi031 ceph-mon[16336]: pgmap v621: 57 pgs: 4 peering, 10 unknown, 1 active+recovering+undersized+remapped, 42 active+clean; 6.5 GiB data, 23 GiB used, 782 GiB / 805 GiB avail; 87 MiB/s wr, 1.39k op/s; 1146/317868 objects misplaced (0.361%); 4.7 MiB/s, 75 objects/s recovering 2024-08-19T14:18:17.935 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:17 smithi031 ceph-mon[16336]: 3.17 scrub starts 2024-08-19T14:18:17.936 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:17 smithi031 ceph-mon[16336]: 3.b scrub starts 2024-08-19T14:18:17.936 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:17 smithi031 ceph-mon[16336]: 3.21 scrub starts 2024-08-19T14:18:17.936 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:17 smithi031 ceph-mon[16336]: 3.17 scrub ok 2024-08-19T14:18:17.936 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:17 smithi031 ceph-mon[16336]: 3.2 scrub starts 2024-08-19T14:18:17.936 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:17 smithi031 ceph-mon[16336]: 3.21 scrub ok 2024-08-19T14:18:17.936 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:17 smithi031 ceph-mon[16336]: 3.1b scrub starts 2024-08-19T14:18:17.936 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:17 smithi031 ceph-mon[16336]: 3.5 scrub ok 2024-08-19T14:18:17.936 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:17 smithi031 ceph-mon[16336]: 3.2 scrub ok 2024-08-19T14:18:17.936 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:17 smithi031 ceph-mon[16336]: 3.c scrub starts 2024-08-19T14:18:17.936 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:17 smithi031 ceph-mon[16336]: 3.b scrub ok 2024-08-19T14:18:17.936 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:17 smithi031 ceph-mon[16336]: 3.8 scrub ok 2024-08-19T14:18:17.936 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:17 smithi031 ceph-mon[16336]: 3.7 scrub starts 2024-08-19T14:18:17.936 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:17 smithi031 ceph-mon[16336]: 3.0 scrub starts 2024-08-19T14:18:17.937 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:17 smithi031 ceph-mon[16336]: osdmap e272: 12 total, 12 up, 9 in 2024-08-19T14:18:17.953 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:18:17.986 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:17.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:17.988 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:18:18.000 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:18:18.000 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:18:18.041 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:18:18.041 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:18:18.061 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:18:18.080 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:18:18.081 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:18:18.088 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:18:18.109 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:18:18.118 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:18:18.118 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:18:18.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:18.131 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:18.131 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:18:18.153 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:18:18.153 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:18:18.183 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:18:18.184 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:18:18.211 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:18:18.211 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:18:18.231 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:18:18.249 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:18:18.250 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:18:18.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:17 smithi032 ceph-mon[20004]: pgmap v621: 57 pgs: 4 peering, 10 unknown, 1 active+recovering+undersized+remapped, 42 active+clean; 6.5 GiB data, 23 GiB used, 782 GiB / 805 GiB avail; 87 MiB/s wr, 1.39k op/s; 1146/317868 objects misplaced (0.361%); 4.7 MiB/s, 75 objects/s recovering 2024-08-19T14:18:18.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:17 smithi032 ceph-mon[20004]: 3.17 scrub starts 2024-08-19T14:18:18.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:17 smithi032 ceph-mon[20004]: 3.b scrub starts 2024-08-19T14:18:18.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:17 smithi032 ceph-mon[20004]: 3.21 scrub starts 2024-08-19T14:18:18.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:17 smithi032 ceph-mon[20004]: 3.17 scrub ok 2024-08-19T14:18:18.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:17 smithi032 ceph-mon[20004]: 3.2 scrub starts 2024-08-19T14:18:18.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:17 smithi032 ceph-mon[20004]: 3.21 scrub ok 2024-08-19T14:18:18.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:17 smithi032 ceph-mon[20004]: 3.1b scrub starts 2024-08-19T14:18:18.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:17 smithi032 ceph-mon[20004]: 3.5 scrub ok 2024-08-19T14:18:18.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:17 smithi032 ceph-mon[20004]: 3.2 scrub ok 2024-08-19T14:18:18.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:17 smithi032 ceph-mon[20004]: 3.c scrub starts 2024-08-19T14:18:18.259 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:17 smithi032 ceph-mon[20004]: 3.b scrub ok 2024-08-19T14:18:18.259 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:17 smithi032 ceph-mon[20004]: 3.8 scrub ok 2024-08-19T14:18:18.259 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:17 smithi032 ceph-mon[20004]: 3.7 scrub starts 2024-08-19T14:18:18.259 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:17 smithi032 ceph-mon[20004]: 3.0 scrub starts 2024-08-19T14:18:18.259 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:17 smithi032 ceph-mon[20004]: osdmap e272: 12 total, 12 up, 9 in 2024-08-19T14:18:18.272 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:18.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:18.273 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:18:18.283 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:18:18.283 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:18:18.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:17 smithi045 ceph-mon[20143]: pgmap v621: 57 pgs: 4 peering, 10 unknown, 1 active+recovering+undersized+remapped, 42 active+clean; 6.5 GiB data, 23 GiB used, 782 GiB / 805 GiB avail; 87 MiB/s wr, 1.39k op/s; 1146/317868 objects misplaced (0.361%); 4.7 MiB/s, 75 objects/s recovering 2024-08-19T14:18:18.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:17 smithi045 ceph-mon[20143]: 3.17 scrub starts 2024-08-19T14:18:18.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:17 smithi045 ceph-mon[20143]: 3.b scrub starts 2024-08-19T14:18:18.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:17 smithi045 ceph-mon[20143]: 3.21 scrub starts 2024-08-19T14:18:18.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:17 smithi045 ceph-mon[20143]: 3.17 scrub ok 2024-08-19T14:18:18.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:17 smithi045 ceph-mon[20143]: 3.2 scrub starts 2024-08-19T14:18:18.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:17 smithi045 ceph-mon[20143]: 3.21 scrub ok 2024-08-19T14:18:18.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:17 smithi045 ceph-mon[20143]: 3.1b scrub starts 2024-08-19T14:18:18.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:17 smithi045 ceph-mon[20143]: 3.5 scrub ok 2024-08-19T14:18:18.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:17 smithi045 ceph-mon[20143]: 3.2 scrub ok 2024-08-19T14:18:18.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:17 smithi045 ceph-mon[20143]: 3.c scrub starts 2024-08-19T14:18:18.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:17 smithi045 ceph-mon[20143]: 3.b scrub ok 2024-08-19T14:18:18.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:17 smithi045 ceph-mon[20143]: 3.8 scrub ok 2024-08-19T14:18:18.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:17 smithi045 ceph-mon[20143]: 3.7 scrub starts 2024-08-19T14:18:18.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:17 smithi045 ceph-mon[20143]: 3.0 scrub starts 2024-08-19T14:18:18.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:17 smithi045 ceph-mon[20143]: osdmap e272: 12 total, 12 up, 9 in 2024-08-19T14:18:18.315 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:18:18.315 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:18:18.317 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 76 16 115672 115656 95.1003 98.0625 0.0091134 0.0105112 2024-08-19T14:18:18.347 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:18:18.347 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:18:18.374 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:18:18.375 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:18:18.375 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:18:18.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:18.421 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:18.422 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:18:18.523 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:18:18.564 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:18.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:18.565 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:18:18.666 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:18:18.701 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:18.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:18.702 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:18:18.803 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:18:18.819 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-19T14:18:18.838 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:18.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:18.842 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:18:18.943 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:18:18.975 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:18.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:18.976 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:18:19.077 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:18:19.079 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:18 smithi031 ceph-mon[16336]: 3.9 scrub starts 2024-08-19T14:18:19.079 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:18 smithi031 ceph-mon[16336]: 3.20 scrub starts 2024-08-19T14:18:19.079 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:18 smithi031 ceph-mon[16336]: 3.13 scrub starts 2024-08-19T14:18:19.079 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:18 smithi031 ceph-mon[16336]: pgmap v624: 57 pgs: 1 active+clean+scrubbing, 4 peering, 4 remapped+peering, 6 unknown, 42 active+clean; 6.6 GiB data, 23 GiB used, 782 GiB / 805 GiB avail; 2.5 KiB/s rd, 85 MiB/s wr, 1.37k op/s; 10 MiB/s, 161 objects/s recovering 2024-08-19T14:18:19.079 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:18 smithi031 ceph-mon[16336]: 3.0 scrub ok 2024-08-19T14:18:19.080 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:18 smithi031 ceph-mon[16336]: 3.1b scrub ok 2024-08-19T14:18:19.080 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:18 smithi031 ceph-mon[16336]: 3.11 scrub starts 2024-08-19T14:18:19.080 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:18 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1909705076' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:18:19.080 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:18 smithi031 ceph-mon[16336]: 3.c scrub ok 2024-08-19T14:18:19.080 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:18 smithi031 ceph-mon[16336]: 3.16 scrub starts 2024-08-19T14:18:19.080 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:18 smithi031 ceph-mon[16336]: 3.20 scrub ok 2024-08-19T14:18:19.080 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:18 smithi031 ceph-mon[16336]: 3.7 scrub ok 2024-08-19T14:18:19.080 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:18 smithi031 ceph-mon[16336]: 3.1e scrub starts 2024-08-19T14:18:19.080 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:18 smithi031 ceph-mon[16336]: 3.e scrub starts 2024-08-19T14:18:19.080 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:18 smithi031 ceph-mon[16336]: osdmap e273: 12 total, 12 up, 9 in 2024-08-19T14:18:19.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:19.106 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:19.106 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:18:19.207 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:18:19.215 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:18:19.216 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-08-19T14:18:19.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:19.243 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:19.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:18:19.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:18 smithi032 ceph-mon[20004]: 3.9 scrub starts 2024-08-19T14:18:19.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:18 smithi032 ceph-mon[20004]: 3.20 scrub starts 2024-08-19T14:18:19.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:18 smithi032 ceph-mon[20004]: 3.13 scrub starts 2024-08-19T14:18:19.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:18 smithi032 ceph-mon[20004]: pgmap v624: 57 pgs: 1 active+clean+scrubbing, 4 peering, 4 remapped+peering, 6 unknown, 42 active+clean; 6.6 GiB data, 23 GiB used, 782 GiB / 805 GiB avail; 2.5 KiB/s rd, 85 MiB/s wr, 1.37k op/s; 10 MiB/s, 161 objects/s recovering 2024-08-19T14:18:19.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:18 smithi032 ceph-mon[20004]: 3.0 scrub ok 2024-08-19T14:18:19.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:18 smithi032 ceph-mon[20004]: 3.1b scrub ok 2024-08-19T14:18:19.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:18 smithi032 ceph-mon[20004]: 3.11 scrub starts 2024-08-19T14:18:19.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:18 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1909705076' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:18:19.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:18 smithi032 ceph-mon[20004]: 3.c scrub ok 2024-08-19T14:18:19.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:18 smithi032 ceph-mon[20004]: 3.16 scrub starts 2024-08-19T14:18:19.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:18 smithi032 ceph-mon[20004]: 3.20 scrub ok 2024-08-19T14:18:19.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:18 smithi032 ceph-mon[20004]: 3.7 scrub ok 2024-08-19T14:18:19.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:18 smithi032 ceph-mon[20004]: 3.1e scrub starts 2024-08-19T14:18:19.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:18 smithi032 ceph-mon[20004]: 3.e scrub starts 2024-08-19T14:18:19.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:18 smithi032 ceph-mon[20004]: osdmap e273: 12 total, 12 up, 9 in 2024-08-19T14:18:19.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:18 smithi045 ceph-mon[20143]: 3.9 scrub starts 2024-08-19T14:18:19.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:18 smithi045 ceph-mon[20143]: 3.20 scrub starts 2024-08-19T14:18:19.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:18 smithi045 ceph-mon[20143]: 3.13 scrub starts 2024-08-19T14:18:19.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:18 smithi045 ceph-mon[20143]: pgmap v624: 57 pgs: 1 active+clean+scrubbing, 4 peering, 4 remapped+peering, 6 unknown, 42 active+clean; 6.6 GiB data, 23 GiB used, 782 GiB / 805 GiB avail; 2.5 KiB/s rd, 85 MiB/s wr, 1.37k op/s; 10 MiB/s, 161 objects/s recovering 2024-08-19T14:18:19.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:18 smithi045 ceph-mon[20143]: 3.0 scrub ok 2024-08-19T14:18:19.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:18 smithi045 ceph-mon[20143]: 3.1b scrub ok 2024-08-19T14:18:19.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:18 smithi045 ceph-mon[20143]: 3.11 scrub starts 2024-08-19T14:18:19.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:18 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1909705076' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:18:19.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:18 smithi045 ceph-mon[20143]: 3.c scrub ok 2024-08-19T14:18:19.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:18 smithi045 ceph-mon[20143]: 3.16 scrub starts 2024-08-19T14:18:19.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:18 smithi045 ceph-mon[20143]: 3.20 scrub ok 2024-08-19T14:18:19.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:18 smithi045 ceph-mon[20143]: 3.7 scrub ok 2024-08-19T14:18:19.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:18 smithi045 ceph-mon[20143]: 3.1e scrub starts 2024-08-19T14:18:19.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:18 smithi045 ceph-mon[20143]: 3.e scrub starts 2024-08-19T14:18:19.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:18 smithi045 ceph-mon[20143]: osdmap e273: 12 total, 12 up, 9 in 2024-08-19T14:18:19.317 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 77 16 117435 117419 95.2961 110.188 0.00946984 0.0104892 2024-08-19T14:18:19.344 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:18:19.376 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:19.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:19.378 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:18:19.479 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:18:19.513 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:19.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:19.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:18:19.615 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:18:19.648 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:19.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:19.649 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:18:19.750 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:18:19.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:19.784 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:19.784 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:18:19.885 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:18:19.899 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:19 smithi031 ceph-mon[16336]: 3.9 scrub ok 2024-08-19T14:18:19.899 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:19 smithi031 ceph-mon[16336]: 3.13 scrub ok 2024-08-19T14:18:19.899 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:19 smithi031 ceph-mon[16336]: 1.14 scrub starts 2024-08-19T14:18:19.899 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:19 smithi031 ceph-mon[16336]: 1.14 scrub ok 2024-08-19T14:18:19.899 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:19 smithi031 ceph-mon[16336]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-08-19T14:18:19.899 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:19 smithi031 ceph-mon[16336]: 3.11 scrub ok 2024-08-19T14:18:19.899 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:19 smithi031 ceph-mon[16336]: 3.14 scrub starts 2024-08-19T14:18:19.900 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:19 smithi031 ceph-mon[16336]: 3.16 scrub ok 2024-08-19T14:18:19.900 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:19 smithi031 ceph-mon[16336]: 3.1e scrub ok 2024-08-19T14:18:19.900 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:19 smithi031 ceph-mon[16336]: 3.1c scrub starts 2024-08-19T14:18:19.900 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:19 smithi031 ceph-mon[16336]: 3.e scrub ok 2024-08-19T14:18:19.900 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:19 smithi031 ceph-mon[16336]: from='client.15393 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:18:19.900 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:19 smithi031 ceph-mon[16336]: 3.22 scrub starts 2024-08-19T14:18:19.900 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:19 smithi031 ceph-mon[16336]: 1.b scrub starts 2024-08-19T14:18:19.900 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:19 smithi031 ceph-mon[16336]: 1.b scrub ok 2024-08-19T14:18:19.901 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:18:19.901 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:18:19.912 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:19.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:19.913 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:18:20.014 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:18:20.047 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:20.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:20.048 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:18:20.149 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:18:20.177 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:20.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:20.178 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:18:20.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:19 smithi032 ceph-mon[20004]: 3.9 scrub ok 2024-08-19T14:18:20.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:19 smithi032 ceph-mon[20004]: 3.13 scrub ok 2024-08-19T14:18:20.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:19 smithi032 ceph-mon[20004]: 1.14 scrub starts 2024-08-19T14:18:20.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:19 smithi032 ceph-mon[20004]: 1.14 scrub ok 2024-08-19T14:18:20.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:19 smithi032 ceph-mon[20004]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-08-19T14:18:20.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:19 smithi032 ceph-mon[20004]: 3.11 scrub ok 2024-08-19T14:18:20.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:19 smithi032 ceph-mon[20004]: 3.14 scrub starts 2024-08-19T14:18:20.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:19 smithi032 ceph-mon[20004]: 3.16 scrub ok 2024-08-19T14:18:20.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:19 smithi032 ceph-mon[20004]: 3.1e scrub ok 2024-08-19T14:18:20.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:19 smithi032 ceph-mon[20004]: 3.1c scrub starts 2024-08-19T14:18:20.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:19 smithi032 ceph-mon[20004]: 3.e scrub ok 2024-08-19T14:18:20.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:19 smithi032 ceph-mon[20004]: from='client.15393 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:18:20.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:19 smithi032 ceph-mon[20004]: 3.22 scrub starts 2024-08-19T14:18:20.259 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:19 smithi032 ceph-mon[20004]: 1.b scrub starts 2024-08-19T14:18:20.259 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:19 smithi032 ceph-mon[20004]: 1.b scrub ok 2024-08-19T14:18:20.278 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:18:20.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:19 smithi045 ceph-mon[20143]: 3.9 scrub ok 2024-08-19T14:18:20.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:19 smithi045 ceph-mon[20143]: 3.13 scrub ok 2024-08-19T14:18:20.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:19 smithi045 ceph-mon[20143]: 1.14 scrub starts 2024-08-19T14:18:20.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:19 smithi045 ceph-mon[20143]: 1.14 scrub ok 2024-08-19T14:18:20.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:19 smithi045 ceph-mon[20143]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-08-19T14:18:20.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:19 smithi045 ceph-mon[20143]: 3.11 scrub ok 2024-08-19T14:18:20.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:19 smithi045 ceph-mon[20143]: 3.14 scrub starts 2024-08-19T14:18:20.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:19 smithi045 ceph-mon[20143]: 3.16 scrub ok 2024-08-19T14:18:20.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:19 smithi045 ceph-mon[20143]: 3.1e scrub ok 2024-08-19T14:18:20.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:19 smithi045 ceph-mon[20143]: 3.1c scrub starts 2024-08-19T14:18:20.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:19 smithi045 ceph-mon[20143]: 3.e scrub ok 2024-08-19T14:18:20.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:19 smithi045 ceph-mon[20143]: from='client.15393 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:18:20.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:19 smithi045 ceph-mon[20143]: 3.22 scrub starts 2024-08-19T14:18:20.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:19 smithi045 ceph-mon[20143]: 1.b scrub starts 2024-08-19T14:18:20.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:19 smithi045 ceph-mon[20143]: 1.b scrub ok 2024-08-19T14:18:20.300 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:20.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:20.301 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:18:20.318 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 78 16 118893 118877 95.2426 91.125 0.00234471 0.0104656 2024-08-19T14:18:20.402 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:18:20.423 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:20.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:20.424 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:18:20.525 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:18:20.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:20.548 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:20.548 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:18:20.648 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:18:20.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:20.671 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:20.671 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:18:20.773 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:18:20.795 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:20.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:20.797 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:18:20.898 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:18:20.912 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-08-19T14:18:20.919 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:20.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:20.920 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:18:21.021 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:18:21.047 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:21.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:21.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:18:21.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:20 smithi031 ceph-mon[16336]: 3.1f scrub starts 2024-08-19T14:18:21.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:20 smithi031 ceph-mon[16336]: 3.1a scrub starts 2024-08-19T14:18:21.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:20 smithi031 ceph-mon[16336]: 3.4 scrub starts 2024-08-19T14:18:21.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:20 smithi031 ceph-mon[16336]: 3.22 scrub ok 2024-08-19T14:18:21.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:20 smithi031 ceph-mon[16336]: 1.d scrub starts 2024-08-19T14:18:21.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:20 smithi031 ceph-mon[16336]: 1.d scrub ok 2024-08-19T14:18:21.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:20 smithi031 ceph-mon[16336]: 3.10 scrub starts 2024-08-19T14:18:21.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:20 smithi031 ceph-mon[16336]: pgmap v626: 57 pgs: 4 peering, 3 active+clean+scrubbing, 3 remapped+peering, 2 unknown, 45 active+clean; 6.9 GiB data, 24 GiB used, 781 GiB / 805 GiB avail; 5.5 KiB/s rd, 6 op/s 2024-08-19T14:18:21.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:20 smithi031 ceph-mon[16336]: 3.14 scrub ok 2024-08-19T14:18:21.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:20 smithi031 ceph-mon[16336]: 3.1f scrub ok 2024-08-19T14:18:21.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:20 smithi031 ceph-mon[16336]: 3.3 scrub starts 2024-08-19T14:18:21.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:20 smithi031 ceph-mon[16336]: 3.1 scrub starts 2024-08-19T14:18:21.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:20 smithi031 ceph-mon[16336]: 1.12 scrub starts 2024-08-19T14:18:21.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:20 smithi031 ceph-mon[16336]: 1.12 scrub ok 2024-08-19T14:18:21.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:20 smithi031 ceph-mon[16336]: 3.1a scrub ok 2024-08-19T14:18:21.118 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:20 smithi031 ceph-mon[16336]: 3.1c scrub ok 2024-08-19T14:18:21.118 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:20 smithi031 ceph-mon[16336]: 3.3 scrub ok 2024-08-19T14:18:21.118 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:20 smithi031 ceph-mon[16336]: 1.c scrub starts 2024-08-19T14:18:21.118 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:20 smithi031 ceph-mon[16336]: 1.c scrub ok 2024-08-19T14:18:21.118 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:20 smithi031 ceph-mon[16336]: 3.10 scrub ok 2024-08-19T14:18:21.118 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:20 smithi031 ceph-mon[16336]: 1.5 scrub starts 2024-08-19T14:18:21.118 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:20 smithi031 ceph-mon[16336]: 1.5 scrub ok 2024-08-19T14:18:21.150 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:18:21.172 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:21.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:21.173 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:18:21.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:20 smithi032 ceph-mon[20004]: 3.1f scrub starts 2024-08-19T14:18:21.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:20 smithi032 ceph-mon[20004]: 3.1a scrub starts 2024-08-19T14:18:21.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:20 smithi032 ceph-mon[20004]: 3.4 scrub starts 2024-08-19T14:18:21.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:20 smithi032 ceph-mon[20004]: 3.22 scrub ok 2024-08-19T14:18:21.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:20 smithi032 ceph-mon[20004]: 1.d scrub starts 2024-08-19T14:18:21.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:20 smithi032 ceph-mon[20004]: 1.d scrub ok 2024-08-19T14:18:21.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:20 smithi032 ceph-mon[20004]: 3.10 scrub starts 2024-08-19T14:18:21.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:20 smithi032 ceph-mon[20004]: pgmap v626: 57 pgs: 4 peering, 3 active+clean+scrubbing, 3 remapped+peering, 2 unknown, 45 active+clean; 6.9 GiB data, 24 GiB used, 781 GiB / 805 GiB avail; 5.5 KiB/s rd, 6 op/s 2024-08-19T14:18:21.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:20 smithi032 ceph-mon[20004]: 3.14 scrub ok 2024-08-19T14:18:21.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:20 smithi032 ceph-mon[20004]: 3.1f scrub ok 2024-08-19T14:18:21.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:20 smithi032 ceph-mon[20004]: 3.3 scrub starts 2024-08-19T14:18:21.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:20 smithi032 ceph-mon[20004]: 3.1 scrub starts 2024-08-19T14:18:21.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:20 smithi032 ceph-mon[20004]: 1.12 scrub starts 2024-08-19T14:18:21.259 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:20 smithi032 ceph-mon[20004]: 1.12 scrub ok 2024-08-19T14:18:21.259 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:20 smithi032 ceph-mon[20004]: 3.1a scrub ok 2024-08-19T14:18:21.259 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:20 smithi032 ceph-mon[20004]: 3.1c scrub ok 2024-08-19T14:18:21.259 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:20 smithi032 ceph-mon[20004]: 3.3 scrub ok 2024-08-19T14:18:21.259 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:20 smithi032 ceph-mon[20004]: 1.c scrub starts 2024-08-19T14:18:21.259 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:20 smithi032 ceph-mon[20004]: 1.c scrub ok 2024-08-19T14:18:21.259 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:20 smithi032 ceph-mon[20004]: 3.10 scrub ok 2024-08-19T14:18:21.259 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:20 smithi032 ceph-mon[20004]: 1.5 scrub starts 2024-08-19T14:18:21.259 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:20 smithi032 ceph-mon[20004]: 1.5 scrub ok 2024-08-19T14:18:21.275 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:18:21.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:20 smithi045 ceph-mon[20143]: 3.1f scrub starts 2024-08-19T14:18:21.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:20 smithi045 ceph-mon[20143]: 3.1a scrub starts 2024-08-19T14:18:21.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:20 smithi045 ceph-mon[20143]: 3.4 scrub starts 2024-08-19T14:18:21.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:20 smithi045 ceph-mon[20143]: 3.22 scrub ok 2024-08-19T14:18:21.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:20 smithi045 ceph-mon[20143]: 1.d scrub starts 2024-08-19T14:18:21.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:20 smithi045 ceph-mon[20143]: 1.d scrub ok 2024-08-19T14:18:21.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:20 smithi045 ceph-mon[20143]: 3.10 scrub starts 2024-08-19T14:18:21.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:20 smithi045 ceph-mon[20143]: pgmap v626: 57 pgs: 4 peering, 3 active+clean+scrubbing, 3 remapped+peering, 2 unknown, 45 active+clean; 6.9 GiB data, 24 GiB used, 781 GiB / 805 GiB avail; 5.5 KiB/s rd, 6 op/s 2024-08-19T14:18:21.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:20 smithi045 ceph-mon[20143]: 3.14 scrub ok 2024-08-19T14:18:21.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:20 smithi045 ceph-mon[20143]: 3.1f scrub ok 2024-08-19T14:18:21.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:20 smithi045 ceph-mon[20143]: 3.3 scrub starts 2024-08-19T14:18:21.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:20 smithi045 ceph-mon[20143]: 3.1 scrub starts 2024-08-19T14:18:21.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:20 smithi045 ceph-mon[20143]: 1.12 scrub starts 2024-08-19T14:18:21.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:20 smithi045 ceph-mon[20143]: 1.12 scrub ok 2024-08-19T14:18:21.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:20 smithi045 ceph-mon[20143]: 3.1a scrub ok 2024-08-19T14:18:21.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:20 smithi045 ceph-mon[20143]: 3.1c scrub ok 2024-08-19T14:18:21.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:20 smithi045 ceph-mon[20143]: 3.3 scrub ok 2024-08-19T14:18:21.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:20 smithi045 ceph-mon[20143]: 1.c scrub starts 2024-08-19T14:18:21.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:20 smithi045 ceph-mon[20143]: 1.c scrub ok 2024-08-19T14:18:21.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:20 smithi045 ceph-mon[20143]: 3.10 scrub ok 2024-08-19T14:18:21.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:20 smithi045 ceph-mon[20143]: 1.5 scrub starts 2024-08-19T14:18:21.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:20 smithi045 ceph-mon[20143]: 1.5 scrub ok 2024-08-19T14:18:21.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:21.299 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:21.299 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:18:21.318 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 79 16 118893 118877 94.037 0 - 0.0104656 2024-08-19T14:18:21.377 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":625,"stamp":"2024-08-19T14:18:18.690966+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7351577893,"num_objects":112171,"num_object_clones":0,"num_object_copies":336513,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":112171,"num_whiteouts":0,"num_read":22,"num_read_kb":18,"num_write":105156,"num_write_kb":6728427,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28641,"num_bytes_recovered":1877324328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":182354,"ondisk_log_size":182354,"up":168,"acting":165,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":182,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":24869932,"kb_used_data":22347216,"kb_used_omap":14,"kb_used_meta":2522545,"kb_avail":818836436,"statfs":{"total":863955320832,"available":838488510464,"internally_reserved":0,"allocated":22883549184,"data_stored":22871914936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14353,"internal_metadata":2583087087},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,7,13,3,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":517090053,"num_objects":7892,"num_object_clones":0,"num_object_copies":23676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1146,"num_objects_unfound":0,"num_objects_dirty":7892,"num_whiteouts":0,"num_read":22,"num_read_kb":18,"num_write":879,"num_write_kb":54699,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2629,"num_bytes_recovered":-172122104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.887333"},"pg_stats":[{"pgid":"3.23","version":"269'4209","reported_seq":6403,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-08-19T14:18:15.808680+0000","last_change":"2024-08-19T14:17:46.799697+0000","last_active":"2024-08-19T14:18:15.808680+0000","last_peered":"2024-08-19T14:18:15.808680+0000","last_clean":"2024-08-19T14:18:15.808680+0000","last_became_active":"2024-08-19T14:17:46.797296+0000","last_became_peered":"2024-08-19T14:17:46.797296+0000","last_unstale":"2024-08-19T14:18:15.808680+0000","last_undegraded":"2024-08-19T14:18:15.808680+0000","last_fullsized":"2024-08-19T14:18:15.808680+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":4209,"log_dups_size":0,"ondisk_log_size":4209,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:19.018064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":114376704,"num_objects":1745,"num_object_clones":0,"num_object_copies":5235,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1745,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1745,"num_write_kb":111696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1031,"num_bytes_recovered":67567616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"268'4273","reported_seq":7400,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-08-19T14:18:14.183946+0000","last_change":"2024-08-19T14:17:49.689584+0000","last_active":"2024-08-19T14:18:14.183946+0000","last_peered":"2024-08-19T14:18:14.183946+0000","last_clean":"2024-08-19T14:18:14.183946+0000","last_became_active":"2024-08-19T14:17:49.687020+0000","last_became_peered":"2024-08-19T14:17:49.687020+0000","last_unstale":"2024-08-19T14:18:14.183946+0000","last_undegraded":"2024-08-19T14:18:14.183946+0000","last_fullsized":"2024-08-19T14:18:14.183946+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":4273,"log_dups_size":0,"ondisk_log_size":4273,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:18.393875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":114917376,"num_objects":1754,"num_object_clones":0,"num_object_copies":5262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1754,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1754,"num_write_kb":112224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1037,"num_bytes_recovered":67960832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,9],"acting":[3,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"268'4157","reported_seq":8194,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-08-19T14:18:14.804358+0000","last_change":"2024-08-19T14:17:49.682222+0000","last_active":"2024-08-19T14:18:14.804358+0000","last_peered":"2024-08-19T14:18:14.804358+0000","last_clean":"2024-08-19T14:18:14.804358+0000","last_became_active":"2024-08-19T14:17:49.680746+0000","last_became_peered":"2024-08-19T14:17:49.680746+0000","last_unstale":"2024-08-19T14:18:14.804358+0000","last_undegraded":"2024-08-19T14:18:14.804358+0000","last_fullsized":"2024-08-19T14:18:14.804358+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":4157,"log_dups_size":0,"ondisk_log_size":4157,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:16.371238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":111984640,"num_objects":1709,"num_object_clones":0,"num_object_copies":5127,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1709,"num_write_kb":109360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":978,"num_bytes_recovered":64094208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,9],"acting":[8,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.20","version":"272'4201","reported_seq":6328,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-08-19T14:18:18.197984+0000","last_change":"2024-08-19T14:18:18.133721+0000","last_active":"2024-08-19T14:18:18.197984+0000","last_peered":"2024-08-19T14:18:18.197984+0000","last_clean":"2024-08-19T14:18:18.197984+0000","last_became_active":"2024-08-19T14:17:43.686357+0000","last_became_peered":"2024-08-19T14:17:43.686357+0000","last_unstale":"2024-08-19T14:18:18.197984+0000","last_undegraded":"2024-08-19T14:18:18.197984+0000","last_fullsized":"2024-08-19T14:18:18.197984+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":6,"last_scrub":"272'4197","last_scrub_stamp":"2024-08-19T14:18:18.133535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:18.133535+0000","objects_scrubbed":1747,"log_size":4201,"log_dups_size":0,"ondisk_log_size":4201,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:28.095319+0000","scrub_duration":374,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":115015680,"num_objects":1755,"num_object_clones":0,"num_object_copies":5265,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,11],"acting":[9,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1f","version":"269'5378","reported_seq":13513,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-08-19T14:18:15.808174+0000","last_change":"2024-08-19T14:18:12.804413+0000","last_active":"2024-08-19T14:18:15.808174+0000","last_peered":"2024-08-19T14:18:15.808174+0000","last_clean":"2024-08-19T14:18:15.808174+0000","last_became_active":"2024-08-19T14:18:12.800420+0000","last_became_peered":"2024-08-19T14:18:12.800420+0000","last_unstale":"2024-08-19T14:18:15.808174+0000","last_undegraded":"2024-08-19T14:18:15.808174+0000","last_fullsized":"2024-08-19T14:18:15.808174+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":5378,"log_dups_size":0,"ondisk_log_size":5378,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:17.999363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222593024,"num_objects":3397,"num_object_clones":0,"num_object_copies":10191,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3397,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3397,"num_write_kb":217376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":131203072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"268'5275","reported_seq":13507,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-08-19T14:18:14.216027+0000","last_change":"2024-08-19T14:18:11.845740+0000","last_active":"2024-08-19T14:18:14.216027+0000","last_peered":"2024-08-19T14:18:14.216027+0000","last_clean":"2024-08-19T14:18:14.216027+0000","last_became_active":"2024-08-19T14:18:11.801671+0000","last_became_peered":"2024-08-19T14:18:11.801671+0000","last_unstale":"2024-08-19T14:18:14.216027+0000","last_undegraded":"2024-08-19T14:18:14.216027+0000","last_fullsized":"2024-08-19T14:18:14.216027+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":5275,"log_dups_size":0,"ondisk_log_size":5275,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:17.423791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":216137728,"num_objects":3298,"num_object_clones":0,"num_object_copies":9894,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3298,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3298,"num_write_kb":211072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2028,"num_bytes_recovered":132907008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,8],"acting":[3,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"269'5443","reported_seq":11601,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-08-19T14:18:14.813870+0000","last_change":"2024-08-19T14:17:59.738739+0000","last_active":"2024-08-19T14:18:14.813870+0000","last_peered":"2024-08-19T14:18:14.813870+0000","last_clean":"2024-08-19T14:18:14.813870+0000","last_became_active":"2024-08-19T14:17:59.735912+0000","last_became_peered":"2024-08-19T14:17:59.735912+0000","last_unstale":"2024-08-19T14:18:14.813870+0000","last_undegraded":"2024-08-19T14:18:14.813870+0000","last_fullsized":"2024-08-19T14:18:14.813870+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":5443,"log_dups_size":0,"ondisk_log_size":5443,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:15.339195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":224624640,"num_objects":3428,"num_object_clones":0,"num_object_copies":10284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3428,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3428,"num_write_kb":219360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2011,"num_bytes_recovered":131792896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,5],"acting":[8,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"270'5443","reported_seq":11349,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-08-19T14:18:16.577736+0000","last_change":"2024-08-19T14:18:05.761531+0000","last_active":"2024-08-19T14:18:16.577736+0000","last_peered":"2024-08-19T14:18:16.577736+0000","last_clean":"2024-08-19T14:18:16.577736+0000","last_became_active":"2024-08-19T14:18:05.757137+0000","last_became_peered":"2024-08-19T14:18:05.757137+0000","last_unstale":"2024-08-19T14:18:16.577736+0000","last_undegraded":"2024-08-19T14:18:16.577736+0000","last_fullsized":"2024-08-19T14:18:16.577736+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":5443,"log_dups_size":0,"ondisk_log_size":5443,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":228884480,"num_objects":3493,"num_object_clones":0,"num_object_copies":10479,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3493,"num_write_kb":223520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1926,"num_bytes_recovered":126222336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"272'5604","reported_seq":11754,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-08-19T14:18:17.986554+0000","last_change":"2024-08-19T14:18:17.849009+0000","last_active":"2024-08-19T14:18:17.986554+0000","last_peered":"2024-08-19T14:18:17.986554+0000","last_clean":"2024-08-19T14:18:17.986554+0000","last_became_active":"2024-08-19T14:18:06.761591+0000","last_became_peered":"2024-08-19T14:18:06.761591+0000","last_unstale":"2024-08-19T14:18:17.986554+0000","last_undegraded":"2024-08-19T14:18:17.986554+0000","last_fullsized":"2024-08-19T14:18:17.986554+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":5,"last_scrub":"272'5596","last_scrub_stamp":"2024-08-19T14:18:17.848788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:17.848788+0000","objects_scrubbed":3567,"log_size":5604,"log_dups_size":0,"ondisk_log_size":5604,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:35.007855+0000","scrub_duration":920,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235405312,"num_objects":3592,"num_object_clones":0,"num_object_copies":10776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3592,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25,"num_write_kb":1600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,1],"acting":[4,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"269'5419","reported_seq":9558,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-08-19T14:18:14.805512+0000","last_change":"2024-08-19T14:17:46.733798+0000","last_active":"2024-08-19T14:18:14.805512+0000","last_peered":"2024-08-19T14:18:14.805512+0000","last_clean":"2024-08-19T14:18:14.805512+0000","last_became_active":"2024-08-19T14:17:46.731170+0000","last_became_peered":"2024-08-19T14:17:46.731170+0000","last_unstale":"2024-08-19T14:18:14.805512+0000","last_undegraded":"2024-08-19T14:18:14.805512+0000","last_fullsized":"2024-08-19T14:18:14.805512+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":5419,"log_dups_size":0,"ondisk_log_size":5419,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:18.446412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":225902592,"num_objects":3447,"num_object_clones":0,"num_object_copies":10341,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3447,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3447,"num_write_kb":220608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2009,"num_bytes_recovered":131661824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,9],"acting":[8,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.19","version":"272'5164","reported_seq":11571,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-08-19T14:18:18.386698+0000","last_change":"2024-08-19T14:17:46.800856+0000","last_active":"2024-08-19T14:18:18.386698+0000","last_peered":"2024-08-19T14:18:18.386698+0000","last_clean":"2024-08-19T14:18:18.386698+0000","last_became_active":"2024-08-19T14:17:46.796533+0000","last_became_peered":"2024-08-19T14:17:46.796533+0000","last_unstale":"2024-08-19T14:18:18.386698+0000","last_undegraded":"2024-08-19T14:18:18.386698+0000","last_fullsized":"2024-08-19T14:18:18.386698+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":5164,"log_dups_size":0,"ondisk_log_size":5164,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:20.620953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":239894528,"num_objects":3661,"num_object_clones":0,"num_object_copies":10983,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3661,"num_write_kb":234272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1569,"num_bytes_recovered":102825984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"268'4981","reported_seq":9820,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-08-19T14:18:14.226119+0000","last_change":"2024-08-19T14:17:45.732581+0000","last_active":"2024-08-19T14:18:14.226119+0000","last_peered":"2024-08-19T14:18:14.226119+0000","last_clean":"2024-08-19T14:18:14.226119+0000","last_became_active":"2024-08-19T14:17:45.729134+0000","last_became_peered":"2024-08-19T14:17:45.729134+0000","last_unstale":"2024-08-19T14:18:14.226119+0000","last_undegraded":"2024-08-19T14:18:14.226119+0000","last_fullsized":"2024-08-19T14:18:14.226119+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":4981,"log_dups_size":0,"ondisk_log_size":4981,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:14.500490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":224952320,"num_objects":3433,"num_object_clones":0,"num_object_copies":10299,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3433,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3433,"num_write_kb":219680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1572,"num_bytes_recovered":103022592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,1],"acting":[3,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"269'4869","reported_seq":11111,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-08-19T14:18:15.808230+0000","last_change":"2024-08-19T14:17:48.685189+0000","last_active":"2024-08-19T14:18:15.808230+0000","last_peered":"2024-08-19T14:18:15.808230+0000","last_clean":"2024-08-19T14:18:15.808230+0000","last_became_active":"2024-08-19T14:17:48.682321+0000","last_became_peered":"2024-08-19T14:17:48.682321+0000","last_unstale":"2024-08-19T14:18:15.808230+0000","last_undegraded":"2024-08-19T14:18:15.808230+0000","last_fullsized":"2024-08-19T14:18:15.808230+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":4869,"log_dups_size":0,"ondisk_log_size":4869,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222560256,"num_objects":3396,"num_object_clones":0,"num_object_copies":10188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3396,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3396,"num_write_kb":217344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1529,"num_bytes_recovered":100204544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,9],"acting":[0,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"70'139","reported_seq":397,"reported_epoch":272,"state":"peering","last_fresh":"2024-08-19T14:18:17.696946+0000","last_change":"2024-08-19T14:18:17.696946+0000","last_active":"2024-08-19T14:18:17.696946+0000","last_peered":"2024-08-19T14:18:16.822830+0000","last_clean":"2024-08-19T14:13:00.882096+0000","last_became_active":"2024-08-19T14:18:16.821754+0000","last_became_peered":"2024-08-19T14:18:16.821754+0000","last_unstale":"2024-08-19T14:18:17.696946+0000","last_undegraded":"2024-08-19T14:18:17.696946+0000","last_fullsized":"2024-08-19T14:18:17.696946+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:16:29.256959+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:16:29.256959+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17890,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":6,"num_write_kb":89,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":80141,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,1],"acting":[7,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"270'5124","reported_seq":9843,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-08-19T14:18:16.598288+0000","last_change":"2024-08-19T14:17:47.684929+0000","last_active":"2024-08-19T14:18:16.598288+0000","last_peered":"2024-08-19T14:18:16.598288+0000","last_clean":"2024-08-19T14:18:16.598288+0000","last_became_active":"2024-08-19T14:17:47.681397+0000","last_became_peered":"2024-08-19T14:17:47.681397+0000","last_unstale":"2024-08-19T14:18:16.598288+0000","last_undegraded":"2024-08-19T14:18:16.598288+0000","last_fullsized":"2024-08-19T14:18:16.598288+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":5124,"log_dups_size":0,"ondisk_log_size":5124,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235765760,"num_objects":3598,"num_object_clones":0,"num_object_copies":10794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3598,"num_write_kb":230240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1597,"num_bytes_recovered":104660992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"269'4856","reported_seq":9615,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-08-19T14:18:14.805280+0000","last_change":"2024-08-19T14:17:42.700414+0000","last_active":"2024-08-19T14:18:14.805280+0000","last_peered":"2024-08-19T14:18:14.805280+0000","last_clean":"2024-08-19T14:18:14.805280+0000","last_became_active":"2024-08-19T14:17:42.696659+0000","last_became_peered":"2024-08-19T14:17:42.696659+0000","last_unstale":"2024-08-19T14:18:14.805280+0000","last_undegraded":"2024-08-19T14:18:14.805280+0000","last_fullsized":"2024-08-19T14:18:14.805280+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":4856,"log_dups_size":0,"ondisk_log_size":4856,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:19.447837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":221282304,"num_objects":3377,"num_object_clones":0,"num_object_copies":10131,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3377,"num_write_kb":216096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1544,"num_bytes_recovered":101187584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.14","version":"272'5135","reported_seq":9997,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-08-19T14:18:17.984609+0000","last_change":"2024-08-19T14:17:44.694783+0000","last_active":"2024-08-19T14:18:17.984609+0000","last_peered":"2024-08-19T14:18:17.984609+0000","last_clean":"2024-08-19T14:18:17.984609+0000","last_became_active":"2024-08-19T14:17:44.691159+0000","last_became_peered":"2024-08-19T14:17:44.691159+0000","last_unstale":"2024-08-19T14:18:17.984609+0000","last_undegraded":"2024-08-19T14:18:17.984609+0000","last_fullsized":"2024-08-19T14:18:17.984609+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":5135,"log_dups_size":0,"ondisk_log_size":5135,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":238256128,"num_objects":3636,"num_object_clones":0,"num_object_copies":10908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3636,"num_write_kb":232672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1576,"num_bytes_recovered":103284736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,11],"acting":[4,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.11","version":"70'139","reported_seq":275,"reported_epoch":270,"state":"remapped+peering","last_fresh":"2024-08-19T14:18:15.807640+0000","last_change":"2024-08-19T14:18:15.807640+0000","last_active":"2024-08-19T14:13:02.592411+0000","last_peered":"2024-08-19T14:10:40.177137+0000","last_clean":"2024-08-19T14:10:40.177137+0000","last_became_active":"2024-08-19T14:10:32.896866+0000","last_became_peered":"2024-08-19T14:10:32.896866+0000","last_unstale":"2024-08-19T14:18:15.807640+0000","last_undegraded":"2024-08-19T14:18:15.807640+0000","last_fullsized":"2024-08-19T14:18:15.807640+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:16:30.043526+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:16:30.043526+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":5,"num_write":9,"num_write_kb":134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":96169,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"272'5071","reported_seq":9835,"reported_epoch":272,"state":"active+clean+scrubbing","last_fresh":"2024-08-19T14:18:18.186396+0000","last_change":"2024-08-19T14:18:17.761487+0000","last_active":"2024-08-19T14:18:18.186396+0000","last_peered":"2024-08-19T14:18:18.186396+0000","last_clean":"2024-08-19T14:18:18.186396+0000","last_became_active":"2024-08-19T14:17:45.731114+0000","last_became_peered":"2024-08-19T14:17:45.731114+0000","last_unstale":"2024-08-19T14:18:18.186396+0000","last_undegraded":"2024-08-19T14:18:18.186396+0000","last_fullsized":"2024-08-19T14:18:18.186396+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":1899,"log_size":5071,"log_dups_size":0,"ondisk_log_size":5071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235307008,"num_objects":3591,"num_object_clones":0,"num_object_copies":10773,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3591,"num_write_kb":229792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1536,"num_bytes_recovered":100663296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,9],"acting":[7,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.10","version":"70'139","reported_seq":721,"reported_epoch":266,"state":"unknown","last_fresh":"2024-08-19T14:18:11.790612+0000","last_change":"2024-08-19T14:18:13.186717+0000","last_active":"2024-08-19T14:18:13.186717+0000","last_peered":"2024-08-19T14:18:11.790612+0000","last_clean":"2024-08-19T14:18:11.790612+0000","last_became_active":"2024-08-19T14:16:42.250702+0000","last_became_peered":"2024-08-19T14:16:42.250702+0000","last_unstale":"2024-08-19T14:18:11.790612+0000","last_undegraded":"2024-08-19T14:18:11.790612+0000","last_fullsized":"2024-08-19T14:18:11.790612+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:15:59.869266+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:15:59.869266+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,9],"acting":[7,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"269'4950","reported_seq":11431,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-08-19T14:18:15.808584+0000","last_change":"2024-08-19T14:17:47.707364+0000","last_active":"2024-08-19T14:18:15.808584+0000","last_peered":"2024-08-19T14:18:15.808584+0000","last_clean":"2024-08-19T14:18:15.808584+0000","last_became_active":"2024-08-19T14:17:47.703601+0000","last_became_peered":"2024-08-19T14:17:47.703601+0000","last_unstale":"2024-08-19T14:18:15.808584+0000","last_undegraded":"2024-08-19T14:18:15.808584+0000","last_fullsized":"2024-08-19T14:18:15.808584+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":4950,"log_dups_size":0,"ondisk_log_size":4950,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:19.018095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":224985088,"num_objects":3433,"num_object_clones":0,"num_object_copies":10299,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3433,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3433,"num_write_kb":219712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1589,"num_bytes_recovered":104136704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"70'139","reported_seq":639,"reported_epoch":271,"state":"peering","last_fresh":"2024-08-19T14:18:16.822162+0000","last_change":"2024-08-19T14:18:17.710988+0000","last_active":"2024-08-19T14:18:17.710988+0000","last_peered":"2024-08-19T14:18:16.822162+0000","last_clean":"2024-08-19T14:18:13.808301+0000","last_became_active":"2024-08-19T14:18:16.821793+0000","last_became_peered":"2024-08-19T14:18:16.821793+0000","last_unstale":"2024-08-19T14:18:16.822162+0000","last_undegraded":"2024-08-19T14:18:16.822162+0000","last_fullsized":"2024-08-19T14:18:16.811499+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:16:06.679075+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:16:06.679075+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,4,8,11],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"272'5034","reported_seq":8264,"reported_epoch":272,"state":"active+clean+scrubbing","last_fresh":"2024-08-19T14:18:17.986102+0000","last_change":"2024-08-19T14:18:17.918786+0000","last_active":"2024-08-19T14:18:17.986102+0000","last_peered":"2024-08-19T14:18:17.986102+0000","last_clean":"2024-08-19T14:18:17.986102+0000","last_became_active":"2024-08-19T14:17:38.695549+0000","last_became_peered":"2024-08-19T14:17:38.695549+0000","last_unstale":"2024-08-19T14:18:17.986102+0000","last_undegraded":"2024-08-19T14:18:17.986102+0000","last_fullsized":"2024-08-19T14:18:17.986102+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":399,"log_size":5034,"log_dups_size":0,"ondisk_log_size":5034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233472000,"num_objects":3563,"num_object_clones":0,"num_object_copies":10689,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3563,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3563,"num_write_kb":228000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1552,"num_bytes_recovered":101711872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.12","version":"70'139","reported_seq":575,"reported_epoch":270,"state":"remapped+peering","last_fresh":"2024-08-19T14:18:15.807537+0000","last_change":"2024-08-19T14:18:15.807537+0000","last_active":"2024-08-19T14:18:15.807537+0000","last_peered":"2024-08-19T14:18:13.811054+0000","last_clean":"2024-08-19T14:18:13.811054+0000","last_became_active":"2024-08-19T14:18:13.809544+0000","last_became_peered":"2024-08-19T14:18:13.809544+0000","last_unstale":"2024-08-19T14:18:15.807537+0000","last_undegraded":"2024-08-19T14:18:15.807537+0000","last_fullsized":"2024-08-19T14:18:15.807537+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:16:00.592678+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:16:00.592678+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"272'5055","reported_seq":9839,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-08-19T14:18:18.300478+0000","last_change":"2024-08-19T14:17:38.687862+0000","last_active":"2024-08-19T14:18:18.300478+0000","last_peered":"2024-08-19T14:18:18.300478+0000","last_clean":"2024-08-19T14:18:18.300478+0000","last_became_active":"2024-08-19T14:17:38.684742+0000","last_became_peered":"2024-08-19T14:17:38.684742+0000","last_unstale":"2024-08-19T14:18:18.300478+0000","last_undegraded":"2024-08-19T14:18:18.300478+0000","last_fullsized":"2024-08-19T14:18:18.300478+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":5055,"log_dups_size":0,"ondisk_log_size":5055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:18.793755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":236322816,"num_objects":3606,"num_object_clones":0,"num_object_copies":10818,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3606,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3606,"num_write_kb":230784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1551,"num_bytes_recovered":101646336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,3],"acting":[9,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.d","version":"70'139","reported_seq":307,"reported_epoch":272,"state":"peering","last_fresh":"2024-08-19T14:18:16.823037+0000","last_change":"2024-08-19T14:18:18.188720+0000","last_active":"2024-08-19T14:18:18.188720+0000","last_peered":"2024-08-19T14:18:16.823037+0000","last_clean":"2024-08-19T14:13:16.698346+0000","last_became_active":"2024-08-19T14:18:16.822737+0000","last_became_peered":"2024-08-19T14:18:16.822737+0000","last_unstale":"2024-08-19T14:18:16.823037+0000","last_undegraded":"2024-08-19T14:18:16.823037+0000","last_fullsized":"2024-08-19T14:18:16.812828+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:16:30.232687+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:16:30.232687+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":5,"num_write":9,"num_write_kb":134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":96169,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"272'5600","reported_seq":5722,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-08-19T14:18:18.146881+0000","last_change":"2024-08-19T14:17:34.958285+0000","last_active":"2024-08-19T14:18:18.146881+0000","last_peered":"2024-08-19T14:18:18.146881+0000","last_clean":"2024-08-19T14:18:18.146881+0000","last_became_active":"2024-08-19T14:17:34.954276+0000","last_became_peered":"2024-08-19T14:17:34.954276+0000","last_unstale":"2024-08-19T14:18:18.146881+0000","last_undegraded":"2024-08-19T14:18:18.146881+0000","last_fullsized":"2024-08-19T14:18:18.146881+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":5600,"log_dups_size":0,"ondisk_log_size":5600,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:20.724329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":237142016,"num_objects":3618,"num_object_clones":0,"num_object_copies":10854,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3618,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3618,"num_write_kb":231584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-19T14:18:12.787134+0000","last_change":"2024-08-19T14:18:12.787134+0000","last_active":"2024-08-19T14:18:12.787134+0000","last_peered":"2024-08-19T14:18:12.787134+0000","last_clean":"2024-08-19T14:18:12.787134+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-19T14:18:12.787134+0000","last_undegraded":"2024-08-19T14:18:12.787134+0000","last_fullsized":"2024-08-19T14:18:12.787134+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:18:12.787134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:18:12.787134+0000","last_clean_scrub_stamp":"2024-08-19T14:18:12.787134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.e","version":"269'5497","reported_seq":5612,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-08-19T14:18:15.667577+0000","last_change":"2024-08-19T14:17:34.973808+0000","last_active":"2024-08-19T14:18:15.667577+0000","last_peered":"2024-08-19T14:18:15.667577+0000","last_clean":"2024-08-19T14:18:15.667577+0000","last_became_active":"2024-08-19T14:17:34.971738+0000","last_became_peered":"2024-08-19T14:17:34.971738+0000","last_unstale":"2024-08-19T14:18:15.667577+0000","last_undegraded":"2024-08-19T14:18:15.667577+0000","last_fullsized":"2024-08-19T14:18:15.667577+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":5497,"log_dups_size":0,"ondisk_log_size":5497,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:16.424148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":230686720,"num_objects":3520,"num_object_clones":0,"num_object_copies":10560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3520,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3520,"num_write_kb":225280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,0],"acting":[11,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.f","version":"70'139","reported_seq":536,"reported_epoch":270,"state":"remapped+peering","last_fresh":"2024-08-19T14:18:15.808487+0000","last_change":"2024-08-19T14:18:15.808487+0000","last_active":"2024-08-19T14:18:14.807870+0000","last_peered":"2024-08-19T14:18:13.810718+0000","last_clean":"2024-08-19T14:18:13.810718+0000","last_became_active":"2024-08-19T14:18:13.808901+0000","last_became_peered":"2024-08-19T14:18:13.808901+0000","last_unstale":"2024-08-19T14:18:15.808487+0000","last_undegraded":"2024-08-19T14:18:15.808487+0000","last_fullsized":"2024-08-19T14:18:15.808487+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:16:10.672834+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"272'5687","reported_seq":5809,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-08-19T14:18:18.135519+0000","last_change":"2024-08-19T14:17:34.971437+0000","last_active":"2024-08-19T14:18:18.135519+0000","last_peered":"2024-08-19T14:18:18.135519+0000","last_clean":"2024-08-19T14:18:18.135519+0000","last_became_active":"2024-08-19T14:17:34.969333+0000","last_became_peered":"2024-08-19T14:17:34.969333+0000","last_unstale":"2024-08-19T14:18:18.135519+0000","last_undegraded":"2024-08-19T14:18:18.135519+0000","last_fullsized":"2024-08-19T14:18:18.135519+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":5687,"log_dups_size":0,"ondisk_log_size":5687,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:20.724310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":240615424,"num_objects":3671,"num_object_clones":0,"num_object_copies":11013,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3671,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3671,"num_write_kb":234976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,9],"acting":[7,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.e","version":"70'139","reported_seq":287,"reported_epoch":272,"state":"peering","last_fresh":"2024-08-19T14:18:17.693484+0000","last_change":"2024-08-19T14:18:17.693484+0000","last_active":"2024-08-19T14:18:17.693484+0000","last_peered":"2024-08-19T14:18:16.823075+0000","last_clean":"2024-08-19T14:10:40.177137+0000","last_became_active":"2024-08-19T14:18:16.822800+0000","last_became_peered":"2024-08-19T14:18:16.822800+0000","last_unstale":"2024-08-19T14:18:17.693484+0000","last_undegraded":"2024-08-19T14:18:17.693484+0000","last_fullsized":"2024-08-19T14:18:17.693484+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:16:36.969114+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:16:36.969114+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":5,"num_write":8,"num_write_kb":134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":96169,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"270'5412","reported_seq":5530,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-08-19T14:18:16.585070+0000","last_change":"2024-08-19T14:17:34.976154+0000","last_active":"2024-08-19T14:18:16.585070+0000","last_peered":"2024-08-19T14:18:16.585070+0000","last_clean":"2024-08-19T14:18:16.585070+0000","last_became_active":"2024-08-19T14:17:34.973857+0000","last_became_peered":"2024-08-19T14:17:34.973857+0000","last_unstale":"2024-08-19T14:18:16.585070+0000","last_undegraded":"2024-08-19T14:18:16.585070+0000","last_fullsized":"2024-08-19T14:18:16.585070+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":5412,"log_dups_size":0,"ondisk_log_size":5412,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":226852864,"num_objects":3461,"num_object_clones":0,"num_object_copies":10383,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3461,"num_write_kb":221536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,9],"acting":[5,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"0'0","reported_seq":1,"reported_epoch":269,"state":"unknown","last_fresh":"2024-08-19T14:18:14.806438+0000","last_change":"2024-08-19T14:18:15.703140+0000","last_active":"0.000000","last_peered":"0.000000","last_clean":"0.000000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-19T14:18:14.806438+0000","last_undegraded":"2024-08-19T14:18:14.806438+0000","last_fullsized":"2024-08-19T14:18:14.806438+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:16:06.679075+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:16:06.679075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.9","version":"269'4855","reported_seq":4971,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-08-19T14:18:14.819470+0000","last_change":"2024-08-19T14:17:19.745357+0000","last_active":"2024-08-19T14:18:14.819470+0000","last_peered":"2024-08-19T14:18:14.819470+0000","last_clean":"2024-08-19T14:18:14.819470+0000","last_became_active":"2024-08-19T14:17:19.743605+0000","last_became_peered":"2024-08-19T14:17:19.743605+0000","last_unstale":"2024-08-19T14:18:14.819470+0000","last_undegraded":"2024-08-19T14:18:14.819470+0000","last_fullsized":"2024-08-19T14:18:14.819470+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":4855,"log_dups_size":0,"ondisk_log_size":4855,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:17.415643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":219643904,"num_objects":3351,"num_object_clones":0,"num_object_copies":10053,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3351,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3351,"num_write_kb":214496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"272'5543","reported_seq":5661,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-08-19T14:18:17.968126+0000","last_change":"2024-08-19T14:17:34.964717+0000","last_active":"2024-08-19T14:18:17.968126+0000","last_peered":"2024-08-19T14:18:17.968126+0000","last_clean":"2024-08-19T14:18:17.968126+0000","last_became_active":"2024-08-19T14:17:34.962596+0000","last_became_peered":"2024-08-19T14:17:34.962596+0000","last_unstale":"2024-08-19T14:18:17.968126+0000","last_undegraded":"2024-08-19T14:18:17.968126+0000","last_fullsized":"2024-08-19T14:18:17.968126+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":5543,"log_dups_size":0,"ondisk_log_size":5543,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:19.914060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234029056,"num_objects":3571,"num_object_clones":0,"num_object_copies":10713,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3571,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3571,"num_write_kb":228544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,8],"acting":[4,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"70'139","reported_seq":634,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-08-19T14:18:15.807844+0000","last_change":"2024-08-19T14:16:42.249971+0000","last_active":"2024-08-19T14:18:15.807844+0000","last_peered":"2024-08-19T14:18:15.807844+0000","last_clean":"2024-08-19T14:18:15.807844+0000","last_became_active":"2024-08-19T14:16:42.249518+0000","last_became_peered":"2024-08-19T14:16:42.249518+0000","last_unstale":"2024-08-19T14:18:15.807844+0000","last_undegraded":"2024-08-19T14:18:15.807844+0000","last_fullsized":"2024-08-19T14:18:15.807844+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:15:44.541487+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:15:44.541487+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:20.121968+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,8],"acting":[5,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.b","version":"268'5412","reported_seq":5526,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-08-19T14:18:14.230991+0000","last_change":"2024-08-19T14:17:34.975300+0000","last_active":"2024-08-19T14:18:14.230991+0000","last_peered":"2024-08-19T14:18:14.230991+0000","last_clean":"2024-08-19T14:18:14.230991+0000","last_became_active":"2024-08-19T14:17:34.972934+0000","last_became_peered":"2024-08-19T14:17:34.972934+0000","last_unstale":"2024-08-19T14:18:14.230991+0000","last_undegraded":"2024-08-19T14:18:14.230991+0000","last_fullsized":"2024-08-19T14:18:14.230991+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":5412,"log_dups_size":0,"ondisk_log_size":5412,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:15.464409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":224624640,"num_objects":3427,"num_object_clones":0,"num_object_copies":10281,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3427,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3427,"num_write_kb":219360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"70'139","reported_seq":756,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-08-19T14:18:17.700426+0000","last_change":"2024-08-19T14:16:42.252897+0000","last_active":"2024-08-19T14:18:17.700426+0000","last_peered":"2024-08-19T14:18:17.700426+0000","last_clean":"2024-08-19T14:18:17.700426+0000","last_became_active":"2024-08-19T14:16:42.252391+0000","last_became_peered":"2024-08-19T14:16:42.252391+0000","last_unstale":"2024-08-19T14:18:17.700426+0000","last_undegraded":"2024-08-19T14:18:17.700426+0000","last_fullsized":"2024-08-19T14:18:17.700426+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:16:03.918046+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:20.724362+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"269'5093","reported_seq":5209,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-08-19T14:18:15.683913+0000","last_change":"2024-08-19T14:17:19.755178+0000","last_active":"2024-08-19T14:18:15.683913+0000","last_peered":"2024-08-19T14:18:15.683913+0000","last_clean":"2024-08-19T14:18:15.683913+0000","last_became_active":"2024-08-19T14:17:19.753063+0000","last_became_peered":"2024-08-19T14:17:19.753063+0000","last_unstale":"2024-08-19T14:18:15.683913+0000","last_undegraded":"2024-08-19T14:18:15.683913+0000","last_fullsized":"2024-08-19T14:18:15.683913+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":5093,"log_dups_size":0,"ondisk_log_size":5093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:16.424112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":232292352,"num_objects":3544,"num_object_clones":0,"num_object_copies":10632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3544,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3544,"num_write_kb":226848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,4],"acting":[11,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.a","version":"70'139","reported_seq":634,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-08-19T14:18:17.696284+0000","last_change":"2024-08-19T14:16:02.818848+0000","last_active":"2024-08-19T14:18:17.696284+0000","last_peered":"2024-08-19T14:18:17.696284+0000","last_clean":"2024-08-19T14:18:17.696284+0000","last_became_active":"2024-08-19T14:13:22.728779+0000","last_became_peered":"2024-08-19T14:13:22.728779+0000","last_unstale":"2024-08-19T14:18:17.696284+0000","last_undegraded":"2024-08-19T14:18:17.696284+0000","last_fullsized":"2024-08-19T14:18:17.696284+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:16:02.818774+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:16:02.818774+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:20.767528+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,0],"acting":[9,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.6","version":"269'5141","reported_seq":5259,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-08-19T14:18:15.808118+0000","last_change":"2024-08-19T14:17:19.750488+0000","last_active":"2024-08-19T14:18:15.808118+0000","last_peered":"2024-08-19T14:18:15.808118+0000","last_clean":"2024-08-19T14:18:15.808118+0000","last_became_active":"2024-08-19T14:17:19.748180+0000","last_became_peered":"2024-08-19T14:17:19.748180+0000","last_unstale":"2024-08-19T14:18:15.808118+0000","last_undegraded":"2024-08-19T14:18:15.808118+0000","last_fullsized":"2024-08-19T14:18:15.808118+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":5141,"log_dups_size":0,"ondisk_log_size":5141,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:19.018018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":236879872,"num_objects":3614,"num_object_clones":0,"num_object_copies":10842,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3614,"num_write_kb":231328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"70'139","reported_seq":671,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-08-19T14:18:14.808010+0000","last_change":"2024-08-19T14:18:13.806539+0000","last_active":"2024-08-19T14:18:14.808010+0000","last_peered":"2024-08-19T14:18:14.808010+0000","last_clean":"2024-08-19T14:18:14.808010+0000","last_became_active":"2024-08-19T14:18:13.806013+0000","last_became_peered":"2024-08-19T14:18:13.806013+0000","last_unstale":"2024-08-19T14:18:14.808010+0000","last_undegraded":"2024-08-19T14:18:14.808010+0000","last_fullsized":"2024-08-19T14:18:14.808010+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:16:29.256959+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:16:29.256959+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109226,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.7","version":"268'4847","reported_seq":4961,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-08-19T14:18:14.208392+0000","last_change":"2024-08-19T14:17:19.746696+0000","last_active":"2024-08-19T14:18:14.208392+0000","last_peered":"2024-08-19T14:18:14.208392+0000","last_clean":"2024-08-19T14:18:14.208392+0000","last_became_active":"2024-08-19T14:17:19.742604+0000","last_became_peered":"2024-08-19T14:17:19.742604+0000","last_unstale":"2024-08-19T14:18:14.208392+0000","last_undegraded":"2024-08-19T14:18:14.208392+0000","last_fullsized":"2024-08-19T14:18:14.208392+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":4847,"log_dups_size":0,"ondisk_log_size":4847,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:15.464437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":221118464,"num_objects":3374,"num_object_clones":0,"num_object_copies":10122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3374,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3374,"num_write_kb":215936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"70'139","reported_seq":433,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-08-19T14:18:14.807310+0000","last_change":"2024-08-19T14:18:13.815131+0000","last_active":"2024-08-19T14:18:14.807310+0000","last_peered":"2024-08-19T14:18:14.807310+0000","last_clean":"2024-08-19T14:18:14.807310+0000","last_became_active":"2024-08-19T14:18:13.814779+0000","last_became_peered":"2024-08-19T14:18:13.814779+0000","last_unstale":"2024-08-19T14:18:14.807310+0000","last_undegraded":"2024-08-19T14:18:14.807310+0000","last_fullsized":"2024-08-19T14:18:14.807310+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:16:30.232687+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:16:30.232687+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,4],"acting":[11,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"272'5152","reported_seq":5274,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-08-19T14:18:18.327549+0000","last_change":"2024-08-19T14:17:19.744537+0000","last_active":"2024-08-19T14:18:18.327549+0000","last_peered":"2024-08-19T14:18:18.327549+0000","last_clean":"2024-08-19T14:18:18.327549+0000","last_became_active":"2024-08-19T14:17:19.741977+0000","last_became_peered":"2024-08-19T14:17:19.741977+0000","last_unstale":"2024-08-19T14:18:18.327549+0000","last_undegraded":"2024-08-19T14:18:18.327549+0000","last_fullsized":"2024-08-19T14:18:18.327549+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":5152,"log_dups_size":0,"ondisk_log_size":5152,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:18.639297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":239370240,"num_objects":3652,"num_object_clones":0,"num_object_copies":10956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3652,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3652,"num_write_kb":233760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,5],"acting":[1,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"70'139","reported_seq":626,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-08-19T14:18:17.695726+0000","last_change":"2024-08-19T14:18:13.808038+0000","last_active":"2024-08-19T14:18:17.695726+0000","last_peered":"2024-08-19T14:18:17.695726+0000","last_clean":"2024-08-19T14:18:17.695726+0000","last_became_active":"2024-08-19T14:18:13.807711+0000","last_became_peered":"2024-08-19T14:18:13.807711+0000","last_unstale":"2024-08-19T14:18:17.695726+0000","last_undegraded":"2024-08-19T14:18:17.695726+0000","last_fullsized":"2024-08-19T14:18:17.695726+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:16:36.969114+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:16:36.969114+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:20.767499+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,4],"acting":[9,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"270'5013","reported_seq":5157,"reported_epoch":270,"state":"active+clean+scrubbing","last_fresh":"2024-08-19T14:18:16.594604+0000","last_change":"2024-08-19T14:18:16.090679+0000","last_active":"2024-08-19T14:18:16.594604+0000","last_peered":"2024-08-19T14:18:16.594604+0000","last_clean":"2024-08-19T14:18:16.594604+0000","last_became_active":"2024-08-19T14:17:19.745911+0000","last_became_peered":"2024-08-19T14:17:19.745911+0000","last_unstale":"2024-08-19T14:18:16.594604+0000","last_undegraded":"2024-08-19T14:18:16.594604+0000","last_fullsized":"2024-08-19T14:18:16.594604+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":2049,"log_size":5013,"log_dups_size":0,"ondisk_log_size":5013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":231571456,"num_objects":3533,"num_object_clones":0,"num_object_copies":10599,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":226144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"70'139","reported_seq":539,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-08-19T14:18:15.808202+0000","last_change":"2024-08-19T14:18:13.809945+0000","last_active":"2024-08-19T14:18:15.808202+0000","last_peered":"2024-08-19T14:18:15.808202+0000","last_clean":"2024-08-19T14:18:15.808202+0000","last_became_active":"2024-08-19T14:18:13.808446+0000","last_became_peered":"2024-08-19T14:18:13.808446+0000","last_unstale":"2024-08-19T14:18:15.808202+0000","last_undegraded":"2024-08-19T14:18:15.808202+0000","last_fullsized":"2024-08-19T14:18:15.808202+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:16:10.672834+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:20.121936+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"272'4235","reported_seq":4362,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-08-19T14:18:17.929689+0000","last_change":"2024-08-19T14:17:34.971150+0000","last_active":"2024-08-19T14:18:17.929689+0000","last_peered":"2024-08-19T14:18:17.929689+0000","last_clean":"2024-08-19T14:18:17.929689+0000","last_became_active":"2024-08-19T14:17:34.968430+0000","last_became_peered":"2024-08-19T14:17:34.968430+0000","last_unstale":"2024-08-19T14:18:17.929689+0000","last_undegraded":"2024-08-19T14:18:17.929689+0000","last_fullsized":"2024-08-19T14:18:17.929689+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":4235,"log_dups_size":0,"ondisk_log_size":4235,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:19.914025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":116080640,"num_objects":1771,"num_object_clones":0,"num_object_copies":5313,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1771,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1771,"num_write_kb":113360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,11],"acting":[4,0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"70'139","reported_seq":635,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-08-19T14:18:17.696150+0000","last_change":"2024-08-19T14:18:13.809263+0000","last_active":"2024-08-19T14:18:17.696150+0000","last_peered":"2024-08-19T14:18:17.696150+0000","last_clean":"2024-08-19T14:18:17.696150+0000","last_became_active":"2024-08-19T14:18:13.808505+0000","last_became_peered":"2024-08-19T14:18:13.808505+0000","last_unstale":"2024-08-19T14:18:17.696150+0000","last_undegraded":"2024-08-19T14:18:17.696150+0000","last_fullsized":"2024-08-19T14:18:17.696150+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:16:30.043526+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:16:30.043526+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:20.767463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,11],"acting":[9,0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.0","version":"269'4156","reported_seq":4278,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-08-19T14:18:15.662298+0000","last_change":"2024-08-19T14:17:34.964487+0000","last_active":"2024-08-19T14:18:15.662298+0000","last_peered":"2024-08-19T14:18:15.662298+0000","last_clean":"2024-08-19T14:18:15.662298+0000","last_became_active":"2024-08-19T14:17:34.963390+0000","last_became_peered":"2024-08-19T14:17:34.963390+0000","last_unstale":"2024-08-19T14:18:15.662298+0000","last_undegraded":"2024-08-19T14:18:15.662298+0000","last_fullsized":"2024-08-19T14:18:15.662298+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":4156,"log_dups_size":0,"ondisk_log_size":4156,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:16.424131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113311744,"num_objects":1729,"num_object_clones":0,"num_object_copies":5187,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1729,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1729,"num_write_kb":110656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,1],"acting":[11,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.2","version":"70'139","reported_seq":579,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-08-19T14:18:15.808897+0000","last_change":"2024-08-19T14:18:13.809943+0000","last_active":"2024-08-19T14:18:15.808897+0000","last_peered":"2024-08-19T14:18:15.808897+0000","last_clean":"2024-08-19T14:18:15.808897+0000","last_became_active":"2024-08-19T14:18:13.809344+0000","last_became_peered":"2024-08-19T14:18:13.809344+0000","last_unstale":"2024-08-19T14:18:15.808897+0000","last_undegraded":"2024-08-19T14:18:15.808897+0000","last_fullsized":"2024-08-19T14:18:15.808897+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:16:00.592678+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:16:00.592678+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:19.984453+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,8],"acting":[0,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"269'4107","reported_seq":4222,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-08-19T14:18:15.807938+0000","last_change":"2024-08-19T14:17:34.964957+0000","last_active":"2024-08-19T14:18:15.807938+0000","last_peered":"2024-08-19T14:18:15.807938+0000","last_clean":"2024-08-19T14:18:15.807938+0000","last_became_active":"2024-08-19T14:17:34.963892+0000","last_became_peered":"2024-08-19T14:17:34.963892+0000","last_unstale":"2024-08-19T14:18:15.807938+0000","last_undegraded":"2024-08-19T14:18:15.807938+0000","last_fullsized":"2024-08-19T14:18:15.807938+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:16:55.393990+0000","objects_scrubbed":0,"log_size":4107,"log_dups_size":0,"ondisk_log_size":4107,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:19.017984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108707840,"num_objects":1658,"num_object_clones":0,"num_object_copies":4974,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1658,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1658,"num_write_kb":106160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,11],"acting":[0,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"70'139","reported_seq":638,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-08-19T14:18:17.711994+0000","last_change":"2024-08-19T14:18:13.805016+0000","last_active":"2024-08-19T14:18:17.711994+0000","last_peered":"2024-08-19T14:18:17.711994+0000","last_clean":"2024-08-19T14:18:17.711994+0000","last_became_active":"2024-08-19T14:18:13.804721+0000","last_became_peered":"2024-08-19T14:18:13.804721+0000","last_unstale":"2024-08-19T14:18:17.711994+0000","last_undegraded":"2024-08-19T14:18:17.711994+0000","last_fullsized":"2024-08-19T14:18:17.711994+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:16:06.679075+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:16:06.679075+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:20.620986+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"272'4322","reported_seq":4475,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-08-19T14:18:18.284566+0000","last_change":"2024-08-19T14:18:17.084657+0000","last_active":"2024-08-19T14:18:18.284566+0000","last_peered":"2024-08-19T14:18:18.284566+0000","last_clean":"2024-08-19T14:18:18.284566+0000","last_became_active":"2024-08-19T14:17:34.968554+0000","last_became_peered":"2024-08-19T14:17:34.968554+0000","last_unstale":"2024-08-19T14:18:18.284566+0000","last_undegraded":"2024-08-19T14:18:18.284566+0000","last_fullsized":"2024-08-19T14:18:18.284566+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'4295","last_scrub_stamp":"2024-08-19T14:18:17.084447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:17.084447+0000","objects_scrubbed":1731,"log_size":4322,"log_dups_size":0,"ondisk_log_size":4322,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:29.607335+0000","scrub_duration":357,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":115539968,"num_objects":1763,"num_object_clones":0,"num_object_copies":5289,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1763,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32,"num_write_kb":2048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,5],"acting":[9,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.0","version":"70'139","reported_seq":739,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-08-19T14:18:17.695996+0000","last_change":"2024-08-19T14:18:13.807473+0000","last_active":"2024-08-19T14:18:17.695996+0000","last_peered":"2024-08-19T14:18:17.695996+0000","last_clean":"2024-08-19T14:18:17.695996+0000","last_became_active":"2024-08-19T14:18:13.807133+0000","last_became_peered":"2024-08-19T14:18:13.807133+0000","last_unstale":"2024-08-19T14:18:17.695996+0000","last_undegraded":"2024-08-19T14:18:17.695996+0000","last_fullsized":"2024-08-19T14:18:17.695996+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:15:59.869266+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:15:59.869266+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:20.724346+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,9],"acting":[7,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":36,"stat_sum":{"num_bytes":7351107584,"num_objects":112169,"num_object_clones":0,"num_object_copies":336507,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":112169,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105124,"num_write_kb":6727936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28637,"num_bytes_recovered":1876754432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":23041081344,"data_stored":23041081344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":179713,"ondisk_log_size":179713,"up":108,"acting":108,"num_store_stats":9},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":470309,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":22,"num_read_kb":18,"num_write":32,"num_write_kb":491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":569896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2129920,"data_stored":2115712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2641,"ondisk_log_size":2641,"up":60,"acting":57,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187539,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2499644,"kb_used_data":2211628,"kb_used_omap":1,"kb_used_meta":287998,"kb_avail":91245508,"statfs":{"total":95995035648,"available":93435400192,"internally_reserved":0,"allocated":2264707072,"data_stored":2263419288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":294910404},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":343597383769,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":74,"seq":317827579997,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2972772,"kb_used_data":2644496,"kb_used_omap":1,"kb_used_meta":328254,"kb_avail":90772380,"statfs":{"total":95995035648,"available":92950917120,"internally_reserved":0,"allocated":2707963904,"data_stored":2706664670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":336132545},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874338,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2288444,"kb_used_data":2050408,"kb_used_omap":1,"kb_used_meta":238014,"kb_avail":91456708,"statfs":{"total":95995035648,"available":93651668992,"internally_reserved":0,"allocated":2099617792,"data_stored":2098330308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":243726789},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":57,"seq":244813135973,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2509160,"kb_used_data":2185040,"kb_used_omap":1,"kb_used_meta":324094,"kb_avail":91235992,"statfs":{"total":95995035648,"available":93425655808,"internally_reserved":0,"allocated":2237480960,"data_stored":2236181726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":331872714},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":206158430312,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691949,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2796752,"kb_used_data":2590656,"kb_used_omap":1,"kb_used_meta":206078,"kb_avail":90948400,"statfs":{"total":95995035648,"available":93131161600,"internally_reserved":0,"allocated":2652831744,"data_stored":2651536822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":211024325},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986290,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4222692,"kb_used_data":3902736,"kb_used_omap":1,"kb_used_meta":319934,"kb_avail":89522460,"statfs":{"total":95995035648,"available":91670999040,"internally_reserved":0,"allocated":3996401664,"data_stored":3995102430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":327612869},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149815,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2434728,"kb_used_data":2128668,"kb_used_omap":1,"kb_used_meta":306046,"kb_avail":91310424,"statfs":{"total":95995035648,"available":93501874176,"internally_reserved":0,"allocated":2179756032,"data_stored":2178472722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":313391559},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411452,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574977,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2945620,"kb_used_data":2653444,"kb_used_omap":1,"kb_used_meta":292158,"kb_avail":90799532,"statfs":{"total":95995035648,"available":92978720768,"internally_reserved":0,"allocated":2717126656,"data_stored":2715831394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":299170245},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738501,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2200120,"kb_used_data":1980140,"kb_used_omap":1,"kb_used_meta":219966,"kb_avail":91545032,"statfs":{"total":95995035648,"available":93742112768,"internally_reserved":0,"allocated":2027663360,"data_stored":2026375576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":225245637},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2057895936,"data_stored":2057895936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2740715520,"data_stored":2740715520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2221801472,"data_stored":2221801472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":3995598848,"data_stored":3995598848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2672230400,"data_stored":2672230400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2245656576,"data_stored":2245656576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":2112749568,"data_stored":2112749568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":2720595968,"data_stored":2720595968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":2273837056,"data_stored":2273837056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T14:18:21.380 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-08-19T14:18:21.381 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd pool set .mgr pg_num 31 2024-08-19T14:18:21.400 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:18:21.413 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:18:21.421 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:21.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:21.422 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:18:21.522 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:18:21.545 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:21.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:21.546 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:18:21.647 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:18:21.668 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:21.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:21.669 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:18:21.679 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:18:21.697 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:18:21.770 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:18:21.791 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:21.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:21.792 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:18:21.893 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:18:21.915 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:21.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:21.916 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:18:21.993 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:21 smithi031 ceph-mon[16336]: 3.1 scrub ok 2024-08-19T14:18:21.994 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:21 smithi031 ceph-mon[16336]: 3.4 scrub ok 2024-08-19T14:18:21.994 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:21 smithi031 ceph-mon[16336]: 3.15 scrub starts 2024-08-19T14:18:21.994 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:21 smithi031 ceph-mon[16336]: 3.19 scrub starts 2024-08-19T14:18:21.994 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:21 smithi031 ceph-mon[16336]: 3.15 scrub ok 2024-08-19T14:18:21.994 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:21 smithi031 ceph-mon[16336]: 3.a scrub starts 2024-08-19T14:18:21.994 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:21 smithi031 ceph-mon[16336]: 1.7 scrub starts 2024-08-19T14:18:21.994 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:21 smithi031 ceph-mon[16336]: 1.7 scrub ok 2024-08-19T14:18:21.994 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:21 smithi031 ceph-mon[16336]: 3.19 scrub ok 2024-08-19T14:18:21.995 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:21 smithi031 ceph-mon[16336]: 3.18 scrub starts 2024-08-19T14:18:21.995 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:21 smithi031 ceph-mon[16336]: 3.a scrub ok 2024-08-19T14:18:22.016 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:18:22.038 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:22.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:22.039 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:18:22.043 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:18:22.140 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:18:22.162 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:22.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:22.163 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:18:22.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:21 smithi032 ceph-mon[20004]: 3.1 scrub ok 2024-08-19T14:18:22.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:21 smithi032 ceph-mon[20004]: 3.4 scrub ok 2024-08-19T14:18:22.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:21 smithi032 ceph-mon[20004]: 3.15 scrub starts 2024-08-19T14:18:22.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:21 smithi032 ceph-mon[20004]: 3.19 scrub starts 2024-08-19T14:18:22.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:21 smithi032 ceph-mon[20004]: 3.15 scrub ok 2024-08-19T14:18:22.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:21 smithi032 ceph-mon[20004]: 3.a scrub starts 2024-08-19T14:18:22.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:21 smithi032 ceph-mon[20004]: 1.7 scrub starts 2024-08-19T14:18:22.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:21 smithi032 ceph-mon[20004]: 1.7 scrub ok 2024-08-19T14:18:22.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:21 smithi032 ceph-mon[20004]: 3.19 scrub ok 2024-08-19T14:18:22.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:21 smithi032 ceph-mon[20004]: 3.18 scrub starts 2024-08-19T14:18:22.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:21 smithi032 ceph-mon[20004]: 3.a scrub ok 2024-08-19T14:18:22.264 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:18:22.288 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:22.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:22.289 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:18:22.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:21 smithi045 ceph-mon[20143]: 3.1 scrub ok 2024-08-19T14:18:22.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:21 smithi045 ceph-mon[20143]: 3.4 scrub ok 2024-08-19T14:18:22.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:21 smithi045 ceph-mon[20143]: 3.15 scrub starts 2024-08-19T14:18:22.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:21 smithi045 ceph-mon[20143]: 3.19 scrub starts 2024-08-19T14:18:22.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:21 smithi045 ceph-mon[20143]: 3.15 scrub ok 2024-08-19T14:18:22.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:21 smithi045 ceph-mon[20143]: 3.a scrub starts 2024-08-19T14:18:22.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:21 smithi045 ceph-mon[20143]: 1.7 scrub starts 2024-08-19T14:18:22.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:21 smithi045 ceph-mon[20143]: 1.7 scrub ok 2024-08-19T14:18:22.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:21 smithi045 ceph-mon[20143]: 3.19 scrub ok 2024-08-19T14:18:22.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:21 smithi045 ceph-mon[20143]: 3.18 scrub starts 2024-08-19T14:18:22.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:21 smithi045 ceph-mon[20143]: 3.a scrub ok 2024-08-19T14:18:22.318 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:2024-08-19T14:18:22.318515+0000 min lat: 0.00117714 max lat: 2.24324 avg lat: 0.0104656 2024-08-19T14:18:22.318 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-19T14:18:22.318 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 80 16 118893 118877 92.8615 0 - 0.0104656 2024-08-19T14:18:22.390 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:18:22.416 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:22.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:22.417 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:18:22.518 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:18:22.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:22.539 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:22.540 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:18:22.641 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:18:22.664 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:22.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:22.665 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:18:22.697 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:18:22.702 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:18:22.702 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:18:22.766 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:18:22.792 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:22.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:22.793 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:18:22.861 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is unset 2024-08-19T14:18:22.894 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:18:22.918 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:22.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:22.919 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:18:23.020 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:18:23.042 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:23.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:23.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:18:23.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:22 smithi031 ceph-mon[16336]: 3.6 scrub starts 2024-08-19T14:18:23.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:22 smithi031 ceph-mon[16336]: 3.1d scrub starts 2024-08-19T14:18:23.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:22 smithi031 ceph-mon[16336]: 3.6 scrub ok 2024-08-19T14:18:23.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:22 smithi031 ceph-mon[16336]: 1.4 scrub starts 2024-08-19T14:18:23.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:22 smithi031 ceph-mon[16336]: 1.4 scrub ok 2024-08-19T14:18:23.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:22 smithi031 ceph-mon[16336]: 1.3 scrub starts 2024-08-19T14:18:23.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:22 smithi031 ceph-mon[16336]: 1.3 scrub ok 2024-08-19T14:18:23.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:22 smithi031 ceph-mon[16336]: 3.18 scrub ok 2024-08-19T14:18:23.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:22 smithi031 ceph-mon[16336]: 1.1 scrub starts 2024-08-19T14:18:23.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:22 smithi031 ceph-mon[16336]: 1.1 scrub ok 2024-08-19T14:18:23.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:22 smithi031 ceph-mon[16336]: 3.1d scrub ok 2024-08-19T14:18:23.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:22 smithi031 ceph-mon[16336]: pgmap v627: 57 pgs: 4 peering, 3 active+clean+scrubbing, 50 active+clean; 7.2 GiB data, 25 GiB used, 780 GiB / 805 GiB avail; 3.7 KiB/s rd, 4 op/s 2024-08-19T14:18:23.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:22 smithi031 ceph-mon[16336]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-08-19T14:18:23.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:22 smithi031 ceph-mon[16336]: 1.e scrub starts 2024-08-19T14:18:23.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:22 smithi031 ceph-mon[16336]: 1.e scrub ok 2024-08-19T14:18:23.118 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:22 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:18:23.118 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:22 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:18:23.118 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:22 smithi031 ceph-mon[16336]: 1.10 scrub starts 2024-08-19T14:18:23.118 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:22 smithi031 ceph-mon[16336]: 1.10 scrub ok 2024-08-19T14:18:23.118 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:22 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2305990264' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:18:23.144 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:18:23.165 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:23.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:23.167 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:18:23.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:22 smithi032 ceph-mon[20004]: 3.6 scrub starts 2024-08-19T14:18:23.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:22 smithi032 ceph-mon[20004]: 3.1d scrub starts 2024-08-19T14:18:23.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:22 smithi032 ceph-mon[20004]: 3.6 scrub ok 2024-08-19T14:18:23.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:22 smithi032 ceph-mon[20004]: 1.4 scrub starts 2024-08-19T14:18:23.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:22 smithi032 ceph-mon[20004]: 1.4 scrub ok 2024-08-19T14:18:23.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:22 smithi032 ceph-mon[20004]: 1.3 scrub starts 2024-08-19T14:18:23.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:22 smithi032 ceph-mon[20004]: 1.3 scrub ok 2024-08-19T14:18:23.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:22 smithi032 ceph-mon[20004]: 3.18 scrub ok 2024-08-19T14:18:23.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:22 smithi032 ceph-mon[20004]: 1.1 scrub starts 2024-08-19T14:18:23.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:22 smithi032 ceph-mon[20004]: 1.1 scrub ok 2024-08-19T14:18:23.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:22 smithi032 ceph-mon[20004]: 3.1d scrub ok 2024-08-19T14:18:23.259 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:22 smithi032 ceph-mon[20004]: pgmap v627: 57 pgs: 4 peering, 3 active+clean+scrubbing, 50 active+clean; 7.2 GiB data, 25 GiB used, 780 GiB / 805 GiB avail; 3.7 KiB/s rd, 4 op/s 2024-08-19T14:18:23.259 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:22 smithi032 ceph-mon[20004]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-08-19T14:18:23.259 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:22 smithi032 ceph-mon[20004]: 1.e scrub starts 2024-08-19T14:18:23.259 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:22 smithi032 ceph-mon[20004]: 1.e scrub ok 2024-08-19T14:18:23.259 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:22 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:18:23.259 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:22 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:18:23.259 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:22 smithi032 ceph-mon[20004]: 1.10 scrub starts 2024-08-19T14:18:23.259 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:22 smithi032 ceph-mon[20004]: 1.10 scrub ok 2024-08-19T14:18:23.259 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:22 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2305990264' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:18:23.267 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:18:23.288 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:23.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:23.289 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:18:23.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:22 smithi045 ceph-mon[20143]: 3.6 scrub starts 2024-08-19T14:18:23.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:22 smithi045 ceph-mon[20143]: 3.1d scrub starts 2024-08-19T14:18:23.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:22 smithi045 ceph-mon[20143]: 3.6 scrub ok 2024-08-19T14:18:23.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:22 smithi045 ceph-mon[20143]: 1.4 scrub starts 2024-08-19T14:18:23.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:22 smithi045 ceph-mon[20143]: 1.4 scrub ok 2024-08-19T14:18:23.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:22 smithi045 ceph-mon[20143]: 1.3 scrub starts 2024-08-19T14:18:23.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:22 smithi045 ceph-mon[20143]: 1.3 scrub ok 2024-08-19T14:18:23.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:22 smithi045 ceph-mon[20143]: 3.18 scrub ok 2024-08-19T14:18:23.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:22 smithi045 ceph-mon[20143]: 1.1 scrub starts 2024-08-19T14:18:23.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:22 smithi045 ceph-mon[20143]: 1.1 scrub ok 2024-08-19T14:18:23.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:22 smithi045 ceph-mon[20143]: 3.1d scrub ok 2024-08-19T14:18:23.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:22 smithi045 ceph-mon[20143]: pgmap v627: 57 pgs: 4 peering, 3 active+clean+scrubbing, 50 active+clean; 7.2 GiB data, 25 GiB used, 780 GiB / 805 GiB avail; 3.7 KiB/s rd, 4 op/s 2024-08-19T14:18:23.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:22 smithi045 ceph-mon[20143]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-08-19T14:18:23.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:22 smithi045 ceph-mon[20143]: 1.e scrub starts 2024-08-19T14:18:23.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:22 smithi045 ceph-mon[20143]: 1.e scrub ok 2024-08-19T14:18:23.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:22 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:18:23.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:22 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:18:23.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:22 smithi045 ceph-mon[20143]: 1.10 scrub starts 2024-08-19T14:18:23.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:22 smithi045 ceph-mon[20143]: 1.10 scrub ok 2024-08-19T14:18:23.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:22 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2305990264' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:18:23.318 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 81 16 118893 118877 91.7151 0 - 0.0104656 2024-08-19T14:18:23.390 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:18:23.415 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:23.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:23.416 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:18:23.517 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:18:23.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:23.539 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:23.539 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:18:23.640 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:18:23.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:23.667 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:23.668 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:18:23.769 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:18:23.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:23.793 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:23.793 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:18:23.864 INFO:teuthology.orchestra.run.smithi031.stderr:set pool 1 pg_num to 31 2024-08-19T14:18:23.893 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:18:23.915 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:23.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:23.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:18:24.017 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:18:24.040 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:24.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:24.041 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:18:24.051 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:23 smithi031 ceph-mon[16336]: 3.23 scrub starts 2024-08-19T14:18:24.051 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:23 smithi031 ceph-mon[16336]: 3.23 scrub ok 2024-08-19T14:18:24.051 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:23 smithi031 ceph-mon[16336]: 1.13 scrub starts 2024-08-19T14:18:24.051 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:23 smithi031 ceph-mon[16336]: 1.13 scrub ok 2024-08-19T14:18:24.051 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:23 smithi031 ceph-mon[16336]: 3.f scrub starts 2024-08-19T14:18:24.051 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:23 smithi031 ceph-mon[16336]: 1.6 scrub starts 2024-08-19T14:18:24.051 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:23 smithi031 ceph-mon[16336]: 1.6 scrub ok 2024-08-19T14:18:24.051 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:23 smithi031 ceph-mon[16336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:18:24.051 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:23 smithi031 ceph-mon[16336]: Cluster is now healthy 2024-08-19T14:18:24.051 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:23 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2305990264' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:18:24.052 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:23 smithi031 ceph-mon[16336]: osdmap e274: 12 total, 12 up, 9 in 2024-08-19T14:18:24.052 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:23 smithi031 ceph-mon[16336]: 3.f scrub ok 2024-08-19T14:18:24.052 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:23 smithi031 ceph-mon[16336]: 1.f scrub starts 2024-08-19T14:18:24.052 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:23 smithi031 ceph-mon[16336]: 1.f scrub ok 2024-08-19T14:18:24.052 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:23 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1259956446' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]: dispatch 2024-08-19T14:18:24.076 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-08-19T14:18:24.142 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:18:24.168 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:24.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:24.169 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:18:24.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:23 smithi032 ceph-mon[20004]: 3.23 scrub starts 2024-08-19T14:18:24.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:23 smithi032 ceph-mon[20004]: 3.23 scrub ok 2024-08-19T14:18:24.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:23 smithi032 ceph-mon[20004]: 1.13 scrub starts 2024-08-19T14:18:24.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:23 smithi032 ceph-mon[20004]: 1.13 scrub ok 2024-08-19T14:18:24.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:23 smithi032 ceph-mon[20004]: 3.f scrub starts 2024-08-19T14:18:24.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:23 smithi032 ceph-mon[20004]: 1.6 scrub starts 2024-08-19T14:18:24.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:23 smithi032 ceph-mon[20004]: 1.6 scrub ok 2024-08-19T14:18:24.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:23 smithi032 ceph-mon[20004]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:18:24.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:23 smithi032 ceph-mon[20004]: Cluster is now healthy 2024-08-19T14:18:24.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:23 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2305990264' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:18:24.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:23 smithi032 ceph-mon[20004]: osdmap e274: 12 total, 12 up, 9 in 2024-08-19T14:18:24.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:23 smithi032 ceph-mon[20004]: 3.f scrub ok 2024-08-19T14:18:24.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:23 smithi032 ceph-mon[20004]: 1.f scrub starts 2024-08-19T14:18:24.259 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:23 smithi032 ceph-mon[20004]: 1.f scrub ok 2024-08-19T14:18:24.259 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:23 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1259956446' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]: dispatch 2024-08-19T14:18:24.269 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:18:24.294 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:24.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:24.295 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:18:24.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:23 smithi045 ceph-mon[20143]: 3.23 scrub starts 2024-08-19T14:18:24.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:23 smithi045 ceph-mon[20143]: 3.23 scrub ok 2024-08-19T14:18:24.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:23 smithi045 ceph-mon[20143]: 1.13 scrub starts 2024-08-19T14:18:24.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:23 smithi045 ceph-mon[20143]: 1.13 scrub ok 2024-08-19T14:18:24.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:23 smithi045 ceph-mon[20143]: 3.f scrub starts 2024-08-19T14:18:24.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:23 smithi045 ceph-mon[20143]: 1.6 scrub starts 2024-08-19T14:18:24.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:23 smithi045 ceph-mon[20143]: 1.6 scrub ok 2024-08-19T14:18:24.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:23 smithi045 ceph-mon[20143]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:18:24.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:23 smithi045 ceph-mon[20143]: Cluster is now healthy 2024-08-19T14:18:24.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:23 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2305990264' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:18:24.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:23 smithi045 ceph-mon[20143]: osdmap e274: 12 total, 12 up, 9 in 2024-08-19T14:18:24.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:23 smithi045 ceph-mon[20143]: 3.f scrub ok 2024-08-19T14:18:24.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:23 smithi045 ceph-mon[20143]: 1.f scrub starts 2024-08-19T14:18:24.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:23 smithi045 ceph-mon[20143]: 1.f scrub ok 2024-08-19T14:18:24.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:23 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1259956446' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]: dispatch 2024-08-19T14:18:24.318 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 82 16 118893 118877 90.5965 0 - 0.0104656 2024-08-19T14:18:24.395 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:18:24.421 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:24.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:24.422 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:18:24.523 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:18:24.549 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:24.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:24.550 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:18:24.651 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:18:24.673 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:24.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:24.674 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:18:24.779 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:18:24.805 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:24.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:24.806 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:18:24.907 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:18:24.929 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:24.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:24.930 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:18:25.031 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:18:25.056 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:25.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:25.057 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:18:25.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:24 smithi031 ceph-mon[16336]: 3.12 scrub ok 2024-08-19T14:18:25.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:24 smithi031 ceph-mon[16336]: 1.a scrub starts 2024-08-19T14:18:25.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:24 smithi031 ceph-mon[16336]: 1.a scrub ok 2024-08-19T14:18:25.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:24 smithi031 ceph-mon[16336]: pgmap v629: 57 pgs: 2 active+clean+scrubbing, 55 active+clean; 7.2 GiB data, 25 GiB used, 780 GiB / 805 GiB avail 2024-08-19T14:18:25.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:24 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1259956446' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]': finished 2024-08-19T14:18:25.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:24 smithi031 ceph-mon[16336]: osdmap e275: 12 total, 12 up, 9 in 2024-08-19T14:18:25.158 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:18:25.159 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:24 smithi032 ceph-mon[20004]: 3.12 scrub ok 2024-08-19T14:18:25.159 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:24 smithi032 ceph-mon[20004]: 1.a scrub starts 2024-08-19T14:18:25.159 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:24 smithi032 ceph-mon[20004]: 1.a scrub ok 2024-08-19T14:18:25.159 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:24 smithi032 ceph-mon[20004]: pgmap v629: 57 pgs: 2 active+clean+scrubbing, 55 active+clean; 7.2 GiB data, 25 GiB used, 780 GiB / 805 GiB avail 2024-08-19T14:18:25.159 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:24 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1259956446' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]': finished 2024-08-19T14:18:25.159 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:24 smithi032 ceph-mon[20004]: osdmap e275: 12 total, 12 up, 9 in 2024-08-19T14:18:25.168 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:18:25.169 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:18:25.186 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:18:25.187 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:18:25.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:25.187 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:25.188 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:18:25.196 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:24 smithi045 ceph-mon[20143]: 3.12 scrub ok 2024-08-19T14:18:25.197 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:24 smithi045 ceph-mon[20143]: 1.a scrub starts 2024-08-19T14:18:25.197 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:24 smithi045 ceph-mon[20143]: 1.a scrub ok 2024-08-19T14:18:25.197 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:24 smithi045 ceph-mon[20143]: pgmap v629: 57 pgs: 2 active+clean+scrubbing, 55 active+clean; 7.2 GiB data, 25 GiB used, 780 GiB / 805 GiB avail 2024-08-19T14:18:25.197 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:24 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1259956446' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]': finished 2024-08-19T14:18:25.197 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:24 smithi045 ceph-mon[20143]: osdmap e275: 12 total, 12 up, 9 in 2024-08-19T14:18:25.202 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:18:25.202 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:18:25.218 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:18:25.219 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:18:25.236 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:18:25.236 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:18:25.253 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:18:25.253 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:18:25.255 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:18:25.272 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:18:25.272 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:18:25.288 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:18:25.291 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:18:25.292 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:18:25.309 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:25.309 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:18:25.310 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:18:25.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:25.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:18:25.318 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 83 16 118893 118877 89.505 0 - 0.0104656 2024-08-19T14:18:25.326 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:18:25.327 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:18:25.343 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:18:25.344 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:18:25.361 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:18:25.361 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:18:25.412 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:18:25.433 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:25.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:25.434 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:18:25.485 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-19T14:18:25.536 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:18:25.561 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:25.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:25.562 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:18:25.663 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:18:25.688 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:25.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:25.689 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:18:25.790 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:18:25.816 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:25.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:25.817 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:18:25.918 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:18:25.953 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:25.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:25.955 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:18:26.056 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:18:26.086 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:26.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:26.087 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:18:26.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:25 smithi031 ceph-mon[16336]: 1.2 scrub starts 2024-08-19T14:18:26.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:25 smithi031 ceph-mon[16336]: 1.2 scrub ok 2024-08-19T14:18:26.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:25 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1092877242' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:18:26.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:25 smithi031 ceph-mon[16336]: 1.8 scrub starts 2024-08-19T14:18:26.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:25 smithi031 ceph-mon[16336]: 1.8 scrub ok 2024-08-19T14:18:26.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:25 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-08-19T14:18:26.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:25 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-08-19T14:18:26.181 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:18:26.182 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [ 2024-08-19T14:18:26.182 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:26.182 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25057.0:119498 3.1e 3:7aaa9d1a:::benchmark_data_smithi045_59973_object119497:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e276)", 2024-08-19T14:18:26.182 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:18:26.174747+0000", 2024-08-19T14:18:26.182 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.0055043499999999999, 2024-08-19T14:18:26.182 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0055282020000000003, 2024-08-19T14:18:26.182 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:18:26.182 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:18:26.182 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "waiting for sub ops", 2024-08-19T14:18:26.182 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:18:26.182 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25057", 2024-08-19T14:18:26.182 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:18:26.183 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 119498 2024-08-19T14:18:26.183 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:26.183 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:18:26.183 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:26.183 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:18:26.183 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:18:26.174747+0000", 2024-08-19T14:18:26.183 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:18:26.183 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:26.183 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:26.183 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:18:26.183 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:18:26.174747+0000", 2024-08-19T14:18:26.183 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:18:26.183 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:26.183 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:26.184 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:18:26.184 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:18:26.174748+0000", 2024-08-19T14:18:26.184 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2109999999999999e-06 2024-08-19T14:18:26.184 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:26.184 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:26.184 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:18:26.184 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:18:26.174805+0000", 2024-08-19T14:18:26.184 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.63e-05 2024-08-19T14:18:26.184 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:26.184 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:26.184 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:18:26.184 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:18:26.174821+0000", 2024-08-19T14:18:26.184 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6418000000000001e-05 2024-08-19T14:18:26.185 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:26.185 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:26.185 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:18:26.185 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:18:26.174828+0000", 2024-08-19T14:18:26.185 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.092e-06 2024-08-19T14:18:26.185 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:26.185 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:26.185 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:18:26.185 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:18:26.176701+0000", 2024-08-19T14:18:26.185 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0018725079999999999 2024-08-19T14:18:26.185 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:26.185 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:26.185 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:18:26.186 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:18:26.176821+0000", 2024-08-19T14:18:26.186 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012068799999999999 2024-08-19T14:18:26.186 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:26.186 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:26.186 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 8,9", 2024-08-19T14:18:26.186 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:18:26.176957+0000", 2024-08-19T14:18:26.186 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000135492 2024-08-19T14:18:26.186 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:26.186 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:26.187 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:18:26.187 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:18:26.178953+0000", 2024-08-19T14:18:26.187 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0019959679999999999 2024-08-19T14:18:26.187 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:26.187 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:18:26.187 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:26.187 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:26.187 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:26.187 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25057.0:119506 3.1e 3:7e78767a:::benchmark_data_smithi045_59973_object119505:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e276)", 2024-08-19T14:18:26.187 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:18:26.178899+0000", 2024-08-19T14:18:26.187 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.001352998, 2024-08-19T14:18:26.187 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.001468069, 2024-08-19T14:18:26.187 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:18:26.187 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:18:26.188 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "waiting for sub ops", 2024-08-19T14:18:26.188 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:18:26.188 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25057", 2024-08-19T14:18:26.188 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:18:26.188 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 119506 2024-08-19T14:18:26.188 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:26.188 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:18:26.188 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:26.188 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:18:26.188 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:18:26.178899+0000", 2024-08-19T14:18:26.188 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:18:26.188 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:26.188 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:26.189 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:18:26.189 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:18:26.178899+0000", 2024-08-19T14:18:26.189 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:18:26.189 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:26.189 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:26.189 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:18:26.189 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:18:26.178900+0000", 2024-08-19T14:18:26.189 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.457e-06 2024-08-19T14:18:26.189 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:26.189 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:26.190 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:18:26.190 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:18:26.178954+0000", 2024-08-19T14:18:26.190 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.4156999999999997e-05 2024-08-19T14:18:26.190 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:26.190 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:26.190 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:18:26.190 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:18:26.178965+0000", 2024-08-19T14:18:26.190 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1124000000000001e-05 2024-08-19T14:18:26.190 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:26.190 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:26.190 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:18:26.190 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:18:26.178973+0000", 2024-08-19T14:18:26.191 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.6389999999999992e-06 2024-08-19T14:18:26.191 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:26.191 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:26.191 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:18:26.191 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:18:26.179045+0000", 2024-08-19T14:18:26.191 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.2108999999999997e-05 2024-08-19T14:18:26.191 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:26.191 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:26.191 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:18:26.191 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:18:26.179185+0000", 2024-08-19T14:18:26.191 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000139405 2024-08-19T14:18:26.191 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:26.191 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:26.192 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 8,9", 2024-08-19T14:18:26.192 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:18:26.179326+0000", 2024-08-19T14:18:26.192 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00014102799999999999 2024-08-19T14:18:26.192 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:26.192 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:18:26.192 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:26.192 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:26.192 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:26.192 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop_reply(client.25057.0:119498 3.1e e276/265)", 2024-08-19T14:18:26.192 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:18:26.179650+0000", 2024-08-19T14:18:26.192 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.00060205399999999998, 2024-08-19T14:18:26.192 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00078794200000000003, 2024-08-19T14:18:26.192 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:18:26.192 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:18:26.193 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "queued for pg", 2024-08-19T14:18:26.193 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:18:26.193 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:26.193 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:18:26.193 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:18:26.179650+0000", 2024-08-19T14:18:26.193 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:18:26.193 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:26.193 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:26.193 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:18:26.193 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:18:26.179650+0000", 2024-08-19T14:18:26.193 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:18:26.193 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:26.193 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:26.194 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:18:26.194 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:18:26.179650+0000", 2024-08-19T14:18:26.194 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.3200000000000005e-07 2024-08-19T14:18:26.194 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:26.194 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:26.194 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:18:26.194 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:18:26.179654+0000", 2024-08-19T14:18:26.194 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.4699999999999998e-06 2024-08-19T14:18:26.194 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:26.194 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:26.194 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:18:26.194 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:18:26.179667+0000", 2024-08-19T14:18:26.194 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.34e-05 2024-08-19T14:18:26.195 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:26.195 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:26.195 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:18:26.195 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:18:26.179673+0000", 2024-08-19T14:18:26.195 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.0920000000000003e-06 2024-08-19T14:18:26.195 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:26.195 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:18:26.195 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:26.195 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:26.195 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:26.195 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25057.0:119508 3.7 3:e4fa1fa3:::benchmark_data_smithi045_59973_object119507:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e276)", 2024-08-19T14:18:26.195 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:18:26.179626+0000", 2024-08-19T14:18:26.195 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.00062548300000000003, 2024-08-19T14:18:26.196 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00086394100000000001, 2024-08-19T14:18:26.196 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:18:26.196 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:18:26.196 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "waiting for sub ops", 2024-08-19T14:18:26.196 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:18:26.196 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25057", 2024-08-19T14:18:26.196 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:18:26.196 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 119508 2024-08-19T14:18:26.196 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:26.196 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:18:26.196 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:26.196 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:18:26.196 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:18:26.179626+0000", 2024-08-19T14:18:26.196 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:18:26.197 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:26.197 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:26.197 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:18:26.197 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:18:26.179626+0000", 2024-08-19T14:18:26.197 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:18:26.197 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:26.197 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:26.197 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:18:26.197 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:18:26.179627+0000", 2024-08-19T14:18:26.197 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.2200000000000002e-07 2024-08-19T14:18:26.197 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:26.197 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:26.197 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:18:26.197 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:18:26.179713+0000", 2024-08-19T14:18:26.198 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.6038000000000005e-05 2024-08-19T14:18:26.198 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:26.198 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:26.198 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:18:26.198 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:18:26.179724+0000", 2024-08-19T14:18:26.198 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0437e-05 2024-08-19T14:18:26.198 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:26.198 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:26.198 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:18:26.198 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:18:26.179730+0000", 2024-08-19T14:18:26.198 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.0260000000000004e-06 2024-08-19T14:18:26.198 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:26.198 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:26.199 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:18:26.199 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:18:26.179782+0000", 2024-08-19T14:18:26.199 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.2493e-05 2024-08-19T14:18:26.199 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:26.199 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:26.199 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:18:26.199 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:18:26.179920+0000", 2024-08-19T14:18:26.199 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013834399999999999 2024-08-19T14:18:26.199 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:26.199 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:26.199 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 7,11", 2024-08-19T14:18:26.199 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:18:26.180055+0000", 2024-08-19T14:18:26.199 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013425700000000001 2024-08-19T14:18:26.199 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:26.200 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:18:26.200 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:26.200 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:26.200 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:26.200 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25057.0:119509 3.b 3:d5f54209:::benchmark_data_smithi045_59973_object119508:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e276)", 2024-08-19T14:18:26.200 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:18:26.180174+0000", 2024-08-19T14:18:26.200 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 7.7280000000000005e-05, 2024-08-19T14:18:26.200 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000394917, 2024-08-19T14:18:26.200 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:18:26.200 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:18:26.200 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-08-19T14:18:26.200 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:18:26.200 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25057", 2024-08-19T14:18:26.201 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:18:26.201 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 119509 2024-08-19T14:18:26.201 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:26.201 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:18:26.201 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:26.201 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:18:26.201 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:18:26.180174+0000", 2024-08-19T14:18:26.201 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:18:26.201 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:26.201 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:26.201 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:18:26.201 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:18:26.180174+0000", 2024-08-19T14:18:26.201 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:18:26.201 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:26.202 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:26.202 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:18:26.202 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:18:26.180175+0000", 2024-08-19T14:18:26.202 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.9099999999999991e-07 2024-08-19T14:18:26.202 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:26.202 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:26.202 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:18:26.202 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:18:26.180245+0000", 2024-08-19T14:18:26.202 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.9825000000000003e-05 2024-08-19T14:18:26.202 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:26.202 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:26.202 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:18:26.202 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:18:26.180260+0000", 2024-08-19T14:18:26.203 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4440999999999999e-05 2024-08-19T14:18:26.203 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:26.203 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:26.203 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:18:26.203 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:18:26.180267+0000", 2024-08-19T14:18:26.203 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.469e-06 2024-08-19T14:18:26.203 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:26.203 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:26.203 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:18:26.203 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:18:26.180353+0000", 2024-08-19T14:18:26.203 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.6108999999999998e-05 2024-08-19T14:18:26.203 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:26.203 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:26.204 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:18:26.204 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:18:26.180523+0000", 2024-08-19T14:18:26.204 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00017030300000000001 2024-08-19T14:18:26.204 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:26.204 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:18:26.204 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:26.204 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:26.204 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-08-19T14:18:26.204 INFO:teuthology.orchestra.run.smithi031.stdout: "num_ops": 5 2024-08-19T14:18:26.204 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:18:26.204 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:18:26.205 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:25 smithi032 ceph-mon[20004]: 1.2 scrub starts 2024-08-19T14:18:26.205 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:25 smithi032 ceph-mon[20004]: 1.2 scrub ok 2024-08-19T14:18:26.205 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:25 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1092877242' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:18:26.205 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:25 smithi032 ceph-mon[20004]: 1.8 scrub starts 2024-08-19T14:18:26.205 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:25 smithi032 ceph-mon[20004]: 1.8 scrub ok 2024-08-19T14:18:26.206 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:25 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-08-19T14:18:26.206 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:25 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-08-19T14:18:26.237 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:26.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:26.238 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:18:26.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:25 smithi045 ceph-mon[20143]: 1.2 scrub starts 2024-08-19T14:18:26.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:25 smithi045 ceph-mon[20143]: 1.2 scrub ok 2024-08-19T14:18:26.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:25 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1092877242' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:18:26.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:25 smithi045 ceph-mon[20143]: 1.8 scrub starts 2024-08-19T14:18:26.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:25 smithi045 ceph-mon[20143]: 1.8 scrub ok 2024-08-19T14:18:26.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:25 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-08-19T14:18:26.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:25 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-08-19T14:18:26.318 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 84 15 119801 119786 89.1157 9.46875 0.00309416 0.0112173 2024-08-19T14:18:26.340 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:18:26.368 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:26.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:26.369 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:18:26.470 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:18:26.491 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:26.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:26.492 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:18:26.593 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:18:26.625 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:26.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:26.626 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:18:26.727 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:18:26.748 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:26.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:26.749 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:18:26.851 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:18:26.895 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:26.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:26.896 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:18:26.997 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:18:27.031 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:27.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:27.032 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:18:27.133 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:18:27.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:27.168 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:27.168 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:18:27.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:26 smithi032 ceph-mon[20004]: 1.11 scrub starts 2024-08-19T14:18:27.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:26 smithi032 ceph-mon[20004]: 1.11 scrub ok 2024-08-19T14:18:27.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:26 smithi032 ceph-mon[20004]: pgmap v631: 57 pgs: 1 active+clean+scrubbing, 56 active+clean; 7.3 GiB data, 25 GiB used, 780 GiB / 805 GiB avail 2024-08-19T14:18:27.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:26 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]': finished 2024-08-19T14:18:27.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:26 smithi032 ceph-mon[20004]: osdmap e276: 12 total, 12 up, 9 in 2024-08-19T14:18:27.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:26 smithi032 ceph-mon[20004]: 1.0 scrub starts 2024-08-19T14:18:27.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:26 smithi032 ceph-mon[20004]: 1.0 scrub ok 2024-08-19T14:18:27.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:26 smithi032 ceph-mon[20004]: 3.d scrub starts 2024-08-19T14:18:27.269 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:18:27.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:26 smithi045 ceph-mon[20143]: 1.11 scrub starts 2024-08-19T14:18:27.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:26 smithi045 ceph-mon[20143]: 1.11 scrub ok 2024-08-19T14:18:27.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:26 smithi045 ceph-mon[20143]: pgmap v631: 57 pgs: 1 active+clean+scrubbing, 56 active+clean; 7.3 GiB data, 25 GiB used, 780 GiB / 805 GiB avail 2024-08-19T14:18:27.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:26 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]': finished 2024-08-19T14:18:27.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:26 smithi045 ceph-mon[20143]: osdmap e276: 12 total, 12 up, 9 in 2024-08-19T14:18:27.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:26 smithi045 ceph-mon[20143]: 1.0 scrub starts 2024-08-19T14:18:27.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:26 smithi045 ceph-mon[20143]: 1.0 scrub ok 2024-08-19T14:18:27.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:26 smithi045 ceph-mon[20143]: 3.d scrub starts 2024-08-19T14:18:27.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:27.298 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:27.299 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:18:27.318 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 85 16 121503 121487 89.3179 106.312 0.00720696 0.011192 2024-08-19T14:18:27.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:26 smithi031 ceph-mon[16336]: 1.11 scrub starts 2024-08-19T14:18:27.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:26 smithi031 ceph-mon[16336]: 1.11 scrub ok 2024-08-19T14:18:27.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:26 smithi031 ceph-mon[16336]: pgmap v631: 57 pgs: 1 active+clean+scrubbing, 56 active+clean; 7.3 GiB data, 25 GiB used, 780 GiB / 805 GiB avail 2024-08-19T14:18:27.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:26 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]': finished 2024-08-19T14:18:27.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:26 smithi031 ceph-mon[16336]: osdmap e276: 12 total, 12 up, 9 in 2024-08-19T14:18:27.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:26 smithi031 ceph-mon[16336]: 1.0 scrub starts 2024-08-19T14:18:27.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:26 smithi031 ceph-mon[16336]: 1.0 scrub ok 2024-08-19T14:18:27.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:26 smithi031 ceph-mon[16336]: 3.d scrub starts 2024-08-19T14:18:27.400 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:18:27.435 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:27.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:27.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:18:27.540 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:18:27.573 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:27.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:27.574 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:18:27.674 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:18:27.710 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:27.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:27.711 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:18:27.812 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:18:27.848 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:27.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:27.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:18:27.950 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:18:27.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:27.986 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:27.987 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:18:28.088 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:18:28.122 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:28.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:28.123 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:18:28.224 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:18:28.254 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:27 smithi031 ceph-mon[16336]: 3.d scrub ok 2024-08-19T14:18:28.254 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:27 smithi031 ceph-mon[16336]: osdmap e277: 12 total, 12 up, 9 in 2024-08-19T14:18:28.255 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:27 smithi031 ceph-mon[16336]: 1.7 scrub starts 2024-08-19T14:18:28.255 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:27 smithi031 ceph-mon[16336]: 1.7 scrub ok 2024-08-19T14:18:28.255 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:27 smithi031 ceph-mon[16336]: 1.1c scrub starts 2024-08-19T14:18:28.255 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:27 smithi031 ceph-mon[16336]: 1.1c scrub ok 2024-08-19T14:18:28.255 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:27 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-08-19T14:18:28.255 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:27 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-08-19T14:18:28.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:28.256 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:28.256 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:18:28.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:27 smithi032 ceph-mon[20004]: 3.d scrub ok 2024-08-19T14:18:28.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:27 smithi032 ceph-mon[20004]: osdmap e277: 12 total, 12 up, 9 in 2024-08-19T14:18:28.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:27 smithi032 ceph-mon[20004]: 1.7 scrub starts 2024-08-19T14:18:28.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:27 smithi032 ceph-mon[20004]: 1.7 scrub ok 2024-08-19T14:18:28.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:27 smithi032 ceph-mon[20004]: 1.1c scrub starts 2024-08-19T14:18:28.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:27 smithi032 ceph-mon[20004]: 1.1c scrub ok 2024-08-19T14:18:28.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:27 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-08-19T14:18:28.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:27 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-08-19T14:18:28.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:27 smithi045 ceph-mon[20143]: 3.d scrub ok 2024-08-19T14:18:28.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:27 smithi045 ceph-mon[20143]: osdmap e277: 12 total, 12 up, 9 in 2024-08-19T14:18:28.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:27 smithi045 ceph-mon[20143]: 1.7 scrub starts 2024-08-19T14:18:28.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:27 smithi045 ceph-mon[20143]: 1.7 scrub ok 2024-08-19T14:18:28.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:27 smithi045 ceph-mon[20143]: 1.1c scrub starts 2024-08-19T14:18:28.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:27 smithi045 ceph-mon[20143]: 1.1c scrub ok 2024-08-19T14:18:28.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:27 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-08-19T14:18:28.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:27 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-08-19T14:18:28.319 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 86 16 123644 123628 89.8351 133.812 0.00247784 0.0111177 2024-08-19T14:18:28.357 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:18:28.378 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:28.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:28.379 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:18:28.480 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:18:28.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:28.516 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:28.516 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:18:28.617 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:18:28.655 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:28.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:28.656 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:18:28.757 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:18:28.790 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:28.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:28.792 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:18:28.893 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:18:28.927 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:28.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:28.928 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:18:29.029 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:18:29.060 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:29.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:29.061 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:18:29.162 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:18:29.164 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:28 smithi031 ceph-mon[16336]: 1.b scrub starts 2024-08-19T14:18:29.164 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:28 smithi031 ceph-mon[16336]: 1.b scrub ok 2024-08-19T14:18:29.165 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:28 smithi031 ceph-mon[16336]: 1.d scrub starts 2024-08-19T14:18:29.165 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:28 smithi031 ceph-mon[16336]: 1.d scrub ok 2024-08-19T14:18:29.165 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:28 smithi031 ceph-mon[16336]: pgmap v634: 67 pgs: 2 peering, 10 unknown, 1 active+clean+scrubbing, 54 active+clean; 7.3 GiB data, 25 GiB used, 780 GiB / 805 GiB avail 2024-08-19T14:18:29.165 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:28 smithi031 ceph-mon[16336]: 1.6 deep-scrub starts 2024-08-19T14:18:29.165 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:28 smithi031 ceph-mon[16336]: 1.6 deep-scrub ok 2024-08-19T14:18:29.165 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:28 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]': finished 2024-08-19T14:18:29.165 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:28 smithi031 ceph-mon[16336]: osdmap e278: 12 total, 12 up, 9 in 2024-08-19T14:18:29.165 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:28 smithi031 ceph-mon[16336]: 1.e scrub starts 2024-08-19T14:18:29.165 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:28 smithi031 ceph-mon[16336]: 1.e scrub ok 2024-08-19T14:18:29.165 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:28 smithi031 ceph-mon[16336]: 1.8 scrub starts 2024-08-19T14:18:29.165 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:28 smithi031 ceph-mon[16336]: 1.8 scrub ok 2024-08-19T14:18:29.165 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:28 smithi031 ceph-mon[16336]: osdmap e279: 12 total, 12 up, 9 in 2024-08-19T14:18:29.201 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:29.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:29.202 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:18:29.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:28 smithi032 ceph-mon[20004]: 1.b scrub starts 2024-08-19T14:18:29.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:28 smithi032 ceph-mon[20004]: 1.b scrub ok 2024-08-19T14:18:29.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:28 smithi032 ceph-mon[20004]: 1.d scrub starts 2024-08-19T14:18:29.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:28 smithi032 ceph-mon[20004]: 1.d scrub ok 2024-08-19T14:18:29.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:28 smithi032 ceph-mon[20004]: pgmap v634: 67 pgs: 2 peering, 10 unknown, 1 active+clean+scrubbing, 54 active+clean; 7.3 GiB data, 25 GiB used, 780 GiB / 805 GiB avail 2024-08-19T14:18:29.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:28 smithi032 ceph-mon[20004]: 1.6 deep-scrub starts 2024-08-19T14:18:29.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:28 smithi032 ceph-mon[20004]: 1.6 deep-scrub ok 2024-08-19T14:18:29.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:28 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]': finished 2024-08-19T14:18:29.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:28 smithi032 ceph-mon[20004]: osdmap e278: 12 total, 12 up, 9 in 2024-08-19T14:18:29.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:28 smithi032 ceph-mon[20004]: 1.e scrub starts 2024-08-19T14:18:29.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:28 smithi032 ceph-mon[20004]: 1.e scrub ok 2024-08-19T14:18:29.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:28 smithi032 ceph-mon[20004]: 1.8 scrub starts 2024-08-19T14:18:29.259 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:28 smithi032 ceph-mon[20004]: 1.8 scrub ok 2024-08-19T14:18:29.259 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:28 smithi032 ceph-mon[20004]: osdmap e279: 12 total, 12 up, 9 in 2024-08-19T14:18:29.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:28 smithi045 ceph-mon[20143]: 1.b scrub starts 2024-08-19T14:18:29.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:28 smithi045 ceph-mon[20143]: 1.b scrub ok 2024-08-19T14:18:29.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:28 smithi045 ceph-mon[20143]: 1.d scrub starts 2024-08-19T14:18:29.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:28 smithi045 ceph-mon[20143]: 1.d scrub ok 2024-08-19T14:18:29.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:28 smithi045 ceph-mon[20143]: pgmap v634: 67 pgs: 2 peering, 10 unknown, 1 active+clean+scrubbing, 54 active+clean; 7.3 GiB data, 25 GiB used, 780 GiB / 805 GiB avail 2024-08-19T14:18:29.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:28 smithi045 ceph-mon[20143]: 1.6 deep-scrub starts 2024-08-19T14:18:29.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:28 smithi045 ceph-mon[20143]: 1.6 deep-scrub ok 2024-08-19T14:18:29.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:28 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]': finished 2024-08-19T14:18:29.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:28 smithi045 ceph-mon[20143]: osdmap e278: 12 total, 12 up, 9 in 2024-08-19T14:18:29.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:28 smithi045 ceph-mon[20143]: 1.e scrub starts 2024-08-19T14:18:29.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:28 smithi045 ceph-mon[20143]: 1.e scrub ok 2024-08-19T14:18:29.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:28 smithi045 ceph-mon[20143]: 1.8 scrub starts 2024-08-19T14:18:29.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:28 smithi045 ceph-mon[20143]: 1.8 scrub ok 2024-08-19T14:18:29.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:28 smithi045 ceph-mon[20143]: osdmap e279: 12 total, 12 up, 9 in 2024-08-19T14:18:29.303 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:18:29.318 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 87 16 125563 125547 90.181 119.938 0.00254126 0.0110748 2024-08-19T14:18:29.324 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:29.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:29.325 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:18:29.426 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:18:29.458 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:29.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:29.459 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:18:29.510 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-08-19T14:18:29.540 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:18:29.559 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:18:29.591 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:29.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:29.592 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:18:29.693 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:18:29.729 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:29.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:29.730 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:18:29.831 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:18:29.864 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:29.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:29.865 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:18:29.967 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:18:30.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:30.000 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:30.000 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:18:30.101 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:18:30.102 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:29 smithi031 ceph-mon[16336]: 1.c scrub starts 2024-08-19T14:18:30.102 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:29 smithi031 ceph-mon[16336]: 1.c scrub ok 2024-08-19T14:18:30.102 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:29 smithi031 ceph-mon[16336]: 1.5 scrub starts 2024-08-19T14:18:30.103 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:29 smithi031 ceph-mon[16336]: 1.5 scrub ok 2024-08-19T14:18:30.103 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:29 smithi031 ceph-mon[16336]: 1.9 scrub starts 2024-08-19T14:18:30.103 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:29 smithi031 ceph-mon[16336]: 1.9 scrub ok 2024-08-19T14:18:30.103 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:29 smithi031 ceph-mon[16336]: 1.a deep-scrub starts 2024-08-19T14:18:30.103 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:29 smithi031 ceph-mon[16336]: 1.a deep-scrub ok 2024-08-19T14:18:30.103 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:29 smithi031 ceph-mon[16336]: osdmap e280: 12 total, 12 up, 9 in 2024-08-19T14:18:30.137 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:30.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:30.138 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:18:30.177 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:18:30.239 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:18:30.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:29 smithi032 ceph-mon[20004]: 1.c scrub starts 2024-08-19T14:18:30.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:29 smithi032 ceph-mon[20004]: 1.c scrub ok 2024-08-19T14:18:30.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:29 smithi032 ceph-mon[20004]: 1.5 scrub starts 2024-08-19T14:18:30.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:29 smithi032 ceph-mon[20004]: 1.5 scrub ok 2024-08-19T14:18:30.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:29 smithi032 ceph-mon[20004]: 1.9 scrub starts 2024-08-19T14:18:30.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:29 smithi032 ceph-mon[20004]: 1.9 scrub ok 2024-08-19T14:18:30.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:29 smithi032 ceph-mon[20004]: 1.a deep-scrub starts 2024-08-19T14:18:30.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:29 smithi032 ceph-mon[20004]: 1.a deep-scrub ok 2024-08-19T14:18:30.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:29 smithi032 ceph-mon[20004]: osdmap e280: 12 total, 12 up, 9 in 2024-08-19T14:18:30.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:30.270 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:30.270 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:18:30.284 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 7, 8, 9, 11] out_osds: [10, 6, 2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-08-19T14:18:30.284 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-19T14:18:30.284 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-08-19T14:18:30.285 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd dump -f json-pretty 2024-08-19T14:18:30.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:29 smithi045 ceph-mon[20143]: 1.c scrub starts 2024-08-19T14:18:30.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:29 smithi045 ceph-mon[20143]: 1.c scrub ok 2024-08-19T14:18:30.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:29 smithi045 ceph-mon[20143]: 1.5 scrub starts 2024-08-19T14:18:30.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:29 smithi045 ceph-mon[20143]: 1.5 scrub ok 2024-08-19T14:18:30.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:29 smithi045 ceph-mon[20143]: 1.9 scrub starts 2024-08-19T14:18:30.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:29 smithi045 ceph-mon[20143]: 1.9 scrub ok 2024-08-19T14:18:30.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:29 smithi045 ceph-mon[20143]: 1.a deep-scrub starts 2024-08-19T14:18:30.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:29 smithi045 ceph-mon[20143]: 1.a deep-scrub ok 2024-08-19T14:18:30.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:29 smithi045 ceph-mon[20143]: osdmap e280: 12 total, 12 up, 9 in 2024-08-19T14:18:30.319 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 88 16 127585 127569 90.5921 126.375 0.00641513 0.0110345 2024-08-19T14:18:30.371 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:18:30.407 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:30.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:30.408 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:18:30.509 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:18:30.530 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:30.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:30.531 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:18:30.633 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:18:30.666 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:30.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:30.667 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:18:30.768 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:18:30.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:30.794 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:30.794 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:18:30.894 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is set 2024-08-19T14:18:30.896 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:18:30.932 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:30.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:30.933 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:18:31.013 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:18:31.034 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:18:31.084 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:31.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:31.085 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:18:31.186 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:18:31.212 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:30 smithi031 ceph-mon[16336]: pgmap v637: 67 pgs: 2 peering, 1 remapped+peering, 9 unknown, 55 active+clean; 7.4 GiB data, 25 GiB used, 779 GiB / 805 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-08-19T14:18:31.212 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:30 smithi031 ceph-mon[16336]: Health check failed: Reduced data availability: 3 pgs inactive, 1 pg peering (PG_AVAILABILITY) 2024-08-19T14:18:31.212 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:30 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1799237929' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:18:31.213 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:30 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:18:31.213 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:30 smithi031 ceph-mon[16336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:18:31.213 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:30 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:18:31.215 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:31.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:31.216 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:18:31.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:30 smithi032 ceph-mon[20004]: pgmap v637: 67 pgs: 2 peering, 1 remapped+peering, 9 unknown, 55 active+clean; 7.4 GiB data, 25 GiB used, 779 GiB / 805 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-08-19T14:18:31.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:30 smithi032 ceph-mon[20004]: Health check failed: Reduced data availability: 3 pgs inactive, 1 pg peering (PG_AVAILABILITY) 2024-08-19T14:18:31.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:30 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1799237929' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:18:31.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:30 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:18:31.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:30 smithi032 ceph-mon[20004]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:18:31.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:30 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:18:31.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:30 smithi045 ceph-mon[20143]: pgmap v637: 67 pgs: 2 peering, 1 remapped+peering, 9 unknown, 55 active+clean; 7.4 GiB data, 25 GiB used, 779 GiB / 805 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-08-19T14:18:31.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:30 smithi045 ceph-mon[20143]: Health check failed: Reduced data availability: 3 pgs inactive, 1 pg peering (PG_AVAILABILITY) 2024-08-19T14:18:31.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:30 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1799237929' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:18:31.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:30 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:18:31.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:30 smithi045 ceph-mon[20143]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:18:31.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:30 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:18:31.317 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:18:31.319 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 89 16 129431 129415 90.8705 115.375 0.00542782 0.0110006 2024-08-19T14:18:31.352 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:31.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:31.353 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:18:31.454 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:18:31.477 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:31.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:31.478 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:18:31.579 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:18:31.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:31.614 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:31.614 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:18:31.715 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:18:31.736 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:31.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:31.737 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:18:31.838 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:18:31.869 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:31.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:31.870 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:18:31.971 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:18:32.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:32.000 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:32.000 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:18:32.101 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:18:32.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:32.139 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:32.139 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:18:32.240 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:18:32.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:31 smithi032 ceph-mon[20004]: osdmap e281: 12 total, 12 up, 9 in 2024-08-19T14:18:32.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:31 smithi032 ceph-mon[20004]: osdmap e282: 12 total, 12 up, 9 in 2024-08-19T14:18:32.270 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:32.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:32.272 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:18:32.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:31 smithi045 ceph-mon[20143]: osdmap e281: 12 total, 12 up, 9 in 2024-08-19T14:18:32.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:31 smithi045 ceph-mon[20143]: osdmap e282: 12 total, 12 up, 9 in 2024-08-19T14:18:32.319 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Total time run: 90.0057 2024-08-19T14:18:32.319 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Total writes made: 131176 2024-08-19T14:18:32.319 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Write size: 65536 2024-08-19T14:18:32.319 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Object size: 65536 2024-08-19T14:18:32.320 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Bandwidth (MB/sec): 91.0887 2024-08-19T14:18:32.320 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Stddev Bandwidth: 60.9183 2024-08-19T14:18:32.320 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Max bandwidth (MB/sec): 271.688 2024-08-19T14:18:32.320 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Min bandwidth (MB/sec): 0 2024-08-19T14:18:32.320 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Average IOPS: 1457 2024-08-19T14:18:32.320 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Stddev IOPS: 974.708 2024-08-19T14:18:32.320 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Max IOPS: 4347 2024-08-19T14:18:32.320 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Min IOPS: 0 2024-08-19T14:18:32.321 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Average Latency(s): 0.0109746 2024-08-19T14:18:32.321 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Stddev Latency(s): 0.0803483 2024-08-19T14:18:32.321 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Max latency(s): 5.89072 2024-08-19T14:18:32.321 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Min latency(s): 0.00117714 2024-08-19T14:18:32.322 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Cleaning up (deleting benchmark objects) 2024-08-19T14:18:32.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:31 smithi031 ceph-mon[16336]: osdmap e281: 12 total, 12 up, 9 in 2024-08-19T14:18:32.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:31 smithi031 ceph-mon[16336]: osdmap e282: 12 total, 12 up, 9 in 2024-08-19T14:18:32.373 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:18:32.394 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:32.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:32.395 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:18:32.496 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:18:32.529 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:32.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:32.530 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:18:32.631 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:18:32.652 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:32.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:32.653 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:18:32.754 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:18:32.788 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:32.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:32.789 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:18:32.889 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:18:32.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:32.924 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:32.925 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:18:33.025 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:18:33.058 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:33.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:33.059 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:18:33.160 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:18:33.183 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:33.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:33.184 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:18:33.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:32 smithi032 ceph-mon[20004]: pgmap v640: 67 pgs: 2 peering, 1 remapped+peering, 7 unknown, 57 active+clean; 7.7 GiB data, 26 GiB used, 779 GiB / 805 GiB avail; 2.7 KiB/s rd, 52 MiB/s wr, 834 op/s; 71 KiB/s, 0 objects/s recovering 2024-08-19T14:18:33.285 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:18:33.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:32 smithi045 ceph-mon[20143]: pgmap v640: 67 pgs: 2 peering, 1 remapped+peering, 7 unknown, 57 active+clean; 7.7 GiB data, 26 GiB used, 779 GiB / 805 GiB avail; 2.7 KiB/s rd, 52 MiB/s wr, 834 op/s; 71 KiB/s, 0 objects/s recovering 2024-08-19T14:18:33.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:33.321 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:33.321 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:18:33.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:32 smithi031 ceph-mon[16336]: pgmap v640: 67 pgs: 2 peering, 1 remapped+peering, 7 unknown, 57 active+clean; 7.7 GiB data, 26 GiB used, 779 GiB / 805 GiB avail; 2.7 KiB/s rd, 52 MiB/s wr, 834 op/s; 71 KiB/s, 0 objects/s recovering 2024-08-19T14:18:33.422 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:18:33.453 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:33.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:33.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:18:33.555 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:18:33.585 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:33.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:33.586 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:18:33.687 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:18:33.708 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:33.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:33.709 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:18:33.810 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:18:33.844 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:33.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:33.845 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:18:33.946 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:18:33.979 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:33.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:33.980 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:18:34.081 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:18:34.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:34.115 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:34.115 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:18:34.217 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:18:34.246 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:34.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:34.247 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:18:34.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:33 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:18:34.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:33 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:18:34.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:33 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:18:34.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:33 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:18:34.335 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:33 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:18:34.335 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:33 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:18:34.348 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:18:34.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:34.381 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:34.381 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:18:34.482 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:18:34.510 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:34.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:34.511 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:18:34.612 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:18:34.637 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:34.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:34.638 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:18:34.739 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:18:34.762 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:34.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:34.764 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:18:34.865 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:18:34.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:34.897 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:34.897 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:18:34.999 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:18:35.034 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:35.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:35.035 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:18:35.136 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:18:35.170 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:35.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:35.171 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:18:35.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:34 smithi032 ceph-mon[20004]: pgmap v642: 67 pgs: 2 peering, 1 remapped+peering, 3 unknown, 61 active+clean; 7.8 GiB data, 26 GiB used, 778 GiB / 805 GiB avail; 1.8 KiB/s rd, 105 MiB/s wr, 1.80k op/s; 48 KiB/s, 0 objects/s recovering 2024-08-19T14:18:35.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:34 smithi032 ceph-mon[20004]: osdmap e283: 12 total, 12 up, 9 in 2024-08-19T14:18:35.272 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:18:35.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:34 smithi045 ceph-mon[20143]: pgmap v642: 67 pgs: 2 peering, 1 remapped+peering, 3 unknown, 61 active+clean; 7.8 GiB data, 26 GiB used, 778 GiB / 805 GiB avail; 1.8 KiB/s rd, 105 MiB/s wr, 1.80k op/s; 48 KiB/s, 0 objects/s recovering 2024-08-19T14:18:35.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:34 smithi045 ceph-mon[20143]: osdmap e283: 12 total, 12 up, 9 in 2024-08-19T14:18:35.303 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:35.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:35.305 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:18:35.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:34 smithi031 ceph-mon[16336]: pgmap v642: 67 pgs: 2 peering, 1 remapped+peering, 3 unknown, 61 active+clean; 7.8 GiB data, 26 GiB used, 778 GiB / 805 GiB avail; 1.8 KiB/s rd, 105 MiB/s wr, 1.80k op/s; 48 KiB/s, 0 objects/s recovering 2024-08-19T14:18:35.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:34 smithi031 ceph-mon[16336]: osdmap e283: 12 total, 12 up, 9 in 2024-08-19T14:18:35.405 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:18:35.438 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:35.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:35.439 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:18:35.540 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:18:35.572 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:35.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:35.573 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:18:35.675 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:18:35.702 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:35.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:35.703 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:18:35.804 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:18:35.835 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:35.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:35.837 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:18:35.937 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:18:35.972 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:35.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:35.973 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:18:36.051 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:18:36.051 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-08-19T14:18:36.051 INFO:teuthology.orchestra.run.smithi031.stdout: "complaint_time": 30, 2024-08-19T14:18:36.051 INFO:teuthology.orchestra.run.smithi031.stdout: "num_blocked_ops": 0 2024-08-19T14:18:36.051 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:18:36.074 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:18:36.105 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:36.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:36.107 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:18:36.208 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:18:36.238 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:36.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:36.239 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:18:36.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:35 smithi032 ceph-mon[20004]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-08-19T14:18:36.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:35 smithi032 ceph-mon[20004]: osdmap e284: 12 total, 12 up, 9 in 2024-08-19T14:18:36.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:35 smithi045 ceph-mon[20143]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-08-19T14:18:36.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:35 smithi045 ceph-mon[20143]: osdmap e284: 12 total, 12 up, 9 in 2024-08-19T14:18:36.341 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:18:36.349 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:35 smithi031 ceph-mon[16336]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-08-19T14:18:36.349 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:35 smithi031 ceph-mon[16336]: osdmap e284: 12 total, 12 up, 9 in 2024-08-19T14:18:36.367 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:36.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:36.368 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:18:36.469 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:18:36.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:36.503 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:36.503 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:18:36.604 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:18:36.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:36.639 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:36.639 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:18:36.741 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:18:36.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:36.776 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:36.776 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:18:36.791 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-08-19T14:18:36.828 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:18:36.878 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:18:36.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:36.919 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:36.920 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:18:37.021 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:18:37.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:37.053 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:37.053 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:18:37.108 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:36 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:18:37.108 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:36 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:18:37.108 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:36 smithi031 ceph-mon[16336]: pgmap v644: 67 pgs: 2 peering, 65 active+clean; 7.8 GiB data, 26 GiB used, 778 GiB / 805 GiB avail; 1.5 KiB/s rd, 84 MiB/s wr, 1.77k op/s; 39 KiB/s, 0 objects/s recovering 2024-08-19T14:18:37.153 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:18:37.181 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:37.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:37.182 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:18:37.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:36 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:18:37.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:36 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:18:37.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:36 smithi032 ceph-mon[20004]: pgmap v644: 67 pgs: 2 peering, 65 active+clean; 7.8 GiB data, 26 GiB used, 778 GiB / 805 GiB avail; 1.5 KiB/s rd, 84 MiB/s wr, 1.77k op/s; 39 KiB/s, 0 objects/s recovering 2024-08-19T14:18:37.283 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:18:37.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:36 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:18:37.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:36 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:18:37.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:36 smithi045 ceph-mon[20143]: pgmap v644: 67 pgs: 2 peering, 65 active+clean; 7.8 GiB data, 26 GiB used, 778 GiB / 805 GiB avail; 1.5 KiB/s rd, 84 MiB/s wr, 1.77k op/s; 39 KiB/s, 0 objects/s recovering 2024-08-19T14:18:37.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:37.319 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:37.319 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:18:37.404 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:18:37.420 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:18:37.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:37.451 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:37.452 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:18:37.468 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:18:37.552 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:18:37.585 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:37.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:37.586 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:18:37.686 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-19T14:18:37.687 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:18:37.715 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:37.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:37.716 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:18:37.777 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:18:37.777 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:18:37.777 INFO:teuthology.orchestra.run.smithi031.stdout: "epoch": 285, 2024-08-19T14:18:37.777 INFO:teuthology.orchestra.run.smithi031.stdout: "fsid": "ed7d1c6e-5e33-11ef-bccf-c7b262605968", 2024-08-19T14:18:37.777 INFO:teuthology.orchestra.run.smithi031.stdout: "created": "2024-08-19T14:05:22.842169+0000", 2024-08-19T14:18:37.777 INFO:teuthology.orchestra.run.smithi031.stdout: "modified": "2024-08-19T14:18:36.974640+0000", 2024-08-19T14:18:37.777 INFO:teuthology.orchestra.run.smithi031.stdout: "last_up_change": "2024-08-19T14:11:39.491234+0000", 2024-08-19T14:18:37.777 INFO:teuthology.orchestra.run.smithi031.stdout: "last_in_change": "2024-08-19T14:16:38.207673+0000", 2024-08-19T14:18:37.777 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-08-19T14:18:37.777 INFO:teuthology.orchestra.run.smithi031.stdout: "flags_num": 5801984, 2024-08-19T14:18:37.778 INFO:teuthology.orchestra.run.smithi031.stdout: "flags_set": [ 2024-08-19T14:18:37.778 INFO:teuthology.orchestra.run.smithi031.stdout: "noscrub", 2024-08-19T14:18:37.778 INFO:teuthology.orchestra.run.smithi031.stdout: "pglog_hardlimit", 2024-08-19T14:18:37.778 INFO:teuthology.orchestra.run.smithi031.stdout: "purged_snapdirs", 2024-08-19T14:18:37.778 INFO:teuthology.orchestra.run.smithi031.stdout: "recovery_deletes", 2024-08-19T14:18:37.778 INFO:teuthology.orchestra.run.smithi031.stdout: "sortbitwise" 2024-08-19T14:18:37.778 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-08-19T14:18:37.778 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_version": 27, 2024-08-19T14:18:37.778 INFO:teuthology.orchestra.run.smithi031.stdout: "full_ratio": 0.94999998807907104, 2024-08-19T14:18:37.778 INFO:teuthology.orchestra.run.smithi031.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-08-19T14:18:37.778 INFO:teuthology.orchestra.run.smithi031.stdout: "nearfull_ratio": 0.85000002384185791, 2024-08-19T14:18:37.778 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_snapshot": "", 2024-08-19T14:18:37.778 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_max": 3, 2024-08-19T14:18:37.779 INFO:teuthology.orchestra.run.smithi031.stdout: "max_osd": 12, 2024-08-19T14:18:37.779 INFO:teuthology.orchestra.run.smithi031.stdout: "require_min_compat_client": "luminous", 2024-08-19T14:18:37.779 INFO:teuthology.orchestra.run.smithi031.stdout: "min_compat_client": "luminous", 2024-08-19T14:18:37.779 INFO:teuthology.orchestra.run.smithi031.stdout: "require_osd_release": "squid", 2024-08-19T14:18:37.779 INFO:teuthology.orchestra.run.smithi031.stdout: "allow_crimson": false, 2024-08-19T14:18:37.779 INFO:teuthology.orchestra.run.smithi031.stdout: "pools": [ 2024-08-19T14:18:37.779 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.779 INFO:teuthology.orchestra.run.smithi031.stdout: "pool": 1, 2024-08-19T14:18:37.779 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_name": ".mgr", 2024-08-19T14:18:37.779 INFO:teuthology.orchestra.run.smithi031.stdout: "create_time": "2024-08-19T14:08:15.966221+0000", 2024-08-19T14:18:37.779 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-08-19T14:18:37.779 INFO:teuthology.orchestra.run.smithi031.stdout: "flags_names": "hashpspool", 2024-08-19T14:18:37.779 INFO:teuthology.orchestra.run.smithi031.stdout: "type": 1, 2024-08-19T14:18:37.780 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 3, 2024-08-19T14:18:37.780 INFO:teuthology.orchestra.run.smithi031.stdout: "min_size": 2, 2024-08-19T14:18:37.780 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_rule": 0, 2024-08-19T14:18:37.780 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_count": 0, 2024-08-19T14:18:37.780 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_target": 0, 2024-08-19T14:18:37.780 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_barrier": 0, 2024-08-19T14:18:37.780 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-08-19T14:18:37.780 INFO:teuthology.orchestra.run.smithi031.stdout: "object_hash": 2, 2024-08-19T14:18:37.780 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_autoscale_mode": "off", 2024-08-19T14:18:37.781 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num": 31, 2024-08-19T14:18:37.781 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_placement_num": 31, 2024-08-19T14:18:37.781 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_placement_num_target": 31, 2024-08-19T14:18:37.781 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num_target": 31, 2024-08-19T14:18:37.781 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num_pending": 31, 2024-08-19T14:18:37.781 INFO:teuthology.orchestra.run.smithi031.stdout: "last_pg_merge_meta": { 2024-08-19T14:18:37.781 INFO:teuthology.orchestra.run.smithi031.stdout: "source_pgid": "0.0", 2024-08-19T14:18:37.781 INFO:teuthology.orchestra.run.smithi031.stdout: "ready_epoch": 0, 2024-08-19T14:18:37.781 INFO:teuthology.orchestra.run.smithi031.stdout: "last_epoch_started": 0, 2024-08-19T14:18:37.781 INFO:teuthology.orchestra.run.smithi031.stdout: "last_epoch_clean": 0, 2024-08-19T14:18:37.781 INFO:teuthology.orchestra.run.smithi031.stdout: "source_version": "0'0", 2024-08-19T14:18:37.781 INFO:teuthology.orchestra.run.smithi031.stdout: "target_version": "0'0" 2024-08-19T14:18:37.782 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.782 INFO:teuthology.orchestra.run.smithi031.stdout: "last_change": "278", 2024-08-19T14:18:37.782 INFO:teuthology.orchestra.run.smithi031.stdout: "last_force_op_resend": "0", 2024-08-19T14:18:37.782 INFO:teuthology.orchestra.run.smithi031.stdout: "last_force_op_resend_prenautilus": "0", 2024-08-19T14:18:37.782 INFO:teuthology.orchestra.run.smithi031.stdout: "last_force_op_resend_preluminous": "276", 2024-08-19T14:18:37.782 INFO:teuthology.orchestra.run.smithi031.stdout: "auid": 0, 2024-08-19T14:18:37.782 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_mode": "selfmanaged", 2024-08-19T14:18:37.782 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_seq": 0, 2024-08-19T14:18:37.782 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_epoch": 0, 2024-08-19T14:18:37.782 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_snaps": [], 2024-08-19T14:18:37.783 INFO:teuthology.orchestra.run.smithi031.stdout: "removed_snaps": "[]", 2024-08-19T14:18:37.783 INFO:teuthology.orchestra.run.smithi031.stdout: "quota_max_bytes": 0, 2024-08-19T14:18:37.783 INFO:teuthology.orchestra.run.smithi031.stdout: "quota_max_objects": 0, 2024-08-19T14:18:37.783 INFO:teuthology.orchestra.run.smithi031.stdout: "tiers": [], 2024-08-19T14:18:37.783 INFO:teuthology.orchestra.run.smithi031.stdout: "tier_of": -1, 2024-08-19T14:18:37.783 INFO:teuthology.orchestra.run.smithi031.stdout: "read_tier": -1, 2024-08-19T14:18:37.783 INFO:teuthology.orchestra.run.smithi031.stdout: "write_tier": -1, 2024-08-19T14:18:37.783 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_mode": "none", 2024-08-19T14:18:37.783 INFO:teuthology.orchestra.run.smithi031.stdout: "target_max_bytes": 0, 2024-08-19T14:18:37.783 INFO:teuthology.orchestra.run.smithi031.stdout: "target_max_objects": 0, 2024-08-19T14:18:37.783 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-08-19T14:18:37.784 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-08-19T14:18:37.784 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_target_full_ratio_micro": 800000, 2024-08-19T14:18:37.784 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_min_flush_age": 0, 2024-08-19T14:18:37.784 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_min_evict_age": 0, 2024-08-19T14:18:37.784 INFO:teuthology.orchestra.run.smithi031.stdout: "erasure_code_profile": "", 2024-08-19T14:18:37.784 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_params": { 2024-08-19T14:18:37.784 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "none" 2024-08-19T14:18:37.784 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.784 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_period": 0, 2024-08-19T14:18:37.784 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_count": 0, 2024-08-19T14:18:37.785 INFO:teuthology.orchestra.run.smithi031.stdout: "use_gmt_hitset": true, 2024-08-19T14:18:37.785 INFO:teuthology.orchestra.run.smithi031.stdout: "min_read_recency_for_promote": 0, 2024-08-19T14:18:37.785 INFO:teuthology.orchestra.run.smithi031.stdout: "min_write_recency_for_promote": 0, 2024-08-19T14:18:37.785 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_grade_decay_rate": 0, 2024-08-19T14:18:37.785 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_search_last_n": 0, 2024-08-19T14:18:37.785 INFO:teuthology.orchestra.run.smithi031.stdout: "grade_table": [], 2024-08-19T14:18:37.785 INFO:teuthology.orchestra.run.smithi031.stdout: "stripe_width": 0, 2024-08-19T14:18:37.785 INFO:teuthology.orchestra.run.smithi031.stdout: "expected_num_objects": 0, 2024-08-19T14:18:37.785 INFO:teuthology.orchestra.run.smithi031.stdout: "fast_read": false, 2024-08-19T14:18:37.785 INFO:teuthology.orchestra.run.smithi031.stdout: "options": { 2024-08-19T14:18:37.785 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num_max": 32, 2024-08-19T14:18:37.786 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num_min": 1 2024-08-19T14:18:37.786 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.786 INFO:teuthology.orchestra.run.smithi031.stdout: "application_metadata": { 2024-08-19T14:18:37.786 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": {} 2024-08-19T14:18:37.786 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.786 INFO:teuthology.orchestra.run.smithi031.stdout: "read_balance": { 2024-08-19T14:18:37.786 INFO:teuthology.orchestra.run.smithi031.stdout: "score_type": "Fair distribution", 2024-08-19T14:18:37.786 INFO:teuthology.orchestra.run.smithi031.stdout: "score_acting": 2.3199999332427979, 2024-08-19T14:18:37.786 INFO:teuthology.orchestra.run.smithi031.stdout: "score_stable": 2.3199999332427979, 2024-08-19T14:18:37.787 INFO:teuthology.orchestra.run.smithi031.stdout: "optimal_score": 0.75, 2024-08-19T14:18:37.787 INFO:teuthology.orchestra.run.smithi031.stdout: "raw_score_acting": 1.7400000095367432, 2024-08-19T14:18:37.787 INFO:teuthology.orchestra.run.smithi031.stdout: "raw_score_stable": 1.7400000095367432, 2024-08-19T14:18:37.787 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity_weighted": 0.92281585931777954, 2024-08-19T14:18:37.787 INFO:teuthology.orchestra.run.smithi031.stdout: "average_primary_affinity": 0.92000001668930054, 2024-08-19T14:18:37.787 INFO:teuthology.orchestra.run.smithi031.stdout: "average_primary_affinity_weighted": 0.92000001668930054 2024-08-19T14:18:37.787 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:37.787 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.787 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.788 INFO:teuthology.orchestra.run.smithi031.stdout: "pool": 3, 2024-08-19T14:18:37.788 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_name": "unique_pool_1", 2024-08-19T14:18:37.788 INFO:teuthology.orchestra.run.smithi031.stdout: "create_time": "2024-08-19T14:16:55.393520+0000", 2024-08-19T14:18:37.788 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-08-19T14:18:37.788 INFO:teuthology.orchestra.run.smithi031.stdout: "flags_names": "hashpspool", 2024-08-19T14:18:37.788 INFO:teuthology.orchestra.run.smithi031.stdout: "type": 1, 2024-08-19T14:18:37.788 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 3, 2024-08-19T14:18:37.788 INFO:teuthology.orchestra.run.smithi031.stdout: "min_size": 2, 2024-08-19T14:18:37.788 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_rule": 0, 2024-08-19T14:18:37.788 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_count": 0, 2024-08-19T14:18:37.788 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_target": 0, 2024-08-19T14:18:37.788 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_barrier": 0, 2024-08-19T14:18:37.789 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-08-19T14:18:37.789 INFO:teuthology.orchestra.run.smithi031.stdout: "object_hash": 2, 2024-08-19T14:18:37.789 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_autoscale_mode": "off", 2024-08-19T14:18:37.789 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num": 36, 2024-08-19T14:18:37.789 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_placement_num": 36, 2024-08-19T14:18:37.789 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_placement_num_target": 36, 2024-08-19T14:18:37.789 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num_target": 36, 2024-08-19T14:18:37.789 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num_pending": 36, 2024-08-19T14:18:37.789 INFO:teuthology.orchestra.run.smithi031.stdout: "last_pg_merge_meta": { 2024-08-19T14:18:37.789 INFO:teuthology.orchestra.run.smithi031.stdout: "source_pgid": "0.0", 2024-08-19T14:18:37.789 INFO:teuthology.orchestra.run.smithi031.stdout: "ready_epoch": 0, 2024-08-19T14:18:37.789 INFO:teuthology.orchestra.run.smithi031.stdout: "last_epoch_started": 0, 2024-08-19T14:18:37.789 INFO:teuthology.orchestra.run.smithi031.stdout: "last_epoch_clean": 0, 2024-08-19T14:18:37.790 INFO:teuthology.orchestra.run.smithi031.stdout: "source_version": "0'0", 2024-08-19T14:18:37.790 INFO:teuthology.orchestra.run.smithi031.stdout: "target_version": "0'0" 2024-08-19T14:18:37.790 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.790 INFO:teuthology.orchestra.run.smithi031.stdout: "last_change": "246", 2024-08-19T14:18:37.790 INFO:teuthology.orchestra.run.smithi031.stdout: "last_force_op_resend": "0", 2024-08-19T14:18:37.790 INFO:teuthology.orchestra.run.smithi031.stdout: "last_force_op_resend_prenautilus": "0", 2024-08-19T14:18:37.790 INFO:teuthology.orchestra.run.smithi031.stdout: "last_force_op_resend_preluminous": "244", 2024-08-19T14:18:37.790 INFO:teuthology.orchestra.run.smithi031.stdout: "auid": 0, 2024-08-19T14:18:37.790 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_mode": "selfmanaged", 2024-08-19T14:18:37.790 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_seq": 0, 2024-08-19T14:18:37.790 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_epoch": 0, 2024-08-19T14:18:37.790 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_snaps": [], 2024-08-19T14:18:37.790 INFO:teuthology.orchestra.run.smithi031.stdout: "removed_snaps": "[]", 2024-08-19T14:18:37.791 INFO:teuthology.orchestra.run.smithi031.stdout: "quota_max_bytes": 0, 2024-08-19T14:18:37.791 INFO:teuthology.orchestra.run.smithi031.stdout: "quota_max_objects": 0, 2024-08-19T14:18:37.791 INFO:teuthology.orchestra.run.smithi031.stdout: "tiers": [], 2024-08-19T14:18:37.791 INFO:teuthology.orchestra.run.smithi031.stdout: "tier_of": -1, 2024-08-19T14:18:37.791 INFO:teuthology.orchestra.run.smithi031.stdout: "read_tier": -1, 2024-08-19T14:18:37.791 INFO:teuthology.orchestra.run.smithi031.stdout: "write_tier": -1, 2024-08-19T14:18:37.791 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_mode": "none", 2024-08-19T14:18:37.791 INFO:teuthology.orchestra.run.smithi031.stdout: "target_max_bytes": 0, 2024-08-19T14:18:37.791 INFO:teuthology.orchestra.run.smithi031.stdout: "target_max_objects": 0, 2024-08-19T14:18:37.791 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-08-19T14:18:37.791 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-08-19T14:18:37.791 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_target_full_ratio_micro": 800000, 2024-08-19T14:18:37.791 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_min_flush_age": 0, 2024-08-19T14:18:37.791 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_min_evict_age": 0, 2024-08-19T14:18:37.792 INFO:teuthology.orchestra.run.smithi031.stdout: "erasure_code_profile": "", 2024-08-19T14:18:37.792 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_params": { 2024-08-19T14:18:37.792 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "none" 2024-08-19T14:18:37.792 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.792 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_period": 0, 2024-08-19T14:18:37.792 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_count": 0, 2024-08-19T14:18:37.792 INFO:teuthology.orchestra.run.smithi031.stdout: "use_gmt_hitset": true, 2024-08-19T14:18:37.792 INFO:teuthology.orchestra.run.smithi031.stdout: "min_read_recency_for_promote": 0, 2024-08-19T14:18:37.792 INFO:teuthology.orchestra.run.smithi031.stdout: "min_write_recency_for_promote": 0, 2024-08-19T14:18:37.792 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_grade_decay_rate": 0, 2024-08-19T14:18:37.792 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_search_last_n": 0, 2024-08-19T14:18:37.792 INFO:teuthology.orchestra.run.smithi031.stdout: "grade_table": [], 2024-08-19T14:18:37.792 INFO:teuthology.orchestra.run.smithi031.stdout: "stripe_width": 0, 2024-08-19T14:18:37.792 INFO:teuthology.orchestra.run.smithi031.stdout: "expected_num_objects": 0, 2024-08-19T14:18:37.793 INFO:teuthology.orchestra.run.smithi031.stdout: "fast_read": false, 2024-08-19T14:18:37.793 INFO:teuthology.orchestra.run.smithi031.stdout: "options": {}, 2024-08-19T14:18:37.793 INFO:teuthology.orchestra.run.smithi031.stdout: "application_metadata": { 2024-08-19T14:18:37.793 INFO:teuthology.orchestra.run.smithi031.stdout: "rados": {} 2024-08-19T14:18:37.793 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.793 INFO:teuthology.orchestra.run.smithi031.stdout: "read_balance": { 2024-08-19T14:18:37.793 INFO:teuthology.orchestra.run.smithi031.stdout: "score_type": "Fair distribution", 2024-08-19T14:18:37.793 INFO:teuthology.orchestra.run.smithi031.stdout: "score_acting": 2, 2024-08-19T14:18:37.793 INFO:teuthology.orchestra.run.smithi031.stdout: "score_stable": 2, 2024-08-19T14:18:37.793 INFO:teuthology.orchestra.run.smithi031.stdout: "optimal_score": 0.75, 2024-08-19T14:18:37.794 INFO:teuthology.orchestra.run.smithi031.stdout: "raw_score_acting": 1.5, 2024-08-19T14:18:37.794 INFO:teuthology.orchestra.run.smithi031.stdout: "raw_score_stable": 1.5, 2024-08-19T14:18:37.794 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity_weighted": 0.92281585931777954, 2024-08-19T14:18:37.794 INFO:teuthology.orchestra.run.smithi031.stdout: "average_primary_affinity": 0.92000001668930054, 2024-08-19T14:18:37.794 INFO:teuthology.orchestra.run.smithi031.stdout: "average_primary_affinity_weighted": 0.92000001668930054 2024-08-19T14:18:37.794 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:37.794 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:37.794 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-08-19T14:18:37.794 INFO:teuthology.orchestra.run.smithi031.stdout: "osds": [ 2024-08-19T14:18:37.794 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.794 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 0, 2024-08-19T14:18:37.794 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "d7a0fff3-23c6-4ee3-a275-a844862aabfb", 2024-08-19T14:18:37.794 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-08-19T14:18:37.795 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-08-19T14:18:37.795 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-08-19T14:18:37.795 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-08-19T14:18:37.795 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-08-19T14:18:37.795 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-08-19T14:18:37.795 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 8, 2024-08-19T14:18:37.795 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 281, 2024-08-19T14:18:37.795 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-08-19T14:18:37.795 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-08-19T14:18:37.795 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-08-19T14:18:37.795 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:18:37.795 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.795 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:18:37.796 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6802", 2024-08-19T14:18:37.796 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1108945097 2024-08-19T14:18:37.796 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.796 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.796 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:18:37.796 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6803", 2024-08-19T14:18:37.796 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1108945097 2024-08-19T14:18:37.796 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:37.796 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:18:37.797 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.797 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-08-19T14:18:37.797 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:18:37.797 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.797 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:18:37.797 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6804", 2024-08-19T14:18:37.797 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1108945097 2024-08-19T14:18:37.797 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.797 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.797 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:18:37.797 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6805", 2024-08-19T14:18:37.797 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1108945097 2024-08-19T14:18:37.797 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:37.798 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:18:37.798 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.798 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-08-19T14:18:37.798 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:18:37.798 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.798 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:18:37.798 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6808", 2024-08-19T14:18:37.798 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1108945097 2024-08-19T14:18:37.798 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.798 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.798 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:18:37.798 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6809", 2024-08-19T14:18:37.798 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1108945097 2024-08-19T14:18:37.798 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:37.799 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:18:37.799 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.799 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-08-19T14:18:37.799 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:18:37.799 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.799 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:18:37.799 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6806", 2024-08-19T14:18:37.799 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1108945097 2024-08-19T14:18:37.799 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.799 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.799 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:18:37.799 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6807", 2024-08-19T14:18:37.799 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1108945097 2024-08-19T14:18:37.800 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:37.800 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:18:37.800 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.800 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.31:6803/1108945097", 2024-08-19T14:18:37.800 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.31:6805/1108945097", 2024-08-19T14:18:37.800 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.31:6809/1108945097", 2024-08-19T14:18:37.800 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.31:6807/1108945097", 2024-08-19T14:18:37.800 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-08-19T14:18:37.800 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-08-19T14:18:37.800 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-08-19T14:18:37.800 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:18:37.800 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.800 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.800 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 1, 2024-08-19T14:18:37.801 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "145d5bb4-74dd-440f-bcc9-7e9ebda9c5d6", 2024-08-19T14:18:37.801 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-08-19T14:18:37.801 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-08-19T14:18:37.801 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-08-19T14:18:37.801 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-08-19T14:18:37.801 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-08-19T14:18:37.801 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-08-19T14:18:37.801 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 13, 2024-08-19T14:18:37.801 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 281, 2024-08-19T14:18:37.801 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-08-19T14:18:37.801 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-08-19T14:18:37.801 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-08-19T14:18:37.801 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:18:37.801 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.801 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:18:37.802 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6810", 2024-08-19T14:18:37.802 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3236186806 2024-08-19T14:18:37.802 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.802 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.802 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:18:37.802 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6811", 2024-08-19T14:18:37.802 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3236186806 2024-08-19T14:18:37.802 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:37.802 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:18:37.802 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.802 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-08-19T14:18:37.803 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:18:37.803 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.803 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:18:37.803 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6812", 2024-08-19T14:18:37.803 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3236186806 2024-08-19T14:18:37.803 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.803 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.803 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:18:37.803 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6813", 2024-08-19T14:18:37.803 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3236186806 2024-08-19T14:18:37.803 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:37.803 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:18:37.803 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.803 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-08-19T14:18:37.804 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:18:37.804 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.804 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:18:37.804 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6816", 2024-08-19T14:18:37.804 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3236186806 2024-08-19T14:18:37.804 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.804 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.804 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:18:37.804 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6817", 2024-08-19T14:18:37.804 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3236186806 2024-08-19T14:18:37.805 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:37.805 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:18:37.805 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.805 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-08-19T14:18:37.805 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:18:37.805 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.805 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:18:37.805 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6814", 2024-08-19T14:18:37.805 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3236186806 2024-08-19T14:18:37.805 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.805 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.805 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:18:37.806 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6815", 2024-08-19T14:18:37.806 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3236186806 2024-08-19T14:18:37.806 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:37.806 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:18:37.806 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.806 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.31:6811/3236186806", 2024-08-19T14:18:37.806 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.31:6813/3236186806", 2024-08-19T14:18:37.806 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.31:6817/3236186806", 2024-08-19T14:18:37.806 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.31:6815/3236186806", 2024-08-19T14:18:37.807 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-08-19T14:18:37.807 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-08-19T14:18:37.807 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-08-19T14:18:37.807 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:18:37.807 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.807 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.807 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 2, 2024-08-19T14:18:37.807 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "376c18d0-4768-46a0-bb60-e79c88ca909c", 2024-08-19T14:18:37.807 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-08-19T14:18:37.807 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 0, 2024-08-19T14:18:37.808 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 0, 2024-08-19T14:18:37.808 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 0.07379150390625, 2024-08-19T14:18:37.808 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-08-19T14:18:37.808 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-08-19T14:18:37.808 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 18, 2024-08-19T14:18:37.808 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 204, 2024-08-19T14:18:37.808 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-08-19T14:18:37.808 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-08-19T14:18:37.808 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-08-19T14:18:37.809 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:18:37.809 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.809 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:18:37.809 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6818", 2024-08-19T14:18:37.809 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 4045553387 2024-08-19T14:18:37.809 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.809 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.809 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:18:37.809 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6819", 2024-08-19T14:18:37.809 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 4045553387 2024-08-19T14:18:37.809 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:37.809 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:18:37.809 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.809 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-08-19T14:18:37.810 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:18:37.810 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.810 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:18:37.810 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6820", 2024-08-19T14:18:37.810 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 4045553387 2024-08-19T14:18:37.810 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.810 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.810 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:18:37.810 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6821", 2024-08-19T14:18:37.810 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 4045553387 2024-08-19T14:18:37.811 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:37.811 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:18:37.811 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.811 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-08-19T14:18:37.811 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:18:37.811 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.811 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:18:37.811 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6824", 2024-08-19T14:18:37.811 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 4045553387 2024-08-19T14:18:37.811 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.811 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.811 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:18:37.811 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6825", 2024-08-19T14:18:37.812 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 4045553387 2024-08-19T14:18:37.812 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:37.812 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:18:37.812 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.812 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-08-19T14:18:37.812 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:18:37.812 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.813 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:18:37.813 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6822", 2024-08-19T14:18:37.813 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 4045553387 2024-08-19T14:18:37.813 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.813 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.813 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:18:37.813 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6823", 2024-08-19T14:18:37.813 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 4045553387 2024-08-19T14:18:37.813 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:37.813 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:18:37.813 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.813 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.31:6819/4045553387", 2024-08-19T14:18:37.813 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.31:6821/4045553387", 2024-08-19T14:18:37.814 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.31:6825/4045553387", 2024-08-19T14:18:37.814 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.31:6823/4045553387", 2024-08-19T14:18:37.814 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-08-19T14:18:37.814 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-08-19T14:18:37.814 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-08-19T14:18:37.814 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:18:37.814 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.814 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.814 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 3, 2024-08-19T14:18:37.814 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "a787ea9f-b9ba-450f-b388-614cfba90634", 2024-08-19T14:18:37.815 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-08-19T14:18:37.815 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-08-19T14:18:37.815 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 0.6437225341796875, 2024-08-19T14:18:37.815 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-08-19T14:18:37.815 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-08-19T14:18:37.815 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-08-19T14:18:37.815 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 26, 2024-08-19T14:18:37.815 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 280, 2024-08-19T14:18:37.815 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-08-19T14:18:37.815 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-08-19T14:18:37.815 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-08-19T14:18:37.815 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:18:37.815 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.815 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:18:37.816 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6826", 2024-08-19T14:18:37.816 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2288082716 2024-08-19T14:18:37.816 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.816 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.816 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:18:37.816 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6827", 2024-08-19T14:18:37.816 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2288082716 2024-08-19T14:18:37.816 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:37.816 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:18:37.816 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.817 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-08-19T14:18:37.817 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:18:37.817 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.817 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:18:37.817 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6828", 2024-08-19T14:18:37.817 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2288082716 2024-08-19T14:18:37.817 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.818 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.818 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:18:37.818 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6829", 2024-08-19T14:18:37.818 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2288082716 2024-08-19T14:18:37.818 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:37.818 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:18:37.818 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.819 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-08-19T14:18:37.819 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:18:37.819 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.819 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:18:37.819 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6832", 2024-08-19T14:18:37.819 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2288082716 2024-08-19T14:18:37.820 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:18:37.821 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.821 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.821 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:18:37.821 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6833", 2024-08-19T14:18:37.821 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2288082716 2024-08-19T14:18:37.822 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:37.822 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:18:37.822 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.822 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-08-19T14:18:37.822 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:18:37.822 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.822 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:18:37.822 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6830", 2024-08-19T14:18:37.822 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2288082716 2024-08-19T14:18:37.822 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.822 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.822 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:18:37.822 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6831", 2024-08-19T14:18:37.823 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2288082716 2024-08-19T14:18:37.823 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:37.823 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:18:37.823 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.823 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.31:6827/2288082716", 2024-08-19T14:18:37.823 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.31:6829/2288082716", 2024-08-19T14:18:37.823 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.31:6833/2288082716", 2024-08-19T14:18:37.823 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.31:6831/2288082716", 2024-08-19T14:18:37.823 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-08-19T14:18:37.823 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-08-19T14:18:37.823 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-08-19T14:18:37.823 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:18:37.823 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.824 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.824 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 4, 2024-08-19T14:18:37.824 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "7859751a-12f5-4a57-a332-435360479d53", 2024-08-19T14:18:37.824 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-08-19T14:18:37.824 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-08-19T14:18:37.824 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-08-19T14:18:37.824 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-08-19T14:18:37.824 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-08-19T14:18:37.824 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-08-19T14:18:37.824 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 31, 2024-08-19T14:18:37.824 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 281, 2024-08-19T14:18:37.824 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-08-19T14:18:37.824 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-08-19T14:18:37.824 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-08-19T14:18:37.825 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:18:37.825 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.825 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:18:37.825 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6800", 2024-08-19T14:18:37.825 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3934739984 2024-08-19T14:18:37.825 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.825 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.825 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:18:37.825 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6801", 2024-08-19T14:18:37.825 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3934739984 2024-08-19T14:18:37.825 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:37.825 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:18:37.825 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.825 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-08-19T14:18:37.826 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:18:37.826 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.826 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:18:37.826 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6802", 2024-08-19T14:18:37.826 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3934739984 2024-08-19T14:18:37.826 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.826 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.826 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:18:37.826 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6803", 2024-08-19T14:18:37.826 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3934739984 2024-08-19T14:18:37.826 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:37.826 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:18:37.826 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.826 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-08-19T14:18:37.827 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:18:37.827 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.827 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:18:37.827 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6806", 2024-08-19T14:18:37.827 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3934739984 2024-08-19T14:18:37.827 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.827 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.827 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:18:37.827 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6807", 2024-08-19T14:18:37.827 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3934739984 2024-08-19T14:18:37.827 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:37.827 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:18:37.827 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.827 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-08-19T14:18:37.828 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:18:37.828 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.828 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:18:37.828 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6804", 2024-08-19T14:18:37.828 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3934739984 2024-08-19T14:18:37.828 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.828 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.828 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:18:37.828 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6805", 2024-08-19T14:18:37.829 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3934739984 2024-08-19T14:18:37.829 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:37.829 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:18:37.829 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.829 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.32:6801/3934739984", 2024-08-19T14:18:37.829 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.32:6803/3934739984", 2024-08-19T14:18:37.829 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.32:6807/3934739984", 2024-08-19T14:18:37.829 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.32:6805/3934739984", 2024-08-19T14:18:37.829 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-08-19T14:18:37.829 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-08-19T14:18:37.829 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-08-19T14:18:37.829 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:18:37.829 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.830 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.830 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 5, 2024-08-19T14:18:37.830 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "017f3eef-ac60-4b82-8350-919017cb4ccd", 2024-08-19T14:18:37.830 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-08-19T14:18:37.830 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-08-19T14:18:37.830 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-08-19T14:18:37.830 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-08-19T14:18:37.830 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-08-19T14:18:37.830 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-08-19T14:18:37.830 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 40, 2024-08-19T14:18:37.830 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 281, 2024-08-19T14:18:37.830 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-08-19T14:18:37.830 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-08-19T14:18:37.831 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-08-19T14:18:37.831 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:18:37.831 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.831 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:18:37.831 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6808", 2024-08-19T14:18:37.831 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 526388985 2024-08-19T14:18:37.831 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.831 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.831 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:18:37.831 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6809", 2024-08-19T14:18:37.831 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 526388985 2024-08-19T14:18:37.831 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:37.831 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:18:37.831 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.832 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-08-19T14:18:37.832 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:18:37.832 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.832 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:18:37.832 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6810", 2024-08-19T14:18:37.832 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 526388985 2024-08-19T14:18:37.832 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.832 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.832 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:18:37.832 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6811", 2024-08-19T14:18:37.832 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 526388985 2024-08-19T14:18:37.832 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:37.832 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:18:37.832 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.833 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-08-19T14:18:37.833 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:18:37.833 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.833 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:18:37.833 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6814", 2024-08-19T14:18:37.833 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 526388985 2024-08-19T14:18:37.833 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.833 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.833 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:18:37.833 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6815", 2024-08-19T14:18:37.833 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 526388985 2024-08-19T14:18:37.833 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:37.833 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:18:37.833 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.834 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-08-19T14:18:37.834 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:18:37.834 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.834 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:18:37.834 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6812", 2024-08-19T14:18:37.834 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 526388985 2024-08-19T14:18:37.834 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.834 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.834 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:18:37.834 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6813", 2024-08-19T14:18:37.834 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 526388985 2024-08-19T14:18:37.834 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:37.834 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:18:37.835 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.835 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.32:6809/526388985", 2024-08-19T14:18:37.835 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.32:6811/526388985", 2024-08-19T14:18:37.835 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.32:6815/526388985", 2024-08-19T14:18:37.835 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.32:6813/526388985", 2024-08-19T14:18:37.835 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-08-19T14:18:37.835 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-08-19T14:18:37.835 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-08-19T14:18:37.835 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:18:37.835 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.835 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.835 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 6, 2024-08-19T14:18:37.835 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "44aa8229-0268-49d7-bd57-3bf1f384e5e6", 2024-08-19T14:18:37.836 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-08-19T14:18:37.836 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 0, 2024-08-19T14:18:37.836 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 0, 2024-08-19T14:18:37.836 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-08-19T14:18:37.836 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-08-19T14:18:37.836 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-08-19T14:18:37.836 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 48, 2024-08-19T14:18:37.836 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 126, 2024-08-19T14:18:37.836 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-08-19T14:18:37.836 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-08-19T14:18:37.836 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-08-19T14:18:37.836 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:18:37.836 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.836 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:18:37.837 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6816", 2024-08-19T14:18:37.837 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1131035844 2024-08-19T14:18:37.837 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.837 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.837 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:18:37.837 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6817", 2024-08-19T14:18:37.837 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1131035844 2024-08-19T14:18:37.837 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:37.837 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:18:37.837 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.837 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-08-19T14:18:37.837 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:18:37.837 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.838 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:18:37.838 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6818", 2024-08-19T14:18:37.838 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1131035844 2024-08-19T14:18:37.838 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.838 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.838 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:18:37.838 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6819", 2024-08-19T14:18:37.838 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1131035844 2024-08-19T14:18:37.838 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:37.838 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:18:37.838 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.838 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-08-19T14:18:37.838 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:18:37.839 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.839 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:18:37.839 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6822", 2024-08-19T14:18:37.839 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1131035844 2024-08-19T14:18:37.839 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.839 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.839 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:18:37.839 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6823", 2024-08-19T14:18:37.839 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1131035844 2024-08-19T14:18:37.839 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:37.839 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:18:37.839 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.839 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-08-19T14:18:37.840 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:18:37.840 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.840 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:18:37.840 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6820", 2024-08-19T14:18:37.840 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1131035844 2024-08-19T14:18:37.840 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.840 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.840 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:18:37.840 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6821", 2024-08-19T14:18:37.840 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1131035844 2024-08-19T14:18:37.840 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:37.840 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:18:37.841 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.841 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.32:6817/1131035844", 2024-08-19T14:18:37.841 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.32:6819/1131035844", 2024-08-19T14:18:37.841 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.32:6823/1131035844", 2024-08-19T14:18:37.841 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.32:6821/1131035844", 2024-08-19T14:18:37.841 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-08-19T14:18:37.841 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-08-19T14:18:37.841 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-08-19T14:18:37.841 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:18:37.841 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.841 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.842 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 7, 2024-08-19T14:18:37.842 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "263433e5-70b6-4192-86ff-dc2ad95927eb", 2024-08-19T14:18:37.842 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-08-19T14:18:37.842 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-08-19T14:18:37.842 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 0.95001220703125, 2024-08-19T14:18:37.842 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-08-19T14:18:37.842 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-08-19T14:18:37.842 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-08-19T14:18:37.842 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 57, 2024-08-19T14:18:37.842 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 281, 2024-08-19T14:18:37.842 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-08-19T14:18:37.842 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-08-19T14:18:37.842 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-08-19T14:18:37.842 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:18:37.843 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.843 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:18:37.843 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6824", 2024-08-19T14:18:37.843 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2014939947 2024-08-19T14:18:37.843 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.843 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.843 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:18:37.843 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6825", 2024-08-19T14:18:37.843 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2014939947 2024-08-19T14:18:37.844 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:37.844 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:18:37.844 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.844 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-08-19T14:18:37.844 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:18:37.844 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.844 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:18:37.844 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6826", 2024-08-19T14:18:37.844 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2014939947 2024-08-19T14:18:37.844 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.844 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.844 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:18:37.844 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6827", 2024-08-19T14:18:37.844 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2014939947 2024-08-19T14:18:37.845 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:37.845 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:18:37.845 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.845 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-08-19T14:18:37.845 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:18:37.845 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.845 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:18:37.845 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6830", 2024-08-19T14:18:37.845 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2014939947 2024-08-19T14:18:37.845 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.845 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.845 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:18:37.845 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6831", 2024-08-19T14:18:37.845 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2014939947 2024-08-19T14:18:37.846 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:37.846 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:18:37.846 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.846 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-08-19T14:18:37.846 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:18:37.846 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.846 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:18:37.846 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6828", 2024-08-19T14:18:37.846 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2014939947 2024-08-19T14:18:37.846 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.846 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.846 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:18:37.846 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6829", 2024-08-19T14:18:37.847 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2014939947 2024-08-19T14:18:37.847 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:37.847 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:18:37.847 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.847 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.32:6825/2014939947", 2024-08-19T14:18:37.847 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.32:6827/2014939947", 2024-08-19T14:18:37.847 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.32:6831/2014939947", 2024-08-19T14:18:37.847 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.32:6829/2014939947", 2024-08-19T14:18:37.847 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-08-19T14:18:37.847 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-08-19T14:18:37.847 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-08-19T14:18:37.847 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:18:37.848 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.848 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.848 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 8, 2024-08-19T14:18:37.848 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "2bbcad36-0c74-4528-9fb9-a8812afa1662", 2024-08-19T14:18:37.848 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-08-19T14:18:37.848 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-08-19T14:18:37.848 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-08-19T14:18:37.848 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-08-19T14:18:37.848 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-08-19T14:18:37.848 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-08-19T14:18:37.848 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 65, 2024-08-19T14:18:37.849 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 281, 2024-08-19T14:18:37.849 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-08-19T14:18:37.849 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-08-19T14:18:37.849 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-08-19T14:18:37.849 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:18:37.849 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.849 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:18:37.849 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6800", 2024-08-19T14:18:37.849 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1445140892 2024-08-19T14:18:37.849 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.849 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.850 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:18:37.850 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6801", 2024-08-19T14:18:37.850 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1445140892 2024-08-19T14:18:37.850 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:37.850 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:18:37.850 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.850 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-08-19T14:18:37.850 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:18:37.850 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.850 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:18:37.850 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6802", 2024-08-19T14:18:37.850 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1445140892 2024-08-19T14:18:37.851 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.851 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.851 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:18:37.851 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6803", 2024-08-19T14:18:37.851 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1445140892 2024-08-19T14:18:37.851 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:37.851 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:18:37.852 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.852 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-08-19T14:18:37.852 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:18:37.852 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.852 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:18:37.852 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6806", 2024-08-19T14:18:37.852 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1445140892 2024-08-19T14:18:37.852 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.852 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.853 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:18:37.853 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6807", 2024-08-19T14:18:37.853 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1445140892 2024-08-19T14:18:37.853 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:37.853 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:18:37.853 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.853 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-08-19T14:18:37.853 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:18:37.853 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.853 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:18:37.854 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6804", 2024-08-19T14:18:37.854 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1445140892 2024-08-19T14:18:37.854 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.854 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.854 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:18:37.854 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6805", 2024-08-19T14:18:37.854 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1445140892 2024-08-19T14:18:37.854 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:37.855 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:18:37.855 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.855 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.45:6801/1445140892", 2024-08-19T14:18:37.855 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.45:6803/1445140892", 2024-08-19T14:18:37.855 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.45:6807/1445140892", 2024-08-19T14:18:37.855 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.45:6805/1445140892", 2024-08-19T14:18:37.856 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-08-19T14:18:37.856 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-08-19T14:18:37.856 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-08-19T14:18:37.856 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:18:37.856 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.856 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.856 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 9, 2024-08-19T14:18:37.856 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "c8ee4277-3bf9-46bf-9cc0-152c1c4b9508", 2024-08-19T14:18:37.856 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-08-19T14:18:37.856 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-08-19T14:18:37.856 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 0.95001220703125, 2024-08-19T14:18:37.856 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-08-19T14:18:37.856 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-08-19T14:18:37.856 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-08-19T14:18:37.857 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 74, 2024-08-19T14:18:37.857 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 276, 2024-08-19T14:18:37.857 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-08-19T14:18:37.857 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-08-19T14:18:37.857 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-08-19T14:18:37.857 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:18:37.857 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.857 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:18:37.857 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6808", 2024-08-19T14:18:37.857 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1213787169 2024-08-19T14:18:37.857 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.857 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.857 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:18:37.857 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6809", 2024-08-19T14:18:37.858 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1213787169 2024-08-19T14:18:37.858 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:37.858 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:18:37.858 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.858 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-08-19T14:18:37.858 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:18:37.858 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.858 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:18:37.858 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6810", 2024-08-19T14:18:37.858 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1213787169 2024-08-19T14:18:37.858 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.858 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.858 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:18:37.859 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6811", 2024-08-19T14:18:37.859 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1213787169 2024-08-19T14:18:37.859 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:37.859 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:18:37.859 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.859 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-08-19T14:18:37.859 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:18:37.859 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.859 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:18:37.859 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6814", 2024-08-19T14:18:37.859 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1213787169 2024-08-19T14:18:37.859 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.859 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.859 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:18:37.859 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6815", 2024-08-19T14:18:37.860 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1213787169 2024-08-19T14:18:37.860 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:37.860 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:18:37.860 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.860 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-08-19T14:18:37.860 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:18:37.860 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.860 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:18:37.860 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6812", 2024-08-19T14:18:37.860 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1213787169 2024-08-19T14:18:37.860 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.860 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.860 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:18:37.861 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6813", 2024-08-19T14:18:37.861 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1213787169 2024-08-19T14:18:37.861 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:37.861 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:18:37.861 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.861 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.45:6809/1213787169", 2024-08-19T14:18:37.861 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.45:6811/1213787169", 2024-08-19T14:18:37.861 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.45:6815/1213787169", 2024-08-19T14:18:37.861 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.45:6813/1213787169", 2024-08-19T14:18:37.861 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-08-19T14:18:37.861 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-08-19T14:18:37.861 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-08-19T14:18:37.861 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:18:37.861 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.862 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.862 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 10, 2024-08-19T14:18:37.862 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "29031395-5f5c-40f8-ac90-9f6ac48e5c9f", 2024-08-19T14:18:37.862 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-08-19T14:18:37.863 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 0, 2024-08-19T14:18:37.863 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 0, 2024-08-19T14:18:37.863 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-08-19T14:18:37.863 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-08-19T14:18:37.863 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-08-19T14:18:37.863 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 80, 2024-08-19T14:18:37.863 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 115, 2024-08-19T14:18:37.865 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-08-19T14:18:37.866 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-08-19T14:18:37.866 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-08-19T14:18:37.866 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:18:37.866 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.866 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:18:37.866 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6816", 2024-08-19T14:18:37.866 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2461824621 2024-08-19T14:18:37.866 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.866 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.866 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:18:37.866 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6817", 2024-08-19T14:18:37.866 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2461824621 2024-08-19T14:18:37.866 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:37.867 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:18:37.867 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.867 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-08-19T14:18:37.867 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:18:37.867 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.867 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:18:37.867 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6818", 2024-08-19T14:18:37.867 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2461824621 2024-08-19T14:18:37.867 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.867 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.867 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:18:37.867 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6819", 2024-08-19T14:18:37.867 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2461824621 2024-08-19T14:18:37.867 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:37.868 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:18:37.868 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.868 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-08-19T14:18:37.868 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:18:37.868 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.868 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:18:37.868 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6822", 2024-08-19T14:18:37.868 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2461824621 2024-08-19T14:18:37.868 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.868 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.868 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:18:37.868 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6823", 2024-08-19T14:18:37.868 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2461824621 2024-08-19T14:18:37.868 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:37.869 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:18:37.869 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.869 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-08-19T14:18:37.869 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:18:37.869 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.869 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:18:37.869 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6820", 2024-08-19T14:18:37.869 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2461824621 2024-08-19T14:18:37.869 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.869 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.869 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:18:37.869 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6821", 2024-08-19T14:18:37.869 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2461824621 2024-08-19T14:18:37.869 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:37.870 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:18:37.870 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.870 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.45:6817/2461824621", 2024-08-19T14:18:37.870 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.45:6819/2461824621", 2024-08-19T14:18:37.870 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.45:6823/2461824621", 2024-08-19T14:18:37.870 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.45:6821/2461824621", 2024-08-19T14:18:37.870 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-08-19T14:18:37.870 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-08-19T14:18:37.870 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-08-19T14:18:37.871 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:18:37.871 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.871 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.871 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 11, 2024-08-19T14:18:37.871 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "5ef0db4b-0fef-4ea0-a19b-d3677e082c29", 2024-08-19T14:18:37.871 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-08-19T14:18:37.871 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-08-19T14:18:37.871 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-08-19T14:18:37.871 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-08-19T14:18:37.871 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-08-19T14:18:37.871 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-08-19T14:18:37.871 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 86, 2024-08-19T14:18:37.871 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 281, 2024-08-19T14:18:37.871 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-08-19T14:18:37.871 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-08-19T14:18:37.872 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-08-19T14:18:37.872 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:18:37.872 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.872 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:18:37.872 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6824", 2024-08-19T14:18:37.872 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2438396298 2024-08-19T14:18:37.872 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.872 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.872 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:18:37.872 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6825", 2024-08-19T14:18:37.872 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2438396298 2024-08-19T14:18:37.872 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:37.872 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:18:37.872 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.873 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-08-19T14:18:37.873 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:18:37.873 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.873 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:18:37.873 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6826", 2024-08-19T14:18:37.873 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2438396298 2024-08-19T14:18:37.873 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.873 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.873 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:18:37.873 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6827", 2024-08-19T14:18:37.873 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2438396298 2024-08-19T14:18:37.873 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:37.874 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:18:37.874 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.874 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-08-19T14:18:37.874 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:18:37.874 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.874 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:18:37.874 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6830", 2024-08-19T14:18:37.874 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2438396298 2024-08-19T14:18:37.874 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.874 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.874 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:18:37.874 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6831", 2024-08-19T14:18:37.874 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2438396298 2024-08-19T14:18:37.874 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:37.875 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:18:37.875 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.875 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-08-19T14:18:37.875 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:18:37.875 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.875 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:18:37.875 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6828", 2024-08-19T14:18:37.875 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2438396298 2024-08-19T14:18:37.875 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.875 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.875 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:18:37.875 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6829", 2024-08-19T14:18:37.875 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2438396298 2024-08-19T14:18:37.875 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:37.876 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:18:37.876 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.876 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.45:6825/2438396298", 2024-08-19T14:18:37.876 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.45:6827/2438396298", 2024-08-19T14:18:37.876 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.45:6831/2438396298", 2024-08-19T14:18:37.876 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.45:6829/2438396298", 2024-08-19T14:18:37.876 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-08-19T14:18:37.876 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-08-19T14:18:37.876 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-08-19T14:18:37.876 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:18:37.876 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:37.876 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-08-19T14:18:37.877 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_xinfo": [ 2024-08-19T14:18:37.877 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.877 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 0, 2024-08-19T14:18:37.877 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-08-19T14:18:37.877 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-08-19T14:18:37.877 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-08-19T14:18:37.877 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-08-19T14:18:37.877 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-08-19T14:18:37.877 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-08-19T14:17:25.275894+0000", 2024-08-19T14:18:37.877 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-08-19T14:18:37.878 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.878 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.878 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 1, 2024-08-19T14:18:37.878 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-08-19T14:18:37.878 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-08-19T14:18:37.878 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-08-19T14:18:37.878 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-08-19T14:18:37.878 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-08-19T14:18:37.878 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-08-19T14:17:23.848389+0000", 2024-08-19T14:18:37.878 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-08-19T14:18:37.878 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.878 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.878 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 2, 2024-08-19T14:18:37.878 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-08-19T14:18:37.879 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-08-19T14:18:37.879 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-08-19T14:18:37.879 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-08-19T14:18:37.879 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 65536, 2024-08-19T14:18:37.879 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-08-19T14:17:14.130303+0000", 2024-08-19T14:18:37.879 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-08-19T14:18:37.879 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.879 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.879 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 3, 2024-08-19T14:18:37.880 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-08-19T14:18:37.880 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-08-19T14:18:37.880 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-08-19T14:18:37.880 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-08-19T14:18:37.880 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-08-19T14:18:37.880 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-08-19T14:18:36.688561+0000", 2024-08-19T14:18:37.880 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-08-19T14:18:37.880 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.880 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.880 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 4, 2024-08-19T14:18:37.880 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-08-19T14:18:37.880 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-08-19T14:18:37.880 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-08-19T14:18:37.880 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-08-19T14:18:37.881 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-08-19T14:18:37.881 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-08-19T14:18:35.608744+0000", 2024-08-19T14:18:37.881 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-08-19T14:18:37.881 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.881 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.881 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 5, 2024-08-19T14:18:37.881 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-08-19T14:18:37.881 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-08-19T14:18:37.881 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-08-19T14:18:37.881 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-08-19T14:18:37.881 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-08-19T14:18:37.882 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-08-19T14:18:32.988892+0000", 2024-08-19T14:18:37.882 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-08-19T14:18:37.882 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.882 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.882 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 6, 2024-08-19T14:18:37.882 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-08-19T14:18:37.882 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-08-19T14:18:37.882 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-08-19T14:18:37.882 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-08-19T14:18:37.882 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 65536, 2024-08-19T14:18:37.882 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-08-19T14:18:05.687348+0000", 2024-08-19T14:18:37.882 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-08-19T14:18:37.882 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.883 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.883 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 7, 2024-08-19T14:18:37.883 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-08-19T14:18:37.883 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-08-19T14:18:37.883 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-08-19T14:18:37.883 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-08-19T14:18:37.883 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-08-19T14:18:37.883 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-08-19T14:18:09.791947+0000", 2024-08-19T14:18:37.883 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-08-19T14:18:37.883 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.884 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.884 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 8, 2024-08-19T14:18:37.884 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-08-19T14:18:37.884 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-08-19T14:18:37.884 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-08-19T14:18:37.884 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-08-19T14:18:37.884 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-08-19T14:18:37.884 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-08-19T14:17:54.921918+0000", 2024-08-19T14:18:37.884 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-08-19T14:18:37.884 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.884 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.884 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 9, 2024-08-19T14:18:37.885 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-08-19T14:18:37.885 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-08-19T14:18:37.885 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-08-19T14:18:37.885 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-08-19T14:18:37.885 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-08-19T14:18:37.885 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-08-19T14:18:22.053948+0000", 2024-08-19T14:18:37.885 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-08-19T14:18:37.885 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.885 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.885 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 10, 2024-08-19T14:18:37.886 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-08-19T14:18:37.886 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-08-19T14:18:37.886 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-08-19T14:18:37.886 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-08-19T14:18:37.886 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 65536, 2024-08-19T14:18:37.886 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-08-19T14:17:24.968756+0000", 2024-08-19T14:18:37.886 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-08-19T14:18:37.886 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.886 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.887 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 11, 2024-08-19T14:18:37.887 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-08-19T14:18:37.887 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-08-19T14:18:37.887 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-08-19T14:18:37.887 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-08-19T14:18:37.887 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-08-19T14:18:37.887 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-08-19T14:18:05.893546+0000", 2024-08-19T14:18:37.887 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-08-19T14:18:37.887 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:37.888 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-08-19T14:18:37.888 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_upmap": [ 2024-08-19T14:18:37.888 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.888 INFO:teuthology.orchestra.run.smithi031.stdout: "pgid": "1.0", 2024-08-19T14:18:37.888 INFO:teuthology.orchestra.run.smithi031.stdout: "osds": [ 2024-08-19T14:18:37.888 INFO:teuthology.orchestra.run.smithi031.stdout: 2, 2024-08-19T14:18:37.888 INFO:teuthology.orchestra.run.smithi031.stdout: 9, 2024-08-19T14:18:37.888 INFO:teuthology.orchestra.run.smithi031.stdout: 0 2024-08-19T14:18:37.888 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:18:37.888 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.889 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:18:37.889 INFO:teuthology.orchestra.run.smithi031.stdout: "pgid": "3.a", 2024-08-19T14:18:37.889 INFO:teuthology.orchestra.run.smithi031.stdout: "osds": [ 2024-08-19T14:18:37.889 INFO:teuthology.orchestra.run.smithi031.stdout: 6, 2024-08-19T14:18:37.889 INFO:teuthology.orchestra.run.smithi031.stdout: 1, 2024-08-19T14:18:37.889 INFO:teuthology.orchestra.run.smithi031.stdout: 4 2024-08-19T14:18:37.889 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:18:37.889 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:37.889 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-08-19T14:18:37.889 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_upmap_items": [], 2024-08-19T14:18:37.890 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_upmap_primaries": [], 2024-08-19T14:18:37.890 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_temp": [], 2024-08-19T14:18:37.890 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_temp": [], 2024-08-19T14:18:37.890 INFO:teuthology.orchestra.run.smithi031.stdout: "blocklist": { 2024-08-19T14:18:37.890 INFO:teuthology.orchestra.run.smithi031.stdout: "172.21.15.31:0/1451745969": "2024-08-20T14:06:13.660657+0000", 2024-08-19T14:18:37.890 INFO:teuthology.orchestra.run.smithi031.stdout: "172.21.15.31:0/1515698918": "2024-08-20T14:06:13.660657+0000", 2024-08-19T14:18:37.890 INFO:teuthology.orchestra.run.smithi031.stdout: "172.21.15.31:6801/3297539242": "2024-08-20T14:06:13.660657+0000", 2024-08-19T14:18:37.890 INFO:teuthology.orchestra.run.smithi031.stdout: "172.21.15.31:6800/3297539242": "2024-08-20T14:06:13.660657+0000", 2024-08-19T14:18:37.890 INFO:teuthology.orchestra.run.smithi031.stdout: "172.21.15.31:0/2217423818": "2024-08-20T14:05:48.827576+0000", 2024-08-19T14:18:37.890 INFO:teuthology.orchestra.run.smithi031.stdout: "172.21.15.31:0/3353498281": "2024-08-20T14:06:13.660657+0000", 2024-08-19T14:18:37.890 INFO:teuthology.orchestra.run.smithi031.stdout: "172.21.15.31:0/4013594220": "2024-08-20T14:05:48.827576+0000", 2024-08-19T14:18:37.890 INFO:teuthology.orchestra.run.smithi031.stdout: "172.21.15.31:0/1654518065": "2024-08-20T14:05:48.827576+0000", 2024-08-19T14:18:37.890 INFO:teuthology.orchestra.run.smithi031.stdout: "172.21.15.31:6801/2368821258": "2024-08-20T14:05:48.827576+0000", 2024-08-19T14:18:37.890 INFO:teuthology.orchestra.run.smithi031.stdout: "172.21.15.31:6800/2368821258": "2024-08-20T14:05:48.827576+0000" 2024-08-19T14:18:37.891 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.891 INFO:teuthology.orchestra.run.smithi031.stdout: "range_blocklist": {}, 2024-08-19T14:18:37.891 INFO:teuthology.orchestra.run.smithi031.stdout: "erasure_code_profiles": { 2024-08-19T14:18:37.891 INFO:teuthology.orchestra.run.smithi031.stdout: "default": { 2024-08-19T14:18:37.891 INFO:teuthology.orchestra.run.smithi031.stdout: "crush-failure-domain": "osd", 2024-08-19T14:18:37.891 INFO:teuthology.orchestra.run.smithi031.stdout: "k": "2", 2024-08-19T14:18:37.891 INFO:teuthology.orchestra.run.smithi031.stdout: "m": "1", 2024-08-19T14:18:37.891 INFO:teuthology.orchestra.run.smithi031.stdout: "plugin": "jerasure", 2024-08-19T14:18:37.891 INFO:teuthology.orchestra.run.smithi031.stdout: "technique": "reed_sol_van" 2024-08-19T14:18:37.891 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:37.891 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:18:37.891 INFO:teuthology.orchestra.run.smithi031.stdout: "removed_snaps_queue": [], 2024-08-19T14:18:37.891 INFO:teuthology.orchestra.run.smithi031.stdout: "new_removed_snaps": [], 2024-08-19T14:18:37.891 INFO:teuthology.orchestra.run.smithi031.stdout: "new_purged_snaps": [], 2024-08-19T14:18:37.892 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_node_flags": {}, 2024-08-19T14:18:37.892 INFO:teuthology.orchestra.run.smithi031.stdout: "device_class_flags": {}, 2024-08-19T14:18:37.892 INFO:teuthology.orchestra.run.smithi031.stdout: "stretch_mode": { 2024-08-19T14:18:37.892 INFO:teuthology.orchestra.run.smithi031.stdout: "stretch_mode_enabled": false, 2024-08-19T14:18:37.892 INFO:teuthology.orchestra.run.smithi031.stdout: "stretch_bucket_count": 0, 2024-08-19T14:18:37.892 INFO:teuthology.orchestra.run.smithi031.stdout: "degraded_stretch_mode": 0, 2024-08-19T14:18:37.892 INFO:teuthology.orchestra.run.smithi031.stdout: "recovering_stretch_mode": 0, 2024-08-19T14:18:37.892 INFO:teuthology.orchestra.run.smithi031.stdout: "stretch_mode_bucket": 0 2024-08-19T14:18:37.892 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:18:37.892 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:18:37.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:37.896 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:37.896 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:18:37.998 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:18:38.025 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:38.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:38.026 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:18:38.100 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:18:38.101 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:18:38.126 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:18:38.132 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:18:38.133 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:18:38.158 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:38.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:38.159 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:18:38.162 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:18:38.162 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:18:38.195 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:18:38.195 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:18:38.231 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:18:38.231 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:18:38.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:37 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:18:38.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:37 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:18:38.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:37 smithi032 ceph-mon[20004]: osdmap e285: 12 total, 12 up, 9 in 2024-08-19T14:18:38.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:37 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:18:38.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:37 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:18:38.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:37 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3752865995' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-08-19T14:18:38.259 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:18:38.263 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:18:38.264 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:18:38.264 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:37 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:18:38.264 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:37 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:18:38.264 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:37 smithi031 ceph-mon[16336]: osdmap e285: 12 total, 12 up, 9 in 2024-08-19T14:18:38.264 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:37 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:18:38.264 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:37 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:18:38.265 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:37 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3752865995' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-08-19T14:18:38.290 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:38.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:38.291 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:18:38.295 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:18:38.295 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:18:38.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:37 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:18:38.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:37 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:18:38.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:37 smithi045 ceph-mon[20143]: osdmap e285: 12 total, 12 up, 9 in 2024-08-19T14:18:38.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:37 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:18:38.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:37 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:18:38.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:37 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3752865995' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-08-19T14:18:38.328 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:18:38.328 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:18:38.359 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:18:38.360 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:18:38.389 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:18:38.389 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:18:38.392 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:18:38.418 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:38.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:38.419 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:18:38.420 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:18:38.420 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:18:38.451 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:18:38.451 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:18:38.520 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:18:38.552 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:38.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:38.553 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:18:38.654 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:18:38.680 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:38.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:38.681 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:18:38.782 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:18:38.817 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:38.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:38.818 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:18:38.919 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:18:38.941 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:38.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:38.942 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:18:39.043 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:18:39.069 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:39.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:39.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:18:39.172 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:18:39.208 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:39.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:39.210 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:18:39.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:39 smithi032 ceph-mon[20004]: pgmap v647: 67 pgs: 67 active+clean; 7.7 GiB data, 26 GiB used, 778 GiB / 805 GiB avail; 1.5 KiB/s rd, 52 MiB/s wr, 1.67k op/s; 22 KiB/s, 0 objects/s recovering 2024-08-19T14:18:39.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:39 smithi032 ceph-mon[20004]: osdmap e286: 12 total, 12 up, 9 in 2024-08-19T14:18:39.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:39 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2842964366' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:18:39.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:39 smithi045 ceph-mon[20143]: pgmap v647: 67 pgs: 67 active+clean; 7.7 GiB data, 26 GiB used, 778 GiB / 805 GiB avail; 1.5 KiB/s rd, 52 MiB/s wr, 1.67k op/s; 22 KiB/s, 0 objects/s recovering 2024-08-19T14:18:39.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:39 smithi045 ceph-mon[20143]: osdmap e286: 12 total, 12 up, 9 in 2024-08-19T14:18:39.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:39 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2842964366' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:18:39.311 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:18:39.342 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:39.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:39.343 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:18:39.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:39 smithi031 ceph-mon[16336]: pgmap v647: 67 pgs: 67 active+clean; 7.7 GiB data, 26 GiB used, 778 GiB / 805 GiB avail; 1.5 KiB/s rd, 52 MiB/s wr, 1.67k op/s; 22 KiB/s, 0 objects/s recovering 2024-08-19T14:18:39.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:39 smithi031 ceph-mon[16336]: osdmap e286: 12 total, 12 up, 9 in 2024-08-19T14:18:39.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:39 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2842964366' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:18:39.444 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:18:39.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:39.467 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:39.467 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:18:39.569 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:18:39.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:39.604 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:39.604 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:18:39.706 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:18:39.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:39.739 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:39.739 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:18:39.840 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:18:39.872 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:39.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:39.873 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:18:39.974 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:18:40.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:40.008 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:40.008 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:18:40.109 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:18:40.141 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:40.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:40.142 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:18:40.243 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:18:40.275 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:40.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:40.276 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:18:40.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:40 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:18:40.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:40 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:18:40.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:40 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:18:40.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:40 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:18:40.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:40 smithi045 ceph-mon[20143]: osdmap e287: 12 total, 12 up, 9 in 2024-08-19T14:18:40.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:40 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:18:40.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:40 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:18:40.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:40 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:18:40.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:40 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:18:40.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:40 smithi031 ceph-mon[16336]: osdmap e287: 12 total, 12 up, 9 in 2024-08-19T14:18:40.377 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:18:40.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:40.410 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:40.410 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:18:40.449 INFO:tasks.thrashosds.thrasher:j is {'epoch': 285, 'fsid': 'ed7d1c6e-5e33-11ef-bccf-c7b262605968', 'created': '2024-08-19T14:05:22.842169+0000', 'modified': '2024-08-19T14:18:36.974640+0000', 'last_up_change': '2024-08-19T14:11:39.491234+0000', 'last_in_change': '2024-08-19T14:16:38.207673+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 27, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-08-19T14:08:15.966221+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 31, 'pg_placement_num': 31, 'pg_placement_num_target': 31, 'pg_num_target': 31, 'pg_num_pending': 31, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '278', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '276', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.319999933242798, 'score_stable': 2.319999933242798, 'optimal_score': 0.75, 'raw_score_acting': 1.7400000095367432, 'raw_score_stable': 1.7400000095367432, 'primary_affinity_weighted': 0.9228158593177795, 'average_primary_affinity': 0.9200000166893005, 'average_primary_affinity_weighted': 0.9200000166893005}}, {'pool': 3, 'pool_name': 'unique_pool_1', 'create_time': '2024-08-19T14:16:55.393520+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 36, 'pg_placement_num': 36, 'pg_placement_num_target': 36, 'pg_num_target': 36, 'pg_num_pending': 36, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '246', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '244', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 0.75, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9228158593177795, 'average_primary_affinity': 0.9200000166893005, 'average_primary_affinity_weighted': 0.9200000166893005}}], 'osds': [{'osd': 0, 'uuid': 'd7a0fff3-23c6-4ee3-a275-a844862aabfb', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 281, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6802', 'nonce': 1108945097}, {'type': 'v1', 'addr': '172.21.15.31:6803', 'nonce': 1108945097}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6804', 'nonce': 1108945097}, {'type': 'v1', 'addr': '172.21.15.31:6805', 'nonce': 1108945097}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6808', 'nonce': 1108945097}, {'type': 'v1', 'addr': '172.21.15.31:6809', 'nonce': 1108945097}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6806', 'nonce': 1108945097}, {'type': 'v1', 'addr': '172.21.15.31:6807', 'nonce': 1108945097}]}, 'public_addr': '172.21.15.31:6803/1108945097', 'cluster_addr': '172.21.15.31:6805/1108945097', 'heartbeat_back_addr': '172.21.15.31:6809/1108945097', 'heartbeat_front_addr': '172.21.15.31:6807/1108945097', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '145d5bb4-74dd-440f-bcc9-7e9ebda9c5d6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 281, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6810', 'nonce': 3236186806}, {'type': 'v1', 'addr': '172.21.15.31:6811', 'nonce': 3236186806}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6812', 'nonce': 3236186806}, {'type': 'v1', 'addr': '172.21.15.31:6813', 'nonce': 3236186806}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6816', 'nonce': 3236186806}, {'type': 'v1', 'addr': '172.21.15.31:6817', 'nonce': 3236186806}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6814', 'nonce': 3236186806}, {'type': 'v1', 'addr': '172.21.15.31:6815', 'nonce': 3236186806}]}, 'public_addr': '172.21.15.31:6811/3236186806', 'cluster_addr': '172.21.15.31:6813/3236186806', 'heartbeat_back_addr': '172.21.15.31:6817/3236186806', 'heartbeat_front_addr': '172.21.15.31:6815/3236186806', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '376c18d0-4768-46a0-bb60-e79c88ca909c', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 0.07379150390625, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 204, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6818', 'nonce': 4045553387}, {'type': 'v1', 'addr': '172.21.15.31:6819', 'nonce': 4045553387}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6820', 'nonce': 4045553387}, {'type': 'v1', 'addr': '172.21.15.31:6821', 'nonce': 4045553387}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6824', 'nonce': 4045553387}, {'type': 'v1', 'addr': '172.21.15.31:6825', 'nonce': 4045553387}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6822', 'nonce': 4045553387}, {'type': 'v1', 'addr': '172.21.15.31:6823', 'nonce': 4045553387}]}, 'public_addr': '172.21.15.31:6819/4045553387', 'cluster_addr': '172.21.15.31:6821/4045553387', 'heartbeat_back_addr': '172.21.15.31:6825/4045553387', 'heartbeat_front_addr': '172.21.15.31:6823/4045553387', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'a787ea9f-b9ba-450f-b388-614cfba90634', 'up': 1, 'in': 1, 'weight': 0.6437225341796875, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 280, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6826', 'nonce': 2288082716}, {'type': 'v1', 'addr': '172.21.15.31:6827', 'nonce': 2288082716}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6828', 'nonce': 2288082716}, {'type': 'v1', 'addr': '172.21.15.31:6829', 'nonce': 2288082716}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6832', 'nonce': 2288082716}, {'type': 'v1', 'addr': '172.21.15.31:6833', 'nonce': 2288082716}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6830', 'nonce': 2288082716}, {'type': 'v1', 'addr': '172.21.15.31:6831', 'nonce': 2288082716}]}, 'public_addr': '172.21.15.31:6827/2288082716', 'cluster_addr': '172.21.15.31:6829/2288082716', 'heartbeat_back_addr': '172.21.15.31:6833/2288082716', 'heartbeat_front_addr': '172.21.15.31:6831/2288082716', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '7859751a-12f5-4a57-a332-435360479d53', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 281, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6800', 'nonce': 3934739984}, {'type': 'v1', 'addr': '172.21.15.32:6801', 'nonce': 3934739984}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6802', 'nonce': 3934739984}, {'type': 'v1', 'addr': '172.21.15.32:6803', 'nonce': 3934739984}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6806', 'nonce': 3934739984}, {'type': 'v1', 'addr': '172.21.15.32:6807', 'nonce': 3934739984}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6804', 'nonce': 3934739984}, {'type': 'v1', 'addr': '172.21.15.32:6805', 'nonce': 3934739984}]}, 'public_addr': '172.21.15.32:6801/3934739984', 'cluster_addr': '172.21.15.32:6803/3934739984', 'heartbeat_back_addr': '172.21.15.32:6807/3934739984', 'heartbeat_front_addr': '172.21.15.32:6805/3934739984', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '017f3eef-ac60-4b82-8350-919017cb4ccd', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 40, 'up_thru': 281, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6808', 'nonce': 526388985}, {'type': 'v1', 'addr': '172.21.15.32:6809', 'nonce': 526388985}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6810', 'nonce': 526388985}, {'type': 'v1', 'addr': '172.21.15.32:6811', 'nonce': 526388985}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6814', 'nonce': 526388985}, {'type': 'v1', 'addr': '172.21.15.32:6815', 'nonce': 526388985}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6812', 'nonce': 526388985}, {'type': 'v1', 'addr': '172.21.15.32:6813', 'nonce': 526388985}]}, 'public_addr': '172.21.15.32:6809/526388985', 'cluster_addr': '172.21.15.32:6811/526388985', 'heartbeat_back_addr': '172.21.15.32:6815/526388985', 'heartbeat_front_addr': '172.21.15.32:6813/526388985', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '44aa8229-0268-49d7-bd57-3bf1f384e5e6', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 48, 'up_thru': 126, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6816', 'nonce': 1131035844}, {'type': 'v1', 'addr': '172.21.15.32:6817', 'nonce': 1131035844}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6818', 'nonce': 1131035844}, {'type': 'v1', 'addr': '172.21.15.32:6819', 'nonce': 1131035844}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6822', 'nonce': 1131035844}, {'type': 'v1', 'addr': '172.21.15.32:6823', 'nonce': 1131035844}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6820', 'nonce': 1131035844}, {'type': 'v1', 'addr': '172.21.15.32:6821', 'nonce': 1131035844}]}, 'public_addr': '172.21.15.32:6817/1131035844', 'cluster_addr': '172.21.15.32:6819/1131035844', 'heartbeat_back_addr': '172.21.15.32:6823/1131035844', 'heartbeat_front_addr': '172.21.15.32:6821/1131035844', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '263433e5-70b6-4192-86ff-dc2ad95927eb', 'up': 1, 'in': 1, 'weight': 0.95001220703125, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 57, 'up_thru': 281, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6824', 'nonce': 2014939947}, {'type': 'v1', 'addr': '172.21.15.32:6825', 'nonce': 2014939947}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6826', 'nonce': 2014939947}, {'type': 'v1', 'addr': '172.21.15.32:6827', 'nonce': 2014939947}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6830', 'nonce': 2014939947}, {'type': 'v1', 'addr': '172.21.15.32:6831', 'nonce': 2014939947}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6828', 'nonce': 2014939947}, {'type': 'v1', 'addr': '172.21.15.32:6829', 'nonce': 2014939947}]}, 'public_addr': '172.21.15.32:6825/2014939947', 'cluster_addr': '172.21.15.32:6827/2014939947', 'heartbeat_back_addr': '172.21.15.32:6831/2014939947', 'heartbeat_front_addr': '172.21.15.32:6829/2014939947', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': '2bbcad36-0c74-4528-9fb9-a8812afa1662', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 65, 'up_thru': 281, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6800', 'nonce': 1445140892}, {'type': 'v1', 'addr': '172.21.15.45:6801', 'nonce': 1445140892}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6802', 'nonce': 1445140892}, {'type': 'v1', 'addr': '172.21.15.45:6803', 'nonce': 1445140892}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6806', 'nonce': 1445140892}, {'type': 'v1', 'addr': '172.21.15.45:6807', 'nonce': 1445140892}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6804', 'nonce': 1445140892}, {'type': 'v1', 'addr': '172.21.15.45:6805', 'nonce': 1445140892}]}, 'public_addr': '172.21.15.45:6801/1445140892', 'cluster_addr': '172.21.15.45:6803/1445140892', 'heartbeat_back_addr': '172.21.15.45:6807/1445140892', 'heartbeat_front_addr': '172.21.15.45:6805/1445140892', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': 'c8ee4277-3bf9-46bf-9cc0-152c1c4b9508', 'up': 1, 'in': 1, 'weight': 0.95001220703125, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 74, 'up_thru': 276, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6808', 'nonce': 1213787169}, {'type': 'v1', 'addr': '172.21.15.45:6809', 'nonce': 1213787169}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6810', 'nonce': 1213787169}, {'type': 'v1', 'addr': '172.21.15.45:6811', 'nonce': 1213787169}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6814', 'nonce': 1213787169}, {'type': 'v1', 'addr': '172.21.15.45:6815', 'nonce': 1213787169}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6812', 'nonce': 1213787169}, {'type': 'v1', 'addr': '172.21.15.45:6813', 'nonce': 1213787169}]}, 'public_addr': '172.21.15.45:6809/1213787169', 'cluster_addr': '172.21.15.45:6811/1213787169', 'heartbeat_back_addr': '172.21.15.45:6815/1213787169', 'heartbeat_front_addr': '172.21.15.45:6813/1213787169', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '29031395-5f5c-40f8-ac90-9f6ac48e5c9f', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 80, 'up_thru': 115, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6816', 'nonce': 2461824621}, {'type': 'v1', 'addr': '172.21.15.45:6817', 'nonce': 2461824621}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6818', 'nonce': 2461824621}, {'type': 'v1', 'addr': '172.21.15.45:6819', 'nonce': 2461824621}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6822', 'nonce': 2461824621}, {'type': 'v1', 'addr': '172.21.15.45:6823', 'nonce': 2461824621}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6820', 'nonce': 2461824621}, {'type': 'v1', 'addr': '172.21.15.45:6821', 'nonce': 2461824621}]}, 'public_addr': '172.21.15.45:6817/2461824621', 'cluster_addr': '172.21.15.45:6819/2461824621', 'heartbeat_back_addr': '172.21.15.45:6823/2461824621', 'heartbeat_front_addr': '172.21.15.45:6821/2461824621', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': '5ef0db4b-0fef-4ea0-a19b-d3677e082c29', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 86, 'up_thru': 281, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6824', 'nonce': 2438396298}, {'type': 'v1', 'addr': '172.21.15.45:6825', 'nonce': 2438396298}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6826', 'nonce': 2438396298}, {'type': 'v1', 'addr': '172.21.15.45:6827', 'nonce': 2438396298}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6830', 'nonce': 2438396298}, {'type': 'v1', 'addr': '172.21.15.45:6831', 'nonce': 2438396298}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6828', 'nonce': 2438396298}, {'type': 'v1', 'addr': '172.21.15.45:6829', 'nonce': 2438396298}]}, 'public_addr': '172.21.15.45:6825/2438396298', 'cluster_addr': '172.21.15.45:6827/2438396298', 'heartbeat_back_addr': '172.21.15.45:6831/2438396298', 'heartbeat_front_addr': '172.21.15.45:6829/2438396298', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-19T14:17:25.275894+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-19T14:17:23.848389+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-08-19T14:17:14.130303+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-19T14:18:36.688561+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-19T14:18:35.608744+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-19T14:18:32.988892+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-08-19T14:18:05.687348+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-19T14:18:09.791947+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-19T14:17:54.921918+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-19T14:18:22.053948+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-08-19T14:17:24.968756+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-19T14:18:05.893546+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '1.0', 'osds': [2, 9, 0]}, {'pgid': '3.a', 'osds': [6, 1, 4]}], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.31:0/1451745969': '2024-08-20T14:06:13.660657+0000', '172.21.15.31:0/1515698918': '2024-08-20T14:06:13.660657+0000', '172.21.15.31:6801/3297539242': '2024-08-20T14:06:13.660657+0000', '172.21.15.31:6800/3297539242': '2024-08-20T14:06:13.660657+0000', '172.21.15.31:0/2217423818': '2024-08-20T14:05:48.827576+0000', '172.21.15.31:0/3353498281': '2024-08-20T14:06:13.660657+0000', '172.21.15.31:0/4013594220': '2024-08-20T14:05:48.827576+0000', '172.21.15.31:0/1654518065': '2024-08-20T14:05:48.827576+0000', '172.21.15.31:6801/2368821258': '2024-08-20T14:05:48.827576+0000', '172.21.15.31:6800/2368821258': '2024-08-20T14:05:48.827576+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-08-19T14:18:40.449 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-19T14:18:40.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:40 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:18:40.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:40 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:18:40.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:40 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:18:40.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:40 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:18:40.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:40 smithi032 ceph-mon[20004]: osdmap e287: 12 total, 12 up, 9 in 2024-08-19T14:18:40.511 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:18:40.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:40.580 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:40.580 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:18:40.681 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:18:40.711 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:40.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:40.712 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:18:40.813 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:18:40.845 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:40.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:40.846 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:18:40.947 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:18:40.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:40.983 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:40.983 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:18:41.084 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:18:41.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:41.109 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:41.109 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:18:41.210 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:18:41.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:41.246 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:41.247 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:18:41.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:41 smithi045 ceph-mon[20143]: pgmap v650: 67 pgs: 67 active+clean; 7.4 GiB data, 25 GiB used, 780 GiB / 805 GiB avail; 8.0 MiB/s wr, 2.12k op/s 2024-08-19T14:18:41.347 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:18:41.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:41 smithi031 ceph-mon[16336]: pgmap v650: 67 pgs: 67 active+clean; 7.4 GiB data, 25 GiB used, 780 GiB / 805 GiB avail; 8.0 MiB/s wr, 2.12k op/s 2024-08-19T14:18:41.381 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:41.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:41.383 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:18:41.484 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:18:41.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:41 smithi032 ceph-mon[20004]: pgmap v650: 67 pgs: 67 active+clean; 7.4 GiB data, 25 GiB used, 780 GiB / 805 GiB avail; 8.0 MiB/s wr, 2.12k op/s 2024-08-19T14:18:41.517 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:41.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:41.519 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:18:41.620 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:18:41.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:41.652 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:41.653 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:18:41.753 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:18:41.788 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:41.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:41.789 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:18:41.890 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:18:41.923 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:41.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:41.924 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:18:42.025 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:18:42.061 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:42.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:42.062 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:18:42.109 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:18:42.109 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:18:42.164 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:18:42.202 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:42.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:42.203 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:18:42.304 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:18:42.337 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:42.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:42.338 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:18:42.439 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:18:42.476 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:42.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:42.477 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:18:42.578 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:18:42.609 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:42.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:42.610 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:18:42.711 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:18:42.740 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:42.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:42.741 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:18:42.842 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:18:42.938 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-08-19T14:18:42.938 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:18:42.947 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:18:42.985 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:42.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:42.986 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:18:43.087 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:18:43.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:43 smithi031 ceph-mon[16336]: pgmap v651: 67 pgs: 67 active+clean; 7.0 GiB data, 24 GiB used, 780 GiB / 805 GiB avail; 5.4 MiB/s wr, 2.36k op/s 2024-08-19T14:18:43.117 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:43.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:43.119 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:18:43.220 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:18:43.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:43 smithi032 ceph-mon[20004]: pgmap v651: 67 pgs: 67 active+clean; 7.0 GiB data, 24 GiB used, 780 GiB / 805 GiB avail; 5.4 MiB/s wr, 2.36k op/s 2024-08-19T14:18:43.271 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:43.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:43.272 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:18:43.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:43 smithi045 ceph-mon[20143]: pgmap v651: 67 pgs: 67 active+clean; 7.0 GiB data, 24 GiB used, 780 GiB / 805 GiB avail; 5.4 MiB/s wr, 2.36k op/s 2024-08-19T14:18:43.341 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:18:43.373 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:18:43.417 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:43.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:43.418 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:18:43.519 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:18:43.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:43.540 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:43.541 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:18:43.641 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:18:43.677 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:43.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:43.678 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:18:43.779 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:18:43.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:43.810 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:43.810 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:18:43.885 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:18:43.911 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:18:43.963 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:18:43.963 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:18:43.966 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:43.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:43.967 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:18:44.037 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is set 2024-08-19T14:18:44.068 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:18:44.098 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:44.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:44.099 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:18:44.200 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:18:44.235 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:44.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:44.236 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:18:44.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:44 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2373292137' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:18:44.337 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:18:44.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:44 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2373292137' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:18:44.373 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:44.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:44.375 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:18:44.380 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:18:44.381 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-08-19T14:18:44.475 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:18:44.502 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:44.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:44.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:18:44.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:44 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2373292137' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:18:44.604 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:18:44.634 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:44.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:44.635 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:18:44.736 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:18:44.769 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:44.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:44.771 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:18:44.871 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:18:44.905 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:44.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:44.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:18:45.007 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:18:45.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:45.038 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:45.038 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:18:45.052 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-08-19T14:18:45.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:45 smithi031 ceph-mon[16336]: pgmap v652: 67 pgs: 67 active+clean; 6.8 GiB data, 23 GiB used, 781 GiB / 805 GiB avail; 0 B/s wr, 2.16k op/s 2024-08-19T14:18:45.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:45 smithi031 ceph-mon[16336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:18:45.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:45 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2373292137' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:18:45.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:45 smithi031 ceph-mon[16336]: osdmap e288: 12 total, 12 up, 9 in 2024-08-19T14:18:45.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:45 smithi031 ceph-mon[16336]: from='client.25259 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:18:45.139 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:18:45.168 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:45.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:45.169 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:18:45.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:45 smithi032 ceph-mon[20004]: pgmap v652: 67 pgs: 67 active+clean; 6.8 GiB data, 23 GiB used, 781 GiB / 805 GiB avail; 0 B/s wr, 2.16k op/s 2024-08-19T14:18:45.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:45 smithi032 ceph-mon[20004]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:18:45.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:45 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2373292137' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:18:45.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:45 smithi032 ceph-mon[20004]: osdmap e288: 12 total, 12 up, 9 in 2024-08-19T14:18:45.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:45 smithi032 ceph-mon[20004]: from='client.25259 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:18:45.269 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:18:45.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:45 smithi045 ceph-mon[20143]: pgmap v652: 67 pgs: 67 active+clean; 6.8 GiB data, 23 GiB used, 781 GiB / 805 GiB avail; 0 B/s wr, 2.16k op/s 2024-08-19T14:18:45.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:45 smithi045 ceph-mon[20143]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:18:45.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:45 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2373292137' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:18:45.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:45 smithi045 ceph-mon[20143]: osdmap e288: 12 total, 12 up, 9 in 2024-08-19T14:18:45.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:45 smithi045 ceph-mon[20143]: from='client.25259 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:18:45.300 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:45.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:45.301 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:18:45.402 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:18:45.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:45.424 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:45.424 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:18:45.428 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:18:45.525 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:18:45.551 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:45.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:45.553 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:18:45.654 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:18:45.686 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:45.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:45.687 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:18:45.788 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:18:45.823 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:45.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:45.824 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:18:45.925 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:18:45.954 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:45.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:45.955 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:18:45.973 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":653,"stamp":"2024-08-19T14:18:44.034834+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7303356052,"num_objects":111435,"num_object_clones":0,"num_object_copies":334305,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":111435,"num_whiteouts":0,"num_read":21,"num_read_kb":20,"num_write":34260,"num_write_kb":927616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":630451,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":222688,"ondisk_log_size":222688,"up":201,"acting":201,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":210,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":24331424,"kb_used_data":21698492,"kb_used_omap":14,"kb_used_meta":2632241,"kb_avail":819374944,"statfs":{"total":863955320832,"available":839039942656,"internally_reserved":0,"allocated":22219255808,"data_stored":22206999578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14353,"internal_metadata":2695415791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-968622080,"num_objects":-14780,"num_object_clones":0,"num_object_copies":-44340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-14780,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14780,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.204348"},"pg_stats":[{"pgid":"1.0","version":"70'139","reported_seq":781,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-08-19T14:18:39.062857+0000","last_change":"2024-08-19T14:18:25.872921+0000","last_active":"2024-08-19T14:18:39.062857+0000","last_peered":"2024-08-19T14:18:39.062857+0000","last_clean":"2024-08-19T14:18:39.062857+0000","last_became_active":"2024-08-19T14:18:13.807133+0000","last_became_peered":"2024-08-19T14:18:13.807133+0000","last_unstale":"2024-08-19T14:18:39.062857+0000","last_undegraded":"2024-08-19T14:18:39.062857+0000","last_fullsized":"2024-08-19T14:18:39.062857+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:18:25.872808+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:18:25.872808+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:33.201222+0000","scrub_duration":2157,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,9],"acting":[7,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"287'4932","reported_seq":5115,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-08-19T14:18:43.216269+0000","last_change":"2024-08-19T14:18:17.084657+0000","last_active":"2024-08-19T14:18:43.216269+0000","last_peered":"2024-08-19T14:18:43.216269+0000","last_clean":"2024-08-19T14:18:43.216269+0000","last_became_active":"2024-08-19T14:17:34.968554+0000","last_became_peered":"2024-08-19T14:17:34.968554+0000","last_unstale":"2024-08-19T14:18:43.216269+0000","last_undegraded":"2024-08-19T14:18:43.216269+0000","last_fullsized":"2024-08-19T14:18:43.216269+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'4295","last_scrub_stamp":"2024-08-19T14:18:17.084447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:17.084447+0000","objects_scrubbed":1731,"log_size":4932,"log_dups_size":0,"ondisk_log_size":4932,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:29.607335+0000","scrub_duration":357,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107806720,"num_objects":1645,"num_object_clones":0,"num_object_copies":4935,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1645,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":642,"num_write_kb":17792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,5],"acting":[9,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.3","version":"70'139","reported_seq":679,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-08-19T14:18:39.055516+0000","last_change":"2024-08-19T14:18:21.708466+0000","last_active":"2024-08-19T14:18:39.055516+0000","last_peered":"2024-08-19T14:18:39.055516+0000","last_clean":"2024-08-19T14:18:39.055516+0000","last_became_active":"2024-08-19T14:18:13.804721+0000","last_became_peered":"2024-08-19T14:18:13.804721+0000","last_unstale":"2024-08-19T14:18:39.055516+0000","last_undegraded":"2024-08-19T14:18:39.055516+0000","last_fullsized":"2024-08-19T14:18:39.055516+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:18:21.708333+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:18:21.708333+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:45.972923+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"287'4681","reported_seq":4855,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-08-19T14:18:40.888806+0000","last_change":"2024-08-19T14:18:20.248300+0000","last_active":"2024-08-19T14:18:40.888806+0000","last_peered":"2024-08-19T14:18:40.888806+0000","last_clean":"2024-08-19T14:18:40.888806+0000","last_became_active":"2024-08-19T14:17:34.963892+0000","last_became_peered":"2024-08-19T14:17:34.963892+0000","last_unstale":"2024-08-19T14:18:40.888806+0000","last_undegraded":"2024-08-19T14:18:40.888806+0000","last_fullsized":"2024-08-19T14:18:40.888806+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'4227","last_scrub_stamp":"2024-08-19T14:18:20.248075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:20.248075+0000","objects_scrubbed":1786,"log_size":4681,"log_dups_size":0,"ondisk_log_size":4681,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:29.422356+0000","scrub_duration":285,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109838336,"num_objects":1676,"num_object_clones":0,"num_object_copies":5028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1676,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":456,"num_write_kb":11072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,11],"acting":[0,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"70'139","reported_seq":624,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-08-19T14:18:39.004512+0000","last_change":"2024-08-19T14:18:24.001903+0000","last_active":"2024-08-19T14:18:39.004512+0000","last_peered":"2024-08-19T14:18:39.004512+0000","last_clean":"2024-08-19T14:18:39.004512+0000","last_became_active":"2024-08-19T14:18:13.809344+0000","last_became_peered":"2024-08-19T14:18:13.809344+0000","last_unstale":"2024-08-19T14:18:39.004512+0000","last_undegraded":"2024-08-19T14:18:39.004512+0000","last_fullsized":"2024-08-19T14:18:39.004512+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:18:24.001801+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:18:24.001801+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:26.754447+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,8],"acting":[0,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"287'4764","reported_seq":4947,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-08-19T14:18:40.682193+0000","last_change":"2024-08-19T14:18:17.845313+0000","last_active":"2024-08-19T14:18:40.682193+0000","last_peered":"2024-08-19T14:18:40.682193+0000","last_clean":"2024-08-19T14:18:40.682193+0000","last_became_active":"2024-08-19T14:17:34.963390+0000","last_became_peered":"2024-08-19T14:17:34.963390+0000","last_unstale":"2024-08-19T14:18:40.682193+0000","last_undegraded":"2024-08-19T14:18:40.682193+0000","last_fullsized":"2024-08-19T14:18:40.682193+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'4224","last_scrub_stamp":"2024-08-19T14:18:17.845089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:17.845089+0000","objects_scrubbed":1755,"log_size":4764,"log_dups_size":0,"ondisk_log_size":4764,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:24.860639+0000","scrub_duration":426,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":115146752,"num_objects":1757,"num_object_clones":0,"num_object_copies":5271,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":548,"num_write_kb":17600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,1],"acting":[11,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1","version":"70'139","reported_seq":676,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-08-19T14:18:39.004842+0000","last_change":"2024-08-19T14:18:21.795775+0000","last_active":"2024-08-19T14:18:39.004842+0000","last_peered":"2024-08-19T14:18:39.004842+0000","last_clean":"2024-08-19T14:18:39.004842+0000","last_became_active":"2024-08-19T14:18:13.808505+0000","last_became_peered":"2024-08-19T14:18:13.808505+0000","last_unstale":"2024-08-19T14:18:39.004842+0000","last_undegraded":"2024-08-19T14:18:39.004842+0000","last_fullsized":"2024-08-19T14:18:39.004842+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:18:21.795647+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:18:21.795647+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:42.370696+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,11],"acting":[9,0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3","version":"287'4840","reported_seq":5023,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-08-19T14:18:43.013926+0000","last_change":"2024-08-19T14:18:20.238483+0000","last_active":"2024-08-19T14:18:43.013926+0000","last_peered":"2024-08-19T14:18:43.013926+0000","last_clean":"2024-08-19T14:18:43.013926+0000","last_became_active":"2024-08-19T14:17:34.968430+0000","last_became_peered":"2024-08-19T14:17:34.968430+0000","last_unstale":"2024-08-19T14:18:43.013926+0000","last_undegraded":"2024-08-19T14:18:43.013926+0000","last_fullsized":"2024-08-19T14:18:43.013926+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'4292","last_scrub_stamp":"2024-08-19T14:18:20.238336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:20.238336+0000","objects_scrubbed":1842,"log_size":4840,"log_dups_size":0,"ondisk_log_size":4840,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:38.217046+0000","scrub_duration":304,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110493696,"num_objects":1686,"num_object_clones":0,"num_object_copies":5058,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1686,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":548,"num_write_kb":12544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,11],"acting":[4,0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"70'139","reported_seq":597,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-08-19T14:18:39.057770+0000","last_change":"2024-08-19T14:18:27.204083+0000","last_active":"2024-08-19T14:18:39.057770+0000","last_peered":"2024-08-19T14:18:39.057770+0000","last_clean":"2024-08-19T14:18:39.057770+0000","last_became_active":"2024-08-19T14:18:26.878753+0000","last_became_peered":"2024-08-19T14:18:26.878753+0000","last_unstale":"2024-08-19T14:18:39.057770+0000","last_undegraded":"2024-08-19T14:18:39.057770+0000","last_fullsized":"2024-08-19T14:18:39.057770+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:18:27.203987+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:18:27.203987+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:56.868334+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"287'6248","reported_seq":6445,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-08-19T14:18:41.615176+0000","last_change":"2024-08-19T14:18:16.976479+0000","last_active":"2024-08-19T14:18:41.615176+0000","last_peered":"2024-08-19T14:18:41.615176+0000","last_clean":"2024-08-19T14:18:41.615176+0000","last_became_active":"2024-08-19T14:17:19.745911+0000","last_became_peered":"2024-08-19T14:17:19.745911+0000","last_unstale":"2024-08-19T14:18:41.615176+0000","last_undegraded":"2024-08-19T14:18:41.615176+0000","last_fullsized":"2024-08-19T14:18:41.615176+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'5041","last_scrub_stamp":"2024-08-19T14:18:16.976188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:16.976188+0000","objects_scrubbed":3541,"log_size":6248,"log_dups_size":0,"ondisk_log_size":6248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:20.500490+0000","scrub_duration":886,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":230096896,"num_objects":3511,"num_object_clones":0,"num_object_copies":10533,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1236,"num_write_kb":38592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"70'139","reported_seq":680,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-08-19T14:18:39.005037+0000","last_change":"2024-08-19T14:18:27.848950+0000","last_active":"2024-08-19T14:18:39.005037+0000","last_peered":"2024-08-19T14:18:39.005037+0000","last_clean":"2024-08-19T14:18:39.005037+0000","last_became_active":"2024-08-19T14:18:26.874902+0000","last_became_peered":"2024-08-19T14:18:26.874902+0000","last_unstale":"2024-08-19T14:18:39.005037+0000","last_undegraded":"2024-08-19T14:18:39.005037+0000","last_fullsized":"2024-08-19T14:18:39.005037+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:18:27.848870+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:27.848870+0000","last_clean_scrub_stamp":"2024-08-19T14:18:27.848870+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:29.542244+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,4],"acting":[9,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.4","version":"287'6426","reported_seq":6623,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-08-19T14:18:43.400911+0000","last_change":"2024-08-19T14:18:20.377814+0000","last_active":"2024-08-19T14:18:43.400911+0000","last_peered":"2024-08-19T14:18:43.400911+0000","last_clean":"2024-08-19T14:18:43.400911+0000","last_became_active":"2024-08-19T14:17:19.741977+0000","last_became_peered":"2024-08-19T14:17:19.741977+0000","last_unstale":"2024-08-19T14:18:43.400911+0000","last_undegraded":"2024-08-19T14:18:43.400911+0000","last_fullsized":"2024-08-19T14:18:43.400911+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'5234","last_scrub_stamp":"2024-08-19T14:18:20.377712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:20.377712+0000","objects_scrubbed":3737,"log_size":6426,"log_dups_size":0,"ondisk_log_size":6426,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:34.514418+0000","scrub_duration":701,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218759168,"num_objects":3338,"num_object_clones":0,"num_object_copies":10014,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1195,"num_write_kb":25472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,5],"acting":[1,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"70'139","reported_seq":494,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-08-19T14:18:39.056973+0000","last_change":"2024-08-19T14:18:28.501538+0000","last_active":"2024-08-19T14:18:39.056973+0000","last_peered":"2024-08-19T14:18:39.056973+0000","last_clean":"2024-08-19T14:18:39.056973+0000","last_became_active":"2024-08-19T14:18:26.881435+0000","last_became_peered":"2024-08-19T14:18:26.881435+0000","last_unstale":"2024-08-19T14:18:39.056973+0000","last_undegraded":"2024-08-19T14:18:39.056973+0000","last_fullsized":"2024-08-19T14:18:39.056973+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:18:28.501449+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:18:28.501449+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:55.948489+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,4],"acting":[11,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.7","version":"287'6020","reported_seq":6216,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-08-19T14:18:39.239584+0000","last_change":"2024-08-19T14:18:18.152792+0000","last_active":"2024-08-19T14:18:39.239584+0000","last_peered":"2024-08-19T14:18:39.239584+0000","last_clean":"2024-08-19T14:18:39.239584+0000","last_became_active":"2024-08-19T14:17:19.742604+0000","last_became_peered":"2024-08-19T14:17:19.742604+0000","last_unstale":"2024-08-19T14:18:39.239584+0000","last_undegraded":"2024-08-19T14:18:39.239584+0000","last_fullsized":"2024-08-19T14:18:39.239584+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'5087","last_scrub_stamp":"2024-08-19T14:18:18.152521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:18.152521+0000","objects_scrubbed":3609,"log_size":6020,"log_dups_size":0,"ondisk_log_size":6020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:45.684422+0000","scrub_duration":838,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":240189440,"num_objects":3665,"num_object_clones":0,"num_object_copies":10995,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":956,"num_write_kb":32384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"70'139","reported_seq":718,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-08-19T14:18:39.058067+0000","last_change":"2024-08-19T14:18:21.476428+0000","last_active":"2024-08-19T14:18:39.058067+0000","last_peered":"2024-08-19T14:18:39.058067+0000","last_clean":"2024-08-19T14:18:39.058067+0000","last_became_active":"2024-08-19T14:18:13.806013+0000","last_became_peered":"2024-08-19T14:18:13.806013+0000","last_unstale":"2024-08-19T14:18:39.058067+0000","last_undegraded":"2024-08-19T14:18:39.058067+0000","last_fullsized":"2024-08-19T14:18:39.058067+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:18:21.476313+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:18:21.476313+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:29.834715+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.6","version":"287'6292","reported_seq":6490,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-08-19T14:18:40.917058+0000","last_change":"2024-08-19T14:18:21.462540+0000","last_active":"2024-08-19T14:18:40.917058+0000","last_peered":"2024-08-19T14:18:40.917058+0000","last_clean":"2024-08-19T14:18:40.917058+0000","last_became_active":"2024-08-19T14:17:19.748180+0000","last_became_peered":"2024-08-19T14:17:19.748180+0000","last_unstale":"2024-08-19T14:18:40.917058+0000","last_undegraded":"2024-08-19T14:18:40.917058+0000","last_fullsized":"2024-08-19T14:18:40.917058+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'5366","last_scrub_stamp":"2024-08-19T14:18:21.462335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:21.462335+0000","objects_scrubbed":3843,"log_size":6292,"log_dups_size":0,"ondisk_log_size":6292,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:33.613418+0000","scrub_duration":461,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235995136,"num_objects":3601,"num_object_clones":0,"num_object_copies":10803,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3601,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":926,"num_write_kb":21888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.a","version":"70'139","reported_seq":688,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-08-19T14:18:39.004960+0000","last_change":"2024-08-19T14:18:28.865004+0000","last_active":"2024-08-19T14:18:39.004960+0000","last_peered":"2024-08-19T14:18:39.004960+0000","last_clean":"2024-08-19T14:18:39.004960+0000","last_became_active":"2024-08-19T14:18:26.875006+0000","last_became_peered":"2024-08-19T14:18:26.875006+0000","last_unstale":"2024-08-19T14:18:39.004960+0000","last_undegraded":"2024-08-19T14:18:39.004960+0000","last_fullsized":"2024-08-19T14:18:39.004960+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:18:28.864870+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:28.864870+0000","last_clean_scrub_stamp":"2024-08-19T14:18:28.864870+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:43.628160+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,0],"acting":[9,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.8","version":"287'6264","reported_seq":6460,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-08-19T14:18:40.647983+0000","last_change":"2024-08-19T14:18:17.263508+0000","last_active":"2024-08-19T14:18:40.647983+0000","last_peered":"2024-08-19T14:18:40.647983+0000","last_clean":"2024-08-19T14:18:40.647983+0000","last_became_active":"2024-08-19T14:17:19.753063+0000","last_became_peered":"2024-08-19T14:17:19.753063+0000","last_unstale":"2024-08-19T14:18:40.647983+0000","last_undegraded":"2024-08-19T14:18:40.647983+0000","last_fullsized":"2024-08-19T14:18:40.647983+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'5161","last_scrub_stamp":"2024-08-19T14:18:17.262486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:17.262486+0000","objects_scrubbed":3649,"log_size":6264,"log_dups_size":0,"ondisk_log_size":6264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:42.192456+0000","scrub_duration":818,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":237240320,"num_objects":3620,"num_object_clones":0,"num_object_copies":10860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3620,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1117,"num_write_kb":34816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,4],"acting":[11,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.9","version":"70'139","reported_seq":803,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-08-19T14:18:39.063131+0000","last_change":"2024-08-19T14:18:28.744550+0000","last_active":"2024-08-19T14:18:39.063131+0000","last_peered":"2024-08-19T14:18:39.063131+0000","last_clean":"2024-08-19T14:18:39.063131+0000","last_became_active":"2024-08-19T14:18:26.887657+0000","last_became_peered":"2024-08-19T14:18:26.887657+0000","last_unstale":"2024-08-19T14:18:39.063131+0000","last_undegraded":"2024-08-19T14:18:39.063131+0000","last_fullsized":"2024-08-19T14:18:39.063131+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:18:28.744320+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:18:28.744320+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:30.737843+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"287'6610","reported_seq":6806,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-08-19T14:18:39.194635+0000","last_change":"2024-08-19T14:18:17.226643+0000","last_active":"2024-08-19T14:18:39.194635+0000","last_peered":"2024-08-19T14:18:39.194635+0000","last_clean":"2024-08-19T14:18:39.194635+0000","last_became_active":"2024-08-19T14:17:34.972934+0000","last_became_peered":"2024-08-19T14:17:34.972934+0000","last_unstale":"2024-08-19T14:18:39.194635+0000","last_undegraded":"2024-08-19T14:18:39.194635+0000","last_fullsized":"2024-08-19T14:18:39.194635+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'5595","last_scrub_stamp":"2024-08-19T14:18:17.226507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:17.226507+0000","objects_scrubbed":3605,"log_size":6610,"log_dups_size":0,"ondisk_log_size":6610,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:43.427899+0000","scrub_duration":888,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":244776960,"num_objects":3735,"num_object_clones":0,"num_object_copies":11205,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3735,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1048,"num_write_kb":37696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"70'139","reported_seq":692,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-08-19T14:18:39.057192+0000","last_change":"2024-08-19T14:18:28.190126+0000","last_active":"2024-08-19T14:18:39.057192+0000","last_peered":"2024-08-19T14:18:39.057192+0000","last_clean":"2024-08-19T14:18:39.057192+0000","last_became_active":"2024-08-19T14:18:26.887277+0000","last_became_peered":"2024-08-19T14:18:26.887277+0000","last_unstale":"2024-08-19T14:18:39.057192+0000","last_undegraded":"2024-08-19T14:18:39.057192+0000","last_fullsized":"2024-08-19T14:18:39.057192+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:18:28.190009+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:18:28.190009+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:57.986532+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,8],"acting":[5,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"287'6797","reported_seq":6989,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-08-19T14:18:43.006177+0000","last_change":"2024-08-19T14:18:21.329122+0000","last_active":"2024-08-19T14:18:43.006177+0000","last_peered":"2024-08-19T14:18:43.006177+0000","last_clean":"2024-08-19T14:18:43.006177+0000","last_became_active":"2024-08-19T14:17:34.962596+0000","last_became_peered":"2024-08-19T14:17:34.962596+0000","last_unstale":"2024-08-19T14:18:43.006177+0000","last_undegraded":"2024-08-19T14:18:43.006177+0000","last_fullsized":"2024-08-19T14:18:43.006177+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'5664","last_scrub_stamp":"2024-08-19T14:18:21.328957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:21.328957+0000","objects_scrubbed":3688,"log_size":6797,"log_dups_size":0,"ondisk_log_size":6797,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:22.510625+0000","scrub_duration":405,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220790784,"num_objects":3369,"num_object_clones":0,"num_object_copies":10107,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3369,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1133,"num_write_kb":26048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,8],"acting":[4,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"287'6043","reported_seq":6238,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-08-19T14:18:39.826581+0000","last_change":"2024-08-19T14:18:18.202254+0000","last_active":"2024-08-19T14:18:39.826581+0000","last_peered":"2024-08-19T14:18:39.826581+0000","last_clean":"2024-08-19T14:18:39.826581+0000","last_became_active":"2024-08-19T14:17:19.743605+0000","last_became_peered":"2024-08-19T14:17:19.743605+0000","last_unstale":"2024-08-19T14:18:39.826581+0000","last_undegraded":"2024-08-19T14:18:39.826581+0000","last_fullsized":"2024-08-19T14:18:39.826581+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'5067","last_scrub_stamp":"2024-08-19T14:18:18.201966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:18.201966+0000","objects_scrubbed":3534,"log_size":6043,"log_dups_size":0,"ondisk_log_size":6043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:46.145488+0000","scrub_duration":779,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":231538688,"num_objects":3533,"num_object_clones":0,"num_object_copies":10599,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1003,"num_write_kb":32064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.b","version":"70'139","reported_seq":694,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-08-19T14:18:39.056805+0000","last_change":"2024-08-19T14:18:27.506148+0000","last_active":"2024-08-19T14:18:39.056805+0000","last_peered":"2024-08-19T14:18:39.056805+0000","last_clean":"2024-08-19T14:18:39.056805+0000","last_became_active":"2024-08-19T14:18:26.875179+0000","last_became_peered":"2024-08-19T14:18:26.875179+0000","last_unstale":"2024-08-19T14:18:39.056805+0000","last_undegraded":"2024-08-19T14:18:39.056805+0000","last_fullsized":"2024-08-19T14:18:39.056805+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:18:27.506045+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:18:27.506045+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:43.546458+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.c","version":"287'6655","reported_seq":6851,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-08-19T14:18:41.604285+0000","last_change":"2024-08-19T14:18:18.066128+0000","last_active":"2024-08-19T14:18:41.604285+0000","last_peered":"2024-08-19T14:18:41.604285+0000","last_clean":"2024-08-19T14:18:41.604285+0000","last_became_active":"2024-08-19T14:17:34.973857+0000","last_became_peered":"2024-08-19T14:17:34.973857+0000","last_unstale":"2024-08-19T14:18:41.604285+0000","last_undegraded":"2024-08-19T14:18:41.604285+0000","last_fullsized":"2024-08-19T14:18:41.604285+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'5481","last_scrub_stamp":"2024-08-19T14:18:18.065864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:18.065864+0000","objects_scrubbed":3568,"log_size":6655,"log_dups_size":0,"ondisk_log_size":6655,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:25.462037+0000","scrub_duration":959,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":229244928,"num_objects":3498,"num_object_clones":0,"num_object_copies":10494,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1202,"num_write_kb":36224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,9],"acting":[5,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"70'139","reported_seq":348,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-08-19T14:18:39.060298+0000","last_change":"2024-08-19T14:18:27.900166+0000","last_active":"2024-08-19T14:18:39.060298+0000","last_peered":"2024-08-19T14:18:39.060298+0000","last_clean":"2024-08-19T14:18:39.060298+0000","last_became_active":"2024-08-19T14:18:26.884105+0000","last_became_peered":"2024-08-19T14:18:26.884105+0000","last_unstale":"2024-08-19T14:18:39.060298+0000","last_undegraded":"2024-08-19T14:18:39.060298+0000","last_fullsized":"2024-08-19T14:18:39.060298+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:18:27.899965+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:18:27.899965+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:56.474954+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"287'6933","reported_seq":7131,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-08-19T14:18:43.175264+0000","last_change":"2024-08-19T14:18:26.752263+0000","last_active":"2024-08-19T14:18:43.175264+0000","last_peered":"2024-08-19T14:18:43.175264+0000","last_clean":"2024-08-19T14:18:43.175264+0000","last_became_active":"2024-08-19T14:17:34.969333+0000","last_became_peered":"2024-08-19T14:17:34.969333+0000","last_unstale":"2024-08-19T14:18:43.175264+0000","last_undegraded":"2024-08-19T14:18:43.175264+0000","last_fullsized":"2024-08-19T14:18:43.175264+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'5846","last_scrub_stamp":"2024-08-19T14:18:26.752109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:26.752109+0000","objects_scrubbed":3860,"log_size":6933,"log_dups_size":0,"ondisk_log_size":6933,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:56.568372+0000","scrub_duration":5035,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":225247232,"num_objects":3437,"num_object_clones":0,"num_object_copies":10311,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3437,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1101,"num_write_kb":21696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,9],"acting":[7,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"70'139","reported_seq":592,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-08-19T14:18:39.057532+0000","last_change":"2024-08-19T14:18:23.173343+0000","last_active":"2024-08-19T14:18:39.057532+0000","last_peered":"2024-08-19T14:18:39.057532+0000","last_clean":"2024-08-19T14:18:39.057532+0000","last_became_active":"2024-08-19T14:18:18.711708+0000","last_became_peered":"2024-08-19T14:18:18.711708+0000","last_unstale":"2024-08-19T14:18:39.057532+0000","last_undegraded":"2024-08-19T14:18:39.057532+0000","last_fullsized":"2024-08-19T14:18:39.057532+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:18:23.173268+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:18:23.173268+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:51.974300+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"287'6695","reported_seq":6890,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-08-19T14:18:40.688654+0000","last_change":"2024-08-19T14:18:19.171319+0000","last_active":"2024-08-19T14:18:40.688654+0000","last_peered":"2024-08-19T14:18:40.688654+0000","last_clean":"2024-08-19T14:18:40.688654+0000","last_became_active":"2024-08-19T14:17:34.971738+0000","last_became_peered":"2024-08-19T14:17:34.971738+0000","last_unstale":"2024-08-19T14:18:40.688654+0000","last_undegraded":"2024-08-19T14:18:40.688654+0000","last_fullsized":"2024-08-19T14:18:40.688654+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'5670","last_scrub_stamp":"2024-08-19T14:18:19.171123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:19.171123+0000","objects_scrubbed":3656,"log_size":6695,"log_dups_size":0,"ondisk_log_size":6695,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:37.122914+0000","scrub_duration":758,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":232325120,"num_objects":3545,"num_object_clones":0,"num_object_copies":10635,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3545,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1051,"num_write_kb":30080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,0],"acting":[11,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.c","version":"70'139","reported_seq":453,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-08-19T14:18:39.061009+0000","last_change":"2024-08-19T14:18:28.348190+0000","last_active":"2024-08-19T14:18:39.061009+0000","last_peered":"2024-08-19T14:18:39.061009+0000","last_clean":"2024-08-19T14:18:39.061009+0000","last_became_active":"2024-08-19T14:18:26.872942+0000","last_became_peered":"2024-08-19T14:18:26.872942+0000","last_unstale":"2024-08-19T14:18:39.061009+0000","last_undegraded":"2024-08-19T14:18:39.061009+0000","last_fullsized":"2024-08-19T14:18:39.061009+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:18:28.348087+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:18:28.348087+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:39.796688+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,9],"acting":[3,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.f","version":"287'6855","reported_seq":7051,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-08-19T14:18:43.184891+0000","last_change":"2024-08-19T14:18:23.090893+0000","last_active":"2024-08-19T14:18:43.184891+0000","last_peered":"2024-08-19T14:18:43.184891+0000","last_clean":"2024-08-19T14:18:43.184891+0000","last_became_active":"2024-08-19T14:17:34.954276+0000","last_became_peered":"2024-08-19T14:17:34.954276+0000","last_unstale":"2024-08-19T14:18:43.184891+0000","last_undegraded":"2024-08-19T14:18:43.184891+0000","last_fullsized":"2024-08-19T14:18:43.184891+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'5708","last_scrub_stamp":"2024-08-19T14:18:23.090770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:23.090770+0000","objects_scrubbed":3694,"log_size":6855,"log_dups_size":0,"ondisk_log_size":6855,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:33.066761+0000","scrub_duration":369,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218955776,"num_objects":3341,"num_object_clones":0,"num_object_copies":10023,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3341,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1147,"num_write_kb":25408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"70'139","reported_seq":368,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-08-19T14:18:39.063397+0000","last_change":"2024-08-19T14:18:27.752642+0000","last_active":"2024-08-19T14:18:39.063397+0000","last_peered":"2024-08-19T14:18:39.063397+0000","last_clean":"2024-08-19T14:18:39.063397+0000","last_became_active":"2024-08-19T14:18:26.882002+0000","last_became_peered":"2024-08-19T14:18:26.882002+0000","last_unstale":"2024-08-19T14:18:39.063397+0000","last_undegraded":"2024-08-19T14:18:39.063397+0000","last_fullsized":"2024-08-19T14:18:39.063397+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:18:27.751964+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:18:27.751964+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:48.211625+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"287'6299","reported_seq":11159,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-08-19T14:18:43.285166+0000","last_change":"2024-08-19T14:18:20.355375+0000","last_active":"2024-08-19T14:18:43.285166+0000","last_peered":"2024-08-19T14:18:43.285166+0000","last_clean":"2024-08-19T14:18:43.285166+0000","last_became_active":"2024-08-19T14:17:38.684742+0000","last_became_peered":"2024-08-19T14:17:38.684742+0000","last_unstale":"2024-08-19T14:18:43.285166+0000","last_undegraded":"2024-08-19T14:18:43.285166+0000","last_fullsized":"2024-08-19T14:18:43.285166+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":5,"last_scrub":"273'5152","last_scrub_stamp":"2024-08-19T14:18:20.355288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:20.355288+0000","objects_scrubbed":3752,"log_size":6299,"log_dups_size":0,"ondisk_log_size":6299,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:42.543316+0000","scrub_duration":598,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220528640,"num_objects":3365,"num_object_clones":0,"num_object_copies":10095,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1151,"num_write_kb":24448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,3],"acting":[9,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.12","version":"70'139","reported_seq":627,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-08-19T14:18:39.057225+0000","last_change":"2024-08-19T14:18:20.109582+0000","last_active":"2024-08-19T14:18:39.057225+0000","last_peered":"2024-08-19T14:18:39.057225+0000","last_clean":"2024-08-19T14:18:39.057225+0000","last_became_active":"2024-08-19T14:18:18.703594+0000","last_became_peered":"2024-08-19T14:18:18.703594+0000","last_unstale":"2024-08-19T14:18:39.057225+0000","last_undegraded":"2024-08-19T14:18:39.057225+0000","last_fullsized":"2024-08-19T14:18:39.057225+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:18:20.109467+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:18:20.109467+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:21.664555+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"287'6201","reported_seq":9500,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-08-19T14:18:43.006540+0000","last_change":"2024-08-19T14:18:18.845192+0000","last_active":"2024-08-19T14:18:43.006540+0000","last_peered":"2024-08-19T14:18:43.006540+0000","last_clean":"2024-08-19T14:18:43.006540+0000","last_became_active":"2024-08-19T14:17:38.695549+0000","last_became_peered":"2024-08-19T14:17:38.695549+0000","last_unstale":"2024-08-19T14:18:43.006540+0000","last_undegraded":"2024-08-19T14:18:43.006540+0000","last_fullsized":"2024-08-19T14:18:43.006540+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":5,"last_scrub":"273'5072","last_scrub_stamp":"2024-08-19T14:18:18.844791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:18.844791+0000","objects_scrubbed":3594,"log_size":6201,"log_dups_size":0,"ondisk_log_size":6201,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:35.762099+0000","scrub_duration":924,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":216793088,"num_objects":3308,"num_object_clones":0,"num_object_copies":9924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1148,"num_write_kb":27584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.13","version":"70'139","reported_seq":687,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-08-19T14:18:39.055381+0000","last_change":"2024-08-19T14:18:22.659716+0000","last_active":"2024-08-19T14:18:39.055381+0000","last_peered":"2024-08-19T14:18:39.055381+0000","last_clean":"2024-08-19T14:18:39.055381+0000","last_became_active":"2024-08-19T14:18:18.698334+0000","last_became_peered":"2024-08-19T14:18:18.698334+0000","last_unstale":"2024-08-19T14:18:39.055381+0000","last_undegraded":"2024-08-19T14:18:39.055381+0000","last_fullsized":"2024-08-19T14:18:39.055381+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:18:22.659638+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:18:22.659638+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:50.367468+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.12","version":"287'6142","reported_seq":12701,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-08-19T14:18:40.853654+0000","last_change":"2024-08-19T14:18:23.349819+0000","last_active":"2024-08-19T14:18:40.853654+0000","last_peered":"2024-08-19T14:18:40.853654+0000","last_clean":"2024-08-19T14:18:40.853654+0000","last_became_active":"2024-08-19T14:17:47.703601+0000","last_became_peered":"2024-08-19T14:17:47.703601+0000","last_unstale":"2024-08-19T14:18:40.853654+0000","last_undegraded":"2024-08-19T14:18:40.853654+0000","last_fullsized":"2024-08-19T14:18:40.853654+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":5,"last_scrub":"273'5164","last_scrub_stamp":"2024-08-19T14:18:23.349706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:23.349706+0000","objects_scrubbed":3676,"log_size":6142,"log_dups_size":0,"ondisk_log_size":6142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:44.157657+0000","scrub_duration":326,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":227803136,"num_objects":3476,"num_object_clones":0,"num_object_copies":10428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3476,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":978,"num_write_kb":24896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"70'139","reported_seq":782,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-08-19T14:18:39.056867+0000","last_change":"2024-08-19T14:18:22.139657+0000","last_active":"2024-08-19T14:18:39.056867+0000","last_peered":"2024-08-19T14:18:39.056867+0000","last_clean":"2024-08-19T14:18:39.056867+0000","last_became_active":"2024-08-19T14:18:18.704562+0000","last_became_peered":"2024-08-19T14:18:18.704562+0000","last_unstale":"2024-08-19T14:18:39.056867+0000","last_undegraded":"2024-08-19T14:18:39.056867+0000","last_fullsized":"2024-08-19T14:18:39.056867+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:18:22.139587+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:18:22.139587+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:43.235144+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,8],"acting":[5,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"287'6301","reported_seq":11116,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-08-19T14:18:43.173424+0000","last_change":"2024-08-19T14:18:18.597206+0000","last_active":"2024-08-19T14:18:43.173424+0000","last_peered":"2024-08-19T14:18:43.173424+0000","last_clean":"2024-08-19T14:18:43.173424+0000","last_became_active":"2024-08-19T14:17:45.731114+0000","last_became_peered":"2024-08-19T14:17:45.731114+0000","last_unstale":"2024-08-19T14:18:43.173424+0000","last_undegraded":"2024-08-19T14:18:43.173424+0000","last_fullsized":"2024-08-19T14:18:43.173424+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":5,"last_scrub":"272'5093","last_scrub_stamp":"2024-08-19T14:18:18.596762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:18.596762+0000","objects_scrubbed":3558,"log_size":6301,"log_dups_size":0,"ondisk_log_size":6301,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:23.244078+0000","scrub_duration":834,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218365952,"num_objects":3332,"num_object_clones":0,"num_object_copies":9996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1228,"num_write_kb":32064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,9],"acting":[7,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.11","version":"70'139","reported_seq":331,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-08-19T14:18:39.004255+0000","last_change":"2024-08-19T14:18:25.021206+0000","last_active":"2024-08-19T14:18:39.004255+0000","last_peered":"2024-08-19T14:18:39.004255+0000","last_clean":"2024-08-19T14:18:39.004255+0000","last_became_active":"2024-08-19T14:18:18.697894+0000","last_became_peered":"2024-08-19T14:18:18.697894+0000","last_unstale":"2024-08-19T14:18:39.004255+0000","last_undegraded":"2024-08-19T14:18:39.004255+0000","last_fullsized":"2024-08-19T14:18:39.004255+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:18:25.021126+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:18:25.021126+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:32.345902+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.14","version":"287'6404","reported_seq":11341,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-08-19T14:18:43.032969+0000","last_change":"2024-08-19T14:18:19.862358+0000","last_active":"2024-08-19T14:18:43.032969+0000","last_peered":"2024-08-19T14:18:43.032969+0000","last_clean":"2024-08-19T14:18:43.032969+0000","last_became_active":"2024-08-19T14:17:44.691159+0000","last_became_peered":"2024-08-19T14:17:44.691159+0000","last_unstale":"2024-08-19T14:18:43.032969+0000","last_undegraded":"2024-08-19T14:18:43.032969+0000","last_fullsized":"2024-08-19T14:18:43.032969+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":5,"last_scrub":"273'5248","last_scrub_stamp":"2024-08-19T14:18:19.862173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:19.862173+0000","objects_scrubbed":3703,"log_size":6404,"log_dups_size":0,"ondisk_log_size":6404,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:44.248749+0000","scrub_duration":928,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":224985088,"num_objects":3433,"num_object_clones":0,"num_object_copies":10299,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3433,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1196,"num_write_kb":29632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,11],"acting":[4,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.16","version":"70'139","reported_seq":299,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-08-19T14:18:39.055469+0000","last_change":"2024-08-19T14:18:31.900475+0000","last_active":"2024-08-19T14:18:39.055469+0000","last_peered":"2024-08-19T14:18:39.055469+0000","last_clean":"2024-08-19T14:18:39.055469+0000","last_became_active":"2024-08-19T14:18:31.899921+0000","last_became_peered":"2024-08-19T14:18:31.899921+0000","last_unstale":"2024-08-19T14:18:39.055469+0000","last_undegraded":"2024-08-19T14:18:39.055469+0000","last_fullsized":"2024-08-19T14:18:39.055469+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:18:22.807154+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:18:22.807154+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:43.728112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13418,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":48084,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,11],"acting":[1,0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"70'139","reported_seq":428,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-08-19T14:18:39.061240+0000","last_change":"2024-08-19T14:18:31.902162+0000","last_active":"2024-08-19T14:18:39.061240+0000","last_peered":"2024-08-19T14:18:39.061240+0000","last_clean":"2024-08-19T14:18:39.061240+0000","last_became_active":"2024-08-19T14:18:31.901556+0000","last_became_peered":"2024-08-19T14:18:31.901556+0000","last_unstale":"2024-08-19T14:18:39.061240+0000","last_undegraded":"2024-08-19T14:18:39.061240+0000","last_fullsized":"2024-08-19T14:18:39.061240+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:18:21.158420+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:18:21.158420+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:42.648013+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,0],"acting":[8,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.15","version":"287'6052","reported_seq":10891,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-08-19T14:18:39.831447+0000","last_change":"2024-08-19T14:18:20.779335+0000","last_active":"2024-08-19T14:18:39.831447+0000","last_peered":"2024-08-19T14:18:39.831447+0000","last_clean":"2024-08-19T14:18:39.831447+0000","last_became_active":"2024-08-19T14:17:42.696659+0000","last_became_peered":"2024-08-19T14:17:42.696659+0000","last_unstale":"2024-08-19T14:18:39.831447+0000","last_undegraded":"2024-08-19T14:18:39.831447+0000","last_fullsized":"2024-08-19T14:18:39.831447+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":5,"last_scrub":"273'5168","last_scrub_stamp":"2024-08-19T14:18:20.779227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:20.779227+0000","objects_scrubbed":3680,"log_size":6052,"log_dups_size":0,"ondisk_log_size":6052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:30.576171+0000","scrub_duration":384,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":232390656,"num_objects":3546,"num_object_clones":0,"num_object_copies":10638,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3546,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":884,"num_write_kb":24000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.16","version":"287'6354","reported_seq":11152,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-08-19T14:18:41.584508+0000","last_change":"2024-08-19T14:18:18.940376+0000","last_active":"2024-08-19T14:18:41.584508+0000","last_peered":"2024-08-19T14:18:41.584508+0000","last_clean":"2024-08-19T14:18:41.584508+0000","last_became_active":"2024-08-19T14:17:47.681397+0000","last_became_peered":"2024-08-19T14:17:47.681397+0000","last_unstale":"2024-08-19T14:18:41.584508+0000","last_undegraded":"2024-08-19T14:18:41.584508+0000","last_fullsized":"2024-08-19T14:18:41.584508+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":5,"last_scrub":"273'5233","last_scrub_stamp":"2024-08-19T14:18:18.940216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:18.940216+0000","objects_scrubbed":3684,"log_size":6354,"log_dups_size":0,"ondisk_log_size":6354,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:37.548896+0000","scrub_duration":846,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233046016,"num_objects":3556,"num_object_clones":0,"num_object_copies":10668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3556,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1140,"num_write_kb":32384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.14","version":"70'139","reported_seq":445,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-08-19T14:18:39.063077+0000","last_change":"2024-08-19T14:18:18.745307+0000","last_active":"2024-08-19T14:18:39.063077+0000","last_peered":"2024-08-19T14:18:39.063077+0000","last_clean":"2024-08-19T14:18:39.063077+0000","last_became_active":"2024-08-19T14:18:18.703926+0000","last_became_peered":"2024-08-19T14:18:18.703926+0000","last_unstale":"2024-08-19T14:18:39.063077+0000","last_undegraded":"2024-08-19T14:18:39.063077+0000","last_fullsized":"2024-08-19T14:18:39.063077+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:18:18.745191+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:18:18.745191+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:24.644627+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,1],"acting":[7,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"287'6067","reported_seq":12385,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-08-19T14:18:40.899803+0000","last_change":"2024-08-19T14:18:16.720459+0000","last_active":"2024-08-19T14:18:40.899803+0000","last_peered":"2024-08-19T14:18:40.899803+0000","last_clean":"2024-08-19T14:18:40.899803+0000","last_became_active":"2024-08-19T14:17:48.682321+0000","last_became_peered":"2024-08-19T14:17:48.682321+0000","last_unstale":"2024-08-19T14:18:40.899803+0000","last_undegraded":"2024-08-19T14:18:40.899803+0000","last_fullsized":"2024-08-19T14:18:40.899803+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":5,"last_scrub":"270'4919","last_scrub_stamp":"2024-08-19T14:18:16.720122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:16.720122+0000","objects_scrubbed":3405,"log_size":6067,"log_dups_size":0,"ondisk_log_size":6067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:37.591387+0000","scrub_duration":754,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":226623488,"num_objects":3458,"num_object_clones":0,"num_object_copies":10374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3458,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1171,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,9],"acting":[0,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.15","version":"70'139","reported_seq":324,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-08-19T14:18:39.063347+0000","last_change":"2024-08-19T14:18:31.912927+0000","last_active":"2024-08-19T14:18:39.063347+0000","last_peered":"2024-08-19T14:18:39.063347+0000","last_clean":"2024-08-19T14:18:39.063347+0000","last_became_active":"2024-08-19T14:18:31.912496+0000","last_became_peered":"2024-08-19T14:18:31.912496+0000","last_unstale":"2024-08-19T14:18:39.063347+0000","last_undegraded":"2024-08-19T14:18:39.063347+0000","last_fullsized":"2024-08-19T14:18:39.063347+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:18:20.455298+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:18:20.455298+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:43.822943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13418,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":48085,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"70'139","reported_seq":302,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-08-19T14:18:39.059502+0000","last_change":"2024-08-19T14:18:31.901742+0000","last_active":"2024-08-19T14:18:39.059502+0000","last_peered":"2024-08-19T14:18:39.059502+0000","last_clean":"2024-08-19T14:18:39.059502+0000","last_became_active":"2024-08-19T14:18:31.901229+0000","last_became_peered":"2024-08-19T14:18:31.901229+0000","last_unstale":"2024-08-19T14:18:39.059502+0000","last_undegraded":"2024-08-19T14:18:39.059502+0000","last_fullsized":"2024-08-19T14:18:39.059502+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:18:23.815845+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:18:23.815845+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:42.884288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":5,"num_write":8,"num_write_kb":134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":96169,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.18","version":"287'6132","reported_seq":11054,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-08-19T14:18:39.228194+0000","last_change":"2024-08-19T14:18:21.736611+0000","last_active":"2024-08-19T14:18:39.228194+0000","last_peered":"2024-08-19T14:18:39.228194+0000","last_clean":"2024-08-19T14:18:39.228194+0000","last_became_active":"2024-08-19T14:17:45.729134+0000","last_became_peered":"2024-08-19T14:17:45.729134+0000","last_unstale":"2024-08-19T14:18:39.228194+0000","last_undegraded":"2024-08-19T14:18:39.228194+0000","last_fullsized":"2024-08-19T14:18:39.228194+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":5,"last_scrub":"273'5311","last_scrub_stamp":"2024-08-19T14:18:21.736491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:21.736491+0000","objects_scrubbed":3712,"log_size":6132,"log_dups_size":0,"ondisk_log_size":6132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:34.973824+0000","scrub_duration":424,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235470848,"num_objects":3593,"num_object_clones":0,"num_object_copies":10779,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":821,"num_write_kb":22464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,1],"acting":[3,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1b","version":"70'139","reported_seq":654,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-08-19T14:18:39.061085+0000","last_change":"2024-08-19T14:18:31.901557+0000","last_active":"2024-08-19T14:18:39.061085+0000","last_peered":"2024-08-19T14:18:39.061085+0000","last_clean":"2024-08-19T14:18:39.061085+0000","last_became_active":"2024-08-19T14:18:31.901152+0000","last_became_peered":"2024-08-19T14:18:31.901152+0000","last_unstale":"2024-08-19T14:18:39.061085+0000","last_undegraded":"2024-08-19T14:18:39.061085+0000","last_fullsized":"2024-08-19T14:18:39.061085+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:18:19.414519+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:18:19.414519+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:42.647980+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.19","version":"287'6368","reported_seq":12850,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-08-19T14:18:43.356958+0000","last_change":"2024-08-19T14:18:21.161635+0000","last_active":"2024-08-19T14:18:43.356958+0000","last_peered":"2024-08-19T14:18:43.356958+0000","last_clean":"2024-08-19T14:18:43.356958+0000","last_became_active":"2024-08-19T14:17:46.796533+0000","last_became_peered":"2024-08-19T14:17:46.796533+0000","last_unstale":"2024-08-19T14:18:43.356958+0000","last_undegraded":"2024-08-19T14:18:43.356958+0000","last_fullsized":"2024-08-19T14:18:43.356958+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":5,"last_scrub":"273'5249","last_scrub_stamp":"2024-08-19T14:18:21.161536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:21.161536+0000","objects_scrubbed":3748,"log_size":6368,"log_dups_size":0,"ondisk_log_size":6368,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:31.937329+0000","scrub_duration":491,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222494720,"num_objects":3395,"num_object_clones":0,"num_object_copies":10185,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1119,"num_write_kb":24512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.18","version":"70'139","reported_seq":698,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-08-19T14:18:39.056750+0000","last_change":"2024-08-19T14:18:31.903169+0000","last_active":"2024-08-19T14:18:39.056750+0000","last_peered":"2024-08-19T14:18:39.056750+0000","last_clean":"2024-08-19T14:18:39.056750+0000","last_became_active":"2024-08-19T14:18:31.902735+0000","last_became_peered":"2024-08-19T14:18:31.902735+0000","last_unstale":"2024-08-19T14:18:39.056750+0000","last_undegraded":"2024-08-19T14:18:39.056750+0000","last_fullsized":"2024-08-19T14:18:39.056750+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:18:25.178947+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:18:25.178947+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:42.127561+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131344,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":301872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"287'6593","reported_seq":10814,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-08-19T14:18:39.818509+0000","last_change":"2024-08-19T14:18:20.149227+0000","last_active":"2024-08-19T14:18:39.818509+0000","last_peered":"2024-08-19T14:18:39.818509+0000","last_clean":"2024-08-19T14:18:39.818509+0000","last_became_active":"2024-08-19T14:17:46.731170+0000","last_became_peered":"2024-08-19T14:17:46.731170+0000","last_unstale":"2024-08-19T14:18:39.818509+0000","last_undegraded":"2024-08-19T14:18:39.818509+0000","last_fullsized":"2024-08-19T14:18:39.818509+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":5,"last_scrub":"273'5710","last_scrub_stamp":"2024-08-19T14:18:20.148991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:20.148991+0000","objects_scrubbed":3721,"log_size":6593,"log_dups_size":0,"ondisk_log_size":6593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:38.250493+0000","scrub_duration":772,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235470848,"num_objects":3593,"num_object_clones":0,"num_object_copies":10779,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":904,"num_write_kb":24832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,9],"acting":[8,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.19","version":"70'139","reported_seq":646,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-08-19T14:18:39.004238+0000","last_change":"2024-08-19T14:18:31.902933+0000","last_active":"2024-08-19T14:18:39.004238+0000","last_peered":"2024-08-19T14:18:39.004238+0000","last_clean":"2024-08-19T14:18:39.004238+0000","last_became_active":"2024-08-19T14:18:31.902547+0000","last_became_peered":"2024-08-19T14:18:31.902547+0000","last_unstale":"2024-08-19T14:18:39.004238+0000","last_undegraded":"2024-08-19T14:18:39.004238+0000","last_fullsized":"2024-08-19T14:18:39.004238+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:16:03.918046+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:42.075962+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1b","version":"287'6802","reported_seq":12982,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-08-19T14:18:43.006054+0000","last_change":"2024-08-19T14:18:17.849009+0000","last_active":"2024-08-19T14:18:43.006054+0000","last_peered":"2024-08-19T14:18:43.006054+0000","last_clean":"2024-08-19T14:18:43.006054+0000","last_became_active":"2024-08-19T14:18:06.761591+0000","last_became_peered":"2024-08-19T14:18:06.761591+0000","last_unstale":"2024-08-19T14:18:43.006054+0000","last_undegraded":"2024-08-19T14:18:43.006054+0000","last_fullsized":"2024-08-19T14:18:43.006054+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":5,"last_scrub":"272'5596","last_scrub_stamp":"2024-08-19T14:18:17.848788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:17.848788+0000","objects_scrubbed":3567,"log_size":6802,"log_dups_size":0,"ondisk_log_size":6802,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:35.007855+0000","scrub_duration":920,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":219938816,"num_objects":3356,"num_object_clones":0,"num_object_copies":10068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3356,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1223,"num_write_kb":32384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,1],"acting":[4,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1e","version":"70'139","reported_seq":304,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-08-19T14:18:39.057011+0000","last_change":"2024-08-19T14:18:31.903428+0000","last_active":"2024-08-19T14:18:39.057011+0000","last_peered":"2024-08-19T14:18:39.057011+0000","last_clean":"2024-08-19T14:18:39.057011+0000","last_became_active":"2024-08-19T14:18:31.902546+0000","last_became_peered":"2024-08-19T14:18:31.902546+0000","last_unstale":"2024-08-19T14:18:39.057011+0000","last_undegraded":"2024-08-19T14:18:39.057011+0000","last_fullsized":"2024-08-19T14:18:39.057011+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:18:21.964495+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:18:21.964495+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:42.563349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13418,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":48085,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"287'6634","reported_seq":12619,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-08-19T14:18:41.607745+0000","last_change":"2024-08-19T14:18:20.165996+0000","last_active":"2024-08-19T14:18:41.607745+0000","last_peered":"2024-08-19T14:18:41.607745+0000","last_clean":"2024-08-19T14:18:41.607745+0000","last_became_active":"2024-08-19T14:18:05.757137+0000","last_became_peered":"2024-08-19T14:18:05.757137+0000","last_unstale":"2024-08-19T14:18:41.607745+0000","last_undegraded":"2024-08-19T14:18:41.607745+0000","last_fullsized":"2024-08-19T14:18:41.607745+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":5,"last_scrub":"273'5655","last_scrub_stamp":"2024-08-19T14:18:20.165862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:20.165862+0000","objects_scrubbed":3615,"log_size":6634,"log_dups_size":0,"ondisk_log_size":6634,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:38.986213+0000","scrub_duration":1057,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":221118464,"num_objects":3374,"num_object_clones":0,"num_object_copies":10122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3374,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1003,"num_write_kb":24384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"287'6662","reported_seq":12901,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-08-19T14:18:39.820668+0000","last_change":"2024-08-19T14:18:21.800463+0000","last_active":"2024-08-19T14:18:39.820668+0000","last_peered":"2024-08-19T14:18:39.820668+0000","last_clean":"2024-08-19T14:18:39.820668+0000","last_became_active":"2024-08-19T14:17:59.735912+0000","last_became_peered":"2024-08-19T14:17:59.735912+0000","last_unstale":"2024-08-19T14:18:39.820668+0000","last_undegraded":"2024-08-19T14:18:39.820668+0000","last_fullsized":"2024-08-19T14:18:39.820668+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":5,"last_scrub":"273'5747","last_scrub_stamp":"2024-08-19T14:18:21.800353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:21.800353+0000","objects_scrubbed":3688,"log_size":6662,"log_dups_size":0,"ondisk_log_size":6662,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:39.685212+0000","scrub_duration":368,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235798552,"num_objects":3599,"num_object_clones":0,"num_object_copies":10797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3599,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":915,"num_write_kb":26369,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,5],"acting":[8,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1c","version":"70'139","reported_seq":412,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-08-19T14:18:39.061130+0000","last_change":"2024-08-19T14:18:31.903380+0000","last_active":"2024-08-19T14:18:39.061130+0000","last_peered":"2024-08-19T14:18:39.061130+0000","last_clean":"2024-08-19T14:18:39.061130+0000","last_became_active":"2024-08-19T14:18:31.902494+0000","last_became_peered":"2024-08-19T14:18:31.902494+0000","last_unstale":"2024-08-19T14:18:39.061130+0000","last_undegraded":"2024-08-19T14:18:39.061130+0000","last_fullsized":"2024-08-19T14:18:39.061130+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:18:27.323451+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:18:27.323451+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:42.648043+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8945,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":3,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":40071,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,11],"acting":[8,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1e","version":"287'6458","reported_seq":14774,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-08-19T14:18:39.222660+0000","last_change":"2024-08-19T14:18:19.041894+0000","last_active":"2024-08-19T14:18:39.222660+0000","last_peered":"2024-08-19T14:18:39.222660+0000","last_clean":"2024-08-19T14:18:39.222660+0000","last_became_active":"2024-08-19T14:18:11.801671+0000","last_became_peered":"2024-08-19T14:18:11.801671+0000","last_unstale":"2024-08-19T14:18:39.222660+0000","last_undegraded":"2024-08-19T14:18:39.222660+0000","last_fullsized":"2024-08-19T14:18:39.222660+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":5,"last_scrub":"273'5544","last_scrub_stamp":"2024-08-19T14:18:19.041763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:19.041763+0000","objects_scrubbed":3560,"log_size":6458,"log_dups_size":0,"ondisk_log_size":6458,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:29.244531+0000","scrub_duration":775,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":232652800,"num_objects":3550,"num_object_clones":0,"num_object_copies":10650,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3550,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":932,"num_write_kb":29504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,8],"acting":[3,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1d","version":"70'139","reported_seq":334,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-08-19T14:18:39.055448+0000","last_change":"2024-08-19T14:18:31.901595+0000","last_active":"2024-08-19T14:18:39.055448+0000","last_peered":"2024-08-19T14:18:39.055448+0000","last_clean":"2024-08-19T14:18:39.055448+0000","last_became_active":"2024-08-19T14:18:31.901183+0000","last_became_peered":"2024-08-19T14:18:31.901183+0000","last_unstale":"2024-08-19T14:18:39.055448+0000","last_undegraded":"2024-08-19T14:18:39.055448+0000","last_fullsized":"2024-08-19T14:18:39.055448+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:18:19.739352+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:18:19.739352+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:43.728143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13418,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":5,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":48085,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,5],"acting":[1,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"287'6621","reported_seq":14835,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-08-19T14:18:40.895391+0000","last_change":"2024-08-19T14:18:19.910823+0000","last_active":"2024-08-19T14:18:40.895391+0000","last_peered":"2024-08-19T14:18:40.895391+0000","last_clean":"2024-08-19T14:18:40.895391+0000","last_became_active":"2024-08-19T14:18:12.800420+0000","last_became_peered":"2024-08-19T14:18:12.800420+0000","last_unstale":"2024-08-19T14:18:40.895391+0000","last_undegraded":"2024-08-19T14:18:40.895391+0000","last_fullsized":"2024-08-19T14:18:40.895391+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":5,"last_scrub":"273'5615","last_scrub_stamp":"2024-08-19T14:18:19.910511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:19.910511+0000","objects_scrubbed":3633,"log_size":6621,"log_dups_size":0,"ondisk_log_size":6621,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:30.864591+0000","scrub_duration":922,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":224788480,"num_objects":3430,"num_object_clones":0,"num_object_copies":10290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3430,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1039,"num_write_kb":26752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.20","version":"287'4764","reported_seq":6921,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-08-19T14:18:43.257184+0000","last_change":"2024-08-19T14:18:18.133721+0000","last_active":"2024-08-19T14:18:43.257184+0000","last_peered":"2024-08-19T14:18:43.257184+0000","last_clean":"2024-08-19T14:18:43.257184+0000","last_became_active":"2024-08-19T14:17:43.686357+0000","last_became_peered":"2024-08-19T14:17:43.686357+0000","last_unstale":"2024-08-19T14:18:43.257184+0000","last_undegraded":"2024-08-19T14:18:43.257184+0000","last_fullsized":"2024-08-19T14:18:43.257184+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":6,"last_scrub":"272'4197","last_scrub_stamp":"2024-08-19T14:18:18.133535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:18.133535+0000","objects_scrubbed":1747,"log_size":4764,"log_dups_size":0,"ondisk_log_size":4764,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:28.095319+0000","scrub_duration":374,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109969408,"num_objects":1678,"num_object_clones":0,"num_object_copies":5034,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1678,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":571,"num_write_kb":16064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,11],"acting":[9,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.21","version":"287'4744","reported_seq":8842,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-08-19T14:18:39.820112+0000","last_change":"2024-08-19T14:18:16.780918+0000","last_active":"2024-08-19T14:18:39.820112+0000","last_peered":"2024-08-19T14:18:39.820112+0000","last_clean":"2024-08-19T14:18:39.820112+0000","last_became_active":"2024-08-19T14:17:49.680746+0000","last_became_peered":"2024-08-19T14:17:49.680746+0000","last_unstale":"2024-08-19T14:18:39.820112+0000","last_undegraded":"2024-08-19T14:18:39.820112+0000","last_fullsized":"2024-08-19T14:18:39.820112+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":6,"last_scrub":"270'4217","last_scrub_stamp":"2024-08-19T14:18:16.780726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:16.780726+0000","objects_scrubbed":1743,"log_size":4744,"log_dups_size":0,"ondisk_log_size":4744,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:45.170940+0000","scrub_duration":329,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":115867648,"num_objects":1768,"num_object_clones":0,"num_object_copies":5304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1768,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":531,"num_write_kb":17792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,9],"acting":[8,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.22","version":"287'4889","reported_seq":8081,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-08-19T14:18:39.145473+0000","last_change":"2024-08-19T14:18:19.684323+0000","last_active":"2024-08-19T14:18:39.145473+0000","last_peered":"2024-08-19T14:18:39.145473+0000","last_clean":"2024-08-19T14:18:39.145473+0000","last_became_active":"2024-08-19T14:17:49.687020+0000","last_became_peered":"2024-08-19T14:17:49.687020+0000","last_unstale":"2024-08-19T14:18:39.145473+0000","last_undegraded":"2024-08-19T14:18:39.145473+0000","last_fullsized":"2024-08-19T14:18:39.145473+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":6,"last_scrub":"273'4443","last_scrub_stamp":"2024-08-19T14:18:19.684050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:19.684050+0000","objects_scrubbed":1928,"log_size":4889,"log_dups_size":0,"ondisk_log_size":4889,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:39.823259+0000","scrub_duration":369,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":124256256,"num_objects":1896,"num_object_clones":0,"num_object_copies":5688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":452,"num_write_kb":13440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,9],"acting":[3,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"287'4837","reported_seq":7091,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-08-19T14:18:40.831977+0000","last_change":"2024-08-19T14:18:22.194528+0000","last_active":"2024-08-19T14:18:40.831977+0000","last_peered":"2024-08-19T14:18:40.831977+0000","last_clean":"2024-08-19T14:18:40.831977+0000","last_became_active":"2024-08-19T14:17:46.797296+0000","last_became_peered":"2024-08-19T14:17:46.797296+0000","last_unstale":"2024-08-19T14:18:40.831977+0000","last_undegraded":"2024-08-19T14:18:40.831977+0000","last_fullsized":"2024-08-19T14:18:40.831977+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":6,"last_scrub":"273'4320","last_scrub_stamp":"2024-08-19T14:18:22.194414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:22.194414+0000","objects_scrubbed":1877,"log_size":4837,"log_dups_size":0,"ondisk_log_size":4837,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:24.622604+0000","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":115998720,"num_objects":1770,"num_object_clones":0,"num_object_copies":5310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":517,"num_write_kb":13120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":36,"stat_sum":{"num_bytes":7302807576,"num_objects":111433,"num_object_clones":0,"num_object_copies":334299,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":111433,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":34232,"num_write_kb":927169,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":22024368128,"data_stored":22024355912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":218379,"ondisk_log_size":218379,"up":108,"acting":108,"num_store_stats":9},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":548476,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":20,"num_read_kb":19,"num_write":28,"num_write_kb":447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":630451,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1802240,"data_stored":1788032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187544,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2552212,"kb_used_data":2268356,"kb_used_omap":1,"kb_used_meta":283838,"kb_avail":91192940,"statfs":{"total":95995035648,"available":93381570560,"internally_reserved":0,"allocated":2322796544,"data_stored":2321436824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":290650564},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":343597383774,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":74,"seq":317827580002,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2690444,"kb_used_data":2362180,"kb_used_omap":1,"kb_used_meta":328254,"kb_avail":91054708,"statfs":{"total":95995035648,"available":93240020992,"internally_reserved":0,"allocated":2418872320,"data_stored":2417512600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":336132545},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874343,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2450388,"kb_used_data":2180168,"kb_used_omap":1,"kb_used_meta":269950,"kb_avail":91294764,"statfs":{"total":95995035648,"available":93485838336,"internally_reserved":0,"allocated":2232492032,"data_stored":2231128240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":276429253},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":57,"seq":244813135978,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2378260,"kb_used_data":2058252,"kb_used_omap":1,"kb_used_meta":319934,"kb_avail":91366892,"statfs":{"total":95995035648,"available":93559697408,"internally_reserved":0,"allocated":2107650048,"data_stored":2106286776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":327612874},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":206158430317,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691954,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2674716,"kb_used_data":2468624,"kb_used_omap":1,"kb_used_meta":206078,"kb_avail":91070436,"statfs":{"total":95995035648,"available":93256126464,"internally_reserved":0,"allocated":2527870976,"data_stored":2526503632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":211024325},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986295,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3763344,"kb_used_data":3439176,"kb_used_omap":1,"kb_used_meta":324094,"kb_avail":89981808,"statfs":{"total":95995035648,"available":92141371392,"internally_reserved":0,"allocated":3521716224,"data_stored":3520352432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":331872709},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149820,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2686216,"kb_used_data":2366136,"kb_used_omap":1,"kb_used_meta":319934,"kb_avail":91058936,"statfs":{"total":95995035648,"available":93244350464,"internally_reserved":0,"allocated":2422923264,"data_stored":2421567666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":327612871},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411457,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574982,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2811348,"kb_used_data":2496972,"kb_used_omap":1,"kb_used_meta":314366,"kb_avail":90933804,"statfs":{"total":95995035648,"available":93116215296,"internally_reserved":0,"allocated":2556899328,"data_stored":2555536056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":321911237},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738506,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2324496,"kb_used_data":2058628,"kb_used_omap":1,"kb_used_meta":265790,"kb_avail":91420656,"statfs":{"total":95995035648,"available":93614751744,"internally_reserved":0,"allocated":2108035072,"data_stored":2106675352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":272169413},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2071658496,"data_stored":2071658496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2523398144,"data_stored":2523398144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2391277568,"data_stored":2391277568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":3516272640,"data_stored":3516268568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2500202496,"data_stored":2500198424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2091646976,"data_stored":2091646976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":2221150208,"data_stored":2221146136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":2396454912,"data_stored":2396454912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":2312306688,"data_stored":2312306688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T14:18:45.976 INFO:tasks.thrashosds.thrasher:Setting 3.c to [10, 9, 1, 6, 4, 7] 2024-08-19T14:18:45.976 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '3.c', '10', '9', '1', '6', '4', '7'] 2024-08-19T14:18:45.977 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd pg-upmap-items 3.c 10 9 1 6 4 7 2024-08-19T14:18:46.014 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:18:46.015 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:18:46.039 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:18:46.056 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:18:46.084 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:46.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:46.086 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:18:46.187 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:18:46.223 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:46.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:46.224 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:18:46.324 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:18:46.354 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:46.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:46.355 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:18:46.456 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:18:46.491 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:46.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:46.493 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:18:46.594 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:18:46.630 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:46.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:46.631 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:18:46.733 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:18:46.763 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:46.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:46.764 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:18:46.848 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-19T14:18:46.865 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:18:46.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:46.932 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:46.933 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:18:47.034 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:18:47.082 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:47.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:47.083 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:18:47.141 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:18:47.141 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:18:47.180 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:18:47.181 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:18:47.184 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:18:47.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:47.214 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:47.215 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:18:47.215 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:18:47.216 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:18:47.250 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:18:47.250 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:18:47.255 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:18:47.281 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:18:47.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:47 smithi045 ceph-mon[20143]: pgmap v654: 67 pgs: 67 active+clean; 6.6 GiB data, 23 GiB used, 782 GiB / 805 GiB avail; 0 B/s wr, 2.38k op/s 2024-08-19T14:18:47.297 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:18:47.298 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:18:47.299 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:47 smithi031 ceph-mon[16336]: pgmap v654: 67 pgs: 67 active+clean; 6.6 GiB data, 23 GiB used, 782 GiB / 805 GiB avail; 0 B/s wr, 2.38k op/s 2024-08-19T14:18:47.316 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:18:47.327 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:18:47.328 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:18:47.345 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:47.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:47.346 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:18:47.360 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:18:47.360 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:18:47.374 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:47 smithi032 ceph-mon[20004]: pgmap v654: 67 pgs: 67 active+clean; 6.6 GiB data, 23 GiB used, 782 GiB / 805 GiB avail; 0 B/s wr, 2.38k op/s 2024-08-19T14:18:47.393 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:18:47.393 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:18:47.426 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:18:47.427 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:18:47.447 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:18:47.461 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:18:47.461 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:18:47.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:47.480 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:47.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:18:47.494 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:18:47.494 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:18:47.528 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:18:47.529 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:18:47.581 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:18:47.607 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:47.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:47.608 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:18:47.709 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:18:47.738 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:47.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:47.739 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:18:47.840 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:18:47.867 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:47.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:47.868 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:18:47.969 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:18:48.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:48.001 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:48.001 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:18:48.102 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:18:48.133 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:48.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:48.134 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:18:48.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:48 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/4039078676' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:18:48.235 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:18:48.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:48.258 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:48.259 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:18:48.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:48 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/4039078676' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:18:48.360 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:18:48.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:48 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/4039078676' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:18:48.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:48.609 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:48.609 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:18:48.710 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:18:48.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:48.790 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:48.790 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:18:48.890 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:18:48.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:48.920 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:48.920 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:18:49.022 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:18:49.054 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:49.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:49.056 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:18:49.057 INFO:teuthology.orchestra.run.smithi031.stderr:set 3.c pg_upmap_items mapping to [10->9,1->6,4->7] 2024-08-19T14:18:49.157 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:18:49.197 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:49.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:49.198 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:18:49.299 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:18:49.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:49.336 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:49.336 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:18:49.352 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:49 smithi031 ceph-mon[16336]: pgmap v655: 67 pgs: 67 active+clean; 6.4 GiB data, 22 GiB used, 782 GiB / 805 GiB avail; 0 B/s wr, 1.97k op/s 2024-08-19T14:18:49.353 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:49 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2468453014' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "3.c", "id": [10, 9, 1, 6, 4, 7]}]: dispatch 2024-08-19T14:18:49.438 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:18:49.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:49.470 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:49.470 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:18:49.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:49 smithi032 ceph-mon[20004]: pgmap v655: 67 pgs: 67 active+clean; 6.4 GiB data, 22 GiB used, 782 GiB / 805 GiB avail; 0 B/s wr, 1.97k op/s 2024-08-19T14:18:49.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:49 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2468453014' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "3.c", "id": [10, 9, 1, 6, 4, 7]}]: dispatch 2024-08-19T14:18:49.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:49 smithi045 ceph-mon[20143]: pgmap v655: 67 pgs: 67 active+clean; 6.4 GiB data, 22 GiB used, 782 GiB / 805 GiB avail; 0 B/s wr, 1.97k op/s 2024-08-19T14:18:49.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:49 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2468453014' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "3.c", "id": [10, 9, 1, 6, 4, 7]}]: dispatch 2024-08-19T14:18:49.571 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:18:49.601 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:49.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:49.602 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:18:49.703 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:18:49.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:49.736 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:49.736 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:18:49.837 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:18:49.872 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:49.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:49.873 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:18:49.973 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:18:50.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:50.008 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:50.009 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:18:50.066 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is unset 2024-08-19T14:18:50.109 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:18:50.154 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:50.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:50.155 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:18:50.181 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-08-19T14:18:50.256 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:18:50.302 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:50.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:50.303 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:18:50.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:50 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2468453014' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "3.c", "id": [10, 9, 1, 6, 4, 7]}]': finished 2024-08-19T14:18:50.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:50 smithi031 ceph-mon[16336]: osdmap e289: 12 total, 12 up, 9 in 2024-08-19T14:18:50.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:50 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1558174980' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:18:50.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:50 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:18:50.404 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:18:50.434 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:50.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:50.436 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:18:50.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:50 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2468453014' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "3.c", "id": [10, 9, 1, 6, 4, 7]}]': finished 2024-08-19T14:18:50.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:50 smithi032 ceph-mon[20004]: osdmap e289: 12 total, 12 up, 9 in 2024-08-19T14:18:50.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:50 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1558174980' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:18:50.509 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:50 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:18:50.537 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:18:50.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:50 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2468453014' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "3.c", "id": [10, 9, 1, 6, 4, 7]}]': finished 2024-08-19T14:18:50.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:50 smithi045 ceph-mon[20143]: osdmap e289: 12 total, 12 up, 9 in 2024-08-19T14:18:50.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:50 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1558174980' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:18:50.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:50 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:18:50.570 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:50.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:50.571 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:18:50.644 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:18:50.672 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:18:50.701 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:50.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:50.702 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:18:50.713 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:18:50.803 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:18:50.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:50.835 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:50.835 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:18:50.936 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:18:50.961 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:50.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:50.963 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:18:51.064 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:18:51.075 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:18:51.095 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:51.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:51.096 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:18:51.197 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:18:51.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:51.230 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:51.230 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:18:51.331 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:18:51.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:51 smithi031 ceph-mon[16336]: pgmap v657: 67 pgs: 67 active+clean; 6.1 GiB data, 21 GiB used, 783 GiB / 805 GiB avail; 0 B/s wr, 1.88k op/s 2024-08-19T14:18:51.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:51 smithi031 ceph-mon[16336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:18:51.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:51 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:18:51.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:51 smithi031 ceph-mon[16336]: osdmap e290: 12 total, 12 up, 9 in 2024-08-19T14:18:51.369 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:51.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:51.371 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:18:51.472 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:18:51.494 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:51.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:51.495 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:18:51.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:51 smithi032 ceph-mon[20004]: pgmap v657: 67 pgs: 67 active+clean; 6.1 GiB data, 21 GiB used, 783 GiB / 805 GiB avail; 0 B/s wr, 1.88k op/s 2024-08-19T14:18:51.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:51 smithi032 ceph-mon[20004]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:18:51.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:51 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:18:51.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:51 smithi032 ceph-mon[20004]: osdmap e290: 12 total, 12 up, 9 in 2024-08-19T14:18:51.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:51 smithi045 ceph-mon[20143]: pgmap v657: 67 pgs: 67 active+clean; 6.1 GiB data, 21 GiB used, 783 GiB / 805 GiB avail; 0 B/s wr, 1.88k op/s 2024-08-19T14:18:51.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:51 smithi045 ceph-mon[20143]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:18:51.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:51 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:18:51.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:51 smithi045 ceph-mon[20143]: osdmap e290: 12 total, 12 up, 9 in 2024-08-19T14:18:51.595 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:18:51.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:51.626 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:51.627 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:18:51.674 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:18:51.674 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:18:51.705 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:18:51.706 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:18:51.727 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:18:51.739 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:18:51.739 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:18:51.762 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:51.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:51.763 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:18:51.776 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:18:51.776 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:18:51.812 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:18:51.812 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:18:51.846 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:18:51.846 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:18:51.864 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:18:51.876 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:18:51.876 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:18:51.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:51.897 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:51.897 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:18:51.910 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:18:51.911 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:18:51.940 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:18:51.941 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:18:51.969 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:18:51.969 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:18:51.995 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:18:51.995 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:18:51.998 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:18:52.017 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:18:52.018 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:18:52.027 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:52.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:52.028 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:18:52.129 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:18:52.166 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:52.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:52.168 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:18:52.245 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:18:52.245 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:18:52.269 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:18:52.306 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:52.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:52.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:18:52.337 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:52 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3242945535' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:18:52.408 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:18:52.439 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:52.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:52.440 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:18:52.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:52 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3242945535' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:18:52.541 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:18:52.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:52 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3242945535' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:18:52.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:52.573 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:52.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:18:52.674 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:18:52.702 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:52.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:52.703 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:18:52.804 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:18:52.836 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:52.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:52.838 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:18:52.880 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-19T14:18:52.938 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:18:52.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:52.984 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:52.984 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:18:53.026 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-08-19T14:18:53.085 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:18:53.119 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:53.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:53.120 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:18:53.221 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:18:53.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:53.259 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:53.259 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:18:53.263 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:18:53.360 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:18:53.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:53 smithi031 ceph-mon[16336]: pgmap v659: 67 pgs: 67 active+clean; 5.8 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 0 B/s wr, 2.02k op/s 2024-08-19T14:18:53.418 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:53.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:53.419 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:18:53.465 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:18:53.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:53 smithi032 ceph-mon[20004]: pgmap v659: 67 pgs: 67 active+clean; 5.8 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 0 B/s wr, 2.02k op/s 2024-08-19T14:18:53.520 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:18:53.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:53 smithi045 ceph-mon[20143]: pgmap v659: 67 pgs: 67 active+clean; 5.8 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 0 B/s wr, 2.02k op/s 2024-08-19T14:18:53.551 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:53.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:53.552 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:18:53.653 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:18:53.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:53.693 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:53.694 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:18:53.795 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:18:53.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:53.828 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:53.828 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:18:53.929 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:18:53.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:53.976 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:53.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:18:54.078 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:18:54.097 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:18:54.098 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:18:54.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:54.114 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:54.114 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:18:54.215 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:18:54.248 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:54.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:54.249 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:18:54.350 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:18:54.379 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:54.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:54.380 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:18:54.481 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:18:54.516 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:54.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:54.517 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:18:54.618 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:18:54.650 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:54.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:54.651 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:18:54.752 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:18:54.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:54.790 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:54.790 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:18:54.807 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:18:54.891 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:18:54.918 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 7, 8, 9, 11] out_osds: [10, 6, 2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-08-19T14:18:54.918 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-19T14:18:54.919 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_1 2024-08-19T14:18:54.919 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-19T14:18:54.922 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:54.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:54.923 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:18:55.024 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:18:55.079 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:55.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:55.080 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:18:55.090 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is unset 2024-08-19T14:18:55.132 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-08-19T14:18:55.181 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:18:55.216 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:55.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:55.217 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:18:55.318 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:18:55.348 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:55 smithi045 ceph-mon[20143]: 1.1e scrub starts 2024-08-19T14:18:55.348 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:55 smithi045 ceph-mon[20143]: 1.1e scrub ok 2024-08-19T14:18:55.348 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:55 smithi045 ceph-mon[20143]: pgmap v660: 67 pgs: 67 active+clean; 5.6 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 1.92k op/s 2024-08-19T14:18:55.348 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:55 smithi045 ceph-mon[20143]: 1.1b scrub starts 2024-08-19T14:18:55.348 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:55 smithi045 ceph-mon[20143]: 1.1b scrub ok 2024-08-19T14:18:55.348 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:55 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:18:55.348 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:55 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:18:55.349 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:55 smithi045 ceph-mon[20143]: osdmap e291: 12 total, 12 up, 9 in 2024-08-19T14:18:55.349 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:55 smithi045 ceph-mon[20143]: 1.18 scrub starts 2024-08-19T14:18:55.349 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:55 smithi045 ceph-mon[20143]: 1.18 scrub ok 2024-08-19T14:18:55.349 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:55 smithi045 ceph-mon[20143]: 1.19 scrub starts 2024-08-19T14:18:55.349 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:55 smithi045 ceph-mon[20143]: 1.19 scrub ok 2024-08-19T14:18:55.349 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:55 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1331567015' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:18:55.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:55 smithi031 ceph-mon[16336]: 1.1e scrub starts 2024-08-19T14:18:55.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:55 smithi031 ceph-mon[16336]: 1.1e scrub ok 2024-08-19T14:18:55.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:55 smithi031 ceph-mon[16336]: pgmap v660: 67 pgs: 67 active+clean; 5.6 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 1.92k op/s 2024-08-19T14:18:55.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:55 smithi031 ceph-mon[16336]: 1.1b scrub starts 2024-08-19T14:18:55.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:55 smithi031 ceph-mon[16336]: 1.1b scrub ok 2024-08-19T14:18:55.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:55 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:18:55.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:55 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:18:55.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:55 smithi031 ceph-mon[16336]: osdmap e291: 12 total, 12 up, 9 in 2024-08-19T14:18:55.368 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:55 smithi031 ceph-mon[16336]: 1.18 scrub starts 2024-08-19T14:18:55.368 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:55 smithi031 ceph-mon[16336]: 1.18 scrub ok 2024-08-19T14:18:55.368 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:55 smithi031 ceph-mon[16336]: 1.19 scrub starts 2024-08-19T14:18:55.368 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:55 smithi031 ceph-mon[16336]: 1.19 scrub ok 2024-08-19T14:18:55.368 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:55 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1331567015' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:18:55.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:55.375 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:55.376 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:18:55.476 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:18:55.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:55 smithi032 ceph-mon[20004]: 1.1e scrub starts 2024-08-19T14:18:55.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:55 smithi032 ceph-mon[20004]: 1.1e scrub ok 2024-08-19T14:18:55.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:55 smithi032 ceph-mon[20004]: pgmap v660: 67 pgs: 67 active+clean; 5.6 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 1.92k op/s 2024-08-19T14:18:55.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:55 smithi032 ceph-mon[20004]: 1.1b scrub starts 2024-08-19T14:18:55.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:55 smithi032 ceph-mon[20004]: 1.1b scrub ok 2024-08-19T14:18:55.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:55 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:18:55.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:55 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:18:55.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:55 smithi032 ceph-mon[20004]: osdmap e291: 12 total, 12 up, 9 in 2024-08-19T14:18:55.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:55 smithi032 ceph-mon[20004]: 1.18 scrub starts 2024-08-19T14:18:55.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:55 smithi032 ceph-mon[20004]: 1.18 scrub ok 2024-08-19T14:18:55.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:55 smithi032 ceph-mon[20004]: 1.19 scrub starts 2024-08-19T14:18:55.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:55 smithi032 ceph-mon[20004]: 1.19 scrub ok 2024-08-19T14:18:55.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:55 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1331567015' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:18:55.533 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:55.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:55.534 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:18:55.554 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:18:55.635 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:18:55.666 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:55.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:55.667 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:18:55.768 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:18:55.804 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:55.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:55.805 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:18:55.845 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:18:55.864 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:18:55.906 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:18:55.931 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:55.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:55.932 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:18:56.033 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:18:56.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:56.068 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:56.068 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:18:56.162 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:18:56.162 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:18:56.169 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:18:56.197 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:56.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:56.198 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:18:56.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:56 smithi032 ceph-mon[20004]: 1.16 scrub starts 2024-08-19T14:18:56.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:56 smithi032 ceph-mon[20004]: 1.16 scrub ok 2024-08-19T14:18:56.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:56 smithi032 ceph-mon[20004]: 1.1a scrub starts 2024-08-19T14:18:56.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:56 smithi032 ceph-mon[20004]: 1.1a scrub ok 2024-08-19T14:18:56.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:56 smithi032 ceph-mon[20004]: 1.17 scrub starts 2024-08-19T14:18:56.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:56 smithi032 ceph-mon[20004]: 1.17 scrub ok 2024-08-19T14:18:56.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:56 smithi032 ceph-mon[20004]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:18:56.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:56 smithi032 ceph-mon[20004]: Cluster is now healthy 2024-08-19T14:18:56.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:56 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1331567015' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:18:56.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:56 smithi032 ceph-mon[20004]: osdmap e292: 12 total, 12 up, 9 in 2024-08-19T14:18:56.299 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:18:56.332 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:56.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:56.333 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:18:56.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:56 smithi031 ceph-mon[16336]: 1.16 scrub starts 2024-08-19T14:18:56.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:56 smithi031 ceph-mon[16336]: 1.16 scrub ok 2024-08-19T14:18:56.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:56 smithi031 ceph-mon[16336]: 1.1a scrub starts 2024-08-19T14:18:56.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:56 smithi031 ceph-mon[16336]: 1.1a scrub ok 2024-08-19T14:18:56.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:56 smithi031 ceph-mon[16336]: 1.17 scrub starts 2024-08-19T14:18:56.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:56 smithi031 ceph-mon[16336]: 1.17 scrub ok 2024-08-19T14:18:56.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:56 smithi031 ceph-mon[16336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:18:56.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:56 smithi031 ceph-mon[16336]: Cluster is now healthy 2024-08-19T14:18:56.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:56 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1331567015' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:18:56.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:56 smithi031 ceph-mon[16336]: osdmap e292: 12 total, 12 up, 9 in 2024-08-19T14:18:56.434 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:18:56.468 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:56.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:56.469 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:18:56.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:56 smithi045 ceph-mon[20143]: 1.16 scrub starts 2024-08-19T14:18:56.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:56 smithi045 ceph-mon[20143]: 1.16 scrub ok 2024-08-19T14:18:56.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:56 smithi045 ceph-mon[20143]: 1.1a scrub starts 2024-08-19T14:18:56.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:56 smithi045 ceph-mon[20143]: 1.1a scrub ok 2024-08-19T14:18:56.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:56 smithi045 ceph-mon[20143]: 1.17 scrub starts 2024-08-19T14:18:56.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:56 smithi045 ceph-mon[20143]: 1.17 scrub ok 2024-08-19T14:18:56.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:56 smithi045 ceph-mon[20143]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:18:56.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:56 smithi045 ceph-mon[20143]: Cluster is now healthy 2024-08-19T14:18:56.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:56 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1331567015' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:18:56.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:56 smithi045 ceph-mon[20143]: osdmap e292: 12 total, 12 up, 9 in 2024-08-19T14:18:56.570 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:18:56.608 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:56.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:56.609 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:18:56.710 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:18:56.743 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:56.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:56.744 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:18:56.844 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:18:56.870 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-08-19T14:18:56.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:56.876 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:56.876 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:18:56.977 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:18:57.003 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:57.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:57.004 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:18:57.091 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:18:57.092 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:18:57.104 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:18:57.122 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:18:57.122 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:18:57.131 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:57.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:57.133 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:18:57.150 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:18:57.151 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:18:57.177 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:18:57.178 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:18:57.205 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:18:57.205 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:18:57.206 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:57 smithi031 ceph-mon[16336]: 1.1d scrub starts 2024-08-19T14:18:57.206 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:57 smithi031 ceph-mon[16336]: 1.1d scrub ok 2024-08-19T14:18:57.206 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:57 smithi031 ceph-mon[16336]: 1.15 deep-scrub starts 2024-08-19T14:18:57.206 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:57 smithi031 ceph-mon[16336]: 1.15 deep-scrub ok 2024-08-19T14:18:57.206 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:57 smithi031 ceph-mon[16336]: pgmap v663: 67 pgs: 67 active+clean; 5.4 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 1.90k op/s 2024-08-19T14:18:57.206 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:57 smithi031 ceph-mon[16336]: 1.1c scrub starts 2024-08-19T14:18:57.206 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:57 smithi031 ceph-mon[16336]: 1.1c scrub ok 2024-08-19T14:18:57.206 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:57 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/846106138' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:18:57.232 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:18:57.233 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:18:57.233 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:18:57.260 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:18:57.260 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:18:57.279 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:57.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:57.280 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:18:57.290 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:18:57.291 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:18:57.310 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:18:57.314 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:18:57.314 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:18:57.346 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:18:57.346 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:18:57.371 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:18:57.371 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:18:57.382 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:18:57.405 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:18:57.405 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:18:57.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:57.421 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:57.421 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:18:57.447 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:57 smithi032 ceph-mon[20004]: 1.1d scrub starts 2024-08-19T14:18:57.448 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:57 smithi032 ceph-mon[20004]: 1.1d scrub ok 2024-08-19T14:18:57.448 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:57 smithi032 ceph-mon[20004]: 1.15 deep-scrub starts 2024-08-19T14:18:57.448 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:57 smithi032 ceph-mon[20004]: 1.15 deep-scrub ok 2024-08-19T14:18:57.448 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:57 smithi032 ceph-mon[20004]: pgmap v663: 67 pgs: 67 active+clean; 5.4 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 1.90k op/s 2024-08-19T14:18:57.448 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:57 smithi032 ceph-mon[20004]: 1.1c scrub starts 2024-08-19T14:18:57.448 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:57 smithi032 ceph-mon[20004]: 1.1c scrub ok 2024-08-19T14:18:57.448 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:57 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/846106138' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:18:57.522 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:18:57.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:57 smithi045 ceph-mon[20143]: 1.1d scrub starts 2024-08-19T14:18:57.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:57 smithi045 ceph-mon[20143]: 1.1d scrub ok 2024-08-19T14:18:57.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:57 smithi045 ceph-mon[20143]: 1.15 deep-scrub starts 2024-08-19T14:18:57.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:57 smithi045 ceph-mon[20143]: 1.15 deep-scrub ok 2024-08-19T14:18:57.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:57 smithi045 ceph-mon[20143]: pgmap v663: 67 pgs: 67 active+clean; 5.4 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 1.90k op/s 2024-08-19T14:18:57.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:57 smithi045 ceph-mon[20143]: 1.1c scrub starts 2024-08-19T14:18:57.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:57 smithi045 ceph-mon[20143]: 1.1c scrub ok 2024-08-19T14:18:57.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:57 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/846106138' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:18:57.558 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:57.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:57.559 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:18:57.660 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:18:57.663 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:18:57.664 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-08-19T14:18:57.698 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:57.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:57.699 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:18:57.800 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:18:57.805 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-19T14:18:57.841 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:57.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:57.842 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:18:57.925 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:18:57.925 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-08-19T14:18:57.925 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-08-19T14:18:57.925 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-08-19T14:18:57.925 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.925 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:62938 2.2 2.749ad082 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-08-19T14:18:57.926 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:18.730377+0000", 2024-08-19T14:18:57.926 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 339.19262537600002, 2024-08-19T14:18:57.926 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.98852832099999999, 2024-08-19T14:18:57.926 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:18:57.926 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:18:57.926 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:18:57.926 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:18:57.926 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:18:57.926 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:18:57.926 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 62938 2024-08-19T14:18:57.926 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.926 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:18:57.926 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.927 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:18:57.927 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.730377+0000", 2024-08-19T14:18:57.927 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:18:57.927 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.927 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.927 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:18:57.927 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.730377+0000", 2024-08-19T14:18:57.927 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:18:57.927 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.927 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.927 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:18:57.927 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.730380+0000", 2024-08-19T14:18:57.928 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5129999999999999e-06 2024-08-19T14:18:57.928 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.928 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.928 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:18:57.928 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.730416+0000", 2024-08-19T14:18:57.928 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5981e-05 2024-08-19T14:18:57.928 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.928 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.928 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:18:57.928 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.730441+0000", 2024-08-19T14:18:57.928 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4964e-05 2024-08-19T14:18:57.928 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.928 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.928 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:18:57.929 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.730450+0000", 2024-08-19T14:18:57.929 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.5850000000000006e-06 2024-08-19T14:18:57.929 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.929 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.929 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:18:57.929 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.730530+0000", 2024-08-19T14:18:57.929 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.9703999999999994e-05 2024-08-19T14:18:57.929 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.929 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.929 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:18:57.929 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.730544+0000", 2024-08-19T14:18:57.929 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4229e-05 2024-08-19T14:18:57.929 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.929 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.930 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:18:57.930 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.718832+0000", 2024-08-19T14:18:57.930 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.98828802699999996 2024-08-19T14:18:57.930 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.930 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.930 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:18:57.930 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.718905+0000", 2024-08-19T14:18:57.930 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3318000000000005e-05 2024-08-19T14:18:57.930 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:18:57.930 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:18:57.930 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:18:57.930 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.930 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.931 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:62942 2.2 2.8241cf42 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-08-19T14:18:57.931 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:18.731498+0000", 2024-08-19T14:18:57.931 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 339.19150470300002, 2024-08-19T14:18:57.931 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.98803843599999996, 2024-08-19T14:18:57.931 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:18:57.931 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:18:57.931 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:18:57.931 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:18:57.931 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:18:57.931 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:18:57.931 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 62942 2024-08-19T14:18:57.931 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.931 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:18:57.931 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.932 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:18:57.932 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.731498+0000", 2024-08-19T14:18:57.932 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:18:57.932 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.933 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.933 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:18:57.934 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.731498+0000", 2024-08-19T14:18:57.934 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:18:57.934 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.934 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.934 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:18:57.934 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.731499+0000", 2024-08-19T14:18:57.934 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.4600000000000003e-07 2024-08-19T14:18:57.934 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.934 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.934 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:18:57.934 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.731572+0000", 2024-08-19T14:18:57.934 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2946000000000003e-05 2024-08-19T14:18:57.935 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.935 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.935 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:18:57.935 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.731584+0000", 2024-08-19T14:18:57.935 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2272e-05 2024-08-19T14:18:57.935 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.935 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.935 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:18:57.935 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.731591+0000", 2024-08-19T14:18:57.935 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6290000000000001e-06 2024-08-19T14:18:57.935 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.935 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.935 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:18:57.936 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.731642+0000", 2024-08-19T14:18:57.936 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0930999999999997e-05 2024-08-19T14:18:57.936 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.936 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.936 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:18:57.936 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.731650+0000", 2024-08-19T14:18:57.936 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.554e-06 2024-08-19T14:18:57.936 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.936 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.936 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:18:57.936 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.719522+0000", 2024-08-19T14:18:57.936 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.98787217500000002 2024-08-19T14:18:57.936 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.936 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.937 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:18:57.937 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.719536+0000", 2024-08-19T14:18:57.937 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4083000000000001e-05 2024-08-19T14:18:57.937 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:18:57.937 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:18:57.937 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:18:57.937 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.938 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.938 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:62947 2.2 2.528f01a2 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-08-19T14:18:57.938 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:18.735148+0000", 2024-08-19T14:18:57.938 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 339.18785424800001, 2024-08-19T14:18:57.938 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.98444280500000003, 2024-08-19T14:18:57.938 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:18:57.939 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:18:57.939 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:18:57.939 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:18:57.939 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:18:57.939 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:18:57.939 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 62947 2024-08-19T14:18:57.939 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.939 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:18:57.939 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.939 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:18:57.939 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.735148+0000", 2024-08-19T14:18:57.940 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:18:57.940 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.940 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.940 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:18:57.940 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.735148+0000", 2024-08-19T14:18:57.940 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:18:57.940 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.940 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.940 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:18:57.940 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.735150+0000", 2024-08-19T14:18:57.940 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.903e-06 2024-08-19T14:18:57.940 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.940 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.941 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:18:57.941 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.735192+0000", 2024-08-19T14:18:57.941 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1780999999999999e-05 2024-08-19T14:18:57.941 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.941 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.941 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:18:57.941 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.735211+0000", 2024-08-19T14:18:57.941 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9021e-05 2024-08-19T14:18:57.941 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.941 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.941 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:18:57.941 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.735221+0000", 2024-08-19T14:18:57.941 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.9140000000000003e-06 2024-08-19T14:18:57.941 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.941 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.942 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:18:57.942 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.735291+0000", 2024-08-19T14:18:57.942 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0599999999999995e-05 2024-08-19T14:18:57.942 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.942 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.942 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:18:57.942 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.735304+0000", 2024-08-19T14:18:57.942 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2218999999999999e-05 2024-08-19T14:18:57.942 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.942 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.942 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:18:57.942 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.719582+0000", 2024-08-19T14:18:57.942 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.98427827999999995 2024-08-19T14:18:57.943 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.943 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.943 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:18:57.943 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.719591+0000", 2024-08-19T14:18:57.943 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.0869999999999996e-06 2024-08-19T14:18:57.943 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:18:57.943 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:18:57.943 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:18:57.943 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.944 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.944 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:62961 2.2 2.d8a35c52 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-08-19T14:18:57.944 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:18.743735+0000", 2024-08-19T14:18:57.944 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 339.17926750499998, 2024-08-19T14:18:57.944 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.97590511300000005, 2024-08-19T14:18:57.944 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:18:57.944 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:18:57.944 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:18:57.944 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:18:57.944 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:18:57.944 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:18:57.944 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 62961 2024-08-19T14:18:57.945 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.945 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:18:57.945 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.946 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:18:57.946 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.743735+0000", 2024-08-19T14:18:57.946 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:18:57.946 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.946 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.946 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:18:57.947 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.743735+0000", 2024-08-19T14:18:57.947 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:18:57.947 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.947 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.947 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:18:57.947 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.743737+0000", 2024-08-19T14:18:57.947 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1689999999999999e-06 2024-08-19T14:18:57.947 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.947 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.948 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:18:57.948 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.743772+0000", 2024-08-19T14:18:57.948 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5293000000000002e-05 2024-08-19T14:18:57.948 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.948 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.948 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:18:57.948 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.743795+0000", 2024-08-19T14:18:57.949 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2167e-05 2024-08-19T14:18:57.949 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.949 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.950 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:18:57.950 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.743804+0000", 2024-08-19T14:18:57.950 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3379999999999994e-06 2024-08-19T14:18:57.950 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.950 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.950 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:18:57.950 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.743875+0000", 2024-08-19T14:18:57.950 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1073000000000002e-05 2024-08-19T14:18:57.950 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.951 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.951 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:18:57.951 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.743888+0000", 2024-08-19T14:18:57.951 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2622999999999999e-05 2024-08-19T14:18:57.951 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.951 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.951 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:18:57.951 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.719630+0000", 2024-08-19T14:18:57.951 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.97574243299999996 2024-08-19T14:18:57.951 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.951 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.951 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:18:57.951 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.719640+0000", 2024-08-19T14:18:57.951 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0017000000000001e-05 2024-08-19T14:18:57.952 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:18:57.952 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:18:57.952 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:18:57.952 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.952 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.952 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:62991 2.2 2.365dd392 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-08-19T14:18:57.952 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:18.765407+0000", 2024-08-19T14:18:57.952 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 339.15759500500002, 2024-08-19T14:18:57.952 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.95428210599999996, 2024-08-19T14:18:57.952 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:18:57.952 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:18:57.953 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:18:57.953 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:18:57.953 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:18:57.953 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:18:57.953 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 62991 2024-08-19T14:18:57.953 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.953 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:18:57.953 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.953 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:18:57.953 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.765407+0000", 2024-08-19T14:18:57.953 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:18:57.953 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.954 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.954 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:18:57.954 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.765407+0000", 2024-08-19T14:18:57.954 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:18:57.954 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.954 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.954 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:18:57.954 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.765409+0000", 2024-08-19T14:18:57.954 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0950000000000001e-06 2024-08-19T14:18:57.954 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.954 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.955 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:18:57.955 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.765456+0000", 2024-08-19T14:18:57.955 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7061000000000002e-05 2024-08-19T14:18:57.955 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.955 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.955 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:18:57.955 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.765473+0000", 2024-08-19T14:18:57.956 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7377e-05 2024-08-19T14:18:57.956 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.956 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.956 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:18:57.956 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.765480+0000", 2024-08-19T14:18:57.956 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7889999999999997e-06 2024-08-19T14:18:57.956 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.956 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.956 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:18:57.957 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.765536+0000", 2024-08-19T14:18:57.957 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5897999999999999e-05 2024-08-19T14:18:57.957 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.957 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.957 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:18:57.957 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.765545+0000", 2024-08-19T14:18:57.957 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.4939999999999992e-06 2024-08-19T14:18:57.957 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.957 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.957 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:18:57.957 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.719682+0000", 2024-08-19T14:18:57.958 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.95413691499999997 2024-08-19T14:18:57.958 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.958 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.958 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:18:57.958 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.719690+0000", 2024-08-19T14:18:57.959 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4769999999999997e-06 2024-08-19T14:18:57.959 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:18:57.959 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:18:57.959 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:18:57.959 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.959 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.959 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:62995 2.2 2.d0be02c2 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-08-19T14:18:57.959 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:18.768402+0000", 2024-08-19T14:18:57.959 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 339.15460066999998, 2024-08-19T14:18:57.961 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.95133469699999995, 2024-08-19T14:18:57.961 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:18:57.961 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:18:57.961 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:18:57.961 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:18:57.961 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:18:57.961 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:18:57.962 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 62995 2024-08-19T14:18:57.962 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.962 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:18:57.962 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.962 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:18:57.962 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.768402+0000", 2024-08-19T14:18:57.963 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:18:57.963 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.963 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.963 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:18:57.963 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.768402+0000", 2024-08-19T14:18:57.963 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:18:57.963 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.963 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.963 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:18:57.963 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.768402+0000", 2024-08-19T14:18:57.964 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7000000000000004e-07 2024-08-19T14:18:57.964 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.964 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.964 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:18:57.964 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.768439+0000", 2024-08-19T14:18:57.964 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6641999999999999e-05 2024-08-19T14:18:57.964 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.964 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.964 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:18:57.964 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.768450+0000", 2024-08-19T14:18:57.964 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0635999999999999e-05 2024-08-19T14:18:57.964 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.964 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.964 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:18:57.965 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.768455+0000", 2024-08-19T14:18:57.965 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9840000000000001e-06 2024-08-19T14:18:57.965 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.965 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.965 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:18:57.965 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.768498+0000", 2024-08-19T14:18:57.965 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3350000000000003e-05 2024-08-19T14:18:57.965 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.965 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.965 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:18:57.965 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.768504+0000", 2024-08-19T14:18:57.965 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3380000000000001e-06 2024-08-19T14:18:57.965 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.965 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.966 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:18:57.966 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:18:57.967 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.719726+0000", 2024-08-19T14:18:57.968 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.95122155900000005 2024-08-19T14:18:57.968 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.968 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.968 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:18:57.968 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.719736+0000", 2024-08-19T14:18:57.968 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0518e-05 2024-08-19T14:18:57.968 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:18:57.968 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:18:57.968 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:18:57.969 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.969 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.969 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:63016 2.2 2.3ca449b2 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-08-19T14:18:57.969 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:18.789614+0000", 2024-08-19T14:18:57.969 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 339.13338818400001, 2024-08-19T14:18:57.969 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.93017577600000001, 2024-08-19T14:18:57.969 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:18:57.969 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:18:57.969 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:18:57.969 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:18:57.969 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:18:57.969 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:18:57.969 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 63016 2024-08-19T14:18:57.970 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.970 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:18:57.970 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.970 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:18:57.970 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.789614+0000", 2024-08-19T14:18:57.970 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:18:57.970 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.970 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.970 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:18:57.970 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.789614+0000", 2024-08-19T14:18:57.970 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:18:57.970 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.970 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.970 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:18:57.971 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.789616+0000", 2024-08-19T14:18:57.971 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.327e-06 2024-08-19T14:18:57.971 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.971 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.971 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:18:57.971 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.789658+0000", 2024-08-19T14:18:57.971 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1934000000000002e-05 2024-08-19T14:18:57.971 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.971 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.971 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:18:57.971 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.789667+0000", 2024-08-19T14:18:57.971 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.6549999999999999e-06 2024-08-19T14:18:57.971 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.972 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.972 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:18:57.972 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.789672+0000", 2024-08-19T14:18:57.972 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0080000000000002e-06 2024-08-19T14:18:57.972 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.972 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.972 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:18:57.972 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.789732+0000", 2024-08-19T14:18:57.972 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9568999999999998e-05 2024-08-19T14:18:57.972 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.972 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.972 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:18:57.972 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.789742+0000", 2024-08-19T14:18:57.972 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0740000000000001e-05 2024-08-19T14:18:57.973 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.973 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.973 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:18:57.973 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.719765+0000", 2024-08-19T14:18:57.973 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.930022723 2024-08-19T14:18:57.973 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.973 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.973 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:18:57.973 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.719790+0000", 2024-08-19T14:18:57.973 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.482e-05 2024-08-19T14:18:57.973 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:18:57.973 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:18:57.973 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:18:57.973 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.974 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.974 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:63026 2.2 2.519434f2 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-08-19T14:18:57.974 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:18.799709+0000", 2024-08-19T14:18:57.974 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 339.12329327700002, 2024-08-19T14:18:57.974 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.92011866099999995, 2024-08-19T14:18:57.974 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:18:57.974 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:18:57.974 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:18:57.974 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:18:57.974 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:18:57.974 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:18:57.974 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 63026 2024-08-19T14:18:57.975 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.975 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:18:57.975 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.975 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:18:57.975 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.799709+0000", 2024-08-19T14:18:57.975 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:18:57.975 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.975 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.975 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:18:57.975 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.799709+0000", 2024-08-19T14:18:57.975 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:18:57.976 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.976 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.976 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:18:57.976 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.799710+0000", 2024-08-19T14:18:57.976 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.333e-06 2024-08-19T14:18:57.976 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.976 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.976 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:18:57.976 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.799743+0000", 2024-08-19T14:18:57.976 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2845000000000001e-05 2024-08-19T14:18:57.976 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.976 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.976 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:18:57.977 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.799757+0000", 2024-08-19T14:18:57.977 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3655999999999999e-05 2024-08-19T14:18:57.977 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.977 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.977 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:18:57.977 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.799764+0000", 2024-08-19T14:18:57.977 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2760000000000003e-06 2024-08-19T14:18:57.977 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.977 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.977 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:18:57.977 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.799815+0000", 2024-08-19T14:18:57.977 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0269000000000003e-05 2024-08-19T14:18:57.977 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.977 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.978 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:18:57.978 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.799822+0000", 2024-08-19T14:18:57.978 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7840000000000004e-06 2024-08-19T14:18:57.978 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.978 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.978 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:18:57.978 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.719821+0000", 2024-08-19T14:18:57.978 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.91999866900000005 2024-08-19T14:18:57.978 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.978 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.978 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:18:57.978 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.719828+0000", 2024-08-19T14:18:57.978 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8290000000000002e-06 2024-08-19T14:18:57.979 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:18:57.979 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:18:57.979 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:18:57.979 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.979 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.979 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:63031 2.2 2.92065272 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-08-19T14:18:57.979 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:18.809226+0000", 2024-08-19T14:18:57.979 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 339.11377607399999, 2024-08-19T14:18:57.979 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.91063533200000002, 2024-08-19T14:18:57.979 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:18:57.979 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:18:57.979 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:18:57.979 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:18:57.979 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:18:57.980 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:18:57.980 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 63031 2024-08-19T14:18:57.980 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.980 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:18:57.980 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.980 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:18:57.980 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.809226+0000", 2024-08-19T14:18:57.980 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:18:57.980 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.980 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.980 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:18:57.980 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.809226+0000", 2024-08-19T14:18:57.980 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:18:57.980 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.981 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.981 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:18:57.981 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.809228+0000", 2024-08-19T14:18:57.981 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4670000000000001e-06 2024-08-19T14:18:57.981 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.981 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.981 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:18:57.981 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.809267+0000", 2024-08-19T14:18:57.981 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8881000000000003e-05 2024-08-19T14:18:57.981 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.981 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.981 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:18:57.981 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.809282+0000", 2024-08-19T14:18:57.982 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5333e-05 2024-08-19T14:18:57.982 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.982 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.982 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:18:57.982 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.809289+0000", 2024-08-19T14:18:57.982 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7299999999999999e-06 2024-08-19T14:18:57.982 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.982 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.982 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:18:57.982 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.809352+0000", 2024-08-19T14:18:57.982 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3639000000000005e-05 2024-08-19T14:18:57.982 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.982 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.983 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:18:57.983 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.809364+0000", 2024-08-19T14:18:57.983 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1995e-05 2024-08-19T14:18:57.983 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.983 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.983 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:18:57.983 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.719856+0000", 2024-08-19T14:18:57.983 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.91049139999999995 2024-08-19T14:18:57.983 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.984 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.984 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:18:57.984 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.719862+0000", 2024-08-19T14:18:57.984 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8869999999999997e-06 2024-08-19T14:18:57.984 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:18:57.984 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:18:57.984 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:18:57.984 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.984 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.984 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:171107 2.3 2:cdcc5df6:::benchmark_data_smithi045_46415_object47826:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e169)", 2024-08-19T14:18:57.984 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:15:12.315407+0000", 2024-08-19T14:18:57.985 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 225.60759532500001, 2024-08-19T14:18:57.985 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.23111446799999999, 2024-08-19T14:18:57.985 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:18:57.985 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:18:57.985 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:18:57.985 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:18:57.985 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:18:57.985 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:18:57.985 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 171107 2024-08-19T14:18:57.985 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.985 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:18:57.985 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.985 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:18:57.986 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.315407+0000", 2024-08-19T14:18:57.986 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:18:57.986 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.986 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.986 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:18:57.986 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.315407+0000", 2024-08-19T14:18:57.986 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:18:57.986 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.986 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.986 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:18:57.987 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.315408+0000", 2024-08-19T14:18:57.987 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.094e-06 2024-08-19T14:18:57.987 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.987 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.987 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:18:57.987 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.315413+0000", 2024-08-19T14:18:57.987 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2070000000000003e-06 2024-08-19T14:18:57.987 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.987 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.987 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:18:57.987 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.315424+0000", 2024-08-19T14:18:57.987 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0873000000000001e-05 2024-08-19T14:18:57.988 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.988 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.988 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:18:57.988 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.315431+0000", 2024-08-19T14:18:57.988 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1149999999999996e-06 2024-08-19T14:18:57.988 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.988 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.988 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:18:57.988 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.315490+0000", 2024-08-19T14:18:57.989 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8118000000000002e-05 2024-08-19T14:18:57.989 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.989 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.989 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:18:57.989 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.315625+0000", 2024-08-19T14:18:57.989 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000135902 2024-08-19T14:18:57.989 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.989 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.989 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,7", 2024-08-19T14:18:57.989 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.315754+0000", 2024-08-19T14:18:57.989 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012891700000000001 2024-08-19T14:18:57.990 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.990 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.990 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:18:57.990 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.320475+0000", 2024-08-19T14:18:57.990 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0047206069999999999 2024-08-19T14:18:57.990 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.990 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.990 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:18:57.990 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.321696+0000", 2024-08-19T14:18:57.990 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00122135 2024-08-19T14:18:57.990 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.990 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.990 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:18:57.991 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.546437+0000", 2024-08-19T14:18:57.991 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.22474116199999999 2024-08-19T14:18:57.991 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.991 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.991 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:18:57.991 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.546477+0000", 2024-08-19T14:18:57.991 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9263e-05 2024-08-19T14:18:57.991 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.991 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.991 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:18:57.992 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.546522+0000", 2024-08-19T14:18:57.992 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4860000000000001e-05 2024-08-19T14:18:57.992 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:18:57.992 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:18:57.992 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:18:57.992 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.992 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.992 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:171113 2.3 2:cfe749bb:::benchmark_data_smithi045_46415_object47832:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e169)", 2024-08-19T14:18:57.993 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:15:12.322593+0000", 2024-08-19T14:18:57.993 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 225.600409652, 2024-08-19T14:18:57.993 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.228733402, 2024-08-19T14:18:57.993 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:18:57.993 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:18:57.993 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:18:57.993 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:18:57.993 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:18:57.993 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:18:57.993 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 171113 2024-08-19T14:18:57.993 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.994 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:18:57.994 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.994 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:18:57.994 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.322593+0000", 2024-08-19T14:18:57.994 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:18:57.994 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.994 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.994 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:18:57.994 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.322593+0000", 2024-08-19T14:18:57.994 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:18:57.994 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.995 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.995 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:18:57.995 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.322594+0000", 2024-08-19T14:18:57.995 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2240000000000001e-06 2024-08-19T14:18:57.995 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.995 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.995 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:18:57.995 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.322603+0000", 2024-08-19T14:18:57.995 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6910000000000003e-06 2024-08-19T14:18:57.995 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.995 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.995 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:18:57.996 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.322615+0000", 2024-08-19T14:18:57.996 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2181000000000001e-05 2024-08-19T14:18:57.996 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.996 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.996 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:18:57.996 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.322620+0000", 2024-08-19T14:18:57.996 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3070000000000002e-06 2024-08-19T14:18:57.996 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.996 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.996 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:18:57.996 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.322677+0000", 2024-08-19T14:18:57.996 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6407999999999999e-05 2024-08-19T14:18:57.996 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.996 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.997 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:18:57.997 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.322808+0000", 2024-08-19T14:18:57.997 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013162499999999999 2024-08-19T14:18:57.997 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.997 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.998 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,7", 2024-08-19T14:18:57.998 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.322936+0000", 2024-08-19T14:18:57.998 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000127358 2024-08-19T14:18:57.998 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.998 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.998 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:18:57.998 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.332145+0000", 2024-08-19T14:18:57.998 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0092095149999999997 2024-08-19T14:18:57.998 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.998 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.999 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:18:57.999 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.333304+0000", 2024-08-19T14:18:57.999 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0011586330000000001 2024-08-19T14:18:57.999 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.999 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.999 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:18:57.999 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.551228+0000", 2024-08-19T14:18:57.999 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.2179239 2024-08-19T14:18:57.999 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:57.999 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:57.999 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:18:57.999 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.551282+0000", 2024-08-19T14:18:57.999 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4080999999999999e-05 2024-08-19T14:18:57.999 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.000 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.000 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:18:58.000 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.551326+0000", 2024-08-19T14:18:58.000 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4478999999999999e-05 2024-08-19T14:18:58.000 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:18:58.000 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:18:58.000 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:18:58.000 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.000 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.001 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:171114 2.3 2:cce6d5f2:::benchmark_data_smithi045_46415_object47833:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e169)", 2024-08-19T14:18:58.001 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:15:12.322676+0000", 2024-08-19T14:18:58.001 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 225.600326051, 2024-08-19T14:18:58.001 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.228818197, 2024-08-19T14:18:58.001 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:18:58.001 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:18:58.001 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:18:58.001 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:18:58.001 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:18:58.001 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:18:58.001 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 171114 2024-08-19T14:18:58.001 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.001 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:18:58.001 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.002 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:18:58.002 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.322676+0000", 2024-08-19T14:18:58.002 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:18:58.002 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.002 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.002 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:18:58.002 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.322676+0000", 2024-08-19T14:18:58.002 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:18:58.002 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.002 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.002 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:18:58.002 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.322677+0000", 2024-08-19T14:18:58.002 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7999999999999996e-07 2024-08-19T14:18:58.002 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.003 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.003 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:18:58.003 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.322680+0000", 2024-08-19T14:18:58.003 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9519999999999999e-06 2024-08-19T14:18:58.003 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.003 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.003 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:18:58.003 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.322689+0000", 2024-08-19T14:18:58.003 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.6879999999999994e-06 2024-08-19T14:18:58.003 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.003 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.003 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:18:58.003 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.322693+0000", 2024-08-19T14:18:58.003 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9910000000000004e-06 2024-08-19T14:18:58.004 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.004 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.004 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:18:58.004 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.327524+0000", 2024-08-19T14:18:58.004 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0048305259999999999 2024-08-19T14:18:58.004 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.004 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.004 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:18:58.004 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.327692+0000", 2024-08-19T14:18:58.004 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000167956 2024-08-19T14:18:58.004 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.004 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.004 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,7", 2024-08-19T14:18:58.004 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.327829+0000", 2024-08-19T14:18:58.004 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013674999999999999 2024-08-19T14:18:58.005 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.005 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.005 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:18:58.005 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.332259+0000", 2024-08-19T14:18:58.005 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0044300859999999997 2024-08-19T14:18:58.005 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.005 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.005 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:18:58.005 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.333546+0000", 2024-08-19T14:18:58.005 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001287683 2024-08-19T14:18:58.005 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.005 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.005 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:18:58.005 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.551388+0000", 2024-08-19T14:18:58.006 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.21784162100000001 2024-08-19T14:18:58.006 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.006 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.006 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:18:58.006 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.551451+0000", 2024-08-19T14:18:58.006 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2556000000000002e-05 2024-08-19T14:18:58.006 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.006 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.006 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:18:58.006 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.551495+0000", 2024-08-19T14:18:58.006 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3908e-05 2024-08-19T14:18:58.006 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:18:58.006 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:18:58.006 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:18:58.007 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.007 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.007 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:177246 2.3 2:c8628734:::benchmark_data_smithi045_46415_object53965:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-08-19T14:18:58.007 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:15:22.002110+0000", 2024-08-19T14:18:58.007 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 215.92089273299999, 2024-08-19T14:18:58.007 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.24453285999999999, 2024-08-19T14:18:58.007 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:18:58.007 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:18:58.007 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:18:58.007 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:18:58.007 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:18:58.007 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:18:58.007 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 177246 2024-08-19T14:18:58.007 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.008 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:18:58.008 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.008 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:18:58.008 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:22.002110+0000", 2024-08-19T14:18:58.008 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:18:58.008 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.008 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.008 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:18:58.008 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:22.002110+0000", 2024-08-19T14:18:58.008 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:18:58.008 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.008 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.008 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:18:58.009 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:22.002110+0000", 2024-08-19T14:18:58.009 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8699999999999995e-07 2024-08-19T14:18:58.009 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.009 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.009 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:18:58.009 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:22.002113+0000", 2024-08-19T14:18:58.009 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1429999999999998e-06 2024-08-19T14:18:58.009 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.009 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.009 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:18:58.009 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:22.002125+0000", 2024-08-19T14:18:58.009 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1250000000000001e-05 2024-08-19T14:18:58.009 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.010 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.010 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:18:58.010 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:22.002130+0000", 2024-08-19T14:18:58.010 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.31e-06 2024-08-19T14:18:58.010 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.010 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.010 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:18:58.010 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:22.002180+0000", 2024-08-19T14:18:58.010 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0312000000000002e-05 2024-08-19T14:18:58.010 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.010 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.010 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:18:58.010 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:22.002298+0000", 2024-08-19T14:18:58.011 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000117545 2024-08-19T14:18:58.011 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.011 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.011 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,7", 2024-08-19T14:18:58.011 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:22.002416+0000", 2024-08-19T14:18:58.011 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000118029 2024-08-19T14:18:58.011 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.011 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.011 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:18:58.011 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:22.005936+0000", 2024-08-19T14:18:58.011 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0035206149999999999 2024-08-19T14:18:58.011 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.011 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.011 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:18:58.012 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:22.008247+0000", 2024-08-19T14:18:58.012 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0023103109999999998 2024-08-19T14:18:58.012 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.012 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.012 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:18:58.012 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:22.246574+0000", 2024-08-19T14:18:58.012 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.23832711200000001 2024-08-19T14:18:58.012 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.012 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.012 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:18:58.012 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:22.246611+0000", 2024-08-19T14:18:58.012 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6717000000000001e-05 2024-08-19T14:18:58.012 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.012 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.013 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:18:58.013 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:22.246643+0000", 2024-08-19T14:18:58.013 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1928999999999999e-05 2024-08-19T14:18:58.013 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:18:58.013 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:18:58.013 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:18:58.013 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.013 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.013 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:177255 2.3 2:ce66084d:::benchmark_data_smithi045_46415_object53974:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-08-19T14:18:58.013 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:15:22.015290+0000", 2024-08-19T14:18:58.013 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 215.90771254500001, 2024-08-19T14:18:58.013 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.23151259900000001, 2024-08-19T14:18:58.013 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:18:58.013 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:18:58.014 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:18:58.014 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:18:58.014 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:18:58.014 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:18:58.014 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 177255 2024-08-19T14:18:58.014 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.014 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:18:58.014 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.014 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:18:58.014 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:22.015290+0000", 2024-08-19T14:18:58.014 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:18:58.014 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.014 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.014 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:18:58.015 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:22.015290+0000", 2024-08-19T14:18:58.015 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:18:58.015 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.015 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.015 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:18:58.015 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:22.015292+0000", 2024-08-19T14:18:58.015 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.86e-06 2024-08-19T14:18:58.015 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.015 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.015 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:18:58.015 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:22.015302+0000", 2024-08-19T14:18:58.015 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0139999999999999e-05 2024-08-19T14:18:58.015 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.015 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.016 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:18:58.016 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:22.015326+0000", 2024-08-19T14:18:58.016 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4436000000000001e-05 2024-08-19T14:18:58.016 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.016 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.016 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:18:58.016 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:22.015339+0000", 2024-08-19T14:18:58.016 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2782999999999999e-05 2024-08-19T14:18:58.016 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.016 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.016 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:18:58.016 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:22.015483+0000", 2024-08-19T14:18:58.016 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000143848 2024-08-19T14:18:58.017 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.017 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.017 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:18:58.017 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:22.015658+0000", 2024-08-19T14:18:58.017 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00017525799999999999 2024-08-19T14:18:58.017 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.017 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.017 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,7", 2024-08-19T14:18:58.017 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:22.015733+0000", 2024-08-19T14:18:58.017 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4522000000000002e-05 2024-08-19T14:18:58.017 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.017 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.017 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:18:58.018 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:22.019282+0000", 2024-08-19T14:18:58.018 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0035489110000000001 2024-08-19T14:18:58.018 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.018 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.018 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:18:58.018 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:22.019455+0000", 2024-08-19T14:18:58.018 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000173115 2024-08-19T14:18:58.018 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.018 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.018 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:18:58.018 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:22.246698+0000", 2024-08-19T14:18:58.018 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.22724322599999999 2024-08-19T14:18:58.018 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.018 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.019 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:18:58.019 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:22.246757+0000", 2024-08-19T14:18:58.019 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8909000000000001e-05 2024-08-19T14:18:58.019 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.019 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.019 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:18:58.019 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:22.246802+0000", 2024-08-19T14:18:58.019 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5590999999999998e-05 2024-08-19T14:18:58.019 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:18:58.019 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:18:58.020 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:18:58.020 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.020 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.020 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:177265 2.3 2:cd07bc78:::benchmark_data_smithi045_46415_object53984:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-08-19T14:18:58.020 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:15:22.027303+0000", 2024-08-19T14:18:58.020 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 215.89569990999999, 2024-08-19T14:18:58.020 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.21960194099999999, 2024-08-19T14:18:58.020 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:18:58.020 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:18:58.020 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:18:58.020 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:18:58.020 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:18:58.020 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:18:58.021 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 177265 2024-08-19T14:18:58.021 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.021 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:18:58.021 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.021 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:18:58.021 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:22.027303+0000", 2024-08-19T14:18:58.021 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:18:58.021 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.021 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.021 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:18:58.021 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:22.027303+0000", 2024-08-19T14:18:58.021 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:18:58.022 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.022 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.022 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:18:58.022 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:22.027303+0000", 2024-08-19T14:18:58.022 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.2800000000000005e-07 2024-08-19T14:18:58.022 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.022 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.022 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:18:58.022 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:22.027308+0000", 2024-08-19T14:18:58.023 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.826e-06 2024-08-19T14:18:58.023 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.023 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.023 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:18:58.023 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:22.027319+0000", 2024-08-19T14:18:58.023 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.031e-05 2024-08-19T14:18:58.023 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.023 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.023 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:18:58.023 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:22.027324+0000", 2024-08-19T14:18:58.023 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0250000000000002e-06 2024-08-19T14:18:58.023 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.024 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.024 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:18:58.024 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:22.027374+0000", 2024-08-19T14:18:58.024 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0111000000000002e-05 2024-08-19T14:18:58.024 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.024 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.024 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:18:58.024 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:22.027484+0000", 2024-08-19T14:18:58.024 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000110605 2024-08-19T14:18:58.024 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.025 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.025 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,7", 2024-08-19T14:18:58.025 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:22.027601+0000", 2024-08-19T14:18:58.025 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000116884 2024-08-19T14:18:58.025 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.025 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.025 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:18:58.025 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:22.035236+0000", 2024-08-19T14:18:58.025 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.007634353 2024-08-19T14:18:58.025 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.025 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.026 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:18:58.026 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:22.035284+0000", 2024-08-19T14:18:58.026 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8084999999999998e-05 2024-08-19T14:18:58.026 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.026 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.026 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:18:58.026 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:22.246847+0000", 2024-08-19T14:18:58.026 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.21156344399999999 2024-08-19T14:18:58.026 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.026 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.027 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:18:58.027 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:22.246878+0000", 2024-08-19T14:18:58.027 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0636000000000003e-05 2024-08-19T14:18:58.027 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.027 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.027 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:18:58.027 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:22.246904+0000", 2024-08-19T14:18:58.027 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6733999999999998e-05 2024-08-19T14:18:58.028 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:18:58.028 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:18:58.028 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:18:58.028 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.028 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.028 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:223849 2.3 2:ce808b41:::benchmark_data_smithi045_46415_object100568:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e201)", 2024-08-19T14:18:58.028 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:16:18.946749+0000", 2024-08-19T14:18:58.028 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 158.97625304799999, 2024-08-19T14:18:58.028 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.23607239799999999, 2024-08-19T14:18:58.028 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:18:58.028 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:18:58.028 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:18:58.029 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:18:58.029 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:18:58.029 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:18:58.029 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 223849 2024-08-19T14:18:58.029 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.029 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:18:58.029 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.029 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:18:58.029 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:16:18.946749+0000", 2024-08-19T14:18:58.030 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:18:58.030 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.030 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.030 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:18:58.030 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:16:18.946749+0000", 2024-08-19T14:18:58.030 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:18:58.030 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.030 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.030 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:18:58.031 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:16:18.946750+0000", 2024-08-19T14:18:58.031 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.2600000000000001e-07 2024-08-19T14:18:58.031 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.031 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.031 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:18:58.031 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:16:18.946755+0000", 2024-08-19T14:18:58.031 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6210000000000003e-06 2024-08-19T14:18:58.031 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.031 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.032 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:18:58.032 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:16:18.946766+0000", 2024-08-19T14:18:58.032 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1173e-05 2024-08-19T14:18:58.032 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.032 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.032 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:18:58.032 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:16:18.946772+0000", 2024-08-19T14:18:58.032 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8900000000000004e-06 2024-08-19T14:18:58.032 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.032 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.032 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:18:58.032 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:16:18.946812+0000", 2024-08-19T14:18:58.033 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9619000000000001e-05 2024-08-19T14:18:58.033 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.033 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.033 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:18:58.033 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:16:18.946899+0000", 2024-08-19T14:18:58.033 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.7496000000000004e-05 2024-08-19T14:18:58.033 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.033 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.033 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,7", 2024-08-19T14:18:58.033 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:16:18.946970+0000", 2024-08-19T14:18:58.033 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1371999999999994e-05 2024-08-19T14:18:58.033 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.033 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.033 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:18:58.034 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:16:18.951666+0000", 2024-08-19T14:18:58.034 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.004695734 2024-08-19T14:18:58.034 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.034 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.034 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:18:58.034 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:16:18.952768+0000", 2024-08-19T14:18:58.034 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001101802 2024-08-19T14:18:58.034 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.034 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.034 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:18:58.034 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:16:19.182722+0000", 2024-08-19T14:18:58.034 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.22995431499999999 2024-08-19T14:18:58.035 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.035 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.035 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:18:58.035 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:16:19.182776+0000", 2024-08-19T14:18:58.035 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3752999999999998e-05 2024-08-19T14:18:58.035 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.035 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.035 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:18:58.035 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:16:19.182822+0000", 2024-08-19T14:18:58.035 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5697e-05 2024-08-19T14:18:58.035 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:18:58.035 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:18:58.035 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:18:58.035 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.036 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.036 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:223864 2.3 2:c10b07c7:::benchmark_data_smithi045_46415_object100583:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e201)", 2024-08-19T14:18:58.036 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:16:18.962904+0000", 2024-08-19T14:18:58.036 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 158.96009882199999, 2024-08-19T14:18:58.036 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.22006620700000001, 2024-08-19T14:18:58.036 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:18:58.036 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:18:58.036 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:18:58.036 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:18:58.036 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:18:58.036 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:18:58.036 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 223864 2024-08-19T14:18:58.036 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.037 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:18:58.037 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.037 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:18:58.037 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:16:18.962904+0000", 2024-08-19T14:18:58.037 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:18:58.037 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.037 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.037 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:18:58.037 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:16:18.962904+0000", 2024-08-19T14:18:58.037 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:18:58.037 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.037 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.037 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:18:58.037 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:16:18.962905+0000", 2024-08-19T14:18:58.038 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1170000000000001e-06 2024-08-19T14:18:58.038 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.038 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.038 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:18:58.038 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:16:18.962910+0000", 2024-08-19T14:18:58.038 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1880000000000001e-06 2024-08-19T14:18:58.038 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.038 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.038 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:18:58.038 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:16:18.962923+0000", 2024-08-19T14:18:58.038 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3380999999999999e-05 2024-08-19T14:18:58.038 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.038 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.038 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:18:58.039 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:16:18.962931+0000", 2024-08-19T14:18:58.039 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3579999999999997e-06 2024-08-19T14:18:58.039 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.039 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.039 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:18:58.039 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:16:18.962997+0000", 2024-08-19T14:18:58.039 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6303000000000005e-05 2024-08-19T14:18:58.039 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.039 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.039 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:18:58.039 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:16:18.963123+0000", 2024-08-19T14:18:58.039 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012634500000000001 2024-08-19T14:18:58.039 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.039 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.040 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,7", 2024-08-19T14:18:58.040 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:16:18.963248+0000", 2024-08-19T14:18:58.040 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000125165 2024-08-19T14:18:58.040 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.040 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.040 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:18:58.040 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:16:18.966968+0000", 2024-08-19T14:18:58.040 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0037195790000000002 2024-08-19T14:18:58.040 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.040 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.040 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:18:58.040 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:16:18.968245+0000", 2024-08-19T14:18:58.040 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0012766489999999999 2024-08-19T14:18:58.040 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.041 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.041 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:18:58.041 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:16:19.182869+0000", 2024-08-19T14:18:58.041 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.21462381699999999 2024-08-19T14:18:58.041 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.041 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.041 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:18:58.041 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:16:19.182926+0000", 2024-08-19T14:18:58.041 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7349999999999998e-05 2024-08-19T14:18:58.041 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.041 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.041 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:18:58.041 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:16:19.182970+0000", 2024-08-19T14:18:58.041 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3955000000000002e-05 2024-08-19T14:18:58.042 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:18:58.042 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:18:58.042 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:18:58.042 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.042 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.042 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25057.0:64611 3.1c 3.ea59d81c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e246)", 2024-08-19T14:18:58.042 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:17:35.978148+0000", 2024-08-19T14:18:58.042 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 81.944854117999995, 2024-08-19T14:18:58.042 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.98523712200000002, 2024-08-19T14:18:58.042 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:18:58.042 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:18:58.042 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:18:58.042 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:18:58.043 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25057", 2024-08-19T14:18:58.043 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:18:58.043 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 64611 2024-08-19T14:18:58.043 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.043 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:18:58.043 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.043 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:18:58.043 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.978148+0000", 2024-08-19T14:18:58.043 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:18:58.043 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.043 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.043 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:18:58.043 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.978148+0000", 2024-08-19T14:18:58.043 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:18:58.044 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.044 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.044 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:18:58.044 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.978150+0000", 2024-08-19T14:18:58.044 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.903e-06 2024-08-19T14:18:58.044 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.044 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.044 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:18:58.044 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.978235+0000", 2024-08-19T14:18:58.044 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.4945999999999996e-05 2024-08-19T14:18:58.044 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.044 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.044 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:18:58.044 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.978255+0000", 2024-08-19T14:18:58.045 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9361e-05 2024-08-19T14:18:58.045 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.045 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.045 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:18:58.045 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.978263+0000", 2024-08-19T14:18:58.045 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6770000000000001e-06 2024-08-19T14:18:58.045 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.045 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.045 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:18:58.045 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.978330+0000", 2024-08-19T14:18:58.045 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7089999999999996e-05 2024-08-19T14:18:58.045 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.045 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.046 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:18:58.046 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.978341+0000", 2024-08-19T14:18:58.046 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0559e-05 2024-08-19T14:18:58.046 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.046 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.046 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:18:58.046 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:36.963362+0000", 2024-08-19T14:18:58.046 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.98502080999999997 2024-08-19T14:18:58.046 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.046 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.046 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:18:58.046 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:36.963385+0000", 2024-08-19T14:18:58.046 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3776000000000001e-05 2024-08-19T14:18:58.047 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:18:58.047 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:18:58.047 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:18:58.047 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.047 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.047 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25057.0:64634 3.1c 3.60566bc (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e246)", 2024-08-19T14:18:58.047 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:17:36.183428+0000", 2024-08-19T14:18:58.047 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 81.739574570000002, 2024-08-19T14:18:58.047 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.78006474000000003, 2024-08-19T14:18:58.047 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:18:58.047 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:18:58.047 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:18:58.047 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:18:58.047 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25057", 2024-08-19T14:18:58.048 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:18:58.048 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 64634 2024-08-19T14:18:58.048 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.048 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:18:58.048 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.048 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:18:58.048 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:36.183428+0000", 2024-08-19T14:18:58.048 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:18:58.048 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.048 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.048 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:18:58.048 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:36.183428+0000", 2024-08-19T14:18:58.048 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:18:58.048 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.049 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.049 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:18:58.049 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:36.183430+0000", 2024-08-19T14:18:58.049 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8160000000000001e-06 2024-08-19T14:18:58.049 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.049 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.049 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:18:58.049 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:36.183463+0000", 2024-08-19T14:18:58.049 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3254999999999999e-05 2024-08-19T14:18:58.049 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.049 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.049 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:18:58.049 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:36.183482+0000", 2024-08-19T14:18:58.049 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8595000000000001e-05 2024-08-19T14:18:58.050 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.050 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.050 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:18:58.050 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:36.183491+0000", 2024-08-19T14:18:58.050 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.8840000000000006e-06 2024-08-19T14:18:58.050 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.050 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.050 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:18:58.050 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:36.183557+0000", 2024-08-19T14:18:58.050 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.5791999999999996e-05 2024-08-19T14:18:58.050 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.050 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.050 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:18:58.050 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:36.183566+0000", 2024-08-19T14:18:58.051 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.9919999999999996e-06 2024-08-19T14:18:58.051 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.051 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.051 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:18:58.051 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:36.963484+0000", 2024-08-19T14:18:58.051 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.77991771499999996 2024-08-19T14:18:58.051 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.051 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.051 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:18:58.051 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:36.963493+0000", 2024-08-19T14:18:58.051 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6910000000000003e-06 2024-08-19T14:18:58.051 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:18:58.051 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:18:58.051 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:18:58.051 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.052 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.052 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25057.0:68717 3.1d 3:b8ee1297:::benchmark_data_smithi045_59973_object68716:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e251)", 2024-08-19T14:18:58.052 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:17:43.596001+0000", 2024-08-19T14:18:58.052 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 74.327001448999994, 2024-08-19T14:18:58.052 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.22569884800000001, 2024-08-19T14:18:58.052 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:18:58.052 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:18:58.052 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:18:58.052 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:18:58.052 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25057", 2024-08-19T14:18:58.053 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:18:58.053 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 68717 2024-08-19T14:18:58.053 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.053 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:18:58.053 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.053 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:18:58.053 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:43.596001+0000", 2024-08-19T14:18:58.053 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:18:58.053 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.053 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.053 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:18:58.053 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:43.596001+0000", 2024-08-19T14:18:58.054 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:18:58.054 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.054 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.054 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:18:58.054 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:43.596003+0000", 2024-08-19T14:18:58.054 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0540000000000002e-06 2024-08-19T14:18:58.054 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.054 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.054 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:18:58.054 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:43.596035+0000", 2024-08-19T14:18:58.054 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2276000000000001e-05 2024-08-19T14:18:58.054 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.054 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.055 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:18:58.055 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:43.596919+0000", 2024-08-19T14:18:58.055 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00088367600000000001 2024-08-19T14:18:58.055 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.055 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.055 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:18:58.055 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:43.596940+0000", 2024-08-19T14:18:58.055 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1239000000000001e-05 2024-08-19T14:18:58.055 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.055 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.055 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:18:58.056 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:43.598926+0000", 2024-08-19T14:18:58.056 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0019856399999999999 2024-08-19T14:18:58.056 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.056 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.056 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:18:58.056 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:43.599089+0000", 2024-08-19T14:18:58.056 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00016281999999999999 2024-08-19T14:18:58.056 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.056 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.056 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4,8", 2024-08-19T14:18:58.056 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:43.599257+0000", 2024-08-19T14:18:58.056 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000168114 2024-08-19T14:18:58.056 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.056 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.057 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:18:58.057 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:43.604920+0000", 2024-08-19T14:18:58.057 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0056633059999999999 2024-08-19T14:18:58.057 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.057 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.057 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:18:58.057 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:43.607476+0000", 2024-08-19T14:18:58.057 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0025555230000000001 2024-08-19T14:18:58.057 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.057 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.058 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:18:58.058 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:43.820484+0000", 2024-08-19T14:18:58.058 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.21300864999999999 2024-08-19T14:18:58.058 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.058 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.058 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:18:58.058 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:43.820556+0000", 2024-08-19T14:18:58.058 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1452999999999995e-05 2024-08-19T14:18:58.058 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:18:58.058 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:18:58.058 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:18:58.059 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:43.821700+0000", 2024-08-19T14:18:58.059 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001144097 2024-08-19T14:18:58.059 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:18:58.059 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:18:58.059 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:18:58.059 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:18:58.059 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:18:58.059 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:18:58.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:58.100 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:58.100 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:18:58.200 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:18:58.232 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:58.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:58.233 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:18:58.334 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:18:58.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:58 smithi031 ceph-mon[16336]: from='client.25295 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:18:58.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:58.372 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:58.372 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:18:58.405 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:58 smithi032 ceph-mon[20004]: from='client.25295 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:18:58.473 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:18:58.505 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:58.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:58.506 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:18:58.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:58 smithi045 ceph-mon[20143]: from='client.25295 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:18:58.607 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:18:58.639 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:58.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:58.640 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:18:58.741 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:18:58.784 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":663,"stamp":"2024-08-19T14:18:55.835055+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5820918314,"num_objects":88815,"num_object_clones":0,"num_object_copies":266445,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88815,"num_whiteouts":0,"num_read":18,"num_read_kb":17,"num_write":56876,"num_write_kb":927549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":582366,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":245308,"ondisk_log_size":245308,"up":201,"acting":201,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":206,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":20075148,"kb_used_data":17427312,"kb_used_omap":14,"kb_used_meta":2647537,"kb_avail":823631220,"statfs":{"total":863955320832,"available":843398369280,"internally_reserved":0,"allocated":17845567488,"data_stored":17833180729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14353,"internal_metadata":2711078895},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,4,8,7,5],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-746861674,"num_objects":-11396,"num_object_clones":0,"num_object_copies":-34188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-11396,"num_whiteouts":0,"num_read":-3,"num_read_kb":-3,"num_write":11392,"num_write_kb":-67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1,"num_bytes_recovered":-48085,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001777"},"pg_stats":[{"pgid":"1.0","version":"70'139","reported_seq":787,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-08-19T14:18:50.071647+0000","last_change":"2024-08-19T14:18:25.872921+0000","last_active":"2024-08-19T14:18:50.071647+0000","last_peered":"2024-08-19T14:18:50.071647+0000","last_clean":"2024-08-19T14:18:50.071647+0000","last_became_active":"2024-08-19T14:18:13.807133+0000","last_became_peered":"2024-08-19T14:18:13.807133+0000","last_unstale":"2024-08-19T14:18:50.071647+0000","last_undegraded":"2024-08-19T14:18:50.071647+0000","last_fullsized":"2024-08-19T14:18:50.071647+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:18:25.872808+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:18:25.872808+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:33.201222+0000","scrub_duration":2157,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,9],"acting":[7,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"290'5215","reported_seq":5404,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-08-19T14:18:53.237762+0000","last_change":"2024-08-19T14:18:17.084657+0000","last_active":"2024-08-19T14:18:53.237762+0000","last_peered":"2024-08-19T14:18:53.237762+0000","last_clean":"2024-08-19T14:18:53.237762+0000","last_became_active":"2024-08-19T14:17:34.968554+0000","last_became_peered":"2024-08-19T14:17:34.968554+0000","last_unstale":"2024-08-19T14:18:53.237762+0000","last_undegraded":"2024-08-19T14:18:53.237762+0000","last_fullsized":"2024-08-19T14:18:53.237762+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'4295","last_scrub_stamp":"2024-08-19T14:18:17.084447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:17.084447+0000","objects_scrubbed":1731,"log_size":5215,"log_dups_size":0,"ondisk_log_size":5215,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:29.607335+0000","scrub_duration":357,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":89260032,"num_objects":1362,"num_object_clones":0,"num_object_copies":4086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":925,"num_write_kb":17792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,5],"acting":[9,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.3","version":"70'139","reported_seq":685,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-08-19T14:18:50.067192+0000","last_change":"2024-08-19T14:18:21.708466+0000","last_active":"2024-08-19T14:18:50.067192+0000","last_peered":"2024-08-19T14:18:50.067192+0000","last_clean":"2024-08-19T14:18:50.067192+0000","last_became_active":"2024-08-19T14:18:13.804721+0000","last_became_peered":"2024-08-19T14:18:13.804721+0000","last_unstale":"2024-08-19T14:18:50.067192+0000","last_undegraded":"2024-08-19T14:18:50.067192+0000","last_fullsized":"2024-08-19T14:18:50.067192+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:18:21.708333+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:18:21.708333+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:45.972923+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"290'4998","reported_seq":5178,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-08-19T14:18:50.826137+0000","last_change":"2024-08-19T14:18:20.248300+0000","last_active":"2024-08-19T14:18:50.826137+0000","last_peered":"2024-08-19T14:18:50.826137+0000","last_clean":"2024-08-19T14:18:50.826137+0000","last_became_active":"2024-08-19T14:17:34.963892+0000","last_became_peered":"2024-08-19T14:17:34.963892+0000","last_unstale":"2024-08-19T14:18:50.826137+0000","last_undegraded":"2024-08-19T14:18:50.826137+0000","last_fullsized":"2024-08-19T14:18:50.826137+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'4227","last_scrub_stamp":"2024-08-19T14:18:20.248075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:20.248075+0000","objects_scrubbed":1786,"log_size":4998,"log_dups_size":0,"ondisk_log_size":4998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:29.422356+0000","scrub_duration":285,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":89063424,"num_objects":1359,"num_object_clones":0,"num_object_copies":4077,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1359,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":773,"num_write_kb":11072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,11],"acting":[0,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"70'139","reported_seq":630,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-08-19T14:18:50.065772+0000","last_change":"2024-08-19T14:18:24.001903+0000","last_active":"2024-08-19T14:18:50.065772+0000","last_peered":"2024-08-19T14:18:50.065772+0000","last_clean":"2024-08-19T14:18:50.065772+0000","last_became_active":"2024-08-19T14:18:13.809344+0000","last_became_peered":"2024-08-19T14:18:13.809344+0000","last_unstale":"2024-08-19T14:18:50.065772+0000","last_undegraded":"2024-08-19T14:18:50.065772+0000","last_fullsized":"2024-08-19T14:18:50.065772+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:18:24.001801+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:18:24.001801+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:26.754447+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,8],"acting":[0,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"292'5185","reported_seq":5379,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-08-19T14:18:55.708923+0000","last_change":"2024-08-19T14:18:17.845313+0000","last_active":"2024-08-19T14:18:55.708923+0000","last_peered":"2024-08-19T14:18:55.708923+0000","last_clean":"2024-08-19T14:18:55.708923+0000","last_became_active":"2024-08-19T14:17:34.963390+0000","last_became_peered":"2024-08-19T14:17:34.963390+0000","last_unstale":"2024-08-19T14:18:55.708923+0000","last_undegraded":"2024-08-19T14:18:55.708923+0000","last_fullsized":"2024-08-19T14:18:55.708923+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'4224","last_scrub_stamp":"2024-08-19T14:18:17.845089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:17.845089+0000","objects_scrubbed":1755,"log_size":5185,"log_dups_size":0,"ondisk_log_size":5185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:24.860639+0000","scrub_duration":426,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87556096,"num_objects":1336,"num_object_clones":0,"num_object_copies":4008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1336,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":969,"num_write_kb":17600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,1],"acting":[11,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1","version":"70'139","reported_seq":682,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-08-19T14:18:50.066666+0000","last_change":"2024-08-19T14:18:21.795775+0000","last_active":"2024-08-19T14:18:50.066666+0000","last_peered":"2024-08-19T14:18:50.066666+0000","last_clean":"2024-08-19T14:18:50.066666+0000","last_became_active":"2024-08-19T14:18:13.808505+0000","last_became_peered":"2024-08-19T14:18:13.808505+0000","last_unstale":"2024-08-19T14:18:50.066666+0000","last_undegraded":"2024-08-19T14:18:50.066666+0000","last_fullsized":"2024-08-19T14:18:50.066666+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:18:21.795647+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:18:21.795647+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:42.370696+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,11],"acting":[9,0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3","version":"290'5140","reported_seq":5329,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-08-19T14:18:53.040270+0000","last_change":"2024-08-19T14:18:20.238483+0000","last_active":"2024-08-19T14:18:53.040270+0000","last_peered":"2024-08-19T14:18:53.040270+0000","last_clean":"2024-08-19T14:18:53.040270+0000","last_became_active":"2024-08-19T14:17:34.968430+0000","last_became_peered":"2024-08-19T14:17:34.968430+0000","last_unstale":"2024-08-19T14:18:53.040270+0000","last_undegraded":"2024-08-19T14:18:53.040270+0000","last_fullsized":"2024-08-19T14:18:53.040270+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'4292","last_scrub_stamp":"2024-08-19T14:18:20.238336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:20.238336+0000","objects_scrubbed":1842,"log_size":5140,"log_dups_size":0,"ondisk_log_size":5140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:38.217046+0000","scrub_duration":304,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90832896,"num_objects":1386,"num_object_clones":0,"num_object_copies":4158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1386,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":848,"num_write_kb":12544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,11],"acting":[4,0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"70'139","reported_seq":603,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-08-19T14:18:50.070317+0000","last_change":"2024-08-19T14:18:27.204083+0000","last_active":"2024-08-19T14:18:50.070317+0000","last_peered":"2024-08-19T14:18:50.070317+0000","last_clean":"2024-08-19T14:18:50.070317+0000","last_became_active":"2024-08-19T14:18:26.878753+0000","last_became_peered":"2024-08-19T14:18:26.878753+0000","last_unstale":"2024-08-19T14:18:50.070317+0000","last_undegraded":"2024-08-19T14:18:50.070317+0000","last_fullsized":"2024-08-19T14:18:50.070317+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:18:27.203987+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:18:27.203987+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:56.868334+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"290'6862","reported_seq":7065,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-08-19T14:18:51.633054+0000","last_change":"2024-08-19T14:18:16.976479+0000","last_active":"2024-08-19T14:18:51.633054+0000","last_peered":"2024-08-19T14:18:51.633054+0000","last_clean":"2024-08-19T14:18:51.633054+0000","last_became_active":"2024-08-19T14:17:19.745911+0000","last_became_peered":"2024-08-19T14:17:19.745911+0000","last_unstale":"2024-08-19T14:18:51.633054+0000","last_undegraded":"2024-08-19T14:18:51.633054+0000","last_fullsized":"2024-08-19T14:18:51.633054+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'5041","last_scrub_stamp":"2024-08-19T14:18:16.976188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:16.976188+0000","objects_scrubbed":3541,"log_size":6862,"log_dups_size":0,"ondisk_log_size":6862,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:20.500490+0000","scrub_duration":886,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189857792,"num_objects":2897,"num_object_clones":0,"num_object_copies":8691,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1850,"num_write_kb":38592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"70'139","reported_seq":686,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-08-19T14:18:50.066947+0000","last_change":"2024-08-19T14:18:27.848950+0000","last_active":"2024-08-19T14:18:50.066947+0000","last_peered":"2024-08-19T14:18:50.066947+0000","last_clean":"2024-08-19T14:18:50.066947+0000","last_became_active":"2024-08-19T14:18:26.874902+0000","last_became_peered":"2024-08-19T14:18:26.874902+0000","last_unstale":"2024-08-19T14:18:50.066947+0000","last_undegraded":"2024-08-19T14:18:50.066947+0000","last_fullsized":"2024-08-19T14:18:50.066947+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:18:27.848870+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:27.848870+0000","last_clean_scrub_stamp":"2024-08-19T14:18:27.848870+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:29.542244+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,4],"acting":[9,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.4","version":"290'6988","reported_seq":7191,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-08-19T14:18:53.403022+0000","last_change":"2024-08-19T14:18:20.377814+0000","last_active":"2024-08-19T14:18:53.403022+0000","last_peered":"2024-08-19T14:18:53.403022+0000","last_clean":"2024-08-19T14:18:53.403022+0000","last_became_active":"2024-08-19T14:17:19.741977+0000","last_became_peered":"2024-08-19T14:17:19.741977+0000","last_unstale":"2024-08-19T14:18:53.403022+0000","last_undegraded":"2024-08-19T14:18:53.403022+0000","last_fullsized":"2024-08-19T14:18:53.403022+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'5234","last_scrub_stamp":"2024-08-19T14:18:20.377712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:20.377712+0000","objects_scrubbed":3737,"log_size":6988,"log_dups_size":0,"ondisk_log_size":6988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:34.514418+0000","scrub_duration":701,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":181927936,"num_objects":2776,"num_object_clones":0,"num_object_copies":8328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1757,"num_write_kb":25472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,5],"acting":[1,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"70'139","reported_seq":504,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-08-19T14:18:55.091068+0000","last_change":"2024-08-19T14:18:28.501538+0000","last_active":"2024-08-19T14:18:55.091068+0000","last_peered":"2024-08-19T14:18:55.091068+0000","last_clean":"2024-08-19T14:18:55.091068+0000","last_became_active":"2024-08-19T14:18:26.881435+0000","last_became_peered":"2024-08-19T14:18:26.881435+0000","last_unstale":"2024-08-19T14:18:55.091068+0000","last_undegraded":"2024-08-19T14:18:55.091068+0000","last_fullsized":"2024-08-19T14:18:55.091068+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:18:28.501449+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:18:28.501449+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:55.948489+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,4],"acting":[11,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.7","version":"291'6951","reported_seq":7155,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-08-19T14:18:54.261966+0000","last_change":"2024-08-19T14:18:18.152792+0000","last_active":"2024-08-19T14:18:54.261966+0000","last_peered":"2024-08-19T14:18:54.261966+0000","last_clean":"2024-08-19T14:18:54.261966+0000","last_became_active":"2024-08-19T14:17:19.742604+0000","last_became_peered":"2024-08-19T14:17:19.742604+0000","last_unstale":"2024-08-19T14:18:54.261966+0000","last_undegraded":"2024-08-19T14:18:54.261966+0000","last_fullsized":"2024-08-19T14:18:54.261966+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'5087","last_scrub_stamp":"2024-08-19T14:18:18.152521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:18.152521+0000","objects_scrubbed":3609,"log_size":6951,"log_dups_size":0,"ondisk_log_size":6951,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:45.684422+0000","scrub_duration":838,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":179175424,"num_objects":2734,"num_object_clones":0,"num_object_copies":8202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2734,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1887,"num_write_kb":32384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"70'139","reported_seq":728,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-08-19T14:18:55.091939+0000","last_change":"2024-08-19T14:18:21.476428+0000","last_active":"2024-08-19T14:18:55.091939+0000","last_peered":"2024-08-19T14:18:55.091939+0000","last_clean":"2024-08-19T14:18:55.091939+0000","last_became_active":"2024-08-19T14:18:13.806013+0000","last_became_peered":"2024-08-19T14:18:13.806013+0000","last_unstale":"2024-08-19T14:18:55.091939+0000","last_undegraded":"2024-08-19T14:18:55.091939+0000","last_fullsized":"2024-08-19T14:18:55.091939+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:18:21.476313+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:18:21.476313+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:29.834715+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.6","version":"290'6919","reported_seq":7123,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-08-19T14:18:50.874379+0000","last_change":"2024-08-19T14:18:21.462540+0000","last_active":"2024-08-19T14:18:50.874379+0000","last_peered":"2024-08-19T14:18:50.874379+0000","last_clean":"2024-08-19T14:18:50.874379+0000","last_became_active":"2024-08-19T14:17:19.748180+0000","last_became_peered":"2024-08-19T14:17:19.748180+0000","last_unstale":"2024-08-19T14:18:50.874379+0000","last_undegraded":"2024-08-19T14:18:50.874379+0000","last_fullsized":"2024-08-19T14:18:50.874379+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'5366","last_scrub_stamp":"2024-08-19T14:18:21.462335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:21.462335+0000","objects_scrubbed":3843,"log_size":6919,"log_dups_size":0,"ondisk_log_size":6919,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:33.613418+0000","scrub_duration":461,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":194904064,"num_objects":2974,"num_object_clones":0,"num_object_copies":8922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1553,"num_write_kb":21888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.a","version":"70'139","reported_seq":694,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-08-19T14:18:50.066707+0000","last_change":"2024-08-19T14:18:28.865004+0000","last_active":"2024-08-19T14:18:50.066707+0000","last_peered":"2024-08-19T14:18:50.066707+0000","last_clean":"2024-08-19T14:18:50.066707+0000","last_became_active":"2024-08-19T14:18:26.875006+0000","last_became_peered":"2024-08-19T14:18:26.875006+0000","last_unstale":"2024-08-19T14:18:50.066707+0000","last_undegraded":"2024-08-19T14:18:50.066707+0000","last_fullsized":"2024-08-19T14:18:50.066707+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:18:28.864870+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:28.864870+0000","last_clean_scrub_stamp":"2024-08-19T14:18:28.864870+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:43.628160+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,0],"acting":[9,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.8","version":"292'7253","reported_seq":7459,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-08-19T14:18:55.668784+0000","last_change":"2024-08-19T14:18:17.263508+0000","last_active":"2024-08-19T14:18:55.668784+0000","last_peered":"2024-08-19T14:18:55.668784+0000","last_clean":"2024-08-19T14:18:55.668784+0000","last_became_active":"2024-08-19T14:17:19.753063+0000","last_became_peered":"2024-08-19T14:17:19.753063+0000","last_unstale":"2024-08-19T14:18:55.668784+0000","last_undegraded":"2024-08-19T14:18:55.668784+0000","last_fullsized":"2024-08-19T14:18:55.668784+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'5161","last_scrub_stamp":"2024-08-19T14:18:17.262486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:17.262486+0000","objects_scrubbed":3649,"log_size":7253,"log_dups_size":0,"ondisk_log_size":7253,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:42.192456+0000","scrub_duration":818,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":172425216,"num_objects":2631,"num_object_clones":0,"num_object_copies":7893,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2106,"num_write_kb":34816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,4],"acting":[11,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.9","version":"70'139","reported_seq":809,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-08-19T14:18:50.074379+0000","last_change":"2024-08-19T14:18:28.744550+0000","last_active":"2024-08-19T14:18:50.074379+0000","last_peered":"2024-08-19T14:18:50.074379+0000","last_clean":"2024-08-19T14:18:50.074379+0000","last_became_active":"2024-08-19T14:18:26.887657+0000","last_became_peered":"2024-08-19T14:18:26.887657+0000","last_unstale":"2024-08-19T14:18:50.074379+0000","last_undegraded":"2024-08-19T14:18:50.074379+0000","last_fullsized":"2024-08-19T14:18:50.074379+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:18:28.744320+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:18:28.744320+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:30.737843+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"291'7559","reported_seq":7762,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-08-19T14:18:54.251960+0000","last_change":"2024-08-19T14:18:17.226643+0000","last_active":"2024-08-19T14:18:54.251960+0000","last_peered":"2024-08-19T14:18:54.251960+0000","last_clean":"2024-08-19T14:18:54.251960+0000","last_became_active":"2024-08-19T14:17:34.972934+0000","last_became_peered":"2024-08-19T14:17:34.972934+0000","last_unstale":"2024-08-19T14:18:54.251960+0000","last_undegraded":"2024-08-19T14:18:54.251960+0000","last_fullsized":"2024-08-19T14:18:54.251960+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'5595","last_scrub_stamp":"2024-08-19T14:18:17.226507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:17.226507+0000","objects_scrubbed":3605,"log_size":7559,"log_dups_size":0,"ondisk_log_size":7559,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:43.427899+0000","scrub_duration":888,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":182583296,"num_objects":2786,"num_object_clones":0,"num_object_copies":8358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1997,"num_write_kb":37696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"70'139","reported_seq":698,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-08-19T14:18:50.069308+0000","last_change":"2024-08-19T14:18:28.190126+0000","last_active":"2024-08-19T14:18:50.069308+0000","last_peered":"2024-08-19T14:18:50.069308+0000","last_clean":"2024-08-19T14:18:50.069308+0000","last_became_active":"2024-08-19T14:18:26.887277+0000","last_became_peered":"2024-08-19T14:18:26.887277+0000","last_unstale":"2024-08-19T14:18:50.069308+0000","last_undegraded":"2024-08-19T14:18:50.069308+0000","last_fullsized":"2024-08-19T14:18:50.069308+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:18:28.190009+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:18:28.190009+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:57.986532+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,8],"acting":[5,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"290'7375","reported_seq":7573,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-08-19T14:18:53.070160+0000","last_change":"2024-08-19T14:18:21.329122+0000","last_active":"2024-08-19T14:18:53.070160+0000","last_peered":"2024-08-19T14:18:53.070160+0000","last_clean":"2024-08-19T14:18:53.070160+0000","last_became_active":"2024-08-19T14:17:34.962596+0000","last_became_peered":"2024-08-19T14:17:34.962596+0000","last_unstale":"2024-08-19T14:18:53.070160+0000","last_undegraded":"2024-08-19T14:18:53.070160+0000","last_fullsized":"2024-08-19T14:18:53.070160+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'5664","last_scrub_stamp":"2024-08-19T14:18:21.328957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:21.328957+0000","objects_scrubbed":3688,"log_size":7375,"log_dups_size":0,"ondisk_log_size":7375,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:22.510625+0000","scrub_duration":405,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":182910976,"num_objects":2791,"num_object_clones":0,"num_object_copies":8373,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2791,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1711,"num_write_kb":26048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,8],"acting":[4,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"291'6962","reported_seq":7165,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-08-19T14:18:54.830108+0000","last_change":"2024-08-19T14:18:18.202254+0000","last_active":"2024-08-19T14:18:54.830108+0000","last_peered":"2024-08-19T14:18:54.830108+0000","last_clean":"2024-08-19T14:18:54.830108+0000","last_became_active":"2024-08-19T14:17:19.743605+0000","last_became_peered":"2024-08-19T14:17:19.743605+0000","last_unstale":"2024-08-19T14:18:54.830108+0000","last_undegraded":"2024-08-19T14:18:54.830108+0000","last_fullsized":"2024-08-19T14:18:54.830108+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'5067","last_scrub_stamp":"2024-08-19T14:18:18.201966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:18.201966+0000","objects_scrubbed":3534,"log_size":6962,"log_dups_size":0,"ondisk_log_size":6962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:46.145488+0000","scrub_duration":779,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171311104,"num_objects":2614,"num_object_clones":0,"num_object_copies":7842,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1922,"num_write_kb":32064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.b","version":"70'139","reported_seq":704,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-08-19T14:18:55.090757+0000","last_change":"2024-08-19T14:18:27.506148+0000","last_active":"2024-08-19T14:18:55.090757+0000","last_peered":"2024-08-19T14:18:55.090757+0000","last_clean":"2024-08-19T14:18:55.090757+0000","last_became_active":"2024-08-19T14:18:26.875179+0000","last_became_peered":"2024-08-19T14:18:26.875179+0000","last_unstale":"2024-08-19T14:18:55.090757+0000","last_undegraded":"2024-08-19T14:18:55.090757+0000","last_fullsized":"2024-08-19T14:18:55.090757+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:18:27.506045+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:18:27.506045+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:43.546458+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.c","version":"290'7274","reported_seq":7476,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-08-19T14:18:51.611556+0000","last_change":"2024-08-19T14:18:18.066128+0000","last_active":"2024-08-19T14:18:51.611556+0000","last_peered":"2024-08-19T14:18:51.611556+0000","last_clean":"2024-08-19T14:18:51.611556+0000","last_became_active":"2024-08-19T14:17:34.973857+0000","last_became_peered":"2024-08-19T14:17:34.973857+0000","last_unstale":"2024-08-19T14:18:51.611556+0000","last_undegraded":"2024-08-19T14:18:51.611556+0000","last_fullsized":"2024-08-19T14:18:51.611556+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'5481","last_scrub_stamp":"2024-08-19T14:18:18.065864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:18.065864+0000","objects_scrubbed":3568,"log_size":7274,"log_dups_size":0,"ondisk_log_size":7274,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:25.462037+0000","scrub_duration":959,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188678144,"num_objects":2879,"num_object_clones":0,"num_object_copies":8637,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2879,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1821,"num_write_kb":36224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,9],"acting":[5,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"70'139","reported_seq":354,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-08-19T14:18:50.064033+0000","last_change":"2024-08-19T14:18:27.900166+0000","last_active":"2024-08-19T14:18:50.064033+0000","last_peered":"2024-08-19T14:18:50.064033+0000","last_clean":"2024-08-19T14:18:50.064033+0000","last_became_active":"2024-08-19T14:18:26.884105+0000","last_became_peered":"2024-08-19T14:18:26.884105+0000","last_unstale":"2024-08-19T14:18:50.064033+0000","last_undegraded":"2024-08-19T14:18:50.064033+0000","last_fullsized":"2024-08-19T14:18:50.064033+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:18:27.899965+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:18:27.899965+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:56.474954+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"290'7570","reported_seq":7774,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-08-19T14:18:53.265864+0000","last_change":"2024-08-19T14:18:26.752263+0000","last_active":"2024-08-19T14:18:53.265864+0000","last_peered":"2024-08-19T14:18:53.265864+0000","last_clean":"2024-08-19T14:18:53.265864+0000","last_became_active":"2024-08-19T14:17:34.969333+0000","last_became_peered":"2024-08-19T14:17:34.969333+0000","last_unstale":"2024-08-19T14:18:53.265864+0000","last_undegraded":"2024-08-19T14:18:53.265864+0000","last_fullsized":"2024-08-19T14:18:53.265864+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'5846","last_scrub_stamp":"2024-08-19T14:18:26.752109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:26.752109+0000","objects_scrubbed":3860,"log_size":7570,"log_dups_size":0,"ondisk_log_size":7570,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:56.568372+0000","scrub_duration":5035,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":183500800,"num_objects":2800,"num_object_clones":0,"num_object_copies":8400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2800,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1738,"num_write_kb":21696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,9],"acting":[7,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"70'139","reported_seq":598,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-08-19T14:18:50.068529+0000","last_change":"2024-08-19T14:18:23.173343+0000","last_active":"2024-08-19T14:18:50.068529+0000","last_peered":"2024-08-19T14:18:50.068529+0000","last_clean":"2024-08-19T14:18:50.068529+0000","last_became_active":"2024-08-19T14:18:18.711708+0000","last_became_peered":"2024-08-19T14:18:18.711708+0000","last_unstale":"2024-08-19T14:18:50.068529+0000","last_undegraded":"2024-08-19T14:18:50.068529+0000","last_fullsized":"2024-08-19T14:18:50.068529+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:18:23.173268+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:18:23.173268+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:51.974300+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"292'7559","reported_seq":7764,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-08-19T14:18:55.681213+0000","last_change":"2024-08-19T14:18:19.171319+0000","last_active":"2024-08-19T14:18:55.681213+0000","last_peered":"2024-08-19T14:18:55.681213+0000","last_clean":"2024-08-19T14:18:55.681213+0000","last_became_active":"2024-08-19T14:17:34.971738+0000","last_became_peered":"2024-08-19T14:17:34.971738+0000","last_unstale":"2024-08-19T14:18:55.681213+0000","last_undegraded":"2024-08-19T14:18:55.681213+0000","last_fullsized":"2024-08-19T14:18:55.681213+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'5670","last_scrub_stamp":"2024-08-19T14:18:19.171123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:19.171123+0000","objects_scrubbed":3656,"log_size":7559,"log_dups_size":0,"ondisk_log_size":7559,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:37.122914+0000","scrub_duration":758,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":175702016,"num_objects":2681,"num_object_clones":0,"num_object_copies":8043,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2681,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1915,"num_write_kb":30080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,0],"acting":[11,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.c","version":"70'139","reported_seq":461,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-08-19T14:18:54.096380+0000","last_change":"2024-08-19T14:18:28.348190+0000","last_active":"2024-08-19T14:18:54.096380+0000","last_peered":"2024-08-19T14:18:54.096380+0000","last_clean":"2024-08-19T14:18:54.096380+0000","last_became_active":"2024-08-19T14:18:26.872942+0000","last_became_peered":"2024-08-19T14:18:26.872942+0000","last_unstale":"2024-08-19T14:18:54.096380+0000","last_undegraded":"2024-08-19T14:18:54.096380+0000","last_fullsized":"2024-08-19T14:18:54.096380+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:18:28.348087+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:18:28.348087+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:39.796688+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,9],"acting":[3,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.f","version":"290'7436","reported_seq":7638,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-08-19T14:18:53.270800+0000","last_change":"2024-08-19T14:18:23.090893+0000","last_active":"2024-08-19T14:18:53.270800+0000","last_peered":"2024-08-19T14:18:53.270800+0000","last_clean":"2024-08-19T14:18:53.270800+0000","last_became_active":"2024-08-19T14:17:34.954276+0000","last_became_peered":"2024-08-19T14:17:34.954276+0000","last_unstale":"2024-08-19T14:18:53.270800+0000","last_undegraded":"2024-08-19T14:18:53.270800+0000","last_fullsized":"2024-08-19T14:18:53.270800+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'5708","last_scrub_stamp":"2024-08-19T14:18:23.090770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:23.090770+0000","objects_scrubbed":3694,"log_size":7436,"log_dups_size":0,"ondisk_log_size":7436,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:33.066761+0000","scrub_duration":369,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180879360,"num_objects":2760,"num_object_clones":0,"num_object_copies":8280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2760,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1728,"num_write_kb":25408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"70'139","reported_seq":374,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-08-19T14:18:50.072095+0000","last_change":"2024-08-19T14:18:27.752642+0000","last_active":"2024-08-19T14:18:50.072095+0000","last_peered":"2024-08-19T14:18:50.072095+0000","last_clean":"2024-08-19T14:18:50.072095+0000","last_became_active":"2024-08-19T14:18:26.882002+0000","last_became_peered":"2024-08-19T14:18:26.882002+0000","last_unstale":"2024-08-19T14:18:50.072095+0000","last_undegraded":"2024-08-19T14:18:50.072095+0000","last_fullsized":"2024-08-19T14:18:50.072095+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:18:27.751964+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:18:27.751964+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:48.211625+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"290'6867","reported_seq":11733,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-08-19T14:18:53.299679+0000","last_change":"2024-08-19T14:18:20.355375+0000","last_active":"2024-08-19T14:18:53.299679+0000","last_peered":"2024-08-19T14:18:53.299679+0000","last_clean":"2024-08-19T14:18:53.299679+0000","last_became_active":"2024-08-19T14:17:38.684742+0000","last_became_peered":"2024-08-19T14:17:38.684742+0000","last_unstale":"2024-08-19T14:18:53.299679+0000","last_undegraded":"2024-08-19T14:18:53.299679+0000","last_fullsized":"2024-08-19T14:18:53.299679+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":5,"last_scrub":"273'5152","last_scrub_stamp":"2024-08-19T14:18:20.355288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:20.355288+0000","objects_scrubbed":3752,"log_size":6867,"log_dups_size":0,"ondisk_log_size":6867,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:42.543316+0000","scrub_duration":598,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":183304192,"num_objects":2797,"num_object_clones":0,"num_object_copies":8391,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2797,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1719,"num_write_kb":24448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,3],"acting":[9,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.12","version":"70'139","reported_seq":633,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-08-19T14:18:50.068260+0000","last_change":"2024-08-19T14:18:20.109582+0000","last_active":"2024-08-19T14:18:50.068260+0000","last_peered":"2024-08-19T14:18:50.068260+0000","last_clean":"2024-08-19T14:18:50.068260+0000","last_became_active":"2024-08-19T14:18:18.703594+0000","last_became_peered":"2024-08-19T14:18:18.703594+0000","last_unstale":"2024-08-19T14:18:50.068260+0000","last_undegraded":"2024-08-19T14:18:50.068260+0000","last_fullsized":"2024-08-19T14:18:50.068260+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:18:20.109467+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:18:20.109467+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:21.664555+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"290'6781","reported_seq":10086,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-08-19T14:18:53.034380+0000","last_change":"2024-08-19T14:18:18.845192+0000","last_active":"2024-08-19T14:18:53.034380+0000","last_peered":"2024-08-19T14:18:53.034380+0000","last_clean":"2024-08-19T14:18:53.034380+0000","last_became_active":"2024-08-19T14:17:38.695549+0000","last_became_peered":"2024-08-19T14:17:38.695549+0000","last_unstale":"2024-08-19T14:18:53.034380+0000","last_undegraded":"2024-08-19T14:18:53.034380+0000","last_fullsized":"2024-08-19T14:18:53.034380+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":5,"last_scrub":"273'5072","last_scrub_stamp":"2024-08-19T14:18:18.844791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:18.844791+0000","objects_scrubbed":3594,"log_size":6781,"log_dups_size":0,"ondisk_log_size":6781,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:35.762099+0000","scrub_duration":924,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":178782208,"num_objects":2728,"num_object_clones":0,"num_object_copies":8184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2728,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1728,"num_write_kb":27584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.13","version":"70'139","reported_seq":693,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-08-19T14:18:50.067011+0000","last_change":"2024-08-19T14:18:22.659716+0000","last_active":"2024-08-19T14:18:50.067011+0000","last_peered":"2024-08-19T14:18:50.067011+0000","last_clean":"2024-08-19T14:18:50.067011+0000","last_became_active":"2024-08-19T14:18:18.698334+0000","last_became_peered":"2024-08-19T14:18:18.698334+0000","last_unstale":"2024-08-19T14:18:50.067011+0000","last_undegraded":"2024-08-19T14:18:50.067011+0000","last_fullsized":"2024-08-19T14:18:50.067011+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:18:22.659638+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:18:22.659638+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:50.367468+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.12","version":"290'6749","reported_seq":13314,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-08-19T14:18:50.848694+0000","last_change":"2024-08-19T14:18:23.349819+0000","last_active":"2024-08-19T14:18:50.848694+0000","last_peered":"2024-08-19T14:18:50.848694+0000","last_clean":"2024-08-19T14:18:50.848694+0000","last_became_active":"2024-08-19T14:17:47.703601+0000","last_became_peered":"2024-08-19T14:17:47.703601+0000","last_unstale":"2024-08-19T14:18:50.848694+0000","last_undegraded":"2024-08-19T14:18:50.848694+0000","last_fullsized":"2024-08-19T14:18:50.848694+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":5,"last_scrub":"273'5164","last_scrub_stamp":"2024-08-19T14:18:23.349706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:23.349706+0000","objects_scrubbed":3676,"log_size":6749,"log_dups_size":0,"ondisk_log_size":6749,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:44.157657+0000","scrub_duration":326,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188022784,"num_objects":2869,"num_object_clones":0,"num_object_copies":8607,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2869,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1585,"num_write_kb":24896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"70'139","reported_seq":788,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-08-19T14:18:50.068403+0000","last_change":"2024-08-19T14:18:22.139657+0000","last_active":"2024-08-19T14:18:50.068403+0000","last_peered":"2024-08-19T14:18:50.068403+0000","last_clean":"2024-08-19T14:18:50.068403+0000","last_became_active":"2024-08-19T14:18:18.704562+0000","last_became_peered":"2024-08-19T14:18:18.704562+0000","last_unstale":"2024-08-19T14:18:50.068403+0000","last_undegraded":"2024-08-19T14:18:50.068403+0000","last_fullsized":"2024-08-19T14:18:50.068403+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:18:22.139587+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:18:22.139587+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:43.235144+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,8],"acting":[5,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"290'6861","reported_seq":11682,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-08-19T14:18:53.250295+0000","last_change":"2024-08-19T14:18:18.597206+0000","last_active":"2024-08-19T14:18:53.250295+0000","last_peered":"2024-08-19T14:18:53.250295+0000","last_clean":"2024-08-19T14:18:53.250295+0000","last_became_active":"2024-08-19T14:17:45.731114+0000","last_became_peered":"2024-08-19T14:17:45.731114+0000","last_unstale":"2024-08-19T14:18:53.250295+0000","last_undegraded":"2024-08-19T14:18:53.250295+0000","last_fullsized":"2024-08-19T14:18:53.250295+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":5,"last_scrub":"272'5093","last_scrub_stamp":"2024-08-19T14:18:18.596762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:18.596762+0000","objects_scrubbed":3558,"log_size":6861,"log_dups_size":0,"ondisk_log_size":6861,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:23.244078+0000","scrub_duration":834,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":181665792,"num_objects":2772,"num_object_clones":0,"num_object_copies":8316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1788,"num_write_kb":32064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,9],"acting":[7,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.11","version":"70'139","reported_seq":337,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-08-19T14:18:50.064851+0000","last_change":"2024-08-19T14:18:25.021206+0000","last_active":"2024-08-19T14:18:50.064851+0000","last_peered":"2024-08-19T14:18:50.064851+0000","last_clean":"2024-08-19T14:18:50.064851+0000","last_became_active":"2024-08-19T14:18:18.697894+0000","last_became_peered":"2024-08-19T14:18:18.697894+0000","last_unstale":"2024-08-19T14:18:50.064851+0000","last_undegraded":"2024-08-19T14:18:50.064851+0000","last_fullsized":"2024-08-19T14:18:50.064851+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:18:25.021126+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:18:25.021126+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:32.345902+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.14","version":"290'6988","reported_seq":11931,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-08-19T14:18:53.047610+0000","last_change":"2024-08-19T14:18:19.862358+0000","last_active":"2024-08-19T14:18:53.047610+0000","last_peered":"2024-08-19T14:18:53.047610+0000","last_clean":"2024-08-19T14:18:53.047610+0000","last_became_active":"2024-08-19T14:17:44.691159+0000","last_became_peered":"2024-08-19T14:17:44.691159+0000","last_unstale":"2024-08-19T14:18:53.047610+0000","last_undegraded":"2024-08-19T14:18:53.047610+0000","last_fullsized":"2024-08-19T14:18:53.047610+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":5,"last_scrub":"273'5248","last_scrub_stamp":"2024-08-19T14:18:19.862173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:19.862173+0000","objects_scrubbed":3703,"log_size":6988,"log_dups_size":0,"ondisk_log_size":6988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:44.248749+0000","scrub_duration":928,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":186712064,"num_objects":2849,"num_object_clones":0,"num_object_copies":8547,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1780,"num_write_kb":29632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,11],"acting":[4,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.16","version":"70'139","reported_seq":305,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-08-19T14:18:50.067340+0000","last_change":"2024-08-19T14:18:31.900475+0000","last_active":"2024-08-19T14:18:50.067340+0000","last_peered":"2024-08-19T14:18:50.067340+0000","last_clean":"2024-08-19T14:18:50.067340+0000","last_became_active":"2024-08-19T14:18:31.899921+0000","last_became_peered":"2024-08-19T14:18:31.899921+0000","last_unstale":"2024-08-19T14:18:50.067340+0000","last_undegraded":"2024-08-19T14:18:50.067340+0000","last_fullsized":"2024-08-19T14:18:50.067340+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:18:22.807154+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:18:22.807154+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:54.733273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13418,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":48084,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,11],"acting":[1,0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"70'139","reported_seq":444,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-08-19T14:18:54.816182+0000","last_change":"2024-08-19T14:18:54.816222+0000","last_active":"2024-08-19T14:18:54.816182+0000","last_peered":"2024-08-19T14:18:54.816182+0000","last_clean":"2024-08-19T14:18:54.816182+0000","last_became_active":"2024-08-19T14:18:31.901556+0000","last_became_peered":"2024-08-19T14:18:31.901556+0000","last_unstale":"2024-08-19T14:18:54.816182+0000","last_undegraded":"2024-08-19T14:18:54.816182+0000","last_fullsized":"2024-08-19T14:18:54.816182+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:18:54.816110+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:18:54.816110+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:20:09.076175+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,0],"acting":[8,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.15","version":"291'6926","reported_seq":11773,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-08-19T14:18:54.838275+0000","last_change":"2024-08-19T14:18:20.779335+0000","last_active":"2024-08-19T14:18:54.838275+0000","last_peered":"2024-08-19T14:18:54.838275+0000","last_clean":"2024-08-19T14:18:54.838275+0000","last_became_active":"2024-08-19T14:17:42.696659+0000","last_became_peered":"2024-08-19T14:17:42.696659+0000","last_unstale":"2024-08-19T14:18:54.838275+0000","last_undegraded":"2024-08-19T14:18:54.838275+0000","last_fullsized":"2024-08-19T14:18:54.838275+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":5,"last_scrub":"273'5168","last_scrub_stamp":"2024-08-19T14:18:20.779227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:20.779227+0000","objects_scrubbed":3680,"log_size":6926,"log_dups_size":0,"ondisk_log_size":6926,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:30.576171+0000","scrub_duration":384,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":175112192,"num_objects":2672,"num_object_clones":0,"num_object_copies":8016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1758,"num_write_kb":24000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.16","version":"290'6957","reported_seq":11761,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-08-19T14:18:51.607459+0000","last_change":"2024-08-19T14:18:18.940376+0000","last_active":"2024-08-19T14:18:51.607459+0000","last_peered":"2024-08-19T14:18:51.607459+0000","last_clean":"2024-08-19T14:18:51.607459+0000","last_became_active":"2024-08-19T14:17:47.681397+0000","last_became_peered":"2024-08-19T14:17:47.681397+0000","last_unstale":"2024-08-19T14:18:51.607459+0000","last_undegraded":"2024-08-19T14:18:51.607459+0000","last_fullsized":"2024-08-19T14:18:51.607459+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":5,"last_scrub":"273'5233","last_scrub_stamp":"2024-08-19T14:18:18.940216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:18.940216+0000","objects_scrubbed":3684,"log_size":6957,"log_dups_size":0,"ondisk_log_size":6957,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:37.548896+0000","scrub_duration":846,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":193527808,"num_objects":2953,"num_object_clones":0,"num_object_copies":8859,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2953,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1743,"num_write_kb":32384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.14","version":"70'139","reported_seq":451,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-08-19T14:18:50.071887+0000","last_change":"2024-08-19T14:18:18.745307+0000","last_active":"2024-08-19T14:18:50.071887+0000","last_peered":"2024-08-19T14:18:50.071887+0000","last_clean":"2024-08-19T14:18:50.071887+0000","last_became_active":"2024-08-19T14:18:18.703926+0000","last_became_peered":"2024-08-19T14:18:18.703926+0000","last_unstale":"2024-08-19T14:18:50.071887+0000","last_undegraded":"2024-08-19T14:18:50.071887+0000","last_fullsized":"2024-08-19T14:18:50.071887+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:18:18.745191+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:18:18.745191+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:24.644627+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,1],"acting":[7,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"290'6652","reported_seq":12976,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-08-19T14:18:50.881248+0000","last_change":"2024-08-19T14:18:16.720459+0000","last_active":"2024-08-19T14:18:50.881248+0000","last_peered":"2024-08-19T14:18:50.881248+0000","last_clean":"2024-08-19T14:18:50.881248+0000","last_became_active":"2024-08-19T14:17:48.682321+0000","last_became_peered":"2024-08-19T14:17:48.682321+0000","last_unstale":"2024-08-19T14:18:50.881248+0000","last_undegraded":"2024-08-19T14:18:50.881248+0000","last_fullsized":"2024-08-19T14:18:50.881248+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":5,"last_scrub":"270'4919","last_scrub_stamp":"2024-08-19T14:18:16.720122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:16.720122+0000","objects_scrubbed":3405,"log_size":6652,"log_dups_size":0,"ondisk_log_size":6652,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:37.591387+0000","scrub_duration":754,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188284928,"num_objects":2873,"num_object_clones":0,"num_object_copies":8619,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1756,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,9],"acting":[0,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.15","version":"70'139","reported_seq":330,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-08-19T14:18:50.072031+0000","last_change":"2024-08-19T14:18:31.912927+0000","last_active":"2024-08-19T14:18:50.072031+0000","last_peered":"2024-08-19T14:18:50.072031+0000","last_clean":"2024-08-19T14:18:50.072031+0000","last_became_active":"2024-08-19T14:18:31.912496+0000","last_became_peered":"2024-08-19T14:18:31.912496+0000","last_unstale":"2024-08-19T14:18:50.072031+0000","last_undegraded":"2024-08-19T14:18:50.072031+0000","last_fullsized":"2024-08-19T14:18:50.072031+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:18:20.455298+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:18:20.455298+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:54.861776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13418,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":48085,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"70'139","reported_seq":308,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-08-19T14:18:50.064086+0000","last_change":"2024-08-19T14:18:31.901742+0000","last_active":"2024-08-19T14:18:50.064086+0000","last_peered":"2024-08-19T14:18:50.064086+0000","last_clean":"2024-08-19T14:18:50.064086+0000","last_became_active":"2024-08-19T14:18:31.901229+0000","last_became_peered":"2024-08-19T14:18:31.901229+0000","last_unstale":"2024-08-19T14:18:50.064086+0000","last_undegraded":"2024-08-19T14:18:50.064086+0000","last_fullsized":"2024-08-19T14:18:50.064086+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:18:23.815845+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:18:23.815845+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:53.851172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":5,"num_write":8,"num_write_kb":134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":96169,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.18","version":"291'7045","reported_seq":11975,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-08-19T14:18:54.252340+0000","last_change":"2024-08-19T14:18:21.736611+0000","last_active":"2024-08-19T14:18:54.252340+0000","last_peered":"2024-08-19T14:18:54.252340+0000","last_clean":"2024-08-19T14:18:54.252340+0000","last_became_active":"2024-08-19T14:17:45.729134+0000","last_became_peered":"2024-08-19T14:17:45.729134+0000","last_unstale":"2024-08-19T14:18:54.252340+0000","last_undegraded":"2024-08-19T14:18:54.252340+0000","last_fullsized":"2024-08-19T14:18:54.252340+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":5,"last_scrub":"273'5311","last_scrub_stamp":"2024-08-19T14:18:21.736491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:21.736491+0000","objects_scrubbed":3712,"log_size":7045,"log_dups_size":0,"ondisk_log_size":7045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:34.973824+0000","scrub_duration":424,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":175636480,"num_objects":2680,"num_object_clones":0,"num_object_copies":8040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2680,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1734,"num_write_kb":22464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,1],"acting":[3,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1b","version":"70'139","reported_seq":670,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-08-19T14:18:54.089704+0000","last_change":"2024-08-19T14:18:53.860503+0000","last_active":"2024-08-19T14:18:54.089704+0000","last_peered":"2024-08-19T14:18:54.089704+0000","last_clean":"2024-08-19T14:18:54.089704+0000","last_became_active":"2024-08-19T14:18:31.901152+0000","last_became_peered":"2024-08-19T14:18:31.901152+0000","last_unstale":"2024-08-19T14:18:54.089704+0000","last_undegraded":"2024-08-19T14:18:54.089704+0000","last_fullsized":"2024-08-19T14:18:54.089704+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:18:53.860431+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:18:53.860431+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:20:08.015501+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.19","version":"290'6953","reported_seq":13440,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-08-19T14:18:53.383353+0000","last_change":"2024-08-19T14:18:21.161635+0000","last_active":"2024-08-19T14:18:53.383353+0000","last_peered":"2024-08-19T14:18:53.383353+0000","last_clean":"2024-08-19T14:18:53.383353+0000","last_became_active":"2024-08-19T14:17:46.796533+0000","last_became_peered":"2024-08-19T14:17:46.796533+0000","last_unstale":"2024-08-19T14:18:53.383353+0000","last_undegraded":"2024-08-19T14:18:53.383353+0000","last_fullsized":"2024-08-19T14:18:53.383353+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":5,"last_scrub":"273'5249","last_scrub_stamp":"2024-08-19T14:18:21.161536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:21.161536+0000","objects_scrubbed":3748,"log_size":6953,"log_dups_size":0,"ondisk_log_size":6953,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:31.937329+0000","scrub_duration":491,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184156160,"num_objects":2810,"num_object_clones":0,"num_object_copies":8430,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2810,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1704,"num_write_kb":24512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.18","version":"70'139","reported_seq":704,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-08-19T14:18:50.068187+0000","last_change":"2024-08-19T14:18:31.903169+0000","last_active":"2024-08-19T14:18:50.068187+0000","last_peered":"2024-08-19T14:18:50.068187+0000","last_clean":"2024-08-19T14:18:50.068187+0000","last_became_active":"2024-08-19T14:18:31.902735+0000","last_became_peered":"2024-08-19T14:18:31.902735+0000","last_unstale":"2024-08-19T14:18:50.068187+0000","last_undegraded":"2024-08-19T14:18:50.068187+0000","last_fullsized":"2024-08-19T14:18:50.068187+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:18:25.178947+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:18:25.178947+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:53.141159+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131344,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":301872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"291'7510","reported_seq":11739,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-08-19T14:18:54.825388+0000","last_change":"2024-08-19T14:18:20.149227+0000","last_active":"2024-08-19T14:18:54.825388+0000","last_peered":"2024-08-19T14:18:54.825388+0000","last_clean":"2024-08-19T14:18:54.825388+0000","last_became_active":"2024-08-19T14:17:46.731170+0000","last_became_peered":"2024-08-19T14:17:46.731170+0000","last_unstale":"2024-08-19T14:18:54.825388+0000","last_undegraded":"2024-08-19T14:18:54.825388+0000","last_fullsized":"2024-08-19T14:18:54.825388+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":5,"last_scrub":"273'5710","last_scrub_stamp":"2024-08-19T14:18:20.148991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:20.148991+0000","objects_scrubbed":3721,"log_size":7510,"log_dups_size":0,"ondisk_log_size":7510,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:38.250493+0000","scrub_duration":772,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":175374336,"num_objects":2676,"num_object_clones":0,"num_object_copies":8028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2676,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1821,"num_write_kb":24832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,9],"acting":[8,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.19","version":"70'139","reported_seq":652,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-08-19T14:18:50.064745+0000","last_change":"2024-08-19T14:18:31.902933+0000","last_active":"2024-08-19T14:18:50.064745+0000","last_peered":"2024-08-19T14:18:50.064745+0000","last_clean":"2024-08-19T14:18:50.064745+0000","last_became_active":"2024-08-19T14:18:31.902547+0000","last_became_peered":"2024-08-19T14:18:31.902547+0000","last_unstale":"2024-08-19T14:18:50.064745+0000","last_undegraded":"2024-08-19T14:18:50.064745+0000","last_fullsized":"2024-08-19T14:18:50.064745+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:16:03.918046+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:53.154918+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1b","version":"290'7367","reported_seq":13553,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-08-19T14:18:52.978346+0000","last_change":"2024-08-19T14:18:17.849009+0000","last_active":"2024-08-19T14:18:52.978346+0000","last_peered":"2024-08-19T14:18:52.978346+0000","last_clean":"2024-08-19T14:18:52.978346+0000","last_became_active":"2024-08-19T14:18:06.761591+0000","last_became_peered":"2024-08-19T14:18:06.761591+0000","last_unstale":"2024-08-19T14:18:52.978346+0000","last_undegraded":"2024-08-19T14:18:52.978346+0000","last_fullsized":"2024-08-19T14:18:52.978346+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":5,"last_scrub":"272'5596","last_scrub_stamp":"2024-08-19T14:18:17.848788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:17.848788+0000","objects_scrubbed":3567,"log_size":7367,"log_dups_size":0,"ondisk_log_size":7367,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:35.007855+0000","scrub_duration":920,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":182910976,"num_objects":2791,"num_object_clones":0,"num_object_copies":8373,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2791,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1788,"num_write_kb":32384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,1],"acting":[4,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1e","version":"70'139","reported_seq":322,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-08-19T14:18:55.091099+0000","last_change":"2024-08-19T14:18:53.600024+0000","last_active":"2024-08-19T14:18:55.091099+0000","last_peered":"2024-08-19T14:18:55.091099+0000","last_clean":"2024-08-19T14:18:55.091099+0000","last_became_active":"2024-08-19T14:18:31.902546+0000","last_became_peered":"2024-08-19T14:18:31.902546+0000","last_unstale":"2024-08-19T14:18:55.091099+0000","last_undegraded":"2024-08-19T14:18:55.091099+0000","last_fullsized":"2024-08-19T14:18:55.091099+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:18:53.599913+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:18:53.599913+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:20:18.091419+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"290'7195","reported_seq":13186,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-08-19T14:18:51.581149+0000","last_change":"2024-08-19T14:18:20.165996+0000","last_active":"2024-08-19T14:18:51.581149+0000","last_peered":"2024-08-19T14:18:51.581149+0000","last_clean":"2024-08-19T14:18:51.581149+0000","last_became_active":"2024-08-19T14:18:05.757137+0000","last_became_peered":"2024-08-19T14:18:05.757137+0000","last_unstale":"2024-08-19T14:18:51.581149+0000","last_undegraded":"2024-08-19T14:18:51.581149+0000","last_fullsized":"2024-08-19T14:18:51.581149+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":5,"last_scrub":"273'5655","last_scrub_stamp":"2024-08-19T14:18:20.165862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:20.165862+0000","objects_scrubbed":3615,"log_size":7195,"log_dups_size":0,"ondisk_log_size":7195,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:38.986213+0000","scrub_duration":1057,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184352768,"num_objects":2813,"num_object_clones":0,"num_object_copies":8439,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1564,"num_write_kb":24384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"291'7597","reported_seq":13844,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-08-19T14:18:54.838075+0000","last_change":"2024-08-19T14:18:21.800463+0000","last_active":"2024-08-19T14:18:54.838075+0000","last_peered":"2024-08-19T14:18:54.838075+0000","last_clean":"2024-08-19T14:18:54.838075+0000","last_became_active":"2024-08-19T14:17:59.735912+0000","last_became_peered":"2024-08-19T14:17:59.735912+0000","last_unstale":"2024-08-19T14:18:54.838075+0000","last_undegraded":"2024-08-19T14:18:54.838075+0000","last_fullsized":"2024-08-19T14:18:54.838075+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":5,"last_scrub":"273'5747","last_scrub_stamp":"2024-08-19T14:18:21.800353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:21.800353+0000","objects_scrubbed":3688,"log_size":7597,"log_dups_size":0,"ondisk_log_size":7597,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:39.685212+0000","scrub_duration":368,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":174522392,"num_objects":2664,"num_object_clones":0,"num_object_copies":7992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2664,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1850,"num_write_kb":26369,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,5],"acting":[8,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1c","version":"70'139","reported_seq":420,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-08-19T14:18:54.089279+0000","last_change":"2024-08-19T14:18:31.903380+0000","last_active":"2024-08-19T14:18:54.089279+0000","last_peered":"2024-08-19T14:18:54.089279+0000","last_clean":"2024-08-19T14:18:54.089279+0000","last_became_active":"2024-08-19T14:18:31.902494+0000","last_became_peered":"2024-08-19T14:18:31.902494+0000","last_unstale":"2024-08-19T14:18:54.089279+0000","last_undegraded":"2024-08-19T14:18:54.089279+0000","last_fullsized":"2024-08-19T14:18:54.089279+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:18:27.323451+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:18:27.323451+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8945,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":3,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":40071,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,11],"acting":[8,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1e","version":"291'7348","reported_seq":15672,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-08-19T14:18:54.227715+0000","last_change":"2024-08-19T14:18:19.041894+0000","last_active":"2024-08-19T14:18:54.227715+0000","last_peered":"2024-08-19T14:18:54.227715+0000","last_clean":"2024-08-19T14:18:54.227715+0000","last_became_active":"2024-08-19T14:18:11.801671+0000","last_became_peered":"2024-08-19T14:18:11.801671+0000","last_unstale":"2024-08-19T14:18:54.227715+0000","last_undegraded":"2024-08-19T14:18:54.227715+0000","last_fullsized":"2024-08-19T14:18:54.227715+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":5,"last_scrub":"273'5544","last_scrub_stamp":"2024-08-19T14:18:19.041763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:19.041763+0000","objects_scrubbed":3560,"log_size":7348,"log_dups_size":0,"ondisk_log_size":7348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:29.244531+0000","scrub_duration":775,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":174325760,"num_objects":2660,"num_object_clones":0,"num_object_copies":7980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2660,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1822,"num_write_kb":29504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,8],"acting":[3,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1d","version":"70'139","reported_seq":340,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-08-19T14:18:50.067129+0000","last_change":"2024-08-19T14:18:31.901595+0000","last_active":"2024-08-19T14:18:50.067129+0000","last_peered":"2024-08-19T14:18:50.067129+0000","last_clean":"2024-08-19T14:18:50.067129+0000","last_became_active":"2024-08-19T14:18:31.901183+0000","last_became_peered":"2024-08-19T14:18:31.901183+0000","last_unstale":"2024-08-19T14:18:50.067129+0000","last_undegraded":"2024-08-19T14:18:50.067129+0000","last_fullsized":"2024-08-19T14:18:50.067129+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:18:19.739352+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:18:19.739352+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:18:54.733309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13418,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":5,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":48085,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,5],"acting":[1,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"290'7220","reported_seq":15440,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-08-19T14:18:50.933766+0000","last_change":"2024-08-19T14:18:19.910823+0000","last_active":"2024-08-19T14:18:50.933766+0000","last_peered":"2024-08-19T14:18:50.933766+0000","last_clean":"2024-08-19T14:18:50.933766+0000","last_became_active":"2024-08-19T14:18:12.800420+0000","last_became_peered":"2024-08-19T14:18:12.800420+0000","last_unstale":"2024-08-19T14:18:50.933766+0000","last_undegraded":"2024-08-19T14:18:50.933766+0000","last_fullsized":"2024-08-19T14:18:50.933766+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":5,"last_scrub":"273'5615","last_scrub_stamp":"2024-08-19T14:18:19.910511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:19.910511+0000","objects_scrubbed":3633,"log_size":7220,"log_dups_size":0,"ondisk_log_size":7220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:30.864591+0000","scrub_duration":922,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":185532416,"num_objects":2831,"num_object_clones":0,"num_object_copies":8493,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1638,"num_write_kb":26752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.20","version":"290'5054","reported_seq":7217,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-08-19T14:18:53.220947+0000","last_change":"2024-08-19T14:18:18.133721+0000","last_active":"2024-08-19T14:18:53.220947+0000","last_peered":"2024-08-19T14:18:53.220947+0000","last_clean":"2024-08-19T14:18:53.220947+0000","last_became_active":"2024-08-19T14:17:43.686357+0000","last_became_peered":"2024-08-19T14:17:43.686357+0000","last_unstale":"2024-08-19T14:18:53.220947+0000","last_undegraded":"2024-08-19T14:18:53.220947+0000","last_fullsized":"2024-08-19T14:18:53.220947+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":6,"last_scrub":"272'4197","last_scrub_stamp":"2024-08-19T14:18:18.133535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:18.133535+0000","objects_scrubbed":1747,"log_size":5054,"log_dups_size":0,"ondisk_log_size":5054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:28.095319+0000","scrub_duration":374,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90963968,"num_objects":1388,"num_object_clones":0,"num_object_copies":4164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1388,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":861,"num_write_kb":16064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,11],"acting":[9,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.21","version":"291'5171","reported_seq":9277,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-08-19T14:18:54.758252+0000","last_change":"2024-08-19T14:18:16.780918+0000","last_active":"2024-08-19T14:18:54.758252+0000","last_peered":"2024-08-19T14:18:54.758252+0000","last_clean":"2024-08-19T14:18:54.758252+0000","last_became_active":"2024-08-19T14:17:49.680746+0000","last_became_peered":"2024-08-19T14:17:49.680746+0000","last_unstale":"2024-08-19T14:18:54.758252+0000","last_undegraded":"2024-08-19T14:18:54.758252+0000","last_fullsized":"2024-08-19T14:18:54.758252+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":6,"last_scrub":"270'4217","last_scrub_stamp":"2024-08-19T14:18:16.780726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:16.780726+0000","objects_scrubbed":1743,"log_size":5171,"log_dups_size":0,"ondisk_log_size":5171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:45.170940+0000","scrub_duration":329,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87883776,"num_objects":1341,"num_object_clones":0,"num_object_copies":4023,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1341,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":958,"num_write_kb":17792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,9],"acting":[8,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.22","version":"291'5355","reported_seq":8555,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-08-19T14:18:54.248973+0000","last_change":"2024-08-19T14:18:19.684323+0000","last_active":"2024-08-19T14:18:54.248973+0000","last_peered":"2024-08-19T14:18:54.248973+0000","last_clean":"2024-08-19T14:18:54.248973+0000","last_became_active":"2024-08-19T14:17:49.687020+0000","last_became_peered":"2024-08-19T14:17:49.687020+0000","last_unstale":"2024-08-19T14:18:54.248973+0000","last_undegraded":"2024-08-19T14:18:54.248973+0000","last_fullsized":"2024-08-19T14:18:54.248973+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":6,"last_scrub":"273'4443","last_scrub_stamp":"2024-08-19T14:18:19.684050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:19.684050+0000","objects_scrubbed":1928,"log_size":5355,"log_dups_size":0,"ondisk_log_size":5355,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:39.823259+0000","scrub_duration":369,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":93716480,"num_objects":1430,"num_object_clones":0,"num_object_copies":4290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1430,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":918,"num_write_kb":13440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,9],"acting":[3,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"290'5157","reported_seq":7417,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-08-19T14:18:50.860601+0000","last_change":"2024-08-19T14:18:22.194528+0000","last_active":"2024-08-19T14:18:50.860601+0000","last_peered":"2024-08-19T14:18:50.860601+0000","last_clean":"2024-08-19T14:18:50.860601+0000","last_became_active":"2024-08-19T14:17:46.797296+0000","last_became_peered":"2024-08-19T14:17:46.797296+0000","last_unstale":"2024-08-19T14:18:50.860601+0000","last_undegraded":"2024-08-19T14:18:50.860601+0000","last_fullsized":"2024-08-19T14:18:50.860601+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":6,"last_scrub":"273'4320","last_scrub_stamp":"2024-08-19T14:18:22.194414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:22.194414+0000","objects_scrubbed":1877,"log_size":5157,"log_dups_size":0,"ondisk_log_size":5157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:24.622604+0000","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":95027200,"num_objects":1450,"num_object_clones":0,"num_object_copies":4350,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":837,"num_write_kb":13120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":36,"stat_sum":{"num_bytes":5820383256,"num_objects":88813,"num_object_clones":0,"num_object_copies":266439,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88813,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":56852,"num_write_kb":927169,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17624936448,"data_stored":17624924232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":240999,"ondisk_log_size":240999,"up":108,"acting":108,"num_store_stats":9},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":535058,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":16,"num_write":24,"num_write_kb":380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":582366,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1802240,"data_stored":1788032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187547,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1979152,"kb_used_data":1691136,"kb_used_omap":1,"kb_used_meta":287998,"kb_avail":91766000,"statfs":{"total":95995035648,"available":93968384000,"internally_reserved":0,"allocated":1731723264,"data_stored":1730342501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":294910404},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":343597383776,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":74,"seq":317827580004,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2286384,"kb_used_data":1952552,"kb_used_omap":1,"kb_used_meta":333822,"kb_avail":91458768,"statfs":{"total":95995035648,"available":93653778432,"internally_reserved":0,"allocated":1999413248,"data_stored":1998040879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":341834177},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874346,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1899012,"kb_used_data":1629048,"kb_used_omap":1,"kb_used_meta":269950,"kb_avail":91846140,"statfs":{"total":95995035648,"available":94050447360,"internally_reserved":0,"allocated":1668145152,"data_stored":1666764539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":276429253},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":57,"seq":244813135980,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2031352,"kb_used_data":1707184,"kb_used_omap":1,"kb_used_meta":324094,"kb_avail":91713800,"statfs":{"total":95995035648,"available":93914931200,"internally_reserved":0,"allocated":1748156416,"data_stored":1746780495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":331872714},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":206158430320,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691956,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2241212,"kb_used_data":2043444,"kb_used_omap":1,"kb_used_meta":197758,"kb_avail":91503940,"statfs":{"total":95995035648,"available":93700034560,"internally_reserved":0,"allocated":2092486656,"data_stored":2091106663,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":202504645},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,3,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986297,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3179764,"kb_used_data":2841708,"kb_used_omap":1,"kb_used_meta":337982,"kb_avail":90565388,"statfs":{"total":95995035648,"available":92738957312,"internally_reserved":0,"allocated":2909908992,"data_stored":2908532551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":346094021},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,6,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149823,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2105136,"kb_used_data":1785128,"kb_used_omap":1,"kb_used_meta":319934,"kb_avail":91640016,"statfs":{"total":95995035648,"available":93839376384,"internally_reserved":0,"allocated":1827971072,"data_stored":1826598703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":327612871},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411459,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574984,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2381432,"kb_used_data":2075376,"kb_used_omap":1,"kb_used_meta":306046,"kb_avail":91363720,"statfs":{"total":95995035648,"available":93556449280,"internally_reserved":0,"allocated":2125185024,"data_stored":2123809103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":313391557},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738508,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1971704,"kb_used_data":1701736,"kb_used_omap":1,"kb_used_meta":269950,"kb_avail":91773448,"statfs":{"total":95995035648,"available":93976010752,"internally_reserved":0,"allocated":1742577664,"data_stored":1741205295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":276429253},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1711865856,"data_stored":1711865856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2092892160,"data_stored":2092892160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1789591552,"data_stored":1789591552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2882605056,"data_stored":2882600984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2069172224,"data_stored":2069168152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1728053248,"data_stored":1728053248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1662783488,"data_stored":1662779416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1964179456,"data_stored":1964179456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1723793408,"data_stored":1723793408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T14:18:58.788 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-08-19T14:18:58.789 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd pool set unique_pool_1 pg_num 46 2024-08-19T14:18:58.795 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:58.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:58.797 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:18:58.840 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:18:58.898 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:18:58.939 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-08-19T14:18:58.948 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:58.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:58.949 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:18:59.050 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:18:59.080 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:18:59.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:59.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:18:59.182 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:18:59.223 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:59.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:59.226 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:18:59.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:18:59 smithi032 ceph-mon[20004]: pgmap v664: 67 pgs: 67 active+clean; 5.3 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 1.64k op/s 2024-08-19T14:18:59.280 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:18:59 smithi031 ceph-mon[16336]: pgmap v664: 67 pgs: 67 active+clean; 5.3 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 1.64k op/s 2024-08-19T14:18:59.327 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:18:59.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:59.358 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:59.358 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:18:59.362 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:18:59.459 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:18:59.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:59.501 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:18:59.501 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:18:59.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:18:59 smithi045 ceph-mon[20143]: pgmap v664: 67 pgs: 67 active+clean; 5.3 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 1.64k op/s 2024-08-19T14:18:59.585 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:18:59.601 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:18:59.622 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:59.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:59.624 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:18:59.725 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:18:59.754 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:59.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:59.755 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:18:59.856 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:18:59.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:18:59.886 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:18:59.886 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:18:59.987 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:19:00.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:00.021 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:00.021 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:19:00.068 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:19:00.068 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-08-19T14:19:00.068 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-08-19T14:19:00.069 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:19:00.121 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:19:00.127 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:19:00.150 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is set 2024-08-19T14:19:00.154 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:00.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:00.155 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:19:00.256 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:19:00.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:00.279 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:00.280 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:19:00.381 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:19:00.422 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:00.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:00.424 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:19:00.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:00 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:19:00.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:00 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:19:00.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:00 smithi032 ceph-mon[20004]: osdmap e293: 12 total, 12 up, 9 in 2024-08-19T14:19:00.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:00 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1801409192' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:19:00.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:00 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:19:00.524 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:19:00.546 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:00.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:00 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:19:00.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:00 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:19:00.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:00 smithi045 ceph-mon[20143]: osdmap e293: 12 total, 12 up, 9 in 2024-08-19T14:19:00.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:00 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1801409192' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:19:00.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:00 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:19:00.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:00.547 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:19:00.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:00 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:19:00.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:00 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:19:00.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:00 smithi031 ceph-mon[16336]: osdmap e293: 12 total, 12 up, 9 in 2024-08-19T14:19:00.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:00 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1801409192' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:19:00.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:00 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:19:00.648 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:19:00.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:00.681 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:00.681 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:19:00.782 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:19:00.800 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-08-19T14:19:00.806 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:00.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:00.809 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:19:00.910 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:19:00.936 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:00.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:00.937 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:19:01.038 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:19:01.078 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:01.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:01.079 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:19:01.148 INFO:teuthology.orchestra.run.smithi031.stderr:set pool 3 pg_num to 46 2024-08-19T14:19:01.180 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:19:01.205 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:19:01.211 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:01.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:01.212 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:19:01.313 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:19:01.344 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:01.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:01.345 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:19:01.446 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:19:01.479 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:01.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:01.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:19:01.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:01 smithi032 ceph-mon[20004]: pgmap v666: 67 pgs: 67 active+clean; 5.0 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 1.72k op/s 2024-08-19T14:19:01.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:01 smithi032 ceph-mon[20004]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:19:01.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:01 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:19:01.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:01 smithi032 ceph-mon[20004]: osdmap e294: 12 total, 12 up, 9 in 2024-08-19T14:19:01.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:01 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/979372460' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "46"}]: dispatch 2024-08-19T14:19:01.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:01 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "46"}]: dispatch 2024-08-19T14:19:01.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:01 smithi045 ceph-mon[20143]: pgmap v666: 67 pgs: 67 active+clean; 5.0 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 1.72k op/s 2024-08-19T14:19:01.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:01 smithi045 ceph-mon[20143]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:19:01.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:01 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:19:01.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:01 smithi045 ceph-mon[20143]: osdmap e294: 12 total, 12 up, 9 in 2024-08-19T14:19:01.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:01 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/979372460' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "46"}]: dispatch 2024-08-19T14:19:01.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:01 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "46"}]: dispatch 2024-08-19T14:19:01.581 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:19:01.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:01.613 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:01.613 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:19:01.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:01 smithi031 ceph-mon[16336]: pgmap v666: 67 pgs: 67 active+clean; 5.0 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 1.72k op/s 2024-08-19T14:19:01.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:01 smithi031 ceph-mon[16336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:19:01.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:01 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:19:01.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:01 smithi031 ceph-mon[16336]: osdmap e294: 12 total, 12 up, 9 in 2024-08-19T14:19:01.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:01 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/979372460' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "46"}]: dispatch 2024-08-19T14:19:01.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:01 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "46"}]: dispatch 2024-08-19T14:19:01.713 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:19:01.746 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:01.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:01.747 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:19:01.824 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:19:01.824 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-08-19T14:19:01.824 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-08-19T14:19:01.825 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-08-19T14:19:01.825 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:19:01.848 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:19:01.879 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:01.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:01.880 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:19:01.981 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:19:02.017 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:02.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:02.018 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:19:02.120 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:19:02.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:02.150 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:02.150 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:19:02.251 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:19:02.299 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:02.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:02.301 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:19:02.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:02 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "46"}]': finished 2024-08-19T14:19:02.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:02 smithi031 ceph-mon[16336]: osdmap e295: 12 total, 12 up, 9 in 2024-08-19T14:19:02.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:02 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "46"}]: dispatch 2024-08-19T14:19:02.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:02 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "46"}]: dispatch 2024-08-19T14:19:02.402 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:19:02.423 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:02.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:02.424 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:19:02.456 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:02 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "46"}]': finished 2024-08-19T14:19:02.456 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:02 smithi032 ceph-mon[20004]: osdmap e295: 12 total, 12 up, 9 in 2024-08-19T14:19:02.456 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:02 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "46"}]: dispatch 2024-08-19T14:19:02.456 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:02 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "46"}]: dispatch 2024-08-19T14:19:02.482 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-08-19T14:19:02.525 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:19:02.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:02 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "46"}]': finished 2024-08-19T14:19:02.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:02 smithi045 ceph-mon[20143]: osdmap e295: 12 total, 12 up, 9 in 2024-08-19T14:19:02.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:02 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "46"}]: dispatch 2024-08-19T14:19:02.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:02 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "46"}]: dispatch 2024-08-19T14:19:02.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:02.547 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:02.548 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:19:02.648 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:19:02.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:02.671 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:02.671 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:19:02.737 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:19:02.772 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:19:02.793 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:02.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:02.794 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:19:02.803 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:19:02.895 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:19:02.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:02.917 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:02.917 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:19:03.018 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:19:03.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:03.042 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:03.042 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:19:03.143 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:19:03.170 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:03.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:03.172 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:19:03.245 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:19:03.245 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-08-19T14:19:03.245 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-08-19T14:19:03.245 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-08-19T14:19:03.246 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.246 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:8241 2.7 2:edf47d3e:::benchmark_data_smithi045_46415_object8240:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-08-19T14:19:03.246 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:12:51.684165+0000", 2024-08-19T14:19:03.246 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 371.55810106500002, 2024-08-19T14:19:03.246 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.204747761, 2024-08-19T14:19:03.246 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:19:03.246 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:19:03.246 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:19:03.246 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:19:03.246 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:19:03.246 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:19:03.246 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 8241 2024-08-19T14:19:03.246 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.247 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:19:03.247 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.247 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:19:03.247 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.684165+0000", 2024-08-19T14:19:03.247 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:19:03.247 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.247 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.247 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:19:03.247 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.684165+0000", 2024-08-19T14:19:03.247 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:19:03.247 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.247 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.247 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:19:03.247 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.684167+0000", 2024-08-19T14:19:03.248 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.313e-06 2024-08-19T14:19:03.248 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.248 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.248 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:19:03.248 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.684210+0000", 2024-08-19T14:19:03.248 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3049000000000001e-05 2024-08-19T14:19:03.248 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.248 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.248 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:19:03.248 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.684223+0000", 2024-08-19T14:19:03.248 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3019e-05 2024-08-19T14:19:03.248 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.248 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.248 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:19:03.249 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.684230+0000", 2024-08-19T14:19:03.249 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1969999999999999e-06 2024-08-19T14:19:03.249 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.249 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.249 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:19:03.249 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.684270+0000", 2024-08-19T14:19:03.249 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0649999999999999e-05 2024-08-19T14:19:03.249 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.249 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.249 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:19:03.249 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.684340+0000", 2024-08-19T14:19:03.249 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9979000000000002e-05 2024-08-19T14:19:03.249 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.250 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.250 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,10", 2024-08-19T14:19:03.250 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.684464+0000", 2024-08-19T14:19:03.250 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012321599999999999 2024-08-19T14:19:03.250 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.250 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.250 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:19:03.250 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.684888+0000", 2024-08-19T14:19:03.250 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00042467399999999999 2024-08-19T14:19:03.250 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.250 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.250 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:19:03.250 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.685794+0000", 2024-08-19T14:19:03.250 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00090609299999999996 2024-08-19T14:19:03.251 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.251 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.251 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:19:03.251 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.888736+0000", 2024-08-19T14:19:03.251 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20294189100000001 2024-08-19T14:19:03.251 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.251 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.251 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:19:03.251 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.888831+0000", 2024-08-19T14:19:03.252 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.5172000000000003e-05 2024-08-19T14:19:03.252 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.252 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.252 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:19:03.252 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.888913+0000", 2024-08-19T14:19:03.252 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1508000000000006e-05 2024-08-19T14:19:03.252 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:19:03.252 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:19:03.252 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:19:03.252 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.252 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.253 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:8259 2.7 2:e2e1720b:::benchmark_data_smithi045_46415_object8258:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-08-19T14:19:03.253 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:12:51.689217+0000", 2024-08-19T14:19:03.253 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 371.55304919600002, 2024-08-19T14:19:03.253 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.19987796899999999, 2024-08-19T14:19:03.253 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:19:03.253 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:19:03.253 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:19:03.253 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:19:03.253 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:19:03.253 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:19:03.253 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 8259 2024-08-19T14:19:03.253 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.253 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:19:03.254 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.254 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:19:03.254 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.689217+0000", 2024-08-19T14:19:03.254 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:19:03.254 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.254 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.254 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:19:03.254 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.689217+0000", 2024-08-19T14:19:03.254 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:19:03.254 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.254 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.254 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:19:03.254 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.689218+0000", 2024-08-19T14:19:03.254 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0240000000000001e-06 2024-08-19T14:19:03.255 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.255 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.255 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:19:03.255 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.689272+0000", 2024-08-19T14:19:03.255 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3974999999999997e-05 2024-08-19T14:19:03.255 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.255 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.255 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:19:03.255 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.689283+0000", 2024-08-19T14:19:03.255 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0647e-05 2024-08-19T14:19:03.255 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.256 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.256 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:19:03.256 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.689288+0000", 2024-08-19T14:19:03.256 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2390000000000002e-06 2024-08-19T14:19:03.256 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.256 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.256 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:19:03.256 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.689341+0000", 2024-08-19T14:19:03.256 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.342e-05 2024-08-19T14:19:03.256 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.256 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.256 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:19:03.256 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.689464+0000", 2024-08-19T14:19:03.256 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000122991 2024-08-19T14:19:03.257 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.257 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.257 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,10", 2024-08-19T14:19:03.257 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.689610+0000", 2024-08-19T14:19:03.257 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014557200000000001 2024-08-19T14:19:03.257 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.257 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.257 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:19:03.257 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.690297+0000", 2024-08-19T14:19:03.257 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00068681199999999997 2024-08-19T14:19:03.257 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.257 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.257 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:19:03.258 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.690649+0000", 2024-08-19T14:19:03.258 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00035261899999999998 2024-08-19T14:19:03.258 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.258 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.258 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:19:03.258 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.888991+0000", 2024-08-19T14:19:03.258 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.19834179800000001 2024-08-19T14:19:03.258 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.258 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.258 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:19:03.258 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.889045+0000", 2024-08-19T14:19:03.259 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3643000000000001e-05 2024-08-19T14:19:03.259 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.259 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.259 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:19:03.259 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.889095+0000", 2024-08-19T14:19:03.259 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0229000000000003e-05 2024-08-19T14:19:03.259 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:19:03.259 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:19:03.259 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:19:03.259 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.259 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.259 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:8272 2.7 2:ec6eef80:::benchmark_data_smithi045_46415_object8271:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-08-19T14:19:03.259 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:12:51.692591+0000", 2024-08-19T14:19:03.260 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 371.54967537300001, 2024-08-19T14:19:03.260 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.19659718500000001, 2024-08-19T14:19:03.260 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:19:03.260 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:19:03.260 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:19:03.261 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:19:03.261 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:19:03.261 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:19:03.261 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 8272 2024-08-19T14:19:03.261 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.261 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:19:03.261 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.261 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:19:03.261 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.692591+0000", 2024-08-19T14:19:03.262 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:19:03.262 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.262 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.262 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:19:03.262 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.692591+0000", 2024-08-19T14:19:03.262 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:19:03.262 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.262 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.263 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:19:03.263 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.692592+0000", 2024-08-19T14:19:03.263 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.2800000000000005e-07 2024-08-19T14:19:03.263 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.263 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.263 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:19:03.263 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.692661+0000", 2024-08-19T14:19:03.263 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8744000000000005e-05 2024-08-19T14:19:03.263 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.263 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.263 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:19:03.264 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.692675+0000", 2024-08-19T14:19:03.264 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3916e-05 2024-08-19T14:19:03.264 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.264 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.264 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:19:03.264 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.692682+0000", 2024-08-19T14:19:03.264 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5320000000000001e-06 2024-08-19T14:19:03.264 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.264 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.264 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:19:03.264 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.692731+0000", 2024-08-19T14:19:03.265 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8909999999999998e-05 2024-08-19T14:19:03.265 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.265 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.265 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:19:03.265 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.692805+0000", 2024-08-19T14:19:03.265 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4128000000000006e-05 2024-08-19T14:19:03.265 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.265 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.265 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,10", 2024-08-19T14:19:03.265 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.692885+0000", 2024-08-19T14:19:03.265 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.9655999999999998e-05 2024-08-19T14:19:03.265 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.266 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.266 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:19:03.266 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.693416+0000", 2024-08-19T14:19:03.266 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00053101099999999996 2024-08-19T14:19:03.266 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.266 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.266 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:19:03.266 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.694542+0000", 2024-08-19T14:19:03.266 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0011263810000000001 2024-08-19T14:19:03.266 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.266 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.266 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:19:03.266 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.889136+0000", 2024-08-19T14:19:03.267 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.194593922 2024-08-19T14:19:03.267 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.267 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.267 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:19:03.267 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.889164+0000", 2024-08-19T14:19:03.267 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8113999999999999e-05 2024-08-19T14:19:03.267 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.267 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.267 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:19:03.267 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.889188+0000", 2024-08-19T14:19:03.267 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3943000000000001e-05 2024-08-19T14:19:03.267 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:19:03.267 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:19:03.268 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:19:03.268 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.268 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.268 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:8281 2.7 2:edaf96ca:::benchmark_data_smithi045_46415_object8280:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-08-19T14:19:03.268 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:12:51.695230+0000", 2024-08-19T14:19:03.268 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 371.54703672300002, 2024-08-19T14:19:03.269 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.194082428, 2024-08-19T14:19:03.269 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:19:03.269 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:19:03.269 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:19:03.269 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:19:03.270 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:19:03.270 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:19:03.270 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 8281 2024-08-19T14:19:03.270 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.270 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:19:03.270 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.270 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:19:03.270 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.695230+0000", 2024-08-19T14:19:03.270 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:19:03.270 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.270 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.270 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:19:03.270 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.695230+0000", 2024-08-19T14:19:03.271 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:19:03.271 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.271 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.271 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:19:03.271 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.695231+0000", 2024-08-19T14:19:03.271 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0249999999999999e-06 2024-08-19T14:19:03.271 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.271 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.271 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:19:03.271 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.695295+0000", 2024-08-19T14:19:03.271 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4115999999999994e-05 2024-08-19T14:19:03.271 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.271 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.271 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:19:03.272 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.695309+0000", 2024-08-19T14:19:03.272 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.383e-05 2024-08-19T14:19:03.272 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.272 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.272 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:19:03.272 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.695317+0000", 2024-08-19T14:19:03.272 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.0490000000000004e-06 2024-08-19T14:19:03.272 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.272 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.272 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:19:03.272 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.695368+0000", 2024-08-19T14:19:03.272 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1819e-05 2024-08-19T14:19:03.272 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.272 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.273 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:19:03.273 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.695445+0000", 2024-08-19T14:19:03.273 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6705999999999994e-05 2024-08-19T14:19:03.273 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.273 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.273 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,10", 2024-08-19T14:19:03.273 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.695523+0000", 2024-08-19T14:19:03.273 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7989000000000002e-05 2024-08-19T14:19:03.273 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.273 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.273 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:19:03.273 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.696398+0000", 2024-08-19T14:19:03.273 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00087526300000000004 2024-08-19T14:19:03.273 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.273 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.274 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:19:03.274 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.696820+0000", 2024-08-19T14:19:03.274 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00042155799999999999 2024-08-19T14:19:03.274 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.274 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.274 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:19:03.274 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.889226+0000", 2024-08-19T14:19:03.274 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.19240576600000001 2024-08-19T14:19:03.274 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.274 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.274 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:19:03.274 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.889276+0000", 2024-08-19T14:19:03.274 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0389000000000002e-05 2024-08-19T14:19:03.274 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.275 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.275 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:19:03.275 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.889312+0000", 2024-08-19T14:19:03.275 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5917999999999997e-05 2024-08-19T14:19:03.275 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:19:03.275 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:19:03.275 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:19:03.275 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.275 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.275 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:8284 2.7 2:eea07cd5:::benchmark_data_smithi045_46415_object8283:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-08-19T14:19:03.275 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:12:51.695765+0000", 2024-08-19T14:19:03.275 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 371.546501202, 2024-08-19T14:19:03.275 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.19395319999999999, 2024-08-19T14:19:03.275 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:19:03.276 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:19:03.276 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:19:03.276 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:19:03.276 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:19:03.276 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:19:03.276 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 8284 2024-08-19T14:19:03.276 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.276 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:19:03.276 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.276 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:19:03.276 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.695765+0000", 2024-08-19T14:19:03.276 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:19:03.276 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.276 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.277 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:19:03.277 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.695765+0000", 2024-08-19T14:19:03.277 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:19:03.277 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.277 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.277 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:19:03.277 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.695766+0000", 2024-08-19T14:19:03.277 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2399999999999997e-07 2024-08-19T14:19:03.277 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.277 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.277 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:19:03.277 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.695825+0000", 2024-08-19T14:19:03.277 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8702000000000001e-05 2024-08-19T14:19:03.277 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.278 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.278 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:19:03.278 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.695845+0000", 2024-08-19T14:19:03.278 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0500999999999999e-05 2024-08-19T14:19:03.278 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.278 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.278 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:19:03.278 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.695850+0000", 2024-08-19T14:19:03.278 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3319999999999999e-06 2024-08-19T14:19:03.278 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.278 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.278 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:19:03.278 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.695932+0000", 2024-08-19T14:19:03.278 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.2089000000000006e-05 2024-08-19T14:19:03.279 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.279 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.279 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:19:03.279 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.696041+0000", 2024-08-19T14:19:03.279 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000108931 2024-08-19T14:19:03.279 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.279 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.279 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,10", 2024-08-19T14:19:03.279 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.696129+0000", 2024-08-19T14:19:03.279 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.7545000000000003e-05 2024-08-19T14:19:03.279 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.279 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.279 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:19:03.279 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.696793+0000", 2024-08-19T14:19:03.280 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000663946 2024-08-19T14:19:03.280 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.280 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.280 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:19:03.280 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.697250+0000", 2024-08-19T14:19:03.280 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000457541 2024-08-19T14:19:03.280 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.280 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.280 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:19:03.280 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.889620+0000", 2024-08-19T14:19:03.280 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.19236932600000001 2024-08-19T14:19:03.280 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.280 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.280 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:19:03.280 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.889680+0000", 2024-08-19T14:19:03.281 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0417000000000001e-05 2024-08-19T14:19:03.281 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.281 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.281 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:19:03.281 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.889718+0000", 2024-08-19T14:19:03.281 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8145999999999997e-05 2024-08-19T14:19:03.281 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:19:03.281 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:19:03.281 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:19:03.281 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.281 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.281 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:8285 2.7 2:e4063f99:::benchmark_data_smithi045_46415_object8284:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-08-19T14:19:03.281 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:12:51.695865+0000", 2024-08-19T14:19:03.282 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 371.54640133599997, 2024-08-19T14:19:03.282 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.19405726300000001, 2024-08-19T14:19:03.282 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:19:03.282 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:19:03.282 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:19:03.282 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:19:03.282 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:19:03.282 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:19:03.282 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 8285 2024-08-19T14:19:03.282 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.282 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:19:03.282 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.282 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:19:03.282 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.695865+0000", 2024-08-19T14:19:03.283 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:19:03.283 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.283 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.283 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:19:03.283 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.695865+0000", 2024-08-19T14:19:03.283 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:19:03.283 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.283 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.283 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:19:03.283 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.695866+0000", 2024-08-19T14:19:03.283 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8999999999999996e-07 2024-08-19T14:19:03.283 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.283 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.283 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:19:03.283 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.695901+0000", 2024-08-19T14:19:03.284 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5859999999999999e-05 2024-08-19T14:19:03.284 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.284 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.284 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:19:03.284 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.695913+0000", 2024-08-19T14:19:03.284 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1114000000000001e-05 2024-08-19T14:19:03.284 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.284 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.284 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:19:03.284 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.695920+0000", 2024-08-19T14:19:03.284 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1949999999999997e-06 2024-08-19T14:19:03.284 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.284 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.284 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:19:03.285 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.696419+0000", 2024-08-19T14:19:03.285 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00049909300000000004 2024-08-19T14:19:03.285 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.285 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.285 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:19:03.285 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.696489+0000", 2024-08-19T14:19:03.285 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9915999999999999e-05 2024-08-19T14:19:03.285 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.285 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.285 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,10", 2024-08-19T14:19:03.285 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.696578+0000", 2024-08-19T14:19:03.285 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.8825000000000004e-05 2024-08-19T14:19:03.285 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.285 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.286 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:19:03.286 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.696895+0000", 2024-08-19T14:19:03.286 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000317912 2024-08-19T14:19:03.286 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.286 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.286 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:19:03.286 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.697442+0000", 2024-08-19T14:19:03.286 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00054631000000000005 2024-08-19T14:19:03.286 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.286 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.286 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:19:03.286 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.889765+0000", 2024-08-19T14:19:03.286 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.192323409 2024-08-19T14:19:03.286 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.287 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.287 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:19:03.287 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.889882+0000", 2024-08-19T14:19:03.287 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00011684 2024-08-19T14:19:03.287 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.287 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.287 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:19:03.287 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.889922+0000", 2024-08-19T14:19:03.287 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0198999999999999e-05 2024-08-19T14:19:03.287 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:19:03.287 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:19:03.287 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:19:03.287 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.287 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.288 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:8318 2.7 2:e0a01dab:::benchmark_data_smithi045_46415_object8317:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-08-19T14:19:03.288 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:12:51.706539+0000", 2024-08-19T14:19:03.288 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 371.535727495, 2024-08-19T14:19:03.288 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.183777518, 2024-08-19T14:19:03.288 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:19:03.288 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:19:03.288 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:19:03.288 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:19:03.288 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:19:03.288 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:19:03.288 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 8318 2024-08-19T14:19:03.288 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.288 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:19:03.288 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.288 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:19:03.289 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.706539+0000", 2024-08-19T14:19:03.289 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:19:03.289 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.289 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.289 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:19:03.289 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.706539+0000", 2024-08-19T14:19:03.289 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:19:03.289 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.289 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.289 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:19:03.289 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.706541+0000", 2024-08-19T14:19:03.289 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8050000000000001e-06 2024-08-19T14:19:03.289 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.289 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.290 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:19:03.290 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.706582+0000", 2024-08-19T14:19:03.290 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1748000000000001e-05 2024-08-19T14:19:03.290 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.290 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.290 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:19:03.290 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.706605+0000", 2024-08-19T14:19:03.290 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2614000000000001e-05 2024-08-19T14:19:03.290 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.290 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.290 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:19:03.290 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.706615+0000", 2024-08-19T14:19:03.290 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.825e-06 2024-08-19T14:19:03.291 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.291 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.291 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:19:03.291 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.706681+0000", 2024-08-19T14:19:03.291 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.5729999999999996e-05 2024-08-19T14:19:03.291 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.291 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.291 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:19:03.291 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.706811+0000", 2024-08-19T14:19:03.291 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013086200000000001 2024-08-19T14:19:03.291 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.291 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.291 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,10", 2024-08-19T14:19:03.291 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.706948+0000", 2024-08-19T14:19:03.292 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013628800000000001 2024-08-19T14:19:03.292 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.292 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.292 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:19:03.292 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.707654+0000", 2024-08-19T14:19:03.292 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00070644600000000001 2024-08-19T14:19:03.292 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.292 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.292 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:19:03.292 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.707895+0000", 2024-08-19T14:19:03.292 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00024116599999999999 2024-08-19T14:19:03.292 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.292 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.292 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:19:03.293 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.890219+0000", 2024-08-19T14:19:03.293 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.18232368199999999 2024-08-19T14:19:03.293 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.293 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.293 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:19:03.293 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.890279+0000", 2024-08-19T14:19:03.293 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9941000000000001e-05 2024-08-19T14:19:03.293 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.293 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.293 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:19:03.293 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.890316+0000", 2024-08-19T14:19:03.293 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7410999999999998e-05 2024-08-19T14:19:03.293 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:19:03.293 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:19:03.293 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:19:03.294 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.294 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.294 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:48872 2.7 2:e2eb060c:::benchmark_data_smithi045_46415_object48871:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-08-19T14:19:03.294 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:07.265121+0000", 2024-08-19T14:19:03.294 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 355.97714528799997, 2024-08-19T14:19:03.294 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20642575599999999, 2024-08-19T14:19:03.294 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:19:03.294 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:19:03.294 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:19:03.294 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:19:03.294 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:19:03.294 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:19:03.294 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 48872 2024-08-19T14:19:03.295 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.295 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:19:03.295 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.295 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:19:03.295 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.265121+0000", 2024-08-19T14:19:03.295 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:19:03.295 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.295 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.295 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:19:03.295 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.265121+0000", 2024-08-19T14:19:03.295 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:19:03.295 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.295 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.295 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:19:03.295 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.265123+0000", 2024-08-19T14:19:03.296 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.494e-06 2024-08-19T14:19:03.296 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.296 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.296 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:19:03.296 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.265156+0000", 2024-08-19T14:19:03.296 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3651e-05 2024-08-19T14:19:03.296 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.296 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.296 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:19:03.296 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.265167+0000", 2024-08-19T14:19:03.296 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0669000000000001e-05 2024-08-19T14:19:03.296 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.296 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.296 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:19:03.297 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.265174+0000", 2024-08-19T14:19:03.297 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8750000000000002e-06 2024-08-19T14:19:03.297 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.297 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.297 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:19:03.297 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.265293+0000", 2024-08-19T14:19:03.297 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000119067 2024-08-19T14:19:03.297 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.297 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.297 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:19:03.297 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.265408+0000", 2024-08-19T14:19:03.297 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000115519 2024-08-19T14:19:03.297 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.297 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.298 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,10", 2024-08-19T14:19:03.298 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.265541+0000", 2024-08-19T14:19:03.298 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013279200000000001 2024-08-19T14:19:03.298 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.298 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.298 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:19:03.298 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.268514+0000", 2024-08-19T14:19:03.298 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0029729259999999999 2024-08-19T14:19:03.298 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.298 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.298 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:19:03.298 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.268576+0000", 2024-08-19T14:19:03.298 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2145999999999998e-05 2024-08-19T14:19:03.298 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.299 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.299 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:19:03.299 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.471473+0000", 2024-08-19T14:19:03.299 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.202897204 2024-08-19T14:19:03.299 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.299 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.299 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:19:03.299 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.471514+0000", 2024-08-19T14:19:03.299 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0484000000000001e-05 2024-08-19T14:19:03.299 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.299 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.299 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:19:03.299 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.471547+0000", 2024-08-19T14:19:03.300 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2929000000000003e-05 2024-08-19T14:19:03.300 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:19:03.300 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:19:03.300 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:19:03.300 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.300 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.300 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:48876 2.7 2:e761e839:::benchmark_data_smithi045_46415_object48875:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-08-19T14:19:03.300 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:07.265962+0000", 2024-08-19T14:19:03.300 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 355.97630396900001, 2024-08-19T14:19:03.300 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20685887, 2024-08-19T14:19:03.300 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:19:03.300 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:19:03.300 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:19:03.300 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:19:03.300 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:19:03.301 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:19:03.301 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 48876 2024-08-19T14:19:03.301 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.301 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:19:03.301 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.301 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:19:03.301 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.265962+0000", 2024-08-19T14:19:03.301 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:19:03.301 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.301 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.301 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:19:03.301 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.265962+0000", 2024-08-19T14:19:03.301 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:19:03.301 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.302 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.302 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:19:03.302 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.265963+0000", 2024-08-19T14:19:03.302 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0729999999999999e-06 2024-08-19T14:19:03.302 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.302 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.302 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:19:03.302 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.266042+0000", 2024-08-19T14:19:03.302 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.8845999999999997e-05 2024-08-19T14:19:03.302 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.302 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.302 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:19:03.302 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.266060+0000", 2024-08-19T14:19:03.302 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.783e-05 2024-08-19T14:19:03.303 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.303 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.303 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:19:03.303 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.266068+0000", 2024-08-19T14:19:03.303 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7009999999999996e-06 2024-08-19T14:19:03.303 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.303 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.303 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:19:03.303 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.266916+0000", 2024-08-19T14:19:03.303 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00084853200000000004 2024-08-19T14:19:03.303 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.303 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.303 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:19:03.303 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.267096+0000", 2024-08-19T14:19:03.304 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:19:03.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:03 smithi031 ceph-mon[16336]: pgmap v669: 67 pgs: 67 active+clean; 4.8 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 1.82k op/s 2024-08-19T14:19:03.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:03 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "46"}]': finished 2024-08-19T14:19:03.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:03 smithi031 ceph-mon[16336]: osdmap e296: 12 total, 12 up, 9 in 2024-08-19T14:19:03.306 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00017937699999999999 2024-08-19T14:19:03.306 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.306 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.306 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,10", 2024-08-19T14:19:03.306 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.267225+0000", 2024-08-19T14:19:03.306 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000129727 2024-08-19T14:19:03.306 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.306 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.306 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:19:03.306 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.268638+0000", 2024-08-19T14:19:03.306 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0014122640000000001 2024-08-19T14:19:03.306 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.306 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.306 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:19:03.307 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.269422+0000", 2024-08-19T14:19:03.307 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00078479600000000004 2024-08-19T14:19:03.307 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.307 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.307 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:19:03.307 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.472763+0000", 2024-08-19T14:19:03.307 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.203340627 2024-08-19T14:19:03.307 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.307 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.307 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:19:03.307 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.472797+0000", 2024-08-19T14:19:03.307 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.4060999999999998e-05 2024-08-19T14:19:03.307 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.307 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.308 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:19:03.308 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.472821+0000", 2024-08-19T14:19:03.308 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4036000000000002e-05 2024-08-19T14:19:03.308 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:19:03.308 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:19:03.308 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:19:03.308 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.308 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.308 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:48899 2.7 2:ed417148:::benchmark_data_smithi045_46415_object48898:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-08-19T14:19:03.308 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:07.275772+0000", 2024-08-19T14:19:03.308 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 355.96649423000002, 2024-08-19T14:19:03.308 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.19712979899999999, 2024-08-19T14:19:03.308 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:19:03.308 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:19:03.309 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:19:03.309 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:19:03.309 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:19:03.309 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:19:03.309 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 48899 2024-08-19T14:19:03.309 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.309 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:19:03.309 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.309 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:19:03.309 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.275772+0000", 2024-08-19T14:19:03.309 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:19:03.309 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.309 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.309 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:19:03.309 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.275772+0000", 2024-08-19T14:19:03.310 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:19:03.310 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.310 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.310 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:19:03.310 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.275774+0000", 2024-08-19T14:19:03.310 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5009999999999999e-06 2024-08-19T14:19:03.310 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.310 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.310 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:19:03.310 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.275826+0000", 2024-08-19T14:19:03.310 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2824000000000001e-05 2024-08-19T14:19:03.310 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.310 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.310 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:19:03.311 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.275841+0000", 2024-08-19T14:19:03.311 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4691e-05 2024-08-19T14:19:03.311 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.311 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.311 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:19:03.311 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.275849+0000", 2024-08-19T14:19:03.311 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1580000000000005e-06 2024-08-19T14:19:03.311 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.311 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.311 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:19:03.311 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.275913+0000", 2024-08-19T14:19:03.311 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3590999999999995e-05 2024-08-19T14:19:03.311 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.311 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.312 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:19:03.312 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.276130+0000", 2024-08-19T14:19:03.312 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000216901 2024-08-19T14:19:03.312 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.312 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.312 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,10", 2024-08-19T14:19:03.312 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.276273+0000", 2024-08-19T14:19:03.312 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014370100000000001 2024-08-19T14:19:03.312 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.312 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.312 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:19:03.312 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.278099+0000", 2024-08-19T14:19:03.312 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0018257849999999999 2024-08-19T14:19:03.312 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.313 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.313 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:19:03.313 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.281287+0000", 2024-08-19T14:19:03.313 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0031876769999999999 2024-08-19T14:19:03.313 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.313 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.313 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:19:03.313 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.472854+0000", 2024-08-19T14:19:03.313 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.19156674300000001 2024-08-19T14:19:03.313 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.313 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.313 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:19:03.313 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.472881+0000", 2024-08-19T14:19:03.313 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7521999999999998e-05 2024-08-19T14:19:03.313 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.314 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.314 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:19:03.314 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.472902+0000", 2024-08-19T14:19:03.314 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0704999999999999e-05 2024-08-19T14:19:03.314 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:19:03.314 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:19:03.314 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:19:03.314 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.314 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.314 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:48912 2.7 2:e6ffd652:::benchmark_data_smithi045_46415_object48911:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-08-19T14:19:03.314 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:07.279334+0000", 2024-08-19T14:19:03.314 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 355.96293272899999, 2024-08-19T14:19:03.314 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.195037181, 2024-08-19T14:19:03.314 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:19:03.315 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:19:03.315 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:19:03.315 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:19:03.315 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:19:03.315 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:19:03.315 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 48912 2024-08-19T14:19:03.315 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.315 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:19:03.315 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.315 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:19:03.315 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.279334+0000", 2024-08-19T14:19:03.315 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:19:03.315 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.315 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.316 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:19:03.316 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.279334+0000", 2024-08-19T14:19:03.316 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:19:03.316 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.316 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.316 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:19:03.316 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.279335+0000", 2024-08-19T14:19:03.316 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3599999999999999e-06 2024-08-19T14:19:03.316 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.316 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.316 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:19:03.316 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.279409+0000", 2024-08-19T14:19:03.316 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3819000000000006e-05 2024-08-19T14:19:03.316 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.317 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.317 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:19:03.317 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.279420+0000", 2024-08-19T14:19:03.317 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0892e-05 2024-08-19T14:19:03.317 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.317 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.317 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:19:03.317 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.279427+0000", 2024-08-19T14:19:03.317 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2790000000000001e-06 2024-08-19T14:19:03.317 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.317 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.317 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:19:03.317 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.279479+0000", 2024-08-19T14:19:03.317 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2547999999999999e-05 2024-08-19T14:19:03.318 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.318 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.318 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:19:03.318 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.279613+0000", 2024-08-19T14:19:03.318 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013337900000000001 2024-08-19T14:19:03.318 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.318 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.318 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,10", 2024-08-19T14:19:03.318 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.279754+0000", 2024-08-19T14:19:03.318 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000141359 2024-08-19T14:19:03.318 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.318 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.318 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:19:03.318 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.281352+0000", 2024-08-19T14:19:03.319 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0015973070000000001 2024-08-19T14:19:03.319 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.319 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.319 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:19:03.319 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.282744+0000", 2024-08-19T14:19:03.319 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0013921630000000001 2024-08-19T14:19:03.319 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.319 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.319 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:19:03.319 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.474309+0000", 2024-08-19T14:19:03.319 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.19156536800000001 2024-08-19T14:19:03.319 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.319 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.319 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:19:03.320 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.474347+0000", 2024-08-19T14:19:03.320 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.824e-05 2024-08-19T14:19:03.320 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.320 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.320 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:19:03.320 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.474371+0000", 2024-08-19T14:19:03.320 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3467000000000001e-05 2024-08-19T14:19:03.320 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:19:03.320 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:19:03.320 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:19:03.320 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.320 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.320 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:48921 2.7 2:e2434a00:::benchmark_data_smithi045_46415_object48920:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-08-19T14:19:03.321 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:07.283150+0000", 2024-08-19T14:19:03.321 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 355.95911649599998, 2024-08-19T14:19:03.321 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.19130543999999999, 2024-08-19T14:19:03.321 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:19:03.321 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:19:03.321 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:19:03.321 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:19:03.321 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:19:03.321 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:19:03.321 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 48921 2024-08-19T14:19:03.321 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.321 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:19:03.321 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.321 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:19:03.322 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.283150+0000", 2024-08-19T14:19:03.322 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:19:03.322 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.322 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.322 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:19:03.322 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.283150+0000", 2024-08-19T14:19:03.322 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:19:03.322 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.322 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.322 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:19:03.322 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.283151+0000", 2024-08-19T14:19:03.322 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6330000000000001e-06 2024-08-19T14:19:03.322 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.322 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.323 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:19:03.323 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.283177+0000", 2024-08-19T14:19:03.323 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5497e-05 2024-08-19T14:19:03.323 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.323 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.323 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:19:03.323 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.283190+0000", 2024-08-19T14:19:03.323 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3315e-05 2024-08-19T14:19:03.323 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.323 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.323 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:19:03.323 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.283197+0000", 2024-08-19T14:19:03.323 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7479999999999996e-06 2024-08-19T14:19:03.323 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.324 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.324 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:19:03.324 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.283262+0000", 2024-08-19T14:19:03.324 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4550999999999999e-05 2024-08-19T14:19:03.324 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.324 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.324 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:19:03.324 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.283434+0000", 2024-08-19T14:19:03.324 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00017221700000000001 2024-08-19T14:19:03.324 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.324 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.324 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,10", 2024-08-19T14:19:03.324 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.283553+0000", 2024-08-19T14:19:03.324 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0001197 2024-08-19T14:19:03.325 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.325 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.325 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:19:03.325 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.285150+0000", 2024-08-19T14:19:03.325 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00159615 2024-08-19T14:19:03.325 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.325 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.325 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:19:03.325 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.285949+0000", 2024-08-19T14:19:03.325 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00079935399999999995 2024-08-19T14:19:03.325 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.325 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.325 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:19:03.325 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.474403+0000", 2024-08-19T14:19:03.325 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.18845408199999999 2024-08-19T14:19:03.326 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.326 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.326 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:19:03.326 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.474433+0000", 2024-08-19T14:19:03.326 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9584999999999999e-05 2024-08-19T14:19:03.326 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.326 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.326 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:19:03.326 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.474455+0000", 2024-08-19T14:19:03.326 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2608000000000001e-05 2024-08-19T14:19:03.326 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:19:03.326 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:19:03.326 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:19:03.327 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.327 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.327 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:48971 2.7 2:ecf6a539:::benchmark_data_smithi045_46415_object48970:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-08-19T14:19:03.327 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:07.303688+0000", 2024-08-19T14:19:03.327 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 355.93857849400001, 2024-08-19T14:19:03.327 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.172252192, 2024-08-19T14:19:03.327 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:19:03.327 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:19:03.327 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:19:03.327 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:19:03.327 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:19:03.327 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:19:03.327 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 48971 2024-08-19T14:19:03.327 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.328 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:19:03.328 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.328 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:19:03.328 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.303688+0000", 2024-08-19T14:19:03.328 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:19:03.328 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.328 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.328 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:19:03.328 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.303688+0000", 2024-08-19T14:19:03.328 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:19:03.328 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.328 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.328 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:19:03.328 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.303689+0000", 2024-08-19T14:19:03.329 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.652e-06 2024-08-19T14:19:03.329 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.329 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.329 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:19:03.329 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.303729+0000", 2024-08-19T14:19:03.329 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9443000000000002e-05 2024-08-19T14:19:03.329 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.329 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.329 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:19:03.329 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.303744+0000", 2024-08-19T14:19:03.329 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5173e-05 2024-08-19T14:19:03.329 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.329 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.329 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:19:03.329 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.303751+0000", 2024-08-19T14:19:03.330 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.5910000000000001e-06 2024-08-19T14:19:03.330 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.330 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.330 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:19:03.330 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.303807+0000", 2024-08-19T14:19:03.330 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5918000000000002e-05 2024-08-19T14:19:03.330 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.330 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.330 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:19:03.330 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.303888+0000", 2024-08-19T14:19:03.330 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1302000000000002e-05 2024-08-19T14:19:03.330 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.330 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.330 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,10", 2024-08-19T14:19:03.331 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.303973+0000", 2024-08-19T14:19:03.331 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.5055000000000005e-05 2024-08-19T14:19:03.331 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.331 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.331 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:19:03.331 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.305453+0000", 2024-08-19T14:19:03.331 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001480182 2024-08-19T14:19:03.331 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.331 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.331 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:19:03.331 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.306105+0000", 2024-08-19T14:19:03.331 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00065189100000000004 2024-08-19T14:19:03.331 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.331 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.331 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:19:03.332 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.475889+0000", 2024-08-19T14:19:03.332 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.16978352599999999 2024-08-19T14:19:03.332 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.332 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.332 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:19:03.332 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.475919+0000", 2024-08-19T14:19:03.332 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0199e-05 2024-08-19T14:19:03.332 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.332 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.332 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:19:03.332 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.475940+0000", 2024-08-19T14:19:03.332 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.126e-05 2024-08-19T14:19:03.332 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:19:03.332 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:19:03.333 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:19:03.333 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.333 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.333 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:62930 2.7 2.377d1257 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-08-19T14:19:03.333 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:18.725228+0000", 2024-08-19T14:19:03.333 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 344.51703811700003, 2024-08-19T14:19:03.333 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.99330006599999998, 2024-08-19T14:19:03.333 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:19:03.333 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:19:03.333 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:19:03.333 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:19:03.333 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:19:03.333 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:19:03.333 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 62930 2024-08-19T14:19:03.334 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.334 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:19:03.334 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.334 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:19:03.334 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.725228+0000", 2024-08-19T14:19:03.334 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:19:03.334 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.334 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.334 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:19:03.334 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.725228+0000", 2024-08-19T14:19:03.334 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:19:03.334 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.334 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.334 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:19:03.335 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.725231+0000", 2024-08-19T14:19:03.335 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5950000000000001e-06 2024-08-19T14:19:03.335 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.335 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.335 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:19:03.335 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.725270+0000", 2024-08-19T14:19:03.335 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9162000000000002e-05 2024-08-19T14:19:03.335 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.335 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.335 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:19:03.335 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.725291+0000", 2024-08-19T14:19:03.335 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1220999999999999e-05 2024-08-19T14:19:03.335 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.335 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.336 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:19:03.336 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.725303+0000", 2024-08-19T14:19:03.336 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1788e-05 2024-08-19T14:19:03.336 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.336 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.336 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:19:03.336 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.725382+0000", 2024-08-19T14:19:03.336 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.9389999999999997e-05 2024-08-19T14:19:03.336 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.336 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.336 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:19:03.336 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.725395+0000", 2024-08-19T14:19:03.336 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.308e-05 2024-08-19T14:19:03.336 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.337 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.337 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:19:03.337 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.718510+0000", 2024-08-19T14:19:03.337 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.99311438900000004 2024-08-19T14:19:03.337 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.337 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.337 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:19:03.337 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.718528+0000", 2024-08-19T14:19:03.337 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8440999999999998e-05 2024-08-19T14:19:03.337 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:19:03.337 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:19:03.337 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:19:03.337 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.337 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.337 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:62932 2.7 2.84394fd7 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-08-19T14:19:03.338 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:18.727423+0000", 2024-08-19T14:19:03.338 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 344.514842956, 2024-08-19T14:19:03.338 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.99114474299999999, 2024-08-19T14:19:03.338 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:19:03.338 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:19:03.338 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:19:03.338 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:19:03.338 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:19:03.338 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:19:03.338 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 62932 2024-08-19T14:19:03.338 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.338 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:19:03.338 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.338 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:19:03.339 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.727423+0000", 2024-08-19T14:19:03.339 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:19:03.339 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.339 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.339 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:19:03.339 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.727423+0000", 2024-08-19T14:19:03.339 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:19:03.339 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.339 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.339 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:19:03.339 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.727425+0000", 2024-08-19T14:19:03.339 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0870000000000001e-06 2024-08-19T14:19:03.339 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.339 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.340 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:19:03.340 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.727471+0000", 2024-08-19T14:19:03.340 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5300999999999999e-05 2024-08-19T14:19:03.340 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.340 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.340 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:19:03.340 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.727489+0000", 2024-08-19T14:19:03.340 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7909000000000001e-05 2024-08-19T14:19:03.340 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.340 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.340 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:19:03.340 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.727497+0000", 2024-08-19T14:19:03.340 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.2150000000000001e-06 2024-08-19T14:19:03.340 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.341 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.341 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:19:03.341 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.727590+0000", 2024-08-19T14:19:03.341 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3193e-05 2024-08-19T14:19:03.341 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.341 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.341 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:19:03.341 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.727606+0000", 2024-08-19T14:19:03.341 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6345e-05 2024-08-19T14:19:03.341 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.341 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.341 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:19:03.341 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.718558+0000", 2024-08-19T14:19:03.341 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.99095196399999996 2024-08-19T14:19:03.341 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.342 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.342 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:19:03.342 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.718568+0000", 2024-08-19T14:19:03.342 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.7289999999999995e-06 2024-08-19T14:19:03.342 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:19:03.342 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:19:03.342 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:19:03.342 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.342 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.342 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:62948 2.7 2.63c33357 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-08-19T14:19:03.342 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:18.736424+0000", 2024-08-19T14:19:03.342 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 344.505842535, 2024-08-19T14:19:03.342 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.98218305500000003, 2024-08-19T14:19:03.342 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:19:03.343 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:19:03.343 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:19:03.343 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:19:03.343 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:19:03.343 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:19:03.343 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 62948 2024-08-19T14:19:03.343 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.343 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:19:03.343 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.343 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:19:03.343 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.736424+0000", 2024-08-19T14:19:03.343 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:19:03.343 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.343 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.344 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:19:03.344 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.736424+0000", 2024-08-19T14:19:03.344 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:19:03.344 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.344 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.344 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:19:03.344 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.736426+0000", 2024-08-19T14:19:03.344 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9980000000000002e-06 2024-08-19T14:19:03.344 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.344 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.344 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:19:03.344 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.736459+0000", 2024-08-19T14:19:03.344 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3155999999999999e-05 2024-08-19T14:19:03.344 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.344 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.345 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:19:03.345 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.736479+0000", 2024-08-19T14:19:03.345 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0052999999999999e-05 2024-08-19T14:19:03.345 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.345 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.345 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:19:03.345 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.736489+0000", 2024-08-19T14:19:03.345 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.7790000000000007e-06 2024-08-19T14:19:03.345 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.345 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.345 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:19:03.345 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.736567+0000", 2024-08-19T14:19:03.345 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.8240999999999998e-05 2024-08-19T14:19:03.345 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.346 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.346 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:19:03.346 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.736578+0000", 2024-08-19T14:19:03.346 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0813e-05 2024-08-19T14:19:03.346 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.346 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.346 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:19:03.347 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:19:03.348 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:19:03.348 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.718594+0000", 2024-08-19T14:19:03.348 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.98201632900000002 2024-08-19T14:19:03.348 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.348 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.348 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:19:03.348 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.718607+0000", 2024-08-19T14:19:03.348 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2686e-05 2024-08-19T14:19:03.348 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:19:03.348 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:19:03.348 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:19:03.349 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.349 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.349 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:62956 2.7 2.bf121107 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-08-19T14:19:03.349 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:18.741261+0000", 2024-08-19T14:19:03.349 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 344.50100510599998, 2024-08-19T14:19:03.349 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.97739631299999996, 2024-08-19T14:19:03.349 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:19:03.349 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:19:03.349 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:19:03.349 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:19:03.349 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:19:03.349 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:19:03.349 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 62956 2024-08-19T14:19:03.349 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.350 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:19:03.350 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.350 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:19:03.350 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.741261+0000", 2024-08-19T14:19:03.350 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:19:03.350 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.350 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.350 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:19:03.350 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.741261+0000", 2024-08-19T14:19:03.350 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:19:03.350 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.350 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.350 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:19:03.350 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.741263+0000", 2024-08-19T14:19:03.351 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.68e-06 2024-08-19T14:19:03.351 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.351 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.351 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:19:03.351 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.741301+0000", 2024-08-19T14:19:03.351 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8313999999999999e-05 2024-08-19T14:19:03.351 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.351 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.351 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:19:03.351 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.741320+0000", 2024-08-19T14:19:03.351 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.927e-05 2024-08-19T14:19:03.351 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.351 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.351 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:19:03.351 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.741328+0000", 2024-08-19T14:19:03.352 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3000000000000004e-06 2024-08-19T14:19:03.352 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.352 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.352 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:19:03.352 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.741408+0000", 2024-08-19T14:19:03.352 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.9839000000000006e-05 2024-08-19T14:19:03.352 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.352 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.352 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:19:03.352 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.741421+0000", 2024-08-19T14:19:03.352 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2914e-05 2024-08-19T14:19:03.352 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.352 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.353 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:19:03.353 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.718639+0000", 2024-08-19T14:19:03.353 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.97721861399999999 2024-08-19T14:19:03.353 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.353 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.353 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:19:03.353 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.718658+0000", 2024-08-19T14:19:03.353 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8382000000000001e-05 2024-08-19T14:19:03.353 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:19:03.353 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:19:03.353 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:19:03.353 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.353 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.353 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:62963 2.7 2.30acc547 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-08-19T14:19:03.353 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:18.745404+0000", 2024-08-19T14:19:03.354 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 344.49686256299998, 2024-08-19T14:19:03.354 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.97330407100000005, 2024-08-19T14:19:03.354 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:19:03.354 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:19:03.354 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:19:03.354 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:19:03.354 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:19:03.354 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:19:03.354 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 62963 2024-08-19T14:19:03.354 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.354 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:19:03.354 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.354 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:19:03.354 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.745404+0000", 2024-08-19T14:19:03.355 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:19:03.355 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.355 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.355 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:19:03.355 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.745404+0000", 2024-08-19T14:19:03.355 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:19:03.355 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.355 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.355 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:19:03.355 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.745406+0000", 2024-08-19T14:19:03.355 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.328e-06 2024-08-19T14:19:03.355 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.355 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.356 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:19:03.356 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.745440+0000", 2024-08-19T14:19:03.356 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3837000000000001e-05 2024-08-19T14:19:03.356 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.356 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.356 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:19:03.356 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.745474+0000", 2024-08-19T14:19:03.356 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.4196999999999998e-05 2024-08-19T14:19:03.356 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.356 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.356 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:19:03.356 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.745485+0000", 2024-08-19T14:19:03.356 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.095e-05 2024-08-19T14:19:03.356 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.357 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.357 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:19:03.357 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.745568+0000", 2024-08-19T14:19:03.357 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.2770999999999997e-05 2024-08-19T14:19:03.357 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.357 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.357 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:19:03.357 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.745577+0000", 2024-08-19T14:19:03.357 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.1269999999999992e-06 2024-08-19T14:19:03.357 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.357 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.357 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:19:03.357 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.718689+0000", 2024-08-19T14:19:03.357 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.97311253600000003 2024-08-19T14:19:03.358 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.358 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.358 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:19:03.358 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.718708+0000", 2024-08-19T14:19:03.358 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8325000000000002e-05 2024-08-19T14:19:03.358 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:19:03.358 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:19:03.358 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:19:03.358 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.358 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.358 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:62983 2.7 2.e3420e17 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-08-19T14:19:03.358 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:18.759553+0000", 2024-08-19T14:19:03.358 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 344.48271320399999, 2024-08-19T14:19:03.358 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.95919200800000004, 2024-08-19T14:19:03.359 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:19:03.359 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:19:03.359 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:19:03.359 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:19:03.359 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:19:03.359 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:19:03.359 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 62983 2024-08-19T14:19:03.359 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.359 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:19:03.359 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.359 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:19:03.359 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.759553+0000", 2024-08-19T14:19:03.359 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:19:03.359 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.360 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.360 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:19:03.360 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.759553+0000", 2024-08-19T14:19:03.360 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:19:03.360 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.360 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.360 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:19:03.360 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.759555+0000", 2024-08-19T14:19:03.360 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2630000000000002e-06 2024-08-19T14:19:03.360 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.360 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.360 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:19:03.360 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.759595+0000", 2024-08-19T14:19:03.360 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9490000000000003e-05 2024-08-19T14:19:03.360 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.361 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.361 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:19:03.361 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.759614+0000", 2024-08-19T14:19:03.361 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9446999999999999e-05 2024-08-19T14:19:03.361 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.361 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.361 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:19:03.361 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.759622+0000", 2024-08-19T14:19:03.361 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5519999999999999e-06 2024-08-19T14:19:03.361 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.361 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.361 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:19:03.361 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.759684+0000", 2024-08-19T14:19:03.362 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2168000000000005e-05 2024-08-19T14:19:03.362 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.362 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.362 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:19:03.362 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.759693+0000", 2024-08-19T14:19:03.362 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.8920000000000006e-06 2024-08-19T14:19:03.362 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.362 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.362 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:19:03.362 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.718735+0000", 2024-08-19T14:19:03.362 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.95904258099999995 2024-08-19T14:19:03.362 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.362 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.362 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:19:03.363 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.718745+0000", 2024-08-19T14:19:03.363 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.6150000000000003e-06 2024-08-19T14:19:03.363 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:19:03.363 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:19:03.363 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:19:03.363 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.363 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.363 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:63032 2.7 2.24f95377 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-08-19T14:19:03.363 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:18.809762+0000", 2024-08-19T14:19:03.363 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 344.432504521, 2024-08-19T14:19:03.363 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.909118655, 2024-08-19T14:19:03.363 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:19:03.363 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:19:03.363 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:19:03.364 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:19:03.364 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:19:03.364 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:19:03.364 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 63032 2024-08-19T14:19:03.364 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.364 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:19:03.364 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.364 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:19:03.364 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.809762+0000", 2024-08-19T14:19:03.364 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:19:03.364 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.364 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.364 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:19:03.364 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.809762+0000", 2024-08-19T14:19:03.364 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:19:03.365 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.365 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.365 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:19:03.365 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.809764+0000", 2024-08-19T14:19:03.365 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1490000000000001e-06 2024-08-19T14:19:03.365 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.365 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.365 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:19:03.365 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.809805+0000", 2024-08-19T14:19:03.365 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1241e-05 2024-08-19T14:19:03.365 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.365 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.365 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:19:03.365 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.809820+0000", 2024-08-19T14:19:03.366 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4355000000000001e-05 2024-08-19T14:19:03.366 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.366 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.366 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:19:03.366 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.809826+0000", 2024-08-19T14:19:03.366 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4829999999999996e-06 2024-08-19T14:19:03.366 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.366 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.366 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:19:03.366 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.809883+0000", 2024-08-19T14:19:03.366 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6885000000000002e-05 2024-08-19T14:19:03.366 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.366 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.366 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:19:03.367 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.809892+0000", 2024-08-19T14:19:03.367 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.8519999999999993e-06 2024-08-19T14:19:03.367 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.367 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.367 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:19:03.367 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.718868+0000", 2024-08-19T14:19:03.367 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.90897634900000002 2024-08-19T14:19:03.367 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:19:03.367 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:19:03.367 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:19:03.367 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.718880+0000", 2024-08-19T14:19:03.367 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2340999999999999e-05 2024-08-19T14:19:03.367 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:19:03.367 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:19:03.368 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:19:03.368 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:19:03.368 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:19:03.368 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:19:03.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:03.369 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:03.369 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:19:03.369 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:19:03.370 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:19:03.373 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:19:03.373 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:19:03.402 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:19:03.402 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:19:03.432 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:19:03.433 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:19:03.467 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:19:03.467 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:19:03.470 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:19:03.494 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:19:03.494 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:19:03.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:03.503 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:03.504 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:19:03.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:03 smithi032 ceph-mon[20004]: pgmap v669: 67 pgs: 67 active+clean; 4.8 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 1.82k op/s 2024-08-19T14:19:03.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:03 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "46"}]': finished 2024-08-19T14:19:03.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:03 smithi032 ceph-mon[20004]: osdmap e296: 12 total, 12 up, 9 in 2024-08-19T14:19:03.527 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:19:03.528 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:19:03.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:03 smithi045 ceph-mon[20143]: pgmap v669: 67 pgs: 67 active+clean; 4.8 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 1.82k op/s 2024-08-19T14:19:03.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:03 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "46"}]': finished 2024-08-19T14:19:03.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:03 smithi045 ceph-mon[20143]: osdmap e296: 12 total, 12 up, 9 in 2024-08-19T14:19:03.556 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:19:03.556 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:19:03.590 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:19:03.591 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:19:03.604 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:19:03.624 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:19:03.626 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:19:03.640 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:03.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:03.641 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:19:03.654 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:19:03.654 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:19:03.742 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:19:03.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:03.772 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:03.773 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:19:03.874 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:19:03.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:03.913 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:03.914 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:19:04.015 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:19:04.051 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:04.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:04.052 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:19:04.154 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:19:04.187 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:04.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:04.188 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:19:04.270 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-08-19T14:19:04.289 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:19:04.309 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:04.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:04.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:19:04.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:04 smithi031 ceph-mon[16336]: osdmap e297: 12 total, 12 up, 9 in 2024-08-19T14:19:04.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:04 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/47449873' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:19:04.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:04 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "46"}]: dispatch 2024-08-19T14:19:04.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:04 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "46"}]: dispatch 2024-08-19T14:19:04.411 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:19:04.433 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:04.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:04.434 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:19:04.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:04 smithi032 ceph-mon[20004]: osdmap e297: 12 total, 12 up, 9 in 2024-08-19T14:19:04.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:04 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/47449873' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:19:04.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:04 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "46"}]: dispatch 2024-08-19T14:19:04.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:04 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "46"}]: dispatch 2024-08-19T14:19:04.520 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:19:04.535 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:19:04.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:04 smithi045 ceph-mon[20143]: osdmap e297: 12 total, 12 up, 9 in 2024-08-19T14:19:04.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:04 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/47449873' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:19:04.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:04 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "46"}]: dispatch 2024-08-19T14:19:04.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:04 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "46"}]: dispatch 2024-08-19T14:19:04.556 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:04.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:04.557 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:19:04.659 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:19:04.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:04.681 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:04.681 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:19:04.771 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-19T14:19:04.782 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:19:04.802 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:04.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:04.803 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:19:04.904 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:19:04.928 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:04.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:04.929 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:19:05.021 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:19:05.029 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:19:05.051 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:05.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:05.053 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:19:05.106 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:19:05.106 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:19:05.154 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:19:05.176 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:05.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:05.177 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:19:05.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:05 smithi032 ceph-mon[20004]: pgmap v672: 77 pgs: 1 peering, 8 unknown, 68 active+clean; 4.6 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 1.63k op/s 2024-08-19T14:19:05.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:05 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "46"}]': finished 2024-08-19T14:19:05.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:05 smithi032 ceph-mon[20004]: osdmap e298: 12 total, 12 up, 9 in 2024-08-19T14:19:05.278 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:19:05.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:05.304 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:05.304 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:19:05.406 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:19:05.407 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:05 smithi031 ceph-mon[16336]: pgmap v672: 77 pgs: 1 peering, 8 unknown, 68 active+clean; 4.6 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 1.63k op/s 2024-08-19T14:19:05.407 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:05 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "46"}]': finished 2024-08-19T14:19:05.407 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:05 smithi031 ceph-mon[16336]: osdmap e298: 12 total, 12 up, 9 in 2024-08-19T14:19:05.427 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:05.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:05.428 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:19:05.529 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:19:05.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:05 smithi045 ceph-mon[20143]: pgmap v672: 77 pgs: 1 peering, 8 unknown, 68 active+clean; 4.6 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 1.63k op/s 2024-08-19T14:19:05.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:05 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "46"}]': finished 2024-08-19T14:19:05.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:05 smithi045 ceph-mon[20143]: osdmap e298: 12 total, 12 up, 9 in 2024-08-19T14:19:05.550 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:05.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:05.551 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:19:05.652 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:19:05.674 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:05.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:05.675 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:19:05.737 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-08-19T14:19:05.775 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:19:05.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:05.804 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:05.804 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:19:05.905 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:19:05.930 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:05.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:05.931 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:19:05.988 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:19:06.032 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:19:06.057 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:06.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:06.058 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:19:06.159 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:19:06.179 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is set 2024-08-19T14:19:06.187 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:06.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:06.188 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:19:06.289 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:19:06.322 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:06.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:06.323 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:19:06.424 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:19:06.437 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:19:06.447 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:06.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:06.448 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:19:06.450 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:06 smithi032 ceph-mon[20004]: osdmap e299: 12 total, 12 up, 9 in 2024-08-19T14:19:06.451 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:06 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1553237872' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:19:06.451 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:06 smithi032 ceph-mon[20004]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:19:06.451 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:06 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1553237872' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:19:06.451 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:06 smithi032 ceph-mon[20004]: osdmap e300: 12 total, 12 up, 9 in 2024-08-19T14:19:06.455 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:19:06.455 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:19:06.482 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:06 smithi031 ceph-mon[16336]: osdmap e299: 12 total, 12 up, 9 in 2024-08-19T14:19:06.482 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:06 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1553237872' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:19:06.482 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:06 smithi031 ceph-mon[16336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:19:06.482 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:06 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1553237872' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:19:06.482 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:06 smithi031 ceph-mon[16336]: osdmap e300: 12 total, 12 up, 9 in 2024-08-19T14:19:06.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:06 smithi045 ceph-mon[20143]: osdmap e299: 12 total, 12 up, 9 in 2024-08-19T14:19:06.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:06 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1553237872' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:19:06.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:06 smithi045 ceph-mon[20143]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:19:06.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:06 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1553237872' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:19:06.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:06 smithi045 ceph-mon[20143]: osdmap e300: 12 total, 12 up, 9 in 2024-08-19T14:19:06.548 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:19:06.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:06.607 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:06.607 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:19:06.708 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:19:06.733 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:06.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:06.734 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:19:06.835 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:19:06.860 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:06.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:06.861 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:19:06.961 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:19:06.986 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:06.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:06.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:19:07.000 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:19:07.053 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 7, 8, 9, 11] out_osds: [10, 6, 2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-08-19T14:19:07.053 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-19T14:19:07.054 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-08-19T14:19:07.054 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-19T14:19:07.088 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:19:07.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:07.169 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:07.169 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:19:07.170 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-08-19T14:19:07.270 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:19:07.294 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:07.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:07.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:19:07.375 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:07 smithi031 ceph-mon[16336]: pgmap v675: 77 pgs: 1 peering, 9 unknown, 67 active+clean; 4.3 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 266 op/s 2024-08-19T14:19:07.396 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:19:07.472 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:07.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:07.473 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:19:07.474 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:19:07.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:07 smithi032 ceph-mon[20004]: pgmap v675: 77 pgs: 1 peering, 9 unknown, 67 active+clean; 4.3 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 266 op/s 2024-08-19T14:19:07.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:07 smithi045 ceph-mon[20143]: pgmap v675: 77 pgs: 1 peering, 9 unknown, 67 active+clean; 4.3 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 266 op/s 2024-08-19T14:19:07.573 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:19:07.598 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:07.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:07.599 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:19:07.700 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:19:07.716 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:19:07.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:07.727 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:07.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:19:07.748 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:19:07.748 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:19:07.807 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:19:07.807 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:19:07.828 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:19:07.843 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:19:07.843 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:19:07.893 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:19:07.893 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:19:07.898 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:07.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:07.899 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:19:07.926 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:19:07.926 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:19:07.980 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:19:07.981 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:19:08.000 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:19:08.016 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:19:08.016 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:19:08.022 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:19:08.022 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:19:08.028 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:08.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:08.029 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:19:08.054 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:19:08.054 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:19:08.089 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:19:08.089 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:19:08.124 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:19:08.124 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:19:08.130 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:19:08.146 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:19:08.146 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:19:08.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:08.153 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:08.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:19:08.162 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:19:08.162 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:19:08.254 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:19:08.280 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:08.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:08.281 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:19:08.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:08 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:19:08.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:08 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3747719315' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:19:08.382 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:19:08.409 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:08.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:08.410 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:19:08.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:08 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:19:08.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:08 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3747719315' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:19:08.511 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:19:08.537 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:08.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:08.538 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:19:08.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:08 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:19:08.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:08 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3747719315' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:19:08.639 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:19:08.668 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:08.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:08.669 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:19:08.686 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-08-19T14:19:08.770 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:19:08.795 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:08.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:08.797 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:19:08.898 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:19:08.920 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-19T14:19:08.925 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:08.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:08.927 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:19:09.028 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:19:09.051 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:19:09.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:09.053 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:09.053 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:19:09.154 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:19:09.185 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:09.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:09.186 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:19:09.287 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:19:09.319 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:09.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:09.321 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:19:09.422 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:19:09.447 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:09.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:09.448 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:19:09.462 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:09 smithi045 ceph-mon[20143]: pgmap v677: 77 pgs: 1 peering, 2 remapped+peering, 7 unknown, 67 active+clean; 4.1 GiB data, 16 GiB used, 788 GiB / 805 GiB avail 2024-08-19T14:19:09.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:09 smithi032 ceph-mon[20004]: pgmap v677: 77 pgs: 1 peering, 2 remapped+peering, 7 unknown, 67 active+clean; 4.1 GiB data, 16 GiB used, 788 GiB / 805 GiB avail 2024-08-19T14:19:09.549 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:19:09.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:09.574 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:09.575 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:19:09.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:09 smithi031 ceph-mon[16336]: pgmap v677: 77 pgs: 1 peering, 2 remapped+peering, 7 unknown, 67 active+clean; 4.1 GiB data, 16 GiB used, 788 GiB / 805 GiB avail 2024-08-19T14:19:09.676 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:19:09.704 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:19:09.704 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [ 2024-08-19T14:19:09.704 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:09.704 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25057.0:193665 3.1a 3:5f229c26:::benchmark_data_smithi045_59973_object62486:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e300)", 2024-08-19T14:19:09.704 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:19:09.517205+0000", 2024-08-19T14:19:09.704 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.069149475000000002, 2024-08-19T14:19:09.704 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.069177695999999997, 2024-08-19T14:19:09.704 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:19:09.704 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:19:09.704 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "waiting for sub ops", 2024-08-19T14:19:09.704 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:19:09.705 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25057", 2024-08-19T14:19:09.705 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:19:09.705 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 193665 2024-08-19T14:19:09.705 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:09.705 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:19:09.705 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:09.705 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:19:09.705 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:09.517205+0000", 2024-08-19T14:19:09.705 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:09.705 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:09.705 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:09.705 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:19:09.705 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:09.517205+0000", 2024-08-19T14:19:09.705 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:09.706 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:09.706 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:09.706 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:19:09.706 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:09.517206+0000", 2024-08-19T14:19:09.706 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4440000000000001e-06 2024-08-19T14:19:09.706 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:09.706 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:09.706 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:19:09.706 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:09.517213+0000", 2024-08-19T14:19:09.706 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.8789999999999997e-06 2024-08-19T14:19:09.706 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:09.706 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:09.706 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:19:09.706 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:09.517226+0000", 2024-08-19T14:19:09.707 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2651e-05 2024-08-19T14:19:09.707 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:09.707 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:09.707 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:19:09.707 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:09.517235+0000", 2024-08-19T14:19:09.707 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.3149999999999998e-06 2024-08-19T14:19:09.707 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:09.707 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:09.707 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:19:09.707 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:09.517301+0000", 2024-08-19T14:19:09.707 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.5346999999999996e-05 2024-08-19T14:19:09.707 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:09.707 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:09.707 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:19:09.708 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:09.517449+0000", 2024-08-19T14:19:09.708 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014818399999999999 2024-08-19T14:19:09.708 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:09.708 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:09.708 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,9", 2024-08-19T14:19:09.708 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:09.517580+0000", 2024-08-19T14:19:09.708 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013100000000000001 2024-08-19T14:19:09.708 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:09.708 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:09.708 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:19:09.708 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:09.520835+0000", 2024-08-19T14:19:09.708 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0032557850000000002 2024-08-19T14:19:09.708 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:09.708 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:09.708 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:19:09.709 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:09.521266+0000", 2024-08-19T14:19:09.709 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00043092599999999999 2024-08-19T14:19:09.709 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:09.709 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:19:09.709 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:09.709 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:09.709 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-08-19T14:19:09.709 INFO:teuthology.orchestra.run.smithi045.stdout: "num_ops": 1 2024-08-19T14:19:09.709 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:19:09.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:09.720 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:09.721 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:19:09.822 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:19:09.868 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:09.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:09.869 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:19:09.970 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:19:10.018 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:10.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:10.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:19:10.120 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:19:10.152 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:10.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:10.153 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:19:10.254 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:19:10.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:10.282 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:10.282 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:19:10.383 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:19:10.407 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:10.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:10.408 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:19:10.495 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-08-19T14:19:10.509 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:19:10.535 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:10.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:10.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:19:10.637 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:19:10.695 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:10.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:10.696 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:19:10.737 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:19:10.739 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-08-19T14:19:10.798 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:19:10.841 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:19:10.856 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:10.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:10.858 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:19:10.959 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:19:10.985 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:10.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:10.986 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:19:11.087 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:19:11.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:11.115 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:11.116 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:19:11.217 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:19:11.245 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:11.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:11.246 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:19:11.347 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:19:11.375 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:11.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:11.377 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:19:11.478 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:19:11.481 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:11 smithi045 ceph-mon[20143]: pgmap v678: 77 pgs: 7 active+recovering+undersized+remapped, 2 remapped+peering, 1 unknown, 67 active+clean; 4.3 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 824 op/s; 18957/211299 objects misplaced (8.972%); 5.7 MiB/s, 92 objects/s recovering 2024-08-19T14:19:11.481 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:11 smithi045 ceph-mon[20143]: from='client.25313 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:19:11.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:11 smithi032 ceph-mon[20004]: pgmap v678: 77 pgs: 7 active+recovering+undersized+remapped, 2 remapped+peering, 1 unknown, 67 active+clean; 4.3 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 824 op/s; 18957/211299 objects misplaced (8.972%); 5.7 MiB/s, 92 objects/s recovering 2024-08-19T14:19:11.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:11 smithi032 ceph-mon[20004]: from='client.25313 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:19:11.517 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:11.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:11.518 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:19:11.522 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:19:11.522 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [], 2024-08-19T14:19:11.522 INFO:teuthology.orchestra.run.smithi045.stdout: "complaint_time": 30, 2024-08-19T14:19:11.523 INFO:teuthology.orchestra.run.smithi045.stdout: "num_blocked_ops": 0 2024-08-19T14:19:11.523 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:19:11.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:11 smithi031 ceph-mon[16336]: pgmap v678: 77 pgs: 7 active+recovering+undersized+remapped, 2 remapped+peering, 1 unknown, 67 active+clean; 4.3 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 824 op/s; 18957/211299 objects misplaced (8.972%); 5.7 MiB/s, 92 objects/s recovering 2024-08-19T14:19:11.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:11 smithi031 ceph-mon[16336]: from='client.25313 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:19:11.618 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:19:11.666 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:11.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:11.668 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:19:11.768 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:19:11.796 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:11.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:11.797 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:19:11.898 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:19:11.919 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:11.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:11.920 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:19:12.022 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:19:12.049 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:12.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:12.050 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:19:12.151 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:19:12.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:12.190 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:12.190 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:19:12.233 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-08-19T14:19:12.291 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:19:12.316 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:12.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:12.317 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:19:12.418 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:19:12.463 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:12.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:12.464 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:19:12.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:12 smithi045 ceph-mon[20143]: Health check failed: Degraded data redundancy: 4/206112 objects degraded (0.002%), 2 pgs degraded (PG_DEGRADED) 2024-08-19T14:19:12.565 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:19:12.591 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:12.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:12.592 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:19:12.612 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:19:12.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:12 smithi031 ceph-mon[16336]: Health check failed: Degraded data redundancy: 4/206112 objects degraded (0.002%), 2 pgs degraded (PG_DEGRADED) 2024-08-19T14:19:12.676 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":678,"stamp":"2024-08-19T14:19:09.850381+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4616290872,"num_objects":70433,"num_object_clones":0,"num_object_copies":211299,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":18957,"num_objects_unfound":0,"num_objects_dirty":70433,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":75234,"num_write_kb":927169,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":553,"num_bytes_recovered":36044800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":340024,"ondisk_log_size":340024,"up":231,"acting":222,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":235,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":16624356,"kb_used_data":13741936,"kb_used_omap":14,"kb_used_meta":2882161,"kb_avail":827082012,"statfs":{"total":863955320832,"available":846931980288,"internally_reserved":0,"allocated":14071742464,"data_stored":14059033991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14353,"internal_metadata":2951333871},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-324141056,"num_objects":-4946,"num_object_clones":0,"num_object_copies":-14838,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":18957,"num_objects_unfound":0,"num_objects_dirty":-4946,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4946,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":553,"num_bytes_recovered":36044800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001578"},"pg_stats":[{"pgid":"3.2d","version":"300'8120","reported_seq":8626,"reported_epoch":300,"state":"active+recovering+undersized+remapped","last_fresh":"2024-08-19T14:19:08.338570+0000","last_change":"2024-08-19T14:19:06.468072+0000","last_active":"2024-08-19T14:19:08.338570+0000","last_peered":"2024-08-19T14:19:08.338570+0000","last_clean":"2024-08-19T14:19:04.154984+0000","last_became_active":"2024-08-19T14:19:06.428698+0000","last_became_peered":"2024-08-19T14:19:06.428698+0000","last_unstale":"2024-08-19T14:19:08.338570+0000","last_undegraded":"2024-08-19T14:19:08.338570+0000","last_fullsized":"2024-08-19T14:19:06.185552+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":6,"last_scrub":"276'5846","last_scrub_stamp":"2024-08-19T14:18:26.752109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:26.752109+0000","objects_scrubbed":3860,"log_size":8120,"log_dups_size":0,"ondisk_log_size":8120,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"no scrub is scheduled","scrub_duration":5035,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71598080,"num_objects":1093,"num_object_clones":0,"num_object_copies":3279,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2080,"num_objects_unfound":0,"num_objects_dirty":1093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1177,"num_write_kb":10848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":5898240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,0],"acting":[9,0],"avail_no_missing":["9","5","7"],"object_location_counts":[{"shards":"0,9","objects":53},{"shards":"5,7,9","objects":1040}],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2c","version":"300'7924","reported_seq":8477,"reported_epoch":300,"state":"active+recovering+undersized+remapped","last_fresh":"2024-08-19T14:19:09.257683+0000","last_change":"2024-08-19T14:19:06.645126+0000","last_active":"2024-08-19T14:19:09.257683+0000","last_peered":"2024-08-19T14:19:09.257683+0000","last_clean":"2024-08-19T14:19:04.136972+0000","last_became_active":"2024-08-19T14:19:06.580223+0000","last_became_peered":"2024-08-19T14:19:06.580223+0000","last_unstale":"2024-08-19T14:19:09.257683+0000","last_undegraded":"2024-08-19T14:19:09.257683+0000","last_fullsized":"2024-08-19T14:19:06.184290+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":6,"last_scrub":"272'5481","last_scrub_stamp":"2024-08-19T14:18:18.065864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:18.065864+0000","objects_scrubbed":3568,"log_size":7924,"log_dups_size":0,"ondisk_log_size":7924,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":959,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70713344,"num_objects":1079,"num_object_clones":0,"num_object_copies":3237,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3072,"num_objects_unfound":0,"num_objects_dirty":1079,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1271,"num_write_kb":18112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":80,"num_bytes_recovered":5242880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[3,4],"avail_no_missing":["3","5","9"],"object_location_counts":[{"shards":"3,4","objects":55},{"shards":"3,5,9","objects":1024}],"blocked_by":[],"up_primary":11,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2b","version":"297'7992","reported_seq":8217,"reported_epoch":299,"state":"remapped+peering","last_fresh":"2024-08-19T14:19:05.184731+0000","last_change":"2024-08-19T14:19:05.184731+0000","last_active":"2024-08-19T14:19:04.176750+0000","last_peered":"2024-08-19T14:19:04.136830+0000","last_clean":"2024-08-19T14:19:04.136830+0000","last_became_active":"2024-08-19T14:19:03.191001+0000","last_became_peered":"2024-08-19T14:19:03.191001+0000","last_unstale":"2024-08-19T14:19:05.184731+0000","last_undegraded":"2024-08-19T14:19:05.184731+0000","last_fullsized":"2024-08-19T14:19:05.184731+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":6,"last_scrub":"271'5595","last_scrub_stamp":"2024-08-19T14:18:17.226507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:17.226507+0000","objects_scrubbed":3605,"log_size":7992,"log_dups_size":0,"ondisk_log_size":7992,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":888,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76021760,"num_objects":1160,"num_object_clones":0,"num_object_copies":3480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1160,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1232,"num_write_kb":18848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"300'7916","reported_seq":8357,"reported_epoch":300,"state":"active+recovering+undersized+remapped","last_fresh":"2024-08-19T14:19:08.496670+0000","last_change":"2024-08-19T14:19:06.557324+0000","last_active":"2024-08-19T14:19:08.496670+0000","last_peered":"2024-08-19T14:19:08.496670+0000","last_clean":"2024-08-19T14:19:04.151222+0000","last_became_active":"2024-08-19T14:19:06.446972+0000","last_became_peered":"2024-08-19T14:19:06.446972+0000","last_unstale":"2024-08-19T14:19:08.496670+0000","last_undegraded":"2024-08-19T14:19:08.496670+0000","last_fullsized":"2024-08-19T14:19:06.181070+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":6,"last_scrub":"273'5664","last_scrub_stamp":"2024-08-19T14:18:21.328957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:21.328957+0000","objects_scrubbed":3688,"log_size":7916,"log_dups_size":0,"ondisk_log_size":7916,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":405,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71434240,"num_objects":1090,"num_object_clones":0,"num_object_copies":3270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3195,"num_objects_unfound":0,"num_objects_dirty":1090,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1161,"num_write_kb":13024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":3473408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[1,3],"avail_no_missing":["1","4","8"],"object_location_counts":[{"shards":"1,3","objects":25},{"shards":"1,4,8","objects":1065}],"blocked_by":[],"up_primary":5,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"297'7345","reported_seq":7569,"reported_epoch":298,"state":"unknown","last_fresh":"2024-08-19T14:19:04.177683+0000","last_change":"2024-08-19T14:19:04.853689+0000","last_active":"2024-08-19T14:19:04.177683+0000","last_peered":"2024-08-19T14:19:04.169338+0000","last_clean":"2024-08-19T14:19:04.169338+0000","last_became_active":"2024-08-19T14:19:03.185781+0000","last_became_peered":"2024-08-19T14:19:03.185781+0000","last_unstale":"2024-08-19T14:19:04.177683+0000","last_undegraded":"2024-08-19T14:19:04.177683+0000","last_fullsized":"2024-08-19T14:19:04.177683+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":6,"last_scrub":"272'5067","last_scrub_stamp":"2024-08-19T14:18:18.201966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:18.201966+0000","objects_scrubbed":3534,"log_size":7345,"log_dups_size":0,"ondisk_log_size":7345,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":779,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71925760,"num_objects":1098,"num_object_clones":0,"num_object_copies":3294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1098,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1171,"num_write_kb":16032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,1],"acting":[8,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.28","version":"300'7657","reported_seq":8068,"reported_epoch":300,"state":"active+recovering+undersized+remapped","last_fresh":"2024-08-19T14:19:08.491769+0000","last_change":"2024-08-19T14:19:06.600736+0000","last_active":"2024-08-19T14:19:08.491769+0000","last_peered":"2024-08-19T14:19:08.491769+0000","last_clean":"2024-08-19T14:19:04.156917+0000","last_became_active":"2024-08-19T14:19:06.460361+0000","last_became_peered":"2024-08-19T14:19:06.460361+0000","last_unstale":"2024-08-19T14:19:08.491769+0000","last_undegraded":"2024-08-19T14:19:08.491769+0000","last_fullsized":"2024-08-19T14:19:06.181003+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":6,"last_scrub":"271'5161","last_scrub_stamp":"2024-08-19T14:18:17.262486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:17.262486+0000","objects_scrubbed":3649,"log_size":7657,"log_dups_size":0,"ondisk_log_size":7657,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":818,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70746112,"num_objects":1080,"num_object_clones":0,"num_object_copies":3240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3337,"num_objects_unfound":0,"num_objects_dirty":1080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1289,"num_write_kb":17408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":2818048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,8],"acting":[1,3],"avail_no_missing":["1","4","11"],"object_location_counts":[{"shards":"1,3","objects":-33},{"shards":"1,4,8,9,11","objects":1},{"shards":"1,4,11","objects":1112}],"blocked_by":[],"up_primary":9,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.27","version":"300'7399","reported_seq":7970,"reported_epoch":300,"state":"active+recovering+undersized+remapped","last_fresh":"2024-08-19T14:19:09.277408+0000","last_change":"2024-08-19T14:19:06.343220+0000","last_active":"2024-08-19T14:19:09.277408+0000","last_peered":"2024-08-19T14:19:09.277408+0000","last_clean":"2024-08-19T14:19:04.118605+0000","last_became_active":"2024-08-19T14:19:06.292909+0000","last_became_peered":"2024-08-19T14:19:06.292909+0000","last_unstale":"2024-08-19T14:19:09.277408+0000","last_undegraded":"2024-08-19T14:19:09.277408+0000","last_fullsized":"2024-08-19T14:19:06.185617+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":6,"last_scrub":"272'5087","last_scrub_stamp":"2024-08-19T14:18:18.152521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:18.152521+0000","objects_scrubbed":3609,"log_size":7399,"log_dups_size":0,"ondisk_log_size":7399,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":838,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":73367552,"num_objects":1120,"num_object_clones":0,"num_object_copies":3360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":934,"num_objects_unfound":0,"num_objects_dirty":1120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1191,"num_write_kb":16192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":11141120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,11],"acting":[3,11],"avail_no_missing":["3","7","11"],"object_location_counts":[{"shards":"3,7,11","objects":934},{"shards":"3,11","objects":186}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"297'7591","reported_seq":7817,"reported_epoch":299,"state":"remapped+peering","last_fresh":"2024-08-19T14:19:05.184574+0000","last_change":"2024-08-19T14:19:05.184574+0000","last_active":"2024-08-19T14:19:05.184574+0000","last_peered":"2024-08-19T14:19:04.159812+0000","last_clean":"2024-08-19T14:19:04.159812+0000","last_became_active":"2024-08-19T14:19:03.191969+0000","last_became_peered":"2024-08-19T14:19:03.191969+0000","last_unstale":"2024-08-19T14:19:05.184574+0000","last_undegraded":"2024-08-19T14:19:05.184574+0000","last_fullsized":"2024-08-19T14:19:05.184574+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":6,"last_scrub":"273'5366","last_scrub_stamp":"2024-08-19T14:18:21.462335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:21.462335+0000","objects_scrubbed":3843,"log_size":7591,"log_dups_size":0,"ondisk_log_size":7591,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":461,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":74219520,"num_objects":1133,"num_object_clones":0,"num_object_copies":3399,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1131,"num_write_kb":10944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.25","version":"300'7486","reported_seq":8045,"reported_epoch":300,"state":"active+recovering+undersized+remapped","last_fresh":"2024-08-19T14:19:09.281272+0000","last_change":"2024-08-19T14:19:06.556777+0000","last_active":"2024-08-19T14:19:09.281272+0000","last_peered":"2024-08-19T14:19:09.281272+0000","last_clean":"2024-08-19T14:19:04.133634+0000","last_became_active":"2024-08-19T14:19:06.447949+0000","last_became_peered":"2024-08-19T14:19:06.447949+0000","last_unstale":"2024-08-19T14:19:09.281272+0000","last_undegraded":"2024-08-19T14:19:09.281272+0000","last_fullsized":"2024-08-19T14:19:06.185780+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":6,"last_scrub":"271'5041","last_scrub_stamp":"2024-08-19T14:18:16.976188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:16.976188+0000","objects_scrubbed":3541,"log_size":7486,"log_dups_size":0,"ondisk_log_size":7486,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":886,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71761920,"num_objects":1095,"num_object_clones":0,"num_object_copies":3285,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3130,"num_objects_unfound":0,"num_objects_dirty":1095,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1279,"num_write_kb":19296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":5242880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,9],"acting":[3,8],"avail_no_missing":["3","5","7"],"object_location_counts":[{"shards":"3,5,7","objects":1043},{"shards":"3,5,7,8,9","objects":1},{"shards":"3,8","objects":51}],"blocked_by":[],"up_primary":11,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.24","version":"300'7478","reported_seq":7855,"reported_epoch":300,"state":"active+recovering+undersized+remapped","last_fresh":"2024-08-19T14:19:08.488745+0000","last_change":"2024-08-19T14:19:06.721560+0000","last_active":"2024-08-19T14:19:08.488745+0000","last_peered":"2024-08-19T14:19:08.488745+0000","last_clean":"2024-08-19T14:19:04.175568+0000","last_became_active":"2024-08-19T14:19:06.607711+0000","last_became_peered":"2024-08-19T14:19:06.607711+0000","last_unstale":"2024-08-19T14:19:08.488745+0000","last_undegraded":"2024-08-19T14:19:08.488745+0000","last_fullsized":"2024-08-19T14:19:06.181396+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":6,"last_scrub":"273'5234","last_scrub_stamp":"2024-08-19T14:18:20.377712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:20.377712+0000","objects_scrubbed":3737,"log_size":7478,"log_dups_size":0,"ondisk_log_size":7478,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":701,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72810496,"num_objects":1111,"num_object_clones":0,"num_object_copies":3333,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3209,"num_objects_unfound":0,"num_objects_dirty":1111,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1156,"num_write_kb":12736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":2228224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[1,0],"avail_no_missing":["1","5","11"],"object_location_counts":[{"shards":"0,1","objects":41},{"shards":"1,4,5,11","objects":1},{"shards":"1,5,11","objects":1069}],"blocked_by":[],"up_primary":3,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"70'139","reported_seq":807,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-19T14:19:06.196898+0000","last_change":"2024-08-19T14:18:25.872921+0000","last_active":"2024-08-19T14:19:06.196898+0000","last_peered":"2024-08-19T14:19:06.196898+0000","last_clean":"2024-08-19T14:19:06.196898+0000","last_became_active":"2024-08-19T14:18:13.807133+0000","last_became_peered":"2024-08-19T14:18:13.807133+0000","last_unstale":"2024-08-19T14:19:06.196898+0000","last_undegraded":"2024-08-19T14:19:06.196898+0000","last_fullsized":"2024-08-19T14:19:06.196898+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:18:25.872808+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:18:25.872808+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:33.201222+0000","scrub_duration":2157,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,9],"acting":[7,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"300'5495","reported_seq":5708,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-19T14:19:08.060327+0000","last_change":"2024-08-19T14:18:17.084657+0000","last_active":"2024-08-19T14:19:08.060327+0000","last_peered":"2024-08-19T14:19:08.060327+0000","last_clean":"2024-08-19T14:19:08.060327+0000","last_became_active":"2024-08-19T14:17:34.968554+0000","last_became_peered":"2024-08-19T14:17:34.968554+0000","last_unstale":"2024-08-19T14:19:08.060327+0000","last_undegraded":"2024-08-19T14:19:08.060327+0000","last_fullsized":"2024-08-19T14:19:08.060327+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'4295","last_scrub_stamp":"2024-08-19T14:18:17.084447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:17.084447+0000","objects_scrubbed":1731,"log_size":5495,"log_dups_size":0,"ondisk_log_size":5495,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:31.192054+0000","scrub_duration":357,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70909952,"num_objects":1082,"num_object_clones":0,"num_object_copies":3246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1082,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1205,"num_write_kb":17792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,5],"acting":[9,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.3","version":"70'139","reported_seq":705,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-19T14:19:06.181499+0000","last_change":"2024-08-19T14:18:21.708466+0000","last_active":"2024-08-19T14:19:06.181499+0000","last_peered":"2024-08-19T14:19:06.181499+0000","last_clean":"2024-08-19T14:19:06.181499+0000","last_became_active":"2024-08-19T14:18:13.804721+0000","last_became_peered":"2024-08-19T14:18:13.804721+0000","last_unstale":"2024-08-19T14:19:06.181499+0000","last_undegraded":"2024-08-19T14:19:06.181499+0000","last_fullsized":"2024-08-19T14:19:06.181499+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:18:21.708333+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:18:21.708333+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:45.972923+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"297'5332","reported_seq":5534,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-08-19T14:19:05.185234+0000","last_change":"2024-08-19T14:18:20.248300+0000","last_active":"2024-08-19T14:19:05.185234+0000","last_peered":"2024-08-19T14:19:05.185234+0000","last_clean":"2024-08-19T14:19:05.185234+0000","last_became_active":"2024-08-19T14:17:34.963892+0000","last_became_peered":"2024-08-19T14:17:34.963892+0000","last_unstale":"2024-08-19T14:19:05.185234+0000","last_undegraded":"2024-08-19T14:19:05.185234+0000","last_fullsized":"2024-08-19T14:19:05.185234+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'4227","last_scrub_stamp":"2024-08-19T14:18:20.248075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:20.248075+0000","objects_scrubbed":1786,"log_size":5332,"log_dups_size":0,"ondisk_log_size":5332,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:30.227197+0000","scrub_duration":285,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67174400,"num_objects":1025,"num_object_clones":0,"num_object_copies":3075,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1025,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1107,"num_write_kb":11072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,11],"acting":[0,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"70'139","reported_seq":648,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-08-19T14:19:05.184598+0000","last_change":"2024-08-19T14:18:24.001903+0000","last_active":"2024-08-19T14:19:05.184598+0000","last_peered":"2024-08-19T14:19:05.184598+0000","last_clean":"2024-08-19T14:19:05.184598+0000","last_became_active":"2024-08-19T14:18:13.809344+0000","last_became_peered":"2024-08-19T14:18:13.809344+0000","last_unstale":"2024-08-19T14:19:05.184598+0000","last_undegraded":"2024-08-19T14:19:05.184598+0000","last_fullsized":"2024-08-19T14:19:05.184598+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:18:24.001801+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:18:24.001801+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:26.754447+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,8],"acting":[0,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"297'5381","reported_seq":5592,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-08-19T14:19:05.179473+0000","last_change":"2024-08-19T14:18:17.845313+0000","last_active":"2024-08-19T14:19:05.179473+0000","last_peered":"2024-08-19T14:19:05.179473+0000","last_clean":"2024-08-19T14:19:05.179473+0000","last_became_active":"2024-08-19T14:17:34.963390+0000","last_became_peered":"2024-08-19T14:17:34.963390+0000","last_unstale":"2024-08-19T14:19:05.179473+0000","last_undegraded":"2024-08-19T14:19:05.179473+0000","last_fullsized":"2024-08-19T14:19:05.179473+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'4224","last_scrub_stamp":"2024-08-19T14:18:17.845089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:17.845089+0000","objects_scrubbed":1755,"log_size":5381,"log_dups_size":0,"ondisk_log_size":5381,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:27.065936+0000","scrub_duration":426,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":74711040,"num_objects":1140,"num_object_clones":0,"num_object_copies":3420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1140,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1165,"num_write_kb":17600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,1],"acting":[11,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1","version":"70'139","reported_seq":702,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-19T14:19:06.185280+0000","last_change":"2024-08-19T14:18:21.795775+0000","last_active":"2024-08-19T14:19:06.185280+0000","last_peered":"2024-08-19T14:19:06.185280+0000","last_clean":"2024-08-19T14:19:06.185280+0000","last_became_active":"2024-08-19T14:18:13.808505+0000","last_became_peered":"2024-08-19T14:18:13.808505+0000","last_unstale":"2024-08-19T14:19:06.185280+0000","last_undegraded":"2024-08-19T14:19:06.185280+0000","last_fullsized":"2024-08-19T14:19:06.185280+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:18:21.795647+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:18:21.795647+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:42.370696+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,11],"acting":[9,0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3","version":"300'5462","reported_seq":5675,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-19T14:19:07.935333+0000","last_change":"2024-08-19T14:18:20.238483+0000","last_active":"2024-08-19T14:19:07.935333+0000","last_peered":"2024-08-19T14:19:07.935333+0000","last_clean":"2024-08-19T14:19:07.935333+0000","last_became_active":"2024-08-19T14:17:34.968430+0000","last_became_peered":"2024-08-19T14:17:34.968430+0000","last_unstale":"2024-08-19T14:19:07.935333+0000","last_undegraded":"2024-08-19T14:19:07.935333+0000","last_fullsized":"2024-08-19T14:19:07.935333+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'4292","last_scrub_stamp":"2024-08-19T14:18:20.238336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:20.238336+0000","objects_scrubbed":1842,"log_size":5462,"log_dups_size":0,"ondisk_log_size":5462,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:30.768685+0000","scrub_duration":304,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69730304,"num_objects":1064,"num_object_clones":0,"num_object_copies":3192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1064,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1170,"num_write_kb":12544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,11],"acting":[4,0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"70'139","reported_seq":623,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-19T14:19:06.264045+0000","last_change":"2024-08-19T14:18:27.204083+0000","last_active":"2024-08-19T14:19:06.264045+0000","last_peered":"2024-08-19T14:19:06.264045+0000","last_clean":"2024-08-19T14:19:06.264045+0000","last_became_active":"2024-08-19T14:18:26.878753+0000","last_became_peered":"2024-08-19T14:18:26.878753+0000","last_unstale":"2024-08-19T14:19:06.264045+0000","last_undegraded":"2024-08-19T14:19:06.264045+0000","last_fullsized":"2024-08-19T14:19:06.264045+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:18:27.203987+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:18:27.203987+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:56.868334+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"300'7449","reported_seq":7681,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-19T14:19:06.531629+0000","last_change":"2024-08-19T14:19:03.204621+0000","last_active":"2024-08-19T14:19:06.531629+0000","last_peered":"2024-08-19T14:19:06.531629+0000","last_clean":"2024-08-19T14:19:06.531629+0000","last_became_active":"2024-08-19T14:19:03.196276+0000","last_became_peered":"2024-08-19T14:19:03.196276+0000","last_unstale":"2024-08-19T14:19:06.531629+0000","last_undegraded":"2024-08-19T14:19:06.531629+0000","last_fullsized":"2024-08-19T14:19:06.531629+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'5041","last_scrub_stamp":"2024-08-19T14:18:16.976188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:16.976188+0000","objects_scrubbed":3541,"log_size":7449,"log_dups_size":0,"ondisk_log_size":7449,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:10.097830+0000","scrub_duration":886,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":74186752,"num_objects":1132,"num_object_clones":0,"num_object_copies":3396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1132,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1241,"num_write_kb":19296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"70'139","reported_seq":706,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-19T14:19:06.186047+0000","last_change":"2024-08-19T14:18:27.848950+0000","last_active":"2024-08-19T14:19:06.186047+0000","last_peered":"2024-08-19T14:19:06.186047+0000","last_clean":"2024-08-19T14:19:06.186047+0000","last_became_active":"2024-08-19T14:18:26.874902+0000","last_became_peered":"2024-08-19T14:18:26.874902+0000","last_unstale":"2024-08-19T14:19:06.186047+0000","last_undegraded":"2024-08-19T14:19:06.186047+0000","last_fullsized":"2024-08-19T14:19:06.186047+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:18:27.848870+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:27.848870+0000","last_clean_scrub_stamp":"2024-08-19T14:18:27.848870+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:29.542244+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,4],"acting":[9,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.4","version":"300'7466","reported_seq":7699,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-19T14:19:08.359811+0000","last_change":"2024-08-19T14:19:03.187955+0000","last_active":"2024-08-19T14:19:08.359811+0000","last_peered":"2024-08-19T14:19:08.359811+0000","last_clean":"2024-08-19T14:19:08.359811+0000","last_became_active":"2024-08-19T14:19:03.174624+0000","last_became_peered":"2024-08-19T14:19:03.174624+0000","last_unstale":"2024-08-19T14:19:08.359811+0000","last_undegraded":"2024-08-19T14:19:08.359811+0000","last_fullsized":"2024-08-19T14:19:08.359811+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'5234","last_scrub_stamp":"2024-08-19T14:18:20.377712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:20.377712+0000","objects_scrubbed":3737,"log_size":7466,"log_dups_size":0,"ondisk_log_size":7466,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:09.626578+0000","scrub_duration":701,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":73596928,"num_objects":1123,"num_object_clones":0,"num_object_copies":3369,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1143,"num_write_kb":12736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,5],"acting":[1,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"70'139","reported_seq":518,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-08-19T14:19:05.179782+0000","last_change":"2024-08-19T14:18:28.501538+0000","last_active":"2024-08-19T14:19:05.179782+0000","last_peered":"2024-08-19T14:19:05.179782+0000","last_clean":"2024-08-19T14:19:05.179782+0000","last_became_active":"2024-08-19T14:18:26.881435+0000","last_became_peered":"2024-08-19T14:18:26.881435+0000","last_unstale":"2024-08-19T14:19:05.179782+0000","last_undegraded":"2024-08-19T14:19:05.179782+0000","last_fullsized":"2024-08-19T14:19:05.179782+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:18:28.501449+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:18:28.501449+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:55.948489+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,4],"acting":[11,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.7","version":"300'7426","reported_seq":7655,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-19T14:19:09.235924+0000","last_change":"2024-08-19T14:19:03.180110+0000","last_active":"2024-08-19T14:19:09.235924+0000","last_peered":"2024-08-19T14:19:09.235924+0000","last_clean":"2024-08-19T14:19:09.235924+0000","last_became_active":"2024-08-19T14:19:03.171209+0000","last_became_peered":"2024-08-19T14:19:03.171209+0000","last_unstale":"2024-08-19T14:19:09.235924+0000","last_undegraded":"2024-08-19T14:19:09.235924+0000","last_fullsized":"2024-08-19T14:19:09.235924+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'5087","last_scrub_stamp":"2024-08-19T14:18:18.152521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:18.152521+0000","objects_scrubbed":3609,"log_size":7426,"log_dups_size":0,"ondisk_log_size":7426,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:09.305793+0000","scrub_duration":838,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71598080,"num_objects":1092,"num_object_clones":0,"num_object_copies":3276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1092,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1218,"num_write_kb":16192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"70'139","reported_seq":742,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-08-19T14:19:05.180620+0000","last_change":"2024-08-19T14:18:21.476428+0000","last_active":"2024-08-19T14:19:05.180620+0000","last_peered":"2024-08-19T14:19:05.180620+0000","last_clean":"2024-08-19T14:19:05.180620+0000","last_became_active":"2024-08-19T14:18:13.806013+0000","last_became_peered":"2024-08-19T14:18:13.806013+0000","last_unstale":"2024-08-19T14:19:05.180620+0000","last_undegraded":"2024-08-19T14:19:05.180620+0000","last_fullsized":"2024-08-19T14:19:05.180620+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:18:21.476313+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:18:21.476313+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:29.834715+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.6","version":"298'7592","reported_seq":7823,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-08-19T14:19:05.185102+0000","last_change":"2024-08-19T14:19:03.178823+0000","last_active":"2024-08-19T14:19:05.185102+0000","last_peered":"2024-08-19T14:19:05.185102+0000","last_clean":"2024-08-19T14:19:05.185102+0000","last_became_active":"2024-08-19T14:19:03.165923+0000","last_became_peered":"2024-08-19T14:19:03.165923+0000","last_unstale":"2024-08-19T14:19:05.185102+0000","last_undegraded":"2024-08-19T14:19:05.185102+0000","last_fullsized":"2024-08-19T14:19:05.185102+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'5366","last_scrub_stamp":"2024-08-19T14:18:21.462335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:21.462335+0000","objects_scrubbed":3843,"log_size":7592,"log_dups_size":0,"ondisk_log_size":7592,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:10.020833+0000","scrub_duration":461,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":74153984,"num_objects":1131,"num_object_clones":0,"num_object_copies":3393,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1132,"num_write_kb":10944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.a","version":"70'139","reported_seq":714,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-19T14:19:06.185214+0000","last_change":"2024-08-19T14:18:28.865004+0000","last_active":"2024-08-19T14:19:06.185214+0000","last_peered":"2024-08-19T14:19:06.185214+0000","last_clean":"2024-08-19T14:19:06.185214+0000","last_became_active":"2024-08-19T14:18:26.875006+0000","last_became_peered":"2024-08-19T14:18:26.875006+0000","last_unstale":"2024-08-19T14:19:06.185214+0000","last_undegraded":"2024-08-19T14:19:06.185214+0000","last_fullsized":"2024-08-19T14:19:06.185214+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:18:28.864870+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:28.864870+0000","last_clean_scrub_stamp":"2024-08-19T14:18:28.864870+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:43.628160+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,0],"acting":[9,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.8","version":"297'7617","reported_seq":7847,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-08-19T14:19:05.179405+0000","last_change":"2024-08-19T14:19:03.209681+0000","last_active":"2024-08-19T14:19:05.179405+0000","last_peered":"2024-08-19T14:19:05.179405+0000","last_clean":"2024-08-19T14:19:05.179405+0000","last_became_active":"2024-08-19T14:19:03.201496+0000","last_became_peered":"2024-08-19T14:19:03.201496+0000","last_unstale":"2024-08-19T14:19:05.179405+0000","last_undegraded":"2024-08-19T14:19:05.179405+0000","last_fullsized":"2024-08-19T14:19:05.179405+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'5161","last_scrub_stamp":"2024-08-19T14:18:17.262486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:17.262486+0000","objects_scrubbed":3649,"log_size":7617,"log_dups_size":0,"ondisk_log_size":7617,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:09.678841+0000","scrub_duration":818,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":73367552,"num_objects":1119,"num_object_clones":0,"num_object_copies":3357,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1249,"num_write_kb":17408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,4],"acting":[11,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.9","version":"70'139","reported_seq":829,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-19T14:19:06.200749+0000","last_change":"2024-08-19T14:18:28.744550+0000","last_active":"2024-08-19T14:19:06.200749+0000","last_peered":"2024-08-19T14:19:06.200749+0000","last_clean":"2024-08-19T14:19:06.200749+0000","last_became_active":"2024-08-19T14:18:26.887657+0000","last_became_peered":"2024-08-19T14:18:26.887657+0000","last_unstale":"2024-08-19T14:19:06.200749+0000","last_undegraded":"2024-08-19T14:19:06.200749+0000","last_fullsized":"2024-08-19T14:19:06.200749+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:18:28.744320+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:18:28.744320+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:30.737843+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"300'8049","reported_seq":8280,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-19T14:19:09.142888+0000","last_change":"2024-08-19T14:19:03.193085+0000","last_active":"2024-08-19T14:19:09.142888+0000","last_peered":"2024-08-19T14:19:09.142888+0000","last_clean":"2024-08-19T14:19:09.142888+0000","last_became_active":"2024-08-19T14:19:03.182979+0000","last_became_peered":"2024-08-19T14:19:03.182979+0000","last_unstale":"2024-08-19T14:19:09.142888+0000","last_undegraded":"2024-08-19T14:19:09.142888+0000","last_fullsized":"2024-08-19T14:19:09.142888+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'5595","last_scrub_stamp":"2024-08-19T14:18:17.226507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:17.226507+0000","objects_scrubbed":3605,"log_size":8049,"log_dups_size":0,"ondisk_log_size":8049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:09.305828+0000","scrub_duration":888,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72286208,"num_objects":1103,"num_object_clones":0,"num_object_copies":3309,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1288,"num_write_kb":18848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"70'139","reported_seq":718,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-19T14:19:06.263501+0000","last_change":"2024-08-19T14:18:28.190126+0000","last_active":"2024-08-19T14:19:06.263501+0000","last_peered":"2024-08-19T14:19:06.263501+0000","last_clean":"2024-08-19T14:19:06.263501+0000","last_became_active":"2024-08-19T14:18:26.887277+0000","last_became_peered":"2024-08-19T14:18:26.887277+0000","last_unstale":"2024-08-19T14:19:06.263501+0000","last_undegraded":"2024-08-19T14:19:06.263501+0000","last_fullsized":"2024-08-19T14:19:06.263501+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:18:28.190009+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:18:28.190009+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:57.986532+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,8],"acting":[5,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"300'7892","reported_seq":8120,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-19T14:19:08.075439+0000","last_change":"2024-08-19T14:19:03.193011+0000","last_active":"2024-08-19T14:19:08.075439+0000","last_peered":"2024-08-19T14:19:08.075439+0000","last_clean":"2024-08-19T14:19:08.075439+0000","last_became_active":"2024-08-19T14:19:03.181991+0000","last_became_peered":"2024-08-19T14:19:03.181991+0000","last_unstale":"2024-08-19T14:19:08.075439+0000","last_undegraded":"2024-08-19T14:19:08.075439+0000","last_fullsized":"2024-08-19T14:19:08.075439+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'5664","last_scrub_stamp":"2024-08-19T14:18:21.328957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:21.328957+0000","objects_scrubbed":3688,"log_size":7892,"log_dups_size":0,"ondisk_log_size":7892,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:09.845185+0000","scrub_duration":405,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":73007104,"num_objects":1114,"num_object_clones":0,"num_object_copies":3342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1114,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1137,"num_write_kb":13024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,8],"acting":[4,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"298'7339","reported_seq":7565,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-08-19T14:19:04.178595+0000","last_change":"2024-08-19T14:19:03.185633+0000","last_active":"2024-08-19T14:19:04.178595+0000","last_peered":"2024-08-19T14:19:04.178595+0000","last_clean":"2024-08-19T14:19:04.178595+0000","last_became_active":"2024-08-19T14:19:03.177372+0000","last_became_peered":"2024-08-19T14:19:03.177372+0000","last_unstale":"2024-08-19T14:19:04.178595+0000","last_undegraded":"2024-08-19T14:19:04.178595+0000","last_fullsized":"2024-08-19T14:19:04.178595+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'5067","last_scrub_stamp":"2024-08-19T14:18:18.201966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:18.201966+0000","objects_scrubbed":3534,"log_size":7339,"log_dups_size":0,"ondisk_log_size":7339,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":779,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72318976,"num_objects":1103,"num_object_clones":0,"num_object_copies":3309,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1164,"num_write_kb":16032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.b","version":"70'139","reported_seq":718,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-08-19T14:19:05.179424+0000","last_change":"2024-08-19T14:18:27.506148+0000","last_active":"2024-08-19T14:19:05.179424+0000","last_peered":"2024-08-19T14:19:05.179424+0000","last_clean":"2024-08-19T14:19:05.179424+0000","last_became_active":"2024-08-19T14:18:26.875179+0000","last_became_peered":"2024-08-19T14:18:26.875179+0000","last_unstale":"2024-08-19T14:19:05.179424+0000","last_undegraded":"2024-08-19T14:19:05.179424+0000","last_fullsized":"2024-08-19T14:19:05.179424+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:18:27.506045+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:18:27.506045+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:43.546458+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.c","version":"297'7889","reported_seq":8120,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-19T14:19:06.264053+0000","last_change":"2024-08-19T14:19:03.198767+0000","last_active":"2024-08-19T14:19:06.264053+0000","last_peered":"2024-08-19T14:19:06.264053+0000","last_clean":"2024-08-19T14:19:06.264053+0000","last_became_active":"2024-08-19T14:19:03.185720+0000","last_became_peered":"2024-08-19T14:19:03.185720+0000","last_unstale":"2024-08-19T14:19:06.264053+0000","last_undegraded":"2024-08-19T14:19:06.264053+0000","last_fullsized":"2024-08-19T14:19:06.264053+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'5481","last_scrub_stamp":"2024-08-19T14:18:18.065864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:18.065864+0000","objects_scrubbed":3568,"log_size":7889,"log_dups_size":0,"ondisk_log_size":7889,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:10.097810+0000","scrub_duration":959,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":73007104,"num_objects":1114,"num_object_clones":0,"num_object_copies":3342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1114,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1236,"num_write_kb":18112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,9],"acting":[5,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"70'139","reported_seq":374,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-19T14:19:06.181146+0000","last_change":"2024-08-19T14:18:27.900166+0000","last_active":"2024-08-19T14:19:06.181146+0000","last_peered":"2024-08-19T14:19:06.181146+0000","last_clean":"2024-08-19T14:19:06.181146+0000","last_became_active":"2024-08-19T14:18:26.884105+0000","last_became_peered":"2024-08-19T14:18:26.884105+0000","last_unstale":"2024-08-19T14:19:06.181146+0000","last_undegraded":"2024-08-19T14:19:06.181146+0000","last_fullsized":"2024-08-19T14:19:06.181146+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:18:27.899965+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:18:27.899965+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:56.474954+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"300'8098","reported_seq":8331,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-19T14:19:08.076002+0000","last_change":"2024-08-19T14:19:03.192366+0000","last_active":"2024-08-19T14:19:08.076002+0000","last_peered":"2024-08-19T14:19:08.076002+0000","last_clean":"2024-08-19T14:19:08.076002+0000","last_became_active":"2024-08-19T14:19:03.174108+0000","last_became_peered":"2024-08-19T14:19:03.174108+0000","last_unstale":"2024-08-19T14:19:08.076002+0000","last_undegraded":"2024-08-19T14:19:08.076002+0000","last_fullsized":"2024-08-19T14:19:08.076002+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'5846","last_scrub_stamp":"2024-08-19T14:18:26.752109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:26.752109+0000","objects_scrubbed":3860,"log_size":8098,"log_dups_size":0,"ondisk_log_size":8098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:09.914375+0000","scrub_duration":5035,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":73039872,"num_objects":1114,"num_object_clones":0,"num_object_copies":3342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1114,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1154,"num_write_kb":10848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,9],"acting":[7,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"70'139","reported_seq":618,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-19T14:19:06.263489+0000","last_change":"2024-08-19T14:18:23.173343+0000","last_active":"2024-08-19T14:19:06.263489+0000","last_peered":"2024-08-19T14:19:06.263489+0000","last_clean":"2024-08-19T14:19:06.263489+0000","last_became_active":"2024-08-19T14:18:18.711708+0000","last_became_peered":"2024-08-19T14:18:18.711708+0000","last_unstale":"2024-08-19T14:19:06.263489+0000","last_undegraded":"2024-08-19T14:19:06.263489+0000","last_fullsized":"2024-08-19T14:19:06.263489+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:18:23.173268+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:18:23.173268+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:51.974300+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"298'7961","reported_seq":8183,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-08-19T14:19:05.179976+0000","last_change":"2024-08-19T14:18:19.171319+0000","last_active":"2024-08-19T14:19:05.179976+0000","last_peered":"2024-08-19T14:19:05.179976+0000","last_clean":"2024-08-19T14:19:05.179976+0000","last_became_active":"2024-08-19T14:17:34.971738+0000","last_became_peered":"2024-08-19T14:17:34.971738+0000","last_unstale":"2024-08-19T14:19:05.179976+0000","last_undegraded":"2024-08-19T14:19:05.179976+0000","last_fullsized":"2024-08-19T14:19:05.179976+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'5670","last_scrub_stamp":"2024-08-19T14:18:19.171123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:19.171123+0000","objects_scrubbed":3656,"log_size":7961,"log_dups_size":0,"ondisk_log_size":7961,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:38.377950+0000","scrub_duration":758,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":149356544,"num_objects":2279,"num_object_clones":0,"num_object_copies":6837,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2279,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2317,"num_write_kb":30080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,0],"acting":[11,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.c","version":"70'139","reported_seq":479,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-19T14:19:06.185636+0000","last_change":"2024-08-19T14:18:28.348190+0000","last_active":"2024-08-19T14:19:06.185636+0000","last_peered":"2024-08-19T14:19:06.185636+0000","last_clean":"2024-08-19T14:19:06.185636+0000","last_became_active":"2024-08-19T14:18:26.872942+0000","last_became_peered":"2024-08-19T14:18:26.872942+0000","last_unstale":"2024-08-19T14:19:06.185636+0000","last_undegraded":"2024-08-19T14:19:06.185636+0000","last_fullsized":"2024-08-19T14:19:06.185636+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:18:28.348087+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:18:28.348087+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:39.796688+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,9],"acting":[3,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.f","version":"300'8007","reported_seq":8232,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-19T14:19:08.213419+0000","last_change":"2024-08-19T14:18:23.090893+0000","last_active":"2024-08-19T14:19:08.213419+0000","last_peered":"2024-08-19T14:19:08.213419+0000","last_clean":"2024-08-19T14:19:08.213419+0000","last_became_active":"2024-08-19T14:17:34.954276+0000","last_became_peered":"2024-08-19T14:17:34.954276+0000","last_unstale":"2024-08-19T14:19:08.213419+0000","last_undegraded":"2024-08-19T14:19:08.213419+0000","last_fullsized":"2024-08-19T14:19:08.213419+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'5708","last_scrub_stamp":"2024-08-19T14:18:23.090770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:23.090770+0000","objects_scrubbed":3694,"log_size":8007,"log_dups_size":0,"ondisk_log_size":8007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:35.516099+0000","scrub_duration":369,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":143458304,"num_objects":2189,"num_object_clones":0,"num_object_copies":6567,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2299,"num_write_kb":25408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"70'139","reported_seq":394,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-19T14:19:06.197214+0000","last_change":"2024-08-19T14:18:27.752642+0000","last_active":"2024-08-19T14:19:06.197214+0000","last_peered":"2024-08-19T14:19:06.197214+0000","last_clean":"2024-08-19T14:19:06.197214+0000","last_became_active":"2024-08-19T14:18:26.882002+0000","last_became_peered":"2024-08-19T14:18:26.882002+0000","last_unstale":"2024-08-19T14:19:06.197214+0000","last_undegraded":"2024-08-19T14:19:06.197214+0000","last_fullsized":"2024-08-19T14:19:06.197214+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:18:27.751964+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:18:27.751964+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:48.211625+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"300'7466","reported_seq":12355,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-19T14:19:08.289372+0000","last_change":"2024-08-19T14:18:20.355375+0000","last_active":"2024-08-19T14:19:08.289372+0000","last_peered":"2024-08-19T14:19:08.289372+0000","last_clean":"2024-08-19T14:19:08.289372+0000","last_became_active":"2024-08-19T14:17:38.684742+0000","last_became_peered":"2024-08-19T14:17:38.684742+0000","last_unstale":"2024-08-19T14:19:08.289372+0000","last_undegraded":"2024-08-19T14:19:08.289372+0000","last_fullsized":"2024-08-19T14:19:08.289372+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":5,"last_scrub":"273'5152","last_scrub_stamp":"2024-08-19T14:18:20.355288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:20.355288+0000","objects_scrubbed":3752,"log_size":7466,"log_dups_size":0,"ondisk_log_size":7466,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:45.269893+0000","scrub_duration":598,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":144048128,"num_objects":2198,"num_object_clones":0,"num_object_copies":6594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2318,"num_write_kb":24448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,3],"acting":[9,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.12","version":"70'139","reported_seq":653,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-19T14:19:06.331194+0000","last_change":"2024-08-19T14:18:20.109582+0000","last_active":"2024-08-19T14:19:06.331194+0000","last_peered":"2024-08-19T14:19:06.331194+0000","last_clean":"2024-08-19T14:19:06.331194+0000","last_became_active":"2024-08-19T14:18:18.703594+0000","last_became_peered":"2024-08-19T14:18:18.703594+0000","last_unstale":"2024-08-19T14:19:06.331194+0000","last_undegraded":"2024-08-19T14:19:06.331194+0000","last_fullsized":"2024-08-19T14:19:06.331194+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:18:20.109467+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:18:20.109467+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:21.664555+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"300'7374","reported_seq":10702,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-19T14:19:08.032171+0000","last_change":"2024-08-19T14:18:18.845192+0000","last_active":"2024-08-19T14:19:08.032171+0000","last_peered":"2024-08-19T14:19:08.032171+0000","last_clean":"2024-08-19T14:19:08.032171+0000","last_became_active":"2024-08-19T14:17:38.695549+0000","last_became_peered":"2024-08-19T14:17:38.695549+0000","last_unstale":"2024-08-19T14:19:08.032171+0000","last_undegraded":"2024-08-19T14:19:08.032171+0000","last_fullsized":"2024-08-19T14:19:08.032171+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":5,"last_scrub":"273'5072","last_scrub_stamp":"2024-08-19T14:18:18.844791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:18.844791+0000","objects_scrubbed":3594,"log_size":7374,"log_dups_size":0,"ondisk_log_size":7374,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:47.619362+0000","scrub_duration":924,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":139919360,"num_objects":2135,"num_object_clones":0,"num_object_copies":6405,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2321,"num_write_kb":27584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.13","version":"70'139","reported_seq":713,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-19T14:19:06.181295+0000","last_change":"2024-08-19T14:18:22.659716+0000","last_active":"2024-08-19T14:19:06.181295+0000","last_peered":"2024-08-19T14:19:06.181295+0000","last_clean":"2024-08-19T14:19:06.181295+0000","last_became_active":"2024-08-19T14:18:18.698334+0000","last_became_peered":"2024-08-19T14:18:18.698334+0000","last_unstale":"2024-08-19T14:19:06.181295+0000","last_undegraded":"2024-08-19T14:19:06.181295+0000","last_fullsized":"2024-08-19T14:19:06.181295+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:18:22.659638+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:18:22.659638+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:50.367468+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.12","version":"298'7489","reported_seq":14075,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-08-19T14:19:05.184416+0000","last_change":"2024-08-19T14:18:23.349819+0000","last_active":"2024-08-19T14:19:05.184416+0000","last_peered":"2024-08-19T14:19:05.184416+0000","last_clean":"2024-08-19T14:19:05.184416+0000","last_became_active":"2024-08-19T14:17:47.703601+0000","last_became_peered":"2024-08-19T14:17:47.703601+0000","last_unstale":"2024-08-19T14:19:05.184416+0000","last_undegraded":"2024-08-19T14:19:05.184416+0000","last_fullsized":"2024-08-19T14:19:05.184416+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":5,"last_scrub":"273'5164","last_scrub_stamp":"2024-08-19T14:18:23.349706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:23.349706+0000","objects_scrubbed":3676,"log_size":7489,"log_dups_size":0,"ondisk_log_size":7489,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:28.805226+0000","scrub_duration":326,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":139526144,"num_objects":2129,"num_object_clones":0,"num_object_copies":6387,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2129,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2325,"num_write_kb":24896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"70'139","reported_seq":808,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-19T14:19:06.263391+0000","last_change":"2024-08-19T14:18:22.139657+0000","last_active":"2024-08-19T14:19:06.263391+0000","last_peered":"2024-08-19T14:19:06.263391+0000","last_clean":"2024-08-19T14:19:06.263391+0000","last_became_active":"2024-08-19T14:18:18.704562+0000","last_became_peered":"2024-08-19T14:18:18.704562+0000","last_unstale":"2024-08-19T14:19:06.263391+0000","last_undegraded":"2024-08-19T14:19:06.263391+0000","last_fullsized":"2024-08-19T14:19:06.263391+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:18:22.139587+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:18:22.139587+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:43.235144+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,8],"acting":[5,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"300'7430","reported_seq":12274,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-19T14:19:08.252559+0000","last_change":"2024-08-19T14:18:18.597206+0000","last_active":"2024-08-19T14:19:08.252559+0000","last_peered":"2024-08-19T14:19:08.252559+0000","last_clean":"2024-08-19T14:19:08.252559+0000","last_became_active":"2024-08-19T14:17:45.731114+0000","last_became_peered":"2024-08-19T14:17:45.731114+0000","last_unstale":"2024-08-19T14:19:08.252559+0000","last_undegraded":"2024-08-19T14:19:08.252559+0000","last_fullsized":"2024-08-19T14:19:08.252559+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":5,"last_scrub":"272'5093","last_scrub_stamp":"2024-08-19T14:18:18.596762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:18.596762+0000","objects_scrubbed":3558,"log_size":7430,"log_dups_size":0,"ondisk_log_size":7430,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:19.615099+0000","scrub_duration":834,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":144375808,"num_objects":2203,"num_object_clones":0,"num_object_copies":6609,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2203,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2357,"num_write_kb":32064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,9],"acting":[7,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.11","version":"70'139","reported_seq":355,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-08-19T14:19:05.185054+0000","last_change":"2024-08-19T14:18:25.021206+0000","last_active":"2024-08-19T14:19:05.185054+0000","last_peered":"2024-08-19T14:19:05.185054+0000","last_clean":"2024-08-19T14:19:05.185054+0000","last_became_active":"2024-08-19T14:18:18.697894+0000","last_became_peered":"2024-08-19T14:18:18.697894+0000","last_unstale":"2024-08-19T14:19:05.185054+0000","last_undegraded":"2024-08-19T14:19:05.185054+0000","last_fullsized":"2024-08-19T14:19:05.185054+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:18:25.021126+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:18:25.021126+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:32.345902+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.14","version":"300'7597","reported_seq":12563,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-19T14:19:08.017768+0000","last_change":"2024-08-19T14:18:19.862358+0000","last_active":"2024-08-19T14:19:08.017768+0000","last_peered":"2024-08-19T14:19:08.017768+0000","last_clean":"2024-08-19T14:19:08.017768+0000","last_became_active":"2024-08-19T14:17:44.691159+0000","last_became_peered":"2024-08-19T14:17:44.691159+0000","last_unstale":"2024-08-19T14:19:08.017768+0000","last_undegraded":"2024-08-19T14:19:08.017768+0000","last_fullsized":"2024-08-19T14:19:08.017768+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":5,"last_scrub":"273'5248","last_scrub_stamp":"2024-08-19T14:18:19.862173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:19.862173+0000","objects_scrubbed":3703,"log_size":7597,"log_dups_size":0,"ondisk_log_size":7597,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:39.609192+0000","scrub_duration":928,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":2240,"num_object_clones":0,"num_object_copies":6720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2389,"num_write_kb":29632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,11],"acting":[4,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.16","version":"70'139","reported_seq":333,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-19T14:19:06.182240+0000","last_change":"2024-08-19T14:18:54.742773+0000","last_active":"2024-08-19T14:19:06.182240+0000","last_peered":"2024-08-19T14:19:06.182240+0000","last_clean":"2024-08-19T14:19:06.182240+0000","last_became_active":"2024-08-19T14:18:31.899921+0000","last_became_peered":"2024-08-19T14:18:31.899921+0000","last_unstale":"2024-08-19T14:19:06.182240+0000","last_undegraded":"2024-08-19T14:19:06.182240+0000","last_fullsized":"2024-08-19T14:19:06.182240+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:18:54.742287+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:18:54.742287+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:20:08.261401+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,11],"acting":[1,0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"70'139","reported_seq":458,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-08-19T14:19:04.177350+0000","last_change":"2024-08-19T14:18:54.816222+0000","last_active":"2024-08-19T14:19:04.177350+0000","last_peered":"2024-08-19T14:19:04.177350+0000","last_clean":"2024-08-19T14:19:04.177350+0000","last_became_active":"2024-08-19T14:18:31.901556+0000","last_became_peered":"2024-08-19T14:18:31.901556+0000","last_unstale":"2024-08-19T14:19:04.177350+0000","last_undegraded":"2024-08-19T14:19:04.177350+0000","last_fullsized":"2024-08-19T14:19:04.177350+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:18:54.816110+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:18:54.816110+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:20:09.076175+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,0],"acting":[8,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.15","version":"298'7371","reported_seq":12235,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-08-19T14:19:04.202233+0000","last_change":"2024-08-19T14:18:20.779335+0000","last_active":"2024-08-19T14:19:04.202233+0000","last_peered":"2024-08-19T14:19:04.202233+0000","last_clean":"2024-08-19T14:19:04.202233+0000","last_became_active":"2024-08-19T14:17:42.696659+0000","last_became_peered":"2024-08-19T14:17:42.696659+0000","last_unstale":"2024-08-19T14:19:04.202233+0000","last_undegraded":"2024-08-19T14:19:04.202233+0000","last_fullsized":"2024-08-19T14:19:04.202233+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":5,"last_scrub":"273'5168","last_scrub_stamp":"2024-08-19T14:18:20.779227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:20.779227+0000","objects_scrubbed":3680,"log_size":7371,"log_dups_size":0,"ondisk_log_size":7371,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:35.778641+0000","scrub_duration":384,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":145948672,"num_objects":2227,"num_object_clones":0,"num_object_copies":6681,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2203,"num_write_kb":24000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.16","version":"298'7666","reported_seq":12493,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-19T14:19:06.263274+0000","last_change":"2024-08-19T14:18:18.940376+0000","last_active":"2024-08-19T14:19:06.263274+0000","last_peered":"2024-08-19T14:19:06.263274+0000","last_clean":"2024-08-19T14:19:06.263274+0000","last_became_active":"2024-08-19T14:17:47.681397+0000","last_became_peered":"2024-08-19T14:17:47.681397+0000","last_unstale":"2024-08-19T14:19:06.263274+0000","last_undegraded":"2024-08-19T14:19:06.263274+0000","last_fullsized":"2024-08-19T14:19:06.263274+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":5,"last_scrub":"273'5233","last_scrub_stamp":"2024-08-19T14:18:18.940216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:18.940216+0000","objects_scrubbed":3684,"log_size":7666,"log_dups_size":0,"ondisk_log_size":7666,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:22.269871+0000","scrub_duration":846,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":147062784,"num_objects":2244,"num_object_clones":0,"num_object_copies":6732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2244,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2452,"num_write_kb":32384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.14","version":"70'139","reported_seq":471,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-19T14:19:06.196941+0000","last_change":"2024-08-19T14:18:18.745307+0000","last_active":"2024-08-19T14:19:06.196941+0000","last_peered":"2024-08-19T14:19:06.196941+0000","last_clean":"2024-08-19T14:19:06.196941+0000","last_became_active":"2024-08-19T14:18:18.703926+0000","last_became_peered":"2024-08-19T14:18:18.703926+0000","last_unstale":"2024-08-19T14:19:06.196941+0000","last_undegraded":"2024-08-19T14:19:06.196941+0000","last_fullsized":"2024-08-19T14:19:06.196941+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:18:18.745191+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:18:18.745191+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:24.644627+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,1],"acting":[7,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"298'7299","reported_seq":13644,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-08-19T14:19:05.184942+0000","last_change":"2024-08-19T14:18:16.720459+0000","last_active":"2024-08-19T14:19:05.184942+0000","last_peered":"2024-08-19T14:19:05.184942+0000","last_clean":"2024-08-19T14:19:05.184942+0000","last_became_active":"2024-08-19T14:17:48.682321+0000","last_became_peered":"2024-08-19T14:17:48.682321+0000","last_unstale":"2024-08-19T14:19:05.184942+0000","last_undegraded":"2024-08-19T14:19:05.184942+0000","last_fullsized":"2024-08-19T14:19:05.184942+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":5,"last_scrub":"270'4919","last_scrub_stamp":"2024-08-19T14:18:16.720122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:16.720122+0000","objects_scrubbed":3405,"log_size":7299,"log_dups_size":0,"ondisk_log_size":7299,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:40.422232+0000","scrub_duration":754,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":145883136,"num_objects":2226,"num_object_clones":0,"num_object_copies":6678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2403,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,9],"acting":[0,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.15","version":"70'139","reported_seq":358,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-19T14:19:06.197115+0000","last_change":"2024-08-19T14:18:55.827423+0000","last_active":"2024-08-19T14:19:06.197115+0000","last_peered":"2024-08-19T14:19:06.197115+0000","last_clean":"2024-08-19T14:19:06.197115+0000","last_became_active":"2024-08-19T14:18:31.912496+0000","last_became_peered":"2024-08-19T14:18:31.912496+0000","last_unstale":"2024-08-19T14:19:06.197115+0000","last_undegraded":"2024-08-19T14:19:06.197115+0000","last_fullsized":"2024-08-19T14:19:06.197115+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:18:55.827302+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:55.827302+0000","last_clean_scrub_stamp":"2024-08-19T14:18:55.827302+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:20:08.036174+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"70'139","reported_seq":336,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-19T14:19:06.180846+0000","last_change":"2024-08-19T14:18:54.760739+0000","last_active":"2024-08-19T14:19:06.180846+0000","last_peered":"2024-08-19T14:19:06.180846+0000","last_clean":"2024-08-19T14:19:06.180846+0000","last_became_active":"2024-08-19T14:18:31.901229+0000","last_became_peered":"2024-08-19T14:18:31.901229+0000","last_unstale":"2024-08-19T14:19:06.180846+0000","last_undegraded":"2024-08-19T14:19:06.180846+0000","last_fullsized":"2024-08-19T14:19:06.180846+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:18:54.760619+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:18:54.760619+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:59.064566+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.18","version":"300'7596","reported_seq":12547,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-19T14:19:09.272184+0000","last_change":"2024-08-19T14:18:21.736611+0000","last_active":"2024-08-19T14:19:09.272184+0000","last_peered":"2024-08-19T14:19:09.272184+0000","last_clean":"2024-08-19T14:19:09.272184+0000","last_became_active":"2024-08-19T14:17:45.729134+0000","last_became_peered":"2024-08-19T14:17:45.729134+0000","last_unstale":"2024-08-19T14:19:09.272184+0000","last_undegraded":"2024-08-19T14:19:09.272184+0000","last_fullsized":"2024-08-19T14:19:09.272184+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":5,"last_scrub":"273'5311","last_scrub_stamp":"2024-08-19T14:18:21.736491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:21.736491+0000","objects_scrubbed":3712,"log_size":7596,"log_dups_size":0,"ondisk_log_size":7596,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:22.157061+0000","scrub_duration":424,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":139526144,"num_objects":2129,"num_object_clones":0,"num_object_copies":6387,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2129,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2285,"num_write_kb":22464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,1],"acting":[3,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1b","version":"70'139","reported_seq":684,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-08-19T14:19:04.176296+0000","last_change":"2024-08-19T14:18:53.860503+0000","last_active":"2024-08-19T14:19:04.176296+0000","last_peered":"2024-08-19T14:19:04.176296+0000","last_clean":"2024-08-19T14:19:04.176296+0000","last_became_active":"2024-08-19T14:18:31.901152+0000","last_became_peered":"2024-08-19T14:18:31.901152+0000","last_unstale":"2024-08-19T14:19:04.176296+0000","last_undegraded":"2024-08-19T14:19:04.176296+0000","last_fullsized":"2024-08-19T14:19:04.176296+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:18:53.860431+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:18:53.860431+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:20:08.015501+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.19","version":"300'7593","reported_seq":14104,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-19T14:19:08.434048+0000","last_change":"2024-08-19T14:18:21.161635+0000","last_active":"2024-08-19T14:19:08.434048+0000","last_peered":"2024-08-19T14:19:08.434048+0000","last_clean":"2024-08-19T14:19:08.434048+0000","last_became_active":"2024-08-19T14:17:46.796533+0000","last_became_peered":"2024-08-19T14:17:46.796533+0000","last_unstale":"2024-08-19T14:19:08.434048+0000","last_undegraded":"2024-08-19T14:19:08.434048+0000","last_fullsized":"2024-08-19T14:19:08.434048+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":5,"last_scrub":"273'5249","last_scrub_stamp":"2024-08-19T14:18:21.161536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:21.161536+0000","objects_scrubbed":3748,"log_size":7593,"log_dups_size":0,"ondisk_log_size":7593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:35.038752+0000","scrub_duration":491,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":142213120,"num_objects":2170,"num_object_clones":0,"num_object_copies":6510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2170,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2344,"num_write_kb":24512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.18","version":"70'139","reported_seq":732,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-19T14:19:06.263246+0000","last_change":"2024-08-19T14:18:54.126998+0000","last_active":"2024-08-19T14:19:06.263246+0000","last_peered":"2024-08-19T14:19:06.263246+0000","last_clean":"2024-08-19T14:19:06.263246+0000","last_became_active":"2024-08-19T14:18:31.902735+0000","last_became_peered":"2024-08-19T14:18:31.902735+0000","last_unstale":"2024-08-19T14:19:06.263246+0000","last_undegraded":"2024-08-19T14:19:06.263246+0000","last_fullsized":"2024-08-19T14:19:06.263246+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:18:54.126753+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:18:54.126753+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:20:17.241722+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"298'7944","reported_seq":12190,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-08-19T14:19:04.177213+0000","last_change":"2024-08-19T14:18:20.149227+0000","last_active":"2024-08-19T14:19:04.177213+0000","last_peered":"2024-08-19T14:19:04.177213+0000","last_clean":"2024-08-19T14:19:04.177213+0000","last_became_active":"2024-08-19T14:17:46.731170+0000","last_became_peered":"2024-08-19T14:17:46.731170+0000","last_unstale":"2024-08-19T14:19:04.177213+0000","last_undegraded":"2024-08-19T14:19:04.177213+0000","last_fullsized":"2024-08-19T14:19:04.177213+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":5,"last_scrub":"273'5710","last_scrub_stamp":"2024-08-19T14:18:20.148991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:20.148991+0000","objects_scrubbed":3721,"log_size":7944,"log_dups_size":0,"ondisk_log_size":7944,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:46.698048+0000","scrub_duration":772,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146931712,"num_objects":2242,"num_object_clones":0,"num_object_copies":6726,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2242,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2255,"num_write_kb":24832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,9],"acting":[8,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.19","version":"70'139","reported_seq":678,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-08-19T14:19:05.184911+0000","last_change":"2024-08-19T14:18:54.131138+0000","last_active":"2024-08-19T14:19:05.184911+0000","last_peered":"2024-08-19T14:19:05.184911+0000","last_clean":"2024-08-19T14:19:05.184911+0000","last_became_active":"2024-08-19T14:18:31.902547+0000","last_became_peered":"2024-08-19T14:18:31.902547+0000","last_unstale":"2024-08-19T14:19:05.184911+0000","last_undegraded":"2024-08-19T14:19:05.184911+0000","last_fullsized":"2024-08-19T14:19:05.184911+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:18:54.131032+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:18:54.131032+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:20:24.016240+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1b","version":"300'7977","reported_seq":14186,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-19T14:19:07.990714+0000","last_change":"2024-08-19T14:18:17.849009+0000","last_active":"2024-08-19T14:19:07.990714+0000","last_peered":"2024-08-19T14:19:07.990714+0000","last_clean":"2024-08-19T14:19:07.990714+0000","last_became_active":"2024-08-19T14:18:06.761591+0000","last_became_peered":"2024-08-19T14:18:06.761591+0000","last_unstale":"2024-08-19T14:19:07.990714+0000","last_undegraded":"2024-08-19T14:19:07.990714+0000","last_fullsized":"2024-08-19T14:19:07.990714+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":5,"last_scrub":"272'5596","last_scrub_stamp":"2024-08-19T14:18:17.848788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:17.848788+0000","objects_scrubbed":3567,"log_size":7977,"log_dups_size":0,"ondisk_log_size":7977,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:38.566335+0000","scrub_duration":920,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":142934016,"num_objects":2181,"num_object_clones":0,"num_object_copies":6543,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2181,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2398,"num_write_kb":32384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,1],"acting":[4,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1e","version":"70'139","reported_seq":336,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-08-19T14:19:05.179671+0000","last_change":"2024-08-19T14:18:53.600024+0000","last_active":"2024-08-19T14:19:05.179671+0000","last_peered":"2024-08-19T14:19:05.179671+0000","last_clean":"2024-08-19T14:19:05.179671+0000","last_became_active":"2024-08-19T14:18:31.902546+0000","last_became_peered":"2024-08-19T14:18:31.902546+0000","last_unstale":"2024-08-19T14:19:05.179671+0000","last_undegraded":"2024-08-19T14:19:05.179671+0000","last_fullsized":"2024-08-19T14:19:05.179671+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:18:53.599913+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:18:53.599913+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:20:18.091419+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"300'7800","reported_seq":13814,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-19T14:19:06.563966+0000","last_change":"2024-08-19T14:18:20.165996+0000","last_active":"2024-08-19T14:19:06.563966+0000","last_peered":"2024-08-19T14:19:06.563966+0000","last_clean":"2024-08-19T14:19:06.563966+0000","last_became_active":"2024-08-19T14:18:05.757137+0000","last_became_peered":"2024-08-19T14:18:05.757137+0000","last_unstale":"2024-08-19T14:19:06.563966+0000","last_undegraded":"2024-08-19T14:19:06.563966+0000","last_fullsized":"2024-08-19T14:19:06.563966+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":5,"last_scrub":"273'5655","last_scrub_stamp":"2024-08-19T14:18:20.165862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:20.165862+0000","objects_scrubbed":3615,"log_size":7800,"log_dups_size":0,"ondisk_log_size":7800,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:44.126700+0000","scrub_duration":1057,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":144703488,"num_objects":2208,"num_object_clones":0,"num_object_copies":6624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2208,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2169,"num_write_kb":24384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"298'8033","reported_seq":14297,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-08-19T14:19:04.177121+0000","last_change":"2024-08-19T14:18:21.800463+0000","last_active":"2024-08-19T14:19:04.177121+0000","last_peered":"2024-08-19T14:19:04.177121+0000","last_clean":"2024-08-19T14:19:04.177121+0000","last_became_active":"2024-08-19T14:17:59.735912+0000","last_became_peered":"2024-08-19T14:17:59.735912+0000","last_unstale":"2024-08-19T14:19:04.177121+0000","last_undegraded":"2024-08-19T14:19:04.177121+0000","last_fullsized":"2024-08-19T14:19:04.177121+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":5,"last_scrub":"273'5747","last_scrub_stamp":"2024-08-19T14:18:21.800353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:21.800353+0000","objects_scrubbed":3688,"log_size":8033,"log_dups_size":0,"ondisk_log_size":8033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:30.594474+0000","scrub_duration":368,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":145948696,"num_objects":2228,"num_object_clones":0,"num_object_copies":6684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2228,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2286,"num_write_kb":26369,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,5],"acting":[8,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1c","version":"70'139","reported_seq":442,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-08-19T14:19:04.175929+0000","last_change":"2024-08-19T14:18:55.843704+0000","last_active":"2024-08-19T14:19:04.175929+0000","last_peered":"2024-08-19T14:19:04.175929+0000","last_clean":"2024-08-19T14:19:04.175929+0000","last_became_active":"2024-08-19T14:18:31.902494+0000","last_became_peered":"2024-08-19T14:18:31.902494+0000","last_unstale":"2024-08-19T14:19:04.175929+0000","last_undegraded":"2024-08-19T14:19:04.175929+0000","last_fullsized":"2024-08-19T14:19:04.175929+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:18:55.843498+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:18:55.843498+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:20:05.235849+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,11],"acting":[8,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1e","version":"300'7937","reported_seq":16282,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-19T14:19:09.216585+0000","last_change":"2024-08-19T14:18:19.041894+0000","last_active":"2024-08-19T14:19:09.216585+0000","last_peered":"2024-08-19T14:19:09.216585+0000","last_clean":"2024-08-19T14:19:09.216585+0000","last_became_active":"2024-08-19T14:18:11.801671+0000","last_became_peered":"2024-08-19T14:18:11.801671+0000","last_unstale":"2024-08-19T14:19:09.216585+0000","last_undegraded":"2024-08-19T14:19:09.216585+0000","last_fullsized":"2024-08-19T14:19:09.216585+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":5,"last_scrub":"273'5544","last_scrub_stamp":"2024-08-19T14:18:19.041763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:19.041763+0000","objects_scrubbed":3560,"log_size":7937,"log_dups_size":0,"ondisk_log_size":7937,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:33.631537+0000","scrub_duration":775,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":135725056,"num_objects":2071,"num_object_clones":0,"num_object_copies":6213,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2071,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2411,"num_write_kb":29504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,8],"acting":[3,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1d","version":"70'139","reported_seq":368,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-19T14:19:06.181285+0000","last_change":"2024-08-19T14:18:55.752642+0000","last_active":"2024-08-19T14:19:06.181285+0000","last_peered":"2024-08-19T14:19:06.181285+0000","last_clean":"2024-08-19T14:19:06.181285+0000","last_became_active":"2024-08-19T14:18:31.901183+0000","last_became_peered":"2024-08-19T14:18:31.901183+0000","last_unstale":"2024-08-19T14:19:06.181285+0000","last_undegraded":"2024-08-19T14:19:06.181285+0000","last_fullsized":"2024-08-19T14:19:06.181285+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:18:55.752525+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:18:55.752525+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:20:16.513343+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,5],"acting":[1,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"298'7899","reported_seq":16140,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-08-19T14:19:05.184799+0000","last_change":"2024-08-19T14:18:19.910823+0000","last_active":"2024-08-19T14:19:05.184799+0000","last_peered":"2024-08-19T14:19:05.184799+0000","last_clean":"2024-08-19T14:19:05.184799+0000","last_became_active":"2024-08-19T14:18:12.800420+0000","last_became_peered":"2024-08-19T14:18:12.800420+0000","last_unstale":"2024-08-19T14:19:05.184799+0000","last_undegraded":"2024-08-19T14:19:05.184799+0000","last_fullsized":"2024-08-19T14:19:05.184799+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":5,"last_scrub":"273'5615","last_scrub_stamp":"2024-08-19T14:18:19.910511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:19.910511+0000","objects_scrubbed":3633,"log_size":7899,"log_dups_size":0,"ondisk_log_size":7899,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:41.416376+0000","scrub_duration":922,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":141033472,"num_objects":2152,"num_object_clones":0,"num_object_copies":6456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2317,"num_write_kb":26752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.20","version":"300'5371","reported_seq":7557,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-19T14:19:08.299328+0000","last_change":"2024-08-19T14:18:18.133721+0000","last_active":"2024-08-19T14:19:08.299328+0000","last_peered":"2024-08-19T14:19:08.299328+0000","last_clean":"2024-08-19T14:19:08.299328+0000","last_became_active":"2024-08-19T14:17:43.686357+0000","last_became_peered":"2024-08-19T14:17:43.686357+0000","last_unstale":"2024-08-19T14:19:08.299328+0000","last_undegraded":"2024-08-19T14:19:08.299328+0000","last_fullsized":"2024-08-19T14:19:08.299328+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":6,"last_scrub":"272'4197","last_scrub_stamp":"2024-08-19T14:18:18.133535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:18.133535+0000","objects_scrubbed":1747,"log_size":5371,"log_dups_size":0,"ondisk_log_size":5371,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:37.755792+0000","scrub_duration":374,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70189056,"num_objects":1071,"num_object_clones":0,"num_object_copies":3213,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1071,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1178,"num_write_kb":16064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,11],"acting":[9,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.21","version":"297'5392","reported_seq":9515,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-08-19T14:19:04.178302+0000","last_change":"2024-08-19T14:18:16.780918+0000","last_active":"2024-08-19T14:19:04.178302+0000","last_peered":"2024-08-19T14:19:04.178302+0000","last_clean":"2024-08-19T14:19:04.178302+0000","last_became_active":"2024-08-19T14:17:49.680746+0000","last_became_peered":"2024-08-19T14:17:49.680746+0000","last_unstale":"2024-08-19T14:19:04.178302+0000","last_undegraded":"2024-08-19T14:19:04.178302+0000","last_fullsized":"2024-08-19T14:19:04.178302+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":6,"last_scrub":"270'4217","last_scrub_stamp":"2024-08-19T14:18:16.780726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:16.780726+0000","objects_scrubbed":1743,"log_size":5392,"log_dups_size":0,"ondisk_log_size":5392,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:18.069764+0000","scrub_duration":329,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":73400320,"num_objects":1120,"num_object_clones":0,"num_object_copies":3360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1179,"num_write_kb":17792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,9],"acting":[8,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.22","version":"300'5637","reported_seq":8858,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-19T14:19:09.242661+0000","last_change":"2024-08-19T14:18:19.684323+0000","last_active":"2024-08-19T14:19:09.242661+0000","last_peered":"2024-08-19T14:19:09.242661+0000","last_clean":"2024-08-19T14:19:09.242661+0000","last_became_active":"2024-08-19T14:17:49.687020+0000","last_became_peered":"2024-08-19T14:17:49.687020+0000","last_unstale":"2024-08-19T14:19:09.242661+0000","last_undegraded":"2024-08-19T14:19:09.242661+0000","last_fullsized":"2024-08-19T14:19:09.242661+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":6,"last_scrub":"273'4443","last_scrub_stamp":"2024-08-19T14:18:19.684050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:19.684050+0000","objects_scrubbed":1928,"log_size":5637,"log_dups_size":0,"ondisk_log_size":5637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:41.056881+0000","scrub_duration":369,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75235328,"num_objects":1148,"num_object_clones":0,"num_object_copies":3444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1148,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1200,"num_write_kb":13440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,9],"acting":[3,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"297'5481","reported_seq":7762,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-08-19T14:19:05.185784+0000","last_change":"2024-08-19T14:18:22.194528+0000","last_active":"2024-08-19T14:19:05.185784+0000","last_peered":"2024-08-19T14:19:05.185784+0000","last_clean":"2024-08-19T14:19:05.185784+0000","last_became_active":"2024-08-19T14:17:46.797296+0000","last_became_peered":"2024-08-19T14:17:46.797296+0000","last_unstale":"2024-08-19T14:19:05.185784+0000","last_undegraded":"2024-08-19T14:19:05.185784+0000","last_fullsized":"2024-08-19T14:19:05.185784+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":6,"last_scrub":"273'4320","last_scrub_stamp":"2024-08-19T14:18:22.194414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:16:55.393990+0000","last_clean_scrub_stamp":"2024-08-19T14:18:22.194414+0000","objects_scrubbed":1877,"log_size":5481,"log_dups_size":0,"ondisk_log_size":5481,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:19:32.370226+0000","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":73793536,"num_objects":1126,"num_object_clones":0,"num_object_copies":3378,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1161,"num_write_kb":13120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":46,"stat_sum":{"num_bytes":4615700504,"num_objects":70431,"num_object_clones":0,"num_object_copies":211293,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":18957,"num_objects_unfound":0,"num_objects_dirty":70431,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":75234,"num_write_kb":927169,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":553,"num_bytes_recovered":36044800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13991555072,"data_stored":13991542856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":335715,"ondisk_log_size":335715,"up":138,"acting":129,"num_store_stats":9},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1802240,"data_stored":1788032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187549,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1759772,"kb_used_data":1435656,"kb_used_omap":1,"kb_used_meta":324094,"kb_avail":91985380,"statfs":{"total":95995035648,"available":94193029120,"internally_reserved":0,"allocated":1470111744,"data_stored":1468706791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":331872708},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":343597383779,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":74,"seq":317827580007,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1858664,"kb_used_data":1494304,"kb_used_omap":1,"kb_used_meta":364350,"kb_avail":91886488,"statfs":{"total":95995035648,"available":94091763712,"internally_reserved":0,"allocated":1530167296,"data_stored":1528754857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":373094849},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874348,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1628252,"kb_used_data":1358284,"kb_used_omap":1,"kb_used_meta":269950,"kb_avail":92116900,"statfs":{"total":95995035648,"available":94327705600,"internally_reserved":0,"allocated":1390882816,"data_stored":1389473791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":276429253},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":57,"seq":244813135983,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1710708,"kb_used_data":1346344,"kb_used_omap":1,"kb_used_meta":364350,"kb_avail":92034444,"statfs":{"total":95995035648,"available":94243270656,"internally_reserved":0,"allocated":1378656256,"data_stored":1377240265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":373094858},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":206158430323,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691959,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1800756,"kb_used_data":1575200,"kb_used_omap":1,"kb_used_meta":225534,"kb_avail":91944396,"statfs":{"total":95995035648,"available":94151061504,"internally_reserved":0,"allocated":1613004800,"data_stored":1611588629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":230947269},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986300,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2587052,"kb_used_data":2212964,"kb_used_omap":1,"kb_used_meta":374078,"kb_avail":91158100,"statfs":{"total":95995035648,"available":93345894400,"internally_reserved":0,"allocated":2266075136,"data_stored":2264658625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":383056325},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149826,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1755176,"kb_used_data":1408864,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":91989976,"statfs":{"total":95995035648,"available":94197735424,"internally_reserved":0,"allocated":1442676736,"data_stored":1441264297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":354613703},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411462,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574987,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1963824,"kb_used_data":1629864,"kb_used_omap":1,"kb_used_meta":333822,"kb_avail":91781328,"statfs":{"total":95995035648,"available":93984079872,"internally_reserved":0,"allocated":1668980736,"data_stored":1667564745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":341834181},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738511,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1560152,"kb_used_data":1280456,"kb_used_omap":1,"kb_used_meta":279678,"kb_avail":92185000,"statfs":{"total":95995035648,"available":94397440000,"internally_reserved":0,"allocated":1311186944,"data_stored":1309781991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":286390725},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1307312128,"data_stored":1307312128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1655701504,"data_stored":1655701504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1423310848,"data_stored":1423310848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2257981440,"data_stored":2257977368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1607733248,"data_stored":1607729176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1371537408,"data_stored":1371537408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1387335680,"data_stored":1387331608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1514078208,"data_stored":1514078208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1466564608,"data_stored":1466564608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T14:19:12.680 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-08-19T14:19:12.680 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd pool set .mgr pg_num 41 2024-08-19T14:19:12.693 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:19:12.716 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:12.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:12.717 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:19:12.729 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:19:12.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:12 smithi032 ceph-mon[20004]: Health check failed: Degraded data redundancy: 4/206112 objects degraded (0.002%), 2 pgs degraded (PG_DEGRADED) 2024-08-19T14:19:12.819 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:19:12.892 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:19:12.900 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:12.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:12.901 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:19:13.002 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:19:13.026 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:13.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:13.027 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:19:13.128 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:19:13.180 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:19:13.180 INFO:teuthology.orchestra.run.smithi045.stdout: "size": 20, 2024-08-19T14:19:13.180 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 600, 2024-08-19T14:19:13.180 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [ 2024-08-19T14:19:13.180 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.180 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24580.0:10284 2.5 2:a4caf01e:::benchmark_data_smithi045_46415_object10283:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-08-19T14:19:13.180 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:12:52.336926+0000", 2024-08-19T14:19:13.181 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 380.84080217600001, 2024-08-19T14:19:13.181 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.206277084, 2024-08-19T14:19:13.181 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:19:13.181 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:19:13.181 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:19:13.181 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:19:13.181 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24580", 2024-08-19T14:19:13.181 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:19:13.182 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 10284 2024-08-19T14:19:13.182 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.182 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:19:13.182 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.182 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:19:13.182 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.336926+0000", 2024-08-19T14:19:13.182 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:13.182 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.183 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.183 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:19:13.183 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.336926+0000", 2024-08-19T14:19:13.183 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:13.183 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.183 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.183 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:19:13.183 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.336927+0000", 2024-08-19T14:19:13.183 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3370000000000001e-06 2024-08-19T14:19:13.183 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.183 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.183 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:19:13.183 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.336949+0000", 2024-08-19T14:19:13.183 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1399e-05 2024-08-19T14:19:13.184 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.184 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.184 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:19:13.184 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.336961+0000", 2024-08-19T14:19:13.184 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.224e-05 2024-08-19T14:19:13.184 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.185 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.185 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:19:13.185 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.336968+0000", 2024-08-19T14:19:13.185 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.2710000000000003e-06 2024-08-19T14:19:13.185 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.185 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.185 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:19:13.185 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.337134+0000", 2024-08-19T14:19:13.185 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000165809 2024-08-19T14:19:13.185 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.185 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.185 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:19:13.186 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.337251+0000", 2024-08-19T14:19:13.186 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000116848 2024-08-19T14:19:13.186 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.186 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.186 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,4", 2024-08-19T14:19:13.186 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.337803+0000", 2024-08-19T14:19:13.186 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00055147699999999998 2024-08-19T14:19:13.186 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.186 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.186 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:19:13.186 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.339273+0000", 2024-08-19T14:19:13.186 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0014706750000000001 2024-08-19T14:19:13.186 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.187 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.187 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:19:13.187 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.339438+0000", 2024-08-19T14:19:13.187 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00016459299999999999 2024-08-19T14:19:13.187 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.187 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.187 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:19:13.187 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.543107+0000", 2024-08-19T14:19:13.187 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20366880100000001 2024-08-19T14:19:13.187 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.187 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.187 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:19:13.187 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.543162+0000", 2024-08-19T14:19:13.187 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.5075999999999999e-05 2024-08-19T14:19:13.188 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.188 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.188 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:19:13.188 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.543203+0000", 2024-08-19T14:19:13.188 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.1557999999999998e-05 2024-08-19T14:19:13.188 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:13.188 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:19:13.188 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:13.188 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.189 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.189 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24580.0:10301 2.f 2:f82f4cfa:::benchmark_data_smithi045_46415_object10300:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-08-19T14:19:13.189 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:12:52.341171+0000", 2024-08-19T14:19:13.189 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 380.83655733099999, 2024-08-19T14:19:13.189 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20191952899999999, 2024-08-19T14:19:13.189 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:19:13.189 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:19:13.189 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:19:13.189 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:19:13.189 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24580", 2024-08-19T14:19:13.189 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:19:13.189 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 10301 2024-08-19T14:19:13.189 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.190 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:19:13.190 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.190 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:19:13.190 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.341171+0000", 2024-08-19T14:19:13.190 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:13.190 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.190 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.190 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:19:13.190 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.341171+0000", 2024-08-19T14:19:13.190 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:13.191 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.191 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.191 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:19:13.191 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.341172+0000", 2024-08-19T14:19:13.191 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.093e-06 2024-08-19T14:19:13.191 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.191 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.191 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:19:13.191 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.341195+0000", 2024-08-19T14:19:13.191 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2435000000000001e-05 2024-08-19T14:19:13.191 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.191 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.191 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:19:13.192 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.341205+0000", 2024-08-19T14:19:13.192 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0567999999999999e-05 2024-08-19T14:19:13.192 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.192 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.192 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:19:13.192 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.341212+0000", 2024-08-19T14:19:13.192 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.686e-06 2024-08-19T14:19:13.192 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.192 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.192 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:19:13.193 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.341290+0000", 2024-08-19T14:19:13.193 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.8572999999999995e-05 2024-08-19T14:19:13.193 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.193 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.193 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:19:13.193 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.341427+0000", 2024-08-19T14:19:13.193 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000136754 2024-08-19T14:19:13.193 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.193 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.193 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 4,9", 2024-08-19T14:19:13.193 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.341748+0000", 2024-08-19T14:19:13.193 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00032119000000000001 2024-08-19T14:19:13.193 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.193 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.194 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:19:13.194 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.342661+0000", 2024-08-19T14:19:13.194 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000912761 2024-08-19T14:19:13.194 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.194 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.194 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:19:13.194 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.342681+0000", 2024-08-19T14:19:13.194 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0316999999999999e-05 2024-08-19T14:19:13.194 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.194 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.194 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:19:13.195 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.543009+0000", 2024-08-19T14:19:13.195 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.200327793 2024-08-19T14:19:13.195 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.195 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.195 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:19:13.195 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.543053+0000", 2024-08-19T14:19:13.195 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.4036000000000003e-05 2024-08-19T14:19:13.195 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.195 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.196 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:19:13.196 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.543091+0000", 2024-08-19T14:19:13.196 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.7323000000000002e-05 2024-08-19T14:19:13.196 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:13.197 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:19:13.197 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:13.197 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.197 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.197 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24580.0:10310 2.5 2:a1615b0f:::benchmark_data_smithi045_46415_object10309:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-08-19T14:19:13.197 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:12:52.342440+0000", 2024-08-19T14:19:13.197 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 380.83528852799998, 2024-08-19T14:19:13.198 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20090964, 2024-08-19T14:19:13.198 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:19:13.198 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:19:13.198 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:19:13.198 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:19:13.199 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24580", 2024-08-19T14:19:13.199 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:19:13.199 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 10310 2024-08-19T14:19:13.200 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.200 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:19:13.200 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.200 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:19:13.200 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.342440+0000", 2024-08-19T14:19:13.200 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:13.200 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.200 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.200 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:19:13.200 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.342440+0000", 2024-08-19T14:19:13.200 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:13.200 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.200 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.201 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:19:13.201 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.342441+0000", 2024-08-19T14:19:13.201 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.7499999999999998e-07 2024-08-19T14:19:13.201 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.201 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.201 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:19:13.201 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.342463+0000", 2024-08-19T14:19:13.201 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2303999999999999e-05 2024-08-19T14:19:13.201 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.201 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.201 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:19:13.201 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.342477+0000", 2024-08-19T14:19:13.201 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4099e-05 2024-08-19T14:19:13.201 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.202 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.202 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:19:13.202 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.342485+0000", 2024-08-19T14:19:13.202 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.6490000000000008e-06 2024-08-19T14:19:13.202 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.202 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.202 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:19:13.202 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.342571+0000", 2024-08-19T14:19:13.202 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.6238999999999999e-05 2024-08-19T14:19:13.203 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.203 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.203 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:19:13.203 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.343481+0000", 2024-08-19T14:19:13.203 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00091020900000000004 2024-08-19T14:19:13.203 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.203 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.203 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,4", 2024-08-19T14:19:13.203 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.343622+0000", 2024-08-19T14:19:13.203 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014069700000000001 2024-08-19T14:19:13.204 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.204 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.204 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:19:13.204 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.344447+0000", 2024-08-19T14:19:13.204 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00082507499999999998 2024-08-19T14:19:13.204 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.204 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.204 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:19:13.204 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.346610+0000", 2024-08-19T14:19:13.204 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0021627830000000002 2024-08-19T14:19:13.204 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.204 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.204 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:19:13.204 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.543255+0000", 2024-08-19T14:19:13.205 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.19664540699999999 2024-08-19T14:19:13.205 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.205 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.205 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:19:13.205 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.543309+0000", 2024-08-19T14:19:13.206 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4231999999999998e-05 2024-08-19T14:19:13.206 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.206 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.206 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:19:13.206 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.543349+0000", 2024-08-19T14:19:13.206 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.9971000000000001e-05 2024-08-19T14:19:13.206 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:13.206 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:19:13.206 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:13.206 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.206 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.206 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24580.0:10315 2.f 2:fe6f4b87:::benchmark_data_smithi045_46415_object10314:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-08-19T14:19:13.207 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:12:52.343698+0000", 2024-08-19T14:19:13.207 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 380.83403051800002, 2024-08-19T14:19:13.207 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.19949809700000001, 2024-08-19T14:19:13.207 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:19:13.207 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:19:13.207 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:19:13.207 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:19:13.208 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24580", 2024-08-19T14:19:13.208 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:19:13.208 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 10315 2024-08-19T14:19:13.208 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.208 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:19:13.208 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.208 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:19:13.208 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.343698+0000", 2024-08-19T14:19:13.208 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:13.208 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.209 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.209 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:19:13.209 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.343698+0000", 2024-08-19T14:19:13.209 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:13.209 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.209 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.209 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:19:13.209 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.343699+0000", 2024-08-19T14:19:13.209 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.238e-06 2024-08-19T14:19:13.209 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.209 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.209 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:19:13.209 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.343723+0000", 2024-08-19T14:19:13.209 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.4314000000000001e-05 2024-08-19T14:19:13.210 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.210 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.210 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:19:13.210 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.343735+0000", 2024-08-19T14:19:13.210 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2133999999999999e-05 2024-08-19T14:19:13.210 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.210 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.210 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:19:13.210 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.343743+0000", 2024-08-19T14:19:13.210 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.34e-06 2024-08-19T14:19:13.210 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.211 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.211 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:19:13.211 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.344026+0000", 2024-08-19T14:19:13.211 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00028303099999999999 2024-08-19T14:19:13.211 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.211 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.211 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:19:13.211 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.344278+0000", 2024-08-19T14:19:13.211 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00025181699999999999 2024-08-19T14:19:13.211 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.211 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.211 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 4,9", 2024-08-19T14:19:13.211 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.344520+0000", 2024-08-19T14:19:13.211 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000241864 2024-08-19T14:19:13.211 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.212 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.212 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:19:13.212 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.346213+0000", 2024-08-19T14:19:13.212 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001693398 2024-08-19T14:19:13.212 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.212 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.212 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:19:13.212 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.346420+0000", 2024-08-19T14:19:13.212 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00020658300000000001 2024-08-19T14:19:13.212 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.212 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.212 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:19:13.212 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.543130+0000", 2024-08-19T14:19:13.212 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.196710037 2024-08-19T14:19:13.213 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.213 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.213 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:19:13.213 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.543161+0000", 2024-08-19T14:19:13.213 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.1306999999999997e-05 2024-08-19T14:19:13.213 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.213 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.213 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:19:13.213 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.543196+0000", 2024-08-19T14:19:13.214 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.5033999999999997e-05 2024-08-19T14:19:13.214 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:13.214 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:19:13.214 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:13.214 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.214 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.214 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24580.0:10322 2.5 2:ad7f7050:::benchmark_data_smithi045_46415_object10321:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-08-19T14:19:13.214 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:12:52.345086+0000", 2024-08-19T14:19:13.214 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 380.83264243100001, 2024-08-19T14:19:13.214 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.19865560199999999, 2024-08-19T14:19:13.214 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:19:13.214 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:19:13.215 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:19:13.215 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:19:13.215 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24580", 2024-08-19T14:19:13.215 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:19:13.215 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 10322 2024-08-19T14:19:13.215 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.216 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:19:13.216 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.216 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:19:13.216 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.345086+0000", 2024-08-19T14:19:13.216 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:13.216 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.216 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.216 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:19:13.216 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.345086+0000", 2024-08-19T14:19:13.216 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:13.216 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.216 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.216 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:19:13.217 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.345087+0000", 2024-08-19T14:19:13.217 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.8299999999999996e-07 2024-08-19T14:19:13.217 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.217 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.217 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:19:13.217 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.345106+0000", 2024-08-19T14:19:13.217 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.91e-05 2024-08-19T14:19:13.217 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.217 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.217 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:19:13.217 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.345114+0000", 2024-08-19T14:19:13.217 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.6200000000000005e-06 2024-08-19T14:19:13.217 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.218 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.218 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:19:13.218 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.345122+0000", 2024-08-19T14:19:13.218 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.8310000000000001e-06 2024-08-19T14:19:13.218 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.218 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.218 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:19:13.218 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.345281+0000", 2024-08-19T14:19:13.218 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000158386 2024-08-19T14:19:13.218 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.218 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.219 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:19:13.220 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.345396+0000", 2024-08-19T14:19:13.220 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000115299 2024-08-19T14:19:13.220 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.220 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.220 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,4", 2024-08-19T14:19:13.220 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.345628+0000", 2024-08-19T14:19:13.220 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00023218499999999999 2024-08-19T14:19:13.220 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.220 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.220 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:19:13.220 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.347682+0000", 2024-08-19T14:19:13.221 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0020542189999999999 2024-08-19T14:19:13.221 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.221 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.221 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:19:13.221 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.348039+0000", 2024-08-19T14:19:13.221 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00035721800000000002 2024-08-19T14:19:13.221 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.221 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.221 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:19:13.221 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.543654+0000", 2024-08-19T14:19:13.221 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.19561498799999999 2024-08-19T14:19:13.221 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.222 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.222 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:19:13.222 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.543708+0000", 2024-08-19T14:19:13.222 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.3152999999999997e-05 2024-08-19T14:19:13.222 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.222 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.222 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:19:13.222 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.543741+0000", 2024-08-19T14:19:13.222 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.392e-05 2024-08-19T14:19:13.222 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:13.222 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:19:13.223 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:13.223 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.223 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.223 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24580.0:10329 2.5 2:a14cf373:::benchmark_data_smithi045_46415_object10328:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-08-19T14:19:13.223 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:12:52.346732+0000", 2024-08-19T14:19:13.223 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 380.830996094, 2024-08-19T14:19:13.223 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.197173301, 2024-08-19T14:19:13.223 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:19:13.223 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:19:13.223 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:19:13.223 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:19:13.224 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24580", 2024-08-19T14:19:13.224 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:19:13.224 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 10329 2024-08-19T14:19:13.224 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.224 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:19:13.224 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.224 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:19:13.224 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.346732+0000", 2024-08-19T14:19:13.224 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:13.224 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.224 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.225 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:19:13.225 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.346732+0000", 2024-08-19T14:19:13.225 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:13.225 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.225 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.225 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:19:13.225 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.346733+0000", 2024-08-19T14:19:13.225 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.4799999999999997e-07 2024-08-19T14:19:13.225 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.225 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.225 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:19:13.225 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.346755+0000", 2024-08-19T14:19:13.226 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1695000000000002e-05 2024-08-19T14:19:13.226 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.226 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.226 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:19:13.226 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.346768+0000", 2024-08-19T14:19:13.226 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2951e-05 2024-08-19T14:19:13.226 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.226 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.226 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:19:13.226 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.346773+0000", 2024-08-19T14:19:13.227 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4659999999999998e-06 2024-08-19T14:19:13.227 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.227 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.227 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:19:13.227 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.346886+0000", 2024-08-19T14:19:13.227 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000112957 2024-08-19T14:19:13.227 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.227 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.227 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:19:13.227 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.347003+0000", 2024-08-19T14:19:13.227 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000117138 2024-08-19T14:19:13.228 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.228 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.228 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,4", 2024-08-19T14:19:13.228 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.347219+0000", 2024-08-19T14:19:13.228 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00021526999999999999 2024-08-19T14:19:13.228 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.228 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.228 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:19:13.228 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.347921+0000", 2024-08-19T14:19:13.228 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00070217000000000001 2024-08-19T14:19:13.228 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.228 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.229 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:19:13.229 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.349966+0000", 2024-08-19T14:19:13.229 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0020451800000000002 2024-08-19T14:19:13.229 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.229 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.229 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:19:13.229 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.543791+0000", 2024-08-19T14:19:13.229 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.19382521 2024-08-19T14:19:13.229 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.229 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.229 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:19:13.230 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.543848+0000", 2024-08-19T14:19:13.230 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7153e-05 2024-08-19T14:19:13.230 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.230 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.230 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:19:13.230 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.543906+0000", 2024-08-19T14:19:13.230 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7163000000000001e-05 2024-08-19T14:19:13.230 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:13.230 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:19:13.230 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:13.230 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.231 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.231 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24580.0:10333 2.5 2:abe3c6e5:::benchmark_data_smithi045_46415_object10332:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-08-19T14:19:13.231 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:12:52.348039+0000", 2024-08-19T14:19:13.231 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 380.82968924900001, 2024-08-19T14:19:13.231 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.19636310100000001, 2024-08-19T14:19:13.231 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:19:13.231 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:19:13.231 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:19:13.231 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:19:13.231 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24580", 2024-08-19T14:19:13.231 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:19:13.231 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 10333 2024-08-19T14:19:13.232 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.232 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:19:13.232 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.232 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:19:13.232 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.348039+0000", 2024-08-19T14:19:13.232 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:13.232 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.232 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.232 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:19:13.232 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.348039+0000", 2024-08-19T14:19:13.232 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:13.233 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.233 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.233 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:19:13.233 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.348040+0000", 2024-08-19T14:19:13.233 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.0599999999999999e-07 2024-08-19T14:19:13.233 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.233 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.233 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:19:13.233 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.348064+0000", 2024-08-19T14:19:13.233 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.3768e-05 2024-08-19T14:19:13.233 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.233 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.233 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:19:13.234 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.348075+0000", 2024-08-19T14:19:13.234 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1083999999999999e-05 2024-08-19T14:19:13.234 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.234 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.234 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:19:13.234 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.348082+0000", 2024-08-19T14:19:13.234 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.3010000000000001e-06 2024-08-19T14:19:13.234 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.234 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.234 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:19:13.234 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.348125+0000", 2024-08-19T14:19:13.235 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.2559999999999999e-05 2024-08-19T14:19:13.235 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.235 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.235 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:19:13.235 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.348235+0000", 2024-08-19T14:19:13.235 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00011069300000000001 2024-08-19T14:19:13.235 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.235 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.235 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,4", 2024-08-19T14:19:13.235 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.348341+0000", 2024-08-19T14:19:13.235 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00010518099999999999 2024-08-19T14:19:13.235 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.236 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.236 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:19:13.236 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.349463+0000", 2024-08-19T14:19:13.236 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0011227909999999999 2024-08-19T14:19:13.237 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.237 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.237 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:19:13.237 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.350404+0000", 2024-08-19T14:19:13.237 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00094061000000000004 2024-08-19T14:19:13.237 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.237 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.237 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:19:13.237 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.544288+0000", 2024-08-19T14:19:13.237 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.19388386399999999 2024-08-19T14:19:13.237 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.238 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.238 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:19:13.238 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.544356+0000", 2024-08-19T14:19:13.238 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.8329000000000003e-05 2024-08-19T14:19:13.238 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.238 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.238 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:19:13.238 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.544402+0000", 2024-08-19T14:19:13.238 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.6013999999999997e-05 2024-08-19T14:19:13.238 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:13.238 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:19:13.238 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:13.239 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.239 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.239 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24580.0:10336 2.5 2:a6b0c637:::benchmark_data_smithi045_46415_object10335:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-08-19T14:19:13.239 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:12:52.348409+0000", 2024-08-19T14:19:13.239 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 380.82931884599998, 2024-08-19T14:19:13.239 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.19615854299999999, 2024-08-19T14:19:13.239 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:19:13.239 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:19:13.239 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:19:13.239 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:19:13.239 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24580", 2024-08-19T14:19:13.240 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:19:13.240 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 10336 2024-08-19T14:19:13.240 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.240 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:19:13.240 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.240 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:19:13.240 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.348409+0000", 2024-08-19T14:19:13.240 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:13.240 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.240 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.240 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:19:13.240 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.348409+0000", 2024-08-19T14:19:13.241 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:13.241 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.241 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.241 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:19:13.241 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.348410+0000", 2024-08-19T14:19:13.241 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4799999999999997e-07 2024-08-19T14:19:13.241 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.241 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.241 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:19:13.241 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.348431+0000", 2024-08-19T14:19:13.241 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0757000000000001e-05 2024-08-19T14:19:13.241 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.242 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.242 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:19:13.242 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.348442+0000", 2024-08-19T14:19:13.242 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0611000000000001e-05 2024-08-19T14:19:13.242 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.242 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.242 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:19:13.242 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.348449+0000", 2024-08-19T14:19:13.242 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.9949999999999999e-06 2024-08-19T14:19:13.242 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.242 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.243 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:19:13.243 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.348754+0000", 2024-08-19T14:19:13.243 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00030516199999999999 2024-08-19T14:19:13.243 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.243 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.243 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:19:13.243 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.348885+0000", 2024-08-19T14:19:13.243 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013173500000000001 2024-08-19T14:19:13.243 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.243 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.244 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,4", 2024-08-19T14:19:13.244 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.349038+0000", 2024-08-19T14:19:13.244 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015267899999999999 2024-08-19T14:19:13.244 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.244 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.244 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:19:13.244 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.349637+0000", 2024-08-19T14:19:13.244 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00059873499999999998 2024-08-19T14:19:13.244 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.244 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.244 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:19:13.245 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.350437+0000", 2024-08-19T14:19:13.245 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00079989500000000001 2024-08-19T14:19:13.245 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.245 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.245 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:19:13.245 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.544475+0000", 2024-08-19T14:19:13.245 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.19403847499999999 2024-08-19T14:19:13.245 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.245 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.245 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:19:13.245 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.544531+0000", 2024-08-19T14:19:13.246 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.5288000000000003e-05 2024-08-19T14:19:13.246 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.246 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.246 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:19:13.246 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.544568+0000", 2024-08-19T14:19:13.246 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.7462999999999997e-05 2024-08-19T14:19:13.246 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:13.246 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:19:13.246 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:13.246 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.246 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.247 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24580.0:10342 2.5 2:aaeb89b1:::benchmark_data_smithi045_46415_object10341:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-08-19T14:19:13.247 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:12:52.350928+0000", 2024-08-19T14:19:13.247 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 380.82680029800002, 2024-08-19T14:19:13.247 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.19439313599999999, 2024-08-19T14:19:13.247 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:19:13.247 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:19:13.247 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:19:13.247 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:19:13.247 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24580", 2024-08-19T14:19:13.247 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:19:13.247 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 10342 2024-08-19T14:19:13.247 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.247 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:19:13.248 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.248 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:19:13.248 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.350928+0000", 2024-08-19T14:19:13.248 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:13.249 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.249 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.249 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:19:13.250 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.350928+0000", 2024-08-19T14:19:13.250 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:13.250 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.250 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.250 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:19:13.250 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.350929+0000", 2024-08-19T14:19:13.250 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.3399999999999998e-07 2024-08-19T14:19:13.250 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.250 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.250 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:19:13.250 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.350951+0000", 2024-08-19T14:19:13.250 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2436999999999999e-05 2024-08-19T14:19:13.251 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.251 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.251 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:19:13.251 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.350961+0000", 2024-08-19T14:19:13.251 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.3039999999999994e-06 2024-08-19T14:19:13.251 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.251 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.251 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:19:13.251 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.350967+0000", 2024-08-19T14:19:13.251 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.8109999999999997e-06 2024-08-19T14:19:13.251 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.251 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.251 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:19:13.252 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.351045+0000", 2024-08-19T14:19:13.252 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.7655000000000001e-05 2024-08-19T14:19:13.252 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.252 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.252 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:19:13.252 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.351364+0000", 2024-08-19T14:19:13.254 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00031882500000000001 2024-08-19T14:19:13.254 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.254 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.254 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,4", 2024-08-19T14:19:13.254 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.351549+0000", 2024-08-19T14:19:13.255 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00018494900000000001 2024-08-19T14:19:13.255 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.255 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.255 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:19:13.255 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.352871+0000", 2024-08-19T14:19:13.255 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0013225579999999999 2024-08-19T14:19:13.255 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.255 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.255 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:19:13.255 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.352993+0000", 2024-08-19T14:19:13.255 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012199199999999999 2024-08-19T14:19:13.255 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.256 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.256 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:19:13.256 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.545110+0000", 2024-08-19T14:19:13.256 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.192116701 2024-08-19T14:19:13.256 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.256 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.256 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:19:13.256 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.545188+0000", 2024-08-19T14:19:13.256 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.8238000000000005e-05 2024-08-19T14:19:13.256 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.256 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.256 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:19:13.257 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.545321+0000", 2024-08-19T14:19:13.257 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013283199999999999 2024-08-19T14:19:13.257 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:13.257 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:19:13.257 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:13.257 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.257 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.257 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24580.0:88314 2.f 2:fc0d2546:::benchmark_data_smithi045_46415_object88313:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-08-19T14:19:13.257 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:42.807395+0000", 2024-08-19T14:19:13.257 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 330.37033324399999, 2024-08-19T14:19:13.257 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.21007711900000001, 2024-08-19T14:19:13.258 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:19:13.258 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:19:13.258 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:19:13.258 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:19:13.258 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24580", 2024-08-19T14:19:13.258 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:19:13.258 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 88314 2024-08-19T14:19:13.258 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.258 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:19:13.258 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.258 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:19:13.259 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.807395+0000", 2024-08-19T14:19:13.259 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:13.259 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.259 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.259 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:19:13.259 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.807395+0000", 2024-08-19T14:19:13.259 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:13.259 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.259 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.259 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:19:13.259 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.807397+0000", 2024-08-19T14:19:13.259 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1239999999999999e-06 2024-08-19T14:19:13.260 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.260 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.260 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:19:13.260 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.807424+0000", 2024-08-19T14:19:13.260 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.6429000000000001e-05 2024-08-19T14:19:13.260 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.260 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.260 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:19:13.260 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.807445+0000", 2024-08-19T14:19:13.260 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1078e-05 2024-08-19T14:19:13.260 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.260 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.261 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:19:13.261 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.807455+0000", 2024-08-19T14:19:13.261 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0351e-05 2024-08-19T14:19:13.261 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.261 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.261 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:19:13.261 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.807524+0000", 2024-08-19T14:19:13.261 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.9122999999999995e-05 2024-08-19T14:19:13.261 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.261 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.261 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:19:13.262 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.807862+0000", 2024-08-19T14:19:13.262 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000338222 2024-08-19T14:19:13.262 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.262 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.262 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 4,9", 2024-08-19T14:19:13.262 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.808028+0000", 2024-08-19T14:19:13.262 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00016509700000000001 2024-08-19T14:19:13.262 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.262 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.262 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:19:13.262 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.811333+0000", 2024-08-19T14:19:13.262 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0033057300000000002 2024-08-19T14:19:13.263 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.263 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.263 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:19:13.263 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.814174+0000", 2024-08-19T14:19:13.263 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0028408959999999999 2024-08-19T14:19:13.263 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.263 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.263 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:19:13.263 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:43.017401+0000", 2024-08-19T14:19:13.263 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.203226459 2024-08-19T14:19:13.263 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.264 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.264 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:19:13.264 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:43.017443+0000", 2024-08-19T14:19:13.264 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.2438999999999998e-05 2024-08-19T14:19:13.264 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.264 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.264 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:19:13.264 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:43.017472+0000", 2024-08-19T14:19:13.264 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.9170999999999999e-05 2024-08-19T14:19:13.264 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:13.264 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:19:13.264 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:13.265 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.265 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.265 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24580.0:88316 2.5 2:ade27d21:::benchmark_data_smithi045_46415_object88315:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-08-19T14:19:13.265 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:42.807950+0000", 2024-08-19T14:19:13.265 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 330.36977829400001, 2024-08-19T14:19:13.265 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.21254900700000001, 2024-08-19T14:19:13.265 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:19:13.265 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:19:13.265 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:19:13.265 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:19:13.265 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24580", 2024-08-19T14:19:13.265 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:19:13.266 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 88316 2024-08-19T14:19:13.266 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.266 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:19:13.266 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.266 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:19:13.266 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.807950+0000", 2024-08-19T14:19:13.266 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:13.266 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.266 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.266 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:19:13.266 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.807950+0000", 2024-08-19T14:19:13.267 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:13.267 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.267 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.267 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:19:13.267 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.807951+0000", 2024-08-19T14:19:13.267 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.046e-06 2024-08-19T14:19:13.267 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.267 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.267 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:19:13.267 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.807985+0000", 2024-08-19T14:19:13.267 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.3565999999999997e-05 2024-08-19T14:19:13.267 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.268 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.268 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:19:13.268 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.807997+0000", 2024-08-19T14:19:13.268 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2690999999999999e-05 2024-08-19T14:19:13.268 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.268 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.268 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:19:13.268 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.808005+0000", 2024-08-19T14:19:13.268 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.8089999999999993e-06 2024-08-19T14:19:13.268 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.268 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.268 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:19:13.269 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.808063+0000", 2024-08-19T14:19:13.269 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.8108e-05 2024-08-19T14:19:13.269 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.269 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.269 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:19:13.269 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.808366+0000", 2024-08-19T14:19:13.269 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00030297899999999999 2024-08-19T14:19:13.269 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.269 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.269 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,4", 2024-08-19T14:19:13.269 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.808492+0000", 2024-08-19T14:19:13.269 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000125556 2024-08-19T14:19:13.270 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.270 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.270 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:19:13.270 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.811416+0000", 2024-08-19T14:19:13.270 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0029237540000000002 2024-08-19T14:19:13.270 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.270 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.270 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:19:13.270 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.818511+0000", 2024-08-19T14:19:13.270 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0070953980000000002 2024-08-19T14:19:13.270 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.271 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.271 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:19:13.271 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:43.020384+0000", 2024-08-19T14:19:13.271 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.201872945 2024-08-19T14:19:13.271 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.271 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.271 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:19:13.271 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:43.020458+0000", 2024-08-19T14:19:13.271 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.3699e-05 2024-08-19T14:19:13.271 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.271 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.272 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:19:13.272 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:43.020499+0000", 2024-08-19T14:19:13.272 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.1455999999999997e-05 2024-08-19T14:19:13.272 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:13.272 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:19:13.272 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:13.272 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.272 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.272 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24580.0:88321 2.5 2:a4cf6cb2:::benchmark_data_smithi045_46415_object88320:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-08-19T14:19:13.272 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:42.814619+0000", 2024-08-19T14:19:13.272 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 330.363109243, 2024-08-19T14:19:13.272 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20609303900000001, 2024-08-19T14:19:13.273 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:19:13.273 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:19:13.273 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:19:13.273 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:19:13.273 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24580", 2024-08-19T14:19:13.273 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:19:13.273 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 88321 2024-08-19T14:19:13.273 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.273 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:19:13.273 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.273 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:19:13.274 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.814619+0000", 2024-08-19T14:19:13.274 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:13.274 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.274 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.274 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:19:13.274 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.814619+0000", 2024-08-19T14:19:13.274 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:13.274 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.274 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.274 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:19:13.274 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.814621+0000", 2024-08-19T14:19:13.274 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.514e-06 2024-08-19T14:19:13.275 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.275 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.275 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:19:13.275 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.814641+0000", 2024-08-19T14:19:13.275 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0852e-05 2024-08-19T14:19:13.275 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.275 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.275 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:19:13.275 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.814653+0000", 2024-08-19T14:19:13.275 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1293e-05 2024-08-19T14:19:13.275 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.276 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.276 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:19:13.276 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.814660+0000", 2024-08-19T14:19:13.276 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.905e-06 2024-08-19T14:19:13.276 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.276 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.276 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:19:13.276 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.814711+0000", 2024-08-19T14:19:13.276 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1657999999999999e-05 2024-08-19T14:19:13.276 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.276 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.276 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:19:13.277 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.814825+0000", 2024-08-19T14:19:13.277 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000113597 2024-08-19T14:19:13.277 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.277 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.277 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,4", 2024-08-19T14:19:13.277 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.814949+0000", 2024-08-19T14:19:13.277 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012450800000000001 2024-08-19T14:19:13.277 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.277 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.277 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:19:13.277 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.817037+0000", 2024-08-19T14:19:13.278 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0020878089999999999 2024-08-19T14:19:13.278 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.278 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.278 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:19:13.278 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.819696+0000", 2024-08-19T14:19:13.278 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.002659004 2024-08-19T14:19:13.278 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.278 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.278 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:19:13.278 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:43.020593+0000", 2024-08-19T14:19:13.278 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20089647899999999 2024-08-19T14:19:13.278 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.279 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.279 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:19:13.279 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:43.020673+0000", 2024-08-19T14:19:13.279 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.9881000000000003e-05 2024-08-19T14:19:13.279 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.279 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.279 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:19:13.279 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:43.020712+0000", 2024-08-19T14:19:13.279 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.9539000000000002e-05 2024-08-19T14:19:13.279 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:13.279 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:19:13.279 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:13.280 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.280 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.280 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24580.0:88328 2.5 2:a21c7eb0:::benchmark_data_smithi045_46415_object88327:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-08-19T14:19:13.280 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:42.820640+0000", 2024-08-19T14:19:13.280 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 330.35708801800001, 2024-08-19T14:19:13.280 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20083906900000001, 2024-08-19T14:19:13.280 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:19:13.280 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:19:13.280 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:19:13.280 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:19:13.280 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24580", 2024-08-19T14:19:13.281 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:19:13.281 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 88328 2024-08-19T14:19:13.281 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.281 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:19:13.281 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.281 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:19:13.281 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.820640+0000", 2024-08-19T14:19:13.281 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:13.281 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.281 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.281 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:19:13.281 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.820640+0000", 2024-08-19T14:19:13.281 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:13.282 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.282 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.282 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:19:13.282 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.820641+0000", 2024-08-19T14:19:13.282 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.23e-07 2024-08-19T14:19:13.282 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.282 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.282 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:19:13.282 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.820660+0000", 2024-08-19T14:19:13.282 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8788e-05 2024-08-19T14:19:13.282 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.283 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.283 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:19:13.283 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.820670+0000", 2024-08-19T14:19:13.283 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.9089999999999995e-06 2024-08-19T14:19:13.283 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.283 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.283 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:19:13.283 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.820675+0000", 2024-08-19T14:19:13.283 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.8890000000000001e-06 2024-08-19T14:19:13.283 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.283 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.284 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:19:13.284 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.820718+0000", 2024-08-19T14:19:13.284 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.2997999999999997e-05 2024-08-19T14:19:13.284 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.284 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.284 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:19:13.284 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.820811+0000", 2024-08-19T14:19:13.284 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.3219000000000003e-05 2024-08-19T14:19:13.284 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.284 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.284 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,4", 2024-08-19T14:19:13.285 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.820907+0000", 2024-08-19T14:19:13.285 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.6267000000000005e-05 2024-08-19T14:19:13.285 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.285 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.285 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:19:13.285 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.823083+0000", 2024-08-19T14:19:13.285 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0021754299999999999 2024-08-19T14:19:13.285 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.285 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.285 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:19:13.285 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.824768+0000", 2024-08-19T14:19:13.285 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001685048 2024-08-19T14:19:13.285 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.286 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.286 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:19:13.286 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:43.021388+0000", 2024-08-19T14:19:13.286 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.196620295 2024-08-19T14:19:13.286 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.286 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.286 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:19:13.286 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:43.021436+0000", 2024-08-19T14:19:13.286 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.8195000000000002e-05 2024-08-19T14:19:13.286 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.286 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.286 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:19:13.287 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:43.021479+0000", 2024-08-19T14:19:13.287 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.3307999999999999e-05 2024-08-19T14:19:13.287 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:13.287 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:19:13.287 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:13.287 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.287 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.287 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24580.0:209201 2.f 2:fb8d21d4:::benchmark_data_smithi045_46415_object85920:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194)", 2024-08-19T14:19:13.287 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:16:02.057866+0000", 2024-08-19T14:19:13.287 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 191.119862692, 2024-08-19T14:19:13.287 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.21458635300000001, 2024-08-19T14:19:13.288 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:19:13.288 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:19:13.288 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:19:13.288 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:19:13.288 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24580", 2024-08-19T14:19:13.288 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:19:13.288 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 209201 2024-08-19T14:19:13.288 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.288 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:19:13.288 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.288 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:19:13.288 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:16:02.057866+0000", 2024-08-19T14:19:13.288 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:13.289 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.289 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.289 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:19:13.289 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:16:02.057866+0000", 2024-08-19T14:19:13.289 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:13.289 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.289 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.289 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:19:13.289 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:16:02.057867+0000", 2024-08-19T14:19:13.289 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.540000000000001e-07 2024-08-19T14:19:13.289 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.289 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.290 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:19:13.290 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:16:02.057871+0000", 2024-08-19T14:19:13.290 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.1010000000000002e-06 2024-08-19T14:19:13.290 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.290 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.290 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:19:13.290 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:16:02.057887+0000", 2024-08-19T14:19:13.290 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6809000000000001e-05 2024-08-19T14:19:13.290 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.290 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.290 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:19:13.290 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:16:02.057893+0000", 2024-08-19T14:19:13.291 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7880000000000003e-06 2024-08-19T14:19:13.291 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.291 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.291 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:19:13.291 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:16:02.057942+0000", 2024-08-19T14:19:13.291 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.8983999999999997e-05 2024-08-19T14:19:13.291 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.291 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.291 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:19:13.291 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:16:02.058135+0000", 2024-08-19T14:19:13.291 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000192692 2024-08-19T14:19:13.291 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.292 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.292 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 4,9", 2024-08-19T14:19:13.292 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:16:02.058253+0000", 2024-08-19T14:19:13.292 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000118071 2024-08-19T14:19:13.292 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.292 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.292 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:19:13.292 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:16:02.062846+0000", 2024-08-19T14:19:13.292 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.004593209 2024-08-19T14:19:13.292 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.292 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.292 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:19:13.293 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:16:02.062913+0000", 2024-08-19T14:19:13.293 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.6808000000000001e-05 2024-08-19T14:19:13.293 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.293 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.293 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:19:13.293 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:16:02.272371+0000", 2024-08-19T14:19:13.293 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.209458071 2024-08-19T14:19:13.293 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.293 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.293 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:19:13.293 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:16:02.272407+0000", 2024-08-19T14:19:13.293 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.6031000000000001e-05 2024-08-19T14:19:13.294 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.294 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.294 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:19:13.294 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:16:02.272452+0000", 2024-08-19T14:19:13.294 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.4835e-05 2024-08-19T14:19:13.294 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:13.294 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:19:13.294 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:13.294 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.294 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.294 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24580.0:209204 2.f 2:f5b7b692:::benchmark_data_smithi045_46415_object85923:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194)", 2024-08-19T14:19:13.295 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:16:02.063760+0000", 2024-08-19T14:19:13.295 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 191.113968787, 2024-08-19T14:19:13.295 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.21463046399999999, 2024-08-19T14:19:13.295 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:19:13.295 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:19:13.295 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:19:13.295 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:19:13.295 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24580", 2024-08-19T14:19:13.295 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:19:13.295 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 209204 2024-08-19T14:19:13.295 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.296 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:19:13.296 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.296 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:19:13.296 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:16:02.063760+0000", 2024-08-19T14:19:13.296 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:13.296 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.296 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.296 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:19:13.296 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:16:02.063760+0000", 2024-08-19T14:19:13.296 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:13.296 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.296 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.297 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:19:13.297 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:16:02.063761+0000", 2024-08-19T14:19:13.297 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2419999999999999e-06 2024-08-19T14:19:13.297 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.297 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.297 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:19:13.297 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:16:02.063765+0000", 2024-08-19T14:19:13.297 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.0629999999999999e-06 2024-08-19T14:19:13.297 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.297 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.298 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:19:13.298 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:16:02.063775+0000", 2024-08-19T14:19:13.298 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.8239999999999995e-06 2024-08-19T14:19:13.298 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.298 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.298 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:19:13.298 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:16:02.063780+0000", 2024-08-19T14:19:13.298 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.31e-06 2024-08-19T14:19:13.298 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.298 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.299 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:19:13.299 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:16:02.063822+0000", 2024-08-19T14:19:13.299 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.2009999999999999e-05 2024-08-19T14:19:13.299 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.299 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.299 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:19:13.299 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:16:02.063940+0000", 2024-08-19T14:19:13.299 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00011828999999999999 2024-08-19T14:19:13.299 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.299 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.299 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 4,9", 2024-08-19T14:19:13.299 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:16:02.064018+0000", 2024-08-19T14:19:13.300 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.7959000000000004e-05 2024-08-19T14:19:13.300 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.300 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.300 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:19:13.300 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:16:02.067663+0000", 2024-08-19T14:19:13.300 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0036444989999999998 2024-08-19T14:19:13.300 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.300 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.300 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:19:13.300 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:16:02.067679+0000", 2024-08-19T14:19:13.300 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6184999999999998e-05 2024-08-19T14:19:13.301 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.301 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.301 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:19:13.301 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:16:02.278283+0000", 2024-08-19T14:19:13.301 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.210604333 2024-08-19T14:19:13.301 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.301 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.301 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:19:13.301 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:16:02.278327+0000", 2024-08-19T14:19:13.301 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.3619999999999999e-05 2024-08-19T14:19:13.301 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.302 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.302 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:19:13.302 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:16:02.278390+0000", 2024-08-19T14:19:13.302 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.3128999999999998e-05 2024-08-19T14:19:13.302 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:13.302 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:19:13.302 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:13.302 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.302 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.302 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25057.0:64631 3.1a 3.4f01fb1a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e246)", 2024-08-19T14:19:13.303 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:17:36.180454+0000", 2024-08-19T14:19:13.303 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 96.997274511000001, 2024-08-19T14:19:13.303 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.78083698899999998, 2024-08-19T14:19:13.303 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:19:13.303 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:19:13.303 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-08-19T14:19:13.303 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:19:13.303 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25057", 2024-08-19T14:19:13.303 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:19:13.303 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 64631 2024-08-19T14:19:13.304 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.304 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:19:13.304 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.304 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:19:13.304 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.180454+0000", 2024-08-19T14:19:13.304 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:13.304 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.304 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.304 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:19:13.304 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.180454+0000", 2024-08-19T14:19:13.304 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:13.304 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.305 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.305 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:19:13.305 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.180456+0000", 2024-08-19T14:19:13.305 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.5220000000000002e-06 2024-08-19T14:19:13.305 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.305 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.305 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:19:13.305 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.180483+0000", 2024-08-19T14:19:13.305 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.6417000000000001e-05 2024-08-19T14:19:13.305 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.305 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.306 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:19:13.306 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.180499+0000", 2024-08-19T14:19:13.306 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5823000000000001e-05 2024-08-19T14:19:13.306 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.306 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.306 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:19:13.306 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.180510+0000", 2024-08-19T14:19:13.306 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1737e-05 2024-08-19T14:19:13.306 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.306 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.306 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:19:13.307 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.180596+0000", 2024-08-19T14:19:13.307 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.6001000000000006e-05 2024-08-19T14:19:13.307 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.307 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.307 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for peered", 2024-08-19T14:19:13.307 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.180606+0000", 2024-08-19T14:19:13.307 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.3470000000000005e-06 2024-08-19T14:19:13.307 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.307 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.307 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:19:13.307 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.961260+0000", 2024-08-19T14:19:13.307 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.78065392099999997 2024-08-19T14:19:13.308 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.308 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.308 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:19:13.308 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.961291+0000", 2024-08-19T14:19:13.308 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.1220999999999998e-05 2024-08-19T14:19:13.308 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:13.308 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:19:13.309 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:13.309 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.309 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.309 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25057.0:64651 3.1a 3.5fb8049a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e246)", 2024-08-19T14:19:13.309 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:17:36.235642+0000", 2024-08-19T14:19:13.309 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 96.942086454000005, 2024-08-19T14:19:13.309 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.72570858100000002, 2024-08-19T14:19:13.309 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:19:13.309 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:19:13.309 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-08-19T14:19:13.309 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:19:13.310 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25057", 2024-08-19T14:19:13.310 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:19:13.310 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 64651 2024-08-19T14:19:13.310 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.310 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:19:13.310 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.310 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:19:13.310 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.235642+0000", 2024-08-19T14:19:13.310 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:13.310 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.310 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.311 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:19:13.311 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.235642+0000", 2024-08-19T14:19:13.311 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:13.311 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.311 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.311 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:19:13.311 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.235644+0000", 2024-08-19T14:19:13.311 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7829999999999999e-06 2024-08-19T14:19:13.311 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.311 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.311 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:19:13.311 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.235677+0000", 2024-08-19T14:19:13.312 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.3333000000000001e-05 2024-08-19T14:19:13.312 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.312 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.312 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:19:13.312 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.235694+0000", 2024-08-19T14:19:13.312 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6626999999999999e-05 2024-08-19T14:19:13.312 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.312 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.312 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:19:13.312 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.235701+0000", 2024-08-19T14:19:13.312 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.7000000000000008e-06 2024-08-19T14:19:13.312 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.313 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.313 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:19:13.313 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.235758+0000", 2024-08-19T14:19:13.313 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7166999999999997e-05 2024-08-19T14:19:13.313 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.313 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.313 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for peered", 2024-08-19T14:19:13.313 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.235766+0000", 2024-08-19T14:19:13.313 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4059999999999999e-06 2024-08-19T14:19:13.313 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.313 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.314 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:19:13.314 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.961337+0000", 2024-08-19T14:19:13.314 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.72557158099999997 2024-08-19T14:19:13.314 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.314 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.314 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:19:13.314 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.961350+0000", 2024-08-19T14:19:13.314 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2984e-05 2024-08-19T14:19:13.314 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:13.314 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:19:13.314 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:13.315 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.315 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.315 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25057.0:64656 3.1a 3.4e375ba (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e246)", 2024-08-19T14:19:13.315 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:17:36.257659+0000", 2024-08-19T14:19:13.315 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 96.920068982000004, 2024-08-19T14:19:13.315 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.70372575500000001, 2024-08-19T14:19:13.315 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:19:13.315 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:19:13.315 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-08-19T14:19:13.315 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:19:13.315 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25057", 2024-08-19T14:19:13.316 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:19:13.316 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 64656 2024-08-19T14:19:13.316 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.316 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:19:13.316 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.316 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:19:13.316 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.257659+0000", 2024-08-19T14:19:13.316 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:13.316 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.316 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.316 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:19:13.316 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.257659+0000", 2024-08-19T14:19:13.316 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:13.317 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.317 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.317 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:19:13.317 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.257661+0000", 2024-08-19T14:19:13.317 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.821e-06 2024-08-19T14:19:13.317 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.317 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.317 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:19:13.317 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.257681+0000", 2024-08-19T14:19:13.317 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0307999999999999e-05 2024-08-19T14:19:13.317 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.317 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.318 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:19:13.318 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.257693+0000", 2024-08-19T14:19:13.318 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1447000000000001e-05 2024-08-19T14:19:13.318 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.318 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.318 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:19:13.318 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.257699+0000", 2024-08-19T14:19:13.318 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7389999999999996e-06 2024-08-19T14:19:13.318 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.318 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.318 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:19:13.319 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.257751+0000", 2024-08-19T14:19:13.319 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2281999999999999e-05 2024-08-19T14:19:13.319 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.319 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.319 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for peered", 2024-08-19T14:19:13.319 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.257759+0000", 2024-08-19T14:19:13.319 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.2390000000000002e-06 2024-08-19T14:19:13.319 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.319 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.319 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:19:13.319 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.961378+0000", 2024-08-19T14:19:13.319 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.70361908699999998 2024-08-19T14:19:13.320 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.320 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.320 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:19:13.320 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.961385+0000", 2024-08-19T14:19:13.320 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.832e-06 2024-08-19T14:19:13.320 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:13.320 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:19:13.320 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:13.320 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.320 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.320 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25057.0:130084 3.9 3:96bca659:::benchmark_data_smithi045_59973_object130083:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e281)", 2024-08-19T14:19:13.320 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:18:31.614605+0000", 2024-08-19T14:19:13.320 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 41.563123656999998, 2024-08-19T14:19:13.321 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.206588472, 2024-08-19T14:19:13.321 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:19:13.321 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:19:13.321 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:19:13.321 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:19:13.321 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25057", 2024-08-19T14:19:13.321 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:19:13.321 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 130084 2024-08-19T14:19:13.321 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.321 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:19:13.321 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.322 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:19:13.322 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:18:31.614605+0000", 2024-08-19T14:19:13.322 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:13.322 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.322 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.322 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:19:13.322 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:18:31.614605+0000", 2024-08-19T14:19:13.322 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:13.322 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.322 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.322 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:19:13.322 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:18:31.614606+0000", 2024-08-19T14:19:13.323 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3960000000000001e-06 2024-08-19T14:19:13.323 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.323 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.323 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:19:13.323 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:18:31.614627+0000", 2024-08-19T14:19:13.323 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1313000000000001e-05 2024-08-19T14:19:13.323 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.323 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.323 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:19:13.323 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:18:31.614638+0000", 2024-08-19T14:19:13.323 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0686000000000001e-05 2024-08-19T14:19:13.324 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.324 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.324 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:19:13.324 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:18:31.614643+0000", 2024-08-19T14:19:13.324 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0690000000000001e-06 2024-08-19T14:19:13.324 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.324 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.324 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:19:13.324 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:18:31.614719+0000", 2024-08-19T14:19:13.324 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.6252000000000001e-05 2024-08-19T14:19:13.324 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.325 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.325 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:19:13.325 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:18:31.614843+0000", 2024-08-19T14:19:13.325 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012340699999999999 2024-08-19T14:19:13.325 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.325 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.325 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 7,11", 2024-08-19T14:19:13.325 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:18:31.614982+0000", 2024-08-19T14:19:13.325 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013878600000000001 2024-08-19T14:19:13.325 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.325 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.325 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:19:13.326 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:18:31.617402+0000", 2024-08-19T14:19:13.326 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00242034 2024-08-19T14:19:13.326 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.326 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.326 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:19:13.326 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:18:31.617596+0000", 2024-08-19T14:19:13.326 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00019408500000000001 2024-08-19T14:19:13.326 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.326 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.326 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:19:13.326 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:18:31.820984+0000", 2024-08-19T14:19:13.327 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20338821900000001 2024-08-19T14:19:13.327 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.327 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.327 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:19:13.327 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:18:31.821156+0000", 2024-08-19T14:19:13.327 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00017137899999999999 2024-08-19T14:19:13.327 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.327 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.327 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:19:13.327 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:18:31.821193+0000", 2024-08-19T14:19:13.327 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.7540000000000003e-05 2024-08-19T14:19:13.328 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:13.328 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:19:13.328 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:13.328 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.328 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.328 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25057.0:163335 3.1d 3:bec78ced:::benchmark_data_smithi045_59973_object32156:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e288)", 2024-08-19T14:19:13.328 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:18:47.673097+0000", 2024-08-19T14:19:13.328 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 25.504630939999998, 2024-08-19T14:19:13.328 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.207015213, 2024-08-19T14:19:13.328 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:19:13.329 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:19:13.329 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:19:13.329 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:19:13.329 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25057", 2024-08-19T14:19:13.329 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:19:13.329 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 163335 2024-08-19T14:19:13.329 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.329 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:19:13.329 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.329 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:19:13.329 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:18:47.673097+0000", 2024-08-19T14:19:13.330 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:13.330 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.330 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.330 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:19:13.330 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:18:47.673097+0000", 2024-08-19T14:19:13.330 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:13.330 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.330 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.330 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:19:13.330 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:18:47.673098+0000", 2024-08-19T14:19:13.330 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.8299999999999995e-07 2024-08-19T14:19:13.330 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.331 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.331 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:19:13.331 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:18:47.673103+0000", 2024-08-19T14:19:13.331 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.2760000000000002e-06 2024-08-19T14:19:13.331 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.331 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.331 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:19:13.331 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:18:47.673115+0000", 2024-08-19T14:19:13.331 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2676e-05 2024-08-19T14:19:13.331 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.331 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.332 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:19:13.332 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:18:47.673122+0000", 2024-08-19T14:19:13.332 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.9990000000000002e-06 2024-08-19T14:19:13.332 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.332 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.332 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:19:13.332 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:18:47.674044+0000", 2024-08-19T14:19:13.332 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00092164999999999999 2024-08-19T14:19:13.332 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.332 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.332 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:19:13.332 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:18:47.674163+0000", 2024-08-19T14:19:13.332 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000119577 2024-08-19T14:19:13.333 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.333 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.333 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 4,5", 2024-08-19T14:19:13.333 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:18:47.674268+0000", 2024-08-19T14:19:13.333 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00010464 2024-08-19T14:19:13.333 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.333 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.333 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:19:13.333 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:18:47.678891+0000", 2024-08-19T14:19:13.333 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0046226419999999997 2024-08-19T14:19:13.333 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.333 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.334 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:19:13.334 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:18:47.735032+0000", 2024-08-19T14:19:13.334 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.056140931999999998 2024-08-19T14:19:13.334 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.334 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.334 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:19:13.334 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:18:47.880041+0000", 2024-08-19T14:19:13.334 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.14500938999999999 2024-08-19T14:19:13.334 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.334 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.334 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:19:13.334 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:18:47.880073+0000", 2024-08-19T14:19:13.335 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.1695000000000001e-05 2024-08-19T14:19:13.335 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:13.335 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:13.335 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:19:13.335 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:18:47.880113+0000", 2024-08-19T14:19:13.335 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.9852999999999999e-05 2024-08-19T14:19:13.335 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:13.335 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:19:13.335 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:13.335 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:13.335 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:19:13.335 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:19:13.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:13.336 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:13.336 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:19:13.337 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:13 smithi045 ceph-mon[20143]: pgmap v679: 77 pgs: 2 active+recovering+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 67 active+clean; 4.2 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 869 op/s; 4/206112 objects degraded (0.002%); 23732/206112 objects misplaced (11.514%); 8.9 MiB/s, 143 objects/s recovering 2024-08-19T14:19:13.437 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:19:13.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:13.459 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:13.459 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:19:13.560 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:19:13.586 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:13.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:13.587 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:19:13.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:13 smithi031 ceph-mon[16336]: pgmap v679: 77 pgs: 2 active+recovering+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 67 active+clean; 4.2 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 869 op/s; 4/206112 objects degraded (0.002%); 23732/206112 objects misplaced (11.514%); 8.9 MiB/s, 143 objects/s recovering 2024-08-19T14:19:13.687 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:19:13.710 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:13.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:13.711 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:19:13.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:13 smithi032 ceph-mon[20004]: pgmap v679: 77 pgs: 2 active+recovering+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 67 active+clean; 4.2 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 869 op/s; 4/206112 objects degraded (0.002%); 23732/206112 objects misplaced (11.514%); 8.9 MiB/s, 143 objects/s recovering 2024-08-19T14:19:13.812 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:19:13.843 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:13.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:13.844 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:19:13.945 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:19:13.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:13.975 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:13.975 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:19:14.075 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:19:14.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:14.178 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:14.178 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:19:14.279 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:19:14.309 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:14.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:14.310 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:19:14.411 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:19:14.448 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:14.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:14.449 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:19:14.550 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:19:14.579 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:14.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:14.580 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:19:14.641 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-08-19T14:19:14.681 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:19:14.719 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:14.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:14.720 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:19:14.821 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:19:14.854 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:14.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:14.855 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:19:14.956 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:19:14.977 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:14.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:14.978 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:19:14.990 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:19:15.079 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:19:15.102 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:15.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:15.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:19:15.204 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:19:15.229 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:15.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:15.230 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:19:15.330 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:19:15.354 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:15.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:15.355 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:19:15.395 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:15 smithi045 ceph-mon[20143]: pgmap v680: 77 pgs: 2 active+recovering+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 67 active+clean; 4.1 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 3.6 MiB/s wr, 1.04k op/s; 4/200631 objects degraded (0.002%); 22147/200631 objects misplaced (11.039%); 12 MiB/s, 198 objects/s recovering 2024-08-19T14:19:15.456 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:19:15.524 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:15.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:15.525 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:19:15.528 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:19:15.529 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [], 2024-08-19T14:19:15.529 INFO:teuthology.orchestra.run.smithi045.stdout: "num_ops": 0 2024-08-19T14:19:15.529 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:19:15.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:15 smithi031 ceph-mon[16336]: pgmap v680: 77 pgs: 2 active+recovering+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 67 active+clean; 4.1 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 3.6 MiB/s wr, 1.04k op/s; 4/200631 objects degraded (0.002%); 22147/200631 objects misplaced (11.039%); 12 MiB/s, 198 objects/s recovering 2024-08-19T14:19:15.627 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:19:15.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:15.670 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:15.671 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:19:15.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:15 smithi032 ceph-mon[20004]: pgmap v680: 77 pgs: 2 active+recovering+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 67 active+clean; 4.1 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 3.6 MiB/s wr, 1.04k op/s; 4/200631 objects degraded (0.002%); 22147/200631 objects misplaced (11.039%); 12 MiB/s, 198 objects/s recovering 2024-08-19T14:19:15.771 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:19:15.800 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:15.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:15.801 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:19:15.902 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:19:15.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:15.926 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:15.927 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:19:16.028 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:19:16.076 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:16.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:16.078 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:19:16.179 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:19:16.205 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:16.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:16.206 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:19:16.298 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-08-19T14:19:16.307 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:19:16.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:16.374 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:16.374 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:19:16.475 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:19:16.505 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:16.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:16.506 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:19:16.607 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:19:16.635 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:16.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:16.636 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:19:16.648 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:19:16.738 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:19:16.762 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:16.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:16.763 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:19:16.864 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:19:16.887 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:16.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:16.888 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:19:16.989 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:19:17.033 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:17.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:17.034 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:19:17.135 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:19:17.172 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:17.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:17.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:19:17.183 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:19:17.183 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [], 2024-08-19T14:19:17.183 INFO:teuthology.orchestra.run.smithi045.stdout: "complaint_time": 30, 2024-08-19T14:19:17.183 INFO:teuthology.orchestra.run.smithi045.stdout: "num_blocked_ops": 0 2024-08-19T14:19:17.183 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:19:17.274 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:19:17.326 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:17.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:17.327 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:19:17.428 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:19:17.473 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:17.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:17.474 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:19:17.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:17 smithi045 ceph-mon[20143]: pgmap v681: 77 pgs: 2 active+recovering+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 67 active+clean; 4.0 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 4.8 MiB/s wr, 1.17k op/s; 4/195330 objects degraded (0.002%); 20488/195330 objects misplaced (10.489%); 17 MiB/s, 273 objects/s recovering 2024-08-19T14:19:17.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:17 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1276201036' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:19:17.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:17 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:19:17.575 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:19:17.609 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:17.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:17.610 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:19:17.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:17 smithi031 ceph-mon[16336]: pgmap v681: 77 pgs: 2 active+recovering+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 67 active+clean; 4.0 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 4.8 MiB/s wr, 1.17k op/s; 4/195330 objects degraded (0.002%); 20488/195330 objects misplaced (10.489%); 17 MiB/s, 273 objects/s recovering 2024-08-19T14:19:17.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:17 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1276201036' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:19:17.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:17 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:19:17.710 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:19:17.736 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:17.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:17.737 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:19:17.744 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is unset 2024-08-19T14:19:17.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:17 smithi032 ceph-mon[20004]: pgmap v681: 77 pgs: 2 active+recovering+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 67 active+clean; 4.0 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 4.8 MiB/s wr, 1.17k op/s; 4/195330 objects degraded (0.002%); 20488/195330 objects misplaced (10.489%); 17 MiB/s, 273 objects/s recovering 2024-08-19T14:19:17.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:17 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1276201036' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:19:17.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:17 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:19:17.838 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:19:17.861 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:17.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:17.862 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:19:17.964 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:19:17.989 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:17.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:17.990 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:19:18.014 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:19:18.091 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:19:18.116 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:18.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:18.117 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:19:18.186 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-08-19T14:19:18.218 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:19:18.253 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:18.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:18.254 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:19:18.355 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:19:18.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:18.419 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:18.419 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:19:18.520 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:19:18.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:18 smithi045 ceph-mon[20143]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:19:18.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:18 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:19:18.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:18 smithi045 ceph-mon[20143]: osdmap e301: 12 total, 12 up, 9 in 2024-08-19T14:19:18.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:18 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 4/195330 objects degraded (0.002%), 2 pgs degraded (PG_DEGRADED) 2024-08-19T14:19:18.550 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:19:18.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:18.565 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:18.565 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:19:18.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:18 smithi031 ceph-mon[16336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:19:18.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:18 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:19:18.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:18 smithi031 ceph-mon[16336]: osdmap e301: 12 total, 12 up, 9 in 2024-08-19T14:19:18.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:18 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 4/195330 objects degraded (0.002%), 2 pgs degraded (PG_DEGRADED) 2024-08-19T14:19:18.666 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:19:18.693 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:18.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:18.694 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:19:18.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:18 smithi032 ceph-mon[20004]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:19:18.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:18 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:19:18.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:18 smithi032 ceph-mon[20004]: osdmap e301: 12 total, 12 up, 9 in 2024-08-19T14:19:18.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:18 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 4/195330 objects degraded (0.002%), 2 pgs degraded (PG_DEGRADED) 2024-08-19T14:19:18.794 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:19:18.821 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:18.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:18.822 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:19:18.923 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:19:18.973 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:18.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:18.974 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:19:19.075 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:19:19.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:19.131 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:19.131 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:19:19.232 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:19:19.273 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:19.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:19.275 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:19:19.376 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:19:19.405 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:19.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:19.407 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:19:19.508 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:19:19.517 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:19 smithi031 ceph-mon[16336]: pgmap v683: 77 pgs: 2 active+recovering+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 67 active+clean; 3.9 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 6.6 MiB/s wr, 1.30k op/s; 4/191697 objects degraded (0.002%); 19666/191697 objects misplaced (10.259%); 20 MiB/s, 314 objects/s recovering 2024-08-19T14:19:19.533 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:19.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:19.534 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:19:19.635 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:19:19.664 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:19.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:19.666 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:19:19.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:19 smithi032 ceph-mon[20004]: pgmap v683: 77 pgs: 2 active+recovering+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 67 active+clean; 3.9 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 6.6 MiB/s wr, 1.30k op/s; 4/191697 objects degraded (0.002%); 19666/191697 objects misplaced (10.259%); 20 MiB/s, 314 objects/s recovering 2024-08-19T14:19:19.768 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:19:19.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:19 smithi045 ceph-mon[20143]: pgmap v683: 77 pgs: 2 active+recovering+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 67 active+clean; 3.9 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 6.6 MiB/s wr, 1.30k op/s; 4/191697 objects degraded (0.002%); 19666/191697 objects misplaced (10.259%); 20 MiB/s, 314 objects/s recovering 2024-08-19T14:19:19.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:19.797 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:19.797 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:19:19.898 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:19:19.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:19.923 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:19.923 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:19:20.024 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:19:20.054 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:20.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:20.055 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:19:20.156 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:19:20.198 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:20.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:20.199 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:19:20.300 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:19:20.332 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:20.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:20.333 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:19:20.434 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:19:20.473 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:20.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:20.475 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:19:20.576 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:19:20.602 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:20.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:20.603 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:19:20.704 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:19:20.735 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:20.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:20.736 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:19:20.838 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:19:20.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:20.867 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:20.867 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:19:20.968 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:19:20.989 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:20.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:20.990 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:19:21.091 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:19:21.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:21.118 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:21.118 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:19:21.219 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:19:21.242 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:21.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:21.246 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:19:21.348 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:19:21.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:21 smithi031 ceph-mon[16336]: pgmap v684: 77 pgs: 2 active+recovering+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 67 active+clean; 3.7 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 904 op/s; 4/184152 objects degraded (0.002%); 16470/184152 objects misplaced (8.944%); 25 MiB/s, 400 objects/s recovering 2024-08-19T14:19:21.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:21 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1052114441' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "41"}]: dispatch 2024-08-19T14:19:21.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:21 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "41"}]: dispatch 2024-08-19T14:19:21.384 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:21.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:21.385 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:19:21.485 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:19:21.518 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:19:21.518 INFO:teuthology.orchestra.run.smithi045.stdout: "size": 20, 2024-08-19T14:19:21.518 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 600, 2024-08-19T14:19:21.519 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [ 2024-08-19T14:19:21.519 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.519 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:115112 2.a e147/102 2:56fe95bd:::benchmark_data_smithi045_46415_object115111:head v 147'7223, mlcod=147'7220)", 2024-08-19T14:19:21.519 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:14:09.954245+0000", 2024-08-19T14:19:21.519 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 310.67603249699999, 2024-08-19T14:19:21.519 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.17428000299999999, 2024-08-19T14:19:21.519 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:19:21.519 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:19:21.519 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:19:21.519 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:19:21.520 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.520 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:19:21.520 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.954245+0000", 2024-08-19T14:19:21.520 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:21.520 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.520 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.520 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:19:21.520 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.954245+0000", 2024-08-19T14:19:21.520 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:21.520 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.520 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.520 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:19:21.520 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.954246+0000", 2024-08-19T14:19:21.521 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.1100000000000005e-07 2024-08-19T14:19:21.521 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.521 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.521 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:19:21.521 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.954272+0000", 2024-08-19T14:19:21.521 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.5551e-05 2024-08-19T14:19:21.521 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.521 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.521 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:19:21.521 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.954284+0000", 2024-08-19T14:19:21.522 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1717e-05 2024-08-19T14:19:21.522 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.522 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.522 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:19:21.522 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.954291+0000", 2024-08-19T14:19:21.522 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.5279999999999998e-06 2024-08-19T14:19:21.522 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.522 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.522 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:19:21.522 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.955487+0000", 2024-08-19T14:19:21.522 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001196428 2024-08-19T14:19:21.522 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.522 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.523 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:19:21.523 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.955510+0000", 2024-08-19T14:19:21.523 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2396e-05 2024-08-19T14:19:21.523 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.523 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.523 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:19:21.523 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:10.128404+0000", 2024-08-19T14:19:21.523 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.17289379499999999 2024-08-19T14:19:21.523 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.523 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.523 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:19:21.523 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:10.128525+0000", 2024-08-19T14:19:21.523 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000121777 2024-08-19T14:19:21.523 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:21.524 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:19:21.524 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:21.524 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.524 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.524 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:115113 2.a e147/102 2:57a3b2e5:::benchmark_data_smithi045_46415_object115112:head v 147'7224, mlcod=147'7220)", 2024-08-19T14:19:21.524 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:14:09.957896+0000", 2024-08-19T14:19:21.524 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 310.672381637, 2024-08-19T14:19:21.524 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.17072668099999999, 2024-08-19T14:19:21.524 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:19:21.524 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:19:21.524 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:19:21.524 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:19:21.524 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.524 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:19:21.525 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.957896+0000", 2024-08-19T14:19:21.525 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:21.525 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.525 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.525 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:19:21.525 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.957896+0000", 2024-08-19T14:19:21.525 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:21.525 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.525 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.525 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:19:21.525 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.957897+0000", 2024-08-19T14:19:21.525 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.8999999999999995e-07 2024-08-19T14:19:21.525 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.525 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.525 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:19:21.526 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.957925+0000", 2024-08-19T14:19:21.526 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.7433e-05 2024-08-19T14:19:21.526 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.526 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.526 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:19:21.526 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.957936+0000", 2024-08-19T14:19:21.526 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1337e-05 2024-08-19T14:19:21.526 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.526 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.526 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:19:21.526 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.957942+0000", 2024-08-19T14:19:21.526 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.6170000000000003e-06 2024-08-19T14:19:21.526 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.526 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.527 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:19:21.527 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.959198+0000", 2024-08-19T14:19:21.527 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0012551490000000001 2024-08-19T14:19:21.527 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.527 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.527 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:19:21.527 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.959218+0000", 2024-08-19T14:19:21.527 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0621000000000001e-05 2024-08-19T14:19:21.527 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.527 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.527 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:19:21.527 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:10.128532+0000", 2024-08-19T14:19:21.527 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.169313305 2024-08-19T14:19:21.528 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.528 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.528 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:19:21.528 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:10.128623+0000", 2024-08-19T14:19:21.528 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.1428999999999995e-05 2024-08-19T14:19:21.528 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:21.528 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:19:21.528 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:21.528 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.528 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.528 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:115120 2.f e147/102 2:f135df33:::benchmark_data_smithi045_46415_object115119:head v 147'7250, mlcod=147'7248)", 2024-08-19T14:19:21.528 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:14:09.960501+0000", 2024-08-19T14:19:21.528 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 310.66977694100001, 2024-08-19T14:19:21.528 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.15682758499999999, 2024-08-19T14:19:21.528 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:19:21.529 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:19:21.529 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:19:21.529 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:19:21.529 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.529 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:19:21.529 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.960501+0000", 2024-08-19T14:19:21.529 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:21.529 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.529 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.529 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:19:21.529 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.960501+0000", 2024-08-19T14:19:21.529 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:21.529 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.529 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.530 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:19:21.530 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.960502+0000", 2024-08-19T14:19:21.530 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.5199999999999995e-07 2024-08-19T14:19:21.530 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.530 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.530 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:19:21.530 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.960531+0000", 2024-08-19T14:19:21.530 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.9034e-05 2024-08-19T14:19:21.530 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.530 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.531 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:19:21.532 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.960541+0000", 2024-08-19T14:19:21.532 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0596e-05 2024-08-19T14:19:21.532 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.532 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.532 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:19:21.532 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.960548+0000", 2024-08-19T14:19:21.532 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.99e-06 2024-08-19T14:19:21.532 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.532 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.532 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:19:21.532 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.960622+0000", 2024-08-19T14:19:21.532 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.3458e-05 2024-08-19T14:19:21.532 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.532 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.533 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:19:21.533 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.960641+0000", 2024-08-19T14:19:21.533 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9032e-05 2024-08-19T14:19:21.533 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.533 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.533 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:19:21.533 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:10.117268+0000", 2024-08-19T14:19:21.533 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.156626723 2024-08-19T14:19:21.533 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.534 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.534 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:19:21.534 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:10.117329+0000", 2024-08-19T14:19:21.534 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.0900000000000003e-05 2024-08-19T14:19:21.534 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:21.534 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:19:21.534 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:21.534 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.534 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.534 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24580.0:115126 2.d 2:b84de1a6:::benchmark_data_smithi045_46415_object115125:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-08-19T14:19:21.534 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:14:09.961680+0000", 2024-08-19T14:19:21.535 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 310.66859771200001, 2024-08-19T14:19:21.535 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.17432350099999999, 2024-08-19T14:19:21.535 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:19:21.535 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:19:21.535 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:19:21.535 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:19:21.535 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24580", 2024-08-19T14:19:21.535 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:19:21.535 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 115126 2024-08-19T14:19:21.535 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.535 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:19:21.535 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.535 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:19:21.535 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.961680+0000", 2024-08-19T14:19:21.536 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:21.536 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.536 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.536 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:19:21.536 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.961680+0000", 2024-08-19T14:19:21.536 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:21.536 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.536 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.536 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:19:21.536 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.961682+0000", 2024-08-19T14:19:21.536 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6959999999999999e-06 2024-08-19T14:19:21.536 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.536 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.536 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:19:21.537 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.961705+0000", 2024-08-19T14:19:21.537 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.3292e-05 2024-08-19T14:19:21.537 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.537 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.537 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:19:21.537 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.961717+0000", 2024-08-19T14:19:21.537 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1976e-05 2024-08-19T14:19:21.537 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.537 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.537 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:19:21.537 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.961725+0000", 2024-08-19T14:19:21.537 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.7519999999999996e-06 2024-08-19T14:19:21.537 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.537 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.537 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:19:21.538 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:10.119167+0000", 2024-08-19T14:19:21.538 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.157442057 2024-08-19T14:19:21.538 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.538 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.538 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:19:21.538 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:10.119308+0000", 2024-08-19T14:19:21.538 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000140798 2024-08-19T14:19:21.538 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.538 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.538 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 3,4", 2024-08-19T14:19:21.538 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:10.119454+0000", 2024-08-19T14:19:21.538 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014580699999999999 2024-08-19T14:19:21.538 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.538 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.539 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:19:21.539 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:10.131417+0000", 2024-08-19T14:19:21.539 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.011963055 2024-08-19T14:19:21.539 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.539 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.539 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:19:21.539 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:10.131458+0000", 2024-08-19T14:19:21.539 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.1291000000000002e-05 2024-08-19T14:19:21.539 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.539 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.539 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:19:21.539 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:10.135887+0000", 2024-08-19T14:19:21.539 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0044292050000000003 2024-08-19T14:19:21.539 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.540 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.540 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:19:21.540 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:10.135942+0000", 2024-08-19T14:19:21.540 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4707999999999999e-05 2024-08-19T14:19:21.540 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.540 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.540 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:19:21.540 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:10.136004+0000", 2024-08-19T14:19:21.540 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.1864000000000003e-05 2024-08-19T14:19:21.540 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:21.540 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:19:21.540 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:21.540 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.540 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.540 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:115124 2.f e147/102 2:f4df7718:::benchmark_data_smithi045_46415_object115123:head v 147'7251, mlcod=147'7248)", 2024-08-19T14:19:21.541 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:14:09.964066+0000", 2024-08-19T14:19:21.541 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 310.66621145099998, 2024-08-19T14:19:21.541 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.15330069800000001, 2024-08-19T14:19:21.541 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:19:21.541 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:19:21.541 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:19:21.541 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:19:21.541 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.541 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:19:21.541 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.964066+0000", 2024-08-19T14:19:21.541 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:21.541 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.541 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.541 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:19:21.542 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.964066+0000", 2024-08-19T14:19:21.542 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:21.542 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.542 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.542 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:19:21.542 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.964067+0000", 2024-08-19T14:19:21.542 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4000000000000001e-07 2024-08-19T14:19:21.542 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.542 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.542 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:19:21.542 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.964100+0000", 2024-08-19T14:19:21.542 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.2298000000000001e-05 2024-08-19T14:19:21.542 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.542 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.543 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:19:21.543 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.964110+0000", 2024-08-19T14:19:21.543 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0193e-05 2024-08-19T14:19:21.543 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.543 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.543 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:19:21.543 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.964117+0000", 2024-08-19T14:19:21.543 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.827e-06 2024-08-19T14:19:21.543 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.543 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.543 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:19:21.543 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.964209+0000", 2024-08-19T14:19:21.543 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.2227999999999999e-05 2024-08-19T14:19:21.543 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.544 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.544 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:19:21.544 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.964228+0000", 2024-08-19T14:19:21.544 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9381999999999998e-05 2024-08-19T14:19:21.544 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.544 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.544 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:19:21.544 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:10.117338+0000", 2024-08-19T14:19:21.544 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.15311018000000001 2024-08-19T14:19:21.544 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.544 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.544 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:19:21.544 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:10.117367+0000", 2024-08-19T14:19:21.544 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.885e-05 2024-08-19T14:19:21.545 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:21.545 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:19:21.545 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:21.545 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.545 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.545 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24580.0:115129 2.d 2:b66d07ae:::benchmark_data_smithi045_46415_object115128:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-08-19T14:19:21.545 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:14:09.965238+0000", 2024-08-19T14:19:21.545 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 310.66504042000003, 2024-08-19T14:19:21.545 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.17130973899999999, 2024-08-19T14:19:21.545 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:19:21.545 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:19:21.545 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:19:21.545 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:19:21.545 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24580", 2024-08-19T14:19:21.545 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:19:21.546 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 115129 2024-08-19T14:19:21.546 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.546 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:19:21.546 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.546 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:19:21.546 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.965238+0000", 2024-08-19T14:19:21.546 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:21.546 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.546 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.546 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:19:21.546 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.965238+0000", 2024-08-19T14:19:21.546 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:21.546 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.546 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.547 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:19:21.547 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.965239+0000", 2024-08-19T14:19:21.547 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8160000000000001e-06 2024-08-19T14:19:21.547 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.547 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.547 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:19:21.547 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.965263+0000", 2024-08-19T14:19:21.547 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.3937000000000001e-05 2024-08-19T14:19:21.547 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.547 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.547 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:19:21.547 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.965275+0000", 2024-08-19T14:19:21.547 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2009e-05 2024-08-19T14:19:21.547 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.548 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.548 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:19:21.548 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.965283+0000", 2024-08-19T14:19:21.548 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4189999999999996e-06 2024-08-19T14:19:21.548 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.548 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.548 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:19:21.548 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:10.122985+0000", 2024-08-19T14:19:21.548 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.157702077 2024-08-19T14:19:21.548 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.548 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.548 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:19:21.548 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:10.123131+0000", 2024-08-19T14:19:21.548 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014671500000000001 2024-08-19T14:19:21.549 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.549 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.549 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 3,4", 2024-08-19T14:19:21.549 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:10.123284+0000", 2024-08-19T14:19:21.549 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015256199999999999 2024-08-19T14:19:21.549 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.549 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.549 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:19:21.549 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:10.131497+0000", 2024-08-19T14:19:21.549 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0082132809999999994 2024-08-19T14:19:21.549 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.549 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.549 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:19:21.549 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:10.135830+0000", 2024-08-19T14:19:21.549 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0043327130000000002 2024-08-19T14:19:21.550 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.550 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.550 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:19:21.550 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:10.136011+0000", 2024-08-19T14:19:21.550 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00018142399999999999 2024-08-19T14:19:21.550 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.550 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.550 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:19:21.550 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:10.136064+0000", 2024-08-19T14:19:21.550 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2139999999999999e-05 2024-08-19T14:19:21.550 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.550 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.550 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:19:21.550 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:10.136547+0000", 2024-08-19T14:19:21.551 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00048364600000000001 2024-08-19T14:19:21.551 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:21.551 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:19:21.551 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:21.551 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.551 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.551 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:115128 2.0 e147/132 2:08e033d8:::benchmark_data_smithi045_46415_object115127:head v 147'7222, mlcod=147'7220)", 2024-08-19T14:19:21.551 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:14:09.965305+0000", 2024-08-19T14:19:21.551 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 310.66497329100002, 2024-08-19T14:19:21.551 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.152300509, 2024-08-19T14:19:21.551 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:19:21.551 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:19:21.551 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:19:21.551 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:19:21.551 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.552 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:19:21.552 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.965305+0000", 2024-08-19T14:19:21.552 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:21.552 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.552 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.552 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:19:21.552 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.965305+0000", 2024-08-19T14:19:21.552 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:21.552 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.552 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.552 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:19:21.552 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.965305+0000", 2024-08-19T14:19:21.552 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2300000000000001e-07 2024-08-19T14:19:21.552 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.552 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.553 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:19:21.553 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.965331+0000", 2024-08-19T14:19:21.553 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.5709999999999999e-05 2024-08-19T14:19:21.553 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.553 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.553 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:19:21.553 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.965339+0000", 2024-08-19T14:19:21.553 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.8199999999999997e-06 2024-08-19T14:19:21.553 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.553 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.553 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:19:21.553 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.965344+0000", 2024-08-19T14:19:21.553 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1449999999999999e-06 2024-08-19T14:19:21.553 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.554 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.554 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:19:21.554 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.965409+0000", 2024-08-19T14:19:21.554 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.5573000000000004e-05 2024-08-19T14:19:21.554 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.554 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.554 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:19:21.554 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.965428+0000", 2024-08-19T14:19:21.554 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9038e-05 2024-08-19T14:19:21.554 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.554 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.554 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:19:21.554 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:10.117559+0000", 2024-08-19T14:19:21.554 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.15213016200000001 2024-08-19T14:19:21.554 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.555 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.555 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:19:21.555 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:10.117605+0000", 2024-08-19T14:19:21.555 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.6538000000000001e-05 2024-08-19T14:19:21.555 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:21.555 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:19:21.555 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:21.555 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.555 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.555 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:115127 2.f e147/102 2:fb596b92:::benchmark_data_smithi045_46415_object115126:head v 147'7252, mlcod=147'7248)", 2024-08-19T14:19:21.555 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:14:09.967625+0000", 2024-08-19T14:19:21.555 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 310.66265338099998, 2024-08-19T14:19:21.555 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.16869712100000001, 2024-08-19T14:19:21.555 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:19:21.556 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:19:21.556 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:19:21.556 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:19:21.556 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.556 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:19:21.556 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.967625+0000", 2024-08-19T14:19:21.556 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:21.556 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.556 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.556 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:19:21.556 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.967625+0000", 2024-08-19T14:19:21.556 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:21.556 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.556 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.557 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:19:21.557 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.967625+0000", 2024-08-19T14:19:21.557 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.5799999999999998e-07 2024-08-19T14:19:21.557 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.557 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.557 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:19:21.557 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.967650+0000", 2024-08-19T14:19:21.557 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.4702000000000002e-05 2024-08-19T14:19:21.557 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.557 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.557 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:19:21.557 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:10.117223+0000", 2024-08-19T14:19:21.557 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.149573331 2024-08-19T14:19:21.557 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.557 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.558 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:19:21.558 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:10.117232+0000", 2024-08-19T14:19:21.558 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.901e-06 2024-08-19T14:19:21.558 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.558 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.558 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:19:21.558 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:10.117397+0000", 2024-08-19T14:19:21.558 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00016443 2024-08-19T14:19:21.558 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.558 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.558 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:19:21.558 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:10.117417+0000", 2024-08-19T14:19:21.558 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0259e-05 2024-08-19T14:19:21.558 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.559 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.559 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:19:21.559 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:10.136126+0000", 2024-08-19T14:19:21.559 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.018709078000000001 2024-08-19T14:19:21.559 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.559 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.559 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:19:21.559 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:10.136322+0000", 2024-08-19T14:19:21.559 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00019566200000000001 2024-08-19T14:19:21.559 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:21.559 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:19:21.559 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:21.559 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.559 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.559 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:115136 2.0 e147/132 2:02df7cb0:::benchmark_data_smithi045_46415_object115135:head v 147'7223, mlcod=147'7220)", 2024-08-19T14:19:21.560 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:14:09.971911+0000", 2024-08-19T14:19:21.560 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 310.65836738199999, 2024-08-19T14:19:21.560 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.152436297, 2024-08-19T14:19:21.560 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:19:21.560 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:19:21.560 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:19:21.560 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:19:21.560 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.560 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:19:21.560 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.971911+0000", 2024-08-19T14:19:21.560 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:21.560 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.560 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.560 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:19:21.560 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.971911+0000", 2024-08-19T14:19:21.561 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:21.561 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.561 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.561 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:19:21.561 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.971911+0000", 2024-08-19T14:19:21.561 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.4600000000000003e-07 2024-08-19T14:19:21.561 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.561 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.561 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:19:21.561 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:09.972026+0000", 2024-08-19T14:19:21.561 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000114629 2024-08-19T14:19:21.561 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.561 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.561 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:19:21.562 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:10.117028+0000", 2024-08-19T14:19:21.562 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.145002193 2024-08-19T14:19:21.562 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.562 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.562 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:19:21.562 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:10.117039+0000", 2024-08-19T14:19:21.562 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0618000000000001e-05 2024-08-19T14:19:21.562 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.562 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.562 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:19:21.562 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:10.117632+0000", 2024-08-19T14:19:21.562 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00059258399999999995 2024-08-19T14:19:21.562 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.562 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.563 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:19:21.563 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:10.117652+0000", 2024-08-19T14:19:21.563 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0975000000000002e-05 2024-08-19T14:19:21.563 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.563 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.563 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:19:21.563 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:10.124240+0000", 2024-08-19T14:19:21.563 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0065879629999999996 2024-08-19T14:19:21.563 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.563 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.563 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:19:21.563 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:10.124347+0000", 2024-08-19T14:19:21.563 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000106389 2024-08-19T14:19:21.563 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:21.564 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:19:21.564 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:21.564 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.564 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.564 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24580.0:116074 2.d 2:b7e42e23:::benchmark_data_smithi045_46415_object116073:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-08-19T14:19:21.564 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:14:11.206338+0000", 2024-08-19T14:19:21.564 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 309.42393970299997, 2024-08-19T14:19:21.564 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.115659096, 2024-08-19T14:19:21.564 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:19:21.564 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:19:21.564 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:19:21.564 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:19:21.564 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24580", 2024-08-19T14:19:21.564 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:19:21.564 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 116074 2024-08-19T14:19:21.565 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.565 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:19:21.565 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.565 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:19:21.565 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:11.206338+0000", 2024-08-19T14:19:21.565 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:21.565 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.565 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.565 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:19:21.565 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:11.206338+0000", 2024-08-19T14:19:21.565 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:21.565 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.565 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.565 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:19:21.566 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:11.206340+0000", 2024-08-19T14:19:21.566 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.672e-06 2024-08-19T14:19:21.566 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.566 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.566 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:19:21.566 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:11.206363+0000", 2024-08-19T14:19:21.566 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2929e-05 2024-08-19T14:19:21.566 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.566 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.566 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:19:21.566 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:11.206376+0000", 2024-08-19T14:19:21.566 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3044000000000001e-05 2024-08-19T14:19:21.566 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.566 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.567 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:19:21.567 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:11.206384+0000", 2024-08-19T14:19:21.567 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.8089999999999993e-06 2024-08-19T14:19:21.567 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.567 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.567 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:19:21.567 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:11.314413+0000", 2024-08-19T14:19:21.567 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.108029721 2024-08-19T14:19:21.567 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.567 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.567 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:19:21.567 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:11.314553+0000", 2024-08-19T14:19:21.567 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013989800000000001 2024-08-19T14:19:21.567 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.568 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.568 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 3,4", 2024-08-19T14:19:21.568 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:11.314711+0000", 2024-08-19T14:19:21.568 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015746099999999999 2024-08-19T14:19:21.568 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.568 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.568 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:19:21.568 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:11.321791+0000", 2024-08-19T14:19:21.568 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0070803680000000001 2024-08-19T14:19:21.568 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.568 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.568 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:19:21.568 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:11.321804+0000", 2024-08-19T14:19:21.568 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2408999999999999e-05 2024-08-19T14:19:21.568 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.569 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.569 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:19:21.569 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:11.321866+0000", 2024-08-19T14:19:21.569 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.2206999999999996e-05 2024-08-19T14:19:21.569 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.569 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.569 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:19:21.569 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:11.321937+0000", 2024-08-19T14:19:21.569 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.1352000000000004e-05 2024-08-19T14:19:21.569 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.569 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.569 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:19:21.569 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:11.321997+0000", 2024-08-19T14:19:21.569 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.0226000000000002e-05 2024-08-19T14:19:21.570 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:21.570 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:19:21.570 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:21.570 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.570 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.570 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24580.0:116732 2.d 2:bd0455f5:::benchmark_data_smithi045_46415_object116731:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-08-19T14:19:21.570 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:14:12.006637+0000", 2024-08-19T14:19:21.570 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 308.62364072600002, 2024-08-19T14:19:21.570 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.10503797400000001, 2024-08-19T14:19:21.570 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:19:21.570 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:19:21.570 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:19:21.570 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:19:21.570 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24580", 2024-08-19T14:19:21.571 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:19:21.571 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 116732 2024-08-19T14:19:21.571 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.571 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:19:21.571 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.571 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:19:21.571 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:12.006637+0000", 2024-08-19T14:19:21.571 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:21.571 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.571 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.571 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:19:21.571 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:12.006637+0000", 2024-08-19T14:19:21.571 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:21.571 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.572 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.572 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:19:21.572 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:12.006638+0000", 2024-08-19T14:19:21.572 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.8100000000000001e-07 2024-08-19T14:19:21.572 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.572 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.572 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:19:21.572 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:12.006659+0000", 2024-08-19T14:19:21.572 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1398000000000001e-05 2024-08-19T14:19:21.572 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.572 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.572 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:19:21.572 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:12.006670+0000", 2024-08-19T14:19:21.572 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0903999999999999e-05 2024-08-19T14:19:21.572 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.573 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.573 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:19:21.573 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:12.006677+0000", 2024-08-19T14:19:21.573 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.8210000000000004e-06 2024-08-19T14:19:21.573 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.573 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.573 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:19:21.573 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:12.091573+0000", 2024-08-19T14:19:21.573 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.084895913000000003 2024-08-19T14:19:21.573 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.573 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.573 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:19:21.573 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:12.091715+0000", 2024-08-19T14:19:21.573 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000141416 2024-08-19T14:19:21.573 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.574 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.574 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 3,4", 2024-08-19T14:19:21.574 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:12.091964+0000", 2024-08-19T14:19:21.574 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00024988000000000001 2024-08-19T14:19:21.574 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.574 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.574 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:19:21.574 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:12.100726+0000", 2024-08-19T14:19:21.574 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.008761507 2024-08-19T14:19:21.574 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.574 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.574 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:19:21.574 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:12.103900+0000", 2024-08-19T14:19:21.574 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.003173978 2024-08-19T14:19:21.575 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.575 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.575 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:19:21.575 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:12.111566+0000", 2024-08-19T14:19:21.575 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0076658960000000002 2024-08-19T14:19:21.575 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.575 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.575 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:19:21.575 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:12.111627+0000", 2024-08-19T14:19:21.575 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.0887999999999997e-05 2024-08-19T14:19:21.575 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.575 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.575 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:19:21.575 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:12.111675+0000", 2024-08-19T14:19:21.576 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.8492000000000003e-05 2024-08-19T14:19:21.576 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:21.576 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:19:21.576 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:21.576 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.576 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.576 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25057.0:50174 3.10 3.bde586f0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-08-19T14:19:21.576 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:17:20.749720+0000", 2024-08-19T14:19:21.576 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 119.880558045, 2024-08-19T14:19:21.576 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.99597661999999998, 2024-08-19T14:19:21.576 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:19:21.576 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:19:21.576 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-08-19T14:19:21.576 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:19:21.577 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25057", 2024-08-19T14:19:21.577 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:19:21.577 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 50174 2024-08-19T14:19:21.577 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.577 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:19:21.577 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.577 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:19:21.577 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:20.749720+0000", 2024-08-19T14:19:21.577 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:21.577 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.577 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.577 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:19:21.577 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:20.749720+0000", 2024-08-19T14:19:21.577 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:21.577 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.578 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.578 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:19:21.578 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:20.749722+0000", 2024-08-19T14:19:21.578 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.79e-06 2024-08-19T14:19:21.578 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.578 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.578 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:19:21.578 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:20.749747+0000", 2024-08-19T14:19:21.578 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.4967e-05 2024-08-19T14:19:21.578 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.578 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.578 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:19:21.578 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:20.749867+0000", 2024-08-19T14:19:21.578 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000120388 2024-08-19T14:19:21.579 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.579 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.579 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:19:21.579 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:20.749875+0000", 2024-08-19T14:19:21.579 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.1470000000000001e-06 2024-08-19T14:19:21.579 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.579 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.579 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:19:21.579 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:20.749980+0000", 2024-08-19T14:19:21.579 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000104744 2024-08-19T14:19:21.579 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.579 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.579 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for peered", 2024-08-19T14:19:21.579 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:20.749991+0000", 2024-08-19T14:19:21.580 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1014e-05 2024-08-19T14:19:21.580 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.580 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.580 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:19:21.580 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:21.745658+0000", 2024-08-19T14:19:21.580 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.99566752700000005 2024-08-19T14:19:21.580 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.580 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.580 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:19:21.580 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:21.745696+0000", 2024-08-19T14:19:21.580 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.8043000000000002e-05 2024-08-19T14:19:21.580 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:21.580 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:19:21.580 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:21.581 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.581 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.581 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25057.0:50174 3.10 3:0f61a7bd:::benchmark_data_smithi045_59973_object50173:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e238)", 2024-08-19T14:19:21.581 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:17:22.794782+0000", 2024-08-19T14:19:21.581 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 117.835495947, 2024-08-19T14:19:21.581 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.131475602, 2024-08-19T14:19:21.581 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:19:21.581 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:19:21.581 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:19:21.581 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:19:21.581 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25057", 2024-08-19T14:19:21.581 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:19:21.581 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 50174 2024-08-19T14:19:21.581 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.582 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:19:21.582 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.582 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:19:21.582 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:22.794782+0000", 2024-08-19T14:19:21.582 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:21.582 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.582 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.582 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:19:21.582 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:22.794782+0000", 2024-08-19T14:19:21.582 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:21.582 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.582 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.582 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:19:21.582 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:22.794784+0000", 2024-08-19T14:19:21.583 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.4509999999999999e-06 2024-08-19T14:19:21.583 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.583 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.583 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:19:21.583 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:22.794819+0000", 2024-08-19T14:19:21.583 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.4592999999999999e-05 2024-08-19T14:19:21.583 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.583 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.583 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:19:21.583 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:22.794834+0000", 2024-08-19T14:19:21.583 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5082999999999999e-05 2024-08-19T14:19:21.583 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.583 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.583 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:19:21.584 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:22.794846+0000", 2024-08-19T14:19:21.584 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1772e-05 2024-08-19T14:19:21.584 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.584 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.584 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:19:21.584 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:22.796046+0000", 2024-08-19T14:19:21.584 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0012004120000000001 2024-08-19T14:19:21.584 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.584 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.584 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for peered", 2024-08-19T14:19:21.584 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:22.796062+0000", 2024-08-19T14:19:21.584 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5707000000000001e-05 2024-08-19T14:19:21.584 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.584 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.584 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:19:21.585 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:22.915080+0000", 2024-08-19T14:19:21.585 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.11901813999999999 2024-08-19T14:19:21.585 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.585 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.585 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:19:21.585 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:22.915183+0000", 2024-08-19T14:19:21.585 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00010237 2024-08-19T14:19:21.585 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.585 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.585 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 3,5", 2024-08-19T14:19:21.585 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:22.915289+0000", 2024-08-19T14:19:21.585 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000106056 2024-08-19T14:19:21.585 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.585 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.585 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:19:21.586 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:22.916302+0000", 2024-08-19T14:19:21.586 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0010135700000000001 2024-08-19T14:19:21.586 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.586 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.586 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:19:21.586 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:22.917212+0000", 2024-08-19T14:19:21.586 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00091027600000000001 2024-08-19T14:19:21.586 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.586 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.586 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:19:21.586 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:22.925306+0000", 2024-08-19T14:19:21.586 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0080939759999999993 2024-08-19T14:19:21.586 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.586 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.587 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:19:21.587 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:22.925350+0000", 2024-08-19T14:19:21.587 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.3990999999999999e-05 2024-08-19T14:19:21.587 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.587 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.587 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:19:21.587 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:22.926258+0000", 2024-08-19T14:19:21.587 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00090720500000000001 2024-08-19T14:19:21.587 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:21.587 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:19:21.587 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:21.587 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.587 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.587 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25057.0:63341 3.22 3:44f22295:::benchmark_data_smithi045_59973_object63340:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e244)", 2024-08-19T14:19:21.587 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:17:34.014807+0000", 2024-08-19T14:19:21.588 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 106.615470675, 2024-08-19T14:19:21.588 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.97840119400000003, 2024-08-19T14:19:21.588 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:19:21.588 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:19:21.588 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:19:21.588 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:19:21.588 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25057", 2024-08-19T14:19:21.588 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:19:21.588 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 63341 2024-08-19T14:19:21.588 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.588 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:19:21.588 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.588 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:19:21.588 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:34.014807+0000", 2024-08-19T14:19:21.589 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:21.589 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.589 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.589 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:19:21.589 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:34.014807+0000", 2024-08-19T14:19:21.589 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:21.589 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.589 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.589 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:19:21.589 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:34.014809+0000", 2024-08-19T14:19:21.589 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.46e-06 2024-08-19T14:19:21.589 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.589 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.589 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:19:21.590 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:34.014834+0000", 2024-08-19T14:19:21.590 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.4854999999999998e-05 2024-08-19T14:19:21.590 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.590 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.590 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:19:21.590 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:34.014846+0000", 2024-08-19T14:19:21.590 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2258e-05 2024-08-19T14:19:21.590 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.590 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.590 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:19:21.590 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:34.014851+0000", 2024-08-19T14:19:21.590 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2959999999999998e-06 2024-08-19T14:19:21.590 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.590 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.591 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:19:21.591 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:34.014906+0000", 2024-08-19T14:19:21.591 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4738999999999999e-05 2024-08-19T14:19:21.591 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.591 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.591 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for peered", 2024-08-19T14:19:21.591 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:34.014913+0000", 2024-08-19T14:19:21.591 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.1590000000000004e-06 2024-08-19T14:19:21.591 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.591 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.591 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:19:21.591 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:34.989215+0000", 2024-08-19T14:19:21.591 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.97430243299999997 2024-08-19T14:19:21.591 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.592 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.592 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:19:21.592 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:34.989336+0000", 2024-08-19T14:19:21.592 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000120977 2024-08-19T14:19:21.592 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.592 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.592 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 4,5", 2024-08-19T14:19:21.592 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:34.989467+0000", 2024-08-19T14:19:21.592 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000130495 2024-08-19T14:19:21.592 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.592 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.592 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:19:21.592 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:34.990225+0000", 2024-08-19T14:19:21.592 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00075812000000000004 2024-08-19T14:19:21.593 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.593 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.593 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:19:21.593 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:34.990711+0000", 2024-08-19T14:19:21.593 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000485522 2024-08-19T14:19:21.593 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.593 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.593 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:19:21.593 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:34.993150+0000", 2024-08-19T14:19:21.593 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.002439049 2024-08-19T14:19:21.593 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.593 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.593 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:19:21.593 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:34.993184+0000", 2024-08-19T14:19:21.593 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.4035000000000002e-05 2024-08-19T14:19:21.594 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.594 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.594 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:19:21.594 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:34.993208+0000", 2024-08-19T14:19:21.594 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.4796000000000001e-05 2024-08-19T14:19:21.594 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:21.594 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:19:21.594 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:21.594 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.594 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.594 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25057.0:64615 3.20 3.18b2560 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e246)", 2024-08-19T14:19:21.594 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:17:35.982300+0000", 2024-08-19T14:19:21.594 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 104.64797836299999, 2024-08-19T14:19:21.594 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.98262199699999997, 2024-08-19T14:19:21.595 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:19:21.595 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:19:21.595 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-08-19T14:19:21.595 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:19:21.595 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25057", 2024-08-19T14:19:21.595 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:19:21.595 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 64615 2024-08-19T14:19:21.595 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.595 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:19:21.595 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.595 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:19:21.595 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:35.982300+0000", 2024-08-19T14:19:21.595 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:21.595 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.596 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.596 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:19:21.596 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:35.982300+0000", 2024-08-19T14:19:21.596 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:21.596 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.596 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.596 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:19:21.596 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:35.982301+0000", 2024-08-19T14:19:21.596 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.316e-06 2024-08-19T14:19:21.596 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.596 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.596 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:19:21.596 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:35.982323+0000", 2024-08-19T14:19:21.596 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1747e-05 2024-08-19T14:19:21.597 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.597 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.597 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:19:21.597 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:35.982336+0000", 2024-08-19T14:19:21.597 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3315e-05 2024-08-19T14:19:21.597 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.597 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.597 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:19:21.597 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:35.982343+0000", 2024-08-19T14:19:21.597 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4880000000000001e-06 2024-08-19T14:19:21.597 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.597 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.597 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:19:21.597 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:35.982388+0000", 2024-08-19T14:19:21.598 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.4397000000000002e-05 2024-08-19T14:19:21.598 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.598 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.598 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for peered", 2024-08-19T14:19:21.598 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:35.982394+0000", 2024-08-19T14:19:21.598 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.2809999999999997e-06 2024-08-19T14:19:21.598 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.598 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.598 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:19:21.598 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.964911+0000", 2024-08-19T14:19:21.598 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.98251731499999995 2024-08-19T14:19:21.598 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.598 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.598 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:19:21.598 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.964922+0000", 2024-08-19T14:19:21.599 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0138e-05 2024-08-19T14:19:21.599 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:21.599 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:19:21.599 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:21.599 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.599 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.599 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25057.0:64628 3.10 3:0cdbc0d9:::benchmark_data_smithi045_59973_object64627:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e246)", 2024-08-19T14:19:21.599 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:17:36.006171+0000", 2024-08-19T14:19:21.599 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 104.62410656599999, 2024-08-19T14:19:21.599 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.17738801200000001, 2024-08-19T14:19:21.599 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:19:21.599 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:19:21.599 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:19:21.599 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:19:21.600 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25057", 2024-08-19T14:19:21.600 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:19:21.600 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 64628 2024-08-19T14:19:21.600 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.600 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:19:21.600 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.600 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:19:21.600 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.006171+0000", 2024-08-19T14:19:21.600 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:21.600 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.600 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.600 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:19:21.600 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.006171+0000", 2024-08-19T14:19:21.600 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:21.601 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.601 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.601 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:19:21.601 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.006173+0000", 2024-08-19T14:19:21.601 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.666e-06 2024-08-19T14:19:21.601 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.601 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.601 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:19:21.601 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.006203+0000", 2024-08-19T14:19:21.601 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.0429e-05 2024-08-19T14:19:21.601 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.601 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.601 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:19:21.601 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.006218+0000", 2024-08-19T14:19:21.601 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4464000000000001e-05 2024-08-19T14:19:21.602 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.602 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.602 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:19:21.602 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.006225+0000", 2024-08-19T14:19:21.602 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.3339999999999996e-06 2024-08-19T14:19:21.602 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.602 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.602 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:19:21.602 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.006307+0000", 2024-08-19T14:19:21.602 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.1846000000000002e-05 2024-08-19T14:19:21.602 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.602 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.602 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:19:21.602 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.006462+0000", 2024-08-19T14:19:21.603 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000154709 2024-08-19T14:19:21.603 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.603 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.603 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 3,5", 2024-08-19T14:19:21.603 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.006645+0000", 2024-08-19T14:19:21.603 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00018337900000000001 2024-08-19T14:19:21.603 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.604 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.604 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:19:21.604 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.007701+0000", 2024-08-19T14:19:21.604 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001056174 2024-08-19T14:19:21.604 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.604 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.604 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:19:21.604 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.009427+0000", 2024-08-19T14:19:21.604 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0017257419999999999 2024-08-19T14:19:21.604 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.604 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.604 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:19:21.604 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.183128+0000", 2024-08-19T14:19:21.604 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.17370068499999999 2024-08-19T14:19:21.605 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.605 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.605 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:19:21.605 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.183177+0000", 2024-08-19T14:19:21.605 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.9302999999999999e-05 2024-08-19T14:19:21.605 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.605 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.605 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:19:21.605 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.183559+0000", 2024-08-19T14:19:21.605 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00038228099999999999 2024-08-19T14:19:21.605 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:21.605 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:19:21.605 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:21.605 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.606 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.606 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25057.0:64615 3.20 3.18b2560 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e246)", 2024-08-19T14:19:21.606 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:17:36.245637+0000", 2024-08-19T14:19:21.606 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 104.384640968, 2024-08-19T14:19:21.606 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.71969014899999995, 2024-08-19T14:19:21.606 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:19:21.606 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:19:21.606 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-08-19T14:19:21.606 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:19:21.606 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25057", 2024-08-19T14:19:21.606 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:19:21.607 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 64615 2024-08-19T14:19:21.607 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.607 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:19:21.607 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.607 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:19:21.607 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.245637+0000", 2024-08-19T14:19:21.607 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:21.607 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.607 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.607 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:19:21.608 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.245637+0000", 2024-08-19T14:19:21.608 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:21.608 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.608 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.608 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:19:21.608 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.245638+0000", 2024-08-19T14:19:21.608 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1340000000000001e-06 2024-08-19T14:19:21.608 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.608 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.608 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:19:21.608 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.245665+0000", 2024-08-19T14:19:21.608 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.7107e-05 2024-08-19T14:19:21.608 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.608 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.609 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:19:21.609 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.245678+0000", 2024-08-19T14:19:21.609 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2911999999999999e-05 2024-08-19T14:19:21.609 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.609 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.609 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:19:21.609 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.245686+0000", 2024-08-19T14:19:21.609 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.8120000000000008e-06 2024-08-19T14:19:21.609 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.609 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.609 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:19:21.609 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.245751+0000", 2024-08-19T14:19:21.609 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.4845999999999995e-05 2024-08-19T14:19:21.609 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.609 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.610 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for peered", 2024-08-19T14:19:21.610 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.245767+0000", 2024-08-19T14:19:21.610 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6708e-05 2024-08-19T14:19:21.610 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.610 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.610 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:19:21.610 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.965166+0000", 2024-08-19T14:19:21.610 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.71939893200000005 2024-08-19T14:19:21.610 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.610 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.610 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:19:21.610 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.965327+0000", 2024-08-19T14:19:21.610 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000160698 2024-08-19T14:19:21.611 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:21.611 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:19:21.611 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:21.611 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.611 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.611 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25057.0:74596 3.10 3:0ba948e1:::benchmark_data_smithi045_59973_object74595:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e258)", 2024-08-19T14:19:21.611 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:17:51.788236+0000", 2024-08-19T14:19:21.611 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 88.842042413000001, 2024-08-19T14:19:21.611 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.12760691499999999, 2024-08-19T14:19:21.611 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:19:21.611 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:19:21.611 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:19:21.611 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:19:21.611 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25057", 2024-08-19T14:19:21.611 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:19:21.612 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 74596 2024-08-19T14:19:21.612 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.612 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:19:21.612 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.612 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:19:21.612 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:51.788236+0000", 2024-08-19T14:19:21.612 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:21.612 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.612 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.612 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:19:21.612 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:51.788236+0000", 2024-08-19T14:19:21.612 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:21.612 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.612 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.613 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:19:21.613 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:51.788237+0000", 2024-08-19T14:19:21.613 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.201e-06 2024-08-19T14:19:21.613 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.613 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.613 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:19:21.613 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:51.788257+0000", 2024-08-19T14:19:21.613 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0559000000000001e-05 2024-08-19T14:19:21.613 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.613 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.613 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:19:21.613 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:51.788270+0000", 2024-08-19T14:19:21.613 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2999999999999999e-05 2024-08-19T14:19:21.613 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.613 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.614 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:19:21.614 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:51.788277+0000", 2024-08-19T14:19:21.614 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.849e-06 2024-08-19T14:19:21.614 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.614 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.614 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:19:21.614 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:51.788314+0000", 2024-08-19T14:19:21.614 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.7144000000000002e-05 2024-08-19T14:19:21.614 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.614 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.614 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:19:21.614 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:51.788399+0000", 2024-08-19T14:19:21.614 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.4584000000000002e-05 2024-08-19T14:19:21.614 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.615 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.615 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 3,5", 2024-08-19T14:19:21.615 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:51.788510+0000", 2024-08-19T14:19:21.615 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00011137 2024-08-19T14:19:21.615 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.615 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.615 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:19:21.615 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:51.789617+0000", 2024-08-19T14:19:21.615 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001106452 2024-08-19T14:19:21.615 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.615 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.615 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:19:21.615 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:51.792157+0000", 2024-08-19T14:19:21.615 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.002539913 2024-08-19T14:19:21.615 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.616 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.616 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:19:21.616 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:51.915730+0000", 2024-08-19T14:19:21.616 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.12357322499999999 2024-08-19T14:19:21.616 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.616 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.616 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:19:21.616 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:51.915810+0000", 2024-08-19T14:19:21.616 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.0377000000000006e-05 2024-08-19T14:19:21.616 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.616 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.616 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:19:21.616 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:51.915842+0000", 2024-08-19T14:19:21.616 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.2240999999999998e-05 2024-08-19T14:19:21.617 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:21.617 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:19:21.617 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:21.617 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.617 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.617 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25057.0:198788 3.2d 3:b70bb268:::benchmark_data_smithi045_59973_object67609:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e300)", 2024-08-19T14:19:21.617 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:19:14.911166+0000", 2024-08-19T14:19:21.617 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 5.7191115620000001, 2024-08-19T14:19:21.617 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.119812131, 2024-08-19T14:19:21.617 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:19:21.617 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:19:21.617 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:19:21.617 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:19:21.617 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25057", 2024-08-19T14:19:21.618 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:19:21.618 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 198788 2024-08-19T14:19:21.618 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.618 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:19:21.618 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.618 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:19:21.618 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:14.911166+0000", 2024-08-19T14:19:21.618 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:21.618 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.618 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.618 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:19:21.618 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:14.911166+0000", 2024-08-19T14:19:21.619 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:21.619 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.619 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.619 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:19:21.619 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:14.911168+0000", 2024-08-19T14:19:21.619 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2419999999999999e-06 2024-08-19T14:19:21.619 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.619 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.619 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:19:21.619 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:14.911173+0000", 2024-08-19T14:19:21.619 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0710000000000002e-06 2024-08-19T14:19:21.619 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.619 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.619 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:19:21.620 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:14.911185+0000", 2024-08-19T14:19:21.620 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1851999999999999e-05 2024-08-19T14:19:21.620 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.620 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.620 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:19:21.620 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:14.911191+0000", 2024-08-19T14:19:21.620 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.8290000000000002e-06 2024-08-19T14:19:21.620 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.620 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.620 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:19:21.620 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:14.917273+0000", 2024-08-19T14:19:21.620 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0060812139999999997 2024-08-19T14:19:21.620 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.620 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.621 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:19:21.621 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:14.917425+0000", 2024-08-19T14:19:21.621 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015265600000000001 2024-08-19T14:19:21.621 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.621 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.621 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,1", 2024-08-19T14:19:21.621 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:14.917559+0000", 2024-08-19T14:19:21.621 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013383199999999999 2024-08-19T14:19:21.621 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.621 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.621 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:19:21.621 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:14.919724+0000", 2024-08-19T14:19:21.621 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0021649339999999999 2024-08-19T14:19:21.621 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.622 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.622 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:19:21.622 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:14.922275+0000", 2024-08-19T14:19:21.622 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0025514919999999998 2024-08-19T14:19:21.622 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.622 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.622 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:19:21.622 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:15.030121+0000", 2024-08-19T14:19:21.622 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.107845911 2024-08-19T14:19:21.622 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.622 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.622 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:19:21.622 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:15.030186+0000", 2024-08-19T14:19:21.622 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.4787000000000002e-05 2024-08-19T14:19:21.623 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.623 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.623 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:19:21.623 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:15.030978+0000", 2024-08-19T14:19:21.623 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00079231099999999997 2024-08-19T14:19:21.623 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:21.623 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:19:21.623 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:21.623 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.623 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.623 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25057.0:198798 3.2d 3:b554e9c5:::benchmark_data_smithi045_59973_object67619:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e300)", 2024-08-19T14:19:21.623 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:19:14.920334+0000", 2024-08-19T14:19:21.623 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 5.7099435789999999, 2024-08-19T14:19:21.623 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.111162898, 2024-08-19T14:19:21.624 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:19:21.624 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:19:21.624 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:19:21.624 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:19:21.624 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25057", 2024-08-19T14:19:21.624 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:19:21.624 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 198798 2024-08-19T14:19:21.624 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.624 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:19:21.624 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.624 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:19:21.624 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:14.920334+0000", 2024-08-19T14:19:21.624 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:21.624 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.624 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.625 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:19:21.625 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:14.920334+0000", 2024-08-19T14:19:21.625 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:21.625 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.625 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.625 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:19:21.625 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:14.920336+0000", 2024-08-19T14:19:21.625 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.308e-06 2024-08-19T14:19:21.625 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.625 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.625 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:19:21.625 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:14.920342+0000", 2024-08-19T14:19:21.625 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.1449999999999996e-06 2024-08-19T14:19:21.626 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.626 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.626 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:19:21.626 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:14.920354+0000", 2024-08-19T14:19:21.626 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2615e-05 2024-08-19T14:19:21.626 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.626 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.626 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:19:21.626 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:14.920361+0000", 2024-08-19T14:19:21.626 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.2920000000000001e-06 2024-08-19T14:19:21.626 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.626 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.626 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:19:21.627 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:14.920414+0000", 2024-08-19T14:19:21.627 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.3776000000000001e-05 2024-08-19T14:19:21.627 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.627 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.627 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:19:21.627 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:14.920573+0000", 2024-08-19T14:19:21.627 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015845099999999999 2024-08-19T14:19:21.627 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.627 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.627 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,1", 2024-08-19T14:19:21.627 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:14.920716+0000", 2024-08-19T14:19:21.627 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014305599999999999 2024-08-19T14:19:21.627 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.627 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.627 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:19:21.628 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:14.922382+0000", 2024-08-19T14:19:21.628 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0016662859999999999 2024-08-19T14:19:21.628 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.628 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.628 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:19:21.628 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:14.925637+0000", 2024-08-19T14:19:21.628 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0032550740000000002 2024-08-19T14:19:21.628 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.628 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.628 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:19:21.628 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:15.031032+0000", 2024-08-19T14:19:21.628 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.10539465000000001 2024-08-19T14:19:21.628 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.628 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.628 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:19:21.629 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:15.031068+0000", 2024-08-19T14:19:21.629 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.6087000000000002e-05 2024-08-19T14:19:21.629 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:21.629 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:21.629 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:19:21.629 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:15.031497+0000", 2024-08-19T14:19:21.629 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00042915800000000001 2024-08-19T14:19:21.629 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:21.629 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:19:21.629 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:21.629 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:21.629 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:19:21.629 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:19:21.630 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-19T14:19:21.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:21.631 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:21.632 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:19:21.733 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:19:21.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:21 smithi032 ceph-mon[20004]: pgmap v684: 77 pgs: 2 active+recovering+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 67 active+clean; 3.7 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 904 op/s; 4/184152 objects degraded (0.002%); 16470/184152 objects misplaced (8.944%); 25 MiB/s, 400 objects/s recovering 2024-08-19T14:19:21.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:21 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1052114441' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "41"}]: dispatch 2024-08-19T14:19:21.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:21 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "41"}]: dispatch 2024-08-19T14:19:21.763 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:21.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:21.764 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:19:21.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:21 smithi045 ceph-mon[20143]: pgmap v684: 77 pgs: 2 active+recovering+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 67 active+clean; 3.7 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 904 op/s; 4/184152 objects degraded (0.002%); 16470/184152 objects misplaced (8.944%); 25 MiB/s, 400 objects/s recovering 2024-08-19T14:19:21.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:21 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1052114441' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "41"}]: dispatch 2024-08-19T14:19:21.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:21 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "41"}]: dispatch 2024-08-19T14:19:21.852 INFO:teuthology.orchestra.run.smithi031.stderr:Error EINVAL: specified pg_num 41 < pg_num_max 32 2024-08-19T14:19:21.865 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:19:21.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:21.943 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:21.943 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:19:22.044 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:19:22.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:22.098 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:22.098 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:19:22.199 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:19:22.225 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:22.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:22.226 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:19:22.326 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:19:22.352 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:22.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:22.353 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:19:22.454 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:19:22.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:22.483 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:22.483 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:19:22.584 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:19:22.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:22.609 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:22.609 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:19:22.710 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:19:22.733 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:22.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:22.734 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:19:22.835 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:19:22.864 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:22.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:22.865 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:19:22.966 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:19:22.992 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:22.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:22.993 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:19:23.094 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:19:23.120 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:23.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:23.121 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:19:23.222 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:19:23.245 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:23.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:23.246 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:19:23.347 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:19:23.374 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:23.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:23.375 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:19:23.476 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:19:23.516 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:23.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:23.517 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:19:23.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:23 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:19:23.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:23 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:19:23.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:23 smithi031 ceph-mon[16336]: pgmap v685: 77 pgs: 2 active+recovering+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 67 active+clean; 3.6 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 954 op/s; 4/177480 objects degraded (0.002%); 15187/177480 objects misplaced (8.557%); 27 MiB/s, 428 objects/s recovering 2024-08-19T14:19:23.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:23 smithi031 ceph-mon[16336]: osdmap e302: 12 total, 12 up, 9 in 2024-08-19T14:19:23.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:23 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 4/177480 objects degraded (0.002%), 2 pgs degraded (PG_DEGRADED) 2024-08-19T14:19:23.619 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:19:23.641 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:23.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:23.642 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:19:23.743 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:19:23.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:23 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:19:23.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:23 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:19:23.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:23 smithi032 ceph-mon[20004]: pgmap v685: 77 pgs: 2 active+recovering+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 67 active+clean; 3.6 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 954 op/s; 4/177480 objects degraded (0.002%); 15187/177480 objects misplaced (8.557%); 27 MiB/s, 428 objects/s recovering 2024-08-19T14:19:23.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:23 smithi032 ceph-mon[20004]: osdmap e302: 12 total, 12 up, 9 in 2024-08-19T14:19:23.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:23 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 4/177480 objects degraded (0.002%), 2 pgs degraded (PG_DEGRADED) 2024-08-19T14:19:23.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:23.788 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:23.788 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:19:23.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:23 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:19:23.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:23 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:19:23.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:23 smithi045 ceph-mon[20143]: pgmap v685: 77 pgs: 2 active+recovering+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 67 active+clean; 3.6 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 954 op/s; 4/177480 objects degraded (0.002%); 15187/177480 objects misplaced (8.557%); 27 MiB/s, 428 objects/s recovering 2024-08-19T14:19:23.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:23 smithi045 ceph-mon[20143]: osdmap e302: 12 total, 12 up, 9 in 2024-08-19T14:19:23.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:23 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 4/177480 objects degraded (0.002%), 2 pgs degraded (PG_DEGRADED) 2024-08-19T14:19:23.889 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:19:23.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:23.912 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:23.912 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:19:24.014 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:19:24.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:24.036 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:24.037 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:19:24.137 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:19:24.186 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:24.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:24.189 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:19:24.235 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:19:24.290 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:19:24.361 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:24.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:24.362 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:19:24.463 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:19:24.496 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:24.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:24.497 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:19:24.598 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:19:24.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:24.637 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:24.637 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:19:24.737 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:19:24.770 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:24.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:24.771 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:19:24.872 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:19:24.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:24.915 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:24.915 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:19:25.016 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:19:25.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:25.061 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:25.062 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:19:25.162 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:19:25.185 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:25.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:25.186 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:19:25.287 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:19:25.314 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:25.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:25.315 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:19:25.416 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:19:25.454 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:25.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:25.455 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:19:25.556 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:19:25.585 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:25.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:25.586 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:19:25.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:25 smithi031 ceph-mon[16336]: pgmap v687: 77 pgs: 2 active+recovering+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 67 active+clean; 3.5 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 972 op/s; 4/171987 objects degraded (0.002%); 13370/171987 objects misplaced (7.774%); 26 MiB/s, 419 objects/s recovering 2024-08-19T14:19:25.661 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:25 smithi045 ceph-mon[20143]: pgmap v687: 77 pgs: 2 active+recovering+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 67 active+clean; 3.5 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 972 op/s; 4/171987 objects degraded (0.002%); 13370/171987 objects misplaced (7.774%); 26 MiB/s, 419 objects/s recovering 2024-08-19T14:19:25.686 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:19:25.727 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:25.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:25.731 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:19:25.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:25 smithi032 ceph-mon[20004]: pgmap v687: 77 pgs: 2 active+recovering+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 67 active+clean; 3.5 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 972 op/s; 4/171987 objects degraded (0.002%); 13370/171987 objects misplaced (7.774%); 26 MiB/s, 419 objects/s recovering 2024-08-19T14:19:25.831 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:19:25.867 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:25.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:25.868 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:19:25.969 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:19:25.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:25.991 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:25.992 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:19:26.093 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:19:26.117 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:26.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:26.119 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:19:26.220 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:19:26.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:26.266 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:26.267 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:19:26.370 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:19:26.390 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:26.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:26.392 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:19:26.493 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:19:26.528 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:26.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:26.529 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:19:26.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:26 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:19:26.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:26 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:19:26.629 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:19:26.671 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:26.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:26.673 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:19:26.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:26 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:19:26.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:26 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:19:26.773 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:19:26.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:26 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:19:26.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:26 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:19:26.799 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:26.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:26.801 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:19:26.902 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:19:26.940 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:26.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:26.941 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:19:27.042 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:19:27.065 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:27.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:27.066 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:19:27.167 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:19:27.211 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:27.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:27.212 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:19:27.313 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:19:27.334 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:27.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:27.335 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:19:27.436 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:19:27.465 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:27.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:27.466 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:19:27.567 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:19:27.599 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:27.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:27.600 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:19:27.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:27 smithi031 ceph-mon[16336]: pgmap v688: 77 pgs: 2 active+recovering+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 67 active+clean; 3.4 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 957 op/s; 4/167070 objects degraded (0.002%); 12015/167070 objects misplaced (7.192%); 27 MiB/s, 493 objects/s recovering 2024-08-19T14:19:27.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:27 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:19:27.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:27 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:19:27.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:27 smithi031 ceph-mon[16336]: osdmap e303: 12 total, 12 up, 9 in 2024-08-19T14:19:27.701 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:19:27.726 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:27.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:27.727 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:19:27.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:27 smithi032 ceph-mon[20004]: pgmap v688: 77 pgs: 2 active+recovering+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 67 active+clean; 3.4 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 957 op/s; 4/167070 objects degraded (0.002%); 12015/167070 objects misplaced (7.192%); 27 MiB/s, 493 objects/s recovering 2024-08-19T14:19:27.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:27 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:19:27.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:27 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:19:27.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:27 smithi032 ceph-mon[20004]: osdmap e303: 12 total, 12 up, 9 in 2024-08-19T14:19:27.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:27 smithi045 ceph-mon[20143]: pgmap v688: 77 pgs: 2 active+recovering+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 67 active+clean; 3.4 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 957 op/s; 4/167070 objects degraded (0.002%); 12015/167070 objects misplaced (7.192%); 27 MiB/s, 493 objects/s recovering 2024-08-19T14:19:27.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:27 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:19:27.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:27 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:19:27.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:27 smithi045 ceph-mon[20143]: osdmap e303: 12 total, 12 up, 9 in 2024-08-19T14:19:27.828 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:19:27.872 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:27.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:27.874 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:19:27.975 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:19:27.997 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:27.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:27.998 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:19:28.099 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:19:28.128 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:28.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:28.129 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:19:28.230 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:19:28.259 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:28.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:28.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:19:28.361 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:19:28.385 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:28.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:28.386 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:19:28.487 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:19:28.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:28.511 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:28.511 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:19:28.612 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:19:28.643 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:28.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:28.644 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:19:28.745 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:19:28.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:28 smithi032 ceph-mon[20004]: osdmap e304: 12 total, 12 up, 9 in 2024-08-19T14:19:28.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:28 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 4/167070 objects degraded (0.002%), 2 pgs degraded (PG_DEGRADED) 2024-08-19T14:19:28.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:28.776 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:28.777 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:19:28.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:28 smithi045 ceph-mon[20143]: osdmap e304: 12 total, 12 up, 9 in 2024-08-19T14:19:28.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:28 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 4/167070 objects degraded (0.002%), 2 pgs degraded (PG_DEGRADED) 2024-08-19T14:19:28.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:28 smithi031 ceph-mon[16336]: osdmap e304: 12 total, 12 up, 9 in 2024-08-19T14:19:28.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:28 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 4/167070 objects degraded (0.002%), 2 pgs degraded (PG_DEGRADED) 2024-08-19T14:19:28.877 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:19:28.906 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:28.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:28.907 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:19:28.935 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-08-19T14:19:29.008 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:19:29.032 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:29.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:29.033 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:19:29.133 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:19:29.158 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:29.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:29.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:19:29.260 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:19:29.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:29.288 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:29.288 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:19:29.372 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:19:29.389 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:19:29.430 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:29.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:29.431 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:19:29.532 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:19:29.577 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:29.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:29.578 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:19:29.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:29 smithi031 ceph-mon[16336]: pgmap v691: 77 pgs: 2 active+recovering+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 67 active+clean; 3.3 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 769 op/s; 4/163629 objects degraded (0.002%); 11099/163629 objects misplaced (6.783%); 19 MiB/s, 407 objects/s recovering 2024-08-19T14:19:29.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:29 smithi031 ceph-mon[16336]: osdmap e305: 12 total, 12 up, 9 in 2024-08-19T14:19:29.660 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:29 smithi045 ceph-mon[20143]: pgmap v691: 77 pgs: 2 active+recovering+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 67 active+clean; 3.3 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 769 op/s; 4/163629 objects degraded (0.002%); 11099/163629 objects misplaced (6.783%); 19 MiB/s, 407 objects/s recovering 2024-08-19T14:19:29.660 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:29 smithi045 ceph-mon[20143]: osdmap e305: 12 total, 12 up, 9 in 2024-08-19T14:19:29.680 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:19:29.704 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:29.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:29.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:19:29.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:29 smithi032 ceph-mon[20004]: pgmap v691: 77 pgs: 2 active+recovering+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 67 active+clean; 3.3 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 769 op/s; 4/163629 objects degraded (0.002%); 11099/163629 objects misplaced (6.783%); 19 MiB/s, 407 objects/s recovering 2024-08-19T14:19:29.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:29 smithi032 ceph-mon[20004]: osdmap e305: 12 total, 12 up, 9 in 2024-08-19T14:19:29.807 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:19:29.834 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:29.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:29.835 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:19:29.936 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:19:29.947 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:19:29.947 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [], 2024-08-19T14:19:29.947 INFO:teuthology.orchestra.run.smithi045.stdout: "num_ops": 0 2024-08-19T14:19:29.947 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:19:29.957 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:29.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:29.958 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:19:30.059 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:19:30.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:30.081 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:30.082 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:19:30.183 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:19:30.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:30.211 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:30.211 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:19:30.312 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:19:30.342 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:30.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:30.344 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:19:30.445 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:19:30.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:30.471 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:30.471 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:19:30.572 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:19:30.600 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:30.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:30.601 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:19:30.702 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:19:30.732 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:30.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:30.733 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:19:30.834 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:19:30.858 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:30.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:30.860 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:19:30.961 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:19:30.991 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:30.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:30.992 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:19:31.093 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:19:31.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:31.134 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:31.134 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:19:31.235 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:19:31.258 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:31.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:31.259 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:19:31.290 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-19T14:19:31.359 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:19:31.387 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:31.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:31.388 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:19:31.488 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:19:31.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:31.519 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:31.520 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:19:31.555 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:19:31.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:31 smithi031 ceph-mon[16336]: pgmap v693: 77 pgs: 3 active+recovering+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 68 active+clean; 3.2 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 832 op/s; 4/156990 objects degraded (0.003%); 8172/156990 objects misplaced (5.205%); 23 MiB/s, 570 objects/s recovering 2024-08-19T14:19:31.620 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:19:31.647 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:31 smithi045 ceph-mon[20143]: pgmap v693: 77 pgs: 3 active+recovering+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 68 active+clean; 3.2 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 832 op/s; 4/156990 objects degraded (0.003%); 8172/156990 objects misplaced (5.205%); 23 MiB/s, 570 objects/s recovering 2024-08-19T14:19:31.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:31.648 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:31.648 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:19:31.675 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-08-19T14:19:31.749 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:19:31.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:31 smithi032 ceph-mon[20004]: pgmap v693: 77 pgs: 3 active+recovering+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 68 active+clean; 3.2 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 832 op/s; 4/156990 objects degraded (0.003%); 8172/156990 objects misplaced (5.205%); 23 MiB/s, 570 objects/s recovering 2024-08-19T14:19:31.774 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:31.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:31.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:19:31.876 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:19:31.918 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:31.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:31.919 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:19:32.017 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:19:32.020 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:19:32.048 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:32.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:32.049 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:19:32.150 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:19:32.178 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:32.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:32.179 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:19:32.280 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:19:32.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:32.323 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:32.323 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:19:32.424 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:19:32.464 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:32.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:32.465 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:19:32.566 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:19:32.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:32.601 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:32.602 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:19:32.609 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:19:32.609 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [], 2024-08-19T14:19:32.609 INFO:teuthology.orchestra.run.smithi045.stdout: "complaint_time": 30, 2024-08-19T14:19:32.609 INFO:teuthology.orchestra.run.smithi045.stdout: "num_blocked_ops": 0 2024-08-19T14:19:32.609 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:19:32.703 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:19:32.729 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:32.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:32.731 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:19:32.831 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:19:32.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:32.862 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:32.862 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:19:32.964 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:19:32.987 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:32.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:32.988 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:19:33.089 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:19:33.117 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:33.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:33.118 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:19:33.219 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:19:33.246 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:33.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:33.247 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:19:33.290 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-08-19T14:19:33.348 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:19:33.391 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:33.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:33.392 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:19:33.493 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:19:33.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:33.533 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:33.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:19:33.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:33 smithi045 ceph-mon[20143]: pgmap v694: 77 pgs: 3 active+recovering+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 68 active+clean; 3.1 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 910 op/s; 4/150675 objects degraded (0.003%); 7058/150675 objects misplaced (4.684%); 21 MiB/s, 526 objects/s recovering 2024-08-19T14:19:33.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:33 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 4/150675 objects degraded (0.003%), 3 pgs degraded (PG_DEGRADED) 2024-08-19T14:19:33.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:33 smithi045 ceph-mon[20143]: osdmap e306: 12 total, 12 up, 9 in 2024-08-19T14:19:33.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:33 smithi031 ceph-mon[16336]: pgmap v694: 77 pgs: 3 active+recovering+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 68 active+clean; 3.1 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 910 op/s; 4/150675 objects degraded (0.003%); 7058/150675 objects misplaced (4.684%); 21 MiB/s, 526 objects/s recovering 2024-08-19T14:19:33.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:33 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 4/150675 objects degraded (0.003%), 3 pgs degraded (PG_DEGRADED) 2024-08-19T14:19:33.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:33 smithi031 ceph-mon[16336]: osdmap e306: 12 total, 12 up, 9 in 2024-08-19T14:19:33.624 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:19:33.634 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:19:33.670 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:33.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:33.671 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:19:33.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:33 smithi032 ceph-mon[20004]: pgmap v694: 77 pgs: 3 active+recovering+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 68 active+clean; 3.1 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 910 op/s; 4/150675 objects degraded (0.003%); 7058/150675 objects misplaced (4.684%); 21 MiB/s, 526 objects/s recovering 2024-08-19T14:19:33.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:33 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 4/150675 objects degraded (0.003%), 3 pgs degraded (PG_DEGRADED) 2024-08-19T14:19:33.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:33 smithi032 ceph-mon[20004]: osdmap e306: 12 total, 12 up, 9 in 2024-08-19T14:19:33.772 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:19:33.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:33.811 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:33.811 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:19:33.912 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:19:33.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:33.945 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:33.945 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:19:34.046 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:19:34.069 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:34.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:34.070 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:19:34.168 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:19:34.169 INFO:teuthology.orchestra.run.smithi045.stdout: "size": 20, 2024-08-19T14:19:34.169 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 600, 2024-08-19T14:19:34.169 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [ 2024-08-19T14:19:34.169 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.169 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:14 2.4 e106/102 2:291e9767:::benchmark_data_smithi045_46415_object13:head v 106'1, mlcod=0'0)", 2024-08-19T14:19:34.169 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:12:49.835194+0000", 2024-08-19T14:19:34.169 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 404.31675961799999, 2024-08-19T14:19:34.169 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.045178087999999998, 2024-08-19T14:19:34.169 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:19:34.169 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:19:34.169 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:19:34.169 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:19:34.170 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.170 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:19:34.170 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:49.835194+0000", 2024-08-19T14:19:34.170 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:34.170 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.170 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.170 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:19:34.170 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:49.835194+0000", 2024-08-19T14:19:34.170 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:34.170 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.170 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.170 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:19:34.171 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:49.835194+0000", 2024-08-19T14:19:34.171 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.7500000000000001e-07 2024-08-19T14:19:34.171 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.171 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.171 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:19:34.171 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:49.879644+0000", 2024-08-19T14:19:34.171 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.044449610000000001 2024-08-19T14:19:34.171 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.171 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.171 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:19:34.171 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:49.879665+0000", 2024-08-19T14:19:34.171 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1458999999999999e-05 2024-08-19T14:19:34.171 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.171 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.172 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:19:34.172 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:49.879706+0000", 2024-08-19T14:19:34.172 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.0748999999999999e-05 2024-08-19T14:19:34.172 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.172 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.172 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:19:34.172 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:49.879782+0000", 2024-08-19T14:19:34.172 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.6445000000000004e-05 2024-08-19T14:19:34.172 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.172 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.172 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:19:34.172 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:49.879807+0000", 2024-08-19T14:19:34.172 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.4765000000000001e-05 2024-08-19T14:19:34.172 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.173 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.173 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:19:34.173 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:49.880329+0000", 2024-08-19T14:19:34.173 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00052147599999999999 2024-08-19T14:19:34.173 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.173 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.173 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:19:34.173 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:49.880372+0000", 2024-08-19T14:19:34.173 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.3309000000000002e-05 2024-08-19T14:19:34.173 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:34.173 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:19:34.173 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:34.173 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.173 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.174 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:12156 2.4 e107/102 2:28b3607b:::benchmark_data_smithi045_46415_object12155:head v 107'793, mlcod=107'791)", 2024-08-19T14:19:34.174 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:12:53.026834+0000", 2024-08-19T14:19:34.174 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 401.12511915300001, 2024-08-19T14:19:34.174 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0078985359999999994, 2024-08-19T14:19:34.174 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:19:34.174 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:19:34.174 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:19:34.174 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:19:34.174 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.174 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:19:34.174 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:53.026834+0000", 2024-08-19T14:19:34.174 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:34.174 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.174 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.175 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:19:34.175 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:53.026834+0000", 2024-08-19T14:19:34.175 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:34.175 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.175 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.175 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:19:34.175 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:53.026835+0000", 2024-08-19T14:19:34.175 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.5199999999999996e-07 2024-08-19T14:19:34.175 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.175 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.175 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:19:34.175 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:53.034134+0000", 2024-08-19T14:19:34.175 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0072991369999999998 2024-08-19T14:19:34.175 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.176 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.176 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:19:34.176 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:53.034144+0000", 2024-08-19T14:19:34.176 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0370000000000001e-05 2024-08-19T14:19:34.176 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.176 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.176 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:19:34.176 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:53.034149+0000", 2024-08-19T14:19:34.176 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.8130000000000003e-06 2024-08-19T14:19:34.176 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.176 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.176 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:19:34.176 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:53.034196+0000", 2024-08-19T14:19:34.176 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.7159e-05 2024-08-19T14:19:34.176 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.177 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.177 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:19:34.177 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:53.034211+0000", 2024-08-19T14:19:34.177 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4429e-05 2024-08-19T14:19:34.177 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.177 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.177 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:19:34.177 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:53.034701+0000", 2024-08-19T14:19:34.177 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00049043000000000003 2024-08-19T14:19:34.177 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.177 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.177 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:19:34.177 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:53.034733+0000", 2024-08-19T14:19:34.177 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.1545999999999999e-05 2024-08-19T14:19:34.178 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:34.178 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:19:34.178 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:34.178 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.178 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.178 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:38741 2.4 e112/102 2:2ddcb16b:::benchmark_data_smithi045_46415_object38740:head v 112'2448, mlcod=112'2444)", 2024-08-19T14:19:34.178 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:02.435487+0000", 2024-08-19T14:19:34.178 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 391.71646672600002, 2024-08-19T14:19:34.178 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0060987159999999997, 2024-08-19T14:19:34.178 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:19:34.178 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:19:34.178 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:19:34.178 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:19:34.178 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.179 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:19:34.179 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.435487+0000", 2024-08-19T14:19:34.179 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:34.179 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.179 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.179 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:19:34.179 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.435487+0000", 2024-08-19T14:19:34.179 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:34.179 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.179 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.179 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:19:34.179 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.435487+0000", 2024-08-19T14:19:34.179 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2699999999999999e-07 2024-08-19T14:19:34.179 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.179 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.180 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:19:34.180 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.435514+0000", 2024-08-19T14:19:34.180 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.6457000000000001e-05 2024-08-19T14:19:34.180 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.180 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.180 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:19:34.180 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.435524+0000", 2024-08-19T14:19:34.180 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0299000000000001e-05 2024-08-19T14:19:34.180 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.180 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.180 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:19:34.180 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.435531+0000", 2024-08-19T14:19:34.180 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.6529999999999997e-06 2024-08-19T14:19:34.180 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.181 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.181 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:19:34.181 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.435603+0000", 2024-08-19T14:19:34.181 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.1963000000000002e-05 2024-08-19T14:19:34.181 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.181 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.181 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:19:34.181 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.435622+0000", 2024-08-19T14:19:34.181 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8926000000000001e-05 2024-08-19T14:19:34.181 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.181 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.181 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:19:34.181 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.441541+0000", 2024-08-19T14:19:34.181 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0059191479999999999 2024-08-19T14:19:34.181 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.182 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.182 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:19:34.182 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.441585+0000", 2024-08-19T14:19:34.182 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.4743000000000002e-05 2024-08-19T14:19:34.182 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:34.182 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:19:34.182 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:34.182 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.182 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.182 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:40033 2.0 e113/102 2:0512d183:::benchmark_data_smithi045_46415_object40032:head v 113'2436, mlcod=113'2434)", 2024-08-19T14:19:34.182 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:02.895855+0000", 2024-08-19T14:19:34.182 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 391.256098033, 2024-08-19T14:19:34.182 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0095757759999999994, 2024-08-19T14:19:34.183 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:19:34.183 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:19:34.183 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:19:34.183 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:19:34.183 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.183 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:19:34.183 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.895855+0000", 2024-08-19T14:19:34.183 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:34.183 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.183 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.183 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:19:34.183 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.895855+0000", 2024-08-19T14:19:34.183 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:34.183 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.183 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.184 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:19:34.184 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.895856+0000", 2024-08-19T14:19:34.184 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.2099999999999995e-07 2024-08-19T14:19:34.184 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.184 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.184 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:19:34.184 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.895922+0000", 2024-08-19T14:19:34.184 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.5188e-05 2024-08-19T14:19:34.184 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.184 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.184 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:19:34.184 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.895935+0000", 2024-08-19T14:19:34.184 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3601000000000001e-05 2024-08-19T14:19:34.184 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.185 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.185 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:19:34.185 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.895943+0000", 2024-08-19T14:19:34.185 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.9170000000000006e-06 2024-08-19T14:19:34.185 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.185 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.185 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:19:34.185 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.896012+0000", 2024-08-19T14:19:34.185 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.9053000000000004e-05 2024-08-19T14:19:34.185 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.185 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.185 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:19:34.185 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.896035+0000", 2024-08-19T14:19:34.185 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2943999999999999e-05 2024-08-19T14:19:34.186 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.186 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.186 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:19:34.186 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.905344+0000", 2024-08-19T14:19:34.186 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0093086450000000008 2024-08-19T14:19:34.186 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.186 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.186 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:19:34.186 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.905431+0000", 2024-08-19T14:19:34.186 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.7507000000000001e-05 2024-08-19T14:19:34.186 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:34.186 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:19:34.186 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:34.187 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.187 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.187 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:40035 2.0 e113/102 2:0960ca99:::benchmark_data_smithi045_46415_object40034:head v 113'2437, mlcod=113'2434)", 2024-08-19T14:19:34.187 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:02.897909+0000", 2024-08-19T14:19:34.187 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 391.25404495399999, 2024-08-19T14:19:34.187 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0076402179999999998, 2024-08-19T14:19:34.187 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:19:34.187 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:19:34.187 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:19:34.187 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:19:34.187 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.187 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:19:34.187 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.897909+0000", 2024-08-19T14:19:34.187 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:34.188 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.188 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.188 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:19:34.188 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.897909+0000", 2024-08-19T14:19:34.188 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:34.188 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.188 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.188 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:19:34.188 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.897909+0000", 2024-08-19T14:19:34.188 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.33e-07 2024-08-19T14:19:34.188 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.188 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.188 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:19:34.188 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.897950+0000", 2024-08-19T14:19:34.189 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.0683000000000003e-05 2024-08-19T14:19:34.189 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.189 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.189 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:19:34.189 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.897966+0000", 2024-08-19T14:19:34.189 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6209999999999999e-05 2024-08-19T14:19:34.189 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.189 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.189 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:19:34.189 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.897974+0000", 2024-08-19T14:19:34.189 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4569999999999999e-06 2024-08-19T14:19:34.189 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.189 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.189 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:19:34.189 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.898035+0000", 2024-08-19T14:19:34.190 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.1033000000000003e-05 2024-08-19T14:19:34.190 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.190 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.190 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:19:34.190 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.898068+0000", 2024-08-19T14:19:34.190 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.3593000000000002e-05 2024-08-19T14:19:34.190 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.190 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.190 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:19:34.190 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.905435+0000", 2024-08-19T14:19:34.190 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0073662830000000004 2024-08-19T14:19:34.190 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.190 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.190 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:19:34.191 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.905549+0000", 2024-08-19T14:19:34.191 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000114026 2024-08-19T14:19:34.191 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:34.191 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:19:34.191 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:34.191 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.191 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.191 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:40069 2.0 e113/102 2:03e60c7a:::benchmark_data_smithi045_46415_object40068:head v 113'2440, mlcod=113'2436)", 2024-08-19T14:19:34.191 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:02.908487+0000", 2024-08-19T14:19:34.191 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 391.24346618099997, 2024-08-19T14:19:34.191 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0062491439999999999, 2024-08-19T14:19:34.191 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:19:34.191 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:19:34.191 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:19:34.191 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:19:34.192 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.192 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:19:34.192 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.908487+0000", 2024-08-19T14:19:34.192 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:34.192 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.192 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.192 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:19:34.192 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.908487+0000", 2024-08-19T14:19:34.192 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:34.192 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.192 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.192 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:19:34.192 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.908488+0000", 2024-08-19T14:19:34.192 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.8999999999999995e-07 2024-08-19T14:19:34.193 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.193 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.193 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:19:34.193 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.908517+0000", 2024-08-19T14:19:34.193 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.8643999999999999e-05 2024-08-19T14:19:34.193 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.193 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.193 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:19:34.193 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.908533+0000", 2024-08-19T14:19:34.193 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5770999999999999e-05 2024-08-19T14:19:34.193 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.193 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.193 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:19:34.193 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.908540+0000", 2024-08-19T14:19:34.193 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4000000000000003e-06 2024-08-19T14:19:34.194 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.194 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.194 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:19:34.194 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.908593+0000", 2024-08-19T14:19:34.194 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.3146999999999998e-05 2024-08-19T14:19:34.194 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.194 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.194 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:19:34.194 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.908614+0000", 2024-08-19T14:19:34.194 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0900999999999999e-05 2024-08-19T14:19:34.194 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.194 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.194 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:19:34.194 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.914692+0000", 2024-08-19T14:19:34.194 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0060776399999999996 2024-08-19T14:19:34.195 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.195 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.195 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:19:34.195 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.914736+0000", 2024-08-19T14:19:34.195 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.4851000000000001e-05 2024-08-19T14:19:34.195 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:34.195 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:19:34.195 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:34.195 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.196 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.196 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:43484 2.4 e114/102 2:285d10cd:::benchmark_data_smithi045_46415_object43483:head v 114'2732, mlcod=114'2729)", 2024-08-19T14:19:34.196 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:04.598813+0000", 2024-08-19T14:19:34.196 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 389.55314029599998, 2024-08-19T14:19:34.196 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0062693649999999998, 2024-08-19T14:19:34.196 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:19:34.196 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:19:34.196 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:19:34.196 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:19:34.196 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.196 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:19:34.196 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:04.598813+0000", 2024-08-19T14:19:34.196 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:34.196 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.197 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.197 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:19:34.197 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:04.598813+0000", 2024-08-19T14:19:34.197 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:34.197 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.197 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.197 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:19:34.197 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:04.598814+0000", 2024-08-19T14:19:34.197 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.9699999999999994e-07 2024-08-19T14:19:34.197 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.197 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.197 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:19:34.197 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:04.598850+0000", 2024-08-19T14:19:34.197 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.5865000000000003e-05 2024-08-19T14:19:34.197 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.198 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.198 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:19:34.198 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:04.598861+0000", 2024-08-19T14:19:34.198 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1284000000000001e-05 2024-08-19T14:19:34.198 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.198 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.198 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:19:34.198 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:04.598870+0000", 2024-08-19T14:19:34.198 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.9479999999999997e-06 2024-08-19T14:19:34.198 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.198 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.198 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:19:34.198 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:04.598935+0000", 2024-08-19T14:19:34.199 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.4364999999999998e-05 2024-08-19T14:19:34.199 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.199 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.199 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:19:34.199 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:04.598955+0000", 2024-08-19T14:19:34.199 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0849999999999999e-05 2024-08-19T14:19:34.199 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.199 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.199 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:19:34.199 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:04.604957+0000", 2024-08-19T14:19:34.199 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0060020000000000004 2024-08-19T14:19:34.199 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.199 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.199 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:19:34.200 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:04.605083+0000", 2024-08-19T14:19:34.200 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012505599999999999 2024-08-19T14:19:34.200 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:34.200 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:19:34.200 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:34.200 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.200 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.200 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:46495 2.4 e116/102 2:27583565:::benchmark_data_smithi045_46415_object46494:head v 116'2919, mlcod=116'2914)", 2024-08-19T14:19:34.200 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:06.037606+0000", 2024-08-19T14:19:34.200 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 388.11434789999998, 2024-08-19T14:19:34.200 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0079993879999999996, 2024-08-19T14:19:34.200 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:19:34.200 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:19:34.200 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:19:34.200 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:19:34.201 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.201 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:19:34.201 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:06.037606+0000", 2024-08-19T14:19:34.201 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:34.201 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.201 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.201 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:19:34.201 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:06.037606+0000", 2024-08-19T14:19:34.201 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:34.201 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.201 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.201 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:19:34.201 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:06.037606+0000", 2024-08-19T14:19:34.201 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2300000000000001e-07 2024-08-19T14:19:34.202 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.202 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.202 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:19:34.202 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:06.043150+0000", 2024-08-19T14:19:34.202 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0055434869999999997 2024-08-19T14:19:34.202 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.202 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.202 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:19:34.202 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:06.043160+0000", 2024-08-19T14:19:34.202 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0088999999999999e-05 2024-08-19T14:19:34.202 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.202 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.202 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:19:34.202 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:06.043167+0000", 2024-08-19T14:19:34.203 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.3309999999999998e-06 2024-08-19T14:19:34.203 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.203 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.203 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:19:34.203 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:06.043225+0000", 2024-08-19T14:19:34.203 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.8062000000000001e-05 2024-08-19T14:19:34.203 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.203 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.203 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:19:34.203 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:06.043246+0000", 2024-08-19T14:19:34.203 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1316000000000001e-05 2024-08-19T14:19:34.203 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.203 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.203 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:19:34.204 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:06.045577+0000", 2024-08-19T14:19:34.204 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0023308890000000001 2024-08-19T14:19:34.204 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.204 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.204 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:19:34.204 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:06.045605+0000", 2024-08-19T14:19:34.204 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.7691e-05 2024-08-19T14:19:34.204 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:34.204 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:19:34.204 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:34.204 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.204 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.204 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:48912 2.7 e117/102 2:e6ffd652:::benchmark_data_smithi045_46415_object48911:head v 117'2997, mlcod=117'2992)", 2024-08-19T14:19:34.204 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:07.468103+0000", 2024-08-19T14:19:34.204 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 386.68385096899999, 2024-08-19T14:19:34.205 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0061113349999999999, 2024-08-19T14:19:34.205 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:19:34.205 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:19:34.205 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:19:34.205 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:19:34.205 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.205 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:19:34.205 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468103+0000", 2024-08-19T14:19:34.205 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:34.205 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.205 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.205 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:19:34.205 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468103+0000", 2024-08-19T14:19:34.205 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:34.206 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.206 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.206 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:19:34.206 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468103+0000", 2024-08-19T14:19:34.206 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.8000000000000002e-07 2024-08-19T14:19:34.206 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.206 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.206 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:19:34.206 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468136+0000", 2024-08-19T14:19:34.206 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.3411000000000002e-05 2024-08-19T14:19:34.206 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.206 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.206 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:19:34.206 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468143+0000", 2024-08-19T14:19:34.207 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.5869999999999998e-06 2024-08-19T14:19:34.207 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.207 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.207 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:19:34.207 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468146+0000", 2024-08-19T14:19:34.207 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.304e-06 2024-08-19T14:19:34.207 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.207 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.207 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:19:34.207 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.470630+0000", 2024-08-19T14:19:34.207 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0024838009999999999 2024-08-19T14:19:34.207 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.207 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.207 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:19:34.207 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.470643+0000", 2024-08-19T14:19:34.208 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2964e-05 2024-08-19T14:19:34.208 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.208 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.208 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:19:34.208 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.474173+0000", 2024-08-19T14:19:34.208 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.003530423 2024-08-19T14:19:34.208 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.208 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.208 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:19:34.208 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.474214+0000", 2024-08-19T14:19:34.208 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.0565000000000002e-05 2024-08-19T14:19:34.208 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:34.208 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:19:34.208 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:34.209 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.209 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.209 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:48971 2.7 e117/102 2:ecf6a539:::benchmark_data_smithi045_46415_object48970:head v 117'2999, mlcod=117'2992)", 2024-08-19T14:19:34.209 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:07.468206+0000", 2024-08-19T14:19:34.209 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 386.68374703500001, 2024-08-19T14:19:34.209 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0076013820000000003, 2024-08-19T14:19:34.209 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:19:34.209 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:19:34.209 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:19:34.209 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:19:34.209 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.209 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:19:34.209 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468206+0000", 2024-08-19T14:19:34.209 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:34.209 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.210 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.210 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:19:34.210 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468206+0000", 2024-08-19T14:19:34.210 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:34.210 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.210 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.210 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:19:34.210 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468207+0000", 2024-08-19T14:19:34.210 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6899999999999999e-07 2024-08-19T14:19:34.210 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.210 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.210 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:19:34.210 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468248+0000", 2024-08-19T14:19:34.211 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.0985999999999997e-05 2024-08-19T14:19:34.211 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.211 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.211 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:19:34.211 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468254+0000", 2024-08-19T14:19:34.211 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.9769999999999997e-06 2024-08-19T14:19:34.211 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.211 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.211 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:19:34.211 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468257+0000", 2024-08-19T14:19:34.211 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.0259999999999999e-06 2024-08-19T14:19:34.211 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.211 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.211 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:19:34.211 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.471956+0000", 2024-08-19T14:19:34.212 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0036996059999999998 2024-08-19T14:19:34.212 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.212 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.212 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:19:34.212 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.471974+0000", 2024-08-19T14:19:34.212 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8219999999999998e-05 2024-08-19T14:19:34.212 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.212 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.212 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:19:34.212 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.475774+0000", 2024-08-19T14:19:34.212 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0037994729999999998 2024-08-19T14:19:34.212 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.212 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.212 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:19:34.213 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.475808+0000", 2024-08-19T14:19:34.213 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.3924999999999998e-05 2024-08-19T14:19:34.213 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:34.213 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:19:34.213 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:34.213 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.213 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.213 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:48978 2.7 e117/102 2:e11d5fd6:::benchmark_data_smithi045_46415_object48977:head v 117'3000, mlcod=117'2992)", 2024-08-19T14:19:34.213 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:07.468262+0000", 2024-08-19T14:19:34.213 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 386.68369133599998, 2024-08-19T14:19:34.213 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0075744890000000002, 2024-08-19T14:19:34.213 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:19:34.213 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:19:34.213 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:19:34.214 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:19:34.214 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.214 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:19:34.214 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468262+0000", 2024-08-19T14:19:34.214 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:34.214 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.214 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.214 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:19:34.214 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468262+0000", 2024-08-19T14:19:34.214 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:34.214 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.214 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.214 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:19:34.214 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468262+0000", 2024-08-19T14:19:34.215 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.04e-07 2024-08-19T14:19:34.215 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.215 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.215 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:19:34.215 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468309+0000", 2024-08-19T14:19:34.215 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.6913999999999998e-05 2024-08-19T14:19:34.215 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.215 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.215 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:19:34.215 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468315+0000", 2024-08-19T14:19:34.215 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7139999999999998e-06 2024-08-19T14:19:34.215 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.215 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.215 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:19:34.216 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468318+0000", 2024-08-19T14:19:34.216 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.117e-06 2024-08-19T14:19:34.216 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.216 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.216 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:19:34.216 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.472643+0000", 2024-08-19T14:19:34.216 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0043251950000000004 2024-08-19T14:19:34.216 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.216 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.216 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:19:34.216 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.472657+0000", 2024-08-19T14:19:34.216 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3478e-05 2024-08-19T14:19:34.216 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.216 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.217 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:19:34.217 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.475812+0000", 2024-08-19T14:19:34.217 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0031556259999999999 2024-08-19T14:19:34.217 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.217 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.217 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:19:34.217 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.475837+0000", 2024-08-19T14:19:34.217 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.4241000000000001e-05 2024-08-19T14:19:34.217 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:34.217 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:19:34.217 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:34.217 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.217 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.217 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:48995 2.7 e117/102 2:e01c8116:::benchmark_data_smithi045_46415_object48994:head v 117'3001, mlcod=117'2992)", 2024-08-19T14:19:34.218 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:07.468324+0000", 2024-08-19T14:19:34.218 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 386.68362980900002, 2024-08-19T14:19:34.218 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0090529970000000001, 2024-08-19T14:19:34.218 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:19:34.218 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:19:34.218 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:19:34.218 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:19:34.218 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.218 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:19:34.218 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468324+0000", 2024-08-19T14:19:34.218 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:34.218 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.218 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.218 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:19:34.219 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468324+0000", 2024-08-19T14:19:34.219 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:34.219 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.219 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.219 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:19:34.219 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468324+0000", 2024-08-19T14:19:34.219 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.97e-07 2024-08-19T14:19:34.219 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.219 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.219 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:19:34.219 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468388+0000", 2024-08-19T14:19:34.219 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.4523999999999994e-05 2024-08-19T14:19:34.219 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.219 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.219 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:19:34.220 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468395+0000", 2024-08-19T14:19:34.220 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.579e-06 2024-08-19T14:19:34.220 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.220 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.220 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:19:34.220 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468398+0000", 2024-08-19T14:19:34.220 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.2780000000000002e-06 2024-08-19T14:19:34.220 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.220 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.220 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:19:34.220 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.473325+0000", 2024-08-19T14:19:34.220 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00492626 2024-08-19T14:19:34.220 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.220 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.221 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:19:34.221 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.473343+0000", 2024-08-19T14:19:34.221 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8788999999999999e-05 2024-08-19T14:19:34.221 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.221 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.221 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:19:34.221 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.477310+0000", 2024-08-19T14:19:34.221 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0039662819999999998 2024-08-19T14:19:34.221 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.221 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.221 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:19:34.221 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.477377+0000", 2024-08-19T14:19:34.221 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.7088000000000005e-05 2024-08-19T14:19:34.221 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:34.221 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:19:34.221 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:34.222 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.222 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.222 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:48996 2.7 e117/102 2:e8abf67b:::benchmark_data_smithi045_46415_object48995:head v 117'3002, mlcod=117'2992)", 2024-08-19T14:19:34.222 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:07.468406+0000", 2024-08-19T14:19:34.222 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 386.68354780099997, 2024-08-19T14:19:34.222 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0090294899999999994, 2024-08-19T14:19:34.222 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:19:34.222 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:19:34.222 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:19:34.222 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:19:34.222 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.222 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:19:34.222 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468406+0000", 2024-08-19T14:19:34.222 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:34.223 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.223 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.223 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:19:34.223 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468406+0000", 2024-08-19T14:19:34.223 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:34.223 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.223 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.223 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:19:34.223 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468406+0000", 2024-08-19T14:19:34.223 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0900000000000001e-07 2024-08-19T14:19:34.223 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.223 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.223 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:19:34.223 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468447+0000", 2024-08-19T14:19:34.224 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.0992000000000003e-05 2024-08-19T14:19:34.224 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.224 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.224 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:19:34.224 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468452+0000", 2024-08-19T14:19:34.224 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2179999999999998e-06 2024-08-19T14:19:34.224 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.224 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.224 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:19:34.224 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468455+0000", 2024-08-19T14:19:34.224 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.796e-06 2024-08-19T14:19:34.224 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.224 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.224 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:19:34.224 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.474288+0000", 2024-08-19T14:19:34.225 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0058327689999999998 2024-08-19T14:19:34.225 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.225 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.225 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:19:34.225 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.474307+0000", 2024-08-19T14:19:34.225 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9031000000000001e-05 2024-08-19T14:19:34.225 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.225 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.225 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:19:34.225 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.477383+0000", 2024-08-19T14:19:34.225 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0030758790000000001 2024-08-19T14:19:34.225 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.225 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.225 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:19:34.225 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.477435+0000", 2024-08-19T14:19:34.226 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2596000000000003e-05 2024-08-19T14:19:34.226 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:34.226 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:19:34.226 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:34.226 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.226 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.226 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:49005 2.7 e117/102 2:e58e73b6:::benchmark_data_smithi045_46415_object49004:head v 117'3003, mlcod=117'2992)", 2024-08-19T14:19:34.226 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:07.468460+0000", 2024-08-19T14:19:34.226 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 386.68349369999999, 2024-08-19T14:19:34.226 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0098532910000000001, 2024-08-19T14:19:34.226 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:19:34.226 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:19:34.226 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:19:34.226 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:19:34.227 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.227 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:19:34.227 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468460+0000", 2024-08-19T14:19:34.227 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:34.227 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.227 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.227 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:19:34.227 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468460+0000", 2024-08-19T14:19:34.227 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:34.227 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.227 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.227 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:19:34.227 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468460+0000", 2024-08-19T14:19:34.227 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5900000000000001e-07 2024-08-19T14:19:34.227 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.228 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.228 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:19:34.228 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468518+0000", 2024-08-19T14:19:34.228 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.8536000000000003e-05 2024-08-19T14:19:34.228 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.228 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.228 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:19:34.228 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468524+0000", 2024-08-19T14:19:34.228 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.8039999999999998e-06 2024-08-19T14:19:34.228 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.228 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.228 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:19:34.228 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468527+0000", 2024-08-19T14:19:34.228 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.004e-06 2024-08-19T14:19:34.228 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.229 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.229 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:19:34.229 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.475085+0000", 2024-08-19T14:19:34.229 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0065574090000000002 2024-08-19T14:19:34.229 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.229 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.229 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:19:34.229 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.475106+0000", 2024-08-19T14:19:34.229 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1056999999999998e-05 2024-08-19T14:19:34.229 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.229 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.229 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:19:34.229 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.478267+0000", 2024-08-19T14:19:34.229 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0031614049999999999 2024-08-19T14:19:34.229 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.230 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.230 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:19:34.230 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.478313+0000", 2024-08-19T14:19:34.230 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.5917000000000001e-05 2024-08-19T14:19:34.230 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:34.230 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:19:34.230 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:34.230 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.230 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.230 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:49015 2.7 e117/102 2:e4d8c3e1:::benchmark_data_smithi045_46415_object49014:head v 117'3004, mlcod=117'2992)", 2024-08-19T14:19:34.231 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:07.468533+0000", 2024-08-19T14:19:34.231 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 386.68342063599999, 2024-08-19T14:19:34.231 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0098176270000000006, 2024-08-19T14:19:34.231 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:19:34.231 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:19:34.231 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:19:34.231 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:19:34.231 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.231 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:19:34.231 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468533+0000", 2024-08-19T14:19:34.231 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:34.231 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.231 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.231 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:19:34.232 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468533+0000", 2024-08-19T14:19:34.232 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:34.232 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.232 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.232 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:19:34.232 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468533+0000", 2024-08-19T14:19:34.232 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2600000000000001e-07 2024-08-19T14:19:34.232 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.232 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.232 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:19:34.232 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468577+0000", 2024-08-19T14:19:34.232 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.3436000000000002e-05 2024-08-19T14:19:34.232 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.232 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.232 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:19:34.233 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468582+0000", 2024-08-19T14:19:34.233 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4399999999999996e-06 2024-08-19T14:19:34.233 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.233 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.233 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:19:34.233 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468585+0000", 2024-08-19T14:19:34.233 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.8619999999999999e-06 2024-08-19T14:19:34.233 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.233 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.233 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:19:34.233 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.475870+0000", 2024-08-19T14:19:34.233 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0072852919999999996 2024-08-19T14:19:34.233 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.233 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.233 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:19:34.234 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.475894+0000", 2024-08-19T14:19:34.234 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.3858000000000001e-05 2024-08-19T14:19:34.234 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.234 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.234 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:19:34.234 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.478319+0000", 2024-08-19T14:19:34.234 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0024247729999999999 2024-08-19T14:19:34.234 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.234 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.234 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:19:34.234 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.478350+0000", 2024-08-19T14:19:34.234 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.1739999999999998e-05 2024-08-19T14:19:34.234 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:34.234 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:19:34.235 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:34.235 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.235 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.235 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:49016 2.7 e117/102 2:e52959d5:::benchmark_data_smithi045_46415_object49015:head v 117'3005, mlcod=117'2992)", 2024-08-19T14:19:34.235 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:07.468590+0000", 2024-08-19T14:19:34.235 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 386.68336375799998, 2024-08-19T14:19:34.235 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.011903967999999999, 2024-08-19T14:19:34.235 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:19:34.235 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:19:34.235 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:19:34.235 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:19:34.235 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.235 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:19:34.235 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468590+0000", 2024-08-19T14:19:34.235 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:34.236 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.236 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.236 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:19:34.236 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468590+0000", 2024-08-19T14:19:34.236 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:34.236 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.236 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.236 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:19:34.236 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468590+0000", 2024-08-19T14:19:34.236 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.54e-07 2024-08-19T14:19:34.236 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.236 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.236 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:19:34.236 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468632+0000", 2024-08-19T14:19:34.237 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.2459999999999997e-05 2024-08-19T14:19:34.237 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.237 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.237 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:19:34.237 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468638+0000", 2024-08-19T14:19:34.237 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2410000000000003e-06 2024-08-19T14:19:34.237 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:19:34.238 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.238 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.238 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:19:34.238 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468641+0000", 2024-08-19T14:19:34.238 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.061e-06 2024-08-19T14:19:34.238 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.239 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.239 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:19:34.239 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.476571+0000", 2024-08-19T14:19:34.239 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0079302920000000002 2024-08-19T14:19:34.239 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.239 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.239 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:19:34.239 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.476593+0000", 2024-08-19T14:19:34.239 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2045e-05 2024-08-19T14:19:34.239 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.239 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.239 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:19:34.239 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.480447+0000", 2024-08-19T14:19:34.239 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.003853825 2024-08-19T14:19:34.240 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.240 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.240 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:19:34.240 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.480494+0000", 2024-08-19T14:19:34.240 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.689e-05 2024-08-19T14:19:34.240 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:34.240 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:19:34.240 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:34.240 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.240 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.240 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:49017 2.7 e117/102 2:eca28b17:::benchmark_data_smithi045_46415_object49016:head v 117'3006, mlcod=117'2992)", 2024-08-19T14:19:34.240 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:07.468646+0000", 2024-08-19T14:19:34.240 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 386.683307894, 2024-08-19T14:19:34.240 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.011882515999999999, 2024-08-19T14:19:34.241 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:19:34.241 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:19:34.241 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:19:34.241 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:19:34.241 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.241 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:19:34.241 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468646+0000", 2024-08-19T14:19:34.241 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:34.241 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.241 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.241 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:19:34.241 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468646+0000", 2024-08-19T14:19:34.241 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:34.241 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.242 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.242 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:19:34.242 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468646+0000", 2024-08-19T14:19:34.242 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5800000000000001e-07 2024-08-19T14:19:34.242 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.242 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.242 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:19:34.242 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468686+0000", 2024-08-19T14:19:34.242 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.0732999999999998e-05 2024-08-19T14:19:34.242 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.242 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.242 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:19:34.242 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468692+0000", 2024-08-19T14:19:34.242 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0860000000000001e-06 2024-08-19T14:19:34.243 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.243 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.243 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:19:34.243 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468694+0000", 2024-08-19T14:19:34.243 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.864e-06 2024-08-19T14:19:34.243 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.243 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.243 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:19:34.243 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.477463+0000", 2024-08-19T14:19:34.243 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0087685190000000007 2024-08-19T14:19:34.244 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.244 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.244 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:19:34.244 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.477483+0000", 2024-08-19T14:19:34.244 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0449999999999999e-05 2024-08-19T14:19:34.244 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.244 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.244 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:19:34.244 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.480501+0000", 2024-08-19T14:19:34.244 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.003017802 2024-08-19T14:19:34.244 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.244 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.244 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:19:34.244 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.480528+0000", 2024-08-19T14:19:34.245 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.6903999999999999e-05 2024-08-19T14:19:34.245 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:34.245 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:19:34.245 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:34.245 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.245 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.245 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:49027 2.7 e117/102 2:e96ce7ac:::benchmark_data_smithi045_46415_object49026:head v 117'3007, mlcod=117'2992)", 2024-08-19T14:19:34.245 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:07.468699+0000", 2024-08-19T14:19:34.245 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 386.683254482, 2024-08-19T14:19:34.245 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.011858521, 2024-08-19T14:19:34.245 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:19:34.245 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:19:34.245 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:19:34.245 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:19:34.246 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.246 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:19:34.246 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468699+0000", 2024-08-19T14:19:34.246 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:34.246 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.246 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.246 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:19:34.246 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468699+0000", 2024-08-19T14:19:34.246 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:34.246 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.246 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.246 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:19:34.246 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468699+0000", 2024-08-19T14:19:34.246 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5599999999999999e-07 2024-08-19T14:19:34.247 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.247 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.247 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:19:34.247 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468743+0000", 2024-08-19T14:19:34.247 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.3741e-05 2024-08-19T14:19:34.247 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.247 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.247 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:19:34.247 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468748+0000", 2024-08-19T14:19:34.247 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1000000000000003e-06 2024-08-19T14:19:34.247 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.247 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.247 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:19:34.247 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468751+0000", 2024-08-19T14:19:34.248 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.8260000000000001e-06 2024-08-19T14:19:34.248 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.248 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.248 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:19:34.248 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.478397+0000", 2024-08-19T14:19:34.248 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0096462240000000001 2024-08-19T14:19:34.248 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.248 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.248 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:19:34.248 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.478415+0000", 2024-08-19T14:19:34.248 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8450000000000001e-05 2024-08-19T14:19:34.248 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.248 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.248 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:19:34.249 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.480533+0000", 2024-08-19T14:19:34.249 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.002117196 2024-08-19T14:19:34.249 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.249 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.249 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:19:34.249 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.480558+0000", 2024-08-19T14:19:34.249 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.4828e-05 2024-08-19T14:19:34.249 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:34.249 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:19:34.249 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:34.249 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.249 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.249 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:49030 2.7 e117/102 2:e915209e:::benchmark_data_smithi045_46415_object49029:head v 117'3008, mlcod=117'2992)", 2024-08-19T14:19:34.249 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:07.468755+0000", 2024-08-19T14:19:34.249 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 386.68319819499999, 2024-08-19T14:19:34.250 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.015403441, 2024-08-19T14:19:34.250 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:19:34.250 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:19:34.250 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:19:34.250 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:19:34.250 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.250 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:19:34.250 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468755+0000", 2024-08-19T14:19:34.250 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:34.250 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.250 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.250 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:19:34.250 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468755+0000", 2024-08-19T14:19:34.250 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:34.251 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.251 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.251 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:19:34.251 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468755+0000", 2024-08-19T14:19:34.251 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5599999999999999e-07 2024-08-19T14:19:34.251 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.251 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.251 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:19:34.251 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468805+0000", 2024-08-19T14:19:34.251 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.9622e-05 2024-08-19T14:19:34.251 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.251 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.251 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:19:34.251 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468811+0000", 2024-08-19T14:19:34.252 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.6119999999999998e-06 2024-08-19T14:19:34.252 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.252 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.252 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:19:34.252 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468813+0000", 2024-08-19T14:19:34.252 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.7769999999999998e-06 2024-08-19T14:19:34.252 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.252 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.252 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:19:34.252 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.479393+0000", 2024-08-19T14:19:34.252 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.010579646999999999 2024-08-19T14:19:34.252 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.252 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.252 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:19:34.252 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.479412+0000", 2024-08-19T14:19:34.253 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8728000000000001e-05 2024-08-19T14:19:34.253 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.253 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.253 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:19:34.253 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.484094+0000", 2024-08-19T14:19:34.253 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0046820009999999999 2024-08-19T14:19:34.253 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.253 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.253 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:19:34.253 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.484159+0000", 2024-08-19T14:19:34.253 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.4898000000000001e-05 2024-08-19T14:19:34.253 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:34.253 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:19:34.253 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:34.254 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.254 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.254 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:49049 2.7 e117/102 2:eb29c0a1:::benchmark_data_smithi045_46415_object49048:head v 117'3009, mlcod=117'2992)", 2024-08-19T14:19:34.254 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:07.468819+0000", 2024-08-19T14:19:34.254 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 386.68313494799997, 2024-08-19T14:19:34.254 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.015383332, 2024-08-19T14:19:34.254 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:19:34.254 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:19:34.254 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:19:34.254 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:19:34.254 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.254 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:19:34.254 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468819+0000", 2024-08-19T14:19:34.254 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:34.254 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.255 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.255 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:19:34.255 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468819+0000", 2024-08-19T14:19:34.255 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:34.255 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.255 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.255 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:19:34.255 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468819+0000", 2024-08-19T14:19:34.255 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2000000000000001e-07 2024-08-19T14:19:34.255 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.255 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.255 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:19:34.255 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468854+0000", 2024-08-19T14:19:34.255 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.5079000000000001e-05 2024-08-19T14:19:34.256 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.256 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.256 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:19:34.256 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468859+0000", 2024-08-19T14:19:34.256 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1710000000000001e-06 2024-08-19T14:19:34.256 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.256 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.256 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:19:34.256 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468862+0000", 2024-08-19T14:19:34.256 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.0599999999999999e-06 2024-08-19T14:19:34.256 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.256 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.256 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:19:34.256 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.480596+0000", 2024-08-19T14:19:34.256 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.011734058 2024-08-19T14:19:34.257 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.257 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.257 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:19:34.257 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.480615+0000", 2024-08-19T14:19:34.257 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8919e-05 2024-08-19T14:19:34.257 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.257 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.257 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:19:34.257 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.484164+0000", 2024-08-19T14:19:34.257 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.003548486 2024-08-19T14:19:34.257 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:34.257 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:34.257 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:19:34.257 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.484202+0000", 2024-08-19T14:19:34.257 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.8339e-05 2024-08-19T14:19:34.258 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:34.258 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:19:34.258 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:34.258 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:34.258 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:19:34.258 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:19:34.287 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:34.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:34.288 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:19:34.388 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:19:34.429 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:34.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:34.430 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:19:34.531 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:19:34.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:34.563 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:34.563 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:19:34.664 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:19:34.685 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:34.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:34.686 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:19:34.787 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:19:34.817 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:34.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:34.818 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:19:34.919 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:19:34.949 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:34.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:34.951 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:19:34.984 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:34 smithi045 ceph-mon[20143]: osdmap e307: 12 total, 12 up, 9 in 2024-08-19T14:19:34.984 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:34 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:19:34.984 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:34 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:19:35.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:34 smithi032 ceph-mon[20004]: osdmap e307: 12 total, 12 up, 9 in 2024-08-19T14:19:35.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:34 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:19:35.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:34 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:19:35.010 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-08-19T14:19:35.048 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:34 smithi031 ceph-mon[16336]: osdmap e307: 12 total, 12 up, 9 in 2024-08-19T14:19:35.048 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:34 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:19:35.048 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:34 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:19:35.051 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:19:35.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:35.088 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:35.088 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:19:35.189 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:19:35.218 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:35.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:35.219 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:19:35.320 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:19:35.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:35.348 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:35.348 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:19:35.419 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:19:35.450 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:19:35.496 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:35.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:35.497 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:19:35.598 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:19:35.625 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:35.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:35.626 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:19:35.727 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:19:35.752 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:35.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:35.753 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:19:35.821 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:35 smithi045 ceph-mon[20143]: pgmap v697: 77 pgs: 3 active+recovering+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 68 active+clean; 3.0 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 1.03k op/s; 4/145038 objects degraded (0.003%); 5396/145038 objects misplaced (3.720%); 22 MiB/s, 601 objects/s recovering 2024-08-19T14:19:35.821 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:35 smithi045 ceph-mon[20143]: osdmap e308: 12 total, 12 up, 9 in 2024-08-19T14:19:35.821 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:35 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:19:35.821 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:35 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:19:35.853 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:19:35.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:35.898 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:35.898 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:19:35.979 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:19:35.979 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [ 2024-08-19T14:19:35.979 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:35.979 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25057.0:218352 3.0 3:0389e812:::benchmark_data_smithi045_59973_object87173:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e308)", 2024-08-19T14:19:35.979 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:19:35.967007+0000", 2024-08-19T14:19:35.979 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.010224550000000001, 2024-08-19T14:19:35.979 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.010264966, 2024-08-19T14:19:35.980 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:19:35.980 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:19:35.980 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "waiting for sub ops", 2024-08-19T14:19:35.980 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:19:35.980 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25057", 2024-08-19T14:19:35.980 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:19:35.980 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 218352 2024-08-19T14:19:35.980 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:35.980 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:19:35.980 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:35.980 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:19:35.980 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:35.967007+0000", 2024-08-19T14:19:35.981 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:35.981 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:35.981 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:35.981 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:19:35.981 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:35.967007+0000", 2024-08-19T14:19:35.981 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:35.981 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:35.981 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:35.981 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:19:35.981 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:35.967009+0000", 2024-08-19T14:19:35.981 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.606e-06 2024-08-19T14:19:35.981 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:35.981 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:35.981 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:19:35.982 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:35.967015+0000", 2024-08-19T14:19:35.982 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7359999999999998e-06 2024-08-19T14:19:35.982 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:35.982 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:35.982 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:19:35.982 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:35.967028+0000", 2024-08-19T14:19:35.982 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2845e-05 2024-08-19T14:19:35.982 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:35.982 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:35.982 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:19:35.982 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:35.967036+0000", 2024-08-19T14:19:35.982 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.6139999999999992e-06 2024-08-19T14:19:35.982 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:35.983 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:35.983 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:19:35.983 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:35.967321+0000", 2024-08-19T14:19:35.983 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00028457800000000001 2024-08-19T14:19:35.983 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:35.983 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:35.983 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:19:35.983 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:35.967461+0000", 2024-08-19T14:19:35.983 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000140135 2024-08-19T14:19:35.983 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:35.983 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:35.984 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,9", 2024-08-19T14:19:35.984 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:35.967579+0000", 2024-08-19T14:19:35.984 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00011821500000000001 2024-08-19T14:19:35.984 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:35.984 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:35.984 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:19:35.984 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:35.969532+0000", 2024-08-19T14:19:35.984 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0019526789999999999 2024-08-19T14:19:35.984 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:35.984 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:35.984 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:19:35.984 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:35.972075+0000", 2024-08-19T14:19:35.984 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0025429519999999998 2024-08-19T14:19:35.984 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:35.985 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:19:35.985 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:35.985 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:35.985 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:35.985 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25057.0:218359 3.e 3:7385567b:::benchmark_data_smithi045_59973_object87180:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e308)", 2024-08-19T14:19:35.985 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:19:35.970161+0000", 2024-08-19T14:19:35.985 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.0070710879999999997, 2024-08-19T14:19:35.985 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0072042319999999996, 2024-08-19T14:19:35.985 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:19:35.985 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:19:35.985 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "waiting for sub ops", 2024-08-19T14:19:35.985 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:19:35.985 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25057", 2024-08-19T14:19:35.985 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:19:35.985 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 218359 2024-08-19T14:19:35.986 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:35.986 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:19:35.986 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:35.986 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:19:35.986 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:35.970161+0000", 2024-08-19T14:19:35.986 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:35.986 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:35.986 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:35.986 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:19:35.986 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:35.970161+0000", 2024-08-19T14:19:35.986 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:35.986 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:35.987 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:35.987 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:19:35.987 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:35.970162+0000", 2024-08-19T14:19:35.987 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.429e-06 2024-08-19T14:19:35.987 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:35.987 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:35.987 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:19:35.987 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:35.970173+0000", 2024-08-19T14:19:35.987 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.062e-05 2024-08-19T14:19:35.988 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:35.988 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:35.988 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:19:35.988 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:35.970190+0000", 2024-08-19T14:19:35.988 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7064000000000001e-05 2024-08-19T14:19:35.988 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:35.988 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:35.989 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:19:35.989 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:35.970197+0000", 2024-08-19T14:19:35.989 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4059999999999999e-06 2024-08-19T14:19:35.989 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:35.989 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:35.989 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:19:35.989 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:35.970324+0000", 2024-08-19T14:19:35.989 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000126322 2024-08-19T14:19:35.989 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:35.989 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:35.989 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:19:35.989 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:35.970436+0000", 2024-08-19T14:19:35.989 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000112321 2024-08-19T14:19:35.990 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:35.990 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:35.990 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,4", 2024-08-19T14:19:35.990 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:35.970537+0000", 2024-08-19T14:19:35.990 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00010139799999999999 2024-08-19T14:19:35.990 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:35.990 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:35.990 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:19:35.990 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:35.974376+0000", 2024-08-19T14:19:35.990 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0038386190000000001 2024-08-19T14:19:35.990 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:35.990 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:35.991 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:19:35.991 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:35.975096+0000", 2024-08-19T14:19:35.991 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00072007500000000003 2024-08-19T14:19:35.991 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:35.991 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:19:35.991 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:35.991 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:35.991 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:35.991 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.25057.0:218358 3.f e308/244 3:f74eb95c:::benchmark_data_smithi045_59973_object87179:head v 308'8789, mlcod=308'8786)", 2024-08-19T14:19:35.991 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:19:35.971798+0000", 2024-08-19T14:19:35.991 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.0054339560000000002, 2024-08-19T14:19:35.991 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0056475450000000003, 2024-08-19T14:19:35.991 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:19:35.991 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:19:35.992 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-08-19T14:19:35.992 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:19:35.992 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:35.992 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:19:35.992 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:35.971798+0000", 2024-08-19T14:19:35.992 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:35.992 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:35.992 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:35.992 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:19:35.992 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:35.971798+0000", 2024-08-19T14:19:35.992 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:35.992 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:35.992 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:35.993 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:19:35.993 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:35.971799+0000", 2024-08-19T14:19:35.993 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.2099999999999995e-07 2024-08-19T14:19:35.993 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:35.993 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:35.993 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:19:35.993 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:35.971806+0000", 2024-08-19T14:19:35.993 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.9439999999999999e-06 2024-08-19T14:19:35.993 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:35.993 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:35.994 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:19:35.994 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:35.971820+0000", 2024-08-19T14:19:35.994 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4463e-05 2024-08-19T14:19:35.994 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:35.994 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:35.994 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:19:35.994 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:35.971827+0000", 2024-08-19T14:19:35.994 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.737e-06 2024-08-19T14:19:35.994 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:35.994 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:35.994 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:19:35.994 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:35.972706+0000", 2024-08-19T14:19:35.994 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00087907600000000001 2024-08-19T14:19:35.995 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:35.995 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:35.995 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:19:35.998 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:35.972731+0000", 2024-08-19T14:19:35.998 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.5145000000000001e-05 2024-08-19T14:19:35.998 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:35.998 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:19:35.998 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:35.998 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:35.999 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:35.999 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDPGPush(3.25 308/299 [PushOp(3:a6f89c82:::benchmark_data_smithi045_59973_object127930:head, version: 280'5500, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:a6f89c82:::benchmark_data_smithi045_59973_object127930:head@280'5500, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-19T14:19:35.999 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:19:35.974642+0000", 2024-08-19T14:19:35.999 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.002589526, 2024-08-19T14:19:35.999 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0028836510000000001, 2024-08-19T14:19:35.999 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:19:35.999 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:19:35.999 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-08-19T14:19:35.999 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:19:35.999 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:35.999 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:19:35.999 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:35.974642+0000", 2024-08-19T14:19:36.000 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:36.000 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:36.000 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:36.000 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:19:36.000 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:35.974642+0000", 2024-08-19T14:19:36.000 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:36.000 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:36.000 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:36.000 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:19:36.000 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:35.974650+0000", 2024-08-19T14:19:36.000 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.9200000000000004e-06 2024-08-19T14:19:36.000 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:36.000 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:36.001 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:19:36.001 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:35.976262+0000", 2024-08-19T14:19:36.001 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001611641 2024-08-19T14:19:36.001 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:36.001 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:36.001 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:19:36.001 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:35.976282+0000", 2024-08-19T14:19:36.001 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.012e-05 2024-08-19T14:19:36.001 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:36.001 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:36.001 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:19:36.001 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:35.976299+0000", 2024-08-19T14:19:36.002 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6648999999999999e-05 2024-08-19T14:19:36.002 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:36.002 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:36.002 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:19:36.002 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:35.976542+0000", 2024-08-19T14:19:36.002 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00024282199999999999 2024-08-19T14:19:36.002 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:36.002 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:36.002 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:19:36.002 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:35.976555+0000", 2024-08-19T14:19:36.002 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3303e-05 2024-08-19T14:19:36.003 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:36.003 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:19:36.003 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:36.003 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:36.003 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:36.003 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDPGPush(3.2c 308/299 [PushOp(3:3784db6a:::benchmark_data_smithi045_59973_object130920:head, version: 282'6017, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:3784db6a:::benchmark_data_smithi045_59973_object130920:head@282'6017, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-19T14:19:36.003 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:19:35.976330+0000", 2024-08-19T14:19:36.003 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.00090226599999999998, 2024-08-19T14:19:36.003 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001270771, 2024-08-19T14:19:36.003 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:19:36.004 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:19:36.004 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "queued for pg", 2024-08-19T14:19:36.004 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:19:36.004 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:36.004 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:19:36.004 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:35.976330+0000", 2024-08-19T14:19:36.004 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:36.004 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:36.004 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:36.004 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:19:36.004 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:35.976330+0000", 2024-08-19T14:19:36.004 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:36.005 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:36.005 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:36.005 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:19:36.005 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:35.976330+0000", 2024-08-19T14:19:36.005 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.9699999999999996e-07 2024-08-19T14:19:36.005 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:36.005 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:36.005 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:19:36.005 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:35.976366+0000", 2024-08-19T14:19:36.005 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.5858000000000001e-05 2024-08-19T14:19:36.005 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:36.005 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:36.005 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:19:36.006 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:35.976380+0000", 2024-08-19T14:19:36.006 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.393e-05 2024-08-19T14:19:36.006 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:36.006 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:36.006 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:19:36.006 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:35.976394+0000", 2024-08-19T14:19:36.006 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.403e-05 2024-08-19T14:19:36.006 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:36.006 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:19:36.006 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:36.006 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:36.006 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:36.007 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.25057.0:218365 3.2c e308/299 3:366b6f5a:::benchmark_data_smithi045_59973_object87186:head v 308'8307, mlcod=282'6014)", 2024-08-19T14:19:36.007 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:19:35.976418+0000", 2024-08-19T14:19:36.007 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.00081433299999999998, 2024-08-19T14:19:36.007 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0012280959999999999, 2024-08-19T14:19:36.007 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:19:36.007 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:19:36.007 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-08-19T14:19:36.007 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:19:36.007 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:36.007 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:19:36.007 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:35.976418+0000", 2024-08-19T14:19:36.007 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:36.007 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:36.007 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:36.008 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:19:36.008 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:35.976418+0000", 2024-08-19T14:19:36.008 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:36.008 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:36.008 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:36.008 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:19:36.008 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:35.976418+0000", 2024-08-19T14:19:36.008 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.4200000000000002e-07 2024-08-19T14:19:36.008 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:36.008 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:36.008 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:19:36.008 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:35.976423+0000", 2024-08-19T14:19:36.008 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1889999999999998e-06 2024-08-19T14:19:36.008 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:36.009 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:36.009 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:19:36.009 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:35.976430+0000", 2024-08-19T14:19:36.009 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.8940000000000004e-06 2024-08-19T14:19:36.009 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:36.009 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:36.009 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:19:36.009 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:35.976435+0000", 2024-08-19T14:19:36.009 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.9150000000000004e-06 2024-08-19T14:19:36.009 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:36.009 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:36.009 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:19:36.009 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:35.976629+0000", 2024-08-19T14:19:36.009 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00019450700000000001 2024-08-19T14:19:36.010 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:36.010 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:36.010 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:19:36.010 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:35.976652+0000", 2024-08-19T14:19:36.010 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2563999999999999e-05 2024-08-19T14:19:36.010 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:36.010 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:19:36.010 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:36.010 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:36.010 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:36.010 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.25057.0:218366 3.f e308/244)", 2024-08-19T14:19:36.010 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:19:35.977405+0000", 2024-08-19T14:19:36.010 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 4294967295.9998269, 2024-08-19T14:19:36.011 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00029346899999999998, 2024-08-19T14:19:36.011 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:19:36.011 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:19:36.011 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "queued for pg", 2024-08-19T14:19:36.011 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:19:36.011 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:36.011 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:19:36.011 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:35.977405+0000", 2024-08-19T14:19:36.011 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:36.011 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:36.011 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:36.011 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:19:36.011 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:35.977405+0000", 2024-08-19T14:19:36.011 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:19:36.012 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:36.012 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:36.012 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:19:36.012 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:35.977406+0000", 2024-08-19T14:19:36.012 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.33e-07 2024-08-19T14:19:36.012 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:36.012 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:36.012 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:19:36.012 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:35.977412+0000", 2024-08-19T14:19:36.012 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.0340000000000002e-06 2024-08-19T14:19:36.012 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:36.012 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:36.012 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:19:36.012 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:35.977426+0000", 2024-08-19T14:19:36.013 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4124999999999999e-05 2024-08-19T14:19:36.013 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:19:36.013 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:19:36.013 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:19:36.013 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:35.977434+0000", 2024-08-19T14:19:36.013 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.8189999999999992e-06 2024-08-19T14:19:36.013 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:36.013 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:19:36.013 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:36.013 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:19:36.013 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-08-19T14:19:36.013 INFO:teuthology.orchestra.run.smithi045.stdout: "num_ops": 7 2024-08-19T14:19:36.014 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:19:36.014 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:19:36.015 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:35 smithi032 ceph-mon[20004]: pgmap v697: 77 pgs: 3 active+recovering+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 68 active+clean; 3.0 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 1.03k op/s; 4/145038 objects degraded (0.003%); 5396/145038 objects misplaced (3.720%); 22 MiB/s, 601 objects/s recovering 2024-08-19T14:19:36.015 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:35 smithi032 ceph-mon[20004]: osdmap e308: 12 total, 12 up, 9 in 2024-08-19T14:19:36.015 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:35 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:19:36.015 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:35 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:19:36.037 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:36.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:36.038 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:19:36.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:35 smithi031 ceph-mon[16336]: pgmap v697: 77 pgs: 3 active+recovering+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 68 active+clean; 3.0 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 1.03k op/s; 4/145038 objects degraded (0.003%); 5396/145038 objects misplaced (3.720%); 22 MiB/s, 601 objects/s recovering 2024-08-19T14:19:36.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:35 smithi031 ceph-mon[16336]: osdmap e308: 12 total, 12 up, 9 in 2024-08-19T14:19:36.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:35 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:19:36.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:35 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:19:36.139 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:19:36.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:36.183 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:36.183 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:19:36.284 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:19:36.291 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 7, 8, 9, 11] out_osds: [10, 6, 2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-08-19T14:19:36.292 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd reweight 0 1 2024-08-19T14:19:36.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:36.310 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:36.310 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:19:36.411 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:19:36.437 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:36.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:36.438 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:19:36.539 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:19:36.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:36.561 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:36.561 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:19:36.662 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:19:36.693 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:36.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:36.694 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:19:36.794 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:19:36.824 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:36.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:36.825 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:19:36.926 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:19:36.953 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:36.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:36.954 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:19:36.978 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:36 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/997028777' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:19:37.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:36 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/997028777' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:19:37.055 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:19:37.088 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:37.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:37.089 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:19:37.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:36 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/997028777' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:19:37.190 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:19:37.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:37.219 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:37.219 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:19:37.320 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:19:37.346 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:37.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:37.348 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:19:37.379 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-08-19T14:19:37.448 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:19:37.468 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:19:37.468 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:19:37.468 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:19:37.468 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:19:37.468 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:19:37.468 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:19:37.468 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:19:37.468 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:19:37.468 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:19:37.469 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:19:37.469 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:19:37.469 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:19:37.469 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:19:37.469 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:19:37.469 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:19:37.469 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:19:37.469 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:19:37.469 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:19:37.469 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:19:37.469 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:19:37.469 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:19:37.470 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:19:37.470 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:19:37.470 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:19:37.491 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:37.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:37.494 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:19:37.595 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:19:37.615 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:37.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:37.616 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:19:37.708 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:19:37.716 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:19:37.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:37.756 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:37.756 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:19:37.857 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:19:37.902 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:37.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:37.903 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:19:37.986 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:37 smithi045 ceph-mon[20143]: pgmap v699: 77 pgs: 2 active+recovering+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 69 active+clean; 2.8 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 953 op/s; 1/139827 objects degraded (0.001%); 3929/139827 objects misplaced (2.810%); 17 MiB/s, 442 objects/s recovering 2024-08-19T14:19:37.987 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:37 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 1/139827 objects degraded (0.001%), 2 pgs degraded (PG_DEGRADED) 2024-08-19T14:19:38.004 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:19:38.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:37 smithi032 ceph-mon[20004]: pgmap v699: 77 pgs: 2 active+recovering+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 69 active+clean; 2.8 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 953 op/s; 1/139827 objects degraded (0.001%); 3929/139827 objects misplaced (2.810%); 17 MiB/s, 442 objects/s recovering 2024-08-19T14:19:38.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:37 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 1/139827 objects degraded (0.001%), 2 pgs degraded (PG_DEGRADED) 2024-08-19T14:19:38.025 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:38.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:38.026 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:19:38.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:37 smithi031 ceph-mon[16336]: pgmap v699: 77 pgs: 2 active+recovering+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 69 active+clean; 2.8 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 953 op/s; 1/139827 objects degraded (0.001%); 3929/139827 objects misplaced (2.810%); 17 MiB/s, 442 objects/s recovering 2024-08-19T14:19:38.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:37 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 1/139827 objects degraded (0.001%), 2 pgs degraded (PG_DEGRADED) 2024-08-19T14:19:38.127 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:19:38.163 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:38.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:38.164 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:19:38.265 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:19:38.268 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:19:38.268 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:19:38.303 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:38.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:38.304 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:19:38.404 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:19:38.429 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:38.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:38.430 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:19:38.531 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:19:38.573 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:38.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:38.574 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:19:38.675 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:19:38.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:38.707 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:38.708 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:19:38.808 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:19:38.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:38.843 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:38.843 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:19:38.936 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-08-19T14:19:38.944 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:19:38.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:38.966 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:38.966 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:19:39.067 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:19:39.096 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:39.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:39.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:19:39.198 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:19:39.220 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:39.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:39.221 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:19:39.261 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:19:39.322 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:19:39.351 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:39.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:39.352 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:19:39.453 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:19:39.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:39.481 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:39.481 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:19:39.582 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:19:39.624 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:39.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:39.625 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:19:39.726 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:19:39.769 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:39.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:39.770 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:19:39.828 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:19:39.828 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:19:39.871 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:19:39.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:39.915 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:39.915 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:19:40.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:39 smithi032 ceph-mon[20004]: pgmap v700: 77 pgs: 2 active+recovering+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 69 active+clean; 2.8 GiB data, 12 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 824 op/s; 1/135837 objects degraded (0.001%); 3801/135837 objects misplaced (2.798%); 12 MiB/s, 394 objects/s recovering 2024-08-19T14:19:40.016 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:19:40.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:39 smithi045 ceph-mon[20143]: pgmap v700: 77 pgs: 2 active+recovering+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 69 active+clean; 2.8 GiB data, 12 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 824 op/s; 1/135837 objects degraded (0.001%); 3801/135837 objects misplaced (2.798%); 12 MiB/s, 394 objects/s recovering 2024-08-19T14:19:40.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:40.048 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:40.048 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:19:40.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:39 smithi031 ceph-mon[16336]: pgmap v700: 77 pgs: 2 active+recovering+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 69 active+clean; 2.8 GiB data, 12 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 824 op/s; 1/135837 objects degraded (0.001%); 3801/135837 objects misplaced (2.798%); 12 MiB/s, 394 objects/s recovering 2024-08-19T14:19:40.149 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:19:40.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:40.213 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:40.213 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:19:40.314 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:19:40.340 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:40.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:40.342 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:19:40.443 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:19:40.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:40.465 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:40.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:19:40.479 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-08-19T14:19:40.567 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:19:40.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:40.590 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:40.590 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:19:40.691 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:19:40.719 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:40.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:40.720 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:19:40.821 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:19:40.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:40 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:19:40.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:40 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:19:40.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:40 smithi031 ceph-mon[16336]: osdmap e309: 12 total, 12 up, 9 in 2024-08-19T14:19:40.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:40 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:19:40.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:40 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:19:40.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:40 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:19:40.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:40 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:19:40.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:40 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:19:40.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:40.870 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:40.871 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:19:40.971 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:19:40.999 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:41.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:41.000 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:19:41.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:40 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:19:41.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:40 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:19:41.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:40 smithi045 ceph-mon[20143]: osdmap e309: 12 total, 12 up, 9 in 2024-08-19T14:19:41.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:40 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:19:41.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:40 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:19:41.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:40 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:19:41.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:40 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:19:41.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:40 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:19:41.100 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:19:41.135 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:41.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:41.136 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:19:41.237 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:19:41.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:40 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:19:41.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:40 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:19:41.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:40 smithi032 ceph-mon[20004]: osdmap e309: 12 total, 12 up, 9 in 2024-08-19T14:19:41.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:40 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:19:41.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:40 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:19:41.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:40 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:19:41.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:40 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:19:41.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:40 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:19:41.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:41.273 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:41.273 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:19:41.374 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:19:41.405 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:41.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:41.406 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:19:41.507 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:19:41.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:41.544 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:41.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:19:41.645 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:19:41.682 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:41.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:41.683 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:19:41.784 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:19:41.809 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:41.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:41.811 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:19:41.912 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:19:41.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:41.949 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:41.950 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:19:42.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:41 smithi045 ceph-mon[20143]: pgmap v702: 77 pgs: 2 active+recovering+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 69 active+clean; 2.6 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 1.01k op/s; 1/127854 objects degraded (0.001%); 1630/127854 objects misplaced (1.275%); 19 MiB/s, 560 objects/s recovering 2024-08-19T14:19:42.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:41 smithi045 ceph-mon[20143]: osdmap e310: 12 total, 12 up, 9 in 2024-08-19T14:19:42.051 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:19:42.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:42.077 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:42.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:19:42.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:41 smithi031 ceph-mon[16336]: pgmap v702: 77 pgs: 2 active+recovering+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 69 active+clean; 2.6 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 1.01k op/s; 1/127854 objects degraded (0.001%); 1630/127854 objects misplaced (1.275%); 19 MiB/s, 560 objects/s recovering 2024-08-19T14:19:42.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:41 smithi031 ceph-mon[16336]: osdmap e310: 12 total, 12 up, 9 in 2024-08-19T14:19:42.179 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:19:42.208 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:42.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:42.209 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:19:42.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:41 smithi032 ceph-mon[20004]: pgmap v702: 77 pgs: 2 active+recovering+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 69 active+clean; 2.6 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 1.01k op/s; 1/127854 objects degraded (0.001%); 1630/127854 objects misplaced (1.275%); 19 MiB/s, 560 objects/s recovering 2024-08-19T14:19:42.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:41 smithi032 ceph-mon[20004]: osdmap e310: 12 total, 12 up, 9 in 2024-08-19T14:19:42.310 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:19:42.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:42.339 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:42.339 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:19:42.440 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:19:42.465 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:42.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:42.466 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:19:42.567 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:19:42.593 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:42.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:42.594 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:19:42.695 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:19:42.722 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:42.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:42.723 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:19:42.824 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:19:42.867 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:42.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:42.868 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:19:42.969 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:19:43.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:43.009 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:43.010 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:19:43.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:42 smithi045 ceph-mon[20143]: osdmap e311: 12 total, 12 up, 9 in 2024-08-19T14:19:43.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:42 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 1/127854 objects degraded (0.001%), 2 pgs degraded (PG_DEGRADED) 2024-08-19T14:19:43.110 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:19:43.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:42 smithi031 ceph-mon[16336]: osdmap e311: 12 total, 12 up, 9 in 2024-08-19T14:19:43.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:42 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 1/127854 objects degraded (0.001%), 2 pgs degraded (PG_DEGRADED) 2024-08-19T14:19:43.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:43.147 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:43.147 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:19:43.248 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:19:43.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:42 smithi032 ceph-mon[20004]: osdmap e311: 12 total, 12 up, 9 in 2024-08-19T14:19:43.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:42 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 1/127854 objects degraded (0.001%), 2 pgs degraded (PG_DEGRADED) 2024-08-19T14:19:43.282 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:43.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:43.283 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:19:43.384 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:19:43.421 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:43.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:43.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:19:43.524 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:19:43.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:43.555 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:43.555 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:19:43.656 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:19:43.686 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:43.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:43.687 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:19:43.789 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:19:43.835 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:43.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:43.836 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:19:43.937 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:19:43.973 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:43.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:43.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:19:44.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:43 smithi045 ceph-mon[20143]: pgmap v705: 77 pgs: 3 active+recovering+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 69 active+clean; 2.4 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 1.08k op/s; 1/120273 objects degraded (0.001%); 1630/120273 objects misplaced (1.355%); 8.2 MiB/s, 497 objects/s recovering 2024-08-19T14:19:44.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:43 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:19:44.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:43 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:19:44.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:43 smithi045 ceph-mon[20143]: osdmap e312: 12 total, 12 up, 9 in 2024-08-19T14:19:44.075 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:19:44.109 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:44.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:44.110 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:19:44.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:43 smithi031 ceph-mon[16336]: pgmap v705: 77 pgs: 3 active+recovering+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 69 active+clean; 2.4 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 1.08k op/s; 1/120273 objects degraded (0.001%); 1630/120273 objects misplaced (1.355%); 8.2 MiB/s, 497 objects/s recovering 2024-08-19T14:19:44.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:43 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:19:44.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:43 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:19:44.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:43 smithi031 ceph-mon[16336]: osdmap e312: 12 total, 12 up, 9 in 2024-08-19T14:19:44.211 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:19:44.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:44.246 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:44.246 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:19:44.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:43 smithi032 ceph-mon[20004]: pgmap v705: 77 pgs: 3 active+recovering+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 69 active+clean; 2.4 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 1.08k op/s; 1/120273 objects degraded (0.001%); 1630/120273 objects misplaced (1.355%); 8.2 MiB/s, 497 objects/s recovering 2024-08-19T14:19:44.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:43 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:19:44.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:43 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:19:44.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:43 smithi032 ceph-mon[20004]: osdmap e312: 12 total, 12 up, 9 in 2024-08-19T14:19:44.347 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:19:44.377 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:44.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:44.378 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:19:44.479 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:19:44.507 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:44.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:44.508 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:19:44.609 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:19:44.633 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:44.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:44.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:19:44.735 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:19:44.760 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:44.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:44.761 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:19:44.862 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:19:44.893 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:44.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:44.894 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:19:44.995 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:19:45.069 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:44 smithi045 ceph-mon[20143]: osdmap e313: 12 total, 12 up, 9 in 2024-08-19T14:19:45.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:45.106 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:45.106 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:19:45.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:44 smithi031 ceph-mon[16336]: osdmap e313: 12 total, 12 up, 9 in 2024-08-19T14:19:45.207 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:19:45.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:45.236 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:45.237 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:19:45.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:44 smithi032 ceph-mon[20004]: osdmap e313: 12 total, 12 up, 9 in 2024-08-19T14:19:45.338 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:19:45.359 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:45.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:45.360 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:19:45.462 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:19:45.492 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:45.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:45.493 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:19:45.594 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:19:45.628 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:45.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:45.629 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:19:45.730 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:19:45.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:45.755 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:45.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:19:45.856 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:19:45.892 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:45.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:45.893 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:19:45.994 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:19:46.027 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:46.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:46.028 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:19:46.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:45 smithi031 ceph-mon[16336]: pgmap v708: 77 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 71 active+clean; 2.3 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 1.15k op/s; 2/113991 objects degraded (0.002%); 255/113991 objects misplaced (0.224%); 7.2 MiB/s, 465 objects/s recovering 2024-08-19T14:19:46.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:45 smithi031 ceph-mon[16336]: osdmap e314: 12 total, 12 up, 9 in 2024-08-19T14:19:46.128 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:19:46.162 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:46.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:46.163 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:19:46.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:45 smithi032 ceph-mon[20004]: pgmap v708: 77 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 71 active+clean; 2.3 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 1.15k op/s; 2/113991 objects degraded (0.002%); 255/113991 objects misplaced (0.224%); 7.2 MiB/s, 465 objects/s recovering 2024-08-19T14:19:46.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:45 smithi032 ceph-mon[20004]: osdmap e314: 12 total, 12 up, 9 in 2024-08-19T14:19:46.264 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:19:46.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:45 smithi045 ceph-mon[20143]: pgmap v708: 77 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 71 active+clean; 2.3 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 1.15k op/s; 2/113991 objects degraded (0.002%); 255/113991 objects misplaced (0.224%); 7.2 MiB/s, 465 objects/s recovering 2024-08-19T14:19:46.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:45 smithi045 ceph-mon[20143]: osdmap e314: 12 total, 12 up, 9 in 2024-08-19T14:19:46.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:46.301 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:46.301 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:19:46.402 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:19:46.442 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:46.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:46.443 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:19:46.544 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:19:46.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:46.585 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:46.585 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:19:46.686 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:19:46.721 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:46.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:46.722 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:19:46.823 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:19:46.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:46.845 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:46.846 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:19:46.947 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:19:46.981 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:46.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:46.982 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:19:47.082 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:19:47.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:47.104 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:47.105 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:19:47.206 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:19:47.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:47.239 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:47.240 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:19:47.340 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:19:47.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:47.372 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:47.372 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:19:47.473 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:19:47.507 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:47.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:47.508 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:19:47.609 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:19:47.641 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:47.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:47.642 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:19:47.747 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:19:47.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:47.776 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:47.776 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:19:47.877 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:19:47.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:47.905 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:47.905 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:19:48.006 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:19:48.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:48.040 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:48.040 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:19:48.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:47 smithi031 ceph-mon[16336]: pgmap v710: 77 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 71 active+clean; 2.2 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 1.00k op/s; 2/107916 objects degraded (0.002%); 255/107916 objects misplaced (0.236%); 7.0 MiB/s, 470 objects/s recovering 2024-08-19T14:19:48.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:47 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 2/107916 objects degraded (0.002%), 2 pgs degraded (PG_DEGRADED) 2024-08-19T14:19:48.142 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:19:48.170 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:48.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:48.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:19:48.259 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:47 smithi032 ceph-mon[20004]: pgmap v710: 77 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 71 active+clean; 2.2 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 1.00k op/s; 2/107916 objects degraded (0.002%); 255/107916 objects misplaced (0.236%); 7.0 MiB/s, 470 objects/s recovering 2024-08-19T14:19:48.259 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:47 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 2/107916 objects degraded (0.002%), 2 pgs degraded (PG_DEGRADED) 2024-08-19T14:19:48.272 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:19:48.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:47 smithi045 ceph-mon[20143]: pgmap v710: 77 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 71 active+clean; 2.2 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 1.00k op/s; 2/107916 objects degraded (0.002%); 255/107916 objects misplaced (0.236%); 7.0 MiB/s, 470 objects/s recovering 2024-08-19T14:19:48.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:47 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 2/107916 objects degraded (0.002%), 2 pgs degraded (PG_DEGRADED) 2024-08-19T14:19:48.310 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:48.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:48.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:19:48.412 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:19:48.437 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:48.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:48.438 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:19:48.539 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:19:48.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:48.566 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:48.566 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:19:48.666 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:19:48.691 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:48.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:48.692 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:19:48.793 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:19:48.817 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:48.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:48.818 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:19:48.919 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:19:48.943 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:48.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:48.944 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:19:49.045 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:19:49.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:49.076 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:49.077 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:19:49.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:48 smithi031 ceph-mon[16336]: osdmap e315: 12 total, 12 up, 9 in 2024-08-19T14:19:49.178 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:19:49.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:49.201 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:49.202 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:19:49.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:48 smithi032 ceph-mon[20004]: osdmap e315: 12 total, 12 up, 9 in 2024-08-19T14:19:49.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:48 smithi045 ceph-mon[20143]: osdmap e315: 12 total, 12 up, 9 in 2024-08-19T14:19:49.302 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:19:49.325 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:49.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:49.326 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:19:49.427 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:19:49.448 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:49.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:49.449 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:19:49.550 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:19:49.573 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:49.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:49.574 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:19:49.674 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:19:49.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:49.697 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:49.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:19:49.798 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:19:49.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:49.821 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:49.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:19:49.922 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:19:49.953 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:49.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:49.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:19:50.056 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:19:50.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:50.087 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:50.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:19:50.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:49 smithi031 ceph-mon[16336]: pgmap v712: 77 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 72 active+clean; 2.1 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 1.08k op/s; 2/103677 objects degraded (0.002%); 255/103677 objects misplaced (0.246%); 5.6 MiB/s, 443 objects/s recovering 2024-08-19T14:19:50.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:49 smithi031 ceph-mon[16336]: osdmap e316: 12 total, 12 up, 9 in 2024-08-19T14:19:50.188 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:19:50.217 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:50.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:50.218 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:19:50.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:49 smithi032 ceph-mon[20004]: pgmap v712: 77 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 72 active+clean; 2.1 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 1.08k op/s; 2/103677 objects degraded (0.002%); 255/103677 objects misplaced (0.246%); 5.6 MiB/s, 443 objects/s recovering 2024-08-19T14:19:50.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:49 smithi032 ceph-mon[20004]: osdmap e316: 12 total, 12 up, 9 in 2024-08-19T14:19:50.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:49 smithi045 ceph-mon[20143]: pgmap v712: 77 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 72 active+clean; 2.1 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 1.08k op/s; 2/103677 objects degraded (0.002%); 255/103677 objects misplaced (0.246%); 5.6 MiB/s, 443 objects/s recovering 2024-08-19T14:19:50.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:49 smithi045 ceph-mon[20143]: osdmap e316: 12 total, 12 up, 9 in 2024-08-19T14:19:50.319 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:19:50.342 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:50.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:50.343 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:19:50.444 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:19:50.476 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:50.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:50.477 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:19:50.578 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:19:50.599 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:50.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:50.600 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:19:50.701 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:19:50.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:50.736 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:50.736 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:19:50.837 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:19:50.871 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:50.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:50.872 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:19:50.973 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:19:51.004 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:51.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:51.005 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:19:51.106 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:19:51.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:50 smithi031 ceph-mon[16336]: osdmap e317: 12 total, 12 up, 9 in 2024-08-19T14:19:51.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:50 smithi031 ceph-mon[16336]: 3.2c scrub starts 2024-08-19T14:19:51.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:50 smithi031 ceph-mon[16336]: 3.2c scrub ok 2024-08-19T14:19:51.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:50 smithi031 ceph-mon[16336]: 3.2a scrub starts 2024-08-19T14:19:51.138 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:51.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:51.139 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:19:51.240 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:19:51.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:50 smithi032 ceph-mon[20004]: osdmap e317: 12 total, 12 up, 9 in 2024-08-19T14:19:51.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:50 smithi032 ceph-mon[20004]: 3.2c scrub starts 2024-08-19T14:19:51.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:50 smithi032 ceph-mon[20004]: 3.2c scrub ok 2024-08-19T14:19:51.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:50 smithi032 ceph-mon[20004]: 3.2a scrub starts 2024-08-19T14:19:51.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:51.275 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:51.276 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:19:51.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:50 smithi045 ceph-mon[20143]: osdmap e317: 12 total, 12 up, 9 in 2024-08-19T14:19:51.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:50 smithi045 ceph-mon[20143]: 3.2c scrub starts 2024-08-19T14:19:51.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:50 smithi045 ceph-mon[20143]: 3.2c scrub ok 2024-08-19T14:19:51.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:50 smithi045 ceph-mon[20143]: 3.2a scrub starts 2024-08-19T14:19:51.318 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:19:51.376 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:19:51.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:51.415 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:51.415 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:19:51.515 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:19:51.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:51.545 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:51.546 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:19:51.647 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:19:51.685 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:51.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:51.686 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:19:51.787 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:19:51.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:51.820 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:51.820 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:19:51.922 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:19:51.956 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:51.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:51.957 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:19:51.993 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is unset 2024-08-19T14:19:52.058 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:19:52.078 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:52.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:52.080 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:19:52.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:51 smithi031 ceph-mon[16336]: pgmap v715: 77 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 72 active+clean; 1.9 GiB data, 9.8 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 1.10k op/s; 4/94581 objects degraded (0.004%); 1.0 MiB/s, 212 objects/s recovering 2024-08-19T14:19:52.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:51 smithi031 ceph-mon[16336]: 3.13 scrub starts 2024-08-19T14:19:52.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:51 smithi031 ceph-mon[16336]: 3.13 scrub ok 2024-08-19T14:19:52.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:51 smithi031 ceph-mon[16336]: 3.2a scrub ok 2024-08-19T14:19:52.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:51 smithi031 ceph-mon[16336]: 3.25 scrub starts 2024-08-19T14:19:52.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:51 smithi031 ceph-mon[16336]: 3.25 scrub ok 2024-08-19T14:19:52.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:51 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2458932963' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:19:52.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:51 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:19:52.146 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:51 smithi045 ceph-mon[20143]: pgmap v715: 77 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 72 active+clean; 1.9 GiB data, 9.8 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 1.10k op/s; 4/94581 objects degraded (0.004%); 1.0 MiB/s, 212 objects/s recovering 2024-08-19T14:19:52.146 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:51 smithi045 ceph-mon[20143]: 3.13 scrub starts 2024-08-19T14:19:52.146 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:51 smithi045 ceph-mon[20143]: 3.13 scrub ok 2024-08-19T14:19:52.146 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:51 smithi045 ceph-mon[20143]: 3.2a scrub ok 2024-08-19T14:19:52.146 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:51 smithi045 ceph-mon[20143]: 3.25 scrub starts 2024-08-19T14:19:52.146 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:51 smithi045 ceph-mon[20143]: 3.25 scrub ok 2024-08-19T14:19:52.146 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:51 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2458932963' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:19:52.146 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:51 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:19:52.181 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:19:52.211 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:52.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:52.212 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:19:52.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:51 smithi032 ceph-mon[20004]: pgmap v715: 77 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 72 active+clean; 1.9 GiB data, 9.8 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 1.10k op/s; 4/94581 objects degraded (0.004%); 1.0 MiB/s, 212 objects/s recovering 2024-08-19T14:19:52.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:51 smithi032 ceph-mon[20004]: 3.13 scrub starts 2024-08-19T14:19:52.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:51 smithi032 ceph-mon[20004]: 3.13 scrub ok 2024-08-19T14:19:52.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:51 smithi032 ceph-mon[20004]: 3.2a scrub ok 2024-08-19T14:19:52.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:51 smithi032 ceph-mon[20004]: 3.25 scrub starts 2024-08-19T14:19:52.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:51 smithi032 ceph-mon[20004]: 3.25 scrub ok 2024-08-19T14:19:52.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:51 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2458932963' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:19:52.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:51 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:19:52.313 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:19:52.341 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:52.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:52.343 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:19:52.443 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:19:52.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:52.480 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:52.480 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:19:52.581 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:19:52.604 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:52.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:52.605 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:19:52.706 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:19:52.735 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:52.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:52.737 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:19:52.838 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:19:52.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:52.873 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:52.873 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:19:52.974 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:19:52.997 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:52.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:52.998 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:19:53.099 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:19:53.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:52 smithi031 ceph-mon[16336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:19:53.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:52 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:19:53.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:52 smithi031 ceph-mon[16336]: osdmap e318: 12 total, 12 up, 9 in 2024-08-19T14:19:53.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:52 smithi031 ceph-mon[16336]: 1.12 scrub starts 2024-08-19T14:19:53.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:52 smithi031 ceph-mon[16336]: 1.12 scrub ok 2024-08-19T14:19:53.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:52 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 4/94581 objects degraded (0.004%), 2 pgs degraded (PG_DEGRADED) 2024-08-19T14:19:53.132 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:53.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:53.133 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:19:53.234 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:19:53.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:52 smithi032 ceph-mon[20004]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:19:53.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:52 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:19:53.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:52 smithi032 ceph-mon[20004]: osdmap e318: 12 total, 12 up, 9 in 2024-08-19T14:19:53.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:52 smithi032 ceph-mon[20004]: 1.12 scrub starts 2024-08-19T14:19:53.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:52 smithi032 ceph-mon[20004]: 1.12 scrub ok 2024-08-19T14:19:53.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:52 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 4/94581 objects degraded (0.004%), 2 pgs degraded (PG_DEGRADED) 2024-08-19T14:19:53.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:53.270 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:53.270 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:19:53.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:52 smithi045 ceph-mon[20143]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:19:53.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:52 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:19:53.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:52 smithi045 ceph-mon[20143]: osdmap e318: 12 total, 12 up, 9 in 2024-08-19T14:19:53.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:52 smithi045 ceph-mon[20143]: 1.12 scrub starts 2024-08-19T14:19:53.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:52 smithi045 ceph-mon[20143]: 1.12 scrub ok 2024-08-19T14:19:53.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:52 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 4/94581 objects degraded (0.004%), 2 pgs degraded (PG_DEGRADED) 2024-08-19T14:19:53.372 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:19:53.395 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:53.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:53.397 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:19:53.497 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:19:53.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:53.528 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:53.528 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:19:53.629 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:19:53.653 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:53.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:53.654 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:19:53.755 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:19:53.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:53.789 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:53.789 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:19:53.890 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:19:53.921 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:53.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:53.922 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:19:54.023 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:19:54.052 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:54.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:54.053 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:19:54.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:53 smithi031 ceph-mon[16336]: pgmap v717: 77 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 75 active+clean; 1.7 GiB data, 9.4 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 558 op/s; 2/85812 objects degraded (0.002%) 2024-08-19T14:19:54.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:53 smithi031 ceph-mon[16336]: 3.8 scrub starts 2024-08-19T14:19:54.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:53 smithi031 ceph-mon[16336]: 3.8 scrub ok 2024-08-19T14:19:54.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:53 smithi031 ceph-mon[16336]: 3.d deep-scrub starts 2024-08-19T14:19:54.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:53 smithi031 ceph-mon[16336]: 3.c scrub starts 2024-08-19T14:19:54.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:53 smithi031 ceph-mon[16336]: 3.c scrub ok 2024-08-19T14:19:54.154 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:19:54.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:54.184 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:54.184 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:19:54.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:53 smithi032 ceph-mon[20004]: pgmap v717: 77 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 75 active+clean; 1.7 GiB data, 9.4 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 558 op/s; 2/85812 objects degraded (0.002%) 2024-08-19T14:19:54.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:53 smithi032 ceph-mon[20004]: 3.8 scrub starts 2024-08-19T14:19:54.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:53 smithi032 ceph-mon[20004]: 3.8 scrub ok 2024-08-19T14:19:54.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:53 smithi032 ceph-mon[20004]: 3.d deep-scrub starts 2024-08-19T14:19:54.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:53 smithi032 ceph-mon[20004]: 3.c scrub starts 2024-08-19T14:19:54.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:53 smithi032 ceph-mon[20004]: 3.c scrub ok 2024-08-19T14:19:54.284 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:19:54.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:53 smithi045 ceph-mon[20143]: pgmap v717: 77 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 75 active+clean; 1.7 GiB data, 9.4 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 558 op/s; 2/85812 objects degraded (0.002%) 2024-08-19T14:19:54.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:53 smithi045 ceph-mon[20143]: 3.8 scrub starts 2024-08-19T14:19:54.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:53 smithi045 ceph-mon[20143]: 3.8 scrub ok 2024-08-19T14:19:54.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:53 smithi045 ceph-mon[20143]: 3.d deep-scrub starts 2024-08-19T14:19:54.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:53 smithi045 ceph-mon[20143]: 3.c scrub starts 2024-08-19T14:19:54.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:53 smithi045 ceph-mon[20143]: 3.c scrub ok 2024-08-19T14:19:54.306 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:54.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:54.307 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:19:54.408 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:19:54.436 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:54.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:54.437 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:19:54.515 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:19:54.537 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:19:54.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:54.563 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:54.563 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:19:54.664 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:19:54.685 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:54.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:54.686 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:19:54.787 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:19:54.811 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:54.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:54.813 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:19:54.914 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:19:54.945 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:54.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:54.946 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:19:55.047 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:19:55.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:55.082 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:55.082 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:19:55.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:54 smithi031 ceph-mon[16336]: osdmap e319: 12 total, 12 up, 9 in 2024-08-19T14:19:55.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:54 smithi031 ceph-mon[16336]: pgmap v719: 77 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 1 active+clean+scrubbing+deep, 74 active+clean; 1.6 GiB data, 8.7 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 535 op/s; 1/78837 objects degraded (0.001%) 2024-08-19T14:19:55.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:54 smithi031 ceph-mon[16336]: 3.d deep-scrub ok 2024-08-19T14:19:55.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:54 smithi031 ceph-mon[16336]: 3.5 scrub starts 2024-08-19T14:19:55.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:54 smithi031 ceph-mon[16336]: 3.5 scrub ok 2024-08-19T14:19:55.183 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:19:55.197 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:54 smithi045 ceph-mon[20143]: osdmap e319: 12 total, 12 up, 9 in 2024-08-19T14:19:55.197 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:54 smithi045 ceph-mon[20143]: pgmap v719: 77 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 1 active+clean+scrubbing+deep, 74 active+clean; 1.6 GiB data, 8.7 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 535 op/s; 1/78837 objects degraded (0.001%) 2024-08-19T14:19:55.197 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:54 smithi045 ceph-mon[20143]: 3.d deep-scrub ok 2024-08-19T14:19:55.197 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:54 smithi045 ceph-mon[20143]: 3.5 scrub starts 2024-08-19T14:19:55.197 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:54 smithi045 ceph-mon[20143]: 3.5 scrub ok 2024-08-19T14:19:55.222 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:55.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:55.223 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:19:55.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:54 smithi032 ceph-mon[20004]: osdmap e319: 12 total, 12 up, 9 in 2024-08-19T14:19:55.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:54 smithi032 ceph-mon[20004]: pgmap v719: 77 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 1 active+clean+scrubbing+deep, 74 active+clean; 1.6 GiB data, 8.7 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 535 op/s; 1/78837 objects degraded (0.001%) 2024-08-19T14:19:55.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:54 smithi032 ceph-mon[20004]: 3.d deep-scrub ok 2024-08-19T14:19:55.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:54 smithi032 ceph-mon[20004]: 3.5 scrub starts 2024-08-19T14:19:55.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:54 smithi032 ceph-mon[20004]: 3.5 scrub ok 2024-08-19T14:19:55.324 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:19:55.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:55.356 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:55.357 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:19:55.458 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:19:55.495 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:55.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:55.496 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:19:55.597 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:19:55.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:55.619 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:55.619 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:19:55.720 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:19:55.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:55.748 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:55.748 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:19:55.849 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:19:55.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:55.890 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:55.890 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:19:55.991 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:19:56.012 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:56.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:56.014 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:19:56.115 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:19:56.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:55 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:19:56.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:55 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:19:56.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:55 smithi031 ceph-mon[16336]: osdmap e320: 12 total, 12 up, 9 in 2024-08-19T14:19:56.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:55 smithi031 ceph-mon[16336]: 1.14 scrub starts 2024-08-19T14:19:56.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:55 smithi031 ceph-mon[16336]: 1.14 scrub ok 2024-08-19T14:19:56.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:55 smithi031 ceph-mon[16336]: 1.6 scrub starts 2024-08-19T14:19:56.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:55 smithi031 ceph-mon[16336]: 1.6 scrub ok 2024-08-19T14:19:56.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:56.142 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:56.143 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:19:56.246 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:19:56.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:55 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:19:56.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:55 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:19:56.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:55 smithi032 ceph-mon[20004]: osdmap e320: 12 total, 12 up, 9 in 2024-08-19T14:19:56.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:55 smithi032 ceph-mon[20004]: 1.14 scrub starts 2024-08-19T14:19:56.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:55 smithi032 ceph-mon[20004]: 1.14 scrub ok 2024-08-19T14:19:56.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:55 smithi032 ceph-mon[20004]: 1.6 scrub starts 2024-08-19T14:19:56.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:55 smithi032 ceph-mon[20004]: 1.6 scrub ok 2024-08-19T14:19:56.272 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:56.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:56.273 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:19:56.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:55 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:19:56.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:55 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:19:56.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:55 smithi045 ceph-mon[20143]: osdmap e320: 12 total, 12 up, 9 in 2024-08-19T14:19:56.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:55 smithi045 ceph-mon[20143]: 1.14 scrub starts 2024-08-19T14:19:56.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:55 smithi045 ceph-mon[20143]: 1.14 scrub ok 2024-08-19T14:19:56.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:55 smithi045 ceph-mon[20143]: 1.6 scrub starts 2024-08-19T14:19:56.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:55 smithi045 ceph-mon[20143]: 1.6 scrub ok 2024-08-19T14:19:56.374 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:19:56.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:56.398 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:56.399 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:19:56.500 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:19:56.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:56.522 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:56.523 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:19:56.624 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:19:56.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:56.647 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:56.647 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:19:56.749 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:19:56.774 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:56.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:56.776 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:19:56.877 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:19:56.905 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:56.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:56.907 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:19:57.008 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:19:57.037 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:57.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:57.038 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:19:57.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:56 smithi031 ceph-mon[16336]: 3.21 scrub starts 2024-08-19T14:19:57.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:56 smithi031 ceph-mon[16336]: 3.24 scrub starts 2024-08-19T14:19:57.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:56 smithi031 ceph-mon[16336]: osdmap e321: 12 total, 12 up, 9 in 2024-08-19T14:19:57.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:56 smithi031 ceph-mon[16336]: pgmap v722: 77 pgs: 1 creating+peering, 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing+deep, 74 active+clean; 1.5 GiB data, 8.2 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 191 op/s; 1/71685 objects degraded (0.001%) 2024-08-19T14:19:57.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:56 smithi031 ceph-mon[16336]: 3.21 scrub ok 2024-08-19T14:19:57.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:56 smithi031 ceph-mon[16336]: 3.24 scrub ok 2024-08-19T14:19:57.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:56 smithi031 ceph-mon[16336]: 1.9 scrub starts 2024-08-19T14:19:57.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:56 smithi031 ceph-mon[16336]: 1.9 scrub ok 2024-08-19T14:19:57.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:56 smithi031 ceph-mon[16336]: 3.2b scrub starts 2024-08-19T14:19:57.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:56 smithi031 ceph-mon[16336]: 3.16 scrub starts 2024-08-19T14:19:57.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:56 smithi031 ceph-mon[16336]: 3.16 scrub ok 2024-08-19T14:19:57.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:56 smithi031 ceph-mon[16336]: 3.2 deep-scrub starts 2024-08-19T14:19:57.138 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:19:57.173 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:57.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:57.174 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:19:57.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:56 smithi032 ceph-mon[20004]: 3.21 scrub starts 2024-08-19T14:19:57.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:56 smithi032 ceph-mon[20004]: 3.24 scrub starts 2024-08-19T14:19:57.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:56 smithi032 ceph-mon[20004]: osdmap e321: 12 total, 12 up, 9 in 2024-08-19T14:19:57.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:56 smithi032 ceph-mon[20004]: pgmap v722: 77 pgs: 1 creating+peering, 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing+deep, 74 active+clean; 1.5 GiB data, 8.2 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 191 op/s; 1/71685 objects degraded (0.001%) 2024-08-19T14:19:57.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:56 smithi032 ceph-mon[20004]: 3.21 scrub ok 2024-08-19T14:19:57.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:56 smithi032 ceph-mon[20004]: 3.24 scrub ok 2024-08-19T14:19:57.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:56 smithi032 ceph-mon[20004]: 1.9 scrub starts 2024-08-19T14:19:57.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:56 smithi032 ceph-mon[20004]: 1.9 scrub ok 2024-08-19T14:19:57.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:56 smithi032 ceph-mon[20004]: 3.2b scrub starts 2024-08-19T14:19:57.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:56 smithi032 ceph-mon[20004]: 3.16 scrub starts 2024-08-19T14:19:57.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:56 smithi032 ceph-mon[20004]: 3.16 scrub ok 2024-08-19T14:19:57.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:56 smithi032 ceph-mon[20004]: 3.2 deep-scrub starts 2024-08-19T14:19:57.275 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:19:57.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:56 smithi045 ceph-mon[20143]: 3.21 scrub starts 2024-08-19T14:19:57.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:56 smithi045 ceph-mon[20143]: 3.24 scrub starts 2024-08-19T14:19:57.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:56 smithi045 ceph-mon[20143]: osdmap e321: 12 total, 12 up, 9 in 2024-08-19T14:19:57.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:56 smithi045 ceph-mon[20143]: pgmap v722: 77 pgs: 1 creating+peering, 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing+deep, 74 active+clean; 1.5 GiB data, 8.2 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 191 op/s; 1/71685 objects degraded (0.001%) 2024-08-19T14:19:57.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:56 smithi045 ceph-mon[20143]: 3.21 scrub ok 2024-08-19T14:19:57.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:56 smithi045 ceph-mon[20143]: 3.24 scrub ok 2024-08-19T14:19:57.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:56 smithi045 ceph-mon[20143]: 1.9 scrub starts 2024-08-19T14:19:57.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:56 smithi045 ceph-mon[20143]: 1.9 scrub ok 2024-08-19T14:19:57.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:56 smithi045 ceph-mon[20143]: 3.2b scrub starts 2024-08-19T14:19:57.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:56 smithi045 ceph-mon[20143]: 3.16 scrub starts 2024-08-19T14:19:57.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:56 smithi045 ceph-mon[20143]: 3.16 scrub ok 2024-08-19T14:19:57.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:56 smithi045 ceph-mon[20143]: 3.2 deep-scrub starts 2024-08-19T14:19:57.303 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:57.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:57.304 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:19:57.405 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:19:57.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:57.439 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:57.440 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:19:57.541 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:19:57.562 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:57.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:57.563 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:19:57.664 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:19:57.695 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:57.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:57.696 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:19:57.797 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:19:57.826 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:57.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:57.827 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:19:57.928 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:19:57.950 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:57.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:57.951 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:19:58.052 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:19:58.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:58.086 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:58.087 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:19:58.188 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:19:58.215 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:58.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:58.217 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:19:58.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:57 smithi032 ceph-mon[20004]: 3.2b scrub ok 2024-08-19T14:19:58.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:57 smithi032 ceph-mon[20004]: 3.29 scrub starts 2024-08-19T14:19:58.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:57 smithi032 ceph-mon[20004]: 3.29 scrub ok 2024-08-19T14:19:58.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:57 smithi032 ceph-mon[20004]: 3.7 scrub starts 2024-08-19T14:19:58.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:57 smithi032 ceph-mon[20004]: 3.7 scrub ok 2024-08-19T14:19:58.259 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:57 smithi032 ceph-mon[20004]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-08-19T14:19:58.259 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:57 smithi032 ceph-mon[20004]: osdmap e322: 12 total, 12 up, 9 in 2024-08-19T14:19:58.259 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:57 smithi032 ceph-mon[20004]: 3.2 deep-scrub ok 2024-08-19T14:19:58.259 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:57 smithi032 ceph-mon[20004]: 3.a scrub starts 2024-08-19T14:19:58.259 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:57 smithi032 ceph-mon[20004]: 3.a scrub ok 2024-08-19T14:19:58.259 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:57 smithi032 ceph-mon[20004]: 1.10 scrub starts 2024-08-19T14:19:58.259 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:57 smithi032 ceph-mon[20004]: 1.10 scrub ok 2024-08-19T14:19:58.260 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:57 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 1/71685 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-08-19T14:19:58.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:57 smithi045 ceph-mon[20143]: 3.2b scrub ok 2024-08-19T14:19:58.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:57 smithi045 ceph-mon[20143]: 3.29 scrub starts 2024-08-19T14:19:58.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:57 smithi045 ceph-mon[20143]: 3.29 scrub ok 2024-08-19T14:19:58.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:57 smithi045 ceph-mon[20143]: 3.7 scrub starts 2024-08-19T14:19:58.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:57 smithi045 ceph-mon[20143]: 3.7 scrub ok 2024-08-19T14:19:58.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:57 smithi045 ceph-mon[20143]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-08-19T14:19:58.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:57 smithi045 ceph-mon[20143]: osdmap e322: 12 total, 12 up, 9 in 2024-08-19T14:19:58.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:57 smithi045 ceph-mon[20143]: 3.2 deep-scrub ok 2024-08-19T14:19:58.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:57 smithi045 ceph-mon[20143]: 3.a scrub starts 2024-08-19T14:19:58.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:57 smithi045 ceph-mon[20143]: 3.a scrub ok 2024-08-19T14:19:58.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:57 smithi045 ceph-mon[20143]: 1.10 scrub starts 2024-08-19T14:19:58.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:57 smithi045 ceph-mon[20143]: 1.10 scrub ok 2024-08-19T14:19:58.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:57 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 1/71685 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-08-19T14:19:58.318 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:19:58.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:58.348 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:58.348 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:19:58.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:57 smithi031 ceph-mon[16336]: 3.2b scrub ok 2024-08-19T14:19:58.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:57 smithi031 ceph-mon[16336]: 3.29 scrub starts 2024-08-19T14:19:58.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:57 smithi031 ceph-mon[16336]: 3.29 scrub ok 2024-08-19T14:19:58.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:57 smithi031 ceph-mon[16336]: 3.7 scrub starts 2024-08-19T14:19:58.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:57 smithi031 ceph-mon[16336]: 3.7 scrub ok 2024-08-19T14:19:58.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:57 smithi031 ceph-mon[16336]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-08-19T14:19:58.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:57 smithi031 ceph-mon[16336]: osdmap e322: 12 total, 12 up, 9 in 2024-08-19T14:19:58.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:57 smithi031 ceph-mon[16336]: 3.2 deep-scrub ok 2024-08-19T14:19:58.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:57 smithi031 ceph-mon[16336]: 3.a scrub starts 2024-08-19T14:19:58.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:57 smithi031 ceph-mon[16336]: 3.a scrub ok 2024-08-19T14:19:58.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:57 smithi031 ceph-mon[16336]: 1.10 scrub starts 2024-08-19T14:19:58.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:57 smithi031 ceph-mon[16336]: 1.10 scrub ok 2024-08-19T14:19:58.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:57 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 1/71685 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-08-19T14:19:58.449 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:19:58.482 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:58.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:58.483 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:19:58.584 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:19:58.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:58.617 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:58.617 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:19:58.717 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:19:58.755 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:58.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:58.756 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:19:58.857 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:19:58.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:58.887 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:58.887 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:19:58.988 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:19:59.015 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:59.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:59.017 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:19:59.117 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:19:59.146 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:19:59.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:59.147 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:19:59.247 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:19:59.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:58 smithi032 ceph-mon[20004]: 3.28 scrub starts 2024-08-19T14:19:59.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:58 smithi032 ceph-mon[20004]: 3.28 scrub ok 2024-08-19T14:19:59.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:58 smithi032 ceph-mon[20004]: 3.b scrub starts 2024-08-19T14:19:59.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:58 smithi032 ceph-mon[20004]: 3.b scrub ok 2024-08-19T14:19:59.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:58 smithi032 ceph-mon[20004]: pgmap v724: 77 pgs: 1 creating+peering, 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing+deep, 74 active+clean; 1.4 GiB data, 7.9 GiB used, 797 GiB / 805 GiB avail; 1/66705 objects degraded (0.001%) 2024-08-19T14:19:59.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:58 smithi032 ceph-mon[20004]: 1.0 deep-scrub starts 2024-08-19T14:19:59.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:58 smithi032 ceph-mon[20004]: 1.0 deep-scrub ok 2024-08-19T14:19:59.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:58 smithi032 ceph-mon[20004]: 3.6 deep-scrub starts 2024-08-19T14:19:59.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:58 smithi032 ceph-mon[20004]: 3.3 scrub starts 2024-08-19T14:19:59.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:58 smithi032 ceph-mon[20004]: 3.3 scrub ok 2024-08-19T14:19:59.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:58 smithi032 ceph-mon[20004]: 3.2d scrub starts 2024-08-19T14:19:59.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:58 smithi032 ceph-mon[20004]: 3.2d scrub ok 2024-08-19T14:19:59.269 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:59.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:59.270 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:19:59.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:58 smithi045 ceph-mon[20143]: 3.28 scrub starts 2024-08-19T14:19:59.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:58 smithi045 ceph-mon[20143]: 3.28 scrub ok 2024-08-19T14:19:59.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:58 smithi045 ceph-mon[20143]: 3.b scrub starts 2024-08-19T14:19:59.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:58 smithi045 ceph-mon[20143]: 3.b scrub ok 2024-08-19T14:19:59.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:58 smithi045 ceph-mon[20143]: pgmap v724: 77 pgs: 1 creating+peering, 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing+deep, 74 active+clean; 1.4 GiB data, 7.9 GiB used, 797 GiB / 805 GiB avail; 1/66705 objects degraded (0.001%) 2024-08-19T14:19:59.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:58 smithi045 ceph-mon[20143]: 1.0 deep-scrub starts 2024-08-19T14:19:59.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:58 smithi045 ceph-mon[20143]: 1.0 deep-scrub ok 2024-08-19T14:19:59.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:58 smithi045 ceph-mon[20143]: 3.6 deep-scrub starts 2024-08-19T14:19:59.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:58 smithi045 ceph-mon[20143]: 3.3 scrub starts 2024-08-19T14:19:59.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:58 smithi045 ceph-mon[20143]: 3.3 scrub ok 2024-08-19T14:19:59.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:58 smithi045 ceph-mon[20143]: 3.2d scrub starts 2024-08-19T14:19:59.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:58 smithi045 ceph-mon[20143]: 3.2d scrub ok 2024-08-19T14:19:59.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:58 smithi031 ceph-mon[16336]: 3.28 scrub starts 2024-08-19T14:19:59.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:58 smithi031 ceph-mon[16336]: 3.28 scrub ok 2024-08-19T14:19:59.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:58 smithi031 ceph-mon[16336]: 3.b scrub starts 2024-08-19T14:19:59.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:58 smithi031 ceph-mon[16336]: 3.b scrub ok 2024-08-19T14:19:59.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:58 smithi031 ceph-mon[16336]: pgmap v724: 77 pgs: 1 creating+peering, 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing+deep, 74 active+clean; 1.4 GiB data, 7.9 GiB used, 797 GiB / 805 GiB avail; 1/66705 objects degraded (0.001%) 2024-08-19T14:19:59.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:58 smithi031 ceph-mon[16336]: 1.0 deep-scrub starts 2024-08-19T14:19:59.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:58 smithi031 ceph-mon[16336]: 1.0 deep-scrub ok 2024-08-19T14:19:59.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:58 smithi031 ceph-mon[16336]: 3.6 deep-scrub starts 2024-08-19T14:19:59.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:58 smithi031 ceph-mon[16336]: 3.3 scrub starts 2024-08-19T14:19:59.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:58 smithi031 ceph-mon[16336]: 3.3 scrub ok 2024-08-19T14:19:59.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:58 smithi031 ceph-mon[16336]: 3.2d scrub starts 2024-08-19T14:19:59.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:58 smithi031 ceph-mon[16336]: 3.2d scrub ok 2024-08-19T14:19:59.370 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:19:59.406 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:59.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:59.407 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:19:59.508 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:19:59.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:59.535 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:59.535 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:19:59.637 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:19:59.669 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:19:59.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:59.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:19:59.771 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:19:59.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:59.793 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:59.793 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:19:59.894 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:19:59.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:19:59.931 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:19:59.931 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:20:00.032 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:20:00.062 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:00.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:00.063 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:20:00.164 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:20:00.200 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:00.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:00.201 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:20:00.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:59 smithi032 ceph-mon[20004]: 3.19 scrub starts 2024-08-19T14:20:00.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:19:59 smithi032 ceph-mon[20004]: 3.19 scrub ok 2024-08-19T14:20:00.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:59 smithi045 ceph-mon[20143]: 3.19 scrub starts 2024-08-19T14:20:00.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:19:59 smithi045 ceph-mon[20143]: 3.19 scrub ok 2024-08-19T14:20:00.303 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:20:00.335 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:00.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:00.336 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:20:00.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:59 smithi031 ceph-mon[16336]: 3.19 scrub starts 2024-08-19T14:20:00.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:19:59 smithi031 ceph-mon[16336]: 3.19 scrub ok 2024-08-19T14:20:00.437 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:20:00.463 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:00.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:00.464 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:20:00.565 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:20:00.594 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:00.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:00.595 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:20:00.695 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:20:00.731 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:00.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:00.732 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:20:00.834 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:20:00.861 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:00.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:00.862 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:20:00.963 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:20:01.003 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:01.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:01.004 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:20:01.105 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:20:01.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:00 smithi031 ceph-mon[16336]: 1.2 deep-scrub starts 2024-08-19T14:20:01.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:00 smithi031 ceph-mon[16336]: 1.2 deep-scrub ok 2024-08-19T14:20:01.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:00 smithi031 ceph-mon[16336]: 3.6 deep-scrub ok 2024-08-19T14:20:01.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:00 smithi031 ceph-mon[16336]: 3.4 deep-scrub starts 2024-08-19T14:20:01.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:00 smithi031 ceph-mon[16336]: 3.9 scrub starts 2024-08-19T14:20:01.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:00 smithi031 ceph-mon[16336]: pgmap v725: 77 pgs: 77 active+clean; 1.2 GiB data, 7.2 GiB used, 797 GiB / 805 GiB avail 2024-08-19T14:20:01.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:00 smithi031 ceph-mon[16336]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-08-19T14:20:01.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:00 smithi031 ceph-mon[16336]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/66705 objects degraded (0.001%), 1 pg degraded) 2024-08-19T14:20:01.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:00 smithi031 ceph-mon[16336]: Cluster is now healthy 2024-08-19T14:20:01.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:00 smithi031 ceph-mon[16336]: 3.9 scrub ok 2024-08-19T14:20:01.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:00 smithi031 ceph-mon[16336]: overall HEALTH_OK 2024-08-19T14:20:01.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:00 smithi031 ceph-mon[16336]: 3.f deep-scrub starts 2024-08-19T14:20:01.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:00 smithi031 ceph-mon[16336]: 3.26 scrub starts 2024-08-19T14:20:01.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:00 smithi031 ceph-mon[16336]: 3.26 scrub ok 2024-08-19T14:20:01.145 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:01.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:01.147 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:20:01.248 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:20:01.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:00 smithi032 ceph-mon[20004]: 1.2 deep-scrub starts 2024-08-19T14:20:01.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:00 smithi032 ceph-mon[20004]: 1.2 deep-scrub ok 2024-08-19T14:20:01.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:00 smithi032 ceph-mon[20004]: 3.6 deep-scrub ok 2024-08-19T14:20:01.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:00 smithi032 ceph-mon[20004]: 3.4 deep-scrub starts 2024-08-19T14:20:01.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:00 smithi032 ceph-mon[20004]: 3.9 scrub starts 2024-08-19T14:20:01.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:00 smithi032 ceph-mon[20004]: pgmap v725: 77 pgs: 77 active+clean; 1.2 GiB data, 7.2 GiB used, 797 GiB / 805 GiB avail 2024-08-19T14:20:01.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:00 smithi032 ceph-mon[20004]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-08-19T14:20:01.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:00 smithi032 ceph-mon[20004]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/66705 objects degraded (0.001%), 1 pg degraded) 2024-08-19T14:20:01.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:00 smithi032 ceph-mon[20004]: Cluster is now healthy 2024-08-19T14:20:01.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:00 smithi032 ceph-mon[20004]: 3.9 scrub ok 2024-08-19T14:20:01.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:00 smithi032 ceph-mon[20004]: overall HEALTH_OK 2024-08-19T14:20:01.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:00 smithi032 ceph-mon[20004]: 3.f deep-scrub starts 2024-08-19T14:20:01.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:00 smithi032 ceph-mon[20004]: 3.26 scrub starts 2024-08-19T14:20:01.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:00 smithi032 ceph-mon[20004]: 3.26 scrub ok 2024-08-19T14:20:01.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:01.284 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:01.284 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:20:01.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:00 smithi045 ceph-mon[20143]: 1.2 deep-scrub starts 2024-08-19T14:20:01.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:00 smithi045 ceph-mon[20143]: 1.2 deep-scrub ok 2024-08-19T14:20:01.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:00 smithi045 ceph-mon[20143]: 3.6 deep-scrub ok 2024-08-19T14:20:01.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:00 smithi045 ceph-mon[20143]: 3.4 deep-scrub starts 2024-08-19T14:20:01.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:00 smithi045 ceph-mon[20143]: 3.9 scrub starts 2024-08-19T14:20:01.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:00 smithi045 ceph-mon[20143]: pgmap v725: 77 pgs: 77 active+clean; 1.2 GiB data, 7.2 GiB used, 797 GiB / 805 GiB avail 2024-08-19T14:20:01.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:00 smithi045 ceph-mon[20143]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-08-19T14:20:01.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:00 smithi045 ceph-mon[20143]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/66705 objects degraded (0.001%), 1 pg degraded) 2024-08-19T14:20:01.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:00 smithi045 ceph-mon[20143]: Cluster is now healthy 2024-08-19T14:20:01.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:00 smithi045 ceph-mon[20143]: 3.9 scrub ok 2024-08-19T14:20:01.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:00 smithi045 ceph-mon[20143]: overall HEALTH_OK 2024-08-19T14:20:01.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:00 smithi045 ceph-mon[20143]: 3.f deep-scrub starts 2024-08-19T14:20:01.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:00 smithi045 ceph-mon[20143]: 3.26 scrub starts 2024-08-19T14:20:01.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:00 smithi045 ceph-mon[20143]: 3.26 scrub ok 2024-08-19T14:20:01.385 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:20:01.410 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:01.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:01.411 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:20:01.511 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:20:01.535 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:01.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:01.536 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:20:01.637 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:20:01.673 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:01.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:01.674 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:20:01.776 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:20:01.805 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:01.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:01.806 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:20:01.907 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:20:01.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:01.939 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:01.940 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:20:02.041 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:20:02.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:02.071 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:02.071 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:20:02.172 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:20:02.199 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:02.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:02.200 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:20:02.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:01 smithi032 ceph-mon[20004]: 3.4 deep-scrub ok 2024-08-19T14:20:02.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:01 smithi032 ceph-mon[20004]: 3.27 scrub starts 2024-08-19T14:20:02.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:01 smithi032 ceph-mon[20004]: 3.1d scrub starts 2024-08-19T14:20:02.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:01 smithi032 ceph-mon[20004]: 3.27 scrub ok 2024-08-19T14:20:02.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:01 smithi032 ceph-mon[20004]: 3.1d scrub ok 2024-08-19T14:20:02.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:01 smithi032 ceph-mon[20004]: 3.1c scrub starts 2024-08-19T14:20:02.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:01 smithi032 ceph-mon[20004]: 3.1c scrub ok 2024-08-19T14:20:02.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:01 smithi045 ceph-mon[20143]: 3.4 deep-scrub ok 2024-08-19T14:20:02.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:01 smithi045 ceph-mon[20143]: 3.27 scrub starts 2024-08-19T14:20:02.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:01 smithi045 ceph-mon[20143]: 3.1d scrub starts 2024-08-19T14:20:02.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:01 smithi045 ceph-mon[20143]: 3.27 scrub ok 2024-08-19T14:20:02.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:01 smithi045 ceph-mon[20143]: 3.1d scrub ok 2024-08-19T14:20:02.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:01 smithi045 ceph-mon[20143]: 3.1c scrub starts 2024-08-19T14:20:02.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:01 smithi045 ceph-mon[20143]: 3.1c scrub ok 2024-08-19T14:20:02.300 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:20:02.325 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:02.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:02.326 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:20:02.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:01 smithi031 ceph-mon[16336]: 3.4 deep-scrub ok 2024-08-19T14:20:02.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:01 smithi031 ceph-mon[16336]: 3.27 scrub starts 2024-08-19T14:20:02.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:01 smithi031 ceph-mon[16336]: 3.1d scrub starts 2024-08-19T14:20:02.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:01 smithi031 ceph-mon[16336]: 3.27 scrub ok 2024-08-19T14:20:02.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:01 smithi031 ceph-mon[16336]: 3.1d scrub ok 2024-08-19T14:20:02.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:01 smithi031 ceph-mon[16336]: 3.1c scrub starts 2024-08-19T14:20:02.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:01 smithi031 ceph-mon[16336]: 3.1c scrub ok 2024-08-19T14:20:02.427 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:20:02.453 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:02.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:02.454 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:20:02.555 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:20:02.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:02.590 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:02.590 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:20:02.691 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:20:02.712 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:02.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:02.713 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:20:02.814 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:20:02.839 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:02.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:02.840 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:20:02.905 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:20:02.941 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:20:02.975 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:02.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:02.976 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:20:03.077 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:20:03.107 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:03.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:03.108 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:20:03.210 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:20:03.240 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:03.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:03.241 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:20:03.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:02 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:20:03.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:02 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:20:03.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:02 smithi032 ceph-mon[20004]: 3.0 scrub starts 2024-08-19T14:20:03.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:02 smithi032 ceph-mon[20004]: 3.0 scrub ok 2024-08-19T14:20:03.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:02 smithi032 ceph-mon[20004]: pgmap v726: 77 pgs: 77 active+clean; 1014 MiB data, 6.5 GiB used, 798 GiB / 805 GiB avail 2024-08-19T14:20:03.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:02 smithi032 ceph-mon[20004]: osdmap e323: 12 total, 12 up, 9 in 2024-08-19T14:20:03.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:02 smithi032 ceph-mon[20004]: 1.d scrub starts 2024-08-19T14:20:03.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:02 smithi032 ceph-mon[20004]: 1.d scrub ok 2024-08-19T14:20:03.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:02 smithi032 ceph-mon[20004]: 1.f scrub starts 2024-08-19T14:20:03.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:02 smithi032 ceph-mon[20004]: 1.f scrub ok 2024-08-19T14:20:03.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:02 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:20:03.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:02 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:20:03.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:02 smithi045 ceph-mon[20143]: 3.0 scrub starts 2024-08-19T14:20:03.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:02 smithi045 ceph-mon[20143]: 3.0 scrub ok 2024-08-19T14:20:03.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:02 smithi045 ceph-mon[20143]: pgmap v726: 77 pgs: 77 active+clean; 1014 MiB data, 6.5 GiB used, 798 GiB / 805 GiB avail 2024-08-19T14:20:03.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:02 smithi045 ceph-mon[20143]: osdmap e323: 12 total, 12 up, 9 in 2024-08-19T14:20:03.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:02 smithi045 ceph-mon[20143]: 1.d scrub starts 2024-08-19T14:20:03.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:02 smithi045 ceph-mon[20143]: 1.d scrub ok 2024-08-19T14:20:03.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:02 smithi045 ceph-mon[20143]: 1.f scrub starts 2024-08-19T14:20:03.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:02 smithi045 ceph-mon[20143]: 1.f scrub ok 2024-08-19T14:20:03.342 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:20:03.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:02 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:20:03.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:02 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:20:03.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:02 smithi031 ceph-mon[16336]: 3.0 scrub starts 2024-08-19T14:20:03.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:02 smithi031 ceph-mon[16336]: 3.0 scrub ok 2024-08-19T14:20:03.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:02 smithi031 ceph-mon[16336]: pgmap v726: 77 pgs: 77 active+clean; 1014 MiB data, 6.5 GiB used, 798 GiB / 805 GiB avail 2024-08-19T14:20:03.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:02 smithi031 ceph-mon[16336]: osdmap e323: 12 total, 12 up, 9 in 2024-08-19T14:20:03.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:02 smithi031 ceph-mon[16336]: 1.d scrub starts 2024-08-19T14:20:03.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:02 smithi031 ceph-mon[16336]: 1.d scrub ok 2024-08-19T14:20:03.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:02 smithi031 ceph-mon[16336]: 1.f scrub starts 2024-08-19T14:20:03.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:02 smithi031 ceph-mon[16336]: 1.f scrub ok 2024-08-19T14:20:03.379 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:03.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:03.380 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:20:03.481 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:20:03.511 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:03.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:03.512 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:20:03.614 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:20:03.646 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:03.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:03.647 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:20:03.748 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:20:03.779 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:03.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:03.780 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:20:03.881 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:20:03.904 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:03.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:03.905 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:20:04.006 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:20:04.036 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:04.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:04.037 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:20:04.138 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:20:04.162 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:04.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:04.163 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:20:04.234 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:03 smithi031 ceph-mon[16336]: 3.f deep-scrub ok 2024-08-19T14:20:04.235 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:03 smithi031 ceph-mon[16336]: 3.15 deep-scrub starts 2024-08-19T14:20:04.235 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:03 smithi031 ceph-mon[16336]: 3.12 scrub starts 2024-08-19T14:20:04.235 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:03 smithi031 ceph-mon[16336]: 3.1b scrub starts 2024-08-19T14:20:04.235 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:03 smithi031 ceph-mon[16336]: 3.1b scrub ok 2024-08-19T14:20:04.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:03 smithi032 ceph-mon[20004]: 3.f deep-scrub ok 2024-08-19T14:20:04.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:03 smithi032 ceph-mon[20004]: 3.15 deep-scrub starts 2024-08-19T14:20:04.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:03 smithi032 ceph-mon[20004]: 3.12 scrub starts 2024-08-19T14:20:04.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:03 smithi032 ceph-mon[20004]: 3.1b scrub starts 2024-08-19T14:20:04.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:03 smithi032 ceph-mon[20004]: 3.1b scrub ok 2024-08-19T14:20:04.264 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:20:04.292 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:04.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:04.293 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:20:04.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:03 smithi045 ceph-mon[20143]: 3.f deep-scrub ok 2024-08-19T14:20:04.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:03 smithi045 ceph-mon[20143]: 3.15 deep-scrub starts 2024-08-19T14:20:04.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:03 smithi045 ceph-mon[20143]: 3.12 scrub starts 2024-08-19T14:20:04.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:03 smithi045 ceph-mon[20143]: 3.1b scrub starts 2024-08-19T14:20:04.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:03 smithi045 ceph-mon[20143]: 3.1b scrub ok 2024-08-19T14:20:04.394 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:20:04.422 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:04.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:04.423 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:20:04.524 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:20:04.548 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:04.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:04.549 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:20:04.650 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:20:04.679 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:04.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:04.680 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:20:04.781 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:20:04.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:04.803 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:04.803 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:20:04.904 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:20:04.939 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:04.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:04.941 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:20:04.947 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-19T14:20:05.042 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:20:05.070 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:05.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:05.071 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:20:05.172 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:20:05.205 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:05.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:05.206 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:20:05.242 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:20:05.242 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:20:05.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:04 smithi031 ceph-mon[16336]: 3.12 scrub ok 2024-08-19T14:20:05.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:04 smithi031 ceph-mon[16336]: 1.3 scrub starts 2024-08-19T14:20:05.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:04 smithi031 ceph-mon[16336]: 1.3 scrub ok 2024-08-19T14:20:05.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:04 smithi031 ceph-mon[16336]: 3.18 scrub starts 2024-08-19T14:20:05.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:04 smithi031 ceph-mon[16336]: 3.18 scrub ok 2024-08-19T14:20:05.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:04 smithi031 ceph-mon[16336]: pgmap v728: 77 pgs: 77 active+clean; 873 MiB data, 5.9 GiB used, 799 GiB / 805 GiB avail 2024-08-19T14:20:05.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:04 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:20:05.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:04 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:20:05.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:04 smithi031 ceph-mon[16336]: 3.1 scrub starts 2024-08-19T14:20:05.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:04 smithi031 ceph-mon[16336]: 3.1 scrub ok 2024-08-19T14:20:05.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:04 smithi032 ceph-mon[20004]: 3.12 scrub ok 2024-08-19T14:20:05.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:04 smithi032 ceph-mon[20004]: 1.3 scrub starts 2024-08-19T14:20:05.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:04 smithi032 ceph-mon[20004]: 1.3 scrub ok 2024-08-19T14:20:05.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:04 smithi032 ceph-mon[20004]: 3.18 scrub starts 2024-08-19T14:20:05.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:04 smithi032 ceph-mon[20004]: 3.18 scrub ok 2024-08-19T14:20:05.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:04 smithi032 ceph-mon[20004]: pgmap v728: 77 pgs: 77 active+clean; 873 MiB data, 5.9 GiB used, 799 GiB / 805 GiB avail 2024-08-19T14:20:05.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:04 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:20:05.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:04 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:20:05.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:04 smithi032 ceph-mon[20004]: 3.1 scrub starts 2024-08-19T14:20:05.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:04 smithi032 ceph-mon[20004]: 3.1 scrub ok 2024-08-19T14:20:05.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:04 smithi045 ceph-mon[20143]: 3.12 scrub ok 2024-08-19T14:20:05.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:04 smithi045 ceph-mon[20143]: 1.3 scrub starts 2024-08-19T14:20:05.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:04 smithi045 ceph-mon[20143]: 1.3 scrub ok 2024-08-19T14:20:05.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:04 smithi045 ceph-mon[20143]: 3.18 scrub starts 2024-08-19T14:20:05.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:04 smithi045 ceph-mon[20143]: 3.18 scrub ok 2024-08-19T14:20:05.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:04 smithi045 ceph-mon[20143]: pgmap v728: 77 pgs: 77 active+clean; 873 MiB data, 5.9 GiB used, 799 GiB / 805 GiB avail 2024-08-19T14:20:05.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:04 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:20:05.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:04 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:20:05.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:04 smithi045 ceph-mon[20143]: 3.1 scrub starts 2024-08-19T14:20:05.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:04 smithi045 ceph-mon[20143]: 3.1 scrub ok 2024-08-19T14:20:05.306 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:20:05.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:05.329 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:05.329 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:20:05.429 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:20:05.459 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:05.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:05.460 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:20:05.562 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:20:05.596 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:05.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:05.597 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:20:05.698 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:20:05.732 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:05.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:05.733 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:20:05.834 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:20:05.869 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:05.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:05.870 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:20:05.971 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:20:06.002 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:06.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:06.003 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:20:06.104 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:20:06.130 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:06.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:06.131 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:20:06.232 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:20:06.253 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:06.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:06.255 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:20:06.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:05 smithi032 ceph-mon[20004]: 3.15 deep-scrub ok 2024-08-19T14:20:06.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:05 smithi032 ceph-mon[20004]: 1.4 scrub starts 2024-08-19T14:20:06.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:05 smithi032 ceph-mon[20004]: 1.4 scrub ok 2024-08-19T14:20:06.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:05 smithi032 ceph-mon[20004]: 3.1e scrub starts 2024-08-19T14:20:06.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:05 smithi032 ceph-mon[20004]: 3.1e scrub ok 2024-08-19T14:20:06.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:05 smithi032 ceph-mon[20004]: osdmap e324: 12 total, 12 up, 9 in 2024-08-19T14:20:06.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:05 smithi045 ceph-mon[20143]: 3.15 deep-scrub ok 2024-08-19T14:20:06.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:05 smithi045 ceph-mon[20143]: 1.4 scrub starts 2024-08-19T14:20:06.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:05 smithi045 ceph-mon[20143]: 1.4 scrub ok 2024-08-19T14:20:06.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:05 smithi045 ceph-mon[20143]: 3.1e scrub starts 2024-08-19T14:20:06.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:05 smithi045 ceph-mon[20143]: 3.1e scrub ok 2024-08-19T14:20:06.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:05 smithi045 ceph-mon[20143]: osdmap e324: 12 total, 12 up, 9 in 2024-08-19T14:20:06.356 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:20:06.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:05 smithi031 ceph-mon[16336]: 3.15 deep-scrub ok 2024-08-19T14:20:06.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:05 smithi031 ceph-mon[16336]: 1.4 scrub starts 2024-08-19T14:20:06.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:05 smithi031 ceph-mon[16336]: 1.4 scrub ok 2024-08-19T14:20:06.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:05 smithi031 ceph-mon[16336]: 3.1e scrub starts 2024-08-19T14:20:06.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:05 smithi031 ceph-mon[16336]: 3.1e scrub ok 2024-08-19T14:20:06.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:05 smithi031 ceph-mon[16336]: osdmap e324: 12 total, 12 up, 9 in 2024-08-19T14:20:06.386 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:06.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:06.387 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:20:06.487 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:20:06.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:06.523 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:06.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:20:06.625 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:20:06.657 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:06.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:06.658 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:20:06.759 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:20:06.792 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:06.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:06.793 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:20:06.894 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:20:06.924 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:06.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:06.925 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:20:07.026 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:20:07.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:07.062 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:07.063 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:20:07.164 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:20:07.198 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:07.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:07.199 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:20:07.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:06 smithi032 ceph-mon[20004]: pgmap v730: 77 pgs: 77 active+clean; 720 MiB data, 5.6 GiB used, 799 GiB / 805 GiB avail 2024-08-19T14:20:07.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:06 smithi032 ceph-mon[20004]: 1.11 scrub starts 2024-08-19T14:20:07.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:06 smithi032 ceph-mon[20004]: 1.11 scrub ok 2024-08-19T14:20:07.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:06 smithi045 ceph-mon[20143]: pgmap v730: 77 pgs: 77 active+clean; 720 MiB data, 5.6 GiB used, 799 GiB / 805 GiB avail 2024-08-19T14:20:07.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:06 smithi045 ceph-mon[20143]: 1.11 scrub starts 2024-08-19T14:20:07.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:06 smithi045 ceph-mon[20143]: 1.11 scrub ok 2024-08-19T14:20:07.300 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:20:07.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:07.330 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:07.330 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:20:07.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:06 smithi031 ceph-mon[16336]: pgmap v730: 77 pgs: 77 active+clean; 720 MiB data, 5.6 GiB used, 799 GiB / 805 GiB avail 2024-08-19T14:20:07.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:06 smithi031 ceph-mon[16336]: 1.11 scrub starts 2024-08-19T14:20:07.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:06 smithi031 ceph-mon[16336]: 1.11 scrub ok 2024-08-19T14:20:07.431 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:20:07.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:07.470 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:07.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:20:07.572 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:20:07.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:07.599 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:07.599 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:20:07.700 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:20:07.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:07.732 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:07.732 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:20:07.834 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:20:07.863 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:07.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:07.864 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:20:07.965 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:20:07.997 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:07.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:07.998 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:20:08.099 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:20:08.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:07 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:20:08.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:07 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:20:08.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:07 smithi031 ceph-mon[16336]: osdmap e325: 12 total, 12 up, 9 in 2024-08-19T14:20:08.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:07 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:20:08.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:07 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:20:08.135 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:08.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:08.138 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:20:08.239 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:20:08.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:07 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:20:08.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:07 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:20:08.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:07 smithi032 ceph-mon[20004]: osdmap e325: 12 total, 12 up, 9 in 2024-08-19T14:20:08.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:07 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:20:08.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:07 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:20:08.274 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:08.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:08.275 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:20:08.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:07 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:20:08.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:07 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:20:08.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:07 smithi045 ceph-mon[20143]: osdmap e325: 12 total, 12 up, 9 in 2024-08-19T14:20:08.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:07 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:20:08.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:07 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:20:08.376 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:20:08.411 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:08.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:08.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:20:08.513 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:20:08.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:08.538 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:08.538 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:20:08.639 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:20:08.668 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:08.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:08.670 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:20:08.771 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:20:08.801 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:08.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:08.802 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:20:08.903 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:20:08.937 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:08.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:08.938 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:20:09.039 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:20:09.060 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:09.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:09.061 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:20:09.162 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:20:09.196 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:09.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:09.197 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:20:09.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:08 smithi032 ceph-mon[20004]: 3.20 deep-scrub starts 2024-08-19T14:20:09.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:08 smithi032 ceph-mon[20004]: 1.13 scrub starts 2024-08-19T14:20:09.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:08 smithi032 ceph-mon[20004]: 1.13 scrub ok 2024-08-19T14:20:09.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:08 smithi032 ceph-mon[20004]: 1.c deep-scrub starts 2024-08-19T14:20:09.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:08 smithi032 ceph-mon[20004]: 3.1a scrub starts 2024-08-19T14:20:09.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:08 smithi032 ceph-mon[20004]: 3.20 deep-scrub ok 2024-08-19T14:20:09.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:08 smithi032 ceph-mon[20004]: 1.c deep-scrub ok 2024-08-19T14:20:09.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:08 smithi032 ceph-mon[20004]: 3.1a scrub ok 2024-08-19T14:20:09.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:08 smithi032 ceph-mon[20004]: pgmap v732: 77 pgs: 77 active+clean; 608 MiB data, 5.3 GiB used, 799 GiB / 805 GiB avail 2024-08-19T14:20:09.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:08 smithi032 ceph-mon[20004]: osdmap e326: 12 total, 12 up, 9 in 2024-08-19T14:20:09.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:08 smithi032 ceph-mon[20004]: 3.23 scrub starts 2024-08-19T14:20:09.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:08 smithi032 ceph-mon[20004]: 3.23 scrub ok 2024-08-19T14:20:09.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:08 smithi045 ceph-mon[20143]: 3.20 deep-scrub starts 2024-08-19T14:20:09.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:08 smithi045 ceph-mon[20143]: 1.13 scrub starts 2024-08-19T14:20:09.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:08 smithi045 ceph-mon[20143]: 1.13 scrub ok 2024-08-19T14:20:09.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:08 smithi045 ceph-mon[20143]: 1.c deep-scrub starts 2024-08-19T14:20:09.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:08 smithi045 ceph-mon[20143]: 3.1a scrub starts 2024-08-19T14:20:09.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:08 smithi045 ceph-mon[20143]: 3.20 deep-scrub ok 2024-08-19T14:20:09.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:08 smithi045 ceph-mon[20143]: 1.c deep-scrub ok 2024-08-19T14:20:09.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:08 smithi045 ceph-mon[20143]: 3.1a scrub ok 2024-08-19T14:20:09.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:08 smithi045 ceph-mon[20143]: pgmap v732: 77 pgs: 77 active+clean; 608 MiB data, 5.3 GiB used, 799 GiB / 805 GiB avail 2024-08-19T14:20:09.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:08 smithi045 ceph-mon[20143]: osdmap e326: 12 total, 12 up, 9 in 2024-08-19T14:20:09.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:08 smithi045 ceph-mon[20143]: 3.23 scrub starts 2024-08-19T14:20:09.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:08 smithi045 ceph-mon[20143]: 3.23 scrub ok 2024-08-19T14:20:09.297 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:20:09.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:09.327 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:09.327 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:20:09.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:08 smithi031 ceph-mon[16336]: 3.20 deep-scrub starts 2024-08-19T14:20:09.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:08 smithi031 ceph-mon[16336]: 1.13 scrub starts 2024-08-19T14:20:09.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:08 smithi031 ceph-mon[16336]: 1.13 scrub ok 2024-08-19T14:20:09.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:08 smithi031 ceph-mon[16336]: 1.c deep-scrub starts 2024-08-19T14:20:09.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:08 smithi031 ceph-mon[16336]: 3.1a scrub starts 2024-08-19T14:20:09.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:08 smithi031 ceph-mon[16336]: 3.20 deep-scrub ok 2024-08-19T14:20:09.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:08 smithi031 ceph-mon[16336]: 1.c deep-scrub ok 2024-08-19T14:20:09.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:08 smithi031 ceph-mon[16336]: 3.1a scrub ok 2024-08-19T14:20:09.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:08 smithi031 ceph-mon[16336]: pgmap v732: 77 pgs: 77 active+clean; 608 MiB data, 5.3 GiB used, 799 GiB / 805 GiB avail 2024-08-19T14:20:09.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:08 smithi031 ceph-mon[16336]: osdmap e326: 12 total, 12 up, 9 in 2024-08-19T14:20:09.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:08 smithi031 ceph-mon[16336]: 3.23 scrub starts 2024-08-19T14:20:09.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:08 smithi031 ceph-mon[16336]: 3.23 scrub ok 2024-08-19T14:20:09.428 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:20:09.459 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:09.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:09.460 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:20:09.561 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:20:09.582 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:09.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:09.583 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:20:09.684 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:20:09.723 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:09.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:09.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:20:09.825 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:20:09.851 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:09.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:09.852 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:20:09.953 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:20:09.978 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:09.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:09.979 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:20:10.081 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:20:10.116 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:10.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:10.117 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:20:10.218 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:20:10.253 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:10.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:10.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:20:10.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:09 smithi032 ceph-mon[20004]: 1.16 scrub starts 2024-08-19T14:20:10.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:09 smithi032 ceph-mon[20004]: 1.16 scrub ok 2024-08-19T14:20:10.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:09 smithi032 ceph-mon[20004]: 3.e scrub starts 2024-08-19T14:20:10.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:09 smithi032 ceph-mon[20004]: 1.1c scrub starts 2024-08-19T14:20:10.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:09 smithi032 ceph-mon[20004]: 1.1c scrub ok 2024-08-19T14:20:10.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:09 smithi032 ceph-mon[20004]: 3.e scrub ok 2024-08-19T14:20:10.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:09 smithi032 ceph-mon[20004]: 3.17 scrub starts 2024-08-19T14:20:10.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:09 smithi032 ceph-mon[20004]: 3.17 scrub ok 2024-08-19T14:20:10.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:09 smithi032 ceph-mon[20004]: 1.15 deep-scrub starts 2024-08-19T14:20:10.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:09 smithi032 ceph-mon[20004]: 1.15 deep-scrub ok 2024-08-19T14:20:10.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:09 smithi032 ceph-mon[20004]: 1.7 scrub starts 2024-08-19T14:20:10.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:09 smithi032 ceph-mon[20004]: 1.7 scrub ok 2024-08-19T14:20:10.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:09 smithi032 ceph-mon[20004]: 1.1 scrub starts 2024-08-19T14:20:10.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:09 smithi032 ceph-mon[20004]: 1.1 scrub ok 2024-08-19T14:20:10.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:09 smithi045 ceph-mon[20143]: 1.16 scrub starts 2024-08-19T14:20:10.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:09 smithi045 ceph-mon[20143]: 1.16 scrub ok 2024-08-19T14:20:10.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:09 smithi045 ceph-mon[20143]: 3.e scrub starts 2024-08-19T14:20:10.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:09 smithi045 ceph-mon[20143]: 1.1c scrub starts 2024-08-19T14:20:10.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:09 smithi045 ceph-mon[20143]: 1.1c scrub ok 2024-08-19T14:20:10.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:09 smithi045 ceph-mon[20143]: 3.e scrub ok 2024-08-19T14:20:10.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:09 smithi045 ceph-mon[20143]: 3.17 scrub starts 2024-08-19T14:20:10.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:09 smithi045 ceph-mon[20143]: 3.17 scrub ok 2024-08-19T14:20:10.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:09 smithi045 ceph-mon[20143]: 1.15 deep-scrub starts 2024-08-19T14:20:10.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:09 smithi045 ceph-mon[20143]: 1.15 deep-scrub ok 2024-08-19T14:20:10.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:09 smithi045 ceph-mon[20143]: 1.7 scrub starts 2024-08-19T14:20:10.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:09 smithi045 ceph-mon[20143]: 1.7 scrub ok 2024-08-19T14:20:10.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:09 smithi045 ceph-mon[20143]: 1.1 scrub starts 2024-08-19T14:20:10.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:09 smithi045 ceph-mon[20143]: 1.1 scrub ok 2024-08-19T14:20:10.354 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:20:10.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:09 smithi031 ceph-mon[16336]: 1.16 scrub starts 2024-08-19T14:20:10.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:09 smithi031 ceph-mon[16336]: 1.16 scrub ok 2024-08-19T14:20:10.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:09 smithi031 ceph-mon[16336]: 3.e scrub starts 2024-08-19T14:20:10.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:09 smithi031 ceph-mon[16336]: 1.1c scrub starts 2024-08-19T14:20:10.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:09 smithi031 ceph-mon[16336]: 1.1c scrub ok 2024-08-19T14:20:10.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:09 smithi031 ceph-mon[16336]: 3.e scrub ok 2024-08-19T14:20:10.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:09 smithi031 ceph-mon[16336]: 3.17 scrub starts 2024-08-19T14:20:10.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:09 smithi031 ceph-mon[16336]: 3.17 scrub ok 2024-08-19T14:20:10.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:09 smithi031 ceph-mon[16336]: 1.15 deep-scrub starts 2024-08-19T14:20:10.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:09 smithi031 ceph-mon[16336]: 1.15 deep-scrub ok 2024-08-19T14:20:10.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:09 smithi031 ceph-mon[16336]: 1.7 scrub starts 2024-08-19T14:20:10.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:09 smithi031 ceph-mon[16336]: 1.7 scrub ok 2024-08-19T14:20:10.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:09 smithi031 ceph-mon[16336]: 1.1 scrub starts 2024-08-19T14:20:10.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:09 smithi031 ceph-mon[16336]: 1.1 scrub ok 2024-08-19T14:20:10.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:10.382 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:10.383 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:20:10.484 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:20:10.513 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:10.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:10.514 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:20:10.615 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:20:10.642 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:10.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:10.643 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:20:10.745 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:20:10.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:10.767 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:10.767 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:20:10.867 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:20:10.897 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:10.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:10.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:20:10.999 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:20:11.020 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:11.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:11.021 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:20:11.122 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:20:11.153 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:11.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:11.155 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:20:11.256 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:20:11.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:10 smithi032 ceph-mon[20004]: 3.22 deep-scrub starts 2024-08-19T14:20:11.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:10 smithi032 ceph-mon[20004]: pgmap v734: 77 pgs: 77 active+clean; 380 MiB data, 4.5 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 965 op/s 2024-08-19T14:20:11.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:10 smithi032 ceph-mon[20004]: 3.22 deep-scrub ok 2024-08-19T14:20:11.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:10 smithi032 ceph-mon[20004]: 3.14 scrub starts 2024-08-19T14:20:11.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:10 smithi032 ceph-mon[20004]: 3.14 scrub ok 2024-08-19T14:20:11.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:11.277 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:11.277 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:20:11.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:10 smithi045 ceph-mon[20143]: 3.22 deep-scrub starts 2024-08-19T14:20:11.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:10 smithi045 ceph-mon[20143]: pgmap v734: 77 pgs: 77 active+clean; 380 MiB data, 4.5 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 965 op/s 2024-08-19T14:20:11.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:10 smithi045 ceph-mon[20143]: 3.22 deep-scrub ok 2024-08-19T14:20:11.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:10 smithi045 ceph-mon[20143]: 3.14 scrub starts 2024-08-19T14:20:11.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:10 smithi045 ceph-mon[20143]: 3.14 scrub ok 2024-08-19T14:20:11.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:10 smithi031 ceph-mon[16336]: 3.22 deep-scrub starts 2024-08-19T14:20:11.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:10 smithi031 ceph-mon[16336]: pgmap v734: 77 pgs: 77 active+clean; 380 MiB data, 4.5 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 965 op/s 2024-08-19T14:20:11.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:10 smithi031 ceph-mon[16336]: 3.22 deep-scrub ok 2024-08-19T14:20:11.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:10 smithi031 ceph-mon[16336]: 3.14 scrub starts 2024-08-19T14:20:11.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:10 smithi031 ceph-mon[16336]: 3.14 scrub ok 2024-08-19T14:20:11.378 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:20:11.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:11.415 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:11.416 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:20:11.516 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:20:11.542 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:11.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:11.543 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:20:11.571 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Removed 131176 objects 2024-08-19T14:20:11.571 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Clean up completed and total clean up time :99.2485 2024-08-19T14:20:11.644 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:20:11.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:11.667 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:11.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:20:11.768 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:20:11.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:11.791 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:11.791 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:20:11.892 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:20:11.914 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:11.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:11.915 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:20:12.016 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:20:12.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:12.039 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:12.039 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:20:12.140 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:20:12.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:12.164 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:12.164 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:20:12.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:11 smithi032 ceph-mon[20004]: 3.11 scrub starts 2024-08-19T14:20:12.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:11 smithi032 ceph-mon[20004]: 3.11 scrub ok 2024-08-19T14:20:12.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:11 smithi032 ceph-mon[20004]: 1.8 scrub starts 2024-08-19T14:20:12.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:11 smithi032 ceph-mon[20004]: 1.8 scrub ok 2024-08-19T14:20:12.265 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:20:12.287 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:12.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:12.288 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:20:12.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:11 smithi045 ceph-mon[20143]: 3.11 scrub starts 2024-08-19T14:20:12.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:11 smithi045 ceph-mon[20143]: 3.11 scrub ok 2024-08-19T14:20:12.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:11 smithi045 ceph-mon[20143]: 1.8 scrub starts 2024-08-19T14:20:12.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:11 smithi045 ceph-mon[20143]: 1.8 scrub ok 2024-08-19T14:20:12.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:11 smithi031 ceph-mon[16336]: 3.11 scrub starts 2024-08-19T14:20:12.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:11 smithi031 ceph-mon[16336]: 3.11 scrub ok 2024-08-19T14:20:12.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:11 smithi031 ceph-mon[16336]: 1.8 scrub starts 2024-08-19T14:20:12.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:11 smithi031 ceph-mon[16336]: 1.8 scrub ok 2024-08-19T14:20:12.389 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:20:12.414 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:12.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:12.415 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:20:12.516 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:20:12.539 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:12.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:12.540 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:20:12.641 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:20:12.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:12.668 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:12.669 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:20:12.770 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:20:12.795 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:12.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:12.796 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:20:12.898 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:20:12.923 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:12.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:12.924 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:20:13.025 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:20:13.050 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:13.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:13.052 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:20:13.153 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:20:13.167 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:20:13.178 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:13.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:13.179 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:20:13.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:12 smithi031 ceph-mon[16336]: 1.b scrub starts 2024-08-19T14:20:13.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:12 smithi031 ceph-mon[16336]: 1.b scrub ok 2024-08-19T14:20:13.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:12 smithi031 ceph-mon[16336]: pgmap v735: 77 pgs: 77 active+clean; 152 MiB data, 3.9 GiB used, 801 GiB / 805 GiB avail; 0 B/s wr, 1.52k op/s 2024-08-19T14:20:13.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:12 smithi031 ceph-mon[16336]: 3.1f scrub starts 2024-08-19T14:20:13.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:12 smithi031 ceph-mon[16336]: 3.1f scrub ok 2024-08-19T14:20:13.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:12 smithi031 ceph-mon[16336]: 1.a scrub starts 2024-08-19T14:20:13.201 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:12 smithi031 ceph-mon[16336]: 1.a scrub ok 2024-08-19T14:20:13.243 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-08-19T14:20:13.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:12 smithi032 ceph-mon[20004]: 1.b scrub starts 2024-08-19T14:20:13.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:12 smithi032 ceph-mon[20004]: 1.b scrub ok 2024-08-19T14:20:13.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:12 smithi032 ceph-mon[20004]: pgmap v735: 77 pgs: 77 active+clean; 152 MiB data, 3.9 GiB used, 801 GiB / 805 GiB avail; 0 B/s wr, 1.52k op/s 2024-08-19T14:20:13.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:12 smithi032 ceph-mon[20004]: 3.1f scrub starts 2024-08-19T14:20:13.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:12 smithi032 ceph-mon[20004]: 3.1f scrub ok 2024-08-19T14:20:13.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:12 smithi032 ceph-mon[20004]: 1.a scrub starts 2024-08-19T14:20:13.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:12 smithi032 ceph-mon[20004]: 1.a scrub ok 2024-08-19T14:20:13.280 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:20:13.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:12 smithi045 ceph-mon[20143]: 1.b scrub starts 2024-08-19T14:20:13.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:12 smithi045 ceph-mon[20143]: 1.b scrub ok 2024-08-19T14:20:13.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:12 smithi045 ceph-mon[20143]: pgmap v735: 77 pgs: 77 active+clean; 152 MiB data, 3.9 GiB used, 801 GiB / 805 GiB avail; 0 B/s wr, 1.52k op/s 2024-08-19T14:20:13.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:12 smithi045 ceph-mon[20143]: 3.1f scrub starts 2024-08-19T14:20:13.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:12 smithi045 ceph-mon[20143]: 3.1f scrub ok 2024-08-19T14:20:13.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:12 smithi045 ceph-mon[20143]: 1.a scrub starts 2024-08-19T14:20:13.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:12 smithi045 ceph-mon[20143]: 1.a scrub ok 2024-08-19T14:20:13.306 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:13.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:13.307 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:20:13.408 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:20:13.434 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:13.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:13.435 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:20:13.536 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:20:13.562 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:13.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:13.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:20:13.664 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:20:13.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:13.691 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:13.691 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:20:13.791 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:20:13.816 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:13.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:13.817 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:20:13.918 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:20:13.944 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:13.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:13.945 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:20:14.046 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:20:14.070 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:14.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:14.072 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:20:14.173 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:20:14.199 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:14.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:14.200 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:20:14.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:13 smithi032 ceph-mon[20004]: 1.5 scrub starts 2024-08-19T14:20:14.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:13 smithi032 ceph-mon[20004]: 1.5 scrub ok 2024-08-19T14:20:14.292 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:13 smithi031 ceph-mon[16336]: 1.5 scrub starts 2024-08-19T14:20:14.292 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:13 smithi031 ceph-mon[16336]: 1.5 scrub ok 2024-08-19T14:20:14.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:13 smithi045 ceph-mon[20143]: 1.5 scrub starts 2024-08-19T14:20:14.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:13 smithi045 ceph-mon[20143]: 1.5 scrub ok 2024-08-19T14:20:14.301 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:20:14.326 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:14.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:14.327 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:20:14.428 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:20:14.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:14.454 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:14.454 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:20:14.554 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:20:14.580 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:14.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:14.581 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:20:14.682 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:20:14.708 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:14.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:14.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:20:14.810 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:20:14.836 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:14.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:14.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:20:14.938 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:20:14.963 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:14.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:14.964 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:20:15.065 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:20:15.091 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:15.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:15.093 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:20:15.194 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:20:15.203 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:14 smithi031 ceph-mon[16336]: 3.10 scrub starts 2024-08-19T14:20:15.203 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:14 smithi031 ceph-mon[16336]: 3.10 scrub ok 2024-08-19T14:20:15.204 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:14 smithi031 ceph-mon[16336]: pgmap v736: 77 pgs: 77 active+clean; 56 MiB data, 3.4 GiB used, 801 GiB / 805 GiB avail; 0 B/s wr, 1.33k op/s 2024-08-19T14:20:15.219 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:15.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:15.220 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:20:15.259 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:14 smithi032 ceph-mon[20004]: 3.10 scrub starts 2024-08-19T14:20:15.259 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:14 smithi032 ceph-mon[20004]: 3.10 scrub ok 2024-08-19T14:20:15.259 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:14 smithi032 ceph-mon[20004]: pgmap v736: 77 pgs: 77 active+clean; 56 MiB data, 3.4 GiB used, 801 GiB / 805 GiB avail; 0 B/s wr, 1.33k op/s 2024-08-19T14:20:15.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:14 smithi045 ceph-mon[20143]: 3.10 scrub starts 2024-08-19T14:20:15.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:14 smithi045 ceph-mon[20143]: 3.10 scrub ok 2024-08-19T14:20:15.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:14 smithi045 ceph-mon[20143]: pgmap v736: 77 pgs: 77 active+clean; 56 MiB data, 3.4 GiB used, 801 GiB / 805 GiB avail; 0 B/s wr, 1.33k op/s 2024-08-19T14:20:15.321 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:20:15.342 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:15.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:15.344 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:20:15.445 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:20:15.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:15.471 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:15.471 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:20:15.572 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:20:15.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:15.595 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:15.595 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:20:15.697 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:20:15.722 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:15.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:15.748 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:20:15.849 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:20:15.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:15.875 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:15.876 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:20:15.977 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:20:16.003 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:16.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:16.004 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:20:16.105 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:20:16.116 INFO:teuthology.orchestra.run.smithi031.stderr:reweighted osd.0 to 1 (10000) 2024-08-19T14:20:16.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:16.132 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:16.133 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:20:16.233 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:20:16.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:15 smithi032 ceph-mon[20004]: 1.e deep-scrub starts 2024-08-19T14:20:16.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:15 smithi032 ceph-mon[20004]: 1.e deep-scrub ok 2024-08-19T14:20:16.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:15 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/340192759' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-08-19T14:20:16.259 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:16.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:16.260 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:20:16.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:15 smithi045 ceph-mon[20143]: 1.e deep-scrub starts 2024-08-19T14:20:16.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:15 smithi045 ceph-mon[20143]: 1.e deep-scrub ok 2024-08-19T14:20:16.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:15 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/340192759' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-08-19T14:20:16.361 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:20:16.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:15 smithi031 ceph-mon[16336]: 1.e deep-scrub starts 2024-08-19T14:20:16.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:15 smithi031 ceph-mon[16336]: 1.e deep-scrub ok 2024-08-19T14:20:16.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:15 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/340192759' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-08-19T14:20:16.387 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:16.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:16.388 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:20:16.489 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:20:16.514 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:16.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:16.515 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:20:16.617 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:20:16.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:16.643 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:16.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:20:16.745 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:20:16.770 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:16.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:16.771 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:20:16.872 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:20:16.898 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:16.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:16.899 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:20:17.000 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:20:17.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:17.025 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:17.025 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:20:17.126 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:20:17.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:17.151 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:17.151 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:20:17.229 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:16 smithi031 ceph-mon[16336]: pgmap v737: 77 pgs: 77 active+clean; 5.6 MiB data, 3.3 GiB used, 801 GiB / 805 GiB avail; 0 B/s wr, 1.08k op/s 2024-08-19T14:20:17.229 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:16 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/340192759' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-08-19T14:20:17.229 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:16 smithi031 ceph-mon[16336]: osdmap e327: 12 total, 12 up, 9 in 2024-08-19T14:20:17.252 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:20:17.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:16 smithi032 ceph-mon[20004]: pgmap v737: 77 pgs: 77 active+clean; 5.6 MiB data, 3.3 GiB used, 801 GiB / 805 GiB avail; 0 B/s wr, 1.08k op/s 2024-08-19T14:20:17.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:16 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/340192759' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-08-19T14:20:17.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:16 smithi032 ceph-mon[20004]: osdmap e327: 12 total, 12 up, 9 in 2024-08-19T14:20:17.259 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd reweight 1 1 2024-08-19T14:20:17.278 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:17.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:17.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:20:17.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:16 smithi045 ceph-mon[20143]: pgmap v737: 77 pgs: 77 active+clean; 5.6 MiB data, 3.3 GiB used, 801 GiB / 805 GiB avail; 0 B/s wr, 1.08k op/s 2024-08-19T14:20:17.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:16 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/340192759' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-08-19T14:20:17.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:16 smithi045 ceph-mon[20143]: osdmap e327: 12 total, 12 up, 9 in 2024-08-19T14:20:17.380 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:20:17.404 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:17.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:17.405 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:20:17.502 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_1 2024-08-19T14:20:17.503 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd pool rm unique_pool_1 unique_pool_1 --yes-i-really-really-mean-it 2024-08-19T14:20:17.507 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:20:17.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:17.532 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:17.532 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:20:17.633 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:20:17.656 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:20:17.658 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:17.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:17.659 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:20:17.697 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:20:17.706 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:20:17.706 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:20:17.713 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:20:17.733 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:20:17.733 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:20:17.751 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:20:17.751 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:20:17.760 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:20:17.767 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:20:17.767 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:20:17.784 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:17.784 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:20:17.785 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:20:17.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:17.785 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:20:17.801 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:20:17.801 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:20:17.818 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:20:17.819 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:20:17.835 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:20:17.836 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:20:17.853 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:20:17.853 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:20:17.870 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:20:17.871 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:20:17.886 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:20:17.887 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:20:17.888 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:20:17.904 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:20:17.905 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:20:17.912 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:17.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:17.913 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:20:18.014 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:20:18.017 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:17 smithi031 ceph-mon[16336]: 1.1b scrub starts 2024-08-19T14:20:18.017 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:17 smithi031 ceph-mon[16336]: 1.1b scrub ok 2024-08-19T14:20:18.017 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:17 smithi031 ceph-mon[16336]: 1.1a scrub starts 2024-08-19T14:20:18.017 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:17 smithi031 ceph-mon[16336]: 1.1a scrub ok 2024-08-19T14:20:18.017 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:17 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/298187543' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:20:18.038 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:18.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:18.039 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:20:18.140 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:20:18.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:18.163 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:18.163 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:20:18.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:17 smithi032 ceph-mon[20004]: 1.1b scrub starts 2024-08-19T14:20:18.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:17 smithi032 ceph-mon[20004]: 1.1b scrub ok 2024-08-19T14:20:18.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:17 smithi032 ceph-mon[20004]: 1.1a scrub starts 2024-08-19T14:20:18.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:17 smithi032 ceph-mon[20004]: 1.1a scrub ok 2024-08-19T14:20:18.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:17 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/298187543' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:20:18.264 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:20:18.290 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:18.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:18.291 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:20:18.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:17 smithi045 ceph-mon[20143]: 1.1b scrub starts 2024-08-19T14:20:18.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:17 smithi045 ceph-mon[20143]: 1.1b scrub ok 2024-08-19T14:20:18.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:17 smithi045 ceph-mon[20143]: 1.1a scrub starts 2024-08-19T14:20:18.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:17 smithi045 ceph-mon[20143]: 1.1a scrub ok 2024-08-19T14:20:18.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:17 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/298187543' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:20:18.392 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:20:18.418 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:18.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:18.419 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:20:18.520 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:20:18.545 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:18.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:18.546 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:20:18.647 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:20:18.673 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:18.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:18.674 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:20:18.775 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:20:18.799 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:18.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:18.800 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:20:18.901 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:20:18.923 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:18.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:18.924 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:20:18.964 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is set 2024-08-19T14:20:19.025 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:20:19.051 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:19.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:19.052 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:20:19.153 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:20:19.174 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:19.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:19.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:20:19.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:18 smithi032 ceph-mon[20004]: 1.1d scrub starts 2024-08-19T14:20:19.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:18 smithi032 ceph-mon[20004]: 1.1d scrub ok 2024-08-19T14:20:19.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:18 smithi032 ceph-mon[20004]: pgmap v739: 77 pgs: 77 active+clean; 577 KiB data, 3.2 GiB used, 801 GiB / 805 GiB avail; 0 B/s wr, 977 op/s 2024-08-19T14:20:19.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:18 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2663051691' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:20:19.276 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:20:19.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:18 smithi045 ceph-mon[20143]: 1.1d scrub starts 2024-08-19T14:20:19.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:18 smithi045 ceph-mon[20143]: 1.1d scrub ok 2024-08-19T14:20:19.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:18 smithi045 ceph-mon[20143]: pgmap v739: 77 pgs: 77 active+clean; 577 KiB data, 3.2 GiB used, 801 GiB / 805 GiB avail; 0 B/s wr, 977 op/s 2024-08-19T14:20:19.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:18 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2663051691' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:20:19.302 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:19.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:19.303 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:20:19.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:18 smithi031 ceph-mon[16336]: 1.1d scrub starts 2024-08-19T14:20:19.311 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:18 smithi031 ceph-mon[16336]: 1.1d scrub ok 2024-08-19T14:20:19.311 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:18 smithi031 ceph-mon[16336]: pgmap v739: 77 pgs: 77 active+clean; 577 KiB data, 3.2 GiB used, 801 GiB / 805 GiB avail; 0 B/s wr, 977 op/s 2024-08-19T14:20:19.311 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:18 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2663051691' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:20:19.404 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:20:19.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:19.431 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:19.431 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:20:19.532 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:20:19.557 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:19.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:19.558 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:20:19.659 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:20:19.674 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:20:19.682 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:19.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:19.683 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:20:19.784 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:20:19.810 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:19.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:19.811 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:20:19.912 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:20:19.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:19.935 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:19.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:20:19.970 INFO:teuthology.orchestra.run.smithi031.stderr:reweighted osd.1 to 1 (10000) 2024-08-19T14:20:20.036 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:20:20.057 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:20.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:20.058 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:20:20.162 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:20:20.188 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:20.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:20.189 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:20:20.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:19 smithi032 ceph-mon[20004]: 1.1e scrub starts 2024-08-19T14:20:20.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:19 smithi032 ceph-mon[20004]: 1.1e scrub ok 2024-08-19T14:20:20.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:19 smithi032 ceph-mon[20004]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:20:20.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:19 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2663051691' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:20:20.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:19 smithi032 ceph-mon[20004]: osdmap e328: 12 total, 12 up, 9 in 2024-08-19T14:20:20.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:19 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1722918476' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-08-19T14:20:20.259 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:19 smithi031 ceph-mon[16336]: 1.1e scrub starts 2024-08-19T14:20:20.259 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:19 smithi031 ceph-mon[16336]: 1.1e scrub ok 2024-08-19T14:20:20.259 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:19 smithi031 ceph-mon[16336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:20:20.259 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:19 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2663051691' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:20:20.259 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:19 smithi031 ceph-mon[16336]: osdmap e328: 12 total, 12 up, 9 in 2024-08-19T14:20:20.259 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:19 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1722918476' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-08-19T14:20:20.289 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:20:20.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:19 smithi045 ceph-mon[20143]: 1.1e scrub starts 2024-08-19T14:20:20.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:19 smithi045 ceph-mon[20143]: 1.1e scrub ok 2024-08-19T14:20:20.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:19 smithi045 ceph-mon[20143]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:20:20.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:19 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2663051691' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:20:20.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:19 smithi045 ceph-mon[20143]: osdmap e328: 12 total, 12 up, 9 in 2024-08-19T14:20:20.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:19 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1722918476' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-08-19T14:20:20.315 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:20.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:20.316 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:20:20.417 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:20:20.443 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:20.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:20.445 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:20:20.545 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:20:20.567 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:20.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:20.568 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:20:20.670 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:20:20.695 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:20.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:20.696 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:20:20.797 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:20:20.823 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:20.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:20.824 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:20:20.924 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:20:20.924 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-08-19T14:20:20.924 INFO:teuthology.orchestra.run.smithi031.stdout: "complaint_time": 30, 2024-08-19T14:20:20.924 INFO:teuthology.orchestra.run.smithi031.stdout: "num_blocked_ops": 0 2024-08-19T14:20:20.924 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:20:20.924 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:20:20.953 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:20.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:20.954 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:20:21.054 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:20:21.080 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:21.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:21.081 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:20:21.181 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:20:21.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:20 smithi031 ceph-mon[16336]: pgmap v741: 77 pgs: 77 active+clean; 577 KiB data, 3.2 GiB used, 801 GiB / 805 GiB avail; 0 B/s wr, 303 op/s 2024-08-19T14:20:21.201 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:20 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1722918476' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-08-19T14:20:21.201 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:20 smithi031 ceph-mon[16336]: osdmap e329: 12 total, 12 up, 9 in 2024-08-19T14:20:21.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:21.207 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:21.207 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:20:21.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:20 smithi032 ceph-mon[20004]: pgmap v741: 77 pgs: 77 active+clean; 577 KiB data, 3.2 GiB used, 801 GiB / 805 GiB avail; 0 B/s wr, 303 op/s 2024-08-19T14:20:21.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:20 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1722918476' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-08-19T14:20:21.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:20 smithi032 ceph-mon[20004]: osdmap e329: 12 total, 12 up, 9 in 2024-08-19T14:20:21.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:20 smithi045 ceph-mon[20143]: pgmap v741: 77 pgs: 77 active+clean; 577 KiB data, 3.2 GiB used, 801 GiB / 805 GiB avail; 0 B/s wr, 303 op/s 2024-08-19T14:20:21.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:20 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1722918476' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-08-19T14:20:21.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:20 smithi045 ceph-mon[20143]: osdmap e329: 12 total, 12 up, 9 in 2024-08-19T14:20:21.308 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:20:21.329 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:21.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:21.330 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:20:21.431 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:20:21.457 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:21.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:21.458 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:20:21.559 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:20:21.579 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-08-19T14:20:21.585 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:21.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:21.586 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:20:21.647 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-19T14:20:21.687 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:20:21.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:21.712 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:21.713 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:20:21.814 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:20:21.835 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:21.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:21.836 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:20:21.937 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:20:21.947 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd reweight 3 1 2024-08-19T14:20:21.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:21.963 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:21.964 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:20:22.064 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:20:22.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:22.091 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:22.092 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:20:22.192 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:20:22.210 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:20:22.216 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:20:22.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:22.220 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:22.220 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:20:22.260 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:20:22.286 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:20:22.321 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:20:22.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:22.351 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:22.351 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:20:22.452 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:20:22.473 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:22.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:22.474 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:20:22.575 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:20:22.596 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:22.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:22.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:20:22.698 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:20:22.725 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:22.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:22.726 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:20:22.827 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:20:22.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:22.853 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:22.853 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:20:22.953 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:20:22.979 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:22.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:22.980 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:20:22.981 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'unique_pool_1' removed 2024-08-19T14:20:23.080 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:20:23.116 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:23.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:23.117 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:20:23.122 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:22 smithi031 ceph-mon[16336]: pgmap v743: 77 pgs: 77 active+clean; 577 KiB data, 3.2 GiB used, 801 GiB / 805 GiB avail; 0 B/s wr, 13 op/s 2024-08-19T14:20:23.122 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:22 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1285016895' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-08-19T14:20:23.218 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:20:23.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:23.243 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:23.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:20:23.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:22 smithi032 ceph-mon[20004]: pgmap v743: 77 pgs: 77 active+clean; 577 KiB data, 3.2 GiB used, 801 GiB / 805 GiB avail; 0 B/s wr, 13 op/s 2024-08-19T14:20:23.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:22 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1285016895' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-08-19T14:20:23.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:22 smithi045 ceph-mon[20143]: pgmap v743: 77 pgs: 77 active+clean; 577 KiB data, 3.2 GiB used, 801 GiB / 805 GiB avail; 0 B/s wr, 13 op/s 2024-08-19T14:20:23.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:22 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1285016895' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-08-19T14:20:23.344 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:20:23.365 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:23.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:23.366 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:20:23.467 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:20:23.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:23.490 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:23.490 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:20:23.591 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:20:23.613 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:23.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:23.614 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:20:23.714 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:20:23.736 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:23.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:23.737 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:20:23.838 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:20:23.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:23.860 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:23.860 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:20:23.961 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:20:23.986 INFO:teuthology.orchestra.run.smithi031.stderr:reweighted osd.3 to 1 (10000) 2024-08-19T14:20:23.986 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is set 2024-08-19T14:20:23.987 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:23.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:23.988 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:20:24.083 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:23 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1285016895' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-08-19T14:20:24.083 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:23 smithi031 ceph-mon[16336]: osdmap e330: 12 total, 12 up, 9 in 2024-08-19T14:20:24.083 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:23 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1516330677' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:20:24.083 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:23 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/126522565' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-08-19T14:20:24.088 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:20:24.112 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:24.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:24.113 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:20:24.126 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:20:24.126 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 20, 2024-08-19T14:20:24.126 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 600, 2024-08-19T14:20:24.127 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [ 2024-08-19T14:20:24.127 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.127 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:241314 2.1 2:8e6fec04:::benchmark_data_smithi045_46415_object118033:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-08-19T14:20:24.127 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:16:40.787553+0000", 2024-08-19T14:20:24.127 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 223.33702108599999, 2024-08-19T14:20:24.127 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.29700766899999997, 2024-08-19T14:20:24.127 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:20:24.127 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:20:24.127 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:20:24.127 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:20:24.127 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:20:24.128 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:20:24.128 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 241314 2024-08-19T14:20:24.128 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.128 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:20:24.128 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.128 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:20:24.128 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.787553+0000", 2024-08-19T14:20:24.128 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:20:24.128 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.128 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.128 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:20:24.128 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.787553+0000", 2024-08-19T14:20:24.128 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:20:24.128 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.129 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.129 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:20:24.129 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.787554+0000", 2024-08-19T14:20:24.129 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5770000000000001e-06 2024-08-19T14:20:24.129 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.129 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.129 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:20:24.129 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.787560+0000", 2024-08-19T14:20:24.129 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.6930000000000004e-06 2024-08-19T14:20:24.129 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.129 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.129 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:20:24.130 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.787574+0000", 2024-08-19T14:20:24.130 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3828e-05 2024-08-19T14:20:24.130 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.130 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.130 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:20:24.130 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.787582+0000", 2024-08-19T14:20:24.130 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.225e-06 2024-08-19T14:20:24.130 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.130 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.130 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:20:24.130 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.788601+0000", 2024-08-19T14:20:24.130 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.001018833 2024-08-19T14:20:24.131 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.131 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.131 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:20:24.131 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.788734+0000", 2024-08-19T14:20:24.131 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013335699999999999 2024-08-19T14:20:24.131 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.131 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.131 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 1,2,8", 2024-08-19T14:20:24.131 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.788881+0000", 2024-08-19T14:20:24.131 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000146644 2024-08-19T14:20:24.131 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.131 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.132 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:20:24.132 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.793613+0000", 2024-08-19T14:20:24.132 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0047319600000000003 2024-08-19T14:20:24.132 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.132 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.132 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:20:24.132 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.795531+0000", 2024-08-19T14:20:24.132 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.001918335 2024-08-19T14:20:24.132 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.132 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.132 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:20:24.132 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.796093+0000", 2024-08-19T14:20:24.132 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00056226599999999996 2024-08-19T14:20:24.133 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.133 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.133 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:20:24.133 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.084346+0000", 2024-08-19T14:20:24.133 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.28825296 2024-08-19T14:20:24.133 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.133 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.133 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:20:24.133 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.084417+0000", 2024-08-19T14:20:24.133 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.0599999999999995e-05 2024-08-19T14:20:24.133 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.133 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.134 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:20:24.134 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.084560+0000", 2024-08-19T14:20:24.134 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00014339100000000001 2024-08-19T14:20:24.134 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:20:24.134 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:20:24.134 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:20:24.134 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.134 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.134 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:241325 2.1 2:8f07de09:::benchmark_data_smithi045_46415_object118044:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-08-19T14:20:24.134 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:16:40.798902+0000", 2024-08-19T14:20:24.134 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 223.32567203599999, 2024-08-19T14:20:24.134 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.29144186999999999, 2024-08-19T14:20:24.134 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:20:24.135 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:20:24.135 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:20:24.135 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:20:24.135 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:20:24.135 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:20:24.135 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 241325 2024-08-19T14:20:24.135 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.135 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:20:24.135 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.135 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:20:24.135 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.798902+0000", 2024-08-19T14:20:24.135 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:20:24.136 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.136 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.136 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:20:24.136 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.798902+0000", 2024-08-19T14:20:24.136 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:20:24.136 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.136 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.136 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:20:24.136 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.798904+0000", 2024-08-19T14:20:24.136 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.1780000000000002e-06 2024-08-19T14:20:24.136 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.136 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.136 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:20:24.137 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.798914+0000", 2024-08-19T14:20:24.137 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.7480000000000005e-06 2024-08-19T14:20:24.137 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.137 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.137 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:20:24.137 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.798933+0000", 2024-08-19T14:20:24.137 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.9026e-05 2024-08-19T14:20:24.137 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.137 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.137 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:20:24.137 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.798942+0000", 2024-08-19T14:20:24.137 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.3200000000000006e-06 2024-08-19T14:20:24.137 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.137 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.138 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:20:24.138 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.799163+0000", 2024-08-19T14:20:24.138 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000221337 2024-08-19T14:20:24.138 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.138 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.138 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:20:24.138 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.799310+0000", 2024-08-19T14:20:24.138 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00014677399999999999 2024-08-19T14:20:24.138 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.138 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.138 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 1,2,8", 2024-08-19T14:20:24.138 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.799452+0000", 2024-08-19T14:20:24.138 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00014219499999999999 2024-08-19T14:20:24.139 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.139 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.139 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:20:24.139 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.804505+0000", 2024-08-19T14:20:24.139 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.005052384 2024-08-19T14:20:24.139 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.139 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.139 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:20:24.139 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.804768+0000", 2024-08-19T14:20:24.139 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00026380700000000002 2024-08-19T14:20:24.139 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.139 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.139 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:20:24.140 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.805302+0000", 2024-08-19T14:20:24.140 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00053321300000000005 2024-08-19T14:20:24.140 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.140 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.140 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:20:24.140 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.090219+0000", 2024-08-19T14:20:24.140 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.284917107 2024-08-19T14:20:24.140 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.140 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.140 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:20:24.140 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.090271+0000", 2024-08-19T14:20:24.140 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.2289e-05 2024-08-19T14:20:24.140 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.141 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.141 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:20:24.141 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.090343+0000", 2024-08-19T14:20:24.141 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.2491999999999996e-05 2024-08-19T14:20:24.141 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:20:24.141 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:20:24.141 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:20:24.141 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.141 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.141 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:241358 2.1 2:8659030e:::benchmark_data_smithi045_46415_object118077:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-08-19T14:20:24.141 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:16:40.843360+0000", 2024-08-19T14:20:24.141 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 223.28121397500001, 2024-08-19T14:20:24.142 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.24721465500000001, 2024-08-19T14:20:24.142 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:20:24.142 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:20:24.142 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:20:24.142 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:20:24.142 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:20:24.142 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:20:24.142 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 241358 2024-08-19T14:20:24.142 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.142 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:20:24.142 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.142 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:20:24.142 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.843360+0000", 2024-08-19T14:20:24.143 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:20:24.143 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.143 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.143 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:20:24.143 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.843360+0000", 2024-08-19T14:20:24.143 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:20:24.143 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.143 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.143 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:20:24.143 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.843362+0000", 2024-08-19T14:20:24.143 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.0820000000000001e-06 2024-08-19T14:20:24.143 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.143 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.143 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:20:24.144 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.843373+0000", 2024-08-19T14:20:24.144 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1600000000000001e-05 2024-08-19T14:20:24.144 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.144 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.144 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:20:24.144 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.843395+0000", 2024-08-19T14:20:24.144 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.1670000000000001e-05 2024-08-19T14:20:24.144 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.144 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.144 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:20:24.144 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.843405+0000", 2024-08-19T14:20:24.144 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.5030000000000003e-06 2024-08-19T14:20:24.144 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.144 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.145 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:20:24.145 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.843474+0000", 2024-08-19T14:20:24.145 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.9620000000000001e-05 2024-08-19T14:20:24.145 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.145 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.145 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:20:24.145 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.843613+0000", 2024-08-19T14:20:24.145 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013863699999999999 2024-08-19T14:20:24.145 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.145 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.145 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 1,2,8", 2024-08-19T14:20:24.145 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.843771+0000", 2024-08-19T14:20:24.145 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000158691 2024-08-19T14:20:24.146 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.146 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.146 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:20:24.146 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.847962+0000", 2024-08-19T14:20:24.146 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0041908989999999997 2024-08-19T14:20:24.146 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.146 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.146 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:20:24.146 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.848335+0000", 2024-08-19T14:20:24.146 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00037213900000000001 2024-08-19T14:20:24.146 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.146 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.146 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:20:24.147 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.849083+0000", 2024-08-19T14:20:24.147 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00074856300000000005 2024-08-19T14:20:24.147 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.147 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.147 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:20:24.147 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.090392+0000", 2024-08-19T14:20:24.147 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.24130854600000001 2024-08-19T14:20:24.147 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.147 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.147 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:20:24.147 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.090453+0000", 2024-08-19T14:20:24.147 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.1635000000000002e-05 2024-08-19T14:20:24.147 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.148 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.148 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:20:24.148 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.090574+0000", 2024-08-19T14:20:24.148 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012107 2024-08-19T14:20:24.148 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:20:24.148 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:20:24.148 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:20:24.148 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.148 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.148 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:241367 2.1 2:870a5187:::benchmark_data_smithi045_46415_object118086:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-08-19T14:20:24.149 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:16:40.854128+0000", 2024-08-19T14:20:24.149 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 223.27044545000001, 2024-08-19T14:20:24.149 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.239748353, 2024-08-19T14:20:24.149 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:20:24.149 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:20:24.149 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:20:24.149 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:20:24.149 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:20:24.149 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:20:24.149 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 241367 2024-08-19T14:20:24.149 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.149 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:20:24.149 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.150 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:20:24.150 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.854128+0000", 2024-08-19T14:20:24.150 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:20:24.150 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.150 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.150 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:20:24.150 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.854128+0000", 2024-08-19T14:20:24.150 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:20:24.150 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.150 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.150 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:20:24.150 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.854130+0000", 2024-08-19T14:20:24.151 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6139999999999999e-06 2024-08-19T14:20:24.151 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.151 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.151 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:20:24.151 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.854138+0000", 2024-08-19T14:20:24.151 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.9370000000000004e-06 2024-08-19T14:20:24.151 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.151 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.151 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:20:24.151 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.854151+0000", 2024-08-19T14:20:24.151 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3338e-05 2024-08-19T14:20:24.151 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.151 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.151 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:20:24.152 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.854158+0000", 2024-08-19T14:20:24.152 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.3350000000000001e-06 2024-08-19T14:20:24.152 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.152 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.152 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:20:24.152 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.854208+0000", 2024-08-19T14:20:24.152 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.9373000000000003e-05 2024-08-19T14:20:24.152 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.152 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.152 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:20:24.152 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.854301+0000", 2024-08-19T14:20:24.152 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.2990000000000002e-05 2024-08-19T14:20:24.152 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.153 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.153 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 1,2,8", 2024-08-19T14:20:24.153 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.854388+0000", 2024-08-19T14:20:24.153 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.7321e-05 2024-08-19T14:20:24.153 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.153 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.153 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:20:24.153 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.860101+0000", 2024-08-19T14:20:24.153 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0057132839999999999 2024-08-19T14:20:24.153 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.153 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.153 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:20:24.153 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.860163+0000", 2024-08-19T14:20:24.154 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.1896000000000005e-05 2024-08-19T14:20:24.154 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.154 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.154 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:20:24.154 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.860183+0000", 2024-08-19T14:20:24.154 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.0041e-05 2024-08-19T14:20:24.154 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.154 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.154 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:20:24.154 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.093641+0000", 2024-08-19T14:20:24.154 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.23345814300000001 2024-08-19T14:20:24.154 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.154 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.154 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:20:24.155 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.093706+0000", 2024-08-19T14:20:24.155 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.4453000000000001e-05 2024-08-19T14:20:24.155 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.155 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.155 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:20:24.155 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.093877+0000", 2024-08-19T14:20:24.155 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00017062800000000001 2024-08-19T14:20:24.155 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:20:24.155 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:20:24.155 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:20:24.155 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.155 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.155 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:241375 2.1 2:8e818ec7:::benchmark_data_smithi045_46415_object118094:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-08-19T14:20:24.155 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:16:40.863736+0000", 2024-08-19T14:20:24.156 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 223.260837245, 2024-08-19T14:20:24.156 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.23047135799999999, 2024-08-19T14:20:24.156 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:20:24.156 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:20:24.156 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:20:24.157 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:20:24.157 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:20:24.157 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:20:24.158 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 241375 2024-08-19T14:20:24.158 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.158 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:20:24.158 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.158 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:20:24.158 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.863736+0000", 2024-08-19T14:20:24.158 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:20:24.158 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.158 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.158 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:20:24.158 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.863736+0000", 2024-08-19T14:20:24.158 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:20:24.158 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.159 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.159 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:20:24.159 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.863738+0000", 2024-08-19T14:20:24.159 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5820000000000001e-06 2024-08-19T14:20:24.159 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.159 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.159 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:20:24.159 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.863744+0000", 2024-08-19T14:20:24.159 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.0109999999999997e-06 2024-08-19T14:20:24.159 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.159 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.159 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:20:24.159 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.863759+0000", 2024-08-19T14:20:24.160 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5e-05 2024-08-19T14:20:24.160 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.160 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.160 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:20:24.160 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.863766+0000", 2024-08-19T14:20:24.160 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.311e-06 2024-08-19T14:20:24.160 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.160 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.160 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:20:24.160 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.863821+0000", 2024-08-19T14:20:24.160 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.4954999999999998e-05 2024-08-19T14:20:24.160 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.160 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.161 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:20:24.161 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.863953+0000", 2024-08-19T14:20:24.161 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013166600000000001 2024-08-19T14:20:24.161 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.161 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.161 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 1,2,8", 2024-08-19T14:20:24.161 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.864114+0000", 2024-08-19T14:20:24.161 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000161123 2024-08-19T14:20:24.161 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.161 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.161 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:20:24.161 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.868901+0000", 2024-08-19T14:20:24.161 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0047872490000000004 2024-08-19T14:20:24.161 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.162 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.162 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:20:24.162 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.869524+0000", 2024-08-19T14:20:24.162 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00062289799999999998 2024-08-19T14:20:24.162 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.162 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.162 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:20:24.162 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.870384+0000", 2024-08-19T14:20:24.162 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00085993899999999999 2024-08-19T14:20:24.162 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.162 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.162 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:20:24.162 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.093973+0000", 2024-08-19T14:20:24.163 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.22358860999999999 2024-08-19T14:20:24.163 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.163 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.163 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:20:24.163 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.094038+0000", 2024-08-19T14:20:24.163 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.5326000000000004e-05 2024-08-19T14:20:24.163 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.163 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.163 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:20:24.163 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.094208+0000", 2024-08-19T14:20:24.163 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00016968800000000001 2024-08-19T14:20:24.163 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:20:24.163 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:20:24.164 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:20:24.164 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.164 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.164 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25057.0:47774 3.11 3:8e95e964:::benchmark_data_smithi045_59973_object47773:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e234)", 2024-08-19T14:20:24.164 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:17:18.724217+0000", 2024-08-19T14:20:24.164 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 185.40035641899999, 2024-08-19T14:20:24.164 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0342696659999999, 2024-08-19T14:20:24.164 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:20:24.164 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:20:24.164 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:20:24.164 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:20:24.164 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25057", 2024-08-19T14:20:24.164 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:20:24.165 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 47774 2024-08-19T14:20:24.165 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.165 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:20:24.165 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.165 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:20:24.165 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724217+0000", 2024-08-19T14:20:24.165 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:20:24.165 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.165 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.165 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:20:24.165 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724217+0000", 2024-08-19T14:20:24.165 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:20:24.165 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.165 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.166 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:20:24.166 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724218+0000", 2024-08-19T14:20:24.166 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.4499999999999997e-07 2024-08-19T14:20:24.166 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.166 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.166 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:20:24.166 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724254+0000", 2024-08-19T14:20:24.166 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.5914000000000002e-05 2024-08-19T14:20:24.166 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.166 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.166 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:20:24.166 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724263+0000", 2024-08-19T14:20:24.167 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.4290000000000007e-06 2024-08-19T14:20:24.167 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.167 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.167 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:20:24.167 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724270+0000", 2024-08-19T14:20:24.167 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.7959999999999998e-06 2024-08-19T14:20:24.167 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.167 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.167 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:20:24.167 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.774143+0000", 2024-08-19T14:20:24.167 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.049872861999999997 2024-08-19T14:20:24.167 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.167 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.167 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:20:24.167 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.774151+0000", 2024-08-19T14:20:24.168 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.6489999999999997e-06 2024-08-19T14:20:24.168 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.168 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.168 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:20:24.168 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.758417+0000", 2024-08-19T14:20:24.168 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.98426575800000005 2024-08-19T14:20:24.168 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.168 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.168 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:20:24.168 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.758487+0000", 2024-08-19T14:20:24.168 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.9713000000000002e-05 2024-08-19T14:20:24.168 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:20:24.168 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:20:24.168 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:20:24.169 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.169 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.169 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25057.0:47788 3.11 3:8e23edc0:::benchmark_data_smithi045_59973_object47787:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e234)", 2024-08-19T14:20:24.169 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:17:18.724282+0000", 2024-08-19T14:20:24.169 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 185.40029157399999, 2024-08-19T14:20:24.169 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0343071210000001, 2024-08-19T14:20:24.169 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:20:24.169 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:20:24.169 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:20:24.169 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:20:24.169 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25057", 2024-08-19T14:20:24.169 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:20:24.169 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 47788 2024-08-19T14:20:24.169 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.170 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:20:24.170 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.170 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:20:24.170 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724282+0000", 2024-08-19T14:20:24.170 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:20:24.170 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.170 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.170 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:20:24.170 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724282+0000", 2024-08-19T14:20:24.170 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:20:24.170 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.170 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.170 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:20:24.171 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724283+0000", 2024-08-19T14:20:24.171 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.6100000000000001e-07 2024-08-19T14:20:24.171 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.171 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.171 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:20:24.171 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724318+0000", 2024-08-19T14:20:24.171 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.578e-05 2024-08-19T14:20:24.171 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.171 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.171 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:20:24.171 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724326+0000", 2024-08-19T14:20:24.171 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.4170000000000003e-06 2024-08-19T14:20:24.171 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.171 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.172 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:20:24.172 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724331+0000", 2024-08-19T14:20:24.172 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.5820000000000001e-06 2024-08-19T14:20:24.172 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.172 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.172 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:20:24.172 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.774172+0000", 2024-08-19T14:20:24.172 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.049840256999999999 2024-08-19T14:20:24.172 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.173 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.173 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:20:24.173 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.774177+0000", 2024-08-19T14:20:24.173 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.5160000000000002e-06 2024-08-19T14:20:24.173 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.173 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.173 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:20:24.173 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.758533+0000", 2024-08-19T14:20:24.173 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.98435635099999996 2024-08-19T14:20:24.173 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.173 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.173 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:20:24.173 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.758589+0000", 2024-08-19T14:20:24.173 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.5757000000000002e-05 2024-08-19T14:20:24.174 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:20:24.174 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:20:24.174 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:20:24.174 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.174 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.174 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25057.0:47814 3.11 3:8918a6df:::benchmark_data_smithi045_59973_object47813:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e234)", 2024-08-19T14:20:24.174 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:17:18.724406+0000", 2024-08-19T14:20:24.174 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 185.400168135, 2024-08-19T14:20:24.174 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0342602160000001, 2024-08-19T14:20:24.174 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:20:24.174 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:20:24.174 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:20:24.174 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:20:24.175 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25057", 2024-08-19T14:20:24.175 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:20:24.175 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 47814 2024-08-19T14:20:24.175 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.175 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:20:24.175 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.175 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:20:24.175 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724406+0000", 2024-08-19T14:20:24.175 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:20:24.175 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.175 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.175 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:20:24.175 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724406+0000", 2024-08-19T14:20:24.175 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:20:24.176 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.176 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.176 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:20:24.176 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724406+0000", 2024-08-19T14:20:24.176 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.0200000000000003e-07 2024-08-19T14:20:24.176 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.176 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.176 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:20:24.176 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724440+0000", 2024-08-19T14:20:24.176 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.3689000000000002e-05 2024-08-19T14:20:24.176 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.176 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.176 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:20:24.176 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724449+0000", 2024-08-19T14:20:24.177 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.0469999999999999e-06 2024-08-19T14:20:24.177 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.177 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.177 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:20:24.177 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724455+0000", 2024-08-19T14:20:24.177 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.1190000000000002e-06 2024-08-19T14:20:24.177 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.177 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.177 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:20:24.177 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.774196+0000", 2024-08-19T14:20:24.177 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.049740924999999998 2024-08-19T14:20:24.177 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.177 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.177 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:20:24.178 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.774201+0000", 2024-08-19T14:20:24.178 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.3090000000000003e-06 2024-08-19T14:20:24.178 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.178 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.178 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:20:24.178 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.758615+0000", 2024-08-19T14:20:24.178 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.98441362200000004 2024-08-19T14:20:24.178 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.178 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.178 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:20:24.178 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.758666+0000", 2024-08-19T14:20:24.178 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1103000000000002e-05 2024-08-19T14:20:24.178 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:20:24.179 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:20:24.179 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:20:24.179 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.179 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.179 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25057.0:47832 3.11 3:8d79f59c:::benchmark_data_smithi045_59973_object47831:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e234)", 2024-08-19T14:20:24.179 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:17:18.724649+0000", 2024-08-19T14:20:24.179 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 185.39992491999999, 2024-08-19T14:20:24.179 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0340940759999999, 2024-08-19T14:20:24.179 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:20:24.179 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:20:24.179 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:20:24.179 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:20:24.179 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25057", 2024-08-19T14:20:24.179 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:20:24.180 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 47832 2024-08-19T14:20:24.180 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.180 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:20:24.180 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.180 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:20:24.180 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724649+0000", 2024-08-19T14:20:24.180 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:20:24.180 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.180 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.180 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:20:24.180 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724649+0000", 2024-08-19T14:20:24.180 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:20:24.180 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.180 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.181 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:20:24.181 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724649+0000", 2024-08-19T14:20:24.181 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0399999999999996e-07 2024-08-19T14:20:24.181 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.181 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.181 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:20:24.181 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724680+0000", 2024-08-19T14:20:24.181 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.0477e-05 2024-08-19T14:20:24.181 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.181 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.181 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:20:24.181 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724688+0000", 2024-08-19T14:20:24.181 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.7509999999999997e-06 2024-08-19T14:20:24.181 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.182 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.182 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:20:24.182 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724694+0000", 2024-08-19T14:20:24.182 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.7679999999999997e-06 2024-08-19T14:20:24.182 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.182 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.182 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:20:24.182 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.774220+0000", 2024-08-19T14:20:24.182 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.049526175999999998 2024-08-19T14:20:24.182 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.182 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.182 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:20:24.182 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.774226+0000", 2024-08-19T14:20:24.183 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.2820000000000004e-06 2024-08-19T14:20:24.183 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.183 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.183 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:20:24.183 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.758693+0000", 2024-08-19T14:20:24.183 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.98446715699999998 2024-08-19T14:20:24.183 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.183 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.183 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:20:24.183 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.758743+0000", 2024-08-19T14:20:24.183 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.9960999999999998e-05 2024-08-19T14:20:24.183 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:20:24.183 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:20:24.183 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:20:24.184 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.184 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.184 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25057.0:47838 3.16 3:68f5225c:::benchmark_data_smithi045_59973_object47837:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e234)", 2024-08-19T14:20:24.184 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:17:18.777436+0000", 2024-08-19T14:20:24.184 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 185.34713735099999, 2024-08-19T14:20:24.184 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.98807081600000002, 2024-08-19T14:20:24.184 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:20:24.184 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:20:24.184 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:20:24.184 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:20:24.184 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25057", 2024-08-19T14:20:24.184 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:20:24.184 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 47838 2024-08-19T14:20:24.184 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.185 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:20:24.185 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.185 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:20:24.185 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.777436+0000", 2024-08-19T14:20:24.185 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:20:24.185 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.185 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.185 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:20:24.185 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.777436+0000", 2024-08-19T14:20:24.185 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:20:24.185 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.185 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.185 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:20:24.185 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.777438+0000", 2024-08-19T14:20:24.186 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.7489999999999999e-06 2024-08-19T14:20:24.186 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.186 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.186 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:20:24.186 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.777473+0000", 2024-08-19T14:20:24.186 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.5284999999999998e-05 2024-08-19T14:20:24.186 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.186 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.186 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:20:24.186 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.777492+0000", 2024-08-19T14:20:24.186 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.8853999999999999e-05 2024-08-19T14:20:24.186 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.186 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.187 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:20:24.187 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.777500+0000", 2024-08-19T14:20:24.187 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.9759999999999995e-06 2024-08-19T14:20:24.187 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.187 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.187 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:20:24.187 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.778853+0000", 2024-08-19T14:20:24.187 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0013526460000000001 2024-08-19T14:20:24.187 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.187 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.187 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:20:24.187 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.778867+0000", 2024-08-19T14:20:24.187 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4448e-05 2024-08-19T14:20:24.188 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.188 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.188 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:20:24.188 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.759774+0000", 2024-08-19T14:20:24.188 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.98090639700000004 2024-08-19T14:20:24.188 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.188 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.188 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:20:24.188 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.759902+0000", 2024-08-19T14:20:24.188 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000128128 2024-08-19T14:20:24.188 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.188 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.188 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 1,4", 2024-08-19T14:20:24.188 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.760062+0000", 2024-08-19T14:20:24.189 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000159956 2024-08-19T14:20:24.189 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.189 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.189 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:20:24.189 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.762325+0000", 2024-08-19T14:20:24.189 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0022629880000000001 2024-08-19T14:20:24.189 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.189 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.189 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:20:24.189 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.763087+0000", 2024-08-19T14:20:24.189 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00076208899999999997 2024-08-19T14:20:24.189 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.189 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.189 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:20:24.190 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.765400+0000", 2024-08-19T14:20:24.190 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0023133120000000001 2024-08-19T14:20:24.190 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.190 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.190 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:20:24.190 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.765462+0000", 2024-08-19T14:20:24.190 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.2048e-05 2024-08-19T14:20:24.190 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.190 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.190 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:20:24.190 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.765507+0000", 2024-08-19T14:20:24.190 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.494e-05 2024-08-19T14:20:24.190 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:20:24.191 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:20:24.191 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:20:24.191 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.191 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.191 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25057.0:47845 3.16 3:6c285014:::benchmark_data_smithi045_59973_object47844:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e234)", 2024-08-19T14:20:24.191 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:17:18.799489+0000", 2024-08-19T14:20:24.191 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 185.32508504699999, 2024-08-19T14:20:24.191 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.96674353099999999, 2024-08-19T14:20:24.191 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:20:24.191 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:20:24.194 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:20:24.194 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:20:24.194 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25057", 2024-08-19T14:20:24.194 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:20:24.194 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 47845 2024-08-19T14:20:24.194 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.194 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:20:24.194 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.194 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:20:24.194 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.799489+0000", 2024-08-19T14:20:24.194 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:20:24.195 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.195 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.195 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:20:24.195 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.799489+0000", 2024-08-19T14:20:24.195 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:20:24.195 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.195 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.195 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:20:24.195 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.799490+0000", 2024-08-19T14:20:24.195 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.779e-06 2024-08-19T14:20:24.195 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.195 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.195 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:20:24.195 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.799530+0000", 2024-08-19T14:20:24.196 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.9555000000000003e-05 2024-08-19T14:20:24.196 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.196 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.196 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:20:24.196 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.799544+0000", 2024-08-19T14:20:24.196 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4336e-05 2024-08-19T14:20:24.196 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.196 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.196 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:20:24.196 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.799554+0000", 2024-08-19T14:20:24.196 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.5580000000000006e-06 2024-08-19T14:20:24.196 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.196 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.197 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:20:24.197 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.799613+0000", 2024-08-19T14:20:24.197 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.8995e-05 2024-08-19T14:20:24.197 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.197 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.197 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:20:24.197 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.799625+0000", 2024-08-19T14:20:24.197 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2088e-05 2024-08-19T14:20:24.197 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.197 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.197 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:20:24.197 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.760956+0000", 2024-08-19T14:20:24.197 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.96133111299999996 2024-08-19T14:20:24.197 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.198 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.198 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:20:24.198 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.761072+0000", 2024-08-19T14:20:24.198 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000116173 2024-08-19T14:20:24.198 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.198 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.198 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 1,4", 2024-08-19T14:20:24.198 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.761217+0000", 2024-08-19T14:20:24.198 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000144308 2024-08-19T14:20:24.198 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.198 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.198 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:20:24.198 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.763046+0000", 2024-08-19T14:20:24.199 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00182978 2024-08-19T14:20:24.199 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.199 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.199 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:20:24.199 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.763278+0000", 2024-08-19T14:20:24.199 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00023194700000000001 2024-08-19T14:20:24.199 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.199 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.199 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:20:24.199 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.766122+0000", 2024-08-19T14:20:24.199 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0028441439999999998 2024-08-19T14:20:24.199 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.199 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.199 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:20:24.200 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.766190+0000", 2024-08-19T14:20:24.200 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.7595999999999995e-05 2024-08-19T14:20:24.200 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.200 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.200 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:20:24.200 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.766232+0000", 2024-08-19T14:20:24.200 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2159000000000001e-05 2024-08-19T14:20:24.200 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:20:24.200 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:20:24.200 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:20:24.200 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.200 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.200 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25057.0:64601 3.1f 3.eafa94bf (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e246)", 2024-08-19T14:20:24.200 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:17:35.967173+0000", 2024-08-19T14:20:24.201 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 168.157401045, 2024-08-19T14:20:24.201 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.99490361100000002, 2024-08-19T14:20:24.201 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:20:24.201 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:20:24.201 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:20:24.201 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:20:24.201 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25057", 2024-08-19T14:20:24.201 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:20:24.201 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 64601 2024-08-19T14:20:24.201 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.201 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:20:24.201 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.201 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:20:24.201 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:35.967173+0000", 2024-08-19T14:20:24.202 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:20:24.202 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.202 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.202 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:20:24.202 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:35.967173+0000", 2024-08-19T14:20:24.202 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:20:24.202 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.202 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.202 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:20:24.202 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:35.967174+0000", 2024-08-19T14:20:24.202 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.7740000000000001e-06 2024-08-19T14:20:24.202 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.202 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.203 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:20:24.203 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:35.967205+0000", 2024-08-19T14:20:24.203 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.0550999999999999e-05 2024-08-19T14:20:24.203 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.203 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.203 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:20:24.203 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:35.967214+0000", 2024-08-19T14:20:24.203 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.1430000000000004e-06 2024-08-19T14:20:24.203 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.203 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.203 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:20:24.203 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:35.967221+0000", 2024-08-19T14:20:24.203 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.0199999999999997e-06 2024-08-19T14:20:24.203 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.204 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.204 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:20:24.204 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:35.967334+0000", 2024-08-19T14:20:24.204 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000112799 2024-08-19T14:20:24.204 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.204 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.204 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:20:24.204 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:35.967341+0000", 2024-08-19T14:20:24.204 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.3170000000000004e-06 2024-08-19T14:20:24.204 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.204 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.204 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:20:24.204 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:36.962066+0000", 2024-08-19T14:20:24.204 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.99472444500000001 2024-08-19T14:20:24.205 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.205 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.205 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:20:24.205 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:36.962076+0000", 2024-08-19T14:20:24.205 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0562e-05 2024-08-19T14:20:24.205 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:20:24.205 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:20:24.205 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:20:24.205 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.205 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.205 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25057.0:64632 3.1f 3.fd125fbf (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e246)", 2024-08-19T14:20:24.205 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:17:36.181164+0000", 2024-08-19T14:20:24.205 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 167.943409245, 2024-08-19T14:20:24.205 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.78094885599999997, 2024-08-19T14:20:24.206 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:20:24.206 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:20:24.206 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:20:24.206 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:20:24.206 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25057", 2024-08-19T14:20:24.206 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:20:24.206 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 64632 2024-08-19T14:20:24.206 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.206 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:20:24.206 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.206 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:20:24.206 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:36.181164+0000", 2024-08-19T14:20:24.206 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:20:24.207 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.207 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.207 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:20:24.207 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:36.181164+0000", 2024-08-19T14:20:24.207 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:20:24.207 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.207 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.207 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:20:24.207 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:36.181166+0000", 2024-08-19T14:20:24.207 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.001e-06 2024-08-19T14:20:24.207 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.207 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.207 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:20:24.207 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:36.181210+0000", 2024-08-19T14:20:24.208 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4035000000000001e-05 2024-08-19T14:20:24.208 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.208 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.208 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:20:24.208 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:36.181225+0000", 2024-08-19T14:20:24.208 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4620000000000001e-05 2024-08-19T14:20:24.208 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.208 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.208 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:20:24.208 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:36.181235+0000", 2024-08-19T14:20:24.208 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.6709999999999994e-06 2024-08-19T14:20:24.208 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.208 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.209 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:20:24.209 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:36.181309+0000", 2024-08-19T14:20:24.209 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.4191999999999997e-05 2024-08-19T14:20:24.209 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.209 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.209 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:20:24.209 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:36.181321+0000", 2024-08-19T14:20:24.209 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2026e-05 2024-08-19T14:20:24.209 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.209 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.209 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:20:24.209 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:36.962105+0000", 2024-08-19T14:20:24.209 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.780784382 2024-08-19T14:20:24.210 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.210 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.210 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:20:24.210 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:36.962113+0000", 2024-08-19T14:20:24.210 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.9289999999999998e-06 2024-08-19T14:20:24.210 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:20:24.210 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:20:24.210 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:20:24.210 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.210 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.210 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25057.0:188741 3.26 3:65f5d80b:::benchmark_data_smithi045_59973_object57562:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e296)", 2024-08-19T14:20:24.210 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:19:02.154496+0000", 2024-08-19T14:20:24.210 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 81.970077246000002, 2024-08-19T14:20:24.211 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.047062189, 2024-08-19T14:20:24.211 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:20:24.211 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:20:24.211 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:20:24.211 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:20:24.211 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25057", 2024-08-19T14:20:24.211 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:20:24.211 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 188741 2024-08-19T14:20:24.211 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.211 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:20:24.211 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.211 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:20:24.211 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.154496+0000", 2024-08-19T14:20:24.211 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:20:24.212 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.212 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.212 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:20:24.212 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.154496+0000", 2024-08-19T14:20:24.212 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:20:24.212 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.212 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.212 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:20:24.212 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.154497+0000", 2024-08-19T14:20:24.212 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.5000000000000001e-07 2024-08-19T14:20:24.212 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.212 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.212 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:20:24.212 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.154502+0000", 2024-08-19T14:20:24.213 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0690000000000001e-06 2024-08-19T14:20:24.213 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.213 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.213 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:20:24.213 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.154514+0000", 2024-08-19T14:20:24.213 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1922e-05 2024-08-19T14:20:24.213 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.213 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.213 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:20:24.213 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.154522+0000", 2024-08-19T14:20:24.213 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.4730000000000003e-06 2024-08-19T14:20:24.213 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.213 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.213 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:20:24.214 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.275482+0000", 2024-08-19T14:20:24.214 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.12095984899999999 2024-08-19T14:20:24.214 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.214 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.214 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:20:24.214 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.275489+0000", 2024-08-19T14:20:24.214 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.8949999999999998e-06 2024-08-19T14:20:24.214 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.214 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.214 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:20:24.214 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:03.201475+0000", 2024-08-19T14:20:24.214 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.92598516200000003 2024-08-19T14:20:24.214 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.214 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.215 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:20:24.215 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:03.201559+0000", 2024-08-19T14:20:24.215 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.3968999999999995e-05 2024-08-19T14:20:24.215 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:20:24.215 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:20:24.215 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:20:24.215 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.215 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.215 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25057.0:188765 3.26 3:6524f2fe:::benchmark_data_smithi045_59973_object57586:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-08-19T14:20:24.215 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:19:02.192404+0000", 2024-08-19T14:20:24.215 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 81.932170146999994, 2024-08-19T14:20:24.215 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0202571499999999, 2024-08-19T14:20:24.215 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:20:24.215 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:20:24.216 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:20:24.216 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:20:24.216 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25057", 2024-08-19T14:20:24.216 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:20:24.216 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 188765 2024-08-19T14:20:24.216 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.216 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:20:24.216 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.216 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:20:24.216 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.192404+0000", 2024-08-19T14:20:24.216 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:20:24.216 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.216 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.216 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:20:24.217 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.192404+0000", 2024-08-19T14:20:24.217 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:20:24.217 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.217 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.217 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:20:24.217 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.192406+0000", 2024-08-19T14:20:24.217 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.3140000000000002e-06 2024-08-19T14:20:24.217 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.217 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.217 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:20:24.217 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.192417+0000", 2024-08-19T14:20:24.217 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0839000000000001e-05 2024-08-19T14:20:24.217 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.217 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.218 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:20:24.218 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.192440+0000", 2024-08-19T14:20:24.218 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.2911000000000001e-05 2024-08-19T14:20:24.218 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.218 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.218 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:20:24.218 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.192450+0000", 2024-08-19T14:20:24.218 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0292000000000001e-05 2024-08-19T14:20:24.218 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.218 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.218 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:20:24.218 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.275507+0000", 2024-08-19T14:20:24.218 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.083056873000000003 2024-08-19T14:20:24.219 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.219 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.219 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:20:24.219 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.275512+0000", 2024-08-19T14:20:24.219 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.8799999999999999e-06 2024-08-19T14:20:24.219 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.219 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.219 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:20:24.219 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:03.201920+0000", 2024-08-19T14:20:24.219 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.92640795799999998 2024-08-19T14:20:24.219 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.219 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.219 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:20:24.219 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:03.202036+0000", 2024-08-19T14:20:24.220 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000116364 2024-08-19T14:20:24.220 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.220 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.220 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 1,4", 2024-08-19T14:20:24.220 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:03.202142+0000", 2024-08-19T14:20:24.220 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000106115 2024-08-19T14:20:24.220 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.220 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.220 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:20:24.220 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:03.206756+0000", 2024-08-19T14:20:24.220 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0046137319999999997 2024-08-19T14:20:24.220 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.220 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.220 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:20:24.221 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:03.206778+0000", 2024-08-19T14:20:24.221 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.2056999999999999e-05 2024-08-19T14:20:24.221 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.221 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.221 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:20:24.221 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:03.212547+0000", 2024-08-19T14:20:24.221 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0057695740000000004 2024-08-19T14:20:24.221 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.221 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.221 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:20:24.221 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:03.212603+0000", 2024-08-19T14:20:24.221 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.5189999999999998e-05 2024-08-19T14:20:24.221 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.221 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.222 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:20:24.222 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:03.212661+0000", 2024-08-19T14:20:24.222 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.8050999999999997e-05 2024-08-19T14:20:24.222 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:20:24.222 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:20:24.222 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:20:24.222 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.222 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.222 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25057.0:212402 3.12 3:4df4d6c2:::benchmark_data_smithi045_59973_object81223:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e305)", 2024-08-19T14:20:24.222 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:19:29.898935+0000", 2024-08-19T14:20:24.222 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 54.225638998999997, 2024-08-19T14:20:24.222 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.21048530300000001, 2024-08-19T14:20:24.222 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:20:24.223 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:20:24.223 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:20:24.223 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:20:24.223 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25057", 2024-08-19T14:20:24.223 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:20:24.223 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 212402 2024-08-19T14:20:24.223 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.223 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:20:24.223 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.223 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:20:24.223 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:29.898935+0000", 2024-08-19T14:20:24.223 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:20:24.223 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.224 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.224 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:20:24.224 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:29.898935+0000", 2024-08-19T14:20:24.224 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:20:24.224 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.224 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.224 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:20:24.224 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:29.898936+0000", 2024-08-19T14:20:24.224 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1400000000000001e-06 2024-08-19T14:20:24.224 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.224 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.224 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:20:24.224 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:29.898939+0000", 2024-08-19T14:20:24.224 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.2760000000000001e-06 2024-08-19T14:20:24.225 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.225 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.225 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:20:24.225 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:29.898951+0000", 2024-08-19T14:20:24.225 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1977000000000001e-05 2024-08-19T14:20:24.225 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.225 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.225 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:20:24.225 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:29.898957+0000", 2024-08-19T14:20:24.225 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.7010000000000001e-06 2024-08-19T14:20:24.225 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.225 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.225 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:20:24.225 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:29.901409+0000", 2024-08-19T14:20:24.226 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0024526420000000001 2024-08-19T14:20:24.226 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.226 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.226 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:20:24.226 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:29.901520+0000", 2024-08-19T14:20:24.226 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000110568 2024-08-19T14:20:24.226 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.226 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.226 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 1,7", 2024-08-19T14:20:24.226 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:29.901621+0000", 2024-08-19T14:20:24.226 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000100626 2024-08-19T14:20:24.226 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.226 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.226 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:20:24.227 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:29.909377+0000", 2024-08-19T14:20:24.227 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.007756382 2024-08-19T14:20:24.227 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.227 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.227 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:20:24.227 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:29.909483+0000", 2024-08-19T14:20:24.227 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00010644700000000001 2024-08-19T14:20:24.227 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.227 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.227 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:20:24.228 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:30.109317+0000", 2024-08-19T14:20:24.228 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.199834027 2024-08-19T14:20:24.228 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.228 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.228 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:20:24.228 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:30.109374+0000", 2024-08-19T14:20:24.228 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.6801e-05 2024-08-19T14:20:24.228 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.228 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.228 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:20:24.229 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:30.109420+0000", 2024-08-19T14:20:24.229 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.5716000000000001e-05 2024-08-19T14:20:24.229 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:20:24.229 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:20:24.229 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:20:24.229 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.229 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.229 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25057.0:212413 3.12 3:49eb5644:::benchmark_data_smithi045_59973_object81234:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e305)", 2024-08-19T14:20:24.229 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:19:29.909545+0000", 2024-08-19T14:20:24.229 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 54.215029106999999, 2024-08-19T14:20:24.229 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.208006624, 2024-08-19T14:20:24.229 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:20:24.229 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:20:24.229 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:20:24.230 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:20:24.230 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25057", 2024-08-19T14:20:24.230 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:20:24.230 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 212413 2024-08-19T14:20:24.230 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.230 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:20:24.230 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.230 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:20:24.230 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:29.909545+0000", 2024-08-19T14:20:24.230 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:20:24.230 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.230 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.230 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:20:24.231 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:29.909545+0000", 2024-08-19T14:20:24.231 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:20:24.231 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.231 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.231 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:20:24.231 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:29.909546+0000", 2024-08-19T14:20:24.231 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3e-06 2024-08-19T14:20:24.231 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.231 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.231 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:20:24.231 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:29.909551+0000", 2024-08-19T14:20:24.231 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.3870000000000003e-06 2024-08-19T14:20:24.231 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.231 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.232 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:20:24.232 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:29.909561+0000", 2024-08-19T14:20:24.232 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0029e-05 2024-08-19T14:20:24.232 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.232 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.232 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:20:24.232 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:29.909568+0000", 2024-08-19T14:20:24.232 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.5390000000000004e-06 2024-08-19T14:20:24.232 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.232 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.232 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:20:24.232 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:29.911244+0000", 2024-08-19T14:20:24.232 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0016757339999999999 2024-08-19T14:20:24.232 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.233 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.233 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:20:24.233 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:29.911378+0000", 2024-08-19T14:20:24.233 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013467199999999999 2024-08-19T14:20:24.233 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.233 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.233 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 1,7", 2024-08-19T14:20:24.233 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:29.911481+0000", 2024-08-19T14:20:24.233 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000102748 2024-08-19T14:20:24.233 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.233 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.233 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:20:24.233 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:29.919209+0000", 2024-08-19T14:20:24.234 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0077277070000000003 2024-08-19T14:20:24.234 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.234 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.234 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:20:24.234 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:29.919245+0000", 2024-08-19T14:20:24.234 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.6721000000000003e-05 2024-08-19T14:20:24.234 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.234 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.234 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:20:24.234 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:30.117436+0000", 2024-08-19T14:20:24.234 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.198191006 2024-08-19T14:20:24.234 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.234 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.235 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:20:24.235 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:30.117500+0000", 2024-08-19T14:20:24.235 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.3301999999999998e-05 2024-08-19T14:20:24.235 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.235 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.235 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:20:24.235 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:30.117551+0000", 2024-08-19T14:20:24.235 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1479e-05 2024-08-19T14:20:24.235 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:20:24.235 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:20:24.235 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:20:24.235 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.235 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.236 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25057.0:212423 3.12 3:4ddab1a9:::benchmark_data_smithi045_59973_object81244:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e305)", 2024-08-19T14:20:24.236 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:19:29.914909+0000", 2024-08-19T14:20:24.236 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 54.209664490000002, 2024-08-19T14:20:24.236 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.21180168699999999, 2024-08-19T14:20:24.236 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:20:24.236 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:20:24.236 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:20:24.236 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:20:24.236 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25057", 2024-08-19T14:20:24.236 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:20:24.236 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 212423 2024-08-19T14:20:24.236 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.236 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:20:24.236 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.237 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:20:24.237 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:29.914909+0000", 2024-08-19T14:20:24.237 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:20:24.237 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.237 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.237 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:20:24.237 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:29.914909+0000", 2024-08-19T14:20:24.237 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:20:24.237 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.237 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.237 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:20:24.237 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:29.914910+0000", 2024-08-19T14:20:24.237 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.203e-06 2024-08-19T14:20:24.237 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.238 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.238 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:20:24.238 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:29.914913+0000", 2024-08-19T14:20:24.238 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.038e-06 2024-08-19T14:20:24.238 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.238 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.238 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:20:24.238 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:29.914930+0000", 2024-08-19T14:20:24.238 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6467999999999999e-05 2024-08-19T14:20:24.238 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.238 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.238 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:20:24.238 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:29.914937+0000", 2024-08-19T14:20:24.239 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.72e-06 2024-08-19T14:20:24.239 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.239 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.239 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:20:24.239 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:29.915464+0000", 2024-08-19T14:20:24.239 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00052781300000000002 2024-08-19T14:20:24.239 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.239 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.239 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:20:24.239 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:29.915580+0000", 2024-08-19T14:20:24.239 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000116074 2024-08-19T14:20:24.239 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.239 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.239 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 1,7", 2024-08-19T14:20:24.240 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:29.915694+0000", 2024-08-19T14:20:24.240 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000113217 2024-08-19T14:20:24.240 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.240 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.240 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:20:24.240 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:29.919256+0000", 2024-08-19T14:20:24.240 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0035625069999999999 2024-08-19T14:20:24.240 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.240 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.240 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:20:24.240 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:29.919490+0000", 2024-08-19T14:20:24.240 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00023429599999999999 2024-08-19T14:20:24.240 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.240 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.241 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:20:24.241 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:30.126603+0000", 2024-08-19T14:20:24.241 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.20711281100000001 2024-08-19T14:20:24.241 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.241 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.241 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:20:24.241 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:30.126658+0000", 2024-08-19T14:20:24.241 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.4342000000000002e-05 2024-08-19T14:20:24.241 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.241 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.241 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:20:24.241 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:30.126711+0000", 2024-08-19T14:20:24.241 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.3198000000000001e-05 2024-08-19T14:20:24.241 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:20:24.242 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:20:24.242 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:20:24.242 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.242 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.242 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25057.0:212434 3.12 3:4a0157f2:::benchmark_data_smithi045_59973_object81255:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e305)", 2024-08-19T14:20:24.242 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:19:29.924106+0000", 2024-08-19T14:20:24.242 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 54.200467707999998, 2024-08-19T14:20:24.242 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.20653502600000001, 2024-08-19T14:20:24.242 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:20:24.242 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:20:24.242 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:20:24.242 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:20:24.242 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25057", 2024-08-19T14:20:24.243 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:20:24.243 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 212434 2024-08-19T14:20:24.243 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.243 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:20:24.243 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.243 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:20:24.243 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:29.924106+0000", 2024-08-19T14:20:24.243 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:20:24.243 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.243 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.243 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:20:24.243 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:29.924106+0000", 2024-08-19T14:20:24.243 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:20:24.243 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.244 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.244 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:20:24.244 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:29.924107+0000", 2024-08-19T14:20:24.244 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4279999999999999e-06 2024-08-19T14:20:24.244 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.244 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.244 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:20:24.244 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:29.924113+0000", 2024-08-19T14:20:24.244 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.0519999999999998e-06 2024-08-19T14:20:24.244 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.244 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.245 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:20:24.245 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:29.924126+0000", 2024-08-19T14:20:24.245 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2876e-05 2024-08-19T14:20:24.245 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.245 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.245 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:20:24.245 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:20:24.246 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:29.924134+0000", 2024-08-19T14:20:24.246 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.5839999999999997e-06 2024-08-19T14:20:24.246 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.247 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.247 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:20:24.247 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:29.924194+0000", 2024-08-19T14:20:24.247 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.9794999999999999e-05 2024-08-19T14:20:24.247 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.247 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.247 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:20:24.247 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:29.924296+0000", 2024-08-19T14:20:24.247 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00010235599999999999 2024-08-19T14:20:24.247 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.247 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.247 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 1,7", 2024-08-19T14:20:24.247 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:29.924407+0000", 2024-08-19T14:20:24.248 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000110611 2024-08-19T14:20:24.248 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.248 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.248 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:20:24.248 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:30.092599+0000", 2024-08-19T14:20:24.248 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.168191866 2024-08-19T14:20:24.248 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.248 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.248 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:20:24.248 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:30.092610+0000", 2024-08-19T14:20:24.249 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.134e-05 2024-08-19T14:20:24.249 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.249 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.249 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:20:24.249 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:30.130545+0000", 2024-08-19T14:20:24.249 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.037935037999999997 2024-08-19T14:20:24.249 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.249 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.249 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:20:24.249 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:30.130599+0000", 2024-08-19T14:20:24.249 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.4568000000000003e-05 2024-08-19T14:20:24.249 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.249 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.250 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:20:24.250 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:30.130641+0000", 2024-08-19T14:20:24.250 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.1511999999999998e-05 2024-08-19T14:20:24.250 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:20:24.250 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:20:24.250 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:20:24.250 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.250 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.250 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25057.0:216419 3.12 3:4d97b980:::benchmark_data_smithi045_59973_object85240:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e307)", 2024-08-19T14:20:24.250 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:19:33.989271+0000", 2024-08-19T14:20:24.250 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 50.135303033, 2024-08-19T14:20:24.250 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.21231334399999999, 2024-08-19T14:20:24.250 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:20:24.251 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:20:24.251 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:20:24.251 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:20:24.251 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25057", 2024-08-19T14:20:24.251 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:20:24.251 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 216419 2024-08-19T14:20:24.251 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.251 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:20:24.251 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.251 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:20:24.251 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:33.989271+0000", 2024-08-19T14:20:24.251 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:20:24.251 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.251 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.252 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:20:24.252 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:33.989271+0000", 2024-08-19T14:20:24.252 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:20:24.252 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.252 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.252 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:20:24.252 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:33.989272+0000", 2024-08-19T14:20:24.252 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3850000000000001e-06 2024-08-19T14:20:24.252 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.252 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.252 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:20:24.252 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:33.989277+0000", 2024-08-19T14:20:24.252 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.8149999999999996e-06 2024-08-19T14:20:24.252 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.253 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.253 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:20:24.253 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:33.989287+0000", 2024-08-19T14:20:24.253 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0608000000000001e-05 2024-08-19T14:20:24.253 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.253 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.253 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:20:24.253 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:33.989294+0000", 2024-08-19T14:20:24.253 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.8990000000000004e-06 2024-08-19T14:20:24.253 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.253 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.253 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:20:24.253 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:33.990824+0000", 2024-08-19T14:20:24.253 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.001529318 2024-08-19T14:20:24.253 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.254 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.254 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:20:24.254 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:33.990949+0000", 2024-08-19T14:20:24.254 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012494200000000001 2024-08-19T14:20:24.254 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.254 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.254 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 1,7", 2024-08-19T14:20:24.254 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:34.165620+0000", 2024-08-19T14:20:24.254 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.174671086 2024-08-19T14:20:24.254 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.254 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.254 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:20:24.254 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:34.169291+0000", 2024-08-19T14:20:24.255 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0036716800000000001 2024-08-19T14:20:24.255 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.255 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.255 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:20:24.255 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:34.169480+0000", 2024-08-19T14:20:24.255 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00018827699999999999 2024-08-19T14:20:24.255 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.255 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.255 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:20:24.255 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:34.201489+0000", 2024-08-19T14:20:24.255 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.032009092000000003 2024-08-19T14:20:24.255 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.255 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.255 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:20:24.255 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:34.201544+0000", 2024-08-19T14:20:24.256 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.4889999999999998e-05 2024-08-19T14:20:24.256 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:24.256 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:24.256 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:20:24.256 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:34.201584+0000", 2024-08-19T14:20:24.256 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.0352000000000003e-05 2024-08-19T14:20:24.256 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:20:24.256 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:20:24.256 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:20:24.256 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:20:24.256 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:20:24.256 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:20:24.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:23 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1285016895' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-08-19T14:20:24.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:23 smithi032 ceph-mon[20004]: osdmap e330: 12 total, 12 up, 9 in 2024-08-19T14:20:24.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:23 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1516330677' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:20:24.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:23 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/126522565' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-08-19T14:20:24.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:24.271 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:24.271 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:20:24.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:23 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1285016895' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-08-19T14:20:24.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:23 smithi045 ceph-mon[20143]: osdmap e330: 12 total, 12 up, 9 in 2024-08-19T14:20:24.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:23 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1516330677' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:20:24.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:23 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/126522565' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-08-19T14:20:24.372 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:20:24.398 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:24.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:24.399 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:20:24.500 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:20:24.525 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:24.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:24.526 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:20:24.627 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:20:24.652 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:24.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:24.653 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:20:24.754 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:20:24.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:24.781 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:24.781 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:20:24.882 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:20:24.905 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:24.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:24.906 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:20:25.007 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:20:25.033 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:25.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:25.034 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:20:25.135 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:20:25.159 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:25.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:25.160 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:20:25.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:24 smithi032 ceph-mon[20004]: pgmap v745: 31 pgs: 31 active+clean; 577 KiB data, 3.1 GiB used, 801 GiB / 805 GiB avail 2024-08-19T14:20:25.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:24 smithi032 ceph-mon[20004]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:20:25.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:24 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1516330677' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:20:25.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:24 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/126522565' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-08-19T14:20:25.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:24 smithi032 ceph-mon[20004]: osdmap e331: 12 total, 12 up, 9 in 2024-08-19T14:20:25.261 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:20:25.286 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:25.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:25.287 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:20:25.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:24 smithi045 ceph-mon[20143]: pgmap v745: 31 pgs: 31 active+clean; 577 KiB data, 3.1 GiB used, 801 GiB / 805 GiB avail 2024-08-19T14:20:25.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:24 smithi045 ceph-mon[20143]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:20:25.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:24 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1516330677' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:20:25.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:24 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/126522565' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-08-19T14:20:25.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:24 smithi045 ceph-mon[20143]: osdmap e331: 12 total, 12 up, 9 in 2024-08-19T14:20:25.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:24 smithi031 ceph-mon[16336]: pgmap v745: 31 pgs: 31 active+clean; 577 KiB data, 3.1 GiB used, 801 GiB / 805 GiB avail 2024-08-19T14:20:25.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:24 smithi031 ceph-mon[16336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:20:25.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:24 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1516330677' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:20:25.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:24 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/126522565' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-08-19T14:20:25.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:24 smithi031 ceph-mon[16336]: osdmap e331: 12 total, 12 up, 9 in 2024-08-19T14:20:25.389 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:20:25.412 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:25.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:25.413 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:20:25.514 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:20:25.540 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:25.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:25.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:20:25.642 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:20:25.668 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:25.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:25.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:20:25.770 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:20:25.772 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-08-19T14:20:25.773 INFO:tasks.radosbench:Beginning radosbench... 2024-08-19T14:20:25.773 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_2 2024-08-19T14:20:25.773 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd pool create unique_pool_2 16 2024-08-19T14:20:25.797 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:25.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:25.799 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:20:25.900 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:20:25.920 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:25.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:25.921 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:20:26.022 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:20:26.043 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:26.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:26.044 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:20:26.144 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:20:26.153 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd reweight 4 1 2024-08-19T14:20:26.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:26.172 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:26.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:20:26.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:25 smithi032 ceph-mon[20004]: osdmap e332: 12 total, 12 up, 9 in 2024-08-19T14:20:26.273 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:20:26.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:25 smithi045 ceph-mon[20143]: osdmap e332: 12 total, 12 up, 9 in 2024-08-19T14:20:26.299 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:26.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:26.300 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:20:26.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:25 smithi031 ceph-mon[16336]: osdmap e332: 12 total, 12 up, 9 in 2024-08-19T14:20:26.400 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:20:26.425 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:26.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:26.427 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:20:26.528 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:20:26.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:26.550 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:26.550 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:20:26.650 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:20:26.676 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:26.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:26.677 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:20:26.778 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:20:26.801 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:26.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:26.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:20:26.903 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:20:26.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:26.927 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:26.927 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:20:26.936 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-08-19T14:20:26.961 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:20:27.002 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:20:27.012 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:20:27.028 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:20:27.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:27.054 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:27.054 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:20:27.156 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:20:27.181 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:27.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:27.182 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:20:27.204 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:27 smithi031 ceph-mon[16336]: pgmap v748: 31 pgs: 1 unknown, 30 active+clean; 577 KiB data, 3.2 GiB used, 801 GiB / 805 GiB avail 2024-08-19T14:20:27.204 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:27 smithi031 ceph-mon[16336]: osdmap e333: 12 total, 12 up, 9 in 2024-08-19T14:20:27.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:27 smithi032 ceph-mon[20004]: pgmap v748: 31 pgs: 1 unknown, 30 active+clean; 577 KiB data, 3.2 GiB used, 801 GiB / 805 GiB avail 2024-08-19T14:20:27.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:27 smithi032 ceph-mon[20004]: osdmap e333: 12 total, 12 up, 9 in 2024-08-19T14:20:27.283 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:20:27.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:27 smithi045 ceph-mon[20143]: pgmap v748: 31 pgs: 1 unknown, 30 active+clean; 577 KiB data, 3.2 GiB used, 801 GiB / 805 GiB avail 2024-08-19T14:20:27.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:27 smithi045 ceph-mon[20143]: osdmap e333: 12 total, 12 up, 9 in 2024-08-19T14:20:27.309 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:27.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:27.310 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:20:27.411 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:20:27.437 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:27.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:27.438 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:20:27.478 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-19T14:20:27.539 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:20:27.598 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:27.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:27.599 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:20:27.700 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:20:27.721 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:27.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:27.722 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:20:27.823 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:20:27.844 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:27.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:27.845 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:20:27.946 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:20:27.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:27.976 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:27.976 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:20:28.076 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:20:28.102 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:28.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:28.103 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:20:28.196 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'unique_pool_2' created 2024-08-19T14:20:28.203 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:20:28.227 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:28.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:28.228 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:20:28.329 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:20:28.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:28.351 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:28.351 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:20:28.452 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:20:28.474 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:28.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:28.475 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:20:28.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:28 smithi031 ceph-mon[16336]: osdmap e334: 12 total, 12 up, 9 in 2024-08-19T14:20:28.493 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:28 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1290503000' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-08-19T14:20:28.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:28 smithi032 ceph-mon[20004]: osdmap e334: 12 total, 12 up, 9 in 2024-08-19T14:20:28.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:28 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1290503000' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-08-19T14:20:28.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:28 smithi045 ceph-mon[20143]: osdmap e334: 12 total, 12 up, 9 in 2024-08-19T14:20:28.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:28 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1290503000' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-08-19T14:20:28.576 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:20:28.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:28.599 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:28.599 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:20:28.701 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:20:28.722 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:28.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:28.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:20:28.825 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:20:28.837 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd pool application enable unique_pool_2 rados --yes-i-really-mean-it || true 2024-08-19T14:20:28.850 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:28.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:28.851 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:20:28.952 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:20:28.977 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:28.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:28.978 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:20:29.079 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:20:29.101 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:20:29.101 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:20:29.104 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:29.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:29.105 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:20:29.105 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:20:29.106 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:20:29.126 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:20:29.127 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:20:29.149 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:20:29.150 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:20:29.150 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:20:29.168 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:20:29.168 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:20:29.186 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:20:29.186 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:20:29.197 INFO:teuthology.orchestra.run.smithi031.stderr:reweighted osd.4 to 1 (10000) 2024-08-19T14:20:29.205 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:20:29.206 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:20:29.206 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:20:29.225 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:20:29.226 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:20:29.231 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:29.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:29.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:20:29.244 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:20:29.244 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:20:29.262 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:20:29.262 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:20:29.281 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:20:29.281 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:20:29.305 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:20:29.305 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:20:29.322 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:20:29.323 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:20:29.334 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:20:29.356 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:29.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:29.357 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:20:29.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:29 smithi031 ceph-mon[16336]: pgmap v751: 31 pgs: 1 active+remapped, 30 active+clean; 577 KiB data, 3.2 GiB used, 801 GiB / 805 GiB avail 2024-08-19T14:20:29.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:29 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1290503000' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-08-19T14:20:29.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:29 smithi031 ceph-mon[16336]: osdmap e335: 12 total, 12 up, 9 in 2024-08-19T14:20:29.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:29 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/837859077' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-08-19T14:20:29.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:29 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1531781797' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:20:29.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:29 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/837859077' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-08-19T14:20:29.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:29 smithi031 ceph-mon[16336]: osdmap e336: 12 total, 12 up, 9 in 2024-08-19T14:20:29.458 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:20:29.484 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:29.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:29.485 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:20:29.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:29 smithi032 ceph-mon[20004]: pgmap v751: 31 pgs: 1 active+remapped, 30 active+clean; 577 KiB data, 3.2 GiB used, 801 GiB / 805 GiB avail 2024-08-19T14:20:29.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:29 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1290503000' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-08-19T14:20:29.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:29 smithi032 ceph-mon[20004]: osdmap e335: 12 total, 12 up, 9 in 2024-08-19T14:20:29.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:29 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/837859077' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-08-19T14:20:29.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:29 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1531781797' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:20:29.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:29 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/837859077' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-08-19T14:20:29.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:29 smithi032 ceph-mon[20004]: osdmap e336: 12 total, 12 up, 9 in 2024-08-19T14:20:29.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:29 smithi045 ceph-mon[20143]: pgmap v751: 31 pgs: 1 active+remapped, 30 active+clean; 577 KiB data, 3.2 GiB used, 801 GiB / 805 GiB avail 2024-08-19T14:20:29.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:29 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1290503000' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-08-19T14:20:29.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:29 smithi045 ceph-mon[20143]: osdmap e335: 12 total, 12 up, 9 in 2024-08-19T14:20:29.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:29 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/837859077' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-08-19T14:20:29.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:29 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1531781797' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:20:29.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:29 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/837859077' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-08-19T14:20:29.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:29 smithi045 ceph-mon[20143]: osdmap e336: 12 total, 12 up, 9 in 2024-08-19T14:20:29.586 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:20:29.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:29.609 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:29.609 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:20:29.710 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:20:29.736 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:29.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:29.737 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:20:29.838 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:20:29.864 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:29.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:29.865 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:20:29.966 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:20:29.987 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:29.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:29.988 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:20:30.020 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:20:30.020 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:20:30.089 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:20:30.115 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:30.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:30.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:20:30.208 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is unset 2024-08-19T14:20:30.217 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:20:30.239 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:30.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:30.240 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:20:30.341 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:20:30.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:30 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2375993584' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:20:30.367 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:30.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:30.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:20:30.469 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:20:30.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:30.491 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:30.491 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:20:30.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:30 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2375993584' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:20:30.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:30 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2375993584' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:20:30.592 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:20:30.618 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:30.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:30.619 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:20:30.720 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:20:30.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:30.743 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:30.743 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:20:30.845 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:20:30.870 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:30.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:30.871 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:20:30.973 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:20:30.998 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:30.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:30.999 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:20:31.100 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:20:31.126 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:31.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:31.127 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:20:31.228 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:20:31.254 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:31.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:31.255 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:20:31.356 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:20:31.382 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:31.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:31.383 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:20:31.484 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:20:31.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:31 smithi032 ceph-mon[20004]: pgmap v754: 47 pgs: 6 creating+peering, 9 unknown, 32 active+clean; 577 KiB data, 3.2 GiB used, 801 GiB / 805 GiB avail 2024-08-19T14:20:31.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:31 smithi032 ceph-mon[20004]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:20:31.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:31 smithi032 ceph-mon[20004]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-19T14:20:31.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:31 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2375993584' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:20:31.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:31 smithi032 ceph-mon[20004]: osdmap e337: 12 total, 12 up, 9 in 2024-08-19T14:20:31.509 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:31.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:31.510 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:20:31.533 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:31 smithi031 ceph-mon[16336]: pgmap v754: 47 pgs: 6 creating+peering, 9 unknown, 32 active+clean; 577 KiB data, 3.2 GiB used, 801 GiB / 805 GiB avail 2024-08-19T14:20:31.533 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:31 smithi031 ceph-mon[16336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:20:31.533 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:31 smithi031 ceph-mon[16336]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-19T14:20:31.533 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:31 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2375993584' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:20:31.533 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:31 smithi031 ceph-mon[16336]: osdmap e337: 12 total, 12 up, 9 in 2024-08-19T14:20:31.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:31 smithi045 ceph-mon[20143]: pgmap v754: 47 pgs: 6 creating+peering, 9 unknown, 32 active+clean; 577 KiB data, 3.2 GiB used, 801 GiB / 805 GiB avail 2024-08-19T14:20:31.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:31 smithi045 ceph-mon[20143]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:20:31.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:31 smithi045 ceph-mon[20143]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-19T14:20:31.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:31 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2375993584' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:20:31.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:31 smithi045 ceph-mon[20143]: osdmap e337: 12 total, 12 up, 9 in 2024-08-19T14:20:31.563 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd reweight 5 1 2024-08-19T14:20:31.612 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:20:31.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:31.634 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:31.634 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:20:31.735 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:20:31.757 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:31.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:31.758 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:20:31.859 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:20:31.881 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:31.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:31.882 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:20:31.983 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:20:32.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:32.007 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:32.007 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:20:32.108 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:20:32.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:32.130 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:32.130 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:20:32.224 INFO:teuthology.orchestra.run.smithi031.stderr:enabled application 'rados' on pool 'unique_pool_2' 2024-08-19T14:20:32.231 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:20:32.256 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:32.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:32.257 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:20:32.358 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:20:32.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:32 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2279231192' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-08-19T14:20:32.383 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:32.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:32.384 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:20:32.485 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:20:32.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:32 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2279231192' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-08-19T14:20:32.511 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:32.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:32.512 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:20:32.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:32 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2279231192' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-08-19T14:20:32.613 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:20:32.634 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:32.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:32.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:20:32.736 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:20:32.758 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:32.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:32.759 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:20:32.780 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-19T14:20:32.860 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:20:32.882 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:32.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:32.883 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:20:32.984 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:20:33.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:33.007 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:33.007 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:20:33.108 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:20:33.134 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:33.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:33.135 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:20:33.236 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:20:33.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:33.263 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:33.263 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:20:33.363 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:20:33.370 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-08-19T14:20:33.385 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:33.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:33.386 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:20:33.403 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:20:33.404 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:20:33.487 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:20:33.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:33 smithi032 ceph-mon[20004]: pgmap v756: 47 pgs: 6 creating+peering, 3 unknown, 38 active+clean; 577 KiB data, 3.2 GiB used, 801 GiB / 805 GiB avail 2024-08-19T14:20:33.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:33 smithi032 ceph-mon[20004]: 1.17 scrub starts 2024-08-19T14:20:33.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:33 smithi032 ceph-mon[20004]: 1.17 scrub ok 2024-08-19T14:20:33.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:33 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2279231192' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-08-19T14:20:33.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:33 smithi032 ceph-mon[20004]: osdmap e338: 12 total, 12 up, 9 in 2024-08-19T14:20:33.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:33.514 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:33.514 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:20:33.527 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:33 smithi031 ceph-mon[16336]: pgmap v756: 47 pgs: 6 creating+peering, 3 unknown, 38 active+clean; 577 KiB data, 3.2 GiB used, 801 GiB / 805 GiB avail 2024-08-19T14:20:33.527 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:33 smithi031 ceph-mon[16336]: 1.17 scrub starts 2024-08-19T14:20:33.527 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:33 smithi031 ceph-mon[16336]: 1.17 scrub ok 2024-08-19T14:20:33.527 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:33 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2279231192' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-08-19T14:20:33.527 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:33 smithi031 ceph-mon[16336]: osdmap e338: 12 total, 12 up, 9 in 2024-08-19T14:20:33.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:33 smithi045 ceph-mon[20143]: pgmap v756: 47 pgs: 6 creating+peering, 3 unknown, 38 active+clean; 577 KiB data, 3.2 GiB used, 801 GiB / 805 GiB avail 2024-08-19T14:20:33.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:33 smithi045 ceph-mon[20143]: 1.17 scrub starts 2024-08-19T14:20:33.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:33 smithi045 ceph-mon[20143]: 1.17 scrub ok 2024-08-19T14:20:33.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:33 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2279231192' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-08-19T14:20:33.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:33 smithi045 ceph-mon[20143]: osdmap e338: 12 total, 12 up, 9 in 2024-08-19T14:20:33.615 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:20:33.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:33.642 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:33.642 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:20:33.744 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:20:33.769 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:33.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:33.770 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:20:33.875 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:20:33.901 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:33.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:33.902 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:20:33.989 DEBUG:teuthology.orchestra.run.smithi045:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_2 bench 90 write' 2024-08-19T14:20:33.992 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-08-19T14:20:33.992 INFO:teuthology.orchestra.run:waiting for 3000 2024-08-19T14:20:34.002 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:20:34.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:34.025 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:34.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:20:34.044 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:hints = 1 2024-08-19T14:20:34.044 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-08-19T14:20:34.045 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Object prefix: benchmark_data_smithi045_71562 2024-08-19T14:20:34.045 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-19T14:20:34.045 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 0 0 0 0 0 0 - 0 2024-08-19T14:20:34.126 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:20:34.164 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:34.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:34.165 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:20:34.266 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:20:34.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:34.303 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:34.303 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:20:34.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:34 smithi031 ceph-mon[16336]: 1.19 scrub starts 2024-08-19T14:20:34.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:34 smithi031 ceph-mon[16336]: 1.19 scrub ok 2024-08-19T14:20:34.405 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:20:34.415 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:20:34.433 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:34.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:34.435 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:20:34.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:34 smithi032 ceph-mon[20004]: 1.19 scrub starts 2024-08-19T14:20:34.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:34 smithi032 ceph-mon[20004]: 1.19 scrub ok 2024-08-19T14:20:34.536 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:20:34.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:34 smithi045 ceph-mon[20143]: 1.19 scrub starts 2024-08-19T14:20:34.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:34 smithi045 ceph-mon[20143]: 1.19 scrub ok 2024-08-19T14:20:34.550 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:20:34.564 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:34.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:34.566 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:20:34.667 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:20:34.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:34.691 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:34.691 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:20:34.793 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:20:34.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:34.815 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:34.815 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:20:34.916 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:20:34.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:34.942 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:34.942 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:20:35.043 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:20:35.044 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 1 16 3107 3091 193.177 193.188 0.00246675 0.00516064 2024-08-19T14:20:35.075 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:35.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:35.076 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:20:35.177 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:20:35.197 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:20:35.197 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:20:35.204 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:20:35.207 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:35.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:35.209 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:20:35.243 INFO:teuthology.orchestra.run.smithi031.stderr:reweighted osd.5 to 1 (10000) 2024-08-19T14:20:35.243 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is unset 2024-08-19T14:20:35.311 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:20:35.343 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:35.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:35.345 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:20:35.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:35 smithi031 ceph-mon[16336]: pgmap v758: 47 pgs: 47 active+clean; 577 KiB data, 3.2 GiB used, 801 GiB / 805 GiB avail 2024-08-19T14:20:35.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:35 smithi031 ceph-mon[16336]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-19T14:20:35.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:35 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2807047474' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:20:35.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:35 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:20:35.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:35 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2840533248' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-08-19T14:20:35.445 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:20:35.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:35.467 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:35.467 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:20:35.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:35 smithi032 ceph-mon[20004]: pgmap v758: 47 pgs: 47 active+clean; 577 KiB data, 3.2 GiB used, 801 GiB / 805 GiB avail 2024-08-19T14:20:35.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:35 smithi032 ceph-mon[20004]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-19T14:20:35.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:35 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2807047474' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:20:35.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:35 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:20:35.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:35 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2840533248' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-08-19T14:20:35.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:35 smithi045 ceph-mon[20143]: pgmap v758: 47 pgs: 47 active+clean; 577 KiB data, 3.2 GiB used, 801 GiB / 805 GiB avail 2024-08-19T14:20:35.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:35 smithi045 ceph-mon[20143]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-19T14:20:35.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:35 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2807047474' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:20:35.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:35 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:20:35.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:35 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2840533248' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-08-19T14:20:35.568 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:20:35.593 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:35.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:35.594 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:20:35.695 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:20:35.728 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:35.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:35.729 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:20:35.818 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-08-19T14:20:35.834 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:20:35.865 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:35.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:35.866 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:20:35.966 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:20:35.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:35.999 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:35.999 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:20:36.044 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 2 16 6605 6589 205.892 218.625 0.00451064 0.00484824 2024-08-19T14:20:36.100 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:20:36.137 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:36.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:36.139 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:20:36.240 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:20:36.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:36.282 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:36.282 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:20:36.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:36 smithi031 ceph-mon[16336]: 1.7 scrub starts 2024-08-19T14:20:36.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:36 smithi031 ceph-mon[16336]: 1.7 scrub ok 2024-08-19T14:20:36.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:36 smithi031 ceph-mon[16336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:20:36.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:36 smithi031 ceph-mon[16336]: Cluster is now healthy 2024-08-19T14:20:36.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:36 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:20:36.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:36 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2840533248' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-08-19T14:20:36.368 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:36 smithi031 ceph-mon[16336]: osdmap e339: 12 total, 12 up, 9 in 2024-08-19T14:20:36.383 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:20:36.417 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:36.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:36.418 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:20:36.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:36 smithi032 ceph-mon[20004]: 1.7 scrub starts 2024-08-19T14:20:36.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:36 smithi032 ceph-mon[20004]: 1.7 scrub ok 2024-08-19T14:20:36.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:36 smithi032 ceph-mon[20004]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:20:36.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:36 smithi032 ceph-mon[20004]: Cluster is now healthy 2024-08-19T14:20:36.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:36 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:20:36.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:36 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2840533248' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-08-19T14:20:36.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:36 smithi032 ceph-mon[20004]: osdmap e339: 12 total, 12 up, 9 in 2024-08-19T14:20:36.519 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:20:36.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:36 smithi045 ceph-mon[20143]: 1.7 scrub starts 2024-08-19T14:20:36.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:36 smithi045 ceph-mon[20143]: 1.7 scrub ok 2024-08-19T14:20:36.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:36 smithi045 ceph-mon[20143]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:20:36.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:36 smithi045 ceph-mon[20143]: Cluster is now healthy 2024-08-19T14:20:36.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:36 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:20:36.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:36 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2840533248' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-08-19T14:20:36.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:36 smithi045 ceph-mon[20143]: osdmap e339: 12 total, 12 up, 9 in 2024-08-19T14:20:36.549 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:36.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:36.550 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:20:36.651 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:20:36.681 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:36.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:36.682 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:20:36.783 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:20:36.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:36.814 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:36.815 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:20:36.914 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:20:36.914 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:20:36.916 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:20:36.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:36.949 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:36.949 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:20:36.950 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:20:36.950 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:20:36.986 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:20:36.987 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:20:37.021 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:20:37.021 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:20:37.044 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 3 16 10587 10571 220.213 248.875 0.00314925 0.00453437 2024-08-19T14:20:37.050 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:20:37.056 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:20:37.056 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:20:37.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:37.081 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:37.082 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:20:37.088 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:20:37.088 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:20:37.127 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:20:37.127 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:20:37.162 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:20:37.163 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:20:37.182 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:20:37.190 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:20:37.191 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:20:37.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:37.204 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:37.204 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:20:37.207 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:20:37.208 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:20:37.231 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:20:37.231 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:20:37.266 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:20:37.267 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:20:37.305 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:20:37.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:37.339 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:37.340 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:20:37.441 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:20:37.475 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:37.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:37.477 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:20:37.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:37 smithi032 ceph-mon[20004]: pgmap v760: 47 pgs: 47 active+clean; 110 MiB data, 3.2 GiB used, 801 GiB / 805 GiB avail; 18 MiB/s wr, 292 op/s 2024-08-19T14:20:37.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:37 smithi032 ceph-mon[20004]: 1.18 scrub starts 2024-08-19T14:20:37.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:37 smithi032 ceph-mon[20004]: 1.18 scrub ok 2024-08-19T14:20:37.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:37 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1902300437' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:20:37.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:37 smithi045 ceph-mon[20143]: pgmap v760: 47 pgs: 47 active+clean; 110 MiB data, 3.2 GiB used, 801 GiB / 805 GiB avail; 18 MiB/s wr, 292 op/s 2024-08-19T14:20:37.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:37 smithi045 ceph-mon[20143]: 1.18 scrub starts 2024-08-19T14:20:37.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:37 smithi045 ceph-mon[20143]: 1.18 scrub ok 2024-08-19T14:20:37.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:37 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1902300437' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:20:37.577 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:20:37.613 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:37.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:37.615 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:20:37.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:37 smithi031 ceph-mon[16336]: pgmap v760: 47 pgs: 47 active+clean; 110 MiB data, 3.2 GiB used, 801 GiB / 805 GiB avail; 18 MiB/s wr, 292 op/s 2024-08-19T14:20:37.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:37 smithi031 ceph-mon[16336]: 1.18 scrub starts 2024-08-19T14:20:37.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:37 smithi031 ceph-mon[16336]: 1.18 scrub ok 2024-08-19T14:20:37.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:37 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1902300437' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:20:37.716 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:20:37.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:37.748 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:37.748 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:20:37.849 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:20:37.880 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:37.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:37.881 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:20:37.983 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:20:38.016 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:38.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:38.017 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:20:38.044 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 4 16 14273 14257 222.751 230.375 0.00219053 0.00448362 2024-08-19T14:20:38.118 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:20:38.150 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:38.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:38.151 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:20:38.252 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:20:38.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:38.281 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:38.281 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:20:38.306 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd reweight 7 1 2024-08-19T14:20:38.321 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:20:38.382 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:20:38.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:38.410 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:38.410 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:20:38.511 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:20:38.533 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:38.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:38.534 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:20:38.635 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:20:38.668 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:38.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:38.669 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:20:38.686 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-19T14:20:38.770 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:20:38.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:38.809 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:38.809 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:20:38.911 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:20:38.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:38.942 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:38.943 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:20:39.043 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:20:39.044 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 5 16 17092 17076 213.435 176.188 0.00299352 0.00468026 2024-08-19T14:20:39.075 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:39.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:39.076 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:20:39.177 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:20:39.208 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:39.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:39.209 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:20:39.310 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:20:39.338 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:39.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:39.340 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:20:39.441 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:20:39.464 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:39.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:39.465 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:20:39.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:39 smithi045 ceph-mon[20143]: pgmap v761: 47 pgs: 47 active+clean; 139 MiB data, 3.5 GiB used, 801 GiB / 805 GiB avail; 18 MiB/s wr, 288 op/s 2024-08-19T14:20:39.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:39 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:20:39.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:39 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:20:39.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:39 smithi045 ceph-mon[20143]: osdmap e340: 12 total, 12 up, 9 in 2024-08-19T14:20:39.565 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:20:39.599 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:39.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:39.600 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:20:39.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:39 smithi031 ceph-mon[16336]: pgmap v761: 47 pgs: 47 active+clean; 139 MiB data, 3.5 GiB used, 801 GiB / 805 GiB avail; 18 MiB/s wr, 288 op/s 2024-08-19T14:20:39.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:39 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:20:39.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:39 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:20:39.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:39 smithi031 ceph-mon[16336]: osdmap e340: 12 total, 12 up, 9 in 2024-08-19T14:20:39.701 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:20:39.732 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:39.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:39.734 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:20:39.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:39 smithi032 ceph-mon[20004]: pgmap v761: 47 pgs: 47 active+clean; 139 MiB data, 3.5 GiB used, 801 GiB / 805 GiB avail; 18 MiB/s wr, 288 op/s 2024-08-19T14:20:39.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:39 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:20:39.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:39 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:20:39.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:39 smithi032 ceph-mon[20004]: osdmap e340: 12 total, 12 up, 9 in 2024-08-19T14:20:39.835 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:20:39.857 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:39.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:39.858 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:20:39.959 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:20:39.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:39.990 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:39.990 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:20:40.044 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 6 16 20100 20084 209.193 188 0.00327308 0.00477518 2024-08-19T14:20:40.091 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:20:40.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:40.124 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:40.124 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:20:40.225 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:20:40.255 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:40.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:40.256 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:20:40.357 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:20:40.388 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:40.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:40.389 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:20:40.490 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:20:40.521 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:40.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:40.522 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:20:40.623 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:20:40.646 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:40.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:40.648 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:20:40.748 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:20:40.770 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:40.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:40.772 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:20:40.873 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:20:40.917 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:40.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:40.918 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:20:40.961 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:20:41.019 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:20:41.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:41.043 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:41.043 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:20:41.044 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 7 16 22174 22158 197.823 129.625 0.00502387 0.00503824 2024-08-19T14:20:41.144 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:20:41.174 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:41.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:41.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:20:41.228 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:20:41.228 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 20, 2024-08-19T14:20:41.229 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 600, 2024-08-19T14:20:41.229 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [ 2024-08-19T14:20:41.229 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.229 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:40314 2.6 2:65a7b859:::benchmark_data_smithi045_46415_object40313:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-08-19T14:20:41.229 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:13:02.993308+0000", 2024-08-19T14:20:41.229 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 457.21866774799997, 2024-08-19T14:20:41.229 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.20540061800000001, 2024-08-19T14:20:41.229 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:20:41.229 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:20:41.230 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:20:41.230 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:20:41.230 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:20:41.230 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:20:41.230 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 40314 2024-08-19T14:20:41.230 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.230 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:20:41.230 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.230 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:20:41.230 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.993308+0000", 2024-08-19T14:20:41.230 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:20:41.230 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.230 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.231 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:20:41.231 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.993308+0000", 2024-08-19T14:20:41.231 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:20:41.231 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.231 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.231 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:20:41.231 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.993309+0000", 2024-08-19T14:20:41.231 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1200000000000001e-06 2024-08-19T14:20:41.231 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.231 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.231 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:20:41.231 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.993470+0000", 2024-08-19T14:20:41.232 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00016027899999999999 2024-08-19T14:20:41.232 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.232 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.232 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:20:41.232 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.993482+0000", 2024-08-19T14:20:41.232 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2276000000000001e-05 2024-08-19T14:20:41.232 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.232 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.232 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:20:41.232 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.993487+0000", 2024-08-19T14:20:41.232 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.4199999999999998e-06 2024-08-19T14:20:41.232 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.232 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.233 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:20:41.233 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.993536+0000", 2024-08-19T14:20:41.233 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.9051999999999997e-05 2024-08-19T14:20:41.233 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.233 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.233 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:20:41.233 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.993628+0000", 2024-08-19T14:20:41.233 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.1546999999999996e-05 2024-08-19T14:20:41.233 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.233 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.233 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,8", 2024-08-19T14:20:41.233 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.993732+0000", 2024-08-19T14:20:41.233 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000104467 2024-08-19T14:20:41.233 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.234 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.234 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:20:41.234 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.997603+0000", 2024-08-19T14:20:41.234 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0038706529999999999 2024-08-19T14:20:41.234 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.234 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.234 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:20:41.234 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.997654+0000", 2024-08-19T14:20:41.234 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0628999999999999e-05 2024-08-19T14:20:41.234 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.234 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.234 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:20:41.234 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.198613+0000", 2024-08-19T14:20:41.234 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.20095934500000001 2024-08-19T14:20:41.235 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.235 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.235 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:20:41.235 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.198666+0000", 2024-08-19T14:20:41.235 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.3340000000000001e-05 2024-08-19T14:20:41.235 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.235 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.235 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:20:41.235 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.198709+0000", 2024-08-19T14:20:41.235 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2490000000000001e-05 2024-08-19T14:20:41.235 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:20:41.235 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:20:41.235 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:20:41.235 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.236 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.236 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:40315 2.9 2:95c123d2:::benchmark_data_smithi045_46415_object40314:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-08-19T14:20:41.236 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:13:02.993698+0000", 2024-08-19T14:20:41.236 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 457.21827808099999, 2024-08-19T14:20:41.236 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.20320638299999999, 2024-08-19T14:20:41.236 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:20:41.236 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:20:41.236 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:20:41.236 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:20:41.236 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:20:41.236 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:20:41.236 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 40315 2024-08-19T14:20:41.236 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.237 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:20:41.237 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.237 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:20:41.237 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.993698+0000", 2024-08-19T14:20:41.237 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:20:41.237 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.237 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.237 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:20:41.238 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.993698+0000", 2024-08-19T14:20:41.238 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:20:41.238 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.238 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.238 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:20:41.238 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.993699+0000", 2024-08-19T14:20:41.238 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.71e-07 2024-08-19T14:20:41.238 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.238 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.238 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:20:41.239 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.993770+0000", 2024-08-19T14:20:41.239 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.1186000000000006e-05 2024-08-19T14:20:41.239 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.239 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.239 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:20:41.239 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.993782+0000", 2024-08-19T14:20:41.239 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1822e-05 2024-08-19T14:20:41.239 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.239 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.239 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:20:41.239 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.993789+0000", 2024-08-19T14:20:41.239 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.3640000000000001e-06 2024-08-19T14:20:41.239 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.240 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.240 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:20:41.240 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.993844+0000", 2024-08-19T14:20:41.240 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.4688999999999998e-05 2024-08-19T14:20:41.240 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.240 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.240 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:20:41.240 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.993997+0000", 2024-08-19T14:20:41.240 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000153709 2024-08-19T14:20:41.240 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.240 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.241 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,7", 2024-08-19T14:20:41.241 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.994134+0000", 2024-08-19T14:20:41.241 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013627600000000001 2024-08-19T14:20:41.241 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.241 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.241 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:20:41.241 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.995858+0000", 2024-08-19T14:20:41.241 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00172466 2024-08-19T14:20:41.241 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.241 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.241 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:20:41.241 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.996156+0000", 2024-08-19T14:20:41.241 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000297764 2024-08-19T14:20:41.241 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.241 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.242 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:20:41.242 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.196808+0000", 2024-08-19T14:20:41.242 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.20065148399999999 2024-08-19T14:20:41.242 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.242 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.242 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:20:41.242 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.196850+0000", 2024-08-19T14:20:41.242 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.1934000000000002e-05 2024-08-19T14:20:41.242 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.242 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.242 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:20:41.242 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.196904+0000", 2024-08-19T14:20:41.242 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.4623999999999997e-05 2024-08-19T14:20:41.243 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:20:41.243 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:20:41.243 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:20:41.243 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.243 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.243 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:40316 2.6 2:6b76d263:::benchmark_data_smithi045_46415_object40315:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-08-19T14:20:41.243 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:13:02.994308+0000", 2024-08-19T14:20:41.243 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 457.21766747200002, 2024-08-19T14:20:41.243 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.20450618700000001, 2024-08-19T14:20:41.243 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:20:41.243 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:20:41.243 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:20:41.243 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:20:41.243 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:20:41.244 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:20:41.245 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 40316 2024-08-19T14:20:41.246 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.246 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:20:41.246 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.246 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:20:41.246 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.994308+0000", 2024-08-19T14:20:41.246 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:20:41.246 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.246 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.246 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:20:41.246 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.994308+0000", 2024-08-19T14:20:41.246 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:20:41.246 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.246 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.246 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:20:41.247 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.994309+0000", 2024-08-19T14:20:41.247 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.7899999999999997e-07 2024-08-19T14:20:41.247 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.247 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.247 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:20:41.247 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.994385+0000", 2024-08-19T14:20:41.247 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.6071999999999999e-05 2024-08-19T14:20:41.247 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.247 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.247 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:20:41.247 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.994398+0000", 2024-08-19T14:20:41.248 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2228e-05 2024-08-19T14:20:41.248 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.248 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.248 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:20:41.248 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.994405+0000", 2024-08-19T14:20:41.248 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.3189999999999997e-06 2024-08-19T14:20:41.248 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.248 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.248 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:20:41.248 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.995083+0000", 2024-08-19T14:20:41.248 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000678117 2024-08-19T14:20:41.248 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.248 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.249 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:20:41.249 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.995204+0000", 2024-08-19T14:20:41.249 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012103399999999999 2024-08-19T14:20:41.249 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.249 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.249 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,8", 2024-08-19T14:20:41.249 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.995352+0000", 2024-08-19T14:20:41.249 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00014803400000000001 2024-08-19T14:20:41.249 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.249 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.249 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:20:41.249 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.997636+0000", 2024-08-19T14:20:41.249 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0022837830000000002 2024-08-19T14:20:41.249 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.250 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.250 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:20:41.250 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.997659+0000", 2024-08-19T14:20:41.250 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.3473e-05 2024-08-19T14:20:41.250 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.250 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.250 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:20:41.250 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.198750+0000", 2024-08-19T14:20:41.250 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.201090301 2024-08-19T14:20:41.251 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.251 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.251 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:20:41.251 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.198780+0000", 2024-08-19T14:20:41.251 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.0688999999999997e-05 2024-08-19T14:20:41.251 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.251 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.251 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:20:41.251 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.198815+0000", 2024-08-19T14:20:41.251 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.4258000000000003e-05 2024-08-19T14:20:41.251 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:20:41.251 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:20:41.251 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:20:41.251 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.252 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.252 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:40320 2.6 2:6546d5b0:::benchmark_data_smithi045_46415_object40319:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-08-19T14:20:41.252 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:13:02.995448+0000", 2024-08-19T14:20:41.252 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 457.21652757599998, 2024-08-19T14:20:41.252 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.20356038500000001, 2024-08-19T14:20:41.252 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:20:41.252 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:20:41.252 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:20:41.252 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:20:41.252 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:20:41.252 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:20:41.252 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 40320 2024-08-19T14:20:41.252 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.252 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:20:41.253 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.253 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:20:41.253 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.995448+0000", 2024-08-19T14:20:41.253 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:20:41.253 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.253 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.253 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:20:41.253 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.995448+0000", 2024-08-19T14:20:41.253 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:20:41.253 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.253 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.253 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:20:41.253 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.995449+0000", 2024-08-19T14:20:41.253 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.7000000000000004e-07 2024-08-19T14:20:41.253 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.254 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.254 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:20:41.254 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.995511+0000", 2024-08-19T14:20:41.254 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.2249000000000007e-05 2024-08-19T14:20:41.254 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.254 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.254 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:20:41.254 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.995520+0000", 2024-08-19T14:20:41.254 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.8799999999999997e-06 2024-08-19T14:20:41.254 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.254 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.254 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:20:41.254 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.995525+0000", 2024-08-19T14:20:41.254 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.6659999999999999e-06 2024-08-19T14:20:41.255 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.255 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.255 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:20:41.255 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.996261+0000", 2024-08-19T14:20:41.255 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000736351 2024-08-19T14:20:41.255 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.255 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.255 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:20:41.255 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.996382+0000", 2024-08-19T14:20:41.255 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000120983 2024-08-19T14:20:41.255 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.255 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.255 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,8", 2024-08-19T14:20:41.255 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.996517+0000", 2024-08-19T14:20:41.256 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013479900000000001 2024-08-19T14:20:41.256 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.256 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.256 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:20:41.256 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.997767+0000", 2024-08-19T14:20:41.256 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0012497739999999999 2024-08-19T14:20:41.256 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.256 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.256 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:20:41.256 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.999297+0000", 2024-08-19T14:20:41.256 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0015301150000000001 2024-08-19T14:20:41.256 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.256 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.256 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:20:41.256 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.198909+0000", 2024-08-19T14:20:41.257 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.199612074 2024-08-19T14:20:41.257 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.257 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.257 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:20:41.257 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.198968+0000", 2024-08-19T14:20:41.257 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.8801999999999997e-05 2024-08-19T14:20:41.257 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.257 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.257 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:20:41.257 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.199009+0000", 2024-08-19T14:20:41.257 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.0921999999999999e-05 2024-08-19T14:20:41.257 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:20:41.257 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:20:41.257 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:20:41.257 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.258 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.258 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:40322 2.9 2:99a1eedb:::benchmark_data_smithi045_46415_object40321:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-08-19T14:20:41.258 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:13:02.996325+0000", 2024-08-19T14:20:41.258 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 457.21565088, 2024-08-19T14:20:41.258 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.20255289300000001, 2024-08-19T14:20:41.258 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:20:41.258 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:20:41.258 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:20:41.258 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:20:41.258 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:20:41.258 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:20:41.258 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 40322 2024-08-19T14:20:41.258 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.258 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:20:41.259 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.259 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:20:41.259 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.996325+0000", 2024-08-19T14:20:41.259 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:20:41.259 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.259 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.259 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:20:41.259 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.996325+0000", 2024-08-19T14:20:41.259 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:20:41.259 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.259 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.259 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:20:41.259 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.996326+0000", 2024-08-19T14:20:41.259 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0470000000000001e-06 2024-08-19T14:20:41.260 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.260 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.260 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:20:41.260 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.996355+0000", 2024-08-19T14:20:41.260 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.9167999999999999e-05 2024-08-19T14:20:41.260 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.260 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.260 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:20:41.260 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.996366+0000", 2024-08-19T14:20:41.260 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0679999999999999e-05 2024-08-19T14:20:41.260 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.260 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.260 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:20:41.260 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.996373+0000", 2024-08-19T14:20:41.261 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.5229999999999998e-06 2024-08-19T14:20:41.261 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.261 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.261 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:20:41.261 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.996428+0000", 2024-08-19T14:20:41.261 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.4580000000000003e-05 2024-08-19T14:20:41.261 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.261 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.261 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:20:41.261 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.996560+0000", 2024-08-19T14:20:41.261 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013169399999999999 2024-08-19T14:20:41.261 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.261 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.261 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,7", 2024-08-19T14:20:41.262 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.996700+0000", 2024-08-19T14:20:41.262 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000139973 2024-08-19T14:20:41.262 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.262 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.262 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:20:41.262 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.998183+0000", 2024-08-19T14:20:41.262 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0014834609999999999 2024-08-19T14:20:41.262 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.262 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.262 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:20:41.262 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:02.998340+0000", 2024-08-19T14:20:41.262 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000156658 2024-08-19T14:20:41.262 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.263 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.263 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:20:41.263 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.198775+0000", 2024-08-19T14:20:41.263 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.20043515100000001 2024-08-19T14:20:41.263 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.263 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.263 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:20:41.263 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.198828+0000", 2024-08-19T14:20:41.263 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.2556999999999998e-05 2024-08-19T14:20:41.263 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.263 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.264 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:20:41.264 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:13:03.198878+0000", 2024-08-19T14:20:41.264 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0401000000000001e-05 2024-08-19T14:20:41.264 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:20:41.264 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:20:41.264 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:20:41.264 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.264 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.264 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:241301 2.4 2:26065fe2:::benchmark_data_smithi045_46415_object118020:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-08-19T14:20:41.264 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:16:40.774899+0000", 2024-08-19T14:20:41.264 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 239.43707730899999, 2024-08-19T14:20:41.264 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.29898516600000002, 2024-08-19T14:20:41.264 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:20:41.264 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:20:41.265 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:20:41.265 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:20:41.265 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:20:41.265 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:20:41.265 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 241301 2024-08-19T14:20:41.265 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.265 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:20:41.265 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.265 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:20:41.266 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.774899+0000", 2024-08-19T14:20:41.266 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:20:41.266 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.266 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.267 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:20:41.267 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.774899+0000", 2024-08-19T14:20:41.267 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:20:41.267 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.267 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.267 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:20:41.267 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.774900+0000", 2024-08-19T14:20:41.267 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6190000000000001e-06 2024-08-19T14:20:41.267 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.267 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.267 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:20:41.267 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.774907+0000", 2024-08-19T14:20:41.268 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.6540000000000002e-06 2024-08-19T14:20:41.268 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.268 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.268 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:20:41.268 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.774921+0000", 2024-08-19T14:20:41.268 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4355000000000001e-05 2024-08-19T14:20:41.268 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.268 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.268 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:20:41.268 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.774928+0000", 2024-08-19T14:20:41.268 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.2339999999999997e-06 2024-08-19T14:20:41.268 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.269 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.269 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:20:41.269 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.780109+0000", 2024-08-19T14:20:41.269 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0051809409999999997 2024-08-19T14:20:41.269 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.269 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.269 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:20:41.269 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.780240+0000", 2024-08-19T14:20:41.269 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000131006 2024-08-19T14:20:41.269 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.269 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.269 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,2,8", 2024-08-19T14:20:41.270 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.780378+0000", 2024-08-19T14:20:41.270 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013800999999999999 2024-08-19T14:20:41.270 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.270 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.270 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:20:41.270 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.794340+0000", 2024-08-19T14:20:41.270 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.013961559 2024-08-19T14:20:41.270 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.270 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.270 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:20:41.271 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.802616+0000", 2024-08-19T14:20:41.271 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0082755490000000001 2024-08-19T14:20:41.271 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.271 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.271 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:20:41.271 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.809365+0000", 2024-08-19T14:20:41.271 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0067497039999999996 2024-08-19T14:20:41.271 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.271 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.271 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:20:41.272 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.073684+0000", 2024-08-19T14:20:41.272 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.26431834199999998 2024-08-19T14:20:41.272 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.272 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.272 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:20:41.272 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.073749+0000", 2024-08-19T14:20:41.272 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.5343000000000001e-05 2024-08-19T14:20:41.272 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.272 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.272 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:20:41.272 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.073884+0000", 2024-08-19T14:20:41.273 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013485 2024-08-19T14:20:41.273 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:20:41.273 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:20:41.273 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:20:41.273 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.273 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.273 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:241310 2.4 2:2df264d1:::benchmark_data_smithi045_46415_object118029:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-08-19T14:20:41.273 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:16:40.777745+0000", 2024-08-19T14:20:41.273 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 239.43423082000001, 2024-08-19T14:20:41.273 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.34201599500000002, 2024-08-19T14:20:41.273 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:20:41.273 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:20:41.274 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:20:41.274 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:20:41.274 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:20:41.274 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:20:41.274 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 241310 2024-08-19T14:20:41.274 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.274 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:20:41.274 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.274 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:20:41.274 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.777745+0000", 2024-08-19T14:20:41.275 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:20:41.275 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.275 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.275 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:20:41.275 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.777745+0000", 2024-08-19T14:20:41.275 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:20:41.275 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.275 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.275 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:20:41.275 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.777746+0000", 2024-08-19T14:20:41.275 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.5999999999999991e-07 2024-08-19T14:20:41.276 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.276 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.276 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:20:41.276 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.777751+0000", 2024-08-19T14:20:41.276 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.6369999999999998e-06 2024-08-19T14:20:41.276 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.276 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.276 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:20:41.277 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.777763+0000", 2024-08-19T14:20:41.277 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2279000000000001e-05 2024-08-19T14:20:41.277 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.277 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.277 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:20:41.277 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.777771+0000", 2024-08-19T14:20:41.277 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.6109999999999996e-06 2024-08-19T14:20:41.277 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.277 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.277 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:20:41.277 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.788985+0000", 2024-08-19T14:20:41.278 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.011214636 2024-08-19T14:20:41.278 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.278 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.278 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:20:41.278 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.789090+0000", 2024-08-19T14:20:41.278 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000104583 2024-08-19T14:20:41.278 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.278 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.278 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,2,8", 2024-08-19T14:20:41.278 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.789223+0000", 2024-08-19T14:20:41.278 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013313299999999999 2024-08-19T14:20:41.278 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.279 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.279 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:20:41.279 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.802660+0000", 2024-08-19T14:20:41.279 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.013437065999999999 2024-08-19T14:20:41.279 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.279 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.279 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:20:41.279 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.808077+0000", 2024-08-19T14:20:41.279 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0054167759999999999 2024-08-19T14:20:41.279 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.279 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.280 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:20:41.280 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.812929+0000", 2024-08-19T14:20:41.280 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0048518399999999996 2024-08-19T14:20:41.280 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.280 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.280 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:20:41.280 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.119563+0000", 2024-08-19T14:20:41.280 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.30663414300000003 2024-08-19T14:20:41.280 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.280 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.280 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:20:41.280 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.119627+0000", 2024-08-19T14:20:41.280 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.4622999999999994e-05 2024-08-19T14:20:41.281 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.281 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.281 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:20:41.281 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.119761+0000", 2024-08-19T14:20:41.281 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013370799999999999 2024-08-19T14:20:41.281 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:20:41.281 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:20:41.281 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:20:41.281 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.281 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.281 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:241313 2.4 2:22ac58a2:::benchmark_data_smithi045_46415_object118032:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-08-19T14:20:41.281 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:16:40.785407+0000", 2024-08-19T14:20:41.281 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 239.42656897699999, 2024-08-19T14:20:41.281 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.33459377200000001, 2024-08-19T14:20:41.282 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:20:41.282 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:20:41.282 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:20:41.282 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:20:41.282 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:20:41.282 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:20:41.282 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 241313 2024-08-19T14:20:41.282 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.282 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:20:41.282 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.282 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:20:41.283 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.785407+0000", 2024-08-19T14:20:41.283 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:20:41.283 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.283 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.283 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:20:41.283 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.785407+0000", 2024-08-19T14:20:41.283 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:20:41.283 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.283 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.283 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:20:41.283 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.785409+0000", 2024-08-19T14:20:41.283 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6109999999999999e-06 2024-08-19T14:20:41.283 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.284 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.284 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:20:41.284 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.785414+0000", 2024-08-19T14:20:41.284 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1919999999999996e-06 2024-08-19T14:20:41.284 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.284 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.284 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:20:41.284 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.785424+0000", 2024-08-19T14:20:41.284 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0719e-05 2024-08-19T14:20:41.284 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.284 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.284 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:20:41.285 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.785430+0000", 2024-08-19T14:20:41.285 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.8019999999999997e-06 2024-08-19T14:20:41.285 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.285 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.285 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:20:41.285 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.794371+0000", 2024-08-19T14:20:41.285 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0089406929999999996 2024-08-19T14:20:41.286 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.286 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.286 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:20:41.286 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.794483+0000", 2024-08-19T14:20:41.286 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000111539 2024-08-19T14:20:41.287 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.287 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.287 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,2,8", 2024-08-19T14:20:41.287 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.794630+0000", 2024-08-19T14:20:41.287 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00014782700000000001 2024-08-19T14:20:41.287 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.287 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.287 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:20:41.287 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.802700+0000", 2024-08-19T14:20:41.287 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0080701290000000005 2024-08-19T14:20:41.287 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.287 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.287 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:20:41.288 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.808117+0000", 2024-08-19T14:20:41.288 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0054165469999999999 2024-08-19T14:20:41.288 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.288 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.288 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:20:41.288 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.812933+0000", 2024-08-19T14:20:41.288 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0048158460000000004 2024-08-19T14:20:41.288 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.288 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.288 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:20:41.288 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.119824+0000", 2024-08-19T14:20:41.288 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.30689145899999998 2024-08-19T14:20:41.288 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.288 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.289 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:20:41.289 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.119884+0000", 2024-08-19T14:20:41.289 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.9642000000000003e-05 2024-08-19T14:20:41.289 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.289 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.289 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:20:41.289 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.120001+0000", 2024-08-19T14:20:41.289 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000116766 2024-08-19T14:20:41.289 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:20:41.289 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:20:41.289 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:20:41.289 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.290 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.290 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:241315 2.4 2:24bb1245:::benchmark_data_smithi045_46415_object118034:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-08-19T14:20:41.290 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:16:40.789772+0000", 2024-08-19T14:20:41.290 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 239.42220353100001, 2024-08-19T14:20:41.290 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.33090881500000002, 2024-08-19T14:20:41.290 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:20:41.290 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:20:41.290 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:20:41.290 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:20:41.290 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:20:41.290 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:20:41.290 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 241315 2024-08-19T14:20:41.291 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.291 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:20:41.291 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.291 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:20:41.291 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.789772+0000", 2024-08-19T14:20:41.291 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:20:41.291 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.291 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.291 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:20:41.291 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.789772+0000", 2024-08-19T14:20:41.291 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:20:41.291 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.291 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.291 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:20:41.292 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.789774+0000", 2024-08-19T14:20:41.292 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.483e-06 2024-08-19T14:20:41.292 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.292 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.292 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:20:41.292 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.789780+0000", 2024-08-19T14:20:41.292 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.9390000000000002e-06 2024-08-19T14:20:41.292 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.292 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.292 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:20:41.292 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.789796+0000", 2024-08-19T14:20:41.292 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5889999999999999e-05 2024-08-19T14:20:41.292 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.292 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.293 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:20:41.293 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.789803+0000", 2024-08-19T14:20:41.293 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.3900000000000004e-06 2024-08-19T14:20:41.293 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.293 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.293 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:20:41.293 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.802735+0000", 2024-08-19T14:20:41.293 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.012932332 2024-08-19T14:20:41.293 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.293 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.293 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:20:41.293 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.802891+0000", 2024-08-19T14:20:41.293 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000155301 2024-08-19T14:20:41.294 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.294 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.294 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,2,8", 2024-08-19T14:20:41.294 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.803031+0000", 2024-08-19T14:20:41.294 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000140414 2024-08-19T14:20:41.294 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.294 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.294 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:20:41.294 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.809352+0000", 2024-08-19T14:20:41.294 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0063210699999999998 2024-08-19T14:20:41.294 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.294 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.294 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:20:41.295 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.812937+0000", 2024-08-19T14:20:41.295 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0035844050000000001 2024-08-19T14:20:41.295 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.295 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.295 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:20:41.295 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.812965+0000", 2024-08-19T14:20:41.295 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.8099999999999999e-05 2024-08-19T14:20:41.295 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.295 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.295 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:20:41.295 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.120323+0000", 2024-08-19T14:20:41.295 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.30735870700000001 2024-08-19T14:20:41.295 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.296 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.296 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:20:41.296 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.120566+0000", 2024-08-19T14:20:41.296 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00024232400000000001 2024-08-19T14:20:41.296 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.296 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.296 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:20:41.296 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.120681+0000", 2024-08-19T14:20:41.296 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00011546 2024-08-19T14:20:41.296 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:20:41.296 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:20:41.296 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:20:41.296 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.296 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.297 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:241316 2.4 2:2a938701:::benchmark_data_smithi045_46415_object118035:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-08-19T14:20:41.297 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:16:40.789816+0000", 2024-08-19T14:20:41.297 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 239.422159459, 2024-08-19T14:20:41.297 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.33160296900000003, 2024-08-19T14:20:41.297 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:20:41.297 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:20:41.297 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:20:41.297 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:20:41.297 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:20:41.297 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:20:41.297 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 241316 2024-08-19T14:20:41.297 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.297 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:20:41.297 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.298 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:20:41.298 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.789816+0000", 2024-08-19T14:20:41.298 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:20:41.298 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.298 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.298 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:20:41.298 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.789816+0000", 2024-08-19T14:20:41.298 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:20:41.298 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.298 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.298 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:20:41.298 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.789817+0000", 2024-08-19T14:20:41.298 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.0699999999999998e-07 2024-08-19T14:20:41.299 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.299 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.299 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:20:41.299 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.789819+0000", 2024-08-19T14:20:41.299 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.4380000000000002e-06 2024-08-19T14:20:41.299 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.299 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.299 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:20:41.299 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.789826+0000", 2024-08-19T14:20:41.299 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.4060000000000002e-06 2024-08-19T14:20:41.299 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.299 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.299 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:20:41.300 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.789831+0000", 2024-08-19T14:20:41.300 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.8960000000000002e-06 2024-08-19T14:20:41.300 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.300 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.300 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:20:41.300 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.812990+0000", 2024-08-19T14:20:41.300 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.023159863999999999 2024-08-19T14:20:41.300 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.300 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.300 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:20:41.300 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.813089+0000", 2024-08-19T14:20:41.300 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.8478000000000001e-05 2024-08-19T14:20:41.300 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.300 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.301 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,2,8", 2024-08-19T14:20:41.301 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.813175+0000", 2024-08-19T14:20:41.301 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.5834000000000005e-05 2024-08-19T14:20:41.301 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.301 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.301 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:20:41.301 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.818905+0000", 2024-08-19T14:20:41.301 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0057298870000000003 2024-08-19T14:20:41.301 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.301 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.301 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:20:41.301 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.820238+0000", 2024-08-19T14:20:41.301 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.001333603 2024-08-19T14:20:41.301 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.302 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.302 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:20:41.302 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.831561+0000", 2024-08-19T14:20:41.302 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.011322334 2024-08-19T14:20:41.302 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.302 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.302 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:20:41.302 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.120731+0000", 2024-08-19T14:20:41.302 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.289170123 2024-08-19T14:20:41.302 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.302 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.302 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:20:41.302 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.121237+0000", 2024-08-19T14:20:41.302 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00050635100000000004 2024-08-19T14:20:41.303 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.303 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.303 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:20:41.303 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.121419+0000", 2024-08-19T14:20:41.303 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00018234800000000001 2024-08-19T14:20:41.303 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:20:41.303 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:20:41.303 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:20:41.303 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.303 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.303 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:241344 2.4 2:2b291422:::benchmark_data_smithi045_46415_object118063:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-08-19T14:20:41.303 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:16:40.823839+0000", 2024-08-19T14:20:41.303 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 239.38813664200001, 2024-08-19T14:20:41.303 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.29983483399999999, 2024-08-19T14:20:41.304 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:20:41.304 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:20:41.304 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:20:41.304 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:20:41.304 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:20:41.304 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:20:41.304 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 241344 2024-08-19T14:20:41.304 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.304 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:20:41.304 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.304 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:20:41.304 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.823839+0000", 2024-08-19T14:20:41.304 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:20:41.304 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.305 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.305 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:20:41.305 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.823839+0000", 2024-08-19T14:20:41.305 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:20:41.305 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.305 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.305 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:20:41.305 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.823841+0000", 2024-08-19T14:20:41.305 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.607e-06 2024-08-19T14:20:41.305 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.305 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.305 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:20:41.305 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.823847+0000", 2024-08-19T14:20:41.305 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.7620000000000001e-06 2024-08-19T14:20:41.306 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.306 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.306 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:20:41.306 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.823865+0000", 2024-08-19T14:20:41.306 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.8134999999999998e-05 2024-08-19T14:20:41.306 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.306 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.306 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:20:41.306 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.823872+0000", 2024-08-19T14:20:41.306 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.7039999999999994e-06 2024-08-19T14:20:41.306 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.306 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.306 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:20:41.306 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.824022+0000", 2024-08-19T14:20:41.307 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000149405 2024-08-19T14:20:41.307 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.307 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.307 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:20:41.307 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.824180+0000", 2024-08-19T14:20:41.307 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00015828199999999999 2024-08-19T14:20:41.307 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.307 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.307 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,2,8", 2024-08-19T14:20:41.307 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.824332+0000", 2024-08-19T14:20:41.307 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00015196799999999999 2024-08-19T14:20:41.307 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.307 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.307 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:20:41.308 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.831601+0000", 2024-08-19T14:20:41.308 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0072684150000000003 2024-08-19T14:20:41.308 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.308 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.308 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:20:41.308 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.832795+0000", 2024-08-19T14:20:41.308 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0011945739999999999 2024-08-19T14:20:41.308 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.308 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.308 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:20:41.308 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.843115+0000", 2024-08-19T14:20:41.308 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.010319488 2024-08-19T14:20:41.308 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.309 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.309 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:20:41.309 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.121468+0000", 2024-08-19T14:20:41.309 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.27835382199999997 2024-08-19T14:20:41.309 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.309 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.309 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:20:41.309 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.123536+0000", 2024-08-19T14:20:41.309 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.002067892 2024-08-19T14:20:41.309 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.309 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.309 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:20:41.309 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.123674+0000", 2024-08-19T14:20:41.309 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013778 2024-08-19T14:20:41.310 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:20:41.310 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:20:41.310 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:20:41.310 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.310 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.310 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:241352 2.4 2:258e31fc:::benchmark_data_smithi045_46415_object118071:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-08-19T14:20:41.310 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:16:40.833613+0000", 2024-08-19T14:20:41.310 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 239.37836273299999, 2024-08-19T14:20:41.310 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.290304479, 2024-08-19T14:20:41.310 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:20:41.310 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:20:41.310 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:20:41.310 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:20:41.311 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:20:41.311 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:20:41.311 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 241352 2024-08-19T14:20:41.311 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.311 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:20:41.311 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.311 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:20:41.311 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.833613+0000", 2024-08-19T14:20:41.311 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:20:41.311 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.311 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.311 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:20:41.311 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.833613+0000", 2024-08-19T14:20:41.311 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:20:41.312 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.312 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.312 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:20:41.312 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.833615+0000", 2024-08-19T14:20:41.312 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4839999999999999e-06 2024-08-19T14:20:41.312 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.312 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.312 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:20:41.312 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.833620+0000", 2024-08-19T14:20:41.312 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.5740000000000003e-06 2024-08-19T14:20:41.312 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.312 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.312 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:20:41.312 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.833635+0000", 2024-08-19T14:20:41.313 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4893e-05 2024-08-19T14:20:41.313 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.313 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.313 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:20:41.313 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.833642+0000", 2024-08-19T14:20:41.313 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.2620000000000001e-06 2024-08-19T14:20:41.313 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.313 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.313 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:20:41.313 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.836631+0000", 2024-08-19T14:20:41.313 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0029890400000000001 2024-08-19T14:20:41.313 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.313 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.313 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:20:41.314 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.836733+0000", 2024-08-19T14:20:41.314 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000101532 2024-08-19T14:20:41.314 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.314 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.314 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,2,8", 2024-08-19T14:20:41.314 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.836833+0000", 2024-08-19T14:20:41.314 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.9652000000000001e-05 2024-08-19T14:20:41.314 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.314 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.314 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:20:41.314 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.841532+0000", 2024-08-19T14:20:41.314 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0046991769999999997 2024-08-19T14:20:41.315 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.315 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.315 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:20:41.315 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.843144+0000", 2024-08-19T14:20:41.315 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0016120139999999999 2024-08-19T14:20:41.315 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.315 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.315 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:20:41.315 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.843187+0000", 2024-08-19T14:20:41.315 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.3198000000000002e-05 2024-08-19T14:20:41.315 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.315 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.315 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:20:41.315 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.123725+0000", 2024-08-19T14:20:41.316 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.28053793599999999 2024-08-19T14:20:41.316 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.316 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.316 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:20:41.316 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.123784+0000", 2024-08-19T14:20:41.316 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.8931000000000002e-05 2024-08-19T14:20:41.316 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.316 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.316 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:20:41.316 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.123918+0000", 2024-08-19T14:20:41.316 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000133786 2024-08-19T14:20:41.316 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:20:41.316 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:20:41.316 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:20:41.317 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.317 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.317 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:241376 2.4 2:259a9829:::benchmark_data_smithi045_46415_object118095:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-08-19T14:20:41.317 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:16:40.865722+0000", 2024-08-19T14:20:41.317 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 239.346253784, 2024-08-19T14:20:41.317 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.25843650499999998, 2024-08-19T14:20:41.317 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:20:41.317 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:20:41.317 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:20:41.317 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:20:41.317 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:20:41.317 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:20:41.317 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 241376 2024-08-19T14:20:41.317 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.317 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:20:41.318 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.318 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:20:41.318 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.865722+0000", 2024-08-19T14:20:41.318 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:20:41.318 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.318 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.318 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:20:41.318 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.865722+0000", 2024-08-19T14:20:41.318 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:20:41.318 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.318 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.318 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:20:41.318 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.865724+0000", 2024-08-19T14:20:41.319 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6619999999999999e-06 2024-08-19T14:20:41.319 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.319 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.319 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:20:41.319 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.865729+0000", 2024-08-19T14:20:41.319 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7040000000000002e-06 2024-08-19T14:20:41.319 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.319 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.319 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:20:41.319 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.865739+0000", 2024-08-19T14:20:41.319 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0541e-05 2024-08-19T14:20:41.319 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.319 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.319 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:20:41.319 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.865745+0000", 2024-08-19T14:20:41.320 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.6609999999999997e-06 2024-08-19T14:20:41.320 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.320 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.320 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:20:41.320 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.865926+0000", 2024-08-19T14:20:41.320 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00018153000000000001 2024-08-19T14:20:41.320 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.320 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.320 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:20:41.320 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.866094+0000", 2024-08-19T14:20:41.320 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000167844 2024-08-19T14:20:41.320 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.320 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.320 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,2,8", 2024-08-19T14:20:41.321 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.866236+0000", 2024-08-19T14:20:41.321 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00014171600000000001 2024-08-19T14:20:41.321 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.321 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.321 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:20:41.321 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.871503+0000", 2024-08-19T14:20:41.321 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0052669960000000004 2024-08-19T14:20:41.321 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.321 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.321 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:20:41.321 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.871600+0000", 2024-08-19T14:20:41.321 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.7027000000000005e-05 2024-08-19T14:20:41.321 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.321 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.322 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:20:41.322 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.872043+0000", 2024-08-19T14:20:41.322 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00044304300000000001 2024-08-19T14:20:41.322 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.322 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.322 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:20:41.322 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.123962+0000", 2024-08-19T14:20:41.322 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.25191884199999998 2024-08-19T14:20:41.322 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.322 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.322 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:20:41.322 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.124022+0000", 2024-08-19T14:20:41.322 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.0331999999999998e-05 2024-08-19T14:20:41.323 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.323 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.323 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:20:41.323 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.124159+0000", 2024-08-19T14:20:41.323 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013660699999999999 2024-08-19T14:20:41.323 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:20:41.323 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:20:41.323 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:20:41.323 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.323 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.323 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25057.0:47840 3.14 3:2da72587:::benchmark_data_smithi045_59973_object47839:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e234)", 2024-08-19T14:20:41.323 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:17:18.780442+0000", 2024-08-19T14:20:41.323 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 201.43153378599999, 2024-08-19T14:20:41.323 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.98449229999999999, 2024-08-19T14:20:41.324 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:20:41.324 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:20:41.324 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:20:41.324 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:20:41.324 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25057", 2024-08-19T14:20:41.324 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:20:41.324 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 47840 2024-08-19T14:20:41.324 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.324 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:20:41.324 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.324 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:20:41.324 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.780442+0000", 2024-08-19T14:20:41.324 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:20:41.324 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.325 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.325 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:20:41.325 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.780442+0000", 2024-08-19T14:20:41.325 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:20:41.325 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.325 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.325 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:20:41.325 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.780444+0000", 2024-08-19T14:20:41.325 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.216e-06 2024-08-19T14:20:41.325 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.325 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.325 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:20:41.325 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.780527+0000", 2024-08-19T14:20:41.326 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.3096000000000005e-05 2024-08-19T14:20:41.326 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.326 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.326 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:20:41.326 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.780548+0000", 2024-08-19T14:20:41.326 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.0424999999999998e-05 2024-08-19T14:20:41.326 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.326 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.326 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:20:41.326 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.780559+0000", 2024-08-19T14:20:41.326 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1579999999999999e-05 2024-08-19T14:20:41.326 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.326 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.326 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:20:41.327 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.815099+0000", 2024-08-19T14:20:41.327 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.034539489999999999 2024-08-19T14:20:41.327 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.327 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.327 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:20:41.327 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.815109+0000", 2024-08-19T14:20:41.327 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0036e-05 2024-08-19T14:20:41.327 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.327 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.327 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:20:41.327 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.756630+0000", 2024-08-19T14:20:41.327 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.94152143899999996 2024-08-19T14:20:41.327 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.327 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.328 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:20:41.328 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.756793+0000", 2024-08-19T14:20:41.328 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00016217400000000001 2024-08-19T14:20:41.328 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.328 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.328 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 5,11", 2024-08-19T14:20:41.328 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.756983+0000", 2024-08-19T14:20:41.328 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00019000000000000001 2024-08-19T14:20:41.328 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.328 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.328 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:20:41.328 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.759245+0000", 2024-08-19T14:20:41.328 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0022623140000000001 2024-08-19T14:20:41.329 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.329 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.329 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:20:41.329 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.762237+0000", 2024-08-19T14:20:41.329 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0029923850000000002 2024-08-19T14:20:41.329 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.329 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.329 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:20:41.329 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.764795+0000", 2024-08-19T14:20:41.329 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0025578240000000002 2024-08-19T14:20:41.329 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.329 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.329 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:20:41.329 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.764889+0000", 2024-08-19T14:20:41.330 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.3927999999999999e-05 2024-08-19T14:20:41.330 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.330 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.330 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:20:41.330 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.764934+0000", 2024-08-19T14:20:41.330 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.5392999999999997e-05 2024-08-19T14:20:41.330 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:20:41.330 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:20:41.330 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:20:41.330 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.330 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.330 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25057.0:188768 3.24 3:25c22bc2:::benchmark_data_smithi045_59973_object57589:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-08-19T14:20:41.330 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:19:02.299148+0000", 2024-08-19T14:20:41.330 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 97.912827699999994, 2024-08-19T14:20:41.331 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.92656360999999998, 2024-08-19T14:20:41.331 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:20:41.331 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:20:41.331 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:20:41.331 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:20:41.331 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25057", 2024-08-19T14:20:41.331 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:20:41.331 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 188768 2024-08-19T14:20:41.331 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.331 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:20:41.331 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.331 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:20:41.331 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.299148+0000", 2024-08-19T14:20:41.331 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:20:41.332 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.332 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.332 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:20:41.332 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.299148+0000", 2024-08-19T14:20:41.332 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:20:41.332 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.332 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.332 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:20:41.332 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.299150+0000", 2024-08-19T14:20:41.332 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.869e-06 2024-08-19T14:20:41.332 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.332 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.332 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:20:41.332 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.299157+0000", 2024-08-19T14:20:41.333 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.2699999999999999e-06 2024-08-19T14:20:41.333 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.333 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.333 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:20:41.333 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.299173+0000", 2024-08-19T14:20:41.333 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6099e-05 2024-08-19T14:20:41.333 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.333 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.333 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:20:41.333 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.299184+0000", 2024-08-19T14:20:41.333 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0523000000000001e-05 2024-08-19T14:20:41.333 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.333 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.333 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:20:41.333 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.299253+0000", 2024-08-19T14:20:41.334 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.8752999999999997e-05 2024-08-19T14:20:41.334 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.334 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.334 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:20:41.334 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.299265+0000", 2024-08-19T14:20:41.334 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2295e-05 2024-08-19T14:20:41.334 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.334 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.334 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:20:41.334 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:03.218622+0000", 2024-08-19T14:20:41.334 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.91935708900000002 2024-08-19T14:20:41.334 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.334 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.335 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:20:41.335 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:03.218724+0000", 2024-08-19T14:20:41.335 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00010233700000000001 2024-08-19T14:20:41.335 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.335 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.335 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 5,11", 2024-08-19T14:20:41.335 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:03.218822+0000", 2024-08-19T14:20:41.335 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.7435999999999993e-05 2024-08-19T14:20:41.335 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.335 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.335 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:20:41.335 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:03.220396+0000", 2024-08-19T14:20:41.335 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0015744450000000001 2024-08-19T14:20:41.336 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.336 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.336 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:20:41.336 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:03.223542+0000", 2024-08-19T14:20:41.336 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0031455879999999999 2024-08-19T14:20:41.336 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.336 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.336 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:20:41.336 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:03.225592+0000", 2024-08-19T14:20:41.336 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0020502789999999999 2024-08-19T14:20:41.336 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.336 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.336 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:20:41.336 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:03.225650+0000", 2024-08-19T14:20:41.337 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.7472000000000001e-05 2024-08-19T14:20:41.337 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.337 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.337 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:20:41.337 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:03.225712+0000", 2024-08-19T14:20:41.337 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.2155000000000004e-05 2024-08-19T14:20:41.337 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:20:41.337 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:20:41.337 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:20:41.337 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.337 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.337 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25057.0:212408 3.2a 3:55a220e0:::benchmark_data_smithi045_59973_object81229:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e305)", 2024-08-19T14:20:41.337 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:19:29.899756+0000", 2024-08-19T14:20:41.337 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 70.312220104000005, 2024-08-19T14:20:41.338 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.22791488500000001, 2024-08-19T14:20:41.338 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:20:41.338 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:20:41.338 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:20:41.338 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:20:41.338 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25057", 2024-08-19T14:20:41.338 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:20:41.338 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 212408 2024-08-19T14:20:41.338 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.338 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:20:41.338 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.338 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:20:41.338 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:29.899756+0000", 2024-08-19T14:20:41.338 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:20:41.339 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.339 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.339 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:20:41.339 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:29.899756+0000", 2024-08-19T14:20:41.339 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:20:41.339 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.339 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.339 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:20:41.339 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:29.899757+0000", 2024-08-19T14:20:41.339 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6509999999999999e-06 2024-08-19T14:20:41.339 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.339 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.339 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:20:41.339 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:29.899763+0000", 2024-08-19T14:20:41.340 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.9109999999999998e-06 2024-08-19T14:20:41.340 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.340 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.340 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:20:41.340 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:29.899776+0000", 2024-08-19T14:20:41.340 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3038e-05 2024-08-19T14:20:41.340 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.340 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.340 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:20:41.340 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:29.899785+0000", 2024-08-19T14:20:41.340 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.5040000000000002e-06 2024-08-19T14:20:41.340 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.340 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.340 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:20:41.341 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:30.107412+0000", 2024-08-19T14:20:41.341 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.20762661900000001 2024-08-19T14:20:41.341 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.341 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.341 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:20:41.341 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:30.108207+0000", 2024-08-19T14:20:41.341 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00079542899999999995 2024-08-19T14:20:41.341 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.341 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.341 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,5,7", 2024-08-19T14:20:41.341 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:30.108349+0000", 2024-08-19T14:20:41.341 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00014198800000000001 2024-08-19T14:20:41.341 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.341 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.342 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:20:41.342 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:30.121649+0000", 2024-08-19T14:20:41.342 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.013299989 2024-08-19T14:20:41.342 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.342 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.342 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:20:41.342 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:30.121692+0000", 2024-08-19T14:20:41.342 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2718e-05 2024-08-19T14:20:41.342 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.342 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.342 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:20:41.342 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:30.121958+0000", 2024-08-19T14:20:41.342 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00026637699999999999 2024-08-19T14:20:41.343 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.343 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.343 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:20:41.343 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:30.126991+0000", 2024-08-19T14:20:41.343 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0050324819999999996 2024-08-19T14:20:41.343 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.343 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.343 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:20:41.343 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:30.127092+0000", 2024-08-19T14:20:41.343 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000101635 2024-08-19T14:20:41.343 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.343 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.343 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:20:41.343 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:30.127671+0000", 2024-08-19T14:20:41.344 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00057854399999999995 2024-08-19T14:20:41.344 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:20:41.344 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:20:41.344 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:20:41.344 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.344 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.344 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "MOSDPGPushReply(3.2a 305/299 [PushReplyOp(3:54cf2ba1:::benchmark_data_smithi045_59973_object107528:head)])", 2024-08-19T14:20:41.344 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:19:29.912075+0000", 2024-08-19T14:20:41.344 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 70.299901061, 2024-08-19T14:20:41.344 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.21486738699999999, 2024-08-19T14:20:41.344 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:20:41.344 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:20:41.344 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:20:41.344 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:20:41.345 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.345 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:20:41.345 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:29.912075+0000", 2024-08-19T14:20:41.345 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:20:41.345 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.345 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.345 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:20:41.345 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:29.912075+0000", 2024-08-19T14:20:41.345 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:20:41.345 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.345 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.345 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:20:41.345 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:29.912075+0000", 2024-08-19T14:20:41.345 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.9400000000000001e-07 2024-08-19T14:20:41.345 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.346 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.346 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:20:41.346 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:29.912079+0000", 2024-08-19T14:20:41.346 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.8299999999999998e-06 2024-08-19T14:20:41.346 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.346 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.346 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:20:41.346 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:29.912087+0000", 2024-08-19T14:20:41.346 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.5299999999999999e-06 2024-08-19T14:20:41.346 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.346 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.346 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:20:41.346 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:29.912093+0000", 2024-08-19T14:20:41.347 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.0159999999999997e-06 2024-08-19T14:20:41.347 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.347 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.347 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:20:41.347 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:30.115839+0000", 2024-08-19T14:20:41.347 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.20374692999999999 2024-08-19T14:20:41.347 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.347 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.347 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:20:41.347 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:30.126942+0000", 2024-08-19T14:20:41.347 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.011102786999999999 2024-08-19T14:20:41.347 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:20:41.347 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:20:41.347 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:20:41.348 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.348 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.348 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "MOSDPGPushReply(3.2a 305/299 [PushReplyOp(3:54cf2ba1:::benchmark_data_smithi045_59973_object107528:head)])", 2024-08-19T14:20:41.348 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:19:29.912114+0000", 2024-08-19T14:20:41.348 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 70.299861586999995, 2024-08-19T14:20:41.348 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.21602574799999999, 2024-08-19T14:20:41.348 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:20:41.348 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:20:41.348 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:20:41.348 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:20:41.348 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.348 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:20:41.348 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:29.912114+0000", 2024-08-19T14:20:41.348 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:20:41.349 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.349 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.349 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:20:41.349 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:29.912114+0000", 2024-08-19T14:20:41.349 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:20:41.349 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.349 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.349 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:20:41.349 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:29.912115+0000", 2024-08-19T14:20:41.349 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.8099999999999999e-07 2024-08-19T14:20:41.349 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.349 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.349 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:20:41.350 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:29.912117+0000", 2024-08-19T14:20:41.350 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.8590000000000001e-06 2024-08-19T14:20:41.350 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.350 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.350 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:20:41.350 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:29.912128+0000", 2024-08-19T14:20:41.350 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0369e-05 2024-08-19T14:20:41.350 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.350 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.350 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:20:41.350 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:29.912133+0000", 2024-08-19T14:20:41.350 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.2580000000000003e-06 2024-08-19T14:20:41.350 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.350 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.351 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:20:41.351 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:30.127777+0000", 2024-08-19T14:20:41.351 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.21564402799999999 2024-08-19T14:20:41.351 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.351 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.351 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:20:41.351 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:30.128140+0000", 2024-08-19T14:20:41.351 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00036295300000000001 2024-08-19T14:20:41.351 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:20:41.351 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:20:41.351 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:20:41.351 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.351 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.351 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "MOSDPGPushReply(3.2a 305/299 [PushReplyOp(3:54cf2ba1:::benchmark_data_smithi045_59973_object107528:head)])", 2024-08-19T14:20:41.352 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:19:29.918107+0000", 2024-08-19T14:20:41.352 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 70.293868978999996, 2024-08-19T14:20:41.352 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.21061088, 2024-08-19T14:20:41.352 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:20:41.352 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:20:41.352 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:20:41.352 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:20:41.352 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.352 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:20:41.352 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:29.918107+0000", 2024-08-19T14:20:41.352 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:20:41.352 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.352 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.353 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:20:41.353 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:29.918107+0000", 2024-08-19T14:20:41.353 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:20:41.353 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.353 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.353 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:20:41.353 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:29.918108+0000", 2024-08-19T14:20:41.353 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.0999999999999998e-07 2024-08-19T14:20:41.353 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.353 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.353 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:20:41.353 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:29.918112+0000", 2024-08-19T14:20:41.353 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.6759999999999998e-06 2024-08-19T14:20:41.353 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.354 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.354 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:20:41.354 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:29.918123+0000", 2024-08-19T14:20:41.354 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0587999999999999e-05 2024-08-19T14:20:41.354 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.354 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.354 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:20:41.354 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:29.918130+0000", 2024-08-19T14:20:41.354 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.4370000000000001e-06 2024-08-19T14:20:41.354 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.354 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.354 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:20:41.355 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:30.128168+0000", 2024-08-19T14:20:41.355 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.210038061 2024-08-19T14:20:41.355 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.355 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.355 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:20:41.355 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:30.128718+0000", 2024-08-19T14:20:41.355 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00054950799999999998 2024-08-19T14:20:41.355 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:20:41.355 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:20:41.355 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:20:41.355 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.355 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.355 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25057.0:216420 3.2a 3:56b46cf0:::benchmark_data_smithi045_59973_object85241:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e307)", 2024-08-19T14:20:41.355 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:19:33.992801+0000", 2024-08-19T14:20:41.356 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 66.219174527000007, 2024-08-19T14:20:41.356 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.203694439, 2024-08-19T14:20:41.356 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:20:41.356 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:20:41.356 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:20:41.356 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:20:41.356 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25057", 2024-08-19T14:20:41.356 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:20:41.356 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 216420 2024-08-19T14:20:41.356 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.356 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:20:41.356 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.356 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:20:41.356 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:33.992801+0000", 2024-08-19T14:20:41.357 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:20:41.357 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.357 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.357 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:20:41.357 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:33.992801+0000", 2024-08-19T14:20:41.357 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:20:41.357 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.357 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.357 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:20:41.357 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:33.992803+0000", 2024-08-19T14:20:41.357 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.271e-06 2024-08-19T14:20:41.357 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.357 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.357 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:20:41.358 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:33.992808+0000", 2024-08-19T14:20:41.358 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0150000000000003e-06 2024-08-19T14:20:41.358 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.358 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.358 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:20:41.358 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:34.165983+0000", 2024-08-19T14:20:41.358 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.173175053 2024-08-19T14:20:41.358 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.358 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.358 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:20:41.358 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:34.165994+0000", 2024-08-19T14:20:41.358 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1433999999999999e-05 2024-08-19T14:20:41.358 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.358 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.359 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:20:41.359 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:34.166454+0000", 2024-08-19T14:20:41.359 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000460178 2024-08-19T14:20:41.359 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.359 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.359 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:20:41.359 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:34.166592+0000", 2024-08-19T14:20:41.359 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000137723 2024-08-19T14:20:41.359 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.359 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.359 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,5,7", 2024-08-19T14:20:41.359 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:34.166722+0000", 2024-08-19T14:20:41.359 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012988499999999999 2024-08-19T14:20:41.359 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.360 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.360 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:20:41.360 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:34.193210+0000", 2024-08-19T14:20:41.360 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.026488385999999999 2024-08-19T14:20:41.360 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.360 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.360 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:20:41.360 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:34.193253+0000", 2024-08-19T14:20:41.360 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2543999999999998e-05 2024-08-19T14:20:41.360 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.360 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.360 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:20:41.360 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:34.193339+0000", 2024-08-19T14:20:41.360 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.6163999999999997e-05 2024-08-19T14:20:41.361 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.361 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.361 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:20:41.361 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:34.195209+0000", 2024-08-19T14:20:41.361 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0018700380000000001 2024-08-19T14:20:41.361 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.361 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.361 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:20:41.361 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:34.196017+0000", 2024-08-19T14:20:41.361 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00080749499999999998 2024-08-19T14:20:41.361 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:20:41.361 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:20:41.361 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:20:41.361 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:34.196496+0000", 2024-08-19T14:20:41.361 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00047925300000000001 2024-08-19T14:20:41.362 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:20:41.362 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:20:41.362 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:20:41.362 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:20:41.362 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:20:41.362 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:20:41.362 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:20:41.383 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:41.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:41.384 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:20:41.485 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:20:41.507 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:41.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:41.508 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:20:41.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:41 smithi045 ceph-mon[20143]: pgmap v763: 47 pgs: 47 active+clean; 755 MiB data, 5.2 GiB used, 799 GiB / 805 GiB avail; 99 MiB/s wr, 1.58k op/s 2024-08-19T14:20:41.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:41 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:20:41.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:41 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:20:41.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:41 smithi045 ceph-mon[20143]: osdmap e341: 12 total, 12 up, 9 in 2024-08-19T14:20:41.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:41 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:20:41.609 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:20:41.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:41 smithi031 ceph-mon[16336]: pgmap v763: 47 pgs: 47 active+clean; 755 MiB data, 5.2 GiB used, 799 GiB / 805 GiB avail; 99 MiB/s wr, 1.58k op/s 2024-08-19T14:20:41.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:41 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:20:41.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:41 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:20:41.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:41 smithi031 ceph-mon[16336]: osdmap e341: 12 total, 12 up, 9 in 2024-08-19T14:20:41.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:41 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:20:41.631 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:41.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:41.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:20:41.729 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:41 smithi032 ceph-mon[20004]: pgmap v763: 47 pgs: 47 active+clean; 755 MiB data, 5.2 GiB used, 799 GiB / 805 GiB avail; 99 MiB/s wr, 1.58k op/s 2024-08-19T14:20:41.729 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:41 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:20:41.729 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:41 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:20:41.730 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:41 smithi032 ceph-mon[20004]: osdmap e341: 12 total, 12 up, 9 in 2024-08-19T14:20:41.730 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:41 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:20:41.733 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:20:41.761 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:41.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:41.762 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:20:41.865 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:20:41.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:41.889 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:41.889 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:20:41.990 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:20:42.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:42.022 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:42.022 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:20:42.045 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 8 16 23481 23465 183.305 81.6875 0.00850657 0.00545021 2024-08-19T14:20:42.123 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:20:42.158 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:42.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:42.159 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:20:42.260 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:20:42.287 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:42.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:42.288 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:20:42.389 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:20:42.423 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:42.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:42.424 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:20:42.525 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:20:42.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:42.547 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:42.547 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:20:42.648 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:20:42.684 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:42.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:42.685 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:20:42.786 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:20:42.818 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:42.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:42.819 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:20:42.920 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:20:42.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:42.961 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:42.961 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:20:43.045 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 9 15 25863 25848 179.485 148.938 0.00398931 0.00556672 2024-08-19T14:20:43.061 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:20:43.089 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:43.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:43.090 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:20:43.190 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:20:43.223 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:43.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:43.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:20:43.268 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:20:43.324 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:20:43.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:43.352 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:43.352 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:20:43.453 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:20:43.484 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:43.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:43.485 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:20:43.586 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:20:43.607 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:43.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:43.608 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:20:43.709 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:20:43.738 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:43.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:43.739 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:20:43.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:43 smithi032 ceph-mon[20004]: pgmap v765: 47 pgs: 47 active+clean; 1.1 GiB data, 6.2 GiB used, 798 GiB / 805 GiB avail; 170 MiB/s wr, 2.73k op/s 2024-08-19T14:20:43.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:43 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:20:43.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:43 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:20:43.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:43 smithi045 ceph-mon[20143]: pgmap v765: 47 pgs: 47 active+clean; 1.1 GiB data, 6.2 GiB used, 798 GiB / 805 GiB avail; 170 MiB/s wr, 2.73k op/s 2024-08-19T14:20:43.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:43 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:20:43.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:43 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:20:43.840 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:20:43.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:43.863 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:43.863 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:20:43.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:43 smithi031 ceph-mon[16336]: pgmap v765: 47 pgs: 47 active+clean; 1.1 GiB data, 6.2 GiB used, 798 GiB / 805 GiB avail; 170 MiB/s wr, 2.73k op/s 2024-08-19T14:20:43.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:43 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:20:43.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:43 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:20:43.964 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:20:43.992 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:43.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:43.993 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:20:44.045 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 10 15 28530 28515 178.204 166.688 0.00515109 0.0056068 2024-08-19T14:20:44.094 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:20:44.124 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:44.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:44.125 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:20:44.226 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:20:44.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:44.258 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:44.258 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:20:44.360 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:20:44.386 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:44.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:44.388 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:20:44.489 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:20:44.518 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:44.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:44.519 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:20:44.621 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:20:44.650 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:44.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:44.651 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:20:44.752 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:20:44.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:44 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:20:44.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:44 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:20:44.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:44 smithi032 ceph-mon[20004]: osdmap e342: 12 total, 12 up, 9 in 2024-08-19T14:20:44.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:44.783 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:44.784 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:20:44.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:44 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:20:44.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:44 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:20:44.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:44 smithi045 ceph-mon[20143]: osdmap e342: 12 total, 12 up, 9 in 2024-08-19T14:20:44.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:44 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:20:44.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:44 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:20:44.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:44 smithi031 ceph-mon[16336]: osdmap e342: 12 total, 12 up, 9 in 2024-08-19T14:20:44.884 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:20:44.914 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:44.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:44.916 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:20:45.017 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:20:45.045 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 11 16 31159 31143 176.934 164.25 0.00625246 0.00564672 2024-08-19T14:20:45.047 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:45.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:45.048 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:20:45.149 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:20:45.179 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:45.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:45.180 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:20:45.281 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:20:45.311 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:45.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:45.312 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:20:45.413 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:20:45.436 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:45.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:45.437 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:20:45.537 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:20:45.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:45.560 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:45.560 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:20:45.661 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:20:45.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:45.690 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:45.691 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:20:45.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:45 smithi032 ceph-mon[20004]: pgmap v767: 47 pgs: 47 active+clean; 1.3 GiB data, 6.9 GiB used, 798 GiB / 805 GiB avail; 196 MiB/s wr, 3.13k op/s 2024-08-19T14:20:45.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:45 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:20:45.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:45 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:20:45.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:45 smithi032 ceph-mon[20004]: osdmap e343: 12 total, 12 up, 9 in 2024-08-19T14:20:45.791 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:20:45.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:45 smithi045 ceph-mon[20143]: pgmap v767: 47 pgs: 47 active+clean; 1.3 GiB data, 6.9 GiB used, 798 GiB / 805 GiB avail; 196 MiB/s wr, 3.13k op/s 2024-08-19T14:20:45.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:45 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:20:45.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:45 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:20:45.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:45 smithi045 ceph-mon[20143]: osdmap e343: 12 total, 12 up, 9 in 2024-08-19T14:20:45.813 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:45.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:45.814 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:20:45.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:45 smithi031 ceph-mon[16336]: pgmap v767: 47 pgs: 47 active+clean; 1.3 GiB data, 6.9 GiB used, 798 GiB / 805 GiB avail; 196 MiB/s wr, 3.13k op/s 2024-08-19T14:20:45.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:45 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:20:45.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:45 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:20:45.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:45 smithi031 ceph-mon[16336]: osdmap e343: 12 total, 12 up, 9 in 2024-08-19T14:20:45.915 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:20:45.944 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:45.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:45.945 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:20:46.029 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-08-19T14:20:46.032 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:20:46.045 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 12 16 32660 32644 170.006 93.8125 0.00190209 0.00581414 2024-08-19T14:20:46.046 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:20:46.069 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:46.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:46.070 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:20:46.171 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:20:46.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:46.194 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:46.194 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:20:46.296 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:20:46.321 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:46.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:46.323 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:20:46.424 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:20:46.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:46.527 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:46.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:20:46.628 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:20:46.662 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:46.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:46.663 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:20:46.765 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:20:46.789 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:20:46.794 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:46.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:46.795 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:20:46.896 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:20:46.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:46.918 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:46.918 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:20:47.020 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:20:47.045 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 13 16 33496 33480 160.947 52.25 0.0039034 0.00620611 2024-08-19T14:20:47.055 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:47.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:47.056 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:20:47.157 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:20:47.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:47.179 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:47.180 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:20:47.281 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:20:47.310 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:47.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:47.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:20:47.412 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:20:47.433 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:47.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:47.435 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:20:47.493 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:20:47.535 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:20:47.548 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:47 smithi031 ceph-mon[16336]: pgmap v769: 47 pgs: 47 active+clean; 1.8 GiB data, 8.0 GiB used, 797 GiB / 805 GiB avail; 174 MiB/s wr, 2.78k op/s 2024-08-19T14:20:47.548 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:47 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:20:47.548 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:47 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:20:47.548 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:47 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:20:47.548 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:47 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1461393782' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-08-19T14:20:47.548 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:47 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3665549058' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:20:47.549 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:20:47.549 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:20:47.572 INFO:teuthology.orchestra.run.smithi031.stderr:reweighted osd.7 to 1 (10000) 2024-08-19T14:20:47.576 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is set 2024-08-19T14:20:47.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:47.580 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:47.580 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:20:47.587 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:20:47.587 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:20:47.634 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:20:47.634 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:20:47.676 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:20:47.676 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:20:47.681 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:20:47.707 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:20:47.707 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:20:47.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:47.711 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:47.711 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:20:47.742 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:20:47.742 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:20:47.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:47 smithi032 ceph-mon[20004]: pgmap v769: 47 pgs: 47 active+clean; 1.8 GiB data, 8.0 GiB used, 797 GiB / 805 GiB avail; 174 MiB/s wr, 2.78k op/s 2024-08-19T14:20:47.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:47 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:20:47.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:47 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:20:47.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:47 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:20:47.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:47 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1461393782' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-08-19T14:20:47.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:47 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3665549058' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:20:47.777 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:20:47.778 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:20:47.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:47 smithi045 ceph-mon[20143]: pgmap v769: 47 pgs: 47 active+clean; 1.8 GiB data, 8.0 GiB used, 797 GiB / 805 GiB avail; 174 MiB/s wr, 2.78k op/s 2024-08-19T14:20:47.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:47 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:20:47.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:47 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:20:47.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:47 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:20:47.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:47 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1461393782' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-08-19T14:20:47.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:47 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3665549058' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:20:47.809 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:20:47.809 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:20:47.813 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:20:47.830 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:20:47.830 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:20:47.835 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:47.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:47.836 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:20:47.856 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:20:47.856 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:20:47.891 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:20:47.891 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:20:47.924 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:20:47.924 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:20:47.937 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:20:47.966 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:47.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:47.967 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:20:48.045 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 14 16 35254 35238 157.299 109.875 0.00482455 0.00635284 2024-08-19T14:20:48.068 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:20:48.099 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:48.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:48.100 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:20:48.202 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:20:48.233 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:48.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:48.234 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:20:48.267 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:20:48.268 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:20:48.334 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:20:48.365 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:48.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:48.366 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:20:48.467 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:20:48.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:48.489 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:48.489 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:20:48.590 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:20:48.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:48.612 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:48.613 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:20:48.714 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:20:48.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:48.750 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:48.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:20:48.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:48 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3302222484' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:20:48.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:48 smithi032 ceph-mon[20004]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:20:48.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:48 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1461393782' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-08-19T14:20:48.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:48 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3665549058' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:20:48.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:48 smithi032 ceph-mon[20004]: osdmap e344: 12 total, 12 up, 9 in 2024-08-19T14:20:48.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:48 smithi032 ceph-mon[20004]: osdmap e345: 12 total, 12 up, 9 in 2024-08-19T14:20:48.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:48 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3302222484' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:20:48.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:48 smithi045 ceph-mon[20143]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:20:48.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:48 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1461393782' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-08-19T14:20:48.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:48 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3665549058' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:20:48.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:48 smithi045 ceph-mon[20143]: osdmap e344: 12 total, 12 up, 9 in 2024-08-19T14:20:48.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:48 smithi045 ceph-mon[20143]: osdmap e345: 12 total, 12 up, 9 in 2024-08-19T14:20:48.851 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:20:48.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:48 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3302222484' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:20:48.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:48 smithi031 ceph-mon[16336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:20:48.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:48 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1461393782' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-08-19T14:20:48.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:48 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3665549058' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:20:48.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:48 smithi031 ceph-mon[16336]: osdmap e344: 12 total, 12 up, 9 in 2024-08-19T14:20:48.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:48 smithi031 ceph-mon[16336]: osdmap e345: 12 total, 12 up, 9 in 2024-08-19T14:20:48.881 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:48.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:48.887 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:20:48.988 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:20:49.022 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:49.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:49.023 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:20:49.045 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 15 16 37479 37463 156.083 139.062 0.0071007 0.00640119 2024-08-19T14:20:49.124 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:20:49.145 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:49.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:49.146 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:20:49.248 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:20:49.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:49.280 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:49.281 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:20:49.382 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:20:49.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:49.414 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:49.414 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:20:49.515 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:20:49.542 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:49.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:49.543 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:20:49.644 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:20:49.676 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:49.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:49.677 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:20:49.764 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-08-19T14:20:49.777 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:20:49.806 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:49.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:49.807 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:20:49.908 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:20:49.936 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:49.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:49.937 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:20:50.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:49 smithi032 ceph-mon[20004]: pgmap v772: 47 pgs: 47 active+clean; 1.8 GiB data, 8.3 GiB used, 796 GiB / 805 GiB avail; 119 MiB/s wr, 1.90k op/s 2024-08-19T14:20:50.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:49 smithi032 ceph-mon[20004]: osdmap e346: 12 total, 12 up, 9 in 2024-08-19T14:20:50.038 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:20:50.045 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 16 16 39653 39637 154.819 135.875 0.00417461 0.00645441 2024-08-19T14:20:50.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:49 smithi045 ceph-mon[20143]: pgmap v772: 47 pgs: 47 active+clean; 1.8 GiB data, 8.3 GiB used, 796 GiB / 805 GiB avail; 119 MiB/s wr, 1.90k op/s 2024-08-19T14:20:50.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:49 smithi045 ceph-mon[20143]: osdmap e346: 12 total, 12 up, 9 in 2024-08-19T14:20:50.070 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:49 smithi031 ceph-mon[16336]: pgmap v772: 47 pgs: 47 active+clean; 1.8 GiB data, 8.3 GiB used, 796 GiB / 805 GiB avail; 119 MiB/s wr, 1.90k op/s 2024-08-19T14:20:50.070 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:49 smithi031 ceph-mon[16336]: osdmap e346: 12 total, 12 up, 9 in 2024-08-19T14:20:50.071 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:50.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:50.072 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:20:50.173 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:20:50.201 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:50.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:50.202 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:20:50.303 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:20:50.340 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:50.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:50.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:20:50.442 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:20:50.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:50.490 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:50.490 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:20:50.592 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:20:50.627 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:50.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:50.628 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:20:50.729 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:20:50.759 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:50.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:50.760 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:20:50.861 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:20:50.894 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:50.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:50.895 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:20:50.946 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-19T14:20:50.996 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:20:51.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:50 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:20:51.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:50 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:20:51.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:50 smithi032 ceph-mon[20004]: osdmap e347: 12 total, 12 up, 9 in 2024-08-19T14:20:51.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:50 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:20:51.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:50 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:20:51.022 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:51.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:51.023 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:20:51.045 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 17 16 42141 42125 154.858 155.5 0.00554731 0.00645299 2024-08-19T14:20:51.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:50 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:20:51.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:50 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:20:51.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:50 smithi045 ceph-mon[20143]: osdmap e347: 12 total, 12 up, 9 in 2024-08-19T14:20:51.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:50 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:20:51.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:50 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:20:51.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:50 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:20:51.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:50 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:20:51.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:50 smithi031 ceph-mon[16336]: osdmap e347: 12 total, 12 up, 9 in 2024-08-19T14:20:51.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:50 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:20:51.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:50 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:20:51.124 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:20:51.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:51.171 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:51.171 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:20:51.216 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd reweight 8 1 2024-08-19T14:20:51.261 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:20:51.272 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:20:51.301 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:51.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:51.303 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:20:51.404 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:20:51.443 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:51.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:51.445 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:20:51.546 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:20:51.580 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:51.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:51.581 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:20:51.682 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:20:51.710 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:51.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:51.711 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:20:51.812 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:20:51.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:51.836 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:51.836 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:20:51.920 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:20:51.920 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:20:51.936 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:20:51.952 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:51 smithi031 ceph-mon[16336]: pgmap v775: 47 pgs: 47 active+clean; 2.1 GiB data, 9.5 GiB used, 795 GiB / 805 GiB avail; 97 MiB/s wr, 1.56k op/s 2024-08-19T14:20:51.952 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:51 smithi031 ceph-mon[16336]: osdmap e348: 12 total, 12 up, 9 in 2024-08-19T14:20:51.969 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:51.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:51.970 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:20:52.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:51 smithi032 ceph-mon[20004]: pgmap v775: 47 pgs: 47 active+clean; 2.1 GiB data, 9.5 GiB used, 795 GiB / 805 GiB avail; 97 MiB/s wr, 1.56k op/s 2024-08-19T14:20:52.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:51 smithi032 ceph-mon[20004]: osdmap e348: 12 total, 12 up, 9 in 2024-08-19T14:20:52.045 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 18 16 43845 43829 152.171 106.5 0.00554675 0.00656732 2024-08-19T14:20:52.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:51 smithi045 ceph-mon[20143]: pgmap v775: 47 pgs: 47 active+clean; 2.1 GiB data, 9.5 GiB used, 795 GiB / 805 GiB avail; 97 MiB/s wr, 1.56k op/s 2024-08-19T14:20:52.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:51 smithi045 ceph-mon[20143]: osdmap e348: 12 total, 12 up, 9 in 2024-08-19T14:20:52.071 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:20:52.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:52.106 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:52.106 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:20:52.196 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:20:52.196 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:20:52.207 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:20:52.230 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:52.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:52.231 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:20:52.332 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:20:52.362 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:52.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:52.363 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:20:52.464 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:20:52.482 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:20:52.497 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:52.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:52.498 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:20:52.599 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:20:52.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:52.638 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:52.638 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:20:52.739 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:20:52.766 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:52.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:52.767 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:20:52.784 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-08-19T14:20:52.868 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:20:52.895 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:52.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:52.896 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:20:52.997 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:20:53.024 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:53.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:53.025 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:20:53.046 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 19 15 46050 46035 151.418 137.875 0.00797135 0.00659976 2024-08-19T14:20:53.127 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:20:53.153 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:53.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:53.154 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:20:53.255 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:20:53.283 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:53.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:53.284 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:20:53.385 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:20:53.417 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:53.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:53.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:20:53.519 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:20:53.548 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:53.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:53.549 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:20:53.650 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:20:53.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:53.688 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:53.689 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:20:53.789 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:20:53.819 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:53.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:53.820 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:20:53.921 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:20:53.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:53.961 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:53.961 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:20:54.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:53 smithi032 ceph-mon[20004]: pgmap v777: 47 pgs: 47 active+clean; 2.3 GiB data, 10 GiB used, 795 GiB / 805 GiB avail; 137 MiB/s wr, 2.19k op/s 2024-08-19T14:20:54.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:53 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:20:54.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:53 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:20:54.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:53 smithi032 ceph-mon[20004]: osdmap e349: 12 total, 12 up, 9 in 2024-08-19T14:20:54.046 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:2024-08-19T14:20:54.046293+0000 min lat: 0.00106793 max lat: 0.816389 avg lat: 0.00663776 2024-08-19T14:20:54.046 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-19T14:20:54.046 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 20 16 48198 48182 150.556 134.188 0.00669761 0.00663776 2024-08-19T14:20:54.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:53 smithi045 ceph-mon[20143]: pgmap v777: 47 pgs: 47 active+clean; 2.3 GiB data, 10 GiB used, 795 GiB / 805 GiB avail; 137 MiB/s wr, 2.19k op/s 2024-08-19T14:20:54.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:53 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:20:54.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:53 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:20:54.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:53 smithi045 ceph-mon[20143]: osdmap e349: 12 total, 12 up, 9 in 2024-08-19T14:20:54.047 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:53 smithi031 ceph-mon[16336]: pgmap v777: 47 pgs: 47 active+clean; 2.3 GiB data, 10 GiB used, 795 GiB / 805 GiB avail; 137 MiB/s wr, 2.19k op/s 2024-08-19T14:20:54.047 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:53 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:20:54.047 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:53 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:20:54.047 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:53 smithi031 ceph-mon[16336]: osdmap e349: 12 total, 12 up, 9 in 2024-08-19T14:20:54.062 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:20:54.086 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:54.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:54.088 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:20:54.189 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:20:54.222 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:54.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:54.223 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:20:54.324 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:20:54.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:54.353 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:54.354 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:20:54.454 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:20:54.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:54.507 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:54.507 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:20:54.568 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:20:54.588 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:20:54.608 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:20:54.635 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:54.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:54.636 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:20:54.738 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:20:54.767 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is set 2024-08-19T14:20:54.789 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:54.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:54.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:20:54.893 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:20:54.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:54.919 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:54.919 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:20:55.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:54 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2734826008' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:20:55.021 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:20:55.046 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 21 16 49716 49700 147.904 94.875 0.00552094 0.00675618 2024-08-19T14:20:55.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:54 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2734826008' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:20:55.048 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:55.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:55.050 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:20:55.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:54 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2734826008' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:20:55.151 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:20:55.190 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:55.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:55.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:20:55.292 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:20:55.320 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:55.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:55.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:20:55.422 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:20:55.457 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:55.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:55.458 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:20:55.559 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:20:55.588 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:55.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:55.589 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:20:55.689 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:20:55.729 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:55.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:55.730 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:20:55.770 INFO:teuthology.orchestra.run.smithi031.stderr:reweighted osd.8 to 1 (10000) 2024-08-19T14:20:55.831 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:20:55.834 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:55 smithi031 ceph-mon[16336]: pgmap v779: 47 pgs: 47 active+clean; 2.5 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 115 MiB/s wr, 1.83k op/s 2024-08-19T14:20:55.834 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:55 smithi031 ceph-mon[16336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:20:55.834 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:55 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2734826008' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:20:55.834 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:55 smithi031 ceph-mon[16336]: osdmap e350: 12 total, 12 up, 9 in 2024-08-19T14:20:55.834 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:55 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2014076621' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-08-19T14:20:55.868 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:55.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:55.869 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:20:55.970 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:20:56.007 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:56.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:56.008 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:20:56.046 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 22 16 51897 51881 147.377 136.312 0.00482543 0.00678092 2024-08-19T14:20:56.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:55 smithi045 ceph-mon[20143]: pgmap v779: 47 pgs: 47 active+clean; 2.5 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 115 MiB/s wr, 1.83k op/s 2024-08-19T14:20:56.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:55 smithi045 ceph-mon[20143]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:20:56.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:55 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2734826008' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:20:56.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:55 smithi045 ceph-mon[20143]: osdmap e350: 12 total, 12 up, 9 in 2024-08-19T14:20:56.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:55 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2014076621' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-08-19T14:20:56.109 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:20:56.137 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:56.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:56.138 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:20:56.239 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:20:56.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:55 smithi032 ceph-mon[20004]: pgmap v779: 47 pgs: 47 active+clean; 2.5 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 115 MiB/s wr, 1.83k op/s 2024-08-19T14:20:56.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:55 smithi032 ceph-mon[20004]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:20:56.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:55 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2734826008' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:20:56.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:55 smithi032 ceph-mon[20004]: osdmap e350: 12 total, 12 up, 9 in 2024-08-19T14:20:56.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:55 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2014076621' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-08-19T14:20:56.269 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:56.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:56.270 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:20:56.293 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:20:56.293 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:20:56.371 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:20:56.404 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:56.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:56.405 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:20:56.506 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:20:56.542 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:56.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:56.543 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:20:56.644 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:20:56.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:56.671 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:56.671 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:20:56.772 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:20:56.801 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:56.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:56.803 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:20:56.815 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd reweight 9 1 2024-08-19T14:20:56.903 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:20:56.924 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:56.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:56.925 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:20:56.973 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:20:56.973 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:20:56.990 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:20:56.990 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:20:57.006 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:20:57.007 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:20:57.026 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:20:57.039 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:56 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2014076621' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-08-19T14:20:57.040 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:56 smithi031 ceph-mon[16336]: osdmap e351: 12 total, 12 up, 9 in 2024-08-19T14:20:57.042 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:20:57.042 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:20:57.046 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 23 16 53691 53675 145.844 112.125 0.00585056 0.00685251 2024-08-19T14:20:57.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:56 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2014076621' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-08-19T14:20:57.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:56 smithi045 ceph-mon[20143]: osdmap e351: 12 total, 12 up, 9 in 2024-08-19T14:20:57.068 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:20:57.068 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:20:57.070 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:57.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:57.071 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:20:57.089 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:20:57.089 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:20:57.119 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:20:57.120 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:20:57.153 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:20:57.153 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:20:57.171 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:20:57.185 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:20:57.186 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:20:57.198 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:57.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:57.199 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:20:57.219 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:20:57.219 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:20:57.253 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:20:57.253 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:20:57.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:56 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2014076621' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-08-19T14:20:57.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:56 smithi032 ceph-mon[20004]: osdmap e351: 12 total, 12 up, 9 in 2024-08-19T14:20:57.287 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:20:57.288 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:20:57.300 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:20:57.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:57.326 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:57.326 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:20:57.428 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:20:57.450 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:57.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:57.451 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:20:57.552 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:20:57.579 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:57.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:57.580 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:20:57.630 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-19T14:20:57.681 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:20:57.706 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:57.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:57.707 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:20:57.778 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:57 smithi032 ceph-mon[20004]: pgmap v782: 47 pgs: 47 active+clean; 3.0 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 162 MiB/s wr, 2.60k op/s 2024-08-19T14:20:57.778 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:57 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1863245716' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:20:57.807 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:20:57.869 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:57.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:57.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:20:57.972 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:20:58.003 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:58.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:58.004 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:20:58.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:57 smithi045 ceph-mon[20143]: pgmap v782: 47 pgs: 47 active+clean; 3.0 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 162 MiB/s wr, 2.60k op/s 2024-08-19T14:20:58.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:57 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1863245716' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:20:58.046 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 24 15 55282 55267 143.912 99.5 0.00595355 0.00694474 2024-08-19T14:20:58.105 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:20:58.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:57 smithi031 ceph-mon[16336]: pgmap v782: 47 pgs: 47 active+clean; 3.0 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 162 MiB/s wr, 2.60k op/s 2024-08-19T14:20:58.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:57 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1863245716' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:20:58.138 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:58.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:58.139 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:20:58.240 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:20:58.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:58.272 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:58.272 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:20:58.373 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:20:58.394 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:58.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:58.395 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:20:58.495 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:20:58.516 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:58.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:58.517 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:20:58.618 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:20:58.651 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:58.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:58.652 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:20:58.753 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:20:58.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:58.775 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:58.775 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:20:58.876 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:20:58.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:58.905 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:58.905 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:20:59.006 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:20:59.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:59.029 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:59.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:20:59.046 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 25 16 56550 56534 141.323 79.1875 0.00204428 0.00702858 2024-08-19T14:20:59.129 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:20:59.158 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:59.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:59.159 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:20:59.260 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:20:59.281 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:59.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:59.282 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:20:59.383 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:20:59.411 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:59.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:59.412 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:20:59.513 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:20:59.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:59.539 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:59.539 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:20:59.640 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:20:59.667 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:20:59.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:59.668 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:20:59.770 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:20:59.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:59.804 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:20:59.804 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:20:59.905 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:20:59.937 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:20:59.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:20:59.938 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:21:00.039 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:21:00.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:20:59 smithi045 ceph-mon[20143]: pgmap v783: 47 pgs: 47 active+clean; 3.0 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 109 MiB/s wr, 1.75k op/s 2024-08-19T14:21:00.046 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 26 16 58177 58161 139.798 101.688 0.00420466 0.00714869 2024-08-19T14:21:00.068 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:00.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:00.069 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:21:00.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:20:59 smithi031 ceph-mon[16336]: pgmap v783: 47 pgs: 47 active+clean; 3.0 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 109 MiB/s wr, 1.75k op/s 2024-08-19T14:21:00.170 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:21:00.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:00.192 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:00.193 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:21:00.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:20:59 smithi032 ceph-mon[20004]: pgmap v783: 47 pgs: 47 active+clean; 3.0 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 109 MiB/s wr, 1.75k op/s 2024-08-19T14:21:00.294 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:21:00.335 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:00.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:00.336 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:21:00.438 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:21:00.466 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:00.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:00.468 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:21:00.569 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:21:00.596 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:00.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:00.596 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:21:00.698 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:21:00.725 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:00.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:00.727 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:21:00.827 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:21:00.852 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:00.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:00.853 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:21:00.954 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:21:00.988 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:00.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:00.989 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:21:01.046 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 27 16 60034 60018 138.918 116.062 0.00444485 0.00718811 2024-08-19T14:21:01.089 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:21:01.113 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:01.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:01.114 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:21:01.215 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:21:01.254 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:01.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:01.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:21:01.356 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:21:01.393 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:01.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:01.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:21:01.496 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:21:01.525 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:01.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:01.527 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:21:01.628 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:21:01.654 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:01.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:01.655 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:21:01.756 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:21:01.785 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:01.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:01.786 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:21:01.887 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:21:01.916 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:01.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:01.917 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:21:02.018 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:21:02.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:01 smithi045 ceph-mon[20143]: pgmap v784: 47 pgs: 47 active+clean; 3.3 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 139 MiB/s wr, 2.23k op/s 2024-08-19T14:21:02.046 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 28 16 61810 61794 137.921 111 0.00969109 0.00724611 2024-08-19T14:21:02.052 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:02.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:02.053 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:21:02.118 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:01 smithi031 ceph-mon[16336]: pgmap v784: 47 pgs: 47 active+clean; 3.3 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 139 MiB/s wr, 2.23k op/s 2024-08-19T14:21:02.154 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:21:02.190 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:02.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:02.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:21:02.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:01 smithi032 ceph-mon[20004]: pgmap v784: 47 pgs: 47 active+clean; 3.3 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 139 MiB/s wr, 2.23k op/s 2024-08-19T14:21:02.292 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:21:02.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:02.327 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:02.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:21:02.428 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:21:02.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:02.457 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:02.457 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:21:02.558 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:21:02.580 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-08-19T14:21:02.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:02.622 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:02.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:21:02.628 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:21:02.665 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:21:02.724 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:21:02.751 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:02.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:02.752 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:21:02.853 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:21:02.881 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:02.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:02.882 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:21:02.983 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:21:03.027 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:03.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:03.028 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:21:03.046 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 29 16 63658 63642 137.148 115.5 0.0037382 0.0072861 2024-08-19T14:21:03.128 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:21:03.158 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:03.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:03.159 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:21:03.261 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:21:03.289 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:03.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:03.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:21:03.391 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:21:03.419 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:03.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:03.420 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:21:03.521 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:21:03.545 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:03.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:03.546 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:21:03.647 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:21:03.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:03.690 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:03.690 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:21:03.791 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:21:03.795 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is unset 2024-08-19T14:21:03.837 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:03.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:03.839 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:21:03.940 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:21:03.969 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:03.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:03.970 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:21:04.046 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 30 16 65227 65211 135.845 98.0625 0.00488439 0.00735684 2024-08-19T14:21:04.071 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:21:04.106 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:04.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:04.107 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:21:04.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:03 smithi031 ceph-mon[16336]: pgmap v785: 47 pgs: 47 active+clean; 3.5 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 128 MiB/s wr, 2.04k op/s 2024-08-19T14:21:04.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:03 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3017663999' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:21:04.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:03 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:21:04.185 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:21:04.208 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:21:04.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:04.252 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:04.252 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:21:04.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:03 smithi032 ceph-mon[20004]: pgmap v785: 47 pgs: 47 active+clean; 3.5 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 128 MiB/s wr, 2.04k op/s 2024-08-19T14:21:04.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:03 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3017663999' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:21:04.259 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:03 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:21:04.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:03 smithi045 ceph-mon[20143]: pgmap v785: 47 pgs: 47 active+clean; 3.5 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 128 MiB/s wr, 2.04k op/s 2024-08-19T14:21:04.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:03 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3017663999' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:21:04.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:03 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:21:04.353 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:21:04.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:04.377 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:04.378 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:21:04.479 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:21:04.499 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:04.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:04.500 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:21:04.601 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:21:04.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:04.636 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:04.637 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:21:04.738 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:21:04.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:04.776 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:04.776 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:21:04.802 INFO:teuthology.orchestra.run.smithi031.stderr:reweighted osd.9 to 1 (10000) 2024-08-19T14:21:04.877 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:21:04.904 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:04.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:04.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:21:05.006 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:21:05.027 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:05.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:05.028 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:21:05.047 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 31 16 66544 66528 134.117 82.3125 0.00185856 0.00741908 2024-08-19T14:21:05.098 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:04 smithi031 ceph-mon[16336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:21:05.098 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:04 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:21:05.098 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:04 smithi031 ceph-mon[16336]: osdmap e352: 12 total, 12 up, 9 in 2024-08-19T14:21:05.098 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:04 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2971532572' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-08-19T14:21:05.129 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:21:05.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:05.152 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:05.152 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:21:05.254 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:21:05.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:04 smithi032 ceph-mon[20004]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:21:05.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:04 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:21:05.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:04 smithi032 ceph-mon[20004]: osdmap e352: 12 total, 12 up, 9 in 2024-08-19T14:21:05.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:04 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2971532572' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-08-19T14:21:05.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:05.276 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:05.277 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:21:05.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:04 smithi045 ceph-mon[20143]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:21:05.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:04 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:21:05.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:04 smithi045 ceph-mon[20143]: osdmap e352: 12 total, 12 up, 9 in 2024-08-19T14:21:05.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:04 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2971532572' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-08-19T14:21:05.377 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:21:05.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:05.400 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:05.400 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:21:05.501 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:21:05.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:05.525 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:05.525 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:21:05.626 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:21:05.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:05.650 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:05.650 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:21:05.751 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:21:05.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:05.774 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:05.775 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:21:05.878 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:21:05.904 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:05.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:05.905 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:21:06.006 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:21:06.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:06.028 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:06.028 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:21:06.044 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:05 smithi031 ceph-mon[16336]: pgmap v787: 47 pgs: 47 active+clean; 3.6 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 110 MiB/s wr, 1.77k op/s 2024-08-19T14:21:06.044 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:05 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2971532572' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-08-19T14:21:06.044 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:05 smithi031 ceph-mon[16336]: osdmap e353: 12 total, 12 up, 9 in 2024-08-19T14:21:06.044 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:05 smithi031 ceph-mon[16336]: osdmap e354: 12 total, 12 up, 9 in 2024-08-19T14:21:06.047 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 32 16 66550 66534 129.938 0.375 0.00239219 0.00741865 2024-08-19T14:21:06.073 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd reweight 11 1 2024-08-19T14:21:06.107 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:21:06.119 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:21:06.129 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:21:06.154 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:06.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:06.155 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:21:06.256 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:21:06.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:05 smithi032 ceph-mon[20004]: pgmap v787: 47 pgs: 47 active+clean; 3.6 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 110 MiB/s wr, 1.77k op/s 2024-08-19T14:21:06.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:05 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2971532572' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-08-19T14:21:06.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:05 smithi032 ceph-mon[20004]: osdmap e353: 12 total, 12 up, 9 in 2024-08-19T14:21:06.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:05 smithi032 ceph-mon[20004]: osdmap e354: 12 total, 12 up, 9 in 2024-08-19T14:21:06.280 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:06.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:06.281 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:21:06.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:05 smithi045 ceph-mon[20143]: pgmap v787: 47 pgs: 47 active+clean; 3.6 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 110 MiB/s wr, 1.77k op/s 2024-08-19T14:21:06.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:05 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2971532572' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-08-19T14:21:06.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:05 smithi045 ceph-mon[20143]: osdmap e353: 12 total, 12 up, 9 in 2024-08-19T14:21:06.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:05 smithi045 ceph-mon[20143]: osdmap e354: 12 total, 12 up, 9 in 2024-08-19T14:21:06.383 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:21:06.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:06.405 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:06.405 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:21:06.505 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:21:06.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:06.532 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:06.533 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:21:06.633 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:21:06.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:06.655 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:06.655 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:21:06.756 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:21:06.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:06.778 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:06.778 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:21:06.785 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:21:06.785 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-08-19T14:21:06.785 INFO:teuthology.orchestra.run.smithi031.stdout: "num_ops": 0 2024-08-19T14:21:06.785 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:21:06.879 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:21:06.905 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:06.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:06.906 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:21:06.983 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:21:06.984 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:21:07.001 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:21:07.001 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:21:07.007 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:21:07.018 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:21:07.019 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:21:07.029 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:07.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:07.030 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:21:07.036 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:21:07.036 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:21:07.047 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 33 16 66550 66534 126 0 - 0.00741865 2024-08-19T14:21:07.053 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:21:07.053 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:21:07.070 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:21:07.071 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:21:07.088 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:21:07.088 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:21:07.106 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:21:07.106 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:21:07.123 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:21:07.123 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:21:07.131 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:21:07.141 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:21:07.141 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:21:07.154 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:07.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:07.155 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:21:07.158 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:21:07.158 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:21:07.176 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:21:07.176 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:21:07.256 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:21:07.288 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:07.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:07.289 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:21:07.390 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:21:07.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:07.420 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:07.420 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:21:07.459 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-19T14:21:07.522 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:21:07.544 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:07.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:07.545 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:21:07.646 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:21:07.673 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:07.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:07.674 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:21:07.775 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:21:07.812 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:07.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:07.813 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:21:07.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:07 smithi031 ceph-mon[16336]: pgmap v790: 47 pgs: 47 active+clean; 4.0 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 110 MiB/s wr, 1.76k op/s 2024-08-19T14:21:07.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:07 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:21:07.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:07 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:21:07.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:07 smithi031 ceph-mon[16336]: osdmap e355: 12 total, 12 up, 9 in 2024-08-19T14:21:07.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:07 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/468235265' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:21:07.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:07 smithi031 ceph-mon[16336]: 1.d scrub starts 2024-08-19T14:21:07.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:07 smithi031 ceph-mon[16336]: 1.d scrub ok 2024-08-19T14:21:07.914 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:21:07.942 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:07.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:07.943 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:21:08.019 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-08-19T14:21:08.044 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:21:08.048 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 34 16 67731 67715 124.465 36.9062 0.00816559 0.00802774 2024-08-19T14:21:08.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:08.074 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:08.074 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:21:08.175 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:21:08.201 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:08.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:08.202 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:21:08.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:07 smithi032 ceph-mon[20004]: pgmap v790: 47 pgs: 47 active+clean; 4.0 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 110 MiB/s wr, 1.76k op/s 2024-08-19T14:21:08.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:07 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:21:08.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:07 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:21:08.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:07 smithi032 ceph-mon[20004]: osdmap e355: 12 total, 12 up, 9 in 2024-08-19T14:21:08.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:07 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/468235265' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:21:08.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:07 smithi032 ceph-mon[20004]: 1.d scrub starts 2024-08-19T14:21:08.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:07 smithi032 ceph-mon[20004]: 1.d scrub ok 2024-08-19T14:21:08.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:07 smithi045 ceph-mon[20143]: pgmap v790: 47 pgs: 47 active+clean; 4.0 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 110 MiB/s wr, 1.76k op/s 2024-08-19T14:21:08.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:07 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:21:08.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:07 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:21:08.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:07 smithi045 ceph-mon[20143]: osdmap e355: 12 total, 12 up, 9 in 2024-08-19T14:21:08.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:07 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/468235265' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:21:08.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:07 smithi045 ceph-mon[20143]: 1.d scrub starts 2024-08-19T14:21:08.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:07 smithi045 ceph-mon[20143]: 1.d scrub ok 2024-08-19T14:21:08.302 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:21:08.329 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:08.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:08.330 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:21:08.430 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:21:08.432 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:21:08.456 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:08.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:08.457 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:21:08.495 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:21:08.519 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:21:08.558 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:21:08.598 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:08.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:08.599 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:21:08.701 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:21:08.729 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:08.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:08.730 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:21:08.831 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:21:08.859 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:08.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:08.860 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:21:08.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:08 smithi031 ceph-mon[16336]: 1.10 scrub starts 2024-08-19T14:21:08.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:08 smithi031 ceph-mon[16336]: 1.10 scrub ok 2024-08-19T14:21:08.960 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:21:08.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:08.988 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:08.989 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:21:09.047 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 35 16 69274 69258 123.664 96.4375 0.0162461 0.00808106 2024-08-19T14:21:09.089 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:21:09.116 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:09.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:09.117 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:21:09.218 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:21:09.245 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:09.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:09.246 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:21:09.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:08 smithi032 ceph-mon[20004]: 1.10 scrub starts 2024-08-19T14:21:09.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:08 smithi032 ceph-mon[20004]: 1.10 scrub ok 2024-08-19T14:21:09.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:08 smithi045 ceph-mon[20143]: 1.10 scrub starts 2024-08-19T14:21:09.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:08 smithi045 ceph-mon[20143]: 1.10 scrub ok 2024-08-19T14:21:09.347 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:21:09.370 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:09.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:09.371 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:21:09.407 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:21:09.407 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:21:09.471 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:21:09.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:09.497 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:09.497 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:21:09.598 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:21:09.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:09.621 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:09.621 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:21:09.722 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:21:09.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:09.744 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:09.744 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:21:09.845 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:21:09.890 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:09.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:09.891 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:21:09.992 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:21:10.020 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:10.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:10.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:21:10.047 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 36 16 70292 70276 121.996 63.625 0.00597292 0.00819217 2024-08-19T14:21:10.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:09 smithi031 ceph-mon[16336]: pgmap v792: 47 pgs: 47 active+clean; 4.0 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 88 MiB/s wr, 1.41k op/s 2024-08-19T14:21:10.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:09 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2192302376' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-08-19T14:21:10.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:09 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-08-19T14:21:10.122 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:21:10.147 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:10.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:10.148 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:21:10.249 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:21:10.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:09 smithi032 ceph-mon[20004]: pgmap v792: 47 pgs: 47 active+clean; 4.0 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 88 MiB/s wr, 1.41k op/s 2024-08-19T14:21:10.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:09 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2192302376' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-08-19T14:21:10.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:09 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-08-19T14:21:10.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:10.278 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:10.278 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:21:10.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:09 smithi045 ceph-mon[20143]: pgmap v792: 47 pgs: 47 active+clean; 4.0 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 88 MiB/s wr, 1.41k op/s 2024-08-19T14:21:10.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:09 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2192302376' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-08-19T14:21:10.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:09 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-08-19T14:21:10.378 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:21:10.400 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:21:10.409 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:10.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:10.411 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:21:10.512 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:21:10.555 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:10.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:10.556 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:21:10.657 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:21:10.682 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:10.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:10.683 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:21:10.783 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:21:10.811 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:10.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:10.812 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:21:10.913 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:21:10.945 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:10.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:10.946 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:21:11.047 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:21:11.048 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 37 16 71925 71909 121.457 102.062 0.0359552 0.00822815 2024-08-19T14:21:11.075 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:11.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:11.076 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:21:11.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:10 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-08-19T14:21:11.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:10 smithi031 ceph-mon[16336]: osdmap e356: 12 total, 12 up, 9 in 2024-08-19T14:21:11.177 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:21:11.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:11.209 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:11.209 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:21:11.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:10 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-08-19T14:21:11.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:10 smithi032 ceph-mon[20004]: osdmap e356: 12 total, 12 up, 9 in 2024-08-19T14:21:11.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:10 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-08-19T14:21:11.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:10 smithi045 ceph-mon[20143]: osdmap e356: 12 total, 12 up, 9 in 2024-08-19T14:21:11.311 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:21:11.346 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:11.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:11.347 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:21:11.448 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:21:11.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:11.470 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:11.470 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:21:11.571 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:21:11.599 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:11.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:11.600 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:21:11.701 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:21:11.728 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:11.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:11.729 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:21:11.813 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-08-19T14:21:11.830 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:21:11.904 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:11.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:11.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:21:12.006 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:21:12.021 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:11 smithi031 ceph-mon[16336]: pgmap v794: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 4.1 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 62 MiB/s wr, 990 op/s; 4104/203715 objects misplaced (2.015%); 453 KiB/s, 7 objects/s recovering 2024-08-19T14:21:12.021 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:11 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:21:12.021 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:11 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:21:12.021 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:11 smithi031 ceph-mon[16336]: osdmap e357: 12 total, 12 up, 9 in 2024-08-19T14:21:12.031 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:12.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:12.033 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:21:12.048 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 38 15 73456 73441 120.78 95.75 0.007849 0.0082698 2024-08-19T14:21:12.133 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:21:12.176 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:12.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:12.177 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:21:12.214 INFO:teuthology.orchestra.run.smithi031.stderr:reweighted osd.11 to 1 (10000) 2024-08-19T14:21:12.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:11 smithi032 ceph-mon[20004]: pgmap v794: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 4.1 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 62 MiB/s wr, 990 op/s; 4104/203715 objects misplaced (2.015%); 453 KiB/s, 7 objects/s recovering 2024-08-19T14:21:12.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:11 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:21:12.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:11 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:21:12.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:11 smithi032 ceph-mon[20004]: osdmap e357: 12 total, 12 up, 9 in 2024-08-19T14:21:12.278 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:21:12.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:11 smithi045 ceph-mon[20143]: pgmap v794: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 4.1 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 62 MiB/s wr, 990 op/s; 4104/203715 objects misplaced (2.015%); 453 KiB/s, 7 objects/s recovering 2024-08-19T14:21:12.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:11 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:21:12.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:11 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:21:12.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:11 smithi045 ceph-mon[20143]: osdmap e357: 12 total, 12 up, 9 in 2024-08-19T14:21:12.315 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:12.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:12.316 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:21:12.416 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:21:12.445 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:12.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:12.446 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:21:12.547 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:21:12.576 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:12.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:12.578 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:21:12.679 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:21:12.710 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:12.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:12.711 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:21:12.812 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:21:12.836 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is unset 2024-08-19T14:21:12.840 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:12.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:12.841 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:21:12.942 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:21:12.967 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:12.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:12.968 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:21:12.994 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-08-19T14:21:12.995 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-19T14:21:13.048 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 39 16 74950 74934 120.076 93.3125 0.0056035 0.00832322 2024-08-19T14:21:13.069 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:21:13.097 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:13.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:13.099 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:21:13.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:12 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/271677855' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:21:13.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:12 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:21:13.200 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:21:13.227 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:13.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:13.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:21:13.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:12 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/271677855' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:21:13.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:12 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:21:13.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:12 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/271677855' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:21:13.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:12 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:21:13.329 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:21:13.357 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:13.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:13.358 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:21:13.459 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:21:13.484 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:13.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:13.486 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:21:13.590 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:21:13.616 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:13.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:13.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:21:13.693 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:21:13.718 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:21:13.763 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:21:13.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:13.773 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:13.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:21:13.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:13 smithi031 ceph-mon[16336]: pgmap v796: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 4.3 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 50 MiB/s wr, 798 op/s; 4104/209319 objects misplaced (1.961%); 384 KiB/s, 5 objects/s recovering 2024-08-19T14:21:13.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:13 smithi031 ceph-mon[16336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:21:13.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:13 smithi031 ceph-mon[16336]: Cluster is now healthy 2024-08-19T14:21:13.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:13 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:21:13.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:13 smithi031 ceph-mon[16336]: osdmap e358: 12 total, 12 up, 9 in 2024-08-19T14:21:13.874 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:21:13.901 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:13.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:13.902 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:21:14.003 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:21:14.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:14.030 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:14.031 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:21:14.048 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:2024-08-19T14:21:14.048190+0000 min lat: 0.00106793 max lat: 2.42694 avg lat: 0.00837596 2024-08-19T14:21:14.048 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-19T14:21:14.048 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 40 16 76377 76361 119.303 89.1875 0.0268599 0.00837596 2024-08-19T14:21:14.132 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:21:14.171 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:14.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:14.172 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:21:14.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:13 smithi032 ceph-mon[20004]: pgmap v796: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 4.3 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 50 MiB/s wr, 798 op/s; 4104/209319 objects misplaced (1.961%); 384 KiB/s, 5 objects/s recovering 2024-08-19T14:21:14.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:13 smithi032 ceph-mon[20004]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:21:14.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:13 smithi032 ceph-mon[20004]: Cluster is now healthy 2024-08-19T14:21:14.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:13 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:21:14.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:13 smithi032 ceph-mon[20004]: osdmap e358: 12 total, 12 up, 9 in 2024-08-19T14:21:14.273 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:21:14.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:13 smithi045 ceph-mon[20143]: pgmap v796: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 4.3 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 50 MiB/s wr, 798 op/s; 4104/209319 objects misplaced (1.961%); 384 KiB/s, 5 objects/s recovering 2024-08-19T14:21:14.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:13 smithi045 ceph-mon[20143]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:21:14.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:13 smithi045 ceph-mon[20143]: Cluster is now healthy 2024-08-19T14:21:14.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:13 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:21:14.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:13 smithi045 ceph-mon[20143]: osdmap e358: 12 total, 12 up, 9 in 2024-08-19T14:21:14.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:14.303 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:14.303 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:21:14.404 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:21:14.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:14.440 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:14.440 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:21:14.529 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:21:14.529 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:21:14.541 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:21:14.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:14.577 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:14.577 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:21:14.578 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:21:14.578 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:21:14.606 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:21:14.607 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:21:14.640 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:21:14.640 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:21:14.669 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:21:14.669 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:21:14.679 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:21:14.691 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:21:14.691 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:21:14.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:14.702 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:14.702 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:21:14.709 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:21:14.709 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:21:14.727 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:21:14.728 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:21:14.744 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:21:14.745 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:21:14.763 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:21:14.763 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:21:14.780 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:21:14.781 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:21:14.798 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:21:14.798 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:21:14.803 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:21:14.824 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:14.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:14.826 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:21:14.926 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:21:14.950 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:14.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:14.951 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:21:15.048 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 41 16 77496 77480 118.099 69.9375 0.00494104 0.00846289 2024-08-19T14:21:15.052 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:21:15.080 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:15.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:15.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:21:15.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:14 smithi031 ceph-mon[16336]: 1.12 scrub starts 2024-08-19T14:21:15.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:14 smithi031 ceph-mon[16336]: 1.12 scrub ok 2024-08-19T14:21:15.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:14 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/735855500' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:21:15.182 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:21:15.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:15.222 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:15.222 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:21:15.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:14 smithi032 ceph-mon[20004]: 1.12 scrub starts 2024-08-19T14:21:15.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:14 smithi032 ceph-mon[20004]: 1.12 scrub ok 2024-08-19T14:21:15.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:14 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/735855500' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:21:15.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:14 smithi045 ceph-mon[20143]: 1.12 scrub starts 2024-08-19T14:21:15.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:14 smithi045 ceph-mon[20143]: 1.12 scrub ok 2024-08-19T14:21:15.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:14 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/735855500' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:21:15.313 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:21:15.314 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-08-19T14:21:15.324 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:21:15.355 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:15.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:15.356 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:21:15.457 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:21:15.484 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:15.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:15.485 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:21:15.586 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:21:15.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:15.609 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:15.609 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:21:15.662 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-19T14:21:15.710 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:21:15.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:15.734 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:15.734 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:21:15.756 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:21:15.835 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:21:15.860 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:15.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:15.861 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:21:15.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:15 smithi031 ceph-mon[16336]: pgmap v798: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 4.4 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 64 MiB/s wr, 1.03k op/s; 4104/214725 objects misplaced (1.911%); 384 KiB/s, 5 objects/s recovering 2024-08-19T14:21:15.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:15 smithi031 ceph-mon[16336]: 1.f scrub starts 2024-08-19T14:21:15.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:15 smithi031 ceph-mon[16336]: 1.f scrub ok 2024-08-19T14:21:15.962 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:21:16.000 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:16.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:16.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:21:16.048 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 42 16 78458 78442 116.719 60.125 0.00419178 0.00856301 2024-08-19T14:21:16.101 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:21:16.136 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:16.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:16.137 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:21:16.238 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:21:16.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:15 smithi032 ceph-mon[20004]: pgmap v798: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 4.4 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 64 MiB/s wr, 1.03k op/s; 4104/214725 objects misplaced (1.911%); 384 KiB/s, 5 objects/s recovering 2024-08-19T14:21:16.259 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:15 smithi032 ceph-mon[20004]: 1.f scrub starts 2024-08-19T14:21:16.259 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:15 smithi032 ceph-mon[20004]: 1.f scrub ok 2024-08-19T14:21:16.267 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:16.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:16.268 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:21:16.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:15 smithi045 ceph-mon[20143]: pgmap v798: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 4.4 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 64 MiB/s wr, 1.03k op/s; 4104/214725 objects misplaced (1.911%); 384 KiB/s, 5 objects/s recovering 2024-08-19T14:21:16.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:15 smithi045 ceph-mon[20143]: 1.f scrub starts 2024-08-19T14:21:16.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:15 smithi045 ceph-mon[20143]: 1.f scrub ok 2024-08-19T14:21:16.369 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:21:16.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:16.397 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:16.398 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:21:16.498 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:21:16.525 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:16.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:16.526 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:21:16.627 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:21:16.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:16.657 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:16.657 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:21:16.758 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:21:16.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:16.780 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:16.780 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:21:16.881 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:21:16.909 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:16.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:16.910 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:21:17.011 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:21:17.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:17.040 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:17.041 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:21:17.048 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 43 16 79681 79665 115.782 76.4375 0.0143229 0.00863245 2024-08-19T14:21:17.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:16 smithi031 ceph-mon[16336]: from='client.25414 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:21:17.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:16 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:21:17.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:16 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:21:17.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:16 smithi031 ceph-mon[16336]: osdmap e359: 12 total, 12 up, 9 in 2024-08-19T14:21:17.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:16 smithi031 ceph-mon[16336]: 1.11 deep-scrub starts 2024-08-19T14:21:17.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:16 smithi031 ceph-mon[16336]: 1.11 deep-scrub ok 2024-08-19T14:21:17.141 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:21:17.166 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:17.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:17.167 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:21:17.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:16 smithi032 ceph-mon[20004]: from='client.25414 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:21:17.403 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:16 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:21:17.404 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:16 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:21:17.404 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:16 smithi032 ceph-mon[20004]: osdmap e359: 12 total, 12 up, 9 in 2024-08-19T14:21:17.404 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:16 smithi032 ceph-mon[20004]: 1.11 deep-scrub starts 2024-08-19T14:21:17.404 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:16 smithi032 ceph-mon[20004]: 1.11 deep-scrub ok 2024-08-19T14:21:17.404 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:21:17.405 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:16 smithi045 ceph-mon[20143]: from='client.25414 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:21:17.405 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:16 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:21:17.405 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:16 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:21:17.405 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:16 smithi045 ceph-mon[20143]: osdmap e359: 12 total, 12 up, 9 in 2024-08-19T14:21:17.406 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:16 smithi045 ceph-mon[20143]: 1.11 deep-scrub starts 2024-08-19T14:21:17.406 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:16 smithi045 ceph-mon[20143]: 1.11 deep-scrub ok 2024-08-19T14:21:17.441 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:17.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:17.443 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:21:17.544 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:21:17.581 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:17.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:17.582 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:21:17.683 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:21:17.713 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:17.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:17.714 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:21:17.815 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:21:17.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:17.855 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:17.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:21:17.956 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:21:17.983 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:17.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:17.984 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:21:18.048 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 44 16 81134 81118 115.214 90.8125 0.0138998 0.00867466 2024-08-19T14:21:18.085 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:21:18.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:18.115 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:18.116 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:21:18.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:17 smithi031 ceph-mon[16336]: pgmap v800: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 4.7 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 89 MiB/s wr, 1.42k op/s; 3989/229251 objects misplaced (1.740%); 1.2 MiB/s, 19 objects/s recovering 2024-08-19T14:21:18.216 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:21:18.244 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:18.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:18.245 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:21:18.253 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":798,"stamp":"2024-08-19T14:21:13.891273+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4691198496,"num_objects":71575,"num_object_clones":0,"num_object_copies":214725,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4104,"num_objects_unfound":0,"num_objects_dirty":71575,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71573,"num_write_kb":4580672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":2359296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75882,"ondisk_log_size":75882,"up":141,"acting":140,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":142,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":17339788,"kb_used_data":13735584,"kb_used_omap":14,"kb_used_meta":3604081,"kb_avail":826366580,"statfs":{"total":863955320832,"available":846199377920,"internally_reserved":0,"allocated":14065238016,"data_stored":14050377052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14353,"internal_metadata":3690579951},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":405929984,"num_objects":6194,"num_object_clones":0,"num_object_copies":18582,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4104,"num_objects_unfound":0,"num_objects_dirty":6194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":396416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":2359296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.003077"},"pg_stats":[{"pgid":"4.5","version":"356'4530","reported_seq":4582,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-08-19T14:21:10.777632+0000","last_change":"2024-08-19T14:20:29.268214+0000","last_active":"2024-08-19T14:21:10.777632+0000","last_peered":"2024-08-19T14:21:10.777632+0000","last_clean":"2024-08-19T14:21:10.777632+0000","last_became_active":"2024-08-19T14:20:29.267549+0000","last_became_peered":"2024-08-19T14:20:29.267549+0000","last_unstale":"2024-08-19T14:21:10.777632+0000","last_undegraded":"2024-08-19T14:21:10.777632+0000","last_fullsized":"2024-08-19T14:21:10.777632+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":4530,"log_dups_size":0,"ondisk_log_size":4530,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:57.640230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":296878080,"num_objects":4530,"num_object_clones":0,"num_object_copies":13590,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4530,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4530,"num_write_kb":289920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,0],"acting":[11,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"70'139","reported_seq":932,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-08-19T14:21:12.837871+0000","last_change":"2024-08-19T14:19:58.072227+0000","last_active":"2024-08-19T14:21:12.837871+0000","last_peered":"2024-08-19T14:21:12.837871+0000","last_clean":"2024-08-19T14:21:12.837871+0000","last_became_active":"2024-08-19T14:18:13.807133+0000","last_became_peered":"2024-08-19T14:18:13.807133+0000","last_unstale":"2024-08-19T14:21:12.837871+0000","last_undegraded":"2024-08-19T14:21:12.837871+0000","last_fullsized":"2024-08-19T14:21:12.837871+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:19:58.072083+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:19:58.072083+0000","last_clean_scrub_stamp":"2024-08-19T14:19:58.072083+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:17.231298+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,9],"acting":[7,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"355'4307","reported_seq":4431,"reported_epoch":355,"state":"active+recovering+undersized+remapped","last_fresh":"2024-08-19T14:21:08.973120+0000","last_change":"2024-08-19T14:21:07.239135+0000","last_active":"2024-08-19T14:21:08.973120+0000","last_peered":"2024-08-19T14:21:08.973120+0000","last_clean":"2024-08-19T14:21:04.784061+0000","last_became_active":"2024-08-19T14:21:07.213050+0000","last_became_peered":"2024-08-19T14:21:07.213050+0000","last_unstale":"2024-08-19T14:21:08.973120+0000","last_undegraded":"2024-08-19T14:21:08.973120+0000","last_fullsized":"2024-08-19T14:21:06.805412+0000","mapping_epoch":354,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":4307,"log_dups_size":0,"ondisk_log_size":4307,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282263552,"num_objects":4307,"num_object_clones":0,"num_object_copies":12921,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4104,"num_objects_unfound":0,"num_objects_dirty":4307,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4307,"num_write_kb":275648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":2359296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,7],"acting":[1,7],"avail_no_missing":["1","5","7"],"object_location_counts":[{"shards":"1,5,7","objects":4104},{"shards":"1,7","objects":203}],"blocked_by":[],"up_primary":9,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"70'139","reported_seq":824,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-08-19T14:21:06.805331+0000","last_change":"2024-08-19T14:20:03.714917+0000","last_active":"2024-08-19T14:21:06.805331+0000","last_peered":"2024-08-19T14:21:06.805331+0000","last_clean":"2024-08-19T14:21:06.805331+0000","last_became_active":"2024-08-19T14:18:13.804721+0000","last_became_peered":"2024-08-19T14:18:13.804721+0000","last_unstale":"2024-08-19T14:21:06.805331+0000","last_undegraded":"2024-08-19T14:21:06.805331+0000","last_fullsized":"2024-08-19T14:21:06.805331+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:03.714818+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:03.714818+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:29.563399+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"355'4359","reported_seq":4409,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-08-19T14:21:08.970339+0000","last_change":"2024-08-19T14:20:29.268110+0000","last_active":"2024-08-19T14:21:08.970339+0000","last_peered":"2024-08-19T14:21:08.970339+0000","last_clean":"2024-08-19T14:21:08.970339+0000","last_became_active":"2024-08-19T14:20:29.267811+0000","last_became_peered":"2024-08-19T14:20:29.267811+0000","last_unstale":"2024-08-19T14:21:08.970339+0000","last_undegraded":"2024-08-19T14:21:08.970339+0000","last_fullsized":"2024-08-19T14:21:08.970339+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":4359,"log_dups_size":0,"ondisk_log_size":4359,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:46.516076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":285671424,"num_objects":4359,"num_object_clones":0,"num_object_copies":13077,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4359,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4359,"num_write_kb":278976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"70'139","reported_seq":779,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-08-19T14:21:12.837586+0000","last_change":"2024-08-19T14:20:50.742676+0000","last_active":"2024-08-19T14:21:12.837586+0000","last_peered":"2024-08-19T14:21:12.837586+0000","last_clean":"2024-08-19T14:21:12.837586+0000","last_became_active":"2024-08-19T14:20:50.742354+0000","last_became_peered":"2024-08-19T14:20:50.742354+0000","last_unstale":"2024-08-19T14:21:12.837586+0000","last_undegraded":"2024-08-19T14:21:12.837586+0000","last_fullsized":"2024-08-19T14:21:12.837586+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:19:59.302931+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:19:59.302931+0000","last_clean_scrub_stamp":"2024-08-19T14:19:59.302931+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:18.923389+0000","scrub_duration":114,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"355'4336","reported_seq":4386,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-08-19T14:21:08.932094+0000","last_change":"2024-08-19T14:20:29.267640+0000","last_active":"2024-08-19T14:21:08.932094+0000","last_peered":"2024-08-19T14:21:08.932094+0000","last_clean":"2024-08-19T14:21:08.932094+0000","last_became_active":"2024-08-19T14:20:29.267331+0000","last_became_peered":"2024-08-19T14:20:29.267331+0000","last_unstale":"2024-08-19T14:21:08.932094+0000","last_undegraded":"2024-08-19T14:21:08.932094+0000","last_fullsized":"2024-08-19T14:21:08.932094+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":4336,"log_dups_size":0,"ondisk_log_size":4336,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:46.504191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284164096,"num_objects":4336,"num_object_clones":0,"num_object_copies":13008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4336,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4336,"num_write_kb":277504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"70'139","reported_seq":827,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-08-19T14:21:12.837512+0000","last_change":"2024-08-19T14:20:09.507422+0000","last_active":"2024-08-19T14:21:12.837512+0000","last_peered":"2024-08-19T14:21:12.837512+0000","last_clean":"2024-08-19T14:21:12.837512+0000","last_became_active":"2024-08-19T14:18:13.808505+0000","last_became_peered":"2024-08-19T14:18:13.808505+0000","last_unstale":"2024-08-19T14:21:12.837512+0000","last_undegraded":"2024-08-19T14:21:12.837512+0000","last_fullsized":"2024-08-19T14:21:12.837512+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:09.507256+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:09.507256+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:29.309519+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,11],"acting":[9,0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"358'4731","reported_seq":4787,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-08-19T14:21:13.377053+0000","last_change":"2024-08-19T14:20:29.268766+0000","last_active":"2024-08-19T14:21:13.377053+0000","last_peered":"2024-08-19T14:21:13.377053+0000","last_clean":"2024-08-19T14:21:13.377053+0000","last_became_active":"2024-08-19T14:20:29.268443+0000","last_became_peered":"2024-08-19T14:20:29.268443+0000","last_unstale":"2024-08-19T14:21:13.377053+0000","last_undegraded":"2024-08-19T14:21:13.377053+0000","last_fullsized":"2024-08-19T14:21:13.377053+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":4731,"log_dups_size":0,"ondisk_log_size":4731,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:39.688611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":310050816,"num_objects":4731,"num_object_clones":0,"num_object_copies":14193,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4731,"num_write_kb":302784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,11],"acting":[9,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.7","version":"70'139","reported_seq":465,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-08-19T14:21:06.951335+0000","last_change":"2024-08-19T14:20:34.937317+0000","last_active":"2024-08-19T14:21:06.951335+0000","last_peered":"2024-08-19T14:21:06.951335+0000","last_clean":"2024-08-19T14:21:06.951335+0000","last_became_active":"2024-08-19T14:20:28.204384+0000","last_became_peered":"2024-08-19T14:20:28.204384+0000","last_unstale":"2024-08-19T14:21:06.951335+0000","last_undegraded":"2024-08-19T14:21:06.951335+0000","last_fullsized":"2024-08-19T14:21:06.951335+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:34.937192+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:20:34.937192+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:40.330609+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"356'4461","reported_seq":4513,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-08-19T14:21:10.772220+0000","last_change":"2024-08-19T14:20:29.267833+0000","last_active":"2024-08-19T14:21:10.772220+0000","last_peered":"2024-08-19T14:21:10.772220+0000","last_clean":"2024-08-19T14:21:10.772220+0000","last_became_active":"2024-08-19T14:20:29.267379+0000","last_became_peered":"2024-08-19T14:20:29.267379+0000","last_unstale":"2024-08-19T14:21:10.772220+0000","last_undegraded":"2024-08-19T14:21:10.772220+0000","last_fullsized":"2024-08-19T14:21:10.772220+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":4461,"log_dups_size":0,"ondisk_log_size":4461,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:28.819534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":292356096,"num_objects":4461,"num_object_clones":0,"num_object_copies":13383,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4461,"num_write_kb":285504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,1],"acting":[11,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.6","version":"70'139","reported_seq":831,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-08-19T14:21:12.839238+0000","last_change":"2024-08-19T14:19:55.512078+0000","last_active":"2024-08-19T14:21:12.839238+0000","last_peered":"2024-08-19T14:21:12.839238+0000","last_clean":"2024-08-19T14:21:12.839238+0000","last_became_active":"2024-08-19T14:18:26.874902+0000","last_became_peered":"2024-08-19T14:18:26.874902+0000","last_unstale":"2024-08-19T14:21:12.839238+0000","last_undegraded":"2024-08-19T14:21:12.839238+0000","last_fullsized":"2024-08-19T14:21:12.839238+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:19:55.511974+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:27.848870+0000","last_clean_scrub_stamp":"2024-08-19T14:19:55.511974+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:16.706651+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,4],"acting":[9,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"355'4339","reported_seq":4389,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-08-19T14:21:09.313131+0000","last_change":"2024-08-19T14:20:29.267682+0000","last_active":"2024-08-19T14:21:09.313131+0000","last_peered":"2024-08-19T14:21:09.313131+0000","last_clean":"2024-08-19T14:21:09.313131+0000","last_became_active":"2024-08-19T14:20:29.267360+0000","last_became_peered":"2024-08-19T14:20:29.267360+0000","last_unstale":"2024-08-19T14:21:09.313131+0000","last_undegraded":"2024-08-19T14:21:09.313131+0000","last_fullsized":"2024-08-19T14:21:09.313131+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":4339,"log_dups_size":0,"ondisk_log_size":4339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:51.551392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284360704,"num_objects":4339,"num_object_clones":0,"num_object_copies":13017,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4339,"num_write_kb":277696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"70'139","reported_seq":641,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-08-19T14:21:09.826426+0000","last_change":"2024-08-19T14:20:12.731993+0000","last_active":"2024-08-19T14:21:09.826426+0000","last_peered":"2024-08-19T14:21:09.826426+0000","last_clean":"2024-08-19T14:21:09.826426+0000","last_became_active":"2024-08-19T14:18:26.881435+0000","last_became_peered":"2024-08-19T14:18:26.881435+0000","last_unstale":"2024-08-19T14:21:09.826426+0000","last_undegraded":"2024-08-19T14:21:09.826426+0000","last_fullsized":"2024-08-19T14:21:09.826426+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:12.731850+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:12.731850+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:37.351039+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,4],"acting":[11,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.1","version":"356'4530","reported_seq":4582,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-08-19T14:21:10.775812+0000","last_change":"2024-08-19T14:20:29.268808+0000","last_active":"2024-08-19T14:21:10.775812+0000","last_peered":"2024-08-19T14:21:10.775812+0000","last_clean":"2024-08-19T14:21:10.775812+0000","last_became_active":"2024-08-19T14:20:29.268465+0000","last_became_peered":"2024-08-19T14:20:29.268465+0000","last_unstale":"2024-08-19T14:21:10.775812+0000","last_undegraded":"2024-08-19T14:21:10.775812+0000","last_fullsized":"2024-08-19T14:21:10.775812+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":4530,"log_dups_size":0,"ondisk_log_size":4530,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:57.700841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":296878080,"num_objects":4530,"num_object_clones":0,"num_object_copies":13590,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4530,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4530,"num_write_kb":289920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.4","version":"70'139","reported_seq":865,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-08-19T14:21:09.826277+0000","last_change":"2024-08-19T14:20:04.729769+0000","last_active":"2024-08-19T14:21:09.826277+0000","last_peered":"2024-08-19T14:21:09.826277+0000","last_clean":"2024-08-19T14:21:09.826277+0000","last_became_active":"2024-08-19T14:18:13.806013+0000","last_became_peered":"2024-08-19T14:18:13.806013+0000","last_unstale":"2024-08-19T14:21:09.826277+0000","last_undegraded":"2024-08-19T14:21:09.826277+0000","last_fullsized":"2024-08-19T14:21:09.826277+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:04.729563+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:04.729563+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:26.771562+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.f","version":"355'4254","reported_seq":4304,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-08-19T14:21:08.955048+0000","last_change":"2024-08-19T14:20:29.212298+0000","last_active":"2024-08-19T14:21:08.955048+0000","last_peered":"2024-08-19T14:21:08.955048+0000","last_clean":"2024-08-19T14:21:08.955048+0000","last_became_active":"2024-08-19T14:20:29.212003+0000","last_became_peered":"2024-08-19T14:20:29.212003+0000","last_unstale":"2024-08-19T14:21:08.955048+0000","last_undegraded":"2024-08-19T14:21:08.955048+0000","last_fullsized":"2024-08-19T14:21:08.955048+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":4254,"log_dups_size":0,"ondisk_log_size":4254,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:52.621283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":278790144,"num_objects":4254,"num_object_clones":0,"num_object_copies":12762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4254,"num_write_kb":272256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,11],"acting":[1,3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"70'139","reported_seq":839,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-08-19T14:21:12.837491+0000","last_change":"2024-08-19T14:20:12.570453+0000","last_active":"2024-08-19T14:21:12.837491+0000","last_peered":"2024-08-19T14:21:12.837491+0000","last_clean":"2024-08-19T14:21:12.837491+0000","last_became_active":"2024-08-19T14:18:26.875006+0000","last_became_peered":"2024-08-19T14:18:26.875006+0000","last_unstale":"2024-08-19T14:21:12.837491+0000","last_undegraded":"2024-08-19T14:21:12.837491+0000","last_fullsized":"2024-08-19T14:21:12.837491+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:12.570352+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:28.864870+0000","last_clean_scrub_stamp":"2024-08-19T14:20:12.570352+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:32.193930+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,0],"acting":[9,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.c","version":"358'4647","reported_seq":4703,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-08-19T14:21:13.141958+0000","last_change":"2024-08-19T14:20:29.266724+0000","last_active":"2024-08-19T14:21:13.141958+0000","last_peered":"2024-08-19T14:21:13.141958+0000","last_clean":"2024-08-19T14:21:13.141958+0000","last_became_active":"2024-08-19T14:20:29.266373+0000","last_became_peered":"2024-08-19T14:20:29.266373+0000","last_unstale":"2024-08-19T14:21:13.141958+0000","last_undegraded":"2024-08-19T14:21:13.141958+0000","last_fullsized":"2024-08-19T14:21:13.141958+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":4647,"log_dups_size":0,"ondisk_log_size":4647,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:48.006455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":304545792,"num_objects":4647,"num_object_clones":0,"num_object_copies":13941,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4647,"num_write_kb":297408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,8],"acting":[4,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"70'139","reported_seq":954,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-08-19T14:21:12.837566+0000","last_change":"2024-08-19T14:19:56.106047+0000","last_active":"2024-08-19T14:21:12.837566+0000","last_peered":"2024-08-19T14:21:12.837566+0000","last_clean":"2024-08-19T14:21:12.837566+0000","last_became_active":"2024-08-19T14:18:26.887657+0000","last_became_peered":"2024-08-19T14:18:26.887657+0000","last_unstale":"2024-08-19T14:21:12.837566+0000","last_undegraded":"2024-08-19T14:21:12.837566+0000","last_fullsized":"2024-08-19T14:21:12.837566+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:19:56.105880+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:19:56.105880+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:22.062694+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"358'4639","reported_seq":4695,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-08-19T14:21:13.119464+0000","last_change":"2024-08-19T14:20:29.208992+0000","last_active":"2024-08-19T14:21:13.119464+0000","last_peered":"2024-08-19T14:21:13.119464+0000","last_clean":"2024-08-19T14:21:13.119464+0000","last_became_active":"2024-08-19T14:20:29.208599+0000","last_became_peered":"2024-08-19T14:20:29.208599+0000","last_unstale":"2024-08-19T14:21:13.119464+0000","last_undegraded":"2024-08-19T14:21:13.119464+0000","last_fullsized":"2024-08-19T14:21:13.119464+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":4639,"log_dups_size":0,"ondisk_log_size":4639,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:55.930633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":304021504,"num_objects":4639,"num_object_clones":0,"num_object_copies":13917,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4639,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4639,"num_write_kb":296896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,1],"acting":[4,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"70'139","reported_seq":841,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-08-19T14:21:10.834672+0000","last_change":"2024-08-19T14:20:11.390093+0000","last_active":"2024-08-19T14:21:10.834672+0000","last_peered":"2024-08-19T14:21:10.834672+0000","last_clean":"2024-08-19T14:21:10.834672+0000","last_became_active":"2024-08-19T14:18:26.887277+0000","last_became_peered":"2024-08-19T14:18:26.887277+0000","last_unstale":"2024-08-19T14:21:10.834672+0000","last_undegraded":"2024-08-19T14:21:10.834672+0000","last_fullsized":"2024-08-19T14:21:10.834672+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:11.389938+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:11.389938+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:21.144007+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,8],"acting":[5,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"358'4637","reported_seq":4693,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-08-19T14:21:13.129442+0000","last_change":"2024-08-19T14:20:29.209096+0000","last_active":"2024-08-19T14:21:13.129442+0000","last_peered":"2024-08-19T14:21:13.129442+0000","last_clean":"2024-08-19T14:21:13.129442+0000","last_became_active":"2024-08-19T14:20:29.208748+0000","last_became_peered":"2024-08-19T14:20:29.208748+0000","last_unstale":"2024-08-19T14:21:13.129442+0000","last_undegraded":"2024-08-19T14:21:13.129442+0000","last_fullsized":"2024-08-19T14:21:13.129442+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:56.597251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":303890432,"num_objects":4637,"num_object_clones":0,"num_object_copies":13911,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4637,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":296768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"70'139","reported_seq":841,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-08-19T14:21:09.826365+0000","last_change":"2024-08-19T14:20:11.733961+0000","last_active":"2024-08-19T14:21:09.826365+0000","last_peered":"2024-08-19T14:21:09.826365+0000","last_clean":"2024-08-19T14:21:09.826365+0000","last_became_active":"2024-08-19T14:18:26.875179+0000","last_became_peered":"2024-08-19T14:18:26.875179+0000","last_unstale":"2024-08-19T14:21:09.826365+0000","last_undegraded":"2024-08-19T14:21:09.826365+0000","last_fullsized":"2024-08-19T14:21:09.826365+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:11.733865+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:11.733865+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:41.250361+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.b","version":"357'4469","reported_seq":4523,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-08-19T14:21:11.170977+0000","last_change":"2024-08-19T14:20:29.207845+0000","last_active":"2024-08-19T14:21:11.170977+0000","last_peered":"2024-08-19T14:21:11.170977+0000","last_clean":"2024-08-19T14:21:11.170977+0000","last_became_active":"2024-08-19T14:20:29.207372+0000","last_became_peered":"2024-08-19T14:20:29.207372+0000","last_unstale":"2024-08-19T14:21:11.170977+0000","last_undegraded":"2024-08-19T14:21:11.170977+0000","last_fullsized":"2024-08-19T14:21:11.170977+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":4469,"log_dups_size":0,"ondisk_log_size":4469,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:33.577033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":292880384,"num_objects":4469,"num_object_clones":0,"num_object_copies":13407,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4469,"num_write_kb":286016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"70'139","reported_seq":499,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-08-19T14:21:12.833857+0000","last_change":"2024-08-19T14:20:15.322051+0000","last_active":"2024-08-19T14:21:12.833857+0000","last_peered":"2024-08-19T14:21:12.833857+0000","last_clean":"2024-08-19T14:21:12.833857+0000","last_became_active":"2024-08-19T14:18:26.884105+0000","last_became_peered":"2024-08-19T14:18:26.884105+0000","last_unstale":"2024-08-19T14:21:12.833857+0000","last_undegraded":"2024-08-19T14:21:12.833857+0000","last_fullsized":"2024-08-19T14:21:12.833857+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:15.321945+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:20:15.321945+0000","last_clean_scrub_stamp":"2024-08-19T14:20:15.321945+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:43.904541+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"355'4415","reported_seq":4465,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-08-19T14:21:08.962634+0000","last_change":"2024-08-19T14:20:29.269723+0000","last_active":"2024-08-19T14:21:08.962634+0000","last_peered":"2024-08-19T14:21:08.962634+0000","last_clean":"2024-08-19T14:21:08.962634+0000","last_became_active":"2024-08-19T14:20:29.269432+0000","last_became_peered":"2024-08-19T14:20:29.269432+0000","last_unstale":"2024-08-19T14:21:08.962634+0000","last_undegraded":"2024-08-19T14:21:08.962634+0000","last_fullsized":"2024-08-19T14:21:08.962634+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":4415,"log_dups_size":0,"ondisk_log_size":4415,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:44.639451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":289341440,"num_objects":4415,"num_object_clones":0,"num_object_copies":13245,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4415,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4415,"num_write_kb":282560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.f","version":"70'139","reported_seq":741,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-08-19T14:21:10.834873+0000","last_change":"2024-08-19T14:20:02.301201+0000","last_active":"2024-08-19T14:21:10.834873+0000","last_peered":"2024-08-19T14:21:10.834873+0000","last_clean":"2024-08-19T14:21:10.834873+0000","last_became_active":"2024-08-19T14:18:18.711708+0000","last_became_peered":"2024-08-19T14:18:18.711708+0000","last_unstale":"2024-08-19T14:21:10.834873+0000","last_undegraded":"2024-08-19T14:21:10.834873+0000","last_fullsized":"2024-08-19T14:21:10.834873+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:02.301096+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:20:02.301096+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"356'4479","reported_seq":4531,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-08-19T14:21:10.771261+0000","last_change":"2024-08-19T14:20:29.212320+0000","last_active":"2024-08-19T14:21:10.771261+0000","last_peered":"2024-08-19T14:21:10.771261+0000","last_clean":"2024-08-19T14:21:10.771261+0000","last_became_active":"2024-08-19T14:20:29.211975+0000","last_became_peered":"2024-08-19T14:20:29.211975+0000","last_unstale":"2024-08-19T14:21:10.771261+0000","last_undegraded":"2024-08-19T14:21:10.771261+0000","last_fullsized":"2024-08-19T14:21:10.771261+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":4479,"log_dups_size":0,"ondisk_log_size":4479,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:28.509656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":293535744,"num_objects":4479,"num_object_clones":0,"num_object_copies":13437,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4479,"num_write_kb":286656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,3],"acting":[11,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.c","version":"70'139","reported_seq":598,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-08-19T14:21:06.951224+0000","last_change":"2024-08-19T14:20:07.839218+0000","last_active":"2024-08-19T14:21:06.951224+0000","last_peered":"2024-08-19T14:21:06.951224+0000","last_clean":"2024-08-19T14:21:06.951224+0000","last_became_active":"2024-08-19T14:18:26.872942+0000","last_became_peered":"2024-08-19T14:18:26.872942+0000","last_unstale":"2024-08-19T14:21:06.951224+0000","last_undegraded":"2024-08-19T14:21:06.951224+0000","last_fullsized":"2024-08-19T14:21:06.951224+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:07.839076+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:20:07.839076+0000","last_clean_scrub_stamp":"2024-08-19T14:20:07.839076+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:33.185668+0000","scrub_duration":62,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,9],"acting":[3,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"356'4440","reported_seq":4492,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-08-19T14:21:10.768937+0000","last_change":"2024-08-19T14:20:29.268318+0000","last_active":"2024-08-19T14:21:10.768937+0000","last_peered":"2024-08-19T14:21:10.768937+0000","last_clean":"2024-08-19T14:21:10.768937+0000","last_became_active":"2024-08-19T14:20:29.267442+0000","last_became_peered":"2024-08-19T14:20:29.267442+0000","last_unstale":"2024-08-19T14:21:10.768937+0000","last_undegraded":"2024-08-19T14:21:10.768937+0000","last_fullsized":"2024-08-19T14:21:10.768937+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":4440,"log_dups_size":0,"ondisk_log_size":4440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:47.715801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":290979840,"num_objects":4440,"num_object_clones":0,"num_object_copies":13320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4440,"num_write_kb":284160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,9],"acting":[11,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.d","version":"70'139","reported_seq":527,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-08-19T14:21:12.838101+0000","last_change":"2024-08-19T14:21:07.155294+0000","last_active":"2024-08-19T14:21:12.838101+0000","last_peered":"2024-08-19T14:21:12.838101+0000","last_clean":"2024-08-19T14:21:12.838101+0000","last_became_active":"2024-08-19T14:18:26.882002+0000","last_became_peered":"2024-08-19T14:18:26.882002+0000","last_unstale":"2024-08-19T14:21:12.838101+0000","last_undegraded":"2024-08-19T14:21:12.838101+0000","last_fullsized":"2024-08-19T14:21:12.838101+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:07.155157+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:21:07.155157+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:22:11.274708+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"70'139","reported_seq":776,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-08-19T14:21:10.836045+0000","last_change":"2024-08-19T14:19:52.410055+0000","last_active":"2024-08-19T14:21:10.836045+0000","last_peered":"2024-08-19T14:21:10.836045+0000","last_clean":"2024-08-19T14:21:10.836045+0000","last_became_active":"2024-08-19T14:18:18.703594+0000","last_became_peered":"2024-08-19T14:18:18.703594+0000","last_unstale":"2024-08-19T14:21:10.836045+0000","last_undegraded":"2024-08-19T14:21:10.836045+0000","last_fullsized":"2024-08-19T14:21:10.836045+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:19:52.409915+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:19:52.409915+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"70'139","reported_seq":832,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-08-19T14:21:06.805230+0000","last_change":"2024-08-19T14:20:07.692054+0000","last_active":"2024-08-19T14:21:06.805230+0000","last_peered":"2024-08-19T14:21:06.805230+0000","last_clean":"2024-08-19T14:21:06.805230+0000","last_became_active":"2024-08-19T14:18:18.698334+0000","last_became_peered":"2024-08-19T14:18:18.698334+0000","last_unstale":"2024-08-19T14:21:06.805230+0000","last_undegraded":"2024-08-19T14:21:06.805230+0000","last_fullsized":"2024-08-19T14:21:06.805230+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:07.691932+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:07.691932+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:36.578885+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.10","version":"70'139","reported_seq":939,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-08-19T14:21:10.834619+0000","last_change":"2024-08-19T14:21:07.768770+0000","last_active":"2024-08-19T14:21:10.834619+0000","last_peered":"2024-08-19T14:21:10.834619+0000","last_clean":"2024-08-19T14:21:10.834619+0000","last_became_active":"2024-08-19T14:18:18.704562+0000","last_became_peered":"2024-08-19T14:18:18.704562+0000","last_unstale":"2024-08-19T14:21:10.834619+0000","last_undegraded":"2024-08-19T14:21:10.834619+0000","last_fullsized":"2024-08-19T14:21:10.834619+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:07.768697+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:21:07.768697+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:22:10.360289+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,8],"acting":[5,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"70'139","reported_seq":480,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-08-19T14:21:10.838815+0000","last_change":"2024-08-19T14:20:06.002140+0000","last_active":"2024-08-19T14:21:10.838815+0000","last_peered":"2024-08-19T14:21:10.838815+0000","last_clean":"2024-08-19T14:21:10.838815+0000","last_became_active":"2024-08-19T14:18:18.697894+0000","last_became_peered":"2024-08-19T14:18:18.697894+0000","last_unstale":"2024-08-19T14:21:10.838815+0000","last_undegraded":"2024-08-19T14:21:10.838815+0000","last_fullsized":"2024-08-19T14:21:10.838815+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:06.001905+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:06.001905+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:13.776545+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"70'139","reported_seq":461,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-08-19T14:21:06.805542+0000","last_change":"2024-08-19T14:20:28.205212+0000","last_active":"2024-08-19T14:21:06.805542+0000","last_peered":"2024-08-19T14:21:06.805542+0000","last_clean":"2024-08-19T14:21:06.805542+0000","last_became_active":"2024-08-19T14:20:28.204724+0000","last_became_peered":"2024-08-19T14:20:28.204724+0000","last_unstale":"2024-08-19T14:21:06.805542+0000","last_undegraded":"2024-08-19T14:21:06.805542+0000","last_fullsized":"2024-08-19T14:21:06.805542+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:08.680021+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:08.680021+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:28.441081+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"70'139","reported_seq":583,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-08-19T14:21:09.854393+0000","last_change":"2024-08-19T14:20:31.931658+0000","last_active":"2024-08-19T14:21:09.854393+0000","last_peered":"2024-08-19T14:21:09.854393+0000","last_clean":"2024-08-19T14:21:09.854393+0000","last_became_active":"2024-08-19T14:18:31.901556+0000","last_became_peered":"2024-08-19T14:18:31.901556+0000","last_unstale":"2024-08-19T14:21:09.854393+0000","last_undegraded":"2024-08-19T14:21:09.854393+0000","last_fullsized":"2024-08-19T14:21:09.854393+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:31.931562+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:20:31.931562+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:47.687921+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,0],"acting":[8,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.14","version":"70'139","reported_seq":596,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-08-19T14:21:12.838043+0000","last_change":"2024-08-19T14:19:55.130518+0000","last_active":"2024-08-19T14:21:12.838043+0000","last_peered":"2024-08-19T14:21:12.838043+0000","last_clean":"2024-08-19T14:21:12.838043+0000","last_became_active":"2024-08-19T14:18:18.703926+0000","last_became_peered":"2024-08-19T14:18:18.703926+0000","last_unstale":"2024-08-19T14:21:12.838043+0000","last_undegraded":"2024-08-19T14:21:12.838043+0000","last_fullsized":"2024-08-19T14:21:12.838043+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:19:55.130420+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:19:55.130420+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:12.906155+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,1],"acting":[7,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.15","version":"70'139","reported_seq":483,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-08-19T14:21:12.838072+0000","last_change":"2024-08-19T14:20:09.181046+0000","last_active":"2024-08-19T14:21:12.838072+0000","last_peered":"2024-08-19T14:21:12.838072+0000","last_clean":"2024-08-19T14:21:12.838072+0000","last_became_active":"2024-08-19T14:18:31.912496+0000","last_became_peered":"2024-08-19T14:18:31.912496+0000","last_unstale":"2024-08-19T14:21:12.838072+0000","last_undegraded":"2024-08-19T14:21:12.838072+0000","last_fullsized":"2024-08-19T14:21:12.838072+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:09.180923+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:20:09.180923+0000","last_clean_scrub_stamp":"2024-08-19T14:20:09.180923+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:21.100428+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"70'139","reported_seq":461,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-08-19T14:21:12.833534+0000","last_change":"2024-08-19T14:20:17.333347+0000","last_active":"2024-08-19T14:21:12.833534+0000","last_peered":"2024-08-19T14:21:12.833534+0000","last_clean":"2024-08-19T14:21:12.833534+0000","last_became_active":"2024-08-19T14:18:31.901229+0000","last_became_peered":"2024-08-19T14:18:31.901229+0000","last_unstale":"2024-08-19T14:21:12.833534+0000","last_undegraded":"2024-08-19T14:21:12.833534+0000","last_fullsized":"2024-08-19T14:21:12.833534+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:17.333260+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:17.333260+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:23.889383+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1b","version":"70'139","reported_seq":809,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-08-19T14:21:09.854224+0000","last_change":"2024-08-19T14:20:16.776086+0000","last_active":"2024-08-19T14:21:09.854224+0000","last_peered":"2024-08-19T14:21:09.854224+0000","last_clean":"2024-08-19T14:21:09.854224+0000","last_became_active":"2024-08-19T14:18:31.901152+0000","last_became_peered":"2024-08-19T14:18:31.901152+0000","last_unstale":"2024-08-19T14:21:09.854224+0000","last_undegraded":"2024-08-19T14:21:09.854224+0000","last_fullsized":"2024-08-19T14:21:09.854224+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:16.775987+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:16.775987+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:46.485727+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.18","version":"70'139","reported_seq":855,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-08-19T14:21:10.834383+0000","last_change":"2024-08-19T14:20:36.510554+0000","last_active":"2024-08-19T14:21:10.834383+0000","last_peered":"2024-08-19T14:21:10.834383+0000","last_clean":"2024-08-19T14:21:10.834383+0000","last_became_active":"2024-08-19T14:18:31.902735+0000","last_became_peered":"2024-08-19T14:18:31.902735+0000","last_unstale":"2024-08-19T14:21:10.834383+0000","last_undegraded":"2024-08-19T14:21:10.834383+0000","last_fullsized":"2024-08-19T14:21:10.834383+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:36.510442+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:36.510442+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:53.706735+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.19","version":"70'139","reported_seq":803,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-08-19T14:21:10.838738+0000","last_change":"2024-08-19T14:20:32.919627+0000","last_active":"2024-08-19T14:21:10.838738+0000","last_peered":"2024-08-19T14:21:10.838738+0000","last_clean":"2024-08-19T14:21:10.838738+0000","last_became_active":"2024-08-19T14:18:31.902547+0000","last_became_peered":"2024-08-19T14:18:31.902547+0000","last_unstale":"2024-08-19T14:21:10.838738+0000","last_undegraded":"2024-08-19T14:21:10.838738+0000","last_fullsized":"2024-08-19T14:21:10.838738+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:32.919545+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:20:32.919545+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:35.916636+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1e","version":"70'139","reported_seq":459,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-08-19T14:21:09.826400+0000","last_change":"2024-08-19T14:20:18.771836+0000","last_active":"2024-08-19T14:21:09.826400+0000","last_peered":"2024-08-19T14:21:09.826400+0000","last_clean":"2024-08-19T14:21:09.826400+0000","last_became_active":"2024-08-19T14:18:31.902546+0000","last_became_peered":"2024-08-19T14:18:31.902546+0000","last_unstale":"2024-08-19T14:21:09.826400+0000","last_undegraded":"2024-08-19T14:21:09.826400+0000","last_fullsized":"2024-08-19T14:21:09.826400+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:18.771704+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:18.771704+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:47.154907+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1c","version":"70'139","reported_seq":567,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-08-19T14:21:09.855465+0000","last_change":"2024-08-19T14:20:08.824736+0000","last_active":"2024-08-19T14:21:09.855465+0000","last_peered":"2024-08-19T14:21:09.855465+0000","last_clean":"2024-08-19T14:21:09.855465+0000","last_became_active":"2024-08-19T14:18:31.902494+0000","last_became_peered":"2024-08-19T14:18:31.902494+0000","last_unstale":"2024-08-19T14:21:09.855465+0000","last_undegraded":"2024-08-19T14:21:09.855465+0000","last_fullsized":"2024-08-19T14:21:09.855465+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:08.824505+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:08.824505+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:32.216640+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,11],"acting":[8,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1d","version":"70'139","reported_seq":487,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-08-19T14:21:06.805323+0000","last_change":"2024-08-19T14:20:17.725615+0000","last_active":"2024-08-19T14:21:06.805323+0000","last_peered":"2024-08-19T14:21:06.805323+0000","last_clean":"2024-08-19T14:21:06.805323+0000","last_became_active":"2024-08-19T14:18:31.901183+0000","last_became_peered":"2024-08-19T14:18:31.901183+0000","last_unstale":"2024-08-19T14:21:06.805323+0000","last_undegraded":"2024-08-19T14:21:06.805323+0000","last_fullsized":"2024-08-19T14:21:06.805323+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:17.725477+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:17.725477+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:25.545335+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,5],"acting":[1,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":4690608128,"num_objects":71573,"num_object_clones":0,"num_object_copies":214719,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4104,"num_objects_unfound":0,"num_objects_dirty":71573,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71573,"num_write_kb":4580672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":2359296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14219870208,"data_stored":14219870208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71573,"ondisk_log_size":71573,"up":48,"acting":47,"num_store_stats":9},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1802240,"data_stored":1788032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187574,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2358992,"kb_used_data":1976580,"kb_used_omap":1,"kb_used_meta":382398,"kb_avail":91386160,"statfs":{"total":95995035648,"available":93579427840,"internally_reserved":0,"allocated":2024017920,"data_stored":2022369274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":391576004},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":343597383804,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":74,"seq":317827580032,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1378076,"kb_used_data":927632,"kb_used_omap":1,"kb_used_meta":450430,"kb_avail":92367076,"statfs":{"total":95995035648,"available":94583885824,"internally_reserved":0,"allocated":949895168,"data_stored":948242346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":461240769},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874373,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":656144,"kb_used_data":282052,"kb_used_omap":1,"kb_used_meta":374078,"kb_avail":93089008,"statfs":{"total":95995035648,"available":95323144192,"internally_reserved":0,"allocated":288821248,"data_stored":287172602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":383056325},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":57,"seq":244813136008,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2845104,"kb_used_data":2412708,"kb_used_omap":1,"kb_used_meta":432382,"kb_avail":90900048,"statfs":{"total":95995035648,"available":93081649152,"internally_reserved":0,"allocated":2470612992,"data_stored":2468952392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":442759626},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":206158430347,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691984,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1475672,"kb_used_data":1137676,"kb_used_omap":1,"kb_used_meta":337982,"kb_avail":92269480,"statfs":{"total":95995035648,"available":94483947520,"internally_reserved":0,"allocated":1164980224,"data_stored":1163328026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":346094021},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986325,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1926812,"kb_used_data":1472208,"kb_used_omap":1,"kb_used_meta":454590,"kb_avail":91818340,"statfs":{"total":95995035648,"available":94021980160,"internally_reserved":0,"allocated":1507540992,"data_stored":1505888170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":465500613},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149850,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2059904,"kb_used_data":1653880,"kb_used_omap":1,"kb_used_meta":406014,"kb_avail":91685248,"statfs":{"total":95995035648,"available":93885693952,"internally_reserved":0,"allocated":1693573120,"data_stored":1691928692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":415758791},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411488,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575011,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3142864,"kb_used_data":2714624,"kb_used_omap":1,"kb_used_meta":428222,"kb_avail":90602288,"statfs":{"total":95995035648,"available":92776742912,"internally_reserved":0,"allocated":2779774976,"data_stored":2778126996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":438499781},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738536,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1496220,"kb_used_data":1158224,"kb_used_omap":1,"kb_used_meta":337982,"kb_avail":92248932,"statfs":{"total":95995035648,"available":94462906368,"internally_reserved":0,"allocated":1186021376,"data_stored":1184368554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":346094021},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1185087488,"data_stored":1185087488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2828402688,"data_stored":2828402688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1703608320,"data_stored":1703608320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1529282560,"data_stored":1529282560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1180631040,"data_stored":1180631040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2483355648,"data_stored":2483355648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":284622848,"data_stored":284622848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":970391552,"data_stored":970391552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":2054488064,"data_stored":2054488064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T14:21:18.255 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-19T14:21:18.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:17 smithi032 ceph-mon[20004]: pgmap v800: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 4.7 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 89 MiB/s wr, 1.42k op/s; 3989/229251 objects misplaced (1.740%); 1.2 MiB/s, 19 objects/s recovering 2024-08-19T14:21:18.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:17 smithi045 ceph-mon[20143]: pgmap v800: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 4.7 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 89 MiB/s wr, 1.42k op/s; 3989/229251 objects misplaced (1.740%); 1.2 MiB/s, 19 objects/s recovering 2024-08-19T14:21:18.346 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:21:18.375 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:18.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:18.376 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:21:18.477 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:21:18.505 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:18.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:18.506 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:21:18.607 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:21:18.635 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:18.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:18.636 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:21:18.737 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:21:18.763 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:18.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:18.765 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:21:18.845 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:21:18.846 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 20, 2024-08-19T14:21:18.846 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 600, 2024-08-19T14:21:18.846 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [ 2024-08-19T14:21:18.846 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.846 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25057.0:63310 3.1b 3:de62eaa4:::benchmark_data_smithi045_59973_object63309:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e244)", 2024-08-19T14:21:18.846 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:17:33.888643+0000", 2024-08-19T14:21:18.846 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 224.948486637, 2024-08-19T14:21:18.846 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.11976188, 2024-08-19T14:21:18.846 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:21:18.846 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:21:18.846 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:21:18.847 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:21:18.847 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25057", 2024-08-19T14:21:18.847 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:21:18.847 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 63310 2024-08-19T14:21:18.847 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.847 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:21:18.847 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.847 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:21:18.847 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:33.888643+0000", 2024-08-19T14:21:18.847 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:21:18.847 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.847 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.848 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:21:18.848 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:33.888643+0000", 2024-08-19T14:21:18.848 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:21:18.848 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.848 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.848 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:21:18.848 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:33.888648+0000", 2024-08-19T14:21:18.848 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4959999999999998e-06 2024-08-19T14:21:18.848 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.848 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.848 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:21:18.849 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:33.888850+0000", 2024-08-19T14:21:18.849 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00020193199999999999 2024-08-19T14:21:18.849 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.849 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.849 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:21:18.849 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:33.888866+0000", 2024-08-19T14:21:18.849 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6195e-05 2024-08-19T14:21:18.849 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.849 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.849 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:21:18.849 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:33.888873+0000", 2024-08-19T14:21:18.849 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.3669999999999999e-06 2024-08-19T14:21:18.850 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.850 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.850 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:21:18.850 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:33.975006+0000", 2024-08-19T14:21:18.850 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.086132499000000001 2024-08-19T14:21:18.850 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.850 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.850 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:21:18.850 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:33.975015+0000", 2024-08-19T14:21:18.850 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.0690000000000008e-06 2024-08-19T14:21:18.850 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.850 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.851 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:21:18.851 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:35.008332+0000", 2024-08-19T14:21:18.851 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0333170140000001 2024-08-19T14:21:18.851 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.851 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.851 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:21:18.851 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:35.008405+0000", 2024-08-19T14:21:18.851 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.3307999999999997e-05 2024-08-19T14:21:18.851 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:21:18.851 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:21:18.852 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:21:18.852 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.852 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.852 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25057.0:63337 3.1b 3:da914815:::benchmark_data_smithi045_59973_object63336:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e244)", 2024-08-19T14:21:18.852 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:17:33.958921+0000", 2024-08-19T14:21:18.853 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 224.878208758, 2024-08-19T14:21:18.853 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0657095919999999, 2024-08-19T14:21:18.853 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:21:18.853 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:21:18.853 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:21:18.854 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:21:18.854 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25057", 2024-08-19T14:21:18.854 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:21:18.854 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 63337 2024-08-19T14:21:18.854 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.854 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:21:18.854 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.855 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:21:18.855 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:33.958921+0000", 2024-08-19T14:21:18.855 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:21:18.855 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.855 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.855 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:21:18.855 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:33.958921+0000", 2024-08-19T14:21:18.855 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:21:18.855 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.856 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.856 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:21:18.856 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:33.958923+0000", 2024-08-19T14:21:18.856 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.1629999999999999e-06 2024-08-19T14:21:18.856 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.856 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.856 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:21:18.856 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:33.958968+0000", 2024-08-19T14:21:18.856 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4407999999999999e-05 2024-08-19T14:21:18.856 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.856 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.856 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:21:18.856 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:33.958990+0000", 2024-08-19T14:21:18.857 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.1851000000000001e-05 2024-08-19T14:21:18.857 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.857 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.857 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:21:18.857 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:33.958999+0000", 2024-08-19T14:21:18.857 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.7270000000000002e-06 2024-08-19T14:21:18.857 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.857 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.857 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:21:18.857 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:33.975034+0000", 2024-08-19T14:21:18.857 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.016035089999999998 2024-08-19T14:21:18.857 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.857 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.857 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:21:18.858 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:33.975040+0000", 2024-08-19T14:21:18.858 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.2900000000000002e-06 2024-08-19T14:21:18.858 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.858 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.858 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:21:18.858 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:35.008794+0000", 2024-08-19T14:21:18.858 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.033754399 2024-08-19T14:21:18.858 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.858 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.858 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:21:18.858 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:35.008913+0000", 2024-08-19T14:21:18.858 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000119225 2024-08-19T14:21:18.858 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.858 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.859 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,4", 2024-08-19T14:21:18.859 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:35.009041+0000", 2024-08-19T14:21:18.859 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000127618 2024-08-19T14:21:18.859 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.859 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.859 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:21:18.859 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:35.010327+0000", 2024-08-19T14:21:18.859 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0012860160000000001 2024-08-19T14:21:18.859 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.859 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.860 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:18.860 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:35.013480+0000", 2024-08-19T14:21:18.860 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0031529219999999998 2024-08-19T14:21:18.860 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.860 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.860 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:18.860 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:35.024451+0000", 2024-08-19T14:21:18.860 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.010971353999999999 2024-08-19T14:21:18.860 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.860 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.860 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:21:18.860 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:35.024584+0000", 2024-08-19T14:21:18.860 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000133127 2024-08-19T14:21:18.860 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.861 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.861 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:21:18.861 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:35.024631+0000", 2024-08-19T14:21:18.861 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.6402000000000001e-05 2024-08-19T14:21:18.861 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:21:18.861 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:21:18.861 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:21:18.861 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.861 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.861 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25057.0:64625 3.15 3:a948fc7c:::benchmark_data_smithi045_59973_object64624:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e246)", 2024-08-19T14:21:18.861 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:17:36.003296+0000", 2024-08-19T14:21:18.861 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 222.83383428299999, 2024-08-19T14:21:18.861 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.18316766400000001, 2024-08-19T14:21:18.861 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:21:18.862 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:21:18.862 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:21:18.862 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:21:18.862 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25057", 2024-08-19T14:21:18.862 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:21:18.862 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 64625 2024-08-19T14:21:18.862 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.862 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:21:18.862 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.862 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:21:18.863 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:36.003296+0000", 2024-08-19T14:21:18.863 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:21:18.863 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.863 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.863 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:21:18.863 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:36.003296+0000", 2024-08-19T14:21:18.863 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:21:18.863 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.863 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.863 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:21:18.863 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:36.003297+0000", 2024-08-19T14:21:18.863 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4479999999999999e-06 2024-08-19T14:21:18.863 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.863 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.864 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:21:18.864 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:36.003664+0000", 2024-08-19T14:21:18.864 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00036658900000000001 2024-08-19T14:21:18.864 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.864 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.864 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:21:18.864 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:36.003675+0000", 2024-08-19T14:21:18.864 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1625e-05 2024-08-19T14:21:18.864 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.864 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.864 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:21:18.864 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:36.003682+0000", 2024-08-19T14:21:18.864 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.1439999999999997e-06 2024-08-19T14:21:18.864 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.865 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.865 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:21:18.865 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:36.006392+0000", 2024-08-19T14:21:18.865 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0027093270000000001 2024-08-19T14:21:18.865 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.865 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.865 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:21:18.865 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:36.006540+0000", 2024-08-19T14:21:18.865 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00014818599999999999 2024-08-19T14:21:18.865 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.865 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.866 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 7,8,11", 2024-08-19T14:21:18.866 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:36.007779+0000", 2024-08-19T14:21:18.866 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0012390420000000001 2024-08-19T14:21:18.866 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.866 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.866 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:18.866 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:36.181996+0000", 2024-08-19T14:21:18.866 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.174217449 2024-08-19T14:21:18.866 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.866 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.867 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:18.867 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:36.182051+0000", 2024-08-19T14:21:18.867 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.452e-05 2024-08-19T14:21:18.867 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.867 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.867 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:18.867 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:36.182098+0000", 2024-08-19T14:21:18.867 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7271000000000002e-05 2024-08-19T14:21:18.867 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.867 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.867 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:21:18.867 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:36.185733+0000", 2024-08-19T14:21:18.867 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.003634918 2024-08-19T14:21:18.867 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.868 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.868 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:21:18.868 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:36.185795+0000", 2024-08-19T14:21:18.868 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.1451999999999994e-05 2024-08-19T14:21:18.868 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.868 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.868 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:21:18.868 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:36.186463+0000", 2024-08-19T14:21:18.868 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00066869300000000004 2024-08-19T14:21:18.868 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:21:18.868 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:21:18.869 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:21:18.869 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.869 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.869 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25057.0:188722 3.27 3:e4cb8e0c:::benchmark_data_smithi045_59973_object57543:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e296)", 2024-08-19T14:21:18.869 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:19:02.154320+0000", 2024-08-19T14:21:18.869 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 136.68281033299999, 2024-08-19T14:21:18.870 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0480230699999999, 2024-08-19T14:21:18.870 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:21:18.870 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:21:18.870 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:21:18.870 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:21:18.871 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25057", 2024-08-19T14:21:18.871 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:21:18.871 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 188722 2024-08-19T14:21:18.872 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.872 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:21:18.872 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.872 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:21:18.873 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.154320+0000", 2024-08-19T14:21:18.873 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:21:18.873 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.873 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.873 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:21:18.873 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.154320+0000", 2024-08-19T14:21:18.873 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:21:18.874 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.874 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.874 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:21:18.874 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.154321+0000", 2024-08-19T14:21:18.874 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.127e-06 2024-08-19T14:21:18.874 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.875 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.875 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:21:18.875 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.154328+0000", 2024-08-19T14:21:18.875 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.542e-06 2024-08-19T14:21:18.875 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.875 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.875 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:21:18.876 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.154340+0000", 2024-08-19T14:21:18.876 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1946999999999999e-05 2024-08-19T14:21:18.876 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.876 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.876 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:21:18.876 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.154348+0000", 2024-08-19T14:21:18.877 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.2490000000000001e-06 2024-08-19T14:21:18.877 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.877 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.877 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:21:18.877 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.286665+0000", 2024-08-19T14:21:18.877 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.13231673599999999 2024-08-19T14:21:18.878 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.878 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.878 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:21:18.878 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.286695+0000", 2024-08-19T14:21:18.878 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.0270999999999999e-05 2024-08-19T14:21:18.879 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.879 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.879 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:21:18.879 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:03.202268+0000", 2024-08-19T14:21:18.879 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.91557253000000005 2024-08-19T14:21:18.879 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.880 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.880 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:21:18.880 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:03.202343+0000", 2024-08-19T14:21:18.880 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.4667999999999997e-05 2024-08-19T14:21:18.880 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:21:18.880 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:21:18.880 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:21:18.880 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.880 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.881 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25057.0:188762 3.27 3:e4fc7775:::benchmark_data_smithi045_59973_object57583:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-08-19T14:21:18.881 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:19:02.183993+0000", 2024-08-19T14:21:18.881 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 136.653137337, 2024-08-19T14:21:18.881 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0247527810000001, 2024-08-19T14:21:18.881 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:21:18.881 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:21:18.881 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:21:18.881 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:21:18.881 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25057", 2024-08-19T14:21:18.881 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:21:18.881 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 188762 2024-08-19T14:21:18.881 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.881 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:21:18.881 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.881 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:21:18.882 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.183993+0000", 2024-08-19T14:21:18.882 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:21:18.882 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.882 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.882 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:21:18.882 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.183993+0000", 2024-08-19T14:21:18.882 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:21:18.882 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.882 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.882 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:21:18.882 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.183995+0000", 2024-08-19T14:21:18.882 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.0609999999999998e-06 2024-08-19T14:21:18.882 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.883 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.883 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:21:18.883 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.184003+0000", 2024-08-19T14:21:18.883 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.3939999999999996e-06 2024-08-19T14:21:18.883 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.883 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.883 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:21:18.883 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.184019+0000", 2024-08-19T14:21:18.883 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6030000000000001e-05 2024-08-19T14:21:18.883 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.883 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.883 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:21:18.883 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.184028+0000", 2024-08-19T14:21:18.883 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.3579999999999992e-06 2024-08-19T14:21:18.884 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.884 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.884 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:21:18.884 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.286800+0000", 2024-08-19T14:21:18.884 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.10277198799999999 2024-08-19T14:21:18.884 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.884 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.884 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:21:18.884 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.286823+0000", 2024-08-19T14:21:18.884 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.3050000000000001e-05 2024-08-19T14:21:18.884 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.884 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.884 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:21:18.885 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:03.202680+0000", 2024-08-19T14:21:18.885 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.91585707800000005 2024-08-19T14:21:18.885 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.885 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.885 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:21:18.885 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:03.202789+0000", 2024-08-19T14:21:18.885 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000108059 2024-08-19T14:21:18.885 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.885 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.885 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 7,11", 2024-08-19T14:21:18.885 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:03.202904+0000", 2024-08-19T14:21:18.885 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00011581100000000001 2024-08-19T14:21:18.885 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.886 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.886 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:21:18.886 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:03.204648+0000", 2024-08-19T14:21:18.886 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.001744076 2024-08-19T14:21:18.886 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.886 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.886 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:18.886 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:03.205732+0000", 2024-08-19T14:21:18.886 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0010840190000000001 2024-08-19T14:21:18.886 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.886 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.886 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:18.886 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:03.208610+0000", 2024-08-19T14:21:18.887 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.002877424 2024-08-19T14:21:18.887 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.887 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.887 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:21:18.887 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:03.208699+0000", 2024-08-19T14:21:18.887 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.9506000000000007e-05 2024-08-19T14:21:18.887 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.887 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.887 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:21:18.887 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:03.208745+0000", 2024-08-19T14:21:18.887 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.5927000000000003e-05 2024-08-19T14:21:18.887 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:21:18.887 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:21:18.887 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:21:18.887 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.888 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.888 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25057.0:208920 3.18 3:1a07a1d6:::benchmark_data_smithi045_59973_object77741:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e302)", 2024-08-19T14:21:18.888 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:19:25.901712+0000", 2024-08-19T14:21:18.888 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 112.935417749, 2024-08-19T14:21:18.888 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.18483865499999999, 2024-08-19T14:21:18.888 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:21:18.888 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:21:18.888 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:21:18.889 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:21:18.889 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25057", 2024-08-19T14:21:18.889 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:21:18.889 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 208920 2024-08-19T14:21:18.889 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.889 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:21:18.889 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.889 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:21:18.889 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:25.901712+0000", 2024-08-19T14:21:18.889 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:21:18.889 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.889 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.890 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:21:18.890 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:25.901712+0000", 2024-08-19T14:21:18.890 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:21:18.890 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.890 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.890 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:21:18.890 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:25.901714+0000", 2024-08-19T14:21:18.890 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.669e-06 2024-08-19T14:21:18.890 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.891 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.891 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:21:18.891 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:25.901720+0000", 2024-08-19T14:21:18.891 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.0140000000000004e-06 2024-08-19T14:21:18.891 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.891 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.891 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:21:18.891 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:25.901732+0000", 2024-08-19T14:21:18.891 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2663999999999999e-05 2024-08-19T14:21:18.891 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.891 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.891 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:21:18.891 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:25.901740+0000", 2024-08-19T14:21:18.892 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.3150000000000003e-06 2024-08-19T14:21:18.892 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.892 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.892 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:21:18.892 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:25.902237+0000", 2024-08-19T14:21:18.892 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00049720199999999997 2024-08-19T14:21:18.893 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.893 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.893 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:21:18.893 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:25.902348+0000", 2024-08-19T14:21:18.893 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000111307 2024-08-19T14:21:18.894 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.894 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.894 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 1,9", 2024-08-19T14:21:18.894 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:25.902448+0000", 2024-08-19T14:21:18.894 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.9936e-05 2024-08-19T14:21:18.894 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.895 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.895 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:21:18.895 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:25.908225+0000", 2024-08-19T14:21:18.895 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0057768910000000001 2024-08-19T14:21:18.895 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.895 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.896 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:18.896 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:25.912784+0000", 2024-08-19T14:21:18.896 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0045585540000000003 2024-08-19T14:21:18.896 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.896 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.896 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:18.897 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:26.086413+0000", 2024-08-19T14:21:18.897 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.17362950999999999 2024-08-19T14:21:18.897 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.897 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.897 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:21:18.897 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:26.086498+0000", 2024-08-19T14:21:18.897 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.5313000000000001e-05 2024-08-19T14:21:18.897 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.898 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.898 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:21:18.898 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:26.086551+0000", 2024-08-19T14:21:18.898 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.2280000000000001e-05 2024-08-19T14:21:18.898 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:21:18.898 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:21:18.899 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:21:18.899 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.899 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.899 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25057.0:216423 3.18 3:1df398bb:::benchmark_data_smithi045_59973_object85244:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e307)", 2024-08-19T14:21:18.899 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:19:33.993641+0000", 2024-08-19T14:21:18.899 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 104.843488674, 2024-08-19T14:21:18.900 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.18904570800000001, 2024-08-19T14:21:18.900 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:21:18.900 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:21:18.900 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:21:18.900 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:21:18.901 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25057", 2024-08-19T14:21:18.901 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:21:18.901 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 216423 2024-08-19T14:21:18.901 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.902 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:21:18.902 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.902 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:21:18.902 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:33.993641+0000", 2024-08-19T14:21:18.902 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:21:18.902 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.902 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.902 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:21:18.902 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:33.993641+0000", 2024-08-19T14:21:18.902 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:21:18.902 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.902 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.903 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:21:18.903 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:33.993642+0000", 2024-08-19T14:21:18.903 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0720000000000001e-06 2024-08-19T14:21:18.903 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.903 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.903 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:21:18.903 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:33.993646+0000", 2024-08-19T14:21:18.903 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7100000000000001e-06 2024-08-19T14:21:18.903 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.903 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.903 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:21:18.904 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:33.993654+0000", 2024-08-19T14:21:18.904 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.4090000000000003e-06 2024-08-19T14:21:18.904 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.904 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.904 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:21:18.904 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:33.993659+0000", 2024-08-19T14:21:18.904 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.9899999999999997e-06 2024-08-19T14:21:18.904 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.904 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.904 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:21:18.904 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:33.993700+0000", 2024-08-19T14:21:18.904 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.0173999999999999e-05 2024-08-19T14:21:18.904 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.904 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.905 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:21:18.905 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:33.993774+0000", 2024-08-19T14:21:18.905 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.4704999999999997e-05 2024-08-19T14:21:18.905 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.905 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.905 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 1,9", 2024-08-19T14:21:18.906 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:33.993838+0000", 2024-08-19T14:21:18.906 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.3281999999999995e-05 2024-08-19T14:21:18.906 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:21:18.908 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.908 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.909 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:21:18.909 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:34.176586+0000", 2024-08-19T14:21:18.909 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.182747994 2024-08-19T14:21:18.909 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.909 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.909 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:18.910 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:34.176625+0000", 2024-08-19T14:21:18.910 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.8995999999999997e-05 2024-08-19T14:21:18.910 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.911 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.911 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:18.911 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:34.182590+0000", 2024-08-19T14:21:18.911 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.005965662 2024-08-19T14:21:18.911 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.911 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.912 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:21:18.912 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:34.182643+0000", 2024-08-19T14:21:18.912 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.2670999999999998e-05 2024-08-19T14:21:18.913 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.913 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.913 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:21:18.913 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:34.182687+0000", 2024-08-19T14:21:18.913 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4042999999999998e-05 2024-08-19T14:21:18.913 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:21:18.914 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:21:18.914 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:21:18.914 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.914 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.914 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25057.0:216925 3.25 3:a76b9d69:::benchmark_data_smithi045_59973_object85746:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e307)", 2024-08-19T14:21:18.915 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:19:34.561921+0000", 2024-08-19T14:21:18.915 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 104.27520899699999, 2024-08-19T14:21:18.915 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.208080195, 2024-08-19T14:21:18.915 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:21:18.915 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:21:18.916 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:21:18.916 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:21:18.916 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25057", 2024-08-19T14:21:18.916 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:21:18.917 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 216925 2024-08-19T14:21:18.917 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.917 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:21:18.917 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.917 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:21:18.917 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:34.561921+0000", 2024-08-19T14:21:18.918 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:21:18.918 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.918 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.918 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:21:18.918 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:34.561921+0000", 2024-08-19T14:21:18.919 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:21:18.919 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.919 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.919 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:21:18.919 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:34.561922+0000", 2024-08-19T14:21:18.919 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4589999999999999e-06 2024-08-19T14:21:18.919 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.919 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.919 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:21:18.919 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:34.561926+0000", 2024-08-19T14:21:18.920 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.799e-06 2024-08-19T14:21:18.920 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.920 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.920 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:21:18.920 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:34.561935+0000", 2024-08-19T14:21:18.920 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.217e-06 2024-08-19T14:21:18.920 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.920 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.920 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:21:18.920 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:34.561941+0000", 2024-08-19T14:21:18.920 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.8130000000000001e-06 2024-08-19T14:21:18.920 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.920 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.920 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:21:18.921 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:34.721674+0000", 2024-08-19T14:21:18.921 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.15973267599999999 2024-08-19T14:21:18.921 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.921 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.921 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:21:18.921 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:34.721844+0000", 2024-08-19T14:21:18.921 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000170271 2024-08-19T14:21:18.921 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.921 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.921 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 8,9,11", 2024-08-19T14:21:18.921 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:34.722139+0000", 2024-08-19T14:21:18.921 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00029475399999999999 2024-08-19T14:21:18.921 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.922 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.922 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:18.922 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:34.726021+0000", 2024-08-19T14:21:18.922 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.003881865 2024-08-19T14:21:18.922 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.922 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.922 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:18.922 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:34.734915+0000", 2024-08-19T14:21:18.922 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0088943110000000002 2024-08-19T14:21:18.922 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.922 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.922 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:18.922 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:34.734993+0000", 2024-08-19T14:21:18.923 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.7733999999999998e-05 2024-08-19T14:21:18.923 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.923 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.923 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:21:18.923 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:34.769714+0000", 2024-08-19T14:21:18.923 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.034721476000000001 2024-08-19T14:21:18.923 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.923 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.923 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:21:18.923 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:34.769792+0000", 2024-08-19T14:21:18.923 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.7800999999999996e-05 2024-08-19T14:21:18.923 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.923 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.923 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:21:18.924 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:34.770001+0000", 2024-08-19T14:21:18.924 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00020901900000000001 2024-08-19T14:21:18.924 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:21:18.924 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:21:18.924 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:21:18.924 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.924 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.924 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25333.0:22719 4.0 4:0fc4203e:::benchmark_data_smithi045_71562_object22718:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-08-19T14:21:18.924 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:20:41.358778+0000", 2024-08-19T14:21:18.924 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 37.478351764000003, 2024-08-19T14:21:18.924 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.32258211599999997, 2024-08-19T14:21:18.924 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:21:18.924 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:21:18.924 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:21:18.924 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:21:18.925 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25333", 2024-08-19T14:21:18.925 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:21:18.925 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 22719 2024-08-19T14:21:18.925 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.925 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:21:18.925 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.925 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:21:18.925 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.358778+0000", 2024-08-19T14:21:18.925 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:21:18.925 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.925 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.925 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:21:18.925 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.358778+0000", 2024-08-19T14:21:18.925 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:21:18.926 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.926 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.926 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:21:18.926 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.358789+0000", 2024-08-19T14:21:18.926 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0446e-05 2024-08-19T14:21:18.926 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.926 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.926 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:21:18.926 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.358854+0000", 2024-08-19T14:21:18.926 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.5469000000000006e-05 2024-08-19T14:21:18.926 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.926 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.927 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:21:18.927 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.358877+0000", 2024-08-19T14:21:18.927 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.3285000000000002e-05 2024-08-19T14:21:18.927 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.927 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.927 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:21:18.927 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.358884+0000", 2024-08-19T14:21:18.927 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.229e-06 2024-08-19T14:21:18.927 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.927 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.927 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:21:18.927 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.358934+0000", 2024-08-19T14:21:18.927 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0878999999999999e-05 2024-08-19T14:21:18.927 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.928 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.928 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:21:18.928 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.359078+0000", 2024-08-19T14:21:18.928 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000143805 2024-08-19T14:21:18.928 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.928 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.928 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,7", 2024-08-19T14:21:18.928 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.359265+0000", 2024-08-19T14:21:18.928 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00018657799999999999 2024-08-19T14:21:18.928 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.928 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.928 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:18.928 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.360662+0000", 2024-08-19T14:21:18.928 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.001397162 2024-08-19T14:21:18.929 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.929 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.929 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:21:18.929 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.360902+0000", 2024-08-19T14:21:18.929 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000239614 2024-08-19T14:21:18.929 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.929 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.929 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:21:18.930 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:18.934 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.681249+0000", 2024-08-19T14:21:18.934 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.32034695800000001 2024-08-19T14:21:18.934 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.934 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.934 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:21:18.934 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.681311+0000", 2024-08-19T14:21:18.934 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.2346000000000002e-05 2024-08-19T14:21:18.934 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.934 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.934 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:21:18.934 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.681360+0000", 2024-08-19T14:21:18.935 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.9345000000000003e-05 2024-08-19T14:21:18.935 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:21:18.935 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:21:18.935 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:21:18.935 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.935 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.935 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25333.0:56498 4.0 4:0f82e6b4:::benchmark_data_smithi045_71562_object56497:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-08-19T14:21:18.935 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:20:58.878784+0000", 2024-08-19T14:21:18.935 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 19.958346095, 2024-08-19T14:21:18.935 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.18928798299999999, 2024-08-19T14:21:18.935 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:21:18.935 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:21:18.935 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:21:18.935 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:21:18.936 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25333", 2024-08-19T14:21:18.936 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:21:18.936 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 56498 2024-08-19T14:21:18.936 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.936 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:21:18.936 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.936 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:21:18.936 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:58.878784+0000", 2024-08-19T14:21:18.936 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:21:18.936 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.936 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.936 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:21:18.936 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:58.878784+0000", 2024-08-19T14:21:18.936 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:21:18.937 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.937 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.937 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:21:18.937 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:58.878785+0000", 2024-08-19T14:21:18.937 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6160000000000001e-06 2024-08-19T14:21:18.937 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.937 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.937 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:21:18.937 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:58.878867+0000", 2024-08-19T14:21:18.937 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.1923999999999997e-05 2024-08-19T14:21:18.937 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.937 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.937 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:21:18.937 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:58.878880+0000", 2024-08-19T14:21:18.938 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2598e-05 2024-08-19T14:21:18.938 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.938 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.938 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:21:18.938 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:58.878887+0000", 2024-08-19T14:21:18.938 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.0199999999999997e-06 2024-08-19T14:21:18.938 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.938 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.938 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:21:18.938 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:58.878940+0000", 2024-08-19T14:21:18.938 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.2954999999999997e-05 2024-08-19T14:21:18.938 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.938 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.938 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:21:18.939 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:58.879060+0000", 2024-08-19T14:21:18.939 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00011979100000000001 2024-08-19T14:21:18.939 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.939 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.939 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,7", 2024-08-19T14:21:18.939 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:58.879200+0000", 2024-08-19T14:21:18.939 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00014064899999999999 2024-08-19T14:21:18.939 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.939 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.939 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:21:18.939 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:58.881795+0000", 2024-08-19T14:21:18.939 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0025949549999999999 2024-08-19T14:21:18.939 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.939 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.940 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:18.940 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:58.885495+0000", 2024-08-19T14:21:18.940 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0037001040000000001 2024-08-19T14:21:18.940 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.940 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.940 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:18.940 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:59.067923+0000", 2024-08-19T14:21:18.940 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.18242719499999999 2024-08-19T14:21:18.940 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.940 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.940 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:21:18.940 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:59.068024+0000", 2024-08-19T14:21:18.940 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000101557 2024-08-19T14:21:18.941 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.941 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.941 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:21:18.941 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:59.068072+0000", 2024-08-19T14:21:18.941 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7618999999999999e-05 2024-08-19T14:21:18.941 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:21:18.941 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:21:18.941 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:21:18.941 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.941 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.941 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25333.0:56501 4.0 4:099fdd3c:::benchmark_data_smithi045_71562_object56500:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-08-19T14:21:18.941 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:20:58.882145+0000", 2024-08-19T14:21:18.941 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 19.954984971999998, 2024-08-19T14:21:18.941 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.186117589, 2024-08-19T14:21:18.942 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:21:18.942 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:21:18.942 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:21:18.942 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:21:18.942 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25333", 2024-08-19T14:21:18.942 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:21:18.942 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 56501 2024-08-19T14:21:18.942 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.942 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:21:18.942 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.942 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:21:18.942 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:58.882145+0000", 2024-08-19T14:21:18.942 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:21:18.943 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.943 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.943 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:21:18.943 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:58.882145+0000", 2024-08-19T14:21:18.943 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:21:18.943 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.943 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.943 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:21:18.943 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:58.882147+0000", 2024-08-19T14:21:18.944 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.7859999999999999e-06 2024-08-19T14:21:18.944 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.944 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.944 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:21:18.944 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:58.882174+0000", 2024-08-19T14:21:18.944 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.7492e-05 2024-08-19T14:21:18.944 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.944 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.944 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:21:18.944 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:58.882187+0000", 2024-08-19T14:21:18.944 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2495e-05 2024-08-19T14:21:18.944 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.944 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.944 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:21:18.945 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:58.882194+0000", 2024-08-19T14:21:18.945 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.0910000000000004e-06 2024-08-19T14:21:18.945 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.945 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.945 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:21:18.945 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:58.882258+0000", 2024-08-19T14:21:18.945 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.4004000000000006e-05 2024-08-19T14:21:18.945 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.945 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.945 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:21:18.945 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:58.882403+0000", 2024-08-19T14:21:18.945 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00014511799999999999 2024-08-19T14:21:18.945 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.945 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.945 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,7", 2024-08-19T14:21:18.946 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:58.882563+0000", 2024-08-19T14:21:18.946 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000160144 2024-08-19T14:21:18.946 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.946 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.946 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:21:18.946 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:58.885522+0000", 2024-08-19T14:21:18.946 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0029586759999999999 2024-08-19T14:21:18.946 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.946 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.946 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:18.946 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:58.885646+0000", 2024-08-19T14:21:18.946 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012395 2024-08-19T14:21:18.946 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.947 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.947 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:18.947 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:59.068118+0000", 2024-08-19T14:21:18.947 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.18247282300000001 2024-08-19T14:21:18.947 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.947 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.947 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:21:18.947 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:59.068212+0000", 2024-08-19T14:21:18.947 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.3626000000000001e-05 2024-08-19T14:21:18.947 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.947 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.947 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:21:18.947 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:59.068262+0000", 2024-08-19T14:21:18.948 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0383999999999997e-05 2024-08-19T14:21:18.948 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:21:18.948 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:21:18.948 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:21:18.948 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.948 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.948 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25333.0:59928 4.0 4:01eb36bc:::benchmark_data_smithi045_71562_object59927:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-08-19T14:21:18.948 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:21:00.987826+0000", 2024-08-19T14:21:18.948 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 17.849303510999999, 2024-08-19T14:21:18.948 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.20708318000000001, 2024-08-19T14:21:18.948 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:21:18.948 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:21:18.948 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:21:18.948 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:21:18.949 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25333", 2024-08-19T14:21:18.949 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:21:18.949 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 59928 2024-08-19T14:21:18.949 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.949 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:21:18.949 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.949 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:21:18.949 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.987826+0000", 2024-08-19T14:21:18.949 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:21:18.949 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.949 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.949 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:21:18.949 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.987826+0000", 2024-08-19T14:21:18.950 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:21:18.950 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.950 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.950 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:21:18.950 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.987835+0000", 2024-08-19T14:21:18.950 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.7809999999999994e-06 2024-08-19T14:21:18.950 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.950 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.950 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:21:18.950 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.987889+0000", 2024-08-19T14:21:18.950 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.3498000000000002e-05 2024-08-19T14:21:18.950 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.950 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.950 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:21:18.951 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.987901+0000", 2024-08-19T14:21:18.951 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2661e-05 2024-08-19T14:21:18.951 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.951 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.951 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:21:18.951 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.987909+0000", 2024-08-19T14:21:18.951 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.3019999999999997e-06 2024-08-19T14:21:18.951 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.951 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.951 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:21:18.951 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.987976+0000", 2024-08-19T14:21:18.951 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.7602999999999996e-05 2024-08-19T14:21:18.951 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.952 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.952 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:21:18.952 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.988101+0000", 2024-08-19T14:21:18.952 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000124902 2024-08-19T14:21:18.952 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.952 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.952 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,7", 2024-08-19T14:21:18.952 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.988524+0000", 2024-08-19T14:21:18.952 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000422922 2024-08-19T14:21:18.952 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.952 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.952 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:18.952 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.997402+0000", 2024-08-19T14:21:18.952 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0088783839999999996 2024-08-19T14:21:18.953 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.953 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.953 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:21:18.953 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.997415+0000", 2024-08-19T14:21:18.953 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2125e-05 2024-08-19T14:21:18.953 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.953 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.953 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:18.953 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.194820+0000", 2024-08-19T14:21:18.953 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.19740545000000001 2024-08-19T14:21:18.953 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.953 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.953 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:21:18.953 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.194855+0000", 2024-08-19T14:21:18.953 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.4551000000000002e-05 2024-08-19T14:21:18.954 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.954 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.954 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:21:18.954 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.194910+0000", 2024-08-19T14:21:18.954 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.5000999999999997e-05 2024-08-19T14:21:18.954 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:21:18.954 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:21:18.954 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:21:18.954 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.954 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.954 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25333.0:59930 4.0 4:053c7972:::benchmark_data_smithi045_71562_object59929:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-08-19T14:21:18.954 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:21:00.988067+0000", 2024-08-19T14:21:18.954 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 17.849062695000001, 2024-08-19T14:21:18.955 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.20703280199999999, 2024-08-19T14:21:18.955 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:21:18.955 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:21:18.955 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:21:18.955 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:21:18.955 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25333", 2024-08-19T14:21:18.955 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:21:18.955 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 59930 2024-08-19T14:21:18.955 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.955 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:21:18.955 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.955 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:21:18.955 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.988067+0000", 2024-08-19T14:21:18.955 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:21:18.956 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.956 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.956 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:21:18.956 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.988067+0000", 2024-08-19T14:21:18.956 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:21:18.956 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.956 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.956 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:21:18.956 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.988068+0000", 2024-08-19T14:21:18.956 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.6700000000000002e-07 2024-08-19T14:21:18.956 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.956 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.956 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:21:18.956 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.988128+0000", 2024-08-19T14:21:18.957 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.0041000000000003e-05 2024-08-19T14:21:18.957 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.957 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.957 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:21:18.957 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.988150+0000", 2024-08-19T14:21:18.957 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.2262000000000001e-05 2024-08-19T14:21:18.957 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.957 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.957 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:21:18.957 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.988157+0000", 2024-08-19T14:21:18.957 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.567e-06 2024-08-19T14:21:18.957 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.957 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.957 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:21:18.958 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.990871+0000", 2024-08-19T14:21:18.958 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.002714475 2024-08-19T14:21:18.958 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.958 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.958 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:21:18.958 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.990996+0000", 2024-08-19T14:21:18.958 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012425700000000001 2024-08-19T14:21:18.958 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.958 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.958 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,7", 2024-08-19T14:21:18.958 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.991126+0000", 2024-08-19T14:21:18.958 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000130548 2024-08-19T14:21:18.958 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.958 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.959 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:21:18.959 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.997419+0000", 2024-08-19T14:21:18.959 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0062931339999999997 2024-08-19T14:21:18.959 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.959 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.959 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:18.959 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.997461+0000", 2024-08-19T14:21:18.959 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2061999999999998e-05 2024-08-19T14:21:18.959 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.959 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.959 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:18.959 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.194959+0000", 2024-08-19T14:21:18.959 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.197497439 2024-08-19T14:21:18.959 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.959 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.960 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:21:18.960 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.195017+0000", 2024-08-19T14:21:18.960 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.8272e-05 2024-08-19T14:21:18.960 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.960 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.960 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:21:18.960 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.195100+0000", 2024-08-19T14:21:18.960 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.2878000000000002e-05 2024-08-19T14:21:18.960 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:21:18.960 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:21:18.960 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:21:18.960 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.960 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.960 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25333.0:59932 4.0 4:0ae6f13c:::benchmark_data_smithi045_71562_object59931:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-08-19T14:21:18.961 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:21:00.988175+0000", 2024-08-19T14:21:18.961 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 17.848954567, 2024-08-19T14:21:18.961 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.21384358000000001, 2024-08-19T14:21:18.961 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:21:18.961 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:21:18.961 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:21:18.961 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:21:18.961 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25333", 2024-08-19T14:21:18.961 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:21:18.961 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 59932 2024-08-19T14:21:18.961 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.961 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:21:18.961 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.961 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:21:18.962 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.988175+0000", 2024-08-19T14:21:18.962 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:21:18.962 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.962 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.962 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:21:18.962 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.988175+0000", 2024-08-19T14:21:18.962 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:21:18.962 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.962 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.962 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:21:18.962 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.988176+0000", 2024-08-19T14:21:18.962 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.8999999999999997e-07 2024-08-19T14:21:18.962 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.963 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.963 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:21:18.963 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.988205+0000", 2024-08-19T14:21:18.963 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.9047000000000002e-05 2024-08-19T14:21:18.963 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.963 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.963 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:21:18.963 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.988212+0000", 2024-08-19T14:21:18.963 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.2250000000000003e-06 2024-08-19T14:21:18.963 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.963 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.963 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:21:18.963 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.988218+0000", 2024-08-19T14:21:18.963 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.4670000000000003e-06 2024-08-19T14:21:18.964 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.964 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.964 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:21:18.964 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.993116+0000", 2024-08-19T14:21:18.964 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0048980830000000001 2024-08-19T14:21:18.964 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.964 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.964 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:21:18.964 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.993256+0000", 2024-08-19T14:21:18.964 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000140426 2024-08-19T14:21:18.964 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.964 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.964 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,7", 2024-08-19T14:21:18.964 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.993350+0000", 2024-08-19T14:21:18.965 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.4026999999999999e-05 2024-08-19T14:21:18.965 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.965 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.965 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:21:18.965 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.997424+0000", 2024-08-19T14:21:18.965 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0040743929999999999 2024-08-19T14:21:18.965 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.965 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.965 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:18.965 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.002435+0000", 2024-08-19T14:21:18.965 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0050106919999999998 2024-08-19T14:21:18.965 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.965 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.965 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:18.966 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.201944+0000", 2024-08-19T14:21:18.966 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.19950880200000001 2024-08-19T14:21:18.966 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.966 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.966 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:21:18.966 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.201978+0000", 2024-08-19T14:21:18.966 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.4379999999999999e-05 2024-08-19T14:21:18.966 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.966 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.966 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:21:18.966 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.202019+0000", 2024-08-19T14:21:18.966 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.0547999999999999e-05 2024-08-19T14:21:18.966 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:21:18.966 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:21:18.967 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:21:18.967 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.967 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.967 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25333.0:59939 4.0 4:022bf8ca:::benchmark_data_smithi045_71562_object59938:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-08-19T14:21:18.967 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:21:00.992948+0000", 2024-08-19T14:21:18.967 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 17.844181535000001, 2024-08-19T14:21:18.967 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.20916093399999999, 2024-08-19T14:21:18.967 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:21:18.967 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:21:18.967 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:21:18.967 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:21:18.967 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25333", 2024-08-19T14:21:18.967 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:21:18.967 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 59939 2024-08-19T14:21:18.968 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.968 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:21:18.968 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.968 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:21:18.968 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.992948+0000", 2024-08-19T14:21:18.968 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:21:18.968 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.968 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.968 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:21:18.968 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.992948+0000", 2024-08-19T14:21:18.968 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:21:18.968 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.968 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.968 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:21:18.969 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.992950+0000", 2024-08-19T14:21:18.969 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.668e-06 2024-08-19T14:21:18.969 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.969 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.969 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:21:18.969 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.992997+0000", 2024-08-19T14:21:18.969 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7491999999999999e-05 2024-08-19T14:21:18.969 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.969 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.969 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:21:18.969 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.993017+0000", 2024-08-19T14:21:18.969 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.9837999999999999e-05 2024-08-19T14:21:18.969 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.969 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.970 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:21:18.970 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.993025+0000", 2024-08-19T14:21:18.970 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.2710000000000003e-06 2024-08-19T14:21:18.970 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.970 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.970 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:21:18.970 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.995120+0000", 2024-08-19T14:21:18.970 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.002095436 2024-08-19T14:21:18.970 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.970 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.970 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:21:18.970 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.995232+0000", 2024-08-19T14:21:18.970 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000111691 2024-08-19T14:21:18.971 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.971 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.971 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,7", 2024-08-19T14:21:18.971 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.995370+0000", 2024-08-19T14:21:18.971 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013791000000000001 2024-08-19T14:21:18.971 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.971 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.971 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:21:18.971 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.001953+0000", 2024-08-19T14:21:18.971 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0065829340000000004 2024-08-19T14:21:18.971 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.971 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.971 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:18.971 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.002472+0000", 2024-08-19T14:21:18.972 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00051963200000000001 2024-08-19T14:21:18.972 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.972 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.972 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:18.972 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.202049+0000", 2024-08-19T14:21:18.972 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.19957635800000001 2024-08-19T14:21:18.972 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.972 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.972 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:21:18.972 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.202076+0000", 2024-08-19T14:21:18.973 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.7314999999999999e-05 2024-08-19T14:21:18.973 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.973 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.973 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:21:18.973 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.202109+0000", 2024-08-19T14:21:18.973 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.3389000000000002e-05 2024-08-19T14:21:18.973 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:21:18.973 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:21:18.973 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:21:18.973 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.973 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.973 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25333.0:59946 4.0 4:0b0e7328:::benchmark_data_smithi045_71562_object59945:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-08-19T14:21:18.973 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:21:00.995937+0000", 2024-08-19T14:21:18.974 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 17.841192738, 2024-08-19T14:21:18.974 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.20888220900000001, 2024-08-19T14:21:18.974 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:21:18.974 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:21:18.974 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:21:18.974 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:21:18.974 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25333", 2024-08-19T14:21:18.974 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:21:18.974 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 59946 2024-08-19T14:21:18.974 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.974 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:21:18.974 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.974 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:21:18.975 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.995937+0000", 2024-08-19T14:21:18.975 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:21:18.975 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.975 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.975 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:21:18.975 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.995937+0000", 2024-08-19T14:21:18.975 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:21:18.975 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.975 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.975 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:21:18.975 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.995939+0000", 2024-08-19T14:21:18.975 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5510000000000001e-06 2024-08-19T14:21:18.975 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.975 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.976 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:21:18.976 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.996006+0000", 2024-08-19T14:21:18.976 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.6914000000000003e-05 2024-08-19T14:21:18.976 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.976 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.976 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:21:18.976 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.996018+0000", 2024-08-19T14:21:18.976 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2361000000000001e-05 2024-08-19T14:21:18.976 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.976 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.976 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:21:18.976 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.996025+0000", 2024-08-19T14:21:18.976 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.199e-06 2024-08-19T14:21:18.977 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.977 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.977 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:21:18.977 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.997492+0000", 2024-08-19T14:21:18.977 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.001466572 2024-08-19T14:21:18.977 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.977 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.977 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:21:18.977 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.997618+0000", 2024-08-19T14:21:18.977 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012644699999999999 2024-08-19T14:21:18.977 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.977 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.977 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,7", 2024-08-19T14:21:18.977 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.997835+0000", 2024-08-19T14:21:18.978 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000216357 2024-08-19T14:21:18.978 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.978 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.978 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:21:18.978 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.001957+0000", 2024-08-19T14:21:18.978 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0041229559999999997 2024-08-19T14:21:18.978 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.978 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.978 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:18.978 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.005478+0000", 2024-08-19T14:21:18.978 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.003520535 2024-08-19T14:21:18.978 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.978 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.978 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:18.979 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.204708+0000", 2024-08-19T14:21:18.979 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.199229884 2024-08-19T14:21:18.979 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.979 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.979 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:21:18.979 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.204761+0000", 2024-08-19T14:21:18.979 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.3443999999999999e-05 2024-08-19T14:21:18.979 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.979 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.979 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:21:18.979 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.204819+0000", 2024-08-19T14:21:18.979 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.7989000000000003e-05 2024-08-19T14:21:18.979 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:21:18.979 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:21:18.980 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:21:18.980 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.980 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.980 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25333.0:59952 4.0 4:0d22fdcc:::benchmark_data_smithi045_71562_object59951:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-08-19T14:21:18.980 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:21:00.997380+0000", 2024-08-19T14:21:18.980 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 17.839750323000001, 2024-08-19T14:21:18.980 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.207624002, 2024-08-19T14:21:18.980 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:21:18.980 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:21:18.980 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:21:18.980 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:21:18.980 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25333", 2024-08-19T14:21:18.980 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:21:18.980 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 59952 2024-08-19T14:21:18.981 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.981 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:21:18.981 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.981 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:21:18.981 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.997380+0000", 2024-08-19T14:21:18.981 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:21:18.981 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.981 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.981 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:21:18.981 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.997380+0000", 2024-08-19T14:21:18.981 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:21:18.981 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.981 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.981 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:21:18.982 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.997381+0000", 2024-08-19T14:21:18.982 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1310000000000001e-06 2024-08-19T14:21:18.982 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.982 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.982 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:21:18.982 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.997419+0000", 2024-08-19T14:21:18.982 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.8781e-05 2024-08-19T14:21:18.982 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.982 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.982 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:21:18.982 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.997433+0000", 2024-08-19T14:21:18.982 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3301e-05 2024-08-19T14:21:18.982 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.983 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.983 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:21:18.983 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.997440+0000", 2024-08-19T14:21:18.983 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.289e-06 2024-08-19T14:21:18.983 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.983 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.983 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:21:18.983 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.999803+0000", 2024-08-19T14:21:18.983 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.002363066 2024-08-19T14:21:18.983 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.983 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.983 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:21:18.983 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.999943+0000", 2024-08-19T14:21:18.984 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013990000000000001 2024-08-19T14:21:18.984 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.984 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.984 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,7", 2024-08-19T14:21:18.984 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.000083+0000", 2024-08-19T14:21:18.984 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000139963 2024-08-19T14:21:18.984 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.984 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.984 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:21:18.984 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.002202+0000", 2024-08-19T14:21:18.984 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0021193430000000001 2024-08-19T14:21:18.984 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.984 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.985 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:18.985 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.006637+0000", 2024-08-19T14:21:18.985 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0044344629999999996 2024-08-19T14:21:18.985 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.985 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.985 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:18.985 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.204885+0000", 2024-08-19T14:21:18.985 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.198248705 2024-08-19T14:21:18.985 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.985 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.985 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:21:18.985 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.204945+0000", 2024-08-19T14:21:18.985 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.9762000000000001e-05 2024-08-19T14:21:18.986 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.986 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.986 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:21:18.986 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.205004+0000", 2024-08-19T14:21:18.986 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.8298000000000003e-05 2024-08-19T14:21:18.986 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:21:18.986 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:21:18.986 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:21:18.986 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.986 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.986 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25333.0:59956 4.0 4:0ac3aada:::benchmark_data_smithi045_71562_object59955:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-08-19T14:21:18.986 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:21:01.002823+0000", 2024-08-19T14:21:18.986 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 17.83430697, 2024-08-19T14:21:18.987 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.20233593899999999, 2024-08-19T14:21:18.987 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:21:18.987 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:21:18.987 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:21:18.987 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:21:18.987 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25333", 2024-08-19T14:21:18.987 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:21:18.987 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 59956 2024-08-19T14:21:18.987 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.987 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:21:18.987 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.987 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:21:18.987 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.002823+0000", 2024-08-19T14:21:18.988 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:21:18.988 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.988 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.988 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:21:18.988 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.002823+0000", 2024-08-19T14:21:18.988 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:21:18.988 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.988 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.988 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:21:18.988 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.002824+0000", 2024-08-19T14:21:18.988 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.401e-06 2024-08-19T14:21:18.988 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.988 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.989 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:21:18.989 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.002905+0000", 2024-08-19T14:21:18.989 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.0476999999999995e-05 2024-08-19T14:21:18.989 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.989 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.989 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:21:18.989 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.002917+0000", 2024-08-19T14:21:18.989 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2581e-05 2024-08-19T14:21:18.989 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.989 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.989 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:21:18.989 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.002925+0000", 2024-08-19T14:21:18.989 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.391e-06 2024-08-19T14:21:18.990 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.990 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.990 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:21:18.990 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.003010+0000", 2024-08-19T14:21:18.990 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.5099000000000006e-05 2024-08-19T14:21:18.990 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.990 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.990 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:21:18.990 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.003192+0000", 2024-08-19T14:21:18.990 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00018232799999999999 2024-08-19T14:21:18.990 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.990 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.990 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,7", 2024-08-19T14:21:18.991 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.003343+0000", 2024-08-19T14:21:18.991 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000151163 2024-08-19T14:21:18.991 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.991 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.991 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:21:18.991 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.005697+0000", 2024-08-19T14:21:18.991 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.002353749 2024-08-19T14:21:18.991 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.991 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.991 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:18.991 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.006706+0000", 2024-08-19T14:21:18.991 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.001009397 2024-08-19T14:21:18.991 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.991 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.992 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:18.992 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.205041+0000", 2024-08-19T14:21:18.992 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.198334387 2024-08-19T14:21:18.992 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.992 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.992 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:21:18.992 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.205091+0000", 2024-08-19T14:21:18.992 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0374999999999998e-05 2024-08-19T14:21:18.992 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.992 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.992 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:21:18.992 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.205159+0000", 2024-08-19T14:21:18.992 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.7590999999999997e-05 2024-08-19T14:21:18.993 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:21:18.993 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:21:18.993 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:21:18.993 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.993 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.993 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25333.0:59962 4.0 4:05eece39:::benchmark_data_smithi045_71562_object59961:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-08-19T14:21:18.993 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:21:01.006823+0000", 2024-08-19T14:21:18.993 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 17.830307131000001, 2024-08-19T14:21:18.993 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.20119857399999999, 2024-08-19T14:21:18.993 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:21:18.993 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:21:18.993 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:21:18.993 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:21:18.994 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25333", 2024-08-19T14:21:18.994 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:21:18.994 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 59962 2024-08-19T14:21:18.994 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.994 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:21:18.994 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.994 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:21:18.994 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.006823+0000", 2024-08-19T14:21:18.994 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:21:18.994 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.994 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.994 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:21:18.994 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.006823+0000", 2024-08-19T14:21:18.995 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:21:18.995 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.995 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.995 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:21:18.995 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.006824+0000", 2024-08-19T14:21:18.995 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5319999999999999e-06 2024-08-19T14:21:18.995 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.995 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.995 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:21:18.995 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.006881+0000", 2024-08-19T14:21:18.995 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.6805999999999998e-05 2024-08-19T14:21:18.995 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.995 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.996 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:21:18.996 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.006892+0000", 2024-08-19T14:21:18.996 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0997e-05 2024-08-19T14:21:18.996 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.996 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.996 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:21:18.996 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.006897+0000", 2024-08-19T14:21:18.996 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1370000000000001e-06 2024-08-19T14:21:18.996 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.996 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.996 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:21:18.996 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.006947+0000", 2024-08-19T14:21:18.996 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.9484999999999998e-05 2024-08-19T14:21:18.996 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.997 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.997 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:21:18.997 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.007191+0000", 2024-08-19T14:21:18.997 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000244718 2024-08-19T14:21:18.997 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.997 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.997 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,7", 2024-08-19T14:21:18.997 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.007328+0000", 2024-08-19T14:21:18.997 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000136143 2024-08-19T14:21:18.997 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.997 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.997 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:21:18.997 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.010086+0000", 2024-08-19T14:21:18.998 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0027586080000000001 2024-08-19T14:21:18.998 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.998 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.998 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:18.998 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.011443+0000", 2024-08-19T14:21:18.998 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00135698 2024-08-19T14:21:18.998 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.998 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.998 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:18.998 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.207902+0000", 2024-08-19T14:21:18.998 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.19645927599999999 2024-08-19T14:21:18.998 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.998 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.999 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:21:18.999 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.207961+0000", 2024-08-19T14:21:18.999 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.8989000000000001e-05 2024-08-19T14:21:18.999 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.999 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.999 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:21:18.999 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.208021+0000", 2024-08-19T14:21:18.999 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.9902999999999999e-05 2024-08-19T14:21:18.999 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:21:18.999 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:21:18.999 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:21:18.999 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:18.999 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:18.999 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25333.0:65675 4.0 4:09d644dc:::benchmark_data_smithi045_71562_object65674:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e352)", 2024-08-19T14:21:19.000 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:21:04.339245+0000", 2024-08-19T14:21:19.000 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 14.497884708000001, 2024-08-19T14:21:19.000 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.18343664800000001, 2024-08-19T14:21:19.000 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:21:19.000 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:21:19.000 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:21:19.000 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:21:19.000 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25333", 2024-08-19T14:21:19.000 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:21:19.000 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 65675 2024-08-19T14:21:19.000 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:19.000 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:21:19.000 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:19.001 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:21:19.001 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:04.339245+0000", 2024-08-19T14:21:19.001 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:21:19.001 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:19.001 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:19.001 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:21:19.001 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:04.339245+0000", 2024-08-19T14:21:19.001 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:21:19.001 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:19.001 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:19.001 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:21:19.001 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:04.339263+0000", 2024-08-19T14:21:19.001 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.8048999999999999e-05 2024-08-19T14:21:19.001 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:19.002 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:19.002 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:21:19.002 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:04.339332+0000", 2024-08-19T14:21:19.002 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.9060000000000006e-05 2024-08-19T14:21:19.002 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:19.002 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:19.002 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:21:19.002 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:04.339356+0000", 2024-08-19T14:21:19.002 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.3824000000000001e-05 2024-08-19T14:21:19.002 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:19.002 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:19.002 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:21:19.002 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:04.339363+0000", 2024-08-19T14:21:19.003 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.2860000000000002e-06 2024-08-19T14:21:19.003 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:19.003 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:19.003 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:21:19.003 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:04.339424+0000", 2024-08-19T14:21:19.003 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.0909999999999998e-05 2024-08-19T14:21:19.003 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:19.003 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:19.003 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:21:19.003 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:04.339548+0000", 2024-08-19T14:21:19.003 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012341199999999999 2024-08-19T14:21:19.003 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:19.003 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:19.004 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,7", 2024-08-19T14:21:19.004 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:04.339678+0000", 2024-08-19T14:21:19.004 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013075700000000001 2024-08-19T14:21:19.004 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:19.004 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:19.004 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:21:19.004 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:04.342744+0000", 2024-08-19T14:21:19.004 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0030652140000000001 2024-08-19T14:21:19.004 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:19.004 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:19.004 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:19.004 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:04.343193+0000", 2024-08-19T14:21:19.004 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00044979999999999998 2024-08-19T14:21:19.005 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:19.005 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:19.005 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:19.005 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:04.522529+0000", 2024-08-19T14:21:19.005 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.17933595699999999 2024-08-19T14:21:19.005 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:19.005 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:19.005 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:21:19.005 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:04.522629+0000", 2024-08-19T14:21:19.005 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.9619999999999998e-05 2024-08-19T14:21:19.005 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:21:19.005 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:21:19.005 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:21:19.005 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:04.522682+0000", 2024-08-19T14:21:19.006 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.2759000000000001e-05 2024-08-19T14:21:19.006 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:21:19.006 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:21:19.006 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:21:19.006 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:21:19.006 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:21:19.006 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:21:19.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:19.008 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:19.008 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:21:19.048 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 45 16 82537 82521 114.602 87.6875 0.00819582 0.00872153 2024-08-19T14:21:19.108 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:21:19.137 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:19.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:19.138 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:21:19.239 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:21:19.266 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:19.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:19.267 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:21:19.368 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:21:19.403 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:19.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:19.404 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:21:19.506 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:21:19.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:19.545 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:19.545 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:21:19.646 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:21:19.672 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:19.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:19.673 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:21:19.759 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:21:19.774 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:21:19.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:19.857 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:19.858 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:21:19.962 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:21:19.999 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:19.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:20.000 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:21:20.048 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 46 16 83955 83939 114.038 88.625 0.0142702 0.0087642 2024-08-19T14:21:20.101 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:21:20.112 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:19 smithi032 ceph-mon[20004]: pgmap v801: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 4.7 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 66 MiB/s wr, 1.05k op/s; 3989/230481 objects misplaced (1.731%); 1.0 MiB/s, 16 objects/s recovering 2024-08-19T14:21:20.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:19 smithi031 ceph-mon[16336]: pgmap v801: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 4.7 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 66 MiB/s wr, 1.05k op/s; 3989/230481 objects misplaced (1.731%); 1.0 MiB/s, 16 objects/s recovering 2024-08-19T14:21:20.129 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:20.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:20.130 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:21:20.231 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:21:20.262 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:20.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:20.263 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:21:20.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:19 smithi045 ceph-mon[20143]: pgmap v801: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 4.7 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 66 MiB/s wr, 1.05k op/s; 3989/230481 objects misplaced (1.731%); 1.0 MiB/s, 16 objects/s recovering 2024-08-19T14:21:20.367 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:21:20.402 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:20.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:20.403 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:21:20.504 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:21:20.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:20.548 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:20.548 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:21:20.649 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:21:20.675 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:20.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:20.676 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:21:20.777 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:21:20.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:20.812 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:20.812 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:21:20.913 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:21:20.915 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:21:20.960 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:20.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:20.961 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:21:20.966 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-08-19T14:21:21.048 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 47 16 85375 85359 113.499 88.75 0.00970252 0.0088059 2024-08-19T14:21:21.063 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:21:21.091 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:21.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:21.092 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:21:21.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:20 smithi031 ceph-mon[16336]: pgmap v802: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 4.9 GiB data, 18 GiB used, 786 GiB / 805 GiB avail; 85 MiB/s wr, 1.36k op/s; 3854/241932 objects misplaced (1.593%); 2.0 MiB/s, 31 objects/s recovering 2024-08-19T14:21:21.193 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:21:21.219 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:21.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:21.220 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:21:21.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:20 smithi032 ceph-mon[20004]: pgmap v802: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 4.9 GiB data, 18 GiB used, 786 GiB / 805 GiB avail; 85 MiB/s wr, 1.36k op/s; 3854/241932 objects misplaced (1.593%); 2.0 MiB/s, 31 objects/s recovering 2024-08-19T14:21:21.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:20 smithi045 ceph-mon[20143]: pgmap v802: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 4.9 GiB data, 18 GiB used, 786 GiB / 805 GiB avail; 85 MiB/s wr, 1.36k op/s; 3854/241932 objects misplaced (1.593%); 2.0 MiB/s, 31 objects/s recovering 2024-08-19T14:21:21.321 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:21:21.349 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:21:21.350 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:21.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:21.351 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:21:21.452 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:21:21.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:21.502 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:21.502 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:21:21.603 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:21:21.629 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:21.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:21.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:21:21.731 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:21:21.758 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:21.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:21.759 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:21:21.860 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:21:21.872 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is set 2024-08-19T14:21:21.888 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:21.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:21.889 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:21:21.989 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:21:22.019 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:22.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:22.020 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:21:22.036 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:21 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:21:22.036 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:21 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:21:22.036 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:21 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3518403023' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:21:22.048 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 48 16 86660 86644 112.808 80.3125 0.00460159 0.00885894 2024-08-19T14:21:22.063 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:21:22.064 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-08-19T14:21:22.064 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:22.064 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25333.0:86675 4.c 4:3a487de1:::benchmark_data_smithi045_71562_object86674:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e360)", 2024-08-19T14:21:22.064 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:21:22.062141+0000", 2024-08-19T14:21:22.064 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.00022476899999999999, 2024-08-19T14:21:22.064 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00025707799999999999, 2024-08-19T14:21:22.064 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:21:22.064 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:21:22.064 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:21:22.064 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:21:22.064 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25333", 2024-08-19T14:21:22.064 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:21:22.064 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 86675 2024-08-19T14:21:22.065 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:22.065 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:21:22.065 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:22.065 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:21:22.065 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:21:22.062141+0000", 2024-08-19T14:21:22.065 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:21:22.065 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:22.065 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:22.065 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:21:22.065 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:21:22.062141+0000", 2024-08-19T14:21:22.065 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:21:22.065 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:22.065 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:22.065 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:21:22.065 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:21:22.062142+0000", 2024-08-19T14:21:22.066 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.398e-06 2024-08-19T14:21:22.066 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:22.066 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:22.066 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:21:22.066 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:21:22.062184+0000", 2024-08-19T14:21:22.066 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1953000000000003e-05 2024-08-19T14:21:22.066 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:22.066 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:22.066 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:21:22.066 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:21:22.062198+0000", 2024-08-19T14:21:22.066 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.418e-05 2024-08-19T14:21:22.066 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:22.066 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:22.067 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:21:22.067 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:21:22.062206+0000", 2024-08-19T14:21:22.067 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5159999999999997e-06 2024-08-19T14:21:22.067 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:22.067 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:22.067 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:21:22.067 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:21:22.062262+0000", 2024-08-19T14:21:22.067 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6511999999999997e-05 2024-08-19T14:21:22.067 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:21:22.067 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:21:22.067 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:21:22.067 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:21:22.067 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-08-19T14:21:22.067 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 1 2024-08-19T14:21:22.067 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:21:22.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:21 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:21:22.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:21 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:21:22.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:21 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3518403023' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:21:22.121 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:21:22.145 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:22.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:22.147 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:21:22.223 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:21:22.248 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:21:22.276 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:22.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:22.277 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:21:22.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:21 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:21:22.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:21 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:21:22.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:21 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3518403023' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:21:22.378 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:21:22.419 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:22.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:22.420 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:21:22.521 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:21:22.552 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:22.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:22.553 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:21:22.654 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:21:22.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:22.691 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:22.691 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:21:22.792 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:21:22.832 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:22.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:22.833 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:21:22.905 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-08-19T14:21:22.935 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:21:22.962 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:21:22.963 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-08-19T14:21:22.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:22.968 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:22.968 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:21:23.048 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 49 16 87948 87932 112.148 80.5 0.00685161 0.00891241 2024-08-19T14:21:23.069 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:21:23.088 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:22 smithi031 ceph-mon[16336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:21:23.089 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:22 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3518403023' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:21:23.089 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:22 smithi031 ceph-mon[16336]: osdmap e360: 12 total, 12 up, 9 in 2024-08-19T14:21:23.089 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:22 smithi031 ceph-mon[16336]: pgmap v804: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.1 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 91 MiB/s wr, 1.46k op/s; 3854/249780 objects misplaced (1.543%); 2.0 MiB/s, 31 objects/s recovering 2024-08-19T14:21:23.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:23.097 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:23.098 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:21:23.199 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:21:23.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:23.228 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:23.228 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:21:23.251 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:21:23.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:22 smithi032 ceph-mon[20004]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:21:23.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:22 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3518403023' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:21:23.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:22 smithi032 ceph-mon[20004]: osdmap e360: 12 total, 12 up, 9 in 2024-08-19T14:21:23.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:22 smithi032 ceph-mon[20004]: pgmap v804: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.1 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 91 MiB/s wr, 1.46k op/s; 3854/249780 objects misplaced (1.543%); 2.0 MiB/s, 31 objects/s recovering 2024-08-19T14:21:23.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:22 smithi045 ceph-mon[20143]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:21:23.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:22 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3518403023' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:21:23.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:22 smithi045 ceph-mon[20143]: osdmap e360: 12 total, 12 up, 9 in 2024-08-19T14:21:23.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:22 smithi045 ceph-mon[20143]: pgmap v804: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.1 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 91 MiB/s wr, 1.46k op/s; 3854/249780 objects misplaced (1.543%); 2.0 MiB/s, 31 objects/s recovering 2024-08-19T14:21:23.329 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:21:23.365 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:23.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:23.366 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:21:23.467 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:21:23.488 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:23.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:23.489 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:21:23.589 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:21:23.623 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:23.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:23.625 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:21:23.699 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:21:23.700 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:21:23.725 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:21:23.733 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:21:23.734 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:21:23.751 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:23.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:23.752 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:21:23.769 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:21:23.770 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:21:23.802 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:21:23.802 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:21:23.819 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:21:23.819 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-08-19T14:21:23.819 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-08-19T14:21:23.819 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-08-19T14:21:23.819 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:21:23.840 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:21:23.841 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:21:23.853 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:21:23.878 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:23.879 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:21:23.880 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:21:23.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:23.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:21:23.909 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:21:23.909 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:21:23.940 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:21:23.940 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:21:23.973 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:21:23.974 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:21:23.981 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:21:24.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:23 smithi032 ceph-mon[20004]: from='client.25423 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:21:24.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:23 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1302123946' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:21:24.008 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:21:24.008 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:21:24.020 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:24.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:24.021 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:21:24.041 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:21:24.041 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:21:24.048 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 50 15 89068 89053 111.307 70.0625 0.0189104 0.00897974 2024-08-19T14:21:24.078 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:23 smithi031 ceph-mon[16336]: from='client.25423 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:21:24.078 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:23 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1302123946' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:21:24.078 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:21:24.078 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:21:24.122 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:21:24.147 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:24.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:24.148 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:21:24.249 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:21:24.285 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:24.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:24.286 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:21:24.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:23 smithi045 ceph-mon[20143]: from='client.25423 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:21:24.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:23 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1302123946' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:21:24.387 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:21:24.419 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:24.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:24.420 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:21:24.452 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-08-19T14:21:24.490 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":804,"stamp":"2024-08-19T14:21:21.892995+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5456986656,"num_objects":83260,"num_object_clones":0,"num_object_copies":249780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3854,"num_objects_unfound":0,"num_objects_dirty":83260,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":83258,"num_write_kb":5328512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":286,"num_bytes_recovered":18743296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87567,"ondisk_log_size":87567,"up":141,"acting":140,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":142,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":19501516,"kb_used_data":15851520,"kb_used_omap":14,"kb_used_meta":3649905,"kb_avail":824204852,"statfs":{"total":863955320832,"available":843985768448,"internally_reserved":0,"allocated":16231956480,"data_stored":16216994808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14353,"internal_metadata":3737503727},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,6,4,5,2,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":765788160,"num_objects":11685,"num_object_clones":0,"num_object_copies":35055,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-250,"num_objects_unfound":0,"num_objects_dirty":11685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11685,"num_write_kb":747840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":250,"num_bytes_recovered":16384000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001721"},"pg_stats":[{"pgid":"4.5","version":"359'5371","reported_seq":5429,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-08-19T14:21:20.764646+0000","last_change":"2024-08-19T14:20:29.268214+0000","last_active":"2024-08-19T14:21:20.764646+0000","last_peered":"2024-08-19T14:21:20.764646+0000","last_clean":"2024-08-19T14:21:20.764646+0000","last_became_active":"2024-08-19T14:20:29.267549+0000","last_became_peered":"2024-08-19T14:20:29.267549+0000","last_unstale":"2024-08-19T14:21:20.764646+0000","last_undegraded":"2024-08-19T14:21:20.764646+0000","last_fullsized":"2024-08-19T14:21:20.764646+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":5371,"log_dups_size":0,"ondisk_log_size":5371,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:57.640230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":351993856,"num_objects":5371,"num_object_clones":0,"num_object_copies":16113,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5371,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5371,"num_write_kb":343744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,0],"acting":[11,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"70'139","reported_seq":934,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-08-19T14:21:15.879548+0000","last_change":"2024-08-19T14:19:58.072227+0000","last_active":"2024-08-19T14:21:15.879548+0000","last_peered":"2024-08-19T14:21:15.879548+0000","last_clean":"2024-08-19T14:21:15.879548+0000","last_became_active":"2024-08-19T14:18:13.807133+0000","last_became_peered":"2024-08-19T14:18:13.807133+0000","last_unstale":"2024-08-19T14:21:15.879548+0000","last_undegraded":"2024-08-19T14:21:15.879548+0000","last_fullsized":"2024-08-19T14:21:15.879548+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:19:58.072083+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:19:58.072083+0000","last_clean_scrub_stamp":"2024-08-19T14:19:58.072083+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:17.231298+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,9],"acting":[7,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"359'5154","reported_seq":5787,"reported_epoch":359,"state":"active+recovering+undersized+remapped","last_fresh":"2024-08-19T14:21:19.004365+0000","last_change":"2024-08-19T14:21:07.239135+0000","last_active":"2024-08-19T14:21:19.004365+0000","last_peered":"2024-08-19T14:21:19.004365+0000","last_clean":"2024-08-19T14:21:04.784061+0000","last_became_active":"2024-08-19T14:21:07.213050+0000","last_became_peered":"2024-08-19T14:21:07.213050+0000","last_unstale":"2024-08-19T14:21:19.004365+0000","last_undegraded":"2024-08-19T14:21:19.004365+0000","last_fullsized":"2024-08-19T14:21:06.805412+0000","mapping_epoch":354,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":5154,"log_dups_size":0,"ondisk_log_size":5154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":337772544,"num_objects":5154,"num_object_clones":0,"num_object_copies":15462,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3854,"num_objects_unfound":0,"num_objects_dirty":5154,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5154,"num_write_kb":329856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":286,"num_bytes_recovered":18743296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,7],"acting":[1,7],"avail_no_missing":["1","5","7"],"object_location_counts":[{"shards":"1,5,7","objects":3854},{"shards":"1,7","objects":1300}],"blocked_by":[],"up_primary":9,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"70'139","reported_seq":832,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-08-19T14:21:15.876207+0000","last_change":"2024-08-19T14:20:03.714917+0000","last_active":"2024-08-19T14:21:15.876207+0000","last_peered":"2024-08-19T14:21:15.876207+0000","last_clean":"2024-08-19T14:21:15.876207+0000","last_became_active":"2024-08-19T14:18:13.804721+0000","last_became_peered":"2024-08-19T14:18:13.804721+0000","last_unstale":"2024-08-19T14:21:15.876207+0000","last_undegraded":"2024-08-19T14:21:15.876207+0000","last_fullsized":"2024-08-19T14:21:15.876207+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:03.714818+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:03.714818+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:29.563399+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"359'5210","reported_seq":5268,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-08-19T14:21:19.020767+0000","last_change":"2024-08-19T14:20:29.268110+0000","last_active":"2024-08-19T14:21:19.020767+0000","last_peered":"2024-08-19T14:21:19.020767+0000","last_clean":"2024-08-19T14:21:19.020767+0000","last_became_active":"2024-08-19T14:20:29.267811+0000","last_became_peered":"2024-08-19T14:20:29.267811+0000","last_unstale":"2024-08-19T14:21:19.020767+0000","last_undegraded":"2024-08-19T14:21:19.020767+0000","last_fullsized":"2024-08-19T14:21:19.020767+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":5210,"log_dups_size":0,"ondisk_log_size":5210,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:46.516076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":341442560,"num_objects":5210,"num_object_clones":0,"num_object_copies":15630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5210,"num_write_kb":333440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"70'139","reported_seq":781,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-08-19T14:21:15.877570+0000","last_change":"2024-08-19T14:20:50.742676+0000","last_active":"2024-08-19T14:21:15.877570+0000","last_peered":"2024-08-19T14:21:15.877570+0000","last_clean":"2024-08-19T14:21:15.877570+0000","last_became_active":"2024-08-19T14:20:50.742354+0000","last_became_peered":"2024-08-19T14:20:50.742354+0000","last_unstale":"2024-08-19T14:21:15.877570+0000","last_undegraded":"2024-08-19T14:21:15.877570+0000","last_fullsized":"2024-08-19T14:21:15.877570+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:19:59.302931+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:19:59.302931+0000","last_clean_scrub_stamp":"2024-08-19T14:19:59.302931+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:18.923389+0000","scrub_duration":114,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"359'5168","reported_seq":5226,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-08-19T14:21:18.969896+0000","last_change":"2024-08-19T14:20:29.267640+0000","last_active":"2024-08-19T14:21:18.969896+0000","last_peered":"2024-08-19T14:21:18.969896+0000","last_clean":"2024-08-19T14:21:18.969896+0000","last_became_active":"2024-08-19T14:20:29.267331+0000","last_became_peered":"2024-08-19T14:20:29.267331+0000","last_unstale":"2024-08-19T14:21:18.969896+0000","last_undegraded":"2024-08-19T14:21:18.969896+0000","last_fullsized":"2024-08-19T14:21:18.969896+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":5168,"log_dups_size":0,"ondisk_log_size":5168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:46.504191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":338690048,"num_objects":5168,"num_object_clones":0,"num_object_copies":15504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5168,"num_write_kb":330752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"70'139","reported_seq":829,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-08-19T14:21:15.876018+0000","last_change":"2024-08-19T14:20:09.507422+0000","last_active":"2024-08-19T14:21:15.876018+0000","last_peered":"2024-08-19T14:21:15.876018+0000","last_clean":"2024-08-19T14:21:15.876018+0000","last_became_active":"2024-08-19T14:18:13.808505+0000","last_became_peered":"2024-08-19T14:18:13.808505+0000","last_unstale":"2024-08-19T14:21:15.876018+0000","last_undegraded":"2024-08-19T14:21:15.876018+0000","last_fullsized":"2024-08-19T14:21:15.876018+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:09.507256+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:09.507256+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:29.309519+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,11],"acting":[9,0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"359'5139","reported_seq":5197,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-08-19T14:21:18.376858+0000","last_change":"2024-08-19T14:20:29.268766+0000","last_active":"2024-08-19T14:21:18.376858+0000","last_peered":"2024-08-19T14:21:18.376858+0000","last_clean":"2024-08-19T14:21:18.376858+0000","last_became_active":"2024-08-19T14:20:29.268443+0000","last_became_peered":"2024-08-19T14:20:29.268443+0000","last_unstale":"2024-08-19T14:21:18.376858+0000","last_undegraded":"2024-08-19T14:21:18.376858+0000","last_fullsized":"2024-08-19T14:21:18.376858+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":5139,"log_dups_size":0,"ondisk_log_size":5139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:39.688611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":336789504,"num_objects":5139,"num_object_clones":0,"num_object_copies":15417,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5139,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5139,"num_write_kb":328896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,11],"acting":[9,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.7","version":"70'139","reported_seq":473,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-08-19T14:21:15.876746+0000","last_change":"2024-08-19T14:20:34.937317+0000","last_active":"2024-08-19T14:21:15.876746+0000","last_peered":"2024-08-19T14:21:15.876746+0000","last_clean":"2024-08-19T14:21:15.876746+0000","last_became_active":"2024-08-19T14:20:28.204384+0000","last_became_peered":"2024-08-19T14:20:28.204384+0000","last_unstale":"2024-08-19T14:21:15.876746+0000","last_undegraded":"2024-08-19T14:21:15.876746+0000","last_fullsized":"2024-08-19T14:21:15.876746+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:34.937192+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:20:34.937192+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:40.330609+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"359'5288","reported_seq":5346,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-08-19T14:21:20.736303+0000","last_change":"2024-08-19T14:20:29.267833+0000","last_active":"2024-08-19T14:21:20.736303+0000","last_peered":"2024-08-19T14:21:20.736303+0000","last_clean":"2024-08-19T14:21:20.736303+0000","last_became_active":"2024-08-19T14:20:29.267379+0000","last_became_peered":"2024-08-19T14:20:29.267379+0000","last_unstale":"2024-08-19T14:21:20.736303+0000","last_undegraded":"2024-08-19T14:21:20.736303+0000","last_fullsized":"2024-08-19T14:21:20.736303+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":5288,"log_dups_size":0,"ondisk_log_size":5288,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:28.819534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":346554368,"num_objects":5288,"num_object_clones":0,"num_object_copies":15864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5288,"num_write_kb":338432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,1],"acting":[11,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.6","version":"70'139","reported_seq":833,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-08-19T14:21:15.876186+0000","last_change":"2024-08-19T14:19:55.512078+0000","last_active":"2024-08-19T14:21:15.876186+0000","last_peered":"2024-08-19T14:21:15.876186+0000","last_clean":"2024-08-19T14:21:15.876186+0000","last_became_active":"2024-08-19T14:18:26.874902+0000","last_became_peered":"2024-08-19T14:18:26.874902+0000","last_unstale":"2024-08-19T14:21:15.876186+0000","last_undegraded":"2024-08-19T14:21:15.876186+0000","last_fullsized":"2024-08-19T14:21:15.876186+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:19:55.511974+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:27.848870+0000","last_clean_scrub_stamp":"2024-08-19T14:19:55.511974+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:16.706651+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,4],"acting":[9,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"359'5168","reported_seq":5226,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-08-19T14:21:19.461482+0000","last_change":"2024-08-19T14:20:29.267682+0000","last_active":"2024-08-19T14:21:19.461482+0000","last_peered":"2024-08-19T14:21:19.461482+0000","last_clean":"2024-08-19T14:21:19.461482+0000","last_became_active":"2024-08-19T14:20:29.267360+0000","last_became_peered":"2024-08-19T14:20:29.267360+0000","last_unstale":"2024-08-19T14:21:19.461482+0000","last_undegraded":"2024-08-19T14:21:19.461482+0000","last_fullsized":"2024-08-19T14:21:19.461482+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":5168,"log_dups_size":0,"ondisk_log_size":5168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:51.551392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":338690048,"num_objects":5168,"num_object_clones":0,"num_object_copies":15504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5168,"num_write_kb":330752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"70'139","reported_seq":647,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-08-19T14:21:15.875770+0000","last_change":"2024-08-19T14:20:12.731993+0000","last_active":"2024-08-19T14:21:15.875770+0000","last_peered":"2024-08-19T14:21:15.875770+0000","last_clean":"2024-08-19T14:21:15.875770+0000","last_became_active":"2024-08-19T14:18:26.881435+0000","last_became_peered":"2024-08-19T14:18:26.881435+0000","last_unstale":"2024-08-19T14:21:15.875770+0000","last_undegraded":"2024-08-19T14:21:15.875770+0000","last_fullsized":"2024-08-19T14:21:15.875770+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:12.731850+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:12.731850+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:37.351039+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,4],"acting":[11,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.1","version":"359'5420","reported_seq":5478,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-08-19T14:21:20.778845+0000","last_change":"2024-08-19T14:20:29.268808+0000","last_active":"2024-08-19T14:21:20.778845+0000","last_peered":"2024-08-19T14:21:20.778845+0000","last_clean":"2024-08-19T14:21:20.778845+0000","last_became_active":"2024-08-19T14:20:29.268465+0000","last_became_peered":"2024-08-19T14:20:29.268465+0000","last_unstale":"2024-08-19T14:21:20.778845+0000","last_undegraded":"2024-08-19T14:21:20.778845+0000","last_fullsized":"2024-08-19T14:21:20.778845+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":5420,"log_dups_size":0,"ondisk_log_size":5420,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:57.700841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":355205120,"num_objects":5420,"num_object_clones":0,"num_object_copies":16260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5420,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5420,"num_write_kb":346880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.4","version":"70'139","reported_seq":871,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-08-19T14:21:15.875517+0000","last_change":"2024-08-19T14:20:04.729769+0000","last_active":"2024-08-19T14:21:15.875517+0000","last_peered":"2024-08-19T14:21:15.875517+0000","last_clean":"2024-08-19T14:21:15.875517+0000","last_became_active":"2024-08-19T14:18:13.806013+0000","last_became_peered":"2024-08-19T14:18:13.806013+0000","last_unstale":"2024-08-19T14:21:15.875517+0000","last_undegraded":"2024-08-19T14:21:15.875517+0000","last_fullsized":"2024-08-19T14:21:15.875517+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:04.729563+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:04.729563+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:26.771562+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.f","version":"359'5125","reported_seq":5183,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-08-19T14:21:19.012015+0000","last_change":"2024-08-19T14:20:29.212298+0000","last_active":"2024-08-19T14:21:19.012015+0000","last_peered":"2024-08-19T14:21:19.012015+0000","last_clean":"2024-08-19T14:21:19.012015+0000","last_became_active":"2024-08-19T14:20:29.212003+0000","last_became_peered":"2024-08-19T14:20:29.212003+0000","last_unstale":"2024-08-19T14:21:19.012015+0000","last_undegraded":"2024-08-19T14:21:19.012015+0000","last_fullsized":"2024-08-19T14:21:19.012015+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":5125,"log_dups_size":0,"ondisk_log_size":5125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:52.621283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335872000,"num_objects":5125,"num_object_clones":0,"num_object_copies":15375,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5125,"num_write_kb":328000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,11],"acting":[1,3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"70'139","reported_seq":841,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-08-19T14:21:15.876263+0000","last_change":"2024-08-19T14:20:12.570453+0000","last_active":"2024-08-19T14:21:15.876263+0000","last_peered":"2024-08-19T14:21:15.876263+0000","last_clean":"2024-08-19T14:21:15.876263+0000","last_became_active":"2024-08-19T14:18:26.875006+0000","last_became_peered":"2024-08-19T14:18:26.875006+0000","last_unstale":"2024-08-19T14:21:15.876263+0000","last_undegraded":"2024-08-19T14:21:15.876263+0000","last_fullsized":"2024-08-19T14:21:15.876263+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:12.570352+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:28.864870+0000","last_clean_scrub_stamp":"2024-08-19T14:20:12.570352+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:32.193930+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,0],"acting":[9,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.c","version":"359'5036","reported_seq":5094,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-08-19T14:21:18.124236+0000","last_change":"2024-08-19T14:20:29.266724+0000","last_active":"2024-08-19T14:21:18.124236+0000","last_peered":"2024-08-19T14:21:18.124236+0000","last_clean":"2024-08-19T14:21:18.124236+0000","last_became_active":"2024-08-19T14:20:29.266373+0000","last_became_peered":"2024-08-19T14:20:29.266373+0000","last_unstale":"2024-08-19T14:21:18.124236+0000","last_undegraded":"2024-08-19T14:21:18.124236+0000","last_fullsized":"2024-08-19T14:21:18.124236+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":5036,"log_dups_size":0,"ondisk_log_size":5036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:48.006455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330039296,"num_objects":5036,"num_object_clones":0,"num_object_copies":15108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5036,"num_write_kb":322304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,8],"acting":[4,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"70'139","reported_seq":956,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-08-19T14:21:15.877453+0000","last_change":"2024-08-19T14:19:56.106047+0000","last_active":"2024-08-19T14:21:15.877453+0000","last_peered":"2024-08-19T14:21:15.877453+0000","last_clean":"2024-08-19T14:21:15.877453+0000","last_became_active":"2024-08-19T14:18:26.887657+0000","last_became_peered":"2024-08-19T14:18:26.887657+0000","last_unstale":"2024-08-19T14:21:15.877453+0000","last_undegraded":"2024-08-19T14:21:15.877453+0000","last_fullsized":"2024-08-19T14:21:15.877453+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:19:56.105880+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:19:56.105880+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:22.062694+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"359'4997","reported_seq":5055,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-08-19T14:21:18.151598+0000","last_change":"2024-08-19T14:20:29.208992+0000","last_active":"2024-08-19T14:21:18.151598+0000","last_peered":"2024-08-19T14:21:18.151598+0000","last_clean":"2024-08-19T14:21:18.151598+0000","last_became_active":"2024-08-19T14:20:29.208599+0000","last_became_peered":"2024-08-19T14:20:29.208599+0000","last_unstale":"2024-08-19T14:21:18.151598+0000","last_undegraded":"2024-08-19T14:21:18.151598+0000","last_fullsized":"2024-08-19T14:21:18.151598+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":4997,"log_dups_size":0,"ondisk_log_size":4997,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:55.930633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327483392,"num_objects":4997,"num_object_clones":0,"num_object_copies":14991,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4997,"num_write_kb":319808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,1],"acting":[4,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"70'139","reported_seq":845,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-08-19T14:21:15.877549+0000","last_change":"2024-08-19T14:20:11.390093+0000","last_active":"2024-08-19T14:21:15.877549+0000","last_peered":"2024-08-19T14:21:15.877549+0000","last_clean":"2024-08-19T14:21:15.877549+0000","last_became_active":"2024-08-19T14:18:26.887277+0000","last_became_peered":"2024-08-19T14:18:26.887277+0000","last_unstale":"2024-08-19T14:21:15.877549+0000","last_undegraded":"2024-08-19T14:21:15.877549+0000","last_fullsized":"2024-08-19T14:21:15.877549+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:11.389938+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:11.389938+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:21.144007+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,8],"acting":[5,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"359'5009","reported_seq":5066,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-08-19T14:21:18.151462+0000","last_change":"2024-08-19T14:20:29.209096+0000","last_active":"2024-08-19T14:21:18.151462+0000","last_peered":"2024-08-19T14:21:18.151462+0000","last_clean":"2024-08-19T14:21:18.151462+0000","last_became_active":"2024-08-19T14:20:29.208748+0000","last_became_peered":"2024-08-19T14:20:29.208748+0000","last_unstale":"2024-08-19T14:21:18.151462+0000","last_undegraded":"2024-08-19T14:21:18.151462+0000","last_fullsized":"2024-08-19T14:21:18.151462+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":5009,"log_dups_size":0,"ondisk_log_size":5009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:56.597251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328269824,"num_objects":5009,"num_object_clones":0,"num_object_copies":15027,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5009,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5009,"num_write_kb":320576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"70'139","reported_seq":847,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-08-19T14:21:15.875524+0000","last_change":"2024-08-19T14:20:11.733961+0000","last_active":"2024-08-19T14:21:15.875524+0000","last_peered":"2024-08-19T14:21:15.875524+0000","last_clean":"2024-08-19T14:21:15.875524+0000","last_became_active":"2024-08-19T14:18:26.875179+0000","last_became_peered":"2024-08-19T14:18:26.875179+0000","last_unstale":"2024-08-19T14:21:15.875524+0000","last_undegraded":"2024-08-19T14:21:15.875524+0000","last_fullsized":"2024-08-19T14:21:15.875524+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:11.733865+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:11.733865+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:41.250361+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.b","version":"359'5333","reported_seq":5391,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-08-19T14:21:21.167875+0000","last_change":"2024-08-19T14:20:29.207845+0000","last_active":"2024-08-19T14:21:21.167875+0000","last_peered":"2024-08-19T14:21:21.167875+0000","last_clean":"2024-08-19T14:21:21.167875+0000","last_became_active":"2024-08-19T14:20:29.207372+0000","last_became_peered":"2024-08-19T14:20:29.207372+0000","last_unstale":"2024-08-19T14:21:21.167875+0000","last_undegraded":"2024-08-19T14:21:21.167875+0000","last_fullsized":"2024-08-19T14:21:21.167875+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":5333,"log_dups_size":0,"ondisk_log_size":5333,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:33.577033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":349503488,"num_objects":5333,"num_object_clones":0,"num_object_copies":15999,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5333,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5333,"num_write_kb":341312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"70'139","reported_seq":501,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-08-19T14:21:15.876330+0000","last_change":"2024-08-19T14:20:15.322051+0000","last_active":"2024-08-19T14:21:15.876330+0000","last_peered":"2024-08-19T14:21:15.876330+0000","last_clean":"2024-08-19T14:21:15.876330+0000","last_became_active":"2024-08-19T14:18:26.884105+0000","last_became_peered":"2024-08-19T14:18:26.884105+0000","last_unstale":"2024-08-19T14:21:15.876330+0000","last_undegraded":"2024-08-19T14:21:15.876330+0000","last_fullsized":"2024-08-19T14:21:15.876330+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:15.321945+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:20:15.321945+0000","last_clean_scrub_stamp":"2024-08-19T14:20:15.321945+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:43.904541+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"359'5241","reported_seq":5299,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-08-19T14:21:19.018078+0000","last_change":"2024-08-19T14:20:29.269723+0000","last_active":"2024-08-19T14:21:19.018078+0000","last_peered":"2024-08-19T14:21:19.018078+0000","last_clean":"2024-08-19T14:21:19.018078+0000","last_became_active":"2024-08-19T14:20:29.269432+0000","last_became_peered":"2024-08-19T14:20:29.269432+0000","last_unstale":"2024-08-19T14:21:19.018078+0000","last_undegraded":"2024-08-19T14:21:19.018078+0000","last_fullsized":"2024-08-19T14:21:19.018078+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":5241,"log_dups_size":0,"ondisk_log_size":5241,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:44.639451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":343474176,"num_objects":5241,"num_object_clones":0,"num_object_copies":15723,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5241,"num_write_kb":335424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.f","version":"70'139","reported_seq":753,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-08-19T14:21:15.878182+0000","last_change":"2024-08-19T14:21:14.765419+0000","last_active":"2024-08-19T14:21:15.878182+0000","last_peered":"2024-08-19T14:21:15.878182+0000","last_clean":"2024-08-19T14:21:15.878182+0000","last_became_active":"2024-08-19T14:18:18.711708+0000","last_became_peered":"2024-08-19T14:18:18.711708+0000","last_unstale":"2024-08-19T14:21:15.878182+0000","last_undegraded":"2024-08-19T14:21:15.878182+0000","last_fullsized":"2024-08-19T14:21:15.878182+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:14.765340+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:21:14.765340+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:22:24.415219+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"359'5313","reported_seq":5371,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-08-19T14:21:20.774165+0000","last_change":"2024-08-19T14:20:29.212320+0000","last_active":"2024-08-19T14:21:20.774165+0000","last_peered":"2024-08-19T14:21:20.774165+0000","last_clean":"2024-08-19T14:21:20.774165+0000","last_became_active":"2024-08-19T14:20:29.211975+0000","last_became_peered":"2024-08-19T14:20:29.211975+0000","last_unstale":"2024-08-19T14:21:20.774165+0000","last_undegraded":"2024-08-19T14:21:20.774165+0000","last_fullsized":"2024-08-19T14:21:20.774165+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":5313,"log_dups_size":0,"ondisk_log_size":5313,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:28.509656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":348192768,"num_objects":5313,"num_object_clones":0,"num_object_copies":15939,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5313,"num_write_kb":340032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,3],"acting":[11,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.c","version":"70'139","reported_seq":606,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-08-19T14:21:15.876991+0000","last_change":"2024-08-19T14:20:07.839218+0000","last_active":"2024-08-19T14:21:15.876991+0000","last_peered":"2024-08-19T14:21:15.876991+0000","last_clean":"2024-08-19T14:21:15.876991+0000","last_became_active":"2024-08-19T14:18:26.872942+0000","last_became_peered":"2024-08-19T14:18:26.872942+0000","last_unstale":"2024-08-19T14:21:15.876991+0000","last_undegraded":"2024-08-19T14:21:15.876991+0000","last_fullsized":"2024-08-19T14:21:15.876991+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:07.839076+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:20:07.839076+0000","last_clean_scrub_stamp":"2024-08-19T14:20:07.839076+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:33.185668+0000","scrub_duration":62,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,9],"acting":[3,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"359'5286","reported_seq":5344,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-08-19T14:21:20.781971+0000","last_change":"2024-08-19T14:20:29.268318+0000","last_active":"2024-08-19T14:21:20.781971+0000","last_peered":"2024-08-19T14:21:20.781971+0000","last_clean":"2024-08-19T14:21:20.781971+0000","last_became_active":"2024-08-19T14:20:29.267442+0000","last_became_peered":"2024-08-19T14:20:29.267442+0000","last_unstale":"2024-08-19T14:21:20.781971+0000","last_undegraded":"2024-08-19T14:21:20.781971+0000","last_fullsized":"2024-08-19T14:21:20.781971+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":5286,"log_dups_size":0,"ondisk_log_size":5286,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:47.715801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":346423296,"num_objects":5286,"num_object_clones":0,"num_object_copies":15858,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5286,"num_write_kb":338304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,9],"acting":[11,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.d","version":"70'139","reported_seq":529,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-08-19T14:21:15.878284+0000","last_change":"2024-08-19T14:21:07.155294+0000","last_active":"2024-08-19T14:21:15.878284+0000","last_peered":"2024-08-19T14:21:15.878284+0000","last_clean":"2024-08-19T14:21:15.878284+0000","last_became_active":"2024-08-19T14:18:26.882002+0000","last_became_peered":"2024-08-19T14:18:26.882002+0000","last_unstale":"2024-08-19T14:21:15.878284+0000","last_undegraded":"2024-08-19T14:21:15.878284+0000","last_fullsized":"2024-08-19T14:21:15.878284+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:07.155157+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:21:07.155157+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:22:11.274708+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"70'139","reported_seq":788,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-08-19T14:21:15.879238+0000","last_change":"2024-08-19T14:21:13.777891+0000","last_active":"2024-08-19T14:21:15.879238+0000","last_peered":"2024-08-19T14:21:15.879238+0000","last_clean":"2024-08-19T14:21:15.879238+0000","last_became_active":"2024-08-19T14:18:18.703594+0000","last_became_peered":"2024-08-19T14:18:18.703594+0000","last_unstale":"2024-08-19T14:21:15.879238+0000","last_undegraded":"2024-08-19T14:21:15.879238+0000","last_fullsized":"2024-08-19T14:21:15.879238+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:13.777777+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:21:13.777777+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:22:38.473560+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"70'139","reported_seq":840,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-08-19T14:21:15.876006+0000","last_change":"2024-08-19T14:20:07.692054+0000","last_active":"2024-08-19T14:21:15.876006+0000","last_peered":"2024-08-19T14:21:15.876006+0000","last_clean":"2024-08-19T14:21:15.876006+0000","last_became_active":"2024-08-19T14:18:18.698334+0000","last_became_peered":"2024-08-19T14:18:18.698334+0000","last_unstale":"2024-08-19T14:21:15.876006+0000","last_undegraded":"2024-08-19T14:21:15.876006+0000","last_fullsized":"2024-08-19T14:21:15.876006+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:07.691932+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:07.691932+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:36.578885+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.10","version":"70'139","reported_seq":943,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-08-19T14:21:15.877464+0000","last_change":"2024-08-19T14:21:07.768770+0000","last_active":"2024-08-19T14:21:15.877464+0000","last_peered":"2024-08-19T14:21:15.877464+0000","last_clean":"2024-08-19T14:21:15.877464+0000","last_became_active":"2024-08-19T14:18:18.704562+0000","last_became_peered":"2024-08-19T14:18:18.704562+0000","last_unstale":"2024-08-19T14:21:15.877464+0000","last_undegraded":"2024-08-19T14:21:15.877464+0000","last_fullsized":"2024-08-19T14:21:15.877464+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:07.768697+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:21:07.768697+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:22:10.360289+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,8],"acting":[5,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"70'139","reported_seq":492,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-08-19T14:21:16.071775+0000","last_change":"2024-08-19T14:21:16.071812+0000","last_active":"2024-08-19T14:21:16.071775+0000","last_peered":"2024-08-19T14:21:16.071775+0000","last_clean":"2024-08-19T14:21:16.071775+0000","last_became_active":"2024-08-19T14:18:18.697894+0000","last_became_peered":"2024-08-19T14:18:18.697894+0000","last_unstale":"2024-08-19T14:21:16.071775+0000","last_undegraded":"2024-08-19T14:21:16.071775+0000","last_fullsized":"2024-08-19T14:21:16.071775+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:16.071703+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:21:16.071703+0000","last_clean_scrub_stamp":"2024-08-19T14:21:16.071703+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:22:18.184091+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"70'139","reported_seq":469,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-08-19T14:21:15.952894+0000","last_change":"2024-08-19T14:20:28.205212+0000","last_active":"2024-08-19T14:21:15.952894+0000","last_peered":"2024-08-19T14:21:15.952894+0000","last_clean":"2024-08-19T14:21:15.952894+0000","last_became_active":"2024-08-19T14:20:28.204724+0000","last_became_peered":"2024-08-19T14:20:28.204724+0000","last_unstale":"2024-08-19T14:21:15.952894+0000","last_undegraded":"2024-08-19T14:21:15.952894+0000","last_fullsized":"2024-08-19T14:21:15.952894+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:08.680021+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:08.680021+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:28.441081+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"70'139","reported_seq":589,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-08-19T14:21:15.878263+0000","last_change":"2024-08-19T14:20:31.931658+0000","last_active":"2024-08-19T14:21:15.878263+0000","last_peered":"2024-08-19T14:21:15.878263+0000","last_clean":"2024-08-19T14:21:15.878263+0000","last_became_active":"2024-08-19T14:18:31.901556+0000","last_became_peered":"2024-08-19T14:18:31.901556+0000","last_unstale":"2024-08-19T14:21:15.878263+0000","last_undegraded":"2024-08-19T14:21:15.878263+0000","last_fullsized":"2024-08-19T14:21:15.878263+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:31.931562+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:20:31.931562+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:47.687921+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,0],"acting":[8,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.14","version":"70'139","reported_seq":598,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-08-19T14:21:15.877901+0000","last_change":"2024-08-19T14:19:55.130518+0000","last_active":"2024-08-19T14:21:15.877901+0000","last_peered":"2024-08-19T14:21:15.877901+0000","last_clean":"2024-08-19T14:21:15.877901+0000","last_became_active":"2024-08-19T14:18:18.703926+0000","last_became_peered":"2024-08-19T14:18:18.703926+0000","last_unstale":"2024-08-19T14:21:15.877901+0000","last_undegraded":"2024-08-19T14:21:15.877901+0000","last_fullsized":"2024-08-19T14:21:15.877901+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:19:55.130420+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:19:55.130420+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,1],"acting":[7,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.15","version":"70'139","reported_seq":485,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-08-19T14:21:15.878186+0000","last_change":"2024-08-19T14:20:09.181046+0000","last_active":"2024-08-19T14:21:15.878186+0000","last_peered":"2024-08-19T14:21:15.878186+0000","last_clean":"2024-08-19T14:21:15.878186+0000","last_became_active":"2024-08-19T14:18:31.912496+0000","last_became_peered":"2024-08-19T14:18:31.912496+0000","last_unstale":"2024-08-19T14:21:15.878186+0000","last_undegraded":"2024-08-19T14:21:15.878186+0000","last_fullsized":"2024-08-19T14:21:15.878186+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:09.180923+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:20:09.180923+0000","last_clean_scrub_stamp":"2024-08-19T14:20:09.180923+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:21.100428+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"70'139","reported_seq":463,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-08-19T14:21:15.876167+0000","last_change":"2024-08-19T14:20:17.333347+0000","last_active":"2024-08-19T14:21:15.876167+0000","last_peered":"2024-08-19T14:21:15.876167+0000","last_clean":"2024-08-19T14:21:15.876167+0000","last_became_active":"2024-08-19T14:18:31.901229+0000","last_became_peered":"2024-08-19T14:18:31.901229+0000","last_unstale":"2024-08-19T14:21:15.876167+0000","last_undegraded":"2024-08-19T14:21:15.876167+0000","last_fullsized":"2024-08-19T14:21:15.876167+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:17.333260+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:17.333260+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:23.889383+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1b","version":"70'139","reported_seq":815,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-08-19T14:21:15.878092+0000","last_change":"2024-08-19T14:20:16.776086+0000","last_active":"2024-08-19T14:21:15.878092+0000","last_peered":"2024-08-19T14:21:15.878092+0000","last_clean":"2024-08-19T14:21:15.878092+0000","last_became_active":"2024-08-19T14:18:31.901152+0000","last_became_peered":"2024-08-19T14:18:31.901152+0000","last_unstale":"2024-08-19T14:21:15.878092+0000","last_undegraded":"2024-08-19T14:21:15.878092+0000","last_fullsized":"2024-08-19T14:21:15.878092+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:16.775987+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:16.775987+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:46.485727+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.18","version":"70'139","reported_seq":859,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-08-19T14:21:15.877366+0000","last_change":"2024-08-19T14:20:36.510554+0000","last_active":"2024-08-19T14:21:15.877366+0000","last_peered":"2024-08-19T14:21:15.877366+0000","last_clean":"2024-08-19T14:21:15.877366+0000","last_became_active":"2024-08-19T14:18:31.902735+0000","last_became_peered":"2024-08-19T14:18:31.902735+0000","last_unstale":"2024-08-19T14:21:15.877366+0000","last_undegraded":"2024-08-19T14:21:15.877366+0000","last_fullsized":"2024-08-19T14:21:15.877366+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:36.510442+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:36.510442+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:53.706735+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.19","version":"70'139","reported_seq":807,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-08-19T14:21:15.877321+0000","last_change":"2024-08-19T14:20:32.919627+0000","last_active":"2024-08-19T14:21:15.877321+0000","last_peered":"2024-08-19T14:21:15.877321+0000","last_clean":"2024-08-19T14:21:15.877321+0000","last_became_active":"2024-08-19T14:18:31.902547+0000","last_became_peered":"2024-08-19T14:18:31.902547+0000","last_unstale":"2024-08-19T14:21:15.877321+0000","last_undegraded":"2024-08-19T14:21:15.877321+0000","last_fullsized":"2024-08-19T14:21:15.877321+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:32.919545+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:20:32.919545+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:35.916636+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1e","version":"70'139","reported_seq":465,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-08-19T14:21:15.875791+0000","last_change":"2024-08-19T14:20:18.771836+0000","last_active":"2024-08-19T14:21:15.875791+0000","last_peered":"2024-08-19T14:21:15.875791+0000","last_clean":"2024-08-19T14:21:15.875791+0000","last_became_active":"2024-08-19T14:18:31.902546+0000","last_became_peered":"2024-08-19T14:18:31.902546+0000","last_unstale":"2024-08-19T14:21:15.875791+0000","last_undegraded":"2024-08-19T14:21:15.875791+0000","last_fullsized":"2024-08-19T14:21:15.875791+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:18.771704+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:18.771704+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:47.154907+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1c","version":"70'139","reported_seq":573,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-08-19T14:21:15.879651+0000","last_change":"2024-08-19T14:20:08.824736+0000","last_active":"2024-08-19T14:21:15.879651+0000","last_peered":"2024-08-19T14:21:15.879651+0000","last_clean":"2024-08-19T14:21:15.879651+0000","last_became_active":"2024-08-19T14:18:31.902494+0000","last_became_peered":"2024-08-19T14:18:31.902494+0000","last_unstale":"2024-08-19T14:21:15.879651+0000","last_undegraded":"2024-08-19T14:21:15.879651+0000","last_fullsized":"2024-08-19T14:21:15.879651+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:08.824505+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:08.824505+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:32.216640+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,11],"acting":[8,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1d","version":"70'139","reported_seq":495,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-08-19T14:21:15.875265+0000","last_change":"2024-08-19T14:20:17.725615+0000","last_active":"2024-08-19T14:21:15.875265+0000","last_peered":"2024-08-19T14:21:15.875265+0000","last_clean":"2024-08-19T14:21:15.875265+0000","last_became_active":"2024-08-19T14:18:31.901183+0000","last_became_peered":"2024-08-19T14:18:31.901183+0000","last_unstale":"2024-08-19T14:21:15.875265+0000","last_undegraded":"2024-08-19T14:21:15.875265+0000","last_fullsized":"2024-08-19T14:21:15.875265+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:17.725477+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:17.725477+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:25.545335+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,5],"acting":[1,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":5456396288,"num_objects":83258,"num_object_clones":0,"num_object_copies":249774,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3854,"num_objects_unfound":0,"num_objects_dirty":83258,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":83258,"num_write_kb":5328512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":286,"num_bytes_recovered":18743296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16361259008,"data_stored":16361259008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83258,"ondisk_log_size":83258,"up":48,"acting":47,"num_store_stats":9},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1802240,"data_stored":1788032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187576,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2750512,"kb_used_data":2362536,"kb_used_omap":1,"kb_used_meta":387966,"kb_avail":90994640,"statfs":{"total":95995035648,"available":93178511360,"internally_reserved":0,"allocated":2419236864,"data_stored":2417575640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":397277636},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":343597383805,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":74,"seq":317827580033,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1488880,"kb_used_data":1034280,"kb_used_omap":1,"kb_used_meta":454590,"kb_avail":92256272,"statfs":{"total":95995035648,"available":94470422528,"internally_reserved":0,"allocated":1059102720,"data_stored":1057441496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":465500609},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874375,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":713840,"kb_used_data":335592,"kb_used_omap":1,"kb_used_meta":378238,"kb_avail":93031312,"statfs":{"total":95995035648,"available":95264063488,"internally_reserved":0,"allocated":343646208,"data_stored":341984984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":387316165},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":57,"seq":244813136009,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3040060,"kb_used_data":2611824,"kb_used_omap":1,"kb_used_meta":428222,"kb_avail":90705092,"statfs":{"total":95995035648,"available":92882014208,"internally_reserved":0,"allocated":2674507776,"data_stored":2672843000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":438499786},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":206158430349,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691986,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1655096,"kb_used_data":1304624,"kb_used_omap":1,"kb_used_meta":350462,"kb_avail":92090056,"statfs":{"total":95995035648,"available":94300217344,"internally_reserved":0,"allocated":1335934976,"data_stored":1334270200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":358873541},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986326,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2062384,"kb_used_data":1598056,"kb_used_omap":1,"kb_used_meta":464318,"kb_avail":91682768,"statfs":{"total":95995035648,"available":93883154432,"internally_reserved":0,"allocated":1636409344,"data_stored":1634748120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":475462085},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,1,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149852,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2386480,"kb_used_data":1976296,"kb_used_omap":1,"kb_used_meta":410174,"kb_avail":91358672,"statfs":{"total":95995035648,"available":93551280128,"internally_reserved":0,"allocated":2023727104,"data_stored":2022065880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":420018631},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411490,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575013,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3695096,"kb_used_data":3252976,"kb_used_omap":1,"kb_used_meta":442110,"kb_avail":90050056,"statfs":{"total":95995035648,"available":92211257344,"internally_reserved":0,"allocated":3331047424,"data_stored":3329382648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":452721093},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,1,2,2,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738538,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1709168,"kb_used_data":1375336,"kb_used_omap":1,"kb_used_meta":333822,"kb_avail":92035984,"statfs":{"total":95995035648,"available":94244847616,"internally_reserved":0,"allocated":1408344064,"data_stored":1406682840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":341834181},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1407451136,"data_stored":1407451136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3371302912,"data_stored":3371302912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2031878144,"data_stored":2031878144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1655111680,"data_stored":1655111680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1345126400,"data_stored":1345126400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2685992960,"data_stored":2685992960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":339738624,"data_stored":339738624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1079967744,"data_stored":1079967744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":2444689408,"data_stored":2444689408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T14:21:24.493 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph status --format=json 2024-08-19T14:21:24.521 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:21:24.557 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:24.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:24.558 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:21:24.659 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:21:24.701 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:24.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:24.702 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:21:24.731 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:21:24.786 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-19T14:21:24.804 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:21:24.827 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:24.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:24.828 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:21:24.929 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:21:24.970 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:24.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:24.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:21:25.049 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 51 16 90082 90066 110.365 63.3125 0.0198453 0.00905578 2024-08-19T14:21:25.073 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:21:25.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:25.095 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:25.095 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:21:25.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:24 smithi031 ceph-mon[16336]: pgmap v805: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.2 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 66 MiB/s wr, 1.06k op/s; 3854/254820 objects misplaced (1.512%); 1.1 MiB/s, 16 objects/s recovering 2024-08-19T14:21:25.156 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:24 smithi032 ceph-mon[20004]: pgmap v805: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.2 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 66 MiB/s wr, 1.06k op/s; 3854/254820 objects misplaced (1.512%); 1.1 MiB/s, 16 objects/s recovering 2024-08-19T14:21:25.196 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:21:25.197 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:24 smithi045 ceph-mon[20143]: pgmap v805: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.2 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 66 MiB/s wr, 1.06k op/s; 3854/254820 objects misplaced (1.512%); 1.1 MiB/s, 16 objects/s recovering 2024-08-19T14:21:25.224 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:25.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:25.225 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:21:25.297 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:21:25.298 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:21:25.326 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:21:25.363 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:25.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:25.364 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:21:25.465 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:21:25.489 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:25.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:25.490 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:21:25.591 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:21:25.618 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:25.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:25.619 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:21:25.719 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:21:25.766 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:25.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:25.767 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:21:25.868 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:21:25.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:25.900 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:25.900 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:21:26.001 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:21:26.027 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:26.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:26.028 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:21:26.048 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 52 15 91258 91243 109.658 73.5625 0.00757212 0.00911472 2024-08-19T14:21:26.128 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:21:26.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:26.164 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:26.164 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:21:26.258 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-08-19T14:21:26.265 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:21:26.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:26.294 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:26.294 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:21:26.366 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:21:26.368 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:21:26.395 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:21:26.450 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:26.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:26.451 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:21:26.552 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:21:26.574 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:21:26.579 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:26.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:26.580 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:21:26.681 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:21:26.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:26.705 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:26.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:21:26.806 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:21:26.832 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:26.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:26.833 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:21:26.934 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:21:26.963 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:26.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:26.964 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:21:27.000 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:26 smithi032 ceph-mon[20004]: pgmap v806: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.4 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 80 MiB/s wr, 1.29k op/s; 3737/267852 objects misplaced (1.395%); 1.6 MiB/s, 25 objects/s recovering 2024-08-19T14:21:27.048 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 53 16 92371 92355 108.9 69.5 0.00414159 0.00917569 2024-08-19T14:21:27.065 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:21:27.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:27.101 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:27.101 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:21:27.137 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:21:27.137 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:21:27.203 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:21:27.238 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:27.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:27.239 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:21:27.277 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:26 smithi031 ceph-mon[16336]: pgmap v806: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.4 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 80 MiB/s wr, 1.29k op/s; 3737/267852 objects misplaced (1.395%); 1.6 MiB/s, 25 objects/s recovering 2024-08-19T14:21:27.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:26 smithi045 ceph-mon[20143]: pgmap v806: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.4 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 80 MiB/s wr, 1.29k op/s; 3737/267852 objects misplaced (1.395%); 1.6 MiB/s, 25 objects/s recovering 2024-08-19T14:21:27.340 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:21:27.374 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:27.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:27.375 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:21:27.477 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:21:27.508 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:27.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:27.509 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:21:27.611 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:21:27.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:27.650 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:27.650 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:21:27.751 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:21:27.775 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:27.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:27.776 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:21:27.863 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-08-19T14:21:27.877 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:21:27.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:27.934 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:27.935 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:21:28.036 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:21:28.048 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 54 16 93666 93650 108.382 80.9375 0.00708233 0.00922159 2024-08-19T14:21:28.070 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:28.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:28.071 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:21:28.172 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:21:28.202 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:21:28.203 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:28.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:28.204 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:21:28.305 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:21:28.329 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:28.329 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:21:28.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:28.330 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:21:28.431 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:21:28.478 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:28.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:28.479 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:21:28.502 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:21:28.502 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"ed7d1c6e-5e33-11ef-bccf-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub flag(s) set","count":7},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":865,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":360,"num_osds":12,"num_up_osds":12,"osd_up_since":1724076699,"num_in_osds":9,"osd_in_since":1724076998,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":46},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":47,"num_pools":2,"num_objects":89284,"data_bytes":5851775520,"bytes_used":21133037568,"bytes_avail":842822283264,"bytes_total":863955320832,"misplaced_objects":3737,"misplaced_total":267852,"misplaced_ratio":0.013951734539969835,"recovering_objects_per_sec":25,"recovering_bytes_per_sec":1651014,"recovering_keys_per_sec":0,"num_objects_recovered":252,"num_bytes_recovered":16515072,"num_keys_recovered":0,"write_bytes_sec":84300013,"read_op_per_sec":0,"write_op_per_sec":1286},"fsmap":{"epoch":1,"btime":"2024-08-19T14:05:22:841940+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.31:8443/"}},"servicemap":{"epoch":20,"modified":"2024-08-19T14:20:27.873686+0000","services":{}},"progress_events":{}} 2024-08-19T14:21:28.580 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:21:28.607 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:28.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:28.608 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:21:28.709 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:21:28.745 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is set 2024-08-19T14:21:28.749 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:28.749 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:21:28.750 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:21:28.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:28.750 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:21:28.770 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:28 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1966188376' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-08-19T14:21:28.770 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:28 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1707959587' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:21:28.851 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:21:28.876 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:28.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:28.877 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:21:28.978 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:21:29.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:28 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1966188376' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-08-19T14:21:29.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:28 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1707959587' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:21:29.014 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:29.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:29.015 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:21:29.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:28 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1966188376' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-08-19T14:21:29.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:28 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1707959587' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:21:29.048 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 55 16 94762 94746 107.657 68.5 0.0109322 0.00928234 2024-08-19T14:21:29.116 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:21:29.142 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:29.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:29.143 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:21:29.245 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:21:29.272 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:29.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:29.274 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:21:29.375 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:21:29.401 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:29.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:29.402 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:21:29.440 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-08-19T14:21:29.503 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:21:29.528 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-08-19T14:21:29.529 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-19T14:21:29.532 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:29.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:29.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:21:29.635 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:21:29.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:29.682 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:29.682 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:21:29.780 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:21:29.783 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:21:29.818 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:29.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:29.819 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:21:29.920 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:21:29.947 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:29.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:29.948 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:21:29.983 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:21:30.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:29 smithi032 ceph-mon[20004]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:21:30.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:29 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1707959587' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:21:30.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:29 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:21:30.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:29 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:21:30.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:29 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:21:30.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:29 smithi032 ceph-mon[20004]: osdmap e361: 12 total, 12 up, 9 in 2024-08-19T14:21:30.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:29 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:21:30.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:29 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:21:30.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:29 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:21:30.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:29 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:21:30.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:29 smithi045 ceph-mon[20143]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:21:30.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:29 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1707959587' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:21:30.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:29 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:21:30.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:29 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:21:30.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:29 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:21:30.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:29 smithi045 ceph-mon[20143]: osdmap e361: 12 total, 12 up, 9 in 2024-08-19T14:21:30.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:29 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:21:30.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:29 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:21:30.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:29 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:21:30.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:29 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:21:30.048 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:21:30.049 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 56 16 96018 96002 107.136 78.5 0.0149381 0.00932907 2024-08-19T14:21:30.074 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:30.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:30.075 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:21:30.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:29 smithi031 ceph-mon[16336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:21:30.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:29 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1707959587' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:21:30.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:29 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:21:30.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:29 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:21:30.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:29 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:21:30.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:29 smithi031 ceph-mon[16336]: osdmap e361: 12 total, 12 up, 9 in 2024-08-19T14:21:30.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:29 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:21:30.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:29 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:21:30.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:29 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:21:30.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:29 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:21:30.175 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:21:30.204 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:30.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:30.205 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:21:30.306 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:21:30.340 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:30.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:30.341 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:21:30.356 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:21:30.357 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:21:30.441 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:21:30.486 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:30.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:30.487 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:21:30.588 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:21:30.613 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:21:30.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:30.614 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:30.614 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:21:30.636 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:21:30.636 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:21:30.674 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:21:30.675 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:21:30.706 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:21:30.706 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:21:30.716 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:21:30.738 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:21:30.739 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:21:30.740 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:30.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:30.742 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:21:30.843 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:21:30.870 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:30.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:30.871 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:21:30.971 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:21:30.971 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:21:30.971 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:30 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/24691596' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:21:30.972 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:21:30.994 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:30.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:30.995 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:21:31.006 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:21:31.006 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:21:31.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:30 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/24691596' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:21:31.037 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:21:31.037 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:21:31.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:30 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/24691596' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:21:31.049 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 57 16 97231 97215 106.586 75.8125 0.00835508 0.00937767 2024-08-19T14:21:31.073 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:21:31.073 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:21:31.085 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-08-19T14:21:31.096 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:21:31.111 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:21:31.111 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:21:31.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:31.133 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:31.134 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:21:31.143 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:21:31.143 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:21:31.172 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:21:31.172 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:21:31.206 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:21:31.206 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:21:31.235 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:21:31.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:31.272 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:31.272 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:21:31.373 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:21:31.403 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:31.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:31.405 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:21:31.457 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:21:31.505 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:21:31.535 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:31.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:31.536 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:21:31.637 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:21:31.665 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:31.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:31.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:21:31.768 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:21:31.801 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:31.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:31.803 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:21:31.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:31 smithi031 ceph-mon[16336]: pgmap v809: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.7 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 77 MiB/s wr, 1.23k op/s; 3603/279357 objects misplaced (1.290%); 2.0 MiB/s, 31 objects/s recovering 2024-08-19T14:21:31.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:31 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:21:31.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:31 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:21:31.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:31 smithi031 ceph-mon[16336]: osdmap e362: 12 total, 12 up, 9 in 2024-08-19T14:21:31.903 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:21:31.933 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:31 smithi032 ceph-mon[20004]: pgmap v809: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.7 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 77 MiB/s wr, 1.23k op/s; 3603/279357 objects misplaced (1.290%); 2.0 MiB/s, 31 objects/s recovering 2024-08-19T14:21:31.933 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:31 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:21:31.933 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:31 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:21:31.933 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:31 smithi032 ceph-mon[20004]: osdmap e362: 12 total, 12 up, 9 in 2024-08-19T14:21:31.934 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:31.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:31.935 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:21:31.992 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-19T14:21:32.036 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:21:32.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:31 smithi045 ceph-mon[20143]: pgmap v809: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.7 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 77 MiB/s wr, 1.23k op/s; 3603/279357 objects misplaced (1.290%); 2.0 MiB/s, 31 objects/s recovering 2024-08-19T14:21:32.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:31 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:21:32.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:31 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:21:32.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:31 smithi045 ceph-mon[20143]: osdmap e362: 12 total, 12 up, 9 in 2024-08-19T14:21:32.049 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 58 16 98528 98512 106.146 81.0625 0.00700852 0.00941626 2024-08-19T14:21:32.065 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:32.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:32.066 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:21:32.167 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:21:32.198 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:32.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:32.200 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:21:32.301 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:21:32.329 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:32.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:32.331 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:21:32.432 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:21:32.456 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:32.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:32.457 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:21:32.558 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:21:32.584 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:32.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:32.585 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:21:32.686 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:21:32.717 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:32.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:32.719 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:21:32.820 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:21:32.852 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:32.853 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:21:32.853 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-08-19T14:21:32.853 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-08-19T14:21:32.853 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:21:32.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:32.854 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:21:32.955 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:21:32.982 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:32.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:32.983 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:21:33.049 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 59 16 99758 99742 105.65 76.875 0.00891865 0.00946056 2024-08-19T14:21:33.084 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:21:33.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:33.110 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:33.110 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:21:33.211 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:21:33.241 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:33.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:33.242 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:21:33.343 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:21:33.370 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:33.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:33.371 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:21:33.472 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:21:33.495 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:33.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:33.496 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:21:33.580 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-08-19T14:21:33.597 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:21:33.652 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:33.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:33.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:21:33.754 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:21:33.779 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:33.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:33.780 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:21:33.846 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:21:33.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:33 smithi031 ceph-mon[16336]: pgmap v811: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.8 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 82 MiB/s wr, 1.31k op/s; 3603/286182 objects misplaced (1.259%); 2.0 MiB/s, 31 objects/s recovering 2024-08-19T14:21:33.881 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:21:33.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:33.916 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:33.916 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:21:33.931 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:21:34.017 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:21:34.043 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:34.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:34.045 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:21:34.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:33 smithi045 ceph-mon[20143]: pgmap v811: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.8 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 82 MiB/s wr, 1.31k op/s; 3603/286182 objects misplaced (1.259%); 2.0 MiB/s, 31 objects/s recovering 2024-08-19T14:21:34.049 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:2024-08-19T14:21:34.049771+0000 min lat: 0.00106793 max lat: 2.42694 avg lat: 0.0095067 2024-08-19T14:21:34.049 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-19T14:21:34.049 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 60 16 100956 100940 105.137 74.875 0.0060752 0.0095067 2024-08-19T14:21:34.062 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:33 smithi032 ceph-mon[20004]: pgmap v811: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.8 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 82 MiB/s wr, 1.31k op/s; 3603/286182 objects misplaced (1.259%); 2.0 MiB/s, 31 objects/s recovering 2024-08-19T14:21:34.065 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:21:34.068 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-08-19T14:21:34.146 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:21:34.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:34.185 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:34.185 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:21:34.287 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:21:34.324 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:34.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:34.325 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:21:34.426 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:21:34.461 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:34.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:34.462 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:21:34.563 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:21:34.592 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:34.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:34.593 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:21:34.694 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:21:34.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:34.720 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:34.720 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:21:34.766 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:21:34.766 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-08-19T14:21:34.766 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-08-19T14:21:34.766 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-08-19T14:21:34.766 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:21:34.821 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:21:34.851 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:34.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:34.852 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:21:34.875 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":812,"stamp":"2024-08-19T14:21:33.899293+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6353977888,"num_objects":96947,"num_object_clones":0,"num_object_copies":290841,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3603,"num_objects_unfound":0,"num_objects_dirty":96947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":96945,"num_write_kb":6204480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":537,"num_bytes_recovered":35192832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":101254,"ondisk_log_size":101254,"up":141,"acting":140,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":142,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":22331880,"kb_used_data":18661064,"kb_used_omap":14,"kb_used_meta":3670705,"kb_avail":821374488,"statfs":{"total":863955320832,"available":841087475712,"internally_reserved":0,"allocated":19108929536,"data_stored":19093875592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14353,"internal_metadata":3758802927},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":502202368,"num_objects":7663,"num_object_clones":0,"num_object_copies":22989,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-134,"num_objects_unfound":0,"num_objects_dirty":7663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7663,"num_write_kb":490432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":134,"num_bytes_recovered":8781824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.004605"},"pg_stats":[{"pgid":"4.5","version":"362'6115","reported_seq":6179,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-08-19T14:21:30.777356+0000","last_change":"2024-08-19T14:20:29.268214+0000","last_active":"2024-08-19T14:21:30.777356+0000","last_peered":"2024-08-19T14:21:30.777356+0000","last_clean":"2024-08-19T14:21:30.777356+0000","last_became_active":"2024-08-19T14:20:29.267549+0000","last_became_peered":"2024-08-19T14:20:29.267549+0000","last_unstale":"2024-08-19T14:21:30.777356+0000","last_undegraded":"2024-08-19T14:21:30.777356+0000","last_fullsized":"2024-08-19T14:21:30.777356+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":6115,"log_dups_size":0,"ondisk_log_size":6115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:57.640230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400752640,"num_objects":6115,"num_object_clones":0,"num_object_copies":18345,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6115,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6115,"num_write_kb":391360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,0],"acting":[11,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"70'139","reported_seq":940,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-08-19T14:21:30.754216+0000","last_change":"2024-08-19T14:19:58.072227+0000","last_active":"2024-08-19T14:21:30.754216+0000","last_peered":"2024-08-19T14:21:30.754216+0000","last_clean":"2024-08-19T14:21:30.754216+0000","last_became_active":"2024-08-19T14:18:13.807133+0000","last_became_peered":"2024-08-19T14:18:13.807133+0000","last_unstale":"2024-08-19T14:21:30.754216+0000","last_undegraded":"2024-08-19T14:21:30.754216+0000","last_fullsized":"2024-08-19T14:21:30.754216+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:19:58.072083+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:19:58.072083+0000","last_clean_scrub_stamp":"2024-08-19T14:19:58.072083+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,9],"acting":[7,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"361'5951","reported_seq":7090,"reported_epoch":361,"state":"active+recovering+undersized+remapped","last_fresh":"2024-08-19T14:21:29.066378+0000","last_change":"2024-08-19T14:21:07.239135+0000","last_active":"2024-08-19T14:21:29.066378+0000","last_peered":"2024-08-19T14:21:29.066378+0000","last_clean":"2024-08-19T14:21:04.784061+0000","last_became_active":"2024-08-19T14:21:07.213050+0000","last_became_peered":"2024-08-19T14:21:07.213050+0000","last_unstale":"2024-08-19T14:21:29.066378+0000","last_undegraded":"2024-08-19T14:21:29.066378+0000","last_fullsized":"2024-08-19T14:21:06.805412+0000","mapping_epoch":354,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":5951,"log_dups_size":0,"ondisk_log_size":5951,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":390004736,"num_objects":5951,"num_object_clones":0,"num_object_copies":17853,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3603,"num_objects_unfound":0,"num_objects_dirty":5951,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5951,"num_write_kb":380864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":537,"num_bytes_recovered":35192832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,7],"acting":[1,7],"avail_no_missing":["1","5","7"],"object_location_counts":[{"shards":"1,5,7","objects":3603},{"shards":"1,7","objects":2348}],"blocked_by":[],"up_primary":9,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"70'139","reported_seq":837,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-08-19T14:21:28.749844+0000","last_change":"2024-08-19T14:20:03.714917+0000","last_active":"2024-08-19T14:21:28.749844+0000","last_peered":"2024-08-19T14:21:28.749844+0000","last_clean":"2024-08-19T14:21:28.749844+0000","last_became_active":"2024-08-19T14:18:13.804721+0000","last_became_peered":"2024-08-19T14:18:13.804721+0000","last_unstale":"2024-08-19T14:21:28.749844+0000","last_undegraded":"2024-08-19T14:21:28.749844+0000","last_fullsized":"2024-08-19T14:21:28.749844+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:03.714818+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:03.714818+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:29.563399+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"361'5970","reported_seq":6030,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-08-19T14:21:29.062350+0000","last_change":"2024-08-19T14:20:29.268110+0000","last_active":"2024-08-19T14:21:29.062350+0000","last_peered":"2024-08-19T14:21:29.062350+0000","last_clean":"2024-08-19T14:21:29.062350+0000","last_became_active":"2024-08-19T14:20:29.267811+0000","last_became_peered":"2024-08-19T14:20:29.267811+0000","last_unstale":"2024-08-19T14:21:29.062350+0000","last_undegraded":"2024-08-19T14:21:29.062350+0000","last_fullsized":"2024-08-19T14:21:29.062350+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":5970,"log_dups_size":0,"ondisk_log_size":5970,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:46.516076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":391249920,"num_objects":5970,"num_object_clones":0,"num_object_copies":17910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5970,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5970,"num_write_kb":382080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"70'139","reported_seq":787,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-08-19T14:21:30.754181+0000","last_change":"2024-08-19T14:20:50.742676+0000","last_active":"2024-08-19T14:21:30.754181+0000","last_peered":"2024-08-19T14:21:30.754181+0000","last_clean":"2024-08-19T14:21:30.754181+0000","last_became_active":"2024-08-19T14:20:50.742354+0000","last_became_peered":"2024-08-19T14:20:50.742354+0000","last_unstale":"2024-08-19T14:21:30.754181+0000","last_undegraded":"2024-08-19T14:21:30.754181+0000","last_fullsized":"2024-08-19T14:21:30.754181+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:19:59.302931+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:19:59.302931+0000","last_clean_scrub_stamp":"2024-08-19T14:19:59.302931+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":114,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"361'5928","reported_seq":5990,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-08-19T14:21:29.040303+0000","last_change":"2024-08-19T14:20:29.267640+0000","last_active":"2024-08-19T14:21:29.040303+0000","last_peered":"2024-08-19T14:21:29.040303+0000","last_clean":"2024-08-19T14:21:29.040303+0000","last_became_active":"2024-08-19T14:20:29.267331+0000","last_became_peered":"2024-08-19T14:20:29.267331+0000","last_unstale":"2024-08-19T14:21:29.040303+0000","last_undegraded":"2024-08-19T14:21:29.040303+0000","last_fullsized":"2024-08-19T14:21:29.040303+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":5928,"log_dups_size":0,"ondisk_log_size":5928,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:46.504191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":388497408,"num_objects":5928,"num_object_clones":0,"num_object_copies":17784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5928,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5928,"num_write_kb":379392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"70'139","reported_seq":836,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-08-19T14:21:30.752764+0000","last_change":"2024-08-19T14:20:09.507422+0000","last_active":"2024-08-19T14:21:30.752764+0000","last_peered":"2024-08-19T14:21:30.752764+0000","last_clean":"2024-08-19T14:21:30.752764+0000","last_became_active":"2024-08-19T14:18:13.808505+0000","last_became_peered":"2024-08-19T14:18:13.808505+0000","last_unstale":"2024-08-19T14:21:30.752764+0000","last_undegraded":"2024-08-19T14:21:30.752764+0000","last_fullsized":"2024-08-19T14:21:30.752764+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:09.507256+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:09.507256+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,11],"acting":[9,0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"362'6301","reported_seq":6365,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-08-19T14:21:33.384491+0000","last_change":"2024-08-19T14:20:29.268766+0000","last_active":"2024-08-19T14:21:33.384491+0000","last_peered":"2024-08-19T14:21:33.384491+0000","last_clean":"2024-08-19T14:21:33.384491+0000","last_became_active":"2024-08-19T14:20:29.268443+0000","last_became_peered":"2024-08-19T14:20:29.268443+0000","last_unstale":"2024-08-19T14:21:33.384491+0000","last_undegraded":"2024-08-19T14:21:33.384491+0000","last_fullsized":"2024-08-19T14:21:33.384491+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":6301,"log_dups_size":0,"ondisk_log_size":6301,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:39.688611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":412942336,"num_objects":6301,"num_object_clones":0,"num_object_copies":18903,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6301,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6301,"num_write_kb":403264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,11],"acting":[9,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.7","version":"70'139","reported_seq":477,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-08-19T14:21:28.757351+0000","last_change":"2024-08-19T14:20:34.937317+0000","last_active":"2024-08-19T14:21:28.757351+0000","last_peered":"2024-08-19T14:21:28.757351+0000","last_clean":"2024-08-19T14:21:28.757351+0000","last_became_active":"2024-08-19T14:20:28.204384+0000","last_became_peered":"2024-08-19T14:20:28.204384+0000","last_unstale":"2024-08-19T14:21:28.757351+0000","last_undegraded":"2024-08-19T14:21:28.757351+0000","last_fullsized":"2024-08-19T14:21:28.757351+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:34.937192+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:20:34.937192+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:40.330609+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"362'6018","reported_seq":6082,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-08-19T14:21:30.785099+0000","last_change":"2024-08-19T14:20:29.267833+0000","last_active":"2024-08-19T14:21:30.785099+0000","last_peered":"2024-08-19T14:21:30.785099+0000","last_clean":"2024-08-19T14:21:30.785099+0000","last_became_active":"2024-08-19T14:20:29.267379+0000","last_became_peered":"2024-08-19T14:20:29.267379+0000","last_unstale":"2024-08-19T14:21:30.785099+0000","last_undegraded":"2024-08-19T14:21:30.785099+0000","last_fullsized":"2024-08-19T14:21:30.785099+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":6018,"log_dups_size":0,"ondisk_log_size":6018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":394395648,"num_objects":6018,"num_object_clones":0,"num_object_copies":18054,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6018,"num_write_kb":385152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,1],"acting":[11,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.6","version":"70'139","reported_seq":839,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-08-19T14:21:30.753559+0000","last_change":"2024-08-19T14:19:55.512078+0000","last_active":"2024-08-19T14:21:30.753559+0000","last_peered":"2024-08-19T14:21:30.753559+0000","last_clean":"2024-08-19T14:21:30.753559+0000","last_became_active":"2024-08-19T14:18:26.874902+0000","last_became_peered":"2024-08-19T14:18:26.874902+0000","last_unstale":"2024-08-19T14:21:30.753559+0000","last_undegraded":"2024-08-19T14:21:30.753559+0000","last_fullsized":"2024-08-19T14:21:30.753559+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:19:55.511974+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:27.848870+0000","last_clean_scrub_stamp":"2024-08-19T14:19:55.511974+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,4],"acting":[9,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"361'5930","reported_seq":5991,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-08-19T14:21:29.437485+0000","last_change":"2024-08-19T14:20:29.267682+0000","last_active":"2024-08-19T14:21:29.437485+0000","last_peered":"2024-08-19T14:21:29.437485+0000","last_clean":"2024-08-19T14:21:29.437485+0000","last_became_active":"2024-08-19T14:20:29.267360+0000","last_became_peered":"2024-08-19T14:20:29.267360+0000","last_unstale":"2024-08-19T14:21:29.437485+0000","last_undegraded":"2024-08-19T14:21:29.437485+0000","last_fullsized":"2024-08-19T14:21:29.437485+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":5930,"log_dups_size":0,"ondisk_log_size":5930,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:51.551392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":388628480,"num_objects":5930,"num_object_clones":0,"num_object_copies":17790,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5930,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5930,"num_write_kb":379520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"70'139","reported_seq":653,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-08-19T14:21:30.750425+0000","last_change":"2024-08-19T14:20:12.731993+0000","last_active":"2024-08-19T14:21:30.750425+0000","last_peered":"2024-08-19T14:21:30.750425+0000","last_clean":"2024-08-19T14:21:30.750425+0000","last_became_active":"2024-08-19T14:18:26.881435+0000","last_became_peered":"2024-08-19T14:18:26.881435+0000","last_unstale":"2024-08-19T14:21:30.750425+0000","last_undegraded":"2024-08-19T14:21:30.750425+0000","last_fullsized":"2024-08-19T14:21:30.750425+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:12.731850+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:12.731850+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:37.351039+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,4],"acting":[11,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.1","version":"362'6182","reported_seq":6246,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-08-19T14:21:30.771631+0000","last_change":"2024-08-19T14:20:29.268808+0000","last_active":"2024-08-19T14:21:30.771631+0000","last_peered":"2024-08-19T14:21:30.771631+0000","last_clean":"2024-08-19T14:21:30.771631+0000","last_became_active":"2024-08-19T14:20:29.268465+0000","last_became_peered":"2024-08-19T14:20:29.268465+0000","last_unstale":"2024-08-19T14:21:30.771631+0000","last_undegraded":"2024-08-19T14:21:30.771631+0000","last_fullsized":"2024-08-19T14:21:30.771631+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":6182,"log_dups_size":0,"ondisk_log_size":6182,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:57.700841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":405143552,"num_objects":6182,"num_object_clones":0,"num_object_copies":18546,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6182,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6182,"num_write_kb":395648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.4","version":"70'139","reported_seq":878,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-08-19T14:21:30.750131+0000","last_change":"2024-08-19T14:20:04.729769+0000","last_active":"2024-08-19T14:21:30.750131+0000","last_peered":"2024-08-19T14:21:30.750131+0000","last_clean":"2024-08-19T14:21:30.750131+0000","last_became_active":"2024-08-19T14:18:13.806013+0000","last_became_peered":"2024-08-19T14:18:13.806013+0000","last_unstale":"2024-08-19T14:21:30.750131+0000","last_undegraded":"2024-08-19T14:21:30.750131+0000","last_fullsized":"2024-08-19T14:21:30.750131+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:04.729563+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:04.729563+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:32.113541+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.f","version":"361'5921","reported_seq":5982,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-08-19T14:21:29.062452+0000","last_change":"2024-08-19T14:20:29.212298+0000","last_active":"2024-08-19T14:21:29.062452+0000","last_peered":"2024-08-19T14:21:29.062452+0000","last_clean":"2024-08-19T14:21:29.062452+0000","last_became_active":"2024-08-19T14:20:29.212003+0000","last_became_peered":"2024-08-19T14:20:29.212003+0000","last_unstale":"2024-08-19T14:21:29.062452+0000","last_undegraded":"2024-08-19T14:21:29.062452+0000","last_fullsized":"2024-08-19T14:21:29.062452+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":5921,"log_dups_size":0,"ondisk_log_size":5921,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:52.621283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":388038656,"num_objects":5921,"num_object_clones":0,"num_object_copies":17763,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5921,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5921,"num_write_kb":378944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,11],"acting":[1,3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"70'139","reported_seq":847,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-08-19T14:21:30.752973+0000","last_change":"2024-08-19T14:20:12.570453+0000","last_active":"2024-08-19T14:21:30.752973+0000","last_peered":"2024-08-19T14:21:30.752973+0000","last_clean":"2024-08-19T14:21:30.752973+0000","last_became_active":"2024-08-19T14:18:26.875006+0000","last_became_peered":"2024-08-19T14:18:26.875006+0000","last_unstale":"2024-08-19T14:21:30.752973+0000","last_undegraded":"2024-08-19T14:21:30.752973+0000","last_fullsized":"2024-08-19T14:21:30.752973+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:12.570352+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:28.864870+0000","last_clean_scrub_stamp":"2024-08-19T14:20:12.570352+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:32.193930+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,0],"acting":[9,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.c","version":"362'6173","reported_seq":6237,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-08-19T14:21:33.129556+0000","last_change":"2024-08-19T14:20:29.266724+0000","last_active":"2024-08-19T14:21:33.129556+0000","last_peered":"2024-08-19T14:21:33.129556+0000","last_clean":"2024-08-19T14:21:33.129556+0000","last_became_active":"2024-08-19T14:20:29.266373+0000","last_became_peered":"2024-08-19T14:20:29.266373+0000","last_unstale":"2024-08-19T14:21:33.129556+0000","last_undegraded":"2024-08-19T14:21:33.129556+0000","last_fullsized":"2024-08-19T14:21:33.129556+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":6173,"log_dups_size":0,"ondisk_log_size":6173,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:48.006455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":404553728,"num_objects":6173,"num_object_clones":0,"num_object_copies":18519,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6173,"num_write_kb":395072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,8],"acting":[4,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"70'139","reported_seq":962,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-08-19T14:21:30.753907+0000","last_change":"2024-08-19T14:19:56.106047+0000","last_active":"2024-08-19T14:21:30.753907+0000","last_peered":"2024-08-19T14:21:30.753907+0000","last_clean":"2024-08-19T14:21:30.753907+0000","last_became_active":"2024-08-19T14:18:26.887657+0000","last_became_peered":"2024-08-19T14:18:26.887657+0000","last_unstale":"2024-08-19T14:21:30.753907+0000","last_undegraded":"2024-08-19T14:21:30.753907+0000","last_fullsized":"2024-08-19T14:21:30.753907+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:19:56.105880+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:19:56.105880+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"362'6157","reported_seq":6220,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-08-19T14:21:33.156644+0000","last_change":"2024-08-19T14:20:29.208992+0000","last_active":"2024-08-19T14:21:33.156644+0000","last_peered":"2024-08-19T14:21:33.156644+0000","last_clean":"2024-08-19T14:21:33.156644+0000","last_became_active":"2024-08-19T14:20:29.208599+0000","last_became_peered":"2024-08-19T14:20:29.208599+0000","last_unstale":"2024-08-19T14:21:33.156644+0000","last_undegraded":"2024-08-19T14:21:33.156644+0000","last_fullsized":"2024-08-19T14:21:33.156644+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":6157,"log_dups_size":0,"ondisk_log_size":6157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:55.930633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403505152,"num_objects":6157,"num_object_clones":0,"num_object_copies":18471,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6157,"num_write_kb":394048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,1],"acting":[4,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"70'139","reported_seq":851,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-08-19T14:21:30.753216+0000","last_change":"2024-08-19T14:20:11.390093+0000","last_active":"2024-08-19T14:21:30.753216+0000","last_peered":"2024-08-19T14:21:30.753216+0000","last_clean":"2024-08-19T14:21:30.753216+0000","last_became_active":"2024-08-19T14:18:26.887277+0000","last_became_peered":"2024-08-19T14:18:26.887277+0000","last_unstale":"2024-08-19T14:21:30.753216+0000","last_undegraded":"2024-08-19T14:21:30.753216+0000","last_fullsized":"2024-08-19T14:21:30.753216+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:11.389938+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:11.389938+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:34.809098+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,8],"acting":[5,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"362'6162","reported_seq":6225,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-08-19T14:21:33.137600+0000","last_change":"2024-08-19T14:20:29.209096+0000","last_active":"2024-08-19T14:21:33.137600+0000","last_peered":"2024-08-19T14:21:33.137600+0000","last_clean":"2024-08-19T14:21:33.137600+0000","last_became_active":"2024-08-19T14:20:29.208748+0000","last_became_peered":"2024-08-19T14:20:29.208748+0000","last_unstale":"2024-08-19T14:21:33.137600+0000","last_undegraded":"2024-08-19T14:21:33.137600+0000","last_fullsized":"2024-08-19T14:21:33.137600+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":6162,"log_dups_size":0,"ondisk_log_size":6162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:56.597251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403832832,"num_objects":6162,"num_object_clones":0,"num_object_copies":18486,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6162,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6162,"num_write_kb":394368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"70'139","reported_seq":853,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-08-19T14:21:30.750066+0000","last_change":"2024-08-19T14:20:11.733961+0000","last_active":"2024-08-19T14:21:30.750066+0000","last_peered":"2024-08-19T14:21:30.750066+0000","last_clean":"2024-08-19T14:21:30.750066+0000","last_became_active":"2024-08-19T14:18:26.875179+0000","last_became_peered":"2024-08-19T14:18:26.875179+0000","last_unstale":"2024-08-19T14:21:30.750066+0000","last_undegraded":"2024-08-19T14:21:30.750066+0000","last_fullsized":"2024-08-19T14:21:30.750066+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:11.733865+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:11.733865+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:41.250361+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.b","version":"362'6072","reported_seq":6136,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-08-19T14:21:31.186154+0000","last_change":"2024-08-19T14:20:29.207845+0000","last_active":"2024-08-19T14:21:31.186154+0000","last_peered":"2024-08-19T14:21:31.186154+0000","last_clean":"2024-08-19T14:21:31.186154+0000","last_became_active":"2024-08-19T14:20:29.207372+0000","last_became_peered":"2024-08-19T14:20:29.207372+0000","last_unstale":"2024-08-19T14:21:31.186154+0000","last_undegraded":"2024-08-19T14:21:31.186154+0000","last_fullsized":"2024-08-19T14:21:31.186154+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":6072,"log_dups_size":0,"ondisk_log_size":6072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:33.577033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":397934592,"num_objects":6072,"num_object_clones":0,"num_object_copies":18216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6072,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6072,"num_write_kb":388608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"70'139","reported_seq":507,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-08-19T14:21:30.752279+0000","last_change":"2024-08-19T14:20:15.322051+0000","last_active":"2024-08-19T14:21:30.752279+0000","last_peered":"2024-08-19T14:21:30.752279+0000","last_clean":"2024-08-19T14:21:30.752279+0000","last_became_active":"2024-08-19T14:18:26.884105+0000","last_became_peered":"2024-08-19T14:18:26.884105+0000","last_unstale":"2024-08-19T14:21:30.752279+0000","last_undegraded":"2024-08-19T14:21:30.752279+0000","last_fullsized":"2024-08-19T14:21:30.752279+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:15.321945+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:20:15.321945+0000","last_clean_scrub_stamp":"2024-08-19T14:20:15.321945+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:43.904541+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"361'5957","reported_seq":6019,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-08-19T14:21:29.030339+0000","last_change":"2024-08-19T14:20:29.269723+0000","last_active":"2024-08-19T14:21:29.030339+0000","last_peered":"2024-08-19T14:21:29.030339+0000","last_clean":"2024-08-19T14:21:29.030339+0000","last_became_active":"2024-08-19T14:20:29.269432+0000","last_became_peered":"2024-08-19T14:20:29.269432+0000","last_unstale":"2024-08-19T14:21:29.030339+0000","last_undegraded":"2024-08-19T14:21:29.030339+0000","last_fullsized":"2024-08-19T14:21:29.030339+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":5957,"log_dups_size":0,"ondisk_log_size":5957,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:44.639451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":390397952,"num_objects":5957,"num_object_clones":0,"num_object_copies":17871,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5957,"num_write_kb":381248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.f","version":"70'139","reported_seq":759,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-08-19T14:21:30.753704+0000","last_change":"2024-08-19T14:21:14.765419+0000","last_active":"2024-08-19T14:21:30.753704+0000","last_peered":"2024-08-19T14:21:30.753704+0000","last_clean":"2024-08-19T14:21:30.753704+0000","last_became_active":"2024-08-19T14:18:18.711708+0000","last_became_peered":"2024-08-19T14:18:18.711708+0000","last_unstale":"2024-08-19T14:21:30.753704+0000","last_undegraded":"2024-08-19T14:21:30.753704+0000","last_fullsized":"2024-08-19T14:21:30.753704+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:14.765340+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:21:14.765340+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:22:24.415219+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"362'6089","reported_seq":6153,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-08-19T14:21:30.790782+0000","last_change":"2024-08-19T14:20:29.212320+0000","last_active":"2024-08-19T14:21:30.790782+0000","last_peered":"2024-08-19T14:21:30.790782+0000","last_clean":"2024-08-19T14:21:30.790782+0000","last_became_active":"2024-08-19T14:20:29.211975+0000","last_became_peered":"2024-08-19T14:20:29.211975+0000","last_unstale":"2024-08-19T14:21:30.790782+0000","last_undegraded":"2024-08-19T14:21:30.790782+0000","last_fullsized":"2024-08-19T14:21:30.790782+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":6089,"log_dups_size":0,"ondisk_log_size":6089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":399048704,"num_objects":6089,"num_object_clones":0,"num_object_copies":18267,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6089,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6089,"num_write_kb":389696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,3],"acting":[11,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.c","version":"70'139","reported_seq":610,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-08-19T14:21:28.757520+0000","last_change":"2024-08-19T14:20:07.839218+0000","last_active":"2024-08-19T14:21:28.757520+0000","last_peered":"2024-08-19T14:21:28.757520+0000","last_clean":"2024-08-19T14:21:28.757520+0000","last_became_active":"2024-08-19T14:18:26.872942+0000","last_became_peered":"2024-08-19T14:18:26.872942+0000","last_unstale":"2024-08-19T14:21:28.757520+0000","last_undegraded":"2024-08-19T14:21:28.757520+0000","last_fullsized":"2024-08-19T14:21:28.757520+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:07.839076+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:20:07.839076+0000","last_clean_scrub_stamp":"2024-08-19T14:20:07.839076+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:33.185668+0000","scrub_duration":62,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,9],"acting":[3,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"362'6019","reported_seq":6083,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-08-19T14:21:30.749992+0000","last_change":"2024-08-19T14:20:29.268318+0000","last_active":"2024-08-19T14:21:30.749992+0000","last_peered":"2024-08-19T14:21:30.749992+0000","last_clean":"2024-08-19T14:21:30.749992+0000","last_became_active":"2024-08-19T14:20:29.267442+0000","last_became_peered":"2024-08-19T14:20:29.267442+0000","last_unstale":"2024-08-19T14:21:30.749992+0000","last_undegraded":"2024-08-19T14:21:30.749992+0000","last_fullsized":"2024-08-19T14:21:30.749992+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":6019,"log_dups_size":0,"ondisk_log_size":6019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:47.715801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":394461184,"num_objects":6019,"num_object_clones":0,"num_object_copies":18057,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6019,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6019,"num_write_kb":385216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,9],"acting":[11,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.d","version":"70'139","reported_seq":535,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-08-19T14:21:30.754089+0000","last_change":"2024-08-19T14:21:07.155294+0000","last_active":"2024-08-19T14:21:30.754089+0000","last_peered":"2024-08-19T14:21:30.754089+0000","last_clean":"2024-08-19T14:21:30.754089+0000","last_became_active":"2024-08-19T14:18:26.882002+0000","last_became_peered":"2024-08-19T14:18:26.882002+0000","last_unstale":"2024-08-19T14:21:30.754089+0000","last_undegraded":"2024-08-19T14:21:30.754089+0000","last_fullsized":"2024-08-19T14:21:30.754089+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:07.155157+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:21:07.155157+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:22:11.274708+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"70'139","reported_seq":794,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-08-19T14:21:30.757889+0000","last_change":"2024-08-19T14:21:13.777891+0000","last_active":"2024-08-19T14:21:30.757889+0000","last_peered":"2024-08-19T14:21:30.757889+0000","last_clean":"2024-08-19T14:21:30.757889+0000","last_became_active":"2024-08-19T14:18:18.703594+0000","last_became_peered":"2024-08-19T14:18:18.703594+0000","last_unstale":"2024-08-19T14:21:30.757889+0000","last_undegraded":"2024-08-19T14:21:30.757889+0000","last_fullsized":"2024-08-19T14:21:30.757889+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:13.777777+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:21:13.777777+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:22:38.473560+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"70'139","reported_seq":844,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-08-19T14:21:28.749499+0000","last_change":"2024-08-19T14:20:07.692054+0000","last_active":"2024-08-19T14:21:28.749499+0000","last_peered":"2024-08-19T14:21:28.749499+0000","last_clean":"2024-08-19T14:21:28.749499+0000","last_became_active":"2024-08-19T14:18:18.698334+0000","last_became_peered":"2024-08-19T14:18:18.698334+0000","last_unstale":"2024-08-19T14:21:28.749499+0000","last_undegraded":"2024-08-19T14:21:28.749499+0000","last_fullsized":"2024-08-19T14:21:28.749499+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:07.691932+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:07.691932+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:36.578885+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.10","version":"70'139","reported_seq":949,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-08-19T14:21:30.753147+0000","last_change":"2024-08-19T14:21:07.768770+0000","last_active":"2024-08-19T14:21:30.753147+0000","last_peered":"2024-08-19T14:21:30.753147+0000","last_clean":"2024-08-19T14:21:30.753147+0000","last_became_active":"2024-08-19T14:18:18.704562+0000","last_became_peered":"2024-08-19T14:18:18.704562+0000","last_unstale":"2024-08-19T14:21:30.753147+0000","last_undegraded":"2024-08-19T14:21:30.753147+0000","last_fullsized":"2024-08-19T14:21:30.753147+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:07.768697+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:21:07.768697+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:22:10.360289+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,8],"acting":[5,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"70'139","reported_seq":498,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-08-19T14:21:30.757631+0000","last_change":"2024-08-19T14:21:16.071812+0000","last_active":"2024-08-19T14:21:30.757631+0000","last_peered":"2024-08-19T14:21:30.757631+0000","last_clean":"2024-08-19T14:21:30.757631+0000","last_became_active":"2024-08-19T14:18:18.697894+0000","last_became_peered":"2024-08-19T14:18:18.697894+0000","last_unstale":"2024-08-19T14:21:30.757631+0000","last_undegraded":"2024-08-19T14:21:30.757631+0000","last_fullsized":"2024-08-19T14:21:30.757631+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:16.071703+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:21:16.071703+0000","last_clean_scrub_stamp":"2024-08-19T14:21:16.071703+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:22:18.184091+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"70'139","reported_seq":473,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-08-19T14:21:28.752103+0000","last_change":"2024-08-19T14:20:28.205212+0000","last_active":"2024-08-19T14:21:28.752103+0000","last_peered":"2024-08-19T14:21:28.752103+0000","last_clean":"2024-08-19T14:21:28.752103+0000","last_became_active":"2024-08-19T14:20:28.204724+0000","last_became_peered":"2024-08-19T14:20:28.204724+0000","last_unstale":"2024-08-19T14:21:28.752103+0000","last_undegraded":"2024-08-19T14:21:28.752103+0000","last_fullsized":"2024-08-19T14:21:28.752103+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:08.680021+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:08.680021+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"70'139","reported_seq":593,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-08-19T14:21:28.771078+0000","last_change":"2024-08-19T14:20:31.931658+0000","last_active":"2024-08-19T14:21:28.771078+0000","last_peered":"2024-08-19T14:21:28.771078+0000","last_clean":"2024-08-19T14:21:28.771078+0000","last_became_active":"2024-08-19T14:18:31.901556+0000","last_became_peered":"2024-08-19T14:18:31.901556+0000","last_unstale":"2024-08-19T14:21:28.771078+0000","last_undegraded":"2024-08-19T14:21:28.771078+0000","last_fullsized":"2024-08-19T14:21:28.771078+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:31.931562+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:20:31.931562+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:47.687921+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,0],"acting":[8,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.14","version":"70'139","reported_seq":604,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-08-19T14:21:30.754423+0000","last_change":"2024-08-19T14:19:55.130518+0000","last_active":"2024-08-19T14:21:30.754423+0000","last_peered":"2024-08-19T14:21:30.754423+0000","last_clean":"2024-08-19T14:21:30.754423+0000","last_became_active":"2024-08-19T14:18:18.703926+0000","last_became_peered":"2024-08-19T14:18:18.703926+0000","last_unstale":"2024-08-19T14:21:30.754423+0000","last_undegraded":"2024-08-19T14:21:30.754423+0000","last_fullsized":"2024-08-19T14:21:30.754423+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:19:55.130420+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:19:55.130420+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,1],"acting":[7,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.15","version":"70'139","reported_seq":491,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-08-19T14:21:30.753993+0000","last_change":"2024-08-19T14:20:09.181046+0000","last_active":"2024-08-19T14:21:30.753993+0000","last_peered":"2024-08-19T14:21:30.753993+0000","last_clean":"2024-08-19T14:21:30.753993+0000","last_became_active":"2024-08-19T14:18:31.912496+0000","last_became_peered":"2024-08-19T14:18:31.912496+0000","last_unstale":"2024-08-19T14:21:30.753993+0000","last_undegraded":"2024-08-19T14:21:30.753993+0000","last_fullsized":"2024-08-19T14:21:30.753993+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:09.180923+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:20:09.180923+0000","last_clean_scrub_stamp":"2024-08-19T14:20:09.180923+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"70'139","reported_seq":469,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-08-19T14:21:30.752097+0000","last_change":"2024-08-19T14:20:17.333347+0000","last_active":"2024-08-19T14:21:30.752097+0000","last_peered":"2024-08-19T14:21:30.752097+0000","last_clean":"2024-08-19T14:21:30.752097+0000","last_became_active":"2024-08-19T14:18:31.901229+0000","last_became_peered":"2024-08-19T14:18:31.901229+0000","last_unstale":"2024-08-19T14:21:30.752097+0000","last_undegraded":"2024-08-19T14:21:30.752097+0000","last_fullsized":"2024-08-19T14:21:30.752097+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:17.333260+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:17.333260+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:34.399780+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1b","version":"70'139","reported_seq":819,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-08-19T14:21:28.770907+0000","last_change":"2024-08-19T14:20:16.776086+0000","last_active":"2024-08-19T14:21:28.770907+0000","last_peered":"2024-08-19T14:21:28.770907+0000","last_clean":"2024-08-19T14:21:28.770907+0000","last_became_active":"2024-08-19T14:18:31.901152+0000","last_became_peered":"2024-08-19T14:18:31.901152+0000","last_unstale":"2024-08-19T14:21:28.770907+0000","last_undegraded":"2024-08-19T14:21:28.770907+0000","last_fullsized":"2024-08-19T14:21:28.770907+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:16.775987+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:16.775987+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:46.485727+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.18","version":"70'139","reported_seq":865,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-08-19T14:21:30.753111+0000","last_change":"2024-08-19T14:20:36.510554+0000","last_active":"2024-08-19T14:21:30.753111+0000","last_peered":"2024-08-19T14:21:30.753111+0000","last_clean":"2024-08-19T14:21:30.753111+0000","last_became_active":"2024-08-19T14:18:31.902735+0000","last_became_peered":"2024-08-19T14:18:31.902735+0000","last_unstale":"2024-08-19T14:21:30.753111+0000","last_undegraded":"2024-08-19T14:21:30.753111+0000","last_fullsized":"2024-08-19T14:21:30.753111+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:36.510442+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:36.510442+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:53.706735+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.19","version":"70'139","reported_seq":813,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-08-19T14:21:30.757588+0000","last_change":"2024-08-19T14:20:32.919627+0000","last_active":"2024-08-19T14:21:30.757588+0000","last_peered":"2024-08-19T14:21:30.757588+0000","last_clean":"2024-08-19T14:21:30.757588+0000","last_became_active":"2024-08-19T14:18:31.902547+0000","last_became_peered":"2024-08-19T14:18:31.902547+0000","last_unstale":"2024-08-19T14:21:30.757588+0000","last_undegraded":"2024-08-19T14:21:30.757588+0000","last_fullsized":"2024-08-19T14:21:30.757588+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:32.919545+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:20:32.919545+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:35.916636+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1e","version":"70'139","reported_seq":471,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-08-19T14:21:30.750239+0000","last_change":"2024-08-19T14:20:18.771836+0000","last_active":"2024-08-19T14:21:30.750239+0000","last_peered":"2024-08-19T14:21:30.750239+0000","last_clean":"2024-08-19T14:21:30.750239+0000","last_became_active":"2024-08-19T14:18:31.902546+0000","last_became_peered":"2024-08-19T14:18:31.902546+0000","last_unstale":"2024-08-19T14:21:30.750239+0000","last_undegraded":"2024-08-19T14:21:30.750239+0000","last_fullsized":"2024-08-19T14:21:30.750239+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:18.771704+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:18.771704+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:47.154907+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1c","version":"70'139","reported_seq":577,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-08-19T14:21:28.772625+0000","last_change":"2024-08-19T14:20:08.824736+0000","last_active":"2024-08-19T14:21:28.772625+0000","last_peered":"2024-08-19T14:21:28.772625+0000","last_clean":"2024-08-19T14:21:28.772625+0000","last_became_active":"2024-08-19T14:18:31.902494+0000","last_became_peered":"2024-08-19T14:18:31.902494+0000","last_unstale":"2024-08-19T14:21:28.772625+0000","last_undegraded":"2024-08-19T14:21:28.772625+0000","last_fullsized":"2024-08-19T14:21:28.772625+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:08.824505+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:08.824505+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:32.216640+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,11],"acting":[8,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1d","version":"70'139","reported_seq":499,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-08-19T14:21:28.752670+0000","last_change":"2024-08-19T14:20:17.725615+0000","last_active":"2024-08-19T14:21:28.752670+0000","last_peered":"2024-08-19T14:21:28.752670+0000","last_clean":"2024-08-19T14:21:28.752670+0000","last_became_active":"2024-08-19T14:18:31.901183+0000","last_became_peered":"2024-08-19T14:18:31.901183+0000","last_unstale":"2024-08-19T14:21:28.752670+0000","last_undegraded":"2024-08-19T14:21:28.752670+0000","last_fullsized":"2024-08-19T14:21:28.752670+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:17.725477+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:17.725477+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,5],"acting":[1,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":6353387520,"num_objects":96945,"num_object_clones":0,"num_object_copies":290835,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3603,"num_objects_unfound":0,"num_objects_dirty":96945,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":96945,"num_write_kb":6204480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":537,"num_bytes_recovered":35192832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19211812864,"data_stored":19211812864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96945,"ondisk_log_size":96945,"up":48,"acting":47,"num_store_stats":9},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1802240,"data_stored":1788032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187578,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3102728,"kb_used_data":2706432,"kb_used_omap":1,"kb_used_meta":396286,"kb_avail":90642424,"statfs":{"total":95995035648,"available":92817842176,"internally_reserved":0,"allocated":2771386368,"data_stored":2769716742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":405797316},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":343597383808,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":74,"seq":317827580036,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1801556,"kb_used_data":1359436,"kb_used_omap":1,"kb_used_meta":442110,"kb_avail":91943596,"statfs":{"total":95995035648,"available":94150242304,"internally_reserved":0,"allocated":1392062464,"data_stored":1390388662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":452721089},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874377,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":765388,"kb_used_data":382976,"kb_used_omap":1,"kb_used_meta":382398,"kb_avail":92979764,"statfs":{"total":95995035648,"available":95211278336,"internally_reserved":0,"allocated":392167424,"data_stored":390497798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":391576005},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":57,"seq":244813136012,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3637984,"kb_used_data":3205588,"kb_used_omap":1,"kb_used_meta":432382,"kb_avail":90107168,"statfs":{"total":95995035648,"available":92269740032,"internally_reserved":0,"allocated":3282522112,"data_stored":3280844758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":442759626},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":206158430351,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691988,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1787488,"kb_used_data":1445332,"kb_used_omap":1,"kb_used_meta":342142,"kb_avail":91957664,"statfs":{"total":95995035648,"available":94164647936,"internally_reserved":0,"allocated":1480019968,"data_stored":1478342614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":350353861},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986329,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2439572,"kb_used_data":1971084,"kb_used_omap":1,"kb_used_meta":468478,"kb_avail":91305580,"statfs":{"total":95995035648,"available":93496913920,"internally_reserved":0,"allocated":2018390016,"data_stored":2016716214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":479721925},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149854,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2684300,"kb_used_data":2274112,"kb_used_omap":1,"kb_used_meta":410174,"kb_avail":91060852,"statfs":{"total":95995035648,"available":93246312448,"internally_reserved":0,"allocated":2328690688,"data_stored":2327021062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":420018631},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411492,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575015,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4202244,"kb_used_data":3751804,"kb_used_omap":1,"kb_used_meta":450430,"kb_avail":89542908,"statfs":{"total":95995035648,"available":91691937792,"internally_reserved":0,"allocated":3841847296,"data_stored":3840178344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":461240773},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738540,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1910620,"kb_used_data":1564300,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":91834532,"statfs":{"total":95995035648,"available":94038560768,"internally_reserved":0,"allocated":1601843200,"data_stored":1600169398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":354613701},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1600323584,"data_stored":1600323584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3875209216,"data_stored":3875209216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2333933568,"data_stored":2333933568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2035089408,"data_stored":2035089408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1489698816,"data_stored":1489698816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3290300416,"data_stored":3290300416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":388235264,"data_stored":388235264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1409351680,"data_stored":1409351680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":2789670912,"data_stored":2789670912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T14:21:34.952 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:21:34.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:34.982 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:34.982 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:21:35.049 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 61 16 101969 101953 104.451 63.3125 0.00923934 0.00956972 2024-08-19T14:21:35.083 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:21:35.110 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:35.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:35.114 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:21:35.215 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:21:35.239 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:35.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:35.240 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:21:35.341 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:21:35.366 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:35.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:35.368 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:21:35.468 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:21:35.506 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:35.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:35.507 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:21:35.521 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:21:35.608 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:21:35.641 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:35.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:35.642 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:21:35.660 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-08-19T14:21:35.744 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:21:35.771 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is unset 2024-08-19T14:21:35.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:35.776 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:35.776 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:21:35.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:35 smithi031 ceph-mon[16336]: pgmap v812: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.9 GiB data, 21 GiB used, 783 GiB / 805 GiB avail; 60 MiB/s wr, 957 op/s; 3603/290841 objects misplaced (1.239%); 1.0 MiB/s, 16 objects/s recovering 2024-08-19T14:21:35.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:35 smithi031 ceph-mon[16336]: from='client.25447 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:21:35.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:35 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2455510632' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:21:35.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:35 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:21:35.877 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:21:35.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:35.903 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:35.903 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:21:36.006 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:21:36.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:35 smithi032 ceph-mon[20004]: pgmap v812: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.9 GiB data, 21 GiB used, 783 GiB / 805 GiB avail; 60 MiB/s wr, 957 op/s; 3603/290841 objects misplaced (1.239%); 1.0 MiB/s, 16 objects/s recovering 2024-08-19T14:21:36.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:35 smithi032 ceph-mon[20004]: from='client.25447 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:21:36.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:35 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2455510632' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:21:36.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:35 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:21:36.036 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:21:36.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:35 smithi045 ceph-mon[20143]: pgmap v812: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.9 GiB data, 21 GiB used, 783 GiB / 805 GiB avail; 60 MiB/s wr, 957 op/s; 3603/290841 objects misplaced (1.239%); 1.0 MiB/s, 16 objects/s recovering 2024-08-19T14:21:36.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:35 smithi045 ceph-mon[20143]: from='client.25447 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:21:36.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:35 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2455510632' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:21:36.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:35 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:21:36.049 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 62 16 103137 103121 103.944 73 0.0147054 0.00961611 2024-08-19T14:21:36.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:36.050 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:36.051 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:21:36.152 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:21:36.180 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:36.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:36.181 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:21:36.282 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:21:36.320 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:36.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:36.321 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:21:36.422 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:21:36.446 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:36.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:36.449 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:21:36.549 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:21:36.579 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:36.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:36.580 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:21:36.656 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:21:36.656 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-08-19T14:21:36.656 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-08-19T14:21:36.656 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-08-19T14:21:36.656 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.656 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:8241 2.7 2:edf47d3e:::benchmark_data_smithi045_46415_object8240:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-08-19T14:21:36.656 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:12:51.684165+0000", 2024-08-19T14:21:36.656 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 524.96941780500003, 2024-08-19T14:21:36.656 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.204747761, 2024-08-19T14:21:36.656 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:21:36.657 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:21:36.657 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:21:36.657 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:21:36.657 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:21:36.657 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:21:36.657 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 8241 2024-08-19T14:21:36.657 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.657 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:21:36.657 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.657 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:21:36.657 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.684165+0000", 2024-08-19T14:21:36.657 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:21:36.657 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.657 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.658 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:21:36.658 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.684165+0000", 2024-08-19T14:21:36.658 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:21:36.658 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.658 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.658 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:21:36.658 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.684167+0000", 2024-08-19T14:21:36.658 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.313e-06 2024-08-19T14:21:36.658 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.658 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.658 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:21:36.658 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.684210+0000", 2024-08-19T14:21:36.658 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3049000000000001e-05 2024-08-19T14:21:36.659 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.659 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.659 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:21:36.659 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.684223+0000", 2024-08-19T14:21:36.659 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3019e-05 2024-08-19T14:21:36.659 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.659 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.659 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:21:36.659 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.684230+0000", 2024-08-19T14:21:36.659 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1969999999999999e-06 2024-08-19T14:21:36.659 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.659 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.660 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:21:36.660 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.684270+0000", 2024-08-19T14:21:36.660 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0649999999999999e-05 2024-08-19T14:21:36.660 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.660 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.660 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:21:36.660 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.684340+0000", 2024-08-19T14:21:36.660 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9979000000000002e-05 2024-08-19T14:21:36.660 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.660 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.660 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,10", 2024-08-19T14:21:36.660 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.684464+0000", 2024-08-19T14:21:36.660 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012321599999999999 2024-08-19T14:21:36.660 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.661 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.661 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:21:36.661 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.684888+0000", 2024-08-19T14:21:36.661 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00042467399999999999 2024-08-19T14:21:36.661 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.661 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.661 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:36.661 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.685794+0000", 2024-08-19T14:21:36.661 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00090609299999999996 2024-08-19T14:21:36.661 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.661 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.661 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:36.661 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.888736+0000", 2024-08-19T14:21:36.662 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20294189100000001 2024-08-19T14:21:36.662 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.662 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.662 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:21:36.662 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.888831+0000", 2024-08-19T14:21:36.662 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.5172000000000003e-05 2024-08-19T14:21:36.662 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.662 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.662 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:21:36.662 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.888913+0000", 2024-08-19T14:21:36.662 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1508000000000006e-05 2024-08-19T14:21:36.662 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:21:36.664 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:21:36.664 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:21:36.664 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.664 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.664 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:8259 2.7 2:e2e1720b:::benchmark_data_smithi045_46415_object8258:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-08-19T14:21:36.665 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:12:51.689217+0000", 2024-08-19T14:21:36.665 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 524.96436593600004, 2024-08-19T14:21:36.665 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.19987796899999999, 2024-08-19T14:21:36.665 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:21:36.665 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:21:36.665 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:21:36.665 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:21:36.665 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:21:36.665 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:21:36.665 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 8259 2024-08-19T14:21:36.665 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.665 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:21:36.665 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.665 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:21:36.666 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.689217+0000", 2024-08-19T14:21:36.666 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:21:36.666 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.666 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.666 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:21:36.666 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.689217+0000", 2024-08-19T14:21:36.666 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:21:36.666 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.666 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.666 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:21:36.666 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.689218+0000", 2024-08-19T14:21:36.666 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0240000000000001e-06 2024-08-19T14:21:36.666 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.667 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.667 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:21:36.667 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.689272+0000", 2024-08-19T14:21:36.667 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3974999999999997e-05 2024-08-19T14:21:36.667 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.667 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.667 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:21:36.667 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.689283+0000", 2024-08-19T14:21:36.667 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0647e-05 2024-08-19T14:21:36.667 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.667 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.667 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:21:36.667 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.689288+0000", 2024-08-19T14:21:36.667 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2390000000000002e-06 2024-08-19T14:21:36.668 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.668 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.668 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:21:36.668 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.689341+0000", 2024-08-19T14:21:36.668 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.342e-05 2024-08-19T14:21:36.668 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.668 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.668 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:21:36.669 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.689464+0000", 2024-08-19T14:21:36.669 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000122991 2024-08-19T14:21:36.669 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.669 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.669 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,10", 2024-08-19T14:21:36.669 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.689610+0000", 2024-08-19T14:21:36.669 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014557200000000001 2024-08-19T14:21:36.669 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.669 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.669 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:21:36.669 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.690297+0000", 2024-08-19T14:21:36.669 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00068681199999999997 2024-08-19T14:21:36.669 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.670 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.670 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:36.670 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.690649+0000", 2024-08-19T14:21:36.670 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00035261899999999998 2024-08-19T14:21:36.670 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.670 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.671 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:36.671 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.888991+0000", 2024-08-19T14:21:36.671 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.19834179800000001 2024-08-19T14:21:36.671 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.671 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.671 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:21:36.671 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.889045+0000", 2024-08-19T14:21:36.671 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3643000000000001e-05 2024-08-19T14:21:36.671 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.671 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.671 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:21:36.671 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.889095+0000", 2024-08-19T14:21:36.671 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0229000000000003e-05 2024-08-19T14:21:36.672 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:21:36.672 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:21:36.672 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:21:36.672 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.672 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.672 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:8272 2.7 2:ec6eef80:::benchmark_data_smithi045_46415_object8271:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-08-19T14:21:36.672 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:12:51.692591+0000", 2024-08-19T14:21:36.672 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 524.96099211299997, 2024-08-19T14:21:36.672 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.19659718500000001, 2024-08-19T14:21:36.672 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:21:36.672 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:21:36.672 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:21:36.673 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:21:36.673 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:21:36.673 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:21:36.674 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 8272 2024-08-19T14:21:36.674 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.674 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:21:36.674 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.674 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:21:36.674 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.692591+0000", 2024-08-19T14:21:36.674 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:21:36.674 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.674 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.674 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:21:36.674 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.692591+0000", 2024-08-19T14:21:36.674 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:21:36.675 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.675 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.675 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:21:36.675 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.692592+0000", 2024-08-19T14:21:36.675 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.2800000000000005e-07 2024-08-19T14:21:36.675 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.675 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.675 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:21:36.675 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.692661+0000", 2024-08-19T14:21:36.675 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8744000000000005e-05 2024-08-19T14:21:36.675 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.675 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.675 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:21:36.676 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.692675+0000", 2024-08-19T14:21:36.676 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3916e-05 2024-08-19T14:21:36.676 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.676 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.676 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:21:36.676 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.692682+0000", 2024-08-19T14:21:36.676 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5320000000000001e-06 2024-08-19T14:21:36.676 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.676 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.676 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:21:36.676 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.692731+0000", 2024-08-19T14:21:36.676 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8909999999999998e-05 2024-08-19T14:21:36.676 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.676 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.677 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:21:36.677 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.692805+0000", 2024-08-19T14:21:36.677 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4128000000000006e-05 2024-08-19T14:21:36.677 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.677 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.677 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,10", 2024-08-19T14:21:36.677 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.692885+0000", 2024-08-19T14:21:36.677 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.9655999999999998e-05 2024-08-19T14:21:36.677 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.677 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.677 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:21:36.677 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.693416+0000", 2024-08-19T14:21:36.677 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00053101099999999996 2024-08-19T14:21:36.677 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.678 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.678 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:36.678 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.694542+0000", 2024-08-19T14:21:36.678 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0011263810000000001 2024-08-19T14:21:36.678 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.678 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.678 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:36.678 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.889136+0000", 2024-08-19T14:21:36.679 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.194593922 2024-08-19T14:21:36.679 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.679 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.679 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:21:36.679 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.889164+0000", 2024-08-19T14:21:36.679 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8113999999999999e-05 2024-08-19T14:21:36.679 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.679 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.679 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:21:36.679 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.889188+0000", 2024-08-19T14:21:36.679 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3943000000000001e-05 2024-08-19T14:21:36.679 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:21:36.679 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:21:36.680 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:21:36.680 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.680 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.680 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:8281 2.7 2:edaf96ca:::benchmark_data_smithi045_46415_object8280:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-08-19T14:21:36.680 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:12:51.695230+0000", 2024-08-19T14:21:36.680 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 524.95835346299998, 2024-08-19T14:21:36.680 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.194082428, 2024-08-19T14:21:36.680 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:21:36.680 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:21:36.680 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:21:36.680 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:21:36.681 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:21:36.681 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:21:36.681 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 8281 2024-08-19T14:21:36.681 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.681 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:21:36.681 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.681 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:21:36.681 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.695230+0000", 2024-08-19T14:21:36.681 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:21:36.681 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.681 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.681 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:21:36.681 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.695230+0000", 2024-08-19T14:21:36.681 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:21:36.682 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.682 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.682 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:21:36.682 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.695231+0000", 2024-08-19T14:21:36.682 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0249999999999999e-06 2024-08-19T14:21:36.682 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.682 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.682 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:21:36.682 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.695295+0000", 2024-08-19T14:21:36.682 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4115999999999994e-05 2024-08-19T14:21:36.682 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.682 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.682 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:21:36.683 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.695309+0000", 2024-08-19T14:21:36.683 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.383e-05 2024-08-19T14:21:36.683 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.683 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.683 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:21:36.683 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.695317+0000", 2024-08-19T14:21:36.683 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.0490000000000004e-06 2024-08-19T14:21:36.683 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.683 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.683 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:21:36.683 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.695368+0000", 2024-08-19T14:21:36.683 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1819e-05 2024-08-19T14:21:36.683 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.683 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.683 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:21:36.684 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.695445+0000", 2024-08-19T14:21:36.684 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6705999999999994e-05 2024-08-19T14:21:36.684 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.684 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.684 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,10", 2024-08-19T14:21:36.684 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.695523+0000", 2024-08-19T14:21:36.684 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7989000000000002e-05 2024-08-19T14:21:36.684 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.684 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.684 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:21:36.684 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.696398+0000", 2024-08-19T14:21:36.684 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00087526300000000004 2024-08-19T14:21:36.684 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.684 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.685 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:36.685 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.696820+0000", 2024-08-19T14:21:36.685 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00042155799999999999 2024-08-19T14:21:36.685 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.685 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.685 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:36.685 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.889226+0000", 2024-08-19T14:21:36.685 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.19240576600000001 2024-08-19T14:21:36.685 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.685 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.686 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:21:36.686 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.889276+0000", 2024-08-19T14:21:36.686 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0389000000000002e-05 2024-08-19T14:21:36.686 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.686 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.686 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:21:36.686 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.889312+0000", 2024-08-19T14:21:36.686 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5917999999999997e-05 2024-08-19T14:21:36.686 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:21:36.686 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:21:36.686 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:21:36.686 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.686 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.686 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:8284 2.7 2:eea07cd5:::benchmark_data_smithi045_46415_object8283:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-08-19T14:21:36.687 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:12:51.695765+0000", 2024-08-19T14:21:36.687 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 524.95781794200002, 2024-08-19T14:21:36.687 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.19395319999999999, 2024-08-19T14:21:36.687 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:21:36.687 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:21:36.687 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:21:36.687 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:21:36.687 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:21:36.687 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:21:36.687 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 8284 2024-08-19T14:21:36.687 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.687 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:21:36.687 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.687 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:21:36.688 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.695765+0000", 2024-08-19T14:21:36.688 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:21:36.688 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.688 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.688 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:21:36.688 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.695765+0000", 2024-08-19T14:21:36.688 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:21:36.688 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.688 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.688 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:21:36.688 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.695766+0000", 2024-08-19T14:21:36.688 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2399999999999997e-07 2024-08-19T14:21:36.688 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.689 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.689 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:21:36.689 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.695825+0000", 2024-08-19T14:21:36.689 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8702000000000001e-05 2024-08-19T14:21:36.689 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.689 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.689 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:21:36.689 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.695845+0000", 2024-08-19T14:21:36.689 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0500999999999999e-05 2024-08-19T14:21:36.689 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.689 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.689 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:21:36.689 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.695850+0000", 2024-08-19T14:21:36.689 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3319999999999999e-06 2024-08-19T14:21:36.690 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.690 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.690 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:21:36.690 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.695932+0000", 2024-08-19T14:21:36.690 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.2089000000000006e-05 2024-08-19T14:21:36.690 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.690 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.690 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:21:36.690 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.696041+0000", 2024-08-19T14:21:36.690 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000108931 2024-08-19T14:21:36.690 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.690 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.690 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,10", 2024-08-19T14:21:36.691 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.696129+0000", 2024-08-19T14:21:36.691 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.7545000000000003e-05 2024-08-19T14:21:36.691 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.691 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.691 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:21:36.691 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.696793+0000", 2024-08-19T14:21:36.691 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000663946 2024-08-19T14:21:36.691 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.691 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.691 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:36.691 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.697250+0000", 2024-08-19T14:21:36.691 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000457541 2024-08-19T14:21:36.691 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.691 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.692 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:36.692 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.889620+0000", 2024-08-19T14:21:36.692 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.19236932600000001 2024-08-19T14:21:36.692 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.692 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.692 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:21:36.692 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.889680+0000", 2024-08-19T14:21:36.692 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0417000000000001e-05 2024-08-19T14:21:36.692 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.692 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.692 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:21:36.692 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.889718+0000", 2024-08-19T14:21:36.692 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8145999999999997e-05 2024-08-19T14:21:36.692 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:21:36.693 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:21:36.693 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:21:36.693 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.693 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.693 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:8285 2.7 2:e4063f99:::benchmark_data_smithi045_46415_object8284:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-08-19T14:21:36.693 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:12:51.695865+0000", 2024-08-19T14:21:36.693 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 524.95771807599999, 2024-08-19T14:21:36.693 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.19405726300000001, 2024-08-19T14:21:36.693 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:21:36.693 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:21:36.693 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:21:36.693 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:21:36.693 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:21:36.693 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:21:36.694 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 8285 2024-08-19T14:21:36.694 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.694 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:21:36.694 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.694 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:21:36.694 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.695865+0000", 2024-08-19T14:21:36.694 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:21:36.694 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.694 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.694 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:21:36.695 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.695865+0000", 2024-08-19T14:21:36.695 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:21:36.695 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.695 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.695 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:21:36.695 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.695866+0000", 2024-08-19T14:21:36.695 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8999999999999996e-07 2024-08-19T14:21:36.695 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.695 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.696 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:21:36.696 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.695901+0000", 2024-08-19T14:21:36.696 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5859999999999999e-05 2024-08-19T14:21:36.696 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.696 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.696 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:21:36.696 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.695913+0000", 2024-08-19T14:21:36.696 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1114000000000001e-05 2024-08-19T14:21:36.696 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.696 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.696 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:21:36.696 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.695920+0000", 2024-08-19T14:21:36.696 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1949999999999997e-06 2024-08-19T14:21:36.696 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.697 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.697 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:21:36.697 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.696419+0000", 2024-08-19T14:21:36.697 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00049909300000000004 2024-08-19T14:21:36.697 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.697 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.697 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:21:36.697 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.696489+0000", 2024-08-19T14:21:36.697 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9915999999999999e-05 2024-08-19T14:21:36.697 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.697 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.697 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,10", 2024-08-19T14:21:36.697 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.696578+0000", 2024-08-19T14:21:36.697 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.8825000000000004e-05 2024-08-19T14:21:36.698 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.698 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.698 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:21:36.698 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.696895+0000", 2024-08-19T14:21:36.698 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000317912 2024-08-19T14:21:36.698 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.698 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.698 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:36.698 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.697442+0000", 2024-08-19T14:21:36.698 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00054631000000000005 2024-08-19T14:21:36.698 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.698 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.698 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:36.699 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.889765+0000", 2024-08-19T14:21:36.699 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.192323409 2024-08-19T14:21:36.699 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.699 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.699 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:21:36.699 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.889882+0000", 2024-08-19T14:21:36.699 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00011684 2024-08-19T14:21:36.699 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.699 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.699 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:21:36.700 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.889922+0000", 2024-08-19T14:21:36.700 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0198999999999999e-05 2024-08-19T14:21:36.700 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:21:36.700 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:21:36.700 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:21:36.700 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.700 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.700 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:8318 2.7 2:e0a01dab:::benchmark_data_smithi045_46415_object8317:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-08-19T14:21:36.700 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:12:51.706539+0000", 2024-08-19T14:21:36.700 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 524.94704423500002, 2024-08-19T14:21:36.700 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.183777518, 2024-08-19T14:21:36.700 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:21:36.700 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:21:36.700 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:21:36.701 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:21:36.701 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:21:36.701 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:21:36.701 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 8318 2024-08-19T14:21:36.701 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.701 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:21:36.701 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.701 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:21:36.701 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.706539+0000", 2024-08-19T14:21:36.701 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:21:36.701 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.701 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.701 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:21:36.701 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.706539+0000", 2024-08-19T14:21:36.702 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:21:36.702 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.702 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.702 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:21:36.702 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.706541+0000", 2024-08-19T14:21:36.702 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8050000000000001e-06 2024-08-19T14:21:36.702 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.702 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.702 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:21:36.702 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.706582+0000", 2024-08-19T14:21:36.702 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1748000000000001e-05 2024-08-19T14:21:36.702 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.702 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.702 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:21:36.703 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.706605+0000", 2024-08-19T14:21:36.703 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2614000000000001e-05 2024-08-19T14:21:36.703 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.703 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.703 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:21:36.703 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.706615+0000", 2024-08-19T14:21:36.703 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.825e-06 2024-08-19T14:21:36.703 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.703 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.703 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:21:36.703 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.706681+0000", 2024-08-19T14:21:36.703 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.5729999999999996e-05 2024-08-19T14:21:36.703 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.704 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.704 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:21:36.704 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.706811+0000", 2024-08-19T14:21:36.704 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013086200000000001 2024-08-19T14:21:36.704 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.704 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.704 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,10", 2024-08-19T14:21:36.704 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.706948+0000", 2024-08-19T14:21:36.704 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013628800000000001 2024-08-19T14:21:36.704 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.704 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.704 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:21:36.704 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.707654+0000", 2024-08-19T14:21:36.704 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00070644600000000001 2024-08-19T14:21:36.705 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.705 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.705 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:36.705 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.707895+0000", 2024-08-19T14:21:36.705 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00024116599999999999 2024-08-19T14:21:36.705 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.705 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.705 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:36.705 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.890219+0000", 2024-08-19T14:21:36.705 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.18232368199999999 2024-08-19T14:21:36.705 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.705 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.705 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:21:36.705 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.890279+0000", 2024-08-19T14:21:36.706 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9941000000000001e-05 2024-08-19T14:21:36.706 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.706 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.706 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:21:36.706 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:12:51.890316+0000", 2024-08-19T14:21:36.706 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7410999999999998e-05 2024-08-19T14:21:36.706 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:21:36.706 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:21:36.706 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:21:36.706 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.706 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.706 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:48872 2.7 2:e2eb060c:::benchmark_data_smithi045_46415_object48871:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-08-19T14:21:36.706 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:07.265121+0000", 2024-08-19T14:21:36.706 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 509.38846202799999, 2024-08-19T14:21:36.707 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20642575599999999, 2024-08-19T14:21:36.707 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:21:36.707 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:21:36.707 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:21:36.707 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:21:36.707 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:21:36.707 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:21:36.707 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 48872 2024-08-19T14:21:36.707 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.707 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:21:36.707 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.707 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:21:36.707 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.265121+0000", 2024-08-19T14:21:36.707 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:21:36.708 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.708 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.708 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:21:36.708 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.265121+0000", 2024-08-19T14:21:36.708 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:21:36.708 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.708 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.708 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:21:36.708 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.265123+0000", 2024-08-19T14:21:36.708 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.494e-06 2024-08-19T14:21:36.708 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.708 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.709 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:21:36.709 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.265156+0000", 2024-08-19T14:21:36.709 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3651e-05 2024-08-19T14:21:36.709 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.709 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.709 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:21:36.709 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.265167+0000", 2024-08-19T14:21:36.709 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0669000000000001e-05 2024-08-19T14:21:36.710 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.710 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.710 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:21:36.710 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.265174+0000", 2024-08-19T14:21:36.710 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8750000000000002e-06 2024-08-19T14:21:36.710 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.710 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.710 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:21:36.710 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.265293+0000", 2024-08-19T14:21:36.710 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000119067 2024-08-19T14:21:36.710 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.710 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.711 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:21:36.711 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.265408+0000", 2024-08-19T14:21:36.711 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000115519 2024-08-19T14:21:36.711 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.711 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.711 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,10", 2024-08-19T14:21:36.711 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.265541+0000", 2024-08-19T14:21:36.711 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013279200000000001 2024-08-19T14:21:36.711 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.711 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.711 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:21:36.711 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.268514+0000", 2024-08-19T14:21:36.711 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0029729259999999999 2024-08-19T14:21:36.711 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.712 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.712 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:36.712 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.268576+0000", 2024-08-19T14:21:36.712 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2145999999999998e-05 2024-08-19T14:21:36.712 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.712 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.712 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:36.712 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.471473+0000", 2024-08-19T14:21:36.712 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.202897204 2024-08-19T14:21:36.712 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.712 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.712 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:21:36.712 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.471514+0000", 2024-08-19T14:21:36.712 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0484000000000001e-05 2024-08-19T14:21:36.712 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.713 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.713 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:21:36.713 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.471547+0000", 2024-08-19T14:21:36.713 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2929000000000003e-05 2024-08-19T14:21:36.713 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:21:36.713 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:21:36.713 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:21:36.713 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.713 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.713 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:48876 2.7 2:e761e839:::benchmark_data_smithi045_46415_object48875:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-08-19T14:21:36.713 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:07.265962+0000", 2024-08-19T14:21:36.713 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 509.38762070899998, 2024-08-19T14:21:36.713 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20685887, 2024-08-19T14:21:36.713 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:21:36.714 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:21:36.714 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:21:36.714 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:21:36.714 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:21:36.714 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:21:36.714 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 48876 2024-08-19T14:21:36.714 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.714 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:21:36.714 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.714 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:21:36.714 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.265962+0000", 2024-08-19T14:21:36.714 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:21:36.714 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.714 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.715 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:21:36.715 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.265962+0000", 2024-08-19T14:21:36.715 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:21:36.715 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.715 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.715 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:21:36.715 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.265963+0000", 2024-08-19T14:21:36.715 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0729999999999999e-06 2024-08-19T14:21:36.715 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.715 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.715 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:21:36.715 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.266042+0000", 2024-08-19T14:21:36.715 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.8845999999999997e-05 2024-08-19T14:21:36.715 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.716 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.716 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:21:36.716 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.266060+0000", 2024-08-19T14:21:36.716 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.783e-05 2024-08-19T14:21:36.716 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.716 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.716 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:21:36.716 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.266068+0000", 2024-08-19T14:21:36.716 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7009999999999996e-06 2024-08-19T14:21:36.716 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.716 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.716 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:21:36.716 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.266916+0000", 2024-08-19T14:21:36.716 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00084853200000000004 2024-08-19T14:21:36.717 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.717 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.717 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:21:36.717 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.267096+0000", 2024-08-19T14:21:36.717 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00017937699999999999 2024-08-19T14:21:36.717 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.717 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.717 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,10", 2024-08-19T14:21:36.717 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.267225+0000", 2024-08-19T14:21:36.717 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000129727 2024-08-19T14:21:36.717 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.717 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.717 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:21:36.717 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.268638+0000", 2024-08-19T14:21:36.718 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0014122640000000001 2024-08-19T14:21:36.718 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.718 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.718 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:36.718 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.269422+0000", 2024-08-19T14:21:36.718 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00078479600000000004 2024-08-19T14:21:36.718 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.718 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.718 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:36.718 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.472763+0000", 2024-08-19T14:21:36.718 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.203340627 2024-08-19T14:21:36.718 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.718 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.718 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:21:36.719 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.472797+0000", 2024-08-19T14:21:36.719 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.4060999999999998e-05 2024-08-19T14:21:36.719 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.719 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.719 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:21:36.719 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.472821+0000", 2024-08-19T14:21:36.719 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4036000000000002e-05 2024-08-19T14:21:36.719 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:21:36.719 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:21:36.719 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:21:36.719 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.719 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.719 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:48899 2.7 2:ed417148:::benchmark_data_smithi045_46415_object48898:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-08-19T14:21:36.719 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:07.275772+0000", 2024-08-19T14:21:36.720 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 509.37781096999998, 2024-08-19T14:21:36.720 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.19712979899999999, 2024-08-19T14:21:36.720 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:21:36.720 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:21:36.720 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:21:36.720 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:21:36.720 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:21:36.720 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:21:36.720 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 48899 2024-08-19T14:21:36.720 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.720 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:21:36.720 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.720 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:21:36.720 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.275772+0000", 2024-08-19T14:21:36.721 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:21:36.721 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.721 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.721 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:21:36.721 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.275772+0000", 2024-08-19T14:21:36.721 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:21:36.721 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.721 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.721 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:21:36.721 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.275774+0000", 2024-08-19T14:21:36.721 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5009999999999999e-06 2024-08-19T14:21:36.721 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.721 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.721 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:21:36.722 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.275826+0000", 2024-08-19T14:21:36.722 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2824000000000001e-05 2024-08-19T14:21:36.722 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.722 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.722 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:21:36.722 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.275841+0000", 2024-08-19T14:21:36.722 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4691e-05 2024-08-19T14:21:36.722 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.722 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.722 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:21:36.722 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.275849+0000", 2024-08-19T14:21:36.722 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1580000000000005e-06 2024-08-19T14:21:36.722 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.723 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.723 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:21:36.723 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.275913+0000", 2024-08-19T14:21:36.723 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3590999999999995e-05 2024-08-19T14:21:36.723 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.723 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.723 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:21:36.723 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.276130+0000", 2024-08-19T14:21:36.723 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000216901 2024-08-19T14:21:36.723 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.723 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.723 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,10", 2024-08-19T14:21:36.723 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.276273+0000", 2024-08-19T14:21:36.723 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014370100000000001 2024-08-19T14:21:36.723 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.724 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.724 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:21:36.724 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.278099+0000", 2024-08-19T14:21:36.724 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0018257849999999999 2024-08-19T14:21:36.724 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.724 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.724 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:36.724 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.281287+0000", 2024-08-19T14:21:36.724 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0031876769999999999 2024-08-19T14:21:36.724 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.724 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.724 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:36.724 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.472854+0000", 2024-08-19T14:21:36.725 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.19156674300000001 2024-08-19T14:21:36.725 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.725 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.725 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:21:36.725 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.472881+0000", 2024-08-19T14:21:36.725 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7521999999999998e-05 2024-08-19T14:21:36.725 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.725 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.725 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:21:36.725 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.472902+0000", 2024-08-19T14:21:36.725 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0704999999999999e-05 2024-08-19T14:21:36.725 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:21:36.725 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:21:36.725 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:21:36.725 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.726 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.726 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:48912 2.7 2:e6ffd652:::benchmark_data_smithi045_46415_object48911:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-08-19T14:21:36.726 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:07.279334+0000", 2024-08-19T14:21:36.726 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 509.37424946900001, 2024-08-19T14:21:36.726 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.195037181, 2024-08-19T14:21:36.726 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:21:36.726 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:21:36.726 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:21:36.726 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:21:36.726 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:21:36.726 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:21:36.726 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 48912 2024-08-19T14:21:36.726 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.727 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:21:36.727 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.727 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:21:36.727 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.279334+0000", 2024-08-19T14:21:36.727 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:21:36.727 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.727 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.727 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:21:36.727 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.279334+0000", 2024-08-19T14:21:36.727 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:21:36.727 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.727 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.727 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:21:36.727 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.279335+0000", 2024-08-19T14:21:36.727 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3599999999999999e-06 2024-08-19T14:21:36.728 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.728 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.728 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:21:36.728 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.279409+0000", 2024-08-19T14:21:36.728 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3819000000000006e-05 2024-08-19T14:21:36.728 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.728 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.728 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:21:36.728 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.279420+0000", 2024-08-19T14:21:36.728 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0892e-05 2024-08-19T14:21:36.728 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.728 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.728 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:21:36.728 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.279427+0000", 2024-08-19T14:21:36.729 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2790000000000001e-06 2024-08-19T14:21:36.729 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.729 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.729 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:21:36.729 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.279479+0000", 2024-08-19T14:21:36.729 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2547999999999999e-05 2024-08-19T14:21:36.729 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.729 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.729 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:21:36.729 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.279613+0000", 2024-08-19T14:21:36.729 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013337900000000001 2024-08-19T14:21:36.729 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.729 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.729 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,10", 2024-08-19T14:21:36.729 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.279754+0000", 2024-08-19T14:21:36.730 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000141359 2024-08-19T14:21:36.730 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.730 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.730 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:21:36.730 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.281352+0000", 2024-08-19T14:21:36.730 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0015973070000000001 2024-08-19T14:21:36.730 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.730 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.730 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:21:36.732 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:36.732 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.282744+0000", 2024-08-19T14:21:36.732 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0013921630000000001 2024-08-19T14:21:36.732 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.732 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.732 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:36.732 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.474309+0000", 2024-08-19T14:21:36.732 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.19156536800000001 2024-08-19T14:21:36.733 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.733 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.733 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:21:36.733 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.474347+0000", 2024-08-19T14:21:36.733 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.824e-05 2024-08-19T14:21:36.733 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.733 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.733 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:21:36.733 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.474371+0000", 2024-08-19T14:21:36.733 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3467000000000001e-05 2024-08-19T14:21:36.733 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:21:36.733 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:21:36.733 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:21:36.733 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.734 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.734 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:48921 2.7 2:e2434a00:::benchmark_data_smithi045_46415_object48920:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-08-19T14:21:36.734 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:07.283150+0000", 2024-08-19T14:21:36.734 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 509.370433236, 2024-08-19T14:21:36.734 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.19130543999999999, 2024-08-19T14:21:36.734 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:21:36.734 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:21:36.734 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:21:36.734 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:21:36.734 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:21:36.734 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:21:36.734 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 48921 2024-08-19T14:21:36.734 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.735 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:21:36.735 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.735 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:21:36.735 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.283150+0000", 2024-08-19T14:21:36.735 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:21:36.735 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.735 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.735 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:21:36.735 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.283150+0000", 2024-08-19T14:21:36.735 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:21:36.735 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.735 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.735 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:21:36.735 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.283151+0000", 2024-08-19T14:21:36.736 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6330000000000001e-06 2024-08-19T14:21:36.736 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.736 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.736 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:21:36.736 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.283177+0000", 2024-08-19T14:21:36.736 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5497e-05 2024-08-19T14:21:36.736 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.736 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.736 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:21:36.737 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.283190+0000", 2024-08-19T14:21:36.737 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3315e-05 2024-08-19T14:21:36.737 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.737 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.737 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:21:36.737 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.283197+0000", 2024-08-19T14:21:36.737 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7479999999999996e-06 2024-08-19T14:21:36.737 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.737 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.737 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:21:36.737 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.283262+0000", 2024-08-19T14:21:36.737 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4550999999999999e-05 2024-08-19T14:21:36.737 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.738 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.738 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:21:36.738 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.283434+0000", 2024-08-19T14:21:36.738 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00017221700000000001 2024-08-19T14:21:36.738 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.738 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.738 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,10", 2024-08-19T14:21:36.738 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.283553+0000", 2024-08-19T14:21:36.738 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0001197 2024-08-19T14:21:36.738 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.738 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.738 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:21:36.738 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.285150+0000", 2024-08-19T14:21:36.739 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00159615 2024-08-19T14:21:36.739 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.739 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.739 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:36.739 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.285949+0000", 2024-08-19T14:21:36.739 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00079935399999999995 2024-08-19T14:21:36.739 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.739 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.739 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:36.739 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.474403+0000", 2024-08-19T14:21:36.739 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.18845408199999999 2024-08-19T14:21:36.739 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.739 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.739 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:21:36.740 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.474433+0000", 2024-08-19T14:21:36.740 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9584999999999999e-05 2024-08-19T14:21:36.740 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.740 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.740 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:21:36.740 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.474455+0000", 2024-08-19T14:21:36.740 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2608000000000001e-05 2024-08-19T14:21:36.740 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:21:36.740 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:21:36.740 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:21:36.740 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.740 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.740 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:48971 2.7 2:ecf6a539:::benchmark_data_smithi045_46415_object48970:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-08-19T14:21:36.740 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:07.303688+0000", 2024-08-19T14:21:36.741 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 509.34989523399997, 2024-08-19T14:21:36.741 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.172252192, 2024-08-19T14:21:36.741 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:21:36.741 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:21:36.741 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:21:36.741 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:21:36.741 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:21:36.741 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:21:36.741 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 48971 2024-08-19T14:21:36.741 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.741 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:21:36.741 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.741 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:21:36.741 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.303688+0000", 2024-08-19T14:21:36.742 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:21:36.742 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.742 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.742 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:21:36.742 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.303688+0000", 2024-08-19T14:21:36.742 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:21:36.742 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.742 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.742 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:21:36.742 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.303689+0000", 2024-08-19T14:21:36.742 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.652e-06 2024-08-19T14:21:36.742 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.742 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.743 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:21:36.743 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.303729+0000", 2024-08-19T14:21:36.743 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9443000000000002e-05 2024-08-19T14:21:36.743 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.743 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.743 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:21:36.743 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.303744+0000", 2024-08-19T14:21:36.743 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5173e-05 2024-08-19T14:21:36.743 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.743 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.743 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:21:36.743 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.303751+0000", 2024-08-19T14:21:36.743 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.5910000000000001e-06 2024-08-19T14:21:36.743 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.743 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.744 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:21:36.744 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.303807+0000", 2024-08-19T14:21:36.744 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5918000000000002e-05 2024-08-19T14:21:36.744 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.744 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.744 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:21:36.744 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.303888+0000", 2024-08-19T14:21:36.744 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1302000000000002e-05 2024-08-19T14:21:36.744 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.744 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.744 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,10", 2024-08-19T14:21:36.744 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.303973+0000", 2024-08-19T14:21:36.744 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.5055000000000005e-05 2024-08-19T14:21:36.744 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.745 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.745 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:21:36.745 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.305453+0000", 2024-08-19T14:21:36.745 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001480182 2024-08-19T14:21:36.745 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.745 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.745 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:36.745 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.306105+0000", 2024-08-19T14:21:36.745 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00065189100000000004 2024-08-19T14:21:36.745 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.745 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.745 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:36.745 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.475889+0000", 2024-08-19T14:21:36.745 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.16978352599999999 2024-08-19T14:21:36.746 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.746 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.746 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:21:36.746 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.475919+0000", 2024-08-19T14:21:36.746 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0199e-05 2024-08-19T14:21:36.746 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.746 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.746 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:21:36.746 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:07.475940+0000", 2024-08-19T14:21:36.746 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.126e-05 2024-08-19T14:21:36.746 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:21:36.746 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:21:36.746 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:21:36.746 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.747 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.747 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:62930 2.7 2.377d1257 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-08-19T14:21:36.747 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:18.725228+0000", 2024-08-19T14:21:36.747 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 497.92835485699999, 2024-08-19T14:21:36.747 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.99330006599999998, 2024-08-19T14:21:36.747 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:21:36.747 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:21:36.747 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:21:36.747 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:21:36.747 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:21:36.747 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:21:36.747 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 62930 2024-08-19T14:21:36.747 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.747 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:21:36.748 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.748 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:21:36.748 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.725228+0000", 2024-08-19T14:21:36.748 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:21:36.748 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.748 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.748 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:21:36.748 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.725228+0000", 2024-08-19T14:21:36.748 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:21:36.748 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.748 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.748 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:21:36.748 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.725231+0000", 2024-08-19T14:21:36.748 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5950000000000001e-06 2024-08-19T14:21:36.749 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.749 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.749 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:21:36.749 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.725270+0000", 2024-08-19T14:21:36.749 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9162000000000002e-05 2024-08-19T14:21:36.749 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.749 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.749 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:21:36.749 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.725291+0000", 2024-08-19T14:21:36.749 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1220999999999999e-05 2024-08-19T14:21:36.749 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.749 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.749 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:21:36.750 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.725303+0000", 2024-08-19T14:21:36.750 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1788e-05 2024-08-19T14:21:36.750 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.750 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.750 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:21:36.750 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.725382+0000", 2024-08-19T14:21:36.750 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.9389999999999997e-05 2024-08-19T14:21:36.750 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.750 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.750 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:21:36.750 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.725395+0000", 2024-08-19T14:21:36.750 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.308e-05 2024-08-19T14:21:36.750 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.750 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.751 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:21:36.751 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.718510+0000", 2024-08-19T14:21:36.751 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.99311438900000004 2024-08-19T14:21:36.751 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.751 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.751 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:21:36.751 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.718528+0000", 2024-08-19T14:21:36.751 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8440999999999998e-05 2024-08-19T14:21:36.751 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:21:36.751 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:21:36.751 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:21:36.751 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.751 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.751 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:62932 2.7 2.84394fd7 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-08-19T14:21:36.752 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:18.727423+0000", 2024-08-19T14:21:36.752 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 497.92615969600001, 2024-08-19T14:21:36.752 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.99114474299999999, 2024-08-19T14:21:36.752 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:21:36.752 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:21:36.752 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:21:36.752 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:21:36.752 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:21:36.752 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:21:36.752 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 62932 2024-08-19T14:21:36.752 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.752 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:21:36.752 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.752 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:21:36.752 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.727423+0000", 2024-08-19T14:21:36.753 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:21:36.753 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.753 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.753 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:21:36.753 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.727423+0000", 2024-08-19T14:21:36.753 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:21:36.753 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.753 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.753 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:21:36.753 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.727425+0000", 2024-08-19T14:21:36.753 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0870000000000001e-06 2024-08-19T14:21:36.753 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.753 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.753 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:21:36.754 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.727471+0000", 2024-08-19T14:21:36.754 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5300999999999999e-05 2024-08-19T14:21:36.754 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.754 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.754 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:21:36.754 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.727489+0000", 2024-08-19T14:21:36.754 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7909000000000001e-05 2024-08-19T14:21:36.754 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.754 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.754 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:21:36.754 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.727497+0000", 2024-08-19T14:21:36.754 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.2150000000000001e-06 2024-08-19T14:21:36.754 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.755 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.755 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:21:36.755 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.727590+0000", 2024-08-19T14:21:36.755 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3193e-05 2024-08-19T14:21:36.755 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.755 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.755 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:21:36.755 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.727606+0000", 2024-08-19T14:21:36.755 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6345e-05 2024-08-19T14:21:36.755 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.755 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.755 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:21:36.755 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.718558+0000", 2024-08-19T14:21:36.755 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.99095196399999996 2024-08-19T14:21:36.755 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.756 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.756 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:21:36.756 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.718568+0000", 2024-08-19T14:21:36.756 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.7289999999999995e-06 2024-08-19T14:21:36.756 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:21:36.757 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:21:36.757 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:21:36.757 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.757 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.757 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:62948 2.7 2.63c33357 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-08-19T14:21:36.757 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:18.736424+0000", 2024-08-19T14:21:36.758 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 497.91715927500002, 2024-08-19T14:21:36.758 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.98218305500000003, 2024-08-19T14:21:36.758 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:21:36.758 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:21:36.758 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:21:36.758 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:21:36.758 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:21:36.758 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:21:36.758 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 62948 2024-08-19T14:21:36.758 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.758 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:21:36.758 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.758 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:21:36.758 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.736424+0000", 2024-08-19T14:21:36.759 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:21:36.759 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.759 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.759 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:21:36.759 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.736424+0000", 2024-08-19T14:21:36.759 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:21:36.759 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.759 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.759 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:21:36.759 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.736426+0000", 2024-08-19T14:21:36.759 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9980000000000002e-06 2024-08-19T14:21:36.759 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.759 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.760 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:21:36.760 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.736459+0000", 2024-08-19T14:21:36.760 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3155999999999999e-05 2024-08-19T14:21:36.760 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.760 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.760 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:21:36.760 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.736479+0000", 2024-08-19T14:21:36.760 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0052999999999999e-05 2024-08-19T14:21:36.761 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.761 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.761 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:21:36.761 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.736489+0000", 2024-08-19T14:21:36.761 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.7790000000000007e-06 2024-08-19T14:21:36.761 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.761 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.761 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:21:36.761 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.736567+0000", 2024-08-19T14:21:36.761 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.8240999999999998e-05 2024-08-19T14:21:36.761 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.761 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.761 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:21:36.761 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.736578+0000", 2024-08-19T14:21:36.761 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0813e-05 2024-08-19T14:21:36.762 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.762 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.762 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:21:36.762 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.718594+0000", 2024-08-19T14:21:36.762 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.98201632900000002 2024-08-19T14:21:36.762 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.762 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.762 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:21:36.762 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.718607+0000", 2024-08-19T14:21:36.762 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2686e-05 2024-08-19T14:21:36.762 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:21:36.762 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:21:36.762 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:21:36.763 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.763 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.763 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:62956 2.7 2.bf121107 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-08-19T14:21:36.763 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:18.741261+0000", 2024-08-19T14:21:36.763 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 497.912321846, 2024-08-19T14:21:36.763 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.97739631299999996, 2024-08-19T14:21:36.763 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:21:36.763 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:21:36.763 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:21:36.763 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:21:36.763 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:21:36.763 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:21:36.763 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 62956 2024-08-19T14:21:36.764 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.764 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:21:36.764 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.764 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:21:36.764 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.741261+0000", 2024-08-19T14:21:36.764 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:21:36.764 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.764 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.764 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:21:36.764 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.741261+0000", 2024-08-19T14:21:36.764 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:21:36.764 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.764 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.764 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:21:36.764 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.741263+0000", 2024-08-19T14:21:36.765 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.68e-06 2024-08-19T14:21:36.765 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.765 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.765 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:21:36.765 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.741301+0000", 2024-08-19T14:21:36.765 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8313999999999999e-05 2024-08-19T14:21:36.765 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.765 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.765 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:21:36.765 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.741320+0000", 2024-08-19T14:21:36.765 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.927e-05 2024-08-19T14:21:36.765 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.765 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.766 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:21:36.766 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.741328+0000", 2024-08-19T14:21:36.766 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3000000000000004e-06 2024-08-19T14:21:36.766 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.766 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.766 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:21:36.766 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.741408+0000", 2024-08-19T14:21:36.766 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.9839000000000006e-05 2024-08-19T14:21:36.766 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.766 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.766 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:21:36.766 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.741421+0000", 2024-08-19T14:21:36.766 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2914e-05 2024-08-19T14:21:36.766 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.767 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.767 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:21:36.767 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.718639+0000", 2024-08-19T14:21:36.767 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.97721861399999999 2024-08-19T14:21:36.767 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.767 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.767 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:21:36.767 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.718658+0000", 2024-08-19T14:21:36.767 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8382000000000001e-05 2024-08-19T14:21:36.767 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:21:36.767 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:21:36.767 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:21:36.767 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.767 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.768 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:62963 2.7 2.30acc547 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-08-19T14:21:36.768 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:18.745404+0000", 2024-08-19T14:21:36.768 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 497.908179303, 2024-08-19T14:21:36.768 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.97330407100000005, 2024-08-19T14:21:36.768 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:21:36.768 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:21:36.768 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:21:36.768 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:21:36.768 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:21:36.768 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:21:36.768 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 62963 2024-08-19T14:21:36.768 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.768 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:21:36.768 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.769 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:21:36.769 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.745404+0000", 2024-08-19T14:21:36.769 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:21:36.769 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.769 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.769 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:21:36.769 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.745404+0000", 2024-08-19T14:21:36.769 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:21:36.769 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.769 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.769 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:21:36.769 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.745406+0000", 2024-08-19T14:21:36.769 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.328e-06 2024-08-19T14:21:36.769 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.769 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.770 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:21:36.770 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.745440+0000", 2024-08-19T14:21:36.770 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3837000000000001e-05 2024-08-19T14:21:36.770 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.770 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.770 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:21:36.770 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.745474+0000", 2024-08-19T14:21:36.770 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.4196999999999998e-05 2024-08-19T14:21:36.770 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.770 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.770 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:21:36.770 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.745485+0000", 2024-08-19T14:21:36.770 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.095e-05 2024-08-19T14:21:36.770 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.771 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.771 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:21:36.771 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.745568+0000", 2024-08-19T14:21:36.771 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.2770999999999997e-05 2024-08-19T14:21:36.771 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.771 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.771 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:21:36.771 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.745577+0000", 2024-08-19T14:21:36.771 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.1269999999999992e-06 2024-08-19T14:21:36.771 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.771 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.771 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:21:36.771 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.718689+0000", 2024-08-19T14:21:36.771 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.97311253600000003 2024-08-19T14:21:36.772 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.772 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.772 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:21:36.772 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.718708+0000", 2024-08-19T14:21:36.772 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8325000000000002e-05 2024-08-19T14:21:36.772 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:21:36.772 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:21:36.772 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:21:36.772 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.772 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.772 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:62983 2.7 2.e3420e17 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-08-19T14:21:36.772 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:18.759553+0000", 2024-08-19T14:21:36.772 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 497.89402994400001, 2024-08-19T14:21:36.772 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.95919200800000004, 2024-08-19T14:21:36.773 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:21:36.773 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:21:36.773 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:21:36.773 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:21:36.773 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:21:36.773 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:21:36.773 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 62983 2024-08-19T14:21:36.773 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.773 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:21:36.773 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.773 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:21:36.773 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.759553+0000", 2024-08-19T14:21:36.773 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:21:36.773 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.773 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.774 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:21:36.774 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.759553+0000", 2024-08-19T14:21:36.774 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:21:36.774 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.774 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.774 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:21:36.774 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.759555+0000", 2024-08-19T14:21:36.774 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2630000000000002e-06 2024-08-19T14:21:36.774 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.774 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.774 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:21:36.774 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.759595+0000", 2024-08-19T14:21:36.774 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9490000000000003e-05 2024-08-19T14:21:36.774 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.775 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.775 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:21:36.775 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.759614+0000", 2024-08-19T14:21:36.775 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9446999999999999e-05 2024-08-19T14:21:36.775 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.775 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.775 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:21:36.775 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.759622+0000", 2024-08-19T14:21:36.775 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5519999999999999e-06 2024-08-19T14:21:36.775 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.775 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.775 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:21:36.775 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.759684+0000", 2024-08-19T14:21:36.775 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2168000000000005e-05 2024-08-19T14:21:36.776 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.776 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.776 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:21:36.776 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.759693+0000", 2024-08-19T14:21:36.776 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.8920000000000006e-06 2024-08-19T14:21:36.776 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.776 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.776 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:21:36.776 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.718735+0000", 2024-08-19T14:21:36.776 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.95904258099999995 2024-08-19T14:21:36.776 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.776 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.776 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:21:36.776 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.718745+0000", 2024-08-19T14:21:36.777 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.6150000000000003e-06 2024-08-19T14:21:36.777 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:21:36.777 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:21:36.777 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:21:36.777 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.777 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.777 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:63032 2.7 2.24f95377 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-08-19T14:21:36.777 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:18.809762+0000", 2024-08-19T14:21:36.777 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 497.84382126100002, 2024-08-19T14:21:36.777 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.909118655, 2024-08-19T14:21:36.777 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:21:36.777 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:21:36.777 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:21:36.777 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:21:36.777 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:21:36.778 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:21:36.778 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 63032 2024-08-19T14:21:36.778 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.778 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:21:36.778 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.778 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:21:36.778 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.809762+0000", 2024-08-19T14:21:36.778 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:21:36.778 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.778 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.778 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:21:36.778 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.809762+0000", 2024-08-19T14:21:36.778 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:21:36.779 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.779 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.779 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:21:36.779 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.809764+0000", 2024-08-19T14:21:36.779 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1490000000000001e-06 2024-08-19T14:21:36.779 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.779 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.779 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:21:36.779 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.809805+0000", 2024-08-19T14:21:36.779 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1241e-05 2024-08-19T14:21:36.779 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.779 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.779 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:21:36.779 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.809820+0000", 2024-08-19T14:21:36.779 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4355000000000001e-05 2024-08-19T14:21:36.780 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.780 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.780 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:21:36.780 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.809826+0000", 2024-08-19T14:21:36.780 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4829999999999996e-06 2024-08-19T14:21:36.780 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.780 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.780 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:21:36.780 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.809883+0000", 2024-08-19T14:21:36.780 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6885000000000002e-05 2024-08-19T14:21:36.780 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.780 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.780 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:21:36.780 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.809892+0000", 2024-08-19T14:21:36.781 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.8519999999999993e-06 2024-08-19T14:21:36.781 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.781 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.781 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:21:36.781 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.718868+0000", 2024-08-19T14:21:36.781 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.90897634900000002 2024-08-19T14:21:36.781 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:21:36.781 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:21:36.781 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:21:36.781 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.718880+0000", 2024-08-19T14:21:36.781 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2340999999999999e-05 2024-08-19T14:21:36.781 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:21:36.781 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:21:36.781 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:21:36.781 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:21:36.782 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:21:36.782 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:21:36.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:36.783 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:36.783 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:21:36.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:36 smithi031 ceph-mon[16336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:21:36.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:36 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:21:36.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:36 smithi031 ceph-mon[16336]: osdmap e363: 12 total, 12 up, 9 in 2024-08-19T14:21:36.885 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:21:36.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:36.907 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:36.907 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:21:37.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:36 smithi032 ceph-mon[20004]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:21:37.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:36 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:21:37.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:36 smithi032 ceph-mon[20004]: osdmap e363: 12 total, 12 up, 9 in 2024-08-19T14:21:37.008 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:21:37.039 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:37.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:37.040 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:21:37.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:36 smithi045 ceph-mon[20143]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:21:37.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:36 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:21:37.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:36 smithi045 ceph-mon[20143]: osdmap e363: 12 total, 12 up, 9 in 2024-08-19T14:21:37.049 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 63 16 104078 104062 103.227 58.8125 0.00897197 0.0096812 2024-08-19T14:21:37.112 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:21:37.140 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:21:37.167 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:37.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:37.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:21:37.269 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:21:37.293 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:37.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:37.294 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:21:37.395 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:21:37.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:37.451 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:37.452 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:21:37.553 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:21:37.594 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-08-19T14:21:37.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:37.595 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:37.595 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:21:37.696 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:21:37.722 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:37.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:37.723 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:21:37.737 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:21:37.737 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:21:37.772 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:21:37.773 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:21:37.808 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:21:37.808 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:21:37.808 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:37 smithi031 ceph-mon[16336]: pgmap v814: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 6.2 GiB data, 22 GiB used, 783 GiB / 805 GiB avail; 89 MiB/s wr, 1.42k op/s; 3496/303483 objects misplaced (1.152%); 2.1 MiB/s, 33 objects/s recovering 2024-08-19T14:21:37.808 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:37 smithi031 ceph-mon[16336]: 4.b scrub starts 2024-08-19T14:21:37.808 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:37 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3417516603' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:21:37.824 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:21:37.845 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:21:37.846 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:21:37.854 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:37.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:37.855 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:21:37.878 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-19T14:21:37.878 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:21:37.879 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:21:37.914 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:21:37.915 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:21:37.949 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:21:37.949 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:21:37.956 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:21:37.974 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:21:37.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:37.982 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:37.983 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:21:37.985 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:21:37.986 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:21:38.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:37 smithi032 ceph-mon[20004]: pgmap v814: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 6.2 GiB data, 22 GiB used, 783 GiB / 805 GiB avail; 89 MiB/s wr, 1.42k op/s; 3496/303483 objects misplaced (1.152%); 2.1 MiB/s, 33 objects/s recovering 2024-08-19T14:21:38.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:37 smithi032 ceph-mon[20004]: 4.b scrub starts 2024-08-19T14:21:38.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:37 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3417516603' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:21:38.023 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:21:38.023 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:21:38.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:37 smithi045 ceph-mon[20143]: pgmap v814: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 6.2 GiB data, 22 GiB used, 783 GiB / 805 GiB avail; 89 MiB/s wr, 1.42k op/s; 3496/303483 objects misplaced (1.152%); 2.1 MiB/s, 33 objects/s recovering 2024-08-19T14:21:38.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:37 smithi045 ceph-mon[20143]: 4.b scrub starts 2024-08-19T14:21:38.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:37 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3417516603' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:21:38.049 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 64 16 105144 105128 102.655 66.625 0.00459477 0.00973583 2024-08-19T14:21:38.058 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:21:38.058 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:21:38.084 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:21:38.092 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:21:38.092 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:21:38.110 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:38.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:38.111 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:21:38.125 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:21:38.125 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:21:38.212 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:21:38.242 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:38.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:38.243 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:21:38.345 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:21:38.371 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:38.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:38.372 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:21:38.473 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:21:38.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:38.501 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:38.501 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:21:38.509 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:21:38.509 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:21:38.602 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:21:38.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:38.639 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:38.639 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:21:38.740 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:21:38.768 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:38.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:38.769 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:21:38.870 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:21:38.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:38.910 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:38.910 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:21:38.989 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-19T14:21:39.011 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:21:39.036 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:39.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:39.037 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:21:39.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:38 smithi045 ceph-mon[20143]: 1.8 scrub starts 2024-08-19T14:21:39.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:38 smithi045 ceph-mon[20143]: 1.8 scrub ok 2024-08-19T14:21:39.049 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 65 16 106023 106007 101.921 54.9375 0.00994797 0.00980695 2024-08-19T14:21:39.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:38 smithi031 ceph-mon[16336]: 1.8 scrub starts 2024-08-19T14:21:39.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:38 smithi031 ceph-mon[16336]: 1.8 scrub ok 2024-08-19T14:21:39.130 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:38 smithi032 ceph-mon[20004]: 1.8 scrub starts 2024-08-19T14:21:39.130 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:38 smithi032 ceph-mon[20004]: 1.8 scrub ok 2024-08-19T14:21:39.137 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:21:39.158 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:39.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:39.159 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:21:39.180 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-08-19T14:21:39.260 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:21:39.295 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:39.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:39.296 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:21:39.397 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:21:39.425 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:39.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:39.426 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:21:39.528 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:21:39.558 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:21:39.563 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:39.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:39.564 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:21:39.665 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:21:39.693 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:39.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:39.694 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:21:39.795 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:21:39.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:39.824 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:39.824 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:21:39.925 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:21:39.945 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:39 smithi032 ceph-mon[20004]: pgmap v815: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 6.2 GiB data, 22 GiB used, 783 GiB / 805 GiB avail; 66 MiB/s wr, 1.05k op/s; 3496/304593 objects misplaced (1.148%); 856 KiB/s, 13 objects/s recovering 2024-08-19T14:21:39.945 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:39 smithi032 ceph-mon[20004]: 1.1c scrub starts 2024-08-19T14:21:39.946 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:39 smithi032 ceph-mon[20004]: 1.1c scrub ok 2024-08-19T14:21:39.946 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:39 smithi032 ceph-mon[20004]: 4.b scrub ok 2024-08-19T14:21:39.946 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:39 smithi032 ceph-mon[20004]: 1.4 scrub starts 2024-08-19T14:21:39.946 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:39 smithi032 ceph-mon[20004]: 1.4 scrub ok 2024-08-19T14:21:39.969 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:39.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:39.970 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:21:40.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:39 smithi045 ceph-mon[20143]: pgmap v815: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 6.2 GiB data, 22 GiB used, 783 GiB / 805 GiB avail; 66 MiB/s wr, 1.05k op/s; 3496/304593 objects misplaced (1.148%); 856 KiB/s, 13 objects/s recovering 2024-08-19T14:21:40.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:39 smithi045 ceph-mon[20143]: 1.1c scrub starts 2024-08-19T14:21:40.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:39 smithi045 ceph-mon[20143]: 1.1c scrub ok 2024-08-19T14:21:40.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:39 smithi045 ceph-mon[20143]: 4.b scrub ok 2024-08-19T14:21:40.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:39 smithi045 ceph-mon[20143]: 1.4 scrub starts 2024-08-19T14:21:40.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:39 smithi045 ceph-mon[20143]: 1.4 scrub ok 2024-08-19T14:21:40.049 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 66 16 107135 107119 101.43 69.5 0.00768591 0.00985465 2024-08-19T14:21:40.071 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:21:40.099 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:40.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:40.100 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:21:40.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:39 smithi031 ceph-mon[16336]: pgmap v815: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 6.2 GiB data, 22 GiB used, 783 GiB / 805 GiB avail; 66 MiB/s wr, 1.05k op/s; 3496/304593 objects misplaced (1.148%); 856 KiB/s, 13 objects/s recovering 2024-08-19T14:21:40.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:39 smithi031 ceph-mon[16336]: 1.1c scrub starts 2024-08-19T14:21:40.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:39 smithi031 ceph-mon[16336]: 1.1c scrub ok 2024-08-19T14:21:40.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:39 smithi031 ceph-mon[16336]: 4.b scrub ok 2024-08-19T14:21:40.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:39 smithi031 ceph-mon[16336]: 1.4 scrub starts 2024-08-19T14:21:40.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:39 smithi031 ceph-mon[16336]: 1.4 scrub ok 2024-08-19T14:21:40.201 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:21:40.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:40.230 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:40.231 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:21:40.332 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:21:40.357 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:40.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:40.358 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:21:40.458 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:21:40.485 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:40.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:40.486 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:21:40.587 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:21:40.616 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:40.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:40.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:21:40.684 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:21:40.685 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:21:40.717 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:21:40.745 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:40.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:40.746 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:21:40.835 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:21:40.847 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:21:40.848 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:21:40.896 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:40.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:40.897 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:21:40.999 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:21:41.000 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:40 smithi031 ceph-mon[16336]: 1.c scrub starts 2024-08-19T14:21:41.000 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:40 smithi031 ceph-mon[16336]: 1.c scrub ok 2024-08-19T14:21:41.000 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:40 smithi031 ceph-mon[16336]: 1.19 scrub starts 2024-08-19T14:21:41.000 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:40 smithi031 ceph-mon[16336]: 1.19 scrub ok 2024-08-19T14:21:41.000 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:40 smithi031 ceph-mon[16336]: 1.1a scrub starts 2024-08-19T14:21:41.000 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:40 smithi031 ceph-mon[16336]: 1.1a scrub ok 2024-08-19T14:21:41.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:41.032 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:41.033 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:21:41.049 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 67 16 108300 108284 101.003 72.8125 0.00614896 0.0098962 2024-08-19T14:21:41.063 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:40 smithi032 ceph-mon[20004]: 1.c scrub starts 2024-08-19T14:21:41.063 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:40 smithi032 ceph-mon[20004]: 1.c scrub ok 2024-08-19T14:21:41.063 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:40 smithi032 ceph-mon[20004]: 1.19 scrub starts 2024-08-19T14:21:41.063 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:40 smithi032 ceph-mon[20004]: 1.19 scrub ok 2024-08-19T14:21:41.063 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:40 smithi032 ceph-mon[20004]: 1.1a scrub starts 2024-08-19T14:21:41.063 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:40 smithi032 ceph-mon[20004]: 1.1a scrub ok 2024-08-19T14:21:41.134 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:21:41.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:41.172 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:41.172 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:21:41.273 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:21:41.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:40 smithi045 ceph-mon[20143]: 1.c scrub starts 2024-08-19T14:21:41.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:40 smithi045 ceph-mon[20143]: 1.c scrub ok 2024-08-19T14:21:41.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:40 smithi045 ceph-mon[20143]: 1.19 scrub starts 2024-08-19T14:21:41.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:40 smithi045 ceph-mon[20143]: 1.19 scrub ok 2024-08-19T14:21:41.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:40 smithi045 ceph-mon[20143]: 1.1a scrub starts 2024-08-19T14:21:41.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:40 smithi045 ceph-mon[20143]: 1.1a scrub ok 2024-08-19T14:21:41.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:41.313 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:41.313 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:21:41.414 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:21:41.449 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:41.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:41.451 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:21:41.516 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-08-19T14:21:41.551 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:21:41.589 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:41.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:41.590 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:21:41.691 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:21:41.720 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:41.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:41.721 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:21:41.822 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:21:41.850 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:21:41.851 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:41.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:41.852 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:21:41.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:41 smithi031 ceph-mon[16336]: pgmap v816: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 6.4 GiB data, 23 GiB used, 782 GiB / 805 GiB avail; 79 MiB/s wr, 1.27k op/s; 3310/314217 objects misplaced (1.053%); 2.0 MiB/s, 32 objects/s recovering 2024-08-19T14:21:41.953 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:21:41.990 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:41.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:41.991 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:21:42.050 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 68 16 109381 109365 100.511 67.5625 0.0109122 0.00994528 2024-08-19T14:21:42.093 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:21:42.117 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:42.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:42.118 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:21:42.124 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:41 smithi032 ceph-mon[20004]: pgmap v816: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 6.4 GiB data, 23 GiB used, 782 GiB / 805 GiB avail; 79 MiB/s wr, 1.27k op/s; 3310/314217 objects misplaced (1.053%); 2.0 MiB/s, 32 objects/s recovering 2024-08-19T14:21:42.197 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:21:42.198 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-08-19T14:21:42.219 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:21:42.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:42.251 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:42.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:21:42.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:41 smithi045 ceph-mon[20143]: pgmap v816: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 6.4 GiB data, 23 GiB used, 782 GiB / 805 GiB avail; 79 MiB/s wr, 1.27k op/s; 3310/314217 objects misplaced (1.053%); 2.0 MiB/s, 32 objects/s recovering 2024-08-19T14:21:42.352 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:21:42.378 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:42.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:42.379 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:21:42.417 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:21:42.418 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:21:42.481 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:21:42.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:42.504 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:42.505 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:21:42.605 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:21:42.640 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:42.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:42.641 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:21:42.742 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:21:42.769 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:21:42.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:42.770 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:42.770 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:21:42.812 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is unset 2024-08-19T14:21:42.871 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:21:42.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:42.897 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:42.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:21:42.970 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":817,"stamp":"2024-08-19T14:21:41.901214+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6991118880,"num_objects":106669,"num_object_clones":0,"num_object_copies":320007,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3310,"num_objects_unfound":0,"num_objects_dirty":106669,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":106667,"num_write_kb":6826688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":830,"num_bytes_recovered":54394880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110976,"ondisk_log_size":110976,"up":141,"acting":140,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":142,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":24091576,"kb_used_data":20431856,"kb_used_omap":14,"kb_used_meta":3659633,"kb_avail":819614792,"statfs":{"total":863955320832,"available":839285547008,"internally_reserved":0,"allocated":20922220544,"data_stored":20907107636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14353,"internal_metadata":3747465199},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,4,4,5,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":738918400,"num_objects":11275,"num_object_clones":0,"num_object_copies":33825,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-293,"num_objects_unfound":0,"num_objects_dirty":11275,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11275,"num_write_kb":721600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":293,"num_bytes_recovered":19202048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.004874"},"pg_stats":[{"pgid":"4.5","version":"363'6782","reported_seq":6848,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-08-19T14:21:40.739052+0000","last_change":"2024-08-19T14:20:29.268214+0000","last_active":"2024-08-19T14:21:40.739052+0000","last_peered":"2024-08-19T14:21:40.739052+0000","last_clean":"2024-08-19T14:21:40.739052+0000","last_became_active":"2024-08-19T14:20:29.267549+0000","last_became_peered":"2024-08-19T14:20:29.267549+0000","last_unstale":"2024-08-19T14:21:40.739052+0000","last_undegraded":"2024-08-19T14:21:40.739052+0000","last_fullsized":"2024-08-19T14:21:40.739052+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":6782,"log_dups_size":0,"ondisk_log_size":6782,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:57.640230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":444465152,"num_objects":6782,"num_object_clones":0,"num_object_copies":20346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6782,"num_write_kb":434048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,0],"acting":[11,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"70'139","reported_seq":942,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-08-19T14:21:35.773272+0000","last_change":"2024-08-19T14:19:58.072227+0000","last_active":"2024-08-19T14:21:35.773272+0000","last_peered":"2024-08-19T14:21:35.773272+0000","last_clean":"2024-08-19T14:21:35.773272+0000","last_became_active":"2024-08-19T14:18:13.807133+0000","last_became_peered":"2024-08-19T14:18:13.807133+0000","last_unstale":"2024-08-19T14:21:35.773272+0000","last_undegraded":"2024-08-19T14:21:35.773272+0000","last_fullsized":"2024-08-19T14:21:35.773272+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:19:58.072083+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:19:58.072083+0000","last_clean_scrub_stamp":"2024-08-19T14:19:58.072083+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,9],"acting":[7,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"363'6705","reported_seq":8434,"reported_epoch":363,"state":"active+recovering+undersized+remapped","last_fresh":"2024-08-19T14:21:39.110159+0000","last_change":"2024-08-19T14:21:07.239135+0000","last_active":"2024-08-19T14:21:39.110159+0000","last_peered":"2024-08-19T14:21:39.110159+0000","last_clean":"2024-08-19T14:21:04.784061+0000","last_became_active":"2024-08-19T14:21:07.213050+0000","last_became_peered":"2024-08-19T14:21:07.213050+0000","last_unstale":"2024-08-19T14:21:39.110159+0000","last_undegraded":"2024-08-19T14:21:39.110159+0000","last_fullsized":"2024-08-19T14:21:06.805412+0000","mapping_epoch":354,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":6705,"log_dups_size":0,"ondisk_log_size":6705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":439418880,"num_objects":6705,"num_object_clones":0,"num_object_copies":20115,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3310,"num_objects_unfound":0,"num_objects_dirty":6705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6705,"num_write_kb":429120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":830,"num_bytes_recovered":54394880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,7],"acting":[1,7],"avail_no_missing":["1","5","7"],"object_location_counts":[{"shards":"1,5,7","objects":3310},{"shards":"1,7","objects":3395}],"blocked_by":[],"up_primary":9,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"70'139","reported_seq":841,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-08-19T14:21:35.778564+0000","last_change":"2024-08-19T14:20:03.714917+0000","last_active":"2024-08-19T14:21:35.778564+0000","last_peered":"2024-08-19T14:21:35.778564+0000","last_clean":"2024-08-19T14:21:35.778564+0000","last_became_active":"2024-08-19T14:18:13.804721+0000","last_became_peered":"2024-08-19T14:18:13.804721+0000","last_unstale":"2024-08-19T14:21:35.778564+0000","last_undegraded":"2024-08-19T14:21:35.778564+0000","last_fullsized":"2024-08-19T14:21:35.778564+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:03.714818+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:03.714818+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"363'6643","reported_seq":6709,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-08-19T14:21:39.099051+0000","last_change":"2024-08-19T14:20:29.268110+0000","last_active":"2024-08-19T14:21:39.099051+0000","last_peered":"2024-08-19T14:21:39.099051+0000","last_clean":"2024-08-19T14:21:39.099051+0000","last_became_active":"2024-08-19T14:20:29.267811+0000","last_became_peered":"2024-08-19T14:20:29.267811+0000","last_unstale":"2024-08-19T14:21:39.099051+0000","last_undegraded":"2024-08-19T14:21:39.099051+0000","last_fullsized":"2024-08-19T14:21:39.099051+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":6643,"log_dups_size":0,"ondisk_log_size":6643,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:46.516076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435355648,"num_objects":6643,"num_object_clones":0,"num_object_copies":19929,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6643,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6643,"num_write_kb":425152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"70'139","reported_seq":789,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-08-19T14:21:35.773166+0000","last_change":"2024-08-19T14:20:50.742676+0000","last_active":"2024-08-19T14:21:35.773166+0000","last_peered":"2024-08-19T14:21:35.773166+0000","last_clean":"2024-08-19T14:21:35.773166+0000","last_became_active":"2024-08-19T14:20:50.742354+0000","last_became_peered":"2024-08-19T14:20:50.742354+0000","last_unstale":"2024-08-19T14:21:35.773166+0000","last_undegraded":"2024-08-19T14:21:35.773166+0000","last_fullsized":"2024-08-19T14:21:35.773166+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:19:59.302931+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:19:59.302931+0000","last_clean_scrub_stamp":"2024-08-19T14:19:59.302931+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":114,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"363'6612","reported_seq":6678,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-08-19T14:21:39.099702+0000","last_change":"2024-08-19T14:20:29.267640+0000","last_active":"2024-08-19T14:21:39.099702+0000","last_peered":"2024-08-19T14:21:39.099702+0000","last_clean":"2024-08-19T14:21:39.099702+0000","last_became_active":"2024-08-19T14:20:29.267331+0000","last_became_peered":"2024-08-19T14:20:29.267331+0000","last_unstale":"2024-08-19T14:21:39.099702+0000","last_undegraded":"2024-08-19T14:21:39.099702+0000","last_fullsized":"2024-08-19T14:21:39.099702+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":6612,"log_dups_size":0,"ondisk_log_size":6612,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:46.504191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":433324032,"num_objects":6612,"num_object_clones":0,"num_object_copies":19836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6612,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6612,"num_write_kb":423168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"70'139","reported_seq":838,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-08-19T14:21:35.776041+0000","last_change":"2024-08-19T14:20:09.507422+0000","last_active":"2024-08-19T14:21:35.776041+0000","last_peered":"2024-08-19T14:21:35.776041+0000","last_clean":"2024-08-19T14:21:35.776041+0000","last_became_active":"2024-08-19T14:18:13.808505+0000","last_became_peered":"2024-08-19T14:18:13.808505+0000","last_unstale":"2024-08-19T14:21:35.776041+0000","last_undegraded":"2024-08-19T14:21:35.776041+0000","last_fullsized":"2024-08-19T14:21:35.776041+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:09.507256+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:09.507256+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,11],"acting":[9,0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"363'6660","reported_seq":6726,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-08-19T14:21:38.261277+0000","last_change":"2024-08-19T14:20:29.268766+0000","last_active":"2024-08-19T14:21:38.261277+0000","last_peered":"2024-08-19T14:21:38.261277+0000","last_clean":"2024-08-19T14:21:38.261277+0000","last_became_active":"2024-08-19T14:20:29.268443+0000","last_became_peered":"2024-08-19T14:20:29.268443+0000","last_unstale":"2024-08-19T14:21:38.261277+0000","last_undegraded":"2024-08-19T14:21:38.261277+0000","last_fullsized":"2024-08-19T14:21:38.261277+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":6660,"log_dups_size":0,"ondisk_log_size":6660,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:39.688611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436469760,"num_objects":6660,"num_object_clones":0,"num_object_copies":19980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6660,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6660,"num_write_kb":426240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,11],"acting":[9,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.7","version":"70'139","reported_seq":481,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-08-19T14:21:35.775448+0000","last_change":"2024-08-19T14:20:34.937317+0000","last_active":"2024-08-19T14:21:35.775448+0000","last_peered":"2024-08-19T14:21:35.775448+0000","last_clean":"2024-08-19T14:21:35.775448+0000","last_became_active":"2024-08-19T14:20:28.204384+0000","last_became_peered":"2024-08-19T14:20:28.204384+0000","last_unstale":"2024-08-19T14:21:35.775448+0000","last_undegraded":"2024-08-19T14:21:35.775448+0000","last_fullsized":"2024-08-19T14:21:35.775448+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:34.937192+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:20:34.937192+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:40.330609+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"363'6716","reported_seq":6777,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-08-19T14:21:40.787849+0000","last_change":"2024-08-19T14:20:29.267833+0000","last_active":"2024-08-19T14:21:40.787849+0000","last_peered":"2024-08-19T14:21:40.787849+0000","last_clean":"2024-08-19T14:21:40.787849+0000","last_became_active":"2024-08-19T14:20:29.267379+0000","last_became_peered":"2024-08-19T14:20:29.267379+0000","last_unstale":"2024-08-19T14:21:40.787849+0000","last_undegraded":"2024-08-19T14:21:40.787849+0000","last_fullsized":"2024-08-19T14:21:40.787849+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":6716,"log_dups_size":0,"ondisk_log_size":6716,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":440139776,"num_objects":6716,"num_object_clones":0,"num_object_copies":20148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6716,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6716,"num_write_kb":429824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,1],"acting":[11,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.6","version":"70'139","reported_seq":841,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-08-19T14:21:35.776056+0000","last_change":"2024-08-19T14:19:55.512078+0000","last_active":"2024-08-19T14:21:35.776056+0000","last_peered":"2024-08-19T14:21:35.776056+0000","last_clean":"2024-08-19T14:21:35.776056+0000","last_became_active":"2024-08-19T14:18:26.874902+0000","last_became_peered":"2024-08-19T14:18:26.874902+0000","last_unstale":"2024-08-19T14:21:35.776056+0000","last_undegraded":"2024-08-19T14:21:35.776056+0000","last_fullsized":"2024-08-19T14:21:35.776056+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:19:55.511974+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:27.848870+0000","last_clean_scrub_stamp":"2024-08-19T14:19:55.511974+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,4],"acting":[9,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"363'6607","reported_seq":6672,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-08-19T14:21:39.480428+0000","last_change":"2024-08-19T14:20:29.267682+0000","last_active":"2024-08-19T14:21:39.480428+0000","last_peered":"2024-08-19T14:21:39.480428+0000","last_clean":"2024-08-19T14:21:39.480428+0000","last_became_active":"2024-08-19T14:20:29.267360+0000","last_became_peered":"2024-08-19T14:20:29.267360+0000","last_unstale":"2024-08-19T14:21:39.480428+0000","last_undegraded":"2024-08-19T14:21:39.480428+0000","last_fullsized":"2024-08-19T14:21:39.480428+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":6607,"log_dups_size":0,"ondisk_log_size":6607,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:51.551392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":432996352,"num_objects":6607,"num_object_clones":0,"num_object_copies":19821,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6607,"num_write_kb":422848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"70'139","reported_seq":655,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-08-19T14:21:35.769559+0000","last_change":"2024-08-19T14:20:12.731993+0000","last_active":"2024-08-19T14:21:35.769559+0000","last_peered":"2024-08-19T14:21:35.769559+0000","last_clean":"2024-08-19T14:21:35.769559+0000","last_became_active":"2024-08-19T14:18:26.881435+0000","last_became_peered":"2024-08-19T14:18:26.881435+0000","last_unstale":"2024-08-19T14:21:35.769559+0000","last_undegraded":"2024-08-19T14:21:35.769559+0000","last_fullsized":"2024-08-19T14:21:35.769559+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:12.731850+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:12.731850+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:37.351039+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,4],"acting":[11,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.1","version":"363'6834","reported_seq":6900,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-08-19T14:21:40.773369+0000","last_change":"2024-08-19T14:20:29.268808+0000","last_active":"2024-08-19T14:21:40.773369+0000","last_peered":"2024-08-19T14:21:40.773369+0000","last_clean":"2024-08-19T14:21:40.773369+0000","last_became_active":"2024-08-19T14:20:29.268465+0000","last_became_peered":"2024-08-19T14:20:29.268465+0000","last_unstale":"2024-08-19T14:21:40.773369+0000","last_undegraded":"2024-08-19T14:21:40.773369+0000","last_fullsized":"2024-08-19T14:21:40.773369+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":6834,"log_dups_size":0,"ondisk_log_size":6834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:57.700841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":447873024,"num_objects":6834,"num_object_clones":0,"num_object_copies":20502,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6834,"num_write_kb":437376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.4","version":"70'139","reported_seq":888,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-08-19T14:21:39.314022+0000","last_change":"2024-08-19T14:21:39.314063+0000","last_active":"2024-08-19T14:21:39.314022+0000","last_peered":"2024-08-19T14:21:39.314022+0000","last_clean":"2024-08-19T14:21:39.314022+0000","last_became_active":"2024-08-19T14:18:13.806013+0000","last_became_peered":"2024-08-19T14:18:13.806013+0000","last_unstale":"2024-08-19T14:21:39.314022+0000","last_undegraded":"2024-08-19T14:21:39.314022+0000","last_fullsized":"2024-08-19T14:21:39.314022+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:39.313941+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:21:39.313941+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:22:45.184594+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.f","version":"363'6619","reported_seq":6685,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-08-19T14:21:39.176419+0000","last_change":"2024-08-19T14:20:29.212298+0000","last_active":"2024-08-19T14:21:39.176419+0000","last_peered":"2024-08-19T14:21:39.176419+0000","last_clean":"2024-08-19T14:21:39.176419+0000","last_became_active":"2024-08-19T14:20:29.212003+0000","last_became_peered":"2024-08-19T14:20:29.212003+0000","last_unstale":"2024-08-19T14:21:39.176419+0000","last_undegraded":"2024-08-19T14:21:39.176419+0000","last_fullsized":"2024-08-19T14:21:39.176419+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":6619,"log_dups_size":0,"ondisk_log_size":6619,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:52.621283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":433782784,"num_objects":6619,"num_object_clones":0,"num_object_copies":19857,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6619,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6619,"num_write_kb":423616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,11],"acting":[1,3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"70'139","reported_seq":849,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-08-19T14:21:35.776017+0000","last_change":"2024-08-19T14:20:12.570453+0000","last_active":"2024-08-19T14:21:35.776017+0000","last_peered":"2024-08-19T14:21:35.776017+0000","last_clean":"2024-08-19T14:21:35.776017+0000","last_became_active":"2024-08-19T14:18:26.875006+0000","last_became_peered":"2024-08-19T14:18:26.875006+0000","last_unstale":"2024-08-19T14:21:35.776017+0000","last_undegraded":"2024-08-19T14:21:35.776017+0000","last_fullsized":"2024-08-19T14:21:35.776017+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:12.570352+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:28.864870+0000","last_clean_scrub_stamp":"2024-08-19T14:20:12.570352+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,0],"acting":[9,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.c","version":"363'6502","reported_seq":6568,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-08-19T14:21:38.133361+0000","last_change":"2024-08-19T14:20:29.266724+0000","last_active":"2024-08-19T14:21:38.133361+0000","last_peered":"2024-08-19T14:21:38.133361+0000","last_clean":"2024-08-19T14:21:38.133361+0000","last_became_active":"2024-08-19T14:20:29.266373+0000","last_became_peered":"2024-08-19T14:20:29.266373+0000","last_unstale":"2024-08-19T14:21:38.133361+0000","last_undegraded":"2024-08-19T14:21:38.133361+0000","last_fullsized":"2024-08-19T14:21:38.133361+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":6502,"log_dups_size":0,"ondisk_log_size":6502,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:48.006455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":426115072,"num_objects":6502,"num_object_clones":0,"num_object_copies":19506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6502,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6502,"num_write_kb":416128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,8],"acting":[4,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"70'139","reported_seq":964,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-08-19T14:21:35.773269+0000","last_change":"2024-08-19T14:19:56.106047+0000","last_active":"2024-08-19T14:21:35.773269+0000","last_peered":"2024-08-19T14:21:35.773269+0000","last_clean":"2024-08-19T14:21:35.773269+0000","last_became_active":"2024-08-19T14:18:26.887657+0000","last_became_peered":"2024-08-19T14:18:26.887657+0000","last_unstale":"2024-08-19T14:21:35.773269+0000","last_undegraded":"2024-08-19T14:21:35.773269+0000","last_fullsized":"2024-08-19T14:21:35.773269+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:19:56.105880+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:19:56.105880+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"363'6497","reported_seq":6563,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-08-19T14:21:38.154163+0000","last_change":"2024-08-19T14:20:29.208992+0000","last_active":"2024-08-19T14:21:38.154163+0000","last_peered":"2024-08-19T14:21:38.154163+0000","last_clean":"2024-08-19T14:21:38.154163+0000","last_became_active":"2024-08-19T14:20:29.208599+0000","last_became_peered":"2024-08-19T14:20:29.208599+0000","last_unstale":"2024-08-19T14:21:38.154163+0000","last_undegraded":"2024-08-19T14:21:38.154163+0000","last_fullsized":"2024-08-19T14:21:38.154163+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":6497,"log_dups_size":0,"ondisk_log_size":6497,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:55.930633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":425787392,"num_objects":6497,"num_object_clones":0,"num_object_copies":19491,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6497,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6497,"num_write_kb":415808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,1],"acting":[4,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"70'139","reported_seq":861,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-08-19T14:21:37.951759+0000","last_change":"2024-08-19T14:21:37.951800+0000","last_active":"2024-08-19T14:21:37.951759+0000","last_peered":"2024-08-19T14:21:37.951759+0000","last_clean":"2024-08-19T14:21:37.951759+0000","last_became_active":"2024-08-19T14:18:26.887277+0000","last_became_peered":"2024-08-19T14:18:26.887277+0000","last_unstale":"2024-08-19T14:21:37.951759+0000","last_undegraded":"2024-08-19T14:21:37.951759+0000","last_fullsized":"2024-08-19T14:21:37.951759+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:37.951694+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:21:37.951694+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:03.860882+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,8],"acting":[5,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"363'6482","reported_seq":6548,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-08-19T14:21:38.138701+0000","last_change":"2024-08-19T14:20:29.209096+0000","last_active":"2024-08-19T14:21:38.138701+0000","last_peered":"2024-08-19T14:21:38.138701+0000","last_clean":"2024-08-19T14:21:38.138701+0000","last_became_active":"2024-08-19T14:20:29.208748+0000","last_became_peered":"2024-08-19T14:20:29.208748+0000","last_unstale":"2024-08-19T14:21:38.138701+0000","last_undegraded":"2024-08-19T14:21:38.138701+0000","last_fullsized":"2024-08-19T14:21:38.138701+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":6482,"log_dups_size":0,"ondisk_log_size":6482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:56.597251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":424804352,"num_objects":6482,"num_object_clones":0,"num_object_copies":19446,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6482,"num_write_kb":414848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"70'139","reported_seq":855,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-08-19T14:21:35.769404+0000","last_change":"2024-08-19T14:20:11.733961+0000","last_active":"2024-08-19T14:21:35.769404+0000","last_peered":"2024-08-19T14:21:35.769404+0000","last_clean":"2024-08-19T14:21:35.769404+0000","last_became_active":"2024-08-19T14:18:26.875179+0000","last_became_peered":"2024-08-19T14:18:26.875179+0000","last_unstale":"2024-08-19T14:21:35.769404+0000","last_undegraded":"2024-08-19T14:21:35.769404+0000","last_fullsized":"2024-08-19T14:21:35.769404+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:11.733865+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:11.733865+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:41.250361+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.b","version":"363'6748","reported_seq":6889,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-08-19T14:21:41.189879+0000","last_change":"2024-08-19T14:21:39.089676+0000","last_active":"2024-08-19T14:21:41.189879+0000","last_peered":"2024-08-19T14:21:41.189879+0000","last_clean":"2024-08-19T14:21:41.189879+0000","last_became_active":"2024-08-19T14:20:29.207372+0000","last_became_peered":"2024-08-19T14:20:29.207372+0000","last_unstale":"2024-08-19T14:21:41.189879+0000","last_undegraded":"2024-08-19T14:21:41.189879+0000","last_fullsized":"2024-08-19T14:21:41.189879+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"363'6615","last_scrub_stamp":"2024-08-19T14:21:39.089400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:21:39.089400+0000","objects_scrubbed":6552,"log_size":6748,"log_dups_size":0,"ondisk_log_size":6748,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:05.632119+0000","scrub_duration":1907,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":442236928,"num_objects":6748,"num_object_clones":0,"num_object_copies":20244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6748,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6748,"num_write_kb":431872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"70'139","reported_seq":509,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-08-19T14:21:35.769557+0000","last_change":"2024-08-19T14:20:15.322051+0000","last_active":"2024-08-19T14:21:35.769557+0000","last_peered":"2024-08-19T14:21:35.769557+0000","last_clean":"2024-08-19T14:21:35.769557+0000","last_became_active":"2024-08-19T14:18:26.884105+0000","last_became_peered":"2024-08-19T14:18:26.884105+0000","last_unstale":"2024-08-19T14:21:35.769557+0000","last_undegraded":"2024-08-19T14:21:35.769557+0000","last_fullsized":"2024-08-19T14:21:35.769557+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:15.321945+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:20:15.321945+0000","last_clean_scrub_stamp":"2024-08-19T14:20:15.321945+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:43.904541+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"363'6724","reported_seq":6790,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-08-19T14:21:39.086702+0000","last_change":"2024-08-19T14:20:29.269723+0000","last_active":"2024-08-19T14:21:39.086702+0000","last_peered":"2024-08-19T14:21:39.086702+0000","last_clean":"2024-08-19T14:21:39.086702+0000","last_became_active":"2024-08-19T14:20:29.269432+0000","last_became_peered":"2024-08-19T14:20:29.269432+0000","last_unstale":"2024-08-19T14:21:39.086702+0000","last_undegraded":"2024-08-19T14:21:39.086702+0000","last_fullsized":"2024-08-19T14:21:39.086702+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":6724,"log_dups_size":0,"ondisk_log_size":6724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:44.639451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":440664064,"num_objects":6724,"num_object_clones":0,"num_object_copies":20172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":430336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.f","version":"70'139","reported_seq":761,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-08-19T14:21:35.770935+0000","last_change":"2024-08-19T14:21:14.765419+0000","last_active":"2024-08-19T14:21:35.770935+0000","last_peered":"2024-08-19T14:21:35.770935+0000","last_clean":"2024-08-19T14:21:35.770935+0000","last_became_active":"2024-08-19T14:18:18.711708+0000","last_became_peered":"2024-08-19T14:18:18.711708+0000","last_unstale":"2024-08-19T14:21:35.770935+0000","last_undegraded":"2024-08-19T14:21:35.770935+0000","last_fullsized":"2024-08-19T14:21:35.770935+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:14.765340+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:21:14.765340+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:22:24.415219+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"363'6795","reported_seq":6861,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-08-19T14:21:40.760783+0000","last_change":"2024-08-19T14:20:29.212320+0000","last_active":"2024-08-19T14:21:40.760783+0000","last_peered":"2024-08-19T14:21:40.760783+0000","last_clean":"2024-08-19T14:21:40.760783+0000","last_became_active":"2024-08-19T14:20:29.211975+0000","last_became_peered":"2024-08-19T14:20:29.211975+0000","last_unstale":"2024-08-19T14:21:40.760783+0000","last_undegraded":"2024-08-19T14:21:40.760783+0000","last_fullsized":"2024-08-19T14:21:40.760783+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":6795,"log_dups_size":0,"ondisk_log_size":6795,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":445317120,"num_objects":6795,"num_object_clones":0,"num_object_copies":20385,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6795,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6795,"num_write_kb":434880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,3],"acting":[11,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.c","version":"70'139","reported_seq":614,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-08-19T14:21:35.776759+0000","last_change":"2024-08-19T14:20:07.839218+0000","last_active":"2024-08-19T14:21:35.776759+0000","last_peered":"2024-08-19T14:21:35.776759+0000","last_clean":"2024-08-19T14:21:35.776759+0000","last_became_active":"2024-08-19T14:18:26.872942+0000","last_became_peered":"2024-08-19T14:18:26.872942+0000","last_unstale":"2024-08-19T14:21:35.776759+0000","last_undegraded":"2024-08-19T14:21:35.776759+0000","last_fullsized":"2024-08-19T14:21:35.776759+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:07.839076+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:20:07.839076+0000","last_clean_scrub_stamp":"2024-08-19T14:20:07.839076+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:39.146515+0000","scrub_duration":62,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,9],"acting":[3,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"363'6741","reported_seq":6807,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-08-19T14:21:40.762624+0000","last_change":"2024-08-19T14:20:29.268318+0000","last_active":"2024-08-19T14:21:40.762624+0000","last_peered":"2024-08-19T14:21:40.762624+0000","last_clean":"2024-08-19T14:21:40.762624+0000","last_became_active":"2024-08-19T14:20:29.267442+0000","last_became_peered":"2024-08-19T14:20:29.267442+0000","last_unstale":"2024-08-19T14:21:40.762624+0000","last_undegraded":"2024-08-19T14:21:40.762624+0000","last_fullsized":"2024-08-19T14:21:40.762624+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":6741,"log_dups_size":0,"ondisk_log_size":6741,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:47.715801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":441778176,"num_objects":6741,"num_object_clones":0,"num_object_copies":20223,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6741,"num_write_kb":431424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,9],"acting":[11,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.d","version":"70'139","reported_seq":537,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-08-19T14:21:35.774121+0000","last_change":"2024-08-19T14:21:07.155294+0000","last_active":"2024-08-19T14:21:35.774121+0000","last_peered":"2024-08-19T14:21:35.774121+0000","last_clean":"2024-08-19T14:21:35.774121+0000","last_became_active":"2024-08-19T14:18:26.882002+0000","last_became_peered":"2024-08-19T14:18:26.882002+0000","last_unstale":"2024-08-19T14:21:35.774121+0000","last_undegraded":"2024-08-19T14:21:35.774121+0000","last_fullsized":"2024-08-19T14:21:35.774121+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:07.155157+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:21:07.155157+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:22:11.274708+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"70'139","reported_seq":796,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-08-19T14:21:35.773998+0000","last_change":"2024-08-19T14:21:13.777891+0000","last_active":"2024-08-19T14:21:35.773998+0000","last_peered":"2024-08-19T14:21:35.773998+0000","last_clean":"2024-08-19T14:21:35.773998+0000","last_became_active":"2024-08-19T14:18:18.703594+0000","last_became_peered":"2024-08-19T14:18:18.703594+0000","last_unstale":"2024-08-19T14:21:35.773998+0000","last_undegraded":"2024-08-19T14:21:35.773998+0000","last_fullsized":"2024-08-19T14:21:35.773998+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:13.777777+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:21:13.777777+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:22:38.473560+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"70'139","reported_seq":848,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-08-19T14:21:35.778232+0000","last_change":"2024-08-19T14:20:07.692054+0000","last_active":"2024-08-19T14:21:35.778232+0000","last_peered":"2024-08-19T14:21:35.778232+0000","last_clean":"2024-08-19T14:21:35.778232+0000","last_became_active":"2024-08-19T14:18:18.698334+0000","last_became_peered":"2024-08-19T14:18:18.698334+0000","last_unstale":"2024-08-19T14:21:35.778232+0000","last_undegraded":"2024-08-19T14:21:35.778232+0000","last_fullsized":"2024-08-19T14:21:35.778232+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:07.691932+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:07.691932+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:36.578885+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.10","version":"70'139","reported_seq":951,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-08-19T14:21:35.770665+0000","last_change":"2024-08-19T14:21:07.768770+0000","last_active":"2024-08-19T14:21:35.770665+0000","last_peered":"2024-08-19T14:21:35.770665+0000","last_clean":"2024-08-19T14:21:35.770665+0000","last_became_active":"2024-08-19T14:18:18.704562+0000","last_became_peered":"2024-08-19T14:18:18.704562+0000","last_unstale":"2024-08-19T14:21:35.770665+0000","last_undegraded":"2024-08-19T14:21:35.770665+0000","last_fullsized":"2024-08-19T14:21:35.770665+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:07.768697+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:21:07.768697+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:22:10.360289+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,8],"acting":[5,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"70'139","reported_seq":500,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-08-19T14:21:35.777589+0000","last_change":"2024-08-19T14:21:16.071812+0000","last_active":"2024-08-19T14:21:35.777589+0000","last_peered":"2024-08-19T14:21:35.777589+0000","last_clean":"2024-08-19T14:21:35.777589+0000","last_became_active":"2024-08-19T14:18:18.697894+0000","last_became_peered":"2024-08-19T14:18:18.697894+0000","last_unstale":"2024-08-19T14:21:35.777589+0000","last_undegraded":"2024-08-19T14:21:35.777589+0000","last_fullsized":"2024-08-19T14:21:35.777589+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:16.071703+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:21:16.071703+0000","last_clean_scrub_stamp":"2024-08-19T14:21:16.071703+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:22:18.184091+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"70'139","reported_seq":477,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-08-19T14:21:35.784772+0000","last_change":"2024-08-19T14:20:28.205212+0000","last_active":"2024-08-19T14:21:35.784772+0000","last_peered":"2024-08-19T14:21:35.784772+0000","last_clean":"2024-08-19T14:21:35.784772+0000","last_became_active":"2024-08-19T14:20:28.204724+0000","last_became_peered":"2024-08-19T14:20:28.204724+0000","last_unstale":"2024-08-19T14:21:35.784772+0000","last_undegraded":"2024-08-19T14:21:35.784772+0000","last_fullsized":"2024-08-19T14:21:35.784772+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:08.680021+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:08.680021+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"70'139","reported_seq":597,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-08-19T14:21:35.789524+0000","last_change":"2024-08-19T14:20:31.931658+0000","last_active":"2024-08-19T14:21:35.789524+0000","last_peered":"2024-08-19T14:21:35.789524+0000","last_clean":"2024-08-19T14:21:35.789524+0000","last_became_active":"2024-08-19T14:18:31.901556+0000","last_became_peered":"2024-08-19T14:18:31.901556+0000","last_unstale":"2024-08-19T14:21:35.789524+0000","last_undegraded":"2024-08-19T14:21:35.789524+0000","last_fullsized":"2024-08-19T14:21:35.789524+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:31.931562+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:20:31.931562+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:47.687921+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,0],"acting":[8,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.14","version":"70'139","reported_seq":606,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-08-19T14:21:35.773654+0000","last_change":"2024-08-19T14:19:55.130518+0000","last_active":"2024-08-19T14:21:35.773654+0000","last_peered":"2024-08-19T14:21:35.773654+0000","last_clean":"2024-08-19T14:21:35.773654+0000","last_became_active":"2024-08-19T14:18:18.703926+0000","last_became_peered":"2024-08-19T14:18:18.703926+0000","last_unstale":"2024-08-19T14:21:35.773654+0000","last_undegraded":"2024-08-19T14:21:35.773654+0000","last_fullsized":"2024-08-19T14:21:35.773654+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:19:55.130420+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:19:55.130420+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,1],"acting":[7,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.15","version":"70'139","reported_seq":493,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-08-19T14:21:35.773489+0000","last_change":"2024-08-19T14:20:09.181046+0000","last_active":"2024-08-19T14:21:35.773489+0000","last_peered":"2024-08-19T14:21:35.773489+0000","last_clean":"2024-08-19T14:21:35.773489+0000","last_became_active":"2024-08-19T14:18:31.912496+0000","last_became_peered":"2024-08-19T14:18:31.912496+0000","last_unstale":"2024-08-19T14:21:35.773489+0000","last_undegraded":"2024-08-19T14:21:35.773489+0000","last_fullsized":"2024-08-19T14:21:35.773489+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:09.180923+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:20:09.180923+0000","last_clean_scrub_stamp":"2024-08-19T14:20:09.180923+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"70'139","reported_seq":471,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-08-19T14:21:35.769266+0000","last_change":"2024-08-19T14:20:17.333347+0000","last_active":"2024-08-19T14:21:35.769266+0000","last_peered":"2024-08-19T14:21:35.769266+0000","last_clean":"2024-08-19T14:21:35.769266+0000","last_became_active":"2024-08-19T14:18:31.901229+0000","last_became_peered":"2024-08-19T14:18:31.901229+0000","last_unstale":"2024-08-19T14:21:35.769266+0000","last_undegraded":"2024-08-19T14:21:35.769266+0000","last_fullsized":"2024-08-19T14:21:35.769266+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:17.333260+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:17.333260+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:40.389837+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1b","version":"70'139","reported_seq":823,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-08-19T14:21:35.789174+0000","last_change":"2024-08-19T14:20:16.776086+0000","last_active":"2024-08-19T14:21:35.789174+0000","last_peered":"2024-08-19T14:21:35.789174+0000","last_clean":"2024-08-19T14:21:35.789174+0000","last_became_active":"2024-08-19T14:18:31.901152+0000","last_became_peered":"2024-08-19T14:18:31.901152+0000","last_unstale":"2024-08-19T14:21:35.789174+0000","last_undegraded":"2024-08-19T14:21:35.789174+0000","last_fullsized":"2024-08-19T14:21:35.789174+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:16.775987+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:16.775987+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:46.485727+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.18","version":"70'139","reported_seq":867,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-08-19T14:21:35.770766+0000","last_change":"2024-08-19T14:20:36.510554+0000","last_active":"2024-08-19T14:21:35.770766+0000","last_peered":"2024-08-19T14:21:35.770766+0000","last_clean":"2024-08-19T14:21:35.770766+0000","last_became_active":"2024-08-19T14:18:31.902735+0000","last_became_peered":"2024-08-19T14:18:31.902735+0000","last_unstale":"2024-08-19T14:21:35.770766+0000","last_undegraded":"2024-08-19T14:21:35.770766+0000","last_fullsized":"2024-08-19T14:21:35.770766+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:36.510442+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:36.510442+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:53.706735+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.19","version":"70'139","reported_seq":823,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-08-19T14:21:40.219911+0000","last_change":"2024-08-19T14:21:40.219949+0000","last_active":"2024-08-19T14:21:40.219911+0000","last_peered":"2024-08-19T14:21:40.219911+0000","last_clean":"2024-08-19T14:21:40.219911+0000","last_became_active":"2024-08-19T14:18:31.902547+0000","last_became_peered":"2024-08-19T14:18:31.902547+0000","last_unstale":"2024-08-19T14:21:40.219911+0000","last_undegraded":"2024-08-19T14:21:40.219911+0000","last_fullsized":"2024-08-19T14:21:40.219911+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:40.219830+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:21:40.219830+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:05.414864+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1e","version":"70'139","reported_seq":473,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-08-19T14:21:35.769540+0000","last_change":"2024-08-19T14:20:18.771836+0000","last_active":"2024-08-19T14:21:35.769540+0000","last_peered":"2024-08-19T14:21:35.769540+0000","last_clean":"2024-08-19T14:21:35.769540+0000","last_became_active":"2024-08-19T14:18:31.902546+0000","last_became_peered":"2024-08-19T14:18:31.902546+0000","last_unstale":"2024-08-19T14:21:35.769540+0000","last_undegraded":"2024-08-19T14:21:35.769540+0000","last_fullsized":"2024-08-19T14:21:35.769540+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:18.771704+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:18.771704+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:47.154907+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1c","version":"70'139","reported_seq":589,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-08-19T14:21:38.753623+0000","last_change":"2024-08-19T14:21:38.753663+0000","last_active":"2024-08-19T14:21:38.753623+0000","last_peered":"2024-08-19T14:21:38.753623+0000","last_clean":"2024-08-19T14:21:38.753623+0000","last_became_active":"2024-08-19T14:18:31.902494+0000","last_became_peered":"2024-08-19T14:18:31.902494+0000","last_unstale":"2024-08-19T14:21:38.753623+0000","last_undegraded":"2024-08-19T14:21:38.753623+0000","last_fullsized":"2024-08-19T14:21:38.753623+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:38.753553+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:21:38.753553+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:22:46.720742+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,11],"acting":[8,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1d","version":"70'139","reported_seq":503,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-08-19T14:21:35.772788+0000","last_change":"2024-08-19T14:20:17.725615+0000","last_active":"2024-08-19T14:21:35.772788+0000","last_peered":"2024-08-19T14:21:35.772788+0000","last_clean":"2024-08-19T14:21:35.772788+0000","last_became_active":"2024-08-19T14:18:31.901183+0000","last_became_peered":"2024-08-19T14:18:31.901183+0000","last_unstale":"2024-08-19T14:21:35.772788+0000","last_undegraded":"2024-08-19T14:21:35.772788+0000","last_fullsized":"2024-08-19T14:21:35.772788+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:17.725477+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:17.725477+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,5],"acting":[1,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":6990528512,"num_objects":106667,"num_object_clones":0,"num_object_copies":320001,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3310,"num_objects_unfound":0,"num_objects_dirty":106667,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":106667,"num_write_kb":6826688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":830,"num_bytes_recovered":54394880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":21004288000,"data_stored":21004288000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":106667,"ondisk_log_size":106667,"up":48,"acting":47,"num_store_stats":9},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1802240,"data_stored":1788032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187580,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3421088,"kb_used_data":3024792,"kb_used_omap":1,"kb_used_meta":396286,"kb_avail":90324064,"statfs":{"total":95995035648,"available":92491841536,"internally_reserved":0,"allocated":3097387008,"data_stored":3095708980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":405797316},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":343597383809,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":74,"seq":317827580037,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1915424,"kb_used_data":1464984,"kb_used_omap":1,"kb_used_meta":450430,"kb_avail":91829728,"statfs":{"total":95995035648,"available":94033641472,"internally_reserved":0,"allocated":1500143616,"data_stored":1498465588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":461240769},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874379,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":808544,"kb_used_data":426136,"kb_used_omap":1,"kb_used_meta":382398,"kb_avail":92936608,"statfs":{"total":95995035648,"available":95167086592,"internally_reserved":0,"allocated":436363264,"data_stored":434685236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":391576005},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":57,"seq":244813136013,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3807080,"kb_used_data":3378848,"kb_used_omap":1,"kb_used_meta":428222,"kb_avail":89938072,"statfs":{"total":95995035648,"available":92096585728,"internally_reserved":0,"allocated":3459940352,"data_stored":3458258772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":438499786},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":206158430353,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691990,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1917992,"kb_used_data":1584160,"kb_used_omap":1,"kb_used_meta":333822,"kb_avail":91827160,"statfs":{"total":95995035648,"available":94031011840,"internally_reserved":0,"allocated":1622179840,"data_stored":1620498260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":341834181},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986330,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2543008,"kb_used_data":2082840,"kb_used_omap":1,"kb_used_meta":460158,"kb_avail":91202144,"statfs":{"total":95995035648,"available":93390995456,"internally_reserved":0,"allocated":2132828160,"data_stored":2131150132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":471202245},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149856,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2946208,"kb_used_data":2536024,"kb_used_omap":1,"kb_used_meta":410174,"kb_avail":90798944,"statfs":{"total":95995035648,"available":92978118656,"internally_reserved":0,"allocated":2596888576,"data_stored":2595210548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":420018631},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,4,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411494,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575017,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4674984,"kb_used_data":4214816,"kb_used_omap":1,"kb_used_meta":460158,"kb_avail":89070168,"statfs":{"total":95995035648,"available":91207852032,"internally_reserved":0,"allocated":4315971584,"data_stored":4314290004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":471202245},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,4,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738542,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2057248,"kb_used_data":1719256,"kb_used_omap":1,"kb_used_meta":337982,"kb_avail":91687904,"statfs":{"total":95995035648,"available":93888413696,"internally_reserved":0,"allocated":1760518144,"data_stored":1758840116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":346094021},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1775435776,"data_stored":1775435776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4341628928,"data_stored":4341628928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2600599552,"data_stored":2600599552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2144600064,"data_stored":2144600064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1627914240,"data_stored":1627914240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3459383296,"data_stored":3459383296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":433192960,"data_stored":433192960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1510932480,"data_stored":1510932480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":3110600704,"data_stored":3110600704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T14:21:42.973 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph status --format=json 2024-08-19T14:21:42.999 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:21:43.022 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:43.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:43.024 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:21:43.050 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 69 16 110336 110320 99.919 59.6875 0.0137589 0.0100033 2024-08-19T14:21:43.069 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:42 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/480781314' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:21:43.069 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:42 smithi031 ceph-mon[16336]: 4.9 scrub starts 2024-08-19T14:21:43.124 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:21:43.157 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:42 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/480781314' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:21:43.157 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:42 smithi032 ceph-mon[20004]: 4.9 scrub starts 2024-08-19T14:21:43.175 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:43.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:43.176 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:21:43.194 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-08-19T14:21:43.224 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:42 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/480781314' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:21:43.224 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:42 smithi045 ceph-mon[20143]: 4.9 scrub starts 2024-08-19T14:21:43.276 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:21:43.323 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:43.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:43.324 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:21:43.425 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:21:43.466 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:43.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:43.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:21:43.566 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:21:43.568 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:21:43.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:43.623 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:43.623 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:21:43.683 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:21:43.695 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:21:43.724 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:21:43.750 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:43.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:43.751 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:21:43.812 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:43 smithi031 ceph-mon[16336]: pgmap v817: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 6.5 GiB data, 23 GiB used, 782 GiB / 805 GiB avail; 70 MiB/s wr, 1.13k op/s; 3310/320007 objects misplaced (1.034%); 1.8 MiB/s, 29 objects/s recovering 2024-08-19T14:21:43.813 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:43 smithi031 ceph-mon[16336]: from='client.25462 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:21:43.813 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:43 smithi031 ceph-mon[16336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:21:43.813 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:43 smithi031 ceph-mon[16336]: Cluster is now healthy 2024-08-19T14:21:43.813 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:43 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/480781314' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:21:43.813 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:43 smithi031 ceph-mon[16336]: osdmap e364: 12 total, 12 up, 9 in 2024-08-19T14:21:43.852 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:21:43.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:43.883 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:43.883 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:21:43.984 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:21:44.006 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:43 smithi045 ceph-mon[20143]: pgmap v817: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 6.5 GiB data, 23 GiB used, 782 GiB / 805 GiB avail; 70 MiB/s wr, 1.13k op/s; 3310/320007 objects misplaced (1.034%); 1.8 MiB/s, 29 objects/s recovering 2024-08-19T14:21:44.007 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:43 smithi045 ceph-mon[20143]: from='client.25462 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:21:44.007 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:43 smithi045 ceph-mon[20143]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:21:44.007 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:43 smithi045 ceph-mon[20143]: Cluster is now healthy 2024-08-19T14:21:44.007 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:43 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/480781314' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:21:44.007 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:43 smithi045 ceph-mon[20143]: osdmap e364: 12 total, 12 up, 9 in 2024-08-19T14:21:44.014 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:44.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:44.016 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:21:44.050 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 70 16 111396 111380 99.4379 66.25 0.0201236 0.0100509 2024-08-19T14:21:44.117 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:21:44.154 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:44.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:44.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:21:44.159 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:21:44.159 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:21:44.256 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:21:44.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:43 smithi032 ceph-mon[20004]: pgmap v817: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 6.5 GiB data, 23 GiB used, 782 GiB / 805 GiB avail; 70 MiB/s wr, 1.13k op/s; 3310/320007 objects misplaced (1.034%); 1.8 MiB/s, 29 objects/s recovering 2024-08-19T14:21:44.259 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:43 smithi032 ceph-mon[20004]: from='client.25462 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:21:44.259 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:43 smithi032 ceph-mon[20004]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:21:44.259 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:43 smithi032 ceph-mon[20004]: Cluster is now healthy 2024-08-19T14:21:44.259 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:43 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/480781314' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:21:44.259 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:43 smithi032 ceph-mon[20004]: osdmap e364: 12 total, 12 up, 9 in 2024-08-19T14:21:44.283 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:44.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:44.284 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:21:44.385 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:21:44.411 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:44.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:44.412 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:21:44.461 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:21:44.461 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:21:44.462 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:21:44.462 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:21:44.478 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:21:44.478 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:21:44.512 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:21:44.512 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:21:44.514 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:21:44.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:44.540 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:44.540 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:21:44.545 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:21:44.545 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:21:44.586 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:21:44.587 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:21:44.624 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:21:44.624 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:21:44.641 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:21:44.667 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:21:44.668 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:21:44.668 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:44.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:44.670 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:21:44.713 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:21:44.714 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:21:44.753 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:21:44.754 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:21:44.770 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:21:44.787 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:21:44.787 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:21:44.796 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:44.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:44.797 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:21:44.830 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:21:44.831 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:21:44.897 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:21:44.927 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:44.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:44.929 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:21:45.030 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:21:45.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:44 smithi045 ceph-mon[20143]: 4.9 scrub ok 2024-08-19T14:21:45.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:44 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/115325940' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:21:45.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:44 smithi045 ceph-mon[20143]: 1.e scrub starts 2024-08-19T14:21:45.045 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:44 smithi045 ceph-mon[20143]: 1.e scrub ok 2024-08-19T14:21:45.050 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 71 16 112428 112412 98.9458 64.5 0.0065944 0.0101009 2024-08-19T14:21:45.063 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:45.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:45.064 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:21:45.088 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-08-19T14:21:45.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:44 smithi031 ceph-mon[16336]: 4.9 scrub ok 2024-08-19T14:21:45.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:44 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/115325940' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:21:45.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:44 smithi031 ceph-mon[16336]: 1.e scrub starts 2024-08-19T14:21:45.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:44 smithi031 ceph-mon[16336]: 1.e scrub ok 2024-08-19T14:21:45.165 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:21:45.190 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:21:45.191 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"ed7d1c6e-5e33-11ef-bccf-c7b262605968","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":882,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":364,"num_osds":12,"num_up_osds":12,"osd_up_since":1724076699,"num_in_osds":9,"osd_in_since":1724076998,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":46},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":47,"num_pools":2,"num_objects":107954,"data_bytes":7075332640,"bytes_used":25084043264,"bytes_avail":838871277568,"bytes_total":863955320832,"misplaced_objects":3310,"misplaced_total":323862,"misplaced_ratio":0.01022040251712149,"recovering_objects_per_sec":22,"recovering_bytes_per_sec":1498285,"recovering_keys_per_sec":0,"num_objects_recovered":186,"num_bytes_recovered":12189696,"num_keys_recovered":0,"write_bytes_sec":69388351,"read_op_per_sec":0,"write_op_per_sec":1058},"fsmap":{"epoch":1,"btime":"2024-08-19T14:05:22:841940+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.31:8443/"}},"servicemap":{"epoch":20,"modified":"2024-08-19T14:20:27.873686+0000","services":{}},"progress_events":{}} 2024-08-19T14:21:45.194 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:45.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:45.196 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:21:45.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:44 smithi032 ceph-mon[20004]: 4.9 scrub ok 2024-08-19T14:21:45.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:44 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/115325940' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:21:45.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:44 smithi032 ceph-mon[20004]: 1.e scrub starts 2024-08-19T14:21:45.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:44 smithi032 ceph-mon[20004]: 1.e scrub ok 2024-08-19T14:21:45.297 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:21:45.327 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:45.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:45.329 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:21:45.430 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:21:45.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:45.467 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:45.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:21:45.473 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:21:45.568 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:21:45.582 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-19T14:21:45.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:45.601 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:45.601 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:21:45.702 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:21:45.726 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:45.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:45.727 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:21:45.828 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:21:45.859 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:45.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:45.860 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:21:45.899 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:45 smithi045 ceph-mon[20143]: pgmap v819: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 6.6 GiB data, 23 GiB used, 781 GiB / 805 GiB avail; 66 MiB/s wr, 1.06k op/s; 3310/323862 objects misplaced (1.022%); 1.4 MiB/s, 22 objects/s recovering 2024-08-19T14:21:45.899 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:45 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/844160251' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-08-19T14:21:45.945 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:45 smithi031 ceph-mon[16336]: pgmap v819: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 6.6 GiB data, 23 GiB used, 781 GiB / 805 GiB avail; 66 MiB/s wr, 1.06k op/s; 3310/323862 objects misplaced (1.022%); 1.4 MiB/s, 22 objects/s recovering 2024-08-19T14:21:45.945 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:45 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/844160251' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-08-19T14:21:45.961 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:21:45.990 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:45.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:45.991 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:21:46.043 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:21:46.043 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [], 2024-08-19T14:21:46.044 INFO:teuthology.orchestra.run.smithi045.stdout: "complaint_time": 30, 2024-08-19T14:21:46.044 INFO:teuthology.orchestra.run.smithi045.stdout: "num_blocked_ops": 0 2024-08-19T14:21:46.044 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:21:46.050 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 72 16 113584 113568 98.5749 72.25 0.0144136 0.0101405 2024-08-19T14:21:46.092 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:21:46.114 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:46.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:46.115 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:21:46.216 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:21:46.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:45 smithi032 ceph-mon[20004]: pgmap v819: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 6.6 GiB data, 23 GiB used, 781 GiB / 805 GiB avail; 66 MiB/s wr, 1.06k op/s; 3310/323862 objects misplaced (1.022%); 1.4 MiB/s, 22 objects/s recovering 2024-08-19T14:21:46.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:45 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/844160251' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-08-19T14:21:46.258 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:46.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:46.259 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:21:46.360 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:21:46.389 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-08-19T14:21:46.389 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-19T14:21:46.405 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:46.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:46.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:21:46.507 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:21:46.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:46.543 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:46.543 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:21:46.643 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:21:46.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:46.675 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:46.676 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:21:46.778 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:21:46.807 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:46.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:46.808 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:21:46.909 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:21:46.936 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:21:46.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:46.955 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:46.955 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:21:46.969 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:21:47.052 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 73 16 114669 114653 98.1535 67.8125 0.00873728 0.0101833 2024-08-19T14:21:47.056 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:21:47.078 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:46 smithi031 ceph-mon[16336]: 1.7 deep-scrub starts 2024-08-19T14:21:47.079 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:46 smithi031 ceph-mon[16336]: 1.7 deep-scrub ok 2024-08-19T14:21:47.079 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:46 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:21:47.079 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:46 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:21:47.081 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:47.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:47.082 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:21:47.112 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:46 smithi045 ceph-mon[20143]: 1.7 deep-scrub starts 2024-08-19T14:21:47.112 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:46 smithi045 ceph-mon[20143]: 1.7 deep-scrub ok 2024-08-19T14:21:47.112 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:46 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:21:47.112 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:46 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:21:47.157 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-08-19T14:21:47.183 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:21:47.211 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:47.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:47.212 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:21:47.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:46 smithi032 ceph-mon[20004]: 1.7 deep-scrub starts 2024-08-19T14:21:47.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:46 smithi032 ceph-mon[20004]: 1.7 deep-scrub ok 2024-08-19T14:21:47.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:46 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:21:47.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:46 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:21:47.313 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:21:47.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:47.339 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:47.339 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:21:47.440 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:21:47.479 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:47.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:47.481 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:21:47.501 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:21:47.581 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:21:47.610 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:47.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:47.611 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:21:47.712 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:21:47.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:47.741 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:47.741 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:21:47.843 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:21:47.871 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:47.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:47.872 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:21:47.957 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:47 smithi045 ceph-mon[20143]: pgmap v820: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 6.8 GiB data, 24 GiB used, 781 GiB / 805 GiB avail; 66 MiB/s wr, 1.05k op/s; 3181/334935 objects misplaced (0.950%); 2.0 MiB/s, 31 objects/s recovering 2024-08-19T14:21:48.074 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:47 smithi045 ceph-mon[20143]: osdmap e365: 12 total, 12 up, 9 in 2024-08-19T14:21:48.074 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:47 smithi045 ceph-mon[20143]: 4.3 scrub starts 2024-08-19T14:21:48.074 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:21:48.075 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:47 smithi031 ceph-mon[16336]: pgmap v820: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 6.8 GiB data, 24 GiB used, 781 GiB / 805 GiB avail; 66 MiB/s wr, 1.05k op/s; 3181/334935 objects misplaced (0.950%); 2.0 MiB/s, 31 objects/s recovering 2024-08-19T14:21:48.075 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:47 smithi031 ceph-mon[16336]: osdmap e365: 12 total, 12 up, 9 in 2024-08-19T14:21:48.075 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:47 smithi031 ceph-mon[16336]: 4.3 scrub starts 2024-08-19T14:21:48.076 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 74 16 115698 115682 97.6961 64.3125 0.014154 0.0102309 2024-08-19T14:21:48.097 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:48.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:48.098 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:21:48.105 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:21:48.105 INFO:teuthology.orchestra.run.smithi045.stdout: "size": 20, 2024-08-19T14:21:48.105 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 600, 2024-08-19T14:21:48.105 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [ 2024-08-19T14:21:48.105 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.106 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24580.0:10284 2.5 2:a4caf01e:::benchmark_data_smithi045_46415_object10283:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-08-19T14:21:48.106 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:12:52.336926+0000", 2024-08-19T14:21:48.106 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 535.76553811500003, 2024-08-19T14:21:48.106 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.206277084, 2024-08-19T14:21:48.106 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:21:48.106 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:21:48.106 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:21:48.106 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:21:48.106 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24580", 2024-08-19T14:21:48.106 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:21:48.106 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 10284 2024-08-19T14:21:48.106 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.106 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:21:48.106 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.107 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:21:48.107 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.336926+0000", 2024-08-19T14:21:48.107 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:21:48.107 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.107 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.107 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:21:48.107 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.336926+0000", 2024-08-19T14:21:48.107 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:21:48.107 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.107 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.107 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:21:48.107 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.336927+0000", 2024-08-19T14:21:48.107 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3370000000000001e-06 2024-08-19T14:21:48.108 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.108 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.108 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:21:48.108 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.336949+0000", 2024-08-19T14:21:48.108 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1399e-05 2024-08-19T14:21:48.108 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.108 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.108 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:21:48.108 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.336961+0000", 2024-08-19T14:21:48.108 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.224e-05 2024-08-19T14:21:48.108 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.108 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.108 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:21:48.108 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.336968+0000", 2024-08-19T14:21:48.109 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.2710000000000003e-06 2024-08-19T14:21:48.109 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.109 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.109 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:21:48.109 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.337134+0000", 2024-08-19T14:21:48.109 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000165809 2024-08-19T14:21:48.109 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.109 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.109 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:21:48.109 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.337251+0000", 2024-08-19T14:21:48.109 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000116848 2024-08-19T14:21:48.109 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.109 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.109 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,4", 2024-08-19T14:21:48.110 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.337803+0000", 2024-08-19T14:21:48.110 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00055147699999999998 2024-08-19T14:21:48.110 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.110 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.110 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:21:48.110 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.339273+0000", 2024-08-19T14:21:48.110 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0014706750000000001 2024-08-19T14:21:48.110 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.110 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.110 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:48.110 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.339438+0000", 2024-08-19T14:21:48.110 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00016459299999999999 2024-08-19T14:21:48.111 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.111 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.111 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:48.111 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.543107+0000", 2024-08-19T14:21:48.111 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20366880100000001 2024-08-19T14:21:48.111 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.111 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.111 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:21:48.111 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.543162+0000", 2024-08-19T14:21:48.111 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.5075999999999999e-05 2024-08-19T14:21:48.111 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.111 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.111 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:21:48.111 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.543203+0000", 2024-08-19T14:21:48.112 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.1557999999999998e-05 2024-08-19T14:21:48.112 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:21:48.112 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:21:48.112 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:21:48.112 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.112 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.112 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24580.0:10301 2.f 2:f82f4cfa:::benchmark_data_smithi045_46415_object10300:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-08-19T14:21:48.112 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:12:52.341171+0000", 2024-08-19T14:21:48.112 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 535.76129327000001, 2024-08-19T14:21:48.112 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20191952899999999, 2024-08-19T14:21:48.112 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:21:48.112 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:21:48.112 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:21:48.112 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:21:48.113 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24580", 2024-08-19T14:21:48.113 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:21:48.113 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 10301 2024-08-19T14:21:48.113 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.113 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:21:48.113 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.113 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:21:48.113 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.341171+0000", 2024-08-19T14:21:48.113 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:21:48.113 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.113 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.113 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:21:48.113 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.341171+0000", 2024-08-19T14:21:48.114 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:21:48.114 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.114 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.114 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:21:48.114 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.341172+0000", 2024-08-19T14:21:48.114 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.093e-06 2024-08-19T14:21:48.114 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.114 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.114 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:21:48.114 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.341195+0000", 2024-08-19T14:21:48.114 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2435000000000001e-05 2024-08-19T14:21:48.114 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.114 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.114 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:21:48.115 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.341205+0000", 2024-08-19T14:21:48.115 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0567999999999999e-05 2024-08-19T14:21:48.115 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.115 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.115 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:21:48.115 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.341212+0000", 2024-08-19T14:21:48.115 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.686e-06 2024-08-19T14:21:48.115 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.115 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.115 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:21:48.115 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.341290+0000", 2024-08-19T14:21:48.115 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.8572999999999995e-05 2024-08-19T14:21:48.115 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.115 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.116 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:21:48.116 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.341427+0000", 2024-08-19T14:21:48.116 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000136754 2024-08-19T14:21:48.116 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.116 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.116 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 4,9", 2024-08-19T14:21:48.116 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.341748+0000", 2024-08-19T14:21:48.116 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00032119000000000001 2024-08-19T14:21:48.116 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.116 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.116 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:48.116 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.342661+0000", 2024-08-19T14:21:48.116 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000912761 2024-08-19T14:21:48.116 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.117 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.117 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:21:48.117 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.342681+0000", 2024-08-19T14:21:48.117 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0316999999999999e-05 2024-08-19T14:21:48.117 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.117 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.117 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:48.117 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.543009+0000", 2024-08-19T14:21:48.117 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.200327793 2024-08-19T14:21:48.117 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.117 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.117 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:21:48.117 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.543053+0000", 2024-08-19T14:21:48.117 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.4036000000000003e-05 2024-08-19T14:21:48.118 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.118 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.118 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:21:48.118 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.543091+0000", 2024-08-19T14:21:48.118 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.7323000000000002e-05 2024-08-19T14:21:48.118 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:21:48.118 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:21:48.118 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:21:48.118 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.118 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.118 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24580.0:10310 2.5 2:a1615b0f:::benchmark_data_smithi045_46415_object10309:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-08-19T14:21:48.118 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:12:52.342440+0000", 2024-08-19T14:21:48.119 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 535.76002446699999, 2024-08-19T14:21:48.119 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20090964, 2024-08-19T14:21:48.119 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:21:48.119 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:21:48.119 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:21:48.119 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:21:48.120 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24580", 2024-08-19T14:21:48.120 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:21:48.120 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 10310 2024-08-19T14:21:48.120 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.120 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:21:48.120 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.120 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:21:48.120 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.342440+0000", 2024-08-19T14:21:48.120 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:21:48.120 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.120 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.120 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:21:48.121 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.342440+0000", 2024-08-19T14:21:48.121 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:21:48.121 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.121 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.121 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:21:48.121 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.342441+0000", 2024-08-19T14:21:48.121 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.7499999999999998e-07 2024-08-19T14:21:48.121 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.121 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.121 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:21:48.121 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.342463+0000", 2024-08-19T14:21:48.121 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2303999999999999e-05 2024-08-19T14:21:48.121 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.121 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.122 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:21:48.122 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.342477+0000", 2024-08-19T14:21:48.122 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4099e-05 2024-08-19T14:21:48.122 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.122 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.122 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:21:48.122 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.342485+0000", 2024-08-19T14:21:48.122 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.6490000000000008e-06 2024-08-19T14:21:48.122 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.122 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.122 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:21:48.122 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.342571+0000", 2024-08-19T14:21:48.122 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.6238999999999999e-05 2024-08-19T14:21:48.122 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.123 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.123 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:21:48.123 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.343481+0000", 2024-08-19T14:21:48.123 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00091020900000000004 2024-08-19T14:21:48.123 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.123 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.123 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,4", 2024-08-19T14:21:48.123 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.343622+0000", 2024-08-19T14:21:48.123 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014069700000000001 2024-08-19T14:21:48.123 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.123 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.123 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:21:48.123 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.344447+0000", 2024-08-19T14:21:48.123 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00082507499999999998 2024-08-19T14:21:48.124 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.124 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.124 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:48.124 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.346610+0000", 2024-08-19T14:21:48.124 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0021627830000000002 2024-08-19T14:21:48.124 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.124 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.124 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:48.124 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.543255+0000", 2024-08-19T14:21:48.124 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.19664540699999999 2024-08-19T14:21:48.124 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.124 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.124 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:21:48.124 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.543309+0000", 2024-08-19T14:21:48.125 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4231999999999998e-05 2024-08-19T14:21:48.125 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.125 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.125 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:21:48.125 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.543349+0000", 2024-08-19T14:21:48.125 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.9971000000000001e-05 2024-08-19T14:21:48.125 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:21:48.125 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:21:48.125 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:21:48.125 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.125 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.125 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24580.0:10315 2.f 2:fe6f4b87:::benchmark_data_smithi045_46415_object10314:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-08-19T14:21:48.125 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:12:52.343698+0000", 2024-08-19T14:21:48.125 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 535.75876645699998, 2024-08-19T14:21:48.126 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.19949809700000001, 2024-08-19T14:21:48.126 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:21:48.126 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:21:48.126 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:21:48.126 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:21:48.126 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24580", 2024-08-19T14:21:48.126 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:21:48.126 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 10315 2024-08-19T14:21:48.126 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.126 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:21:48.126 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.126 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:21:48.126 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.343698+0000", 2024-08-19T14:21:48.127 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:21:48.127 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.127 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.127 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:21:48.127 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.343698+0000", 2024-08-19T14:21:48.127 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:21:48.127 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.127 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.127 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:21:48.127 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.343699+0000", 2024-08-19T14:21:48.127 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.238e-06 2024-08-19T14:21:48.127 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.127 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.127 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:21:48.128 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.343723+0000", 2024-08-19T14:21:48.128 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.4314000000000001e-05 2024-08-19T14:21:48.128 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.128 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.128 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:21:48.128 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.343735+0000", 2024-08-19T14:21:48.128 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2133999999999999e-05 2024-08-19T14:21:48.128 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.128 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.128 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:21:48.128 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.343743+0000", 2024-08-19T14:21:48.128 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.34e-06 2024-08-19T14:21:48.128 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.128 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.129 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:21:48.129 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.344026+0000", 2024-08-19T14:21:48.129 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00028303099999999999 2024-08-19T14:21:48.129 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.129 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.129 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:21:48.129 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.344278+0000", 2024-08-19T14:21:48.129 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00025181699999999999 2024-08-19T14:21:48.129 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.129 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.129 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 4,9", 2024-08-19T14:21:48.129 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.344520+0000", 2024-08-19T14:21:48.129 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000241864 2024-08-19T14:21:48.129 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.130 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.130 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:21:48.130 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.346213+0000", 2024-08-19T14:21:48.130 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001693398 2024-08-19T14:21:48.130 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.130 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.130 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:48.130 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.346420+0000", 2024-08-19T14:21:48.130 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00020658300000000001 2024-08-19T14:21:48.130 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.130 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.130 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:48.130 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.543130+0000", 2024-08-19T14:21:48.130 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.196710037 2024-08-19T14:21:48.131 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.131 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.131 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:21:48.131 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.543161+0000", 2024-08-19T14:21:48.131 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.1306999999999997e-05 2024-08-19T14:21:48.131 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.131 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.131 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:21:48.131 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.543196+0000", 2024-08-19T14:21:48.131 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.5033999999999997e-05 2024-08-19T14:21:48.131 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:21:48.131 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:21:48.131 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:21:48.131 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.131 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.132 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24580.0:10322 2.5 2:ad7f7050:::benchmark_data_smithi045_46415_object10321:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-08-19T14:21:48.132 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:12:52.345086+0000", 2024-08-19T14:21:48.132 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 535.75737836999997, 2024-08-19T14:21:48.132 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.19865560199999999, 2024-08-19T14:21:48.132 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:21:48.132 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:21:48.132 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:21:48.132 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:21:48.132 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24580", 2024-08-19T14:21:48.132 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:21:48.132 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 10322 2024-08-19T14:21:48.132 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.132 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:21:48.132 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.133 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:21:48.133 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.345086+0000", 2024-08-19T14:21:48.133 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:21:48.133 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.133 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.133 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:21:48.133 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.345086+0000", 2024-08-19T14:21:48.133 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:21:48.133 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.133 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.133 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:21:48.133 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.345087+0000", 2024-08-19T14:21:48.133 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.8299999999999996e-07 2024-08-19T14:21:48.133 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.134 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.134 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:21:48.134 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.345106+0000", 2024-08-19T14:21:48.134 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.91e-05 2024-08-19T14:21:48.134 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.134 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.134 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:21:48.134 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.345114+0000", 2024-08-19T14:21:48.134 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.6200000000000005e-06 2024-08-19T14:21:48.134 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.134 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.134 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:21:48.134 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.345122+0000", 2024-08-19T14:21:48.135 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.8310000000000001e-06 2024-08-19T14:21:48.135 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.135 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.135 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:21:48.135 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.345281+0000", 2024-08-19T14:21:48.135 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000158386 2024-08-19T14:21:48.135 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.135 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.135 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:21:48.135 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.345396+0000", 2024-08-19T14:21:48.135 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000115299 2024-08-19T14:21:48.135 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.135 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.135 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,4", 2024-08-19T14:21:48.136 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.345628+0000", 2024-08-19T14:21:48.136 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00023218499999999999 2024-08-19T14:21:48.136 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.136 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.136 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:21:48.136 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.347682+0000", 2024-08-19T14:21:48.136 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0020542189999999999 2024-08-19T14:21:48.136 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.136 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.136 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:48.136 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.348039+0000", 2024-08-19T14:21:48.136 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00035721800000000002 2024-08-19T14:21:48.136 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.136 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.137 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:48.137 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.543654+0000", 2024-08-19T14:21:48.137 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.19561498799999999 2024-08-19T14:21:48.137 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.137 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.137 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:21:48.137 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.543708+0000", 2024-08-19T14:21:48.137 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.3152999999999997e-05 2024-08-19T14:21:48.137 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.137 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.137 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:21:48.137 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:52.543741+0000", 2024-08-19T14:21:48.137 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.392e-05 2024-08-19T14:21:48.137 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:21:48.138 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:21:48.138 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:21:48.138 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.138 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.138 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24580.0:88314 2.f 2:fc0d2546:::benchmark_data_smithi045_46415_object88313:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-08-19T14:21:48.138 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:42.807395+0000", 2024-08-19T14:21:48.138 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 485.29506918300001, 2024-08-19T14:21:48.138 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.21007711900000001, 2024-08-19T14:21:48.138 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:21:48.138 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:21:48.138 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:21:48.138 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:21:48.138 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24580", 2024-08-19T14:21:48.139 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:21:48.139 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 88314 2024-08-19T14:21:48.139 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.139 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:21:48.139 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.139 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:21:48.139 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.807395+0000", 2024-08-19T14:21:48.139 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:21:48.139 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.139 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.139 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:21:48.139 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.807395+0000", 2024-08-19T14:21:48.139 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:21:48.140 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.140 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.140 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:21:48.140 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.807397+0000", 2024-08-19T14:21:48.140 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1239999999999999e-06 2024-08-19T14:21:48.140 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.140 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.140 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:21:48.140 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.807424+0000", 2024-08-19T14:21:48.140 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.6429000000000001e-05 2024-08-19T14:21:48.141 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.141 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.141 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:21:48.141 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.807445+0000", 2024-08-19T14:21:48.141 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1078e-05 2024-08-19T14:21:48.141 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.141 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.141 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:21:48.141 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.807455+0000", 2024-08-19T14:21:48.141 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0351e-05 2024-08-19T14:21:48.141 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.141 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.141 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:21:48.141 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.807524+0000", 2024-08-19T14:21:48.142 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.9122999999999995e-05 2024-08-19T14:21:48.142 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.142 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.142 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:21:48.142 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.807862+0000", 2024-08-19T14:21:48.142 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000338222 2024-08-19T14:21:48.142 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.142 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.142 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 4,9", 2024-08-19T14:21:48.142 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.808028+0000", 2024-08-19T14:21:48.142 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00016509700000000001 2024-08-19T14:21:48.142 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.142 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.143 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:21:48.143 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.811333+0000", 2024-08-19T14:21:48.143 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0033057300000000002 2024-08-19T14:21:48.143 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.143 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.143 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:48.143 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.814174+0000", 2024-08-19T14:21:48.143 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0028408959999999999 2024-08-19T14:21:48.143 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.143 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.143 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:48.143 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:43.017401+0000", 2024-08-19T14:21:48.143 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.203226459 2024-08-19T14:21:48.143 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.143 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.144 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:21:48.144 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:43.017443+0000", 2024-08-19T14:21:48.144 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.2438999999999998e-05 2024-08-19T14:21:48.144 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.144 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.144 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:21:48.144 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:43.017472+0000", 2024-08-19T14:21:48.144 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.9170999999999999e-05 2024-08-19T14:21:48.144 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:21:48.144 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:21:48.144 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:21:48.144 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.144 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.144 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24580.0:88316 2.5 2:ade27d21:::benchmark_data_smithi045_46415_object88315:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-08-19T14:21:48.145 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:42.807950+0000", 2024-08-19T14:21:48.145 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 485.29451423299997, 2024-08-19T14:21:48.145 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.21254900700000001, 2024-08-19T14:21:48.145 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:21:48.145 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:21:48.145 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:21:48.145 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:21:48.145 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24580", 2024-08-19T14:21:48.145 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:21:48.145 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 88316 2024-08-19T14:21:48.145 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.145 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:21:48.145 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.145 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:21:48.146 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.807950+0000", 2024-08-19T14:21:48.146 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:21:48.146 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.146 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.146 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:21:48.146 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.807950+0000", 2024-08-19T14:21:48.146 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:21:48.146 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.146 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.146 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:21:48.146 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.807951+0000", 2024-08-19T14:21:48.146 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.046e-06 2024-08-19T14:21:48.146 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.147 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.147 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:21:48.147 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.807985+0000", 2024-08-19T14:21:48.147 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.3565999999999997e-05 2024-08-19T14:21:48.147 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.147 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.147 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:21:48.147 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.807997+0000", 2024-08-19T14:21:48.147 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2690999999999999e-05 2024-08-19T14:21:48.147 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.147 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.147 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:21:48.147 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.808005+0000", 2024-08-19T14:21:48.147 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.8089999999999993e-06 2024-08-19T14:21:48.148 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.148 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.148 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:21:48.148 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.808063+0000", 2024-08-19T14:21:48.148 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.8108e-05 2024-08-19T14:21:48.148 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.148 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.148 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:21:48.148 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.808366+0000", 2024-08-19T14:21:48.148 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00030297899999999999 2024-08-19T14:21:48.148 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.148 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.148 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,4", 2024-08-19T14:21:48.149 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.808492+0000", 2024-08-19T14:21:48.149 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000125556 2024-08-19T14:21:48.149 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.149 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.149 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:21:48.149 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.811416+0000", 2024-08-19T14:21:48.149 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0029237540000000002 2024-08-19T14:21:48.149 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.149 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.149 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:48.149 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.818511+0000", 2024-08-19T14:21:48.149 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0070953980000000002 2024-08-19T14:21:48.149 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.149 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.149 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:48.150 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:43.020384+0000", 2024-08-19T14:21:48.150 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.201872945 2024-08-19T14:21:48.150 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.150 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.150 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:21:48.150 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:43.020458+0000", 2024-08-19T14:21:48.150 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.3699e-05 2024-08-19T14:21:48.150 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.150 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.150 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:21:48.150 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:43.020499+0000", 2024-08-19T14:21:48.150 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.1455999999999997e-05 2024-08-19T14:21:48.150 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:21:48.151 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:21:48.151 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:21:48.151 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.151 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.151 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24580.0:88321 2.5 2:a4cf6cb2:::benchmark_data_smithi045_46415_object88320:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-08-19T14:21:48.151 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:42.814619+0000", 2024-08-19T14:21:48.151 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 485.28784518200001, 2024-08-19T14:21:48.151 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20609303900000001, 2024-08-19T14:21:48.151 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:21:48.151 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:21:48.151 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:21:48.151 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:21:48.152 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24580", 2024-08-19T14:21:48.152 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:21:48.152 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 88321 2024-08-19T14:21:48.152 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.152 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:21:48.152 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.152 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:21:48.152 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.814619+0000", 2024-08-19T14:21:48.152 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:21:48.152 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.153 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.153 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:21:48.153 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.814619+0000", 2024-08-19T14:21:48.153 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:21:48.153 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.153 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.153 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:21:48.153 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.814621+0000", 2024-08-19T14:21:48.153 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.514e-06 2024-08-19T14:21:48.153 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.153 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.153 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:21:48.153 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.814641+0000", 2024-08-19T14:21:48.153 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0852e-05 2024-08-19T14:21:48.154 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.154 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.154 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:21:48.154 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.814653+0000", 2024-08-19T14:21:48.154 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1293e-05 2024-08-19T14:21:48.154 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.154 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.154 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:21:48.154 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.814660+0000", 2024-08-19T14:21:48.154 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.905e-06 2024-08-19T14:21:48.154 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.154 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.154 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:21:48.155 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.814711+0000", 2024-08-19T14:21:48.155 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1657999999999999e-05 2024-08-19T14:21:48.155 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.155 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.155 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:21:48.155 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.814825+0000", 2024-08-19T14:21:48.155 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000113597 2024-08-19T14:21:48.155 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.155 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.155 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,4", 2024-08-19T14:21:48.155 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.814949+0000", 2024-08-19T14:21:48.155 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012450800000000001 2024-08-19T14:21:48.155 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.155 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.156 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:21:48.156 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.817037+0000", 2024-08-19T14:21:48.156 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0020878089999999999 2024-08-19T14:21:48.156 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.156 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.156 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:48.156 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.819696+0000", 2024-08-19T14:21:48.156 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.002659004 2024-08-19T14:21:48.156 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.156 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.156 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:48.156 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:43.020593+0000", 2024-08-19T14:21:48.156 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20089647899999999 2024-08-19T14:21:48.156 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.157 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.157 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:21:48.157 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:43.020673+0000", 2024-08-19T14:21:48.157 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.9881000000000003e-05 2024-08-19T14:21:48.157 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.157 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.157 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:21:48.157 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:43.020712+0000", 2024-08-19T14:21:48.157 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.9539000000000002e-05 2024-08-19T14:21:48.157 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:21:48.157 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:21:48.157 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:21:48.157 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.157 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.158 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24580.0:88328 2.5 2:a21c7eb0:::benchmark_data_smithi045_46415_object88327:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-08-19T14:21:48.158 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:42.820640+0000", 2024-08-19T14:21:48.158 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 485.28182395699997, 2024-08-19T14:21:48.158 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20083906900000001, 2024-08-19T14:21:48.158 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:21:48.158 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:21:48.158 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:21:48.158 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:21:48.158 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24580", 2024-08-19T14:21:48.158 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:21:48.158 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 88328 2024-08-19T14:21:48.158 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.158 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:21:48.159 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.159 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:21:48.159 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.820640+0000", 2024-08-19T14:21:48.159 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:21:48.159 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.159 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.159 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:21:48.159 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.820640+0000", 2024-08-19T14:21:48.159 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:21:48.159 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.159 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.159 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:21:48.159 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.820641+0000", 2024-08-19T14:21:48.159 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.23e-07 2024-08-19T14:21:48.160 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.160 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.160 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:21:48.160 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.820660+0000", 2024-08-19T14:21:48.160 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8788e-05 2024-08-19T14:21:48.160 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.160 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.160 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:21:48.160 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.820670+0000", 2024-08-19T14:21:48.160 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.9089999999999995e-06 2024-08-19T14:21:48.161 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.161 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.161 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:21:48.161 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.820675+0000", 2024-08-19T14:21:48.161 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.8890000000000001e-06 2024-08-19T14:21:48.161 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.161 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.161 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:21:48.161 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.820718+0000", 2024-08-19T14:21:48.161 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.2997999999999997e-05 2024-08-19T14:21:48.161 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.161 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.161 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:21:48.162 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.820811+0000", 2024-08-19T14:21:48.162 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.3219000000000003e-05 2024-08-19T14:21:48.162 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.162 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.164 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,4", 2024-08-19T14:21:48.164 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.820907+0000", 2024-08-19T14:21:48.164 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.6267000000000005e-05 2024-08-19T14:21:48.164 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.165 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.165 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:21:48.165 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.823083+0000", 2024-08-19T14:21:48.165 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0021754299999999999 2024-08-19T14:21:48.165 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.165 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.165 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:48.165 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:42.824768+0000", 2024-08-19T14:21:48.165 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001685048 2024-08-19T14:21:48.165 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.165 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.165 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:48.165 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:43.021388+0000", 2024-08-19T14:21:48.166 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.196620295 2024-08-19T14:21:48.166 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.166 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.166 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:21:48.166 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:43.021436+0000", 2024-08-19T14:21:48.166 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.8195000000000002e-05 2024-08-19T14:21:48.166 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.166 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.166 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:21:48.166 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:43.021479+0000", 2024-08-19T14:21:48.166 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.3307999999999999e-05 2024-08-19T14:21:48.166 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:21:48.166 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:21:48.167 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:21:48.167 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.167 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.167 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24580.0:209201 2.f 2:fb8d21d4:::benchmark_data_smithi045_46415_object85920:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194)", 2024-08-19T14:21:48.167 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:16:02.057866+0000", 2024-08-19T14:21:48.167 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 346.04459863099999, 2024-08-19T14:21:48.167 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.21458635300000001, 2024-08-19T14:21:48.167 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:21:48.167 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:21:48.167 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:21:48.167 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:21:48.167 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24580", 2024-08-19T14:21:48.167 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:21:48.168 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 209201 2024-08-19T14:21:48.168 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.168 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:21:48.168 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.168 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:21:48.168 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:16:02.057866+0000", 2024-08-19T14:21:48.168 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:21:48.168 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.168 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.168 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:21:48.168 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:16:02.057866+0000", 2024-08-19T14:21:48.168 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:21:48.169 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.169 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.169 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:21:48.169 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:16:02.057867+0000", 2024-08-19T14:21:48.169 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.540000000000001e-07 2024-08-19T14:21:48.169 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.169 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.169 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:21:48.169 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:16:02.057871+0000", 2024-08-19T14:21:48.169 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.1010000000000002e-06 2024-08-19T14:21:48.169 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.169 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.169 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:21:48.170 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:16:02.057887+0000", 2024-08-19T14:21:48.170 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6809000000000001e-05 2024-08-19T14:21:48.170 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.170 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.170 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:21:48.170 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:16:02.057893+0000", 2024-08-19T14:21:48.170 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7880000000000003e-06 2024-08-19T14:21:48.170 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.170 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.170 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:21:48.170 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:16:02.057942+0000", 2024-08-19T14:21:48.170 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.8983999999999997e-05 2024-08-19T14:21:48.171 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.171 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.171 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:21:48.171 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:16:02.058135+0000", 2024-08-19T14:21:48.171 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000192692 2024-08-19T14:21:48.171 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.171 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.171 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 4,9", 2024-08-19T14:21:48.171 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:16:02.058253+0000", 2024-08-19T14:21:48.171 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000118071 2024-08-19T14:21:48.171 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.171 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.171 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:48.171 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:16:02.062846+0000", 2024-08-19T14:21:48.172 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.004593209 2024-08-19T14:21:48.172 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.172 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.172 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:21:48.172 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:16:02.062913+0000", 2024-08-19T14:21:48.172 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.6808000000000001e-05 2024-08-19T14:21:48.172 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.172 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.172 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:48.172 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:16:02.272371+0000", 2024-08-19T14:21:48.172 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.209458071 2024-08-19T14:21:48.172 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.172 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.172 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:21:48.173 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:16:02.272407+0000", 2024-08-19T14:21:48.173 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.6031000000000001e-05 2024-08-19T14:21:48.173 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.173 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.173 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:21:48.173 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:16:02.272452+0000", 2024-08-19T14:21:48.173 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.4835e-05 2024-08-19T14:21:48.173 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:21:48.173 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:21:48.173 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:21:48.173 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.173 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.173 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24580.0:209204 2.f 2:f5b7b692:::benchmark_data_smithi045_46415_object85923:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194)", 2024-08-19T14:21:48.173 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:16:02.063760+0000", 2024-08-19T14:21:48.174 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 346.03870472599999, 2024-08-19T14:21:48.174 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.21463046399999999, 2024-08-19T14:21:48.174 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:21:48.174 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:21:48.174 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:21:48.174 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:21:48.174 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24580", 2024-08-19T14:21:48.174 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:21:48.174 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 209204 2024-08-19T14:21:48.174 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.174 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:21:48.174 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.174 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:21:48.175 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:16:02.063760+0000", 2024-08-19T14:21:48.175 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:21:48.175 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.175 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.175 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:21:48.175 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:16:02.063760+0000", 2024-08-19T14:21:48.175 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:21:48.175 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.175 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.175 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:21:48.175 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:16:02.063761+0000", 2024-08-19T14:21:48.175 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2419999999999999e-06 2024-08-19T14:21:48.175 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.175 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.176 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:21:48.176 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:16:02.063765+0000", 2024-08-19T14:21:48.176 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.0629999999999999e-06 2024-08-19T14:21:48.176 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.176 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.176 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:21:48.176 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:16:02.063775+0000", 2024-08-19T14:21:48.176 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.8239999999999995e-06 2024-08-19T14:21:48.176 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.176 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.176 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:21:48.176 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:16:02.063780+0000", 2024-08-19T14:21:48.176 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.31e-06 2024-08-19T14:21:48.176 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.177 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.177 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:21:48.177 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:16:02.063822+0000", 2024-08-19T14:21:48.177 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.2009999999999999e-05 2024-08-19T14:21:48.177 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.177 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.177 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:21:48.177 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:16:02.063940+0000", 2024-08-19T14:21:48.177 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00011828999999999999 2024-08-19T14:21:48.177 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.177 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.177 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 4,9", 2024-08-19T14:21:48.177 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:16:02.064018+0000", 2024-08-19T14:21:48.177 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.7959000000000004e-05 2024-08-19T14:21:48.177 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.178 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.178 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:48.178 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:16:02.067663+0000", 2024-08-19T14:21:48.178 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0036444989999999998 2024-08-19T14:21:48.178 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.178 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.178 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:21:48.178 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:16:02.067679+0000", 2024-08-19T14:21:48.178 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6184999999999998e-05 2024-08-19T14:21:48.178 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.178 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.178 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:48.178 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:16:02.278283+0000", 2024-08-19T14:21:48.179 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.210604333 2024-08-19T14:21:48.179 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.179 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.179 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:21:48.179 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:16:02.278327+0000", 2024-08-19T14:21:48.179 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.3619999999999999e-05 2024-08-19T14:21:48.179 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.179 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.179 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:21:48.179 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:16:02.278390+0000", 2024-08-19T14:21:48.179 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.3128999999999998e-05 2024-08-19T14:21:48.179 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:21:48.179 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:21:48.180 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:21:48.180 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.180 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.180 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25057.0:64631 3.1a 3.4f01fb1a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e246)", 2024-08-19T14:21:48.180 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:17:36.180454+0000", 2024-08-19T14:21:48.180 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 251.92201044999999, 2024-08-19T14:21:48.180 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.78083698899999998, 2024-08-19T14:21:48.180 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:21:48.180 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:21:48.180 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-08-19T14:21:48.180 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:21:48.180 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25057", 2024-08-19T14:21:48.180 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:21:48.180 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 64631 2024-08-19T14:21:48.181 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.181 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:21:48.181 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.181 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:21:48.181 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.180454+0000", 2024-08-19T14:21:48.181 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:21:48.181 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.181 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.181 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:21:48.181 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.180454+0000", 2024-08-19T14:21:48.181 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:21:48.181 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.181 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.181 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:21:48.182 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.180456+0000", 2024-08-19T14:21:48.182 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.5220000000000002e-06 2024-08-19T14:21:48.182 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.182 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.182 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:21:48.182 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.180483+0000", 2024-08-19T14:21:48.182 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.6417000000000001e-05 2024-08-19T14:21:48.182 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.182 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.182 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:21:48.182 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.180499+0000", 2024-08-19T14:21:48.182 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5823000000000001e-05 2024-08-19T14:21:48.182 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.182 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.183 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:21:48.183 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.180510+0000", 2024-08-19T14:21:48.183 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1737e-05 2024-08-19T14:21:48.183 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.183 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.183 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:21:48.183 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.180596+0000", 2024-08-19T14:21:48.183 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.6001000000000006e-05 2024-08-19T14:21:48.183 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.183 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.183 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for peered", 2024-08-19T14:21:48.183 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.180606+0000", 2024-08-19T14:21:48.183 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.3470000000000005e-06 2024-08-19T14:21:48.184 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.184 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.184 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:21:48.184 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.961260+0000", 2024-08-19T14:21:48.184 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.78065392099999997 2024-08-19T14:21:48.184 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.184 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.184 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:21:48.184 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.961291+0000", 2024-08-19T14:21:48.184 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.1220999999999998e-05 2024-08-19T14:21:48.184 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:21:48.184 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:21:48.184 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:21:48.184 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.185 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.185 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25057.0:64651 3.1a 3.5fb8049a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e246)", 2024-08-19T14:21:48.185 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:17:36.235642+0000", 2024-08-19T14:21:48.185 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 251.86682239300001, 2024-08-19T14:21:48.185 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.72570858100000002, 2024-08-19T14:21:48.185 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:21:48.185 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:21:48.185 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-08-19T14:21:48.185 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:21:48.185 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25057", 2024-08-19T14:21:48.185 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:21:48.185 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 64651 2024-08-19T14:21:48.185 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.185 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:21:48.186 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.186 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:21:48.186 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.235642+0000", 2024-08-19T14:21:48.186 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:21:48.186 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.186 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.186 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:21:48.186 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.235642+0000", 2024-08-19T14:21:48.186 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:21:48.186 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.186 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.186 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:21:48.186 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.235644+0000", 2024-08-19T14:21:48.186 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7829999999999999e-06 2024-08-19T14:21:48.187 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.187 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.187 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:21:48.187 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.235677+0000", 2024-08-19T14:21:48.187 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.3333000000000001e-05 2024-08-19T14:21:48.187 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.187 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.187 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:21:48.187 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.235694+0000", 2024-08-19T14:21:48.187 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6626999999999999e-05 2024-08-19T14:21:48.187 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.187 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.187 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:21:48.188 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.235701+0000", 2024-08-19T14:21:48.188 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.7000000000000008e-06 2024-08-19T14:21:48.188 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.188 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.188 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:21:48.188 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.235758+0000", 2024-08-19T14:21:48.188 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7166999999999997e-05 2024-08-19T14:21:48.188 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.188 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.188 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for peered", 2024-08-19T14:21:48.188 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.235766+0000", 2024-08-19T14:21:48.188 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4059999999999999e-06 2024-08-19T14:21:48.188 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.188 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.189 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:21:48.189 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.961337+0000", 2024-08-19T14:21:48.189 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.72557158099999997 2024-08-19T14:21:48.189 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.189 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.189 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:21:48.189 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.961350+0000", 2024-08-19T14:21:48.189 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2984e-05 2024-08-19T14:21:48.189 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:21:48.189 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:21:48.189 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:21:48.189 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.189 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.189 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25057.0:64656 3.1a 3.4e375ba (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e246)", 2024-08-19T14:21:48.190 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:17:36.257659+0000", 2024-08-19T14:21:48.190 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 251.84480492099999, 2024-08-19T14:21:48.190 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.70372575500000001, 2024-08-19T14:21:48.190 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:21:48.190 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:21:48.190 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-08-19T14:21:48.190 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:21:48.190 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25057", 2024-08-19T14:21:48.190 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:21:48.190 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 64656 2024-08-19T14:21:48.190 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.190 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:21:48.190 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.191 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:21:48.191 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.257659+0000", 2024-08-19T14:21:48.191 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:21:48.191 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.191 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.191 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:21:48.191 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.257659+0000", 2024-08-19T14:21:48.191 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:21:48.191 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.191 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.191 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:21:48.191 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.257661+0000", 2024-08-19T14:21:48.191 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.821e-06 2024-08-19T14:21:48.191 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.192 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.192 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:21:48.192 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.257681+0000", 2024-08-19T14:21:48.192 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0307999999999999e-05 2024-08-19T14:21:48.192 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.192 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.192 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:21:48.192 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.257693+0000", 2024-08-19T14:21:48.192 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1447000000000001e-05 2024-08-19T14:21:48.192 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.192 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.192 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:21:48.192 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.257699+0000", 2024-08-19T14:21:48.192 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7389999999999996e-06 2024-08-19T14:21:48.193 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.193 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.193 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:21:48.193 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.257751+0000", 2024-08-19T14:21:48.193 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2281999999999999e-05 2024-08-19T14:21:48.193 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.193 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.193 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for peered", 2024-08-19T14:21:48.193 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.257759+0000", 2024-08-19T14:21:48.193 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.2390000000000002e-06 2024-08-19T14:21:48.193 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.193 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.193 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:21:48.194 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.961378+0000", 2024-08-19T14:21:48.194 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.70361908699999998 2024-08-19T14:21:48.194 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.194 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.194 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:21:48.194 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.961385+0000", 2024-08-19T14:21:48.194 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.832e-06 2024-08-19T14:21:48.194 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:21:48.194 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:21:48.194 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:21:48.194 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.194 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.194 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25057.0:130084 3.9 3:96bca659:::benchmark_data_smithi045_59973_object130083:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e281)", 2024-08-19T14:21:48.195 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:18:31.614605+0000", 2024-08-19T14:21:48.195 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 196.48785959599999, 2024-08-19T14:21:48.195 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.206588472, 2024-08-19T14:21:48.195 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:21:48.195 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:21:48.195 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:21:48.195 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:21:48.195 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25057", 2024-08-19T14:21:48.195 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:21:48.195 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 130084 2024-08-19T14:21:48.195 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.195 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:21:48.195 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.195 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:21:48.196 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:18:31.614605+0000", 2024-08-19T14:21:48.196 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:21:48.196 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.196 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.196 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:21:48.196 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:18:31.614605+0000", 2024-08-19T14:21:48.196 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:21:48.196 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.196 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.196 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:21:48.196 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:18:31.614606+0000", 2024-08-19T14:21:48.196 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3960000000000001e-06 2024-08-19T14:21:48.196 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.197 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.197 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:21:48.197 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:18:31.614627+0000", 2024-08-19T14:21:48.197 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1313000000000001e-05 2024-08-19T14:21:48.197 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.197 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.197 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:21:48.197 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:18:31.614638+0000", 2024-08-19T14:21:48.197 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0686000000000001e-05 2024-08-19T14:21:48.197 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.197 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.197 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:21:48.197 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:18:31.614643+0000", 2024-08-19T14:21:48.197 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0690000000000001e-06 2024-08-19T14:21:48.198 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.198 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.198 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:21:48.198 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:18:31.614719+0000", 2024-08-19T14:21:48.198 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.6252000000000001e-05 2024-08-19T14:21:48.198 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.198 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.198 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:21:48.198 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:18:31.614843+0000", 2024-08-19T14:21:48.198 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012340699999999999 2024-08-19T14:21:48.198 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.198 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.198 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 7,11", 2024-08-19T14:21:48.199 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:18:31.614982+0000", 2024-08-19T14:21:48.199 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013878600000000001 2024-08-19T14:21:48.199 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.199 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.199 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:21:48.199 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:18:31.617402+0000", 2024-08-19T14:21:48.199 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00242034 2024-08-19T14:21:48.199 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.199 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.199 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:48.199 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:18:31.617596+0000", 2024-08-19T14:21:48.199 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00019408500000000001 2024-08-19T14:21:48.199 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.199 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.200 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:48.200 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:18:31.820984+0000", 2024-08-19T14:21:48.200 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20338821900000001 2024-08-19T14:21:48.200 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.200 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.200 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:21:48.200 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:18:31.821156+0000", 2024-08-19T14:21:48.200 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00017137899999999999 2024-08-19T14:21:48.200 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.200 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.200 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:21:48.200 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:18:31.821193+0000", 2024-08-19T14:21:48.200 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.7540000000000003e-05 2024-08-19T14:21:48.201 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:21:48.201 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:21:48.201 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:21:48.201 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.201 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.201 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25057.0:163335 3.1d 3:bec78ced:::benchmark_data_smithi045_59973_object32156:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e288)", 2024-08-19T14:21:48.201 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:18:47.673097+0000", 2024-08-19T14:21:48.201 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 180.42936687900001, 2024-08-19T14:21:48.201 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.207015213, 2024-08-19T14:21:48.201 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:21:48.201 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:21:48.201 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:21:48.201 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:21:48.201 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25057", 2024-08-19T14:21:48.202 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:21:48.202 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 163335 2024-08-19T14:21:48.202 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.202 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:21:48.202 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.202 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:21:48.202 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:18:47.673097+0000", 2024-08-19T14:21:48.202 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:21:48.202 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.202 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.202 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:21:48.202 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:18:47.673097+0000", 2024-08-19T14:21:48.202 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:21:48.202 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.203 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.203 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:21:48.203 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:18:47.673098+0000", 2024-08-19T14:21:48.203 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.8299999999999995e-07 2024-08-19T14:21:48.203 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.203 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.203 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:21:48.203 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:18:47.673103+0000", 2024-08-19T14:21:48.203 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.2760000000000002e-06 2024-08-19T14:21:48.203 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.203 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.203 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:21:48.203 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:18:47.673115+0000", 2024-08-19T14:21:48.204 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2676e-05 2024-08-19T14:21:48.204 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.204 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.204 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:21:48.204 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:18:47.673122+0000", 2024-08-19T14:21:48.204 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.9990000000000002e-06 2024-08-19T14:21:48.204 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.204 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.204 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:21:48.204 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:18:47.674044+0000", 2024-08-19T14:21:48.204 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00092164999999999999 2024-08-19T14:21:48.204 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.204 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.204 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:21:48.204 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:18:47.674163+0000", 2024-08-19T14:21:48.205 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000119577 2024-08-19T14:21:48.205 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.205 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.205 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 4,5", 2024-08-19T14:21:48.205 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:18:47.674268+0000", 2024-08-19T14:21:48.205 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00010464 2024-08-19T14:21:48.205 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.205 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.205 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:21:48.205 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:18:47.678891+0000", 2024-08-19T14:21:48.205 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0046226419999999997 2024-08-19T14:21:48.205 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.205 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.205 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:48.206 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:18:47.735032+0000", 2024-08-19T14:21:48.206 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.056140931999999998 2024-08-19T14:21:48.206 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.206 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.206 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:48.206 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:18:47.880041+0000", 2024-08-19T14:21:48.206 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.14500938999999999 2024-08-19T14:21:48.206 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.206 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.206 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:21:48.206 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:18:47.880073+0000", 2024-08-19T14:21:48.206 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.1695000000000001e-05 2024-08-19T14:21:48.206 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.207 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.207 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:21:48.207 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:18:47.880113+0000", 2024-08-19T14:21:48.207 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.9852999999999999e-05 2024-08-19T14:21:48.207 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:21:48.207 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:21:48.207 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:21:48.207 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.207 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.207 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25057.0:212414 3.1a 3:58a5364f:::benchmark_data_smithi045_59973_object81235:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e305)", 2024-08-19T14:21:48.207 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:19:29.907348+0000", 2024-08-19T14:21:48.207 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 138.195115872, 2024-08-19T14:21:48.207 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.19817712800000001, 2024-08-19T14:21:48.207 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:21:48.208 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:21:48.208 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:21:48.209 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:21:48.210 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25057", 2024-08-19T14:21:48.210 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:21:48.210 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 212414 2024-08-19T14:21:48.210 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.210 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:21:48.210 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.210 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:21:48.210 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:29.907348+0000", 2024-08-19T14:21:48.210 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:21:48.210 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.210 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.210 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:21:48.211 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:29.907348+0000", 2024-08-19T14:21:48.211 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:21:48.211 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.211 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.211 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:21:48.211 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:29.907349+0000", 2024-08-19T14:21:48.211 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.1299999999999998e-07 2024-08-19T14:21:48.211 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.211 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.211 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:21:48.211 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:29.907354+0000", 2024-08-19T14:21:48.211 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.78e-06 2024-08-19T14:21:48.211 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.211 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.212 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:21:48.212 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:29.907366+0000", 2024-08-19T14:21:48.212 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1430000000000001e-05 2024-08-19T14:21:48.212 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.212 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.212 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:21:48.212 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:29.907371+0000", 2024-08-19T14:21:48.212 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.3000000000000001e-06 2024-08-19T14:21:48.212 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.212 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.212 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:21:48.212 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:29.907460+0000", 2024-08-19T14:21:48.212 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.9044999999999999e-05 2024-08-19T14:21:48.212 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.213 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.213 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:21:48.213 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:29.907531+0000", 2024-08-19T14:21:48.213 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.1620999999999997e-05 2024-08-19T14:21:48.213 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.213 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.213 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,9", 2024-08-19T14:21:48.213 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:29.907593+0000", 2024-08-19T14:21:48.213 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.1859999999999994e-05 2024-08-19T14:21:48.213 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.213 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.213 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:48.213 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:29.911966+0000", 2024-08-19T14:21:48.213 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0043727549999999999 2024-08-19T14:21:48.214 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.214 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.214 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:21:48.214 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:29.912043+0000", 2024-08-19T14:21:48.214 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.6963000000000002e-05 2024-08-19T14:21:48.214 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.214 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.214 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:48.214 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:30.105414+0000", 2024-08-19T14:21:48.214 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.193371022 2024-08-19T14:21:48.214 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.214 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.214 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:21:48.215 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:30.105474+0000", 2024-08-19T14:21:48.215 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.0081999999999998e-05 2024-08-19T14:21:48.215 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.215 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.215 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:21:48.215 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:30.105526+0000", 2024-08-19T14:21:48.215 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1357000000000003e-05 2024-08-19T14:21:48.215 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:21:48.215 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:21:48.215 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:21:48.215 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.215 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.215 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25057.0:212429 3.1a 3:5cc1fd18:::benchmark_data_smithi045_59973_object81250:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e305)", 2024-08-19T14:21:48.216 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:19:29.920895+0000", 2024-08-19T14:21:48.216 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 138.18156947599999, 2024-08-19T14:21:48.216 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.201169661, 2024-08-19T14:21:48.216 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:21:48.216 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:21:48.216 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:21:48.216 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:21:48.216 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25057", 2024-08-19T14:21:48.216 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:21:48.216 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 212429 2024-08-19T14:21:48.216 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.216 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:21:48.216 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.216 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:21:48.217 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:29.920895+0000", 2024-08-19T14:21:48.217 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:21:48.217 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.217 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.217 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:21:48.217 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:29.920895+0000", 2024-08-19T14:21:48.217 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:21:48.217 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.217 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.217 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:21:48.217 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:29.920896+0000", 2024-08-19T14:21:48.217 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.561e-06 2024-08-19T14:21:48.217 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.218 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.218 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:21:48.218 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:29.920903+0000", 2024-08-19T14:21:48.218 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.7329999999999997e-06 2024-08-19T14:21:48.218 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.218 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.218 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:21:48.218 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:29.920916+0000", 2024-08-19T14:21:48.218 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3186999999999999e-05 2024-08-19T14:21:48.218 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.218 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.218 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:21:48.218 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:29.920924+0000", 2024-08-19T14:21:48.219 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.0330000000000008e-06 2024-08-19T14:21:48.219 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.219 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.219 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:21:48.219 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:29.920998+0000", 2024-08-19T14:21:48.219 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.3480999999999997e-05 2024-08-19T14:21:48.219 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.219 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.219 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:21:48.219 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:29.921084+0000", 2024-08-19T14:21:48.219 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.6697e-05 2024-08-19T14:21:48.219 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.219 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.219 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,9", 2024-08-19T14:21:48.220 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:29.921167+0000", 2024-08-19T14:21:48.220 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.2146999999999998e-05 2024-08-19T14:21:48.220 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.220 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.220 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:21:48.220 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:29.928838+0000", 2024-08-19T14:21:48.220 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0076718439999999997 2024-08-19T14:21:48.220 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.220 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.220 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:48.220 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:29.929101+0000", 2024-08-19T14:21:48.220 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00026302200000000002 2024-08-19T14:21:48.220 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.220 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.221 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:48.221 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:30.122006+0000", 2024-08-19T14:21:48.221 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.19290453199999999 2024-08-19T14:21:48.221 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.221 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.221 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:21:48.221 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:30.122039+0000", 2024-08-19T14:21:48.221 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.2509000000000003e-05 2024-08-19T14:21:48.221 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.221 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.221 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:21:48.221 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:30.122064+0000", 2024-08-19T14:21:48.221 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.5914999999999998e-05 2024-08-19T14:21:48.221 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:21:48.222 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:21:48.222 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:21:48.222 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.222 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.222 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25057.0:216413 3.1a 3:5fc494fd:::benchmark_data_smithi045_59973_object85234:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e307)", 2024-08-19T14:21:48.222 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:19:33.987755+0000", 2024-08-19T14:21:48.222 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 134.114708949, 2024-08-19T14:21:48.222 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20454685, 2024-08-19T14:21:48.222 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:21:48.222 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:21:48.222 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:21:48.222 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:21:48.222 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25057", 2024-08-19T14:21:48.223 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:21:48.223 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 216413 2024-08-19T14:21:48.223 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.223 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:21:48.223 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.223 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:21:48.223 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:33.987755+0000", 2024-08-19T14:21:48.223 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:21:48.223 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.223 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.223 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:21:48.223 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:33.987755+0000", 2024-08-19T14:21:48.223 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:21:48.224 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.224 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.224 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:21:48.224 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:33.987756+0000", 2024-08-19T14:21:48.224 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1200000000000003e-07 2024-08-19T14:21:48.224 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.224 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.224 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:21:48.224 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:33.987759+0000", 2024-08-19T14:21:48.224 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.213e-06 2024-08-19T14:21:48.224 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.224 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.224 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:21:48.224 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:33.987768+0000", 2024-08-19T14:21:48.225 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.5089999999999993e-06 2024-08-19T14:21:48.225 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.225 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.225 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:21:48.225 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:33.987774+0000", 2024-08-19T14:21:48.225 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.8159999999999996e-06 2024-08-19T14:21:48.225 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.225 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.225 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:21:48.225 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:33.988656+0000", 2024-08-19T14:21:48.225 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000881747 2024-08-19T14:21:48.225 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.225 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.226 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:21:48.226 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:33.988760+0000", 2024-08-19T14:21:48.226 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000104345 2024-08-19T14:21:48.226 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.226 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.226 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,9", 2024-08-19T14:21:48.226 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:33.988862+0000", 2024-08-19T14:21:48.226 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00010182400000000001 2024-08-19T14:21:48.226 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.226 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.226 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:21:48.226 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:33.992105+0000", 2024-08-19T14:21:48.226 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0032423069999999998 2024-08-19T14:21:48.226 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.227 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.227 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:48.227 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:33.998218+0000", 2024-08-19T14:21:48.227 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0061136009999999998 2024-08-19T14:21:48.227 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.227 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.227 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:48.227 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:34.191451+0000", 2024-08-19T14:21:48.227 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.193233241 2024-08-19T14:21:48.227 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.227 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.227 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:21:48.227 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:34.191530+0000", 2024-08-19T14:21:48.227 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.8672999999999997e-05 2024-08-19T14:21:48.228 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.228 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.228 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:21:48.228 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:34.192302+0000", 2024-08-19T14:21:48.228 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00077206200000000001 2024-08-19T14:21:48.228 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:21:48.228 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:21:48.228 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:21:48.228 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.228 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.228 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25057.0:216426 3.1a 3:582bcfa1:::benchmark_data_smithi045_59973_object85247:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e307)", 2024-08-19T14:21:48.228 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:19:33.994645+0000", 2024-08-19T14:21:48.228 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 134.107818791, 2024-08-19T14:21:48.229 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.19960349199999999, 2024-08-19T14:21:48.229 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:21:48.229 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:21:48.229 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:21:48.229 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:21:48.229 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25057", 2024-08-19T14:21:48.229 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:21:48.229 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 216426 2024-08-19T14:21:48.229 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.229 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:21:48.229 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.229 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:21:48.229 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:33.994645+0000", 2024-08-19T14:21:48.229 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:21:48.230 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.230 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.230 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:21:48.230 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:33.994645+0000", 2024-08-19T14:21:48.230 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:21:48.230 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.230 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.230 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:21:48.230 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:33.994647+0000", 2024-08-19T14:21:48.230 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.221e-06 2024-08-19T14:21:48.230 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.230 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.230 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:21:48.231 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:33.994652+0000", 2024-08-19T14:21:48.231 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.865e-06 2024-08-19T14:21:48.231 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.231 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.231 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:21:48.231 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:33.994667+0000", 2024-08-19T14:21:48.231 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5651999999999999e-05 2024-08-19T14:21:48.231 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.231 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.231 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:21:48.231 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:33.994674+0000", 2024-08-19T14:21:48.231 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.9999999999999999e-06 2024-08-19T14:21:48.231 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.231 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.232 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:21:48.232 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:33.994730+0000", 2024-08-19T14:21:48.232 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.5401e-05 2024-08-19T14:21:48.232 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.232 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.232 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:21:48.232 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:33.994837+0000", 2024-08-19T14:21:48.232 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00010730000000000001 2024-08-19T14:21:48.232 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.232 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.232 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,9", 2024-08-19T14:21:48.232 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:33.994947+0000", 2024-08-19T14:21:48.232 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000109887 2024-08-19T14:21:48.233 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.233 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.233 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:21:48.233 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:33.998233+0000", 2024-08-19T14:21:48.233 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.003285708 2024-08-19T14:21:48.233 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.233 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.233 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:48.233 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:33.999576+0000", 2024-08-19T14:21:48.233 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001343386 2024-08-19T14:21:48.233 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.233 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.233 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:21:48.233 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:34.194092+0000", 2024-08-19T14:21:48.233 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.194516153 2024-08-19T14:21:48.234 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.234 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.234 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:21:48.234 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:34.194199+0000", 2024-08-19T14:21:48.234 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000106914 2024-08-19T14:21:48.234 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:21:48.234 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:21:48.234 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:21:48.234 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:34.194249+0000", 2024-08-19T14:21:48.234 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0005e-05 2024-08-19T14:21:48.234 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:21:48.234 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:21:48.234 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:21:48.235 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:21:48.235 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:21:48.235 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:21:48.235 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:21:48.236 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-08-19T14:21:48.237 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:21:48.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:47 smithi032 ceph-mon[20004]: pgmap v820: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 6.8 GiB data, 24 GiB used, 781 GiB / 805 GiB avail; 66 MiB/s wr, 1.05k op/s; 3181/334935 objects misplaced (0.950%); 2.0 MiB/s, 31 objects/s recovering 2024-08-19T14:21:48.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:47 smithi032 ceph-mon[20004]: osdmap e365: 12 total, 12 up, 9 in 2024-08-19T14:21:48.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:47 smithi032 ceph-mon[20004]: 4.3 scrub starts 2024-08-19T14:21:48.266 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:48.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:48.267 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:21:48.368 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:21:48.408 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:48.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:48.409 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:21:48.510 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:21:48.539 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:48.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:48.540 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:21:48.641 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:21:48.670 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:48.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:48.671 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:21:48.772 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:21:48.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:48.793 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:48.794 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:21:48.842 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is set 2024-08-19T14:21:48.845 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:21:48.856 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:48 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/654515354' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:21:48.856 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:48 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:21:48.894 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:21:48.933 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:48.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:48.934 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:21:48.935 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":822,"stamp":"2024-08-19T14:21:47.902735+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7339115040,"num_objects":111979,"num_object_clones":0,"num_object_copies":335937,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3181,"num_objects_unfound":0,"num_objects_dirty":111979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":111977,"num_write_kb":7166528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":959,"num_bytes_recovered":62849024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":116286,"ondisk_log_size":116286,"up":141,"acting":140,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":142,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":25183708,"kb_used_data":21467076,"kb_used_omap":14,"kb_used_meta":3716529,"kb_avail":818522660,"statfs":{"total":863955320832,"available":838167203840,"internally_reserved":0,"allocated":21982285824,"data_stored":21967143334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14353,"internal_metadata":3805726703},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":474480640,"num_objects":7240,"num_object_clones":0,"num_object_copies":21720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-129,"num_objects_unfound":0,"num_objects_dirty":7240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7240,"num_write_kb":463360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":129,"num_bytes_recovered":8454144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002015"},"pg_stats":[{"pgid":"4.5","version":"364'7096","reported_seq":7164,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-08-19T14:21:45.718462+0000","last_change":"2024-08-19T14:20:29.268214+0000","last_active":"2024-08-19T14:21:45.718462+0000","last_peered":"2024-08-19T14:21:45.718462+0000","last_clean":"2024-08-19T14:21:45.718462+0000","last_became_active":"2024-08-19T14:20:29.267549+0000","last_became_peered":"2024-08-19T14:20:29.267549+0000","last_unstale":"2024-08-19T14:21:45.718462+0000","last_undegraded":"2024-08-19T14:21:45.718462+0000","last_fullsized":"2024-08-19T14:21:45.718462+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":7096,"log_dups_size":0,"ondisk_log_size":7096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:57.640230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":465043456,"num_objects":7096,"num_object_clones":0,"num_object_copies":21288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7096,"num_write_kb":454144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,0],"acting":[11,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"70'139","reported_seq":944,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-08-19T14:21:42.825256+0000","last_change":"2024-08-19T14:19:58.072227+0000","last_active":"2024-08-19T14:21:42.825256+0000","last_peered":"2024-08-19T14:21:42.825256+0000","last_clean":"2024-08-19T14:21:42.825256+0000","last_became_active":"2024-08-19T14:18:13.807133+0000","last_became_peered":"2024-08-19T14:18:13.807133+0000","last_unstale":"2024-08-19T14:21:42.825256+0000","last_undegraded":"2024-08-19T14:21:42.825256+0000","last_fullsized":"2024-08-19T14:21:42.825256+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:19:58.072083+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:19:58.072083+0000","last_clean_scrub_stamp":"2024-08-19T14:19:58.072083+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,9],"acting":[7,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"364'7079","reported_seq":9068,"reported_epoch":364,"state":"active+recovering+undersized+remapped","last_fresh":"2024-08-19T14:21:44.194944+0000","last_change":"2024-08-19T14:21:07.239135+0000","last_active":"2024-08-19T14:21:44.194944+0000","last_peered":"2024-08-19T14:21:44.194944+0000","last_clean":"2024-08-19T14:21:04.784061+0000","last_became_active":"2024-08-19T14:21:07.213050+0000","last_became_peered":"2024-08-19T14:21:07.213050+0000","last_unstale":"2024-08-19T14:21:44.194944+0000","last_undegraded":"2024-08-19T14:21:44.194944+0000","last_fullsized":"2024-08-19T14:21:06.805412+0000","mapping_epoch":354,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":7079,"log_dups_size":0,"ondisk_log_size":7079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":463929344,"num_objects":7079,"num_object_clones":0,"num_object_copies":21237,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3181,"num_objects_unfound":0,"num_objects_dirty":7079,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7079,"num_write_kb":453056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":959,"num_bytes_recovered":62849024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,7],"acting":[1,7],"avail_no_missing":["1","5","7"],"object_location_counts":[{"shards":"1,5,7","objects":3181},{"shards":"1,7","objects":3898}],"blocked_by":[],"up_primary":9,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"70'139","reported_seq":843,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-08-19T14:21:42.821321+0000","last_change":"2024-08-19T14:20:03.714917+0000","last_active":"2024-08-19T14:21:42.821321+0000","last_peered":"2024-08-19T14:21:42.821321+0000","last_clean":"2024-08-19T14:21:42.821321+0000","last_became_active":"2024-08-19T14:18:13.804721+0000","last_became_peered":"2024-08-19T14:18:13.804721+0000","last_unstale":"2024-08-19T14:21:42.821321+0000","last_undegraded":"2024-08-19T14:21:42.821321+0000","last_fullsized":"2024-08-19T14:21:42.821321+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:03.714818+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:03.714818+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"364'7005","reported_seq":7073,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-08-19T14:21:44.174552+0000","last_change":"2024-08-19T14:20:29.268110+0000","last_active":"2024-08-19T14:21:44.174552+0000","last_peered":"2024-08-19T14:21:44.174552+0000","last_clean":"2024-08-19T14:21:44.174552+0000","last_became_active":"2024-08-19T14:20:29.267811+0000","last_became_peered":"2024-08-19T14:20:29.267811+0000","last_unstale":"2024-08-19T14:21:44.174552+0000","last_undegraded":"2024-08-19T14:21:44.174552+0000","last_fullsized":"2024-08-19T14:21:44.174552+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":7005,"log_dups_size":0,"ondisk_log_size":7005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:46.516076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459079680,"num_objects":7005,"num_object_clones":0,"num_object_copies":21015,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7005,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7005,"num_write_kb":448320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"70'139","reported_seq":791,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-08-19T14:21:42.825197+0000","last_change":"2024-08-19T14:20:50.742676+0000","last_active":"2024-08-19T14:21:42.825197+0000","last_peered":"2024-08-19T14:21:42.825197+0000","last_clean":"2024-08-19T14:21:42.825197+0000","last_became_active":"2024-08-19T14:20:50.742354+0000","last_became_peered":"2024-08-19T14:20:50.742354+0000","last_unstale":"2024-08-19T14:21:42.825197+0000","last_undegraded":"2024-08-19T14:21:42.825197+0000","last_fullsized":"2024-08-19T14:21:42.825197+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:19:59.302931+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:19:59.302931+0000","last_clean_scrub_stamp":"2024-08-19T14:19:59.302931+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":114,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"364'6956","reported_seq":7023,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-08-19T14:21:44.147919+0000","last_change":"2024-08-19T14:20:29.267640+0000","last_active":"2024-08-19T14:21:44.147919+0000","last_peered":"2024-08-19T14:21:44.147919+0000","last_clean":"2024-08-19T14:21:44.147919+0000","last_became_active":"2024-08-19T14:20:29.267331+0000","last_became_peered":"2024-08-19T14:20:29.267331+0000","last_unstale":"2024-08-19T14:21:44.147919+0000","last_undegraded":"2024-08-19T14:21:44.147919+0000","last_fullsized":"2024-08-19T14:21:44.147919+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":6956,"log_dups_size":0,"ondisk_log_size":6956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:46.504191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":455868416,"num_objects":6956,"num_object_clones":0,"num_object_copies":20868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6956,"num_write_kb":445184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"70'139","reported_seq":840,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-08-19T14:21:42.812799+0000","last_change":"2024-08-19T14:20:09.507422+0000","last_active":"2024-08-19T14:21:42.812799+0000","last_peered":"2024-08-19T14:21:42.812799+0000","last_clean":"2024-08-19T14:21:42.812799+0000","last_became_active":"2024-08-19T14:18:13.808505+0000","last_became_peered":"2024-08-19T14:18:13.808505+0000","last_unstale":"2024-08-19T14:21:42.812799+0000","last_undegraded":"2024-08-19T14:21:42.812799+0000","last_fullsized":"2024-08-19T14:21:42.812799+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:09.507256+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:09.507256+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,11],"acting":[9,0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"364'6989","reported_seq":7057,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-08-19T14:21:43.398652+0000","last_change":"2024-08-19T14:20:29.268766+0000","last_active":"2024-08-19T14:21:43.398652+0000","last_peered":"2024-08-19T14:21:43.398652+0000","last_clean":"2024-08-19T14:21:43.398652+0000","last_became_active":"2024-08-19T14:20:29.268443+0000","last_became_peered":"2024-08-19T14:20:29.268443+0000","last_unstale":"2024-08-19T14:21:43.398652+0000","last_undegraded":"2024-08-19T14:21:43.398652+0000","last_fullsized":"2024-08-19T14:21:43.398652+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":6989,"log_dups_size":0,"ondisk_log_size":6989,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":458031104,"num_objects":6989,"num_object_clones":0,"num_object_copies":20967,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6989,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6989,"num_write_kb":447296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,11],"acting":[9,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.7","version":"70'139","reported_seq":483,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-08-19T14:21:42.834218+0000","last_change":"2024-08-19T14:20:34.937317+0000","last_active":"2024-08-19T14:21:42.834218+0000","last_peered":"2024-08-19T14:21:42.834218+0000","last_clean":"2024-08-19T14:21:42.834218+0000","last_became_active":"2024-08-19T14:20:28.204384+0000","last_became_peered":"2024-08-19T14:20:28.204384+0000","last_unstale":"2024-08-19T14:21:42.834218+0000","last_undegraded":"2024-08-19T14:21:42.834218+0000","last_fullsized":"2024-08-19T14:21:42.834218+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:34.937192+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:20:34.937192+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"364'7041","reported_seq":7109,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-08-19T14:21:45.804410+0000","last_change":"2024-08-19T14:20:29.267833+0000","last_active":"2024-08-19T14:21:45.804410+0000","last_peered":"2024-08-19T14:21:45.804410+0000","last_clean":"2024-08-19T14:21:45.804410+0000","last_became_active":"2024-08-19T14:20:29.267379+0000","last_became_peered":"2024-08-19T14:20:29.267379+0000","last_unstale":"2024-08-19T14:21:45.804410+0000","last_undegraded":"2024-08-19T14:21:45.804410+0000","last_fullsized":"2024-08-19T14:21:45.804410+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":7041,"log_dups_size":0,"ondisk_log_size":7041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":461438976,"num_objects":7041,"num_object_clones":0,"num_object_copies":21123,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7041,"num_write_kb":450624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,1],"acting":[11,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.6","version":"70'139","reported_seq":843,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-08-19T14:21:42.812958+0000","last_change":"2024-08-19T14:19:55.512078+0000","last_active":"2024-08-19T14:21:42.812958+0000","last_peered":"2024-08-19T14:21:42.812958+0000","last_clean":"2024-08-19T14:21:42.812958+0000","last_became_active":"2024-08-19T14:18:26.874902+0000","last_became_peered":"2024-08-19T14:18:26.874902+0000","last_unstale":"2024-08-19T14:21:42.812958+0000","last_undegraded":"2024-08-19T14:21:42.812958+0000","last_fullsized":"2024-08-19T14:21:42.812958+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:19:55.511974+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:27.848870+0000","last_clean_scrub_stamp":"2024-08-19T14:19:55.511974+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,4],"acting":[9,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"364'6946","reported_seq":7014,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-08-19T14:21:44.444470+0000","last_change":"2024-08-19T14:20:29.267682+0000","last_active":"2024-08-19T14:21:44.444470+0000","last_peered":"2024-08-19T14:21:44.444470+0000","last_clean":"2024-08-19T14:21:44.444470+0000","last_became_active":"2024-08-19T14:20:29.267360+0000","last_became_peered":"2024-08-19T14:20:29.267360+0000","last_unstale":"2024-08-19T14:21:44.444470+0000","last_undegraded":"2024-08-19T14:21:44.444470+0000","last_fullsized":"2024-08-19T14:21:44.444470+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":6946,"log_dups_size":0,"ondisk_log_size":6946,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:51.551392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":455213056,"num_objects":6946,"num_object_clones":0,"num_object_copies":20838,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6946,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6946,"num_write_kb":444544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"70'139","reported_seq":657,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-08-19T14:21:42.814226+0000","last_change":"2024-08-19T14:20:12.731993+0000","last_active":"2024-08-19T14:21:42.814226+0000","last_peered":"2024-08-19T14:21:42.814226+0000","last_clean":"2024-08-19T14:21:42.814226+0000","last_became_active":"2024-08-19T14:18:26.881435+0000","last_became_peered":"2024-08-19T14:18:26.881435+0000","last_unstale":"2024-08-19T14:21:42.814226+0000","last_undegraded":"2024-08-19T14:21:42.814226+0000","last_fullsized":"2024-08-19T14:21:42.814226+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:12.731850+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:12.731850+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,4],"acting":[11,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.1","version":"364'7150","reported_seq":7218,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-08-19T14:21:45.815173+0000","last_change":"2024-08-19T14:20:29.268808+0000","last_active":"2024-08-19T14:21:45.815173+0000","last_peered":"2024-08-19T14:21:45.815173+0000","last_clean":"2024-08-19T14:21:45.815173+0000","last_became_active":"2024-08-19T14:20:29.268465+0000","last_became_peered":"2024-08-19T14:20:29.268465+0000","last_unstale":"2024-08-19T14:21:45.815173+0000","last_undegraded":"2024-08-19T14:21:45.815173+0000","last_fullsized":"2024-08-19T14:21:45.815173+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":7150,"log_dups_size":0,"ondisk_log_size":7150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:57.700841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":468582400,"num_objects":7150,"num_object_clones":0,"num_object_copies":21450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7150,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7150,"num_write_kb":457600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.4","version":"70'139","reported_seq":890,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-08-19T14:21:42.814136+0000","last_change":"2024-08-19T14:21:39.314063+0000","last_active":"2024-08-19T14:21:42.814136+0000","last_peered":"2024-08-19T14:21:42.814136+0000","last_clean":"2024-08-19T14:21:42.814136+0000","last_became_active":"2024-08-19T14:18:13.806013+0000","last_became_peered":"2024-08-19T14:18:13.806013+0000","last_unstale":"2024-08-19T14:21:42.814136+0000","last_undegraded":"2024-08-19T14:21:42.814136+0000","last_fullsized":"2024-08-19T14:21:42.814136+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:39.313941+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:21:39.313941+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:22:45.184594+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.f","version":"364'6966","reported_seq":7034,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-08-19T14:21:44.196712+0000","last_change":"2024-08-19T14:20:29.212298+0000","last_active":"2024-08-19T14:21:44.196712+0000","last_peered":"2024-08-19T14:21:44.196712+0000","last_clean":"2024-08-19T14:21:44.196712+0000","last_became_active":"2024-08-19T14:20:29.212003+0000","last_became_peered":"2024-08-19T14:20:29.212003+0000","last_unstale":"2024-08-19T14:21:44.196712+0000","last_undegraded":"2024-08-19T14:21:44.196712+0000","last_fullsized":"2024-08-19T14:21:44.196712+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":6966,"log_dups_size":0,"ondisk_log_size":6966,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:52.621283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":456523776,"num_objects":6966,"num_object_clones":0,"num_object_copies":20898,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6966,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6966,"num_write_kb":445824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,11],"acting":[1,3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"70'139","reported_seq":851,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-08-19T14:21:42.812852+0000","last_change":"2024-08-19T14:20:12.570453+0000","last_active":"2024-08-19T14:21:42.812852+0000","last_peered":"2024-08-19T14:21:42.812852+0000","last_clean":"2024-08-19T14:21:42.812852+0000","last_became_active":"2024-08-19T14:18:26.875006+0000","last_became_peered":"2024-08-19T14:18:26.875006+0000","last_unstale":"2024-08-19T14:21:42.812852+0000","last_undegraded":"2024-08-19T14:21:42.812852+0000","last_fullsized":"2024-08-19T14:21:42.812852+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:12.570352+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:28.864870+0000","last_clean_scrub_stamp":"2024-08-19T14:20:12.570352+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,0],"acting":[9,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.c","version":"364'6826","reported_seq":6894,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-08-19T14:21:43.092963+0000","last_change":"2024-08-19T14:20:29.266724+0000","last_active":"2024-08-19T14:21:43.092963+0000","last_peered":"2024-08-19T14:21:43.092963+0000","last_clean":"2024-08-19T14:21:43.092963+0000","last_became_active":"2024-08-19T14:20:29.266373+0000","last_became_peered":"2024-08-19T14:20:29.266373+0000","last_unstale":"2024-08-19T14:21:43.092963+0000","last_undegraded":"2024-08-19T14:21:43.092963+0000","last_fullsized":"2024-08-19T14:21:43.092963+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":6826,"log_dups_size":0,"ondisk_log_size":6826,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:48.006455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":447348736,"num_objects":6826,"num_object_clones":0,"num_object_copies":20478,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6826,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6826,"num_write_kb":436864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,8],"acting":[4,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"70'139","reported_seq":967,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-08-19T14:21:42.825247+0000","last_change":"2024-08-19T14:19:56.106047+0000","last_active":"2024-08-19T14:21:42.825247+0000","last_peered":"2024-08-19T14:21:42.825247+0000","last_clean":"2024-08-19T14:21:42.825247+0000","last_became_active":"2024-08-19T14:18:26.887657+0000","last_became_peered":"2024-08-19T14:18:26.887657+0000","last_unstale":"2024-08-19T14:21:42.825247+0000","last_undegraded":"2024-08-19T14:21:42.825247+0000","last_fullsized":"2024-08-19T14:21:42.825247+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:19:56.105880+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:19:56.105880+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"364'6825","reported_seq":6893,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-08-19T14:21:43.125384+0000","last_change":"2024-08-19T14:20:29.208992+0000","last_active":"2024-08-19T14:21:43.125384+0000","last_peered":"2024-08-19T14:21:43.125384+0000","last_clean":"2024-08-19T14:21:43.125384+0000","last_became_active":"2024-08-19T14:20:29.208599+0000","last_became_peered":"2024-08-19T14:20:29.208599+0000","last_unstale":"2024-08-19T14:21:43.125384+0000","last_undegraded":"2024-08-19T14:21:43.125384+0000","last_fullsized":"2024-08-19T14:21:43.125384+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":6825,"log_dups_size":0,"ondisk_log_size":6825,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:55.930633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":447283200,"num_objects":6825,"num_object_clones":0,"num_object_copies":20475,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6825,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6825,"num_write_kb":436800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,1],"acting":[4,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"70'139","reported_seq":864,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-08-19T14:21:42.814168+0000","last_change":"2024-08-19T14:21:37.951800+0000","last_active":"2024-08-19T14:21:42.814168+0000","last_peered":"2024-08-19T14:21:42.814168+0000","last_clean":"2024-08-19T14:21:42.814168+0000","last_became_active":"2024-08-19T14:18:26.887277+0000","last_became_peered":"2024-08-19T14:18:26.887277+0000","last_unstale":"2024-08-19T14:21:42.814168+0000","last_undegraded":"2024-08-19T14:21:42.814168+0000","last_fullsized":"2024-08-19T14:21:42.814168+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:37.951694+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:21:37.951694+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:03.860882+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,8],"acting":[5,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"364'6786","reported_seq":6853,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-08-19T14:21:43.162445+0000","last_change":"2024-08-19T14:20:29.209096+0000","last_active":"2024-08-19T14:21:43.162445+0000","last_peered":"2024-08-19T14:21:43.162445+0000","last_clean":"2024-08-19T14:21:43.162445+0000","last_became_active":"2024-08-19T14:20:29.208748+0000","last_became_peered":"2024-08-19T14:20:29.208748+0000","last_unstale":"2024-08-19T14:21:43.162445+0000","last_undegraded":"2024-08-19T14:21:43.162445+0000","last_fullsized":"2024-08-19T14:21:43.162445+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":6786,"log_dups_size":0,"ondisk_log_size":6786,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:56.597251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":444727296,"num_objects":6786,"num_object_clones":0,"num_object_copies":20358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6786,"num_write_kb":434304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"70'139","reported_seq":857,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-08-19T14:21:42.814134+0000","last_change":"2024-08-19T14:20:11.733961+0000","last_active":"2024-08-19T14:21:42.814134+0000","last_peered":"2024-08-19T14:21:42.814134+0000","last_clean":"2024-08-19T14:21:42.814134+0000","last_became_active":"2024-08-19T14:18:26.875179+0000","last_became_peered":"2024-08-19T14:18:26.875179+0000","last_unstale":"2024-08-19T14:21:42.814134+0000","last_undegraded":"2024-08-19T14:21:42.814134+0000","last_fullsized":"2024-08-19T14:21:42.814134+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:11.733865+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:11.733865+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.b","version":"364'7082","reported_seq":7225,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-08-19T14:21:46.214036+0000","last_change":"2024-08-19T14:21:39.089676+0000","last_active":"2024-08-19T14:21:46.214036+0000","last_peered":"2024-08-19T14:21:46.214036+0000","last_clean":"2024-08-19T14:21:46.214036+0000","last_became_active":"2024-08-19T14:20:29.207372+0000","last_became_peered":"2024-08-19T14:20:29.207372+0000","last_unstale":"2024-08-19T14:21:46.214036+0000","last_undegraded":"2024-08-19T14:21:46.214036+0000","last_fullsized":"2024-08-19T14:21:46.214036+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"363'6615","last_scrub_stamp":"2024-08-19T14:21:39.089400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:21:39.089400+0000","objects_scrubbed":6552,"log_size":7082,"log_dups_size":0,"ondisk_log_size":7082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:05.632119+0000","scrub_duration":1907,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":464125952,"num_objects":7082,"num_object_clones":0,"num_object_copies":21246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7082,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7082,"num_write_kb":453248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"70'139","reported_seq":511,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-08-19T14:21:42.818056+0000","last_change":"2024-08-19T14:20:15.322051+0000","last_active":"2024-08-19T14:21:42.818056+0000","last_peered":"2024-08-19T14:21:42.818056+0000","last_clean":"2024-08-19T14:21:42.818056+0000","last_became_active":"2024-08-19T14:18:26.884105+0000","last_became_peered":"2024-08-19T14:18:26.884105+0000","last_unstale":"2024-08-19T14:21:42.818056+0000","last_undegraded":"2024-08-19T14:21:42.818056+0000","last_fullsized":"2024-08-19T14:21:42.818056+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:15.321945+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:20:15.321945+0000","last_clean_scrub_stamp":"2024-08-19T14:20:15.321945+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:43.904541+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"364'7088","reported_seq":7156,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-08-19T14:21:44.163454+0000","last_change":"2024-08-19T14:20:29.269723+0000","last_active":"2024-08-19T14:21:44.163454+0000","last_peered":"2024-08-19T14:21:44.163454+0000","last_clean":"2024-08-19T14:21:44.163454+0000","last_became_active":"2024-08-19T14:20:29.269432+0000","last_became_peered":"2024-08-19T14:20:29.269432+0000","last_unstale":"2024-08-19T14:21:44.163454+0000","last_undegraded":"2024-08-19T14:21:44.163454+0000","last_fullsized":"2024-08-19T14:21:44.163454+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":7088,"log_dups_size":0,"ondisk_log_size":7088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:44.639451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":464519168,"num_objects":7088,"num_object_clones":0,"num_object_copies":21264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7088,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7088,"num_write_kb":453632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.f","version":"70'139","reported_seq":763,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-08-19T14:21:42.814333+0000","last_change":"2024-08-19T14:21:14.765419+0000","last_active":"2024-08-19T14:21:42.814333+0000","last_peered":"2024-08-19T14:21:42.814333+0000","last_clean":"2024-08-19T14:21:42.814333+0000","last_became_active":"2024-08-19T14:18:18.711708+0000","last_became_peered":"2024-08-19T14:18:18.711708+0000","last_unstale":"2024-08-19T14:21:42.814333+0000","last_undegraded":"2024-08-19T14:21:42.814333+0000","last_fullsized":"2024-08-19T14:21:42.814333+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:14.765340+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:21:14.765340+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:22:24.415219+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"364'7110","reported_seq":7257,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-08-19T14:21:45.788321+0000","last_change":"2024-08-19T14:21:43.974700+0000","last_active":"2024-08-19T14:21:45.788321+0000","last_peered":"2024-08-19T14:21:45.788321+0000","last_clean":"2024-08-19T14:21:45.788321+0000","last_became_active":"2024-08-19T14:20:29.211975+0000","last_became_peered":"2024-08-19T14:20:29.211975+0000","last_unstale":"2024-08-19T14:21:45.788321+0000","last_undegraded":"2024-08-19T14:21:45.788321+0000","last_fullsized":"2024-08-19T14:21:45.788321+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"364'6991","last_scrub_stamp":"2024-08-19T14:21:43.974613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:21:43.974613+0000","objects_scrubbed":6934,"log_size":7110,"log_dups_size":0,"ondisk_log_size":7110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:07.463188+0000","scrub_duration":1682,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":465960960,"num_objects":7110,"num_object_clones":0,"num_object_copies":21330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7110,"num_write_kb":455040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,3],"acting":[11,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.c","version":"70'139","reported_seq":624,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-08-19T14:21:42.829939+0000","last_change":"2024-08-19T14:21:40.181472+0000","last_active":"2024-08-19T14:21:42.829939+0000","last_peered":"2024-08-19T14:21:42.829939+0000","last_clean":"2024-08-19T14:21:42.829939+0000","last_became_active":"2024-08-19T14:18:26.872942+0000","last_became_peered":"2024-08-19T14:18:26.872942+0000","last_unstale":"2024-08-19T14:21:42.829939+0000","last_undegraded":"2024-08-19T14:21:42.829939+0000","last_fullsized":"2024-08-19T14:21:42.829939+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:40.181356+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:20:07.839076+0000","last_clean_scrub_stamp":"2024-08-19T14:21:40.181356+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:22:52.009381+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,9],"acting":[3,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"364'7032","reported_seq":7100,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-08-19T14:21:45.799530+0000","last_change":"2024-08-19T14:20:29.268318+0000","last_active":"2024-08-19T14:21:45.799530+0000","last_peered":"2024-08-19T14:21:45.799530+0000","last_clean":"2024-08-19T14:21:45.799530+0000","last_became_active":"2024-08-19T14:20:29.267442+0000","last_became_peered":"2024-08-19T14:20:29.267442+0000","last_unstale":"2024-08-19T14:21:45.799530+0000","last_undegraded":"2024-08-19T14:21:45.799530+0000","last_fullsized":"2024-08-19T14:21:45.799530+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":7032,"log_dups_size":0,"ondisk_log_size":7032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:47.715801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":460849152,"num_objects":7032,"num_object_clones":0,"num_object_copies":21096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7032,"num_write_kb":450048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,9],"acting":[11,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.d","version":"70'139","reported_seq":539,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-08-19T14:21:42.825376+0000","last_change":"2024-08-19T14:21:07.155294+0000","last_active":"2024-08-19T14:21:42.825376+0000","last_peered":"2024-08-19T14:21:42.825376+0000","last_clean":"2024-08-19T14:21:42.825376+0000","last_became_active":"2024-08-19T14:18:26.882002+0000","last_became_peered":"2024-08-19T14:18:26.882002+0000","last_unstale":"2024-08-19T14:21:42.825376+0000","last_undegraded":"2024-08-19T14:21:42.825376+0000","last_fullsized":"2024-08-19T14:21:42.825376+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:07.155157+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:21:07.155157+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:22:11.274708+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"70'139","reported_seq":798,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-08-19T14:21:42.816180+0000","last_change":"2024-08-19T14:21:13.777891+0000","last_active":"2024-08-19T14:21:42.816180+0000","last_peered":"2024-08-19T14:21:42.816180+0000","last_clean":"2024-08-19T14:21:42.816180+0000","last_became_active":"2024-08-19T14:18:18.703594+0000","last_became_peered":"2024-08-19T14:18:18.703594+0000","last_unstale":"2024-08-19T14:21:42.816180+0000","last_undegraded":"2024-08-19T14:21:42.816180+0000","last_fullsized":"2024-08-19T14:21:42.816180+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:13.777777+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:21:13.777777+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:22:38.473560+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"70'139","reported_seq":850,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-08-19T14:21:42.821144+0000","last_change":"2024-08-19T14:20:07.692054+0000","last_active":"2024-08-19T14:21:42.821144+0000","last_peered":"2024-08-19T14:21:42.821144+0000","last_clean":"2024-08-19T14:21:42.821144+0000","last_became_active":"2024-08-19T14:18:18.698334+0000","last_became_peered":"2024-08-19T14:18:18.698334+0000","last_unstale":"2024-08-19T14:21:42.821144+0000","last_undegraded":"2024-08-19T14:21:42.821144+0000","last_fullsized":"2024-08-19T14:21:42.821144+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:07.691932+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:07.691932+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.10","version":"70'139","reported_seq":953,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-08-19T14:21:42.813991+0000","last_change":"2024-08-19T14:21:07.768770+0000","last_active":"2024-08-19T14:21:42.813991+0000","last_peered":"2024-08-19T14:21:42.813991+0000","last_clean":"2024-08-19T14:21:42.813991+0000","last_became_active":"2024-08-19T14:18:18.704562+0000","last_became_peered":"2024-08-19T14:18:18.704562+0000","last_unstale":"2024-08-19T14:21:42.813991+0000","last_undegraded":"2024-08-19T14:21:42.813991+0000","last_fullsized":"2024-08-19T14:21:42.813991+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:07.768697+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:21:07.768697+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:22:10.360289+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,8],"acting":[5,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"70'139","reported_seq":502,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-08-19T14:21:42.828263+0000","last_change":"2024-08-19T14:21:16.071812+0000","last_active":"2024-08-19T14:21:42.828263+0000","last_peered":"2024-08-19T14:21:42.828263+0000","last_clean":"2024-08-19T14:21:42.828263+0000","last_became_active":"2024-08-19T14:18:18.697894+0000","last_became_peered":"2024-08-19T14:18:18.697894+0000","last_unstale":"2024-08-19T14:21:42.828263+0000","last_undegraded":"2024-08-19T14:21:42.828263+0000","last_fullsized":"2024-08-19T14:21:42.828263+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:16.071703+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:21:16.071703+0000","last_clean_scrub_stamp":"2024-08-19T14:21:16.071703+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:22:18.184091+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"70'139","reported_seq":479,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-08-19T14:21:42.828224+0000","last_change":"2024-08-19T14:20:28.205212+0000","last_active":"2024-08-19T14:21:42.828224+0000","last_peered":"2024-08-19T14:21:42.828224+0000","last_clean":"2024-08-19T14:21:42.828224+0000","last_became_active":"2024-08-19T14:20:28.204724+0000","last_became_peered":"2024-08-19T14:20:28.204724+0000","last_unstale":"2024-08-19T14:21:42.828224+0000","last_undegraded":"2024-08-19T14:21:42.828224+0000","last_fullsized":"2024-08-19T14:21:42.828224+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:08.680021+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:08.680021+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"70'139","reported_seq":599,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-08-19T14:21:42.877192+0000","last_change":"2024-08-19T14:20:31.931658+0000","last_active":"2024-08-19T14:21:42.877192+0000","last_peered":"2024-08-19T14:21:42.877192+0000","last_clean":"2024-08-19T14:21:42.877192+0000","last_became_active":"2024-08-19T14:18:31.901556+0000","last_became_peered":"2024-08-19T14:18:31.901556+0000","last_unstale":"2024-08-19T14:21:42.877192+0000","last_undegraded":"2024-08-19T14:21:42.877192+0000","last_fullsized":"2024-08-19T14:21:42.877192+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:31.931562+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:20:31.931562+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:47.687921+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,0],"acting":[8,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.14","version":"70'139","reported_seq":608,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-08-19T14:21:42.827440+0000","last_change":"2024-08-19T14:19:55.130518+0000","last_active":"2024-08-19T14:21:42.827440+0000","last_peered":"2024-08-19T14:21:42.827440+0000","last_clean":"2024-08-19T14:21:42.827440+0000","last_became_active":"2024-08-19T14:18:18.703926+0000","last_became_peered":"2024-08-19T14:18:18.703926+0000","last_unstale":"2024-08-19T14:21:42.827440+0000","last_undegraded":"2024-08-19T14:21:42.827440+0000","last_fullsized":"2024-08-19T14:21:42.827440+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:19:55.130420+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:19:55.130420+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,1],"acting":[7,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.15","version":"70'139","reported_seq":495,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-08-19T14:21:42.825439+0000","last_change":"2024-08-19T14:20:09.181046+0000","last_active":"2024-08-19T14:21:42.825439+0000","last_peered":"2024-08-19T14:21:42.825439+0000","last_clean":"2024-08-19T14:21:42.825439+0000","last_became_active":"2024-08-19T14:18:31.912496+0000","last_became_peered":"2024-08-19T14:18:31.912496+0000","last_unstale":"2024-08-19T14:21:42.825439+0000","last_undegraded":"2024-08-19T14:21:42.825439+0000","last_fullsized":"2024-08-19T14:21:42.825439+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:09.180923+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:20:09.180923+0000","last_clean_scrub_stamp":"2024-08-19T14:20:09.180923+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"70'139","reported_seq":481,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-08-19T14:21:42.817956+0000","last_change":"2024-08-19T14:21:40.430835+0000","last_active":"2024-08-19T14:21:42.817956+0000","last_peered":"2024-08-19T14:21:42.817956+0000","last_clean":"2024-08-19T14:21:42.817956+0000","last_became_active":"2024-08-19T14:18:31.901229+0000","last_became_peered":"2024-08-19T14:18:31.901229+0000","last_unstale":"2024-08-19T14:21:42.817956+0000","last_undegraded":"2024-08-19T14:21:42.817956+0000","last_fullsized":"2024-08-19T14:21:42.817956+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:40.430714+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:21:40.430714+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:04.072653+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1b","version":"70'139","reported_seq":825,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-08-19T14:21:42.877072+0000","last_change":"2024-08-19T14:20:16.776086+0000","last_active":"2024-08-19T14:21:42.877072+0000","last_peered":"2024-08-19T14:21:42.877072+0000","last_clean":"2024-08-19T14:21:42.877072+0000","last_became_active":"2024-08-19T14:18:31.901152+0000","last_became_peered":"2024-08-19T14:18:31.901152+0000","last_unstale":"2024-08-19T14:21:42.877072+0000","last_undegraded":"2024-08-19T14:21:42.877072+0000","last_fullsized":"2024-08-19T14:21:42.877072+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:16.775987+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:16.775987+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:46.485727+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.18","version":"70'139","reported_seq":869,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-08-19T14:21:42.814000+0000","last_change":"2024-08-19T14:20:36.510554+0000","last_active":"2024-08-19T14:21:42.814000+0000","last_peered":"2024-08-19T14:21:42.814000+0000","last_clean":"2024-08-19T14:21:42.814000+0000","last_became_active":"2024-08-19T14:18:31.902735+0000","last_became_peered":"2024-08-19T14:18:31.902735+0000","last_unstale":"2024-08-19T14:21:42.814000+0000","last_undegraded":"2024-08-19T14:21:42.814000+0000","last_fullsized":"2024-08-19T14:21:42.814000+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:36.510442+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:36.510442+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:53.706735+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.19","version":"70'139","reported_seq":825,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-08-19T14:21:42.828128+0000","last_change":"2024-08-19T14:21:40.219949+0000","last_active":"2024-08-19T14:21:42.828128+0000","last_peered":"2024-08-19T14:21:42.828128+0000","last_clean":"2024-08-19T14:21:42.828128+0000","last_became_active":"2024-08-19T14:18:31.902547+0000","last_became_peered":"2024-08-19T14:18:31.902547+0000","last_unstale":"2024-08-19T14:21:42.828128+0000","last_undegraded":"2024-08-19T14:21:42.828128+0000","last_fullsized":"2024-08-19T14:21:42.828128+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:40.219830+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:21:40.219830+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:05.414864+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1e","version":"70'139","reported_seq":475,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-08-19T14:21:42.814199+0000","last_change":"2024-08-19T14:20:18.771836+0000","last_active":"2024-08-19T14:21:42.814199+0000","last_peered":"2024-08-19T14:21:42.814199+0000","last_clean":"2024-08-19T14:21:42.814199+0000","last_became_active":"2024-08-19T14:18:31.902546+0000","last_became_peered":"2024-08-19T14:18:31.902546+0000","last_unstale":"2024-08-19T14:21:42.814199+0000","last_undegraded":"2024-08-19T14:21:42.814199+0000","last_fullsized":"2024-08-19T14:21:42.814199+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:18.771704+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:18.771704+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:47.154907+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1c","version":"70'139","reported_seq":591,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-08-19T14:21:42.879543+0000","last_change":"2024-08-19T14:21:38.753663+0000","last_active":"2024-08-19T14:21:42.879543+0000","last_peered":"2024-08-19T14:21:42.879543+0000","last_clean":"2024-08-19T14:21:42.879543+0000","last_became_active":"2024-08-19T14:18:31.902494+0000","last_became_peered":"2024-08-19T14:18:31.902494+0000","last_unstale":"2024-08-19T14:21:42.879543+0000","last_undegraded":"2024-08-19T14:21:42.879543+0000","last_fullsized":"2024-08-19T14:21:42.879543+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:38.753553+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:21:38.753553+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:22:46.720742+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,11],"acting":[8,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1d","version":"70'139","reported_seq":505,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-08-19T14:21:42.823354+0000","last_change":"2024-08-19T14:20:17.725615+0000","last_active":"2024-08-19T14:21:42.823354+0000","last_peered":"2024-08-19T14:21:42.823354+0000","last_clean":"2024-08-19T14:21:42.823354+0000","last_became_active":"2024-08-19T14:18:31.901183+0000","last_became_peered":"2024-08-19T14:18:31.901183+0000","last_unstale":"2024-08-19T14:21:42.823354+0000","last_undegraded":"2024-08-19T14:21:42.823354+0000","last_fullsized":"2024-08-19T14:21:42.823354+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:17.725477+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:17.725477+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,5],"acting":[1,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":7338524672,"num_objects":111977,"num_object_clones":0,"num_object_copies":335931,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3181,"num_objects_unfound":0,"num_objects_dirty":111977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":111977,"num_write_kb":7166528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":959,"num_bytes_recovered":62849024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":22059941888,"data_stored":22059941888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111977,"ondisk_log_size":111977,"up":48,"acting":47,"num_store_stats":9},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1802240,"data_stored":1788032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187581,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3569772,"kb_used_data":3163748,"kb_used_omap":1,"kb_used_meta":406014,"kb_avail":90175380,"statfs":{"total":95995035648,"available":92339589120,"internally_reserved":0,"allocated":3239677952,"data_stored":3237995698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":415758788},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":343597383811,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":74,"seq":317827580038,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2014496,"kb_used_data":1559896,"kb_used_omap":1,"kb_used_meta":454590,"kb_avail":91730656,"statfs":{"total":95995035648,"available":93932191744,"internally_reserved":0,"allocated":1597333504,"data_stored":1595655476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":465500609},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874380,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":817584,"kb_used_data":447652,"kb_used_omap":1,"kb_used_meta":369918,"kb_avail":92927568,"statfs":{"total":95995035648,"available":95157829632,"internally_reserved":0,"allocated":458395648,"data_stored":456713394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":378796485},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":57,"seq":244813136014,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4004664,"kb_used_data":3550060,"kb_used_omap":1,"kb_used_meta":454590,"kb_avail":89740488,"statfs":{"total":95995035648,"available":91894259712,"internally_reserved":0,"allocated":3635261440,"data_stored":3633575634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":465500618},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":206158430354,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691991,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1986868,"kb_used_data":1648876,"kb_used_omap":1,"kb_used_meta":337982,"kb_avail":91758284,"statfs":{"total":95995035648,"available":93960482816,"internally_reserved":0,"allocated":1688449024,"data_stored":1686763218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":346094021},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986331,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2650912,"kb_used_data":2186584,"kb_used_omap":1,"kb_used_meta":464318,"kb_avail":91094240,"statfs":{"total":95995035648,"available":93280501760,"internally_reserved":0,"allocated":2239062016,"data_stored":2237383988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":475462085},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149857,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3076720,"kb_used_data":2662372,"kb_used_omap":1,"kb_used_meta":414334,"kb_avail":90668432,"statfs":{"total":95995035648,"available":92844474368,"internally_reserved":0,"allocated":2726268928,"data_stored":2724586674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":424278471},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411495,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575018,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4898232,"kb_used_data":4425580,"kb_used_omap":1,"kb_used_meta":472638,"kb_avail":88846920,"statfs":{"total":95995035648,"available":90979246080,"internally_reserved":0,"allocated":4531793920,"data_stored":4530108114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":483981765},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738543,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2164460,"kb_used_data":1822308,"kb_used_omap":1,"kb_used_meta":342142,"kb_avail":91580692,"statfs":{"total":95995035648,"available":93778628608,"internally_reserved":0,"allocated":1866043392,"data_stored":1864361138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":350353861},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1861681152,"data_stored":1861681152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4569759744,"data_stored":4569759744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2733637632,"data_stored":2733637632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2247032832,"data_stored":2247032832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1693450240,"data_stored":1693450240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3636330496,"data_stored":3636330496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":455278592,"data_stored":455278592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1607794688,"data_stored":1607794688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":3254976512,"data_stored":3254976512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T14:21:49.035 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:21:49.050 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 75 16 116558 116542 97.1101 53.75 0.0193947 0.0102931 2024-08-19T14:21:49.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:49.061 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:49.062 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:21:49.163 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:21:49.202 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:49.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:49.203 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:21:49.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:48 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/654515354' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:21:49.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:48 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:21:49.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:48 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/654515354' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:21:49.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:48 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:21:49.304 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:21:49.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:49.335 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:49.335 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:21:49.436 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:21:49.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:49.479 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:49.479 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:21:49.580 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:21:49.607 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:49.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:49.608 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:21:49.709 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:21:49.736 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:49.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:49.737 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:21:49.737 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:21:49.838 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:21:49.865 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:49.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:49.867 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:21:49.967 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:21:50.000 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:50.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:50.001 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:21:50.050 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 76 16 117661 117645 96.7393 68.9375 0.00618271 0.0103325 2024-08-19T14:21:50.102 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:21:50.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:49 smithi031 ceph-mon[16336]: pgmap v822: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 6.8 GiB data, 24 GiB used, 781 GiB / 805 GiB avail; 57 MiB/s wr, 904 op/s; 3181/335937 objects misplaced (0.947%); 1.0 MiB/s, 16 objects/s recovering 2024-08-19T14:21:50.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:49 smithi031 ceph-mon[16336]: from='client.15756 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:21:50.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:49 smithi031 ceph-mon[16336]: 1.1b scrub starts 2024-08-19T14:21:50.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:49 smithi031 ceph-mon[16336]: 1.1b scrub ok 2024-08-19T14:21:50.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:49 smithi031 ceph-mon[16336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:21:50.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:49 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:21:50.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:49 smithi031 ceph-mon[16336]: osdmap e366: 12 total, 12 up, 9 in 2024-08-19T14:21:50.123 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:50.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:50.124 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:21:50.158 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:49 smithi032 ceph-mon[20004]: pgmap v822: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 6.8 GiB data, 24 GiB used, 781 GiB / 805 GiB avail; 57 MiB/s wr, 904 op/s; 3181/335937 objects misplaced (0.947%); 1.0 MiB/s, 16 objects/s recovering 2024-08-19T14:21:50.158 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:49 smithi032 ceph-mon[20004]: from='client.15756 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:21:50.158 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:49 smithi032 ceph-mon[20004]: 1.1b scrub starts 2024-08-19T14:21:50.158 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:49 smithi032 ceph-mon[20004]: 1.1b scrub ok 2024-08-19T14:21:50.158 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:49 smithi032 ceph-mon[20004]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:21:50.159 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:49 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:21:50.159 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:49 smithi032 ceph-mon[20004]: osdmap e366: 12 total, 12 up, 9 in 2024-08-19T14:21:50.224 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:21:50.249 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:50.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:50.250 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:21:50.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:49 smithi045 ceph-mon[20143]: pgmap v822: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 6.8 GiB data, 24 GiB used, 781 GiB / 805 GiB avail; 57 MiB/s wr, 904 op/s; 3181/335937 objects misplaced (0.947%); 1.0 MiB/s, 16 objects/s recovering 2024-08-19T14:21:50.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:49 smithi045 ceph-mon[20143]: from='client.15756 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:21:50.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:49 smithi045 ceph-mon[20143]: 1.1b scrub starts 2024-08-19T14:21:50.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:49 smithi045 ceph-mon[20143]: 1.1b scrub ok 2024-08-19T14:21:50.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:49 smithi045 ceph-mon[20143]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:21:50.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:49 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:21:50.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:49 smithi045 ceph-mon[20143]: osdmap e366: 12 total, 12 up, 9 in 2024-08-19T14:21:50.351 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:21:50.376 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:50.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:50.377 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:21:50.385 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:21:50.386 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:21:50.421 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:21:50.422 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:21:50.448 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:21:50.448 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:21:50.478 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:21:50.484 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:21:50.484 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:21:50.519 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:50.519 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:21:50.520 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:21:50.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:50.520 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:21:50.553 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:21:50.553 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:21:50.586 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:21:50.587 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:21:50.618 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:21:50.618 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:21:50.621 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:21:50.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:50.649 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:50.650 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:21:50.650 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:21:50.651 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:21:50.683 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:21:50.683 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:21:50.719 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:21:50.719 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:21:50.750 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:21:50.752 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:21:50.752 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:21:50.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:50.776 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:50.776 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:21:50.877 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:21:50.911 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:50.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:50.913 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:21:50.971 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:50 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3488856992' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:21:51.011 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:50 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3488856992' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:21:51.014 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:21:51.047 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:51.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:51.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:21:51.050 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 77 16 118517 118501 96.1777 53.5 0.00946877 0.0103929 2024-08-19T14:21:51.148 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:21:51.161 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:50 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3488856992' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:21:51.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:51.187 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:51.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:21:51.202 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-08-19T14:21:51.290 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:21:51.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:51.334 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:51.334 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:21:51.435 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:21:51.470 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:51.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:51.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:21:51.559 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:21:51.572 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:21:51.616 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:51.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:51.618 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:21:51.627 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-19T14:21:51.719 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:21:51.747 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:51.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:51.748 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:21:51.849 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:21:51.877 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:51.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:51.879 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:21:51.938 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-19T14:21:51.979 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:21:51.985 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:51 smithi045 ceph-mon[20143]: pgmap v824: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 7.0 GiB data, 25 GiB used, 780 GiB / 805 GiB avail; 76 MiB/s wr, 1.22k op/s; 3037/345894 objects misplaced (0.878%); 2.4 MiB/s, 38 objects/s recovering 2024-08-19T14:21:51.985 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:51 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:21:51.985 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:51 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:21:52.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:52.010 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:52.011 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:21:52.052 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 78 16 119433 119417 95.6786 57.25 0.00652566 0.0104295 2024-08-19T14:21:52.079 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:21:52.112 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:21:52.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:51 smithi031 ceph-mon[16336]: pgmap v824: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 7.0 GiB data, 25 GiB used, 780 GiB / 805 GiB avail; 76 MiB/s wr, 1.22k op/s; 3037/345894 objects misplaced (0.878%); 2.4 MiB/s, 38 objects/s recovering 2024-08-19T14:21:52.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:51 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:21:52.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:51 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:21:52.138 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:52.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:52.139 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:21:52.161 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:51 smithi032 ceph-mon[20004]: pgmap v824: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 7.0 GiB data, 25 GiB used, 780 GiB / 805 GiB avail; 76 MiB/s wr, 1.22k op/s; 3037/345894 objects misplaced (0.878%); 2.4 MiB/s, 38 objects/s recovering 2024-08-19T14:21:52.162 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:51 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:21:52.162 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:51 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:21:52.239 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:21:52.264 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:52.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:52.265 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:21:52.366 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:21:52.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:52.420 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:52.420 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:21:52.440 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:21:52.441 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:21:52.520 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:21:52.546 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:52.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:52.547 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:21:52.648 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:21:52.678 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:52.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:52.679 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:21:52.753 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:21:52.780 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:21:52.802 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:52.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:52.803 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:21:52.904 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:21:52.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:52.931 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:52.931 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:21:53.033 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:21:53.050 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 79 16 120306 120290 95.1581 54.5625 0.0213798 0.010504 2024-08-19T14:21:53.056 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:53.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:53.057 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:21:53.103 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-08-19T14:21:53.127 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:52 smithi045 ceph-mon[20143]: osdmap e367: 12 total, 12 up, 9 in 2024-08-19T14:21:53.127 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:52 smithi045 ceph-mon[20143]: pgmap v826: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 7.2 GiB data, 25 GiB used, 780 GiB / 805 GiB avail; 57 MiB/s wr, 919 op/s; 3037/351486 objects misplaced (0.864%); 1.5 MiB/s, 23 objects/s recovering 2024-08-19T14:21:53.135 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:52 smithi031 ceph-mon[16336]: osdmap e367: 12 total, 12 up, 9 in 2024-08-19T14:21:53.135 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:52 smithi031 ceph-mon[16336]: pgmap v826: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 7.2 GiB data, 25 GiB used, 780 GiB / 805 GiB avail; 57 MiB/s wr, 919 op/s; 3037/351486 objects misplaced (0.864%); 1.5 MiB/s, 23 objects/s recovering 2024-08-19T14:21:53.158 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:21:53.187 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:53.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:53.188 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:21:53.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:52 smithi032 ceph-mon[20004]: osdmap e367: 12 total, 12 up, 9 in 2024-08-19T14:21:53.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:52 smithi032 ceph-mon[20004]: pgmap v826: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 7.2 GiB data, 25 GiB used, 780 GiB / 805 GiB avail; 57 MiB/s wr, 919 op/s; 3037/351486 objects misplaced (0.864%); 1.5 MiB/s, 23 objects/s recovering 2024-08-19T14:21:53.289 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:21:53.315 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:53.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:53.316 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:21:53.416 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:21:53.445 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:21:53.451 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:53.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:53.452 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:21:53.553 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:21:53.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-08-19T14:21:53.576+0000 7fa73b7fe640 0 --2- 172.21.15.31:0/2718564704 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa73c1124b0 0x7fa73c06c930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-08-19T14:21:53.588 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:21:53.592 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:53.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:53.593 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:21:53.694 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:21:53.726 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:53.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:53.727 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:21:53.828 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:21:53.864 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:53.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:53.865 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:21:53.895 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is set 2024-08-19T14:21:53.928 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:21:53.929 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-08-19T14:21:53.966 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:21:53.992 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:53.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:53.993 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:21:54.004 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:21:54.005 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:21:54.051 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:2024-08-19T14:21:54.051393+0000 min lat: 0.00106793 max lat: 2.42694 avg lat: 0.0105511 2024-08-19T14:21:54.051 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-19T14:21:54.051 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 80 16 121289 121273 94.7365 61.4375 0.00756796 0.0105511 2024-08-19T14:21:54.094 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:21:54.123 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:54.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:54.124 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:21:54.139 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:53 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:21:54.139 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:53 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:21:54.139 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:53 smithi031 ceph-mon[16336]: osdmap e368: 12 total, 12 up, 9 in 2024-08-19T14:21:54.139 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:53 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3251288606' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:21:54.140 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:53 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:21:54.225 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:21:54.252 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:54.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:54.253 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:21:54.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:53 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:21:54.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:53 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:21:54.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:53 smithi032 ceph-mon[20004]: osdmap e368: 12 total, 12 up, 9 in 2024-08-19T14:21:54.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:53 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3251288606' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:21:54.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:53 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:21:54.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:53 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:21:54.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:53 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:21:54.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:53 smithi045 ceph-mon[20143]: osdmap e368: 12 total, 12 up, 9 in 2024-08-19T14:21:54.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:53 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3251288606' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:21:54.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:53 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:21:54.354 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:21:54.385 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:54.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:54.386 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:21:54.487 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:21:54.512 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:54.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:54.513 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:21:54.615 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:21:54.648 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:54.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:54.649 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:21:54.699 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-08-19T14:21:54.750 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:21:54.785 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:54.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:54.786 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:21:54.882 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":829,"stamp":"2024-08-19T14:21:53.904196+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7752450592,"num_objects":118286,"num_object_clones":0,"num_object_copies":354858,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3037,"num_objects_unfound":0,"num_objects_dirty":118286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":118284,"num_write_kb":7570176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1103,"num_bytes_recovered":72286208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":122593,"ondisk_log_size":122593,"up":141,"acting":140,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":142,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":26581492,"kb_used_data":22805144,"kb_used_omap":14,"kb_used_meta":3776241,"kb_avail":817124876,"statfs":{"total":863955320832,"available":836735873024,"internally_reserved":0,"allocated":23352467456,"data_stored":23337228418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14353,"internal_metadata":3866871791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,2,4,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":324534272,"num_objects":4952,"num_object_clones":0,"num_object_copies":14856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-144,"num_objects_unfound":0,"num_objects_dirty":4952,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4952,"num_write_kb":316928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":144,"num_bytes_recovered":9437184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.064312"},"pg_stats":[{"pgid":"4.5","version":"366'7422","reported_seq":7494,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-08-19T14:21:50.800141+0000","last_change":"2024-08-19T14:20:29.268214+0000","last_active":"2024-08-19T14:21:50.800141+0000","last_peered":"2024-08-19T14:21:50.800141+0000","last_clean":"2024-08-19T14:21:50.800141+0000","last_became_active":"2024-08-19T14:20:29.267549+0000","last_became_peered":"2024-08-19T14:20:29.267549+0000","last_unstale":"2024-08-19T14:21:50.800141+0000","last_undegraded":"2024-08-19T14:21:50.800141+0000","last_fullsized":"2024-08-19T14:21:50.800141+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":7422,"log_dups_size":0,"ondisk_log_size":7422,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:57.640230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":486408192,"num_objects":7422,"num_object_clones":0,"num_object_copies":22266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7422,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7422,"num_write_kb":475008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,0],"acting":[11,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"70'139","reported_seq":952,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-08-19T14:21:52.882898+0000","last_change":"2024-08-19T14:19:58.072227+0000","last_active":"2024-08-19T14:21:52.882898+0000","last_peered":"2024-08-19T14:21:52.882898+0000","last_clean":"2024-08-19T14:21:52.882898+0000","last_became_active":"2024-08-19T14:18:13.807133+0000","last_became_peered":"2024-08-19T14:18:13.807133+0000","last_unstale":"2024-08-19T14:21:52.882898+0000","last_undegraded":"2024-08-19T14:21:52.882898+0000","last_fullsized":"2024-08-19T14:21:52.882898+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:19:58.072083+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:19:58.072083+0000","last_clean_scrub_stamp":"2024-08-19T14:19:58.072083+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,9],"acting":[7,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"366'7419","reported_seq":9699,"reported_epoch":366,"state":"active+recovering+undersized+remapped","last_fresh":"2024-08-19T14:21:49.228088+0000","last_change":"2024-08-19T14:21:07.239135+0000","last_active":"2024-08-19T14:21:49.228088+0000","last_peered":"2024-08-19T14:21:49.228088+0000","last_clean":"2024-08-19T14:21:04.784061+0000","last_became_active":"2024-08-19T14:21:07.213050+0000","last_became_peered":"2024-08-19T14:21:07.213050+0000","last_unstale":"2024-08-19T14:21:49.228088+0000","last_undegraded":"2024-08-19T14:21:49.228088+0000","last_fullsized":"2024-08-19T14:21:06.805412+0000","mapping_epoch":354,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":7419,"log_dups_size":0,"ondisk_log_size":7419,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":486211584,"num_objects":7419,"num_object_clones":0,"num_object_copies":22257,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3037,"num_objects_unfound":0,"num_objects_dirty":7419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7419,"num_write_kb":474816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1103,"num_bytes_recovered":72286208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,7],"acting":[1,7],"avail_no_missing":["1","5","7"],"object_location_counts":[{"shards":"1,5,7","objects":3037},{"shards":"1,7","objects":4382}],"blocked_by":[],"up_primary":9,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"70'139","reported_seq":847,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-08-19T14:21:48.853105+0000","last_change":"2024-08-19T14:20:03.714917+0000","last_active":"2024-08-19T14:21:48.853105+0000","last_peered":"2024-08-19T14:21:48.853105+0000","last_clean":"2024-08-19T14:21:48.853105+0000","last_became_active":"2024-08-19T14:18:13.804721+0000","last_became_peered":"2024-08-19T14:18:13.804721+0000","last_unstale":"2024-08-19T14:21:48.853105+0000","last_undegraded":"2024-08-19T14:21:48.853105+0000","last_fullsized":"2024-08-19T14:21:48.853105+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:03.714818+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:03.714818+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"366'7351","reported_seq":7423,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-08-19T14:21:49.233870+0000","last_change":"2024-08-19T14:20:29.268110+0000","last_active":"2024-08-19T14:21:49.233870+0000","last_peered":"2024-08-19T14:21:49.233870+0000","last_clean":"2024-08-19T14:21:49.233870+0000","last_became_active":"2024-08-19T14:20:29.267811+0000","last_became_peered":"2024-08-19T14:20:29.267811+0000","last_unstale":"2024-08-19T14:21:49.233870+0000","last_undegraded":"2024-08-19T14:21:49.233870+0000","last_fullsized":"2024-08-19T14:21:49.233870+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":7351,"log_dups_size":0,"ondisk_log_size":7351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":481755136,"num_objects":7351,"num_object_clones":0,"num_object_copies":22053,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7351,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7351,"num_write_kb":470464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"70'139","reported_seq":799,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-08-19T14:21:52.881324+0000","last_change":"2024-08-19T14:20:50.742676+0000","last_active":"2024-08-19T14:21:52.881324+0000","last_peered":"2024-08-19T14:21:52.881324+0000","last_clean":"2024-08-19T14:21:52.881324+0000","last_became_active":"2024-08-19T14:20:50.742354+0000","last_became_peered":"2024-08-19T14:20:50.742354+0000","last_unstale":"2024-08-19T14:21:52.881324+0000","last_undegraded":"2024-08-19T14:21:52.881324+0000","last_fullsized":"2024-08-19T14:21:52.881324+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:19:59.302931+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:19:59.302931+0000","last_clean_scrub_stamp":"2024-08-19T14:19:59.302931+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":114,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"366'7273","reported_seq":7345,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-08-19T14:21:49.215798+0000","last_change":"2024-08-19T14:20:29.267640+0000","last_active":"2024-08-19T14:21:49.215798+0000","last_peered":"2024-08-19T14:21:49.215798+0000","last_clean":"2024-08-19T14:21:49.215798+0000","last_became_active":"2024-08-19T14:20:29.267331+0000","last_became_peered":"2024-08-19T14:20:29.267331+0000","last_unstale":"2024-08-19T14:21:49.215798+0000","last_undegraded":"2024-08-19T14:21:49.215798+0000","last_fullsized":"2024-08-19T14:21:49.215798+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":7273,"log_dups_size":0,"ondisk_log_size":7273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":476643328,"num_objects":7273,"num_object_clones":0,"num_object_copies":21819,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7273,"num_write_kb":465472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"70'139","reported_seq":848,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-08-19T14:21:52.880296+0000","last_change":"2024-08-19T14:20:09.507422+0000","last_active":"2024-08-19T14:21:52.880296+0000","last_peered":"2024-08-19T14:21:52.880296+0000","last_clean":"2024-08-19T14:21:52.880296+0000","last_became_active":"2024-08-19T14:18:13.808505+0000","last_became_peered":"2024-08-19T14:18:13.808505+0000","last_unstale":"2024-08-19T14:21:52.880296+0000","last_undegraded":"2024-08-19T14:21:52.880296+0000","last_fullsized":"2024-08-19T14:21:52.880296+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:09.507256+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:09.507256+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,11],"acting":[9,0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"368'7596","reported_seq":7672,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-08-19T14:21:53.407640+0000","last_change":"2024-08-19T14:20:29.268766+0000","last_active":"2024-08-19T14:21:53.407640+0000","last_peered":"2024-08-19T14:21:53.407640+0000","last_clean":"2024-08-19T14:21:53.407640+0000","last_became_active":"2024-08-19T14:20:29.268443+0000","last_became_peered":"2024-08-19T14:20:29.268443+0000","last_unstale":"2024-08-19T14:21:53.407640+0000","last_undegraded":"2024-08-19T14:21:53.407640+0000","last_fullsized":"2024-08-19T14:21:53.407640+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":7596,"log_dups_size":0,"ondisk_log_size":7596,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":497811456,"num_objects":7596,"num_object_clones":0,"num_object_copies":22788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7596,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7596,"num_write_kb":486144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,11],"acting":[9,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.7","version":"70'139","reported_seq":495,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-08-19T14:21:48.841936+0000","last_change":"2024-08-19T14:21:46.057764+0000","last_active":"2024-08-19T14:21:48.841936+0000","last_peered":"2024-08-19T14:21:48.841936+0000","last_clean":"2024-08-19T14:21:48.841936+0000","last_became_active":"2024-08-19T14:20:28.204384+0000","last_became_peered":"2024-08-19T14:20:28.204384+0000","last_unstale":"2024-08-19T14:21:48.841936+0000","last_undegraded":"2024-08-19T14:21:48.841936+0000","last_fullsized":"2024-08-19T14:21:48.841936+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:46.057624+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:21:46.057624+0000","last_clean_scrub_stamp":"2024-08-19T14:21:46.057624+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:02.689278+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"366'7337","reported_seq":7465,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-08-19T14:21:50.791599+0000","last_change":"2024-08-19T14:21:48.933576+0000","last_active":"2024-08-19T14:21:50.791599+0000","last_peered":"2024-08-19T14:21:50.791599+0000","last_clean":"2024-08-19T14:21:50.791599+0000","last_became_active":"2024-08-19T14:20:29.267379+0000","last_became_peered":"2024-08-19T14:20:29.267379+0000","last_unstale":"2024-08-19T14:21:50.791599+0000","last_undegraded":"2024-08-19T14:21:50.791599+0000","last_fullsized":"2024-08-19T14:21:50.791599+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":5249,"log_size":7337,"log_dups_size":0,"ondisk_log_size":7337,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:53.903409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":480837632,"num_objects":7337,"num_object_clones":0,"num_object_copies":22011,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7337,"num_write_kb":469568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,1],"acting":[11,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.6","version":"70'139","reported_seq":852,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-08-19T14:21:52.880592+0000","last_change":"2024-08-19T14:19:55.512078+0000","last_active":"2024-08-19T14:21:52.880592+0000","last_peered":"2024-08-19T14:21:52.880592+0000","last_clean":"2024-08-19T14:21:52.880592+0000","last_became_active":"2024-08-19T14:18:26.874902+0000","last_became_peered":"2024-08-19T14:18:26.874902+0000","last_unstale":"2024-08-19T14:21:52.880592+0000","last_undegraded":"2024-08-19T14:21:52.880592+0000","last_fullsized":"2024-08-19T14:21:52.880592+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:19:55.511974+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:27.848870+0000","last_clean_scrub_stamp":"2024-08-19T14:19:55.511974+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,4],"acting":[9,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"366'7257","reported_seq":7329,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-08-19T14:21:49.493812+0000","last_change":"2024-08-19T14:20:29.267682+0000","last_active":"2024-08-19T14:21:49.493812+0000","last_peered":"2024-08-19T14:21:49.493812+0000","last_clean":"2024-08-19T14:21:49.493812+0000","last_became_active":"2024-08-19T14:20:29.267360+0000","last_became_peered":"2024-08-19T14:20:29.267360+0000","last_unstale":"2024-08-19T14:21:49.493812+0000","last_undegraded":"2024-08-19T14:21:49.493812+0000","last_fullsized":"2024-08-19T14:21:49.493812+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":7257,"log_dups_size":0,"ondisk_log_size":7257,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:51.551392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":475594752,"num_objects":7257,"num_object_clones":0,"num_object_copies":21771,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7257,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7257,"num_write_kb":464448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"70'139","reported_seq":661,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-08-19T14:21:48.850609+0000","last_change":"2024-08-19T14:20:12.731993+0000","last_active":"2024-08-19T14:21:48.850609+0000","last_peered":"2024-08-19T14:21:48.850609+0000","last_clean":"2024-08-19T14:21:48.850609+0000","last_became_active":"2024-08-19T14:18:26.881435+0000","last_became_peered":"2024-08-19T14:18:26.881435+0000","last_unstale":"2024-08-19T14:21:48.850609+0000","last_undegraded":"2024-08-19T14:21:48.850609+0000","last_fullsized":"2024-08-19T14:21:48.850609+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:12.731850+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:12.731850+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,4],"acting":[11,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.1","version":"366'7447","reported_seq":7519,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-08-19T14:21:50.725449+0000","last_change":"2024-08-19T14:20:29.268808+0000","last_active":"2024-08-19T14:21:50.725449+0000","last_peered":"2024-08-19T14:21:50.725449+0000","last_clean":"2024-08-19T14:21:50.725449+0000","last_became_active":"2024-08-19T14:20:29.268465+0000","last_became_peered":"2024-08-19T14:20:29.268465+0000","last_unstale":"2024-08-19T14:21:50.725449+0000","last_undegraded":"2024-08-19T14:21:50.725449+0000","last_fullsized":"2024-08-19T14:21:50.725449+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":7447,"log_dups_size":0,"ondisk_log_size":7447,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:57.700841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":488046592,"num_objects":7447,"num_object_clones":0,"num_object_copies":22341,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7447,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7447,"num_write_kb":476608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.4","version":"70'139","reported_seq":894,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-08-19T14:21:48.892637+0000","last_change":"2024-08-19T14:21:39.314063+0000","last_active":"2024-08-19T14:21:48.892637+0000","last_peered":"2024-08-19T14:21:48.892637+0000","last_clean":"2024-08-19T14:21:48.892637+0000","last_became_active":"2024-08-19T14:18:13.806013+0000","last_became_peered":"2024-08-19T14:18:13.806013+0000","last_unstale":"2024-08-19T14:21:48.892637+0000","last_undegraded":"2024-08-19T14:21:48.892637+0000","last_fullsized":"2024-08-19T14:21:48.892637+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:39.313941+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:21:39.313941+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:22:45.184594+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.f","version":"366'7281","reported_seq":7353,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-08-19T14:21:49.154596+0000","last_change":"2024-08-19T14:20:29.212298+0000","last_active":"2024-08-19T14:21:49.154596+0000","last_peered":"2024-08-19T14:21:49.154596+0000","last_clean":"2024-08-19T14:21:49.154596+0000","last_became_active":"2024-08-19T14:20:29.212003+0000","last_became_peered":"2024-08-19T14:20:29.212003+0000","last_unstale":"2024-08-19T14:21:49.154596+0000","last_undegraded":"2024-08-19T14:21:49.154596+0000","last_fullsized":"2024-08-19T14:21:49.154596+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":7281,"log_dups_size":0,"ondisk_log_size":7281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:52.621283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":477167616,"num_objects":7281,"num_object_clones":0,"num_object_copies":21843,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7281,"num_write_kb":465984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,11],"acting":[1,3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"70'139","reported_seq":860,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-08-19T14:21:52.880436+0000","last_change":"2024-08-19T14:20:12.570453+0000","last_active":"2024-08-19T14:21:52.880436+0000","last_peered":"2024-08-19T14:21:52.880436+0000","last_clean":"2024-08-19T14:21:52.880436+0000","last_became_active":"2024-08-19T14:18:26.875006+0000","last_became_peered":"2024-08-19T14:18:26.875006+0000","last_unstale":"2024-08-19T14:21:52.880436+0000","last_undegraded":"2024-08-19T14:21:52.880436+0000","last_fullsized":"2024-08-19T14:21:52.880436+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:12.570352+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:28.864870+0000","last_clean_scrub_stamp":"2024-08-19T14:20:12.570352+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,0],"acting":[9,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.c","version":"368'7472","reported_seq":7548,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-08-19T14:21:53.115811+0000","last_change":"2024-08-19T14:20:29.266724+0000","last_active":"2024-08-19T14:21:53.115811+0000","last_peered":"2024-08-19T14:21:53.115811+0000","last_clean":"2024-08-19T14:21:53.115811+0000","last_became_active":"2024-08-19T14:20:29.266373+0000","last_became_peered":"2024-08-19T14:20:29.266373+0000","last_unstale":"2024-08-19T14:21:53.115811+0000","last_undegraded":"2024-08-19T14:21:53.115811+0000","last_fullsized":"2024-08-19T14:21:53.115811+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":7472,"log_dups_size":0,"ondisk_log_size":7472,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":489684992,"num_objects":7472,"num_object_clones":0,"num_object_copies":22416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7472,"num_write_kb":478208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,8],"acting":[4,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"70'139","reported_seq":975,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-08-19T14:21:52.881355+0000","last_change":"2024-08-19T14:19:56.106047+0000","last_active":"2024-08-19T14:21:52.881355+0000","last_peered":"2024-08-19T14:21:52.881355+0000","last_clean":"2024-08-19T14:21:52.881355+0000","last_became_active":"2024-08-19T14:18:26.887657+0000","last_became_peered":"2024-08-19T14:18:26.887657+0000","last_unstale":"2024-08-19T14:21:52.881355+0000","last_undegraded":"2024-08-19T14:21:52.881355+0000","last_fullsized":"2024-08-19T14:21:52.881355+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:19:56.105880+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:19:56.105880+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"368'7454","reported_seq":7530,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-08-19T14:21:53.147177+0000","last_change":"2024-08-19T14:20:29.208992+0000","last_active":"2024-08-19T14:21:53.147177+0000","last_peered":"2024-08-19T14:21:53.147177+0000","last_clean":"2024-08-19T14:21:53.147177+0000","last_became_active":"2024-08-19T14:20:29.208599+0000","last_became_peered":"2024-08-19T14:20:29.208599+0000","last_unstale":"2024-08-19T14:21:53.147177+0000","last_undegraded":"2024-08-19T14:21:53.147177+0000","last_fullsized":"2024-08-19T14:21:53.147177+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":7454,"log_dups_size":0,"ondisk_log_size":7454,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:55.930633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":488505344,"num_objects":7454,"num_object_clones":0,"num_object_copies":22362,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7454,"num_write_kb":477056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,1],"acting":[4,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"70'139","reported_seq":868,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-08-19T14:21:48.852364+0000","last_change":"2024-08-19T14:21:37.951800+0000","last_active":"2024-08-19T14:21:48.852364+0000","last_peered":"2024-08-19T14:21:48.852364+0000","last_clean":"2024-08-19T14:21:48.852364+0000","last_became_active":"2024-08-19T14:18:26.887277+0000","last_became_peered":"2024-08-19T14:18:26.887277+0000","last_unstale":"2024-08-19T14:21:48.852364+0000","last_undegraded":"2024-08-19T14:21:48.852364+0000","last_fullsized":"2024-08-19T14:21:48.852364+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:37.951694+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:21:37.951694+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:03.860882+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,8],"acting":[5,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"368'7383","reported_seq":7459,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-08-19T14:21:53.132034+0000","last_change":"2024-08-19T14:20:29.209096+0000","last_active":"2024-08-19T14:21:53.132034+0000","last_peered":"2024-08-19T14:21:53.132034+0000","last_clean":"2024-08-19T14:21:53.132034+0000","last_became_active":"2024-08-19T14:20:29.208748+0000","last_became_peered":"2024-08-19T14:20:29.208748+0000","last_unstale":"2024-08-19T14:21:53.132034+0000","last_undegraded":"2024-08-19T14:21:53.132034+0000","last_fullsized":"2024-08-19T14:21:53.132034+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":7383,"log_dups_size":0,"ondisk_log_size":7383,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:56.597251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483852288,"num_objects":7383,"num_object_clones":0,"num_object_copies":22149,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7383,"num_write_kb":472512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"70'139","reported_seq":861,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-08-19T14:21:48.893029+0000","last_change":"2024-08-19T14:20:11.733961+0000","last_active":"2024-08-19T14:21:48.893029+0000","last_peered":"2024-08-19T14:21:48.893029+0000","last_clean":"2024-08-19T14:21:48.893029+0000","last_became_active":"2024-08-19T14:18:26.875179+0000","last_became_peered":"2024-08-19T14:18:26.875179+0000","last_unstale":"2024-08-19T14:21:48.893029+0000","last_undegraded":"2024-08-19T14:21:48.893029+0000","last_fullsized":"2024-08-19T14:21:48.893029+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:11.733865+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:11.733865+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.b","version":"366'7399","reported_seq":7546,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-08-19T14:21:51.215110+0000","last_change":"2024-08-19T14:21:39.089676+0000","last_active":"2024-08-19T14:21:51.215110+0000","last_peered":"2024-08-19T14:21:51.215110+0000","last_clean":"2024-08-19T14:21:51.215110+0000","last_became_active":"2024-08-19T14:20:29.207372+0000","last_became_peered":"2024-08-19T14:20:29.207372+0000","last_unstale":"2024-08-19T14:21:51.215110+0000","last_undegraded":"2024-08-19T14:21:51.215110+0000","last_fullsized":"2024-08-19T14:21:51.215110+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"363'6615","last_scrub_stamp":"2024-08-19T14:21:39.089400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:21:39.089400+0000","objects_scrubbed":6552,"log_size":7399,"log_dups_size":0,"ondisk_log_size":7399,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:05.632119+0000","scrub_duration":1907,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484900864,"num_objects":7399,"num_object_clones":0,"num_object_copies":22197,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7399,"num_write_kb":473536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"70'139","reported_seq":527,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-08-19T14:21:52.877388+0000","last_change":"2024-08-19T14:21:44.434481+0000","last_active":"2024-08-19T14:21:52.877388+0000","last_peered":"2024-08-19T14:21:52.877388+0000","last_clean":"2024-08-19T14:21:52.877388+0000","last_became_active":"2024-08-19T14:18:26.884105+0000","last_became_peered":"2024-08-19T14:18:26.884105+0000","last_unstale":"2024-08-19T14:21:52.877388+0000","last_undegraded":"2024-08-19T14:21:52.877388+0000","last_fullsized":"2024-08-19T14:21:52.877388+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:44.434382+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:20:15.321945+0000","last_clean_scrub_stamp":"2024-08-19T14:21:44.434382+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:13.544241+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"366'7423","reported_seq":7494,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-08-19T14:21:49.203764+0000","last_change":"2024-08-19T14:20:29.269723+0000","last_active":"2024-08-19T14:21:49.203764+0000","last_peered":"2024-08-19T14:21:49.203764+0000","last_clean":"2024-08-19T14:21:49.203764+0000","last_became_active":"2024-08-19T14:20:29.269432+0000","last_became_peered":"2024-08-19T14:20:29.269432+0000","last_unstale":"2024-08-19T14:21:49.203764+0000","last_undegraded":"2024-08-19T14:21:49.203764+0000","last_fullsized":"2024-08-19T14:21:49.203764+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":7423,"log_dups_size":0,"ondisk_log_size":7423,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":486473728,"num_objects":7423,"num_object_clones":0,"num_object_copies":22269,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7423,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7423,"num_write_kb":475072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.f","version":"70'139","reported_seq":767,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-08-19T14:21:48.852457+0000","last_change":"2024-08-19T14:21:14.765419+0000","last_active":"2024-08-19T14:21:48.852457+0000","last_peered":"2024-08-19T14:21:48.852457+0000","last_clean":"2024-08-19T14:21:48.852457+0000","last_became_active":"2024-08-19T14:18:18.711708+0000","last_became_peered":"2024-08-19T14:18:18.711708+0000","last_unstale":"2024-08-19T14:21:48.852457+0000","last_undegraded":"2024-08-19T14:21:48.852457+0000","last_fullsized":"2024-08-19T14:21:48.852457+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:14.765340+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:21:14.765340+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:22:24.415219+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"366'7394","reported_seq":7545,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-08-19T14:21:50.782007+0000","last_change":"2024-08-19T14:21:43.974700+0000","last_active":"2024-08-19T14:21:50.782007+0000","last_peered":"2024-08-19T14:21:50.782007+0000","last_clean":"2024-08-19T14:21:50.782007+0000","last_became_active":"2024-08-19T14:20:29.211975+0000","last_became_peered":"2024-08-19T14:20:29.211975+0000","last_unstale":"2024-08-19T14:21:50.782007+0000","last_undegraded":"2024-08-19T14:21:50.782007+0000","last_fullsized":"2024-08-19T14:21:50.782007+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"364'6991","last_scrub_stamp":"2024-08-19T14:21:43.974613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:21:43.974613+0000","objects_scrubbed":6934,"log_size":7394,"log_dups_size":0,"ondisk_log_size":7394,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:07.463188+0000","scrub_duration":1682,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484573184,"num_objects":7394,"num_object_clones":0,"num_object_copies":22182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7394,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7394,"num_write_kb":473216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,3],"acting":[11,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.c","version":"70'139","reported_seq":628,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-08-19T14:21:48.841925+0000","last_change":"2024-08-19T14:21:40.181472+0000","last_active":"2024-08-19T14:21:48.841925+0000","last_peered":"2024-08-19T14:21:48.841925+0000","last_clean":"2024-08-19T14:21:48.841925+0000","last_became_active":"2024-08-19T14:18:26.872942+0000","last_became_peered":"2024-08-19T14:18:26.872942+0000","last_unstale":"2024-08-19T14:21:48.841925+0000","last_undegraded":"2024-08-19T14:21:48.841925+0000","last_fullsized":"2024-08-19T14:21:48.841925+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:40.181356+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:20:07.839076+0000","last_clean_scrub_stamp":"2024-08-19T14:21:40.181356+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:22:52.009381+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,9],"acting":[3,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"366'7376","reported_seq":7448,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-08-19T14:21:50.791142+0000","last_change":"2024-08-19T14:20:29.268318+0000","last_active":"2024-08-19T14:21:50.791142+0000","last_peered":"2024-08-19T14:21:50.791142+0000","last_clean":"2024-08-19T14:21:50.791142+0000","last_became_active":"2024-08-19T14:20:29.267442+0000","last_became_peered":"2024-08-19T14:20:29.267442+0000","last_unstale":"2024-08-19T14:21:50.791142+0000","last_undegraded":"2024-08-19T14:21:50.791142+0000","last_fullsized":"2024-08-19T14:21:50.791142+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":7376,"log_dups_size":0,"ondisk_log_size":7376,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:54.296540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483393536,"num_objects":7376,"num_object_clones":0,"num_object_copies":22128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7376,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7376,"num_write_kb":472064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,9],"acting":[11,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.d","version":"70'139","reported_seq":547,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-08-19T14:21:52.881668+0000","last_change":"2024-08-19T14:21:07.155294+0000","last_active":"2024-08-19T14:21:52.881668+0000","last_peered":"2024-08-19T14:21:52.881668+0000","last_clean":"2024-08-19T14:21:52.881668+0000","last_became_active":"2024-08-19T14:18:26.882002+0000","last_became_peered":"2024-08-19T14:18:26.882002+0000","last_unstale":"2024-08-19T14:21:52.881668+0000","last_undegraded":"2024-08-19T14:21:52.881668+0000","last_fullsized":"2024-08-19T14:21:52.881668+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:07.155157+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:21:07.155157+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:22:11.274708+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"70'139","reported_seq":802,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-08-19T14:21:48.855324+0000","last_change":"2024-08-19T14:21:13.777891+0000","last_active":"2024-08-19T14:21:48.855324+0000","last_peered":"2024-08-19T14:21:48.855324+0000","last_clean":"2024-08-19T14:21:48.855324+0000","last_became_active":"2024-08-19T14:18:18.703594+0000","last_became_peered":"2024-08-19T14:18:18.703594+0000","last_unstale":"2024-08-19T14:21:48.855324+0000","last_undegraded":"2024-08-19T14:21:48.855324+0000","last_fullsized":"2024-08-19T14:21:48.855324+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:13.777777+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:21:13.777777+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:22:38.473560+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"70'139","reported_seq":854,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-08-19T14:21:48.852935+0000","last_change":"2024-08-19T14:20:07.692054+0000","last_active":"2024-08-19T14:21:48.852935+0000","last_peered":"2024-08-19T14:21:48.852935+0000","last_clean":"2024-08-19T14:21:48.852935+0000","last_became_active":"2024-08-19T14:18:18.698334+0000","last_became_peered":"2024-08-19T14:18:18.698334+0000","last_unstale":"2024-08-19T14:21:48.852935+0000","last_undegraded":"2024-08-19T14:21:48.852935+0000","last_fullsized":"2024-08-19T14:21:48.852935+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:07.691932+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:07.691932+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.10","version":"70'139","reported_seq":957,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-08-19T14:21:48.852250+0000","last_change":"2024-08-19T14:21:07.768770+0000","last_active":"2024-08-19T14:21:48.852250+0000","last_peered":"2024-08-19T14:21:48.852250+0000","last_clean":"2024-08-19T14:21:48.852250+0000","last_became_active":"2024-08-19T14:18:18.704562+0000","last_became_peered":"2024-08-19T14:18:18.704562+0000","last_unstale":"2024-08-19T14:21:48.852250+0000","last_undegraded":"2024-08-19T14:21:48.852250+0000","last_fullsized":"2024-08-19T14:21:48.852250+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:07.768697+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:21:07.768697+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:22:10.360289+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,8],"acting":[5,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"70'139","reported_seq":506,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-08-19T14:21:48.860352+0000","last_change":"2024-08-19T14:21:16.071812+0000","last_active":"2024-08-19T14:21:48.860352+0000","last_peered":"2024-08-19T14:21:48.860352+0000","last_clean":"2024-08-19T14:21:48.860352+0000","last_became_active":"2024-08-19T14:18:18.697894+0000","last_became_peered":"2024-08-19T14:18:18.697894+0000","last_unstale":"2024-08-19T14:21:48.860352+0000","last_undegraded":"2024-08-19T14:21:48.860352+0000","last_fullsized":"2024-08-19T14:21:48.860352+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:16.071703+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:21:16.071703+0000","last_clean_scrub_stamp":"2024-08-19T14:21:16.071703+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:22:18.184091+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"70'139","reported_seq":483,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-08-19T14:21:48.853932+0000","last_change":"2024-08-19T14:20:28.205212+0000","last_active":"2024-08-19T14:21:48.853932+0000","last_peered":"2024-08-19T14:21:48.853932+0000","last_clean":"2024-08-19T14:21:48.853932+0000","last_became_active":"2024-08-19T14:20:28.204724+0000","last_became_peered":"2024-08-19T14:20:28.204724+0000","last_unstale":"2024-08-19T14:21:48.853932+0000","last_undegraded":"2024-08-19T14:21:48.853932+0000","last_fullsized":"2024-08-19T14:21:48.853932+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:08.680021+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:08.680021+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"70'139","reported_seq":603,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-08-19T14:21:48.840548+0000","last_change":"2024-08-19T14:20:31.931658+0000","last_active":"2024-08-19T14:21:48.840548+0000","last_peered":"2024-08-19T14:21:48.840548+0000","last_clean":"2024-08-19T14:21:48.840548+0000","last_became_active":"2024-08-19T14:18:31.901556+0000","last_became_peered":"2024-08-19T14:18:31.901556+0000","last_unstale":"2024-08-19T14:21:48.840548+0000","last_undegraded":"2024-08-19T14:21:48.840548+0000","last_fullsized":"2024-08-19T14:21:48.840548+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:31.931562+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:20:31.931562+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,0],"acting":[8,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.14","version":"70'139","reported_seq":616,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-08-19T14:21:52.881461+0000","last_change":"2024-08-19T14:19:55.130518+0000","last_active":"2024-08-19T14:21:52.881461+0000","last_peered":"2024-08-19T14:21:52.881461+0000","last_clean":"2024-08-19T14:21:52.881461+0000","last_became_active":"2024-08-19T14:18:18.703926+0000","last_became_peered":"2024-08-19T14:18:18.703926+0000","last_unstale":"2024-08-19T14:21:52.881461+0000","last_undegraded":"2024-08-19T14:21:52.881461+0000","last_fullsized":"2024-08-19T14:21:52.881461+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:19:55.130420+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:19:55.130420+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,1],"acting":[7,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.15","version":"70'139","reported_seq":503,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-08-19T14:21:52.881580+0000","last_change":"2024-08-19T14:20:09.181046+0000","last_active":"2024-08-19T14:21:52.881580+0000","last_peered":"2024-08-19T14:21:52.881580+0000","last_clean":"2024-08-19T14:21:52.881580+0000","last_became_active":"2024-08-19T14:18:31.912496+0000","last_became_peered":"2024-08-19T14:18:31.912496+0000","last_unstale":"2024-08-19T14:21:52.881580+0000","last_undegraded":"2024-08-19T14:21:52.881580+0000","last_fullsized":"2024-08-19T14:21:52.881580+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:09.180923+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:20:09.180923+0000","last_clean_scrub_stamp":"2024-08-19T14:20:09.180923+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"70'139","reported_seq":489,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-08-19T14:21:52.877134+0000","last_change":"2024-08-19T14:21:40.430835+0000","last_active":"2024-08-19T14:21:52.877134+0000","last_peered":"2024-08-19T14:21:52.877134+0000","last_clean":"2024-08-19T14:21:52.877134+0000","last_became_active":"2024-08-19T14:18:31.901229+0000","last_became_peered":"2024-08-19T14:18:31.901229+0000","last_unstale":"2024-08-19T14:21:52.877134+0000","last_undegraded":"2024-08-19T14:21:52.877134+0000","last_fullsized":"2024-08-19T14:21:52.877134+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:40.430714+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:21:40.430714+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:04.072653+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1b","version":"70'139","reported_seq":837,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-08-19T14:21:48.840493+0000","last_change":"2024-08-19T14:21:48.693864+0000","last_active":"2024-08-19T14:21:48.840493+0000","last_peered":"2024-08-19T14:21:48.840493+0000","last_clean":"2024-08-19T14:21:48.840493+0000","last_became_active":"2024-08-19T14:18:31.901152+0000","last_became_peered":"2024-08-19T14:18:31.901152+0000","last_unstale":"2024-08-19T14:21:48.840493+0000","last_undegraded":"2024-08-19T14:21:48.840493+0000","last_fullsized":"2024-08-19T14:21:48.840493+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:48.693761+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:21:48.693761+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:07.571130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.18","version":"70'139","reported_seq":873,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-08-19T14:21:48.852246+0000","last_change":"2024-08-19T14:20:36.510554+0000","last_active":"2024-08-19T14:21:48.852246+0000","last_peered":"2024-08-19T14:21:48.852246+0000","last_clean":"2024-08-19T14:21:48.852246+0000","last_became_active":"2024-08-19T14:18:31.902735+0000","last_became_peered":"2024-08-19T14:18:31.902735+0000","last_unstale":"2024-08-19T14:21:48.852246+0000","last_undegraded":"2024-08-19T14:21:48.852246+0000","last_fullsized":"2024-08-19T14:21:48.852246+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:36.510442+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:36.510442+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:53.706735+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.19","version":"70'139","reported_seq":829,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-08-19T14:21:48.860420+0000","last_change":"2024-08-19T14:21:40.219949+0000","last_active":"2024-08-19T14:21:48.860420+0000","last_peered":"2024-08-19T14:21:48.860420+0000","last_clean":"2024-08-19T14:21:48.860420+0000","last_became_active":"2024-08-19T14:18:31.902547+0000","last_became_peered":"2024-08-19T14:18:31.902547+0000","last_unstale":"2024-08-19T14:21:48.860420+0000","last_undegraded":"2024-08-19T14:21:48.860420+0000","last_fullsized":"2024-08-19T14:21:48.860420+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:40.219830+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:21:40.219830+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:05.414864+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1e","version":"70'139","reported_seq":479,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-08-19T14:21:48.892648+0000","last_change":"2024-08-19T14:20:18.771836+0000","last_active":"2024-08-19T14:21:48.892648+0000","last_peered":"2024-08-19T14:21:48.892648+0000","last_clean":"2024-08-19T14:21:48.892648+0000","last_became_active":"2024-08-19T14:18:31.902546+0000","last_became_peered":"2024-08-19T14:18:31.902546+0000","last_unstale":"2024-08-19T14:21:48.892648+0000","last_undegraded":"2024-08-19T14:21:48.892648+0000","last_fullsized":"2024-08-19T14:21:48.892648+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:18.771704+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:18.771704+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1c","version":"70'139","reported_seq":595,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-08-19T14:21:48.840592+0000","last_change":"2024-08-19T14:21:38.753663+0000","last_active":"2024-08-19T14:21:48.840592+0000","last_peered":"2024-08-19T14:21:48.840592+0000","last_clean":"2024-08-19T14:21:48.840592+0000","last_became_active":"2024-08-19T14:18:31.902494+0000","last_became_peered":"2024-08-19T14:18:31.902494+0000","last_unstale":"2024-08-19T14:21:48.840592+0000","last_undegraded":"2024-08-19T14:21:48.840592+0000","last_fullsized":"2024-08-19T14:21:48.840592+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:38.753553+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:21:38.753553+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:22:46.720742+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,11],"acting":[8,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1d","version":"70'139","reported_seq":509,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-08-19T14:21:48.850782+0000","last_change":"2024-08-19T14:20:17.725615+0000","last_active":"2024-08-19T14:21:48.850782+0000","last_peered":"2024-08-19T14:21:48.850782+0000","last_clean":"2024-08-19T14:21:48.850782+0000","last_became_active":"2024-08-19T14:18:31.901183+0000","last_became_peered":"2024-08-19T14:18:31.901183+0000","last_unstale":"2024-08-19T14:21:48.850782+0000","last_undegraded":"2024-08-19T14:21:48.850782+0000","last_fullsized":"2024-08-19T14:21:48.850782+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:17.725477+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:17.725477+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,5],"acting":[1,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":7751860224,"num_objects":118284,"num_object_clones":0,"num_object_copies":354852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3037,"num_objects_unfound":0,"num_objects_dirty":118284,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":118284,"num_write_kb":7570176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1103,"num_bytes_recovered":72286208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":23420731392,"data_stored":23420731392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118284,"ondisk_log_size":118284,"up":48,"acting":47,"num_store_stats":9},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1802240,"data_stored":1788032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187582,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3718984,"kb_used_data":3304636,"kb_used_omap":1,"kb_used_meta":414334,"kb_avail":90026168,"statfs":{"total":95995035648,"available":92186796032,"internally_reserved":0,"allocated":3383947264,"data_stored":3382256608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":424278468},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":343597383812,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":74,"seq":317827580040,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2193616,"kb_used_data":1743176,"kb_used_omap":1,"kb_used_meta":450430,"kb_avail":91551536,"statfs":{"total":95995035648,"available":93748772864,"internally_reserved":0,"allocated":1785012224,"data_stored":1783317392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":461240769},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874381,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":839304,"kb_used_data":469372,"kb_used_omap":1,"kb_used_meta":369918,"kb_avail":92905848,"statfs":{"total":95995035648,"available":95135588352,"internally_reserved":0,"allocated":480636928,"data_stored":478946272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":378796485},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":57,"seq":244813136016,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4308780,"kb_used_data":3866652,"kb_used_omap":1,"kb_used_meta":442110,"kb_avail":89436372,"statfs":{"total":95995035648,"available":91582844928,"internally_reserved":0,"allocated":3959451648,"data_stored":3957749088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":452721098},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":206158430355,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691992,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2049292,"kb_used_data":1707140,"kb_used_omap":1,"kb_used_meta":342142,"kb_avail":91695860,"statfs":{"total":95995035648,"available":93896560640,"internally_reserved":0,"allocated":1748111360,"data_stored":1746417152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":350353861},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986333,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2851600,"kb_used_data":2387272,"kb_used_omap":1,"kb_used_meta":464318,"kb_avail":90893552,"statfs":{"total":95995035648,"available":93074997248,"internally_reserved":0,"allocated":2444566528,"data_stored":2442871696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":475462085},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149858,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3208712,"kb_used_data":2784636,"kb_used_omap":1,"kb_used_meta":424062,"kb_avail":90536440,"statfs":{"total":95995035648,"available":92709314560,"internally_reserved":0,"allocated":2851467264,"data_stored":2849776608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":434239943},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411496,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575019,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5144896,"kb_used_data":4640312,"kb_used_omap":1,"kb_used_meta":504574,"kb_avail":88600256,"statfs":{"total":95995035648,"available":90726662144,"internally_reserved":0,"allocated":4751679488,"data_stored":4749989506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":516684229},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738544,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2266308,"kb_used_data":1901948,"kb_used_omap":1,"kb_used_meta":364350,"kb_avail":91478844,"statfs":{"total":95995035648,"available":93674336256,"internally_reserved":0,"allocated":1947594752,"data_stored":1945904096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":373094853},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1943928832,"data_stored":1943928832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4783079424,"data_stored":4783079424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2859597824,"data_stored":2859597824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2451767296,"data_stored":2451767296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1752694784,"data_stored":1752694784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3965190144,"data_stored":3965190144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":477429760,"data_stored":477429760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1792344064,"data_stored":1792344064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":3394699264,"data_stored":3394699264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T14:21:54.885 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph status --format=json 2024-08-19T14:21:54.887 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:21:54.916 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:54 smithi045 ceph-mon[20143]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:21:54.916 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:54 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:21:54.916 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:54 smithi045 ceph-mon[20143]: osdmap e369: 12 total, 12 up, 9 in 2024-08-19T14:21:54.917 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:54 smithi045 ceph-mon[20143]: pgmap v829: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 7.2 GiB data, 25 GiB used, 779 GiB / 805 GiB avail; 61 MiB/s wr, 977 op/s; 3037/354858 objects misplaced (0.856%); 1.8 MiB/s, 28 objects/s recovering 2024-08-19T14:21:54.917 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:54 smithi045 ceph-mon[20143]: from='client.25486 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:21:54.925 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:54.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:54.926 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:21:55.027 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:21:55.042 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:21:55.051 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 81 16 122465 122449 94.4742 73.5 0.0135029 0.0105801 2024-08-19T14:21:55.075 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:55.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:55.076 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:21:55.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:54 smithi031 ceph-mon[16336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:21:55.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:54 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:21:55.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:54 smithi031 ceph-mon[16336]: osdmap e369: 12 total, 12 up, 9 in 2024-08-19T14:21:55.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:54 smithi031 ceph-mon[16336]: pgmap v829: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 7.2 GiB data, 25 GiB used, 779 GiB / 805 GiB avail; 61 MiB/s wr, 977 op/s; 3037/354858 objects misplaced (0.856%); 1.8 MiB/s, 28 objects/s recovering 2024-08-19T14:21:55.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:54 smithi031 ceph-mon[16336]: from='client.25486 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:21:55.176 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:21:55.225 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:55.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:55.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:21:55.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:54 smithi032 ceph-mon[20004]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:21:55.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:54 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:21:55.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:54 smithi032 ceph-mon[20004]: osdmap e369: 12 total, 12 up, 9 in 2024-08-19T14:21:55.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:54 smithi032 ceph-mon[20004]: pgmap v829: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 7.2 GiB data, 25 GiB used, 779 GiB / 805 GiB avail; 61 MiB/s wr, 977 op/s; 3037/354858 objects misplaced (0.856%); 1.8 MiB/s, 28 objects/s recovering 2024-08-19T14:21:55.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:54 smithi032 ceph-mon[20004]: from='client.25486 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:21:55.327 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:21:55.355 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:55.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:55.356 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:21:55.457 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:21:55.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:55.486 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:55.486 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:21:55.587 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:21:55.595 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:21:55.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:55.649 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:55.649 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:21:55.684 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:21:55.685 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:21:55.750 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:21:55.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:55.785 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:55.785 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:21:55.886 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:21:55.919 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:55.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:55.920 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:21:56.021 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:21:56.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:55 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:21:56.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:55 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:21:56.051 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 82 16 123576 123560 94.1688 69.4375 0.00653653 0.0106149 2024-08-19T14:21:56.051 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:56.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:56.052 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:21:56.153 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:21:56.155 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:55 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:21:56.155 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:55 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:21:56.185 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:56.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:56.187 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:21:56.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:55 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:21:56.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:55 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:21:56.288 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:21:56.314 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:56.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:56.315 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:21:56.416 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:21:56.432 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-08-19T14:21:56.432 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:21:56.433 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:21:56.436 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:21:56.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:56.440 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:56.440 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:21:56.451 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:21:56.451 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:21:56.472 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:21:56.472 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:21:56.492 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:21:56.492 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:21:56.510 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:21:56.510 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:21:56.535 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:21:56.535 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:21:56.541 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:21:56.571 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:21:56.571 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:21:56.580 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:56.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:56.581 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:21:56.608 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:21:56.609 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:21:56.644 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:21:56.644 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:21:56.681 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:21:56.681 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:21:56.681 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:21:56.708 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:21:56.709 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:21:56.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:56.723 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:56.724 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:21:56.744 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:21:56.745 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:21:56.783 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:21:56.824 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:21:56.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:56.846 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:56.846 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:21:56.947 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:21:56.976 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:56.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:56.977 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:21:57.051 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 83 16 124381 124365 93.6404 50.3125 0.0097816 0.0106732 2024-08-19T14:21:57.077 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:21:57.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:57.108 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:57.108 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:21:57.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:56 smithi031 ceph-mon[16336]: osdmap e370: 12 total, 12 up, 9 in 2024-08-19T14:21:57.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:56 smithi031 ceph-mon[16336]: pgmap v831: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 7.4 GiB data, 26 GiB used, 779 GiB / 805 GiB avail; 70 MiB/s wr, 1.12k op/s; 2890/365067 objects misplaced (0.792%); 2.3 MiB/s, 36 objects/s recovering 2024-08-19T14:21:57.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:56 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1382963652' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:21:57.210 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:21:57.231 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:57.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:57.232 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:21:57.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:56 smithi032 ceph-mon[20004]: osdmap e370: 12 total, 12 up, 9 in 2024-08-19T14:21:57.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:56 smithi032 ceph-mon[20004]: pgmap v831: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 7.4 GiB data, 26 GiB used, 779 GiB / 805 GiB avail; 70 MiB/s wr, 1.12k op/s; 2890/365067 objects misplaced (0.792%); 2.3 MiB/s, 36 objects/s recovering 2024-08-19T14:21:57.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:56 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1382963652' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:21:57.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:56 smithi045 ceph-mon[20143]: osdmap e370: 12 total, 12 up, 9 in 2024-08-19T14:21:57.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:56 smithi045 ceph-mon[20143]: pgmap v831: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 7.4 GiB data, 26 GiB used, 779 GiB / 805 GiB avail; 70 MiB/s wr, 1.12k op/s; 2890/365067 objects misplaced (0.792%); 2.3 MiB/s, 36 objects/s recovering 2024-08-19T14:21:57.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:56 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1382963652' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:21:57.333 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:21:57.367 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:57.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:57.368 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:21:57.469 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:21:57.503 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:57.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:57.504 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:21:57.590 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-19T14:21:57.605 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:21:57.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:57.639 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:57.639 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:21:57.740 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:21:57.765 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:57.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:57.766 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:21:57.777 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:21:57.777 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [], 2024-08-19T14:21:57.777 INFO:teuthology.orchestra.run.smithi045.stdout: "num_ops": 0 2024-08-19T14:21:57.777 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:21:57.866 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:21:57.895 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:57.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:57.896 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:21:57.997 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:21:58.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:58.048 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:58.049 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:21:58.051 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 84 16 125278 125262 93.193 56.0625 0.00744606 0.010725 2024-08-19T14:21:58.149 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:21:58.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:58.175 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:58.176 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:21:58.276 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:21:58.278 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:21:58.278 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"ed7d1c6e-5e33-11ef-bccf-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub,nodeep-scrub flag(s) set","count":20},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":895,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":370,"num_osds":12,"num_up_osds":12,"osd_up_since":1724076699,"num_in_osds":9,"osd_in_since":1724076998,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":46},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":47,"num_pools":2,"num_objects":121993,"data_bytes":7995392544,"bytes_used":27900555264,"bytes_avail":836054765568,"bytes_total":863955320832,"misplaced_objects":2890,"misplaced_total":365979,"misplaced_ratio":0.0078966279485981443,"recovering_objects_per_sec":24,"recovering_bytes_per_sec":1605297,"recovering_keys_per_sec":0,"num_objects_recovered":147,"num_bytes_recovered":9633792,"num_keys_recovered":0,"write_bytes_sec":52756410,"read_op_per_sec":0,"write_op_per_sec":804},"fsmap":{"epoch":1,"btime":"2024-08-19T14:05:22:841940+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.31:8443/"}},"servicemap":{"epoch":20,"modified":"2024-08-19T14:20:27.873686+0000","services":{}},"progress_events":{}} 2024-08-19T14:21:58.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:58.308 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:58.309 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:21:58.410 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:21:58.441 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:58.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:58.442 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:21:58.505 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-08-19T14:21:58.542 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:21:58.575 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:58.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:58.576 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:21:58.678 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:21:58.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:58.728 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:58.728 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:21:58.830 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:21:58.852 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:21:58.858 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:58.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:58.863 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:21:58.964 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:21:58.996 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:58.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:58.997 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:21:59.051 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 85 16 126341 126325 92.8782 66.4375 0.00494652 0.0107596 2024-08-19T14:21:59.098 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:21:59.103 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-08-19T14:21:59.104 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-19T14:21:59.123 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:59.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:59.124 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:21:59.147 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:21:59.225 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:21:59.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:59.250 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:59.250 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:21:59.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:58 smithi032 ceph-mon[20004]: pgmap v832: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 7.4 GiB data, 26 GiB used, 779 GiB / 805 GiB avail; 50 MiB/s wr, 804 op/s; 2890/365979 objects misplaced (0.790%); 1.5 MiB/s, 24 objects/s recovering 2024-08-19T14:21:59.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:21:58 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/209175218' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-08-19T14:21:59.259 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:58 smithi045 ceph-mon[20143]: pgmap v832: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 7.4 GiB data, 26 GiB used, 779 GiB / 805 GiB avail; 50 MiB/s wr, 804 op/s; 2890/365979 objects misplaced (0.790%); 1.5 MiB/s, 24 objects/s recovering 2024-08-19T14:21:59.259 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:21:58 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/209175218' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-08-19T14:21:59.293 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:58 smithi031 ceph-mon[16336]: pgmap v832: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 7.4 GiB data, 26 GiB used, 779 GiB / 805 GiB avail; 50 MiB/s wr, 804 op/s; 2890/365979 objects misplaced (0.790%); 1.5 MiB/s, 24 objects/s recovering 2024-08-19T14:21:59.293 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:21:58 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/209175218' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-08-19T14:21:59.352 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:21:59.382 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:59.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:59.383 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:21:59.397 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:21:59.397 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [], 2024-08-19T14:21:59.397 INFO:teuthology.orchestra.run.smithi045.stdout: "complaint_time": 30, 2024-08-19T14:21:59.397 INFO:teuthology.orchestra.run.smithi045.stdout: "num_blocked_ops": 0 2024-08-19T14:21:59.398 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:21:59.485 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:21:59.510 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:21:59.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:59.512 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:21:59.613 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:21:59.641 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:59.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:59.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:21:59.743 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:21:59.769 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:21:59.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:59.804 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:21:59.804 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:21:59.904 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:21:59.931 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:21:59.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:21:59.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:22:00.033 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:22:00.051 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 86 16 127314 127298 92.5052 60.8125 0.0115636 0.0108056 2024-08-19T14:22:00.060 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:00.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:00.062 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:22:00.163 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:22:00.202 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:00.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:00.204 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:22:00.243 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-08-19T14:22:00.305 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:22:00.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:00.344 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:00.344 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:22:00.446 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:22:00.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:00.477 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:00.478 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:22:00.578 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:22:00.599 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:22:00.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:00.617 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:00.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:22:00.640 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:22:00.643 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-08-19T14:22:00.655 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:22:00.718 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:22:00.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:00.753 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:00.753 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:22:00.855 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:22:00.887 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:00.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:00.888 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:22:00.978 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is unset 2024-08-19T14:22:00.989 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:22:01.012 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:01.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:01.013 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:22:01.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:00 smithi045 ceph-mon[20143]: pgmap v833: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 7.6 GiB data, 27 GiB used, 778 GiB / 805 GiB avail; 70 MiB/s wr, 1.13k op/s; 2731/375231 objects misplaced (0.728%); 2.7 MiB/s, 43 objects/s recovering 2024-08-19T14:22:01.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:00 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2657441848' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:22:01.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:00 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:22:01.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:00 smithi045 ceph-mon[20143]: from='client.25544 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:22:01.051 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 87 16 128310 128294 92.1574 62.25 0.0186659 0.0108469 2024-08-19T14:22:01.114 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:22:01.148 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:01.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:01.149 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:22:01.181 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:22:01.182 INFO:teuthology.orchestra.run.smithi045.stdout: "size": 20, 2024-08-19T14:22:01.182 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 600, 2024-08-19T14:22:01.182 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [ 2024-08-19T14:22:01.182 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.182 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:14 2.4 e106/102 2:291e9767:::benchmark_data_smithi045_46415_object13:head v 106'1, mlcod=0'0)", 2024-08-19T14:22:01.182 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:12:49.835194+0000", 2024-08-19T14:22:01.182 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 551.34297991899996, 2024-08-19T14:22:01.182 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.045178087999999998, 2024-08-19T14:22:01.182 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:22:01.182 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:22:01.183 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:22:01.183 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:22:01.183 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.183 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:22:01.183 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:49.835194+0000", 2024-08-19T14:22:01.183 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:01.183 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.183 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.183 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:22:01.183 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:49.835194+0000", 2024-08-19T14:22:01.183 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:01.183 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.183 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.184 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:22:01.184 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:49.835194+0000", 2024-08-19T14:22:01.184 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.7500000000000001e-07 2024-08-19T14:22:01.184 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.184 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.184 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:22:01.184 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:49.879644+0000", 2024-08-19T14:22:01.184 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.044449610000000001 2024-08-19T14:22:01.184 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.184 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.184 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:22:01.184 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:49.879665+0000", 2024-08-19T14:22:01.184 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1458999999999999e-05 2024-08-19T14:22:01.184 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.185 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.185 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:22:01.185 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:49.879706+0000", 2024-08-19T14:22:01.185 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.0748999999999999e-05 2024-08-19T14:22:01.185 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.185 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.185 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:22:01.185 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:49.879782+0000", 2024-08-19T14:22:01.185 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.6445000000000004e-05 2024-08-19T14:22:01.185 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.185 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.186 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:22:01.186 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:49.879807+0000", 2024-08-19T14:22:01.186 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.4765000000000001e-05 2024-08-19T14:22:01.186 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.186 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.186 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:22:01.186 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:49.880329+0000", 2024-08-19T14:22:01.186 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00052147599999999999 2024-08-19T14:22:01.186 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.186 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.186 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:22:01.186 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:49.880372+0000", 2024-08-19T14:22:01.186 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.3309000000000002e-05 2024-08-19T14:22:01.187 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:01.187 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:22:01.187 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:01.187 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.187 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.187 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:12156 2.4 e107/102 2:28b3607b:::benchmark_data_smithi045_46415_object12155:head v 107'793, mlcod=107'791)", 2024-08-19T14:22:01.187 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:12:53.026834+0000", 2024-08-19T14:22:01.187 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 548.15133945399998, 2024-08-19T14:22:01.187 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0078985359999999994, 2024-08-19T14:22:01.187 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:22:01.187 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:22:01.187 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:22:01.187 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:22:01.188 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.188 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:22:01.188 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:53.026834+0000", 2024-08-19T14:22:01.188 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:01.188 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.188 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.188 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:22:01.188 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:53.026834+0000", 2024-08-19T14:22:01.188 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:01.188 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.188 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.188 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:22:01.188 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:53.026835+0000", 2024-08-19T14:22:01.189 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.5199999999999996e-07 2024-08-19T14:22:01.189 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.189 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.189 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:22:01.189 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:53.034134+0000", 2024-08-19T14:22:01.189 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0072991369999999998 2024-08-19T14:22:01.189 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.189 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.189 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:22:01.189 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:53.034144+0000", 2024-08-19T14:22:01.189 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0370000000000001e-05 2024-08-19T14:22:01.189 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.189 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.189 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:22:01.190 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:53.034149+0000", 2024-08-19T14:22:01.190 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.8130000000000003e-06 2024-08-19T14:22:01.190 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.190 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.190 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:22:01.190 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:53.034196+0000", 2024-08-19T14:22:01.190 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.7159e-05 2024-08-19T14:22:01.190 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.190 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.190 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:22:01.190 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:53.034211+0000", 2024-08-19T14:22:01.190 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4429e-05 2024-08-19T14:22:01.191 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.191 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.191 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:22:01.191 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:53.034701+0000", 2024-08-19T14:22:01.191 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00049043000000000003 2024-08-19T14:22:01.191 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.191 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.191 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:22:01.191 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:12:53.034733+0000", 2024-08-19T14:22:01.191 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.1545999999999999e-05 2024-08-19T14:22:01.191 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:01.191 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:22:01.191 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:01.192 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.192 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.192 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:38741 2.4 e112/102 2:2ddcb16b:::benchmark_data_smithi045_46415_object38740:head v 112'2448, mlcod=112'2444)", 2024-08-19T14:22:01.192 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:02.435487+0000", 2024-08-19T14:22:01.192 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 538.74268702699999, 2024-08-19T14:22:01.192 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0060987159999999997, 2024-08-19T14:22:01.192 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:22:01.192 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:22:01.192 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:22:01.192 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:22:01.192 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.192 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:22:01.192 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.435487+0000", 2024-08-19T14:22:01.192 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:01.193 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.193 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.193 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:22:01.193 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.435487+0000", 2024-08-19T14:22:01.193 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:01.193 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.193 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.193 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:22:01.193 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.435487+0000", 2024-08-19T14:22:01.193 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2699999999999999e-07 2024-08-19T14:22:01.193 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.193 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.193 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:22:01.194 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.435514+0000", 2024-08-19T14:22:01.194 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.6457000000000001e-05 2024-08-19T14:22:01.194 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.194 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.194 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:22:01.194 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.435524+0000", 2024-08-19T14:22:01.194 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0299000000000001e-05 2024-08-19T14:22:01.194 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.194 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.194 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:22:01.195 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.435531+0000", 2024-08-19T14:22:01.195 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.6529999999999997e-06 2024-08-19T14:22:01.195 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.195 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.195 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:22:01.195 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.435603+0000", 2024-08-19T14:22:01.195 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.1963000000000002e-05 2024-08-19T14:22:01.195 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.195 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.195 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:22:01.195 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.435622+0000", 2024-08-19T14:22:01.195 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8926000000000001e-05 2024-08-19T14:22:01.195 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.195 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.196 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:22:01.196 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.441541+0000", 2024-08-19T14:22:01.196 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0059191479999999999 2024-08-19T14:22:01.196 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.196 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.196 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:22:01.196 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.441585+0000", 2024-08-19T14:22:01.196 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.4743000000000002e-05 2024-08-19T14:22:01.196 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:01.196 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:22:01.196 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:01.196 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.197 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.197 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:40033 2.0 e113/102 2:0512d183:::benchmark_data_smithi045_46415_object40032:head v 113'2436, mlcod=113'2434)", 2024-08-19T14:22:01.197 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:02.895855+0000", 2024-08-19T14:22:01.197 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 538.28231833400002, 2024-08-19T14:22:01.197 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0095757759999999994, 2024-08-19T14:22:01.197 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:22:01.197 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:22:01.197 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:22:01.197 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:22:01.197 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.197 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:22:01.197 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.895855+0000", 2024-08-19T14:22:01.197 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:01.197 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.198 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.198 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:22:01.198 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.895855+0000", 2024-08-19T14:22:01.198 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:01.198 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.198 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.198 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:22:01.198 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.895856+0000", 2024-08-19T14:22:01.198 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.2099999999999995e-07 2024-08-19T14:22:01.198 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.198 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.198 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:22:01.199 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.895922+0000", 2024-08-19T14:22:01.199 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.5188e-05 2024-08-19T14:22:01.199 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.199 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.199 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:22:01.199 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.895935+0000", 2024-08-19T14:22:01.199 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3601000000000001e-05 2024-08-19T14:22:01.199 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.199 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.199 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:22:01.199 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.895943+0000", 2024-08-19T14:22:01.199 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.9170000000000006e-06 2024-08-19T14:22:01.199 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.200 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.200 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:22:01.200 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.896012+0000", 2024-08-19T14:22:01.200 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.9053000000000004e-05 2024-08-19T14:22:01.200 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.200 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.200 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:22:01.200 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.896035+0000", 2024-08-19T14:22:01.200 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2943999999999999e-05 2024-08-19T14:22:01.200 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.200 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.200 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:22:01.201 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.905344+0000", 2024-08-19T14:22:01.201 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0093086450000000008 2024-08-19T14:22:01.201 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.201 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.201 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:22:01.201 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.905431+0000", 2024-08-19T14:22:01.201 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.7507000000000001e-05 2024-08-19T14:22:01.201 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:01.201 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:22:01.201 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:01.201 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.201 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.201 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:40035 2.0 e113/102 2:0960ca99:::benchmark_data_smithi045_46415_object40034:head v 113'2437, mlcod=113'2434)", 2024-08-19T14:22:01.202 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:02.897909+0000", 2024-08-19T14:22:01.202 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 538.28026525500002, 2024-08-19T14:22:01.202 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0076402179999999998, 2024-08-19T14:22:01.202 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:22:01.202 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:22:01.202 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:22:01.202 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:22:01.202 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.202 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:22:01.202 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.897909+0000", 2024-08-19T14:22:01.202 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:01.202 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.203 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.203 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:22:01.203 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.897909+0000", 2024-08-19T14:22:01.203 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:01.203 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.203 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.203 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:22:01.203 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.897909+0000", 2024-08-19T14:22:01.203 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.33e-07 2024-08-19T14:22:01.203 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.203 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.203 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:22:01.203 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.897950+0000", 2024-08-19T14:22:01.203 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.0683000000000003e-05 2024-08-19T14:22:01.203 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.204 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.204 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:22:01.204 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.897966+0000", 2024-08-19T14:22:01.204 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6209999999999999e-05 2024-08-19T14:22:01.204 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.204 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.204 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:22:01.204 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.897974+0000", 2024-08-19T14:22:01.204 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4569999999999999e-06 2024-08-19T14:22:01.204 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.204 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.204 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:22:01.204 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.898035+0000", 2024-08-19T14:22:01.204 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.1033000000000003e-05 2024-08-19T14:22:01.205 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.205 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.205 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:22:01.205 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.898068+0000", 2024-08-19T14:22:01.205 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.3593000000000002e-05 2024-08-19T14:22:01.205 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.205 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.205 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:22:01.205 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.905435+0000", 2024-08-19T14:22:01.205 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0073662830000000004 2024-08-19T14:22:01.205 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.205 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.205 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:22:01.205 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.905549+0000", 2024-08-19T14:22:01.206 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000114026 2024-08-19T14:22:01.206 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:01.206 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:22:01.206 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:01.206 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.206 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.206 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:40069 2.0 e113/102 2:03e60c7a:::benchmark_data_smithi045_46415_object40068:head v 113'2440, mlcod=113'2436)", 2024-08-19T14:22:01.206 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:02.908487+0000", 2024-08-19T14:22:01.206 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 538.269686482, 2024-08-19T14:22:01.206 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0062491439999999999, 2024-08-19T14:22:01.206 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:22:01.206 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:22:01.206 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:22:01.207 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:22:01.207 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.207 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:22:01.207 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.908487+0000", 2024-08-19T14:22:01.207 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:01.207 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.207 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.207 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:22:01.207 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.908487+0000", 2024-08-19T14:22:01.207 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:01.207 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.207 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.207 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:22:01.208 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.908488+0000", 2024-08-19T14:22:01.208 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.8999999999999995e-07 2024-08-19T14:22:01.208 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.208 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.208 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:22:01.208 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.908517+0000", 2024-08-19T14:22:01.208 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.8643999999999999e-05 2024-08-19T14:22:01.208 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.208 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.208 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:22:01.208 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.908533+0000", 2024-08-19T14:22:01.208 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5770999999999999e-05 2024-08-19T14:22:01.208 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.208 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.209 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:22:01.209 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.908540+0000", 2024-08-19T14:22:01.209 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4000000000000003e-06 2024-08-19T14:22:01.209 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.209 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.209 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:22:01.209 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.908593+0000", 2024-08-19T14:22:01.209 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.3146999999999998e-05 2024-08-19T14:22:01.209 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.209 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.209 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:22:01.209 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.908614+0000", 2024-08-19T14:22:01.209 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0900999999999999e-05 2024-08-19T14:22:01.209 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.209 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.210 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:22:01.210 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.914692+0000", 2024-08-19T14:22:01.210 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0060776399999999996 2024-08-19T14:22:01.210 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.210 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.210 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:22:01.210 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:02.914736+0000", 2024-08-19T14:22:01.210 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.4851000000000001e-05 2024-08-19T14:22:01.210 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:01.210 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:22:01.210 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:01.210 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.210 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.211 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:43484 2.4 e114/102 2:285d10cd:::benchmark_data_smithi045_46415_object43483:head v 114'2732, mlcod=114'2729)", 2024-08-19T14:22:01.211 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:04.598813+0000", 2024-08-19T14:22:01.211 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 536.579360597, 2024-08-19T14:22:01.211 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0062693649999999998, 2024-08-19T14:22:01.211 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:22:01.211 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:22:01.211 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:22:01.211 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:22:01.211 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.211 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:22:01.212 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:04.598813+0000", 2024-08-19T14:22:01.212 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:01.212 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.212 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.212 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:22:01.212 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:04.598813+0000", 2024-08-19T14:22:01.212 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:01.213 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.213 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.213 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:22:01.213 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:04.598814+0000", 2024-08-19T14:22:01.213 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.9699999999999994e-07 2024-08-19T14:22:01.213 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.213 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.213 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:22:01.213 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:04.598850+0000", 2024-08-19T14:22:01.213 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.5865000000000003e-05 2024-08-19T14:22:01.213 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.213 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.214 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:22:01.214 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:04.598861+0000", 2024-08-19T14:22:01.214 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1284000000000001e-05 2024-08-19T14:22:01.214 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.214 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.214 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:22:01.214 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:04.598870+0000", 2024-08-19T14:22:01.214 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.9479999999999997e-06 2024-08-19T14:22:01.214 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.214 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.214 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:22:01.214 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:04.598935+0000", 2024-08-19T14:22:01.214 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.4364999999999998e-05 2024-08-19T14:22:01.215 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.215 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.215 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:22:01.215 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:04.598955+0000", 2024-08-19T14:22:01.215 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0849999999999999e-05 2024-08-19T14:22:01.215 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.215 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.215 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:22:01.215 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:04.604957+0000", 2024-08-19T14:22:01.215 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0060020000000000004 2024-08-19T14:22:01.215 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.215 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.215 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:22:01.215 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:04.605083+0000", 2024-08-19T14:22:01.216 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012505599999999999 2024-08-19T14:22:01.216 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:01.216 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:22:01.216 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:01.216 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.216 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.216 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:46495 2.4 e116/102 2:27583565:::benchmark_data_smithi045_46415_object46494:head v 116'2919, mlcod=116'2914)", 2024-08-19T14:22:01.216 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:06.037606+0000", 2024-08-19T14:22:01.216 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 535.14056820099995, 2024-08-19T14:22:01.216 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0079993879999999996, 2024-08-19T14:22:01.216 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:22:01.216 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:22:01.216 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:22:01.216 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:22:01.217 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.217 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:22:01.217 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:06.037606+0000", 2024-08-19T14:22:01.217 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:01.217 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.217 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.217 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:22:01.217 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:06.037606+0000", 2024-08-19T14:22:01.218 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:01.218 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.218 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.218 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:22:01.218 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:06.037606+0000", 2024-08-19T14:22:01.218 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2300000000000001e-07 2024-08-19T14:22:01.218 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.218 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.218 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:22:01.218 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:06.043150+0000", 2024-08-19T14:22:01.218 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0055434869999999997 2024-08-19T14:22:01.218 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.219 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.219 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:22:01.219 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:06.043160+0000", 2024-08-19T14:22:01.219 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0088999999999999e-05 2024-08-19T14:22:01.219 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.219 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.219 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:22:01.219 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:06.043167+0000", 2024-08-19T14:22:01.219 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.3309999999999998e-06 2024-08-19T14:22:01.219 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.219 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.219 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:22:01.219 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:06.043225+0000", 2024-08-19T14:22:01.220 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.8062000000000001e-05 2024-08-19T14:22:01.220 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.220 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.220 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:22:01.220 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:06.043246+0000", 2024-08-19T14:22:01.220 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1316000000000001e-05 2024-08-19T14:22:01.220 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.220 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.220 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:22:01.220 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:06.045577+0000", 2024-08-19T14:22:01.220 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0023308890000000001 2024-08-19T14:22:01.220 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.220 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.220 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:22:01.221 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:06.045605+0000", 2024-08-19T14:22:01.221 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.7691e-05 2024-08-19T14:22:01.221 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:01.221 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:22:01.221 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:01.221 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.221 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.221 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:48912 2.7 e117/102 2:e6ffd652:::benchmark_data_smithi045_46415_object48911:head v 117'2997, mlcod=117'2992)", 2024-08-19T14:22:01.221 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:07.468103+0000", 2024-08-19T14:22:01.221 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 533.71007126999996, 2024-08-19T14:22:01.221 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0061113349999999999, 2024-08-19T14:22:01.221 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:22:01.221 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:22:01.221 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:22:01.222 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:22:01.222 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.222 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:22:01.222 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468103+0000", 2024-08-19T14:22:01.222 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:01.222 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.222 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.222 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:22:01.222 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468103+0000", 2024-08-19T14:22:01.222 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:01.222 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.222 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.222 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:22:01.223 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468103+0000", 2024-08-19T14:22:01.223 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.8000000000000002e-07 2024-08-19T14:22:01.223 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.223 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.223 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:22:01.223 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468136+0000", 2024-08-19T14:22:01.223 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.3411000000000002e-05 2024-08-19T14:22:01.223 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.223 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.223 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:22:01.223 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468143+0000", 2024-08-19T14:22:01.223 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.5869999999999998e-06 2024-08-19T14:22:01.223 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.223 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.224 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:22:01.224 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468146+0000", 2024-08-19T14:22:01.224 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.304e-06 2024-08-19T14:22:01.224 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.224 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.224 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:22:01.224 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.470630+0000", 2024-08-19T14:22:01.224 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0024838009999999999 2024-08-19T14:22:01.224 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.224 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.224 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:22:01.224 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.470643+0000", 2024-08-19T14:22:01.224 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2964e-05 2024-08-19T14:22:01.224 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.224 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.225 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:22:01.225 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.474173+0000", 2024-08-19T14:22:01.225 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.003530423 2024-08-19T14:22:01.225 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.225 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.225 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:22:01.225 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.474214+0000", 2024-08-19T14:22:01.225 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.0565000000000002e-05 2024-08-19T14:22:01.225 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:01.225 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:22:01.225 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:01.225 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.225 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.225 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:48971 2.7 e117/102 2:ecf6a539:::benchmark_data_smithi045_46415_object48970:head v 117'2999, mlcod=117'2992)", 2024-08-19T14:22:01.226 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:07.468206+0000", 2024-08-19T14:22:01.226 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 533.70996733599998, 2024-08-19T14:22:01.226 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0076013820000000003, 2024-08-19T14:22:01.226 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:22:01.226 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:22:01.226 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:22:01.226 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:22:01.226 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.226 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:22:01.226 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468206+0000", 2024-08-19T14:22:01.226 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:01.226 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.226 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.227 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:22:01.227 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468206+0000", 2024-08-19T14:22:01.227 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:01.227 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.227 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.227 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:22:01.227 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468207+0000", 2024-08-19T14:22:01.227 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6899999999999999e-07 2024-08-19T14:22:01.227 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.227 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.227 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:22:01.227 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468248+0000", 2024-08-19T14:22:01.227 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.0985999999999997e-05 2024-08-19T14:22:01.227 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.228 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.228 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:22:01.228 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468254+0000", 2024-08-19T14:22:01.228 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.9769999999999997e-06 2024-08-19T14:22:01.228 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.228 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.228 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:22:01.228 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468257+0000", 2024-08-19T14:22:01.228 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.0259999999999999e-06 2024-08-19T14:22:01.228 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.228 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.228 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:22:01.228 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.471956+0000", 2024-08-19T14:22:01.228 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0036996059999999998 2024-08-19T14:22:01.229 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.229 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.229 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:22:01.229 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.471974+0000", 2024-08-19T14:22:01.229 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8219999999999998e-05 2024-08-19T14:22:01.229 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.229 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.229 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:22:01.229 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.475774+0000", 2024-08-19T14:22:01.229 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0037994729999999998 2024-08-19T14:22:01.229 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.229 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.229 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:22:01.229 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.475808+0000", 2024-08-19T14:22:01.229 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.3924999999999998e-05 2024-08-19T14:22:01.230 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:01.230 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:22:01.230 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:01.230 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.230 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.230 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:48978 2.7 e117/102 2:e11d5fd6:::benchmark_data_smithi045_46415_object48977:head v 117'3000, mlcod=117'2992)", 2024-08-19T14:22:01.230 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:07.468262+0000", 2024-08-19T14:22:01.230 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 533.709911637, 2024-08-19T14:22:01.230 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0075744890000000002, 2024-08-19T14:22:01.230 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:22:01.230 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:22:01.230 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:22:01.230 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:22:01.231 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.231 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:22:01.231 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468262+0000", 2024-08-19T14:22:01.231 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:01.231 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.231 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.231 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:22:01.231 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468262+0000", 2024-08-19T14:22:01.231 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:01.231 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.231 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.231 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:22:01.232 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468262+0000", 2024-08-19T14:22:01.232 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.04e-07 2024-08-19T14:22:01.232 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.232 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.232 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:22:01.232 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468309+0000", 2024-08-19T14:22:01.232 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.6913999999999998e-05 2024-08-19T14:22:01.232 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.232 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.232 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:22:01.232 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468315+0000", 2024-08-19T14:22:01.232 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7139999999999998e-06 2024-08-19T14:22:01.232 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.233 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.233 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:22:01.233 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468318+0000", 2024-08-19T14:22:01.233 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.117e-06 2024-08-19T14:22:01.233 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.233 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.233 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:22:01.233 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.472643+0000", 2024-08-19T14:22:01.233 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0043251950000000004 2024-08-19T14:22:01.233 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.233 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.233 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:22:01.233 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.472657+0000", 2024-08-19T14:22:01.234 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3478e-05 2024-08-19T14:22:01.234 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.234 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.234 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:22:01.234 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.475812+0000", 2024-08-19T14:22:01.234 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0031556259999999999 2024-08-19T14:22:01.234 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.234 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.234 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:22:01.234 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.475837+0000", 2024-08-19T14:22:01.234 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.4241000000000001e-05 2024-08-19T14:22:01.234 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:01.234 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:22:01.235 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:01.235 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.235 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.235 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:48995 2.7 e117/102 2:e01c8116:::benchmark_data_smithi045_46415_object48994:head v 117'3001, mlcod=117'2992)", 2024-08-19T14:22:01.235 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:07.468324+0000", 2024-08-19T14:22:01.235 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 533.70985011000005, 2024-08-19T14:22:01.235 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0090529970000000001, 2024-08-19T14:22:01.235 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:22:01.235 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:22:01.235 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:22:01.235 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:22:01.235 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.236 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:22:01.236 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468324+0000", 2024-08-19T14:22:01.236 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:01.236 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.236 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.236 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:22:01.236 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468324+0000", 2024-08-19T14:22:01.236 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:01.236 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.236 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.236 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:22:01.236 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468324+0000", 2024-08-19T14:22:01.236 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.97e-07 2024-08-19T14:22:01.237 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.237 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.237 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:22:01.237 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468388+0000", 2024-08-19T14:22:01.237 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.4523999999999994e-05 2024-08-19T14:22:01.237 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.237 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.237 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:22:01.237 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468395+0000", 2024-08-19T14:22:01.237 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.579e-06 2024-08-19T14:22:01.237 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.237 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.237 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:22:01.238 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468398+0000", 2024-08-19T14:22:01.238 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.2780000000000002e-06 2024-08-19T14:22:01.238 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.238 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.238 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:22:01.238 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.473325+0000", 2024-08-19T14:22:01.238 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00492626 2024-08-19T14:22:01.238 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.238 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.238 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:22:01.238 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.473343+0000", 2024-08-19T14:22:01.238 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8788999999999999e-05 2024-08-19T14:22:01.238 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.239 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.239 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:22:01.239 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.477310+0000", 2024-08-19T14:22:01.239 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0039662819999999998 2024-08-19T14:22:01.239 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.239 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.239 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:22:01.239 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.477377+0000", 2024-08-19T14:22:01.239 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.7088000000000005e-05 2024-08-19T14:22:01.239 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:01.240 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:22:01.240 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:01.240 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.240 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.240 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:48996 2.7 e117/102 2:e8abf67b:::benchmark_data_smithi045_46415_object48995:head v 117'3002, mlcod=117'2992)", 2024-08-19T14:22:01.240 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:07.468406+0000", 2024-08-19T14:22:01.240 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 533.709768102, 2024-08-19T14:22:01.240 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0090294899999999994, 2024-08-19T14:22:01.240 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:22:01.240 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:22:01.240 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:22:01.240 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:22:01.240 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.240 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:22:01.241 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468406+0000", 2024-08-19T14:22:01.241 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:01.241 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.241 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.241 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:22:01.241 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468406+0000", 2024-08-19T14:22:01.241 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:01.241 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.241 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.241 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:22:01.241 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468406+0000", 2024-08-19T14:22:01.241 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0900000000000001e-07 2024-08-19T14:22:01.241 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.242 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.242 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:22:01.242 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468447+0000", 2024-08-19T14:22:01.242 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.0992000000000003e-05 2024-08-19T14:22:01.242 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.242 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.242 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:22:01.242 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468452+0000", 2024-08-19T14:22:01.242 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2179999999999998e-06 2024-08-19T14:22:01.242 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.242 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.242 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:22:01.242 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468455+0000", 2024-08-19T14:22:01.243 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.796e-06 2024-08-19T14:22:01.243 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.243 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.243 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:22:01.243 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.474288+0000", 2024-08-19T14:22:01.243 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0058327689999999998 2024-08-19T14:22:01.243 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.243 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.243 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:22:01.243 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.474307+0000", 2024-08-19T14:22:01.243 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9031000000000001e-05 2024-08-19T14:22:01.243 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.243 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.243 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:22:01.244 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.477383+0000", 2024-08-19T14:22:01.244 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0030758790000000001 2024-08-19T14:22:01.244 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.244 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.244 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:22:01.244 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.477435+0000", 2024-08-19T14:22:01.244 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2596000000000003e-05 2024-08-19T14:22:01.244 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:01.244 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:22:01.244 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:01.244 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.244 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.244 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:49005 2.7 e117/102 2:e58e73b6:::benchmark_data_smithi045_46415_object49004:head v 117'3003, mlcod=117'2992)", 2024-08-19T14:22:01.244 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:07.468460+0000", 2024-08-19T14:22:01.245 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 533.70971400099995, 2024-08-19T14:22:01.245 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0098532910000000001, 2024-08-19T14:22:01.245 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:22:01.245 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:22:01.245 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:22:01.245 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:22:01.245 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.245 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:22:01.245 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468460+0000", 2024-08-19T14:22:01.245 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:01.245 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.245 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.245 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:22:01.246 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468460+0000", 2024-08-19T14:22:01.246 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:01.246 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.246 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.246 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:22:01.246 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468460+0000", 2024-08-19T14:22:01.246 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5900000000000001e-07 2024-08-19T14:22:01.246 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.246 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.246 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:22:01.246 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468518+0000", 2024-08-19T14:22:01.246 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.8536000000000003e-05 2024-08-19T14:22:01.246 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.246 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.247 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:22:01.247 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468524+0000", 2024-08-19T14:22:01.247 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.8039999999999998e-06 2024-08-19T14:22:01.247 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.247 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.247 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:22:01.247 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468527+0000", 2024-08-19T14:22:01.247 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.004e-06 2024-08-19T14:22:01.247 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.247 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.247 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:22:01.247 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.475085+0000", 2024-08-19T14:22:01.247 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0065574090000000002 2024-08-19T14:22:01.247 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.248 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.248 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:22:01.248 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.475106+0000", 2024-08-19T14:22:01.248 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1056999999999998e-05 2024-08-19T14:22:01.248 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.248 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.248 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:22:01.248 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.478267+0000", 2024-08-19T14:22:01.248 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0031614049999999999 2024-08-19T14:22:01.248 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.248 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.248 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:22:01.248 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.478313+0000", 2024-08-19T14:22:01.249 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.5917000000000001e-05 2024-08-19T14:22:01.249 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:01.249 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:22:01.249 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:01.249 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.249 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.249 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:49015 2.7 e117/102 2:e4d8c3e1:::benchmark_data_smithi045_46415_object49014:head v 117'3004, mlcod=117'2992)", 2024-08-19T14:22:01.249 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:07.468533+0000", 2024-08-19T14:22:01.249 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 533.70964093700002, 2024-08-19T14:22:01.249 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0098176270000000006, 2024-08-19T14:22:01.249 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:22:01.249 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:22:01.249 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:22:01.250 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:22:01.250 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.250 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:22:01.250 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468533+0000", 2024-08-19T14:22:01.250 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:01.250 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.250 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.250 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:22:01.250 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468533+0000", 2024-08-19T14:22:01.250 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:01.250 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.250 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.250 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:22:01.251 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468533+0000", 2024-08-19T14:22:01.251 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2600000000000001e-07 2024-08-19T14:22:01.251 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.251 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.251 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:22:01.251 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468577+0000", 2024-08-19T14:22:01.251 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.3436000000000002e-05 2024-08-19T14:22:01.251 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.251 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.251 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:22:01.251 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468582+0000", 2024-08-19T14:22:01.251 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4399999999999996e-06 2024-08-19T14:22:01.251 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.251 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.252 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:22:01.252 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468585+0000", 2024-08-19T14:22:01.252 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.8619999999999999e-06 2024-08-19T14:22:01.252 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.252 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.252 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:22:01.252 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.475870+0000", 2024-08-19T14:22:01.252 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0072852919999999996 2024-08-19T14:22:01.252 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.252 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.252 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:22:01.252 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.475894+0000", 2024-08-19T14:22:01.252 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.3858000000000001e-05 2024-08-19T14:22:01.253 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.253 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.253 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:22:01.253 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.478319+0000", 2024-08-19T14:22:01.253 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0024247729999999999 2024-08-19T14:22:01.253 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.253 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.253 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:22:01.253 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.478350+0000", 2024-08-19T14:22:01.253 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.1739999999999998e-05 2024-08-19T14:22:01.253 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:01.253 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:22:01.253 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:01.254 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.254 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.254 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:49016 2.7 e117/102 2:e52959d5:::benchmark_data_smithi045_46415_object49015:head v 117'3005, mlcod=117'2992)", 2024-08-19T14:22:01.254 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:07.468590+0000", 2024-08-19T14:22:01.254 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 533.70958405900001, 2024-08-19T14:22:01.254 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.011903967999999999, 2024-08-19T14:22:01.254 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:22:01.254 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:22:01.254 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:22:01.254 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:22:01.254 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.254 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:22:01.254 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468590+0000", 2024-08-19T14:22:01.254 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:01.255 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.255 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.255 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:22:01.255 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468590+0000", 2024-08-19T14:22:01.255 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:01.255 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.255 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.255 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:22:01.255 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468590+0000", 2024-08-19T14:22:01.255 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.54e-07 2024-08-19T14:22:01.255 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.255 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.255 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:22:01.256 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468632+0000", 2024-08-19T14:22:01.256 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.2459999999999997e-05 2024-08-19T14:22:01.256 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.256 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.256 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:22:01.256 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468638+0000", 2024-08-19T14:22:01.256 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2410000000000003e-06 2024-08-19T14:22:01.256 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.256 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:22:01.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:00 smithi032 ceph-mon[20004]: pgmap v833: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 7.6 GiB data, 27 GiB used, 778 GiB / 805 GiB avail; 70 MiB/s wr, 1.13k op/s; 2731/375231 objects misplaced (0.728%); 2.7 MiB/s, 43 objects/s recovering 2024-08-19T14:22:01.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:00 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2657441848' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:22:01.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:00 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:22:01.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:00 smithi032 ceph-mon[20004]: from='client.25544 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:22:01.258 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.258 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:22:01.258 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468641+0000", 2024-08-19T14:22:01.258 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.061e-06 2024-08-19T14:22:01.258 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.258 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.258 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:22:01.258 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.476571+0000", 2024-08-19T14:22:01.259 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0079302920000000002 2024-08-19T14:22:01.259 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.259 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.259 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:22:01.259 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.476593+0000", 2024-08-19T14:22:01.259 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2045e-05 2024-08-19T14:22:01.259 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.259 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.259 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:22:01.259 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.480447+0000", 2024-08-19T14:22:01.259 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.003853825 2024-08-19T14:22:01.259 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.259 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.259 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:22:01.260 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.480494+0000", 2024-08-19T14:22:01.260 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.689e-05 2024-08-19T14:22:01.260 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:01.260 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:22:01.260 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:01.260 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.260 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.260 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:49017 2.7 e117/102 2:eca28b17:::benchmark_data_smithi045_46415_object49016:head v 117'3006, mlcod=117'2992)", 2024-08-19T14:22:01.260 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:07.468646+0000", 2024-08-19T14:22:01.260 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 533.70952819499996, 2024-08-19T14:22:01.260 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.011882515999999999, 2024-08-19T14:22:01.260 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:22:01.260 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:22:01.260 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:22:01.261 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:22:01.261 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.261 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:22:01.261 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468646+0000", 2024-08-19T14:22:01.261 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:01.261 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.261 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.261 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:22:01.261 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468646+0000", 2024-08-19T14:22:01.261 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:01.261 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.261 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.261 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:22:01.261 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468646+0000", 2024-08-19T14:22:01.262 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5800000000000001e-07 2024-08-19T14:22:01.262 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.262 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.262 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:22:01.262 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468686+0000", 2024-08-19T14:22:01.262 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.0732999999999998e-05 2024-08-19T14:22:01.262 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.262 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.262 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:22:01.262 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468692+0000", 2024-08-19T14:22:01.262 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0860000000000001e-06 2024-08-19T14:22:01.262 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.262 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.262 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:22:01.263 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468694+0000", 2024-08-19T14:22:01.263 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.864e-06 2024-08-19T14:22:01.263 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.263 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.263 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:22:01.263 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.477463+0000", 2024-08-19T14:22:01.263 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0087685190000000007 2024-08-19T14:22:01.263 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.263 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.263 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:22:01.263 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.477483+0000", 2024-08-19T14:22:01.263 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0449999999999999e-05 2024-08-19T14:22:01.263 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.264 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.264 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:22:01.264 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.480501+0000", 2024-08-19T14:22:01.264 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.003017802 2024-08-19T14:22:01.264 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.264 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.264 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:22:01.264 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.480528+0000", 2024-08-19T14:22:01.264 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.6903999999999999e-05 2024-08-19T14:22:01.264 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:01.264 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:22:01.264 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:01.264 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.265 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.265 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:49027 2.7 e117/102 2:e96ce7ac:::benchmark_data_smithi045_46415_object49026:head v 117'3007, mlcod=117'2992)", 2024-08-19T14:22:01.265 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:07.468699+0000", 2024-08-19T14:22:01.265 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 533.70947478300002, 2024-08-19T14:22:01.265 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.011858521, 2024-08-19T14:22:01.265 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:22:01.265 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:22:01.265 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:22:01.265 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:22:01.265 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.265 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:22:01.265 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468699+0000", 2024-08-19T14:22:01.265 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:01.265 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.266 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.266 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:22:01.266 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468699+0000", 2024-08-19T14:22:01.266 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:01.266 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.266 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.266 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:22:01.266 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468699+0000", 2024-08-19T14:22:01.266 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5599999999999999e-07 2024-08-19T14:22:01.266 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.266 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.266 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:22:01.266 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468743+0000", 2024-08-19T14:22:01.267 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.3741e-05 2024-08-19T14:22:01.267 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.267 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.267 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:22:01.267 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468748+0000", 2024-08-19T14:22:01.267 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1000000000000003e-06 2024-08-19T14:22:01.267 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.267 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.267 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:22:01.267 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468751+0000", 2024-08-19T14:22:01.267 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.8260000000000001e-06 2024-08-19T14:22:01.267 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.267 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.268 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:22:01.268 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.478397+0000", 2024-08-19T14:22:01.268 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0096462240000000001 2024-08-19T14:22:01.268 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.268 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.268 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:22:01.268 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.478415+0000", 2024-08-19T14:22:01.268 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8450000000000001e-05 2024-08-19T14:22:01.268 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.268 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.268 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:22:01.268 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.480533+0000", 2024-08-19T14:22:01.268 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.002117196 2024-08-19T14:22:01.268 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.269 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.269 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:22:01.269 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.480558+0000", 2024-08-19T14:22:01.269 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.4828e-05 2024-08-19T14:22:01.269 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:01.269 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:22:01.269 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:01.269 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.269 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.269 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:49030 2.7 e117/102 2:e915209e:::benchmark_data_smithi045_46415_object49029:head v 117'3008, mlcod=117'2992)", 2024-08-19T14:22:01.269 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:07.468755+0000", 2024-08-19T14:22:01.269 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 533.70941849600001, 2024-08-19T14:22:01.269 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.015403441, 2024-08-19T14:22:01.269 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:22:01.270 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:22:01.270 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:22:01.270 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:22:01.270 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.270 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:22:01.270 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468755+0000", 2024-08-19T14:22:01.270 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:01.270 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.270 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.270 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:22:01.270 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468755+0000", 2024-08-19T14:22:01.270 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:01.271 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.271 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.271 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:22:01.271 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468755+0000", 2024-08-19T14:22:01.271 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5599999999999999e-07 2024-08-19T14:22:01.271 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.271 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.271 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:22:01.271 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468805+0000", 2024-08-19T14:22:01.271 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.9622e-05 2024-08-19T14:22:01.271 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.271 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.271 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:22:01.271 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468811+0000", 2024-08-19T14:22:01.271 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.6119999999999998e-06 2024-08-19T14:22:01.272 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.272 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.272 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:22:01.272 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468813+0000", 2024-08-19T14:22:01.272 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.7769999999999998e-06 2024-08-19T14:22:01.272 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.272 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.272 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:22:01.272 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.479393+0000", 2024-08-19T14:22:01.272 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.010579646999999999 2024-08-19T14:22:01.272 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.272 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.273 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:22:01.273 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.479412+0000", 2024-08-19T14:22:01.273 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8728000000000001e-05 2024-08-19T14:22:01.273 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.273 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.273 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:22:01.273 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.484094+0000", 2024-08-19T14:22:01.273 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0046820009999999999 2024-08-19T14:22:01.273 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.273 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.273 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:22:01.273 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.484159+0000", 2024-08-19T14:22:01.273 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.4898000000000001e-05 2024-08-19T14:22:01.274 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:01.274 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:22:01.274 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:01.274 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.274 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.274 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24580.0:49049 2.7 e117/102 2:eb29c0a1:::benchmark_data_smithi045_46415_object49048:head v 117'3009, mlcod=117'2992)", 2024-08-19T14:22:01.274 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:13:07.468819+0000", 2024-08-19T14:22:01.274 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 533.70935524900005, 2024-08-19T14:22:01.274 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.015383332, 2024-08-19T14:22:01.274 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:22:01.274 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:22:01.274 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:22:01.274 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:22:01.274 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.275 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:22:01.275 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468819+0000", 2024-08-19T14:22:01.275 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:01.275 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.275 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.275 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:22:01.275 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468819+0000", 2024-08-19T14:22:01.275 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:01.275 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.275 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.275 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:22:01.275 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468819+0000", 2024-08-19T14:22:01.275 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2000000000000001e-07 2024-08-19T14:22:01.275 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.276 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.276 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:22:01.276 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468854+0000", 2024-08-19T14:22:01.276 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.5079000000000001e-05 2024-08-19T14:22:01.276 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.276 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.276 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:22:01.276 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468859+0000", 2024-08-19T14:22:01.276 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1710000000000001e-06 2024-08-19T14:22:01.276 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.276 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.276 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:22:01.276 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.468862+0000", 2024-08-19T14:22:01.276 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.0599999999999999e-06 2024-08-19T14:22:01.277 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.277 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.277 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:22:01.277 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.480596+0000", 2024-08-19T14:22:01.277 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.011734058 2024-08-19T14:22:01.277 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.277 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.277 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:22:01.277 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.480615+0000", 2024-08-19T14:22:01.277 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8919e-05 2024-08-19T14:22:01.277 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.277 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.277 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:22:01.277 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.484164+0000", 2024-08-19T14:22:01.278 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.003548486 2024-08-19T14:22:01.278 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:01.278 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:01.278 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:22:01.278 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:13:07.484202+0000", 2024-08-19T14:22:01.278 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.8339e-05 2024-08-19T14:22:01.278 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:01.278 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:22:01.278 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:01.278 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:01.278 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:22:01.278 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:22:01.320 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:01.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:01.321 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:22:01.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:00 smithi031 ceph-mon[16336]: pgmap v833: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 7.6 GiB data, 27 GiB used, 778 GiB / 805 GiB avail; 70 MiB/s wr, 1.13k op/s; 2731/375231 objects misplaced (0.728%); 2.7 MiB/s, 43 objects/s recovering 2024-08-19T14:22:01.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:00 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2657441848' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:22:01.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:00 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:22:01.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:00 smithi031 ceph-mon[16336]: from='client.25544 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:22:01.422 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:22:01.453 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:01.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:01.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:22:01.555 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:22:01.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:01.578 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:01.578 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:22:01.678 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:22:01.711 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:01.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:01.712 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:22:01.813 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:22:01.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:01.842 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:01.843 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:22:01.944 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:22:01.965 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:01.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:01.966 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:22:02.037 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":833,"stamp":"2024-08-19T14:21:59.905386+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8197505568,"num_objects":125077,"num_object_clones":0,"num_object_copies":375231,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2731,"num_objects_unfound":0,"num_objects_dirty":125077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":125075,"num_write_kb":8004800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":92340224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":129384,"ondisk_log_size":129384,"up":141,"acting":140,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":142,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":27927524,"kb_used_data":24072076,"kb_used_omap":14,"kb_used_meta":3855345,"kb_avail":815778844,"statfs":{"total":863955320832,"available":835357536256,"internally_reserved":0,"allocated":24649805824,"data_stored":24634436830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14353,"internal_metadata":3947874287},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,4,6,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":518717440,"num_objects":7915,"num_object_clones":0,"num_object_copies":23745,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-306,"num_objects_unfound":0,"num_objects_dirty":7915,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7915,"num_write_kb":506560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":306,"num_bytes_recovered":20054016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.028983"},"pg_stats":[{"pgid":"4.5","version":"369'7711","reported_seq":7789,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-08-19T14:21:55.780913+0000","last_change":"2024-08-19T14:20:29.268214+0000","last_active":"2024-08-19T14:21:55.780913+0000","last_peered":"2024-08-19T14:21:55.780913+0000","last_clean":"2024-08-19T14:21:55.780913+0000","last_became_active":"2024-08-19T14:20:29.267549+0000","last_became_peered":"2024-08-19T14:20:29.267549+0000","last_unstale":"2024-08-19T14:21:55.780913+0000","last_undegraded":"2024-08-19T14:21:55.780913+0000","last_fullsized":"2024-08-19T14:21:55.780913+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":7711,"log_dups_size":0,"ondisk_log_size":7711,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:57.640230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":505348096,"num_objects":7711,"num_object_clones":0,"num_object_copies":23133,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7711,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7711,"num_write_kb":493504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,0],"acting":[11,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"70'139","reported_seq":956,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-08-19T14:21:55.901044+0000","last_change":"2024-08-19T14:19:58.072227+0000","last_active":"2024-08-19T14:21:55.901044+0000","last_peered":"2024-08-19T14:21:55.901044+0000","last_clean":"2024-08-19T14:21:55.901044+0000","last_became_active":"2024-08-19T14:18:13.807133+0000","last_became_peered":"2024-08-19T14:18:13.807133+0000","last_unstale":"2024-08-19T14:21:55.901044+0000","last_undegraded":"2024-08-19T14:21:55.901044+0000","last_fullsized":"2024-08-19T14:21:55.901044+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:19:58.072083+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:19:58.072083+0000","last_clean_scrub_stamp":"2024-08-19T14:19:58.072083+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,9],"acting":[7,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"370'8038","reported_seq":10935,"reported_epoch":370,"state":"active+recovering+undersized+remapped","last_fresh":"2024-08-19T14:21:59.265991+0000","last_change":"2024-08-19T14:21:07.239135+0000","last_active":"2024-08-19T14:21:59.265991+0000","last_peered":"2024-08-19T14:21:59.265991+0000","last_clean":"2024-08-19T14:21:04.784061+0000","last_became_active":"2024-08-19T14:21:07.213050+0000","last_became_peered":"2024-08-19T14:21:07.213050+0000","last_unstale":"2024-08-19T14:21:59.265991+0000","last_undegraded":"2024-08-19T14:21:59.265991+0000","last_fullsized":"2024-08-19T14:21:06.805412+0000","mapping_epoch":354,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":8038,"log_dups_size":0,"ondisk_log_size":8038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":526778368,"num_objects":8038,"num_object_clones":0,"num_object_copies":24114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2731,"num_objects_unfound":0,"num_objects_dirty":8038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8038,"num_write_kb":514432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":92340224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,7],"acting":[1,7],"avail_no_missing":["1","5","7"],"object_location_counts":[{"shards":"1,5,7","objects":2731},{"shards":"1,7","objects":5307}],"blocked_by":[],"up_primary":9,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"70'139","reported_seq":855,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-08-19T14:21:55.902346+0000","last_change":"2024-08-19T14:20:03.714917+0000","last_active":"2024-08-19T14:21:55.902346+0000","last_peered":"2024-08-19T14:21:55.902346+0000","last_clean":"2024-08-19T14:21:55.902346+0000","last_became_active":"2024-08-19T14:18:13.804721+0000","last_became_peered":"2024-08-19T14:18:13.804721+0000","last_unstale":"2024-08-19T14:21:55.902346+0000","last_undegraded":"2024-08-19T14:21:55.902346+0000","last_fullsized":"2024-08-19T14:21:55.902346+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:03.714818+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:03.714818+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"370'7967","reported_seq":8047,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-08-19T14:21:59.219236+0000","last_change":"2024-08-19T14:20:29.268110+0000","last_active":"2024-08-19T14:21:59.219236+0000","last_peered":"2024-08-19T14:21:59.219236+0000","last_clean":"2024-08-19T14:21:59.219236+0000","last_became_active":"2024-08-19T14:20:29.267811+0000","last_became_peered":"2024-08-19T14:20:29.267811+0000","last_unstale":"2024-08-19T14:21:59.219236+0000","last_undegraded":"2024-08-19T14:21:59.219236+0000","last_fullsized":"2024-08-19T14:21:59.219236+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":7967,"log_dups_size":0,"ondisk_log_size":7967,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":522125312,"num_objects":7967,"num_object_clones":0,"num_object_copies":23901,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7967,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7967,"num_write_kb":509888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"70'139","reported_seq":803,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-08-19T14:21:55.900026+0000","last_change":"2024-08-19T14:20:50.742676+0000","last_active":"2024-08-19T14:21:55.900026+0000","last_peered":"2024-08-19T14:21:55.900026+0000","last_clean":"2024-08-19T14:21:55.900026+0000","last_became_active":"2024-08-19T14:20:50.742354+0000","last_became_peered":"2024-08-19T14:20:50.742354+0000","last_unstale":"2024-08-19T14:21:55.900026+0000","last_undegraded":"2024-08-19T14:21:55.900026+0000","last_fullsized":"2024-08-19T14:21:55.900026+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:19:59.302931+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:19:59.302931+0000","last_clean_scrub_stamp":"2024-08-19T14:19:59.302931+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":114,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"370'7879","reported_seq":7959,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-08-19T14:21:59.153620+0000","last_change":"2024-08-19T14:20:29.267640+0000","last_active":"2024-08-19T14:21:59.153620+0000","last_peered":"2024-08-19T14:21:59.153620+0000","last_clean":"2024-08-19T14:21:59.153620+0000","last_became_active":"2024-08-19T14:20:29.267331+0000","last_became_peered":"2024-08-19T14:20:29.267331+0000","last_unstale":"2024-08-19T14:21:59.153620+0000","last_undegraded":"2024-08-19T14:21:59.153620+0000","last_fullsized":"2024-08-19T14:21:59.153620+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":7879,"log_dups_size":0,"ondisk_log_size":7879,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":516358144,"num_objects":7879,"num_object_clones":0,"num_object_copies":23637,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7879,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7879,"num_write_kb":504256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"70'139","reported_seq":852,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-08-19T14:21:55.895310+0000","last_change":"2024-08-19T14:20:09.507422+0000","last_active":"2024-08-19T14:21:55.895310+0000","last_peered":"2024-08-19T14:21:55.895310+0000","last_clean":"2024-08-19T14:21:55.895310+0000","last_became_active":"2024-08-19T14:18:13.808505+0000","last_became_peered":"2024-08-19T14:18:13.808505+0000","last_unstale":"2024-08-19T14:21:55.895310+0000","last_undegraded":"2024-08-19T14:21:55.895310+0000","last_fullsized":"2024-08-19T14:21:55.895310+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:09.507256+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:09.507256+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,11],"acting":[9,0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"370'7901","reported_seq":7981,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-08-19T14:21:58.375129+0000","last_change":"2024-08-19T14:20:29.268766+0000","last_active":"2024-08-19T14:21:58.375129+0000","last_peered":"2024-08-19T14:21:58.375129+0000","last_clean":"2024-08-19T14:21:58.375129+0000","last_became_active":"2024-08-19T14:20:29.268443+0000","last_became_peered":"2024-08-19T14:20:29.268443+0000","last_unstale":"2024-08-19T14:21:58.375129+0000","last_undegraded":"2024-08-19T14:21:58.375129+0000","last_fullsized":"2024-08-19T14:21:58.375129+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":7901,"log_dups_size":0,"ondisk_log_size":7901,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":517799936,"num_objects":7901,"num_object_clones":0,"num_object_copies":23703,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7901,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7901,"num_write_kb":505664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,11],"acting":[9,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.7","version":"70'139","reported_seq":503,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-08-19T14:21:55.901509+0000","last_change":"2024-08-19T14:21:46.057764+0000","last_active":"2024-08-19T14:21:55.901509+0000","last_peered":"2024-08-19T14:21:55.901509+0000","last_clean":"2024-08-19T14:21:55.901509+0000","last_became_active":"2024-08-19T14:20:28.204384+0000","last_became_peered":"2024-08-19T14:20:28.204384+0000","last_unstale":"2024-08-19T14:21:55.901509+0000","last_undegraded":"2024-08-19T14:21:55.901509+0000","last_fullsized":"2024-08-19T14:21:55.901509+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:46.057624+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:21:46.057624+0000","last_clean_scrub_stamp":"2024-08-19T14:21:46.057624+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:02.689278+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"369'7686","reported_seq":7820,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-08-19T14:21:55.806802+0000","last_change":"2024-08-19T14:21:48.933576+0000","last_active":"2024-08-19T14:21:55.806802+0000","last_peered":"2024-08-19T14:21:55.806802+0000","last_clean":"2024-08-19T14:21:55.806802+0000","last_became_active":"2024-08-19T14:20:29.267379+0000","last_became_peered":"2024-08-19T14:20:29.267379+0000","last_unstale":"2024-08-19T14:21:55.806802+0000","last_undegraded":"2024-08-19T14:21:55.806802+0000","last_fullsized":"2024-08-19T14:21:55.806802+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":5249,"log_size":7686,"log_dups_size":0,"ondisk_log_size":7686,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:59.288972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503709696,"num_objects":7686,"num_object_clones":0,"num_object_copies":23058,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7686,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7686,"num_write_kb":491904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,1],"acting":[11,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.6","version":"70'139","reported_seq":856,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-08-19T14:21:55.899923+0000","last_change":"2024-08-19T14:19:55.512078+0000","last_active":"2024-08-19T14:21:55.899923+0000","last_peered":"2024-08-19T14:21:55.899923+0000","last_clean":"2024-08-19T14:21:55.899923+0000","last_became_active":"2024-08-19T14:18:26.874902+0000","last_became_peered":"2024-08-19T14:18:26.874902+0000","last_unstale":"2024-08-19T14:21:55.899923+0000","last_undegraded":"2024-08-19T14:21:55.899923+0000","last_fullsized":"2024-08-19T14:21:55.899923+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:19:55.511974+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:27.848870+0000","last_clean_scrub_stamp":"2024-08-19T14:19:55.511974+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,4],"acting":[9,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"370'7863","reported_seq":7942,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-08-19T14:21:59.494128+0000","last_change":"2024-08-19T14:20:29.267682+0000","last_active":"2024-08-19T14:21:59.494128+0000","last_peered":"2024-08-19T14:21:59.494128+0000","last_clean":"2024-08-19T14:21:59.494128+0000","last_became_active":"2024-08-19T14:20:29.267360+0000","last_became_peered":"2024-08-19T14:20:29.267360+0000","last_unstale":"2024-08-19T14:21:59.494128+0000","last_undegraded":"2024-08-19T14:21:59.494128+0000","last_fullsized":"2024-08-19T14:21:59.494128+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":7863,"log_dups_size":0,"ondisk_log_size":7863,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:22:04.078731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":515309568,"num_objects":7863,"num_object_clones":0,"num_object_copies":23589,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7863,"num_write_kb":503232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"70'139","reported_seq":667,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-08-19T14:21:53.895571+0000","last_change":"2024-08-19T14:20:12.731993+0000","last_active":"2024-08-19T14:21:53.895571+0000","last_peered":"2024-08-19T14:21:53.895571+0000","last_clean":"2024-08-19T14:21:53.895571+0000","last_became_active":"2024-08-19T14:18:26.881435+0000","last_became_peered":"2024-08-19T14:18:26.881435+0000","last_unstale":"2024-08-19T14:21:53.895571+0000","last_undegraded":"2024-08-19T14:21:53.895571+0000","last_fullsized":"2024-08-19T14:21:53.895571+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:12.731850+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:12.731850+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:54.295344+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,4],"acting":[11,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.1","version":"369'7784","reported_seq":7862,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-08-19T14:21:55.815526+0000","last_change":"2024-08-19T14:20:29.268808+0000","last_active":"2024-08-19T14:21:55.815526+0000","last_peered":"2024-08-19T14:21:55.815526+0000","last_clean":"2024-08-19T14:21:55.815526+0000","last_became_active":"2024-08-19T14:20:29.268465+0000","last_became_peered":"2024-08-19T14:20:29.268465+0000","last_unstale":"2024-08-19T14:21:55.815526+0000","last_undegraded":"2024-08-19T14:21:55.815526+0000","last_fullsized":"2024-08-19T14:21:55.815526+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":7784,"log_dups_size":0,"ondisk_log_size":7784,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:57.700841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":510132224,"num_objects":7784,"num_object_clones":0,"num_object_copies":23352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7784,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7784,"num_write_kb":498176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.4","version":"70'139","reported_seq":900,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-08-19T14:21:53.894268+0000","last_change":"2024-08-19T14:21:39.314063+0000","last_active":"2024-08-19T14:21:53.894268+0000","last_peered":"2024-08-19T14:21:53.894268+0000","last_clean":"2024-08-19T14:21:53.894268+0000","last_became_active":"2024-08-19T14:18:13.806013+0000","last_became_peered":"2024-08-19T14:18:13.806013+0000","last_unstale":"2024-08-19T14:21:53.894268+0000","last_undegraded":"2024-08-19T14:21:53.894268+0000","last_fullsized":"2024-08-19T14:21:53.894268+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:39.313941+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:21:39.313941+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:22:45.184594+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.f","version":"370'7851","reported_seq":7931,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-08-19T14:21:59.266689+0000","last_change":"2024-08-19T14:20:29.212298+0000","last_active":"2024-08-19T14:21:59.266689+0000","last_peered":"2024-08-19T14:21:59.266689+0000","last_clean":"2024-08-19T14:21:59.266689+0000","last_became_active":"2024-08-19T14:20:29.212003+0000","last_became_peered":"2024-08-19T14:20:29.212003+0000","last_unstale":"2024-08-19T14:21:59.266689+0000","last_undegraded":"2024-08-19T14:21:59.266689+0000","last_fullsized":"2024-08-19T14:21:59.266689+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":7851,"log_dups_size":0,"ondisk_log_size":7851,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":514523136,"num_objects":7851,"num_object_clones":0,"num_object_copies":23553,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7851,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7851,"num_write_kb":502464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,11],"acting":[1,3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"70'139","reported_seq":864,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-08-19T14:21:55.895415+0000","last_change":"2024-08-19T14:20:12.570453+0000","last_active":"2024-08-19T14:21:55.895415+0000","last_peered":"2024-08-19T14:21:55.895415+0000","last_clean":"2024-08-19T14:21:55.895415+0000","last_became_active":"2024-08-19T14:18:26.875006+0000","last_became_peered":"2024-08-19T14:18:26.875006+0000","last_unstale":"2024-08-19T14:21:55.895415+0000","last_undegraded":"2024-08-19T14:21:55.895415+0000","last_fullsized":"2024-08-19T14:21:55.895415+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:12.570352+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:28.864870+0000","last_clean_scrub_stamp":"2024-08-19T14:20:12.570352+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,0],"acting":[9,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.c","version":"370'7803","reported_seq":7883,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-08-19T14:21:58.142299+0000","last_change":"2024-08-19T14:20:29.266724+0000","last_active":"2024-08-19T14:21:58.142299+0000","last_peered":"2024-08-19T14:21:58.142299+0000","last_clean":"2024-08-19T14:21:58.142299+0000","last_became_active":"2024-08-19T14:20:29.266373+0000","last_became_peered":"2024-08-19T14:20:29.266373+0000","last_unstale":"2024-08-19T14:21:58.142299+0000","last_undegraded":"2024-08-19T14:21:58.142299+0000","last_fullsized":"2024-08-19T14:21:58.142299+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":7803,"log_dups_size":0,"ondisk_log_size":7803,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:59.546960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":511377408,"num_objects":7803,"num_object_clones":0,"num_object_copies":23409,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7803,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7803,"num_write_kb":499392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,8],"acting":[4,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"70'139","reported_seq":979,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-08-19T14:21:55.899154+0000","last_change":"2024-08-19T14:19:56.106047+0000","last_active":"2024-08-19T14:21:55.899154+0000","last_peered":"2024-08-19T14:21:55.899154+0000","last_clean":"2024-08-19T14:21:55.899154+0000","last_became_active":"2024-08-19T14:18:26.887657+0000","last_became_peered":"2024-08-19T14:18:26.887657+0000","last_unstale":"2024-08-19T14:21:55.899154+0000","last_undegraded":"2024-08-19T14:21:55.899154+0000","last_fullsized":"2024-08-19T14:21:55.899154+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:19:56.105880+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:19:56.105880+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"370'7763","reported_seq":7843,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-08-19T14:21:58.166969+0000","last_change":"2024-08-19T14:20:29.208992+0000","last_active":"2024-08-19T14:21:58.166969+0000","last_peered":"2024-08-19T14:21:58.166969+0000","last_clean":"2024-08-19T14:21:58.166969+0000","last_became_active":"2024-08-19T14:20:29.208599+0000","last_became_peered":"2024-08-19T14:20:29.208599+0000","last_unstale":"2024-08-19T14:21:58.166969+0000","last_undegraded":"2024-08-19T14:21:58.166969+0000","last_fullsized":"2024-08-19T14:21:58.166969+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":7763,"log_dups_size":0,"ondisk_log_size":7763,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":508755968,"num_objects":7763,"num_object_clones":0,"num_object_copies":23289,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7763,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7763,"num_write_kb":496832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,1],"acting":[4,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"70'139","reported_seq":876,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-08-19T14:21:55.897654+0000","last_change":"2024-08-19T14:21:37.951800+0000","last_active":"2024-08-19T14:21:55.897654+0000","last_peered":"2024-08-19T14:21:55.897654+0000","last_clean":"2024-08-19T14:21:55.897654+0000","last_became_active":"2024-08-19T14:18:26.887277+0000","last_became_peered":"2024-08-19T14:18:26.887277+0000","last_unstale":"2024-08-19T14:21:55.897654+0000","last_undegraded":"2024-08-19T14:21:55.897654+0000","last_fullsized":"2024-08-19T14:21:55.897654+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:37.951694+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:21:37.951694+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:03.860882+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,8],"acting":[5,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"370'7688","reported_seq":7768,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-08-19T14:21:58.130393+0000","last_change":"2024-08-19T14:20:29.209096+0000","last_active":"2024-08-19T14:21:58.130393+0000","last_peered":"2024-08-19T14:21:58.130393+0000","last_clean":"2024-08-19T14:21:58.130393+0000","last_became_active":"2024-08-19T14:20:29.208748+0000","last_became_peered":"2024-08-19T14:20:29.208748+0000","last_unstale":"2024-08-19T14:21:58.130393+0000","last_undegraded":"2024-08-19T14:21:58.130393+0000","last_fullsized":"2024-08-19T14:21:58.130393+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":7688,"log_dups_size":0,"ondisk_log_size":7688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503840768,"num_objects":7688,"num_object_clones":0,"num_object_copies":23064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7688,"num_write_kb":492032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"70'139","reported_seq":867,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-08-19T14:21:53.894449+0000","last_change":"2024-08-19T14:20:11.733961+0000","last_active":"2024-08-19T14:21:53.894449+0000","last_peered":"2024-08-19T14:21:53.894449+0000","last_clean":"2024-08-19T14:21:53.894449+0000","last_became_active":"2024-08-19T14:18:26.875179+0000","last_became_peered":"2024-08-19T14:18:26.875179+0000","last_unstale":"2024-08-19T14:21:53.894449+0000","last_undegraded":"2024-08-19T14:21:53.894449+0000","last_fullsized":"2024-08-19T14:21:53.894449+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:11.733865+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:11.733865+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:54.295384+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.b","version":"370'7703","reported_seq":7858,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-08-19T14:21:56.228966+0000","last_change":"2024-08-19T14:21:39.089676+0000","last_active":"2024-08-19T14:21:56.228966+0000","last_peered":"2024-08-19T14:21:56.228966+0000","last_clean":"2024-08-19T14:21:56.228966+0000","last_became_active":"2024-08-19T14:20:29.207372+0000","last_became_peered":"2024-08-19T14:20:29.207372+0000","last_unstale":"2024-08-19T14:21:56.228966+0000","last_undegraded":"2024-08-19T14:21:56.228966+0000","last_fullsized":"2024-08-19T14:21:56.228966+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"363'6615","last_scrub_stamp":"2024-08-19T14:21:39.089400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:21:39.089400+0000","objects_scrubbed":6552,"log_size":7703,"log_dups_size":0,"ondisk_log_size":7703,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:05.632119+0000","scrub_duration":1907,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":504823808,"num_objects":7703,"num_object_clones":0,"num_object_copies":23109,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7703,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7703,"num_write_kb":492992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"70'139","reported_seq":531,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-08-19T14:21:55.900366+0000","last_change":"2024-08-19T14:21:44.434481+0000","last_active":"2024-08-19T14:21:55.900366+0000","last_peered":"2024-08-19T14:21:55.900366+0000","last_clean":"2024-08-19T14:21:55.900366+0000","last_became_active":"2024-08-19T14:18:26.884105+0000","last_became_peered":"2024-08-19T14:18:26.884105+0000","last_unstale":"2024-08-19T14:21:55.900366+0000","last_undegraded":"2024-08-19T14:21:55.900366+0000","last_fullsized":"2024-08-19T14:21:55.900366+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:44.434382+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:20:15.321945+0000","last_clean_scrub_stamp":"2024-08-19T14:21:44.434382+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:13.544241+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"370'8036","reported_seq":8116,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-08-19T14:21:59.209540+0000","last_change":"2024-08-19T14:20:29.269723+0000","last_active":"2024-08-19T14:21:59.209540+0000","last_peered":"2024-08-19T14:21:59.209540+0000","last_clean":"2024-08-19T14:21:59.209540+0000","last_became_active":"2024-08-19T14:20:29.269432+0000","last_became_peered":"2024-08-19T14:20:29.269432+0000","last_unstale":"2024-08-19T14:21:59.209540+0000","last_undegraded":"2024-08-19T14:21:59.209540+0000","last_fullsized":"2024-08-19T14:21:59.209540+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":8036,"log_dups_size":0,"ondisk_log_size":8036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":526647296,"num_objects":8036,"num_object_clones":0,"num_object_copies":24108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8036,"num_write_kb":514304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.f","version":"70'139","reported_seq":775,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-08-19T14:21:55.897864+0000","last_change":"2024-08-19T14:21:14.765419+0000","last_active":"2024-08-19T14:21:55.897864+0000","last_peered":"2024-08-19T14:21:55.897864+0000","last_clean":"2024-08-19T14:21:55.897864+0000","last_became_active":"2024-08-19T14:18:18.711708+0000","last_became_peered":"2024-08-19T14:18:18.711708+0000","last_unstale":"2024-08-19T14:21:55.897864+0000","last_undegraded":"2024-08-19T14:21:55.897864+0000","last_fullsized":"2024-08-19T14:21:55.897864+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:14.765340+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:21:14.765340+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:22:24.415219+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"369'7709","reported_seq":7866,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-08-19T14:21:55.815518+0000","last_change":"2024-08-19T14:21:43.974700+0000","last_active":"2024-08-19T14:21:55.815518+0000","last_peered":"2024-08-19T14:21:55.815518+0000","last_clean":"2024-08-19T14:21:55.815518+0000","last_became_active":"2024-08-19T14:20:29.211975+0000","last_became_peered":"2024-08-19T14:20:29.211975+0000","last_unstale":"2024-08-19T14:21:55.815518+0000","last_undegraded":"2024-08-19T14:21:55.815518+0000","last_fullsized":"2024-08-19T14:21:55.815518+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"364'6991","last_scrub_stamp":"2024-08-19T14:21:43.974613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:21:43.974613+0000","objects_scrubbed":6934,"log_size":7709,"log_dups_size":0,"ondisk_log_size":7709,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:07.463188+0000","scrub_duration":1682,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":505217024,"num_objects":7709,"num_object_clones":0,"num_object_copies":23127,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7709,"num_write_kb":493376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,3],"acting":[11,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.c","version":"70'139","reported_seq":636,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-08-19T14:21:55.906314+0000","last_change":"2024-08-19T14:21:40.181472+0000","last_active":"2024-08-19T14:21:55.906314+0000","last_peered":"2024-08-19T14:21:55.906314+0000","last_clean":"2024-08-19T14:21:55.906314+0000","last_became_active":"2024-08-19T14:18:26.872942+0000","last_became_peered":"2024-08-19T14:18:26.872942+0000","last_unstale":"2024-08-19T14:21:55.906314+0000","last_undegraded":"2024-08-19T14:21:55.906314+0000","last_fullsized":"2024-08-19T14:21:55.906314+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:40.181356+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:20:07.839076+0000","last_clean_scrub_stamp":"2024-08-19T14:21:40.181356+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:22:52.009381+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,9],"acting":[3,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"369'7693","reported_seq":7771,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-08-19T14:21:55.812913+0000","last_change":"2024-08-19T14:20:29.268318+0000","last_active":"2024-08-19T14:21:55.812913+0000","last_peered":"2024-08-19T14:21:55.812913+0000","last_clean":"2024-08-19T14:21:55.812913+0000","last_became_active":"2024-08-19T14:20:29.267442+0000","last_became_peered":"2024-08-19T14:20:29.267442+0000","last_unstale":"2024-08-19T14:21:55.812913+0000","last_undegraded":"2024-08-19T14:21:55.812913+0000","last_fullsized":"2024-08-19T14:21:55.812913+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":7693,"log_dups_size":0,"ondisk_log_size":7693,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":504168448,"num_objects":7693,"num_object_clones":0,"num_object_copies":23079,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7693,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7693,"num_write_kb":492352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,9],"acting":[11,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.d","version":"70'139","reported_seq":551,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-08-19T14:21:55.899570+0000","last_change":"2024-08-19T14:21:07.155294+0000","last_active":"2024-08-19T14:21:55.899570+0000","last_peered":"2024-08-19T14:21:55.899570+0000","last_clean":"2024-08-19T14:21:55.899570+0000","last_became_active":"2024-08-19T14:18:26.882002+0000","last_became_peered":"2024-08-19T14:18:26.882002+0000","last_unstale":"2024-08-19T14:21:55.899570+0000","last_undegraded":"2024-08-19T14:21:55.899570+0000","last_fullsized":"2024-08-19T14:21:55.899570+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:07.155157+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:21:07.155157+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:22:11.274708+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"70'139","reported_seq":810,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-08-19T14:21:55.902421+0000","last_change":"2024-08-19T14:21:13.777891+0000","last_active":"2024-08-19T14:21:55.902421+0000","last_peered":"2024-08-19T14:21:55.902421+0000","last_clean":"2024-08-19T14:21:55.902421+0000","last_became_active":"2024-08-19T14:18:18.703594+0000","last_became_peered":"2024-08-19T14:18:18.703594+0000","last_unstale":"2024-08-19T14:21:55.902421+0000","last_undegraded":"2024-08-19T14:21:55.902421+0000","last_fullsized":"2024-08-19T14:21:55.902421+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:13.777777+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:21:13.777777+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:22:38.473560+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"70'139","reported_seq":862,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-08-19T14:21:55.902216+0000","last_change":"2024-08-19T14:20:07.692054+0000","last_active":"2024-08-19T14:21:55.902216+0000","last_peered":"2024-08-19T14:21:55.902216+0000","last_clean":"2024-08-19T14:21:55.902216+0000","last_became_active":"2024-08-19T14:18:18.698334+0000","last_became_peered":"2024-08-19T14:18:18.698334+0000","last_unstale":"2024-08-19T14:21:55.902216+0000","last_undegraded":"2024-08-19T14:21:55.902216+0000","last_fullsized":"2024-08-19T14:21:55.902216+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:07.691932+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:07.691932+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.10","version":"70'139","reported_seq":965,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-08-19T14:21:55.897534+0000","last_change":"2024-08-19T14:21:07.768770+0000","last_active":"2024-08-19T14:21:55.897534+0000","last_peered":"2024-08-19T14:21:55.897534+0000","last_clean":"2024-08-19T14:21:55.897534+0000","last_became_active":"2024-08-19T14:18:18.704562+0000","last_became_peered":"2024-08-19T14:18:18.704562+0000","last_unstale":"2024-08-19T14:21:55.897534+0000","last_undegraded":"2024-08-19T14:21:55.897534+0000","last_fullsized":"2024-08-19T14:21:55.897534+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:07.768697+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:21:07.768697+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:22:10.360289+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,8],"acting":[5,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"70'139","reported_seq":514,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-08-19T14:21:55.905395+0000","last_change":"2024-08-19T14:21:16.071812+0000","last_active":"2024-08-19T14:21:55.905395+0000","last_peered":"2024-08-19T14:21:55.905395+0000","last_clean":"2024-08-19T14:21:55.905395+0000","last_became_active":"2024-08-19T14:18:18.697894+0000","last_became_peered":"2024-08-19T14:18:18.697894+0000","last_unstale":"2024-08-19T14:21:55.905395+0000","last_undegraded":"2024-08-19T14:21:55.905395+0000","last_fullsized":"2024-08-19T14:21:55.905395+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:16.071703+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:21:16.071703+0000","last_clean_scrub_stamp":"2024-08-19T14:21:16.071703+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:22:18.184091+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"70'139","reported_seq":491,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-08-19T14:21:55.905120+0000","last_change":"2024-08-19T14:20:28.205212+0000","last_active":"2024-08-19T14:21:55.905120+0000","last_peered":"2024-08-19T14:21:55.905120+0000","last_clean":"2024-08-19T14:21:55.905120+0000","last_became_active":"2024-08-19T14:20:28.204724+0000","last_became_peered":"2024-08-19T14:20:28.204724+0000","last_unstale":"2024-08-19T14:21:55.905120+0000","last_undegraded":"2024-08-19T14:21:55.905120+0000","last_fullsized":"2024-08-19T14:21:55.905120+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:08.680021+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:08.680021+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"70'139","reported_seq":609,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-08-19T14:21:53.893028+0000","last_change":"2024-08-19T14:20:31.931658+0000","last_active":"2024-08-19T14:21:53.893028+0000","last_peered":"2024-08-19T14:21:53.893028+0000","last_clean":"2024-08-19T14:21:53.893028+0000","last_became_active":"2024-08-19T14:18:31.901556+0000","last_became_peered":"2024-08-19T14:18:31.901556+0000","last_unstale":"2024-08-19T14:21:53.893028+0000","last_undegraded":"2024-08-19T14:21:53.893028+0000","last_fullsized":"2024-08-19T14:21:53.893028+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:31.931562+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:20:31.931562+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:54.686161+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,0],"acting":[8,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.14","version":"70'139","reported_seq":620,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-08-19T14:21:55.899227+0000","last_change":"2024-08-19T14:19:55.130518+0000","last_active":"2024-08-19T14:21:55.899227+0000","last_peered":"2024-08-19T14:21:55.899227+0000","last_clean":"2024-08-19T14:21:55.899227+0000","last_became_active":"2024-08-19T14:18:18.703926+0000","last_became_peered":"2024-08-19T14:18:18.703926+0000","last_unstale":"2024-08-19T14:21:55.899227+0000","last_undegraded":"2024-08-19T14:21:55.899227+0000","last_fullsized":"2024-08-19T14:21:55.899227+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:19:55.130420+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:19:55.130420+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,1],"acting":[7,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.15","version":"70'139","reported_seq":507,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-08-19T14:21:55.899472+0000","last_change":"2024-08-19T14:20:09.181046+0000","last_active":"2024-08-19T14:21:55.899472+0000","last_peered":"2024-08-19T14:21:55.899472+0000","last_clean":"2024-08-19T14:21:55.899472+0000","last_became_active":"2024-08-19T14:18:31.912496+0000","last_became_peered":"2024-08-19T14:18:31.912496+0000","last_unstale":"2024-08-19T14:21:55.899472+0000","last_undegraded":"2024-08-19T14:21:55.899472+0000","last_fullsized":"2024-08-19T14:21:55.899472+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:09.180923+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:20:09.180923+0000","last_clean_scrub_stamp":"2024-08-19T14:20:09.180923+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"70'139","reported_seq":493,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-08-19T14:21:55.898555+0000","last_change":"2024-08-19T14:21:40.430835+0000","last_active":"2024-08-19T14:21:55.898555+0000","last_peered":"2024-08-19T14:21:55.898555+0000","last_clean":"2024-08-19T14:21:55.898555+0000","last_became_active":"2024-08-19T14:18:31.901229+0000","last_became_peered":"2024-08-19T14:18:31.901229+0000","last_unstale":"2024-08-19T14:21:55.898555+0000","last_undegraded":"2024-08-19T14:21:55.898555+0000","last_fullsized":"2024-08-19T14:21:55.898555+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:40.430714+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:21:40.430714+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:04.072653+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1b","version":"70'139","reported_seq":843,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-08-19T14:21:53.892116+0000","last_change":"2024-08-19T14:21:48.693864+0000","last_active":"2024-08-19T14:21:53.892116+0000","last_peered":"2024-08-19T14:21:53.892116+0000","last_clean":"2024-08-19T14:21:53.892116+0000","last_became_active":"2024-08-19T14:18:31.901152+0000","last_became_peered":"2024-08-19T14:18:31.901152+0000","last_unstale":"2024-08-19T14:21:53.892116+0000","last_undegraded":"2024-08-19T14:21:53.892116+0000","last_fullsized":"2024-08-19T14:21:53.892116+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:48.693761+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:21:48.693761+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:07.571130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.18","version":"70'139","reported_seq":881,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-08-19T14:21:55.897533+0000","last_change":"2024-08-19T14:20:36.510554+0000","last_active":"2024-08-19T14:21:55.897533+0000","last_peered":"2024-08-19T14:21:55.897533+0000","last_clean":"2024-08-19T14:21:55.897533+0000","last_became_active":"2024-08-19T14:18:31.902735+0000","last_became_peered":"2024-08-19T14:18:31.902735+0000","last_unstale":"2024-08-19T14:21:55.897533+0000","last_undegraded":"2024-08-19T14:21:55.897533+0000","last_fullsized":"2024-08-19T14:21:55.897533+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:36.510442+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:36.510442+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:59.078431+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.19","version":"70'139","reported_seq":837,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-08-19T14:21:55.905253+0000","last_change":"2024-08-19T14:21:40.219949+0000","last_active":"2024-08-19T14:21:55.905253+0000","last_peered":"2024-08-19T14:21:55.905253+0000","last_clean":"2024-08-19T14:21:55.905253+0000","last_became_active":"2024-08-19T14:18:31.902547+0000","last_became_peered":"2024-08-19T14:18:31.902547+0000","last_unstale":"2024-08-19T14:21:55.905253+0000","last_undegraded":"2024-08-19T14:21:55.905253+0000","last_fullsized":"2024-08-19T14:21:55.905253+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:40.219830+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:21:40.219830+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:05.414864+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1e","version":"70'139","reported_seq":485,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-08-19T14:21:53.894295+0000","last_change":"2024-08-19T14:20:18.771836+0000","last_active":"2024-08-19T14:21:53.894295+0000","last_peered":"2024-08-19T14:21:53.894295+0000","last_clean":"2024-08-19T14:21:53.894295+0000","last_became_active":"2024-08-19T14:18:31.902546+0000","last_became_peered":"2024-08-19T14:18:31.902546+0000","last_unstale":"2024-08-19T14:21:53.894295+0000","last_undegraded":"2024-08-19T14:21:53.894295+0000","last_fullsized":"2024-08-19T14:21:53.894295+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:18.771704+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:18.771704+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:21:54.295418+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1c","version":"70'139","reported_seq":601,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-08-19T14:21:53.893555+0000","last_change":"2024-08-19T14:21:38.753663+0000","last_active":"2024-08-19T14:21:53.893555+0000","last_peered":"2024-08-19T14:21:53.893555+0000","last_clean":"2024-08-19T14:21:53.893555+0000","last_became_active":"2024-08-19T14:18:31.902494+0000","last_became_peered":"2024-08-19T14:18:31.902494+0000","last_unstale":"2024-08-19T14:21:53.893555+0000","last_undegraded":"2024-08-19T14:21:53.893555+0000","last_fullsized":"2024-08-19T14:21:53.893555+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:38.753553+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:21:38.753553+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:22:46.720742+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,11],"acting":[8,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1d","version":"70'139","reported_seq":517,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-08-19T14:21:55.899201+0000","last_change":"2024-08-19T14:20:17.725615+0000","last_active":"2024-08-19T14:21:55.899201+0000","last_peered":"2024-08-19T14:21:55.899201+0000","last_clean":"2024-08-19T14:21:55.899201+0000","last_became_active":"2024-08-19T14:18:31.901183+0000","last_became_peered":"2024-08-19T14:18:31.901183+0000","last_unstale":"2024-08-19T14:21:55.899201+0000","last_undegraded":"2024-08-19T14:21:55.899201+0000","last_fullsized":"2024-08-19T14:21:55.899201+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:17.725477+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:17.725477+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,5],"acting":[1,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":8196915200,"num_objects":125075,"num_object_clones":0,"num_object_copies":375225,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2731,"num_objects_unfound":0,"num_objects_dirty":125075,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":125075,"num_write_kb":8004800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":92340224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24722866176,"data_stored":24722866176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125075,"ondisk_log_size":125075,"up":48,"acting":47,"num_store_stats":9},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1802240,"data_stored":1788032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187583,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3856108,"kb_used_data":3445920,"kb_used_omap":1,"kb_used_meta":410174,"kb_avail":89889044,"statfs":{"total":95995035648,"available":92046381056,"internally_reserved":0,"allocated":3528622080,"data_stored":3526918846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":420018628},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":343597383813,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":74,"seq":317827580041,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2305976,"kb_used_data":1833324,"kb_used_omap":1,"kb_used_meta":472638,"kb_avail":91439176,"statfs":{"total":95995035648,"available":93633716224,"internally_reserved":0,"allocated":1877323776,"data_stored":1875616366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":483981761},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874382,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":860200,"kb_used_data":490272,"kb_used_omap":1,"kb_used_meta":369918,"kb_avail":92884952,"statfs":{"total":95995035648,"available":95114190848,"internally_reserved":0,"allocated":502038528,"data_stored":500335294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":378796485},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":57,"seq":244813136017,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4477820,"kb_used_data":4027380,"kb_used_omap":1,"kb_used_meta":450430,"kb_avail":89267332,"statfs":{"total":95995035648,"available":91409747968,"internally_reserved":0,"allocated":4124037120,"data_stored":4122326158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":461240778},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":206158430357,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691993,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2114624,"kb_used_data":1768308,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":91630528,"statfs":{"total":95995035648,"available":93829660672,"internally_reserved":0,"allocated":1810747392,"data_stored":1809036430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":354613701},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986334,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2957304,"kb_used_data":2488812,"kb_used_omap":1,"kb_used_meta":468478,"kb_avail":90787848,"statfs":{"total":95995035648,"available":92966756352,"internally_reserved":0,"allocated":2548543488,"data_stored":2546836078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":479721925},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149860,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3453876,"kb_used_data":3017324,"kb_used_omap":1,"kb_used_meta":436542,"kb_avail":90291276,"statfs":{"total":95995035648,"available":92458266624,"internally_reserved":0,"allocated":3089739776,"data_stored":3088032366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":447019463},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411497,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575021,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5565948,"kb_used_data":5021108,"kb_used_omap":1,"kb_used_meta":544830,"kb_avail":88179204,"statfs":{"total":95995035648,"available":90295504896,"internally_reserved":0,"allocated":5141614592,"data_stored":5139903630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":557906373},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738545,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2335668,"kb_used_data":1979628,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":91409484,"statfs":{"total":95995035648,"available":93603311616,"internally_reserved":0,"allocated":2027139072,"data_stored":2025431662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":364575173},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2025586688,"data_stored":2025586688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5181865984,"data_stored":5181865984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3095265280,"data_stored":3095265280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2554789888,"data_stored":2554789888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1811742720,"data_stored":1811742720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4130865152,"data_stored":4130865152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":498663424,"data_stored":498663424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1884028928,"data_stored":1884028928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":3540058112,"data_stored":3540058112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T14:22:02.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:01 smithi045 ceph-mon[20143]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:22:02.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:01 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:22:02.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:01 smithi045 ceph-mon[20143]: osdmap e371: 12 total, 12 up, 9 in 2024-08-19T14:22:02.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:01 smithi045 ceph-mon[20143]: 1.5 scrub starts 2024-08-19T14:22:02.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:01 smithi045 ceph-mon[20143]: 1.5 scrub ok 2024-08-19T14:22:02.051 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 88 16 129278 129262 91.7976 60.5 0.00879964 0.0108887 2024-08-19T14:22:02.066 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:22:02.096 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:02.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:02.097 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:22:02.198 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:22:02.221 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:02.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:02.222 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:22:02.225 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:01 smithi032 ceph-mon[20004]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:22:02.225 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:01 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:22:02.225 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:01 smithi032 ceph-mon[20004]: osdmap e371: 12 total, 12 up, 9 in 2024-08-19T14:22:02.225 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:01 smithi032 ceph-mon[20004]: 1.5 scrub starts 2024-08-19T14:22:02.225 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:01 smithi032 ceph-mon[20004]: 1.5 scrub ok 2024-08-19T14:22:02.237 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-08-19T14:22:02.323 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:22:02.353 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:02.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:02.354 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:22:02.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:01 smithi031 ceph-mon[16336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:22:02.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:01 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:22:02.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:01 smithi031 ceph-mon[16336]: osdmap e371: 12 total, 12 up, 9 in 2024-08-19T14:22:02.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:01 smithi031 ceph-mon[16336]: 1.5 scrub starts 2024-08-19T14:22:02.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:01 smithi031 ceph-mon[16336]: 1.5 scrub ok 2024-08-19T14:22:02.454 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:22:02.481 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:02.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:02.483 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:22:02.584 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:22:02.619 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:22:02.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:02.620 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:02.620 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:22:02.721 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:22:02.746 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:02.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:02.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:22:02.848 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:22:02.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:02.881 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:02.881 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:22:02.982 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:22:03.018 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:03.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:03.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:22:03.052 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 89 16 130256 130240 91.4529 61.125 0.00790337 0.0109301 2024-08-19T14:22:03.120 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:22:03.153 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:03.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:03.154 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:22:03.255 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:22:03.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:03 smithi032 ceph-mon[20004]: pgmap v835: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 7.7 GiB data, 27 GiB used, 778 GiB / 805 GiB avail; 67 MiB/s wr, 1.07k op/s; 2731/380613 objects misplaced (0.718%); 2.4 MiB/s, 38 objects/s recovering 2024-08-19T14:22:03.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:03 smithi032 ceph-mon[20004]: 1.b scrub starts 2024-08-19T14:22:03.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:03 smithi032 ceph-mon[20004]: 1.b scrub ok 2024-08-19T14:22:03.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=57} crc {auth=3844387402,config=2267314355,health=1997216375,kv=2659915664}) 2024-08-19T14:22:03.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=3452950460}) 2024-08-19T14:22:03.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=466025714}) 2024-08-19T14:22:03.259 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {kv=21,logm=79} crc {kv=3988016596,logm=1133634557}) 2024-08-19T14:22:03.259 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4010328061}) 2024-08-19T14:22:03.259 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2545751690}) 2024-08-19T14:22:03.259 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2494271256}) 2024-08-19T14:22:03.259 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2014744068}) 2024-08-19T14:22:03.259 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2299111062}) 2024-08-19T14:22:03.259 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3981635033}) 2024-08-19T14:22:03.259 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2160154212}) 2024-08-19T14:22:03.259 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3930611401}) 2024-08-19T14:22:03.259 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2195799963}) 2024-08-19T14:22:03.259 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2756774531}) 2024-08-19T14:22:03.259 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4188362844}) 2024-08-19T14:22:03.259 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2865977224}) 2024-08-19T14:22:03.260 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2121769585}) 2024-08-19T14:22:03.260 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4003465848}) 2024-08-19T14:22:03.260 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=337143848}) 2024-08-19T14:22:03.260 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=893072438}) 2024-08-19T14:22:03.260 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=458116227}) 2024-08-19T14:22:03.260 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2202830698}) 2024-08-19T14:22:03.260 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3293976021}) 2024-08-19T14:22:03.260 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3089587075}) 2024-08-19T14:22:03.260 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2600338745}) 2024-08-19T14:22:03.260 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1130315361}) 2024-08-19T14:22:03.260 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=370516271}) 2024-08-19T14:22:03.260 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1590105172}) 2024-08-19T14:22:03.260 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2352943092}) 2024-08-19T14:22:03.260 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1432877632}) 2024-08-19T14:22:03.261 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1730333550}) 2024-08-19T14:22:03.261 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2212512019}) 2024-08-19T14:22:03.261 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1306363937}) 2024-08-19T14:22:03.261 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1001553709}) 2024-08-19T14:22:03.261 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2918764624}) 2024-08-19T14:22:03.261 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3243238377}) 2024-08-19T14:22:03.261 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3088076938}) 2024-08-19T14:22:03.261 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=512644529}) 2024-08-19T14:22:03.262 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3265875966}) 2024-08-19T14:22:03.262 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1069486677}) 2024-08-19T14:22:03.262 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=634251794}) 2024-08-19T14:22:03.262 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=74,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=2} crc {logm=3566302676,mdsmap=2474562478,mgr=2924679688,mgr_command_descs=1300230762,mgr_metadata=482233419,mgrstat=2812917215}) 2024-08-19T14:22:03.262 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=100} crc {mgrstat=2394606258}) 2024-08-19T14:22:03.262 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=100} crc {mgrstat=3628492418}) 2024-08-19T14:22:03.262 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=11,mon_config_key=89} crc {mgrstat=2901543591,mon_config_key=4072413482}) 2024-08-19T14:22:03.262 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2431280192}) 2024-08-19T14:22:03.262 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=87,monmap=5,osd_metadata=8} crc {mon_config_key=803278344,monmap=3193798142,osd_metadata=3733428623}) 2024-08-19T14:22:03.262 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {osd_metadata=4,osd_pg_creating=1,osdmap=95} crc {osd_metadata=1462670740,osd_pg_creating=4165095012,osdmap=2399748274}) 2024-08-19T14:22:03.262 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3159391742}) 2024-08-19T14:22:03.262 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=664319008}) 2024-08-19T14:22:03.262 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3438549530}) 2024-08-19T14:22:03.263 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1289214110}) 2024-08-19T14:22:03.263 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=268078775}) 2024-08-19T14:22:03.263 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=272830975}) 2024-08-19T14:22:03.263 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=50} crc {osdmap=2450976987}) 2024-08-19T14:22:03.282 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:03.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:03.283 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:22:03.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:03 smithi045 ceph-mon[20143]: pgmap v835: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 7.7 GiB data, 27 GiB used, 778 GiB / 805 GiB avail; 67 MiB/s wr, 1.07k op/s; 2731/380613 objects misplaced (0.718%); 2.4 MiB/s, 38 objects/s recovering 2024-08-19T14:22:03.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:03 smithi045 ceph-mon[20143]: 1.b scrub starts 2024-08-19T14:22:03.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:03 smithi045 ceph-mon[20143]: 1.b scrub ok 2024-08-19T14:22:03.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=57} crc {auth=3844387402,config=2267314355,health=1997216375,kv=2659915664}) 2024-08-19T14:22:03.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=3452950460}) 2024-08-19T14:22:03.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=466025714}) 2024-08-19T14:22:03.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {kv=21,logm=79} crc {kv=3988016596,logm=1133634557}) 2024-08-19T14:22:03.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4010328061}) 2024-08-19T14:22:03.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2545751690}) 2024-08-19T14:22:03.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2494271256}) 2024-08-19T14:22:03.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2014744068}) 2024-08-19T14:22:03.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2299111062}) 2024-08-19T14:22:03.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3981635033}) 2024-08-19T14:22:03.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2160154212}) 2024-08-19T14:22:03.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3930611401}) 2024-08-19T14:22:03.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2195799963}) 2024-08-19T14:22:03.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2756774531}) 2024-08-19T14:22:03.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4188362844}) 2024-08-19T14:22:03.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2865977224}) 2024-08-19T14:22:03.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2121769585}) 2024-08-19T14:22:03.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4003465848}) 2024-08-19T14:22:03.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=337143848}) 2024-08-19T14:22:03.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=893072438}) 2024-08-19T14:22:03.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=458116227}) 2024-08-19T14:22:03.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2202830698}) 2024-08-19T14:22:03.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3293976021}) 2024-08-19T14:22:03.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3089587075}) 2024-08-19T14:22:03.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2600338745}) 2024-08-19T14:22:03.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1130315361}) 2024-08-19T14:22:03.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=370516271}) 2024-08-19T14:22:03.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1590105172}) 2024-08-19T14:22:03.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2352943092}) 2024-08-19T14:22:03.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1432877632}) 2024-08-19T14:22:03.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1730333550}) 2024-08-19T14:22:03.300 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2212512019}) 2024-08-19T14:22:03.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1306363937}) 2024-08-19T14:22:03.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1001553709}) 2024-08-19T14:22:03.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2918764624}) 2024-08-19T14:22:03.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3243238377}) 2024-08-19T14:22:03.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3088076938}) 2024-08-19T14:22:03.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=512644529}) 2024-08-19T14:22:03.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3265875966}) 2024-08-19T14:22:03.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1069486677}) 2024-08-19T14:22:03.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=634251794}) 2024-08-19T14:22:03.302 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=74,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=2} crc {logm=3566302676,mdsmap=2474562478,mgr=2924679688,mgr_command_descs=1300230762,mgr_metadata=482233419,mgrstat=2812917215}) 2024-08-19T14:22:03.302 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=100} crc {mgrstat=2394606258}) 2024-08-19T14:22:03.302 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=100} crc {mgrstat=3628492418}) 2024-08-19T14:22:03.302 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=11,mon_config_key=89} crc {mgrstat=2901543591,mon_config_key=4072413482}) 2024-08-19T14:22:03.302 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2431280192}) 2024-08-19T14:22:03.302 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=87,monmap=5,osd_metadata=8} crc {mon_config_key=803278344,monmap=3193798142,osd_metadata=3733428623}) 2024-08-19T14:22:03.302 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {osd_metadata=4,osd_pg_creating=1,osdmap=95} crc {osd_metadata=1462670740,osd_pg_creating=4165095012,osdmap=2399748274}) 2024-08-19T14:22:03.303 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3159391742}) 2024-08-19T14:22:03.303 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=664319008}) 2024-08-19T14:22:03.303 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3438549530}) 2024-08-19T14:22:03.303 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1289214110}) 2024-08-19T14:22:03.303 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=268078775}) 2024-08-19T14:22:03.303 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=272830975}) 2024-08-19T14:22:03.303 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=50} crc {osdmap=2450976987}) 2024-08-19T14:22:03.303 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:02 smithi031 ceph-mon[16336]: pgmap v835: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 7.7 GiB data, 27 GiB used, 778 GiB / 805 GiB avail; 67 MiB/s wr, 1.07k op/s; 2731/380613 objects misplaced (0.718%); 2.4 MiB/s, 38 objects/s recovering 2024-08-19T14:22:03.303 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:02 smithi031 ceph-mon[16336]: 1.b scrub starts 2024-08-19T14:22:03.303 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:02 smithi031 ceph-mon[16336]: 1.b scrub ok 2024-08-19T14:22:03.303 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:02 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=57} crc {auth=3844387402,config=2267314355,health=1997216375,kv=2659915664}) 2024-08-19T14:22:03.303 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:02 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=3452950460}) 2024-08-19T14:22:03.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:02 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=466025714}) 2024-08-19T14:22:03.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:02 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {kv=21,logm=79} crc {kv=3988016596,logm=1133634557}) 2024-08-19T14:22:03.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:02 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4010328061}) 2024-08-19T14:22:03.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:02 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2545751690}) 2024-08-19T14:22:03.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:02 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2494271256}) 2024-08-19T14:22:03.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:02 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2014744068}) 2024-08-19T14:22:03.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:02 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2299111062}) 2024-08-19T14:22:03.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:02 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3981635033}) 2024-08-19T14:22:03.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:02 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2160154212}) 2024-08-19T14:22:03.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:02 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3930611401}) 2024-08-19T14:22:03.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:02 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2195799963}) 2024-08-19T14:22:03.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:02 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2756774531}) 2024-08-19T14:22:03.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:02 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4188362844}) 2024-08-19T14:22:03.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:02 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2865977224}) 2024-08-19T14:22:03.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:02 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2121769585}) 2024-08-19T14:22:03.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:02 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4003465848}) 2024-08-19T14:22:03.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:02 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=337143848}) 2024-08-19T14:22:03.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:02 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=893072438}) 2024-08-19T14:22:03.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:02 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=458116227}) 2024-08-19T14:22:03.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:02 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2202830698}) 2024-08-19T14:22:03.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:02 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3293976021}) 2024-08-19T14:22:03.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:02 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3089587075}) 2024-08-19T14:22:03.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:02 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2600338745}) 2024-08-19T14:22:03.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:02 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1130315361}) 2024-08-19T14:22:03.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:02 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=370516271}) 2024-08-19T14:22:03.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:02 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1590105172}) 2024-08-19T14:22:03.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:02 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2352943092}) 2024-08-19T14:22:03.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:02 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1432877632}) 2024-08-19T14:22:03.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:02 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1730333550}) 2024-08-19T14:22:03.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:02 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2212512019}) 2024-08-19T14:22:03.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:02 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1306363937}) 2024-08-19T14:22:03.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:02 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1001553709}) 2024-08-19T14:22:03.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:02 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2918764624}) 2024-08-19T14:22:03.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:02 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3243238377}) 2024-08-19T14:22:03.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:02 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3088076938}) 2024-08-19T14:22:03.307 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:02 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=512644529}) 2024-08-19T14:22:03.307 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:02 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3265875966}) 2024-08-19T14:22:03.307 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:02 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1069486677}) 2024-08-19T14:22:03.307 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:02 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=634251794}) 2024-08-19T14:22:03.307 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:02 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=74,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=2} crc {logm=3566302676,mdsmap=2474562478,mgr=2924679688,mgr_command_descs=1300230762,mgr_metadata=482233419,mgrstat=2812917215}) 2024-08-19T14:22:03.307 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:02 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=100} crc {mgrstat=2394606258}) 2024-08-19T14:22:03.307 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:02 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=100} crc {mgrstat=3628492418}) 2024-08-19T14:22:03.307 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:02 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=11,mon_config_key=89} crc {mgrstat=2901543591,mon_config_key=4072413482}) 2024-08-19T14:22:03.307 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:02 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2431280192}) 2024-08-19T14:22:03.307 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:02 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=87,monmap=5,osd_metadata=8} crc {mon_config_key=803278344,monmap=3193798142,osd_metadata=3733428623}) 2024-08-19T14:22:03.307 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:02 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {osd_metadata=4,osd_pg_creating=1,osdmap=95} crc {osd_metadata=1462670740,osd_pg_creating=4165095012,osdmap=2399748274}) 2024-08-19T14:22:03.307 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:02 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3159391742}) 2024-08-19T14:22:03.307 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:02 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=664319008}) 2024-08-19T14:22:03.308 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:02 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3438549530}) 2024-08-19T14:22:03.308 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:02 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1289214110}) 2024-08-19T14:22:03.308 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:02 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=268078775}) 2024-08-19T14:22:03.308 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:02 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=272830975}) 2024-08-19T14:22:03.308 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:02 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=50} crc {osdmap=2450976987}) 2024-08-19T14:22:03.373 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:22:03.373 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [ 2024-08-19T14:22:03.373 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:03.373 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:130532 4.3 4:c864c607:::benchmark_data_smithi045_71562_object130531:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e371)", 2024-08-19T14:22:03.373 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:22:03.369497+0000", 2024-08-19T14:22:03.374 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.0020285490000000002, 2024-08-19T14:22:03.374 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0020546150000000001, 2024-08-19T14:22:03.374 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:22:03.375 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:22:03.375 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "waiting for sub ops", 2024-08-19T14:22:03.375 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:22:03.375 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:22:03.375 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:22:03.375 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 130532 2024-08-19T14:22:03.375 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:03.375 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:22:03.375 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:03.375 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:22:03.375 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:03.369497+0000", 2024-08-19T14:22:03.375 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:03.376 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:03.376 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:03.376 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:22:03.376 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:03.369497+0000", 2024-08-19T14:22:03.376 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:03.376 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:03.376 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:03.376 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:22:03.376 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:03.369498+0000", 2024-08-19T14:22:03.376 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3200000000000001e-06 2024-08-19T14:22:03.377 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:03.377 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:03.377 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:22:03.377 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:03.369541+0000", 2024-08-19T14:22:03.377 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.2960000000000002e-05 2024-08-19T14:22:03.377 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:03.377 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:03.377 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:22:03.377 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:03.369554+0000", 2024-08-19T14:22:03.377 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3015e-05 2024-08-19T14:22:03.377 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:03.377 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:03.377 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:22:03.377 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:03.369562+0000", 2024-08-19T14:22:03.378 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.7440000000000007e-06 2024-08-19T14:22:03.378 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:03.378 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:03.378 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:22:03.378 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:03.369618+0000", 2024-08-19T14:22:03.378 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.5435e-05 2024-08-19T14:22:03.378 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:03.378 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:03.378 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:22:03.378 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:03.369760+0000", 2024-08-19T14:22:03.379 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000142309 2024-08-19T14:22:03.379 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:03.379 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:03.379 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,7", 2024-08-19T14:22:03.379 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:03.369907+0000", 2024-08-19T14:22:03.379 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014682200000000001 2024-08-19T14:22:03.379 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:03.379 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:22:03.379 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:03.379 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:03.379 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:03.379 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop_reply(client.25333.0:130519 4.1 e371/335)", 2024-08-19T14:22:03.379 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:22:03.369774+0000", 2024-08-19T14:22:03.380 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.001752183, 2024-08-19T14:22:03.380 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001823996, 2024-08-19T14:22:03.380 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:22:03.380 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:22:03.380 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "queued for pg", 2024-08-19T14:22:03.380 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:22:03.380 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:03.380 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:22:03.380 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:03.369774+0000", 2024-08-19T14:22:03.380 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:03.380 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:03.380 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:03.381 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:22:03.381 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:03.369774+0000", 2024-08-19T14:22:03.381 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:03.381 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:03.381 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:03.381 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:22:03.381 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:03.369774+0000", 2024-08-19T14:22:03.381 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.6199999999999997e-07 2024-08-19T14:22:03.381 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:03.381 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:03.381 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:22:03.381 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:03.369778+0000", 2024-08-19T14:22:03.381 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.16e-06 2024-08-19T14:22:03.382 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:03.382 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:03.382 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:22:03.382 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:03.369790+0000", 2024-08-19T14:22:03.382 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1068e-05 2024-08-19T14:22:03.382 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:03.382 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:03.382 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:22:03.382 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:03.369796+0000", 2024-08-19T14:22:03.383 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.9419999999999998e-06 2024-08-19T14:22:03.383 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:03.383 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:22:03.383 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:03.383 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:03.383 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:03.383 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.25333.0:130524 4.f e371/335 4:f8d7e21e:::benchmark_data_smithi045_71562_object130523:head v 371'8103, mlcod=371'8100)", 2024-08-19T14:22:03.383 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:22:03.369885+0000", 2024-08-19T14:22:03.383 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.001641202, 2024-08-19T14:22:03.383 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00174789, 2024-08-19T14:22:03.383 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:22:03.383 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:22:03.383 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-08-19T14:22:03.383 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:22:03.384 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:03.384 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:22:03.384 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:03.369885+0000", 2024-08-19T14:22:03.384 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:03.384 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:03.384 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:03.384 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:22:03.384 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:03.369885+0000", 2024-08-19T14:22:03.384 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:03.384 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:03.384 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:03.384 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:22:03.384 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:03.369885+0000", 2024-08-19T14:22:03.384 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.4499999999999997e-07 2024-08-19T14:22:03.385 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:03.385 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:03.385 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:22:03.385 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:03.369928+0000", 2024-08-19T14:22:03.385 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.2846999999999998e-05 2024-08-19T14:22:03.385 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:03.385 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:03.385 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:22:03.385 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:03.369938+0000", 2024-08-19T14:22:03.385 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.9690000000000006e-06 2024-08-19T14:22:03.385 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:03.386 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:03.386 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:22:03.386 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:03.369945+0000", 2024-08-19T14:22:03.386 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.5869999999999998e-06 2024-08-19T14:22:03.386 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:03.386 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:03.386 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:22:03.386 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:03.369987+0000", 2024-08-19T14:22:03.386 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.1987999999999998e-05 2024-08-19T14:22:03.386 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:03.386 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:03.386 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:22:03.386 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:03.370005+0000", 2024-08-19T14:22:03.387 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8715999999999998e-05 2024-08-19T14:22:03.387 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:03.387 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:22:03.387 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:03.387 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:03.387 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:03.387 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:130517 4.1 4:8982d263:::benchmark_data_smithi045_71562_object130516:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e371)", 2024-08-19T14:22:03.387 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:22:03.358491+0000", 2024-08-19T14:22:03.387 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.013034567, 2024-08-19T14:22:03.387 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.013175901, 2024-08-19T14:22:03.387 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:22:03.387 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:22:03.387 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "waiting for sub ops", 2024-08-19T14:22:03.387 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:22:03.388 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:22:03.388 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:22:03.388 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 130517 2024-08-19T14:22:03.388 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:03.388 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:22:03.388 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:03.388 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:22:03.388 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:03.358491+0000", 2024-08-19T14:22:03.388 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:03.388 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:03.388 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:03.388 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:22:03.388 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:03.358491+0000", 2024-08-19T14:22:03.389 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:03.389 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:03.389 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:03.389 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:22:03.389 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:03.358492+0000", 2024-08-19T14:22:03.389 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.7199999999999997e-07 2024-08-19T14:22:03.389 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:03.389 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:03.389 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:22:03.389 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:03.358545+0000", 2024-08-19T14:22:03.389 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.3307999999999999e-05 2024-08-19T14:22:03.389 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:03.389 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:03.389 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:22:03.390 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:22:03.391 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:03.358556+0000", 2024-08-19T14:22:03.391 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0855e-05 2024-08-19T14:22:03.391 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:03.391 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:03.391 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:22:03.391 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:03.358563+0000", 2024-08-19T14:22:03.391 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.0360000000000001e-06 2024-08-19T14:22:03.392 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:03.392 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:03.392 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:22:03.392 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:03.361571+0000", 2024-08-19T14:22:03.392 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0030075869999999999 2024-08-19T14:22:03.392 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:03.392 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:03.392 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:22:03.392 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:03.361701+0000", 2024-08-19T14:22:03.392 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012967799999999999 2024-08-19T14:22:03.392 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:03.392 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:03.392 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 4,5", 2024-08-19T14:22:03.393 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:03.361809+0000", 2024-08-19T14:22:03.393 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000108124 2024-08-19T14:22:03.393 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:03.393 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:03.393 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:22:03.393 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:03.369245+0000", 2024-08-19T14:22:03.393 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0074359839999999996 2024-08-19T14:22:03.393 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:03.393 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:03.393 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:22:03.393 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:03.369289+0000", 2024-08-19T14:22:03.393 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.4461e-05 2024-08-19T14:22:03.393 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:03.393 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:22:03.394 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:03.394 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:03.394 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:03.394 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:130519 4.1 4:82ef59df:::benchmark_data_smithi045_71562_object130518:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e371)", 2024-08-19T14:22:03.394 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:22:03.359392+0000", 2024-08-19T14:22:03.394 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.012134000000000001, 2024-08-19T14:22:03.394 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.012323967, 2024-08-19T14:22:03.394 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:22:03.394 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:22:03.394 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "waiting for sub ops", 2024-08-19T14:22:03.394 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:22:03.395 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:22:03.395 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:22:03.395 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 130519 2024-08-19T14:22:03.395 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:03.395 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:22:03.395 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:03.395 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:22:03.395 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:03.359392+0000", 2024-08-19T14:22:03.395 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:03.395 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:03.395 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:03.395 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:22:03.395 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:03.359392+0000", 2024-08-19T14:22:03.395 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:03.396 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:03.396 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:03.396 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:22:03.396 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:03.359392+0000", 2024-08-19T14:22:03.396 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.3600000000000004e-07 2024-08-19T14:22:03.396 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:03.396 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:03.396 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:22:03.396 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:03.359434+0000", 2024-08-19T14:22:03.396 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.1255999999999999e-05 2024-08-19T14:22:03.396 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:03.396 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:03.396 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:22:03.396 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:03.359439+0000", 2024-08-19T14:22:03.396 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.5380000000000002e-06 2024-08-19T14:22:03.397 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:03.397 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:03.397 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:22:03.397 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:03.359443+0000", 2024-08-19T14:22:03.397 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.6940000000000001e-06 2024-08-19T14:22:03.397 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:03.397 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:03.397 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:22:03.397 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:03.365887+0000", 2024-08-19T14:22:03.397 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0064441200000000002 2024-08-19T14:22:03.397 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:03.397 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:03.397 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:22:03.398 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:03.366052+0000", 2024-08-19T14:22:03.398 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00016468 2024-08-19T14:22:03.398 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:03.398 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:03.398 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 4,5", 2024-08-19T14:22:03.398 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:03.366144+0000", 2024-08-19T14:22:03.398 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.2419999999999999e-05 2024-08-19T14:22:03.398 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:03.398 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:22:03.398 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:03.398 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:03.398 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:03.398 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:130521 4.9 4:9f151a75:::benchmark_data_smithi045_71562_object130520:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e371)", 2024-08-19T14:22:03.399 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:22:03.359736+0000", 2024-08-19T14:22:03.399 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.011789739, 2024-08-19T14:22:03.399 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.012019442999999999, 2024-08-19T14:22:03.399 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:22:03.399 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:22:03.399 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "waiting for sub ops", 2024-08-19T14:22:03.399 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:22:03.399 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:22:03.399 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:22:03.399 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 130521 2024-08-19T14:22:03.399 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:03.399 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:22:03.399 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:03.399 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:22:03.400 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:03.359736+0000", 2024-08-19T14:22:03.400 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:03.400 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:03.400 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:03.400 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:22:03.400 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:03.359736+0000", 2024-08-19T14:22:03.400 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:03.400 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:03.400 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:03.400 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:22:03.400 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:03.359737+0000", 2024-08-19T14:22:03.400 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.5799999999999998e-07 2024-08-19T14:22:03.400 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:03.400 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:03.401 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:22:03.401 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:03.359780+0000", 2024-08-19T14:22:03.401 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.3127000000000002e-05 2024-08-19T14:22:03.401 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:03.401 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:03.401 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:22:03.401 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:03.359801+0000", 2024-08-19T14:22:03.401 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0771000000000002e-05 2024-08-19T14:22:03.401 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:03.401 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:03.401 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:22:03.401 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:03.359808+0000", 2024-08-19T14:22:03.401 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4839999999999998e-06 2024-08-19T14:22:03.401 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:03.402 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:03.402 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:22:03.402 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:03.369298+0000", 2024-08-19T14:22:03.402 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0094897750000000006 2024-08-19T14:22:03.402 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:03.402 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:03.402 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:22:03.402 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:03.369475+0000", 2024-08-19T14:22:03.402 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00017734900000000001 2024-08-19T14:22:03.402 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:03.402 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:03.402 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,3", 2024-08-19T14:22:03.402 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:03.369558+0000", 2024-08-19T14:22:03.402 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.2520000000000003e-05 2024-08-19T14:22:03.403 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:03.403 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:22:03.403 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:03.403 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:03.403 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:03.403 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:130526 4.1 4:8f49e6d5:::benchmark_data_smithi045_71562_object130525:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e371)", 2024-08-19T14:22:03.403 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:22:03.361671+0000", 2024-08-19T14:22:03.403 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.0098544979999999997, 2024-08-19T14:22:03.403 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.010123721, 2024-08-19T14:22:03.403 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:22:03.403 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:22:03.403 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "waiting for sub ops", 2024-08-19T14:22:03.403 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:22:03.403 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:22:03.404 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:22:03.404 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 130526 2024-08-19T14:22:03.404 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:03.404 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:22:03.404 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:03.404 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:22:03.404 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:03.361671+0000", 2024-08-19T14:22:03.404 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:03.404 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:03.404 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:03.404 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:22:03.404 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:03.361671+0000", 2024-08-19T14:22:03.404 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:03.404 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:03.404 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:03.405 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:22:03.405 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:03.361672+0000", 2024-08-19T14:22:03.405 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.158e-06 2024-08-19T14:22:03.405 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:03.405 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:03.405 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:22:03.405 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:03.361732+0000", 2024-08-19T14:22:03.405 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.9539e-05 2024-08-19T14:22:03.405 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:03.405 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:03.405 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:22:03.405 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:03.361744+0000", 2024-08-19T14:22:03.405 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2062000000000001e-05 2024-08-19T14:22:03.405 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:03.406 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:03.406 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:22:03.406 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:03.361751+0000", 2024-08-19T14:22:03.406 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.2880000000000004e-06 2024-08-19T14:22:03.406 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:03.406 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:03.406 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:22:03.406 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:03.369322+0000", 2024-08-19T14:22:03.406 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0075707309999999998 2024-08-19T14:22:03.406 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:03.406 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:03.406 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:22:03.406 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:03.369470+0000", 2024-08-19T14:22:03.406 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014835400000000001 2024-08-19T14:22:03.407 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:03.407 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:03.407 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 4,5", 2024-08-19T14:22:03.407 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:03.369769+0000", 2024-08-19T14:22:03.407 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00029828800000000001 2024-08-19T14:22:03.407 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:03.407 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:22:03.407 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:03.407 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:03.407 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:03.407 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:130527 4.9 4.b65b149 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e371)", 2024-08-19T14:22:03.407 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:22:03.361766+0000", 2024-08-19T14:22:03.407 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.0097599769999999995, 2024-08-19T14:22:03.408 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.010068598, 2024-08-19T14:22:03.408 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:22:03.408 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:22:03.408 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "queued for pg", 2024-08-19T14:22:03.408 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:22:03.408 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:22:03.408 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:22:03.408 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 130527 2024-08-19T14:22:03.408 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:03.408 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:22:03.408 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:03.408 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:22:03.408 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:03.361766+0000", 2024-08-19T14:22:03.408 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:03.409 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:03.409 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:03.409 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:22:03.409 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:03.361766+0000", 2024-08-19T14:22:03.409 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:03.409 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:03.409 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:03.409 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:22:03.409 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:03.361766+0000", 2024-08-19T14:22:03.409 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.06e-07 2024-08-19T14:22:03.409 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:03.409 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:03.409 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:22:03.409 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:03.361808+0000", 2024-08-19T14:22:03.410 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.1953000000000003e-05 2024-08-19T14:22:03.410 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:03.410 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:03.410 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:22:03.410 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:03.361817+0000", 2024-08-19T14:22:03.410 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.918e-06 2024-08-19T14:22:03.410 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:03.410 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:03.410 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:22:03.410 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:03.361823+0000", 2024-08-19T14:22:03.410 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.6219999999999997e-06 2024-08-19T14:22:03.410 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:03.410 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:22:03.410 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:03.411 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:03.411 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:03.411 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.25333.0:130523 4.f e371/335 4:fedf71a6:::benchmark_data_smithi045_71562_object130522:head v 371'8102, mlcod=371'8100)", 2024-08-19T14:22:03.411 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:22:03.365429+0000", 2024-08-19T14:22:03.411 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.0060962459999999996, 2024-08-19T14:22:03.411 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0064347349999999996, 2024-08-19T14:22:03.411 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:22:03.411 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:22:03.411 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-08-19T14:22:03.411 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:22:03.411 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:03.411 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:22:03.411 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:03.365429+0000", 2024-08-19T14:22:03.411 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:03.412 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:03.412 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:03.412 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:22:03.412 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:03.365429+0000", 2024-08-19T14:22:03.412 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:03.412 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:03.412 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:03.412 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:22:03.412 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:03.365430+0000", 2024-08-19T14:22:03.412 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4700000000000001e-07 2024-08-19T14:22:03.412 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:03.412 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:03.413 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:22:03.413 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:03.365484+0000", 2024-08-19T14:22:03.413 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4156999999999997e-05 2024-08-19T14:22:03.413 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:03.413 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:03.413 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:22:03.413 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:03.365494+0000", 2024-08-19T14:22:03.413 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.5699999999999999e-06 2024-08-19T14:22:03.413 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:03.413 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:03.413 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:22:03.413 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:03.365501+0000", 2024-08-19T14:22:03.413 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.635e-06 2024-08-19T14:22:03.413 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:03.414 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:03.414 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:22:03.414 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:03.365555+0000", 2024-08-19T14:22:03.414 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4023000000000001e-05 2024-08-19T14:22:03.414 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:03.414 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:03.414 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:22:03.414 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:03.365578+0000", 2024-08-19T14:22:03.414 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2915999999999999e-05 2024-08-19T14:22:03.414 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:03.414 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:22:03.414 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:03.414 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:03.415 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-08-19T14:22:03.415 INFO:teuthology.orchestra.run.smithi045.stdout: "num_ops": 9 2024-08-19T14:22:03.415 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:22:03.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:03.416 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:03.417 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:22:03.518 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:22:03.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:03.561 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:03.561 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:22:03.662 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:22:03.698 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:03.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:03.700 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:22:03.801 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:22:03.831 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:03.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:03.832 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:22:03.933 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:22:03.960 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:03.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:03.961 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:22:04.052 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 90 16 131172 131156 91.0729 57.25 0.00441992 0.010973 2024-08-19T14:22:04.062 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:22:04.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:04.084 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:04.084 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:22:04.186 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:22:04.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:04.212 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:04.213 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:22:04.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:04 smithi045 ceph-mon[20143]: 1.17 scrub starts 2024-08-19T14:22:04.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:04 smithi045 ceph-mon[20143]: 1.17 scrub ok 2024-08-19T14:22:04.314 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:22:04.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:04.337 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:04.337 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:22:04.352 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-08-19T14:22:04.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:04 smithi031 ceph-mon[16336]: 1.17 scrub starts 2024-08-19T14:22:04.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:04 smithi031 ceph-mon[16336]: 1.17 scrub ok 2024-08-19T14:22:04.437 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:22:04.459 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:04.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:04.460 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:22:04.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:04 smithi032 ceph-mon[20004]: 1.17 scrub starts 2024-08-19T14:22:04.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:04 smithi032 ceph-mon[20004]: 1.17 scrub ok 2024-08-19T14:22:04.561 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:22:04.582 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:04.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:04.583 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:22:04.605 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:22:04.684 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:22:04.705 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:04.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:04.707 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:22:04.808 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:22:04.829 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:04.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:04.830 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:22:04.931 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:22:04.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:04.953 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:04.953 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:22:05.041 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-19T14:22:05.052 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Total time run: 90.0627 2024-08-19T14:22:05.052 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Total writes made: 131172 2024-08-19T14:22:05.052 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Write size: 65536 2024-08-19T14:22:05.052 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Object size: 65536 2024-08-19T14:22:05.052 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Bandwidth (MB/sec): 91.0283 2024-08-19T14:22:05.052 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Stddev Bandwidth: 44.1621 2024-08-19T14:22:05.052 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Max bandwidth (MB/sec): 248.875 2024-08-19T14:22:05.052 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Min bandwidth (MB/sec): 0 2024-08-19T14:22:05.052 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Average IOPS: 1456 2024-08-19T14:22:05.052 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Stddev IOPS: 706.601 2024-08-19T14:22:05.052 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Max IOPS: 3982 2024-08-19T14:22:05.053 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Min IOPS: 0 2024-08-19T14:22:05.053 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Average Latency(s): 0.0109813 2024-08-19T14:22:05.053 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Stddev Latency(s): 0.0312255 2024-08-19T14:22:05.053 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Max latency(s): 2.42694 2024-08-19T14:22:05.053 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Min latency(s): 0.00106793 2024-08-19T14:22:05.054 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:22:05.059 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Cleaning up (deleting benchmark objects) 2024-08-19T14:22:05.082 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:05.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:05.083 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:22:05.111 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:22:05.111 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [], 2024-08-19T14:22:05.111 INFO:teuthology.orchestra.run.smithi045.stdout: "complaint_time": 30, 2024-08-19T14:22:05.111 INFO:teuthology.orchestra.run.smithi045.stdout: "num_blocked_ops": 0 2024-08-19T14:22:05.111 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:22:05.184 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:22:05.207 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:05.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:05.208 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:22:05.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:05 smithi045 ceph-mon[20143]: pgmap v836: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 7.8 GiB data, 27 GiB used, 777 GiB / 805 GiB avail; 50 MiB/s wr, 804 op/s; 2731/384411 objects misplaced (0.710%); 1.2 MiB/s, 19 objects/s recovering 2024-08-19T14:22:05.310 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:22:05.336 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:05.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:05.337 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:22:05.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:05 smithi031 ceph-mon[16336]: pgmap v836: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 7.8 GiB data, 27 GiB used, 777 GiB / 805 GiB avail; 50 MiB/s wr, 804 op/s; 2731/384411 objects misplaced (0.710%); 1.2 MiB/s, 19 objects/s recovering 2024-08-19T14:22:05.437 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:22:05.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:05.463 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:05.463 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:22:05.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:05 smithi032 ceph-mon[20004]: pgmap v836: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 7.8 GiB data, 27 GiB used, 777 GiB / 805 GiB avail; 50 MiB/s wr, 804 op/s; 2731/384411 objects misplaced (0.710%); 1.2 MiB/s, 19 objects/s recovering 2024-08-19T14:22:05.564 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:22:05.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:05.602 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:05.602 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:22:05.702 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:22:05.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:05.734 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:05.734 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:22:05.835 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:22:05.873 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:05.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:05.875 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:22:05.907 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-08-19T14:22:05.976 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:22:06.004 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:06.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:06.005 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:22:06.106 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:22:06.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:06.141 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:06.141 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:22:06.242 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:22:06.247 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:22:06.274 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:06.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:06.275 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:22:06.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:06 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:22:06.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:06 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:22:06.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:06 smithi045 ceph-mon[20143]: osdmap e372: 12 total, 12 up, 9 in 2024-08-19T14:22:06.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:06 smithi045 ceph-mon[20143]: 4.3 scrub starts 2024-08-19T14:22:06.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:06 smithi045 ceph-mon[20143]: 4.c scrub starts 2024-08-19T14:22:06.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:06 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:22:06.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:06 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:22:06.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:06 smithi031 ceph-mon[16336]: osdmap e372: 12 total, 12 up, 9 in 2024-08-19T14:22:06.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:06 smithi031 ceph-mon[16336]: 4.3 scrub starts 2024-08-19T14:22:06.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:06 smithi031 ceph-mon[16336]: 4.c scrub starts 2024-08-19T14:22:06.376 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:22:06.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:06.401 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:06.402 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:22:06.503 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:22:06.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:06 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:22:06.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:06 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:22:06.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:06 smithi032 ceph-mon[20004]: osdmap e372: 12 total, 12 up, 9 in 2024-08-19T14:22:06.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:06 smithi032 ceph-mon[20004]: 4.3 scrub starts 2024-08-19T14:22:06.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:06 smithi032 ceph-mon[20004]: 4.c scrub starts 2024-08-19T14:22:06.537 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:06.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:06.538 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:22:06.642 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:22:06.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:06.675 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:06.675 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:22:06.776 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:22:06.806 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:22:06.806 INFO:teuthology.orchestra.run.smithi045.stdout: "size": 20, 2024-08-19T14:22:06.806 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 600, 2024-08-19T14:22:06.806 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [ 2024-08-19T14:22:06.806 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.807 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24580.0:159035 2.c 2:32456cd8:::benchmark_data_smithi045_46415_object35754:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e167)", 2024-08-19T14:22:06.807 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:14:59.419140+0000", 2024-08-19T14:22:06.807 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 427.38447446999999, 2024-08-19T14:22:06.807 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.21467188200000001, 2024-08-19T14:22:06.807 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:22:06.807 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:22:06.807 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:22:06.807 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:22:06.807 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24580", 2024-08-19T14:22:06.807 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:22:06.807 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 159035 2024-08-19T14:22:06.807 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.807 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:22:06.807 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.808 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:22:06.808 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.419140+0000", 2024-08-19T14:22:06.808 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:06.808 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.808 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.808 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:22:06.808 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.419140+0000", 2024-08-19T14:22:06.808 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:06.808 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.808 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.808 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:22:06.808 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.419141+0000", 2024-08-19T14:22:06.808 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4079999999999999e-06 2024-08-19T14:22:06.808 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.809 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.809 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:22:06.809 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.419146+0000", 2024-08-19T14:22:06.809 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0610000000000003e-06 2024-08-19T14:22:06.809 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.809 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.809 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:22:06.809 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.419163+0000", 2024-08-19T14:22:06.809 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6770999999999999e-05 2024-08-19T14:22:06.810 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.810 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.810 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:22:06.810 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.419172+0000", 2024-08-19T14:22:06.810 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.276e-06 2024-08-19T14:22:06.810 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.810 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.810 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:22:06.810 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.419223+0000", 2024-08-19T14:22:06.811 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1588999999999997e-05 2024-08-19T14:22:06.811 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.811 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.811 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:22:06.811 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.419320+0000", 2024-08-19T14:22:06.811 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.7237000000000004e-05 2024-08-19T14:22:06.811 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.811 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.811 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,1", 2024-08-19T14:22:06.811 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.419386+0000", 2024-08-19T14:22:06.811 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.6093999999999994e-05 2024-08-19T14:22:06.811 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.811 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.812 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:22:06.812 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.422728+0000", 2024-08-19T14:22:06.812 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0033416489999999999 2024-08-19T14:22:06.812 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.812 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.812 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:22:06.813 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.439269+0000", 2024-08-19T14:22:06.813 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.016540764999999999 2024-08-19T14:22:06.813 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.813 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.813 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:22:06.813 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.633710+0000", 2024-08-19T14:22:06.813 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.194440742 2024-08-19T14:22:06.813 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.813 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.813 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:22:06.813 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.633767+0000", 2024-08-19T14:22:06.813 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7210999999999998e-05 2024-08-19T14:22:06.813 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.814 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.814 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:22:06.814 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.633812+0000", 2024-08-19T14:22:06.814 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.5079e-05 2024-08-19T14:22:06.814 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:06.814 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:22:06.814 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:06.814 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.814 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.814 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24580.0:159041 2.c 2:326ec62a:::benchmark_data_smithi045_46415_object35760:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e167)", 2024-08-19T14:22:06.814 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:14:59.424792+0000", 2024-08-19T14:22:06.814 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 427.37882196599998, 2024-08-19T14:22:06.814 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.25135169800000001, 2024-08-19T14:22:06.814 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:22:06.815 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:22:06.815 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:22:06.815 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:22:06.815 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24580", 2024-08-19T14:22:06.815 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:22:06.815 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 159041 2024-08-19T14:22:06.815 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.815 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:22:06.815 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.815 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:22:06.815 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.424792+0000", 2024-08-19T14:22:06.815 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:06.815 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.816 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.816 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:22:06.816 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.424792+0000", 2024-08-19T14:22:06.816 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:06.816 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.816 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.816 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:22:06.816 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.424794+0000", 2024-08-19T14:22:06.816 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.883e-06 2024-08-19T14:22:06.816 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.816 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.816 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:22:06.816 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.424800+0000", 2024-08-19T14:22:06.816 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4349999999999996e-06 2024-08-19T14:22:06.817 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.817 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.817 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:22:06.817 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.424813+0000", 2024-08-19T14:22:06.817 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3619e-05 2024-08-19T14:22:06.817 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.817 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.817 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:22:06.817 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.424821+0000", 2024-08-19T14:22:06.817 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4649999999999997e-06 2024-08-19T14:22:06.817 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.817 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.817 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:22:06.817 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.424864+0000", 2024-08-19T14:22:06.818 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.2786e-05 2024-08-19T14:22:06.818 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.818 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.818 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:22:06.818 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.424944+0000", 2024-08-19T14:22:06.818 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.0628000000000001e-05 2024-08-19T14:22:06.818 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.818 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.818 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,1", 2024-08-19T14:22:06.818 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.425008+0000", 2024-08-19T14:22:06.818 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.3223000000000001e-05 2024-08-19T14:22:06.818 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.818 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.818 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:22:06.819 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.429480+0000", 2024-08-19T14:22:06.819 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0044724830000000002 2024-08-19T14:22:06.819 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.819 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.819 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:22:06.819 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.452535+0000", 2024-08-19T14:22:06.819 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.023055386000000001 2024-08-19T14:22:06.819 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.819 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.819 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:22:06.819 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.676047+0000", 2024-08-19T14:22:06.819 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.22351151399999999 2024-08-19T14:22:06.819 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.819 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.820 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:22:06.820 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.676101+0000", 2024-08-19T14:22:06.820 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4237000000000003e-05 2024-08-19T14:22:06.820 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.820 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.820 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:22:06.820 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.676144+0000", 2024-08-19T14:22:06.820 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.3038999999999999e-05 2024-08-19T14:22:06.820 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:06.820 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:22:06.820 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:06.820 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.820 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.820 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24580.0:159045 2.c 2:30c9429c:::benchmark_data_smithi045_46415_object35764:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e167)", 2024-08-19T14:22:06.821 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:14:59.429858+0000", 2024-08-19T14:22:06.821 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 427.37375601299999, 2024-08-19T14:22:06.821 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.24643119899999999, 2024-08-19T14:22:06.821 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:22:06.821 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:22:06.821 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:22:06.821 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:22:06.821 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24580", 2024-08-19T14:22:06.821 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:22:06.821 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 159045 2024-08-19T14:22:06.821 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.821 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:22:06.821 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.821 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:22:06.822 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.429858+0000", 2024-08-19T14:22:06.822 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:06.822 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.822 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.822 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:22:06.822 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.429858+0000", 2024-08-19T14:22:06.822 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:06.822 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.822 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.822 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:22:06.822 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.429860+0000", 2024-08-19T14:22:06.822 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.531e-06 2024-08-19T14:22:06.822 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.822 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.823 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:22:06.823 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.429865+0000", 2024-08-19T14:22:06.823 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0989999999999998e-06 2024-08-19T14:22:06.823 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.823 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.823 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:22:06.823 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.429875+0000", 2024-08-19T14:22:06.823 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0043e-05 2024-08-19T14:22:06.823 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.823 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.823 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:22:06.823 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.429891+0000", 2024-08-19T14:22:06.823 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5539000000000001e-05 2024-08-19T14:22:06.823 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.824 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.824 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:22:06.824 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.429951+0000", 2024-08-19T14:22:06.824 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.0528000000000001e-05 2024-08-19T14:22:06.824 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.824 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.824 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:22:06.824 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.430061+0000", 2024-08-19T14:22:06.824 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000110203 2024-08-19T14:22:06.824 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.824 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.824 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,1", 2024-08-19T14:22:06.824 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.430129+0000", 2024-08-19T14:22:06.824 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.7970000000000001e-05 2024-08-19T14:22:06.825 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.825 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.825 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:22:06.825 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.435692+0000", 2024-08-19T14:22:06.825 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0055628550000000002 2024-08-19T14:22:06.825 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.825 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.825 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:22:06.825 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.459086+0000", 2024-08-19T14:22:06.825 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.023393942000000001 2024-08-19T14:22:06.825 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.825 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.825 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:22:06.825 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.676201+0000", 2024-08-19T14:22:06.826 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.21711492499999999 2024-08-19T14:22:06.826 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.826 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.826 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:22:06.826 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.676252+0000", 2024-08-19T14:22:06.826 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1334e-05 2024-08-19T14:22:06.826 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.826 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.826 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:22:06.826 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.676290+0000", 2024-08-19T14:22:06.826 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.7230000000000001e-05 2024-08-19T14:22:06.826 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:06.826 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:22:06.827 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:06.827 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.827 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.827 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25057.0:63327 3.1e 3:79dc30b5:::benchmark_data_smithi045_59973_object63326:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e244)", 2024-08-19T14:22:06.827 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:17:33.888774+0000", 2024-08-19T14:22:06.827 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 272.914840008, 2024-08-19T14:22:06.827 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0936514980000001, 2024-08-19T14:22:06.827 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:22:06.827 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:22:06.827 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-08-19T14:22:06.827 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:22:06.827 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25057", 2024-08-19T14:22:06.827 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:22:06.827 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 63327 2024-08-19T14:22:06.827 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.828 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:22:06.828 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.828 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:22:06.828 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:33.888774+0000", 2024-08-19T14:22:06.828 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:06.828 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.828 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.828 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:22:06.828 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:33.888774+0000", 2024-08-19T14:22:06.828 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:06.828 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.828 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.828 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:22:06.828 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:33.888776+0000", 2024-08-19T14:22:06.829 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9190000000000002e-06 2024-08-19T14:22:06.829 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.829 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.829 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:22:06.829 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:33.888802+0000", 2024-08-19T14:22:06.829 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.5134e-05 2024-08-19T14:22:06.829 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.829 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.829 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:22:06.829 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:33.888817+0000", 2024-08-19T14:22:06.829 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5753e-05 2024-08-19T14:22:06.829 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.829 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.829 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:22:06.830 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:33.888827+0000", 2024-08-19T14:22:06.830 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0117e-05 2024-08-19T14:22:06.830 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.830 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.830 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:22:06.830 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:33.969428+0000", 2024-08-19T14:22:06.830 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.080600464999999996 2024-08-19T14:22:06.830 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.830 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.830 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for peered", 2024-08-19T14:22:06.830 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:33.969436+0000", 2024-08-19T14:22:06.830 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.8730000000000007e-06 2024-08-19T14:22:06.830 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.831 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.831 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:22:06.831 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:34.982371+0000", 2024-08-19T14:22:06.831 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.012934805 2024-08-19T14:22:06.831 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.831 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.831 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:22:06.831 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:34.982426+0000", 2024-08-19T14:22:06.831 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.5432000000000001e-05 2024-08-19T14:22:06.831 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:06.831 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:22:06.831 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:06.831 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.831 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.831 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:22707 4.5 4:aa5c4d32:::benchmark_data_smithi045_71562_object22706:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-08-19T14:22:06.832 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:20:41.355284+0000", 2024-08-19T14:22:06.832 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 85.448330142000003, 2024-08-19T14:22:06.832 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.32540933900000002, 2024-08-19T14:22:06.832 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:22:06.832 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:22:06.832 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:22:06.832 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:22:06.832 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:22:06.832 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:22:06.832 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 22707 2024-08-19T14:22:06.832 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.832 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:22:06.832 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.832 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:22:06.833 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.355284+0000", 2024-08-19T14:22:06.833 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:06.833 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.833 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.833 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:22:06.833 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.355284+0000", 2024-08-19T14:22:06.833 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:06.833 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.833 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.833 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:22:06.833 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.355285+0000", 2024-08-19T14:22:06.833 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.2699999999999998e-07 2024-08-19T14:22:06.833 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.833 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.834 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:22:06.834 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.355304+0000", 2024-08-19T14:22:06.834 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8834e-05 2024-08-19T14:22:06.834 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.834 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.834 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:22:06.834 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.355315+0000", 2024-08-19T14:22:06.834 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1388e-05 2024-08-19T14:22:06.834 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.834 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.834 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:22:06.834 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.355326+0000", 2024-08-19T14:22:06.834 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0101999999999999e-05 2024-08-19T14:22:06.834 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.835 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.835 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:22:06.835 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.355381+0000", 2024-08-19T14:22:06.835 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.5114999999999997e-05 2024-08-19T14:22:06.835 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.835 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.836 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:22:06.836 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.355481+0000", 2024-08-19T14:22:06.837 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000100448 2024-08-19T14:22:06.837 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.837 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.837 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,7", 2024-08-19T14:22:06.837 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.355563+0000", 2024-08-19T14:22:06.837 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.1891999999999995e-05 2024-08-19T14:22:06.837 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.837 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.837 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:22:06.837 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.356429+0000", 2024-08-19T14:22:06.837 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00086571700000000001 2024-08-19T14:22:06.837 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.837 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.837 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:22:06.838 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.356898+0000", 2024-08-19T14:22:06.838 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000469171 2024-08-19T14:22:06.838 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.838 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.838 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:22:06.838 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.680624+0000", 2024-08-19T14:22:06.838 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.32372564599999998 2024-08-19T14:22:06.838 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.838 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.838 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:22:06.838 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.680661+0000", 2024-08-19T14:22:06.838 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.7589999999999998e-05 2024-08-19T14:22:06.838 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.839 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.839 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:22:06.839 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.680694+0000", 2024-08-19T14:22:06.839 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.2509000000000003e-05 2024-08-19T14:22:06.839 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:06.839 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:22:06.839 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:06.839 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.839 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.839 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:22723 4.5 4:a03b0b8c:::benchmark_data_smithi045_71562_object22722:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-08-19T14:22:06.839 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:20:41.360250+0000", 2024-08-19T14:22:06.839 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 85.443364809000002, 2024-08-19T14:22:06.839 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.32180027, 2024-08-19T14:22:06.839 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:22:06.839 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:22:06.840 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:22:06.840 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:22:06.840 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:22:06.840 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:22:06.840 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 22723 2024-08-19T14:22:06.840 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.840 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:22:06.840 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.840 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:22:06.840 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.360250+0000", 2024-08-19T14:22:06.840 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:06.840 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.840 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.840 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:22:06.841 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.360250+0000", 2024-08-19T14:22:06.841 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:06.841 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.841 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.841 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:22:06.841 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.360251+0000", 2024-08-19T14:22:06.841 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7740000000000001e-06 2024-08-19T14:22:06.841 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.841 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.841 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:22:06.841 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.360271+0000", 2024-08-19T14:22:06.841 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9755999999999999e-05 2024-08-19T14:22:06.841 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.841 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.842 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:22:06.842 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.360280+0000", 2024-08-19T14:22:06.842 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.268e-06 2024-08-19T14:22:06.842 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.842 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.842 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:22:06.842 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.360286+0000", 2024-08-19T14:22:06.842 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2700000000000004e-06 2024-08-19T14:22:06.842 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.842 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.842 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:22:06.842 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.360339+0000", 2024-08-19T14:22:06.842 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.3644999999999999e-05 2024-08-19T14:22:06.843 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.843 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.846 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:22:06.846 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.360424+0000", 2024-08-19T14:22:06.846 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.4493999999999994e-05 2024-08-19T14:22:06.846 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.846 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.846 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,7", 2024-08-19T14:22:06.846 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.360529+0000", 2024-08-19T14:22:06.846 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00010503200000000001 2024-08-19T14:22:06.846 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.846 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.847 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:22:06.847 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.362147+0000", 2024-08-19T14:22:06.847 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001617718 2024-08-19T14:22:06.847 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.847 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.847 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:22:06.847 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.362188+0000", 2024-08-19T14:22:06.847 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.1553e-05 2024-08-19T14:22:06.847 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.847 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.847 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:22:06.847 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.681950+0000", 2024-08-19T14:22:06.847 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.31976217299999998 2024-08-19T14:22:06.847 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.848 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.848 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:22:06.848 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.682008+0000", 2024-08-19T14:22:06.848 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7494000000000002e-05 2024-08-19T14:22:06.848 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.848 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.848 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:22:06.848 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.682050+0000", 2024-08-19T14:22:06.848 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.2092999999999998e-05 2024-08-19T14:22:06.848 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:06.848 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:22:06.848 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:06.848 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.848 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.848 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:22725 4.5 4:adb3fee7:::benchmark_data_smithi045_71562_object22724:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-08-19T14:22:06.849 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:20:41.361334+0000", 2024-08-19T14:22:06.849 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 85.442280835000005, 2024-08-19T14:22:06.849 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.32179169200000002, 2024-08-19T14:22:06.849 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:22:06.849 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:22:06.849 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:22:06.849 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:22:06.849 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:22:06.849 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:22:06.849 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 22725 2024-08-19T14:22:06.849 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.849 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:22:06.849 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.849 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:22:06.850 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.361334+0000", 2024-08-19T14:22:06.850 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:06.850 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.850 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.850 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:22:06.850 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.361334+0000", 2024-08-19T14:22:06.850 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:06.850 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.850 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.850 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:22:06.850 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.361334+0000", 2024-08-19T14:22:06.850 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.61e-07 2024-08-19T14:22:06.850 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.850 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.851 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:22:06.851 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.361353+0000", 2024-08-19T14:22:06.851 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9074999999999999e-05 2024-08-19T14:22:06.851 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.851 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.851 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:22:06.851 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.361365+0000", 2024-08-19T14:22:06.851 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1498000000000001e-05 2024-08-19T14:22:06.851 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.851 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.851 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:22:06.851 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.361371+0000", 2024-08-19T14:22:06.851 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.1290000000000001e-06 2024-08-19T14:22:06.851 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.852 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.852 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:22:06.852 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.361415+0000", 2024-08-19T14:22:06.852 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.3445999999999997e-05 2024-08-19T14:22:06.852 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.852 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.852 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:22:06.852 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.361542+0000", 2024-08-19T14:22:06.852 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012710400000000001 2024-08-19T14:22:06.852 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.852 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.852 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,7", 2024-08-19T14:22:06.852 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.361632+0000", 2024-08-19T14:22:06.852 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.9963999999999994e-05 2024-08-19T14:22:06.853 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.853 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.853 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:22:06.853 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.362288+0000", 2024-08-19T14:22:06.853 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00065625500000000001 2024-08-19T14:22:06.853 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.853 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.853 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:22:06.853 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.362699+0000", 2024-08-19T14:22:06.853 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00041120600000000001 2024-08-19T14:22:06.853 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.853 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.853 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:22:06.853 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.683027+0000", 2024-08-19T14:22:06.854 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.32032799699999998 2024-08-19T14:22:06.854 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.854 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.854 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:22:06.854 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.683084+0000", 2024-08-19T14:22:06.854 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7054e-05 2024-08-19T14:22:06.854 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.854 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.854 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:22:06.854 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.683125+0000", 2024-08-19T14:22:06.854 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.1202999999999998e-05 2024-08-19T14:22:06.854 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:06.854 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:22:06.854 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:06.854 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.855 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.855 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:32454 4.3 4:c5fede70:::benchmark_data_smithi045_71562_object32453:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e343)", 2024-08-19T14:22:06.855 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:20:45.676484+0000", 2024-08-19T14:22:06.855 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 81.127130882000003, 2024-08-19T14:22:06.855 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.212526989, 2024-08-19T14:22:06.855 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:22:06.855 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:22:06.855 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:22:06.855 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:22:06.855 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:22:06.855 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:22:06.855 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 32454 2024-08-19T14:22:06.855 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.855 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:22:06.856 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.856 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:22:06.856 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.676484+0000", 2024-08-19T14:22:06.856 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:06.856 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.856 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.856 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:22:06.856 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.676484+0000", 2024-08-19T14:22:06.856 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:06.856 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.856 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.856 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:22:06.856 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.676484+0000", 2024-08-19T14:22:06.856 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1900000000000003e-07 2024-08-19T14:22:06.857 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.857 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.857 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:22:06.857 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.676506+0000", 2024-08-19T14:22:06.857 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1515e-05 2024-08-19T14:22:06.857 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.857 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.857 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:22:06.857 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.676513+0000", 2024-08-19T14:22:06.857 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.7570000000000004e-06 2024-08-19T14:22:06.857 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.857 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.857 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:22:06.857 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.676519+0000", 2024-08-19T14:22:06.858 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4419999999999997e-06 2024-08-19T14:22:06.858 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.858 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.858 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:22:06.858 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.676567+0000", 2024-08-19T14:22:06.858 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.8439999999999997e-05 2024-08-19T14:22:06.858 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.858 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.858 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:22:06.858 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.676672+0000", 2024-08-19T14:22:06.858 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000104991 2024-08-19T14:22:06.858 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.858 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.858 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,7", 2024-08-19T14:22:06.859 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.676835+0000", 2024-08-19T14:22:06.859 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00016259600000000001 2024-08-19T14:22:06.859 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.859 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.859 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:22:06.859 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.678852+0000", 2024-08-19T14:22:06.859 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.002017511 2024-08-19T14:22:06.859 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.859 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.859 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:22:06.859 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.678903+0000", 2024-08-19T14:22:06.859 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0387999999999999e-05 2024-08-19T14:22:06.859 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.859 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.860 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:22:06.860 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.888915+0000", 2024-08-19T14:22:06.860 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.21001176899999999 2024-08-19T14:22:06.860 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.860 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.860 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:22:06.860 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.888969+0000", 2024-08-19T14:22:06.860 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.401e-05 2024-08-19T14:22:06.860 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.860 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.860 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:22:06.860 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.889011+0000", 2024-08-19T14:22:06.860 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.2051000000000001e-05 2024-08-19T14:22:06.860 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:06.861 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:22:06.861 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:06.861 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.861 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.861 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:32457 4.9 4:9a19dbd1:::benchmark_data_smithi045_71562_object32456:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e343)", 2024-08-19T14:22:06.861 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:20:45.677306+0000", 2024-08-19T14:22:06.861 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 81.126308796999993, 2024-08-19T14:22:06.861 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.21092156100000001, 2024-08-19T14:22:06.861 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:22:06.861 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:22:06.861 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:22:06.861 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:22:06.861 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:22:06.861 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:22:06.862 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 32457 2024-08-19T14:22:06.862 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.862 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:22:06.862 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.862 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:22:06.862 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.677306+0000", 2024-08-19T14:22:06.862 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:06.862 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.862 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.862 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:22:06.862 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.677306+0000", 2024-08-19T14:22:06.862 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:06.862 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.863 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.863 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:22:06.863 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.677307+0000", 2024-08-19T14:22:06.863 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.026e-06 2024-08-19T14:22:06.863 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.863 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.863 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:22:06.863 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.677329+0000", 2024-08-19T14:22:06.863 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2163000000000001e-05 2024-08-19T14:22:06.863 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.863 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.863 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:22:06.863 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.677346+0000", 2024-08-19T14:22:06.863 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6858999999999999e-05 2024-08-19T14:22:06.863 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.864 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.864 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:22:06.864 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.677354+0000", 2024-08-19T14:22:06.864 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.2609999999999993e-06 2024-08-19T14:22:06.864 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.864 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.864 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:22:06.864 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.678032+0000", 2024-08-19T14:22:06.864 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00067848099999999998 2024-08-19T14:22:06.864 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.864 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.864 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:22:06.864 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.678159+0000", 2024-08-19T14:22:06.864 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000126904 2024-08-19T14:22:06.865 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.865 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.865 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,3", 2024-08-19T14:22:06.865 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.678310+0000", 2024-08-19T14:22:06.865 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015097799999999999 2024-08-19T14:22:06.865 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.865 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.865 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:22:06.865 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.680731+0000", 2024-08-19T14:22:06.865 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0024209969999999998 2024-08-19T14:22:06.865 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.865 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.865 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:22:06.865 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.681072+0000", 2024-08-19T14:22:06.866 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00034038500000000001 2024-08-19T14:22:06.866 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.866 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.866 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:22:06.866 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.888152+0000", 2024-08-19T14:22:06.866 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20708003799999999 2024-08-19T14:22:06.866 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.866 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.866 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:22:06.866 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.888194+0000", 2024-08-19T14:22:06.866 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.1817999999999998e-05 2024-08-19T14:22:06.866 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.866 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.867 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:22:06.867 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.888227+0000", 2024-08-19T14:22:06.867 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.3651e-05 2024-08-19T14:22:06.867 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:06.867 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:22:06.867 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:06.867 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.867 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.867 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:32458 4.9 4:9e35a421:::benchmark_data_smithi045_71562_object32457:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e343)", 2024-08-19T14:22:06.867 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:20:45.677368+0000", 2024-08-19T14:22:06.867 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 81.126246304000006, 2024-08-19T14:22:06.867 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.21210032300000001, 2024-08-19T14:22:06.867 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:22:06.867 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:22:06.868 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:22:06.868 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:22:06.868 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:22:06.868 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:22:06.868 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 32458 2024-08-19T14:22:06.868 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.868 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:22:06.868 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.868 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:22:06.868 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.677368+0000", 2024-08-19T14:22:06.868 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:06.868 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.868 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.868 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:22:06.869 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.677368+0000", 2024-08-19T14:22:06.869 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:06.869 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.869 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.869 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:22:06.869 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.677369+0000", 2024-08-19T14:22:06.869 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.9999999999999997e-07 2024-08-19T14:22:06.869 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.869 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.869 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:22:06.869 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.677399+0000", 2024-08-19T14:22:06.869 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.0094e-05 2024-08-19T14:22:06.869 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.869 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.869 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:22:06.870 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.677413+0000", 2024-08-19T14:22:06.870 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3869999999999999e-05 2024-08-19T14:22:06.870 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.870 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.870 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:22:06.870 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.677420+0000", 2024-08-19T14:22:06.870 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.0679999999999999e-06 2024-08-19T14:22:06.870 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.870 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.870 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:22:06.870 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.679186+0000", 2024-08-19T14:22:06.870 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0017659889999999999 2024-08-19T14:22:06.870 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.871 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.871 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:22:06.871 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.679263+0000", 2024-08-19T14:22:06.871 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.7045000000000005e-05 2024-08-19T14:22:06.871 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.871 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.871 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,3", 2024-08-19T14:22:06.871 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.679340+0000", 2024-08-19T14:22:06.871 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.7371000000000002e-05 2024-08-19T14:22:06.871 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.871 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.871 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:22:06.871 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.680788+0000", 2024-08-19T14:22:06.871 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0014473439999999999 2024-08-19T14:22:06.872 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.872 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.872 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:22:06.872 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.681667+0000", 2024-08-19T14:22:06.872 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00087960100000000004 2024-08-19T14:22:06.872 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.872 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.872 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:22:06.872 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.889382+0000", 2024-08-19T14:22:06.872 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20771527200000001 2024-08-19T14:22:06.872 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.872 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.872 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:22:06.872 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.889434+0000", 2024-08-19T14:22:06.873 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1198e-05 2024-08-19T14:22:06.873 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.873 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.873 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:22:06.873 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.889468+0000", 2024-08-19T14:22:06.873 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.4870999999999999e-05 2024-08-19T14:22:06.873 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:06.873 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:22:06.873 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:06.873 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.873 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.873 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:32461 4.3 4:c504f85a:::benchmark_data_smithi045_71562_object32460:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e343)", 2024-08-19T14:22:06.873 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:20:45.677433+0000", 2024-08-19T14:22:06.873 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 81.126181711000001, 2024-08-19T14:22:06.874 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.21174120199999999, 2024-08-19T14:22:06.874 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:22:06.874 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:22:06.874 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:22:06.874 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:22:06.874 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:22:06.874 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:22:06.874 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 32461 2024-08-19T14:22:06.874 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.874 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:22:06.874 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.874 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:22:06.874 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.677433+0000", 2024-08-19T14:22:06.874 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:06.875 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.875 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.875 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:22:06.875 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.677433+0000", 2024-08-19T14:22:06.875 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:06.875 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.875 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.875 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:22:06.875 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.677433+0000", 2024-08-19T14:22:06.875 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.5899999999999996e-07 2024-08-19T14:22:06.875 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.875 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.875 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:22:06.875 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.677450+0000", 2024-08-19T14:22:06.876 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6702e-05 2024-08-19T14:22:06.876 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.876 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.876 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:22:06.876 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.677459+0000", 2024-08-19T14:22:06.876 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.8759999999999994e-06 2024-08-19T14:22:06.876 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.876 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.876 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:22:06.876 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.677464+0000", 2024-08-19T14:22:06.876 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.5130000000000004e-06 2024-08-19T14:22:06.876 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.876 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.876 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:22:06.876 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.677931+0000", 2024-08-19T14:22:06.877 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000467054 2024-08-19T14:22:06.877 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.877 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.877 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:22:06.877 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.678021+0000", 2024-08-19T14:22:06.877 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.0017000000000002e-05 2024-08-19T14:22:06.877 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.877 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.877 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,7", 2024-08-19T14:22:06.877 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.678110+0000", 2024-08-19T14:22:06.877 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.8369999999999996e-05 2024-08-19T14:22:06.877 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.877 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.877 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:22:06.878 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.678991+0000", 2024-08-19T14:22:06.878 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00088159499999999999 2024-08-19T14:22:06.878 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.878 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.878 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:22:06.878 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.679917+0000", 2024-08-19T14:22:06.878 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00092540099999999996 2024-08-19T14:22:06.878 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.878 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.878 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:22:06.878 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.889075+0000", 2024-08-19T14:22:06.878 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20915857400000001 2024-08-19T14:22:06.878 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.878 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.879 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:22:06.879 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.889128+0000", 2024-08-19T14:22:06.879 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.3029000000000003e-05 2024-08-19T14:22:06.879 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.879 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.879 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:22:06.879 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.889174+0000", 2024-08-19T14:22:06.879 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.5512000000000001e-05 2024-08-19T14:22:06.879 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:06.879 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:22:06.879 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:06.879 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.879 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:06.880 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:06.880 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:22:06.881 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:32464 4.3 4:c66cc914:::benchmark_data_smithi045_71562_object32463:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e343)", 2024-08-19T14:22:06.881 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:20:45.678951+0000", 2024-08-19T14:22:06.881 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 81.124663216000002, 2024-08-19T14:22:06.881 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.21038989499999999, 2024-08-19T14:22:06.882 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:22:06.882 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:22:06.882 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:22:06.882 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:22:06.882 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:22:06.882 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:22:06.882 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 32464 2024-08-19T14:22:06.882 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.882 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:22:06.882 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.882 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:22:06.882 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.678951+0000", 2024-08-19T14:22:06.882 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:06.883 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.883 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.883 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:22:06.883 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.678951+0000", 2024-08-19T14:22:06.883 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:06.883 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.883 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.883 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:22:06.883 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.678952+0000", 2024-08-19T14:22:06.883 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.4600000000000003e-07 2024-08-19T14:22:06.883 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.883 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.883 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:22:06.883 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.678969+0000", 2024-08-19T14:22:06.884 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6935999999999998e-05 2024-08-19T14:22:06.884 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.884 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.884 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:22:06.884 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.678980+0000", 2024-08-19T14:22:06.884 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1389000000000001e-05 2024-08-19T14:22:06.884 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.884 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.884 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:22:06.884 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.678987+0000", 2024-08-19T14:22:06.884 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.6900000000000003e-06 2024-08-19T14:22:06.884 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.884 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.884 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:22:06.885 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.679014+0000", 2024-08-19T14:22:06.885 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.7317999999999998e-05 2024-08-19T14:22:06.885 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.885 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.885 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:22:06.885 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.679082+0000", 2024-08-19T14:22:06.885 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.7349999999999997e-05 2024-08-19T14:22:06.885 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.885 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.885 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,7", 2024-08-19T14:22:06.885 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.679154+0000", 2024-08-19T14:22:06.885 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.2293999999999996e-05 2024-08-19T14:22:06.885 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.886 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.886 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:22:06.886 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.680076+0000", 2024-08-19T14:22:06.886 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00092186600000000003 2024-08-19T14:22:06.886 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.886 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.886 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:22:06.886 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.680498+0000", 2024-08-19T14:22:06.886 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00042202899999999998 2024-08-19T14:22:06.886 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.886 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.886 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:22:06.886 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.889235+0000", 2024-08-19T14:22:06.886 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20873746200000001 2024-08-19T14:22:06.887 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.887 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.887 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:22:06.887 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.889297+0000", 2024-08-19T14:22:06.887 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.1418e-05 2024-08-19T14:22:06.887 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.887 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.887 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:22:06.887 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.889341+0000", 2024-08-19T14:22:06.887 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.4297e-05 2024-08-19T14:22:06.887 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:06.887 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:22:06.887 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:06.887 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.888 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.888 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:32465 4.9 4:99a9de10:::benchmark_data_smithi045_71562_object32464:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e343)", 2024-08-19T14:22:06.888 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:20:45.679784+0000", 2024-08-19T14:22:06.888 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 81.123830235, 2024-08-19T14:22:06.888 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20989022900000001, 2024-08-19T14:22:06.888 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:22:06.888 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:22:06.888 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:22:06.888 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:22:06.888 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:22:06.888 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:22:06.888 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 32465 2024-08-19T14:22:06.888 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.889 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:22:06.889 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.889 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:22:06.889 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.679784+0000", 2024-08-19T14:22:06.889 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:06.889 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.889 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.889 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:22:06.889 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.679784+0000", 2024-08-19T14:22:06.889 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:06.889 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.889 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.889 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:22:06.889 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.679785+0000", 2024-08-19T14:22:06.890 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.5199999999999995e-07 2024-08-19T14:22:06.890 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.890 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.890 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:22:06.890 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.679804+0000", 2024-08-19T14:22:06.890 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9136000000000001e-05 2024-08-19T14:22:06.890 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.890 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.890 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:22:06.890 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.679814+0000", 2024-08-19T14:22:06.890 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.6719999999999999e-06 2024-08-19T14:22:06.890 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.890 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.890 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:22:06.891 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.679819+0000", 2024-08-19T14:22:06.891 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0560000000000004e-06 2024-08-19T14:22:06.891 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.891 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.891 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:22:06.891 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.679881+0000", 2024-08-19T14:22:06.891 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.1948999999999999e-05 2024-08-19T14:22:06.891 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.891 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.891 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:22:06.891 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.680007+0000", 2024-08-19T14:22:06.891 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000125559 2024-08-19T14:22:06.891 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.891 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.892 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,3", 2024-08-19T14:22:06.892 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.680144+0000", 2024-08-19T14:22:06.892 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000137322 2024-08-19T14:22:06.892 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.892 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.892 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:22:06.892 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.680839+0000", 2024-08-19T14:22:06.892 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00069560900000000003 2024-08-19T14:22:06.892 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.892 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.892 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:22:06.892 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.681799+0000", 2024-08-19T14:22:06.892 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00095974500000000002 2024-08-19T14:22:06.892 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.892 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.893 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:22:06.893 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.889578+0000", 2024-08-19T14:22:06.893 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20777911900000001 2024-08-19T14:22:06.893 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.893 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.893 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:22:06.893 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.889630+0000", 2024-08-19T14:22:06.893 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2098999999999997e-05 2024-08-19T14:22:06.893 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.893 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.893 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:22:06.893 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.889674+0000", 2024-08-19T14:22:06.893 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.4011000000000003e-05 2024-08-19T14:22:06.894 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:06.894 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:22:06.894 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:06.894 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.894 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.894 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:116285 4.8 4:15b9fca8:::benchmark_data_smithi045_71562_object116284:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-08-19T14:22:06.894 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:21:48.600383+0000", 2024-08-19T14:22:06.894 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 18.203231593999998, 2024-08-19T14:22:06.894 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.25002218100000001, 2024-08-19T14:22:06.894 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:22:06.894 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:22:06.894 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:22:06.894 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:22:06.894 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:22:06.895 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:22:06.895 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 116285 2024-08-19T14:22:06.895 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.895 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:22:06.895 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.895 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:22:06.895 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.600383+0000", 2024-08-19T14:22:06.895 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:06.895 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.895 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.895 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:22:06.895 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.600383+0000", 2024-08-19T14:22:06.895 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:06.895 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.896 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.896 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:22:06.896 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.600384+0000", 2024-08-19T14:22:06.896 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.3200000000000003e-07 2024-08-19T14:22:06.896 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.896 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.896 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:22:06.896 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.600403+0000", 2024-08-19T14:22:06.896 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9326000000000001e-05 2024-08-19T14:22:06.896 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.896 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.896 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:22:06.896 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.600414+0000", 2024-08-19T14:22:06.896 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1129e-05 2024-08-19T14:22:06.896 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.897 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.897 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:22:06.897 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.600421+0000", 2024-08-19T14:22:06.897 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.8589999999999999e-06 2024-08-19T14:22:06.897 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.897 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.897 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:22:06.897 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.600478+0000", 2024-08-19T14:22:06.897 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.6502000000000002e-05 2024-08-19T14:22:06.897 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.897 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.897 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:22:06.897 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.600573+0000", 2024-08-19T14:22:06.897 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.5165000000000001e-05 2024-08-19T14:22:06.898 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.898 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.898 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 7,9", 2024-08-19T14:22:06.898 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.600654+0000", 2024-08-19T14:22:06.898 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.1002999999999997e-05 2024-08-19T14:22:06.898 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.898 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.898 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:22:06.898 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.607545+0000", 2024-08-19T14:22:06.898 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0068915039999999997 2024-08-19T14:22:06.898 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.898 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.898 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:22:06.898 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.610300+0000", 2024-08-19T14:22:06.899 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0027545199999999999 2024-08-19T14:22:06.899 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.899 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.899 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:22:06.899 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.850322+0000", 2024-08-19T14:22:06.899 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.24002195200000001 2024-08-19T14:22:06.899 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.899 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.899 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:22:06.899 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.850364+0000", 2024-08-19T14:22:06.899 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.2582e-05 2024-08-19T14:22:06.899 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.899 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.899 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:22:06.900 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.850405+0000", 2024-08-19T14:22:06.900 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.0707000000000002e-05 2024-08-19T14:22:06.900 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:06.900 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:22:06.900 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:06.900 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.900 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.900 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:116286 4.8 4:198dc556:::benchmark_data_smithi045_71562_object116285:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-08-19T14:22:06.900 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:21:48.600439+0000", 2024-08-19T14:22:06.900 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 18.203175566999999, 2024-08-19T14:22:06.900 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.25012546899999999, 2024-08-19T14:22:06.900 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:22:06.900 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:22:06.900 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:22:06.901 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:22:06.901 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:22:06.901 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:22:06.901 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 116286 2024-08-19T14:22:06.901 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.901 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:22:06.901 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.901 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:22:06.901 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.600439+0000", 2024-08-19T14:22:06.901 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:06.901 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.901 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.901 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:22:06.901 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.600439+0000", 2024-08-19T14:22:06.901 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:06.902 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.902 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.902 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:22:06.902 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.600439+0000", 2024-08-19T14:22:06.902 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2900000000000004e-07 2024-08-19T14:22:06.902 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.902 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.902 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:22:06.902 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.600458+0000", 2024-08-19T14:22:06.902 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8729999999999999e-05 2024-08-19T14:22:06.902 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.902 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.902 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:22:06.903 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.600466+0000", 2024-08-19T14:22:06.903 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.1030000000000002e-06 2024-08-19T14:22:06.903 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.903 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.903 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:22:06.903 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.600472+0000", 2024-08-19T14:22:06.903 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4380000000000003e-06 2024-08-19T14:22:06.903 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.903 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.903 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:22:06.903 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.603745+0000", 2024-08-19T14:22:06.903 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0032732350000000002 2024-08-19T14:22:06.903 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.903 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.903 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:22:06.904 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.603905+0000", 2024-08-19T14:22:06.904 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015985500000000001 2024-08-19T14:22:06.904 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.904 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.904 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 7,9", 2024-08-19T14:22:06.904 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.604060+0000", 2024-08-19T14:22:06.904 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015485799999999999 2024-08-19T14:22:06.904 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.904 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.904 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:22:06.904 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.607871+0000", 2024-08-19T14:22:06.904 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0038114540000000001 2024-08-19T14:22:06.904 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.904 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.904 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:22:06.905 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.610365+0000", 2024-08-19T14:22:06.905 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0024943449999999998 2024-08-19T14:22:06.905 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.905 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.905 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:22:06.905 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.850477+0000", 2024-08-19T14:22:06.905 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.24011110099999999 2024-08-19T14:22:06.905 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.905 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.905 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:22:06.905 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.850528+0000", 2024-08-19T14:22:06.905 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1684999999999997e-05 2024-08-19T14:22:06.905 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.905 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.906 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:22:06.906 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.850564+0000", 2024-08-19T14:22:06.906 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.6136000000000001e-05 2024-08-19T14:22:06.906 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:06.906 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:22:06.906 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:06.906 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.906 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.906 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:116297 4.8 4:17724d94:::benchmark_data_smithi045_71562_object116296:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-08-19T14:22:06.906 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:21:48.612374+0000", 2024-08-19T14:22:06.906 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 18.191240489999998, 2024-08-19T14:22:06.906 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.28029534299999997, 2024-08-19T14:22:06.906 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:22:06.906 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:22:06.907 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:22:06.907 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:22:06.907 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:22:06.907 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:22:06.907 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 116297 2024-08-19T14:22:06.907 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.907 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:22:06.907 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.907 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:22:06.907 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.612374+0000", 2024-08-19T14:22:06.907 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:06.907 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.907 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.907 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:22:06.908 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.612374+0000", 2024-08-19T14:22:06.908 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:06.908 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.908 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.908 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:22:06.908 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.612375+0000", 2024-08-19T14:22:06.908 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2869999999999999e-06 2024-08-19T14:22:06.908 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.908 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.908 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:22:06.908 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.612393+0000", 2024-08-19T14:22:06.908 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7864e-05 2024-08-19T14:22:06.908 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.908 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.909 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:22:06.909 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.612404+0000", 2024-08-19T14:22:06.909 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1377e-05 2024-08-19T14:22:06.909 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.909 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.909 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:22:06.909 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.612410+0000", 2024-08-19T14:22:06.909 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.3730000000000001e-06 2024-08-19T14:22:06.909 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.909 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.909 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:22:06.909 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.612460+0000", 2024-08-19T14:22:06.909 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.9675999999999997e-05 2024-08-19T14:22:06.910 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.910 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.910 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:22:06.910 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.612561+0000", 2024-08-19T14:22:06.910 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00010157099999999999 2024-08-19T14:22:06.910 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.910 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.910 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 7,9", 2024-08-19T14:22:06.910 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.612676+0000", 2024-08-19T14:22:06.910 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00011491 2024-08-19T14:22:06.910 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.910 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.910 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:22:06.910 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.616157+0000", 2024-08-19T14:22:06.911 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00348073 2024-08-19T14:22:06.911 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.911 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.911 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:22:06.911 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.616249+0000", 2024-08-19T14:22:06.911 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.2085999999999999e-05 2024-08-19T14:22:06.911 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.911 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.911 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:22:06.911 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.850604+0000", 2024-08-19T14:22:06.911 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.23435562900000001 2024-08-19T14:22:06.911 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.911 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.911 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:22:06.912 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.892613+0000", 2024-08-19T14:22:06.912 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.042008839999999999 2024-08-19T14:22:06.912 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.912 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.912 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:22:06.912 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.892669+0000", 2024-08-19T14:22:06.912 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.5999999999999999e-05 2024-08-19T14:22:06.912 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:06.912 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:22:06.912 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:06.912 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.912 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.912 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:116299 4.3 4:c0da230c:::benchmark_data_smithi045_71562_object116298:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-08-19T14:22:06.912 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:21:48.612978+0000", 2024-08-19T14:22:06.913 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 18.190636201, 2024-08-19T14:22:06.913 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.23536219899999999, 2024-08-19T14:22:06.913 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:22:06.913 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:22:06.913 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:22:06.913 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:22:06.913 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:22:06.913 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:22:06.913 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 116299 2024-08-19T14:22:06.913 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.913 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:22:06.913 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.913 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:22:06.913 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.612978+0000", 2024-08-19T14:22:06.914 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:06.914 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.914 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.914 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:22:06.914 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.612978+0000", 2024-08-19T14:22:06.914 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:06.914 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.914 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.914 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:22:06.914 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.612979+0000", 2024-08-19T14:22:06.914 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.7599999999999996e-07 2024-08-19T14:22:06.914 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.914 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.914 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:22:06.915 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.612998+0000", 2024-08-19T14:22:06.915 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8527000000000001e-05 2024-08-19T14:22:06.915 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.915 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.915 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:22:06.915 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.613008+0000", 2024-08-19T14:22:06.915 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.079e-05 2024-08-19T14:22:06.915 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.915 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.915 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:22:06.915 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.613014+0000", 2024-08-19T14:22:06.915 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4210000000000003e-06 2024-08-19T14:22:06.915 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.915 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.916 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:22:06.916 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.613062+0000", 2024-08-19T14:22:06.916 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.8674999999999997e-05 2024-08-19T14:22:06.916 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.916 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.916 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:22:06.916 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.613200+0000", 2024-08-19T14:22:06.916 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013715400000000001 2024-08-19T14:22:06.916 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.916 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.916 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,7", 2024-08-19T14:22:06.916 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.613317+0000", 2024-08-19T14:22:06.916 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00011734399999999999 2024-08-19T14:22:06.916 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.916 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.917 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:22:06.917 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.616326+0000", 2024-08-19T14:22:06.917 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0030090450000000001 2024-08-19T14:22:06.917 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.917 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.917 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:22:06.917 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.621117+0000", 2024-08-19T14:22:06.917 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0047912160000000001 2024-08-19T14:22:06.917 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.917 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.917 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:22:06.917 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.848233+0000", 2024-08-19T14:22:06.917 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.22711604099999999 2024-08-19T14:22:06.917 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.918 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.918 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:22:06.918 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.848288+0000", 2024-08-19T14:22:06.918 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4972999999999997e-05 2024-08-19T14:22:06.918 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.918 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.918 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:22:06.918 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.848340+0000", 2024-08-19T14:22:06.918 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2237000000000001e-05 2024-08-19T14:22:06.918 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:06.918 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:22:06.918 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:06.918 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.918 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.919 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:116311 4.8 4:16b96ac4:::benchmark_data_smithi045_71562_object116310:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-08-19T14:22:06.919 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:21:48.627253+0000", 2024-08-19T14:22:06.919 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 18.176361252, 2024-08-19T14:22:06.919 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.26561561299999997, 2024-08-19T14:22:06.919 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:22:06.919 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:22:06.919 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:22:06.919 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:22:06.919 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:22:06.919 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:22:06.919 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 116311 2024-08-19T14:22:06.919 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.919 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:22:06.919 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.920 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:22:06.920 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.627253+0000", 2024-08-19T14:22:06.920 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:06.920 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.920 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.920 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:22:06.920 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.627253+0000", 2024-08-19T14:22:06.920 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:06.920 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.920 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.920 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:22:06.920 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.627254+0000", 2024-08-19T14:22:06.920 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4000000000000001e-07 2024-08-19T14:22:06.920 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.921 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.921 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:22:06.921 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.627293+0000", 2024-08-19T14:22:06.921 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.8742000000000003e-05 2024-08-19T14:22:06.921 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.921 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.921 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:22:06.921 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.627302+0000", 2024-08-19T14:22:06.921 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.2639999999999997e-06 2024-08-19T14:22:06.921 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.921 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.921 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:22:06.921 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.627307+0000", 2024-08-19T14:22:06.921 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0209999999999999e-06 2024-08-19T14:22:06.922 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.922 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.922 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:22:06.922 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.627378+0000", 2024-08-19T14:22:06.922 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.0529000000000002e-05 2024-08-19T14:22:06.922 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.922 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.922 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:22:06.922 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.627458+0000", 2024-08-19T14:22:06.922 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.0259999999999994e-05 2024-08-19T14:22:06.922 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.922 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.922 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 7,9", 2024-08-19T14:22:06.923 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.627624+0000", 2024-08-19T14:22:06.923 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00016623300000000001 2024-08-19T14:22:06.923 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.923 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.923 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:22:06.923 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.631383+0000", 2024-08-19T14:22:06.923 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0037587599999999999 2024-08-19T14:22:06.923 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.923 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.923 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:22:06.923 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.631513+0000", 2024-08-19T14:22:06.923 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013022199999999999 2024-08-19T14:22:06.923 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.923 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.923 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:22:06.924 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.892784+0000", 2024-08-19T14:22:06.924 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.26127100399999997 2024-08-19T14:22:06.924 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.924 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.924 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:22:06.924 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.892832+0000", 2024-08-19T14:22:06.924 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.7970999999999999e-05 2024-08-19T14:22:06.924 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.924 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.924 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:22:06.924 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.892869+0000", 2024-08-19T14:22:06.924 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.6866999999999998e-05 2024-08-19T14:22:06.924 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:06.925 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:22:06.925 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:06.925 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.925 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.925 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:116334 4.8 4:1f777a70:::benchmark_data_smithi045_71562_object116333:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-08-19T14:22:06.925 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:21:48.650934+0000", 2024-08-19T14:22:06.925 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 18.152680544999999, 2024-08-19T14:22:06.925 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.242080929, 2024-08-19T14:22:06.925 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:22:06.925 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:22:06.925 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:22:06.925 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:22:06.925 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:22:06.925 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:22:06.926 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 116334 2024-08-19T14:22:06.926 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.926 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:22:06.926 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.926 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:22:06.926 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.650934+0000", 2024-08-19T14:22:06.926 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:06.926 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.926 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.926 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:22:06.926 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.650934+0000", 2024-08-19T14:22:06.926 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:06.926 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.926 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.927 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:22:06.927 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.650935+0000", 2024-08-19T14:22:06.927 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.26e-06 2024-08-19T14:22:06.927 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.927 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.927 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:22:06.927 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.650954+0000", 2024-08-19T14:22:06.927 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9000000000000001e-05 2024-08-19T14:22:06.927 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.927 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.927 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:22:06.927 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.650964+0000", 2024-08-19T14:22:06.927 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0253e-05 2024-08-19T14:22:06.927 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.928 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.928 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:22:06.928 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.650970+0000", 2024-08-19T14:22:06.928 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1499999999999998e-06 2024-08-19T14:22:06.928 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.928 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.928 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:22:06.928 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.651014+0000", 2024-08-19T14:22:06.928 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.4391000000000002e-05 2024-08-19T14:22:06.928 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.928 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.928 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:22:06.928 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.651134+0000", 2024-08-19T14:22:06.928 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000119535 2024-08-19T14:22:06.929 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.929 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.929 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 7,9", 2024-08-19T14:22:06.929 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.651262+0000", 2024-08-19T14:22:06.929 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012826100000000001 2024-08-19T14:22:06.929 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.929 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.929 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:22:06.929 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.654276+0000", 2024-08-19T14:22:06.929 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0030142839999999999 2024-08-19T14:22:06.929 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.929 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.929 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:22:06.929 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.655021+0000", 2024-08-19T14:22:06.930 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00074536299999999997 2024-08-19T14:22:06.930 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.930 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.930 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:22:06.930 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.892914+0000", 2024-08-19T14:22:06.930 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.23789228400000001 2024-08-19T14:22:06.930 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.930 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.930 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:22:06.930 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.892980+0000", 2024-08-19T14:22:06.930 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.6193999999999997e-05 2024-08-19T14:22:06.930 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.930 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.931 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:22:06.931 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.893015+0000", 2024-08-19T14:22:06.931 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.4953999999999998e-05 2024-08-19T14:22:06.931 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:06.931 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:22:06.931 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:06.931 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.931 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.931 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:116354 4.8 4:103642ea:::benchmark_data_smithi045_71562_object116353:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-08-19T14:22:06.931 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:21:48.673582+0000", 2024-08-19T14:22:06.931 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 18.130032872000001, 2024-08-19T14:22:06.931 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.21960154700000001, 2024-08-19T14:22:06.931 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:22:06.931 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:22:06.932 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:22:06.932 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:22:06.932 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:22:06.932 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:22:06.932 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 116354 2024-08-19T14:22:06.932 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.932 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:22:06.932 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.932 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:22:06.932 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.673582+0000", 2024-08-19T14:22:06.932 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:06.932 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.932 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.932 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:22:06.932 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.673582+0000", 2024-08-19T14:22:06.933 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:22:06.933 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.933 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.933 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:22:06.933 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.673583+0000", 2024-08-19T14:22:06.933 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.279e-06 2024-08-19T14:22:06.933 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.933 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.933 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:22:06.933 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.673633+0000", 2024-08-19T14:22:06.933 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.9845000000000001e-05 2024-08-19T14:22:06.933 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.933 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.934 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:22:06.934 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.673644+0000", 2024-08-19T14:22:06.934 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1333e-05 2024-08-19T14:22:06.934 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.934 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.934 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:22:06.934 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.673649+0000", 2024-08-19T14:22:06.934 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2000000000000002e-06 2024-08-19T14:22:06.934 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.934 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.934 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:22:06.934 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.673703+0000", 2024-08-19T14:22:06.934 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.3471999999999999e-05 2024-08-19T14:22:06.934 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.935 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.935 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:22:06.935 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.673845+0000", 2024-08-19T14:22:06.935 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014190399999999999 2024-08-19T14:22:06.935 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.935 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.935 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 7,9", 2024-08-19T14:22:06.935 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.673956+0000", 2024-08-19T14:22:06.935 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00011128000000000001 2024-08-19T14:22:06.935 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.935 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.935 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:22:06.935 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.676763+0000", 2024-08-19T14:22:06.935 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.002807038 2024-08-19T14:22:06.936 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.936 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.936 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:22:06.936 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.677752+0000", 2024-08-19T14:22:06.936 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00098899899999999995 2024-08-19T14:22:06.936 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.936 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.936 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:22:06.936 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.893093+0000", 2024-08-19T14:22:06.936 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.21534098300000001 2024-08-19T14:22:06.936 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.936 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.936 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:22:06.936 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.893143+0000", 2024-08-19T14:22:06.937 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.9877999999999999e-05 2024-08-19T14:22:06.937 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:22:06.937 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:22:06.937 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:22:06.937 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.893183+0000", 2024-08-19T14:22:06.937 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.0336000000000001e-05 2024-08-19T14:22:06.937 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:06.937 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:22:06.937 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:06.937 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:22:06.937 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:22:06.937 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:22:06.981 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:22:07.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:07.011 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:07.012 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:22:07.113 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:22:07.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:07.146 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:07.147 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:22:07.248 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:22:07.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:07.279 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:07.279 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:22:07.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:07 smithi045 ceph-mon[20143]: pgmap v838: 47 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 45 active+clean; 8.0 GiB data, 28 GiB used, 777 GiB / 805 GiB avail; 69 MiB/s wr, 1.13k op/s; 2581/391677 objects misplaced (0.659%); 2.4 MiB/s, 38 objects/s recovering 2024-08-19T14:22:07.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:07 smithi045 ceph-mon[20143]: 4.0 scrub starts 2024-08-19T14:22:07.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:07 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:22:07.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:07 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:22:07.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:07 smithi045 ceph-mon[20143]: 4.d scrub starts 2024-08-19T14:22:07.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:07 smithi031 ceph-mon[16336]: pgmap v838: 47 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 45 active+clean; 8.0 GiB data, 28 GiB used, 777 GiB / 805 GiB avail; 69 MiB/s wr, 1.13k op/s; 2581/391677 objects misplaced (0.659%); 2.4 MiB/s, 38 objects/s recovering 2024-08-19T14:22:07.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:07 smithi031 ceph-mon[16336]: 4.0 scrub starts 2024-08-19T14:22:07.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:07 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:22:07.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:07 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:22:07.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:07 smithi031 ceph-mon[16336]: 4.d scrub starts 2024-08-19T14:22:07.380 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:22:07.418 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:07.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:07.419 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:22:07.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:07 smithi032 ceph-mon[20004]: pgmap v838: 47 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 45 active+clean; 8.0 GiB data, 28 GiB used, 777 GiB / 805 GiB avail; 69 MiB/s wr, 1.13k op/s; 2581/391677 objects misplaced (0.659%); 2.4 MiB/s, 38 objects/s recovering 2024-08-19T14:22:07.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:07 smithi032 ceph-mon[20004]: 4.0 scrub starts 2024-08-19T14:22:07.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:07 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:22:07.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:07 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:22:07.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:07 smithi032 ceph-mon[20004]: 4.d scrub starts 2024-08-19T14:22:07.520 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:22:07.551 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:07.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:07.552 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:22:07.654 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:22:07.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:07.690 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:07.690 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:22:07.791 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:22:07.829 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:07.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:07.830 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:22:07.931 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:22:07.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:07.956 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:07.956 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:22:08.049 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:22:08.057 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:22:08.083 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:08.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:08.084 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:22:08.171 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-08-19T14:22:08.184 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:22:08.213 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:08.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:08.214 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:22:08.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:08 smithi045 ceph-mon[20143]: osdmap e373: 12 total, 12 up, 9 in 2024-08-19T14:22:08.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:08 smithi045 ceph-mon[20143]: 1.18 scrub starts 2024-08-19T14:22:08.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:08 smithi045 ceph-mon[20143]: 1.18 scrub ok 2024-08-19T14:22:08.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:08 smithi045 ceph-mon[20143]: 4.c scrub ok 2024-08-19T14:22:08.315 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:22:08.338 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:08.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:08.340 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:22:08.351 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:08 smithi031 ceph-mon[16336]: osdmap e373: 12 total, 12 up, 9 in 2024-08-19T14:22:08.352 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:08 smithi031 ceph-mon[16336]: 1.18 scrub starts 2024-08-19T14:22:08.352 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:08 smithi031 ceph-mon[16336]: 1.18 scrub ok 2024-08-19T14:22:08.352 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:08 smithi031 ceph-mon[16336]: 4.c scrub ok 2024-08-19T14:22:08.441 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:22:08.469 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:08.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:08.470 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:22:08.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:08 smithi032 ceph-mon[20004]: osdmap e373: 12 total, 12 up, 9 in 2024-08-19T14:22:08.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:08 smithi032 ceph-mon[20004]: 1.18 scrub starts 2024-08-19T14:22:08.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:08 smithi032 ceph-mon[20004]: 1.18 scrub ok 2024-08-19T14:22:08.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:08 smithi032 ceph-mon[20004]: 4.c scrub ok 2024-08-19T14:22:08.570 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:22:08.594 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:08.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:08.595 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:22:08.675 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:22:08.696 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:22:08.720 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:22:08.720 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:22:08.760 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:22:08.761 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:22:08.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:08.775 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:08.775 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:22:08.794 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:22:08.794 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:22:08.825 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:22:08.825 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:22:08.860 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:22:08.860 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:22:08.877 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:22:08.891 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:22:08.892 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:22:08.924 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:08.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:08.925 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:22:08.932 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:22:08.932 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:22:08.968 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:22:08.969 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:22:09.006 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:22:09.006 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:22:09.026 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:22:09.049 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:22:09.049 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:22:09.062 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:09.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:09.063 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:22:09.087 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:22:09.087 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:22:09.120 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:22:09.120 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:22:09.164 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:22:09.196 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:09.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:09.197 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:22:09.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:09 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:22:09.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:09 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:22:09.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:09 smithi045 ceph-mon[20143]: 4.3 scrub ok 2024-08-19T14:22:09.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:09 smithi045 ceph-mon[20143]: pgmap v840: 47 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 45 active+clean; 8.0 GiB data, 28 GiB used, 777 GiB / 805 GiB avail; 40 MiB/s wr, 681 op/s; 2581/391920 objects misplaced (0.659%); 1.4 MiB/s, 21 objects/s recovering 2024-08-19T14:22:09.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:09 smithi045 ceph-mon[20143]: Health check failed: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-08-19T14:22:09.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:09 smithi045 ceph-mon[20143]: osdmap e374: 12 total, 12 up, 9 in 2024-08-19T14:22:09.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:09 smithi045 ceph-mon[20143]: 4.d scrub ok 2024-08-19T14:22:09.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:09 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3446431877' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:22:09.298 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:22:09.324 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:09.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:09.325 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:22:09.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:09 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:22:09.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:09 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:22:09.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:09 smithi031 ceph-mon[16336]: 4.3 scrub ok 2024-08-19T14:22:09.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:09 smithi031 ceph-mon[16336]: pgmap v840: 47 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 45 active+clean; 8.0 GiB data, 28 GiB used, 777 GiB / 805 GiB avail; 40 MiB/s wr, 681 op/s; 2581/391920 objects misplaced (0.659%); 1.4 MiB/s, 21 objects/s recovering 2024-08-19T14:22:09.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:09 smithi031 ceph-mon[16336]: Health check failed: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-08-19T14:22:09.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:09 smithi031 ceph-mon[16336]: osdmap e374: 12 total, 12 up, 9 in 2024-08-19T14:22:09.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:09 smithi031 ceph-mon[16336]: 4.d scrub ok 2024-08-19T14:22:09.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:09 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3446431877' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:22:09.426 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:22:09.465 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:09.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:09.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:22:09.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:09 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:22:09.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:09 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:22:09.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:09 smithi032 ceph-mon[20004]: 4.3 scrub ok 2024-08-19T14:22:09.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:09 smithi032 ceph-mon[20004]: pgmap v840: 47 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 45 active+clean; 8.0 GiB data, 28 GiB used, 777 GiB / 805 GiB avail; 40 MiB/s wr, 681 op/s; 2581/391920 objects misplaced (0.659%); 1.4 MiB/s, 21 objects/s recovering 2024-08-19T14:22:09.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:09 smithi032 ceph-mon[20004]: Health check failed: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-08-19T14:22:09.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:09 smithi032 ceph-mon[20004]: osdmap e374: 12 total, 12 up, 9 in 2024-08-19T14:22:09.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:09 smithi032 ceph-mon[20004]: 4.d scrub ok 2024-08-19T14:22:09.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:09 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3446431877' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:22:09.567 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:22:09.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:09.597 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:09.597 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:22:09.698 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:22:09.728 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:09.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:09.729 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:22:09.830 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:22:09.856 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:09.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:09.857 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:22:09.958 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:22:09.982 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:09.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:09.984 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:22:10.019 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-19T14:22:10.060 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:22:10.084 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:22:10.106 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:10.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:10.107 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:22:10.208 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:22:10.230 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:10.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:10.231 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:22:10.236 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:22:10.237 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-08-19T14:22:10.332 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:22:10.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:10 smithi031 ceph-mon[16336]: 4.0 scrub ok 2024-08-19T14:22:10.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:10 smithi031 ceph-mon[16336]: 1.1e scrub starts 2024-08-19T14:22:10.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:10 smithi031 ceph-mon[16336]: 1.1e scrub ok 2024-08-19T14:22:10.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:10.379 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:10.379 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:22:10.480 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:22:10.507 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:10.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:10 smithi032 ceph-mon[20004]: 4.0 scrub ok 2024-08-19T14:22:10.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:10 smithi032 ceph-mon[20004]: 1.1e scrub starts 2024-08-19T14:22:10.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:10 smithi032 ceph-mon[20004]: 1.1e scrub ok 2024-08-19T14:22:10.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:10.509 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:22:10.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:10 smithi045 ceph-mon[20143]: 4.0 scrub ok 2024-08-19T14:22:10.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:10 smithi045 ceph-mon[20143]: 1.1e scrub starts 2024-08-19T14:22:10.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:10 smithi045 ceph-mon[20143]: 1.1e scrub ok 2024-08-19T14:22:10.610 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:22:10.641 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:10.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:10.642 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:22:10.742 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:22:10.769 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:10.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:10.770 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:22:10.871 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:22:10.914 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:10.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:10.915 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:22:11.016 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:22:11.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:11.043 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:11.043 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:22:11.144 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:22:11.188 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:11.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:11.189 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:22:11.219 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:11 smithi031 ceph-mon[16336]: pgmap v842: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+clean+scrubbing, 44 active+clean; 7.9 GiB data, 27 GiB used, 777 GiB / 805 GiB avail; 170 B/s rd, 32 MiB/s wr, 931 op/s; 2339/385869 objects misplaced (0.606%); 4.1 MiB/s, 65 objects/s recovering 2024-08-19T14:22:11.219 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:11 smithi031 ceph-mon[16336]: from='client.25507 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:22:11.290 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:22:11.315 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:11.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:11.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:22:11.385 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:22:11.385 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:22:11.417 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:22:11.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:11.468 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:11.469 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:22:11.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:11 smithi032 ceph-mon[20004]: pgmap v842: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+clean+scrubbing, 44 active+clean; 7.9 GiB data, 27 GiB used, 777 GiB / 805 GiB avail; 170 B/s rd, 32 MiB/s wr, 931 op/s; 2339/385869 objects misplaced (0.606%); 4.1 MiB/s, 65 objects/s recovering 2024-08-19T14:22:11.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:11 smithi032 ceph-mon[20004]: from='client.25507 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:22:11.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:11 smithi045 ceph-mon[20143]: pgmap v842: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+clean+scrubbing, 44 active+clean; 7.9 GiB data, 27 GiB used, 777 GiB / 805 GiB avail; 170 B/s rd, 32 MiB/s wr, 931 op/s; 2339/385869 objects misplaced (0.606%); 4.1 MiB/s, 65 objects/s recovering 2024-08-19T14:22:11.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:11 smithi045 ceph-mon[20143]: from='client.25507 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:22:11.570 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:22:11.596 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:11.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:11.597 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:22:11.698 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:22:11.732 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:11.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:11.733 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:22:11.812 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":842,"stamp":"2024-08-19T14:22:09.907559+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8429830712,"num_objects":128623,"num_object_clones":0,"num_object_copies":385869,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2339,"num_objects_unfound":0,"num_objects_dirty":128623,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":133725,"num_write_kb":8395009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1801,"num_bytes_recovered":118030336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":138034,"ondisk_log_size":138034,"up":141,"acting":140,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":142,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":28766848,"kb_used_data":24909660,"kb_used_omap":14,"kb_used_meta":3856753,"kb_avail":814939520,"statfs":{"total":863955320832,"available":834498068480,"internally_reserved":0,"allocated":25507491840,"data_stored":25491997428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14353,"internal_metadata":3949316079},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,4,8,6,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":31784984,"num_objects":486,"num_object_clones":0,"num_object_copies":1458,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-392,"num_objects_unfound":0,"num_objects_dirty":486,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5590,"num_write_kb":194369,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":392,"num_bytes_recovered":25690112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001330"},"pg_stats":[{"pgid":"4.5","version":"372'8252","reported_seq":8336,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-08-19T14:22:05.818367+0000","last_change":"2024-08-19T14:20:29.268214+0000","last_active":"2024-08-19T14:22:05.818367+0000","last_peered":"2024-08-19T14:22:05.818367+0000","last_clean":"2024-08-19T14:22:05.818367+0000","last_became_active":"2024-08-19T14:20:29.267549+0000","last_became_peered":"2024-08-19T14:20:29.267549+0000","last_unstale":"2024-08-19T14:22:05.818367+0000","last_undegraded":"2024-08-19T14:22:05.818367+0000","last_fullsized":"2024-08-19T14:22:05.818367+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":8252,"log_dups_size":0,"ondisk_log_size":8252,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":533987328,"num_objects":8148,"num_object_clones":0,"num_object_copies":24444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8148,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8252,"num_write_kb":524800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,0],"acting":[11,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"70'139","reported_seq":964,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-08-19T14:22:08.052479+0000","last_change":"2024-08-19T14:19:58.072227+0000","last_active":"2024-08-19T14:22:08.052479+0000","last_peered":"2024-08-19T14:22:08.052479+0000","last_clean":"2024-08-19T14:22:08.052479+0000","last_became_active":"2024-08-19T14:18:13.807133+0000","last_became_peered":"2024-08-19T14:18:13.807133+0000","last_unstale":"2024-08-19T14:22:08.052479+0000","last_undegraded":"2024-08-19T14:22:08.052479+0000","last_fullsized":"2024-08-19T14:22:08.052479+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:19:58.072083+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:19:58.072083+0000","last_clean_scrub_stamp":"2024-08-19T14:19:58.072083+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,9],"acting":[7,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"374'8593","reported_seq":12286,"reported_epoch":374,"state":"active+recovering+undersized+remapped","last_fresh":"2024-08-19T14:22:09.276692+0000","last_change":"2024-08-19T14:21:07.239135+0000","last_active":"2024-08-19T14:22:09.276692+0000","last_peered":"2024-08-19T14:22:09.276692+0000","last_clean":"2024-08-19T14:21:04.784061+0000","last_became_active":"2024-08-19T14:21:07.213050+0000","last_became_peered":"2024-08-19T14:21:07.213050+0000","last_unstale":"2024-08-19T14:22:09.276692+0000","last_undegraded":"2024-08-19T14:22:09.276692+0000","last_fullsized":"2024-08-19T14:21:06.805412+0000","mapping_epoch":354,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":8593,"log_dups_size":0,"ondisk_log_size":8593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":529858560,"num_objects":8085,"num_object_clones":0,"num_object_copies":24255,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2339,"num_objects_unfound":0,"num_objects_dirty":8085,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8593,"num_write_kb":533696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1801,"num_bytes_recovered":118030336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,7],"acting":[1,7],"avail_no_missing":["1","5","7"],"object_location_counts":[{"shards":"1,5,7","objects":2339},{"shards":"1,7","objects":5746}],"blocked_by":[],"up_primary":9,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"70'139","reported_seq":863,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-08-19T14:22:08.050748+0000","last_change":"2024-08-19T14:20:03.714917+0000","last_active":"2024-08-19T14:22:08.050748+0000","last_peered":"2024-08-19T14:22:08.050748+0000","last_clean":"2024-08-19T14:22:08.050748+0000","last_became_active":"2024-08-19T14:18:13.804721+0000","last_became_peered":"2024-08-19T14:18:13.804721+0000","last_unstale":"2024-08-19T14:22:08.050748+0000","last_undegraded":"2024-08-19T14:22:08.050748+0000","last_fullsized":"2024-08-19T14:22:08.050748+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:03.714818+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:03.714818+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"374'8460","reported_seq":8548,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-08-19T14:22:09.272763+0000","last_change":"2024-08-19T14:20:29.268110+0000","last_active":"2024-08-19T14:22:09.272763+0000","last_peered":"2024-08-19T14:22:09.272763+0000","last_clean":"2024-08-19T14:22:09.272763+0000","last_became_active":"2024-08-19T14:20:29.267811+0000","last_became_peered":"2024-08-19T14:20:29.267811+0000","last_unstale":"2024-08-19T14:22:09.272763+0000","last_undegraded":"2024-08-19T14:22:09.272763+0000","last_fullsized":"2024-08-19T14:22:09.272763+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":8460,"log_dups_size":0,"ondisk_log_size":8460,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":523370496,"num_objects":7986,"num_object_clones":0,"num_object_copies":23958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8460,"num_write_kb":526272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"70'139","reported_seq":811,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-08-19T14:22:08.048416+0000","last_change":"2024-08-19T14:20:50.742676+0000","last_active":"2024-08-19T14:22:08.048416+0000","last_peered":"2024-08-19T14:22:08.048416+0000","last_clean":"2024-08-19T14:22:08.048416+0000","last_became_active":"2024-08-19T14:20:50.742354+0000","last_became_peered":"2024-08-19T14:20:50.742354+0000","last_unstale":"2024-08-19T14:22:08.048416+0000","last_undegraded":"2024-08-19T14:22:08.048416+0000","last_fullsized":"2024-08-19T14:22:08.048416+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:19:59.302931+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:19:59.302931+0000","last_clean_scrub_stamp":"2024-08-19T14:19:59.302931+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":114,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"374'8425","reported_seq":8513,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-08-19T14:22:09.258565+0000","last_change":"2024-08-19T14:20:29.267640+0000","last_active":"2024-08-19T14:22:09.258565+0000","last_peered":"2024-08-19T14:22:09.258565+0000","last_clean":"2024-08-19T14:22:09.258565+0000","last_became_active":"2024-08-19T14:20:29.267331+0000","last_became_peered":"2024-08-19T14:20:29.267331+0000","last_unstale":"2024-08-19T14:22:09.258565+0000","last_undegraded":"2024-08-19T14:22:09.258565+0000","last_fullsized":"2024-08-19T14:22:09.258565+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":8425,"log_dups_size":0,"ondisk_log_size":8425,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":521732096,"num_objects":7961,"num_object_clones":0,"num_object_copies":23883,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7961,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8425,"num_write_kb":524352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"70'139","reported_seq":860,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-08-19T14:22:08.049050+0000","last_change":"2024-08-19T14:20:09.507422+0000","last_active":"2024-08-19T14:22:08.049050+0000","last_peered":"2024-08-19T14:22:08.049050+0000","last_clean":"2024-08-19T14:22:08.049050+0000","last_became_active":"2024-08-19T14:18:13.808505+0000","last_became_peered":"2024-08-19T14:18:13.808505+0000","last_unstale":"2024-08-19T14:22:08.049050+0000","last_undegraded":"2024-08-19T14:22:08.049050+0000","last_fullsized":"2024-08-19T14:22:08.049050+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:09.507256+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:09.507256+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,11],"acting":[9,0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"374'8467","reported_seq":8555,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-08-19T14:22:08.390350+0000","last_change":"2024-08-19T14:20:29.268766+0000","last_active":"2024-08-19T14:22:08.390350+0000","last_peered":"2024-08-19T14:22:08.390350+0000","last_clean":"2024-08-19T14:22:08.390350+0000","last_became_active":"2024-08-19T14:20:29.268443+0000","last_became_peered":"2024-08-19T14:20:29.268443+0000","last_unstale":"2024-08-19T14:22:08.390350+0000","last_undegraded":"2024-08-19T14:22:08.390350+0000","last_fullsized":"2024-08-19T14:22:08.390350+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":8467,"log_dups_size":0,"ondisk_log_size":8467,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":527237120,"num_objects":8045,"num_object_clones":0,"num_object_copies":24135,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8045,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8467,"num_write_kb":528384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,11],"acting":[9,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.7","version":"70'139","reported_seq":511,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-08-19T14:22:08.047085+0000","last_change":"2024-08-19T14:21:46.057764+0000","last_active":"2024-08-19T14:22:08.047085+0000","last_peered":"2024-08-19T14:22:08.047085+0000","last_clean":"2024-08-19T14:22:08.047085+0000","last_became_active":"2024-08-19T14:20:28.204384+0000","last_became_peered":"2024-08-19T14:20:28.204384+0000","last_unstale":"2024-08-19T14:22:08.047085+0000","last_undegraded":"2024-08-19T14:22:08.047085+0000","last_fullsized":"2024-08-19T14:22:08.047085+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:46.057624+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:21:46.057624+0000","last_clean_scrub_stamp":"2024-08-19T14:21:46.057624+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:02.689278+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"372'8227","reported_seq":8386,"reported_epoch":372,"state":"active+clean+scrubbing","last_fresh":"2024-08-19T14:22:05.834586+0000","last_change":"2024-08-19T14:22:05.430451+0000","last_active":"2024-08-19T14:22:05.834586+0000","last_peered":"2024-08-19T14:22:05.834586+0000","last_clean":"2024-08-19T14:22:05.834586+0000","last_became_active":"2024-08-19T14:20:29.267379+0000","last_became_peered":"2024-08-19T14:20:29.267379+0000","last_unstale":"2024-08-19T14:22:05.834586+0000","last_undegraded":"2024-08-19T14:22:05.834586+0000","last_fullsized":"2024-08-19T14:22:05.834586+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":1499,"log_size":8227,"log_dups_size":0,"ondisk_log_size":8227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":532348928,"num_objects":8123,"num_object_clones":0,"num_object_copies":24369,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8227,"num_write_kb":523200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,1],"acting":[11,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.6","version":"70'139","reported_seq":864,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-08-19T14:22:08.049125+0000","last_change":"2024-08-19T14:19:55.512078+0000","last_active":"2024-08-19T14:22:08.049125+0000","last_peered":"2024-08-19T14:22:08.049125+0000","last_clean":"2024-08-19T14:22:08.049125+0000","last_became_active":"2024-08-19T14:18:26.874902+0000","last_became_peered":"2024-08-19T14:18:26.874902+0000","last_unstale":"2024-08-19T14:22:08.049125+0000","last_undegraded":"2024-08-19T14:22:08.049125+0000","last_fullsized":"2024-08-19T14:22:08.049125+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:19:55.511974+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:27.848870+0000","last_clean_scrub_stamp":"2024-08-19T14:19:55.511974+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,4],"acting":[9,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"374'8394","reported_seq":8568,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-08-19T14:22:09.539077+0000","last_change":"2024-08-19T14:22:08.536948+0000","last_active":"2024-08-19T14:22:09.539077+0000","last_peered":"2024-08-19T14:22:09.539077+0000","last_clean":"2024-08-19T14:22:09.539077+0000","last_became_active":"2024-08-19T14:20:29.267360+0000","last_became_peered":"2024-08-19T14:20:29.267360+0000","last_unstale":"2024-08-19T14:22:09.539077+0000","last_undegraded":"2024-08-19T14:22:09.539077+0000","last_fullsized":"2024-08-19T14:22:09.539077+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"374'8331","last_scrub_stamp":"2024-08-19T14:22:08.536620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:22:08.536620+0000","objects_scrubbed":7995,"log_size":8394,"log_dups_size":0,"ondisk_log_size":8394,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:19.976024+0000","scrub_duration":2478,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":514195456,"num_objects":7846,"num_object_clones":0,"num_object_copies":23538,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8394,"num_write_kb":519680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"70'139","reported_seq":681,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-08-19T14:22:05.065305+0000","last_change":"2024-08-19T14:22:01.354585+0000","last_active":"2024-08-19T14:22:05.065305+0000","last_peered":"2024-08-19T14:22:05.065305+0000","last_clean":"2024-08-19T14:22:05.065305+0000","last_became_active":"2024-08-19T14:18:26.881435+0000","last_became_peered":"2024-08-19T14:18:26.881435+0000","last_unstale":"2024-08-19T14:22:05.065305+0000","last_undegraded":"2024-08-19T14:22:05.065305+0000","last_fullsized":"2024-08-19T14:22:05.065305+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:22:01.354343+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:22:01.354343+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:30.081548+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,4],"acting":[11,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.1","version":"372'8299","reported_seq":8383,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-08-19T14:22:05.845815+0000","last_change":"2024-08-19T14:20:29.268808+0000","last_active":"2024-08-19T14:22:05.845815+0000","last_peered":"2024-08-19T14:22:05.845815+0000","last_clean":"2024-08-19T14:22:05.845815+0000","last_became_active":"2024-08-19T14:20:29.268465+0000","last_became_peered":"2024-08-19T14:20:29.268465+0000","last_unstale":"2024-08-19T14:22:05.845815+0000","last_undegraded":"2024-08-19T14:22:05.845815+0000","last_fullsized":"2024-08-19T14:22:05.845815+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":8299,"log_dups_size":0,"ondisk_log_size":8299,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":537591808,"num_objects":8203,"num_object_clones":0,"num_object_copies":24609,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8203,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8299,"num_write_kb":528064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.4","version":"70'139","reported_seq":906,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-08-19T14:22:05.065014+0000","last_change":"2024-08-19T14:21:39.314063+0000","last_active":"2024-08-19T14:22:05.065014+0000","last_peered":"2024-08-19T14:22:05.065014+0000","last_clean":"2024-08-19T14:22:05.065014+0000","last_became_active":"2024-08-19T14:18:13.806013+0000","last_became_peered":"2024-08-19T14:18:13.806013+0000","last_unstale":"2024-08-19T14:22:05.065014+0000","last_undegraded":"2024-08-19T14:22:05.065014+0000","last_fullsized":"2024-08-19T14:22:05.065014+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:39.313941+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:21:39.313941+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:22:45.184594+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.f","version":"374'8385","reported_seq":8473,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-08-19T14:22:09.257933+0000","last_change":"2024-08-19T14:20:29.212298+0000","last_active":"2024-08-19T14:22:09.257933+0000","last_peered":"2024-08-19T14:22:09.257933+0000","last_clean":"2024-08-19T14:22:09.257933+0000","last_became_active":"2024-08-19T14:20:29.212003+0000","last_became_peered":"2024-08-19T14:20:29.212003+0000","last_unstale":"2024-08-19T14:22:09.257933+0000","last_undegraded":"2024-08-19T14:22:09.257933+0000","last_fullsized":"2024-08-19T14:22:09.257933+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":8385,"log_dups_size":0,"ondisk_log_size":8385,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":518848512,"num_objects":7917,"num_object_clones":0,"num_object_copies":23751,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7917,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8385,"num_write_kb":521664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,11],"acting":[1,3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"70'139","reported_seq":872,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-08-19T14:22:08.049058+0000","last_change":"2024-08-19T14:20:12.570453+0000","last_active":"2024-08-19T14:22:08.049058+0000","last_peered":"2024-08-19T14:22:08.049058+0000","last_clean":"2024-08-19T14:22:08.049058+0000","last_became_active":"2024-08-19T14:18:26.875006+0000","last_became_peered":"2024-08-19T14:18:26.875006+0000","last_unstale":"2024-08-19T14:22:08.049058+0000","last_undegraded":"2024-08-19T14:22:08.049058+0000","last_fullsized":"2024-08-19T14:22:08.049058+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:12.570352+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:28.864870+0000","last_clean_scrub_stamp":"2024-08-19T14:20:12.570352+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,0],"acting":[9,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.c","version":"374'8379","reported_seq":8555,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-08-19T14:22:08.155888+0000","last_change":"2024-08-19T14:22:07.487434+0000","last_active":"2024-08-19T14:22:08.155888+0000","last_peered":"2024-08-19T14:22:08.155888+0000","last_clean":"2024-08-19T14:22:08.155888+0000","last_became_active":"2024-08-19T14:20:29.266373+0000","last_became_peered":"2024-08-19T14:20:29.266373+0000","last_unstale":"2024-08-19T14:22:08.155888+0000","last_undegraded":"2024-08-19T14:22:08.155888+0000","last_fullsized":"2024-08-19T14:22:08.155888+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"373'8336","last_scrub_stamp":"2024-08-19T14:22:07.487270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:22:07.487270+0000","objects_scrubbed":8079,"log_size":8379,"log_dups_size":0,"ondisk_log_size":8379,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:14.053549+0000","scrub_duration":1972,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":523173888,"num_objects":7983,"num_object_clones":0,"num_object_copies":23949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7983,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8379,"num_write_kb":523584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,8],"acting":[4,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"70'139","reported_seq":987,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-08-19T14:22:08.048182+0000","last_change":"2024-08-19T14:19:56.106047+0000","last_active":"2024-08-19T14:22:08.048182+0000","last_peered":"2024-08-19T14:22:08.048182+0000","last_clean":"2024-08-19T14:22:08.048182+0000","last_became_active":"2024-08-19T14:18:26.887657+0000","last_became_peered":"2024-08-19T14:18:26.887657+0000","last_unstale":"2024-08-19T14:22:08.048182+0000","last_undegraded":"2024-08-19T14:22:08.048182+0000","last_fullsized":"2024-08-19T14:22:08.048182+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:19:56.105880+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:19:56.105880+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"374'8301","reported_seq":8458,"reported_epoch":374,"state":"active+clean+scrubbing","last_fresh":"2024-08-19T14:22:08.178234+0000","last_change":"2024-08-19T14:22:06.566433+0000","last_active":"2024-08-19T14:22:08.178234+0000","last_peered":"2024-08-19T14:22:08.178234+0000","last_clean":"2024-08-19T14:22:08.178234+0000","last_became_active":"2024-08-19T14:20:29.208599+0000","last_became_peered":"2024-08-19T14:20:29.208599+0000","last_unstale":"2024-08-19T14:22:08.178234+0000","last_undegraded":"2024-08-19T14:22:08.178234+0000","last_fullsized":"2024-08-19T14:22:08.178234+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":6449,"log_size":8301,"log_dups_size":0,"ondisk_log_size":8301,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 1s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520880152,"num_objects":7949,"num_object_clones":0,"num_object_copies":23847,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7949,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8301,"num_write_kb":519937,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,1],"acting":[4,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"70'139","reported_seq":880,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-08-19T14:22:05.061427+0000","last_change":"2024-08-19T14:21:37.951800+0000","last_active":"2024-08-19T14:22:05.061427+0000","last_peered":"2024-08-19T14:22:05.061427+0000","last_clean":"2024-08-19T14:22:05.061427+0000","last_became_active":"2024-08-19T14:18:26.887277+0000","last_became_peered":"2024-08-19T14:18:26.887277+0000","last_unstale":"2024-08-19T14:22:05.061427+0000","last_undegraded":"2024-08-19T14:22:05.061427+0000","last_fullsized":"2024-08-19T14:22:05.061427+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:37.951694+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:21:37.951694+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:03.860882+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,8],"acting":[5,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"374'8246","reported_seq":8334,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-08-19T14:22:08.169788+0000","last_change":"2024-08-19T14:20:29.209096+0000","last_active":"2024-08-19T14:22:08.169788+0000","last_peered":"2024-08-19T14:22:08.169788+0000","last_clean":"2024-08-19T14:22:08.169788+0000","last_became_active":"2024-08-19T14:20:29.208748+0000","last_became_peered":"2024-08-19T14:20:29.208748+0000","last_unstale":"2024-08-19T14:22:08.169788+0000","last_undegraded":"2024-08-19T14:22:08.169788+0000","last_fullsized":"2024-08-19T14:22:08.169788+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":8246,"log_dups_size":0,"ondisk_log_size":8246,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":517210112,"num_objects":7892,"num_object_clones":0,"num_object_copies":23676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7892,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8246,"num_write_kb":516416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"70'139","reported_seq":881,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-08-19T14:22:05.061210+0000","last_change":"2024-08-19T14:22:02.413212+0000","last_active":"2024-08-19T14:22:05.061210+0000","last_peered":"2024-08-19T14:22:05.061210+0000","last_clean":"2024-08-19T14:22:05.061210+0000","last_became_active":"2024-08-19T14:18:26.875179+0000","last_became_peered":"2024-08-19T14:18:26.875179+0000","last_unstale":"2024-08-19T14:22:05.061210+0000","last_undegraded":"2024-08-19T14:22:05.061210+0000","last_fullsized":"2024-08-19T14:22:05.061210+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:22:02.413091+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:22:02.413091+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:23.082273+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.b","version":"372'8253","reported_seq":8408,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-08-19T14:22:06.202378+0000","last_change":"2024-08-19T14:21:39.089676+0000","last_active":"2024-08-19T14:22:06.202378+0000","last_peered":"2024-08-19T14:22:06.202378+0000","last_clean":"2024-08-19T14:22:06.202378+0000","last_became_active":"2024-08-19T14:20:29.207372+0000","last_became_peered":"2024-08-19T14:20:29.207372+0000","last_unstale":"2024-08-19T14:22:06.202378+0000","last_undegraded":"2024-08-19T14:22:06.202378+0000","last_fullsized":"2024-08-19T14:22:06.202378+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"363'6615","last_scrub_stamp":"2024-08-19T14:21:39.089400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:21:39.089400+0000","objects_scrubbed":6552,"log_size":8253,"log_dups_size":0,"ondisk_log_size":8253,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:05.632119+0000","scrub_duration":1907,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":530120704,"num_objects":8089,"num_object_clones":0,"num_object_copies":24267,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8089,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8253,"num_write_kb":522944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"70'139","reported_seq":539,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-08-19T14:22:08.050527+0000","last_change":"2024-08-19T14:21:44.434481+0000","last_active":"2024-08-19T14:22:08.050527+0000","last_peered":"2024-08-19T14:22:08.050527+0000","last_clean":"2024-08-19T14:22:08.050527+0000","last_became_active":"2024-08-19T14:18:26.884105+0000","last_became_peered":"2024-08-19T14:18:26.884105+0000","last_unstale":"2024-08-19T14:22:08.050527+0000","last_undegraded":"2024-08-19T14:22:08.050527+0000","last_fullsized":"2024-08-19T14:22:08.050527+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:44.434382+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:20:15.321945+0000","last_clean_scrub_stamp":"2024-08-19T14:21:44.434382+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:13.544241+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"374'8542","reported_seq":8630,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-08-19T14:22:09.260378+0000","last_change":"2024-08-19T14:20:29.269723+0000","last_active":"2024-08-19T14:22:09.260378+0000","last_peered":"2024-08-19T14:22:09.260378+0000","last_clean":"2024-08-19T14:22:09.260378+0000","last_became_active":"2024-08-19T14:20:29.269432+0000","last_became_peered":"2024-08-19T14:20:29.269432+0000","last_unstale":"2024-08-19T14:22:09.260378+0000","last_undegraded":"2024-08-19T14:22:09.260378+0000","last_fullsized":"2024-08-19T14:22:09.260378+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":8542,"log_dups_size":0,"ondisk_log_size":8542,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":529137664,"num_objects":8074,"num_object_clones":0,"num_object_copies":24222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8074,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8542,"num_write_kb":531712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.f","version":"70'139","reported_seq":779,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-08-19T14:22:05.061496+0000","last_change":"2024-08-19T14:21:14.765419+0000","last_active":"2024-08-19T14:22:05.061496+0000","last_peered":"2024-08-19T14:22:05.061496+0000","last_clean":"2024-08-19T14:22:05.061496+0000","last_became_active":"2024-08-19T14:18:18.711708+0000","last_became_peered":"2024-08-19T14:18:18.711708+0000","last_unstale":"2024-08-19T14:22:05.061496+0000","last_undegraded":"2024-08-19T14:22:05.061496+0000","last_fullsized":"2024-08-19T14:22:05.061496+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:14.765340+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:21:14.765340+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:22:24.415219+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"372'8270","reported_seq":8433,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-08-19T14:22:05.797270+0000","last_change":"2024-08-19T14:21:43.974700+0000","last_active":"2024-08-19T14:22:05.797270+0000","last_peered":"2024-08-19T14:22:05.797270+0000","last_clean":"2024-08-19T14:22:05.797270+0000","last_became_active":"2024-08-19T14:20:29.211975+0000","last_became_peered":"2024-08-19T14:20:29.211975+0000","last_unstale":"2024-08-19T14:22:05.797270+0000","last_undegraded":"2024-08-19T14:22:05.797270+0000","last_fullsized":"2024-08-19T14:22:05.797270+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"364'6991","last_scrub_stamp":"2024-08-19T14:21:43.974613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:21:43.974613+0000","objects_scrubbed":6934,"log_size":8270,"log_dups_size":0,"ondisk_log_size":8270,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:07.463188+0000","scrub_duration":1682,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":536608768,"num_objects":8188,"num_object_clones":0,"num_object_copies":24564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8270,"num_write_kb":526656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,3],"acting":[11,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.c","version":"70'139","reported_seq":644,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-08-19T14:22:08.049594+0000","last_change":"2024-08-19T14:21:40.181472+0000","last_active":"2024-08-19T14:22:08.049594+0000","last_peered":"2024-08-19T14:22:08.049594+0000","last_clean":"2024-08-19T14:22:08.049594+0000","last_became_active":"2024-08-19T14:18:26.872942+0000","last_became_peered":"2024-08-19T14:18:26.872942+0000","last_unstale":"2024-08-19T14:22:08.049594+0000","last_undegraded":"2024-08-19T14:22:08.049594+0000","last_fullsized":"2024-08-19T14:22:08.049594+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:40.181356+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:20:07.839076+0000","last_clean_scrub_stamp":"2024-08-19T14:21:40.181356+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:22:52.009381+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,9],"acting":[3,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"372'8232","reported_seq":8316,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-08-19T14:22:05.823217+0000","last_change":"2024-08-19T14:20:29.268318+0000","last_active":"2024-08-19T14:22:05.823217+0000","last_peered":"2024-08-19T14:22:05.823217+0000","last_clean":"2024-08-19T14:22:05.823217+0000","last_became_active":"2024-08-19T14:20:29.267442+0000","last_became_peered":"2024-08-19T14:20:29.267442+0000","last_unstale":"2024-08-19T14:22:05.823217+0000","last_undegraded":"2024-08-19T14:22:05.823217+0000","last_fullsized":"2024-08-19T14:22:05.823217+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":8232,"log_dups_size":0,"ondisk_log_size":8232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":532938752,"num_objects":8132,"num_object_clones":0,"num_object_copies":24396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8132,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8232,"num_write_kb":523648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,9],"acting":[11,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.d","version":"70'139","reported_seq":559,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-08-19T14:22:08.048197+0000","last_change":"2024-08-19T14:21:07.155294+0000","last_active":"2024-08-19T14:22:08.048197+0000","last_peered":"2024-08-19T14:22:08.048197+0000","last_clean":"2024-08-19T14:22:08.048197+0000","last_became_active":"2024-08-19T14:18:26.882002+0000","last_became_peered":"2024-08-19T14:18:26.882002+0000","last_unstale":"2024-08-19T14:22:08.048197+0000","last_undegraded":"2024-08-19T14:22:08.048197+0000","last_fullsized":"2024-08-19T14:22:08.048197+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:07.155157+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:21:07.155157+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:22:11.274708+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"70'139","reported_seq":814,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-08-19T14:22:05.065344+0000","last_change":"2024-08-19T14:21:13.777891+0000","last_active":"2024-08-19T14:22:05.065344+0000","last_peered":"2024-08-19T14:22:05.065344+0000","last_clean":"2024-08-19T14:22:05.065344+0000","last_became_active":"2024-08-19T14:18:18.703594+0000","last_became_peered":"2024-08-19T14:18:18.703594+0000","last_unstale":"2024-08-19T14:22:05.065344+0000","last_undegraded":"2024-08-19T14:22:05.065344+0000","last_fullsized":"2024-08-19T14:22:05.065344+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:13.777777+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:21:13.777777+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:22:38.473560+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"70'139","reported_seq":870,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-08-19T14:22:08.050568+0000","last_change":"2024-08-19T14:20:07.692054+0000","last_active":"2024-08-19T14:22:08.050568+0000","last_peered":"2024-08-19T14:22:08.050568+0000","last_clean":"2024-08-19T14:22:08.050568+0000","last_became_active":"2024-08-19T14:18:18.698334+0000","last_became_peered":"2024-08-19T14:18:18.698334+0000","last_unstale":"2024-08-19T14:22:08.050568+0000","last_undegraded":"2024-08-19T14:22:08.050568+0000","last_fullsized":"2024-08-19T14:22:08.050568+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:07.691932+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:07.691932+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.10","version":"70'139","reported_seq":969,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-08-19T14:22:05.061303+0000","last_change":"2024-08-19T14:21:07.768770+0000","last_active":"2024-08-19T14:22:05.061303+0000","last_peered":"2024-08-19T14:22:05.061303+0000","last_clean":"2024-08-19T14:22:05.061303+0000","last_became_active":"2024-08-19T14:18:18.704562+0000","last_became_peered":"2024-08-19T14:18:18.704562+0000","last_unstale":"2024-08-19T14:22:05.061303+0000","last_undegraded":"2024-08-19T14:22:05.061303+0000","last_fullsized":"2024-08-19T14:22:05.061303+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:07.768697+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:21:07.768697+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:22:10.360289+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,8],"acting":[5,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"70'139","reported_seq":518,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-08-19T14:22:05.014021+0000","last_change":"2024-08-19T14:21:16.071812+0000","last_active":"2024-08-19T14:22:05.014021+0000","last_peered":"2024-08-19T14:22:05.014021+0000","last_clean":"2024-08-19T14:22:05.014021+0000","last_became_active":"2024-08-19T14:18:18.697894+0000","last_became_peered":"2024-08-19T14:18:18.697894+0000","last_unstale":"2024-08-19T14:22:05.014021+0000","last_undegraded":"2024-08-19T14:22:05.014021+0000","last_fullsized":"2024-08-19T14:22:05.014021+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:16.071703+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:21:16.071703+0000","last_clean_scrub_stamp":"2024-08-19T14:21:16.071703+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:22:18.184091+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"70'139","reported_seq":499,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-08-19T14:22:08.062042+0000","last_change":"2024-08-19T14:20:28.205212+0000","last_active":"2024-08-19T14:22:08.062042+0000","last_peered":"2024-08-19T14:22:08.062042+0000","last_clean":"2024-08-19T14:22:08.062042+0000","last_became_active":"2024-08-19T14:20:28.204724+0000","last_became_peered":"2024-08-19T14:20:28.204724+0000","last_unstale":"2024-08-19T14:22:08.062042+0000","last_undegraded":"2024-08-19T14:22:08.062042+0000","last_fullsized":"2024-08-19T14:22:08.062042+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:08.680021+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:08.680021+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"70'139","reported_seq":621,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-08-19T14:22:02.722438+0000","last_change":"2024-08-19T14:22:02.722505+0000","last_active":"2024-08-19T14:22:02.722438+0000","last_peered":"2024-08-19T14:22:02.722438+0000","last_clean":"2024-08-19T14:22:02.722438+0000","last_became_active":"2024-08-19T14:18:31.901556+0000","last_became_peered":"2024-08-19T14:18:31.901556+0000","last_unstale":"2024-08-19T14:22:02.722438+0000","last_undegraded":"2024-08-19T14:22:02.722438+0000","last_fullsized":"2024-08-19T14:22:02.722438+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:22:02.722345+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:22:02.722345+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:13.381470+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,0],"acting":[8,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.14","version":"70'139","reported_seq":628,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-08-19T14:22:08.048260+0000","last_change":"2024-08-19T14:19:55.130518+0000","last_active":"2024-08-19T14:22:08.048260+0000","last_peered":"2024-08-19T14:22:08.048260+0000","last_clean":"2024-08-19T14:22:08.048260+0000","last_became_active":"2024-08-19T14:18:18.703926+0000","last_became_peered":"2024-08-19T14:18:18.703926+0000","last_unstale":"2024-08-19T14:22:08.048260+0000","last_undegraded":"2024-08-19T14:22:08.048260+0000","last_fullsized":"2024-08-19T14:22:08.048260+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:19:55.130420+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:19:55.130420+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,1],"acting":[7,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.15","version":"70'139","reported_seq":515,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-08-19T14:22:08.048135+0000","last_change":"2024-08-19T14:20:09.181046+0000","last_active":"2024-08-19T14:22:08.048135+0000","last_peered":"2024-08-19T14:22:08.048135+0000","last_clean":"2024-08-19T14:22:08.048135+0000","last_became_active":"2024-08-19T14:18:31.912496+0000","last_became_peered":"2024-08-19T14:18:31.912496+0000","last_unstale":"2024-08-19T14:22:08.048135+0000","last_undegraded":"2024-08-19T14:22:08.048135+0000","last_fullsized":"2024-08-19T14:22:08.048135+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:09.180923+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:20:09.180923+0000","last_clean_scrub_stamp":"2024-08-19T14:20:09.180923+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"70'139","reported_seq":501,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-08-19T14:22:08.050505+0000","last_change":"2024-08-19T14:21:40.430835+0000","last_active":"2024-08-19T14:22:08.050505+0000","last_peered":"2024-08-19T14:22:08.050505+0000","last_clean":"2024-08-19T14:22:08.050505+0000","last_became_active":"2024-08-19T14:18:31.901229+0000","last_became_peered":"2024-08-19T14:18:31.901229+0000","last_unstale":"2024-08-19T14:22:08.050505+0000","last_undegraded":"2024-08-19T14:22:08.050505+0000","last_fullsized":"2024-08-19T14:22:08.050505+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:40.430714+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:21:40.430714+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:04.072653+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1b","version":"70'139","reported_seq":847,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-08-19T14:22:00.975199+0000","last_change":"2024-08-19T14:21:48.693864+0000","last_active":"2024-08-19T14:22:00.975199+0000","last_peered":"2024-08-19T14:22:00.975199+0000","last_clean":"2024-08-19T14:22:00.975199+0000","last_became_active":"2024-08-19T14:18:31.901152+0000","last_became_peered":"2024-08-19T14:18:31.901152+0000","last_unstale":"2024-08-19T14:22:00.975199+0000","last_undegraded":"2024-08-19T14:22:00.975199+0000","last_fullsized":"2024-08-19T14:22:00.975199+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:48.693761+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:21:48.693761+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:07.571130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.18","version":"70'139","reported_seq":885,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-08-19T14:22:05.061293+0000","last_change":"2024-08-19T14:20:36.510554+0000","last_active":"2024-08-19T14:22:05.061293+0000","last_peered":"2024-08-19T14:22:05.061293+0000","last_clean":"2024-08-19T14:22:05.061293+0000","last_became_active":"2024-08-19T14:18:31.902735+0000","last_became_peered":"2024-08-19T14:18:31.902735+0000","last_unstale":"2024-08-19T14:22:05.061293+0000","last_undegraded":"2024-08-19T14:22:05.061293+0000","last_fullsized":"2024-08-19T14:22:05.061293+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:36.510442+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:36.510442+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.19","version":"70'139","reported_seq":841,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-08-19T14:22:05.013913+0000","last_change":"2024-08-19T14:21:40.219949+0000","last_active":"2024-08-19T14:22:05.013913+0000","last_peered":"2024-08-19T14:22:05.013913+0000","last_clean":"2024-08-19T14:22:05.013913+0000","last_became_active":"2024-08-19T14:18:31.902547+0000","last_became_peered":"2024-08-19T14:18:31.902547+0000","last_unstale":"2024-08-19T14:22:05.013913+0000","last_undegraded":"2024-08-19T14:22:05.013913+0000","last_fullsized":"2024-08-19T14:22:05.013913+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:40.219830+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:21:40.219830+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:05.414864+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1e","version":"70'139","reported_seq":491,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-08-19T14:22:05.065774+0000","last_change":"2024-08-19T14:20:18.771836+0000","last_active":"2024-08-19T14:22:05.065774+0000","last_peered":"2024-08-19T14:22:05.065774+0000","last_clean":"2024-08-19T14:22:05.065774+0000","last_became_active":"2024-08-19T14:18:31.902546+0000","last_became_peered":"2024-08-19T14:18:31.902546+0000","last_unstale":"2024-08-19T14:22:05.065774+0000","last_undegraded":"2024-08-19T14:22:05.065774+0000","last_fullsized":"2024-08-19T14:22:05.065774+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:18.771704+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:18.771704+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1c","version":"70'139","reported_seq":605,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-08-19T14:22:00.975417+0000","last_change":"2024-08-19T14:21:38.753663+0000","last_active":"2024-08-19T14:22:00.975417+0000","last_peered":"2024-08-19T14:22:00.975417+0000","last_clean":"2024-08-19T14:22:00.975417+0000","last_became_active":"2024-08-19T14:18:31.902494+0000","last_became_peered":"2024-08-19T14:18:31.902494+0000","last_unstale":"2024-08-19T14:22:00.975417+0000","last_undegraded":"2024-08-19T14:22:00.975417+0000","last_fullsized":"2024-08-19T14:22:00.975417+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:38.753553+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:21:38.753553+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:22:46.720742+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,11],"acting":[8,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1d","version":"70'139","reported_seq":525,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-08-19T14:22:08.052151+0000","last_change":"2024-08-19T14:20:17.725615+0000","last_active":"2024-08-19T14:22:08.052151+0000","last_peered":"2024-08-19T14:22:08.052151+0000","last_clean":"2024-08-19T14:22:08.052151+0000","last_became_active":"2024-08-19T14:18:31.901183+0000","last_became_peered":"2024-08-19T14:18:31.901183+0000","last_unstale":"2024-08-19T14:22:08.052151+0000","last_undegraded":"2024-08-19T14:22:08.052151+0000","last_fullsized":"2024-08-19T14:22:08.052151+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:17.725477+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:17.725477+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,5],"acting":[1,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":8429240344,"num_objects":128621,"num_object_clones":0,"num_object_copies":385863,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2339,"num_objects_unfound":0,"num_objects_dirty":128621,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":133725,"num_write_kb":8395009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1801,"num_bytes_recovered":118030336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":25367621632,"data_stored":25367609416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":133725,"ondisk_log_size":133725,"up":48,"acting":47,"num_store_stats":9},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1802240,"data_stored":1788032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187585,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4076176,"kb_used_data":3670148,"kb_used_omap":1,"kb_used_meta":406014,"kb_avail":89668976,"statfs":{"total":95995035648,"available":91821031424,"internally_reserved":0,"allocated":3758231552,"data_stored":3756515740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":415758788},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,1,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":343597383815,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":74,"seq":317827580043,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2387164,"kb_used_data":1918676,"kb_used_omap":1,"kb_used_meta":468478,"kb_avail":91357988,"statfs":{"total":95995035648,"available":93550579712,"internally_reserved":0,"allocated":1964724224,"data_stored":1963000164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":479721921},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,3,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874384,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":902016,"kb_used_data":527928,"kb_used_omap":1,"kb_used_meta":374078,"kb_avail":92843136,"statfs":{"total":95995035648,"available":95071371264,"internally_reserved":0,"allocated":540598272,"data_stored":538886636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":383056325},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":57,"seq":244813136019,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4592180,"kb_used_data":4123620,"kb_used_omap":1,"kb_used_meta":468478,"kb_avail":89152972,"statfs":{"total":95995035648,"available":91292643328,"internally_reserved":0,"allocated":4222586880,"data_stored":4220859164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":479721930},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":206158430359,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691995,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2177684,"kb_used_data":1843852,"kb_used_omap":1,"kb_used_meta":333822,"kb_avail":91567468,"statfs":{"total":95995035648,"available":93765087232,"internally_reserved":0,"allocated":1888104448,"data_stored":1886385084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":341834181},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986336,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3033052,"kb_used_data":2568724,"kb_used_omap":1,"kb_used_meta":464318,"kb_avail":90712100,"statfs":{"total":95995035648,"available":92889190400,"internally_reserved":0,"allocated":2630373376,"data_stored":2628649316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":475462085},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149862,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3480108,"kb_used_data":3047708,"kb_used_omap":1,"kb_used_meta":432382,"kb_avail":90265044,"statfs":{"total":95995035648,"available":92431405056,"internally_reserved":0,"allocated":3120852992,"data_stored":3119128828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":442759623},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411499,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575023,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5672184,"kb_used_data":5127080,"kb_used_omap":1,"kb_used_meta":544830,"kb_avail":88072968,"statfs":{"total":95995035648,"available":90186719232,"internally_reserved":0,"allocated":5250129920,"data_stored":5248398132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":557906373},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,4,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738547,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2446284,"kb_used_data":2081924,"kb_used_omap":1,"kb_used_meta":364350,"kb_avail":91298868,"statfs":{"total":95995035648,"available":93490040832,"internally_reserved":0,"allocated":2131890176,"data_stored":2130174364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":373094853},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2124480512,"data_stored":2124480512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5213982720,"data_stored":5213978648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3104178176,"data_stored":3104178176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2612334592,"data_stored":2612330520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1877213184,"data_stored":1877213184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4208394240,"data_stored":4208394240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":536150016,"data_stored":536150016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1949175808,"data_stored":1949171736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":3741712384,"data_stored":3741712384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T14:22:11.815 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph status --format=json 2024-08-19T14:22:11.834 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:22:11.859 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:11.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:11.860 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:22:11.961 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:22:11.991 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:11.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:11.993 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:22:12.051 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:22:12.094 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:22:12.118 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:12.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:12.119 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:22:12.220 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:22:12.243 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:12.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:12.244 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:22:12.345 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:22:12.355 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:12 smithi031 ceph-mon[16336]: 4.e scrub starts 2024-08-19T14:22:12.355 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:12 smithi031 ceph-mon[16336]: 1.10 scrub starts 2024-08-19T14:22:12.355 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:12 smithi031 ceph-mon[16336]: 1.10 scrub ok 2024-08-19T14:22:12.355 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:12 smithi031 ceph-mon[16336]: 4.8 scrub starts 2024-08-19T14:22:12.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:12.386 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:12.386 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:22:12.487 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:22:12.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:12 smithi032 ceph-mon[20004]: 4.e scrub starts 2024-08-19T14:22:12.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:12 smithi032 ceph-mon[20004]: 1.10 scrub starts 2024-08-19T14:22:12.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:12 smithi032 ceph-mon[20004]: 1.10 scrub ok 2024-08-19T14:22:12.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:12 smithi032 ceph-mon[20004]: 4.8 scrub starts 2024-08-19T14:22:12.518 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:12.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:12.519 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:22:12.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:12 smithi045 ceph-mon[20143]: 4.e scrub starts 2024-08-19T14:22:12.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:12 smithi045 ceph-mon[20143]: 1.10 scrub starts 2024-08-19T14:22:12.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:12 smithi045 ceph-mon[20143]: 1.10 scrub ok 2024-08-19T14:22:12.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:12 smithi045 ceph-mon[20143]: 4.8 scrub starts 2024-08-19T14:22:12.620 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:22:12.649 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:12.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:12.650 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:22:12.670 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-08-19T14:22:12.680 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:22:12.751 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:22:12.775 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:12.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:12.776 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:22:12.877 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:22:12.905 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:12.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:12.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:22:13.007 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:22:13.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:13.031 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:13.032 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:22:13.132 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:22:13.158 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:13.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:13.160 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:22:13.261 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:22:13.290 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:13.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:13.291 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:22:13.355 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:13 smithi031 ceph-mon[16336]: pgmap v843: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 45 active+clean; 7.7 GiB data, 27 GiB used, 777 GiB / 805 GiB avail; 148 B/s rd, 12 MiB/s wr, 826 op/s; 2339/380439 objects misplaced (0.615%); 2.2 MiB/s, 35 objects/s recovering 2024-08-19T14:22:13.355 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:13 smithi031 ceph-mon[16336]: 4.5 scrub starts 2024-08-19T14:22:13.392 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:22:13.413 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:13.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:13.413 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:22:13.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:13 smithi032 ceph-mon[20004]: pgmap v843: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 45 active+clean; 7.7 GiB data, 27 GiB used, 777 GiB / 805 GiB avail; 148 B/s rd, 12 MiB/s wr, 826 op/s; 2339/380439 objects misplaced (0.615%); 2.2 MiB/s, 35 objects/s recovering 2024-08-19T14:22:13.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:13 smithi032 ceph-mon[20004]: 4.5 scrub starts 2024-08-19T14:22:13.514 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:22:13.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:13 smithi045 ceph-mon[20143]: pgmap v843: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 45 active+clean; 7.7 GiB data, 27 GiB used, 777 GiB / 805 GiB avail; 148 B/s rd, 12 MiB/s wr, 826 op/s; 2339/380439 objects misplaced (0.615%); 2.2 MiB/s, 35 objects/s recovering 2024-08-19T14:22:13.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:13 smithi045 ceph-mon[20143]: 4.5 scrub starts 2024-08-19T14:22:13.555 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:13.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:13.556 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:22:13.657 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:22:13.685 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:13.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:13.686 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:22:13.787 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:22:13.812 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:13.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:13.813 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:22:13.914 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:22:13.942 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:13.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:13.943 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:22:14.044 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:22:14.071 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:14.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:14.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:22:14.171 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is unset 2024-08-19T14:22:14.173 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:22:14.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:14.213 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:14.213 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:22:14.314 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:22:14.345 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:14.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:14.346 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:22:14.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:14 smithi031 ceph-mon[16336]: 4.8 scrub ok 2024-08-19T14:22:14.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:14 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2130883045' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:22:14.446 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:22:14.467 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:14.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:14.468 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:22:14.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:14 smithi032 ceph-mon[20004]: 4.8 scrub ok 2024-08-19T14:22:14.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:14 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2130883045' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:22:14.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:14 smithi045 ceph-mon[20143]: 4.8 scrub ok 2024-08-19T14:22:14.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:14 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2130883045' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:22:14.569 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:22:14.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:14.596 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:14.596 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:22:14.697 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:22:14.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:14.724 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:14.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:22:14.812 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:22:14.825 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:22:14.851 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:14.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:14.853 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:22:14.954 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:22:14.984 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:14.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:14.985 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:22:15.086 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:22:15.116 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:15.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:15.117 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:22:15.218 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:22:15.252 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:15.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:15.253 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:22:15.354 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:22:15.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:15 smithi031 ceph-mon[16336]: 4.e scrub ok 2024-08-19T14:22:15.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:15 smithi031 ceph-mon[16336]: pgmap v844: 47 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 45 active+clean; 7.7 GiB data, 27 GiB used, 778 GiB / 805 GiB avail; 127 B/s rd, 2.9 MiB/s wr, 712 op/s; 2339/376803 objects misplaced (0.621%); 1.9 MiB/s, 30 objects/s recovering 2024-08-19T14:22:15.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:15 smithi031 ceph-mon[16336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:22:15.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:15 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2130883045' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:22:15.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:15 smithi031 ceph-mon[16336]: osdmap e375: 12 total, 12 up, 9 in 2024-08-19T14:22:15.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:15 smithi031 ceph-mon[16336]: 4.5 scrub ok 2024-08-19T14:22:15.377 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:15.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:15.378 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:22:15.480 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:22:15.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:15 smithi032 ceph-mon[20004]: 4.e scrub ok 2024-08-19T14:22:15.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:15 smithi032 ceph-mon[20004]: pgmap v844: 47 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 45 active+clean; 7.7 GiB data, 27 GiB used, 778 GiB / 805 GiB avail; 127 B/s rd, 2.9 MiB/s wr, 712 op/s; 2339/376803 objects misplaced (0.621%); 1.9 MiB/s, 30 objects/s recovering 2024-08-19T14:22:15.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:15 smithi032 ceph-mon[20004]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:22:15.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:15 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2130883045' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:22:15.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:15 smithi032 ceph-mon[20004]: osdmap e375: 12 total, 12 up, 9 in 2024-08-19T14:22:15.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:15 smithi032 ceph-mon[20004]: 4.5 scrub ok 2024-08-19T14:22:15.510 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:15.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:15.512 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:22:15.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:15 smithi045 ceph-mon[20143]: 4.e scrub ok 2024-08-19T14:22:15.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:15 smithi045 ceph-mon[20143]: pgmap v844: 47 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 45 active+clean; 7.7 GiB data, 27 GiB used, 778 GiB / 805 GiB avail; 127 B/s rd, 2.9 MiB/s wr, 712 op/s; 2339/376803 objects misplaced (0.621%); 1.9 MiB/s, 30 objects/s recovering 2024-08-19T14:22:15.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:15 smithi045 ceph-mon[20143]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:22:15.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:15 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2130883045' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:22:15.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:15 smithi045 ceph-mon[20143]: osdmap e375: 12 total, 12 up, 9 in 2024-08-19T14:22:15.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:15 smithi045 ceph-mon[20143]: 4.5 scrub ok 2024-08-19T14:22:15.612 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:22:15.636 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:15.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:15.637 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:22:15.738 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:22:15.803 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:15.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:15.804 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:22:15.905 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:22:15.933 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:15.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:15.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:22:16.035 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:22:16.059 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:22:16.061 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:16.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:16.062 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:22:16.163 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:22:16.201 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:16.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:16.202 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:22:16.303 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:22:16.348 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:16.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:16.350 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:22:16.385 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:22:16.451 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:22:16.474 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:16.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:16.475 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:22:16.491 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:22:16.491 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"ed7d1c6e-5e33-11ef-bccf-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 1 pg undersized","count":1},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":913,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":375,"num_osds":12,"num_up_osds":12,"osd_up_since":1724076699,"num_in_osds":9,"osd_in_since":1724076998,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":45},{"state_name":"active+clean+scrubbing","count":1},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":47,"num_pools":2,"num_objects":122311,"data_bytes":8016167480,"bytes_used":28383465472,"bytes_avail":835571855360,"bytes_total":863955320832,"misplaced_objects":2196,"misplaced_total":366933,"misplaced_ratio":0.0059847438088152342,"recovering_objects_per_sec":48,"recovering_bytes_per_sec":3153041,"recovering_keys_per_sec":0,"num_objects_recovered":385,"num_bytes_recovered":25231360,"num_keys_recovered":0,"read_bytes_sec":127,"write_bytes_sec":1711779,"read_op_per_sec":0,"write_op_per_sec":1093},"fsmap":{"epoch":1,"btime":"2024-08-19T14:05:22:841940+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.31:8443/"}},"servicemap":{"epoch":22,"modified":"2024-08-19T14:22:15.909544+0000","services":{"mon":{"daemons":{"summary":"","c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-08-19T14:22:16.577 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:22:16.604 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:16.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:16.605 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:22:16.706 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:22:16.743 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:16.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:16.744 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:22:16.842 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:22:16.842 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:22:16.845 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:22:16.870 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:16.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:16.871 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:22:16.972 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:22:17.000 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:17.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:17.001 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:22:17.102 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:22:17.130 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:17.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:17.131 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:22:17.232 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:22:17.258 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:17.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:17.259 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:22:17.361 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:22:17.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:17 smithi031 ceph-mon[16336]: pgmap v846: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 45 active+clean; 7.5 GiB data, 26 GiB used, 778 GiB / 805 GiB avail; 127 B/s rd, 1.6 MiB/s wr, 1.09k op/s; 2196/366933 objects misplaced (0.598%); 3.0 MiB/s, 48 objects/s recovering 2024-08-19T14:22:17.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:17 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2707633518' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-08-19T14:22:17.386 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:17.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:17.388 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:22:17.426 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-19T14:22:17.489 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:22:17.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:17 smithi032 ceph-mon[20004]: pgmap v846: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 45 active+clean; 7.5 GiB data, 26 GiB used, 778 GiB / 805 GiB avail; 127 B/s rd, 1.6 MiB/s wr, 1.09k op/s; 2196/366933 objects misplaced (0.598%); 3.0 MiB/s, 48 objects/s recovering 2024-08-19T14:22:17.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:17 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2707633518' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-08-19T14:22:17.514 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:17.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:17.515 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:22:17.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:17 smithi045 ceph-mon[20143]: pgmap v846: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 45 active+clean; 7.5 GiB data, 26 GiB used, 778 GiB / 805 GiB avail; 127 B/s rd, 1.6 MiB/s wr, 1.09k op/s; 2196/366933 objects misplaced (0.598%); 3.0 MiB/s, 48 objects/s recovering 2024-08-19T14:22:17.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:17 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2707633518' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-08-19T14:22:17.612 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-08-19T14:22:17.616 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:22:17.641 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:17.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:17.642 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:22:17.742 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:22:17.766 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:17.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:17.767 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:22:17.868 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:22:17.895 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:17.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:17.896 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:22:17.997 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:22:18.021 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:18.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:18.022 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:22:18.123 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:22:18.145 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:18.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:18.146 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:22:18.247 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:22:18.274 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:18.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:18.275 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:22:18.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:18 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:22:18.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:18 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:22:18.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:18 smithi031 ceph-mon[16336]: osdmap e376: 12 total, 12 up, 9 in 2024-08-19T14:22:18.376 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:22:18.412 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:18.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:18.413 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:22:18.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:18 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:22:18.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:18 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:22:18.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:18 smithi032 ceph-mon[20004]: osdmap e376: 12 total, 12 up, 9 in 2024-08-19T14:22:18.514 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:22:18.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:18.542 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:18.542 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:22:18.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:18 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:22:18.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:18 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:22:18.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:18 smithi045 ceph-mon[20143]: osdmap e376: 12 total, 12 up, 9 in 2024-08-19T14:22:18.642 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:22:18.733 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:18.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:18.735 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:22:18.836 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:22:18.863 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:18.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:18.864 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:22:18.965 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:22:18.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:18.994 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:18.994 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:22:19.096 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:22:19.109 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-08-19T14:22:19.109 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-19T14:22:19.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:19.123 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:19.123 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:22:19.224 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:22:19.253 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:19.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:19.254 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:22:19.355 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:22:19.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:19 smithi031 ceph-mon[16336]: pgmap v848: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 45 active+clean; 7.4 GiB data, 26 GiB used, 778 GiB / 805 GiB avail; 0 B/s wr, 826 op/s; 2196/366030 objects misplaced (0.600%); 1.1 MiB/s, 17 objects/s recovering 2024-08-19T14:22:19.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:19.402 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:19.402 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:22:19.503 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:22:19.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:19 smithi032 ceph-mon[20004]: pgmap v848: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 45 active+clean; 7.4 GiB data, 26 GiB used, 778 GiB / 805 GiB avail; 0 B/s wr, 826 op/s; 2196/366030 objects misplaced (0.600%); 1.1 MiB/s, 17 objects/s recovering 2024-08-19T14:22:19.532 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:19.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:19.533 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:22:19.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:19 smithi045 ceph-mon[20143]: pgmap v848: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 45 active+clean; 7.4 GiB data, 26 GiB used, 778 GiB / 805 GiB avail; 0 B/s wr, 826 op/s; 2196/366030 objects misplaced (0.600%); 1.1 MiB/s, 17 objects/s recovering 2024-08-19T14:22:19.634 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:22:19.659 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:19.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:19.660 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:22:19.674 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:22:19.691 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:22:19.725 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:22:19.727 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:22:19.727 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:22:19.761 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:22:19.767 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:22:19.768 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:22:19.799 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:19.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:19.801 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:22:19.801 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:22:19.801 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:22:19.835 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:22:19.835 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:22:19.865 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:22:19.865 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:22:19.895 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:22:19.895 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:22:19.902 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:22:19.930 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:19.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:19.932 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:22:19.932 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:22:19.932 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:22:19.971 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:22:19.971 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:22:20.006 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:22:20.006 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:22:20.033 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:22:20.040 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:22:20.041 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:22:20.073 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:20.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:20.074 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:22:20.083 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:22:20.084 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:22:20.119 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:22:20.119 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:22:20.174 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:22:20.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:20.211 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:20.211 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:22:20.246 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:20 smithi031 ceph-mon[16336]: 4.1 scrub starts 2024-08-19T14:22:20.246 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:20 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/4225858166' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:22:20.313 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:22:20.355 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:20.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:20.356 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:22:20.408 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:20 smithi032 ceph-mon[20004]: 4.1 scrub starts 2024-08-19T14:22:20.408 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:20 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/4225858166' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:22:20.457 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:22:20.484 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:20.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:20.485 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:22:20.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:20 smithi045 ceph-mon[20143]: 4.1 scrub starts 2024-08-19T14:22:20.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:20 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/4225858166' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:22:20.586 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:22:20.612 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:20.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:20.613 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:22:20.714 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:22:20.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:20.742 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:20.743 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:22:20.844 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:22:20.866 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:20.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:20.867 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:22:20.968 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:22:21.026 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:21.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:21.027 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:22:21.128 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:22:21.154 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:21.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:21.155 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:22:21.256 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:22:21.276 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:21.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:21.277 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:22:21.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:21 smithi031 ceph-mon[16336]: pgmap v849: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 7.3 GiB data, 26 GiB used, 779 GiB / 805 GiB avail; 0 B/s wr, 966 op/s; 1990/357228 objects misplaced (0.557%); 2.7 MiB/s, 43 objects/s recovering 2024-08-19T14:22:21.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:21 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:22:21.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:21 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:22:21.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:21 smithi031 ceph-mon[16336]: osdmap e377: 12 total, 12 up, 9 in 2024-08-19T14:22:21.378 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:22:21.399 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:21.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:21.400 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:22:21.501 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:22:21.509 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:21 smithi032 ceph-mon[20004]: pgmap v849: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 7.3 GiB data, 26 GiB used, 779 GiB / 805 GiB avail; 0 B/s wr, 966 op/s; 1990/357228 objects misplaced (0.557%); 2.7 MiB/s, 43 objects/s recovering 2024-08-19T14:22:21.509 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:21 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:22:21.509 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:21 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:22:21.509 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:21 smithi032 ceph-mon[20004]: osdmap e377: 12 total, 12 up, 9 in 2024-08-19T14:22:21.531 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:21.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:21.532 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:22:21.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:21 smithi045 ceph-mon[20143]: pgmap v849: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 7.3 GiB data, 26 GiB used, 779 GiB / 805 GiB avail; 0 B/s wr, 966 op/s; 1990/357228 objects misplaced (0.557%); 2.7 MiB/s, 43 objects/s recovering 2024-08-19T14:22:21.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:21 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:22:21.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:21 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:22:21.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:21 smithi045 ceph-mon[20143]: osdmap e377: 12 total, 12 up, 9 in 2024-08-19T14:22:21.633 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:22:21.665 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:21.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:21.666 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:22:21.767 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:22:21.788 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:21.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:21.789 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:22:21.890 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:22:21.923 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:21.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:21.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:22:22.025 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:22:22.050 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:22.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:22.052 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:22:22.153 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:22:22.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:22.180 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:22.180 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:22:22.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:22 smithi032 ceph-mon[20004]: 4.1 scrub ok 2024-08-19T14:22:22.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:22 smithi032 ceph-mon[20004]: 1.11 scrub starts 2024-08-19T14:22:22.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:22 smithi032 ceph-mon[20004]: 1.11 scrub ok 2024-08-19T14:22:22.281 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:22:22.303 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:22.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:22.304 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:22:22.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:22 smithi031 ceph-mon[16336]: 4.1 scrub ok 2024-08-19T14:22:22.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:22 smithi031 ceph-mon[16336]: 1.11 scrub starts 2024-08-19T14:22:22.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:22 smithi031 ceph-mon[16336]: 1.11 scrub ok 2024-08-19T14:22:22.405 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:22:22.427 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:22.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:22.428 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:22:22.529 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:22:22.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:22 smithi045 ceph-mon[20143]: 4.1 scrub ok 2024-08-19T14:22:22.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:22 smithi045 ceph-mon[20143]: 1.11 scrub starts 2024-08-19T14:22:22.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:22 smithi045 ceph-mon[20143]: 1.11 scrub ok 2024-08-19T14:22:22.554 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:22.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:22.555 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:22:22.655 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:22:22.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:22.700 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:22.700 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:22:22.748 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:22:22.748 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 20, 2024-08-19T14:22:22.748 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 600, 2024-08-19T14:22:22.749 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [ 2024-08-19T14:22:22.749 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.749 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:241314 2.1 2:8e6fec04:::benchmark_data_smithi045_46415_object118033:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-08-19T14:22:22.749 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:16:40.787553+0000", 2024-08-19T14:22:22.749 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 341.64082442300003, 2024-08-19T14:22:22.749 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.29700766899999997, 2024-08-19T14:22:22.749 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:22:22.749 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:22:22.749 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:22:22.749 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:22:22.749 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:22:22.749 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:22:22.749 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 241314 2024-08-19T14:22:22.750 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.750 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:22:22.750 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.750 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:22:22.750 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.787553+0000", 2024-08-19T14:22:22.750 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:22:22.750 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.750 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.750 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:22:22.750 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.787553+0000", 2024-08-19T14:22:22.750 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:22:22.750 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.750 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.751 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:22:22.751 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.787554+0000", 2024-08-19T14:22:22.751 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5770000000000001e-06 2024-08-19T14:22:22.751 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.751 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.751 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:22:22.751 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.787560+0000", 2024-08-19T14:22:22.751 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.6930000000000004e-06 2024-08-19T14:22:22.751 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.751 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.751 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:22:22.751 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.787574+0000", 2024-08-19T14:22:22.751 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3828e-05 2024-08-19T14:22:22.751 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.752 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.752 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:22:22.752 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.787582+0000", 2024-08-19T14:22:22.752 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.225e-06 2024-08-19T14:22:22.752 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.752 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.752 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:22:22.752 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.788601+0000", 2024-08-19T14:22:22.752 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.001018833 2024-08-19T14:22:22.752 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.752 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.752 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:22:22.752 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.788734+0000", 2024-08-19T14:22:22.753 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013335699999999999 2024-08-19T14:22:22.753 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.753 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.753 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 1,2,8", 2024-08-19T14:22:22.753 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.788881+0000", 2024-08-19T14:22:22.753 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000146644 2024-08-19T14:22:22.753 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.753 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.753 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:22:22.753 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.793613+0000", 2024-08-19T14:22:22.753 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0047319600000000003 2024-08-19T14:22:22.753 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.753 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.754 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:22:22.754 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.795531+0000", 2024-08-19T14:22:22.754 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.001918335 2024-08-19T14:22:22.754 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.754 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.754 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:22:22.754 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.796093+0000", 2024-08-19T14:22:22.754 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00056226599999999996 2024-08-19T14:22:22.754 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.754 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.754 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:22:22.754 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.084346+0000", 2024-08-19T14:22:22.754 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.28825296 2024-08-19T14:22:22.755 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.755 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.755 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:22:22.755 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.084417+0000", 2024-08-19T14:22:22.755 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.0599999999999995e-05 2024-08-19T14:22:22.755 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.755 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.755 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:22:22.755 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.084560+0000", 2024-08-19T14:22:22.755 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00014339100000000001 2024-08-19T14:22:22.755 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:22:22.755 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:22:22.755 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:22:22.755 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.756 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.756 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:241325 2.1 2:8f07de09:::benchmark_data_smithi045_46415_object118044:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-08-19T14:22:22.756 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:16:40.798902+0000", 2024-08-19T14:22:22.756 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 341.62947537299999, 2024-08-19T14:22:22.756 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.29144186999999999, 2024-08-19T14:22:22.756 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:22:22.756 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:22:22.756 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:22:22.756 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:22:22.756 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:22:22.756 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:22:22.756 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 241325 2024-08-19T14:22:22.756 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.757 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:22:22.757 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.757 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:22:22.757 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.798902+0000", 2024-08-19T14:22:22.757 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:22:22.757 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.757 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.757 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:22:22.757 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.798902+0000", 2024-08-19T14:22:22.757 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:22:22.757 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.757 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.757 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:22:22.757 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.798904+0000", 2024-08-19T14:22:22.758 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.1780000000000002e-06 2024-08-19T14:22:22.758 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.758 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.758 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:22:22.758 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.798914+0000", 2024-08-19T14:22:22.758 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.7480000000000005e-06 2024-08-19T14:22:22.758 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.758 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.758 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:22:22.758 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.798933+0000", 2024-08-19T14:22:22.758 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.9026e-05 2024-08-19T14:22:22.758 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.758 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.759 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:22:22.759 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.798942+0000", 2024-08-19T14:22:22.759 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.3200000000000006e-06 2024-08-19T14:22:22.759 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.759 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.759 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:22:22.759 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.799163+0000", 2024-08-19T14:22:22.759 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000221337 2024-08-19T14:22:22.759 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.759 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.759 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:22:22.759 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.799310+0000", 2024-08-19T14:22:22.759 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00014677399999999999 2024-08-19T14:22:22.759 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.760 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.760 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 1,2,8", 2024-08-19T14:22:22.760 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.799452+0000", 2024-08-19T14:22:22.760 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00014219499999999999 2024-08-19T14:22:22.760 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.760 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.760 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:22:22.760 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.804505+0000", 2024-08-19T14:22:22.760 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.005052384 2024-08-19T14:22:22.760 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.760 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.760 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:22:22.760 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.804768+0000", 2024-08-19T14:22:22.761 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00026380700000000002 2024-08-19T14:22:22.761 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.761 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.761 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:22:22.761 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.805302+0000", 2024-08-19T14:22:22.761 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00053321300000000005 2024-08-19T14:22:22.761 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.761 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.761 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:22:22.761 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.090219+0000", 2024-08-19T14:22:22.761 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.284917107 2024-08-19T14:22:22.761 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.761 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.761 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:22:22.761 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.090271+0000", 2024-08-19T14:22:22.762 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.2289e-05 2024-08-19T14:22:22.762 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.762 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.762 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:22:22.762 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.090343+0000", 2024-08-19T14:22:22.762 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.2491999999999996e-05 2024-08-19T14:22:22.762 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:22:22.762 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:22:22.762 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:22:22.762 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.762 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.762 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:241358 2.1 2:8659030e:::benchmark_data_smithi045_46415_object118077:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-08-19T14:22:22.763 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:16:40.843360+0000", 2024-08-19T14:22:22.763 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 341.58501731199999, 2024-08-19T14:22:22.763 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.24721465500000001, 2024-08-19T14:22:22.763 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:22:22.763 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:22:22.763 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:22:22.763 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:22:22.763 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:22:22.763 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:22:22.763 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 241358 2024-08-19T14:22:22.763 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.763 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:22:22.763 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.763 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:22:22.764 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.843360+0000", 2024-08-19T14:22:22.764 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:22:22.764 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.764 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.764 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:22:22.764 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.843360+0000", 2024-08-19T14:22:22.764 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:22:22.764 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.764 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.764 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:22:22.764 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.843362+0000", 2024-08-19T14:22:22.764 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.0820000000000001e-06 2024-08-19T14:22:22.764 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.765 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.765 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:22:22.765 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.843373+0000", 2024-08-19T14:22:22.765 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1600000000000001e-05 2024-08-19T14:22:22.765 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.765 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.765 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:22:22.765 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.843395+0000", 2024-08-19T14:22:22.765 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.1670000000000001e-05 2024-08-19T14:22:22.765 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.765 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.765 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:22:22.765 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.843405+0000", 2024-08-19T14:22:22.765 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.5030000000000003e-06 2024-08-19T14:22:22.765 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.766 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.766 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:22:22.766 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.843474+0000", 2024-08-19T14:22:22.766 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.9620000000000001e-05 2024-08-19T14:22:22.766 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.766 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.766 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:22:22.766 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.843613+0000", 2024-08-19T14:22:22.766 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013863699999999999 2024-08-19T14:22:22.766 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.766 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.766 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 1,2,8", 2024-08-19T14:22:22.766 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.843771+0000", 2024-08-19T14:22:22.767 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000158691 2024-08-19T14:22:22.767 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.767 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.767 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:22:22.767 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.847962+0000", 2024-08-19T14:22:22.767 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0041908989999999997 2024-08-19T14:22:22.767 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.767 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.767 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:22:22.767 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.848335+0000", 2024-08-19T14:22:22.767 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00037213900000000001 2024-08-19T14:22:22.767 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.767 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.768 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:22:22.768 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.849083+0000", 2024-08-19T14:22:22.768 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00074856300000000005 2024-08-19T14:22:22.768 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.768 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.768 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:22:22.768 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.090392+0000", 2024-08-19T14:22:22.768 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.24130854600000001 2024-08-19T14:22:22.768 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.768 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.768 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:22:22.768 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.090453+0000", 2024-08-19T14:22:22.768 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.1635000000000002e-05 2024-08-19T14:22:22.768 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.769 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.769 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:22:22.769 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.090574+0000", 2024-08-19T14:22:22.769 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012107 2024-08-19T14:22:22.769 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:22:22.769 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:22:22.769 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:22:22.769 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.769 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.769 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:241367 2.1 2:870a5187:::benchmark_data_smithi045_46415_object118086:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-08-19T14:22:22.769 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:16:40.854128+0000", 2024-08-19T14:22:22.769 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 341.57424878699999, 2024-08-19T14:22:22.769 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.239748353, 2024-08-19T14:22:22.769 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:22:22.770 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:22:22.770 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:22:22.770 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:22:22.770 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:22:22.770 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:22:22.770 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 241367 2024-08-19T14:22:22.770 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.770 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:22:22.770 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.770 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:22:22.770 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.854128+0000", 2024-08-19T14:22:22.770 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:22:22.770 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.771 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.771 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:22:22.771 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.854128+0000", 2024-08-19T14:22:22.771 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:22:22.771 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.771 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.771 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:22:22.771 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.854130+0000", 2024-08-19T14:22:22.771 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6139999999999999e-06 2024-08-19T14:22:22.771 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.771 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.771 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:22:22.771 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.854138+0000", 2024-08-19T14:22:22.771 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.9370000000000004e-06 2024-08-19T14:22:22.772 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.772 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.772 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:22:22.772 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.854151+0000", 2024-08-19T14:22:22.772 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3338e-05 2024-08-19T14:22:22.772 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.772 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.772 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:22:22.772 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.854158+0000", 2024-08-19T14:22:22.772 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.3350000000000001e-06 2024-08-19T14:22:22.772 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.772 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.772 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:22:22.773 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.854208+0000", 2024-08-19T14:22:22.773 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.9373000000000003e-05 2024-08-19T14:22:22.773 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.773 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.773 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:22:22.773 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.854301+0000", 2024-08-19T14:22:22.773 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.2990000000000002e-05 2024-08-19T14:22:22.773 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.773 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.773 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 1,2,8", 2024-08-19T14:22:22.773 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.854388+0000", 2024-08-19T14:22:22.773 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.7321e-05 2024-08-19T14:22:22.773 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.773 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.774 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:22:22.774 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.860101+0000", 2024-08-19T14:22:22.774 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0057132839999999999 2024-08-19T14:22:22.774 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.774 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.774 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:22:22.774 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.860163+0000", 2024-08-19T14:22:22.774 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.1896000000000005e-05 2024-08-19T14:22:22.774 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.774 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.774 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:22:22.774 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.860183+0000", 2024-08-19T14:22:22.774 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.0041e-05 2024-08-19T14:22:22.774 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.775 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.775 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:22:22.775 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.093641+0000", 2024-08-19T14:22:22.775 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.23345814300000001 2024-08-19T14:22:22.775 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.775 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.775 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:22:22.775 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.093706+0000", 2024-08-19T14:22:22.775 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.4453000000000001e-05 2024-08-19T14:22:22.775 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.775 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.775 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:22:22.775 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.093877+0000", 2024-08-19T14:22:22.775 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00017062800000000001 2024-08-19T14:22:22.776 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:22:22.776 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:22:22.776 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:22:22.776 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.776 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.776 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:241375 2.1 2:8e818ec7:::benchmark_data_smithi045_46415_object118094:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-08-19T14:22:22.776 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:16:40.863736+0000", 2024-08-19T14:22:22.776 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 341.56464058199998, 2024-08-19T14:22:22.776 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.23047135799999999, 2024-08-19T14:22:22.776 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:22:22.776 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:22:22.776 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:22:22.776 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:22:22.776 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:22:22.777 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:22:22.777 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 241375 2024-08-19T14:22:22.777 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.777 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:22:22.777 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.777 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:22:22.777 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.863736+0000", 2024-08-19T14:22:22.777 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:22:22.777 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.777 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.777 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:22:22.777 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.863736+0000", 2024-08-19T14:22:22.777 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:22:22.777 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.778 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.778 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:22:22.778 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.863738+0000", 2024-08-19T14:22:22.778 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5820000000000001e-06 2024-08-19T14:22:22.778 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.778 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.778 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:22:22.778 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.863744+0000", 2024-08-19T14:22:22.778 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.0109999999999997e-06 2024-08-19T14:22:22.778 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.778 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.778 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:22:22.778 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.863759+0000", 2024-08-19T14:22:22.778 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5e-05 2024-08-19T14:22:22.779 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.779 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.779 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:22:22.779 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.863766+0000", 2024-08-19T14:22:22.779 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.311e-06 2024-08-19T14:22:22.779 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.779 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.779 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:22:22.779 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.863821+0000", 2024-08-19T14:22:22.779 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.4954999999999998e-05 2024-08-19T14:22:22.779 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.779 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.779 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:22:22.779 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.863953+0000", 2024-08-19T14:22:22.780 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013166600000000001 2024-08-19T14:22:22.780 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.780 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.780 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 1,2,8", 2024-08-19T14:22:22.780 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.864114+0000", 2024-08-19T14:22:22.780 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000161123 2024-08-19T14:22:22.780 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.780 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.780 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:22:22.780 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.868901+0000", 2024-08-19T14:22:22.780 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0047872490000000004 2024-08-19T14:22:22.780 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.780 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.780 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:22:22.781 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.869524+0000", 2024-08-19T14:22:22.781 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00062289799999999998 2024-08-19T14:22:22.781 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.781 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.781 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:22:22.781 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.870384+0000", 2024-08-19T14:22:22.781 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00085993899999999999 2024-08-19T14:22:22.781 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.781 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.781 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:22:22.781 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.093973+0000", 2024-08-19T14:22:22.781 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.22358860999999999 2024-08-19T14:22:22.781 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.781 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.782 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:22:22.782 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.094038+0000", 2024-08-19T14:22:22.782 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.5326000000000004e-05 2024-08-19T14:22:22.782 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.782 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.782 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:22:22.782 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.094208+0000", 2024-08-19T14:22:22.782 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00016968800000000001 2024-08-19T14:22:22.782 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:22:22.782 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:22:22.782 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:22:22.782 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.782 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.782 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25057.0:47774 3.11 3:8e95e964:::benchmark_data_smithi045_59973_object47773:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e234)", 2024-08-19T14:22:22.783 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:17:18.724217+0000", 2024-08-19T14:22:22.783 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 303.70415975600002, 2024-08-19T14:22:22.783 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0342696659999999, 2024-08-19T14:22:22.783 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:22:22.783 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:22:22.783 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:22:22.783 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:22:22.783 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25057", 2024-08-19T14:22:22.783 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:22:22.783 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 47774 2024-08-19T14:22:22.783 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.783 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:22:22.783 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.783 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:22:22.784 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724217+0000", 2024-08-19T14:22:22.784 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:22:22.784 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.784 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.784 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:22:22.784 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724217+0000", 2024-08-19T14:22:22.784 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:22:22.784 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.784 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.784 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:22:22.784 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724218+0000", 2024-08-19T14:22:22.784 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.4499999999999997e-07 2024-08-19T14:22:22.784 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.785 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.785 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:22:22.785 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724254+0000", 2024-08-19T14:22:22.785 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.5914000000000002e-05 2024-08-19T14:22:22.785 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.785 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.785 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:22:22.785 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724263+0000", 2024-08-19T14:22:22.785 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.4290000000000007e-06 2024-08-19T14:22:22.785 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.785 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.785 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:22:22.785 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724270+0000", 2024-08-19T14:22:22.785 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.7959999999999998e-06 2024-08-19T14:22:22.786 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.786 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.786 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:22:22.786 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.774143+0000", 2024-08-19T14:22:22.786 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.049872861999999997 2024-08-19T14:22:22.786 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.786 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.786 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:22:22.786 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.774151+0000", 2024-08-19T14:22:22.786 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.6489999999999997e-06 2024-08-19T14:22:22.786 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.786 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.786 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:22:22.787 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.758417+0000", 2024-08-19T14:22:22.787 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.98426575800000005 2024-08-19T14:22:22.787 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.787 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.787 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:22:22.787 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.758487+0000", 2024-08-19T14:22:22.787 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.9713000000000002e-05 2024-08-19T14:22:22.787 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:22:22.787 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:22:22.787 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:22:22.787 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.787 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.787 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25057.0:47788 3.11 3:8e23edc0:::benchmark_data_smithi045_59973_object47787:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e234)", 2024-08-19T14:22:22.787 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:17:18.724282+0000", 2024-08-19T14:22:22.788 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 303.70409491100003, 2024-08-19T14:22:22.788 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0343071210000001, 2024-08-19T14:22:22.788 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:22:22.788 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:22:22.788 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:22:22.789 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:22:22.789 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25057", 2024-08-19T14:22:22.789 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:22:22.789 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 47788 2024-08-19T14:22:22.789 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.790 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:22:22.790 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.790 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:22:22.790 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724282+0000", 2024-08-19T14:22:22.790 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:22:22.790 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.790 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.790 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:22:22.790 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724282+0000", 2024-08-19T14:22:22.790 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:22:22.790 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.790 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.790 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:22:22.791 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724283+0000", 2024-08-19T14:22:22.791 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.6100000000000001e-07 2024-08-19T14:22:22.791 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.791 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.791 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:22:22.791 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724318+0000", 2024-08-19T14:22:22.791 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.578e-05 2024-08-19T14:22:22.791 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.791 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.791 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:22:22.791 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724326+0000", 2024-08-19T14:22:22.791 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.4170000000000003e-06 2024-08-19T14:22:22.791 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.792 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.792 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:22:22.792 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724331+0000", 2024-08-19T14:22:22.792 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.5820000000000001e-06 2024-08-19T14:22:22.792 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.792 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.792 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:22:22.792 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.774172+0000", 2024-08-19T14:22:22.792 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.049840256999999999 2024-08-19T14:22:22.792 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.792 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.792 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:22:22.792 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.774177+0000", 2024-08-19T14:22:22.793 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.5160000000000002e-06 2024-08-19T14:22:22.793 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.793 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.793 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:22:22.793 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.758533+0000", 2024-08-19T14:22:22.793 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.98435635099999996 2024-08-19T14:22:22.793 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.793 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.793 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:22:22.793 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.758589+0000", 2024-08-19T14:22:22.793 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.5757000000000002e-05 2024-08-19T14:22:22.793 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:22:22.793 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:22:22.794 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:22:22.795 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.795 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.795 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25057.0:47814 3.11 3:8918a6df:::benchmark_data_smithi045_59973_object47813:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e234)", 2024-08-19T14:22:22.795 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:17:18.724406+0000", 2024-08-19T14:22:22.795 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 303.70397147199998, 2024-08-19T14:22:22.795 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0342602160000001, 2024-08-19T14:22:22.795 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:22:22.795 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:22:22.795 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:22:22.795 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:22:22.795 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25057", 2024-08-19T14:22:22.796 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:22:22.796 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 47814 2024-08-19T14:22:22.796 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.796 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:22:22.796 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.796 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:22:22.796 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724406+0000", 2024-08-19T14:22:22.796 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:22:22.796 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.796 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.796 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:22:22.796 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724406+0000", 2024-08-19T14:22:22.796 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:22:22.796 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.797 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.797 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:22:22.797 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724406+0000", 2024-08-19T14:22:22.797 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.0200000000000003e-07 2024-08-19T14:22:22.797 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.797 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.797 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:22:22.797 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724440+0000", 2024-08-19T14:22:22.797 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.3689000000000002e-05 2024-08-19T14:22:22.797 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.797 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.797 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:22:22.798 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724449+0000", 2024-08-19T14:22:22.798 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.0469999999999999e-06 2024-08-19T14:22:22.798 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.798 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.798 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:22:22.798 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724455+0000", 2024-08-19T14:22:22.798 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.1190000000000002e-06 2024-08-19T14:22:22.798 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.798 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.798 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:22:22.798 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.774196+0000", 2024-08-19T14:22:22.798 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.049740924999999998 2024-08-19T14:22:22.798 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.799 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.799 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:22:22.799 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.774201+0000", 2024-08-19T14:22:22.799 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.3090000000000003e-06 2024-08-19T14:22:22.799 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.799 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.799 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:22:22.799 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.758615+0000", 2024-08-19T14:22:22.799 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.98441362200000004 2024-08-19T14:22:22.799 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.799 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.799 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:22:22.799 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.758666+0000", 2024-08-19T14:22:22.800 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1103000000000002e-05 2024-08-19T14:22:22.800 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:22:22.800 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:22:22.800 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:22:22.800 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.800 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.800 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25057.0:47832 3.11 3:8d79f59c:::benchmark_data_smithi045_59973_object47831:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e234)", 2024-08-19T14:22:22.800 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:17:18.724649+0000", 2024-08-19T14:22:22.800 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 303.70372825700002, 2024-08-19T14:22:22.800 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0340940759999999, 2024-08-19T14:22:22.800 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:22:22.800 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:22:22.800 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:22:22.801 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:22:22.801 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25057", 2024-08-19T14:22:22.801 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:22:22.801 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 47832 2024-08-19T14:22:22.801 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.801 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:22:22.801 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.801 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:22:22.801 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724649+0000", 2024-08-19T14:22:22.801 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:22:22.801 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.801 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.801 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:22:22.801 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724649+0000", 2024-08-19T14:22:22.802 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:22:22.802 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.802 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.802 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:22:22.802 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724649+0000", 2024-08-19T14:22:22.802 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0399999999999996e-07 2024-08-19T14:22:22.802 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.802 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.802 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:22:22.802 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724680+0000", 2024-08-19T14:22:22.802 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.0477e-05 2024-08-19T14:22:22.802 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.803 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.803 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:22:22.803 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724688+0000", 2024-08-19T14:22:22.803 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.7509999999999997e-06 2024-08-19T14:22:22.803 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.803 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.803 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:22:22.803 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724694+0000", 2024-08-19T14:22:22.803 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.7679999999999997e-06 2024-08-19T14:22:22.803 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.803 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.803 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:22:22.803 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.774220+0000", 2024-08-19T14:22:22.803 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.049526175999999998 2024-08-19T14:22:22.803 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.804 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.804 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:22:22.804 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.774226+0000", 2024-08-19T14:22:22.804 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.2820000000000004e-06 2024-08-19T14:22:22.804 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.804 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.804 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:22:22.804 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.758693+0000", 2024-08-19T14:22:22.804 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.98446715699999998 2024-08-19T14:22:22.804 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.804 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.804 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:22:22.804 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.758743+0000", 2024-08-19T14:22:22.804 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.9960999999999998e-05 2024-08-19T14:22:22.805 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:22:22.805 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:22:22.805 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:22:22.805 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.805 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.805 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25057.0:47838 3.16 3:68f5225c:::benchmark_data_smithi045_59973_object47837:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e234)", 2024-08-19T14:22:22.805 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:17:18.777436+0000", 2024-08-19T14:22:22.805 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 303.65094068799999, 2024-08-19T14:22:22.805 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.98807081600000002, 2024-08-19T14:22:22.805 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:22:22.805 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:22:22.805 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:22:22.805 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:22:22.805 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25057", 2024-08-19T14:22:22.805 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:22:22.806 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 47838 2024-08-19T14:22:22.806 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.806 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:22:22.806 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.806 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:22:22.806 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.777436+0000", 2024-08-19T14:22:22.806 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:22:22.806 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.806 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.806 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:22:22.806 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.777436+0000", 2024-08-19T14:22:22.806 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:22:22.807 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.807 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.807 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:22:22.807 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.777438+0000", 2024-08-19T14:22:22.807 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.7489999999999999e-06 2024-08-19T14:22:22.807 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.807 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.807 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:22:22.807 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.777473+0000", 2024-08-19T14:22:22.807 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.5284999999999998e-05 2024-08-19T14:22:22.807 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.807 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.808 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:22:22.808 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.777492+0000", 2024-08-19T14:22:22.808 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.8853999999999999e-05 2024-08-19T14:22:22.808 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.808 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.808 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:22:22.808 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.777500+0000", 2024-08-19T14:22:22.808 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.9759999999999995e-06 2024-08-19T14:22:22.808 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.808 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.808 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:22:22.808 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.778853+0000", 2024-08-19T14:22:22.808 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0013526460000000001 2024-08-19T14:22:22.808 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.809 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.809 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:22:22.809 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.778867+0000", 2024-08-19T14:22:22.809 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4448e-05 2024-08-19T14:22:22.809 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.809 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.809 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:22:22.809 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.759774+0000", 2024-08-19T14:22:22.809 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.98090639700000004 2024-08-19T14:22:22.809 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.809 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.809 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:22:22.809 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.759902+0000", 2024-08-19T14:22:22.810 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000128128 2024-08-19T14:22:22.810 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.810 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.810 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 1,4", 2024-08-19T14:22:22.810 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.760062+0000", 2024-08-19T14:22:22.810 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000159956 2024-08-19T14:22:22.810 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.810 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.810 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:22:22.810 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.762325+0000", 2024-08-19T14:22:22.810 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0022629880000000001 2024-08-19T14:22:22.810 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.810 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.811 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:22:22.811 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.763087+0000", 2024-08-19T14:22:22.811 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00076208899999999997 2024-08-19T14:22:22.811 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.811 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.811 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:22:22.811 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.765400+0000", 2024-08-19T14:22:22.811 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0023133120000000001 2024-08-19T14:22:22.811 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.811 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.811 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:22:22.811 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.765462+0000", 2024-08-19T14:22:22.811 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.2048e-05 2024-08-19T14:22:22.811 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.811 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.812 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:22:22.812 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.765507+0000", 2024-08-19T14:22:22.812 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.494e-05 2024-08-19T14:22:22.812 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:22:22.812 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:22:22.812 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:22:22.812 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.812 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.812 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25057.0:47845 3.16 3:6c285014:::benchmark_data_smithi045_59973_object47844:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e234)", 2024-08-19T14:22:22.812 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:17:18.799489+0000", 2024-08-19T14:22:22.812 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 303.62888838399999, 2024-08-19T14:22:22.812 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.96674353099999999, 2024-08-19T14:22:22.812 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:22:22.812 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:22:22.813 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:22:22.813 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:22:22.813 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25057", 2024-08-19T14:22:22.813 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:22:22.813 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 47845 2024-08-19T14:22:22.813 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.813 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:22:22.813 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.813 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:22:22.813 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.799489+0000", 2024-08-19T14:22:22.813 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:22:22.813 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.814 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.814 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:22:22.814 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.799489+0000", 2024-08-19T14:22:22.814 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:22:22.814 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.814 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.814 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:22:22.814 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.799490+0000", 2024-08-19T14:22:22.814 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.779e-06 2024-08-19T14:22:22.814 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.814 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.814 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:22:22.814 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.799530+0000", 2024-08-19T14:22:22.815 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.9555000000000003e-05 2024-08-19T14:22:22.815 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.815 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.815 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:22:22.815 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.799544+0000", 2024-08-19T14:22:22.815 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4336e-05 2024-08-19T14:22:22.815 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.815 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.815 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:22:22.815 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.799554+0000", 2024-08-19T14:22:22.815 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.5580000000000006e-06 2024-08-19T14:22:22.815 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.815 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.816 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:22:22.816 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.799613+0000", 2024-08-19T14:22:22.816 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.8995e-05 2024-08-19T14:22:22.816 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.816 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.816 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:22:22.816 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.799625+0000", 2024-08-19T14:22:22.816 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2088e-05 2024-08-19T14:22:22.816 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.816 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.816 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:22:22.816 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.760956+0000", 2024-08-19T14:22:22.816 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.96133111299999996 2024-08-19T14:22:22.816 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.817 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.817 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:22:22.817 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.761072+0000", 2024-08-19T14:22:22.817 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000116173 2024-08-19T14:22:22.817 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.817 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.817 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 1,4", 2024-08-19T14:22:22.817 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.761217+0000", 2024-08-19T14:22:22.817 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000144308 2024-08-19T14:22:22.817 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.817 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.817 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:22:22.817 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.763046+0000", 2024-08-19T14:22:22.817 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00182978 2024-08-19T14:22:22.818 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.818 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.818 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:22:22.818 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.763278+0000", 2024-08-19T14:22:22.818 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00023194700000000001 2024-08-19T14:22:22.818 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.818 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.818 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:22:22.818 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.766122+0000", 2024-08-19T14:22:22.818 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0028441439999999998 2024-08-19T14:22:22.818 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.818 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.818 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:22:22.819 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.766190+0000", 2024-08-19T14:22:22.819 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.7595999999999995e-05 2024-08-19T14:22:22.819 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.819 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.819 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:22:22.819 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.766232+0000", 2024-08-19T14:22:22.819 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2159000000000001e-05 2024-08-19T14:22:22.819 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:22:22.819 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:22:22.819 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:22:22.819 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.819 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.819 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25057.0:64601 3.1f 3.eafa94bf (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e246)", 2024-08-19T14:22:22.819 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:17:35.967173+0000", 2024-08-19T14:22:22.820 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 286.46120438200001, 2024-08-19T14:22:22.820 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.99490361100000002, 2024-08-19T14:22:22.820 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:22:22.820 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:22:22.820 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:22:22.820 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:22:22.820 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25057", 2024-08-19T14:22:22.820 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:22:22.820 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 64601 2024-08-19T14:22:22.820 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.820 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:22:22.820 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.820 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:22:22.821 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:35.967173+0000", 2024-08-19T14:22:22.821 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:22:22.821 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.821 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.821 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:22:22.821 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:35.967173+0000", 2024-08-19T14:22:22.821 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:22:22.821 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.821 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.821 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:22:22.821 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:35.967174+0000", 2024-08-19T14:22:22.821 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.7740000000000001e-06 2024-08-19T14:22:22.821 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.822 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.822 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:22:22.822 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:35.967205+0000", 2024-08-19T14:22:22.822 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.0550999999999999e-05 2024-08-19T14:22:22.822 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.822 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.822 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:22:22.822 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:35.967214+0000", 2024-08-19T14:22:22.822 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.1430000000000004e-06 2024-08-19T14:22:22.822 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.822 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.822 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:22:22.822 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:35.967221+0000", 2024-08-19T14:22:22.823 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.0199999999999997e-06 2024-08-19T14:22:22.823 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.823 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.823 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:22:22.823 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:35.967334+0000", 2024-08-19T14:22:22.823 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000112799 2024-08-19T14:22:22.823 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.823 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.823 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:22:22.823 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:35.967341+0000", 2024-08-19T14:22:22.823 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.3170000000000004e-06 2024-08-19T14:22:22.823 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.823 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.824 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:22:22.824 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:36.962066+0000", 2024-08-19T14:22:22.824 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.99472444500000001 2024-08-19T14:22:22.824 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.824 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.824 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:22:22.824 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:36.962076+0000", 2024-08-19T14:22:22.824 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0562e-05 2024-08-19T14:22:22.824 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:22:22.824 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:22:22.824 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:22:22.824 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.824 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.825 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25057.0:64632 3.1f 3.fd125fbf (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e246)", 2024-08-19T14:22:22.825 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:17:36.181164+0000", 2024-08-19T14:22:22.825 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 286.24721258199997, 2024-08-19T14:22:22.825 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.78094885599999997, 2024-08-19T14:22:22.825 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:22:22.825 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:22:22.825 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:22:22.825 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:22:22.825 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25057", 2024-08-19T14:22:22.825 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:22:22.825 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 64632 2024-08-19T14:22:22.825 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.825 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:22:22.825 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.826 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:22:22.826 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:36.181164+0000", 2024-08-19T14:22:22.826 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:22:22.826 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.826 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.826 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:22:22.826 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:36.181164+0000", 2024-08-19T14:22:22.826 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:22:22.826 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.826 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.826 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:22:22.826 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:36.181166+0000", 2024-08-19T14:22:22.826 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.001e-06 2024-08-19T14:22:22.827 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.827 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.827 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:22:22.827 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:36.181210+0000", 2024-08-19T14:22:22.827 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4035000000000001e-05 2024-08-19T14:22:22.827 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.827 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.827 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:22:22.827 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:36.181225+0000", 2024-08-19T14:22:22.827 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4620000000000001e-05 2024-08-19T14:22:22.827 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.827 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.827 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:22:22.827 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:36.181235+0000", 2024-08-19T14:22:22.827 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.6709999999999994e-06 2024-08-19T14:22:22.828 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.828 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.828 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:22:22.828 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:36.181309+0000", 2024-08-19T14:22:22.828 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.4191999999999997e-05 2024-08-19T14:22:22.828 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.828 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.828 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:22:22.828 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:36.181321+0000", 2024-08-19T14:22:22.828 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2026e-05 2024-08-19T14:22:22.828 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.828 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.828 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:22:22.828 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:36.962105+0000", 2024-08-19T14:22:22.828 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.780784382 2024-08-19T14:22:22.829 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.829 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.829 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:22:22.829 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:36.962113+0000", 2024-08-19T14:22:22.829 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.9289999999999998e-06 2024-08-19T14:22:22.829 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:22:22.829 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:22:22.829 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:22:22.829 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.829 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.829 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25057.0:188741 3.26 3:65f5d80b:::benchmark_data_smithi045_59973_object57562:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e296)", 2024-08-19T14:22:22.829 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:19:02.154496+0000", 2024-08-19T14:22:22.829 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 200.27388058299999, 2024-08-19T14:22:22.829 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.047062189, 2024-08-19T14:22:22.830 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:22:22.830 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:22:22.830 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:22:22.830 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:22:22.830 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25057", 2024-08-19T14:22:22.830 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:22:22.830 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 188741 2024-08-19T14:22:22.830 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.830 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:22:22.830 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.830 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:22:22.830 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.154496+0000", 2024-08-19T14:22:22.830 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:22:22.831 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.831 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.831 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:22:22.831 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.154496+0000", 2024-08-19T14:22:22.831 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:22:22.831 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.831 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.831 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:22:22.831 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.154497+0000", 2024-08-19T14:22:22.831 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.5000000000000001e-07 2024-08-19T14:22:22.831 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.831 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.831 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:22:22.831 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.154502+0000", 2024-08-19T14:22:22.832 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0690000000000001e-06 2024-08-19T14:22:22.832 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.832 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.832 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:22:22.832 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.154514+0000", 2024-08-19T14:22:22.832 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1922e-05 2024-08-19T14:22:22.832 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.832 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.832 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:22:22.832 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.154522+0000", 2024-08-19T14:22:22.832 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.4730000000000003e-06 2024-08-19T14:22:22.832 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.832 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.832 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:22:22.833 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.275482+0000", 2024-08-19T14:22:22.833 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.12095984899999999 2024-08-19T14:22:22.833 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.833 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.833 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:22:22.833 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.275489+0000", 2024-08-19T14:22:22.833 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.8949999999999998e-06 2024-08-19T14:22:22.833 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.833 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.833 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:22:22.833 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:03.201475+0000", 2024-08-19T14:22:22.833 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.92598516200000003 2024-08-19T14:22:22.833 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.833 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.833 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:22:22.834 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:03.201559+0000", 2024-08-19T14:22:22.834 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.3968999999999995e-05 2024-08-19T14:22:22.834 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:22:22.834 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:22:22.834 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:22:22.834 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.834 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.834 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25057.0:188765 3.26 3:6524f2fe:::benchmark_data_smithi045_59973_object57586:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-08-19T14:22:22.834 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:19:02.192404+0000", 2024-08-19T14:22:22.834 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 200.235973484, 2024-08-19T14:22:22.834 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0202571499999999, 2024-08-19T14:22:22.834 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:22:22.835 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:22:22.835 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:22:22.835 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:22:22.835 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25057", 2024-08-19T14:22:22.835 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:22:22.835 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 188765 2024-08-19T14:22:22.835 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.835 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:22:22.835 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.835 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:22:22.835 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.192404+0000", 2024-08-19T14:22:22.835 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:22:22.835 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.835 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.836 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:22:22.836 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.192404+0000", 2024-08-19T14:22:22.836 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:22:22.836 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.836 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.836 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:22:22.836 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.192406+0000", 2024-08-19T14:22:22.836 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.3140000000000002e-06 2024-08-19T14:22:22.836 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.836 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.836 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:22:22.836 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.192417+0000", 2024-08-19T14:22:22.836 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0839000000000001e-05 2024-08-19T14:22:22.836 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.837 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.837 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:22:22.837 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.192440+0000", 2024-08-19T14:22:22.837 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.2911000000000001e-05 2024-08-19T14:22:22.837 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.837 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.837 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:22:22.837 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.192450+0000", 2024-08-19T14:22:22.837 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0292000000000001e-05 2024-08-19T14:22:22.837 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.837 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.837 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:22:22.837 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.275507+0000", 2024-08-19T14:22:22.838 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.083056873000000003 2024-08-19T14:22:22.838 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.838 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.838 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:22:22.838 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.275512+0000", 2024-08-19T14:22:22.838 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.8799999999999999e-06 2024-08-19T14:22:22.838 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.838 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.838 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:22:22.838 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:03.201920+0000", 2024-08-19T14:22:22.838 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.92640795799999998 2024-08-19T14:22:22.838 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.838 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.838 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:22:22.839 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:03.202036+0000", 2024-08-19T14:22:22.839 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000116364 2024-08-19T14:22:22.839 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.839 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.839 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 1,4", 2024-08-19T14:22:22.839 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:03.202142+0000", 2024-08-19T14:22:22.839 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000106115 2024-08-19T14:22:22.839 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.839 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.839 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:22:22.839 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:03.206756+0000", 2024-08-19T14:22:22.839 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0046137319999999997 2024-08-19T14:22:22.839 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.840 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.840 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:22:22.840 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:03.206778+0000", 2024-08-19T14:22:22.840 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.2056999999999999e-05 2024-08-19T14:22:22.840 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.840 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.840 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:22:22.840 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:03.212547+0000", 2024-08-19T14:22:22.840 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0057695740000000004 2024-08-19T14:22:22.840 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.840 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.840 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:22:22.840 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:03.212603+0000", 2024-08-19T14:22:22.841 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.5189999999999998e-05 2024-08-19T14:22:22.841 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.841 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.841 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:22:22.841 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:03.212661+0000", 2024-08-19T14:22:22.841 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.8050999999999997e-05 2024-08-19T14:22:22.841 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:22:22.841 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:22:22.841 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:22:22.841 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.841 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.841 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25333.0:22685 4.b 4:d7f4804b:::benchmark_data_smithi045_71562_object22684:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-08-19T14:22:22.841 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:20:41.347757+0000", 2024-08-19T14:22:22.842 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 101.08061973700001, 2024-08-19T14:22:22.842 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.33048444500000002, 2024-08-19T14:22:22.842 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:22:22.842 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:22:22.842 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:22:22.842 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:22:22.842 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25333", 2024-08-19T14:22:22.842 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:22:22.842 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 22685 2024-08-19T14:22:22.842 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.842 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:22:22.842 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.842 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:22:22.843 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.347757+0000", 2024-08-19T14:22:22.843 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:22:22.843 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.843 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.843 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:22:22.843 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.347757+0000", 2024-08-19T14:22:22.843 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:22:22.843 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.843 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.843 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:22:22.843 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.347759+0000", 2024-08-19T14:22:22.843 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4410000000000001e-06 2024-08-19T14:22:22.843 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.844 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.844 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:22:22.844 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.347940+0000", 2024-08-19T14:22:22.844 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00018171800000000001 2024-08-19T14:22:22.844 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.844 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.844 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:22:22.844 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.347953+0000", 2024-08-19T14:22:22.844 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2724e-05 2024-08-19T14:22:22.844 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.844 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.844 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:22:22.844 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.347960+0000", 2024-08-19T14:22:22.844 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.3320000000000003e-06 2024-08-19T14:22:22.845 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.845 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.845 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:22:22.845 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.348009+0000", 2024-08-19T14:22:22.845 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.8882999999999999e-05 2024-08-19T14:22:22.845 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.845 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.845 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:22:22.845 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.348129+0000", 2024-08-19T14:22:22.845 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012009 2024-08-19T14:22:22.845 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.845 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.845 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 1,4", 2024-08-19T14:22:22.846 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.348257+0000", 2024-08-19T14:22:22.846 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012718399999999999 2024-08-19T14:22:22.846 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.846 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.846 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:22:22.846 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.349748+0000", 2024-08-19T14:22:22.846 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.001491579 2024-08-19T14:22:22.846 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.846 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.846 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:22:22.846 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.350166+0000", 2024-08-19T14:22:22.846 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00041812000000000002 2024-08-19T14:22:22.846 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.847 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.847 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:22:22.847 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.351389+0000", 2024-08-19T14:22:22.847 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0012227119999999999 2024-08-19T14:22:22.847 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.847 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.847 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:22:22.847 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.678185+0000", 2024-08-19T14:22:22.847 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.32679576199999999 2024-08-19T14:22:22.847 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.847 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.847 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:22:22.847 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.678242+0000", 2024-08-19T14:22:22.848 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.6900000000000001e-05 2024-08-19T14:22:22.848 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:22:22.848 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:22:22.848 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:22:22.848 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.848 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.848 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop_reply(client.25333.0:22685 4.b e341/335 ondisk, result = 0)", 2024-08-19T14:22:22.848 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:20:41.350582+0000", 2024-08-19T14:22:22.848 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 101.077795319, 2024-08-19T14:22:22.848 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.32767402200000001, 2024-08-19T14:22:22.848 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:22:22.848 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:22:22.848 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-08-19T14:22:22.848 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:22:22.848 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.849 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:22:22.849 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.350582+0000", 2024-08-19T14:22:22.849 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:22:22.849 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.849 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.849 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:22:22.849 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.350582+0000", 2024-08-19T14:22:22.849 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:22:22.849 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.849 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.849 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:22:22.849 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.350582+0000", 2024-08-19T14:22:22.849 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.1999999999999999e-07 2024-08-19T14:22:22.849 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.850 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.850 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:22:22.850 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.350587+0000", 2024-08-19T14:22:22.850 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.6870000000000002e-06 2024-08-19T14:22:22.850 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.850 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.850 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:22:22.850 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.350598+0000", 2024-08-19T14:22:22.850 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0781999999999999e-05 2024-08-19T14:22:22.850 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.850 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.850 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:22:22.850 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.350605+0000", 2024-08-19T14:22:22.851 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.0419999999999997e-06 2024-08-19T14:22:22.851 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.851 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.851 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:22:22.851 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.351377+0000", 2024-08-19T14:22:22.851 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00077201899999999998 2024-08-19T14:22:22.851 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.851 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.851 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:22:22.851 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.351384+0000", 2024-08-19T14:22:22.851 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.917e-06 2024-08-19T14:22:22.851 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.851 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.852 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:22:22.852 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.678256+0000", 2024-08-19T14:22:22.852 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.32687185499999999 2024-08-19T14:22:22.852 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:22:22.852 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:22:22.852 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:22:22.852 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.852 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.852 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.25333.0:22687 4.7 e341/335 4:efd17a0f:::benchmark_data_smithi045_71562_object22686:head v 341'1401, mlcod=341'1397)", 2024-08-19T14:22:22.852 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:20:41.351262+0000", 2024-08-19T14:22:22.852 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 101.077115208, 2024-08-19T14:22:22.852 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.32996297600000002, 2024-08-19T14:22:22.852 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:22:22.852 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:22:22.853 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:22:22.853 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:22:22.853 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.853 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:22:22.853 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.351262+0000", 2024-08-19T14:22:22.853 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:22:22.853 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.853 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.853 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:22:22.853 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.351262+0000", 2024-08-19T14:22:22.853 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:22:22.853 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.853 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.854 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:22:22.854 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.351262+0000", 2024-08-19T14:22:22.854 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.0900000000000002e-07 2024-08-19T14:22:22.854 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.854 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.854 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:22:22.854 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.351289+0000", 2024-08-19T14:22:22.854 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.7036999999999999e-05 2024-08-19T14:22:22.854 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.854 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.854 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:22:22.854 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.351299+0000", 2024-08-19T14:22:22.854 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.9299999999999998e-06 2024-08-19T14:22:22.854 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.855 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.855 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:22:22.855 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.351305+0000", 2024-08-19T14:22:22.855 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.9229999999999999e-06 2024-08-19T14:22:22.855 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.855 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.855 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:22:22.855 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.351361+0000", 2024-08-19T14:22:22.855 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.6311999999999999e-05 2024-08-19T14:22:22.855 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.855 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.855 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:22:22.855 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.351380+0000", 2024-08-19T14:22:22.855 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.861e-05 2024-08-19T14:22:22.856 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.856 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.856 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:22:22.856 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.681181+0000", 2024-08-19T14:22:22.856 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.32980058099999998 2024-08-19T14:22:22.856 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.856 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.856 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:22:22.856 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.681225+0000", 2024-08-19T14:22:22.856 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4174000000000001e-05 2024-08-19T14:22:22.856 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:22:22.856 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:22:22.856 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:22:22.856 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.857 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.857 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.25333.0:22691 4.7 e341/335 4:e60184b4:::benchmark_data_smithi045_71562_object22690:head v 341'1402, mlcod=341'1397)", 2024-08-19T14:22:22.857 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:20:41.352142+0000", 2024-08-19T14:22:22.857 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 101.07623499100001, 2024-08-19T14:22:22.857 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.32972854099999999, 2024-08-19T14:22:22.857 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:22:22.857 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:22:22.857 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:22:22.857 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:22:22.857 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.857 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:22:22.857 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.352142+0000", 2024-08-19T14:22:22.857 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:22:22.857 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.858 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.858 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:22:22.858 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.352142+0000", 2024-08-19T14:22:22.858 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:22:22.858 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.858 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.858 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:22:22.858 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.352143+0000", 2024-08-19T14:22:22.858 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.5199999999999996e-07 2024-08-19T14:22:22.858 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.858 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.858 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:22:22.858 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.352196+0000", 2024-08-19T14:22:22.859 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.3468999999999999e-05 2024-08-19T14:22:22.859 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.859 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.859 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:22:22.859 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.678332+0000", 2024-08-19T14:22:22.859 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.32613542000000001 2024-08-19T14:22:22.859 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.859 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.859 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:22:22.859 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.678342+0000", 2024-08-19T14:22:22.859 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0696999999999999e-05 2024-08-19T14:22:22.859 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.859 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.859 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:22:22.860 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.679002+0000", 2024-08-19T14:22:22.860 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000659945 2024-08-19T14:22:22.860 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.860 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.860 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:22:22.860 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.679024+0000", 2024-08-19T14:22:22.860 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.1693999999999999e-05 2024-08-19T14:22:22.860 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.860 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.860 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:22:22.860 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.681816+0000", 2024-08-19T14:22:22.860 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0027922590000000001 2024-08-19T14:22:22.860 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.860 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.861 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:22:22.861 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.681871+0000", 2024-08-19T14:22:22.861 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.4305000000000003e-05 2024-08-19T14:22:22.861 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:22:22.861 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:22:22.861 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:22:22.861 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.861 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.862 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25333.0:22700 4.b 4:d65b8e70:::benchmark_data_smithi045_71562_object22699:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-08-19T14:22:22.862 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:20:41.353478+0000", 2024-08-19T14:22:22.862 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 101.07489877099999, 2024-08-19T14:22:22.862 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.32867489300000002, 2024-08-19T14:22:22.862 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:22:22.862 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:22:22.862 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:22:22.862 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:22:22.862 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25333", 2024-08-19T14:22:22.862 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:22:22.862 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 22700 2024-08-19T14:22:22.862 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.862 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:22:22.863 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.863 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:22:22.863 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.353478+0000", 2024-08-19T14:22:22.863 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:22:22.863 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.863 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.863 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:22:22.863 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.353478+0000", 2024-08-19T14:22:22.863 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:22:22.863 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.864 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.864 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:22:22.864 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.353480+0000", 2024-08-19T14:22:22.864 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6980000000000001e-06 2024-08-19T14:22:22.864 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.864 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.864 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:22:22.864 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.353537+0000", 2024-08-19T14:22:22.864 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.6724000000000001e-05 2024-08-19T14:22:22.864 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.864 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.864 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:22:22.864 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.677867+0000", 2024-08-19T14:22:22.864 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.32433008200000002 2024-08-19T14:22:22.865 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.865 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.865 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:22:22.865 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.677885+0000", 2024-08-19T14:22:22.865 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.8405000000000001e-05 2024-08-19T14:22:22.865 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.865 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.865 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:22:22.865 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.678314+0000", 2024-08-19T14:22:22.865 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00042872500000000003 2024-08-19T14:22:22.865 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.865 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.865 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:22:22.865 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.678522+0000", 2024-08-19T14:22:22.866 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00020833599999999999 2024-08-19T14:22:22.866 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.866 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.866 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 1,4", 2024-08-19T14:22:22.866 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.678691+0000", 2024-08-19T14:22:22.866 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00016871200000000001 2024-08-19T14:22:22.866 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.866 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.866 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:22:22.866 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.681852+0000", 2024-08-19T14:22:22.866 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0031611849999999999 2024-08-19T14:22:22.866 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.866 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.867 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:22:22.867 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.681921+0000", 2024-08-19T14:22:22.867 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.9191999999999998e-05 2024-08-19T14:22:22.867 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.867 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.867 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:22:22.867 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.682059+0000", 2024-08-19T14:22:22.867 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013730400000000001 2024-08-19T14:22:22.867 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.867 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.867 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:22:22.867 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.682113+0000", 2024-08-19T14:22:22.867 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.4840999999999999e-05 2024-08-19T14:22:22.867 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:22.868 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:22.868 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:22:22.868 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.682153+0000", 2024-08-19T14:22:22.868 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.9688999999999999e-05 2024-08-19T14:22:22.868 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:22:22.868 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:22:22.868 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:22:22.868 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:22:22.868 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:22:22.868 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:22:22.868 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:22:22.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:22.904 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:22.904 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:22:23.005 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:22:23.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:23.038 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:23.039 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:22:23.131 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is set 2024-08-19T14:22:23.139 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:22:23.165 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:23.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:23.166 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:22:23.267 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:22:23.292 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:23.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:23.293 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:22:23.394 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:22:23.426 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:23.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:23.427 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:22:23.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:23 smithi032 ceph-mon[20004]: pgmap v851: 47 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 45 active+clean; 7.2 GiB data, 25 GiB used, 779 GiB / 805 GiB avail; 0 B/s wr, 1.05k op/s; 1990/352017 objects misplaced (0.565%); 2.8 MiB/s, 44 objects/s recovering 2024-08-19T14:22:23.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:23 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/348430429' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:22:23.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:23 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:22:23.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:23 smithi032 ceph-mon[20004]: from='client.25577 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:22:23.528 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:22:23.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:23 smithi045 ceph-mon[20143]: pgmap v851: 47 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 45 active+clean; 7.2 GiB data, 25 GiB used, 779 GiB / 805 GiB avail; 0 B/s wr, 1.05k op/s; 1990/352017 objects misplaced (0.565%); 2.8 MiB/s, 44 objects/s recovering 2024-08-19T14:22:23.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:23 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/348430429' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:22:23.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:23 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:22:23.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:23 smithi045 ceph-mon[20143]: from='client.25577 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:22:23.553 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:23.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:23.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:22:23.574 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:23 smithi031 ceph-mon[16336]: pgmap v851: 47 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 45 active+clean; 7.2 GiB data, 25 GiB used, 779 GiB / 805 GiB avail; 0 B/s wr, 1.05k op/s; 1990/352017 objects misplaced (0.565%); 2.8 MiB/s, 44 objects/s recovering 2024-08-19T14:22:23.574 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:23 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/348430429' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:22:23.575 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:23 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:22:23.575 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:23 smithi031 ceph-mon[16336]: from='client.25577 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:22:23.655 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:22:23.683 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:23.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:23.684 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:22:23.763 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:22:23.765 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-08-19T14:22:23.785 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:22:23.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:23.811 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:23.812 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:22:23.913 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:22:23.948 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:23.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:23.949 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:22:24.050 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:22:24.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:24.074 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:24.074 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:22:24.174 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:22:24.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:24.223 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:24.223 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:22:24.324 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:22:24.350 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:24.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:24.351 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:22:24.452 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:22:24.475 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:24.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:24.477 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:22:24.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:24 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:22:24.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:24 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:22:24.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:24 smithi032 ceph-mon[20004]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:22:24.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:24 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:22:24.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:24 smithi032 ceph-mon[20004]: osdmap e378: 12 total, 12 up, 9 in 2024-08-19T14:22:24.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:24 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:22:24.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:24 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:22:24.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:24 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:22:24.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:24 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:22:24.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:24 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:22:24.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:24 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:22:24.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:24 smithi045 ceph-mon[20143]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:22:24.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:24 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:22:24.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:24 smithi045 ceph-mon[20143]: osdmap e378: 12 total, 12 up, 9 in 2024-08-19T14:22:24.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:24 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:22:24.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:24 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:22:24.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:24 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:22:24.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:24 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:22:24.577 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:22:24.602 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:24.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:24.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:22:24.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:24 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:22:24.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:24 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:22:24.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:24 smithi031 ceph-mon[16336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:22:24.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:24 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:22:24.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:24 smithi031 ceph-mon[16336]: osdmap e378: 12 total, 12 up, 9 in 2024-08-19T14:22:24.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:24 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:22:24.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:24 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:22:24.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:24 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:22:24.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:24 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:22:24.704 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:22:24.732 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:24.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:24.733 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:22:24.834 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:22:24.861 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:24.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:24.862 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:22:24.963 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:22:25.002 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:25.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:25.003 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:22:25.104 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:22:25.131 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:25.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:25.132 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:22:25.233 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:22:25.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:25.272 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:25.272 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:22:25.373 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:22:25.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:25.400 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:25.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:22:25.501 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:22:25.532 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:25.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:25.533 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:22:25.634 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:22:25.674 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:25.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:25.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:22:25.776 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:22:25.819 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:25.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:25.820 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:22:25.921 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:22:25.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:25.959 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:25.959 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:22:26.060 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:22:26.084 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:26.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:26.085 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:22:26.186 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:22:26.213 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:26.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:26.214 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:22:26.315 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:22:26.345 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:26.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:26.346 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:22:26.447 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:22:26.482 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:26.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:26.483 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:22:26.584 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:22:26.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:26.610 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:26.610 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:22:26.675 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":851,"stamp":"2024-08-19T14:22:21.910915+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7690322488,"num_objects":117339,"num_object_clones":0,"num_object_copies":352017,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1990,"num_objects_unfound":0,"num_objects_dirty":117339,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":145009,"num_write_kb":8395009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2150,"num_bytes_recovered":140902400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149318,"ondisk_log_size":149318,"up":141,"acting":140,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":142,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":26693236,"kb_used_data":22779484,"kb_used_omap":14,"kb_used_meta":3913649,"kb_avail":817013132,"statfs":{"total":863955320832,"available":836621447168,"internally_reserved":0,"allocated":23326191616,"data_stored":23310563122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14353,"internal_metadata":4007577583},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-541458432,"num_objects":-8262,"num_object_clones":0,"num_object_copies":-24786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-349,"num_objects_unfound":0,"num_objects_dirty":-8262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8262,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":349,"num_bytes_recovered":22872064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.844678"},"pg_stats":[{"pgid":"4.5","version":"377'9163","reported_seq":9342,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-08-19T14:22:20.861003+0000","last_change":"2024-08-19T14:22:14.211513+0000","last_active":"2024-08-19T14:22:20.861003+0000","last_peered":"2024-08-19T14:22:20.861003+0000","last_clean":"2024-08-19T14:22:20.861003+0000","last_became_active":"2024-08-19T14:20:29.267549+0000","last_became_peered":"2024-08-19T14:20:29.267549+0000","last_unstale":"2024-08-19T14:22:20.861003+0000","last_undegraded":"2024-08-19T14:22:20.861003+0000","last_fullsized":"2024-08-19T14:22:20.861003+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"375'8747","last_scrub_stamp":"2024-08-19T14:22:14.211361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:22:14.211361+0000","objects_scrubbed":7717,"log_size":9163,"log_dups_size":0,"ondisk_log_size":9163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:26.018227+0000","scrub_duration":1909,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":474284032,"num_objects":7237,"num_object_clones":0,"num_object_copies":21711,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7237,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9163,"num_write_kb":524800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,0],"acting":[11,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"70'139","reported_seq":968,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-08-19T14:22:17.102470+0000","last_change":"2024-08-19T14:19:58.072227+0000","last_active":"2024-08-19T14:22:17.102470+0000","last_peered":"2024-08-19T14:22:17.102470+0000","last_clean":"2024-08-19T14:22:17.102470+0000","last_became_active":"2024-08-19T14:18:13.807133+0000","last_became_peered":"2024-08-19T14:18:13.807133+0000","last_unstale":"2024-08-19T14:22:17.102470+0000","last_undegraded":"2024-08-19T14:22:17.102470+0000","last_fullsized":"2024-08-19T14:22:17.102470+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:19:58.072083+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:19:58.072083+0000","last_clean_scrub_stamp":"2024-08-19T14:19:58.072083+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,9],"acting":[7,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"376'9238","reported_seq":13633,"reported_epoch":376,"state":"active+recovering+undersized+remapped","last_fresh":"2024-08-19T14:22:19.322785+0000","last_change":"2024-08-19T14:21:07.239135+0000","last_active":"2024-08-19T14:22:19.322785+0000","last_peered":"2024-08-19T14:22:19.322785+0000","last_clean":"2024-08-19T14:21:04.784061+0000","last_became_active":"2024-08-19T14:21:07.213050+0000","last_became_peered":"2024-08-19T14:21:07.213050+0000","last_unstale":"2024-08-19T14:22:19.322785+0000","last_undegraded":"2024-08-19T14:22:19.322785+0000","last_fullsized":"2024-08-19T14:21:06.805412+0000","mapping_epoch":354,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":9238,"log_dups_size":0,"ondisk_log_size":9238,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":487587840,"num_objects":7440,"num_object_clones":0,"num_object_copies":22320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1990,"num_objects_unfound":0,"num_objects_dirty":7440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9238,"num_write_kb":533696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2150,"num_bytes_recovered":140902400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,7],"acting":[1,7],"avail_no_missing":["1","5","7"],"object_location_counts":[{"shards":"1,5,7","objects":1990},{"shards":"1,7","objects":5450}],"blocked_by":[],"up_primary":9,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"70'139","reported_seq":867,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-08-19T14:22:17.102449+0000","last_change":"2024-08-19T14:20:03.714917+0000","last_active":"2024-08-19T14:22:17.102449+0000","last_peered":"2024-08-19T14:22:17.102449+0000","last_clean":"2024-08-19T14:22:17.102449+0000","last_became_active":"2024-08-19T14:18:13.804721+0000","last_became_peered":"2024-08-19T14:18:13.804721+0000","last_unstale":"2024-08-19T14:22:17.102449+0000","last_undegraded":"2024-08-19T14:22:17.102449+0000","last_fullsized":"2024-08-19T14:22:17.102449+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:03.714818+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:03.714818+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"376'9045","reported_seq":9137,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-08-19T14:22:19.327148+0000","last_change":"2024-08-19T14:20:29.268110+0000","last_active":"2024-08-19T14:22:19.327148+0000","last_peered":"2024-08-19T14:22:19.327148+0000","last_clean":"2024-08-19T14:22:19.327148+0000","last_became_active":"2024-08-19T14:20:29.267811+0000","last_became_peered":"2024-08-19T14:20:29.267811+0000","last_unstale":"2024-08-19T14:22:19.327148+0000","last_undegraded":"2024-08-19T14:22:19.327148+0000","last_fullsized":"2024-08-19T14:22:19.327148+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":9045,"log_dups_size":0,"ondisk_log_size":9045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":485031936,"num_objects":7401,"num_object_clones":0,"num_object_copies":22203,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9045,"num_write_kb":526272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"70'139","reported_seq":815,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-08-19T14:22:17.099861+0000","last_change":"2024-08-19T14:20:50.742676+0000","last_active":"2024-08-19T14:22:17.099861+0000","last_peered":"2024-08-19T14:22:17.099861+0000","last_clean":"2024-08-19T14:22:17.099861+0000","last_became_active":"2024-08-19T14:20:50.742354+0000","last_became_peered":"2024-08-19T14:20:50.742354+0000","last_unstale":"2024-08-19T14:22:17.099861+0000","last_undegraded":"2024-08-19T14:22:17.099861+0000","last_fullsized":"2024-08-19T14:22:17.099861+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:19:59.302931+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:19:59.302931+0000","last_clean_scrub_stamp":"2024-08-19T14:19:59.302931+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":114,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"376'9013","reported_seq":9105,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-08-19T14:22:19.283355+0000","last_change":"2024-08-19T14:20:29.267640+0000","last_active":"2024-08-19T14:22:19.283355+0000","last_peered":"2024-08-19T14:22:19.283355+0000","last_clean":"2024-08-19T14:22:19.283355+0000","last_became_active":"2024-08-19T14:20:29.267331+0000","last_became_peered":"2024-08-19T14:20:29.267331+0000","last_unstale":"2024-08-19T14:22:19.283355+0000","last_undegraded":"2024-08-19T14:22:19.283355+0000","last_fullsized":"2024-08-19T14:22:19.283355+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":9013,"log_dups_size":0,"ondisk_log_size":9013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483196928,"num_objects":7373,"num_object_clones":0,"num_object_copies":22119,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9013,"num_write_kb":524352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"70'139","reported_seq":864,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-08-19T14:22:17.096374+0000","last_change":"2024-08-19T14:20:09.507422+0000","last_active":"2024-08-19T14:22:17.096374+0000","last_peered":"2024-08-19T14:22:17.096374+0000","last_clean":"2024-08-19T14:22:17.096374+0000","last_became_active":"2024-08-19T14:18:13.808505+0000","last_became_peered":"2024-08-19T14:18:13.808505+0000","last_unstale":"2024-08-19T14:22:17.096374+0000","last_undegraded":"2024-08-19T14:22:17.096374+0000","last_fullsized":"2024-08-19T14:22:17.096374+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:09.507256+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:09.507256+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,11],"acting":[9,0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"376'9069","reported_seq":9161,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-08-19T14:22:18.412571+0000","last_change":"2024-08-19T14:20:29.268766+0000","last_active":"2024-08-19T14:22:18.412571+0000","last_peered":"2024-08-19T14:22:18.412571+0000","last_clean":"2024-08-19T14:22:18.412571+0000","last_became_active":"2024-08-19T14:20:29.268443+0000","last_became_peered":"2024-08-19T14:20:29.268443+0000","last_unstale":"2024-08-19T14:22:18.412571+0000","last_undegraded":"2024-08-19T14:22:18.412571+0000","last_fullsized":"2024-08-19T14:22:18.412571+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":9069,"log_dups_size":0,"ondisk_log_size":9069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":487784448,"num_objects":7443,"num_object_clones":0,"num_object_copies":22329,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9069,"num_write_kb":528384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,11],"acting":[9,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.7","version":"70'139","reported_seq":515,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-08-19T14:22:17.095428+0000","last_change":"2024-08-19T14:21:46.057764+0000","last_active":"2024-08-19T14:22:17.095428+0000","last_peered":"2024-08-19T14:22:17.095428+0000","last_clean":"2024-08-19T14:22:17.095428+0000","last_became_active":"2024-08-19T14:20:28.204384+0000","last_became_peered":"2024-08-19T14:20:28.204384+0000","last_unstale":"2024-08-19T14:22:17.095428+0000","last_undegraded":"2024-08-19T14:22:17.095428+0000","last_fullsized":"2024-08-19T14:22:17.095428+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:46.057624+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:21:46.057624+0000","last_clean_scrub_stamp":"2024-08-19T14:21:46.057624+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:02.689278+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"377'9103","reported_seq":9341,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-08-19T14:22:20.739466+0000","last_change":"2024-08-19T14:22:07.840154+0000","last_active":"2024-08-19T14:22:20.739466+0000","last_peered":"2024-08-19T14:22:20.739466+0000","last_clean":"2024-08-19T14:22:20.739466+0000","last_became_active":"2024-08-19T14:20:29.267379+0000","last_became_peered":"2024-08-19T14:20:29.267379+0000","last_unstale":"2024-08-19T14:22:20.739466+0000","last_undegraded":"2024-08-19T14:22:20.739466+0000","last_fullsized":"2024-08-19T14:22:20.739466+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"373'8338","last_scrub_stamp":"2024-08-19T14:22:07.840021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:22:07.840021+0000","objects_scrubbed":8073,"log_size":9103,"log_dups_size":0,"ondisk_log_size":9103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:13.679336+0000","scrub_duration":2409,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":474939392,"num_objects":7247,"num_object_clones":0,"num_object_copies":21741,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7247,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9103,"num_write_kb":523200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,1],"acting":[11,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.6","version":"70'139","reported_seq":868,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-08-19T14:22:17.097205+0000","last_change":"2024-08-19T14:19:55.512078+0000","last_active":"2024-08-19T14:22:17.097205+0000","last_peered":"2024-08-19T14:22:17.097205+0000","last_clean":"2024-08-19T14:22:17.097205+0000","last_became_active":"2024-08-19T14:18:26.874902+0000","last_became_peered":"2024-08-19T14:18:26.874902+0000","last_unstale":"2024-08-19T14:22:17.097205+0000","last_undegraded":"2024-08-19T14:22:17.097205+0000","last_fullsized":"2024-08-19T14:22:17.097205+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:19:55.511974+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:27.848870+0000","last_clean_scrub_stamp":"2024-08-19T14:19:55.511974+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,4],"acting":[9,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"376'8988","reported_seq":9168,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-08-19T14:22:19.571907+0000","last_change":"2024-08-19T14:22:08.536948+0000","last_active":"2024-08-19T14:22:19.571907+0000","last_peered":"2024-08-19T14:22:19.571907+0000","last_clean":"2024-08-19T14:22:19.571907+0000","last_became_active":"2024-08-19T14:20:29.267360+0000","last_became_peered":"2024-08-19T14:20:29.267360+0000","last_unstale":"2024-08-19T14:22:19.571907+0000","last_undegraded":"2024-08-19T14:22:19.571907+0000","last_fullsized":"2024-08-19T14:22:19.571907+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"374'8331","last_scrub_stamp":"2024-08-19T14:22:08.536620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:22:08.536620+0000","objects_scrubbed":7995,"log_size":8988,"log_dups_size":0,"ondisk_log_size":8988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:19.976024+0000","scrub_duration":2478,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":475267072,"num_objects":7252,"num_object_clones":0,"num_object_copies":21756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8988,"num_write_kb":519680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"70'139","reported_seq":691,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-08-19T14:22:20.118616+0000","last_change":"2024-08-19T14:22:01.354585+0000","last_active":"2024-08-19T14:22:20.118616+0000","last_peered":"2024-08-19T14:22:20.118616+0000","last_clean":"2024-08-19T14:22:20.118616+0000","last_became_active":"2024-08-19T14:18:26.881435+0000","last_became_peered":"2024-08-19T14:18:26.881435+0000","last_unstale":"2024-08-19T14:22:20.118616+0000","last_undegraded":"2024-08-19T14:22:20.118616+0000","last_fullsized":"2024-08-19T14:22:20.118616+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:22:01.354343+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:22:01.354343+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:30.081548+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,4],"acting":[11,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.1","version":"377'9213","reported_seq":9381,"reported_epoch":377,"state":"active+clean+scrubbing","last_fresh":"2024-08-19T14:22:20.871571+0000","last_change":"2024-08-19T14:22:19.291604+0000","last_active":"2024-08-19T14:22:20.871571+0000","last_peered":"2024-08-19T14:22:20.871571+0000","last_clean":"2024-08-19T14:22:20.871571+0000","last_became_active":"2024-08-19T14:20:29.268465+0000","last_became_peered":"2024-08-19T14:20:29.268465+0000","last_unstale":"2024-08-19T14:22:20.871571+0000","last_undegraded":"2024-08-19T14:22:20.871571+0000","last_fullsized":"2024-08-19T14:22:20.871571+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":6999,"log_size":9213,"log_dups_size":0,"ondisk_log_size":9213,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 1s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":477691904,"num_objects":7289,"num_object_clones":0,"num_object_copies":21867,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9213,"num_write_kb":528064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.4","version":"70'139","reported_seq":916,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-08-19T14:22:20.118490+0000","last_change":"2024-08-19T14:21:39.314063+0000","last_active":"2024-08-19T14:22:20.118490+0000","last_peered":"2024-08-19T14:22:20.118490+0000","last_clean":"2024-08-19T14:22:20.118490+0000","last_became_active":"2024-08-19T14:18:13.806013+0000","last_became_peered":"2024-08-19T14:18:13.806013+0000","last_unstale":"2024-08-19T14:22:20.118490+0000","last_undegraded":"2024-08-19T14:22:20.118490+0000","last_fullsized":"2024-08-19T14:22:20.118490+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:39.313941+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:21:39.313941+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:22:45.184594+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.f","version":"376'8962","reported_seq":9054,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-08-19T14:22:19.327237+0000","last_change":"2024-08-19T14:20:29.212298+0000","last_active":"2024-08-19T14:22:19.327237+0000","last_peered":"2024-08-19T14:22:19.327237+0000","last_clean":"2024-08-19T14:22:19.327237+0000","last_became_active":"2024-08-19T14:20:29.212003+0000","last_became_peered":"2024-08-19T14:20:29.212003+0000","last_unstale":"2024-08-19T14:22:19.327237+0000","last_undegraded":"2024-08-19T14:22:19.327237+0000","last_fullsized":"2024-08-19T14:22:19.327237+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":8962,"log_dups_size":0,"ondisk_log_size":8962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":481034240,"num_objects":7340,"num_object_clones":0,"num_object_copies":22020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8962,"num_write_kb":521664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,11],"acting":[1,3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"70'139","reported_seq":876,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-08-19T14:22:17.096450+0000","last_change":"2024-08-19T14:20:12.570453+0000","last_active":"2024-08-19T14:22:17.096450+0000","last_peered":"2024-08-19T14:22:17.096450+0000","last_clean":"2024-08-19T14:22:17.096450+0000","last_became_active":"2024-08-19T14:18:26.875006+0000","last_became_peered":"2024-08-19T14:18:26.875006+0000","last_unstale":"2024-08-19T14:22:17.096450+0000","last_undegraded":"2024-08-19T14:22:17.096450+0000","last_fullsized":"2024-08-19T14:22:17.096450+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:12.570352+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:28.864870+0000","last_clean_scrub_stamp":"2024-08-19T14:20:12.570352+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,0],"acting":[9,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.c","version":"376'8977","reported_seq":9157,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-08-19T14:22:18.179258+0000","last_change":"2024-08-19T14:22:07.487434+0000","last_active":"2024-08-19T14:22:18.179258+0000","last_peered":"2024-08-19T14:22:18.179258+0000","last_clean":"2024-08-19T14:22:18.179258+0000","last_became_active":"2024-08-19T14:20:29.266373+0000","last_became_peered":"2024-08-19T14:20:29.266373+0000","last_unstale":"2024-08-19T14:22:18.179258+0000","last_undegraded":"2024-08-19T14:22:18.179258+0000","last_fullsized":"2024-08-19T14:22:18.179258+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"373'8336","last_scrub_stamp":"2024-08-19T14:22:07.487270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:22:07.487270+0000","objects_scrubbed":8079,"log_size":8977,"log_dups_size":0,"ondisk_log_size":8977,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:14.053549+0000","scrub_duration":1972,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483983360,"num_objects":7385,"num_object_clones":0,"num_object_copies":22155,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8977,"num_write_kb":523584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,8],"acting":[4,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"70'139","reported_seq":991,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-08-19T14:22:17.098660+0000","last_change":"2024-08-19T14:19:56.106047+0000","last_active":"2024-08-19T14:22:17.098660+0000","last_peered":"2024-08-19T14:22:17.098660+0000","last_clean":"2024-08-19T14:22:17.098660+0000","last_became_active":"2024-08-19T14:18:26.887657+0000","last_became_peered":"2024-08-19T14:18:26.887657+0000","last_unstale":"2024-08-19T14:22:17.098660+0000","last_undegraded":"2024-08-19T14:22:17.098660+0000","last_fullsized":"2024-08-19T14:22:17.098660+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:19:56.105880+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:19:56.105880+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"376'8877","reported_seq":9059,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-08-19T14:22:18.174773+0000","last_change":"2024-08-19T14:22:08.529439+0000","last_active":"2024-08-19T14:22:18.174773+0000","last_peered":"2024-08-19T14:22:18.174773+0000","last_clean":"2024-08-19T14:22:18.174773+0000","last_became_active":"2024-08-19T14:20:29.208599+0000","last_became_peered":"2024-08-19T14:20:29.208599+0000","last_unstale":"2024-08-19T14:22:18.174773+0000","last_undegraded":"2024-08-19T14:22:18.174773+0000","last_fullsized":"2024-08-19T14:22:18.174773+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"374'8320","last_scrub_stamp":"2024-08-19T14:22:08.529109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:22:08.529109+0000","objects_scrubbed":7983,"log_size":8877,"log_dups_size":0,"ondisk_log_size":8877,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:30.745931+0000","scrub_duration":1962,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483131416,"num_objects":7373,"num_object_clones":0,"num_object_copies":22119,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7373,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8877,"num_write_kb":519937,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,1],"acting":[4,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"70'139","reported_seq":890,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-08-19T14:22:20.133331+0000","last_change":"2024-08-19T14:21:37.951800+0000","last_active":"2024-08-19T14:22:20.133331+0000","last_peered":"2024-08-19T14:22:20.133331+0000","last_clean":"2024-08-19T14:22:20.133331+0000","last_became_active":"2024-08-19T14:18:26.887277+0000","last_became_peered":"2024-08-19T14:18:26.887277+0000","last_unstale":"2024-08-19T14:22:20.133331+0000","last_undegraded":"2024-08-19T14:22:20.133331+0000","last_fullsized":"2024-08-19T14:22:20.133331+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:37.951694+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:21:37.951694+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:03.860882+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,8],"acting":[5,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"376'8858","reported_seq":9035,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-08-19T14:22:18.086341+0000","last_change":"2024-08-19T14:22:13.656515+0000","last_active":"2024-08-19T14:22:18.086341+0000","last_peered":"2024-08-19T14:22:18.086341+0000","last_clean":"2024-08-19T14:22:18.086341+0000","last_became_active":"2024-08-19T14:20:29.208748+0000","last_became_peered":"2024-08-19T14:20:29.208748+0000","last_unstale":"2024-08-19T14:22:18.086341+0000","last_undegraded":"2024-08-19T14:22:18.086341+0000","last_fullsized":"2024-08-19T14:22:18.086341+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"374'8571","last_scrub_stamp":"2024-08-19T14:22:13.656335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:22:13.656335+0000","objects_scrubbed":7643,"log_size":8858,"log_dups_size":0,"ondisk_log_size":8858,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:30.767371+0000","scrub_duration":3043,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":477102080,"num_objects":7280,"num_object_clones":0,"num_object_copies":21840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8858,"num_write_kb":516416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"70'139","reported_seq":891,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-08-19T14:22:20.118488+0000","last_change":"2024-08-19T14:22:02.413212+0000","last_active":"2024-08-19T14:22:20.118488+0000","last_peered":"2024-08-19T14:22:20.118488+0000","last_clean":"2024-08-19T14:22:20.118488+0000","last_became_active":"2024-08-19T14:18:26.875179+0000","last_became_peered":"2024-08-19T14:18:26.875179+0000","last_unstale":"2024-08-19T14:22:20.118488+0000","last_undegraded":"2024-08-19T14:22:20.118488+0000","last_fullsized":"2024-08-19T14:22:20.118488+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:22:02.413091+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:22:02.413091+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:23.082273+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.b","version":"377'9137","reported_seq":9306,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-08-19T14:22:21.189205+0000","last_change":"2024-08-19T14:21:39.089676+0000","last_active":"2024-08-19T14:22:21.189205+0000","last_peered":"2024-08-19T14:22:21.189205+0000","last_clean":"2024-08-19T14:22:21.189205+0000","last_became_active":"2024-08-19T14:20:29.207372+0000","last_became_peered":"2024-08-19T14:20:29.207372+0000","last_unstale":"2024-08-19T14:22:21.189205+0000","last_undegraded":"2024-08-19T14:22:21.189205+0000","last_fullsized":"2024-08-19T14:22:21.189205+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"363'6615","last_scrub_stamp":"2024-08-19T14:21:39.089400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:21:39.089400+0000","objects_scrubbed":6552,"log_size":9137,"log_dups_size":0,"ondisk_log_size":9137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:05.632119+0000","scrub_duration":1907,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":472186880,"num_objects":7205,"num_object_clones":0,"num_object_copies":21615,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9137,"num_write_kb":522944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"70'139","reported_seq":543,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-08-19T14:22:17.096503+0000","last_change":"2024-08-19T14:21:44.434481+0000","last_active":"2024-08-19T14:22:17.096503+0000","last_peered":"2024-08-19T14:22:17.096503+0000","last_clean":"2024-08-19T14:22:17.096503+0000","last_became_active":"2024-08-19T14:18:26.884105+0000","last_became_peered":"2024-08-19T14:18:26.884105+0000","last_unstale":"2024-08-19T14:22:17.096503+0000","last_undegraded":"2024-08-19T14:22:17.096503+0000","last_fullsized":"2024-08-19T14:22:17.096503+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:44.434382+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:20:15.321945+0000","last_clean_scrub_stamp":"2024-08-19T14:21:44.434382+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:13.544241+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"376'9109","reported_seq":9201,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-08-19T14:22:19.322252+0000","last_change":"2024-08-19T14:20:29.269723+0000","last_active":"2024-08-19T14:22:19.322252+0000","last_peered":"2024-08-19T14:22:19.322252+0000","last_clean":"2024-08-19T14:22:19.322252+0000","last_became_active":"2024-08-19T14:20:29.269432+0000","last_became_peered":"2024-08-19T14:20:29.269432+0000","last_unstale":"2024-08-19T14:22:19.322252+0000","last_undegraded":"2024-08-19T14:22:19.322252+0000","last_fullsized":"2024-08-19T14:22:19.322252+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":9109,"log_dups_size":0,"ondisk_log_size":9109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":491978752,"num_objects":7507,"num_object_clones":0,"num_object_copies":22521,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7507,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9109,"num_write_kb":531712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.f","version":"70'139","reported_seq":789,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-08-19T14:22:20.133394+0000","last_change":"2024-08-19T14:21:14.765419+0000","last_active":"2024-08-19T14:22:20.133394+0000","last_peered":"2024-08-19T14:22:20.133394+0000","last_clean":"2024-08-19T14:22:20.133394+0000","last_became_active":"2024-08-19T14:18:18.711708+0000","last_became_peered":"2024-08-19T14:18:18.711708+0000","last_unstale":"2024-08-19T14:22:20.133394+0000","last_undegraded":"2024-08-19T14:22:20.133394+0000","last_fullsized":"2024-08-19T14:22:20.133394+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:14.765340+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:21:14.765340+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:22:24.415219+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"377'9154","reported_seq":9327,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-08-19T14:22:20.747364+0000","last_change":"2024-08-19T14:21:43.974700+0000","last_active":"2024-08-19T14:22:20.747364+0000","last_peered":"2024-08-19T14:22:20.747364+0000","last_clean":"2024-08-19T14:22:20.747364+0000","last_became_active":"2024-08-19T14:20:29.211975+0000","last_became_peered":"2024-08-19T14:20:29.211975+0000","last_unstale":"2024-08-19T14:22:20.747364+0000","last_undegraded":"2024-08-19T14:22:20.747364+0000","last_fullsized":"2024-08-19T14:22:20.747364+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"364'6991","last_scrub_stamp":"2024-08-19T14:21:43.974613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:21:43.974613+0000","objects_scrubbed":6934,"log_size":9154,"log_dups_size":0,"ondisk_log_size":9154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:07.463188+0000","scrub_duration":1682,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":478674944,"num_objects":7304,"num_object_clones":0,"num_object_copies":21912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7304,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9154,"num_write_kb":526656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,3],"acting":[11,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.c","version":"70'139","reported_seq":648,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-08-19T14:22:17.104260+0000","last_change":"2024-08-19T14:21:40.181472+0000","last_active":"2024-08-19T14:22:17.104260+0000","last_peered":"2024-08-19T14:22:17.104260+0000","last_clean":"2024-08-19T14:22:17.104260+0000","last_became_active":"2024-08-19T14:18:26.872942+0000","last_became_peered":"2024-08-19T14:18:26.872942+0000","last_unstale":"2024-08-19T14:22:17.104260+0000","last_undegraded":"2024-08-19T14:22:17.104260+0000","last_fullsized":"2024-08-19T14:22:17.104260+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:40.181356+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:20:07.839076+0000","last_clean_scrub_stamp":"2024-08-19T14:21:40.181356+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:22:52.009381+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,9],"acting":[3,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"377'9103","reported_seq":9284,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-08-19T14:22:20.762693+0000","last_change":"2024-08-19T14:22:12.682046+0000","last_active":"2024-08-19T14:22:20.762693+0000","last_peered":"2024-08-19T14:22:20.762693+0000","last_clean":"2024-08-19T14:22:20.762693+0000","last_became_active":"2024-08-19T14:20:29.267442+0000","last_became_peered":"2024-08-19T14:20:29.267442+0000","last_unstale":"2024-08-19T14:22:20.762693+0000","last_undegraded":"2024-08-19T14:22:20.762693+0000","last_fullsized":"2024-08-19T14:22:20.762693+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"374'8663","last_scrub_stamp":"2024-08-19T14:22:12.681916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:22:12.681916+0000","objects_scrubbed":7729,"log_size":9103,"log_dups_size":0,"ondisk_log_size":9103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:34.638254+0000","scrub_duration":1366,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":475856896,"num_objects":7261,"num_object_clones":0,"num_object_copies":21783,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7261,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9103,"num_write_kb":523648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,9],"acting":[11,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.d","version":"70'139","reported_seq":563,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-08-19T14:22:17.100184+0000","last_change":"2024-08-19T14:21:07.155294+0000","last_active":"2024-08-19T14:22:17.100184+0000","last_peered":"2024-08-19T14:22:17.100184+0000","last_clean":"2024-08-19T14:22:17.100184+0000","last_became_active":"2024-08-19T14:18:26.882002+0000","last_became_peered":"2024-08-19T14:18:26.882002+0000","last_unstale":"2024-08-19T14:22:17.100184+0000","last_undegraded":"2024-08-19T14:22:17.100184+0000","last_fullsized":"2024-08-19T14:22:17.100184+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:07.155157+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:21:07.155157+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"70'139","reported_seq":824,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-08-19T14:22:20.136216+0000","last_change":"2024-08-19T14:21:13.777891+0000","last_active":"2024-08-19T14:22:20.136216+0000","last_peered":"2024-08-19T14:22:20.136216+0000","last_clean":"2024-08-19T14:22:20.136216+0000","last_became_active":"2024-08-19T14:18:18.703594+0000","last_became_peered":"2024-08-19T14:18:18.703594+0000","last_unstale":"2024-08-19T14:22:20.136216+0000","last_undegraded":"2024-08-19T14:22:20.136216+0000","last_fullsized":"2024-08-19T14:22:20.136216+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:13.777777+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:21:13.777777+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:22:38.473560+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"70'139","reported_seq":874,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-08-19T14:22:17.097247+0000","last_change":"2024-08-19T14:20:07.692054+0000","last_active":"2024-08-19T14:22:17.097247+0000","last_peered":"2024-08-19T14:22:17.097247+0000","last_clean":"2024-08-19T14:22:17.097247+0000","last_became_active":"2024-08-19T14:18:18.698334+0000","last_became_peered":"2024-08-19T14:18:18.698334+0000","last_unstale":"2024-08-19T14:22:17.097247+0000","last_undegraded":"2024-08-19T14:22:17.097247+0000","last_fullsized":"2024-08-19T14:22:17.097247+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:07.691932+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:07.691932+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.10","version":"70'139","reported_seq":987,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-08-19T14:22:20.133229+0000","last_change":"2024-08-19T14:22:11.260404+0000","last_active":"2024-08-19T14:22:20.133229+0000","last_peered":"2024-08-19T14:22:20.133229+0000","last_clean":"2024-08-19T14:22:20.133229+0000","last_became_active":"2024-08-19T14:18:18.704562+0000","last_became_peered":"2024-08-19T14:18:18.704562+0000","last_unstale":"2024-08-19T14:22:20.133229+0000","last_undegraded":"2024-08-19T14:22:20.133229+0000","last_fullsized":"2024-08-19T14:22:20.133229+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:22:11.260286+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:22:11.260286+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:36.552471+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,8],"acting":[5,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"70'139","reported_seq":536,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-08-19T14:22:21.104560+0000","last_change":"2024-08-19T14:22:21.104595+0000","last_active":"2024-08-19T14:22:21.104560+0000","last_peered":"2024-08-19T14:22:21.104560+0000","last_clean":"2024-08-19T14:22:21.104560+0000","last_became_active":"2024-08-19T14:18:18.697894+0000","last_became_peered":"2024-08-19T14:18:18.697894+0000","last_unstale":"2024-08-19T14:22:21.104560+0000","last_undegraded":"2024-08-19T14:22:21.104560+0000","last_fullsized":"2024-08-19T14:22:21.104560+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:22:21.104491+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:21:16.071703+0000","last_clean_scrub_stamp":"2024-08-19T14:22:21.104491+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:45.856425+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"70'139","reported_seq":503,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-08-19T14:22:17.097457+0000","last_change":"2024-08-19T14:20:28.205212+0000","last_active":"2024-08-19T14:22:17.097457+0000","last_peered":"2024-08-19T14:22:17.097457+0000","last_clean":"2024-08-19T14:22:17.097457+0000","last_became_active":"2024-08-19T14:20:28.204724+0000","last_became_peered":"2024-08-19T14:20:28.204724+0000","last_unstale":"2024-08-19T14:22:17.097457+0000","last_undegraded":"2024-08-19T14:22:17.097457+0000","last_fullsized":"2024-08-19T14:22:17.097457+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:08.680021+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:08.680021+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"70'139","reported_seq":631,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-08-19T14:22:17.101004+0000","last_change":"2024-08-19T14:22:02.722505+0000","last_active":"2024-08-19T14:22:17.101004+0000","last_peered":"2024-08-19T14:22:17.101004+0000","last_clean":"2024-08-19T14:22:17.101004+0000","last_became_active":"2024-08-19T14:18:31.901556+0000","last_became_peered":"2024-08-19T14:18:31.901556+0000","last_unstale":"2024-08-19T14:22:17.101004+0000","last_undegraded":"2024-08-19T14:22:17.101004+0000","last_fullsized":"2024-08-19T14:22:17.101004+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:22:02.722345+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:22:02.722345+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:13.381470+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,0],"acting":[8,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.14","version":"70'139","reported_seq":632,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-08-19T14:22:17.100471+0000","last_change":"2024-08-19T14:19:55.130518+0000","last_active":"2024-08-19T14:22:17.100471+0000","last_peered":"2024-08-19T14:22:17.100471+0000","last_clean":"2024-08-19T14:22:17.100471+0000","last_became_active":"2024-08-19T14:18:18.703926+0000","last_became_peered":"2024-08-19T14:18:18.703926+0000","last_unstale":"2024-08-19T14:22:17.100471+0000","last_undegraded":"2024-08-19T14:22:17.100471+0000","last_fullsized":"2024-08-19T14:22:17.100471+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:19:55.130420+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:19:55.130420+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,1],"acting":[7,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.15","version":"70'139","reported_seq":519,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-08-19T14:22:17.098662+0000","last_change":"2024-08-19T14:20:09.181046+0000","last_active":"2024-08-19T14:22:17.098662+0000","last_peered":"2024-08-19T14:22:17.098662+0000","last_clean":"2024-08-19T14:22:17.098662+0000","last_became_active":"2024-08-19T14:18:31.912496+0000","last_became_peered":"2024-08-19T14:18:31.912496+0000","last_unstale":"2024-08-19T14:22:17.098662+0000","last_undegraded":"2024-08-19T14:22:17.098662+0000","last_fullsized":"2024-08-19T14:22:17.098662+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:09.180923+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:20:09.180923+0000","last_clean_scrub_stamp":"2024-08-19T14:20:09.180923+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"70'139","reported_seq":505,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-08-19T14:22:17.096039+0000","last_change":"2024-08-19T14:21:40.430835+0000","last_active":"2024-08-19T14:22:17.096039+0000","last_peered":"2024-08-19T14:22:17.096039+0000","last_clean":"2024-08-19T14:22:17.096039+0000","last_became_active":"2024-08-19T14:18:31.901229+0000","last_became_peered":"2024-08-19T14:18:31.901229+0000","last_unstale":"2024-08-19T14:22:17.096039+0000","last_undegraded":"2024-08-19T14:22:17.096039+0000","last_fullsized":"2024-08-19T14:22:17.096039+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:40.430714+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:21:40.430714+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:04.072653+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1b","version":"70'139","reported_seq":857,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-08-19T14:22:17.097958+0000","last_change":"2024-08-19T14:21:48.693864+0000","last_active":"2024-08-19T14:22:17.097958+0000","last_peered":"2024-08-19T14:22:17.097958+0000","last_clean":"2024-08-19T14:22:17.097958+0000","last_became_active":"2024-08-19T14:18:31.901152+0000","last_became_peered":"2024-08-19T14:18:31.901152+0000","last_unstale":"2024-08-19T14:22:17.097958+0000","last_undegraded":"2024-08-19T14:22:17.097958+0000","last_fullsized":"2024-08-19T14:22:17.097958+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:48.693761+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:21:48.693761+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:07.571130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.18","version":"70'139","reported_seq":903,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-08-19T14:22:20.133222+0000","last_change":"2024-08-19T14:22:07.254887+0000","last_active":"2024-08-19T14:22:20.133222+0000","last_peered":"2024-08-19T14:22:20.133222+0000","last_clean":"2024-08-19T14:22:20.133222+0000","last_became_active":"2024-08-19T14:18:31.902735+0000","last_became_peered":"2024-08-19T14:18:31.902735+0000","last_unstale":"2024-08-19T14:22:20.133222+0000","last_undegraded":"2024-08-19T14:22:20.133222+0000","last_fullsized":"2024-08-19T14:22:20.133222+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:22:07.254767+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:22:07.254767+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:11.195110+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.19","version":"70'139","reported_seq":851,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-08-19T14:22:20.115632+0000","last_change":"2024-08-19T14:21:40.219949+0000","last_active":"2024-08-19T14:22:20.115632+0000","last_peered":"2024-08-19T14:22:20.115632+0000","last_clean":"2024-08-19T14:22:20.115632+0000","last_became_active":"2024-08-19T14:18:31.902547+0000","last_became_peered":"2024-08-19T14:18:31.902547+0000","last_unstale":"2024-08-19T14:22:20.115632+0000","last_undegraded":"2024-08-19T14:22:20.115632+0000","last_fullsized":"2024-08-19T14:22:20.115632+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:40.219830+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:21:40.219830+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:05.414864+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1e","version":"70'139","reported_seq":509,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-08-19T14:22:20.118534+0000","last_change":"2024-08-19T14:22:09.404540+0000","last_active":"2024-08-19T14:22:20.118534+0000","last_peered":"2024-08-19T14:22:20.118534+0000","last_clean":"2024-08-19T14:22:20.118534+0000","last_became_active":"2024-08-19T14:18:31.902546+0000","last_became_peered":"2024-08-19T14:18:31.902546+0000","last_unstale":"2024-08-19T14:22:20.118534+0000","last_undegraded":"2024-08-19T14:22:20.118534+0000","last_fullsized":"2024-08-19T14:22:20.118534+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:22:09.404433+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:22:09.404433+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:13.587290+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1c","version":"70'139","reported_seq":615,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-08-19T14:22:17.101608+0000","last_change":"2024-08-19T14:21:38.753663+0000","last_active":"2024-08-19T14:22:17.101608+0000","last_peered":"2024-08-19T14:22:17.101608+0000","last_clean":"2024-08-19T14:22:17.101608+0000","last_became_active":"2024-08-19T14:18:31.902494+0000","last_became_peered":"2024-08-19T14:18:31.902494+0000","last_unstale":"2024-08-19T14:22:17.101608+0000","last_undegraded":"2024-08-19T14:22:17.101608+0000","last_fullsized":"2024-08-19T14:22:17.101608+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:38.753553+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:21:38.753553+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:22:46.720742+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,11],"acting":[8,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1d","version":"70'139","reported_seq":529,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-08-19T14:22:17.096994+0000","last_change":"2024-08-19T14:20:17.725615+0000","last_active":"2024-08-19T14:22:17.096994+0000","last_peered":"2024-08-19T14:22:17.096994+0000","last_clean":"2024-08-19T14:22:17.096994+0000","last_became_active":"2024-08-19T14:18:31.901183+0000","last_became_peered":"2024-08-19T14:18:31.901183+0000","last_unstale":"2024-08-19T14:22:17.096994+0000","last_undegraded":"2024-08-19T14:22:17.096994+0000","last_fullsized":"2024-08-19T14:22:17.096994+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:17.725477+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:17.725477+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,5],"acting":[1,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":7689732120,"num_objects":117337,"num_object_clones":0,"num_object_copies":352011,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1990,"num_objects_unfound":0,"num_objects_dirty":117337,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":145009,"num_write_kb":8395009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2150,"num_bytes_recovered":140902400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":23226626048,"data_stored":23226613832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":145009,"ondisk_log_size":145009,"up":48,"acting":47,"num_store_stats":9},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1802240,"data_stored":1788032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187588,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3705740,"kb_used_data":3273344,"kb_used_omap":1,"kb_used_meta":432382,"kb_avail":90039412,"statfs":{"total":95995035648,"available":92200357888,"internally_reserved":0,"allocated":3351904256,"data_stored":3350167514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":442759620},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":343597383817,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":74,"seq":317827580045,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2246532,"kb_used_data":1782200,"kb_used_omap":1,"kb_used_meta":464318,"kb_avail":91498620,"statfs":{"total":95995035648,"available":93694586880,"internally_reserved":0,"allocated":1824972800,"data_stored":1823236162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":475462081},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874387,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":850108,"kb_used_data":471860,"kb_used_omap":1,"kb_used_meta":378238,"kb_avail":92895044,"statfs":{"total":95995035648,"available":95124525056,"internally_reserved":0,"allocated":483184640,"data_stored":481452074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":387316165},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":57,"seq":244813136021,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4286472,"kb_used_data":3813820,"kb_used_omap":1,"kb_used_meta":472638,"kb_avail":89458680,"statfs":{"total":95995035648,"available":91605688320,"internally_reserved":0,"allocated":3905351680,"data_stored":3903615562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":483981770},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":206158430361,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691998,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2021776,"kb_used_data":1675464,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":91723376,"statfs":{"total":95995035648,"available":93924737024,"internally_reserved":0,"allocated":1715675136,"data_stored":1713934842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":354613701},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986338,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2867588,"kb_used_data":2372728,"kb_used_omap":1,"kb_used_meta":494846,"kb_avail":90877564,"statfs":{"total":95995035648,"available":93058625536,"internally_reserved":0,"allocated":2429673472,"data_stored":2427936834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":506722757},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149864,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3245756,"kb_used_data":2817524,"kb_used_omap":1,"kb_used_meta":428222,"kb_avail":90499396,"statfs":{"total":95995035648,"available":92671381504,"internally_reserved":0,"allocated":2885144576,"data_stored":2883412010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":438499783},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411502,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575025,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5251912,"kb_used_data":4715392,"kb_used_omap":1,"kb_used_meta":536510,"kb_avail":88493240,"statfs":{"total":95995035648,"available":90617077760,"internally_reserved":0,"allocated":4828561408,"data_stored":4826821218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":549386693},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738550,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2217352,"kb_used_data":1857152,"kb_used_omap":1,"kb_used_meta":360190,"kb_avail":91527800,"statfs":{"total":95995035648,"available":93724467200,"internally_reserved":0,"allocated":1901723648,"data_stored":1899986906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":368835013},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1894842368,"data_stored":1894842368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4823846912,"data_stored":4823842840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2871132160,"data_stored":2871132160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2416316416,"data_stored":2416312344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1706819584,"data_stored":1706819584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3889889280,"data_stored":3889889280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":478019584,"data_stored":478019584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1811156992,"data_stored":1811152920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":3334602752,"data_stored":3334602752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T14:22:26.711 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:22:26.752 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:26.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:26.753 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:22:26.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:26 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:22:26.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:26 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:22:26.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:26 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:22:26.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:26 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:22:26.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:26 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:22:26.853 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:22:26.891 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:26.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:26.892 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:22:26.993 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:22:27.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:26 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:22:27.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:26 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:22:27.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:26 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:22:27.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:26 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:22:27.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:26 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:22:27.021 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:27.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:27.022 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:22:27.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:26 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:22:27.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:26 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:22:27.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:26 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:22:27.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:26 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:22:27.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:26 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:22:27.122 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:22:27.148 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:27.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:27.149 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:22:27.180 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-08-19T14:22:27.250 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:22:27.276 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:27.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:27.277 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:22:27.378 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:22:27.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:27.405 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:27.406 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:22:27.506 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:22:27.529 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:27.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:27.530 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:22:27.597 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:22:27.631 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:22:27.656 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:27.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:27.657 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:22:27.759 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:22:27.781 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-19T14:22:27.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:27.805 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:27.805 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:22:27.843 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:27 smithi031 ceph-mon[16336]: pgmap v854: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 6.9 GiB data, 25 GiB used, 780 GiB / 805 GiB avail; 0 B/s wr, 1.09k op/s; 1772/339876 objects misplaced (0.521%); 3.3 MiB/s, 52 objects/s recovering 2024-08-19T14:22:27.906 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:22:27.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:27.964 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:27.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:22:28.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:27 smithi032 ceph-mon[20004]: pgmap v854: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 6.9 GiB data, 25 GiB used, 780 GiB / 805 GiB avail; 0 B/s wr, 1.09k op/s; 1772/339876 objects misplaced (0.521%); 3.3 MiB/s, 52 objects/s recovering 2024-08-19T14:22:28.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:27 smithi045 ceph-mon[20143]: pgmap v854: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 6.9 GiB data, 25 GiB used, 780 GiB / 805 GiB avail; 0 B/s wr, 1.09k op/s; 1772/339876 objects misplaced (0.521%); 3.3 MiB/s, 52 objects/s recovering 2024-08-19T14:22:28.066 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:22:28.082 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:22:28.088 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:28.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:28.089 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:22:28.190 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:22:28.214 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:28.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:28.215 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:22:28.242 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:22:28.242 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [ 2024-08-19T14:22:28.242 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:28.242 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.25333.0:152612 4.d e378/335 4:b39fd589:::benchmark_data_smithi045_71562_object21437:head v 378'9495, mlcod=378'9489)", 2024-08-19T14:22:28.242 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:22:28.224604+0000", 2024-08-19T14:22:28.242 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.014950382999999999, 2024-08-19T14:22:28.242 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.01498405, 2024-08-19T14:22:28.242 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:22:28.242 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:22:28.242 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-08-19T14:22:28.243 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:22:28.243 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:28.243 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:22:28.243 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:28.224604+0000", 2024-08-19T14:22:28.243 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:22:28.243 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:28.243 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:28.243 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:22:28.243 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:28.224604+0000", 2024-08-19T14:22:28.243 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:22:28.243 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:28.243 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:28.243 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:22:28.243 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:28.224605+0000", 2024-08-19T14:22:28.244 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.6200000000000006e-07 2024-08-19T14:22:28.244 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:28.244 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:28.244 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:22:28.244 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:28.224612+0000", 2024-08-19T14:22:28.244 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.4840000000000001e-06 2024-08-19T14:22:28.244 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:28.244 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:28.244 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:22:28.244 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:28.224623+0000", 2024-08-19T14:22:28.244 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1528999999999999e-05 2024-08-19T14:22:28.244 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:28.244 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:28.245 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:22:28.245 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:28.224630+0000", 2024-08-19T14:22:28.245 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.1940000000000001e-06 2024-08-19T14:22:28.245 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:28.245 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:28.245 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:22:28.245 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:28.229809+0000", 2024-08-19T14:22:28.245 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0051785859999999998 2024-08-19T14:22:28.245 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:28.245 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:28.245 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:22:28.245 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:28.229832+0000", 2024-08-19T14:22:28.245 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.2991000000000001e-05 2024-08-19T14:22:28.245 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:22:28.246 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:22:28.246 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:22:28.246 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:28.246 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:28.246 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25333.0:152614 4.7 4:efc04532:::benchmark_data_smithi045_71562_object21439:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e378)", 2024-08-19T14:22:28.246 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:22:28.228352+0000", 2024-08-19T14:22:28.246 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.01120238, 2024-08-19T14:22:28.246 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.011307759000000001, 2024-08-19T14:22:28.246 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:22:28.246 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:22:28.246 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "waiting for sub ops", 2024-08-19T14:22:28.246 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:22:28.247 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25333", 2024-08-19T14:22:28.247 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:22:28.247 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 152614 2024-08-19T14:22:28.247 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:28.247 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:22:28.247 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:28.247 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:22:28.247 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:28.228352+0000", 2024-08-19T14:22:28.247 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:22:28.247 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:28.247 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:28.247 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:22:28.247 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:28.228352+0000", 2024-08-19T14:22:28.248 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:22:28.248 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:28.248 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:28.248 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:22:28.248 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:28.228354+0000", 2024-08-19T14:22:28.248 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.519e-06 2024-08-19T14:22:28.248 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:28.248 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:28.248 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:22:28.248 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:28.228361+0000", 2024-08-19T14:22:28.248 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.7340000000000002e-06 2024-08-19T14:22:28.248 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:28.248 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:28.248 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:22:28.249 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:28.228377+0000", 2024-08-19T14:22:28.249 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6308999999999999e-05 2024-08-19T14:22:28.249 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:28.249 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:28.249 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:22:28.249 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:28.228386+0000", 2024-08-19T14:22:28.249 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.8049999999999996e-06 2024-08-19T14:22:28.249 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:28.249 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:28.249 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:22:28.249 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:28.228446+0000", 2024-08-19T14:22:28.249 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.0315000000000001e-05 2024-08-19T14:22:28.249 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:28.249 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:28.250 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:22:28.250 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:28.228591+0000", 2024-08-19T14:22:28.250 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00014479599999999999 2024-08-19T14:22:28.250 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:28.250 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:28.250 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,7", 2024-08-19T14:22:28.250 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:28.228699+0000", 2024-08-19T14:22:28.250 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000108186 2024-08-19T14:22:28.250 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:28.250 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:28.250 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:22:28.250 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:28.237411+0000", 2024-08-19T14:22:28.251 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0087122520000000002 2024-08-19T14:22:28.251 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:28.251 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:28.251 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:22:28.251 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:28.237770+0000", 2024-08-19T14:22:28.251 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000358477 2024-08-19T14:22:28.251 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:22:28.251 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:22:28.251 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:22:28.251 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:28.251 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:28.251 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25333.0:152617 4.f 4:fe11132f:::benchmark_data_smithi045_71562_object21442:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e378)", 2024-08-19T14:22:28.251 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:22:28.229938+0000", 2024-08-19T14:22:28.251 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.0096170549999999994, 2024-08-19T14:22:28.252 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0098180890000000003, 2024-08-19T14:22:28.252 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:22:28.252 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:22:28.252 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "waiting for sub ops", 2024-08-19T14:22:28.252 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:22:28.252 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25333", 2024-08-19T14:22:28.252 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:22:28.252 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 152617 2024-08-19T14:22:28.252 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:28.252 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:22:28.252 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:28.252 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:22:28.252 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:28.229938+0000", 2024-08-19T14:22:28.252 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:22:28.253 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:28.253 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:28.253 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:22:28.253 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:28.229938+0000", 2024-08-19T14:22:28.253 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:22:28.253 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:28.253 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:28.253 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:22:28.253 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:28.229939+0000", 2024-08-19T14:22:28.253 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2890000000000001e-06 2024-08-19T14:22:28.253 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:28.253 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:28.253 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:22:28.254 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:28.229944+0000", 2024-08-19T14:22:28.254 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1109999999999999e-06 2024-08-19T14:22:28.254 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:28.254 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:28.254 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:22:28.254 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:28.229965+0000", 2024-08-19T14:22:28.254 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.1242000000000001e-05 2024-08-19T14:22:28.254 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:28.254 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:28.254 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:22:28.254 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:28.229973+0000", 2024-08-19T14:22:28.254 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.5900000000000002e-06 2024-08-19T14:22:28.254 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:28.255 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:28.255 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:22:28.255 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:28.230570+0000", 2024-08-19T14:22:28.255 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00059710799999999995 2024-08-19T14:22:28.255 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:28.255 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:28.255 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:22:28.255 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:28.230707+0000", 2024-08-19T14:22:28.255 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013744000000000001 2024-08-19T14:22:28.255 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:28.255 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:28.255 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,11", 2024-08-19T14:22:28.255 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:28.230832+0000", 2024-08-19T14:22:28.256 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012452700000000001 2024-08-19T14:22:28.256 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:22:28.256 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:22:28.256 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:22:28.256 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:28.256 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:28.256 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.25333.0:152619 4.d e378/335 4:bc5482e8:::benchmark_data_smithi045_71562_object21444:head v 378'9496, mlcod=378'9489)", 2024-08-19T14:22:28.256 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:22:28.235165+0000", 2024-08-19T14:22:28.256 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.0043900140000000002, 2024-08-19T14:22:28.257 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0046613699999999997, 2024-08-19T14:22:28.257 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:22:28.257 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:22:28.257 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-08-19T14:22:28.257 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:22:28.257 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:28.257 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:22:28.257 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:28.235165+0000", 2024-08-19T14:22:28.257 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:22:28.257 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:28.257 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:28.257 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:22:28.257 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:28.235165+0000", 2024-08-19T14:22:28.258 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:22:28.258 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:28.258 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:28.258 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:22:28.258 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:28.235165+0000", 2024-08-19T14:22:28.258 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.3300000000000002e-07 2024-08-19T14:22:28.258 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:28.258 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:28.258 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:22:28.258 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:28.235170+0000", 2024-08-19T14:22:28.258 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.775e-06 2024-08-19T14:22:28.258 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:28.258 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:28.259 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:22:28.259 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:28.235179+0000", 2024-08-19T14:22:28.259 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.4599999999999992e-06 2024-08-19T14:22:28.259 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:28.259 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:28.259 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:22:28.259 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:28.235194+0000", 2024-08-19T14:22:28.259 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4679e-05 2024-08-19T14:22:28.259 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:28.259 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:28.259 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:22:28.259 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:28.236513+0000", 2024-08-19T14:22:28.259 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0013186389999999999 2024-08-19T14:22:28.259 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:28.260 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:28.260 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:22:28.260 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:28.236535+0000", 2024-08-19T14:22:28.260 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.2201e-05 2024-08-19T14:22:28.260 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:22:28.260 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:22:28.260 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:22:28.260 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:28.260 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:28.260 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25333.0:152620 4.7 4:e10dc43e:::benchmark_data_smithi045_71562_object21445:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e378)", 2024-08-19T14:22:28.260 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:22:28.237524+0000", 2024-08-19T14:22:28.260 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.0020302530000000001, 2024-08-19T14:22:28.260 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0023690690000000001, 2024-08-19T14:22:28.261 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:22:28.261 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:22:28.261 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "waiting for sub ops", 2024-08-19T14:22:28.261 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:22:28.261 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25333", 2024-08-19T14:22:28.261 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:22:28.261 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 152620 2024-08-19T14:22:28.261 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:28.261 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:22:28.261 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:28.261 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:22:28.261 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:28.237524+0000", 2024-08-19T14:22:28.261 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:22:28.261 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:28.262 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:28.262 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:22:28.262 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:28.237524+0000", 2024-08-19T14:22:28.262 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:22:28.262 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:28.262 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:28.262 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:22:28.262 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:28.237526+0000", 2024-08-19T14:22:28.262 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.753e-06 2024-08-19T14:22:28.262 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:28.262 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:28.262 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:22:28.262 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:28.237532+0000", 2024-08-19T14:22:28.262 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.5450000000000003e-06 2024-08-19T14:22:28.263 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:28.263 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:28.263 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:22:28.263 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:28.237544+0000", 2024-08-19T14:22:28.263 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2573e-05 2024-08-19T14:22:28.263 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:28.263 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:28.263 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:22:28.263 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:28.237552+0000", 2024-08-19T14:22:28.263 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.7980000000000006e-06 2024-08-19T14:22:28.263 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:28.263 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:28.263 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:22:28.263 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:28.237799+0000", 2024-08-19T14:22:28.264 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000246956 2024-08-19T14:22:28.264 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:28.264 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:28.264 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:22:28.264 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:28.237929+0000", 2024-08-19T14:22:28.264 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013023700000000001 2024-08-19T14:22:28.264 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:28.264 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:28.264 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,7", 2024-08-19T14:22:28.264 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:28.238037+0000", 2024-08-19T14:22:28.264 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000107255 2024-08-19T14:22:28.264 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:22:28.264 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:22:28.265 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:22:28.265 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:28.265 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:28.265 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25333.0:152623 4.7 4.b71ed377 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e378)", 2024-08-19T14:22:28.265 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:22:28.237573+0000", 2024-08-19T14:22:28.265 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.0019811849999999999, 2024-08-19T14:22:28.265 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0023935699999999998, 2024-08-19T14:22:28.265 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:22:28.265 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:22:28.265 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "queued for pg", 2024-08-19T14:22:28.265 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:22:28.265 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25333", 2024-08-19T14:22:28.265 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:22:28.266 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 152623 2024-08-19T14:22:28.266 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:28.266 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:22:28.266 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:28.266 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:22:28.266 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:28.237573+0000", 2024-08-19T14:22:28.266 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:22:28.266 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:28.266 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:28.266 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:22:28.266 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:28.237573+0000", 2024-08-19T14:22:28.266 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:22:28.266 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:28.267 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:28.267 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:22:28.267 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:28.237574+0000", 2024-08-19T14:22:28.267 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4200000000000001e-07 2024-08-19T14:22:28.267 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:28.267 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:28.267 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:22:28.267 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:28.237576+0000", 2024-08-19T14:22:28.267 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.5679999999999998e-06 2024-08-19T14:22:28.267 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:28.267 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:28.267 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:22:28.267 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:28.237583+0000", 2024-08-19T14:22:28.267 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.5549999999999999e-06 2024-08-19T14:22:28.268 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:28.268 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:28.268 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:22:28.268 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:28.237588+0000", 2024-08-19T14:22:28.268 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0839999999999999e-06 2024-08-19T14:22:28.268 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:22:28.268 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:22:28.268 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:22:28.268 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:28.268 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:28.268 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.25333.0:152626 4.d e378/335)", 2024-08-19T14:22:28.268 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:22:28.237652+0000", 2024-08-19T14:22:28.268 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.0019029170000000001, 2024-08-19T14:22:28.268 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0023690880000000001, 2024-08-19T14:22:28.269 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:22:28.269 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:22:28.269 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "queued for pg", 2024-08-19T14:22:28.269 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:22:28.269 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:28.269 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:22:28.269 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:28.237652+0000", 2024-08-19T14:22:28.269 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:22:28.269 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:28.269 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:28.269 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:22:28.269 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:28.237652+0000", 2024-08-19T14:22:28.269 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:22:28.269 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:28.270 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:28.270 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:22:28.270 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:28.237652+0000", 2024-08-19T14:22:28.270 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.41e-07 2024-08-19T14:22:28.270 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:28.270 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:28.270 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:22:28.270 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:28.237655+0000", 2024-08-19T14:22:28.270 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.2399999999999999e-06 2024-08-19T14:22:28.270 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:28.270 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:28.270 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:22:28.270 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:28.237661+0000", 2024-08-19T14:22:28.271 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.0820000000000004e-06 2024-08-19T14:22:28.271 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:28.271 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:28.271 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:22:28.271 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:28.237666+0000", 2024-08-19T14:22:28.271 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2320000000000003e-06 2024-08-19T14:22:28.271 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:22:28.271 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:22:28.271 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:22:28.271 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:28.271 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:28.271 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.25333.0:152621 4.9 e378/335 4:956167ea:::benchmark_data_smithi045_71562_object21446:head v 378'9556, mlcod=378'9554)", 2024-08-19T14:22:28.271 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:22:28.238350+0000", 2024-08-19T14:22:28.271 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.001204882, 2024-08-19T14:22:28.272 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.001723231, 2024-08-19T14:22:28.272 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:22:28.272 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:22:28.272 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-08-19T14:22:28.272 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:22:28.272 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:28.272 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:22:28.272 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:28.238350+0000", 2024-08-19T14:22:28.272 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:22:28.272 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:28.272 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:28.272 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:22:28.272 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:28.238350+0000", 2024-08-19T14:22:28.273 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:22:28.273 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:28.273 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:28.273 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:22:28.273 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:28.238350+0000", 2024-08-19T14:22:28.273 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.2900000000000003e-07 2024-08-19T14:22:28.273 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:28.273 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:28.273 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:22:28.273 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:28.238357+0000", 2024-08-19T14:22:28.273 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.331e-06 2024-08-19T14:22:28.273 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:28.274 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:28.274 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:22:28.274 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:28.238371+0000", 2024-08-19T14:22:28.274 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3784000000000001e-05 2024-08-19T14:22:28.274 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:28.274 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:28.274 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:22:28.274 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:28.238386+0000", 2024-08-19T14:22:28.274 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5526e-05 2024-08-19T14:22:28.274 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:28.274 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:28.274 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:22:28.275 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:28.238434+0000", 2024-08-19T14:22:28.275 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.8158999999999997e-05 2024-08-19T14:22:28.275 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:28.275 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:28.275 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:22:28.275 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:28.238453+0000", 2024-08-19T14:22:28.275 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.8533999999999999e-05 2024-08-19T14:22:28.275 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:22:28.275 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:22:28.275 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:22:28.275 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:28.275 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:28.275 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.25333.0:152628 4.3 e378/335 4:c53e5543:::benchmark_data_smithi045_71562_object21453:head v 378'9518, mlcod=378'9516)", 2024-08-19T14:22:28.276 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:22:28.238406+0000", 2024-08-19T14:22:28.276 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.0011485429999999999, 2024-08-19T14:22:28.276 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.001725734, 2024-08-19T14:22:28.276 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:22:28.276 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:22:28.276 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-08-19T14:22:28.276 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:22:28.276 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:28.276 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:22:28.276 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:28.238406+0000", 2024-08-19T14:22:28.276 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:22:28.276 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:28.276 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:28.276 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:22:28.276 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:28.238406+0000", 2024-08-19T14:22:28.277 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:22:28.277 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:28.277 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:28.277 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:22:28.277 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:28.238406+0000", 2024-08-19T14:22:28.277 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.0600000000000001e-07 2024-08-19T14:22:28.277 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:28.277 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:28.277 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:22:28.277 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:28.238410+0000", 2024-08-19T14:22:28.277 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.8139999999999998e-06 2024-08-19T14:22:28.277 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:28.277 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:28.278 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:22:28.278 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:28.238417+0000", 2024-08-19T14:22:28.278 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.4189999999999999e-06 2024-08-19T14:22:28.278 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:28.278 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:28.278 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:22:28.278 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:28.238421+0000", 2024-08-19T14:22:28.278 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.5310000000000003e-06 2024-08-19T14:22:28.278 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:28.279 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:28.279 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:22:28.279 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:28.238682+0000", 2024-08-19T14:22:28.279 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00026089100000000002 2024-08-19T14:22:28.279 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:28.279 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:28.279 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:22:28.279 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:28.238700+0000", 2024-08-19T14:22:28.279 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.8371000000000001e-05 2024-08-19T14:22:28.279 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:22:28.279 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:22:28.279 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:22:28.279 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:22:28.279 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-08-19T14:22:28.280 INFO:teuthology.orchestra.run.smithi031.stdout: "num_ops": 9 2024-08-19T14:22:28.280 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:22:28.316 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:22:28.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:28.366 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:28.366 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:22:28.467 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:22:28.494 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:28.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:28.495 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:22:28.596 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:22:28.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:28.634 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:28.634 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:22:28.735 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:22:28.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:28.765 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:28.765 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:22:28.866 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:22:28.892 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:28.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:28.893 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:22:28.995 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:22:29.020 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:29.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:29.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:22:29.122 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:22:29.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:29.151 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:29.151 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:22:29.234 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-08-19T14:22:29.252 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:22:29.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:29.277 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:29.277 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:22:29.278 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:22:29.354 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:22:29.378 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:22:29.422 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:29.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:29.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:22:29.524 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:22:29.548 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:29.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:29.550 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:22:29.651 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:22:29.678 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-19T14:22:29.687 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:29.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:29.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:22:29.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:29 smithi031 ceph-mon[16336]: pgmap v855: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 6.9 GiB data, 25 GiB used, 780 GiB / 805 GiB avail; 0 B/s wr, 757 op/s; 1772/339042 objects misplaced (0.523%); 1.7 MiB/s, 27 objects/s recovering 2024-08-19T14:22:29.789 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:22:29.836 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:29.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:29.837 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:22:29.938 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:22:29.964 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:29.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:29.965 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:22:30.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:29 smithi032 ceph-mon[20004]: pgmap v855: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 6.9 GiB data, 25 GiB used, 780 GiB / 805 GiB avail; 0 B/s wr, 757 op/s; 1772/339042 objects misplaced (0.523%); 1.7 MiB/s, 27 objects/s recovering 2024-08-19T14:22:30.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:29 smithi045 ceph-mon[20143]: pgmap v855: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 6.9 GiB data, 25 GiB used, 780 GiB / 805 GiB avail; 0 B/s wr, 757 op/s; 1772/339042 objects misplaced (0.523%); 1.7 MiB/s, 27 objects/s recovering 2024-08-19T14:22:30.065 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:22:30.090 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:30.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:30.091 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:22:30.192 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:22:30.223 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:30.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:30.224 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:22:30.325 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:22:30.364 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:30.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:30.365 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:22:30.466 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:22:30.491 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:30.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:30.492 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:22:30.593 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:22:30.619 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:30.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:30.620 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:22:30.721 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:22:30.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:30.761 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:30.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:22:30.807 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:22:30.807 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:22:30.807 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:22:30.807 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:22:30.807 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:22:30.808 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:22:30.808 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:22:30.808 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:22:30.808 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:22:30.808 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:22:30.808 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:22:30.808 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:22:30.808 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:22:30.808 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:22:30.809 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:22:30.809 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:22:30.809 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:22:30.809 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:22:30.809 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:22:30.809 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:22:30.809 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:22:30.809 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:22:30.809 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:22:30.809 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:22:30.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:30 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/20898575' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:22:30.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:30 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2304397335' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:22:30.862 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:22:30.886 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:30.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:30.887 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:22:30.988 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:22:31.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:30 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/20898575' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:22:31.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:30 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2304397335' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:22:31.013 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:31.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:31.014 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:22:31.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:30 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/20898575' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:22:31.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:30 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2304397335' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:22:31.115 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:22:31.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:31.141 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:31.141 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:22:31.242 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:22:31.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:31.275 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:31.275 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:22:31.378 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:22:31.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:31.406 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:31.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:22:31.508 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:22:31.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:31.536 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:31.536 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:22:31.636 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:22:31.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:31.669 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:31.669 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:22:31.770 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:22:31.808 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:31.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:31.809 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:22:31.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:31 smithi031 ceph-mon[16336]: pgmap v856: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 6.7 GiB data, 24 GiB used, 780 GiB / 805 GiB avail; 0 B/s wr, 916 op/s; 1594/330279 objects misplaced (0.483%); 2.5 MiB/s, 40 objects/s recovering 2024-08-19T14:22:31.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:31 smithi031 ceph-mon[16336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:22:31.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:31 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2304397335' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:22:31.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:31 smithi031 ceph-mon[16336]: osdmap e379: 12 total, 12 up, 9 in 2024-08-19T14:22:31.910 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:22:31.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:31.947 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:31.947 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:22:31.990 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:31 smithi032 ceph-mon[20004]: pgmap v856: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 6.7 GiB data, 24 GiB used, 780 GiB / 805 GiB avail; 0 B/s wr, 916 op/s; 1594/330279 objects misplaced (0.483%); 2.5 MiB/s, 40 objects/s recovering 2024-08-19T14:22:31.990 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:31 smithi032 ceph-mon[20004]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:22:31.990 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:31 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2304397335' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:22:31.990 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:31 smithi032 ceph-mon[20004]: osdmap e379: 12 total, 12 up, 9 in 2024-08-19T14:22:32.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:31 smithi045 ceph-mon[20143]: pgmap v856: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 6.7 GiB data, 24 GiB used, 780 GiB / 805 GiB avail; 0 B/s wr, 916 op/s; 1594/330279 objects misplaced (0.483%); 2.5 MiB/s, 40 objects/s recovering 2024-08-19T14:22:32.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:31 smithi045 ceph-mon[20143]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:22:32.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:31 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2304397335' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:22:32.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:31 smithi045 ceph-mon[20143]: osdmap e379: 12 total, 12 up, 9 in 2024-08-19T14:22:32.048 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:22:32.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:32.084 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:32.084 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:22:32.185 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:22:32.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:32.210 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:32.210 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:22:32.268 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is set 2024-08-19T14:22:32.311 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:22:32.346 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:32.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:32.347 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:22:32.448 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:22:32.482 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:32.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:32.483 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:22:32.584 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:22:32.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:32.610 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:32.610 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:22:32.711 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:22:32.742 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:32.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:32.743 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:22:32.844 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:22:32.873 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:32.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:32.874 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:22:32.975 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:22:32.976 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:22:33.038 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:33.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:33.040 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:22:33.141 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:22:33.167 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:33.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:33.168 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:22:33.269 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:22:33.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:33.322 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:33.322 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:22:33.423 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:22:33.445 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:33.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:33.446 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:22:33.547 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:22:33.549 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:22:33.571 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:33.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:33.572 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:22:33.596 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:22:33.596 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:22:33.673 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:22:33.699 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:33.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:33.700 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:22:33.802 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:22:33.826 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:33.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:33.827 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:22:33.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:33 smithi031 ceph-mon[16336]: pgmap v858: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 6.6 GiB data, 24 GiB used, 781 GiB / 805 GiB avail; 0 B/s wr, 1.02k op/s; 1594/325191 objects misplaced (0.490%); 2.8 MiB/s, 45 objects/s recovering 2024-08-19T14:22:33.927 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:22:33.957 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:33.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:33.958 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:22:34.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:33 smithi032 ceph-mon[20004]: pgmap v858: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 6.6 GiB data, 24 GiB used, 781 GiB / 805 GiB avail; 0 B/s wr, 1.02k op/s; 1594/325191 objects misplaced (0.490%); 2.8 MiB/s, 45 objects/s recovering 2024-08-19T14:22:34.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:33 smithi045 ceph-mon[20143]: pgmap v858: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 6.6 GiB data, 24 GiB used, 781 GiB / 805 GiB avail; 0 B/s wr, 1.02k op/s; 1594/325191 objects misplaced (0.490%); 2.8 MiB/s, 45 objects/s recovering 2024-08-19T14:22:34.060 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:22:34.087 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:34.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:34.088 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:22:34.162 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:22:34.189 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:22:34.252 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:34.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:34.253 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:22:34.293 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-08-19T14:22:34.354 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:22:34.380 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:34.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:34.381 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:22:34.482 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:22:34.509 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:34.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:34.510 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:22:34.611 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:22:34.640 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:34.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:34.641 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:22:34.742 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:22:34.766 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:34.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:34.767 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:22:34.855 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:22:34.868 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:22:34.893 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:34.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:34.894 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:22:34.929 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:22:34.989 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-19T14:22:34.994 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:22:35.075 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:35.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:35.076 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:22:35.177 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:22:35.199 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:22:35.199 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-08-19T14:22:35.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:35.201 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:35.202 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:22:35.303 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:22:35.330 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:35.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:35.331 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:22:35.432 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:22:35.459 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:35.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:35.460 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:22:35.561 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:22:35.585 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:35.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:35.586 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:22:35.687 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:22:35.715 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:35.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:35.716 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:22:35.755 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:35 smithi031 ceph-mon[16336]: pgmap v859: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 6.5 GiB data, 24 GiB used, 781 GiB / 805 GiB avail; 0 B/s wr, 902 op/s; 1594/321762 objects misplaced (0.495%); 2.5 MiB/s, 39 objects/s recovering 2024-08-19T14:22:35.816 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:22:35.844 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:35.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:35.845 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:22:35.946 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:22:35.970 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:35.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:35.971 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:22:36.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:35 smithi032 ceph-mon[20004]: pgmap v859: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 6.5 GiB data, 24 GiB used, 781 GiB / 805 GiB avail; 0 B/s wr, 902 op/s; 1594/321762 objects misplaced (0.495%); 2.5 MiB/s, 39 objects/s recovering 2024-08-19T14:22:36.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:35 smithi045 ceph-mon[20143]: pgmap v859: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 6.5 GiB data, 24 GiB used, 781 GiB / 805 GiB avail; 0 B/s wr, 902 op/s; 1594/321762 objects misplaced (0.495%); 2.5 MiB/s, 39 objects/s recovering 2024-08-19T14:22:36.073 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:22:36.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:36.097 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:36.097 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:22:36.198 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:22:36.219 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:36.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:36.220 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:22:36.321 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:22:36.343 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:36.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:36.344 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:22:36.445 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:22:36.483 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:36.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:36.485 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:22:36.586 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:22:36.620 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:36.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:36.621 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:22:36.722 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:22:36.747 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:36.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:36.748 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:22:36.849 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:22:36.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:36 smithi031 ceph-mon[16336]: from='client.15849 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:22:36.875 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:36.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:36.876 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:22:36.905 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:36 smithi045 ceph-mon[20143]: from='client.15849 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:22:36.977 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:22:37.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:36 smithi032 ceph-mon[20004]: from='client.15849 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:22:37.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:37.014 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:37.015 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:22:37.116 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:22:37.137 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:37.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:37.138 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:22:37.239 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:22:37.260 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:37.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:37.261 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:22:37.362 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:22:37.400 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:37.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:37.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:22:37.477 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:22:37.477 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:22:37.502 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:22:37.529 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:37.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:37.530 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:22:37.631 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:22:37.655 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:37.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:37.656 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:22:37.757 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:22:37.781 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:37.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:37.783 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:22:37.825 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:37 smithi031 ceph-mon[16336]: pgmap v860: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 6.3 GiB data, 23 GiB used, 781 GiB / 805 GiB avail; 0 B/s wr, 936 op/s; 1399/311772 objects misplaced (0.449%); 2.3 MiB/s, 37 objects/s recovering 2024-08-19T14:22:37.883 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:22:37.905 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:37.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:37.906 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:22:38.007 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:22:38.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:37 smithi032 ceph-mon[20004]: pgmap v860: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 6.3 GiB data, 23 GiB used, 781 GiB / 805 GiB avail; 0 B/s wr, 936 op/s; 1399/311772 objects misplaced (0.449%); 2.3 MiB/s, 37 objects/s recovering 2024-08-19T14:22:38.032 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:38.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:38.033 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:22:38.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:37 smithi045 ceph-mon[20143]: pgmap v860: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 6.3 GiB data, 23 GiB used, 781 GiB / 805 GiB avail; 0 B/s wr, 936 op/s; 1399/311772 objects misplaced (0.449%); 2.3 MiB/s, 37 objects/s recovering 2024-08-19T14:22:38.134 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:22:38.158 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:38.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:38.159 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:22:38.260 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:22:38.304 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:38.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:38.305 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:22:38.406 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:22:38.435 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:38.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:38.436 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:22:38.536 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:22:38.562 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:38.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:38.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:22:38.664 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:22:38.679 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:38 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3513791398' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:22:38.695 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:38.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:38.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:22:38.717 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-08-19T14:22:38.797 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:22:38.813 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:22:38.813 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:22:38.813 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:22:38.813 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:22:38.813 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:22:38.813 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:22:38.813 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:22:38.813 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:22:38.814 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:22:38.814 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:22:38.814 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:22:38.814 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:22:38.814 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:22:38.814 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:22:38.814 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:22:38.814 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:22:38.815 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:22:38.815 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:22:38.815 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:22:38.815 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:22:38.815 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:22:38.815 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:22:38.815 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:22:38.815 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:22:38.819 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:38.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:38.820 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:22:38.921 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:22:38.949 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:38.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:38.950 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:22:39.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:38 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3513791398' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:22:39.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:38 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3513791398' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:22:39.050 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:22:39.078 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:39.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:39.079 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:22:39.180 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:22:39.210 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:39.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:39.211 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:22:39.312 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:22:39.335 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:39.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:39.336 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:22:39.437 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:22:39.464 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:39.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:39.465 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:22:39.508 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:22:39.566 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:22:39.592 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:39.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:39.593 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:22:39.694 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:22:39.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:39.727 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:39.728 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:22:39.829 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:22:39.859 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:39.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:39.861 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:22:39.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:39 smithi031 ceph-mon[16336]: pgmap v861: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 6.3 GiB data, 23 GiB used, 782 GiB / 805 GiB avail; 0 B/s wr, 936 op/s; 1399/310926 objects misplaced (0.450%); 2.3 MiB/s, 37 objects/s recovering 2024-08-19T14:22:39.962 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":859,"stamp":"2024-08-19T14:22:33.916025+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7029391928,"num_objects":107254,"num_object_clones":0,"num_object_copies":321762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1594,"num_objects_unfound":0,"num_objects_dirty":107254,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":155094,"num_write_kb":8395009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2546,"num_bytes_recovered":166854656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159403,"ondisk_log_size":159403,"up":141,"acting":140,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":142,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":24713180,"kb_used_data":20788236,"kb_used_omap":14,"kb_used_meta":3924785,"kb_avail":818993188,"statfs":{"total":863955320832,"available":838649024512,"internally_reserved":0,"allocated":21287153664,"data_stored":21271441400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14353,"internal_metadata":4018980847},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,7,3,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-591462400,"num_objects":-9025,"num_object_clones":0,"num_object_copies":-27075,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-396,"num_objects_unfound":0,"num_objects_dirty":-9025,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9025,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":396,"num_bytes_recovered":25952256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.004020"},"pg_stats":[{"pgid":"4.5","version":"379'9747","reported_seq":9930,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-08-19T14:22:30.822051+0000","last_change":"2024-08-19T14:22:14.211513+0000","last_active":"2024-08-19T14:22:30.822051+0000","last_peered":"2024-08-19T14:22:30.822051+0000","last_clean":"2024-08-19T14:22:30.822051+0000","last_became_active":"2024-08-19T14:20:29.267549+0000","last_became_peered":"2024-08-19T14:20:29.267549+0000","last_unstale":"2024-08-19T14:22:30.822051+0000","last_undegraded":"2024-08-19T14:22:30.822051+0000","last_fullsized":"2024-08-19T14:22:30.822051+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"375'8747","last_scrub_stamp":"2024-08-19T14:22:14.211361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:22:14.211361+0000","objects_scrubbed":7717,"log_size":9747,"log_dups_size":0,"ondisk_log_size":9747,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:26.018227+0000","scrub_duration":1909,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436011008,"num_objects":6653,"num_object_clones":0,"num_object_copies":19959,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9747,"num_write_kb":524800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,0],"acting":[11,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"70'139","reported_seq":974,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-08-19T14:22:30.577488+0000","last_change":"2024-08-19T14:19:58.072227+0000","last_active":"2024-08-19T14:22:30.577488+0000","last_peered":"2024-08-19T14:22:30.577488+0000","last_clean":"2024-08-19T14:22:30.577488+0000","last_became_active":"2024-08-19T14:18:13.807133+0000","last_became_peered":"2024-08-19T14:18:13.807133+0000","last_unstale":"2024-08-19T14:22:30.577488+0000","last_undegraded":"2024-08-19T14:22:30.577488+0000","last_fullsized":"2024-08-19T14:22:30.577488+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:19:58.072083+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:19:58.072083+0000","last_clean_scrub_stamp":"2024-08-19T14:19:58.072083+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,9],"acting":[7,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"378'9803","reported_seq":14994,"reported_epoch":378,"state":"active+recovering+undersized+remapped","last_fresh":"2024-08-19T14:22:29.367935+0000","last_change":"2024-08-19T14:21:07.239135+0000","last_active":"2024-08-19T14:22:29.367935+0000","last_peered":"2024-08-19T14:22:29.367935+0000","last_clean":"2024-08-19T14:21:04.784061+0000","last_became_active":"2024-08-19T14:21:07.213050+0000","last_became_peered":"2024-08-19T14:21:07.213050+0000","last_unstale":"2024-08-19T14:22:29.367935+0000","last_undegraded":"2024-08-19T14:22:29.367935+0000","last_fullsized":"2024-08-19T14:21:06.805412+0000","mapping_epoch":354,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":9803,"log_dups_size":0,"ondisk_log_size":9803,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":450560000,"num_objects":6875,"num_object_clones":0,"num_object_copies":20625,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1594,"num_objects_unfound":0,"num_objects_dirty":6875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9803,"num_write_kb":533696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2546,"num_bytes_recovered":166854656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,7],"acting":[1,7],"avail_no_missing":["1","5","7"],"object_location_counts":[{"shards":"1,5,7","objects":1594},{"shards":"1,7","objects":5281}],"blocked_by":[],"up_primary":9,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"70'139","reported_seq":871,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-08-19T14:22:23.133493+0000","last_change":"2024-08-19T14:20:03.714917+0000","last_active":"2024-08-19T14:22:23.133493+0000","last_peered":"2024-08-19T14:22:23.133493+0000","last_clean":"2024-08-19T14:22:23.133493+0000","last_became_active":"2024-08-19T14:18:13.804721+0000","last_became_peered":"2024-08-19T14:18:13.804721+0000","last_unstale":"2024-08-19T14:22:23.133493+0000","last_undegraded":"2024-08-19T14:22:23.133493+0000","last_fullsized":"2024-08-19T14:22:23.133493+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:03.714818+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:03.714818+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"378'9605","reported_seq":9699,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-08-19T14:22:29.297256+0000","last_change":"2024-08-19T14:20:29.268110+0000","last_active":"2024-08-19T14:22:29.297256+0000","last_peered":"2024-08-19T14:22:29.297256+0000","last_clean":"2024-08-19T14:22:29.297256+0000","last_became_active":"2024-08-19T14:20:29.267811+0000","last_became_peered":"2024-08-19T14:20:29.267811+0000","last_unstale":"2024-08-19T14:22:29.297256+0000","last_undegraded":"2024-08-19T14:22:29.297256+0000","last_fullsized":"2024-08-19T14:22:29.297256+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":9605,"log_dups_size":0,"ondisk_log_size":9605,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":448331776,"num_objects":6841,"num_object_clones":0,"num_object_copies":20523,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6841,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9605,"num_write_kb":526272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"70'139","reported_seq":821,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-08-19T14:22:30.576944+0000","last_change":"2024-08-19T14:20:50.742676+0000","last_active":"2024-08-19T14:22:30.576944+0000","last_peered":"2024-08-19T14:22:30.576944+0000","last_clean":"2024-08-19T14:22:30.576944+0000","last_became_active":"2024-08-19T14:20:50.742354+0000","last_became_peered":"2024-08-19T14:20:50.742354+0000","last_unstale":"2024-08-19T14:22:30.576944+0000","last_undegraded":"2024-08-19T14:22:30.576944+0000","last_fullsized":"2024-08-19T14:22:30.576944+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:19:59.302931+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:19:59.302931+0000","last_clean_scrub_stamp":"2024-08-19T14:19:59.302931+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":114,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"378'9576","reported_seq":9672,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-08-19T14:22:29.353154+0000","last_change":"2024-08-19T14:20:29.267640+0000","last_active":"2024-08-19T14:22:29.353154+0000","last_peered":"2024-08-19T14:22:29.353154+0000","last_clean":"2024-08-19T14:22:29.353154+0000","last_became_active":"2024-08-19T14:20:29.267331+0000","last_became_peered":"2024-08-19T14:20:29.267331+0000","last_unstale":"2024-08-19T14:22:29.353154+0000","last_undegraded":"2024-08-19T14:22:29.353154+0000","last_fullsized":"2024-08-19T14:22:29.353154+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":9576,"log_dups_size":0,"ondisk_log_size":9576,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":446300160,"num_objects":6810,"num_object_clones":0,"num_object_copies":20430,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6810,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9576,"num_write_kb":524352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"70'139","reported_seq":870,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-08-19T14:22:30.581848+0000","last_change":"2024-08-19T14:20:09.507422+0000","last_active":"2024-08-19T14:22:30.581848+0000","last_peered":"2024-08-19T14:22:30.581848+0000","last_clean":"2024-08-19T14:22:30.581848+0000","last_became_active":"2024-08-19T14:18:13.808505+0000","last_became_peered":"2024-08-19T14:18:13.808505+0000","last_unstale":"2024-08-19T14:22:30.581848+0000","last_undegraded":"2024-08-19T14:22:30.581848+0000","last_fullsized":"2024-08-19T14:22:30.581848+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:09.507256+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:09.507256+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,11],"acting":[9,0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"379'9932","reported_seq":10031,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-08-19T14:22:33.417911+0000","last_change":"2024-08-19T14:20:29.268766+0000","last_active":"2024-08-19T14:22:33.417911+0000","last_peered":"2024-08-19T14:22:33.417911+0000","last_clean":"2024-08-19T14:22:33.417911+0000","last_became_active":"2024-08-19T14:20:29.268443+0000","last_became_peered":"2024-08-19T14:20:29.268443+0000","last_unstale":"2024-08-19T14:22:33.417911+0000","last_undegraded":"2024-08-19T14:22:33.417911+0000","last_fullsized":"2024-08-19T14:22:33.417911+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":9932,"log_dups_size":0,"ondisk_log_size":9932,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":431226880,"num_objects":6580,"num_object_clones":0,"num_object_copies":19740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6580,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9932,"num_write_kb":528384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,11],"acting":[9,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.7","version":"70'139","reported_seq":519,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-08-19T14:22:23.132787+0000","last_change":"2024-08-19T14:21:46.057764+0000","last_active":"2024-08-19T14:22:23.132787+0000","last_peered":"2024-08-19T14:22:23.132787+0000","last_clean":"2024-08-19T14:22:23.132787+0000","last_became_active":"2024-08-19T14:20:28.204384+0000","last_became_peered":"2024-08-19T14:20:28.204384+0000","last_unstale":"2024-08-19T14:22:23.132787+0000","last_undegraded":"2024-08-19T14:22:23.132787+0000","last_fullsized":"2024-08-19T14:22:23.132787+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:46.057624+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:21:46.057624+0000","last_clean_scrub_stamp":"2024-08-19T14:21:46.057624+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:02.689278+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"379'9678","reported_seq":9922,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-08-19T14:22:30.881560+0000","last_change":"2024-08-19T14:22:07.840154+0000","last_active":"2024-08-19T14:22:30.881560+0000","last_peered":"2024-08-19T14:22:30.881560+0000","last_clean":"2024-08-19T14:22:30.881560+0000","last_became_active":"2024-08-19T14:20:29.267379+0000","last_became_peered":"2024-08-19T14:20:29.267379+0000","last_unstale":"2024-08-19T14:22:30.881560+0000","last_undegraded":"2024-08-19T14:22:30.881560+0000","last_fullsized":"2024-08-19T14:22:30.881560+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"373'8338","last_scrub_stamp":"2024-08-19T14:22:07.840021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:22:07.840021+0000","objects_scrubbed":8073,"log_size":9678,"log_dups_size":0,"ondisk_log_size":9678,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:13.679336+0000","scrub_duration":2409,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":437256192,"num_objects":6672,"num_object_clones":0,"num_object_copies":20016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9678,"num_write_kb":523200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,1],"acting":[11,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.6","version":"70'139","reported_seq":874,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-08-19T14:22:30.585373+0000","last_change":"2024-08-19T14:19:55.512078+0000","last_active":"2024-08-19T14:22:30.585373+0000","last_peered":"2024-08-19T14:22:30.585373+0000","last_clean":"2024-08-19T14:22:30.585373+0000","last_became_active":"2024-08-19T14:18:26.874902+0000","last_became_peered":"2024-08-19T14:18:26.874902+0000","last_unstale":"2024-08-19T14:22:30.585373+0000","last_undegraded":"2024-08-19T14:22:30.585373+0000","last_fullsized":"2024-08-19T14:22:30.585373+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:19:55.511974+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:27.848870+0000","last_clean_scrub_stamp":"2024-08-19T14:19:55.511974+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,4],"acting":[9,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"378'9522","reported_seq":9707,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-08-19T14:22:29.539717+0000","last_change":"2024-08-19T14:22:08.536948+0000","last_active":"2024-08-19T14:22:29.539717+0000","last_peered":"2024-08-19T14:22:29.539717+0000","last_clean":"2024-08-19T14:22:29.539717+0000","last_became_active":"2024-08-19T14:20:29.267360+0000","last_became_peered":"2024-08-19T14:20:29.267360+0000","last_unstale":"2024-08-19T14:22:29.539717+0000","last_undegraded":"2024-08-19T14:22:29.539717+0000","last_fullsized":"2024-08-19T14:22:29.539717+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"374'8331","last_scrub_stamp":"2024-08-19T14:22:08.536620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:22:08.536620+0000","objects_scrubbed":7995,"log_size":9522,"log_dups_size":0,"ondisk_log_size":9522,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:19.976024+0000","scrub_duration":2478,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":440270848,"num_objects":6718,"num_object_clones":0,"num_object_copies":20154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6718,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9522,"num_write_kb":519680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"70'139","reported_seq":695,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-08-19T14:22:30.575839+0000","last_change":"2024-08-19T14:22:01.354585+0000","last_active":"2024-08-19T14:22:30.575839+0000","last_peered":"2024-08-19T14:22:30.575839+0000","last_clean":"2024-08-19T14:22:30.575839+0000","last_became_active":"2024-08-19T14:18:26.881435+0000","last_became_peered":"2024-08-19T14:18:26.881435+0000","last_unstale":"2024-08-19T14:22:30.575839+0000","last_undegraded":"2024-08-19T14:22:30.575839+0000","last_fullsized":"2024-08-19T14:22:30.575839+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:22:01.354343+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:22:01.354343+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:30.081548+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,4],"acting":[11,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.1","version":"379'9772","reported_seq":9950,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-08-19T14:22:30.870653+0000","last_change":"2024-08-19T14:22:20.917940+0000","last_active":"2024-08-19T14:22:30.870653+0000","last_peered":"2024-08-19T14:22:30.870653+0000","last_clean":"2024-08-19T14:22:30.870653+0000","last_became_active":"2024-08-19T14:20:29.268465+0000","last_became_peered":"2024-08-19T14:20:29.268465+0000","last_unstale":"2024-08-19T14:22:30.870653+0000","last_undegraded":"2024-08-19T14:22:30.870653+0000","last_fullsized":"2024-08-19T14:22:30.870653+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"377'9214","last_scrub_stamp":"2024-08-19T14:22:20.917842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:22:20.917842+0000","objects_scrubbed":7332,"log_size":9772,"log_dups_size":0,"ondisk_log_size":9772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:39.609654+0000","scrub_duration":1625,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":441057280,"num_objects":6730,"num_object_clones":0,"num_object_copies":20190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9772,"num_write_kb":528064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.4","version":"70'139","reported_seq":920,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-08-19T14:22:30.575768+0000","last_change":"2024-08-19T14:21:39.314063+0000","last_active":"2024-08-19T14:22:30.575768+0000","last_peered":"2024-08-19T14:22:30.575768+0000","last_clean":"2024-08-19T14:22:30.575768+0000","last_became_active":"2024-08-19T14:18:13.806013+0000","last_became_peered":"2024-08-19T14:18:13.806013+0000","last_unstale":"2024-08-19T14:22:30.575768+0000","last_undegraded":"2024-08-19T14:22:30.575768+0000","last_fullsized":"2024-08-19T14:22:30.575768+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:39.313941+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:21:39.313941+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:22:45.184594+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.f","version":"378'9520","reported_seq":9616,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-08-19T14:22:29.355597+0000","last_change":"2024-08-19T14:20:29.212298+0000","last_active":"2024-08-19T14:22:29.355597+0000","last_peered":"2024-08-19T14:22:29.355597+0000","last_clean":"2024-08-19T14:22:29.355597+0000","last_became_active":"2024-08-19T14:20:29.212003+0000","last_became_peered":"2024-08-19T14:20:29.212003+0000","last_unstale":"2024-08-19T14:22:29.355597+0000","last_undegraded":"2024-08-19T14:22:29.355597+0000","last_fullsized":"2024-08-19T14:22:29.355597+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":9520,"log_dups_size":0,"ondisk_log_size":9520,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":444465152,"num_objects":6782,"num_object_clones":0,"num_object_copies":20346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9520,"num_write_kb":521664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,11],"acting":[1,3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"70'139","reported_seq":882,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-08-19T14:22:30.581855+0000","last_change":"2024-08-19T14:20:12.570453+0000","last_active":"2024-08-19T14:22:30.581855+0000","last_peered":"2024-08-19T14:22:30.581855+0000","last_clean":"2024-08-19T14:22:30.581855+0000","last_became_active":"2024-08-19T14:18:26.875006+0000","last_became_peered":"2024-08-19T14:18:26.875006+0000","last_unstale":"2024-08-19T14:22:30.581855+0000","last_undegraded":"2024-08-19T14:22:30.581855+0000","last_fullsized":"2024-08-19T14:22:30.581855+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:12.570352+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:28.864870+0000","last_clean_scrub_stamp":"2024-08-19T14:20:12.570352+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,0],"acting":[9,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.c","version":"379'9755","reported_seq":9941,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-08-19T14:22:33.200331+0000","last_change":"2024-08-19T14:22:07.487434+0000","last_active":"2024-08-19T14:22:33.200331+0000","last_peered":"2024-08-19T14:22:33.200331+0000","last_clean":"2024-08-19T14:22:33.200331+0000","last_became_active":"2024-08-19T14:20:29.266373+0000","last_became_peered":"2024-08-19T14:20:29.266373+0000","last_unstale":"2024-08-19T14:22:33.200331+0000","last_undegraded":"2024-08-19T14:22:33.200331+0000","last_fullsized":"2024-08-19T14:22:33.200331+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"373'8336","last_scrub_stamp":"2024-08-19T14:22:07.487270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:22:07.487270+0000","objects_scrubbed":8079,"log_size":9755,"log_dups_size":0,"ondisk_log_size":9755,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:14.053549+0000","scrub_duration":1972,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":432996352,"num_objects":6607,"num_object_clones":0,"num_object_copies":19821,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9755,"num_write_kb":523584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,8],"acting":[4,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"70'139","reported_seq":997,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-08-19T14:22:30.576951+0000","last_change":"2024-08-19T14:19:56.106047+0000","last_active":"2024-08-19T14:22:30.576951+0000","last_peered":"2024-08-19T14:22:30.576951+0000","last_clean":"2024-08-19T14:22:30.576951+0000","last_became_active":"2024-08-19T14:18:26.887657+0000","last_became_peered":"2024-08-19T14:18:26.887657+0000","last_unstale":"2024-08-19T14:22:30.576951+0000","last_undegraded":"2024-08-19T14:22:30.576951+0000","last_fullsized":"2024-08-19T14:22:30.576951+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:19:56.105880+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:19:56.105880+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"379'9777","reported_seq":9965,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-08-19T14:22:33.215121+0000","last_change":"2024-08-19T14:22:08.529439+0000","last_active":"2024-08-19T14:22:33.215121+0000","last_peered":"2024-08-19T14:22:33.215121+0000","last_clean":"2024-08-19T14:22:33.215121+0000","last_became_active":"2024-08-19T14:20:29.208599+0000","last_became_peered":"2024-08-19T14:20:29.208599+0000","last_unstale":"2024-08-19T14:22:33.215121+0000","last_undegraded":"2024-08-19T14:22:33.215121+0000","last_fullsized":"2024-08-19T14:22:33.215121+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"374'8320","last_scrub_stamp":"2024-08-19T14:22:08.529109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:22:08.529109+0000","objects_scrubbed":7983,"log_size":9777,"log_dups_size":0,"ondisk_log_size":9777,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:30.745931+0000","scrub_duration":1962,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":424149016,"num_objects":6473,"num_object_clones":0,"num_object_copies":19419,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6473,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9777,"num_write_kb":519937,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,1],"acting":[4,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"70'139","reported_seq":894,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-08-19T14:22:30.573065+0000","last_change":"2024-08-19T14:21:37.951800+0000","last_active":"2024-08-19T14:22:30.573065+0000","last_peered":"2024-08-19T14:22:30.573065+0000","last_clean":"2024-08-19T14:22:30.573065+0000","last_became_active":"2024-08-19T14:18:26.887277+0000","last_became_peered":"2024-08-19T14:18:26.887277+0000","last_unstale":"2024-08-19T14:22:30.573065+0000","last_undegraded":"2024-08-19T14:22:30.573065+0000","last_fullsized":"2024-08-19T14:22:30.573065+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:37.951694+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:21:37.951694+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:03.860882+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,8],"acting":[5,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"379'9688","reported_seq":9870,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-08-19T14:22:33.220398+0000","last_change":"2024-08-19T14:22:13.656515+0000","last_active":"2024-08-19T14:22:33.220398+0000","last_peered":"2024-08-19T14:22:33.220398+0000","last_clean":"2024-08-19T14:22:33.220398+0000","last_became_active":"2024-08-19T14:20:29.208748+0000","last_became_peered":"2024-08-19T14:20:29.208748+0000","last_unstale":"2024-08-19T14:22:33.220398+0000","last_undegraded":"2024-08-19T14:22:33.220398+0000","last_fullsized":"2024-08-19T14:22:33.220398+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"374'8571","last_scrub_stamp":"2024-08-19T14:22:13.656335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:22:13.656335+0000","objects_scrubbed":7643,"log_size":9688,"log_dups_size":0,"ondisk_log_size":9688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:30.767371+0000","scrub_duration":3043,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":422707200,"num_objects":6450,"num_object_clones":0,"num_object_copies":19350,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9688,"num_write_kb":516416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"70'139","reported_seq":895,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-08-19T14:22:30.575717+0000","last_change":"2024-08-19T14:22:02.413212+0000","last_active":"2024-08-19T14:22:30.575717+0000","last_peered":"2024-08-19T14:22:30.575717+0000","last_clean":"2024-08-19T14:22:30.575717+0000","last_became_active":"2024-08-19T14:18:26.875179+0000","last_became_peered":"2024-08-19T14:18:26.875179+0000","last_unstale":"2024-08-19T14:22:30.575717+0000","last_undegraded":"2024-08-19T14:22:30.575717+0000","last_fullsized":"2024-08-19T14:22:30.575717+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:22:02.413091+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:22:02.413091+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:23.082273+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.b","version":"379'9672","reported_seq":9845,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-08-19T14:22:31.233478+0000","last_change":"2024-08-19T14:21:39.089676+0000","last_active":"2024-08-19T14:22:31.233478+0000","last_peered":"2024-08-19T14:22:31.233478+0000","last_clean":"2024-08-19T14:22:31.233478+0000","last_became_active":"2024-08-19T14:20:29.207372+0000","last_became_peered":"2024-08-19T14:20:29.207372+0000","last_unstale":"2024-08-19T14:22:31.233478+0000","last_undegraded":"2024-08-19T14:22:31.233478+0000","last_fullsized":"2024-08-19T14:22:31.233478+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"363'6615","last_scrub_stamp":"2024-08-19T14:21:39.089400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:21:39.089400+0000","objects_scrubbed":6552,"log_size":9672,"log_dups_size":0,"ondisk_log_size":9672,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:05.632119+0000","scrub_duration":1907,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":437125120,"num_objects":6670,"num_object_clones":0,"num_object_copies":20010,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6670,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9672,"num_write_kb":522944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"70'139","reported_seq":549,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-08-19T14:22:30.576109+0000","last_change":"2024-08-19T14:21:44.434481+0000","last_active":"2024-08-19T14:22:30.576109+0000","last_peered":"2024-08-19T14:22:30.576109+0000","last_clean":"2024-08-19T14:22:30.576109+0000","last_became_active":"2024-08-19T14:18:26.884105+0000","last_became_peered":"2024-08-19T14:18:26.884105+0000","last_unstale":"2024-08-19T14:22:30.576109+0000","last_undegraded":"2024-08-19T14:22:30.576109+0000","last_fullsized":"2024-08-19T14:22:30.576109+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:44.434382+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:20:15.321945+0000","last_clean_scrub_stamp":"2024-08-19T14:21:44.434382+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:13.544241+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"378'9683","reported_seq":9779,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-08-19T14:22:29.379986+0000","last_change":"2024-08-19T14:20:29.269723+0000","last_active":"2024-08-19T14:22:29.379986+0000","last_peered":"2024-08-19T14:22:29.379986+0000","last_clean":"2024-08-19T14:22:29.379986+0000","last_became_active":"2024-08-19T14:20:29.269432+0000","last_became_peered":"2024-08-19T14:20:29.269432+0000","last_unstale":"2024-08-19T14:22:29.379986+0000","last_undegraded":"2024-08-19T14:22:29.379986+0000","last_fullsized":"2024-08-19T14:22:29.379986+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":9683,"log_dups_size":0,"ondisk_log_size":9683,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":454361088,"num_objects":6933,"num_object_clones":0,"num_object_copies":20799,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6933,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9683,"num_write_kb":531712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.f","version":"70'139","reported_seq":793,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-08-19T14:22:30.573212+0000","last_change":"2024-08-19T14:21:14.765419+0000","last_active":"2024-08-19T14:22:30.573212+0000","last_peered":"2024-08-19T14:22:30.573212+0000","last_clean":"2024-08-19T14:22:30.573212+0000","last_became_active":"2024-08-19T14:18:18.711708+0000","last_became_peered":"2024-08-19T14:18:18.711708+0000","last_unstale":"2024-08-19T14:22:30.573212+0000","last_undegraded":"2024-08-19T14:22:30.573212+0000","last_fullsized":"2024-08-19T14:22:30.573212+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:14.765340+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:21:14.765340+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:22:33.271624+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"379'9693","reported_seq":9870,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-08-19T14:22:30.864808+0000","last_change":"2024-08-19T14:21:43.974700+0000","last_active":"2024-08-19T14:22:30.864808+0000","last_peered":"2024-08-19T14:22:30.864808+0000","last_clean":"2024-08-19T14:22:30.864808+0000","last_became_active":"2024-08-19T14:20:29.211975+0000","last_became_peered":"2024-08-19T14:20:29.211975+0000","last_unstale":"2024-08-19T14:22:30.864808+0000","last_undegraded":"2024-08-19T14:22:30.864808+0000","last_fullsized":"2024-08-19T14:22:30.864808+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"364'6991","last_scrub_stamp":"2024-08-19T14:21:43.974613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:21:43.974613+0000","objects_scrubbed":6934,"log_size":9693,"log_dups_size":0,"ondisk_log_size":9693,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:07.463188+0000","scrub_duration":1682,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":443351040,"num_objects":6765,"num_object_clones":0,"num_object_copies":20295,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6765,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9693,"num_write_kb":526656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,3],"acting":[11,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.c","version":"70'139","reported_seq":652,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-08-19T14:22:23.132826+0000","last_change":"2024-08-19T14:21:40.181472+0000","last_active":"2024-08-19T14:22:23.132826+0000","last_peered":"2024-08-19T14:22:23.132826+0000","last_clean":"2024-08-19T14:22:23.132826+0000","last_became_active":"2024-08-19T14:18:26.872942+0000","last_became_peered":"2024-08-19T14:18:26.872942+0000","last_unstale":"2024-08-19T14:22:23.132826+0000","last_undegraded":"2024-08-19T14:22:23.132826+0000","last_fullsized":"2024-08-19T14:22:23.132826+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:40.181356+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:20:07.839076+0000","last_clean_scrub_stamp":"2024-08-19T14:21:40.181356+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:22:52.009381+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,9],"acting":[3,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"379'9671","reported_seq":9856,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-08-19T14:22:30.908093+0000","last_change":"2024-08-19T14:22:12.682046+0000","last_active":"2024-08-19T14:22:30.908093+0000","last_peered":"2024-08-19T14:22:30.908093+0000","last_clean":"2024-08-19T14:22:30.908093+0000","last_became_active":"2024-08-19T14:20:29.267442+0000","last_became_peered":"2024-08-19T14:20:29.267442+0000","last_unstale":"2024-08-19T14:22:30.908093+0000","last_undegraded":"2024-08-19T14:22:30.908093+0000","last_fullsized":"2024-08-19T14:22:30.908093+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"374'8663","last_scrub_stamp":"2024-08-19T14:22:12.681916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:22:12.681916+0000","objects_scrubbed":7729,"log_size":9671,"log_dups_size":0,"ondisk_log_size":9671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:34.638254+0000","scrub_duration":1366,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":438632448,"num_objects":6693,"num_object_clones":0,"num_object_copies":20079,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6693,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9671,"num_write_kb":523648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,9],"acting":[11,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.d","version":"70'139","reported_seq":569,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-08-19T14:22:30.577373+0000","last_change":"2024-08-19T14:21:07.155294+0000","last_active":"2024-08-19T14:22:30.577373+0000","last_peered":"2024-08-19T14:22:30.577373+0000","last_clean":"2024-08-19T14:22:30.577373+0000","last_became_active":"2024-08-19T14:18:26.882002+0000","last_became_peered":"2024-08-19T14:18:26.882002+0000","last_unstale":"2024-08-19T14:22:30.577373+0000","last_undegraded":"2024-08-19T14:22:30.577373+0000","last_fullsized":"2024-08-19T14:22:30.577373+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:07.155157+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:21:07.155157+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"70'139","reported_seq":828,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-08-19T14:22:30.573215+0000","last_change":"2024-08-19T14:21:13.777891+0000","last_active":"2024-08-19T14:22:30.573215+0000","last_peered":"2024-08-19T14:22:30.573215+0000","last_clean":"2024-08-19T14:22:30.573215+0000","last_became_active":"2024-08-19T14:18:18.703594+0000","last_became_peered":"2024-08-19T14:18:18.703594+0000","last_unstale":"2024-08-19T14:22:30.573215+0000","last_undegraded":"2024-08-19T14:22:30.573215+0000","last_fullsized":"2024-08-19T14:22:30.573215+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:13.777777+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:21:13.777777+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:22:38.473560+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"70'139","reported_seq":878,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-08-19T14:22:23.133335+0000","last_change":"2024-08-19T14:20:07.692054+0000","last_active":"2024-08-19T14:22:23.133335+0000","last_peered":"2024-08-19T14:22:23.133335+0000","last_clean":"2024-08-19T14:22:23.133335+0000","last_became_active":"2024-08-19T14:18:18.698334+0000","last_became_peered":"2024-08-19T14:18:18.698334+0000","last_unstale":"2024-08-19T14:22:23.133335+0000","last_undegraded":"2024-08-19T14:22:23.133335+0000","last_fullsized":"2024-08-19T14:22:23.133335+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:07.691932+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:07.691932+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.10","version":"70'139","reported_seq":991,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-08-19T14:22:30.572942+0000","last_change":"2024-08-19T14:22:11.260404+0000","last_active":"2024-08-19T14:22:30.572942+0000","last_peered":"2024-08-19T14:22:30.572942+0000","last_clean":"2024-08-19T14:22:30.572942+0000","last_became_active":"2024-08-19T14:18:18.704562+0000","last_became_peered":"2024-08-19T14:18:18.704562+0000","last_unstale":"2024-08-19T14:22:30.572942+0000","last_undegraded":"2024-08-19T14:22:30.572942+0000","last_fullsized":"2024-08-19T14:22:30.572942+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:22:11.260286+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:22:11.260286+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:36.552471+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,8],"acting":[5,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"70'139","reported_seq":540,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-08-19T14:22:30.580913+0000","last_change":"2024-08-19T14:22:21.104595+0000","last_active":"2024-08-19T14:22:30.580913+0000","last_peered":"2024-08-19T14:22:30.580913+0000","last_clean":"2024-08-19T14:22:30.580913+0000","last_became_active":"2024-08-19T14:18:18.697894+0000","last_became_peered":"2024-08-19T14:18:18.697894+0000","last_unstale":"2024-08-19T14:22:30.580913+0000","last_undegraded":"2024-08-19T14:22:30.580913+0000","last_fullsized":"2024-08-19T14:22:30.580913+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:22:21.104491+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:21:16.071703+0000","last_clean_scrub_stamp":"2024-08-19T14:22:21.104491+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:45.856425+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"70'139","reported_seq":507,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-08-19T14:22:23.147631+0000","last_change":"2024-08-19T14:20:28.205212+0000","last_active":"2024-08-19T14:22:23.147631+0000","last_peered":"2024-08-19T14:22:23.147631+0000","last_clean":"2024-08-19T14:22:23.147631+0000","last_became_active":"2024-08-19T14:20:28.204724+0000","last_became_peered":"2024-08-19T14:20:28.204724+0000","last_unstale":"2024-08-19T14:22:23.147631+0000","last_undegraded":"2024-08-19T14:22:23.147631+0000","last_fullsized":"2024-08-19T14:22:23.147631+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:08.680021+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:08.680021+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"70'139","reported_seq":635,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-08-19T14:22:23.128821+0000","last_change":"2024-08-19T14:22:02.722505+0000","last_active":"2024-08-19T14:22:23.128821+0000","last_peered":"2024-08-19T14:22:23.128821+0000","last_clean":"2024-08-19T14:22:23.128821+0000","last_became_active":"2024-08-19T14:18:31.901556+0000","last_became_peered":"2024-08-19T14:18:31.901556+0000","last_unstale":"2024-08-19T14:22:23.128821+0000","last_undegraded":"2024-08-19T14:22:23.128821+0000","last_fullsized":"2024-08-19T14:22:23.128821+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:22:02.722345+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:22:02.722345+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:13.381470+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,0],"acting":[8,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.14","version":"70'139","reported_seq":638,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-08-19T14:22:30.577193+0000","last_change":"2024-08-19T14:19:55.130518+0000","last_active":"2024-08-19T14:22:30.577193+0000","last_peered":"2024-08-19T14:22:30.577193+0000","last_clean":"2024-08-19T14:22:30.577193+0000","last_became_active":"2024-08-19T14:18:18.703926+0000","last_became_peered":"2024-08-19T14:18:18.703926+0000","last_unstale":"2024-08-19T14:22:30.577193+0000","last_undegraded":"2024-08-19T14:22:30.577193+0000","last_fullsized":"2024-08-19T14:22:30.577193+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:19:55.130420+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:19:55.130420+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,1],"acting":[7,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.15","version":"70'139","reported_seq":525,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-08-19T14:22:30.577285+0000","last_change":"2024-08-19T14:20:09.181046+0000","last_active":"2024-08-19T14:22:30.577285+0000","last_peered":"2024-08-19T14:22:30.577285+0000","last_clean":"2024-08-19T14:22:30.577285+0000","last_became_active":"2024-08-19T14:18:31.912496+0000","last_became_peered":"2024-08-19T14:18:31.912496+0000","last_unstale":"2024-08-19T14:22:30.577285+0000","last_undegraded":"2024-08-19T14:22:30.577285+0000","last_fullsized":"2024-08-19T14:22:30.577285+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:09.180923+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:20:09.180923+0000","last_clean_scrub_stamp":"2024-08-19T14:20:09.180923+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"70'139","reported_seq":511,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-08-19T14:22:30.575899+0000","last_change":"2024-08-19T14:21:40.430835+0000","last_active":"2024-08-19T14:22:30.575899+0000","last_peered":"2024-08-19T14:22:30.575899+0000","last_clean":"2024-08-19T14:22:30.575899+0000","last_became_active":"2024-08-19T14:18:31.901229+0000","last_became_peered":"2024-08-19T14:18:31.901229+0000","last_unstale":"2024-08-19T14:22:30.575899+0000","last_undegraded":"2024-08-19T14:22:30.575899+0000","last_fullsized":"2024-08-19T14:22:30.575899+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:40.430714+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:21:40.430714+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:04.072653+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1b","version":"70'139","reported_seq":861,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-08-19T14:22:23.128754+0000","last_change":"2024-08-19T14:21:48.693864+0000","last_active":"2024-08-19T14:22:23.128754+0000","last_peered":"2024-08-19T14:22:23.128754+0000","last_clean":"2024-08-19T14:22:23.128754+0000","last_became_active":"2024-08-19T14:18:31.901152+0000","last_became_peered":"2024-08-19T14:18:31.901152+0000","last_unstale":"2024-08-19T14:22:23.128754+0000","last_undegraded":"2024-08-19T14:22:23.128754+0000","last_fullsized":"2024-08-19T14:22:23.128754+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:48.693761+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:21:48.693761+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:07.571130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.18","version":"70'139","reported_seq":907,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-08-19T14:22:30.572935+0000","last_change":"2024-08-19T14:22:07.254887+0000","last_active":"2024-08-19T14:22:30.572935+0000","last_peered":"2024-08-19T14:22:30.572935+0000","last_clean":"2024-08-19T14:22:30.572935+0000","last_became_active":"2024-08-19T14:18:31.902735+0000","last_became_peered":"2024-08-19T14:18:31.902735+0000","last_unstale":"2024-08-19T14:22:30.572935+0000","last_undegraded":"2024-08-19T14:22:30.572935+0000","last_fullsized":"2024-08-19T14:22:30.572935+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:22:07.254767+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:22:07.254767+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:11.195110+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.19","version":"70'139","reported_seq":855,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-08-19T14:22:30.580799+0000","last_change":"2024-08-19T14:21:40.219949+0000","last_active":"2024-08-19T14:22:30.580799+0000","last_peered":"2024-08-19T14:22:30.580799+0000","last_clean":"2024-08-19T14:22:30.580799+0000","last_became_active":"2024-08-19T14:18:31.902547+0000","last_became_peered":"2024-08-19T14:18:31.902547+0000","last_unstale":"2024-08-19T14:22:30.580799+0000","last_undegraded":"2024-08-19T14:22:30.580799+0000","last_fullsized":"2024-08-19T14:22:30.580799+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:40.219830+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:21:40.219830+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:05.414864+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1e","version":"70'139","reported_seq":513,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-08-19T14:22:30.575800+0000","last_change":"2024-08-19T14:22:09.404540+0000","last_active":"2024-08-19T14:22:30.575800+0000","last_peered":"2024-08-19T14:22:30.575800+0000","last_clean":"2024-08-19T14:22:30.575800+0000","last_became_active":"2024-08-19T14:18:31.902546+0000","last_became_peered":"2024-08-19T14:18:31.902546+0000","last_unstale":"2024-08-19T14:22:30.575800+0000","last_undegraded":"2024-08-19T14:22:30.575800+0000","last_fullsized":"2024-08-19T14:22:30.575800+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:22:09.404433+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:22:09.404433+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:13.587290+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1c","version":"70'139","reported_seq":619,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-08-19T14:22:23.128886+0000","last_change":"2024-08-19T14:21:38.753663+0000","last_active":"2024-08-19T14:22:23.128886+0000","last_peered":"2024-08-19T14:22:23.128886+0000","last_clean":"2024-08-19T14:22:23.128886+0000","last_became_active":"2024-08-19T14:18:31.902494+0000","last_became_peered":"2024-08-19T14:18:31.902494+0000","last_unstale":"2024-08-19T14:22:23.128886+0000","last_undegraded":"2024-08-19T14:22:23.128886+0000","last_fullsized":"2024-08-19T14:22:23.128886+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:38.753553+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:21:38.753553+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:22:46.720742+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,11],"acting":[8,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1d","version":"70'139","reported_seq":533,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-08-19T14:22:23.133552+0000","last_change":"2024-08-19T14:20:17.725615+0000","last_active":"2024-08-19T14:22:23.133552+0000","last_peered":"2024-08-19T14:22:23.133552+0000","last_clean":"2024-08-19T14:22:23.133552+0000","last_became_active":"2024-08-19T14:18:31.901183+0000","last_became_peered":"2024-08-19T14:18:31.901183+0000","last_unstale":"2024-08-19T14:22:23.133552+0000","last_undegraded":"2024-08-19T14:22:23.133552+0000","last_fullsized":"2024-08-19T14:22:23.133552+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:17.725477+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:17.725477+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,5],"acting":[1,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":7028801560,"num_objects":107252,"num_object_clones":0,"num_object_copies":321756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1594,"num_objects_unfound":0,"num_objects_dirty":107252,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":155094,"num_write_kb":8395009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2546,"num_bytes_recovered":166854656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":21179609088,"data_stored":21179596872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155094,"ondisk_log_size":155094,"up":48,"acting":47,"num_store_stats":9},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1802240,"data_stored":1788032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187590,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3455764,"kb_used_data":3023372,"kb_used_omap":1,"kb_used_meta":432382,"kb_avail":90289388,"statfs":{"total":95995035648,"available":92456333312,"internally_reserved":0,"allocated":3095932928,"data_stored":3094192010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":442759620},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":343597383820,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":74,"seq":317827580048,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2075748,"kb_used_data":1593372,"kb_used_omap":1,"kb_used_meta":482366,"kb_avail":91669404,"statfs":{"total":95995035648,"available":93869469696,"internally_reserved":0,"allocated":1631612928,"data_stored":1629863712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":493943233},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874389,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":817684,"kb_used_data":439436,"kb_used_omap":1,"kb_used_meta":378238,"kb_avail":92927468,"statfs":{"total":95995035648,"available":95157727232,"internally_reserved":0,"allocated":449982464,"data_stored":448241546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":387316165},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":57,"seq":244813136024,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3851688,"kb_used_data":3383200,"kb_used_omap":1,"kb_used_meta":468478,"kb_avail":89893464,"statfs":{"total":95995035648,"available":92050907136,"internally_reserved":0,"allocated":3464396800,"data_stored":3462648104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":479721930},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":206158430363,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692000,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1904044,"kb_used_data":1566048,"kb_used_omap":1,"kb_used_meta":337982,"kb_avail":91841108,"statfs":{"total":95995035648,"available":94045294592,"internally_reserved":0,"allocated":1603633152,"data_stored":1601884456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":346094021},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986341,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2600740,"kb_used_data":2105884,"kb_used_omap":1,"kb_used_meta":494846,"kb_avail":91144412,"statfs":{"total":95995035648,"available":93331877888,"internally_reserved":0,"allocated":2156425216,"data_stored":2154676000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":506722757},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149866,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3050964,"kb_used_data":2608844,"kb_used_omap":1,"kb_used_meta":442110,"kb_avail":90694188,"statfs":{"total":95995035648,"available":92870848512,"internally_reserved":0,"allocated":2671456256,"data_stored":2669715338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":452721095},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411504,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575027,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4893088,"kb_used_data":4356504,"kb_used_omap":1,"kb_used_meta":536510,"kb_avail":88852064,"statfs":{"total":95995035648,"available":90984513536,"internally_reserved":0,"allocated":4461060096,"data_stored":4459311554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":549386693},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,4,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738552,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2063460,"kb_used_data":1711576,"kb_used_omap":1,"kb_used_meta":351870,"kb_avail":91681692,"statfs":{"total":95995035648,"available":93882052608,"internally_reserved":0,"allocated":1752653824,"data_stored":1750908680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":360315333},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1746468864,"data_stored":1746468864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4453896192,"data_stored":4453892120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2659975168,"data_stored":2659975168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2139164672,"data_stored":2139160600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1592328192,"data_stored":1592328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3442278400,"data_stored":3442278400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":444596224,"data_stored":444596224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1623396352,"data_stored":1623392280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":3077505024,"data_stored":3077505024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T14:22:39.964 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph status --format=json 2024-08-19T14:22:39.965 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:22:39.990 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:39.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:39.991 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:22:40.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:39 smithi032 ceph-mon[20004]: pgmap v861: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 6.3 GiB data, 23 GiB used, 782 GiB / 805 GiB avail; 0 B/s wr, 936 op/s; 1399/310926 objects misplaced (0.450%); 2.3 MiB/s, 37 objects/s recovering 2024-08-19T14:22:40.008 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:22:40.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:39 smithi045 ceph-mon[20143]: pgmap v861: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 6.3 GiB data, 23 GiB used, 782 GiB / 805 GiB avail; 0 B/s wr, 936 op/s; 1399/310926 objects misplaced (0.450%); 2.3 MiB/s, 37 objects/s recovering 2024-08-19T14:22:40.092 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:22:40.123 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:40.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:40.124 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:22:40.225 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:22:40.247 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:40.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:40.248 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:22:40.349 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:22:40.376 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:40.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:40.377 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:22:40.478 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:22:40.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:40.523 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:40.524 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:22:40.625 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:22:40.662 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:40.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:40.663 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:22:40.765 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:22:40.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:40.793 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:40.793 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:22:40.894 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:22:40.919 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:40.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:40.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:22:41.003 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:22:41.003 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-08-19T14:22:41.003 INFO:teuthology.orchestra.run.smithi031.stdout: "num_ops": 0 2024-08-19T14:22:41.003 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:22:41.010 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:22:41.021 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:22:41.046 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:41.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:41.047 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:22:41.148 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:22:41.175 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:41.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:41.176 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:22:41.277 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:22:41.320 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:41.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:41.321 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:22:41.421 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:22:41.455 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:41.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:41.456 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:22:41.468 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:22:41.556 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:22:41.622 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is unset 2024-08-19T14:22:41.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:41.625 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:41.626 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:22:41.644 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:41 smithi031 ceph-mon[16336]: pgmap v862: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 6.2 GiB data, 23 GiB used, 782 GiB / 805 GiB avail; 0 B/s wr, 905 op/s; 1114/303108 objects misplaced (0.368%); 3.0 MiB/s, 47 objects/s recovering 2024-08-19T14:22:41.644 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:41 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3429993729' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:22:41.727 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:22:41.771 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:41.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:41.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:22:41.874 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:22:41.905 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:41.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:41.906 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:22:42.007 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:22:42.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:41 smithi032 ceph-mon[20004]: pgmap v862: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 6.2 GiB data, 23 GiB used, 782 GiB / 805 GiB avail; 0 B/s wr, 905 op/s; 1114/303108 objects misplaced (0.368%); 3.0 MiB/s, 47 objects/s recovering 2024-08-19T14:22:42.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:41 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3429993729' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:22:42.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:41 smithi045 ceph-mon[20143]: pgmap v862: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 6.2 GiB data, 23 GiB used, 782 GiB / 805 GiB avail; 0 B/s wr, 905 op/s; 1114/303108 objects misplaced (0.368%); 3.0 MiB/s, 47 objects/s recovering 2024-08-19T14:22:42.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:41 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3429993729' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:22:42.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:42.050 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:42.050 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:22:42.152 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:22:42.179 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:42.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:42.180 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:22:42.280 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:22:42.309 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:22:42.310 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"ed7d1c6e-5e33-11ef-bccf-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"nodeep-scrub flag(s) set","count":12},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 1 pg undersized","count":1},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":939,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":380,"num_osds":12,"num_up_osds":12,"osd_up_since":1724076699,"num_in_osds":9,"osd_in_since":1724076998,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":46},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":47,"num_pools":2,"num_objects":101036,"data_bytes":6621889080,"bytes_used":24227549184,"bytes_avail":839727771648,"bytes_total":863955320832,"misplaced_objects":1114,"misplaced_total":303108,"misplaced_ratio":0.0036752576639349673,"recovering_objects_per_sec":47,"recovering_bytes_per_sec":3144822,"recovering_keys_per_sec":0,"num_objects_recovered":480,"num_bytes_recovered":31457280,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":905},"fsmap":{"epoch":1,"btime":"2024-08-19T14:05:22:841940+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.31:8443/"}},"servicemap":{"epoch":22,"modified":"2024-08-19T14:22:15.909544+0000","services":{"mon":{"daemons":{"summary":"","c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-08-19T14:22:42.318 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:42.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:42.319 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:22:42.420 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:22:42.443 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:42.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:42.444 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:22:42.545 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:22:42.570 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:42.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:42.571 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:22:42.672 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:22:42.706 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:42.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:42.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:22:42.809 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:22:42.835 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:42.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:42.836 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:22:42.937 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:22:42.960 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:42.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:42.961 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:22:42.996 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:42 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:22:42.996 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:42 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:22:42.996 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:42 smithi031 ceph-mon[16336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:22:42.996 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:42 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3429993729' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:22:42.996 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:42 smithi031 ceph-mon[16336]: osdmap e380: 12 total, 12 up, 9 in 2024-08-19T14:22:42.996 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:42 smithi031 ceph-mon[16336]: 1.f scrub starts 2024-08-19T14:22:42.996 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:42 smithi031 ceph-mon[16336]: 1.f scrub ok 2024-08-19T14:22:42.996 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:42 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1524567095' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-08-19T14:22:43.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:42 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:22:43.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:42 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:22:43.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:42 smithi032 ceph-mon[20004]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:22:43.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:42 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3429993729' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:22:43.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:42 smithi032 ceph-mon[20004]: osdmap e380: 12 total, 12 up, 9 in 2024-08-19T14:22:43.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:42 smithi032 ceph-mon[20004]: 1.f scrub starts 2024-08-19T14:22:43.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:42 smithi032 ceph-mon[20004]: 1.f scrub ok 2024-08-19T14:22:43.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:42 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1524567095' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-08-19T14:22:43.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:42 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:22:43.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:42 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:22:43.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:42 smithi045 ceph-mon[20143]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:22:43.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:42 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3429993729' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:22:43.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:42 smithi045 ceph-mon[20143]: osdmap e380: 12 total, 12 up, 9 in 2024-08-19T14:22:43.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:42 smithi045 ceph-mon[20143]: 1.f scrub starts 2024-08-19T14:22:43.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:42 smithi045 ceph-mon[20143]: 1.f scrub ok 2024-08-19T14:22:43.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:42 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1524567095' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-08-19T14:22:43.062 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:22:43.083 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:43.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:43.084 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:22:43.109 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-08-19T14:22:43.185 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:22:43.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:43.211 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:43.211 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:22:43.312 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:22:43.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:43.339 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:43.339 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:22:43.440 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:22:43.488 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:43.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:43.489 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:22:43.590 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:22:43.618 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:43.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:43.619 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:22:43.720 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:22:43.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:43.747 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:43.747 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:22:43.848 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:22:43.874 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:43.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:43.875 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:22:43.976 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:22:44.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:44.004 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:44.004 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:22:44.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:43 smithi032 ceph-mon[20004]: pgmap v864: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 6.1 GiB data, 22 GiB used, 782 GiB / 805 GiB avail; 0 B/s wr, 885 op/s; 1114/298611 objects misplaced (0.373%); 3.0 MiB/s, 47 objects/s recovering 2024-08-19T14:22:44.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:43 smithi045 ceph-mon[20143]: pgmap v864: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 6.1 GiB data, 22 GiB used, 782 GiB / 805 GiB avail; 0 B/s wr, 885 op/s; 1114/298611 objects misplaced (0.373%); 3.0 MiB/s, 47 objects/s recovering 2024-08-19T14:22:44.105 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:22:44.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:43 smithi031 ceph-mon[16336]: pgmap v864: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 6.1 GiB data, 22 GiB used, 782 GiB / 805 GiB avail; 0 B/s wr, 885 op/s; 1114/298611 objects misplaced (0.373%); 3.0 MiB/s, 47 objects/s recovering 2024-08-19T14:22:44.128 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:44.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:44.129 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:22:44.229 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:22:44.251 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:44.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:44.252 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:22:44.353 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:22:44.382 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:44.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:44.383 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:22:44.486 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:22:44.508 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:44.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:44.509 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:22:44.575 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-08-19T14:22:44.576 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-19T14:22:44.584 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:22:44.610 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:22:44.632 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:44.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:44.633 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:22:44.643 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:22:44.735 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:22:44.760 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:44.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:44.761 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:22:44.862 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:22:44.898 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:44.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:44.900 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:22:45.000 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:22:45.025 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:45.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:45.026 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:22:45.026 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-19T14:22:45.127 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:22:45.152 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:45.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:45.153 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:22:45.254 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:22:45.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:45.305 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:45.305 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:22:45.406 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:22:45.450 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:45.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:45.451 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:22:45.552 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:22:45.578 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:45.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:45.579 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:22:45.596 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:22:45.596 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:22:45.667 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:22:45.667 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:22:45.667 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:22:45.667 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:22:45.667 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:22:45.668 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:22:45.668 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:22:45.668 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:22:45.675 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:22:45.675 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:22:45.680 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:22:45.710 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:22:45.710 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:22:45.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:45.718 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:45.718 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:22:45.749 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:22:45.749 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:22:45.787 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:22:45.787 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:22:45.819 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:22:45.822 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:22:45.822 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:22:45.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:45.855 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:45.856 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:22:45.856 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:22:45.856 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:22:45.857 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:45 smithi031 ceph-mon[16336]: pgmap v865: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 6.0 GiB data, 22 GiB used, 783 GiB / 805 GiB avail; 0 B/s wr, 876 op/s; 1114/295446 objects misplaced (0.377%); 3.0 MiB/s, 47 objects/s recovering 2024-08-19T14:22:45.857 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:45 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:22:45.857 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:45 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:22:45.857 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:45 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/873062869' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:22:45.890 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:22:45.890 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:22:45.932 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:22:45.933 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:22:45.956 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:22:45.981 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:45.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:45.982 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:22:46.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:45 smithi032 ceph-mon[20004]: pgmap v865: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 6.0 GiB data, 22 GiB used, 783 GiB / 805 GiB avail; 0 B/s wr, 876 op/s; 1114/295446 objects misplaced (0.377%); 3.0 MiB/s, 47 objects/s recovering 2024-08-19T14:22:46.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:45 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:22:46.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:45 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:22:46.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:45 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/873062869' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:22:46.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:45 smithi045 ceph-mon[20143]: pgmap v865: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 6.0 GiB data, 22 GiB used, 783 GiB / 805 GiB avail; 0 B/s wr, 876 op/s; 1114/295446 objects misplaced (0.377%); 3.0 MiB/s, 47 objects/s recovering 2024-08-19T14:22:46.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:45 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:22:46.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:45 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:22:46.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:45 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/873062869' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:22:46.083 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:22:46.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:46.122 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:46.122 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:22:46.223 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:22:46.249 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:46.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:46.250 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:22:46.351 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:22:46.377 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:46.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:46.378 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:22:46.479 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:22:46.504 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:46.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:46.505 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:22:46.605 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:22:46.631 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:46.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:46.632 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:22:46.646 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:22:46.647 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:22:46.733 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:22:46.767 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:46.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:46.768 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:22:46.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:46 smithi031 ceph-mon[16336]: osdmap e381: 12 total, 12 up, 9 in 2024-08-19T14:22:46.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:46 smithi031 ceph-mon[16336]: 1.4 scrub starts 2024-08-19T14:22:46.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:46 smithi031 ceph-mon[16336]: 1.4 scrub ok 2024-08-19T14:22:46.869 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:22:46.894 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:46.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:46.895 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:22:46.996 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:22:47.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:46 smithi032 ceph-mon[20004]: osdmap e381: 12 total, 12 up, 9 in 2024-08-19T14:22:47.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:46 smithi032 ceph-mon[20004]: 1.4 scrub starts 2024-08-19T14:22:47.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:46 smithi032 ceph-mon[20004]: 1.4 scrub ok 2024-08-19T14:22:47.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:47.038 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:47.038 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:22:47.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:46 smithi045 ceph-mon[20143]: osdmap e381: 12 total, 12 up, 9 in 2024-08-19T14:22:47.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:46 smithi045 ceph-mon[20143]: 1.4 scrub starts 2024-08-19T14:22:47.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:46 smithi045 ceph-mon[20143]: 1.4 scrub ok 2024-08-19T14:22:47.106 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-08-19T14:22:47.139 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:22:47.162 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:47.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:47.163 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:22:47.264 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:22:47.288 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:47.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:47.289 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:22:47.390 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:22:47.461 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:47.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:47.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:22:47.563 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:22:47.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:47.593 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:47.593 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:22:47.694 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:22:47.718 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:47.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:47.719 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:22:47.820 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:22:47.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:47.879 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:47.879 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:22:47.980 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:22:48.004 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:47 smithi031 ceph-mon[16336]: pgmap v867: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.9 GiB data, 22 GiB used, 783 GiB / 805 GiB avail; 0 B/s wr, 849 op/s; 885/290529 objects misplaced (0.305%); 4.0 MiB/s, 64 objects/s recovering 2024-08-19T14:22:48.004 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:47 smithi031 ceph-mon[16336]: 1.12 scrub starts 2024-08-19T14:22:48.004 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:47 smithi031 ceph-mon[16336]: 1.12 scrub ok 2024-08-19T14:22:48.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:47 smithi032 ceph-mon[20004]: pgmap v867: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.9 GiB data, 22 GiB used, 783 GiB / 805 GiB avail; 0 B/s wr, 849 op/s; 885/290529 objects misplaced (0.305%); 4.0 MiB/s, 64 objects/s recovering 2024-08-19T14:22:48.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:47 smithi032 ceph-mon[20004]: 1.12 scrub starts 2024-08-19T14:22:48.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:47 smithi032 ceph-mon[20004]: 1.12 scrub ok 2024-08-19T14:22:48.023 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:48.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:48.024 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:22:48.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:47 smithi045 ceph-mon[20143]: pgmap v867: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.9 GiB data, 22 GiB used, 783 GiB / 805 GiB avail; 0 B/s wr, 849 op/s; 885/290529 objects misplaced (0.305%); 4.0 MiB/s, 64 objects/s recovering 2024-08-19T14:22:48.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:47 smithi045 ceph-mon[20143]: 1.12 scrub starts 2024-08-19T14:22:48.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:47 smithi045 ceph-mon[20143]: 1.12 scrub ok 2024-08-19T14:22:48.124 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:22:48.149 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:48.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:48.150 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:22:48.252 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:22:48.271 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:22:48.290 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:48.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:48.291 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:22:48.392 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:22:48.417 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:48.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:48.418 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:22:48.519 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:22:48.544 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:48.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:48.545 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:22:48.637 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:22:48.640 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-08-19T14:22:48.640 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:22:48.647 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:22:48.670 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:48.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:48.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:22:48.696 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is unset 2024-08-19T14:22:48.772 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:22:48.794 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:48.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:48.796 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:22:48.847 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:48 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3258105175' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:22:48.896 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:22:48.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:48.955 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:48.955 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:22:49.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:48 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3258105175' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:22:49.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:48 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3258105175' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:22:49.056 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:22:49.100 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:49.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:49.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:22:49.202 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:22:49.229 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:49.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:49.230 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:22:49.331 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:22:49.492 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":868,"stamp":"2024-08-19T14:22:47.921352+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6238831160,"num_objects":95191,"num_object_clones":0,"num_object_copies":285573,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":885,"num_objects_unfound":0,"num_objects_dirty":95191,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":167157,"num_write_kb":8395009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3255,"num_bytes_recovered":213319680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":165066,"ondisk_log_size":165066,"up":141,"acting":140,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":142,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":22628300,"kb_used_data":18620084,"kb_used_omap":14,"kb_used_meta":4008049,"kb_avail":821078068,"statfs":{"total":863955320832,"available":840783941632,"internally_reserved":0,"allocated":19066966016,"data_stored":19051195088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14353,"internal_metadata":4104243183},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,4,17,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-383057920,"num_objects":-5845,"num_object_clones":0,"num_object_copies":-17535,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-229,"num_objects_unfound":0,"num_objects_dirty":-5845,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5845,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":229,"num_bytes_recovered":15007744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003846"},"pg_stats":[{"pgid":"4.5","version":"381'10604","reported_seq":10789,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-08-19T14:22:45.919031+0000","last_change":"2024-08-19T14:22:14.211513+0000","last_active":"2024-08-19T14:22:45.919031+0000","last_peered":"2024-08-19T14:22:45.919031+0000","last_clean":"2024-08-19T14:22:45.919031+0000","last_became_active":"2024-08-19T14:20:29.267549+0000","last_became_peered":"2024-08-19T14:20:29.267549+0000","last_unstale":"2024-08-19T14:22:45.919031+0000","last_undegraded":"2024-08-19T14:22:45.919031+0000","last_fullsized":"2024-08-19T14:22:45.919031+0000","mapping_epoch":335,"log_start":"339'600","ondisk_log_start":"339'600","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"375'8747","last_scrub_stamp":"2024-08-19T14:22:14.211361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:22:14.211361+0000","objects_scrubbed":7717,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:26.018227+0000","scrub_duration":1909,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":379846656,"num_objects":5796,"num_object_clones":0,"num_object_copies":17388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5796,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10604,"num_write_kb":524800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,0],"acting":[11,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"70'139","reported_seq":977,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-08-19T14:22:41.628424+0000","last_change":"2024-08-19T14:19:58.072227+0000","last_active":"2024-08-19T14:22:41.628424+0000","last_peered":"2024-08-19T14:22:41.628424+0000","last_clean":"2024-08-19T14:22:41.628424+0000","last_became_active":"2024-08-19T14:18:13.807133+0000","last_became_peered":"2024-08-19T14:18:13.807133+0000","last_unstale":"2024-08-19T14:22:41.628424+0000","last_undegraded":"2024-08-19T14:22:41.628424+0000","last_fullsized":"2024-08-19T14:22:41.628424+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:19:58.072083+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:19:58.072083+0000","last_clean_scrub_stamp":"2024-08-19T14:19:58.072083+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,9],"acting":[7,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"380'10623","reported_seq":17237,"reported_epoch":380,"state":"active+recovering+undersized+remapped","last_fresh":"2024-08-19T14:22:44.446454+0000","last_change":"2024-08-19T14:21:07.239135+0000","last_active":"2024-08-19T14:22:44.446454+0000","last_peered":"2024-08-19T14:22:44.446454+0000","last_clean":"2024-08-19T14:21:04.784061+0000","last_became_active":"2024-08-19T14:21:07.213050+0000","last_became_peered":"2024-08-19T14:21:07.213050+0000","last_unstale":"2024-08-19T14:22:44.446454+0000","last_undegraded":"2024-08-19T14:22:44.446454+0000","last_fullsized":"2024-08-19T14:21:06.805412+0000","mapping_epoch":354,"log_start":"339'600","ondisk_log_start":"339'600","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":396820480,"num_objects":6055,"num_object_clones":0,"num_object_copies":18165,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":885,"num_objects_unfound":0,"num_objects_dirty":6055,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10623,"num_write_kb":533696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3255,"num_bytes_recovered":213319680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,7],"acting":[1,7],"avail_no_missing":["1","5","7"],"object_location_counts":[{"shards":"1,5,7","objects":885},{"shards":"1,7","objects":5170}],"blocked_by":[],"up_primary":9,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"70'139","reported_seq":875,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-08-19T14:22:41.636529+0000","last_change":"2024-08-19T14:20:03.714917+0000","last_active":"2024-08-19T14:22:41.636529+0000","last_peered":"2024-08-19T14:22:41.636529+0000","last_clean":"2024-08-19T14:22:41.636529+0000","last_became_active":"2024-08-19T14:18:13.804721+0000","last_became_peered":"2024-08-19T14:18:13.804721+0000","last_unstale":"2024-08-19T14:22:41.636529+0000","last_undegraded":"2024-08-19T14:22:41.636529+0000","last_fullsized":"2024-08-19T14:22:41.636529+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:03.714818+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:03.714818+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"380'10463","reported_seq":10562,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-08-19T14:22:44.375696+0000","last_change":"2024-08-19T14:20:29.268110+0000","last_active":"2024-08-19T14:22:44.375696+0000","last_peered":"2024-08-19T14:22:44.375696+0000","last_clean":"2024-08-19T14:22:44.375696+0000","last_became_active":"2024-08-19T14:20:29.267811+0000","last_became_peered":"2024-08-19T14:20:29.267811+0000","last_unstale":"2024-08-19T14:22:44.375696+0000","last_undegraded":"2024-08-19T14:22:44.375696+0000","last_fullsized":"2024-08-19T14:22:44.375696+0000","mapping_epoch":335,"log_start":"339'400","ondisk_log_start":"339'400","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":392101888,"num_objects":5983,"num_object_clones":0,"num_object_copies":17949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5983,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10463,"num_write_kb":526272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"70'139","reported_seq":823,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-08-19T14:22:41.625134+0000","last_change":"2024-08-19T14:20:50.742676+0000","last_active":"2024-08-19T14:22:41.625134+0000","last_peered":"2024-08-19T14:22:41.625134+0000","last_clean":"2024-08-19T14:22:41.625134+0000","last_became_active":"2024-08-19T14:20:50.742354+0000","last_became_peered":"2024-08-19T14:20:50.742354+0000","last_unstale":"2024-08-19T14:22:41.625134+0000","last_undegraded":"2024-08-19T14:22:41.625134+0000","last_fullsized":"2024-08-19T14:22:41.625134+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:19:59.302931+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:19:59.302931+0000","last_clean_scrub_stamp":"2024-08-19T14:19:59.302931+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":114,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"380'10461","reported_seq":10562,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-08-19T14:22:44.452139+0000","last_change":"2024-08-19T14:20:29.267640+0000","last_active":"2024-08-19T14:22:44.452139+0000","last_peered":"2024-08-19T14:22:44.452139+0000","last_clean":"2024-08-19T14:22:44.452139+0000","last_became_active":"2024-08-19T14:20:29.267331+0000","last_became_peered":"2024-08-19T14:20:29.267331+0000","last_unstale":"2024-08-19T14:22:44.452139+0000","last_undegraded":"2024-08-19T14:22:44.452139+0000","last_fullsized":"2024-08-19T14:22:44.452139+0000","mapping_epoch":335,"log_start":"339'400","ondisk_log_start":"339'400","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":388300800,"num_objects":5925,"num_object_clones":0,"num_object_copies":17775,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5925,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10461,"num_write_kb":524352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"70'139","reported_seq":872,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-08-19T14:22:41.627380+0000","last_change":"2024-08-19T14:20:09.507422+0000","last_active":"2024-08-19T14:22:41.627380+0000","last_peered":"2024-08-19T14:22:41.627380+0000","last_clean":"2024-08-19T14:22:41.627380+0000","last_became_active":"2024-08-19T14:18:13.808505+0000","last_became_peered":"2024-08-19T14:18:13.808505+0000","last_unstale":"2024-08-19T14:22:41.627380+0000","last_undegraded":"2024-08-19T14:22:41.627380+0000","last_fullsized":"2024-08-19T14:22:41.627380+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:09.507256+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:09.507256+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,11],"acting":[9,0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"380'10465","reported_seq":10566,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-08-19T14:22:43.375707+0000","last_change":"2024-08-19T14:20:29.268766+0000","last_active":"2024-08-19T14:22:43.375707+0000","last_peered":"2024-08-19T14:22:43.375707+0000","last_clean":"2024-08-19T14:22:43.375707+0000","last_became_active":"2024-08-19T14:20:29.268443+0000","last_became_peered":"2024-08-19T14:20:29.268443+0000","last_unstale":"2024-08-19T14:22:43.375707+0000","last_undegraded":"2024-08-19T14:22:43.375707+0000","last_fullsized":"2024-08-19T14:22:43.375707+0000","mapping_epoch":335,"log_start":"339'400","ondisk_log_start":"339'400","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":396296192,"num_objects":6047,"num_object_clones":0,"num_object_copies":18141,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6047,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10465,"num_write_kb":528384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,11],"acting":[9,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.7","version":"70'139","reported_seq":524,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-08-19T14:22:41.645205+0000","last_change":"2024-08-19T14:21:46.057764+0000","last_active":"2024-08-19T14:22:41.645205+0000","last_peered":"2024-08-19T14:22:41.645205+0000","last_clean":"2024-08-19T14:22:41.645205+0000","last_became_active":"2024-08-19T14:20:28.204384+0000","last_became_peered":"2024-08-19T14:20:28.204384+0000","last_unstale":"2024-08-19T14:22:41.645205+0000","last_undegraded":"2024-08-19T14:22:41.645205+0000","last_fullsized":"2024-08-19T14:22:41.645205+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:46.057624+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:21:46.057624+0000","last_clean_scrub_stamp":"2024-08-19T14:21:46.057624+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:02.689278+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"381'10513","reported_seq":10761,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-08-19T14:22:45.847639+0000","last_change":"2024-08-19T14:22:07.840154+0000","last_active":"2024-08-19T14:22:45.847639+0000","last_peered":"2024-08-19T14:22:45.847639+0000","last_clean":"2024-08-19T14:22:45.847639+0000","last_became_active":"2024-08-19T14:20:29.267379+0000","last_became_peered":"2024-08-19T14:20:29.267379+0000","last_unstale":"2024-08-19T14:22:45.847639+0000","last_undegraded":"2024-08-19T14:22:45.847639+0000","last_fullsized":"2024-08-19T14:22:45.847639+0000","mapping_epoch":335,"log_start":"339'500","ondisk_log_start":"339'500","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"373'8338","last_scrub_stamp":"2024-08-19T14:22:07.840021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:22:07.840021+0000","objects_scrubbed":8073,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:13.679336+0000","scrub_duration":2409,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":382533632,"num_objects":5837,"num_object_clones":0,"num_object_copies":17511,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5837,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10513,"num_write_kb":523200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,1],"acting":[11,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.6","version":"70'139","reported_seq":876,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-08-19T14:22:41.627482+0000","last_change":"2024-08-19T14:19:55.512078+0000","last_active":"2024-08-19T14:22:41.627482+0000","last_peered":"2024-08-19T14:22:41.627482+0000","last_clean":"2024-08-19T14:22:41.627482+0000","last_became_active":"2024-08-19T14:18:26.874902+0000","last_became_peered":"2024-08-19T14:18:26.874902+0000","last_unstale":"2024-08-19T14:22:41.627482+0000","last_undegraded":"2024-08-19T14:22:41.627482+0000","last_fullsized":"2024-08-19T14:22:41.627482+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:19:55.511974+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:27.848870+0000","last_clean_scrub_stamp":"2024-08-19T14:19:55.511974+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,4],"acting":[9,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"380'10364","reported_seq":10551,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-08-19T14:22:44.628800+0000","last_change":"2024-08-19T14:22:08.536948+0000","last_active":"2024-08-19T14:22:44.628800+0000","last_peered":"2024-08-19T14:22:44.628800+0000","last_clean":"2024-08-19T14:22:44.628800+0000","last_became_active":"2024-08-19T14:20:29.267360+0000","last_became_peered":"2024-08-19T14:20:29.267360+0000","last_unstale":"2024-08-19T14:22:44.628800+0000","last_undegraded":"2024-08-19T14:22:44.628800+0000","last_fullsized":"2024-08-19T14:22:44.628800+0000","mapping_epoch":335,"log_start":"339'300","ondisk_log_start":"339'300","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"374'8331","last_scrub_stamp":"2024-08-19T14:22:08.536620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:22:08.536620+0000","objects_scrubbed":7995,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:19.976024+0000","scrub_duration":2478,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":385089536,"num_objects":5876,"num_object_clones":0,"num_object_copies":17628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5876,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10364,"num_write_kb":519680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"70'139","reported_seq":700,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-08-19T14:22:45.668997+0000","last_change":"2024-08-19T14:22:01.354585+0000","last_active":"2024-08-19T14:22:45.668997+0000","last_peered":"2024-08-19T14:22:45.668997+0000","last_clean":"2024-08-19T14:22:45.668997+0000","last_became_active":"2024-08-19T14:18:26.881435+0000","last_became_peered":"2024-08-19T14:18:26.881435+0000","last_unstale":"2024-08-19T14:22:45.668997+0000","last_undegraded":"2024-08-19T14:22:45.668997+0000","last_fullsized":"2024-08-19T14:22:45.668997+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:22:01.354343+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:22:01.354343+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:30.081548+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,4],"acting":[11,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.1","version":"381'10596","reported_seq":10780,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-08-19T14:22:45.897216+0000","last_change":"2024-08-19T14:22:20.917940+0000","last_active":"2024-08-19T14:22:45.897216+0000","last_peered":"2024-08-19T14:22:45.897216+0000","last_clean":"2024-08-19T14:22:45.897216+0000","last_became_active":"2024-08-19T14:20:29.268465+0000","last_became_peered":"2024-08-19T14:20:29.268465+0000","last_unstale":"2024-08-19T14:22:45.897216+0000","last_undegraded":"2024-08-19T14:22:45.897216+0000","last_fullsized":"2024-08-19T14:22:45.897216+0000","mapping_epoch":335,"log_start":"339'500","ondisk_log_start":"339'500","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"377'9214","last_scrub_stamp":"2024-08-19T14:22:20.917842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:22:20.917842+0000","objects_scrubbed":7332,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:39.609654+0000","scrub_duration":1625,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":387055616,"num_objects":5906,"num_object_clones":0,"num_object_copies":17718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5906,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10596,"num_write_kb":528064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.4","version":"70'139","reported_seq":924,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-08-19T14:22:45.668936+0000","last_change":"2024-08-19T14:21:39.314063+0000","last_active":"2024-08-19T14:22:45.668936+0000","last_peered":"2024-08-19T14:22:45.668936+0000","last_clean":"2024-08-19T14:22:45.668936+0000","last_became_active":"2024-08-19T14:18:13.806013+0000","last_became_peered":"2024-08-19T14:18:13.806013+0000","last_unstale":"2024-08-19T14:22:45.668936+0000","last_undegraded":"2024-08-19T14:22:45.668936+0000","last_fullsized":"2024-08-19T14:22:45.668936+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:39.313941+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:21:39.313941+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.f","version":"380'10324","reported_seq":10425,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-08-19T14:22:44.445831+0000","last_change":"2024-08-19T14:20:29.212298+0000","last_active":"2024-08-19T14:22:44.445831+0000","last_peered":"2024-08-19T14:22:44.445831+0000","last_clean":"2024-08-19T14:22:44.445831+0000","last_became_active":"2024-08-19T14:20:29.212003+0000","last_became_peered":"2024-08-19T14:20:29.212003+0000","last_unstale":"2024-08-19T14:22:44.445831+0000","last_undegraded":"2024-08-19T14:22:44.445831+0000","last_fullsized":"2024-08-19T14:22:44.445831+0000","mapping_epoch":335,"log_start":"339'300","ondisk_log_start":"339'300","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":391774208,"num_objects":5978,"num_object_clones":0,"num_object_copies":17934,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10324,"num_write_kb":521664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,11],"acting":[1,3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"70'139","reported_seq":884,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-08-19T14:22:41.630822+0000","last_change":"2024-08-19T14:20:12.570453+0000","last_active":"2024-08-19T14:22:41.630822+0000","last_peered":"2024-08-19T14:22:41.630822+0000","last_clean":"2024-08-19T14:22:41.630822+0000","last_became_active":"2024-08-19T14:18:26.875006+0000","last_became_peered":"2024-08-19T14:18:26.875006+0000","last_unstale":"2024-08-19T14:22:41.630822+0000","last_undegraded":"2024-08-19T14:22:41.630822+0000","last_fullsized":"2024-08-19T14:22:41.630822+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:12.570352+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:28.864870+0000","last_clean_scrub_stamp":"2024-08-19T14:20:12.570352+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,0],"acting":[9,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.c","version":"380'10273","reported_seq":10461,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-08-19T14:22:43.206169+0000","last_change":"2024-08-19T14:22:07.487434+0000","last_active":"2024-08-19T14:22:43.206169+0000","last_peered":"2024-08-19T14:22:43.206169+0000","last_clean":"2024-08-19T14:22:43.206169+0000","last_became_active":"2024-08-19T14:20:29.266373+0000","last_became_peered":"2024-08-19T14:20:29.266373+0000","last_unstale":"2024-08-19T14:22:43.206169+0000","last_undegraded":"2024-08-19T14:22:43.206169+0000","last_fullsized":"2024-08-19T14:22:43.206169+0000","mapping_epoch":335,"log_start":"338'200","ondisk_log_start":"338'200","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"373'8336","last_scrub_stamp":"2024-08-19T14:22:07.487270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:22:07.487270+0000","objects_scrubbed":8079,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:14.053549+0000","scrub_duration":1972,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":399048704,"num_objects":6089,"num_object_clones":0,"num_object_copies":18267,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6089,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10273,"num_write_kb":523584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,8],"acting":[4,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"70'139","reported_seq":999,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-08-19T14:22:41.624997+0000","last_change":"2024-08-19T14:19:56.106047+0000","last_active":"2024-08-19T14:22:41.624997+0000","last_peered":"2024-08-19T14:22:41.624997+0000","last_clean":"2024-08-19T14:22:41.624997+0000","last_became_active":"2024-08-19T14:18:26.887657+0000","last_became_peered":"2024-08-19T14:18:26.887657+0000","last_unstale":"2024-08-19T14:22:41.624997+0000","last_undegraded":"2024-08-19T14:22:41.624997+0000","last_fullsized":"2024-08-19T14:22:41.624997+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:19:56.105880+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:19:56.105880+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"380'10251","reported_seq":10441,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-08-19T14:22:43.210557+0000","last_change":"2024-08-19T14:22:08.529439+0000","last_active":"2024-08-19T14:22:43.210557+0000","last_peered":"2024-08-19T14:22:43.210557+0000","last_clean":"2024-08-19T14:22:43.210557+0000","last_became_active":"2024-08-19T14:20:29.208599+0000","last_became_peered":"2024-08-19T14:20:29.208599+0000","last_unstale":"2024-08-19T14:22:43.210557+0000","last_undegraded":"2024-08-19T14:22:43.210557+0000","last_fullsized":"2024-08-19T14:22:43.210557+0000","mapping_epoch":335,"log_start":"338'200","ondisk_log_start":"338'200","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"374'8320","last_scrub_stamp":"2024-08-19T14:22:08.529109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:22:08.529109+0000","objects_scrubbed":7983,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:30.745931+0000","scrub_duration":1962,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393084952,"num_objects":5999,"num_object_clones":0,"num_object_copies":17997,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5999,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10251,"num_write_kb":519937,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,1],"acting":[4,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"70'139","reported_seq":898,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-08-19T14:22:45.668396+0000","last_change":"2024-08-19T14:21:37.951800+0000","last_active":"2024-08-19T14:22:45.668396+0000","last_peered":"2024-08-19T14:22:45.668396+0000","last_clean":"2024-08-19T14:22:45.668396+0000","last_became_active":"2024-08-19T14:18:26.887277+0000","last_became_peered":"2024-08-19T14:18:26.887277+0000","last_unstale":"2024-08-19T14:22:45.668396+0000","last_undegraded":"2024-08-19T14:22:45.668396+0000","last_fullsized":"2024-08-19T14:22:45.668396+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:37.951694+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:21:37.951694+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:03.860882+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,8],"acting":[5,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"380'10219","reported_seq":10400,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-08-19T14:22:43.194033+0000","last_change":"2024-08-19T14:22:13.656515+0000","last_active":"2024-08-19T14:22:43.194033+0000","last_peered":"2024-08-19T14:22:43.194033+0000","last_clean":"2024-08-19T14:22:43.194033+0000","last_became_active":"2024-08-19T14:20:29.208748+0000","last_became_peered":"2024-08-19T14:20:29.208748+0000","last_unstale":"2024-08-19T14:22:43.194033+0000","last_undegraded":"2024-08-19T14:22:43.194033+0000","last_fullsized":"2024-08-19T14:22:43.194033+0000","mapping_epoch":335,"log_start":"338'200","ondisk_log_start":"338'200","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"374'8571","last_scrub_stamp":"2024-08-19T14:22:13.656335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:22:13.656335+0000","objects_scrubbed":7643,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:30.767371+0000","scrub_duration":3043,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":387907584,"num_objects":5919,"num_object_clones":0,"num_object_copies":17757,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5919,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10219,"num_write_kb":516416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"70'139","reported_seq":900,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-08-19T14:22:45.668872+0000","last_change":"2024-08-19T14:22:02.413212+0000","last_active":"2024-08-19T14:22:45.668872+0000","last_peered":"2024-08-19T14:22:45.668872+0000","last_clean":"2024-08-19T14:22:45.668872+0000","last_became_active":"2024-08-19T14:18:26.875179+0000","last_became_peered":"2024-08-19T14:18:26.875179+0000","last_unstale":"2024-08-19T14:22:45.668872+0000","last_undegraded":"2024-08-19T14:22:45.668872+0000","last_fullsized":"2024-08-19T14:22:45.668872+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:22:02.413091+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:22:02.413091+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:23.082273+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.b","version":"381'10484","reported_seq":10661,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-08-19T14:22:46.242814+0000","last_change":"2024-08-19T14:21:39.089676+0000","last_active":"2024-08-19T14:22:46.242814+0000","last_peered":"2024-08-19T14:22:46.242814+0000","last_clean":"2024-08-19T14:22:46.242814+0000","last_became_active":"2024-08-19T14:20:29.207372+0000","last_became_peered":"2024-08-19T14:20:29.207372+0000","last_unstale":"2024-08-19T14:22:46.242814+0000","last_undegraded":"2024-08-19T14:22:46.242814+0000","last_fullsized":"2024-08-19T14:22:46.242814+0000","mapping_epoch":335,"log_start":"339'400","ondisk_log_start":"339'400","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"363'6615","last_scrub_stamp":"2024-08-19T14:21:39.089400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:21:39.089400+0000","objects_scrubbed":6552,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:05.632119+0000","scrub_duration":1907,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":383909888,"num_objects":5858,"num_object_clones":0,"num_object_copies":17574,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10484,"num_write_kb":522944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"70'139","reported_seq":551,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-08-19T14:22:41.630933+0000","last_change":"2024-08-19T14:21:44.434481+0000","last_active":"2024-08-19T14:22:41.630933+0000","last_peered":"2024-08-19T14:22:41.630933+0000","last_clean":"2024-08-19T14:22:41.630933+0000","last_became_active":"2024-08-19T14:18:26.884105+0000","last_became_peered":"2024-08-19T14:18:26.884105+0000","last_unstale":"2024-08-19T14:22:41.630933+0000","last_undegraded":"2024-08-19T14:22:41.630933+0000","last_fullsized":"2024-08-19T14:22:41.630933+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:44.434382+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:20:15.321945+0000","last_clean_scrub_stamp":"2024-08-19T14:21:44.434382+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:13.544241+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"380'10517","reported_seq":10618,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-08-19T14:22:44.377738+0000","last_change":"2024-08-19T14:20:29.269723+0000","last_active":"2024-08-19T14:22:44.377738+0000","last_peered":"2024-08-19T14:22:44.377738+0000","last_clean":"2024-08-19T14:22:44.377738+0000","last_became_active":"2024-08-19T14:20:29.269432+0000","last_became_peered":"2024-08-19T14:20:29.269432+0000","last_unstale":"2024-08-19T14:22:44.377738+0000","last_undegraded":"2024-08-19T14:22:44.377738+0000","last_fullsized":"2024-08-19T14:22:44.377738+0000","mapping_epoch":335,"log_start":"339'500","ondisk_log_start":"339'500","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":399704064,"num_objects":6099,"num_object_clones":0,"num_object_copies":18297,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6099,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10517,"num_write_kb":531712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.f","version":"70'139","reported_seq":806,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-08-19T14:22:45.668529+0000","last_change":"2024-08-19T14:22:42.293240+0000","last_active":"2024-08-19T14:22:45.668529+0000","last_peered":"2024-08-19T14:22:45.668529+0000","last_clean":"2024-08-19T14:22:45.668529+0000","last_became_active":"2024-08-19T14:18:18.711708+0000","last_became_peered":"2024-08-19T14:18:18.711708+0000","last_unstale":"2024-08-19T14:22:45.668529+0000","last_undegraded":"2024-08-19T14:22:45.668529+0000","last_fullsized":"2024-08-19T14:22:45.668529+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:22:42.293171+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:22:42.293171+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:24:10.029244+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"381'10531","reported_seq":10713,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-08-19T14:22:45.899917+0000","last_change":"2024-08-19T14:21:43.974700+0000","last_active":"2024-08-19T14:22:45.899917+0000","last_peered":"2024-08-19T14:22:45.899917+0000","last_clean":"2024-08-19T14:22:45.899917+0000","last_became_active":"2024-08-19T14:20:29.211975+0000","last_became_peered":"2024-08-19T14:20:29.211975+0000","last_unstale":"2024-08-19T14:22:45.899917+0000","last_undegraded":"2024-08-19T14:22:45.899917+0000","last_fullsized":"2024-08-19T14:22:45.899917+0000","mapping_epoch":335,"log_start":"339'500","ondisk_log_start":"339'500","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"364'6991","last_scrub_stamp":"2024-08-19T14:21:43.974613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:21:43.974613+0000","objects_scrubbed":6934,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:07.463188+0000","scrub_duration":1682,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":388431872,"num_objects":5927,"num_object_clones":0,"num_object_copies":17781,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5927,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10531,"num_write_kb":526656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,3],"acting":[11,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.c","version":"70'139","reported_seq":656,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-08-19T14:22:41.629841+0000","last_change":"2024-08-19T14:21:40.181472+0000","last_active":"2024-08-19T14:22:41.629841+0000","last_peered":"2024-08-19T14:22:41.629841+0000","last_clean":"2024-08-19T14:22:41.629841+0000","last_became_active":"2024-08-19T14:18:26.872942+0000","last_became_peered":"2024-08-19T14:18:26.872942+0000","last_unstale":"2024-08-19T14:22:41.629841+0000","last_undegraded":"2024-08-19T14:22:41.629841+0000","last_fullsized":"2024-08-19T14:22:41.629841+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:40.181356+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:20:07.839076+0000","last_clean_scrub_stamp":"2024-08-19T14:21:40.181356+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:22:52.009381+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,9],"acting":[3,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"381'10469","reported_seq":10659,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-08-19T14:22:45.910992+0000","last_change":"2024-08-19T14:22:12.682046+0000","last_active":"2024-08-19T14:22:45.910992+0000","last_peered":"2024-08-19T14:22:45.910992+0000","last_clean":"2024-08-19T14:22:45.910992+0000","last_became_active":"2024-08-19T14:20:29.267442+0000","last_became_peered":"2024-08-19T14:20:29.267442+0000","last_unstale":"2024-08-19T14:22:45.910992+0000","last_undegraded":"2024-08-19T14:22:45.910992+0000","last_fullsized":"2024-08-19T14:22:45.910992+0000","mapping_epoch":335,"log_start":"339'400","ondisk_log_start":"339'400","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"374'8663","last_scrub_stamp":"2024-08-19T14:22:12.681916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:22:12.681916+0000","objects_scrubbed":7729,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:34.638254+0000","scrub_duration":1366,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":386334720,"num_objects":5895,"num_object_clones":0,"num_object_copies":17685,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5895,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10469,"num_write_kb":523648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,9],"acting":[11,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.d","version":"70'139","reported_seq":571,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-08-19T14:22:41.625384+0000","last_change":"2024-08-19T14:21:07.155294+0000","last_active":"2024-08-19T14:22:41.625384+0000","last_peered":"2024-08-19T14:22:41.625384+0000","last_clean":"2024-08-19T14:22:41.625384+0000","last_became_active":"2024-08-19T14:18:26.882002+0000","last_became_peered":"2024-08-19T14:18:26.882002+0000","last_unstale":"2024-08-19T14:22:41.625384+0000","last_undegraded":"2024-08-19T14:22:41.625384+0000","last_fullsized":"2024-08-19T14:22:41.625384+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:07.155157+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:21:07.155157+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"70'139","reported_seq":832,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-08-19T14:22:45.671914+0000","last_change":"2024-08-19T14:21:13.777891+0000","last_active":"2024-08-19T14:22:45.671914+0000","last_peered":"2024-08-19T14:22:45.671914+0000","last_clean":"2024-08-19T14:22:45.671914+0000","last_became_active":"2024-08-19T14:18:18.703594+0000","last_became_peered":"2024-08-19T14:18:18.703594+0000","last_unstale":"2024-08-19T14:22:45.671914+0000","last_undegraded":"2024-08-19T14:22:45.671914+0000","last_fullsized":"2024-08-19T14:22:45.671914+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:13.777777+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:21:13.777777+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:22:46.272172+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"70'139","reported_seq":882,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-08-19T14:22:41.636031+0000","last_change":"2024-08-19T14:20:07.692054+0000","last_active":"2024-08-19T14:22:41.636031+0000","last_peered":"2024-08-19T14:22:41.636031+0000","last_clean":"2024-08-19T14:22:41.636031+0000","last_became_active":"2024-08-19T14:18:18.698334+0000","last_became_peered":"2024-08-19T14:18:18.698334+0000","last_unstale":"2024-08-19T14:22:41.636031+0000","last_undegraded":"2024-08-19T14:22:41.636031+0000","last_fullsized":"2024-08-19T14:22:41.636031+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:07.691932+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:07.691932+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.10","version":"70'139","reported_seq":995,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-08-19T14:22:45.668266+0000","last_change":"2024-08-19T14:22:11.260404+0000","last_active":"2024-08-19T14:22:45.668266+0000","last_peered":"2024-08-19T14:22:45.668266+0000","last_clean":"2024-08-19T14:22:45.668266+0000","last_became_active":"2024-08-19T14:18:18.704562+0000","last_became_peered":"2024-08-19T14:18:18.704562+0000","last_unstale":"2024-08-19T14:22:45.668266+0000","last_undegraded":"2024-08-19T14:22:45.668266+0000","last_fullsized":"2024-08-19T14:22:45.668266+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:22:11.260286+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:22:11.260286+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:36.552471+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,8],"acting":[5,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"70'139","reported_seq":544,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-08-19T14:22:45.682291+0000","last_change":"2024-08-19T14:22:21.104595+0000","last_active":"2024-08-19T14:22:45.682291+0000","last_peered":"2024-08-19T14:22:45.682291+0000","last_clean":"2024-08-19T14:22:45.682291+0000","last_became_active":"2024-08-19T14:18:18.697894+0000","last_became_peered":"2024-08-19T14:18:18.697894+0000","last_unstale":"2024-08-19T14:22:45.682291+0000","last_undegraded":"2024-08-19T14:22:45.682291+0000","last_fullsized":"2024-08-19T14:22:45.682291+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:22:21.104491+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:21:16.071703+0000","last_clean_scrub_stamp":"2024-08-19T14:22:21.104491+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:45.856425+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"70'139","reported_seq":511,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-08-19T14:22:41.631523+0000","last_change":"2024-08-19T14:20:28.205212+0000","last_active":"2024-08-19T14:22:41.631523+0000","last_peered":"2024-08-19T14:22:41.631523+0000","last_clean":"2024-08-19T14:22:41.631523+0000","last_became_active":"2024-08-19T14:20:28.204724+0000","last_became_peered":"2024-08-19T14:20:28.204724+0000","last_unstale":"2024-08-19T14:22:41.631523+0000","last_undegraded":"2024-08-19T14:22:41.631523+0000","last_fullsized":"2024-08-19T14:22:41.631523+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:08.680021+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:08.680021+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"70'139","reported_seq":639,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-08-19T14:22:41.621521+0000","last_change":"2024-08-19T14:22:02.722505+0000","last_active":"2024-08-19T14:22:41.621521+0000","last_peered":"2024-08-19T14:22:41.621521+0000","last_clean":"2024-08-19T14:22:41.621521+0000","last_became_active":"2024-08-19T14:18:31.901556+0000","last_became_peered":"2024-08-19T14:18:31.901556+0000","last_unstale":"2024-08-19T14:22:41.621521+0000","last_undegraded":"2024-08-19T14:22:41.621521+0000","last_fullsized":"2024-08-19T14:22:41.621521+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:22:02.722345+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:22:02.722345+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:13.381470+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,0],"acting":[8,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.14","version":"70'139","reported_seq":640,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-08-19T14:22:41.625106+0000","last_change":"2024-08-19T14:19:55.130518+0000","last_active":"2024-08-19T14:22:41.625106+0000","last_peered":"2024-08-19T14:22:41.625106+0000","last_clean":"2024-08-19T14:22:41.625106+0000","last_became_active":"2024-08-19T14:18:18.703926+0000","last_became_peered":"2024-08-19T14:18:18.703926+0000","last_unstale":"2024-08-19T14:22:41.625106+0000","last_undegraded":"2024-08-19T14:22:41.625106+0000","last_fullsized":"2024-08-19T14:22:41.625106+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:19:55.130420+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:19:55.130420+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,1],"acting":[7,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.15","version":"70'139","reported_seq":527,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-08-19T14:22:41.625373+0000","last_change":"2024-08-19T14:20:09.181046+0000","last_active":"2024-08-19T14:22:41.625373+0000","last_peered":"2024-08-19T14:22:41.625373+0000","last_clean":"2024-08-19T14:22:41.625373+0000","last_became_active":"2024-08-19T14:18:31.912496+0000","last_became_peered":"2024-08-19T14:18:31.912496+0000","last_unstale":"2024-08-19T14:22:41.625373+0000","last_undegraded":"2024-08-19T14:22:41.625373+0000","last_fullsized":"2024-08-19T14:22:41.625373+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:09.180923+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:20:09.180923+0000","last_clean_scrub_stamp":"2024-08-19T14:20:09.180923+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"70'139","reported_seq":513,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-08-19T14:22:41.630742+0000","last_change":"2024-08-19T14:21:40.430835+0000","last_active":"2024-08-19T14:22:41.630742+0000","last_peered":"2024-08-19T14:22:41.630742+0000","last_clean":"2024-08-19T14:22:41.630742+0000","last_became_active":"2024-08-19T14:18:31.901229+0000","last_became_peered":"2024-08-19T14:18:31.901229+0000","last_unstale":"2024-08-19T14:22:41.630742+0000","last_undegraded":"2024-08-19T14:22:41.630742+0000","last_fullsized":"2024-08-19T14:22:41.630742+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:40.430714+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:21:40.430714+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:04.072653+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1b","version":"70'139","reported_seq":865,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-08-19T14:22:41.621285+0000","last_change":"2024-08-19T14:21:48.693864+0000","last_active":"2024-08-19T14:22:41.621285+0000","last_peered":"2024-08-19T14:22:41.621285+0000","last_clean":"2024-08-19T14:22:41.621285+0000","last_became_active":"2024-08-19T14:18:31.901152+0000","last_became_peered":"2024-08-19T14:18:31.901152+0000","last_unstale":"2024-08-19T14:22:41.621285+0000","last_undegraded":"2024-08-19T14:22:41.621285+0000","last_fullsized":"2024-08-19T14:22:41.621285+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:48.693761+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:21:48.693761+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:07.571130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.18","version":"70'139","reported_seq":911,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-08-19T14:22:45.668266+0000","last_change":"2024-08-19T14:22:07.254887+0000","last_active":"2024-08-19T14:22:45.668266+0000","last_peered":"2024-08-19T14:22:45.668266+0000","last_clean":"2024-08-19T14:22:45.668266+0000","last_became_active":"2024-08-19T14:18:31.902735+0000","last_became_peered":"2024-08-19T14:18:31.902735+0000","last_unstale":"2024-08-19T14:22:45.668266+0000","last_undegraded":"2024-08-19T14:22:45.668266+0000","last_fullsized":"2024-08-19T14:22:45.668266+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:22:07.254767+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:22:07.254767+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:11.195110+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.19","version":"70'139","reported_seq":859,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-08-19T14:22:45.682472+0000","last_change":"2024-08-19T14:21:40.219949+0000","last_active":"2024-08-19T14:22:45.682472+0000","last_peered":"2024-08-19T14:22:45.682472+0000","last_clean":"2024-08-19T14:22:45.682472+0000","last_became_active":"2024-08-19T14:18:31.902547+0000","last_became_peered":"2024-08-19T14:18:31.902547+0000","last_unstale":"2024-08-19T14:22:45.682472+0000","last_undegraded":"2024-08-19T14:22:45.682472+0000","last_fullsized":"2024-08-19T14:22:45.682472+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:40.219830+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:21:40.219830+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:05.414864+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1e","version":"70'139","reported_seq":518,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-08-19T14:22:45.668812+0000","last_change":"2024-08-19T14:22:09.404540+0000","last_active":"2024-08-19T14:22:45.668812+0000","last_peered":"2024-08-19T14:22:45.668812+0000","last_clean":"2024-08-19T14:22:45.668812+0000","last_became_active":"2024-08-19T14:18:31.902546+0000","last_became_peered":"2024-08-19T14:18:31.902546+0000","last_unstale":"2024-08-19T14:22:45.668812+0000","last_undegraded":"2024-08-19T14:22:45.668812+0000","last_fullsized":"2024-08-19T14:22:45.668812+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:22:09.404433+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:22:09.404433+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:13.587290+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1c","version":"70'139","reported_seq":623,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-08-19T14:22:41.621495+0000","last_change":"2024-08-19T14:21:38.753663+0000","last_active":"2024-08-19T14:22:41.621495+0000","last_peered":"2024-08-19T14:22:41.621495+0000","last_clean":"2024-08-19T14:22:41.621495+0000","last_became_active":"2024-08-19T14:18:31.902494+0000","last_became_peered":"2024-08-19T14:18:31.902494+0000","last_unstale":"2024-08-19T14:22:41.621495+0000","last_undegraded":"2024-08-19T14:22:41.621495+0000","last_fullsized":"2024-08-19T14:22:41.621495+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:38.753553+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:21:38.753553+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:22:46.720742+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,11],"acting":[8,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1d","version":"70'139","reported_seq":537,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-08-19T14:22:41.631526+0000","last_change":"2024-08-19T14:20:17.725615+0000","last_active":"2024-08-19T14:22:41.631526+0000","last_peered":"2024-08-19T14:22:41.631526+0000","last_clean":"2024-08-19T14:22:41.631526+0000","last_became_active":"2024-08-19T14:18:31.901183+0000","last_became_peered":"2024-08-19T14:18:31.901183+0000","last_unstale":"2024-08-19T14:22:41.631526+0000","last_undegraded":"2024-08-19T14:22:41.631526+0000","last_fullsized":"2024-08-19T14:22:41.631526+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:17.725477+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:17.725477+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,5],"acting":[1,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":6238240792,"num_objects":95189,"num_object_clones":0,"num_object_copies":285567,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":885,"num_objects_unfound":0,"num_objects_dirty":95189,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":167157,"num_write_kb":8395009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3255,"num_bytes_recovered":213319680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18946994176,"data_stored":18946981960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160757,"ondisk_log_size":160757,"up":48,"acting":47,"num_store_stats":9},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1802240,"data_stored":1788032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187593,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3089644,"kb_used_data":2653092,"kb_used_omap":1,"kb_used_meta":436542,"kb_avail":90655508,"statfs":{"total":95995035648,"available":92831240192,"internally_reserved":0,"allocated":2716766208,"data_stored":2715016888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":447019460},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":343597383823,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":74,"seq":317827580050,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1986420,"kb_used_data":1495720,"kb_used_omap":1,"kb_used_meta":490686,"kb_avail":91758732,"statfs":{"total":95995035648,"available":93960941568,"internally_reserved":0,"allocated":1531617280,"data_stored":1529863888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":502462913},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874392,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":770668,"kb_used_data":388260,"kb_used_omap":1,"kb_used_meta":382398,"kb_avail":92974484,"statfs":{"total":95995035648,"available":95205871616,"internally_reserved":0,"allocated":397578240,"data_stored":395828920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":391576005},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":57,"seq":244813136026,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3569592,"kb_used_data":3101100,"kb_used_omap":1,"kb_used_meta":468478,"kb_avail":90175560,"statfs":{"total":95995035648,"available":92339773440,"internally_reserved":0,"allocated":3175526400,"data_stored":3173773528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":479721930},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":206158430366,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692003,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1771588,"kb_used_data":1407224,"kb_used_omap":1,"kb_used_meta":364350,"kb_avail":91973564,"statfs":{"total":95995035648,"available":94180929536,"internally_reserved":0,"allocated":1440997376,"data_stored":1439240328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":373094853},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986343,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2422644,"kb_used_data":1936104,"kb_used_omap":1,"kb_used_meta":486526,"kb_avail":91322508,"statfs":{"total":95995035648,"available":93514248192,"internally_reserved":0,"allocated":1982570496,"data_stored":1980817104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":498203077},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149869,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2741484,"kb_used_data":2295204,"kb_used_omap":1,"kb_used_meta":446270,"kb_avail":91003668,"statfs":{"total":95995035648,"available":93187756032,"internally_reserved":0,"allocated":2350288896,"data_stored":2348539576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":456980935},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411507,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575030,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4400568,"kb_used_data":3832112,"kb_used_omap":1,"kb_used_meta":568446,"kb_avail":89344584,"statfs":{"total":95995035648,"available":91488854016,"internally_reserved":0,"allocated":3924082688,"data_stored":3922325744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":582089157},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,14],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738555,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1875692,"kb_used_data":1511268,"kb_used_omap":1,"kb_used_meta":364350,"kb_avail":91869460,"statfs":{"total":95995035648,"available":94074327040,"internally_reserved":0,"allocated":1547538432,"data_stored":1545789112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":373094853},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1527316480,"data_stored":1527316480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3913748480,"data_stored":3913744408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2334785536,"data_stored":2334785536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1970278400,"data_stored":1970274328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1430323200,"data_stored":1430323200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3156541440,"data_stored":3156541440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":392036352,"data_stored":392036352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1523847168,"data_stored":1523843096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":2698117120,"data_stored":2698117120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T14:22:49.494 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:22:49.495 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:22:49.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:49.523 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:49.523 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:22:49.624 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:22:49.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:49.652 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:49.652 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:22:49.753 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:22:49.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:49.797 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:49.797 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:22:49.897 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:22:49.925 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:49.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:49.926 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:22:50.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:49 smithi032 ceph-mon[20004]: pgmap v868: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.8 GiB data, 22 GiB used, 783 GiB / 805 GiB avail; 0 B/s wr, 730 op/s; 885/285573 objects misplaced (0.310%); 1.8 MiB/s, 28 objects/s recovering 2024-08-19T14:22:50.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:49 smithi032 ceph-mon[20004]: from='client.25555 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:22:50.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:49 smithi032 ceph-mon[20004]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:22:50.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:49 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3258105175' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:22:50.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:49 smithi032 ceph-mon[20004]: osdmap e382: 12 total, 12 up, 9 in 2024-08-19T14:22:50.026 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:22:50.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:49 smithi045 ceph-mon[20143]: pgmap v868: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.8 GiB data, 22 GiB used, 783 GiB / 805 GiB avail; 0 B/s wr, 730 op/s; 885/285573 objects misplaced (0.310%); 1.8 MiB/s, 28 objects/s recovering 2024-08-19T14:22:50.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:49 smithi045 ceph-mon[20143]: from='client.25555 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:22:50.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:49 smithi045 ceph-mon[20143]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:22:50.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:49 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3258105175' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:22:50.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:49 smithi045 ceph-mon[20143]: osdmap e382: 12 total, 12 up, 9 in 2024-08-19T14:22:50.066 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:50.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:50.067 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:22:50.085 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:49 smithi031 ceph-mon[16336]: pgmap v868: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.8 GiB data, 22 GiB used, 783 GiB / 805 GiB avail; 0 B/s wr, 730 op/s; 885/285573 objects misplaced (0.310%); 1.8 MiB/s, 28 objects/s recovering 2024-08-19T14:22:50.086 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:49 smithi031 ceph-mon[16336]: from='client.25555 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:22:50.086 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:49 smithi031 ceph-mon[16336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:22:50.086 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:49 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3258105175' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:22:50.086 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:49 smithi031 ceph-mon[16336]: osdmap e382: 12 total, 12 up, 9 in 2024-08-19T14:22:50.134 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-08-19T14:22:50.168 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:22:50.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:50.207 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:50.207 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:22:50.308 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:22:50.333 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:50.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:50.334 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:22:50.435 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:22:50.456 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:50.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:50.457 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:22:50.558 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:22:50.592 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:50.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:50.593 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:22:50.695 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:22:50.728 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:50.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:50.730 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:22:50.831 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:22:50.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:50 smithi031 ceph-mon[16336]: 1.1c scrub starts 2024-08-19T14:22:50.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:50 smithi031 ceph-mon[16336]: 1.1c scrub ok 2024-08-19T14:22:50.870 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:50.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:50.872 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:22:50.973 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:22:51.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:50 smithi032 ceph-mon[20004]: 1.1c scrub starts 2024-08-19T14:22:51.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:50 smithi032 ceph-mon[20004]: 1.1c scrub ok 2024-08-19T14:22:51.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:51.015 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:51.015 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:22:51.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:50 smithi045 ceph-mon[20143]: 1.1c scrub starts 2024-08-19T14:22:51.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:50 smithi045 ceph-mon[20143]: 1.1c scrub ok 2024-08-19T14:22:51.053 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:22:51.092 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:22:51.116 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:22:51.139 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:51.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:51.140 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:22:51.242 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:22:51.287 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:51.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:51.289 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:22:51.390 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:22:51.412 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:51.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:51.413 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:22:51.514 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:22:51.538 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:51.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:51.539 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:22:51.640 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:22:51.664 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:51.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:51.666 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:22:51.728 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:51 smithi031 ceph-mon[16336]: pgmap v870: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.6 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 0 B/s wr, 891 op/s; 637/277200 objects misplaced (0.230%); 3.7 MiB/s, 59 objects/s recovering 2024-08-19T14:22:51.767 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:22:51.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:51.795 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:51.795 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:22:51.824 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:22:51.824 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:22:51.862 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:22:51.862 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:22:51.896 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:22:51.905 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:22:51.905 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:22:51.920 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:51.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:51.922 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:22:51.955 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:22:51.955 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:22:51.983 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:22:51.984 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:22:52.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:51 smithi032 ceph-mon[20004]: pgmap v870: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.6 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 0 B/s wr, 891 op/s; 637/277200 objects misplaced (0.230%); 3.7 MiB/s, 59 objects/s recovering 2024-08-19T14:22:52.019 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:22:52.019 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:22:52.023 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:22:52.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:52.046 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:52.046 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:22:52.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:51 smithi045 ceph-mon[20143]: pgmap v870: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.6 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 0 B/s wr, 891 op/s; 637/277200 objects misplaced (0.230%); 3.7 MiB/s, 59 objects/s recovering 2024-08-19T14:22:52.147 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:22:52.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:52.186 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:52.186 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:22:52.193 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:22:52.193 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [ 2024-08-19T14:22:52.193 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:52.193 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25333.0:173824 4.0 4:0c880876:::benchmark_data_smithi045_71562_object42649:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e382)", 2024-08-19T14:22:52.193 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:22:52.183393+0000", 2024-08-19T14:22:52.193 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.0084964229999999995, 2024-08-19T14:22:52.193 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0085466940000000005, 2024-08-19T14:22:52.193 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:22:52.193 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:22:52.194 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "waiting for sub ops", 2024-08-19T14:22:52.194 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:22:52.194 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25333", 2024-08-19T14:22:52.194 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:22:52.194 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 173824 2024-08-19T14:22:52.194 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:52.194 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:22:52.194 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:52.194 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:22:52.194 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:52.183393+0000", 2024-08-19T14:22:52.194 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:22:52.194 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:52.194 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:52.195 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:22:52.195 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:52.183393+0000", 2024-08-19T14:22:52.195 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:22:52.195 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:52.195 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:52.195 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:22:52.195 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:52.183394+0000", 2024-08-19T14:22:52.195 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.176e-06 2024-08-19T14:22:52.195 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:52.195 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:52.195 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:22:52.195 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:52.183399+0000", 2024-08-19T14:22:52.195 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1150000000000002e-06 2024-08-19T14:22:52.195 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:52.196 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:52.196 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:22:52.196 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:52.183412+0000", 2024-08-19T14:22:52.196 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3195e-05 2024-08-19T14:22:52.196 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:52.196 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:52.196 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:22:52.196 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:52.183420+0000", 2024-08-19T14:22:52.196 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.3239999999999997e-06 2024-08-19T14:22:52.196 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:52.196 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:52.196 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:22:52.196 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:52.183468+0000", 2024-08-19T14:22:52.196 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.8452999999999998e-05 2024-08-19T14:22:52.197 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:52.197 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:52.197 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:22:52.197 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:52.183584+0000", 2024-08-19T14:22:52.197 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000115631 2024-08-19T14:22:52.197 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:52.197 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:52.197 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,7", 2024-08-19T14:22:52.197 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:52.183697+0000", 2024-08-19T14:22:52.197 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00011325800000000001 2024-08-19T14:22:52.197 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:52.197 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:52.197 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:22:52.198 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:52.189095+0000", 2024-08-19T14:22:52.198 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0053975430000000003 2024-08-19T14:22:52.198 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:52.198 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:52.198 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:22:52.198 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:52.189902+0000", 2024-08-19T14:22:52.198 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00080699099999999996 2024-08-19T14:22:52.198 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:22:52.198 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:22:52.198 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:22:52.198 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:52.198 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:52.198 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25333.0:173825 4.0 4:004f3c19:::benchmark_data_smithi045_71562_object42650:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e382)", 2024-08-19T14:22:52.198 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:22:52.190402+0000", 2024-08-19T14:22:52.199 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.001487436, 2024-08-19T14:22:52.199 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0016264459999999999, 2024-08-19T14:22:52.199 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:22:52.199 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:22:52.199 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "waiting for sub ops", 2024-08-19T14:22:52.199 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:22:52.199 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25333", 2024-08-19T14:22:52.199 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:22:52.199 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 173825 2024-08-19T14:22:52.199 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:52.199 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:22:52.199 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:52.199 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:22:52.199 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:52.190402+0000", 2024-08-19T14:22:52.200 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:22:52.200 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:52.200 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:52.200 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:22:52.200 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:52.190402+0000", 2024-08-19T14:22:52.200 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:22:52.200 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:52.200 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:52.200 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:22:52.200 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:52.190404+0000", 2024-08-19T14:22:52.200 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.829e-06 2024-08-19T14:22:52.200 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:52.200 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:52.200 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:22:52.201 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:52.190412+0000", 2024-08-19T14:22:52.201 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.4130000000000006e-06 2024-08-19T14:22:52.201 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:52.201 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:52.201 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:22:52.201 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:52.190429+0000", 2024-08-19T14:22:52.201 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.7367000000000002e-05 2024-08-19T14:22:52.201 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:52.201 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:52.201 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:22:52.201 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:52.190438+0000", 2024-08-19T14:22:52.201 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.4829999999999999e-06 2024-08-19T14:22:52.201 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:52.201 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:52.202 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:22:52.202 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:52.190507+0000", 2024-08-19T14:22:52.202 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.9327999999999998e-05 2024-08-19T14:22:52.202 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:52.202 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:52.202 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:22:52.202 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:52.190833+0000", 2024-08-19T14:22:52.202 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00032585399999999999 2024-08-19T14:22:52.202 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:52.202 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:52.202 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,7", 2024-08-19T14:22:52.202 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:52.191445+0000", 2024-08-19T14:22:52.202 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00061179099999999998 2024-08-19T14:22:52.202 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:22:52.203 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:22:52.203 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:22:52.203 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:52.203 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:52.203 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.25333.0:173828 4.9 e382/335)", 2024-08-19T14:22:52.203 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:22:52.190890+0000", 2024-08-19T14:22:52.203 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.00099880200000000002, 2024-08-19T14:22:52.203 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00120239, 2024-08-19T14:22:52.203 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:22:52.203 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:22:52.203 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "queued for pg", 2024-08-19T14:22:52.203 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:22:52.203 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:52.203 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:22:52.203 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:52.190890+0000", 2024-08-19T14:22:52.204 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:22:52.204 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:52.204 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:52.204 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:22:52.204 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:52.190890+0000", 2024-08-19T14:22:52.204 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:22:52.204 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:52.204 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:52.204 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:22:52.204 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:52.190891+0000", 2024-08-19T14:22:52.204 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.3699999999999999e-07 2024-08-19T14:22:52.204 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:52.204 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:52.204 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:22:52.205 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:52.190897+0000", 2024-08-19T14:22:52.205 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.0039999999999996e-06 2024-08-19T14:22:52.205 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:52.205 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:52.205 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:22:52.205 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:52.190909+0000", 2024-08-19T14:22:52.205 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1413000000000001e-05 2024-08-19T14:22:52.205 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:22:52.205 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:22:52.205 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:22:52.205 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:52.190916+0000", 2024-08-19T14:22:52.205 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.1219999999999997e-06 2024-08-19T14:22:52.205 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:22:52.205 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:22:52.205 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:22:52.206 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:22:52.206 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-08-19T14:22:52.206 INFO:teuthology.orchestra.run.smithi031.stdout: "num_ops": 3 2024-08-19T14:22:52.206 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:22:52.254 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:22:52.255 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:22:52.287 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:22:52.292 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:22:52.293 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:22:52.320 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:52.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:52.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:22:52.326 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:22:52.326 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:22:52.357 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:22:52.357 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:22:52.397 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:22:52.397 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:22:52.422 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:22:52.427 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:22:52.427 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:22:52.465 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:52.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:52.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:22:52.495 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-19T14:22:52.567 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:22:52.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:52.593 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:52.593 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:22:52.695 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:22:52.717 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:52.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:52.719 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:22:52.796 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:52 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3311029898' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:22:52.796 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:52 smithi031 ceph-mon[16336]: 1.c deep-scrub starts 2024-08-19T14:22:52.797 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:52 smithi031 ceph-mon[16336]: 1.c deep-scrub ok 2024-08-19T14:22:52.819 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:22:52.843 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:52.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:52.844 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:22:52.941 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-19T14:22:52.944 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:22:52.968 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:52.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:52.969 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:22:53.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:52 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3311029898' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:22:53.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:52 smithi032 ceph-mon[20004]: 1.c deep-scrub starts 2024-08-19T14:22:53.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:52 smithi032 ceph-mon[20004]: 1.c deep-scrub ok 2024-08-19T14:22:53.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:52 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3311029898' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:22:53.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:52 smithi045 ceph-mon[20143]: 1.c deep-scrub starts 2024-08-19T14:22:53.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:52 smithi045 ceph-mon[20143]: 1.c deep-scrub ok 2024-08-19T14:22:53.070 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:22:53.116 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:53.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:53.117 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:22:53.218 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:22:53.245 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:53.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:53.246 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:22:53.343 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-08-19T14:22:53.347 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:22:53.369 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:53.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:53.371 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:22:53.472 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:22:53.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:53.509 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:53.509 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:22:53.610 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:22:53.635 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:53.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:53.636 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:22:53.737 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:22:53.765 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:53.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:53.767 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:22:53.868 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:22:53.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:53.895 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:53.895 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:22:53.996 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:22:54.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:53 smithi032 ceph-mon[20004]: pgmap v871: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.5 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 0 B/s wr, 973 op/s; 637/272067 objects misplaced (0.234%); 3.7 MiB/s, 59 objects/s recovering 2024-08-19T14:22:54.022 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:54.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:54.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:22:54.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:53 smithi045 ceph-mon[20143]: pgmap v871: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.5 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 0 B/s wr, 973 op/s; 637/272067 objects misplaced (0.234%); 3.7 MiB/s, 59 objects/s recovering 2024-08-19T14:22:54.086 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:53 smithi031 ceph-mon[16336]: pgmap v871: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.5 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 0 B/s wr, 973 op/s; 637/272067 objects misplaced (0.234%); 3.7 MiB/s, 59 objects/s recovering 2024-08-19T14:22:54.123 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:22:54.147 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:54.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:54.148 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:22:54.186 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:22:54.211 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:22:54.249 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:22:54.253 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:22:54.273 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:54.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:54.274 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:22:54.375 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:22:54.428 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:54.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:54.430 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:22:54.531 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:22:54.552 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:54.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:54.553 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:22:54.654 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:22:54.676 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:54.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:54.677 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:22:54.777 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:22:54.802 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:54.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:54.804 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:22:54.905 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:22:54.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:54.943 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:54.943 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:22:55.044 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:22:55.069 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:55.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:55.071 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:22:55.172 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:22:55.198 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:55.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:55.199 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:22:55.300 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:22:55.323 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:55.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:55.324 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:22:55.426 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:22:55.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:55.472 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:55.472 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:22:55.574 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:22:55.614 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:55.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:55.616 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:22:55.698 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:22:55.699 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-08-19T14:22:55.699 INFO:teuthology.orchestra.run.smithi031.stdout: "complaint_time": 30, 2024-08-19T14:22:55.699 INFO:teuthology.orchestra.run.smithi031.stdout: "num_blocked_ops": 0 2024-08-19T14:22:55.699 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:22:55.705 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:22:55.707 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-08-19T14:22:55.717 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:22:55.721 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is set 2024-08-19T14:22:55.740 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:55.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:55.741 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:22:55.842 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:22:55.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:55 smithi031 ceph-mon[16336]: pgmap v872: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.4 GiB data, 20 GiB used, 784 GiB / 805 GiB avail; 0 B/s wr, 983 op/s; 476/266157 objects misplaced (0.179%); 3.1 MiB/s, 49 objects/s recovering 2024-08-19T14:22:55.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:55 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/580936782' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:22:55.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:55 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:22:55.876 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:55.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:55.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:22:55.978 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:22:56.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:55 smithi032 ceph-mon[20004]: pgmap v872: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.4 GiB data, 20 GiB used, 784 GiB / 805 GiB avail; 0 B/s wr, 983 op/s; 476/266157 objects misplaced (0.179%); 3.1 MiB/s, 49 objects/s recovering 2024-08-19T14:22:56.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:55 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/580936782' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:22:56.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:55 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:22:56.014 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:56.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:56.015 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:22:56.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:55 smithi045 ceph-mon[20143]: pgmap v872: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.4 GiB data, 20 GiB used, 784 GiB / 805 GiB avail; 0 B/s wr, 983 op/s; 476/266157 objects misplaced (0.179%); 3.1 MiB/s, 49 objects/s recovering 2024-08-19T14:22:56.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:55 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/580936782' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:22:56.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:55 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:22:56.115 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:22:56.144 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:56.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:56.145 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:22:56.150 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:22:56.246 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:22:56.270 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:56.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:56.271 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:22:56.372 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:22:56.415 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:56.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:56.416 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:22:56.517 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:22:56.544 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:56.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:56.545 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:22:56.646 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:22:56.685 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:56.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:56.686 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:22:56.787 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:22:56.816 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:56.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:56.817 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:22:56.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:56 smithi031 ceph-mon[16336]: from='client.15882 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:22:56.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:56 smithi031 ceph-mon[16336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:22:56.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:56 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:22:56.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:56 smithi031 ceph-mon[16336]: osdmap e383: 12 total, 12 up, 9 in 2024-08-19T14:22:56.918 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:22:56.939 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:56.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:56.940 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:22:57.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:56 smithi032 ceph-mon[20004]: from='client.15882 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:22:57.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:56 smithi032 ceph-mon[20004]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:22:57.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:56 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:22:57.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:56 smithi032 ceph-mon[20004]: osdmap e383: 12 total, 12 up, 9 in 2024-08-19T14:22:57.040 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:22:57.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:56 smithi045 ceph-mon[20143]: from='client.15882 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:22:57.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:56 smithi045 ceph-mon[20143]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:22:57.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:56 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:22:57.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:56 smithi045 ceph-mon[20143]: osdmap e383: 12 total, 12 up, 9 in 2024-08-19T14:22:57.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:57.069 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:57.069 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:22:57.171 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:22:57.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:57.205 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:57.205 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:22:57.306 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:22:57.335 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:57.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:57.337 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:22:57.428 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-08-19T14:22:57.438 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:22:57.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:57.518 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:57.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:22:57.620 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:22:57.642 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:57.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:57.643 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:22:57.723 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:57 smithi031 ceph-mon[16336]: pgmap v874: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.4 GiB data, 20 GiB used, 784 GiB / 805 GiB avail; 0 B/s wr, 842 op/s; 476/265353 objects misplaced (0.179%); 3.2 MiB/s, 51 objects/s recovering 2024-08-19T14:22:57.744 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:22:57.771 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:57.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:57.772 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:22:57.872 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:22:57.895 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:57.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:57.896 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:22:57.898 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":872,"stamp":"2024-08-19T14:22:53.922805+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5814682168,"num_objects":88719,"num_object_clones":0,"num_object_copies":266157,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":476,"num_objects_unfound":0,"num_objects_dirty":88719,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":173629,"num_write_kb":8395009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3664,"num_bytes_recovered":240123904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":165138,"ondisk_log_size":165138,"up":141,"acting":140,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":142,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":21394072,"kb_used_data":17388608,"kb_used_omap":14,"kb_used_meta":4005297,"kb_avail":822312296,"statfs":{"total":863955320832,"available":842047791104,"internally_reserved":0,"allocated":17805934592,"data_stored":17790092222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14353,"internal_metadata":4101425135},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,6,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-532414464,"num_objects":-8124,"num_object_clones":0,"num_object_copies":-24372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-409,"num_objects_unfound":0,"num_objects_dirty":-8124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8124,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":409,"num_bytes_recovered":26804224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.257677"},"pg_stats":[{"pgid":"4.5","version":"382'10906","reported_seq":11095,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-08-19T14:22:50.919658+0000","last_change":"2024-08-19T14:22:14.211513+0000","last_active":"2024-08-19T14:22:50.919658+0000","last_peered":"2024-08-19T14:22:50.919658+0000","last_clean":"2024-08-19T14:22:50.919658+0000","last_became_active":"2024-08-19T14:20:29.267549+0000","last_became_peered":"2024-08-19T14:20:29.267549+0000","last_unstale":"2024-08-19T14:22:50.919658+0000","last_undegraded":"2024-08-19T14:22:50.919658+0000","last_fullsized":"2024-08-19T14:22:50.919658+0000","mapping_epoch":335,"log_start":"339'900","ondisk_log_start":"339'900","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"375'8747","last_scrub_stamp":"2024-08-19T14:22:14.211361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:22:14.211361+0000","objects_scrubbed":7717,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:26.018227+0000","scrub_duration":1909,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":360054784,"num_objects":5494,"num_object_clones":0,"num_object_copies":16482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10906,"num_write_kb":524800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,0],"acting":[11,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"70'139","reported_seq":981,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-08-19T14:22:48.701871+0000","last_change":"2024-08-19T14:19:58.072227+0000","last_active":"2024-08-19T14:22:48.701871+0000","last_peered":"2024-08-19T14:22:48.701871+0000","last_clean":"2024-08-19T14:22:48.701871+0000","last_became_active":"2024-08-19T14:18:13.807133+0000","last_became_peered":"2024-08-19T14:18:13.807133+0000","last_unstale":"2024-08-19T14:22:48.701871+0000","last_undegraded":"2024-08-19T14:22:48.701871+0000","last_fullsized":"2024-08-19T14:22:48.701871+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:19:58.072083+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:19:58.072083+0000","last_clean_scrub_stamp":"2024-08-19T14:19:58.072083+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,9],"acting":[7,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"382'11050","reported_seq":18486,"reported_epoch":382,"state":"active+recovering+undersized+remapped","last_fresh":"2024-08-19T14:22:52.595387+0000","last_change":"2024-08-19T14:21:07.239135+0000","last_active":"2024-08-19T14:22:52.595387+0000","last_peered":"2024-08-19T14:22:52.595387+0000","last_clean":"2024-08-19T14:21:04.784061+0000","last_became_active":"2024-08-19T14:21:07.213050+0000","last_became_peered":"2024-08-19T14:21:07.213050+0000","last_unstale":"2024-08-19T14:22:52.595387+0000","last_undegraded":"2024-08-19T14:22:52.595387+0000","last_fullsized":"2024-08-19T14:21:06.805412+0000","mapping_epoch":354,"log_start":"340'1000","ondisk_log_start":"340'1000","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":368836608,"num_objects":5628,"num_object_clones":0,"num_object_copies":16884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":476,"num_objects_unfound":0,"num_objects_dirty":5628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11050,"num_write_kb":533696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3664,"num_bytes_recovered":240123904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,7],"acting":[1,7],"avail_no_missing":["1","5","7"],"object_location_counts":[{"shards":"1,5,7","objects":476},{"shards":"1,7","objects":5152}],"blocked_by":[],"up_primary":9,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"70'139","reported_seq":879,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-08-19T14:22:48.702726+0000","last_change":"2024-08-19T14:20:03.714917+0000","last_active":"2024-08-19T14:22:48.702726+0000","last_peered":"2024-08-19T14:22:48.702726+0000","last_clean":"2024-08-19T14:22:48.702726+0000","last_became_active":"2024-08-19T14:18:13.804721+0000","last_became_peered":"2024-08-19T14:18:13.804721+0000","last_unstale":"2024-08-19T14:22:48.702726+0000","last_undegraded":"2024-08-19T14:22:48.702726+0000","last_fullsized":"2024-08-19T14:22:48.702726+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:03.714818+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:03.714818+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"382'10948","reported_seq":11053,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-08-19T14:22:52.614314+0000","last_change":"2024-08-19T14:20:29.268110+0000","last_active":"2024-08-19T14:22:52.614314+0000","last_peered":"2024-08-19T14:22:52.614314+0000","last_clean":"2024-08-19T14:22:52.614314+0000","last_became_active":"2024-08-19T14:20:29.267811+0000","last_became_peered":"2024-08-19T14:20:29.267811+0000","last_unstale":"2024-08-19T14:22:52.614314+0000","last_undegraded":"2024-08-19T14:22:52.614314+0000","last_fullsized":"2024-08-19T14:22:52.614314+0000","mapping_epoch":335,"log_start":"339'900","ondisk_log_start":"339'900","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":360316928,"num_objects":5498,"num_object_clones":0,"num_object_copies":16494,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10948,"num_write_kb":526272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"70'139","reported_seq":827,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-08-19T14:22:48.698378+0000","last_change":"2024-08-19T14:20:50.742676+0000","last_active":"2024-08-19T14:22:48.698378+0000","last_peered":"2024-08-19T14:22:48.698378+0000","last_clean":"2024-08-19T14:22:48.698378+0000","last_became_active":"2024-08-19T14:20:50.742354+0000","last_became_peered":"2024-08-19T14:20:50.742354+0000","last_unstale":"2024-08-19T14:22:48.698378+0000","last_undegraded":"2024-08-19T14:22:48.698378+0000","last_fullsized":"2024-08-19T14:22:48.698378+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:19:59.302931+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:19:59.302931+0000","last_clean_scrub_stamp":"2024-08-19T14:19:59.302931+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":114,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"382'10921","reported_seq":11024,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-08-19T14:22:52.625850+0000","last_change":"2024-08-19T14:20:29.267640+0000","last_active":"2024-08-19T14:22:52.625850+0000","last_peered":"2024-08-19T14:22:52.625850+0000","last_clean":"2024-08-19T14:22:52.625850+0000","last_became_active":"2024-08-19T14:20:29.267331+0000","last_became_peered":"2024-08-19T14:20:29.267331+0000","last_unstale":"2024-08-19T14:22:52.625850+0000","last_undegraded":"2024-08-19T14:22:52.625850+0000","last_fullsized":"2024-08-19T14:22:52.625850+0000","mapping_epoch":335,"log_start":"339'900","ondisk_log_start":"339'900","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358154240,"num_objects":5465,"num_object_clones":0,"num_object_copies":16395,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5465,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10921,"num_write_kb":524352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"70'139","reported_seq":876,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-08-19T14:22:48.698411+0000","last_change":"2024-08-19T14:20:09.507422+0000","last_active":"2024-08-19T14:22:48.698411+0000","last_peered":"2024-08-19T14:22:48.698411+0000","last_clean":"2024-08-19T14:22:48.698411+0000","last_became_active":"2024-08-19T14:18:13.808505+0000","last_became_peered":"2024-08-19T14:18:13.808505+0000","last_unstale":"2024-08-19T14:22:48.698411+0000","last_undegraded":"2024-08-19T14:22:48.698411+0000","last_fullsized":"2024-08-19T14:22:48.698411+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:09.507256+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:09.507256+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,11],"acting":[9,0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"382'11041","reported_seq":11146,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-08-19T14:22:53.378084+0000","last_change":"2024-08-19T14:20:29.268766+0000","last_active":"2024-08-19T14:22:53.378084+0000","last_peered":"2024-08-19T14:22:53.378084+0000","last_clean":"2024-08-19T14:22:53.378084+0000","last_became_active":"2024-08-19T14:20:29.268443+0000","last_became_peered":"2024-08-19T14:20:29.268443+0000","last_unstale":"2024-08-19T14:22:53.378084+0000","last_undegraded":"2024-08-19T14:22:53.378084+0000","last_fullsized":"2024-08-19T14:22:53.378084+0000","mapping_epoch":335,"log_start":"340'1000","ondisk_log_start":"340'1000","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358547456,"num_objects":5471,"num_object_clones":0,"num_object_copies":16413,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5471,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11041,"num_write_kb":528384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,11],"acting":[9,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.7","version":"70'139","reported_seq":528,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-08-19T14:22:48.702196+0000","last_change":"2024-08-19T14:21:46.057764+0000","last_active":"2024-08-19T14:22:48.702196+0000","last_peered":"2024-08-19T14:22:48.702196+0000","last_clean":"2024-08-19T14:22:48.702196+0000","last_became_active":"2024-08-19T14:20:28.204384+0000","last_became_peered":"2024-08-19T14:20:28.204384+0000","last_unstale":"2024-08-19T14:22:48.702196+0000","last_undegraded":"2024-08-19T14:22:48.702196+0000","last_fullsized":"2024-08-19T14:22:48.702196+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:46.057624+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:21:46.057624+0000","last_clean_scrub_stamp":"2024-08-19T14:21:46.057624+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:02.689278+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"382'10779","reported_seq":11029,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-08-19T14:22:50.896937+0000","last_change":"2024-08-19T14:22:07.840154+0000","last_active":"2024-08-19T14:22:50.896937+0000","last_peered":"2024-08-19T14:22:50.896937+0000","last_clean":"2024-08-19T14:22:50.896937+0000","last_became_active":"2024-08-19T14:20:29.267379+0000","last_became_peered":"2024-08-19T14:20:29.267379+0000","last_unstale":"2024-08-19T14:22:50.896937+0000","last_undegraded":"2024-08-19T14:22:50.896937+0000","last_fullsized":"2024-08-19T14:22:50.896937+0000","mapping_epoch":335,"log_start":"339'700","ondisk_log_start":"339'700","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"373'8338","last_scrub_stamp":"2024-08-19T14:22:07.840021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:22:07.840021+0000","objects_scrubbed":8073,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:13.679336+0000","scrub_duration":2409,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":365101056,"num_objects":5571,"num_object_clones":0,"num_object_copies":16713,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5571,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10779,"num_write_kb":523200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,1],"acting":[11,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.6","version":"70'139","reported_seq":880,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-08-19T14:22:48.698490+0000","last_change":"2024-08-19T14:19:55.512078+0000","last_active":"2024-08-19T14:22:48.698490+0000","last_peered":"2024-08-19T14:22:48.698490+0000","last_clean":"2024-08-19T14:22:48.698490+0000","last_became_active":"2024-08-19T14:18:26.874902+0000","last_became_peered":"2024-08-19T14:18:26.874902+0000","last_unstale":"2024-08-19T14:22:48.698490+0000","last_undegraded":"2024-08-19T14:22:48.698490+0000","last_fullsized":"2024-08-19T14:22:48.698490+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:19:55.511974+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:27.848870+0000","last_clean_scrub_stamp":"2024-08-19T14:19:55.511974+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,4],"acting":[9,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"382'10637","reported_seq":10830,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-08-19T14:22:49.659022+0000","last_change":"2024-08-19T14:22:08.536948+0000","last_active":"2024-08-19T14:22:49.659022+0000","last_peered":"2024-08-19T14:22:49.659022+0000","last_clean":"2024-08-19T14:22:49.659022+0000","last_became_active":"2024-08-19T14:20:29.267360+0000","last_became_peered":"2024-08-19T14:20:29.267360+0000","last_unstale":"2024-08-19T14:22:49.659022+0000","last_undegraded":"2024-08-19T14:22:49.659022+0000","last_fullsized":"2024-08-19T14:22:49.659022+0000","mapping_epoch":335,"log_start":"339'600","ondisk_log_start":"339'600","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"374'8331","last_scrub_stamp":"2024-08-19T14:22:08.536620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:22:08.536620+0000","objects_scrubbed":7995,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:19.976024+0000","scrub_duration":2478,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":367198208,"num_objects":5603,"num_object_clones":0,"num_object_copies":16809,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5603,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10637,"num_write_kb":519680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"70'139","reported_seq":702,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-08-19T14:22:48.697411+0000","last_change":"2024-08-19T14:22:01.354585+0000","last_active":"2024-08-19T14:22:48.697411+0000","last_peered":"2024-08-19T14:22:48.697411+0000","last_clean":"2024-08-19T14:22:48.697411+0000","last_became_active":"2024-08-19T14:18:26.881435+0000","last_became_peered":"2024-08-19T14:18:26.881435+0000","last_unstale":"2024-08-19T14:22:48.697411+0000","last_undegraded":"2024-08-19T14:22:48.697411+0000","last_fullsized":"2024-08-19T14:22:48.697411+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:22:01.354343+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:22:01.354343+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:30.081548+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,4],"acting":[11,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.1","version":"382'10882","reported_seq":11068,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-08-19T14:22:50.896958+0000","last_change":"2024-08-19T14:22:20.917940+0000","last_active":"2024-08-19T14:22:50.896958+0000","last_peered":"2024-08-19T14:22:50.896958+0000","last_clean":"2024-08-19T14:22:50.896958+0000","last_became_active":"2024-08-19T14:20:29.268465+0000","last_became_peered":"2024-08-19T14:20:29.268465+0000","last_unstale":"2024-08-19T14:22:50.896958+0000","last_undegraded":"2024-08-19T14:22:50.896958+0000","last_fullsized":"2024-08-19T14:22:50.896958+0000","mapping_epoch":335,"log_start":"339'800","ondisk_log_start":"339'800","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"377'9214","last_scrub_stamp":"2024-08-19T14:22:20.917842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:22:20.917842+0000","objects_scrubbed":7332,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:39.609654+0000","scrub_duration":1625,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":368312320,"num_objects":5620,"num_object_clones":0,"num_object_copies":16860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5620,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10882,"num_write_kb":528064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.4","version":"70'139","reported_seq":934,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-08-19T14:22:48.697252+0000","last_change":"2024-08-19T14:22:46.380189+0000","last_active":"2024-08-19T14:22:48.697252+0000","last_peered":"2024-08-19T14:22:48.697252+0000","last_clean":"2024-08-19T14:22:48.697252+0000","last_became_active":"2024-08-19T14:18:13.806013+0000","last_became_peered":"2024-08-19T14:18:13.806013+0000","last_unstale":"2024-08-19T14:22:48.697252+0000","last_undegraded":"2024-08-19T14:22:48.697252+0000","last_fullsized":"2024-08-19T14:22:48.697252+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:22:46.380119+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:22:46.380119+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:49.643921+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.f","version":"382'10725","reported_seq":10830,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-08-19T14:22:52.487763+0000","last_change":"2024-08-19T14:20:29.212298+0000","last_active":"2024-08-19T14:22:52.487763+0000","last_peered":"2024-08-19T14:22:52.487763+0000","last_clean":"2024-08-19T14:22:52.487763+0000","last_became_active":"2024-08-19T14:20:29.212003+0000","last_became_peered":"2024-08-19T14:20:29.212003+0000","last_unstale":"2024-08-19T14:22:52.487763+0000","last_undegraded":"2024-08-19T14:22:52.487763+0000","last_fullsized":"2024-08-19T14:22:52.487763+0000","mapping_epoch":335,"log_start":"339'700","ondisk_log_start":"339'700","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":365494272,"num_objects":5577,"num_object_clones":0,"num_object_copies":16731,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10725,"num_write_kb":521664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,11],"acting":[1,3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"70'139","reported_seq":888,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-08-19T14:22:48.698460+0000","last_change":"2024-08-19T14:20:12.570453+0000","last_active":"2024-08-19T14:22:48.698460+0000","last_peered":"2024-08-19T14:22:48.698460+0000","last_clean":"2024-08-19T14:22:48.698460+0000","last_became_active":"2024-08-19T14:18:26.875006+0000","last_became_peered":"2024-08-19T14:18:26.875006+0000","last_unstale":"2024-08-19T14:22:48.698460+0000","last_undegraded":"2024-08-19T14:22:48.698460+0000","last_fullsized":"2024-08-19T14:22:48.698460+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:12.570352+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:28.864870+0000","last_clean_scrub_stamp":"2024-08-19T14:20:12.570352+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,0],"acting":[9,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.c","version":"382'10846","reported_seq":11039,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-08-19T14:22:53.222213+0000","last_change":"2024-08-19T14:22:07.487434+0000","last_active":"2024-08-19T14:22:53.222213+0000","last_peered":"2024-08-19T14:22:53.222213+0000","last_clean":"2024-08-19T14:22:53.222213+0000","last_became_active":"2024-08-19T14:20:29.266373+0000","last_became_peered":"2024-08-19T14:20:29.266373+0000","last_unstale":"2024-08-19T14:22:53.222213+0000","last_undegraded":"2024-08-19T14:22:53.222213+0000","last_fullsized":"2024-08-19T14:22:53.222213+0000","mapping_epoch":335,"log_start":"339'800","ondisk_log_start":"339'800","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"373'8336","last_scrub_stamp":"2024-08-19T14:22:07.487270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:22:07.487270+0000","objects_scrubbed":8079,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:14.053549+0000","scrub_duration":1972,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":361496576,"num_objects":5516,"num_object_clones":0,"num_object_copies":16548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5516,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10846,"num_write_kb":523584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,8],"acting":[4,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"70'139","reported_seq":1003,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-08-19T14:22:48.698373+0000","last_change":"2024-08-19T14:19:56.106047+0000","last_active":"2024-08-19T14:22:48.698373+0000","last_peered":"2024-08-19T14:22:48.698373+0000","last_clean":"2024-08-19T14:22:48.698373+0000","last_became_active":"2024-08-19T14:18:26.887657+0000","last_became_peered":"2024-08-19T14:18:26.887657+0000","last_unstale":"2024-08-19T14:22:48.698373+0000","last_undegraded":"2024-08-19T14:22:48.698373+0000","last_fullsized":"2024-08-19T14:22:48.698373+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:19:56.105880+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:19:56.105880+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"382'10794","reported_seq":10988,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-08-19T14:22:53.188644+0000","last_change":"2024-08-19T14:22:08.529439+0000","last_active":"2024-08-19T14:22:53.188644+0000","last_peered":"2024-08-19T14:22:53.188644+0000","last_clean":"2024-08-19T14:22:53.188644+0000","last_became_active":"2024-08-19T14:20:29.208599+0000","last_became_peered":"2024-08-19T14:20:29.208599+0000","last_unstale":"2024-08-19T14:22:53.188644+0000","last_undegraded":"2024-08-19T14:22:53.188644+0000","last_fullsized":"2024-08-19T14:22:53.188644+0000","mapping_epoch":335,"log_start":"339'700","ondisk_log_start":"339'700","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"374'8320","last_scrub_stamp":"2024-08-19T14:22:08.529109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:22:08.529109+0000","objects_scrubbed":7983,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:30.745931+0000","scrub_duration":1962,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":357498904,"num_objects":5456,"num_object_clones":0,"num_object_copies":16368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5456,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10794,"num_write_kb":519937,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,1],"acting":[4,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"70'139","reported_seq":900,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-08-19T14:22:48.713294+0000","last_change":"2024-08-19T14:21:37.951800+0000","last_active":"2024-08-19T14:22:48.713294+0000","last_peered":"2024-08-19T14:22:48.713294+0000","last_clean":"2024-08-19T14:22:48.713294+0000","last_became_active":"2024-08-19T14:18:26.887277+0000","last_became_peered":"2024-08-19T14:18:26.887277+0000","last_unstale":"2024-08-19T14:22:48.713294+0000","last_undegraded":"2024-08-19T14:22:48.713294+0000","last_fullsized":"2024-08-19T14:22:48.713294+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:37.951694+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:21:37.951694+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:03.860882+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,8],"acting":[5,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"382'10766","reported_seq":10955,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-08-19T14:22:53.230252+0000","last_change":"2024-08-19T14:22:13.656515+0000","last_active":"2024-08-19T14:22:53.230252+0000","last_peered":"2024-08-19T14:22:53.230252+0000","last_clean":"2024-08-19T14:22:53.230252+0000","last_became_active":"2024-08-19T14:20:29.208748+0000","last_became_peered":"2024-08-19T14:20:29.208748+0000","last_unstale":"2024-08-19T14:22:53.230252+0000","last_undegraded":"2024-08-19T14:22:53.230252+0000","last_fullsized":"2024-08-19T14:22:53.230252+0000","mapping_epoch":335,"log_start":"339'700","ondisk_log_start":"339'700","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"374'8571","last_scrub_stamp":"2024-08-19T14:22:13.656335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:22:13.656335+0000","objects_scrubbed":7643,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:30.767371+0000","scrub_duration":3043,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":352059392,"num_objects":5372,"num_object_clones":0,"num_object_copies":16116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5372,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10766,"num_write_kb":516416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"70'139","reported_seq":902,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-08-19T14:22:48.697240+0000","last_change":"2024-08-19T14:22:02.413212+0000","last_active":"2024-08-19T14:22:48.697240+0000","last_peered":"2024-08-19T14:22:48.697240+0000","last_clean":"2024-08-19T14:22:48.697240+0000","last_became_active":"2024-08-19T14:18:26.875179+0000","last_became_peered":"2024-08-19T14:18:26.875179+0000","last_unstale":"2024-08-19T14:22:48.697240+0000","last_undegraded":"2024-08-19T14:22:48.697240+0000","last_fullsized":"2024-08-19T14:22:48.697240+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:22:02.413091+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:22:02.413091+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:23.082273+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.b","version":"382'10744","reported_seq":10923,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-08-19T14:22:51.260390+0000","last_change":"2024-08-19T14:21:39.089676+0000","last_active":"2024-08-19T14:22:51.260390+0000","last_peered":"2024-08-19T14:22:51.260390+0000","last_clean":"2024-08-19T14:22:51.260390+0000","last_became_active":"2024-08-19T14:20:29.207372+0000","last_became_peered":"2024-08-19T14:20:29.207372+0000","last_unstale":"2024-08-19T14:22:51.260390+0000","last_undegraded":"2024-08-19T14:22:51.260390+0000","last_fullsized":"2024-08-19T14:22:51.260390+0000","mapping_epoch":335,"log_start":"339'700","ondisk_log_start":"339'700","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"363'6615","last_scrub_stamp":"2024-08-19T14:21:39.089400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:21:39.089400+0000","objects_scrubbed":6552,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:05.632119+0000","scrub_duration":1907,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":366870528,"num_objects":5598,"num_object_clones":0,"num_object_copies":16794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10744,"num_write_kb":522944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"70'139","reported_seq":556,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-08-19T14:22:48.697524+0000","last_change":"2024-08-19T14:21:44.434481+0000","last_active":"2024-08-19T14:22:48.697524+0000","last_peered":"2024-08-19T14:22:48.697524+0000","last_clean":"2024-08-19T14:22:48.697524+0000","last_became_active":"2024-08-19T14:18:26.884105+0000","last_became_peered":"2024-08-19T14:18:26.884105+0000","last_unstale":"2024-08-19T14:22:48.697524+0000","last_undegraded":"2024-08-19T14:22:48.697524+0000","last_fullsized":"2024-08-19T14:22:48.697524+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:44.434382+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:20:15.321945+0000","last_clean_scrub_stamp":"2024-08-19T14:21:44.434382+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:13.544241+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"382'10993","reported_seq":11098,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-08-19T14:22:52.580846+0000","last_change":"2024-08-19T14:20:29.269723+0000","last_active":"2024-08-19T14:22:52.580846+0000","last_peered":"2024-08-19T14:22:52.580846+0000","last_clean":"2024-08-19T14:22:52.580846+0000","last_became_active":"2024-08-19T14:20:29.269432+0000","last_became_peered":"2024-08-19T14:20:29.269432+0000","last_unstale":"2024-08-19T14:22:52.580846+0000","last_undegraded":"2024-08-19T14:22:52.580846+0000","last_fullsized":"2024-08-19T14:22:52.580846+0000","mapping_epoch":335,"log_start":"339'900","ondisk_log_start":"339'900","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":368508928,"num_objects":5623,"num_object_clones":0,"num_object_copies":16869,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5623,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10993,"num_write_kb":531712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.f","version":"70'139","reported_seq":808,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-08-19T14:22:48.713414+0000","last_change":"2024-08-19T14:22:42.293240+0000","last_active":"2024-08-19T14:22:48.713414+0000","last_peered":"2024-08-19T14:22:48.713414+0000","last_clean":"2024-08-19T14:22:48.713414+0000","last_became_active":"2024-08-19T14:18:18.711708+0000","last_became_peered":"2024-08-19T14:18:18.711708+0000","last_unstale":"2024-08-19T14:22:48.713414+0000","last_undegraded":"2024-08-19T14:22:48.713414+0000","last_fullsized":"2024-08-19T14:22:48.713414+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:22:42.293171+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:22:42.293171+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:24:10.029244+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"382'10852","reported_seq":11034,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-08-19T14:22:50.902466+0000","last_change":"2024-08-19T14:21:43.974700+0000","last_active":"2024-08-19T14:22:50.902466+0000","last_peered":"2024-08-19T14:22:50.902466+0000","last_clean":"2024-08-19T14:22:50.902466+0000","last_became_active":"2024-08-19T14:20:29.211975+0000","last_became_peered":"2024-08-19T14:20:29.211975+0000","last_unstale":"2024-08-19T14:22:50.902466+0000","last_undegraded":"2024-08-19T14:22:50.902466+0000","last_fullsized":"2024-08-19T14:22:50.902466+0000","mapping_epoch":335,"log_start":"339'800","ondisk_log_start":"339'800","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"364'6991","last_scrub_stamp":"2024-08-19T14:21:43.974613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:21:43.974613+0000","objects_scrubbed":6934,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:07.463188+0000","scrub_duration":1682,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":367394816,"num_objects":5606,"num_object_clones":0,"num_object_copies":16818,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5606,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10852,"num_write_kb":526656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,3],"acting":[11,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.c","version":"70'139","reported_seq":661,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-08-19T14:22:48.702129+0000","last_change":"2024-08-19T14:21:40.181472+0000","last_active":"2024-08-19T14:22:48.702129+0000","last_peered":"2024-08-19T14:22:48.702129+0000","last_clean":"2024-08-19T14:22:48.702129+0000","last_became_active":"2024-08-19T14:18:26.872942+0000","last_became_peered":"2024-08-19T14:18:26.872942+0000","last_unstale":"2024-08-19T14:22:48.702129+0000","last_undegraded":"2024-08-19T14:22:48.702129+0000","last_fullsized":"2024-08-19T14:22:48.702129+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:40.181356+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:20:07.839076+0000","last_clean_scrub_stamp":"2024-08-19T14:21:40.181356+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:22:52.009381+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,9],"acting":[3,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"382'10745","reported_seq":10937,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-08-19T14:22:50.900834+0000","last_change":"2024-08-19T14:22:12.682046+0000","last_active":"2024-08-19T14:22:50.900834+0000","last_peered":"2024-08-19T14:22:50.900834+0000","last_clean":"2024-08-19T14:22:50.900834+0000","last_became_active":"2024-08-19T14:20:29.267442+0000","last_became_peered":"2024-08-19T14:20:29.267442+0000","last_unstale":"2024-08-19T14:22:50.900834+0000","last_undegraded":"2024-08-19T14:22:50.900834+0000","last_fullsized":"2024-08-19T14:22:50.900834+0000","mapping_epoch":335,"log_start":"339'700","ondisk_log_start":"339'700","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"374'8663","last_scrub_stamp":"2024-08-19T14:22:12.681916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:22:12.681916+0000","objects_scrubbed":7729,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:34.638254+0000","scrub_duration":1366,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":368246784,"num_objects":5619,"num_object_clones":0,"num_object_copies":16857,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5619,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10745,"num_write_kb":523648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,9],"acting":[11,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.d","version":"70'139","reported_seq":576,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-08-19T14:22:48.701798+0000","last_change":"2024-08-19T14:21:07.155294+0000","last_active":"2024-08-19T14:22:48.701798+0000","last_peered":"2024-08-19T14:22:48.701798+0000","last_clean":"2024-08-19T14:22:48.701798+0000","last_became_active":"2024-08-19T14:18:26.882002+0000","last_became_peered":"2024-08-19T14:18:26.882002+0000","last_unstale":"2024-08-19T14:22:48.701798+0000","last_undegraded":"2024-08-19T14:22:48.701798+0000","last_fullsized":"2024-08-19T14:22:48.701798+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:07.155157+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:21:07.155157+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"70'139","reported_seq":842,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-08-19T14:22:48.716568+0000","last_change":"2024-08-19T14:22:47.256611+0000","last_active":"2024-08-19T14:22:48.716568+0000","last_peered":"2024-08-19T14:22:48.716568+0000","last_clean":"2024-08-19T14:22:48.716568+0000","last_became_active":"2024-08-19T14:18:18.703594+0000","last_became_peered":"2024-08-19T14:18:18.703594+0000","last_unstale":"2024-08-19T14:22:48.716568+0000","last_undegraded":"2024-08-19T14:22:48.716568+0000","last_fullsized":"2024-08-19T14:22:48.716568+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:22:47.256536+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:22:47.256536+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:24:02.876164+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"70'139","reported_seq":886,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-08-19T14:22:48.702555+0000","last_change":"2024-08-19T14:20:07.692054+0000","last_active":"2024-08-19T14:22:48.702555+0000","last_peered":"2024-08-19T14:22:48.702555+0000","last_clean":"2024-08-19T14:22:48.702555+0000","last_became_active":"2024-08-19T14:18:18.698334+0000","last_became_peered":"2024-08-19T14:18:18.698334+0000","last_unstale":"2024-08-19T14:22:48.702555+0000","last_undegraded":"2024-08-19T14:22:48.702555+0000","last_fullsized":"2024-08-19T14:22:48.702555+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:07.691932+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:07.691932+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.10","version":"70'139","reported_seq":997,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-08-19T14:22:48.713189+0000","last_change":"2024-08-19T14:22:11.260404+0000","last_active":"2024-08-19T14:22:48.713189+0000","last_peered":"2024-08-19T14:22:48.713189+0000","last_clean":"2024-08-19T14:22:48.713189+0000","last_became_active":"2024-08-19T14:18:18.704562+0000","last_became_peered":"2024-08-19T14:18:18.704562+0000","last_unstale":"2024-08-19T14:22:48.713189+0000","last_undegraded":"2024-08-19T14:22:48.713189+0000","last_fullsized":"2024-08-19T14:22:48.713189+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:22:11.260286+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:22:11.260286+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:36.552471+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,8],"acting":[5,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"70'139","reported_seq":546,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-08-19T14:22:48.707237+0000","last_change":"2024-08-19T14:22:21.104595+0000","last_active":"2024-08-19T14:22:48.707237+0000","last_peered":"2024-08-19T14:22:48.707237+0000","last_clean":"2024-08-19T14:22:48.707237+0000","last_became_active":"2024-08-19T14:18:18.697894+0000","last_became_peered":"2024-08-19T14:18:18.697894+0000","last_unstale":"2024-08-19T14:22:48.707237+0000","last_undegraded":"2024-08-19T14:22:48.707237+0000","last_fullsized":"2024-08-19T14:22:48.707237+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:22:21.104491+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:21:16.071703+0000","last_clean_scrub_stamp":"2024-08-19T14:22:21.104491+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:45.856425+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"70'139","reported_seq":515,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-08-19T14:22:48.701689+0000","last_change":"2024-08-19T14:20:28.205212+0000","last_active":"2024-08-19T14:22:48.701689+0000","last_peered":"2024-08-19T14:22:48.701689+0000","last_clean":"2024-08-19T14:22:48.701689+0000","last_became_active":"2024-08-19T14:20:28.204724+0000","last_became_peered":"2024-08-19T14:20:28.204724+0000","last_unstale":"2024-08-19T14:22:48.701689+0000","last_undegraded":"2024-08-19T14:22:48.701689+0000","last_fullsized":"2024-08-19T14:22:48.701689+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:08.680021+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:08.680021+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"70'139","reported_seq":643,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-08-19T14:22:48.702629+0000","last_change":"2024-08-19T14:22:02.722505+0000","last_active":"2024-08-19T14:22:48.702629+0000","last_peered":"2024-08-19T14:22:48.702629+0000","last_clean":"2024-08-19T14:22:48.702629+0000","last_became_active":"2024-08-19T14:18:31.901556+0000","last_became_peered":"2024-08-19T14:18:31.901556+0000","last_unstale":"2024-08-19T14:22:48.702629+0000","last_undegraded":"2024-08-19T14:22:48.702629+0000","last_fullsized":"2024-08-19T14:22:48.702629+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:22:02.722345+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:22:02.722345+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:13.381470+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,0],"acting":[8,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.14","version":"70'139","reported_seq":644,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-08-19T14:22:48.698563+0000","last_change":"2024-08-19T14:19:55.130518+0000","last_active":"2024-08-19T14:22:48.698563+0000","last_peered":"2024-08-19T14:22:48.698563+0000","last_clean":"2024-08-19T14:22:48.698563+0000","last_became_active":"2024-08-19T14:18:18.703926+0000","last_became_peered":"2024-08-19T14:18:18.703926+0000","last_unstale":"2024-08-19T14:22:48.698563+0000","last_undegraded":"2024-08-19T14:22:48.698563+0000","last_fullsized":"2024-08-19T14:22:48.698563+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:19:55.130420+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:19:55.130420+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,1],"acting":[7,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.15","version":"70'139","reported_seq":531,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-08-19T14:22:48.701691+0000","last_change":"2024-08-19T14:20:09.181046+0000","last_active":"2024-08-19T14:22:48.701691+0000","last_peered":"2024-08-19T14:22:48.701691+0000","last_clean":"2024-08-19T14:22:48.701691+0000","last_became_active":"2024-08-19T14:18:31.912496+0000","last_became_peered":"2024-08-19T14:18:31.912496+0000","last_unstale":"2024-08-19T14:22:48.701691+0000","last_undegraded":"2024-08-19T14:22:48.701691+0000","last_fullsized":"2024-08-19T14:22:48.701691+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:09.180923+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:20:09.180923+0000","last_clean_scrub_stamp":"2024-08-19T14:20:09.180923+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"70'139","reported_seq":517,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-08-19T14:22:48.697322+0000","last_change":"2024-08-19T14:21:40.430835+0000","last_active":"2024-08-19T14:22:48.697322+0000","last_peered":"2024-08-19T14:22:48.697322+0000","last_clean":"2024-08-19T14:22:48.697322+0000","last_became_active":"2024-08-19T14:18:31.901229+0000","last_became_peered":"2024-08-19T14:18:31.901229+0000","last_unstale":"2024-08-19T14:22:48.697322+0000","last_undegraded":"2024-08-19T14:22:48.697322+0000","last_fullsized":"2024-08-19T14:22:48.697322+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:40.430714+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:21:40.430714+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:04.072653+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1b","version":"70'139","reported_seq":869,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-08-19T14:22:48.702519+0000","last_change":"2024-08-19T14:21:48.693864+0000","last_active":"2024-08-19T14:22:48.702519+0000","last_peered":"2024-08-19T14:22:48.702519+0000","last_clean":"2024-08-19T14:22:48.702519+0000","last_became_active":"2024-08-19T14:18:31.901152+0000","last_became_peered":"2024-08-19T14:18:31.901152+0000","last_unstale":"2024-08-19T14:22:48.702519+0000","last_undegraded":"2024-08-19T14:22:48.702519+0000","last_fullsized":"2024-08-19T14:22:48.702519+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:48.693761+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:21:48.693761+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:07.571130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.18","version":"70'139","reported_seq":913,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-08-19T14:22:48.713181+0000","last_change":"2024-08-19T14:22:07.254887+0000","last_active":"2024-08-19T14:22:48.713181+0000","last_peered":"2024-08-19T14:22:48.713181+0000","last_clean":"2024-08-19T14:22:48.713181+0000","last_became_active":"2024-08-19T14:18:31.902735+0000","last_became_peered":"2024-08-19T14:18:31.902735+0000","last_unstale":"2024-08-19T14:22:48.713181+0000","last_undegraded":"2024-08-19T14:22:48.713181+0000","last_fullsized":"2024-08-19T14:22:48.713181+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:22:07.254767+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:22:07.254767+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:11.195110+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.19","version":"70'139","reported_seq":861,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-08-19T14:22:48.707113+0000","last_change":"2024-08-19T14:21:40.219949+0000","last_active":"2024-08-19T14:22:48.707113+0000","last_peered":"2024-08-19T14:22:48.707113+0000","last_clean":"2024-08-19T14:22:48.707113+0000","last_became_active":"2024-08-19T14:18:31.902547+0000","last_became_peered":"2024-08-19T14:18:31.902547+0000","last_unstale":"2024-08-19T14:22:48.707113+0000","last_undegraded":"2024-08-19T14:22:48.707113+0000","last_fullsized":"2024-08-19T14:22:48.707113+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:40.219830+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:21:40.219830+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:05.414864+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1e","version":"70'139","reported_seq":520,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-08-19T14:22:48.697391+0000","last_change":"2024-08-19T14:22:09.404540+0000","last_active":"2024-08-19T14:22:48.697391+0000","last_peered":"2024-08-19T14:22:48.697391+0000","last_clean":"2024-08-19T14:22:48.697391+0000","last_became_active":"2024-08-19T14:18:31.902546+0000","last_became_peered":"2024-08-19T14:18:31.902546+0000","last_unstale":"2024-08-19T14:22:48.697391+0000","last_undegraded":"2024-08-19T14:22:48.697391+0000","last_fullsized":"2024-08-19T14:22:48.697391+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:22:09.404433+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:22:09.404433+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:13.587290+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1c","version":"70'139","reported_seq":635,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-08-19T14:22:49.619735+0000","last_change":"2024-08-19T14:22:49.619773+0000","last_active":"2024-08-19T14:22:49.619735+0000","last_peered":"2024-08-19T14:22:49.619735+0000","last_clean":"2024-08-19T14:22:49.619735+0000","last_became_active":"2024-08-19T14:18:31.902494+0000","last_became_peered":"2024-08-19T14:18:31.902494+0000","last_unstale":"2024-08-19T14:22:49.619735+0000","last_undegraded":"2024-08-19T14:22:49.619735+0000","last_fullsized":"2024-08-19T14:22:49.619735+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:22:49.619657+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:22:49.619657+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:58.403527+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,11],"acting":[8,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1d","version":"70'139","reported_seq":541,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-08-19T14:22:48.704717+0000","last_change":"2024-08-19T14:20:17.725615+0000","last_active":"2024-08-19T14:22:48.704717+0000","last_peered":"2024-08-19T14:22:48.704717+0000","last_clean":"2024-08-19T14:22:48.704717+0000","last_became_active":"2024-08-19T14:18:31.901183+0000","last_became_peered":"2024-08-19T14:18:31.901183+0000","last_unstale":"2024-08-19T14:22:48.704717+0000","last_undegraded":"2024-08-19T14:22:48.704717+0000","last_fullsized":"2024-08-19T14:22:48.704717+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:17.725477+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:17.725477+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,5],"acting":[1,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":5814091800,"num_objects":88717,"num_object_clones":0,"num_object_copies":266151,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":476,"num_objects_unfound":0,"num_objects_dirty":88717,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":173629,"num_write_kb":8395009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3664,"num_bytes_recovered":240123904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17674612736,"data_stored":17674600520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160829,"ondisk_log_size":160829,"up":48,"acting":47,"num_store_stats":9},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1802240,"data_stored":1788032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187594,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2952328,"kb_used_data":2524092,"kb_used_omap":1,"kb_used_meta":428222,"kb_avail":90792824,"statfs":{"total":95995035648,"available":92971851776,"internally_reserved":0,"allocated":2584670208,"data_stored":2582912486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":438499780},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":343597383824,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":74,"seq":317827580052,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1873672,"kb_used_data":1387136,"kb_used_omap":1,"kb_used_meta":486526,"kb_avail":91871480,"statfs":{"total":95995035648,"available":94076395520,"internally_reserved":0,"allocated":1420427264,"data_stored":1418665470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":498203073},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874393,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":752520,"kb_used_data":370108,"kb_used_omap":1,"kb_used_meta":382398,"kb_avail":92992632,"statfs":{"total":95995035648,"available":95224455168,"internally_reserved":0,"allocated":378990592,"data_stored":377232870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":391576005},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":57,"seq":244813136028,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3290508,"kb_used_data":2822020,"kb_used_omap":1,"kb_used_meta":468478,"kb_avail":90454644,"statfs":{"total":95995035648,"available":92625555456,"internally_reserved":0,"allocated":2889748480,"data_stored":2887987206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":479721930},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,3,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":206158430367,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692004,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1702028,"kb_used_data":1350148,"kb_used_omap":1,"kb_used_meta":351870,"kb_avail":92043124,"statfs":{"total":95995035648,"available":94252158976,"internally_reserved":0,"allocated":1382551552,"data_stored":1380790278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":360315333},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986345,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2263432,"kb_used_data":1763008,"kb_used_omap":1,"kb_used_meta":500414,"kb_avail":91481720,"statfs":{"total":95995035648,"available":93677281280,"internally_reserved":0,"allocated":1805320192,"data_stored":1803558398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":512424389},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149870,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2639240,"kb_used_data":2188796,"kb_used_omap":1,"kb_used_meta":450430,"kb_avail":91105912,"statfs":{"total":95995035648,"available":93292453888,"internally_reserved":0,"allocated":2241327104,"data_stored":2239569382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":461240775},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411508,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575032,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4131728,"kb_used_data":3546568,"kb_used_omap":1,"kb_used_meta":585086,"kb_avail":89613424,"statfs":{"total":95995035648,"available":91764146176,"internally_reserved":0,"allocated":3631685632,"data_stored":3629920286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":599128517},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738556,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1788616,"kb_used_data":1436732,"kb_used_omap":1,"kb_used_meta":351870,"kb_avail":91956536,"statfs":{"total":95995035648,"available":94163492864,"internally_reserved":0,"allocated":1471213568,"data_stored":1469455846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":360315333},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1451622400,"data_stored":1451622400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3617460224,"data_stored":3617456152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2222325760,"data_stored":2222325760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1792544768,"data_stored":1792540696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1373765632,"data_stored":1373765632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2865233920,"data_stored":2865233920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":372375552,"data_stored":372375552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1412042752,"data_stored":1412038680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":2567241728,"data_stored":2567241728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T14:22:57.900 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph status --format=json 2024-08-19T14:22:57.996 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:22:58.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:57 smithi032 ceph-mon[20004]: pgmap v874: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.4 GiB data, 20 GiB used, 784 GiB / 805 GiB avail; 0 B/s wr, 842 op/s; 476/265353 objects misplaced (0.179%); 3.2 MiB/s, 51 objects/s recovering 2024-08-19T14:22:58.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:58.026 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:58.026 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:22:58.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:57 smithi045 ceph-mon[20143]: pgmap v874: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.4 GiB data, 20 GiB used, 784 GiB / 805 GiB avail; 0 B/s wr, 842 op/s; 476/265353 objects misplaced (0.179%); 3.2 MiB/s, 51 objects/s recovering 2024-08-19T14:22:58.128 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:22:58.151 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:58.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:58.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:22:58.253 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:22:58.277 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:58.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:58.278 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:22:58.380 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:22:58.402 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:58.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:58.403 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:22:58.416 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:22:58.422 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:22:58.451 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:22:58.504 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:22:58.534 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:58.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:58.535 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:22:58.636 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:22:58.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:58.669 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:58.669 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:22:58.770 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:22:58.798 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:58.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:58.799 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:22:58.900 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:22:58.929 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:58.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:58.930 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:22:59.032 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:22:59.056 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:59.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:59.057 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:22:59.158 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:22:59.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:59.193 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:22:59.193 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:22:59.295 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:22:59.319 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:59.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:59.320 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:22:59.421 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:22:59.445 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:59.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:59.446 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:22:59.473 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:22:59.473 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:22:59.515 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:22:59.516 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:22:59.547 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:22:59.557 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:22:59.558 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:22:59.570 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:22:59.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:59.571 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:22:59.607 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:22:59.608 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:22:59.647 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:22:59.648 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:22:59.672 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:22:59.683 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:22:59.683 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:22:59.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:59.721 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:59.721 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:22:59.740 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:22:59.740 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:22:59.785 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:22:59.786 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:22:59.818 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:22:59.818 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:22:59.823 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:22:59.855 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:22:59.856 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:22:59.869 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:22:59.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:22:59.870 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:22:59.882 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:22:59.882 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:22:59.893 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:22:59.893 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:22:59.934 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:22:59.935 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:22:59.971 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:23:00.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:59 smithi032 ceph-mon[20004]: pgmap v875: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.2 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 930 op/s; 176/256404 objects misplaced (0.069%); 4.8 MiB/s, 76 objects/s recovering 2024-08-19T14:23:00.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:22:59 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/386244912' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:23:00.009 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:00.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:00.010 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:23:00.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:59 smithi045 ceph-mon[20143]: pgmap v875: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.2 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 930 op/s; 176/256404 objects misplaced (0.069%); 4.8 MiB/s, 76 objects/s recovering 2024-08-19T14:23:00.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:22:59 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/386244912' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:23:00.080 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:59 smithi031 ceph-mon[16336]: pgmap v875: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.2 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 930 op/s; 176/256404 objects misplaced (0.069%); 4.8 MiB/s, 76 objects/s recovering 2024-08-19T14:23:00.080 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:22:59 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/386244912' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:23:00.110 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:23:00.139 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:00.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:00.140 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:23:00.152 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:23:00.152 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"ed7d1c6e-5e33-11ef-bccf-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub flag(s) set","count":7},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 1 pg undersized","count":1},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":957,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":383,"num_osds":12,"num_up_osds":12,"osd_up_since":1724076699,"num_in_osds":9,"osd_in_since":1724076998,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":46},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":47,"num_pools":2,"num_objects":85468,"data_bytes":5601624632,"bytes_used":21377884160,"bytes_avail":842577436672,"bytes_total":863955320832,"misplaced_objects":176,"misplaced_total":256404,"misplaced_ratio":0.00068641674856866508,"recovering_objects_per_sec":76,"recovering_bytes_per_sec":5035733,"recovering_keys_per_sec":0,"num_objects_recovered":709,"num_bytes_recovered":46465024,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":930},"fsmap":{"epoch":1,"btime":"2024-08-19T14:05:22:841940+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.31:8443/"}},"servicemap":{"epoch":23,"modified":"2024-08-19T14:22:55.923383+0000","services":{"mgr":{"daemons":{"summary":"","x":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-08-19T14:23:00.241 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:23:00.268 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:00.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:00.269 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:23:00.352 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-19T14:23:00.370 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:23:00.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:00.442 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:00.442 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:23:00.543 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:23:00.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:00.584 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:00.585 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:23:00.686 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:23:00.716 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:00.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:00.718 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:23:00.819 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:23:00.852 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:00.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:00.854 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:23:00.951 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:00 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1859634227' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-08-19T14:23:00.954 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:23:00.977 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:00.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:00.978 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:23:01.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:00 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1859634227' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-08-19T14:23:01.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:00 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1859634227' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-08-19T14:23:01.078 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:23:01.100 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:01.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:01.102 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:23:01.203 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:23:01.226 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:01.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:01.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:23:01.328 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:23:01.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:01.350 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:01.350 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:23:01.431 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-08-19T14:23:01.431 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-19T14:23:01.451 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:23:01.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:01.473 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:01.473 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:23:01.574 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:23:01.595 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:01.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:01.596 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:23:01.697 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:23:01.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:01.719 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:01.719 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:23:01.820 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:23:01.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:01.851 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:01.851 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:23:01.952 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:23:01.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:01.989 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:01.989 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:23:02.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:01 smithi032 ceph-mon[20004]: pgmap v876: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.1 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 829 op/s; 176/252294 objects misplaced (0.070%); 2.9 MiB/s, 46 objects/s recovering 2024-08-19T14:23:02.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:01 smithi032 ceph-mon[20004]: osdmap e384: 12 total, 12 up, 9 in 2024-08-19T14:23:02.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:01 smithi032 ceph-mon[20004]: osdmap e385: 12 total, 12 up, 9 in 2024-08-19T14:23:02.033 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:01 smithi031 ceph-mon[16336]: pgmap v876: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.1 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 829 op/s; 176/252294 objects misplaced (0.070%); 2.9 MiB/s, 46 objects/s recovering 2024-08-19T14:23:02.033 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:01 smithi031 ceph-mon[16336]: osdmap e384: 12 total, 12 up, 9 in 2024-08-19T14:23:02.033 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:01 smithi031 ceph-mon[16336]: osdmap e385: 12 total, 12 up, 9 in 2024-08-19T14:23:02.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:01 smithi045 ceph-mon[20143]: pgmap v876: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.1 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 829 op/s; 176/252294 objects misplaced (0.070%); 2.9 MiB/s, 46 objects/s recovering 2024-08-19T14:23:02.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:01 smithi045 ceph-mon[20143]: osdmap e384: 12 total, 12 up, 9 in 2024-08-19T14:23:02.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:01 smithi045 ceph-mon[20143]: osdmap e385: 12 total, 12 up, 9 in 2024-08-19T14:23:02.084 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-08-19T14:23:02.090 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:23:02.142 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:23:02.145 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:02.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:02.146 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:23:02.231 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:23:02.247 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:23:02.302 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:02.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:02.304 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:23:02.405 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:23:02.431 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:02.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:02.432 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:23:02.437 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:23:02.533 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:23:02.557 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:02.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:02.558 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:23:02.659 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:23:02.687 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:02.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:02.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:23:02.789 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:23:02.826 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:02.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:02.827 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:23:02.928 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:23:02.952 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:02.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:02.953 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:23:03.000 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:23:03.054 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:23:03.092 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:03.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:03.093 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:23:03.187 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:23:03.187 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:23:03.193 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:23:03.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:03.224 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:03.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:23:03.326 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:23:03.350 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:03.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:03.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:23:03.453 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:23:03.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:03.484 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:03.484 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:23:03.586 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:23:03.611 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:03.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:03.612 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:23:03.713 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:23:03.747 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:03.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:03.748 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:23:03.849 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:23:03.879 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:03.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:03.880 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:23:03.946 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:03 smithi032 ceph-mon[20004]: pgmap v879: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.0 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 968 op/s; 176/247308 objects misplaced (0.071%); 3.0 MiB/s, 48 objects/s recovering 2024-08-19T14:23:03.981 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:23:03.985 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-08-19T14:23:04.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:03 smithi045 ceph-mon[20143]: pgmap v879: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.0 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 968 op/s; 176/247308 objects misplaced (0.071%); 3.0 MiB/s, 48 objects/s recovering 2024-08-19T14:23:04.050 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:04.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:04.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:23:04.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:03 smithi031 ceph-mon[16336]: pgmap v879: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.0 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 968 op/s; 176/247308 objects misplaced (0.071%); 3.0 MiB/s, 48 objects/s recovering 2024-08-19T14:23:04.152 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:23:04.176 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:04.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:04.177 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:23:04.278 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:23:04.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:04.314 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:04.314 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:23:04.328 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:23:04.415 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:23:04.440 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:04.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:04.441 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:23:04.542 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:23:04.580 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:04.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:04.581 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:23:04.682 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:23:04.711 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:04.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:04.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:23:04.813 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:23:04.848 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:04.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:04.849 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:23:04.950 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:23:04.986 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:04.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:04.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:23:05.087 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:23:05.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:04 smithi031 ceph-mon[16336]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-08-19T14:23:05.118 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:05.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:05.119 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:23:05.220 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:23:05.246 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:05.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:05.247 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:23:05.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:04 smithi032 ceph-mon[20004]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-08-19T14:23:05.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:04 smithi045 ceph-mon[20143]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-08-19T14:23:05.322 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:23:05.323 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-08-19T14:23:05.348 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:23:05.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:05.374 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:05.374 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:23:05.468 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:23:05.468 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:23:05.475 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:23:05.506 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:05.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:05.507 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:23:05.608 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:23:05.632 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:05.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:05.633 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:23:05.734 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:23:05.772 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:05.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:05.773 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:23:05.874 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:23:05.903 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:05 smithi031 ceph-mon[16336]: pgmap v880: 47 pgs: 47 active+clean; 4.9 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 1.03k op/s; 3.7 MiB/s, 59 objects/s recovering 2024-08-19T14:23:05.903 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:05 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:23:05.903 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:05 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:23:05.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:05.907 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:05.907 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:23:06.008 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:23:06.034 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:06.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:06.036 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:23:06.137 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:23:06.168 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:06.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:06.169 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:23:06.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:05 smithi032 ceph-mon[20004]: pgmap v880: 47 pgs: 47 active+clean; 4.9 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 1.03k op/s; 3.7 MiB/s, 59 objects/s recovering 2024-08-19T14:23:06.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:05 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:23:06.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:05 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:23:06.270 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:23:06.294 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:06.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:06.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:23:06.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:05 smithi045 ceph-mon[20143]: pgmap v880: 47 pgs: 47 active+clean; 4.9 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 1.03k op/s; 3.7 MiB/s, 59 objects/s recovering 2024-08-19T14:23:06.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:05 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:23:06.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:05 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:23:06.361 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":880,"stamp":"2024-08-19T14:23:03.927656+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5254742584,"num_objects":80175,"num_object_clones":0,"num_object_copies":240525,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80175,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":182173,"num_write_kb":8395009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4140,"num_bytes_recovered":271319040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":165082,"ondisk_log_size":165082,"up":141,"acting":141,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":142,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":19811508,"kb_used_data":15772568,"kb_used_omap":14,"kb_used_meta":4038641,"kb_avail":823894860,"statfs":{"total":863955320832,"available":843668336640,"internally_reserved":0,"allocated":16151109632,"data_stored":16135191110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14353,"internal_metadata":4135569391},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5],"upper_bound":8},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-542375936,"num_objects":-8276,"num_object_clones":0,"num_object_copies":-24828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-476,"num_objects_unfound":0,"num_objects_dirty":-8276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8276,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":476,"num_bytes_recovered":31195136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.004310"},"pg_stats":[{"pgid":"4.5","version":"384'11442","reported_seq":11636,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-08-19T14:23:00.881897+0000","last_change":"2024-08-19T14:22:14.211513+0000","last_active":"2024-08-19T14:23:00.881897+0000","last_peered":"2024-08-19T14:23:00.881897+0000","last_clean":"2024-08-19T14:23:00.881897+0000","last_became_active":"2024-08-19T14:20:29.267549+0000","last_became_peered":"2024-08-19T14:20:29.267549+0000","last_unstale":"2024-08-19T14:23:00.881897+0000","last_undegraded":"2024-08-19T14:23:00.881897+0000","last_fullsized":"2024-08-19T14:23:00.881897+0000","mapping_epoch":335,"log_start":"341'1400","ondisk_log_start":"341'1400","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"375'8747","last_scrub_stamp":"2024-08-19T14:22:14.211361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:22:14.211361+0000","objects_scrubbed":7717,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:26.018227+0000","scrub_duration":1909,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":324927488,"num_objects":4958,"num_object_clones":0,"num_object_copies":14874,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4958,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11442,"num_write_kb":524800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,0],"acting":[11,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"70'139","reported_seq":987,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-08-19T14:23:01.740723+0000","last_change":"2024-08-19T14:19:58.072227+0000","last_active":"2024-08-19T14:23:01.740723+0000","last_peered":"2024-08-19T14:23:01.740723+0000","last_clean":"2024-08-19T14:23:01.740723+0000","last_became_active":"2024-08-19T14:18:13.807133+0000","last_became_peered":"2024-08-19T14:18:13.807133+0000","last_unstale":"2024-08-19T14:23:01.740723+0000","last_undegraded":"2024-08-19T14:23:01.740723+0000","last_fullsized":"2024-08-19T14:23:01.740723+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:19:58.072083+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:19:58.072083+0000","last_clean_scrub_stamp":"2024-08-19T14:19:58.072083+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,9],"acting":[7,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"385'11619","reported_seq":20020,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-08-19T14:23:03.444436+0000","last_change":"2024-08-19T14:23:01.772236+0000","last_active":"2024-08-19T14:23:03.444436+0000","last_peered":"2024-08-19T14:23:03.444436+0000","last_clean":"2024-08-19T14:23:03.444436+0000","last_became_active":"2024-08-19T14:23:01.741503+0000","last_became_peered":"2024-08-19T14:23:01.741503+0000","last_unstale":"2024-08-19T14:23:03.444436+0000","last_undegraded":"2024-08-19T14:23:03.444436+0000","last_fullsized":"2024-08-19T14:23:03.444436+0000","mapping_epoch":384,"log_start":"341'1600","ondisk_log_start":"341'1600","created":335,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":331546624,"num_objects":5059,"num_object_clones":0,"num_object_copies":15177,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11619,"num_write_kb":533696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4140,"num_bytes_recovered":271319040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,7],"acting":[9,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.3","version":"70'139","reported_seq":885,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-08-19T14:23:01.740598+0000","last_change":"2024-08-19T14:20:03.714917+0000","last_active":"2024-08-19T14:23:01.740598+0000","last_peered":"2024-08-19T14:23:01.740598+0000","last_clean":"2024-08-19T14:23:01.740598+0000","last_became_active":"2024-08-19T14:18:13.804721+0000","last_became_peered":"2024-08-19T14:18:13.804721+0000","last_unstale":"2024-08-19T14:23:01.740598+0000","last_undegraded":"2024-08-19T14:23:01.740598+0000","last_fullsized":"2024-08-19T14:23:01.740598+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:03.714818+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:03.714818+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:06.462600+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"385'11466","reported_seq":11577,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-08-19T14:23:02.650553+0000","last_change":"2024-08-19T14:20:29.268110+0000","last_active":"2024-08-19T14:23:02.650553+0000","last_peered":"2024-08-19T14:23:02.650553+0000","last_clean":"2024-08-19T14:23:02.650553+0000","last_became_active":"2024-08-19T14:20:29.267811+0000","last_became_peered":"2024-08-19T14:20:29.267811+0000","last_unstale":"2024-08-19T14:23:02.650553+0000","last_undegraded":"2024-08-19T14:23:02.650553+0000","last_fullsized":"2024-08-19T14:23:02.650553+0000","mapping_epoch":335,"log_start":"341'1400","ondisk_log_start":"341'1400","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:06.462668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":326369280,"num_objects":4980,"num_object_clones":0,"num_object_copies":14940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4980,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11466,"num_write_kb":526272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"70'139","reported_seq":834,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-08-19T14:23:01.740700+0000","last_change":"2024-08-19T14:20:50.742676+0000","last_active":"2024-08-19T14:23:01.740700+0000","last_peered":"2024-08-19T14:23:01.740700+0000","last_clean":"2024-08-19T14:23:01.740700+0000","last_became_active":"2024-08-19T14:20:50.742354+0000","last_became_peered":"2024-08-19T14:20:50.742354+0000","last_unstale":"2024-08-19T14:23:01.740700+0000","last_undegraded":"2024-08-19T14:23:01.740700+0000","last_fullsized":"2024-08-19T14:23:01.740700+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:19:59.302931+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:19:59.302931+0000","last_clean_scrub_stamp":"2024-08-19T14:19:59.302931+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":114,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"385'11442","reported_seq":11553,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-08-19T14:23:02.608940+0000","last_change":"2024-08-19T14:20:29.267640+0000","last_active":"2024-08-19T14:23:02.608940+0000","last_peered":"2024-08-19T14:23:02.608940+0000","last_clean":"2024-08-19T14:23:02.608940+0000","last_became_active":"2024-08-19T14:20:29.267331+0000","last_became_peered":"2024-08-19T14:20:29.267331+0000","last_unstale":"2024-08-19T14:23:02.608940+0000","last_undegraded":"2024-08-19T14:23:02.608940+0000","last_fullsized":"2024-08-19T14:23:02.608940+0000","mapping_epoch":335,"log_start":"341'1400","ondisk_log_start":"341'1400","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:06.462651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":324009984,"num_objects":4944,"num_object_clones":0,"num_object_copies":14832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4944,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11442,"num_write_kb":524352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"70'139","reported_seq":882,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-08-19T14:23:01.739305+0000","last_change":"2024-08-19T14:20:09.507422+0000","last_active":"2024-08-19T14:23:01.739305+0000","last_peered":"2024-08-19T14:23:01.739305+0000","last_clean":"2024-08-19T14:23:01.739305+0000","last_became_active":"2024-08-19T14:18:13.808505+0000","last_became_peered":"2024-08-19T14:18:13.808505+0000","last_unstale":"2024-08-19T14:23:01.739305+0000","last_undegraded":"2024-08-19T14:23:01.739305+0000","last_fullsized":"2024-08-19T14:23:01.739305+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:09.507256+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:09.507256+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,11],"acting":[9,0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"385'11540","reported_seq":11650,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-08-19T14:23:03.431365+0000","last_change":"2024-08-19T14:20:29.268766+0000","last_active":"2024-08-19T14:23:03.431365+0000","last_peered":"2024-08-19T14:23:03.431365+0000","last_clean":"2024-08-19T14:23:03.431365+0000","last_became_active":"2024-08-19T14:20:29.268443+0000","last_became_peered":"2024-08-19T14:20:29.268443+0000","last_unstale":"2024-08-19T14:23:03.431365+0000","last_undegraded":"2024-08-19T14:23:03.431365+0000","last_fullsized":"2024-08-19T14:23:03.431365+0000","mapping_epoch":335,"log_start":"341'1500","ondisk_log_start":"341'1500","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":325844992,"num_objects":4972,"num_object_clones":0,"num_object_copies":14916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11540,"num_write_kb":528384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,11],"acting":[9,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.7","version":"70'139","reported_seq":530,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-08-19T14:22:55.726670+0000","last_change":"2024-08-19T14:21:46.057764+0000","last_active":"2024-08-19T14:22:55.726670+0000","last_peered":"2024-08-19T14:22:55.726670+0000","last_clean":"2024-08-19T14:22:55.726670+0000","last_became_active":"2024-08-19T14:20:28.204384+0000","last_became_peered":"2024-08-19T14:20:28.204384+0000","last_unstale":"2024-08-19T14:22:55.726670+0000","last_undegraded":"2024-08-19T14:22:55.726670+0000","last_fullsized":"2024-08-19T14:22:55.726670+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:46.057624+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:21:46.057624+0000","last_clean_scrub_stamp":"2024-08-19T14:21:46.057624+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:02.689278+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"384'11316","reported_seq":11570,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-08-19T14:23:00.912717+0000","last_change":"2024-08-19T14:22:07.840154+0000","last_active":"2024-08-19T14:23:00.912717+0000","last_peered":"2024-08-19T14:23:00.912717+0000","last_clean":"2024-08-19T14:23:00.912717+0000","last_became_active":"2024-08-19T14:20:29.267379+0000","last_became_peered":"2024-08-19T14:20:29.267379+0000","last_unstale":"2024-08-19T14:23:00.912717+0000","last_undegraded":"2024-08-19T14:23:00.912717+0000","last_fullsized":"2024-08-19T14:23:00.912717+0000","mapping_epoch":335,"log_start":"341'1300","ondisk_log_start":"341'1300","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"373'8338","last_scrub_stamp":"2024-08-19T14:22:07.840021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:22:07.840021+0000","objects_scrubbed":8073,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:13.679336+0000","scrub_duration":2409,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":329908224,"num_objects":5034,"num_object_clones":0,"num_object_copies":15102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5034,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11316,"num_write_kb":523200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,1],"acting":[11,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.6","version":"70'139","reported_seq":886,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-08-19T14:23:01.739406+0000","last_change":"2024-08-19T14:19:55.512078+0000","last_active":"2024-08-19T14:23:01.739406+0000","last_peered":"2024-08-19T14:23:01.739406+0000","last_clean":"2024-08-19T14:23:01.739406+0000","last_became_active":"2024-08-19T14:18:26.874902+0000","last_became_peered":"2024-08-19T14:18:26.874902+0000","last_unstale":"2024-08-19T14:23:01.739406+0000","last_undegraded":"2024-08-19T14:23:01.739406+0000","last_fullsized":"2024-08-19T14:23:01.739406+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:19:55.511974+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:27.848870+0000","last_clean_scrub_stamp":"2024-08-19T14:19:55.511974+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,4],"acting":[9,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"383'11211","reported_seq":11406,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-08-19T14:22:59.678995+0000","last_change":"2024-08-19T14:22:08.536948+0000","last_active":"2024-08-19T14:22:59.678995+0000","last_peered":"2024-08-19T14:22:59.678995+0000","last_clean":"2024-08-19T14:22:59.678995+0000","last_became_active":"2024-08-19T14:20:29.267360+0000","last_became_peered":"2024-08-19T14:20:29.267360+0000","last_unstale":"2024-08-19T14:22:59.678995+0000","last_undegraded":"2024-08-19T14:22:59.678995+0000","last_fullsized":"2024-08-19T14:22:59.678995+0000","mapping_epoch":335,"log_start":"340'1200","ondisk_log_start":"340'1200","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"374'8331","last_scrub_stamp":"2024-08-19T14:22:08.536620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:22:08.536620+0000","objects_scrubbed":7995,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:19.976024+0000","scrub_duration":2478,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":329580544,"num_objects":5029,"num_object_clones":0,"num_object_copies":15087,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5029,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11211,"num_write_kb":519680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"70'139","reported_seq":706,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-08-19T14:23:00.736792+0000","last_change":"2024-08-19T14:22:01.354585+0000","last_active":"2024-08-19T14:23:00.736792+0000","last_peered":"2024-08-19T14:23:00.736792+0000","last_clean":"2024-08-19T14:23:00.736792+0000","last_became_active":"2024-08-19T14:18:26.881435+0000","last_became_peered":"2024-08-19T14:18:26.881435+0000","last_unstale":"2024-08-19T14:23:00.736792+0000","last_undegraded":"2024-08-19T14:23:00.736792+0000","last_fullsized":"2024-08-19T14:23:00.736792+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:22:01.354343+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:22:01.354343+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:30.081548+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,4],"acting":[11,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.1","version":"384'11470","reported_seq":11660,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-08-19T14:23:00.869782+0000","last_change":"2024-08-19T14:22:20.917940+0000","last_active":"2024-08-19T14:23:00.869782+0000","last_peered":"2024-08-19T14:23:00.869782+0000","last_clean":"2024-08-19T14:23:00.869782+0000","last_became_active":"2024-08-19T14:20:29.268465+0000","last_became_peered":"2024-08-19T14:20:29.268465+0000","last_unstale":"2024-08-19T14:23:00.869782+0000","last_undegraded":"2024-08-19T14:23:00.869782+0000","last_fullsized":"2024-08-19T14:23:00.869782+0000","mapping_epoch":335,"log_start":"341'1400","ondisk_log_start":"341'1400","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"377'9214","last_scrub_stamp":"2024-08-19T14:22:20.917842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:22:20.917842+0000","objects_scrubbed":7332,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:39.609654+0000","scrub_duration":1625,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":329777152,"num_objects":5032,"num_object_clones":0,"num_object_copies":15096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11470,"num_write_kb":528064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.4","version":"70'139","reported_seq":938,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-08-19T14:23:00.736642+0000","last_change":"2024-08-19T14:22:46.380189+0000","last_active":"2024-08-19T14:23:00.736642+0000","last_peered":"2024-08-19T14:23:00.736642+0000","last_clean":"2024-08-19T14:23:00.736642+0000","last_became_active":"2024-08-19T14:18:13.806013+0000","last_became_peered":"2024-08-19T14:18:13.806013+0000","last_unstale":"2024-08-19T14:23:00.736642+0000","last_undegraded":"2024-08-19T14:23:00.736642+0000","last_fullsized":"2024-08-19T14:23:00.736642+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:22:46.380119+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:22:46.380119+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:49.643921+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.f","version":"385'11266","reported_seq":11377,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-08-19T14:23:02.648813+0000","last_change":"2024-08-19T14:20:29.212298+0000","last_active":"2024-08-19T14:23:02.648813+0000","last_peered":"2024-08-19T14:23:02.648813+0000","last_clean":"2024-08-19T14:23:02.648813+0000","last_became_active":"2024-08-19T14:20:29.212003+0000","last_became_peered":"2024-08-19T14:20:29.212003+0000","last_unstale":"2024-08-19T14:23:02.648813+0000","last_undegraded":"2024-08-19T14:23:02.648813+0000","last_fullsized":"2024-08-19T14:23:02.648813+0000","mapping_epoch":335,"log_start":"340'1200","ondisk_log_start":"340'1200","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:06.462685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330039296,"num_objects":5036,"num_object_clones":0,"num_object_copies":15108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11266,"num_write_kb":521664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,11],"acting":[1,3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"70'139","reported_seq":894,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-08-19T14:23:01.739298+0000","last_change":"2024-08-19T14:20:12.570453+0000","last_active":"2024-08-19T14:23:01.739298+0000","last_peered":"2024-08-19T14:23:01.739298+0000","last_clean":"2024-08-19T14:23:01.739298+0000","last_became_active":"2024-08-19T14:18:26.875006+0000","last_became_peered":"2024-08-19T14:18:26.875006+0000","last_unstale":"2024-08-19T14:23:01.739298+0000","last_undegraded":"2024-08-19T14:23:01.739298+0000","last_fullsized":"2024-08-19T14:23:01.739298+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:12.570352+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:28.864870+0000","last_clean_scrub_stamp":"2024-08-19T14:20:12.570352+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,0],"acting":[9,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.c","version":"385'11379","reported_seq":11578,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-08-19T14:23:03.197664+0000","last_change":"2024-08-19T14:22:07.487434+0000","last_active":"2024-08-19T14:23:03.197664+0000","last_peered":"2024-08-19T14:23:03.197664+0000","last_clean":"2024-08-19T14:23:03.197664+0000","last_became_active":"2024-08-19T14:20:29.266373+0000","last_became_peered":"2024-08-19T14:20:29.266373+0000","last_unstale":"2024-08-19T14:23:03.197664+0000","last_undegraded":"2024-08-19T14:23:03.197664+0000","last_fullsized":"2024-08-19T14:23:03.197664+0000","mapping_epoch":335,"log_start":"341'1300","ondisk_log_start":"341'1300","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"373'8336","last_scrub_stamp":"2024-08-19T14:22:07.487270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:22:07.487270+0000","objects_scrubbed":8079,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:14.053549+0000","scrub_duration":1972,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":326565888,"num_objects":4983,"num_object_clones":0,"num_object_copies":14949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4983,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11379,"num_write_kb":523584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,8],"acting":[4,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"70'139","reported_seq":1009,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-08-19T14:23:01.740630+0000","last_change":"2024-08-19T14:19:56.106047+0000","last_active":"2024-08-19T14:23:01.740630+0000","last_peered":"2024-08-19T14:23:01.740630+0000","last_clean":"2024-08-19T14:23:01.740630+0000","last_became_active":"2024-08-19T14:18:26.887657+0000","last_became_peered":"2024-08-19T14:18:26.887657+0000","last_unstale":"2024-08-19T14:23:01.740630+0000","last_undegraded":"2024-08-19T14:23:01.740630+0000","last_fullsized":"2024-08-19T14:23:01.740630+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:19:56.105880+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:19:56.105880+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"385'11324","reported_seq":11524,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-08-19T14:23:03.230584+0000","last_change":"2024-08-19T14:22:08.529439+0000","last_active":"2024-08-19T14:23:03.230584+0000","last_peered":"2024-08-19T14:23:03.230584+0000","last_clean":"2024-08-19T14:23:03.230584+0000","last_became_active":"2024-08-19T14:20:29.208599+0000","last_became_peered":"2024-08-19T14:20:29.208599+0000","last_unstale":"2024-08-19T14:23:03.230584+0000","last_undegraded":"2024-08-19T14:23:03.230584+0000","last_fullsized":"2024-08-19T14:23:03.230584+0000","mapping_epoch":335,"log_start":"340'1300","ondisk_log_start":"340'1300","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"374'8320","last_scrub_stamp":"2024-08-19T14:22:08.529109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:22:08.529109+0000","objects_scrubbed":7983,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:30.745931+0000","scrub_duration":1962,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":322764824,"num_objects":4926,"num_object_clones":0,"num_object_copies":14778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4926,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11324,"num_write_kb":519937,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,1],"acting":[4,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"70'139","reported_seq":904,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-08-19T14:23:00.738774+0000","last_change":"2024-08-19T14:21:37.951800+0000","last_active":"2024-08-19T14:23:00.738774+0000","last_peered":"2024-08-19T14:23:00.738774+0000","last_clean":"2024-08-19T14:23:00.738774+0000","last_became_active":"2024-08-19T14:18:26.887277+0000","last_became_peered":"2024-08-19T14:18:26.887277+0000","last_unstale":"2024-08-19T14:23:00.738774+0000","last_undegraded":"2024-08-19T14:23:00.738774+0000","last_fullsized":"2024-08-19T14:23:00.738774+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:37.951694+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:21:37.951694+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:03.860882+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,8],"acting":[5,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"385'11272","reported_seq":11467,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-08-19T14:23:03.228228+0000","last_change":"2024-08-19T14:22:13.656515+0000","last_active":"2024-08-19T14:23:03.228228+0000","last_peered":"2024-08-19T14:23:03.228228+0000","last_clean":"2024-08-19T14:23:03.228228+0000","last_became_active":"2024-08-19T14:20:29.208748+0000","last_became_peered":"2024-08-19T14:20:29.208748+0000","last_unstale":"2024-08-19T14:23:03.228228+0000","last_undegraded":"2024-08-19T14:23:03.228228+0000","last_fullsized":"2024-08-19T14:23:03.228228+0000","mapping_epoch":335,"log_start":"340'1200","ondisk_log_start":"340'1200","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"374'8571","last_scrub_stamp":"2024-08-19T14:22:13.656335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:22:13.656335+0000","objects_scrubbed":7643,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:30.767371+0000","scrub_duration":3043,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":318898176,"num_objects":4866,"num_object_clones":0,"num_object_copies":14598,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11272,"num_write_kb":516416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"70'139","reported_seq":906,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-08-19T14:23:00.736758+0000","last_change":"2024-08-19T14:22:02.413212+0000","last_active":"2024-08-19T14:23:00.736758+0000","last_peered":"2024-08-19T14:23:00.736758+0000","last_clean":"2024-08-19T14:23:00.736758+0000","last_became_active":"2024-08-19T14:18:26.875179+0000","last_became_peered":"2024-08-19T14:18:26.875179+0000","last_unstale":"2024-08-19T14:23:00.736758+0000","last_undegraded":"2024-08-19T14:23:00.736758+0000","last_fullsized":"2024-08-19T14:23:00.736758+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:22:02.413091+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:22:02.413091+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:23.082273+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.b","version":"384'11262","reported_seq":11446,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-08-19T14:23:01.255104+0000","last_change":"2024-08-19T14:21:39.089676+0000","last_active":"2024-08-19T14:23:01.255104+0000","last_peered":"2024-08-19T14:23:01.255104+0000","last_clean":"2024-08-19T14:23:01.255104+0000","last_became_active":"2024-08-19T14:20:29.207372+0000","last_became_peered":"2024-08-19T14:20:29.207372+0000","last_unstale":"2024-08-19T14:23:01.255104+0000","last_undegraded":"2024-08-19T14:23:01.255104+0000","last_fullsized":"2024-08-19T14:23:01.255104+0000","mapping_epoch":335,"log_start":"340'1200","ondisk_log_start":"340'1200","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"363'6615","last_scrub_stamp":"2024-08-19T14:21:39.089400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:21:39.089400+0000","objects_scrubbed":6552,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:05.632119+0000","scrub_duration":1907,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332922880,"num_objects":5080,"num_object_clones":0,"num_object_copies":15240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11262,"num_write_kb":522944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"70'139","reported_seq":562,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-08-19T14:23:01.783552+0000","last_change":"2024-08-19T14:21:44.434481+0000","last_active":"2024-08-19T14:23:01.783552+0000","last_peered":"2024-08-19T14:23:01.783552+0000","last_clean":"2024-08-19T14:23:01.783552+0000","last_became_active":"2024-08-19T14:18:26.884105+0000","last_became_peered":"2024-08-19T14:18:26.884105+0000","last_unstale":"2024-08-19T14:23:01.783552+0000","last_undegraded":"2024-08-19T14:23:01.783552+0000","last_fullsized":"2024-08-19T14:23:01.783552+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:44.434382+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:20:15.321945+0000","last_clean_scrub_stamp":"2024-08-19T14:21:44.434382+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:13.544241+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"385'11501","reported_seq":11612,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-08-19T14:23:02.617834+0000","last_change":"2024-08-19T14:20:29.269723+0000","last_active":"2024-08-19T14:23:02.617834+0000","last_peered":"2024-08-19T14:23:02.617834+0000","last_clean":"2024-08-19T14:23:02.617834+0000","last_became_active":"2024-08-19T14:20:29.269432+0000","last_became_peered":"2024-08-19T14:20:29.269432+0000","last_unstale":"2024-08-19T14:23:02.617834+0000","last_undegraded":"2024-08-19T14:23:02.617834+0000","last_fullsized":"2024-08-19T14:23:02.617834+0000","mapping_epoch":335,"log_start":"341'1500","ondisk_log_start":"341'1500","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:06.462634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335216640,"num_objects":5115,"num_object_clones":0,"num_object_copies":15345,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5115,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11501,"num_write_kb":531712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.f","version":"70'139","reported_seq":812,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-08-19T14:23:00.738992+0000","last_change":"2024-08-19T14:22:42.293240+0000","last_active":"2024-08-19T14:23:00.738992+0000","last_peered":"2024-08-19T14:23:00.738992+0000","last_clean":"2024-08-19T14:23:00.738992+0000","last_became_active":"2024-08-19T14:18:18.711708+0000","last_became_peered":"2024-08-19T14:18:18.711708+0000","last_unstale":"2024-08-19T14:23:00.738992+0000","last_undegraded":"2024-08-19T14:23:00.738992+0000","last_fullsized":"2024-08-19T14:23:00.738992+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:22:42.293171+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:22:42.293171+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:24:10.029244+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"384'11394","reported_seq":11582,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-08-19T14:23:00.925779+0000","last_change":"2024-08-19T14:21:43.974700+0000","last_active":"2024-08-19T14:23:00.925779+0000","last_peered":"2024-08-19T14:23:00.925779+0000","last_clean":"2024-08-19T14:23:00.925779+0000","last_became_active":"2024-08-19T14:20:29.211975+0000","last_became_peered":"2024-08-19T14:20:29.211975+0000","last_unstale":"2024-08-19T14:23:00.925779+0000","last_undegraded":"2024-08-19T14:23:00.925779+0000","last_fullsized":"2024-08-19T14:23:00.925779+0000","mapping_epoch":335,"log_start":"341'1300","ondisk_log_start":"341'1300","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"364'6991","last_scrub_stamp":"2024-08-19T14:21:43.974613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:21:43.974613+0000","objects_scrubbed":6934,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:07.463188+0000","scrub_duration":1682,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":331874304,"num_objects":5064,"num_object_clones":0,"num_object_copies":15192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5064,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11394,"num_write_kb":526656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,3],"acting":[11,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.c","version":"70'139","reported_seq":671,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-08-19T14:22:55.729411+0000","last_change":"2024-08-19T14:22:52.164697+0000","last_active":"2024-08-19T14:22:55.729411+0000","last_peered":"2024-08-19T14:22:55.729411+0000","last_clean":"2024-08-19T14:22:55.729411+0000","last_became_active":"2024-08-19T14:18:26.872942+0000","last_became_peered":"2024-08-19T14:18:26.872942+0000","last_unstale":"2024-08-19T14:22:55.729411+0000","last_undegraded":"2024-08-19T14:22:55.729411+0000","last_fullsized":"2024-08-19T14:22:55.729411+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:22:52.164586+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:22:52.164586+0000","last_clean_scrub_stamp":"2024-08-19T14:22:52.164586+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:58.654533+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,9],"acting":[3,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"384'11269","reported_seq":11465,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-08-19T14:23:00.887376+0000","last_change":"2024-08-19T14:22:12.682046+0000","last_active":"2024-08-19T14:23:00.887376+0000","last_peered":"2024-08-19T14:23:00.887376+0000","last_clean":"2024-08-19T14:23:00.887376+0000","last_became_active":"2024-08-19T14:20:29.267442+0000","last_became_peered":"2024-08-19T14:20:29.267442+0000","last_unstale":"2024-08-19T14:23:00.887376+0000","last_undegraded":"2024-08-19T14:23:00.887376+0000","last_fullsized":"2024-08-19T14:23:00.887376+0000","mapping_epoch":335,"log_start":"340'1200","ondisk_log_start":"340'1200","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"374'8663","last_scrub_stamp":"2024-08-19T14:22:12.681916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:22:12.681916+0000","objects_scrubbed":7729,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:34.638254+0000","scrub_duration":1366,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":333905920,"num_objects":5095,"num_object_clones":0,"num_object_copies":15285,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5095,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11269,"num_write_kb":523648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,9],"acting":[11,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.d","version":"70'139","reported_seq":582,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-08-19T14:23:01.740911+0000","last_change":"2024-08-19T14:21:07.155294+0000","last_active":"2024-08-19T14:23:01.740911+0000","last_peered":"2024-08-19T14:23:01.740911+0000","last_clean":"2024-08-19T14:23:01.740911+0000","last_became_active":"2024-08-19T14:18:26.882002+0000","last_became_peered":"2024-08-19T14:18:26.882002+0000","last_unstale":"2024-08-19T14:23:01.740911+0000","last_undegraded":"2024-08-19T14:23:01.740911+0000","last_fullsized":"2024-08-19T14:23:01.740911+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:07.155157+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:21:07.155157+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"70'139","reported_seq":847,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-08-19T14:23:00.742736+0000","last_change":"2024-08-19T14:22:47.256611+0000","last_active":"2024-08-19T14:23:00.742736+0000","last_peered":"2024-08-19T14:23:00.742736+0000","last_clean":"2024-08-19T14:23:00.742736+0000","last_became_active":"2024-08-19T14:18:18.703594+0000","last_became_peered":"2024-08-19T14:18:18.703594+0000","last_unstale":"2024-08-19T14:23:00.742736+0000","last_undegraded":"2024-08-19T14:23:00.742736+0000","last_fullsized":"2024-08-19T14:23:00.742736+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:22:47.256536+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:22:47.256536+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:24:02.876164+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"70'139","reported_seq":893,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-08-19T14:23:01.740505+0000","last_change":"2024-08-19T14:20:07.692054+0000","last_active":"2024-08-19T14:23:01.740505+0000","last_peered":"2024-08-19T14:23:01.740505+0000","last_clean":"2024-08-19T14:23:01.740505+0000","last_became_active":"2024-08-19T14:18:18.698334+0000","last_became_peered":"2024-08-19T14:18:18.698334+0000","last_unstale":"2024-08-19T14:23:01.740505+0000","last_undegraded":"2024-08-19T14:23:01.740505+0000","last_fullsized":"2024-08-19T14:23:01.740505+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:07.691932+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:07.691932+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:06.462617+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.10","version":"70'139","reported_seq":1002,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-08-19T14:23:00.738720+0000","last_change":"2024-08-19T14:22:11.260404+0000","last_active":"2024-08-19T14:23:00.738720+0000","last_peered":"2024-08-19T14:23:00.738720+0000","last_clean":"2024-08-19T14:23:00.738720+0000","last_became_active":"2024-08-19T14:18:18.704562+0000","last_became_peered":"2024-08-19T14:18:18.704562+0000","last_unstale":"2024-08-19T14:23:00.738720+0000","last_undegraded":"2024-08-19T14:23:00.738720+0000","last_fullsized":"2024-08-19T14:23:00.738720+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:22:11.260286+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:22:11.260286+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:36.552471+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,8],"acting":[5,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"70'139","reported_seq":551,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-08-19T14:23:00.746928+0000","last_change":"2024-08-19T14:22:21.104595+0000","last_active":"2024-08-19T14:23:00.746928+0000","last_peered":"2024-08-19T14:23:00.746928+0000","last_clean":"2024-08-19T14:23:00.746928+0000","last_became_active":"2024-08-19T14:18:18.697894+0000","last_became_peered":"2024-08-19T14:18:18.697894+0000","last_unstale":"2024-08-19T14:23:00.746928+0000","last_undegraded":"2024-08-19T14:23:00.746928+0000","last_fullsized":"2024-08-19T14:23:00.746928+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:22:21.104491+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:21:16.071703+0000","last_clean_scrub_stamp":"2024-08-19T14:22:21.104491+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:45.856425+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"70'139","reported_seq":521,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-08-19T14:23:01.740755+0000","last_change":"2024-08-19T14:20:28.205212+0000","last_active":"2024-08-19T14:23:01.740755+0000","last_peered":"2024-08-19T14:23:01.740755+0000","last_clean":"2024-08-19T14:23:01.740755+0000","last_became_active":"2024-08-19T14:20:28.204724+0000","last_became_peered":"2024-08-19T14:20:28.204724+0000","last_unstale":"2024-08-19T14:23:01.740755+0000","last_undegraded":"2024-08-19T14:23:01.740755+0000","last_fullsized":"2024-08-19T14:23:01.740755+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:08.680021+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:08.680021+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:06.462583+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"70'139","reported_seq":645,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-08-19T14:22:55.719359+0000","last_change":"2024-08-19T14:22:02.722505+0000","last_active":"2024-08-19T14:22:55.719359+0000","last_peered":"2024-08-19T14:22:55.719359+0000","last_clean":"2024-08-19T14:22:55.719359+0000","last_became_active":"2024-08-19T14:18:31.901556+0000","last_became_peered":"2024-08-19T14:18:31.901556+0000","last_unstale":"2024-08-19T14:22:55.719359+0000","last_undegraded":"2024-08-19T14:22:55.719359+0000","last_fullsized":"2024-08-19T14:22:55.719359+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:22:02.722345+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:22:02.722345+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:13.381470+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,0],"acting":[8,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.14","version":"70'139","reported_seq":651,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-08-19T14:23:01.740800+0000","last_change":"2024-08-19T14:19:55.130518+0000","last_active":"2024-08-19T14:23:01.740800+0000","last_peered":"2024-08-19T14:23:01.740800+0000","last_clean":"2024-08-19T14:23:01.740800+0000","last_became_active":"2024-08-19T14:18:18.703926+0000","last_became_peered":"2024-08-19T14:18:18.703926+0000","last_unstale":"2024-08-19T14:23:01.740800+0000","last_undegraded":"2024-08-19T14:23:01.740800+0000","last_fullsized":"2024-08-19T14:23:01.740800+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:19:55.130420+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:19:55.130420+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,1],"acting":[7,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.15","version":"70'139","reported_seq":537,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-08-19T14:23:01.740937+0000","last_change":"2024-08-19T14:20:09.181046+0000","last_active":"2024-08-19T14:23:01.740937+0000","last_peered":"2024-08-19T14:23:01.740937+0000","last_clean":"2024-08-19T14:23:01.740937+0000","last_became_active":"2024-08-19T14:18:31.912496+0000","last_became_peered":"2024-08-19T14:18:31.912496+0000","last_unstale":"2024-08-19T14:23:01.740937+0000","last_undegraded":"2024-08-19T14:23:01.740937+0000","last_fullsized":"2024-08-19T14:23:01.740937+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:09.180923+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:20:09.180923+0000","last_clean_scrub_stamp":"2024-08-19T14:20:09.180923+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"70'139","reported_seq":523,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-08-19T14:23:01.783247+0000","last_change":"2024-08-19T14:21:40.430835+0000","last_active":"2024-08-19T14:23:01.783247+0000","last_peered":"2024-08-19T14:23:01.783247+0000","last_clean":"2024-08-19T14:23:01.783247+0000","last_became_active":"2024-08-19T14:18:31.901229+0000","last_became_peered":"2024-08-19T14:18:31.901229+0000","last_unstale":"2024-08-19T14:23:01.783247+0000","last_undegraded":"2024-08-19T14:23:01.783247+0000","last_fullsized":"2024-08-19T14:23:01.783247+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:40.430714+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:21:40.430714+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:04.072653+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1b","version":"70'139","reported_seq":871,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-08-19T14:22:55.719161+0000","last_change":"2024-08-19T14:21:48.693864+0000","last_active":"2024-08-19T14:22:55.719161+0000","last_peered":"2024-08-19T14:22:55.719161+0000","last_clean":"2024-08-19T14:22:55.719161+0000","last_became_active":"2024-08-19T14:18:31.901152+0000","last_became_peered":"2024-08-19T14:18:31.901152+0000","last_unstale":"2024-08-19T14:22:55.719161+0000","last_undegraded":"2024-08-19T14:22:55.719161+0000","last_fullsized":"2024-08-19T14:22:55.719161+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:48.693761+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:21:48.693761+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:07.571130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.18","version":"70'139","reported_seq":917,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-08-19T14:23:00.738479+0000","last_change":"2024-08-19T14:22:07.254887+0000","last_active":"2024-08-19T14:23:00.738479+0000","last_peered":"2024-08-19T14:23:00.738479+0000","last_clean":"2024-08-19T14:23:00.738479+0000","last_became_active":"2024-08-19T14:18:31.902735+0000","last_became_peered":"2024-08-19T14:18:31.902735+0000","last_unstale":"2024-08-19T14:23:00.738479+0000","last_undegraded":"2024-08-19T14:23:00.738479+0000","last_fullsized":"2024-08-19T14:23:00.738479+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:22:07.254767+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:22:07.254767+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:11.195110+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.19","version":"70'139","reported_seq":865,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-08-19T14:23:00.746826+0000","last_change":"2024-08-19T14:21:40.219949+0000","last_active":"2024-08-19T14:23:00.746826+0000","last_peered":"2024-08-19T14:23:00.746826+0000","last_clean":"2024-08-19T14:23:00.746826+0000","last_became_active":"2024-08-19T14:18:31.902547+0000","last_became_peered":"2024-08-19T14:18:31.902547+0000","last_unstale":"2024-08-19T14:23:00.746826+0000","last_undegraded":"2024-08-19T14:23:00.746826+0000","last_fullsized":"2024-08-19T14:23:00.746826+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:40.219830+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:21:40.219830+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:05.414864+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1e","version":"70'139","reported_seq":524,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-08-19T14:23:00.736711+0000","last_change":"2024-08-19T14:22:09.404540+0000","last_active":"2024-08-19T14:23:00.736711+0000","last_peered":"2024-08-19T14:23:00.736711+0000","last_clean":"2024-08-19T14:23:00.736711+0000","last_became_active":"2024-08-19T14:18:31.902546+0000","last_became_peered":"2024-08-19T14:18:31.902546+0000","last_unstale":"2024-08-19T14:23:00.736711+0000","last_undegraded":"2024-08-19T14:23:00.736711+0000","last_fullsized":"2024-08-19T14:23:00.736711+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:22:09.404433+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:22:09.404433+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:13.587290+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1c","version":"70'139","reported_seq":637,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-08-19T14:22:55.719237+0000","last_change":"2024-08-19T14:22:49.619773+0000","last_active":"2024-08-19T14:22:55.719237+0000","last_peered":"2024-08-19T14:22:55.719237+0000","last_clean":"2024-08-19T14:22:55.719237+0000","last_became_active":"2024-08-19T14:18:31.902494+0000","last_became_peered":"2024-08-19T14:18:31.902494+0000","last_unstale":"2024-08-19T14:22:55.719237+0000","last_undegraded":"2024-08-19T14:22:55.719237+0000","last_fullsized":"2024-08-19T14:22:55.719237+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:22:49.619657+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:22:49.619657+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:58.403527+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,11],"acting":[8,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1d","version":"70'139","reported_seq":547,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-08-19T14:23:01.740608+0000","last_change":"2024-08-19T14:20:17.725615+0000","last_active":"2024-08-19T14:23:01.740608+0000","last_peered":"2024-08-19T14:23:01.740608+0000","last_clean":"2024-08-19T14:23:01.740608+0000","last_became_active":"2024-08-19T14:18:31.901183+0000","last_became_peered":"2024-08-19T14:18:31.901183+0000","last_unstale":"2024-08-19T14:23:01.740608+0000","last_undegraded":"2024-08-19T14:23:01.740608+0000","last_fullsized":"2024-08-19T14:23:01.740608+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:17.725477+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:17.725477+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:06.462564+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,5],"acting":[1,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":5254152216,"num_objects":80173,"num_object_clones":0,"num_object_copies":240519,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80173,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":182173,"num_write_kb":8395009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4140,"num_bytes_recovered":271319040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16033656832,"data_stored":16033644616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160773,"ondisk_log_size":160773,"up":48,"acting":48,"num_store_stats":9},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1802240,"data_stored":1788032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187596,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2708240,"kb_used_data":2280008,"kb_used_omap":1,"kb_used_meta":428222,"kb_avail":91036912,"statfs":{"total":95995035648,"available":93221797888,"internally_reserved":0,"allocated":2334728192,"data_stored":2332966244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":438499780},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":343597383826,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":74,"seq":317827580054,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1762672,"kb_used_data":1284388,"kb_used_omap":1,"kb_used_meta":478206,"kb_avail":91982480,"statfs":{"total":95995035648,"available":94190059520,"internally_reserved":0,"allocated":1315213312,"data_stored":1313438819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":489683393},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874395,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":704272,"kb_used_data":334280,"kb_used_omap":1,"kb_used_meta":369918,"kb_avail":93040880,"statfs":{"total":95995035648,"available":95273861120,"internally_reserved":0,"allocated":342302720,"data_stored":340540772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":378796485},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":57,"seq":244813136030,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3060084,"kb_used_data":2555496,"kb_used_omap":1,"kb_used_meta":504574,"kb_avail":90685068,"statfs":{"total":95995035648,"available":92861509632,"internally_reserved":0,"allocated":2616827904,"data_stored":2615053931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":516684234},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":206158430369,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692006,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1618984,"kb_used_data":1244892,"kb_used_omap":1,"kb_used_meta":374078,"kb_avail":92126168,"statfs":{"total":95995035648,"available":94337196032,"internally_reserved":0,"allocated":1274769408,"data_stored":1272999682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":383056325},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986347,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2101808,"kb_used_data":1593060,"kb_used_omap":1,"kb_used_meta":508734,"kb_avail":91643344,"statfs":{"total":95995035648,"available":93842784256,"internally_reserved":0,"allocated":1631293440,"data_stored":1629518947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":520944069},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149872,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2420112,"kb_used_data":1973832,"kb_used_omap":1,"kb_used_meta":446270,"kb_avail":91325040,"statfs":{"total":95995035648,"available":93516840960,"internally_reserved":0,"allocated":2021203968,"data_stored":2019442020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":456980935},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411510,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575034,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3780856,"kb_used_data":3212396,"kb_used_omap":1,"kb_used_meta":568446,"kb_avail":89964296,"statfs":{"total":95995035648,"available":92123439104,"internally_reserved":0,"allocated":3289493504,"data_stored":3287715459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":582089157},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738558,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1654480,"kb_used_data":1294216,"kb_used_omap":1,"kb_used_meta":360190,"kb_avail":92090672,"statfs":{"total":95995035648,"available":94300848128,"internally_reserved":0,"allocated":1325277184,"data_stored":1323515236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":368835013},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1310588928,"data_stored":1310588928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3277656064,"data_stored":3277651992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2005401600,"data_stored":2005401600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1618743296,"data_stored":1618739224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1269956608,"data_stored":1269956608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2593390592,"data_stored":2593390592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":336265216,"data_stored":336265216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1305874432,"data_stored":1305870360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":2315780096,"data_stored":2315780096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T14:23:06.396 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:23:06.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:06.417 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:06.418 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:23:06.519 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:23:06.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:06.560 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:06.560 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:23:06.574 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:23:06.660 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:23:06.686 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:06.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:06.687 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:23:06.761 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-08-19T14:23:06.788 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:23:06.814 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:06.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:06.815 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:23:06.916 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:23:06.960 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:06.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:06.961 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:23:07.011 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:06 smithi032 ceph-mon[20004]: from='client.25637 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:23:07.011 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:06 smithi032 ceph-mon[20004]: osdmap e386: 12 total, 12 up, 9 in 2024-08-19T14:23:07.011 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:06 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:23:07.011 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:06 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:23:07.017 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:06 smithi031 ceph-mon[16336]: from='client.25637 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:23:07.017 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:06 smithi031 ceph-mon[16336]: osdmap e386: 12 total, 12 up, 9 in 2024-08-19T14:23:07.017 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:06 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:23:07.017 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:06 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:23:07.062 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:23:07.098 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:07.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:07.099 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:23:07.103 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:23:07.108 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:06 smithi045 ceph-mon[20143]: from='client.25637 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:23:07.108 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:06 smithi045 ceph-mon[20143]: osdmap e386: 12 total, 12 up, 9 in 2024-08-19T14:23:07.108 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:06 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:23:07.108 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:06 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:23:07.192 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:23:07.193 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:23:07.200 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:23:07.223 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:07.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:07.225 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:23:07.233 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:23:07.234 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:23:07.267 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:23:07.268 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:23:07.300 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:23:07.300 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:23:07.325 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:23:07.327 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:23:07.327 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:23:07.351 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:23:07.351 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:07.351 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:23:07.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:07.352 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:23:07.377 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:23:07.377 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:23:07.399 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:23:07.399 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:23:07.419 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:23:07.420 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:23:07.436 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:23:07.436 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:23:07.453 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:23:07.454 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:23:07.454 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:23:07.470 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:23:07.470 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:23:07.475 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:07.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:07.476 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:23:07.576 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:23:07.611 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:07.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:07.612 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:23:07.713 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:23:07.729 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:23:07.729 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-08-19T14:23:07.729 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-08-19T14:23:07.729 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-08-19T14:23:07.729 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.729 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25057.0:50176 3.14 3.f91a7cf4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-08-19T14:23:07.730 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:17:20.751296+0000", 2024-08-19T14:23:07.730 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 346.97461341000002, 2024-08-19T14:23:07.730 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.99210832500000001, 2024-08-19T14:23:07.730 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:23:07.730 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:23:07.730 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:23:07.730 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:23:07.730 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25057", 2024-08-19T14:23:07.730 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:23:07.730 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 50176 2024-08-19T14:23:07.730 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.730 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:23:07.730 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.731 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:23:07.731 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:20.751296+0000", 2024-08-19T14:23:07.731 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:07.731 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.731 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.731 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:23:07.731 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:20.751296+0000", 2024-08-19T14:23:07.731 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:07.731 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.731 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.731 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:23:07.731 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:20.751297+0000", 2024-08-19T14:23:07.731 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.04e-06 2024-08-19T14:23:07.731 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.732 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.732 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:23:07.732 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:20.751354+0000", 2024-08-19T14:23:07.732 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6588000000000001e-05 2024-08-19T14:23:07.732 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.732 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.732 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:23:07.732 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:20.751365+0000", 2024-08-19T14:23:07.732 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0889e-05 2024-08-19T14:23:07.732 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.732 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.732 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:23:07.732 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:20.751371+0000", 2024-08-19T14:23:07.732 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6879999999999996e-06 2024-08-19T14:23:07.733 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.733 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.733 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:23:07.733 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:20.751421+0000", 2024-08-19T14:23:07.733 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0519000000000002e-05 2024-08-19T14:23:07.733 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.733 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.733 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:23:07.733 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:20.751430+0000", 2024-08-19T14:23:07.733 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.1200000000000008e-06 2024-08-19T14:23:07.733 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.733 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.733 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:23:07.733 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:21.743385+0000", 2024-08-19T14:23:07.734 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.99195432400000005 2024-08-19T14:23:07.734 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.734 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.734 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:23:07.734 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:21.743405+0000", 2024-08-19T14:23:07.734 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0157e-05 2024-08-19T14:23:07.734 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:07.734 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:23:07.734 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:07.734 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.734 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.734 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25057.0:50187 3.11 3.8832571 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-08-19T14:23:07.734 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:17:20.755221+0000", 2024-08-19T14:23:07.734 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 346.970689225, 2024-08-19T14:23:07.735 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.98787882999999999, 2024-08-19T14:23:07.735 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:23:07.735 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:23:07.735 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:23:07.735 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:23:07.735 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25057", 2024-08-19T14:23:07.735 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:23:07.735 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 50187 2024-08-19T14:23:07.735 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.735 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:23:07.735 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.735 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:23:07.735 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:20.755221+0000", 2024-08-19T14:23:07.735 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:07.735 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.735 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.736 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:23:07.736 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:20.755221+0000", 2024-08-19T14:23:07.736 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:07.736 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.736 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.736 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:23:07.736 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:20.755222+0000", 2024-08-19T14:23:07.736 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.133e-06 2024-08-19T14:23:07.736 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.736 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.736 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:23:07.736 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:20.755306+0000", 2024-08-19T14:23:07.736 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.4384999999999999e-05 2024-08-19T14:23:07.736 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.737 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.737 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:23:07.737 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:20.755319+0000", 2024-08-19T14:23:07.737 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3368e-05 2024-08-19T14:23:07.737 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.737 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.737 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:23:07.737 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:20.755327+0000", 2024-08-19T14:23:07.737 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.9529999999999999e-06 2024-08-19T14:23:07.737 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.737 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.737 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:23:07.737 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:20.755394+0000", 2024-08-19T14:23:07.738 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6836999999999997e-05 2024-08-19T14:23:07.738 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.738 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.738 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:23:07.738 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:20.755404+0000", 2024-08-19T14:23:07.738 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.022e-05 2024-08-19T14:23:07.738 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.738 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.738 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:23:07.738 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:21.743056+0000", 2024-08-19T14:23:07.738 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.98765108499999998 2024-08-19T14:23:07.738 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.738 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.739 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:23:07.739 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:21.743099+0000", 2024-08-19T14:23:07.739 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3849e-05 2024-08-19T14:23:07.739 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:07.739 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:23:07.739 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:07.739 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.739 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.739 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25057.0:50187 3.11 3:8ea4c110:::benchmark_data_smithi045_59973_object50186:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e237)", 2024-08-19T14:23:07.739 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:17:22.758606+0000", 2024-08-19T14:23:07.739 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 344.967303727, 2024-08-19T14:23:07.739 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.22265618500000001, 2024-08-19T14:23:07.739 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:23:07.739 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:23:07.740 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:23:07.740 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:23:07.740 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25057", 2024-08-19T14:23:07.740 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:23:07.740 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 50187 2024-08-19T14:23:07.740 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.740 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:23:07.740 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.740 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:23:07.740 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:22.758606+0000", 2024-08-19T14:23:07.740 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:07.740 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.740 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.740 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:23:07.741 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:22.758606+0000", 2024-08-19T14:23:07.741 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:07.741 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.741 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.741 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:23:07.741 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:22.758608+0000", 2024-08-19T14:23:07.741 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.967e-06 2024-08-19T14:23:07.741 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.741 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.741 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:23:07.741 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:22.758658+0000", 2024-08-19T14:23:07.741 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9842000000000002e-05 2024-08-19T14:23:07.741 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.741 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.741 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:23:07.742 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:22.758674+0000", 2024-08-19T14:23:07.742 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6022999999999999e-05 2024-08-19T14:23:07.742 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.742 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.742 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:23:07.742 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:22.758686+0000", 2024-08-19T14:23:07.742 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1895999999999999e-05 2024-08-19T14:23:07.742 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.742 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.742 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:23:07.742 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:22.759828+0000", 2024-08-19T14:23:07.742 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001142636 2024-08-19T14:23:07.742 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.742 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.743 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:23:07.743 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:22.759845+0000", 2024-08-19T14:23:07.743 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6158e-05 2024-08-19T14:23:07.743 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.743 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.743 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:23:07.743 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:22.881339+0000", 2024-08-19T14:23:07.744 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.121494108 2024-08-19T14:23:07.744 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.744 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.744 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:23:07.744 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:22.881449+0000", 2024-08-19T14:23:07.744 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000110331 2024-08-19T14:23:07.745 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.745 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.745 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0,3", 2024-08-19T14:23:07.745 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:22.881562+0000", 2024-08-19T14:23:07.745 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00011250499999999999 2024-08-19T14:23:07.745 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.745 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.745 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:23:07.745 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:22.882268+0000", 2024-08-19T14:23:07.745 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00070606200000000003 2024-08-19T14:23:07.745 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.745 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.745 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:23:07.745 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:22.891817+0000", 2024-08-19T14:23:07.746 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0095496739999999993 2024-08-19T14:23:07.746 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.746 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.746 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:23:07.746 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:22.980253+0000", 2024-08-19T14:23:07.746 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.088435402999999996 2024-08-19T14:23:07.746 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.746 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.746 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:23:07.746 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:22.980330+0000", 2024-08-19T14:23:07.746 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7447000000000006e-05 2024-08-19T14:23:07.746 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.747 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.747 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:23:07.747 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:22.981262+0000", 2024-08-19T14:23:07.747 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00093213300000000003 2024-08-19T14:23:07.747 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:07.747 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:23:07.747 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:07.747 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.747 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.747 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25057.0:63305 3.1a 3:594f84d9:::benchmark_data_smithi045_59973_object63304:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e244)", 2024-08-19T14:23:07.747 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:17:33.888645+0000", 2024-08-19T14:23:07.747 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 333.83726479000001, 2024-08-19T14:23:07.747 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0982461939999999, 2024-08-19T14:23:07.747 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:23:07.748 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:23:07.748 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:23:07.748 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:23:07.748 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25057", 2024-08-19T14:23:07.748 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:23:07.748 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 63305 2024-08-19T14:23:07.748 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.748 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:23:07.748 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.748 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:23:07.748 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:33.888645+0000", 2024-08-19T14:23:07.748 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:07.748 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.749 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.749 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:23:07.749 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:33.888645+0000", 2024-08-19T14:23:07.749 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:07.749 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.749 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.749 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:23:07.749 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:33.888646+0000", 2024-08-19T14:23:07.749 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2550000000000001e-06 2024-08-19T14:23:07.749 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.749 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.749 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:23:07.749 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:33.888765+0000", 2024-08-19T14:23:07.750 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00011885 2024-08-19T14:23:07.750 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.750 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.750 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:23:07.750 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:33.888786+0000", 2024-08-19T14:23:07.750 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0828000000000001e-05 2024-08-19T14:23:07.750 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.750 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.750 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:23:07.750 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:33.888795+0000", 2024-08-19T14:23:07.750 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.1649999999999995e-06 2024-08-19T14:23:07.750 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.750 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.750 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:23:07.751 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:33.996820+0000", 2024-08-19T14:23:07.751 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.108024672 2024-08-19T14:23:07.751 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.751 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.751 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:23:07.751 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:33.996825+0000", 2024-08-19T14:23:07.751 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8149999999999996e-06 2024-08-19T14:23:07.751 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.751 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.751 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:23:07.751 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:34.986807+0000", 2024-08-19T14:23:07.751 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.98998204300000003 2024-08-19T14:23:07.751 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.751 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.752 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:23:07.752 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:34.986891+0000", 2024-08-19T14:23:07.752 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.4566000000000003e-05 2024-08-19T14:23:07.752 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:07.752 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:23:07.752 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:07.752 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.752 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.752 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25057.0:63311 3.1a 3:5f4435a5:::benchmark_data_smithi045_59973_object63310:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e244)", 2024-08-19T14:23:07.752 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:17:33.888830+0000", 2024-08-19T14:23:07.752 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 333.83707976199997, 2024-08-19T14:23:07.752 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.098523232, 2024-08-19T14:23:07.752 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:23:07.753 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:23:07.753 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:23:07.753 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:23:07.753 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25057", 2024-08-19T14:23:07.753 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:23:07.753 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 63311 2024-08-19T14:23:07.753 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.753 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:23:07.753 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.753 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:23:07.753 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:33.888830+0000", 2024-08-19T14:23:07.753 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:07.753 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.753 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.754 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:23:07.754 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:33.888830+0000", 2024-08-19T14:23:07.754 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:07.754 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.754 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.754 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:23:07.754 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:33.888831+0000", 2024-08-19T14:23:07.754 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.99e-07 2024-08-19T14:23:07.754 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.754 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.754 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:23:07.754 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:33.888901+0000", 2024-08-19T14:23:07.754 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9948000000000002e-05 2024-08-19T14:23:07.755 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.755 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.755 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:23:07.755 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:33.888911+0000", 2024-08-19T14:23:07.755 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0679000000000001e-05 2024-08-19T14:23:07.755 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.755 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.755 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:23:07.755 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:33.888918+0000", 2024-08-19T14:23:07.755 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.5880000000000003e-06 2024-08-19T14:23:07.755 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.755 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.755 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:23:07.755 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:33.997358+0000", 2024-08-19T14:23:07.756 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.108439887 2024-08-19T14:23:07.756 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.756 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.756 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:23:07.756 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:33.997362+0000", 2024-08-19T14:23:07.756 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1219999999999997e-06 2024-08-19T14:23:07.756 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.756 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.756 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:23:07.756 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:34.987287+0000", 2024-08-19T14:23:07.756 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.989925481 2024-08-19T14:23:07.756 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.756 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.756 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:23:07.757 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:34.987353+0000", 2024-08-19T14:23:07.757 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.5827999999999994e-05 2024-08-19T14:23:07.757 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:07.757 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:23:07.757 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:07.757 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.757 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.757 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25057.0:64606 3.1b 3.702ddf7b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e246)", 2024-08-19T14:23:07.757 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:17:35.969630+0000", 2024-08-19T14:23:07.757 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 331.75628005499999, 2024-08-19T14:23:07.757 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.99329972499999997, 2024-08-19T14:23:07.757 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:23:07.757 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:23:07.757 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:23:07.758 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:23:07.758 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25057", 2024-08-19T14:23:07.758 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:23:07.758 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 64606 2024-08-19T14:23:07.758 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.758 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:23:07.758 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.758 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:23:07.758 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.969630+0000", 2024-08-19T14:23:07.758 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:07.758 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.758 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.759 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:23:07.759 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.969630+0000", 2024-08-19T14:23:07.759 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:07.759 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.759 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.759 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:23:07.759 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.969632+0000", 2024-08-19T14:23:07.759 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9069999999999999e-06 2024-08-19T14:23:07.759 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.759 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.759 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:23:07.759 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.969667+0000", 2024-08-19T14:23:07.759 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5187e-05 2024-08-19T14:23:07.759 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.760 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.760 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:23:07.760 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.969678+0000", 2024-08-19T14:23:07.760 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0849000000000001e-05 2024-08-19T14:23:07.760 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.760 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.760 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:23:07.760 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.969685+0000", 2024-08-19T14:23:07.760 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5039999999999996e-06 2024-08-19T14:23:07.760 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.760 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.760 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:23:07.760 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.969734+0000", 2024-08-19T14:23:07.760 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8690000000000003e-05 2024-08-19T14:23:07.761 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.761 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.761 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:23:07.761 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.969744+0000", 2024-08-19T14:23:07.761 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0183999999999999e-05 2024-08-19T14:23:07.761 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.761 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.761 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:23:07.761 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:36.962899+0000", 2024-08-19T14:23:07.761 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.99315520700000004 2024-08-19T14:23:07.761 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.761 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.761 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:23:07.761 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:36.962929+0000", 2024-08-19T14:23:07.762 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0196999999999999e-05 2024-08-19T14:23:07.762 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:07.762 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:23:07.762 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:07.762 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.762 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.762 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25057.0:64610 3.1b 3.810f60bb (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e246)", 2024-08-19T14:23:07.762 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:17:35.977398+0000", 2024-08-19T14:23:07.762 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 331.74851133099997, 2024-08-19T14:23:07.762 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.98556503799999995, 2024-08-19T14:23:07.762 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:23:07.762 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:23:07.762 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:23:07.763 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:23:07.763 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25057", 2024-08-19T14:23:07.763 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:23:07.763 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 64610 2024-08-19T14:23:07.763 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.763 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:23:07.763 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.763 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:23:07.763 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.977398+0000", 2024-08-19T14:23:07.763 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:07.764 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.764 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.764 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:23:07.764 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.977398+0000", 2024-08-19T14:23:07.764 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:07.764 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.764 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.764 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:23:07.764 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.977400+0000", 2024-08-19T14:23:07.764 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.469e-06 2024-08-19T14:23:07.764 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.764 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.764 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:23:07.764 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.977457+0000", 2024-08-19T14:23:07.765 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7043999999999998e-05 2024-08-19T14:23:07.765 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.765 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.765 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:23:07.765 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.977465+0000", 2024-08-19T14:23:07.765 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.2320000000000001e-06 2024-08-19T14:23:07.765 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.765 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.765 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:23:07.765 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.977470+0000", 2024-08-19T14:23:07.765 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5759999999999999e-06 2024-08-19T14:23:07.765 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.765 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.765 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:23:07.766 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.978418+0000", 2024-08-19T14:23:07.766 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00094818599999999995 2024-08-19T14:23:07.766 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.766 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.766 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:23:07.766 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.978424+0000", 2024-08-19T14:23:07.766 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2979999999999997e-06 2024-08-19T14:23:07.766 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.766 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.766 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:23:07.766 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:36.962956+0000", 2024-08-19T14:23:07.766 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.98453159300000004 2024-08-19T14:23:07.766 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.766 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.767 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:23:07.767 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:36.962963+0000", 2024-08-19T14:23:07.767 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6399999999999997e-06 2024-08-19T14:23:07.767 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:07.767 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:23:07.767 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:07.767 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.767 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.767 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25057.0:64616 3.1b 3.cab7ee3b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e246)", 2024-08-19T14:23:07.767 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:17:35.982530+0000", 2024-08-19T14:23:07.767 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 331.74338008799998, 2024-08-19T14:23:07.767 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.98047576599999997, 2024-08-19T14:23:07.767 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:23:07.768 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:23:07.768 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:23:07.768 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:23:07.768 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25057", 2024-08-19T14:23:07.768 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:23:07.768 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 64616 2024-08-19T14:23:07.768 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.768 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:23:07.768 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.768 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:23:07.768 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.982530+0000", 2024-08-19T14:23:07.768 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:07.768 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.768 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.769 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:23:07.769 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.982530+0000", 2024-08-19T14:23:07.769 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:07.769 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.769 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.769 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:23:07.769 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.982531+0000", 2024-08-19T14:23:07.769 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.626e-06 2024-08-19T14:23:07.769 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.769 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.769 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:23:07.769 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.982589+0000", 2024-08-19T14:23:07.769 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8013000000000002e-05 2024-08-19T14:23:07.769 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.769 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.770 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:23:07.770 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.982603+0000", 2024-08-19T14:23:07.770 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.384e-05 2024-08-19T14:23:07.770 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.770 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.770 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:23:07.770 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.982611+0000", 2024-08-19T14:23:07.770 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6329999999999996e-06 2024-08-19T14:23:07.770 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.770 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.770 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:23:07.770 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.982674+0000", 2024-08-19T14:23:07.770 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3575e-05 2024-08-19T14:23:07.771 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.771 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.771 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:23:07.771 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.982686+0000", 2024-08-19T14:23:07.771 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1929999999999999e-05 2024-08-19T14:23:07.771 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.771 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.771 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:23:07.771 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:36.962998+0000", 2024-08-19T14:23:07.771 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.98031178900000004 2024-08-19T14:23:07.771 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.771 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.771 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:23:07.771 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:36.963005+0000", 2024-08-19T14:23:07.772 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3599999999999998e-06 2024-08-19T14:23:07.772 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:07.772 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:23:07.772 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:07.772 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.772 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.772 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25057.0:64585 3.1e 3:78822400:::benchmark_data_smithi045_59973_object64584:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e248)", 2024-08-19T14:23:07.772 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:17:37.897689+0000", 2024-08-19T14:23:07.772 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 329.82822043800002, 2024-08-19T14:23:07.772 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.207104121, 2024-08-19T14:23:07.772 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:23:07.772 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:23:07.772 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:23:07.772 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:23:07.773 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25057", 2024-08-19T14:23:07.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:07.773 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:07.773 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:23:07.774 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:23:07.774 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 64585 2024-08-19T14:23:07.774 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.774 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:23:07.774 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.775 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:23:07.775 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:37.897689+0000", 2024-08-19T14:23:07.775 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:07.775 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.775 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.775 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:23:07.775 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:37.897689+0000", 2024-08-19T14:23:07.775 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:07.775 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.775 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.776 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:23:07.776 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:37.897690+0000", 2024-08-19T14:23:07.776 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1900000000000003e-07 2024-08-19T14:23:07.776 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.776 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.776 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:23:07.776 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:37.897735+0000", 2024-08-19T14:23:07.776 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4876000000000002e-05 2024-08-19T14:23:07.776 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.776 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.776 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:23:07.776 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:37.897742+0000", 2024-08-19T14:23:07.777 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5429999999999996e-06 2024-08-19T14:23:07.777 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.777 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.777 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:23:07.777 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:37.897748+0000", 2024-08-19T14:23:07.777 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3510000000000001e-06 2024-08-19T14:23:07.777 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.777 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.777 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:23:07.777 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:37.961806+0000", 2024-08-19T14:23:07.778 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.064058459999999998 2024-08-19T14:23:07.778 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.778 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.778 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for degraded object", 2024-08-19T14:23:07.778 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:37.962240+0000", 2024-08-19T14:23:07.778 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00043381799999999999 2024-08-19T14:23:07.778 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.778 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.778 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:23:07.778 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:38.104692+0000", 2024-08-19T14:23:07.778 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.14245212400000001 2024-08-19T14:23:07.778 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.779 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.779 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:23:07.779 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:38.104793+0000", 2024-08-19T14:23:07.779 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00010143 2024-08-19T14:23:07.779 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:07.779 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:23:07.779 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:07.779 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.779 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.779 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25057.0:64589 3.1e 3:7d191b55:::benchmark_data_smithi045_59973_object64588:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e248)", 2024-08-19T14:23:07.779 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:17:37.897758+0000", 2024-08-19T14:23:07.779 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 329.82815201599999, 2024-08-19T14:23:07.779 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20971097999999999, 2024-08-19T14:23:07.779 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:23:07.779 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:23:07.780 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:23:07.780 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:23:07.780 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25057", 2024-08-19T14:23:07.780 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:23:07.780 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 64589 2024-08-19T14:23:07.780 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.780 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:23:07.780 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.780 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:23:07.780 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:37.897758+0000", 2024-08-19T14:23:07.780 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:07.780 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.780 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.781 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:23:07.781 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:37.897758+0000", 2024-08-19T14:23:07.781 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:07.781 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.781 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.781 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:23:07.781 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:37.897758+0000", 2024-08-19T14:23:07.781 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3300000000000003e-07 2024-08-19T14:23:07.781 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.781 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.781 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:23:07.781 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:37.897792+0000", 2024-08-19T14:23:07.781 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3327000000000001e-05 2024-08-19T14:23:07.781 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.781 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.782 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:23:07.782 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:37.897800+0000", 2024-08-19T14:23:07.782 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.5129999999999996e-06 2024-08-19T14:23:07.782 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.782 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.782 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:23:07.782 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:37.897805+0000", 2024-08-19T14:23:07.782 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9869999999999999e-06 2024-08-19T14:23:07.782 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.782 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.782 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:23:07.782 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:37.962303+0000", 2024-08-19T14:23:07.782 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.064497909000000006 2024-08-19T14:23:07.783 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.783 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.783 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for degraded object", 2024-08-19T14:23:07.783 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:37.962758+0000", 2024-08-19T14:23:07.783 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00045486799999999998 2024-08-19T14:23:07.783 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.783 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.783 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:23:07.783 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:38.107362+0000", 2024-08-19T14:23:07.783 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.144604434 2024-08-19T14:23:07.784 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.784 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.784 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:23:07.784 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:38.107469+0000", 2024-08-19T14:23:07.784 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00010650900000000001 2024-08-19T14:23:07.784 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:07.784 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:23:07.784 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:07.784 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.784 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.784 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25057.0:64659 3.1e 3:7bada211:::benchmark_data_smithi045_59973_object64658:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e248)", 2024-08-19T14:23:07.784 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:17:37.897812+0000", 2024-08-19T14:23:07.785 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 329.82809765500002, 2024-08-19T14:23:07.785 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.21203138599999999, 2024-08-19T14:23:07.785 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:23:07.785 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:23:07.785 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:23:07.785 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:23:07.785 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25057", 2024-08-19T14:23:07.785 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:23:07.785 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 64659 2024-08-19T14:23:07.785 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.785 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:23:07.785 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.785 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:23:07.785 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:37.897812+0000", 2024-08-19T14:23:07.786 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:07.786 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.786 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.786 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:23:07.786 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:37.897812+0000", 2024-08-19T14:23:07.786 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:07.786 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.786 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.786 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:23:07.786 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:37.897813+0000", 2024-08-19T14:23:07.786 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2199999999999999e-07 2024-08-19T14:23:07.786 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.786 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.786 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:23:07.787 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:37.897845+0000", 2024-08-19T14:23:07.787 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2447999999999998e-05 2024-08-19T14:23:07.787 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.787 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.787 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:23:07.787 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:37.897852+0000", 2024-08-19T14:23:07.787 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6640000000000001e-06 2024-08-19T14:23:07.787 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.787 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.787 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:23:07.787 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:37.897856+0000", 2024-08-19T14:23:07.787 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7180000000000004e-06 2024-08-19T14:23:07.788 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.788 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.788 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:23:07.788 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:37.962786+0000", 2024-08-19T14:23:07.788 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.064930102000000003 2024-08-19T14:23:07.788 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.788 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.788 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:23:07.788 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:37.962879+0000", 2024-08-19T14:23:07.788 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.2677999999999996e-05 2024-08-19T14:23:07.788 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.788 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.788 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 3,8,9", 2024-08-19T14:23:07.788 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:37.963005+0000", 2024-08-19T14:23:07.789 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012557399999999999 2024-08-19T14:23:07.789 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.789 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.789 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:23:07.789 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:37.964594+0000", 2024-08-19T14:23:07.789 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001589008 2024-08-19T14:23:07.789 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.789 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.789 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:23:07.789 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:37.969399+0000", 2024-08-19T14:23:07.789 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0048054600000000001 2024-08-19T14:23:07.789 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.789 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.789 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:23:07.789 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:37.971415+0000", 2024-08-19T14:23:07.790 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0020157700000000001 2024-08-19T14:23:07.790 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.790 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.790 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:23:07.790 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:38.107519+0000", 2024-08-19T14:23:07.790 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.136104371 2024-08-19T14:23:07.790 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.790 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.790 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:23:07.790 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:38.108261+0000", 2024-08-19T14:23:07.790 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00074117400000000004 2024-08-19T14:23:07.790 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.791 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.791 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:23:07.791 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:38.109844+0000", 2024-08-19T14:23:07.791 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001582997 2024-08-19T14:23:07.791 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:07.791 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:23:07.791 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:07.791 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.791 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.791 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25057.0:188779 3.2a 3:54ce2294:::benchmark_data_smithi045_59973_object57600:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-08-19T14:23:07.791 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:19:02.351958+0000", 2024-08-19T14:23:07.791 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 245.37395185899999, 2024-08-19T14:23:07.791 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.86379963100000001, 2024-08-19T14:23:07.792 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:23:07.792 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:23:07.792 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:23:07.792 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:23:07.792 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25057", 2024-08-19T14:23:07.792 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:23:07.792 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 188779 2024-08-19T14:23:07.792 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.792 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:23:07.792 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.792 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:23:07.792 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:19:02.351958+0000", 2024-08-19T14:23:07.792 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:07.792 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.793 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.793 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:23:07.793 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:19:02.351958+0000", 2024-08-19T14:23:07.793 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:07.793 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.793 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.793 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:23:07.793 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:19:02.351959+0000", 2024-08-19T14:23:07.793 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.3200000000000004e-07 2024-08-19T14:23:07.793 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.793 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.793 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:23:07.793 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:19:02.351962+0000", 2024-08-19T14:23:07.793 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.32e-06 2024-08-19T14:23:07.794 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.794 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.794 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:23:07.794 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:19:02.351970+0000", 2024-08-19T14:23:07.794 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6790000000000005e-06 2024-08-19T14:23:07.794 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.794 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.794 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:23:07.794 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:19:02.351974+0000", 2024-08-19T14:23:07.794 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5979999999999999e-06 2024-08-19T14:23:07.794 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.794 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.794 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:23:07.795 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:19:02.352047+0000", 2024-08-19T14:23:07.795 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2187999999999994e-05 2024-08-19T14:23:07.795 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.795 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.795 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:23:07.795 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:19:02.352057+0000", 2024-08-19T14:23:07.795 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0642000000000001e-05 2024-08-19T14:23:07.795 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.795 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.795 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:23:07.795 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:19:03.210075+0000", 2024-08-19T14:23:07.795 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.85801831699999997 2024-08-19T14:23:07.795 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.795 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.796 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:23:07.796 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:19:03.210203+0000", 2024-08-19T14:23:07.796 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012764300000000001 2024-08-19T14:23:07.796 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.796 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.796 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 1,8", 2024-08-19T14:23:07.796 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:19:03.210318+0000", 2024-08-19T14:23:07.796 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000114954 2024-08-19T14:23:07.796 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.796 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.796 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:23:07.796 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:19:03.212413+0000", 2024-08-19T14:23:07.796 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0020953249999999999 2024-08-19T14:23:07.797 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.797 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.797 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:23:07.797 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:19:03.212766+0000", 2024-08-19T14:23:07.797 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00035248899999999999 2024-08-19T14:23:07.797 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.797 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.797 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:23:07.797 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:19:03.215663+0000", 2024-08-19T14:23:07.797 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0028968959999999999 2024-08-19T14:23:07.797 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.797 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.797 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:23:07.797 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:19:03.215714+0000", 2024-08-19T14:23:07.797 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0719e-05 2024-08-19T14:23:07.798 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.798 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.798 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:23:07.798 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:19:03.215758+0000", 2024-08-19T14:23:07.798 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4029000000000001e-05 2024-08-19T14:23:07.798 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:07.798 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:23:07.798 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:07.798 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.798 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.798 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25333.0:54277 4.d 4:b77eedff:::benchmark_data_smithi045_71562_object54276:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-08-19T14:23:07.798 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:20:57.352004+0000", 2024-08-19T14:23:07.798 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 130.373906084, 2024-08-19T14:23:07.799 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.209036954, 2024-08-19T14:23:07.799 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:23:07.799 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:23:07.799 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:23:07.799 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:23:07.799 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25333", 2024-08-19T14:23:07.799 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:23:07.799 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 54277 2024-08-19T14:23:07.799 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.799 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:23:07.799 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.799 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:23:07.799 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:57.352004+0000", 2024-08-19T14:23:07.799 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:07.800 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.800 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.800 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:23:07.800 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:57.352004+0000", 2024-08-19T14:23:07.800 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:07.800 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.800 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.800 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:23:07.800 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:57.352005+0000", 2024-08-19T14:23:07.800 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.035e-06 2024-08-19T14:23:07.800 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.800 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.800 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:23:07.800 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:57.352034+0000", 2024-08-19T14:23:07.801 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9580000000000001e-05 2024-08-19T14:23:07.801 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.801 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.801 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:23:07.801 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:57.352044+0000", 2024-08-19T14:23:07.801 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0104e-05 2024-08-19T14:23:07.801 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.801 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.801 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:23:07.801 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:57.352052+0000", 2024-08-19T14:23:07.801 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.148e-06 2024-08-19T14:23:07.801 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.801 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.801 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:23:07.802 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:57.352111+0000", 2024-08-19T14:23:07.802 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9392000000000003e-05 2024-08-19T14:23:07.802 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.802 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.802 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:23:07.802 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:57.352268+0000", 2024-08-19T14:23:07.802 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000157372 2024-08-19T14:23:07.802 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.802 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.802 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 1,9", 2024-08-19T14:23:07.802 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:57.352417+0000", 2024-08-19T14:23:07.802 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014890199999999999 2024-08-19T14:23:07.802 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.802 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.803 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:23:07.803 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:57.354049+0000", 2024-08-19T14:23:07.803 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0016321650000000001 2024-08-19T14:23:07.803 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.803 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.803 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:23:07.803 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:57.354867+0000", 2024-08-19T14:23:07.803 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00081795800000000003 2024-08-19T14:23:07.803 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.803 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.803 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:23:07.803 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:57.560974+0000", 2024-08-19T14:23:07.803 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20610681 2024-08-19T14:23:07.803 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.804 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.804 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:23:07.804 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:57.561017+0000", 2024-08-19T14:23:07.804 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2531000000000003e-05 2024-08-19T14:23:07.804 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.804 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.804 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:23:07.804 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:57.561041+0000", 2024-08-19T14:23:07.804 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3957000000000001e-05 2024-08-19T14:23:07.804 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:07.804 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:23:07.804 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:07.804 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.805 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.805 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25333.0:56673 4.e 4:7feebc47:::benchmark_data_smithi045_71562_object56672:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-08-19T14:23:07.805 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:20:59.123818+0000", 2024-08-19T14:23:07.805 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 128.60209208200001, 2024-08-19T14:23:07.805 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20982393999999999, 2024-08-19T14:23:07.805 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:23:07.805 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:23:07.805 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:23:07.805 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:23:07.805 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25333", 2024-08-19T14:23:07.805 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:23:07.805 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 56673 2024-08-19T14:23:07.805 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.805 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:23:07.805 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.806 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:23:07.806 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:59.123818+0000", 2024-08-19T14:23:07.806 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:07.806 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.806 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.806 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:23:07.806 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:59.123818+0000", 2024-08-19T14:23:07.806 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:07.806 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.806 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.806 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:23:07.806 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:59.123819+0000", 2024-08-19T14:23:07.807 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2079999999999999e-06 2024-08-19T14:23:07.807 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.807 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.807 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:23:07.807 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:59.123880+0000", 2024-08-19T14:23:07.807 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0890000000000001e-05 2024-08-19T14:23:07.807 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.807 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.807 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:23:07.807 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:59.123895+0000", 2024-08-19T14:23:07.807 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5284000000000001e-05 2024-08-19T14:23:07.807 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.807 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.807 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:23:07.808 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:59.123903+0000", 2024-08-19T14:23:07.808 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.0299999999999994e-06 2024-08-19T14:23:07.808 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.808 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.808 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:23:07.808 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:59.124075+0000", 2024-08-19T14:23:07.808 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00017151100000000001 2024-08-19T14:23:07.808 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.808 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.808 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:23:07.808 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:59.124207+0000", 2024-08-19T14:23:07.808 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013249600000000001 2024-08-19T14:23:07.808 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.808 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.809 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 1,3", 2024-08-19T14:23:07.809 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:59.124291+0000", 2024-08-19T14:23:07.809 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.4339999999999995e-05 2024-08-19T14:23:07.809 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.809 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.809 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:23:07.809 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:59.125565+0000", 2024-08-19T14:23:07.809 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0012739489999999999 2024-08-19T14:23:07.809 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.809 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.809 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:23:07.809 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:59.128881+0000", 2024-08-19T14:23:07.809 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.003316079 2024-08-19T14:23:07.809 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.809 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.810 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:23:07.810 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:59.333544+0000", 2024-08-19T14:23:07.810 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20466241900000001 2024-08-19T14:23:07.810 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.810 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.810 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:23:07.810 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:59.333604+0000", 2024-08-19T14:23:07.810 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0569000000000002e-05 2024-08-19T14:23:07.810 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.810 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.810 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:23:07.810 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:59.333642+0000", 2024-08-19T14:23:07.810 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7165000000000001e-05 2024-08-19T14:23:07.810 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:07.811 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:23:07.811 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:07.811 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.811 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.811 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25333.0:56679 4.e 4:7b928f24:::benchmark_data_smithi045_71562_object56678:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-08-19T14:23:07.811 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:20:59.126484+0000", 2024-08-19T14:23:07.811 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 128.59942622099999, 2024-08-19T14:23:07.811 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20890175799999999, 2024-08-19T14:23:07.811 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:23:07.811 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:23:07.811 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:23:07.811 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:23:07.811 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25333", 2024-08-19T14:23:07.811 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:23:07.811 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 56679 2024-08-19T14:23:07.812 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.812 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:23:07.812 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.812 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:23:07.812 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:59.126484+0000", 2024-08-19T14:23:07.812 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:07.812 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.812 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.812 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:23:07.812 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:59.126484+0000", 2024-08-19T14:23:07.812 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:07.812 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.812 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.812 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:23:07.812 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:59.126484+0000", 2024-08-19T14:23:07.813 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.5600000000000004e-07 2024-08-19T14:23:07.813 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.813 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.813 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:23:07.813 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:59.126523+0000", 2024-08-19T14:23:07.813 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8637999999999998e-05 2024-08-19T14:23:07.813 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.813 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.813 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:23:07.813 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:59.126552+0000", 2024-08-19T14:23:07.813 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8824999999999999e-05 2024-08-19T14:23:07.813 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.813 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.813 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:23:07.813 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:59.126560+0000", 2024-08-19T14:23:07.813 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.9540000000000004e-06 2024-08-19T14:23:07.814 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.814 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.814 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:23:07.814 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:59.126617+0000", 2024-08-19T14:23:07.814 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7201999999999999e-05 2024-08-19T14:23:07.814 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.814 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.814 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:23:07.814 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:59.126734+0000", 2024-08-19T14:23:07.814 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000116554 2024-08-19T14:23:07.814 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.814 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.814 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 1,3", 2024-08-19T14:23:07.814 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:59.126853+0000", 2024-08-19T14:23:07.814 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000119385 2024-08-19T14:23:07.815 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.815 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.815 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:23:07.815 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:59.131576+0000", 2024-08-19T14:23:07.815 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0047225990000000001 2024-08-19T14:23:07.815 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.815 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.815 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:23:07.815 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:59.132201+0000", 2024-08-19T14:23:07.815 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00062569700000000004 2024-08-19T14:23:07.815 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.815 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.815 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:23:07.815 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:59.335309+0000", 2024-08-19T14:23:07.815 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20310735299999999 2024-08-19T14:23:07.816 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.816 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.816 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:23:07.816 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:59.335348+0000", 2024-08-19T14:23:07.816 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9536999999999998e-05 2024-08-19T14:23:07.816 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.817 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.817 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:23:07.817 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:59.335385+0000", 2024-08-19T14:23:07.817 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7157999999999999e-05 2024-08-19T14:23:07.817 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:07.817 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:23:07.817 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:07.817 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.817 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.817 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25333.0:56683 4.e 4:7f9bbff2:::benchmark_data_smithi045_71562_object56682:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-08-19T14:23:07.818 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:20:59.129285+0000", 2024-08-19T14:23:07.818 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 128.596624921, 2024-08-19T14:23:07.818 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20620688400000001, 2024-08-19T14:23:07.818 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:23:07.818 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:23:07.818 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:23:07.818 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:23:07.818 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25333", 2024-08-19T14:23:07.818 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:23:07.818 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 56683 2024-08-19T14:23:07.818 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.818 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:23:07.818 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.818 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:23:07.819 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:59.129285+0000", 2024-08-19T14:23:07.819 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:07.819 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.819 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.819 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:23:07.819 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:59.129285+0000", 2024-08-19T14:23:07.819 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:07.819 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.819 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.819 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:23:07.819 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:59.129286+0000", 2024-08-19T14:23:07.819 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.99e-07 2024-08-19T14:23:07.819 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.820 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.820 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:23:07.820 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:59.129374+0000", 2024-08-19T14:23:07.820 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.8918999999999993e-05 2024-08-19T14:23:07.820 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.820 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.820 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:23:07.820 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:59.129385+0000", 2024-08-19T14:23:07.820 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0995999999999999e-05 2024-08-19T14:23:07.820 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.820 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.820 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:23:07.820 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:59.129393+0000", 2024-08-19T14:23:07.820 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4220000000000003e-06 2024-08-19T14:23:07.821 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.821 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.821 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:23:07.821 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:59.129444+0000", 2024-08-19T14:23:07.821 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1236000000000002e-05 2024-08-19T14:23:07.821 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.821 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.821 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:23:07.821 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:59.129562+0000", 2024-08-19T14:23:07.821 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000118299 2024-08-19T14:23:07.821 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.821 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.821 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 1,3", 2024-08-19T14:23:07.821 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:59.129700+0000", 2024-08-19T14:23:07.822 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013714 2024-08-19T14:23:07.822 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.822 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.822 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:23:07.822 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:59.131712+0000", 2024-08-19T14:23:07.822 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0020121480000000001 2024-08-19T14:23:07.822 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.822 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.822 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:23:07.822 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:59.132244+0000", 2024-08-19T14:23:07.822 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00053220000000000003 2024-08-19T14:23:07.822 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.822 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.823 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:23:07.823 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:59.335431+0000", 2024-08-19T14:23:07.823 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.203187118 2024-08-19T14:23:07.823 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.823 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.823 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:23:07.823 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:59.335468+0000", 2024-08-19T14:23:07.823 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7020999999999997e-05 2024-08-19T14:23:07.823 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.823 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.823 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:23:07.823 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:59.335492+0000", 2024-08-19T14:23:07.823 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3686e-05 2024-08-19T14:23:07.823 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:07.824 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:23:07.824 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:07.824 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.824 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.824 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25333.0:56691 4.e 4:7f9706d8:::benchmark_data_smithi045_71562_object56690:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-08-19T14:23:07.824 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:20:59.132973+0000", 2024-08-19T14:23:07.824 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 128.59293665000001, 2024-08-19T14:23:07.824 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20674016100000001, 2024-08-19T14:23:07.824 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:23:07.824 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:23:07.824 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:23:07.824 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:23:07.825 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25333", 2024-08-19T14:23:07.825 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:23:07.825 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 56691 2024-08-19T14:23:07.825 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.825 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:23:07.825 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.826 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:23:07.826 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:59.132973+0000", 2024-08-19T14:23:07.826 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:07.826 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.826 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.826 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:23:07.826 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:59.132973+0000", 2024-08-19T14:23:07.826 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:07.826 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.826 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.826 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:23:07.826 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:59.132974+0000", 2024-08-19T14:23:07.826 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6000000000000002e-07 2024-08-19T14:23:07.826 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.827 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.827 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:23:07.827 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:59.133023+0000", 2024-08-19T14:23:07.827 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9069000000000001e-05 2024-08-19T14:23:07.827 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.827 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.827 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:23:07.827 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:59.133035+0000", 2024-08-19T14:23:07.827 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1989e-05 2024-08-19T14:23:07.827 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.827 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.827 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:23:07.827 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:59.133042+0000", 2024-08-19T14:23:07.827 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9179999999999997e-06 2024-08-19T14:23:07.828 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.828 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.828 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:23:07.828 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:59.133089+0000", 2024-08-19T14:23:07.828 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7290000000000003e-05 2024-08-19T14:23:07.828 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.828 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.828 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:23:07.828 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:59.133179+0000", 2024-08-19T14:23:07.828 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.9870000000000005e-05 2024-08-19T14:23:07.828 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.828 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.828 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 1,3", 2024-08-19T14:23:07.829 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:59.133261+0000", 2024-08-19T14:23:07.829 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.2013000000000002e-05 2024-08-19T14:23:07.829 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.829 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.829 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:23:07.829 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:59.134810+0000", 2024-08-19T14:23:07.829 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0015485329999999999 2024-08-19T14:23:07.829 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.829 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.829 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:23:07.829 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:59.135214+0000", 2024-08-19T14:23:07.829 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00040444800000000001 2024-08-19T14:23:07.829 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.829 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.830 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:23:07.830 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:59.339654+0000", 2024-08-19T14:23:07.830 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.204439597 2024-08-19T14:23:07.830 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.830 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.830 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:23:07.830 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:59.339687+0000", 2024-08-19T14:23:07.830 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3061000000000001e-05 2024-08-19T14:23:07.830 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.830 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.830 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:23:07.830 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:59.339713+0000", 2024-08-19T14:23:07.830 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6512999999999998e-05 2024-08-19T14:23:07.830 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:07.830 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:23:07.831 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:07.831 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.831 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.831 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25333.0:149067 4.e 4:7e247c1a:::benchmark_data_smithi045_71562_object17892:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e378)", 2024-08-19T14:23:07.831 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:22:24.357990+0000", 2024-08-19T14:23:07.831 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 43.367919297, 2024-08-19T14:23:07.831 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.22065488899999999, 2024-08-19T14:23:07.831 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:23:07.831 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:23:07.831 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:23:07.831 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:23:07.832 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25333", 2024-08-19T14:23:07.832 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:23:07.832 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 149067 2024-08-19T14:23:07.832 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.832 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:23:07.832 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.832 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:23:07.832 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:22:24.357990+0000", 2024-08-19T14:23:07.832 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:07.832 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.832 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.832 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:23:07.832 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:22:24.357990+0000", 2024-08-19T14:23:07.832 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:07.833 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.833 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.833 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:23:07.833 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:22:24.357992+0000", 2024-08-19T14:23:07.833 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.499e-06 2024-08-19T14:23:07.833 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.833 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.833 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:23:07.833 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:22:24.357998+0000", 2024-08-19T14:23:07.833 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1210000000000003e-06 2024-08-19T14:23:07.833 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.833 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.833 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:23:07.834 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:22:24.358011+0000", 2024-08-19T14:23:07.834 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2959000000000001e-05 2024-08-19T14:23:07.834 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.834 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.834 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:23:07.834 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:22:24.358019+0000", 2024-08-19T14:23:07.834 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1529999999999997e-06 2024-08-19T14:23:07.834 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.834 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.834 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:23:07.834 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:22:24.358081+0000", 2024-08-19T14:23:07.834 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1508999999999997e-05 2024-08-19T14:23:07.834 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.835 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.835 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:23:07.835 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:22:24.358218+0000", 2024-08-19T14:23:07.835 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013742 2024-08-19T14:23:07.835 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.835 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.835 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 1,3", 2024-08-19T14:23:07.835 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:22:24.358334+0000", 2024-08-19T14:23:07.835 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000116048 2024-08-19T14:23:07.835 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.835 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.835 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:23:07.835 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:22:24.367414+0000", 2024-08-19T14:23:07.836 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0090802490000000003 2024-08-19T14:23:07.836 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.836 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.836 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:23:07.836 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:22:24.368374+0000", 2024-08-19T14:23:07.836 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00095930899999999999 2024-08-19T14:23:07.836 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.836 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.836 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:23:07.836 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:22:24.578569+0000", 2024-08-19T14:23:07.836 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.210195359 2024-08-19T14:23:07.836 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.836 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.836 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:23:07.836 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:22:24.578607+0000", 2024-08-19T14:23:07.837 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7969999999999997e-05 2024-08-19T14:23:07.837 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.837 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.837 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:23:07.837 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:22:24.578645+0000", 2024-08-19T14:23:07.837 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8293000000000001e-05 2024-08-19T14:23:07.837 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:07.837 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:23:07.837 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:07.837 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.837 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.837 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25333.0:149070 4.e 4:77cd6059:::benchmark_data_smithi045_71562_object17895:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e378)", 2024-08-19T14:23:07.837 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:22:24.358621+0000", 2024-08-19T14:23:07.837 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 43.367288352999999, 2024-08-19T14:23:07.838 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.223820359, 2024-08-19T14:23:07.838 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:23:07.838 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:23:07.838 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:23:07.838 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:23:07.838 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25333", 2024-08-19T14:23:07.838 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:23:07.838 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 149070 2024-08-19T14:23:07.838 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.838 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:23:07.838 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.838 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:23:07.839 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:22:24.358621+0000", 2024-08-19T14:23:07.839 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:07.839 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.839 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.839 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:23:07.839 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:22:24.358621+0000", 2024-08-19T14:23:07.839 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:07.839 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.839 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.839 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:23:07.839 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:22:24.358622+0000", 2024-08-19T14:23:07.839 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.0999999999999997e-07 2024-08-19T14:23:07.839 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.839 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.840 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:23:07.840 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:22:24.358627+0000", 2024-08-19T14:23:07.840 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3760000000000001e-06 2024-08-19T14:23:07.840 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.840 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.840 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:23:07.840 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:22:24.358642+0000", 2024-08-19T14:23:07.840 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5582000000000001e-05 2024-08-19T14:23:07.840 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.840 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.840 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:23:07.840 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:22:24.358650+0000", 2024-08-19T14:23:07.840 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5079999999999999e-06 2024-08-19T14:23:07.841 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.841 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.841 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:23:07.841 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:22:24.362772+0000", 2024-08-19T14:23:07.841 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0041220220000000004 2024-08-19T14:23:07.841 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.841 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.841 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:23:07.841 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:22:24.362918+0000", 2024-08-19T14:23:07.841 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014636200000000001 2024-08-19T14:23:07.841 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.841 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.841 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 1,3", 2024-08-19T14:23:07.841 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:22:24.363057+0000", 2024-08-19T14:23:07.842 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013936800000000001 2024-08-19T14:23:07.842 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.842 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.842 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:23:07.842 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:22:24.367511+0000", 2024-08-19T14:23:07.842 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0044538939999999999 2024-08-19T14:23:07.842 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.842 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.842 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:23:07.842 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:22:24.379873+0000", 2024-08-19T14:23:07.842 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.01236145 2024-08-19T14:23:07.842 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.842 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.842 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:23:07.843 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:22:24.582321+0000", 2024-08-19T14:23:07.843 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.202448131 2024-08-19T14:23:07.843 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.843 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.843 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:23:07.843 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:22:24.582389+0000", 2024-08-19T14:23:07.843 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7778000000000001e-05 2024-08-19T14:23:07.843 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.843 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.843 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:23:07.843 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:22:24.582442+0000", 2024-08-19T14:23:07.843 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3078000000000002e-05 2024-08-19T14:23:07.843 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:07.843 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:23:07.844 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:07.844 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.844 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.844 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25333.0:149082 4.e 4:766b7f6c:::benchmark_data_smithi045_71562_object17907:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e378)", 2024-08-19T14:23:07.844 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:22:24.373601+0000", 2024-08-19T14:23:07.844 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 43.352308991000001, 2024-08-19T14:23:07.844 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.208952306, 2024-08-19T14:23:07.844 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:23:07.844 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:23:07.844 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:23:07.844 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:23:07.844 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25333", 2024-08-19T14:23:07.844 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:23:07.844 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 149082 2024-08-19T14:23:07.845 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.845 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:23:07.845 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.845 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:23:07.845 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:22:24.373601+0000", 2024-08-19T14:23:07.845 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:07.845 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.845 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.845 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:23:07.845 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:22:24.373601+0000", 2024-08-19T14:23:07.845 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:07.845 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.845 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.845 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:23:07.846 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:22:24.373602+0000", 2024-08-19T14:23:07.846 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4000000000000001e-07 2024-08-19T14:23:07.846 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.846 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.846 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:23:07.846 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:22:24.373609+0000", 2024-08-19T14:23:07.846 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1559999999999998e-06 2024-08-19T14:23:07.846 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.846 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.846 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:23:07.846 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:22:24.373619+0000", 2024-08-19T14:23:07.846 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0664e-05 2024-08-19T14:23:07.846 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.847 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.847 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:23:07.847 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:22:24.373626+0000", 2024-08-19T14:23:07.847 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.562e-06 2024-08-19T14:23:07.847 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.847 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.847 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:23:07.847 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:22:24.373680+0000", 2024-08-19T14:23:07.847 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4151999999999999e-05 2024-08-19T14:23:07.847 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.847 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.847 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:23:07.847 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:22:24.373789+0000", 2024-08-19T14:23:07.847 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00010857300000000001 2024-08-19T14:23:07.848 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.848 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.848 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 1,3", 2024-08-19T14:23:07.848 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:22:24.373892+0000", 2024-08-19T14:23:07.848 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000103099 2024-08-19T14:23:07.848 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.848 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.848 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:23:07.848 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:22:24.378370+0000", 2024-08-19T14:23:07.848 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0044786519999999996 2024-08-19T14:23:07.848 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.848 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.848 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:23:07.848 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:22:24.383595+0000", 2024-08-19T14:23:07.849 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0052250700000000001 2024-08-19T14:23:07.849 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.849 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.849 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:23:07.849 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:22:24.582481+0000", 2024-08-19T14:23:07.849 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.198885068 2024-08-19T14:23:07.849 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.849 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.849 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:23:07.849 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:22:24.582521+0000", 2024-08-19T14:23:07.849 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0574999999999997e-05 2024-08-19T14:23:07.849 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:07.849 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:07.849 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:23:07.849 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:22:24.582553+0000", 2024-08-19T14:23:07.850 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1995000000000001e-05 2024-08-19T14:23:07.850 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:07.850 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:23:07.850 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:07.850 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:07.850 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:23:07.850 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:23:07.850 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is set 2024-08-19T14:23:07.874 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:23:07.898 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:07.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:07.899 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:23:08.000 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:23:08.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:07 smithi032 ceph-mon[20004]: pgmap v882: 47 pgs: 47 active+clean; 4.9 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 691 op/s; 1.8 MiB/s, 29 objects/s recovering 2024-08-19T14:23:08.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:07 smithi032 ceph-mon[20004]: osdmap e387: 12 total, 12 up, 9 in 2024-08-19T14:23:08.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:07 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/842154956' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:23:08.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:07 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1601614307' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:23:08.028 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:08.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:08.030 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:23:08.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:07 smithi031 ceph-mon[16336]: pgmap v882: 47 pgs: 47 active+clean; 4.9 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 691 op/s; 1.8 MiB/s, 29 objects/s recovering 2024-08-19T14:23:08.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:07 smithi031 ceph-mon[16336]: osdmap e387: 12 total, 12 up, 9 in 2024-08-19T14:23:08.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:07 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/842154956' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:23:08.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:07 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1601614307' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:23:08.131 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:23:08.174 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:08.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:08.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:23:08.276 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:23:08.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:07 smithi045 ceph-mon[20143]: pgmap v882: 47 pgs: 47 active+clean; 4.9 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 691 op/s; 1.8 MiB/s, 29 objects/s recovering 2024-08-19T14:23:08.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:07 smithi045 ceph-mon[20143]: osdmap e387: 12 total, 12 up, 9 in 2024-08-19T14:23:08.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:07 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/842154956' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:23:08.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:07 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1601614307' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:23:08.355 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:08.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:08.357 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:23:08.457 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:23:08.486 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:08.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:08.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:23:08.588 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:23:08.626 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:08.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:08.628 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:23:08.729 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:23:08.747 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-08-19T14:23:08.762 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:08.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:08.763 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:23:08.864 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:23:08.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:08.893 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:08.893 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:23:08.902 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:08 smithi031 ceph-mon[16336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:23:08.902 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:08 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/842154956' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:23:08.902 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:08 smithi031 ceph-mon[16336]: osdmap e388: 12 total, 12 up, 9 in 2024-08-19T14:23:08.994 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:23:09.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:08 smithi032 ceph-mon[20004]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:23:09.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:08 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/842154956' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:23:09.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:08 smithi032 ceph-mon[20004]: osdmap e388: 12 total, 12 up, 9 in 2024-08-19T14:23:09.016 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:09.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:09.017 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:23:09.118 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:23:09.136 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:23:09.152 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:09.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:09.153 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:23:09.254 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:23:09.288 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:09.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:09.289 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:23:09.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:08 smithi045 ceph-mon[20143]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:23:09.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:08 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/842154956' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:23:09.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:08 smithi045 ceph-mon[20143]: osdmap e388: 12 total, 12 up, 9 in 2024-08-19T14:23:09.365 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-19T14:23:09.390 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:23:09.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:09.440 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:09.440 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:23:09.541 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:23:09.594 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:09.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:09.595 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:23:09.696 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:23:09.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:09.731 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:09.731 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:23:09.739 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:23:09.739 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-08-19T14:23:09.739 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:09.739 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.25333.0:189065 4.1 e388/335 4:8f0eace9:::benchmark_data_smithi045_71562_object57890:head v 388'11925, mlcod=388'11917)", 2024-08-19T14:23:09.739 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:23:09.724813+0000", 2024-08-19T14:23:09.739 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.011199929000000001, 2024-08-19T14:23:09.739 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011228597, 2024-08-19T14:23:09.739 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:23:09.740 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:23:09.740 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-08-19T14:23:09.740 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:23:09.740 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:09.740 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:23:09.740 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:09.724813+0000", 2024-08-19T14:23:09.740 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:09.740 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:09.740 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:09.740 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:23:09.740 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:09.724813+0000", 2024-08-19T14:23:09.740 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:09.741 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:09.741 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:09.741 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:23:09.741 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:09.724814+0000", 2024-08-19T14:23:09.741 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4099999999999998e-07 2024-08-19T14:23:09.741 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:09.741 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:09.741 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:23:09.742 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:09.724820+0000", 2024-08-19T14:23:09.742 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5799999999999999e-06 2024-08-19T14:23:09.742 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:09.742 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:09.742 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:23:09.742 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:09.724834+0000", 2024-08-19T14:23:09.742 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3913999999999999e-05 2024-08-19T14:23:09.742 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:09.742 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:09.742 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:23:09.742 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:09.724840+0000", 2024-08-19T14:23:09.742 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1670000000000004e-06 2024-08-19T14:23:09.743 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:09.743 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:09.743 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:23:09.743 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:09.732908+0000", 2024-08-19T14:23:09.743 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0080685870000000003 2024-08-19T14:23:09.743 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:09.743 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:09.743 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:23:09.743 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:09.732928+0000", 2024-08-19T14:23:09.743 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9956e-05 2024-08-19T14:23:09.743 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:09.743 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:23:09.743 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:09.744 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:09.744 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:09.744 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.25333.0:189070 4.2 e388/335 4:4aa97868:::benchmark_data_smithi045_71562_object57895:head v 388'11900, mlcod=388'11898)", 2024-08-19T14:23:09.744 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:23:09.727468+0000", 2024-08-19T14:23:09.744 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.0085452570000000005, 2024-08-19T14:23:09.744 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0086383529999999997, 2024-08-19T14:23:09.744 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:23:09.744 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:23:09.744 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-08-19T14:23:09.745 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:23:09.745 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:09.745 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:23:09.745 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:09.727468+0000", 2024-08-19T14:23:09.745 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:09.745 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:09.745 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:09.745 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:23:09.745 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:09.727468+0000", 2024-08-19T14:23:09.745 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:09.745 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:09.746 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:09.746 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:23:09.746 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:09.727469+0000", 2024-08-19T14:23:09.746 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3600000000000002e-07 2024-08-19T14:23:09.746 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:09.746 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:09.746 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:23:09.747 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:09.727477+0000", 2024-08-19T14:23:09.747 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.0739999999999993e-06 2024-08-19T14:23:09.747 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:09.747 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:09.747 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:23:09.747 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:09.727496+0000", 2024-08-19T14:23:09.747 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8749e-05 2024-08-19T14:23:09.747 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:09.747 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:09.747 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:23:09.747 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:09.727504+0000", 2024-08-19T14:23:09.747 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6549999999999993e-06 2024-08-19T14:23:09.747 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:09.748 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:09.748 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:23:09.748 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:09.727583+0000", 2024-08-19T14:23:09.748 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.8559000000000005e-05 2024-08-19T14:23:09.748 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:09.748 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:09.748 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:23:09.748 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:09.727606+0000", 2024-08-19T14:23:09.748 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2877000000000001e-05 2024-08-19T14:23:09.748 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:09.748 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:23:09.749 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:09.749 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:09.749 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:09.749 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.25333.0:189066 4.a e388/335 4:587ff24e:::benchmark_data_smithi045_71562_object57891:head v 388'11926, mlcod=388'11923)", 2024-08-19T14:23:09.749 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:23:09.729459+0000", 2024-08-19T14:23:09.749 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.0065542919999999998, 2024-08-19T14:23:09.749 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0067052459999999998, 2024-08-19T14:23:09.749 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:23:09.749 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:23:09.749 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-08-19T14:23:09.749 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:23:09.749 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:09.749 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:23:09.750 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:09.729459+0000", 2024-08-19T14:23:09.750 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:09.750 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:09.750 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:09.750 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:23:09.750 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:09.729459+0000", 2024-08-19T14:23:09.750 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:09.750 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:09.750 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:09.750 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:23:09.750 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:09.729460+0000", 2024-08-19T14:23:09.750 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.9900000000000009e-07 2024-08-19T14:23:09.751 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:09.751 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:09.751 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:23:09.751 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:09.729467+0000", 2024-08-19T14:23:09.751 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1350000000000003e-06 2024-08-19T14:23:09.751 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:09.751 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:09.751 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:23:09.751 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:09.729480+0000", 2024-08-19T14:23:09.751 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3124e-05 2024-08-19T14:23:09.751 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:09.751 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:09.751 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:23:09.752 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:09.729488+0000", 2024-08-19T14:23:09.752 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6210000000000004e-06 2024-08-19T14:23:09.752 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:09.752 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:09.752 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:23:09.752 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:09.729654+0000", 2024-08-19T14:23:09.752 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00016593200000000001 2024-08-19T14:23:09.752 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:09.752 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:09.752 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:23:09.752 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:09.729703+0000", 2024-08-19T14:23:09.752 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9224000000000002e-05 2024-08-19T14:23:09.752 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:09.753 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:23:09.753 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:09.753 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:09.753 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:09.753 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.25333.0:189073 4.2 e388/335 4:450b3fb6:::benchmark_data_smithi045_71562_object57898:head v 388'11901, mlcod=388'11898)", 2024-08-19T14:23:09.753 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:23:09.734528+0000", 2024-08-19T14:23:09.753 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.0014850060000000001, 2024-08-19T14:23:09.753 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0016913060000000001, 2024-08-19T14:23:09.753 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:23:09.753 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:23:09.753 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-08-19T14:23:09.753 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:23:09.753 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:09.753 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:23:09.754 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:09.734528+0000", 2024-08-19T14:23:09.754 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:09.754 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:09.754 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:09.754 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:23:09.754 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:09.734528+0000", 2024-08-19T14:23:09.754 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:09.754 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:09.754 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:09.754 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:23:09.754 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:09.734529+0000", 2024-08-19T14:23:09.754 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.7199999999999997e-07 2024-08-19T14:23:09.754 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:09.755 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:09.755 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:23:09.755 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:09.734536+0000", 2024-08-19T14:23:09.755 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8399999999999997e-06 2024-08-19T14:23:09.755 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:09.755 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:09.755 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:23:09.755 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:09.734548+0000", 2024-08-19T14:23:09.755 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2103000000000001e-05 2024-08-19T14:23:09.755 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:09.755 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:09.755 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:23:09.755 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:09.734555+0000", 2024-08-19T14:23:09.756 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9539999999999998e-06 2024-08-19T14:23:09.756 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:09.756 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:09.756 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:23:09.756 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:09.734619+0000", 2024-08-19T14:23:09.756 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4288999999999994e-05 2024-08-19T14:23:09.756 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:09.756 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:09.756 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:23:09.756 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:09.734638+0000", 2024-08-19T14:23:09.756 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8885999999999998e-05 2024-08-19T14:23:09.756 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:09.756 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:23:09.757 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:09.757 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:09.757 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-08-19T14:23:09.757 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 4 2024-08-19T14:23:09.757 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:23:09.795 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:23:09.831 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:23:09.855 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:09.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:09.856 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:23:09.957 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:23:09.985 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:09.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:09.986 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:23:10.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:09 smithi032 ceph-mon[20004]: pgmap v885: 47 pgs: 47 active+clean; 4.7 GiB data, 18 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 872 op/s; 1.8 MiB/s, 29 objects/s recovering 2024-08-19T14:23:10.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:09 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:23:10.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:09 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:23:10.087 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:23:10.108 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:09 smithi031 ceph-mon[16336]: pgmap v885: 47 pgs: 47 active+clean; 4.7 GiB data, 18 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 872 op/s; 1.8 MiB/s, 29 objects/s recovering 2024-08-19T14:23:10.108 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:09 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:23:10.108 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:09 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:23:10.111 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:10.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:10.112 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:23:10.213 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:23:10.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:10.248 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:10.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:23:10.279 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:23:10.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:09 smithi045 ceph-mon[20143]: pgmap v885: 47 pgs: 47 active+clean; 4.7 GiB data, 18 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 872 op/s; 1.8 MiB/s, 29 objects/s recovering 2024-08-19T14:23:10.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:09 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:23:10.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:09 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:23:10.349 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:23:10.375 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:10.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:10.376 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:23:10.477 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:23:10.502 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:10.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:10.504 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:23:10.605 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:23:10.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:10.627 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:10.627 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:23:10.681 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:23:10.693 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:23:10.694 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-08-19T14:23:10.729 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:23:10.747 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-08-19T14:23:10.758 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:10.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:10.759 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:23:10.860 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:23:10.887 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:10.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:10.888 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:23:10.989 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:23:11.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:10 smithi032 ceph-mon[20004]: osdmap e389: 12 total, 12 up, 9 in 2024-08-19T14:23:11.014 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:11.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:11.015 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:23:11.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:10 smithi031 ceph-mon[16336]: osdmap e389: 12 total, 12 up, 9 in 2024-08-19T14:23:11.083 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-19T14:23:11.116 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:23:11.120 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:23:11.142 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:11.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:11.143 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:23:11.244 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:23:11.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:11.282 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:11.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:23:11.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:10 smithi045 ceph-mon[20143]: osdmap e389: 12 total, 12 up, 9 in 2024-08-19T14:23:11.383 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:23:11.406 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:11.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:11.407 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:23:11.509 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:23:11.537 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:11.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:11.538 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:23:11.544 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:23:11.545 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:23:11.581 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:23:11.582 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:23:11.617 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:23:11.617 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:23:11.639 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:23:11.658 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:23:11.658 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:23:11.678 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:11.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:11.679 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:23:11.690 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:23:11.690 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:23:11.696 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:23:11.696 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-08-19T14:23:11.696 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-08-19T14:23:11.696 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-08-19T14:23:11.697 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:23:11.726 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:23:11.727 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:23:11.759 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:23:11.759 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:23:11.780 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:23:11.794 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:23:11.795 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:23:11.803 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:11.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:11.804 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:23:11.825 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:23:11.825 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:23:11.859 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:23:11.860 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:23:11.895 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:23:11.895 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:23:11.895 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:11 smithi031 ceph-mon[16336]: pgmap v887: 47 pgs: 47 active+clean; 4.6 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 1.07k op/s 2024-08-19T14:23:11.895 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:11 smithi031 ceph-mon[16336]: from='client.25649 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:23:11.895 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:11 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:23:11.896 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:11 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:23:11.896 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:11 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3440124734' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:23:11.904 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:23:11.929 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:23:11.930 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:23:11.934 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:11.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:11.935 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:23:12.035 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:23:12.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:12.072 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:12.073 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:23:12.095 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":887,"stamp":"2024-08-19T14:23:09.928920+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4948820536,"num_objects":75507,"num_object_clones":0,"num_object_copies":226521,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75507,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":186841,"num_write_kb":8395009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4140,"num_bytes_recovered":271319040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":165050,"ondisk_log_size":165050,"up":141,"acting":141,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":142,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":18872016,"kb_used_data":14824952,"kb_used_omap":14,"kb_used_meta":4046961,"kb_avail":824834352,"statfs":{"total":863955320832,"available":844630376448,"internally_reserved":0,"allocated":15180750848,"data_stored":15164730912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14353,"internal_metadata":4144089071},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,3,0,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-290979840,"num_objects":-4440,"num_object_clones":0,"num_object_copies":-13320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4440,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.129810"},"pg_stats":[{"pgid":"4.5","version":"386'11708","reported_seq":11906,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-08-19T14:23:05.939724+0000","last_change":"2024-08-19T14:22:14.211513+0000","last_active":"2024-08-19T14:23:05.939724+0000","last_peered":"2024-08-19T14:23:05.939724+0000","last_clean":"2024-08-19T14:23:05.939724+0000","last_became_active":"2024-08-19T14:20:29.267549+0000","last_became_peered":"2024-08-19T14:20:29.267549+0000","last_unstale":"2024-08-19T14:23:05.939724+0000","last_undegraded":"2024-08-19T14:23:05.939724+0000","last_fullsized":"2024-08-19T14:23:05.939724+0000","mapping_epoch":335,"log_start":"341'1700","ondisk_log_start":"341'1700","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"375'8747","last_scrub_stamp":"2024-08-19T14:22:14.211361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:22:14.211361+0000","objects_scrubbed":7717,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:26.018227+0000","scrub_duration":1909,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":307494912,"num_objects":4692,"num_object_clones":0,"num_object_copies":14076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4692,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11708,"num_write_kb":524800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,0],"acting":[11,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"70'139","reported_seq":993,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-08-19T14:23:07.828482+0000","last_change":"2024-08-19T14:19:58.072227+0000","last_active":"2024-08-19T14:23:07.828482+0000","last_peered":"2024-08-19T14:23:07.828482+0000","last_clean":"2024-08-19T14:23:07.828482+0000","last_became_active":"2024-08-19T14:18:13.807133+0000","last_became_peered":"2024-08-19T14:18:13.807133+0000","last_unstale":"2024-08-19T14:23:07.828482+0000","last_undegraded":"2024-08-19T14:23:07.828482+0000","last_fullsized":"2024-08-19T14:23:07.828482+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:19:58.072083+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:19:58.072083+0000","last_clean_scrub_stamp":"2024-08-19T14:19:58.072083+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:08.833284+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,9],"acting":[7,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"388'11889","reported_seq":20298,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-08-19T14:23:08.422722+0000","last_change":"2024-08-19T14:23:01.772236+0000","last_active":"2024-08-19T14:23:08.422722+0000","last_peered":"2024-08-19T14:23:08.422722+0000","last_clean":"2024-08-19T14:23:08.422722+0000","last_became_active":"2024-08-19T14:23:01.741503+0000","last_became_peered":"2024-08-19T14:23:01.741503+0000","last_unstale":"2024-08-19T14:23:08.422722+0000","last_undegraded":"2024-08-19T14:23:08.422722+0000","last_fullsized":"2024-08-19T14:23:08.422722+0000","mapping_epoch":384,"log_start":"342'1800","ondisk_log_start":"342'1800","created":335,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:13.215866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":313851904,"num_objects":4789,"num_object_clones":0,"num_object_copies":14367,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11889,"num_write_kb":533696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4140,"num_bytes_recovered":271319040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,7],"acting":[9,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.3","version":"70'139","reported_seq":889,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-19T14:23:06.828098+0000","last_change":"2024-08-19T14:20:03.714917+0000","last_active":"2024-08-19T14:23:06.828098+0000","last_peered":"2024-08-19T14:23:06.828098+0000","last_clean":"2024-08-19T14:23:06.828098+0000","last_became_active":"2024-08-19T14:18:13.804721+0000","last_became_peered":"2024-08-19T14:18:13.804721+0000","last_unstale":"2024-08-19T14:23:06.828098+0000","last_undegraded":"2024-08-19T14:23:06.828098+0000","last_fullsized":"2024-08-19T14:23:06.828098+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:03.714818+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:03.714818+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"387'11746","reported_seq":11861,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-19T14:23:07.611500+0000","last_change":"2024-08-19T14:20:29.268110+0000","last_active":"2024-08-19T14:23:07.611500+0000","last_peered":"2024-08-19T14:23:07.611500+0000","last_clean":"2024-08-19T14:23:07.611500+0000","last_became_active":"2024-08-19T14:20:29.267811+0000","last_became_peered":"2024-08-19T14:20:29.267811+0000","last_unstale":"2024-08-19T14:23:07.611500+0000","last_undegraded":"2024-08-19T14:23:07.611500+0000","last_fullsized":"2024-08-19T14:23:07.611500+0000","mapping_epoch":335,"log_start":"342'1700","ondisk_log_start":"342'1700","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":308019200,"num_objects":4700,"num_object_clones":0,"num_object_copies":14100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4700,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11746,"num_write_kb":526272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"70'139","reported_seq":840,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-08-19T14:23:07.828329+0000","last_change":"2024-08-19T14:20:50.742676+0000","last_active":"2024-08-19T14:23:07.828329+0000","last_peered":"2024-08-19T14:23:07.828329+0000","last_clean":"2024-08-19T14:23:07.828329+0000","last_became_active":"2024-08-19T14:20:50.742354+0000","last_became_peered":"2024-08-19T14:20:50.742354+0000","last_unstale":"2024-08-19T14:23:07.828329+0000","last_undegraded":"2024-08-19T14:23:07.828329+0000","last_fullsized":"2024-08-19T14:23:07.828329+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:19:59.302931+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:19:59.302931+0000","last_clean_scrub_stamp":"2024-08-19T14:19:59.302931+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:08.833315+0000","scrub_duration":114,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"387'11726","reported_seq":11841,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-19T14:23:07.677848+0000","last_change":"2024-08-19T14:20:29.267640+0000","last_active":"2024-08-19T14:23:07.677848+0000","last_peered":"2024-08-19T14:23:07.677848+0000","last_clean":"2024-08-19T14:23:07.677848+0000","last_became_active":"2024-08-19T14:20:29.267331+0000","last_became_peered":"2024-08-19T14:20:29.267331+0000","last_unstale":"2024-08-19T14:23:07.677848+0000","last_undegraded":"2024-08-19T14:23:07.677848+0000","last_fullsized":"2024-08-19T14:23:07.677848+0000","mapping_epoch":335,"log_start":"342'1700","ondisk_log_start":"342'1700","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":305397760,"num_objects":4660,"num_object_clones":0,"num_object_copies":13980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4660,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11726,"num_write_kb":524352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"70'139","reported_seq":888,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-08-19T14:23:07.826679+0000","last_change":"2024-08-19T14:20:09.507422+0000","last_active":"2024-08-19T14:23:07.826679+0000","last_peered":"2024-08-19T14:23:07.826679+0000","last_clean":"2024-08-19T14:23:07.826679+0000","last_became_active":"2024-08-19T14:18:13.808505+0000","last_became_peered":"2024-08-19T14:18:13.808505+0000","last_unstale":"2024-08-19T14:23:07.826679+0000","last_undegraded":"2024-08-19T14:23:07.826679+0000","last_fullsized":"2024-08-19T14:23:07.826679+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:09.507256+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:09.507256+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,11],"acting":[9,0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"388'11823","reported_seq":11940,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-08-19T14:23:08.436393+0000","last_change":"2024-08-19T14:20:29.268766+0000","last_active":"2024-08-19T14:23:08.436393+0000","last_peered":"2024-08-19T14:23:08.436393+0000","last_clean":"2024-08-19T14:23:08.436393+0000","last_became_active":"2024-08-19T14:20:29.268443+0000","last_became_peered":"2024-08-19T14:20:29.268443+0000","last_unstale":"2024-08-19T14:23:08.436393+0000","last_undegraded":"2024-08-19T14:23:08.436393+0000","last_fullsized":"2024-08-19T14:23:08.436393+0000","mapping_epoch":335,"log_start":"342'1800","ondisk_log_start":"342'1800","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:13.215834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":307298304,"num_objects":4689,"num_object_clones":0,"num_object_copies":14067,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4689,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11823,"num_write_kb":528384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,11],"acting":[9,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.7","version":"70'139","reported_seq":540,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-08-19T14:23:07.830439+0000","last_change":"2024-08-19T14:21:46.057764+0000","last_active":"2024-08-19T14:23:07.830439+0000","last_peered":"2024-08-19T14:23:07.830439+0000","last_clean":"2024-08-19T14:23:07.830439+0000","last_became_active":"2024-08-19T14:20:28.204384+0000","last_became_peered":"2024-08-19T14:20:28.204384+0000","last_unstale":"2024-08-19T14:23:07.830439+0000","last_undegraded":"2024-08-19T14:23:07.830439+0000","last_fullsized":"2024-08-19T14:23:07.830439+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:46.057624+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:21:46.057624+0000","last_clean_scrub_stamp":"2024-08-19T14:21:46.057624+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"386'11590","reported_seq":11848,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-08-19T14:23:05.899483+0000","last_change":"2024-08-19T14:22:07.840154+0000","last_active":"2024-08-19T14:23:05.899483+0000","last_peered":"2024-08-19T14:23:05.899483+0000","last_clean":"2024-08-19T14:23:05.899483+0000","last_became_active":"2024-08-19T14:20:29.267379+0000","last_became_peered":"2024-08-19T14:20:29.267379+0000","last_unstale":"2024-08-19T14:23:05.899483+0000","last_undegraded":"2024-08-19T14:23:05.899483+0000","last_fullsized":"2024-08-19T14:23:05.899483+0000","mapping_epoch":335,"log_start":"341'1500","ondisk_log_start":"341'1500","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"373'8338","last_scrub_stamp":"2024-08-19T14:22:07.840021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:22:07.840021+0000","objects_scrubbed":8073,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:13.679336+0000","scrub_duration":2409,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":311951360,"num_objects":4760,"num_object_clones":0,"num_object_copies":14280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4760,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11590,"num_write_kb":523200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,1],"acting":[11,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.6","version":"70'139","reported_seq":892,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-08-19T14:23:07.826954+0000","last_change":"2024-08-19T14:19:55.512078+0000","last_active":"2024-08-19T14:23:07.826954+0000","last_peered":"2024-08-19T14:23:07.826954+0000","last_clean":"2024-08-19T14:23:07.826954+0000","last_became_active":"2024-08-19T14:18:26.874902+0000","last_became_peered":"2024-08-19T14:18:26.874902+0000","last_unstale":"2024-08-19T14:23:07.826954+0000","last_undegraded":"2024-08-19T14:23:07.826954+0000","last_fullsized":"2024-08-19T14:23:07.826954+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:19:55.511974+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:27.848870+0000","last_clean_scrub_stamp":"2024-08-19T14:19:55.511974+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,4],"acting":[9,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"388'11719","reported_seq":11924,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-08-19T14:23:09.640080+0000","last_change":"2024-08-19T14:22:08.536948+0000","last_active":"2024-08-19T14:23:09.640080+0000","last_peered":"2024-08-19T14:23:09.640080+0000","last_clean":"2024-08-19T14:23:09.640080+0000","last_became_active":"2024-08-19T14:20:29.267360+0000","last_became_peered":"2024-08-19T14:20:29.267360+0000","last_unstale":"2024-08-19T14:23:09.640080+0000","last_undegraded":"2024-08-19T14:23:09.640080+0000","last_fullsized":"2024-08-19T14:23:09.640080+0000","mapping_epoch":335,"log_start":"342'1700","ondisk_log_start":"342'1700","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"374'8331","last_scrub_stamp":"2024-08-19T14:22:08.536620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:22:08.536620+0000","objects_scrubbed":7995,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:19.976024+0000","scrub_duration":2478,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":296288256,"num_objects":4521,"num_object_clones":0,"num_object_copies":13563,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4521,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11719,"num_write_kb":519680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"70'139","reported_seq":710,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-08-19T14:23:05.805057+0000","last_change":"2024-08-19T14:22:01.354585+0000","last_active":"2024-08-19T14:23:05.805057+0000","last_peered":"2024-08-19T14:23:05.805057+0000","last_clean":"2024-08-19T14:23:05.805057+0000","last_became_active":"2024-08-19T14:18:26.881435+0000","last_became_peered":"2024-08-19T14:18:26.881435+0000","last_unstale":"2024-08-19T14:23:05.805057+0000","last_undegraded":"2024-08-19T14:23:05.805057+0000","last_fullsized":"2024-08-19T14:23:05.805057+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:22:01.354343+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:22:01.354343+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:30.081548+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,4],"acting":[11,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.1","version":"386'11726","reported_seq":11920,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-08-19T14:23:05.939747+0000","last_change":"2024-08-19T14:22:20.917940+0000","last_active":"2024-08-19T14:23:05.939747+0000","last_peered":"2024-08-19T14:23:05.939747+0000","last_clean":"2024-08-19T14:23:05.939747+0000","last_became_active":"2024-08-19T14:20:29.268465+0000","last_became_peered":"2024-08-19T14:20:29.268465+0000","last_unstale":"2024-08-19T14:23:05.939747+0000","last_undegraded":"2024-08-19T14:23:05.939747+0000","last_fullsized":"2024-08-19T14:23:05.939747+0000","mapping_epoch":335,"log_start":"341'1700","ondisk_log_start":"341'1700","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"377'9214","last_scrub_stamp":"2024-08-19T14:22:20.917842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:22:20.917842+0000","objects_scrubbed":7332,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:39.609654+0000","scrub_duration":1625,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":312999936,"num_objects":4776,"num_object_clones":0,"num_object_copies":14328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11726,"num_write_kb":528064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.4","version":"70'139","reported_seq":942,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-08-19T14:23:05.804940+0000","last_change":"2024-08-19T14:22:46.380189+0000","last_active":"2024-08-19T14:23:05.804940+0000","last_peered":"2024-08-19T14:23:05.804940+0000","last_clean":"2024-08-19T14:23:05.804940+0000","last_became_active":"2024-08-19T14:18:13.806013+0000","last_became_peered":"2024-08-19T14:18:13.806013+0000","last_unstale":"2024-08-19T14:23:05.804940+0000","last_undegraded":"2024-08-19T14:23:05.804940+0000","last_fullsized":"2024-08-19T14:23:05.804940+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:22:46.380119+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:22:46.380119+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:49.643921+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.f","version":"387'11558","reported_seq":11673,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-19T14:23:07.699669+0000","last_change":"2024-08-19T14:20:29.212298+0000","last_active":"2024-08-19T14:23:07.699669+0000","last_peered":"2024-08-19T14:23:07.699669+0000","last_clean":"2024-08-19T14:23:07.699669+0000","last_became_active":"2024-08-19T14:20:29.212003+0000","last_became_peered":"2024-08-19T14:20:29.212003+0000","last_unstale":"2024-08-19T14:23:07.699669+0000","last_undegraded":"2024-08-19T14:23:07.699669+0000","last_fullsized":"2024-08-19T14:23:07.699669+0000","mapping_epoch":335,"log_start":"341'1500","ondisk_log_start":"341'1500","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":310902784,"num_objects":4744,"num_object_clones":0,"num_object_copies":14232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11558,"num_write_kb":521664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,11],"acting":[1,3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"70'139","reported_seq":900,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-08-19T14:23:07.826822+0000","last_change":"2024-08-19T14:20:12.570453+0000","last_active":"2024-08-19T14:23:07.826822+0000","last_peered":"2024-08-19T14:23:07.826822+0000","last_clean":"2024-08-19T14:23:07.826822+0000","last_became_active":"2024-08-19T14:18:26.875006+0000","last_became_peered":"2024-08-19T14:18:26.875006+0000","last_unstale":"2024-08-19T14:23:07.826822+0000","last_undegraded":"2024-08-19T14:23:07.826822+0000","last_fullsized":"2024-08-19T14:23:07.826822+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:12.570352+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:28.864870+0000","last_clean_scrub_stamp":"2024-08-19T14:20:12.570352+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,0],"acting":[9,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.c","version":"388'11680","reported_seq":11884,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-08-19T14:23:08.234626+0000","last_change":"2024-08-19T14:22:07.487434+0000","last_active":"2024-08-19T14:23:08.234626+0000","last_peered":"2024-08-19T14:23:08.234626+0000","last_clean":"2024-08-19T14:23:08.234626+0000","last_became_active":"2024-08-19T14:20:29.266373+0000","last_became_peered":"2024-08-19T14:20:29.266373+0000","last_unstale":"2024-08-19T14:23:08.234626+0000","last_undegraded":"2024-08-19T14:23:08.234626+0000","last_fullsized":"2024-08-19T14:23:08.234626+0000","mapping_epoch":335,"log_start":"341'1600","ondisk_log_start":"341'1600","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"373'8336","last_scrub_stamp":"2024-08-19T14:22:07.487270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:22:07.487270+0000","objects_scrubbed":8079,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:14.053549+0000","scrub_duration":1972,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":306839552,"num_objects":4682,"num_object_clones":0,"num_object_copies":14046,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11680,"num_write_kb":523584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,8],"acting":[4,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"70'139","reported_seq":1015,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-08-19T14:23:07.828398+0000","last_change":"2024-08-19T14:19:56.106047+0000","last_active":"2024-08-19T14:23:07.828398+0000","last_peered":"2024-08-19T14:23:07.828398+0000","last_clean":"2024-08-19T14:23:07.828398+0000","last_became_active":"2024-08-19T14:18:26.887657+0000","last_became_peered":"2024-08-19T14:18:26.887657+0000","last_unstale":"2024-08-19T14:23:07.828398+0000","last_undegraded":"2024-08-19T14:23:07.828398+0000","last_fullsized":"2024-08-19T14:23:07.828398+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:19:56.105880+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:19:56.105880+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:08.833373+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"388'11602","reported_seq":11808,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-08-19T14:23:08.222335+0000","last_change":"2024-08-19T14:22:08.529439+0000","last_active":"2024-08-19T14:23:08.222335+0000","last_peered":"2024-08-19T14:23:08.222335+0000","last_clean":"2024-08-19T14:23:08.222335+0000","last_became_active":"2024-08-19T14:20:29.208599+0000","last_became_peered":"2024-08-19T14:20:29.208599+0000","last_unstale":"2024-08-19T14:23:08.222335+0000","last_undegraded":"2024-08-19T14:23:08.222335+0000","last_fullsized":"2024-08-19T14:23:08.222335+0000","mapping_epoch":335,"log_start":"341'1600","ondisk_log_start":"341'1600","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"374'8320","last_scrub_stamp":"2024-08-19T14:22:08.529109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:22:08.529109+0000","objects_scrubbed":7983,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:30.745931+0000","scrub_duration":1962,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":304545816,"num_objects":4648,"num_object_clones":0,"num_object_copies":13944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4648,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11602,"num_write_kb":519937,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,1],"acting":[4,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"70'139","reported_seq":908,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-08-19T14:23:05.805952+0000","last_change":"2024-08-19T14:21:37.951800+0000","last_active":"2024-08-19T14:23:05.805952+0000","last_peered":"2024-08-19T14:23:05.805952+0000","last_clean":"2024-08-19T14:23:05.805952+0000","last_became_active":"2024-08-19T14:18:26.887277+0000","last_became_peered":"2024-08-19T14:18:26.887277+0000","last_unstale":"2024-08-19T14:23:05.805952+0000","last_undegraded":"2024-08-19T14:23:05.805952+0000","last_fullsized":"2024-08-19T14:23:05.805952+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:37.951694+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:21:37.951694+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,8],"acting":[5,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"388'11553","reported_seq":11754,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-08-19T14:23:08.244366+0000","last_change":"2024-08-19T14:22:13.656515+0000","last_active":"2024-08-19T14:23:08.244366+0000","last_peered":"2024-08-19T14:23:08.244366+0000","last_clean":"2024-08-19T14:23:08.244366+0000","last_became_active":"2024-08-19T14:20:29.208748+0000","last_became_peered":"2024-08-19T14:20:29.208748+0000","last_unstale":"2024-08-19T14:23:08.244366+0000","last_undegraded":"2024-08-19T14:23:08.244366+0000","last_fullsized":"2024-08-19T14:23:08.244366+0000","mapping_epoch":335,"log_start":"341'1500","ondisk_log_start":"341'1500","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"374'8571","last_scrub_stamp":"2024-08-19T14:22:13.656335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:22:13.656335+0000","objects_scrubbed":7643,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:30.767371+0000","scrub_duration":3043,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":300482560,"num_objects":4585,"num_object_clones":0,"num_object_copies":13755,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11553,"num_write_kb":516416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"70'139","reported_seq":910,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-08-19T14:23:05.805133+0000","last_change":"2024-08-19T14:22:02.413212+0000","last_active":"2024-08-19T14:23:05.805133+0000","last_peered":"2024-08-19T14:23:05.805133+0000","last_clean":"2024-08-19T14:23:05.805133+0000","last_became_active":"2024-08-19T14:18:26.875179+0000","last_became_peered":"2024-08-19T14:18:26.875179+0000","last_unstale":"2024-08-19T14:23:05.805133+0000","last_undegraded":"2024-08-19T14:23:05.805133+0000","last_fullsized":"2024-08-19T14:23:05.805133+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:22:02.413091+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:22:02.413091+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:23.082273+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.b","version":"386'11540","reported_seq":11728,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-08-19T14:23:06.276155+0000","last_change":"2024-08-19T14:21:39.089676+0000","last_active":"2024-08-19T14:23:06.276155+0000","last_peered":"2024-08-19T14:23:06.276155+0000","last_clean":"2024-08-19T14:23:06.276155+0000","last_became_active":"2024-08-19T14:20:29.207372+0000","last_became_peered":"2024-08-19T14:20:29.207372+0000","last_unstale":"2024-08-19T14:23:06.276155+0000","last_undegraded":"2024-08-19T14:23:06.276155+0000","last_fullsized":"2024-08-19T14:23:06.276155+0000","mapping_epoch":335,"log_start":"341'1500","ondisk_log_start":"341'1500","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"363'6615","last_scrub_stamp":"2024-08-19T14:21:39.089400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:21:39.089400+0000","objects_scrubbed":6552,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1907,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":314703872,"num_objects":4802,"num_object_clones":0,"num_object_copies":14406,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11540,"num_write_kb":522944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"70'139","reported_seq":568,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-08-19T14:23:07.830417+0000","last_change":"2024-08-19T14:21:44.434481+0000","last_active":"2024-08-19T14:23:07.830417+0000","last_peered":"2024-08-19T14:23:07.830417+0000","last_clean":"2024-08-19T14:23:07.830417+0000","last_became_active":"2024-08-19T14:18:26.884105+0000","last_became_peered":"2024-08-19T14:18:26.884105+0000","last_unstale":"2024-08-19T14:23:07.830417+0000","last_undegraded":"2024-08-19T14:23:07.830417+0000","last_fullsized":"2024-08-19T14:23:07.830417+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:44.434382+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:20:15.321945+0000","last_clean_scrub_stamp":"2024-08-19T14:21:44.434382+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:13.544241+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"387'11772","reported_seq":11887,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-19T14:23:07.678560+0000","last_change":"2024-08-19T14:20:29.269723+0000","last_active":"2024-08-19T14:23:07.678560+0000","last_peered":"2024-08-19T14:23:07.678560+0000","last_clean":"2024-08-19T14:23:07.678560+0000","last_became_active":"2024-08-19T14:20:29.269432+0000","last_became_peered":"2024-08-19T14:20:29.269432+0000","last_unstale":"2024-08-19T14:23:07.678560+0000","last_undegraded":"2024-08-19T14:23:07.678560+0000","last_fullsized":"2024-08-19T14:23:07.678560+0000","mapping_epoch":335,"log_start":"342'1700","ondisk_log_start":"342'1700","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":317456384,"num_objects":4844,"num_object_clones":0,"num_object_copies":14532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4844,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11772,"num_write_kb":531712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.f","version":"70'139","reported_seq":816,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-08-19T14:23:05.806402+0000","last_change":"2024-08-19T14:22:42.293240+0000","last_active":"2024-08-19T14:23:05.806402+0000","last_peered":"2024-08-19T14:23:05.806402+0000","last_clean":"2024-08-19T14:23:05.806402+0000","last_became_active":"2024-08-19T14:18:18.711708+0000","last_became_peered":"2024-08-19T14:18:18.711708+0000","last_unstale":"2024-08-19T14:23:05.806402+0000","last_undegraded":"2024-08-19T14:23:05.806402+0000","last_fullsized":"2024-08-19T14:23:05.806402+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:22:42.293171+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:22:42.293171+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:24:10.029244+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"386'11668","reported_seq":11860,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-08-19T14:23:05.830731+0000","last_change":"2024-08-19T14:21:43.974700+0000","last_active":"2024-08-19T14:23:05.830731+0000","last_peered":"2024-08-19T14:23:05.830731+0000","last_clean":"2024-08-19T14:23:05.830731+0000","last_became_active":"2024-08-19T14:20:29.211975+0000","last_became_peered":"2024-08-19T14:20:29.211975+0000","last_unstale":"2024-08-19T14:23:05.830731+0000","last_undegraded":"2024-08-19T14:23:05.830731+0000","last_fullsized":"2024-08-19T14:23:05.830731+0000","mapping_epoch":335,"log_start":"341'1600","ondisk_log_start":"341'1600","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"364'6991","last_scrub_stamp":"2024-08-19T14:21:43.974613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:21:43.974613+0000","objects_scrubbed":6934,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:07.463188+0000","scrub_duration":1682,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":313917440,"num_objects":4790,"num_object_clones":0,"num_object_copies":14370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4790,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11668,"num_write_kb":526656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,3],"acting":[11,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.c","version":"70'139","reported_seq":681,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-08-19T14:23:07.834171+0000","last_change":"2024-08-19T14:22:52.164697+0000","last_active":"2024-08-19T14:23:07.834171+0000","last_peered":"2024-08-19T14:23:07.834171+0000","last_clean":"2024-08-19T14:23:07.834171+0000","last_became_active":"2024-08-19T14:18:26.872942+0000","last_became_peered":"2024-08-19T14:18:26.872942+0000","last_unstale":"2024-08-19T14:23:07.834171+0000","last_undegraded":"2024-08-19T14:23:07.834171+0000","last_fullsized":"2024-08-19T14:23:07.834171+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:22:52.164586+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:22:52.164586+0000","last_clean_scrub_stamp":"2024-08-19T14:22:52.164586+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:58.654533+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,9],"acting":[3,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"386'11541","reported_seq":11741,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-08-19T14:23:05.932644+0000","last_change":"2024-08-19T14:22:12.682046+0000","last_active":"2024-08-19T14:23:05.932644+0000","last_peered":"2024-08-19T14:23:05.932644+0000","last_clean":"2024-08-19T14:23:05.932644+0000","last_became_active":"2024-08-19T14:20:29.267442+0000","last_became_peered":"2024-08-19T14:20:29.267442+0000","last_unstale":"2024-08-19T14:23:05.932644+0000","last_undegraded":"2024-08-19T14:23:05.932644+0000","last_fullsized":"2024-08-19T14:23:05.932644+0000","mapping_epoch":335,"log_start":"341'1500","ondisk_log_start":"341'1500","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"374'8663","last_scrub_stamp":"2024-08-19T14:22:12.681916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:22:12.681916+0000","objects_scrubbed":7729,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:34.638254+0000","scrub_duration":1366,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":316080128,"num_objects":4823,"num_object_clones":0,"num_object_copies":14469,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4823,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11541,"num_write_kb":523648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,9],"acting":[11,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.d","version":"70'139","reported_seq":588,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-08-19T14:23:07.828708+0000","last_change":"2024-08-19T14:21:07.155294+0000","last_active":"2024-08-19T14:23:07.828708+0000","last_peered":"2024-08-19T14:23:07.828708+0000","last_clean":"2024-08-19T14:23:07.828708+0000","last_became_active":"2024-08-19T14:18:26.882002+0000","last_became_peered":"2024-08-19T14:18:26.882002+0000","last_unstale":"2024-08-19T14:23:07.828708+0000","last_undegraded":"2024-08-19T14:23:07.828708+0000","last_fullsized":"2024-08-19T14:23:07.828708+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:07.155157+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:21:07.155157+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:08.833401+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"70'139","reported_seq":851,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-08-19T14:23:05.809751+0000","last_change":"2024-08-19T14:22:47.256611+0000","last_active":"2024-08-19T14:23:05.809751+0000","last_peered":"2024-08-19T14:23:05.809751+0000","last_clean":"2024-08-19T14:23:05.809751+0000","last_became_active":"2024-08-19T14:18:18.703594+0000","last_became_peered":"2024-08-19T14:18:18.703594+0000","last_unstale":"2024-08-19T14:23:05.809751+0000","last_undegraded":"2024-08-19T14:23:05.809751+0000","last_fullsized":"2024-08-19T14:23:05.809751+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:22:47.256536+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:22:47.256536+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:24:02.876164+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"70'139","reported_seq":897,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-19T14:23:06.827934+0000","last_change":"2024-08-19T14:20:07.692054+0000","last_active":"2024-08-19T14:23:06.827934+0000","last_peered":"2024-08-19T14:23:06.827934+0000","last_clean":"2024-08-19T14:23:06.827934+0000","last_became_active":"2024-08-19T14:18:18.698334+0000","last_became_peered":"2024-08-19T14:18:18.698334+0000","last_unstale":"2024-08-19T14:23:06.827934+0000","last_undegraded":"2024-08-19T14:23:06.827934+0000","last_fullsized":"2024-08-19T14:23:06.827934+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:07.691932+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:07.691932+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.10","version":"70'139","reported_seq":1006,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-08-19T14:23:05.805847+0000","last_change":"2024-08-19T14:22:11.260404+0000","last_active":"2024-08-19T14:23:05.805847+0000","last_peered":"2024-08-19T14:23:05.805847+0000","last_clean":"2024-08-19T14:23:05.805847+0000","last_became_active":"2024-08-19T14:18:18.704562+0000","last_became_peered":"2024-08-19T14:18:18.704562+0000","last_unstale":"2024-08-19T14:23:05.805847+0000","last_undegraded":"2024-08-19T14:23:05.805847+0000","last_fullsized":"2024-08-19T14:23:05.805847+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:22:11.260286+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:22:11.260286+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:36.552471+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,8],"acting":[5,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"70'139","reported_seq":555,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-08-19T14:23:05.811058+0000","last_change":"2024-08-19T14:22:21.104595+0000","last_active":"2024-08-19T14:23:05.811058+0000","last_peered":"2024-08-19T14:23:05.811058+0000","last_clean":"2024-08-19T14:23:05.811058+0000","last_became_active":"2024-08-19T14:18:18.697894+0000","last_became_peered":"2024-08-19T14:18:18.697894+0000","last_unstale":"2024-08-19T14:23:05.811058+0000","last_undegraded":"2024-08-19T14:23:05.811058+0000","last_fullsized":"2024-08-19T14:23:05.811058+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:22:21.104491+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:21:16.071703+0000","last_clean_scrub_stamp":"2024-08-19T14:22:21.104491+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:45.856425+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"70'139","reported_seq":525,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-19T14:23:06.823888+0000","last_change":"2024-08-19T14:20:28.205212+0000","last_active":"2024-08-19T14:23:06.823888+0000","last_peered":"2024-08-19T14:23:06.823888+0000","last_clean":"2024-08-19T14:23:06.823888+0000","last_became_active":"2024-08-19T14:20:28.204724+0000","last_became_peered":"2024-08-19T14:20:28.204724+0000","last_unstale":"2024-08-19T14:23:06.823888+0000","last_undegraded":"2024-08-19T14:23:06.823888+0000","last_fullsized":"2024-08-19T14:23:06.823888+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:08.680021+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:08.680021+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"70'139","reported_seq":649,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-08-19T14:23:01.783503+0000","last_change":"2024-08-19T14:22:02.722505+0000","last_active":"2024-08-19T14:23:01.783503+0000","last_peered":"2024-08-19T14:23:01.783503+0000","last_clean":"2024-08-19T14:23:01.783503+0000","last_became_active":"2024-08-19T14:18:31.901556+0000","last_became_peered":"2024-08-19T14:18:31.901556+0000","last_unstale":"2024-08-19T14:23:01.783503+0000","last_undegraded":"2024-08-19T14:23:01.783503+0000","last_fullsized":"2024-08-19T14:23:01.783503+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:22:02.722345+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:22:02.722345+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:13.381470+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,0],"acting":[8,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.14","version":"70'139","reported_seq":657,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-08-19T14:23:07.828641+0000","last_change":"2024-08-19T14:19:55.130518+0000","last_active":"2024-08-19T14:23:07.828641+0000","last_peered":"2024-08-19T14:23:07.828641+0000","last_clean":"2024-08-19T14:23:07.828641+0000","last_became_active":"2024-08-19T14:18:18.703926+0000","last_became_peered":"2024-08-19T14:18:18.703926+0000","last_unstale":"2024-08-19T14:23:07.828641+0000","last_undegraded":"2024-08-19T14:23:07.828641+0000","last_fullsized":"2024-08-19T14:23:07.828641+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:19:55.130420+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:19:55.130420+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:08.833249+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,1],"acting":[7,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.15","version":"70'139","reported_seq":543,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-08-19T14:23:07.828643+0000","last_change":"2024-08-19T14:20:09.181046+0000","last_active":"2024-08-19T14:23:07.828643+0000","last_peered":"2024-08-19T14:23:07.828643+0000","last_clean":"2024-08-19T14:23:07.828643+0000","last_became_active":"2024-08-19T14:18:31.912496+0000","last_became_peered":"2024-08-19T14:18:31.912496+0000","last_unstale":"2024-08-19T14:23:07.828643+0000","last_undegraded":"2024-08-19T14:23:07.828643+0000","last_fullsized":"2024-08-19T14:23:07.828643+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:09.180923+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:20:09.180923+0000","last_clean_scrub_stamp":"2024-08-19T14:20:09.180923+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:08.833343+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"70'139","reported_seq":529,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-08-19T14:23:07.829895+0000","last_change":"2024-08-19T14:21:40.430835+0000","last_active":"2024-08-19T14:23:07.829895+0000","last_peered":"2024-08-19T14:23:07.829895+0000","last_clean":"2024-08-19T14:23:07.829895+0000","last_became_active":"2024-08-19T14:18:31.901229+0000","last_became_peered":"2024-08-19T14:18:31.901229+0000","last_unstale":"2024-08-19T14:23:07.829895+0000","last_undegraded":"2024-08-19T14:23:07.829895+0000","last_fullsized":"2024-08-19T14:23:07.829895+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:40.430714+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:21:40.430714+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:10.893871+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1b","version":"70'139","reported_seq":875,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-08-19T14:23:01.783396+0000","last_change":"2024-08-19T14:21:48.693864+0000","last_active":"2024-08-19T14:23:01.783396+0000","last_peered":"2024-08-19T14:23:01.783396+0000","last_clean":"2024-08-19T14:23:01.783396+0000","last_became_active":"2024-08-19T14:18:31.901152+0000","last_became_peered":"2024-08-19T14:18:31.901152+0000","last_unstale":"2024-08-19T14:23:01.783396+0000","last_undegraded":"2024-08-19T14:23:01.783396+0000","last_fullsized":"2024-08-19T14:23:01.783396+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:48.693761+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:21:48.693761+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:07.571130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.18","version":"70'139","reported_seq":921,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-08-19T14:23:05.805734+0000","last_change":"2024-08-19T14:22:07.254887+0000","last_active":"2024-08-19T14:23:05.805734+0000","last_peered":"2024-08-19T14:23:05.805734+0000","last_clean":"2024-08-19T14:23:05.805734+0000","last_became_active":"2024-08-19T14:18:31.902735+0000","last_became_peered":"2024-08-19T14:18:31.902735+0000","last_unstale":"2024-08-19T14:23:05.805734+0000","last_undegraded":"2024-08-19T14:23:05.805734+0000","last_fullsized":"2024-08-19T14:23:05.805734+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:22:07.254767+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:22:07.254767+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:11.195110+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.19","version":"70'139","reported_seq":869,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-08-19T14:23:05.810916+0000","last_change":"2024-08-19T14:21:40.219949+0000","last_active":"2024-08-19T14:23:05.810916+0000","last_peered":"2024-08-19T14:23:05.810916+0000","last_clean":"2024-08-19T14:23:05.810916+0000","last_became_active":"2024-08-19T14:18:31.902547+0000","last_became_peered":"2024-08-19T14:18:31.902547+0000","last_unstale":"2024-08-19T14:23:05.810916+0000","last_undegraded":"2024-08-19T14:23:05.810916+0000","last_fullsized":"2024-08-19T14:23:05.810916+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:21:40.219830+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:21:40.219830+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:10.467382+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1e","version":"70'139","reported_seq":528,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-08-19T14:23:05.804988+0000","last_change":"2024-08-19T14:22:09.404540+0000","last_active":"2024-08-19T14:23:05.804988+0000","last_peered":"2024-08-19T14:23:05.804988+0000","last_clean":"2024-08-19T14:23:05.804988+0000","last_became_active":"2024-08-19T14:18:31.902546+0000","last_became_peered":"2024-08-19T14:18:31.902546+0000","last_unstale":"2024-08-19T14:23:05.804988+0000","last_undegraded":"2024-08-19T14:23:05.804988+0000","last_fullsized":"2024-08-19T14:23:05.804988+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:22:09.404433+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:22:09.404433+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:13.587290+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1c","version":"70'139","reported_seq":641,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-08-19T14:23:01.783581+0000","last_change":"2024-08-19T14:22:49.619773+0000","last_active":"2024-08-19T14:23:01.783581+0000","last_peered":"2024-08-19T14:23:01.783581+0000","last_clean":"2024-08-19T14:23:01.783581+0000","last_became_active":"2024-08-19T14:18:31.902494+0000","last_became_peered":"2024-08-19T14:18:31.902494+0000","last_unstale":"2024-08-19T14:23:01.783581+0000","last_undegraded":"2024-08-19T14:23:01.783581+0000","last_fullsized":"2024-08-19T14:23:01.783581+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:22:49.619657+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:22:49.619657+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:58.403527+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,11],"acting":[8,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1d","version":"70'139","reported_seq":551,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-19T14:23:06.822873+0000","last_change":"2024-08-19T14:20:17.725615+0000","last_active":"2024-08-19T14:23:06.822873+0000","last_peered":"2024-08-19T14:23:06.822873+0000","last_clean":"2024-08-19T14:23:06.822873+0000","last_became_active":"2024-08-19T14:18:31.901183+0000","last_became_peered":"2024-08-19T14:18:31.901183+0000","last_unstale":"2024-08-19T14:23:06.822873+0000","last_undegraded":"2024-08-19T14:23:06.822873+0000","last_fullsized":"2024-08-19T14:23:06.822873+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:20:17.725477+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:20:17.725477+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,5],"acting":[1,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":4948230168,"num_objects":75505,"num_object_clones":0,"num_object_copies":226515,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75505,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":186841,"num_write_kb":8395009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4140,"num_bytes_recovered":271319040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15049306112,"data_stored":15049293896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160741,"ondisk_log_size":160741,"up":48,"acting":48,"num_store_stats":9},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1802240,"data_stored":1788032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187597,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2598184,"kb_used_data":2161632,"kb_used_omap":1,"kb_used_meta":436542,"kb_avail":91146968,"statfs":{"total":95995035648,"available":93334495232,"internally_reserved":0,"allocated":2213511168,"data_stored":2211740747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":447019460},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":343597383827,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":74,"seq":317827580055,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1694932,"kb_used_data":1212552,"kb_used_omap":1,"kb_used_meta":482366,"kb_avail":92050220,"statfs":{"total":95995035648,"available":94259425280,"internally_reserved":0,"allocated":1241653248,"data_stored":1239866090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":493943233},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874396,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":688104,"kb_used_data":318176,"kb_used_omap":1,"kb_used_meta":369918,"kb_avail":93057048,"statfs":{"total":95995035648,"available":95290417152,"internally_reserved":0,"allocated":325812224,"data_stored":324041803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":378796485},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":57,"seq":244813136031,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2907864,"kb_used_data":2411596,"kb_used_omap":1,"kb_used_meta":496254,"kb_avail":90837288,"statfs":{"total":95995035648,"available":93017382912,"internally_reserved":0,"allocated":2469474304,"data_stored":2467687666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":508164554},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":206158430371,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692007,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1564220,"kb_used_data":1185972,"kb_used_omap":1,"kb_used_meta":378238,"kb_avail":92180932,"statfs":{"total":95995035648,"available":94393274368,"internally_reserved":0,"allocated":1214435328,"data_stored":1212657150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":387316165},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986348,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2003412,"kb_used_data":1502984,"kb_used_omap":1,"kb_used_meta":500414,"kb_avail":91741740,"statfs":{"total":95995035648,"available":93943541760,"internally_reserved":0,"allocated":1539055616,"data_stored":1537268458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":512424389},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149874,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2209296,"kb_used_data":1767172,"kb_used_omap":1,"kb_used_meta":442110,"kb_avail":91535856,"statfs":{"total":95995035648,"available":93732716544,"internally_reserved":0,"allocated":1809584128,"data_stored":1807801042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":452721095},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,3,0,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411511,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575035,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3609932,"kb_used_data":3033156,"kb_used_omap":1,"kb_used_meta":576766,"kb_avail":90135220,"statfs":{"total":95995035648,"available":92298465280,"internally_reserved":0,"allocated":3105951744,"data_stored":3104165289,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":590608837},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738559,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1596072,"kb_used_data":1231712,"kb_used_omap":1,"kb_used_meta":364350,"kb_avail":92149080,"statfs":{"total":95995035648,"available":94360657920,"internally_reserved":0,"allocated":1261273088,"data_stored":1259502667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":373094853},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1241710592,"data_stored":1241710592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3091730432,"data_stored":3091726360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1792278528,"data_stored":1792278528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1525223424,"data_stored":1525219352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1206452224,"data_stored":1206452224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2443902976,"data_stored":2443902976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":319553536,"data_stored":319553536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1232080896,"data_stored":1232076824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":2196373504,"data_stored":2196373504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T14:23:12.097 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-08-19T14:23:12.098 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-19T14:23:12.098 INFO:tasks.thrashosds.thrasher:inject_pause on osd.10 2024-08-19T14:23:12.098 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-08-19T14:23:12.098 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-08-19T14:23:12.098 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.10 config set filestore_inject_stall 3 2024-08-19T14:23:12.122 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:11 smithi045 ceph-mon[20143]: pgmap v887: 47 pgs: 47 active+clean; 4.6 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 1.07k op/s 2024-08-19T14:23:12.122 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:11 smithi045 ceph-mon[20143]: from='client.25649 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:23:12.122 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:11 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:23:12.122 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:11 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:23:12.122 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:11 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3440124734' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:23:12.173 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:23:12.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:12.207 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:12.207 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:23:12.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:11 smithi032 ceph-mon[20004]: pgmap v887: 47 pgs: 47 active+clean; 4.6 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 1.07k op/s 2024-08-19T14:23:12.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:11 smithi032 ceph-mon[20004]: from='client.25649 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:23:12.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:11 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:23:12.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:11 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:23:12.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:11 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3440124734' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:23:12.307 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:23:12.339 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:12.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:12.340 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:23:12.393 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-08-19T14:23:12.431 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:23:12.441 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:23:12.474 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:12.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:12.476 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:23:12.577 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:23:12.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:12.611 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:12.611 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:23:12.713 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:23:12.750 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:12.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:12.751 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:23:12.756 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:23:12.851 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:23:12.892 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:12.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:12.893 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:23:12.995 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:23:13.033 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:13.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:13.034 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:23:13.060 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:23:13.060 INFO:teuthology.orchestra.run.smithi045.stdout: "success": "" 2024-08-19T14:23:13.060 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:23:13.131 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:12 smithi032 ceph-mon[20004]: osdmap e390: 12 total, 12 up, 9 in 2024-08-19T14:23:13.132 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:12 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:23:13.132 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:12 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:23:13.135 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:23:13.143 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:12 smithi031 ceph-mon[16336]: osdmap e390: 12 total, 12 up, 9 in 2024-08-19T14:23:13.144 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:12 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:23:13.145 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:12 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:23:13.169 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:13.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:13.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:23:13.198 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:23:13.271 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:23:13.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:12 smithi045 ceph-mon[20143]: osdmap e390: 12 total, 12 up, 9 in 2024-08-19T14:23:13.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:12 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:23:13.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:12 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:23:13.298 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:13.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:13.299 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:23:13.382 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:23:13.382 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:23:13.400 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:23:13.440 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:13.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:13.441 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:23:13.542 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:23:13.567 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:13.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:13.568 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:23:13.669 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:23:13.705 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:13.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:13.706 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:23:13.807 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:23:13.834 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:13.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:13.835 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:23:13.936 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:23:13.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:13.966 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:13.967 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:23:14.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:13 smithi032 ceph-mon[20004]: pgmap v889: 47 pgs: 47 active+clean; 4.5 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 889 op/s 2024-08-19T14:23:14.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:13 smithi032 ceph-mon[20004]: osdmap e391: 12 total, 12 up, 9 in 2024-08-19T14:23:14.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:13 smithi045 ceph-mon[20143]: pgmap v889: 47 pgs: 47 active+clean; 4.5 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 889 op/s 2024-08-19T14:23:14.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:13 smithi045 ceph-mon[20143]: osdmap e391: 12 total, 12 up, 9 in 2024-08-19T14:23:14.064 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-08-19T14:23:14.067 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:23:14.094 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:14.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:14.095 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:23:14.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:13 smithi031 ceph-mon[16336]: pgmap v889: 47 pgs: 47 active+clean; 4.5 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 889 op/s 2024-08-19T14:23:14.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:13 smithi031 ceph-mon[16336]: osdmap e391: 12 total, 12 up, 9 in 2024-08-19T14:23:14.196 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:23:14.219 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:14.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:14.220 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:23:14.321 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:23:14.349 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:14.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:14.350 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:23:14.438 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:23:14.451 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:23:14.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:14.481 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:14.481 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:23:14.583 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:23:14.609 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:14.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:14.610 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:23:14.711 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:23:14.751 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:14.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:14.752 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:23:14.853 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:23:14.872 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:14 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2300734251' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:23:14.872 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:14 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:23:14.872 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:14 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:23:14.883 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is unset 2024-08-19T14:23:14.887 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:14.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:14.888 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:23:14.988 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:23:15.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:15.029 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:15.029 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:23:15.035 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:23:15.035 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:23:15.130 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:23:15.156 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:15.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:15.157 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:23:15.191 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:23:15.235 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:14 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2300734251' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:23:15.235 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:14 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:23:15.235 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:14 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:23:15.258 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:23:15.284 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:15.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:15.285 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:23:15.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:14 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2300734251' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:23:15.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:14 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:23:15.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:14 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:23:15.386 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:23:15.424 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:15.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:15.426 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:23:15.527 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:23:15.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:15.563 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:15.563 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:23:15.664 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:23:15.690 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:15.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:15.691 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:23:15.792 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:23:15.819 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:15.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:15.820 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:23:15.921 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:23:15.949 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:15.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:15.950 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:23:16.035 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-08-19T14:23:16.051 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:23:16.087 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:16.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:16.088 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:23:16.189 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:23:16.212 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:16.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:16.214 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:23:16.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:15 smithi032 ceph-mon[20004]: pgmap v891: 47 pgs: 47 active+clean; 4.3 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 1.01k op/s 2024-08-19T14:23:16.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:15 smithi032 ceph-mon[20004]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:23:16.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:15 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2300734251' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:23:16.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:15 smithi032 ceph-mon[20004]: osdmap e392: 12 total, 12 up, 9 in 2024-08-19T14:23:16.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:15 smithi045 ceph-mon[20143]: pgmap v891: 47 pgs: 47 active+clean; 4.3 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 1.01k op/s 2024-08-19T14:23:16.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:15 smithi045 ceph-mon[20143]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:23:16.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:15 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2300734251' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:23:16.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:15 smithi045 ceph-mon[20143]: osdmap e392: 12 total, 12 up, 9 in 2024-08-19T14:23:16.315 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:23:16.341 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:16.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:16.342 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:23:16.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:15 smithi031 ceph-mon[16336]: pgmap v891: 47 pgs: 47 active+clean; 4.3 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 1.01k op/s 2024-08-19T14:23:16.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:15 smithi031 ceph-mon[16336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:23:16.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:15 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2300734251' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:23:16.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:15 smithi031 ceph-mon[16336]: osdmap e392: 12 total, 12 up, 9 in 2024-08-19T14:23:16.404 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:23:16.443 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:23:16.464 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:16.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:16.465 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:23:16.566 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:23:16.598 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:16.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:16.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:23:16.700 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:23:16.733 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:16.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:16.734 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:23:16.834 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:23:16.855 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:16.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:16.856 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:23:16.880 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:23:16.947 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:23:16.947 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:23:16.957 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:23:16.996 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:16.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:16.997 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:23:17.098 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:23:17.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:16 smithi031 ceph-mon[16336]: 1.1a scrub starts 2024-08-19T14:23:17.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:16 smithi031 ceph-mon[16336]: 1.1a scrub ok 2024-08-19T14:23:17.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:16 smithi031 ceph-mon[16336]: 1.14 scrub starts 2024-08-19T14:23:17.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:16 smithi031 ceph-mon[16336]: 1.14 scrub ok 2024-08-19T14:23:17.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:16 smithi031 ceph-mon[16336]: 1.7 scrub starts 2024-08-19T14:23:17.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:16 smithi031 ceph-mon[16336]: 1.7 scrub ok 2024-08-19T14:23:17.137 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:17.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:17.138 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:23:17.239 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:23:17.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:16 smithi032 ceph-mon[20004]: 1.1a scrub starts 2024-08-19T14:23:17.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:16 smithi032 ceph-mon[20004]: 1.1a scrub ok 2024-08-19T14:23:17.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:16 smithi032 ceph-mon[20004]: 1.14 scrub starts 2024-08-19T14:23:17.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:16 smithi032 ceph-mon[20004]: 1.14 scrub ok 2024-08-19T14:23:17.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:16 smithi032 ceph-mon[20004]: 1.7 scrub starts 2024-08-19T14:23:17.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:16 smithi032 ceph-mon[20004]: 1.7 scrub ok 2024-08-19T14:23:17.267 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:17.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:17.269 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:23:17.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:16 smithi045 ceph-mon[20143]: 1.1a scrub starts 2024-08-19T14:23:17.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:16 smithi045 ceph-mon[20143]: 1.1a scrub ok 2024-08-19T14:23:17.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:16 smithi045 ceph-mon[20143]: 1.14 scrub starts 2024-08-19T14:23:17.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:16 smithi045 ceph-mon[20143]: 1.14 scrub ok 2024-08-19T14:23:17.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:16 smithi045 ceph-mon[20143]: 1.7 scrub starts 2024-08-19T14:23:17.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:16 smithi045 ceph-mon[20143]: 1.7 scrub ok 2024-08-19T14:23:17.369 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:23:17.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:17.413 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:17.413 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:23:17.515 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:23:17.530 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:23:17.530 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:23:17.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:17.552 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:17.552 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:23:17.571 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:23:17.571 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:23:17.601 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:23:17.602 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:23:17.632 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:23:17.633 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:23:17.641 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-08-19T14:23:17.653 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:23:17.669 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:23:17.670 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:23:17.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:17.686 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:17.686 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:23:17.704 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:23:17.704 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:23:17.735 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:23:17.735 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:23:17.768 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:23:17.769 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:23:17.787 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:23:17.799 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:23:17.799 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:23:17.818 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:17.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:17.819 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:23:17.829 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:23:17.829 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:23:17.857 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:23:17.857 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:23:17.886 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:17 smithi032 ceph-mon[20004]: pgmap v893: 47 pgs: 47 active+clean; 4.3 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 784 op/s 2024-08-19T14:23:17.886 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:17 smithi032 ceph-mon[20004]: 1.e scrub starts 2024-08-19T14:23:17.886 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:17 smithi032 ceph-mon[20004]: 1.e scrub ok 2024-08-19T14:23:17.886 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:17 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2611191148' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:23:17.890 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:23:17.890 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:23:17.921 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:23:17.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:17.957 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:17.957 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:23:18.009 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:23:18.058 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:23:18.101 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:18.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:18.103 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:23:18.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:17 smithi031 ceph-mon[16336]: pgmap v893: 47 pgs: 47 active+clean; 4.3 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 784 op/s 2024-08-19T14:23:18.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:17 smithi031 ceph-mon[16336]: 1.e scrub starts 2024-08-19T14:23:18.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:17 smithi031 ceph-mon[16336]: 1.e scrub ok 2024-08-19T14:23:18.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:17 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2611191148' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:23:18.203 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:23:18.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:18.228 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:18.228 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:23:18.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:17 smithi045 ceph-mon[20143]: pgmap v893: 47 pgs: 47 active+clean; 4.3 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 784 op/s 2024-08-19T14:23:18.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:17 smithi045 ceph-mon[20143]: 1.e scrub starts 2024-08-19T14:23:18.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:17 smithi045 ceph-mon[20143]: 1.e scrub ok 2024-08-19T14:23:18.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:17 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2611191148' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:23:18.329 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:23:18.360 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:18.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:18.361 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:23:18.462 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:23:18.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:18.491 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:18.491 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:23:18.592 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:23:18.608 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:23:18.608 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-08-19T14:23:18.608 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-08-19T14:23:18.609 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-08-19T14:23:18.609 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:18.609 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:62930 2.7 2.377d1257 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-08-19T14:23:18.609 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:18.725228+0000", 2024-08-19T14:23:18.609 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 599.88138229499998, 2024-08-19T14:23:18.609 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.99330006599999998, 2024-08-19T14:23:18.609 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:23:18.609 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:23:18.610 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:23:18.610 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:23:18.610 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:23:18.610 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:23:18.610 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 62930 2024-08-19T14:23:18.610 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:18.610 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:23:18.610 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:18.610 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:23:18.611 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.725228+0000", 2024-08-19T14:23:18.611 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:18.611 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:18.611 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:18.611 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:23:18.611 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.725228+0000", 2024-08-19T14:23:18.611 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:18.611 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:18.612 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:18.612 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:23:18.612 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.725231+0000", 2024-08-19T14:23:18.612 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5950000000000001e-06 2024-08-19T14:23:18.612 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:18.612 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:18.612 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:23:18.613 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.725270+0000", 2024-08-19T14:23:18.613 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9162000000000002e-05 2024-08-19T14:23:18.613 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:18.613 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:18.613 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:23:18.613 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.725291+0000", 2024-08-19T14:23:18.613 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1220999999999999e-05 2024-08-19T14:23:18.613 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:18.614 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:18.614 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:23:18.614 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.725303+0000", 2024-08-19T14:23:18.614 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1788e-05 2024-08-19T14:23:18.614 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:18.614 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:18.614 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:23:18.614 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.725382+0000", 2024-08-19T14:23:18.615 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.9389999999999997e-05 2024-08-19T14:23:18.615 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:18.615 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:18.615 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:23:18.615 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.725395+0000", 2024-08-19T14:23:18.615 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.308e-05 2024-08-19T14:23:18.615 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:18.615 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:18.615 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:23:18.616 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.718510+0000", 2024-08-19T14:23:18.616 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.99311438900000004 2024-08-19T14:23:18.616 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:18.616 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:18.616 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:23:18.616 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.718528+0000", 2024-08-19T14:23:18.616 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8440999999999998e-05 2024-08-19T14:23:18.616 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:18.616 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:23:18.617 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:18.617 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:18.617 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:18.617 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:62932 2.7 2.84394fd7 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-08-19T14:23:18.617 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:18.727423+0000", 2024-08-19T14:23:18.617 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 599.87918713399995, 2024-08-19T14:23:18.617 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.99114474299999999, 2024-08-19T14:23:18.617 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:23:18.618 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:23:18.618 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:23:18.618 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:23:18.618 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:23:18.618 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:23:18.618 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 62932 2024-08-19T14:23:18.618 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:18.619 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:23:18.619 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:18.619 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:23:18.619 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.727423+0000", 2024-08-19T14:23:18.619 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:18.619 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:18.619 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:18.619 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:23:18.619 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.727423+0000", 2024-08-19T14:23:18.620 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:18.620 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:18.620 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:18.620 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:23:18.620 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.727425+0000", 2024-08-19T14:23:18.620 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0870000000000001e-06 2024-08-19T14:23:18.620 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:18.620 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:18.620 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:23:18.621 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.727471+0000", 2024-08-19T14:23:18.621 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5300999999999999e-05 2024-08-19T14:23:18.621 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:18.621 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:18.622 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:23:18.622 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.727489+0000", 2024-08-19T14:23:18.622 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7909000000000001e-05 2024-08-19T14:23:18.622 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:18.622 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:18.622 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:23:18.622 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.727497+0000", 2024-08-19T14:23:18.622 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.2150000000000001e-06 2024-08-19T14:23:18.622 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:18.623 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:18.623 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:23:18.623 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.727590+0000", 2024-08-19T14:23:18.623 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3193e-05 2024-08-19T14:23:18.623 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:18.623 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:18.623 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:23:18.623 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.727606+0000", 2024-08-19T14:23:18.624 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6345e-05 2024-08-19T14:23:18.624 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:18.624 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:18.624 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:23:18.624 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.718558+0000", 2024-08-19T14:23:18.624 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.99095196399999996 2024-08-19T14:23:18.624 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:18.624 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:18.624 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:23:18.625 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.718568+0000", 2024-08-19T14:23:18.625 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.7289999999999995e-06 2024-08-19T14:23:18.625 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:18.625 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:23:18.625 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:18.625 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:18.625 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:18.625 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:62948 2.7 2.63c33357 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-08-19T14:23:18.626 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:18.736424+0000", 2024-08-19T14:23:18.626 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 599.87018671299995, 2024-08-19T14:23:18.626 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.98218305500000003, 2024-08-19T14:23:18.626 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:23:18.626 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:23:18.626 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:23:18.626 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:23:18.627 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:23:18.627 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:23:18.627 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 62948 2024-08-19T14:23:18.627 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:18.627 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:23:18.627 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:18.627 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:23:18.627 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.736424+0000", 2024-08-19T14:23:18.627 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:18.627 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:18.627 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:18.627 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:23:18.627 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.736424+0000", 2024-08-19T14:23:18.627 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:18.628 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:18.628 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:18.628 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:23:18.628 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.736426+0000", 2024-08-19T14:23:18.628 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9980000000000002e-06 2024-08-19T14:23:18.628 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:18.628 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:18.628 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:23:18.628 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.736459+0000", 2024-08-19T14:23:18.628 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3155999999999999e-05 2024-08-19T14:23:18.628 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:18.628 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:18.628 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:23:18.628 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.736479+0000", 2024-08-19T14:23:18.628 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0052999999999999e-05 2024-08-19T14:23:18.629 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:18.629 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:18.629 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:23:18.629 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.736489+0000", 2024-08-19T14:23:18.629 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.7790000000000007e-06 2024-08-19T14:23:18.629 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:18.629 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:18.629 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:23:18.629 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.736567+0000", 2024-08-19T14:23:18.629 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.8240999999999998e-05 2024-08-19T14:23:18.629 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:18.629 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:18.629 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:23:18.629 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.736578+0000", 2024-08-19T14:23:18.630 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0813e-05 2024-08-19T14:23:18.630 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:18.630 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:18.630 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:23:18.630 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.718594+0000", 2024-08-19T14:23:18.630 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.98201632900000002 2024-08-19T14:23:18.630 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:18.630 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:18.631 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:23:18.631 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.718607+0000", 2024-08-19T14:23:18.631 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2686e-05 2024-08-19T14:23:18.631 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:18.632 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:23:18.632 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:18.632 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:18.632 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:18.632 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:62956 2.7 2.bf121107 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-08-19T14:23:18.632 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:18.741261+0000", 2024-08-19T14:23:18.632 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 599.86534928399999, 2024-08-19T14:23:18.632 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.97739631299999996, 2024-08-19T14:23:18.632 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:23:18.632 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:23:18.632 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:23:18.632 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:23:18.632 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:23:18.632 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:23:18.633 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 62956 2024-08-19T14:23:18.633 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:18.633 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:23:18.633 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:18.633 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:23:18.633 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.741261+0000", 2024-08-19T14:23:18.633 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:18.633 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:18.633 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:18.633 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:23:18.633 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.741261+0000", 2024-08-19T14:23:18.633 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:18.633 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:18.633 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:18.634 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:23:18.634 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.741263+0000", 2024-08-19T14:23:18.634 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.68e-06 2024-08-19T14:23:18.634 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:18.634 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:18.634 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:23:18.634 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.741301+0000", 2024-08-19T14:23:18.634 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8313999999999999e-05 2024-08-19T14:23:18.634 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:18.634 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:18.634 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:23:18.634 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.741320+0000", 2024-08-19T14:23:18.634 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.927e-05 2024-08-19T14:23:18.634 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:18.635 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:18.635 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:23:18.635 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.741328+0000", 2024-08-19T14:23:18.635 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3000000000000004e-06 2024-08-19T14:23:18.635 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:18.635 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:18.635 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:23:18.635 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.741408+0000", 2024-08-19T14:23:18.635 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.9839000000000006e-05 2024-08-19T14:23:18.635 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:18.635 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:18.635 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:23:18.635 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.741421+0000", 2024-08-19T14:23:18.635 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2914e-05 2024-08-19T14:23:18.635 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:18.636 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:18.636 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:23:18.636 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.718639+0000", 2024-08-19T14:23:18.636 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.97721861399999999 2024-08-19T14:23:18.636 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:18.636 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:18.636 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:23:18.636 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.718658+0000", 2024-08-19T14:23:18.636 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8382000000000001e-05 2024-08-19T14:23:18.636 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:18.636 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:23:18.636 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:18.636 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:18.636 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:18.637 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:62963 2.7 2.30acc547 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-08-19T14:23:18.637 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:18.745404+0000", 2024-08-19T14:23:18.637 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 599.86120674100005, 2024-08-19T14:23:18.637 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.97330407100000005, 2024-08-19T14:23:18.637 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:23:18.637 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:23:18.637 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:23:18.637 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:23:18.637 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:23:18.637 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:23:18.637 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 62963 2024-08-19T14:23:18.637 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:18.638 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:23:18.638 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:18.638 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:23:18.638 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.745404+0000", 2024-08-19T14:23:18.638 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:18.638 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:18.638 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:18.638 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:23:18.638 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.745404+0000", 2024-08-19T14:23:18.638 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:18.638 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:18.638 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:18.639 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:23:18.639 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.745406+0000", 2024-08-19T14:23:18.639 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.328e-06 2024-08-19T14:23:18.639 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:18.639 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:18.639 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:23:18.639 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.745440+0000", 2024-08-19T14:23:18.639 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3837000000000001e-05 2024-08-19T14:23:18.639 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:18.639 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:18.639 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:23:18.639 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.745474+0000", 2024-08-19T14:23:18.639 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.4196999999999998e-05 2024-08-19T14:23:18.639 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:18.639 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:18.640 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:23:18.640 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.745485+0000", 2024-08-19T14:23:18.640 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.095e-05 2024-08-19T14:23:18.640 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:18.640 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:18.640 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:23:18.640 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.745568+0000", 2024-08-19T14:23:18.640 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.2770999999999997e-05 2024-08-19T14:23:18.640 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:18.640 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:18.640 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:23:18.640 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.745577+0000", 2024-08-19T14:23:18.640 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.1269999999999992e-06 2024-08-19T14:23:18.640 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:18.641 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:18.641 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:23:18.641 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.718689+0000", 2024-08-19T14:23:18.641 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.97311253600000003 2024-08-19T14:23:18.641 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:18.641 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:18.641 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:23:18.641 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.718708+0000", 2024-08-19T14:23:18.641 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8325000000000002e-05 2024-08-19T14:23:18.641 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:18.641 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:23:18.641 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:18.641 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:18.641 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:18.641 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:62983 2.7 2.e3420e17 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-08-19T14:23:18.642 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:18.759553+0000", 2024-08-19T14:23:18.642 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 599.847057382, 2024-08-19T14:23:18.642 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.95919200800000004, 2024-08-19T14:23:18.642 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:23:18.642 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:23:18.642 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:23:18.642 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:23:18.642 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:23:18.642 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:23:18.642 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 62983 2024-08-19T14:23:18.642 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:18.642 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:23:18.642 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:18.642 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:23:18.643 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.759553+0000", 2024-08-19T14:23:18.643 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:18.643 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:18.643 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:18.643 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:23:18.643 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.759553+0000", 2024-08-19T14:23:18.643 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:18.643 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:18.643 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:18.643 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:23:18.643 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.759555+0000", 2024-08-19T14:23:18.643 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2630000000000002e-06 2024-08-19T14:23:18.643 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:18.643 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:18.644 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:23:18.644 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.759595+0000", 2024-08-19T14:23:18.644 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9490000000000003e-05 2024-08-19T14:23:18.644 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:18.644 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:18.644 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:23:18.644 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.759614+0000", 2024-08-19T14:23:18.644 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9446999999999999e-05 2024-08-19T14:23:18.644 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:18.644 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:18.644 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:23:18.644 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.759622+0000", 2024-08-19T14:23:18.644 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5519999999999999e-06 2024-08-19T14:23:18.644 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:18.644 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:18.645 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:23:18.645 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.759684+0000", 2024-08-19T14:23:18.645 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2168000000000005e-05 2024-08-19T14:23:18.645 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:18.645 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:18.645 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:23:18.645 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.759693+0000", 2024-08-19T14:23:18.645 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.8920000000000006e-06 2024-08-19T14:23:18.645 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:18.645 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:18.645 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:23:18.645 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.718735+0000", 2024-08-19T14:23:18.645 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.95904258099999995 2024-08-19T14:23:18.645 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:18.646 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:18.646 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:23:18.646 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.718745+0000", 2024-08-19T14:23:18.646 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.6150000000000003e-06 2024-08-19T14:23:18.646 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:18.646 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:23:18.646 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:18.646 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:18.646 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:18.646 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:63032 2.7 2.24f95377 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-08-19T14:23:18.647 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:13:18.809762+0000", 2024-08-19T14:23:18.647 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 599.79684869899995, 2024-08-19T14:23:18.647 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.909118655, 2024-08-19T14:23:18.647 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:23:18.647 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:23:18.647 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:23:18.647 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:23:18.647 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:23:18.647 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:23:18.647 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 63032 2024-08-19T14:23:18.647 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:18.647 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:23:18.647 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:18.647 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:23:18.648 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.809762+0000", 2024-08-19T14:23:18.648 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:18.648 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:18.648 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:18.648 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:23:18.648 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.809762+0000", 2024-08-19T14:23:18.648 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:18.648 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:18.648 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:18.648 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:23:18.648 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.809764+0000", 2024-08-19T14:23:18.648 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1490000000000001e-06 2024-08-19T14:23:18.648 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:18.648 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:18.648 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:23:18.649 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.809805+0000", 2024-08-19T14:23:18.649 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1241e-05 2024-08-19T14:23:18.649 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:18.649 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:18.649 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:23:18.649 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.809820+0000", 2024-08-19T14:23:18.649 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4355000000000001e-05 2024-08-19T14:23:18.649 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:18.649 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:18.649 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:23:18.649 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.809826+0000", 2024-08-19T14:23:18.649 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4829999999999996e-06 2024-08-19T14:23:18.649 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:18.649 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:18.650 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:23:18.650 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.809883+0000", 2024-08-19T14:23:18.650 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6885000000000002e-05 2024-08-19T14:23:18.650 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:18.650 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:18.650 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:23:18.650 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:18.809892+0000", 2024-08-19T14:23:18.650 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.8519999999999993e-06 2024-08-19T14:23:18.650 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:18.650 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:18.650 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:23:18.650 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.718868+0000", 2024-08-19T14:23:18.650 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.90897634900000002 2024-08-19T14:23:18.650 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:18.650 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:18.651 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:23:18.651 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:13:19.718880+0000", 2024-08-19T14:23:18.651 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2340999999999999e-05 2024-08-19T14:23:18.651 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:18.651 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:23:18.652 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:18.652 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:18.652 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:23:18.652 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:23:18.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:18.653 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:18.653 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:23:18.754 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:23:18.777 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:18.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:18.778 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:23:18.835 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-19T14:23:18.880 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:23:18.893 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:18 smithi031 ceph-mon[16336]: 4.9 scrub starts 2024-08-19T14:23:18.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:18.937 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:18.937 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:23:19.038 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:23:19.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:19.079 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:19.080 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:23:19.096 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 7, 8, 9, 11] out_osds: [10, 6, 2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-08-19T14:23:19.097 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-19T14:23:19.097 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-08-19T14:23:19.097 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph tell osd.0 'trim stale osdmaps' 2024-08-19T14:23:19.175 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:18 smithi031 ceph-mon[16336]: pgmap v894: 47 pgs: 47 active+clean; 4.1 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 998 op/s 2024-08-19T14:23:19.179 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:18 smithi032 ceph-mon[20004]: 4.9 scrub starts 2024-08-19T14:23:19.179 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:18 smithi032 ceph-mon[20004]: pgmap v894: 47 pgs: 47 active+clean; 4.1 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 998 op/s 2024-08-19T14:23:19.181 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:23:19.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:19.232 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:19.232 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:23:19.291 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:23:19.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:18 smithi045 ceph-mon[20143]: 4.9 scrub starts 2024-08-19T14:23:19.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:18 smithi045 ceph-mon[20143]: pgmap v894: 47 pgs: 47 active+clean; 4.1 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 998 op/s 2024-08-19T14:23:19.333 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:23:19.371 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:19.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:19.372 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:23:19.453 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-08-19T14:23:19.474 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:23:19.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:19.501 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:19.501 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:23:19.602 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:23:19.626 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:19.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:19.627 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:23:19.728 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:23:19.755 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:19.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:19.756 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:23:19.838 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:23:19.857 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:23:19.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:19.887 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:19.887 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:23:19.932 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:23:19.988 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:23:20.015 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:20.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:20.016 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:23:20.077 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:19 smithi031 ceph-mon[16336]: 1.1e scrub starts 2024-08-19T14:23:20.077 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:19 smithi031 ceph-mon[16336]: 1.1e scrub ok 2024-08-19T14:23:20.077 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:19 smithi031 ceph-mon[16336]: 1.19 scrub starts 2024-08-19T14:23:20.077 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:19 smithi031 ceph-mon[16336]: 1.19 scrub ok 2024-08-19T14:23:20.077 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:19 smithi031 ceph-mon[16336]: 4.9 scrub ok 2024-08-19T14:23:20.077 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:19 smithi031 ceph-mon[16336]: 1.1b scrub starts 2024-08-19T14:23:20.077 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:19 smithi031 ceph-mon[16336]: 1.1b scrub ok 2024-08-19T14:23:20.077 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:19 smithi031 ceph-mon[16336]: 4.c scrub starts 2024-08-19T14:23:20.078 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:19 smithi031 ceph-mon[16336]: 1.8 scrub starts 2024-08-19T14:23:20.078 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:19 smithi031 ceph-mon[16336]: 1.8 scrub ok 2024-08-19T14:23:20.078 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:19 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:23:20.078 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:19 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:23:20.117 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:23:20.152 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:19 smithi032 ceph-mon[20004]: 1.1e scrub starts 2024-08-19T14:23:20.152 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:19 smithi032 ceph-mon[20004]: 1.1e scrub ok 2024-08-19T14:23:20.152 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:19 smithi032 ceph-mon[20004]: 1.19 scrub starts 2024-08-19T14:23:20.152 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:19 smithi032 ceph-mon[20004]: 1.19 scrub ok 2024-08-19T14:23:20.152 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:19 smithi032 ceph-mon[20004]: 4.9 scrub ok 2024-08-19T14:23:20.152 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:19 smithi032 ceph-mon[20004]: 1.1b scrub starts 2024-08-19T14:23:20.152 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:19 smithi032 ceph-mon[20004]: 1.1b scrub ok 2024-08-19T14:23:20.152 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:19 smithi032 ceph-mon[20004]: 4.c scrub starts 2024-08-19T14:23:20.153 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:19 smithi032 ceph-mon[20004]: 1.8 scrub starts 2024-08-19T14:23:20.153 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:19 smithi032 ceph-mon[20004]: 1.8 scrub ok 2024-08-19T14:23:20.153 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:19 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:23:20.153 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:19 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:23:20.160 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:20.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:20.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:23:20.262 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:23:20.286 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:20.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:20.396 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:23:20.397 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:19 smithi045 ceph-mon[20143]: 1.1e scrub starts 2024-08-19T14:23:20.397 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:19 smithi045 ceph-mon[20143]: 1.1e scrub ok 2024-08-19T14:23:20.397 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:19 smithi045 ceph-mon[20143]: 1.19 scrub starts 2024-08-19T14:23:20.397 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:19 smithi045 ceph-mon[20143]: 1.19 scrub ok 2024-08-19T14:23:20.397 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:19 smithi045 ceph-mon[20143]: 4.9 scrub ok 2024-08-19T14:23:20.397 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:19 smithi045 ceph-mon[20143]: 1.1b scrub starts 2024-08-19T14:23:20.397 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:19 smithi045 ceph-mon[20143]: 1.1b scrub ok 2024-08-19T14:23:20.397 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:19 smithi045 ceph-mon[20143]: 4.c scrub starts 2024-08-19T14:23:20.397 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:19 smithi045 ceph-mon[20143]: 1.8 scrub starts 2024-08-19T14:23:20.397 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:19 smithi045 ceph-mon[20143]: 1.8 scrub ok 2024-08-19T14:23:20.397 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:19 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:23:20.397 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:19 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:23:20.434 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:23:20.434 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-08-19T14:23:20.434 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:20.434 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.25333.0:199209 4.a e393/335)", 2024-08-19T14:23:20.435 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:23:20.432514+0000", 2024-08-19T14:23:20.435 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.00030586600000000002, 2024-08-19T14:23:20.435 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000328196, 2024-08-19T14:23:20.435 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:23:20.435 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:23:20.435 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "queued for pg", 2024-08-19T14:23:20.435 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:23:20.435 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:20.435 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:23:20.435 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.432514+0000", 2024-08-19T14:23:20.435 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:20.435 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:20.435 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:20.435 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:23:20.436 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.432514+0000", 2024-08-19T14:23:20.436 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:20.436 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:20.436 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:20.436 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:23:20.436 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.432514+0000", 2024-08-19T14:23:20.436 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.5899999999999996e-07 2024-08-19T14:23:20.436 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:20.436 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:20.436 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:23:20.436 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.432520+0000", 2024-08-19T14:23:20.436 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0859999999999998e-06 2024-08-19T14:23:20.436 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:20.436 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:20.437 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:23:20.437 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.432529+0000", 2024-08-19T14:23:20.437 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6419999999999996e-06 2024-08-19T14:23:20.437 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:20.437 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:20.437 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:23:20.437 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.432535+0000", 2024-08-19T14:23:20.437 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1449999999999996e-06 2024-08-19T14:23:20.437 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:20.437 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:23:20.437 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:20.437 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:20.437 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:20.437 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.25333.0:199211 4.0 e393/335 4:0328d802:::benchmark_data_smithi045_71562_object68036:head v 393'12372, mlcod=393'12370)", 2024-08-19T14:23:20.438 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:23:20.432561+0000", 2024-08-19T14:23:20.438 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.00025807100000000002, 2024-08-19T14:23:20.438 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00033368999999999998, 2024-08-19T14:23:20.438 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:23:20.438 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:23:20.438 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-08-19T14:23:20.438 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:23:20.438 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:20.438 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:23:20.438 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.432561+0000", 2024-08-19T14:23:20.438 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:20.438 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:20.439 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:20.439 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:23:20.439 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.432561+0000", 2024-08-19T14:23:20.439 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:20.439 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:20.439 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:20.439 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:23:20.439 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.432562+0000", 2024-08-19T14:23:20.439 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5999999999999999e-07 2024-08-19T14:23:20.439 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:20.439 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:20.439 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:23:20.439 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.432566+0000", 2024-08-19T14:23:20.439 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9160000000000003e-06 2024-08-19T14:23:20.440 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:20.440 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:20.440 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:23:20.440 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.432573+0000", 2024-08-19T14:23:20.440 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8730000000000001e-06 2024-08-19T14:23:20.440 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:20.440 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:20.440 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:23:20.440 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.432577+0000", 2024-08-19T14:23:20.440 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7539999999999997e-06 2024-08-19T14:23:20.440 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:20.440 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:20.440 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:23:20.441 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.432641+0000", 2024-08-19T14:23:20.441 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3168000000000002e-05 2024-08-19T14:23:20.441 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:20.441 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:20.441 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:23:20.441 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.432658+0000", 2024-08-19T14:23:20.441 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.696e-05 2024-08-19T14:23:20.441 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:20.441 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:23:20.441 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:20.441 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:20.441 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:20.441 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.25333.0:199201 4.3 e393/335 4:c29e55f1:::benchmark_data_smithi045_71562_object68026:head v 393'12421, mlcod=393'12417)", 2024-08-19T14:23:20.441 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:23:20.423815+0000", 2024-08-19T14:23:20.442 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.0090044430000000009, 2024-08-19T14:23:20.442 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0091408140000000006, 2024-08-19T14:23:20.442 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:23:20.442 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:23:20.442 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-08-19T14:23:20.442 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:23:20.442 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:20.442 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:23:20.442 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.423815+0000", 2024-08-19T14:23:20.443 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:20.443 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:20.443 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:20.443 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:23:20.443 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.423815+0000", 2024-08-19T14:23:20.443 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:20.443 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:20.444 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:20.444 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:23:20.444 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.423816+0000", 2024-08-19T14:23:20.444 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4400000000000001e-07 2024-08-19T14:23:20.444 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:20.444 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:20.444 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:23:20.444 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.423821+0000", 2024-08-19T14:23:20.444 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4450000000000004e-06 2024-08-19T14:23:20.444 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:20.444 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:20.444 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:23:20.445 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.423830+0000", 2024-08-19T14:23:20.445 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.5350000000000004e-06 2024-08-19T14:23:20.445 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:20.445 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:20.445 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:23:20.445 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.423835+0000", 2024-08-19T14:23:20.445 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5509999999999999e-06 2024-08-19T14:23:20.445 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:20.445 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:20.445 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:23:20.445 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.427052+0000", 2024-08-19T14:23:20.445 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0032168660000000001 2024-08-19T14:23:20.445 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:20.445 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:20.446 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:23:20.446 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.427073+0000", 2024-08-19T14:23:20.446 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1186e-05 2024-08-19T14:23:20.446 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:20.446 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:23:20.446 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:20.446 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:20.446 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:20.446 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.25333.0:199203 4.3 e393/335 4:cd445660:::benchmark_data_smithi045_71562_object68028:head v 393'12422, mlcod=393'12418)", 2024-08-19T14:23:20.446 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:23:20.428085+0000", 2024-08-19T14:23:20.446 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.0047348090000000004, 2024-08-19T14:23:20.446 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0049349620000000002, 2024-08-19T14:23:20.446 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:23:20.446 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:23:20.447 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-08-19T14:23:20.447 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:23:20.447 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:20.447 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:23:20.447 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.428085+0000", 2024-08-19T14:23:20.447 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:20.447 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:20.447 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:20.447 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:23:20.447 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.428085+0000", 2024-08-19T14:23:20.447 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:20.447 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:20.448 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:20.448 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:23:20.448 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.428086+0000", 2024-08-19T14:23:20.448 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.8800000000000002e-07 2024-08-19T14:23:20.448 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:20.448 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:20.448 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:23:20.448 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.428092+0000", 2024-08-19T14:23:20.448 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8709999999999999e-06 2024-08-19T14:23:20.448 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:20.448 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:20.448 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:23:20.448 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.428104+0000", 2024-08-19T14:23:20.448 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1800999999999999e-05 2024-08-19T14:23:20.448 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:20.449 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:20.449 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:23:20.449 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.428111+0000", 2024-08-19T14:23:20.449 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3740000000000003e-06 2024-08-19T14:23:20.449 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:20.449 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:20.449 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:23:20.449 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.431910+0000", 2024-08-19T14:23:20.449 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.003799846 2024-08-19T14:23:20.449 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:20.449 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:20.449 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:23:20.449 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.431967+0000", 2024-08-19T14:23:20.449 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7059999999999999e-05 2024-08-19T14:23:20.450 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:20.450 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:23:20.450 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:20.450 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:20.450 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:20.450 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.25333.0:199204 4.7 e393/335 4:efea476c:::benchmark_data_smithi045_71562_object68029:head v 393'12501, mlcod=393'12498)", 2024-08-19T14:23:20.450 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:23:20.428192+0000", 2024-08-19T14:23:20.450 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.004627511, 2024-08-19T14:23:20.450 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0048857240000000001, 2024-08-19T14:23:20.450 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:23:20.450 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:23:20.450 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-08-19T14:23:20.450 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:23:20.450 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:20.451 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:23:20.451 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.428192+0000", 2024-08-19T14:23:20.451 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:20.451 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:20.451 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:20.451 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:23:20.451 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.428192+0000", 2024-08-19T14:23:20.451 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:20.451 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:20.451 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:20.451 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:23:20.451 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.428193+0000", 2024-08-19T14:23:20.451 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.9999999999999996e-07 2024-08-19T14:23:20.451 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:20.452 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:20.452 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:23:20.452 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.428198+0000", 2024-08-19T14:23:20.452 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9520000000000002e-06 2024-08-19T14:23:20.452 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:20.452 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:20.452 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:23:20.452 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.428208+0000", 2024-08-19T14:23:20.452 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0475999999999999e-05 2024-08-19T14:23:20.452 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:20.452 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:20.452 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:23:20.452 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.428215+0000", 2024-08-19T14:23:20.452 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3550000000000001e-06 2024-08-19T14:23:20.452 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:20.453 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:20.453 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:23:20.453 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.428263+0000", 2024-08-19T14:23:20.453 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8044999999999998e-05 2024-08-19T14:23:20.453 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:20.453 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:20.453 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:23:20.453 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.428284+0000", 2024-08-19T14:23:20.453 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1401999999999999e-05 2024-08-19T14:23:20.453 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:20.453 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:23:20.453 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:20.453 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:20.453 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:20.454 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.25333.0:199214 4.6 e393/384 4:6caacb0a:::benchmark_data_smithi045_71562_object68039:head v 393'12584, mlcod=393'12582)", 2024-08-19T14:23:20.454 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:23:20.432333+0000", 2024-08-19T14:23:20.454 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.00048635600000000002, 2024-08-19T14:23:20.454 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000800624, 2024-08-19T14:23:20.454 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:23:20.454 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:23:20.454 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-08-19T14:23:20.454 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:23:20.454 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:20.454 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:23:20.454 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.432333+0000", 2024-08-19T14:23:20.454 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:20.454 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:20.454 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:20.455 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:23:20.455 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.432333+0000", 2024-08-19T14:23:20.455 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:20.455 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:20.455 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:20.455 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:23:20.455 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.432334+0000", 2024-08-19T14:23:20.455 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0690000000000001e-06 2024-08-19T14:23:20.455 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:20.455 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:20.455 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:23:20.455 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.432341+0000", 2024-08-19T14:23:20.455 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.5919999999999997e-06 2024-08-19T14:23:20.455 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:20.455 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:20.456 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:23:20.456 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.432357+0000", 2024-08-19T14:23:20.456 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6373000000000001e-05 2024-08-19T14:23:20.456 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:20.456 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:20.456 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:23:20.456 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.432364+0000", 2024-08-19T14:23:20.456 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.5799999999999997e-06 2024-08-19T14:23:20.456 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:20.456 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:20.456 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:23:20.456 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.432438+0000", 2024-08-19T14:23:20.456 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4014e-05 2024-08-19T14:23:20.456 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:20.457 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:20.457 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:23:20.457 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.432461+0000", 2024-08-19T14:23:20.457 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.353e-05 2024-08-19T14:23:20.457 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:20.457 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:23:20.457 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:20.457 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:20.457 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-08-19T14:23:20.457 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 6 2024-08-19T14:23:20.457 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:23:20.497 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:23:20.521 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:20.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:20.522 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:23:20.623 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:23:20.638 INFO:teuthology.orchestra.run.smithi031.stderr:no valid command found; 10 closest matches: 2024-08-19T14:23:20.638 INFO:teuthology.orchestra.run.smithi031.stderr:0 2024-08-19T14:23:20.638 INFO:teuthology.orchestra.run.smithi031.stderr:1 2024-08-19T14:23:20.638 INFO:teuthology.orchestra.run.smithi031.stderr:2 2024-08-19T14:23:20.639 INFO:teuthology.orchestra.run.smithi031.stderr:abort 2024-08-19T14:23:20.639 INFO:teuthology.orchestra.run.smithi031.stderr:assert 2024-08-19T14:23:20.639 INFO:teuthology.orchestra.run.smithi031.stderr:bench [] [] [] [] 2024-08-19T14:23:20.639 INFO:teuthology.orchestra.run.smithi031.stderr:bluefs debug_inject_read_zeros 2024-08-19T14:23:20.639 INFO:teuthology.orchestra.run.smithi031.stderr:bluefs files list 2024-08-19T14:23:20.639 INFO:teuthology.orchestra.run.smithi031.stderr:bluefs stats 2024-08-19T14:23:20.639 INFO:teuthology.orchestra.run.smithi031.stderr:bluestore allocator dump block 2024-08-19T14:23:20.639 INFO:teuthology.orchestra.run.smithi031.stderr:Error EINVAL: invalid command 2024-08-19T14:23:20.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:20.650 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:20.651 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:23:20.675 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:23:20.751 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:23:20.801 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:20.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:20.802 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:23:20.903 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:23:20.924 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is unset 2024-08-19T14:23:20.964 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:20.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:20.965 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:23:21.001 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:20 smithi031 ceph-mon[16336]: osdmap e393: 12 total, 12 up, 9 in 2024-08-19T14:23:21.001 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:20 smithi031 ceph-mon[16336]: pgmap v896: 47 pgs: 47 active+clean; 4.0 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 943 op/s 2024-08-19T14:23:21.001 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:20 smithi031 ceph-mon[16336]: 4.c scrub ok 2024-08-19T14:23:21.001 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:20 smithi031 ceph-mon[16336]: 1.18 scrub starts 2024-08-19T14:23:21.002 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:20 smithi031 ceph-mon[16336]: 1.18 scrub ok 2024-08-19T14:23:21.002 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:20 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/482054557' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:23:21.002 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:20 smithi031 ceph-mon[16336]: 1.1d scrub starts 2024-08-19T14:23:21.002 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:20 smithi031 ceph-mon[16336]: 1.1d scrub ok 2024-08-19T14:23:21.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:20 smithi032 ceph-mon[20004]: osdmap e393: 12 total, 12 up, 9 in 2024-08-19T14:23:21.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:20 smithi032 ceph-mon[20004]: pgmap v896: 47 pgs: 47 active+clean; 4.0 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 943 op/s 2024-08-19T14:23:21.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:20 smithi032 ceph-mon[20004]: 4.c scrub ok 2024-08-19T14:23:21.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:20 smithi032 ceph-mon[20004]: 1.18 scrub starts 2024-08-19T14:23:21.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:20 smithi032 ceph-mon[20004]: 1.18 scrub ok 2024-08-19T14:23:21.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:20 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/482054557' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:23:21.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:20 smithi032 ceph-mon[20004]: 1.1d scrub starts 2024-08-19T14:23:21.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:20 smithi032 ceph-mon[20004]: 1.1d scrub ok 2024-08-19T14:23:21.066 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:23:21.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:21.101 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:21.102 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:23:21.185 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-08-19T14:23:21.202 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:23:21.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:21.267 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:21.267 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:23:21.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:20 smithi045 ceph-mon[20143]: osdmap e393: 12 total, 12 up, 9 in 2024-08-19T14:23:21.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:20 smithi045 ceph-mon[20143]: pgmap v896: 47 pgs: 47 active+clean; 4.0 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 943 op/s 2024-08-19T14:23:21.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:20 smithi045 ceph-mon[20143]: 4.c scrub ok 2024-08-19T14:23:21.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:20 smithi045 ceph-mon[20143]: 1.18 scrub starts 2024-08-19T14:23:21.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:20 smithi045 ceph-mon[20143]: 1.18 scrub ok 2024-08-19T14:23:21.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:20 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/482054557' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:23:21.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:20 smithi045 ceph-mon[20143]: 1.1d scrub starts 2024-08-19T14:23:21.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:20 smithi045 ceph-mon[20143]: 1.1d scrub ok 2024-08-19T14:23:21.367 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:23:21.410 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:21.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:21.411 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:23:21.461 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-19T14:23:21.462 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-19T14:23:21.462 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph tell osd.1 'trim stale osdmaps' 2024-08-19T14:23:21.511 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:23:21.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:21.555 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:21.555 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:23:21.587 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:23:21.656 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:23:21.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:21.703 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:21.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:23:21.804 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:23:21.829 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:21.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:21.830 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:23:21.931 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:23:21.961 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:21.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:21.962 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:23:22.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:21 smithi032 ceph-mon[20004]: 1.17 scrub starts 2024-08-19T14:23:22.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:21 smithi032 ceph-mon[20004]: 1.17 scrub ok 2024-08-19T14:23:22.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:21 smithi032 ceph-mon[20004]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:23:22.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:21 smithi032 ceph-mon[20004]: Cluster is now healthy 2024-08-19T14:23:22.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:21 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/482054557' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:23:22.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:21 smithi032 ceph-mon[20004]: osdmap e394: 12 total, 12 up, 9 in 2024-08-19T14:23:22.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:21 smithi032 ceph-mon[20004]: 1.16 scrub starts 2024-08-19T14:23:22.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:21 smithi032 ceph-mon[20004]: 1.16 scrub ok 2024-08-19T14:23:22.062 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:23:22.094 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:22.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:22.095 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:23:22.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:21 smithi031 ceph-mon[16336]: 1.17 scrub starts 2024-08-19T14:23:22.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:21 smithi031 ceph-mon[16336]: 1.17 scrub ok 2024-08-19T14:23:22.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:21 smithi031 ceph-mon[16336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:23:22.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:21 smithi031 ceph-mon[16336]: Cluster is now healthy 2024-08-19T14:23:22.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:21 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/482054557' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:23:22.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:21 smithi031 ceph-mon[16336]: osdmap e394: 12 total, 12 up, 9 in 2024-08-19T14:23:22.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:21 smithi031 ceph-mon[16336]: 1.16 scrub starts 2024-08-19T14:23:22.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:21 smithi031 ceph-mon[16336]: 1.16 scrub ok 2024-08-19T14:23:22.172 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:23:22.172 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-08-19T14:23:22.172 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-08-19T14:23:22.172 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-08-19T14:23:22.172 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:23:22.196 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:23:22.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:22.233 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:22.233 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:23:22.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:21 smithi045 ceph-mon[20143]: 1.17 scrub starts 2024-08-19T14:23:22.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:21 smithi045 ceph-mon[20143]: 1.17 scrub ok 2024-08-19T14:23:22.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:21 smithi045 ceph-mon[20143]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:23:22.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:21 smithi045 ceph-mon[20143]: Cluster is now healthy 2024-08-19T14:23:22.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:21 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/482054557' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:23:22.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:21 smithi045 ceph-mon[20143]: osdmap e394: 12 total, 12 up, 9 in 2024-08-19T14:23:22.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:21 smithi045 ceph-mon[20143]: 1.16 scrub starts 2024-08-19T14:23:22.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:21 smithi045 ceph-mon[20143]: 1.16 scrub ok 2024-08-19T14:23:22.333 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:23:22.372 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:22.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:22.373 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:23:22.475 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:23:22.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:22.506 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:22.506 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:23:22.608 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:23:22.633 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:22.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:22.634 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:23:22.735 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:23:22.759 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:22.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:22.760 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:23:22.862 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:23:22.889 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:22.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:22.890 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:23:22.930 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-08-19T14:23:22.990 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:23:23.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:22 smithi032 ceph-mon[20004]: 4.3 scrub starts 2024-08-19T14:23:23.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:22 smithi032 ceph-mon[20004]: 4.b scrub starts 2024-08-19T14:23:23.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:22 smithi032 ceph-mon[20004]: pgmap v898: 47 pgs: 47 active+clean; 3.9 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 933 op/s 2024-08-19T14:23:23.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:22 smithi032 ceph-mon[20004]: 1.0 scrub starts 2024-08-19T14:23:23.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:22 smithi032 ceph-mon[20004]: 1.0 scrub ok 2024-08-19T14:23:23.016 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:23.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:23.018 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:23:23.119 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:23:23.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:23.150 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:23.150 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:23:23.251 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:23:23.274 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:23:23.279 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:23.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:23.280 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:23:23.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:22 smithi045 ceph-mon[20143]: 4.3 scrub starts 2024-08-19T14:23:23.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:22 smithi045 ceph-mon[20143]: 4.b scrub starts 2024-08-19T14:23:23.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:22 smithi045 ceph-mon[20143]: pgmap v898: 47 pgs: 47 active+clean; 3.9 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 933 op/s 2024-08-19T14:23:23.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:22 smithi045 ceph-mon[20143]: 1.0 scrub starts 2024-08-19T14:23:23.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:22 smithi045 ceph-mon[20143]: 1.0 scrub ok 2024-08-19T14:23:23.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:22 smithi031 ceph-mon[16336]: 4.3 scrub starts 2024-08-19T14:23:23.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:22 smithi031 ceph-mon[16336]: 4.b scrub starts 2024-08-19T14:23:23.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:22 smithi031 ceph-mon[16336]: pgmap v898: 47 pgs: 47 active+clean; 3.9 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 933 op/s 2024-08-19T14:23:23.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:22 smithi031 ceph-mon[16336]: 1.0 scrub starts 2024-08-19T14:23:23.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:22 smithi031 ceph-mon[16336]: 1.0 scrub ok 2024-08-19T14:23:23.381 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:23:23.415 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:23.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:23.416 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:23:23.517 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:23:23.544 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:23.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:23.545 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:23:23.646 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:23:23.667 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:23.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:23.668 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:23:23.768 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:23:23.790 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:23.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:23.791 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:23:23.892 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:23:23.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:23.915 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:23.915 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:23:23.962 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:23:23.962 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-08-19T14:23:23.962 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-08-19T14:23:23.962 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-08-19T14:23:23.962 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:23.962 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:115128 2.0 2:08e033d8:::benchmark_data_smithi045_46415_object115127:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-08-19T14:23:23.962 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:14:09.964413+0000", 2024-08-19T14:23:23.963 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 553.99563640700001, 2024-08-19T14:23:23.963 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.15347317799999999, 2024-08-19T14:23:23.963 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:23:23.963 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:23:23.963 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:23:23.963 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:23:23.963 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:23:23.963 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:23:23.963 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 115128 2024-08-19T14:23:23.963 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:23.963 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:23:23.963 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:23.963 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:23:23.963 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:09.964413+0000", 2024-08-19T14:23:23.964 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:23.964 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:23.964 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:23.964 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:23:23.964 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:09.964413+0000", 2024-08-19T14:23:23.964 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:23.964 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:23.964 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:23.964 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:23:23.964 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:09.964414+0000", 2024-08-19T14:23:23.964 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.689e-06 2024-08-19T14:23:23.964 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:23.964 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:23.964 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:23:23.965 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:09.964452+0000", 2024-08-19T14:23:23.965 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7565999999999999e-05 2024-08-19T14:23:23.965 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:23.965 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:23.965 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:23:23.965 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:09.964467+0000", 2024-08-19T14:23:23.965 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5160999999999999e-05 2024-08-19T14:23:23.965 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:23.965 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:23.965 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:23:23.965 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:09.964475+0000", 2024-08-19T14:23:23.965 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.276e-06 2024-08-19T14:23:23.965 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:23.965 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:23.966 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:23:23.966 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:09.964534+0000", 2024-08-19T14:23:23.966 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8727999999999997e-05 2024-08-19T14:23:23.966 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:23.966 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:23.966 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:23:23.966 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:09.964675+0000", 2024-08-19T14:23:23.966 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000140705 2024-08-19T14:23:23.966 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:23.966 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:23.966 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 9,11", 2024-08-19T14:23:23.966 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:09.964830+0000", 2024-08-19T14:23:23.966 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00015500900000000001 2024-08-19T14:23:23.967 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:23.967 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:23.967 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:23:23.967 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:09.967628+0000", 2024-08-19T14:23:23.967 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0027980909999999999 2024-08-19T14:23:23.967 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:23.967 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:23.967 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:23:23.967 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:09.968335+0000", 2024-08-19T14:23:23.967 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00070710399999999996 2024-08-19T14:23:23.967 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:23.967 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:23.967 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:23:23.968 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:10.117809+0000", 2024-08-19T14:23:23.968 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.149473572 2024-08-19T14:23:23.968 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:23.968 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:23.968 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:23:23.968 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:10.117849+0000", 2024-08-19T14:23:23.968 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0431e-05 2024-08-19T14:23:23.968 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:23.968 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:23.968 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:23:23.968 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:10.117886+0000", 2024-08-19T14:23:23.968 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6845999999999999e-05 2024-08-19T14:23:23.968 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:23.968 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:23:23.969 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:23.969 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:23.969 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:23.969 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:115136 2.0 2:02df7cb0:::benchmark_data_smithi045_46415_object115135:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-08-19T14:23:23.969 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:14:09.971356+0000", 2024-08-19T14:23:23.969 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 553.98869291699998, 2024-08-19T14:23:23.969 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.15309395200000001, 2024-08-19T14:23:23.969 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:23:23.969 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:23:23.969 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:23:23.969 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:23:23.969 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:23:23.969 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:23:23.970 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 115136 2024-08-19T14:23:23.970 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:23.970 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:23:23.970 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:23.970 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:23:23.970 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:09.971356+0000", 2024-08-19T14:23:23.970 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:23.970 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:23.970 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:23.970 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:23:23.970 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:09.971356+0000", 2024-08-19T14:23:23.970 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:23.970 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:23.971 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:23.971 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:23:23.971 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:09.971357+0000", 2024-08-19T14:23:23.971 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.172e-06 2024-08-19T14:23:23.971 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:23.971 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:23.971 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:23:23.971 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:09.971398+0000", 2024-08-19T14:23:23.971 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0955999999999999e-05 2024-08-19T14:23:23.971 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:23.971 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:23.971 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:23:23.971 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:09.971410+0000", 2024-08-19T14:23:23.971 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.135e-05 2024-08-19T14:23:23.972 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:23.972 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:23.972 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:23:23.972 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:09.971415+0000", 2024-08-19T14:23:23.972 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1819999999999997e-06 2024-08-19T14:23:23.972 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:23.972 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:23.972 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:23:23.972 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:09.971457+0000", 2024-08-19T14:23:23.972 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1972999999999999e-05 2024-08-19T14:23:23.972 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:23.972 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:23.972 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:23:23.972 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:09.971584+0000", 2024-08-19T14:23:23.973 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012695800000000001 2024-08-19T14:23:23.973 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:23.973 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:23.973 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 9,11", 2024-08-19T14:23:23.973 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:09.971773+0000", 2024-08-19T14:23:23.973 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000189101 2024-08-19T14:23:23.973 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:23.973 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:23.973 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:23:23.973 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:09.975215+0000", 2024-08-19T14:23:23.973 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.003441871 2024-08-19T14:23:23.973 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:23.973 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:23.973 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:23:23.974 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:10.123256+0000", 2024-08-19T14:23:23.974 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.14804097999999999 2024-08-19T14:23:23.974 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:23.974 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:23.974 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:23:23.974 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:10.124383+0000", 2024-08-19T14:23:23.974 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00112725 2024-08-19T14:23:23.974 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:23.974 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:23.974 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:23:23.974 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:10.124421+0000", 2024-08-19T14:23:23.974 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8031999999999998e-05 2024-08-19T14:23:23.974 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:23.974 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:23.975 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:23:23.975 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:14:10.124450+0000", 2024-08-19T14:23:23.975 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9127000000000001e-05 2024-08-19T14:23:23.975 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:23.975 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:23:23.975 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:23.975 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:23.975 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:23.975 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:171517 2.8 2:17a35dc4:::benchmark_data_smithi045_46415_object48236:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e169)", 2024-08-19T14:23:23.975 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:15:12.905580+0000", 2024-08-19T14:23:23.975 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 491.054469568, 2024-08-19T14:23:23.975 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.133369984, 2024-08-19T14:23:23.975 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:23:23.975 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:23:23.976 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:23:23.976 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:23:23.976 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:23:23.976 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:23:23.976 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 171517 2024-08-19T14:23:23.976 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:23.976 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:23:23.976 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:23.976 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:23:23.976 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.905580+0000", 2024-08-19T14:23:23.976 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:23.976 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:23.976 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:23.977 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:23:23.977 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.905580+0000", 2024-08-19T14:23:23.977 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:23.977 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:23.977 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:23.977 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:23:23.977 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.905580+0000", 2024-08-19T14:23:23.977 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6199999999999997e-07 2024-08-19T14:23:23.977 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:23.977 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:23.977 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:23:23.977 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.905583+0000", 2024-08-19T14:23:23.977 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2789999999999999e-06 2024-08-19T14:23:23.978 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:23.978 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:23.978 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:23:23.978 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:13.004022+0000", 2024-08-19T14:23:23.978 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.098438756000000002 2024-08-19T14:23:23.978 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:23.978 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:23.978 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:23:23.978 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:13.004038+0000", 2024-08-19T14:23:23.978 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6038000000000001e-05 2024-08-19T14:23:23.978 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:23.978 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:23.978 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:23:23.978 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:13.018509+0000", 2024-08-19T14:23:23.979 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.014470642000000001 2024-08-19T14:23:23.979 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:23.979 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:23.979 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:23:23.979 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:13.018652+0000", 2024-08-19T14:23:23.979 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000143048 2024-08-19T14:23:23.979 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:23.979 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:23.979 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0,9", 2024-08-19T14:23:23.979 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:13.018802+0000", 2024-08-19T14:23:23.979 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000149748 2024-08-19T14:23:23.979 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:23.979 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:23.979 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:23:23.979 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:13.023512+0000", 2024-08-19T14:23:23.980 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0047101729999999998 2024-08-19T14:23:23.980 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:23.980 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:23.980 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:23:23.980 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:13.028445+0000", 2024-08-19T14:23:23.980 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0049330479999999998 2024-08-19T14:23:23.980 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:23.980 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:23.980 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:23:23.980 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:13.038830+0000", 2024-08-19T14:23:23.980 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010384882999999999 2024-08-19T14:23:23.980 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:23.980 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:23.981 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:23:23.981 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:13.038884+0000", 2024-08-19T14:23:23.981 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4664999999999999e-05 2024-08-19T14:23:23.981 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:23.981 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:23.981 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:23:23.981 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:13.038950+0000", 2024-08-19T14:23:23.981 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.5042000000000005e-05 2024-08-19T14:23:23.981 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:23.981 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:23:23.981 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:23.981 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:23.981 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:23.981 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25057.0:50181 3.13 3.d14b4293 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-08-19T14:23:23.982 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:17:20.752507+0000", 2024-08-19T14:23:23.982 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 363.20754167299998, 2024-08-19T14:23:23.982 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.99194691599999996, 2024-08-19T14:23:23.982 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:23:23.982 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:23:23.982 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:23:23.982 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:23:23.982 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25057", 2024-08-19T14:23:23.982 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:23:23.982 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 50181 2024-08-19T14:23:23.982 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:23.982 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:23:23.982 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:23.982 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:23:23.983 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:20.752507+0000", 2024-08-19T14:23:23.983 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:23.983 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:23.983 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:23.983 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:23:23.983 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:20.752507+0000", 2024-08-19T14:23:23.983 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:23.983 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:23.983 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:23.983 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:23:23.983 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:20.752509+0000", 2024-08-19T14:23:23.983 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.358e-06 2024-08-19T14:23:23.983 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:23.984 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:23.984 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:23:23.984 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:20.752560+0000", 2024-08-19T14:23:23.984 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1412999999999997e-05 2024-08-19T14:23:23.984 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:23.984 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:23.984 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:23:23.984 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:20.752574+0000", 2024-08-19T14:23:23.984 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3804e-05 2024-08-19T14:23:23.984 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:23.984 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:23.984 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:23:23.984 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:20.752581+0000", 2024-08-19T14:23:23.984 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9530000000000002e-06 2024-08-19T14:23:23.985 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:23.985 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:23.985 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:23:23.985 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:20.752627+0000", 2024-08-19T14:23:23.985 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5701000000000002e-05 2024-08-19T14:23:23.985 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:23.985 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:23.985 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:23:23.985 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:20.752631+0000", 2024-08-19T14:23:23.985 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7979999999999996e-06 2024-08-19T14:23:23.985 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:23.985 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:23.985 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:23:23.985 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:21.744442+0000", 2024-08-19T14:23:23.986 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.99181026900000002 2024-08-19T14:23:23.986 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:23.986 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:23.986 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:23:23.986 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:21.744454+0000", 2024-08-19T14:23:23.986 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2619999999999999e-05 2024-08-19T14:23:23.986 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:23.986 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:23:23.986 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:23.986 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:23.986 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:23.986 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25057.0:63326 3.1d 3:bcf760da:::benchmark_data_smithi045_59973_object63325:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e244)", 2024-08-19T14:23:23.986 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:17:33.888941+0000", 2024-08-19T14:23:23.987 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 350.07110768799998, 2024-08-19T14:23:23.987 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0967315230000001, 2024-08-19T14:23:23.987 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:23:23.987 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:23:23.987 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:23:23.987 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:23:23.987 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25057", 2024-08-19T14:23:23.987 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:23:23.987 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 63326 2024-08-19T14:23:23.987 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:23.987 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:23:23.987 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:23.987 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:23:23.987 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:33.888941+0000", 2024-08-19T14:23:23.988 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:23.988 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:23.988 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:23.988 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:23:23.988 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:33.888941+0000", 2024-08-19T14:23:23.988 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:23.988 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:23.988 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:23.988 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:23:23.988 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:33.888943+0000", 2024-08-19T14:23:23.988 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9539999999999998e-06 2024-08-19T14:23:23.988 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:23.988 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:23.988 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:23:23.989 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:33.888990+0000", 2024-08-19T14:23:23.989 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6193000000000003e-05 2024-08-19T14:23:23.989 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:23.989 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:23.989 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:23:23.989 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:33.889005+0000", 2024-08-19T14:23:23.989 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5722999999999998e-05 2024-08-19T14:23:23.989 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:23.989 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:23.989 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:23:23.989 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:33.889013+0000", 2024-08-19T14:23:23.989 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3860000000000001e-06 2024-08-19T14:23:23.989 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:23.989 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:23.989 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:23:23.990 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:33.984609+0000", 2024-08-19T14:23:23.990 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.095596212999999999 2024-08-19T14:23:23.990 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:23.990 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:23.990 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:23:23.990 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:33.984618+0000", 2024-08-19T14:23:23.990 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.5079999999999994e-06 2024-08-19T14:23:23.990 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:23.991 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:23.991 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:23:23.991 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:34.985599+0000", 2024-08-19T14:23:23.991 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0009801739999999 2024-08-19T14:23:23.991 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:23.991 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:23.991 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:23:23.991 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:34.985673+0000", 2024-08-19T14:23:23.991 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4371999999999999e-05 2024-08-19T14:23:23.991 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:23.991 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:23:23.991 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:23.991 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:23.991 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:23.992 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25057.0:130072 3.f 3:f7f93fc4:::benchmark_data_smithi045_59973_object130071:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e281)", 2024-08-19T14:23:23.992 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:18:31.609517+0000", 2024-08-19T14:23:23.992 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 292.350532499, 2024-08-19T14:23:23.992 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.21221680200000001, 2024-08-19T14:23:23.992 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:23:23.992 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:23:23.992 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:23:23.992 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:23:23.992 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25057", 2024-08-19T14:23:23.992 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:23:23.992 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 130072 2024-08-19T14:23:23.992 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:23.992 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:23:23.993 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:23.993 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:23:23.993 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.609517+0000", 2024-08-19T14:23:23.993 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:23.993 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:23.993 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:23.993 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:23:23.993 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.609517+0000", 2024-08-19T14:23:23.993 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:23.993 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:23.993 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:23.993 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:23:23.993 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.609517+0000", 2024-08-19T14:23:23.994 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.7899999999999997e-07 2024-08-19T14:23:23.994 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:23.994 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:23.994 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:23:23.994 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.609574+0000", 2024-08-19T14:23:23.994 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6422000000000003e-05 2024-08-19T14:23:23.994 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:23.994 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:23.994 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:23:23.994 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.609588+0000", 2024-08-19T14:23:23.994 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.403e-05 2024-08-19T14:23:23.994 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:23.994 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:23.994 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:23:23.995 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.609595+0000", 2024-08-19T14:23:23.995 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8970000000000002e-06 2024-08-19T14:23:23.995 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:23.995 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:23.995 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:23:23.995 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.610985+0000", 2024-08-19T14:23:23.995 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0013905949999999999 2024-08-19T14:23:23.995 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:23.995 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:23.995 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:23:23.995 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.611111+0000", 2024-08-19T14:23:23.995 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012530700000000001 2024-08-19T14:23:23.995 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:23.995 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:23.996 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 8,11", 2024-08-19T14:23:23.996 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.611234+0000", 2024-08-19T14:23:23.996 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012355100000000001 2024-08-19T14:23:23.996 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:23.996 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:23.996 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:23:23.996 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.616670+0000", 2024-08-19T14:23:23.996 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0054355339999999997 2024-08-19T14:23:23.996 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:23.996 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:23.996 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:23:23.996 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.616703+0000", 2024-08-19T14:23:23.996 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3281999999999997e-05 2024-08-19T14:23:23.996 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:23.997 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:23.997 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:23:23.997 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.821464+0000", 2024-08-19T14:23:23.997 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20476105 2024-08-19T14:23:23.997 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:23.997 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:23.997 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:23:23.997 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.821687+0000", 2024-08-19T14:23:23.997 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000223241 2024-08-19T14:23:23.997 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:23.997 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:23.997 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:23:23.997 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.821733+0000", 2024-08-19T14:23:23.997 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6013999999999997e-05 2024-08-19T14:23:23.998 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:23.998 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:23:23.998 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:23.998 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:23.998 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:23.998 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25057.0:130075 3.f 3:f32a0229:::benchmark_data_smithi045_59973_object130074:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e281)", 2024-08-19T14:23:23.998 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:18:31.611127+0000", 2024-08-19T14:23:23.998 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 292.34892225900001, 2024-08-19T14:23:23.998 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.21073734099999999, 2024-08-19T14:23:23.998 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:23:23.998 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:23:23.998 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:23:23.998 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:23:23.999 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25057", 2024-08-19T14:23:23.999 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:23:23.999 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 130075 2024-08-19T14:23:23.999 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:23.999 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:23:23.999 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:23.999 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:23:23.999 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.611127+0000", 2024-08-19T14:23:23.999 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:23.999 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:23.999 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:23.999 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:23:23.999 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.611127+0000", 2024-08-19T14:23:23.999 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:24.000 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.000 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.000 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:23:24.000 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.611128+0000", 2024-08-19T14:23:24.000 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0640000000000001e-06 2024-08-19T14:23:24.000 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.000 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.000 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:23:24.000 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.611271+0000", 2024-08-19T14:23:24.000 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000143027 2024-08-19T14:23:24.000 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.000 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.000 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:23:24.000 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.611284+0000", 2024-08-19T14:23:24.001 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2979000000000001e-05 2024-08-19T14:23:24.001 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.001 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.001 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:23:24.001 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.611291+0000", 2024-08-19T14:23:24.001 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2810000000000003e-06 2024-08-19T14:23:24.001 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.001 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.001 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:23:24.001 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.614035+0000", 2024-08-19T14:23:24.001 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0027435020000000001 2024-08-19T14:23:24.001 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.001 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.002 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:23:24.002 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.614198+0000", 2024-08-19T14:23:24.002 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00016282100000000001 2024-08-19T14:23:24.002 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.002 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.002 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 8,11", 2024-08-19T14:23:24.002 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.614337+0000", 2024-08-19T14:23:24.002 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000139266 2024-08-19T14:23:24.002 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.002 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.002 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:23:24.002 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.617108+0000", 2024-08-19T14:23:24.002 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0027711960000000001 2024-08-19T14:23:24.002 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.003 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.003 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:23:24.003 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.617630+0000", 2024-08-19T14:23:24.003 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00052161000000000004 2024-08-19T14:23:24.003 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.003 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.003 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:23:24.003 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.821780+0000", 2024-08-19T14:23:24.003 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.204150741 2024-08-19T14:23:24.003 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.003 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.003 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:23:24.003 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.821812+0000", 2024-08-19T14:23:24.003 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1785999999999997e-05 2024-08-19T14:23:24.004 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.004 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.004 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:23:24.004 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.821864+0000", 2024-08-19T14:23:24.004 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:23 smithi032 ceph-mon[20004]: 4.3 scrub ok 2024-08-19T14:23:24.004 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:23 smithi032 ceph-mon[20004]: 1.6 scrub starts 2024-08-19T14:23:24.004 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:23 smithi032 ceph-mon[20004]: 1.6 scrub ok 2024-08-19T14:23:24.005 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2067999999999997e-05 2024-08-19T14:23:24.005 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:24.005 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:23:24.005 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:24.006 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.006 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.006 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25057.0:130097 3.f 3:f04c1e35:::benchmark_data_smithi045_59973_object130096:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e281)", 2024-08-19T14:23:24.006 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:18:31.621483+0000", 2024-08-19T14:23:24.006 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 292.338566582, 2024-08-19T14:23:24.006 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.200549171, 2024-08-19T14:23:24.006 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:23:24.006 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:23:24.006 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:23:24.006 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:23:24.006 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25057", 2024-08-19T14:23:24.006 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:23:24.006 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 130097 2024-08-19T14:23:24.007 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.007 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:23:24.007 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.007 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:23:24.007 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.621483+0000", 2024-08-19T14:23:24.007 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:24.007 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.007 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.007 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:23:24.007 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.621483+0000", 2024-08-19T14:23:24.007 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:24.007 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.007 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.008 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:23:24.008 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.621484+0000", 2024-08-19T14:23:24.008 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3200000000000001e-06 2024-08-19T14:23:24.008 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.008 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.008 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:23:24.008 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.621516+0000", 2024-08-19T14:23:24.008 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2369000000000001e-05 2024-08-19T14:23:24.008 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.008 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.008 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:23:24.008 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.621533+0000", 2024-08-19T14:23:24.008 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6511000000000002e-05 2024-08-19T14:23:24.009 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.009 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.009 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:23:24.009 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.621540+0000", 2024-08-19T14:23:24.009 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4580000000000004e-06 2024-08-19T14:23:24.009 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.009 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.009 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:23:24.009 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.621617+0000", 2024-08-19T14:23:24.009 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6584999999999999e-05 2024-08-19T14:23:24.009 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.009 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.009 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:23:24.009 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.621760+0000", 2024-08-19T14:23:24.010 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014331399999999999 2024-08-19T14:23:24.010 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.010 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.010 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 8,11", 2024-08-19T14:23:24.010 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.621903+0000", 2024-08-19T14:23:24.010 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000142927 2024-08-19T14:23:24.010 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.010 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.010 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:23:24.010 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.626175+0000", 2024-08-19T14:23:24.010 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0042722639999999996 2024-08-19T14:23:24.010 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.011 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.011 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:23:24.011 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.626353+0000", 2024-08-19T14:23:24.011 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000178152 2024-08-19T14:23:24.011 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.011 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.011 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:23:24.011 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.821907+0000", 2024-08-19T14:23:24.011 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.195553372 2024-08-19T14:23:24.011 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.011 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.011 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:23:24.011 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.821996+0000", 2024-08-19T14:23:24.011 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.8832000000000006e-05 2024-08-19T14:23:24.012 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.012 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.012 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:23:24.012 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.822032+0000", 2024-08-19T14:23:24.012 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6066999999999999e-05 2024-08-19T14:23:24.012 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:24.012 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:23:24.012 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:24.012 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.012 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.012 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25057.0:130120 3.f 3:f699ea53:::benchmark_data_smithi045_59973_object130119:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e281)", 2024-08-19T14:23:24.012 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:18:31.629426+0000", 2024-08-19T14:23:24.012 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 292.330623285, 2024-08-19T14:23:24.013 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.19275684200000001, 2024-08-19T14:23:24.013 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:23:24.013 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:23:24.013 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:23:24.013 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:23:24.013 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25057", 2024-08-19T14:23:24.013 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:23:24.013 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 130120 2024-08-19T14:23:24.013 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.013 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:23:24.013 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.013 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:23:24.013 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.629426+0000", 2024-08-19T14:23:24.014 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:24.014 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.014 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.014 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:23:24.014 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.629426+0000", 2024-08-19T14:23:24.014 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:24.014 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.014 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.014 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:23:24.014 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.629427+0000", 2024-08-19T14:23:24.014 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.274e-06 2024-08-19T14:23:24.014 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.014 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.014 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:23:24.015 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.629476+0000", 2024-08-19T14:23:24.015 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8590000000000001e-05 2024-08-19T14:23:24.015 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.015 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.015 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:23:24.015 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.629487+0000", 2024-08-19T14:23:24.015 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1508999999999999e-05 2024-08-19T14:23:24.015 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.015 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.015 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:23:24.015 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.629493+0000", 2024-08-19T14:23:24.015 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.677e-06 2024-08-19T14:23:24.015 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.016 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.016 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:23:24.016 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.629543+0000", 2024-08-19T14:23:24.016 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0630000000000001e-05 2024-08-19T14:23:24.016 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.016 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.016 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:23:24.016 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.629643+0000", 2024-08-19T14:23:24.016 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.9978999999999999e-05 2024-08-19T14:23:24.016 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.016 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.016 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 8,11", 2024-08-19T14:23:24.016 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.629728+0000", 2024-08-19T14:23:24.016 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.4127000000000003e-05 2024-08-19T14:23:24.017 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.017 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.017 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:23:24.017 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.631709+0000", 2024-08-19T14:23:24.017 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0019811009999999999 2024-08-19T14:23:24.017 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.017 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.017 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:23:24.017 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.632189+0000", 2024-08-19T14:23:24.017 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00047994299999999998 2024-08-19T14:23:24.017 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.017 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.017 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:23:24.018 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.822073+0000", 2024-08-19T14:23:24.018 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.18988412800000001 2024-08-19T14:23:24.018 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.018 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.018 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:23:24.018 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.822143+0000", 2024-08-19T14:23:24.018 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0588999999999998e-05 2024-08-19T14:23:24.018 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.018 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.018 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:23:24.018 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.822183+0000", 2024-08-19T14:23:24.018 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9295000000000002e-05 2024-08-19T14:23:24.018 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:24.019 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:23:24.019 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:24.019 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.019 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.019 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25057.0:130174 3.f 3:f33afdbe:::benchmark_data_smithi045_59973_object130173:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e281)", 2024-08-19T14:23:24.019 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:18:31.660169+0000", 2024-08-19T14:23:24.019 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 292.29987987300001, 2024-08-19T14:23:24.019 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.164501495, 2024-08-19T14:23:24.019 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:23:24.019 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:23:24.019 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:23:24.019 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:23:24.020 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25057", 2024-08-19T14:23:24.020 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:23:24.020 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 130174 2024-08-19T14:23:24.020 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.020 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:23:24.020 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.020 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:23:24.020 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.660169+0000", 2024-08-19T14:23:24.020 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:24.020 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.020 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.020 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:23:24.020 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.660169+0000", 2024-08-19T14:23:24.020 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:24.021 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.021 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.021 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:23:24.021 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.660170+0000", 2024-08-19T14:23:24.021 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.119e-06 2024-08-19T14:23:24.021 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.021 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.021 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:23:24.021 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.660216+0000", 2024-08-19T14:23:24.021 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5192999999999999e-05 2024-08-19T14:23:24.021 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.021 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.021 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:23:24.022 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.660227+0000", 2024-08-19T14:23:24.022 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1496e-05 2024-08-19T14:23:24.022 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.022 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.022 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:23:24.022 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.660232+0000", 2024-08-19T14:23:24.022 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1919999999999996e-06 2024-08-19T14:23:24.022 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.022 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.022 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:23:24.022 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.660278+0000", 2024-08-19T14:23:24.022 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.566e-05 2024-08-19T14:23:24.022 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.022 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.023 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:23:24.023 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.660368+0000", 2024-08-19T14:23:24.023 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.0274999999999999e-05 2024-08-19T14:23:24.023 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.023 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.023 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 8,11", 2024-08-19T14:23:24.023 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.660458+0000", 2024-08-19T14:23:24.023 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.9733000000000003e-05 2024-08-19T14:23:24.023 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.023 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.023 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:23:24.023 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.662221+0000", 2024-08-19T14:23:24.023 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0017634409999999999 2024-08-19T14:23:24.024 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.024 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.024 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:23:24.024 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.662766+0000", 2024-08-19T14:23:24.024 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00054489399999999998 2024-08-19T14:23:24.024 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.024 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.024 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:23:24.024 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.822238+0000", 2024-08-19T14:23:24.024 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.159471785 2024-08-19T14:23:24.024 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.024 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.024 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:23:24.024 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.822286+0000", 2024-08-19T14:23:24.025 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7633999999999998e-05 2024-08-19T14:23:24.025 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.025 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.025 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:23:24.025 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.824671+0000", 2024-08-19T14:23:24.025 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0023850730000000001 2024-08-19T14:23:24.025 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:24.025 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:23:24.025 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:24.025 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.025 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.025 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.25333.0:199905 4.8 e394/335 4:1a3c705b:::benchmark_data_smithi045_71562_object68730:head v 394'12448, mlcod=394'12446)", 2024-08-19T14:23:24.025 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:23:21.239568+0000", 2024-08-19T14:23:24.025 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 2.72048142, 2024-08-19T14:23:24.026 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.023507703000000001, 2024-08-19T14:23:24.026 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:23:24.026 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:23:24.026 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:23:24.026 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:23:24.026 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.026 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:23:24.026 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:21.239568+0000", 2024-08-19T14:23:24.026 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:24.026 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.026 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.026 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:23:24.026 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:21.239568+0000", 2024-08-19T14:23:24.027 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:24.027 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.027 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.027 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:23:24.027 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:21.239568+0000", 2024-08-19T14:23:24.027 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.2099999999999995e-07 2024-08-19T14:23:24.027 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.027 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.027 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:23:24.027 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:21.239575+0000", 2024-08-19T14:23:24.027 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.319e-06 2024-08-19T14:23:24.027 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.027 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.027 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:23:24.028 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:21.239588+0000", 2024-08-19T14:23:24.028 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3022e-05 2024-08-19T14:23:24.028 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.028 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.028 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:23:24.028 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:21.239595+0000", 2024-08-19T14:23:24.028 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.418e-06 2024-08-19T14:23:24.028 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.028 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.028 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:23:24.028 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:21.239748+0000", 2024-08-19T14:23:24.028 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000152392 2024-08-19T14:23:24.028 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.028 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.029 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:23:24.029 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:21.239797+0000", 2024-08-19T14:23:24.029 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9397999999999997e-05 2024-08-19T14:23:24.029 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.029 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.029 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:23:24.029 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:21.262869+0000", 2024-08-19T14:23:24.029 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.023071627000000001 2024-08-19T14:23:24.029 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.029 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.029 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:23:24.029 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:21.263075+0000", 2024-08-19T14:23:24.029 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000206706 2024-08-19T14:23:24.029 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:24.030 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:23:24.030 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:24.030 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.030 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.030 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.25333.0:200001 4.8 e394/335 4:11dc5e1f:::benchmark_data_smithi045_71562_object68826:head v 394'12453, mlcod=394'12450)", 2024-08-19T14:23:24.030 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:23:21.340721+0000", 2024-08-19T14:23:24.030 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 2.6193280369999998, 2024-08-19T14:23:24.030 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.026309081000000002, 2024-08-19T14:23:24.030 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:23:24.030 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:23:24.030 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:23:24.030 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:23:24.030 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.031 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:23:24.031 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:21.340721+0000", 2024-08-19T14:23:24.031 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:24.031 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.031 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.031 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:23:24.031 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:21.340721+0000", 2024-08-19T14:23:24.031 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:24.031 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.031 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.031 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:23:24.031 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:21.340722+0000", 2024-08-19T14:23:24.031 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9999999999999997e-07 2024-08-19T14:23:24.031 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.032 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.032 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:23:24.032 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:21.340728+0000", 2024-08-19T14:23:24.032 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0750000000000003e-06 2024-08-19T14:23:24.032 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.032 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.032 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:23:24.032 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:21.340742+0000", 2024-08-19T14:23:24.032 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4395e-05 2024-08-19T14:23:24.032 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.032 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.032 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:23:24.032 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:21.340748+0000", 2024-08-19T14:23:24.032 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6250000000000004e-06 2024-08-19T14:23:24.033 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.033 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.033 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:23:24.033 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:21.347043+0000", 2024-08-19T14:23:24.033 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0062948759999999996 2024-08-19T14:23:24.033 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.033 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.033 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:23:24.033 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:21.347061+0000", 2024-08-19T14:23:24.033 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8814999999999999e-05 2024-08-19T14:23:24.033 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.033 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.033 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:23:24.033 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:21.366994+0000", 2024-08-19T14:23:24.034 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.019933000999999999 2024-08-19T14:23:24.034 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.034 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.034 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:23:24.034 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:21.367030+0000", 2024-08-19T14:23:24.034 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5694000000000001e-05 2024-08-19T14:23:24.034 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:24.034 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:23:24.034 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:24.034 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.034 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.034 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.25333.0:200005 4.8 e394/335 4:15fca165:::benchmark_data_smithi045_71562_object68830:head v 394'12454, mlcod=394'12450)", 2024-08-19T14:23:24.034 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:23:21.344444+0000", 2024-08-19T14:23:24.035 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 2.6156054040000001, 2024-08-19T14:23:24.035 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.022620388000000002, 2024-08-19T14:23:24.035 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:23:24.035 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:23:24.035 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:23:24.035 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:23:24.035 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.035 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:23:24.035 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:21.344444+0000", 2024-08-19T14:23:24.035 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:24.035 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.035 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.035 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:23:24.035 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:21.344444+0000", 2024-08-19T14:23:24.036 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:24.036 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.036 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.036 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:23:24.036 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:21.344444+0000", 2024-08-19T14:23:24.036 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6000000000000003e-07 2024-08-19T14:23:24.036 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.036 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.036 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:23:24.036 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:21.344450+0000", 2024-08-19T14:23:24.036 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3290000000000001e-06 2024-08-19T14:23:24.036 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.036 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.036 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:23:24.037 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:21.344465+0000", 2024-08-19T14:23:24.037 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4885000000000001e-05 2024-08-19T14:23:24.037 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.037 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.037 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:23:24.037 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:21.344472+0000", 2024-08-19T14:23:24.037 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3710000000000002e-06 2024-08-19T14:23:24.037 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.037 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.037 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:23:24.037 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:21.352230+0000", 2024-08-19T14:23:24.037 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0077578159999999998 2024-08-19T14:23:24.037 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.038 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.038 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:23:24.038 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:21.352293+0000", 2024-08-19T14:23:24.038 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3627000000000006e-05 2024-08-19T14:23:24.038 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.038 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.038 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:23:24.038 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:21.367035+0000", 2024-08-19T14:23:24.038 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.01474109 2024-08-19T14:23:24.038 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.038 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.038 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:23:24.038 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:21.367064+0000", 2024-08-19T14:23:24.038 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.951e-05 2024-08-19T14:23:24.039 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:24.039 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:23:24.039 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:24.039 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.039 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.039 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.25333.0:200023 4.7 e394/335 4:edf974f9:::benchmark_data_smithi045_71562_object68848:head v 394'12559, mlcod=394'12556)", 2024-08-19T14:23:24.039 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:23:21.363109+0000", 2024-08-19T14:23:24.039 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 2.5969399859999998, 2024-08-19T14:23:24.039 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.030995016, 2024-08-19T14:23:24.039 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:23:24.039 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:23:24.039 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:23:24.039 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:23:24.039 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.040 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:23:24.040 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:21.363109+0000", 2024-08-19T14:23:24.040 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:24.040 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.040 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.040 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:23:24.040 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:21.363109+0000", 2024-08-19T14:23:24.040 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:24.040 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.040 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.040 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:23:24.040 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:21.363110+0000", 2024-08-19T14:23:24.040 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.3099999999999996e-07 2024-08-19T14:23:24.040 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.041 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.041 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:23:24.041 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:21.363116+0000", 2024-08-19T14:23:24.041 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6110000000000001e-06 2024-08-19T14:23:24.041 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.041 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.041 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:23:24.041 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:21.363125+0000", 2024-08-19T14:23:24.041 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3200000000000006e-06 2024-08-19T14:23:24.041 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.041 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.041 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:23:24.041 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:21.363131+0000", 2024-08-19T14:23:24.041 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3090000000000001e-06 2024-08-19T14:23:24.042 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.042 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.042 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:23:24.042 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:21.363186+0000", 2024-08-19T14:23:24.042 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5102999999999997e-05 2024-08-19T14:23:24.042 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.042 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.042 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:23:24.042 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:21.363203+0000", 2024-08-19T14:23:24.042 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7108999999999998e-05 2024-08-19T14:23:24.042 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.042 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.042 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:23:24.043 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:21.393653+0000", 2024-08-19T14:23:24.043 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.030449397 2024-08-19T14:23:24.043 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.043 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.043 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:23:24.043 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:21.394104+0000", 2024-08-19T14:23:24.043 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00045133599999999999 2024-08-19T14:23:24.043 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:24.043 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:23:24.043 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:24.043 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.043 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.043 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.25333.0:200036 4.7 e394/335 4:e46c656f:::benchmark_data_smithi045_71562_object68861:head v 394'12560, mlcod=394'12557)", 2024-08-19T14:23:24.043 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:23:21.377027+0000", 2024-08-19T14:23:24.044 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 2.583022454, 2024-08-19T14:23:24.044 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.026535385000000002, 2024-08-19T14:23:24.044 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:23:24.044 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:23:24.044 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:23:24.044 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:23:24.044 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.044 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:23:24.044 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:21.377027+0000", 2024-08-19T14:23:24.044 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:24.044 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.044 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.044 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:23:24.044 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:21.377027+0000", 2024-08-19T14:23:24.045 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:24.045 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.045 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.045 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:23:24.045 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:21.377027+0000", 2024-08-19T14:23:24.045 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0800000000000004e-07 2024-08-19T14:23:24.045 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.045 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.045 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:23:24.045 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:21.377033+0000", 2024-08-19T14:23:24.045 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6640000000000003e-06 2024-08-19T14:23:24.045 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.045 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.045 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:23:24.046 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:21.377044+0000", 2024-08-19T14:23:24.046 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1467999999999999e-05 2024-08-19T14:23:24.046 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.046 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.046 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:23:24.046 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:21.377051+0000", 2024-08-19T14:23:24.046 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3110000000000002e-06 2024-08-19T14:23:24.046 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.046 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.046 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:23:24.046 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:21.379599+0000", 2024-08-19T14:23:24.046 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0025484330000000001 2024-08-19T14:23:24.046 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.047 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.047 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:23:24.047 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:21.379656+0000", 2024-08-19T14:23:24.047 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7136000000000003e-05 2024-08-19T14:23:24.047 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.047 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.047 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:23:24.047 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:21.403349+0000", 2024-08-19T14:23:24.047 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.023692656999999999 2024-08-19T14:23:24.047 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.047 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.047 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:23:24.047 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:21.403562+0000", 2024-08-19T14:23:24.047 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00021310799999999999 2024-08-19T14:23:24.048 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:24.048 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:23:24.048 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:24.048 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.048 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.048 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.25333.0:200041 4.7 e394/335 4:e0ce4afe:::benchmark_data_smithi045_71562_object68866:head v 394'12561, mlcod=394'12557)", 2024-08-19T14:23:24.048 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:23:21.382740+0000", 2024-08-19T14:23:24.048 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 2.577309053, 2024-08-19T14:23:24.048 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.021000439999999999, 2024-08-19T14:23:24.048 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:23:24.048 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:23:24.048 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:23:24.048 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:23:24.049 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.049 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:23:24.049 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:21.382740+0000", 2024-08-19T14:23:24.049 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:24.049 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.049 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.049 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:23:24.049 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:21.382740+0000", 2024-08-19T14:23:24.049 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:24.049 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.049 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.049 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:23:24.049 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:21.382741+0000", 2024-08-19T14:23:24.049 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3e-07 2024-08-19T14:23:24.050 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.050 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.050 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:23:24.050 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:21.382747+0000", 2024-08-19T14:23:24.050 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7989999999999999e-06 2024-08-19T14:23:24.050 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.050 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.050 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:23:24.050 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:21.382762+0000", 2024-08-19T14:23:24.050 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5036e-05 2024-08-19T14:23:24.050 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.050 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.051 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:23:24.051 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:21.382769+0000", 2024-08-19T14:23:24.051 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2520000000000002e-06 2024-08-19T14:23:24.051 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.051 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.051 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:23:24.051 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:21.388503+0000", 2024-08-19T14:23:24.051 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0057344459999999998 2024-08-19T14:23:24.051 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.051 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.051 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:23:24.051 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:21.388525+0000", 2024-08-19T14:23:24.051 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1420000000000002e-05 2024-08-19T14:23:24.051 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.052 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.052 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:23:24.052 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:21.403566+0000", 2024-08-19T14:23:24.052 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.015041292 2024-08-19T14:23:24.052 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.052 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.052 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:23:24.052 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:21.403740+0000", 2024-08-19T14:23:24.052 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000174465 2024-08-19T14:23:24.052 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:24.052 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:23:24.052 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:24.052 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.053 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.053 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.25333.0:200045 4.7 e394/335 4:e52b6a4e:::benchmark_data_smithi045_71562_object68870:head v 394'12562, mlcod=394'12557)", 2024-08-19T14:23:24.053 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:23:21.388073+0000", 2024-08-19T14:23:24.053 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 2.5719756199999999, 2024-08-19T14:23:24.053 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.028846528999999999, 2024-08-19T14:23:24.053 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:23:24.053 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:23:24.053 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:23:24.053 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:23:24.053 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.053 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:23:24.053 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:21.388073+0000", 2024-08-19T14:23:24.053 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:24.054 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.054 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.054 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:23:24.054 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:21.388073+0000", 2024-08-19T14:23:24.054 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:24.054 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.054 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.054 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:23:24.054 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:21.388074+0000", 2024-08-19T14:23:24.054 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.9999999999999996e-07 2024-08-19T14:23:24.054 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.054 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.054 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:23:24.054 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:21.388083+0000", 2024-08-19T14:23:24.055 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.2199999999999992e-06 2024-08-19T14:23:24.055 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.055 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.055 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:23:24.055 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:21.388097+0000", 2024-08-19T14:23:24.055 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4797000000000001e-05 2024-08-19T14:23:24.055 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.055 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.055 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:23:24.055 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:21.388105+0000", 2024-08-19T14:23:24.055 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.8210000000000002e-06 2024-08-19T14:23:24.055 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.055 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.056 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:23:24.056 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:21.394137+0000", 2024-08-19T14:23:24.056 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0060316370000000003 2024-08-19T14:23:24.056 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.056 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.056 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:23:24.056 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:21.394158+0000", 2024-08-19T14:23:24.056 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1390999999999999e-05 2024-08-19T14:23:24.056 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.056 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.056 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:23:24.056 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:21.416876+0000", 2024-08-19T14:23:24.056 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.022717970000000001 2024-08-19T14:23:24.056 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.057 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.057 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:23:24.057 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:21.416920+0000", 2024-08-19T14:23:24.057 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3792999999999999e-05 2024-08-19T14:23:24.057 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:24.057 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:23:24.057 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:24.057 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.057 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.057 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.25333.0:200053 4.7 e394/335 4:e08091cb:::benchmark_data_smithi045_71562_object68878:head v 394'12563, mlcod=394'12557)", 2024-08-19T14:23:24.057 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:23:21.395524+0000", 2024-08-19T14:23:24.057 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 2.5645255329999999, 2024-08-19T14:23:24.057 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.021432270999999999, 2024-08-19T14:23:24.057 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:23:24.057 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:23:24.058 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:23:24.058 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:23:24.058 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.058 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:23:24.058 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:21.395524+0000", 2024-08-19T14:23:24.058 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:24.058 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.058 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.058 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:23:24.058 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:21.395524+0000", 2024-08-19T14:23:24.058 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:24.058 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.058 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.059 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:23:24.059 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:21.395524+0000", 2024-08-19T14:23:24.059 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3900000000000003e-07 2024-08-19T14:23:24.059 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.059 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.059 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:23:24.059 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:21.395537+0000", 2024-08-19T14:23:24.059 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2531000000000001e-05 2024-08-19T14:23:24.059 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.059 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.059 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:23:24.059 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:21.395548+0000", 2024-08-19T14:23:24.059 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0787e-05 2024-08-19T14:23:24.060 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.060 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.060 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:23:24.060 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:21.395555+0000", 2024-08-19T14:23:24.060 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7819999999999996e-06 2024-08-19T14:23:24.060 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.060 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.060 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:23:24.060 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:21.403855+0000", 2024-08-19T14:23:24.060 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0083007589999999996 2024-08-19T14:23:24.060 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.060 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.060 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:23:24.060 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:21.403941+0000", 2024-08-19T14:23:24.060 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.5426000000000005e-05 2024-08-19T14:23:24.061 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.061 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.061 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:23:24.061 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:21.416923+0000", 2024-08-19T14:23:24.061 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.012982295 2024-08-19T14:23:24.061 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.061 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.061 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:23:24.061 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:21.416956+0000", 2024-08-19T14:23:24.061 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2752000000000001e-05 2024-08-19T14:23:24.061 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:24.061 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:23:24.061 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:24.062 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.062 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.062 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.25333.0:200651 4.8 e394/335 4:16bdd370:::benchmark_data_smithi045_71562_object69476:head v 394'12492, mlcod=394'12489)", 2024-08-19T14:23:24.062 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:23:22.175135+0000", 2024-08-19T14:23:24.062 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 1.7849143490000001, 2024-08-19T14:23:24.062 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.022032408999999999, 2024-08-19T14:23:24.062 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:23:24.062 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:23:24.062 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:23:24.062 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:23:24.062 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.062 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:23:24.062 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:22.175135+0000", 2024-08-19T14:23:24.063 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:24.063 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.063 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.063 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:23:24.063 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:22.175135+0000", 2024-08-19T14:23:24.063 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:24.063 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.063 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.063 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:23:24.063 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:22.175136+0000", 2024-08-19T14:23:24.063 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.035e-06 2024-08-19T14:23:24.063 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.064 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.064 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:23:24.064 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:22.175145+0000", 2024-08-19T14:23:24.064 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3149999999999998e-06 2024-08-19T14:23:24.064 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.064 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.064 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:23:24.064 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:22.175164+0000", 2024-08-19T14:23:24.064 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8695e-05 2024-08-19T14:23:24.064 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.064 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.064 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:23:24.064 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:22.175173+0000", 2024-08-19T14:23:24.065 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.8089999999999999e-06 2024-08-19T14:23:24.065 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.065 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.065 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:23:24.065 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:22.177248+0000", 2024-08-19T14:23:24.065 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0020755550000000002 2024-08-19T14:23:24.065 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.065 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.065 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:23:24.065 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:22.177270+0000", 2024-08-19T14:23:24.065 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2089000000000001e-05 2024-08-19T14:23:24.065 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.065 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.065 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:23:24.066 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:22.197129+0000", 2024-08-19T14:23:24.066 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.019858974000000001 2024-08-19T14:23:24.066 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.066 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.066 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:23:24.066 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:22.197167+0000", 2024-08-19T14:23:24.066 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7936999999999999e-05 2024-08-19T14:23:24.066 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:24.066 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:23:24.066 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:24.066 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.066 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.067 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.25333.0:200660 4.0 e394/335 4:00dcdf49:::benchmark_data_smithi045_71562_object69485:head v 394'12440, mlcod=394'12438)", 2024-08-19T14:23:24.067 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:23:22.184549+0000", 2024-08-19T14:23:24.067 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 1.775499862, 2024-08-19T14:23:24.067 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.022341271999999999, 2024-08-19T14:23:24.067 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:23:24.067 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:23:24.067 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:23:24.067 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:23:24.067 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.067 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:23:24.067 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:22.184549+0000", 2024-08-19T14:23:24.067 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:24.067 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.068 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.068 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:23:24.068 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:22.184549+0000", 2024-08-19T14:23:24.068 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:23:24.068 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.068 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.068 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:23:24.068 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:22.184550+0000", 2024-08-19T14:23:24.068 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3200000000000004e-07 2024-08-19T14:23:24.068 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.068 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.068 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:23:24.068 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:22.184557+0000", 2024-08-19T14:23:24.069 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3289999999999996e-06 2024-08-19T14:23:24.069 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.069 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.069 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:23:24.069 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:22.184568+0000", 2024-08-19T14:23:24.069 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0552e-05 2024-08-19T14:23:24.069 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.069 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.069 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:23:24.069 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:22.184574+0000", 2024-08-19T14:23:24.069 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.5150000000000003e-06 2024-08-19T14:23:24.069 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.069 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.069 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:23:24.070 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:22.185534+0000", 2024-08-19T14:23:24.070 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00095983400000000001 2024-08-19T14:23:24.070 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.070 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.070 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:23:24.070 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:22.185552+0000", 2024-08-19T14:23:24.070 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7529000000000001e-05 2024-08-19T14:23:24.070 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.070 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.070 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:23:24.070 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:22.206799+0000", 2024-08-19T14:23:24.070 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.021246787999999999 2024-08-19T14:23:24.070 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:23:24.071 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:23:24.071 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:23:24.071 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:22.206890+0000", 2024-08-19T14:23:24.071 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.1992999999999998e-05 2024-08-19T14:23:24.071 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:24.071 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:23:24.071 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:24.071 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:23:24.071 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:23:24.071 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:23:24.071 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:23:24.093 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:24.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:24.094 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:23:24.178 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:23:24.194 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:23:24.220 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:24.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:24.221 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:23:24.250 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:23:24.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:23 smithi045 ceph-mon[20143]: 4.3 scrub ok 2024-08-19T14:23:24.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:23 smithi045 ceph-mon[20143]: 1.6 scrub starts 2024-08-19T14:23:24.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:23 smithi045 ceph-mon[20143]: 1.6 scrub ok 2024-08-19T14:23:24.323 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:23:24.341 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:23 smithi031 ceph-mon[16336]: 4.3 scrub ok 2024-08-19T14:23:24.341 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:23 smithi031 ceph-mon[16336]: 1.6 scrub starts 2024-08-19T14:23:24.341 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:23 smithi031 ceph-mon[16336]: 1.6 scrub ok 2024-08-19T14:23:24.345 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:24.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:24.347 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:23:24.448 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:23:24.474 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:24.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:24.475 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:23:24.577 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:23:24.611 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:24.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:24.612 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:23:24.713 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:23:24.739 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:24.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:24.740 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:23:24.840 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:23:24.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:24.882 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:24.882 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:23:24.925 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-08-19T14:23:24.946 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:23:24.947 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:23:24.983 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:23:24.986 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:23:24.986 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:23:25.008 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:25.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:25.009 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:23:25.024 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:23:25.025 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:23:25.059 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:23:25.060 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:23:25.110 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:23:25.111 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:23:25.112 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:23:25.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:25.143 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:25.143 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:23:25.144 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:23:25.144 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:23:25.144 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:24 smithi031 ceph-mon[16336]: 4.b scrub ok 2024-08-19T14:23:25.144 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:24 smithi031 ceph-mon[16336]: 1.b scrub starts 2024-08-19T14:23:25.144 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:24 smithi031 ceph-mon[16336]: 1.b scrub ok 2024-08-19T14:23:25.144 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:24 smithi031 ceph-mon[16336]: pgmap v899: 47 pgs: 47 active+clean; 3.8 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 1.14k op/s 2024-08-19T14:23:25.144 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:24 smithi031 ceph-mon[16336]: 1.2 scrub starts 2024-08-19T14:23:25.145 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:24 smithi031 ceph-mon[16336]: 1.2 scrub ok 2024-08-19T14:23:25.145 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:24 smithi031 ceph-mon[16336]: 1.1 scrub starts 2024-08-19T14:23:25.145 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:24 smithi031 ceph-mon[16336]: 1.1 scrub ok 2024-08-19T14:23:25.145 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:24 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3994355506' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:23:25.178 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:23:25.178 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:23:25.197 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:24 smithi045 ceph-mon[20143]: 4.b scrub ok 2024-08-19T14:23:25.197 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:24 smithi045 ceph-mon[20143]: 1.b scrub starts 2024-08-19T14:23:25.198 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:24 smithi045 ceph-mon[20143]: 1.b scrub ok 2024-08-19T14:23:25.198 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:24 smithi045 ceph-mon[20143]: pgmap v899: 47 pgs: 47 active+clean; 3.8 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 1.14k op/s 2024-08-19T14:23:25.198 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:24 smithi045 ceph-mon[20143]: 1.2 scrub starts 2024-08-19T14:23:25.198 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:24 smithi045 ceph-mon[20143]: 1.2 scrub ok 2024-08-19T14:23:25.198 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:24 smithi045 ceph-mon[20143]: 1.1 scrub starts 2024-08-19T14:23:25.198 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:24 smithi045 ceph-mon[20143]: 1.1 scrub ok 2024-08-19T14:23:25.198 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:24 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3994355506' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:23:25.214 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:23:25.214 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:23:25.244 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:23:25.254 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:23:25.254 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:23:25.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:24 smithi032 ceph-mon[20004]: 4.b scrub ok 2024-08-19T14:23:25.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:24 smithi032 ceph-mon[20004]: 1.b scrub starts 2024-08-19T14:23:25.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:24 smithi032 ceph-mon[20004]: 1.b scrub ok 2024-08-19T14:23:25.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:24 smithi032 ceph-mon[20004]: pgmap v899: 47 pgs: 47 active+clean; 3.8 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 1.14k op/s 2024-08-19T14:23:25.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:24 smithi032 ceph-mon[20004]: 1.2 scrub starts 2024-08-19T14:23:25.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:24 smithi032 ceph-mon[20004]: 1.2 scrub ok 2024-08-19T14:23:25.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:24 smithi032 ceph-mon[20004]: 1.1 scrub starts 2024-08-19T14:23:25.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:24 smithi032 ceph-mon[20004]: 1.1 scrub ok 2024-08-19T14:23:25.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:24 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3994355506' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:23:25.261 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:23:25.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:25.288 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:25.288 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:23:25.298 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:23:25.298 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:23:25.332 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:23:25.333 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:23:25.373 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:23:25.373 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:23:25.389 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:23:25.404 INFO:teuthology.orchestra.run.smithi031.stderr:no valid command found; 10 closest matches: 2024-08-19T14:23:25.408 INFO:teuthology.orchestra.run.smithi031.stderr:0 2024-08-19T14:23:25.408 INFO:teuthology.orchestra.run.smithi031.stderr:1 2024-08-19T14:23:25.408 INFO:teuthology.orchestra.run.smithi031.stderr:2 2024-08-19T14:23:25.408 INFO:teuthology.orchestra.run.smithi031.stderr:abort 2024-08-19T14:23:25.408 INFO:teuthology.orchestra.run.smithi031.stderr:assert 2024-08-19T14:23:25.409 INFO:teuthology.orchestra.run.smithi031.stderr:bench [] [] [] [] 2024-08-19T14:23:25.409 INFO:teuthology.orchestra.run.smithi031.stderr:bluefs debug_inject_read_zeros 2024-08-19T14:23:25.409 INFO:teuthology.orchestra.run.smithi031.stderr:bluefs files list 2024-08-19T14:23:25.409 INFO:teuthology.orchestra.run.smithi031.stderr:bluefs stats 2024-08-19T14:23:25.409 INFO:teuthology.orchestra.run.smithi031.stderr:bluestore allocator dump block 2024-08-19T14:23:25.409 INFO:teuthology.orchestra.run.smithi031.stderr:Error EINVAL: invalid command 2024-08-19T14:23:25.433 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:25.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:25.434 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:23:25.535 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:23:25.565 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:25.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:25.566 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:23:25.667 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:23:25.705 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:25.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:25.706 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:23:25.807 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:23:25.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:25.844 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:25.844 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:23:25.945 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:23:25.971 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:25.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:25.972 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:23:26.073 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:23:26.097 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:26.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:26.099 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:23:26.100 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-19T14:23:26.151 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:25 smithi031 ceph-mon[16336]: 4.0 scrub starts 2024-08-19T14:23:26.151 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:25 smithi031 ceph-mon[16336]: 1.15 scrub starts 2024-08-19T14:23:26.151 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:25 smithi031 ceph-mon[16336]: 1.15 scrub ok 2024-08-19T14:23:26.151 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:25 smithi031 ceph-mon[16336]: 1.a scrub starts 2024-08-19T14:23:26.152 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:25 smithi031 ceph-mon[16336]: 1.a scrub ok 2024-08-19T14:23:26.199 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:23:26.223 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:25 smithi045 ceph-mon[20143]: 4.0 scrub starts 2024-08-19T14:23:26.223 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:25 smithi045 ceph-mon[20143]: 1.15 scrub starts 2024-08-19T14:23:26.223 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:25 smithi045 ceph-mon[20143]: 1.15 scrub ok 2024-08-19T14:23:26.223 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:25 smithi045 ceph-mon[20143]: 1.a scrub starts 2024-08-19T14:23:26.223 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:25 smithi045 ceph-mon[20143]: 1.a scrub ok 2024-08-19T14:23:26.236 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:23:26.236 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:23:26.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:26.255 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:26.255 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:23:26.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:25 smithi032 ceph-mon[20004]: 4.0 scrub starts 2024-08-19T14:23:26.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:25 smithi032 ceph-mon[20004]: 1.15 scrub starts 2024-08-19T14:23:26.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:25 smithi032 ceph-mon[20004]: 1.15 scrub ok 2024-08-19T14:23:26.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:25 smithi032 ceph-mon[20004]: 1.a scrub starts 2024-08-19T14:23:26.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:25 smithi032 ceph-mon[20004]: 1.a scrub ok 2024-08-19T14:23:26.355 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:23:26.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:26.389 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:26.389 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:23:26.490 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:23:26.530 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:26.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:26.531 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:23:26.633 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:23:26.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:26.655 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:26.655 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:23:26.756 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:23:26.783 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:26.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:26.785 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:23:26.885 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:23:26.929 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:26.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:26.930 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:23:26.939 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-08-19T14:23:27.031 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:23:27.066 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:27.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:27.068 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:23:27.169 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:23:27.198 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:27.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:27.199 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:23:27.236 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:26 smithi031 ceph-mon[16336]: pgmap v900: 47 pgs: 47 active+clean; 3.8 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 781 op/s 2024-08-19T14:23:27.236 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:26 smithi031 ceph-mon[16336]: 4.0 scrub ok 2024-08-19T14:23:27.236 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:26 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:23:27.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:26 smithi032 ceph-mon[20004]: pgmap v900: 47 pgs: 47 active+clean; 3.8 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 781 op/s 2024-08-19T14:23:27.259 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:26 smithi032 ceph-mon[20004]: 4.0 scrub ok 2024-08-19T14:23:27.259 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:26 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:23:27.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:26 smithi045 ceph-mon[20143]: pgmap v900: 47 pgs: 47 active+clean; 3.8 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 781 op/s 2024-08-19T14:23:27.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:26 smithi045 ceph-mon[20143]: 4.0 scrub ok 2024-08-19T14:23:27.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:26 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:23:27.300 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:23:27.325 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:27.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:27.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:23:27.328 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:23:27.428 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:23:27.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:27.460 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:27.460 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:23:27.561 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:23:27.585 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-19T14:23:27.585 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-19T14:23:27.586 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph tell osd.3 'trim stale osdmaps' 2024-08-19T14:23:27.616 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:23:27.629 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:27.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:27.632 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:23:27.733 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:23:27.755 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:27.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:27.756 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:23:27.857 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:23:27.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:27.912 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:27.912 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:23:28.013 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:23:28.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:28.038 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:28.038 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:23:28.066 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:23:28.066 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:23:28.139 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:23:28.140 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:27 smithi031 ceph-mon[16336]: 1.3 scrub starts 2024-08-19T14:23:28.140 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:27 smithi031 ceph-mon[16336]: 1.3 scrub ok 2024-08-19T14:23:28.141 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:27 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:23:28.141 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:27 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:23:28.141 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:27 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:23:28.165 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:28.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:28.167 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:23:28.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:27 smithi032 ceph-mon[20004]: 1.3 scrub starts 2024-08-19T14:23:28.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:27 smithi032 ceph-mon[20004]: 1.3 scrub ok 2024-08-19T14:23:28.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:27 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:23:28.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:27 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:23:28.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:27 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:23:28.267 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:23:28.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:28.291 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:28.292 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:23:28.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:27 smithi045 ceph-mon[20143]: 1.3 scrub starts 2024-08-19T14:23:28.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:27 smithi045 ceph-mon[20143]: 1.3 scrub ok 2024-08-19T14:23:28.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:27 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:23:28.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:27 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:23:28.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:27 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:23:28.340 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:23:28.393 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:23:28.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:28.418 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:28.419 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:23:28.520 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:23:28.546 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:28.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:28.547 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:23:28.648 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:23:28.686 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:28.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:28.690 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:23:28.791 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:23:28.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:28.821 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:28.821 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:23:28.828 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-08-19T14:23:28.922 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:23:28.959 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:28.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:28.961 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:23:28.963 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is set 2024-08-19T14:23:28.999 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:23:29.000 INFO:teuthology.orchestra.run.smithi031.stderr:no valid command found; 10 closest matches: 2024-08-19T14:23:29.000 INFO:teuthology.orchestra.run.smithi031.stderr:0 2024-08-19T14:23:29.000 INFO:teuthology.orchestra.run.smithi031.stderr:1 2024-08-19T14:23:29.000 INFO:teuthology.orchestra.run.smithi031.stderr:2 2024-08-19T14:23:29.000 INFO:teuthology.orchestra.run.smithi031.stderr:abort 2024-08-19T14:23:29.000 INFO:teuthology.orchestra.run.smithi031.stderr:assert 2024-08-19T14:23:29.001 INFO:teuthology.orchestra.run.smithi031.stderr:bench [] [] [] [] 2024-08-19T14:23:29.001 INFO:teuthology.orchestra.run.smithi031.stderr:bluefs debug_inject_read_zeros 2024-08-19T14:23:29.002 INFO:teuthology.orchestra.run.smithi031.stderr:bluefs files list 2024-08-19T14:23:29.002 INFO:teuthology.orchestra.run.smithi031.stderr:bluefs stats 2024-08-19T14:23:29.002 INFO:teuthology.orchestra.run.smithi031.stderr:bluestore allocator dump block 2024-08-19T14:23:29.002 INFO:teuthology.orchestra.run.smithi031.stderr:Error EINVAL: invalid command 2024-08-19T14:23:29.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:28 smithi045 ceph-mon[20143]: pgmap v901: 47 pgs: 47 active+clean; 3.6 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 868 op/s 2024-08-19T14:23:29.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:28 smithi045 ceph-mon[20143]: 4.2 scrub starts 2024-08-19T14:23:29.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:28 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/376547747' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:23:29.062 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:23:29.088 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:29.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:29.089 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:23:29.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:28 smithi031 ceph-mon[16336]: pgmap v901: 47 pgs: 47 active+clean; 3.6 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 868 op/s 2024-08-19T14:23:29.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:28 smithi031 ceph-mon[16336]: 4.2 scrub starts 2024-08-19T14:23:29.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:28 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/376547747' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:23:29.178 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:23:29.189 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:23:29.216 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:29.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:29.217 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:23:29.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:28 smithi032 ceph-mon[20004]: pgmap v901: 47 pgs: 47 active+clean; 3.6 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 868 op/s 2024-08-19T14:23:29.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:28 smithi032 ceph-mon[20004]: 4.2 scrub starts 2024-08-19T14:23:29.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:28 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/376547747' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:23:29.318 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:23:29.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:29.346 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:29.346 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:23:29.447 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:23:29.478 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:29.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:29.479 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:23:29.580 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:23:29.605 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:29.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:29.606 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:23:29.707 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:23:29.723 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:23:29.723 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:23:29.734 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:29.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:29.735 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:23:29.836 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:23:29.857 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:29.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:29.858 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:23:29.959 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:23:29.991 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:29.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:29.992 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:23:30.093 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:23:30.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:29 smithi031 ceph-mon[16336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:23:30.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:29 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/376547747' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:23:30.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:29 smithi031 ceph-mon[16336]: osdmap e395: 12 total, 12 up, 9 in 2024-08-19T14:23:30.130 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:30.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:30.131 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:23:30.232 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:23:30.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:29 smithi032 ceph-mon[20004]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:23:30.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:29 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/376547747' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:23:30.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:29 smithi032 ceph-mon[20004]: osdmap e395: 12 total, 12 up, 9 in 2024-08-19T14:23:30.267 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:30.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:30.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:23:30.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:29 smithi045 ceph-mon[20143]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:23:30.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:29 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/376547747' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:23:30.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:29 smithi045 ceph-mon[20143]: osdmap e395: 12 total, 12 up, 9 in 2024-08-19T14:23:30.369 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:23:30.375 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-08-19T14:23:30.406 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:30.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:30.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:23:30.408 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-19T14:23:30.409 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-19T14:23:30.409 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph tell osd.4 'trim stale osdmaps' 2024-08-19T14:23:30.470 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:23:30.507 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:23:30.534 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:30.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:30.535 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:23:30.636 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:23:30.661 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:30.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:30.662 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:23:30.699 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:23:30.762 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:23:30.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:30.791 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:30.791 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:23:30.893 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:23:30.920 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:30.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:30.921 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:23:31.022 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:23:31.048 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:31.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:31.049 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:23:31.150 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:23:31.182 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:23:31.182 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:23:31.190 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:23:31.216 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:31.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:31.218 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:23:31.227 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:23:31.228 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:23:31.228 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:30 smithi031 ceph-mon[16336]: pgmap v903: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 3.5 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 942 op/s 2024-08-19T14:23:31.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:30 smithi032 ceph-mon[20004]: pgmap v903: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 3.5 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 942 op/s 2024-08-19T14:23:31.266 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:23:31.266 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:23:31.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:30 smithi045 ceph-mon[20143]: pgmap v903: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 3.5 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 942 op/s 2024-08-19T14:23:31.300 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:23:31.300 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:23:31.318 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:23:31.338 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:23:31.338 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:23:31.345 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:31.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:31.346 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:23:31.371 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:23:31.371 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:23:31.402 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:23:31.403 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:23:31.441 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:23:31.441 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:23:31.447 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:23:31.471 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:31.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:31.472 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:23:31.479 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:23:31.480 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:23:31.510 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:23:31.510 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:23:31.541 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:23:31.541 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:23:31.573 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:23:31.596 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:23:31.596 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:23:31.603 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:31.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:31.604 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:23:31.705 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:23:31.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:31.754 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:31.755 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:23:31.855 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:23:31.860 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-19T14:23:31.880 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:31.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:31.881 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:23:31.890 INFO:teuthology.orchestra.run.smithi031.stderr:no valid command found; 10 closest matches: 2024-08-19T14:23:31.890 INFO:teuthology.orchestra.run.smithi031.stderr:0 2024-08-19T14:23:31.890 INFO:teuthology.orchestra.run.smithi031.stderr:1 2024-08-19T14:23:31.890 INFO:teuthology.orchestra.run.smithi031.stderr:2 2024-08-19T14:23:31.890 INFO:teuthology.orchestra.run.smithi031.stderr:abort 2024-08-19T14:23:31.890 INFO:teuthology.orchestra.run.smithi031.stderr:assert 2024-08-19T14:23:31.890 INFO:teuthology.orchestra.run.smithi031.stderr:bench [] [] [] [] 2024-08-19T14:23:31.890 INFO:teuthology.orchestra.run.smithi031.stderr:bluefs debug_inject_read_zeros 2024-08-19T14:23:31.891 INFO:teuthology.orchestra.run.smithi031.stderr:bluefs files list 2024-08-19T14:23:31.891 INFO:teuthology.orchestra.run.smithi031.stderr:bluefs stats 2024-08-19T14:23:31.891 INFO:teuthology.orchestra.run.smithi031.stderr:bluestore allocator dump block 2024-08-19T14:23:31.891 INFO:teuthology.orchestra.run.smithi031.stderr:Error EINVAL: invalid command 2024-08-19T14:23:31.982 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:23:32.007 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:32.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:32.008 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:23:32.109 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:23:32.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:31 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1983170946' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:23:32.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:31 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:23:32.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:31 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:23:32.130 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:32.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:32.131 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:23:32.232 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:23:32.256 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:32.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:32.257 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:23:32.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:31 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1983170946' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:23:32.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:31 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:23:32.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:31 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:23:32.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:31 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1983170946' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:23:32.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:31 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:23:32.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:31 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:23:32.358 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:23:32.384 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:32.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:32.385 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:23:32.486 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:23:32.510 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:23:32.510 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [ 2024-08-19T14:23:32.510 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:32.510 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:208838 4.2 4:44a01794:::benchmark_data_smithi045_71562_object77663:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e395)", 2024-08-19T14:23:32.510 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:23:31.732244+0000", 2024-08-19T14:23:32.510 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.00029090499999999999, 2024-08-19T14:23:32.510 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000318914, 2024-08-19T14:23:32.510 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:23:32.510 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:23:32.510 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "waiting for sub ops", 2024-08-19T14:23:32.510 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:23:32.510 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:23:32.511 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:23:32.511 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 208838 2024-08-19T14:23:32.511 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:32.511 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:23:32.511 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:32.511 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:23:32.511 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:23:31.732244+0000", 2024-08-19T14:23:32.511 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:23:32.511 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:32.511 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:32.511 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:23:32.511 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:23:31.732244+0000", 2024-08-19T14:23:32.511 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:23:32.511 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:32.512 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:32.512 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:23:32.512 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:23:31.732245+0000", 2024-08-19T14:23:32.512 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.215e-06 2024-08-19T14:23:32.512 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:32.512 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:32.512 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:23:32.512 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:23:31.732251+0000", 2024-08-19T14:23:32.512 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.8950000000000003e-06 2024-08-19T14:23:32.512 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:32.512 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:32.512 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:23:32.512 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:23:31.732262+0000", 2024-08-19T14:23:32.512 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1584e-05 2024-08-19T14:23:32.513 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:32.513 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:32.513 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:23:32.513 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:23:31.732269+0000", 2024-08-19T14:23:32.513 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.7109999999999998e-06 2024-08-19T14:23:32.513 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:32.513 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:32.513 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:23:32.513 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:23:31.732326+0000", 2024-08-19T14:23:32.513 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.6724000000000001e-05 2024-08-19T14:23:32.513 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:32.513 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:32.513 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:23:32.514 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:23:31.732411+0000", 2024-08-19T14:23:32.514 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.4598000000000006e-05 2024-08-19T14:23:32.514 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:32.514 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:32.514 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 5,11", 2024-08-19T14:23:32.514 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:23:31.732502+0000", 2024-08-19T14:23:32.514 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.1236000000000005e-05 2024-08-19T14:23:32.514 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:23:32.514 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:23:32.514 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:23:32.514 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:23:32.514 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-08-19T14:23:32.514 INFO:teuthology.orchestra.run.smithi045.stdout: "num_ops": 1 2024-08-19T14:23:32.515 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:23:32.517 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:32.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:32.518 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:23:32.619 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:23:32.647 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:32.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:32.648 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:23:32.749 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:23:32.772 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:32.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:32.773 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:23:32.874 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:23:32.902 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:32.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:32.903 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:23:33.005 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:23:33.033 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:33.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:33.034 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:23:33.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:32 smithi031 ceph-mon[16336]: pgmap v904: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 3.4 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 846 op/s 2024-08-19T14:23:33.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:32 smithi031 ceph-mon[16336]: osdmap e396: 12 total, 12 up, 9 in 2024-08-19T14:23:33.134 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:23:33.169 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:33.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:33.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:23:33.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:32 smithi032 ceph-mon[20004]: pgmap v904: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 3.4 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 846 op/s 2024-08-19T14:23:33.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:32 smithi032 ceph-mon[20004]: osdmap e396: 12 total, 12 up, 9 in 2024-08-19T14:23:33.271 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:23:33.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:32 smithi045 ceph-mon[20143]: pgmap v904: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 3.4 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 846 op/s 2024-08-19T14:23:33.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:32 smithi045 ceph-mon[20143]: osdmap e396: 12 total, 12 up, 9 in 2024-08-19T14:23:33.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:33.300 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:33.300 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:23:33.401 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:23:33.425 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:33.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:33.426 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:23:33.527 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:23:33.558 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:33.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:33.559 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:23:33.660 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:23:33.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:33.699 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:33.699 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:23:33.800 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:23:33.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:33.829 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:33.830 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:23:33.931 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:23:33.968 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:33.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:33.970 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:23:33.972 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-08-19T14:23:34.070 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:23:34.101 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:34.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:34.102 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:23:34.203 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:23:34.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:34.237 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:34.238 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:23:34.308 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:23:34.338 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:23:34.380 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:34.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:34.381 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:23:34.412 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-19T14:23:34.413 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-19T14:23:34.413 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph tell osd.5 'trim stale osdmaps' 2024-08-19T14:23:34.482 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:23:34.507 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:34.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:34.509 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:23:34.610 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:23:34.636 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:34.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:34.637 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:23:34.738 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:23:34.765 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:34.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:34.766 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:23:34.848 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:23:34.848 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [], 2024-08-19T14:23:34.848 INFO:teuthology.orchestra.run.smithi045.stdout: "complaint_time": 30, 2024-08-19T14:23:34.848 INFO:teuthology.orchestra.run.smithi045.stdout: "num_blocked_ops": 0 2024-08-19T14:23:34.849 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:23:34.867 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:23:34.893 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:34.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:34.894 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:23:34.970 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:23:34.981 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:23:34.995 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:23:35.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:35.021 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:35.021 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:23:35.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:34 smithi031 ceph-mon[16336]: pgmap v906: 47 pgs: 47 active+clean; 3.2 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 1.03k op/s 2024-08-19T14:23:35.121 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:23:35.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:35.148 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:35.149 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:23:35.250 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:23:35.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:34 smithi032 ceph-mon[20004]: pgmap v906: 47 pgs: 47 active+clean; 3.2 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 1.03k op/s 2024-08-19T14:23:35.278 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:35.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:35.279 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:23:35.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:34 smithi045 ceph-mon[20143]: pgmap v906: 47 pgs: 47 active+clean; 3.2 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 1.03k op/s 2024-08-19T14:23:35.380 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:23:35.407 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:35.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:35.408 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:23:35.509 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:23:35.534 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-08-19T14:23:35.536 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:35.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:35.537 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:23:35.638 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:23:35.662 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:35.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:35.663 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:23:35.764 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:23:35.786 INFO:teuthology.orchestra.run.smithi031.stderr:no valid command found; 10 closest matches: 2024-08-19T14:23:35.787 INFO:teuthology.orchestra.run.smithi031.stderr:0 2024-08-19T14:23:35.787 INFO:teuthology.orchestra.run.smithi031.stderr:1 2024-08-19T14:23:35.787 INFO:teuthology.orchestra.run.smithi031.stderr:2 2024-08-19T14:23:35.787 INFO:teuthology.orchestra.run.smithi031.stderr:abort 2024-08-19T14:23:35.787 INFO:teuthology.orchestra.run.smithi031.stderr:assert 2024-08-19T14:23:35.787 INFO:teuthology.orchestra.run.smithi031.stderr:bench [] [] [] [] 2024-08-19T14:23:35.787 INFO:teuthology.orchestra.run.smithi031.stderr:bluefs debug_inject_read_zeros 2024-08-19T14:23:35.787 INFO:teuthology.orchestra.run.smithi031.stderr:bluefs files list 2024-08-19T14:23:35.787 INFO:teuthology.orchestra.run.smithi031.stderr:bluefs stats 2024-08-19T14:23:35.787 INFO:teuthology.orchestra.run.smithi031.stderr:bluestore allocator dump block 2024-08-19T14:23:35.787 INFO:teuthology.orchestra.run.smithi031.stderr:Error EINVAL: invalid command 2024-08-19T14:23:35.805 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:35.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:35.806 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:23:35.876 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:23:35.908 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:23:35.939 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:35.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:35.940 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:23:36.041 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:23:36.076 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:36.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:36.077 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:23:36.177 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:23:36.235 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:36.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:36.236 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:23:36.337 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:23:36.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:36.378 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:36.378 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:23:36.401 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:23:36.401 INFO:teuthology.orchestra.run.smithi045.stdout: "size": 20, 2024-08-19T14:23:36.402 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 600, 2024-08-19T14:23:36.402 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [ 2024-08-19T14:23:36.402 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.402 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25057.0:50174 3.10 3.bde586f0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-08-19T14:23:36.402 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:17:20.749720+0000", 2024-08-19T14:23:36.402 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 375.64922521599999, 2024-08-19T14:23:36.402 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.99597661999999998, 2024-08-19T14:23:36.402 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:23:36.402 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:23:36.402 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-08-19T14:23:36.402 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:23:36.402 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25057", 2024-08-19T14:23:36.403 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:23:36.403 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 50174 2024-08-19T14:23:36.403 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.403 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:23:36.403 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.403 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:23:36.403 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:20.749720+0000", 2024-08-19T14:23:36.403 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:23:36.403 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.403 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.403 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:23:36.403 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:20.749720+0000", 2024-08-19T14:23:36.403 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:23:36.403 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.404 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.404 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:23:36.404 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:20.749722+0000", 2024-08-19T14:23:36.404 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.79e-06 2024-08-19T14:23:36.404 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.404 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.404 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:23:36.404 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:20.749747+0000", 2024-08-19T14:23:36.404 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.4967e-05 2024-08-19T14:23:36.404 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.404 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.404 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:23:36.404 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:20.749867+0000", 2024-08-19T14:23:36.405 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000120388 2024-08-19T14:23:36.405 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.405 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.405 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:23:36.405 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:20.749875+0000", 2024-08-19T14:23:36.405 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.1470000000000001e-06 2024-08-19T14:23:36.405 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.405 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.405 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:23:36.405 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:20.749980+0000", 2024-08-19T14:23:36.405 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000104744 2024-08-19T14:23:36.405 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.405 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.405 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for peered", 2024-08-19T14:23:36.406 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:20.749991+0000", 2024-08-19T14:23:36.406 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1014e-05 2024-08-19T14:23:36.406 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.406 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.406 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:23:36.406 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:21.745658+0000", 2024-08-19T14:23:36.406 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.99566752700000005 2024-08-19T14:23:36.406 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.406 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.406 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:23:36.406 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:21.745696+0000", 2024-08-19T14:23:36.406 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.8043000000000002e-05 2024-08-19T14:23:36.406 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:23:36.407 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:23:36.407 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:23:36.407 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.407 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.407 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25057.0:63341 3.22 3:44f22295:::benchmark_data_smithi045_59973_object63340:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e244)", 2024-08-19T14:23:36.407 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:17:34.014807+0000", 2024-08-19T14:23:36.407 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 362.38413784599999, 2024-08-19T14:23:36.407 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.97840119400000003, 2024-08-19T14:23:36.407 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:23:36.407 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:23:36.408 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:23:36.408 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:23:36.408 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25057", 2024-08-19T14:23:36.408 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:23:36.408 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 63341 2024-08-19T14:23:36.408 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.409 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:23:36.409 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.409 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:23:36.409 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:34.014807+0000", 2024-08-19T14:23:36.409 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:23:36.409 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.409 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.409 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:23:36.409 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:34.014807+0000", 2024-08-19T14:23:36.410 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:23:36.410 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.410 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.410 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:23:36.410 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:34.014809+0000", 2024-08-19T14:23:36.410 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.46e-06 2024-08-19T14:23:36.410 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.410 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.410 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:23:36.410 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:34.014834+0000", 2024-08-19T14:23:36.410 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.4854999999999998e-05 2024-08-19T14:23:36.410 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.410 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.411 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:23:36.411 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:34.014846+0000", 2024-08-19T14:23:36.411 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2258e-05 2024-08-19T14:23:36.411 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.411 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.411 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:23:36.411 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:34.014851+0000", 2024-08-19T14:23:36.411 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2959999999999998e-06 2024-08-19T14:23:36.411 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.411 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.411 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:23:36.411 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:34.014906+0000", 2024-08-19T14:23:36.411 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4738999999999999e-05 2024-08-19T14:23:36.412 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.412 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.412 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for peered", 2024-08-19T14:23:36.412 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:34.014913+0000", 2024-08-19T14:23:36.412 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.1590000000000004e-06 2024-08-19T14:23:36.412 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.412 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.412 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:23:36.412 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:34.989215+0000", 2024-08-19T14:23:36.412 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.97430243299999997 2024-08-19T14:23:36.412 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.412 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.412 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:23:36.412 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:34.989336+0000", 2024-08-19T14:23:36.413 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000120977 2024-08-19T14:23:36.413 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.413 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.413 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 4,5", 2024-08-19T14:23:36.413 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:34.989467+0000", 2024-08-19T14:23:36.413 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000130495 2024-08-19T14:23:36.413 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.413 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.413 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:23:36.413 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:34.990225+0000", 2024-08-19T14:23:36.413 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00075812000000000004 2024-08-19T14:23:36.413 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.413 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.413 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:23:36.414 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:34.990711+0000", 2024-08-19T14:23:36.414 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000485522 2024-08-19T14:23:36.414 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.414 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.414 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:23:36.414 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:34.993150+0000", 2024-08-19T14:23:36.414 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.002439049 2024-08-19T14:23:36.414 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.414 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.414 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:23:36.414 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:34.993184+0000", 2024-08-19T14:23:36.414 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.4035000000000002e-05 2024-08-19T14:23:36.414 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.415 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.415 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:23:36.415 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:34.993208+0000", 2024-08-19T14:23:36.415 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.4796000000000001e-05 2024-08-19T14:23:36.415 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:23:36.415 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:23:36.415 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:23:36.415 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.415 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.415 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25057.0:64615 3.20 3.18b2560 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e246)", 2024-08-19T14:23:36.415 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:17:35.982300+0000", 2024-08-19T14:23:36.415 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 360.416645534, 2024-08-19T14:23:36.415 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.98262199699999997, 2024-08-19T14:23:36.415 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:23:36.416 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:23:36.416 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-08-19T14:23:36.416 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:23:36.416 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25057", 2024-08-19T14:23:36.416 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:23:36.416 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 64615 2024-08-19T14:23:36.416 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.416 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:23:36.416 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.416 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:23:36.416 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:35.982300+0000", 2024-08-19T14:23:36.416 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:23:36.416 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.416 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.417 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:23:36.417 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:35.982300+0000", 2024-08-19T14:23:36.417 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:23:36.417 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.417 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.417 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:23:36.417 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:35.982301+0000", 2024-08-19T14:23:36.418 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.316e-06 2024-08-19T14:23:36.418 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.418 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.418 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:23:36.418 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:35.982323+0000", 2024-08-19T14:23:36.418 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1747e-05 2024-08-19T14:23:36.418 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.418 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.418 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:23:36.419 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:35.982336+0000", 2024-08-19T14:23:36.419 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3315e-05 2024-08-19T14:23:36.419 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.419 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.419 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:23:36.419 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:35.982343+0000", 2024-08-19T14:23:36.419 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4880000000000001e-06 2024-08-19T14:23:36.419 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.419 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.419 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:23:36.419 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:35.982388+0000", 2024-08-19T14:23:36.419 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.4397000000000002e-05 2024-08-19T14:23:36.419 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.420 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.420 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for peered", 2024-08-19T14:23:36.420 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:35.982394+0000", 2024-08-19T14:23:36.420 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.2809999999999997e-06 2024-08-19T14:23:36.420 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.420 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.420 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:23:36.420 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.964911+0000", 2024-08-19T14:23:36.420 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.98251731499999995 2024-08-19T14:23:36.420 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.420 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.420 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:23:36.420 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.964922+0000", 2024-08-19T14:23:36.420 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0138e-05 2024-08-19T14:23:36.421 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:23:36.421 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:23:36.421 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:23:36.421 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.421 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.421 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25057.0:64615 3.20 3.18b2560 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e246)", 2024-08-19T14:23:36.421 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:17:36.245637+0000", 2024-08-19T14:23:36.421 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 360.15330813899999, 2024-08-19T14:23:36.421 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.71969014899999995, 2024-08-19T14:23:36.421 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:23:36.421 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:23:36.421 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-08-19T14:23:36.421 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:23:36.421 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25057", 2024-08-19T14:23:36.422 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:23:36.422 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 64615 2024-08-19T14:23:36.422 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.422 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:23:36.422 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.422 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:23:36.422 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.245637+0000", 2024-08-19T14:23:36.422 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:23:36.422 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.422 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.422 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:23:36.422 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.245637+0000", 2024-08-19T14:23:36.422 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:23:36.423 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.423 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.423 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:23:36.423 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.245638+0000", 2024-08-19T14:23:36.423 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1340000000000001e-06 2024-08-19T14:23:36.423 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.423 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.423 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:23:36.423 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.245665+0000", 2024-08-19T14:23:36.423 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.7107e-05 2024-08-19T14:23:36.423 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.423 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.423 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:23:36.423 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.245678+0000", 2024-08-19T14:23:36.424 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2911999999999999e-05 2024-08-19T14:23:36.424 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.424 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.424 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:23:36.424 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.245686+0000", 2024-08-19T14:23:36.424 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.8120000000000008e-06 2024-08-19T14:23:36.424 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.424 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.424 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:23:36.424 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.245751+0000", 2024-08-19T14:23:36.424 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.4845999999999995e-05 2024-08-19T14:23:36.424 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.424 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.424 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for peered", 2024-08-19T14:23:36.425 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.245767+0000", 2024-08-19T14:23:36.425 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6708e-05 2024-08-19T14:23:36.425 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.425 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.425 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:23:36.425 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.965166+0000", 2024-08-19T14:23:36.425 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.71939893200000005 2024-08-19T14:23:36.425 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.425 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.425 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:23:36.425 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.965327+0000", 2024-08-19T14:23:36.425 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000160698 2024-08-19T14:23:36.425 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:23:36.425 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:23:36.426 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:23:36.426 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.426 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.426 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:66232 4.6 4.fa550576 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e353)", 2024-08-19T14:23:36.426 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:21:04.816011+0000", 2024-08-19T14:23:36.426 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 151.58293374199999, 2024-08-19T14:23:36.426 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.98858874100000005, 2024-08-19T14:23:36.426 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:23:36.426 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:23:36.426 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-08-19T14:23:36.426 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:23:36.426 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:23:36.426 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:23:36.426 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 66232 2024-08-19T14:23:36.427 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.427 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:23:36.427 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.427 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:23:36.427 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.816011+0000", 2024-08-19T14:23:36.427 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:23:36.427 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.427 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.427 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:23:36.427 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.816011+0000", 2024-08-19T14:23:36.427 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:23:36.427 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.427 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.427 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:23:36.428 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.816013+0000", 2024-08-19T14:23:36.428 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.35e-06 2024-08-19T14:23:36.428 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.428 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.428 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:23:36.428 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.816032+0000", 2024-08-19T14:23:36.428 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9745000000000002e-05 2024-08-19T14:23:36.428 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.428 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.428 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:23:36.428 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.816046+0000", 2024-08-19T14:23:36.428 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3472000000000001e-05 2024-08-19T14:23:36.428 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.429 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.429 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:23:36.429 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.816052+0000", 2024-08-19T14:23:36.429 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.1929999999999998e-06 2024-08-19T14:23:36.429 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.429 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.429 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:23:36.429 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.816105+0000", 2024-08-19T14:23:36.429 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2914000000000002e-05 2024-08-19T14:23:36.429 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.429 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.429 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for peered", 2024-08-19T14:23:36.429 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.816116+0000", 2024-08-19T14:23:36.429 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0772e-05 2024-08-19T14:23:36.430 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.430 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.430 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:23:36.430 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.804584+0000", 2024-08-19T14:23:36.430 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.98846780999999995 2024-08-19T14:23:36.430 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.430 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.430 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:23:36.430 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.804600+0000", 2024-08-19T14:23:36.430 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6484999999999999e-05 2024-08-19T14:23:36.430 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:23:36.430 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:23:36.430 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:23:36.430 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.431 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.431 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:66234 4.6 4.4176ed76 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e353)", 2024-08-19T14:23:36.431 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:21:04.818467+0000", 2024-08-19T14:23:36.431 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 151.58047842799999, 2024-08-19T14:23:36.431 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.98617420499999997, 2024-08-19T14:23:36.431 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:23:36.431 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:23:36.431 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-08-19T14:23:36.431 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:23:36.431 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:23:36.431 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:23:36.431 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 66234 2024-08-19T14:23:36.431 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.431 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:23:36.432 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.432 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:23:36.432 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.818467+0000", 2024-08-19T14:23:36.432 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:23:36.432 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.432 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.432 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:23:36.432 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.818467+0000", 2024-08-19T14:23:36.432 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:23:36.432 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.432 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.432 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:23:36.432 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.818468+0000", 2024-08-19T14:23:36.433 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.0699999999999996e-07 2024-08-19T14:23:36.433 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.433 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.433 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:23:36.433 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.818487+0000", 2024-08-19T14:23:36.433 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8958999999999999e-05 2024-08-19T14:23:36.433 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.433 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.433 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:23:36.433 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.818500+0000", 2024-08-19T14:23:36.433 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3098999999999999e-05 2024-08-19T14:23:36.433 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.433 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.434 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:23:36.434 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.818507+0000", 2024-08-19T14:23:36.434 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.8369999999999997e-06 2024-08-19T14:23:36.434 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.434 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.434 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:23:36.434 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.818561+0000", 2024-08-19T14:23:36.434 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.3165999999999999e-05 2024-08-19T14:23:36.434 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.434 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.434 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for peered", 2024-08-19T14:23:36.434 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.818568+0000", 2024-08-19T14:23:36.434 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.8599999999999993e-06 2024-08-19T14:23:36.435 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.435 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.435 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:23:36.435 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.804628+0000", 2024-08-19T14:23:36.435 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.98605925000000005 2024-08-19T14:23:36.435 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.435 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.435 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:23:36.435 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.804641+0000", 2024-08-19T14:23:36.435 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3127e-05 2024-08-19T14:23:36.435 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:23:36.435 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:23:36.435 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:23:36.435 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.436 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.436 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:66252 4.6 4.4ffe49a6 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e353)", 2024-08-19T14:23:36.436 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:21:04.825959+0000", 2024-08-19T14:23:36.436 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 151.5729863, 2024-08-19T14:23:36.436 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.97870372400000005, 2024-08-19T14:23:36.436 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:23:36.436 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:23:36.436 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-08-19T14:23:36.436 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:23:36.436 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:23:36.436 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:23:36.436 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 66252 2024-08-19T14:23:36.436 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.436 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:23:36.436 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.437 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:23:36.437 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.825959+0000", 2024-08-19T14:23:36.437 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:23:36.437 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.437 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.437 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:23:36.437 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.825959+0000", 2024-08-19T14:23:36.437 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:23:36.437 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.437 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.437 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:23:36.437 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.825960+0000", 2024-08-19T14:23:36.437 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.13e-06 2024-08-19T14:23:36.438 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.438 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.438 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:23:36.438 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.825979+0000", 2024-08-19T14:23:36.438 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9284999999999999e-05 2024-08-19T14:23:36.438 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.438 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.438 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:23:36.438 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.825991+0000", 2024-08-19T14:23:36.438 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1894e-05 2024-08-19T14:23:36.438 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.438 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.438 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:23:36.438 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.825998+0000", 2024-08-19T14:23:36.439 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.1080000000000004e-06 2024-08-19T14:23:36.439 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.439 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.439 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:23:36.439 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.826050+0000", 2024-08-19T14:23:36.439 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2061000000000002e-05 2024-08-19T14:23:36.439 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.439 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.439 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for peered", 2024-08-19T14:23:36.439 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.826057+0000", 2024-08-19T14:23:36.439 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.4389999999999997e-06 2024-08-19T14:23:36.439 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.439 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.439 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:23:36.440 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.804657+0000", 2024-08-19T14:23:36.440 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.97859989999999997 2024-08-19T14:23:36.440 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.440 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.440 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:23:36.440 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.804663+0000", 2024-08-19T14:23:36.440 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.9070000000000004e-06 2024-08-19T14:23:36.440 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:23:36.440 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:23:36.440 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:23:36.440 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.440 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.440 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:66266 4.6 4.2a59efa6 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e353)", 2024-08-19T14:23:36.441 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:21:04.834727+0000", 2024-08-19T14:23:36.441 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 151.56421766899999, 2024-08-19T14:23:36.441 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.96997657800000003, 2024-08-19T14:23:36.441 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:23:36.441 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:23:36.441 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-08-19T14:23:36.441 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:23:36.441 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:23:36.441 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:23:36.441 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 66266 2024-08-19T14:23:36.441 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.441 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:23:36.441 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.441 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:23:36.442 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.834727+0000", 2024-08-19T14:23:36.442 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:23:36.442 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.442 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.442 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:23:36.442 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.834727+0000", 2024-08-19T14:23:36.442 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:23:36.442 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.442 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.442 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:23:36.442 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.834729+0000", 2024-08-19T14:23:36.442 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5579999999999999e-06 2024-08-19T14:23:36.442 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.442 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.443 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:23:36.443 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.834755+0000", 2024-08-19T14:23:36.443 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.6080000000000001e-05 2024-08-19T14:23:36.443 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.443 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.443 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:23:36.443 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.834781+0000", 2024-08-19T14:23:36.443 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.5605999999999998e-05 2024-08-19T14:23:36.443 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.443 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.443 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:23:36.443 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.834812+0000", 2024-08-19T14:23:36.443 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.1229999999999997e-05 2024-08-19T14:23:36.443 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.444 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.444 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:23:36.444 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.834865+0000", 2024-08-19T14:23:36.444 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2611000000000002e-05 2024-08-19T14:23:36.444 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.444 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.444 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for peered", 2024-08-19T14:23:36.444 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.834870+0000", 2024-08-19T14:23:36.444 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.5169999999999999e-06 2024-08-19T14:23:36.444 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.444 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.444 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:23:36.444 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.804695+0000", 2024-08-19T14:23:36.444 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.96982486499999998 2024-08-19T14:23:36.444 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.445 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.445 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:23:36.445 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.804704+0000", 2024-08-19T14:23:36.445 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.1109999999999997e-06 2024-08-19T14:23:36.445 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:23:36.445 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:23:36.445 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:23:36.445 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.445 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.445 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:66269 4.6 4.f7812866 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e353)", 2024-08-19T14:23:36.445 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:21:04.835697+0000", 2024-08-19T14:23:36.445 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 151.56324775900001, 2024-08-19T14:23:36.445 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.96905406199999999, 2024-08-19T14:23:36.445 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:23:36.446 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:23:36.446 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-08-19T14:23:36.446 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:23:36.446 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:23:36.446 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:23:36.446 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 66269 2024-08-19T14:23:36.446 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.446 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:23:36.446 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.446 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:23:36.446 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.835697+0000", 2024-08-19T14:23:36.446 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:23:36.446 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.446 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.447 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:23:36.447 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.835697+0000", 2024-08-19T14:23:36.447 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:23:36.447 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.447 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.447 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:23:36.447 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.835698+0000", 2024-08-19T14:23:36.447 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.5600000000000004e-07 2024-08-19T14:23:36.447 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.447 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.447 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:23:36.447 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.835719+0000", 2024-08-19T14:23:36.447 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0975000000000002e-05 2024-08-19T14:23:36.447 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.448 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.448 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:23:36.448 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.835732+0000", 2024-08-19T14:23:36.448 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2690999999999999e-05 2024-08-19T14:23:36.448 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.448 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.448 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:23:36.448 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.835739+0000", 2024-08-19T14:23:36.448 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.2069999999999998e-06 2024-08-19T14:23:36.448 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.448 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.448 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:23:36.448 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.835781+0000", 2024-08-19T14:23:36.448 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.1743999999999999e-05 2024-08-19T14:23:36.449 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.449 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.449 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for peered", 2024-08-19T14:23:36.449 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.835785+0000", 2024-08-19T14:23:36.449 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.4320000000000001e-06 2024-08-19T14:23:36.449 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.449 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.449 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:23:36.449 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.804742+0000", 2024-08-19T14:23:36.449 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.96895661200000005 2024-08-19T14:23:36.449 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.449 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.449 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:23:36.449 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.804751+0000", 2024-08-19T14:23:36.449 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.4450000000000002e-06 2024-08-19T14:23:36.450 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:23:36.450 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:23:36.450 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:23:36.450 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.450 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.450 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:66273 4.6 4.a206dc36 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e353)", 2024-08-19T14:23:36.450 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:21:04.837306+0000", 2024-08-19T14:23:36.450 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 151.56163933799999, 2024-08-19T14:23:36.450 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.96748620299999999, 2024-08-19T14:23:36.450 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:23:36.450 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:23:36.450 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-08-19T14:23:36.450 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:23:36.450 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:23:36.451 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:23:36.451 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 66273 2024-08-19T14:23:36.451 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.451 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:23:36.451 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.451 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:23:36.451 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.837306+0000", 2024-08-19T14:23:36.451 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:23:36.451 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.451 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.451 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:23:36.451 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.837306+0000", 2024-08-19T14:23:36.451 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:23:36.451 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.452 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.452 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:23:36.452 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.837307+0000", 2024-08-19T14:23:36.452 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.226e-06 2024-08-19T14:23:36.452 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.452 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.452 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:23:36.452 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.837330+0000", 2024-08-19T14:23:36.452 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.3516999999999999e-05 2024-08-19T14:23:36.452 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.452 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.452 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:23:36.452 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.837344+0000", 2024-08-19T14:23:36.452 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3548e-05 2024-08-19T14:23:36.453 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.453 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.453 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:23:36.453 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.837351+0000", 2024-08-19T14:23:36.453 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.3799999999999996e-06 2024-08-19T14:23:36.453 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.453 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.453 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:23:36.453 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.837396+0000", 2024-08-19T14:23:36.453 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.4087e-05 2024-08-19T14:23:36.453 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.453 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.453 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for peered", 2024-08-19T14:23:36.453 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.837400+0000", 2024-08-19T14:23:36.454 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.9389999999999996e-06 2024-08-19T14:23:36.454 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.454 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.454 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:23:36.454 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.804780+0000", 2024-08-19T14:23:36.454 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.96737990600000001 2024-08-19T14:23:36.454 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.454 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.454 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:23:36.454 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.804792+0000", 2024-08-19T14:23:36.454 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1600000000000001e-05 2024-08-19T14:23:36.454 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:23:36.454 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:23:36.454 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:23:36.455 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.455 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.455 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:66344 4.6 4.4505c966 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e353)", 2024-08-19T14:23:36.455 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:21:04.878110+0000", 2024-08-19T14:23:36.455 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 151.52083460200001, 2024-08-19T14:23:36.455 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.92671215799999995, 2024-08-19T14:23:36.455 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:23:36.455 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:23:36.455 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-08-19T14:23:36.455 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:23:36.455 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:23:36.455 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:23:36.455 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 66344 2024-08-19T14:23:36.455 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.456 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:23:36.456 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.456 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:23:36.456 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.878110+0000", 2024-08-19T14:23:36.456 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:23:36.456 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.456 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.456 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:23:36.456 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.878110+0000", 2024-08-19T14:23:36.456 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:23:36.456 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.456 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.456 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:23:36.456 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.878112+0000", 2024-08-19T14:23:36.457 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3689999999999999e-06 2024-08-19T14:23:36.457 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.457 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.457 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:23:36.457 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.878135+0000", 2024-08-19T14:23:36.457 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2932999999999999e-05 2024-08-19T14:23:36.457 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.457 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.457 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:23:36.457 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.878149+0000", 2024-08-19T14:23:36.457 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4239999999999999e-05 2024-08-19T14:23:36.457 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.457 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.457 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:23:36.458 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.878156+0000", 2024-08-19T14:23:36.458 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.1350000000000003e-06 2024-08-19T14:23:36.458 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.458 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.458 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:23:36.458 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.878212+0000", 2024-08-19T14:23:36.458 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.6112000000000001e-05 2024-08-19T14:23:36.458 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.458 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.458 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for peered", 2024-08-19T14:23:36.458 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.878222+0000", 2024-08-19T14:23:36.458 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.2939999999999994e-06 2024-08-19T14:23:36.458 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.458 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.459 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:23:36.459 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.804814+0000", 2024-08-19T14:23:36.459 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.92659212300000005 2024-08-19T14:23:36.459 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.459 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.459 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:23:36.459 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.804823+0000", 2024-08-19T14:23:36.459 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.952e-06 2024-08-19T14:23:36.459 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:23:36.459 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:23:36.459 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:23:36.459 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.459 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.459 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:66352 4.6 4.84354a56 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e353)", 2024-08-19T14:23:36.460 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:21:04.884189+0000", 2024-08-19T14:23:36.460 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 151.514756398, 2024-08-19T14:23:36.460 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.92066624200000002, 2024-08-19T14:23:36.460 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:23:36.460 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:23:36.460 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-08-19T14:23:36.460 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:23:36.460 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:23:36.460 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:23:36.460 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 66352 2024-08-19T14:23:36.460 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.460 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:23:36.460 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.460 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:23:36.461 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.884189+0000", 2024-08-19T14:23:36.461 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:23:36.461 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.461 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.462 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:23:36.462 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.884189+0000", 2024-08-19T14:23:36.462 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:23:36.462 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.462 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.462 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:23:36.462 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.884190+0000", 2024-08-19T14:23:36.462 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2729999999999999e-06 2024-08-19T14:23:36.462 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.462 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.462 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:23:36.462 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.884209+0000", 2024-08-19T14:23:36.462 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.946e-05 2024-08-19T14:23:36.462 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.463 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.463 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:23:36.463 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.884229+0000", 2024-08-19T14:23:36.463 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9086e-05 2024-08-19T14:23:36.463 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.463 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.463 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:23:36.463 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.884237+0000", 2024-08-19T14:23:36.463 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.5760000000000006e-06 2024-08-19T14:23:36.463 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.463 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.463 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:23:36.463 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.884321+0000", 2024-08-19T14:23:36.463 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.3695000000000004e-05 2024-08-19T14:23:36.464 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.464 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.464 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for peered", 2024-08-19T14:23:36.464 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.884329+0000", 2024-08-19T14:23:36.464 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.6100000000000006e-06 2024-08-19T14:23:36.464 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.464 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.464 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:23:36.464 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.804845+0000", 2024-08-19T14:23:36.464 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.92051592900000001 2024-08-19T14:23:36.464 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.464 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.464 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:23:36.464 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.804855+0000", 2024-08-19T14:23:36.465 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.6129999999999993e-06 2024-08-19T14:23:36.465 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:23:36.465 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:23:36.465 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:23:36.465 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.465 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.465 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:66355 4.6 4.67c06196 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e353)", 2024-08-19T14:23:36.465 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:21:04.884802+0000", 2024-08-19T14:23:36.465 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 151.51414292999999, 2024-08-19T14:23:36.465 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.92009072300000005, 2024-08-19T14:23:36.465 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:23:36.465 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:23:36.465 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-08-19T14:23:36.465 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:23:36.466 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:23:36.466 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:23:36.466 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 66355 2024-08-19T14:23:36.466 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.466 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:23:36.466 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.466 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:23:36.466 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.884802+0000", 2024-08-19T14:23:36.466 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:23:36.466 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.466 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.466 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:23:36.466 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.884802+0000", 2024-08-19T14:23:36.467 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:23:36.467 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.467 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.467 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:23:36.467 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.884803+0000", 2024-08-19T14:23:36.467 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.6599999999999994e-07 2024-08-19T14:23:36.467 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.467 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.467 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:23:36.467 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.884827+0000", 2024-08-19T14:23:36.467 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.4334000000000001e-05 2024-08-19T14:23:36.467 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.467 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.467 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:23:36.467 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.884844+0000", 2024-08-19T14:23:36.468 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6330000000000001e-05 2024-08-19T14:23:36.468 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.468 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.468 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:23:36.468 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.884852+0000", 2024-08-19T14:23:36.468 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.9149999999999996e-06 2024-08-19T14:23:36.468 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.468 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.468 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:23:36.468 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.884910+0000", 2024-08-19T14:23:36.468 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7828000000000003e-05 2024-08-19T14:23:36.468 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.468 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.468 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for peered", 2024-08-19T14:23:36.469 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.884918+0000", 2024-08-19T14:23:36.469 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.8349999999999993e-06 2024-08-19T14:23:36.469 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.469 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.469 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:23:36.469 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.804883+0000", 2024-08-19T14:23:36.469 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.91996511400000003 2024-08-19T14:23:36.469 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.469 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.469 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:23:36.469 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.804893+0000", 2024-08-19T14:23:36.469 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.4010000000000003e-06 2024-08-19T14:23:36.469 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:23:36.469 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:23:36.470 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:23:36.470 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.470 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.470 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:66439 4.6 4.e8f25516 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e353)", 2024-08-19T14:23:36.470 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:21:04.937010+0000", 2024-08-19T14:23:36.470 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 151.461935482, 2024-08-19T14:23:36.470 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.86793240199999999, 2024-08-19T14:23:36.470 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:23:36.470 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:23:36.470 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-08-19T14:23:36.470 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:23:36.470 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:23:36.470 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:23:36.470 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 66439 2024-08-19T14:23:36.471 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.471 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:23:36.471 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.471 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:23:36.471 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.937010+0000", 2024-08-19T14:23:36.471 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:23:36.471 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.471 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.471 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:23:36.471 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.937010+0000", 2024-08-19T14:23:36.471 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:23:36.471 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.471 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.471 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:23:36.472 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.937011+0000", 2024-08-19T14:23:36.472 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5120000000000001e-06 2024-08-19T14:23:36.472 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.472 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.472 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:23:36.472 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.937029+0000", 2024-08-19T14:23:36.472 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7706e-05 2024-08-19T14:23:36.472 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.472 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.472 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:23:36.472 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.937043+0000", 2024-08-19T14:23:36.472 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4214e-05 2024-08-19T14:23:36.472 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.472 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.472 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:23:36.473 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.937050+0000", 2024-08-19T14:23:36.473 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4139999999999997e-06 2024-08-19T14:23:36.473 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.473 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.473 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:23:36.473 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.937095+0000", 2024-08-19T14:23:36.473 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.4833999999999998e-05 2024-08-19T14:23:36.473 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.473 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.473 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for peered", 2024-08-19T14:23:36.473 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.937102+0000", 2024-08-19T14:23:36.473 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.3740000000000003e-06 2024-08-19T14:23:36.473 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.473 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.474 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:23:36.474 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.804932+0000", 2024-08-19T14:23:36.474 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.86783048900000004 2024-08-19T14:23:36.474 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.474 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.474 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:23:36.474 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.804942+0000", 2024-08-19T14:23:36.474 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.859e-06 2024-08-19T14:23:36.474 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:23:36.474 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:23:36.474 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:23:36.474 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.474 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.475 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:66443 4.6 4.31b8f586 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e353)", 2024-08-19T14:23:36.475 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:21:04.938019+0000", 2024-08-19T14:23:36.475 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 151.46092633800001, 2024-08-19T14:23:36.475 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.86697104899999999, 2024-08-19T14:23:36.475 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:23:36.475 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:23:36.475 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-08-19T14:23:36.475 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:23:36.475 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:23:36.475 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:23:36.477 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 66443 2024-08-19T14:23:36.477 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.478 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:23:36.478 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.478 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:23:36.478 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.938019+0000", 2024-08-19T14:23:36.478 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:23:36.478 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.478 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.478 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:23:36.478 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.938019+0000", 2024-08-19T14:23:36.478 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:23:36.478 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.478 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.478 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:23:36.479 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.938020+0000", 2024-08-19T14:23:36.479 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.09e-07 2024-08-19T14:23:36.479 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.479 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.479 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:23:36.479 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.938059+0000", 2024-08-19T14:23:36.479 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.9134000000000002e-05 2024-08-19T14:23:36.479 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.479 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.479 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:23:36.479 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.938076+0000", 2024-08-19T14:23:36.479 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.766e-05 2024-08-19T14:23:36.479 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.479 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.480 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:23:36.480 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.938081+0000", 2024-08-19T14:23:36.480 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0969999999999996e-06 2024-08-19T14:23:36.480 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.480 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.480 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:23:36.480 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.938126+0000", 2024-08-19T14:23:36.480 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.4736e-05 2024-08-19T14:23:36.480 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.480 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.480 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for peered", 2024-08-19T14:23:36.480 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.938134+0000", 2024-08-19T14:23:36.480 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.464e-06 2024-08-19T14:23:36.480 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.481 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.481 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:23:36.481 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.804980+0000", 2024-08-19T14:23:36.481 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.86684594400000003 2024-08-19T14:23:36.481 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.481 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.481 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:23:36.481 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.804990+0000", 2024-08-19T14:23:36.481 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0205e-05 2024-08-19T14:23:36.481 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:23:36.481 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:23:36.481 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:23:36.481 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.481 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.481 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:66472 4.6 4.b9e97686 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e353)", 2024-08-19T14:23:36.482 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:21:04.958071+0000", 2024-08-19T14:23:36.482 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 151.44087379499999, 2024-08-19T14:23:36.482 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.84697957599999996, 2024-08-19T14:23:36.482 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:23:36.482 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:23:36.482 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-08-19T14:23:36.482 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:23:36.482 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:23:36.482 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:23:36.482 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 66472 2024-08-19T14:23:36.482 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.482 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:23:36.482 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.483 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:23:36.483 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.958071+0000", 2024-08-19T14:23:36.483 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:23:36.483 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.483 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.483 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:23:36.483 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.958071+0000", 2024-08-19T14:23:36.483 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:23:36.483 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.483 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.483 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:23:36.483 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.958073+0000", 2024-08-19T14:23:36.483 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.533e-06 2024-08-19T14:23:36.483 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.484 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.484 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:23:36.484 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.958126+0000", 2024-08-19T14:23:36.484 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.3430999999999997e-05 2024-08-19T14:23:36.484 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.484 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.484 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:23:36.484 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.958142+0000", 2024-08-19T14:23:36.484 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5259000000000001e-05 2024-08-19T14:23:36.484 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.484 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.484 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:23:36.484 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.958148+0000", 2024-08-19T14:23:36.484 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.7109999999999998e-06 2024-08-19T14:23:36.485 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.485 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.485 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:23:36.485 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.958204+0000", 2024-08-19T14:23:36.485 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.5282999999999999e-05 2024-08-19T14:23:36.485 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.485 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.485 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for peered", 2024-08-19T14:23:36.485 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.958211+0000", 2024-08-19T14:23:36.485 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.5950000000000001e-06 2024-08-19T14:23:36.485 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.485 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.485 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:23:36.486 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.805037+0000", 2024-08-19T14:23:36.486 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.84682633500000004 2024-08-19T14:23:36.486 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.486 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.486 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:23:36.486 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.805051+0000", 2024-08-19T14:23:36.486 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3429e-05 2024-08-19T14:23:36.486 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:23:36.486 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:23:36.486 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:23:36.486 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.486 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.486 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:66507 4.6 4.d2914886 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e353)", 2024-08-19T14:23:36.486 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:21:04.985293+0000", 2024-08-19T14:23:36.487 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 151.41365241599999, 2024-08-19T14:23:36.487 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.81979336000000003, 2024-08-19T14:23:36.487 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:23:36.487 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:23:36.487 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-08-19T14:23:36.487 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:23:36.487 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:23:36.487 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:23:36.487 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 66507 2024-08-19T14:23:36.487 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.487 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:23:36.487 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.487 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:23:36.487 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.985293+0000", 2024-08-19T14:23:36.488 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:23:36.488 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.488 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.488 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:23:36.488 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.985293+0000", 2024-08-19T14:23:36.488 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:23:36.488 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.488 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.488 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:23:36.488 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.985294+0000", 2024-08-19T14:23:36.488 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.06e-06 2024-08-19T14:23:36.488 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.488 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.488 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:23:36.489 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.985312+0000", 2024-08-19T14:23:36.489 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8604999999999999e-05 2024-08-19T14:23:36.489 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.489 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.489 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:23:36.489 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.985324+0000", 2024-08-19T14:23:36.489 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1511e-05 2024-08-19T14:23:36.489 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.489 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.489 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:23:36.489 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.985330+0000", 2024-08-19T14:23:36.489 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.2230000000000004e-06 2024-08-19T14:23:36.489 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.490 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.490 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:23:36.490 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.985375+0000", 2024-08-19T14:23:36.490 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.4808999999999997e-05 2024-08-19T14:23:36.490 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.490 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.490 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for peered", 2024-08-19T14:23:36.490 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.985380+0000", 2024-08-19T14:23:36.490 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4480000000000002e-06 2024-08-19T14:23:36.490 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.490 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.490 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:23:36.490 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.805076+0000", 2024-08-19T14:23:36.490 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.81969612000000003 2024-08-19T14:23:36.491 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.491 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.491 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:23:36.491 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.805086+0000", 2024-08-19T14:23:36.491 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.5840000000000001e-06 2024-08-19T14:23:36.491 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:23:36.491 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:23:36.491 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:23:36.491 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.491 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.491 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:66512 4.6 4.e5cfc156 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e353)", 2024-08-19T14:23:36.491 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:21:04.992563+0000", 2024-08-19T14:23:36.491 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 151.406381735, 2024-08-19T14:23:36.492 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.812551829, 2024-08-19T14:23:36.492 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:23:36.492 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:23:36.492 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-08-19T14:23:36.492 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:23:36.492 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:23:36.492 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:23:36.492 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 66512 2024-08-19T14:23:36.492 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.492 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:23:36.492 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.492 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:23:36.492 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.992563+0000", 2024-08-19T14:23:36.493 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:23:36.493 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.493 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.493 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:23:36.493 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.992563+0000", 2024-08-19T14:23:36.493 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:23:36.493 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.493 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.493 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:23:36.493 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.992564+0000", 2024-08-19T14:23:36.493 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.7600000000000006e-07 2024-08-19T14:23:36.493 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.493 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.494 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:23:36.494 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.992582+0000", 2024-08-19T14:23:36.494 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7362000000000001e-05 2024-08-19T14:23:36.494 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.494 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.494 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:23:36.494 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.992594+0000", 2024-08-19T14:23:36.494 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1956999999999999e-05 2024-08-19T14:23:36.494 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.494 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.494 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:23:36.494 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.992601+0000", 2024-08-19T14:23:36.494 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.2599999999999999e-06 2024-08-19T14:23:36.494 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.495 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.495 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:23:36.495 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.992644+0000", 2024-08-19T14:23:36.495 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.3111000000000001e-05 2024-08-19T14:23:36.495 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.495 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.495 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for peered", 2024-08-19T14:23:36.495 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.992649+0000", 2024-08-19T14:23:36.495 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.9960000000000001e-06 2024-08-19T14:23:36.495 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.495 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.495 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:23:36.495 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.805107+0000", 2024-08-19T14:23:36.496 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.81245753099999996 2024-08-19T14:23:36.496 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.496 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.496 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:23:36.496 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.805115+0000", 2024-08-19T14:23:36.496 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.636e-06 2024-08-19T14:23:36.496 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:23:36.496 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:23:36.496 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:23:36.496 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.496 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.496 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:66538 4.6 4.94521946 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e353)", 2024-08-19T14:23:36.496 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:21:05.030374+0000", 2024-08-19T14:23:36.496 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 151.368570737, 2024-08-19T14:23:36.497 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.77477269400000004, 2024-08-19T14:23:36.497 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:23:36.497 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:23:36.497 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-08-19T14:23:36.497 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:23:36.497 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:23:36.497 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:23:36.497 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 66538 2024-08-19T14:23:36.497 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.497 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:23:36.497 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.497 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:23:36.497 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.030374+0000", 2024-08-19T14:23:36.497 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:23:36.498 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.498 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.498 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:23:36.498 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.030374+0000", 2024-08-19T14:23:36.498 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:23:36.498 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.498 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.498 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:23:36.498 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.030376+0000", 2024-08-19T14:23:36.498 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1850000000000001e-06 2024-08-19T14:23:36.498 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.498 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.498 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:23:36.498 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.030394+0000", 2024-08-19T14:23:36.499 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8159999999999999e-05 2024-08-19T14:23:36.499 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.499 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.499 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:23:36.499 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.030406+0000", 2024-08-19T14:23:36.499 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2004000000000001e-05 2024-08-19T14:23:36.499 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.499 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.499 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:23:36.499 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.030411+0000", 2024-08-19T14:23:36.499 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2000000000000002e-06 2024-08-19T14:23:36.499 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.499 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.499 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:23:36.500 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.030463+0000", 2024-08-19T14:23:36.500 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2227e-05 2024-08-19T14:23:36.500 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.500 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.500 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for peered", 2024-08-19T14:23:36.500 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.030473+0000", 2024-08-19T14:23:36.500 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0353000000000001e-05 2024-08-19T14:23:36.500 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.500 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.500 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:23:36.500 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.805138+0000", 2024-08-19T14:23:36.500 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.77466448700000001 2024-08-19T14:23:36.500 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.500 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.501 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:23:36.501 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.805147+0000", 2024-08-19T14:23:36.501 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.0780000000000002e-06 2024-08-19T14:23:36.501 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:23:36.501 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:23:36.501 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:23:36.501 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.501 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.501 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:66550 4.6 4.8249c486 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e353)", 2024-08-19T14:23:36.501 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:21:05.061787+0000", 2024-08-19T14:23:36.501 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 151.337158085, 2024-08-19T14:23:36.501 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.74339024399999998, 2024-08-19T14:23:36.501 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:23:36.501 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:23:36.502 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-08-19T14:23:36.502 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:23:36.502 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:23:36.502 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:23:36.502 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 66550 2024-08-19T14:23:36.502 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.502 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:23:36.502 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.502 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:23:36.502 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.061787+0000", 2024-08-19T14:23:36.502 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:23:36.502 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.502 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.502 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:23:36.503 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.061787+0000", 2024-08-19T14:23:36.503 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:23:36.503 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.503 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.503 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:23:36.503 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.061788+0000", 2024-08-19T14:23:36.503 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.263e-06 2024-08-19T14:23:36.503 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.503 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.503 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:23:36.503 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.061829+0000", 2024-08-19T14:23:36.503 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.0988000000000001e-05 2024-08-19T14:23:36.503 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.503 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.503 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:23:36.504 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.061840+0000", 2024-08-19T14:23:36.504 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.112e-05 2024-08-19T14:23:36.504 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.504 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.504 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:23:36.504 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.061846+0000", 2024-08-19T14:23:36.504 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.3299999999999998e-06 2024-08-19T14:23:36.504 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.504 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.504 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:23:36.504 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.061902+0000", 2024-08-19T14:23:36.504 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.6368e-05 2024-08-19T14:23:36.504 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.504 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.504 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for peered", 2024-08-19T14:23:36.505 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.061911+0000", 2024-08-19T14:23:36.505 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.3910000000000004e-06 2024-08-19T14:23:36.505 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.505 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.505 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:23:36.505 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.805169+0000", 2024-08-19T14:23:36.505 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.74325783099999998 2024-08-19T14:23:36.505 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:36.505 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:36.505 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:23:36.505 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.805177+0000", 2024-08-19T14:23:36.505 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.9529999999999999e-06 2024-08-19T14:23:36.505 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:23:36.505 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:23:36.506 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:23:36.506 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:23:36.506 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:23:36.506 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:23:36.506 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:23:36.534 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:36.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:36.535 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:23:36.635 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:23:36.670 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:36.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:36.671 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:23:36.772 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:23:36.792 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-19T14:23:36.793 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-19T14:23:36.793 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph tell osd.7 'trim stale osdmaps' 2024-08-19T14:23:36.799 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:36.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:36.800 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:23:36.853 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:23:36.901 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:23:36.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:36.923 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:36.923 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:23:36.983 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is set 2024-08-19T14:23:36.997 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:36 smithi031 ceph-mon[16336]: pgmap v907: 47 pgs: 47 active+clean; 3.2 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 745 op/s 2024-08-19T14:23:36.997 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:36 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/4287793998' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:23:36.997 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:36 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:23:37.024 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:23:37.044 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:37.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:37.045 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:23:37.146 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:23:37.174 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:37.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:37.175 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:23:37.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:36 smithi032 ceph-mon[20004]: pgmap v907: 47 pgs: 47 active+clean; 3.2 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 745 op/s 2024-08-19T14:23:37.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:36 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/4287793998' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:23:37.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:36 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:23:37.276 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:23:37.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:36 smithi045 ceph-mon[20143]: pgmap v907: 47 pgs: 47 active+clean; 3.2 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 745 op/s 2024-08-19T14:23:37.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:36 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/4287793998' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:23:37.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:36 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:23:37.302 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:37.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:37.303 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:23:37.404 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:23:37.409 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-08-19T14:23:37.432 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:37.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:37.434 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:23:37.535 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:23:37.574 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:37.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:37.575 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:23:37.676 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:23:37.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:37.707 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:37.707 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:23:37.743 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:23:37.782 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:23:37.795 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:23:37.808 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:23:37.833 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:37.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:37.834 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:23:37.935 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:23:37.964 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:37.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:37.965 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:23:38.066 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:23:38.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:38.096 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:38.097 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:23:38.150 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:37 smithi045 ceph-mon[20143]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:23:38.150 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:37 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:23:38.150 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:37 smithi045 ceph-mon[20143]: osdmap e397: 12 total, 12 up, 9 in 2024-08-19T14:23:38.198 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:23:38.205 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:37 smithi031 ceph-mon[16336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:23:38.205 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:37 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:23:38.206 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:37 smithi031 ceph-mon[16336]: osdmap e397: 12 total, 12 up, 9 in 2024-08-19T14:23:38.246 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:38.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:38.247 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:23:38.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:37 smithi032 ceph-mon[20004]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:23:38.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:37 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:23:38.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:37 smithi032 ceph-mon[20004]: osdmap e397: 12 total, 12 up, 9 in 2024-08-19T14:23:38.283 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:23:38.283 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [], 2024-08-19T14:23:38.283 INFO:teuthology.orchestra.run.smithi045.stdout: "num_ops": 0 2024-08-19T14:23:38.283 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:23:38.348 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:23:38.372 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:38.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:38.373 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:23:38.474 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:23:38.516 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:38.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:38.517 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:23:38.618 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:23:38.633 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:23:38.633 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:23:38.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:38.644 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:38.644 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:23:38.675 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:23:38.675 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:23:38.719 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:23:38.719 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:23:38.745 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:23:38.757 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:23:38.758 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:23:38.769 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:38.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:38.770 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:23:38.798 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:23:38.799 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:23:38.839 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:23:38.840 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:23:38.870 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:23:38.876 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:23:38.876 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:23:38.893 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:38.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:38.894 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:23:38.914 INFO:teuthology.orchestra.run.smithi031.stderr:no valid command found; 10 closest matches: 2024-08-19T14:23:38.914 INFO:teuthology.orchestra.run.smithi031.stderr:0 2024-08-19T14:23:38.915 INFO:teuthology.orchestra.run.smithi031.stderr:1 2024-08-19T14:23:38.915 INFO:teuthology.orchestra.run.smithi031.stderr:2 2024-08-19T14:23:38.915 INFO:teuthology.orchestra.run.smithi031.stderr:abort 2024-08-19T14:23:38.915 INFO:teuthology.orchestra.run.smithi031.stderr:assert 2024-08-19T14:23:38.915 INFO:teuthology.orchestra.run.smithi031.stderr:bench [] [] [] [] 2024-08-19T14:23:38.915 INFO:teuthology.orchestra.run.smithi031.stderr:bluefs debug_inject_read_zeros 2024-08-19T14:23:38.915 INFO:teuthology.orchestra.run.smithi031.stderr:bluefs files list 2024-08-19T14:23:38.915 INFO:teuthology.orchestra.run.smithi031.stderr:bluefs stats 2024-08-19T14:23:38.915 INFO:teuthology.orchestra.run.smithi031.stderr:bluestore allocator dump block 2024-08-19T14:23:38.915 INFO:teuthology.orchestra.run.smithi031.stderr:Error EINVAL: invalid command 2024-08-19T14:23:38.916 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:23:38.916 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:23:38.954 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-08-19T14:23:38.962 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:23:38.963 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:23:38.995 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:23:39.002 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:23:39.002 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:23:39.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:39.027 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:39.028 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:23:39.036 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:23:39.036 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:23:39.072 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:23:39.072 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:23:39.128 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:23:39.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:39.166 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:39.166 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:23:39.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:39 smithi032 ceph-mon[20004]: pgmap v909: 47 pgs: 47 active+clean; 3.1 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 913 op/s 2024-08-19T14:23:39.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:39 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3861715230' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:23:39.267 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:23:39.294 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:23:39.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:39 smithi045 ceph-mon[20143]: pgmap v909: 47 pgs: 47 active+clean; 3.1 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 913 op/s 2024-08-19T14:23:39.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:39 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3861715230' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:23:39.296 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:39.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:39.298 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:23:39.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:39 smithi031 ceph-mon[16336]: pgmap v909: 47 pgs: 47 active+clean; 3.1 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 913 op/s 2024-08-19T14:23:39.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:39 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3861715230' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:23:39.399 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:23:39.426 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:39.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:39.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:23:39.528 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:23:39.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:39.568 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:39.568 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:23:39.669 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:23:39.702 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-19T14:23:39.703 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:39.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:39.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:23:39.806 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:23:39.822 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:23:39.823 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:23:39.830 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:39.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:39.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:23:39.931 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:23:39.955 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:39.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:39.956 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:23:40.057 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:23:40.083 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:40.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:40.084 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:23:40.185 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:23:40.214 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:40.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:40.215 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:23:40.316 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:23:40.341 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:40.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:40.342 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:23:40.423 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-19T14:23:40.424 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-19T14:23:40.424 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph tell osd.8 'trim stale osdmaps' 2024-08-19T14:23:40.442 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:23:40.511 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-08-19T14:23:40.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:40.535 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:40.535 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:23:40.617 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:23:40.636 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:23:40.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:40.706 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:40.706 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:23:40.807 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:23:40.808 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:23:40.828 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:40.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:40.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:23:40.930 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:23:40.952 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:40.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:40.953 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:23:41.054 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:23:41.084 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:41 smithi031 ceph-mon[16336]: pgmap v910: 47 pgs: 47 active+clean; 3.0 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 918 op/s 2024-08-19T14:23:41.094 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:41.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:41.096 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:23:41.197 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:23:41.217 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:41 smithi045 ceph-mon[20143]: pgmap v910: 47 pgs: 47 active+clean; 3.0 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 918 op/s 2024-08-19T14:23:41.224 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:41.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:41.225 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:23:41.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:41 smithi032 ceph-mon[20004]: pgmap v910: 47 pgs: 47 active+clean; 3.0 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 918 op/s 2024-08-19T14:23:41.282 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:23:41.325 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:23:41.354 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:23:41.354 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:23:41.371 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:41.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:41.372 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:23:41.473 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:23:41.499 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:41.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:41.500 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:23:41.601 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:23:41.636 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:41.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:41.637 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:23:41.738 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:23:41.767 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:41.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:41.768 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:23:41.869 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:23:41.888 INFO:teuthology.orchestra.run.smithi031.stderr:no valid command found; 10 closest matches: 2024-08-19T14:23:41.888 INFO:teuthology.orchestra.run.smithi031.stderr:0 2024-08-19T14:23:41.889 INFO:teuthology.orchestra.run.smithi031.stderr:1 2024-08-19T14:23:41.889 INFO:teuthology.orchestra.run.smithi031.stderr:2 2024-08-19T14:23:41.889 INFO:teuthology.orchestra.run.smithi031.stderr:abort 2024-08-19T14:23:41.889 INFO:teuthology.orchestra.run.smithi031.stderr:assert 2024-08-19T14:23:41.889 INFO:teuthology.orchestra.run.smithi031.stderr:bench [] [] [] [] 2024-08-19T14:23:41.889 INFO:teuthology.orchestra.run.smithi031.stderr:bluefs debug_inject_read_zeros 2024-08-19T14:23:41.889 INFO:teuthology.orchestra.run.smithi031.stderr:bluefs files list 2024-08-19T14:23:41.889 INFO:teuthology.orchestra.run.smithi031.stderr:bluefs stats 2024-08-19T14:23:41.889 INFO:teuthology.orchestra.run.smithi031.stderr:bluestore allocator dump block 2024-08-19T14:23:41.889 INFO:teuthology.orchestra.run.smithi031.stderr:Error EINVAL: invalid command 2024-08-19T14:23:41.892 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:41.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:41.893 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:23:41.932 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:23:41.994 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:23:42.020 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is unset 2024-08-19T14:23:42.021 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:42.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:42.023 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:23:42.056 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-08-19T14:23:42.123 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:23:42.149 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:42.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:42.151 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:23:42.251 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:23:42.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:42.286 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:42.286 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:23:42.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:42 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/4046864880' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:23:42.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:42 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:23:42.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:42 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/4046864880' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:23:42.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:42 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:23:42.387 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:23:42.397 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:23:42.417 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:42.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:42.418 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:23:42.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:42 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/4046864880' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:23:42.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:42 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:23:42.519 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:23:42.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:42.550 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:42.550 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:23:42.652 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:23:42.677 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:42.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:42.678 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:23:42.779 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:23:42.808 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:42.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:42.809 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:23:42.910 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:23:42.936 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:42.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:42.937 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:23:43.038 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:23:43.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:43 smithi045 ceph-mon[20143]: pgmap v911: 47 pgs: 47 active+clean; 2.8 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 913 op/s 2024-08-19T14:23:43.053 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:43 smithi045 ceph-mon[20143]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:23:43.053 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:43 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:23:43.053 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:43 smithi045 ceph-mon[20143]: osdmap e398: 12 total, 12 up, 9 in 2024-08-19T14:23:43.053 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:43 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:23:43.053 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:43 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:23:43.068 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:43.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:43.069 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:23:43.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:43 smithi031 ceph-mon[16336]: pgmap v911: 47 pgs: 47 active+clean; 2.8 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 913 op/s 2024-08-19T14:23:43.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:43 smithi031 ceph-mon[16336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:23:43.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:43 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:23:43.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:43 smithi031 ceph-mon[16336]: osdmap e398: 12 total, 12 up, 9 in 2024-08-19T14:23:43.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:43 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:23:43.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:43 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:23:43.170 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:23:43.194 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:23:43.194 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:23:43.198 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:43.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:43.199 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:23:43.300 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:23:43.331 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-19T14:23:43.331 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-19T14:23:43.332 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph tell osd.9 'trim stale osdmaps' 2024-08-19T14:23:43.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:43.335 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:43.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:23:43.437 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:23:43.462 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:43.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:43.464 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:23:43.468 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:23:43.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:43 smithi032 ceph-mon[20004]: pgmap v911: 47 pgs: 47 active+clean; 2.8 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 913 op/s 2024-08-19T14:23:43.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:43 smithi032 ceph-mon[20004]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:23:43.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:43 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:23:43.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:43 smithi032 ceph-mon[20004]: osdmap e398: 12 total, 12 up, 9 in 2024-08-19T14:23:43.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:43 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:23:43.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:43 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:23:43.566 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:23:43.587 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:43.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:43.588 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:23:43.689 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:23:43.718 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:43.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:43.719 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:23:43.820 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:23:43.852 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:43.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:43.853 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:23:43.954 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:23:43.980 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:43.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:43.981 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:23:43.983 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-08-19T14:23:44.039 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:23:44.053 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:23:44.053 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:23:44.082 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:23:44.092 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:23:44.092 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:23:44.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:44.115 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:44.115 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:23:44.123 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:44 smithi031 ceph-mon[16336]: osdmap e399: 12 total, 12 up, 9 in 2024-08-19T14:23:44.123 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:44 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3023279468' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:23:44.124 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:23:44.124 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:23:44.156 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:23:44.157 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:23:44.191 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:23:44.192 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:23:44.216 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:23:44.219 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:23:44.219 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:23:44.253 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:44.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:44.254 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:23:44.264 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:23:44.265 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:23:44.289 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:23:44.290 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:23:44.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:44 smithi045 ceph-mon[20143]: osdmap e399: 12 total, 12 up, 9 in 2024-08-19T14:23:44.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:44 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3023279468' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:23:44.320 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:23:44.321 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:23:44.349 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:23:44.355 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:23:44.357 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:23:44.357 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:23:44.379 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:44.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:44.380 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:23:44.394 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:23:44.394 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:23:44.428 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:23:44.428 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:23:44.480 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:23:44.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:44 smithi032 ceph-mon[20004]: osdmap e399: 12 total, 12 up, 9 in 2024-08-19T14:23:44.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:44 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3023279468' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:23:44.510 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:44.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:44.511 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:23:44.612 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:23:44.642 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:44.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:44.643 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:23:44.744 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:23:44.769 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:44.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:44.770 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:23:44.839 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-19T14:23:44.858 INFO:teuthology.orchestra.run.smithi031.stderr:no valid command found; 10 closest matches: 2024-08-19T14:23:44.858 INFO:teuthology.orchestra.run.smithi031.stderr:0 2024-08-19T14:23:44.858 INFO:teuthology.orchestra.run.smithi031.stderr:1 2024-08-19T14:23:44.858 INFO:teuthology.orchestra.run.smithi031.stderr:2 2024-08-19T14:23:44.858 INFO:teuthology.orchestra.run.smithi031.stderr:abort 2024-08-19T14:23:44.858 INFO:teuthology.orchestra.run.smithi031.stderr:assert 2024-08-19T14:23:44.858 INFO:teuthology.orchestra.run.smithi031.stderr:bench [] [] [] [] 2024-08-19T14:23:44.858 INFO:teuthology.orchestra.run.smithi031.stderr:bluefs debug_inject_read_zeros 2024-08-19T14:23:44.858 INFO:teuthology.orchestra.run.smithi031.stderr:bluefs files list 2024-08-19T14:23:44.859 INFO:teuthology.orchestra.run.smithi031.stderr:bluefs stats 2024-08-19T14:23:44.859 INFO:teuthology.orchestra.run.smithi031.stderr:bluestore allocator dump block 2024-08-19T14:23:44.859 INFO:teuthology.orchestra.run.smithi031.stderr:Error EINVAL: invalid command 2024-08-19T14:23:44.870 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:23:44.901 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:44.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:44.902 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:23:44.910 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:23:44.911 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [], 2024-08-19T14:23:44.911 INFO:teuthology.orchestra.run.smithi045.stdout: "complaint_time": 30, 2024-08-19T14:23:44.911 INFO:teuthology.orchestra.run.smithi045.stdout: "num_blocked_ops": 0 2024-08-19T14:23:44.911 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:23:45.004 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:23:45.030 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:45.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:45.031 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:23:45.132 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:23:45.153 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:45.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:45.154 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:23:45.255 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:23:45.285 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:45.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:45.286 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:23:45.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:45 smithi045 ceph-mon[20143]: 4.5 scrub starts 2024-08-19T14:23:45.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:45 smithi045 ceph-mon[20143]: pgmap v914: 47 pgs: 47 active+clean; 2.7 GiB data, 12 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 1.07k op/s 2024-08-19T14:23:45.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:45 smithi045 ceph-mon[20143]: 4.5 scrub ok 2024-08-19T14:23:45.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:45 smithi031 ceph-mon[16336]: 4.5 scrub starts 2024-08-19T14:23:45.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:45 smithi031 ceph-mon[16336]: pgmap v914: 47 pgs: 47 active+clean; 2.7 GiB data, 12 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 1.07k op/s 2024-08-19T14:23:45.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:45 smithi031 ceph-mon[16336]: 4.5 scrub ok 2024-08-19T14:23:45.387 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:23:45.415 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:45.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:45.416 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:23:45.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:45 smithi032 ceph-mon[20004]: 4.5 scrub starts 2024-08-19T14:23:45.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:45 smithi032 ceph-mon[20004]: pgmap v914: 47 pgs: 47 active+clean; 2.7 GiB data, 12 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 1.07k op/s 2024-08-19T14:23:45.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:45 smithi032 ceph-mon[20004]: 4.5 scrub ok 2024-08-19T14:23:45.517 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:23:45.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:45.544 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:45.545 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:23:45.561 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-08-19T14:23:45.645 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:23:45.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:45.685 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:45.686 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:23:45.786 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:23:45.827 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:45.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:45.828 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:23:45.929 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:23:45.938 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:23:45.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:45.963 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:45.964 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:23:46.065 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:23:46.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:46.097 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:46.097 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:23:46.199 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:23:46.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:46.228 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:46.228 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:23:46.330 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:23:46.366 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:46.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:46 smithi031 ceph-mon[16336]: 1.9 scrub starts 2024-08-19T14:23:46.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:46 smithi031 ceph-mon[16336]: 1.9 scrub ok 2024-08-19T14:23:46.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:46 smithi031 ceph-mon[16336]: 1.5 scrub starts 2024-08-19T14:23:46.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:46 smithi031 ceph-mon[16336]: 1.5 scrub ok 2024-08-19T14:23:46.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:46.369 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:23:46.378 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:46 smithi045 ceph-mon[20143]: 1.9 scrub starts 2024-08-19T14:23:46.379 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:46 smithi045 ceph-mon[20143]: 1.9 scrub ok 2024-08-19T14:23:46.379 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:46 smithi045 ceph-mon[20143]: 1.5 scrub starts 2024-08-19T14:23:46.379 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:46 smithi045 ceph-mon[20143]: 1.5 scrub ok 2024-08-19T14:23:46.470 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:23:46.499 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:46.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:46.501 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:23:46.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:46 smithi032 ceph-mon[20004]: 1.9 scrub starts 2024-08-19T14:23:46.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:46 smithi032 ceph-mon[20004]: 1.9 scrub ok 2024-08-19T14:23:46.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:46 smithi032 ceph-mon[20004]: 1.5 scrub starts 2024-08-19T14:23:46.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:46 smithi032 ceph-mon[20004]: 1.5 scrub ok 2024-08-19T14:23:46.518 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:23:46.518 INFO:teuthology.orchestra.run.smithi045.stdout: "size": 20, 2024-08-19T14:23:46.518 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 600, 2024-08-19T14:23:46.519 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [ 2024-08-19T14:23:46.519 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.519 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24580.0:159035 2.c 2:32456cd8:::benchmark_data_smithi045_46415_object35754:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e167)", 2024-08-19T14:23:46.519 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:14:59.419140+0000", 2024-08-19T14:23:46.519 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 527.09512508, 2024-08-19T14:23:46.519 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.21467188200000001, 2024-08-19T14:23:46.519 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:23:46.519 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:23:46.520 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:23:46.520 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:23:46.520 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24580", 2024-08-19T14:23:46.520 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:23:46.520 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 159035 2024-08-19T14:23:46.520 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.520 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:23:46.520 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.520 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:23:46.521 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.419140+0000", 2024-08-19T14:23:46.521 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:23:46.521 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.521 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.521 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:23:46.521 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.419140+0000", 2024-08-19T14:23:46.521 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:23:46.521 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.522 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.522 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:23:46.522 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.419141+0000", 2024-08-19T14:23:46.522 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4079999999999999e-06 2024-08-19T14:23:46.522 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.522 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.522 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:23:46.522 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.419146+0000", 2024-08-19T14:23:46.523 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0610000000000003e-06 2024-08-19T14:23:46.523 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.523 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.523 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:23:46.523 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.419163+0000", 2024-08-19T14:23:46.523 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6770999999999999e-05 2024-08-19T14:23:46.523 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.523 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.524 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:23:46.524 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.419172+0000", 2024-08-19T14:23:46.524 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.276e-06 2024-08-19T14:23:46.524 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.524 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.524 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:23:46.524 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.419223+0000", 2024-08-19T14:23:46.524 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1588999999999997e-05 2024-08-19T14:23:46.525 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.525 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.525 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:23:46.525 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.419320+0000", 2024-08-19T14:23:46.525 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.7237000000000004e-05 2024-08-19T14:23:46.525 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.525 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.525 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,1", 2024-08-19T14:23:46.525 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.419386+0000", 2024-08-19T14:23:46.525 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.6093999999999994e-05 2024-08-19T14:23:46.525 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.525 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.526 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:23:46.526 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.422728+0000", 2024-08-19T14:23:46.526 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0033416489999999999 2024-08-19T14:23:46.526 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.526 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.526 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:23:46.526 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.439269+0000", 2024-08-19T14:23:46.526 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.016540764999999999 2024-08-19T14:23:46.526 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.526 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.526 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:23:46.526 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.633710+0000", 2024-08-19T14:23:46.526 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.194440742 2024-08-19T14:23:46.527 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.527 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.527 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:23:46.527 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.633767+0000", 2024-08-19T14:23:46.527 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7210999999999998e-05 2024-08-19T14:23:46.527 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.527 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.527 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:23:46.527 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.633812+0000", 2024-08-19T14:23:46.527 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.5079e-05 2024-08-19T14:23:46.527 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:23:46.527 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:23:46.527 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:23:46.527 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.528 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.528 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24580.0:159041 2.c 2:326ec62a:::benchmark_data_smithi045_46415_object35760:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e167)", 2024-08-19T14:23:46.528 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:14:59.424792+0000", 2024-08-19T14:23:46.528 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 527.08947257600005, 2024-08-19T14:23:46.528 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.25135169800000001, 2024-08-19T14:23:46.528 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:23:46.528 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:23:46.528 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:23:46.528 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:23:46.528 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24580", 2024-08-19T14:23:46.528 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:23:46.528 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 159041 2024-08-19T14:23:46.528 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.528 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:23:46.529 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.529 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:23:46.529 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.424792+0000", 2024-08-19T14:23:46.529 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:23:46.529 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.529 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.529 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:23:46.529 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.424792+0000", 2024-08-19T14:23:46.529 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:23:46.529 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.529 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.529 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:23:46.530 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.424794+0000", 2024-08-19T14:23:46.530 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.883e-06 2024-08-19T14:23:46.530 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.530 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.530 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:23:46.530 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.424800+0000", 2024-08-19T14:23:46.530 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4349999999999996e-06 2024-08-19T14:23:46.530 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.530 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.530 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:23:46.530 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.424813+0000", 2024-08-19T14:23:46.530 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3619e-05 2024-08-19T14:23:46.530 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.531 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.531 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:23:46.531 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.424821+0000", 2024-08-19T14:23:46.531 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4649999999999997e-06 2024-08-19T14:23:46.531 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.531 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.531 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:23:46.531 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.424864+0000", 2024-08-19T14:23:46.531 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.2786e-05 2024-08-19T14:23:46.531 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.531 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.531 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:23:46.531 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.424944+0000", 2024-08-19T14:23:46.531 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.0628000000000001e-05 2024-08-19T14:23:46.531 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.532 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.532 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,1", 2024-08-19T14:23:46.532 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.425008+0000", 2024-08-19T14:23:46.532 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.3223000000000001e-05 2024-08-19T14:23:46.532 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.532 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.532 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:23:46.532 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.429480+0000", 2024-08-19T14:23:46.532 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0044724830000000002 2024-08-19T14:23:46.532 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.532 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.532 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:23:46.532 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.452535+0000", 2024-08-19T14:23:46.532 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.023055386000000001 2024-08-19T14:23:46.532 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.533 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.533 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:23:46.533 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.676047+0000", 2024-08-19T14:23:46.533 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.22351151399999999 2024-08-19T14:23:46.533 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.533 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.533 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:23:46.533 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.676101+0000", 2024-08-19T14:23:46.533 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4237000000000003e-05 2024-08-19T14:23:46.533 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.533 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.533 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:23:46.533 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.676144+0000", 2024-08-19T14:23:46.533 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.3038999999999999e-05 2024-08-19T14:23:46.534 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:23:46.534 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:23:46.534 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:23:46.534 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.534 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.534 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24580.0:159045 2.c 2:30c9429c:::benchmark_data_smithi045_46415_object35764:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e167)", 2024-08-19T14:23:46.534 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:14:59.429858+0000", 2024-08-19T14:23:46.534 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 527.08440662299995, 2024-08-19T14:23:46.534 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.24643119899999999, 2024-08-19T14:23:46.534 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:23:46.535 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:23:46.535 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:23:46.535 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:23:46.535 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24580", 2024-08-19T14:23:46.535 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:23:46.535 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 159045 2024-08-19T14:23:46.536 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.536 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:23:46.536 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.536 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:23:46.536 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.429858+0000", 2024-08-19T14:23:46.536 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:23:46.536 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.536 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.536 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:23:46.536 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.429858+0000", 2024-08-19T14:23:46.537 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:23:46.537 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.537 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.537 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:23:46.537 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.429860+0000", 2024-08-19T14:23:46.537 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.531e-06 2024-08-19T14:23:46.537 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.537 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.537 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:23:46.537 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.429865+0000", 2024-08-19T14:23:46.537 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0989999999999998e-06 2024-08-19T14:23:46.537 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.537 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.537 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:23:46.538 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.429875+0000", 2024-08-19T14:23:46.538 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0043e-05 2024-08-19T14:23:46.538 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.538 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.538 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:23:46.538 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.429891+0000", 2024-08-19T14:23:46.538 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5539000000000001e-05 2024-08-19T14:23:46.538 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.538 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.538 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:23:46.539 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.429951+0000", 2024-08-19T14:23:46.539 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.0528000000000001e-05 2024-08-19T14:23:46.539 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.539 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.539 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:23:46.539 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.430061+0000", 2024-08-19T14:23:46.539 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000110203 2024-08-19T14:23:46.539 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.539 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.539 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,1", 2024-08-19T14:23:46.539 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.430129+0000", 2024-08-19T14:23:46.540 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.7970000000000001e-05 2024-08-19T14:23:46.540 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.540 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.540 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:23:46.540 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.435692+0000", 2024-08-19T14:23:46.540 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0055628550000000002 2024-08-19T14:23:46.540 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.540 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.540 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:23:46.540 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.459086+0000", 2024-08-19T14:23:46.540 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.023393942000000001 2024-08-19T14:23:46.540 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.540 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.540 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:23:46.540 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.676201+0000", 2024-08-19T14:23:46.541 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.21711492499999999 2024-08-19T14:23:46.541 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.541 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.541 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:23:46.541 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.676252+0000", 2024-08-19T14:23:46.541 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1334e-05 2024-08-19T14:23:46.541 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.541 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.541 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:23:46.541 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:14:59.676290+0000", 2024-08-19T14:23:46.541 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.7230000000000001e-05 2024-08-19T14:23:46.541 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:23:46.541 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:23:46.541 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:23:46.542 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.542 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.542 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25057.0:63327 3.1e 3:79dc30b5:::benchmark_data_smithi045_59973_object63326:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e244)", 2024-08-19T14:23:46.542 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:17:33.888774+0000", 2024-08-19T14:23:46.542 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 372.62549061800001, 2024-08-19T14:23:46.542 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0936514980000001, 2024-08-19T14:23:46.542 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:23:46.542 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:23:46.542 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-08-19T14:23:46.542 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:23:46.542 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25057", 2024-08-19T14:23:46.542 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:23:46.542 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 63327 2024-08-19T14:23:46.542 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.543 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:23:46.543 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.543 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:23:46.543 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:33.888774+0000", 2024-08-19T14:23:46.543 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:23:46.543 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.543 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.543 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:23:46.543 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:33.888774+0000", 2024-08-19T14:23:46.543 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:23:46.543 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.543 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.543 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:23:46.543 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:33.888776+0000", 2024-08-19T14:23:46.544 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9190000000000002e-06 2024-08-19T14:23:46.544 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.544 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.544 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:23:46.544 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:33.888802+0000", 2024-08-19T14:23:46.544 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.5134e-05 2024-08-19T14:23:46.544 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.544 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.544 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:23:46.544 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:33.888817+0000", 2024-08-19T14:23:46.544 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5753e-05 2024-08-19T14:23:46.544 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.544 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.544 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:23:46.544 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:33.888827+0000", 2024-08-19T14:23:46.545 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0117e-05 2024-08-19T14:23:46.545 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.545 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.545 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:23:46.545 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:33.969428+0000", 2024-08-19T14:23:46.545 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.080600464999999996 2024-08-19T14:23:46.545 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.545 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.545 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for peered", 2024-08-19T14:23:46.545 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:33.969436+0000", 2024-08-19T14:23:46.545 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.8730000000000007e-06 2024-08-19T14:23:46.545 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.545 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.545 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:23:46.546 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:34.982371+0000", 2024-08-19T14:23:46.546 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.012934805 2024-08-19T14:23:46.546 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.546 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.546 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:23:46.546 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:34.982426+0000", 2024-08-19T14:23:46.546 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.5432000000000001e-05 2024-08-19T14:23:46.546 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:23:46.546 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:23:46.546 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:23:46.546 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.546 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.546 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:22707 4.5 4:aa5c4d32:::benchmark_data_smithi045_71562_object22706:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-08-19T14:23:46.547 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:20:41.355284+0000", 2024-08-19T14:23:46.547 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 185.15898075199999, 2024-08-19T14:23:46.547 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.32540933900000002, 2024-08-19T14:23:46.547 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:23:46.547 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:23:46.547 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:23:46.547 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:23:46.547 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:23:46.547 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:23:46.547 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 22707 2024-08-19T14:23:46.548 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.548 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:23:46.548 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.548 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:23:46.548 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.355284+0000", 2024-08-19T14:23:46.548 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:23:46.548 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.548 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.548 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:23:46.548 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.355284+0000", 2024-08-19T14:23:46.548 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:23:46.548 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.548 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.548 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:23:46.549 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.355285+0000", 2024-08-19T14:23:46.549 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.2699999999999998e-07 2024-08-19T14:23:46.549 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.549 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.549 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:23:46.549 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.355304+0000", 2024-08-19T14:23:46.549 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8834e-05 2024-08-19T14:23:46.549 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.549 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.549 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:23:46.549 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.355315+0000", 2024-08-19T14:23:46.549 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1388e-05 2024-08-19T14:23:46.549 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.549 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.549 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:23:46.550 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.355326+0000", 2024-08-19T14:23:46.550 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0101999999999999e-05 2024-08-19T14:23:46.550 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.550 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.550 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:23:46.550 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.355381+0000", 2024-08-19T14:23:46.550 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.5114999999999997e-05 2024-08-19T14:23:46.550 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.550 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.550 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:23:46.550 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.355481+0000", 2024-08-19T14:23:46.550 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000100448 2024-08-19T14:23:46.550 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.551 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.551 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,7", 2024-08-19T14:23:46.551 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.355563+0000", 2024-08-19T14:23:46.551 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.1891999999999995e-05 2024-08-19T14:23:46.551 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.551 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.551 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:23:46.551 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.356429+0000", 2024-08-19T14:23:46.551 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00086571700000000001 2024-08-19T14:23:46.551 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.551 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.551 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:23:46.551 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.356898+0000", 2024-08-19T14:23:46.551 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000469171 2024-08-19T14:23:46.552 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.552 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.552 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:23:46.552 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.680624+0000", 2024-08-19T14:23:46.552 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.32372564599999998 2024-08-19T14:23:46.552 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.552 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.552 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:23:46.552 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.680661+0000", 2024-08-19T14:23:46.552 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.7589999999999998e-05 2024-08-19T14:23:46.552 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.552 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.552 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:23:46.552 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.680694+0000", 2024-08-19T14:23:46.553 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.2509000000000003e-05 2024-08-19T14:23:46.553 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:23:46.553 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:23:46.553 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:23:46.553 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.553 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.554 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:22723 4.5 4:a03b0b8c:::benchmark_data_smithi045_71562_object22722:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-08-19T14:23:46.554 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:20:41.360250+0000", 2024-08-19T14:23:46.554 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 185.15401541899999, 2024-08-19T14:23:46.554 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.32180027, 2024-08-19T14:23:46.554 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:23:46.554 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:23:46.554 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:23:46.554 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:23:46.554 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:23:46.554 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:23:46.555 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 22723 2024-08-19T14:23:46.555 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.555 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:23:46.555 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.555 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:23:46.555 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.360250+0000", 2024-08-19T14:23:46.555 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:23:46.555 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.555 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.555 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:23:46.555 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.360250+0000", 2024-08-19T14:23:46.555 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:23:46.555 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.555 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.555 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:23:46.556 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.360251+0000", 2024-08-19T14:23:46.556 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7740000000000001e-06 2024-08-19T14:23:46.556 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.556 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.556 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:23:46.556 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.360271+0000", 2024-08-19T14:23:46.556 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9755999999999999e-05 2024-08-19T14:23:46.556 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.556 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.556 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:23:46.556 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.360280+0000", 2024-08-19T14:23:46.556 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.268e-06 2024-08-19T14:23:46.556 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.556 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.557 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:23:46.557 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.360286+0000", 2024-08-19T14:23:46.557 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2700000000000004e-06 2024-08-19T14:23:46.557 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.557 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.557 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:23:46.557 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.360339+0000", 2024-08-19T14:23:46.557 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.3644999999999999e-05 2024-08-19T14:23:46.557 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.557 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.557 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:23:46.557 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.360424+0000", 2024-08-19T14:23:46.557 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.4493999999999994e-05 2024-08-19T14:23:46.557 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.557 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.558 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,7", 2024-08-19T14:23:46.558 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.360529+0000", 2024-08-19T14:23:46.558 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00010503200000000001 2024-08-19T14:23:46.558 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.558 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.558 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:23:46.558 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.362147+0000", 2024-08-19T14:23:46.558 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001617718 2024-08-19T14:23:46.558 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.558 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.558 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:23:46.559 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.362188+0000", 2024-08-19T14:23:46.559 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.1553e-05 2024-08-19T14:23:46.559 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.559 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.559 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:23:46.559 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.681950+0000", 2024-08-19T14:23:46.559 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.31976217299999998 2024-08-19T14:23:46.559 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.559 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.559 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:23:46.559 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.682008+0000", 2024-08-19T14:23:46.559 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7494000000000002e-05 2024-08-19T14:23:46.560 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.560 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.560 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:23:46.560 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.682050+0000", 2024-08-19T14:23:46.560 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.2092999999999998e-05 2024-08-19T14:23:46.560 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:23:46.560 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:23:46.560 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:23:46.560 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.560 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.560 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:22725 4.5 4:adb3fee7:::benchmark_data_smithi045_71562_object22724:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-08-19T14:23:46.560 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:20:41.361334+0000", 2024-08-19T14:23:46.560 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 185.15293144500001, 2024-08-19T14:23:46.561 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.32179169200000002, 2024-08-19T14:23:46.561 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:23:46.561 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:23:46.561 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:23:46.561 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:23:46.561 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:23:46.561 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:23:46.561 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 22725 2024-08-19T14:23:46.561 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.561 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:23:46.561 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.561 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:23:46.561 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.361334+0000", 2024-08-19T14:23:46.561 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:23:46.562 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.562 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.562 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:23:46.562 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.361334+0000", 2024-08-19T14:23:46.562 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:23:46.562 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.562 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.562 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:23:46.562 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.361334+0000", 2024-08-19T14:23:46.562 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.61e-07 2024-08-19T14:23:46.562 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.562 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.562 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:23:46.563 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.361353+0000", 2024-08-19T14:23:46.563 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9074999999999999e-05 2024-08-19T14:23:46.563 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.563 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.563 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:23:46.563 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.361365+0000", 2024-08-19T14:23:46.563 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1498000000000001e-05 2024-08-19T14:23:46.563 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.563 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.563 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:23:46.563 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.361371+0000", 2024-08-19T14:23:46.563 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.1290000000000001e-06 2024-08-19T14:23:46.563 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.563 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.563 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:23:46.564 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.361415+0000", 2024-08-19T14:23:46.564 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.3445999999999997e-05 2024-08-19T14:23:46.564 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.564 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.564 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:23:46.564 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.361542+0000", 2024-08-19T14:23:46.564 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012710400000000001 2024-08-19T14:23:46.564 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.564 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.564 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,7", 2024-08-19T14:23:46.564 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.361632+0000", 2024-08-19T14:23:46.564 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.9963999999999994e-05 2024-08-19T14:23:46.564 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.564 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.564 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:23:46.565 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.362288+0000", 2024-08-19T14:23:46.565 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00065625500000000001 2024-08-19T14:23:46.565 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.565 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.565 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:23:46.565 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.362699+0000", 2024-08-19T14:23:46.565 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00041120600000000001 2024-08-19T14:23:46.565 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.565 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.565 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:23:46.565 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.683027+0000", 2024-08-19T14:23:46.565 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.32032799699999998 2024-08-19T14:23:46.565 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.565 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.566 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:23:46.566 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.683084+0000", 2024-08-19T14:23:46.566 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7054e-05 2024-08-19T14:23:46.566 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.566 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.566 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:23:46.566 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.683125+0000", 2024-08-19T14:23:46.566 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.1202999999999998e-05 2024-08-19T14:23:46.566 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:23:46.566 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:23:46.566 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:23:46.566 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.566 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.567 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:32454 4.3 4:c5fede70:::benchmark_data_smithi045_71562_object32453:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e343)", 2024-08-19T14:23:46.567 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:20:45.676484+0000", 2024-08-19T14:23:46.567 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 180.837781492, 2024-08-19T14:23:46.567 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.212526989, 2024-08-19T14:23:46.567 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:23:46.567 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:23:46.567 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:23:46.567 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:23:46.567 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:23:46.567 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:23:46.567 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 32454 2024-08-19T14:23:46.567 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.567 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:23:46.567 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.567 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:23:46.568 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.676484+0000", 2024-08-19T14:23:46.568 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:23:46.568 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.568 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.568 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:23:46.568 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.676484+0000", 2024-08-19T14:23:46.568 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:23:46.568 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.568 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.568 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:23:46.568 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.676484+0000", 2024-08-19T14:23:46.568 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1900000000000003e-07 2024-08-19T14:23:46.568 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.568 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.569 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:23:46.569 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.676506+0000", 2024-08-19T14:23:46.569 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1515e-05 2024-08-19T14:23:46.569 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.569 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.569 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:23:46.569 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.676513+0000", 2024-08-19T14:23:46.569 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.7570000000000004e-06 2024-08-19T14:23:46.569 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.569 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.569 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:23:46.569 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.676519+0000", 2024-08-19T14:23:46.569 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4419999999999997e-06 2024-08-19T14:23:46.569 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.570 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.570 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:23:46.570 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.676567+0000", 2024-08-19T14:23:46.570 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.8439999999999997e-05 2024-08-19T14:23:46.570 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.570 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.570 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:23:46.570 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.676672+0000", 2024-08-19T14:23:46.570 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000104991 2024-08-19T14:23:46.570 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.570 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.570 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,7", 2024-08-19T14:23:46.570 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.676835+0000", 2024-08-19T14:23:46.570 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00016259600000000001 2024-08-19T14:23:46.571 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.571 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.571 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:23:46.571 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.678852+0000", 2024-08-19T14:23:46.571 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.002017511 2024-08-19T14:23:46.571 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.571 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.571 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:23:46.571 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.678903+0000", 2024-08-19T14:23:46.571 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0387999999999999e-05 2024-08-19T14:23:46.571 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.571 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.571 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:23:46.571 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.888915+0000", 2024-08-19T14:23:46.571 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.21001176899999999 2024-08-19T14:23:46.572 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.572 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.572 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:23:46.572 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.888969+0000", 2024-08-19T14:23:46.572 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.401e-05 2024-08-19T14:23:46.572 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.572 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.572 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:23:46.572 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.889011+0000", 2024-08-19T14:23:46.572 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.2051000000000001e-05 2024-08-19T14:23:46.572 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:23:46.572 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:23:46.572 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:23:46.572 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.573 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.573 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:116285 4.8 4:15b9fca8:::benchmark_data_smithi045_71562_object116284:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-08-19T14:23:46.573 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:21:48.600383+0000", 2024-08-19T14:23:46.573 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 117.913882204, 2024-08-19T14:23:46.573 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.25002218100000001, 2024-08-19T14:23:46.573 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:23:46.573 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:23:46.573 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:23:46.573 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:23:46.573 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:23:46.573 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:23:46.574 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 116285 2024-08-19T14:23:46.574 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.574 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:23:46.574 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.574 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:23:46.574 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.600383+0000", 2024-08-19T14:23:46.574 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:23:46.574 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.574 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.574 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:23:46.574 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.600383+0000", 2024-08-19T14:23:46.574 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:23:46.574 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.574 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.575 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:23:46.575 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.600384+0000", 2024-08-19T14:23:46.575 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.3200000000000003e-07 2024-08-19T14:23:46.575 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.575 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.575 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:23:46.575 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.600403+0000", 2024-08-19T14:23:46.575 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9326000000000001e-05 2024-08-19T14:23:46.575 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.575 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.575 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:23:46.576 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.600414+0000", 2024-08-19T14:23:46.576 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1129e-05 2024-08-19T14:23:46.576 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.576 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.576 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:23:46.576 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.600421+0000", 2024-08-19T14:23:46.576 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.8589999999999999e-06 2024-08-19T14:23:46.576 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.576 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.576 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:23:46.576 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.600478+0000", 2024-08-19T14:23:46.576 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.6502000000000002e-05 2024-08-19T14:23:46.576 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.576 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.577 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:23:46.577 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.600573+0000", 2024-08-19T14:23:46.577 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.5165000000000001e-05 2024-08-19T14:23:46.577 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.577 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.577 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 7,9", 2024-08-19T14:23:46.577 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.600654+0000", 2024-08-19T14:23:46.577 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.1002999999999997e-05 2024-08-19T14:23:46.577 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.577 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.577 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:23:46.577 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.607545+0000", 2024-08-19T14:23:46.577 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0068915039999999997 2024-08-19T14:23:46.578 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.578 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.578 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:23:46.578 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.610300+0000", 2024-08-19T14:23:46.578 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0027545199999999999 2024-08-19T14:23:46.578 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.578 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.578 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:23:46.578 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.850322+0000", 2024-08-19T14:23:46.578 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.24002195200000001 2024-08-19T14:23:46.578 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.578 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.578 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:23:46.579 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.850364+0000", 2024-08-19T14:23:46.580 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.2582e-05 2024-08-19T14:23:46.580 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.580 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.580 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:23:46.580 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.850405+0000", 2024-08-19T14:23:46.580 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.0707000000000002e-05 2024-08-19T14:23:46.580 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:23:46.580 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:23:46.580 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:23:46.580 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.580 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.580 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:116286 4.8 4:198dc556:::benchmark_data_smithi045_71562_object116285:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-08-19T14:23:46.581 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:21:48.600439+0000", 2024-08-19T14:23:46.581 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 117.913826177, 2024-08-19T14:23:46.581 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.25012546899999999, 2024-08-19T14:23:46.581 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:23:46.581 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:23:46.581 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:23:46.581 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:23:46.581 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:23:46.581 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:23:46.581 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 116286 2024-08-19T14:23:46.581 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.581 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:23:46.581 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.581 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:23:46.581 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.600439+0000", 2024-08-19T14:23:46.582 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:23:46.582 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.582 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.582 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:23:46.582 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.600439+0000", 2024-08-19T14:23:46.582 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:23:46.582 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.582 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.582 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:23:46.582 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.600439+0000", 2024-08-19T14:23:46.582 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2900000000000004e-07 2024-08-19T14:23:46.582 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.582 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.583 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:23:46.583 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.600458+0000", 2024-08-19T14:23:46.583 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8729999999999999e-05 2024-08-19T14:23:46.583 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.583 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.583 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:23:46.583 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.600466+0000", 2024-08-19T14:23:46.583 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.1030000000000002e-06 2024-08-19T14:23:46.583 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.583 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.583 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:23:46.584 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.600472+0000", 2024-08-19T14:23:46.584 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4380000000000003e-06 2024-08-19T14:23:46.584 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.584 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.584 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:23:46.584 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.603745+0000", 2024-08-19T14:23:46.584 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0032732350000000002 2024-08-19T14:23:46.584 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.584 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.584 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:23:46.584 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.603905+0000", 2024-08-19T14:23:46.584 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015985500000000001 2024-08-19T14:23:46.584 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.584 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.585 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 7,9", 2024-08-19T14:23:46.585 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.604060+0000", 2024-08-19T14:23:46.585 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015485799999999999 2024-08-19T14:23:46.585 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.585 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.585 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:23:46.585 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.607871+0000", 2024-08-19T14:23:46.585 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0038114540000000001 2024-08-19T14:23:46.585 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.585 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.585 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:23:46.585 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.610365+0000", 2024-08-19T14:23:46.585 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0024943449999999998 2024-08-19T14:23:46.586 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.586 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.586 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:23:46.586 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.850477+0000", 2024-08-19T14:23:46.586 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.24011110099999999 2024-08-19T14:23:46.586 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.586 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.586 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:23:46.586 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.850528+0000", 2024-08-19T14:23:46.586 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1684999999999997e-05 2024-08-19T14:23:46.586 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.586 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.586 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:23:46.587 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.850564+0000", 2024-08-19T14:23:46.587 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.6136000000000001e-05 2024-08-19T14:23:46.587 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:23:46.587 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:23:46.587 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:23:46.587 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.587 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.587 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:116297 4.8 4:17724d94:::benchmark_data_smithi045_71562_object116296:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-08-19T14:23:46.587 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:21:48.612374+0000", 2024-08-19T14:23:46.587 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 117.9018911, 2024-08-19T14:23:46.587 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.28029534299999997, 2024-08-19T14:23:46.587 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:23:46.587 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:23:46.587 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:23:46.588 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:23:46.588 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:23:46.588 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:23:46.588 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 116297 2024-08-19T14:23:46.588 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.588 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:23:46.588 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.588 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:23:46.588 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.612374+0000", 2024-08-19T14:23:46.588 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:23:46.588 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.588 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.588 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:23:46.588 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.612374+0000", 2024-08-19T14:23:46.588 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:23:46.589 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.589 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.589 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:23:46.589 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.612375+0000", 2024-08-19T14:23:46.589 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2869999999999999e-06 2024-08-19T14:23:46.589 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.589 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.589 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:23:46.589 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.612393+0000", 2024-08-19T14:23:46.589 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7864e-05 2024-08-19T14:23:46.589 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.589 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.589 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:23:46.589 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.612404+0000", 2024-08-19T14:23:46.590 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1377e-05 2024-08-19T14:23:46.590 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.590 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.590 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:23:46.590 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.612410+0000", 2024-08-19T14:23:46.590 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.3730000000000001e-06 2024-08-19T14:23:46.590 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.590 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.590 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:23:46.590 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.612460+0000", 2024-08-19T14:23:46.590 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.9675999999999997e-05 2024-08-19T14:23:46.590 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.590 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.590 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:23:46.591 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.612561+0000", 2024-08-19T14:23:46.591 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00010157099999999999 2024-08-19T14:23:46.591 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.591 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.591 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 7,9", 2024-08-19T14:23:46.591 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.612676+0000", 2024-08-19T14:23:46.591 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00011491 2024-08-19T14:23:46.591 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.591 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.591 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:23:46.591 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.616157+0000", 2024-08-19T14:23:46.591 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00348073 2024-08-19T14:23:46.591 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.591 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.592 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:23:46.592 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.616249+0000", 2024-08-19T14:23:46.592 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.2085999999999999e-05 2024-08-19T14:23:46.592 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.592 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.592 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:23:46.592 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.850604+0000", 2024-08-19T14:23:46.592 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.23435562900000001 2024-08-19T14:23:46.592 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.592 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.592 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:23:46.592 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.892613+0000", 2024-08-19T14:23:46.592 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.042008839999999999 2024-08-19T14:23:46.592 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.592 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.593 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:23:46.593 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.892669+0000", 2024-08-19T14:23:46.593 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.5999999999999999e-05 2024-08-19T14:23:46.593 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:23:46.593 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:23:46.593 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:23:46.593 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.593 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.593 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:116299 4.3 4:c0da230c:::benchmark_data_smithi045_71562_object116298:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-08-19T14:23:46.593 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:21:48.612978+0000", 2024-08-19T14:23:46.593 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 117.90128681100001, 2024-08-19T14:23:46.593 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.23536219899999999, 2024-08-19T14:23:46.593 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:23:46.593 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:23:46.594 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:23:46.594 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:23:46.594 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:23:46.594 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:23:46.594 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 116299 2024-08-19T14:23:46.594 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.594 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:23:46.594 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.594 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:23:46.594 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.612978+0000", 2024-08-19T14:23:46.594 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:23:46.595 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.595 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.595 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:23:46.595 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.612978+0000", 2024-08-19T14:23:46.595 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:23:46.595 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.595 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.595 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:23:46.595 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.612979+0000", 2024-08-19T14:23:46.595 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.7599999999999996e-07 2024-08-19T14:23:46.595 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.595 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.596 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:23:46.596 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.612998+0000", 2024-08-19T14:23:46.596 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8527000000000001e-05 2024-08-19T14:23:46.596 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.596 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.596 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:23:46.596 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.613008+0000", 2024-08-19T14:23:46.596 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.079e-05 2024-08-19T14:23:46.596 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.596 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.596 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:23:46.596 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.613014+0000", 2024-08-19T14:23:46.596 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4210000000000003e-06 2024-08-19T14:23:46.596 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.596 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.597 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:23:46.597 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.613062+0000", 2024-08-19T14:23:46.597 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.8674999999999997e-05 2024-08-19T14:23:46.597 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.597 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.597 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:23:46.597 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.613200+0000", 2024-08-19T14:23:46.597 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013715400000000001 2024-08-19T14:23:46.597 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.597 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.597 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,7", 2024-08-19T14:23:46.597 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.613317+0000", 2024-08-19T14:23:46.597 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00011734399999999999 2024-08-19T14:23:46.598 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.598 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.598 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:23:46.598 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.616326+0000", 2024-08-19T14:23:46.598 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0030090450000000001 2024-08-19T14:23:46.598 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.599 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.599 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:23:46.599 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.621117+0000", 2024-08-19T14:23:46.599 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0047912160000000001 2024-08-19T14:23:46.599 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.599 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.600 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:23:46.600 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.848233+0000", 2024-08-19T14:23:46.600 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.22711604099999999 2024-08-19T14:23:46.600 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.600 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.600 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:23:46.600 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.848288+0000", 2024-08-19T14:23:46.600 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4972999999999997e-05 2024-08-19T14:23:46.600 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.600 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.600 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:23:46.600 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.848340+0000", 2024-08-19T14:23:46.600 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2237000000000001e-05 2024-08-19T14:23:46.600 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:23:46.601 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:23:46.601 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:23:46.601 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.601 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.601 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:116311 4.8 4:16b96ac4:::benchmark_data_smithi045_71562_object116310:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-08-19T14:23:46.601 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:21:48.627253+0000", 2024-08-19T14:23:46.601 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 117.88701186199999, 2024-08-19T14:23:46.601 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.26561561299999997, 2024-08-19T14:23:46.601 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:23:46.601 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:23:46.601 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:23:46.601 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:23:46.601 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:23:46.601 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:23:46.602 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 116311 2024-08-19T14:23:46.602 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.602 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:23:46.602 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.602 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:23:46.602 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.627253+0000", 2024-08-19T14:23:46.602 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:23:46.602 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.602 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.602 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:23:46.602 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.627253+0000", 2024-08-19T14:23:46.602 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:23:46.603 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.603 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.603 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:23:46.603 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.627254+0000", 2024-08-19T14:23:46.603 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4000000000000001e-07 2024-08-19T14:23:46.603 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.604 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.604 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:23:46.604 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.627293+0000", 2024-08-19T14:23:46.604 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.8742000000000003e-05 2024-08-19T14:23:46.604 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.604 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.604 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:23:46.605 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.627302+0000", 2024-08-19T14:23:46.605 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.2639999999999997e-06 2024-08-19T14:23:46.605 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.605 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.605 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:23:46.605 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.627307+0000", 2024-08-19T14:23:46.606 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0209999999999999e-06 2024-08-19T14:23:46.606 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.606 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.606 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:23:46.606 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.627378+0000", 2024-08-19T14:23:46.606 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.0529000000000002e-05 2024-08-19T14:23:46.607 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.607 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.607 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:23:46.607 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.627458+0000", 2024-08-19T14:23:46.607 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.0259999999999994e-05 2024-08-19T14:23:46.607 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.608 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.608 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 7,9", 2024-08-19T14:23:46.608 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.627624+0000", 2024-08-19T14:23:46.608 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00016623300000000001 2024-08-19T14:23:46.608 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.608 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.608 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:23:46.608 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.631383+0000", 2024-08-19T14:23:46.609 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0037587599999999999 2024-08-19T14:23:46.609 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.609 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.609 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:23:46.609 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.631513+0000", 2024-08-19T14:23:46.609 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013022199999999999 2024-08-19T14:23:46.609 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.609 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.610 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:23:46.610 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.892784+0000", 2024-08-19T14:23:46.610 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.26127100399999997 2024-08-19T14:23:46.610 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.610 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.610 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:23:46.611 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.892832+0000", 2024-08-19T14:23:46.611 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.7970999999999999e-05 2024-08-19T14:23:46.611 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.611 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.611 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:23:46.611 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.892869+0000", 2024-08-19T14:23:46.612 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.6866999999999998e-05 2024-08-19T14:23:46.612 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:23:46.612 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:23:46.612 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:23:46.612 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.612 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.612 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:116334 4.8 4:1f777a70:::benchmark_data_smithi045_71562_object116333:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-08-19T14:23:46.612 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:21:48.650934+0000", 2024-08-19T14:23:46.612 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 117.863331155, 2024-08-19T14:23:46.612 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.242080929, 2024-08-19T14:23:46.612 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:23:46.612 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:23:46.613 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:23:46.613 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:23:46.613 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:23:46.613 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:23:46.613 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 116334 2024-08-19T14:23:46.613 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.613 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:23:46.613 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.613 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:23:46.613 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.650934+0000", 2024-08-19T14:23:46.613 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:23:46.613 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.613 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.613 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:23:46.614 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.650934+0000", 2024-08-19T14:23:46.614 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:23:46.614 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.614 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.614 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:23:46.614 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.650935+0000", 2024-08-19T14:23:46.614 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.26e-06 2024-08-19T14:23:46.614 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.614 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.614 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:23:46.614 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.650954+0000", 2024-08-19T14:23:46.614 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9000000000000001e-05 2024-08-19T14:23:46.614 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.615 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.615 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:23:46.615 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.650964+0000", 2024-08-19T14:23:46.615 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0253e-05 2024-08-19T14:23:46.615 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.615 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.615 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:23:46.615 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.650970+0000", 2024-08-19T14:23:46.615 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1499999999999998e-06 2024-08-19T14:23:46.615 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.615 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.615 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:23:46.615 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.651014+0000", 2024-08-19T14:23:46.615 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.4391000000000002e-05 2024-08-19T14:23:46.616 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.616 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.616 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:23:46.616 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.651134+0000", 2024-08-19T14:23:46.616 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000119535 2024-08-19T14:23:46.616 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.616 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.616 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 7,9", 2024-08-19T14:23:46.616 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.651262+0000", 2024-08-19T14:23:46.616 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012826100000000001 2024-08-19T14:23:46.616 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.616 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.616 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:23:46.616 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.654276+0000", 2024-08-19T14:23:46.616 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0030142839999999999 2024-08-19T14:23:46.617 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.617 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.617 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:23:46.617 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.655021+0000", 2024-08-19T14:23:46.617 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00074536299999999997 2024-08-19T14:23:46.617 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.617 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.617 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:23:46.617 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.892914+0000", 2024-08-19T14:23:46.617 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.23789228400000001 2024-08-19T14:23:46.617 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.617 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.617 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:23:46.617 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.892980+0000", 2024-08-19T14:23:46.618 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.6193999999999997e-05 2024-08-19T14:23:46.618 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.618 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.618 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:23:46.618 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.893015+0000", 2024-08-19T14:23:46.618 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.4953999999999998e-05 2024-08-19T14:23:46.618 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:23:46.618 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:23:46.618 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:23:46.618 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.618 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.618 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:116354 4.8 4:103642ea:::benchmark_data_smithi045_71562_object116353:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-08-19T14:23:46.618 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:21:48.673582+0000", 2024-08-19T14:23:46.619 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 117.840683482, 2024-08-19T14:23:46.619 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.21960154700000001, 2024-08-19T14:23:46.619 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:23:46.619 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:23:46.619 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:23:46.619 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:23:46.619 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:23:46.619 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:23:46.619 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 116354 2024-08-19T14:23:46.619 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.619 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:23:46.619 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.619 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:23:46.619 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.673582+0000", 2024-08-19T14:23:46.620 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:23:46.620 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.620 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.620 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:23:46.620 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.673582+0000", 2024-08-19T14:23:46.620 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:23:46.620 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.620 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.620 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:23:46.620 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.673583+0000", 2024-08-19T14:23:46.620 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.279e-06 2024-08-19T14:23:46.620 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.620 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.620 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:23:46.621 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.673633+0000", 2024-08-19T14:23:46.621 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.9845000000000001e-05 2024-08-19T14:23:46.621 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.621 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.621 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:23:46.621 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.673644+0000", 2024-08-19T14:23:46.621 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1333e-05 2024-08-19T14:23:46.621 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.621 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.621 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:23:46.621 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.673649+0000", 2024-08-19T14:23:46.621 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2000000000000002e-06 2024-08-19T14:23:46.621 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.621 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.622 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:23:46.622 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.673703+0000", 2024-08-19T14:23:46.622 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.3471999999999999e-05 2024-08-19T14:23:46.622 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.622 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.622 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:23:46.622 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.673845+0000", 2024-08-19T14:23:46.622 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014190399999999999 2024-08-19T14:23:46.622 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.622 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.622 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 7,9", 2024-08-19T14:23:46.622 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.673956+0000", 2024-08-19T14:23:46.622 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00011128000000000001 2024-08-19T14:23:46.622 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.623 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.623 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:23:46.623 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.676763+0000", 2024-08-19T14:23:46.623 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.002807038 2024-08-19T14:23:46.623 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.623 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.623 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:23:46.623 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.677752+0000", 2024-08-19T14:23:46.623 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00098899899999999995 2024-08-19T14:23:46.623 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.623 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.623 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:23:46.623 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.893093+0000", 2024-08-19T14:23:46.624 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.21534098300000001 2024-08-19T14:23:46.624 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.624 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.624 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:23:46.624 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.893143+0000", 2024-08-19T14:23:46.624 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.9877999999999999e-05 2024-08-19T14:23:46.624 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.624 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.624 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:23:46.624 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.893183+0000", 2024-08-19T14:23:46.624 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.0336000000000001e-05 2024-08-19T14:23:46.624 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:23:46.624 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:23:46.624 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:23:46.624 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.625 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.625 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:165624 4.3 4:cbfa230a:::benchmark_data_smithi045_71562_object34449:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-08-19T14:23:46.625 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:22:42.872335+0000", 2024-08-19T14:23:46.625 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 63.641930074999998, 2024-08-19T14:23:46.625 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.21952618099999999, 2024-08-19T14:23:46.625 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:23:46.625 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:23:46.625 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:23:46.625 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:23:46.625 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:23:46.625 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:23:46.625 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 165624 2024-08-19T14:23:46.625 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.625 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:23:46.625 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.626 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:23:46.626 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.872335+0000", 2024-08-19T14:23:46.626 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:23:46.626 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.626 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.626 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:23:46.626 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.872335+0000", 2024-08-19T14:23:46.626 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:23:46.626 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.626 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.626 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:23:46.626 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.872336+0000", 2024-08-19T14:23:46.626 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1370000000000001e-06 2024-08-19T14:23:46.627 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.627 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.627 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:23:46.627 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.872341+0000", 2024-08-19T14:23:46.627 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.7840000000000003e-06 2024-08-19T14:23:46.627 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.627 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.627 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:23:46.627 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.872351+0000", 2024-08-19T14:23:46.627 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0254999999999999e-05 2024-08-19T14:23:46.627 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.627 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.627 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:23:46.628 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.872357+0000", 2024-08-19T14:23:46.628 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.6330000000000001e-06 2024-08-19T14:23:46.628 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.628 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.628 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:23:46.628 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.872406+0000", 2024-08-19T14:23:46.628 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.8958000000000001e-05 2024-08-19T14:23:46.628 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.628 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.628 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:23:46.628 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.872515+0000", 2024-08-19T14:23:46.628 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00010927199999999999 2024-08-19T14:23:46.628 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.628 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.628 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,7", 2024-08-19T14:23:46.629 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.872634+0000", 2024-08-19T14:23:46.629 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000118534 2024-08-19T14:23:46.629 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.629 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.629 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:23:46.629 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.876720+0000", 2024-08-19T14:23:46.629 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0040864370000000001 2024-08-19T14:23:46.629 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.629 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.629 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:23:46.629 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.878480+0000", 2024-08-19T14:23:46.629 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001760345 2024-08-19T14:23:46.629 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.629 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.630 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:23:46.630 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:43.091797+0000", 2024-08-19T14:23:46.630 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.21331655499999999 2024-08-19T14:23:46.630 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.630 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.630 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:23:46.630 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:43.091830+0000", 2024-08-19T14:23:46.630 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.3401000000000001e-05 2024-08-19T14:23:46.630 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.630 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.631 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:23:46.631 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:43.091861+0000", 2024-08-19T14:23:46.631 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.0870000000000001e-05 2024-08-19T14:23:46.631 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:23:46.631 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:23:46.631 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:23:46.631 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.631 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.631 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:165636 4.3 4:c997d720:::benchmark_data_smithi045_71562_object34461:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-08-19T14:23:46.631 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:22:42.884498+0000", 2024-08-19T14:23:46.631 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 63.629767229000002, 2024-08-19T14:23:46.631 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.22208514900000001, 2024-08-19T14:23:46.631 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:23:46.631 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:23:46.632 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:23:46.632 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:23:46.632 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:23:46.632 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:23:46.632 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 165636 2024-08-19T14:23:46.632 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.632 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:23:46.632 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.632 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:23:46.632 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.884498+0000", 2024-08-19T14:23:46.632 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:23:46.632 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.632 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.632 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:23:46.633 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.884498+0000", 2024-08-19T14:23:46.633 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:23:46.633 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.633 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.633 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:23:46.633 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.884499+0000", 2024-08-19T14:23:46.633 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.238e-06 2024-08-19T14:23:46.633 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.633 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.633 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:23:46.633 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.884504+0000", 2024-08-19T14:23:46.633 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2959999999999998e-06 2024-08-19T14:23:46.633 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.633 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.634 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:23:46.634 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.884515+0000", 2024-08-19T14:23:46.634 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0995999999999999e-05 2024-08-19T14:23:46.634 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.634 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.634 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:23:46.634 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.884523+0000", 2024-08-19T14:23:46.634 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.294e-06 2024-08-19T14:23:46.634 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.634 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.634 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:23:46.634 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.884580+0000", 2024-08-19T14:23:46.634 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7224e-05 2024-08-19T14:23:46.635 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.635 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.635 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:23:46.635 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.884704+0000", 2024-08-19T14:23:46.635 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000124046 2024-08-19T14:23:46.635 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.635 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.635 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,7", 2024-08-19T14:23:46.635 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.884836+0000", 2024-08-19T14:23:46.635 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000132478 2024-08-19T14:23:46.635 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.636 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.636 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:23:46.636 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.888660+0000", 2024-08-19T14:23:46.636 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0038240399999999999 2024-08-19T14:23:46.636 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.636 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.636 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:23:46.636 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.889616+0000", 2024-08-19T14:23:46.636 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00095516899999999996 2024-08-19T14:23:46.636 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.636 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.636 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:23:46.636 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:43.106482+0000", 2024-08-19T14:23:46.636 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.21686655299999999 2024-08-19T14:23:46.637 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.637 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.637 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:23:46.637 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:43.106539+0000", 2024-08-19T14:23:46.637 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.6366999999999998e-05 2024-08-19T14:23:46.637 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.637 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.637 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:23:46.637 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:43.106583+0000", 2024-08-19T14:23:46.637 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.4447999999999999e-05 2024-08-19T14:23:46.637 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:23:46.637 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:23:46.637 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:23:46.637 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.638 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.638 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:165645 4.3 4:c4ef6944:::benchmark_data_smithi045_71562_object34470:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-08-19T14:23:46.638 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:22:42.892140+0000", 2024-08-19T14:23:46.638 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 63.622124653999997, 2024-08-19T14:23:46.638 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.223849192, 2024-08-19T14:23:46.638 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:23:46.638 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:23:46.638 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:23:46.638 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:23:46.638 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:23:46.638 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:23:46.639 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 165645 2024-08-19T14:23:46.639 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.639 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:23:46.639 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.639 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:23:46.639 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.892140+0000", 2024-08-19T14:23:46.639 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:23:46.639 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.639 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.639 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:23:46.640 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.892140+0000", 2024-08-19T14:23:46.640 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:23:46.640 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.640 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.640 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:23:46.640 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.892142+0000", 2024-08-19T14:23:46.640 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.127e-06 2024-08-19T14:23:46.640 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.640 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.640 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:23:46.640 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.892146+0000", 2024-08-19T14:23:46.640 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.7949999999999998e-06 2024-08-19T14:23:46.640 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.640 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.641 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:23:46.641 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.892157+0000", 2024-08-19T14:23:46.641 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0387000000000001e-05 2024-08-19T14:23:46.641 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.641 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.641 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:23:46.641 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.892162+0000", 2024-08-19T14:23:46.641 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.6269999999999997e-06 2024-08-19T14:23:46.641 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.642 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.642 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:23:46.642 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.892203+0000", 2024-08-19T14:23:46.642 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.0238999999999999e-05 2024-08-19T14:23:46.643 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:23:46.644 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.644 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.644 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:23:46.644 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.892284+0000", 2024-08-19T14:23:46.644 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.1198000000000004e-05 2024-08-19T14:23:46.644 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.644 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.645 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,7", 2024-08-19T14:23:46.645 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.892359+0000", 2024-08-19T14:23:46.645 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4783000000000006e-05 2024-08-19T14:23:46.645 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.645 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.645 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:23:46.645 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.895461+0000", 2024-08-19T14:23:46.645 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0031028869999999999 2024-08-19T14:23:46.645 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.645 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.645 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:23:46.645 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.897428+0000", 2024-08-19T14:23:46.646 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0019669069999999999 2024-08-19T14:23:46.646 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.646 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.646 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:23:46.646 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:43.115899+0000", 2024-08-19T14:23:46.646 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.21847051100000001 2024-08-19T14:23:46.646 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.646 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.646 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:23:46.646 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:43.115943+0000", 2024-08-19T14:23:46.646 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.4181000000000003e-05 2024-08-19T14:23:46.646 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.646 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.647 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:23:46.647 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:43.115990+0000", 2024-08-19T14:23:46.647 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.655e-05 2024-08-19T14:23:46.647 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:23:46.647 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:23:46.647 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:23:46.647 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.647 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.647 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:210364 4.9 4:9e22782b:::benchmark_data_smithi045_71562_object79189:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e396)", 2024-08-19T14:23:46.647 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:23:33.561916+0000", 2024-08-19T14:23:46.647 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 12.952349486999999, 2024-08-19T14:23:46.647 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.21356883800000001, 2024-08-19T14:23:46.648 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:23:46.648 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:23:46.648 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:23:46.648 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:23:46.648 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:23:46.648 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:23:46.648 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 210364 2024-08-19T14:23:46.648 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.648 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:23:46.648 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.649 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:23:46.649 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:23:33.561916+0000", 2024-08-19T14:23:46.649 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:23:46.649 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.649 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.649 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:23:46.649 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:23:33.561916+0000", 2024-08-19T14:23:46.649 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:23:46.649 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.649 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.649 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:23:46.649 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:23:33.561916+0000", 2024-08-19T14:23:46.649 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.9999999999999997e-07 2024-08-19T14:23:46.649 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.649 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.650 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:23:46.650 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:23:33.561920+0000", 2024-08-19T14:23:46.650 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.6669999999999998e-06 2024-08-19T14:23:46.650 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.650 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.650 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:23:46.650 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:23:33.561929+0000", 2024-08-19T14:23:46.650 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.4700000000000008e-06 2024-08-19T14:23:46.650 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.650 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.650 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:23:46.650 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:23:33.561937+0000", 2024-08-19T14:23:46.650 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.1829999999999997e-06 2024-08-19T14:23:46.651 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.651 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.651 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:23:46.651 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:23:33.561977+0000", 2024-08-19T14:23:46.651 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.0119e-05 2024-08-19T14:23:46.651 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.651 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.651 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:23:46.651 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:23:33.562051+0000", 2024-08-19T14:23:46.651 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4740000000000006e-05 2024-08-19T14:23:46.651 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.651 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.651 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,3", 2024-08-19T14:23:46.651 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:23:33.562117+0000", 2024-08-19T14:23:46.652 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.5875000000000002e-05 2024-08-19T14:23:46.652 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.652 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.652 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:23:46.652 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:23:33.566295+0000", 2024-08-19T14:23:46.652 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0041779959999999998 2024-08-19T14:23:46.652 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.652 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.652 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:23:46.652 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:23:33.568994+0000", 2024-08-19T14:23:46.652 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0026991010000000002 2024-08-19T14:23:46.652 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.652 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.652 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:23:46.653 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:23:33.775415+0000", 2024-08-19T14:23:46.653 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20642043299999999 2024-08-19T14:23:46.653 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.653 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.653 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:23:46.653 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:23:33.775448+0000", 2024-08-19T14:23:46.653 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.3578999999999998e-05 2024-08-19T14:23:46.653 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.653 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.653 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:23:46.653 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:23:33.775484+0000", 2024-08-19T14:23:46.654 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.5975e-05 2024-08-19T14:23:46.654 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:23:46.654 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:23:46.654 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:23:46.654 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.654 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.654 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:210377 4.9 4:9c01a247:::benchmark_data_smithi045_71562_object79202:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e396)", 2024-08-19T14:23:46.655 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:23:33.573172+0000", 2024-08-19T14:23:46.655 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 12.941093312, 2024-08-19T14:23:46.655 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.212114408, 2024-08-19T14:23:46.655 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:23:46.655 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:23:46.655 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:23:46.655 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:23:46.655 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:23:46.655 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:23:46.655 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 210377 2024-08-19T14:23:46.655 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.655 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:23:46.655 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.655 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:23:46.655 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:23:33.573172+0000", 2024-08-19T14:23:46.656 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:23:46.656 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.656 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.656 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:23:46.656 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:23:33.573172+0000", 2024-08-19T14:23:46.656 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:23:46.656 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.656 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.656 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:23:46.656 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:23:33.573173+0000", 2024-08-19T14:23:46.656 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.204e-06 2024-08-19T14:23:46.656 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.656 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.656 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:23:46.656 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:23:33.573178+0000", 2024-08-19T14:23:46.657 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.8199999999999996e-06 2024-08-19T14:23:46.657 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.657 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.657 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:23:46.657 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:23:33.573192+0000", 2024-08-19T14:23:46.657 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3845e-05 2024-08-19T14:23:46.657 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.657 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.657 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:23:46.657 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:23:33.573199+0000", 2024-08-19T14:23:46.657 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.0299999999999996e-06 2024-08-19T14:23:46.657 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.657 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.657 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:23:46.658 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:23:33.573254+0000", 2024-08-19T14:23:46.658 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4882000000000001e-05 2024-08-19T14:23:46.658 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.658 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.658 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:23:46.658 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:23:33.573357+0000", 2024-08-19T14:23:46.658 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000103181 2024-08-19T14:23:46.658 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.658 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.658 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,3", 2024-08-19T14:23:46.658 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:23:33.573461+0000", 2024-08-19T14:23:46.658 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00010435200000000001 2024-08-19T14:23:46.659 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.659 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.659 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:23:46.659 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:23:33.578930+0000", 2024-08-19T14:23:46.659 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.005469186 2024-08-19T14:23:46.659 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.659 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.659 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:23:46.659 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:23:33.584484+0000", 2024-08-19T14:23:46.659 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0055532860000000002 2024-08-19T14:23:46.659 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.659 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.659 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:23:46.660 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:23:33.785202+0000", 2024-08-19T14:23:46.660 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20071872700000001 2024-08-19T14:23:46.660 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.660 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.660 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:23:46.660 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:23:33.785247+0000", 2024-08-19T14:23:46.660 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.4570000000000002e-05 2024-08-19T14:23:46.660 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:23:46.660 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:23:46.660 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:23:46.660 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:23:33.785286+0000", 2024-08-19T14:23:46.660 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.9325e-05 2024-08-19T14:23:46.661 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:23:46.661 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:23:46.661 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:23:46.661 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:23:46.661 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:23:46.661 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:23:46.663 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-19T14:23:46.663 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-19T14:23:46.663 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph tell osd.11 'trim stale osdmaps' 2024-08-19T14:23:46.685 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:46.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:46.686 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:23:46.694 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:23:46.787 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:23:46.825 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:46.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:46.827 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:23:46.928 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:23:46.967 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:46.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:46.968 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:23:47.068 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:23:47.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:47.129 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:47.129 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:23:47.171 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:47 smithi031 ceph-mon[16336]: pgmap v915: 47 pgs: 47 active+clean; 2.7 GiB data, 12 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 791 op/s 2024-08-19T14:23:47.171 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:47 smithi031 ceph-mon[16336]: 1.10 scrub starts 2024-08-19T14:23:47.171 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:47 smithi031 ceph-mon[16336]: 1.10 scrub ok 2024-08-19T14:23:47.230 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:23:47.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:47.261 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:47.261 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:23:47.363 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:23:47.384 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:47 smithi045 ceph-mon[20143]: pgmap v915: 47 pgs: 47 active+clean; 2.7 GiB data, 12 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 791 op/s 2024-08-19T14:23:47.384 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:47 smithi045 ceph-mon[20143]: 1.10 scrub starts 2024-08-19T14:23:47.384 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:47 smithi045 ceph-mon[20143]: 1.10 scrub ok 2024-08-19T14:23:47.388 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:23:47.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:47.415 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:47.416 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:23:47.426 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-08-19T14:23:47.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:47 smithi032 ceph-mon[20004]: pgmap v915: 47 pgs: 47 active+clean; 2.7 GiB data, 12 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 791 op/s 2024-08-19T14:23:47.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:47 smithi032 ceph-mon[20004]: 1.10 scrub starts 2024-08-19T14:23:47.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:47 smithi032 ceph-mon[20004]: 1.10 scrub ok 2024-08-19T14:23:47.517 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:23:47.543 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:47.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:47.544 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:23:47.646 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:23:47.671 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:47.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:47.673 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:23:47.773 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:23:47.839 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:47.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:47.840 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:23:47.941 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:23:47.996 INFO:teuthology.orchestra.run.smithi031.stderr:no valid command found; 10 closest matches: 2024-08-19T14:23:47.996 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:47.997 INFO:teuthology.orchestra.run.smithi031.stderr:0 2024-08-19T14:23:47.997 INFO:teuthology.orchestra.run.smithi031.stderr:1 2024-08-19T14:23:47.997 INFO:teuthology.orchestra.run.smithi031.stderr:2 2024-08-19T14:23:47.997 INFO:teuthology.orchestra.run.smithi031.stderr:abort 2024-08-19T14:23:47.997 INFO:teuthology.orchestra.run.smithi031.stderr:assert 2024-08-19T14:23:47.997 INFO:teuthology.orchestra.run.smithi031.stderr:bench [] [] [] [] 2024-08-19T14:23:47.997 INFO:teuthology.orchestra.run.smithi031.stderr:bluefs debug_inject_read_zeros 2024-08-19T14:23:47.997 INFO:teuthology.orchestra.run.smithi031.stderr:bluefs files list 2024-08-19T14:23:47.997 INFO:teuthology.orchestra.run.smithi031.stderr:bluefs stats 2024-08-19T14:23:47.997 INFO:teuthology.orchestra.run.smithi031.stderr:bluestore allocator dump block 2024-08-19T14:23:47.998 INFO:teuthology.orchestra.run.smithi031.stderr:Error EINVAL: invalid command 2024-08-19T14:23:47.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:47.998 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:23:48.062 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is unset 2024-08-19T14:23:48.100 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:23:48.131 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:48.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:48.132 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:23:48.228 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:23:48.232 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:23:48.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:48.319 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:48.320 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:23:48.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:48 smithi031 ceph-mon[16336]: 4.d scrub starts 2024-08-19T14:23:48.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:48 smithi031 ceph-mon[16336]: 4.2 scrub starts 2024-08-19T14:23:48.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:48 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2585242893' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:23:48.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:48 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:23:48.421 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:23:48.437 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:23:48.445 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:48.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:48.446 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:23:48.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:48 smithi032 ceph-mon[20004]: 4.d scrub starts 2024-08-19T14:23:48.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:48 smithi032 ceph-mon[20004]: 4.2 scrub starts 2024-08-19T14:23:48.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:48 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2585242893' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:23:48.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:48 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:23:48.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:48 smithi045 ceph-mon[20143]: 4.d scrub starts 2024-08-19T14:23:48.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:48 smithi045 ceph-mon[20143]: 4.2 scrub starts 2024-08-19T14:23:48.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:48 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2585242893' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:23:48.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:48 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:23:48.547 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:23:48.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:48.583 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:48.583 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:23:48.684 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:23:48.719 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:48.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:48.720 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:23:48.821 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:23:48.848 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:48.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:48.849 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:23:48.950 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:23:48.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:48.988 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:48.988 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:23:49.089 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:23:49.117 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:49.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:49.118 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:23:49.219 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:23:49.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:49.243 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:49.243 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:23:49.245 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:49 smithi031 ceph-mon[16336]: 4.d scrub ok 2024-08-19T14:23:49.245 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:49 smithi031 ceph-mon[16336]: 4.2 scrub ok 2024-08-19T14:23:49.245 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:49 smithi031 ceph-mon[16336]: pgmap v916: 47 pgs: 47 active+clean; 2.5 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 932 op/s 2024-08-19T14:23:49.245 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:49 smithi031 ceph-mon[16336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:23:49.245 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:49 smithi031 ceph-mon[16336]: Cluster is now healthy 2024-08-19T14:23:49.245 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:49 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:23:49.246 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:49 smithi031 ceph-mon[16336]: osdmap e400: 12 total, 12 up, 9 in 2024-08-19T14:23:49.246 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:49 smithi031 ceph-mon[16336]: 4.e scrub starts 2024-08-19T14:23:49.344 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:23:49.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:49.386 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:49.387 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:23:49.488 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:23:49.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:49 smithi032 ceph-mon[20004]: 4.d scrub ok 2024-08-19T14:23:49.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:49 smithi032 ceph-mon[20004]: 4.2 scrub ok 2024-08-19T14:23:49.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:49 smithi032 ceph-mon[20004]: pgmap v916: 47 pgs: 47 active+clean; 2.5 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 932 op/s 2024-08-19T14:23:49.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:49 smithi032 ceph-mon[20004]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:23:49.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:49 smithi032 ceph-mon[20004]: Cluster is now healthy 2024-08-19T14:23:49.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:49 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:23:49.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:49 smithi032 ceph-mon[20004]: osdmap e400: 12 total, 12 up, 9 in 2024-08-19T14:23:49.509 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:49 smithi032 ceph-mon[20004]: 4.e scrub starts 2024-08-19T14:23:49.514 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:49.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:49.515 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:23:49.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:49 smithi045 ceph-mon[20143]: 4.d scrub ok 2024-08-19T14:23:49.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:49 smithi045 ceph-mon[20143]: 4.2 scrub ok 2024-08-19T14:23:49.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:49 smithi045 ceph-mon[20143]: pgmap v916: 47 pgs: 47 active+clean; 2.5 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 932 op/s 2024-08-19T14:23:49.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:49 smithi045 ceph-mon[20143]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:23:49.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:49 smithi045 ceph-mon[20143]: Cluster is now healthy 2024-08-19T14:23:49.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:49 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:23:49.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:49 smithi045 ceph-mon[20143]: osdmap e400: 12 total, 12 up, 9 in 2024-08-19T14:23:49.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:49 smithi045 ceph-mon[20143]: 4.e scrub starts 2024-08-19T14:23:49.581 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:23:49.581 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [ 2024-08-19T14:23:49.581 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:23:49.581 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.25333.0:225089 4.5 e400/335 4:a15c4882:::benchmark_data_smithi045_71562_object93914:head v 400'14150, mlcod=400'14147)", 2024-08-19T14:23:49.581 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:23:49.574615+0000", 2024-08-19T14:23:49.581 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.0051001500000000003, 2024-08-19T14:23:49.581 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0051298730000000001, 2024-08-19T14:23:49.581 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:23:49.582 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:23:49.582 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-08-19T14:23:49.582 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:23:49.582 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:23:49.582 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:23:49.582 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:23:49.574615+0000", 2024-08-19T14:23:49.582 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:23:49.582 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:23:49.582 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:23:49.582 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:23:49.582 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:23:49.574615+0000", 2024-08-19T14:23:49.583 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:23:49.583 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:23:49.583 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:23:49.583 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:23:49.587 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:23:49.574616+0000", 2024-08-19T14:23:49.587 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.1500000000000003e-07 2024-08-19T14:23:49.587 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:23:49.587 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:23:49.587 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:23:49.587 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:23:49.574622+0000", 2024-08-19T14:23:49.588 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.0000000000000002e-06 2024-08-19T14:23:49.588 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:23:49.588 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:23:49.588 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:23:49.588 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:23:49.574636+0000", 2024-08-19T14:23:49.589 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4398e-05 2024-08-19T14:23:49.589 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:23:49.589 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:23:49.589 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:23:49.589 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:23:49.574643+0000", 2024-08-19T14:23:49.589 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.9970000000000001e-06 2024-08-19T14:23:49.589 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:23:49.589 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:23:49.589 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:23:49.590 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:23:49.574708+0000", 2024-08-19T14:23:49.590 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.5310999999999999e-05 2024-08-19T14:23:49.590 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:23:49.590 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:23:49.590 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:23:49.590 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:23:49.574729+0000", 2024-08-19T14:23:49.590 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.016e-05 2024-08-19T14:23:49.590 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:23:49.590 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:23:49.590 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:23:49.591 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:23:49.591 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:23:49.591 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25333.0:225094 4.b 4.bc0f211b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e400)", 2024-08-19T14:23:49.591 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:23:49.579415+0000", 2024-08-19T14:23:49.591 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.00029983700000000001, 2024-08-19T14:23:49.591 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00039812700000000002, 2024-08-19T14:23:49.591 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:23:49.591 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:23:49.591 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "queued for pg", 2024-08-19T14:23:49.591 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:23:49.591 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25333", 2024-08-19T14:23:49.591 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:23:49.591 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 225094 2024-08-19T14:23:49.591 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:23:49.592 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:23:49.592 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:23:49.592 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:23:49.592 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:23:49.579415+0000", 2024-08-19T14:23:49.592 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:23:49.592 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:23:49.592 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:23:49.592 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:23:49.593 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:23:49.579415+0000", 2024-08-19T14:23:49.593 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:23:49.593 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:23:49.593 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:23:49.593 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:23:49.593 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:23:49.579417+0000", 2024-08-19T14:23:49.593 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.1490000000000001e-06 2024-08-19T14:23:49.593 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:23:49.593 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:23:49.593 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:23:49.593 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:23:49.579424+0000", 2024-08-19T14:23:49.594 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.7039999999999997e-06 2024-08-19T14:23:49.594 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:23:49.594 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:23:49.594 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:23:49.594 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:23:49.579437+0000", 2024-08-19T14:23:49.594 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2679e-05 2024-08-19T14:23:49.594 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:23:49.594 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:23:49.594 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:23:49.595 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:23:49.579445+0000", 2024-08-19T14:23:49.595 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.3420000000000008e-06 2024-08-19T14:23:49.595 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:23:49.595 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:23:49.595 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:23:49.595 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:23:49.595 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:23:49.595 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop_reply(client.25333.0:225091 4.b e400/335)", 2024-08-19T14:23:49.595 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:23:49.579476+0000", 2024-08-19T14:23:49.596 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.000239034, 2024-08-19T14:23:49.596 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00039929800000000002, 2024-08-19T14:23:49.596 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:23:49.596 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:23:49.596 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "queued for pg", 2024-08-19T14:23:49.596 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:23:49.596 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:23:49.596 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:23:49.596 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:23:49.579476+0000", 2024-08-19T14:23:49.597 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:23:49.597 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:23:49.597 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:23:49.597 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:23:49.598 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:23:49.579476+0000", 2024-08-19T14:23:49.598 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:23:49.598 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:23:49.598 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:23:49.598 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:23:49.598 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:23:49.579477+0000", 2024-08-19T14:23:49.598 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.6000000000000004e-07 2024-08-19T14:23:49.599 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:23:49.599 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:23:49.599 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:23:49.599 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:23:49.579480+0000", 2024-08-19T14:23:49.599 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.2449999999999998e-06 2024-08-19T14:23:49.600 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:23:49.600 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:23:49.600 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:23:49.600 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:23:49.579487+0000", 2024-08-19T14:23:49.600 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.1840000000000002e-06 2024-08-19T14:23:49.601 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:23:49.601 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:23:49.601 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:23:49.601 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:23:49.579492+0000", 2024-08-19T14:23:49.601 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0100000000000003e-06 2024-08-19T14:23:49.601 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:23:49.601 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:23:49.601 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:23:49.601 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:23:49.601 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:23:49.601 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.25333.0:225093 4.5 e400/335)", 2024-08-19T14:23:49.601 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:23:49.579506+0000", 2024-08-19T14:23:49.601 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.000208891, 2024-08-19T14:23:49.602 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00041669599999999998, 2024-08-19T14:23:49.602 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:23:49.602 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:23:49.602 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:23:49.602 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:23:49.602 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:23:49.602 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:23:49.602 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:23:49.579506+0000", 2024-08-19T14:23:49.602 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:23:49.603 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:23:49.603 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:23:49.603 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:23:49.603 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:23:49.579506+0000", 2024-08-19T14:23:49.604 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:23:49.604 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:23:49.604 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:23:49.604 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:23:49.604 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:23:49.579506+0000", 2024-08-19T14:23:49.604 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.1300000000000001e-07 2024-08-19T14:23:49.605 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:23:49.605 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:23:49.606 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:23:49.606 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:23:49.579512+0000", 2024-08-19T14:23:49.606 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.2290000000000002e-06 2024-08-19T14:23:49.606 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:23:49.606 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:23:49.606 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:23:49.606 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:23:49.579518+0000", 2024-08-19T14:23:49.606 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.3280000000000002e-06 2024-08-19T14:23:49.606 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:23:49.606 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:23:49.606 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:23:49.606 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:23:49.579522+0000", 2024-08-19T14:23:49.606 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.3329999999999998e-06 2024-08-19T14:23:49.607 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:23:49.607 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:23:49.607 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:23:49.607 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:23:49.579901+0000", 2024-08-19T14:23:49.607 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00037844300000000001 2024-08-19T14:23:49.607 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:23:49.607 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:23:49.607 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:23:49.607 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:23:49.607 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:23:49.607 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25333.0:225091 4.b 4:d49d6518:::benchmark_data_smithi045_71562_object93916:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e400)", 2024-08-19T14:23:49.608 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:23:49.572766+0000", 2024-08-19T14:23:49.608 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.0069486620000000004, 2024-08-19T14:23:49.608 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0072102110000000002, 2024-08-19T14:23:49.608 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:23:49.608 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:23:49.608 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "waiting for sub ops", 2024-08-19T14:23:49.608 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:23:49.608 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25333", 2024-08-19T14:23:49.608 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:23:49.608 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 225091 2024-08-19T14:23:49.608 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:23:49.608 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:23:49.608 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:23:49.609 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:23:49.609 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:23:49.572766+0000", 2024-08-19T14:23:49.609 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:23:49.609 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:23:49.609 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:23:49.609 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:23:49.609 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:23:49.572766+0000", 2024-08-19T14:23:49.609 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:23:49.609 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:23:49.609 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:23:49.609 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:23:49.609 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:23:49.572768+0000", 2024-08-19T14:23:49.610 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.48e-06 2024-08-19T14:23:49.610 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:23:49.610 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:23:49.610 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:23:49.610 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:23:49.572774+0000", 2024-08-19T14:23:49.610 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.2620000000000004e-06 2024-08-19T14:23:49.610 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:23:49.610 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:23:49.610 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:23:49.610 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:23:49.572792+0000", 2024-08-19T14:23:49.610 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.7827999999999999e-05 2024-08-19T14:23:49.610 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:23:49.610 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:23:49.611 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:23:49.611 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:23:49.572800+0000", 2024-08-19T14:23:49.611 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.2970000000000004e-06 2024-08-19T14:23:49.611 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:23:49.611 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:23:49.611 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:23:49.611 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:23:49.573038+0000", 2024-08-19T14:23:49.611 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00023729099999999999 2024-08-19T14:23:49.611 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:23:49.611 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:23:49.611 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:23:49.611 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:23:49.573338+0000", 2024-08-19T14:23:49.611 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000300656 2024-08-19T14:23:49.611 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:23:49.612 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:23:49.612 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 1,4", 2024-08-19T14:23:49.612 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:23:49.573698+0000", 2024-08-19T14:23:49.612 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00035978699999999998 2024-08-19T14:23:49.612 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:23:49.612 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:23:49.612 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:23:49.612 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:23:49.579943+0000", 2024-08-19T14:23:49.612 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0062445790000000001 2024-08-19T14:23:49.612 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:23:49.612 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:23:49.612 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:23:49.612 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:23:49.612 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-08-19T14:23:49.613 INFO:teuthology.orchestra.run.smithi031.stdout: "num_ops": 5 2024-08-19T14:23:49.613 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:23:49.616 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:23:49.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:49.644 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:49.644 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:23:49.746 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:23:49.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:49.799 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:49.799 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:23:49.901 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:23:49.927 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:49.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:49.929 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:23:50.030 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:23:50.060 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:50.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:50.061 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:23:50.163 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:23:50.193 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:50.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:50.195 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:23:50.296 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:23:50.324 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:50.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:50.325 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:23:50.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:50 smithi031 ceph-mon[16336]: 1.11 deep-scrub starts 2024-08-19T14:23:50.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:50 smithi031 ceph-mon[16336]: 1.11 deep-scrub ok 2024-08-19T14:23:50.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:50 smithi031 ceph-mon[16336]: 4.e scrub ok 2024-08-19T14:23:50.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:50 smithi031 ceph-mon[16336]: 1.d scrub starts 2024-08-19T14:23:50.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:50 smithi031 ceph-mon[16336]: 1.d scrub ok 2024-08-19T14:23:50.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:50 smithi031 ceph-mon[16336]: 1.13 scrub starts 2024-08-19T14:23:50.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:50 smithi031 ceph-mon[16336]: 1.13 scrub ok 2024-08-19T14:23:50.408 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:50 smithi032 ceph-mon[20004]: 1.11 deep-scrub starts 2024-08-19T14:23:50.408 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:50 smithi032 ceph-mon[20004]: 1.11 deep-scrub ok 2024-08-19T14:23:50.408 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:50 smithi032 ceph-mon[20004]: 4.e scrub ok 2024-08-19T14:23:50.408 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:50 smithi032 ceph-mon[20004]: 1.d scrub starts 2024-08-19T14:23:50.408 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:50 smithi032 ceph-mon[20004]: 1.d scrub ok 2024-08-19T14:23:50.408 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:50 smithi032 ceph-mon[20004]: 1.13 scrub starts 2024-08-19T14:23:50.409 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:50 smithi032 ceph-mon[20004]: 1.13 scrub ok 2024-08-19T14:23:50.426 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:23:50.456 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:50.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:50.457 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:23:50.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:50 smithi045 ceph-mon[20143]: 1.11 deep-scrub starts 2024-08-19T14:23:50.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:50 smithi045 ceph-mon[20143]: 1.11 deep-scrub ok 2024-08-19T14:23:50.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:50 smithi045 ceph-mon[20143]: 4.e scrub ok 2024-08-19T14:23:50.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:50 smithi045 ceph-mon[20143]: 1.d scrub starts 2024-08-19T14:23:50.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:50 smithi045 ceph-mon[20143]: 1.d scrub ok 2024-08-19T14:23:50.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:50 smithi045 ceph-mon[20143]: 1.13 scrub starts 2024-08-19T14:23:50.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:50 smithi045 ceph-mon[20143]: 1.13 scrub ok 2024-08-19T14:23:50.557 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:23:50.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:50.597 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:50.597 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:23:50.698 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:23:50.732 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:50.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:50.733 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:23:50.834 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:23:50.867 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:50.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:50.869 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:23:50.970 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:23:51.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:51.006 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:51.006 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:23:51.029 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-19T14:23:51.081 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-19T14:23:51.082 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-19T14:23:51.107 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:23:51.136 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:51.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:51.137 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:23:51.238 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:23:51.261 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:51.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:51.262 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:23:51.363 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:23:51.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:51 smithi031 ceph-mon[16336]: pgmap v918: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 2.4 GiB data, 11 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 930 op/s 2024-08-19T14:23:51.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:51 smithi031 ceph-mon[16336]: 4.6 scrub starts 2024-08-19T14:23:51.388 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:51.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:51.389 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:23:51.489 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:23:51.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:51 smithi032 ceph-mon[20004]: pgmap v918: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 2.4 GiB data, 11 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 930 op/s 2024-08-19T14:23:51.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:51 smithi032 ceph-mon[20004]: 4.6 scrub starts 2024-08-19T14:23:51.520 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:51.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:51.521 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:23:51.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:51 smithi045 ceph-mon[20143]: pgmap v918: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 2.4 GiB data, 11 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 930 op/s 2024-08-19T14:23:51.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:51 smithi045 ceph-mon[20143]: 4.6 scrub starts 2024-08-19T14:23:51.622 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:23:51.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:51.646 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:51.646 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:23:51.747 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:23:51.773 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:51.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:51.774 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:23:51.875 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:23:51.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:51.898 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:51.899 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:23:51.999 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:23:52.046 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:52.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:52.048 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:23:52.149 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:23:52.189 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:52.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:52.190 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:23:52.291 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:23:52.317 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:52.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:52.318 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:23:52.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:52 smithi031 ceph-mon[16336]: 4.6 scrub ok 2024-08-19T14:23:52.418 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:23:52.452 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:52.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:52.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:23:52.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:52 smithi032 ceph-mon[20004]: 4.6 scrub ok 2024-08-19T14:23:52.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:52 smithi045 ceph-mon[20143]: 4.6 scrub ok 2024-08-19T14:23:52.553 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:23:52.590 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:52.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:52.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:23:52.692 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:23:52.719 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:52.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:52.720 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:23:52.821 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:23:52.858 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:52.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:52.862 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:23:52.963 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:23:52.986 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:52.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:52.987 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:23:53.088 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:23:53.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:53.110 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:53.110 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:23:53.211 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:23:53.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:53.250 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:53.250 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:23:53.351 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:23:53.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:53 smithi031 ceph-mon[16336]: pgmap v919: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 2.3 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 901 op/s 2024-08-19T14:23:53.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:53.383 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:53.383 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:23:53.484 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:23:53.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:53 smithi032 ceph-mon[20004]: pgmap v919: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 2.3 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 901 op/s 2024-08-19T14:23:53.514 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:53.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:53.516 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:23:53.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:53 smithi045 ceph-mon[20143]: pgmap v919: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 2.3 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 901 op/s 2024-08-19T14:23:53.616 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:23:53.643 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:53.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:53.645 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:23:53.746 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:23:53.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:53.772 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:53.773 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:23:53.873 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:23:53.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:53.912 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:53.913 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:23:54.014 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:23:54.016 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:23:54.022 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-08-19T14:23:54.036 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:54.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:54.037 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:23:54.138 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:23:54.198 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:54.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:54.202 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:23:54.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:54 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:23:54.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:54 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:23:54.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:54 smithi045 ceph-mon[20143]: osdmap e401: 12 total, 12 up, 9 in 2024-08-19T14:23:54.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:54 smithi045 ceph-mon[20143]: 4.a scrub starts 2024-08-19T14:23:54.303 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:23:54.328 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:54.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:54.329 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:23:54.364 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:54 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:23:54.364 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:54 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:23:54.365 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:54 smithi031 ceph-mon[16336]: osdmap e401: 12 total, 12 up, 9 in 2024-08-19T14:23:54.365 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:54 smithi031 ceph-mon[16336]: 4.a scrub starts 2024-08-19T14:23:54.429 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:23:54.487 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:54.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:54.488 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:23:54.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:54 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:23:54.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:54 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:23:54.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:54 smithi032 ceph-mon[20004]: osdmap e401: 12 total, 12 up, 9 in 2024-08-19T14:23:54.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:54 smithi032 ceph-mon[20004]: 4.a scrub starts 2024-08-19T14:23:54.589 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:23:54.629 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:23:54.641 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:54.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:54.642 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:23:54.712 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:23:54.712 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:23:54.730 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:23:54.731 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:23:54.742 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:23:54.748 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:23:54.748 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:23:54.759 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:23:54.764 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:54.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:54.765 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:23:54.769 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:23:54.769 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:23:54.794 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:23:54.795 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:23:54.811 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:23:54.811 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:23:54.828 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:23:54.828 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:23:54.848 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:23:54.848 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:23:54.866 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:23:54.877 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:23:54.878 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:23:54.911 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:23:54.911 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:23:54.930 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:54.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:54.931 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:23:54.947 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:23:54.947 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:23:54.983 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:23:54.984 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:23:55.031 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:23:55.057 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:55.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:55.058 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:23:55.160 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:23:55.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:55.196 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:55.196 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:23:55.230 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:55 smithi031 ceph-mon[16336]: pgmap v921: 47 pgs: 47 active+clean; 2.1 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 1.15k op/s 2024-08-19T14:23:55.230 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:55 smithi031 ceph-mon[16336]: 4.8 scrub starts 2024-08-19T14:23:55.230 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:55 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/4278777116' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:23:55.297 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:23:55.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:55.324 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:55.324 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:23:55.376 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:55 smithi045 ceph-mon[20143]: pgmap v921: 47 pgs: 47 active+clean; 2.1 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 1.15k op/s 2024-08-19T14:23:55.376 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:55 smithi045 ceph-mon[20143]: 4.8 scrub starts 2024-08-19T14:23:55.376 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:55 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/4278777116' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:23:55.425 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:23:55.447 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:55.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:55.448 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:23:55.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:55 smithi032 ceph-mon[20004]: pgmap v921: 47 pgs: 47 active+clean; 2.1 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 1.15k op/s 2024-08-19T14:23:55.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:55 smithi032 ceph-mon[20004]: 4.8 scrub starts 2024-08-19T14:23:55.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:55 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/4278777116' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:23:55.548 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:23:55.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:55.573 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:55.574 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:23:55.675 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:23:55.707 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:55.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:55.708 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:23:55.810 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:23:55.837 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:55.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:55.838 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:23:55.939 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:23:55.967 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:55.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:55.968 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:23:56.069 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:23:56.083 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 7, 8, 9, 11] out_osds: [10, 6, 2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-08-19T14:23:56.083 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-19T14:23:56.083 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2024-08-19T14:23:56.084 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-08-19T14:23:56.084 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-08-19T14:23:56.084 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.4 config set filestore_inject_stall 3 2024-08-19T14:23:56.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:56.114 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:56.114 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:23:56.215 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:23:56.255 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:56.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:56.256 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:23:56.357 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:23:56.392 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:56.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:56.393 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:23:56.424 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:56 smithi031 ceph-mon[16336]: 4.8 scrub ok 2024-08-19T14:23:56.424 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:56 smithi031 ceph-mon[16336]: 4.a scrub ok 2024-08-19T14:23:56.424 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:56 smithi031 ceph-mon[16336]: 4.4 scrub starts 2024-08-19T14:23:56.451 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:23:56.494 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:23:56.509 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:56 smithi032 ceph-mon[20004]: 4.8 scrub ok 2024-08-19T14:23:56.510 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:56 smithi032 ceph-mon[20004]: 4.a scrub ok 2024-08-19T14:23:56.510 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:56 smithi032 ceph-mon[20004]: 4.4 scrub starts 2024-08-19T14:23:56.529 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:56.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:56.533 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:23:56.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:56 smithi045 ceph-mon[20143]: 4.8 scrub ok 2024-08-19T14:23:56.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:56 smithi045 ceph-mon[20143]: 4.a scrub ok 2024-08-19T14:23:56.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:56 smithi045 ceph-mon[20143]: 4.4 scrub starts 2024-08-19T14:23:56.633 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:23:56.667 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:56.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:56.669 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:23:56.770 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:23:56.801 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:56.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:56.802 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:23:56.849 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:23:56.849 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:23:56.903 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:23:56.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:56.935 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:56.935 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:23:57.036 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:23:57.039 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:23:57.039 INFO:teuthology.orchestra.run.smithi032.stdout: "success": "" 2024-08-19T14:23:57.039 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:23:57.072 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:57.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:57.073 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:23:57.134 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is set 2024-08-19T14:23:57.173 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:23:57.208 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:57.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:57.209 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:23:57.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:57 smithi032 ceph-mon[20004]: pgmap v922: 47 pgs: 47 active+clean; 2.1 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 799 op/s 2024-08-19T14:23:57.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:57 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2047375168' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:23:57.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:57 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:23:57.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:57 smithi032 ceph-mon[20004]: 4.4 scrub ok 2024-08-19T14:23:57.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:57 smithi032 ceph-mon[20004]: 4.1 scrub starts 2024-08-19T14:23:57.310 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:23:57.343 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:57.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:57.344 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:23:57.444 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:23:57.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:57.480 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:57.480 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:23:57.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:57 smithi045 ceph-mon[20143]: pgmap v922: 47 pgs: 47 active+clean; 2.1 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 799 op/s 2024-08-19T14:23:57.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:57 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2047375168' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:23:57.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:57 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:23:57.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:57 smithi045 ceph-mon[20143]: 4.4 scrub ok 2024-08-19T14:23:57.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:57 smithi045 ceph-mon[20143]: 4.1 scrub starts 2024-08-19T14:23:57.581 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:23:57.605 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:57.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:57.606 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:23:57.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:57 smithi031 ceph-mon[16336]: pgmap v922: 47 pgs: 47 active+clean; 2.1 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 799 op/s 2024-08-19T14:23:57.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:57 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2047375168' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:23:57.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:57 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:23:57.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:57 smithi031 ceph-mon[16336]: 4.4 scrub ok 2024-08-19T14:23:57.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:57 smithi031 ceph-mon[16336]: 4.1 scrub starts 2024-08-19T14:23:57.681 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-08-19T14:23:57.708 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:23:57.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:57.735 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:57.735 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:23:57.836 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:23:57.869 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:57.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:57.870 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:23:57.971 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:23:58.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:58.017 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:58.017 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:23:58.118 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:23:58.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:58.154 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:58.155 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:23:58.255 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:23:58.291 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:58.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:58.293 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:23:58.342 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:23:58.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:58 smithi031 ceph-mon[16336]: 4.1 scrub ok 2024-08-19T14:23:58.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:58 smithi031 ceph-mon[16336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:23:58.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:58 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:23:58.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:58 smithi031 ceph-mon[16336]: osdmap e402: 12 total, 12 up, 9 in 2024-08-19T14:23:58.394 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:23:58.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:58.431 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:58.431 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:23:58.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:58 smithi032 ceph-mon[20004]: 4.1 scrub ok 2024-08-19T14:23:58.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:58 smithi032 ceph-mon[20004]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:23:58.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:58 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:23:58.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:58 smithi032 ceph-mon[20004]: osdmap e402: 12 total, 12 up, 9 in 2024-08-19T14:23:58.533 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:23:58.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:58 smithi045 ceph-mon[20143]: 4.1 scrub ok 2024-08-19T14:23:58.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:58 smithi045 ceph-mon[20143]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:23:58.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:58 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:23:58.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:58 smithi045 ceph-mon[20143]: osdmap e402: 12 total, 12 up, 9 in 2024-08-19T14:23:58.558 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:58.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:58.559 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:23:58.572 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:23:58.660 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:23:58.686 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:58.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:58.687 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:23:58.788 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:23:58.814 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:58.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:58.815 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:23:58.916 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:23:58.964 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:58.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:58.965 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:23:59.066 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:23:59.092 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:59.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:59.093 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:23:59.189 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:23:59.189 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:23:59.193 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:23:59.202 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:23:59.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:59.219 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:59.220 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:23:59.321 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:23:59.347 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:59.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:59.348 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:23:59.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:23:59 smithi031 ceph-mon[16336]: pgmap v924: 47 pgs: 47 active+clean; 2.0 GiB data, 10 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 867 op/s 2024-08-19T14:23:59.448 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:23:59.473 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:59.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:59.474 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:23:59.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:23:59 smithi032 ceph-mon[20004]: pgmap v924: 47 pgs: 47 active+clean; 2.0 GiB data, 10 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 867 op/s 2024-08-19T14:23:59.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:23:59 smithi045 ceph-mon[20143]: pgmap v924: 47 pgs: 47 active+clean; 2.0 GiB data, 10 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 867 op/s 2024-08-19T14:23:59.576 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:23:59.617 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:23:59.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:59.618 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:23:59.719 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:23:59.743 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:23:59.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:59.744 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:23:59.845 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:23:59.869 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:23:59.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:23:59.870 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:23:59.971 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:24:00.002 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:00.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:00.003 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:24:00.105 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:24:00.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:00.141 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:00.141 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:24:00.242 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:24:00.250 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-08-19T14:24:00.266 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:00.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:00.267 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:24:00.369 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:24:00.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:00.393 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:00.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:24:00.489 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-19T14:24:00.494 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:24:00.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:00.593 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:00.594 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:24:00.694 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:24:00.718 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:00.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:00.719 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:24:00.821 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:24:00.843 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:00.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:00.844 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:24:00.944 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:24:00.966 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:00.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:00.967 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:24:01.068 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:24:01.094 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:01.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:01.096 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:24:01.197 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:24:01.228 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:01.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:01.229 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:24:01.312 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:24:01.312 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:24:01.330 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:24:01.343 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:24:01.355 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:01.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:01.356 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:24:01.359 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:24:01.359 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:24:01.394 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:01 smithi031 ceph-mon[16336]: pgmap v925: 47 pgs: 47 active+clean; 1.9 GiB data, 9.8 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 884 op/s 2024-08-19T14:24:01.395 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:24:01.395 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:24:01.423 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:24:01.436 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:24:01.436 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:24:01.457 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:24:01.475 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:24:01.475 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:24:01.497 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:01.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:01.498 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:24:01.507 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:24:01.507 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:24:01.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:01 smithi032 ceph-mon[20004]: pgmap v925: 47 pgs: 47 active+clean; 1.9 GiB data, 9.8 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 884 op/s 2024-08-19T14:24:01.538 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:24:01.539 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:24:01.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:01 smithi045 ceph-mon[20143]: pgmap v925: 47 pgs: 47 active+clean; 1.9 GiB data, 9.8 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 884 op/s 2024-08-19T14:24:01.569 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:24:01.569 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:24:01.598 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:24:01.598 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:24:01.598 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:24:01.627 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:01.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:01.627 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:24:01.630 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:24:01.630 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:24:01.663 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:24:01.664 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:24:01.705 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:24:01.705 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:24:01.728 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:24:01.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:01.751 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:01.751 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:24:01.852 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:24:01.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:01.893 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:01.893 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:24:01.994 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:24:02.021 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:02.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:02.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:24:02.124 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:24:02.149 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:02.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:02.150 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:24:02.234 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:02 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/680640766' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:24:02.250 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:24:02.280 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:02.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:02.281 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:24:02.382 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:24:02.412 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:02.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:02.413 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:24:02.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:02 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/680640766' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:24:02.514 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:24:02.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:02 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/680640766' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:24:02.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:02.547 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:02.547 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:24:02.649 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:24:02.680 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:02.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:02.681 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:24:02.782 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:24:02.807 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:02.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:02.808 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:24:02.861 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 7, 8, 9, 11] out_osds: [10, 6, 2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-08-19T14:24:02.861 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-19T14:24:02.862 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-08-19T14:24:02.862 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-19T14:24:02.909 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:24:02.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:02.968 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:02.969 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:24:03.070 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:24:03.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:03.128 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:03.128 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:24:03.158 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:24:03.159 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [ 2024-08-19T14:24:03.159 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:03.159 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25333.0:237207 4.4 4:2233b2cb:::benchmark_data_smithi045_71562_object106032:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e402)", 2024-08-19T14:24:03.159 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:24:03.143933+0000", 2024-08-19T14:24:03.159 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.010116114000000001, 2024-08-19T14:24:03.159 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.010147883999999999, 2024-08-19T14:24:03.159 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:24:03.159 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:24:03.159 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "waiting for sub ops", 2024-08-19T14:24:03.159 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:24:03.159 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25333", 2024-08-19T14:24:03.159 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:24:03.160 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 237207 2024-08-19T14:24:03.160 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:03.160 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:24:03.160 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:03.160 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:24:03.160 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:03.143933+0000", 2024-08-19T14:24:03.160 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:24:03.160 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:03.160 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:03.160 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:24:03.160 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:03.143933+0000", 2024-08-19T14:24:03.160 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:24:03.160 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:03.160 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:03.161 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:24:03.161 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:03.143934+0000", 2024-08-19T14:24:03.161 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.584e-06 2024-08-19T14:24:03.161 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:03.161 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:03.161 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:24:03.161 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:03.143940+0000", 2024-08-19T14:24:03.161 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.8109999999999999e-06 2024-08-19T14:24:03.161 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:03.161 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:03.161 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:24:03.161 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:03.143958+0000", 2024-08-19T14:24:03.162 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.8002000000000001e-05 2024-08-19T14:24:03.162 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:03.162 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:03.162 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:24:03.162 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:03.143968+0000", 2024-08-19T14:24:03.162 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.6900000000000004e-06 2024-08-19T14:24:03.162 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:03.163 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:03.163 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:24:03.163 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:03.150600+0000", 2024-08-19T14:24:03.163 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0066325020000000002 2024-08-19T14:24:03.163 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:03.163 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:03.163 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:24:03.163 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:03.150732+0000", 2024-08-19T14:24:03.163 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013191099999999999 2024-08-19T14:24:03.163 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:03.163 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:03.163 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,7", 2024-08-19T14:24:03.164 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:03.151407+0000", 2024-08-19T14:24:03.164 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00067511700000000003 2024-08-19T14:24:03.164 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:24:03.164 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:24:03.164 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:24:03.164 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:03.164 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:03.164 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25333.0:237210 4.4 4.20fb66a4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e402)", 2024-08-19T14:24:03.164 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:24:03.144684+0000", 2024-08-19T14:24:03.164 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.0093653810000000007, 2024-08-19T14:24:03.164 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0094761299999999993, 2024-08-19T14:24:03.164 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:24:03.164 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:24:03.165 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "queued for pg", 2024-08-19T14:24:03.165 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:24:03.165 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25333", 2024-08-19T14:24:03.165 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:24:03.165 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 237210 2024-08-19T14:24:03.165 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:03.165 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:24:03.165 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:03.165 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:24:03.165 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:03.144684+0000", 2024-08-19T14:24:03.165 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:24:03.165 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:03.165 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:03.165 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:24:03.166 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:03.144684+0000", 2024-08-19T14:24:03.166 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:24:03.166 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:03.166 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:03.166 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:24:03.166 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:03.144685+0000", 2024-08-19T14:24:03.166 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.9399999999999993e-07 2024-08-19T14:24:03.166 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:03.166 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:03.166 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:24:03.166 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:03.144689+0000", 2024-08-19T14:24:03.166 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7770000000000002e-06 2024-08-19T14:24:03.166 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:03.167 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:03.167 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:24:03.167 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:03.144705+0000", 2024-08-19T14:24:03.167 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5449000000000001e-05 2024-08-19T14:24:03.167 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:03.167 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:03.167 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:24:03.167 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:03.144712+0000", 2024-08-19T14:24:03.167 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.3440000000000003e-06 2024-08-19T14:24:03.167 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:24:03.167 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:24:03.167 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:24:03.167 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:03.167 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:03.168 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.25333.0:237211 4.e e402/335 4:780d1c9f:::benchmark_data_smithi045_71562_object106036:head v 402'14598, mlcod=402'14595)", 2024-08-19T14:24:03.168 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:24:03.145585+0000", 2024-08-19T14:24:03.168 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.0084642810000000006, 2024-08-19T14:24:03.168 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0086346349999999999, 2024-08-19T14:24:03.168 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:24:03.168 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:24:03.168 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-08-19T14:24:03.168 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:24:03.168 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:03.168 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:24:03.168 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:03.145585+0000", 2024-08-19T14:24:03.168 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:24:03.168 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:03.169 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:03.169 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:24:03.169 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:03.145585+0000", 2024-08-19T14:24:03.169 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:24:03.169 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:03.169 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:03.169 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:24:03.169 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:03.145585+0000", 2024-08-19T14:24:03.169 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.1299999999999999e-07 2024-08-19T14:24:03.169 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:03.169 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:03.169 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:24:03.169 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:03.145591+0000", 2024-08-19T14:24:03.169 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.8039999999999998e-06 2024-08-19T14:24:03.170 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:03.170 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:03.170 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:24:03.170 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:03.145605+0000", 2024-08-19T14:24:03.170 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3502e-05 2024-08-19T14:24:03.170 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:03.170 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:03.170 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:24:03.170 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:03.145611+0000", 2024-08-19T14:24:03.170 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.6429999999999998e-06 2024-08-19T14:24:03.170 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:03.170 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:03.170 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:24:03.171 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:03.146979+0000", 2024-08-19T14:24:03.171 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0013677419999999999 2024-08-19T14:24:03.171 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:03.171 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:03.171 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:24:03.171 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:03.147004+0000", 2024-08-19T14:24:03.171 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.5227000000000001e-05 2024-08-19T14:24:03.171 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:24:03.171 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:24:03.171 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:24:03.171 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:03.171 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:03.171 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.25333.0:237213 4.d e402/335 4:bc9081d7:::benchmark_data_smithi045_71562_object106038:head v 402'14663, mlcod=402'14659)", 2024-08-19T14:24:03.171 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:24:03.150247+0000", 2024-08-19T14:24:03.172 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.003802279, 2024-08-19T14:24:03.172 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0040314870000000003, 2024-08-19T14:24:03.172 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:24:03.172 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:24:03.172 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-08-19T14:24:03.172 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:24:03.172 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:03.172 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:24:03.172 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:03.150247+0000", 2024-08-19T14:24:03.172 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:24:03.172 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:03.172 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:03.172 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:24:03.173 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:03.150247+0000", 2024-08-19T14:24:03.173 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:24:03.173 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:03.173 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:03.173 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:24:03.173 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:03.150248+0000", 2024-08-19T14:24:03.173 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.3399999999999997e-07 2024-08-19T14:24:03.173 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:03.173 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:03.173 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:24:03.173 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:03.150258+0000", 2024-08-19T14:24:03.173 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0499999999999999e-05 2024-08-19T14:24:03.173 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:03.173 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:03.174 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:24:03.174 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:03.150271+0000", 2024-08-19T14:24:03.174 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2405000000000001e-05 2024-08-19T14:24:03.174 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:03.174 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:03.174 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:24:03.174 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:03.150277+0000", 2024-08-19T14:24:03.174 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.7490000000000001e-06 2024-08-19T14:24:03.174 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:03.174 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:03.174 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:24:03.174 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:03.150653+0000", 2024-08-19T14:24:03.174 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00037558200000000001 2024-08-19T14:24:03.175 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:03.175 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:03.175 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:24:03.175 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:03.150671+0000", 2024-08-19T14:24:03.175 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.8655999999999999e-05 2024-08-19T14:24:03.175 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:24:03.175 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:24:03.175 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:24:03.175 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:03.175 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:03.175 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25333.0:237217 4.f 4:f383a297:::benchmark_data_smithi045_71562_object106042:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e402)", 2024-08-19T14:24:03.175 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:24:03.151927+0000", 2024-08-19T14:24:03.175 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.0021224339999999999, 2024-08-19T14:24:03.175 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0024101719999999999, 2024-08-19T14:24:03.176 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:24:03.176 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:24:03.176 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "waiting for sub ops", 2024-08-19T14:24:03.176 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:24:03.176 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25333", 2024-08-19T14:24:03.176 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:24:03.176 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 237217 2024-08-19T14:24:03.176 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:03.176 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:24:03.176 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:03.176 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:24:03.176 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:03.151927+0000", 2024-08-19T14:24:03.176 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:24:03.177 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:03.177 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:03.177 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:24:03.177 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:03.151927+0000", 2024-08-19T14:24:03.177 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:24:03.177 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:03.177 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:03.177 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:24:03.177 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:03.151928+0000", 2024-08-19T14:24:03.177 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2219999999999999e-06 2024-08-19T14:24:03.177 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:03.177 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:03.177 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:24:03.177 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:03.151932+0000", 2024-08-19T14:24:03.178 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.5549999999999996e-06 2024-08-19T14:24:03.178 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:03.178 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:03.178 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:24:03.178 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:03.151943+0000", 2024-08-19T14:24:03.178 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0905e-05 2024-08-19T14:24:03.178 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:03.178 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:03.178 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:24:03.178 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:03.151950+0000", 2024-08-19T14:24:03.178 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.294e-06 2024-08-19T14:24:03.178 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:03.178 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:03.178 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:24:03.179 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:03.152977+0000", 2024-08-19T14:24:03.179 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.001026669 2024-08-19T14:24:03.179 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:03.179 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:03.179 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:24:03.179 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:03.153058+0000", 2024-08-19T14:24:03.179 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.0952999999999996e-05 2024-08-19T14:24:03.179 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:03.179 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:03.179 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,11", 2024-08-19T14:24:03.179 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:03.153130+0000", 2024-08-19T14:24:03.179 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.1454999999999999e-05 2024-08-19T14:24:03.179 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:24:03.179 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:24:03.180 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:24:03.180 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:03.180 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:03.180 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.25333.0:237215 4.6 e402/384 4:6f0696bc:::benchmark_data_smithi045_71562_object106040:head v 402'15041, mlcod=402'15039)", 2024-08-19T14:24:03.180 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:24:03.152397+0000", 2024-08-19T14:24:03.180 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.001652349, 2024-08-19T14:24:03.180 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0020105100000000001, 2024-08-19T14:24:03.180 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:24:03.180 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:24:03.180 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-08-19T14:24:03.180 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:24:03.180 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:03.180 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:24:03.181 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:03.152397+0000", 2024-08-19T14:24:03.181 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:24:03.181 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:03.181 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:03.181 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:24:03.181 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:03.152397+0000", 2024-08-19T14:24:03.181 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:24:03.181 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:03.181 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:03.181 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:24:03.181 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:03.152397+0000", 2024-08-19T14:24:03.181 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.54e-07 2024-08-19T14:24:03.181 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:03.182 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:03.182 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:24:03.182 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:03.152412+0000", 2024-08-19T14:24:03.182 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4096e-05 2024-08-19T14:24:03.182 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:03.182 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:03.182 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:24:03.182 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:03.152423+0000", 2024-08-19T14:24:03.182 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1863e-05 2024-08-19T14:24:03.182 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:03.182 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:03.182 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:24:03.182 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:03.152431+0000", 2024-08-19T14:24:03.183 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.2949999999999996e-06 2024-08-19T14:24:03.183 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:03.183 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:03.183 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:24:03.183 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:03.152482+0000", 2024-08-19T14:24:03.183 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1344999999999997e-05 2024-08-19T14:24:03.184 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:03.184 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:03.184 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:24:03.184 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:03.152502+0000", 2024-08-19T14:24:03.184 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.9627999999999999e-05 2024-08-19T14:24:03.184 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:24:03.184 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:24:03.184 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:24:03.184 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:03.184 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:03.185 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25333.0:237220 4.4 4.a3cac534 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e402)", 2024-08-19T14:24:03.185 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:24:03.152902+0000", 2024-08-19T14:24:03.185 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.001146825, 2024-08-19T14:24:03.185 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.001562019, 2024-08-19T14:24:03.185 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:24:03.185 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:24:03.185 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "queued for pg", 2024-08-19T14:24:03.185 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:24:03.185 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25333", 2024-08-19T14:24:03.185 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:24:03.185 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 237220 2024-08-19T14:24:03.185 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:03.185 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:24:03.185 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:03.186 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:24:03.186 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:03.152902+0000", 2024-08-19T14:24:03.186 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:24:03.186 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:03.186 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:03.186 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:24:03.186 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:03.152902+0000", 2024-08-19T14:24:03.186 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:24:03.186 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:03.186 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:03.186 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:24:03.186 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:03.152903+0000", 2024-08-19T14:24:03.187 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.1100000000000005e-07 2024-08-19T14:24:03.187 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:03.187 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:03.187 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:24:03.187 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:03.152906+0000", 2024-08-19T14:24:03.187 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.0299999999999998e-06 2024-08-19T14:24:03.187 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:03.187 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:03.187 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:24:03.187 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:03.152917+0000", 2024-08-19T14:24:03.187 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0978999999999999e-05 2024-08-19T14:24:03.187 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:03.187 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:03.187 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:24:03.188 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:03.152922+0000", 2024-08-19T14:24:03.188 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.3669999999999996e-06 2024-08-19T14:24:03.188 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:24:03.188 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:24:03.188 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:24:03.188 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:03.188 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:03.188 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25333.0:237221 4.f 4.e535555f (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e402)", 2024-08-19T14:24:03.188 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:24:03.152933+0000", 2024-08-19T14:24:03.189 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.001115963, 2024-08-19T14:24:03.189 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.001583947, 2024-08-19T14:24:03.189 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:24:03.189 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:24:03.189 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "queued for pg", 2024-08-19T14:24:03.189 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:24:03.189 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25333", 2024-08-19T14:24:03.189 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:24:03.189 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 237221 2024-08-19T14:24:03.189 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:03.189 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:24:03.189 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:03.189 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:24:03.189 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:03.152933+0000", 2024-08-19T14:24:03.190 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:24:03.190 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:03.190 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:03.190 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:24:03.190 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:03.152933+0000", 2024-08-19T14:24:03.190 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:24:03.190 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:03.190 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:03.190 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:24:03.190 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:03.152933+0000", 2024-08-19T14:24:03.190 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.3099999999999999e-07 2024-08-19T14:24:03.190 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:03.190 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:03.191 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:24:03.191 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:03.152935+0000", 2024-08-19T14:24:03.191 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.779e-06 2024-08-19T14:24:03.191 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:03.191 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:03.191 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:24:03.191 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:03.152939+0000", 2024-08-19T14:24:03.191 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.1849999999999997e-06 2024-08-19T14:24:03.191 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:03.191 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:03.191 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:24:03.192 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:03.152943+0000", 2024-08-19T14:24:03.192 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.303e-06 2024-08-19T14:24:03.192 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:24:03.192 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:24:03.192 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:24:03.192 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:03.192 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:03.192 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25333.0:237205 4.4 4:2e08c544:::benchmark_data_smithi045_71562_object106030:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e402)", 2024-08-19T14:24:03.192 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:24:03.140050+0000", 2024-08-19T14:24:03.192 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.013998564, 2024-08-19T14:24:03.192 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.014525163000000001, 2024-08-19T14:24:03.192 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:24:03.192 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:24:03.193 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "waiting for sub ops", 2024-08-19T14:24:03.193 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:24:03.193 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25333", 2024-08-19T14:24:03.193 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:24:03.193 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 237205 2024-08-19T14:24:03.193 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:03.193 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:24:03.193 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:03.193 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:24:03.193 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:03.140050+0000", 2024-08-19T14:24:03.193 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:24:03.193 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:03.194 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:03.194 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:24:03.194 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:03.140050+0000", 2024-08-19T14:24:03.194 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:24:03.194 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:03.194 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:03.194 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:24:03.194 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:03.140051+0000", 2024-08-19T14:24:03.194 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.8100000000000003e-07 2024-08-19T14:24:03.194 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:03.194 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:03.194 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:24:03.194 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:03.140054+0000", 2024-08-19T14:24:03.195 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.1879999999999998e-06 2024-08-19T14:24:03.195 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:03.195 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:03.195 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:24:03.195 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:03.140064+0000", 2024-08-19T14:24:03.195 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.4839999999999993e-06 2024-08-19T14:24:03.195 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:03.195 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:03.195 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:24:03.195 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:03.140073+0000", 2024-08-19T14:24:03.195 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.8300000000000008e-06 2024-08-19T14:24:03.195 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:03.195 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:03.195 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:24:03.195 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:03.140125+0000", 2024-08-19T14:24:03.196 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1505000000000002e-05 2024-08-19T14:24:03.196 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:03.196 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:03.196 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:24:03.196 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:03.140262+0000", 2024-08-19T14:24:03.196 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013685100000000001 2024-08-19T14:24:03.196 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:03.196 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:03.196 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,7", 2024-08-19T14:24:03.196 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:03.140387+0000", 2024-08-19T14:24:03.196 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000125661 2024-08-19T14:24:03.196 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:03.196 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:03.197 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:24:03.197 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:03.150532+0000", 2024-08-19T14:24:03.197 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.010144851999999999 2024-08-19T14:24:03.197 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:03.197 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:03.197 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:24:03.197 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:03.150568+0000", 2024-08-19T14:24:03.197 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.5840000000000002e-05 2024-08-19T14:24:03.197 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:24:03.197 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:24:03.197 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:24:03.197 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:24:03.197 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-08-19T14:24:03.197 INFO:teuthology.orchestra.run.smithi031.stdout: "num_ops": 9 2024-08-19T14:24:03.197 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:24:03.198 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is set 2024-08-19T14:24:03.229 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:24:03.253 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:03.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:03.254 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:24:03.355 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:24:03.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:03 smithi031 ceph-mon[16336]: pgmap v926: 47 pgs: 47 active+clean; 1.8 GiB data, 9.6 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 856 op/s 2024-08-19T14:24:03.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:03 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2622294244' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:24:03.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:03 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:24:03.385 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:03.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:03.386 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:24:03.487 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:24:03.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:03 smithi032 ceph-mon[20004]: pgmap v926: 47 pgs: 47 active+clean; 1.8 GiB data, 9.6 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 856 op/s 2024-08-19T14:24:03.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:03 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2622294244' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:24:03.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:03 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:24:03.514 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:03.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:03.515 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:24:03.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:03 smithi045 ceph-mon[20143]: pgmap v926: 47 pgs: 47 active+clean; 1.8 GiB data, 9.6 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 856 op/s 2024-08-19T14:24:03.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:03 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2622294244' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:24:03.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:03 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:24:03.616 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:24:03.644 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:03.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:03.645 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:24:03.746 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:24:03.779 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:03.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:03.780 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:24:03.881 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:24:03.915 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:03.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:03.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:24:04.017 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:24:04.046 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:04.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:04.048 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:24:04.148 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:24:04.181 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:04.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:04.182 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:24:04.283 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:24:04.308 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:04.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:04.309 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:24:04.410 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:24:04.439 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:04.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:04.440 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:24:04.459 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:04 smithi031 ceph-mon[16336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:24:04.459 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:04 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:24:04.459 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:04 smithi031 ceph-mon[16336]: osdmap e403: 12 total, 12 up, 9 in 2024-08-19T14:24:04.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:04 smithi032 ceph-mon[20004]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:24:04.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:04 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:24:04.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:04 smithi032 ceph-mon[20004]: osdmap e403: 12 total, 12 up, 9 in 2024-08-19T14:24:04.541 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:24:04.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:04 smithi045 ceph-mon[20143]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:24:04.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:04 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:24:04.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:04 smithi045 ceph-mon[20143]: osdmap e403: 12 total, 12 up, 9 in 2024-08-19T14:24:04.562 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:04.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:04.563 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:24:04.664 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:24:04.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:04.710 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:04.710 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:24:04.799 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-08-19T14:24:04.811 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:24:04.879 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:24:04.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:04.904 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:04.904 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:24:04.907 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:24:05.005 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:24:05.027 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:05.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:05.028 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:24:05.129 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:24:05.159 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:05.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:05.160 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:24:05.261 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:24:05.283 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:05.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:05.284 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:24:05.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:05 smithi031 ceph-mon[16336]: pgmap v928: 47 pgs: 47 active+clean; 1.6 GiB data, 9.0 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 1.08k op/s 2024-08-19T14:24:05.385 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:24:05.409 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:05.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:05.410 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:24:05.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:05 smithi032 ceph-mon[20004]: pgmap v928: 47 pgs: 47 active+clean; 1.6 GiB data, 9.0 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 1.08k op/s 2024-08-19T14:24:05.512 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:24:05.537 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:05.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:05.538 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:24:05.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:05 smithi045 ceph-mon[20143]: pgmap v928: 47 pgs: 47 active+clean; 1.6 GiB data, 9.0 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 1.08k op/s 2024-08-19T14:24:05.597 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:24:05.639 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:24:05.640 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:24:05.665 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:05.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:05.666 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:24:05.767 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:24:05.792 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:05.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:05.793 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:24:05.893 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:24:05.894 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-08-19T14:24:05.894 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:24:05.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:05.920 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:05.920 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:24:06.021 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:24:06.055 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:06.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:06.056 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:24:06.157 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:24:06.183 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:06.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:06.184 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:24:06.285 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:24:06.293 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-19T14:24:06.308 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:06.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:06.309 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:24:06.329 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:06 smithi031 ceph-mon[16336]: from='client.25760 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:24:06.410 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:24:06.440 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:06.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:06.441 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:24:06.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:06 smithi032 ceph-mon[20004]: from='client.25760 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:24:06.542 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:24:06.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:06 smithi045 ceph-mon[20143]: from='client.25760 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:24:06.568 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:06.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:06.569 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:24:06.670 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:24:06.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:06.715 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:06.715 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:24:06.816 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:24:06.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:06.851 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:06.851 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:24:06.952 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":928,"stamp":"2024-08-19T14:24:03.945297+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1697841720,"num_objects":25901,"num_object_clones":0,"num_object_copies":77703,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25901,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":236447,"num_write_kb":8395009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4140,"num_bytes_recovered":271319040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":165156,"ondisk_log_size":165156,"up":141,"acting":141,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":142,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":9454744,"kb_used_data":5192520,"kb_used_omap":14,"kb_used_meta":4262129,"kb_avail":834251624,"statfs":{"total":863955320832,"available":854273662976,"internally_reserved":0,"allocated":5317140480,"data_stored":5300528794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14353,"internal_metadata":4364421103},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,3,4,4,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-566165504,"num_objects":-8639,"num_object_clones":0,"num_object_copies":-25917,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8639,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8639,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001763"},"pg_stats":[{"pgid":"4.5","version":"402'14728","reported_seq":14992,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-08-19T14:24:01.010572+0000","last_change":"2024-08-19T14:23:44.384935+0000","last_active":"2024-08-19T14:24:01.010572+0000","last_peered":"2024-08-19T14:24:01.010572+0000","last_clean":"2024-08-19T14:24:01.010572+0000","last_became_active":"2024-08-19T14:20:29.267549+0000","last_became_peered":"2024-08-19T14:20:29.267549+0000","last_unstale":"2024-08-19T14:24:01.010572+0000","last_undegraded":"2024-08-19T14:24:01.010572+0000","last_fullsized":"2024-08-19T14:24:01.010572+0000","mapping_epoch":335,"log_start":"357'4700","ondisk_log_start":"357'4700","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"399'13841","last_scrub_stamp":"2024-08-19T14:23:44.384705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:23:44.384705+0000","objects_scrubbed":2589,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:24:49.673047+0000","scrub_duration":817,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109576192,"num_objects":1672,"num_object_clones":0,"num_object_copies":5016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14728,"num_write_kb":524800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,0],"acting":[11,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"70'139","reported_seq":1031,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-08-19T14:24:03.187600+0000","last_change":"2024-08-19T14:23:22.006193+0000","last_active":"2024-08-19T14:24:03.187600+0000","last_peered":"2024-08-19T14:24:03.187600+0000","last_clean":"2024-08-19T14:24:03.187600+0000","last_became_active":"2024-08-19T14:18:13.807133+0000","last_became_peered":"2024-08-19T14:18:13.807133+0000","last_unstale":"2024-08-19T14:24:03.187600+0000","last_undegraded":"2024-08-19T14:24:03.187600+0000","last_fullsized":"2024-08-19T14:24:03.187600+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:23:22.006116+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:19:58.072083+0000","last_clean_scrub_stamp":"2024-08-19T14:23:22.006116+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:24:47.877236+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,9],"acting":[7,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"403'15059","reported_seq":23530,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-08-19T14:24:03.489999+0000","last_change":"2024-08-19T14:23:51.058049+0000","last_active":"2024-08-19T14:24:03.489999+0000","last_peered":"2024-08-19T14:24:03.489999+0000","last_clean":"2024-08-19T14:24:03.489999+0000","last_became_active":"2024-08-19T14:23:01.741503+0000","last_became_peered":"2024-08-19T14:23:01.741503+0000","last_unstale":"2024-08-19T14:24:03.489999+0000","last_undegraded":"2024-08-19T14:24:03.489999+0000","last_fullsized":"2024-08-19T14:24:03.489999+0000","mapping_epoch":384,"log_start":"359'5000","ondisk_log_start":"359'5000","created":335,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'14336","last_scrub_stamp":"2024-08-19T14:23:51.057894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:23:51.057894+0000","objects_scrubbed":2372,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:24:59.441268+0000","scrub_duration":622,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106102784,"num_objects":1619,"num_object_clones":0,"num_object_copies":4857,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1619,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15059,"num_write_kb":533696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4140,"num_bytes_recovered":271319040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,7],"acting":[9,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.3","version":"70'139","reported_seq":928,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-08-19T14:23:57.149900+0000","last_change":"2024-08-19T14:23:27.488105+0000","last_active":"2024-08-19T14:23:57.149900+0000","last_peered":"2024-08-19T14:23:57.149900+0000","last_clean":"2024-08-19T14:23:57.149900+0000","last_became_active":"2024-08-19T14:18:13.804721+0000","last_became_peered":"2024-08-19T14:18:13.804721+0000","last_unstale":"2024-08-19T14:23:57.149900+0000","last_undegraded":"2024-08-19T14:23:57.149900+0000","last_fullsized":"2024-08-19T14:23:57.149900+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:23:27.487983+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:23:27.487983+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:24:39.468752+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"402'14858","reported_seq":15003,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-08-19T14:24:02.990402+0000","last_change":"2024-08-19T14:20:29.268110+0000","last_active":"2024-08-19T14:24:02.990402+0000","last_peered":"2024-08-19T14:24:02.990402+0000","last_clean":"2024-08-19T14:24:02.990402+0000","last_became_active":"2024-08-19T14:20:29.267811+0000","last_became_peered":"2024-08-19T14:20:29.267811+0000","last_unstale":"2024-08-19T14:24:02.990402+0000","last_undegraded":"2024-08-19T14:24:02.990402+0000","last_fullsized":"2024-08-19T14:24:02.990402+0000","mapping_epoch":335,"log_start":"358'4800","ondisk_log_start":"358'4800","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:24:03.551457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104071168,"num_objects":1588,"num_object_clones":0,"num_object_copies":4764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14858,"num_write_kb":526272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"70'139","reported_seq":878,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-08-19T14:24:03.187307+0000","last_change":"2024-08-19T14:23:24.042754+0000","last_active":"2024-08-19T14:24:03.187307+0000","last_peered":"2024-08-19T14:24:03.187307+0000","last_clean":"2024-08-19T14:24:03.187307+0000","last_became_active":"2024-08-19T14:20:50.742354+0000","last_became_peered":"2024-08-19T14:20:50.742354+0000","last_unstale":"2024-08-19T14:24:03.187307+0000","last_undegraded":"2024-08-19T14:24:03.187307+0000","last_fullsized":"2024-08-19T14:24:03.187307+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:23:24.042679+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:19:59.302931+0000","last_clean_scrub_stamp":"2024-08-19T14:23:24.042679+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:24:27.949990+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"402'14792","reported_seq":14964,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-08-19T14:24:02.927754+0000","last_change":"2024-08-19T14:23:56.303974+0000","last_active":"2024-08-19T14:24:02.927754+0000","last_peered":"2024-08-19T14:24:02.927754+0000","last_clean":"2024-08-19T14:24:02.927754+0000","last_became_active":"2024-08-19T14:20:29.267331+0000","last_became_peered":"2024-08-19T14:20:29.267331+0000","last_unstale":"2024-08-19T14:24:02.927754+0000","last_undegraded":"2024-08-19T14:24:02.927754+0000","last_fullsized":"2024-08-19T14:24:02.927754+0000","mapping_epoch":335,"log_start":"358'4700","ondisk_log_start":"358'4700","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"401'14428","last_scrub_stamp":"2024-08-19T14:23:56.303276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:23:56.303276+0000","objects_scrubbed":1976,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:25:16.405284+0000","scrub_duration":793,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104464384,"num_objects":1594,"num_object_clones":0,"num_object_copies":4782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1594,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14792,"num_write_kb":524352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"70'139","reported_seq":927,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-08-19T14:24:03.180676+0000","last_change":"2024-08-19T14:23:24.392988+0000","last_active":"2024-08-19T14:24:03.180676+0000","last_peered":"2024-08-19T14:24:03.180676+0000","last_clean":"2024-08-19T14:24:03.180676+0000","last_became_active":"2024-08-19T14:18:13.808505+0000","last_became_peered":"2024-08-19T14:18:13.808505+0000","last_unstale":"2024-08-19T14:24:03.180676+0000","last_undegraded":"2024-08-19T14:24:03.180676+0000","last_fullsized":"2024-08-19T14:24:03.180676+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:23:24.392874+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:23:24.392874+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:24:30.094427+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,11],"acting":[9,0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"403'14995","reported_seq":15209,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-08-19T14:24:03.499266+0000","last_change":"2024-08-19T14:23:47.764099+0000","last_active":"2024-08-19T14:24:03.499266+0000","last_peered":"2024-08-19T14:24:03.499266+0000","last_clean":"2024-08-19T14:24:03.499266+0000","last_became_active":"2024-08-19T14:20:29.268443+0000","last_became_peered":"2024-08-19T14:20:29.268443+0000","last_unstale":"2024-08-19T14:24:03.499266+0000","last_undegraded":"2024-08-19T14:24:03.499266+0000","last_fullsized":"2024-08-19T14:24:03.499266+0000","mapping_epoch":335,"log_start":"358'4900","ondisk_log_start":"358'4900","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"399'14070","last_scrub_stamp":"2024-08-19T14:23:47.764022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:23:47.764022+0000","objects_scrubbed":2456,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:25:03.882630+0000","scrub_duration":437,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":99418112,"num_objects":1517,"num_object_clones":0,"num_object_copies":4551,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14995,"num_write_kb":528384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,11],"acting":[9,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.7","version":"70'139","reported_seq":578,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-08-19T14:24:03.185270+0000","last_change":"2024-08-19T14:23:16.203977+0000","last_active":"2024-08-19T14:24:03.185270+0000","last_peered":"2024-08-19T14:24:03.185270+0000","last_clean":"2024-08-19T14:24:03.185270+0000","last_became_active":"2024-08-19T14:20:28.204384+0000","last_became_peered":"2024-08-19T14:20:28.204384+0000","last_unstale":"2024-08-19T14:24:03.185270+0000","last_undegraded":"2024-08-19T14:24:03.185270+0000","last_fullsized":"2024-08-19T14:24:03.185270+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:23:16.203732+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:21:46.057624+0000","last_clean_scrub_stamp":"2024-08-19T14:23:16.203732+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:24:27.551036+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"402'14634","reported_seq":14969,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-08-19T14:24:01.000244+0000","last_change":"2024-08-19T14:23:22.613580+0000","last_active":"2024-08-19T14:24:01.000244+0000","last_peered":"2024-08-19T14:24:01.000244+0000","last_clean":"2024-08-19T14:24:01.000244+0000","last_became_active":"2024-08-19T14:20:29.267379+0000","last_became_peered":"2024-08-19T14:20:29.267379+0000","last_unstale":"2024-08-19T14:24:01.000244+0000","last_undegraded":"2024-08-19T14:24:01.000244+0000","last_fullsized":"2024-08-19T14:24:01.000244+0000","mapping_epoch":335,"log_start":"357'4600","ondisk_log_start":"357'4600","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"394'12559","last_scrub_stamp":"2024-08-19T14:23:22.613324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:23:22.613324+0000","objects_scrubbed":3829,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:24:25.181152+0000","scrub_duration":1081,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112459776,"num_objects":1716,"num_object_clones":0,"num_object_copies":5148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1716,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14634,"num_write_kb":523200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,1],"acting":[11,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.6","version":"70'139","reported_seq":931,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-08-19T14:24:03.180844+0000","last_change":"2024-08-19T14:23:23.375634+0000","last_active":"2024-08-19T14:24:03.180844+0000","last_peered":"2024-08-19T14:24:03.180844+0000","last_clean":"2024-08-19T14:24:03.180844+0000","last_became_active":"2024-08-19T14:18:26.874902+0000","last_became_peered":"2024-08-19T14:18:26.874902+0000","last_unstale":"2024-08-19T14:24:03.180844+0000","last_undegraded":"2024-08-19T14:24:03.180844+0000","last_fullsized":"2024-08-19T14:24:03.180844+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:23:23.375546+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:27.848870+0000","last_clean_scrub_stamp":"2024-08-19T14:23:23.375546+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:24:25.026779+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,4],"acting":[9,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"403'14737","reported_seq":15016,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-08-19T14:24:03.736138+0000","last_change":"2024-08-19T14:23:26.217160+0000","last_active":"2024-08-19T14:24:03.736138+0000","last_peered":"2024-08-19T14:24:03.736138+0000","last_clean":"2024-08-19T14:24:03.736138+0000","last_became_active":"2024-08-19T14:20:29.267360+0000","last_became_peered":"2024-08-19T14:20:29.267360+0000","last_unstale":"2024-08-19T14:24:03.736138+0000","last_undegraded":"2024-08-19T14:24:03.736138+0000","last_fullsized":"2024-08-19T14:24:03.736138+0000","mapping_epoch":335,"log_start":"358'4700","ondisk_log_start":"358'4700","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"394'12651","last_scrub_stamp":"2024-08-19T14:23:26.216619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:23:26.216619+0000","objects_scrubbed":3626,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:24:44.882612+0000","scrub_duration":1260,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98500608,"num_objects":1503,"num_object_clones":0,"num_object_copies":4509,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14737,"num_write_kb":519680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"70'139","reported_seq":750,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-08-19T14:23:57.135361+0000","last_change":"2024-08-19T14:23:45.496873+0000","last_active":"2024-08-19T14:23:57.135361+0000","last_peered":"2024-08-19T14:23:57.135361+0000","last_clean":"2024-08-19T14:23:57.135361+0000","last_became_active":"2024-08-19T14:18:26.881435+0000","last_became_peered":"2024-08-19T14:18:26.881435+0000","last_unstale":"2024-08-19T14:23:57.135361+0000","last_undegraded":"2024-08-19T14:23:57.135361+0000","last_fullsized":"2024-08-19T14:23:57.135361+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:23:45.496751+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:23:45.496751+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:24:46.294715+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,4],"acting":[11,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.1","version":"402'14854","reported_seq":15106,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-08-19T14:24:00.830811+0000","last_change":"2024-08-19T14:23:56.911837+0000","last_active":"2024-08-19T14:24:00.830811+0000","last_peered":"2024-08-19T14:24:00.830811+0000","last_clean":"2024-08-19T14:24:00.830811+0000","last_became_active":"2024-08-19T14:20:29.268465+0000","last_became_peered":"2024-08-19T14:20:29.268465+0000","last_unstale":"2024-08-19T14:24:00.830811+0000","last_undegraded":"2024-08-19T14:24:00.830811+0000","last_fullsized":"2024-08-19T14:24:00.830811+0000","mapping_epoch":335,"log_start":"358'4800","ondisk_log_start":"358'4800","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"401'14629","last_scrub_stamp":"2024-08-19T14:23:56.911497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:23:56.911497+0000","objects_scrubbed":1881,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:25:24.115176+0000","scrub_duration":423,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108003328,"num_objects":1648,"num_object_clones":0,"num_object_copies":4944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1648,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14854,"num_write_kb":528064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.4","version":"70'139","reported_seq":975,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-08-19T14:23:57.135186+0000","last_change":"2024-08-19T14:22:46.380189+0000","last_active":"2024-08-19T14:23:57.135186+0000","last_peered":"2024-08-19T14:23:57.135186+0000","last_clean":"2024-08-19T14:23:57.135186+0000","last_became_active":"2024-08-19T14:18:13.806013+0000","last_became_peered":"2024-08-19T14:18:13.806013+0000","last_unstale":"2024-08-19T14:23:57.135186+0000","last_undegraded":"2024-08-19T14:23:57.135186+0000","last_fullsized":"2024-08-19T14:23:57.135186+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:22:46.380119+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:22:46.380119+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.f","version":"402'14758","reported_seq":14903,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-08-19T14:24:02.979237+0000","last_change":"2024-08-19T14:20:29.212298+0000","last_active":"2024-08-19T14:24:02.979237+0000","last_peered":"2024-08-19T14:24:02.979237+0000","last_clean":"2024-08-19T14:24:02.979237+0000","last_became_active":"2024-08-19T14:20:29.212003+0000","last_became_peered":"2024-08-19T14:20:29.212003+0000","last_unstale":"2024-08-19T14:24:02.979237+0000","last_undegraded":"2024-08-19T14:24:02.979237+0000","last_fullsized":"2024-08-19T14:24:02.979237+0000","mapping_epoch":335,"log_start":"358'4700","ondisk_log_start":"358'4700","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:24:03.551493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":101187584,"num_objects":1544,"num_object_clones":0,"num_object_copies":4632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1544,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14758,"num_write_kb":521664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,11],"acting":[1,3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"70'139","reported_seq":939,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-08-19T14:24:03.180792+0000","last_change":"2024-08-19T14:23:25.404078+0000","last_active":"2024-08-19T14:24:03.180792+0000","last_peered":"2024-08-19T14:24:03.180792+0000","last_clean":"2024-08-19T14:24:03.180792+0000","last_became_active":"2024-08-19T14:18:26.875006+0000","last_became_peered":"2024-08-19T14:18:26.875006+0000","last_unstale":"2024-08-19T14:24:03.180792+0000","last_undegraded":"2024-08-19T14:24:03.180792+0000","last_fullsized":"2024-08-19T14:24:03.180792+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:23:25.403959+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:28.864870+0000","last_clean_scrub_stamp":"2024-08-19T14:23:25.403959+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:24:36.949528+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,0],"acting":[9,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.c","version":"402'14709","reported_seq":14991,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-08-19T14:24:02.714225+0000","last_change":"2024-08-19T14:23:19.956127+0000","last_active":"2024-08-19T14:24:02.714225+0000","last_peered":"2024-08-19T14:24:02.714225+0000","last_clean":"2024-08-19T14:24:02.714225+0000","last_became_active":"2024-08-19T14:20:29.266373+0000","last_became_peered":"2024-08-19T14:20:29.266373+0000","last_unstale":"2024-08-19T14:24:02.714225+0000","last_undegraded":"2024-08-19T14:24:02.714225+0000","last_fullsized":"2024-08-19T14:24:02.714225+0000","mapping_epoch":335,"log_start":"358'4700","ondisk_log_start":"358'4700","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"393'12368","last_scrub_stamp":"2024-08-19T14:23:19.955924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:23:19.955924+0000","objects_scrubbed":4015,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:24:21.233858+0000","scrub_duration":1085,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108331008,"num_objects":1653,"num_object_clones":0,"num_object_copies":4959,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14709,"num_write_kb":523584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,8],"acting":[4,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"70'139","reported_seq":1054,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-08-19T14:24:03.187151+0000","last_change":"2024-08-19T14:23:44.965612+0000","last_active":"2024-08-19T14:24:03.187151+0000","last_peered":"2024-08-19T14:24:03.187151+0000","last_clean":"2024-08-19T14:24:03.187151+0000","last_became_active":"2024-08-19T14:18:26.887657+0000","last_became_peered":"2024-08-19T14:18:26.887657+0000","last_unstale":"2024-08-19T14:24:03.187151+0000","last_undegraded":"2024-08-19T14:24:03.187151+0000","last_fullsized":"2024-08-19T14:24:03.187151+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:23:44.965485+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:23:44.965485+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:24:53.391995+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"402'14639","reported_seq":14906,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-08-19T14:24:02.716417+0000","last_change":"2024-08-19T14:23:47.756240+0000","last_active":"2024-08-19T14:24:02.716417+0000","last_peered":"2024-08-19T14:24:02.716417+0000","last_clean":"2024-08-19T14:24:02.716417+0000","last_became_active":"2024-08-19T14:20:29.208599+0000","last_became_peered":"2024-08-19T14:20:29.208599+0000","last_unstale":"2024-08-19T14:24:02.716417+0000","last_undegraded":"2024-08-19T14:24:02.716417+0000","last_fullsized":"2024-08-19T14:24:02.716417+0000","mapping_epoch":335,"log_start":"357'4600","ondisk_log_start":"357'4600","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"399'13776","last_scrub_stamp":"2024-08-19T14:23:47.756057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:23:47.756057+0000","objects_scrubbed":2492,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:24:57.431228+0000","scrub_duration":670,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":105512984,"num_objects":1611,"num_object_clones":0,"num_object_copies":4833,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1611,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14639,"num_write_kb":519937,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,1],"acting":[4,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"70'139","reported_seq":949,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-08-19T14:23:57.137216+0000","last_change":"2024-08-19T14:23:19.245681+0000","last_active":"2024-08-19T14:23:57.137216+0000","last_peered":"2024-08-19T14:23:57.137216+0000","last_clean":"2024-08-19T14:23:57.137216+0000","last_became_active":"2024-08-19T14:18:26.887277+0000","last_became_peered":"2024-08-19T14:18:26.887277+0000","last_unstale":"2024-08-19T14:23:57.137216+0000","last_undegraded":"2024-08-19T14:23:57.137216+0000","last_fullsized":"2024-08-19T14:23:57.137216+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:23:19.245574+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:23:19.245574+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:24:39.015396+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,8],"acting":[5,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"402'14579","reported_seq":14837,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-08-19T14:24:02.728578+0000","last_change":"2024-08-19T14:23:48.815007+0000","last_active":"2024-08-19T14:24:02.728578+0000","last_peered":"2024-08-19T14:24:02.728578+0000","last_clean":"2024-08-19T14:24:02.728578+0000","last_became_active":"2024-08-19T14:20:29.208748+0000","last_became_peered":"2024-08-19T14:20:29.208748+0000","last_unstale":"2024-08-19T14:24:02.728578+0000","last_undegraded":"2024-08-19T14:24:02.728578+0000","last_fullsized":"2024-08-19T14:24:02.728578+0000","mapping_epoch":335,"log_start":"357'4500","ondisk_log_start":"357'4500","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'13814","last_scrub_stamp":"2024-08-19T14:23:48.814645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:23:48.814645+0000","objects_scrubbed":2349,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:24:56.662744+0000","scrub_duration":693,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102170624,"num_objects":1559,"num_object_clones":0,"num_object_copies":4677,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1559,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14579,"num_write_kb":516416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"70'139","reported_seq":950,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-08-19T14:23:57.135116+0000","last_change":"2024-08-19T14:23:23.564599+0000","last_active":"2024-08-19T14:23:57.135116+0000","last_peered":"2024-08-19T14:23:57.135116+0000","last_clean":"2024-08-19T14:23:57.135116+0000","last_became_active":"2024-08-19T14:18:26.875179+0000","last_became_peered":"2024-08-19T14:18:26.875179+0000","last_unstale":"2024-08-19T14:23:57.135116+0000","last_undegraded":"2024-08-19T14:23:57.135116+0000","last_fullsized":"2024-08-19T14:23:57.135116+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:23:23.564518+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:23:23.564518+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:24:28.111207+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.b","version":"402'14678","reported_seq":14946,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-08-19T14:24:01.270618+0000","last_change":"2024-08-19T14:23:23.481298+0000","last_active":"2024-08-19T14:24:01.270618+0000","last_peered":"2024-08-19T14:24:01.270618+0000","last_clean":"2024-08-19T14:24:01.270618+0000","last_became_active":"2024-08-19T14:20:29.207372+0000","last_became_peered":"2024-08-19T14:20:29.207372+0000","last_unstale":"2024-08-19T14:24:01.270618+0000","last_undegraded":"2024-08-19T14:24:01.270618+0000","last_fullsized":"2024-08-19T14:24:01.270618+0000","mapping_epoch":335,"log_start":"357'4600","ondisk_log_start":"357'4600","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"394'12554","last_scrub_stamp":"2024-08-19T14:23:23.481035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:23:23.481035+0000","objects_scrubbed":3836,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:24:52.522452+0000","scrub_duration":1875,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":1664,"num_object_clones":0,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1664,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14678,"num_write_kb":522944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"70'139","reported_seq":604,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-08-19T14:23:57.136842+0000","last_change":"2024-08-19T14:23:16.834552+0000","last_active":"2024-08-19T14:23:57.136842+0000","last_peered":"2024-08-19T14:23:57.136842+0000","last_clean":"2024-08-19T14:23:57.136842+0000","last_became_active":"2024-08-19T14:18:26.884105+0000","last_became_peered":"2024-08-19T14:18:26.884105+0000","last_unstale":"2024-08-19T14:23:57.136842+0000","last_undegraded":"2024-08-19T14:23:57.136842+0000","last_fullsized":"2024-08-19T14:23:57.136842+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:23:16.834478+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:20:15.321945+0000","last_clean_scrub_stamp":"2024-08-19T14:23:16.834478+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:24:31.654430+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"402'15017","reported_seq":15190,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-08-19T14:24:02.983748+0000","last_change":"2024-08-19T14:23:54.872969+0000","last_active":"2024-08-19T14:24:02.983748+0000","last_peered":"2024-08-19T14:24:02.983748+0000","last_clean":"2024-08-19T14:24:02.983748+0000","last_became_active":"2024-08-19T14:20:29.269432+0000","last_became_peered":"2024-08-19T14:20:29.269432+0000","last_unstale":"2024-08-19T14:24:02.983748+0000","last_undegraded":"2024-08-19T14:24:02.983748+0000","last_fullsized":"2024-08-19T14:24:02.983748+0000","mapping_epoch":335,"log_start":"359'5000","ondisk_log_start":"359'5000","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"401'14514","last_scrub_stamp":"2024-08-19T14:23:54.872659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:23:54.872659+0000","objects_scrubbed":2127,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:25:22.243401+0000","scrub_duration":1279,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104792064,"num_objects":1599,"num_object_clones":0,"num_object_copies":4797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15017,"num_write_kb":531712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.f","version":"70'139","reported_seq":848,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-08-19T14:23:57.137511+0000","last_change":"2024-08-19T14:22:42.293240+0000","last_active":"2024-08-19T14:23:57.137511+0000","last_peered":"2024-08-19T14:23:57.137511+0000","last_clean":"2024-08-19T14:23:57.137511+0000","last_became_active":"2024-08-19T14:18:18.711708+0000","last_became_peered":"2024-08-19T14:18:18.711708+0000","last_unstale":"2024-08-19T14:23:57.137511+0000","last_undegraded":"2024-08-19T14:23:57.137511+0000","last_fullsized":"2024-08-19T14:23:57.137511+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:22:42.293171+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:22:42.293171+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:24:10.029244+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"402'14744","reported_seq":15017,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-08-19T14:24:00.788204+0000","last_change":"2024-08-19T14:23:18.636632+0000","last_active":"2024-08-19T14:24:00.788204+0000","last_peered":"2024-08-19T14:24:00.788204+0000","last_clean":"2024-08-19T14:24:00.788204+0000","last_became_active":"2024-08-19T14:20:29.211975+0000","last_became_peered":"2024-08-19T14:20:29.211975+0000","last_unstale":"2024-08-19T14:24:00.788204+0000","last_undegraded":"2024-08-19T14:24:00.788204+0000","last_fullsized":"2024-08-19T14:24:00.788204+0000","mapping_epoch":335,"log_start":"358'4700","ondisk_log_start":"358'4700","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"392'12385","last_scrub_stamp":"2024-08-19T14:23:18.636481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:23:18.636481+0000","objects_scrubbed":4109,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:24:36.643654+0000","scrub_duration":1122,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112328704,"num_objects":1714,"num_object_clones":0,"num_object_copies":5142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1714,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14744,"num_write_kb":526656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,3],"acting":[11,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.c","version":"70'139","reported_seq":711,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-08-19T14:24:03.189940+0000","last_change":"2024-08-19T14:22:52.164697+0000","last_active":"2024-08-19T14:24:03.189940+0000","last_peered":"2024-08-19T14:24:03.189940+0000","last_clean":"2024-08-19T14:24:03.189940+0000","last_became_active":"2024-08-19T14:18:26.872942+0000","last_became_peered":"2024-08-19T14:18:26.872942+0000","last_unstale":"2024-08-19T14:24:03.189940+0000","last_undegraded":"2024-08-19T14:24:03.189940+0000","last_fullsized":"2024-08-19T14:24:03.189940+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:22:52.164586+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:22:52.164586+0000","last_clean_scrub_stamp":"2024-08-19T14:22:52.164586+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:24:05.322410+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,9],"acting":[3,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"402'14666","reported_seq":14926,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-08-19T14:24:01.010593+0000","last_change":"2024-08-19T14:23:54.734521+0000","last_active":"2024-08-19T14:24:01.010593+0000","last_peered":"2024-08-19T14:24:01.010593+0000","last_clean":"2024-08-19T14:24:01.010593+0000","last_became_active":"2024-08-19T14:20:29.267442+0000","last_became_peered":"2024-08-19T14:20:29.267442+0000","last_unstale":"2024-08-19T14:24:01.010593+0000","last_undegraded":"2024-08-19T14:24:01.010593+0000","last_fullsized":"2024-08-19T14:24:01.010593+0000","mapping_epoch":335,"log_start":"357'4600","ondisk_log_start":"357'4600","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"401'14297","last_scrub_stamp":"2024-08-19T14:23:54.734387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:23:54.734387+0000","objects_scrubbed":2072,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:25:14.711064+0000","scrub_duration":300,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":111280128,"num_objects":1698,"num_object_clones":0,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1698,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14666,"num_write_kb":523648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,9],"acting":[11,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.d","version":"70'139","reported_seq":626,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-08-19T14:24:03.187428+0000","last_change":"2024-08-19T14:23:48.931295+0000","last_active":"2024-08-19T14:24:03.187428+0000","last_peered":"2024-08-19T14:24:03.187428+0000","last_clean":"2024-08-19T14:24:03.187428+0000","last_became_active":"2024-08-19T14:18:26.882002+0000","last_became_peered":"2024-08-19T14:18:26.882002+0000","last_unstale":"2024-08-19T14:24:03.187428+0000","last_undegraded":"2024-08-19T14:24:03.187428+0000","last_fullsized":"2024-08-19T14:24:03.187428+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:23:48.931225+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:23:48.931225+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:24:50.112289+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"70'139","reported_seq":883,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-08-19T14:23:57.137346+0000","last_change":"2024-08-19T14:22:47.256611+0000","last_active":"2024-08-19T14:23:57.137346+0000","last_peered":"2024-08-19T14:23:57.137346+0000","last_clean":"2024-08-19T14:23:57.137346+0000","last_became_active":"2024-08-19T14:18:18.703594+0000","last_became_peered":"2024-08-19T14:18:18.703594+0000","last_unstale":"2024-08-19T14:23:57.137346+0000","last_undegraded":"2024-08-19T14:23:57.137346+0000","last_fullsized":"2024-08-19T14:23:57.137346+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:22:47.256536+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:22:47.256536+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:24:02.876164+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"70'139","reported_seq":935,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-08-19T14:23:57.149648+0000","last_change":"2024-08-19T14:23:49.551005+0000","last_active":"2024-08-19T14:23:57.149648+0000","last_peered":"2024-08-19T14:23:57.149648+0000","last_clean":"2024-08-19T14:23:57.149648+0000","last_became_active":"2024-08-19T14:18:18.698334+0000","last_became_peered":"2024-08-19T14:18:18.698334+0000","last_unstale":"2024-08-19T14:23:57.149648+0000","last_undegraded":"2024-08-19T14:23:57.149648+0000","last_fullsized":"2024-08-19T14:23:57.149648+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:23:49.550697+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:23:49.550697+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:25:03.915358+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.10","version":"70'139","reported_seq":1046,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-08-19T14:23:57.137154+0000","last_change":"2024-08-19T14:23:46.008812+0000","last_active":"2024-08-19T14:23:57.137154+0000","last_peered":"2024-08-19T14:23:57.137154+0000","last_clean":"2024-08-19T14:23:57.137154+0000","last_became_active":"2024-08-19T14:18:18.704562+0000","last_became_peered":"2024-08-19T14:18:18.704562+0000","last_unstale":"2024-08-19T14:23:57.137154+0000","last_undegraded":"2024-08-19T14:23:57.137154+0000","last_fullsized":"2024-08-19T14:23:57.137154+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:23:46.008709+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:23:46.008709+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:24:50.614402+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,8],"acting":[5,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"70'139","reported_seq":595,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-08-19T14:23:57.146036+0000","last_change":"2024-08-19T14:23:48.687131+0000","last_active":"2024-08-19T14:23:57.146036+0000","last_peered":"2024-08-19T14:23:57.146036+0000","last_clean":"2024-08-19T14:23:57.146036+0000","last_became_active":"2024-08-19T14:18:18.697894+0000","last_became_peered":"2024-08-19T14:18:18.697894+0000","last_unstale":"2024-08-19T14:23:57.146036+0000","last_undegraded":"2024-08-19T14:23:57.146036+0000","last_fullsized":"2024-08-19T14:23:57.146036+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:23:48.687011+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:23:48.687011+0000","last_clean_scrub_stamp":"2024-08-19T14:23:48.687011+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:24:49.333312+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"70'139","reported_seq":564,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-08-19T14:23:57.150308+0000","last_change":"2024-08-19T14:23:21.381100+0000","last_active":"2024-08-19T14:23:57.150308+0000","last_peered":"2024-08-19T14:23:57.150308+0000","last_clean":"2024-08-19T14:23:57.150308+0000","last_became_active":"2024-08-19T14:20:28.204724+0000","last_became_peered":"2024-08-19T14:20:28.204724+0000","last_unstale":"2024-08-19T14:23:57.150308+0000","last_undegraded":"2024-08-19T14:23:57.150308+0000","last_fullsized":"2024-08-19T14:23:57.150308+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:23:21.380985+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:23:21.380985+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:24:35.720673+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"70'139","reported_seq":692,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-08-19T14:23:57.147109+0000","last_change":"2024-08-19T14:23:20.731091+0000","last_active":"2024-08-19T14:23:57.147109+0000","last_peered":"2024-08-19T14:23:57.147109+0000","last_clean":"2024-08-19T14:23:57.147109+0000","last_became_active":"2024-08-19T14:18:31.901556+0000","last_became_peered":"2024-08-19T14:18:31.901556+0000","last_unstale":"2024-08-19T14:23:57.147109+0000","last_undegraded":"2024-08-19T14:23:57.147109+0000","last_fullsized":"2024-08-19T14:23:57.147109+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:23:20.731018+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:23:20.731018+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:24:43.756077+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,0],"acting":[8,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.14","version":"70'139","reported_seq":695,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-08-19T14:24:03.190421+0000","last_change":"2024-08-19T14:23:15.864067+0000","last_active":"2024-08-19T14:24:03.190421+0000","last_peered":"2024-08-19T14:24:03.190421+0000","last_clean":"2024-08-19T14:24:03.190421+0000","last_became_active":"2024-08-19T14:18:18.703926+0000","last_became_peered":"2024-08-19T14:18:18.703926+0000","last_unstale":"2024-08-19T14:24:03.190421+0000","last_undegraded":"2024-08-19T14:24:03.190421+0000","last_fullsized":"2024-08-19T14:24:03.190421+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:23:15.863945+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:23:15.863945+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:24:21.930991+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,1],"acting":[7,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.15","version":"70'139","reported_seq":582,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-08-19T14:24:03.187367+0000","last_change":"2024-08-19T14:23:25.015912+0000","last_active":"2024-08-19T14:24:03.187367+0000","last_peered":"2024-08-19T14:24:03.187367+0000","last_clean":"2024-08-19T14:24:03.187367+0000","last_became_active":"2024-08-19T14:18:31.912496+0000","last_became_peered":"2024-08-19T14:18:31.912496+0000","last_unstale":"2024-08-19T14:24:03.187367+0000","last_undegraded":"2024-08-19T14:24:03.187367+0000","last_fullsized":"2024-08-19T14:24:03.187367+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:23:25.015801+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:20:09.180923+0000","last_clean_scrub_stamp":"2024-08-19T14:23:25.015801+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:24:33.214524+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"70'139","reported_seq":566,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-08-19T14:23:57.136731+0000","last_change":"2024-08-19T14:23:15.839549+0000","last_active":"2024-08-19T14:23:57.136731+0000","last_peered":"2024-08-19T14:23:57.136731+0000","last_clean":"2024-08-19T14:23:57.136731+0000","last_became_active":"2024-08-19T14:18:31.901229+0000","last_became_peered":"2024-08-19T14:18:31.901229+0000","last_unstale":"2024-08-19T14:23:57.136731+0000","last_undegraded":"2024-08-19T14:23:57.136731+0000","last_fullsized":"2024-08-19T14:23:57.136731+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:23:15.839423+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:23:15.839423+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:24:26.744739+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1b","version":"70'139","reported_seq":918,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-08-19T14:23:57.147033+0000","last_change":"2024-08-19T14:23:18.756861+0000","last_active":"2024-08-19T14:23:57.147033+0000","last_peered":"2024-08-19T14:23:57.147033+0000","last_clean":"2024-08-19T14:23:57.147033+0000","last_became_active":"2024-08-19T14:18:31.901152+0000","last_became_peered":"2024-08-19T14:18:31.901152+0000","last_unstale":"2024-08-19T14:23:57.147033+0000","last_undegraded":"2024-08-19T14:23:57.147033+0000","last_fullsized":"2024-08-19T14:23:57.147033+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:23:18.756745+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:23:18.756745+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:24:23.113444+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.18","version":"70'139","reported_seq":962,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-08-19T14:23:57.137109+0000","last_change":"2024-08-19T14:23:20.217662+0000","last_active":"2024-08-19T14:23:57.137109+0000","last_peered":"2024-08-19T14:23:57.137109+0000","last_clean":"2024-08-19T14:23:57.137109+0000","last_became_active":"2024-08-19T14:18:31.902735+0000","last_became_peered":"2024-08-19T14:18:31.902735+0000","last_unstale":"2024-08-19T14:23:57.137109+0000","last_undegraded":"2024-08-19T14:23:57.137109+0000","last_fullsized":"2024-08-19T14:23:57.137109+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:23:20.217551+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:23:20.217551+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:24:31.850512+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.19","version":"70'139","reported_seq":910,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-08-19T14:23:57.146392+0000","last_change":"2024-08-19T14:23:18.563553+0000","last_active":"2024-08-19T14:23:57.146392+0000","last_peered":"2024-08-19T14:23:57.146392+0000","last_clean":"2024-08-19T14:23:57.146392+0000","last_became_active":"2024-08-19T14:18:31.902547+0000","last_became_peered":"2024-08-19T14:18:31.902547+0000","last_unstale":"2024-08-19T14:23:57.146392+0000","last_undegraded":"2024-08-19T14:23:57.146392+0000","last_fullsized":"2024-08-19T14:23:57.146392+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:23:18.563420+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:23:18.563420+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:24:42.281542+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1e","version":"70'139","reported_seq":568,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-08-19T14:23:57.135303+0000","last_change":"2024-08-19T14:23:18.518254+0000","last_active":"2024-08-19T14:23:57.135303+0000","last_peered":"2024-08-19T14:23:57.135303+0000","last_clean":"2024-08-19T14:23:57.135303+0000","last_became_active":"2024-08-19T14:18:31.902546+0000","last_became_peered":"2024-08-19T14:18:31.902546+0000","last_unstale":"2024-08-19T14:23:57.135303+0000","last_undegraded":"2024-08-19T14:23:57.135303+0000","last_fullsized":"2024-08-19T14:23:57.135303+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:23:18.518183+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:23:18.518183+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:24:26.587350+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1c","version":"70'139","reported_seq":676,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-08-19T14:23:57.150873+0000","last_change":"2024-08-19T14:22:49.619773+0000","last_active":"2024-08-19T14:23:57.150873+0000","last_peered":"2024-08-19T14:23:57.150873+0000","last_clean":"2024-08-19T14:23:57.150873+0000","last_became_active":"2024-08-19T14:18:31.902494+0000","last_became_peered":"2024-08-19T14:18:31.902494+0000","last_unstale":"2024-08-19T14:23:57.150873+0000","last_undegraded":"2024-08-19T14:23:57.150873+0000","last_fullsized":"2024-08-19T14:23:57.150873+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:22:49.619657+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:22:49.619657+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:23:58.403527+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,11],"acting":[8,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1d","version":"70'139","reported_seq":590,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-08-19T14:23:57.156131+0000","last_change":"2024-08-19T14:23:20.375564+0000","last_active":"2024-08-19T14:23:57.156131+0000","last_peered":"2024-08-19T14:23:57.156131+0000","last_clean":"2024-08-19T14:23:57.156131+0000","last_became_active":"2024-08-19T14:18:31.901183+0000","last_became_peered":"2024-08-19T14:18:31.901183+0000","last_unstale":"2024-08-19T14:23:57.156131+0000","last_undegraded":"2024-08-19T14:23:57.156131+0000","last_fullsized":"2024-08-19T14:23:57.156131+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:23:20.375451+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:23:20.375451+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:24:21.979354+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,5],"acting":[1,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":1697251352,"num_objects":25899,"num_object_clones":0,"num_object_copies":77697,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25899,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":236447,"num_write_kb":8395009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4140,"num_bytes_recovered":271319040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5185810432,"data_stored":5185798216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160847,"ondisk_log_size":160847,"up":48,"acting":48,"num_store_stats":9},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1802240,"data_stored":1788032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187608,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1211252,"kb_used_data":769132,"kb_used_omap":1,"kb_used_meta":442110,"kb_avail":92533900,"statfs":{"total":95995035648,"available":94754713600,"internally_reserved":0,"allocated":787591168,"data_stored":785748912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":452721092},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":343597383838,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":74,"seq":317827580066,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":928712,"kb_used_data":406076,"kb_used_omap":1,"kb_used_meta":522622,"kb_avail":92816440,"statfs":{"total":95995035648,"available":95044034560,"internally_reserved":0,"allocated":415821824,"data_stored":413971241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":535165377},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874407,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":505460,"kb_used_data":123052,"kb_used_omap":1,"kb_used_meta":382398,"kb_avail":93239692,"statfs":{"total":95995035648,"available":95477444608,"internally_reserved":0,"allocated":126005248,"data_stored":124162992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":391576005},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":57,"seq":244813136043,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1382140,"kb_used_data":813684,"kb_used_omap":1,"kb_used_meta":568446,"kb_avail":92363012,"statfs":{"total":95995035648,"available":94579724288,"internally_reserved":0,"allocated":833212416,"data_stored":831366608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":582089162},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":206158430381,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692018,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":862012,"kb_used_data":479604,"kb_used_omap":1,"kb_used_meta":382398,"kb_avail":92883140,"statfs":{"total":95995035648,"available":95112335360,"internally_reserved":0,"allocated":491114496,"data_stored":489268688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":391576005},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986359,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1030072,"kb_used_data":521328,"kb_used_omap":1,"kb_used_meta":508734,"kb_avail":92715080,"statfs":{"total":95995035648,"available":94940241920,"internally_reserved":0,"allocated":533839872,"data_stored":531993544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":520944069},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149885,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1066884,"kb_used_data":602552,"kb_used_omap":1,"kb_used_meta":464318,"kb_avail":92678268,"statfs":{"total":95995035648,"available":94902546432,"internally_reserved":0,"allocated":617013248,"data_stored":615166737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":475462087},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411522,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575046,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1628864,"kb_used_data":1038200,"kb_used_omap":1,"kb_used_meta":590654,"kb_avail":92116288,"statfs":{"total":95995035648,"available":94327078912,"internally_reserved":0,"allocated":1063116800,"data_stored":1061266920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":604830149},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,2,4,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738570,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":839348,"kb_used_data":438892,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":92905804,"statfs":{"total":95995035648,"available":95135543296,"internally_reserved":0,"allocated":449425408,"data_stored":447583152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":410057157},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":434241536,"data_stored":434241536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1038356480,"data_stored":1038352408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":601096192,"data_stored":601096192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":520884224,"data_stored":520880152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":478019584,"data_stored":478019584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":813236224,"data_stored":813236224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":117899264,"data_stored":117899264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":408686592,"data_stored":408682520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":773390336,"data_stored":773390336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T14:24:06.955 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd pool set .mgr pgp_num 41 2024-08-19T14:24:06.955 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:24:06.977 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:06.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:06.978 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:24:07.079 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:24:07.100 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:07.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:07.101 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:24:07.123 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:24:07.123 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-08-19T14:24:07.123 INFO:teuthology.orchestra.run.smithi031.stdout: "complaint_time": 30, 2024-08-19T14:24:07.123 INFO:teuthology.orchestra.run.smithi031.stdout: "num_blocked_ops": 0 2024-08-19T14:24:07.123 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:24:07.202 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:24:07.231 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:07.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:07.232 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:24:07.333 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:24:07.358 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:07.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:07.360 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:24:07.461 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:24:07.462 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:07 smithi031 ceph-mon[16336]: pgmap v929: 47 pgs: 47 active+clean; 1.6 GiB data, 9.0 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 810 op/s 2024-08-19T14:24:07.462 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:07 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:24:07.462 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:07 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:24:07.485 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:07.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:07.486 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:24:07.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:07 smithi032 ceph-mon[20004]: pgmap v929: 47 pgs: 47 active+clean; 1.6 GiB data, 9.0 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 810 op/s 2024-08-19T14:24:07.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:07 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:24:07.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:07 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:24:07.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:07 smithi045 ceph-mon[20143]: pgmap v929: 47 pgs: 47 active+clean; 1.6 GiB data, 9.0 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 810 op/s 2024-08-19T14:24:07.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:07 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:24:07.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:07 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:24:07.587 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:24:07.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:07.617 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:07.617 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:24:07.719 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:24:07.765 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:07.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:07.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:24:07.814 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-08-19T14:24:07.867 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:24:07.894 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:07.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:07.895 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:24:07.996 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:24:08.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:08.037 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:08.037 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:24:08.108 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:24:08.138 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:24:08.146 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:24:08.146 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:24:08.162 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:08.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:08.163 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:24:08.186 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:24:08.187 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:24:08.218 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:24:08.218 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:24:08.250 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:24:08.250 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:24:08.263 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:24:08.282 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:24:08.282 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:24:08.288 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:08.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:08.289 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:24:08.318 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:08 smithi031 ceph-mon[16336]: osdmap e404: 12 total, 12 up, 9 in 2024-08-19T14:24:08.318 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:08 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3333243117' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:24:08.318 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:08 smithi031 ceph-mon[16336]: from='client.25717 172.21.15.31:0/3333243117' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:24:08.319 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:24:08.319 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:24:08.390 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:24:08.412 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:08.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:08.414 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:24:08.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:08 smithi032 ceph-mon[20004]: osdmap e404: 12 total, 12 up, 9 in 2024-08-19T14:24:08.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:08 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3333243117' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:24:08.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:08 smithi032 ceph-mon[20004]: from='client.25717 172.21.15.31:0/3333243117' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:24:08.515 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:24:08.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:08.537 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:08.538 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:24:08.538 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:24:08.538 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:24:08.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:08 smithi045 ceph-mon[20143]: osdmap e404: 12 total, 12 up, 9 in 2024-08-19T14:24:08.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:08 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3333243117' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:24:08.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:08 smithi045 ceph-mon[20143]: from='client.25717 172.21.15.31:0/3333243117' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:24:08.555 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:24:08.555 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:24:08.572 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:24:08.572 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:24:08.591 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:24:08.591 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:24:08.622 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:24:08.622 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:24:08.638 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:24:08.641 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:24:08.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:08.663 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:08.663 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:24:08.664 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:24:08.664 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:24:08.764 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:24:08.788 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:08.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:08.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:24:08.890 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:24:08.916 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:08.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:08.916 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:24:09.018 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:24:09.049 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:09.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:09.050 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:24:09.150 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:24:09.176 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:09.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:09.177 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:24:09.234 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is unset 2024-08-19T14:24:09.278 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:24:09.301 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:09.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:09.302 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:24:09.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:09 smithi031 ceph-mon[16336]: pgmap v931: 47 pgs: 47 active+clean; 1.4 GiB data, 8.6 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 877 op/s 2024-08-19T14:24:09.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:09 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:24:09.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:09 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:24:09.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:09 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1777501458' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:24:09.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:09 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:24:09.403 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:24:09.438 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:09.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:09.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:24:09.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:09 smithi032 ceph-mon[20004]: pgmap v931: 47 pgs: 47 active+clean; 1.4 GiB data, 8.6 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 877 op/s 2024-08-19T14:24:09.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:09 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:24:09.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:09 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:24:09.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:09 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1777501458' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:24:09.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:09 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:24:09.540 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:24:09.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:09 smithi045 ceph-mon[20143]: pgmap v931: 47 pgs: 47 active+clean; 1.4 GiB data, 8.6 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 877 op/s 2024-08-19T14:24:09.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:09 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:24:09.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:09 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:24:09.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:09 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1777501458' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:24:09.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:09 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:24:09.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:09.570 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:09.570 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:24:09.671 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:24:09.695 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:09.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:09.696 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:24:09.796 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:24:09.822 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:09.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:09.823 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:24:09.924 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:24:09.962 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:09.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:09.963 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:24:10.064 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:24:10.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:10.100 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:10.101 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:24:10.202 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:24:10.237 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:10.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:10.238 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:24:10.323 INFO:teuthology.orchestra.run.smithi031.stderr:Error EINVAL: specified pgp_num 41 > pg_num 31 2024-08-19T14:24:10.339 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:24:10.362 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:10.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:10.363 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:24:10.464 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:24:10.500 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:10.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:10.501 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:24:10.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:10 smithi032 ceph-mon[20004]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:24:10.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:10 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:24:10.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:10 smithi032 ceph-mon[20004]: osdmap e405: 12 total, 12 up, 9 in 2024-08-19T14:24:10.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:10 smithi045 ceph-mon[20143]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:24:10.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:10 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:24:10.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:10 smithi045 ceph-mon[20143]: osdmap e405: 12 total, 12 up, 9 in 2024-08-19T14:24:10.602 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:24:10.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:10 smithi031 ceph-mon[16336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:24:10.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:10 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:24:10.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:10 smithi031 ceph-mon[16336]: osdmap e405: 12 total, 12 up, 9 in 2024-08-19T14:24:10.632 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:10.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:10.633 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:24:10.720 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:24:10.734 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:24:10.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:10.761 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:10.761 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:24:10.862 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:24:10.895 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:10.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:10.896 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:24:10.997 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:24:11.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:11.054 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:11.054 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:24:11.155 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:24:11.185 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:11.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:11.186 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:24:11.287 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:24:11.313 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:11.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:11.314 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:24:11.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:11 smithi031 ceph-mon[16336]: pgmap v933: 47 pgs: 47 active+clean; 1.3 GiB data, 8.2 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 795 op/s 2024-08-19T14:24:11.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:11 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3885192892' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "41"}]: dispatch 2024-08-19T14:24:11.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:11 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "41"}]: dispatch 2024-08-19T14:24:11.415 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:24:11.449 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:11.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:11.450 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:24:11.487 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-19T14:24:11.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:11 smithi032 ceph-mon[20004]: pgmap v933: 47 pgs: 47 active+clean; 1.3 GiB data, 8.2 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 795 op/s 2024-08-19T14:24:11.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:11 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3885192892' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "41"}]: dispatch 2024-08-19T14:24:11.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:11 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "41"}]: dispatch 2024-08-19T14:24:11.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:11 smithi045 ceph-mon[20143]: pgmap v933: 47 pgs: 47 active+clean; 1.3 GiB data, 8.2 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 795 op/s 2024-08-19T14:24:11.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:11 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3885192892' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "41"}]: dispatch 2024-08-19T14:24:11.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:11 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "41"}]: dispatch 2024-08-19T14:24:11.551 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:24:11.574 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:11.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:11.575 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:24:11.676 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:24:11.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:11.718 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:11.718 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:24:11.739 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:24:11.819 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:24:11.850 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:11.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:11.851 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:24:11.951 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:24:11.988 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:11.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:11.989 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:24:12.036 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:24:12.036 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:24:12.090 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:24:12.117 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:12.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:12.119 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:24:12.219 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:24:12.260 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:12.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:12.261 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:24:12.362 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:24:12.398 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:12.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:12.399 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:24:12.500 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:24:12.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:12 smithi032 ceph-mon[20004]: 1.4 scrub starts 2024-08-19T14:24:12.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:12 smithi032 ceph-mon[20004]: 1.4 scrub ok 2024-08-19T14:24:12.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:12 smithi032 ceph-mon[20004]: 4.7 scrub starts 2024-08-19T14:24:12.534 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:12.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:12.535 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:24:12.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:12 smithi045 ceph-mon[20143]: 1.4 scrub starts 2024-08-19T14:24:12.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:12 smithi045 ceph-mon[20143]: 1.4 scrub ok 2024-08-19T14:24:12.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:12 smithi045 ceph-mon[20143]: 4.7 scrub starts 2024-08-19T14:24:12.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:12 smithi031 ceph-mon[16336]: 1.4 scrub starts 2024-08-19T14:24:12.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:12 smithi031 ceph-mon[16336]: 1.4 scrub ok 2024-08-19T14:24:12.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:12 smithi031 ceph-mon[16336]: 4.7 scrub starts 2024-08-19T14:24:12.636 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:24:12.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:12.674 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:12.675 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:24:12.716 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-08-19T14:24:12.765 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-19T14:24:12.775 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:24:12.835 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:24:12.852 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:12.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:12.854 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:24:12.955 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:24:13.018 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:13.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:13.019 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:24:13.121 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:24:13.143 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:13.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:13.144 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:24:13.245 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:24:13.268 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:13.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:13.269 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:24:13.370 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:24:13.396 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:13.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:13.397 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:24:13.491 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:24:13.497 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:24:13.505 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:13 smithi031 ceph-mon[16336]: pgmap v934: 47 pgs: 47 active+clean; 1.2 GiB data, 7.9 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 764 op/s 2024-08-19T14:24:13.505 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:13 smithi031 ceph-mon[16336]: 4.7 scrub ok 2024-08-19T14:24:13.505 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:24:13.505 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:24:13.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:13 smithi032 ceph-mon[20004]: pgmap v934: 47 pgs: 47 active+clean; 1.2 GiB data, 7.9 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 764 op/s 2024-08-19T14:24:13.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:13 smithi032 ceph-mon[20004]: 4.7 scrub ok 2024-08-19T14:24:13.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:13 smithi045 ceph-mon[20143]: pgmap v934: 47 pgs: 47 active+clean; 1.2 GiB data, 7.9 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 764 op/s 2024-08-19T14:24:13.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:13 smithi045 ceph-mon[20143]: 4.7 scrub ok 2024-08-19T14:24:13.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:13.554 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:13.554 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:24:13.554 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:24:13.555 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:24:13.556 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:24:13.597 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:24:13.598 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:24:13.629 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:24:13.630 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:24:13.655 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:24:13.664 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:24:13.665 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:24:13.703 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:24:13.703 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:24:13.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:13.705 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:13.705 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:24:13.738 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:24:13.738 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:24:13.769 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:24:13.770 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:24:13.800 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:24:13.800 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:24:13.806 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:24:13.836 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:24:13.836 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:24:13.864 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:13.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:13.865 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:24:13.936 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:24:13.937 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:24:13.965 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:24:13.966 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:24:13.966 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:24:14.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:14.004 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:14.005 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:24:14.106 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:24:14.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:14.130 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:14.130 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:24:14.231 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:24:14.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:14.259 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:14.259 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:24:14.360 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:24:14.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:14 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:24:14.368 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:14 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:24:14.368 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:14 smithi031 ceph-mon[16336]: osdmap e406: 12 total, 12 up, 9 in 2024-08-19T14:24:14.368 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:14 smithi031 ceph-mon[16336]: 1.c scrub starts 2024-08-19T14:24:14.368 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:14 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/426286640' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:24:14.368 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:14 smithi031 ceph-mon[16336]: 1.c scrub ok 2024-08-19T14:24:14.395 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:14.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:14.396 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:24:14.497 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:24:14.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:14 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:24:14.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:14 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:24:14.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:14 smithi032 ceph-mon[20004]: osdmap e406: 12 total, 12 up, 9 in 2024-08-19T14:24:14.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:14 smithi032 ceph-mon[20004]: 1.c scrub starts 2024-08-19T14:24:14.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:14 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/426286640' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:24:14.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:14 smithi032 ceph-mon[20004]: 1.c scrub ok 2024-08-19T14:24:14.521 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:14.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:14.522 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:24:14.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:14 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:24:14.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:14 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:24:14.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:14 smithi045 ceph-mon[20143]: osdmap e406: 12 total, 12 up, 9 in 2024-08-19T14:24:14.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:14 smithi045 ceph-mon[20143]: 1.c scrub starts 2024-08-19T14:24:14.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:14 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/426286640' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:24:14.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:14 smithi045 ceph-mon[20143]: 1.c scrub ok 2024-08-19T14:24:14.623 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:24:14.647 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:14.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:14.648 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:24:14.750 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:24:14.773 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:14.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:14.774 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:24:14.875 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:24:14.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:14.899 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:14.899 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:24:15.000 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:24:15.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:15.029 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:15.029 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:24:15.131 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:24:15.155 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:15.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:15.156 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:24:15.257 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:24:15.260 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:24:15.261 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-08-19T14:24:15.261 INFO:teuthology.orchestra.run.smithi031.stdout: "num_ops": 0 2024-08-19T14:24:15.261 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:24:15.267 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is unset 2024-08-19T14:24:15.286 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:15.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:15.287 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:24:15.388 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:24:15.414 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:15.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:15.415 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:24:15.517 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:24:15.545 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:15.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:15.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:24:15.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:15 smithi045 ceph-mon[20143]: pgmap v936: 47 pgs: 47 active+clean; 1.0 GiB data, 7.3 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 1.07k op/s 2024-08-19T14:24:15.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:15 smithi045 ceph-mon[20143]: 1.12 scrub starts 2024-08-19T14:24:15.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:15 smithi045 ceph-mon[20143]: 1.12 scrub ok 2024-08-19T14:24:15.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:15 smithi045 ceph-mon[20143]: 1.1c scrub starts 2024-08-19T14:24:15.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:15 smithi045 ceph-mon[20143]: 1.1c scrub ok 2024-08-19T14:24:15.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:15 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/826653160' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:24:15.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:15 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:24:15.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:15 smithi031 ceph-mon[16336]: pgmap v936: 47 pgs: 47 active+clean; 1.0 GiB data, 7.3 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 1.07k op/s 2024-08-19T14:24:15.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:15 smithi031 ceph-mon[16336]: 1.12 scrub starts 2024-08-19T14:24:15.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:15 smithi031 ceph-mon[16336]: 1.12 scrub ok 2024-08-19T14:24:15.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:15 smithi031 ceph-mon[16336]: 1.1c scrub starts 2024-08-19T14:24:15.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:15 smithi031 ceph-mon[16336]: 1.1c scrub ok 2024-08-19T14:24:15.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:15 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/826653160' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:24:15.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:15 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:24:15.648 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:24:15.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:15.687 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:15.687 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:24:15.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:15 smithi032 ceph-mon[20004]: pgmap v936: 47 pgs: 47 active+clean; 1.0 GiB data, 7.3 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 1.07k op/s 2024-08-19T14:24:15.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:15 smithi032 ceph-mon[20004]: 1.12 scrub starts 2024-08-19T14:24:15.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:15 smithi032 ceph-mon[20004]: 1.12 scrub ok 2024-08-19T14:24:15.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:15 smithi032 ceph-mon[20004]: 1.1c scrub starts 2024-08-19T14:24:15.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:15 smithi032 ceph-mon[20004]: 1.1c scrub ok 2024-08-19T14:24:15.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:15 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/826653160' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:24:15.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:15 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:24:15.787 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:24:15.810 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:15.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:15.811 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:24:15.912 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:24:15.935 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:15.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:15.936 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:24:16.038 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:24:16.061 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:16.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:16.062 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:24:16.163 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:24:16.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:16.190 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:16.190 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:24:16.292 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:24:16.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:16.318 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:16.319 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:24:16.420 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:24:16.444 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:16.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:16.446 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:24:16.466 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:16 smithi031 ceph-mon[16336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:24:16.466 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:16 smithi031 ceph-mon[16336]: Cluster is now healthy 2024-08-19T14:24:16.466 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:16 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:24:16.466 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:16 smithi031 ceph-mon[16336]: osdmap e407: 12 total, 12 up, 9 in 2024-08-19T14:24:16.488 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 7, 8, 9, 11] out_osds: [10, 6, 2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-08-19T14:24:16.488 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-19T14:24:16.488 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-08-19T14:24:16.488 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-19T14:24:16.513 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-08-19T14:24:16.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:16 smithi045 ceph-mon[20143]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:24:16.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:16 smithi045 ceph-mon[20143]: Cluster is now healthy 2024-08-19T14:24:16.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:16 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:24:16.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:16 smithi045 ceph-mon[20143]: osdmap e407: 12 total, 12 up, 9 in 2024-08-19T14:24:16.547 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:24:16.569 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:16.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:16.570 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:24:16.671 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:24:16.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:16.694 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:16.695 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:24:16.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:16 smithi032 ceph-mon[20004]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:24:16.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:16 smithi032 ceph-mon[20004]: Cluster is now healthy 2024-08-19T14:24:16.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:16 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:24:16.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:16 smithi032 ceph-mon[20004]: osdmap e407: 12 total, 12 up, 9 in 2024-08-19T14:24:16.795 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:24:16.829 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:16.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:16.830 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:24:16.931 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:24:16.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:16.971 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:16.971 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:24:17.006 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:24:17.072 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:24:17.096 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:17.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:17.097 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:24:17.199 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:24:17.223 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:17.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:17.224 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:24:17.316 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:24:17.324 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:24:17.389 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:17.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:17.390 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:24:17.391 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:24:17.477 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:17 smithi031 ceph-mon[16336]: pgmap v938: 47 pgs: 47 active+clean; 1.0 GiB data, 7.3 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 661 op/s 2024-08-19T14:24:17.477 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:17 smithi031 ceph-mon[16336]: 1.f scrub starts 2024-08-19T14:24:17.477 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:17 smithi031 ceph-mon[16336]: 1.f scrub ok 2024-08-19T14:24:17.492 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:24:17.517 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:17.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:17.518 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:24:17.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:17 smithi045 ceph-mon[20143]: pgmap v938: 47 pgs: 47 active+clean; 1.0 GiB data, 7.3 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 661 op/s 2024-08-19T14:24:17.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:17 smithi045 ceph-mon[20143]: 1.f scrub starts 2024-08-19T14:24:17.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:17 smithi045 ceph-mon[20143]: 1.f scrub ok 2024-08-19T14:24:17.619 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:24:17.642 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:17.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:17.643 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:24:17.744 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:24:17.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:17 smithi032 ceph-mon[20004]: pgmap v938: 47 pgs: 47 active+clean; 1.0 GiB data, 7.3 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 661 op/s 2024-08-19T14:24:17.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:17 smithi032 ceph-mon[20004]: 1.f scrub starts 2024-08-19T14:24:17.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:17 smithi032 ceph-mon[20004]: 1.f scrub ok 2024-08-19T14:24:17.770 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:17.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:17.771 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:24:17.872 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:24:17.924 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:17.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:17.926 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:24:18.026 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:24:18.056 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:18.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:18.057 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:24:18.157 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:24:18.192 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:18.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:18.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:24:18.295 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:24:18.318 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:18.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:18.319 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:24:18.368 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:24:18.368 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-08-19T14:24:18.368 INFO:teuthology.orchestra.run.smithi031.stdout: "complaint_time": 30, 2024-08-19T14:24:18.368 INFO:teuthology.orchestra.run.smithi031.stdout: "num_blocked_ops": 0 2024-08-19T14:24:18.368 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:24:18.420 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:24:18.442 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:18.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:18.443 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:24:18.458 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:24:18.459 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-08-19T14:24:18.544 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:24:18.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:18 smithi045 ceph-mon[20143]: 4.f deep-scrub starts 2024-08-19T14:24:18.568 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:18.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:18.569 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:24:18.608 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:18 smithi031 ceph-mon[16336]: 4.f deep-scrub starts 2024-08-19T14:24:18.669 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:24:18.695 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:18.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:18.696 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:24:18.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:18 smithi032 ceph-mon[20004]: 4.f deep-scrub starts 2024-08-19T14:24:18.797 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:24:18.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:18.819 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:18.819 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:24:18.819 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:24:18.920 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:24:18.947 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:18.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:18.948 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:24:19.049 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:24:19.073 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:19.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:19.075 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:24:19.175 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:24:19.182 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-19T14:24:19.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:19.198 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:19.198 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:24:19.300 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:24:19.326 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:19.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:19.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:24:19.415 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":939,"stamp":"2024-08-19T14:24:17.950391+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":934216248,"num_objects":14249,"num_object_clones":0,"num_object_copies":42747,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14249,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":248099,"num_write_kb":8395009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4140,"num_bytes_recovered":271319040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":165208,"ondisk_log_size":165208,"up":141,"acting":141,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":142,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":7315352,"kb_used_data":3073656,"kb_used_omap":14,"kb_used_meta":4241329,"kb_avail":836391016,"statfs":{"total":863955320832,"available":856464400384,"internally_reserved":0,"allocated":3147423744,"data_stored":3130660128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14353,"internal_metadata":4343121903},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,3,3,2,9,2,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-467075072,"num_objects":-7127,"num_object_clones":0,"num_object_copies":-21381,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-7127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7127,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003567"},"pg_stats":[{"pgid":"4.5","version":"407'15593","reported_seq":15867,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-08-19T14:24:16.003993+0000","last_change":"2024-08-19T14:23:44.384935+0000","last_active":"2024-08-19T14:24:16.003993+0000","last_peered":"2024-08-19T14:24:16.003993+0000","last_clean":"2024-08-19T14:24:16.003993+0000","last_became_active":"2024-08-19T14:20:29.267549+0000","last_became_peered":"2024-08-19T14:20:29.267549+0000","last_unstale":"2024-08-19T14:24:16.003993+0000","last_undegraded":"2024-08-19T14:24:16.003993+0000","last_fullsized":"2024-08-19T14:24:16.003993+0000","mapping_epoch":335,"log_start":"360'5500","ondisk_log_start":"360'5500","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"399'13841","last_scrub_stamp":"2024-08-19T14:23:44.384705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:23:44.384705+0000","objects_scrubbed":2589,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:24:49.673047+0000","scrub_duration":817,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52887552,"num_objects":807,"num_object_clones":0,"num_object_copies":2421,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":807,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15593,"num_write_kb":524800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,0],"acting":[11,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"70'139","reported_seq":1037,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-19T14:24:13.263616+0000","last_change":"2024-08-19T14:23:22.006193+0000","last_active":"2024-08-19T14:24:13.263616+0000","last_peered":"2024-08-19T14:24:13.263616+0000","last_clean":"2024-08-19T14:24:13.263616+0000","last_became_active":"2024-08-19T14:18:13.807133+0000","last_became_peered":"2024-08-19T14:18:13.807133+0000","last_unstale":"2024-08-19T14:24:13.263616+0000","last_undegraded":"2024-08-19T14:24:13.263616+0000","last_fullsized":"2024-08-19T14:24:13.263616+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:23:22.006116+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:19:58.072083+0000","last_clean_scrub_stamp":"2024-08-19T14:23:22.006116+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:24:47.877236+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,9],"acting":[7,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"406'15665","reported_seq":24141,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-19T14:24:13.501624+0000","last_change":"2024-08-19T14:23:51.058049+0000","last_active":"2024-08-19T14:24:13.501624+0000","last_peered":"2024-08-19T14:24:13.501624+0000","last_clean":"2024-08-19T14:24:13.501624+0000","last_became_active":"2024-08-19T14:23:01.741503+0000","last_became_peered":"2024-08-19T14:23:01.741503+0000","last_unstale":"2024-08-19T14:24:13.501624+0000","last_undegraded":"2024-08-19T14:24:13.501624+0000","last_fullsized":"2024-08-19T14:24:13.501624+0000","mapping_epoch":384,"log_start":"360'5600","ondisk_log_start":"360'5600","created":335,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'14336","last_scrub_stamp":"2024-08-19T14:23:51.057894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:23:51.057894+0000","objects_scrubbed":2372,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:24:59.441268+0000","scrub_duration":622,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66387968,"num_objects":1013,"num_object_clones":0,"num_object_copies":3039,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1013,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15665,"num_write_kb":533696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4140,"num_bytes_recovered":271319040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,7],"acting":[9,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.3","version":"70'139","reported_seq":934,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-08-19T14:24:09.246351+0000","last_change":"2024-08-19T14:23:27.488105+0000","last_active":"2024-08-19T14:24:09.246351+0000","last_peered":"2024-08-19T14:24:09.246351+0000","last_clean":"2024-08-19T14:24:09.246351+0000","last_became_active":"2024-08-19T14:18:13.804721+0000","last_became_peered":"2024-08-19T14:18:13.804721+0000","last_unstale":"2024-08-19T14:24:09.246351+0000","last_undegraded":"2024-08-19T14:24:09.246351+0000","last_fullsized":"2024-08-19T14:24:09.246351+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:23:27.487983+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:23:27.487983+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:24:39.468752+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"405'15440","reported_seq":15611,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-08-19T14:24:13.043550+0000","last_change":"2024-08-19T14:24:12.168472+0000","last_active":"2024-08-19T14:24:13.043550+0000","last_peered":"2024-08-19T14:24:13.043550+0000","last_clean":"2024-08-19T14:24:13.043550+0000","last_became_active":"2024-08-19T14:20:29.267811+0000","last_became_peered":"2024-08-19T14:20:29.267811+0000","last_unstale":"2024-08-19T14:24:13.043550+0000","last_undegraded":"2024-08-19T14:24:13.043550+0000","last_fullsized":"2024-08-19T14:24:13.043550+0000","mapping_epoch":335,"log_start":"359'5400","ondisk_log_start":"359'5400","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'15389","last_scrub_stamp":"2024-08-19T14:24:12.168208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:24:12.168208+0000","objects_scrubbed":1079,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:25:25.313065+0000","scrub_duration":696,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65929216,"num_objects":1006,"num_object_clones":0,"num_object_copies":3018,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1006,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15440,"num_write_kb":526272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"70'139","reported_seq":884,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-19T14:24:13.263498+0000","last_change":"2024-08-19T14:23:24.042754+0000","last_active":"2024-08-19T14:24:13.263498+0000","last_peered":"2024-08-19T14:24:13.263498+0000","last_clean":"2024-08-19T14:24:13.263498+0000","last_became_active":"2024-08-19T14:20:50.742354+0000","last_became_peered":"2024-08-19T14:20:50.742354+0000","last_unstale":"2024-08-19T14:24:13.263498+0000","last_undegraded":"2024-08-19T14:24:13.263498+0000","last_fullsized":"2024-08-19T14:24:13.263498+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:23:24.042679+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:19:59.302931+0000","last_clean_scrub_stamp":"2024-08-19T14:23:24.042679+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:24:27.949990+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"405'15344","reported_seq":15522,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-08-19T14:24:13.040275+0000","last_change":"2024-08-19T14:23:56.303974+0000","last_active":"2024-08-19T14:24:13.040275+0000","last_peered":"2024-08-19T14:24:13.040275+0000","last_clean":"2024-08-19T14:24:13.040275+0000","last_became_active":"2024-08-19T14:20:29.267331+0000","last_became_peered":"2024-08-19T14:20:29.267331+0000","last_unstale":"2024-08-19T14:24:13.040275+0000","last_undegraded":"2024-08-19T14:24:13.040275+0000","last_fullsized":"2024-08-19T14:24:13.040275+0000","mapping_epoch":335,"log_start":"359'5300","ondisk_log_start":"359'5300","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"401'14428","last_scrub_stamp":"2024-08-19T14:23:56.303276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:23:56.303276+0000","objects_scrubbed":1976,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:25:16.405284+0000","scrub_duration":793,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68288512,"num_objects":1042,"num_object_clones":0,"num_object_copies":3126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1042,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15344,"num_write_kb":524352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"70'139","reported_seq":933,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-19T14:24:13.256778+0000","last_change":"2024-08-19T14:23:24.392988+0000","last_active":"2024-08-19T14:24:13.256778+0000","last_peered":"2024-08-19T14:24:13.256778+0000","last_clean":"2024-08-19T14:24:13.256778+0000","last_became_active":"2024-08-19T14:18:13.808505+0000","last_became_peered":"2024-08-19T14:18:13.808505+0000","last_unstale":"2024-08-19T14:24:13.256778+0000","last_undegraded":"2024-08-19T14:24:13.256778+0000","last_fullsized":"2024-08-19T14:24:13.256778+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:23:24.392874+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:23:24.392874+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:24:30.094427+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,11],"acting":[9,0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"406'15527","reported_seq":15747,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-19T14:24:13.472667+0000","last_change":"2024-08-19T14:23:47.764099+0000","last_active":"2024-08-19T14:24:13.472667+0000","last_peered":"2024-08-19T14:24:13.472667+0000","last_clean":"2024-08-19T14:24:13.472667+0000","last_became_active":"2024-08-19T14:20:29.268443+0000","last_became_peered":"2024-08-19T14:20:29.268443+0000","last_unstale":"2024-08-19T14:24:13.472667+0000","last_undegraded":"2024-08-19T14:24:13.472667+0000","last_fullsized":"2024-08-19T14:24:13.472667+0000","mapping_epoch":335,"log_start":"360'5500","ondisk_log_start":"360'5500","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"399'14070","last_scrub_stamp":"2024-08-19T14:23:47.764022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:23:47.764022+0000","objects_scrubbed":2456,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:25:03.882630+0000","scrub_duration":437,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64552960,"num_objects":985,"num_object_clones":0,"num_object_copies":2955,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15527,"num_write_kb":528384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,11],"acting":[9,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.7","version":"70'139","reported_seq":584,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-19T14:24:13.258004+0000","last_change":"2024-08-19T14:23:16.203977+0000","last_active":"2024-08-19T14:24:13.258004+0000","last_peered":"2024-08-19T14:24:13.258004+0000","last_clean":"2024-08-19T14:24:13.258004+0000","last_became_active":"2024-08-19T14:20:28.204384+0000","last_became_peered":"2024-08-19T14:20:28.204384+0000","last_unstale":"2024-08-19T14:24:13.258004+0000","last_undegraded":"2024-08-19T14:24:13.258004+0000","last_fullsized":"2024-08-19T14:24:13.258004+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:23:16.203732+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:21:46.057624+0000","last_clean_scrub_stamp":"2024-08-19T14:23:16.203732+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:24:27.551036+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"407'15475","reported_seq":15822,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-08-19T14:24:16.002363+0000","last_change":"2024-08-19T14:23:22.613580+0000","last_active":"2024-08-19T14:24:16.002363+0000","last_peered":"2024-08-19T14:24:16.002363+0000","last_clean":"2024-08-19T14:24:16.002363+0000","last_became_active":"2024-08-19T14:20:29.267379+0000","last_became_peered":"2024-08-19T14:20:29.267379+0000","last_unstale":"2024-08-19T14:24:16.002363+0000","last_undegraded":"2024-08-19T14:24:16.002363+0000","last_fullsized":"2024-08-19T14:24:16.002363+0000","mapping_epoch":335,"log_start":"360'5400","ondisk_log_start":"360'5400","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"394'12559","last_scrub_stamp":"2024-08-19T14:23:22.613324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:23:22.613324+0000","objects_scrubbed":3829,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:24:25.181152+0000","scrub_duration":1081,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":57344000,"num_objects":875,"num_object_clones":0,"num_object_copies":2625,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15475,"num_write_kb":523200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,1],"acting":[11,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.6","version":"70'139","reported_seq":937,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-19T14:24:13.257043+0000","last_change":"2024-08-19T14:23:23.375634+0000","last_active":"2024-08-19T14:24:13.257043+0000","last_peered":"2024-08-19T14:24:13.257043+0000","last_clean":"2024-08-19T14:24:13.257043+0000","last_became_active":"2024-08-19T14:18:26.874902+0000","last_became_peered":"2024-08-19T14:18:26.874902+0000","last_unstale":"2024-08-19T14:24:13.257043+0000","last_undegraded":"2024-08-19T14:24:13.257043+0000","last_fullsized":"2024-08-19T14:24:13.257043+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:23:23.375546+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:27.848870+0000","last_clean_scrub_stamp":"2024-08-19T14:23:23.375546+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:24:25.026779+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,4],"acting":[9,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"406'15288","reported_seq":15573,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-19T14:24:13.732077+0000","last_change":"2024-08-19T14:23:26.217160+0000","last_active":"2024-08-19T14:24:13.732077+0000","last_peered":"2024-08-19T14:24:13.732077+0000","last_clean":"2024-08-19T14:24:13.732077+0000","last_became_active":"2024-08-19T14:20:29.267360+0000","last_became_peered":"2024-08-19T14:20:29.267360+0000","last_unstale":"2024-08-19T14:24:13.732077+0000","last_undegraded":"2024-08-19T14:24:13.732077+0000","last_fullsized":"2024-08-19T14:24:13.732077+0000","mapping_epoch":335,"log_start":"359'5200","ondisk_log_start":"359'5200","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"394'12651","last_scrub_stamp":"2024-08-19T14:23:26.216619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:23:26.216619+0000","objects_scrubbed":3626,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:24:44.882612+0000","scrub_duration":1260,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62390272,"num_objects":952,"num_object_clones":0,"num_object_copies":2856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":952,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15288,"num_write_kb":519680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"70'139","reported_seq":760,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-08-19T14:24:15.272265+0000","last_change":"2024-08-19T14:23:45.496873+0000","last_active":"2024-08-19T14:24:15.272265+0000","last_peered":"2024-08-19T14:24:15.272265+0000","last_clean":"2024-08-19T14:24:15.272265+0000","last_became_active":"2024-08-19T14:18:26.881435+0000","last_became_peered":"2024-08-19T14:18:26.881435+0000","last_unstale":"2024-08-19T14:24:15.272265+0000","last_undegraded":"2024-08-19T14:24:15.272265+0000","last_fullsized":"2024-08-19T14:24:15.272265+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:23:45.496751+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:23:45.496751+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:24:46.294715+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,4],"acting":[11,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.1","version":"407'15675","reported_seq":15937,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-08-19T14:24:16.028935+0000","last_change":"2024-08-19T14:23:56.911837+0000","last_active":"2024-08-19T14:24:16.028935+0000","last_peered":"2024-08-19T14:24:16.028935+0000","last_clean":"2024-08-19T14:24:16.028935+0000","last_became_active":"2024-08-19T14:20:29.268465+0000","last_became_peered":"2024-08-19T14:20:29.268465+0000","last_unstale":"2024-08-19T14:24:16.028935+0000","last_undegraded":"2024-08-19T14:24:16.028935+0000","last_fullsized":"2024-08-19T14:24:16.028935+0000","mapping_epoch":335,"log_start":"360'5600","ondisk_log_start":"360'5600","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"401'14629","last_scrub_stamp":"2024-08-19T14:23:56.911497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:23:56.911497+0000","objects_scrubbed":1881,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:25:24.115176+0000","scrub_duration":423,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54198272,"num_objects":827,"num_object_clones":0,"num_object_copies":2481,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":827,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15675,"num_write_kb":528064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.4","version":"70'139","reported_seq":993,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-08-19T14:24:15.272279+0000","last_change":"2024-08-19T14:24:11.345984+0000","last_active":"2024-08-19T14:24:15.272279+0000","last_peered":"2024-08-19T14:24:15.272279+0000","last_clean":"2024-08-19T14:24:15.272279+0000","last_became_active":"2024-08-19T14:18:13.806013+0000","last_became_peered":"2024-08-19T14:18:13.806013+0000","last_unstale":"2024-08-19T14:24:15.272279+0000","last_undegraded":"2024-08-19T14:24:15.272279+0000","last_fullsized":"2024-08-19T14:24:15.272279+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:11.345873+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:24:11.345873+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:25:14.696871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.f","version":"405'15334","reported_seq":15483,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-08-19T14:24:13.067873+0000","last_change":"2024-08-19T14:20:29.212298+0000","last_active":"2024-08-19T14:24:13.067873+0000","last_peered":"2024-08-19T14:24:13.067873+0000","last_clean":"2024-08-19T14:24:13.067873+0000","last_became_active":"2024-08-19T14:20:29.212003+0000","last_became_peered":"2024-08-19T14:20:29.212003+0000","last_unstale":"2024-08-19T14:24:13.067873+0000","last_undegraded":"2024-08-19T14:24:13.067873+0000","last_fullsized":"2024-08-19T14:24:13.067873+0000","mapping_epoch":335,"log_start":"359'5300","ondisk_log_start":"359'5300","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:20:28.194308+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63438848,"num_objects":968,"num_object_clones":0,"num_object_copies":2904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":968,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15334,"num_write_kb":521664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,11],"acting":[1,3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"70'139","reported_seq":945,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-19T14:24:13.256861+0000","last_change":"2024-08-19T14:23:25.404078+0000","last_active":"2024-08-19T14:24:13.256861+0000","last_peered":"2024-08-19T14:24:13.256861+0000","last_clean":"2024-08-19T14:24:13.256861+0000","last_became_active":"2024-08-19T14:18:26.875006+0000","last_became_peered":"2024-08-19T14:18:26.875006+0000","last_unstale":"2024-08-19T14:24:13.256861+0000","last_undegraded":"2024-08-19T14:24:13.256861+0000","last_fullsized":"2024-08-19T14:24:13.256861+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:23:25.403959+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:28.864870+0000","last_clean_scrub_stamp":"2024-08-19T14:23:25.403959+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:24:36.949528+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,0],"acting":[9,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.c","version":"407'15586","reported_seq":15878,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-08-19T14:24:17.649072+0000","last_change":"2024-08-19T14:23:19.956127+0000","last_active":"2024-08-19T14:24:17.649072+0000","last_peered":"2024-08-19T14:24:17.649072+0000","last_clean":"2024-08-19T14:24:17.649072+0000","last_became_active":"2024-08-19T14:20:29.266373+0000","last_became_peered":"2024-08-19T14:20:29.266373+0000","last_unstale":"2024-08-19T14:24:17.649072+0000","last_undegraded":"2024-08-19T14:24:17.649072+0000","last_fullsized":"2024-08-19T14:24:17.649072+0000","mapping_epoch":335,"log_start":"360'5500","ondisk_log_start":"360'5500","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"393'12368","last_scrub_stamp":"2024-08-19T14:23:19.955924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:23:19.955924+0000","objects_scrubbed":4015,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:24:21.233858+0000","scrub_duration":1085,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50855936,"num_objects":776,"num_object_clones":0,"num_object_copies":2328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15586,"num_write_kb":523584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,8],"acting":[4,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"70'139","reported_seq":1060,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-19T14:24:13.263569+0000","last_change":"2024-08-19T14:23:44.965612+0000","last_active":"2024-08-19T14:24:13.263569+0000","last_peered":"2024-08-19T14:24:13.263569+0000","last_clean":"2024-08-19T14:24:13.263569+0000","last_became_active":"2024-08-19T14:18:26.887657+0000","last_became_peered":"2024-08-19T14:18:26.887657+0000","last_unstale":"2024-08-19T14:24:13.263569+0000","last_undegraded":"2024-08-19T14:24:13.263569+0000","last_fullsized":"2024-08-19T14:24:13.263569+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:23:44.965485+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:23:44.965485+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:24:53.391995+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"407'15514","reported_seq":15791,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-08-19T14:24:17.733856+0000","last_change":"2024-08-19T14:23:47.756240+0000","last_active":"2024-08-19T14:24:17.733856+0000","last_peered":"2024-08-19T14:24:17.733856+0000","last_clean":"2024-08-19T14:24:17.733856+0000","last_became_active":"2024-08-19T14:20:29.208599+0000","last_became_peered":"2024-08-19T14:20:29.208599+0000","last_unstale":"2024-08-19T14:24:17.733856+0000","last_undegraded":"2024-08-19T14:24:17.733856+0000","last_fullsized":"2024-08-19T14:24:17.733856+0000","mapping_epoch":335,"log_start":"360'5500","ondisk_log_start":"360'5500","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"399'13776","last_scrub_stamp":"2024-08-19T14:23:47.756057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:23:47.756057+0000","objects_scrubbed":2492,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:24:57.431228+0000","scrub_duration":670,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48168984,"num_objects":736,"num_object_clones":0,"num_object_copies":2208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":736,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15514,"num_write_kb":519937,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,1],"acting":[4,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"70'139","reported_seq":959,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-08-19T14:24:15.269752+0000","last_change":"2024-08-19T14:23:19.245681+0000","last_active":"2024-08-19T14:24:15.269752+0000","last_peered":"2024-08-19T14:24:15.269752+0000","last_clean":"2024-08-19T14:24:15.269752+0000","last_became_active":"2024-08-19T14:18:26.887277+0000","last_became_peered":"2024-08-19T14:18:26.887277+0000","last_unstale":"2024-08-19T14:24:15.269752+0000","last_undegraded":"2024-08-19T14:24:15.269752+0000","last_fullsized":"2024-08-19T14:24:15.269752+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:23:19.245574+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:23:19.245574+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:24:39.015396+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,8],"acting":[5,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"407'15388","reported_seq":15659,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-08-19T14:24:17.685457+0000","last_change":"2024-08-19T14:23:48.815007+0000","last_active":"2024-08-19T14:24:17.685457+0000","last_peered":"2024-08-19T14:24:17.685457+0000","last_clean":"2024-08-19T14:24:17.685457+0000","last_became_active":"2024-08-19T14:20:29.208748+0000","last_became_peered":"2024-08-19T14:20:29.208748+0000","last_unstale":"2024-08-19T14:24:17.685457+0000","last_undegraded":"2024-08-19T14:24:17.685457+0000","last_fullsized":"2024-08-19T14:24:17.685457+0000","mapping_epoch":335,"log_start":"359'5300","ondisk_log_start":"359'5300","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'13814","last_scrub_stamp":"2024-08-19T14:23:48.814645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:23:48.814645+0000","objects_scrubbed":2349,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:24:56.662744+0000","scrub_duration":693,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49152000,"num_objects":750,"num_object_clones":0,"num_object_copies":2250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15388,"num_write_kb":516416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"70'139","reported_seq":960,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-08-19T14:24:15.272109+0000","last_change":"2024-08-19T14:23:23.564599+0000","last_active":"2024-08-19T14:24:15.272109+0000","last_peered":"2024-08-19T14:24:15.272109+0000","last_clean":"2024-08-19T14:24:15.272109+0000","last_became_active":"2024-08-19T14:18:26.875179+0000","last_became_peered":"2024-08-19T14:18:26.875179+0000","last_unstale":"2024-08-19T14:24:15.272109+0000","last_undegraded":"2024-08-19T14:24:15.272109+0000","last_fullsized":"2024-08-19T14:24:15.272109+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:23:23.564518+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:23:23.564518+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:24:28.111207+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.b","version":"407'15523","reported_seq":15801,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-08-19T14:24:16.381677+0000","last_change":"2024-08-19T14:23:23.481298+0000","last_active":"2024-08-19T14:24:16.381677+0000","last_peered":"2024-08-19T14:24:16.381677+0000","last_clean":"2024-08-19T14:24:16.381677+0000","last_became_active":"2024-08-19T14:20:29.207372+0000","last_became_peered":"2024-08-19T14:20:29.207372+0000","last_unstale":"2024-08-19T14:24:16.381677+0000","last_undegraded":"2024-08-19T14:24:16.381677+0000","last_fullsized":"2024-08-19T14:24:16.381677+0000","mapping_epoch":335,"log_start":"360'5500","ondisk_log_start":"360'5500","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"394'12554","last_scrub_stamp":"2024-08-19T14:23:23.481035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:23:23.481035+0000","objects_scrubbed":3836,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:24:52.522452+0000","scrub_duration":1875,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53673984,"num_objects":819,"num_object_clones":0,"num_object_copies":2457,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15523,"num_write_kb":522944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"70'139","reported_seq":614,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-08-19T14:24:15.271396+0000","last_change":"2024-08-19T14:23:16.834552+0000","last_active":"2024-08-19T14:24:15.271396+0000","last_peered":"2024-08-19T14:24:15.271396+0000","last_clean":"2024-08-19T14:24:15.271396+0000","last_became_active":"2024-08-19T14:18:26.884105+0000","last_became_peered":"2024-08-19T14:18:26.884105+0000","last_unstale":"2024-08-19T14:24:15.271396+0000","last_undegraded":"2024-08-19T14:24:15.271396+0000","last_fullsized":"2024-08-19T14:24:15.271396+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:23:16.834478+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:20:15.321945+0000","last_clean_scrub_stamp":"2024-08-19T14:23:16.834478+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:24:31.654430+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"405'15620","reported_seq":15801,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-08-19T14:24:13.035613+0000","last_change":"2024-08-19T14:23:54.872969+0000","last_active":"2024-08-19T14:24:13.035613+0000","last_peered":"2024-08-19T14:24:13.035613+0000","last_clean":"2024-08-19T14:24:13.035613+0000","last_became_active":"2024-08-19T14:20:29.269432+0000","last_became_peered":"2024-08-19T14:20:29.269432+0000","last_unstale":"2024-08-19T14:24:13.035613+0000","last_undegraded":"2024-08-19T14:24:13.035613+0000","last_fullsized":"2024-08-19T14:24:13.035613+0000","mapping_epoch":335,"log_start":"360'5600","ondisk_log_start":"360'5600","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"401'14514","last_scrub_stamp":"2024-08-19T14:23:54.872659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:23:54.872659+0000","objects_scrubbed":2127,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:25:22.243401+0000","scrub_duration":1279,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65273856,"num_objects":996,"num_object_clones":0,"num_object_copies":2988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":996,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15620,"num_write_kb":531712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.f","version":"70'139","reported_seq":866,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-08-19T14:24:16.076198+0000","last_change":"2024-08-19T14:24:16.076239+0000","last_active":"2024-08-19T14:24:16.076198+0000","last_peered":"2024-08-19T14:24:16.076198+0000","last_clean":"2024-08-19T14:24:16.076198+0000","last_became_active":"2024-08-19T14:18:18.711708+0000","last_became_peered":"2024-08-19T14:18:18.711708+0000","last_unstale":"2024-08-19T14:24:16.076198+0000","last_undegraded":"2024-08-19T14:24:16.076198+0000","last_fullsized":"2024-08-19T14:24:16.076198+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:16.076143+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:24:16.076143+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:25:32.997797+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"407'15600","reported_seq":15883,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-08-19T14:24:15.971563+0000","last_change":"2024-08-19T14:23:18.636632+0000","last_active":"2024-08-19T14:24:15.971563+0000","last_peered":"2024-08-19T14:24:15.971563+0000","last_clean":"2024-08-19T14:24:15.971563+0000","last_became_active":"2024-08-19T14:20:29.211975+0000","last_became_peered":"2024-08-19T14:20:29.211975+0000","last_unstale":"2024-08-19T14:24:15.971563+0000","last_undegraded":"2024-08-19T14:24:15.971563+0000","last_fullsized":"2024-08-19T14:24:15.971563+0000","mapping_epoch":335,"log_start":"360'5500","ondisk_log_start":"360'5500","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"392'12385","last_scrub_stamp":"2024-08-19T14:23:18.636481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:23:18.636481+0000","objects_scrubbed":4109,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:24:36.643654+0000","scrub_duration":1122,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56229888,"num_objects":858,"num_object_clones":0,"num_object_copies":2574,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15600,"num_write_kb":526656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,3],"acting":[11,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.c","version":"70'139","reported_seq":725,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-19T14:24:13.465009+0000","last_change":"2024-08-19T14:24:13.465173+0000","last_active":"2024-08-19T14:24:13.465009+0000","last_peered":"2024-08-19T14:24:13.465009+0000","last_clean":"2024-08-19T14:24:13.465009+0000","last_became_active":"2024-08-19T14:18:26.872942+0000","last_became_peered":"2024-08-19T14:18:26.872942+0000","last_unstale":"2024-08-19T14:24:13.465009+0000","last_undegraded":"2024-08-19T14:24:13.465009+0000","last_fullsized":"2024-08-19T14:24:13.465009+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:13.463716+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:22:52.164586+0000","last_clean_scrub_stamp":"2024-08-19T14:24:13.463716+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:25:23.123046+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,9],"acting":[3,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"407'15527","reported_seq":15797,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-08-19T14:24:16.018618+0000","last_change":"2024-08-19T14:23:54.734521+0000","last_active":"2024-08-19T14:24:16.018618+0000","last_peered":"2024-08-19T14:24:16.018618+0000","last_clean":"2024-08-19T14:24:16.018618+0000","last_became_active":"2024-08-19T14:20:29.267442+0000","last_became_peered":"2024-08-19T14:20:29.267442+0000","last_unstale":"2024-08-19T14:24:16.018618+0000","last_undegraded":"2024-08-19T14:24:16.018618+0000","last_fullsized":"2024-08-19T14:24:16.018618+0000","mapping_epoch":335,"log_start":"360'5500","ondisk_log_start":"360'5500","created":335,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"401'14297","last_scrub_stamp":"2024-08-19T14:23:54.734387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:20:28.194308+0000","last_clean_scrub_stamp":"2024-08-19T14:23:54.734387+0000","objects_scrubbed":2072,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:25:14.711064+0000","scrub_duration":300,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54853632,"num_objects":837,"num_object_clones":0,"num_object_copies":2511,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":837,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15527,"num_write_kb":523648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,9],"acting":[11,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.d","version":"70'139","reported_seq":632,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-19T14:24:13.263731+0000","last_change":"2024-08-19T14:23:48.931295+0000","last_active":"2024-08-19T14:24:13.263731+0000","last_peered":"2024-08-19T14:24:13.263731+0000","last_clean":"2024-08-19T14:24:13.263731+0000","last_became_active":"2024-08-19T14:18:26.882002+0000","last_became_peered":"2024-08-19T14:18:26.882002+0000","last_unstale":"2024-08-19T14:24:13.263731+0000","last_undegraded":"2024-08-19T14:24:13.263731+0000","last_fullsized":"2024-08-19T14:24:13.263731+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:23:48.931225+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:23:48.931225+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:24:50.112289+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"70'139","reported_seq":901,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-08-19T14:24:15.273836+0000","last_change":"2024-08-19T14:24:14.035778+0000","last_active":"2024-08-19T14:24:15.273836+0000","last_peered":"2024-08-19T14:24:15.273836+0000","last_clean":"2024-08-19T14:24:15.273836+0000","last_became_active":"2024-08-19T14:18:18.703594+0000","last_became_peered":"2024-08-19T14:18:18.703594+0000","last_unstale":"2024-08-19T14:24:15.273836+0000","last_undegraded":"2024-08-19T14:24:15.273836+0000","last_fullsized":"2024-08-19T14:24:15.273836+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:14.035665+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:24:14.035665+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:25:35.876300+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"70'139","reported_seq":941,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-08-19T14:24:09.246196+0000","last_change":"2024-08-19T14:23:49.551005+0000","last_active":"2024-08-19T14:24:09.246196+0000","last_peered":"2024-08-19T14:24:09.246196+0000","last_clean":"2024-08-19T14:24:09.246196+0000","last_became_active":"2024-08-19T14:18:18.698334+0000","last_became_peered":"2024-08-19T14:18:18.698334+0000","last_unstale":"2024-08-19T14:24:09.246196+0000","last_undegraded":"2024-08-19T14:24:09.246196+0000","last_fullsized":"2024-08-19T14:24:09.246196+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:23:49.550697+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:23:49.550697+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:25:03.915358+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.10","version":"70'139","reported_seq":1056,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-08-19T14:24:15.269664+0000","last_change":"2024-08-19T14:23:46.008812+0000","last_active":"2024-08-19T14:24:15.269664+0000","last_peered":"2024-08-19T14:24:15.269664+0000","last_clean":"2024-08-19T14:24:15.269664+0000","last_became_active":"2024-08-19T14:18:18.704562+0000","last_became_peered":"2024-08-19T14:18:18.704562+0000","last_unstale":"2024-08-19T14:24:15.269664+0000","last_undegraded":"2024-08-19T14:24:15.269664+0000","last_fullsized":"2024-08-19T14:24:15.269664+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:23:46.008709+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:23:46.008709+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:24:50.614402+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,8],"acting":[5,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"70'139","reported_seq":605,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-08-19T14:24:15.270215+0000","last_change":"2024-08-19T14:23:48.687131+0000","last_active":"2024-08-19T14:24:15.270215+0000","last_peered":"2024-08-19T14:24:15.270215+0000","last_clean":"2024-08-19T14:24:15.270215+0000","last_became_active":"2024-08-19T14:18:18.697894+0000","last_became_peered":"2024-08-19T14:18:18.697894+0000","last_unstale":"2024-08-19T14:24:15.270215+0000","last_undegraded":"2024-08-19T14:24:15.270215+0000","last_fullsized":"2024-08-19T14:24:15.270215+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:23:48.687011+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:23:48.687011+0000","last_clean_scrub_stamp":"2024-08-19T14:23:48.687011+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:24:49.333312+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"70'139","reported_seq":570,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-08-19T14:24:09.265962+0000","last_change":"2024-08-19T14:23:21.381100+0000","last_active":"2024-08-19T14:24:09.265962+0000","last_peered":"2024-08-19T14:24:09.265962+0000","last_clean":"2024-08-19T14:24:09.265962+0000","last_became_active":"2024-08-19T14:20:28.204724+0000","last_became_peered":"2024-08-19T14:20:28.204724+0000","last_unstale":"2024-08-19T14:24:09.265962+0000","last_undegraded":"2024-08-19T14:24:09.265962+0000","last_fullsized":"2024-08-19T14:24:09.265962+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:23:21.380985+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:23:21.380985+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:24:35.720673+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"70'139","reported_seq":700,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-19T14:24:13.290172+0000","last_change":"2024-08-19T14:23:20.731091+0000","last_active":"2024-08-19T14:24:13.290172+0000","last_peered":"2024-08-19T14:24:13.290172+0000","last_clean":"2024-08-19T14:24:13.290172+0000","last_became_active":"2024-08-19T14:18:31.901556+0000","last_became_peered":"2024-08-19T14:18:31.901556+0000","last_unstale":"2024-08-19T14:24:13.290172+0000","last_undegraded":"2024-08-19T14:24:13.290172+0000","last_fullsized":"2024-08-19T14:24:13.290172+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:23:20.731018+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:23:20.731018+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:24:43.756077+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,0],"acting":[8,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.14","version":"70'139","reported_seq":701,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-19T14:24:13.263682+0000","last_change":"2024-08-19T14:23:15.864067+0000","last_active":"2024-08-19T14:24:13.263682+0000","last_peered":"2024-08-19T14:24:13.263682+0000","last_clean":"2024-08-19T14:24:13.263682+0000","last_became_active":"2024-08-19T14:18:18.703926+0000","last_became_peered":"2024-08-19T14:18:18.703926+0000","last_unstale":"2024-08-19T14:24:13.263682+0000","last_undegraded":"2024-08-19T14:24:13.263682+0000","last_fullsized":"2024-08-19T14:24:13.263682+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:23:15.863945+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:23:15.863945+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:24:21.930991+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,1],"acting":[7,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.15","version":"70'139","reported_seq":588,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-19T14:24:13.263637+0000","last_change":"2024-08-19T14:23:25.015912+0000","last_active":"2024-08-19T14:24:13.263637+0000","last_peered":"2024-08-19T14:24:13.263637+0000","last_clean":"2024-08-19T14:24:13.263637+0000","last_became_active":"2024-08-19T14:18:31.912496+0000","last_became_peered":"2024-08-19T14:18:31.912496+0000","last_unstale":"2024-08-19T14:24:13.263637+0000","last_undegraded":"2024-08-19T14:24:13.263637+0000","last_fullsized":"2024-08-19T14:24:13.263637+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:23:25.015801+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:20:09.180923+0000","last_clean_scrub_stamp":"2024-08-19T14:23:25.015801+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:24:33.214524+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"70'139","reported_seq":576,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-08-19T14:24:15.271285+0000","last_change":"2024-08-19T14:23:15.839549+0000","last_active":"2024-08-19T14:24:15.271285+0000","last_peered":"2024-08-19T14:24:15.271285+0000","last_clean":"2024-08-19T14:24:15.271285+0000","last_became_active":"2024-08-19T14:18:31.901229+0000","last_became_peered":"2024-08-19T14:18:31.901229+0000","last_unstale":"2024-08-19T14:24:15.271285+0000","last_undegraded":"2024-08-19T14:24:15.271285+0000","last_fullsized":"2024-08-19T14:24:15.271285+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:23:15.839423+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:23:15.839423+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:24:26.744739+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1b","version":"70'139","reported_seq":926,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-19T14:24:13.290030+0000","last_change":"2024-08-19T14:23:18.756861+0000","last_active":"2024-08-19T14:24:13.290030+0000","last_peered":"2024-08-19T14:24:13.290030+0000","last_clean":"2024-08-19T14:24:13.290030+0000","last_became_active":"2024-08-19T14:18:31.901152+0000","last_became_peered":"2024-08-19T14:18:31.901152+0000","last_unstale":"2024-08-19T14:24:13.290030+0000","last_undegraded":"2024-08-19T14:24:13.290030+0000","last_fullsized":"2024-08-19T14:24:13.290030+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:23:18.756745+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:23:18.756745+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:24:23.113444+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.18","version":"70'139","reported_seq":972,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-08-19T14:24:15.269553+0000","last_change":"2024-08-19T14:23:20.217662+0000","last_active":"2024-08-19T14:24:15.269553+0000","last_peered":"2024-08-19T14:24:15.269553+0000","last_clean":"2024-08-19T14:24:15.269553+0000","last_became_active":"2024-08-19T14:18:31.902735+0000","last_became_peered":"2024-08-19T14:18:31.902735+0000","last_unstale":"2024-08-19T14:24:15.269553+0000","last_undegraded":"2024-08-19T14:24:15.269553+0000","last_fullsized":"2024-08-19T14:24:15.269553+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:23:20.217551+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:23:20.217551+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:24:31.850512+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.19","version":"70'139","reported_seq":920,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-08-19T14:24:15.270104+0000","last_change":"2024-08-19T14:23:18.563553+0000","last_active":"2024-08-19T14:24:15.270104+0000","last_peered":"2024-08-19T14:24:15.270104+0000","last_clean":"2024-08-19T14:24:15.270104+0000","last_became_active":"2024-08-19T14:18:31.902547+0000","last_became_peered":"2024-08-19T14:18:31.902547+0000","last_unstale":"2024-08-19T14:24:15.270104+0000","last_undegraded":"2024-08-19T14:24:15.270104+0000","last_fullsized":"2024-08-19T14:24:15.270104+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:23:18.563420+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:23:18.563420+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:24:42.281542+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1e","version":"70'139","reported_seq":578,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-08-19T14:24:15.272257+0000","last_change":"2024-08-19T14:23:18.518254+0000","last_active":"2024-08-19T14:24:15.272257+0000","last_peered":"2024-08-19T14:24:15.272257+0000","last_clean":"2024-08-19T14:24:15.272257+0000","last_became_active":"2024-08-19T14:18:31.902546+0000","last_became_peered":"2024-08-19T14:18:31.902546+0000","last_unstale":"2024-08-19T14:24:15.272257+0000","last_undegraded":"2024-08-19T14:24:15.272257+0000","last_fullsized":"2024-08-19T14:24:15.272257+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:23:18.518183+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:23:18.518183+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:24:26.587350+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1c","version":"70'139","reported_seq":692,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-19T14:24:14.461271+0000","last_change":"2024-08-19T14:24:14.461312+0000","last_active":"2024-08-19T14:24:14.461271+0000","last_peered":"2024-08-19T14:24:14.461271+0000","last_clean":"2024-08-19T14:24:14.461271+0000","last_became_active":"2024-08-19T14:18:31.902494+0000","last_became_peered":"2024-08-19T14:18:31.902494+0000","last_unstale":"2024-08-19T14:24:14.461271+0000","last_undegraded":"2024-08-19T14:24:14.461271+0000","last_fullsized":"2024-08-19T14:24:14.461271+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:14.461176+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:24:14.461176+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:25:26.070165+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,11],"acting":[8,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1d","version":"70'139","reported_seq":596,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-08-19T14:24:09.246359+0000","last_change":"2024-08-19T14:23:20.375564+0000","last_active":"2024-08-19T14:24:09.246359+0000","last_peered":"2024-08-19T14:24:09.246359+0000","last_clean":"2024-08-19T14:24:09.246359+0000","last_became_active":"2024-08-19T14:18:31.901183+0000","last_became_peered":"2024-08-19T14:18:31.901183+0000","last_unstale":"2024-08-19T14:24:09.246359+0000","last_undegraded":"2024-08-19T14:24:09.246359+0000","last_fullsized":"2024-08-19T14:24:09.246359+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:23:20.375451+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:23:20.375451+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:24:21.979354+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,5],"acting":[1,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":933625880,"num_objects":14247,"num_object_clones":0,"num_object_copies":42741,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14247,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":248099,"num_write_kb":8395009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4140,"num_bytes_recovered":271319040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3008049152,"data_stored":3008036936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160899,"ondisk_log_size":160899,"up":48,"acting":48,"num_store_stats":9},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1802240,"data_stored":1788032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187611,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":849524,"kb_used_data":399080,"kb_used_omap":1,"kb_used_meta":450430,"kb_avail":92895628,"statfs":{"total":95995035648,"available":95125123072,"internally_reserved":0,"allocated":408657920,"data_stored":406794539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":461240772},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,1,1,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":343597383841,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":74,"seq":317827580068,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":776492,"kb_used_data":262176,"kb_used_omap":1,"kb_used_meta":514302,"kb_avail":92968660,"statfs":{"total":95995035648,"available":95199907840,"internally_reserved":0,"allocated":268468224,"data_stored":266605026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":526645697},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874410,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":437544,"kb_used_data":67612,"kb_used_omap":1,"kb_used_meta":369918,"kb_avail":93307608,"statfs":{"total":95995035648,"available":95546990592,"internally_reserved":0,"allocated":69234688,"data_stored":67375562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":378796485},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":57,"seq":244813136045,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1095328,"kb_used_data":526872,"kb_used_omap":1,"kb_used_meta":568446,"kb_avail":92649824,"statfs":{"total":95995035648,"available":94873419776,"internally_reserved":0,"allocated":539516928,"data_stored":537658455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":582089162},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":206158430384,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692021,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":662136,"kb_used_data":283888,"kb_used_omap":1,"kb_used_meta":378238,"kb_avail":93083016,"statfs":{"total":95995035648,"available":95317008384,"internally_reserved":0,"allocated":290701312,"data_stored":288834379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":387316165},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986362,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":761076,"kb_used_data":248172,"kb_used_omap":1,"kb_used_meta":512894,"kb_avail":92984076,"statfs":{"total":95995035648,"available":95215693824,"internally_reserved":0,"allocated":254128128,"data_stored":252260675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":525203909},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,0,5],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149887,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":846568,"kb_used_data":386268,"kb_used_omap":1,"kb_used_meta":460158,"kb_avail":92898584,"statfs":{"total":95995035648,"available":95128150016,"internally_reserved":0,"allocated":395538432,"data_stored":393679306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":471202247},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411525,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575048,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1266340,"kb_used_data":675548,"kb_used_omap":1,"kb_used_meta":590654,"kb_avail":92478812,"statfs":{"total":95995035648,"available":94698303488,"internally_reserved":0,"allocated":691761152,"data_stored":689898607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":604830149},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,1,3,1,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738573,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":620344,"kb_used_data":224040,"kb_used_omap":1,"kb_used_meta":396286,"kb_avail":93124808,"statfs":{"total":95995035648,"available":95359803392,"internally_reserved":0,"allocated":229416960,"data_stored":227553579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":405797317},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":209321984,"data_stored":209321984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":664866816,"data_stored":664862744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":383582208,"data_stored":383582208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":244846592,"data_stored":244842520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":278331392,"data_stored":278331392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":519831552,"data_stored":519831552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":62521344,"data_stored":62521344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":261754880,"data_stored":261750808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":382992384,"data_stored":382992384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T14:24:19.417 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-08-19T14:24:19.418 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd pool set .mgr pg_num 51 2024-08-19T14:24:19.427 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:24:19.449 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:19.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:19.450 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:24:19.461 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:19 smithi031 ceph-mon[16336]: pgmap v939: 47 pgs: 47 active+clean; 891 MiB data, 7.0 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 890 op/s 2024-08-19T14:24:19.461 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:19 smithi031 ceph-mon[16336]: from='client.25735 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:24:19.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:19 smithi045 ceph-mon[20143]: pgmap v939: 47 pgs: 47 active+clean; 891 MiB data, 7.0 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 890 op/s 2024-08-19T14:24:19.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:19 smithi045 ceph-mon[20143]: from='client.25735 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:24:19.552 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:24:19.579 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:19.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:19.580 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:24:19.681 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:24:19.708 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:19.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:19.709 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:24:19.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:19 smithi032 ceph-mon[20004]: pgmap v939: 47 pgs: 47 active+clean; 891 MiB data, 7.0 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 890 op/s 2024-08-19T14:24:19.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:19 smithi032 ceph-mon[20004]: from='client.25735 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:24:19.809 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:24:19.832 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:19.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:19.833 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:24:19.934 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:24:19.955 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:24:19.956 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:24:19.957 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:19.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:19.959 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:24:19.999 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:24:20.000 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:24:20.030 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:24:20.030 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:24:20.060 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:24:20.078 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:24:20.078 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:24:20.081 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:20.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:20.082 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:24:20.103 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:24:20.104 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:24:20.122 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:24:20.122 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:24:20.144 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:24:20.145 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:24:20.167 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:24:20.167 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:24:20.182 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:24:20.197 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:24:20.197 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:24:20.206 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:20.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:20.207 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:24:20.227 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:24:20.227 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:24:20.268 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:24:20.268 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:24:20.304 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:24:20.304 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:24:20.308 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:24:20.352 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:20.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:20.353 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:24:20.454 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:24:20.478 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:20.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:20.479 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:24:20.580 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:24:20.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:20.608 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:20.608 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:24:20.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:20 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:24:20.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:20 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:24:20.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:20 smithi031 ceph-mon[16336]: osdmap e408: 12 total, 12 up, 9 in 2024-08-19T14:24:20.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:20 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3199217773' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:24:20.658 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:20 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:24:20.658 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:20 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:24:20.658 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:20 smithi032 ceph-mon[20004]: osdmap e408: 12 total, 12 up, 9 in 2024-08-19T14:24:20.658 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:20 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3199217773' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:24:20.709 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:24:20.742 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:20.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:20.743 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:24:20.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:20 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:24:20.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:20 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:24:20.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:20 smithi045 ceph-mon[20143]: osdmap e408: 12 total, 12 up, 9 in 2024-08-19T14:24:20.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:20 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3199217773' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:24:20.843 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:24:20.867 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:20.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:20.868 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:24:20.969 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:24:20.995 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:20.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:20.996 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:24:21.097 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:24:21.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:21.127 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:21.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:24:21.228 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:24:21.258 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:21.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:21.259 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:24:21.360 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:24:21.389 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:21.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:21.390 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:24:21.491 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:24:21.520 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:21.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:21.521 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:24:21.574 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:21 smithi031 ceph-mon[16336]: pgmap v941: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 765 MiB data, 6.6 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 830 op/s 2024-08-19T14:24:21.575 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-08-19T14:24:21.621 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:24:21.646 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:21.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:21.647 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:24:21.748 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:24:21.815 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:21 smithi032 ceph-mon[20004]: pgmap v941: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 765 MiB data, 6.6 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 830 op/s 2024-08-19T14:24:21.816 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:21 smithi045 ceph-mon[20143]: pgmap v941: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 765 MiB data, 6.6 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 830 op/s 2024-08-19T14:24:21.844 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:21.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:21.845 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:24:21.910 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:24:21.946 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:24:21.969 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:21.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:21.970 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:24:22.071 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:24:22.095 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:22.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:22.096 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:24:22.197 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:24:22.237 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:22.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:22.238 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:24:22.339 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:24:22.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:22.367 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:22.367 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:24:22.469 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:24:22.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:22.503 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:22.503 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:24:22.604 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:24:22.627 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:22.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:22.628 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:24:22.730 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:24:22.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:22.763 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:22.764 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:24:22.865 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:24:22.891 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:22.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:22.892 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:24:22.993 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:24:23.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:23.025 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:23.026 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:24:23.101 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:24:23.106 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:24:23.126 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:24:23.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:23.185 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:23.185 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:24:23.286 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:24:23.309 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:23.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:23.310 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:24:23.411 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:24:23.433 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:23.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:23.434 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:24:23.530 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:23 smithi031 ceph-mon[16336]: pgmap v942: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 674 MiB data, 6.3 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 769 op/s 2024-08-19T14:24:23.530 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:23 smithi031 ceph-mon[16336]: 4.c scrub starts 2024-08-19T14:24:23.530 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:23 smithi031 ceph-mon[16336]: 1.1d scrub starts 2024-08-19T14:24:23.530 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:23 smithi031 ceph-mon[16336]: 1.1d scrub ok 2024-08-19T14:24:23.530 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:23 smithi031 ceph-mon[16336]: 1.14 scrub starts 2024-08-19T14:24:23.530 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:23 smithi031 ceph-mon[16336]: 1.14 scrub ok 2024-08-19T14:24:23.535 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:24:23.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:23.560 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:23.561 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:24:23.662 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:24:23.684 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:23.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:23.685 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:24:23.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:23 smithi032 ceph-mon[20004]: pgmap v942: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 674 MiB data, 6.3 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 769 op/s 2024-08-19T14:24:23.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:23 smithi032 ceph-mon[20004]: 4.c scrub starts 2024-08-19T14:24:23.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:23 smithi032 ceph-mon[20004]: 1.1d scrub starts 2024-08-19T14:24:23.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:23 smithi032 ceph-mon[20004]: 1.1d scrub ok 2024-08-19T14:24:23.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:23 smithi032 ceph-mon[20004]: 1.14 scrub starts 2024-08-19T14:24:23.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:23 smithi032 ceph-mon[20004]: 1.14 scrub ok 2024-08-19T14:24:23.786 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:24:23.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:23 smithi045 ceph-mon[20143]: pgmap v942: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 674 MiB data, 6.3 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 769 op/s 2024-08-19T14:24:23.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:23 smithi045 ceph-mon[20143]: 4.c scrub starts 2024-08-19T14:24:23.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:23 smithi045 ceph-mon[20143]: 1.1d scrub starts 2024-08-19T14:24:23.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:23 smithi045 ceph-mon[20143]: 1.1d scrub ok 2024-08-19T14:24:23.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:23 smithi045 ceph-mon[20143]: 1.14 scrub starts 2024-08-19T14:24:23.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:23 smithi045 ceph-mon[20143]: 1.14 scrub ok 2024-08-19T14:24:23.819 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:23.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:23.821 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:24:23.921 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:24:23.964 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:23.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:23.965 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:24:23.967 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:24:23.967 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:24:24.066 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:24:24.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:24.088 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:24.089 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:24:24.189 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:24:24.224 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:24.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:24.225 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:24:24.326 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:24:24.328 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is set 2024-08-19T14:24:24.353 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:24.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:24.354 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:24:24.455 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:24:24.479 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:24.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:24.480 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:24:24.508 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:24:24.532 INFO:teuthology.orchestra.run.smithi031.stderr:Error EINVAL: specified pg_num 51 < pg_num_max 32 2024-08-19T14:24:24.581 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:24:24.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:24 smithi031 ceph-mon[16336]: 4.c scrub ok 2024-08-19T14:24:24.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:24 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1674282014' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:24:24.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:24 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:24:24.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:24.625 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:24.625 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:24:24.726 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:24:24.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:24 smithi032 ceph-mon[20004]: 4.c scrub ok 2024-08-19T14:24:24.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:24 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1674282014' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:24:24.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:24 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:24:24.765 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:24.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:24.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:24:24.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:24 smithi045 ceph-mon[20143]: 4.c scrub ok 2024-08-19T14:24:24.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:24 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1674282014' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:24:24.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:24 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:24:24.867 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:24:24.895 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:24.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:24.896 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:24:24.914 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-08-19T14:24:24.997 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:24:25.026 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:25.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:25.027 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:24:25.128 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:24:25.166 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:25.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:25.167 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:24:25.268 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:24:25.295 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:25.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:25.296 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:24:25.397 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:24:25.433 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:25.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:25.434 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:24:25.535 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:24:25.565 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:25.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:25.566 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:24:25.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:25 smithi031 ceph-mon[16336]: pgmap v943: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 531 MiB data, 5.9 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 971 op/s 2024-08-19T14:24:25.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:25 smithi031 ceph-mon[16336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:24:25.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:25 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:24:25.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:25 smithi031 ceph-mon[16336]: osdmap e409: 12 total, 12 up, 9 in 2024-08-19T14:24:25.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:25 smithi031 ceph-mon[16336]: 4.f deep-scrub ok 2024-08-19T14:24:25.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:25 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/982599227' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "51"}]: dispatch 2024-08-19T14:24:25.667 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:24:25.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:25.692 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:25.693 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:24:25.697 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:25 smithi045 ceph-mon[20143]: pgmap v943: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 531 MiB data, 5.9 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 971 op/s 2024-08-19T14:24:25.697 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:25 smithi045 ceph-mon[20143]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:24:25.697 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:25 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:24:25.697 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:25 smithi045 ceph-mon[20143]: osdmap e409: 12 total, 12 up, 9 in 2024-08-19T14:24:25.697 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:25 smithi045 ceph-mon[20143]: 4.f deep-scrub ok 2024-08-19T14:24:25.697 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:25 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/982599227' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "51"}]: dispatch 2024-08-19T14:24:25.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:25 smithi032 ceph-mon[20004]: pgmap v943: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 531 MiB data, 5.9 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 971 op/s 2024-08-19T14:24:25.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:25 smithi032 ceph-mon[20004]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:24:25.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:25 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:24:25.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:25 smithi032 ceph-mon[20004]: osdmap e409: 12 total, 12 up, 9 in 2024-08-19T14:24:25.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:25 smithi032 ceph-mon[20004]: 4.f deep-scrub ok 2024-08-19T14:24:25.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:25 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/982599227' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "51"}]: dispatch 2024-08-19T14:24:25.793 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:24:25.818 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:25.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:25.819 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:24:25.920 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:24:25.949 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:25.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:25.950 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:24:26.051 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:24:26.078 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:26.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:26.079 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:24:26.180 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:24:26.202 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:26.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:26.203 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:24:26.304 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:24:26.336 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:26.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:26.337 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:24:26.438 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:24:26.470 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:26.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:26.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:24:26.571 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:24:26.601 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:26.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:26.602 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:24:26.703 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:24:26.725 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:26.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:26.726 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:24:26.827 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:24:26.852 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:26.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:26.853 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:24:26.954 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:24:26.982 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:26.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:26.983 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:24:27.085 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:24:27.116 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:27.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:27.117 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:24:27.218 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:24:27.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:27.252 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:27.252 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:24:27.353 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:24:27.385 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:27.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:27.386 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:24:27.487 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:24:27.512 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:27.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:27.513 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:24:27.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:27 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:24:27.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:27 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:24:27.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:27 smithi045 ceph-mon[20143]: pgmap v945: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 531 MiB data, 5.9 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 719 op/s 2024-08-19T14:24:27.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:27 smithi045 ceph-mon[20143]: osdmap e410: 12 total, 12 up, 9 in 2024-08-19T14:24:27.614 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:24:27.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:27 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:24:27.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:27 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:24:27.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:27 smithi031 ceph-mon[16336]: pgmap v945: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 531 MiB data, 5.9 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 719 op/s 2024-08-19T14:24:27.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:27 smithi031 ceph-mon[16336]: osdmap e410: 12 total, 12 up, 9 in 2024-08-19T14:24:27.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:27.643 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:27.644 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:24:27.745 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:24:27.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:27 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:24:27.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:27 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:24:27.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:27 smithi032 ceph-mon[20004]: pgmap v945: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 531 MiB data, 5.9 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 719 op/s 2024-08-19T14:24:27.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:27 smithi032 ceph-mon[20004]: osdmap e410: 12 total, 12 up, 9 in 2024-08-19T14:24:27.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:27.782 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:27.782 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:24:27.883 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:24:27.915 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:27.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:27.916 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:24:28.017 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:24:28.043 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:28.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:28.044 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:24:28.145 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:24:28.203 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:28.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:28.204 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:24:28.305 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:24:28.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:28.339 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:28.339 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:24:28.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:28 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:24:28.440 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:24:28.467 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:28.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:28.468 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:24:28.569 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:24:28.594 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:28.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:28.596 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:24:28.697 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:24:28.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:28.723 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:28.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:24:28.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:28 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:24:28.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:28 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:24:28.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:28 smithi032 ceph-mon[20004]: osdmap e411: 12 total, 12 up, 9 in 2024-08-19T14:24:28.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:28 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:24:28.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:28 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:24:28.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:28 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:24:28.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:28 smithi045 ceph-mon[20143]: osdmap e411: 12 total, 12 up, 9 in 2024-08-19T14:24:28.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:28 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:24:28.825 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:24:28.852 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:28.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:28.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:24:28.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:28 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:24:28.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:28 smithi031 ceph-mon[16336]: osdmap e411: 12 total, 12 up, 9 in 2024-08-19T14:24:28.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:28 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:24:28.926 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-19T14:24:28.949 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:24:28.954 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:24:28.963 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:24:28.987 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-19T14:24:29.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:29.035 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:29.036 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:24:29.138 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:24:29.174 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:29.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:29.175 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:24:29.277 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:24:29.301 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:29.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:29.303 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:24:29.404 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:24:29.440 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:29.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:29.442 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:24:29.515 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:29 smithi031 ceph-mon[16336]: pgmap v948: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 387 MiB data, 5.5 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 764 op/s 2024-08-19T14:24:29.515 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:29 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:24:29.515 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:29 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:24:29.515 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:29 smithi031 ceph-mon[16336]: osdmap e412: 12 total, 12 up, 9 in 2024-08-19T14:24:29.515 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:29 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:24:29.515 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:29 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:24:29.515 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:29 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:24:29.543 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:24:29.568 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:29.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:29.569 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:24:29.670 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:24:29.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:29.712 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:29.712 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:24:29.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:29 smithi032 ceph-mon[20004]: pgmap v948: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 387 MiB data, 5.5 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 764 op/s 2024-08-19T14:24:29.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:29 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:24:29.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:29 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:24:29.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:29 smithi032 ceph-mon[20004]: osdmap e412: 12 total, 12 up, 9 in 2024-08-19T14:24:29.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:29 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:24:29.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:29 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:24:29.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:29 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:24:29.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:29 smithi045 ceph-mon[20143]: pgmap v948: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 387 MiB data, 5.5 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 764 op/s 2024-08-19T14:24:29.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:29 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:24:29.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:29 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:24:29.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:29 smithi045 ceph-mon[20143]: osdmap e412: 12 total, 12 up, 9 in 2024-08-19T14:24:29.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:29 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:24:29.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:29 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:24:29.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:29 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:24:29.813 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:24:29.842 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:24:29.843 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:24:29.844 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:29.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:29.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:24:29.880 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:24:29.880 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:24:29.921 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:24:29.921 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:24:29.947 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:24:29.970 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:29.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:29.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:24:30.063 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:24:30.063 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:24:30.066 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:24:30.072 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:24:30.094 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:30.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:30.095 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:24:30.170 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:24:30.170 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:24:30.196 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:24:30.208 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:24:30.208 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:24:30.244 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:30.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:30.245 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:24:30.248 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:24:30.249 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:24:30.282 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:24:30.282 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:24:30.312 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:24:30.313 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:24:30.345 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:24:30.346 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:24:30.346 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:24:30.379 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:30.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:30.380 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:24:30.382 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:24:30.382 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:24:30.416 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:24:30.416 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:24:30.446 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:24:30.447 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:24:30.447 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:30 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3901865635' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:24:30.481 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:24:30.523 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:30.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:30.524 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:24:30.625 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:24:30.652 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:30.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:30.653 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:24:30.754 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:24:30.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:30 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3901865635' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:24:30.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:30.787 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:30.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:24:30.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:30 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3901865635' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:24:30.844 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-08-19T14:24:30.888 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:24:30.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:30.935 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:30.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:24:31.036 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:24:31.067 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:31.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:31.068 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:24:31.169 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:24:31.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:31.206 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:31.207 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:24:31.308 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:24:31.333 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:31.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:31.334 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:24:31.435 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:24:31.459 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:31.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:31.460 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:24:31.562 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:24:31.586 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:31.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:31.588 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:24:31.688 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:24:31.716 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:31.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:31.717 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:24:31.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:31 smithi032 ceph-mon[20004]: pgmap v950: 47 pgs: 47 active+clean; 270 MiB data, 5.1 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 742 op/s 2024-08-19T14:24:31.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:31 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:24:31.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:31 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:24:31.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:31 smithi045 ceph-mon[20143]: pgmap v950: 47 pgs: 47 active+clean; 270 MiB data, 5.1 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 742 op/s 2024-08-19T14:24:31.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:31 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:24:31.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:31 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:24:31.818 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:24:31.843 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:31.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:31.844 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:24:31.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:31 smithi031 ceph-mon[16336]: pgmap v950: 47 pgs: 47 active+clean; 270 MiB data, 5.1 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 742 op/s 2024-08-19T14:24:31.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:31 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:24:31.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:31 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:24:31.944 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:24:31.965 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:31.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:31.966 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:24:32.067 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:24:32.092 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:32.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:32.094 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:24:32.195 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:24:32.232 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:32.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:32.233 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:24:32.334 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:24:32.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:32.364 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:32.364 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:24:32.389 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is set 2024-08-19T14:24:32.462 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Removed 131172 objects 2024-08-19T14:24:32.462 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Clean up completed and total clean up time :147.403 2024-08-19T14:24:32.465 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:24:32.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:32.486 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:32.487 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:24:32.587 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:24:32.609 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:32.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:32.610 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:24:32.614 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:32 smithi031 ceph-mon[16336]: osdmap e413: 12 total, 12 up, 9 in 2024-08-19T14:24:32.614 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:32 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/957270518' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:24:32.614 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:32 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:24:32.710 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:24:32.731 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:32.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:32.732 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:24:32.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:32 smithi032 ceph-mon[20004]: osdmap e413: 12 total, 12 up, 9 in 2024-08-19T14:24:32.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:32 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/957270518' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:24:32.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:32 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:24:32.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:32 smithi045 ceph-mon[20143]: osdmap e413: 12 total, 12 up, 9 in 2024-08-19T14:24:32.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:32 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/957270518' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:24:32.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:32 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:24:32.833 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:24:32.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:32.856 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:32.856 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:24:32.957 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:24:32.982 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:32.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:32.987 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:24:33.088 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:24:33.114 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:33.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:33.115 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:24:33.214 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:24:33.216 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:24:33.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:33.241 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:33.242 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:24:33.343 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:24:33.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:33.365 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:33.365 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:24:33.466 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:33 smithi031 ceph-mon[16336]: pgmap v952: 47 pgs: 47 active+clean; 164 MiB data, 4.8 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 820 op/s 2024-08-19T14:24:33.466 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:33 smithi031 ceph-mon[16336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:24:33.466 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:33 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:24:33.466 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:33 smithi031 ceph-mon[16336]: osdmap e414: 12 total, 12 up, 9 in 2024-08-19T14:24:33.466 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:24:33.493 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:33.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:33.494 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:24:33.595 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:24:33.620 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:33.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:33.621 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:24:33.704 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:24:33.704 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-08-19T14:24:33.705 INFO:teuthology.orchestra.run.smithi031.stdout: "complaint_time": 30, 2024-08-19T14:24:33.705 INFO:teuthology.orchestra.run.smithi031.stdout: "num_blocked_ops": 0 2024-08-19T14:24:33.705 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:24:33.722 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:24:33.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:33.748 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:33.749 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:24:33.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:33 smithi032 ceph-mon[20004]: pgmap v952: 47 pgs: 47 active+clean; 164 MiB data, 4.8 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 820 op/s 2024-08-19T14:24:33.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:33 smithi032 ceph-mon[20004]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:24:33.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:33 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:24:33.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:33 smithi032 ceph-mon[20004]: osdmap e414: 12 total, 12 up, 9 in 2024-08-19T14:24:33.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:33 smithi045 ceph-mon[20143]: pgmap v952: 47 pgs: 47 active+clean; 164 MiB data, 4.8 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 820 op/s 2024-08-19T14:24:33.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:33 smithi045 ceph-mon[20143]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:24:33.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:33 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:24:33.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:33 smithi045 ceph-mon[20143]: osdmap e414: 12 total, 12 up, 9 in 2024-08-19T14:24:33.849 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:24:33.875 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:33.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:33.876 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:24:33.927 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 7, 8, 9, 11] out_osds: [10, 6, 2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-08-19T14:24:33.928 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd reweight 0 1 2024-08-19T14:24:33.977 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:24:34.002 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:34.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:34.003 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:24:34.104 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:24:34.130 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:34.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:34.131 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:24:34.232 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:24:34.257 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:34.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:34.258 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:24:34.343 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-08-19T14:24:34.359 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:24:34.385 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:34.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:34.386 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:24:34.418 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:24:34.487 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:24:34.508 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:34.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:34.509 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:24:34.610 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:24:34.631 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:34.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:34.632 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:24:34.643 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:24:34.734 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:24:34.755 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:34.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:34.757 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:24:34.859 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:24:34.880 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:34.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:34.882 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:24:34.950 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:24:34.983 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:24:34.991 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:24:35.008 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:35.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:35.009 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:24:35.110 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:24:35.134 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:35.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:35.135 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:24:35.213 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-19T14:24:35.236 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:24:35.262 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:35.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:35.263 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:24:35.364 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:24:35.389 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:35.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:35.390 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:24:35.391 INFO:teuthology.orchestra.run.smithi031.stderr:reweighted osd.0 to 1 (10000) 2024-08-19T14:24:35.491 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:24:35.511 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:35.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:35.512 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:24:35.613 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:24:35.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:35 smithi031 ceph-mon[16336]: pgmap v954: 47 pgs: 47 active+clean; 24 MiB data, 4.4 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 969 op/s 2024-08-19T14:24:35.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:35 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:24:35.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:35 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:24:35.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:35 smithi031 ceph-mon[16336]: osdmap e415: 12 total, 12 up, 9 in 2024-08-19T14:24:35.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:35 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/862714147' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-08-19T14:24:35.634 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:35.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:35.635 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:24:35.736 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:24:35.757 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:35.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:35 smithi032 ceph-mon[20004]: pgmap v954: 47 pgs: 47 active+clean; 24 MiB data, 4.4 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 969 op/s 2024-08-19T14:24:35.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:35 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:24:35.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:35 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:24:35.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:35 smithi032 ceph-mon[20004]: osdmap e415: 12 total, 12 up, 9 in 2024-08-19T14:24:35.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:35 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/862714147' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-08-19T14:24:35.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:35.759 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:24:35.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:35 smithi045 ceph-mon[20143]: pgmap v954: 47 pgs: 47 active+clean; 24 MiB data, 4.4 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 969 op/s 2024-08-19T14:24:35.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:35 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:24:35.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:35 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:24:35.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:35 smithi045 ceph-mon[20143]: osdmap e415: 12 total, 12 up, 9 in 2024-08-19T14:24:35.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:35 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/862714147' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-08-19T14:24:35.860 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:24:35.886 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:35.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:35.887 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:24:35.988 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:24:36.015 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:36.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:36.016 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:24:36.083 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd reweight 1 1 2024-08-19T14:24:36.117 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:24:36.146 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:36.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:36.147 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:24:36.248 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:24:36.274 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:36.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:36.276 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:24:36.291 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:24:36.291 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 20, 2024-08-19T14:24:36.291 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 600, 2024-08-19T14:24:36.291 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [ 2024-08-19T14:24:36.291 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.291 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25057.0:63310 3.1b 3:de62eaa4:::benchmark_data_smithi045_59973_object63309:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e244)", 2024-08-19T14:24:36.291 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:17:33.888643+0000", 2024-08-19T14:24:36.291 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 422.40028929599998, 2024-08-19T14:24:36.292 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.11976188, 2024-08-19T14:24:36.292 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:24:36.292 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:24:36.292 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:24:36.292 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:24:36.292 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25057", 2024-08-19T14:24:36.292 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:24:36.292 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 63310 2024-08-19T14:24:36.292 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.292 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:24:36.292 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.292 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:24:36.293 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:33.888643+0000", 2024-08-19T14:24:36.293 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:24:36.293 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.293 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.293 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:24:36.293 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:33.888643+0000", 2024-08-19T14:24:36.293 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:24:36.293 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.293 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.293 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:24:36.293 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:33.888648+0000", 2024-08-19T14:24:36.293 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4959999999999998e-06 2024-08-19T14:24:36.293 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.294 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.294 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:24:36.294 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:33.888850+0000", 2024-08-19T14:24:36.294 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00020193199999999999 2024-08-19T14:24:36.294 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.294 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.294 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:24:36.294 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:33.888866+0000", 2024-08-19T14:24:36.295 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6195e-05 2024-08-19T14:24:36.295 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.295 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.295 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:24:36.295 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:33.888873+0000", 2024-08-19T14:24:36.295 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.3669999999999999e-06 2024-08-19T14:24:36.295 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.295 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.295 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:24:36.296 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:33.975006+0000", 2024-08-19T14:24:36.296 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.086132499000000001 2024-08-19T14:24:36.296 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.296 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.296 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:24:36.296 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:33.975015+0000", 2024-08-19T14:24:36.296 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.0690000000000008e-06 2024-08-19T14:24:36.296 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.296 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.296 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:24:36.296 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:35.008332+0000", 2024-08-19T14:24:36.296 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0333170140000001 2024-08-19T14:24:36.296 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.297 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.297 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:24:36.297 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:35.008405+0000", 2024-08-19T14:24:36.297 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.3307999999999997e-05 2024-08-19T14:24:36.297 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:24:36.297 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:24:36.297 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:24:36.297 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.297 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.297 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25057.0:63337 3.1b 3:da914815:::benchmark_data_smithi045_59973_object63336:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e244)", 2024-08-19T14:24:36.297 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:17:33.958921+0000", 2024-08-19T14:24:36.297 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 422.33001141699998, 2024-08-19T14:24:36.297 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0657095919999999, 2024-08-19T14:24:36.298 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:24:36.298 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:24:36.298 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:24:36.298 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:24:36.298 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25057", 2024-08-19T14:24:36.298 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:24:36.298 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 63337 2024-08-19T14:24:36.298 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.298 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:24:36.298 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.298 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:24:36.298 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:33.958921+0000", 2024-08-19T14:24:36.298 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:24:36.299 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.299 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.299 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:24:36.299 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:33.958921+0000", 2024-08-19T14:24:36.299 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:24:36.299 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.299 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.299 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:24:36.299 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:33.958923+0000", 2024-08-19T14:24:36.299 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.1629999999999999e-06 2024-08-19T14:24:36.299 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.299 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.299 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:24:36.299 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:33.958968+0000", 2024-08-19T14:24:36.300 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4407999999999999e-05 2024-08-19T14:24:36.300 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.300 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.300 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:24:36.300 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:33.958990+0000", 2024-08-19T14:24:36.300 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.1851000000000001e-05 2024-08-19T14:24:36.300 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.300 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.300 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:24:36.300 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:33.958999+0000", 2024-08-19T14:24:36.300 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.7270000000000002e-06 2024-08-19T14:24:36.300 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.300 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.301 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:24:36.301 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:33.975034+0000", 2024-08-19T14:24:36.301 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.016035089999999998 2024-08-19T14:24:36.301 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.301 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.301 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:24:36.301 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:33.975040+0000", 2024-08-19T14:24:36.301 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.2900000000000002e-06 2024-08-19T14:24:36.301 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.301 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.301 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:24:36.301 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:35.008794+0000", 2024-08-19T14:24:36.301 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.033754399 2024-08-19T14:24:36.301 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.302 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.302 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:24:36.302 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:35.008913+0000", 2024-08-19T14:24:36.302 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000119225 2024-08-19T14:24:36.302 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.302 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.302 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,4", 2024-08-19T14:24:36.302 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:35.009041+0000", 2024-08-19T14:24:36.302 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000127618 2024-08-19T14:24:36.302 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.302 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.302 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:24:36.302 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:35.010327+0000", 2024-08-19T14:24:36.303 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0012860160000000001 2024-08-19T14:24:36.303 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.303 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.303 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:24:36.303 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:35.013480+0000", 2024-08-19T14:24:36.303 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0031529219999999998 2024-08-19T14:24:36.303 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.303 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.303 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:24:36.303 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:35.024451+0000", 2024-08-19T14:24:36.303 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.010971353999999999 2024-08-19T14:24:36.303 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.303 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.304 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:24:36.304 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:35.024584+0000", 2024-08-19T14:24:36.304 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000133127 2024-08-19T14:24:36.304 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.304 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.304 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:24:36.304 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:35.024631+0000", 2024-08-19T14:24:36.304 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.6402000000000001e-05 2024-08-19T14:24:36.304 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:24:36.304 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:24:36.304 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:24:36.304 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.304 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.304 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25057.0:188722 3.27 3:e4cb8e0c:::benchmark_data_smithi045_59973_object57543:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e296)", 2024-08-19T14:24:36.305 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:19:02.154320+0000", 2024-08-19T14:24:36.305 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 334.13461299199997, 2024-08-19T14:24:36.305 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0480230699999999, 2024-08-19T14:24:36.305 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:24:36.305 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:24:36.305 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:24:36.305 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:24:36.305 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25057", 2024-08-19T14:24:36.306 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:24:36.306 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 188722 2024-08-19T14:24:36.306 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.306 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:24:36.306 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.306 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:24:36.306 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.154320+0000", 2024-08-19T14:24:36.306 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:24:36.306 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.306 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.306 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:24:36.306 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.154320+0000", 2024-08-19T14:24:36.307 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:24:36.307 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.307 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.307 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:24:36.307 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.154321+0000", 2024-08-19T14:24:36.307 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.127e-06 2024-08-19T14:24:36.307 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.307 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.307 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:24:36.307 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.154328+0000", 2024-08-19T14:24:36.307 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.542e-06 2024-08-19T14:24:36.307 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.308 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.308 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:24:36.308 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.154340+0000", 2024-08-19T14:24:36.308 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1946999999999999e-05 2024-08-19T14:24:36.308 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.308 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.308 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:24:36.308 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.154348+0000", 2024-08-19T14:24:36.308 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.2490000000000001e-06 2024-08-19T14:24:36.308 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.308 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.308 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:24:36.309 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.286665+0000", 2024-08-19T14:24:36.309 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.13231673599999999 2024-08-19T14:24:36.309 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.309 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.309 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:24:36.309 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.286695+0000", 2024-08-19T14:24:36.309 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.0270999999999999e-05 2024-08-19T14:24:36.309 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.309 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.309 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:24:36.309 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:03.202268+0000", 2024-08-19T14:24:36.309 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.91557253000000005 2024-08-19T14:24:36.309 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.309 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.310 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:24:36.310 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:03.202343+0000", 2024-08-19T14:24:36.310 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.4667999999999997e-05 2024-08-19T14:24:36.310 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:24:36.310 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:24:36.310 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:24:36.310 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.310 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.310 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25057.0:188762 3.27 3:e4fc7775:::benchmark_data_smithi045_59973_object57583:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-08-19T14:24:36.310 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:19:02.183993+0000", 2024-08-19T14:24:36.310 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 334.10493999599998, 2024-08-19T14:24:36.310 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0247527810000001, 2024-08-19T14:24:36.310 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:24:36.311 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:24:36.311 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:24:36.311 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:24:36.311 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25057", 2024-08-19T14:24:36.311 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:24:36.311 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 188762 2024-08-19T14:24:36.311 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.311 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:24:36.311 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.311 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:24:36.311 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.183993+0000", 2024-08-19T14:24:36.311 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:24:36.311 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.311 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.312 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:24:36.312 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.183993+0000", 2024-08-19T14:24:36.312 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:24:36.312 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.312 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.312 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:24:36.312 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.183995+0000", 2024-08-19T14:24:36.312 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.0609999999999998e-06 2024-08-19T14:24:36.312 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.312 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.312 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:24:36.312 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.184003+0000", 2024-08-19T14:24:36.313 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.3939999999999996e-06 2024-08-19T14:24:36.313 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.313 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.313 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:24:36.313 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.184019+0000", 2024-08-19T14:24:36.313 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6030000000000001e-05 2024-08-19T14:24:36.313 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.313 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.313 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:24:36.313 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.184028+0000", 2024-08-19T14:24:36.313 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.3579999999999992e-06 2024-08-19T14:24:36.313 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.313 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.313 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:24:36.314 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.286800+0000", 2024-08-19T14:24:36.314 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.10277198799999999 2024-08-19T14:24:36.314 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.314 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.314 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:24:36.315 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.286823+0000", 2024-08-19T14:24:36.315 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.3050000000000001e-05 2024-08-19T14:24:36.315 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.315 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.315 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:24:36.315 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:03.202680+0000", 2024-08-19T14:24:36.315 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.91585707800000005 2024-08-19T14:24:36.315 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.315 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.315 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:24:36.315 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:03.202789+0000", 2024-08-19T14:24:36.315 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000108059 2024-08-19T14:24:36.315 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.316 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.316 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 7,11", 2024-08-19T14:24:36.316 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:03.202904+0000", 2024-08-19T14:24:36.316 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00011581100000000001 2024-08-19T14:24:36.316 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.316 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.316 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:24:36.316 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:03.204648+0000", 2024-08-19T14:24:36.316 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.001744076 2024-08-19T14:24:36.316 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.316 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.316 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:24:36.316 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:03.205732+0000", 2024-08-19T14:24:36.316 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0010840190000000001 2024-08-19T14:24:36.317 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.317 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.317 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:24:36.317 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:03.208610+0000", 2024-08-19T14:24:36.317 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.002877424 2024-08-19T14:24:36.317 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.317 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.317 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:24:36.317 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:03.208699+0000", 2024-08-19T14:24:36.317 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.9506000000000007e-05 2024-08-19T14:24:36.317 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.317 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.317 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:24:36.318 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:03.208745+0000", 2024-08-19T14:24:36.318 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.5927000000000003e-05 2024-08-19T14:24:36.318 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:24:36.318 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:24:36.318 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:24:36.318 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.318 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.318 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25057.0:216925 3.25 3:a76b9d69:::benchmark_data_smithi045_59973_object85746:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e307)", 2024-08-19T14:24:36.318 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:19:34.561921+0000", 2024-08-19T14:24:36.318 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 301.727011656, 2024-08-19T14:24:36.318 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.208080195, 2024-08-19T14:24:36.318 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:24:36.319 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:24:36.319 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:24:36.319 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:24:36.319 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25057", 2024-08-19T14:24:36.319 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:24:36.319 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 216925 2024-08-19T14:24:36.319 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.319 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:24:36.319 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.319 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:24:36.319 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:34.561921+0000", 2024-08-19T14:24:36.319 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:24:36.319 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.320 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.320 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:24:36.320 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:34.561921+0000", 2024-08-19T14:24:36.320 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:24:36.320 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.320 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.320 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:24:36.320 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:34.561922+0000", 2024-08-19T14:24:36.320 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4589999999999999e-06 2024-08-19T14:24:36.320 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.320 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.320 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:24:36.320 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:34.561926+0000", 2024-08-19T14:24:36.321 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.799e-06 2024-08-19T14:24:36.321 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.321 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.321 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:24:36.321 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:34.561935+0000", 2024-08-19T14:24:36.321 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.217e-06 2024-08-19T14:24:36.321 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.321 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.321 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:24:36.321 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:34.561941+0000", 2024-08-19T14:24:36.321 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.8130000000000001e-06 2024-08-19T14:24:36.321 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.321 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.321 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:24:36.322 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:34.721674+0000", 2024-08-19T14:24:36.322 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.15973267599999999 2024-08-19T14:24:36.322 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.323 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.323 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:24:36.323 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:34.721844+0000", 2024-08-19T14:24:36.323 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000170271 2024-08-19T14:24:36.323 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.324 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.324 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 8,9,11", 2024-08-19T14:24:36.324 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:34.722139+0000", 2024-08-19T14:24:36.324 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00029475399999999999 2024-08-19T14:24:36.324 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.324 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.324 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:24:36.324 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:34.726021+0000", 2024-08-19T14:24:36.324 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.003881865 2024-08-19T14:24:36.324 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.324 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.324 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:24:36.324 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:34.734915+0000", 2024-08-19T14:24:36.324 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0088943110000000002 2024-08-19T14:24:36.325 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.325 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.325 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:24:36.325 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:34.734993+0000", 2024-08-19T14:24:36.325 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.7733999999999998e-05 2024-08-19T14:24:36.325 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.325 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.325 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:24:36.325 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:34.769714+0000", 2024-08-19T14:24:36.325 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.034721476000000001 2024-08-19T14:24:36.325 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.326 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.326 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:24:36.326 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:34.769792+0000", 2024-08-19T14:24:36.326 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.7800999999999996e-05 2024-08-19T14:24:36.326 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.326 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.326 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:24:36.326 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:34.770001+0000", 2024-08-19T14:24:36.326 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00020901900000000001 2024-08-19T14:24:36.326 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:24:36.326 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:24:36.326 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:24:36.326 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.326 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.327 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25333.0:22719 4.0 4:0fc4203e:::benchmark_data_smithi045_71562_object22718:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-08-19T14:24:36.327 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:20:41.358778+0000", 2024-08-19T14:24:36.327 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 234.930154423, 2024-08-19T14:24:36.327 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.32258211599999997, 2024-08-19T14:24:36.327 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:24:36.327 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:24:36.327 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:24:36.327 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:24:36.327 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25333", 2024-08-19T14:24:36.327 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:24:36.327 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 22719 2024-08-19T14:24:36.327 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.327 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:24:36.328 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.328 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:24:36.328 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.358778+0000", 2024-08-19T14:24:36.328 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:24:36.328 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.328 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.328 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:24:36.328 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.358778+0000", 2024-08-19T14:24:36.328 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:24:36.328 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.328 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.328 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:24:36.328 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.358789+0000", 2024-08-19T14:24:36.328 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0446e-05 2024-08-19T14:24:36.329 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.329 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.329 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:24:36.329 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.358854+0000", 2024-08-19T14:24:36.329 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.5469000000000006e-05 2024-08-19T14:24:36.329 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.329 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.329 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:24:36.329 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.358877+0000", 2024-08-19T14:24:36.329 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.3285000000000002e-05 2024-08-19T14:24:36.329 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.329 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.330 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:24:36.330 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.358884+0000", 2024-08-19T14:24:36.330 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.229e-06 2024-08-19T14:24:36.330 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.330 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.330 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:24:36.330 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.358934+0000", 2024-08-19T14:24:36.330 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0878999999999999e-05 2024-08-19T14:24:36.330 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.330 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.330 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:24:36.330 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.359078+0000", 2024-08-19T14:24:36.331 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000143805 2024-08-19T14:24:36.331 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.331 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.331 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,7", 2024-08-19T14:24:36.331 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.359265+0000", 2024-08-19T14:24:36.331 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00018657799999999999 2024-08-19T14:24:36.331 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.331 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.331 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:24:36.331 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.360662+0000", 2024-08-19T14:24:36.331 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.001397162 2024-08-19T14:24:36.331 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.331 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.331 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:24:36.332 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.360902+0000", 2024-08-19T14:24:36.332 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000239614 2024-08-19T14:24:36.332 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.332 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.332 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:24:36.332 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.681249+0000", 2024-08-19T14:24:36.332 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.32034695800000001 2024-08-19T14:24:36.332 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.332 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.332 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:24:36.332 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.681311+0000", 2024-08-19T14:24:36.332 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.2346000000000002e-05 2024-08-19T14:24:36.332 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.332 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.333 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:24:36.333 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.681360+0000", 2024-08-19T14:24:36.333 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.9345000000000003e-05 2024-08-19T14:24:36.333 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:24:36.333 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:24:36.333 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:24:36.333 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.333 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.333 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25333.0:59928 4.0 4:01eb36bc:::benchmark_data_smithi045_71562_object59927:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-08-19T14:24:36.333 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:21:00.987826+0000", 2024-08-19T14:24:36.333 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 215.30110617, 2024-08-19T14:24:36.333 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.20708318000000001, 2024-08-19T14:24:36.333 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:24:36.334 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:24:36.334 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:24:36.334 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:24:36.334 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25333", 2024-08-19T14:24:36.334 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:24:36.334 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 59928 2024-08-19T14:24:36.334 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.334 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:24:36.334 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.334 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:24:36.334 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.987826+0000", 2024-08-19T14:24:36.334 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:24:36.334 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.335 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.335 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:24:36.335 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.987826+0000", 2024-08-19T14:24:36.335 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:24:36.335 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.335 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.335 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:24:36.335 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.987835+0000", 2024-08-19T14:24:36.335 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.7809999999999994e-06 2024-08-19T14:24:36.335 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.335 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.335 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:24:36.336 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.987889+0000", 2024-08-19T14:24:36.336 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.3498000000000002e-05 2024-08-19T14:24:36.336 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.336 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.336 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:24:36.336 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.987901+0000", 2024-08-19T14:24:36.336 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2661e-05 2024-08-19T14:24:36.336 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.336 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.336 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:24:36.336 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.987909+0000", 2024-08-19T14:24:36.336 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.3019999999999997e-06 2024-08-19T14:24:36.337 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.337 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.337 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:24:36.337 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.987976+0000", 2024-08-19T14:24:36.337 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.7602999999999996e-05 2024-08-19T14:24:36.337 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.337 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.337 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:24:36.337 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.988101+0000", 2024-08-19T14:24:36.337 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000124902 2024-08-19T14:24:36.337 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.337 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.338 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,7", 2024-08-19T14:24:36.338 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.988524+0000", 2024-08-19T14:24:36.338 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000422922 2024-08-19T14:24:36.338 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.338 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.338 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:24:36.338 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.997402+0000", 2024-08-19T14:24:36.338 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0088783839999999996 2024-08-19T14:24:36.338 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.339 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.339 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:24:36.339 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.997415+0000", 2024-08-19T14:24:36.339 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2125e-05 2024-08-19T14:24:36.339 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.339 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.339 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:24:36.339 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.194820+0000", 2024-08-19T14:24:36.339 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.19740545000000001 2024-08-19T14:24:36.339 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.339 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.339 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:24:36.339 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.194855+0000", 2024-08-19T14:24:36.340 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.4551000000000002e-05 2024-08-19T14:24:36.340 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.340 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.340 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:24:36.340 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.194910+0000", 2024-08-19T14:24:36.340 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.5000999999999997e-05 2024-08-19T14:24:36.340 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:24:36.340 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:24:36.340 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:24:36.340 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.340 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.340 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25333.0:59930 4.0 4:053c7972:::benchmark_data_smithi045_71562_object59929:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-08-19T14:24:36.340 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:21:00.988067+0000", 2024-08-19T14:24:36.341 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 215.300865354, 2024-08-19T14:24:36.341 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.20703280199999999, 2024-08-19T14:24:36.341 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:24:36.341 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:24:36.341 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:24:36.341 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:24:36.341 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25333", 2024-08-19T14:24:36.341 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:24:36.341 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 59930 2024-08-19T14:24:36.341 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.341 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:24:36.341 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.341 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:24:36.342 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.988067+0000", 2024-08-19T14:24:36.342 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:24:36.342 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.342 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.342 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:24:36.342 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.988067+0000", 2024-08-19T14:24:36.342 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:24:36.342 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.342 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.342 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:24:36.342 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.988068+0000", 2024-08-19T14:24:36.342 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.6700000000000002e-07 2024-08-19T14:24:36.343 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.343 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.343 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:24:36.343 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.988128+0000", 2024-08-19T14:24:36.343 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.0041000000000003e-05 2024-08-19T14:24:36.343 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.343 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.343 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:24:36.343 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.988150+0000", 2024-08-19T14:24:36.343 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.2262000000000001e-05 2024-08-19T14:24:36.343 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.343 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.343 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:24:36.343 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.988157+0000", 2024-08-19T14:24:36.344 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.567e-06 2024-08-19T14:24:36.344 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.344 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.344 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:24:36.344 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.990871+0000", 2024-08-19T14:24:36.344 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.002714475 2024-08-19T14:24:36.344 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.344 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.344 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:24:36.344 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.990996+0000", 2024-08-19T14:24:36.344 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012425700000000001 2024-08-19T14:24:36.344 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.344 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.344 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,7", 2024-08-19T14:24:36.345 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.991126+0000", 2024-08-19T14:24:36.345 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000130548 2024-08-19T14:24:36.345 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.345 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.345 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:24:36.345 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.997419+0000", 2024-08-19T14:24:36.345 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0062931339999999997 2024-08-19T14:24:36.345 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.345 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.345 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:24:36.345 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.997461+0000", 2024-08-19T14:24:36.345 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2061999999999998e-05 2024-08-19T14:24:36.345 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.345 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.346 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:24:36.346 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.194959+0000", 2024-08-19T14:24:36.346 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.197497439 2024-08-19T14:24:36.346 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.346 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.346 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:24:36.346 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.195017+0000", 2024-08-19T14:24:36.346 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.8272e-05 2024-08-19T14:24:36.346 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.346 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.346 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:24:36.346 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.195100+0000", 2024-08-19T14:24:36.346 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.2878000000000002e-05 2024-08-19T14:24:36.347 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:24:36.347 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:24:36.347 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:24:36.347 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.347 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.347 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25333.0:59932 4.0 4:0ae6f13c:::benchmark_data_smithi045_71562_object59931:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-08-19T14:24:36.347 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:21:00.988175+0000", 2024-08-19T14:24:36.347 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 215.300757226, 2024-08-19T14:24:36.347 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.21384358000000001, 2024-08-19T14:24:36.347 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:24:36.347 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:24:36.347 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:24:36.348 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:24:36.348 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25333", 2024-08-19T14:24:36.348 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:24:36.348 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 59932 2024-08-19T14:24:36.348 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.348 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:24:36.348 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.348 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:24:36.348 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.988175+0000", 2024-08-19T14:24:36.348 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:24:36.348 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.348 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.348 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:24:36.348 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.988175+0000", 2024-08-19T14:24:36.349 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:24:36.349 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.349 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.349 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:24:36.349 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.988176+0000", 2024-08-19T14:24:36.349 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.8999999999999997e-07 2024-08-19T14:24:36.349 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.349 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.349 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:24:36.349 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.988205+0000", 2024-08-19T14:24:36.349 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.9047000000000002e-05 2024-08-19T14:24:36.349 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.349 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.349 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:24:36.350 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.988212+0000", 2024-08-19T14:24:36.350 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.2250000000000003e-06 2024-08-19T14:24:36.350 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.350 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.350 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:24:36.350 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.988218+0000", 2024-08-19T14:24:36.350 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.4670000000000003e-06 2024-08-19T14:24:36.350 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.350 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.350 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:24:36.351 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.993116+0000", 2024-08-19T14:24:36.351 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0048980830000000001 2024-08-19T14:24:36.351 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.351 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.351 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:24:36.351 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.993256+0000", 2024-08-19T14:24:36.351 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000140426 2024-08-19T14:24:36.351 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.351 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.351 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,7", 2024-08-19T14:24:36.352 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.993350+0000", 2024-08-19T14:24:36.352 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.4026999999999999e-05 2024-08-19T14:24:36.352 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.352 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.352 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:24:36.352 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.997424+0000", 2024-08-19T14:24:36.352 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0040743929999999999 2024-08-19T14:24:36.352 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.352 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.352 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:24:36.352 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.002435+0000", 2024-08-19T14:24:36.352 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0050106919999999998 2024-08-19T14:24:36.353 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.353 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.353 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:24:36.353 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.201944+0000", 2024-08-19T14:24:36.353 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.19950880200000001 2024-08-19T14:24:36.353 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.353 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.353 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:24:36.353 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.201978+0000", 2024-08-19T14:24:36.353 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.4379999999999999e-05 2024-08-19T14:24:36.354 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.354 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.354 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:24:36.354 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.202019+0000", 2024-08-19T14:24:36.354 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.0547999999999999e-05 2024-08-19T14:24:36.354 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:24:36.354 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:24:36.354 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:24:36.354 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.354 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.354 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25333.0:59939 4.0 4:022bf8ca:::benchmark_data_smithi045_71562_object59938:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-08-19T14:24:36.355 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:21:00.992948+0000", 2024-08-19T14:24:36.355 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 215.295984194, 2024-08-19T14:24:36.355 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.20916093399999999, 2024-08-19T14:24:36.355 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:24:36.355 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:24:36.355 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:24:36.355 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:24:36.355 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25333", 2024-08-19T14:24:36.355 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:24:36.355 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 59939 2024-08-19T14:24:36.355 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.356 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:24:36.356 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.356 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:24:36.356 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.992948+0000", 2024-08-19T14:24:36.356 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:24:36.356 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.356 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.356 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:24:36.356 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.992948+0000", 2024-08-19T14:24:36.356 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:24:36.356 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.357 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.357 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:24:36.357 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.992950+0000", 2024-08-19T14:24:36.357 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.668e-06 2024-08-19T14:24:36.357 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.357 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.357 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:24:36.357 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.992997+0000", 2024-08-19T14:24:36.357 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7491999999999999e-05 2024-08-19T14:24:36.357 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.358 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.358 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:24:36.358 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.993017+0000", 2024-08-19T14:24:36.358 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.9837999999999999e-05 2024-08-19T14:24:36.358 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.358 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.358 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:24:36.358 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.993025+0000", 2024-08-19T14:24:36.358 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.2710000000000003e-06 2024-08-19T14:24:36.359 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.359 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.359 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:24:36.359 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.995120+0000", 2024-08-19T14:24:36.359 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.002095436 2024-08-19T14:24:36.359 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.359 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.359 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:24:36.359 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.995232+0000", 2024-08-19T14:24:36.359 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000111691 2024-08-19T14:24:36.360 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.360 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.360 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,7", 2024-08-19T14:24:36.360 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.995370+0000", 2024-08-19T14:24:36.360 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013791000000000001 2024-08-19T14:24:36.360 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.360 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.360 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:24:36.360 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.001953+0000", 2024-08-19T14:24:36.360 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0065829340000000004 2024-08-19T14:24:36.360 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.360 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.361 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:24:36.361 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.002472+0000", 2024-08-19T14:24:36.361 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00051963200000000001 2024-08-19T14:24:36.361 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.361 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.361 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:24:36.361 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.202049+0000", 2024-08-19T14:24:36.361 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.19957635800000001 2024-08-19T14:24:36.361 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.361 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.361 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:24:36.361 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.202076+0000", 2024-08-19T14:24:36.362 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.7314999999999999e-05 2024-08-19T14:24:36.362 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.362 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.362 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:24:36.362 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.202109+0000", 2024-08-19T14:24:36.362 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.3389000000000002e-05 2024-08-19T14:24:36.362 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:24:36.362 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:24:36.362 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:24:36.362 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.362 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.362 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25333.0:59946 4.0 4:0b0e7328:::benchmark_data_smithi045_71562_object59945:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-08-19T14:24:36.363 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:21:00.995937+0000", 2024-08-19T14:24:36.363 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 215.292995397, 2024-08-19T14:24:36.363 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.20888220900000001, 2024-08-19T14:24:36.363 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:24:36.363 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:24:36.363 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:24:36.363 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:24:36.363 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25333", 2024-08-19T14:24:36.363 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:24:36.363 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 59946 2024-08-19T14:24:36.363 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.363 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:24:36.364 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.364 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:24:36.364 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.995937+0000", 2024-08-19T14:24:36.364 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:24:36.364 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.364 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.364 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:24:36.364 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.995937+0000", 2024-08-19T14:24:36.364 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:24:36.364 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.364 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.365 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:24:36.365 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.995939+0000", 2024-08-19T14:24:36.365 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5510000000000001e-06 2024-08-19T14:24:36.365 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.365 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.365 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:24:36.365 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.996006+0000", 2024-08-19T14:24:36.365 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.6914000000000003e-05 2024-08-19T14:24:36.365 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.365 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.365 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:24:36.365 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.996018+0000", 2024-08-19T14:24:36.366 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2361000000000001e-05 2024-08-19T14:24:36.366 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.366 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.366 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:24:36.366 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.996025+0000", 2024-08-19T14:24:36.366 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.199e-06 2024-08-19T14:24:36.366 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.366 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.366 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:24:36.366 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.997492+0000", 2024-08-19T14:24:36.366 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.001466572 2024-08-19T14:24:36.366 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.367 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.367 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:24:36.367 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.997618+0000", 2024-08-19T14:24:36.367 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012644699999999999 2024-08-19T14:24:36.367 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.367 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.367 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,7", 2024-08-19T14:24:36.367 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.997835+0000", 2024-08-19T14:24:36.367 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000216357 2024-08-19T14:24:36.367 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.367 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.367 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:24:36.367 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.001957+0000", 2024-08-19T14:24:36.367 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0041229559999999997 2024-08-19T14:24:36.368 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.368 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.368 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:24:36.368 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.005478+0000", 2024-08-19T14:24:36.368 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.003520535 2024-08-19T14:24:36.368 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.368 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.368 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:24:36.368 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.204708+0000", 2024-08-19T14:24:36.368 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.199229884 2024-08-19T14:24:36.368 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.368 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.368 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:24:36.368 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.204761+0000", 2024-08-19T14:24:36.369 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.3443999999999999e-05 2024-08-19T14:24:36.369 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.369 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.369 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:24:36.369 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.204819+0000", 2024-08-19T14:24:36.369 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.7989000000000003e-05 2024-08-19T14:24:36.369 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:24:36.369 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:24:36.369 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:24:36.369 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.369 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.369 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25333.0:59952 4.0 4:0d22fdcc:::benchmark_data_smithi045_71562_object59951:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-08-19T14:24:36.369 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:21:00.997380+0000", 2024-08-19T14:24:36.369 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 215.29155298200001, 2024-08-19T14:24:36.370 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.207624002, 2024-08-19T14:24:36.370 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:24:36.370 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:24:36.370 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:24:36.370 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:24:36.370 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25333", 2024-08-19T14:24:36.370 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:24:36.370 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 59952 2024-08-19T14:24:36.370 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.370 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:24:36.370 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.370 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:24:36.371 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.997380+0000", 2024-08-19T14:24:36.371 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:24:36.371 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.371 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.371 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:24:36.371 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.997380+0000", 2024-08-19T14:24:36.371 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:24:36.371 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.371 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.371 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:24:36.371 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.997381+0000", 2024-08-19T14:24:36.371 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1310000000000001e-06 2024-08-19T14:24:36.371 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.372 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.372 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:24:36.372 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.997419+0000", 2024-08-19T14:24:36.372 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.8781e-05 2024-08-19T14:24:36.372 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.372 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.372 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:24:36.372 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.997433+0000", 2024-08-19T14:24:36.372 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3301e-05 2024-08-19T14:24:36.372 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.372 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.372 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:24:36.372 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.997440+0000", 2024-08-19T14:24:36.373 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.289e-06 2024-08-19T14:24:36.373 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.373 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.373 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:24:36.373 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.999803+0000", 2024-08-19T14:24:36.373 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.002363066 2024-08-19T14:24:36.373 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.373 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.373 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:24:36.373 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.999943+0000", 2024-08-19T14:24:36.373 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013990000000000001 2024-08-19T14:24:36.373 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.374 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.374 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,7", 2024-08-19T14:24:36.374 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.000083+0000", 2024-08-19T14:24:36.374 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000139963 2024-08-19T14:24:36.374 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.374 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.374 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:24:36.374 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.002202+0000", 2024-08-19T14:24:36.374 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0021193430000000001 2024-08-19T14:24:36.374 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.374 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.374 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:24:36.375 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.006637+0000", 2024-08-19T14:24:36.375 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0044344629999999996 2024-08-19T14:24:36.375 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.375 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.375 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:24:36.375 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.204885+0000", 2024-08-19T14:24:36.375 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.198248705 2024-08-19T14:24:36.375 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.375 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.375 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:24:36.375 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.204945+0000", 2024-08-19T14:24:36.375 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.9762000000000001e-05 2024-08-19T14:24:36.375 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.375 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.376 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:24:36.376 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.205004+0000", 2024-08-19T14:24:36.376 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.8298000000000003e-05 2024-08-19T14:24:36.376 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:24:36.376 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:24:36.376 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:24:36.376 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.376 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.376 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25333.0:59956 4.0 4:0ac3aada:::benchmark_data_smithi045_71562_object59955:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-08-19T14:24:36.376 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:21:01.002823+0000", 2024-08-19T14:24:36.376 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 215.28610962900001, 2024-08-19T14:24:36.376 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.20233593899999999, 2024-08-19T14:24:36.376 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:24:36.377 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:24:36.377 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:24:36.377 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:24:36.377 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25333", 2024-08-19T14:24:36.377 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:24:36.377 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 59956 2024-08-19T14:24:36.377 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.377 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:24:36.377 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.377 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:24:36.377 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.002823+0000", 2024-08-19T14:24:36.377 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:24:36.377 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.377 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.378 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:24:36.378 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.002823+0000", 2024-08-19T14:24:36.378 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:24:36.378 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.378 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.378 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:24:36.378 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.002824+0000", 2024-08-19T14:24:36.378 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.401e-06 2024-08-19T14:24:36.378 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.378 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.378 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:24:36.378 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.002905+0000", 2024-08-19T14:24:36.378 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.0476999999999995e-05 2024-08-19T14:24:36.379 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.379 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.379 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:24:36.379 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.002917+0000", 2024-08-19T14:24:36.379 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2581e-05 2024-08-19T14:24:36.379 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.379 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.379 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:24:36.379 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.002925+0000", 2024-08-19T14:24:36.379 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.391e-06 2024-08-19T14:24:36.379 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.379 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.379 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:24:36.379 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.003010+0000", 2024-08-19T14:24:36.380 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.5099000000000006e-05 2024-08-19T14:24:36.380 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.380 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.380 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:24:36.380 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.003192+0000", 2024-08-19T14:24:36.380 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00018232799999999999 2024-08-19T14:24:36.380 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.380 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.380 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,7", 2024-08-19T14:24:36.380 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.003343+0000", 2024-08-19T14:24:36.380 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000151163 2024-08-19T14:24:36.380 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.380 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.380 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:24:36.381 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.005697+0000", 2024-08-19T14:24:36.381 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.002353749 2024-08-19T14:24:36.381 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.381 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.381 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:24:36.381 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.006706+0000", 2024-08-19T14:24:36.381 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.001009397 2024-08-19T14:24:36.381 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.381 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.381 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:24:36.381 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.205041+0000", 2024-08-19T14:24:36.381 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.198334387 2024-08-19T14:24:36.381 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.381 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.382 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:24:36.382 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.205091+0000", 2024-08-19T14:24:36.382 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0374999999999998e-05 2024-08-19T14:24:36.382 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.382 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.382 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:24:36.382 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.205159+0000", 2024-08-19T14:24:36.382 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.7590999999999997e-05 2024-08-19T14:24:36.382 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:24:36.382 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:24:36.382 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:24:36.382 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.382 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.383 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25333.0:59962 4.0 4:05eece39:::benchmark_data_smithi045_71562_object59961:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-08-19T14:24:36.383 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:21:01.006823+0000", 2024-08-19T14:24:36.383 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 215.28210978999999, 2024-08-19T14:24:36.383 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.20119857399999999, 2024-08-19T14:24:36.383 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:24:36.383 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:24:36.383 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:24:36.383 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:24:36.383 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25333", 2024-08-19T14:24:36.383 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:24:36.383 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 59962 2024-08-19T14:24:36.383 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.383 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:24:36.384 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.384 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:24:36.384 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.006823+0000", 2024-08-19T14:24:36.384 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:24:36.384 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.384 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.384 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:24:36.384 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.006823+0000", 2024-08-19T14:24:36.384 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:24:36.384 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.384 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.384 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:24:36.384 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.006824+0000", 2024-08-19T14:24:36.385 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5319999999999999e-06 2024-08-19T14:24:36.385 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.385 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.385 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:24:36.385 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.006881+0000", 2024-08-19T14:24:36.385 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.6805999999999998e-05 2024-08-19T14:24:36.385 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.385 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.385 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:24:36.385 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.006892+0000", 2024-08-19T14:24:36.385 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0997e-05 2024-08-19T14:24:36.385 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.385 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.385 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:24:36.386 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.006897+0000", 2024-08-19T14:24:36.386 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1370000000000001e-06 2024-08-19T14:24:36.386 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.386 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.386 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:24:36.386 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.006947+0000", 2024-08-19T14:24:36.386 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.9484999999999998e-05 2024-08-19T14:24:36.386 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.386 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.386 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:24:36.386 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.007191+0000", 2024-08-19T14:24:36.386 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000244718 2024-08-19T14:24:36.386 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.387 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.387 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,7", 2024-08-19T14:24:36.387 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.007328+0000", 2024-08-19T14:24:36.387 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000136143 2024-08-19T14:24:36.387 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.387 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.387 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:24:36.387 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.010086+0000", 2024-08-19T14:24:36.387 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0027586080000000001 2024-08-19T14:24:36.387 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.387 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.387 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:24:36.387 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.011443+0000", 2024-08-19T14:24:36.388 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00135698 2024-08-19T14:24:36.388 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.388 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.388 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:24:36.388 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.207902+0000", 2024-08-19T14:24:36.388 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.19645927599999999 2024-08-19T14:24:36.388 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.388 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.388 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:24:36.388 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.207961+0000", 2024-08-19T14:24:36.388 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.8989000000000001e-05 2024-08-19T14:24:36.388 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.388 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.389 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:24:36.389 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.208021+0000", 2024-08-19T14:24:36.389 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.9902999999999999e-05 2024-08-19T14:24:36.389 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:24:36.389 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:24:36.389 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:24:36.389 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.389 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.389 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.25333.0:149067 4.e e378/335 4:7e247c1a:::benchmark_data_smithi045_71562_object17892:head v 378'9178, mlcod=378'9176)", 2024-08-19T14:24:36.389 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:22:24.358485+0000", 2024-08-19T14:24:36.390 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 131.930447878, 2024-08-19T14:24:36.390 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.21993733300000001, 2024-08-19T14:24:36.390 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:24:36.390 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:24:36.390 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:24:36.390 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:24:36.390 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.390 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:24:36.390 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:24.358485+0000", 2024-08-19T14:24:36.390 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:24:36.390 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.390 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.390 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:24:36.391 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:24.358485+0000", 2024-08-19T14:24:36.391 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:24:36.391 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.391 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.391 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:24:36.391 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:24.358485+0000", 2024-08-19T14:24:36.391 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.9500000000000001e-07 2024-08-19T14:24:36.391 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.391 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.391 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:24:36.391 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:24.358493+0000", 2024-08-19T14:24:36.391 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.9470000000000003e-06 2024-08-19T14:24:36.391 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.392 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.392 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:24:36.392 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:24.358505+0000", 2024-08-19T14:24:36.392 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2082000000000001e-05 2024-08-19T14:24:36.392 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.392 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.392 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:24:36.392 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:24.358513+0000", 2024-08-19T14:24:36.392 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.7489999999999998e-06 2024-08-19T14:24:36.392 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.392 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.392 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:24:36.392 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:24.358590+0000", 2024-08-19T14:24:36.393 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.7158999999999998e-05 2024-08-19T14:24:36.393 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.393 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.393 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:24:36.393 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:24.358610+0000", 2024-08-19T14:24:36.393 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.9456000000000001e-05 2024-08-19T14:24:36.393 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.393 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.393 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:24:36.393 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:24.578385+0000", 2024-08-19T14:24:36.393 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.21977474999999999 2024-08-19T14:24:36.393 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.393 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.393 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:24:36.394 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:24.578422+0000", 2024-08-19T14:24:36.394 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7394999999999997e-05 2024-08-19T14:24:36.394 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:24:36.394 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:24:36.394 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:24:36.394 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.394 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.394 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.25333.0:149070 4.e e378/335 4:77cd6059:::benchmark_data_smithi045_71562_object17895:head v 378'9179, mlcod=378'9176)", 2024-08-19T14:24:36.394 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:22:24.363184+0000", 2024-08-19T14:24:36.394 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 131.92574887999999, 2024-08-19T14:24:36.394 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.21893797600000001, 2024-08-19T14:24:36.394 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:24:36.394 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:24:36.395 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:24:36.395 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:24:36.395 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.395 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:24:36.395 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:24.363184+0000", 2024-08-19T14:24:36.395 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:24:36.395 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.395 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.395 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:24:36.395 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:24.363184+0000", 2024-08-19T14:24:36.395 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:24:36.395 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.395 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.395 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:24:36.396 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:24.363185+0000", 2024-08-19T14:24:36.396 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.9400000000000004e-07 2024-08-19T14:24:36.396 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.396 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.396 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:24:36.396 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:24.363192+0000", 2024-08-19T14:24:36.396 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.34e-06 2024-08-19T14:24:36.396 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.396 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.396 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:24:36.396 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:24.363204+0000", 2024-08-19T14:24:36.396 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1934e-05 2024-08-19T14:24:36.396 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.397 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.397 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:24:36.397 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:24.363211+0000", 2024-08-19T14:24:36.397 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.7109999999999998e-06 2024-08-19T14:24:36.397 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.397 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.397 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:24:36.397 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:24.366560+0000", 2024-08-19T14:24:36.397 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.003349552 2024-08-19T14:24:36.397 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.397 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.397 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:24:36.397 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:24.366586+0000", 2024-08-19T14:24:36.397 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.6336999999999999e-05 2024-08-19T14:24:36.398 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.398 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.398 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:24:36.398 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:24.582085+0000", 2024-08-19T14:24:36.398 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.21549889 2024-08-19T14:24:36.398 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.398 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.398 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:24:36.398 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:24.582122+0000", 2024-08-19T14:24:36.398 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.6318e-05 2024-08-19T14:24:36.398 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:24:36.398 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:24:36.399 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:24:36.400 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:24:36.400 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.400 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.400 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.25333.0:149073 4.f e378/335 4:fc7f52b3:::benchmark_data_smithi045_71562_object17898:head v 378'9244, mlcod=378'9241)", 2024-08-19T14:24:36.400 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:22:24.367356+0000", 2024-08-19T14:24:36.400 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 131.92157618600001, 2024-08-19T14:24:36.400 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.20626061600000001, 2024-08-19T14:24:36.400 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:24:36.400 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:24:36.400 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:24:36.400 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:24:36.401 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.401 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:24:36.401 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:24.367356+0000", 2024-08-19T14:24:36.401 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:24:36.401 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.401 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.401 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:24:36.401 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:24.367356+0000", 2024-08-19T14:24:36.401 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:24:36.401 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.401 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.401 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:24:36.402 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:24.367357+0000", 2024-08-19T14:24:36.402 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.4300000000000001e-07 2024-08-19T14:24:36.402 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.402 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.402 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:24:36.402 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:24.367363+0000", 2024-08-19T14:24:36.402 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.2389999999999999e-06 2024-08-19T14:24:36.402 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.402 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.402 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:24:36.402 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:24.565241+0000", 2024-08-19T14:24:36.402 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.19787778 2024-08-19T14:24:36.402 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.403 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.403 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:24:36.403 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:24.565247+0000", 2024-08-19T14:24:36.403 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.3499999999999996e-06 2024-08-19T14:24:36.403 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.403 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.403 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:24:36.403 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:24.565560+0000", 2024-08-19T14:24:36.403 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00031315899999999997 2024-08-19T14:24:36.403 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.403 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.403 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:24:36.404 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:24.565581+0000", 2024-08-19T14:24:36.404 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.1067999999999999e-05 2024-08-19T14:24:36.404 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.404 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.404 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:24:36.404 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:24.573490+0000", 2024-08-19T14:24:36.404 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0079087540000000005 2024-08-19T14:24:36.404 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.404 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.404 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:24:36.404 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:24.573617+0000", 2024-08-19T14:24:36.404 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012732299999999999 2024-08-19T14:24:36.404 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:24:36.405 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:24:36.405 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:24:36.405 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.405 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.405 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.25333.0:149082 4.e e378/335 4:766b7f6c:::benchmark_data_smithi045_71562_object17907:head v 378'9180, mlcod=378'9176)", 2024-08-19T14:24:36.405 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:22:24.374332+0000", 2024-08-19T14:24:36.405 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 131.914600466, 2024-08-19T14:24:36.405 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.20781767900000001, 2024-08-19T14:24:36.405 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:24:36.405 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:24:36.405 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:24:36.405 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:24:36.406 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.406 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:24:36.406 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:24.374332+0000", 2024-08-19T14:24:36.406 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:24:36.406 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.406 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.406 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:24:36.406 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:24.374332+0000", 2024-08-19T14:24:36.406 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:24:36.406 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.406 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.406 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:24:36.406 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:24.374333+0000", 2024-08-19T14:24:36.407 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.105e-06 2024-08-19T14:24:36.407 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.407 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.407 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:24:36.407 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:24.374353+0000", 2024-08-19T14:24:36.407 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.9695e-05 2024-08-19T14:24:36.407 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.407 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.407 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:24:36.407 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:24.565211+0000", 2024-08-19T14:24:36.407 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.19085808200000001 2024-08-19T14:24:36.407 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.407 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.408 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:24:36.408 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:24.565224+0000", 2024-08-19T14:24:36.408 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3305999999999999e-05 2024-08-19T14:24:36.408 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.408 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.408 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:24:36.408 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:24.569252+0000", 2024-08-19T14:24:36.408 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.004027976 2024-08-19T14:24:36.408 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.408 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.408 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:24:36.409 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:24.569277+0000", 2024-08-19T14:24:36.409 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.4561000000000001e-05 2024-08-19T14:24:36.409 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.409 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.409 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:24:36.409 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:24.582125+0000", 2024-08-19T14:24:36.409 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.012848613 2024-08-19T14:24:36.409 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.409 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.409 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:24:36.409 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:24.582150+0000", 2024-08-19T14:24:36.409 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.4341e-05 2024-08-19T14:24:36.410 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:24:36.410 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:24:36.410 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:24:36.410 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.410 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.410 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25333.0:212509 4.0 4:06a04a03:::benchmark_data_smithi045_71562_object81334:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e396)", 2024-08-19T14:24:36.410 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:23:35.781886+0000", 2024-08-19T14:24:36.410 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 60.507046443, 2024-08-19T14:24:36.410 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.21969265399999999, 2024-08-19T14:24:36.410 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:24:36.410 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:24:36.410 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:24:36.410 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:24:36.411 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25333", 2024-08-19T14:24:36.411 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:24:36.411 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 212509 2024-08-19T14:24:36.411 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.411 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:24:36.411 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.411 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:24:36.411 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:23:35.781886+0000", 2024-08-19T14:24:36.411 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:24:36.411 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.411 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.411 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:24:36.411 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:23:35.781886+0000", 2024-08-19T14:24:36.412 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:24:36.412 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.412 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.412 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:24:36.412 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:23:35.781887+0000", 2024-08-19T14:24:36.412 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3799999999999999e-06 2024-08-19T14:24:36.412 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.412 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.412 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:24:36.412 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:23:35.781893+0000", 2024-08-19T14:24:36.412 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1399999999999999e-06 2024-08-19T14:24:36.412 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.412 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.413 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:24:36.413 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:23:35.781903+0000", 2024-08-19T14:24:36.413 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0902e-05 2024-08-19T14:24:36.413 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.413 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.413 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:24:36.413 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:23:35.781911+0000", 2024-08-19T14:24:36.413 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.6680000000000001e-06 2024-08-19T14:24:36.413 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.413 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.413 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:24:36.413 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:23:35.782005+0000", 2024-08-19T14:24:36.413 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.3586999999999997e-05 2024-08-19T14:24:36.414 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.414 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.414 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:24:36.414 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:23:35.782128+0000", 2024-08-19T14:24:36.414 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012343900000000001 2024-08-19T14:24:36.414 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.414 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.414 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,7", 2024-08-19T14:24:36.414 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:23:35.782260+0000", 2024-08-19T14:24:36.414 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013141100000000001 2024-08-19T14:24:36.414 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.414 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.414 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:24:36.415 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:23:35.790786+0000", 2024-08-19T14:24:36.415 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0085262970000000004 2024-08-19T14:24:36.415 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.415 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.415 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:24:36.415 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:23:35.808252+0000", 2024-08-19T14:24:36.415 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.01746578 2024-08-19T14:24:36.415 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.415 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.415 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:24:36.415 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:23:36.001465+0000", 2024-08-19T14:24:36.415 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.19321337599999999 2024-08-19T14:24:36.415 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.415 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.416 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:24:36.416 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:23:36.001529+0000", 2024-08-19T14:24:36.416 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.3825000000000006e-05 2024-08-19T14:24:36.416 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.416 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.416 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:24:36.416 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:23:36.001579+0000", 2024-08-19T14:24:36.416 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.9849000000000003e-05 2024-08-19T14:24:36.416 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:24:36.416 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:24:36.416 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:24:36.416 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.416 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.417 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25333.0:212525 4.0 4:05272b03:::benchmark_data_smithi045_71562_object81350:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e396)", 2024-08-19T14:24:36.417 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:23:35.799140+0000", 2024-08-19T14:24:36.417 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 60.489792583000003, 2024-08-19T14:24:36.417 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.206006202, 2024-08-19T14:24:36.417 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:24:36.417 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:24:36.417 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:24:36.417 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:24:36.417 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25333", 2024-08-19T14:24:36.417 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:24:36.417 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 212525 2024-08-19T14:24:36.417 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.417 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:24:36.417 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.418 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:24:36.418 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:23:35.799140+0000", 2024-08-19T14:24:36.418 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:24:36.418 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.418 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.418 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:24:36.418 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:23:35.799140+0000", 2024-08-19T14:24:36.418 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:24:36.418 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.418 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.418 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:24:36.418 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:23:35.799141+0000", 2024-08-19T14:24:36.418 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.497e-06 2024-08-19T14:24:36.419 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.419 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.419 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:24:36.419 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:23:35.799147+0000", 2024-08-19T14:24:36.419 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.9070000000000004e-06 2024-08-19T14:24:36.419 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.419 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.419 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:24:36.419 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:23:35.799167+0000", 2024-08-19T14:24:36.419 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.9881000000000002e-05 2024-08-19T14:24:36.419 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.419 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.419 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:24:36.419 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:23:35.799175+0000", 2024-08-19T14:24:36.420 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.0209999999999999e-06 2024-08-19T14:24:36.420 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.420 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.420 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:24:36.420 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:23:35.799334+0000", 2024-08-19T14:24:36.420 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00015849399999999999 2024-08-19T14:24:36.420 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.420 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.420 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:24:36.420 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:23:35.799637+0000", 2024-08-19T14:24:36.420 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00030332999999999998 2024-08-19T14:24:36.420 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.420 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.420 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,7", 2024-08-19T14:24:36.421 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:23:35.800018+0000", 2024-08-19T14:24:36.421 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00038109199999999998 2024-08-19T14:24:36.421 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.421 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.421 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:24:36.421 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:23:35.805503+0000", 2024-08-19T14:24:36.421 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0054849080000000001 2024-08-19T14:24:36.421 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.421 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.421 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:24:36.421 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:23:35.808296+0000", 2024-08-19T14:24:36.421 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0027926130000000002 2024-08-19T14:24:36.421 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.422 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.422 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:24:36.422 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:23:36.005042+0000", 2024-08-19T14:24:36.422 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.19674630600000001 2024-08-19T14:24:36.422 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.422 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.422 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:24:36.422 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:23:36.005100+0000", 2024-08-19T14:24:36.422 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.8354999999999999e-05 2024-08-19T14:24:36.422 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:24:36.422 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:24:36.422 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:24:36.422 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:23:36.005146+0000", 2024-08-19T14:24:36.423 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.5797999999999998e-05 2024-08-19T14:24:36.423 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:24:36.423 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:24:36.423 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:24:36.423 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:24:36.423 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:24:36.423 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:24:36.448 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:36.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:36.449 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:24:36.549 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:24:36.576 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:36.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:36.577 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:24:36.589 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:36 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/862714147' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-08-19T14:24:36.589 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:36 smithi031 ceph-mon[16336]: osdmap e416: 12 total, 12 up, 9 in 2024-08-19T14:24:36.616 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:24:36.616 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:24:36.616 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:24:36.616 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:24:36.624 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:24:36.624 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:24:36.640 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:24:36.640 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:24:36.658 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:24:36.658 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:24:36.675 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:24:36.675 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:24:36.678 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:24:36.692 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:24:36.692 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:24:36.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:36.703 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:36.703 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:24:36.710 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:24:36.710 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:24:36.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:36 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/862714147' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-08-19T14:24:36.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:36 smithi032 ceph-mon[20004]: osdmap e416: 12 total, 12 up, 9 in 2024-08-19T14:24:36.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:36 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/862714147' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-08-19T14:24:36.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:36 smithi045 ceph-mon[20143]: osdmap e416: 12 total, 12 up, 9 in 2024-08-19T14:24:36.803 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:24:36.829 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:36.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:36.830 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:24:36.931 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:24:36.952 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:36.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:36.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:24:37.023 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:24:37.023 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:24:37.054 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:24:37.079 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:37.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:37.080 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:24:37.181 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:24:37.206 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:37.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:37.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:24:37.309 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:24:37.331 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:24:37.331 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:24:37.333 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:37.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:37.334 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:24:37.348 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:24:37.348 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:24:37.365 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:24:37.365 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:24:37.384 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:24:37.385 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:24:37.435 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:24:37.460 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:37.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:37.461 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:24:37.561 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:24:37.585 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:37.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:37.586 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:24:37.687 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:24:37.712 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:37.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:37.713 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:24:37.751 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:37 smithi031 ceph-mon[16336]: pgmap v957: 47 pgs: 47 active+clean; 24 MiB data, 4.3 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 555 op/s 2024-08-19T14:24:37.752 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:37 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1045082970' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:24:37.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:37 smithi032 ceph-mon[20004]: pgmap v957: 47 pgs: 47 active+clean; 24 MiB data, 4.3 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 555 op/s 2024-08-19T14:24:37.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:37 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1045082970' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:24:37.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:37 smithi045 ceph-mon[20143]: pgmap v957: 47 pgs: 47 active+clean; 24 MiB data, 4.3 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 555 op/s 2024-08-19T14:24:37.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:37 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1045082970' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:24:37.815 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:24:37.838 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:37.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:37.839 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:24:37.940 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:24:37.965 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:37.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:37.966 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:24:38.047 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_2 2024-08-19T14:24:38.047 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd pool rm unique_pool_2 unique_pool_2 --yes-i-really-really-mean-it 2024-08-19T14:24:38.066 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:24:38.091 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:38.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:38.092 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:24:38.193 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:24:38.214 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:38.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:38.215 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:24:38.317 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:24:38.342 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:38.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:38.343 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:24:38.444 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:24:38.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:38.468 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:38.468 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:24:38.569 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:24:38.595 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:38.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:38.597 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:24:38.698 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:24:38.723 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:38.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:38.724 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:24:38.825 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:24:38.845 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-08-19T14:24:38.846 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:38.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:38.848 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:24:38.949 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:24:38.971 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:38.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:38.972 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:24:39.052 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:24:39.073 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:24:39.095 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:39.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:39.096 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:24:39.111 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:24:39.196 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:24:39.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:39.219 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:39.219 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:24:39.321 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:24:39.346 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:39.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:39.348 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:24:39.408 INFO:teuthology.orchestra.run.smithi031.stderr:reweighted osd.1 to 1 (10000) 2024-08-19T14:24:39.408 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is unset 2024-08-19T14:24:39.449 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:24:39.470 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:39.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:39.471 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:24:39.501 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:39 smithi032 ceph-mon[20004]: pgmap v958: 47 pgs: 47 active+clean; 577 KiB data, 4.2 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 435 op/s 2024-08-19T14:24:39.501 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:39 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2493383209' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-08-19T14:24:39.501 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:39 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1681700212' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:24:39.563 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:39 smithi031 ceph-mon[16336]: pgmap v958: 47 pgs: 47 active+clean; 577 KiB data, 4.2 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 435 op/s 2024-08-19T14:24:39.563 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:39 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2493383209' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-08-19T14:24:39.563 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:39 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1681700212' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:24:39.573 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:24:39.595 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:39.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:39.596 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:24:39.697 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:24:39.709 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:24:39.710 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:24:39.723 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:39.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:39.724 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:24:39.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:39 smithi045 ceph-mon[20143]: pgmap v958: 47 pgs: 47 active+clean; 577 KiB data, 4.2 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 435 op/s 2024-08-19T14:24:39.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:39 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2493383209' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-08-19T14:24:39.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:39 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1681700212' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:24:39.825 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:24:39.846 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:39.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:39.847 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:24:39.948 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:24:39.973 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:39.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:39.974 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:24:40.075 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:24:40.077 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd reweight 3 1 2024-08-19T14:24:40.137 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:40.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:40.138 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:24:40.239 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:24:40.260 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:40.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:40.261 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:24:40.287 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:24:40.362 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:24:40.386 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:40.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:40.387 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:24:40.488 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:24:40.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:40 smithi032 ceph-mon[20004]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:24:40.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:40 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2493383209' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-08-19T14:24:40.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:40 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1681700212' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:24:40.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:40 smithi032 ceph-mon[20004]: osdmap e417: 12 total, 12 up, 9 in 2024-08-19T14:24:40.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:40 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1676690981' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-08-19T14:24:40.510 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:40.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:40.511 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:24:40.559 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-08-19T14:24:40.578 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:40 smithi031 ceph-mon[16336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:24:40.578 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:40 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2493383209' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-08-19T14:24:40.578 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:40 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1681700212' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:24:40.578 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:40 smithi031 ceph-mon[16336]: osdmap e417: 12 total, 12 up, 9 in 2024-08-19T14:24:40.578 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:40 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1676690981' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-08-19T14:24:40.612 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:24:40.633 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:40.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:40.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:24:40.735 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:24:40.758 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:40.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:40.759 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:24:40.761 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'unique_pool_2' removed 2024-08-19T14:24:40.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:40 smithi045 ceph-mon[20143]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:24:40.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:40 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2493383209' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-08-19T14:24:40.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:40 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1681700212' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:24:40.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:40 smithi045 ceph-mon[20143]: osdmap e417: 12 total, 12 up, 9 in 2024-08-19T14:24:40.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:40 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1676690981' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-08-19T14:24:40.815 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:24:40.860 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:24:40.884 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:40.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:40.885 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:24:40.986 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:24:41.011 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:41.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:41.012 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:24:41.113 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:24:41.135 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:41.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:41.136 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:24:41.237 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:24:41.259 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:41.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:41.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:24:41.318 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:24:41.319 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:24:41.361 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-08-19T14:24:41.362 INFO:tasks.radosbench:Beginning radosbench... 2024-08-19T14:24:41.362 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_3 2024-08-19T14:24:41.363 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd pool create unique_pool_3 16 2024-08-19T14:24:41.363 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:24:41.385 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:41.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:41.386 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:24:41.413 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:24:41.414 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:24:41.486 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:24:41.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:41 smithi032 ceph-mon[20004]: pgmap v960: 47 pgs: 47 active+clean; 577 KiB data, 4.2 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 61 op/s 2024-08-19T14:24:41.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:41 smithi032 ceph-mon[20004]: 1.18 scrub starts 2024-08-19T14:24:41.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:41 smithi032 ceph-mon[20004]: 1.18 scrub ok 2024-08-19T14:24:41.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:41 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1676690981' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-08-19T14:24:41.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:41 smithi032 ceph-mon[20004]: osdmap e418: 12 total, 12 up, 9 in 2024-08-19T14:24:41.508 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:41.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:41.509 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:24:41.610 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:24:41.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:41 smithi031 ceph-mon[16336]: pgmap v960: 47 pgs: 47 active+clean; 577 KiB data, 4.2 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 61 op/s 2024-08-19T14:24:41.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:41 smithi031 ceph-mon[16336]: 1.18 scrub starts 2024-08-19T14:24:41.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:41 smithi031 ceph-mon[16336]: 1.18 scrub ok 2024-08-19T14:24:41.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:41 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1676690981' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-08-19T14:24:41.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:41 smithi031 ceph-mon[16336]: osdmap e418: 12 total, 12 up, 9 in 2024-08-19T14:24:41.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:41.634 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:41.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:24:41.735 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:24:41.757 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:41.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:41.758 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:24:41.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:41 smithi045 ceph-mon[20143]: pgmap v960: 47 pgs: 47 active+clean; 577 KiB data, 4.2 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 61 op/s 2024-08-19T14:24:41.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:41 smithi045 ceph-mon[20143]: 1.18 scrub starts 2024-08-19T14:24:41.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:41 smithi045 ceph-mon[20143]: 1.18 scrub ok 2024-08-19T14:24:41.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:41 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1676690981' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-08-19T14:24:41.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:41 smithi045 ceph-mon[20143]: osdmap e418: 12 total, 12 up, 9 in 2024-08-19T14:24:41.859 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:24:41.883 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:41.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:41.884 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:24:41.920 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:24:41.921 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:24:41.930 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:24:41.939 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:24:41.940 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:24:41.961 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:24:41.962 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:24:41.972 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-08-19T14:24:41.979 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:24:41.979 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:24:41.984 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:24:41.995 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:24:41.996 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:24:42.012 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:24:42.012 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:24:42.029 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:24:42.029 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:24:42.036 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:42.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:42.038 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:24:42.046 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:24:42.046 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:24:42.063 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:24:42.063 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:24:42.080 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:24:42.080 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:24:42.097 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:24:42.098 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:24:42.115 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:24:42.115 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:24:42.139 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:24:42.165 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:42.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:42.166 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:24:42.236 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:24:42.266 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:24:42.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:42.289 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:42.289 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:24:42.390 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:24:42.412 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:42.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:42.413 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:24:42.514 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:24:42.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:42.536 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:42.536 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:24:42.561 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:42 smithi031 ceph-mon[16336]: 1.16 scrub starts 2024-08-19T14:24:42.561 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:42 smithi031 ceph-mon[16336]: 1.16 scrub ok 2024-08-19T14:24:42.561 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:42 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/4252726276' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:24:42.606 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-19T14:24:42.637 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:24:42.659 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:42.659 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:42 smithi032 ceph-mon[20004]: 1.16 scrub starts 2024-08-19T14:24:42.659 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:42 smithi032 ceph-mon[20004]: 1.16 scrub ok 2024-08-19T14:24:42.660 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:42 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/4252726276' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:24:42.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:42.660 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:24:42.695 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:24:42.695 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-08-19T14:24:42.696 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-08-19T14:24:42.696 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-08-19T14:24:42.696 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.696 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25057.0:50176 3.14 3.f91a7cf4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-08-19T14:24:42.696 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:17:20.751296+0000", 2024-08-19T14:24:42.696 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 441.94216214199997, 2024-08-19T14:24:42.696 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.99210832500000001, 2024-08-19T14:24:42.696 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:24:42.696 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:24:42.696 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:24:42.696 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:24:42.696 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25057", 2024-08-19T14:24:42.696 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:24:42.697 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 50176 2024-08-19T14:24:42.697 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.697 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:24:42.697 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.697 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:24:42.698 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:20.751296+0000", 2024-08-19T14:24:42.698 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:24:42.698 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.698 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.698 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:24:42.699 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:20.751296+0000", 2024-08-19T14:24:42.699 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:24:42.699 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.699 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.699 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:24:42.699 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:20.751297+0000", 2024-08-19T14:24:42.699 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.04e-06 2024-08-19T14:24:42.699 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.699 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.699 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:24:42.699 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:20.751354+0000", 2024-08-19T14:24:42.699 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6588000000000001e-05 2024-08-19T14:24:42.700 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.700 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.700 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:24:42.700 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:20.751365+0000", 2024-08-19T14:24:42.700 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0889e-05 2024-08-19T14:24:42.700 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.700 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.700 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:24:42.700 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:20.751371+0000", 2024-08-19T14:24:42.700 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6879999999999996e-06 2024-08-19T14:24:42.700 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.700 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.700 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:24:42.700 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:20.751421+0000", 2024-08-19T14:24:42.701 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0519000000000002e-05 2024-08-19T14:24:42.701 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.701 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.701 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:24:42.701 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:20.751430+0000", 2024-08-19T14:24:42.701 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.1200000000000008e-06 2024-08-19T14:24:42.701 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.701 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.702 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:24:42.702 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:21.743385+0000", 2024-08-19T14:24:42.702 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.99195432400000005 2024-08-19T14:24:42.702 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.702 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.702 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:24:42.702 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:21.743405+0000", 2024-08-19T14:24:42.702 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0157e-05 2024-08-19T14:24:42.702 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:24:42.702 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:24:42.702 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:24:42.702 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.702 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.703 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25057.0:50187 3.11 3.8832571 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-08-19T14:24:42.703 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:17:20.755221+0000", 2024-08-19T14:24:42.703 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 441.93823795700001, 2024-08-19T14:24:42.703 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.98787882999999999, 2024-08-19T14:24:42.703 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:24:42.703 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:24:42.704 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:24:42.704 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:24:42.704 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25057", 2024-08-19T14:24:42.704 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:24:42.704 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 50187 2024-08-19T14:24:42.704 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.704 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:24:42.704 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.704 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:24:42.704 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:20.755221+0000", 2024-08-19T14:24:42.705 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:24:42.705 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.705 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.705 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:24:42.705 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:20.755221+0000", 2024-08-19T14:24:42.705 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:24:42.705 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.705 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.705 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:24:42.705 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:20.755222+0000", 2024-08-19T14:24:42.705 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.133e-06 2024-08-19T14:24:42.705 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.706 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.706 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:24:42.706 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:20.755306+0000", 2024-08-19T14:24:42.706 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.4384999999999999e-05 2024-08-19T14:24:42.706 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.706 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.707 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:24:42.707 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:20.755319+0000", 2024-08-19T14:24:42.707 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3368e-05 2024-08-19T14:24:42.707 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.707 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.708 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:24:42.708 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:20.755327+0000", 2024-08-19T14:24:42.708 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.9529999999999999e-06 2024-08-19T14:24:42.708 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.708 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.708 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:24:42.708 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:20.755394+0000", 2024-08-19T14:24:42.708 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6836999999999997e-05 2024-08-19T14:24:42.708 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.708 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.708 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:24:42.709 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:20.755404+0000", 2024-08-19T14:24:42.709 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.022e-05 2024-08-19T14:24:42.709 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.709 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.709 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:24:42.709 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:21.743056+0000", 2024-08-19T14:24:42.709 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.98765108499999998 2024-08-19T14:24:42.709 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.709 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.709 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:24:42.709 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:21.743099+0000", 2024-08-19T14:24:42.709 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3849e-05 2024-08-19T14:24:42.709 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:24:42.710 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:24:42.710 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:24:42.710 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.710 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.710 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25057.0:50187 3.11 3:8ea4c110:::benchmark_data_smithi045_59973_object50186:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e237)", 2024-08-19T14:24:42.710 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:17:22.758606+0000", 2024-08-19T14:24:42.710 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 439.93485245900001, 2024-08-19T14:24:42.710 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.22265618500000001, 2024-08-19T14:24:42.710 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:24:42.710 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:24:42.710 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:24:42.710 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:24:42.710 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25057", 2024-08-19T14:24:42.711 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:24:42.711 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 50187 2024-08-19T14:24:42.711 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.711 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:24:42.711 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.711 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:24:42.711 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:22.758606+0000", 2024-08-19T14:24:42.711 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:24:42.711 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.711 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.711 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:24:42.711 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:22.758606+0000", 2024-08-19T14:24:42.711 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:24:42.711 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.712 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.712 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:24:42.712 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:22.758608+0000", 2024-08-19T14:24:42.712 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.967e-06 2024-08-19T14:24:42.712 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.712 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.712 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:24:42.712 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:22.758658+0000", 2024-08-19T14:24:42.712 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9842000000000002e-05 2024-08-19T14:24:42.712 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.712 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.712 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:24:42.712 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:22.758674+0000", 2024-08-19T14:24:42.712 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6022999999999999e-05 2024-08-19T14:24:42.712 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.713 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.713 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:24:42.713 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:22.758686+0000", 2024-08-19T14:24:42.713 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1895999999999999e-05 2024-08-19T14:24:42.713 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.713 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.713 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:24:42.713 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:22.759828+0000", 2024-08-19T14:24:42.713 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001142636 2024-08-19T14:24:42.713 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.713 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.713 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:24:42.713 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:22.759845+0000", 2024-08-19T14:24:42.713 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6158e-05 2024-08-19T14:24:42.714 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.714 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.714 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:24:42.714 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:22.881339+0000", 2024-08-19T14:24:42.714 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.121494108 2024-08-19T14:24:42.714 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.714 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.714 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:24:42.715 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:22.881449+0000", 2024-08-19T14:24:42.715 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000110331 2024-08-19T14:24:42.715 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.715 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.715 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0,3", 2024-08-19T14:24:42.715 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:22.881562+0000", 2024-08-19T14:24:42.715 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00011250499999999999 2024-08-19T14:24:42.715 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.715 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.715 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:24:42.715 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:22.882268+0000", 2024-08-19T14:24:42.716 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00070606200000000003 2024-08-19T14:24:42.716 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.716 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.716 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:24:42.716 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:22.891817+0000", 2024-08-19T14:24:42.716 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0095496739999999993 2024-08-19T14:24:42.716 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.716 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.716 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:24:42.716 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:22.980253+0000", 2024-08-19T14:24:42.716 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.088435402999999996 2024-08-19T14:24:42.716 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.716 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.716 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:24:42.716 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:22.980330+0000", 2024-08-19T14:24:42.717 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7447000000000006e-05 2024-08-19T14:24:42.717 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.717 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.717 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:24:42.717 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:22.981262+0000", 2024-08-19T14:24:42.717 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00093213300000000003 2024-08-19T14:24:42.717 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:24:42.717 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:24:42.717 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:24:42.717 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.717 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.717 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25057.0:63305 3.1a 3:594f84d9:::benchmark_data_smithi045_59973_object63304:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e244)", 2024-08-19T14:24:42.717 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:17:33.888645+0000", 2024-08-19T14:24:42.717 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 428.80481352200002, 2024-08-19T14:24:42.718 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0982461939999999, 2024-08-19T14:24:42.718 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:24:42.718 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:24:42.718 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:24:42.718 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:24:42.718 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25057", 2024-08-19T14:24:42.718 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:24:42.718 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 63305 2024-08-19T14:24:42.718 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.719 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:24:42.719 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.719 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:24:42.719 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:33.888645+0000", 2024-08-19T14:24:42.719 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:24:42.719 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.719 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.719 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:24:42.719 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:33.888645+0000", 2024-08-19T14:24:42.719 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:24:42.719 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.719 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.719 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:24:42.719 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:33.888646+0000", 2024-08-19T14:24:42.720 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2550000000000001e-06 2024-08-19T14:24:42.720 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.720 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.720 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:24:42.720 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:33.888765+0000", 2024-08-19T14:24:42.720 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00011885 2024-08-19T14:24:42.720 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.720 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.720 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:24:42.720 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:33.888786+0000", 2024-08-19T14:24:42.720 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0828000000000001e-05 2024-08-19T14:24:42.720 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.720 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.721 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:24:42.721 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:33.888795+0000", 2024-08-19T14:24:42.721 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.1649999999999995e-06 2024-08-19T14:24:42.721 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.721 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.721 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:24:42.721 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:33.996820+0000", 2024-08-19T14:24:42.721 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.108024672 2024-08-19T14:24:42.721 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.721 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.721 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:24:42.721 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:33.996825+0000", 2024-08-19T14:24:42.721 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8149999999999996e-06 2024-08-19T14:24:42.722 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.722 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.722 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:24:42.722 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:34.986807+0000", 2024-08-19T14:24:42.722 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.98998204300000003 2024-08-19T14:24:42.722 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.722 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.722 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:24:42.722 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:34.986891+0000", 2024-08-19T14:24:42.722 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.4566000000000003e-05 2024-08-19T14:24:42.722 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:24:42.722 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:24:42.722 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:24:42.723 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.723 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.723 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25057.0:63311 3.1a 3:5f4435a5:::benchmark_data_smithi045_59973_object63310:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e244)", 2024-08-19T14:24:42.723 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:17:33.888830+0000", 2024-08-19T14:24:42.723 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 428.80462849399999, 2024-08-19T14:24:42.723 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.098523232, 2024-08-19T14:24:42.723 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:24:42.723 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:24:42.723 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:24:42.723 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:24:42.723 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25057", 2024-08-19T14:24:42.723 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:24:42.723 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 63311 2024-08-19T14:24:42.723 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.724 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:24:42.724 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.724 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:24:42.724 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:33.888830+0000", 2024-08-19T14:24:42.724 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:24:42.724 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.724 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.724 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:24:42.724 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:33.888830+0000", 2024-08-19T14:24:42.724 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:24:42.724 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.724 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.724 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:24:42.725 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:33.888831+0000", 2024-08-19T14:24:42.725 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.99e-07 2024-08-19T14:24:42.725 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.725 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.725 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:24:42.725 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:33.888901+0000", 2024-08-19T14:24:42.726 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9948000000000002e-05 2024-08-19T14:24:42.726 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.726 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.726 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:24:42.726 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:33.888911+0000", 2024-08-19T14:24:42.727 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0679000000000001e-05 2024-08-19T14:24:42.727 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.727 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.727 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:24:42.727 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:33.888918+0000", 2024-08-19T14:24:42.727 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.5880000000000003e-06 2024-08-19T14:24:42.727 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.728 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.728 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:24:42.728 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:33.997358+0000", 2024-08-19T14:24:42.728 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.108439887 2024-08-19T14:24:42.728 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.728 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.728 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:24:42.728 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:33.997362+0000", 2024-08-19T14:24:42.729 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1219999999999997e-06 2024-08-19T14:24:42.729 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.729 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.729 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:24:42.729 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:34.987287+0000", 2024-08-19T14:24:42.729 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.989925481 2024-08-19T14:24:42.729 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.729 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.729 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:24:42.729 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:34.987353+0000", 2024-08-19T14:24:42.729 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.5827999999999994e-05 2024-08-19T14:24:42.729 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:24:42.729 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:24:42.729 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:24:42.730 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.730 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.730 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25057.0:64606 3.1b 3.702ddf7b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e246)", 2024-08-19T14:24:42.730 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:17:35.969630+0000", 2024-08-19T14:24:42.730 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 426.723828787, 2024-08-19T14:24:42.730 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.99329972499999997, 2024-08-19T14:24:42.730 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:24:42.730 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:24:42.730 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:24:42.730 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:24:42.730 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25057", 2024-08-19T14:24:42.730 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:24:42.730 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 64606 2024-08-19T14:24:42.731 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.731 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:24:42.731 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.731 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:24:42.731 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.969630+0000", 2024-08-19T14:24:42.732 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:24:42.733 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.733 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.733 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:24:42.733 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.969630+0000", 2024-08-19T14:24:42.733 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:24:42.733 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.733 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.733 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:24:42.733 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.969632+0000", 2024-08-19T14:24:42.733 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9069999999999999e-06 2024-08-19T14:24:42.733 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.734 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.734 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:24:42.734 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.969667+0000", 2024-08-19T14:24:42.734 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5187e-05 2024-08-19T14:24:42.734 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.734 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.734 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:24:42.734 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.969678+0000", 2024-08-19T14:24:42.734 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0849000000000001e-05 2024-08-19T14:24:42.735 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.735 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.735 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:24:42.735 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.969685+0000", 2024-08-19T14:24:42.735 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5039999999999996e-06 2024-08-19T14:24:42.735 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.735 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.736 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:24:42.736 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.969734+0000", 2024-08-19T14:24:42.736 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8690000000000003e-05 2024-08-19T14:24:42.736 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.736 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.736 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:24:42.736 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.969744+0000", 2024-08-19T14:24:42.736 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0183999999999999e-05 2024-08-19T14:24:42.736 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.736 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.736 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:24:42.737 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:36.962899+0000", 2024-08-19T14:24:42.737 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.99315520700000004 2024-08-19T14:24:42.737 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.737 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.737 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:24:42.737 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:36.962929+0000", 2024-08-19T14:24:42.737 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0196999999999999e-05 2024-08-19T14:24:42.737 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:24:42.737 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:24:42.737 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:24:42.738 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.738 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.738 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25057.0:64610 3.1b 3.810f60bb (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e246)", 2024-08-19T14:24:42.738 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:17:35.977398+0000", 2024-08-19T14:24:42.738 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 426.71606006299999, 2024-08-19T14:24:42.738 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.98556503799999995, 2024-08-19T14:24:42.738 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:24:42.738 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:24:42.738 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:24:42.738 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:24:42.739 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25057", 2024-08-19T14:24:42.739 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:24:42.739 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 64610 2024-08-19T14:24:42.739 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.739 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:24:42.739 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.739 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:24:42.739 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.977398+0000", 2024-08-19T14:24:42.739 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:24:42.739 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.739 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.739 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:24:42.739 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.977398+0000", 2024-08-19T14:24:42.739 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:24:42.740 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.740 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.740 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:24:42.740 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.977400+0000", 2024-08-19T14:24:42.740 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.469e-06 2024-08-19T14:24:42.740 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.740 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.740 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:24:42.740 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.977457+0000", 2024-08-19T14:24:42.740 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7043999999999998e-05 2024-08-19T14:24:42.740 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.740 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.740 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:24:42.740 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.977465+0000", 2024-08-19T14:24:42.741 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.2320000000000001e-06 2024-08-19T14:24:42.741 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.741 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.741 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:24:42.741 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.977470+0000", 2024-08-19T14:24:42.741 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5759999999999999e-06 2024-08-19T14:24:42.741 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.741 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.741 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:24:42.742 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.978418+0000", 2024-08-19T14:24:42.742 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00094818599999999995 2024-08-19T14:24:42.742 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.742 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.742 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:24:42.742 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.978424+0000", 2024-08-19T14:24:42.742 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2979999999999997e-06 2024-08-19T14:24:42.742 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.742 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.742 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:24:42.742 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:36.962956+0000", 2024-08-19T14:24:42.742 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.98453159300000004 2024-08-19T14:24:42.742 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.743 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.743 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:24:42.743 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:36.962963+0000", 2024-08-19T14:24:42.743 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6399999999999997e-06 2024-08-19T14:24:42.743 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:24:42.743 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:24:42.743 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:24:42.743 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.743 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.743 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25057.0:64616 3.1b 3.cab7ee3b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e246)", 2024-08-19T14:24:42.743 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:17:35.982530+0000", 2024-08-19T14:24:42.743 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 426.71092881999999, 2024-08-19T14:24:42.743 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.98047576599999997, 2024-08-19T14:24:42.743 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:24:42.744 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:24:42.744 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:24:42.744 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:24:42.744 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25057", 2024-08-19T14:24:42.744 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:24:42.744 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 64616 2024-08-19T14:24:42.744 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.744 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:24:42.744 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.744 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:24:42.744 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.982530+0000", 2024-08-19T14:24:42.744 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:24:42.744 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.744 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.745 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:24:42.745 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.982530+0000", 2024-08-19T14:24:42.745 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:24:42.745 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.745 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.745 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:24:42.745 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.982531+0000", 2024-08-19T14:24:42.745 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.626e-06 2024-08-19T14:24:42.745 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.745 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.745 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:24:42.745 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.982589+0000", 2024-08-19T14:24:42.745 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8013000000000002e-05 2024-08-19T14:24:42.745 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.745 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.746 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:24:42.746 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.982603+0000", 2024-08-19T14:24:42.746 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.384e-05 2024-08-19T14:24:42.746 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.746 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.746 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:24:42.746 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.982611+0000", 2024-08-19T14:24:42.746 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6329999999999996e-06 2024-08-19T14:24:42.746 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.746 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.747 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:24:42.747 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.982674+0000", 2024-08-19T14:24:42.747 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3575e-05 2024-08-19T14:24:42.747 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.747 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.747 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:24:42.748 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.982686+0000", 2024-08-19T14:24:42.748 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1929999999999999e-05 2024-08-19T14:24:42.748 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.748 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.748 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:24:42.748 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:36.962998+0000", 2024-08-19T14:24:42.748 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.98031178900000004 2024-08-19T14:24:42.748 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.749 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.749 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:24:42.749 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:36.963005+0000", 2024-08-19T14:24:42.749 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3599999999999998e-06 2024-08-19T14:24:42.749 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:24:42.749 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:24:42.749 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:24:42.749 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.749 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.749 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25057.0:64589 3.1e 3:7d191b55:::benchmark_data_smithi045_59973_object64588:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e248)", 2024-08-19T14:24:42.749 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:17:37.897758+0000", 2024-08-19T14:24:42.749 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 424.795700748, 2024-08-19T14:24:42.749 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20971097999999999, 2024-08-19T14:24:42.749 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:24:42.750 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:24:42.750 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:24:42.750 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:24:42.751 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25057", 2024-08-19T14:24:42.751 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:24:42.751 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 64589 2024-08-19T14:24:42.751 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.751 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:24:42.751 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.751 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:24:42.752 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:37.897758+0000", 2024-08-19T14:24:42.752 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:24:42.752 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.752 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.752 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:24:42.752 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:37.897758+0000", 2024-08-19T14:24:42.752 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:24:42.752 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.752 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.752 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:24:42.752 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:37.897758+0000", 2024-08-19T14:24:42.752 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3300000000000003e-07 2024-08-19T14:24:42.752 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.753 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.753 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:24:42.753 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:37.897792+0000", 2024-08-19T14:24:42.753 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3327000000000001e-05 2024-08-19T14:24:42.753 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.753 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.753 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:24:42.753 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:37.897800+0000", 2024-08-19T14:24:42.753 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.5129999999999996e-06 2024-08-19T14:24:42.753 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.753 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.753 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:24:42.754 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:37.897805+0000", 2024-08-19T14:24:42.754 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9869999999999999e-06 2024-08-19T14:24:42.754 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.754 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.754 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:24:42.754 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:37.962303+0000", 2024-08-19T14:24:42.754 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.064497909000000006 2024-08-19T14:24:42.754 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.754 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.754 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for degraded object", 2024-08-19T14:24:42.754 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:37.962758+0000", 2024-08-19T14:24:42.754 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00045486799999999998 2024-08-19T14:24:42.754 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.755 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.755 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:24:42.755 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:38.107362+0000", 2024-08-19T14:24:42.755 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.144604434 2024-08-19T14:24:42.755 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.755 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.755 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:24:42.755 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:38.107469+0000", 2024-08-19T14:24:42.755 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00010650900000000001 2024-08-19T14:24:42.755 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:24:42.755 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:24:42.756 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:24:42.756 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.756 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.756 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25057.0:64659 3.1e 3:7bada211:::benchmark_data_smithi045_59973_object64658:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e248)", 2024-08-19T14:24:42.756 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:17:37.897812+0000", 2024-08-19T14:24:42.756 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 424.79564638699998, 2024-08-19T14:24:42.756 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.21203138599999999, 2024-08-19T14:24:42.756 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:24:42.756 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:24:42.756 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:24:42.756 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:24:42.756 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25057", 2024-08-19T14:24:42.756 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:24:42.756 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 64659 2024-08-19T14:24:42.757 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.757 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:24:42.757 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.757 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:24:42.757 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:37.897812+0000", 2024-08-19T14:24:42.757 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:24:42.757 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.757 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.757 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:24:42.757 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:37.897812+0000", 2024-08-19T14:24:42.757 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:24:42.758 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.758 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.758 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:24:42.758 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:37.897813+0000", 2024-08-19T14:24:42.758 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2199999999999999e-07 2024-08-19T14:24:42.758 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.758 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.758 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:24:42.759 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:37.897845+0000", 2024-08-19T14:24:42.759 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2447999999999998e-05 2024-08-19T14:24:42.759 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.759 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.759 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:24:42.759 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:37.897852+0000", 2024-08-19T14:24:42.759 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6640000000000001e-06 2024-08-19T14:24:42.759 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.760 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.760 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:24:42.760 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:37.897856+0000", 2024-08-19T14:24:42.760 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7180000000000004e-06 2024-08-19T14:24:42.760 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.760 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.760 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:24:42.760 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:37.962786+0000", 2024-08-19T14:24:42.760 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.064930102000000003 2024-08-19T14:24:42.760 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.760 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.760 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:24:42.760 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:37.962879+0000", 2024-08-19T14:24:42.760 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.2677999999999996e-05 2024-08-19T14:24:42.761 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.761 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.761 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 3,8,9", 2024-08-19T14:24:42.761 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:37.963005+0000", 2024-08-19T14:24:42.761 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012557399999999999 2024-08-19T14:24:42.761 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.761 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.761 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:24:42.761 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:37.964594+0000", 2024-08-19T14:24:42.761 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001589008 2024-08-19T14:24:42.761 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.761 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.761 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:24:42.762 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:37.969399+0000", 2024-08-19T14:24:42.762 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0048054600000000001 2024-08-19T14:24:42.762 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.762 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.762 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:24:42.762 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:37.971415+0000", 2024-08-19T14:24:42.762 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0020157700000000001 2024-08-19T14:24:42.762 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.762 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.762 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:24:42.762 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:38.107519+0000", 2024-08-19T14:24:42.762 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.136104371 2024-08-19T14:24:42.762 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.763 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.763 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:24:42.763 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:38.108261+0000", 2024-08-19T14:24:42.763 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00074117400000000004 2024-08-19T14:24:42.763 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.763 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.763 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:24:42.763 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:38.109844+0000", 2024-08-19T14:24:42.763 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001582997 2024-08-19T14:24:42.763 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:24:42.763 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:24:42.764 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:24:42.764 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.764 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.764 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25057.0:188779 3.2a 3:54ce2294:::benchmark_data_smithi045_59973_object57600:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-08-19T14:24:42.764 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:19:02.351958+0000", 2024-08-19T14:24:42.764 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 340.341500591, 2024-08-19T14:24:42.764 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.86379963100000001, 2024-08-19T14:24:42.765 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:24:42.765 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:24:42.765 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:24:42.765 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:24:42.765 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25057", 2024-08-19T14:24:42.765 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:24:42.765 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 188779 2024-08-19T14:24:42.765 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.765 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:24:42.766 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.766 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:24:42.766 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:19:02.351958+0000", 2024-08-19T14:24:42.766 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:24:42.766 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.766 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.766 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:24:42.766 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:19:02.351958+0000", 2024-08-19T14:24:42.766 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:24:42.766 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.766 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.767 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:24:42.767 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:19:02.351959+0000", 2024-08-19T14:24:42.767 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.3200000000000004e-07 2024-08-19T14:24:42.767 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.767 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.767 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:24:42.767 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:19:02.351962+0000", 2024-08-19T14:24:42.767 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.32e-06 2024-08-19T14:24:42.767 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.767 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.767 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:24:42.767 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:19:02.351970+0000", 2024-08-19T14:24:42.767 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6790000000000005e-06 2024-08-19T14:24:42.767 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.768 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.768 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:24:42.768 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:19:02.351974+0000", 2024-08-19T14:24:42.768 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5979999999999999e-06 2024-08-19T14:24:42.768 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.768 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.768 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:24:42.768 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:19:02.352047+0000", 2024-08-19T14:24:42.768 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2187999999999994e-05 2024-08-19T14:24:42.768 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.768 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.768 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:24:42.768 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:19:02.352057+0000", 2024-08-19T14:24:42.769 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0642000000000001e-05 2024-08-19T14:24:42.769 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.769 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.769 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:24:42.769 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:19:03.210075+0000", 2024-08-19T14:24:42.769 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.85801831699999997 2024-08-19T14:24:42.769 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.769 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.769 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:24:42.769 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:19:03.210203+0000", 2024-08-19T14:24:42.769 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012764300000000001 2024-08-19T14:24:42.770 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.770 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.770 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 1,8", 2024-08-19T14:24:42.770 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:19:03.210318+0000", 2024-08-19T14:24:42.770 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000114954 2024-08-19T14:24:42.770 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.770 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.770 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:24:42.770 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:19:03.212413+0000", 2024-08-19T14:24:42.770 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0020953249999999999 2024-08-19T14:24:42.770 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.770 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.770 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:24:42.771 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:19:03.212766+0000", 2024-08-19T14:24:42.771 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00035248899999999999 2024-08-19T14:24:42.771 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.771 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.771 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:24:42.771 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:19:03.215663+0000", 2024-08-19T14:24:42.772 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0028968959999999999 2024-08-19T14:24:42.772 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.772 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.772 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:24:42.772 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:19:03.215714+0000", 2024-08-19T14:24:42.772 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0719e-05 2024-08-19T14:24:42.772 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.773 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.773 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:24:42.773 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:19:03.215758+0000", 2024-08-19T14:24:42.773 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4029000000000001e-05 2024-08-19T14:24:42.773 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:24:42.773 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:24:42.774 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:24:42.774 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.774 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.774 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25333.0:54277 4.d 4:b77eedff:::benchmark_data_smithi045_71562_object54276:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-08-19T14:24:42.774 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:20:57.352004+0000", 2024-08-19T14:24:42.774 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 225.34145481600001, 2024-08-19T14:24:42.775 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.209036954, 2024-08-19T14:24:42.775 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:24:42.775 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:24:42.775 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:24:42.775 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:24:42.775 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25333", 2024-08-19T14:24:42.776 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:24:42.776 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 54277 2024-08-19T14:24:42.776 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.776 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:24:42.776 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.776 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:24:42.777 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:57.352004+0000", 2024-08-19T14:24:42.777 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:24:42.777 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.777 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.777 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:24:42.777 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:57.352004+0000", 2024-08-19T14:24:42.778 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:24:42.778 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.778 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.778 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:24:42.778 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:57.352005+0000", 2024-08-19T14:24:42.778 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.035e-06 2024-08-19T14:24:42.779 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.779 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.779 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:24:42.779 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:57.352034+0000", 2024-08-19T14:24:42.779 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9580000000000001e-05 2024-08-19T14:24:42.779 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.780 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.780 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:24:42.780 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:57.352044+0000", 2024-08-19T14:24:42.780 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0104e-05 2024-08-19T14:24:42.780 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.780 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.781 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:24:42.781 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:57.352052+0000", 2024-08-19T14:24:42.781 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.148e-06 2024-08-19T14:24:42.781 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.781 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.781 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:24:42.782 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:57.352111+0000", 2024-08-19T14:24:42.782 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9392000000000003e-05 2024-08-19T14:24:42.782 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.782 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.782 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:24:42.783 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:57.352268+0000", 2024-08-19T14:24:42.783 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000157372 2024-08-19T14:24:42.783 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.783 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.783 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 1,9", 2024-08-19T14:24:42.783 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:57.352417+0000", 2024-08-19T14:24:42.784 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014890199999999999 2024-08-19T14:24:42.784 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.784 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.785 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:24:42.785 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:57.354049+0000", 2024-08-19T14:24:42.785 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0016321650000000001 2024-08-19T14:24:42.785 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.785 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.785 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:24:42.786 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:57.354867+0000", 2024-08-19T14:24:42.786 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00081795800000000003 2024-08-19T14:24:42.786 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.786 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.786 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:24:42.786 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:57.560974+0000", 2024-08-19T14:24:42.787 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20610681 2024-08-19T14:24:42.787 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.787 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.787 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:24:42.787 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:57.561017+0000", 2024-08-19T14:24:42.787 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2531000000000003e-05 2024-08-19T14:24:42.788 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.788 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.788 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:24:42.788 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:57.561041+0000", 2024-08-19T14:24:42.788 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3957000000000001e-05 2024-08-19T14:24:42.788 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:24:42.788 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:24:42.788 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:24:42.788 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.789 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.789 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25333.0:56673 4.e 4:7feebc47:::benchmark_data_smithi045_71562_object56672:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-08-19T14:24:42.789 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:20:59.123818+0000", 2024-08-19T14:24:42.789 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 223.569640814, 2024-08-19T14:24:42.789 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20982393999999999, 2024-08-19T14:24:42.789 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:24:42.789 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:24:42.789 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:24:42.789 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:24:42.789 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25333", 2024-08-19T14:24:42.789 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:24:42.789 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 56673 2024-08-19T14:24:42.789 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.789 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:24:42.790 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.790 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:24:42.790 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:59.123818+0000", 2024-08-19T14:24:42.790 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:24:42.790 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.790 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.790 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:24:42.790 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:59.123818+0000", 2024-08-19T14:24:42.790 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:24:42.790 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.791 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.791 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:24:42.791 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:59.123819+0000", 2024-08-19T14:24:42.791 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2079999999999999e-06 2024-08-19T14:24:42.791 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.791 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.791 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:24:42.791 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:59.123880+0000", 2024-08-19T14:24:42.791 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0890000000000001e-05 2024-08-19T14:24:42.791 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.791 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.791 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:24:42.791 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:59.123895+0000", 2024-08-19T14:24:42.791 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5284000000000001e-05 2024-08-19T14:24:42.792 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.792 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.792 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:24:42.792 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:59.123903+0000", 2024-08-19T14:24:42.792 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.0299999999999994e-06 2024-08-19T14:24:42.792 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.792 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.792 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:24:42.792 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:59.124075+0000", 2024-08-19T14:24:42.792 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00017151100000000001 2024-08-19T14:24:42.792 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.792 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.792 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:24:42.793 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:59.124207+0000", 2024-08-19T14:24:42.793 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013249600000000001 2024-08-19T14:24:42.793 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.793 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.793 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 1,3", 2024-08-19T14:24:42.793 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:59.124291+0000", 2024-08-19T14:24:42.793 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.4339999999999995e-05 2024-08-19T14:24:42.793 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.793 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.793 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:24:42.793 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:59.125565+0000", 2024-08-19T14:24:42.793 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0012739489999999999 2024-08-19T14:24:42.793 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.793 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.793 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:24:42.794 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:59.128881+0000", 2024-08-19T14:24:42.794 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.003316079 2024-08-19T14:24:42.794 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.794 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.794 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:24:42.794 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:59.333544+0000", 2024-08-19T14:24:42.794 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20466241900000001 2024-08-19T14:24:42.794 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.794 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.794 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:24:42.794 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:59.333604+0000", 2024-08-19T14:24:42.794 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0569000000000002e-05 2024-08-19T14:24:42.794 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.795 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.795 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:24:42.795 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:20:59.333642+0000", 2024-08-19T14:24:42.795 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7165000000000001e-05 2024-08-19T14:24:42.795 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:24:42.795 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:24:42.795 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:24:42.795 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.795 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.795 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25333.0:149067 4.e 4:7e247c1a:::benchmark_data_smithi045_71562_object17892:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e378)", 2024-08-19T14:24:42.795 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:22:24.357990+0000", 2024-08-19T14:24:42.795 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 138.335468029, 2024-08-19T14:24:42.795 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.22065488899999999, 2024-08-19T14:24:42.795 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:24:42.796 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:24:42.796 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:24:42.796 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:24:42.796 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25333", 2024-08-19T14:24:42.796 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:24:42.796 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 149067 2024-08-19T14:24:42.796 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.796 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:24:42.796 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.796 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:24:42.796 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:22:24.357990+0000", 2024-08-19T14:24:42.796 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:24:42.796 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.796 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.797 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:24:42.797 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:22:24.357990+0000", 2024-08-19T14:24:42.797 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:24:42.797 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.797 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.797 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:24:42.797 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:22:24.357992+0000", 2024-08-19T14:24:42.797 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.499e-06 2024-08-19T14:24:42.797 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.797 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.797 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:24:42.797 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:22:24.357998+0000", 2024-08-19T14:24:42.797 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1210000000000003e-06 2024-08-19T14:24:42.797 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.798 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.798 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:24:42.798 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:22:24.358011+0000", 2024-08-19T14:24:42.798 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2959000000000001e-05 2024-08-19T14:24:42.798 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.798 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.798 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:24:42.798 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:22:24.358019+0000", 2024-08-19T14:24:42.798 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1529999999999997e-06 2024-08-19T14:24:42.798 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.799 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.799 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:24:42.799 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:22:24.358081+0000", 2024-08-19T14:24:42.799 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1508999999999997e-05 2024-08-19T14:24:42.799 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.799 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.799 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:24:42.799 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:22:24.358218+0000", 2024-08-19T14:24:42.799 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013742 2024-08-19T14:24:42.799 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.799 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.799 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 1,3", 2024-08-19T14:24:42.799 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:22:24.358334+0000", 2024-08-19T14:24:42.799 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000116048 2024-08-19T14:24:42.800 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.800 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.800 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:24:42.800 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:22:24.367414+0000", 2024-08-19T14:24:42.800 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0090802490000000003 2024-08-19T14:24:42.800 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.800 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.800 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:24:42.800 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:22:24.368374+0000", 2024-08-19T14:24:42.801 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00095930899999999999 2024-08-19T14:24:42.801 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.801 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.801 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:24:42.801 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:22:24.578569+0000", 2024-08-19T14:24:42.801 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.210195359 2024-08-19T14:24:42.801 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.801 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.801 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:24:42.801 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:22:24.578607+0000", 2024-08-19T14:24:42.801 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7969999999999997e-05 2024-08-19T14:24:42.801 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.802 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.802 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:24:42.802 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:22:24.578645+0000", 2024-08-19T14:24:42.802 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8293000000000001e-05 2024-08-19T14:24:42.802 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:24:42.802 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:24:42.802 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:24:42.802 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.802 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.802 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25333.0:149070 4.e 4:77cd6059:::benchmark_data_smithi045_71562_object17895:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e378)", 2024-08-19T14:24:42.803 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:22:24.358621+0000", 2024-08-19T14:24:42.803 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 138.334837085, 2024-08-19T14:24:42.803 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.223820359, 2024-08-19T14:24:42.803 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:24:42.803 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:24:42.803 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:24:42.803 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:24:42.803 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25333", 2024-08-19T14:24:42.803 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:24:42.803 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 149070 2024-08-19T14:24:42.803 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.803 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:24:42.803 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.803 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:24:42.803 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:22:24.358621+0000", 2024-08-19T14:24:42.804 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:24:42.804 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.804 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.804 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:24:42.804 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:22:24.358621+0000", 2024-08-19T14:24:42.804 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:24:42.804 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.804 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.804 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:24:42.805 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:22:24.358622+0000", 2024-08-19T14:24:42.805 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.0999999999999997e-07 2024-08-19T14:24:42.805 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.805 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.805 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:24:42.805 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:22:24.358627+0000", 2024-08-19T14:24:42.805 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3760000000000001e-06 2024-08-19T14:24:42.805 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.805 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.805 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:24:42.806 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:22:24.358642+0000", 2024-08-19T14:24:42.806 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5582000000000001e-05 2024-08-19T14:24:42.806 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.806 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.806 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:24:42.806 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:22:24.358650+0000", 2024-08-19T14:24:42.806 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5079999999999999e-06 2024-08-19T14:24:42.806 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.807 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.807 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:24:42.807 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:22:24.362772+0000", 2024-08-19T14:24:42.807 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0041220220000000004 2024-08-19T14:24:42.807 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.807 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.807 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:24:42.807 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:22:24.362918+0000", 2024-08-19T14:24:42.807 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014636200000000001 2024-08-19T14:24:42.808 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.808 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.808 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 1,3", 2024-08-19T14:24:42.808 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:22:24.363057+0000", 2024-08-19T14:24:42.808 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013936800000000001 2024-08-19T14:24:42.808 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.808 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.808 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:24:42.809 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:22:24.367511+0000", 2024-08-19T14:24:42.809 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0044538939999999999 2024-08-19T14:24:42.809 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.809 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.809 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:24:42.809 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:22:24.379873+0000", 2024-08-19T14:24:42.809 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.01236145 2024-08-19T14:24:42.809 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.809 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.809 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:24:42.809 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:22:24.582321+0000", 2024-08-19T14:24:42.809 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.202448131 2024-08-19T14:24:42.810 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.810 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.810 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:24:42.810 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:22:24.582389+0000", 2024-08-19T14:24:42.810 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7778000000000001e-05 2024-08-19T14:24:42.810 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.810 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.810 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:24:42.810 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:22:24.582442+0000", 2024-08-19T14:24:42.810 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3078000000000002e-05 2024-08-19T14:24:42.810 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:24:42.810 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:24:42.810 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:24:42.811 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.811 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.811 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25333.0:228222 4.d 4:b862cd65:::benchmark_data_smithi045_71562_object97047:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e400)", 2024-08-19T14:24:42.811 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:23:52.884596+0000", 2024-08-19T14:24:42.811 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 49.808862712, 2024-08-19T14:24:42.811 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.215425644, 2024-08-19T14:24:42.811 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:24:42.811 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:24:42.811 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:24:42.811 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:24:42.811 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25333", 2024-08-19T14:24:42.811 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:24:42.811 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 228222 2024-08-19T14:24:42.811 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.812 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:24:42.812 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.812 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:24:42.812 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:52.884596+0000", 2024-08-19T14:24:42.812 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:24:42.812 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.812 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.812 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:24:42.812 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:52.884596+0000", 2024-08-19T14:24:42.812 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:24:42.812 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.812 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.812 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:24:42.812 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:52.884597+0000", 2024-08-19T14:24:42.813 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2449999999999999e-06 2024-08-19T14:24:42.813 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.813 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.813 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:24:42.813 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:52.884602+0000", 2024-08-19T14:24:42.813 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1780000000000002e-06 2024-08-19T14:24:42.813 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.813 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.813 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:24:42.813 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:52.884613+0000", 2024-08-19T14:24:42.813 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0916e-05 2024-08-19T14:24:42.813 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.813 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.813 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:24:42.814 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:52.884619+0000", 2024-08-19T14:24:42.814 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6840000000000001e-06 2024-08-19T14:24:42.814 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.814 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.814 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:24:42.814 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:52.884673+0000", 2024-08-19T14:24:42.814 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4629999999999997e-05 2024-08-19T14:24:42.814 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.814 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.814 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:24:42.814 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:52.884798+0000", 2024-08-19T14:24:42.814 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000124523 2024-08-19T14:24:42.819 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.819 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.819 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 1,9", 2024-08-19T14:24:42.819 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:52.884882+0000", 2024-08-19T14:24:42.819 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.3702000000000006e-05 2024-08-19T14:24:42.819 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.819 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.819 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:24:42.819 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:52.889076+0000", 2024-08-19T14:24:42.820 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0041941749999999996 2024-08-19T14:24:42.820 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.820 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.820 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:24:42.820 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:52.889174+0000", 2024-08-19T14:24:42.820 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.7757999999999995e-05 2024-08-19T14:24:42.820 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.820 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.820 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:24:42.820 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:53.099923+0000", 2024-08-19T14:24:42.820 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.21074900399999999 2024-08-19T14:24:42.820 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.820 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.820 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:24:42.821 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:53.099977+0000", 2024-08-19T14:24:42.821 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4799999999999997e-05 2024-08-19T14:24:42.821 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.821 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.821 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:24:42.821 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:53.100021+0000", 2024-08-19T14:24:42.821 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4029000000000001e-05 2024-08-19T14:24:42.821 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:24:42.821 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:24:42.821 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:24:42.821 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.821 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.821 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25333.0:228230 4.e 4:7f46c467:::benchmark_data_smithi045_71562_object97055:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e400)", 2024-08-19T14:24:42.821 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:23:52.892067+0000", 2024-08-19T14:24:42.822 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 49.801391258000002, 2024-08-19T14:24:42.822 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.223615182, 2024-08-19T14:24:42.822 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:24:42.822 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:24:42.822 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:24:42.822 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:24:42.822 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25333", 2024-08-19T14:24:42.822 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:24:42.822 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 228230 2024-08-19T14:24:42.822 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.822 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:24:42.822 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.822 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:24:42.822 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:52.892067+0000", 2024-08-19T14:24:42.823 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:24:42.823 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.823 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.823 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:24:42.823 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:52.892067+0000", 2024-08-19T14:24:42.823 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:24:42.823 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.823 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.823 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:24:42.823 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:52.892069+0000", 2024-08-19T14:24:42.823 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2750000000000001e-06 2024-08-19T14:24:42.823 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.823 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.823 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:24:42.823 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:52.892073+0000", 2024-08-19T14:24:42.824 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3880000000000002e-06 2024-08-19T14:24:42.824 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.824 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.824 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:24:42.824 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:52.892083+0000", 2024-08-19T14:24:42.824 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.798e-06 2024-08-19T14:24:42.824 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.824 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.824 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:24:42.824 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:52.892090+0000", 2024-08-19T14:24:42.824 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9650000000000002e-06 2024-08-19T14:24:42.824 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.824 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.824 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:24:42.825 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:52.892145+0000", 2024-08-19T14:24:42.825 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4994999999999998e-05 2024-08-19T14:24:42.825 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.825 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.825 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:24:42.825 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:52.892270+0000", 2024-08-19T14:24:42.825 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012498000000000001 2024-08-19T14:24:42.825 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.825 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.825 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 1,3", 2024-08-19T14:24:42.825 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:52.892388+0000", 2024-08-19T14:24:42.825 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00011871600000000001 2024-08-19T14:24:42.825 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.825 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.825 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:24:42.826 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:52.898094+0000", 2024-08-19T14:24:42.826 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0057056449999999996 2024-08-19T14:24:42.826 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.826 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.826 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:24:42.826 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:52.898531+0000", 2024-08-19T14:24:42.826 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00043701699999999999 2024-08-19T14:24:42.826 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.826 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.826 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:24:42.826 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:53.115594+0000", 2024-08-19T14:24:42.826 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.21706310500000001 2024-08-19T14:24:42.826 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.826 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.827 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:24:42.827 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:53.115650+0000", 2024-08-19T14:24:42.827 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5398000000000001e-05 2024-08-19T14:24:42.827 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.827 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.827 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:24:42.827 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:53.115682+0000", 2024-08-19T14:24:42.827 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.29e-05 2024-08-19T14:24:42.827 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:24:42.827 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:24:42.827 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:24:42.827 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.827 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.827 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25333.0:228231 4.d 4:b8b5ade8:::benchmark_data_smithi045_71562_object97056:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e400)", 2024-08-19T14:24:42.828 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:23:52.892113+0000", 2024-08-19T14:24:42.828 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 49.801345718999997, 2024-08-19T14:24:42.828 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.22368043300000001, 2024-08-19T14:24:42.828 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:24:42.828 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:24:42.828 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:24:42.828 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:24:42.828 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25333", 2024-08-19T14:24:42.828 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:24:42.828 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 228231 2024-08-19T14:24:42.828 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.828 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:24:42.828 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.828 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:24:42.829 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:52.892113+0000", 2024-08-19T14:24:42.829 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:24:42.829 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.829 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.829 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:24:42.829 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:52.892113+0000", 2024-08-19T14:24:42.829 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:24:42.829 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.829 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.829 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:24:42.829 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:52.892113+0000", 2024-08-19T14:24:42.829 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.44e-07 2024-08-19T14:24:42.829 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.829 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.829 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:24:42.830 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:52.892116+0000", 2024-08-19T14:24:42.830 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8289999999999999e-06 2024-08-19T14:24:42.830 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.830 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.830 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:24:42.830 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:52.892124+0000", 2024-08-19T14:24:42.830 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.0520000000000002e-06 2024-08-19T14:24:42.830 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.830 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.830 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:24:42.830 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:52.892131+0000", 2024-08-19T14:24:42.830 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.387e-06 2024-08-19T14:24:42.830 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.831 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.831 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:24:42.831 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:52.892234+0000", 2024-08-19T14:24:42.831 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000103662 2024-08-19T14:24:42.831 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.831 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.831 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:24:42.831 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:52.892337+0000", 2024-08-19T14:24:42.831 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000103099 2024-08-19T14:24:42.831 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.831 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.831 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 1,9", 2024-08-19T14:24:42.831 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:52.892445+0000", 2024-08-19T14:24:42.831 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000107695 2024-08-19T14:24:42.831 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.832 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.832 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:24:42.832 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:52.897668+0000", 2024-08-19T14:24:42.832 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0052227059999999997 2024-08-19T14:24:42.832 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.832 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.832 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:24:42.832 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:52.898542+0000", 2024-08-19T14:24:42.832 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00087437400000000003 2024-08-19T14:24:42.832 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.832 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.832 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:24:42.832 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:53.115682+0000", 2024-08-19T14:24:42.832 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.217139837 2024-08-19T14:24:42.833 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.833 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.833 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:24:42.833 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:53.115748+0000", 2024-08-19T14:24:42.833 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.5968000000000003e-05 2024-08-19T14:24:42.833 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.833 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.833 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:24:42.833 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:53.115793+0000", 2024-08-19T14:24:42.833 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.528e-05 2024-08-19T14:24:42.833 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:24:42.833 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:24:42.833 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:24:42.833 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.833 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.834 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25333.0:228245 4.e 4:79a51a6c:::benchmark_data_smithi045_71562_object97070:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e400)", 2024-08-19T14:24:42.834 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:23:52.904041+0000", 2024-08-19T14:24:42.834 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 49.789417049999997, 2024-08-19T14:24:42.834 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.21178170299999999, 2024-08-19T14:24:42.834 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:24:42.834 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:24:42.834 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:24:42.834 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:24:42.834 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25333", 2024-08-19T14:24:42.834 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:24:42.834 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 228245 2024-08-19T14:24:42.834 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.834 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:24:42.835 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.835 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:24:42.835 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:52.904041+0000", 2024-08-19T14:24:42.835 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:24:42.835 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.835 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.835 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:24:42.835 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:52.904041+0000", 2024-08-19T14:24:42.835 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:24:42.835 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.835 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.835 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:24:42.835 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:52.904042+0000", 2024-08-19T14:24:42.835 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.2900000000000002e-07 2024-08-19T14:24:42.835 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.836 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.836 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:24:42.836 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:52.904047+0000", 2024-08-19T14:24:42.836 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.233e-06 2024-08-19T14:24:42.836 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.836 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.836 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:24:42.836 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:52.904057+0000", 2024-08-19T14:24:42.836 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.9599999999999995e-06 2024-08-19T14:24:42.836 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.836 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.836 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:24:42.836 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:52.904063+0000", 2024-08-19T14:24:42.836 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6250000000000001e-06 2024-08-19T14:24:42.837 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.837 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.837 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:24:42.837 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:52.904115+0000", 2024-08-19T14:24:42.837 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2243000000000001e-05 2024-08-19T14:24:42.837 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.837 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.837 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:24:42.837 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:52.904205+0000", 2024-08-19T14:24:42.837 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.9593999999999996e-05 2024-08-19T14:24:42.837 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.837 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.837 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 1,3", 2024-08-19T14:24:42.837 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:52.904289+0000", 2024-08-19T14:24:42.837 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.407e-05 2024-08-19T14:24:42.838 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.838 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.838 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:24:42.838 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:52.909116+0000", 2024-08-19T14:24:42.838 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0048268970000000001 2024-08-19T14:24:42.838 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.838 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.838 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:24:42.838 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:52.910757+0000", 2024-08-19T14:24:42.838 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001640938 2024-08-19T14:24:42.838 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.838 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.838 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:24:42.838 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:53.115735+0000", 2024-08-19T14:24:42.839 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20497812400000001 2024-08-19T14:24:42.839 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.839 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.839 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:24:42.839 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:53.115791+0000", 2024-08-19T14:24:42.839 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5603000000000003e-05 2024-08-19T14:24:42.839 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.839 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.839 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:24:42.839 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:53.115823+0000", 2024-08-19T14:24:42.839 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2487000000000003e-05 2024-08-19T14:24:42.839 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:24:42.839 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:24:42.839 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:24:42.840 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.840 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.840 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25333.0:228254 4.e 4:7cbbafef:::benchmark_data_smithi045_71562_object97079:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e400)", 2024-08-19T14:24:42.840 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:23:52.914210+0000", 2024-08-19T14:24:42.840 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 49.779248963999997, 2024-08-19T14:24:42.840 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.21500191399999999, 2024-08-19T14:24:42.840 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:24:42.840 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:24:42.840 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:24:42.840 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:24:42.840 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25333", 2024-08-19T14:24:42.840 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:24:42.840 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 228254 2024-08-19T14:24:42.840 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.841 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:24:42.841 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.841 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:24:42.841 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:52.914210+0000", 2024-08-19T14:24:42.841 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:24:42.841 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.841 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.841 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:24:42.841 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:52.914210+0000", 2024-08-19T14:24:42.841 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:24:42.841 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.841 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.841 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:24:42.841 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:52.914210+0000", 2024-08-19T14:24:42.841 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.33e-07 2024-08-19T14:24:42.842 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.842 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.842 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:24:42.842 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:52.914214+0000", 2024-08-19T14:24:42.842 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6260000000000001e-06 2024-08-19T14:24:42.842 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.842 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.842 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:24:42.842 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:52.914223+0000", 2024-08-19T14:24:42.842 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.2730000000000008e-06 2024-08-19T14:24:42.842 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.842 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.842 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:24:42.842 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:52.914228+0000", 2024-08-19T14:24:42.843 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8670000000000002e-06 2024-08-19T14:24:42.843 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.843 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.843 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:24:42.843 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:52.914277+0000", 2024-08-19T14:24:42.843 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8992000000000001e-05 2024-08-19T14:24:42.843 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.843 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.843 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:24:42.843 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:52.914411+0000", 2024-08-19T14:24:42.843 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000133483 2024-08-19T14:24:42.843 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.843 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.843 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 1,3", 2024-08-19T14:24:42.843 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:52.914537+0000", 2024-08-19T14:24:42.844 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000126162 2024-08-19T14:24:42.844 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.844 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.844 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:24:42.844 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:52.918897+0000", 2024-08-19T14:24:42.844 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.004360442 2024-08-19T14:24:42.844 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.844 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.844 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:24:42.844 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:52.920423+0000", 2024-08-19T14:24:42.844 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0015253090000000001 2024-08-19T14:24:42.844 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.844 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.844 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:24:42.845 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:53.129157+0000", 2024-08-19T14:24:42.845 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20873423399999999 2024-08-19T14:24:42.845 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.845 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.845 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:24:42.845 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:53.129189+0000", 2024-08-19T14:24:42.845 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1786999999999999e-05 2024-08-19T14:24:42.845 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:42.845 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:42.845 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:24:42.845 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:53.129211+0000", 2024-08-19T14:24:42.845 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2806000000000002e-05 2024-08-19T14:24:42.845 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:24:42.845 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:24:42.845 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:24:42.846 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:24:42.846 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:24:42.846 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:24:42.846 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:24:42.847 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:42 smithi045 ceph-mon[20143]: 1.16 scrub starts 2024-08-19T14:24:42.847 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:42 smithi045 ceph-mon[20143]: 1.16 scrub ok 2024-08-19T14:24:42.847 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:42 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/4252726276' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:24:42.872 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:42.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:42.873 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:24:42.974 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:24:42.994 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:42.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:42.995 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:24:43.096 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:24:43.117 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:43.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:43.118 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:24:43.219 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:24:43.242 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:43.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:43.243 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:24:43.344 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:24:43.365 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:43.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:43.367 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:24:43.468 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:24:43.493 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:43.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:43.494 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:24:43.576 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'unique_pool_3' created 2024-08-19T14:24:43.577 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-08-19T14:24:43.594 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:24:43.616 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:43.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:43.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:24:43.718 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:24:43.744 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:43.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:43.746 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:24:43.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:43 smithi045 ceph-mon[20143]: pgmap v962: 31 pgs: 31 active+clean; 577 KiB data, 4.2 GiB used, 800 GiB / 805 GiB avail 2024-08-19T14:24:43.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:43 smithi045 ceph-mon[20143]: 1.3 scrub starts 2024-08-19T14:24:43.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:43 smithi045 ceph-mon[20143]: 1.3 scrub ok 2024-08-19T14:24:43.829 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:24:43.847 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:24:43.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:43 smithi031 ceph-mon[16336]: pgmap v962: 31 pgs: 31 active+clean; 577 KiB data, 4.2 GiB used, 800 GiB / 805 GiB avail 2024-08-19T14:24:43.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:43 smithi031 ceph-mon[16336]: 1.3 scrub starts 2024-08-19T14:24:43.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:43 smithi031 ceph-mon[16336]: 1.3 scrub ok 2024-08-19T14:24:43.872 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:43.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:43.873 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:24:43.951 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:43 smithi032 ceph-mon[20004]: pgmap v962: 31 pgs: 31 active+clean; 577 KiB data, 4.2 GiB used, 800 GiB / 805 GiB avail 2024-08-19T14:24:43.951 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:43 smithi032 ceph-mon[20004]: 1.3 scrub starts 2024-08-19T14:24:43.951 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:43 smithi032 ceph-mon[20004]: 1.3 scrub ok 2024-08-19T14:24:43.973 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:24:43.995 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:43.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:43.997 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:24:44.098 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:24:44.123 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:44.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:44.125 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:24:44.162 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd pool application enable unique_pool_3 rados --yes-i-really-mean-it || true 2024-08-19T14:24:44.176 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:24:44.225 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:24:44.246 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:44.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:44.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:24:44.336 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:24:44.336 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-08-19T14:24:44.336 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-08-19T14:24:44.336 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:24:44.349 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:24:44.370 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:44.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:44.371 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:24:44.472 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:24:44.493 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:44.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:44.494 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:24:44.583 INFO:teuthology.orchestra.run.smithi031.stderr:reweighted osd.3 to 1 (10000) 2024-08-19T14:24:44.595 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:24:44.620 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:44.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:44.621 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:24:44.705 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:44 smithi031 ceph-mon[16336]: 1.19 scrub starts 2024-08-19T14:24:44.705 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:44 smithi031 ceph-mon[16336]: 1.19 scrub ok 2024-08-19T14:24:44.705 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:44 smithi031 ceph-mon[16336]: 1.2 scrub starts 2024-08-19T14:24:44.705 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:44 smithi031 ceph-mon[16336]: 1.2 scrub ok 2024-08-19T14:24:44.705 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:44 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2964695056' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-08-19T14:24:44.705 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:44 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2964695056' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-08-19T14:24:44.706 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:44 smithi031 ceph-mon[16336]: osdmap e419: 12 total, 12 up, 9 in 2024-08-19T14:24:44.706 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:44 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:24:44.706 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:44 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:24:44.706 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:44 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1725537756' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-08-19T14:24:44.722 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:24:44.747 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:44.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:44.748 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:24:44.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:44 smithi045 ceph-mon[20143]: 1.19 scrub starts 2024-08-19T14:24:44.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:44 smithi045 ceph-mon[20143]: 1.19 scrub ok 2024-08-19T14:24:44.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:44 smithi045 ceph-mon[20143]: 1.2 scrub starts 2024-08-19T14:24:44.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:44 smithi045 ceph-mon[20143]: 1.2 scrub ok 2024-08-19T14:24:44.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:44 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2964695056' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-08-19T14:24:44.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:44 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2964695056' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-08-19T14:24:44.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:44 smithi045 ceph-mon[20143]: osdmap e419: 12 total, 12 up, 9 in 2024-08-19T14:24:44.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:44 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:24:44.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:44 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:24:44.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:44 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1725537756' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-08-19T14:24:44.849 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:24:44.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:44.875 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:44.875 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:24:44.888 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:24:44.976 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:24:45.002 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:45.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:45.004 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:24:45.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:44 smithi032 ceph-mon[20004]: 1.19 scrub starts 2024-08-19T14:24:45.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:44 smithi032 ceph-mon[20004]: 1.19 scrub ok 2024-08-19T14:24:45.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:44 smithi032 ceph-mon[20004]: 1.2 scrub starts 2024-08-19T14:24:45.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:44 smithi032 ceph-mon[20004]: 1.2 scrub ok 2024-08-19T14:24:45.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:44 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2964695056' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-08-19T14:24:45.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:44 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2964695056' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-08-19T14:24:45.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:44 smithi032 ceph-mon[20004]: osdmap e419: 12 total, 12 up, 9 in 2024-08-19T14:24:45.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:44 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:24:45.009 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:44 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:24:45.009 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:44 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1725537756' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-08-19T14:24:45.105 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:24:45.118 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-08-19T14:24:45.129 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:45.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:45.130 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:24:45.231 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:24:45.257 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:45.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:45.258 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:24:45.272 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:24:45.283 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd reweight 4 1 2024-08-19T14:24:45.359 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:24:45.370 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:24:45.380 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:45.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:45.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:24:45.482 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:24:45.503 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:45.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:45.504 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:24:45.529 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:45 smithi031 ceph-mon[16336]: pgmap v964: 47 pgs: 1 creating+peering, 15 unknown, 31 active+clean; 577 KiB data, 4.2 GiB used, 800 GiB / 805 GiB avail 2024-08-19T14:24:45.530 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:45 smithi031 ceph-mon[16336]: 1.15 scrub starts 2024-08-19T14:24:45.530 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:45 smithi031 ceph-mon[16336]: 1.15 scrub ok 2024-08-19T14:24:45.530 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:45 smithi031 ceph-mon[16336]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-19T14:24:45.530 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:45 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1725537756' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-08-19T14:24:45.530 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:45 smithi031 ceph-mon[16336]: osdmap e420: 12 total, 12 up, 9 in 2024-08-19T14:24:45.530 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:45 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2963813452' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:24:45.584 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is unset 2024-08-19T14:24:45.604 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:24:45.630 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:45.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:45.631 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:24:45.732 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:24:45.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:45.758 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:45.759 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:24:45.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:45 smithi032 ceph-mon[20004]: pgmap v964: 47 pgs: 1 creating+peering, 15 unknown, 31 active+clean; 577 KiB data, 4.2 GiB used, 800 GiB / 805 GiB avail 2024-08-19T14:24:45.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:45 smithi032 ceph-mon[20004]: 1.15 scrub starts 2024-08-19T14:24:45.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:45 smithi032 ceph-mon[20004]: 1.15 scrub ok 2024-08-19T14:24:45.760 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:45 smithi032 ceph-mon[20004]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-19T14:24:45.760 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:45 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1725537756' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-08-19T14:24:45.760 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:45 smithi032 ceph-mon[20004]: osdmap e420: 12 total, 12 up, 9 in 2024-08-19T14:24:45.760 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:45 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2963813452' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:24:45.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:45 smithi045 ceph-mon[20143]: pgmap v964: 47 pgs: 1 creating+peering, 15 unknown, 31 active+clean; 577 KiB data, 4.2 GiB used, 800 GiB / 805 GiB avail 2024-08-19T14:24:45.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:45 smithi045 ceph-mon[20143]: 1.15 scrub starts 2024-08-19T14:24:45.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:45 smithi045 ceph-mon[20143]: 1.15 scrub ok 2024-08-19T14:24:45.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:45 smithi045 ceph-mon[20143]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-19T14:24:45.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:45 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1725537756' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-08-19T14:24:45.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:45 smithi045 ceph-mon[20143]: osdmap e420: 12 total, 12 up, 9 in 2024-08-19T14:24:45.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:45 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2963813452' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:24:45.859 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:24:45.880 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:45.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:45.881 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:24:45.956 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:24:45.956 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-08-19T14:24:45.956 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-08-19T14:24:45.956 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-08-19T14:24:45.956 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:24:45.976 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:24:45.983 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:24:46.008 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:46.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:46.009 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:24:46.110 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:24:46.135 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:46.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:46.136 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:24:46.237 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:24:46.260 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:46.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:46.261 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:24:46.352 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:24:46.362 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:24:46.388 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:46.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:46.389 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:24:46.490 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:24:46.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:46.516 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:46.516 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:24:46.588 INFO:teuthology.orchestra.run.smithi031.stderr:enabled application 'rados' on pool 'unique_pool_3' 2024-08-19T14:24:46.588 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:46 smithi031 ceph-mon[16336]: 1.1e scrub starts 2024-08-19T14:24:46.588 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:46 smithi031 ceph-mon[16336]: 1.1e scrub ok 2024-08-19T14:24:46.588 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:46 smithi031 ceph-mon[16336]: 1.1a scrub starts 2024-08-19T14:24:46.589 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:46 smithi031 ceph-mon[16336]: 1.1a scrub ok 2024-08-19T14:24:46.589 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:46 smithi031 ceph-mon[16336]: 1.8 scrub starts 2024-08-19T14:24:46.589 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:46 smithi031 ceph-mon[16336]: 1.8 scrub ok 2024-08-19T14:24:46.589 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:46 smithi031 ceph-mon[16336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:24:46.589 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:46 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2963813452' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:24:46.589 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:46 smithi031 ceph-mon[16336]: osdmap e421: 12 total, 12 up, 9 in 2024-08-19T14:24:46.589 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:46 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1932820951' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-08-19T14:24:46.617 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:24:46.641 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:46.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:46.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:24:46.657 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-08-19T14:24:46.744 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:24:46.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:46 smithi032 ceph-mon[20004]: 1.1e scrub starts 2024-08-19T14:24:46.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:46 smithi032 ceph-mon[20004]: 1.1e scrub ok 2024-08-19T14:24:46.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:46 smithi032 ceph-mon[20004]: 1.1a scrub starts 2024-08-19T14:24:46.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:46 smithi032 ceph-mon[20004]: 1.1a scrub ok 2024-08-19T14:24:46.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:46 smithi032 ceph-mon[20004]: 1.8 scrub starts 2024-08-19T14:24:46.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:46 smithi032 ceph-mon[20004]: 1.8 scrub ok 2024-08-19T14:24:46.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:46 smithi032 ceph-mon[20004]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:24:46.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:46 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2963813452' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:24:46.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:46 smithi032 ceph-mon[20004]: osdmap e421: 12 total, 12 up, 9 in 2024-08-19T14:24:46.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:46 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1932820951' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-08-19T14:24:46.771 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:46.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:46.772 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:24:46.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:46 smithi045 ceph-mon[20143]: 1.1e scrub starts 2024-08-19T14:24:46.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:46 smithi045 ceph-mon[20143]: 1.1e scrub ok 2024-08-19T14:24:46.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:46 smithi045 ceph-mon[20143]: 1.1a scrub starts 2024-08-19T14:24:46.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:46 smithi045 ceph-mon[20143]: 1.1a scrub ok 2024-08-19T14:24:46.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:46 smithi045 ceph-mon[20143]: 1.8 scrub starts 2024-08-19T14:24:46.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:46 smithi045 ceph-mon[20143]: 1.8 scrub ok 2024-08-19T14:24:46.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:46 smithi045 ceph-mon[20143]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:24:46.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:46 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2963813452' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:24:46.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:46 smithi045 ceph-mon[20143]: osdmap e421: 12 total, 12 up, 9 in 2024-08-19T14:24:46.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:46 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1932820951' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-08-19T14:24:46.873 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:24:46.896 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:46.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:46.897 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:24:46.955 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:24:46.997 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:24:47.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:47.019 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:47.019 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:24:47.120 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:24:47.143 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:47.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:47.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:24:47.245 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:24:47.266 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:47.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:47.267 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:24:47.368 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:24:47.391 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:47.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:47.392 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:24:47.493 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:24:47.514 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:47.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:47.515 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:24:47.565 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:24:47.565 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-08-19T14:24:47.565 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-08-19T14:24:47.566 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-08-19T14:24:47.566 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.566 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:171107 2.3 2:cdcc5df6:::benchmark_data_smithi045_46415_object47826:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e169)", 2024-08-19T14:24:47.566 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:15:12.315407+0000", 2024-08-19T14:24:47.566 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 575.24779939899997, 2024-08-19T14:24:47.566 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.23111446799999999, 2024-08-19T14:24:47.566 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:24:47.566 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:24:47.566 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:24:47.566 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:24:47.567 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:24:47.567 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:24:47.567 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 171107 2024-08-19T14:24:47.567 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.567 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:24:47.567 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.567 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:24:47.567 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.315407+0000", 2024-08-19T14:24:47.567 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:24:47.567 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.567 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.568 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:24:47.568 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.315407+0000", 2024-08-19T14:24:47.568 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:24:47.568 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.568 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.568 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:24:47.568 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.315408+0000", 2024-08-19T14:24:47.568 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.094e-06 2024-08-19T14:24:47.568 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.568 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.568 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:24:47.569 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.315413+0000", 2024-08-19T14:24:47.569 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2070000000000003e-06 2024-08-19T14:24:47.569 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.569 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.569 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:24:47.569 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.315424+0000", 2024-08-19T14:24:47.569 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0873000000000001e-05 2024-08-19T14:24:47.569 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.569 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.569 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:24:47.570 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.315431+0000", 2024-08-19T14:24:47.570 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1149999999999996e-06 2024-08-19T14:24:47.570 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.570 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.570 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:24:47.570 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.315490+0000", 2024-08-19T14:24:47.570 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8118000000000002e-05 2024-08-19T14:24:47.570 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.570 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.570 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:24:47.570 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.315625+0000", 2024-08-19T14:24:47.570 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000135902 2024-08-19T14:24:47.570 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.571 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.571 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,7", 2024-08-19T14:24:47.571 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.315754+0000", 2024-08-19T14:24:47.571 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012891700000000001 2024-08-19T14:24:47.571 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.571 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.571 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:24:47.571 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.320475+0000", 2024-08-19T14:24:47.571 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0047206069999999999 2024-08-19T14:24:47.571 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.571 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.572 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:24:47.572 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.321696+0000", 2024-08-19T14:24:47.572 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00122135 2024-08-19T14:24:47.572 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.572 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.572 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:24:47.572 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.546437+0000", 2024-08-19T14:24:47.572 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.22474116199999999 2024-08-19T14:24:47.572 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.572 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.572 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:24:47.572 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.546477+0000", 2024-08-19T14:24:47.573 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9263e-05 2024-08-19T14:24:47.573 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.573 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.573 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:24:47.573 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.546522+0000", 2024-08-19T14:24:47.573 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4860000000000001e-05 2024-08-19T14:24:47.573 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:24:47.573 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:24:47.573 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:24:47.573 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.573 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.574 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:171113 2.3 2:cfe749bb:::benchmark_data_smithi045_46415_object47832:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e169)", 2024-08-19T14:24:47.574 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:15:12.322593+0000", 2024-08-19T14:24:47.574 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 575.24061372599999, 2024-08-19T14:24:47.574 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.228733402, 2024-08-19T14:24:47.574 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:24:47.574 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:24:47.574 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:24:47.574 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:24:47.574 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:24:47.574 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:24:47.575 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 171113 2024-08-19T14:24:47.575 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.575 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:24:47.575 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.575 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:24:47.575 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.322593+0000", 2024-08-19T14:24:47.575 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:24:47.575 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.575 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.575 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:24:47.575 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.322593+0000", 2024-08-19T14:24:47.576 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:24:47.576 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.576 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.576 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:24:47.576 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.322594+0000", 2024-08-19T14:24:47.576 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2240000000000001e-06 2024-08-19T14:24:47.576 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.576 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.576 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:24:47.576 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.322603+0000", 2024-08-19T14:24:47.576 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6910000000000003e-06 2024-08-19T14:24:47.577 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.577 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.577 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:24:47.577 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.322615+0000", 2024-08-19T14:24:47.577 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2181000000000001e-05 2024-08-19T14:24:47.577 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.577 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.577 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:24:47.577 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.322620+0000", 2024-08-19T14:24:47.577 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3070000000000002e-06 2024-08-19T14:24:47.577 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.577 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.578 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:24:47.578 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.322677+0000", 2024-08-19T14:24:47.578 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6407999999999999e-05 2024-08-19T14:24:47.578 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.578 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.578 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:24:47.578 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.322808+0000", 2024-08-19T14:24:47.578 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013162499999999999 2024-08-19T14:24:47.578 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.578 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.578 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,7", 2024-08-19T14:24:47.579 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.322936+0000", 2024-08-19T14:24:47.579 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000127358 2024-08-19T14:24:47.579 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.579 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.579 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:24:47.579 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.332145+0000", 2024-08-19T14:24:47.579 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0092095149999999997 2024-08-19T14:24:47.579 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.579 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.579 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:24:47.579 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.333304+0000", 2024-08-19T14:24:47.579 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0011586330000000001 2024-08-19T14:24:47.580 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.580 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.580 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:24:47.580 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.551228+0000", 2024-08-19T14:24:47.580 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.2179239 2024-08-19T14:24:47.580 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.580 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.580 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:24:47.580 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.551282+0000", 2024-08-19T14:24:47.580 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4080999999999999e-05 2024-08-19T14:24:47.580 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.580 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.581 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:24:47.581 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.551326+0000", 2024-08-19T14:24:47.581 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4478999999999999e-05 2024-08-19T14:24:47.581 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:24:47.581 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:24:47.581 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:24:47.581 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.581 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.581 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:171114 2.3 2:cce6d5f2:::benchmark_data_smithi045_46415_object47833:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e169)", 2024-08-19T14:24:47.581 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:15:12.322676+0000", 2024-08-19T14:24:47.581 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 575.24053012499996, 2024-08-19T14:24:47.582 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.228818197, 2024-08-19T14:24:47.582 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:24:47.582 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:24:47.582 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:24:47.582 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:24:47.582 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:24:47.582 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:24:47.582 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 171114 2024-08-19T14:24:47.582 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.582 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:24:47.582 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.583 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:24:47.583 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.322676+0000", 2024-08-19T14:24:47.583 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:24:47.583 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.583 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.583 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:24:47.583 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.322676+0000", 2024-08-19T14:24:47.583 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:24:47.583 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.583 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.583 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:24:47.584 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.322677+0000", 2024-08-19T14:24:47.584 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7999999999999996e-07 2024-08-19T14:24:47.584 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.584 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.584 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:24:47.584 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.322680+0000", 2024-08-19T14:24:47.584 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9519999999999999e-06 2024-08-19T14:24:47.584 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.584 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.584 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:24:47.584 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.322689+0000", 2024-08-19T14:24:47.585 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.6879999999999994e-06 2024-08-19T14:24:47.585 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.585 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.585 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:24:47.585 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.322693+0000", 2024-08-19T14:24:47.585 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9910000000000004e-06 2024-08-19T14:24:47.585 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.585 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.585 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:24:47.585 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.327524+0000", 2024-08-19T14:24:47.585 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0048305259999999999 2024-08-19T14:24:47.585 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.586 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.586 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:24:47.586 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.327692+0000", 2024-08-19T14:24:47.586 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000167956 2024-08-19T14:24:47.586 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.586 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.586 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,7", 2024-08-19T14:24:47.586 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.327829+0000", 2024-08-19T14:24:47.586 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013674999999999999 2024-08-19T14:24:47.586 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.587 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.587 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:24:47.587 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.332259+0000", 2024-08-19T14:24:47.587 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0044300859999999997 2024-08-19T14:24:47.587 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.587 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.587 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:24:47.587 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.333546+0000", 2024-08-19T14:24:47.587 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001287683 2024-08-19T14:24:47.587 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.587 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.587 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:24:47.588 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.551388+0000", 2024-08-19T14:24:47.588 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.21784162100000001 2024-08-19T14:24:47.588 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.588 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.588 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:24:47.588 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.551451+0000", 2024-08-19T14:24:47.588 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2556000000000002e-05 2024-08-19T14:24:47.588 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.588 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.588 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:24:47.588 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:12.551495+0000", 2024-08-19T14:24:47.589 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3908e-05 2024-08-19T14:24:47.589 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:24:47.589 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:24:47.589 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:24:47.589 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.589 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.589 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:177246 2.3 2:c8628734:::benchmark_data_smithi045_46415_object53965:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-08-19T14:24:47.589 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:15:22.002110+0000", 2024-08-19T14:24:47.589 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 565.56109680700001, 2024-08-19T14:24:47.589 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.24453285999999999, 2024-08-19T14:24:47.589 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:24:47.589 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:24:47.590 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:24:47.590 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:24:47.590 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:24:47.590 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:24:47.590 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 177246 2024-08-19T14:24:47.590 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.590 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:24:47.590 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.590 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:24:47.591 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:22.002110+0000", 2024-08-19T14:24:47.591 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:24:47.591 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.591 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.591 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:24:47.591 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:22.002110+0000", 2024-08-19T14:24:47.591 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:24:47.591 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.591 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.591 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:24:47.591 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:22.002110+0000", 2024-08-19T14:24:47.591 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8699999999999995e-07 2024-08-19T14:24:47.592 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.592 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.592 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:24:47.592 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:22.002113+0000", 2024-08-19T14:24:47.592 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1429999999999998e-06 2024-08-19T14:24:47.592 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.592 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.592 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:24:47.592 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:22.002125+0000", 2024-08-19T14:24:47.592 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1250000000000001e-05 2024-08-19T14:24:47.592 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.593 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.593 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:24:47.593 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:22.002130+0000", 2024-08-19T14:24:47.593 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.31e-06 2024-08-19T14:24:47.593 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.593 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.593 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:24:47.593 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:22.002180+0000", 2024-08-19T14:24:47.593 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0312000000000002e-05 2024-08-19T14:24:47.593 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.593 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.593 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:24:47.593 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:22.002298+0000", 2024-08-19T14:24:47.593 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000117545 2024-08-19T14:24:47.594 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.594 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.594 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,7", 2024-08-19T14:24:47.594 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:22.002416+0000", 2024-08-19T14:24:47.594 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000118029 2024-08-19T14:24:47.594 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.594 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.594 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:24:47.594 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:22.005936+0000", 2024-08-19T14:24:47.594 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0035206149999999999 2024-08-19T14:24:47.594 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.594 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.595 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:24:47.595 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:22.008247+0000", 2024-08-19T14:24:47.595 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0023103109999999998 2024-08-19T14:24:47.595 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.595 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.595 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:24:47.595 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:22.246574+0000", 2024-08-19T14:24:47.595 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.23832711200000001 2024-08-19T14:24:47.595 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.595 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.595 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:24:47.595 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:22.246611+0000", 2024-08-19T14:24:47.596 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6717000000000001e-05 2024-08-19T14:24:47.596 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.596 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.596 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:24:47.596 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:22.246643+0000", 2024-08-19T14:24:47.596 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1928999999999999e-05 2024-08-19T14:24:47.596 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:24:47.596 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:24:47.596 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:24:47.596 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.596 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.596 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:177255 2.3 2:ce66084d:::benchmark_data_smithi045_46415_object53974:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-08-19T14:24:47.596 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:15:22.015290+0000", 2024-08-19T14:24:47.597 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 565.54791661900003, 2024-08-19T14:24:47.597 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.23151259900000001, 2024-08-19T14:24:47.597 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:24:47.597 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:24:47.597 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:24:47.597 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:24:47.597 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:24:47.597 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:24:47.597 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 177255 2024-08-19T14:24:47.597 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.597 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:24:47.597 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.597 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:24:47.598 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:22.015290+0000", 2024-08-19T14:24:47.598 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:24:47.598 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.598 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.598 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:24:47.598 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:22.015290+0000", 2024-08-19T14:24:47.598 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:24:47.598 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.598 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.598 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:24:47.598 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:22.015292+0000", 2024-08-19T14:24:47.598 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.86e-06 2024-08-19T14:24:47.599 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.599 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.599 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:24:47.600 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:22.015302+0000", 2024-08-19T14:24:47.600 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0139999999999999e-05 2024-08-19T14:24:47.600 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.601 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.601 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:24:47.601 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:22.015326+0000", 2024-08-19T14:24:47.601 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4436000000000001e-05 2024-08-19T14:24:47.601 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.601 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.601 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:24:47.601 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:22.015339+0000", 2024-08-19T14:24:47.601 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2782999999999999e-05 2024-08-19T14:24:47.601 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.601 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.602 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:24:47.602 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:22.015483+0000", 2024-08-19T14:24:47.602 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000143848 2024-08-19T14:24:47.602 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.602 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.602 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:24:47.602 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:22.015658+0000", 2024-08-19T14:24:47.602 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00017525799999999999 2024-08-19T14:24:47.602 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.602 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.602 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,7", 2024-08-19T14:24:47.602 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:22.015733+0000", 2024-08-19T14:24:47.603 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4522000000000002e-05 2024-08-19T14:24:47.603 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.603 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.603 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:24:47.603 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:22.019282+0000", 2024-08-19T14:24:47.603 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0035489110000000001 2024-08-19T14:24:47.603 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.603 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.603 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:24:47.603 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:22.019455+0000", 2024-08-19T14:24:47.603 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000173115 2024-08-19T14:24:47.603 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.603 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.604 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:24:47.604 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:22.246698+0000", 2024-08-19T14:24:47.604 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.22724322599999999 2024-08-19T14:24:47.604 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.604 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.604 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:24:47.604 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:22.246757+0000", 2024-08-19T14:24:47.604 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8909000000000001e-05 2024-08-19T14:24:47.604 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.604 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.604 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:24:47.604 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:22.246802+0000", 2024-08-19T14:24:47.604 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5590999999999998e-05 2024-08-19T14:24:47.605 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:24:47.605 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:24:47.605 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:24:47.605 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.605 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.605 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:177265 2.3 2:cd07bc78:::benchmark_data_smithi045_46415_object53984:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-08-19T14:24:47.605 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:15:22.027303+0000", 2024-08-19T14:24:47.605 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 565.53590398400002, 2024-08-19T14:24:47.605 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.21960194099999999, 2024-08-19T14:24:47.605 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:24:47.605 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:24:47.606 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:24:47.606 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:24:47.606 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:24:47.606 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:24:47.606 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 177265 2024-08-19T14:24:47.606 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.606 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:24:47.606 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.606 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:24:47.606 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:22.027303+0000", 2024-08-19T14:24:47.606 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:24:47.607 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.607 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.607 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:24:47.607 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:22.027303+0000", 2024-08-19T14:24:47.607 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:24:47.607 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.607 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.607 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:24:47.607 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:22.027303+0000", 2024-08-19T14:24:47.607 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.2800000000000005e-07 2024-08-19T14:24:47.607 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.607 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.607 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:24:47.608 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:22.027308+0000", 2024-08-19T14:24:47.608 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.826e-06 2024-08-19T14:24:47.608 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.608 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.608 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:24:47.608 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:22.027319+0000", 2024-08-19T14:24:47.608 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.031e-05 2024-08-19T14:24:47.608 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.608 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.608 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:24:47.608 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:22.027324+0000", 2024-08-19T14:24:47.608 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0250000000000002e-06 2024-08-19T14:24:47.608 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.609 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.609 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:24:47.609 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:22.027374+0000", 2024-08-19T14:24:47.609 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0111000000000002e-05 2024-08-19T14:24:47.609 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.609 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.609 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:24:47.609 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:22.027484+0000", 2024-08-19T14:24:47.609 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000110605 2024-08-19T14:24:47.609 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.609 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.609 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,7", 2024-08-19T14:24:47.610 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:22.027601+0000", 2024-08-19T14:24:47.610 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000116884 2024-08-19T14:24:47.610 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.610 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.610 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:24:47.610 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:22.035236+0000", 2024-08-19T14:24:47.610 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.007634353 2024-08-19T14:24:47.610 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.610 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.610 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:24:47.610 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:22.035284+0000", 2024-08-19T14:24:47.611 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8084999999999998e-05 2024-08-19T14:24:47.611 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.611 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.611 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:24:47.611 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:22.246847+0000", 2024-08-19T14:24:47.611 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.21156344399999999 2024-08-19T14:24:47.611 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.611 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.611 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:24:47.611 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:22.246878+0000", 2024-08-19T14:24:47.611 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0636000000000003e-05 2024-08-19T14:24:47.611 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.612 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.612 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:24:47.612 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:15:22.246904+0000", 2024-08-19T14:24:47.612 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6733999999999998e-05 2024-08-19T14:24:47.612 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:24:47.612 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:24:47.612 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:24:47.612 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.612 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.612 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:223849 2.3 2:ce808b41:::benchmark_data_smithi045_46415_object100568:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e201)", 2024-08-19T14:24:47.612 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:16:18.946749+0000", 2024-08-19T14:24:47.612 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 508.61645712199999, 2024-08-19T14:24:47.613 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.23607239799999999, 2024-08-19T14:24:47.613 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:24:47.613 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:24:47.613 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:24:47.613 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:24:47.613 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:24:47.613 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:24:47.613 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 223849 2024-08-19T14:24:47.613 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.613 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:24:47.613 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.613 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:24:47.614 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:16:18.946749+0000", 2024-08-19T14:24:47.614 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:24:47.614 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.614 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.614 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:24:47.614 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:16:18.946749+0000", 2024-08-19T14:24:47.614 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:24:47.614 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.614 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.614 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:24:47.615 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:16:18.946750+0000", 2024-08-19T14:24:47.615 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.2600000000000001e-07 2024-08-19T14:24:47.615 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.615 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.615 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:24:47.615 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:16:18.946755+0000", 2024-08-19T14:24:47.615 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6210000000000003e-06 2024-08-19T14:24:47.615 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.615 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.615 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:24:47.615 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:16:18.946766+0000", 2024-08-19T14:24:47.615 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1173e-05 2024-08-19T14:24:47.615 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.616 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.616 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:24:47.616 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:16:18.946772+0000", 2024-08-19T14:24:47.616 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8900000000000004e-06 2024-08-19T14:24:47.616 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.616 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.616 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:24:47.616 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:16:18.946812+0000", 2024-08-19T14:24:47.616 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9619000000000001e-05 2024-08-19T14:24:47.616 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.616 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.616 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:24:47.616 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:16:18.946899+0000", 2024-08-19T14:24:47.617 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.7496000000000004e-05 2024-08-19T14:24:47.617 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.617 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.617 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,7", 2024-08-19T14:24:47.617 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:16:18.946970+0000", 2024-08-19T14:24:47.617 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1371999999999994e-05 2024-08-19T14:24:47.617 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.617 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.617 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:24:47.617 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:16:18.951666+0000", 2024-08-19T14:24:47.617 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.004695734 2024-08-19T14:24:47.618 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.618 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.618 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:24:47.618 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:16:18.952768+0000", 2024-08-19T14:24:47.618 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001101802 2024-08-19T14:24:47.618 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.618 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.618 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:24:47.618 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:16:19.182722+0000", 2024-08-19T14:24:47.618 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.22995431499999999 2024-08-19T14:24:47.618 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.619 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.619 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:24:47.619 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:16:19.182776+0000", 2024-08-19T14:24:47.619 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3752999999999998e-05 2024-08-19T14:24:47.619 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.619 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.619 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:24:47.619 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:16:19.182822+0000", 2024-08-19T14:24:47.619 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5697e-05 2024-08-19T14:24:47.619 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:24:47.620 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:24:47.620 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:24:47.620 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.620 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.620 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24580.0:223864 2.3 2:c10b07c7:::benchmark_data_smithi045_46415_object100583:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e201)", 2024-08-19T14:24:47.620 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:16:18.962904+0000", 2024-08-19T14:24:47.620 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 508.60030289600002, 2024-08-19T14:24:47.620 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.22006620700000001, 2024-08-19T14:24:47.620 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:24:47.620 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:24:47.620 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:24:47.620 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:24:47.621 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24580", 2024-08-19T14:24:47.621 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:24:47.621 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 223864 2024-08-19T14:24:47.621 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.621 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:24:47.621 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.621 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:24:47.621 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:16:18.962904+0000", 2024-08-19T14:24:47.621 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:24:47.621 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.621 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.622 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:24:47.622 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:16:18.962904+0000", 2024-08-19T14:24:47.622 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:24:47.622 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.622 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.622 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:24:47.622 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:16:18.962905+0000", 2024-08-19T14:24:47.622 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1170000000000001e-06 2024-08-19T14:24:47.622 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.622 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.623 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:24:47.623 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:16:18.962910+0000", 2024-08-19T14:24:47.623 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1880000000000001e-06 2024-08-19T14:24:47.623 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.623 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.623 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:24:47.623 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:16:18.962923+0000", 2024-08-19T14:24:47.623 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3380999999999999e-05 2024-08-19T14:24:47.623 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.623 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.623 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:24:47.624 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:16:18.962931+0000", 2024-08-19T14:24:47.624 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3579999999999997e-06 2024-08-19T14:24:47.624 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.624 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.624 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:24:47.624 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:16:18.962997+0000", 2024-08-19T14:24:47.624 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6303000000000005e-05 2024-08-19T14:24:47.624 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.624 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.624 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:24:47.624 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:16:18.963123+0000", 2024-08-19T14:24:47.624 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012634500000000001 2024-08-19T14:24:47.625 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.625 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.625 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,7", 2024-08-19T14:24:47.625 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:16:18.963248+0000", 2024-08-19T14:24:47.625 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000125165 2024-08-19T14:24:47.625 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.625 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.626 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:24:47.626 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:16:18.966968+0000", 2024-08-19T14:24:47.626 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0037195790000000002 2024-08-19T14:24:47.626 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.626 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.626 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:24:47.626 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:16:18.968245+0000", 2024-08-19T14:24:47.626 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0012766489999999999 2024-08-19T14:24:47.626 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.626 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.626 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:24:47.627 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:16:19.182869+0000", 2024-08-19T14:24:47.627 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.21462381699999999 2024-08-19T14:24:47.627 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.627 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.627 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:24:47.627 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:16:19.182926+0000", 2024-08-19T14:24:47.627 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7349999999999998e-05 2024-08-19T14:24:47.627 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.627 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.627 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:24:47.627 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:16:19.182970+0000", 2024-08-19T14:24:47.627 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3955000000000002e-05 2024-08-19T14:24:47.628 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:24:47.628 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:24:47.628 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:24:47.628 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.628 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.628 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25057.0:64611 3.1c 3.ea59d81c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e246)", 2024-08-19T14:24:47.628 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:17:35.978148+0000", 2024-08-19T14:24:47.628 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 431.58505819200002, 2024-08-19T14:24:47.628 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.98523712200000002, 2024-08-19T14:24:47.628 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:24:47.628 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:24:47.628 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:24:47.628 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:24:47.629 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25057", 2024-08-19T14:24:47.629 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:24:47.629 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 64611 2024-08-19T14:24:47.629 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.629 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:24:47.629 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.629 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:24:47.629 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.978148+0000", 2024-08-19T14:24:47.629 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:24:47.629 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.629 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.630 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:24:47.630 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.978148+0000", 2024-08-19T14:24:47.630 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:24:47.630 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.630 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.630 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:24:47.630 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.978150+0000", 2024-08-19T14:24:47.630 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.903e-06 2024-08-19T14:24:47.631 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.631 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.631 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:24:47.631 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.978235+0000", 2024-08-19T14:24:47.631 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.4945999999999996e-05 2024-08-19T14:24:47.631 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.631 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.631 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:24:47.631 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.978255+0000", 2024-08-19T14:24:47.631 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9361e-05 2024-08-19T14:24:47.631 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.631 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.632 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:24:47.632 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.978263+0000", 2024-08-19T14:24:47.632 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6770000000000001e-06 2024-08-19T14:24:47.632 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.632 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.632 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:24:47.632 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.978330+0000", 2024-08-19T14:24:47.632 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7089999999999996e-05 2024-08-19T14:24:47.632 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.632 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.632 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:24:47.633 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.978341+0000", 2024-08-19T14:24:47.633 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0559e-05 2024-08-19T14:24:47.633 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.633 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.633 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:24:47.633 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:36.963362+0000", 2024-08-19T14:24:47.633 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.98502080999999997 2024-08-19T14:24:47.633 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.633 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.633 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:24:47.633 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:36.963385+0000", 2024-08-19T14:24:47.633 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3776000000000001e-05 2024-08-19T14:24:47.634 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:24:47.634 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:24:47.634 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:24:47.634 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.634 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.634 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25057.0:64634 3.1c 3.60566bc (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e246)", 2024-08-19T14:24:47.634 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:17:36.183428+0000", 2024-08-19T14:24:47.634 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 431.379778644, 2024-08-19T14:24:47.634 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.78006474000000003, 2024-08-19T14:24:47.634 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:24:47.634 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:24:47.635 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:24:47.635 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:24:47.635 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25057", 2024-08-19T14:24:47.635 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:24:47.635 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 64634 2024-08-19T14:24:47.635 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.635 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:24:47.635 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.635 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:24:47.635 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:36.183428+0000", 2024-08-19T14:24:47.635 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:24:47.635 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.636 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.636 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:24:47.636 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:36.183428+0000", 2024-08-19T14:24:47.636 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:24:47.636 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.636 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.636 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:24:47.636 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:36.183430+0000", 2024-08-19T14:24:47.636 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8160000000000001e-06 2024-08-19T14:24:47.636 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.636 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.636 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:24:47.637 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:36.183463+0000", 2024-08-19T14:24:47.637 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3254999999999999e-05 2024-08-19T14:24:47.637 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.637 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.637 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:24:47.637 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:36.183482+0000", 2024-08-19T14:24:47.637 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8595000000000001e-05 2024-08-19T14:24:47.637 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.637 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.637 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:24:47.637 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:36.183491+0000", 2024-08-19T14:24:47.638 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.8840000000000006e-06 2024-08-19T14:24:47.638 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.638 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.638 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:24:47.638 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:36.183557+0000", 2024-08-19T14:24:47.638 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.5791999999999996e-05 2024-08-19T14:24:47.638 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.638 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.639 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:24:47.639 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:36.183566+0000", 2024-08-19T14:24:47.639 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.9919999999999996e-06 2024-08-19T14:24:47.639 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.639 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.639 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:24:47.639 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:36.963484+0000", 2024-08-19T14:24:47.639 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.77991771499999996 2024-08-19T14:24:47.639 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.639 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.639 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:24:47.639 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:36.963493+0000", 2024-08-19T14:24:47.640 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6910000000000003e-06 2024-08-19T14:24:47.640 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:24:47.640 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:24:47.640 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:24:47.640 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.640 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.640 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25057.0:68717 3.1d 3:b8ee1297:::benchmark_data_smithi045_59973_object68716:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e251)", 2024-08-19T14:24:47.640 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:17:43.596001+0000", 2024-08-19T14:24:47.640 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 423.96720552300002, 2024-08-19T14:24:47.640 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.22569884800000001, 2024-08-19T14:24:47.640 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:24:47.640 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:24:47.641 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:24:47.641 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:24:47.641 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25057", 2024-08-19T14:24:47.641 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:24:47.641 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 68717 2024-08-19T14:24:47.641 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.641 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:24:47.641 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.641 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:24:47.641 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:43.596001+0000", 2024-08-19T14:24:47.641 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:24:47.642 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.642 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.642 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:24:47.642 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:43.596001+0000", 2024-08-19T14:24:47.642 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:24:47.642 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.642 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.642 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:24:47.642 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:43.596003+0000", 2024-08-19T14:24:47.642 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0540000000000002e-06 2024-08-19T14:24:47.642 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.643 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.643 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:24:47.643 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:43.596035+0000", 2024-08-19T14:24:47.643 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2276000000000001e-05 2024-08-19T14:24:47.643 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.643 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.643 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:24:47.643 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:43.596919+0000", 2024-08-19T14:24:47.643 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00088367600000000001 2024-08-19T14:24:47.643 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.643 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.643 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:24:47.643 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:43.596940+0000", 2024-08-19T14:24:47.644 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1239000000000001e-05 2024-08-19T14:24:47.644 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.644 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.644 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:24:47.644 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:43.598926+0000", 2024-08-19T14:24:47.644 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0019856399999999999 2024-08-19T14:24:47.644 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.644 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.644 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:24:47.644 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:43.599089+0000", 2024-08-19T14:24:47.644 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00016281999999999999 2024-08-19T14:24:47.645 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.645 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.645 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4,8", 2024-08-19T14:24:47.645 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:43.599257+0000", 2024-08-19T14:24:47.645 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000168114 2024-08-19T14:24:47.645 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.645 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.645 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:24:47.645 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:43.604920+0000", 2024-08-19T14:24:47.645 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0056633059999999999 2024-08-19T14:24:47.645 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.645 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.645 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:24:47.646 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:43.607476+0000", 2024-08-19T14:24:47.646 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0025555230000000001 2024-08-19T14:24:47.646 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.646 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.646 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:24:47.646 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:43.820484+0000", 2024-08-19T14:24:47.646 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.21300864999999999 2024-08-19T14:24:47.646 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.646 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.646 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:24:47.646 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:43.820556+0000", 2024-08-19T14:24:47.647 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1452999999999995e-05 2024-08-19T14:24:47.647 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.647 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.647 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:24:47.647 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:43.821700+0000", 2024-08-19T14:24:47.647 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001144097 2024-08-19T14:24:47.647 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:24:47.647 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:24:47.647 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:24:47.647 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.647 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.647 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.25333.0:199209 4.a e393/335 4:5129952e:::benchmark_data_smithi045_71562_object68034:head v 393'12658, mlcod=393'12656)", 2024-08-19T14:24:47.648 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:23:20.429575+0000", 2024-08-19T14:24:47.648 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 87.133631046999994, 2024-08-19T14:24:47.648 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.035312189000000001, 2024-08-19T14:24:47.648 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:24:47.648 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:24:47.648 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:24:47.648 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:24:47.648 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.648 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:24:47.648 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.429575+0000", 2024-08-19T14:24:47.648 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:24:47.648 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.648 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.649 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:24:47.649 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.429575+0000", 2024-08-19T14:24:47.649 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:24:47.649 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.649 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.649 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:24:47.649 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.429577+0000", 2024-08-19T14:24:47.649 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.176e-06 2024-08-19T14:24:47.649 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.649 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.649 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:24:47.649 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.429585+0000", 2024-08-19T14:24:47.650 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.7940000000000008e-06 2024-08-19T14:24:47.650 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.650 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.650 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:24:47.650 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.429606+0000", 2024-08-19T14:24:47.650 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0582e-05 2024-08-19T14:24:47.650 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.650 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.650 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:24:47.650 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.429615+0000", 2024-08-19T14:24:47.650 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.2289999999999992e-06 2024-08-19T14:24:47.651 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.651 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.651 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:24:47.651 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.429820+0000", 2024-08-19T14:24:47.651 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000204685 2024-08-19T14:24:47.651 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.651 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.651 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:24:47.651 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.429880+0000", 2024-08-19T14:24:47.651 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0291000000000003e-05 2024-08-19T14:24:47.651 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.651 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.652 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:24:47.652 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.464837+0000", 2024-08-19T14:24:47.652 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.034957249000000003 2024-08-19T14:24:47.652 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.652 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.652 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:24:47.652 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.464888+0000", 2024-08-19T14:24:47.652 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0182999999999997e-05 2024-08-19T14:24:47.652 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:24:47.652 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:24:47.652 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:24:47.652 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.652 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.653 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.25333.0:199215 4.2 e393/335 4:48847619:::benchmark_data_smithi045_71562_object68040:head v 393'12514, mlcod=393'12512)", 2024-08-19T14:24:47.653 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:23:20.435110+0000", 2024-08-19T14:24:47.653 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 87.128096348, 2024-08-19T14:24:47.653 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.034628711999999999, 2024-08-19T14:24:47.653 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:24:47.653 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:24:47.653 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:24:47.653 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:24:47.653 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.653 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:24:47.654 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.435110+0000", 2024-08-19T14:24:47.654 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:24:47.654 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.654 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.654 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:24:47.654 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.435110+0000", 2024-08-19T14:24:47.654 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:24:47.654 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.654 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.654 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:24:47.654 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.435112+0000", 2024-08-19T14:24:47.655 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7099999999999999e-06 2024-08-19T14:24:47.655 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.655 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.655 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:24:47.655 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.435121+0000", 2024-08-19T14:24:47.655 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.1500000000000005e-06 2024-08-19T14:24:47.655 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.655 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.655 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:24:47.655 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.435141+0000", 2024-08-19T14:24:47.655 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0305e-05 2024-08-19T14:24:47.655 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.656 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.656 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:24:47.656 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.435151+0000", 2024-08-19T14:24:47.656 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.4779999999999997e-06 2024-08-19T14:24:47.656 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.656 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.656 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:24:47.656 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.435321+0000", 2024-08-19T14:24:47.656 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00016970900000000001 2024-08-19T14:24:47.656 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.656 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.656 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:24:47.657 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.435380+0000", 2024-08-19T14:24:47.657 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9379000000000002e-05 2024-08-19T14:24:47.657 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.657 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.657 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:24:47.657 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.469652+0000", 2024-08-19T14:24:47.657 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.034272576999999999 2024-08-19T14:24:47.657 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.657 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.657 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:24:47.657 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.469739+0000", 2024-08-19T14:24:47.657 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6403999999999995e-05 2024-08-19T14:24:47.658 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:24:47.658 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:24:47.658 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:24:47.658 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.658 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.658 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.25333.0:199217 4.2 e393/335 4:4318f9c5:::benchmark_data_smithi045_71562_object68042:head v 393'12515, mlcod=393'12512)", 2024-08-19T14:24:47.658 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:23:20.447787+0000", 2024-08-19T14:24:47.658 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 87.115419392999996, 2024-08-19T14:24:47.658 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.027496152999999999, 2024-08-19T14:24:47.658 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:24:47.659 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:24:47.659 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:24:47.659 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:24:47.659 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.659 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:24:47.659 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.447787+0000", 2024-08-19T14:24:47.659 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:24:47.659 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.659 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.659 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:24:47.659 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.447787+0000", 2024-08-19T14:24:47.660 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:24:47.660 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.660 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.660 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:24:47.660 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.447789+0000", 2024-08-19T14:24:47.660 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6589999999999999e-06 2024-08-19T14:24:47.660 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.660 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.660 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:24:47.660 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.447800+0000", 2024-08-19T14:24:47.660 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1474000000000001e-05 2024-08-19T14:24:47.660 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.661 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.661 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:24:47.661 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.447825+0000", 2024-08-19T14:24:47.661 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4542999999999999e-05 2024-08-19T14:24:47.661 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.661 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.661 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:24:47.661 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.447836+0000", 2024-08-19T14:24:47.661 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1056999999999999e-05 2024-08-19T14:24:47.661 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.661 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.662 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:24:47.662 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.448421+0000", 2024-08-19T14:24:47.662 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00058501699999999996 2024-08-19T14:24:47.662 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.662 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.662 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:24:47.662 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.448543+0000", 2024-08-19T14:24:47.662 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012214600000000001 2024-08-19T14:24:47.662 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.662 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.662 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:24:47.662 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.475107+0000", 2024-08-19T14:24:47.663 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.026563902 2024-08-19T14:24:47.663 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.663 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.663 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:24:47.663 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.475283+0000", 2024-08-19T14:24:47.663 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000176355 2024-08-19T14:24:47.663 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:24:47.663 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:24:47.663 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:24:47.663 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.663 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.663 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.25333.0:199212 4.a e393/335 4:59dd3d59:::benchmark_data_smithi045_71562_object68037:head v 393'12659, mlcod=393'12656)", 2024-08-19T14:24:47.663 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:23:20.447875+0000", 2024-08-19T14:24:47.664 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 87.115331522999995, 2024-08-19T14:24:47.664 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.027226127999999999, 2024-08-19T14:24:47.664 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:24:47.664 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:24:47.664 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:24:47.664 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:24:47.664 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.664 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:24:47.664 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.447875+0000", 2024-08-19T14:24:47.664 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:24:47.664 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.664 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.664 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:24:47.665 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.447875+0000", 2024-08-19T14:24:47.665 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:24:47.665 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.665 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.665 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:24:47.665 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.447876+0000", 2024-08-19T14:24:47.665 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2499999999999995e-07 2024-08-19T14:24:47.665 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.665 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.665 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:24:47.665 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.447881+0000", 2024-08-19T14:24:47.665 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6300000000000003e-06 2024-08-19T14:24:47.665 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.666 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.666 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:24:47.666 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.447893+0000", 2024-08-19T14:24:47.666 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1925e-05 2024-08-19T14:24:47.666 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.666 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.666 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:24:47.666 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.447904+0000", 2024-08-19T14:24:47.666 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.079e-05 2024-08-19T14:24:47.666 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.666 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.666 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:24:47.667 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.448487+0000", 2024-08-19T14:24:47.667 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00058326799999999996 2024-08-19T14:24:47.667 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.667 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.667 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:24:47.667 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.448604+0000", 2024-08-19T14:24:47.667 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000117249 2024-08-19T14:24:47.667 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.667 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.667 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:24:47.667 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.474937+0000", 2024-08-19T14:24:47.668 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.026332923000000001 2024-08-19T14:24:47.668 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.668 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.668 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:24:47.668 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.475101+0000", 2024-08-19T14:24:47.668 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00016381800000000001 2024-08-19T14:24:47.668 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:24:47.668 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:24:47.668 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:24:47.668 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.668 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.668 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.25333.0:199216 4.a e393/335 4:513c9c84:::benchmark_data_smithi045_71562_object68041:head v 393'12660, mlcod=393'12656)", 2024-08-19T14:24:47.668 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:23:20.448008+0000", 2024-08-19T14:24:47.669 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 87.115198938999995, 2024-08-19T14:24:47.669 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.027462936, 2024-08-19T14:24:47.669 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:24:47.669 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:24:47.669 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:24:47.669 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:24:47.669 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.669 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:24:47.669 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.448008+0000", 2024-08-19T14:24:47.669 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:24:47.669 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.669 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.669 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:24:47.670 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.448008+0000", 2024-08-19T14:24:47.670 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:24:47.670 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.670 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.670 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:24:47.670 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.448008+0000", 2024-08-19T14:24:47.670 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2e-07 2024-08-19T14:24:47.670 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.670 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.670 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:24:47.670 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.448014+0000", 2024-08-19T14:24:47.671 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9950000000000002e-06 2024-08-19T14:24:47.671 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.671 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.671 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:24:47.671 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.448047+0000", 2024-08-19T14:24:47.671 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2805999999999997e-05 2024-08-19T14:24:47.671 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.671 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.671 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:24:47.671 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.448054+0000", 2024-08-19T14:24:47.671 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0310000000000001e-06 2024-08-19T14:24:47.671 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.672 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.672 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:24:47.672 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.457249+0000", 2024-08-19T14:24:47.672 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0091954410000000004 2024-08-19T14:24:47.672 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.672 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.672 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:24:47.672 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.457273+0000", 2024-08-19T14:24:47.672 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3149000000000001e-05 2024-08-19T14:24:47.672 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.672 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.673 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:24:47.673 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.475286+0000", 2024-08-19T14:24:47.673 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.018013936000000001 2024-08-19T14:24:47.673 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.673 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.673 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:24:47.673 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.475470+0000", 2024-08-19T14:24:47.673 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00018405799999999999 2024-08-19T14:24:47.673 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:24:47.673 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:24:47.673 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:24:47.674 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.674 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.674 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.25333.0:199224 4.2 e393/335 4:40860986:::benchmark_data_smithi045_71562_object68049:head v 393'12516, mlcod=393'12512)", 2024-08-19T14:24:47.674 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:23:20.449339+0000", 2024-08-19T14:24:47.674 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 87.113867756999994, 2024-08-19T14:24:47.674 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.026285112999999999, 2024-08-19T14:24:47.674 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:24:47.674 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:24:47.674 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:24:47.674 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:24:47.674 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.674 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:24:47.675 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.449339+0000", 2024-08-19T14:24:47.675 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:24:47.675 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.675 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.675 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:24:47.675 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.449339+0000", 2024-08-19T14:24:47.675 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:24:47.675 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.675 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.675 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:24:47.675 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.449339+0000", 2024-08-19T14:24:47.676 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1500000000000004e-07 2024-08-19T14:24:47.676 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.676 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.676 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:24:47.676 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.449346+0000", 2024-08-19T14:24:47.676 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2979999999999997e-06 2024-08-19T14:24:47.676 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.676 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.676 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:24:47.676 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.449379+0000", 2024-08-19T14:24:47.676 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2984000000000001e-05 2024-08-19T14:24:47.676 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.676 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.677 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:24:47.677 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.449386+0000", 2024-08-19T14:24:47.677 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2099999999999996e-06 2024-08-19T14:24:47.677 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.677 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.677 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:24:47.677 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.460203+0000", 2024-08-19T14:24:47.677 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010816684 2024-08-19T14:24:47.677 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.677 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.677 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:24:47.678 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.460249+0000", 2024-08-19T14:24:47.678 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5979000000000001e-05 2024-08-19T14:24:47.678 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.678 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.678 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:24:47.678 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.475474+0000", 2024-08-19T14:24:47.678 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.015224995999999999 2024-08-19T14:24:47.678 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.678 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.678 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:24:47.678 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:20.475624+0000", 2024-08-19T14:24:47.678 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000150247 2024-08-19T14:24:47.678 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:24:47.679 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:24:47.679 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:24:47.679 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.679 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.679 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.25333.0:226948 4.2 e400/335 4:460a4f05:::benchmark_data_smithi045_71562_object95773:head v 400'14271, mlcod=400'14269)", 2024-08-19T14:24:47.679 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:23:51.461086+0000", 2024-08-19T14:24:47.679 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 56.102120870999997, 2024-08-19T14:24:47.679 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.026992828999999999, 2024-08-19T14:24:47.679 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:24:47.679 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:24:47.679 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:24:47.679 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:24:47.679 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.680 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:24:47.680 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:51.461086+0000", 2024-08-19T14:24:47.680 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:24:47.680 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.680 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.680 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:24:47.680 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:51.461086+0000", 2024-08-19T14:24:47.680 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:24:47.680 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.680 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.680 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:24:47.680 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:51.461086+0000", 2024-08-19T14:24:47.681 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.44e-07 2024-08-19T14:24:47.681 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.681 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.681 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:24:47.681 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:51.461091+0000", 2024-08-19T14:24:47.681 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0969999999999996e-06 2024-08-19T14:24:47.681 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.681 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.681 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:24:47.681 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:51.461103+0000", 2024-08-19T14:24:47.681 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1284000000000001e-05 2024-08-19T14:24:47.681 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.682 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.682 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:24:47.682 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:51.461109+0000", 2024-08-19T14:24:47.682 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1029999999999998e-06 2024-08-19T14:24:47.682 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.682 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.682 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:24:47.682 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:51.464048+0000", 2024-08-19T14:24:47.682 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0029392350000000001 2024-08-19T14:24:47.682 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.682 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.683 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:24:47.683 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:51.464074+0000", 2024-08-19T14:24:47.683 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.62e-05 2024-08-19T14:24:47.683 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.683 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.683 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:24:47.683 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:51.488020+0000", 2024-08-19T14:24:47.683 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.023945468000000001 2024-08-19T14:24:47.683 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.683 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.683 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:24:47.683 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:51.488078+0000", 2024-08-19T14:24:47.683 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8798000000000002e-05 2024-08-19T14:24:47.684 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:24:47.684 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:24:47.684 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:24:47.684 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.684 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.684 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.25333.0:237032 4.2 e402/335 4:42d01a60:::benchmark_data_smithi045_71562_object105857:head v 402'14949, mlcod=402'14944)", 2024-08-19T14:24:47.684 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:24:02.936398+0000", 2024-08-19T14:24:47.684 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 44.626808406000002, 2024-08-19T14:24:47.684 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.026140351999999999, 2024-08-19T14:24:47.684 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:24:47.684 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:24:47.684 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:24:47.684 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:24:47.685 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.685 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:24:47.685 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:02.936398+0000", 2024-08-19T14:24:47.685 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:24:47.685 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.685 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.685 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:24:47.685 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:02.936398+0000", 2024-08-19T14:24:47.685 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:24:47.685 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.685 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.685 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:24:47.686 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:02.936399+0000", 2024-08-19T14:24:47.686 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4399999999999999e-07 2024-08-19T14:24:47.686 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.686 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.686 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:24:47.686 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:02.936404+0000", 2024-08-19T14:24:47.686 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4090000000000002e-06 2024-08-19T14:24:47.686 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.686 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.686 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:24:47.686 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:02.936416+0000", 2024-08-19T14:24:47.687 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1819e-05 2024-08-19T14:24:47.687 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.687 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.687 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:24:47.687 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:02.936422+0000", 2024-08-19T14:24:47.687 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1190000000000002e-06 2024-08-19T14:24:47.687 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.687 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.687 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:24:47.687 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:02.940908+0000", 2024-08-19T14:24:47.687 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0044859799999999997 2024-08-19T14:24:47.687 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.688 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.688 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:24:47.688 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:02.940928+0000", 2024-08-19T14:24:47.688 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9820999999999999e-05 2024-08-19T14:24:47.688 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.688 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.688 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:24:47.688 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:02.962442+0000", 2024-08-19T14:24:47.688 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.021514432999999999 2024-08-19T14:24:47.688 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.688 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.688 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:24:47.689 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:02.962538+0000", 2024-08-19T14:24:47.689 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.6026999999999994e-05 2024-08-19T14:24:47.689 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:24:47.689 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:24:47.689 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:24:47.689 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.689 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.689 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.25333.0:237040 4.2 e402/335 4:436f9d22:::benchmark_data_smithi045_71562_object105865:head v 402'14950, mlcod=402'14945)", 2024-08-19T14:24:47.689 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:24:02.943241+0000", 2024-08-19T14:24:47.689 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 44.619965194000002, 2024-08-19T14:24:47.689 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.027968839999999998, 2024-08-19T14:24:47.689 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:24:47.690 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:24:47.690 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:24:47.690 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:24:47.690 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.690 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:24:47.690 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:02.943241+0000", 2024-08-19T14:24:47.690 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:24:47.690 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.690 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.690 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:24:47.690 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:02.943241+0000", 2024-08-19T14:24:47.691 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:24:47.691 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.691 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.691 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:24:47.691 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:02.943242+0000", 2024-08-19T14:24:47.691 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1699999999999997e-07 2024-08-19T14:24:47.691 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.691 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.691 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:24:47.691 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:02.943248+0000", 2024-08-19T14:24:47.691 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8680000000000004e-06 2024-08-19T14:24:47.691 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.692 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.692 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:24:47.692 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:02.943260+0000", 2024-08-19T14:24:47.692 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1815e-05 2024-08-19T14:24:47.692 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.692 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.692 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:24:47.692 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:02.943266+0000", 2024-08-19T14:24:47.692 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0850000000000002e-06 2024-08-19T14:24:47.692 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.692 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.692 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:24:47.693 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:02.948388+0000", 2024-08-19T14:24:47.693 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0051223989999999997 2024-08-19T14:24:47.693 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.693 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.693 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:24:47.693 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:02.948407+0000", 2024-08-19T14:24:47.693 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8958999999999999e-05 2024-08-19T14:24:47.693 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.693 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.693 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:24:47.693 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:02.971164+0000", 2024-08-19T14:24:47.694 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.022757329999999999 2024-08-19T14:24:47.694 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:47.694 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:47.694 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:24:47.694 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:02.971210+0000", 2024-08-19T14:24:47.694 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5667000000000002e-05 2024-08-19T14:24:47.694 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:24:47.694 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:24:47.694 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:24:47.694 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:24:47.694 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:24:47.694 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:24:47.695 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:24:47.696 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:24:47.696 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:24:47.696 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:24:47.697 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:24:47.697 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:24:47.697 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:24:47.698 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:24:47.698 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:24:47.698 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:24:47.699 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:24:47.699 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:24:47.699 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:24:47.705 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:24:47.705 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:24:47.720 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:47.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:47.721 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:24:47.722 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:24:47.723 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:24:47.740 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:24:47.740 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:24:47.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:47 smithi032 ceph-mon[20004]: 1.b scrub starts 2024-08-19T14:24:47.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:47 smithi032 ceph-mon[20004]: 1.b scrub ok 2024-08-19T14:24:47.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:47 smithi032 ceph-mon[20004]: pgmap v967: 47 pgs: 1 creating+peering, 14 unknown, 32 active+clean; 577 KiB data, 4.2 GiB used, 800 GiB / 805 GiB avail 2024-08-19T14:24:47.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:47 smithi032 ceph-mon[20004]: 1.6 scrub starts 2024-08-19T14:24:47.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:47 smithi032 ceph-mon[20004]: 1.6 scrub ok 2024-08-19T14:24:47.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:47 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1932820951' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-08-19T14:24:47.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:47 smithi032 ceph-mon[20004]: osdmap e422: 12 total, 12 up, 9 in 2024-08-19T14:24:47.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:47 smithi032 ceph-mon[20004]: 1.5 scrub starts 2024-08-19T14:24:47.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:47 smithi032 ceph-mon[20004]: 1.5 scrub ok 2024-08-19T14:24:47.758 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:24:47.759 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:24:47.759 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:47 smithi031 ceph-mon[16336]: 1.b scrub starts 2024-08-19T14:24:47.759 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:47 smithi031 ceph-mon[16336]: 1.b scrub ok 2024-08-19T14:24:47.759 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:47 smithi031 ceph-mon[16336]: pgmap v967: 47 pgs: 1 creating+peering, 14 unknown, 32 active+clean; 577 KiB data, 4.2 GiB used, 800 GiB / 805 GiB avail 2024-08-19T14:24:47.759 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:47 smithi031 ceph-mon[16336]: 1.6 scrub starts 2024-08-19T14:24:47.759 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:47 smithi031 ceph-mon[16336]: 1.6 scrub ok 2024-08-19T14:24:47.759 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:47 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1932820951' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-08-19T14:24:47.759 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:47 smithi031 ceph-mon[16336]: osdmap e422: 12 total, 12 up, 9 in 2024-08-19T14:24:47.759 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:47 smithi031 ceph-mon[16336]: 1.5 scrub starts 2024-08-19T14:24:47.760 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:47 smithi031 ceph-mon[16336]: 1.5 scrub ok 2024-08-19T14:24:47.777 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:24:47.777 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:24:47.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:47 smithi045 ceph-mon[20143]: 1.b scrub starts 2024-08-19T14:24:47.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:47 smithi045 ceph-mon[20143]: 1.b scrub ok 2024-08-19T14:24:47.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:47 smithi045 ceph-mon[20143]: pgmap v967: 47 pgs: 1 creating+peering, 14 unknown, 32 active+clean; 577 KiB data, 4.2 GiB used, 800 GiB / 805 GiB avail 2024-08-19T14:24:47.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:47 smithi045 ceph-mon[20143]: 1.6 scrub starts 2024-08-19T14:24:47.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:47 smithi045 ceph-mon[20143]: 1.6 scrub ok 2024-08-19T14:24:47.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:47 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1932820951' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-08-19T14:24:47.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:47 smithi045 ceph-mon[20143]: osdmap e422: 12 total, 12 up, 9 in 2024-08-19T14:24:47.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:47 smithi045 ceph-mon[20143]: 1.5 scrub starts 2024-08-19T14:24:47.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:47 smithi045 ceph-mon[20143]: 1.5 scrub ok 2024-08-19T14:24:47.822 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:24:47.843 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:47.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:47.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:24:47.945 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:24:47.970 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:47.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:47.971 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:24:47.995 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:24:47.996 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:24:48.071 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:24:48.096 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:48.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:48.097 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:24:48.198 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:24:48.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:48.223 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:48.223 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:24:48.255 DEBUG:teuthology.orchestra.run.smithi045:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_3 bench 90 write' 2024-08-19T14:24:48.266 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-08-19T14:24:48.266 INFO:teuthology.orchestra.run:waiting for 3000 2024-08-19T14:24:48.315 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:hints = 1 2024-08-19T14:24:48.315 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-08-19T14:24:48.315 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Object prefix: benchmark_data_smithi045_85309 2024-08-19T14:24:48.316 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-19T14:24:48.316 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 0 0 0 0 0 0 - 0 2024-08-19T14:24:48.324 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:24:48.327 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-19T14:24:48.358 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:48.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:48.359 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:24:48.460 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:24:48.494 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:48.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:48.496 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:24:48.533 INFO:teuthology.orchestra.run.smithi031.stderr:reweighted osd.4 to 1 (10000) 2024-08-19T14:24:48.577 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-08-19T14:24:48.596 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:24:48.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:48 smithi031 ceph-mon[16336]: 1.1b scrub starts 2024-08-19T14:24:48.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:48 smithi031 ceph-mon[16336]: 1.1b scrub ok 2024-08-19T14:24:48.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:48 smithi031 ceph-mon[16336]: 1.7 scrub starts 2024-08-19T14:24:48.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:48 smithi031 ceph-mon[16336]: 1.7 scrub ok 2024-08-19T14:24:48.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:48 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3565881632' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:24:48.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:48 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2527472491' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-08-19T14:24:48.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:48 smithi031 ceph-mon[16336]: 1.0 scrub starts 2024-08-19T14:24:48.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:48 smithi031 ceph-mon[16336]: 1.0 scrub ok 2024-08-19T14:24:48.628 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:48.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:48.629 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:24:48.730 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:24:48.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:48.770 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:48.770 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:24:48.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:48 smithi045 ceph-mon[20143]: 1.1b scrub starts 2024-08-19T14:24:48.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:48 smithi045 ceph-mon[20143]: 1.1b scrub ok 2024-08-19T14:24:48.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:48 smithi045 ceph-mon[20143]: 1.7 scrub starts 2024-08-19T14:24:48.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:48 smithi045 ceph-mon[20143]: 1.7 scrub ok 2024-08-19T14:24:48.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:48 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3565881632' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:24:48.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:48 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2527472491' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-08-19T14:24:48.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:48 smithi045 ceph-mon[20143]: 1.0 scrub starts 2024-08-19T14:24:48.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:48 smithi045 ceph-mon[20143]: 1.0 scrub ok 2024-08-19T14:24:48.870 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:24:48.905 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:48.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:48.906 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:24:49.007 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:24:49.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:48 smithi032 ceph-mon[20004]: 1.1b scrub starts 2024-08-19T14:24:49.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:48 smithi032 ceph-mon[20004]: 1.1b scrub ok 2024-08-19T14:24:49.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:48 smithi032 ceph-mon[20004]: 1.7 scrub starts 2024-08-19T14:24:49.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:48 smithi032 ceph-mon[20004]: 1.7 scrub ok 2024-08-19T14:24:49.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:48 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3565881632' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:24:49.009 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:48 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2527472491' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-08-19T14:24:49.009 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:48 smithi032 ceph-mon[20004]: 1.0 scrub starts 2024-08-19T14:24:49.009 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:48 smithi032 ceph-mon[20004]: 1.0 scrub ok 2024-08-19T14:24:49.009 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:24:49.045 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:49.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:49.046 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:24:49.147 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:24:49.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:49.182 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:49.182 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:24:49.283 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:24:49.316 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 1 16 4734 4718 294.859 294.875 0.00357707 0.003381 2024-08-19T14:24:49.317 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:49.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:49.318 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:24:49.330 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd reweight 5 1 2024-08-19T14:24:49.401 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:24:49.419 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:24:49.452 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:49.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:49.453 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:24:49.554 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:24:49.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:49.582 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:49.582 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:24:49.585 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:24:49.585 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:24:49.683 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:24:49.704 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:49.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:49.705 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:24:49.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:49 smithi032 ceph-mon[20004]: pgmap v969: 47 pgs: 1 creating+peering, 4 unknown, 42 active+clean; 577 KiB data, 4.2 GiB used, 800 GiB / 805 GiB avail 2024-08-19T14:24:49.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:49 smithi032 ceph-mon[20004]: 1.1 scrub starts 2024-08-19T14:24:49.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:49 smithi032 ceph-mon[20004]: 1.1 scrub ok 2024-08-19T14:24:49.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:49 smithi032 ceph-mon[20004]: 1.17 scrub starts 2024-08-19T14:24:49.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:49 smithi032 ceph-mon[20004]: 1.17 scrub ok 2024-08-19T14:24:49.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:49 smithi032 ceph-mon[20004]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-19T14:24:49.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:49 smithi032 ceph-mon[20004]: Cluster is now healthy 2024-08-19T14:24:49.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:49 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2527472491' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-08-19T14:24:49.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:49 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:24:49.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:49 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:24:49.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:49 smithi032 ceph-mon[20004]: osdmap e423: 12 total, 12 up, 9 in 2024-08-19T14:24:49.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:49 smithi032 ceph-mon[20004]: osdmap e424: 12 total, 12 up, 9 in 2024-08-19T14:24:49.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:49 smithi045 ceph-mon[20143]: pgmap v969: 47 pgs: 1 creating+peering, 4 unknown, 42 active+clean; 577 KiB data, 4.2 GiB used, 800 GiB / 805 GiB avail 2024-08-19T14:24:49.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:49 smithi045 ceph-mon[20143]: 1.1 scrub starts 2024-08-19T14:24:49.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:49 smithi045 ceph-mon[20143]: 1.1 scrub ok 2024-08-19T14:24:49.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:49 smithi045 ceph-mon[20143]: 1.17 scrub starts 2024-08-19T14:24:49.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:49 smithi045 ceph-mon[20143]: 1.17 scrub ok 2024-08-19T14:24:49.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:49 smithi045 ceph-mon[20143]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-19T14:24:49.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:49 smithi045 ceph-mon[20143]: Cluster is now healthy 2024-08-19T14:24:49.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:49 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2527472491' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-08-19T14:24:49.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:49 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:24:49.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:49 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:24:49.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:49 smithi045 ceph-mon[20143]: osdmap e423: 12 total, 12 up, 9 in 2024-08-19T14:24:49.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:49 smithi045 ceph-mon[20143]: osdmap e424: 12 total, 12 up, 9 in 2024-08-19T14:24:49.806 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:24:49.827 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:49.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:49.828 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:24:49.858 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:49 smithi031 ceph-mon[16336]: pgmap v969: 47 pgs: 1 creating+peering, 4 unknown, 42 active+clean; 577 KiB data, 4.2 GiB used, 800 GiB / 805 GiB avail 2024-08-19T14:24:49.859 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:49 smithi031 ceph-mon[16336]: 1.1 scrub starts 2024-08-19T14:24:49.859 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:49 smithi031 ceph-mon[16336]: 1.1 scrub ok 2024-08-19T14:24:49.859 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:49 smithi031 ceph-mon[16336]: 1.17 scrub starts 2024-08-19T14:24:49.859 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:49 smithi031 ceph-mon[16336]: 1.17 scrub ok 2024-08-19T14:24:49.859 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:49 smithi031 ceph-mon[16336]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-19T14:24:49.859 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:49 smithi031 ceph-mon[16336]: Cluster is now healthy 2024-08-19T14:24:49.859 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:49 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2527472491' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-08-19T14:24:49.859 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:49 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:24:49.859 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:49 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:24:49.859 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:49 smithi031 ceph-mon[16336]: osdmap e423: 12 total, 12 up, 9 in 2024-08-19T14:24:49.859 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:49 smithi031 ceph-mon[16336]: osdmap e424: 12 total, 12 up, 9 in 2024-08-19T14:24:49.925 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:24:49.929 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:24:49.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:49.951 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:49.952 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:24:50.053 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:24:50.086 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:50.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:50.087 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:24:50.188 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:24:50.224 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:50.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:50.225 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:24:50.316 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 2 16 7095 7079 221.203 147.562 0.00379765 0.00451327 2024-08-19T14:24:50.326 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:24:50.358 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:50.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:50.360 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:24:50.461 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:24:50.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:50.495 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:50.495 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:24:50.552 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is set 2024-08-19T14:24:50.596 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:24:50.631 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:50.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:50.632 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:24:50.635 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-08-19T14:24:50.723 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:24:50.733 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:24:50.755 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:50 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/842521394' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:24:50.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:50.790 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:50.790 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:24:50.813 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:50 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/842521394' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:24:50.891 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:24:50.919 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:50.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:50.920 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:24:51.021 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:24:51.025 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:24:51.043 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:51.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:51.044 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:24:51.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:50 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/842521394' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:24:51.144 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:24:51.173 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:51.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:51.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:24:51.275 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:24:51.312 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:51.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:51.313 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:24:51.316 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 3 16 10302 10286 214.273 200.438 0.00328868 0.00464503 2024-08-19T14:24:51.323 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:24:51.414 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:24:51.447 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:51.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:51.448 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:24:51.549 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:24:51.554 INFO:teuthology.orchestra.run.smithi031.stderr:reweighted osd.5 to 1 (10000) 2024-08-19T14:24:51.590 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:51.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:51.592 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:24:51.612 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:24:51.612 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:24:51.693 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:24:51.716 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:51 smithi032 ceph-mon[20004]: pgmap v972: 47 pgs: 47 active+clean; 19 MiB data, 4.3 GiB used, 800 GiB / 805 GiB avail; 4.1 MiB/s wr, 66 op/s 2024-08-19T14:24:51.716 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:51 smithi032 ceph-mon[20004]: 1.a scrub starts 2024-08-19T14:24:51.716 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:51 smithi032 ceph-mon[20004]: 1.a scrub ok 2024-08-19T14:24:51.716 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:51 smithi032 ceph-mon[20004]: 1.d deep-scrub starts 2024-08-19T14:24:51.716 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:51 smithi032 ceph-mon[20004]: 1.d deep-scrub ok 2024-08-19T14:24:51.716 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:51 smithi032 ceph-mon[20004]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:24:51.716 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:51 smithi032 ceph-mon[20004]: 1.11 deep-scrub starts 2024-08-19T14:24:51.717 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:51 smithi032 ceph-mon[20004]: 1.11 deep-scrub ok 2024-08-19T14:24:51.717 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:51 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/842521394' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:24:51.717 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:51 smithi032 ceph-mon[20004]: osdmap e425: 12 total, 12 up, 9 in 2024-08-19T14:24:51.717 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:51 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/4056507060' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-08-19T14:24:51.725 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:51.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:51.726 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:24:51.827 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:24:51.859 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:51.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:51.860 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:24:51.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:51 smithi031 ceph-mon[16336]: pgmap v972: 47 pgs: 47 active+clean; 19 MiB data, 4.3 GiB used, 800 GiB / 805 GiB avail; 4.1 MiB/s wr, 66 op/s 2024-08-19T14:24:51.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:51 smithi031 ceph-mon[16336]: 1.a scrub starts 2024-08-19T14:24:51.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:51 smithi031 ceph-mon[16336]: 1.a scrub ok 2024-08-19T14:24:51.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:51 smithi031 ceph-mon[16336]: 1.d deep-scrub starts 2024-08-19T14:24:51.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:51 smithi031 ceph-mon[16336]: 1.d deep-scrub ok 2024-08-19T14:24:51.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:51 smithi031 ceph-mon[16336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:24:51.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:51 smithi031 ceph-mon[16336]: 1.11 deep-scrub starts 2024-08-19T14:24:51.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:51 smithi031 ceph-mon[16336]: 1.11 deep-scrub ok 2024-08-19T14:24:51.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:51 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/842521394' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:24:51.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:51 smithi031 ceph-mon[16336]: osdmap e425: 12 total, 12 up, 9 in 2024-08-19T14:24:51.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:51 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/4056507060' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-08-19T14:24:51.961 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:24:51.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:51.996 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:51.996 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:24:52.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:51 smithi045 ceph-mon[20143]: pgmap v972: 47 pgs: 47 active+clean; 19 MiB data, 4.3 GiB used, 800 GiB / 805 GiB avail; 4.1 MiB/s wr, 66 op/s 2024-08-19T14:24:52.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:51 smithi045 ceph-mon[20143]: 1.a scrub starts 2024-08-19T14:24:52.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:51 smithi045 ceph-mon[20143]: 1.a scrub ok 2024-08-19T14:24:52.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:51 smithi045 ceph-mon[20143]: 1.d deep-scrub starts 2024-08-19T14:24:52.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:51 smithi045 ceph-mon[20143]: 1.d deep-scrub ok 2024-08-19T14:24:52.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:51 smithi045 ceph-mon[20143]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:24:52.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:51 smithi045 ceph-mon[20143]: 1.11 deep-scrub starts 2024-08-19T14:24:52.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:51 smithi045 ceph-mon[20143]: 1.11 deep-scrub ok 2024-08-19T14:24:52.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:51 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/842521394' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:24:52.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:51 smithi045 ceph-mon[20143]: osdmap e425: 12 total, 12 up, 9 in 2024-08-19T14:24:52.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:51 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/4056507060' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-08-19T14:24:52.097 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:24:52.099 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:24:52.100 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:24:52.126 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:24:52.126 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:24:52.131 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:52.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:52.132 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:24:52.153 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:24:52.153 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:24:52.183 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:24:52.184 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:24:52.216 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:24:52.217 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:24:52.233 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:24:52.254 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:24:52.254 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:24:52.264 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:52.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:52.265 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:24:52.285 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:24:52.285 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:24:52.315 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:24:52.316 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:24:52.316 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 4 16 13851 13835 216.152 221.812 0.00543388 0.00462042 2024-08-19T14:24:52.344 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:24:52.345 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:24:52.366 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:24:52.371 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-08-19T14:24:52.377 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:24:52.377 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:24:52.397 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:52.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:52.398 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:24:52.406 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:24:52.406 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:24:52.433 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:24:52.433 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:24:52.499 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:24:52.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:52.598 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:52.599 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:24:52.694 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd reweight 7 1 2024-08-19T14:24:52.699 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:24:52.715 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:52 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/4056507060' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-08-19T14:24:52.715 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:52 smithi031 ceph-mon[16336]: osdmap e426: 12 total, 12 up, 9 in 2024-08-19T14:24:52.715 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:52 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/629878946' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:24:52.722 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:52.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:52.723 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:24:52.804 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:24:52.824 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:24:52.854 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:52.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:52.855 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:24:52.944 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:52 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/4056507060' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-08-19T14:24:52.944 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:52 smithi032 ceph-mon[20004]: osdmap e426: 12 total, 12 up, 9 in 2024-08-19T14:24:52.945 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:52 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/629878946' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:24:52.956 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:24:52.977 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:52.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:52.978 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:24:53.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:52 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/4056507060' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-08-19T14:24:53.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:52 smithi045 ceph-mon[20143]: osdmap e426: 12 total, 12 up, 9 in 2024-08-19T14:24:53.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:52 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/629878946' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:24:53.079 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:24:53.102 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:53.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:53.103 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:24:53.204 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:24:53.241 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:53.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:53.242 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:24:53.254 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-19T14:24:53.316 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 5 16 16353 16337 204.194 156.375 0.00175081 0.00489077 2024-08-19T14:24:53.343 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:24:53.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:53.375 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:53.375 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:24:53.413 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:24:53.413 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-08-19T14:24:53.413 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-08-19T14:24:53.413 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [] 2024-08-19T14:24:53.413 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:24:53.476 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:24:53.512 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:53.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:53.513 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:24:53.613 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:24:53.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:53.648 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:53.649 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:24:53.750 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:24:53.782 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:53.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:53.783 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:24:53.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:53 smithi031 ceph-mon[16336]: pgmap v975: 47 pgs: 47 active+clean; 419 MiB data, 5.1 GiB used, 800 GiB / 805 GiB avail; 105 MiB/s wr, 1.67k op/s 2024-08-19T14:24:53.884 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:24:53.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:53.917 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:53.917 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:24:54.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:53 smithi032 ceph-mon[20004]: pgmap v975: 47 pgs: 47 active+clean; 419 MiB data, 5.1 GiB used, 800 GiB / 805 GiB avail; 105 MiB/s wr, 1.67k op/s 2024-08-19T14:24:54.019 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:24:54.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:53 smithi045 ceph-mon[20143]: pgmap v975: 47 pgs: 47 active+clean; 419 MiB data, 5.1 GiB used, 800 GiB / 805 GiB avail; 105 MiB/s wr, 1.67k op/s 2024-08-19T14:24:54.051 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:54.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:54.052 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:24:54.153 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:24:54.185 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:54.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:54.186 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:24:54.287 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:24:54.316 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 6 16 20071 20055 208.887 232.375 0.00490161 0.00478157 2024-08-19T14:24:54.318 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:54.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:54.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:24:54.377 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:24:54.384 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:24:54.420 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:24:54.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:54.463 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:54.463 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:24:54.465 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-08-19T14:24:54.564 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:24:54.597 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:54.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:54.598 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:24:54.699 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:24:54.720 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:54.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:54.721 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:24:54.823 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:24:54.857 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:54.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:54.858 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:24:54.902 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:24:54.959 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:24:54.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:54.987 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:54.987 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:24:55.090 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:24:55.125 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:55.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:55.126 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:24:55.227 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:24:55.265 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:55.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:55.266 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:24:55.316 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 7 16 23104 23088 206.124 189.562 0.00187983 0.00481309 2024-08-19T14:24:55.367 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:24:55.388 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:55.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:55.389 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:24:55.490 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:24:55.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:55.527 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:55.527 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:24:55.549 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:24:55.549 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-08-19T14:24:55.549 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:55.549 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:23411 5.9 5:90ebfdb5:::benchmark_data_smithi045_85309_object23410:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e426)", 2024-08-19T14:24:55.549 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:24:55.546268+0000", 2024-08-19T14:24:55.549 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.00085894300000000003, 2024-08-19T14:24:55.549 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00089271200000000004, 2024-08-19T14:24:55.549 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:24:55.549 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:24:55.549 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "waiting for sub ops", 2024-08-19T14:24:55.549 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:24:55.550 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:24:55.550 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:24:55.550 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 23411 2024-08-19T14:24:55.550 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:55.550 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:24:55.550 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:55.550 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:24:55.550 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.546268+0000", 2024-08-19T14:24:55.550 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:24:55.550 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:55.550 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:55.550 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:24:55.550 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.546268+0000", 2024-08-19T14:24:55.551 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:24:55.551 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:55.551 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:55.551 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:24:55.551 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.546269+0000", 2024-08-19T14:24:55.551 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.282e-06 2024-08-19T14:24:55.551 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:55.551 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:55.551 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:24:55.552 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.546335+0000", 2024-08-19T14:24:55.552 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6528999999999999e-05 2024-08-19T14:24:55.552 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:55.552 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:55.552 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:24:55.552 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.546351+0000", 2024-08-19T14:24:55.552 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5909999999999998e-05 2024-08-19T14:24:55.552 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:55.552 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:55.552 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:24:55.552 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.546359+0000", 2024-08-19T14:24:55.552 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2880000000000004e-06 2024-08-19T14:24:55.552 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:55.552 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:55.553 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:24:55.553 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.546411+0000", 2024-08-19T14:24:55.553 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2938e-05 2024-08-19T14:24:55.553 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:55.553 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:55.553 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:24:55.553 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.546659+0000", 2024-08-19T14:24:55.553 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00024755900000000001 2024-08-19T14:24:55.553 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:55.554 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:55.554 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 9,11", 2024-08-19T14:24:55.554 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.546808+0000", 2024-08-19T14:24:55.554 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014882099999999999 2024-08-19T14:24:55.554 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:24:55.554 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:24:55.554 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:24:55.554 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:55.554 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:55.555 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.25813.0:23407 5.1 e426/419 5:83415e7b:::benchmark_data_smithi045_85309_object23406:head v 426'1474, mlcod=426'1471)", 2024-08-19T14:24:55.555 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:24:55.546395+0000", 2024-08-19T14:24:55.555 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.00073189900000000005, 2024-08-19T14:24:55.555 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00084377900000000001, 2024-08-19T14:24:55.555 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:24:55.555 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:24:55.555 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-08-19T14:24:55.555 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:24:55.555 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:55.555 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:24:55.555 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.546395+0000", 2024-08-19T14:24:55.555 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:24:55.555 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:55.555 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:55.556 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:24:55.556 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.546395+0000", 2024-08-19T14:24:55.556 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:24:55.556 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:55.556 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:55.556 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:24:55.556 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.546395+0000", 2024-08-19T14:24:55.556 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4499999999999997e-07 2024-08-19T14:24:55.556 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:55.556 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:55.556 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:24:55.556 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.546417+0000", 2024-08-19T14:24:55.556 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1617999999999999e-05 2024-08-19T14:24:55.557 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:55.557 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:55.557 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:24:55.557 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.546428+0000", 2024-08-19T14:24:55.557 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0927999999999999e-05 2024-08-19T14:24:55.557 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:55.557 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:55.557 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:24:55.557 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.546434+0000", 2024-08-19T14:24:55.557 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6239999999999996e-06 2024-08-19T14:24:55.557 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:55.558 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:55.558 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:24:55.558 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.546500+0000", 2024-08-19T14:24:55.558 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6097000000000001e-05 2024-08-19T14:24:55.558 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:55.558 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:55.558 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:24:55.558 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.546520+0000", 2024-08-19T14:24:55.558 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9137999999999999e-05 2024-08-19T14:24:55.558 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:24:55.558 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:24:55.558 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:24:55.558 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:55.559 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:55.559 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:23415 5.9 5.dea881c9 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e426)", 2024-08-19T14:24:55.559 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:24:55.546685+0000", 2024-08-19T14:24:55.559 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.00044161899999999997, 2024-08-19T14:24:55.559 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00060967499999999995, 2024-08-19T14:24:55.559 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:24:55.559 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:24:55.559 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "queued for pg", 2024-08-19T14:24:55.559 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:24:55.559 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:24:55.559 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:24:55.559 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 23415 2024-08-19T14:24:55.559 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:55.559 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:24:55.560 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:55.560 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:24:55.560 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.546685+0000", 2024-08-19T14:24:55.560 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:24:55.560 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:55.560 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:55.560 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:24:55.560 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.546685+0000", 2024-08-19T14:24:55.560 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:24:55.560 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:55.560 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:55.560 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:24:55.560 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.546686+0000", 2024-08-19T14:24:55.560 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8800000000000002e-07 2024-08-19T14:24:55.561 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:55.561 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:55.561 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:24:55.561 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.546740+0000", 2024-08-19T14:24:55.561 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4021000000000003e-05 2024-08-19T14:24:55.561 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:55.561 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:55.561 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:24:55.561 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.546748+0000", 2024-08-19T14:24:55.561 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.96e-06 2024-08-19T14:24:55.561 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:55.561 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:55.561 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:24:55.561 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.546752+0000", 2024-08-19T14:24:55.562 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6959999999999996e-06 2024-08-19T14:24:55.562 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:24:55.562 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:24:55.562 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:24:55.562 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:55.562 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:55.562 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.25813.0:23410 5.d e426/419 5:b6e6d674:::benchmark_data_smithi045_85309_object23409:head v 426'1443, mlcod=426'1441)", 2024-08-19T14:24:55.562 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:24:55.546559+0000", 2024-08-19T14:24:55.562 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.00056725999999999999, 2024-08-19T14:24:55.562 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00078926699999999996, 2024-08-19T14:24:55.562 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:24:55.562 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:24:55.562 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-08-19T14:24:55.563 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:24:55.563 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:55.563 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:24:55.563 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.546559+0000", 2024-08-19T14:24:55.563 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:24:55.563 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:55.563 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:55.563 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:24:55.563 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.546559+0000", 2024-08-19T14:24:55.563 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:24:55.563 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:55.563 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:55.563 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:24:55.563 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.546560+0000", 2024-08-19T14:24:55.564 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.2399999999999996e-07 2024-08-19T14:24:55.564 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:55.564 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:55.564 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:24:55.564 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.546866+0000", 2024-08-19T14:24:55.564 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000305999 2024-08-19T14:24:55.564 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:55.564 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:55.564 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:24:55.564 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.546874+0000", 2024-08-19T14:24:55.564 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4710000000000001e-06 2024-08-19T14:24:55.564 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:55.564 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:55.564 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:24:55.565 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.546884+0000", 2024-08-19T14:24:55.565 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0611000000000001e-05 2024-08-19T14:24:55.565 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:55.565 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:55.565 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:24:55.565 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.546932+0000", 2024-08-19T14:24:55.565 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7809000000000003e-05 2024-08-19T14:24:55.565 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:55.565 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:55.565 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:24:55.565 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.546952+0000", 2024-08-19T14:24:55.565 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0041e-05 2024-08-19T14:24:55.565 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:24:55.565 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:24:55.566 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:24:55.566 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:55.566 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:55.566 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.25813.0:23412 5.2 e426/419 5:41491314:::benchmark_data_smithi045_85309_object23411:head v 426'1450, mlcod=426'1447)", 2024-08-19T14:24:55.566 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:24:55.546911+0000", 2024-08-19T14:24:55.566 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.00021497500000000001, 2024-08-19T14:24:55.567 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00049956800000000004, 2024-08-19T14:24:55.567 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:24:55.567 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:24:55.567 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-08-19T14:24:55.567 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:24:55.567 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:55.567 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:24:55.567 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.546911+0000", 2024-08-19T14:24:55.568 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:24:55.568 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:55.568 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:55.568 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:24:55.568 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.546911+0000", 2024-08-19T14:24:55.568 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:24:55.568 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:55.568 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:55.568 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:24:55.568 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.546912+0000", 2024-08-19T14:24:55.568 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6100000000000001e-07 2024-08-19T14:24:55.568 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:55.569 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:55.569 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:24:55.569 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.546938+0000", 2024-08-19T14:24:55.569 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6486999999999999e-05 2024-08-19T14:24:55.569 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:55.569 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:55.569 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:24:55.569 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.546945+0000", 2024-08-19T14:24:55.569 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.888e-06 2024-08-19T14:24:55.569 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:55.569 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:55.569 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:24:55.569 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.546951+0000", 2024-08-19T14:24:55.569 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8749999999999997e-06 2024-08-19T14:24:55.569 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:55.570 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:55.570 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:24:55.570 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.547029+0000", 2024-08-19T14:24:55.570 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7507000000000002e-05 2024-08-19T14:24:55.570 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:24:55.570 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:24:55.570 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:24:55.570 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.547050+0000", 2024-08-19T14:24:55.570 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1409000000000001e-05 2024-08-19T14:24:55.570 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:24:55.571 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:24:55.571 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:24:55.571 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:24:55.571 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-08-19T14:24:55.571 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 5 2024-08-19T14:24:55.571 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:24:55.578 INFO:teuthology.orchestra.run.smithi031.stderr:reweighted osd.7 to 1 (10000) 2024-08-19T14:24:55.628 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:24:55.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:55.665 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:55.665 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:24:55.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:55 smithi032 ceph-mon[20004]: pgmap v976: 47 pgs: 47 active+clean; 800 MiB data, 6.3 GiB used, 798 GiB / 805 GiB avail; 145 MiB/s wr, 2.32k op/s 2024-08-19T14:24:55.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:55 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/411959101' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-08-19T14:24:55.766 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:24:55.799 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:55.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:55.801 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:24:55.848 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:55 smithi045 ceph-mon[20143]: pgmap v976: 47 pgs: 47 active+clean; 800 MiB data, 6.3 GiB used, 798 GiB / 805 GiB avail; 145 MiB/s wr, 2.32k op/s 2024-08-19T14:24:55.848 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:55 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/411959101' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-08-19T14:24:55.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:55 smithi031 ceph-mon[16336]: pgmap v976: 47 pgs: 47 active+clean; 800 MiB data, 6.3 GiB used, 798 GiB / 805 GiB avail; 145 MiB/s wr, 2.32k op/s 2024-08-19T14:24:55.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:55 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/411959101' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-08-19T14:24:55.902 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:24:55.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:55.938 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:55.939 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:24:56.040 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:24:56.062 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:56.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:56.063 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:24:56.164 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:24:56.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:56.199 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:56.200 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:24:56.255 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-08-19T14:24:56.272 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:24:56.295 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd reweight 8 1 2024-08-19T14:24:56.300 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:24:56.316 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 8 15 25498 25483 199.068 149.688 0.00197026 0.00501779 2024-08-19T14:24:56.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:56.343 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:56.343 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:24:56.444 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:24:56.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:56.482 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:56.483 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:24:56.584 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:24:56.598 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is set 2024-08-19T14:24:56.617 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:56.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:56.618 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:24:56.666 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:24:56.719 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:24:56.752 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:56.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:56.754 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:24:56.855 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:24:56.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:56 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/411959101' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-08-19T14:24:56.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:56 smithi031 ceph-mon[16336]: osdmap e427: 12 total, 12 up, 9 in 2024-08-19T14:24:56.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:56 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2472016711' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:24:56.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:56.896 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:56.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:24:56.997 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:24:57.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:56 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/411959101' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-08-19T14:24:57.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:56 smithi032 ceph-mon[20004]: osdmap e427: 12 total, 12 up, 9 in 2024-08-19T14:24:57.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:56 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2472016711' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:24:57.036 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:57.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:57.038 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:24:57.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:56 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/411959101' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-08-19T14:24:57.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:56 smithi045 ceph-mon[20143]: osdmap e427: 12 total, 12 up, 9 in 2024-08-19T14:24:57.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:56 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2472016711' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:24:57.138 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:24:57.173 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:57.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:57.174 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:24:57.275 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:24:57.305 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:57.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:57.306 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:24:57.316 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 9 16 28515 28499 197.892 188.5 0.00421517 0.00504831 2024-08-19T14:24:57.352 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:24:57.407 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:24:57.442 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:57.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:57.443 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:24:57.501 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:24:57.501 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-08-19T14:24:57.502 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-08-19T14:24:57.502 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-08-19T14:24:57.502 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:24:57.543 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:24:57.578 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:57.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:57.579 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:24:57.649 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:24:57.680 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:24:57.713 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:57.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:57.714 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:24:57.815 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:24:57.845 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:57.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:57.847 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:24:57.856 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:57 smithi032 ceph-mon[20004]: pgmap v978: 47 pgs: 47 active+clean; 865 MiB data, 6.6 GiB used, 798 GiB / 805 GiB avail; 141 MiB/s wr, 2.25k op/s 2024-08-19T14:24:57.856 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:57 smithi032 ceph-mon[20004]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:24:57.856 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:57 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2472016711' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:24:57.856 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:57 smithi032 ceph-mon[20004]: osdmap e428: 12 total, 12 up, 9 in 2024-08-19T14:24:57.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:57 smithi031 ceph-mon[16336]: pgmap v978: 47 pgs: 47 active+clean; 865 MiB data, 6.6 GiB used, 798 GiB / 805 GiB avail; 141 MiB/s wr, 2.25k op/s 2024-08-19T14:24:57.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:57 smithi031 ceph-mon[16336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:24:57.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:57 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2472016711' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:24:57.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:57 smithi031 ceph-mon[16336]: osdmap e428: 12 total, 12 up, 9 in 2024-08-19T14:24:57.948 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:24:57.981 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:57.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:57.983 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:24:58.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:57 smithi045 ceph-mon[20143]: pgmap v978: 47 pgs: 47 active+clean; 865 MiB data, 6.6 GiB used, 798 GiB / 805 GiB avail; 141 MiB/s wr, 2.25k op/s 2024-08-19T14:24:58.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:57 smithi045 ceph-mon[20143]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:24:58.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:57 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2472016711' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:24:58.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:57 smithi045 ceph-mon[20143]: osdmap e428: 12 total, 12 up, 9 in 2024-08-19T14:24:58.083 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:24:58.115 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:58.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:58.117 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:24:58.218 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:24:58.252 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:58.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:58.253 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:24:58.317 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 10 16 31530 31514 196.944 188.438 0.00208213 0.00506405 2024-08-19T14:24:58.353 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:24:58.381 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:58.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:58.382 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:24:58.483 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:24:58.516 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:58.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:58.517 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:24:58.618 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:24:58.648 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:58.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:58.650 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:24:58.750 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:24:58.775 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:58.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:58.776 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:24:58.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:58 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/754777445' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:24:58.877 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:24:58.908 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:58.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:58.909 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:24:59.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:58 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/754777445' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:24:59.010 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:24:59.045 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:59.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:59.046 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:24:59.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:58 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/754777445' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:24:59.147 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:24:59.181 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:59.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:59.182 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:24:59.283 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:24:59.316 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 11 16 34096 34080 193.619 160.375 0.00348619 0.00515961 2024-08-19T14:24:59.318 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:59.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:59.319 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:24:59.358 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-19T14:24:59.365 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:24:59.366 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:24:59.366 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:24:59.366 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:24:59.366 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:24:59.366 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:24:59.366 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:24:59.366 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:24:59.367 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:24:59.367 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:24:59.367 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:24:59.367 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:24:59.367 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:24:59.367 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:24:59.367 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:24:59.367 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:24:59.367 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:24:59.368 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:24:59.368 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:24:59.368 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:24:59.368 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:24:59.368 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:24:59.368 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:24:59.368 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:24:59.420 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:24:59.457 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:59.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:59.458 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:24:59.559 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:24:59.593 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:59.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:59.594 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:24:59.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:24:59 smithi031 ceph-mon[16336]: pgmap v980: 47 pgs: 47 active+clean; 1.4 GiB data, 8.3 GiB used, 796 GiB / 805 GiB avail; 190 MiB/s wr, 3.05k op/s 2024-08-19T14:24:59.695 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:24:59.726 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:24:59.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:59.727 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:24:59.828 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:24:59.850 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:24:59.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:59.851 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:24:59.952 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:24:59.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:24:59.989 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:24:59.989 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:25:00.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:24:59 smithi032 ceph-mon[20004]: pgmap v980: 47 pgs: 47 active+clean; 1.4 GiB data, 8.3 GiB used, 796 GiB / 805 GiB avail; 190 MiB/s wr, 3.05k op/s 2024-08-19T14:25:00.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:24:59 smithi045 ceph-mon[20143]: pgmap v980: 47 pgs: 47 active+clean; 1.4 GiB data, 8.3 GiB used, 796 GiB / 805 GiB avail; 190 MiB/s wr, 3.05k op/s 2024-08-19T14:25:00.090 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:25:00.119 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:00.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:00.120 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:25:00.221 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:25:00.253 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:00.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:00.254 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:25:00.286 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-08-19T14:25:00.316 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 12 16 36414 36398 189.556 144.875 0.00205035 0.00526972 2024-08-19T14:25:00.355 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:25:00.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:00.386 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:00.387 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:25:00.487 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:25:00.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:00.529 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:00.529 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:25:00.631 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:25:00.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:00.664 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:00.664 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:25:00.708 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:25:00.765 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:25:00.807 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:00.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:00.808 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:25:00.909 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:25:00.929 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:00.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:00.930 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:25:01.031 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:25:01.053 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:01.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:01.054 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:25:01.155 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:25:01.185 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:01.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:01.186 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:25:01.287 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:25:01.317 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 13 16 38670 38654 185.82 141 0.00663699 0.00537704 2024-08-19T14:25:01.318 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:01.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:01.319 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:25:01.420 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:25:01.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:01.456 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:01.456 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:25:01.557 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:25:01.592 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:01.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:01.593 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:25:01.694 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:25:01.735 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:01.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:01.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:25:01.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:01 smithi032 ceph-mon[20004]: pgmap v981: 47 pgs: 47 active+clean; 1.7 GiB data, 9.2 GiB used, 795 GiB / 805 GiB avail; 165 MiB/s wr, 2.64k op/s 2024-08-19T14:25:01.837 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:25:01.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:01 smithi031 ceph-mon[16336]: pgmap v981: 47 pgs: 47 active+clean; 1.7 GiB data, 9.2 GiB used, 795 GiB / 805 GiB avail; 165 MiB/s wr, 2.64k op/s 2024-08-19T14:25:01.874 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:01.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:01.875 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:25:01.932 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:25:01.941 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:25:01.941 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:25:01.976 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:25:02.000 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:02.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:02.002 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:25:02.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:01 smithi045 ceph-mon[20143]: pgmap v981: 47 pgs: 47 active+clean; 1.7 GiB data, 9.2 GiB used, 795 GiB / 805 GiB avail; 165 MiB/s wr, 2.64k op/s 2024-08-19T14:25:02.103 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:25:02.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:02.124 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:02.124 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:25:02.225 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:25:02.260 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:02.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:02.261 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:25:02.317 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 14 16 40674 40658 181.493 125.25 0.00422211 0.00550539 2024-08-19T14:25:02.362 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:25:02.391 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:02.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:02.393 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:25:02.494 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:25:02.526 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:02.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:02.527 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:25:02.623 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:02 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3150316989' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-08-19T14:25:02.623 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:02 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-08-19T14:25:02.623 INFO:teuthology.orchestra.run.smithi031.stderr:reweighted osd.8 to 1 (10000) 2024-08-19T14:25:02.628 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:25:02.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:02.668 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:02.668 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:25:02.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:02 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3150316989' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-08-19T14:25:02.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:02 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-08-19T14:25:02.769 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:25:02.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:02.803 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:02.803 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:25:02.904 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:25:02.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:02.930 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:02.930 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:25:03.031 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:25:03.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:02 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3150316989' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-08-19T14:25:03.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:02 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-08-19T14:25:03.065 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:03.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:03.066 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:25:03.103 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-08-19T14:25:03.167 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:25:03.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:03.197 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:03.197 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:25:03.298 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:25:03.317 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 15 16 43104 43088 179.517 151.875 0.00478542 0.00556595 2024-08-19T14:25:03.335 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:03.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:03.336 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:25:03.366 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd reweight 9 1 2024-08-19T14:25:03.437 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:25:03.475 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:25:03.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:03.488 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:03.488 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:25:03.589 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:25:03.622 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:03.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:03.623 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:25:03.633 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is unset 2024-08-19T14:25:03.724 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:25:03.749 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:25:03.754 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:03.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:03.755 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:25:03.856 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:25:03.894 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:03.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:03.896 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:25:03.995 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:03 smithi031 ceph-mon[16336]: pgmap v982: 47 pgs: 47 active+clean; 2.2 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 181 MiB/s wr, 2.90k op/s 2024-08-19T14:25:03.995 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:03 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-08-19T14:25:03.995 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:03 smithi031 ceph-mon[16336]: osdmap e429: 12 total, 12 up, 9 in 2024-08-19T14:25:03.995 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:03 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2384573089' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:25:03.995 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:03 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:25:03.997 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:25:04.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:03 smithi032 ceph-mon[20004]: pgmap v982: 47 pgs: 47 active+clean; 2.2 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 181 MiB/s wr, 2.90k op/s 2024-08-19T14:25:04.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:03 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-08-19T14:25:04.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:03 smithi032 ceph-mon[20004]: osdmap e429: 12 total, 12 up, 9 in 2024-08-19T14:25:04.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:03 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2384573089' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:25:04.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:03 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:25:04.009 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:03 smithi045 ceph-mon[20143]: pgmap v982: 47 pgs: 47 active+clean; 2.2 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 181 MiB/s wr, 2.90k op/s 2024-08-19T14:25:04.009 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:03 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-08-19T14:25:04.009 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:03 smithi045 ceph-mon[20143]: osdmap e429: 12 total, 12 up, 9 in 2024-08-19T14:25:04.009 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:03 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2384573089' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:25:04.009 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:03 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:25:04.035 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:04.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:04.036 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:25:04.137 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:25:04.159 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:04.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:04.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:25:04.261 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:25:04.292 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:04.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:04.293 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:25:04.317 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 16 16 45121 45105 176.176 126.062 0.0387413 0.00566703 2024-08-19T14:25:04.394 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:25:04.426 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:04.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:04.427 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:25:04.529 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:25:04.562 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:04.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:04.563 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:25:04.664 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:25:04.685 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:04.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:04.686 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:25:04.787 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:25:04.810 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:04.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:04.812 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:25:04.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:04 smithi031 ceph-mon[16336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:25:04.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:04 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:25:04.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:04 smithi031 ceph-mon[16336]: osdmap e430: 12 total, 12 up, 9 in 2024-08-19T14:25:04.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:04 smithi031 ceph-mon[16336]: 1.9 scrub starts 2024-08-19T14:25:04.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:04 smithi031 ceph-mon[16336]: 1.9 scrub ok 2024-08-19T14:25:04.912 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:25:04.933 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:04.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:04.934 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:25:04.958 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:04 smithi045 ceph-mon[20143]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:25:04.959 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:04 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:25:04.959 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:04 smithi045 ceph-mon[20143]: osdmap e430: 12 total, 12 up, 9 in 2024-08-19T14:25:04.959 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:04 smithi045 ceph-mon[20143]: 1.9 scrub starts 2024-08-19T14:25:04.959 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:04 smithi045 ceph-mon[20143]: 1.9 scrub ok 2024-08-19T14:25:05.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:04 smithi032 ceph-mon[20004]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:25:05.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:04 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:25:05.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:04 smithi032 ceph-mon[20004]: osdmap e430: 12 total, 12 up, 9 in 2024-08-19T14:25:05.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:04 smithi032 ceph-mon[20004]: 1.9 scrub starts 2024-08-19T14:25:05.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:04 smithi032 ceph-mon[20004]: 1.9 scrub ok 2024-08-19T14:25:05.035 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:25:05.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:05.074 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:05.074 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:25:05.175 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:25:05.199 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:05.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:05.200 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:25:05.300 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:25:05.301 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:25:05.301 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:25:05.317 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 17 16 46751 46735 171.804 101.875 0.00840102 0.00581628 2024-08-19T14:25:05.334 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:05.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:05.335 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:25:05.436 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:25:05.457 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:05.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:05.458 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:25:05.559 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:25:05.594 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:05.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:05.595 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:25:05.696 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:25:05.718 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:05.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:05.719 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:25:05.820 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:25:05.851 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:05.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:05.852 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:25:05.953 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:25:05.982 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:05.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:05.983 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:25:06.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:05 smithi032 ceph-mon[20004]: pgmap v985: 47 pgs: 47 active+clean; 2.5 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 186 MiB/s wr, 2.97k op/s 2024-08-19T14:25:06.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:05 smithi032 ceph-mon[20004]: 1.e scrub starts 2024-08-19T14:25:06.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:05 smithi032 ceph-mon[20004]: 1.e scrub ok 2024-08-19T14:25:06.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:05 smithi032 ceph-mon[20004]: 1.13 scrub starts 2024-08-19T14:25:06.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:05 smithi032 ceph-mon[20004]: 1.13 scrub ok 2024-08-19T14:25:06.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:05 smithi045 ceph-mon[20143]: pgmap v985: 47 pgs: 47 active+clean; 2.5 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 186 MiB/s wr, 2.97k op/s 2024-08-19T14:25:06.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:05 smithi045 ceph-mon[20143]: 1.e scrub starts 2024-08-19T14:25:06.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:05 smithi045 ceph-mon[20143]: 1.e scrub ok 2024-08-19T14:25:06.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:05 smithi045 ceph-mon[20143]: 1.13 scrub starts 2024-08-19T14:25:06.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:05 smithi045 ceph-mon[20143]: 1.13 scrub ok 2024-08-19T14:25:06.084 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:25:06.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:05 smithi031 ceph-mon[16336]: pgmap v985: 47 pgs: 47 active+clean; 2.5 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 186 MiB/s wr, 2.97k op/s 2024-08-19T14:25:06.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:05 smithi031 ceph-mon[16336]: 1.e scrub starts 2024-08-19T14:25:06.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:05 smithi031 ceph-mon[16336]: 1.e scrub ok 2024-08-19T14:25:06.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:05 smithi031 ceph-mon[16336]: 1.13 scrub starts 2024-08-19T14:25:06.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:05 smithi031 ceph-mon[16336]: 1.13 scrub ok 2024-08-19T14:25:06.117 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:06.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:06.118 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:25:06.128 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-08-19T14:25:06.219 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:25:06.249 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:06.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:06.251 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:25:06.317 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 18 15 49067 49052 170.305 144.812 0.00532508 0.00586737 2024-08-19T14:25:06.352 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:25:06.380 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:06.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:06.381 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:25:06.482 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:25:06.510 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:06.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:06.511 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:25:06.575 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:25:06.612 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:25:06.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:06.651 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:06.652 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:25:06.695 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:25:06.753 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:25:06.759 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:25:06.790 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:06.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:06.791 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:25:06.837 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:06 smithi031 ceph-mon[16336]: 1.10 scrub starts 2024-08-19T14:25:06.838 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:06 smithi031 ceph-mon[16336]: 1.10 scrub ok 2024-08-19T14:25:06.892 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:25:06.925 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:06.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:06.926 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:25:07.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:06 smithi032 ceph-mon[20004]: 1.10 scrub starts 2024-08-19T14:25:07.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:06 smithi032 ceph-mon[20004]: 1.10 scrub ok 2024-08-19T14:25:07.027 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:25:07.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:06 smithi045 ceph-mon[20143]: 1.10 scrub starts 2024-08-19T14:25:07.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:06 smithi045 ceph-mon[20143]: 1.10 scrub ok 2024-08-19T14:25:07.057 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:07.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:07.059 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:25:07.162 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:25:07.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:07.199 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:07.200 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:25:07.301 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:25:07.317 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 19 16 51354 51338 168.859 142.875 0.00326869 0.00591765 2024-08-19T14:25:07.331 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:07.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:07.332 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:25:07.433 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:25:07.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:07.457 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:07.458 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:25:07.559 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:25:07.592 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:07.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:07.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:25:07.674 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:25:07.675 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:25:07.694 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:25:07.732 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:07.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:07.733 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:25:07.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:07 smithi045 ceph-mon[20143]: pgmap v986: 47 pgs: 47 active+clean; 2.5 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 139 MiB/s wr, 2.23k op/s 2024-08-19T14:25:07.834 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:25:07.854 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:07 smithi031 ceph-mon[16336]: pgmap v986: 47 pgs: 47 active+clean; 2.5 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 139 MiB/s wr, 2.23k op/s 2024-08-19T14:25:07.870 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:07.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:07.871 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:25:07.972 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:25:08.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:08.003 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:08.003 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:25:08.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:07 smithi032 ceph-mon[20004]: pgmap v986: 47 pgs: 47 active+clean; 2.5 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 139 MiB/s wr, 2.23k op/s 2024-08-19T14:25:08.104 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:25:08.121 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:25:08.121 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:25:08.122 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:25:08.122 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:25:08.122 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:25:08.122 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:25:08.122 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:25:08.122 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:25:08.122 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:25:08.122 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:25:08.123 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:25:08.123 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:25:08.123 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:25:08.123 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:25:08.123 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:25:08.123 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:25:08.123 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:25:08.123 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:25:08.123 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:25:08.124 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:25:08.124 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:25:08.124 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:25:08.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:08.138 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:08.139 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:25:08.141 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:25:08.141 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:25:08.240 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:25:08.262 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:08.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:08.263 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:25:08.318 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:2024-08-19T14:25:08.318193+0000 min lat: 0.00112215 max lat: 0.26573 avg lat: 0.00598588 2024-08-19T14:25:08.319 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-19T14:25:08.319 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 20 16 53190 53174 166.153 114.75 0.00265716 0.00598588 2024-08-19T14:25:08.343 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-08-19T14:25:08.363 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:25:08.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:08.386 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:08.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:25:08.488 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:25:08.518 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:08.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:08.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:25:08.586 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-19T14:25:08.622 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:25:08.642 INFO:teuthology.orchestra.run.smithi031.stderr:reweighted osd.9 to 1 (10000) 2024-08-19T14:25:08.657 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:08.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:08.658 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:25:08.715 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:25:08.759 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:25:08.798 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:08.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:08.799 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:25:08.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:08 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3537142291' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:25:08.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:08 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3220086961' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-08-19T14:25:08.900 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:25:08.931 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:08.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:08.933 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:25:09.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:08 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3537142291' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:25:09.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:08 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3220086961' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-08-19T14:25:09.010 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:08 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3537142291' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:25:09.010 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:08 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3220086961' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-08-19T14:25:09.033 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:25:09.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:09.063 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:09.063 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:25:09.164 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:25:09.196 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:09.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:09.197 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:25:09.298 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:25:09.318 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 21 16 55133 55117 164.023 121.438 0.00521353 0.00609262 2024-08-19T14:25:09.330 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:25:09.331 INFO:teuthology.orchestra.run.smithi045.stdout: "size": 20, 2024-08-19T14:25:09.331 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 600, 2024-08-19T14:25:09.331 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [ 2024-08-19T14:25:09.331 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.331 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24580.0:209201 2.f 2:fb8d21d4:::benchmark_data_smithi045_46415_object85920:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194)", 2024-08-19T14:25:09.331 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:16:02.057866+0000", 2024-08-19T14:25:09.331 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 547.26863392099995, 2024-08-19T14:25:09.331 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.21458635300000001, 2024-08-19T14:25:09.331 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:25:09.331 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:25:09.331 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:25:09.332 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:25:09.332 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24580", 2024-08-19T14:25:09.332 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:25:09.332 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 209201 2024-08-19T14:25:09.332 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.332 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:25:09.332 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.332 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:25:09.332 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:16:02.057866+0000", 2024-08-19T14:25:09.332 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:25:09.332 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.332 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.332 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:25:09.332 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:16:02.057866+0000", 2024-08-19T14:25:09.333 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:25:09.333 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.333 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.333 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:25:09.333 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:16:02.057867+0000", 2024-08-19T14:25:09.333 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.540000000000001e-07 2024-08-19T14:25:09.333 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.333 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.333 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:25:09.333 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:16:02.057871+0000", 2024-08-19T14:25:09.333 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.1010000000000002e-06 2024-08-19T14:25:09.333 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.333 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.333 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:25:09.334 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:16:02.057887+0000", 2024-08-19T14:25:09.334 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6809000000000001e-05 2024-08-19T14:25:09.334 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.334 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.334 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:25:09.334 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:16:02.057893+0000", 2024-08-19T14:25:09.334 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7880000000000003e-06 2024-08-19T14:25:09.334 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.334 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.334 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:25:09.334 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:16:02.057942+0000", 2024-08-19T14:25:09.334 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.8983999999999997e-05 2024-08-19T14:25:09.334 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.335 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.335 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:25:09.335 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:16:02.058135+0000", 2024-08-19T14:25:09.335 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000192692 2024-08-19T14:25:09.335 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.335 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.335 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 4,9", 2024-08-19T14:25:09.335 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:16:02.058253+0000", 2024-08-19T14:25:09.335 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000118071 2024-08-19T14:25:09.335 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.335 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.335 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:09.335 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:16:02.062846+0000", 2024-08-19T14:25:09.335 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.004593209 2024-08-19T14:25:09.336 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.336 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.336 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:25:09.336 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:16:02.062913+0000", 2024-08-19T14:25:09.336 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.6808000000000001e-05 2024-08-19T14:25:09.336 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.336 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.336 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:09.336 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:16:02.272371+0000", 2024-08-19T14:25:09.336 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.209458071 2024-08-19T14:25:09.336 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.336 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.336 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:25:09.337 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:16:02.272407+0000", 2024-08-19T14:25:09.337 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.6031000000000001e-05 2024-08-19T14:25:09.337 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.337 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.337 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:25:09.337 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:16:02.272452+0000", 2024-08-19T14:25:09.337 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.4835e-05 2024-08-19T14:25:09.337 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:09.337 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:25:09.337 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:09.337 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.337 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.337 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24580.0:209204 2.f 2:f5b7b692:::benchmark_data_smithi045_46415_object85923:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194)", 2024-08-19T14:25:09.337 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:16:02.063760+0000", 2024-08-19T14:25:09.338 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 547.26274001599995, 2024-08-19T14:25:09.338 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.21463046399999999, 2024-08-19T14:25:09.338 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:25:09.338 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:25:09.338 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:25:09.338 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:25:09.338 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24580", 2024-08-19T14:25:09.338 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:25:09.338 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 209204 2024-08-19T14:25:09.338 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.338 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:25:09.338 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.338 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:25:09.339 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:16:02.063760+0000", 2024-08-19T14:25:09.339 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:25:09.339 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.339 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.339 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:25:09.339 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:16:02.063760+0000", 2024-08-19T14:25:09.339 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:25:09.339 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.339 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.339 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:25:09.339 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:16:02.063761+0000", 2024-08-19T14:25:09.339 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2419999999999999e-06 2024-08-19T14:25:09.339 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.339 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.340 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:25:09.340 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:16:02.063765+0000", 2024-08-19T14:25:09.340 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.0629999999999999e-06 2024-08-19T14:25:09.340 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.340 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.340 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:25:09.340 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:16:02.063775+0000", 2024-08-19T14:25:09.340 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.8239999999999995e-06 2024-08-19T14:25:09.340 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.340 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.340 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:25:09.340 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:16:02.063780+0000", 2024-08-19T14:25:09.340 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.31e-06 2024-08-19T14:25:09.340 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.341 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.341 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:25:09.341 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:16:02.063822+0000", 2024-08-19T14:25:09.341 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.2009999999999999e-05 2024-08-19T14:25:09.341 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.341 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.341 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:25:09.341 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:16:02.063940+0000", 2024-08-19T14:25:09.341 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00011828999999999999 2024-08-19T14:25:09.341 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.341 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.341 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 4,9", 2024-08-19T14:25:09.341 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:16:02.064018+0000", 2024-08-19T14:25:09.342 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.7959000000000004e-05 2024-08-19T14:25:09.342 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.342 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.342 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:09.342 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:16:02.067663+0000", 2024-08-19T14:25:09.342 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0036444989999999998 2024-08-19T14:25:09.342 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.342 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.342 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:25:09.342 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:16:02.067679+0000", 2024-08-19T14:25:09.342 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6184999999999998e-05 2024-08-19T14:25:09.342 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.342 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.343 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:09.343 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:16:02.278283+0000", 2024-08-19T14:25:09.343 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.210604333 2024-08-19T14:25:09.343 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.343 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.343 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:25:09.343 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:16:02.278327+0000", 2024-08-19T14:25:09.343 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.3619999999999999e-05 2024-08-19T14:25:09.343 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.343 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.343 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:25:09.343 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:16:02.278390+0000", 2024-08-19T14:25:09.343 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.3128999999999998e-05 2024-08-19T14:25:09.344 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:09.344 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:25:09.344 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:09.344 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.344 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.344 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25057.0:64631 3.1a 3.4f01fb1a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e246)", 2024-08-19T14:25:09.344 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:17:36.180454+0000", 2024-08-19T14:25:09.344 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 453.14604573999998, 2024-08-19T14:25:09.344 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.78083698899999998, 2024-08-19T14:25:09.344 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:25:09.344 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:25:09.344 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-08-19T14:25:09.344 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:25:09.345 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25057", 2024-08-19T14:25:09.345 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:25:09.345 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 64631 2024-08-19T14:25:09.345 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.345 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:25:09.345 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:09.347 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:25:09.347 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.180454+0000", 2024-08-19T14:25:09.347 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:25:09.347 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.347 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.347 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:25:09.347 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.180454+0000", 2024-08-19T14:25:09.347 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:25:09.347 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.347 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.347 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:25:09.347 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.180456+0000", 2024-08-19T14:25:09.348 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.5220000000000002e-06 2024-08-19T14:25:09.348 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.348 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.348 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:25:09.348 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.180483+0000", 2024-08-19T14:25:09.348 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.6417000000000001e-05 2024-08-19T14:25:09.348 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.348 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.348 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:25:09.348 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.180499+0000", 2024-08-19T14:25:09.348 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5823000000000001e-05 2024-08-19T14:25:09.348 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.348 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.348 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:25:09.349 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.180510+0000", 2024-08-19T14:25:09.349 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1737e-05 2024-08-19T14:25:09.349 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.349 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.349 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:25:09.349 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.180596+0000", 2024-08-19T14:25:09.349 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.6001000000000006e-05 2024-08-19T14:25:09.349 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.349 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.349 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for peered", 2024-08-19T14:25:09.349 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.180606+0000", 2024-08-19T14:25:09.349 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.3470000000000005e-06 2024-08-19T14:25:09.349 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.349 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.350 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:25:09.350 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.961260+0000", 2024-08-19T14:25:09.350 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.78065392099999997 2024-08-19T14:25:09.350 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.350 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.350 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:25:09.350 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.961291+0000", 2024-08-19T14:25:09.350 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.1220999999999998e-05 2024-08-19T14:25:09.350 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:09.350 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:25:09.350 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:09.350 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.350 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.350 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25057.0:64651 3.1a 3.5fb8049a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e246)", 2024-08-19T14:25:09.351 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:17:36.235642+0000", 2024-08-19T14:25:09.351 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 453.09085768300002, 2024-08-19T14:25:09.351 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.72570858100000002, 2024-08-19T14:25:09.351 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:25:09.351 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:25:09.351 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-08-19T14:25:09.351 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:25:09.351 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25057", 2024-08-19T14:25:09.351 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:25:09.351 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 64651 2024-08-19T14:25:09.351 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.351 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:25:09.351 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.351 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:25:09.352 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.235642+0000", 2024-08-19T14:25:09.352 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:25:09.352 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.352 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.352 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:25:09.352 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.235642+0000", 2024-08-19T14:25:09.352 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:25:09.352 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.352 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.352 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:25:09.352 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.235644+0000", 2024-08-19T14:25:09.352 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7829999999999999e-06 2024-08-19T14:25:09.352 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.352 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.353 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:25:09.353 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.235677+0000", 2024-08-19T14:25:09.353 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.3333000000000001e-05 2024-08-19T14:25:09.353 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.353 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.353 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:25:09.353 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.235694+0000", 2024-08-19T14:25:09.353 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6626999999999999e-05 2024-08-19T14:25:09.353 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.353 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.353 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:25:09.353 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.235701+0000", 2024-08-19T14:25:09.353 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.7000000000000008e-06 2024-08-19T14:25:09.353 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.353 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.354 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:25:09.354 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.235758+0000", 2024-08-19T14:25:09.354 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7166999999999997e-05 2024-08-19T14:25:09.354 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.354 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.354 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for peered", 2024-08-19T14:25:09.354 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.235766+0000", 2024-08-19T14:25:09.354 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4059999999999999e-06 2024-08-19T14:25:09.354 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.354 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.354 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:25:09.354 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.961337+0000", 2024-08-19T14:25:09.354 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.72557158099999997 2024-08-19T14:25:09.355 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.355 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.355 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:25:09.355 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.961350+0000", 2024-08-19T14:25:09.355 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2984e-05 2024-08-19T14:25:09.355 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:09.355 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:25:09.355 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:09.355 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.355 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.355 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25057.0:64656 3.1a 3.4e375ba (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e246)", 2024-08-19T14:25:09.355 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:17:36.257659+0000", 2024-08-19T14:25:09.355 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 453.06884021100001, 2024-08-19T14:25:09.356 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.70372575500000001, 2024-08-19T14:25:09.356 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:25:09.356 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:25:09.356 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-08-19T14:25:09.356 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:25:09.356 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25057", 2024-08-19T14:25:09.356 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:25:09.356 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 64656 2024-08-19T14:25:09.356 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.356 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:25:09.356 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.356 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:25:09.356 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.257659+0000", 2024-08-19T14:25:09.356 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:25:09.356 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.357 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.357 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:25:09.357 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.257659+0000", 2024-08-19T14:25:09.357 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:25:09.357 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.357 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.357 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:25:09.357 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.257661+0000", 2024-08-19T14:25:09.357 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.821e-06 2024-08-19T14:25:09.357 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.357 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.357 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:25:09.357 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.257681+0000", 2024-08-19T14:25:09.358 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0307999999999999e-05 2024-08-19T14:25:09.358 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.358 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.358 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:25:09.358 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.257693+0000", 2024-08-19T14:25:09.358 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1447000000000001e-05 2024-08-19T14:25:09.358 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.358 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.358 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:25:09.358 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.257699+0000", 2024-08-19T14:25:09.358 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7389999999999996e-06 2024-08-19T14:25:09.358 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.358 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.359 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:25:09.359 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.257751+0000", 2024-08-19T14:25:09.359 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2281999999999999e-05 2024-08-19T14:25:09.359 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.359 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.359 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for peered", 2024-08-19T14:25:09.359 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.257759+0000", 2024-08-19T14:25:09.359 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.2390000000000002e-06 2024-08-19T14:25:09.359 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.359 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.359 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:25:09.359 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.961378+0000", 2024-08-19T14:25:09.359 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.70361908699999998 2024-08-19T14:25:09.359 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.360 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.360 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:25:09.360 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:36.961385+0000", 2024-08-19T14:25:09.360 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.832e-06 2024-08-19T14:25:09.360 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:09.360 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:25:09.360 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:09.360 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.360 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.360 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25057.0:130084 3.9 3:96bca659:::benchmark_data_smithi045_59973_object130083:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e281)", 2024-08-19T14:25:09.360 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:18:31.614605+0000", 2024-08-19T14:25:09.360 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 397.71189488599998, 2024-08-19T14:25:09.360 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.206588472, 2024-08-19T14:25:09.360 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:25:09.361 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:25:09.361 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:25:09.361 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:25:09.361 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25057", 2024-08-19T14:25:09.361 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:25:09.361 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 130084 2024-08-19T14:25:09.361 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.361 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:25:09.361 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.361 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:25:09.361 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:18:31.614605+0000", 2024-08-19T14:25:09.361 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:25:09.361 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.361 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.362 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:25:09.362 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:18:31.614605+0000", 2024-08-19T14:25:09.362 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:25:09.362 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.362 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.362 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:25:09.362 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:18:31.614606+0000", 2024-08-19T14:25:09.362 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3960000000000001e-06 2024-08-19T14:25:09.362 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.362 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.362 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:25:09.362 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:18:31.614627+0000", 2024-08-19T14:25:09.362 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1313000000000001e-05 2024-08-19T14:25:09.363 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.363 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.363 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:25:09.363 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:18:31.614638+0000", 2024-08-19T14:25:09.363 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0686000000000001e-05 2024-08-19T14:25:09.363 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.363 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.363 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:25:09.363 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:18:31.614643+0000", 2024-08-19T14:25:09.363 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0690000000000001e-06 2024-08-19T14:25:09.363 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.363 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.363 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:25:09.363 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:18:31.614719+0000", 2024-08-19T14:25:09.364 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.6252000000000001e-05 2024-08-19T14:25:09.364 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.364 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.364 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:25:09.364 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:18:31.614843+0000", 2024-08-19T14:25:09.364 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012340699999999999 2024-08-19T14:25:09.364 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.364 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.364 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 7,11", 2024-08-19T14:25:09.364 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:18:31.614982+0000", 2024-08-19T14:25:09.364 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013878600000000001 2024-08-19T14:25:09.364 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.364 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.365 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:25:09.365 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:18:31.617402+0000", 2024-08-19T14:25:09.365 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00242034 2024-08-19T14:25:09.365 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.365 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.365 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:09.365 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:18:31.617596+0000", 2024-08-19T14:25:09.365 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00019408500000000001 2024-08-19T14:25:09.365 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.365 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.365 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:09.365 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:18:31.820984+0000", 2024-08-19T14:25:09.365 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20338821900000001 2024-08-19T14:25:09.366 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.366 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.366 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:25:09.366 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:18:31.821156+0000", 2024-08-19T14:25:09.366 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00017137899999999999 2024-08-19T14:25:09.366 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.366 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.366 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:25:09.366 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:18:31.821193+0000", 2024-08-19T14:25:09.366 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.7540000000000003e-05 2024-08-19T14:25:09.366 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:09.366 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:25:09.366 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:09.366 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.367 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.367 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25057.0:163335 3.1d 3:bec78ced:::benchmark_data_smithi045_59973_object32156:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e288)", 2024-08-19T14:25:09.367 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:18:47.673097+0000", 2024-08-19T14:25:09.367 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 381.653402169, 2024-08-19T14:25:09.367 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.207015213, 2024-08-19T14:25:09.367 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:25:09.367 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:25:09.367 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:25:09.367 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:25:09.367 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25057", 2024-08-19T14:25:09.367 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:25:09.367 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 163335 2024-08-19T14:25:09.367 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.367 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:25:09.368 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.368 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:25:09.368 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:18:47.673097+0000", 2024-08-19T14:25:09.368 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:25:09.368 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.368 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.368 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:25:09.368 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:18:47.673097+0000", 2024-08-19T14:25:09.368 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:25:09.368 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.368 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.368 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:25:09.368 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:18:47.673098+0000", 2024-08-19T14:25:09.368 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.8299999999999995e-07 2024-08-19T14:25:09.369 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.369 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.369 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:25:09.369 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:18:47.673103+0000", 2024-08-19T14:25:09.369 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.2760000000000002e-06 2024-08-19T14:25:09.369 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.369 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.369 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:25:09.369 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:18:47.673115+0000", 2024-08-19T14:25:09.369 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2676e-05 2024-08-19T14:25:09.369 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.369 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.369 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:25:09.369 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:18:47.673122+0000", 2024-08-19T14:25:09.369 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.9990000000000002e-06 2024-08-19T14:25:09.370 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.370 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.370 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:25:09.370 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:18:47.674044+0000", 2024-08-19T14:25:09.370 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00092164999999999999 2024-08-19T14:25:09.370 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.370 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.370 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:25:09.370 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:18:47.674163+0000", 2024-08-19T14:25:09.370 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000119577 2024-08-19T14:25:09.370 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.370 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.370 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 4,5", 2024-08-19T14:25:09.371 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:18:47.674268+0000", 2024-08-19T14:25:09.371 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00010464 2024-08-19T14:25:09.371 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.371 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.371 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:25:09.371 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:18:47.678891+0000", 2024-08-19T14:25:09.371 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0046226419999999997 2024-08-19T14:25:09.371 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.371 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.371 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:09.371 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:18:47.735032+0000", 2024-08-19T14:25:09.371 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.056140931999999998 2024-08-19T14:25:09.371 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.371 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.372 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:09.372 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:18:47.880041+0000", 2024-08-19T14:25:09.372 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.14500938999999999 2024-08-19T14:25:09.372 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.372 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.372 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:25:09.372 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:18:47.880073+0000", 2024-08-19T14:25:09.372 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.1695000000000001e-05 2024-08-19T14:25:09.372 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.372 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.372 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:25:09.372 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:18:47.880113+0000", 2024-08-19T14:25:09.372 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.9852999999999999e-05 2024-08-19T14:25:09.372 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:09.373 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:25:09.373 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:09.373 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.373 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.373 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25057.0:212414 3.1a 3:58a5364f:::benchmark_data_smithi045_59973_object81235:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e305)", 2024-08-19T14:25:09.373 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:19:29.907348+0000", 2024-08-19T14:25:09.373 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 339.41915116199999, 2024-08-19T14:25:09.373 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.19817712800000001, 2024-08-19T14:25:09.373 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:25:09.373 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:25:09.373 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:25:09.373 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:25:09.373 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25057", 2024-08-19T14:25:09.373 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:25:09.374 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 212414 2024-08-19T14:25:09.374 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.374 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:25:09.374 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.374 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:25:09.374 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:29.907348+0000", 2024-08-19T14:25:09.374 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:25:09.374 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.374 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.374 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:25:09.374 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:29.907348+0000", 2024-08-19T14:25:09.374 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:25:09.374 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.374 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.375 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:25:09.375 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:29.907349+0000", 2024-08-19T14:25:09.375 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.1299999999999998e-07 2024-08-19T14:25:09.375 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.375 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.375 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:25:09.375 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:29.907354+0000", 2024-08-19T14:25:09.375 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.78e-06 2024-08-19T14:25:09.375 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.375 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.375 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:25:09.375 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:29.907366+0000", 2024-08-19T14:25:09.375 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1430000000000001e-05 2024-08-19T14:25:09.376 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.376 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.376 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:25:09.376 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:29.907371+0000", 2024-08-19T14:25:09.376 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.3000000000000001e-06 2024-08-19T14:25:09.376 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.376 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.376 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:25:09.376 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:29.907460+0000", 2024-08-19T14:25:09.376 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.9044999999999999e-05 2024-08-19T14:25:09.376 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.376 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.376 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:25:09.376 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:29.907531+0000", 2024-08-19T14:25:09.376 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.1620999999999997e-05 2024-08-19T14:25:09.377 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.377 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.377 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,9", 2024-08-19T14:25:09.377 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:29.907593+0000", 2024-08-19T14:25:09.377 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.1859999999999994e-05 2024-08-19T14:25:09.377 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.377 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.377 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:09.377 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:29.911966+0000", 2024-08-19T14:25:09.377 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0043727549999999999 2024-08-19T14:25:09.377 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.377 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.377 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:25:09.377 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:29.912043+0000", 2024-08-19T14:25:09.377 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.6963000000000002e-05 2024-08-19T14:25:09.378 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.378 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.378 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:09.378 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:30.105414+0000", 2024-08-19T14:25:09.378 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.193371022 2024-08-19T14:25:09.378 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.378 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.378 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:25:09.378 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:30.105474+0000", 2024-08-19T14:25:09.378 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.0081999999999998e-05 2024-08-19T14:25:09.378 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.378 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.378 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:25:09.378 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:30.105526+0000", 2024-08-19T14:25:09.379 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1357000000000003e-05 2024-08-19T14:25:09.379 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:09.379 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:25:09.379 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:09.379 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.379 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.379 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25057.0:212429 3.1a 3:5cc1fd18:::benchmark_data_smithi045_59973_object81250:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e305)", 2024-08-19T14:25:09.379 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:19:29.920895+0000", 2024-08-19T14:25:09.379 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 339.40560476600001, 2024-08-19T14:25:09.379 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.201169661, 2024-08-19T14:25:09.379 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:25:09.379 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:25:09.379 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:25:09.379 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:25:09.380 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25057", 2024-08-19T14:25:09.380 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:25:09.380 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 212429 2024-08-19T14:25:09.380 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.380 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:25:09.380 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.380 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:25:09.380 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:29.920895+0000", 2024-08-19T14:25:09.380 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:25:09.380 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.380 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.380 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:25:09.380 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:29.920895+0000", 2024-08-19T14:25:09.380 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:25:09.381 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.381 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.381 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:25:09.381 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:29.920896+0000", 2024-08-19T14:25:09.381 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.561e-06 2024-08-19T14:25:09.381 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.381 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.381 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:25:09.381 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:29.920903+0000", 2024-08-19T14:25:09.381 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.7329999999999997e-06 2024-08-19T14:25:09.381 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.381 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.381 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:25:09.381 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:29.920916+0000", 2024-08-19T14:25:09.382 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3186999999999999e-05 2024-08-19T14:25:09.382 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.382 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.382 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:25:09.382 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:29.920924+0000", 2024-08-19T14:25:09.382 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.0330000000000008e-06 2024-08-19T14:25:09.382 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.382 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.382 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:25:09.382 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:29.920998+0000", 2024-08-19T14:25:09.382 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.3480999999999997e-05 2024-08-19T14:25:09.382 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.382 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.382 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:25:09.383 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:29.921084+0000", 2024-08-19T14:25:09.383 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.6697e-05 2024-08-19T14:25:09.383 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.383 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.383 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,9", 2024-08-19T14:25:09.383 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:29.921167+0000", 2024-08-19T14:25:09.383 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.2146999999999998e-05 2024-08-19T14:25:09.383 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.383 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.383 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:25:09.383 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:29.928838+0000", 2024-08-19T14:25:09.383 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0076718439999999997 2024-08-19T14:25:09.383 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.383 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.384 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:09.384 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:29.929101+0000", 2024-08-19T14:25:09.384 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00026302200000000002 2024-08-19T14:25:09.384 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.384 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.384 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:09.384 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:30.122006+0000", 2024-08-19T14:25:09.384 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.19290453199999999 2024-08-19T14:25:09.384 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.384 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.384 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:25:09.384 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:30.122039+0000", 2024-08-19T14:25:09.384 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.2509000000000003e-05 2024-08-19T14:25:09.384 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.384 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.385 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:25:09.385 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:30.122064+0000", 2024-08-19T14:25:09.385 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.5914999999999998e-05 2024-08-19T14:25:09.385 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:09.385 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:25:09.385 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:09.385 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.385 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.385 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25057.0:216413 3.1a 3:5fc494fd:::benchmark_data_smithi045_59973_object85234:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e307)", 2024-08-19T14:25:09.385 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:19:33.987755+0000", 2024-08-19T14:25:09.385 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 335.33874423899999, 2024-08-19T14:25:09.385 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20454685, 2024-08-19T14:25:09.385 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:25:09.385 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:25:09.386 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:25:09.386 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:25:09.386 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25057", 2024-08-19T14:25:09.386 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:25:09.386 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 216413 2024-08-19T14:25:09.386 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.386 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:25:09.386 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.386 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:25:09.386 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:33.987755+0000", 2024-08-19T14:25:09.386 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:25:09.386 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.386 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.386 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:25:09.387 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:33.987755+0000", 2024-08-19T14:25:09.387 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:25:09.387 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.387 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.387 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:25:09.387 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:33.987756+0000", 2024-08-19T14:25:09.387 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1200000000000003e-07 2024-08-19T14:25:09.387 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.387 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.387 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:25:09.387 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:33.987759+0000", 2024-08-19T14:25:09.387 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.213e-06 2024-08-19T14:25:09.387 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.387 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.388 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:25:09.388 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:33.987768+0000", 2024-08-19T14:25:09.388 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.5089999999999993e-06 2024-08-19T14:25:09.388 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.388 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.388 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:25:09.388 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:09.389 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:25:09.389 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:33.987774+0000", 2024-08-19T14:25:09.389 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.8159999999999996e-06 2024-08-19T14:25:09.389 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.390 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.390 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:25:09.390 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:33.988656+0000", 2024-08-19T14:25:09.390 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000881747 2024-08-19T14:25:09.390 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.390 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.390 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:25:09.390 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:33.988760+0000", 2024-08-19T14:25:09.390 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000104345 2024-08-19T14:25:09.390 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.390 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.390 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,9", 2024-08-19T14:25:09.390 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:33.988862+0000", 2024-08-19T14:25:09.391 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00010182400000000001 2024-08-19T14:25:09.391 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.391 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.391 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:25:09.391 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:33.992105+0000", 2024-08-19T14:25:09.391 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0032423069999999998 2024-08-19T14:25:09.391 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.391 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.391 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:09.391 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:33.998218+0000", 2024-08-19T14:25:09.391 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0061136009999999998 2024-08-19T14:25:09.391 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.391 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.392 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:09.392 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:34.191451+0000", 2024-08-19T14:25:09.392 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.193233241 2024-08-19T14:25:09.392 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.392 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.392 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:25:09.392 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:34.191530+0000", 2024-08-19T14:25:09.392 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.8672999999999997e-05 2024-08-19T14:25:09.392 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.392 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.392 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:25:09.392 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:34.192302+0000", 2024-08-19T14:25:09.392 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00077206200000000001 2024-08-19T14:25:09.392 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:09.393 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:25:09.393 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:09.393 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.393 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.393 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25057.0:216426 3.1a 3:582bcfa1:::benchmark_data_smithi045_59973_object85247:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e307)", 2024-08-19T14:25:09.393 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:19:33.994645+0000", 2024-08-19T14:25:09.393 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 335.33185408100002, 2024-08-19T14:25:09.393 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.19960349199999999, 2024-08-19T14:25:09.393 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:25:09.393 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:25:09.393 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:25:09.393 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:25:09.393 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25057", 2024-08-19T14:25:09.393 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:25:09.394 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 216426 2024-08-19T14:25:09.394 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.394 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:25:09.394 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.394 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:25:09.394 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:33.994645+0000", 2024-08-19T14:25:09.394 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:25:09.394 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.394 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.394 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:25:09.394 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:33.994645+0000", 2024-08-19T14:25:09.394 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:25:09.394 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.394 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.395 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:25:09.395 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:33.994647+0000", 2024-08-19T14:25:09.395 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.221e-06 2024-08-19T14:25:09.395 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.395 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.395 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:25:09.395 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:33.994652+0000", 2024-08-19T14:25:09.395 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.865e-06 2024-08-19T14:25:09.395 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.395 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.395 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:25:09.395 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:33.994667+0000", 2024-08-19T14:25:09.395 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5651999999999999e-05 2024-08-19T14:25:09.395 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.396 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.396 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:25:09.396 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:33.994674+0000", 2024-08-19T14:25:09.396 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.9999999999999999e-06 2024-08-19T14:25:09.396 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.396 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.396 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:25:09.396 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:33.994730+0000", 2024-08-19T14:25:09.396 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.5401e-05 2024-08-19T14:25:09.396 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.396 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.396 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:25:09.396 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:33.994837+0000", 2024-08-19T14:25:09.396 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00010730000000000001 2024-08-19T14:25:09.397 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.397 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.397 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,9", 2024-08-19T14:25:09.397 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:33.994947+0000", 2024-08-19T14:25:09.397 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000109887 2024-08-19T14:25:09.397 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.397 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.397 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:25:09.397 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:33.998233+0000", 2024-08-19T14:25:09.397 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.003285708 2024-08-19T14:25:09.397 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.397 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.397 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:09.398 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:33.999576+0000", 2024-08-19T14:25:09.398 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001343386 2024-08-19T14:25:09.398 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.398 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.398 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:09.398 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:34.194092+0000", 2024-08-19T14:25:09.398 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.194516153 2024-08-19T14:25:09.398 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.398 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.398 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:25:09.398 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:34.194199+0000", 2024-08-19T14:25:09.398 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000106914 2024-08-19T14:25:09.398 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.398 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.398 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:25:09.399 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:19:34.194249+0000", 2024-08-19T14:25:09.399 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0005e-05 2024-08-19T14:25:09.399 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:09.399 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:25:09.399 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:09.399 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.399 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.399 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.25813.0:21367 5.2 e426/419 5:474db56a:::benchmark_data_smithi045_85309_object21366:head v 426'1313, mlcod=426'1309)", 2024-08-19T14:25:09.399 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:24:54.706590+0000", 2024-08-19T14:25:09.399 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 14.619909995, 2024-08-19T14:25:09.399 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.041282434999999999, 2024-08-19T14:25:09.399 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:25:09.399 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:25:09.400 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:25:09.400 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:25:09.400 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.400 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:25:09.400 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:24:54.706590+0000", 2024-08-19T14:25:09.400 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:25:09.400 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.400 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.400 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:25:09.400 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:24:54.706590+0000", 2024-08-19T14:25:09.401 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:25:09.401 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.401 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.401 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:25:09.401 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:24:54.706590+0000", 2024-08-19T14:25:09.401 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.5499999999999999e-07 2024-08-19T14:25:09.401 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.401 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.401 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:25:09.401 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:24:54.745199+0000", 2024-08-19T14:25:09.401 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.038608635000000002 2024-08-19T14:25:09.401 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.401 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.401 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:25:09.402 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:24:54.745242+0000", 2024-08-19T14:25:09.402 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.3300000000000002e-05 2024-08-19T14:25:09.402 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.402 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.402 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:25:09.402 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:24:54.745265+0000", 2024-08-19T14:25:09.402 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.3589000000000001e-05 2024-08-19T14:25:09.402 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.402 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.402 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:25:09.402 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:24:54.745360+0000", 2024-08-19T14:25:09.402 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.4569999999999997e-05 2024-08-19T14:25:09.402 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.402 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.403 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:25:09.403 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:24:54.745383+0000", 2024-08-19T14:25:09.403 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2827e-05 2024-08-19T14:25:09.403 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.403 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.403 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:25:09.403 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:24:54.747842+0000", 2024-08-19T14:25:09.403 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0024593039999999998 2024-08-19T14:25:09.403 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.403 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.403 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:25:09.403 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:24:54.747872+0000", 2024-08-19T14:25:09.403 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.9855000000000001e-05 2024-08-19T14:25:09.403 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:09.404 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:25:09.404 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:09.404 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.404 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.404 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25813.0:25054 5.a 5:57c5c564:::benchmark_data_smithi045_85309_object25053:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e427)", 2024-08-19T14:25:09.404 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:24:56.169146+0000", 2024-08-19T14:25:09.404 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 13.157353498000001, 2024-08-19T14:25:09.404 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.018414824999999999, 2024-08-19T14:25:09.404 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:25:09.404 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:25:09.404 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:25:09.404 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:25:09.404 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25813", 2024-08-19T14:25:09.405 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:25:09.405 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 25054 2024-08-19T14:25:09.405 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.405 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:25:09.405 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.405 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:25:09.405 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:24:56.169146+0000", 2024-08-19T14:25:09.405 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:25:09.405 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.405 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.405 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:25:09.405 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:24:56.169146+0000", 2024-08-19T14:25:09.405 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:25:09.405 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.406 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.406 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:25:09.406 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:24:56.169146+0000", 2024-08-19T14:25:09.406 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.7e-07 2024-08-19T14:25:09.406 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.406 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.406 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:25:09.406 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:24:56.169166+0000", 2024-08-19T14:25:09.406 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9344e-05 2024-08-19T14:25:09.406 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.406 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.406 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:25:09.406 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:24:56.169173+0000", 2024-08-19T14:25:09.406 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.5320000000000001e-06 2024-08-19T14:25:09.407 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.407 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.407 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:25:09.407 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:24:56.169178+0000", 2024-08-19T14:25:09.407 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.8189999999999999e-06 2024-08-19T14:25:09.407 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.407 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.407 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:25:09.407 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:24:56.183665+0000", 2024-08-19T14:25:09.407 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.014486483 2024-08-19T14:25:09.407 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.407 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.407 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:25:09.407 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:24:56.183814+0000", 2024-08-19T14:25:09.407 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000149481 2024-08-19T14:25:09.408 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.408 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.408 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 4,11", 2024-08-19T14:25:09.408 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:24:56.183956+0000", 2024-08-19T14:25:09.408 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000141982 2024-08-19T14:25:09.408 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.408 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.408 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:25:09.408 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:24:56.186042+0000", 2024-08-19T14:25:09.408 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.002085852 2024-08-19T14:25:09.408 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.408 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.408 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:09.408 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:24:56.187358+0000", 2024-08-19T14:25:09.409 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001315798 2024-08-19T14:25:09.409 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.409 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.409 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:09.409 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:24:56.187461+0000", 2024-08-19T14:25:09.409 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000103427 2024-08-19T14:25:09.409 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.409 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.409 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:25:09.409 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:24:56.187520+0000", 2024-08-19T14:25:09.409 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.8480000000000003e-05 2024-08-19T14:25:09.409 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.409 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.409 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:25:09.409 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:24:56.187561+0000", 2024-08-19T14:25:09.410 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.1257000000000002e-05 2024-08-19T14:25:09.410 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:09.410 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:25:09.410 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:09.410 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.410 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.410 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25813.0:27775 5.a 5:5985053d:::benchmark_data_smithi045_85309_object27774:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e428)", 2024-08-19T14:25:09.410 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:24:57.050754+0000", 2024-08-19T14:25:09.410 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 12.275745663, 2024-08-19T14:25:09.410 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.023383247999999999, 2024-08-19T14:25:09.410 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:25:09.410 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:25:09.410 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:25:09.410 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:25:09.411 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25813", 2024-08-19T14:25:09.411 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:25:09.411 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 27775 2024-08-19T14:25:09.411 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.411 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:25:09.411 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.411 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:25:09.411 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:24:57.050754+0000", 2024-08-19T14:25:09.411 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:25:09.411 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.411 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.411 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:25:09.411 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:24:57.050754+0000", 2024-08-19T14:25:09.411 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:25:09.412 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.412 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.413 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:25:09.413 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:24:57.050755+0000", 2024-08-19T14:25:09.413 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.3500000000000005e-07 2024-08-19T14:25:09.413 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.413 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.413 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:25:09.413 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:24:57.050773+0000", 2024-08-19T14:25:09.414 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8519999999999999e-05 2024-08-19T14:25:09.414 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.414 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.414 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:25:09.414 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:24:57.050786+0000", 2024-08-19T14:25:09.414 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2313999999999999e-05 2024-08-19T14:25:09.414 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.414 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.414 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:25:09.414 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:24:57.050794+0000", 2024-08-19T14:25:09.414 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.0299999999999994e-06 2024-08-19T14:25:09.414 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.414 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.414 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:25:09.414 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:24:57.052213+0000", 2024-08-19T14:25:09.415 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0014194260000000001 2024-08-19T14:25:09.415 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.415 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.415 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:25:09.415 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:24:57.052341+0000", 2024-08-19T14:25:09.415 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012757799999999999 2024-08-19T14:25:09.415 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.415 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.415 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 4,11", 2024-08-19T14:25:09.415 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:24:57.052469+0000", 2024-08-19T14:25:09.415 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000128317 2024-08-19T14:25:09.415 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.415 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.415 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:25:09.416 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:24:57.054235+0000", 2024-08-19T14:25:09.416 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00176621 2024-08-19T14:25:09.416 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.416 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.416 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:09.416 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:24:57.056766+0000", 2024-08-19T14:25:09.416 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0025309870000000002 2024-08-19T14:25:09.416 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.416 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.416 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:09.416 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:24:57.074074+0000", 2024-08-19T14:25:09.416 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.01730777 2024-08-19T14:25:09.416 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.416 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.417 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:25:09.417 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:24:57.074110+0000", 2024-08-19T14:25:09.417 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.5747000000000002e-05 2024-08-19T14:25:09.417 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.417 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.417 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:25:09.417 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:24:57.074137+0000", 2024-08-19T14:25:09.417 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.7413999999999999e-05 2024-08-19T14:25:09.417 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:09.417 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:25:09.417 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:09.417 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.417 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.417 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25813.0:27783 5.a 5:59e8ddec:::benchmark_data_smithi045_85309_object27782:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e428)", 2024-08-19T14:25:09.417 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:24:57.053671+0000", 2024-08-19T14:25:09.418 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 12.272828399, 2024-08-19T14:25:09.418 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.020628061999999999, 2024-08-19T14:25:09.418 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:25:09.418 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:25:09.418 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:25:09.418 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:25:09.418 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25813", 2024-08-19T14:25:09.418 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:25:09.418 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 27783 2024-08-19T14:25:09.418 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.418 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:25:09.418 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.418 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:25:09.419 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:24:57.053671+0000", 2024-08-19T14:25:09.419 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:25:09.419 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.419 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.419 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:25:09.419 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:24:57.053671+0000", 2024-08-19T14:25:09.419 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:25:09.419 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.419 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.419 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:25:09.419 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:24:57.053672+0000", 2024-08-19T14:25:09.419 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3340000000000001e-06 2024-08-19T14:25:09.419 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.419 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.420 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:25:09.420 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:24:57.053696+0000", 2024-08-19T14:25:09.420 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.3289999999999999e-05 2024-08-19T14:25:09.420 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.420 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.420 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:25:09.420 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:24:57.053706+0000", 2024-08-19T14:25:09.420 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.064e-05 2024-08-19T14:25:09.420 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.420 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.421 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:25:09.421 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:24:57.053714+0000", 2024-08-19T14:25:09.421 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.379e-06 2024-08-19T14:25:09.421 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.421 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.421 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:25:09.421 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:24:57.054311+0000", 2024-08-19T14:25:09.421 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00059688000000000005 2024-08-19T14:25:09.421 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.421 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.421 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:25:09.421 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:24:57.054437+0000", 2024-08-19T14:25:09.421 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000126787 2024-08-19T14:25:09.422 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.422 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.422 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 4,11", 2024-08-19T14:25:09.422 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:24:57.054577+0000", 2024-08-19T14:25:09.422 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013915999999999999 2024-08-19T14:25:09.422 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.422 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.422 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:25:09.422 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:24:57.056313+0000", 2024-08-19T14:25:09.422 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001736788 2024-08-19T14:25:09.422 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.423 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.423 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:09.423 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:24:57.056811+0000", 2024-08-19T14:25:09.423 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00049740400000000001 2024-08-19T14:25:09.423 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.423 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.423 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:09.423 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:24:57.074209+0000", 2024-08-19T14:25:09.423 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.017398252999999999 2024-08-19T14:25:09.423 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.423 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.424 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:25:09.424 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:24:57.074261+0000", 2024-08-19T14:25:09.424 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1646e-05 2024-08-19T14:25:09.424 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.424 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.424 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:25:09.424 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:24:57.074299+0000", 2024-08-19T14:25:09.424 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.8501000000000003e-05 2024-08-19T14:25:09.424 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:09.424 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:25:09.424 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:09.424 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.425 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.425 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25813.0:27819 5.a 5:5bb5a4a1:::benchmark_data_smithi045_85309_object27818:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e428)", 2024-08-19T14:25:09.425 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:24:57.092958+0000", 2024-08-19T14:25:09.425 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 12.233541281999999, 2024-08-19T14:25:09.425 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.020719834999999999, 2024-08-19T14:25:09.425 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:25:09.425 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:25:09.425 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:25:09.425 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:25:09.425 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25813", 2024-08-19T14:25:09.426 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:25:09.426 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 27819 2024-08-19T14:25:09.426 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.426 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:25:09.426 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.426 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:25:09.426 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:24:57.092958+0000", 2024-08-19T14:25:09.426 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:25:09.426 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.426 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.426 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:25:09.426 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:24:57.092958+0000", 2024-08-19T14:25:09.427 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:25:09.427 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.427 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.427 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:25:09.427 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:24:57.092960+0000", 2024-08-19T14:25:09.427 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4449999999999999e-06 2024-08-19T14:25:09.427 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.427 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.427 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:25:09.427 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:24:57.092984+0000", 2024-08-19T14:25:09.427 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.4561000000000001e-05 2024-08-19T14:25:09.427 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.428 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.428 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:25:09.428 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:24:57.092995+0000", 2024-08-19T14:25:09.428 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0603e-05 2024-08-19T14:25:09.428 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.428 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.428 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:25:09.428 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:24:57.093001+0000", 2024-08-19T14:25:09.428 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.3890000000000002e-06 2024-08-19T14:25:09.428 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.428 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.429 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:25:09.429 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:24:57.093054+0000", 2024-08-19T14:25:09.429 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.3069999999999998e-05 2024-08-19T14:25:09.429 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.429 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.429 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:25:09.429 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:24:57.093193+0000", 2024-08-19T14:25:09.429 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013843500000000001 2024-08-19T14:25:09.429 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.429 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.430 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 4,11", 2024-08-19T14:25:09.430 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:24:57.093322+0000", 2024-08-19T14:25:09.430 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012927699999999999 2024-08-19T14:25:09.430 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.430 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.430 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:25:09.430 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:24:57.094576+0000", 2024-08-19T14:25:09.430 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001254156 2024-08-19T14:25:09.430 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.430 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.430 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:09.431 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:24:57.094956+0000", 2024-08-19T14:25:09.431 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00038025500000000002 2024-08-19T14:25:09.431 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.431 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.431 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:09.431 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:24:57.113583+0000", 2024-08-19T14:25:09.431 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.018626183000000001 2024-08-19T14:25:09.431 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.431 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.431 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:25:09.432 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:24:57.113634+0000", 2024-08-19T14:25:09.432 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1539000000000002e-05 2024-08-19T14:25:09.432 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.432 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.432 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:25:09.432 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:24:57.113678+0000", 2024-08-19T14:25:09.432 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.3921999999999997e-05 2024-08-19T14:25:09.432 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:09.432 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:25:09.432 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:09.432 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.432 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.433 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25813.0:27827 5.a 5:576a7d40:::benchmark_data_smithi045_85309_object27826:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e428)", 2024-08-19T14:25:09.433 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:24:57.095208+0000", 2024-08-19T14:25:09.433 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 12.231291552, 2024-08-19T14:25:09.433 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.018623132000000001, 2024-08-19T14:25:09.433 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:25:09.433 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:25:09.433 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:25:09.433 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:25:09.433 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25813", 2024-08-19T14:25:09.433 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:25:09.433 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 27827 2024-08-19T14:25:09.433 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.433 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:25:09.433 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.433 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:25:09.433 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:24:57.095208+0000", 2024-08-19T14:25:09.434 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:25:09.434 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.434 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.434 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:25:09.434 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:24:57.095208+0000", 2024-08-19T14:25:09.434 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:25:09.434 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.434 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.434 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:25:09.435 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:24:57.095209+0000", 2024-08-19T14:25:09.435 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2309999999999999e-06 2024-08-19T14:25:09.435 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.435 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.435 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:25:09.436 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:24:57.095230+0000", 2024-08-19T14:25:09.436 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0682999999999998e-05 2024-08-19T14:25:09.436 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.436 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.436 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:25:09.436 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:24:57.095242+0000", 2024-08-19T14:25:09.436 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1916000000000001e-05 2024-08-19T14:25:09.436 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.436 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.436 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:25:09.436 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:24:57.095250+0000", 2024-08-19T14:25:09.436 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.8800000000000008e-06 2024-08-19T14:25:09.437 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.437 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.437 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:25:09.437 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:24:57.095316+0000", 2024-08-19T14:25:09.437 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.5993000000000003e-05 2024-08-19T14:25:09.437 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.437 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.437 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:25:09.437 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:24:57.095444+0000", 2024-08-19T14:25:09.437 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012868299999999999 2024-08-19T14:25:09.437 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.437 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.437 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 4,11", 2024-08-19T14:25:09.437 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:24:57.095588+0000", 2024-08-19T14:25:09.437 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014329800000000001 2024-08-19T14:25:09.438 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.438 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.438 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:25:09.438 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:24:57.097650+0000", 2024-08-19T14:25:09.438 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0020619969999999999 2024-08-19T14:25:09.438 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.438 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.438 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:09.438 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:24:57.097699+0000", 2024-08-19T14:25:09.438 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.9048999999999997e-05 2024-08-19T14:25:09.438 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.438 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.438 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:09.438 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:24:57.113733+0000", 2024-08-19T14:25:09.439 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.016034053 2024-08-19T14:25:09.439 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.439 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.439 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:25:09.439 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:24:57.113791+0000", 2024-08-19T14:25:09.439 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.8060000000000003e-05 2024-08-19T14:25:09.439 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.439 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.439 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:25:09.439 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:24:57.113831+0000", 2024-08-19T14:25:09.439 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.0289e-05 2024-08-19T14:25:09.439 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:09.439 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:25:09.440 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:09.440 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.440 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.440 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25813.0:27828 5.a 5:55c86329:::benchmark_data_smithi045_85309_object27827:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e428)", 2024-08-19T14:25:09.440 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:24:57.096409+0000", 2024-08-19T14:25:09.440 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 12.230090943, 2024-08-19T14:25:09.440 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.01771493, 2024-08-19T14:25:09.440 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:25:09.440 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:25:09.440 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:25:09.440 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:25:09.440 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25813", 2024-08-19T14:25:09.440 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:25:09.440 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 27828 2024-08-19T14:25:09.441 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.441 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:25:09.441 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.441 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:25:09.441 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:24:57.096409+0000", 2024-08-19T14:25:09.441 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:25:09.441 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.441 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.441 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:25:09.441 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:24:57.096409+0000", 2024-08-19T14:25:09.441 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:25:09.441 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.442 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.442 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:25:09.442 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:24:57.096409+0000", 2024-08-19T14:25:09.442 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.8999999999999996e-07 2024-08-19T14:25:09.442 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.442 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.442 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:25:09.442 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:24:57.096429+0000", 2024-08-19T14:25:09.442 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9704e-05 2024-08-19T14:25:09.442 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.442 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.442 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:25:09.443 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:24:57.096444+0000", 2024-08-19T14:25:09.443 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4508000000000001e-05 2024-08-19T14:25:09.443 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.443 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.443 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:25:09.443 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:24:57.096451+0000", 2024-08-19T14:25:09.443 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.6510000000000001e-06 2024-08-19T14:25:09.443 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.443 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.443 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:25:09.443 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:24:57.096586+0000", 2024-08-19T14:25:09.443 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000135167 2024-08-19T14:25:09.443 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.444 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.444 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:25:09.444 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:24:57.096706+0000", 2024-08-19T14:25:09.444 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000119177 2024-08-19T14:25:09.444 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.444 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.444 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 4,11", 2024-08-19T14:25:09.444 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:24:57.096824+0000", 2024-08-19T14:25:09.444 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00011882700000000001 2024-08-19T14:25:09.444 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.444 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.444 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:25:09.444 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:24:57.097775+0000", 2024-08-19T14:25:09.444 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00095043199999999997 2024-08-19T14:25:09.445 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.445 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.445 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:09.445 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:24:57.097930+0000", 2024-08-19T14:25:09.445 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015504399999999999 2024-08-19T14:25:09.445 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.445 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.445 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:09.445 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:24:57.113941+0000", 2024-08-19T14:25:09.445 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.016010739999999999 2024-08-19T14:25:09.445 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.445 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.445 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:25:09.445 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:24:57.114032+0000", 2024-08-19T14:25:09.446 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.1811000000000006e-05 2024-08-19T14:25:09.446 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.446 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.446 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:25:09.446 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:24:57.114124+0000", 2024-08-19T14:25:09.446 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.1179000000000002e-05 2024-08-19T14:25:09.446 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:09.446 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:25:09.446 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:09.446 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.446 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.446 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25813.0:53789 5.a 5:51e0c5dd:::benchmark_data_smithi045_85309_object53788:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e431)", 2024-08-19T14:25:09.447 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:25:08.677076+0000", 2024-08-19T14:25:09.447 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.64942306500000002, 2024-08-19T14:25:09.447 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.019162057999999999, 2024-08-19T14:25:09.447 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:25:09.447 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:25:09.447 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:25:09.447 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:25:09.447 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25813", 2024-08-19T14:25:09.447 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:25:09.447 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 53789 2024-08-19T14:25:09.447 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.447 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:25:09.447 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.447 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:25:09.447 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:08.677076+0000", 2024-08-19T14:25:09.448 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:25:09.448 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.448 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.448 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:25:09.448 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:08.677076+0000", 2024-08-19T14:25:09.448 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:25:09.448 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.448 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.448 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:25:09.448 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:08.677077+0000", 2024-08-19T14:25:09.448 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1799999999999995e-07 2024-08-19T14:25:09.448 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.448 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.448 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:25:09.449 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:08.677142+0000", 2024-08-19T14:25:09.449 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.4516000000000004e-05 2024-08-19T14:25:09.449 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.449 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.449 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:25:09.449 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:08.677151+0000", 2024-08-19T14:25:09.449 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.7589999999999992e-06 2024-08-19T14:25:09.449 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.449 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.449 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:25:09.449 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:08.677158+0000", 2024-08-19T14:25:09.449 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.7460000000000003e-06 2024-08-19T14:25:09.449 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.449 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.449 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:25:09.450 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:08.691081+0000", 2024-08-19T14:25:09.450 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.013922779 2024-08-19T14:25:09.450 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.450 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.450 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:25:09.450 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:08.691205+0000", 2024-08-19T14:25:09.450 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000123977 2024-08-19T14:25:09.450 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.450 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.450 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 4,11", 2024-08-19T14:25:09.450 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:08.691330+0000", 2024-08-19T14:25:09.450 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012475 2024-08-19T14:25:09.450 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.450 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.451 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:25:09.451 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:08.693634+0000", 2024-08-19T14:25:09.451 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0023041419999999999 2024-08-19T14:25:09.451 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.451 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.451 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:09.451 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:08.695701+0000", 2024-08-19T14:25:09.451 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0020676190000000001 2024-08-19T14:25:09.451 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.451 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.451 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:09.451 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:08.696133+0000", 2024-08-19T14:25:09.451 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00043198800000000001 2024-08-19T14:25:09.451 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.452 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.452 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:25:09.452 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:08.696194+0000", 2024-08-19T14:25:09.452 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.05e-05 2024-08-19T14:25:09.452 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.452 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.452 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:25:09.452 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:08.696239+0000", 2024-08-19T14:25:09.452 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.4764e-05 2024-08-19T14:25:09.452 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:09.452 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:25:09.452 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:09.452 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.452 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.453 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25813.0:54567 5.a 5:54d10afe:::benchmark_data_smithi045_85309_object54566:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e431)", 2024-08-19T14:25:09.453 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:25:09.027419+0000", 2024-08-19T14:25:09.453 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.29908040499999999, 2024-08-19T14:25:09.453 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.038295557000000001, 2024-08-19T14:25:09.453 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:25:09.453 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:25:09.453 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:25:09.453 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:25:09.453 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25813", 2024-08-19T14:25:09.453 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:25:09.453 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 54567 2024-08-19T14:25:09.453 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.453 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:25:09.453 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.454 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:25:09.454 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:09.027419+0000", 2024-08-19T14:25:09.454 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:25:09.454 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.454 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.454 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:25:09.454 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:09.027419+0000", 2024-08-19T14:25:09.454 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:25:09.454 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.454 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.454 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:25:09.454 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:09.027421+0000", 2024-08-19T14:25:09.454 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.711e-06 2024-08-19T14:25:09.454 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.455 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.455 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:25:09.455 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:09.027443+0000", 2024-08-19T14:25:09.455 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2475000000000001e-05 2024-08-19T14:25:09.455 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.455 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.455 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:25:09.455 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:09.027456+0000", 2024-08-19T14:25:09.455 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2442999999999999e-05 2024-08-19T14:25:09.455 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.455 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.455 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:25:09.455 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:09.027463+0000", 2024-08-19T14:25:09.456 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.396e-06 2024-08-19T14:25:09.456 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.456 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.456 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:25:09.456 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:09.027786+0000", 2024-08-19T14:25:09.456 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00032303699999999998 2024-08-19T14:25:09.456 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.456 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.456 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:25:09.456 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:09.027920+0000", 2024-08-19T14:25:09.456 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013418499999999999 2024-08-19T14:25:09.457 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.457 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.457 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 4,11", 2024-08-19T14:25:09.457 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:09.028068+0000", 2024-08-19T14:25:09.457 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000147503 2024-08-19T14:25:09.457 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.457 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.457 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:09.457 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:09.030555+0000", 2024-08-19T14:25:09.457 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0024870650000000001 2024-08-19T14:25:09.457 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.457 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.457 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:25:09.457 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:09.030714+0000", 2024-08-19T14:25:09.458 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015934599999999999 2024-08-19T14:25:09.458 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.458 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.458 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:09.458 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:09.065608+0000", 2024-08-19T14:25:09.458 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.034894056999999999 2024-08-19T14:25:09.458 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.458 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.458 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:25:09.458 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:09.065664+0000", 2024-08-19T14:25:09.459 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.5597999999999999e-05 2024-08-19T14:25:09.459 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:09.459 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:09.459 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:25:09.459 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:09.065715+0000", 2024-08-19T14:25:09.459 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0741000000000001e-05 2024-08-19T14:25:09.459 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:09.459 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:25:09.459 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:09.459 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:09.459 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:25:09.459 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:25:09.490 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:25:09.522 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:09.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:09.523 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:25:09.624 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:25:09.649 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd reweight 11 1 2024-08-19T14:25:09.653 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:09.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:09.654 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:25:09.675 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:25:09.755 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:25:09.776 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:09.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:09.777 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:25:09.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:09 smithi031 ceph-mon[16336]: pgmap v987: 47 pgs: 47 active+clean; 2.9 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 154 MiB/s wr, 2.46k op/s 2024-08-19T14:25:09.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:09 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3220086961' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-08-19T14:25:09.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:09 smithi031 ceph-mon[16336]: osdmap e431: 12 total, 12 up, 9 in 2024-08-19T14:25:09.879 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:25:09.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:09.914 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:09.914 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:25:10.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:09 smithi032 ceph-mon[20004]: pgmap v987: 47 pgs: 47 active+clean; 2.9 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 154 MiB/s wr, 2.46k op/s 2024-08-19T14:25:10.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:09 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3220086961' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-08-19T14:25:10.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:09 smithi032 ceph-mon[20004]: osdmap e431: 12 total, 12 up, 9 in 2024-08-19T14:25:10.015 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:25:10.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:09 smithi045 ceph-mon[20143]: pgmap v987: 47 pgs: 47 active+clean; 2.9 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 154 MiB/s wr, 2.46k op/s 2024-08-19T14:25:10.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:09 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3220086961' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-08-19T14:25:10.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:09 smithi045 ceph-mon[20143]: osdmap e431: 12 total, 12 up, 9 in 2024-08-19T14:25:10.048 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:10.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:10.050 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:25:10.151 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:25:10.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:10.185 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:10.185 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:25:10.270 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:25:10.285 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:25:10.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:10.316 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:10.317 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:25:10.320 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 22 16 56990 56974 161.829 116.062 0.0103037 0.00617394 2024-08-19T14:25:10.363 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-08-19T14:25:10.417 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:25:10.448 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:10.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:10.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:25:10.552 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:25:10.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:10.592 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:10.592 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:25:10.679 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is unset 2024-08-19T14:25:10.693 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:25:10.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:10.733 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:10.733 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:25:10.805 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:25:10.835 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:25:10.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:10 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:25:10.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:10 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:25:10.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:10 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:25:10.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:10 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:25:10.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:10 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:25:10.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:10 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:25:10.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:10 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:25:10.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:10 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1287551321' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:25:10.869 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:10.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:10.871 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:25:10.940 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:10 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:25:10.940 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:10 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:25:10.941 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:10 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:25:10.941 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:10 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:25:10.941 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:10 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:25:10.941 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:10 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:25:10.941 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:10 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:25:10.941 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:10 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1287551321' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:25:10.957 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:25:10.972 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:25:11.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:10 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:25:11.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:10 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:25:11.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:10 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:25:11.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:10 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:25:11.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:10 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:25:11.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:10 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:25:11.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:10 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:25:11.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:10 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1287551321' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:25:11.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:11.047 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:11.047 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:25:11.148 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:25:11.173 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:11.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:11.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:25:11.275 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:25:11.304 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:11.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:11.305 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:25:11.319 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 23 16 58930 58914 160.064 121.25 0.00307208 0.00623529 2024-08-19T14:25:11.359 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:25:11.359 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [ 2024-08-19T14:25:11.359 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:11.359 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25813.0:58925 5.e 5:7620478b:::benchmark_data_smithi045_85309_object58924:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e432)", 2024-08-19T14:25:11.359 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:25:11.291634+0000", 2024-08-19T14:25:11.360 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.067102035000000004, 2024-08-19T14:25:11.360 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.067131015000000002, 2024-08-19T14:25:11.360 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:25:11.360 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:25:11.360 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "waiting for sub ops", 2024-08-19T14:25:11.360 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:25:11.360 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25813", 2024-08-19T14:25:11.360 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:25:11.360 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 58925 2024-08-19T14:25:11.360 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:11.360 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:25:11.360 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:11.360 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:25:11.361 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:11.291634+0000", 2024-08-19T14:25:11.361 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:25:11.361 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:11.361 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:11.361 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:25:11.361 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:11.291634+0000", 2024-08-19T14:25:11.361 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:25:11.361 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:11.361 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:11.361 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:25:11.361 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:11.291635+0000", 2024-08-19T14:25:11.361 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6390000000000001e-06 2024-08-19T14:25:11.361 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:11.362 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:11.362 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:25:11.362 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:11.291662+0000", 2024-08-19T14:25:11.362 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.6381e-05 2024-08-19T14:25:11.362 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:11.362 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:11.362 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:25:11.362 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:11.291678+0000", 2024-08-19T14:25:11.362 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6518999999999999e-05 2024-08-19T14:25:11.362 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:11.362 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:11.362 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:25:11.362 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:11.291685+0000", 2024-08-19T14:25:11.363 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.3910000000000003e-06 2024-08-19T14:25:11.363 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:11.363 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:11.363 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:25:11.363 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:11.291730+0000", 2024-08-19T14:25:11.363 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.5738000000000002e-05 2024-08-19T14:25:11.363 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:11.363 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:11.363 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:25:11.363 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:11.291952+0000", 2024-08-19T14:25:11.363 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00022174900000000001 2024-08-19T14:25:11.363 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:11.363 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:11.364 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,8", 2024-08-19T14:25:11.364 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:11.292176+0000", 2024-08-19T14:25:11.364 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00022373299999999999 2024-08-19T14:25:11.364 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:11.364 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:11.364 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:25:11.364 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:11.294126+0000", 2024-08-19T14:25:11.364 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0019497869999999999 2024-08-19T14:25:11.364 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:11.364 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:11.364 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:11.364 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:11.294316+0000", 2024-08-19T14:25:11.364 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000190053 2024-08-19T14:25:11.365 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:11.365 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:25:11.365 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:11.365 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:11.365 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:11.365 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25813.0:58930 5.e 5:7a3efca3:::benchmark_data_smithi045_85309_object58929:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e432)", 2024-08-19T14:25:11.365 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:25:11.300352+0000", 2024-08-19T14:25:11.365 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.058383875000000002, 2024-08-19T14:25:11.365 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.058469057999999997, 2024-08-19T14:25:11.365 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:25:11.365 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:25:11.365 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "waiting for sub ops", 2024-08-19T14:25:11.365 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:25:11.365 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25813", 2024-08-19T14:25:11.366 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:25:11.366 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 58930 2024-08-19T14:25:11.366 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:11.366 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:25:11.366 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:11.366 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:25:11.366 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:11.300352+0000", 2024-08-19T14:25:11.366 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:25:11.366 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:11.366 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:11.366 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:25:11.366 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:11.300352+0000", 2024-08-19T14:25:11.366 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:25:11.366 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:11.367 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:11.367 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:25:11.367 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:11.300353+0000", 2024-08-19T14:25:11.367 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.519e-06 2024-08-19T14:25:11.367 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:11.367 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:11.367 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:25:11.367 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:11.300374+0000", 2024-08-19T14:25:11.367 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0995000000000001e-05 2024-08-19T14:25:11.367 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:11.367 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:11.367 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:25:11.367 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:11.300387+0000", 2024-08-19T14:25:11.368 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2867999999999999e-05 2024-08-19T14:25:11.368 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:11.368 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:11.368 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:25:11.368 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:11.300393+0000", 2024-08-19T14:25:11.368 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.1990000000000003e-06 2024-08-19T14:25:11.368 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:11.368 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:11.368 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:25:11.368 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:11.300452+0000", 2024-08-19T14:25:11.368 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.8879000000000003e-05 2024-08-19T14:25:11.368 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:11.368 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:11.368 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:25:11.369 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:11.300609+0000", 2024-08-19T14:25:11.369 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015657 2024-08-19T14:25:11.369 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:11.369 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:11.369 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,8", 2024-08-19T14:25:11.369 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:11.300778+0000", 2024-08-19T14:25:11.369 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00016886399999999999 2024-08-19T14:25:11.369 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:11.369 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:11.369 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:25:11.369 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:11.302561+0000", 2024-08-19T14:25:11.369 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0017837040000000001 2024-08-19T14:25:11.369 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:11.369 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:11.370 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:11.370 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:11.302824+0000", 2024-08-19T14:25:11.370 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00026281600000000003 2024-08-19T14:25:11.370 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:11.370 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:25:11.370 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:11.370 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:11.370 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-08-19T14:25:11.370 INFO:teuthology.orchestra.run.smithi045.stdout: "num_ops": 2 2024-08-19T14:25:11.370 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:25:11.406 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:25:11.441 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:11.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:11.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:25:11.544 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:25:11.579 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:11.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:11.580 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:25:11.681 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:25:11.682 INFO:teuthology.orchestra.run.smithi031.stderr:reweighted osd.11 to 1 (10000) 2024-08-19T14:25:11.716 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:11.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:11.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:25:11.818 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:25:11.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:11.851 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:11.851 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:25:11.952 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:25:11.982 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:11 smithi045 ceph-mon[20143]: pgmap v989: 47 pgs: 47 active+clean; 3.1 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 125 MiB/s wr, 2.00k op/s 2024-08-19T14:25:11.982 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:11 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:25:11.982 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:11 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:25:11.983 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:11 smithi045 ceph-mon[20143]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:25:11.983 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:11 smithi045 ceph-mon[20143]: Cluster is now healthy 2024-08-19T14:25:11.983 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:11 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1287551321' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:25:11.983 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:11 smithi045 ceph-mon[20143]: osdmap e432: 12 total, 12 up, 9 in 2024-08-19T14:25:11.983 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:11 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/494963449' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-08-19T14:25:11.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:11.987 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:11.987 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:25:12.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:11 smithi032 ceph-mon[20004]: pgmap v989: 47 pgs: 47 active+clean; 3.1 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 125 MiB/s wr, 2.00k op/s 2024-08-19T14:25:12.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:11 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:25:12.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:11 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:25:12.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:11 smithi032 ceph-mon[20004]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:25:12.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:11 smithi032 ceph-mon[20004]: Cluster is now healthy 2024-08-19T14:25:12.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:11 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1287551321' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:25:12.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:11 smithi032 ceph-mon[20004]: osdmap e432: 12 total, 12 up, 9 in 2024-08-19T14:25:12.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:11 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/494963449' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-08-19T14:25:12.030 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-08-19T14:25:12.089 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:25:12.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:11 smithi031 ceph-mon[16336]: pgmap v989: 47 pgs: 47 active+clean; 3.1 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 125 MiB/s wr, 2.00k op/s 2024-08-19T14:25:12.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:11 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:25:12.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:11 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:25:12.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:11 smithi031 ceph-mon[16336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:25:12.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:11 smithi031 ceph-mon[16336]: Cluster is now healthy 2024-08-19T14:25:12.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:11 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1287551321' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:25:12.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:11 smithi031 ceph-mon[16336]: osdmap e432: 12 total, 12 up, 9 in 2024-08-19T14:25:12.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:11 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/494963449' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-08-19T14:25:12.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:12.128 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:12.128 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:25:12.229 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:25:12.263 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:12.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:12.265 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:25:12.320 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 24 16 60768 60752 158.179 114.875 0.00899916 0.00631723 2024-08-19T14:25:12.366 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:25:12.400 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:12.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:12.401 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:25:12.426 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:25:12.502 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:25:12.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:12.537 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:12.537 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:25:12.639 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:25:12.675 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:12.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:12.676 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:25:12.776 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:25:12.801 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:12.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:12.802 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:25:12.854 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:12 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/494963449' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-08-19T14:25:12.854 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:12 smithi045 ceph-mon[20143]: osdmap e433: 12 total, 12 up, 9 in 2024-08-19T14:25:12.904 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:25:12.938 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:12 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/494963449' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-08-19T14:25:12.938 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:12 smithi031 ceph-mon[16336]: osdmap e433: 12 total, 12 up, 9 in 2024-08-19T14:25:12.940 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:12.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:12.941 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:25:13.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:12 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/494963449' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-08-19T14:25:13.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:12 smithi032 ceph-mon[20004]: osdmap e433: 12 total, 12 up, 9 in 2024-08-19T14:25:13.010 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:25:13.010 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [], 2024-08-19T14:25:13.011 INFO:teuthology.orchestra.run.smithi045.stdout: "complaint_time": 30, 2024-08-19T14:25:13.011 INFO:teuthology.orchestra.run.smithi045.stdout: "num_blocked_ops": 0 2024-08-19T14:25:13.011 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:25:13.042 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:25:13.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:13.077 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:13.077 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:25:13.178 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:25:13.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:13.211 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:13.211 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:25:13.311 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:25:13.321 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 25 16 62593 62577 156.414 114.062 0.00256667 0.00636061 2024-08-19T14:25:13.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:13.336 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:13.336 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:25:13.419 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-08-19T14:25:13.420 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-19T14:25:13.438 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:25:13.469 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:13.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:13.470 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:25:13.505 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:25:13.570 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:25:13.603 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:13.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:13.604 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:25:13.703 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-08-19T14:25:13.705 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:25:13.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:13.745 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:13.745 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:25:13.846 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:25:13.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:13.890 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:13.890 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:25:13.970 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:13 smithi031 ceph-mon[16336]: pgmap v992: 47 pgs: 47 active+clean; 3.5 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 164 MiB/s wr, 2.62k op/s 2024-08-19T14:25:13.991 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:25:14.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:13 smithi032 ceph-mon[20004]: pgmap v992: 47 pgs: 47 active+clean; 3.5 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 164 MiB/s wr, 2.62k op/s 2024-08-19T14:25:14.019 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:14.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:14.020 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:25:14.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:13 smithi045 ceph-mon[20143]: pgmap v992: 47 pgs: 47 active+clean; 3.5 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 164 MiB/s wr, 2.62k op/s 2024-08-19T14:25:14.100 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:25:14.121 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:25:14.134 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:25:14.137 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:25:14.137 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:25:14.160 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:14.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:14.161 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:25:14.164 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:25:14.164 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:25:14.196 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:25:14.196 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:25:14.222 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:25:14.222 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:25:14.244 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:25:14.245 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:25:14.261 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:25:14.264 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:25:14.264 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:25:14.282 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:25:14.282 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:25:14.283 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:14.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:14.284 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:25:14.299 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:25:14.299 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:25:14.318 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:25:14.318 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:25:14.320 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 26 16 64315 64299 154.538 107.625 0.00268783 0.00644291 2024-08-19T14:25:14.338 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:25:14.338 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:25:14.352 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:25:14.353 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:25:14.369 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:25:14.370 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:25:14.385 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:25:14.413 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:14.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:14.414 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:25:14.515 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:25:14.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:14.548 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:14.548 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:25:14.648 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:25:14.665 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:25:14.666 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:25:14.682 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:14.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:14.683 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:25:14.784 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:25:14.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:14 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1468497918' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:25:14.806 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:14.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:14.807 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:25:14.908 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:25:14.942 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:14.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:14.943 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:25:14.946 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:14 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1468497918' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:25:14.984 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-19T14:25:15.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:14 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1468497918' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:25:15.044 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:25:15.078 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:15.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:15.080 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:25:15.181 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:25:15.184 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:25:15.185 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-08-19T14:25:15.210 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:15.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:15.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:25:15.312 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:25:15.320 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 27 16 65722 65706 152.071 87.9375 0.00445265 0.00657117 2024-08-19T14:25:15.346 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:15.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:15.347 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:25:15.385 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-08-19T14:25:15.448 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:25:15.483 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:15.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:15.484 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:25:15.585 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:25:15.615 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:15.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:15.616 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:25:15.717 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:25:15.745 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:15.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:15.746 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:25:15.788 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:25:15.847 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:25:15.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:15.880 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:15.881 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:25:15.951 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:15 smithi045 ceph-mon[20143]: pgmap v993: 47 pgs: 47 active+clean; 3.7 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 133 MiB/s wr, 2.13k op/s 2024-08-19T14:25:15.981 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:25:16.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:15 smithi032 ceph-mon[20004]: pgmap v993: 47 pgs: 47 active+clean; 3.7 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 133 MiB/s wr, 2.13k op/s 2024-08-19T14:25:16.011 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:16.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:16.013 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:25:16.051 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:15 smithi031 ceph-mon[16336]: pgmap v993: 47 pgs: 47 active+clean; 3.7 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 133 MiB/s wr, 2.13k op/s 2024-08-19T14:25:16.094 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":993,"stamp":"2024-08-19T14:25:13.962910+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3953852960,"num_objects":60324,"num_object_clones":0,"num_object_copies":180972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":60322,"num_write_kb":3860608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64631,"ondisk_log_size":64631,"up":141,"acting":141,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":142,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":16138856,"kb_used_data":11486472,"kb_used_omap":14,"kb_used_meta":4652273,"kb_avail":827567512,"statfs":{"total":863955320832,"available":847429132288,"internally_reserved":0,"allocated":11762147328,"data_stored":11744377658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14353,"internal_metadata":4763928559},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[8,9,9,5,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":837025792,"num_objects":12772,"num_object_clones":0,"num_object_copies":38316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12772,"num_write_kb":817408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001278"},"pg_stats":[{"pgid":"5.4","version":"433'3687","reported_seq":3725,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-08-19T14:25:11.767130+0000","last_change":"2024-08-19T14:24:44.595230+0000","last_active":"2024-08-19T14:25:11.767130+0000","last_peered":"2024-08-19T14:25:11.767130+0000","last_clean":"2024-08-19T14:25:11.767130+0000","last_became_active":"2024-08-19T14:24:44.594920+0000","last_became_peered":"2024-08-19T14:24:44.594920+0000","last_unstale":"2024-08-19T14:25:11.767130+0000","last_undegraded":"2024-08-19T14:25:11.767130+0000","last_fullsized":"2024-08-19T14:25:11.767130+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:24:43.574783+0000","objects_scrubbed":0,"log_size":3687,"log_dups_size":0,"ondisk_log_size":3687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:13.412273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":241631232,"num_objects":3687,"num_object_clones":0,"num_object_copies":11061,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3687,"num_write_kb":235968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,0],"acting":[5,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"70'139","reported_seq":1100,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-08-19T14:25:11.685461+0000","last_change":"2024-08-19T14:24:48.191443+0000","last_active":"2024-08-19T14:25:11.685461+0000","last_peered":"2024-08-19T14:25:11.685461+0000","last_clean":"2024-08-19T14:25:11.685461+0000","last_became_active":"2024-08-19T14:18:13.807133+0000","last_became_peered":"2024-08-19T14:18:13.807133+0000","last_unstale":"2024-08-19T14:25:11.685461+0000","last_undegraded":"2024-08-19T14:25:11.685461+0000","last_fullsized":"2024-08-19T14:25:11.685461+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:48.191339+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:19:58.072083+0000","last_clean_scrub_stamp":"2024-08-19T14:24:48.191339+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:08.680074+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,9],"acting":[7,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"433'3753","reported_seq":3791,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-08-19T14:25:11.756871+0000","last_change":"2024-08-19T14:24:44.593690+0000","last_active":"2024-08-19T14:25:11.756871+0000","last_peered":"2024-08-19T14:25:11.756871+0000","last_clean":"2024-08-19T14:25:11.756871+0000","last_became_active":"2024-08-19T14:24:44.593405+0000","last_became_peered":"2024-08-19T14:24:44.593405+0000","last_unstale":"2024-08-19T14:25:11.756871+0000","last_undegraded":"2024-08-19T14:25:11.756871+0000","last_fullsized":"2024-08-19T14:25:11.756871+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:24:43.574783+0000","objects_scrubbed":0,"log_size":3753,"log_dups_size":0,"ondisk_log_size":3753,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:25:59.739209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":245956608,"num_objects":3753,"num_object_clones":0,"num_object_copies":11259,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3753,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3753,"num_write_kb":240192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.3","version":"70'139","reported_seq":998,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-08-19T14:25:11.686477+0000","last_change":"2024-08-19T14:24:42.697899+0000","last_active":"2024-08-19T14:25:11.686477+0000","last_peered":"2024-08-19T14:25:11.686477+0000","last_clean":"2024-08-19T14:25:11.686477+0000","last_became_active":"2024-08-19T14:18:13.804721+0000","last_became_peered":"2024-08-19T14:18:13.804721+0000","last_unstale":"2024-08-19T14:25:11.686477+0000","last_undegraded":"2024-08-19T14:25:11.686477+0000","last_fullsized":"2024-08-19T14:25:11.686477+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:42.697801+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:24:42.697801+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:08.817098+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"433'3808","reported_seq":3846,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-08-19T14:25:11.752577+0000","last_change":"2024-08-19T14:24:44.593298+0000","last_active":"2024-08-19T14:25:11.752577+0000","last_peered":"2024-08-19T14:25:11.752577+0000","last_clean":"2024-08-19T14:25:11.752577+0000","last_became_active":"2024-08-19T14:24:44.593002+0000","last_became_peered":"2024-08-19T14:24:44.593002+0000","last_unstale":"2024-08-19T14:25:11.752577+0000","last_undegraded":"2024-08-19T14:25:11.752577+0000","last_fullsized":"2024-08-19T14:25:11.752577+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:24:43.574783+0000","objects_scrubbed":0,"log_size":3808,"log_dups_size":0,"ondisk_log_size":3808,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:25:48.182254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":249561088,"num_objects":3808,"num_object_clones":0,"num_object_copies":11424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3808,"num_write_kb":243712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,8],"acting":[5,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"70'139","reported_seq":947,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-08-19T14:25:11.685561+0000","last_change":"2024-08-19T14:24:43.246467+0000","last_active":"2024-08-19T14:25:11.685561+0000","last_peered":"2024-08-19T14:25:11.685561+0000","last_clean":"2024-08-19T14:25:11.685561+0000","last_became_active":"2024-08-19T14:20:50.742354+0000","last_became_peered":"2024-08-19T14:20:50.742354+0000","last_unstale":"2024-08-19T14:25:11.685561+0000","last_undegraded":"2024-08-19T14:25:11.685561+0000","last_fullsized":"2024-08-19T14:25:11.685561+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:43.246387+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:19:59.302931+0000","last_clean_scrub_stamp":"2024-08-19T14:24:43.246387+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:05.068356+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.5","version":"432'3609","reported_seq":3645,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-08-19T14:25:11.394817+0000","last_change":"2024-08-19T14:24:44.592723+0000","last_active":"2024-08-19T14:25:11.394817+0000","last_peered":"2024-08-19T14:25:11.394817+0000","last_clean":"2024-08-19T14:25:11.394817+0000","last_became_active":"2024-08-19T14:24:44.592424+0000","last_became_peered":"2024-08-19T14:24:44.592424+0000","last_unstale":"2024-08-19T14:25:11.394817+0000","last_undegraded":"2024-08-19T14:25:11.394817+0000","last_fullsized":"2024-08-19T14:25:11.394817+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:24:43.574783+0000","objects_scrubbed":0,"log_size":3609,"log_dups_size":0,"ondisk_log_size":3609,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:03.869565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":236519424,"num_objects":3609,"num_object_clones":0,"num_object_copies":10827,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3609,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3609,"num_write_kb":230976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"70'139","reported_seq":995,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-08-19T14:25:11.686379+0000","last_change":"2024-08-19T14:24:48.179646+0000","last_active":"2024-08-19T14:25:11.686379+0000","last_peered":"2024-08-19T14:25:11.686379+0000","last_clean":"2024-08-19T14:25:11.686379+0000","last_became_active":"2024-08-19T14:18:13.808505+0000","last_became_peered":"2024-08-19T14:18:13.808505+0000","last_unstale":"2024-08-19T14:25:11.686379+0000","last_undegraded":"2024-08-19T14:25:11.686379+0000","last_fullsized":"2024-08-19T14:25:11.686379+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:48.179558+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:24:48.179558+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:25:51.185230+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,11],"acting":[9,0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.3","version":"432'3550","reported_seq":3585,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-08-19T14:25:11.410017+0000","last_change":"2024-08-19T14:24:44.659231+0000","last_active":"2024-08-19T14:25:11.410017+0000","last_peered":"2024-08-19T14:25:11.410017+0000","last_clean":"2024-08-19T14:25:11.410017+0000","last_became_active":"2024-08-19T14:24:44.658960+0000","last_became_peered":"2024-08-19T14:24:44.658960+0000","last_unstale":"2024-08-19T14:25:11.410017+0000","last_undegraded":"2024-08-19T14:25:11.410017+0000","last_fullsized":"2024-08-19T14:25:11.410017+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:24:43.574783+0000","objects_scrubbed":0,"log_size":3550,"log_dups_size":0,"ondisk_log_size":3550,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:02.684044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":232652800,"num_objects":3550,"num_object_clones":0,"num_object_copies":10650,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3550,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3550,"num_write_kb":227200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"70'139","reported_seq":647,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-08-19T14:25:11.691507+0000","last_change":"2024-08-19T14:24:47.412400+0000","last_active":"2024-08-19T14:25:11.691507+0000","last_peered":"2024-08-19T14:25:11.691507+0000","last_clean":"2024-08-19T14:25:11.691507+0000","last_became_active":"2024-08-19T14:20:28.204384+0000","last_became_peered":"2024-08-19T14:20:28.204384+0000","last_unstale":"2024-08-19T14:25:11.691507+0000","last_undegraded":"2024-08-19T14:25:11.691507+0000","last_fullsized":"2024-08-19T14:25:11.691507+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:47.412291+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:21:46.057624+0000","last_clean_scrub_stamp":"2024-08-19T14:24:47.412291+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:11.449015+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"432'3661","reported_seq":3695,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-08-19T14:25:11.401093+0000","last_change":"2024-08-19T14:24:44.592960+0000","last_active":"2024-08-19T14:25:11.401093+0000","last_peered":"2024-08-19T14:25:11.401093+0000","last_clean":"2024-08-19T14:25:11.401093+0000","last_became_active":"2024-08-19T14:24:44.592670+0000","last_became_peered":"2024-08-19T14:24:44.592670+0000","last_unstale":"2024-08-19T14:25:11.401093+0000","last_undegraded":"2024-08-19T14:25:11.401093+0000","last_fullsized":"2024-08-19T14:25:11.401093+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:24:43.574783+0000","objects_scrubbed":0,"log_size":3661,"log_dups_size":0,"ondisk_log_size":3661,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:07.754236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":239927296,"num_objects":3661,"num_object_clones":0,"num_object_copies":10983,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3661,"num_write_kb":234304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,8],"acting":[0,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"70'139","reported_seq":999,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-08-19T14:25:11.686524+0000","last_change":"2024-08-19T14:24:46.222297+0000","last_active":"2024-08-19T14:25:11.686524+0000","last_peered":"2024-08-19T14:25:11.686524+0000","last_clean":"2024-08-19T14:25:11.686524+0000","last_became_active":"2024-08-19T14:18:26.874902+0000","last_became_peered":"2024-08-19T14:18:26.874902+0000","last_unstale":"2024-08-19T14:25:11.686524+0000","last_undegraded":"2024-08-19T14:25:11.686524+0000","last_fullsized":"2024-08-19T14:25:11.686524+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:46.222166+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:27.848870+0000","last_clean_scrub_stamp":"2024-08-19T14:24:46.222166+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:09.431052+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,4],"acting":[9,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.1","version":"433'3926","reported_seq":3964,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-08-19T14:25:12.734556+0000","last_change":"2024-08-19T14:24:44.658485+0000","last_active":"2024-08-19T14:25:12.734556+0000","last_peered":"2024-08-19T14:25:12.734556+0000","last_clean":"2024-08-19T14:25:12.734556+0000","last_became_active":"2024-08-19T14:24:44.657571+0000","last_became_peered":"2024-08-19T14:24:44.657571+0000","last_unstale":"2024-08-19T14:25:12.734556+0000","last_undegraded":"2024-08-19T14:25:12.734556+0000","last_fullsized":"2024-08-19T14:25:12.734556+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:24:43.574783+0000","objects_scrubbed":0,"log_size":3926,"log_dups_size":0,"ondisk_log_size":3926,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:25:50.673564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":257294336,"num_objects":3926,"num_object_clones":0,"num_object_copies":11778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3926,"num_write_kb":251264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"70'139","reported_seq":819,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-08-19T14:25:10.685399+0000","last_change":"2024-08-19T14:24:46.629045+0000","last_active":"2024-08-19T14:25:10.685399+0000","last_peered":"2024-08-19T14:25:10.685399+0000","last_clean":"2024-08-19T14:25:10.685399+0000","last_became_active":"2024-08-19T14:18:26.881435+0000","last_became_peered":"2024-08-19T14:18:26.881435+0000","last_unstale":"2024-08-19T14:25:10.685399+0000","last_undegraded":"2024-08-19T14:25:10.685399+0000","last_fullsized":"2024-08-19T14:25:10.685399+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:46.628960+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:24:46.628960+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:03.429914+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,4],"acting":[11,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.0","version":"433'3974","reported_seq":4012,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-08-19T14:25:13.406556+0000","last_change":"2024-08-19T14:24:44.595889+0000","last_active":"2024-08-19T14:25:13.406556+0000","last_peered":"2024-08-19T14:25:13.406556+0000","last_clean":"2024-08-19T14:25:13.406556+0000","last_became_active":"2024-08-19T14:24:44.595535+0000","last_became_peered":"2024-08-19T14:24:44.595535+0000","last_unstale":"2024-08-19T14:25:13.406556+0000","last_undegraded":"2024-08-19T14:25:13.406556+0000","last_fullsized":"2024-08-19T14:25:13.406556+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:24:43.574783+0000","objects_scrubbed":0,"log_size":3974,"log_dups_size":0,"ondisk_log_size":3974,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:07.707608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260440064,"num_objects":3974,"num_object_clones":0,"num_object_copies":11922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3974,"num_write_kb":254336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"70'139","reported_seq":1043,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-08-19T14:25:10.683560+0000","last_change":"2024-08-19T14:24:11.345984+0000","last_active":"2024-08-19T14:25:10.683560+0000","last_peered":"2024-08-19T14:25:10.683560+0000","last_clean":"2024-08-19T14:25:10.683560+0000","last_became_active":"2024-08-19T14:18:13.806013+0000","last_became_peered":"2024-08-19T14:18:13.806013+0000","last_unstale":"2024-08-19T14:25:10.683560+0000","last_undegraded":"2024-08-19T14:25:10.683560+0000","last_fullsized":"2024-08-19T14:25:10.683560+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:11.345873+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:24:11.345873+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:25:14.696871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.e","version":"433'3895","reported_seq":3933,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-08-19T14:25:13.518761+0000","last_change":"2024-08-19T14:24:44.595518+0000","last_active":"2024-08-19T14:25:13.518761+0000","last_peered":"2024-08-19T14:25:13.518761+0000","last_clean":"2024-08-19T14:25:13.518761+0000","last_became_active":"2024-08-19T14:24:44.595166+0000","last_became_peered":"2024-08-19T14:24:44.595166+0000","last_unstale":"2024-08-19T14:25:13.518761+0000","last_undegraded":"2024-08-19T14:25:13.518761+0000","last_fullsized":"2024-08-19T14:25:13.518761+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:24:43.574783+0000","objects_scrubbed":0,"log_size":3895,"log_dups_size":0,"ondisk_log_size":3895,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:09.480899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":255262720,"num_objects":3895,"num_object_clones":0,"num_object_copies":11685,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3895,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3895,"num_write_kb":249280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,0],"acting":[9,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.a","version":"70'139","reported_seq":1007,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-08-19T14:25:11.686319+0000","last_change":"2024-08-19T14:24:50.196984+0000","last_active":"2024-08-19T14:25:11.686319+0000","last_peered":"2024-08-19T14:25:11.686319+0000","last_clean":"2024-08-19T14:25:11.686319+0000","last_became_active":"2024-08-19T14:18:26.875006+0000","last_became_peered":"2024-08-19T14:18:26.875006+0000","last_unstale":"2024-08-19T14:25:11.686319+0000","last_undegraded":"2024-08-19T14:25:11.686319+0000","last_fullsized":"2024-08-19T14:25:11.686319+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:50.196885+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:28.864870+0000","last_clean_scrub_stamp":"2024-08-19T14:24:50.196885+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:15.234239+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,0],"acting":[9,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.d","version":"432'3726","reported_seq":3762,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-08-19T14:25:11.080664+0000","last_change":"2024-08-19T14:24:44.592424+0000","last_active":"2024-08-19T14:25:11.080664+0000","last_peered":"2024-08-19T14:25:11.080664+0000","last_clean":"2024-08-19T14:25:11.080664+0000","last_became_active":"2024-08-19T14:24:44.592098+0000","last_became_peered":"2024-08-19T14:24:44.592098+0000","last_unstale":"2024-08-19T14:25:11.080664+0000","last_undegraded":"2024-08-19T14:25:11.080664+0000","last_fullsized":"2024-08-19T14:25:11.080664+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:24:43.574783+0000","objects_scrubbed":0,"log_size":3726,"log_dups_size":0,"ondisk_log_size":3726,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:25:54.822583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":244187136,"num_objects":3726,"num_object_clones":0,"num_object_copies":11178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3726,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3726,"num_write_kb":238464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7],"acting":[11,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.9","version":"70'139","reported_seq":1122,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-08-19T14:25:11.685630+0000","last_change":"2024-08-19T14:25:04.127400+0000","last_active":"2024-08-19T14:25:11.685630+0000","last_peered":"2024-08-19T14:25:11.685630+0000","last_clean":"2024-08-19T14:25:11.685630+0000","last_became_active":"2024-08-19T14:18:26.887657+0000","last_became_peered":"2024-08-19T14:18:26.887657+0000","last_unstale":"2024-08-19T14:25:11.685630+0000","last_undegraded":"2024-08-19T14:25:11.685630+0000","last_fullsized":"2024-08-19T14:25:11.685630+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:25:04.127296+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:25:04.127296+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:12.377922+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"433'3979","reported_seq":4016,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-08-19T14:25:13.414373+0000","last_change":"2024-08-19T14:24:44.595877+0000","last_active":"2024-08-19T14:25:13.414373+0000","last_peered":"2024-08-19T14:25:13.414373+0000","last_clean":"2024-08-19T14:25:13.414373+0000","last_became_active":"2024-08-19T14:24:44.595538+0000","last_became_peered":"2024-08-19T14:24:44.595538+0000","last_unstale":"2024-08-19T14:25:13.414373+0000","last_undegraded":"2024-08-19T14:25:13.414373+0000","last_fullsized":"2024-08-19T14:25:13.414373+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:24:43.574783+0000","objects_scrubbed":0,"log_size":3979,"log_dups_size":0,"ondisk_log_size":3979,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:25:46.650773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260767744,"num_objects":3979,"num_object_clones":0,"num_object_copies":11937,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3979,"num_write_kb":254656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"70'139","reported_seq":1019,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-08-19T14:25:11.686772+0000","last_change":"2024-08-19T14:24:45.176469+0000","last_active":"2024-08-19T14:25:11.686772+0000","last_peered":"2024-08-19T14:25:11.686772+0000","last_clean":"2024-08-19T14:25:11.686772+0000","last_became_active":"2024-08-19T14:18:26.887277+0000","last_became_peered":"2024-08-19T14:18:26.887277+0000","last_unstale":"2024-08-19T14:25:11.686772+0000","last_undegraded":"2024-08-19T14:25:11.686772+0000","last_fullsized":"2024-08-19T14:25:11.686772+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:45.176400+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:24:45.176400+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:12.727044+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,8],"acting":[5,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.f","version":"433'3679","reported_seq":3716,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-08-19T14:25:11.762556+0000","last_change":"2024-08-19T14:24:44.593761+0000","last_active":"2024-08-19T14:25:11.762556+0000","last_peered":"2024-08-19T14:25:11.762556+0000","last_clean":"2024-08-19T14:25:11.762556+0000","last_became_active":"2024-08-19T14:24:44.593498+0000","last_became_peered":"2024-08-19T14:24:44.593498+0000","last_unstale":"2024-08-19T14:25:11.762556+0000","last_undegraded":"2024-08-19T14:25:11.762556+0000","last_fullsized":"2024-08-19T14:25:11.762556+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:24:43.574783+0000","objects_scrubbed":0,"log_size":3679,"log_dups_size":0,"ondisk_log_size":3679,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:25:50.363040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":241106944,"num_objects":3679,"num_object_clones":0,"num_object_copies":11037,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3679,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3679,"num_write_kb":235456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,11],"acting":[5,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"70'139","reported_seq":1019,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-08-19T14:25:10.683386+0000","last_change":"2024-08-19T14:24:45.610330+0000","last_active":"2024-08-19T14:25:10.683386+0000","last_peered":"2024-08-19T14:25:10.683386+0000","last_clean":"2024-08-19T14:25:10.683386+0000","last_became_active":"2024-08-19T14:18:26.875179+0000","last_became_peered":"2024-08-19T14:18:26.875179+0000","last_unstale":"2024-08-19T14:25:10.683386+0000","last_undegraded":"2024-08-19T14:25:10.683386+0000","last_fullsized":"2024-08-19T14:25:10.683386+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:45.610239+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:24:45.610239+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:15.465099+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.a","version":"431'3561","reported_seq":3595,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-08-19T14:25:09.967440+0000","last_change":"2024-08-19T14:24:44.592347+0000","last_active":"2024-08-19T14:25:09.967440+0000","last_peered":"2024-08-19T14:25:09.967440+0000","last_clean":"2024-08-19T14:25:09.967440+0000","last_became_active":"2024-08-19T14:24:44.592024+0000","last_became_peered":"2024-08-19T14:24:44.592024+0000","last_unstale":"2024-08-19T14:25:09.967440+0000","last_undegraded":"2024-08-19T14:25:09.967440+0000","last_fullsized":"2024-08-19T14:25:09.967440+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:24:43.574783+0000","objects_scrubbed":0,"log_size":3561,"log_dups_size":0,"ondisk_log_size":3561,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:06.455975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233373696,"num_objects":3561,"num_object_clones":0,"num_object_copies":10683,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3561,"num_write_kb":227904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,11],"acting":[8,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.e","version":"70'139","reported_seq":675,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-08-19T14:25:11.689090+0000","last_change":"2024-08-19T14:25:04.737969+0000","last_active":"2024-08-19T14:25:11.689090+0000","last_peered":"2024-08-19T14:25:11.689090+0000","last_clean":"2024-08-19T14:25:11.689090+0000","last_became_active":"2024-08-19T14:18:26.884105+0000","last_became_peered":"2024-08-19T14:18:26.884105+0000","last_unstale":"2024-08-19T14:25:11.689090+0000","last_undegraded":"2024-08-19T14:25:11.689090+0000","last_fullsized":"2024-08-19T14:25:11.689090+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:25:04.737897+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:20:15.321945+0000","last_clean_scrub_stamp":"2024-08-19T14:25:04.737897+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:25.666670+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.b","version":"433'4007","reported_seq":4044,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-08-19T14:25:13.523831+0000","last_change":"2024-08-19T14:24:44.595612+0000","last_active":"2024-08-19T14:25:13.523831+0000","last_peered":"2024-08-19T14:25:13.523831+0000","last_clean":"2024-08-19T14:25:13.523831+0000","last_became_active":"2024-08-19T14:24:44.595175+0000","last_became_peered":"2024-08-19T14:24:44.595175+0000","last_unstale":"2024-08-19T14:25:13.523831+0000","last_undegraded":"2024-08-19T14:25:13.523831+0000","last_fullsized":"2024-08-19T14:25:13.523831+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:24:43.574783+0000","objects_scrubbed":0,"log_size":4007,"log_dups_size":0,"ondisk_log_size":4007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:25:55.419449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262602752,"num_objects":4007,"num_object_clones":0,"num_object_copies":12021,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4007,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4007,"num_write_kb":256448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,5],"acting":[9,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.f","version":"70'139","reported_seq":919,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-08-19T14:25:11.686961+0000","last_change":"2024-08-19T14:24:16.076239+0000","last_active":"2024-08-19T14:25:11.686961+0000","last_peered":"2024-08-19T14:25:11.686961+0000","last_clean":"2024-08-19T14:25:11.686961+0000","last_became_active":"2024-08-19T14:18:18.711708+0000","last_became_peered":"2024-08-19T14:18:18.711708+0000","last_unstale":"2024-08-19T14:25:11.686961+0000","last_undegraded":"2024-08-19T14:25:11.686961+0000","last_fullsized":"2024-08-19T14:25:11.686961+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:16.076143+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:24:16.076143+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:25:32.997797+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.8","version":"432'3600","reported_seq":3636,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-08-19T14:25:11.093036+0000","last_change":"2024-08-19T14:24:44.591109+0000","last_active":"2024-08-19T14:25:11.093036+0000","last_peered":"2024-08-19T14:25:11.093036+0000","last_clean":"2024-08-19T14:25:11.093036+0000","last_became_active":"2024-08-19T14:24:44.590679+0000","last_became_peered":"2024-08-19T14:24:44.590679+0000","last_unstale":"2024-08-19T14:25:11.093036+0000","last_undegraded":"2024-08-19T14:25:11.093036+0000","last_fullsized":"2024-08-19T14:25:11.093036+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:24:43.574783+0000","objects_scrubbed":0,"log_size":3600,"log_dups_size":0,"ondisk_log_size":3600,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:25:52.297517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235929600,"num_objects":3600,"num_object_clones":0,"num_object_copies":10800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3600,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3600,"num_write_kb":230400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.c","version":"70'139","reported_seq":780,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-08-19T14:25:11.691228+0000","last_change":"2024-08-19T14:24:13.465173+0000","last_active":"2024-08-19T14:25:11.691228+0000","last_peered":"2024-08-19T14:25:11.691228+0000","last_clean":"2024-08-19T14:25:11.691228+0000","last_became_active":"2024-08-19T14:18:26.872942+0000","last_became_peered":"2024-08-19T14:18:26.872942+0000","last_unstale":"2024-08-19T14:25:11.691228+0000","last_undegraded":"2024-08-19T14:25:11.691228+0000","last_fullsized":"2024-08-19T14:25:11.691228+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:13.463716+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:22:52.164586+0000","last_clean_scrub_stamp":"2024-08-19T14:24:13.463716+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:25:23.123046+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,9],"acting":[3,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"433'3907","reported_seq":3945,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-08-19T14:25:13.602713+0000","last_change":"2024-08-19T14:24:44.592293+0000","last_active":"2024-08-19T14:25:13.602713+0000","last_peered":"2024-08-19T14:25:13.602713+0000","last_clean":"2024-08-19T14:25:13.602713+0000","last_became_active":"2024-08-19T14:24:44.591973+0000","last_became_peered":"2024-08-19T14:24:44.591973+0000","last_unstale":"2024-08-19T14:25:13.602713+0000","last_undegraded":"2024-08-19T14:25:13.602713+0000","last_fullsized":"2024-08-19T14:25:13.602713+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:24:43.574783+0000","objects_scrubbed":0,"log_size":3907,"log_dups_size":0,"ondisk_log_size":3907,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:25:54.081269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256049152,"num_objects":3907,"num_object_clones":0,"num_object_copies":11721,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3907,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3907,"num_write_kb":250048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,9],"acting":[7,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"70'139","reported_seq":695,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-08-19T14:25:11.685866+0000","last_change":"2024-08-19T14:24:50.276533+0000","last_active":"2024-08-19T14:25:11.685866+0000","last_peered":"2024-08-19T14:25:11.685866+0000","last_clean":"2024-08-19T14:25:11.685866+0000","last_became_active":"2024-08-19T14:18:26.882002+0000","last_became_peered":"2024-08-19T14:18:26.882002+0000","last_unstale":"2024-08-19T14:25:11.685866+0000","last_undegraded":"2024-08-19T14:25:11.685866+0000","last_fullsized":"2024-08-19T14:25:11.685866+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:50.276408+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:24:50.276408+0000","last_clean_scrub_stamp":"2024-08-19T14:24:50.276408+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:06.161471+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"70'139","reported_seq":953,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-08-19T14:25:11.686894+0000","last_change":"2024-08-19T14:24:14.035778+0000","last_active":"2024-08-19T14:25:11.686894+0000","last_peered":"2024-08-19T14:25:11.686894+0000","last_clean":"2024-08-19T14:25:11.686894+0000","last_became_active":"2024-08-19T14:18:18.703594+0000","last_became_peered":"2024-08-19T14:18:18.703594+0000","last_unstale":"2024-08-19T14:25:11.686894+0000","last_undegraded":"2024-08-19T14:25:11.686894+0000","last_fullsized":"2024-08-19T14:25:11.686894+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:14.035665+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:24:14.035665+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:25:35.876300+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"70'139","reported_seq":1005,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-08-19T14:25:11.686366+0000","last_change":"2024-08-19T14:25:04.742939+0000","last_active":"2024-08-19T14:25:11.686366+0000","last_peered":"2024-08-19T14:25:11.686366+0000","last_clean":"2024-08-19T14:25:11.686366+0000","last_became_active":"2024-08-19T14:18:18.698334+0000","last_became_peered":"2024-08-19T14:18:18.698334+0000","last_unstale":"2024-08-19T14:25:11.686366+0000","last_undegraded":"2024-08-19T14:25:11.686366+0000","last_fullsized":"2024-08-19T14:25:11.686366+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:25:04.742869+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:25:04.742869+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:34.199161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.10","version":"70'139","reported_seq":1116,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-08-19T14:25:11.686674+0000","last_change":"2024-08-19T14:25:06.143718+0000","last_active":"2024-08-19T14:25:11.686674+0000","last_peered":"2024-08-19T14:25:11.686674+0000","last_clean":"2024-08-19T14:25:11.686674+0000","last_became_active":"2024-08-19T14:18:18.704562+0000","last_became_peered":"2024-08-19T14:18:18.704562+0000","last_unstale":"2024-08-19T14:25:11.686674+0000","last_undegraded":"2024-08-19T14:25:11.686674+0000","last_fullsized":"2024-08-19T14:25:11.686674+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:25:06.143606+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:25:06.143606+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:16.549538+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,8],"acting":[5,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"70'139","reported_seq":663,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-08-19T14:25:10.680758+0000","last_change":"2024-08-19T14:24:50.547549+0000","last_active":"2024-08-19T14:25:10.680758+0000","last_peered":"2024-08-19T14:25:10.680758+0000","last_clean":"2024-08-19T14:25:10.680758+0000","last_became_active":"2024-08-19T14:18:18.697894+0000","last_became_peered":"2024-08-19T14:18:18.697894+0000","last_unstale":"2024-08-19T14:25:10.680758+0000","last_undegraded":"2024-08-19T14:25:10.680758+0000","last_fullsized":"2024-08-19T14:25:10.680758+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:50.547428+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:24:50.547428+0000","last_clean_scrub_stamp":"2024-08-19T14:24:50.547428+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:14.355547+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"70'139","reported_seq":634,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-08-19T14:25:11.686598+0000","last_change":"2024-08-19T14:24:41.729459+0000","last_active":"2024-08-19T14:25:11.686598+0000","last_peered":"2024-08-19T14:25:11.686598+0000","last_clean":"2024-08-19T14:25:11.686598+0000","last_became_active":"2024-08-19T14:20:28.204724+0000","last_became_peered":"2024-08-19T14:20:28.204724+0000","last_unstale":"2024-08-19T14:25:11.686598+0000","last_undegraded":"2024-08-19T14:25:11.686598+0000","last_fullsized":"2024-08-19T14:25:11.686598+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:41.729377+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:24:41.729377+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:09.621000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"70'139","reported_seq":758,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-08-19T14:25:08.649862+0000","last_change":"2024-08-19T14:24:48.428636+0000","last_active":"2024-08-19T14:25:08.649862+0000","last_peered":"2024-08-19T14:25:08.649862+0000","last_clean":"2024-08-19T14:25:08.649862+0000","last_became_active":"2024-08-19T14:18:31.901556+0000","last_became_peered":"2024-08-19T14:18:31.901556+0000","last_unstale":"2024-08-19T14:25:08.649862+0000","last_undegraded":"2024-08-19T14:25:08.649862+0000","last_fullsized":"2024-08-19T14:25:08.649862+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:48.428515+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:24:48.428515+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:13.841914+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,0],"acting":[8,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.14","version":"70'139","reported_seq":763,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-08-19T14:25:11.685632+0000","last_change":"2024-08-19T14:24:23.204225+0000","last_active":"2024-08-19T14:25:11.685632+0000","last_peered":"2024-08-19T14:25:11.685632+0000","last_clean":"2024-08-19T14:25:11.685632+0000","last_became_active":"2024-08-19T14:18:18.703926+0000","last_became_peered":"2024-08-19T14:18:18.703926+0000","last_unstale":"2024-08-19T14:25:11.685632+0000","last_undegraded":"2024-08-19T14:25:11.685632+0000","last_fullsized":"2024-08-19T14:25:11.685632+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:23.204112+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:24:23.204112+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:25:30.300500+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,1],"acting":[7,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.15","version":"70'139","reported_seq":650,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-08-19T14:25:11.685799+0000","last_change":"2024-08-19T14:24:44.255453+0000","last_active":"2024-08-19T14:25:11.685799+0000","last_peered":"2024-08-19T14:25:11.685799+0000","last_clean":"2024-08-19T14:25:11.685799+0000","last_became_active":"2024-08-19T14:18:31.912496+0000","last_became_peered":"2024-08-19T14:18:31.912496+0000","last_unstale":"2024-08-19T14:25:11.685799+0000","last_undegraded":"2024-08-19T14:25:11.685799+0000","last_fullsized":"2024-08-19T14:25:11.685799+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:44.255304+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:20:09.180923+0000","last_clean_scrub_stamp":"2024-08-19T14:24:44.255304+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:25:54.443311+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"70'139","reported_seq":636,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-08-19T14:25:11.688949+0000","last_change":"2024-08-19T14:24:44.775890+0000","last_active":"2024-08-19T14:25:11.688949+0000","last_peered":"2024-08-19T14:25:11.688949+0000","last_clean":"2024-08-19T14:25:11.688949+0000","last_became_active":"2024-08-19T14:18:31.901229+0000","last_became_peered":"2024-08-19T14:18:31.901229+0000","last_unstale":"2024-08-19T14:25:11.688949+0000","last_undegraded":"2024-08-19T14:25:11.688949+0000","last_fullsized":"2024-08-19T14:25:11.688949+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:44.775817+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:24:44.775817+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:25:48.718462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1b","version":"70'139","reported_seq":984,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-08-19T14:25:08.649537+0000","last_change":"2024-08-19T14:24:47.410923+0000","last_active":"2024-08-19T14:25:08.649537+0000","last_peered":"2024-08-19T14:25:08.649537+0000","last_clean":"2024-08-19T14:25:08.649537+0000","last_became_active":"2024-08-19T14:18:31.901152+0000","last_became_peered":"2024-08-19T14:18:31.901152+0000","last_unstale":"2024-08-19T14:25:08.649537+0000","last_undegraded":"2024-08-19T14:25:08.649537+0000","last_fullsized":"2024-08-19T14:25:08.649537+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:47.410831+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:24:47.410831+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:07.850070+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.18","version":"70'139","reported_seq":1032,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-08-19T14:25:11.686567+0000","last_change":"2024-08-19T14:24:40.113681+0000","last_active":"2024-08-19T14:25:11.686567+0000","last_peered":"2024-08-19T14:25:11.686567+0000","last_clean":"2024-08-19T14:25:11.686567+0000","last_became_active":"2024-08-19T14:18:31.902735+0000","last_became_peered":"2024-08-19T14:18:31.902735+0000","last_unstale":"2024-08-19T14:25:11.686567+0000","last_undegraded":"2024-08-19T14:25:11.686567+0000","last_fullsized":"2024-08-19T14:25:11.686567+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:40.113600+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:24:40.113600+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:25:58.753284+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.19","version":"70'139","reported_seq":978,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-08-19T14:25:10.680653+0000","last_change":"2024-08-19T14:24:42.577775+0000","last_active":"2024-08-19T14:25:10.680653+0000","last_peered":"2024-08-19T14:25:10.680653+0000","last_clean":"2024-08-19T14:25:10.680653+0000","last_became_active":"2024-08-19T14:18:31.902547+0000","last_became_peered":"2024-08-19T14:18:31.902547+0000","last_unstale":"2024-08-19T14:25:10.680653+0000","last_undegraded":"2024-08-19T14:25:10.680653+0000","last_fullsized":"2024-08-19T14:25:10.680653+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:42.577693+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:24:42.577693+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:25:43.680823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1e","version":"70'139","reported_seq":637,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-08-19T14:25:10.683661+0000","last_change":"2024-08-19T14:24:44.651398+0000","last_active":"2024-08-19T14:25:10.683661+0000","last_peered":"2024-08-19T14:25:10.683661+0000","last_clean":"2024-08-19T14:25:10.683661+0000","last_became_active":"2024-08-19T14:18:31.902546+0000","last_became_peered":"2024-08-19T14:18:31.902546+0000","last_unstale":"2024-08-19T14:25:10.683661+0000","last_undegraded":"2024-08-19T14:25:10.683661+0000","last_fullsized":"2024-08-19T14:25:10.683661+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:44.651299+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:24:44.651299+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:13.740488+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1c","version":"70'139","reported_seq":742,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-08-19T14:25:08.650679+0000","last_change":"2024-08-19T14:24:14.461312+0000","last_active":"2024-08-19T14:25:08.650679+0000","last_peered":"2024-08-19T14:25:08.650679+0000","last_clean":"2024-08-19T14:25:08.650679+0000","last_became_active":"2024-08-19T14:18:31.902494+0000","last_became_peered":"2024-08-19T14:18:31.902494+0000","last_unstale":"2024-08-19T14:25:08.650679+0000","last_undegraded":"2024-08-19T14:25:08.650679+0000","last_fullsized":"2024-08-19T14:25:08.650679+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:14.461176+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:24:14.461176+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:25:26.070165+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,11],"acting":[8,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1d","version":"70'139","reported_seq":660,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-08-19T14:25:11.686540+0000","last_change":"2024-08-19T14:24:22.588391+0000","last_active":"2024-08-19T14:25:11.686540+0000","last_peered":"2024-08-19T14:25:11.686540+0000","last_clean":"2024-08-19T14:25:11.686540+0000","last_became_active":"2024-08-19T14:18:31.901183+0000","last_became_peered":"2024-08-19T14:18:31.901183+0000","last_unstale":"2024-08-19T14:25:11.686540+0000","last_undegraded":"2024-08-19T14:25:11.686540+0000","last_fullsized":"2024-08-19T14:25:11.686540+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:22.588266+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:24:22.588266+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:25:26.637645+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,5],"acting":[1,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":3953262592,"num_objects":60322,"num_object_clones":0,"num_object_copies":180966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":60322,"num_write_kb":3860608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11879383040,"data_stored":11879383040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":60322,"ondisk_log_size":60322,"up":48,"acting":48,"num_store_stats":9},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1802240,"data_stored":1788032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187622,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1669908,"kb_used_data":1151436,"kb_used_omap":1,"kb_used_meta":518462,"kb_avail":92075244,"statfs":{"total":95995035648,"available":94285049856,"internally_reserved":0,"allocated":1179070464,"data_stored":1177102403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":530905540},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":343597383852,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":74,"seq":317827580080,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1806000,"kb_used_data":1251428,"kb_used_omap":1,"kb_used_meta":554558,"kb_avail":91939152,"statfs":{"total":95995035648,"available":94145691648,"internally_reserved":0,"allocated":1281462272,"data_stored":1279485759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":567867841},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874421,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1516760,"kb_used_data":1106572,"kb_used_omap":1,"kb_used_meta":410174,"kb_avail":92228392,"statfs":{"total":95995035648,"available":94441873408,"internally_reserved":0,"allocated":1133129728,"data_stored":1131161667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":420018629},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":57,"seq":244813136057,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1855100,"kb_used_data":1260268,"kb_used_omap":1,"kb_used_meta":594814,"kb_avail":91890052,"statfs":{"total":95995035648,"available":94095413248,"internally_reserved":0,"allocated":1290514432,"data_stored":1288534367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":609089994},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":206158430395,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692032,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2079340,"kb_used_data":1664992,"kb_used_omap":1,"kb_used_meta":414334,"kb_avail":91665812,"statfs":{"total":95995035648,"available":93865791488,"internally_reserved":0,"allocated":1704951808,"data_stored":1702975990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":424278469},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,3,0,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986373,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2050544,"kb_used_data":1482084,"kb_used_omap":1,"kb_used_meta":568446,"kb_avail":91694608,"statfs":{"total":95995035648,"available":93895278592,"internally_reserved":0,"allocated":1517654016,"data_stored":1515677503,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":582089157},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149899,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1039532,"kb_used_data":503012,"kb_used_omap":1,"kb_used_meta":536510,"kb_avail":92705620,"statfs":{"total":95995035648,"available":94930554880,"internally_reserved":0,"allocated":515084288,"data_stored":513107775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":549386695},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411536,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575060,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1621044,"kb_used_data":994284,"kb_used_omap":1,"kb_used_meta":626750,"kb_avail":92124108,"statfs":{"total":95995035648,"available":94335086592,"internally_reserved":0,"allocated":1018146816,"data_stored":1016166751,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":641792453},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738584,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2500628,"kb_used_data":2072396,"kb_used_omap":1,"kb_used_meta":428222,"kb_avail":91244524,"statfs":{"total":95995035648,"available":93434392576,"internally_reserved":0,"allocated":2122133504,"data_stored":2120165443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":438499781},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2164195328,"data_stored":2164195328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1032060928,"data_stored":1032060928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":515899392,"data_stored":515899392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1519845376,"data_stored":1519845376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1711276032,"data_stored":1711276032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1303904256,"data_stored":1303904256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1150681088,"data_stored":1150681088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1284833280,"data_stored":1284833280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1196687360,"data_stored":1196687360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T14:25:16.097 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-19T14:25:16.113 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:25:16.121 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:25:16.144 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:16.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:16.145 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:25:16.246 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:25:16.277 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:16.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:16.279 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:25:16.320 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 28 16 67588 67572 150.805 116.625 0.00427099 0.00662644 2024-08-19T14:25:16.367 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:25:16.368 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:25:16.382 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:25:16.414 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:16.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:16.415 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:25:16.517 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:25:16.548 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:16.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:16.549 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:25:16.650 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:25:16.684 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:16.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:16.685 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:25:16.744 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:25:16.786 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:25:16.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:16.808 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:16.809 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:25:16.910 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:25:16.935 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:16.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:16.936 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:25:17.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:16 smithi032 ceph-mon[20004]: from='client.16239 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:25:17.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:16 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:25:17.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:16 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:25:17.037 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:25:17.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:16 smithi045 ceph-mon[20143]: from='client.16239 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:25:17.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:16 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:25:17.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:16 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:25:17.067 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:17.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:17.068 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:25:17.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:16 smithi031 ceph-mon[16336]: from='client.16239 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:25:17.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:16 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:25:17.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:16 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:25:17.166 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-08-19T14:25:17.168 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:25:17.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:17.199 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:17.199 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:25:17.300 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:25:17.321 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 29 16 69229 69213 149.137 102.562 0.00699121 0.00669979 2024-08-19T14:25:17.340 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:17.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:17.341 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:25:17.442 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:25:17.454 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:25:17.484 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:17.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:17.485 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:25:17.580 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:25:17.586 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:25:17.617 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:17.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:17.618 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:25:17.719 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:25:17.721 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is set 2024-08-19T14:25:17.748 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:17.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:17.749 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:25:17.850 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:25:17.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:17.884 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:17.884 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:25:17.945 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:25:17.946 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:17 smithi031 ceph-mon[16336]: pgmap v994: 47 pgs: 47 active+clean; 3.7 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-08-19T14:25:17.946 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:17 smithi031 ceph-mon[16336]: osdmap e434: 12 total, 12 up, 9 in 2024-08-19T14:25:17.946 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:17 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2549815630' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:25:17.946 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:17 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:25:17.947 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-08-19T14:25:17.985 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:25:18.006 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:17 smithi045 ceph-mon[20143]: pgmap v994: 47 pgs: 47 active+clean; 3.7 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-08-19T14:25:18.007 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:17 smithi045 ceph-mon[20143]: osdmap e434: 12 total, 12 up, 9 in 2024-08-19T14:25:18.007 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:17 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2549815630' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:25:18.007 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:17 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:25:18.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:17 smithi032 ceph-mon[20004]: pgmap v994: 47 pgs: 47 active+clean; 3.7 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-08-19T14:25:18.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:17 smithi032 ceph-mon[20004]: osdmap e434: 12 total, 12 up, 9 in 2024-08-19T14:25:18.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:17 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2549815630' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:25:18.009 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:17 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:25:18.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:18.018 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:18.018 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:25:18.120 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:25:18.149 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:18.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:18.150 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:25:18.170 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:25:18.170 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:25:18.251 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:25:18.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:18.284 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:18.285 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:25:18.321 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 30 16 70999 70983 147.853 110.625 0.00627836 0.00675929 2024-08-19T14:25:18.385 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:25:18.417 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:18.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:18.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:25:18.519 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:25:18.555 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:18.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:18.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:25:18.657 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:25:18.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:18.691 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:18.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:25:18.734 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":996,"stamp":"2024-08-19T14:25:17.718425+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4306764320,"num_objects":65709,"num_object_clones":0,"num_object_copies":197127,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":65707,"num_write_kb":4205248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70016,"ondisk_log_size":70016,"up":141,"acting":141,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":142,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":17090932,"kb_used_data":12398300,"kb_used_omap":14,"kb_used_meta":4692529,"kb_avail":826615436,"statfs":{"total":863955320832,"available":846454206464,"internally_reserved":0,"allocated":12695859200,"data_stored":12678059927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14353,"internal_metadata":4805150703},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,5,4,4],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":942604288,"num_objects":14383,"num_object_clones":0,"num_object_copies":43149,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14383,"num_write_kb":920512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.038727"},"pg_stats":[{"pgid":"5.4","version":"434'4226","reported_seq":4266,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-08-19T14:25:16.767517+0000","last_change":"2024-08-19T14:24:44.595230+0000","last_active":"2024-08-19T14:25:16.767517+0000","last_peered":"2024-08-19T14:25:16.767517+0000","last_clean":"2024-08-19T14:25:16.767517+0000","last_became_active":"2024-08-19T14:24:44.594920+0000","last_became_peered":"2024-08-19T14:24:44.594920+0000","last_unstale":"2024-08-19T14:25:16.767517+0000","last_undegraded":"2024-08-19T14:25:16.767517+0000","last_fullsized":"2024-08-19T14:25:16.767517+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:24:43.574783+0000","objects_scrubbed":0,"log_size":4226,"log_dups_size":0,"ondisk_log_size":4226,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:13.412273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":276955136,"num_objects":4226,"num_object_clones":0,"num_object_copies":12678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4226,"num_write_kb":270464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,0],"acting":[5,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"70'139","reported_seq":1100,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-08-19T14:25:11.685461+0000","last_change":"2024-08-19T14:24:48.191443+0000","last_active":"2024-08-19T14:25:11.685461+0000","last_peered":"2024-08-19T14:25:11.685461+0000","last_clean":"2024-08-19T14:25:11.685461+0000","last_became_active":"2024-08-19T14:18:13.807133+0000","last_became_peered":"2024-08-19T14:18:13.807133+0000","last_unstale":"2024-08-19T14:25:11.685461+0000","last_undegraded":"2024-08-19T14:25:11.685461+0000","last_fullsized":"2024-08-19T14:25:11.685461+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:48.191339+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:19:58.072083+0000","last_clean_scrub_stamp":"2024-08-19T14:24:48.191339+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:08.680074+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,9],"acting":[7,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"434'4311","reported_seq":4351,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-08-19T14:25:16.738296+0000","last_change":"2024-08-19T14:24:44.593690+0000","last_active":"2024-08-19T14:25:16.738296+0000","last_peered":"2024-08-19T14:25:16.738296+0000","last_clean":"2024-08-19T14:25:16.738296+0000","last_became_active":"2024-08-19T14:24:44.593405+0000","last_became_peered":"2024-08-19T14:24:44.593405+0000","last_unstale":"2024-08-19T14:25:16.738296+0000","last_undegraded":"2024-08-19T14:25:16.738296+0000","last_fullsized":"2024-08-19T14:25:16.738296+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:24:43.574783+0000","objects_scrubbed":0,"log_size":4311,"log_dups_size":0,"ondisk_log_size":4311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:25:59.739209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282525696,"num_objects":4311,"num_object_clones":0,"num_object_copies":12933,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4311,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4311,"num_write_kb":275904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.3","version":"70'139","reported_seq":998,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-08-19T14:25:11.686477+0000","last_change":"2024-08-19T14:24:42.697899+0000","last_active":"2024-08-19T14:25:11.686477+0000","last_peered":"2024-08-19T14:25:11.686477+0000","last_clean":"2024-08-19T14:25:11.686477+0000","last_became_active":"2024-08-19T14:18:13.804721+0000","last_became_peered":"2024-08-19T14:18:13.804721+0000","last_unstale":"2024-08-19T14:25:11.686477+0000","last_undegraded":"2024-08-19T14:25:11.686477+0000","last_fullsized":"2024-08-19T14:25:11.686477+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:42.697801+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:24:42.697801+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:08.817098+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"434'4335","reported_seq":4375,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-08-19T14:25:16.744377+0000","last_change":"2024-08-19T14:24:44.593298+0000","last_active":"2024-08-19T14:25:16.744377+0000","last_peered":"2024-08-19T14:25:16.744377+0000","last_clean":"2024-08-19T14:25:16.744377+0000","last_became_active":"2024-08-19T14:24:44.593002+0000","last_became_peered":"2024-08-19T14:24:44.593002+0000","last_unstale":"2024-08-19T14:25:16.744377+0000","last_undegraded":"2024-08-19T14:25:16.744377+0000","last_fullsized":"2024-08-19T14:25:16.744377+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:24:43.574783+0000","objects_scrubbed":0,"log_size":4335,"log_dups_size":0,"ondisk_log_size":4335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:25:48.182254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284098560,"num_objects":4335,"num_object_clones":0,"num_object_copies":13005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4335,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4335,"num_write_kb":277440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,8],"acting":[5,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"70'139","reported_seq":947,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-08-19T14:25:11.685561+0000","last_change":"2024-08-19T14:24:43.246467+0000","last_active":"2024-08-19T14:25:11.685561+0000","last_peered":"2024-08-19T14:25:11.685561+0000","last_clean":"2024-08-19T14:25:11.685561+0000","last_became_active":"2024-08-19T14:20:50.742354+0000","last_became_peered":"2024-08-19T14:20:50.742354+0000","last_unstale":"2024-08-19T14:25:11.685561+0000","last_undegraded":"2024-08-19T14:25:11.685561+0000","last_fullsized":"2024-08-19T14:25:11.685561+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:43.246387+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:19:59.302931+0000","last_clean_scrub_stamp":"2024-08-19T14:24:43.246387+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:05.068356+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.5","version":"433'4146","reported_seq":4184,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-08-19T14:25:16.401527+0000","last_change":"2024-08-19T14:24:44.592723+0000","last_active":"2024-08-19T14:25:16.401527+0000","last_peered":"2024-08-19T14:25:16.401527+0000","last_clean":"2024-08-19T14:25:16.401527+0000","last_became_active":"2024-08-19T14:24:44.592424+0000","last_became_peered":"2024-08-19T14:24:44.592424+0000","last_unstale":"2024-08-19T14:25:16.401527+0000","last_undegraded":"2024-08-19T14:25:16.401527+0000","last_fullsized":"2024-08-19T14:25:16.401527+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:24:43.574783+0000","objects_scrubbed":0,"log_size":4146,"log_dups_size":0,"ondisk_log_size":4146,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:03.869565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":271712256,"num_objects":4146,"num_object_clones":0,"num_object_copies":12438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4146,"num_write_kb":265344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"70'139","reported_seq":995,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-08-19T14:25:11.686379+0000","last_change":"2024-08-19T14:24:48.179646+0000","last_active":"2024-08-19T14:25:11.686379+0000","last_peered":"2024-08-19T14:25:11.686379+0000","last_clean":"2024-08-19T14:25:11.686379+0000","last_became_active":"2024-08-19T14:18:13.808505+0000","last_became_peered":"2024-08-19T14:18:13.808505+0000","last_unstale":"2024-08-19T14:25:11.686379+0000","last_undegraded":"2024-08-19T14:25:11.686379+0000","last_fullsized":"2024-08-19T14:25:11.686379+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:48.179558+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:24:48.179558+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:25:51.185230+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,11],"acting":[9,0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.3","version":"433'4110","reported_seq":4148,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-08-19T14:25:16.410243+0000","last_change":"2024-08-19T14:24:44.659231+0000","last_active":"2024-08-19T14:25:16.410243+0000","last_peered":"2024-08-19T14:25:16.410243+0000","last_clean":"2024-08-19T14:25:16.410243+0000","last_became_active":"2024-08-19T14:24:44.658960+0000","last_became_peered":"2024-08-19T14:24:44.658960+0000","last_unstale":"2024-08-19T14:25:16.410243+0000","last_undegraded":"2024-08-19T14:25:16.410243+0000","last_fullsized":"2024-08-19T14:25:16.410243+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:24:43.574783+0000","objects_scrubbed":0,"log_size":4110,"log_dups_size":0,"ondisk_log_size":4110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:02.684044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269352960,"num_objects":4110,"num_object_clones":0,"num_object_copies":12330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4110,"num_write_kb":263040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"70'139","reported_seq":647,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-08-19T14:25:11.691507+0000","last_change":"2024-08-19T14:24:47.412400+0000","last_active":"2024-08-19T14:25:11.691507+0000","last_peered":"2024-08-19T14:25:11.691507+0000","last_clean":"2024-08-19T14:25:11.691507+0000","last_became_active":"2024-08-19T14:20:28.204384+0000","last_became_peered":"2024-08-19T14:20:28.204384+0000","last_unstale":"2024-08-19T14:25:11.691507+0000","last_undegraded":"2024-08-19T14:25:11.691507+0000","last_fullsized":"2024-08-19T14:25:11.691507+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:47.412291+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:21:46.057624+0000","last_clean_scrub_stamp":"2024-08-19T14:24:47.412291+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:11.449015+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"433'4185","reported_seq":4223,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-08-19T14:25:16.408128+0000","last_change":"2024-08-19T14:24:44.592960+0000","last_active":"2024-08-19T14:25:16.408128+0000","last_peered":"2024-08-19T14:25:16.408128+0000","last_clean":"2024-08-19T14:25:16.408128+0000","last_became_active":"2024-08-19T14:24:44.592670+0000","last_became_peered":"2024-08-19T14:24:44.592670+0000","last_unstale":"2024-08-19T14:25:16.408128+0000","last_undegraded":"2024-08-19T14:25:16.408128+0000","last_fullsized":"2024-08-19T14:25:16.408128+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:24:43.574783+0000","objects_scrubbed":0,"log_size":4185,"log_dups_size":0,"ondisk_log_size":4185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:07.754236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":274268160,"num_objects":4185,"num_object_clones":0,"num_object_copies":12555,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4185,"num_write_kb":267840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,8],"acting":[0,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"70'139","reported_seq":999,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-08-19T14:25:11.686524+0000","last_change":"2024-08-19T14:24:46.222297+0000","last_active":"2024-08-19T14:25:11.686524+0000","last_peered":"2024-08-19T14:25:11.686524+0000","last_clean":"2024-08-19T14:25:11.686524+0000","last_became_active":"2024-08-19T14:18:26.874902+0000","last_became_peered":"2024-08-19T14:18:26.874902+0000","last_unstale":"2024-08-19T14:25:11.686524+0000","last_undegraded":"2024-08-19T14:25:11.686524+0000","last_fullsized":"2024-08-19T14:25:11.686524+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:46.222166+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:27.848870+0000","last_clean_scrub_stamp":"2024-08-19T14:24:46.222166+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:09.431052+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,4],"acting":[9,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.1","version":"433'3926","reported_seq":3964,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-08-19T14:25:12.734556+0000","last_change":"2024-08-19T14:24:44.658485+0000","last_active":"2024-08-19T14:25:12.734556+0000","last_peered":"2024-08-19T14:25:12.734556+0000","last_clean":"2024-08-19T14:25:12.734556+0000","last_became_active":"2024-08-19T14:24:44.657571+0000","last_became_peered":"2024-08-19T14:24:44.657571+0000","last_unstale":"2024-08-19T14:25:12.734556+0000","last_undegraded":"2024-08-19T14:25:12.734556+0000","last_fullsized":"2024-08-19T14:25:12.734556+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:24:43.574783+0000","objects_scrubbed":0,"log_size":3926,"log_dups_size":0,"ondisk_log_size":3926,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:25:50.673564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":257294336,"num_objects":3926,"num_object_clones":0,"num_object_copies":11778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3926,"num_write_kb":251264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"70'139","reported_seq":821,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-08-19T14:25:11.685722+0000","last_change":"2024-08-19T14:24:46.629045+0000","last_active":"2024-08-19T14:25:11.685722+0000","last_peered":"2024-08-19T14:25:11.685722+0000","last_clean":"2024-08-19T14:25:11.685722+0000","last_became_active":"2024-08-19T14:18:26.881435+0000","last_became_peered":"2024-08-19T14:18:26.881435+0000","last_unstale":"2024-08-19T14:25:11.685722+0000","last_undegraded":"2024-08-19T14:25:11.685722+0000","last_fullsized":"2024-08-19T14:25:11.685722+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:46.628960+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:24:46.628960+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:03.429914+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,4],"acting":[11,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.0","version":"433'3974","reported_seq":4012,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-08-19T14:25:13.406556+0000","last_change":"2024-08-19T14:24:44.595889+0000","last_active":"2024-08-19T14:25:13.406556+0000","last_peered":"2024-08-19T14:25:13.406556+0000","last_clean":"2024-08-19T14:25:13.406556+0000","last_became_active":"2024-08-19T14:24:44.595535+0000","last_became_peered":"2024-08-19T14:24:44.595535+0000","last_unstale":"2024-08-19T14:25:13.406556+0000","last_undegraded":"2024-08-19T14:25:13.406556+0000","last_fullsized":"2024-08-19T14:25:13.406556+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:24:43.574783+0000","objects_scrubbed":0,"log_size":3974,"log_dups_size":0,"ondisk_log_size":3974,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:07.707608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260440064,"num_objects":3974,"num_object_clones":0,"num_object_copies":11922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3974,"num_write_kb":254336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"70'139","reported_seq":1045,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-08-19T14:25:11.685536+0000","last_change":"2024-08-19T14:24:11.345984+0000","last_active":"2024-08-19T14:25:11.685536+0000","last_peered":"2024-08-19T14:25:11.685536+0000","last_clean":"2024-08-19T14:25:11.685536+0000","last_became_active":"2024-08-19T14:18:13.806013+0000","last_became_peered":"2024-08-19T14:18:13.806013+0000","last_unstale":"2024-08-19T14:25:11.685536+0000","last_undegraded":"2024-08-19T14:25:11.685536+0000","last_fullsized":"2024-08-19T14:25:11.685536+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:11.345873+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:24:11.345873+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:25:14.696871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.e","version":"433'3895","reported_seq":3933,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-08-19T14:25:13.518761+0000","last_change":"2024-08-19T14:24:44.595518+0000","last_active":"2024-08-19T14:25:13.518761+0000","last_peered":"2024-08-19T14:25:13.518761+0000","last_clean":"2024-08-19T14:25:13.518761+0000","last_became_active":"2024-08-19T14:24:44.595166+0000","last_became_peered":"2024-08-19T14:24:44.595166+0000","last_unstale":"2024-08-19T14:25:13.518761+0000","last_undegraded":"2024-08-19T14:25:13.518761+0000","last_fullsized":"2024-08-19T14:25:13.518761+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:24:43.574783+0000","objects_scrubbed":0,"log_size":3895,"log_dups_size":0,"ondisk_log_size":3895,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:09.480899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":255262720,"num_objects":3895,"num_object_clones":0,"num_object_copies":11685,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3895,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3895,"num_write_kb":249280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,0],"acting":[9,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.a","version":"70'139","reported_seq":1007,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-08-19T14:25:11.686319+0000","last_change":"2024-08-19T14:24:50.196984+0000","last_active":"2024-08-19T14:25:11.686319+0000","last_peered":"2024-08-19T14:25:11.686319+0000","last_clean":"2024-08-19T14:25:11.686319+0000","last_became_active":"2024-08-19T14:18:26.875006+0000","last_became_peered":"2024-08-19T14:18:26.875006+0000","last_unstale":"2024-08-19T14:25:11.686319+0000","last_undegraded":"2024-08-19T14:25:11.686319+0000","last_fullsized":"2024-08-19T14:25:11.686319+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:50.196885+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:28.864870+0000","last_clean_scrub_stamp":"2024-08-19T14:24:50.196885+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:15.234239+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,0],"acting":[9,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.d","version":"433'4272","reported_seq":4310,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-08-19T14:25:16.077746+0000","last_change":"2024-08-19T14:24:44.592424+0000","last_active":"2024-08-19T14:25:16.077746+0000","last_peered":"2024-08-19T14:25:16.077746+0000","last_clean":"2024-08-19T14:25:16.077746+0000","last_became_active":"2024-08-19T14:24:44.592098+0000","last_became_peered":"2024-08-19T14:24:44.592098+0000","last_unstale":"2024-08-19T14:25:16.077746+0000","last_undegraded":"2024-08-19T14:25:16.077746+0000","last_fullsized":"2024-08-19T14:25:16.077746+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:24:43.574783+0000","objects_scrubbed":0,"log_size":4272,"log_dups_size":0,"ondisk_log_size":4272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:25:54.822583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":279969792,"num_objects":4272,"num_object_clones":0,"num_object_copies":12816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4272,"num_write_kb":273408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7],"acting":[11,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.9","version":"70'139","reported_seq":1122,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-08-19T14:25:11.685630+0000","last_change":"2024-08-19T14:25:04.127400+0000","last_active":"2024-08-19T14:25:11.685630+0000","last_peered":"2024-08-19T14:25:11.685630+0000","last_clean":"2024-08-19T14:25:11.685630+0000","last_became_active":"2024-08-19T14:18:26.887657+0000","last_became_peered":"2024-08-19T14:18:26.887657+0000","last_unstale":"2024-08-19T14:25:11.685630+0000","last_undegraded":"2024-08-19T14:25:11.685630+0000","last_fullsized":"2024-08-19T14:25:11.685630+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:25:04.127296+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:25:04.127296+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:12.377922+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"433'3979","reported_seq":4016,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-08-19T14:25:13.414373+0000","last_change":"2024-08-19T14:24:44.595877+0000","last_active":"2024-08-19T14:25:13.414373+0000","last_peered":"2024-08-19T14:25:13.414373+0000","last_clean":"2024-08-19T14:25:13.414373+0000","last_became_active":"2024-08-19T14:24:44.595538+0000","last_became_peered":"2024-08-19T14:24:44.595538+0000","last_unstale":"2024-08-19T14:25:13.414373+0000","last_undegraded":"2024-08-19T14:25:13.414373+0000","last_fullsized":"2024-08-19T14:25:13.414373+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:24:43.574783+0000","objects_scrubbed":0,"log_size":3979,"log_dups_size":0,"ondisk_log_size":3979,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:25:46.650773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260767744,"num_objects":3979,"num_object_clones":0,"num_object_copies":11937,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3979,"num_write_kb":254656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"70'139","reported_seq":1021,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-08-19T14:25:16.712718+0000","last_change":"2024-08-19T14:24:45.176469+0000","last_active":"2024-08-19T14:25:16.712718+0000","last_peered":"2024-08-19T14:25:16.712718+0000","last_clean":"2024-08-19T14:25:16.712718+0000","last_became_active":"2024-08-19T14:18:26.887277+0000","last_became_peered":"2024-08-19T14:18:26.887277+0000","last_unstale":"2024-08-19T14:25:16.712718+0000","last_undegraded":"2024-08-19T14:25:16.712718+0000","last_fullsized":"2024-08-19T14:25:16.712718+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:45.176400+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:24:45.176400+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:12.727044+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,8],"acting":[5,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.f","version":"434'4211","reported_seq":4251,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-08-19T14:25:16.764950+0000","last_change":"2024-08-19T14:24:44.593761+0000","last_active":"2024-08-19T14:25:16.764950+0000","last_peered":"2024-08-19T14:25:16.764950+0000","last_clean":"2024-08-19T14:25:16.764950+0000","last_became_active":"2024-08-19T14:24:44.593498+0000","last_became_peered":"2024-08-19T14:24:44.593498+0000","last_unstale":"2024-08-19T14:25:16.764950+0000","last_undegraded":"2024-08-19T14:25:16.764950+0000","last_fullsized":"2024-08-19T14:25:16.764950+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:24:43.574783+0000","objects_scrubbed":0,"log_size":4211,"log_dups_size":0,"ondisk_log_size":4211,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:25:50.363040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":275972096,"num_objects":4211,"num_object_clones":0,"num_object_copies":12633,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4211,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4211,"num_write_kb":269504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,11],"acting":[5,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"70'139","reported_seq":1021,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-08-19T14:25:11.685580+0000","last_change":"2024-08-19T14:24:45.610330+0000","last_active":"2024-08-19T14:25:11.685580+0000","last_peered":"2024-08-19T14:25:11.685580+0000","last_clean":"2024-08-19T14:25:11.685580+0000","last_became_active":"2024-08-19T14:18:26.875179+0000","last_became_peered":"2024-08-19T14:18:26.875179+0000","last_unstale":"2024-08-19T14:25:11.685580+0000","last_undegraded":"2024-08-19T14:25:11.685580+0000","last_fullsized":"2024-08-19T14:25:11.685580+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:45.610239+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:24:45.610239+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:15.465099+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.a","version":"433'4085","reported_seq":4123,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-08-19T14:25:14.979808+0000","last_change":"2024-08-19T14:24:44.592347+0000","last_active":"2024-08-19T14:25:14.979808+0000","last_peered":"2024-08-19T14:25:14.979808+0000","last_clean":"2024-08-19T14:25:14.979808+0000","last_became_active":"2024-08-19T14:24:44.592024+0000","last_became_peered":"2024-08-19T14:24:44.592024+0000","last_unstale":"2024-08-19T14:25:14.979808+0000","last_undegraded":"2024-08-19T14:25:14.979808+0000","last_fullsized":"2024-08-19T14:25:14.979808+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:24:43.574783+0000","objects_scrubbed":0,"log_size":4085,"log_dups_size":0,"ondisk_log_size":4085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:06.455975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":267714560,"num_objects":4085,"num_object_clones":0,"num_object_copies":12255,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4085,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4085,"num_write_kb":261440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,11],"acting":[8,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.e","version":"70'139","reported_seq":675,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-08-19T14:25:11.689090+0000","last_change":"2024-08-19T14:25:04.737969+0000","last_active":"2024-08-19T14:25:11.689090+0000","last_peered":"2024-08-19T14:25:11.689090+0000","last_clean":"2024-08-19T14:25:11.689090+0000","last_became_active":"2024-08-19T14:18:26.884105+0000","last_became_peered":"2024-08-19T14:18:26.884105+0000","last_unstale":"2024-08-19T14:25:11.689090+0000","last_undegraded":"2024-08-19T14:25:11.689090+0000","last_fullsized":"2024-08-19T14:25:11.689090+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:25:04.737897+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:20:15.321945+0000","last_clean_scrub_stamp":"2024-08-19T14:25:04.737897+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:25.666670+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.b","version":"433'4007","reported_seq":4044,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-08-19T14:25:13.523831+0000","last_change":"2024-08-19T14:24:44.595612+0000","last_active":"2024-08-19T14:25:13.523831+0000","last_peered":"2024-08-19T14:25:13.523831+0000","last_clean":"2024-08-19T14:25:13.523831+0000","last_became_active":"2024-08-19T14:24:44.595175+0000","last_became_peered":"2024-08-19T14:24:44.595175+0000","last_unstale":"2024-08-19T14:25:13.523831+0000","last_undegraded":"2024-08-19T14:25:13.523831+0000","last_fullsized":"2024-08-19T14:25:13.523831+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:24:43.574783+0000","objects_scrubbed":0,"log_size":4007,"log_dups_size":0,"ondisk_log_size":4007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:25:55.419449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262602752,"num_objects":4007,"num_object_clones":0,"num_object_copies":12021,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4007,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4007,"num_write_kb":256448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,5],"acting":[9,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.f","version":"70'139","reported_seq":921,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-08-19T14:25:16.712861+0000","last_change":"2024-08-19T14:24:16.076239+0000","last_active":"2024-08-19T14:25:16.712861+0000","last_peered":"2024-08-19T14:25:16.712861+0000","last_clean":"2024-08-19T14:25:16.712861+0000","last_became_active":"2024-08-19T14:18:18.711708+0000","last_became_peered":"2024-08-19T14:18:18.711708+0000","last_unstale":"2024-08-19T14:25:16.712861+0000","last_undegraded":"2024-08-19T14:25:16.712861+0000","last_fullsized":"2024-08-19T14:25:16.712861+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:16.076143+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:24:16.076143+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:25:32.997797+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.8","version":"433'4138","reported_seq":4176,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-08-19T14:25:16.080078+0000","last_change":"2024-08-19T14:24:44.591109+0000","last_active":"2024-08-19T14:25:16.080078+0000","last_peered":"2024-08-19T14:25:16.080078+0000","last_clean":"2024-08-19T14:25:16.080078+0000","last_became_active":"2024-08-19T14:24:44.590679+0000","last_became_peered":"2024-08-19T14:24:44.590679+0000","last_unstale":"2024-08-19T14:25:16.080078+0000","last_undegraded":"2024-08-19T14:25:16.080078+0000","last_fullsized":"2024-08-19T14:25:16.080078+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:24:43.574783+0000","objects_scrubbed":0,"log_size":4138,"log_dups_size":0,"ondisk_log_size":4138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:25:52.297517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":271187968,"num_objects":4138,"num_object_clones":0,"num_object_copies":12414,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4138,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4138,"num_write_kb":264832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.c","version":"70'139","reported_seq":780,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-08-19T14:25:11.691228+0000","last_change":"2024-08-19T14:24:13.465173+0000","last_active":"2024-08-19T14:25:11.691228+0000","last_peered":"2024-08-19T14:25:11.691228+0000","last_clean":"2024-08-19T14:25:11.691228+0000","last_became_active":"2024-08-19T14:18:26.872942+0000","last_became_peered":"2024-08-19T14:18:26.872942+0000","last_unstale":"2024-08-19T14:25:11.691228+0000","last_undegraded":"2024-08-19T14:25:11.691228+0000","last_fullsized":"2024-08-19T14:25:11.691228+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:13.463716+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:22:52.164586+0000","last_clean_scrub_stamp":"2024-08-19T14:24:13.463716+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:25:23.123046+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,9],"acting":[3,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"433'3907","reported_seq":3945,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-08-19T14:25:13.602713+0000","last_change":"2024-08-19T14:24:44.592293+0000","last_active":"2024-08-19T14:25:13.602713+0000","last_peered":"2024-08-19T14:25:13.602713+0000","last_clean":"2024-08-19T14:25:13.602713+0000","last_became_active":"2024-08-19T14:24:44.591973+0000","last_became_peered":"2024-08-19T14:24:44.591973+0000","last_unstale":"2024-08-19T14:25:13.602713+0000","last_undegraded":"2024-08-19T14:25:13.602713+0000","last_fullsized":"2024-08-19T14:25:13.602713+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:24:43.574783+0000","objects_scrubbed":0,"log_size":3907,"log_dups_size":0,"ondisk_log_size":3907,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:25:54.081269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256049152,"num_objects":3907,"num_object_clones":0,"num_object_copies":11721,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3907,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3907,"num_write_kb":250048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,9],"acting":[7,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"70'139","reported_seq":695,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-08-19T14:25:11.685866+0000","last_change":"2024-08-19T14:24:50.276533+0000","last_active":"2024-08-19T14:25:11.685866+0000","last_peered":"2024-08-19T14:25:11.685866+0000","last_clean":"2024-08-19T14:25:11.685866+0000","last_became_active":"2024-08-19T14:18:26.882002+0000","last_became_peered":"2024-08-19T14:18:26.882002+0000","last_unstale":"2024-08-19T14:25:11.685866+0000","last_undegraded":"2024-08-19T14:25:11.685866+0000","last_fullsized":"2024-08-19T14:25:11.685866+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:50.276408+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:24:50.276408+0000","last_clean_scrub_stamp":"2024-08-19T14:24:50.276408+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:06.161471+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"70'139","reported_seq":955,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-08-19T14:25:16.712547+0000","last_change":"2024-08-19T14:24:14.035778+0000","last_active":"2024-08-19T14:25:16.712547+0000","last_peered":"2024-08-19T14:25:16.712547+0000","last_clean":"2024-08-19T14:25:16.712547+0000","last_became_active":"2024-08-19T14:18:18.703594+0000","last_became_peered":"2024-08-19T14:18:18.703594+0000","last_unstale":"2024-08-19T14:25:16.712547+0000","last_undegraded":"2024-08-19T14:25:16.712547+0000","last_fullsized":"2024-08-19T14:25:16.712547+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:14.035665+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:24:14.035665+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:25:35.876300+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"70'139","reported_seq":1005,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-08-19T14:25:11.686366+0000","last_change":"2024-08-19T14:25:04.742939+0000","last_active":"2024-08-19T14:25:11.686366+0000","last_peered":"2024-08-19T14:25:11.686366+0000","last_clean":"2024-08-19T14:25:11.686366+0000","last_became_active":"2024-08-19T14:18:18.698334+0000","last_became_peered":"2024-08-19T14:18:18.698334+0000","last_unstale":"2024-08-19T14:25:11.686366+0000","last_undegraded":"2024-08-19T14:25:11.686366+0000","last_fullsized":"2024-08-19T14:25:11.686366+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:25:04.742869+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:25:04.742869+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:34.199161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.10","version":"70'139","reported_seq":1118,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-08-19T14:25:16.712613+0000","last_change":"2024-08-19T14:25:06.143718+0000","last_active":"2024-08-19T14:25:16.712613+0000","last_peered":"2024-08-19T14:25:16.712613+0000","last_clean":"2024-08-19T14:25:16.712613+0000","last_became_active":"2024-08-19T14:18:18.704562+0000","last_became_peered":"2024-08-19T14:18:18.704562+0000","last_unstale":"2024-08-19T14:25:16.712613+0000","last_undegraded":"2024-08-19T14:25:16.712613+0000","last_fullsized":"2024-08-19T14:25:16.712613+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:25:06.143606+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:25:06.143606+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:16.549538+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,8],"acting":[5,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"70'139","reported_seq":666,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-08-19T14:25:11.687064+0000","last_change":"2024-08-19T14:24:50.547549+0000","last_active":"2024-08-19T14:25:11.687064+0000","last_peered":"2024-08-19T14:25:11.687064+0000","last_clean":"2024-08-19T14:25:11.687064+0000","last_became_active":"2024-08-19T14:18:18.697894+0000","last_became_peered":"2024-08-19T14:18:18.697894+0000","last_unstale":"2024-08-19T14:25:11.687064+0000","last_undegraded":"2024-08-19T14:25:11.687064+0000","last_fullsized":"2024-08-19T14:25:11.687064+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:50.547428+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:24:50.547428+0000","last_clean_scrub_stamp":"2024-08-19T14:24:50.547428+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:14.355547+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"70'139","reported_seq":634,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-08-19T14:25:11.686598+0000","last_change":"2024-08-19T14:24:41.729459+0000","last_active":"2024-08-19T14:25:11.686598+0000","last_peered":"2024-08-19T14:25:11.686598+0000","last_clean":"2024-08-19T14:25:11.686598+0000","last_became_active":"2024-08-19T14:20:28.204724+0000","last_became_peered":"2024-08-19T14:20:28.204724+0000","last_unstale":"2024-08-19T14:25:11.686598+0000","last_undegraded":"2024-08-19T14:25:11.686598+0000","last_fullsized":"2024-08-19T14:25:11.686598+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:41.729377+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:24:41.729377+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:09.621000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"70'139","reported_seq":762,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-08-19T14:25:11.692945+0000","last_change":"2024-08-19T14:24:48.428636+0000","last_active":"2024-08-19T14:25:11.692945+0000","last_peered":"2024-08-19T14:25:11.692945+0000","last_clean":"2024-08-19T14:25:11.692945+0000","last_became_active":"2024-08-19T14:18:31.901556+0000","last_became_peered":"2024-08-19T14:18:31.901556+0000","last_unstale":"2024-08-19T14:25:11.692945+0000","last_undegraded":"2024-08-19T14:25:11.692945+0000","last_fullsized":"2024-08-19T14:25:11.692945+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:48.428515+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:24:48.428515+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:13.841914+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,0],"acting":[8,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.14","version":"70'139","reported_seq":763,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-08-19T14:25:11.685632+0000","last_change":"2024-08-19T14:24:23.204225+0000","last_active":"2024-08-19T14:25:11.685632+0000","last_peered":"2024-08-19T14:25:11.685632+0000","last_clean":"2024-08-19T14:25:11.685632+0000","last_became_active":"2024-08-19T14:18:18.703926+0000","last_became_peered":"2024-08-19T14:18:18.703926+0000","last_unstale":"2024-08-19T14:25:11.685632+0000","last_undegraded":"2024-08-19T14:25:11.685632+0000","last_fullsized":"2024-08-19T14:25:11.685632+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:23.204112+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:24:23.204112+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:25:30.300500+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,1],"acting":[7,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.15","version":"70'139","reported_seq":650,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-08-19T14:25:11.685799+0000","last_change":"2024-08-19T14:24:44.255453+0000","last_active":"2024-08-19T14:25:11.685799+0000","last_peered":"2024-08-19T14:25:11.685799+0000","last_clean":"2024-08-19T14:25:11.685799+0000","last_became_active":"2024-08-19T14:18:31.912496+0000","last_became_peered":"2024-08-19T14:18:31.912496+0000","last_unstale":"2024-08-19T14:25:11.685799+0000","last_undegraded":"2024-08-19T14:25:11.685799+0000","last_fullsized":"2024-08-19T14:25:11.685799+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:44.255304+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:20:09.180923+0000","last_clean_scrub_stamp":"2024-08-19T14:24:44.255304+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:25:54.443311+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"70'139","reported_seq":636,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-08-19T14:25:11.688949+0000","last_change":"2024-08-19T14:24:44.775890+0000","last_active":"2024-08-19T14:25:11.688949+0000","last_peered":"2024-08-19T14:25:11.688949+0000","last_clean":"2024-08-19T14:25:11.688949+0000","last_became_active":"2024-08-19T14:18:31.901229+0000","last_became_peered":"2024-08-19T14:18:31.901229+0000","last_unstale":"2024-08-19T14:25:11.688949+0000","last_undegraded":"2024-08-19T14:25:11.688949+0000","last_fullsized":"2024-08-19T14:25:11.688949+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:44.775817+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:24:44.775817+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:25:48.718462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1b","version":"70'139","reported_seq":988,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-08-19T14:25:11.692558+0000","last_change":"2024-08-19T14:24:47.410923+0000","last_active":"2024-08-19T14:25:11.692558+0000","last_peered":"2024-08-19T14:25:11.692558+0000","last_clean":"2024-08-19T14:25:11.692558+0000","last_became_active":"2024-08-19T14:18:31.901152+0000","last_became_peered":"2024-08-19T14:18:31.901152+0000","last_unstale":"2024-08-19T14:25:11.692558+0000","last_undegraded":"2024-08-19T14:25:11.692558+0000","last_fullsized":"2024-08-19T14:25:11.692558+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:47.410831+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:24:47.410831+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:07.850070+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.18","version":"70'139","reported_seq":1034,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-08-19T14:25:16.712681+0000","last_change":"2024-08-19T14:24:40.113681+0000","last_active":"2024-08-19T14:25:16.712681+0000","last_peered":"2024-08-19T14:25:16.712681+0000","last_clean":"2024-08-19T14:25:16.712681+0000","last_became_active":"2024-08-19T14:18:31.902735+0000","last_became_peered":"2024-08-19T14:18:31.902735+0000","last_unstale":"2024-08-19T14:25:16.712681+0000","last_undegraded":"2024-08-19T14:25:16.712681+0000","last_fullsized":"2024-08-19T14:25:16.712681+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:40.113600+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:24:40.113600+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:25:58.753284+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.19","version":"70'139","reported_seq":980,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-08-19T14:25:11.686971+0000","last_change":"2024-08-19T14:24:42.577775+0000","last_active":"2024-08-19T14:25:11.686971+0000","last_peered":"2024-08-19T14:25:11.686971+0000","last_clean":"2024-08-19T14:25:11.686971+0000","last_became_active":"2024-08-19T14:18:31.902547+0000","last_became_peered":"2024-08-19T14:18:31.902547+0000","last_unstale":"2024-08-19T14:25:11.686971+0000","last_undegraded":"2024-08-19T14:25:11.686971+0000","last_fullsized":"2024-08-19T14:25:11.686971+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:42.577693+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:24:42.577693+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:25:43.680823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1e","version":"70'139","reported_seq":639,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-08-19T14:25:11.685800+0000","last_change":"2024-08-19T14:24:44.651398+0000","last_active":"2024-08-19T14:25:11.685800+0000","last_peered":"2024-08-19T14:25:11.685800+0000","last_clean":"2024-08-19T14:25:11.685800+0000","last_became_active":"2024-08-19T14:18:31.902546+0000","last_became_peered":"2024-08-19T14:18:31.902546+0000","last_unstale":"2024-08-19T14:25:11.685800+0000","last_undegraded":"2024-08-19T14:25:11.685800+0000","last_fullsized":"2024-08-19T14:25:11.685800+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:44.651299+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:24:44.651299+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:13.740488+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1c","version":"70'139","reported_seq":746,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-08-19T14:25:11.692547+0000","last_change":"2024-08-19T14:24:14.461312+0000","last_active":"2024-08-19T14:25:11.692547+0000","last_peered":"2024-08-19T14:25:11.692547+0000","last_clean":"2024-08-19T14:25:11.692547+0000","last_became_active":"2024-08-19T14:18:31.902494+0000","last_became_peered":"2024-08-19T14:18:31.902494+0000","last_unstale":"2024-08-19T14:25:11.692547+0000","last_undegraded":"2024-08-19T14:25:11.692547+0000","last_fullsized":"2024-08-19T14:25:11.692547+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:14.461176+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:24:14.461176+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:25:26.070165+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,11],"acting":[8,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1d","version":"70'139","reported_seq":660,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-08-19T14:25:11.686540+0000","last_change":"2024-08-19T14:24:22.588391+0000","last_active":"2024-08-19T14:25:11.686540+0000","last_peered":"2024-08-19T14:25:11.686540+0000","last_clean":"2024-08-19T14:25:11.686540+0000","last_became_active":"2024-08-19T14:18:31.901183+0000","last_became_peered":"2024-08-19T14:18:31.901183+0000","last_unstale":"2024-08-19T14:25:11.686540+0000","last_undegraded":"2024-08-19T14:25:11.686540+0000","last_fullsized":"2024-08-19T14:25:11.686540+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:22.588266+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:24:22.588266+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:25:26.637645+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,5],"acting":[1,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":4306173952,"num_objects":65707,"num_object_clones":0,"num_object_copies":197121,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":65707,"num_write_kb":4205248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12808880128,"data_stored":12808880128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65707,"ondisk_log_size":65707,"up":48,"acting":48,"num_store_stats":9},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1802240,"data_stored":1788032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187623,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1850796,"kb_used_data":1324004,"kb_used_omap":1,"kb_used_meta":526782,"kb_avail":91894356,"statfs":{"total":95995035648,"available":94099820544,"internally_reserved":0,"allocated":1355780096,"data_stored":1353803583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":539425220},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":343597383853,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":74,"seq":317827580080,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1806000,"kb_used_data":1251428,"kb_used_omap":1,"kb_used_meta":554558,"kb_avail":91939152,"statfs":{"total":95995035648,"available":94145691648,"internally_reserved":0,"allocated":1281462272,"data_stored":1279485759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":567867841},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874422,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1695852,"kb_used_data":1289828,"kb_used_omap":1,"kb_used_meta":406014,"kb_avail":92049300,"statfs":{"total":95995035648,"available":94258483200,"internally_reserved":0,"allocated":1320783872,"data_stored":1318807359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":415758789},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":57,"seq":244813136057,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1855100,"kb_used_data":1260268,"kb_used_omap":1,"kb_used_meta":594814,"kb_avail":91890052,"statfs":{"total":95995035648,"available":94095413248,"internally_reserved":0,"allocated":1290514432,"data_stored":1288534367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":609089994},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":206158430396,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692033,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2332148,"kb_used_data":1899756,"kb_used_omap":1,"kb_used_meta":432382,"kb_avail":91413004,"statfs":{"total":95995035648,"available":93606916096,"internally_reserved":0,"allocated":1945350144,"data_stored":1943370079,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":442759621},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986373,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2050544,"kb_used_data":1482084,"kb_used_omap":1,"kb_used_meta":568446,"kb_avail":91694608,"statfs":{"total":95995035648,"available":93895278592,"internally_reserved":0,"allocated":1517654016,"data_stored":1515677503,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":582089157},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149899,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1039532,"kb_used_data":503012,"kb_used_omap":1,"kb_used_meta":536510,"kb_avail":92705620,"statfs":{"total":95995035648,"available":94930554880,"internally_reserved":0,"allocated":515084288,"data_stored":513107775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":549386695},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411537,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575060,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1621044,"kb_used_data":994284,"kb_used_omap":1,"kb_used_meta":626750,"kb_avail":92124108,"statfs":{"total":95995035648,"available":94335086592,"internally_reserved":0,"allocated":1018146816,"data_stored":1016166751,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":641792453},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738585,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2839916,"kb_used_data":2393636,"kb_used_omap":1,"kb_used_meta":446270,"kb_avail":90905236,"statfs":{"total":95995035648,"available":93086961664,"internally_reserved":0,"allocated":2451083264,"data_stored":2449106751,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":456980933},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2485911552,"data_stored":2485911552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1032060928,"data_stored":1032060928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":515899392,"data_stored":515899392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1519845376,"data_stored":1519845376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1964179456,"data_stored":1964179456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1303904256,"data_stored":1303904256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1329659904,"data_stored":1329659904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1284833280,"data_stored":1284833280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1372585984,"data_stored":1372585984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T14:25:18.737 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-08-19T14:25:18.737 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-19T14:25:18.738 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_3 2024-08-19T14:25:18.738 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-19T14:25:18.744 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:25:18.791 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:25:18.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:18.838 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:18.839 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:25:18.849 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-08-19T14:25:18.939 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:25:18.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:18.979 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:18.980 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:25:19.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:18 smithi032 ceph-mon[20004]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:25:19.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:18 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:25:19.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:18 smithi032 ceph-mon[20004]: osdmap e435: 12 total, 12 up, 9 in 2024-08-19T14:25:19.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:18 smithi045 ceph-mon[20143]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:25:19.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:18 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:25:19.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:18 smithi045 ceph-mon[20143]: osdmap e435: 12 total, 12 up, 9 in 2024-08-19T14:25:19.080 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:25:19.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:19.116 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:19.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:25:19.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:18 smithi031 ceph-mon[16336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:25:19.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:18 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:25:19.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:18 smithi031 ceph-mon[16336]: osdmap e435: 12 total, 12 up, 9 in 2024-08-19T14:25:19.217 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:25:19.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:19.255 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:19.255 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:25:19.275 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:25:19.322 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 31 16 72888 72872 146.892 118.062 0.00634297 0.00680221 2024-08-19T14:25:19.356 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:25:19.383 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:19.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:19.384 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:25:19.486 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:25:19.515 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:19.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:19.516 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:25:19.617 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:25:19.648 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:19.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:19.649 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:25:19.725 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:19 smithi031 ceph-mon[16336]: from='client.25888 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:25:19.726 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:19 smithi031 ceph-mon[16336]: pgmap v997: 47 pgs: 47 active+clean; 4.0 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 118 MiB/s wr, 1.89k op/s 2024-08-19T14:25:19.726 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:19 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1941704295' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:25:19.735 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:25:19.735 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:25:19.746 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:25:19.750 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:25:19.765 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:25:19.765 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:25:19.779 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:19.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:19.780 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:25:19.793 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:25:19.794 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:25:19.813 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:25:19.814 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:25:19.833 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:25:19.833 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:25:19.834 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:25:19.834 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:25:19.851 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:25:19.851 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:25:19.868 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:25:19.868 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:25:19.880 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:25:19.885 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:25:19.885 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:25:19.901 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:25:19.902 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:25:19.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:19.903 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:19.903 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:25:19.918 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:25:19.918 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:25:19.936 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:25:19.937 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:25:19.963 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:25:19.963 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:25:20.004 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:25:20.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:19 smithi032 ceph-mon[20004]: from='client.25888 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:25:20.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:19 smithi032 ceph-mon[20004]: pgmap v997: 47 pgs: 47 active+clean; 4.0 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 118 MiB/s wr, 1.89k op/s 2024-08-19T14:25:20.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:19 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1941704295' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:25:20.039 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:20.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:20.040 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:25:20.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:19 smithi045 ceph-mon[20143]: from='client.25888 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:25:20.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:19 smithi045 ceph-mon[20143]: pgmap v997: 47 pgs: 47 active+clean; 4.0 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 118 MiB/s wr, 1.89k op/s 2024-08-19T14:25:20.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:19 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1941704295' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:25:20.141 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:25:20.174 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:20.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:20.175 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:25:20.276 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:25:20.306 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:20.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:20.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:25:20.322 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 32 16 74282 74266 145.024 87.125 0.0145242 0.00689127 2024-08-19T14:25:20.408 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:25:20.441 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:20.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:20.442 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:25:20.543 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:25:20.577 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:20.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:20.578 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:25:20.614 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-08-19T14:25:20.679 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:25:20.712 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:20.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:20.713 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:25:20.729 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:25:20.730 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-08-19T14:25:20.814 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:25:20.854 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:20.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:20.855 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:25:20.956 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:25:20.993 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:20.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:20.995 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:25:21.010 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:25:21.096 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:25:21.125 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:21.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:21.126 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:25:21.206 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-19T14:25:21.226 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:25:21.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:21.287 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:21.287 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:25:21.322 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 33 16 76087 76071 144.048 112.812 0.00737493 0.00693732 2024-08-19T14:25:21.387 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:25:21.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:21.429 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:21.430 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:25:21.531 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:25:21.560 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:21.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:21.561 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:25:21.604 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:25:21.604 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [ 2024-08-19T14:25:21.604 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:21.604 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25813.0:76483 5.8 5:15c21bbf:::benchmark_data_smithi045_85309_object76482:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e435)", 2024-08-19T14:25:21.604 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:25:21.591905+0000", 2024-08-19T14:25:21.605 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.0097360439999999993, 2024-08-19T14:25:21.605 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.009768044, 2024-08-19T14:25:21.605 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:25:21.605 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:25:21.605 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "waiting for sub ops", 2024-08-19T14:25:21.605 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:25:21.605 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25813", 2024-08-19T14:25:21.605 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:25:21.605 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 76483 2024-08-19T14:25:21.605 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:21.605 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:25:21.605 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:21.606 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:25:21.606 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:21.591905+0000", 2024-08-19T14:25:21.606 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:25:21.606 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:21.606 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:21.606 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:25:21.606 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:21.591905+0000", 2024-08-19T14:25:21.606 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:25:21.606 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:21.606 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:21.606 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:25:21.606 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:21.591906+0000", 2024-08-19T14:25:21.607 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3829999999999999e-06 2024-08-19T14:25:21.607 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:21.607 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:21.607 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:25:21.607 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:21.591930+0000", 2024-08-19T14:25:21.607 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.4559e-05 2024-08-19T14:25:21.607 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:21.607 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:21.607 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:25:21.607 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:21.591946+0000", 2024-08-19T14:25:21.608 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5954999999999999e-05 2024-08-19T14:25:21.608 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:21.608 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:21.608 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:25:21.608 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:21.591954+0000", 2024-08-19T14:25:21.608 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.2969999999999998e-06 2024-08-19T14:25:21.608 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:21.608 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:21.608 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:25:21.608 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:21.592464+0000", 2024-08-19T14:25:21.608 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00051013700000000003 2024-08-19T14:25:21.609 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:21.609 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:21.609 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:25:21.609 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:21.592602+0000", 2024-08-19T14:25:21.609 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013804999999999999 2024-08-19T14:25:21.609 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:21.609 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:21.609 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,9", 2024-08-19T14:25:21.609 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:21.592739+0000", 2024-08-19T14:25:21.609 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013739000000000001 2024-08-19T14:25:21.609 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:21.609 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:21.610 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:25:21.610 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:21.601005+0000", 2024-08-19T14:25:21.610 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0082655030000000004 2024-08-19T14:25:21.610 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:21.610 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:21.610 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:21.610 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:21.601183+0000", 2024-08-19T14:25:21.610 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000177722 2024-08-19T14:25:21.610 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:21.610 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:25:21.611 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:21.611 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:21.611 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:21.611 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25813.0:76488 5.8 5:1f174351:::benchmark_data_smithi045_85309_object76487:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e435)", 2024-08-19T14:25:21.611 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:25:21.594518+0000", 2024-08-19T14:25:21.611 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.0071221660000000001, 2024-08-19T14:25:21.611 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0072463900000000001, 2024-08-19T14:25:21.611 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:25:21.611 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:25:21.612 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "waiting for sub ops", 2024-08-19T14:25:21.612 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:25:21.612 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25813", 2024-08-19T14:25:21.612 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:25:21.612 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 76488 2024-08-19T14:25:21.612 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:21.612 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:25:21.612 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:21.612 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:25:21.612 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:21.594518+0000", 2024-08-19T14:25:21.612 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:25:21.613 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:21.613 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:21.613 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:25:21.613 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:21.594518+0000", 2024-08-19T14:25:21.613 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:25:21.613 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:21.613 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:21.613 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:25:21.613 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:21.594520+0000", 2024-08-19T14:25:21.613 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2550000000000001e-06 2024-08-19T14:25:21.613 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:21.614 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:21.614 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:25:21.614 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:21.594544+0000", 2024-08-19T14:25:21.614 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.4712e-05 2024-08-19T14:25:21.614 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:21.614 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:21.614 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:25:21.614 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:21.594560+0000", 2024-08-19T14:25:21.614 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5537999999999999e-05 2024-08-19T14:25:21.614 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:21.614 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:21.614 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:25:21.615 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:21.594568+0000", 2024-08-19T14:25:21.615 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.4440000000000008e-06 2024-08-19T14:25:21.615 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:21.615 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:21.615 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:25:21.615 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:21.595641+0000", 2024-08-19T14:25:21.615 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0010724160000000001 2024-08-19T14:25:21.615 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:21.616 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:21.616 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:25:21.616 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:21.595723+0000", 2024-08-19T14:25:21.616 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.1831999999999998e-05 2024-08-19T14:25:21.616 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:21.616 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:21.616 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,9", 2024-08-19T14:25:21.616 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:21.595797+0000", 2024-08-19T14:25:21.616 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4210999999999998e-05 2024-08-19T14:25:21.616 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:21.616 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:21.617 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:25:21.617 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:21.601148+0000", 2024-08-19T14:25:21.617 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0053509589999999998 2024-08-19T14:25:21.617 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:21.617 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:25:21.617 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:21.617 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:21.617 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:21.617 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25813.0:76489 5.8 5:10d37334:::benchmark_data_smithi045_85309_object76488:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e435)", 2024-08-19T14:25:21.617 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:25:21.594584+0000", 2024-08-19T14:25:21.617 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.0070561790000000001, 2024-08-19T14:25:21.617 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0072561780000000003, 2024-08-19T14:25:21.618 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:25:21.618 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:25:21.618 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "waiting for sub ops", 2024-08-19T14:25:21.618 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:25:21.618 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25813", 2024-08-19T14:25:21.618 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:25:21.618 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 76489 2024-08-19T14:25:21.618 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:21.619 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:25:21.619 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:21.619 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:25:21.619 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:21.594584+0000", 2024-08-19T14:25:21.619 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:25:21.619 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:21.619 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:21.619 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:25:21.619 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:21.594584+0000", 2024-08-19T14:25:21.620 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:25:21.620 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:21.620 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:21.620 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:25:21.620 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:21.594585+0000", 2024-08-19T14:25:21.620 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7899999999999998e-07 2024-08-19T14:25:21.620 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:21.620 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:21.620 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:25:21.620 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:21.594614+0000", 2024-08-19T14:25:21.621 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.9196999999999998e-05 2024-08-19T14:25:21.621 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:21.621 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:21.621 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:25:21.621 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:21.594644+0000", 2024-08-19T14:25:21.621 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.9493e-05 2024-08-19T14:25:21.621 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:21.621 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:21.621 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:25:21.622 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:21.594657+0000", 2024-08-19T14:25:21.622 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3047999999999999e-05 2024-08-19T14:25:21.622 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:21.622 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:21.622 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:25:21.622 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:21.598235+0000", 2024-08-19T14:25:21.622 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0035782259999999999 2024-08-19T14:25:21.622 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:21.622 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:21.622 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:25:21.623 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:21.598349+0000", 2024-08-19T14:25:21.623 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000113986 2024-08-19T14:25:21.623 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:21.623 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:21.623 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,9", 2024-08-19T14:25:21.623 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:21.598477+0000", 2024-08-19T14:25:21.623 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012769600000000001 2024-08-19T14:25:21.623 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:21.623 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:25:21.623 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:21.623 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:21.623 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:21.624 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.25813.0:76493 5.a e435/419 5:5d535b77:::benchmark_data_smithi045_85309_object76492:head v 435'4809, mlcod=435'4807)", 2024-08-19T14:25:21.624 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:25:21.599945+0000", 2024-08-19T14:25:21.624 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.00169593, 2024-08-19T14:25:21.624 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001973032, 2024-08-19T14:25:21.624 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:25:21.624 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:25:21.624 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-08-19T14:25:21.624 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:25:21.624 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:21.624 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:25:21.624 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:21.599945+0000", 2024-08-19T14:25:21.624 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:25:21.625 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:21.625 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:21.625 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:25:21.625 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:21.599945+0000", 2024-08-19T14:25:21.625 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:25:21.625 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:21.625 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:21.625 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:25:21.625 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:21.599946+0000", 2024-08-19T14:25:21.625 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.7000000000000003e-07 2024-08-19T14:25:21.625 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:21.626 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:21.626 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:25:21.626 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:21.599969+0000", 2024-08-19T14:25:21.626 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.3356000000000001e-05 2024-08-19T14:25:21.626 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:21.626 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:21.626 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:25:21.626 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:21.599985+0000", 2024-08-19T14:25:21.626 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.594e-05 2024-08-19T14:25:21.626 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:21.626 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:21.626 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:25:21.627 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:21.599993+0000", 2024-08-19T14:25:21.627 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.8169999999999999e-06 2024-08-19T14:25:21.627 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:21.627 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:21.627 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:25:21.627 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:21.600088+0000", 2024-08-19T14:25:21.627 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.5403999999999997e-05 2024-08-19T14:25:21.627 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:21.627 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:21.627 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:25:21.627 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:21.600108+0000", 2024-08-19T14:25:21.628 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9595000000000001e-05 2024-08-19T14:25:21.628 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:21.628 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:25:21.628 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:21.628 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:21.628 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:21.628 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25813.0:76499 5.8 5:195f49b3:::benchmark_data_smithi045_85309_object76498:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e435)", 2024-08-19T14:25:21.628 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:25:21.601168+0000", 2024-08-19T14:25:21.628 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.00047235099999999998, 2024-08-19T14:25:21.628 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000806319, 2024-08-19T14:25:21.628 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:25:21.628 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:25:21.629 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "waiting for sub ops", 2024-08-19T14:25:21.629 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:25:21.629 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25813", 2024-08-19T14:25:21.629 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:25:21.629 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 76499 2024-08-19T14:25:21.629 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:21.629 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:25:21.629 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:21.629 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:25:21.629 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:21.601168+0000", 2024-08-19T14:25:21.629 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:25:21.629 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:21.629 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:21.630 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:25:21.630 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:21.601168+0000", 2024-08-19T14:25:21.630 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:25:21.630 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:21.630 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:21.630 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:25:21.630 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:21.601170+0000", 2024-08-19T14:25:21.630 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5209999999999999e-06 2024-08-19T14:25:21.630 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:21.630 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:21.630 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:25:21.631 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:21.601198+0000", 2024-08-19T14:25:21.631 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.7821e-05 2024-08-19T14:25:21.631 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:21.631 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:21.631 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:25:21.631 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:21.601211+0000", 2024-08-19T14:25:21.631 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3227000000000001e-05 2024-08-19T14:25:21.631 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:21.631 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:21.631 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:25:21.631 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:21.601219+0000", 2024-08-19T14:25:21.631 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.3850000000000002e-06 2024-08-19T14:25:21.631 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:21.632 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:21.632 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:25:21.632 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:21.601275+0000", 2024-08-19T14:25:21.632 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.6124e-05 2024-08-19T14:25:21.632 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:21.632 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:21.632 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:25:21.632 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:21.601406+0000", 2024-08-19T14:25:21.632 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000130328 2024-08-19T14:25:21.632 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:21.632 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:21.632 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,9", 2024-08-19T14:25:21.632 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:21.601556+0000", 2024-08-19T14:25:21.633 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015055099999999999 2024-08-19T14:25:21.633 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:21.633 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:25:21.633 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:21.633 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:21.633 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-08-19T14:25:21.633 INFO:teuthology.orchestra.run.smithi045.stdout: "num_ops": 5 2024-08-19T14:25:21.633 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:25:21.662 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:25:21.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:21.695 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:21.695 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:25:21.762 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":998,"stamp":"2024-08-19T14:25:19.964259+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4520935968,"num_objects":68977,"num_object_clones":0,"num_object_copies":206931,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68975,"num_write_kb":4414400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":73284,"ondisk_log_size":73284,"up":141,"acting":141,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":142,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":17885396,"kb_used_data":13140028,"kb_used_omap":14,"kb_used_meta":4745265,"kb_avail":825820972,"statfs":{"total":863955320832,"available":845640675328,"internally_reserved":0,"allocated":13455388672,"data_stored":13437555609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14353,"internal_metadata":4859152367},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":799277056,"num_objects":12196,"num_object_clones":0,"num_object_copies":36588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12196,"num_write_kb":780544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001724"},"pg_stats":[{"pgid":"5.4","version":"434'4226","reported_seq":4266,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-08-19T14:25:16.767517+0000","last_change":"2024-08-19T14:24:44.595230+0000","last_active":"2024-08-19T14:25:16.767517+0000","last_peered":"2024-08-19T14:25:16.767517+0000","last_clean":"2024-08-19T14:25:16.767517+0000","last_became_active":"2024-08-19T14:24:44.594920+0000","last_became_peered":"2024-08-19T14:24:44.594920+0000","last_unstale":"2024-08-19T14:25:16.767517+0000","last_undegraded":"2024-08-19T14:25:16.767517+0000","last_fullsized":"2024-08-19T14:25:16.767517+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:24:43.574783+0000","objects_scrubbed":0,"log_size":4226,"log_dups_size":0,"ondisk_log_size":4226,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:13.412273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":276955136,"num_objects":4226,"num_object_clones":0,"num_object_copies":12678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4226,"num_write_kb":270464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,0],"acting":[5,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"70'139","reported_seq":1104,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-08-19T14:25:17.724872+0000","last_change":"2024-08-19T14:24:48.191443+0000","last_active":"2024-08-19T14:25:17.724872+0000","last_peered":"2024-08-19T14:25:17.724872+0000","last_clean":"2024-08-19T14:25:17.724872+0000","last_became_active":"2024-08-19T14:18:13.807133+0000","last_became_peered":"2024-08-19T14:18:13.807133+0000","last_unstale":"2024-08-19T14:25:17.724872+0000","last_undegraded":"2024-08-19T14:25:17.724872+0000","last_fullsized":"2024-08-19T14:25:17.724872+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:48.191339+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:19:58.072083+0000","last_clean_scrub_stamp":"2024-08-19T14:24:48.191339+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:08.680074+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,9],"acting":[7,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"434'4311","reported_seq":4351,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-08-19T14:25:16.738296+0000","last_change":"2024-08-19T14:24:44.593690+0000","last_active":"2024-08-19T14:25:16.738296+0000","last_peered":"2024-08-19T14:25:16.738296+0000","last_clean":"2024-08-19T14:25:16.738296+0000","last_became_active":"2024-08-19T14:24:44.593405+0000","last_became_peered":"2024-08-19T14:24:44.593405+0000","last_unstale":"2024-08-19T14:25:16.738296+0000","last_undegraded":"2024-08-19T14:25:16.738296+0000","last_fullsized":"2024-08-19T14:25:16.738296+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:24:43.574783+0000","objects_scrubbed":0,"log_size":4311,"log_dups_size":0,"ondisk_log_size":4311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:25:59.739209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282525696,"num_objects":4311,"num_object_clones":0,"num_object_copies":12933,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4311,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4311,"num_write_kb":275904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.3","version":"70'139","reported_seq":1002,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-08-19T14:25:17.725897+0000","last_change":"2024-08-19T14:24:42.697899+0000","last_active":"2024-08-19T14:25:17.725897+0000","last_peered":"2024-08-19T14:25:17.725897+0000","last_clean":"2024-08-19T14:25:17.725897+0000","last_became_active":"2024-08-19T14:18:13.804721+0000","last_became_peered":"2024-08-19T14:18:13.804721+0000","last_unstale":"2024-08-19T14:25:17.725897+0000","last_undegraded":"2024-08-19T14:25:17.725897+0000","last_fullsized":"2024-08-19T14:25:17.725897+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:42.697801+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:24:42.697801+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:08.817098+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"434'4335","reported_seq":4375,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-08-19T14:25:16.744377+0000","last_change":"2024-08-19T14:24:44.593298+0000","last_active":"2024-08-19T14:25:16.744377+0000","last_peered":"2024-08-19T14:25:16.744377+0000","last_clean":"2024-08-19T14:25:16.744377+0000","last_became_active":"2024-08-19T14:24:44.593002+0000","last_became_peered":"2024-08-19T14:24:44.593002+0000","last_unstale":"2024-08-19T14:25:16.744377+0000","last_undegraded":"2024-08-19T14:25:16.744377+0000","last_fullsized":"2024-08-19T14:25:16.744377+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:24:43.574783+0000","objects_scrubbed":0,"log_size":4335,"log_dups_size":0,"ondisk_log_size":4335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:25:48.182254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284098560,"num_objects":4335,"num_object_clones":0,"num_object_copies":13005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4335,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4335,"num_write_kb":277440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,8],"acting":[5,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"70'139","reported_seq":951,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-08-19T14:25:17.725103+0000","last_change":"2024-08-19T14:24:43.246467+0000","last_active":"2024-08-19T14:25:17.725103+0000","last_peered":"2024-08-19T14:25:17.725103+0000","last_clean":"2024-08-19T14:25:17.725103+0000","last_became_active":"2024-08-19T14:20:50.742354+0000","last_became_peered":"2024-08-19T14:20:50.742354+0000","last_unstale":"2024-08-19T14:25:17.725103+0000","last_undegraded":"2024-08-19T14:25:17.725103+0000","last_fullsized":"2024-08-19T14:25:17.725103+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:43.246387+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:19:59.302931+0000","last_clean_scrub_stamp":"2024-08-19T14:24:43.246387+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:05.068356+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.5","version":"433'4146","reported_seq":4184,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-08-19T14:25:16.401527+0000","last_change":"2024-08-19T14:24:44.592723+0000","last_active":"2024-08-19T14:25:16.401527+0000","last_peered":"2024-08-19T14:25:16.401527+0000","last_clean":"2024-08-19T14:25:16.401527+0000","last_became_active":"2024-08-19T14:24:44.592424+0000","last_became_peered":"2024-08-19T14:24:44.592424+0000","last_unstale":"2024-08-19T14:25:16.401527+0000","last_undegraded":"2024-08-19T14:25:16.401527+0000","last_fullsized":"2024-08-19T14:25:16.401527+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:24:43.574783+0000","objects_scrubbed":0,"log_size":4146,"log_dups_size":0,"ondisk_log_size":4146,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:03.869565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":271712256,"num_objects":4146,"num_object_clones":0,"num_object_copies":12438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4146,"num_write_kb":265344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"70'139","reported_seq":999,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-08-19T14:25:17.721648+0000","last_change":"2024-08-19T14:24:48.179646+0000","last_active":"2024-08-19T14:25:17.721648+0000","last_peered":"2024-08-19T14:25:17.721648+0000","last_clean":"2024-08-19T14:25:17.721648+0000","last_became_active":"2024-08-19T14:18:13.808505+0000","last_became_peered":"2024-08-19T14:18:13.808505+0000","last_unstale":"2024-08-19T14:25:17.721648+0000","last_undegraded":"2024-08-19T14:25:17.721648+0000","last_fullsized":"2024-08-19T14:25:17.721648+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:48.179558+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:24:48.179558+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:25:51.185230+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,11],"acting":[9,0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.3","version":"433'4110","reported_seq":4148,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-08-19T14:25:16.410243+0000","last_change":"2024-08-19T14:24:44.659231+0000","last_active":"2024-08-19T14:25:16.410243+0000","last_peered":"2024-08-19T14:25:16.410243+0000","last_clean":"2024-08-19T14:25:16.410243+0000","last_became_active":"2024-08-19T14:24:44.658960+0000","last_became_peered":"2024-08-19T14:24:44.658960+0000","last_unstale":"2024-08-19T14:25:16.410243+0000","last_undegraded":"2024-08-19T14:25:16.410243+0000","last_fullsized":"2024-08-19T14:25:16.410243+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:24:43.574783+0000","objects_scrubbed":0,"log_size":4110,"log_dups_size":0,"ondisk_log_size":4110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:02.684044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269352960,"num_objects":4110,"num_object_clones":0,"num_object_copies":12330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4110,"num_write_kb":263040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"70'139","reported_seq":651,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-08-19T14:25:17.728425+0000","last_change":"2024-08-19T14:24:47.412400+0000","last_active":"2024-08-19T14:25:17.728425+0000","last_peered":"2024-08-19T14:25:17.728425+0000","last_clean":"2024-08-19T14:25:17.728425+0000","last_became_active":"2024-08-19T14:20:28.204384+0000","last_became_peered":"2024-08-19T14:20:28.204384+0000","last_unstale":"2024-08-19T14:25:17.728425+0000","last_undegraded":"2024-08-19T14:25:17.728425+0000","last_fullsized":"2024-08-19T14:25:17.728425+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:47.412291+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:21:46.057624+0000","last_clean_scrub_stamp":"2024-08-19T14:24:47.412291+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:11.449015+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"433'4185","reported_seq":4223,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-08-19T14:25:16.408128+0000","last_change":"2024-08-19T14:24:44.592960+0000","last_active":"2024-08-19T14:25:16.408128+0000","last_peered":"2024-08-19T14:25:16.408128+0000","last_clean":"2024-08-19T14:25:16.408128+0000","last_became_active":"2024-08-19T14:24:44.592670+0000","last_became_peered":"2024-08-19T14:24:44.592670+0000","last_unstale":"2024-08-19T14:25:16.408128+0000","last_undegraded":"2024-08-19T14:25:16.408128+0000","last_fullsized":"2024-08-19T14:25:16.408128+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:24:43.574783+0000","objects_scrubbed":0,"log_size":4185,"log_dups_size":0,"ondisk_log_size":4185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:07.754236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":274268160,"num_objects":4185,"num_object_clones":0,"num_object_copies":12555,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4185,"num_write_kb":267840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,8],"acting":[0,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"70'139","reported_seq":1003,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-08-19T14:25:17.721669+0000","last_change":"2024-08-19T14:24:46.222297+0000","last_active":"2024-08-19T14:25:17.721669+0000","last_peered":"2024-08-19T14:25:17.721669+0000","last_clean":"2024-08-19T14:25:17.721669+0000","last_became_active":"2024-08-19T14:18:26.874902+0000","last_became_peered":"2024-08-19T14:18:26.874902+0000","last_unstale":"2024-08-19T14:25:17.721669+0000","last_undegraded":"2024-08-19T14:25:17.721669+0000","last_fullsized":"2024-08-19T14:25:17.721669+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:46.222166+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:27.848870+0000","last_clean_scrub_stamp":"2024-08-19T14:24:46.222166+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:09.431052+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,4],"acting":[9,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.1","version":"435'4483","reported_seq":4525,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-08-19T14:25:17.740073+0000","last_change":"2024-08-19T14:24:44.658485+0000","last_active":"2024-08-19T14:25:17.740073+0000","last_peered":"2024-08-19T14:25:17.740073+0000","last_clean":"2024-08-19T14:25:17.740073+0000","last_became_active":"2024-08-19T14:24:44.657571+0000","last_became_peered":"2024-08-19T14:24:44.657571+0000","last_unstale":"2024-08-19T14:25:17.740073+0000","last_undegraded":"2024-08-19T14:25:17.740073+0000","last_fullsized":"2024-08-19T14:25:17.740073+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:24:43.574783+0000","objects_scrubbed":0,"log_size":4483,"log_dups_size":0,"ondisk_log_size":4483,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:25:50.673564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":293797888,"num_objects":4483,"num_object_clones":0,"num_object_copies":13449,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4483,"num_write_kb":286912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"70'139","reported_seq":821,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-08-19T14:25:11.685722+0000","last_change":"2024-08-19T14:24:46.629045+0000","last_active":"2024-08-19T14:25:11.685722+0000","last_peered":"2024-08-19T14:25:11.685722+0000","last_clean":"2024-08-19T14:25:11.685722+0000","last_became_active":"2024-08-19T14:18:26.881435+0000","last_became_peered":"2024-08-19T14:18:26.881435+0000","last_unstale":"2024-08-19T14:25:11.685722+0000","last_undegraded":"2024-08-19T14:25:11.685722+0000","last_fullsized":"2024-08-19T14:25:11.685722+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:46.628960+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:24:46.628960+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:03.429914+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,4],"acting":[11,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.0","version":"435'4558","reported_seq":4600,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-08-19T14:25:18.413695+0000","last_change":"2024-08-19T14:24:44.595889+0000","last_active":"2024-08-19T14:25:18.413695+0000","last_peered":"2024-08-19T14:25:18.413695+0000","last_clean":"2024-08-19T14:25:18.413695+0000","last_became_active":"2024-08-19T14:24:44.595535+0000","last_became_peered":"2024-08-19T14:24:44.595535+0000","last_unstale":"2024-08-19T14:25:18.413695+0000","last_undegraded":"2024-08-19T14:25:18.413695+0000","last_fullsized":"2024-08-19T14:25:18.413695+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:24:43.574783+0000","objects_scrubbed":0,"log_size":4558,"log_dups_size":0,"ondisk_log_size":4558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:07.707608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":298713088,"num_objects":4558,"num_object_clones":0,"num_object_copies":13674,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4558,"num_write_kb":291712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"70'139","reported_seq":1045,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-08-19T14:25:11.685536+0000","last_change":"2024-08-19T14:24:11.345984+0000","last_active":"2024-08-19T14:25:11.685536+0000","last_peered":"2024-08-19T14:25:11.685536+0000","last_clean":"2024-08-19T14:25:11.685536+0000","last_became_active":"2024-08-19T14:18:13.806013+0000","last_became_peered":"2024-08-19T14:18:13.806013+0000","last_unstale":"2024-08-19T14:25:11.685536+0000","last_undegraded":"2024-08-19T14:25:11.685536+0000","last_fullsized":"2024-08-19T14:25:11.685536+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:11.345873+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:24:11.345873+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:25:14.696871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.e","version":"435'4422","reported_seq":4464,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-08-19T14:25:18.524521+0000","last_change":"2024-08-19T14:24:44.595518+0000","last_active":"2024-08-19T14:25:18.524521+0000","last_peered":"2024-08-19T14:25:18.524521+0000","last_clean":"2024-08-19T14:25:18.524521+0000","last_became_active":"2024-08-19T14:24:44.595166+0000","last_became_peered":"2024-08-19T14:24:44.595166+0000","last_unstale":"2024-08-19T14:25:18.524521+0000","last_undegraded":"2024-08-19T14:25:18.524521+0000","last_fullsized":"2024-08-19T14:25:18.524521+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:24:43.574783+0000","objects_scrubbed":0,"log_size":4422,"log_dups_size":0,"ondisk_log_size":4422,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:09.480899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":289800192,"num_objects":4422,"num_object_clones":0,"num_object_copies":13266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4422,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4422,"num_write_kb":283008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,0],"acting":[9,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.a","version":"70'139","reported_seq":1011,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-08-19T14:25:17.721501+0000","last_change":"2024-08-19T14:24:50.196984+0000","last_active":"2024-08-19T14:25:17.721501+0000","last_peered":"2024-08-19T14:25:17.721501+0000","last_clean":"2024-08-19T14:25:17.721501+0000","last_became_active":"2024-08-19T14:18:26.875006+0000","last_became_peered":"2024-08-19T14:18:26.875006+0000","last_unstale":"2024-08-19T14:25:17.721501+0000","last_undegraded":"2024-08-19T14:25:17.721501+0000","last_fullsized":"2024-08-19T14:25:17.721501+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:50.196885+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:28.864870+0000","last_clean_scrub_stamp":"2024-08-19T14:24:50.196885+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:15.234239+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,0],"acting":[9,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.d","version":"433'4272","reported_seq":4310,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-08-19T14:25:16.077746+0000","last_change":"2024-08-19T14:24:44.592424+0000","last_active":"2024-08-19T14:25:16.077746+0000","last_peered":"2024-08-19T14:25:16.077746+0000","last_clean":"2024-08-19T14:25:16.077746+0000","last_became_active":"2024-08-19T14:24:44.592098+0000","last_became_peered":"2024-08-19T14:24:44.592098+0000","last_unstale":"2024-08-19T14:25:16.077746+0000","last_undegraded":"2024-08-19T14:25:16.077746+0000","last_fullsized":"2024-08-19T14:25:16.077746+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:24:43.574783+0000","objects_scrubbed":0,"log_size":4272,"log_dups_size":0,"ondisk_log_size":4272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:25:54.822583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":279969792,"num_objects":4272,"num_object_clones":0,"num_object_copies":12816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4272,"num_write_kb":273408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7],"acting":[11,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.9","version":"70'139","reported_seq":1126,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-08-19T14:25:17.725594+0000","last_change":"2024-08-19T14:25:04.127400+0000","last_active":"2024-08-19T14:25:17.725594+0000","last_peered":"2024-08-19T14:25:17.725594+0000","last_clean":"2024-08-19T14:25:17.725594+0000","last_became_active":"2024-08-19T14:18:26.887657+0000","last_became_peered":"2024-08-19T14:18:26.887657+0000","last_unstale":"2024-08-19T14:25:17.725594+0000","last_undegraded":"2024-08-19T14:25:17.725594+0000","last_fullsized":"2024-08-19T14:25:17.725594+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:25:04.127296+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:25:04.127296+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:12.377922+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"435'4484","reported_seq":4526,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-08-19T14:25:18.392308+0000","last_change":"2024-08-19T14:24:44.595877+0000","last_active":"2024-08-19T14:25:18.392308+0000","last_peered":"2024-08-19T14:25:18.392308+0000","last_clean":"2024-08-19T14:25:18.392308+0000","last_became_active":"2024-08-19T14:24:44.595538+0000","last_became_peered":"2024-08-19T14:24:44.595538+0000","last_unstale":"2024-08-19T14:25:18.392308+0000","last_undegraded":"2024-08-19T14:25:18.392308+0000","last_fullsized":"2024-08-19T14:25:18.392308+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:24:43.574783+0000","objects_scrubbed":0,"log_size":4484,"log_dups_size":0,"ondisk_log_size":4484,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:25:46.650773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":293863424,"num_objects":4484,"num_object_clones":0,"num_object_copies":13452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4484,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4484,"num_write_kb":286976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"70'139","reported_seq":1021,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-08-19T14:25:16.712718+0000","last_change":"2024-08-19T14:24:45.176469+0000","last_active":"2024-08-19T14:25:16.712718+0000","last_peered":"2024-08-19T14:25:16.712718+0000","last_clean":"2024-08-19T14:25:16.712718+0000","last_became_active":"2024-08-19T14:18:26.887277+0000","last_became_peered":"2024-08-19T14:18:26.887277+0000","last_unstale":"2024-08-19T14:25:16.712718+0000","last_undegraded":"2024-08-19T14:25:16.712718+0000","last_fullsized":"2024-08-19T14:25:16.712718+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:45.176400+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:24:45.176400+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:12.727044+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,8],"acting":[5,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.f","version":"434'4211","reported_seq":4251,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-08-19T14:25:16.764950+0000","last_change":"2024-08-19T14:24:44.593761+0000","last_active":"2024-08-19T14:25:16.764950+0000","last_peered":"2024-08-19T14:25:16.764950+0000","last_clean":"2024-08-19T14:25:16.764950+0000","last_became_active":"2024-08-19T14:24:44.593498+0000","last_became_peered":"2024-08-19T14:24:44.593498+0000","last_unstale":"2024-08-19T14:25:16.764950+0000","last_undegraded":"2024-08-19T14:25:16.764950+0000","last_fullsized":"2024-08-19T14:25:16.764950+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:24:43.574783+0000","objects_scrubbed":0,"log_size":4211,"log_dups_size":0,"ondisk_log_size":4211,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:25:50.363040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":275972096,"num_objects":4211,"num_object_clones":0,"num_object_copies":12633,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4211,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4211,"num_write_kb":269504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,11],"acting":[5,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"70'139","reported_seq":1021,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-08-19T14:25:11.685580+0000","last_change":"2024-08-19T14:24:45.610330+0000","last_active":"2024-08-19T14:25:11.685580+0000","last_peered":"2024-08-19T14:25:11.685580+0000","last_clean":"2024-08-19T14:25:11.685580+0000","last_became_active":"2024-08-19T14:18:26.875179+0000","last_became_peered":"2024-08-19T14:18:26.875179+0000","last_unstale":"2024-08-19T14:25:11.685580+0000","last_undegraded":"2024-08-19T14:25:11.685580+0000","last_fullsized":"2024-08-19T14:25:11.685580+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:45.610239+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:24:45.610239+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:15.465099+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.a","version":"433'4085","reported_seq":4123,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-08-19T14:25:14.979808+0000","last_change":"2024-08-19T14:24:44.592347+0000","last_active":"2024-08-19T14:25:14.979808+0000","last_peered":"2024-08-19T14:25:14.979808+0000","last_clean":"2024-08-19T14:25:14.979808+0000","last_became_active":"2024-08-19T14:24:44.592024+0000","last_became_peered":"2024-08-19T14:24:44.592024+0000","last_unstale":"2024-08-19T14:25:14.979808+0000","last_undegraded":"2024-08-19T14:25:14.979808+0000","last_fullsized":"2024-08-19T14:25:14.979808+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:24:43.574783+0000","objects_scrubbed":0,"log_size":4085,"log_dups_size":0,"ondisk_log_size":4085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:06.455975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":267714560,"num_objects":4085,"num_object_clones":0,"num_object_copies":12255,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4085,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4085,"num_write_kb":261440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,11],"acting":[8,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.e","version":"70'139","reported_seq":679,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-08-19T14:25:17.721664+0000","last_change":"2024-08-19T14:25:04.737969+0000","last_active":"2024-08-19T14:25:17.721664+0000","last_peered":"2024-08-19T14:25:17.721664+0000","last_clean":"2024-08-19T14:25:17.721664+0000","last_became_active":"2024-08-19T14:18:26.884105+0000","last_became_peered":"2024-08-19T14:18:26.884105+0000","last_unstale":"2024-08-19T14:25:17.721664+0000","last_undegraded":"2024-08-19T14:25:17.721664+0000","last_fullsized":"2024-08-19T14:25:17.721664+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:25:04.737897+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:20:15.321945+0000","last_clean_scrub_stamp":"2024-08-19T14:25:04.737897+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:25.666670+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.b","version":"435'4564","reported_seq":4605,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-08-19T14:25:18.530821+0000","last_change":"2024-08-19T14:24:44.595612+0000","last_active":"2024-08-19T14:25:18.530821+0000","last_peered":"2024-08-19T14:25:18.530821+0000","last_clean":"2024-08-19T14:25:18.530821+0000","last_became_active":"2024-08-19T14:24:44.595175+0000","last_became_peered":"2024-08-19T14:24:44.595175+0000","last_unstale":"2024-08-19T14:25:18.530821+0000","last_undegraded":"2024-08-19T14:25:18.530821+0000","last_fullsized":"2024-08-19T14:25:18.530821+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:24:43.574783+0000","objects_scrubbed":0,"log_size":4564,"log_dups_size":0,"ondisk_log_size":4564,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:25:55.419449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299106304,"num_objects":4564,"num_object_clones":0,"num_object_copies":13692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4564,"num_write_kb":292096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,5],"acting":[9,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.f","version":"70'139","reported_seq":921,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-08-19T14:25:16.712861+0000","last_change":"2024-08-19T14:24:16.076239+0000","last_active":"2024-08-19T14:25:16.712861+0000","last_peered":"2024-08-19T14:25:16.712861+0000","last_clean":"2024-08-19T14:25:16.712861+0000","last_became_active":"2024-08-19T14:18:18.711708+0000","last_became_peered":"2024-08-19T14:18:18.711708+0000","last_unstale":"2024-08-19T14:25:16.712861+0000","last_undegraded":"2024-08-19T14:25:16.712861+0000","last_fullsized":"2024-08-19T14:25:16.712861+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:16.076143+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:24:16.076143+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:25:32.997797+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.8","version":"433'4138","reported_seq":4176,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-08-19T14:25:16.080078+0000","last_change":"2024-08-19T14:24:44.591109+0000","last_active":"2024-08-19T14:25:16.080078+0000","last_peered":"2024-08-19T14:25:16.080078+0000","last_clean":"2024-08-19T14:25:16.080078+0000","last_became_active":"2024-08-19T14:24:44.590679+0000","last_became_peered":"2024-08-19T14:24:44.590679+0000","last_unstale":"2024-08-19T14:25:16.080078+0000","last_undegraded":"2024-08-19T14:25:16.080078+0000","last_fullsized":"2024-08-19T14:25:16.080078+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:24:43.574783+0000","objects_scrubbed":0,"log_size":4138,"log_dups_size":0,"ondisk_log_size":4138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:25:52.297517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":271187968,"num_objects":4138,"num_object_clones":0,"num_object_copies":12414,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4138,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4138,"num_write_kb":264832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.c","version":"70'139","reported_seq":784,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-08-19T14:25:17.728689+0000","last_change":"2024-08-19T14:24:13.465173+0000","last_active":"2024-08-19T14:25:17.728689+0000","last_peered":"2024-08-19T14:25:17.728689+0000","last_clean":"2024-08-19T14:25:17.728689+0000","last_became_active":"2024-08-19T14:18:26.872942+0000","last_became_peered":"2024-08-19T14:18:26.872942+0000","last_unstale":"2024-08-19T14:25:17.728689+0000","last_undegraded":"2024-08-19T14:25:17.728689+0000","last_fullsized":"2024-08-19T14:25:17.728689+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:13.463716+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:22:52.164586+0000","last_clean_scrub_stamp":"2024-08-19T14:24:13.463716+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:25:23.123046+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,9],"acting":[3,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"435'4445","reported_seq":4487,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-08-19T14:25:18.600927+0000","last_change":"2024-08-19T14:24:44.592293+0000","last_active":"2024-08-19T14:25:18.600927+0000","last_peered":"2024-08-19T14:25:18.600927+0000","last_clean":"2024-08-19T14:25:18.600927+0000","last_became_active":"2024-08-19T14:24:44.591973+0000","last_became_peered":"2024-08-19T14:24:44.591973+0000","last_unstale":"2024-08-19T14:25:18.600927+0000","last_undegraded":"2024-08-19T14:25:18.600927+0000","last_fullsized":"2024-08-19T14:25:18.600927+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:24:43.574783+0000","objects_scrubbed":0,"log_size":4445,"log_dups_size":0,"ondisk_log_size":4445,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:25:54.081269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":291307520,"num_objects":4445,"num_object_clones":0,"num_object_copies":13335,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4445,"num_write_kb":284480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,9],"acting":[7,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"70'139","reported_seq":699,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-08-19T14:25:17.725285+0000","last_change":"2024-08-19T14:24:50.276533+0000","last_active":"2024-08-19T14:25:17.725285+0000","last_peered":"2024-08-19T14:25:17.725285+0000","last_clean":"2024-08-19T14:25:17.725285+0000","last_became_active":"2024-08-19T14:18:26.882002+0000","last_became_peered":"2024-08-19T14:18:26.882002+0000","last_unstale":"2024-08-19T14:25:17.725285+0000","last_undegraded":"2024-08-19T14:25:17.725285+0000","last_fullsized":"2024-08-19T14:25:17.725285+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:50.276408+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:24:50.276408+0000","last_clean_scrub_stamp":"2024-08-19T14:24:50.276408+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:06.161471+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"70'139","reported_seq":955,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-08-19T14:25:16.712547+0000","last_change":"2024-08-19T14:24:14.035778+0000","last_active":"2024-08-19T14:25:16.712547+0000","last_peered":"2024-08-19T14:25:16.712547+0000","last_clean":"2024-08-19T14:25:16.712547+0000","last_became_active":"2024-08-19T14:18:18.703594+0000","last_became_peered":"2024-08-19T14:18:18.703594+0000","last_unstale":"2024-08-19T14:25:16.712547+0000","last_undegraded":"2024-08-19T14:25:16.712547+0000","last_fullsized":"2024-08-19T14:25:16.712547+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:14.035665+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:24:14.035665+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:25:35.876300+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"70'139","reported_seq":1010,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-08-19T14:25:17.726197+0000","last_change":"2024-08-19T14:25:04.742939+0000","last_active":"2024-08-19T14:25:17.726197+0000","last_peered":"2024-08-19T14:25:17.726197+0000","last_clean":"2024-08-19T14:25:17.726197+0000","last_became_active":"2024-08-19T14:18:18.698334+0000","last_became_peered":"2024-08-19T14:18:18.698334+0000","last_unstale":"2024-08-19T14:25:17.726197+0000","last_undegraded":"2024-08-19T14:25:17.726197+0000","last_fullsized":"2024-08-19T14:25:17.726197+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:25:04.742869+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:25:04.742869+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:34.199161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.10","version":"70'139","reported_seq":1118,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-08-19T14:25:16.712613+0000","last_change":"2024-08-19T14:25:06.143718+0000","last_active":"2024-08-19T14:25:16.712613+0000","last_peered":"2024-08-19T14:25:16.712613+0000","last_clean":"2024-08-19T14:25:16.712613+0000","last_became_active":"2024-08-19T14:18:18.704562+0000","last_became_peered":"2024-08-19T14:18:18.704562+0000","last_unstale":"2024-08-19T14:25:16.712613+0000","last_undegraded":"2024-08-19T14:25:16.712613+0000","last_fullsized":"2024-08-19T14:25:16.712613+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:25:06.143606+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:25:06.143606+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:16.549538+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,8],"acting":[5,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"70'139","reported_seq":666,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-08-19T14:25:11.687064+0000","last_change":"2024-08-19T14:24:50.547549+0000","last_active":"2024-08-19T14:25:11.687064+0000","last_peered":"2024-08-19T14:25:11.687064+0000","last_clean":"2024-08-19T14:25:11.687064+0000","last_became_active":"2024-08-19T14:18:18.697894+0000","last_became_peered":"2024-08-19T14:18:18.697894+0000","last_unstale":"2024-08-19T14:25:11.687064+0000","last_undegraded":"2024-08-19T14:25:11.687064+0000","last_fullsized":"2024-08-19T14:25:11.687064+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:50.547428+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:24:50.547428+0000","last_clean_scrub_stamp":"2024-08-19T14:24:50.547428+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:14.355547+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"70'139","reported_seq":638,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-08-19T14:25:17.726051+0000","last_change":"2024-08-19T14:24:41.729459+0000","last_active":"2024-08-19T14:25:17.726051+0000","last_peered":"2024-08-19T14:25:17.726051+0000","last_clean":"2024-08-19T14:25:17.726051+0000","last_became_active":"2024-08-19T14:20:28.204724+0000","last_became_peered":"2024-08-19T14:20:28.204724+0000","last_unstale":"2024-08-19T14:25:17.726051+0000","last_undegraded":"2024-08-19T14:25:17.726051+0000","last_fullsized":"2024-08-19T14:25:17.726051+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:41.729377+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:24:41.729377+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:09.621000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"70'139","reported_seq":762,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-08-19T14:25:11.692945+0000","last_change":"2024-08-19T14:24:48.428636+0000","last_active":"2024-08-19T14:25:11.692945+0000","last_peered":"2024-08-19T14:25:11.692945+0000","last_clean":"2024-08-19T14:25:11.692945+0000","last_became_active":"2024-08-19T14:18:31.901556+0000","last_became_peered":"2024-08-19T14:18:31.901556+0000","last_unstale":"2024-08-19T14:25:11.692945+0000","last_undegraded":"2024-08-19T14:25:11.692945+0000","last_fullsized":"2024-08-19T14:25:11.692945+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:48.428515+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:24:48.428515+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:13.841914+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,0],"acting":[8,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.14","version":"70'139","reported_seq":768,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-08-19T14:25:17.725164+0000","last_change":"2024-08-19T14:24:23.204225+0000","last_active":"2024-08-19T14:25:17.725164+0000","last_peered":"2024-08-19T14:25:17.725164+0000","last_clean":"2024-08-19T14:25:17.725164+0000","last_became_active":"2024-08-19T14:18:18.703926+0000","last_became_peered":"2024-08-19T14:18:18.703926+0000","last_unstale":"2024-08-19T14:25:17.725164+0000","last_undegraded":"2024-08-19T14:25:17.725164+0000","last_fullsized":"2024-08-19T14:25:17.725164+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:23.204112+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:24:23.204112+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:25:30.300500+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,1],"acting":[7,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.15","version":"70'139","reported_seq":654,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-08-19T14:25:17.724932+0000","last_change":"2024-08-19T14:24:44.255453+0000","last_active":"2024-08-19T14:25:17.724932+0000","last_peered":"2024-08-19T14:25:17.724932+0000","last_clean":"2024-08-19T14:25:17.724932+0000","last_became_active":"2024-08-19T14:18:31.912496+0000","last_became_peered":"2024-08-19T14:18:31.912496+0000","last_unstale":"2024-08-19T14:25:17.724932+0000","last_undegraded":"2024-08-19T14:25:17.724932+0000","last_fullsized":"2024-08-19T14:25:17.724932+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:44.255304+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:20:09.180923+0000","last_clean_scrub_stamp":"2024-08-19T14:24:44.255304+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:25:54.443311+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"70'139","reported_seq":640,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-08-19T14:25:17.721631+0000","last_change":"2024-08-19T14:24:44.775890+0000","last_active":"2024-08-19T14:25:17.721631+0000","last_peered":"2024-08-19T14:25:17.721631+0000","last_clean":"2024-08-19T14:25:17.721631+0000","last_became_active":"2024-08-19T14:18:31.901229+0000","last_became_peered":"2024-08-19T14:18:31.901229+0000","last_unstale":"2024-08-19T14:25:17.721631+0000","last_undegraded":"2024-08-19T14:25:17.721631+0000","last_fullsized":"2024-08-19T14:25:17.721631+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:44.775817+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:24:44.775817+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:25:48.718462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1b","version":"70'139","reported_seq":988,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-08-19T14:25:11.692558+0000","last_change":"2024-08-19T14:24:47.410923+0000","last_active":"2024-08-19T14:25:11.692558+0000","last_peered":"2024-08-19T14:25:11.692558+0000","last_clean":"2024-08-19T14:25:11.692558+0000","last_became_active":"2024-08-19T14:18:31.901152+0000","last_became_peered":"2024-08-19T14:18:31.901152+0000","last_unstale":"2024-08-19T14:25:11.692558+0000","last_undegraded":"2024-08-19T14:25:11.692558+0000","last_fullsized":"2024-08-19T14:25:11.692558+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:47.410831+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:24:47.410831+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:07.850070+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.18","version":"70'139","reported_seq":1034,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-08-19T14:25:16.712681+0000","last_change":"2024-08-19T14:24:40.113681+0000","last_active":"2024-08-19T14:25:16.712681+0000","last_peered":"2024-08-19T14:25:16.712681+0000","last_clean":"2024-08-19T14:25:16.712681+0000","last_became_active":"2024-08-19T14:18:31.902735+0000","last_became_peered":"2024-08-19T14:18:31.902735+0000","last_unstale":"2024-08-19T14:25:16.712681+0000","last_undegraded":"2024-08-19T14:25:16.712681+0000","last_fullsized":"2024-08-19T14:25:16.712681+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:40.113600+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:24:40.113600+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:25:58.753284+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.19","version":"70'139","reported_seq":980,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-08-19T14:25:11.686971+0000","last_change":"2024-08-19T14:24:42.577775+0000","last_active":"2024-08-19T14:25:11.686971+0000","last_peered":"2024-08-19T14:25:11.686971+0000","last_clean":"2024-08-19T14:25:11.686971+0000","last_became_active":"2024-08-19T14:18:31.902547+0000","last_became_peered":"2024-08-19T14:18:31.902547+0000","last_unstale":"2024-08-19T14:25:11.686971+0000","last_undegraded":"2024-08-19T14:25:11.686971+0000","last_fullsized":"2024-08-19T14:25:11.686971+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:42.577693+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:24:42.577693+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:25:43.680823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1e","version":"70'139","reported_seq":639,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-08-19T14:25:11.685800+0000","last_change":"2024-08-19T14:24:44.651398+0000","last_active":"2024-08-19T14:25:11.685800+0000","last_peered":"2024-08-19T14:25:11.685800+0000","last_clean":"2024-08-19T14:25:11.685800+0000","last_became_active":"2024-08-19T14:18:31.902546+0000","last_became_peered":"2024-08-19T14:18:31.902546+0000","last_unstale":"2024-08-19T14:25:11.685800+0000","last_undegraded":"2024-08-19T14:25:11.685800+0000","last_fullsized":"2024-08-19T14:25:11.685800+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:44.651299+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:24:44.651299+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:13.740488+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1c","version":"70'139","reported_seq":746,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-08-19T14:25:11.692547+0000","last_change":"2024-08-19T14:24:14.461312+0000","last_active":"2024-08-19T14:25:11.692547+0000","last_peered":"2024-08-19T14:25:11.692547+0000","last_clean":"2024-08-19T14:25:11.692547+0000","last_became_active":"2024-08-19T14:18:31.902494+0000","last_became_peered":"2024-08-19T14:18:31.902494+0000","last_unstale":"2024-08-19T14:25:11.692547+0000","last_undegraded":"2024-08-19T14:25:11.692547+0000","last_fullsized":"2024-08-19T14:25:11.692547+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:14.461176+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:24:14.461176+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:25:26.070165+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,11],"acting":[8,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1d","version":"70'139","reported_seq":664,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-08-19T14:25:17.725747+0000","last_change":"2024-08-19T14:24:22.588391+0000","last_active":"2024-08-19T14:25:17.725747+0000","last_peered":"2024-08-19T14:25:17.725747+0000","last_clean":"2024-08-19T14:25:17.725747+0000","last_became_active":"2024-08-19T14:18:31.901183+0000","last_became_peered":"2024-08-19T14:18:31.901183+0000","last_unstale":"2024-08-19T14:25:17.725747+0000","last_undegraded":"2024-08-19T14:25:17.725747+0000","last_fullsized":"2024-08-19T14:25:17.725747+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:22.588266+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:24:22.588266+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:25:26.637645+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,5],"acting":[1,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":4520345600,"num_objects":68975,"num_object_clones":0,"num_object_copies":206925,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68975,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68975,"num_write_kb":4414400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13578797056,"data_stored":13578797056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68975,"ondisk_log_size":68975,"up":48,"acting":48,"num_store_stats":9},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1802240,"data_stored":1788032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187623,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1850796,"kb_used_data":1324004,"kb_used_omap":1,"kb_used_meta":526782,"kb_avail":91894356,"statfs":{"total":95995035648,"available":94099820544,"internally_reserved":0,"allocated":1355780096,"data_stored":1353803583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":539425220},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":343597383853,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":74,"seq":317827580081,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1969608,"kb_used_data":1419196,"kb_used_omap":1,"kb_used_meta":550398,"kb_avail":91775544,"statfs":{"total":95995035648,"available":93978157056,"internally_reserved":0,"allocated":1453256704,"data_stored":1451271731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":563608001},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874422,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1695852,"kb_used_data":1289828,"kb_used_omap":1,"kb_used_meta":406014,"kb_avail":92049300,"statfs":{"total":95995035648,"available":94258483200,"internally_reserved":0,"allocated":1320783872,"data_stored":1318807359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":415758789},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":57,"seq":244813136058,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2036564,"kb_used_data":1427844,"kb_used_omap":1,"kb_used_meta":608702,"kb_avail":91708588,"statfs":{"total":95995035648,"available":93909594112,"internally_reserved":0,"allocated":1462112256,"data_stored":1460123731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":623311306},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":206158430397,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692033,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2332148,"kb_used_data":1899756,"kb_used_omap":1,"kb_used_meta":432382,"kb_avail":91413004,"statfs":{"total":95995035648,"available":93606916096,"internally_reserved":0,"allocated":1945350144,"data_stored":1943370079,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":442759621},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986374,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2282552,"kb_used_data":1683568,"kb_used_omap":1,"kb_used_meta":598974,"kb_avail":91462600,"statfs":{"total":95995035648,"available":93657702400,"internally_reserved":0,"allocated":1723973632,"data_stored":1721992914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":613349829},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149900,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1115908,"kb_used_data":575228,"kb_used_omap":1,"kb_used_meta":540670,"kb_avail":92629244,"statfs":{"total":95995035648,"available":94852345856,"internally_reserved":0,"allocated":589033472,"data_stored":587048499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":553646535},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411537,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575061,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1762052,"kb_used_data":1126968,"kb_used_omap":1,"kb_used_meta":635070,"kb_avail":91983100,"statfs":{"total":95995035648,"available":94190694400,"internally_reserved":0,"allocated":1154015232,"data_stored":1152030962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":650312133},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738585,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2839916,"kb_used_data":2393636,"kb_used_omap":1,"kb_used_meta":446270,"kb_avail":90905236,"statfs":{"total":95995035648,"available":93086961664,"internally_reserved":0,"allocated":2451083264,"data_stored":2449106751,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":456980933},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2485911552,"data_stored":2485911552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1171587072,"data_stored":1171587072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":587726848,"data_stored":587726848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1726021632,"data_stored":1726021632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1964179456,"data_stored":1964179456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1477640192,"data_stored":1477640192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1329659904,"data_stored":1329659904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1463484416,"data_stored":1463484416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1372585984,"data_stored":1372585984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T14:25:21.766 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-08-19T14:25:21.766 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd pool set unique_pool_3 pg_num 26 2024-08-19T14:25:21.795 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:25:21.826 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:21.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:21.827 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:25:21.833 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:25:21.928 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:25:21.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:21.961 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:21.961 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:25:21.968 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:21 smithi031 ceph-mon[16336]: pgmap v998: 47 pgs: 47 active+clean; 4.2 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-08-19T14:25:21.968 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:21 smithi031 ceph-mon[16336]: from='client.25964 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:25:22.009 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:21 smithi032 ceph-mon[20004]: pgmap v998: 47 pgs: 47 active+clean; 4.2 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-08-19T14:25:22.009 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:21 smithi032 ceph-mon[20004]: from='client.25964 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:25:22.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:21 smithi045 ceph-mon[20143]: pgmap v998: 47 pgs: 47 active+clean; 4.2 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-08-19T14:25:22.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:21 smithi045 ceph-mon[20143]: from='client.25964 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:25:22.062 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:25:22.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:22.100 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:22.100 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:25:22.201 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:25:22.233 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:22.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:22.234 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:25:22.322 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 34 16 77591 77575 142.575 94 0.00338353 0.00699093 2024-08-19T14:25:22.335 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:25:22.337 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-08-19T14:25:22.359 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:22.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:22.360 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:25:22.461 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:25:22.495 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:22.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:22.497 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:25:22.598 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:25:22.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:22.632 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:22.632 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:25:22.705 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:25:22.733 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:25:22.771 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:22.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:22.772 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:25:22.873 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:25:22.906 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:22.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:22.907 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:25:22.984 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:25:23.008 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:25:23.064 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:23.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:23.065 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:25:23.166 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:25:23.196 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:23.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:23.257 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:25:23.322 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 35 16 78869 78853 140.784 79.875 0.00311365 0.00707261 2024-08-19T14:25:23.357 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:25:23.405 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:23.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:23.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:25:23.451 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:25:23.451 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [], 2024-08-19T14:25:23.452 INFO:teuthology.orchestra.run.smithi045.stdout: "complaint_time": 30, 2024-08-19T14:25:23.452 INFO:teuthology.orchestra.run.smithi045.stdout: "num_blocked_ops": 0 2024-08-19T14:25:23.452 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:25:23.507 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:25:23.544 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:23.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:23.545 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:25:23.646 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:25:23.670 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:23.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:23.671 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:25:23.772 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:25:23.806 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:23.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:23.807 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:25:23.908 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:25:23.931 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:23.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:23.932 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:25:24.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:23 smithi032 ceph-mon[20004]: pgmap v999: 47 pgs: 47 active+clean; 4.5 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 109 MiB/s wr, 1.74k op/s 2024-08-19T14:25:24.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:23 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:25:24.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:23 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:25:24.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:23 smithi032 ceph-mon[20004]: osdmap e436: 12 total, 12 up, 9 in 2024-08-19T14:25:24.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:23 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3008527608' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:25:24.025 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is set 2024-08-19T14:25:24.026 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:23 smithi031 ceph-mon[16336]: pgmap v999: 47 pgs: 47 active+clean; 4.5 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 109 MiB/s wr, 1.74k op/s 2024-08-19T14:25:24.026 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:23 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:25:24.026 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:23 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:25:24.026 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:23 smithi031 ceph-mon[16336]: osdmap e436: 12 total, 12 up, 9 in 2024-08-19T14:25:24.026 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:23 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3008527608' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:25:24.033 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:25:24.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:23 smithi045 ceph-mon[20143]: pgmap v999: 47 pgs: 47 active+clean; 4.5 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 109 MiB/s wr, 1.74k op/s 2024-08-19T14:25:24.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:23 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:25:24.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:23 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:25:24.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:23 smithi045 ceph-mon[20143]: osdmap e436: 12 total, 12 up, 9 in 2024-08-19T14:25:24.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:23 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3008527608' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:25:24.070 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:24.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:24.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:25:24.140 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-08-19T14:25:24.171 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:25:24.198 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:24.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:24.199 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:25:24.300 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:25:24.322 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 36 15 80434 80419 139.592 97.875 0.00753229 0.00715952 2024-08-19T14:25:24.333 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:24.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:24.334 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:25:24.435 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:25:24.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:24.472 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:24.472 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:25:24.544 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:25:24.573 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:25:24.603 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:24.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:24.604 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:25:24.705 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:25:24.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:24.746 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:24.746 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:25:24.847 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:25:24.875 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:24.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:24.876 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:25:24.970 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:24 smithi045 ceph-mon[20143]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:25:24.970 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:24 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3008527608' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:25:24.970 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:24 smithi045 ceph-mon[20143]: osdmap e437: 12 total, 12 up, 9 in 2024-08-19T14:25:24.977 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:25:25.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:24 smithi032 ceph-mon[20004]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:25:25.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:24 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3008527608' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:25:25.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:24 smithi032 ceph-mon[20004]: osdmap e437: 12 total, 12 up, 9 in 2024-08-19T14:25:25.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:25.008 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:25.009 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:25:25.109 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:25:25.115 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:25:25.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:24 smithi031 ceph-mon[16336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:25:25.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:24 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3008527608' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:25:25.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:24 smithi031 ceph-mon[16336]: osdmap e437: 12 total, 12 up, 9 in 2024-08-19T14:25:25.126 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:25:25.126 INFO:teuthology.orchestra.run.smithi045.stdout: "size": 20, 2024-08-19T14:25:25.126 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 600, 2024-08-19T14:25:25.126 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [ 2024-08-19T14:25:25.126 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.126 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25057.0:63327 3.1e 3:79dc30b5:::benchmark_data_smithi045_59973_object63326:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e244)", 2024-08-19T14:25:25.126 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:17:33.888774+0000", 2024-08-19T14:25:25.126 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 471.235322741, 2024-08-19T14:25:25.127 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0936514980000001, 2024-08-19T14:25:25.127 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:25:25.127 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:25:25.127 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-08-19T14:25:25.127 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:25:25.127 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25057", 2024-08-19T14:25:25.127 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:25:25.127 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 63327 2024-08-19T14:25:25.127 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.127 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:25:25.127 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.127 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:25:25.127 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:33.888774+0000", 2024-08-19T14:25:25.128 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:25:25.128 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.128 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.128 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:25:25.128 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:33.888774+0000", 2024-08-19T14:25:25.128 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:25:25.128 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.128 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.128 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:25:25.128 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:33.888776+0000", 2024-08-19T14:25:25.128 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9190000000000002e-06 2024-08-19T14:25:25.128 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.128 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.128 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:25:25.129 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:33.888802+0000", 2024-08-19T14:25:25.129 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.5134e-05 2024-08-19T14:25:25.129 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.129 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.129 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:25:25.129 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:33.888817+0000", 2024-08-19T14:25:25.129 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5753e-05 2024-08-19T14:25:25.129 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.129 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.129 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:25:25.129 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:33.888827+0000", 2024-08-19T14:25:25.129 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0117e-05 2024-08-19T14:25:25.129 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.129 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.130 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:25:25.130 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:33.969428+0000", 2024-08-19T14:25:25.130 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.080600464999999996 2024-08-19T14:25:25.130 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.130 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.130 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for peered", 2024-08-19T14:25:25.130 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:33.969436+0000", 2024-08-19T14:25:25.130 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.8730000000000007e-06 2024-08-19T14:25:25.130 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.130 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.130 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:25:25.130 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:34.982371+0000", 2024-08-19T14:25:25.131 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.012934805 2024-08-19T14:25:25.131 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.131 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.131 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:25:25.131 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:17:34.982426+0000", 2024-08-19T14:25:25.131 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.5432000000000001e-05 2024-08-19T14:25:25.131 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:25.131 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:25:25.131 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:25.131 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.131 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.131 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:22707 4.5 4:aa5c4d32:::benchmark_data_smithi045_71562_object22706:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-08-19T14:25:25.131 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:20:41.355284+0000", 2024-08-19T14:25:25.131 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 283.76881287499998, 2024-08-19T14:25:25.132 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.32540933900000002, 2024-08-19T14:25:25.132 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:25:25.132 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:25:25.132 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:25:25.132 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:25:25.132 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:25:25.132 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:25:25.132 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 22707 2024-08-19T14:25:25.132 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.132 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:25:25.132 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.132 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:25:25.132 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.355284+0000", 2024-08-19T14:25:25.133 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:25:25.133 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.133 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.133 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:25:25.133 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.355284+0000", 2024-08-19T14:25:25.133 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:25:25.133 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.133 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.133 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:25:25.133 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.355285+0000", 2024-08-19T14:25:25.133 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.2699999999999998e-07 2024-08-19T14:25:25.133 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.133 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.133 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:25:25.134 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.355304+0000", 2024-08-19T14:25:25.134 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8834e-05 2024-08-19T14:25:25.134 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.134 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.134 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:25:25.134 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.355315+0000", 2024-08-19T14:25:25.134 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1388e-05 2024-08-19T14:25:25.134 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.134 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.134 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:25:25.134 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.355326+0000", 2024-08-19T14:25:25.134 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0101999999999999e-05 2024-08-19T14:25:25.134 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.135 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.135 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:25:25.135 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.355381+0000", 2024-08-19T14:25:25.135 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.5114999999999997e-05 2024-08-19T14:25:25.135 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.135 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.135 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:25:25.135 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.355481+0000", 2024-08-19T14:25:25.135 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000100448 2024-08-19T14:25:25.135 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.135 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.135 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,7", 2024-08-19T14:25:25.135 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.355563+0000", 2024-08-19T14:25:25.135 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.1891999999999995e-05 2024-08-19T14:25:25.135 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.136 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.136 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:25:25.136 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.356429+0000", 2024-08-19T14:25:25.136 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00086571700000000001 2024-08-19T14:25:25.136 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.136 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.136 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:25.136 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.356898+0000", 2024-08-19T14:25:25.136 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000469171 2024-08-19T14:25:25.136 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.136 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.136 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:25.136 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.680624+0000", 2024-08-19T14:25:25.137 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.32372564599999998 2024-08-19T14:25:25.137 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.137 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.137 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:25:25.137 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.680661+0000", 2024-08-19T14:25:25.137 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.7589999999999998e-05 2024-08-19T14:25:25.137 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.137 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.137 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:25:25.137 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.680694+0000", 2024-08-19T14:25:25.137 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.2509000000000003e-05 2024-08-19T14:25:25.137 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:25.137 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:25:25.137 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:25.137 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.138 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.138 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:22723 4.5 4:a03b0b8c:::benchmark_data_smithi045_71562_object22722:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-08-19T14:25:25.138 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:20:41.360250+0000", 2024-08-19T14:25:25.138 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 283.76384754200001, 2024-08-19T14:25:25.138 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.32180027, 2024-08-19T14:25:25.138 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:25:25.138 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:25:25.138 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:25:25.138 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:25:25.138 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:25:25.138 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:25:25.138 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 22723 2024-08-19T14:25:25.138 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.139 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:25:25.139 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.139 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:25:25.139 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.360250+0000", 2024-08-19T14:25:25.139 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:25:25.139 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.139 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.139 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:25:25.139 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.360250+0000", 2024-08-19T14:25:25.139 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:25:25.139 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.139 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.139 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:25:25.139 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.360251+0000", 2024-08-19T14:25:25.140 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7740000000000001e-06 2024-08-19T14:25:25.140 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.140 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.140 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:25:25.141 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.360271+0000", 2024-08-19T14:25:25.141 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9755999999999999e-05 2024-08-19T14:25:25.141 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.141 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.141 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:25:25.141 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.360280+0000", 2024-08-19T14:25:25.141 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.268e-06 2024-08-19T14:25:25.141 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.141 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.141 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:25:25.141 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.360286+0000", 2024-08-19T14:25:25.141 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2700000000000004e-06 2024-08-19T14:25:25.142 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.142 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.142 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:25:25.142 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.360339+0000", 2024-08-19T14:25:25.142 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.3644999999999999e-05 2024-08-19T14:25:25.142 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.142 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.142 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:25:25.142 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.360424+0000", 2024-08-19T14:25:25.142 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.4493999999999994e-05 2024-08-19T14:25:25.142 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.142 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.142 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,7", 2024-08-19T14:25:25.143 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.360529+0000", 2024-08-19T14:25:25.143 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00010503200000000001 2024-08-19T14:25:25.143 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.143 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.143 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:25:25.143 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.362147+0000", 2024-08-19T14:25:25.143 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001617718 2024-08-19T14:25:25.143 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.143 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.143 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:25.143 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.362188+0000", 2024-08-19T14:25:25.143 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.1553e-05 2024-08-19T14:25:25.143 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.143 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.144 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:25.144 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.681950+0000", 2024-08-19T14:25:25.144 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.31976217299999998 2024-08-19T14:25:25.144 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.144 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.144 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:25:25.144 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.682008+0000", 2024-08-19T14:25:25.144 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7494000000000002e-05 2024-08-19T14:25:25.144 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.144 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.144 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:25:25.144 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.682050+0000", 2024-08-19T14:25:25.144 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.2092999999999998e-05 2024-08-19T14:25:25.144 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:25.145 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:25:25.145 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:25.145 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.145 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.145 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:22725 4.5 4:adb3fee7:::benchmark_data_smithi045_71562_object22724:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-08-19T14:25:25.145 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:20:41.361334+0000", 2024-08-19T14:25:25.145 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 283.76276356800003, 2024-08-19T14:25:25.145 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.32179169200000002, 2024-08-19T14:25:25.145 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:25:25.145 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:25:25.145 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:25:25.145 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:25:25.145 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:25:25.145 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:25:25.146 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 22725 2024-08-19T14:25:25.146 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.146 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:25:25.146 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.146 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:25:25.146 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.361334+0000", 2024-08-19T14:25:25.146 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:25:25.146 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.146 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.146 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:25:25.146 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.361334+0000", 2024-08-19T14:25:25.146 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:25:25.146 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.147 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.147 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:25:25.147 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.361334+0000", 2024-08-19T14:25:25.147 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.61e-07 2024-08-19T14:25:25.147 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.147 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.147 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:25:25.147 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.361353+0000", 2024-08-19T14:25:25.147 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9074999999999999e-05 2024-08-19T14:25:25.148 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.148 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.148 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:25:25.148 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.361365+0000", 2024-08-19T14:25:25.148 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1498000000000001e-05 2024-08-19T14:25:25.148 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.148 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.148 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:25:25.148 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.361371+0000", 2024-08-19T14:25:25.148 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.1290000000000001e-06 2024-08-19T14:25:25.148 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.148 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.148 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:25:25.148 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.361415+0000", 2024-08-19T14:25:25.149 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.3445999999999997e-05 2024-08-19T14:25:25.149 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.149 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.149 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:25:25.149 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.361542+0000", 2024-08-19T14:25:25.149 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012710400000000001 2024-08-19T14:25:25.149 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.149 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.149 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,7", 2024-08-19T14:25:25.149 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.361632+0000", 2024-08-19T14:25:25.149 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.9963999999999994e-05 2024-08-19T14:25:25.149 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.149 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.149 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:25:25.150 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.362288+0000", 2024-08-19T14:25:25.150 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00065625500000000001 2024-08-19T14:25:25.150 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.150 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.150 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:25.150 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.362699+0000", 2024-08-19T14:25:25.150 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00041120600000000001 2024-08-19T14:25:25.150 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.150 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.150 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:25.150 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.683027+0000", 2024-08-19T14:25:25.150 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.32032799699999998 2024-08-19T14:25:25.150 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.151 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.151 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:25:25.151 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.683084+0000", 2024-08-19T14:25:25.151 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7054e-05 2024-08-19T14:25:25.151 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.151 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.151 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:25:25.151 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.683125+0000", 2024-08-19T14:25:25.151 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.1202999999999998e-05 2024-08-19T14:25:25.151 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:25.151 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:25:25.151 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:25.151 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.151 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.151 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:32454 4.3 4:c5fede70:::benchmark_data_smithi045_71562_object32453:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e343)", 2024-08-19T14:25:25.152 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:20:45.676484+0000", 2024-08-19T14:25:25.152 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 279.44761361500002, 2024-08-19T14:25:25.152 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.212526989, 2024-08-19T14:25:25.152 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:25:25.152 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:25:25.152 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:25:25.152 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:25:25.152 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:25:25.152 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:25:25.152 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 32454 2024-08-19T14:25:25.152 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.152 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:25:25.152 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.153 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:25:25.153 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.676484+0000", 2024-08-19T14:25:25.153 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:25:25.153 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.153 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.153 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:25:25.153 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.676484+0000", 2024-08-19T14:25:25.153 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:25:25.153 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.153 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.153 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:25:25.153 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.676484+0000", 2024-08-19T14:25:25.153 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1900000000000003e-07 2024-08-19T14:25:25.154 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.154 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.154 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:25:25.154 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.676506+0000", 2024-08-19T14:25:25.154 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1515e-05 2024-08-19T14:25:25.154 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.154 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.154 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:25:25.154 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.676513+0000", 2024-08-19T14:25:25.154 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.7570000000000004e-06 2024-08-19T14:25:25.154 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.154 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.154 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:25:25.154 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.676519+0000", 2024-08-19T14:25:25.155 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4419999999999997e-06 2024-08-19T14:25:25.155 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.155 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.155 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:25:25.155 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.676567+0000", 2024-08-19T14:25:25.155 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.8439999999999997e-05 2024-08-19T14:25:25.155 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.155 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.155 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:25:25.155 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.676672+0000", 2024-08-19T14:25:25.155 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000104991 2024-08-19T14:25:25.155 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.155 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.156 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,7", 2024-08-19T14:25:25.156 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.676835+0000", 2024-08-19T14:25:25.156 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00016259600000000001 2024-08-19T14:25:25.156 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.156 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.156 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:25:25.156 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.678852+0000", 2024-08-19T14:25:25.156 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.002017511 2024-08-19T14:25:25.156 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.156 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.156 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:25.156 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.678903+0000", 2024-08-19T14:25:25.156 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0387999999999999e-05 2024-08-19T14:25:25.156 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.156 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.157 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:25.157 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.888915+0000", 2024-08-19T14:25:25.157 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.21001176899999999 2024-08-19T14:25:25.157 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.157 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.157 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:25:25.157 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.888969+0000", 2024-08-19T14:25:25.157 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.401e-05 2024-08-19T14:25:25.157 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.157 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.157 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:25:25.157 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:45.889011+0000", 2024-08-19T14:25:25.157 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.2051000000000001e-05 2024-08-19T14:25:25.157 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:25.158 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:25:25.158 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:25.158 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.158 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.158 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:116285 4.8 4:15b9fca8:::benchmark_data_smithi045_71562_object116284:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-08-19T14:25:25.158 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:21:48.600383+0000", 2024-08-19T14:25:25.158 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 216.52371432699999, 2024-08-19T14:25:25.158 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.25002218100000001, 2024-08-19T14:25:25.158 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:25:25.158 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:25:25.158 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:25:25.158 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:25:25.158 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:25:25.159 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:25:25.159 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 116285 2024-08-19T14:25:25.159 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.159 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:25:25.159 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.159 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:25:25.159 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.600383+0000", 2024-08-19T14:25:25.159 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:25:25.159 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.159 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.159 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:25:25.159 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.600383+0000", 2024-08-19T14:25:25.159 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:25:25.159 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.160 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.160 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:25:25.160 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.600384+0000", 2024-08-19T14:25:25.160 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.3200000000000003e-07 2024-08-19T14:25:25.160 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.160 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.160 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:25:25.160 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.600403+0000", 2024-08-19T14:25:25.160 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9326000000000001e-05 2024-08-19T14:25:25.160 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.160 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.160 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:25:25.160 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.600414+0000", 2024-08-19T14:25:25.160 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1129e-05 2024-08-19T14:25:25.161 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.161 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.161 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:25:25.161 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.600421+0000", 2024-08-19T14:25:25.161 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.8589999999999999e-06 2024-08-19T14:25:25.161 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.161 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.161 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:25:25.161 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.600478+0000", 2024-08-19T14:25:25.161 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.6502000000000002e-05 2024-08-19T14:25:25.161 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.161 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.161 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:25:25.161 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.600573+0000", 2024-08-19T14:25:25.162 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.5165000000000001e-05 2024-08-19T14:25:25.162 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.162 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.162 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 7,9", 2024-08-19T14:25:25.162 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.600654+0000", 2024-08-19T14:25:25.162 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.1002999999999997e-05 2024-08-19T14:25:25.162 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.162 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.162 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:25:25.162 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.607545+0000", 2024-08-19T14:25:25.162 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0068915039999999997 2024-08-19T14:25:25.162 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.162 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.163 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:25.163 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.610300+0000", 2024-08-19T14:25:25.163 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0027545199999999999 2024-08-19T14:25:25.163 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.163 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.163 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:25.163 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.850322+0000", 2024-08-19T14:25:25.163 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.24002195200000001 2024-08-19T14:25:25.163 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.163 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.163 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:25:25.163 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.850364+0000", 2024-08-19T14:25:25.163 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.2582e-05 2024-08-19T14:25:25.163 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.164 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.164 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:25:25.164 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.850405+0000", 2024-08-19T14:25:25.164 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.0707000000000002e-05 2024-08-19T14:25:25.164 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:25.164 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:25:25.164 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:25.164 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.164 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.164 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:116286 4.8 4:198dc556:::benchmark_data_smithi045_71562_object116285:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-08-19T14:25:25.164 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:21:48.600439+0000", 2024-08-19T14:25:25.164 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 216.52365829999999, 2024-08-19T14:25:25.164 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.25012546899999999, 2024-08-19T14:25:25.164 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:25:25.165 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:25:25.165 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:25:25.165 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:25:25.165 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:25:25.165 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:25:25.165 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 116286 2024-08-19T14:25:25.165 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.165 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:25:25.165 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.165 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:25:25.165 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.600439+0000", 2024-08-19T14:25:25.165 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:25:25.165 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.165 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.165 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:25:25.166 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.600439+0000", 2024-08-19T14:25:25.166 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:25:25.166 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.166 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.166 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:25:25.166 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.600439+0000", 2024-08-19T14:25:25.166 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2900000000000004e-07 2024-08-19T14:25:25.166 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.166 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.166 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:25:25.166 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.600458+0000", 2024-08-19T14:25:25.166 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8729999999999999e-05 2024-08-19T14:25:25.166 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.166 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.167 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:25:25.167 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.600466+0000", 2024-08-19T14:25:25.167 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.1030000000000002e-06 2024-08-19T14:25:25.167 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.167 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.167 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:25:25.167 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.600472+0000", 2024-08-19T14:25:25.167 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4380000000000003e-06 2024-08-19T14:25:25.167 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.167 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.167 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:25:25.167 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.603745+0000", 2024-08-19T14:25:25.167 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0032732350000000002 2024-08-19T14:25:25.167 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.168 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.168 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:25:25.169 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.603905+0000", 2024-08-19T14:25:25.169 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015985500000000001 2024-08-19T14:25:25.169 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.170 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.170 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 7,9", 2024-08-19T14:25:25.170 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.604060+0000", 2024-08-19T14:25:25.170 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015485799999999999 2024-08-19T14:25:25.170 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.170 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.170 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:25:25.170 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.607871+0000", 2024-08-19T14:25:25.170 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0038114540000000001 2024-08-19T14:25:25.170 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.170 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.170 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:25.171 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.610365+0000", 2024-08-19T14:25:25.171 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0024943449999999998 2024-08-19T14:25:25.171 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.171 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.171 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:25.171 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.850477+0000", 2024-08-19T14:25:25.171 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.24011110099999999 2024-08-19T14:25:25.171 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.171 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.171 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:25:25.171 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.850528+0000", 2024-08-19T14:25:25.171 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1684999999999997e-05 2024-08-19T14:25:25.171 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.171 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.172 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:25:25.172 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.850564+0000", 2024-08-19T14:25:25.172 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.6136000000000001e-05 2024-08-19T14:25:25.172 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:25.172 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:25:25.172 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:25.172 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.172 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.172 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:116297 4.8 4:17724d94:::benchmark_data_smithi045_71562_object116296:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-08-19T14:25:25.172 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:21:48.612374+0000", 2024-08-19T14:25:25.172 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 216.51172322299999, 2024-08-19T14:25:25.172 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.28029534299999997, 2024-08-19T14:25:25.172 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:25:25.172 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:25:25.173 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:25:25.173 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:25:25.173 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:25:25.173 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:25:25.173 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 116297 2024-08-19T14:25:25.173 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.173 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:25:25.173 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.173 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:25:25.173 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.612374+0000", 2024-08-19T14:25:25.173 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:25:25.173 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.173 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.174 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:25:25.174 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.612374+0000", 2024-08-19T14:25:25.174 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:25:25.174 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.174 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.174 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:25:25.174 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.612375+0000", 2024-08-19T14:25:25.174 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2869999999999999e-06 2024-08-19T14:25:25.174 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.174 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.174 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:25:25.174 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.612393+0000", 2024-08-19T14:25:25.174 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7864e-05 2024-08-19T14:25:25.174 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.175 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.175 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:25:25.175 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.612404+0000", 2024-08-19T14:25:25.175 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1377e-05 2024-08-19T14:25:25.175 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.175 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.175 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:25:25.175 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.612410+0000", 2024-08-19T14:25:25.175 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.3730000000000001e-06 2024-08-19T14:25:25.175 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.175 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.175 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:25:25.175 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.612460+0000", 2024-08-19T14:25:25.175 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.9675999999999997e-05 2024-08-19T14:25:25.176 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.176 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.176 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:25:25.176 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.612561+0000", 2024-08-19T14:25:25.176 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00010157099999999999 2024-08-19T14:25:25.176 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.176 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.176 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 7,9", 2024-08-19T14:25:25.176 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.612676+0000", 2024-08-19T14:25:25.176 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00011491 2024-08-19T14:25:25.176 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.176 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.176 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:25.176 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.616157+0000", 2024-08-19T14:25:25.177 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00348073 2024-08-19T14:25:25.177 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.177 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.177 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:25:25.177 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.616249+0000", 2024-08-19T14:25:25.177 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.2085999999999999e-05 2024-08-19T14:25:25.177 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.177 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.177 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:25.177 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.850604+0000", 2024-08-19T14:25:25.177 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.23435562900000001 2024-08-19T14:25:25.177 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.177 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.177 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:25:25.178 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.892613+0000", 2024-08-19T14:25:25.178 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.042008839999999999 2024-08-19T14:25:25.178 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.178 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.178 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:25:25.178 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.892669+0000", 2024-08-19T14:25:25.178 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.5999999999999999e-05 2024-08-19T14:25:25.178 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:25.178 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:25:25.178 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:25.178 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.178 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.178 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:116299 4.3 4:c0da230c:::benchmark_data_smithi045_71562_object116298:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-08-19T14:25:25.179 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:21:48.612978+0000", 2024-08-19T14:25:25.179 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 216.511118934, 2024-08-19T14:25:25.179 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.23536219899999999, 2024-08-19T14:25:25.179 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:25:25.179 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:25:25.179 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:25:25.179 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:25:25.179 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:25:25.179 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:25:25.179 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 116299 2024-08-19T14:25:25.179 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.179 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:25:25.179 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.179 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:25:25.180 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.612978+0000", 2024-08-19T14:25:25.180 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:25:25.180 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.180 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.180 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:25:25.180 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.612978+0000", 2024-08-19T14:25:25.180 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:25:25.180 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.180 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.180 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:25:25.180 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.612979+0000", 2024-08-19T14:25:25.180 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.7599999999999996e-07 2024-08-19T14:25:25.180 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.180 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.181 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:25:25.181 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.612998+0000", 2024-08-19T14:25:25.181 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8527000000000001e-05 2024-08-19T14:25:25.181 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.181 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.181 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:25:25.181 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.613008+0000", 2024-08-19T14:25:25.181 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.079e-05 2024-08-19T14:25:25.181 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.181 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.181 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:25:25.181 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.613014+0000", 2024-08-19T14:25:25.181 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4210000000000003e-06 2024-08-19T14:25:25.181 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.182 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.182 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:25:25.182 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.613062+0000", 2024-08-19T14:25:25.182 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.8674999999999997e-05 2024-08-19T14:25:25.182 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.182 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.182 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:25:25.182 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.613200+0000", 2024-08-19T14:25:25.182 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013715400000000001 2024-08-19T14:25:25.182 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.182 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.182 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,7", 2024-08-19T14:25:25.182 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.613317+0000", 2024-08-19T14:25:25.182 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00011734399999999999 2024-08-19T14:25:25.183 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.183 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.183 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:25:25.183 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.616326+0000", 2024-08-19T14:25:25.183 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0030090450000000001 2024-08-19T14:25:25.183 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.183 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.183 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:25.183 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.621117+0000", 2024-08-19T14:25:25.183 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0047912160000000001 2024-08-19T14:25:25.183 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.183 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.183 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:25.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:25.184 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:25.185 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:25:25.186 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.848233+0000", 2024-08-19T14:25:25.186 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.22711604099999999 2024-08-19T14:25:25.186 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.186 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.186 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:25:25.186 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.848288+0000", 2024-08-19T14:25:25.186 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4972999999999997e-05 2024-08-19T14:25:25.186 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.186 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.186 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:25:25.187 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.848340+0000", 2024-08-19T14:25:25.187 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2237000000000001e-05 2024-08-19T14:25:25.187 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:25.187 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:25:25.187 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:25.187 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.187 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.187 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:116311 4.8 4:16b96ac4:::benchmark_data_smithi045_71562_object116310:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-08-19T14:25:25.187 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:21:48.627253+0000", 2024-08-19T14:25:25.187 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 216.496843985, 2024-08-19T14:25:25.187 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.26561561299999997, 2024-08-19T14:25:25.187 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:25:25.187 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:25:25.187 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:25:25.188 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:25:25.188 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:25:25.188 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:25:25.188 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 116311 2024-08-19T14:25:25.188 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.188 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:25:25.188 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.188 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:25:25.188 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.627253+0000", 2024-08-19T14:25:25.188 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:25:25.188 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.188 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.188 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:25:25.189 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.627253+0000", 2024-08-19T14:25:25.189 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:25:25.189 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.189 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.189 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:25:25.189 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.627254+0000", 2024-08-19T14:25:25.189 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4000000000000001e-07 2024-08-19T14:25:25.189 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.189 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.189 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:25:25.189 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.627293+0000", 2024-08-19T14:25:25.189 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.8742000000000003e-05 2024-08-19T14:25:25.189 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.189 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.189 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:25:25.190 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.627302+0000", 2024-08-19T14:25:25.190 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.2639999999999997e-06 2024-08-19T14:25:25.190 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.190 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.190 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:25:25.190 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.627307+0000", 2024-08-19T14:25:25.190 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0209999999999999e-06 2024-08-19T14:25:25.190 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.190 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.190 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:25:25.190 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.627378+0000", 2024-08-19T14:25:25.190 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.0529000000000002e-05 2024-08-19T14:25:25.190 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.191 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.191 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:25:25.191 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.627458+0000", 2024-08-19T14:25:25.191 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.0259999999999994e-05 2024-08-19T14:25:25.191 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.191 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.191 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 7,9", 2024-08-19T14:25:25.191 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.627624+0000", 2024-08-19T14:25:25.191 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00016623300000000001 2024-08-19T14:25:25.191 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.191 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.191 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:25.191 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.631383+0000", 2024-08-19T14:25:25.191 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0037587599999999999 2024-08-19T14:25:25.192 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.192 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.192 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:25:25.192 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.631513+0000", 2024-08-19T14:25:25.192 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013022199999999999 2024-08-19T14:25:25.192 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.192 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.192 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:25.192 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.892784+0000", 2024-08-19T14:25:25.192 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.26127100399999997 2024-08-19T14:25:25.192 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.192 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.192 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:25:25.192 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.892832+0000", 2024-08-19T14:25:25.193 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.7970999999999999e-05 2024-08-19T14:25:25.193 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.193 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.193 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:25:25.193 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.892869+0000", 2024-08-19T14:25:25.193 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.6866999999999998e-05 2024-08-19T14:25:25.193 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:25.193 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:25:25.193 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:25.193 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.193 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.193 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:116334 4.8 4:1f777a70:::benchmark_data_smithi045_71562_object116333:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-08-19T14:25:25.193 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:21:48.650934+0000", 2024-08-19T14:25:25.193 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 216.47316327799999, 2024-08-19T14:25:25.194 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.242080929, 2024-08-19T14:25:25.194 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:25:25.194 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:25:25.194 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:25:25.194 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:25:25.194 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:25:25.194 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:25:25.194 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 116334 2024-08-19T14:25:25.194 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.194 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:25:25.194 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.194 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:25:25.194 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.650934+0000", 2024-08-19T14:25:25.194 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:25:25.195 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.195 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.195 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:25:25.195 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.650934+0000", 2024-08-19T14:25:25.195 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:25:25.195 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.195 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.195 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:25:25.195 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.650935+0000", 2024-08-19T14:25:25.195 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.26e-06 2024-08-19T14:25:25.195 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.195 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.195 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:25:25.195 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.650954+0000", 2024-08-19T14:25:25.196 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9000000000000001e-05 2024-08-19T14:25:25.196 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.196 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.196 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:25:25.196 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.650964+0000", 2024-08-19T14:25:25.196 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0253e-05 2024-08-19T14:25:25.196 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.196 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.196 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:25:25.196 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.650970+0000", 2024-08-19T14:25:25.196 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1499999999999998e-06 2024-08-19T14:25:25.196 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.196 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.196 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:25:25.196 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.651014+0000", 2024-08-19T14:25:25.197 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.4391000000000002e-05 2024-08-19T14:25:25.197 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.197 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.197 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:25:25.197 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.651134+0000", 2024-08-19T14:25:25.197 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000119535 2024-08-19T14:25:25.197 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.197 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.197 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 7,9", 2024-08-19T14:25:25.197 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.651262+0000", 2024-08-19T14:25:25.197 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012826100000000001 2024-08-19T14:25:25.197 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.197 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.197 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:25:25.198 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.654276+0000", 2024-08-19T14:25:25.198 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0030142839999999999 2024-08-19T14:25:25.198 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.198 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.198 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:25.198 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.655021+0000", 2024-08-19T14:25:25.198 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00074536299999999997 2024-08-19T14:25:25.198 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.198 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.198 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:25.198 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.892914+0000", 2024-08-19T14:25:25.198 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.23789228400000001 2024-08-19T14:25:25.198 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.198 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.199 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:25:25.199 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.892980+0000", 2024-08-19T14:25:25.199 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.6193999999999997e-05 2024-08-19T14:25:25.199 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.199 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.199 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:25:25.199 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.893015+0000", 2024-08-19T14:25:25.199 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.4953999999999998e-05 2024-08-19T14:25:25.199 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:25.199 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:25:25.199 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:25.199 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.199 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.199 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:116354 4.8 4:103642ea:::benchmark_data_smithi045_71562_object116353:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-08-19T14:25:25.200 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:21:48.673582+0000", 2024-08-19T14:25:25.200 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 216.45051560499999, 2024-08-19T14:25:25.200 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.21960154700000001, 2024-08-19T14:25:25.200 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:25:25.200 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:25:25.200 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:25:25.200 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:25:25.200 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:25:25.200 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:25:25.200 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 116354 2024-08-19T14:25:25.200 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.200 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:25:25.200 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.200 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:25:25.201 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.673582+0000", 2024-08-19T14:25:25.201 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:25:25.201 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.201 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.201 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:25:25.201 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.673582+0000", 2024-08-19T14:25:25.201 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:25:25.201 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.201 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.201 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:25:25.201 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.673583+0000", 2024-08-19T14:25:25.201 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.279e-06 2024-08-19T14:25:25.201 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.201 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.202 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:25:25.202 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.673633+0000", 2024-08-19T14:25:25.202 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.9845000000000001e-05 2024-08-19T14:25:25.202 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.202 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.202 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:25:25.202 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.673644+0000", 2024-08-19T14:25:25.202 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1333e-05 2024-08-19T14:25:25.202 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.202 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.202 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:25:25.202 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.673649+0000", 2024-08-19T14:25:25.202 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2000000000000002e-06 2024-08-19T14:25:25.203 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.203 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.203 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:25:25.203 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.673703+0000", 2024-08-19T14:25:25.203 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.3471999999999999e-05 2024-08-19T14:25:25.203 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.203 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.203 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:25:25.203 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.673845+0000", 2024-08-19T14:25:25.203 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014190399999999999 2024-08-19T14:25:25.203 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.203 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.203 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 7,9", 2024-08-19T14:25:25.203 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.673956+0000", 2024-08-19T14:25:25.204 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00011128000000000001 2024-08-19T14:25:25.204 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.204 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.204 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:25:25.204 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.676763+0000", 2024-08-19T14:25:25.204 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.002807038 2024-08-19T14:25:25.204 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.204 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.204 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:25.204 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.677752+0000", 2024-08-19T14:25:25.204 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00098899899999999995 2024-08-19T14:25:25.204 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.204 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.204 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:25.205 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.893093+0000", 2024-08-19T14:25:25.205 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.21534098300000001 2024-08-19T14:25:25.205 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.205 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.205 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:25:25.205 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.893143+0000", 2024-08-19T14:25:25.205 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.9877999999999999e-05 2024-08-19T14:25:25.205 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.205 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.205 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:25:25.205 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.893183+0000", 2024-08-19T14:25:25.205 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.0336000000000001e-05 2024-08-19T14:25:25.205 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:25.206 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:25:25.206 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:25.206 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.206 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.206 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:165624 4.3 4:cbfa230a:::benchmark_data_smithi045_71562_object34449:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-08-19T14:25:25.206 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:22:42.872335+0000", 2024-08-19T14:25:25.206 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 162.25176219799999, 2024-08-19T14:25:25.206 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.21952618099999999, 2024-08-19T14:25:25.206 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:25:25.206 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:25:25.206 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:25:25.206 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:25:25.206 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:25:25.206 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:25:25.207 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 165624 2024-08-19T14:25:25.207 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.207 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:25:25.207 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.207 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:25:25.207 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.872335+0000", 2024-08-19T14:25:25.207 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:25:25.207 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.207 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.207 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:25:25.207 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.872335+0000", 2024-08-19T14:25:25.207 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:25:25.207 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.207 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.208 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:25:25.208 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.872336+0000", 2024-08-19T14:25:25.208 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1370000000000001e-06 2024-08-19T14:25:25.208 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.208 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.208 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:25:25.208 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.872341+0000", 2024-08-19T14:25:25.208 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.7840000000000003e-06 2024-08-19T14:25:25.208 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.208 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.208 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:25:25.208 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.872351+0000", 2024-08-19T14:25:25.208 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0254999999999999e-05 2024-08-19T14:25:25.208 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.209 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.209 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:25:25.209 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.872357+0000", 2024-08-19T14:25:25.209 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.6330000000000001e-06 2024-08-19T14:25:25.209 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.209 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.209 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:25:25.209 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.872406+0000", 2024-08-19T14:25:25.209 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.8958000000000001e-05 2024-08-19T14:25:25.209 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.209 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.209 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:25:25.209 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.872515+0000", 2024-08-19T14:25:25.209 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00010927199999999999 2024-08-19T14:25:25.209 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.210 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.210 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,7", 2024-08-19T14:25:25.210 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.872634+0000", 2024-08-19T14:25:25.210 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000118534 2024-08-19T14:25:25.210 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.210 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.210 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:25:25.210 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.876720+0000", 2024-08-19T14:25:25.210 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0040864370000000001 2024-08-19T14:25:25.210 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.210 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.210 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:25.210 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.878480+0000", 2024-08-19T14:25:25.211 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001760345 2024-08-19T14:25:25.211 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.211 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.211 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:25.211 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:43.091797+0000", 2024-08-19T14:25:25.211 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.21331655499999999 2024-08-19T14:25:25.211 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.211 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.211 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:25:25.211 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:43.091830+0000", 2024-08-19T14:25:25.211 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.3401000000000001e-05 2024-08-19T14:25:25.211 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.211 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.211 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:25:25.211 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:43.091861+0000", 2024-08-19T14:25:25.212 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.0870000000000001e-05 2024-08-19T14:25:25.212 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:25.212 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:25:25.212 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:25.212 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.212 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.212 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:165636 4.3 4:c997d720:::benchmark_data_smithi045_71562_object34461:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-08-19T14:25:25.212 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:22:42.884498+0000", 2024-08-19T14:25:25.212 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 162.239599352, 2024-08-19T14:25:25.212 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.22208514900000001, 2024-08-19T14:25:25.212 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:25:25.212 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:25:25.212 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:25:25.212 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:25:25.213 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:25:25.213 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:25:25.213 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 165636 2024-08-19T14:25:25.213 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.213 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:25:25.213 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.213 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:25:25.213 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.884498+0000", 2024-08-19T14:25:25.213 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:25:25.213 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.213 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.213 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:25:25.213 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.884498+0000", 2024-08-19T14:25:25.213 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:25:25.214 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.214 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.214 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:25:25.214 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.884499+0000", 2024-08-19T14:25:25.214 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.238e-06 2024-08-19T14:25:25.214 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.214 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.214 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:25:25.214 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.884504+0000", 2024-08-19T14:25:25.214 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2959999999999998e-06 2024-08-19T14:25:25.214 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.214 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.215 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:25:25.215 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.884515+0000", 2024-08-19T14:25:25.215 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0995999999999999e-05 2024-08-19T14:25:25.215 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.215 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.215 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:25:25.215 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.884523+0000", 2024-08-19T14:25:25.215 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.294e-06 2024-08-19T14:25:25.215 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.215 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.215 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:25:25.215 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.884580+0000", 2024-08-19T14:25:25.215 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7224e-05 2024-08-19T14:25:25.215 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.216 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.216 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:25:25.216 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.884704+0000", 2024-08-19T14:25:25.216 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000124046 2024-08-19T14:25:25.216 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.216 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.216 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,7", 2024-08-19T14:25:25.216 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.884836+0000", 2024-08-19T14:25:25.216 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000132478 2024-08-19T14:25:25.216 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.216 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.216 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:25:25.216 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.888660+0000", 2024-08-19T14:25:25.216 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0038240399999999999 2024-08-19T14:25:25.217 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.217 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.217 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:25.217 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.889616+0000", 2024-08-19T14:25:25.217 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00095516899999999996 2024-08-19T14:25:25.217 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.217 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.217 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:25.217 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:43.106482+0000", 2024-08-19T14:25:25.217 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.21686655299999999 2024-08-19T14:25:25.217 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.217 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.217 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:25:25.217 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:43.106539+0000", 2024-08-19T14:25:25.218 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.6366999999999998e-05 2024-08-19T14:25:25.218 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.218 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.218 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:25:25.218 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:43.106583+0000", 2024-08-19T14:25:25.218 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.4447999999999999e-05 2024-08-19T14:25:25.218 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:25.218 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:25:25.218 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:25.218 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.218 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.218 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:165645 4.3 4:c4ef6944:::benchmark_data_smithi045_71562_object34470:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-08-19T14:25:25.218 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:22:42.892140+0000", 2024-08-19T14:25:25.219 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 162.23195677699999, 2024-08-19T14:25:25.219 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.223849192, 2024-08-19T14:25:25.219 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:25:25.219 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:25:25.219 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:25:25.219 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:25:25.219 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:25:25.219 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:25:25.219 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 165645 2024-08-19T14:25:25.219 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.219 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:25:25.219 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.219 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:25:25.219 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.892140+0000", 2024-08-19T14:25:25.220 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:25:25.220 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.220 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.220 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:25:25.220 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.892140+0000", 2024-08-19T14:25:25.220 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:25:25.220 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.220 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.220 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:25:25.220 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.892142+0000", 2024-08-19T14:25:25.220 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.127e-06 2024-08-19T14:25:25.220 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.220 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.220 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:25:25.221 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.892146+0000", 2024-08-19T14:25:25.221 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.7949999999999998e-06 2024-08-19T14:25:25.221 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.221 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.221 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:25:25.221 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.892157+0000", 2024-08-19T14:25:25.221 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0387000000000001e-05 2024-08-19T14:25:25.221 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.221 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.221 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:25:25.221 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.892162+0000", 2024-08-19T14:25:25.221 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.6269999999999997e-06 2024-08-19T14:25:25.221 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.221 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.222 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:25:25.222 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.892203+0000", 2024-08-19T14:25:25.222 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.0238999999999999e-05 2024-08-19T14:25:25.222 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.222 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.222 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:25:25.222 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.892284+0000", 2024-08-19T14:25:25.222 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.1198000000000004e-05 2024-08-19T14:25:25.222 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.222 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.222 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,7", 2024-08-19T14:25:25.222 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.892359+0000", 2024-08-19T14:25:25.222 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4783000000000006e-05 2024-08-19T14:25:25.222 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.223 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.223 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:25:25.223 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.895461+0000", 2024-08-19T14:25:25.223 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0031028869999999999 2024-08-19T14:25:25.223 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.223 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.223 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:25.223 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.897428+0000", 2024-08-19T14:25:25.223 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0019669069999999999 2024-08-19T14:25:25.223 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.223 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.223 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:25.223 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:43.115899+0000", 2024-08-19T14:25:25.223 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.21847051100000001 2024-08-19T14:25:25.224 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.224 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.224 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:25:25.224 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:43.115943+0000", 2024-08-19T14:25:25.224 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.4181000000000003e-05 2024-08-19T14:25:25.224 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.224 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.224 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:25:25.224 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:43.115990+0000", 2024-08-19T14:25:25.224 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.655e-05 2024-08-19T14:25:25.224 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:25.224 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:25:25.224 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:25.224 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.225 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.225 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:210364 4.9 4:9e22782b:::benchmark_data_smithi045_71562_object79189:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e396)", 2024-08-19T14:25:25.225 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:23:33.561916+0000", 2024-08-19T14:25:25.225 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 111.56218161, 2024-08-19T14:25:25.225 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.21356883800000001, 2024-08-19T14:25:25.225 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:25:25.225 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:25:25.225 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:25:25.225 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:25:25.225 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:25:25.225 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:25:25.225 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 210364 2024-08-19T14:25:25.225 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.225 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:25:25.226 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.226 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:25:25.226 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:23:33.561916+0000", 2024-08-19T14:25:25.226 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:25:25.226 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.226 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.226 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:25:25.226 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:23:33.561916+0000", 2024-08-19T14:25:25.226 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:25:25.226 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.226 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.226 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:25:25.226 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:23:33.561916+0000", 2024-08-19T14:25:25.226 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.9999999999999997e-07 2024-08-19T14:25:25.227 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.227 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.227 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:25:25.227 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:23:33.561920+0000", 2024-08-19T14:25:25.227 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.6669999999999998e-06 2024-08-19T14:25:25.227 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.227 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.227 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:25:25.227 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:23:33.561929+0000", 2024-08-19T14:25:25.227 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.4700000000000008e-06 2024-08-19T14:25:25.227 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.227 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.227 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:25:25.228 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:23:33.561937+0000", 2024-08-19T14:25:25.228 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.1829999999999997e-06 2024-08-19T14:25:25.228 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.228 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.228 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:25:25.228 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:23:33.561977+0000", 2024-08-19T14:25:25.228 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.0119e-05 2024-08-19T14:25:25.228 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.228 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.228 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:25:25.228 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:23:33.562051+0000", 2024-08-19T14:25:25.228 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4740000000000006e-05 2024-08-19T14:25:25.228 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.228 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.228 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,3", 2024-08-19T14:25:25.229 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:23:33.562117+0000", 2024-08-19T14:25:25.229 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.5875000000000002e-05 2024-08-19T14:25:25.229 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.229 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.229 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:25:25.229 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:23:33.566295+0000", 2024-08-19T14:25:25.229 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0041779959999999998 2024-08-19T14:25:25.229 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.229 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.229 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:25.229 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:23:33.568994+0000", 2024-08-19T14:25:25.229 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0026991010000000002 2024-08-19T14:25:25.229 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.230 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.230 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:25.230 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:23:33.775415+0000", 2024-08-19T14:25:25.230 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20642043299999999 2024-08-19T14:25:25.230 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.230 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.230 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:25:25.230 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:23:33.775448+0000", 2024-08-19T14:25:25.230 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.3578999999999998e-05 2024-08-19T14:25:25.230 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.230 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.230 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:25:25.230 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:23:33.775484+0000", 2024-08-19T14:25:25.230 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.5975e-05 2024-08-19T14:25:25.231 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:25.231 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:25:25.231 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:25.231 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.231 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.231 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:210377 4.9 4:9c01a247:::benchmark_data_smithi045_71562_object79202:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e396)", 2024-08-19T14:25:25.231 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:23:33.573172+0000", 2024-08-19T14:25:25.231 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 111.550925435, 2024-08-19T14:25:25.231 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.212114408, 2024-08-19T14:25:25.231 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:25:25.231 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:25:25.231 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:25:25.231 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:25:25.231 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:25:25.232 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:25:25.232 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 210377 2024-08-19T14:25:25.232 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.232 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:25:25.232 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.232 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:25:25.232 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:23:33.573172+0000", 2024-08-19T14:25:25.232 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:25:25.232 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.232 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.232 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:25:25.232 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:23:33.573172+0000", 2024-08-19T14:25:25.232 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:25:25.232 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.233 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.233 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:25:25.233 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:23:33.573173+0000", 2024-08-19T14:25:25.233 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.204e-06 2024-08-19T14:25:25.233 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.233 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.233 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:25:25.233 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:23:33.573178+0000", 2024-08-19T14:25:25.233 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.8199999999999996e-06 2024-08-19T14:25:25.233 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.233 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.233 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:25:25.233 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:23:33.573192+0000", 2024-08-19T14:25:25.233 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3845e-05 2024-08-19T14:25:25.234 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.234 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.234 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:25:25.234 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:23:33.573199+0000", 2024-08-19T14:25:25.234 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.0299999999999996e-06 2024-08-19T14:25:25.234 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.234 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.234 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:25:25.234 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:23:33.573254+0000", 2024-08-19T14:25:25.234 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4882000000000001e-05 2024-08-19T14:25:25.234 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.234 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.234 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:25:25.234 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:23:33.573357+0000", 2024-08-19T14:25:25.235 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000103181 2024-08-19T14:25:25.235 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.235 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.235 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,3", 2024-08-19T14:25:25.235 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:23:33.573461+0000", 2024-08-19T14:25:25.235 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00010435200000000001 2024-08-19T14:25:25.235 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.235 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.235 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:25:25.235 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:23:33.578930+0000", 2024-08-19T14:25:25.235 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.005469186 2024-08-19T14:25:25.235 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.235 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.235 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:25.236 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:23:33.584484+0000", 2024-08-19T14:25:25.236 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0055532860000000002 2024-08-19T14:25:25.236 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.236 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.236 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:25.236 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:23:33.785202+0000", 2024-08-19T14:25:25.236 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20071872700000001 2024-08-19T14:25:25.236 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.236 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.236 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:25:25.236 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:23:33.785247+0000", 2024-08-19T14:25:25.236 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.4570000000000002e-05 2024-08-19T14:25:25.236 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.236 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.237 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:25:25.237 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:23:33.785286+0000", 2024-08-19T14:25:25.237 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.9325e-05 2024-08-19T14:25:25.237 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:25.237 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:25:25.237 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:25.237 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.237 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.237 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25813.0:58914 5.8 5:1ecf82f6:::benchmark_data_smithi045_85309_object58913:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e432)", 2024-08-19T14:25:25.237 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:25:11.286200+0000", 2024-08-19T14:25:25.237 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 13.837897351000001, 2024-08-19T14:25:25.237 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.079650823999999995, 2024-08-19T14:25:25.237 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:25:25.237 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:25:25.238 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:25:25.238 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:25:25.238 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25813", 2024-08-19T14:25:25.238 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:25:25.238 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 58914 2024-08-19T14:25:25.238 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.238 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:25:25.238 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.238 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:25:25.238 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:11.286200+0000", 2024-08-19T14:25:25.238 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:25:25.238 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.238 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.238 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:25:25.239 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:11.286200+0000", 2024-08-19T14:25:25.239 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:25:25.239 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.239 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.239 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:25:25.239 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:11.286201+0000", 2024-08-19T14:25:25.239 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3740000000000001e-06 2024-08-19T14:25:25.239 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.239 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.239 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:25:25.239 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:11.286224+0000", 2024-08-19T14:25:25.239 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2422e-05 2024-08-19T14:25:25.239 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.239 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.240 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:25:25.240 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:11.286236+0000", 2024-08-19T14:25:25.240 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2054e-05 2024-08-19T14:25:25.240 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.240 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.240 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:25:25.240 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:11.286244+0000", 2024-08-19T14:25:25.240 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.3950000000000001e-06 2024-08-19T14:25:25.240 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.240 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.240 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:25:25.240 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:11.286294+0000", 2024-08-19T14:25:25.240 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.9768999999999997e-05 2024-08-19T14:25:25.240 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.241 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.241 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:25:25.241 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:11.286412+0000", 2024-08-19T14:25:25.241 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000118162 2024-08-19T14:25:25.241 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.241 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.241 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,9", 2024-08-19T14:25:25.241 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:11.286503+0000", 2024-08-19T14:25:25.241 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.0678000000000002e-05 2024-08-19T14:25:25.241 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.241 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.241 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:25:25.241 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:11.289234+0000", 2024-08-19T14:25:25.241 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0027308329999999998 2024-08-19T14:25:25.242 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.242 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.242 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:25.242 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:11.289314+0000", 2024-08-19T14:25:25.242 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.0357000000000003e-05 2024-08-19T14:25:25.242 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.242 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.242 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:25.242 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:11.365717+0000", 2024-08-19T14:25:25.242 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.076403591000000007 2024-08-19T14:25:25.242 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.242 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.242 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:25:25.243 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:11.365792+0000", 2024-08-19T14:25:25.243 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4553000000000003e-05 2024-08-19T14:25:25.243 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.243 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.243 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:25:25.243 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:11.365851+0000", 2024-08-19T14:25:25.243 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.8635999999999999e-05 2024-08-19T14:25:25.243 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:25.243 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:25:25.243 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:25.243 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.243 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.243 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25813.0:59402 5.8 5:1f2551e9:::benchmark_data_smithi045_85309_object59401:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e432)", 2024-08-19T14:25:25.243 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:25:11.580030+0000", 2024-08-19T14:25:25.243 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 13.544067673000001, 2024-08-19T14:25:25.244 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.074530778000000006, 2024-08-19T14:25:25.244 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:25:25.244 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:25:25.244 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:25:25.244 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:25:25.244 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25813", 2024-08-19T14:25:25.244 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:25:25.244 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 59402 2024-08-19T14:25:25.244 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.244 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:25:25.244 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.244 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:25:25.244 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:11.580030+0000", 2024-08-19T14:25:25.244 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:25:25.245 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.245 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.245 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:25:25.245 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:11.580030+0000", 2024-08-19T14:25:25.245 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:25:25.245 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.245 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.245 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:25:25.245 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:11.580031+0000", 2024-08-19T14:25:25.245 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3459999999999999e-06 2024-08-19T14:25:25.245 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.245 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.245 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:25:25.245 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:11.580052+0000", 2024-08-19T14:25:25.246 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1549999999999999e-05 2024-08-19T14:25:25.246 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.246 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.246 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:25:25.246 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:11.580065+0000", 2024-08-19T14:25:25.246 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2713e-05 2024-08-19T14:25:25.246 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.246 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.246 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:25:25.246 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:11.580072+0000", 2024-08-19T14:25:25.246 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.1980000000000004e-06 2024-08-19T14:25:25.246 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.246 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.247 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:25:25.247 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:11.580553+0000", 2024-08-19T14:25:25.247 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000480485 2024-08-19T14:25:25.247 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.247 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.247 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:25:25.247 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:11.580933+0000", 2024-08-19T14:25:25.247 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00038044400000000002 2024-08-19T14:25:25.247 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.247 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.247 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,9", 2024-08-19T14:25:25.247 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:11.581947+0000", 2024-08-19T14:25:25.247 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001014137 2024-08-19T14:25:25.248 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.248 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.248 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:25:25.248 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:11.584073+0000", 2024-08-19T14:25:25.248 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0021252559999999998 2024-08-19T14:25:25.248 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.248 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.248 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:25.248 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:11.590620+0000", 2024-08-19T14:25:25.248 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0065469350000000003 2024-08-19T14:25:25.248 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.248 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.248 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:25.248 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:11.654488+0000", 2024-08-19T14:25:25.249 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.063868526999999994 2024-08-19T14:25:25.249 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.249 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.249 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:25:25.249 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:11.654526+0000", 2024-08-19T14:25:25.249 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.7827000000000002e-05 2024-08-19T14:25:25.249 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.249 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.249 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:25:25.249 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:11.654560+0000", 2024-08-19T14:25:25.249 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.4360000000000003e-05 2024-08-19T14:25:25.249 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:25.249 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:25:25.249 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:25.250 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.250 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.250 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25813.0:59410 5.8 5:1a2825dd:::benchmark_data_smithi045_85309_object59409:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e432)", 2024-08-19T14:25:25.250 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:25:11.585285+0000", 2024-08-19T14:25:25.250 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 13.538811839999999, 2024-08-19T14:25:25.250 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.073216353999999997, 2024-08-19T14:25:25.250 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:25:25.250 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:25:25.250 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:25:25.250 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:25:25.250 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25813", 2024-08-19T14:25:25.250 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:25:25.250 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 59410 2024-08-19T14:25:25.250 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.251 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:25:25.251 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.251 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:25:25.251 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:11.585285+0000", 2024-08-19T14:25:25.251 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:25:25.251 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.251 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.251 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:25:25.251 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:11.585285+0000", 2024-08-19T14:25:25.251 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:25:25.251 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.251 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.251 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:25:25.251 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:11.585287+0000", 2024-08-19T14:25:25.252 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5319999999999999e-06 2024-08-19T14:25:25.252 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.252 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.252 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:25:25.252 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:11.585312+0000", 2024-08-19T14:25:25.252 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.4851999999999999e-05 2024-08-19T14:25:25.252 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.252 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.252 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:25:25.252 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:11.585330+0000", 2024-08-19T14:25:25.252 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7790000000000001e-05 2024-08-19T14:25:25.252 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.252 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.252 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:25:25.253 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:11.585337+0000", 2024-08-19T14:25:25.253 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.8269999999999998e-06 2024-08-19T14:25:25.253 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.253 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.253 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:25:25.253 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:11.585391+0000", 2024-08-19T14:25:25.253 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4078e-05 2024-08-19T14:25:25.253 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.253 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.253 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:25:25.253 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:11.585532+0000", 2024-08-19T14:25:25.253 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000141051 2024-08-19T14:25:25.253 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.253 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.254 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,9", 2024-08-19T14:25:25.254 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:11.585676+0000", 2024-08-19T14:25:25.254 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014307999999999999 2024-08-19T14:25:25.254 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.254 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.254 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:25:25.254 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:11.590631+0000", 2024-08-19T14:25:25.254 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0049549670000000002 2024-08-19T14:25:25.254 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.254 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.254 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:25.254 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:11.590666+0000", 2024-08-19T14:25:25.254 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.5169000000000002e-05 2024-08-19T14:25:25.254 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.255 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.255 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:25.255 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:11.658389+0000", 2024-08-19T14:25:25.255 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.067723344000000005 2024-08-19T14:25:25.255 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.255 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.255 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:25:25.255 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:11.658451+0000", 2024-08-19T14:25:25.255 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.1523e-05 2024-08-19T14:25:25.255 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:25:25.255 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:25:25.255 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:25:25.255 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:25:11.658502+0000", 2024-08-19T14:25:25.255 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1140999999999997e-05 2024-08-19T14:25:25.256 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:25.256 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:25:25.256 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:25.256 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:25:25.256 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:25:25.256 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:25:25.286 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:25:25.316 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:25.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:25.317 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:25:25.322 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 37 16 81908 81892 138.307 92.0625 0.0109201 0.00722604 2024-08-19T14:25:25.418 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:25:25.454 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:25.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:25.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:25:25.556 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:25:25.590 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:25.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:25.592 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:25:25.692 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:25:25.694 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:25:25.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:25.729 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:25.729 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:25:25.830 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:25:25.862 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:25.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:25.863 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:25:25.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:25 smithi031 ceph-mon[16336]: pgmap v1002: 47 pgs: 47 active+clean; 4.7 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 115 MiB/s wr, 1.84k op/s 2024-08-19T14:25:25.963 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:25:25.995 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:25.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:25.996 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:25:26.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:25 smithi032 ceph-mon[20004]: pgmap v1002: 47 pgs: 47 active+clean; 4.7 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 115 MiB/s wr, 1.84k op/s 2024-08-19T14:25:26.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:25 smithi045 ceph-mon[20143]: pgmap v1002: 47 pgs: 47 active+clean; 4.7 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 115 MiB/s wr, 1.84k op/s 2024-08-19T14:25:26.097 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:25:26.120 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:26.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:26.121 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:25:26.222 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:25:26.245 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:26.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:26.246 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:25:26.322 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 38 16 83391 83375 137.107 92.6875 0.00622362 0.00728872 2024-08-19T14:25:26.347 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:25:26.375 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:26.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:26.376 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:25:26.429 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-08-19T14:25:26.477 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:25:26.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:26.514 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:26.514 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:25:26.596 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:25:26.596 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:25:26.597 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:25:26.597 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:25:26.597 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:25:26.597 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:25:26.597 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:25:26.597 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:25:26.597 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:25:26.597 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:25:26.597 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:25:26.597 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:25:26.598 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:25:26.598 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:25:26.601 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:25:26.602 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:25:26.615 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:25:26.637 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:25:26.637 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:25:26.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:26.648 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:26.648 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:25:26.668 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:25:26.669 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:25:26.697 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:25:26.697 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:25:26.739 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:25:26.740 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:25:26.749 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:25:26.783 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:26.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:26.784 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:25:26.884 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:25:26.915 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:26.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:26.916 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:25:27.017 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:25:27.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:26 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3302049999' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:25:27.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:27.057 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:27.058 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:25:27.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:26 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3302049999' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:25:27.158 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:25:27.190 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:27.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:27.192 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:25:27.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:26 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3302049999' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:25:27.292 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:25:27.322 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 39 16 84992 84976 136.157 100.062 0.00451955 0.00734011 2024-08-19T14:25:27.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:27.323 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:27.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:25:27.425 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:25:27.457 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:27.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:27.458 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:25:27.558 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:25:27.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:27.598 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:27.598 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:25:27.669 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:25:27.692 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-19T14:25:27.699 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:25:27.772 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:27.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:27.773 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:25:27.796 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:27 smithi031 ceph-mon[16336]: pgmap v1003: 47 pgs: 47 active+clean; 4.7 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 89 MiB/s wr, 1.43k op/s 2024-08-19T14:25:27.797 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:27 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1871055254' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]: dispatch 2024-08-19T14:25:27.797 INFO:teuthology.orchestra.run.smithi031.stderr:set pool 5 pg_num to 26 2024-08-19T14:25:27.874 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:25:27.902 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:27.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:27.903 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:25:28.004 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:25:28.035 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:28.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:28.036 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:25:28.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:27 smithi045 ceph-mon[20143]: pgmap v1003: 47 pgs: 47 active+clean; 4.7 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 89 MiB/s wr, 1.43k op/s 2024-08-19T14:25:28.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:27 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1871055254' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]: dispatch 2024-08-19T14:25:28.137 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:25:28.169 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:28.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:28.170 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:25:28.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:27 smithi032 ceph-mon[20004]: pgmap v1003: 47 pgs: 47 active+clean; 4.7 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 89 MiB/s wr, 1.43k op/s 2024-08-19T14:25:28.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:27 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1871055254' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]: dispatch 2024-08-19T14:25:28.271 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:25:28.301 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:28.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:28.302 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:25:28.322 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:2024-08-19T14:25:28.322891+0000 min lat: 0.00112215 max lat: 0.26573 avg lat: 0.00739053 2024-08-19T14:25:28.323 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-19T14:25:28.323 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 40 16 86578 86562 135.231 99.125 0.004658 0.00739053 2024-08-19T14:25:28.403 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:25:28.433 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:28.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:28.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:25:28.535 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:25:28.570 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:28.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:28.571 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:25:28.672 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:25:28.704 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:28.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:28.705 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:25:28.805 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:25:28.858 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:28.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:28.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:25:28.946 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:28 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1871055254' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]': finished 2024-08-19T14:25:28.946 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:28 smithi031 ceph-mon[16336]: osdmap e438: 12 total, 12 up, 9 in 2024-08-19T14:25:28.946 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:28 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-08-19T14:25:28.960 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:25:28.982 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:28.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:28.983 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:25:29.011 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:25:29.011 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:25:29.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:28 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1871055254' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]': finished 2024-08-19T14:25:29.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:28 smithi045 ceph-mon[20143]: osdmap e438: 12 total, 12 up, 9 in 2024-08-19T14:25:29.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:28 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-08-19T14:25:29.084 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:25:29.105 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:29.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:29.106 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:25:29.207 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:25:29.228 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:29.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:29.229 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:25:29.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:28 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1871055254' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]': finished 2024-08-19T14:25:29.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:28 smithi032 ceph-mon[20004]: osdmap e438: 12 total, 12 up, 9 in 2024-08-19T14:25:29.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:28 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-08-19T14:25:29.322 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 41 16 87401 87385 133.187 51.4375 0.0915655 0.00741375 2024-08-19T14:25:29.330 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:25:29.351 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:29.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:29.352 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:25:29.363 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:25:29.453 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:25:29.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:29.474 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:29.474 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:25:29.576 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:25:29.576 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:25:29.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:29.603 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:29.603 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:25:29.704 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:25:29.729 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:29.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:29.730 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:25:29.790 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-08-19T14:25:29.831 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:25:29.884 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:29.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:29.885 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:25:29.987 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:25:30.021 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:30.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:30.022 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:25:30.064 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:29 smithi031 ceph-mon[16336]: pgmap v1005: 47 pgs: 47 active+clean; 5.0 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 84 MiB/s wr, 1.34k op/s 2024-08-19T14:25:30.064 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:29 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:25:30.064 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:29 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:25:30.064 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:29 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]': finished 2024-08-19T14:25:30.064 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:29 smithi031 ceph-mon[16336]: osdmap e439: 12 total, 12 up, 9 in 2024-08-19T14:25:30.081 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:25:30.123 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:25:30.172 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:30.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:30.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:25:30.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:29 smithi032 ceph-mon[20004]: pgmap v1005: 47 pgs: 47 active+clean; 5.0 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 84 MiB/s wr, 1.34k op/s 2024-08-19T14:25:30.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:29 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:25:30.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:29 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:25:30.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:29 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]': finished 2024-08-19T14:25:30.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:29 smithi032 ceph-mon[20004]: osdmap e439: 12 total, 12 up, 9 in 2024-08-19T14:25:30.274 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:25:30.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:29 smithi045 ceph-mon[20143]: pgmap v1005: 47 pgs: 47 active+clean; 5.0 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 84 MiB/s wr, 1.34k op/s 2024-08-19T14:25:30.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:29 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:25:30.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:29 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:25:30.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:29 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]': finished 2024-08-19T14:25:30.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:29 smithi045 ceph-mon[20143]: osdmap e439: 12 total, 12 up, 9 in 2024-08-19T14:25:30.308 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:30.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:30.310 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:25:30.322 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 42 16 88417 88401 131.528 63.5 0.0149734 0.00759899 2024-08-19T14:25:30.410 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:25:30.440 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:30.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:30.442 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:25:30.542 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:25:30.578 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:30.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:30.579 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:25:30.680 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:25:30.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:30.712 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:30.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:25:30.814 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:25:30.817 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is unset 2024-08-19T14:25:30.843 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:30.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:30.844 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:25:30.945 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:25:30.954 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:30 smithi031 ceph-mon[16336]: osdmap e440: 12 total, 12 up, 9 in 2024-08-19T14:25:30.954 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:30 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-08-19T14:25:30.954 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:30 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3233725774' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:25:30.967 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:30.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:30.968 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:25:31.069 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:25:31.090 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:31.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:31.091 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:25:31.159 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:25:31.159 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:25:31.159 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:25:31.159 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:25:31.159 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:25:31.159 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:25:31.159 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:25:31.160 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:25:31.160 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:25:31.160 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:25:31.160 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:25:31.160 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:25:31.160 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:25:31.160 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:25:31.160 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:25:31.160 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:25:31.160 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:25:31.160 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:25:31.163 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:25:31.192 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:25:31.216 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:31.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:31.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:25:31.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:30 smithi032 ceph-mon[20004]: osdmap e440: 12 total, 12 up, 9 in 2024-08-19T14:25:31.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:30 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-08-19T14:25:31.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:30 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3233725774' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:25:31.294 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:25:31.294 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:25:31.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:30 smithi045 ceph-mon[20143]: osdmap e440: 12 total, 12 up, 9 in 2024-08-19T14:25:31.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:30 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-08-19T14:25:31.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:30 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3233725774' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:25:31.310 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:25:31.310 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:25:31.318 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:25:31.322 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 43 16 89504 89488 130.049 67.9375 0.00334877 0.00759772 2024-08-19T14:25:31.330 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:25:31.330 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:25:31.338 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:31.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:31.339 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:25:31.440 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:25:31.461 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:31.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:31.462 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:25:31.563 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:25:31.584 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:31.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:31.585 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:25:31.686 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:25:31.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:31.709 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:31.709 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:25:31.810 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:25:31.839 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:31.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:31.840 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:25:31.941 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:25:31.963 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:31.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:31.964 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:25:31.970 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:31 smithi031 ceph-mon[16336]: pgmap v1008: 57 pgs: 10 unknown, 47 active+clean; 5.2 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-08-19T14:25:31.970 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:31 smithi031 ceph-mon[16336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:25:31.970 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:31 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]': finished 2024-08-19T14:25:31.970 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:31 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3233725774' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:25:31.970 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:31 smithi031 ceph-mon[16336]: osdmap e441: 12 total, 12 up, 9 in 2024-08-19T14:25:31.970 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:31 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/337207697' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:25:31.970 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:31 smithi031 ceph-mon[16336]: 5.9 scrub starts 2024-08-19T14:25:31.970 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:31 smithi031 ceph-mon[16336]: 5.5 scrub starts 2024-08-19T14:25:31.970 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:31 smithi031 ceph-mon[16336]: 5.4 scrub starts 2024-08-19T14:25:31.970 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:31 smithi031 ceph-mon[16336]: 5.5 scrub ok 2024-08-19T14:25:31.971 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:31 smithi031 ceph-mon[16336]: 5.4 scrub ok 2024-08-19T14:25:31.971 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:31 smithi031 ceph-mon[16336]: osdmap e442: 12 total, 12 up, 9 in 2024-08-19T14:25:32.064 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:25:32.087 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:32.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:32.088 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:25:32.188 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:25:32.212 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:32.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:32.213 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:25:32.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:31 smithi032 ceph-mon[20004]: pgmap v1008: 57 pgs: 10 unknown, 47 active+clean; 5.2 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-08-19T14:25:32.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:31 smithi032 ceph-mon[20004]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:25:32.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:31 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]': finished 2024-08-19T14:25:32.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:31 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3233725774' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:25:32.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:31 smithi032 ceph-mon[20004]: osdmap e441: 12 total, 12 up, 9 in 2024-08-19T14:25:32.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:31 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/337207697' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:25:32.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:31 smithi032 ceph-mon[20004]: 5.9 scrub starts 2024-08-19T14:25:32.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:31 smithi032 ceph-mon[20004]: 5.5 scrub starts 2024-08-19T14:25:32.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:31 smithi032 ceph-mon[20004]: 5.4 scrub starts 2024-08-19T14:25:32.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:31 smithi032 ceph-mon[20004]: 5.5 scrub ok 2024-08-19T14:25:32.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:31 smithi032 ceph-mon[20004]: 5.4 scrub ok 2024-08-19T14:25:32.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:31 smithi032 ceph-mon[20004]: osdmap e442: 12 total, 12 up, 9 in 2024-08-19T14:25:32.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:31 smithi045 ceph-mon[20143]: pgmap v1008: 57 pgs: 10 unknown, 47 active+clean; 5.2 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-08-19T14:25:32.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:31 smithi045 ceph-mon[20143]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:25:32.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:31 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]': finished 2024-08-19T14:25:32.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:31 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3233725774' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:25:32.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:31 smithi045 ceph-mon[20143]: osdmap e441: 12 total, 12 up, 9 in 2024-08-19T14:25:32.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:31 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/337207697' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:25:32.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:31 smithi045 ceph-mon[20143]: 5.9 scrub starts 2024-08-19T14:25:32.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:31 smithi045 ceph-mon[20143]: 5.5 scrub starts 2024-08-19T14:25:32.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:31 smithi045 ceph-mon[20143]: 5.4 scrub starts 2024-08-19T14:25:32.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:31 smithi045 ceph-mon[20143]: 5.5 scrub ok 2024-08-19T14:25:32.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:31 smithi045 ceph-mon[20143]: 5.4 scrub ok 2024-08-19T14:25:32.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:31 smithi045 ceph-mon[20143]: osdmap e442: 12 total, 12 up, 9 in 2024-08-19T14:25:32.314 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:25:32.323 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 44 16 89504 89488 127.093 0 - 0.00759772 2024-08-19T14:25:32.334 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:32.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:32.335 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:25:32.417 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:25:32.418 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-08-19T14:25:32.418 INFO:teuthology.orchestra.run.smithi031.stdout: "complaint_time": 30, 2024-08-19T14:25:32.418 INFO:teuthology.orchestra.run.smithi031.stdout: "num_blocked_ops": 0 2024-08-19T14:25:32.418 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:25:32.436 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:25:32.457 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:32.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:32.459 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:25:32.560 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:25:32.581 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:32.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:32.582 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:25:32.683 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:25:32.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:32.705 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:32.706 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:25:32.807 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:25:32.830 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:32.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:32.831 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:25:32.932 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:25:32.964 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:32.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:32.965 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:25:33.066 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:25:33.092 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:33.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:33.093 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:25:33.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:32 smithi031 ceph-mon[16336]: 5.9 scrub ok 2024-08-19T14:25:33.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:32 smithi031 ceph-mon[16336]: 5.8 scrub starts 2024-08-19T14:25:33.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:32 smithi031 ceph-mon[16336]: 5.0 scrub starts 2024-08-19T14:25:33.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:32 smithi031 ceph-mon[16336]: 5.1 scrub starts 2024-08-19T14:25:33.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:32 smithi031 ceph-mon[16336]: 1.14 scrub starts 2024-08-19T14:25:33.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:32 smithi031 ceph-mon[16336]: 1.14 scrub ok 2024-08-19T14:25:33.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:32 smithi031 ceph-mon[16336]: 5.8 scrub ok 2024-08-19T14:25:33.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:32 smithi031 ceph-mon[16336]: 5.1 scrub ok 2024-08-19T14:25:33.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:32 smithi031 ceph-mon[16336]: 5.0 scrub ok 2024-08-19T14:25:33.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:32 smithi031 ceph-mon[16336]: 5.3 scrub starts 2024-08-19T14:25:33.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:32 smithi031 ceph-mon[16336]: 5.7 scrub starts 2024-08-19T14:25:33.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:32 smithi031 ceph-mon[16336]: 5.3 scrub ok 2024-08-19T14:25:33.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:32 smithi031 ceph-mon[16336]: 5.7 scrub ok 2024-08-19T14:25:33.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:32 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:25:33.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:32 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:25:33.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:32 smithi031 ceph-mon[16336]: osdmap e443: 12 total, 12 up, 9 in 2024-08-19T14:25:33.193 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:25:33.218 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-08-19T14:25:33.231 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:33.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:33.233 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:25:33.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:32 smithi032 ceph-mon[20004]: 5.9 scrub ok 2024-08-19T14:25:33.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:32 smithi032 ceph-mon[20004]: 5.8 scrub starts 2024-08-19T14:25:33.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:32 smithi032 ceph-mon[20004]: 5.0 scrub starts 2024-08-19T14:25:33.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:32 smithi032 ceph-mon[20004]: 5.1 scrub starts 2024-08-19T14:25:33.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:32 smithi032 ceph-mon[20004]: 1.14 scrub starts 2024-08-19T14:25:33.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:32 smithi032 ceph-mon[20004]: 1.14 scrub ok 2024-08-19T14:25:33.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:32 smithi032 ceph-mon[20004]: 5.8 scrub ok 2024-08-19T14:25:33.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:32 smithi032 ceph-mon[20004]: 5.1 scrub ok 2024-08-19T14:25:33.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:32 smithi032 ceph-mon[20004]: 5.0 scrub ok 2024-08-19T14:25:33.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:32 smithi032 ceph-mon[20004]: 5.3 scrub starts 2024-08-19T14:25:33.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:32 smithi032 ceph-mon[20004]: 5.7 scrub starts 2024-08-19T14:25:33.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:32 smithi032 ceph-mon[20004]: 5.3 scrub ok 2024-08-19T14:25:33.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:32 smithi032 ceph-mon[20004]: 5.7 scrub ok 2024-08-19T14:25:33.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:32 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:25:33.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:32 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:25:33.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:32 smithi032 ceph-mon[20004]: osdmap e443: 12 total, 12 up, 9 in 2024-08-19T14:25:33.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:32 smithi045 ceph-mon[20143]: 5.9 scrub ok 2024-08-19T14:25:33.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:32 smithi045 ceph-mon[20143]: 5.8 scrub starts 2024-08-19T14:25:33.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:32 smithi045 ceph-mon[20143]: 5.0 scrub starts 2024-08-19T14:25:33.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:32 smithi045 ceph-mon[20143]: 5.1 scrub starts 2024-08-19T14:25:33.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:32 smithi045 ceph-mon[20143]: 1.14 scrub starts 2024-08-19T14:25:33.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:32 smithi045 ceph-mon[20143]: 1.14 scrub ok 2024-08-19T14:25:33.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:32 smithi045 ceph-mon[20143]: 5.8 scrub ok 2024-08-19T14:25:33.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:32 smithi045 ceph-mon[20143]: 5.1 scrub ok 2024-08-19T14:25:33.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:32 smithi045 ceph-mon[20143]: 5.0 scrub ok 2024-08-19T14:25:33.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:32 smithi045 ceph-mon[20143]: 5.3 scrub starts 2024-08-19T14:25:33.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:32 smithi045 ceph-mon[20143]: 5.7 scrub starts 2024-08-19T14:25:33.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:32 smithi045 ceph-mon[20143]: 5.3 scrub ok 2024-08-19T14:25:33.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:32 smithi045 ceph-mon[20143]: 5.7 scrub ok 2024-08-19T14:25:33.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:32 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:25:33.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:32 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:25:33.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:32 smithi045 ceph-mon[20143]: osdmap e443: 12 total, 12 up, 9 in 2024-08-19T14:25:33.323 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 45 16 89655 89639 124.479 4.71875 0.037586 0.00802729 2024-08-19T14:25:33.334 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:25:33.379 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:33.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:33.381 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:25:33.481 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:25:33.539 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:33.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:33.540 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:25:33.641 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:25:33.672 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:33.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:33.673 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:25:33.723 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-19T14:25:33.730 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 7, 8, 9, 11] out_osds: [10, 6, 2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-08-19T14:25:33.730 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-19T14:25:33.730 INFO:tasks.thrashosds.thrasher:inject_pause on osd.7 2024-08-19T14:25:33.730 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-08-19T14:25:33.730 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-08-19T14:25:33.731 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.7 config set filestore_inject_stall 3 2024-08-19T14:25:33.773 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:25:33.783 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:25:33.798 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:33.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:33.800 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:25:33.901 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:25:33.926 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:33.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:33.927 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:25:33.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:33 smithi031 ceph-mon[16336]: pgmap v1011: 57 pgs: 10 unknown, 47 active+clean; 4.4 GiB data, 21 GiB used, 784 GiB / 805 GiB avail 2024-08-19T14:25:34.027 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:25:34.068 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:34.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:34.069 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:25:34.137 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:25:34.170 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:25:34.197 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:34.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:34.198 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:25:34.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:33 smithi032 ceph-mon[20004]: pgmap v1011: 57 pgs: 10 unknown, 47 active+clean; 4.4 GiB data, 21 GiB used, 784 GiB / 805 GiB avail 2024-08-19T14:25:34.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:33 smithi045 ceph-mon[20143]: pgmap v1011: 57 pgs: 10 unknown, 47 active+clean; 4.4 GiB data, 21 GiB used, 784 GiB / 805 GiB avail 2024-08-19T14:25:34.299 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:25:34.323 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 46 16 90515 90499 122.941 53.75 0.0113287 0.00812849 2024-08-19T14:25:34.328 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:34.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:34.329 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:25:34.430 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:25:34.455 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:34.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:34.456 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:25:34.460 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:25:34.460 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 20, 2024-08-19T14:25:34.460 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 600, 2024-08-19T14:25:34.461 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [ 2024-08-19T14:25:34.461 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.461 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:241314 2.1 2:8e6fec04:::benchmark_data_smithi045_46415_object118033:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-08-19T14:25:34.461 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:16:40.787553+0000", 2024-08-19T14:25:34.461 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 533.66955075500005, 2024-08-19T14:25:34.461 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.29700766899999997, 2024-08-19T14:25:34.461 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:25:34.461 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:25:34.461 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:25:34.461 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:25:34.461 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:25:34.461 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:25:34.461 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 241314 2024-08-19T14:25:34.462 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.462 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:25:34.462 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.462 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:25:34.462 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.787553+0000", 2024-08-19T14:25:34.462 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:25:34.462 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.462 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.462 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:25:34.462 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.787553+0000", 2024-08-19T14:25:34.462 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:25:34.462 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.463 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.463 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:25:34.463 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.787554+0000", 2024-08-19T14:25:34.463 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5770000000000001e-06 2024-08-19T14:25:34.463 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.463 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.463 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:25:34.463 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.787560+0000", 2024-08-19T14:25:34.463 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.6930000000000004e-06 2024-08-19T14:25:34.463 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.463 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.463 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:25:34.463 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.787574+0000", 2024-08-19T14:25:34.463 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3828e-05 2024-08-19T14:25:34.464 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.464 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.464 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:25:34.464 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.787582+0000", 2024-08-19T14:25:34.464 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.225e-06 2024-08-19T14:25:34.464 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.464 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.464 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:25:34.464 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.788601+0000", 2024-08-19T14:25:34.464 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.001018833 2024-08-19T14:25:34.464 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.464 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.464 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:25:34.464 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.788734+0000", 2024-08-19T14:25:34.465 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013335699999999999 2024-08-19T14:25:34.465 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.465 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.465 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 1,2,8", 2024-08-19T14:25:34.465 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.788881+0000", 2024-08-19T14:25:34.465 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000146644 2024-08-19T14:25:34.465 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.465 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.465 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:34.465 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.793613+0000", 2024-08-19T14:25:34.465 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0047319600000000003 2024-08-19T14:25:34.465 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.465 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.465 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:34.466 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.795531+0000", 2024-08-19T14:25:34.466 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.001918335 2024-08-19T14:25:34.466 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.466 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.466 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:25:34.466 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.796093+0000", 2024-08-19T14:25:34.466 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00056226599999999996 2024-08-19T14:25:34.466 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.466 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.466 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:34.466 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.084346+0000", 2024-08-19T14:25:34.466 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.28825296 2024-08-19T14:25:34.466 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.466 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.467 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:25:34.467 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.084417+0000", 2024-08-19T14:25:34.467 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.0599999999999995e-05 2024-08-19T14:25:34.467 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.467 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.467 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:25:34.467 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.084560+0000", 2024-08-19T14:25:34.467 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00014339100000000001 2024-08-19T14:25:34.467 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:25:34.467 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:25:34.467 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:25:34.467 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.467 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.467 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:241325 2.1 2:8f07de09:::benchmark_data_smithi045_46415_object118044:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-08-19T14:25:34.468 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:16:40.798902+0000", 2024-08-19T14:25:34.468 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 533.65820170500001, 2024-08-19T14:25:34.468 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.29144186999999999, 2024-08-19T14:25:34.468 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:25:34.468 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:25:34.468 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:25:34.468 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:25:34.468 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:25:34.468 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:25:34.468 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 241325 2024-08-19T14:25:34.469 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.469 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:25:34.469 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.469 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:25:34.469 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.798902+0000", 2024-08-19T14:25:34.469 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:25:34.469 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.469 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.469 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:25:34.469 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.798902+0000", 2024-08-19T14:25:34.469 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:25:34.469 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.469 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.469 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:25:34.470 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.798904+0000", 2024-08-19T14:25:34.470 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.1780000000000002e-06 2024-08-19T14:25:34.470 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.470 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.470 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:25:34.470 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.798914+0000", 2024-08-19T14:25:34.470 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.7480000000000005e-06 2024-08-19T14:25:34.470 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.470 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.470 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:25:34.470 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.798933+0000", 2024-08-19T14:25:34.470 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.9026e-05 2024-08-19T14:25:34.470 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.471 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.471 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:25:34.471 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.798942+0000", 2024-08-19T14:25:34.471 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.3200000000000006e-06 2024-08-19T14:25:34.471 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.471 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.471 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:25:34.471 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.799163+0000", 2024-08-19T14:25:34.471 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000221337 2024-08-19T14:25:34.471 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.471 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.471 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:25:34.471 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.799310+0000", 2024-08-19T14:25:34.471 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00014677399999999999 2024-08-19T14:25:34.472 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.472 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.472 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 1,2,8", 2024-08-19T14:25:34.472 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.799452+0000", 2024-08-19T14:25:34.472 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00014219499999999999 2024-08-19T14:25:34.472 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.472 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.472 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:34.472 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.804505+0000", 2024-08-19T14:25:34.472 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.005052384 2024-08-19T14:25:34.472 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.472 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.472 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:25:34.473 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.804768+0000", 2024-08-19T14:25:34.473 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00026380700000000002 2024-08-19T14:25:34.473 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.473 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.473 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:34.473 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.805302+0000", 2024-08-19T14:25:34.473 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00053321300000000005 2024-08-19T14:25:34.473 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.473 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.473 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:34.473 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.090219+0000", 2024-08-19T14:25:34.473 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.284917107 2024-08-19T14:25:34.473 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.473 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.474 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:25:34.474 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.090271+0000", 2024-08-19T14:25:34.474 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.2289e-05 2024-08-19T14:25:34.474 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.474 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.474 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:25:34.474 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.090343+0000", 2024-08-19T14:25:34.474 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.2491999999999996e-05 2024-08-19T14:25:34.474 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:25:34.474 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:25:34.474 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:25:34.474 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.474 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.475 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25057.0:47774 3.11 3:8e95e964:::benchmark_data_smithi045_59973_object47773:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e234)", 2024-08-19T14:25:34.475 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:17:18.724217+0000", 2024-08-19T14:25:34.475 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 495.73288608799999, 2024-08-19T14:25:34.475 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0342696659999999, 2024-08-19T14:25:34.475 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:25:34.475 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:25:34.475 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:25:34.475 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:25:34.476 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25057", 2024-08-19T14:25:34.476 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:25:34.476 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 47774 2024-08-19T14:25:34.476 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.476 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:25:34.476 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.476 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:25:34.476 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724217+0000", 2024-08-19T14:25:34.476 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:25:34.476 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.476 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.476 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:25:34.476 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724217+0000", 2024-08-19T14:25:34.476 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:25:34.477 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.477 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.477 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:25:34.477 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724218+0000", 2024-08-19T14:25:34.477 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.4499999999999997e-07 2024-08-19T14:25:34.477 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.477 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.477 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:25:34.477 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724254+0000", 2024-08-19T14:25:34.477 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.5914000000000002e-05 2024-08-19T14:25:34.477 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.477 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.477 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:25:34.478 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724263+0000", 2024-08-19T14:25:34.478 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.4290000000000007e-06 2024-08-19T14:25:34.478 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.479 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.479 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:25:34.479 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724270+0000", 2024-08-19T14:25:34.479 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.7959999999999998e-06 2024-08-19T14:25:34.479 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.479 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.479 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:25:34.480 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.774143+0000", 2024-08-19T14:25:34.480 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.049872861999999997 2024-08-19T14:25:34.480 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.480 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.480 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:25:34.480 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.774151+0000", 2024-08-19T14:25:34.480 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.6489999999999997e-06 2024-08-19T14:25:34.480 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.480 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.480 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:25:34.480 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.758417+0000", 2024-08-19T14:25:34.480 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.98426575800000005 2024-08-19T14:25:34.480 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.480 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.481 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:25:34.481 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.758487+0000", 2024-08-19T14:25:34.481 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.9713000000000002e-05 2024-08-19T14:25:34.481 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:25:34.481 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:25:34.481 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:25:34.481 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.481 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.481 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25057.0:47788 3.11 3:8e23edc0:::benchmark_data_smithi045_59973_object47787:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e234)", 2024-08-19T14:25:34.481 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:17:18.724282+0000", 2024-08-19T14:25:34.482 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 495.73282124299999, 2024-08-19T14:25:34.482 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0343071210000001, 2024-08-19T14:25:34.482 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:25:34.482 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:25:34.482 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:25:34.482 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:25:34.482 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25057", 2024-08-19T14:25:34.482 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:25:34.482 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 47788 2024-08-19T14:25:34.482 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.482 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:25:34.482 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.482 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:25:34.482 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724282+0000", 2024-08-19T14:25:34.483 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:25:34.483 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.483 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.483 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:25:34.483 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724282+0000", 2024-08-19T14:25:34.483 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:25:34.483 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.483 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.483 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:25:34.483 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724283+0000", 2024-08-19T14:25:34.483 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.6100000000000001e-07 2024-08-19T14:25:34.483 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.483 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.483 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:25:34.484 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724318+0000", 2024-08-19T14:25:34.484 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.578e-05 2024-08-19T14:25:34.484 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.484 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.484 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:25:34.484 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724326+0000", 2024-08-19T14:25:34.484 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.4170000000000003e-06 2024-08-19T14:25:34.484 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.484 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.484 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:25:34.484 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724331+0000", 2024-08-19T14:25:34.484 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.5820000000000001e-06 2024-08-19T14:25:34.484 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.484 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.485 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:25:34.485 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.774172+0000", 2024-08-19T14:25:34.485 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.049840256999999999 2024-08-19T14:25:34.485 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.485 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.485 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:25:34.485 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.774177+0000", 2024-08-19T14:25:34.485 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.5160000000000002e-06 2024-08-19T14:25:34.485 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.485 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.485 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:25:34.485 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.758533+0000", 2024-08-19T14:25:34.485 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.98435635099999996 2024-08-19T14:25:34.485 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.485 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.486 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:25:34.486 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.758589+0000", 2024-08-19T14:25:34.486 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.5757000000000002e-05 2024-08-19T14:25:34.486 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:25:34.486 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:25:34.486 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:25:34.486 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.486 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.486 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25057.0:47814 3.11 3:8918a6df:::benchmark_data_smithi045_59973_object47813:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e234)", 2024-08-19T14:25:34.486 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:17:18.724406+0000", 2024-08-19T14:25:34.486 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 495.732697804, 2024-08-19T14:25:34.487 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0342602160000001, 2024-08-19T14:25:34.487 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:25:34.487 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:25:34.487 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:25:34.487 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:25:34.487 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25057", 2024-08-19T14:25:34.487 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:25:34.487 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 47814 2024-08-19T14:25:34.487 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.487 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:25:34.487 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.488 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:25:34.488 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724406+0000", 2024-08-19T14:25:34.488 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:25:34.488 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.488 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.488 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:25:34.488 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724406+0000", 2024-08-19T14:25:34.488 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:25:34.488 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.488 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.488 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:25:34.488 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724406+0000", 2024-08-19T14:25:34.488 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.0200000000000003e-07 2024-08-19T14:25:34.488 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.489 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.489 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:25:34.489 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724440+0000", 2024-08-19T14:25:34.489 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.3689000000000002e-05 2024-08-19T14:25:34.489 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.489 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.489 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:25:34.489 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724449+0000", 2024-08-19T14:25:34.489 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.0469999999999999e-06 2024-08-19T14:25:34.489 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.489 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.489 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:25:34.489 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724455+0000", 2024-08-19T14:25:34.489 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.1190000000000002e-06 2024-08-19T14:25:34.490 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.490 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.490 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:25:34.490 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.774196+0000", 2024-08-19T14:25:34.490 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.049740924999999998 2024-08-19T14:25:34.490 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.490 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.490 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:25:34.490 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.774201+0000", 2024-08-19T14:25:34.490 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.3090000000000003e-06 2024-08-19T14:25:34.490 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.490 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.490 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:25:34.491 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.758615+0000", 2024-08-19T14:25:34.491 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.98441362200000004 2024-08-19T14:25:34.491 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.491 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.491 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:25:34.491 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.758666+0000", 2024-08-19T14:25:34.491 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1103000000000002e-05 2024-08-19T14:25:34.491 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:25:34.491 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:25:34.491 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:25:34.491 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.491 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.491 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25057.0:47832 3.11 3:8d79f59c:::benchmark_data_smithi045_59973_object47831:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e234)", 2024-08-19T14:25:34.491 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:17:18.724649+0000", 2024-08-19T14:25:34.492 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 495.73245458899999, 2024-08-19T14:25:34.492 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0340940759999999, 2024-08-19T14:25:34.492 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:25:34.492 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:25:34.492 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:25:34.492 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:25:34.492 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25057", 2024-08-19T14:25:34.492 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:25:34.492 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 47832 2024-08-19T14:25:34.492 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.492 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:25:34.492 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.492 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:25:34.492 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724649+0000", 2024-08-19T14:25:34.493 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:25:34.493 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.493 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.493 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:25:34.493 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724649+0000", 2024-08-19T14:25:34.493 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:25:34.493 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.493 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.493 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:25:34.493 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724649+0000", 2024-08-19T14:25:34.493 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0399999999999996e-07 2024-08-19T14:25:34.493 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.493 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.494 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:25:34.494 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724680+0000", 2024-08-19T14:25:34.494 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.0477e-05 2024-08-19T14:25:34.494 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.494 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.494 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:25:34.494 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724688+0000", 2024-08-19T14:25:34.494 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.7509999999999997e-06 2024-08-19T14:25:34.494 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.495 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.495 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:25:34.495 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.724694+0000", 2024-08-19T14:25:34.495 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.7679999999999997e-06 2024-08-19T14:25:34.495 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.495 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.495 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:25:34.495 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.774220+0000", 2024-08-19T14:25:34.495 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.049526175999999998 2024-08-19T14:25:34.495 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.495 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.495 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:25:34.495 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.774226+0000", 2024-08-19T14:25:34.496 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.2820000000000004e-06 2024-08-19T14:25:34.496 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.496 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.496 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:25:34.496 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.758693+0000", 2024-08-19T14:25:34.496 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.98446715699999998 2024-08-19T14:25:34.496 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.496 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.496 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:25:34.496 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.758743+0000", 2024-08-19T14:25:34.496 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.9960999999999998e-05 2024-08-19T14:25:34.496 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:25:34.496 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:25:34.497 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:25:34.497 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.497 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.497 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25057.0:47838 3.16 3:68f5225c:::benchmark_data_smithi045_59973_object47837:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e234)", 2024-08-19T14:25:34.497 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:17:18.777436+0000", 2024-08-19T14:25:34.497 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 495.67966702000001, 2024-08-19T14:25:34.497 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.98807081600000002, 2024-08-19T14:25:34.497 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:25:34.497 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:25:34.497 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:25:34.497 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:25:34.497 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25057", 2024-08-19T14:25:34.497 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:25:34.498 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 47838 2024-08-19T14:25:34.498 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.498 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:25:34.498 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.498 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:25:34.498 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.777436+0000", 2024-08-19T14:25:34.498 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:25:34.498 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.498 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.498 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:25:34.498 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.777436+0000", 2024-08-19T14:25:34.498 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:25:34.499 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.499 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.499 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:25:34.499 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.777438+0000", 2024-08-19T14:25:34.499 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.7489999999999999e-06 2024-08-19T14:25:34.499 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.499 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.499 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:25:34.499 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.777473+0000", 2024-08-19T14:25:34.499 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.5284999999999998e-05 2024-08-19T14:25:34.499 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.499 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.499 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:25:34.500 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.777492+0000", 2024-08-19T14:25:34.500 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.8853999999999999e-05 2024-08-19T14:25:34.500 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.500 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.500 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:25:34.500 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.777500+0000", 2024-08-19T14:25:34.500 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.9759999999999995e-06 2024-08-19T14:25:34.500 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.500 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.500 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:25:34.500 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.778853+0000", 2024-08-19T14:25:34.500 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0013526460000000001 2024-08-19T14:25:34.500 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.500 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.501 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:25:34.501 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.778867+0000", 2024-08-19T14:25:34.501 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4448e-05 2024-08-19T14:25:34.501 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.501 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.501 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:25:34.501 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.759774+0000", 2024-08-19T14:25:34.501 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.98090639700000004 2024-08-19T14:25:34.501 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.501 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.501 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:25:34.501 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.759902+0000", 2024-08-19T14:25:34.501 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000128128 2024-08-19T14:25:34.502 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.502 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.502 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 1,4", 2024-08-19T14:25:34.502 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.760062+0000", 2024-08-19T14:25:34.502 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000159956 2024-08-19T14:25:34.502 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.502 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.502 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:34.502 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.762325+0000", 2024-08-19T14:25:34.502 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0022629880000000001 2024-08-19T14:25:34.502 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.502 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.502 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:25:34.503 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.763087+0000", 2024-08-19T14:25:34.503 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00076208899999999997 2024-08-19T14:25:34.503 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.503 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.503 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:34.503 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.765400+0000", 2024-08-19T14:25:34.503 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0023133120000000001 2024-08-19T14:25:34.503 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.503 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.503 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:25:34.503 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.765462+0000", 2024-08-19T14:25:34.503 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.2048e-05 2024-08-19T14:25:34.503 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.504 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.504 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:25:34.504 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.765507+0000", 2024-08-19T14:25:34.504 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.494e-05 2024-08-19T14:25:34.504 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:25:34.504 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:25:34.504 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:25:34.504 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.504 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.504 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25057.0:47845 3.16 3:6c285014:::benchmark_data_smithi045_59973_object47844:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e234)", 2024-08-19T14:25:34.504 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:17:18.799489+0000", 2024-08-19T14:25:34.504 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 495.65761471600001, 2024-08-19T14:25:34.504 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.96674353099999999, 2024-08-19T14:25:34.505 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:25:34.505 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:25:34.505 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:25:34.505 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:25:34.505 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25057", 2024-08-19T14:25:34.505 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:25:34.505 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 47845 2024-08-19T14:25:34.506 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:25:34.506 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.506 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:25:34.506 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.506 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:25:34.507 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.799489+0000", 2024-08-19T14:25:34.507 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:25:34.507 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.507 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.507 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:25:34.507 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.799489+0000", 2024-08-19T14:25:34.507 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:25:34.507 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.507 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.507 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:25:34.507 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.799490+0000", 2024-08-19T14:25:34.507 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.779e-06 2024-08-19T14:25:34.508 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.508 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.508 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:25:34.508 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.799530+0000", 2024-08-19T14:25:34.508 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.9555000000000003e-05 2024-08-19T14:25:34.508 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.508 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.508 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:25:34.508 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.799544+0000", 2024-08-19T14:25:34.508 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4336e-05 2024-08-19T14:25:34.508 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.508 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.508 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:25:34.509 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.799554+0000", 2024-08-19T14:25:34.509 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.5580000000000006e-06 2024-08-19T14:25:34.509 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.509 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.509 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:25:34.509 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.799613+0000", 2024-08-19T14:25:34.509 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.8995e-05 2024-08-19T14:25:34.509 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.509 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.509 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:25:34.509 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.799625+0000", 2024-08-19T14:25:34.510 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2088e-05 2024-08-19T14:25:34.510 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.510 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.510 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:25:34.510 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.760956+0000", 2024-08-19T14:25:34.510 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.96133111299999996 2024-08-19T14:25:34.510 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.510 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.510 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:25:34.510 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.761072+0000", 2024-08-19T14:25:34.510 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000116173 2024-08-19T14:25:34.510 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.510 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.511 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 1,4", 2024-08-19T14:25:34.511 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.761217+0000", 2024-08-19T14:25:34.511 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000144308 2024-08-19T14:25:34.511 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.511 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.511 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:34.511 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.763046+0000", 2024-08-19T14:25:34.511 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00182978 2024-08-19T14:25:34.511 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.511 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.511 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:25:34.511 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.763278+0000", 2024-08-19T14:25:34.511 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00023194700000000001 2024-08-19T14:25:34.512 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.512 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.512 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:34.512 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.766122+0000", 2024-08-19T14:25:34.512 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0028441439999999998 2024-08-19T14:25:34.512 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.512 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.512 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:25:34.512 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.766190+0000", 2024-08-19T14:25:34.512 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.7595999999999995e-05 2024-08-19T14:25:34.512 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.512 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.513 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:25:34.513 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.766232+0000", 2024-08-19T14:25:34.513 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2159000000000001e-05 2024-08-19T14:25:34.513 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:25:34.513 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:25:34.513 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:25:34.513 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.513 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.513 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25057.0:64601 3.1f 3.eafa94bf (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e246)", 2024-08-19T14:25:34.513 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:17:35.967173+0000", 2024-08-19T14:25:34.513 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 478.48993071400002, 2024-08-19T14:25:34.513 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.99490361100000002, 2024-08-19T14:25:34.513 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:25:34.514 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:25:34.514 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:25:34.514 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:25:34.514 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25057", 2024-08-19T14:25:34.514 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:25:34.514 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 64601 2024-08-19T14:25:34.514 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.514 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:25:34.514 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.514 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:25:34.514 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:35.967173+0000", 2024-08-19T14:25:34.514 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:25:34.514 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.514 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.515 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:25:34.515 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:35.967173+0000", 2024-08-19T14:25:34.515 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:25:34.515 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.515 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.515 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:25:34.515 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:35.967174+0000", 2024-08-19T14:25:34.515 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.7740000000000001e-06 2024-08-19T14:25:34.515 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.515 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.515 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:25:34.515 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:35.967205+0000", 2024-08-19T14:25:34.515 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.0550999999999999e-05 2024-08-19T14:25:34.516 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.516 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.516 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:25:34.516 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:35.967214+0000", 2024-08-19T14:25:34.516 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.1430000000000004e-06 2024-08-19T14:25:34.516 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.516 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.516 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:25:34.516 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:35.967221+0000", 2024-08-19T14:25:34.516 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.0199999999999997e-06 2024-08-19T14:25:34.516 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.516 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.516 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:25:34.517 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:35.967334+0000", 2024-08-19T14:25:34.517 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000112799 2024-08-19T14:25:34.517 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.517 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.517 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:25:34.517 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:35.967341+0000", 2024-08-19T14:25:34.517 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.3170000000000004e-06 2024-08-19T14:25:34.517 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.517 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.517 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:25:34.517 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:36.962066+0000", 2024-08-19T14:25:34.517 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.99472444500000001 2024-08-19T14:25:34.517 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.518 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.518 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:25:34.518 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:36.962076+0000", 2024-08-19T14:25:34.518 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0562e-05 2024-08-19T14:25:34.518 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:25:34.518 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:25:34.518 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:25:34.518 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.518 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.518 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25057.0:64632 3.1f 3.fd125fbf (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e246)", 2024-08-19T14:25:34.518 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:17:36.181164+0000", 2024-08-19T14:25:34.518 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 478.27593891399999, 2024-08-19T14:25:34.518 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.78094885599999997, 2024-08-19T14:25:34.518 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:25:34.519 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:25:34.519 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:25:34.519 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:25:34.519 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25057", 2024-08-19T14:25:34.519 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:25:34.519 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 64632 2024-08-19T14:25:34.519 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.519 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:25:34.519 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.519 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:25:34.519 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:36.181164+0000", 2024-08-19T14:25:34.519 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:25:34.519 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.519 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.520 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:25:34.520 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:36.181164+0000", 2024-08-19T14:25:34.520 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:25:34.520 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.520 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.520 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:25:34.520 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:36.181166+0000", 2024-08-19T14:25:34.520 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.001e-06 2024-08-19T14:25:34.520 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.520 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.520 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:25:34.520 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:36.181210+0000", 2024-08-19T14:25:34.520 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4035000000000001e-05 2024-08-19T14:25:34.520 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.521 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.521 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:25:34.521 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:36.181225+0000", 2024-08-19T14:25:34.521 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4620000000000001e-05 2024-08-19T14:25:34.521 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.521 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.521 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:25:34.521 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:36.181235+0000", 2024-08-19T14:25:34.521 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.6709999999999994e-06 2024-08-19T14:25:34.521 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.521 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.521 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:25:34.521 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:36.181309+0000", 2024-08-19T14:25:34.521 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.4191999999999997e-05 2024-08-19T14:25:34.522 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.522 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.522 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:25:34.522 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:36.181321+0000", 2024-08-19T14:25:34.522 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2026e-05 2024-08-19T14:25:34.522 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.522 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.522 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:25:34.522 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:36.962105+0000", 2024-08-19T14:25:34.522 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.780784382 2024-08-19T14:25:34.522 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.522 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.522 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:25:34.523 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:36.962113+0000", 2024-08-19T14:25:34.523 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.9289999999999998e-06 2024-08-19T14:25:34.523 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:25:34.523 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:25:34.523 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:25:34.523 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.523 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.523 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25057.0:188741 3.26 3:65f5d80b:::benchmark_data_smithi045_59973_object57562:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e296)", 2024-08-19T14:25:34.523 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:19:02.154496+0000", 2024-08-19T14:25:34.523 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 392.30260691500001, 2024-08-19T14:25:34.523 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.047062189, 2024-08-19T14:25:34.523 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:25:34.523 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:25:34.524 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:25:34.524 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:25:34.524 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25057", 2024-08-19T14:25:34.524 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:25:34.524 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 188741 2024-08-19T14:25:34.524 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.524 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:25:34.524 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.524 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:25:34.524 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.154496+0000", 2024-08-19T14:25:34.524 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:25:34.524 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.524 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.525 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:25:34.525 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.154496+0000", 2024-08-19T14:25:34.525 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:25:34.525 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.525 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.525 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:25:34.525 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.154497+0000", 2024-08-19T14:25:34.525 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.5000000000000001e-07 2024-08-19T14:25:34.525 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.525 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.525 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:25:34.525 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.154502+0000", 2024-08-19T14:25:34.525 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0690000000000001e-06 2024-08-19T14:25:34.526 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.526 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.526 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:25:34.526 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.154514+0000", 2024-08-19T14:25:34.526 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1922e-05 2024-08-19T14:25:34.526 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.526 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.526 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:25:34.526 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.154522+0000", 2024-08-19T14:25:34.526 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.4730000000000003e-06 2024-08-19T14:25:34.526 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.526 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.527 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:25:34.527 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.275482+0000", 2024-08-19T14:25:34.527 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.12095984899999999 2024-08-19T14:25:34.527 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.527 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.527 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:25:34.527 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.275489+0000", 2024-08-19T14:25:34.527 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.8949999999999998e-06 2024-08-19T14:25:34.527 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.527 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.527 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:25:34.527 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:03.201475+0000", 2024-08-19T14:25:34.527 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.92598516200000003 2024-08-19T14:25:34.527 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.528 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.528 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:25:34.528 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:03.201559+0000", 2024-08-19T14:25:34.528 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.3968999999999995e-05 2024-08-19T14:25:34.528 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:25:34.528 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:25:34.528 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:25:34.528 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.528 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.528 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25057.0:188765 3.26 3:6524f2fe:::benchmark_data_smithi045_59973_object57586:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-08-19T14:25:34.528 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:19:02.192404+0000", 2024-08-19T14:25:34.528 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 392.26469981600002, 2024-08-19T14:25:34.528 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0202571499999999, 2024-08-19T14:25:34.529 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:25:34.529 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:25:34.529 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:25:34.529 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:25:34.529 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25057", 2024-08-19T14:25:34.529 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:25:34.529 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 188765 2024-08-19T14:25:34.529 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.529 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:25:34.529 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.529 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:25:34.529 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.192404+0000", 2024-08-19T14:25:34.529 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:25:34.529 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.530 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.530 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:25:34.530 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.192404+0000", 2024-08-19T14:25:34.530 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:25:34.530 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.530 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.530 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:25:34.530 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.192406+0000", 2024-08-19T14:25:34.530 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.3140000000000002e-06 2024-08-19T14:25:34.530 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.530 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.530 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:25:34.530 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.192417+0000", 2024-08-19T14:25:34.530 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0839000000000001e-05 2024-08-19T14:25:34.531 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.531 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.531 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:25:34.531 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.192440+0000", 2024-08-19T14:25:34.531 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.2911000000000001e-05 2024-08-19T14:25:34.531 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.531 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.531 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:25:34.531 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.192450+0000", 2024-08-19T14:25:34.531 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0292000000000001e-05 2024-08-19T14:25:34.531 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.531 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.531 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:25:34.532 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.275507+0000", 2024-08-19T14:25:34.532 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.083056873000000003 2024-08-19T14:25:34.532 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.532 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.532 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:25:34.532 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.275512+0000", 2024-08-19T14:25:34.532 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.8799999999999999e-06 2024-08-19T14:25:34.532 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.532 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.532 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:25:34.532 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:03.201920+0000", 2024-08-19T14:25:34.532 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.92640795799999998 2024-08-19T14:25:34.532 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.532 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.533 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:25:34.533 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:03.202036+0000", 2024-08-19T14:25:34.533 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000116364 2024-08-19T14:25:34.533 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.533 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.533 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 1,4", 2024-08-19T14:25:34.533 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:03.202142+0000", 2024-08-19T14:25:34.533 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000106115 2024-08-19T14:25:34.533 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.533 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.533 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:25:34.533 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:03.206756+0000", 2024-08-19T14:25:34.533 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0046137319999999997 2024-08-19T14:25:34.533 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.534 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.534 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:34.534 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:03.206778+0000", 2024-08-19T14:25:34.534 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.2056999999999999e-05 2024-08-19T14:25:34.534 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.534 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.534 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:34.534 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:03.212547+0000", 2024-08-19T14:25:34.534 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0057695740000000004 2024-08-19T14:25:34.534 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.534 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.534 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:25:34.534 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:03.212603+0000", 2024-08-19T14:25:34.535 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.5189999999999998e-05 2024-08-19T14:25:34.535 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.535 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.535 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:25:34.535 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:03.212661+0000", 2024-08-19T14:25:34.535 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.8050999999999997e-05 2024-08-19T14:25:34.535 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:25:34.535 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:25:34.535 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:25:34.535 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.535 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.535 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25333.0:22685 4.b 4:d7f4804b:::benchmark_data_smithi045_71562_object22684:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-08-19T14:25:34.536 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:20:41.347757+0000", 2024-08-19T14:25:34.536 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 293.10934606900003, 2024-08-19T14:25:34.536 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.33048444500000002, 2024-08-19T14:25:34.536 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:25:34.536 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:25:34.536 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:25:34.536 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:25:34.536 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25333", 2024-08-19T14:25:34.536 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:25:34.537 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 22685 2024-08-19T14:25:34.537 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.537 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:25:34.537 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.537 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:25:34.537 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.347757+0000", 2024-08-19T14:25:34.537 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:25:34.537 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.537 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.537 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:25:34.537 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.347757+0000", 2024-08-19T14:25:34.537 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:25:34.537 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.538 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.538 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:25:34.538 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.347759+0000", 2024-08-19T14:25:34.538 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4410000000000001e-06 2024-08-19T14:25:34.538 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.538 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.538 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:25:34.538 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.347940+0000", 2024-08-19T14:25:34.538 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00018171800000000001 2024-08-19T14:25:34.538 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.538 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.538 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:25:34.538 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.347953+0000", 2024-08-19T14:25:34.539 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2724e-05 2024-08-19T14:25:34.539 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.539 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.539 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:25:34.539 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.347960+0000", 2024-08-19T14:25:34.539 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.3320000000000003e-06 2024-08-19T14:25:34.539 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.539 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.539 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:25:34.539 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.348009+0000", 2024-08-19T14:25:34.539 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.8882999999999999e-05 2024-08-19T14:25:34.539 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.539 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.539 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:25:34.540 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.348129+0000", 2024-08-19T14:25:34.540 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012009 2024-08-19T14:25:34.540 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.540 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.540 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 1,4", 2024-08-19T14:25:34.540 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.348257+0000", 2024-08-19T14:25:34.540 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012718399999999999 2024-08-19T14:25:34.540 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.540 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.540 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:25:34.540 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.349748+0000", 2024-08-19T14:25:34.540 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.001491579 2024-08-19T14:25:34.541 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.541 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.541 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:34.541 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.350166+0000", 2024-08-19T14:25:34.541 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00041812000000000002 2024-08-19T14:25:34.541 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.541 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.541 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:34.541 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.351389+0000", 2024-08-19T14:25:34.541 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0012227119999999999 2024-08-19T14:25:34.541 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.541 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.541 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:25:34.541 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.678185+0000", 2024-08-19T14:25:34.542 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.32679576199999999 2024-08-19T14:25:34.542 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.542 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.542 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:25:34.542 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.678242+0000", 2024-08-19T14:25:34.542 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.6900000000000001e-05 2024-08-19T14:25:34.542 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:25:34.542 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:25:34.542 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:25:34.542 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.542 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.542 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop_reply(client.25333.0:22685 4.b e341/335 ondisk, result = 0)", 2024-08-19T14:25:34.542 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:20:41.350582+0000", 2024-08-19T14:25:34.543 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 293.10652165099998, 2024-08-19T14:25:34.543 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.32767402200000001, 2024-08-19T14:25:34.543 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:25:34.543 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:25:34.543 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-08-19T14:25:34.543 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:25:34.543 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.543 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:25:34.543 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.350582+0000", 2024-08-19T14:25:34.543 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:25:34.543 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.543 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.543 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:25:34.544 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.350582+0000", 2024-08-19T14:25:34.544 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:25:34.544 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.544 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.544 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:25:34.544 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.350582+0000", 2024-08-19T14:25:34.544 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.1999999999999999e-07 2024-08-19T14:25:34.544 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.544 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.544 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:25:34.544 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.350587+0000", 2024-08-19T14:25:34.544 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.6870000000000002e-06 2024-08-19T14:25:34.545 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.545 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.545 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:25:34.545 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.350598+0000", 2024-08-19T14:25:34.545 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0781999999999999e-05 2024-08-19T14:25:34.545 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.545 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.545 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:25:34.545 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.350605+0000", 2024-08-19T14:25:34.546 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.0419999999999997e-06 2024-08-19T14:25:34.546 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.546 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.546 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:25:34.546 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.351377+0000", 2024-08-19T14:25:34.546 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00077201899999999998 2024-08-19T14:25:34.546 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.546 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.546 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:25:34.546 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.351384+0000", 2024-08-19T14:25:34.546 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.917e-06 2024-08-19T14:25:34.546 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.546 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.546 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:25:34.547 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.678256+0000", 2024-08-19T14:25:34.547 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.32687185499999999 2024-08-19T14:25:34.547 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:25:34.547 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:25:34.547 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:25:34.547 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.547 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.547 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.25333.0:22687 4.7 e341/335 4:efd17a0f:::benchmark_data_smithi045_71562_object22686:head v 341'1401, mlcod=341'1397)", 2024-08-19T14:25:34.547 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:20:41.351262+0000", 2024-08-19T14:25:34.547 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 293.10584153999997, 2024-08-19T14:25:34.547 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.32996297600000002, 2024-08-19T14:25:34.547 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:25:34.547 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:25:34.547 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:25:34.548 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:25:34.548 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.548 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:25:34.548 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.351262+0000", 2024-08-19T14:25:34.548 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:25:34.548 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.548 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.548 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:25:34.548 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.351262+0000", 2024-08-19T14:25:34.548 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:25:34.548 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.548 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.548 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:25:34.548 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.351262+0000", 2024-08-19T14:25:34.549 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.0900000000000002e-07 2024-08-19T14:25:34.549 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.549 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.549 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:25:34.549 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.351289+0000", 2024-08-19T14:25:34.549 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.7036999999999999e-05 2024-08-19T14:25:34.549 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.549 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.549 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:25:34.549 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.351299+0000", 2024-08-19T14:25:34.549 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.9299999999999998e-06 2024-08-19T14:25:34.549 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.549 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.550 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:25:34.550 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.351305+0000", 2024-08-19T14:25:34.550 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.9229999999999999e-06 2024-08-19T14:25:34.550 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.550 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.550 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:25:34.550 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.351361+0000", 2024-08-19T14:25:34.550 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.6311999999999999e-05 2024-08-19T14:25:34.550 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.550 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.550 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:25:34.550 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.351380+0000", 2024-08-19T14:25:34.550 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.861e-05 2024-08-19T14:25:34.551 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.551 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.551 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:25:34.551 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.681181+0000", 2024-08-19T14:25:34.551 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.32980058099999998 2024-08-19T14:25:34.551 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.551 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.551 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:25:34.551 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.681225+0000", 2024-08-19T14:25:34.551 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4174000000000001e-05 2024-08-19T14:25:34.551 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:25:34.551 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:25:34.551 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:25:34.551 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.552 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.552 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.25333.0:22691 4.7 e341/335 4:e60184b4:::benchmark_data_smithi045_71562_object22690:head v 341'1402, mlcod=341'1397)", 2024-08-19T14:25:34.552 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:20:41.352142+0000", 2024-08-19T14:25:34.552 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 293.104961323, 2024-08-19T14:25:34.552 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.32972854099999999, 2024-08-19T14:25:34.552 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:25:34.552 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:25:34.552 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:25:34.552 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:25:34.552 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.552 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:25:34.552 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.352142+0000", 2024-08-19T14:25:34.552 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:25:34.553 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.553 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.553 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:25:34.553 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.352142+0000", 2024-08-19T14:25:34.553 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:25:34.553 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.553 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.553 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:25:34.553 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.352143+0000", 2024-08-19T14:25:34.553 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.5199999999999996e-07 2024-08-19T14:25:34.553 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.553 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.553 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:25:34.554 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.352196+0000", 2024-08-19T14:25:34.554 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.3468999999999999e-05 2024-08-19T14:25:34.554 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.554 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.554 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:25:34.554 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.678332+0000", 2024-08-19T14:25:34.554 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.32613542000000001 2024-08-19T14:25:34.554 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.554 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.554 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:25:34.554 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.678342+0000", 2024-08-19T14:25:34.554 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0696999999999999e-05 2024-08-19T14:25:34.554 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.554 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.555 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:25:34.555 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.679002+0000", 2024-08-19T14:25:34.555 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000659945 2024-08-19T14:25:34.555 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.555 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.555 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:25:34.555 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.679024+0000", 2024-08-19T14:25:34.555 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.1693999999999999e-05 2024-08-19T14:25:34.555 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.555 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.555 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:25:34.555 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.681816+0000", 2024-08-19T14:25:34.555 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0027922590000000001 2024-08-19T14:25:34.556 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.556 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.556 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:25:34.556 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.681871+0000", 2024-08-19T14:25:34.556 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.4305000000000003e-05 2024-08-19T14:25:34.556 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:25:34.556 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:25:34.556 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:25:34.556 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.556 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.556 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25333.0:22700 4.b 4:d65b8e70:::benchmark_data_smithi045_71562_object22699:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-08-19T14:25:34.556 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:20:41.353478+0000", 2024-08-19T14:25:34.556 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 293.10362510300001, 2024-08-19T14:25:34.556 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.32867489300000002, 2024-08-19T14:25:34.556 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:25:34.557 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:25:34.557 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:25:34.557 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:25:34.557 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25333", 2024-08-19T14:25:34.557 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:25:34.557 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 22700 2024-08-19T14:25:34.557 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.557 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:25:34.557 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.557 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:25:34.557 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.353478+0000", 2024-08-19T14:25:34.557 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:25:34.557 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.558 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.558 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:25:34.558 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.353478+0000", 2024-08-19T14:25:34.558 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:25:34.558 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.558 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.558 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:25:34.558 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.353480+0000", 2024-08-19T14:25:34.558 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6980000000000001e-06 2024-08-19T14:25:34.558 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.558 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.558 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:25:34.559 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.353537+0000", 2024-08-19T14:25:34.559 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.6724000000000001e-05 2024-08-19T14:25:34.559 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.559 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.559 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:25:34.559 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.677867+0000", 2024-08-19T14:25:34.559 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.32433008200000002 2024-08-19T14:25:34.559 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.559 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.559 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:25:34.559 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.677885+0000", 2024-08-19T14:25:34.559 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.8405000000000001e-05 2024-08-19T14:25:34.559 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.559 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.560 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:25:34.560 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.678314+0000", 2024-08-19T14:25:34.560 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00042872500000000003 2024-08-19T14:25:34.560 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.560 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.560 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:25:34.560 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.678522+0000", 2024-08-19T14:25:34.560 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00020833599999999999 2024-08-19T14:25:34.560 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.560 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.560 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 1,4", 2024-08-19T14:25:34.560 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.678691+0000", 2024-08-19T14:25:34.560 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00016871200000000001 2024-08-19T14:25:34.561 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.561 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.561 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:25:34.561 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.681852+0000", 2024-08-19T14:25:34.561 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0031611849999999999 2024-08-19T14:25:34.561 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.561 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.561 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:34.561 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.681921+0000", 2024-08-19T14:25:34.561 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.9191999999999998e-05 2024-08-19T14:25:34.561 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.561 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.561 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:34.561 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.682059+0000", 2024-08-19T14:25:34.562 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013730400000000001 2024-08-19T14:25:34.562 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.562 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.562 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:25:34.562 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.682113+0000", 2024-08-19T14:25:34.562 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.4840999999999999e-05 2024-08-19T14:25:34.562 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.562 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.562 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:25:34.562 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.682153+0000", 2024-08-19T14:25:34.562 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.9688999999999999e-05 2024-08-19T14:25:34.562 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:25:34.563 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:25:34.563 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:25:34.563 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.563 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.563 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25813.0:15525 5.3 5:c601536c:::benchmark_data_smithi045_85309_object15524:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e426)", 2024-08-19T14:25:34.563 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:24:52.832423+0000", 2024-08-19T14:25:34.563 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 41.624679833000002, 2024-08-19T14:25:34.563 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.26183261899999999, 2024-08-19T14:25:34.563 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:25:34.563 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:25:34.563 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:25:34.563 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:25:34.563 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25813", 2024-08-19T14:25:34.564 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:25:34.564 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 15525 2024-08-19T14:25:34.564 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.564 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:25:34.564 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.564 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:25:34.564 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:52.832423+0000", 2024-08-19T14:25:34.564 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:25:34.564 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.564 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.564 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:25:34.564 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:52.832423+0000", 2024-08-19T14:25:34.564 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:25:34.565 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.565 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.565 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:25:34.565 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:52.832424+0000", 2024-08-19T14:25:34.565 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.5899999999999995e-07 2024-08-19T14:25:34.565 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.565 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.565 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:25:34.565 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:25:34.567 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:52.832480+0000", 2024-08-19T14:25:34.567 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.5377999999999997e-05 2024-08-19T14:25:34.567 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.567 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.567 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:25:34.567 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:52.832489+0000", 2024-08-19T14:25:34.567 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.6169999999999996e-06 2024-08-19T14:25:34.567 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.567 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.567 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:25:34.567 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:52.832494+0000", 2024-08-19T14:25:34.568 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.6720000000000003e-06 2024-08-19T14:25:34.568 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.568 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.568 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:25:34.568 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:52.832531+0000", 2024-08-19T14:25:34.568 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7120999999999999e-05 2024-08-19T14:25:34.568 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.568 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.568 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:25:34.568 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:52.832617+0000", 2024-08-19T14:25:34.568 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.5587000000000006e-05 2024-08-19T14:25:34.568 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.569 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.569 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,5", 2024-08-19T14:25:34.569 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:52.832694+0000", 2024-08-19T14:25:34.569 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.7669000000000005e-05 2024-08-19T14:25:34.569 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.569 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.569 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:25:34.569 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:52.834430+0000", 2024-08-19T14:25:34.569 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.001735225 2024-08-19T14:25:34.569 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.569 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.569 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:34.569 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:52.836161+0000", 2024-08-19T14:25:34.569 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0017313960000000001 2024-08-19T14:25:34.570 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.570 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.570 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:34.570 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:53.094158+0000", 2024-08-19T14:25:34.570 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.257997003 2024-08-19T14:25:34.570 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.570 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.571 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:25:34.571 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:53.094209+0000", 2024-08-19T14:25:34.571 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0763000000000002e-05 2024-08-19T14:25:34.571 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.571 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.571 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:25:34.571 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:53.094256+0000", 2024-08-19T14:25:34.571 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7429000000000003e-05 2024-08-19T14:25:34.571 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:25:34.571 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:25:34.571 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:25:34.571 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.571 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.571 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25813.0:15529 5.3 5:c6799b39:::benchmark_data_smithi045_85309_object15528:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e426)", 2024-08-19T14:25:34.572 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:24:52.833224+0000", 2024-08-19T14:25:34.572 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 41.623878924000003, 2024-08-19T14:25:34.572 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.26288265900000002, 2024-08-19T14:25:34.572 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:25:34.572 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:25:34.572 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:25:34.572 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:25:34.572 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25813", 2024-08-19T14:25:34.572 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:25:34.572 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 15529 2024-08-19T14:25:34.572 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.572 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:25:34.572 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.573 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:25:34.573 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:52.833224+0000", 2024-08-19T14:25:34.573 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:25:34.573 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.573 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.573 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:25:34.573 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:52.833224+0000", 2024-08-19T14:25:34.573 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:25:34.573 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.573 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.573 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:25:34.573 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:52.833225+0000", 2024-08-19T14:25:34.574 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.3000000000000001e-07 2024-08-19T14:25:34.574 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.574 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.574 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:25:34.574 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:52.833253+0000", 2024-08-19T14:25:34.574 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.7852999999999999e-05 2024-08-19T14:25:34.574 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.574 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.574 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:25:34.574 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:52.833260+0000", 2024-08-19T14:25:34.574 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.61e-06 2024-08-19T14:25:34.574 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.574 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.575 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:25:34.575 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:52.833266+0000", 2024-08-19T14:25:34.575 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.614e-06 2024-08-19T14:25:34.575 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.575 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.575 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:25:34.575 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:52.833376+0000", 2024-08-19T14:25:34.575 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000110365 2024-08-19T14:25:34.575 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.575 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.575 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:25:34.575 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:52.833512+0000", 2024-08-19T14:25:34.576 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013533800000000001 2024-08-19T14:25:34.576 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.576 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.576 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,5", 2024-08-19T14:25:34.576 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:52.833748+0000", 2024-08-19T14:25:34.576 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00023644700000000001 2024-08-19T14:25:34.576 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.576 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.576 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:25:34.576 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:52.834588+0000", 2024-08-19T14:25:34.576 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00083990900000000001 2024-08-19T14:25:34.576 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.576 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.576 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:34.576 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:52.836202+0000", 2024-08-19T14:25:34.576 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0016136270000000001 2024-08-19T14:25:34.577 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.577 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.577 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:34.577 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:53.096008+0000", 2024-08-19T14:25:34.577 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.25980599500000001 2024-08-19T14:25:34.577 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.577 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.577 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:25:34.577 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:53.096076+0000", 2024-08-19T14:25:34.577 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.8532e-05 2024-08-19T14:25:34.577 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.577 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.577 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:25:34.578 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:53.096107+0000", 2024-08-19T14:25:34.578 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.0839000000000001e-05 2024-08-19T14:25:34.578 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:25:34.578 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:25:34.578 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:25:34.578 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.578 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.578 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25813.0:15536 5.3 5:cba1709d:::benchmark_data_smithi045_85309_object15535:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e426)", 2024-08-19T14:25:34.578 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:24:52.835227+0000", 2024-08-19T14:25:34.578 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 41.621876716999999, 2024-08-19T14:25:34.578 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.26103143200000001, 2024-08-19T14:25:34.578 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:25:34.578 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:25:34.579 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:25:34.579 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:25:34.579 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25813", 2024-08-19T14:25:34.579 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:25:34.579 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 15536 2024-08-19T14:25:34.579 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.579 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:25:34.579 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.579 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:25:34.579 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:52.835227+0000", 2024-08-19T14:25:34.579 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:25:34.579 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.579 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.579 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:25:34.580 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:52.835227+0000", 2024-08-19T14:25:34.580 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:25:34.580 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.580 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.580 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:25:34.580 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:52.835227+0000", 2024-08-19T14:25:34.580 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.54e-07 2024-08-19T14:25:34.580 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.580 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.580 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:25:34.580 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:52.835266+0000", 2024-08-19T14:25:34.580 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.8816000000000002e-05 2024-08-19T14:25:34.580 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.580 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.581 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:25:34.581 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:52.835279+0000", 2024-08-19T14:25:34.581 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.242e-05 2024-08-19T14:25:34.581 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.581 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.581 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:25:34.581 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:52.835285+0000", 2024-08-19T14:25:34.581 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.7039999999999997e-06 2024-08-19T14:25:34.581 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.581 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.581 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:25:34.581 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:52.835329+0000", 2024-08-19T14:25:34.581 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.3108000000000001e-05 2024-08-19T14:25:34.581 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.582 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.582 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:25:34.582 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:52.835399+0000", 2024-08-19T14:25:34.582 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.0006999999999996e-05 2024-08-19T14:25:34.582 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.582 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.582 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,5", 2024-08-19T14:25:34.582 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:52.835576+0000", 2024-08-19T14:25:34.582 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000176995 2024-08-19T14:25:34.582 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.582 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.582 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:25:34.582 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:52.836602+0000", 2024-08-19T14:25:34.582 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0010267399999999999 2024-08-19T14:25:34.583 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.583 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.583 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:34.583 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:52.837148+0000", 2024-08-19T14:25:34.583 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00054603900000000001 2024-08-19T14:25:34.583 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.583 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.583 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:25:34.583 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:53.096162+0000", 2024-08-19T14:25:34.583 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.25901379099999999 2024-08-19T14:25:34.583 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.583 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.583 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:25:34.584 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:53.096217+0000", 2024-08-19T14:25:34.584 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.4599999999999999e-05 2024-08-19T14:25:34.584 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:25:34.584 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:25:34.584 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:25:34.584 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:53.096258+0000", 2024-08-19T14:25:34.584 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.1357999999999999e-05 2024-08-19T14:25:34.584 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:25:34.584 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:25:34.584 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:25:34.584 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:25:34.584 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:25:34.584 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:25:34.595 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:34.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:34.596 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:25:34.696 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:25:34.731 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:34.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:34.732 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:25:34.833 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:25:34.888 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:34.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:34.889 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:25:34.990 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:25:35.019 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:35.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:35.020 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:25:35.121 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:25:35.156 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:35.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:35.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:25:35.258 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:25:35.286 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:35.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:35.287 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:25:35.323 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 47 16 91355 91339 121.443 52.5 0.00489235 0.00821301 2024-08-19T14:25:35.388 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:25:35.414 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:35.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:35.415 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:25:35.516 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:25:35.548 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:35.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:35.549 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:25:35.650 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:25:35.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:35.689 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:35.690 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:25:35.701 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:25:35.791 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:25:35.827 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:35.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:35.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:25:35.930 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:25:35.973 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:35.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:35.974 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:25:36.075 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:25:36.101 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:36.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:36.102 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:25:36.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:35 smithi031 ceph-mon[16336]: pgmap v1013: 57 pgs: 2 active+recovering+undersized+remapped, 1 remapped+peering, 7 unknown, 47 active+clean; 4.5 GiB data, 21 GiB used, 783 GiB / 805 GiB avail; 13997/218793 objects misplaced (6.397%); 555 KiB/s, 8 objects/s recovering 2024-08-19T14:25:36.127 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:35 smithi032 ceph-mon[20004]: pgmap v1013: 57 pgs: 2 active+recovering+undersized+remapped, 1 remapped+peering, 7 unknown, 47 active+clean; 4.5 GiB data, 21 GiB used, 783 GiB / 805 GiB avail; 13997/218793 objects misplaced (6.397%); 555 KiB/s, 8 objects/s recovering 2024-08-19T14:25:36.203 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:25:36.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:36.237 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:36.237 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:25:36.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:35 smithi045 ceph-mon[20143]: pgmap v1013: 57 pgs: 2 active+recovering+undersized+remapped, 1 remapped+peering, 7 unknown, 47 active+clean; 4.5 GiB data, 21 GiB used, 783 GiB / 805 GiB avail; 13997/218793 objects misplaced (6.397%); 555 KiB/s, 8 objects/s recovering 2024-08-19T14:25:36.323 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 48 16 92086 92070 119.864 45.6875 0.0231862 0.00833704 2024-08-19T14:25:36.338 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:25:36.364 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:36.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:36.365 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:25:36.466 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:25:36.492 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:36.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:36.493 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:25:36.595 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:25:36.631 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:36.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:36.632 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:25:36.733 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:25:36.768 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:36.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:36.769 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:25:36.859 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is unset 2024-08-19T14:25:36.870 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:25:36.892 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:36.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:36.893 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:25:36.906 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-08-19T14:25:36.994 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:25:37.024 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:37.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:37.025 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:25:37.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:36 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:25:37.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:36 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:25:37.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:36 smithi031 ceph-mon[16336]: osdmap e444: 12 total, 12 up, 9 in 2024-08-19T14:25:37.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:36 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3580036284' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:25:37.126 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:25:37.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:37.158 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:37.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:25:37.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:36 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:25:37.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:36 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:25:37.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:36 smithi032 ceph-mon[20004]: osdmap e444: 12 total, 12 up, 9 in 2024-08-19T14:25:37.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:36 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3580036284' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:25:37.259 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:25:37.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:37.287 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:37.288 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:25:37.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:36 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:25:37.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:36 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:25:37.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:36 smithi045 ceph-mon[20143]: osdmap e444: 12 total, 12 up, 9 in 2024-08-19T14:25:37.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:36 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3580036284' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:25:37.323 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 49 16 92854 92838 118.398 48 0.0224844 0.00842987 2024-08-19T14:25:37.389 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:25:37.410 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:37.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:37.411 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:25:37.512 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:25:37.540 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:37.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:37.541 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:25:37.642 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:25:37.674 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:37.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:37.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:25:37.708 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:25:37.708 INFO:teuthology.orchestra.run.smithi032.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-08-19T14:25:37.709 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:25:37.742 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:25:37.776 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:25:37.799 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:37.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:37.800 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:25:37.826 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:25:37.900 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:25:37.901 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:37 smithi031 ceph-mon[16336]: pgmap v1015: 57 pgs: 3 active+recovering+undersized+remapped, 1 remapped+peering, 6 unknown, 47 active+clean; 4.6 GiB data, 21 GiB used, 783 GiB / 805 GiB avail; 19307/227916 objects misplaced (8.471%); 1.8 MiB/s, 29 objects/s recovering 2024-08-19T14:25:37.901 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:37 smithi031 ceph-mon[16336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:25:37.902 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:37 smithi031 ceph-mon[16336]: Cluster is now healthy 2024-08-19T14:25:37.902 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:37 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3580036284' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:25:37.902 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:37 smithi031 ceph-mon[16336]: osdmap e445: 12 total, 12 up, 9 in 2024-08-19T14:25:37.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:37.957 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:37.957 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:25:38.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:37 smithi032 ceph-mon[20004]: pgmap v1015: 57 pgs: 3 active+recovering+undersized+remapped, 1 remapped+peering, 6 unknown, 47 active+clean; 4.6 GiB data, 21 GiB used, 783 GiB / 805 GiB avail; 19307/227916 objects misplaced (8.471%); 1.8 MiB/s, 29 objects/s recovering 2024-08-19T14:25:38.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:37 smithi032 ceph-mon[20004]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:25:38.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:37 smithi032 ceph-mon[20004]: Cluster is now healthy 2024-08-19T14:25:38.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:37 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3580036284' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:25:38.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:37 smithi032 ceph-mon[20004]: osdmap e445: 12 total, 12 up, 9 in 2024-08-19T14:25:38.058 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:25:38.086 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:38.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:38.087 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:25:38.188 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:25:38.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:38.225 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:38.226 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:25:38.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:37 smithi045 ceph-mon[20143]: pgmap v1015: 57 pgs: 3 active+recovering+undersized+remapped, 1 remapped+peering, 6 unknown, 47 active+clean; 4.6 GiB data, 21 GiB used, 783 GiB / 805 GiB avail; 19307/227916 objects misplaced (8.471%); 1.8 MiB/s, 29 objects/s recovering 2024-08-19T14:25:38.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:37 smithi045 ceph-mon[20143]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:25:38.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:37 smithi045 ceph-mon[20143]: Cluster is now healthy 2024-08-19T14:25:38.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:37 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3580036284' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:25:38.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:37 smithi045 ceph-mon[20143]: osdmap e445: 12 total, 12 up, 9 in 2024-08-19T14:25:38.323 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 50 16 93626 93610 116.995 48.25 0.0110307 0.00854153 2024-08-19T14:25:38.326 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:25:38.356 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:38.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:38.358 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:25:38.425 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:25:38.426 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:25:38.459 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:25:38.491 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:38.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:38.492 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:25:38.519 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:25:38.519 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:25:38.555 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:25:38.555 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:25:38.592 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:25:38.592 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:25:38.593 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:25:38.622 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:38.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:38.623 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:25:38.630 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:25:38.630 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:25:38.674 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:25:38.674 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:25:38.717 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:25:38.717 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:25:38.724 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:25:38.757 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:25:38.757 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:25:38.790 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:38.791 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:25:38.792 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:25:38.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:38.793 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:25:38.835 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:25:38.835 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:25:38.860 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:25:38.860 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:25:38.878 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:25:38.879 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:25:38.894 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:25:38.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:38.949 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:38.949 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:25:39.050 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:25:39.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:39.083 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:39.084 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:25:39.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:38 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2499712435' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:25:39.185 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:25:39.210 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:39.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:39.211 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:25:39.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:38 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2499712435' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:25:39.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:38 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2499712435' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:25:39.312 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:25:39.323 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 51 16 94500 94484 115.772 54.625 0.00834357 0.00863129 2024-08-19T14:25:39.345 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:39.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:39.346 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:25:39.447 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:25:39.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:39.477 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:39.477 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:25:39.579 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:25:39.607 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:39.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:39.608 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:25:39.709 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:25:39.711 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-19T14:25:39.758 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:39.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:39.759 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:25:39.846 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:25:39.846 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:25:39.860 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:25:39.881 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:39.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:39.882 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:25:39.983 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:25:40.012 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:40.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:40.013 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:25:40.114 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:25:40.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:39 smithi031 ceph-mon[16336]: pgmap v1017: 57 pgs: 10 active+recovering+undersized+remapped, 47 active+clean; 5.6 GiB data, 22 GiB used, 783 GiB / 805 GiB avail; 30 MiB/s wr, 473 op/s; 51302/273996 objects misplaced (18.724%); 12 MiB/s, 191 objects/s recovering 2024-08-19T14:25:40.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:39 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:25:40.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:39 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:25:40.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:40.172 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:40.172 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:25:40.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:39 smithi032 ceph-mon[20004]: pgmap v1017: 57 pgs: 10 active+recovering+undersized+remapped, 47 active+clean; 5.6 GiB data, 22 GiB used, 783 GiB / 805 GiB avail; 30 MiB/s wr, 473 op/s; 51302/273996 objects misplaced (18.724%); 12 MiB/s, 191 objects/s recovering 2024-08-19T14:25:40.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:39 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:25:40.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:39 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:25:40.273 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:25:40.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:39 smithi045 ceph-mon[20143]: pgmap v1017: 57 pgs: 10 active+recovering+undersized+remapped, 47 active+clean; 5.6 GiB data, 22 GiB used, 783 GiB / 805 GiB avail; 30 MiB/s wr, 473 op/s; 51302/273996 objects misplaced (18.724%); 12 MiB/s, 191 objects/s recovering 2024-08-19T14:25:40.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:39 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:25:40.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:39 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:25:40.305 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:40.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:40.307 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:25:40.323 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 52 16 95225 95209 114.417 45.3125 0.0373983 0.00873408 2024-08-19T14:25:40.407 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:25:40.444 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:40.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:40.445 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:25:40.546 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:25:40.576 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:40.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:40.577 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:25:40.678 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:25:40.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:40.720 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:40.720 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:25:40.821 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:25:40.852 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:40.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:40.853 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:25:40.954 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:25:40.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:40.976 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:40.976 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:25:41.077 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:25:41.104 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:41.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:41.106 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:25:41.206 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:25:41.235 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:41.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:41.236 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:25:41.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:40 smithi032 ceph-mon[20004]: osdmap e446: 12 total, 12 up, 9 in 2024-08-19T14:25:41.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:40 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:25:41.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:40 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:25:41.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:40 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:25:41.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:40 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:25:41.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:40 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:25:41.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:40 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:25:41.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:40 smithi045 ceph-mon[20143]: osdmap e446: 12 total, 12 up, 9 in 2024-08-19T14:25:41.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:40 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:25:41.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:40 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:25:41.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:40 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:25:41.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:40 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:25:41.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:40 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:25:41.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:40 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:25:41.324 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 53 16 96030 96014 113.207 50.3125 0.00757315 0.00882769 2024-08-19T14:25:41.337 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:25:41.362 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:41.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:41.363 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:25:41.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:40 smithi031 ceph-mon[16336]: osdmap e446: 12 total, 12 up, 9 in 2024-08-19T14:25:41.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:40 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:25:41.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:40 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:25:41.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:40 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:25:41.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:40 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:25:41.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:40 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:25:41.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:40 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:25:41.464 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:25:41.497 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:41.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:41.498 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:25:41.598 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:25:41.622 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:41.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:41.623 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:25:41.724 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:25:41.767 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:41.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:41.768 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:25:41.869 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:25:41.908 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:41.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:41.909 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:25:42.010 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:25:42.060 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:42.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:42.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:25:42.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:42 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:25:42.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:42 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:25:42.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:42 smithi031 ceph-mon[16336]: osdmap e447: 12 total, 12 up, 9 in 2024-08-19T14:25:42.162 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:25:42.204 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:42.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:42.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:25:42.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:42 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:25:42.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:42 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:25:42.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:42 smithi032 ceph-mon[20004]: osdmap e447: 12 total, 12 up, 9 in 2024-08-19T14:25:42.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:42 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:25:42.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:42 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:25:42.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:42 smithi045 ceph-mon[20143]: osdmap e447: 12 total, 12 up, 9 in 2024-08-19T14:25:42.306 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:25:42.324 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 54 16 96786 96770 111.986 47.25 0.0145077 0.00892366 2024-08-19T14:25:42.332 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:42.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:42.333 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:25:42.434 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:25:42.457 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:42.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:42.458 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:25:42.559 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:25:42.595 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:42.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:42.596 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:25:42.697 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:25:42.728 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:42.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:42.729 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:25:42.830 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:25:42.866 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:42.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:42.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:25:42.968 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:25:42.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:42.998 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:42.998 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:25:43.100 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:25:43.132 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:43.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:43.133 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:25:43.234 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:25:43.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:43 smithi032 ceph-mon[20004]: pgmap v1021: 57 pgs: 10 active+recovering+undersized+remapped, 47 active+clean; 5.8 GiB data, 22 GiB used, 782 GiB / 805 GiB avail; 114 MiB/s wr, 1.82k op/s; 48413/285348 objects misplaced (16.966%); 26 MiB/s, 420 objects/s recovering 2024-08-19T14:25:43.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:43 smithi032 ceph-mon[20004]: osdmap e448: 12 total, 12 up, 9 in 2024-08-19T14:25:43.259 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:43.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:43.260 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:25:43.278 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:43 smithi031 ceph-mon[16336]: pgmap v1021: 57 pgs: 10 active+recovering+undersized+remapped, 47 active+clean; 5.8 GiB data, 22 GiB used, 782 GiB / 805 GiB avail; 114 MiB/s wr, 1.82k op/s; 48413/285348 objects misplaced (16.966%); 26 MiB/s, 420 objects/s recovering 2024-08-19T14:25:43.278 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:43 smithi031 ceph-mon[16336]: osdmap e448: 12 total, 12 up, 9 in 2024-08-19T14:25:43.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:43 smithi045 ceph-mon[20143]: pgmap v1021: 57 pgs: 10 active+recovering+undersized+remapped, 47 active+clean; 5.8 GiB data, 22 GiB used, 782 GiB / 805 GiB avail; 114 MiB/s wr, 1.82k op/s; 48413/285348 objects misplaced (16.966%); 26 MiB/s, 420 objects/s recovering 2024-08-19T14:25:43.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:43 smithi045 ceph-mon[20143]: osdmap e448: 12 total, 12 up, 9 in 2024-08-19T14:25:43.324 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 55 16 97658 97642 110.941 54.5 0.00402177 0.00900847 2024-08-19T14:25:43.360 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:25:43.383 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:43.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:43.385 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:25:43.486 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:25:43.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:43.527 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:43.527 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:25:43.628 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:25:43.649 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:43.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:43.651 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:25:43.752 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:25:43.785 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:43.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:43.786 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:25:43.815 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-08-19T14:25:43.887 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:25:43.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:43.918 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:43.918 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:25:43.919 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:25:44.019 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:25:44.041 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:44.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:44.042 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:25:44.143 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:25:44.178 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:44.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:44.180 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:25:44.281 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:25:44.306 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:44.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:44.308 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:25:44.323 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 56 16 98467 98451 109.863 50.5625 0.0102259 0.00909707 2024-08-19T14:25:44.408 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:25:44.458 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:44.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:44.459 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:25:44.560 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:25:44.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:44.588 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:44.588 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:25:44.650 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:25:44.689 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:25:44.722 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:44.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:44.723 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:25:44.824 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:25:44.853 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:44.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:44.854 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:25:44.956 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:25:44.981 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:44.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:44.983 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:25:45.063 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is set 2024-08-19T14:25:45.063 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:45 smithi031 ceph-mon[16336]: pgmap v1023: 57 pgs: 10 active+recovering+undersized+remapped, 47 active+clean; 5.9 GiB data, 23 GiB used, 782 GiB / 805 GiB avail; 55 MiB/s wr, 872 op/s; 47205/289710 objects misplaced (16.294%); 24 MiB/s, 376 objects/s recovering 2024-08-19T14:25:45.063 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:45 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2831248471' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:25:45.084 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:25:45.119 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:45.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:45.120 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:25:45.221 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:25:45.251 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:45.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:45.252 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:25:45.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:45 smithi045 ceph-mon[20143]: pgmap v1023: 57 pgs: 10 active+recovering+undersized+remapped, 47 active+clean; 5.9 GiB data, 23 GiB used, 782 GiB / 805 GiB avail; 55 MiB/s wr, 872 op/s; 47205/289710 objects misplaced (16.294%); 24 MiB/s, 376 objects/s recovering 2024-08-19T14:25:45.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:45 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2831248471' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:25:45.324 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 57 15 99210 99195 108.751 46.5 0.00547935 0.00918993 2024-08-19T14:25:45.353 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:25:45.376 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:25:45.398 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:45.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:45.400 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:25:45.442 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:25:45.442 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:25:45.501 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:25:45.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:45 smithi032 ceph-mon[20004]: pgmap v1023: 57 pgs: 10 active+recovering+undersized+remapped, 47 active+clean; 5.9 GiB data, 23 GiB used, 782 GiB / 805 GiB avail; 55 MiB/s wr, 872 op/s; 47205/289710 objects misplaced (16.294%); 24 MiB/s, 376 objects/s recovering 2024-08-19T14:25:45.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:45 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2831248471' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:25:45.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:45.562 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:45.562 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:25:45.663 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:25:45.694 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:45.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:45.695 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:25:45.796 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:25:45.824 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:45.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:45.825 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:25:45.927 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:25:45.968 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:45.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:45.969 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:25:46.069 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:25:46.098 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:46.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:46.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:25:46.200 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:25:46.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:46.226 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:46.226 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:25:46.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:46 smithi032 ceph-mon[20004]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:25:46.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:46 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2831248471' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:25:46.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:46 smithi032 ceph-mon[20004]: osdmap e449: 12 total, 12 up, 9 in 2024-08-19T14:25:46.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:46 smithi045 ceph-mon[20143]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:25:46.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:46 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2831248471' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:25:46.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:46 smithi045 ceph-mon[20143]: osdmap e449: 12 total, 12 up, 9 in 2024-08-19T14:25:46.324 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 58 16 99955 99939 107.678 46.5 0.0171036 0.00928162 2024-08-19T14:25:46.327 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:25:46.356 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:46.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:46.357 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:25:46.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:46 smithi031 ceph-mon[16336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:25:46.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:46 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2831248471' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:25:46.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:46 smithi031 ceph-mon[16336]: osdmap e449: 12 total, 12 up, 9 in 2024-08-19T14:25:46.458 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:25:46.487 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:46.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:46.488 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:25:46.589 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:25:46.617 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:46.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:46.619 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:25:46.719 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:25:46.749 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:46.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:46.750 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:25:46.851 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:25:46.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:46.894 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:46.894 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:25:46.995 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:25:47.024 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:47.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:47.025 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:25:47.127 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:25:47.163 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:47.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:47.164 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:25:47.266 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:25:47.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:47.300 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:47.300 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:25:47.324 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 59 16 100815 100799 106.764 53.75 0.0100418 0.00936164 2024-08-19T14:25:47.401 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:25:47.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:47.435 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:47.435 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:25:47.436 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:47 smithi031 ceph-mon[16336]: pgmap v1025: 57 pgs: 10 active+recovering+undersized+remapped, 47 active+clean; 5.9 GiB data, 23 GiB used, 782 GiB / 805 GiB avail; 45 MiB/s wr, 721 op/s; 46699/290835 objects misplaced (16.057%); 23 MiB/s, 365 objects/s recovering 2024-08-19T14:25:47.436 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:47 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:25:47.436 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:47 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:25:47.436 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:47 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:25:47.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:47 smithi032 ceph-mon[20004]: pgmap v1025: 57 pgs: 10 active+recovering+undersized+remapped, 47 active+clean; 5.9 GiB data, 23 GiB used, 782 GiB / 805 GiB avail; 45 MiB/s wr, 721 op/s; 46699/290835 objects misplaced (16.057%); 23 MiB/s, 365 objects/s recovering 2024-08-19T14:25:47.509 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:47 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:25:47.509 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:47 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:25:47.509 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:47 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:25:47.536 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:25:47.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:47 smithi045 ceph-mon[20143]: pgmap v1025: 57 pgs: 10 active+recovering+undersized+remapped, 47 active+clean; 5.9 GiB data, 23 GiB used, 782 GiB / 805 GiB avail; 45 MiB/s wr, 721 op/s; 46699/290835 objects misplaced (16.057%); 23 MiB/s, 365 objects/s recovering 2024-08-19T14:25:47.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:47 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:25:47.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:47 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:25:47.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:47 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:25:47.559 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:47.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:47.560 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:25:47.661 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:25:47.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:47.695 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:47.696 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:25:47.797 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:25:47.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:47.830 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:47.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:25:47.932 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:25:47.970 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:47.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:47.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:25:48.073 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:25:48.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:48.103 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:48.103 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:25:48.204 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:25:48.232 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:48.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:48.233 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:25:48.324 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:2024-08-19T14:25:48.324703+0000 min lat: 0.00112215 max lat: 2.38842 avg lat: 0.00943433 2024-08-19T14:25:48.324 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-19T14:25:48.324 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 60 16 101720 101704 105.927 56.5625 0.00463435 0.00943433 2024-08-19T14:25:48.334 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:25:48.355 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:48.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:48.356 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:25:48.457 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:25:48.494 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:48.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:48.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:25:48.596 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:25:48.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:48.635 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:48.635 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:25:48.736 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:25:48.776 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:48.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:48.778 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:25:48.879 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:25:48.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:48.918 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:48.918 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:25:49.020 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:25:49.053 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:49.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:49.054 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:25:49.155 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:25:49.183 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:49.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:49.184 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:25:49.285 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:25:49.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:49.308 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:49.308 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:25:49.324 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 61 16 102471 102455 104.96 46.9375 0.00550263 0.00950301 2024-08-19T14:25:49.409 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:25:49.444 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:49.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:49.445 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:25:49.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:49 smithi032 ceph-mon[20004]: pgmap v1026: 57 pgs: 10 active+recovering+undersized+remapped, 47 active+clean; 6.1 GiB data, 24 GiB used, 781 GiB / 805 GiB avail; 57 MiB/s wr, 908 op/s; 44433/298011 objects misplaced (14.910%); 31 MiB/s, 498 objects/s recovering 2024-08-19T14:25:49.546 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:25:49.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:49 smithi045 ceph-mon[20143]: pgmap v1026: 57 pgs: 10 active+recovering+undersized+remapped, 47 active+clean; 6.1 GiB data, 24 GiB used, 781 GiB / 805 GiB avail; 57 MiB/s wr, 908 op/s; 44433/298011 objects misplaced (14.910%); 31 MiB/s, 498 objects/s recovering 2024-08-19T14:25:49.585 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:49.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:49.586 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:25:49.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:49 smithi031 ceph-mon[16336]: pgmap v1026: 57 pgs: 10 active+recovering+undersized+remapped, 47 active+clean; 6.1 GiB data, 24 GiB used, 781 GiB / 805 GiB avail; 57 MiB/s wr, 908 op/s; 44433/298011 objects misplaced (14.910%); 31 MiB/s, 498 objects/s recovering 2024-08-19T14:25:49.687 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:25:49.711 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:49.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:49.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:25:49.813 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:25:49.846 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:49.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:49.847 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:25:49.948 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:25:49.994 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:49.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:49.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:25:50.096 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:25:50.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:50.135 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:50.135 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:25:50.236 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:25:50.258 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:50.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:50.259 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:25:50.324 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 62 16 103282 103266 104.084 50.6875 0.0142542 0.00960207 2024-08-19T14:25:50.360 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:25:50.396 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:50.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:50.397 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:25:50.498 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:25:50.529 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:50.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:50.530 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:25:50.630 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:25:50.658 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:50.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:50.659 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:25:50.760 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:25:50.788 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:50.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:50.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:25:50.890 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:25:50.913 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:50.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:50.915 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:25:51.016 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:25:51.038 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 7, 8, 9, 11] out_osds: [10, 6, 2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-08-19T14:25:51.038 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-19T14:25:51.039 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2024-08-19T14:25:51.039 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-08-19T14:25:51.039 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-08-19T14:25:51.039 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.4 config set filestore_inject_stall 3 2024-08-19T14:25:51.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:51.052 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:51.052 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:25:51.152 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:25:51.185 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:51.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:51.186 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:25:51.287 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:25:51.315 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:51.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:51.317 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:25:51.324 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 63 15 104074 104059 103.219 49.5625 0.0145758 0.00968295 2024-08-19T14:25:51.417 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:25:51.418 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:51 smithi031 ceph-mon[16336]: pgmap v1027: 57 pgs: 10 active+recovering+undersized+remapped, 47 active+clean; 6.1 GiB data, 24 GiB used, 781 GiB / 805 GiB avail; 43 MiB/s wr, 690 op/s; 43538/301914 objects misplaced (14.421%); 23 MiB/s, 360 objects/s recovering 2024-08-19T14:25:51.432 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:51 smithi032 ceph-mon[20004]: pgmap v1027: 57 pgs: 10 active+recovering+undersized+remapped, 47 active+clean; 6.1 GiB data, 24 GiB used, 781 GiB / 805 GiB avail; 43 MiB/s wr, 690 op/s; 43538/301914 objects misplaced (14.421%); 23 MiB/s, 360 objects/s recovering 2024-08-19T14:25:51.455 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:51.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:51.456 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:25:51.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:51 smithi045 ceph-mon[20143]: pgmap v1027: 57 pgs: 10 active+recovering+undersized+remapped, 47 active+clean; 6.1 GiB data, 24 GiB used, 781 GiB / 805 GiB avail; 43 MiB/s wr, 690 op/s; 43538/301914 objects misplaced (14.421%); 23 MiB/s, 360 objects/s recovering 2024-08-19T14:25:51.557 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:25:51.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:51.595 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:51.595 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:25:51.697 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:25:51.729 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:51.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:51.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:25:51.832 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:25:51.865 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:51.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:51.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:25:51.968 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:25:52.001 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:25:52.004 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:52.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:52.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:25:52.106 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:25:52.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:52.134 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:52.135 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:25:52.236 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:25:52.259 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:52.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:52.260 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:25:52.324 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 64 16 104839 104823 102.352 47.75 0.211931 0.00975516 2024-08-19T14:25:52.361 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:25:52.386 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:52.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:52.387 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:25:52.488 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:25:52.522 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:52.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:52.524 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:25:52.625 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:25:52.649 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:52.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:52.650 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:25:52.651 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-08-19T14:25:52.751 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:25:52.778 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:52.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:52.780 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:25:52.881 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:25:52.912 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:52.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:52.913 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:25:53.014 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:25:53.049 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:53.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:53.050 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:25:53.151 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:25:53.178 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:53.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:53.179 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:25:53.280 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:25:53.308 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:53.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:53.309 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:25:53.324 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 65 16 105537 105521 101.449 43.625 0.0694944 0.00985193 2024-08-19T14:25:53.410 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:25:53.445 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:53.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:53.446 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:25:53.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:53 smithi032 ceph-mon[20004]: pgmap v1028: 57 pgs: 10 active+recovering+undersized+remapped, 47 active+clean; 6.3 GiB data, 24 GiB used, 780 GiB / 805 GiB avail; 51 MiB/s wr, 810 op/s; 41069/309549 objects misplaced (13.267%); 27 MiB/s, 427 objects/s recovering 2024-08-19T14:25:53.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:53 smithi045 ceph-mon[20143]: pgmap v1028: 57 pgs: 10 active+recovering+undersized+remapped, 47 active+clean; 6.3 GiB data, 24 GiB used, 780 GiB / 805 GiB avail; 51 MiB/s wr, 810 op/s; 41069/309549 objects misplaced (13.267%); 27 MiB/s, 427 objects/s recovering 2024-08-19T14:25:53.546 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:25:53.575 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:53.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:53.576 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:25:53.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:53 smithi031 ceph-mon[16336]: pgmap v1028: 57 pgs: 10 active+recovering+undersized+remapped, 47 active+clean; 6.3 GiB data, 24 GiB used, 780 GiB / 805 GiB avail; 51 MiB/s wr, 810 op/s; 41069/309549 objects misplaced (13.267%); 27 MiB/s, 427 objects/s recovering 2024-08-19T14:25:53.676 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:25:53.697 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:53.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:53.698 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:25:53.799 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:25:53.828 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:53.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:53.829 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:25:53.930 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:25:53.966 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:53.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:53.968 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:25:54.068 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:25:54.091 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:54.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:54.092 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:25:54.193 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:25:54.215 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:54.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:54.216 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:25:54.317 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:25:54.324 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 66 16 106217 106201 100.556 42.5 0.00560258 0.00993994 2024-08-19T14:25:54.347 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:54.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:54.348 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:25:54.449 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:25:54.471 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:54.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:54.472 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:25:54.573 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:25:54.594 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:54.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:54.595 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:25:54.696 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:25:54.738 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:54.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:54.739 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:25:54.840 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:25:54.862 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:54.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:54.863 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:25:54.964 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:25:54.990 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:54.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:54.992 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:25:55.093 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:25:55.125 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:55.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:55.126 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:25:55.227 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:25:55.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:55.263 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:55.263 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:25:55.325 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 67 16 106926 106910 99.7161 44.3125 0.00442105 0.0100224 2024-08-19T14:25:55.364 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:25:55.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:55.402 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:55.402 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:25:55.418 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:55 smithi031 ceph-mon[16336]: pgmap v1029: 57 pgs: 10 active+recovering+undersized+remapped, 47 active+clean; 6.4 GiB data, 25 GiB used, 780 GiB / 805 GiB avail; 51 MiB/s wr, 808 op/s; 39483/313959 objects misplaced (12.576%); 29 MiB/s, 456 objects/s recovering 2024-08-19T14:25:55.504 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:25:55.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:55 smithi032 ceph-mon[20004]: pgmap v1029: 57 pgs: 10 active+recovering+undersized+remapped, 47 active+clean; 6.4 GiB data, 25 GiB used, 780 GiB / 805 GiB avail; 51 MiB/s wr, 808 op/s; 39483/313959 objects misplaced (12.576%); 29 MiB/s, 456 objects/s recovering 2024-08-19T14:25:55.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:55.535 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:55.535 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:25:55.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:55 smithi045 ceph-mon[20143]: pgmap v1029: 57 pgs: 10 active+recovering+undersized+remapped, 47 active+clean; 6.4 GiB data, 25 GiB used, 780 GiB / 805 GiB avail; 51 MiB/s wr, 808 op/s; 39483/313959 objects misplaced (12.576%); 29 MiB/s, 456 objects/s recovering 2024-08-19T14:25:55.636 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:25:55.662 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:55.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:55.663 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:25:55.710 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:25:55.760 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:25:55.763 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:25:55.795 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:55.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:55.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:25:55.897 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:25:55.933 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:55.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:55.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:25:56.035 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:25:56.079 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:56.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:56.080 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:25:56.181 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:25:56.208 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:56.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:56.209 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:25:56.311 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:25:56.325 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 68 16 107735 107719 98.9932 50.5625 0.00479008 0.0100948 2024-08-19T14:25:56.335 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:56.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:56.336 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:25:56.437 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:25:56.461 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:56.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:56.462 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:25:56.563 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:25:56.596 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:56.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:56.597 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:25:56.698 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:25:56.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:56.745 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:56.746 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:25:56.847 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:25:56.876 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:56.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:56.877 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:25:56.978 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:25:57.004 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:57.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:57.005 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:25:57.106 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:25:57.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:57.155 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:57.156 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:25:57.257 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:25:57.305 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:57.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:57.307 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:25:57.325 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 69 16 108253 108237 98.0278 32.375 0.0115586 0.0101963 2024-08-19T14:25:57.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:57 smithi031 ceph-mon[16336]: pgmap v1030: 57 pgs: 10 active+recovering+undersized+remapped, 47 active+clean; 6.4 GiB data, 25 GiB used, 779 GiB / 805 GiB avail; 46 MiB/s wr, 736 op/s; 38815/314940 objects misplaced (12.325%); 27 MiB/s, 426 objects/s recovering 2024-08-19T14:25:57.408 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:25:57.435 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:57.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:57.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:25:57.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:57 smithi032 ceph-mon[20004]: pgmap v1030: 57 pgs: 10 active+recovering+undersized+remapped, 47 active+clean; 6.4 GiB data, 25 GiB used, 779 GiB / 805 GiB avail; 46 MiB/s wr, 736 op/s; 38815/314940 objects misplaced (12.325%); 27 MiB/s, 426 objects/s recovering 2024-08-19T14:25:57.537 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:25:57.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:57 smithi045 ceph-mon[20143]: pgmap v1030: 57 pgs: 10 active+recovering+undersized+remapped, 47 active+clean; 6.4 GiB data, 25 GiB used, 779 GiB / 805 GiB avail; 46 MiB/s wr, 736 op/s; 38815/314940 objects misplaced (12.325%); 27 MiB/s, 426 objects/s recovering 2024-08-19T14:25:57.566 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:57.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:57.567 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:25:57.647 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-19T14:25:57.668 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:25:57.694 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:57.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:57.696 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:25:57.796 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:25:57.842 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:57.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:57.843 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:25:57.944 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:25:57.983 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:57.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:57.984 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:25:58.085 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:25:58.125 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:58.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:58.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:25:58.228 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:25:58.253 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:25:58.253 INFO:teuthology.orchestra.run.smithi032.stdout: "success": "" 2024-08-19T14:25:58.253 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:25:58.260 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:58.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:58.260 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:25:58.325 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 70 16 108969 108953 97.2666 44.75 0.0348249 0.0102709 2024-08-19T14:25:58.362 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:25:58.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:58.390 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:58.391 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:25:58.492 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:25:58.523 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:58.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:58.525 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:25:58.626 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:25:58.647 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:58.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:58.648 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:25:58.749 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:25:58.786 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:58.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:58.787 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:25:58.889 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:25:58.914 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:25:58.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:58.916 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:25:59.017 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:25:59.051 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:59.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:59.052 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:25:59.153 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:25:59.176 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:59.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:59.177 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:25:59.278 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:25:59.315 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:59.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:59.316 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:25:59.325 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 71 16 109647 109631 96.4935 42.375 0.0330217 0.0103576 2024-08-19T14:25:59.416 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:25:59.459 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:59.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:59.460 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:25:59.464 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:25:59 smithi032 ceph-mon[20004]: pgmap v1031: 57 pgs: 10 active+recovering+undersized+remapped, 47 active+clean; 6.5 GiB data, 26 GiB used, 779 GiB / 805 GiB avail; 52 MiB/s wr, 837 op/s; 36093/320991 objects misplaced (11.244%); 34 MiB/s, 549 objects/s recovering 2024-08-19T14:25:59.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:25:59 smithi045 ceph-mon[20143]: pgmap v1031: 57 pgs: 10 active+recovering+undersized+remapped, 47 active+clean; 6.5 GiB data, 26 GiB used, 779 GiB / 805 GiB avail; 52 MiB/s wr, 837 op/s; 36093/320991 objects misplaced (11.244%); 34 MiB/s, 549 objects/s recovering 2024-08-19T14:25:59.561 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:25:59.585 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:59.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:59.587 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:25:59.589 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:25:59 smithi031 ceph-mon[16336]: pgmap v1031: 57 pgs: 10 active+recovering+undersized+remapped, 47 active+clean; 6.5 GiB data, 26 GiB used, 779 GiB / 805 GiB avail; 52 MiB/s wr, 837 op/s; 36093/320991 objects misplaced (11.244%); 34 MiB/s, 549 objects/s recovering 2024-08-19T14:25:59.687 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:25:59.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:59.725 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:25:59.725 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:25:59.826 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:25:59.850 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:59.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:59.851 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:25:59.952 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:25:59.973 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:25:59.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:25:59.975 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:26:00.076 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:26:00.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:00.097 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:00.098 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:26:00.198 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:26:00.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:00.235 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:00.235 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:26:00.325 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 72 16 110062 110046 95.5136 25.9375 0.00942689 0.0104529 2024-08-19T14:26:00.337 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:26:00.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:00.362 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:00.362 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:26:00.464 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:26:00.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:00.499 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:00.499 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:26:00.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:00 smithi032 ceph-mon[20004]: osdmap e450: 12 total, 12 up, 9 in 2024-08-19T14:26:00.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:00 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1953215907' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:26:00.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:00 smithi045 ceph-mon[20143]: osdmap e450: 12 total, 12 up, 9 in 2024-08-19T14:26:00.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:00 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1953215907' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:26:00.600 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:26:00.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:00 smithi031 ceph-mon[16336]: osdmap e450: 12 total, 12 up, 9 in 2024-08-19T14:26:00.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:00 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1953215907' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:26:00.630 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:00.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:00.631 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:26:00.641 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:26:00.641 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:26:00.641 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:26:00.641 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:26:00.642 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:26:00.642 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:26:00.642 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:26:00.642 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:26:00.642 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:26:00.642 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:26:00.642 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:26:00.642 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:26:00.642 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:26:00.642 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:26:00.643 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:26:00.643 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:26:00.643 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:26:00.643 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:26:00.643 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:26:00.643 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:26:00.643 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:26:00.643 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:26:00.643 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:26:00.643 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:26:00.732 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:26:00.755 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:00.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:00.756 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:26:00.857 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:26:00.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:00.891 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:00.892 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:26:00.993 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:26:01.027 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:01.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:01.028 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:26:01.130 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:26:01.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:01.172 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:01.172 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:26:01.274 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:26:01.295 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:01.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:01.296 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:26:01.325 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 73 16 110400 110384 94.4946 21.125 0.00428917 0.01046 2024-08-19T14:26:01.396 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:26:01.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:01.428 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:01.429 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:26:01.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:01 smithi032 ceph-mon[20004]: pgmap v1033: 57 pgs: 10 active+recovering+undersized+remapped, 47 active+clean; 6.6 GiB data, 26 GiB used, 779 GiB / 805 GiB avail; 47 MiB/s wr, 744 op/s; 35146/324255 objects misplaced (10.839%); 32 MiB/s, 508 objects/s recovering 2024-08-19T14:26:01.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:01 smithi032 ceph-mon[20004]: osdmap e451: 12 total, 12 up, 9 in 2024-08-19T14:26:01.530 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:26:01.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:01 smithi045 ceph-mon[20143]: pgmap v1033: 57 pgs: 10 active+recovering+undersized+remapped, 47 active+clean; 6.6 GiB data, 26 GiB used, 779 GiB / 805 GiB avail; 47 MiB/s wr, 744 op/s; 35146/324255 objects misplaced (10.839%); 32 MiB/s, 508 objects/s recovering 2024-08-19T14:26:01.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:01 smithi045 ceph-mon[20143]: osdmap e451: 12 total, 12 up, 9 in 2024-08-19T14:26:01.551 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:01.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:01.553 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:26:01.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:01 smithi031 ceph-mon[16336]: pgmap v1033: 57 pgs: 10 active+recovering+undersized+remapped, 47 active+clean; 6.6 GiB data, 26 GiB used, 779 GiB / 805 GiB avail; 47 MiB/s wr, 744 op/s; 35146/324255 objects misplaced (10.839%); 32 MiB/s, 508 objects/s recovering 2024-08-19T14:26:01.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:01 smithi031 ceph-mon[16336]: osdmap e451: 12 total, 12 up, 9 in 2024-08-19T14:26:01.653 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:26:01.687 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:01.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:01.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:26:01.789 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:26:01.818 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:01.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:01.819 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:26:01.921 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:26:01.942 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:01.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:01.943 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:26:02.044 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:26:02.067 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:02.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:02.068 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:26:02.169 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:26:02.193 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:02.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:02.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:26:02.295 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:26:02.319 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:02.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:02.320 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:26:02.325 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 74 16 110400 110384 93.2177 0 - 0.01046 2024-08-19T14:26:02.420 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:26:02.443 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:02.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:02.444 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:26:02.546 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:26:02.568 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:02.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:02.569 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:26:02.670 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:26:02.702 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:02.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:02.703 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:26:02.804 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:26:02.825 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:02.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:02.827 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:26:02.928 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:26:02.952 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:02.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:02.953 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:26:02.987 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:26:03.054 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:26:03.076 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:03.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:03.077 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:26:03.178 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:26:03.200 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:03.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:03.201 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:26:03.302 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:26:03.325 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 75 16 110400 110384 91.9748 0 - 0.01046 2024-08-19T14:26:03.326 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:03.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:03.327 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:26:03.428 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:26:03.449 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:03.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:03.450 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:26:03.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:03 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:26:03.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:03 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:26:03.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:03 smithi032 ceph-mon[20004]: pgmap v1035: 57 pgs: 10 active+recovering+undersized+remapped, 47 active+clean; 6.7 GiB data, 26 GiB used, 778 GiB / 805 GiB avail; 40 MiB/s wr, 636 op/s; 31980/329250 objects misplaced (9.713%); 35 MiB/s, 564 objects/s recovering 2024-08-19T14:26:03.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:03 smithi032 ceph-mon[20004]: osdmap e452: 12 total, 12 up, 9 in 2024-08-19T14:26:03.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:03 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:26:03.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:03 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:26:03.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:03 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:26:03.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:03 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:26:03.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:03 smithi045 ceph-mon[20143]: pgmap v1035: 57 pgs: 10 active+recovering+undersized+remapped, 47 active+clean; 6.7 GiB data, 26 GiB used, 778 GiB / 805 GiB avail; 40 MiB/s wr, 636 op/s; 31980/329250 objects misplaced (9.713%); 35 MiB/s, 564 objects/s recovering 2024-08-19T14:26:03.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:03 smithi045 ceph-mon[20143]: osdmap e452: 12 total, 12 up, 9 in 2024-08-19T14:26:03.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:03 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:26:03.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:03 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:26:03.552 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:26:03.583 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:03.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:03.584 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:26:03.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:03 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:26:03.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:03 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:26:03.618 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:03 smithi031 ceph-mon[16336]: pgmap v1035: 57 pgs: 10 active+recovering+undersized+remapped, 47 active+clean; 6.7 GiB data, 26 GiB used, 778 GiB / 805 GiB avail; 40 MiB/s wr, 636 op/s; 31980/329250 objects misplaced (9.713%); 35 MiB/s, 564 objects/s recovering 2024-08-19T14:26:03.618 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:03 smithi031 ceph-mon[16336]: osdmap e452: 12 total, 12 up, 9 in 2024-08-19T14:26:03.618 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:03 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:26:03.618 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:03 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:26:03.686 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:26:03.708 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:03.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:03.709 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:26:03.810 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:26:03.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:03.848 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:03.849 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:26:03.950 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:26:03.975 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:03.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:03.976 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:26:04.077 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:26:04.111 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:04.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:04.113 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:26:04.214 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:26:04.236 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:04.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:04.237 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:26:04.325 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 76 16 110400 110384 90.7647 0 - 0.01046 2024-08-19T14:26:04.338 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:26:04.369 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:04.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:04.371 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:26:04.471 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:26:04.496 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:04.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:04.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:26:04.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:04 smithi032 ceph-mon[20004]: osdmap e453: 12 total, 12 up, 9 in 2024-08-19T14:26:04.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:04 smithi045 ceph-mon[20143]: osdmap e453: 12 total, 12 up, 9 in 2024-08-19T14:26:04.599 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:26:04.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:04 smithi031 ceph-mon[16336]: osdmap e453: 12 total, 12 up, 9 in 2024-08-19T14:26:04.620 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:04.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:04.621 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:26:04.722 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:26:04.744 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:04.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:04.745 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:26:04.846 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:26:04.870 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:04.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:04.871 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:26:04.972 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:26:04.996 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:04.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:04.998 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:26:05.099 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:26:05.122 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:05.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:05.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:26:05.224 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:26:05.246 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:05.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:05.247 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:26:05.325 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 77 16 110400 110384 89.586 0 - 0.01046 2024-08-19T14:26:05.348 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:26:05.372 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:05.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:05.373 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:26:05.474 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:26:05.495 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:05.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:05.496 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:26:05.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:05 smithi032 ceph-mon[20004]: pgmap v1038: 57 pgs: 9 active+recovering+undersized+remapped, 48 active+clean; 6.7 GiB data, 26 GiB used, 778 GiB / 805 GiB avail; 29 MiB/s wr, 462 op/s; 30129/330915 objects misplaced (9.105%); 36 MiB/s, 575 objects/s recovering 2024-08-19T14:26:05.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:05 smithi045 ceph-mon[20143]: pgmap v1038: 57 pgs: 9 active+recovering+undersized+remapped, 48 active+clean; 6.7 GiB data, 26 GiB used, 778 GiB / 805 GiB avail; 29 MiB/s wr, 462 op/s; 30129/330915 objects misplaced (9.105%); 36 MiB/s, 575 objects/s recovering 2024-08-19T14:26:05.597 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:26:05.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:05 smithi031 ceph-mon[16336]: pgmap v1038: 57 pgs: 9 active+recovering+undersized+remapped, 48 active+clean; 6.7 GiB data, 26 GiB used, 778 GiB / 805 GiB avail; 29 MiB/s wr, 462 op/s; 30129/330915 objects misplaced (9.105%); 36 MiB/s, 575 objects/s recovering 2024-08-19T14:26:05.619 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:05.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:05.620 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:26:05.721 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:26:05.742 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:05.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:05.743 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:26:05.844 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:26:05.873 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:05.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:05.874 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:26:05.975 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:26:06.000 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:06.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:06.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:26:06.102 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:26:06.123 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:06.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:06.124 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:26:06.226 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:26:06.250 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:06.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:06.251 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:26:06.326 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 78 16 110466 110450 88.4904 0.825 0.0103784 0.011296 2024-08-19T14:26:06.352 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:26:06.378 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:06.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:06.379 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:26:06.480 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:26:06.504 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:06.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:06.505 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:26:06.606 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:26:06.642 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:06.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:06.643 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:26:06.744 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:26:06.770 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:06.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:06.771 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:26:06.873 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:26:06.894 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:06.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:06.895 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:26:06.996 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:26:07.031 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:07.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:07.032 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:26:07.133 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:26:07.158 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:07.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:07.159 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:26:07.260 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:26:07.283 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:07.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:07.285 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:26:07.326 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 79 16 110787 110771 87.6242 20.0625 0.00771389 0.011368 2024-08-19T14:26:07.386 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:26:07.434 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:07.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:07.435 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:26:07.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:07 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:26:07.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:07 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:26:07.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:07 smithi032 ceph-mon[20004]: pgmap v1039: 57 pgs: 9 active+recovering+undersized+remapped, 48 active+clean; 6.7 GiB data, 27 GiB used, 778 GiB / 805 GiB avail; 24 MiB/s wr, 376 op/s; 30126/331041 objects misplaced (9.100%); 29 MiB/s, 459 objects/s recovering 2024-08-19T14:26:07.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:07 smithi032 ceph-mon[20004]: osdmap e454: 12 total, 12 up, 9 in 2024-08-19T14:26:07.536 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:26:07.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:07 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:26:07.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:07 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:26:07.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:07 smithi045 ceph-mon[20143]: pgmap v1039: 57 pgs: 9 active+recovering+undersized+remapped, 48 active+clean; 6.7 GiB data, 27 GiB used, 778 GiB / 805 GiB avail; 24 MiB/s wr, 376 op/s; 30126/331041 objects misplaced (9.100%); 29 MiB/s, 459 objects/s recovering 2024-08-19T14:26:07.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:07 smithi045 ceph-mon[20143]: osdmap e454: 12 total, 12 up, 9 in 2024-08-19T14:26:07.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:07.575 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:07.575 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:26:07.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:07 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:26:07.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:07 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:26:07.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:07 smithi031 ceph-mon[16336]: pgmap v1039: 57 pgs: 9 active+recovering+undersized+remapped, 48 active+clean; 6.7 GiB data, 27 GiB used, 778 GiB / 805 GiB avail; 24 MiB/s wr, 376 op/s; 30126/331041 objects misplaced (9.100%); 29 MiB/s, 459 objects/s recovering 2024-08-19T14:26:07.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:07 smithi031 ceph-mon[16336]: osdmap e454: 12 total, 12 up, 9 in 2024-08-19T14:26:07.676 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:26:07.712 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:07.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:07.713 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:26:07.814 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:26:07.841 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:07.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:07.842 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:26:07.943 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:26:07.970 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:07.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:07.971 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:26:08.071 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:26:08.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:08.111 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:08.111 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:26:08.213 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:26:08.245 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:08.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:08.246 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:26:08.326 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:2024-08-19T14:26:08.326367+0000 min lat: 0.00112215 max lat: 6.38635 avg lat: 0.0114498 2024-08-19T14:26:08.326 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-19T14:26:08.326 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 80 16 111677 111661 87.2242 55.625 0.0114829 0.0114498 2024-08-19T14:26:08.346 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:26:08.373 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:08.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:08.374 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:26:08.475 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:26:08.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:08.505 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:08.506 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:26:08.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:08 smithi045 ceph-mon[20143]: osdmap e455: 12 total, 12 up, 9 in 2024-08-19T14:26:08.607 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:26:08.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:08 smithi031 ceph-mon[16336]: osdmap e455: 12 total, 12 up, 9 in 2024-08-19T14:26:08.644 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:08.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:08.645 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:26:08.746 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:26:08.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:08 smithi032 ceph-mon[20004]: osdmap e455: 12 total, 12 up, 9 in 2024-08-19T14:26:08.778 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:08.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:08.779 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:26:08.880 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:26:08.907 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:08.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:08.908 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:26:09.009 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:26:09.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:09.048 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:09.048 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:26:09.149 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:26:09.186 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:09.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:09.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:26:09.288 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:26:09.310 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 7, 8, 9, 11] out_osds: [10, 6, 2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-08-19T14:26:09.311 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-19T14:26:09.311 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-08-19T14:26:09.311 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-19T14:26:09.325 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:09.326 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 81 16 112779 112763 86.9976 68.875 0.0290046 0.0114904 2024-08-19T14:26:09.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:09.327 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:26:09.427 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:26:09.448 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:09.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:09.449 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:26:09.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:09 smithi045 ceph-mon[20143]: pgmap v1042: 57 pgs: 9 active+recovering+undersized+remapped, 48 active+clean; 6.7 GiB data, 27 GiB used, 778 GiB / 805 GiB avail; 8.2 MiB/s wr, 130 op/s; 26496/331515 objects misplaced (7.992%); 30 MiB/s, 472 objects/s recovering 2024-08-19T14:26:09.550 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:26:09.585 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:09.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:09.586 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:26:09.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:09 smithi031 ceph-mon[16336]: pgmap v1042: 57 pgs: 9 active+recovering+undersized+remapped, 48 active+clean; 6.7 GiB data, 27 GiB used, 778 GiB / 805 GiB avail; 8.2 MiB/s wr, 130 op/s; 26496/331515 objects misplaced (7.992%); 30 MiB/s, 472 objects/s recovering 2024-08-19T14:26:09.687 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:26:09.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:09.720 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:09.720 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:26:09.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:09 smithi032 ceph-mon[20004]: pgmap v1042: 57 pgs: 9 active+recovering+undersized+remapped, 48 active+clean; 6.7 GiB data, 27 GiB used, 778 GiB / 805 GiB avail; 8.2 MiB/s wr, 130 op/s; 26496/331515 objects misplaced (7.992%); 30 MiB/s, 472 objects/s recovering 2024-08-19T14:26:09.821 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:26:09.841 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:09.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:09.842 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:26:09.943 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:26:09.972 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:09.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:09.973 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:26:10.074 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:26:10.111 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:10.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:10.112 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:26:10.213 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:26:10.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:10.252 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:10.252 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:26:10.326 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 82 16 113559 113543 86.5311 48.75 0.00464203 0.0115468 2024-08-19T14:26:10.353 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:26:10.385 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:10.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:10.386 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:26:10.487 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:26:10.521 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:10.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:10.522 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:26:10.623 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:26:10.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:10.655 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:10.655 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:26:10.756 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:26:10.783 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:10.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:10.784 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:26:10.885 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:26:10.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:10.918 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:10.918 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:26:11.019 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:26:11.049 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:11.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:11.050 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:26:11.151 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:26:11.181 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:11.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:11.182 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:26:11.283 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:26:11.319 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:11.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:11.320 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:26:11.326 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 83 16 114532 114516 86.2212 60.8125 0.00525645 0.0115934 2024-08-19T14:26:11.421 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:26:11.460 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:11.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:11.461 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:26:11.562 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:26:11.586 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:11.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:11.587 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:26:11.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:11 smithi031 ceph-mon[16336]: pgmap v1043: 57 pgs: 9 active+recovering+undersized+remapped, 48 active+clean; 6.8 GiB data, 27 GiB used, 778 GiB / 805 GiB avail; 8.3 MiB/s wr, 132 op/s; 24590/332880 objects misplaced (7.387%); 31 MiB/s, 495 objects/s recovering 2024-08-19T14:26:11.688 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:26:11.725 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:11.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:11.726 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:26:11.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:11 smithi032 ceph-mon[20004]: pgmap v1043: 57 pgs: 9 active+recovering+undersized+remapped, 48 active+clean; 6.8 GiB data, 27 GiB used, 778 GiB / 805 GiB avail; 8.3 MiB/s wr, 132 op/s; 24590/332880 objects misplaced (7.387%); 31 MiB/s, 495 objects/s recovering 2024-08-19T14:26:11.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:11 smithi045 ceph-mon[20143]: pgmap v1043: 57 pgs: 9 active+recovering+undersized+remapped, 48 active+clean; 6.8 GiB data, 27 GiB used, 778 GiB / 805 GiB avail; 8.3 MiB/s wr, 132 op/s; 24590/332880 objects misplaced (7.387%); 31 MiB/s, 495 objects/s recovering 2024-08-19T14:26:11.827 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:26:11.856 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:11.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:11.857 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:26:11.958 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:26:11.990 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:11.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:11.991 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:26:12.046 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:26:12.046 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 20, 2024-08-19T14:26:12.047 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 600, 2024-08-19T14:26:12.047 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [ 2024-08-19T14:26:12.047 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.047 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:241301 2.4 2:26065fe2:::benchmark_data_smithi045_46415_object118020:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-08-19T14:26:12.047 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:16:40.774899+0000", 2024-08-19T14:26:12.047 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 570.36468191100005, 2024-08-19T14:26:12.047 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.29898516600000002, 2024-08-19T14:26:12.047 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:26:12.047 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:26:12.047 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:26:12.047 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:26:12.047 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:26:12.047 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:26:12.048 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 241301 2024-08-19T14:26:12.048 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.048 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:26:12.048 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.048 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:26:12.048 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.774899+0000", 2024-08-19T14:26:12.048 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:26:12.048 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.048 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.048 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:26:12.048 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.774899+0000", 2024-08-19T14:26:12.048 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:26:12.048 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.049 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.049 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:26:12.049 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.774900+0000", 2024-08-19T14:26:12.049 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6190000000000001e-06 2024-08-19T14:26:12.049 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.049 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.049 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:26:12.049 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.774907+0000", 2024-08-19T14:26:12.049 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.6540000000000002e-06 2024-08-19T14:26:12.049 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.049 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.049 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:26:12.050 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.774921+0000", 2024-08-19T14:26:12.050 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4355000000000001e-05 2024-08-19T14:26:12.050 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.050 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.050 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:26:12.050 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.774928+0000", 2024-08-19T14:26:12.050 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.2339999999999997e-06 2024-08-19T14:26:12.050 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.050 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.050 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:26:12.050 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.780109+0000", 2024-08-19T14:26:12.050 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0051809409999999997 2024-08-19T14:26:12.051 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.051 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.051 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:26:12.051 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.780240+0000", 2024-08-19T14:26:12.051 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000131006 2024-08-19T14:26:12.051 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.051 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.051 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,2,8", 2024-08-19T14:26:12.051 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.780378+0000", 2024-08-19T14:26:12.051 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013800999999999999 2024-08-19T14:26:12.051 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.051 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.051 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:26:12.052 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.794340+0000", 2024-08-19T14:26:12.052 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.013961559 2024-08-19T14:26:12.052 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.052 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.052 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:26:12.052 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.802616+0000", 2024-08-19T14:26:12.052 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0082755490000000001 2024-08-19T14:26:12.052 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.053 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.053 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:26:12.053 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.809365+0000", 2024-08-19T14:26:12.053 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0067497039999999996 2024-08-19T14:26:12.053 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.053 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.053 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:26:12.053 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.073684+0000", 2024-08-19T14:26:12.053 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.26431834199999998 2024-08-19T14:26:12.053 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.053 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.054 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:26:12.054 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.073749+0000", 2024-08-19T14:26:12.054 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.5343000000000001e-05 2024-08-19T14:26:12.054 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.054 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.054 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:26:12.054 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.073884+0000", 2024-08-19T14:26:12.054 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013485 2024-08-19T14:26:12.054 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:26:12.054 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:26:12.054 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:26:12.054 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.055 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.055 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:241310 2.4 2:2df264d1:::benchmark_data_smithi045_46415_object118029:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-08-19T14:26:12.055 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:16:40.777745+0000", 2024-08-19T14:26:12.055 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 570.36183542200001, 2024-08-19T14:26:12.055 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.34201599500000002, 2024-08-19T14:26:12.055 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:26:12.055 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:26:12.055 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:26:12.055 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:26:12.055 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:26:12.055 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:26:12.055 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 241310 2024-08-19T14:26:12.055 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.056 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:26:12.056 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.056 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:26:12.056 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.777745+0000", 2024-08-19T14:26:12.056 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:26:12.056 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.056 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.056 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:26:12.056 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.777745+0000", 2024-08-19T14:26:12.056 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:26:12.056 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.056 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.056 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:26:12.056 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.777746+0000", 2024-08-19T14:26:12.057 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.5999999999999991e-07 2024-08-19T14:26:12.057 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.057 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.057 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:26:12.057 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.777751+0000", 2024-08-19T14:26:12.057 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.6369999999999998e-06 2024-08-19T14:26:12.057 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.057 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.057 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:26:12.057 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.777763+0000", 2024-08-19T14:26:12.057 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2279000000000001e-05 2024-08-19T14:26:12.057 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.057 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.057 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:26:12.058 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.777771+0000", 2024-08-19T14:26:12.058 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.6109999999999996e-06 2024-08-19T14:26:12.058 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.058 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.058 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:26:12.058 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.788985+0000", 2024-08-19T14:26:12.058 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.011214636 2024-08-19T14:26:12.058 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.058 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.058 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:26:12.058 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.789090+0000", 2024-08-19T14:26:12.058 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000104583 2024-08-19T14:26:12.058 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.059 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.059 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,2,8", 2024-08-19T14:26:12.059 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.789223+0000", 2024-08-19T14:26:12.059 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013313299999999999 2024-08-19T14:26:12.059 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.059 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.059 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:26:12.059 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.802660+0000", 2024-08-19T14:26:12.059 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.013437065999999999 2024-08-19T14:26:12.059 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.059 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.059 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:26:12.059 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.808077+0000", 2024-08-19T14:26:12.060 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0054167759999999999 2024-08-19T14:26:12.060 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.060 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.060 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:26:12.060 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.812929+0000", 2024-08-19T14:26:12.060 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0048518399999999996 2024-08-19T14:26:12.060 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.060 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.060 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:26:12.060 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.119563+0000", 2024-08-19T14:26:12.060 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.30663414300000003 2024-08-19T14:26:12.060 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.060 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.060 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:26:12.061 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.119627+0000", 2024-08-19T14:26:12.061 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.4622999999999994e-05 2024-08-19T14:26:12.061 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.061 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.061 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:26:12.061 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.119761+0000", 2024-08-19T14:26:12.061 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013370799999999999 2024-08-19T14:26:12.061 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:26:12.061 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:26:12.061 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:26:12.061 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.062 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.062 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:241313 2.4 2:22ac58a2:::benchmark_data_smithi045_46415_object118032:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-08-19T14:26:12.062 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:16:40.785407+0000", 2024-08-19T14:26:12.062 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 570.35417357899996, 2024-08-19T14:26:12.062 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.33459377200000001, 2024-08-19T14:26:12.062 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:26:12.062 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:26:12.062 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:26:12.062 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:26:12.062 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:26:12.062 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:26:12.062 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 241313 2024-08-19T14:26:12.062 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.063 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:26:12.063 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.063 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:26:12.063 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.785407+0000", 2024-08-19T14:26:12.063 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:26:12.063 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.063 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.063 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:26:12.063 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.785407+0000", 2024-08-19T14:26:12.063 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:26:12.064 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.064 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.064 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:26:12.064 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.785409+0000", 2024-08-19T14:26:12.064 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6109999999999999e-06 2024-08-19T14:26:12.064 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.064 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.064 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:26:12.064 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.785414+0000", 2024-08-19T14:26:12.064 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1919999999999996e-06 2024-08-19T14:26:12.064 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.064 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.065 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:26:12.065 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.785424+0000", 2024-08-19T14:26:12.065 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0719e-05 2024-08-19T14:26:12.065 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.065 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.065 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:26:12.065 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.785430+0000", 2024-08-19T14:26:12.065 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.8019999999999997e-06 2024-08-19T14:26:12.065 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.065 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.065 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:26:12.065 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.794371+0000", 2024-08-19T14:26:12.065 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0089406929999999996 2024-08-19T14:26:12.065 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.066 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.066 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:26:12.066 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.794483+0000", 2024-08-19T14:26:12.066 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000111539 2024-08-19T14:26:12.066 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.066 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.066 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,2,8", 2024-08-19T14:26:12.066 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.794630+0000", 2024-08-19T14:26:12.066 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00014782700000000001 2024-08-19T14:26:12.066 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.066 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.066 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:26:12.066 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.802700+0000", 2024-08-19T14:26:12.067 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0080701290000000005 2024-08-19T14:26:12.067 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.067 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.067 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:26:12.067 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.808117+0000", 2024-08-19T14:26:12.067 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0054165469999999999 2024-08-19T14:26:12.067 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.067 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.067 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:26:12.067 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.812933+0000", 2024-08-19T14:26:12.067 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0048158460000000004 2024-08-19T14:26:12.067 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.067 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.067 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:26:12.068 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.119824+0000", 2024-08-19T14:26:12.068 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.30689145899999998 2024-08-19T14:26:12.068 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.068 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.068 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:26:12.068 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.119884+0000", 2024-08-19T14:26:12.068 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.9642000000000003e-05 2024-08-19T14:26:12.068 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.068 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.068 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:26:12.068 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.120001+0000", 2024-08-19T14:26:12.068 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000116766 2024-08-19T14:26:12.068 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:26:12.068 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:26:12.069 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:26:12.069 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.069 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.069 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:241315 2.4 2:24bb1245:::benchmark_data_smithi045_46415_object118034:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-08-19T14:26:12.069 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:16:40.789772+0000", 2024-08-19T14:26:12.069 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 570.34980813300001, 2024-08-19T14:26:12.069 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.33090881500000002, 2024-08-19T14:26:12.069 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:26:12.069 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:26:12.069 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:26:12.069 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:26:12.069 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:26:12.069 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:26:12.070 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 241315 2024-08-19T14:26:12.070 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.070 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:26:12.070 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.070 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:26:12.070 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.789772+0000", 2024-08-19T14:26:12.070 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:26:12.070 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.070 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.070 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:26:12.070 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.789772+0000", 2024-08-19T14:26:12.070 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:26:12.071 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.071 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.071 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:26:12.071 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.789774+0000", 2024-08-19T14:26:12.071 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.483e-06 2024-08-19T14:26:12.071 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.071 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.071 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:26:12.071 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.789780+0000", 2024-08-19T14:26:12.071 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.9390000000000002e-06 2024-08-19T14:26:12.071 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.071 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.071 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:26:12.071 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.789796+0000", 2024-08-19T14:26:12.072 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5889999999999999e-05 2024-08-19T14:26:12.072 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.072 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.072 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:26:12.072 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.789803+0000", 2024-08-19T14:26:12.072 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.3900000000000004e-06 2024-08-19T14:26:12.072 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.072 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.072 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:26:12.072 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.802735+0000", 2024-08-19T14:26:12.072 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.012932332 2024-08-19T14:26:12.072 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.072 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.072 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:26:12.073 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.802891+0000", 2024-08-19T14:26:12.073 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000155301 2024-08-19T14:26:12.073 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.073 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.073 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,2,8", 2024-08-19T14:26:12.073 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.803031+0000", 2024-08-19T14:26:12.073 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000140414 2024-08-19T14:26:12.073 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.073 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.073 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:26:12.073 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.809352+0000", 2024-08-19T14:26:12.073 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0063210699999999998 2024-08-19T14:26:12.073 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.073 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.074 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:26:12.074 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.812937+0000", 2024-08-19T14:26:12.074 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0035844050000000001 2024-08-19T14:26:12.074 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.074 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.074 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:26:12.074 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.812965+0000", 2024-08-19T14:26:12.074 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.8099999999999999e-05 2024-08-19T14:26:12.074 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.074 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.074 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:26:12.074 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.120323+0000", 2024-08-19T14:26:12.074 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.30735870700000001 2024-08-19T14:26:12.075 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.075 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.075 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:26:12.075 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.120566+0000", 2024-08-19T14:26:12.075 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00024232400000000001 2024-08-19T14:26:12.075 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.075 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.075 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:26:12.075 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.120681+0000", 2024-08-19T14:26:12.075 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00011546 2024-08-19T14:26:12.075 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:26:12.075 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:26:12.075 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:26:12.076 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.076 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.076 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:241316 2.4 2:2a938701:::benchmark_data_smithi045_46415_object118035:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-08-19T14:26:12.076 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:16:40.789816+0000", 2024-08-19T14:26:12.076 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 570.34976406099997, 2024-08-19T14:26:12.076 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.33160296900000003, 2024-08-19T14:26:12.076 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:26:12.076 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:26:12.076 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:26:12.076 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:26:12.076 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:26:12.076 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:26:12.076 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 241316 2024-08-19T14:26:12.077 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.077 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:26:12.077 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.077 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:26:12.077 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.789816+0000", 2024-08-19T14:26:12.077 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:26:12.077 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.077 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.077 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:26:12.077 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.789816+0000", 2024-08-19T14:26:12.077 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:26:12.077 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.077 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.078 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:26:12.078 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.789817+0000", 2024-08-19T14:26:12.078 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.0699999999999998e-07 2024-08-19T14:26:12.078 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.078 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.078 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:26:12.078 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.789819+0000", 2024-08-19T14:26:12.078 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.4380000000000002e-06 2024-08-19T14:26:12.078 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.078 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.079 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:26:12.079 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.789826+0000", 2024-08-19T14:26:12.079 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.4060000000000002e-06 2024-08-19T14:26:12.079 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.079 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.079 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:26:12.079 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.789831+0000", 2024-08-19T14:26:12.079 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.8960000000000002e-06 2024-08-19T14:26:12.079 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.079 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.079 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:26:12.079 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.812990+0000", 2024-08-19T14:26:12.080 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.023159863999999999 2024-08-19T14:26:12.080 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.080 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.080 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:26:12.080 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.813089+0000", 2024-08-19T14:26:12.080 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.8478000000000001e-05 2024-08-19T14:26:12.080 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.080 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.080 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,2,8", 2024-08-19T14:26:12.080 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.813175+0000", 2024-08-19T14:26:12.080 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.5834000000000005e-05 2024-08-19T14:26:12.080 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.080 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.081 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:26:12.081 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.818905+0000", 2024-08-19T14:26:12.081 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0057298870000000003 2024-08-19T14:26:12.081 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.081 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.081 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:26:12.081 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.820238+0000", 2024-08-19T14:26:12.081 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.001333603 2024-08-19T14:26:12.081 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.081 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.081 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:26:12.081 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.831561+0000", 2024-08-19T14:26:12.081 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.011322334 2024-08-19T14:26:12.081 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.082 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.082 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:26:12.082 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.120731+0000", 2024-08-19T14:26:12.082 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.289170123 2024-08-19T14:26:12.082 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.082 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.082 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:26:12.082 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.121237+0000", 2024-08-19T14:26:12.082 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00050635100000000004 2024-08-19T14:26:12.082 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.082 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.082 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:26:12.083 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.121419+0000", 2024-08-19T14:26:12.083 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00018234800000000001 2024-08-19T14:26:12.083 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:26:12.083 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:26:12.083 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:26:12.083 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.083 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.083 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:241344 2.4 2:2b291422:::benchmark_data_smithi045_46415_object118063:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-08-19T14:26:12.083 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:16:40.823839+0000", 2024-08-19T14:26:12.083 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 570.31574124400004, 2024-08-19T14:26:12.083 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.29983483399999999, 2024-08-19T14:26:12.083 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:26:12.083 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:26:12.084 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:26:12.084 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:26:12.084 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:26:12.084 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:26:12.084 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 241344 2024-08-19T14:26:12.084 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.084 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:26:12.084 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.084 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:26:12.084 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.823839+0000", 2024-08-19T14:26:12.084 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:26:12.084 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.084 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.085 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:26:12.085 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.823839+0000", 2024-08-19T14:26:12.085 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:26:12.085 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.085 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.085 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:26:12.085 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.823841+0000", 2024-08-19T14:26:12.085 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.607e-06 2024-08-19T14:26:12.085 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.085 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.085 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:26:12.085 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.823847+0000", 2024-08-19T14:26:12.085 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.7620000000000001e-06 2024-08-19T14:26:12.085 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.086 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.086 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:26:12.086 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.823865+0000", 2024-08-19T14:26:12.086 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.8134999999999998e-05 2024-08-19T14:26:12.086 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.086 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.086 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:26:12.086 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.823872+0000", 2024-08-19T14:26:12.086 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.7039999999999994e-06 2024-08-19T14:26:12.086 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.086 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.086 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:26:12.086 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.824022+0000", 2024-08-19T14:26:12.087 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000149405 2024-08-19T14:26:12.087 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.087 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.087 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:26:12.087 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.824180+0000", 2024-08-19T14:26:12.087 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00015828199999999999 2024-08-19T14:26:12.087 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.087 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.088 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,2,8", 2024-08-19T14:26:12.088 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.824332+0000", 2024-08-19T14:26:12.088 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00015196799999999999 2024-08-19T14:26:12.088 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.088 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.088 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:26:12.089 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.831601+0000", 2024-08-19T14:26:12.089 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0072684150000000003 2024-08-19T14:26:12.089 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.089 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.089 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:26:12.089 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.832795+0000", 2024-08-19T14:26:12.089 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0011945739999999999 2024-08-19T14:26:12.090 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.090 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.090 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:26:12.090 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.843115+0000", 2024-08-19T14:26:12.090 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.010319488 2024-08-19T14:26:12.090 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.090 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.090 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:26:12.090 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.121468+0000", 2024-08-19T14:26:12.090 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.27835382199999997 2024-08-19T14:26:12.090 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.090 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.090 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:26:12.091 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.123536+0000", 2024-08-19T14:26:12.091 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.002067892 2024-08-19T14:26:12.091 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.091 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.091 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:26:12.091 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.123674+0000", 2024-08-19T14:26:12.091 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013778 2024-08-19T14:26:12.091 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:26:12.091 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:26:12.092 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:26:12.092 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.092 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.092 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:241352 2.4 2:258e31fc:::benchmark_data_smithi045_46415_object118071:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-08-19T14:26:12.092 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:16:40.833613+0000", 2024-08-19T14:26:12.092 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 570.30596733499999, 2024-08-19T14:26:12.092 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.290304479, 2024-08-19T14:26:12.092 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:26:12.092 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:26:12.092 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:26:12.092 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:26:12.092 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:26:12.092 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:26:12.093 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 241352 2024-08-19T14:26:12.093 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.093 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:26:12.093 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.094 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:26:12.094 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.833613+0000", 2024-08-19T14:26:12.094 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:26:12.094 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.094 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.094 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:26:12.094 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.833613+0000", 2024-08-19T14:26:12.094 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:26:12.094 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.094 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.095 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:26:12.095 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.833615+0000", 2024-08-19T14:26:12.095 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4839999999999999e-06 2024-08-19T14:26:12.095 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.095 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.095 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:26:12.095 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.833620+0000", 2024-08-19T14:26:12.095 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.5740000000000003e-06 2024-08-19T14:26:12.095 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.095 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.095 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:26:12.095 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.833635+0000", 2024-08-19T14:26:12.095 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4893e-05 2024-08-19T14:26:12.096 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.096 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.096 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:26:12.096 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.833642+0000", 2024-08-19T14:26:12.096 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.2620000000000001e-06 2024-08-19T14:26:12.096 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.096 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.096 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:26:12.096 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.836631+0000", 2024-08-19T14:26:12.096 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0029890400000000001 2024-08-19T14:26:12.096 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.096 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.096 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:26:12.096 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.836733+0000", 2024-08-19T14:26:12.097 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000101532 2024-08-19T14:26:12.097 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.097 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.097 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,2,8", 2024-08-19T14:26:12.097 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.836833+0000", 2024-08-19T14:26:12.097 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.9652000000000001e-05 2024-08-19T14:26:12.097 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.097 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.097 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:26:12.097 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.841532+0000", 2024-08-19T14:26:12.097 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0046991769999999997 2024-08-19T14:26:12.097 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.097 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.097 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:26:12.098 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.843144+0000", 2024-08-19T14:26:12.098 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0016120139999999999 2024-08-19T14:26:12.098 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.098 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.098 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:26:12.098 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.843187+0000", 2024-08-19T14:26:12.098 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.3198000000000002e-05 2024-08-19T14:26:12.098 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.098 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.098 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:26:12.098 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.123725+0000", 2024-08-19T14:26:12.098 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.28053793599999999 2024-08-19T14:26:12.098 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.099 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.099 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:26:12.099 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.123784+0000", 2024-08-19T14:26:12.099 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.8931000000000002e-05 2024-08-19T14:26:12.099 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.099 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.099 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:26:12.099 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.123918+0000", 2024-08-19T14:26:12.099 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000133786 2024-08-19T14:26:12.099 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:26:12.099 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:26:12.099 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:26:12.099 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.099 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.100 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.24580.0:241376 2.4 2:259a9829:::benchmark_data_smithi045_46415_object118095:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-08-19T14:26:12.100 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:16:40.865722+0000", 2024-08-19T14:26:12.100 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 570.27385838600003, 2024-08-19T14:26:12.100 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.25843650499999998, 2024-08-19T14:26:12.100 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:26:12.100 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:26:12.100 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:26:12.100 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:26:12.100 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.24580", 2024-08-19T14:26:12.100 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/1876936042", 2024-08-19T14:26:12.100 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 241376 2024-08-19T14:26:12.100 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.100 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:26:12.100 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.101 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:26:12.101 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.865722+0000", 2024-08-19T14:26:12.101 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:26:12.101 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.101 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.101 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:26:12.101 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.865722+0000", 2024-08-19T14:26:12.101 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:26:12.101 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.101 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.101 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:26:12.101 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.865724+0000", 2024-08-19T14:26:12.101 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6619999999999999e-06 2024-08-19T14:26:12.101 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.101 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.102 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:26:12.102 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.865729+0000", 2024-08-19T14:26:12.102 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7040000000000002e-06 2024-08-19T14:26:12.102 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.102 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.102 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:26:12.102 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.865739+0000", 2024-08-19T14:26:12.102 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0541e-05 2024-08-19T14:26:12.102 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.102 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.102 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:26:12.102 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.865745+0000", 2024-08-19T14:26:12.102 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.6609999999999997e-06 2024-08-19T14:26:12.103 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.103 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.103 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:26:12.103 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.865926+0000", 2024-08-19T14:26:12.103 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00018153000000000001 2024-08-19T14:26:12.103 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.103 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.103 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:26:12.103 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.866094+0000", 2024-08-19T14:26:12.103 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000167844 2024-08-19T14:26:12.103 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.104 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.104 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,2,8", 2024-08-19T14:26:12.104 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.866236+0000", 2024-08-19T14:26:12.104 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00014171600000000001 2024-08-19T14:26:12.104 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.104 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.104 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:26:12.104 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.871503+0000", 2024-08-19T14:26:12.104 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0052669960000000004 2024-08-19T14:26:12.104 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.104 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.104 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:26:12.104 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.871600+0000", 2024-08-19T14:26:12.105 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.7027000000000005e-05 2024-08-19T14:26:12.105 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.105 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.105 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:26:12.105 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:40.872043+0000", 2024-08-19T14:26:12.105 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00044304300000000001 2024-08-19T14:26:12.105 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.105 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.105 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:26:12.105 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.123962+0000", 2024-08-19T14:26:12.105 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.25191884199999998 2024-08-19T14:26:12.105 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.105 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.105 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:26:12.106 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.124022+0000", 2024-08-19T14:26:12.106 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.0331999999999998e-05 2024-08-19T14:26:12.106 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.106 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.106 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:26:12.106 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:16:41.124159+0000", 2024-08-19T14:26:12.106 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013660699999999999 2024-08-19T14:26:12.106 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:26:12.106 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:26:12.106 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:26:12.106 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.106 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.106 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25057.0:47840 3.14 3:2da72587:::benchmark_data_smithi045_59973_object47839:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e234)", 2024-08-19T14:26:12.107 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:17:18.780442+0000", 2024-08-19T14:26:12.107 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 532.35913838800002, 2024-08-19T14:26:12.107 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.98449229999999999, 2024-08-19T14:26:12.107 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:26:12.107 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:26:12.107 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:26:12.107 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:26:12.107 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25057", 2024-08-19T14:26:12.107 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:26:12.107 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 47840 2024-08-19T14:26:12.107 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.107 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:26:12.107 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.107 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:26:12.107 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.780442+0000", 2024-08-19T14:26:12.108 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:26:12.108 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.108 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.108 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:26:12.108 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.780442+0000", 2024-08-19T14:26:12.108 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:26:12.108 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.108 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.108 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:26:12.108 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.780444+0000", 2024-08-19T14:26:12.108 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.216e-06 2024-08-19T14:26:12.108 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.108 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.108 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:26:12.109 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.780527+0000", 2024-08-19T14:26:12.109 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.3096000000000005e-05 2024-08-19T14:26:12.109 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.109 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.109 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:26:12.109 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.780548+0000", 2024-08-19T14:26:12.109 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.0424999999999998e-05 2024-08-19T14:26:12.109 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.109 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.109 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:26:12.109 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.780559+0000", 2024-08-19T14:26:12.109 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1579999999999999e-05 2024-08-19T14:26:12.109 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.109 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.110 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:26:12.110 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.815099+0000", 2024-08-19T14:26:12.110 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.034539489999999999 2024-08-19T14:26:12.110 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.110 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.110 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:26:12.110 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:18.815109+0000", 2024-08-19T14:26:12.110 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0036e-05 2024-08-19T14:26:12.110 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.110 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.110 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:26:12.110 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.756630+0000", 2024-08-19T14:26:12.110 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.94152143899999996 2024-08-19T14:26:12.111 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.111 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.111 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:26:12.111 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.756793+0000", 2024-08-19T14:26:12.111 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00016217400000000001 2024-08-19T14:26:12.111 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.111 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.111 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 5,11", 2024-08-19T14:26:12.111 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.756983+0000", 2024-08-19T14:26:12.111 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00019000000000000001 2024-08-19T14:26:12.111 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.111 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.111 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:26:12.111 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.759245+0000", 2024-08-19T14:26:12.112 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0022623140000000001 2024-08-19T14:26:12.112 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.112 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.112 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:26:12.112 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.762237+0000", 2024-08-19T14:26:12.112 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0029923850000000002 2024-08-19T14:26:12.112 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.112 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.112 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:26:12.112 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.764795+0000", 2024-08-19T14:26:12.112 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0025578240000000002 2024-08-19T14:26:12.112 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.112 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.112 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:26:12.113 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.764889+0000", 2024-08-19T14:26:12.113 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.3927999999999999e-05 2024-08-19T14:26:12.113 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.113 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.113 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:26:12.113 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:17:19.764934+0000", 2024-08-19T14:26:12.113 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.5392999999999997e-05 2024-08-19T14:26:12.113 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:26:12.113 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:26:12.113 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:26:12.113 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.113 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.113 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25057.0:188768 3.24 3:25c22bc2:::benchmark_data_smithi045_59973_object57589:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-08-19T14:26:12.114 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:19:02.299148+0000", 2024-08-19T14:26:12.114 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 428.84043230200001, 2024-08-19T14:26:12.114 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.92656360999999998, 2024-08-19T14:26:12.114 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:26:12.114 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:26:12.114 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:26:12.114 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:26:12.114 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25057", 2024-08-19T14:26:12.114 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:26:12.114 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 188768 2024-08-19T14:26:12.114 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.114 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:26:12.114 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.115 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:26:12.115 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.299148+0000", 2024-08-19T14:26:12.115 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:26:12.115 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.115 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.115 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:26:12.115 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.299148+0000", 2024-08-19T14:26:12.115 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:26:12.115 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.115 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.115 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:26:12.115 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.299150+0000", 2024-08-19T14:26:12.115 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.869e-06 2024-08-19T14:26:12.115 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.116 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.116 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:26:12.116 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.299157+0000", 2024-08-19T14:26:12.116 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.2699999999999999e-06 2024-08-19T14:26:12.116 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.116 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.116 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:26:12.116 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.299173+0000", 2024-08-19T14:26:12.116 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6099e-05 2024-08-19T14:26:12.116 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.116 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.116 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:26:12.116 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.299184+0000", 2024-08-19T14:26:12.117 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0523000000000001e-05 2024-08-19T14:26:12.117 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.117 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.117 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:26:12.117 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.299253+0000", 2024-08-19T14:26:12.117 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.8752999999999997e-05 2024-08-19T14:26:12.117 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.117 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.117 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:26:12.117 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.299265+0000", 2024-08-19T14:26:12.117 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2295e-05 2024-08-19T14:26:12.117 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.117 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.117 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:26:12.118 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:03.218622+0000", 2024-08-19T14:26:12.118 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.91935708900000002 2024-08-19T14:26:12.118 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.118 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.118 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:26:12.118 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:03.218724+0000", 2024-08-19T14:26:12.118 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00010233700000000001 2024-08-19T14:26:12.118 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.118 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.118 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 5,11", 2024-08-19T14:26:12.118 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:03.218822+0000", 2024-08-19T14:26:12.118 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.7435999999999993e-05 2024-08-19T14:26:12.118 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.118 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.119 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:26:12.119 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:03.220396+0000", 2024-08-19T14:26:12.119 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0015744450000000001 2024-08-19T14:26:12.119 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.119 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.119 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:26:12.119 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:03.223542+0000", 2024-08-19T14:26:12.119 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0031455879999999999 2024-08-19T14:26:12.119 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.119 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.119 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:26:12.119 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:03.225592+0000", 2024-08-19T14:26:12.120 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0020502789999999999 2024-08-19T14:26:12.120 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.120 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.120 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:26:12.120 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:03.225650+0000", 2024-08-19T14:26:12.120 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.7472000000000001e-05 2024-08-19T14:26:12.120 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.120 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.120 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:26:12.120 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:03.225712+0000", 2024-08-19T14:26:12.120 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.2155000000000004e-05 2024-08-19T14:26:12.120 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:26:12.120 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:26:12.120 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:26:12.121 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.121 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.121 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25333.0:22687 4.7 4:efd17a0f:::benchmark_data_smithi045_71562_object22686:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-08-19T14:26:12.121 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:20:41.348541+0000", 2024-08-19T14:26:12.121 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 329.79104001399998, 2024-08-19T14:26:12.121 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.33315307799999999, 2024-08-19T14:26:12.121 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:26:12.121 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:26:12.121 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:26:12.121 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:26:12.121 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25333", 2024-08-19T14:26:12.121 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:26:12.121 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 22687 2024-08-19T14:26:12.121 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.122 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:26:12.122 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.122 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:26:12.122 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.348541+0000", 2024-08-19T14:26:12.122 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:26:12.122 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.122 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.122 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:26:12.122 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.348541+0000", 2024-08-19T14:26:12.122 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:26:12.122 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.122 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.123 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:26:12.123 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.348541+0000", 2024-08-19T14:26:12.123 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.3e-07 2024-08-19T14:26:12.123 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.123 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.123 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:26:12.123 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.348571+0000", 2024-08-19T14:26:12.123 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.0145e-05 2024-08-19T14:26:12.123 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.123 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.123 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:26:12.123 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.348580+0000", 2024-08-19T14:26:12.124 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.3240000000000003e-06 2024-08-19T14:26:12.124 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.124 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.124 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:26:12.124 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.348586+0000", 2024-08-19T14:26:12.124 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.4500000000000001e-06 2024-08-19T14:26:12.124 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.124 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.124 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:26:12.124 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.349519+0000", 2024-08-19T14:26:12.125 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00093283700000000005 2024-08-19T14:26:12.125 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.125 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.125 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:26:12.125 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.349639+0000", 2024-08-19T14:26:12.125 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000119703 2024-08-19T14:26:12.125 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.125 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.125 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,7", 2024-08-19T14:26:12.125 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.350370+0000", 2024-08-19T14:26:12.125 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00073159899999999999 2024-08-19T14:26:12.125 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.125 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.126 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:26:12.126 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.353458+0000", 2024-08-19T14:26:12.126 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0030877280000000001 2024-08-19T14:26:12.126 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.126 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.126 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:26:12.126 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.354430+0000", 2024-08-19T14:26:12.126 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00097207399999999996 2024-08-19T14:26:12.126 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.126 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.126 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:26:12.126 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.681572+0000", 2024-08-19T14:26:12.126 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.32714184499999999 2024-08-19T14:26:12.127 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.127 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.127 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:26:12.127 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.681644+0000", 2024-08-19T14:26:12.127 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.2311000000000006e-05 2024-08-19T14:26:12.127 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.127 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.127 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:26:12.127 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.681694+0000", 2024-08-19T14:26:12.127 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.9332000000000001e-05 2024-08-19T14:26:12.127 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:26:12.127 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:26:12.127 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:26:12.127 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.128 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.128 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25333.0:22691 4.7 4:e60184b4:::benchmark_data_smithi045_71562_object22690:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-08-19T14:26:12.128 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:20:41.350039+0000", 2024-08-19T14:26:12.128 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 329.78954109099999, 2024-08-19T14:26:12.128 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.33211775999999998, 2024-08-19T14:26:12.128 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:26:12.128 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:26:12.128 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:26:12.128 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:26:12.128 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25333", 2024-08-19T14:26:12.128 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:26:12.128 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 22691 2024-08-19T14:26:12.128 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.129 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:26:12.129 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.129 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:26:12.129 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.350039+0000", 2024-08-19T14:26:12.129 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:26:12.129 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.129 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.129 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:26:12.129 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.350039+0000", 2024-08-19T14:26:12.129 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:26:12.129 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.129 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.129 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:26:12.129 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.350040+0000", 2024-08-19T14:26:12.130 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.7599999999999996e-07 2024-08-19T14:26:12.130 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.130 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.130 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:26:12.130 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.350097+0000", 2024-08-19T14:26:12.130 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.7108999999999998e-05 2024-08-19T14:26:12.130 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.130 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.130 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:26:12.130 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.350109+0000", 2024-08-19T14:26:12.130 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1926999999999999e-05 2024-08-19T14:26:12.130 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.130 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.131 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:26:12.131 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.350117+0000", 2024-08-19T14:26:12.131 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.6429999999999995e-06 2024-08-19T14:26:12.131 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.131 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.131 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:26:12.131 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.351286+0000", 2024-08-19T14:26:12.131 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0011685179999999999 2024-08-19T14:26:12.131 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.131 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.131 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:26:12.132 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.351545+0000", 2024-08-19T14:26:12.132 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00025918799999999999 2024-08-19T14:26:12.132 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.132 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.132 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,7", 2024-08-19T14:26:12.132 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.351671+0000", 2024-08-19T14:26:12.132 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012657099999999999 2024-08-19T14:26:12.132 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.132 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.132 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:26:12.132 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.354827+0000", 2024-08-19T14:26:12.132 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.003155808 2024-08-19T14:26:12.132 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.133 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.133 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:26:12.133 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.354902+0000", 2024-08-19T14:26:12.133 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.4691999999999996e-05 2024-08-19T14:26:12.133 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.133 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.133 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:26:12.133 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.682051+0000", 2024-08-19T14:26:12.133 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.32714955600000001 2024-08-19T14:26:12.133 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.133 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.134 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:26:12.134 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.682110+0000", 2024-08-19T14:26:12.134 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.8746999999999998e-05 2024-08-19T14:26:12.134 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.134 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.134 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:26:12.134 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.682157+0000", 2024-08-19T14:26:12.134 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7125e-05 2024-08-19T14:26:12.134 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:26:12.134 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:26:12.134 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:26:12.134 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.134 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.135 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25333.0:22704 4.7 4:e3ca772f:::benchmark_data_smithi045_71562_object22703:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-08-19T14:26:12.135 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:20:41.354754+0000", 2024-08-19T14:26:12.135 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 329.784826366, 2024-08-19T14:26:12.135 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.32755525600000002, 2024-08-19T14:26:12.135 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:26:12.135 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:26:12.135 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:26:12.135 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:26:12.135 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25333", 2024-08-19T14:26:12.135 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:26:12.135 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 22704 2024-08-19T14:26:12.135 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.135 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:26:12.136 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.136 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:26:12.136 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.354754+0000", 2024-08-19T14:26:12.136 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:26:12.136 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.136 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.136 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:26:12.136 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.354754+0000", 2024-08-19T14:26:12.136 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:26:12.136 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.136 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.136 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:26:12.137 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.354755+0000", 2024-08-19T14:26:12.137 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0589999999999999e-06 2024-08-19T14:26:12.137 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.137 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.137 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:26:12.137 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.354883+0000", 2024-08-19T14:26:12.137 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000127515 2024-08-19T14:26:12.137 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.137 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.137 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:26:12.138 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.354893+0000", 2024-08-19T14:26:12.138 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0291e-05 2024-08-19T14:26:12.138 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.138 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.138 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:26:12.138 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.354901+0000", 2024-08-19T14:26:12.138 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.6760000000000007e-06 2024-08-19T14:26:12.138 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.138 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.138 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:26:12.138 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.354944+0000", 2024-08-19T14:26:12.138 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.3155999999999998e-05 2024-08-19T14:26:12.138 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.139 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.139 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:26:12.139 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.355068+0000", 2024-08-19T14:26:12.139 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000124521 2024-08-19T14:26:12.139 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.139 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.139 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,7", 2024-08-19T14:26:12.139 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.355175+0000", 2024-08-19T14:26:12.139 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000106984 2024-08-19T14:26:12.139 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.139 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.139 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:26:12.139 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.357801+0000", 2024-08-19T14:26:12.140 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0026251920000000002 2024-08-19T14:26:12.140 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.140 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.140 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:26:12.140 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.357813+0000", 2024-08-19T14:26:12.140 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2734e-05 2024-08-19T14:26:12.140 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.140 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.140 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:26:12.140 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.682208+0000", 2024-08-19T14:26:12.140 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.324394395 2024-08-19T14:26:12.140 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.141 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.141 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:26:12.141 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.682268+0000", 2024-08-19T14:26:12.141 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.0228e-05 2024-08-19T14:26:12.141 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.141 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.141 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:26:12.141 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.682309+0000", 2024-08-19T14:26:12.141 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.1505000000000003e-05 2024-08-19T14:26:12.141 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:26:12.141 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:26:12.141 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:26:12.141 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.142 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.142 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25333.0:22709 4.7 4:e2a7b456:::benchmark_data_smithi045_71562_object22708:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-08-19T14:26:12.142 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:20:41.355501+0000", 2024-08-19T14:26:12.142 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 329.78407972600002, 2024-08-19T14:26:12.142 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.32904467700000001, 2024-08-19T14:26:12.142 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:26:12.142 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:26:12.142 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:26:12.142 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:26:12.142 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25333", 2024-08-19T14:26:12.142 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:26:12.142 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 22709 2024-08-19T14:26:12.142 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.142 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:26:12.143 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.143 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:26:12.143 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.355501+0000", 2024-08-19T14:26:12.143 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:26:12.143 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.143 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.143 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:26:12.143 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.355501+0000", 2024-08-19T14:26:12.143 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:26:12.143 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.143 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.143 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:26:12.143 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.355501+0000", 2024-08-19T14:26:12.144 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1099999999999996e-07 2024-08-19T14:26:12.144 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.144 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.144 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:26:12.144 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.355554+0000", 2024-08-19T14:26:12.144 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.2317000000000001e-05 2024-08-19T14:26:12.144 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.144 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.144 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:26:12.144 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.355561+0000", 2024-08-19T14:26:12.144 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.6190000000000002e-06 2024-08-19T14:26:12.144 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.144 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.144 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:26:12.145 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.355567+0000", 2024-08-19T14:26:12.145 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.3090000000000003e-06 2024-08-19T14:26:12.145 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.145 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.145 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:26:12.145 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.356458+0000", 2024-08-19T14:26:12.145 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00089112600000000001 2024-08-19T14:26:12.145 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.145 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.145 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:26:12.145 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.356702+0000", 2024-08-19T14:26:12.145 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00024424000000000001 2024-08-19T14:26:12.145 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.145 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.146 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,7", 2024-08-19T14:26:12.146 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.356835+0000", 2024-08-19T14:26:12.146 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000132571 2024-08-19T14:26:12.146 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.146 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.146 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:26:12.146 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.358857+0000", 2024-08-19T14:26:12.146 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0020221280000000002 2024-08-19T14:26:12.146 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.146 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.146 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:26:12.146 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.358903+0000", 2024-08-19T14:26:12.146 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.6088999999999999e-05 2024-08-19T14:26:12.147 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.147 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.147 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:26:12.147 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.684455+0000", 2024-08-19T14:26:12.147 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.32555214500000001 2024-08-19T14:26:12.147 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.147 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.147 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:26:12.147 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.684505+0000", 2024-08-19T14:26:12.147 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.9835e-05 2024-08-19T14:26:12.147 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.147 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.147 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:26:12.147 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.684545+0000", 2024-08-19T14:26:12.148 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.0787000000000001e-05 2024-08-19T14:26:12.148 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:26:12.148 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:26:12.148 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:26:12.148 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.148 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.148 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25333.0:22715 4.7 4:ee462d5b:::benchmark_data_smithi045_71562_object22714:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-08-19T14:26:12.148 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:20:41.357321+0000", 2024-08-19T14:26:12.148 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 329.78225910499998, 2024-08-19T14:26:12.148 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.32736539599999998, 2024-08-19T14:26:12.148 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:26:12.148 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:26:12.148 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:26:12.148 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:26:12.149 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25333", 2024-08-19T14:26:12.149 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:26:12.149 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 22715 2024-08-19T14:26:12.149 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.149 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:26:12.149 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.149 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:26:12.149 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.357321+0000", 2024-08-19T14:26:12.149 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:26:12.149 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.149 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.149 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:26:12.149 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.357321+0000", 2024-08-19T14:26:12.149 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:26:12.150 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.150 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.150 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:26:12.150 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.357322+0000", 2024-08-19T14:26:12.150 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.3099999999999996e-07 2024-08-19T14:26:12.150 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.150 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.150 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:26:12.150 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.357399+0000", 2024-08-19T14:26:12.150 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.6248000000000006e-05 2024-08-19T14:26:12.150 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.150 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.150 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:26:12.151 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.357412+0000", 2024-08-19T14:26:12.151 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3199e-05 2024-08-19T14:26:12.151 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.151 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.151 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:26:12.151 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.357419+0000", 2024-08-19T14:26:12.151 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.7429999999999996e-06 2024-08-19T14:26:12.151 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.151 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.151 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:26:12.151 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.357883+0000", 2024-08-19T14:26:12.151 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00046407799999999998 2024-08-19T14:26:12.151 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.151 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.152 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:26:12.152 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.358001+0000", 2024-08-19T14:26:12.152 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000118293 2024-08-19T14:26:12.152 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.152 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.152 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,7", 2024-08-19T14:26:12.152 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.358145+0000", 2024-08-19T14:26:12.152 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00014417400000000001 2024-08-19T14:26:12.152 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.152 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.152 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:26:12.152 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.359078+0000", 2024-08-19T14:26:12.152 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00093282800000000002 2024-08-19T14:26:12.152 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.153 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.153 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:26:12.153 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.359246+0000", 2024-08-19T14:26:12.153 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00016784699999999999 2024-08-19T14:26:12.153 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.153 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.153 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:26:12.153 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.684596+0000", 2024-08-19T14:26:12.153 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.32535034800000001 2024-08-19T14:26:12.153 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.153 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.153 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:26:12.153 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.684650+0000", 2024-08-19T14:26:12.153 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.3631999999999997e-05 2024-08-19T14:26:12.154 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.154 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.154 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:26:12.154 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.684687+0000", 2024-08-19T14:26:12.154 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7075e-05 2024-08-19T14:26:12.154 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:26:12.154 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:26:12.154 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:26:12.154 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.154 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.154 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25333.0:22726 4.7 4:eca380cd:::benchmark_data_smithi045_71562_object22725:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-08-19T14:26:12.154 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:20:41.361609+0000", 2024-08-19T14:26:12.154 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 329.777971536, 2024-08-19T14:26:12.154 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.32322025399999998, 2024-08-19T14:26:12.155 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:26:12.155 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:26:12.155 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:26:12.155 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:26:12.155 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25333", 2024-08-19T14:26:12.155 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:26:12.155 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 22726 2024-08-19T14:26:12.155 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.155 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:26:12.155 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.155 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:26:12.155 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.361609+0000", 2024-08-19T14:26:12.155 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:26:12.156 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.156 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.156 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:26:12.156 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.361609+0000", 2024-08-19T14:26:12.156 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:26:12.156 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.156 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.156 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:26:12.156 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.361610+0000", 2024-08-19T14:26:12.156 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1060000000000001e-06 2024-08-19T14:26:12.156 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.156 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.156 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:26:12.156 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.361662+0000", 2024-08-19T14:26:12.156 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1965000000000001e-05 2024-08-19T14:26:12.157 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.157 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.157 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:26:12.157 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.361673+0000", 2024-08-19T14:26:12.157 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0577999999999999e-05 2024-08-19T14:26:12.157 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.157 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.157 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:26:12.157 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.361679+0000", 2024-08-19T14:26:12.157 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.2809999999999997e-06 2024-08-19T14:26:12.157 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.157 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.157 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:26:12.158 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.361734+0000", 2024-08-19T14:26:12.158 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.5056999999999998e-05 2024-08-19T14:26:12.158 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.158 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.158 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:26:12.158 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.361872+0000", 2024-08-19T14:26:12.158 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013795999999999999 2024-08-19T14:26:12.158 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.158 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.158 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,7", 2024-08-19T14:26:12.158 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.361987+0000", 2024-08-19T14:26:12.158 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000114725 2024-08-19T14:26:12.158 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.159 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.159 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:26:12.159 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.363128+0000", 2024-08-19T14:26:12.159 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.001141818 2024-08-19T14:26:12.159 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.159 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.159 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:26:12.159 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.363443+0000", 2024-08-19T14:26:12.159 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00031427400000000002 2024-08-19T14:26:12.159 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.159 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.159 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:26:12.159 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.684736+0000", 2024-08-19T14:26:12.159 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.32129368000000003 2024-08-19T14:26:12.160 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.160 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.160 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:26:12.160 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.684791+0000", 2024-08-19T14:26:12.160 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.4759000000000002e-05 2024-08-19T14:26:12.160 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.160 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.160 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:26:12.160 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.684829+0000", 2024-08-19T14:26:12.160 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.8050999999999999e-05 2024-08-19T14:26:12.160 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:26:12.160 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:26:12.160 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:26:12.160 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.161 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.161 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25333.0:22730 4.7 4:e3e08cfc:::benchmark_data_smithi045_71562_object22729:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-08-19T14:26:12.161 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:20:41.363527+0000", 2024-08-19T14:26:12.161 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 329.77605332100001, 2024-08-19T14:26:12.161 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.32152788399999999, 2024-08-19T14:26:12.161 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:26:12.161 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:26:12.161 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:26:12.161 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:26:12.161 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25333", 2024-08-19T14:26:12.161 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:26:12.161 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 22730 2024-08-19T14:26:12.161 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.161 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:26:12.161 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.162 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:26:12.162 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.363527+0000", 2024-08-19T14:26:12.162 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:26:12.162 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.162 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.162 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:26:12.162 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.363527+0000", 2024-08-19T14:26:12.162 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:26:12.162 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.162 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.162 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:26:12.163 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.363528+0000", 2024-08-19T14:26:12.163 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.2699999999999999e-07 2024-08-19T14:26:12.163 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.163 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.163 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:26:12.163 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.363625+0000", 2024-08-19T14:26:12.163 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.6584999999999998e-05 2024-08-19T14:26:12.163 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.163 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.163 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:26:12.163 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.363636+0000", 2024-08-19T14:26:12.163 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1361e-05 2024-08-19T14:26:12.163 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.164 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.164 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:26:12.164 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.363641+0000", 2024-08-19T14:26:12.164 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.8389999999999997e-06 2024-08-19T14:26:12.164 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.164 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.164 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:26:12.164 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.363681+0000", 2024-08-19T14:26:12.164 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.0339999999999997e-05 2024-08-19T14:26:12.164 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.164 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.164 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:26:12.164 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.363804+0000", 2024-08-19T14:26:12.164 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012319800000000001 2024-08-19T14:26:12.164 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.165 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.165 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,7", 2024-08-19T14:26:12.165 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.363954+0000", 2024-08-19T14:26:12.165 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000149309 2024-08-19T14:26:12.165 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.165 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.165 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:26:12.165 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.364732+0000", 2024-08-19T14:26:12.165 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00077795000000000002 2024-08-19T14:26:12.165 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.165 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.165 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:26:12.165 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.364924+0000", 2024-08-19T14:26:12.165 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00019224699999999999 2024-08-19T14:26:12.166 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.166 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.166 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:26:12.166 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.684963+0000", 2024-08-19T14:26:12.166 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.32003914700000002 2024-08-19T14:26:12.166 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.166 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.166 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:26:12.166 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.685016+0000", 2024-08-19T14:26:12.166 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.2636000000000002e-05 2024-08-19T14:26:12.166 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.166 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.166 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:26:12.167 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.685055+0000", 2024-08-19T14:26:12.167 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.9545000000000002e-05 2024-08-19T14:26:12.167 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:26:12.167 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:26:12.167 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:26:12.167 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.167 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.167 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25333.0:166832 4.7 4:e935784b:::benchmark_data_smithi045_71562_object35657:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-08-19T14:26:12.167 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:22:44.337332+0000", 2024-08-19T14:26:12.167 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 206.80224890400001, 2024-08-19T14:26:12.167 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.25423578899999999, 2024-08-19T14:26:12.167 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:26:12.167 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:26:12.167 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:26:12.168 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:26:12.168 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25333", 2024-08-19T14:26:12.168 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:26:12.168 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 166832 2024-08-19T14:26:12.168 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.168 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:26:12.168 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.168 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:26:12.168 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:44.337332+0000", 2024-08-19T14:26:12.168 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:26:12.168 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.168 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.168 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:26:12.169 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:44.337332+0000", 2024-08-19T14:26:12.169 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:26:12.169 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.169 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.169 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:26:12.169 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:44.337332+0000", 2024-08-19T14:26:12.169 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.0999999999999998e-07 2024-08-19T14:26:12.169 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.169 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.169 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:26:12.169 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:44.337336+0000", 2024-08-19T14:26:12.169 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7409999999999998e-06 2024-08-19T14:26:12.169 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.169 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.170 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:26:12.170 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:44.337345+0000", 2024-08-19T14:26:12.170 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.7860000000000002e-06 2024-08-19T14:26:12.170 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.170 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.170 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:26:12.170 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:44.337351+0000", 2024-08-19T14:26:12.170 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.4740000000000002e-06 2024-08-19T14:26:12.170 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.170 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.170 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:26:12.170 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:44.363870+0000", 2024-08-19T14:26:12.170 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.026518832999999999 2024-08-19T14:26:12.171 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.171 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.171 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:26:12.171 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:44.363985+0000", 2024-08-19T14:26:12.171 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000115238 2024-08-19T14:26:12.171 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.171 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.171 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,7", 2024-08-19T14:26:12.171 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:44.364099+0000", 2024-08-19T14:26:12.171 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000113711 2024-08-19T14:26:12.171 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.171 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.171 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:26:12.172 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:44.381736+0000", 2024-08-19T14:26:12.172 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.017637262000000001 2024-08-19T14:26:12.172 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.172 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.172 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:26:12.172 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:44.381800+0000", 2024-08-19T14:26:12.172 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.3183999999999997e-05 2024-08-19T14:26:12.172 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.172 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.172 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:26:12.172 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:44.590743+0000", 2024-08-19T14:26:12.172 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.20894368499999999 2024-08-19T14:26:12.172 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.172 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.173 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:26:12.173 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:44.591515+0000", 2024-08-19T14:26:12.173 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00077177099999999998 2024-08-19T14:26:12.173 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.173 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.173 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:26:12.173 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:44.591567+0000", 2024-08-19T14:26:12.173 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.2394e-05 2024-08-19T14:26:12.173 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:26:12.173 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:26:12.173 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:26:12.173 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.173 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.173 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25333.0:166846 4.7 4:ed0ac338:::benchmark_data_smithi045_71562_object35671:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-08-19T14:26:12.174 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:22:44.354480+0000", 2024-08-19T14:26:12.174 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 206.78510028299999, 2024-08-19T14:26:12.174 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.23725674399999999, 2024-08-19T14:26:12.174 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:26:12.174 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:26:12.174 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:26:12.174 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:26:12.174 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25333", 2024-08-19T14:26:12.174 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:26:12.174 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 166846 2024-08-19T14:26:12.174 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.174 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:26:12.174 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.174 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:26:12.175 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:44.354480+0000", 2024-08-19T14:26:12.175 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:26:12.175 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.175 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.175 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:26:12.175 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:44.354480+0000", 2024-08-19T14:26:12.175 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:26:12.175 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.175 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.175 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:26:12.175 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:44.354481+0000", 2024-08-19T14:26:12.175 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.201e-06 2024-08-19T14:26:12.175 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.175 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.176 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:26:12.176 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:44.354488+0000", 2024-08-19T14:26:12.176 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.049e-06 2024-08-19T14:26:12.176 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.176 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.176 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:26:12.176 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:44.354504+0000", 2024-08-19T14:26:12.176 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6268000000000001e-05 2024-08-19T14:26:12.176 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.176 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.176 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:26:12.176 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:44.354511+0000", 2024-08-19T14:26:12.176 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.4599999999999997e-06 2024-08-19T14:26:12.176 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.177 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.177 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:26:12.177 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:44.369019+0000", 2024-08-19T14:26:12.177 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.014507631 2024-08-19T14:26:12.177 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.177 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.177 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:26:12.177 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:44.369503+0000", 2024-08-19T14:26:12.177 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000483773 2024-08-19T14:26:12.177 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.177 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.177 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,7", 2024-08-19T14:26:12.177 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:44.369832+0000", 2024-08-19T14:26:12.177 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00032930299999999997 2024-08-19T14:26:12.177 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.178 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.178 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:26:12.178 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:44.381855+0000", 2024-08-19T14:26:12.178 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.012023417999999999 2024-08-19T14:26:12.178 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.178 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.178 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:26:12.178 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:44.381880+0000", 2024-08-19T14:26:12.178 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.4318999999999999e-05 2024-08-19T14:26:12.178 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.178 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.178 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:26:12.178 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:44.591617+0000", 2024-08-19T14:26:12.179 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.20973735900000001 2024-08-19T14:26:12.179 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.179 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.179 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:26:12.179 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:44.591700+0000", 2024-08-19T14:26:12.179 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.3028000000000005e-05 2024-08-19T14:26:12.179 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.179 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.179 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:26:12.179 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:44.591737+0000", 2024-08-19T14:26:12.179 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.6934999999999998e-05 2024-08-19T14:26:12.179 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:26:12.179 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:26:12.179 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:26:12.180 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.180 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.180 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25333.0:166852 4.7 4:e69d85db:::benchmark_data_smithi045_71562_object35677:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-08-19T14:26:12.180 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:22:44.362087+0000", 2024-08-19T14:26:12.180 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 206.77749355399999, 2024-08-19T14:26:12.180 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.229806604, 2024-08-19T14:26:12.180 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:26:12.180 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:26:12.180 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:26:12.180 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:26:12.180 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25333", 2024-08-19T14:26:12.180 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:26:12.180 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 166852 2024-08-19T14:26:12.180 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.181 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:26:12.181 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.181 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:26:12.181 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:44.362087+0000", 2024-08-19T14:26:12.181 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:26:12.181 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.181 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.181 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:26:12.181 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:44.362087+0000", 2024-08-19T14:26:12.181 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:26:12.181 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.181 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.181 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:26:12.181 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:44.362089+0000", 2024-08-19T14:26:12.181 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6029999999999999e-06 2024-08-19T14:26:12.182 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.182 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.182 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:26:12.182 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:44.362095+0000", 2024-08-19T14:26:12.182 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.976e-06 2024-08-19T14:26:12.182 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.182 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.182 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:26:12.182 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:44.362109+0000", 2024-08-19T14:26:12.182 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4089e-05 2024-08-19T14:26:12.182 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.182 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.182 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:26:12.182 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:44.362116+0000", 2024-08-19T14:26:12.183 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.4050000000000003e-06 2024-08-19T14:26:12.183 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.183 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.183 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:26:12.183 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:44.375909+0000", 2024-08-19T14:26:12.183 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.013793041000000001 2024-08-19T14:26:12.183 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.183 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.183 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:26:12.183 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:44.376042+0000", 2024-08-19T14:26:12.183 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000132927 2024-08-19T14:26:12.183 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.183 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.183 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,7", 2024-08-19T14:26:12.184 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:44.376150+0000", 2024-08-19T14:26:12.184 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000107854 2024-08-19T14:26:12.184 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.184 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.184 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:26:12.184 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:44.381915+0000", 2024-08-19T14:26:12.184 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0057653469999999997 2024-08-19T14:26:12.184 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.184 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.184 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:26:12.184 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:44.381933+0000", 2024-08-19T14:26:12.184 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.7753000000000001e-05 2024-08-19T14:26:12.184 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.184 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.185 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:26:12.185 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:44.591778+0000", 2024-08-19T14:26:12.185 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.20984497599999999 2024-08-19T14:26:12.185 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.185 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.185 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:26:12.185 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:44.591851+0000", 2024-08-19T14:26:12.185 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.3334e-05 2024-08-19T14:26:12.185 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:26:12.185 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:26:12.185 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:26:12.185 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:44.591894+0000", 2024-08-19T14:26:12.185 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2299000000000003e-05 2024-08-19T14:26:12.185 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:26:12.186 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:26:12.186 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:26:12.186 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:26:12.186 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:26:12.186 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:26:12.186 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:26:12.214 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:12.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:12.215 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:26:12.316 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:26:12.326 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 84 16 115561 115545 85.9603 64.3125 0.00357136 0.0116278 2024-08-19T14:26:12.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:12.366 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:12.366 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:26:12.467 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:26:12.507 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:12.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:12.508 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:26:12.609 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:26:12.643 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:12.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:12.644 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:26:12.745 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:26:12.771 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:12.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:12.772 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:26:12.873 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:26:12.904 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:12.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:12.906 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:26:13.006 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:26:13.019 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:26:13.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:13.029 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:13.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:26:13.130 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:26:13.158 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:13.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:13.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:26:13.260 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:26:13.293 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:13.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:13.294 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:26:13.315 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:13 smithi031 ceph-mon[16336]: pgmap v1044: 57 pgs: 8 active+recovering+undersized+remapped, 49 active+clean; 6.9 GiB data, 27 GiB used, 777 GiB / 805 GiB avail; 25 MiB/s wr, 397 op/s; 20871/340452 objects misplaced (6.130%); 38 MiB/s, 615 objects/s recovering 2024-08-19T14:26:13.326 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 85 16 116373 116357 85.5461 50.75 0.0122683 0.011677 2024-08-19T14:26:13.395 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:26:13.416 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:13.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:13.417 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:26:13.518 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:26:13.547 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:13.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:13.548 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:26:13.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:13 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:26:13.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:13 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:26:13.650 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:26:13.677 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:13.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:13.678 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:26:13.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:13 smithi032 ceph-mon[20004]: pgmap v1044: 57 pgs: 8 active+recovering+undersized+remapped, 49 active+clean; 6.9 GiB data, 27 GiB used, 777 GiB / 805 GiB avail; 25 MiB/s wr, 397 op/s; 20871/340452 objects misplaced (6.130%); 38 MiB/s, 615 objects/s recovering 2024-08-19T14:26:13.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:13 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:26:13.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:13 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:26:13.779 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:26:13.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:13 smithi045 ceph-mon[20143]: pgmap v1044: 57 pgs: 8 active+recovering+undersized+remapped, 49 active+clean; 6.9 GiB data, 27 GiB used, 777 GiB / 805 GiB avail; 25 MiB/s wr, 397 op/s; 20871/340452 objects misplaced (6.130%); 38 MiB/s, 615 objects/s recovering 2024-08-19T14:26:13.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:13 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:26:13.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:13 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:26:13.808 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:13.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:13.809 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:26:13.910 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:26:13.941 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:13.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:13.942 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:26:14.043 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:26:14.080 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:14.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:14.081 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:26:14.182 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:26:14.212 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:14.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:14.213 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:26:14.314 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:26:14.326 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 86 16 117420 117404 85.3122 65.4375 0.0100605 0.0117171 2024-08-19T14:26:14.347 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:14.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:14.348 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:26:14.449 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:26:14.474 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:14.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:14.475 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:26:14.576 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:26:14.606 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:14.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:14.608 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:26:14.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:14 smithi031 ceph-mon[16336]: osdmap e456: 12 total, 12 up, 9 in 2024-08-19T14:26:14.708 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:26:14.730 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:14.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:14.731 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:26:14.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:14 smithi032 ceph-mon[20004]: osdmap e456: 12 total, 12 up, 9 in 2024-08-19T14:26:14.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:14 smithi045 ceph-mon[20143]: osdmap e456: 12 total, 12 up, 9 in 2024-08-19T14:26:14.832 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:26:14.864 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:14.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:14.865 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:26:14.966 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:26:14.990 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:14.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:14.991 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:26:15.092 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:26:15.113 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:15.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:15.114 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:26:15.215 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:26:15.240 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:15.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:15.241 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:26:15.326 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 87 16 118132 118116 84.8431 44.5 0.00315738 0.0117367 2024-08-19T14:26:15.342 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:26:15.364 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:15.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:15.365 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:26:15.466 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:26:15.489 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:15.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:15.490 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:26:15.591 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:26:15.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:15 smithi031 ceph-mon[16336]: pgmap v1046: 57 pgs: 8 active+recovering+undersized+remapped, 49 active+clean; 7.0 GiB data, 28 GiB used, 777 GiB / 805 GiB avail; 39 MiB/s wr, 625 op/s; 19286/345579 objects misplaced (5.581%); 45 MiB/s, 726 objects/s recovering 2024-08-19T14:26:15.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:15 smithi031 ceph-mon[16336]: osdmap e457: 12 total, 12 up, 9 in 2024-08-19T14:26:15.619 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:15.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:15.620 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:26:15.721 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:26:15.745 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:15.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:15.746 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:26:15.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:15 smithi032 ceph-mon[20004]: pgmap v1046: 57 pgs: 8 active+recovering+undersized+remapped, 49 active+clean; 7.0 GiB data, 28 GiB used, 777 GiB / 805 GiB avail; 39 MiB/s wr, 625 op/s; 19286/345579 objects misplaced (5.581%); 45 MiB/s, 726 objects/s recovering 2024-08-19T14:26:15.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:15 smithi032 ceph-mon[20004]: osdmap e457: 12 total, 12 up, 9 in 2024-08-19T14:26:15.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:15 smithi045 ceph-mon[20143]: pgmap v1046: 57 pgs: 8 active+recovering+undersized+remapped, 49 active+clean; 7.0 GiB data, 28 GiB used, 777 GiB / 805 GiB avail; 39 MiB/s wr, 625 op/s; 19286/345579 objects misplaced (5.581%); 45 MiB/s, 726 objects/s recovering 2024-08-19T14:26:15.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:15 smithi045 ceph-mon[20143]: osdmap e457: 12 total, 12 up, 9 in 2024-08-19T14:26:15.847 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:26:15.881 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:15.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:15.882 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:26:15.983 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:26:16.005 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:16.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:16.006 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:26:16.107 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:26:16.130 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:16.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:16.131 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:26:16.232 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:26:16.254 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:16.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:16.255 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:26:16.326 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 88 16 118530 118514 84.1616 24.875 0.00495268 0.0117824 2024-08-19T14:26:16.356 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:26:16.377 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:16.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:16.378 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:26:16.479 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:26:16.505 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:16.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:16.506 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:26:16.607 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:26:16.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:16 smithi031 ceph-mon[16336]: osdmap e458: 12 total, 12 up, 9 in 2024-08-19T14:26:16.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:16 smithi031 ceph-mon[16336]: osdmap e459: 12 total, 12 up, 9 in 2024-08-19T14:26:16.629 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:16.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:16.630 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:26:16.731 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:26:16.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:16 smithi032 ceph-mon[20004]: osdmap e458: 12 total, 12 up, 9 in 2024-08-19T14:26:16.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:16 smithi032 ceph-mon[20004]: osdmap e459: 12 total, 12 up, 9 in 2024-08-19T14:26:16.765 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:16.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:16.766 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:26:16.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:16 smithi045 ceph-mon[20143]: osdmap e458: 12 total, 12 up, 9 in 2024-08-19T14:26:16.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:16 smithi045 ceph-mon[20143]: osdmap e459: 12 total, 12 up, 9 in 2024-08-19T14:26:16.867 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:26:16.897 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:16.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:16.898 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:26:16.999 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:26:17.026 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:17.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:17.027 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:26:17.128 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:26:17.158 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:17.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:17.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:26:17.260 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:26:17.282 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:17.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:17.283 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:26:17.326 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 89 16 119286 119270 83.7468 47.25 0.00530205 0.0119211 2024-08-19T14:26:17.383 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:26:17.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:17.414 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:17.414 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:26:17.515 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:26:17.536 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:17.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:17.537 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:26:17.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:17 smithi031 ceph-mon[16336]: pgmap v1049: 57 pgs: 1 active+undersized+remapped, 7 active+recovering+undersized+remapped, 49 active+clean; 7.1 GiB data, 28 GiB used, 777 GiB / 805 GiB avail; 50 MiB/s wr, 803 op/s; 17836/347355 objects misplaced (5.135%); 39 MiB/s, 618 objects/s recovering 2024-08-19T14:26:17.637 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:26:17.675 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:17.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:17.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:26:17.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:17 smithi032 ceph-mon[20004]: pgmap v1049: 57 pgs: 1 active+undersized+remapped, 7 active+recovering+undersized+remapped, 49 active+clean; 7.1 GiB data, 28 GiB used, 777 GiB / 805 GiB avail; 50 MiB/s wr, 803 op/s; 17836/347355 objects misplaced (5.135%); 39 MiB/s, 618 objects/s recovering 2024-08-19T14:26:17.777 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:26:17.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:17 smithi045 ceph-mon[20143]: pgmap v1049: 57 pgs: 1 active+undersized+remapped, 7 active+recovering+undersized+remapped, 49 active+clean; 7.1 GiB data, 28 GiB used, 777 GiB / 805 GiB avail; 50 MiB/s wr, 803 op/s; 17836/347355 objects misplaced (5.135%); 39 MiB/s, 618 objects/s recovering 2024-08-19T14:26:17.820 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:17.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:17.820 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:26:17.921 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:26:17.965 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:17.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:17.966 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:26:18.067 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:26:18.106 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:18.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:18.107 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:26:18.208 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:26:18.232 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:18.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:18.233 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:26:18.327 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Total time run: 90.0074 2024-08-19T14:26:18.327 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Total writes made: 120250 2024-08-19T14:26:18.327 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Write size: 65536 2024-08-19T14:26:18.327 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Object size: 65536 2024-08-19T14:26:18.327 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Bandwidth (MB/sec): 83.5001 2024-08-19T14:26:18.327 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Stddev Bandwidth: 57.7129 2024-08-19T14:26:18.327 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Max bandwidth (MB/sec): 294.875 2024-08-19T14:26:18.327 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Min bandwidth (MB/sec): 0 2024-08-19T14:26:18.327 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Average IOPS: 1336 2024-08-19T14:26:18.327 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Stddev IOPS: 923.418 2024-08-19T14:26:18.328 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Max IOPS: 4718 2024-08-19T14:26:18.328 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Min IOPS: 0 2024-08-19T14:26:18.328 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Average Latency(s): 0.0119722 2024-08-19T14:26:18.328 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Stddev Latency(s): 0.0760649 2024-08-19T14:26:18.328 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Max latency(s): 6.38635 2024-08-19T14:26:18.328 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Min latency(s): 0.00112215 2024-08-19T14:26:18.331 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Cleaning up (deleting benchmark objects) 2024-08-19T14:26:18.334 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:26:18.362 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:18.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:18.363 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:26:18.464 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:26:18.490 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:18.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:18.492 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:26:18.592 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:26:18.621 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:18.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:18.622 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:26:18.723 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:26:18.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:18.751 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:18.751 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:26:18.852 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:26:18.881 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:18.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:18.882 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:26:18.983 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:26:19.012 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:19.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:19.013 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:26:19.114 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:26:19.143 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:19.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:19.144 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:26:19.246 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:26:19.272 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:19.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:19.274 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:26:19.375 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:26:19.403 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:19.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:19.404 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:26:19.506 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:26:19.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:19.545 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:19.545 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:26:19.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:19 smithi031 ceph-mon[16336]: pgmap v1051: 57 pgs: 1 active+undersized+remapped, 7 active+recovering+undersized+remapped, 49 active+clean; 7.2 GiB data, 28 GiB used, 776 GiB / 805 GiB avail; 49 MiB/s wr, 791 op/s; 15067/354063 objects misplaced (4.255%); 38 MiB/s, 604 objects/s recovering 2024-08-19T14:26:19.646 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:26:19.670 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:19.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:19.671 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:26:19.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:19 smithi032 ceph-mon[20004]: pgmap v1051: 57 pgs: 1 active+undersized+remapped, 7 active+recovering+undersized+remapped, 49 active+clean; 7.2 GiB data, 28 GiB used, 776 GiB / 805 GiB avail; 49 MiB/s wr, 791 op/s; 15067/354063 objects misplaced (4.255%); 38 MiB/s, 604 objects/s recovering 2024-08-19T14:26:19.772 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:26:19.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:19 smithi045 ceph-mon[20143]: pgmap v1051: 57 pgs: 1 active+undersized+remapped, 7 active+recovering+undersized+remapped, 49 active+clean; 7.2 GiB data, 28 GiB used, 776 GiB / 805 GiB avail; 49 MiB/s wr, 791 op/s; 15067/354063 objects misplaced (4.255%); 38 MiB/s, 604 objects/s recovering 2024-08-19T14:26:19.804 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:19.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:19.805 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:26:19.907 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:26:19.939 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:19.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:19.940 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:26:20.041 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:26:20.083 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:20.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:20.084 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:26:20.185 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:26:20.214 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:20.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:20.215 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:26:20.316 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:26:20.351 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:20.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:20.352 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:26:20.453 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:26:20.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:20.491 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:20.491 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:26:20.593 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:26:20.623 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:20.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:20.624 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:26:20.725 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:26:20.754 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:20.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:20.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:26:20.856 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:26:20.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:20.890 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:20.890 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:26:20.994 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:26:21.031 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:21.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:21.032 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:26:21.133 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:26:21.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:21.171 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:21.172 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:26:21.273 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:26:21.320 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-08-19T14:26:21.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:21.322 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:21.322 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:26:21.423 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:26:21.444 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:21.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:21.445 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:26:21.546 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:26:21.569 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:21.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:21.571 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:26:21.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:21 smithi031 ceph-mon[16336]: pgmap v1052: 57 pgs: 1 active+undersized+remapped, 6 active+recovering+undersized+remapped, 50 active+clean; 7.3 GiB data, 29 GiB used, 776 GiB / 805 GiB avail; 42 MiB/s wr, 680 op/s; 14375/357318 objects misplaced (4.023%); 24 MiB/s, 391 objects/s recovering 2024-08-19T14:26:21.672 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:26:21.710 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:21.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:21.711 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:26:21.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:21 smithi032 ceph-mon[20004]: pgmap v1052: 57 pgs: 1 active+undersized+remapped, 6 active+recovering+undersized+remapped, 50 active+clean; 7.3 GiB data, 29 GiB used, 776 GiB / 805 GiB avail; 42 MiB/s wr, 680 op/s; 14375/357318 objects misplaced (4.023%); 24 MiB/s, 391 objects/s recovering 2024-08-19T14:26:21.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:21 smithi045 ceph-mon[20143]: pgmap v1052: 57 pgs: 1 active+undersized+remapped, 6 active+recovering+undersized+remapped, 50 active+clean; 7.3 GiB data, 29 GiB used, 776 GiB / 805 GiB avail; 42 MiB/s wr, 680 op/s; 14375/357318 objects misplaced (4.023%); 24 MiB/s, 391 objects/s recovering 2024-08-19T14:26:21.812 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:26:21.846 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:21.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:21.847 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:26:21.948 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:26:21.978 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:21.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:21.980 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:26:22.081 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:26:22.112 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:22.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:22.113 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:26:22.214 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:26:22.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:22.239 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:22.239 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:26:22.340 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:26:22.362 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:22.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:22.363 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:26:22.464 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:26:22.500 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:22.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:22.501 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:26:22.602 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:26:22.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:22.633 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:22.633 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:26:22.734 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:26:22.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:22 smithi032 ceph-mon[20004]: osdmap e460: 12 total, 12 up, 9 in 2024-08-19T14:26:22.767 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:22.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:22.768 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:26:22.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:22 smithi045 ceph-mon[20143]: osdmap e460: 12 total, 12 up, 9 in 2024-08-19T14:26:22.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:22 smithi031 ceph-mon[16336]: osdmap e460: 12 total, 12 up, 9 in 2024-08-19T14:26:22.868 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:26:22.894 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:22.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:22.895 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:26:22.996 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:26:23.026 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:23.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:23.027 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:26:23.128 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:26:23.162 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:23.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:23.163 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:26:23.264 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:26:23.305 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:23.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:23.306 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:26:23.408 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:26:23.442 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:23.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:23.446 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:26:23.504 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is set 2024-08-19T14:26:23.547 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:26:23.577 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:23.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:23.578 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:26:23.679 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:26:23.716 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:23.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:23.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:26:23.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:23 smithi032 ceph-mon[20004]: pgmap v1054: 57 pgs: 1 peering, 5 active+recovering+undersized+remapped, 51 active+clean; 7.2 GiB data, 29 GiB used, 776 GiB / 805 GiB avail; 154 B/s rd, 41 MiB/s wr, 984 op/s; 11279/353865 objects misplaced (3.187%); 30 MiB/s, 477 objects/s recovering 2024-08-19T14:26:23.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:23 smithi032 ceph-mon[20004]: osdmap e461: 12 total, 12 up, 9 in 2024-08-19T14:26:23.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:23 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2271120982' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:26:23.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:23 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:26:23.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:23 smithi045 ceph-mon[20143]: pgmap v1054: 57 pgs: 1 peering, 5 active+recovering+undersized+remapped, 51 active+clean; 7.2 GiB data, 29 GiB used, 776 GiB / 805 GiB avail; 154 B/s rd, 41 MiB/s wr, 984 op/s; 11279/353865 objects misplaced (3.187%); 30 MiB/s, 477 objects/s recovering 2024-08-19T14:26:23.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:23 smithi045 ceph-mon[20143]: osdmap e461: 12 total, 12 up, 9 in 2024-08-19T14:26:23.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:23 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2271120982' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:26:23.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:23 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:26:23.818 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:26:23.852 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:23.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:23.854 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:26:23.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:23 smithi031 ceph-mon[16336]: pgmap v1054: 57 pgs: 1 peering, 5 active+recovering+undersized+remapped, 51 active+clean; 7.2 GiB data, 29 GiB used, 776 GiB / 805 GiB avail; 154 B/s rd, 41 MiB/s wr, 984 op/s; 11279/353865 objects misplaced (3.187%); 30 MiB/s, 477 objects/s recovering 2024-08-19T14:26:23.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:23 smithi031 ceph-mon[16336]: osdmap e461: 12 total, 12 up, 9 in 2024-08-19T14:26:23.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:23 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2271120982' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:26:23.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:23 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:26:23.955 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:26:23.992 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:23.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:23.993 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:26:24.094 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:26:24.138 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:24.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:24.139 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:26:24.240 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:26:24.270 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:24.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:24.271 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:26:24.372 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:26:24.408 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:24.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:24.410 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:26:24.511 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:26:24.531 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:24.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:24.532 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:26:24.633 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:26:24.689 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:24.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:24.690 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:26:24.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:24 smithi032 ceph-mon[20004]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:26:24.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:24 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:26:24.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:24 smithi032 ceph-mon[20004]: osdmap e462: 12 total, 12 up, 9 in 2024-08-19T14:26:24.791 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:26:24.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:24 smithi045 ceph-mon[20143]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:26:24.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:24 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:26:24.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:24 smithi045 ceph-mon[20143]: osdmap e462: 12 total, 12 up, 9 in 2024-08-19T14:26:24.820 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:24.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:24.821 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:26:24.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:24 smithi031 ceph-mon[16336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:26:24.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:24 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:26:24.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:24 smithi031 ceph-mon[16336]: osdmap e462: 12 total, 12 up, 9 in 2024-08-19T14:26:24.922 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:26:24.953 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:24.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:24.954 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:26:25.058 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:26:25.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:25.094 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:25.095 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:26:25.196 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:26:25.225 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:25.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:25.226 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:26:25.327 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:26:25.354 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:25.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:25.355 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:26:25.456 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:26:25.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:25.502 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:25.502 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:26:25.603 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:26:25.651 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:25.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:25.655 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:26:25.696 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:25 smithi045 ceph-mon[20143]: pgmap v1057: 57 pgs: 1 peering, 5 active+recovering+undersized+remapped, 51 active+clean; 7.1 GiB data, 28 GiB used, 776 GiB / 805 GiB avail; 170 B/s rd, 23 MiB/s wr, 1.05k op/s; 10442/348279 objects misplaced (2.998%); 22 MiB/s, 344 objects/s recovering 2024-08-19T14:26:25.755 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:26:25.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:25 smithi032 ceph-mon[20004]: pgmap v1057: 57 pgs: 1 peering, 5 active+recovering+undersized+remapped, 51 active+clean; 7.1 GiB data, 28 GiB used, 776 GiB / 805 GiB avail; 170 B/s rd, 23 MiB/s wr, 1.05k op/s; 10442/348279 objects misplaced (2.998%); 22 MiB/s, 344 objects/s recovering 2024-08-19T14:26:25.785 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:25.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:25.786 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:26:25.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:25 smithi031 ceph-mon[16336]: pgmap v1057: 57 pgs: 1 peering, 5 active+recovering+undersized+remapped, 51 active+clean; 7.1 GiB data, 28 GiB used, 776 GiB / 805 GiB avail; 170 B/s rd, 23 MiB/s wr, 1.05k op/s; 10442/348279 objects misplaced (2.998%); 22 MiB/s, 344 objects/s recovering 2024-08-19T14:26:25.886 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:26:25.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:25.925 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:25.926 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:26:26.026 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:26:26.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:26.063 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:26.063 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:26:26.165 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:26:26.202 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:26.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:26.204 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:26:26.306 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:26:26.336 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:26.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:26.337 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:26:26.438 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:26:26.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:26.470 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:26.470 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:26:26.571 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:26:26.608 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:26.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:26.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:26:26.710 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:26:26.746 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:26.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:26.747 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:26:26.848 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:26:26.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:26.881 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:26.881 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:26:26.983 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:26:27.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:27.015 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:27.015 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:26:27.116 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:26:27.156 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:27.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:27.157 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:26:27.258 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:26:27.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:27.289 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:27.289 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:26:27.390 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:26:27.416 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:27.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:27.417 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:26:27.517 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:26:27.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:27.545 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:27.545 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:26:27.646 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:26:27.680 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:27.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:27.682 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:26:27.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:27 smithi032 ceph-mon[20004]: pgmap v1058: 57 pgs: 1 peering, 5 active+recovering+undersized+remapped, 51 active+clean; 7.0 GiB data, 28 GiB used, 777 GiB / 805 GiB avail; 170 B/s rd, 10 MiB/s wr, 952 op/s; 10442/346209 objects misplaced (3.016%); 19 MiB/s, 304 objects/s recovering 2024-08-19T14:26:27.783 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:26:27.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:27 smithi045 ceph-mon[20143]: pgmap v1058: 57 pgs: 1 peering, 5 active+recovering+undersized+remapped, 51 active+clean; 7.0 GiB data, 28 GiB used, 777 GiB / 805 GiB avail; 170 B/s rd, 10 MiB/s wr, 952 op/s; 10442/346209 objects misplaced (3.016%); 19 MiB/s, 304 objects/s recovering 2024-08-19T14:26:27.817 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:27.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:27.818 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:26:27.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:27 smithi031 ceph-mon[16336]: pgmap v1058: 57 pgs: 1 peering, 5 active+recovering+undersized+remapped, 51 active+clean; 7.0 GiB data, 28 GiB used, 777 GiB / 805 GiB avail; 170 B/s rd, 10 MiB/s wr, 952 op/s; 10442/346209 objects misplaced (3.016%); 19 MiB/s, 304 objects/s recovering 2024-08-19T14:26:27.919 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:26:27.940 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:27.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:27.941 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:26:28.042 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:26:28.083 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:28.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:28.084 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:26:28.185 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:26:28.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:28.223 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:28.224 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:26:28.325 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:26:28.360 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:28.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:28.361 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:26:28.462 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:26:28.489 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:28.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:28.490 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:26:28.591 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:26:28.627 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:28.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:28.628 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:26:28.729 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:26:28.761 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:28.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:28.762 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:26:28.863 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:26:28.886 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:28.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:28.887 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:26:28.988 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:26:29.018 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:29.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:29.019 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:26:29.119 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:26:29.152 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:29.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:29.153 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:26:29.255 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:26:29.284 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:29.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:29.285 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:26:29.386 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:26:29.408 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:29.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:29.409 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:26:29.510 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:26:29.538 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:29.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:29.539 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:26:29.640 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:26:29.671 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:29.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:29.672 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:26:29.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:29 smithi032 ceph-mon[20004]: pgmap v1059: 57 pgs: 5 active+recovering+undersized+remapped, 52 active+clean; 6.8 GiB data, 28 GiB used, 777 GiB / 805 GiB avail; 5.1 MiB/s wr, 1.20k op/s; 7904/336426 objects misplaced (2.349%); 29 MiB/s, 467 objects/s recovering 2024-08-19T14:26:29.773 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:26:29.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:29 smithi045 ceph-mon[20143]: pgmap v1059: 57 pgs: 5 active+recovering+undersized+remapped, 52 active+clean; 6.8 GiB data, 28 GiB used, 777 GiB / 805 GiB avail; 5.1 MiB/s wr, 1.20k op/s; 7904/336426 objects misplaced (2.349%); 29 MiB/s, 467 objects/s recovering 2024-08-19T14:26:29.803 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:29.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:29.805 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:26:29.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:29 smithi031 ceph-mon[16336]: pgmap v1059: 57 pgs: 5 active+recovering+undersized+remapped, 52 active+clean; 6.8 GiB data, 28 GiB used, 777 GiB / 805 GiB avail; 5.1 MiB/s wr, 1.20k op/s; 7904/336426 objects misplaced (2.349%); 29 MiB/s, 467 objects/s recovering 2024-08-19T14:26:29.906 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:26:29.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:29.940 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:29.940 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:26:30.041 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:26:30.070 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:30.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:30.071 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:26:30.173 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:26:30.201 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:30.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:30.203 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:26:30.304 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:26:30.332 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:30.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:30.333 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:26:30.434 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:26:30.460 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:30.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:30.461 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:26:30.562 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:26:30.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:30.599 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:30.599 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:26:30.690 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:30 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:26:30.690 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:30 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:26:30.691 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:30 smithi031 ceph-mon[16336]: osdmap e463: 12 total, 12 up, 9 in 2024-08-19T14:26:30.700 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:26:30.732 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:30.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:30.733 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:26:30.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:30 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:26:30.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:30 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:26:30.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:30 smithi032 ceph-mon[20004]: osdmap e463: 12 total, 12 up, 9 in 2024-08-19T14:26:30.765 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:26:30.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:30 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:26:30.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:30 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:26:30.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:30 smithi045 ceph-mon[20143]: osdmap e463: 12 total, 12 up, 9 in 2024-08-19T14:26:30.816 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:26:30.834 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:26:30.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:30.863 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:30.863 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:26:30.964 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:26:30.998 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:30.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:30.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:26:31.100 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:26:31.130 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:31.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:31.132 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:26:31.233 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:26:31.262 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:31.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:31.263 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:26:31.364 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:26:31.397 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:31.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:31.399 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:26:31.499 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:26:31.533 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:31.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:31.534 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:26:31.635 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:26:31.665 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:31.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:31.667 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:26:31.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:31 smithi032 ceph-mon[20004]: pgmap v1061: 57 pgs: 5 active+recovering+undersized+remapped, 52 active+clean; 6.7 GiB data, 27 GiB used, 777 GiB / 805 GiB avail; 471 KiB/s wr, 1.04k op/s; 6986/330528 objects misplaced (2.114%); 15 MiB/s, 243 objects/s recovering 2024-08-19T14:26:31.768 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:26:31.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:31 smithi045 ceph-mon[20143]: pgmap v1061: 57 pgs: 5 active+recovering+undersized+remapped, 52 active+clean; 6.7 GiB data, 27 GiB used, 777 GiB / 805 GiB avail; 471 KiB/s wr, 1.04k op/s; 6986/330528 objects misplaced (2.114%); 15 MiB/s, 243 objects/s recovering 2024-08-19T14:26:31.799 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:31.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:31.800 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:26:31.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:31 smithi031 ceph-mon[16336]: pgmap v1061: 57 pgs: 5 active+recovering+undersized+remapped, 52 active+clean; 6.7 GiB data, 27 GiB used, 777 GiB / 805 GiB avail; 471 KiB/s wr, 1.04k op/s; 6986/330528 objects misplaced (2.114%); 15 MiB/s, 243 objects/s recovering 2024-08-19T14:26:31.901 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:26:31.926 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:31.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:31.927 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:26:32.030 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:26:32.065 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:32.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:32.067 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:26:32.168 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:26:32.189 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:32.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:32.190 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:26:32.291 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:26:32.320 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:32.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:32.321 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:26:32.422 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:26:32.443 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:32.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:32.444 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:26:32.545 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:26:32.575 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:32.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:32.576 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:26:32.677 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:26:32.700 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:32.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:32.701 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:26:32.750 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-19T14:26:32.802 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:26:32.834 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:32.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:32.835 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:26:32.936 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:26:32.959 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:32.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:32.960 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:26:33.061 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:26:33.083 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:33.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:33.084 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:26:33.185 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:26:33.207 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:33.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:33.208 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:26:33.308 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:26:33.330 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:33.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:33.331 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:26:33.432 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:26:33.457 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:33.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:33.458 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:26:33.559 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:26:33.580 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:33.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:33.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:26:33.681 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:26:33.709 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:33.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:33.710 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:26:33.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:33 smithi032 ceph-mon[20004]: pgmap v1062: 57 pgs: 5 active+recovering+undersized+remapped, 52 active+clean; 6.6 GiB data, 27 GiB used, 778 GiB / 805 GiB avail; 0 B/s wr, 1.11k op/s; 6986/322881 objects misplaced (2.164%); 13 MiB/s, 215 objects/s recovering 2024-08-19T14:26:33.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:33 smithi032 ceph-mon[20004]: osdmap e464: 12 total, 12 up, 9 in 2024-08-19T14:26:33.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:33 smithi045 ceph-mon[20143]: pgmap v1062: 57 pgs: 5 active+recovering+undersized+remapped, 52 active+clean; 6.6 GiB data, 27 GiB used, 778 GiB / 805 GiB avail; 0 B/s wr, 1.11k op/s; 6986/322881 objects misplaced (2.164%); 13 MiB/s, 215 objects/s recovering 2024-08-19T14:26:33.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:33 smithi045 ceph-mon[20143]: osdmap e464: 12 total, 12 up, 9 in 2024-08-19T14:26:33.811 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:26:33.841 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:33.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:33.842 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:26:33.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:33 smithi031 ceph-mon[16336]: pgmap v1062: 57 pgs: 5 active+recovering+undersized+remapped, 52 active+clean; 6.6 GiB data, 27 GiB used, 778 GiB / 805 GiB avail; 0 B/s wr, 1.11k op/s; 6986/322881 objects misplaced (2.164%); 13 MiB/s, 215 objects/s recovering 2024-08-19T14:26:33.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:33 smithi031 ceph-mon[16336]: osdmap e464: 12 total, 12 up, 9 in 2024-08-19T14:26:33.943 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:26:33.967 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:33.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:33.968 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:26:34.070 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:26:34.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:34.093 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:34.093 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:26:34.194 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:26:34.215 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:34.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:34.216 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:26:34.317 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:26:34.339 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:34.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:34.340 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:26:34.441 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:26:34.474 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:34.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:34.475 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:26:34.576 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:26:34.597 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:34.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:34.598 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:26:34.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:34 smithi031 ceph-mon[16336]: osdmap e465: 12 total, 12 up, 9 in 2024-08-19T14:26:34.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:34 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:26:34.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:34 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:26:34.699 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:26:34.723 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:34.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:34.724 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:26:34.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:34 smithi032 ceph-mon[20004]: osdmap e465: 12 total, 12 up, 9 in 2024-08-19T14:26:34.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:34 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:26:34.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:34 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:26:34.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:34 smithi045 ceph-mon[20143]: osdmap e465: 12 total, 12 up, 9 in 2024-08-19T14:26:34.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:34 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:26:34.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:34 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:26:34.825 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:26:34.849 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:34.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:34.849 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:26:34.950 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:26:34.973 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:34.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:34.976 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:26:35.077 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:26:35.099 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:35.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:35.100 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:26:35.201 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:26:35.223 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:35.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:35.224 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:26:35.326 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:26:35.348 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:35.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:35.349 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:26:35.450 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:26:35.478 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:35.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:35.479 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:26:35.511 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:26:35.580 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:26:35.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:35.602 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:35.602 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:26:35.704 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:26:35.724 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:35.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:35.726 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:26:35.746 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:26:35.746 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:35 smithi031 ceph-mon[16336]: pgmap v1065: 57 pgs: 1 creating+peering, 1 active+undersized+remapped, 3 active+recovering+undersized+remapped, 52 active+clean; 6.4 GiB data, 26 GiB used, 778 GiB / 805 GiB avail; 0 B/s wr, 1.34k op/s; 2383/312348 objects misplaced (0.763%); 25 MiB/s, 403 objects/s recovering 2024-08-19T14:26:35.746 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:35 smithi031 ceph-mon[16336]: osdmap e466: 12 total, 12 up, 9 in 2024-08-19T14:26:35.746 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:35 smithi031 ceph-mon[16336]: Health check failed: Degraded data redundancy: 4 pgs undersized (PG_DEGRADED) 2024-08-19T14:26:35.747 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:35 smithi031 ceph-mon[16336]: osdmap e467: 12 total, 12 up, 9 in 2024-08-19T14:26:35.747 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-08-19T14:26:35.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:35 smithi032 ceph-mon[20004]: pgmap v1065: 57 pgs: 1 creating+peering, 1 active+undersized+remapped, 3 active+recovering+undersized+remapped, 52 active+clean; 6.4 GiB data, 26 GiB used, 778 GiB / 805 GiB avail; 0 B/s wr, 1.34k op/s; 2383/312348 objects misplaced (0.763%); 25 MiB/s, 403 objects/s recovering 2024-08-19T14:26:35.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:35 smithi032 ceph-mon[20004]: osdmap e466: 12 total, 12 up, 9 in 2024-08-19T14:26:35.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:35 smithi032 ceph-mon[20004]: Health check failed: Degraded data redundancy: 4 pgs undersized (PG_DEGRADED) 2024-08-19T14:26:35.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:35 smithi032 ceph-mon[20004]: osdmap e467: 12 total, 12 up, 9 in 2024-08-19T14:26:35.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:35 smithi045 ceph-mon[20143]: pgmap v1065: 57 pgs: 1 creating+peering, 1 active+undersized+remapped, 3 active+recovering+undersized+remapped, 52 active+clean; 6.4 GiB data, 26 GiB used, 778 GiB / 805 GiB avail; 0 B/s wr, 1.34k op/s; 2383/312348 objects misplaced (0.763%); 25 MiB/s, 403 objects/s recovering 2024-08-19T14:26:35.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:35 smithi045 ceph-mon[20143]: osdmap e466: 12 total, 12 up, 9 in 2024-08-19T14:26:35.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:35 smithi045 ceph-mon[20143]: Health check failed: Degraded data redundancy: 4 pgs undersized (PG_DEGRADED) 2024-08-19T14:26:35.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:35 smithi045 ceph-mon[20143]: osdmap e467: 12 total, 12 up, 9 in 2024-08-19T14:26:35.827 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:26:35.848 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:35.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:35.849 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:26:35.950 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:26:35.971 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:35.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:35.972 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:26:36.073 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:26:36.096 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:36.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:36.097 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:26:36.198 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:26:36.221 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:36.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:36.222 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:26:36.323 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:26:36.348 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:36.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:36.348 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:26:36.450 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:26:36.481 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:36.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:36.482 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:26:36.583 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:26:36.608 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:36.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:36.610 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:26:36.711 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:26:36.716 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:36 smithi031 ceph-mon[16336]: from='client.26015 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:26:36.716 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:36 smithi031 ceph-mon[16336]: osdmap e468: 12 total, 12 up, 9 in 2024-08-19T14:26:36.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:36.750 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:36.750 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:26:36.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:36 smithi032 ceph-mon[20004]: from='client.26015 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:26:36.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:36 smithi032 ceph-mon[20004]: osdmap e468: 12 total, 12 up, 9 in 2024-08-19T14:26:36.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:36 smithi045 ceph-mon[20143]: from='client.26015 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:26:36.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:36 smithi045 ceph-mon[20143]: osdmap e468: 12 total, 12 up, 9 in 2024-08-19T14:26:36.852 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:26:36.878 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:36.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:36.879 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:26:36.980 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:26:37.000 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:37.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:37.001 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:26:37.102 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:26:37.134 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:37.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:37.135 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:26:37.236 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:26:37.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:37.270 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:37.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:26:37.371 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:26:37.397 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:37.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:37.398 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:26:37.498 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:26:37.528 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:37.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:37.529 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:26:37.630 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:26:37.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:37.663 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:37.663 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:26:37.738 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:37 smithi031 ceph-mon[16336]: pgmap v1068: 57 pgs: 1 creating+peering, 1 active+undersized+remapped, 3 active+recovering+undersized+remapped, 52 active+clean; 6.3 GiB data, 26 GiB used, 778 GiB / 805 GiB avail; 0 B/s wr, 1.00k op/s; 2383/310845 objects misplaced (0.767%); 33 MiB/s, 528 objects/s recovering 2024-08-19T14:26:37.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:37 smithi032 ceph-mon[20004]: pgmap v1068: 57 pgs: 1 creating+peering, 1 active+undersized+remapped, 3 active+recovering+undersized+remapped, 52 active+clean; 6.3 GiB data, 26 GiB used, 778 GiB / 805 GiB avail; 0 B/s wr, 1.00k op/s; 2383/310845 objects misplaced (0.767%); 33 MiB/s, 528 objects/s recovering 2024-08-19T14:26:37.764 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:26:37.793 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:37.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:37.794 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:26:37.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:37 smithi045 ceph-mon[20143]: pgmap v1068: 57 pgs: 1 creating+peering, 1 active+undersized+remapped, 3 active+recovering+undersized+remapped, 52 active+clean; 6.3 GiB data, 26 GiB used, 778 GiB / 805 GiB avail; 0 B/s wr, 1.00k op/s; 2383/310845 objects misplaced (0.767%); 33 MiB/s, 528 objects/s recovering 2024-08-19T14:26:37.895 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:26:37.929 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:37.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:37.930 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:26:38.031 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:26:38.054 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:38.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:38.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:26:38.157 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:26:38.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:38.186 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:38.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:26:38.288 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:26:38.320 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:38.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:38.321 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:26:38.422 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:26:38.456 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:38.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:38.457 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:26:38.558 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:26:38.587 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:38.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:38.588 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:26:38.690 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:26:38.725 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:38.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:38.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:26:38.828 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:26:38.860 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:38.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:38.861 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:26:38.962 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:26:38.996 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:38.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:38.997 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:26:39.098 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:26:39.133 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:39.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:39.134 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:26:39.235 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:26:39.264 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:39.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:39.265 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:26:39.366 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:26:39.396 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:39.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:39.397 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:26:39.498 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:26:39.526 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:39.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:39.527 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:26:39.609 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:26:39.609 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:26:39.609 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:26:39.610 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:26:39.610 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:26:39.610 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:26:39.610 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:26:39.610 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:26:39.610 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:26:39.610 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:26:39.610 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:26:39.610 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:26:39.610 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:26:39.611 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:26:39.611 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:26:39.611 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:26:39.611 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:26:39.611 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:26:39.611 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:26:39.611 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:26:39.611 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:26:39.611 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:26:39.611 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:26:39.611 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:26:39.628 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:26:39.652 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:39.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:39.653 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:26:39.754 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:26:39.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:39 smithi032 ceph-mon[20004]: pgmap v1070: 57 pgs: 1 creating+peering, 2 active+recovering+undersized+remapped, 54 active+clean; 6.2 GiB data, 26 GiB used, 779 GiB / 805 GiB avail; 0 B/s wr, 616 op/s; 1467/306996 objects misplaced (0.478%); 13 MiB/s, 200 objects/s recovering 2024-08-19T14:26:39.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:39 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/4045446065' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:26:39.775 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:39.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:39.776 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:26:39.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:39 smithi045 ceph-mon[20143]: pgmap v1070: 57 pgs: 1 creating+peering, 2 active+recovering+undersized+remapped, 54 active+clean; 6.2 GiB data, 26 GiB used, 779 GiB / 805 GiB avail; 0 B/s wr, 616 op/s; 1467/306996 objects misplaced (0.478%); 13 MiB/s, 200 objects/s recovering 2024-08-19T14:26:39.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:39 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/4045446065' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:26:39.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:39 smithi031 ceph-mon[16336]: pgmap v1070: 57 pgs: 1 creating+peering, 2 active+recovering+undersized+remapped, 54 active+clean; 6.2 GiB data, 26 GiB used, 779 GiB / 805 GiB avail; 0 B/s wr, 616 op/s; 1467/306996 objects misplaced (0.478%); 13 MiB/s, 200 objects/s recovering 2024-08-19T14:26:39.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:39 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/4045446065' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:26:39.877 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:26:39.913 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:39.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:39.914 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:26:40.015 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:26:40.041 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:40.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:40.042 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:26:40.143 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:26:40.165 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:40.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:40.166 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:26:40.267 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:26:40.293 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:40.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:40.294 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:26:40.395 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:26:40.417 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:40.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:40.418 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:26:40.519 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:26:40.540 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:40.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:40.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:26:40.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:40 smithi031 ceph-mon[16336]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2 pgs undersized) 2024-08-19T14:26:40.642 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:26:40.663 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:40.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:40.664 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:26:40.765 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:26:40.788 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:40.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:40.789 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:26:40.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:40 smithi045 ceph-mon[20143]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2 pgs undersized) 2024-08-19T14:26:40.890 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:26:40.919 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:40.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:40.921 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:26:41.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:40 smithi032 ceph-mon[20004]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2 pgs undersized) 2024-08-19T14:26:41.021 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:26:41.048 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:41.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:41.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:26:41.150 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:26:41.174 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:41.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:41.175 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:26:41.276 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:26:41.311 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:41.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:41.312 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:26:41.413 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:26:41.445 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:41.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:41.446 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:26:41.547 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:26:41.580 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:41.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:41.581 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:26:41.682 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:26:41.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:41.707 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:41.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:26:41.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:41 smithi045 ceph-mon[20143]: pgmap v1071: 57 pgs: 57 active+clean; 6.2 GiB data, 25 GiB used, 779 GiB / 805 GiB avail; 0 B/s wr, 539 op/s; 9.4 MiB/s, 150 objects/s recovering 2024-08-19T14:26:41.808 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:26:41.836 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:41.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:41.837 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:26:41.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:41 smithi031 ceph-mon[16336]: pgmap v1071: 57 pgs: 57 active+clean; 6.2 GiB data, 25 GiB used, 779 GiB / 805 GiB avail; 0 B/s wr, 539 op/s; 9.4 MiB/s, 150 objects/s recovering 2024-08-19T14:26:41.938 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:26:41.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:41.973 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:41.973 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:26:42.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:41 smithi032 ceph-mon[20004]: pgmap v1071: 57 pgs: 57 active+clean; 6.2 GiB data, 25 GiB used, 779 GiB / 805 GiB avail; 0 B/s wr, 539 op/s; 9.4 MiB/s, 150 objects/s recovering 2024-08-19T14:26:42.074 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:26:42.106 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:42.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:42.107 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:26:42.208 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:26:42.240 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:42.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:42.241 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:26:42.342 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:26:42.376 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:42.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:42.377 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:26:42.478 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:26:42.507 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:42.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:42.508 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:26:42.609 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:26:42.630 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:42.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:42.631 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:26:42.732 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:26:42.760 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:42.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:42.761 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:26:42.862 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:26:42.895 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:42.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:42.896 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:26:42.997 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:26:43.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:43.028 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:43.028 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:26:43.129 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:26:43.165 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:43.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:43.166 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:26:43.267 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:26:43.299 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:43.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:43.300 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:26:43.401 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:26:43.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:43.434 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:43.434 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:26:43.535 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:26:43.556 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:43.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:43.557 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:26:43.658 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:26:43.682 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:43.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:43.683 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:26:43.784 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:26:43.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:43 smithi045 ceph-mon[20143]: pgmap v1072: 57 pgs: 57 active+clean; 6.0 GiB data, 25 GiB used, 780 GiB / 805 GiB avail; 0 B/s wr, 775 op/s; 7.5 MiB/s, 119 objects/s recovering 2024-08-19T14:26:43.810 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:43.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:43.811 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:26:43.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:43 smithi031 ceph-mon[16336]: pgmap v1072: 57 pgs: 57 active+clean; 6.0 GiB data, 25 GiB used, 780 GiB / 805 GiB avail; 0 B/s wr, 775 op/s; 7.5 MiB/s, 119 objects/s recovering 2024-08-19T14:26:43.912 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:26:43.944 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:43.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:43.945 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:26:44.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:43 smithi032 ceph-mon[20004]: pgmap v1072: 57 pgs: 57 active+clean; 6.0 GiB data, 25 GiB used, 780 GiB / 805 GiB avail; 0 B/s wr, 775 op/s; 7.5 MiB/s, 119 objects/s recovering 2024-08-19T14:26:44.045 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:26:44.083 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:44.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:44.084 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:26:44.185 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:26:44.212 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:44.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:44.213 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:26:44.315 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:26:44.351 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:44.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:44.352 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:26:44.453 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:26:44.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:44.490 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:44.490 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:26:44.591 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:26:44.620 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:44.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:44.621 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:26:44.722 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:26:44.749 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:44.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:44.750 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:26:44.851 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:26:44.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:44.882 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:44.883 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:26:44.984 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:26:45.016 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:45.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:45.017 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:26:45.118 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:26:45.145 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:45.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:45.146 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:26:45.247 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:26:45.273 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:45.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:45.274 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:26:45.374 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:26:45.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:45.410 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:45.410 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:26:45.511 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:26:45.542 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:45.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:45.543 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:26:45.644 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:26:45.673 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:45.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:45.674 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:26:45.775 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:26:45.804 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:45.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:45.805 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:26:45.906 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:26:45.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:45.934 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:45.935 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:26:46.036 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:26:46.064 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:46.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:46.065 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:26:46.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:45 smithi031 ceph-mon[16336]: pgmap v1073: 57 pgs: 57 active+clean; 5.8 GiB data, 24 GiB used, 781 GiB / 805 GiB avail; 0 B/s wr, 1.04k op/s; 6.6 MiB/s, 105 objects/s recovering 2024-08-19T14:26:46.167 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:26:46.194 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:46.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:46.196 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:26:46.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:45 smithi032 ceph-mon[20004]: pgmap v1073: 57 pgs: 57 active+clean; 5.8 GiB data, 24 GiB used, 781 GiB / 805 GiB avail; 0 B/s wr, 1.04k op/s; 6.6 MiB/s, 105 objects/s recovering 2024-08-19T14:26:46.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:45 smithi045 ceph-mon[20143]: pgmap v1073: 57 pgs: 57 active+clean; 5.8 GiB data, 24 GiB used, 781 GiB / 805 GiB avail; 0 B/s wr, 1.04k op/s; 6.6 MiB/s, 105 objects/s recovering 2024-08-19T14:26:46.296 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:26:46.327 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:46.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:46.328 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:26:46.429 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:26:46.460 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:46.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:46.461 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:26:46.562 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:26:46.601 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:46.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:46.602 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:26:46.703 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:26:46.732 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:46.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:46.733 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:26:46.834 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:26:46.867 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:46.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:46.868 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:26:46.969 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:26:47.001 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:47.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:47.002 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:26:47.103 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:26:47.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:46 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:26:47.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:46 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:26:47.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:46 smithi031 ceph-mon[16336]: osdmap e469: 12 total, 12 up, 9 in 2024-08-19T14:26:47.130 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:47.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:47.131 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:26:47.232 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:26:47.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:46 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:26:47.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:46 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:26:47.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:46 smithi032 ceph-mon[20004]: osdmap e469: 12 total, 12 up, 9 in 2024-08-19T14:26:47.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:47.260 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:47.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:26:47.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:46 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:26:47.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:46 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:26:47.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:46 smithi045 ceph-mon[20143]: osdmap e469: 12 total, 12 up, 9 in 2024-08-19T14:26:47.361 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:26:47.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:47.396 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:47.396 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:26:47.497 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:26:47.525 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:47.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:47.526 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:26:47.627 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:26:47.656 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:47.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:47.657 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:26:47.758 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:26:47.788 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:47.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:47.789 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:26:47.890 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:26:47.920 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:47.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:47.921 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:26:48.022 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:26:48.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:48.056 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:48.056 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:26:48.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:47 smithi031 ceph-mon[16336]: pgmap v1075: 57 pgs: 57 active+clean; 5.7 GiB data, 24 GiB used, 781 GiB / 805 GiB avail; 0 B/s wr, 1.06k op/s; 5.9 MiB/s, 94 objects/s recovering 2024-08-19T14:26:48.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:47 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:26:48.156 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:26:48.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:48.187 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:48.187 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:26:48.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:47 smithi032 ceph-mon[20004]: pgmap v1075: 57 pgs: 57 active+clean; 5.7 GiB data, 24 GiB used, 781 GiB / 805 GiB avail; 0 B/s wr, 1.06k op/s; 5.9 MiB/s, 94 objects/s recovering 2024-08-19T14:26:48.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:47 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:26:48.288 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:26:48.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:47 smithi045 ceph-mon[20143]: pgmap v1075: 57 pgs: 57 active+clean; 5.7 GiB data, 24 GiB used, 781 GiB / 805 GiB avail; 0 B/s wr, 1.06k op/s; 5.9 MiB/s, 94 objects/s recovering 2024-08-19T14:26:48.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:47 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:26:48.319 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:48.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:48.320 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:26:48.421 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:26:48.453 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:48.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:48.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:26:48.555 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:26:48.584 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:48.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:48.585 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:26:48.686 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:26:48.723 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:48.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:48.724 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:26:48.730 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:26:48.825 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:26:48.857 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:48.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:48.858 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:26:48.958 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:26:48.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:48.989 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:48.989 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:26:49.091 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:26:49.116 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:49.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:49.117 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:26:49.218 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:26:49.251 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:49.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:49.252 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:26:49.353 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:26:49.379 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:49.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:49.380 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:26:49.481 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:26:49.514 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:49.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:49.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:26:49.616 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:26:49.650 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:49.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:49.651 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:26:49.752 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:26:49.779 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:49.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:49.780 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:26:49.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:49 smithi045 ceph-mon[20143]: pgmap v1076: 57 pgs: 57 active+clean; 5.5 GiB data, 23 GiB used, 782 GiB / 805 GiB avail; 0 B/s wr, 1.21k op/s; 2.8 MiB/s, 44 objects/s recovering 2024-08-19T14:26:49.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:49 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:26:49.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:49 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:26:49.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:49 smithi031 ceph-mon[16336]: pgmap v1076: 57 pgs: 57 active+clean; 5.5 GiB data, 23 GiB used, 782 GiB / 805 GiB avail; 0 B/s wr, 1.21k op/s; 2.8 MiB/s, 44 objects/s recovering 2024-08-19T14:26:49.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:49 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:26:49.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:49 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:26:49.881 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:26:49.910 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:49.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:49.911 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:26:50.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:49 smithi032 ceph-mon[20004]: pgmap v1076: 57 pgs: 57 active+clean; 5.5 GiB data, 23 GiB used, 782 GiB / 805 GiB avail; 0 B/s wr, 1.21k op/s; 2.8 MiB/s, 44 objects/s recovering 2024-08-19T14:26:50.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:49 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:26:50.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:49 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:26:50.012 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:26:50.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:50.044 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:50.044 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:26:50.145 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:26:50.175 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:50.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:50.176 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:26:50.277 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:26:50.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:50.307 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:50.307 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:26:50.408 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:26:50.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:50.442 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:50.443 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:26:50.544 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:26:50.578 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:50.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:50.580 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:26:50.680 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:26:50.710 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:50.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:50.711 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:26:50.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:50 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:26:50.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:50 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:26:50.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:50 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:26:50.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:50 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:26:50.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:50 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:26:50.812 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:26:50.833 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:50.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:50.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:26:50.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:50 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:26:50.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:50 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:26:50.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:50 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:26:50.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:50 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:26:50.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:50 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:26:50.935 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:26:50.964 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:50.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:50.965 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:26:51.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:50 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:26:51.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:50 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:26:51.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:50 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:26:51.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:50 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:26:51.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:50 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:26:51.066 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:26:51.095 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:51.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:51.096 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:26:51.197 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:26:51.228 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:51.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:51.229 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:26:51.330 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:26:51.365 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:51.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:51.366 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:26:51.467 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:26:51.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:51.502 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:51.502 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:26:51.603 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:26:51.630 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:51.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:51.631 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:26:51.732 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:26:51.762 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:51.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:51.763 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:26:51.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:51 smithi045 ceph-mon[20143]: pgmap v1077: 57 pgs: 57 active+clean; 5.3 GiB data, 22 GiB used, 782 GiB / 805 GiB avail; 0 B/s wr, 1.34k op/s 2024-08-19T14:26:51.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:51 smithi045 ceph-mon[20143]: osdmap e470: 12 total, 12 up, 9 in 2024-08-19T14:26:51.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:51 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:26:51.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:51 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:26:51.864 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:26:51.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:51 smithi031 ceph-mon[16336]: pgmap v1077: 57 pgs: 57 active+clean; 5.3 GiB data, 22 GiB used, 782 GiB / 805 GiB avail; 0 B/s wr, 1.34k op/s 2024-08-19T14:26:51.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:51 smithi031 ceph-mon[16336]: osdmap e470: 12 total, 12 up, 9 in 2024-08-19T14:26:51.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:51 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:26:51.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:51 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:26:51.896 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:51.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:51.898 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:26:51.908 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:51 smithi032 ceph-mon[20004]: pgmap v1077: 57 pgs: 57 active+clean; 5.3 GiB data, 22 GiB used, 782 GiB / 805 GiB avail; 0 B/s wr, 1.34k op/s 2024-08-19T14:26:51.908 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:51 smithi032 ceph-mon[20004]: osdmap e470: 12 total, 12 up, 9 in 2024-08-19T14:26:51.908 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:51 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:26:51.908 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:51 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:26:51.999 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:26:52.034 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:52.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:52.035 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:26:52.136 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:26:52.159 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:52.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:52.160 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:26:52.261 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:26:52.290 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:52.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:52.291 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:26:52.392 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:26:52.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:52.427 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:52.427 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:26:52.528 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:26:52.559 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:52.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:52.560 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:26:52.661 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:26:52.688 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:52.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:52.689 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:26:52.790 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:26:52.823 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:52.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:52.825 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:26:52.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:52 smithi031 ceph-mon[16336]: osdmap e471: 12 total, 12 up, 9 in 2024-08-19T14:26:52.905 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:26:52.925 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:26:52.947 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":1067,"stamp":"2024-08-19T14:26:35.439058+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6790840888,"num_objects":103615,"num_object_clones":0,"num_object_copies":310845,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2383,"num_objects_unfound":0,"num_objects_dirty":103615,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":114558,"num_write_kb":6270273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26914,"num_bytes_recovered":1763835904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":195570,"ondisk_log_size":195570,"up":171,"acting":167,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":183,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":27543384,"kb_used_data":22120496,"kb_used_omap":14,"kb_used_meta":5422769,"kb_avail":816162984,"statfs":{"total":863955320832,"available":835750895616,"internally_reserved":0,"allocated":22651387904,"data_stored":22632462056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14353,"internal_metadata":5552916463},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,2,4,4,5],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-429981696,"num_objects":-6561,"num_object_clones":0,"num_object_copies":-19683,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-4603,"num_objects_unfound":0,"num_objects_dirty":-6561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6561,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2115,"num_bytes_recovered":138608640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.455460"},"pg_stats":[{"pgid":"5.4","version":"463'6953","reported_seq":7095,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-08-19T14:26:31.836031+0000","last_change":"2024-08-19T14:25:31.443869+0000","last_active":"2024-08-19T14:26:31.836031+0000","last_peered":"2024-08-19T14:26:31.836031+0000","last_clean":"2024-08-19T14:26:31.836031+0000","last_became_active":"2024-08-19T14:25:29.845148+0000","last_became_peered":"2024-08-19T14:25:29.845148+0000","last_unstale":"2024-08-19T14:26:31.836031+0000","last_undegraded":"2024-08-19T14:26:31.836031+0000","last_fullsized":"2024-08-19T14:26:31.836031+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'5445","last_scrub_stamp":"2024-08-19T14:25:31.443758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:25:31.443758+0000","objects_scrubbed":2768,"log_size":6953,"log_dups_size":0,"ondisk_log_size":6953,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:36.438640+0000","scrub_duration":277,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":216006656,"num_objects":3296,"num_object_clones":0,"num_object_copies":9888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3296,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1508,"num_write_kb":65152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,0],"acting":[5,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"70'139","reported_seq":1164,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-08-19T14:26:33.442423+0000","last_change":"2024-08-19T14:24:48.191443+0000","last_active":"2024-08-19T14:26:33.442423+0000","last_peered":"2024-08-19T14:26:33.442423+0000","last_clean":"2024-08-19T14:26:33.442423+0000","last_became_active":"2024-08-19T14:18:13.807133+0000","last_became_peered":"2024-08-19T14:18:13.807133+0000","last_unstale":"2024-08-19T14:26:33.442423+0000","last_undegraded":"2024-08-19T14:26:33.442423+0000","last_fullsized":"2024-08-19T14:26:33.442423+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:48.191339+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:19:58.072083+0000","last_clean_scrub_stamp":"2024-08-19T14:24:48.191339+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,9],"acting":[7,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"463'7058","reported_seq":7200,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-08-19T14:26:31.811419+0000","last_change":"2024-08-19T14:25:32.405310+0000","last_active":"2024-08-19T14:26:31.811419+0000","last_peered":"2024-08-19T14:26:31.811419+0000","last_clean":"2024-08-19T14:26:31.811419+0000","last_became_active":"2024-08-19T14:25:29.845782+0000","last_became_peered":"2024-08-19T14:25:29.845782+0000","last_unstale":"2024-08-19T14:26:31.811419+0000","last_undegraded":"2024-08-19T14:26:31.811419+0000","last_fullsized":"2024-08-19T14:26:31.811419+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'5561","last_scrub_stamp":"2024-08-19T14:25:32.405201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:25:32.405201+0000","objects_scrubbed":2780,"log_size":7058,"log_dups_size":0,"ondisk_log_size":7058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:59.012193+0000","scrub_duration":268,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":212795392,"num_objects":3247,"num_object_clones":0,"num_object_copies":9741,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3247,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1497,"num_write_kb":62848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.3","version":"70'139","reported_seq":1063,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-08-19T14:26:33.442002+0000","last_change":"2024-08-19T14:24:42.697899+0000","last_active":"2024-08-19T14:26:33.442002+0000","last_peered":"2024-08-19T14:26:33.442002+0000","last_clean":"2024-08-19T14:26:33.442002+0000","last_became_active":"2024-08-19T14:18:13.804721+0000","last_became_peered":"2024-08-19T14:18:13.804721+0000","last_unstale":"2024-08-19T14:26:33.442002+0000","last_undegraded":"2024-08-19T14:26:33.442002+0000","last_fullsized":"2024-08-19T14:26:33.442002+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:42.697801+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:24:42.697801+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"463'7080","reported_seq":7187,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-08-19T14:26:31.841587+0000","last_change":"2024-08-19T14:25:29.831288+0000","last_active":"2024-08-19T14:26:31.841587+0000","last_peered":"2024-08-19T14:26:31.841587+0000","last_clean":"2024-08-19T14:26:31.841587+0000","last_became_active":"2024-08-19T14:25:29.824222+0000","last_became_peered":"2024-08-19T14:25:29.824222+0000","last_unstale":"2024-08-19T14:26:31.841587+0000","last_undegraded":"2024-08-19T14:26:31.841587+0000","last_fullsized":"2024-08-19T14:26:31.841587+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:24:43.574783+0000","objects_scrubbed":0,"log_size":7080,"log_dups_size":0,"ondisk_log_size":7080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":214171648,"num_objects":3268,"num_object_clones":0,"num_object_copies":9804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3268,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4298,"num_write_kb":242112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,8],"acting":[5,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"70'139","reported_seq":1011,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-08-19T14:26:33.440909+0000","last_change":"2024-08-19T14:24:43.246467+0000","last_active":"2024-08-19T14:26:33.440909+0000","last_peered":"2024-08-19T14:26:33.440909+0000","last_clean":"2024-08-19T14:26:33.440909+0000","last_became_active":"2024-08-19T14:20:50.742354+0000","last_became_peered":"2024-08-19T14:20:50.742354+0000","last_unstale":"2024-08-19T14:26:33.440909+0000","last_undegraded":"2024-08-19T14:26:33.440909+0000","last_fullsized":"2024-08-19T14:26:33.440909+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:43.246387+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:19:59.302931+0000","last_clean_scrub_stamp":"2024-08-19T14:24:43.246387+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.5","version":"463'6833","reported_seq":6974,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-08-19T14:26:31.968595+0000","last_change":"2024-08-19T14:25:31.371771+0000","last_active":"2024-08-19T14:26:31.968595+0000","last_peered":"2024-08-19T14:26:31.968595+0000","last_clean":"2024-08-19T14:26:31.968595+0000","last_became_active":"2024-08-19T14:25:29.826953+0000","last_became_peered":"2024-08-19T14:25:29.826953+0000","last_unstale":"2024-08-19T14:26:31.968595+0000","last_undegraded":"2024-08-19T14:26:31.968595+0000","last_fullsized":"2024-08-19T14:26:31.968595+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":0,"last_scrub":"440'5369","last_scrub_stamp":"2024-08-19T14:25:31.371653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:25:31.371653+0000","objects_scrubbed":2676,"log_size":6833,"log_dups_size":0,"ondisk_log_size":6833,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:52.679547+0000","scrub_duration":281,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":206569472,"num_objects":3152,"num_object_clones":0,"num_object_copies":9456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1464,"num_write_kb":62080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"70'139","reported_seq":1058,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-08-19T14:26:32.453193+0000","last_change":"2024-08-19T14:24:48.179646+0000","last_active":"2024-08-19T14:26:32.453193+0000","last_peered":"2024-08-19T14:26:32.453193+0000","last_clean":"2024-08-19T14:26:32.453193+0000","last_became_active":"2024-08-19T14:18:13.808505+0000","last_became_peered":"2024-08-19T14:18:13.808505+0000","last_unstale":"2024-08-19T14:26:32.453193+0000","last_undegraded":"2024-08-19T14:26:32.453193+0000","last_fullsized":"2024-08-19T14:26:32.453193+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:48.179558+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:24:48.179558+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,11],"acting":[9,0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.3","version":"463'6842","reported_seq":6983,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-08-19T14:26:31.939124+0000","last_change":"2024-08-19T14:25:32.404120+0000","last_active":"2024-08-19T14:26:31.939124+0000","last_peered":"2024-08-19T14:26:31.939124+0000","last_clean":"2024-08-19T14:26:31.939124+0000","last_became_active":"2024-08-19T14:25:29.851182+0000","last_became_peered":"2024-08-19T14:25:29.851182+0000","last_unstale":"2024-08-19T14:26:31.939124+0000","last_undegraded":"2024-08-19T14:26:31.939124+0000","last_fullsized":"2024-08-19T14:26:31.939124+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":0,"last_scrub":"440'5410","last_scrub_stamp":"2024-08-19T14:25:32.403944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:25:32.403944+0000","objects_scrubbed":2699,"log_size":6842,"log_dups_size":0,"ondisk_log_size":6842,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:38.949340+0000","scrub_duration":274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":211091456,"num_objects":3221,"num_object_clones":0,"num_object_copies":9663,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3221,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1432,"num_write_kb":62528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"70'139","reported_seq":711,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-08-19T14:26:33.436769+0000","last_change":"2024-08-19T14:24:47.412400+0000","last_active":"2024-08-19T14:26:33.436769+0000","last_peered":"2024-08-19T14:26:33.436769+0000","last_clean":"2024-08-19T14:26:33.436769+0000","last_became_active":"2024-08-19T14:20:28.204384+0000","last_became_peered":"2024-08-19T14:20:28.204384+0000","last_unstale":"2024-08-19T14:26:33.436769+0000","last_undegraded":"2024-08-19T14:26:33.436769+0000","last_fullsized":"2024-08-19T14:26:33.436769+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:47.412291+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:21:46.057624+0000","last_clean_scrub_stamp":"2024-08-19T14:24:47.412291+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"463'6920","reported_seq":7025,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-08-19T14:26:31.976818+0000","last_change":"2024-08-19T14:25:29.838717+0000","last_active":"2024-08-19T14:26:31.976818+0000","last_peered":"2024-08-19T14:26:31.976818+0000","last_clean":"2024-08-19T14:26:31.976818+0000","last_became_active":"2024-08-19T14:25:29.831300+0000","last_became_peered":"2024-08-19T14:25:29.831300+0000","last_unstale":"2024-08-19T14:26:31.976818+0000","last_undegraded":"2024-08-19T14:26:31.976818+0000","last_fullsized":"2024-08-19T14:26:31.976818+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:24:43.574783+0000","objects_scrubbed":0,"log_size":6920,"log_dups_size":0,"ondisk_log_size":6920,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:33.078994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213221376,"num_objects":3253,"num_object_clones":0,"num_object_copies":9759,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3253,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4213,"num_write_kb":238944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,8],"acting":[0,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"70'139","reported_seq":1062,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-08-19T14:26:32.453211+0000","last_change":"2024-08-19T14:24:46.222297+0000","last_active":"2024-08-19T14:26:32.453211+0000","last_peered":"2024-08-19T14:26:32.453211+0000","last_clean":"2024-08-19T14:26:32.453211+0000","last_became_active":"2024-08-19T14:18:26.874902+0000","last_became_peered":"2024-08-19T14:18:26.874902+0000","last_unstale":"2024-08-19T14:26:32.453211+0000","last_undegraded":"2024-08-19T14:26:32.453211+0000","last_fullsized":"2024-08-19T14:26:32.453211+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:46.222166+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:27.848870+0000","last_clean_scrub_stamp":"2024-08-19T14:24:46.222166+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,4],"acting":[9,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.1","version":"464'7142","reported_seq":7287,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-08-19T14:26:32.778590+0000","last_change":"2024-08-19T14:25:32.082493+0000","last_active":"2024-08-19T14:26:32.778590+0000","last_peered":"2024-08-19T14:26:32.778590+0000","last_clean":"2024-08-19T14:26:32.778590+0000","last_became_active":"2024-08-19T14:25:29.820527+0000","last_became_peered":"2024-08-19T14:25:29.820527+0000","last_unstale":"2024-08-19T14:26:32.778590+0000","last_undegraded":"2024-08-19T14:26:32.778590+0000","last_fullsized":"2024-08-19T14:26:32.778590+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'5651","last_scrub_stamp":"2024-08-19T14:25:32.082374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:25:32.082374+0000","objects_scrubbed":2854,"log_size":7142,"log_dups_size":0,"ondisk_log_size":7142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:40.027475+0000","scrub_duration":280,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":214761472,"num_objects":3277,"num_object_clones":0,"num_object_copies":9831,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3277,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1491,"num_write_kb":61248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"70'139","reported_seq":881,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-08-19T14:26:29.420023+0000","last_change":"2024-08-19T14:24:46.629045+0000","last_active":"2024-08-19T14:26:29.420023+0000","last_peered":"2024-08-19T14:26:29.420023+0000","last_clean":"2024-08-19T14:26:29.420023+0000","last_became_active":"2024-08-19T14:18:26.881435+0000","last_became_peered":"2024-08-19T14:18:26.881435+0000","last_unstale":"2024-08-19T14:26:29.420023+0000","last_undegraded":"2024-08-19T14:26:29.420023+0000","last_fullsized":"2024-08-19T14:26:29.420023+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:46.628960+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:24:46.628960+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,4],"acting":[11,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.0","version":"464'7161","reported_seq":7307,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-08-19T14:26:33.441475+0000","last_change":"2024-08-19T14:25:32.103509+0000","last_active":"2024-08-19T14:26:33.441475+0000","last_peered":"2024-08-19T14:26:33.441475+0000","last_clean":"2024-08-19T14:26:33.441475+0000","last_became_active":"2024-08-19T14:25:29.850301+0000","last_became_peered":"2024-08-19T14:25:29.850301+0000","last_unstale":"2024-08-19T14:26:33.441475+0000","last_undegraded":"2024-08-19T14:26:33.441475+0000","last_fullsized":"2024-08-19T14:26:33.441475+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'5592","last_scrub_stamp":"2024-08-19T14:25:32.103399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:25:32.103399+0000","objects_scrubbed":2856,"log_size":7161,"log_dups_size":0,"ondisk_log_size":7161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:59.127154+0000","scrub_duration":317,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":212402176,"num_objects":3241,"num_object_clones":0,"num_object_copies":9723,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1569,"num_write_kb":62528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"70'139","reported_seq":1106,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-08-19T14:26:29.419791+0000","last_change":"2024-08-19T14:24:11.345984+0000","last_active":"2024-08-19T14:26:29.419791+0000","last_peered":"2024-08-19T14:26:29.419791+0000","last_clean":"2024-08-19T14:26:29.419791+0000","last_became_active":"2024-08-19T14:18:13.806013+0000","last_became_peered":"2024-08-19T14:18:13.806013+0000","last_unstale":"2024-08-19T14:26:29.419791+0000","last_undegraded":"2024-08-19T14:26:29.419791+0000","last_fullsized":"2024-08-19T14:26:29.419791+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:11.345873+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:24:11.345873+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.e","version":"464'8548","reported_seq":8651,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-08-19T14:26:32.825436+0000","last_change":"2024-08-19T14:24:44.595518+0000","last_active":"2024-08-19T14:26:32.825436+0000","last_peered":"2024-08-19T14:26:32.825436+0000","last_clean":"2024-08-19T14:26:32.825436+0000","last_became_active":"2024-08-19T14:24:44.595166+0000","last_became_peered":"2024-08-19T14:24:44.595166+0000","last_unstale":"2024-08-19T14:26:32.825436+0000","last_undegraded":"2024-08-19T14:26:32.825436+0000","last_fullsized":"2024-08-19T14:26:32.825436+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:24:43.574783+0000","objects_scrubbed":0,"log_size":8548,"log_dups_size":0,"ondisk_log_size":8548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":427032576,"num_objects":6516,"num_object_clones":0,"num_object_copies":19548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6516,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8548,"num_write_kb":482048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,0],"acting":[9,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.a","version":"70'139","reported_seq":1070,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-08-19T14:26:32.453099+0000","last_change":"2024-08-19T14:24:50.196984+0000","last_active":"2024-08-19T14:26:32.453099+0000","last_peered":"2024-08-19T14:26:32.453099+0000","last_clean":"2024-08-19T14:26:32.453099+0000","last_became_active":"2024-08-19T14:18:26.875006+0000","last_became_peered":"2024-08-19T14:18:26.875006+0000","last_unstale":"2024-08-19T14:26:32.453099+0000","last_undegraded":"2024-08-19T14:26:32.453099+0000","last_fullsized":"2024-08-19T14:26:32.453099+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:50.196885+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:28.864870+0000","last_clean_scrub_stamp":"2024-08-19T14:24:50.196885+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,0],"acting":[9,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.d","version":"463'8596","reported_seq":8695,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-08-19T14:26:31.116760+0000","last_change":"2024-08-19T14:24:44.592424+0000","last_active":"2024-08-19T14:26:31.116760+0000","last_peered":"2024-08-19T14:26:31.116760+0000","last_clean":"2024-08-19T14:26:31.116760+0000","last_became_active":"2024-08-19T14:24:44.592098+0000","last_became_peered":"2024-08-19T14:24:44.592098+0000","last_unstale":"2024-08-19T14:26:31.116760+0000","last_undegraded":"2024-08-19T14:26:31.116760+0000","last_fullsized":"2024-08-19T14:26:31.116760+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:24:43.574783+0000","objects_scrubbed":0,"log_size":8596,"log_dups_size":0,"ondisk_log_size":8596,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":442040344,"num_objects":6746,"num_object_clones":0,"num_object_copies":20238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6746,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8596,"num_write_kb":490881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7],"acting":[11,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.9","version":"70'139","reported_seq":1187,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-08-19T14:26:33.440957+0000","last_change":"2024-08-19T14:25:04.127400+0000","last_active":"2024-08-19T14:26:33.440957+0000","last_peered":"2024-08-19T14:26:33.440957+0000","last_clean":"2024-08-19T14:26:33.440957+0000","last_became_active":"2024-08-19T14:18:26.887657+0000","last_became_peered":"2024-08-19T14:18:26.887657+0000","last_unstale":"2024-08-19T14:26:33.440957+0000","last_undegraded":"2024-08-19T14:26:33.440957+0000","last_fullsized":"2024-08-19T14:26:33.440957+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:25:04.127296+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:25:04.127296+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"465'8627","reported_seq":8732,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-08-19T14:26:33.508607+0000","last_change":"2024-08-19T14:24:44.595877+0000","last_active":"2024-08-19T14:26:33.508607+0000","last_peered":"2024-08-19T14:26:33.508607+0000","last_clean":"2024-08-19T14:26:33.508607+0000","last_became_active":"2024-08-19T14:24:44.595538+0000","last_became_peered":"2024-08-19T14:24:44.595538+0000","last_unstale":"2024-08-19T14:26:33.508607+0000","last_undegraded":"2024-08-19T14:26:33.508607+0000","last_fullsized":"2024-08-19T14:26:33.508607+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:24:43.574783+0000","objects_scrubbed":0,"log_size":8627,"log_dups_size":0,"ondisk_log_size":8627,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":425132032,"num_objects":6487,"num_object_clones":0,"num_object_copies":19461,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6487,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8627,"num_write_kb":483648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"70'139","reported_seq":1080,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-08-19T14:26:29.420806+0000","last_change":"2024-08-19T14:24:45.176469+0000","last_active":"2024-08-19T14:26:29.420806+0000","last_peered":"2024-08-19T14:26:29.420806+0000","last_clean":"2024-08-19T14:26:29.420806+0000","last_became_active":"2024-08-19T14:18:26.887277+0000","last_became_peered":"2024-08-19T14:18:26.887277+0000","last_unstale":"2024-08-19T14:26:29.420806+0000","last_undegraded":"2024-08-19T14:26:29.420806+0000","last_fullsized":"2024-08-19T14:26:29.420806+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:45.176400+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:24:45.176400+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,8],"acting":[5,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.f","version":"463'8379","reported_seq":8480,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-08-19T14:26:31.869065+0000","last_change":"2024-08-19T14:24:44.593761+0000","last_active":"2024-08-19T14:26:31.869065+0000","last_peered":"2024-08-19T14:26:31.869065+0000","last_clean":"2024-08-19T14:26:31.869065+0000","last_became_active":"2024-08-19T14:24:44.593498+0000","last_became_peered":"2024-08-19T14:24:44.593498+0000","last_unstale":"2024-08-19T14:26:31.869065+0000","last_undegraded":"2024-08-19T14:26:31.869065+0000","last_fullsized":"2024-08-19T14:26:31.869065+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:24:43.574783+0000","objects_scrubbed":0,"log_size":8379,"log_dups_size":0,"ondisk_log_size":8379,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":417136640,"num_objects":6365,"num_object_clones":0,"num_object_copies":19095,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8379,"num_write_kb":471808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,11],"acting":[5,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"70'139","reported_seq":1081,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-08-19T14:26:29.419715+0000","last_change":"2024-08-19T14:24:45.610330+0000","last_active":"2024-08-19T14:26:29.419715+0000","last_peered":"2024-08-19T14:26:29.419715+0000","last_clean":"2024-08-19T14:26:29.419715+0000","last_became_active":"2024-08-19T14:18:26.875179+0000","last_became_peered":"2024-08-19T14:18:26.875179+0000","last_unstale":"2024-08-19T14:26:29.419715+0000","last_undegraded":"2024-08-19T14:26:29.419715+0000","last_fullsized":"2024-08-19T14:26:29.419715+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:45.610239+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:24:45.610239+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.a","version":"466'8642","reported_seq":8749,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-08-19T14:26:34.735473+0000","last_change":"2024-08-19T14:24:44.592347+0000","last_active":"2024-08-19T14:26:34.735473+0000","last_peered":"2024-08-19T14:26:34.735473+0000","last_clean":"2024-08-19T14:26:34.735473+0000","last_became_active":"2024-08-19T14:24:44.592024+0000","last_became_peered":"2024-08-19T14:24:44.592024+0000","last_unstale":"2024-08-19T14:26:34.735473+0000","last_undegraded":"2024-08-19T14:26:34.735473+0000","last_fullsized":"2024-08-19T14:26:34.735473+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:24:43.574783+0000","objects_scrubbed":0,"log_size":8642,"log_dups_size":0,"ondisk_log_size":8642,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":420085760,"num_objects":6410,"num_object_clones":0,"num_object_copies":19230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8642,"num_write_kb":481664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,11],"acting":[8,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.e","version":"70'139","reported_seq":737,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-08-19T14:26:32.450373+0000","last_change":"2024-08-19T14:25:04.737969+0000","last_active":"2024-08-19T14:26:32.450373+0000","last_peered":"2024-08-19T14:26:32.450373+0000","last_clean":"2024-08-19T14:26:32.450373+0000","last_became_active":"2024-08-19T14:18:26.884105+0000","last_became_peered":"2024-08-19T14:18:26.884105+0000","last_unstale":"2024-08-19T14:26:32.450373+0000","last_undegraded":"2024-08-19T14:26:32.450373+0000","last_fullsized":"2024-08-19T14:26:32.450373+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:25:04.737897+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:20:15.321945+0000","last_clean_scrub_stamp":"2024-08-19T14:25:04.737897+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.b","version":"464'8764","reported_seq":8867,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-08-19T14:26:32.921920+0000","last_change":"2024-08-19T14:24:44.595612+0000","last_active":"2024-08-19T14:26:32.921920+0000","last_peered":"2024-08-19T14:26:32.921920+0000","last_clean":"2024-08-19T14:26:32.921920+0000","last_became_active":"2024-08-19T14:24:44.595175+0000","last_became_peered":"2024-08-19T14:24:44.595175+0000","last_unstale":"2024-08-19T14:26:32.921920+0000","last_undegraded":"2024-08-19T14:26:32.921920+0000","last_fullsized":"2024-08-19T14:26:32.921920+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:24:43.574783+0000","objects_scrubbed":0,"log_size":8764,"log_dups_size":0,"ondisk_log_size":8764,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":429522944,"num_objects":6554,"num_object_clones":0,"num_object_copies":19662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8764,"num_write_kb":490176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,5],"acting":[9,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.f","version":"70'139","reported_seq":979,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-08-19T14:26:29.421446+0000","last_change":"2024-08-19T14:24:16.076239+0000","last_active":"2024-08-19T14:26:29.421446+0000","last_peered":"2024-08-19T14:26:29.421446+0000","last_clean":"2024-08-19T14:26:29.421446+0000","last_became_active":"2024-08-19T14:18:18.711708+0000","last_became_peered":"2024-08-19T14:18:18.711708+0000","last_unstale":"2024-08-19T14:26:29.421446+0000","last_undegraded":"2024-08-19T14:26:29.421446+0000","last_fullsized":"2024-08-19T14:26:29.421446+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:16.076143+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:24:16.076143+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.8","version":"463'6850","reported_seq":6992,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-08-19T14:26:31.018927+0000","last_change":"2024-08-19T14:25:32.033723+0000","last_active":"2024-08-19T14:26:31.018927+0000","last_peered":"2024-08-19T14:26:31.018927+0000","last_clean":"2024-08-19T14:26:31.018927+0000","last_became_active":"2024-08-19T14:25:29.849790+0000","last_became_peered":"2024-08-19T14:25:29.849790+0000","last_unstale":"2024-08-19T14:26:31.018927+0000","last_undegraded":"2024-08-19T14:26:31.018927+0000","last_fullsized":"2024-08-19T14:26:31.018927+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":0,"last_scrub":"440'5444","last_scrub_stamp":"2024-08-19T14:25:32.033618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:25:32.033618+0000","objects_scrubbed":2745,"log_size":6850,"log_dups_size":0,"ondisk_log_size":6850,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:51.502381+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213581824,"num_objects":3259,"num_object_clones":0,"num_object_copies":9777,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3259,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1406,"num_write_kb":61440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.c","version":"70'139","reported_seq":844,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-08-19T14:26:33.436636+0000","last_change":"2024-08-19T14:24:13.465173+0000","last_active":"2024-08-19T14:26:33.436636+0000","last_peered":"2024-08-19T14:26:33.436636+0000","last_clean":"2024-08-19T14:26:33.436636+0000","last_became_active":"2024-08-19T14:18:26.872942+0000","last_became_peered":"2024-08-19T14:18:26.872942+0000","last_unstale":"2024-08-19T14:26:33.436636+0000","last_undegraded":"2024-08-19T14:26:33.436636+0000","last_fullsized":"2024-08-19T14:26:33.436636+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:13.463716+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:22:52.164586+0000","last_clean_scrub_stamp":"2024-08-19T14:24:13.463716+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,9],"acting":[3,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"465'7032","reported_seq":7175,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-08-19T14:26:33.705939+0000","last_change":"2024-08-19T14:25:31.300738+0000","last_active":"2024-08-19T14:26:33.705939+0000","last_peered":"2024-08-19T14:26:33.705939+0000","last_clean":"2024-08-19T14:26:33.705939+0000","last_became_active":"2024-08-19T14:25:29.821239+0000","last_became_peered":"2024-08-19T14:25:29.821239+0000","last_unstale":"2024-08-19T14:26:33.705939+0000","last_undegraded":"2024-08-19T14:26:33.705939+0000","last_fullsized":"2024-08-19T14:26:33.705939+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'5530","last_scrub_stamp":"2024-08-19T14:25:31.300597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:25:31.300597+0000","objects_scrubbed":2843,"log_size":7032,"log_dups_size":0,"ondisk_log_size":7032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:41.079822+0000","scrub_duration":284,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":211877888,"num_objects":3233,"num_object_clones":0,"num_object_copies":9699,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3233,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1502,"num_write_kb":60544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,9],"acting":[7,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"70'139","reported_seq":759,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-08-19T14:26:33.441345+0000","last_change":"2024-08-19T14:24:50.276533+0000","last_active":"2024-08-19T14:26:33.441345+0000","last_peered":"2024-08-19T14:26:33.441345+0000","last_clean":"2024-08-19T14:26:33.441345+0000","last_became_active":"2024-08-19T14:18:26.882002+0000","last_became_peered":"2024-08-19T14:18:26.882002+0000","last_unstale":"2024-08-19T14:26:33.441345+0000","last_undegraded":"2024-08-19T14:26:33.441345+0000","last_fullsized":"2024-08-19T14:26:33.441345+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:50.276408+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:24:50.276408+0000","last_clean_scrub_stamp":"2024-08-19T14:24:50.276408+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"463'7069","reported_seq":15802,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-08-19T14:26:31.841495+0000","last_change":"2024-08-19T14:26:22.383179+0000","last_active":"2024-08-19T14:26:31.841495+0000","last_peered":"2024-08-19T14:26:31.841495+0000","last_clean":"2024-08-19T14:26:31.841495+0000","last_became_active":"2024-08-19T14:26:22.378003+0000","last_became_peered":"2024-08-19T14:26:22.378003+0000","last_unstale":"2024-08-19T14:26:31.841495+0000","last_undegraded":"2024-08-19T14:26:31.841495+0000","last_fullsized":"2024-08-19T14:26:31.841495+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":439,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:24:43.574783+0000","objects_scrubbed":0,"log_size":7069,"log_dups_size":0,"ondisk_log_size":7069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:34.256756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213581824,"num_objects":3259,"num_object_clones":0,"num_object_copies":9777,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3259,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4287,"num_write_kb":241472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2872,"num_bytes_recovered":188219392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.12","version":"70'139","reported_seq":1014,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-08-19T14:26:29.421126+0000","last_change":"2024-08-19T14:24:14.035778+0000","last_active":"2024-08-19T14:26:29.421126+0000","last_peered":"2024-08-19T14:26:29.421126+0000","last_clean":"2024-08-19T14:26:29.421126+0000","last_became_active":"2024-08-19T14:18:18.703594+0000","last_became_peered":"2024-08-19T14:18:18.703594+0000","last_unstale":"2024-08-19T14:26:29.421126+0000","last_undegraded":"2024-08-19T14:26:29.421126+0000","last_fullsized":"2024-08-19T14:26:29.421126+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:14.035665+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:24:14.035665+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.17","version":"465'7075","reported_seq":17348,"reported_epoch":465,"state":"active+recovering+undersized+remapped","last_fresh":"2024-08-19T14:26:33.661687+0000","last_change":"2024-08-19T14:25:33.132667+0000","last_active":"2024-08-19T14:26:33.661687+0000","last_peered":"2024-08-19T14:26:33.661687+0000","last_clean":"2024-08-19T14:25:30.809461+0000","last_became_active":"2024-08-19T14:25:33.090680+0000","last_became_peered":"2024-08-19T14:25:33.090680+0000","last_unstale":"2024-08-19T14:26:33.661687+0000","last_undegraded":"2024-08-19T14:26:33.661687+0000","last_fullsized":"2024-08-19T14:25:32.820853+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":439,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:24:43.574783+0000","objects_scrubbed":0,"log_size":7075,"log_dups_size":0,"ondisk_log_size":7075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":212140032,"num_objects":3237,"num_object_clones":0,"num_object_copies":9711,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":951,"num_objects_unfound":0,"num_objects_dirty":3237,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4325,"num_write_kb":241984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2540,"num_bytes_recovered":166461440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8,11],"acting":[1,3],"avail_no_missing":["1","5","9"],"object_location_counts":[{"shards":"1,3","objects":2920},{"shards":"1,5,9","objects":317}],"blocked_by":[],"up_primary":3,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.13","version":"70'139","reported_seq":1070,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-08-19T14:26:33.441893+0000","last_change":"2024-08-19T14:25:04.742939+0000","last_active":"2024-08-19T14:26:33.441893+0000","last_peered":"2024-08-19T14:26:33.441893+0000","last_clean":"2024-08-19T14:26:33.441893+0000","last_became_active":"2024-08-19T14:18:18.698334+0000","last_became_peered":"2024-08-19T14:18:18.698334+0000","last_unstale":"2024-08-19T14:26:33.441893+0000","last_undegraded":"2024-08-19T14:26:33.441893+0000","last_fullsized":"2024-08-19T14:26:33.441893+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:25:04.742869+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:25:04.742869+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:34.199161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.14","version":"464'6886","reported_seq":15222,"reported_epoch":465,"state":"creating+peering","last_fresh":"2024-08-19T14:26:33.041279+0000","last_change":"2024-08-19T14:26:33.834096+0000","last_active":"2024-08-19T14:26:33.834096+0000","last_peered":"2024-08-19T14:26:33.041279+0000","last_clean":"2024-08-19T14:25:30.810288+0000","last_became_active":"2024-08-19T14:25:33.057420+0000","last_became_peered":"2024-08-19T14:25:33.057420+0000","last_unstale":"2024-08-19T14:26:33.041279+0000","last_undegraded":"2024-08-19T14:26:33.041279+0000","last_fullsized":"2024-08-19T14:25:32.822291+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":439,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:24:43.574783+0000","objects_scrubbed":0,"log_size":6886,"log_dups_size":0,"ondisk_log_size":6886,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209453056,"num_objects":3196,"num_object_clones":0,"num_object_copies":9588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4204,"num_write_kb":236800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2740,"num_bytes_recovered":179568640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8,4],"acting":[3,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.10","version":"70'139","reported_seq":1177,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-08-19T14:26:29.420703+0000","last_change":"2024-08-19T14:25:06.143718+0000","last_active":"2024-08-19T14:26:29.420703+0000","last_peered":"2024-08-19T14:26:29.420703+0000","last_clean":"2024-08-19T14:26:29.420703+0000","last_became_active":"2024-08-19T14:18:18.704562+0000","last_became_peered":"2024-08-19T14:18:18.704562+0000","last_unstale":"2024-08-19T14:26:29.420703+0000","last_undegraded":"2024-08-19T14:26:29.420703+0000","last_fullsized":"2024-08-19T14:26:29.420703+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:25:06.143606+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:25:06.143606+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,8],"acting":[5,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"463'6831","reported_seq":15231,"reported_epoch":463,"state":"active+undersized+remapped","last_fresh":"2024-08-19T14:26:31.962455+0000","last_change":"2024-08-19T14:26:31.912960+0000","last_active":"2024-08-19T14:26:31.962455+0000","last_peered":"2024-08-19T14:26:31.962455+0000","last_clean":"2024-08-19T14:25:30.812843+0000","last_became_active":"2024-08-19T14:25:33.071774+0000","last_became_peered":"2024-08-19T14:25:33.071774+0000","last_unstale":"2024-08-19T14:26:31.962455+0000","last_undegraded":"2024-08-19T14:26:31.962455+0000","last_fullsized":"2024-08-19T14:25:32.821623+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":439,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:24:43.574783+0000","objects_scrubbed":0,"log_size":6831,"log_dups_size":0,"ondisk_log_size":6831,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209584128,"num_objects":3198,"num_object_clones":0,"num_object_copies":9594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4172,"num_write_kb":235840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2763,"num_bytes_recovered":181075968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,8],"acting":[0,1],"avail_no_missing":["0","1","4","5","8"],"object_location_counts":[{"shards":"0,1","objects":3198}],"blocked_by":[],"up_primary":5,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.11","version":"70'139","reported_seq":726,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-08-19T14:26:29.422573+0000","last_change":"2024-08-19T14:24:50.547549+0000","last_active":"2024-08-19T14:26:29.422573+0000","last_peered":"2024-08-19T14:26:29.422573+0000","last_clean":"2024-08-19T14:26:29.422573+0000","last_became_active":"2024-08-19T14:18:18.697894+0000","last_became_peered":"2024-08-19T14:18:18.697894+0000","last_unstale":"2024-08-19T14:26:29.422573+0000","last_undegraded":"2024-08-19T14:26:29.422573+0000","last_fullsized":"2024-08-19T14:26:29.422573+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:50.547428+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:24:50.547428+0000","last_clean_scrub_stamp":"2024-08-19T14:24:50.547428+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"466'7009","reported_seq":15438,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-08-19T14:26:34.809163+0000","last_change":"2024-08-19T14:26:16.344708+0000","last_active":"2024-08-19T14:26:34.809163+0000","last_peered":"2024-08-19T14:26:34.809163+0000","last_clean":"2024-08-19T14:26:34.809163+0000","last_became_active":"2024-08-19T14:26:16.340194+0000","last_became_peered":"2024-08-19T14:26:16.340194+0000","last_unstale":"2024-08-19T14:26:34.809163+0000","last_undegraded":"2024-08-19T14:26:34.809163+0000","last_fullsized":"2024-08-19T14:26:34.809163+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":439,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:24:43.574783+0000","objects_scrubbed":0,"log_size":7009,"log_dups_size":0,"ondisk_log_size":7009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:38.871967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":208699392,"num_objects":3185,"num_object_clones":0,"num_object_copies":9555,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4303,"num_write_kb":239584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2769,"num_bytes_recovered":181469184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,3],"acting":[8,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"70'139","reported_seq":699,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-08-19T14:26:33.442178+0000","last_change":"2024-08-19T14:24:41.729459+0000","last_active":"2024-08-19T14:26:33.442178+0000","last_peered":"2024-08-19T14:26:33.442178+0000","last_clean":"2024-08-19T14:26:33.442178+0000","last_became_active":"2024-08-19T14:20:28.204724+0000","last_became_peered":"2024-08-19T14:20:28.204724+0000","last_unstale":"2024-08-19T14:26:33.442178+0000","last_undegraded":"2024-08-19T14:26:33.442178+0000","last_fullsized":"2024-08-19T14:26:33.442178+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:41.729377+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:24:41.729377+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"466'6914","reported_seq":10705,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-08-19T14:26:34.736479+0000","last_change":"2024-08-19T14:26:07.300253+0000","last_active":"2024-08-19T14:26:34.736479+0000","last_peered":"2024-08-19T14:26:34.736479+0000","last_clean":"2024-08-19T14:26:34.736479+0000","last_became_active":"2024-08-19T14:26:07.295750+0000","last_became_peered":"2024-08-19T14:26:07.295750+0000","last_unstale":"2024-08-19T14:26:34.736479+0000","last_undegraded":"2024-08-19T14:26:34.736479+0000","last_fullsized":"2024-08-19T14:26:34.736479+0000","mapping_epoch":454,"log_start":"0'0","ondisk_log_start":"0'0","created":439,"last_epoch_clean":455,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:24:43.574783+0000","objects_scrubbed":0,"log_size":6914,"log_dups_size":0,"ondisk_log_size":6914,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:39.855691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":206700544,"num_objects":3154,"num_object_clones":0,"num_object_copies":9462,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3154,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4238,"num_write_kb":236544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2780,"num_bytes_recovered":182190080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,5],"acting":[8,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.17","version":"70'139","reported_seq":829,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-08-19T14:26:34.443629+0000","last_change":"2024-08-19T14:24:48.428636+0000","last_active":"2024-08-19T14:26:34.443629+0000","last_peered":"2024-08-19T14:26:34.443629+0000","last_clean":"2024-08-19T14:26:34.443629+0000","last_became_active":"2024-08-19T14:18:31.901556+0000","last_became_peered":"2024-08-19T14:18:31.901556+0000","last_unstale":"2024-08-19T14:26:34.443629+0000","last_undegraded":"2024-08-19T14:26:34.443629+0000","last_fullsized":"2024-08-19T14:26:34.443629+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:48.428515+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:24:48.428515+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,0],"acting":[8,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"5.10","version":"463'7060","reported_seq":14785,"reported_epoch":463,"state":"active+recovering+undersized+remapped","last_fresh":"2024-08-19T14:26:31.977082+0000","last_change":"2024-08-19T14:25:33.157444+0000","last_active":"2024-08-19T14:26:31.977082+0000","last_peered":"2024-08-19T14:26:31.977082+0000","last_clean":"2024-08-19T14:25:30.748956+0000","last_became_active":"2024-08-19T14:25:33.061446+0000","last_became_peered":"2024-08-19T14:25:33.061446+0000","last_unstale":"2024-08-19T14:26:31.977082+0000","last_undegraded":"2024-08-19T14:26:31.977082+0000","last_fullsized":"2024-08-19T14:25:32.821549+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":439,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:24:43.574783+0000","objects_scrubbed":0,"log_size":7060,"log_dups_size":0,"ondisk_log_size":7060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":215121920,"num_objects":3283,"num_object_clones":0,"num_object_copies":9849,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":516,"num_objects_unfound":0,"num_objects_dirty":3283,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4297,"num_write_kb":242528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2539,"num_bytes_recovered":166395904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,0],"acting":[0,7],"avail_no_missing":["0","1","4"],"object_location_counts":[{"shards":"0,1,4","objects":258},{"shards":"0,7","objects":3025}],"blocked_by":[],"up_primary":7,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"70'139","reported_seq":836,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-08-19T14:26:33.440911+0000","last_change":"2024-08-19T14:25:31.977479+0000","last_active":"2024-08-19T14:26:33.440911+0000","last_peered":"2024-08-19T14:26:33.440911+0000","last_clean":"2024-08-19T14:26:33.440911+0000","last_became_active":"2024-08-19T14:18:18.703926+0000","last_became_peered":"2024-08-19T14:18:18.703926+0000","last_unstale":"2024-08-19T14:26:33.440911+0000","last_undegraded":"2024-08-19T14:26:33.440911+0000","last_fullsized":"2024-08-19T14:26:33.440911+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:25:31.977358+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:25:31.977358+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,1],"acting":[7,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"464'7153","reported_seq":12992,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-08-19T14:26:32.630076+0000","last_change":"2024-08-19T14:26:00.211612+0000","last_active":"2024-08-19T14:26:32.630076+0000","last_peered":"2024-08-19T14:26:32.630076+0000","last_clean":"2024-08-19T14:26:32.630076+0000","last_became_active":"2024-08-19T14:26:00.207101+0000","last_became_peered":"2024-08-19T14:26:00.207101+0000","last_unstale":"2024-08-19T14:26:32.630076+0000","last_undegraded":"2024-08-19T14:26:32.630076+0000","last_fullsized":"2024-08-19T14:26:32.630076+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":439,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:24:43.574783+0000","objects_scrubbed":0,"log_size":7153,"log_dups_size":0,"ondisk_log_size":7153,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:37.628657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213057536,"num_objects":3251,"num_object_clones":0,"num_object_copies":9753,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4367,"num_write_kb":243776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2860,"num_bytes_recovered":187432960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7],"acting":[4,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.15","version":"70'139","reported_seq":715,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-08-19T14:26:33.441325+0000","last_change":"2024-08-19T14:24:44.255453+0000","last_active":"2024-08-19T14:26:33.441325+0000","last_peered":"2024-08-19T14:26:33.441325+0000","last_clean":"2024-08-19T14:26:33.441325+0000","last_became_active":"2024-08-19T14:18:31.912496+0000","last_became_peered":"2024-08-19T14:18:31.912496+0000","last_unstale":"2024-08-19T14:26:33.441325+0000","last_undegraded":"2024-08-19T14:26:33.441325+0000","last_fullsized":"2024-08-19T14:26:33.441325+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:44.255304+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:20:09.180923+0000","last_clean_scrub_stamp":"2024-08-19T14:24:44.255304+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"70'139","reported_seq":699,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-08-19T14:26:32.450168+0000","last_change":"2024-08-19T14:24:44.775890+0000","last_active":"2024-08-19T14:26:32.450168+0000","last_peered":"2024-08-19T14:26:32.450168+0000","last_clean":"2024-08-19T14:26:32.450168+0000","last_became_active":"2024-08-19T14:18:31.901229+0000","last_became_peered":"2024-08-19T14:18:31.901229+0000","last_unstale":"2024-08-19T14:26:32.450168+0000","last_undegraded":"2024-08-19T14:26:32.450168+0000","last_fullsized":"2024-08-19T14:26:32.450168+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:44.775817+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:24:44.775817+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1b","version":"70'139","reported_seq":1055,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-08-19T14:26:34.441745+0000","last_change":"2024-08-19T14:24:47.410923+0000","last_active":"2024-08-19T14:26:34.441745+0000","last_peered":"2024-08-19T14:26:34.441745+0000","last_clean":"2024-08-19T14:26:34.441745+0000","last_became_active":"2024-08-19T14:18:31.901152+0000","last_became_peered":"2024-08-19T14:18:31.901152+0000","last_unstale":"2024-08-19T14:26:34.441745+0000","last_undegraded":"2024-08-19T14:26:34.441745+0000","last_fullsized":"2024-08-19T14:26:34.441745+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:47.410831+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:24:47.410831+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.18","version":"70'139","reported_seq":1093,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-08-19T14:26:29.420590+0000","last_change":"2024-08-19T14:24:40.113681+0000","last_active":"2024-08-19T14:26:29.420590+0000","last_peered":"2024-08-19T14:26:29.420590+0000","last_clean":"2024-08-19T14:26:29.420590+0000","last_became_active":"2024-08-19T14:18:31.902735+0000","last_became_peered":"2024-08-19T14:18:31.902735+0000","last_unstale":"2024-08-19T14:26:29.420590+0000","last_undegraded":"2024-08-19T14:26:29.420590+0000","last_fullsized":"2024-08-19T14:26:29.420590+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:40.113600+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:24:40.113600+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.19","version":"70'139","reported_seq":1041,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-08-19T14:26:29.422580+0000","last_change":"2024-08-19T14:24:42.577775+0000","last_active":"2024-08-19T14:26:29.422580+0000","last_peered":"2024-08-19T14:26:29.422580+0000","last_clean":"2024-08-19T14:26:29.422580+0000","last_became_active":"2024-08-19T14:18:31.902547+0000","last_became_peered":"2024-08-19T14:18:31.902547+0000","last_unstale":"2024-08-19T14:26:29.422580+0000","last_undegraded":"2024-08-19T14:26:29.422580+0000","last_fullsized":"2024-08-19T14:26:29.422580+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:42.577693+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:24:42.577693+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1e","version":"70'139","reported_seq":699,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-08-19T14:26:29.419795+0000","last_change":"2024-08-19T14:24:44.651398+0000","last_active":"2024-08-19T14:26:29.419795+0000","last_peered":"2024-08-19T14:26:29.419795+0000","last_clean":"2024-08-19T14:26:29.419795+0000","last_became_active":"2024-08-19T14:18:31.902546+0000","last_became_peered":"2024-08-19T14:18:31.902546+0000","last_unstale":"2024-08-19T14:26:29.419795+0000","last_undegraded":"2024-08-19T14:26:29.419795+0000","last_fullsized":"2024-08-19T14:26:29.419795+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:44.651299+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:24:44.651299+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.18","version":"463'6817","reported_seq":13819,"reported_epoch":463,"state":"active+recovering+undersized+remapped","last_fresh":"2024-08-19T14:26:31.978621+0000","last_change":"2024-08-19T14:25:33.185253+0000","last_active":"2024-08-19T14:26:31.978621+0000","last_peered":"2024-08-19T14:26:31.978621+0000","last_clean":"2024-08-19T14:25:30.813243+0000","last_became_active":"2024-08-19T14:25:33.069086+0000","last_became_peered":"2024-08-19T14:25:33.069086+0000","last_unstale":"2024-08-19T14:26:31.978621+0000","last_undegraded":"2024-08-19T14:26:31.978621+0000","last_fullsized":"2024-08-19T14:25:32.821916+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":439,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:24:43.574783+0000","objects_scrubbed":0,"log_size":6817,"log_dups_size":0,"ondisk_log_size":6817,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205029376,"num_objects":3129,"num_object_clones":0,"num_object_copies":9387,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":916,"num_objects_unfound":0,"num_objects_dirty":3129,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4129,"num_write_kb":232224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2298,"num_bytes_recovered":150601728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,1],"acting":[0,11],"avail_no_missing":["0","9","11"],"object_location_counts":[{"shards":"0,9,11","objects":458},{"shards":"0,11","objects":2671}],"blocked_by":[],"up_primary":4,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1c","version":"70'139","reported_seq":813,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-08-19T14:26:34.441955+0000","last_change":"2024-08-19T14:24:14.461312+0000","last_active":"2024-08-19T14:26:34.441955+0000","last_peered":"2024-08-19T14:26:34.441955+0000","last_clean":"2024-08-19T14:26:34.441955+0000","last_became_active":"2024-08-19T14:18:31.902494+0000","last_became_peered":"2024-08-19T14:18:31.902494+0000","last_unstale":"2024-08-19T14:26:34.441955+0000","last_undegraded":"2024-08-19T14:26:34.441955+0000","last_fullsized":"2024-08-19T14:26:34.441955+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:14.461176+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:24:14.461176+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,11],"acting":[8,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"5.19","version":"465'7020","reported_seq":15395,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-08-19T14:26:33.508442+0000","last_change":"2024-08-19T14:26:15.347698+0000","last_active":"2024-08-19T14:26:33.508442+0000","last_peered":"2024-08-19T14:26:33.508442+0000","last_clean":"2024-08-19T14:26:33.508442+0000","last_became_active":"2024-08-19T14:26:15.340594+0000","last_became_peered":"2024-08-19T14:26:15.340594+0000","last_unstale":"2024-08-19T14:26:33.508442+0000","last_undegraded":"2024-08-19T14:26:33.508442+0000","last_fullsized":"2024-08-19T14:26:33.508442+0000","mapping_epoch":457,"log_start":"0'0","ondisk_log_start":"0'0","created":439,"last_epoch_clean":458,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:24:43.574783+0000","objects_scrubbed":0,"log_size":7020,"log_dups_size":0,"ondisk_log_size":7020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:36.955764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209453056,"num_objects":3196,"num_object_clones":0,"num_object_copies":9588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4300,"num_write_kb":239872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2753,"num_bytes_recovered":180420608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1d","version":"70'139","reported_seq":725,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-08-19T14:26:33.442491+0000","last_change":"2024-08-19T14:24:22.588391+0000","last_active":"2024-08-19T14:26:33.442491+0000","last_peered":"2024-08-19T14:26:33.442491+0000","last_clean":"2024-08-19T14:26:33.442491+0000","last_became_active":"2024-08-19T14:18:31.901183+0000","last_became_peered":"2024-08-19T14:18:31.901183+0000","last_unstale":"2024-08-19T14:26:33.442491+0000","last_undegraded":"2024-08-19T14:26:33.442491+0000","last_fullsized":"2024-08-19T14:26:33.442491+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:22.588266+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:24:22.588266+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,5],"acting":[1,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":26,"stat_sum":{"num_bytes":6790250520,"num_objects":103613,"num_object_clones":0,"num_object_copies":310839,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2383,"num_objects_unfound":0,"num_objects_dirty":103613,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":114558,"num_write_kb":6270273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26914,"num_bytes_recovered":1763835904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":22538235904,"data_stored":22538223688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":191261,"ondisk_log_size":191261,"up":78,"acting":74,"num_store_stats":9},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1802240,"data_stored":1788032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187638,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2660828,"kb_used_data":2084048,"kb_used_omap":1,"kb_used_meta":576766,"kb_avail":91084324,"statfs":{"total":95995035648,"available":93270347776,"internally_reserved":0,"allocated":2134065152,"data_stored":2131964405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":590608836},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":343597383868,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":74,"seq":317827580096,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2791652,"kb_used_data":2169048,"kb_used_omap":1,"kb_used_meta":622590,"kb_avail":90953500,"statfs":{"total":95995035648,"available":93136384000,"internally_reserved":0,"allocated":2221105152,"data_stored":2219004367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":637532609},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874438,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3186236,"kb_used_data":2667760,"kb_used_omap":1,"kb_used_meta":518462,"kb_avail":90558916,"statfs":{"total":95995035648,"available":92732329984,"internally_reserved":0,"allocated":2731786240,"data_stored":2729677214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":530905541},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":57,"seq":244813136073,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2534256,"kb_used_data":1867236,"kb_used_omap":1,"kb_used_meta":667006,"kb_avail":91210896,"statfs":{"total":95995035648,"available":93399957504,"internally_reserved":0,"allocated":1912049664,"data_stored":1909941255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":683014602},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":206158430412,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692048,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4033632,"kb_used_data":3511000,"kb_used_omap":1,"kb_used_meta":522622,"kb_avail":89711520,"statfs":{"total":95995035648,"available":91864596480,"internally_reserved":0,"allocated":3595264000,"data_stored":3593159701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":535165381},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986389,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3454116,"kb_used_data":2800984,"kb_used_omap":1,"kb_used_meta":653118,"kb_avail":90291036,"statfs":{"total":95995035648,"available":92458020864,"internally_reserved":0,"allocated":2868207616,"data_stored":2866106831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":668793285},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149915,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1840676,"kb_used_data":1223640,"kb_used_omap":1,"kb_used_meta":617022,"kb_avail":91904476,"statfs":{"total":95995035648,"available":94110183424,"internally_reserved":0,"allocated":1253007360,"data_stored":1250906575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":631830983},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411552,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575076,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2910700,"kb_used_data":2215904,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":90834452,"statfs":{"total":95995035648,"available":93014478848,"internally_reserved":0,"allocated":2269085696,"data_stored":2266981359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738600,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4131288,"kb_used_data":3580876,"kb_used_omap":1,"kb_used_meta":550398,"kb_avail":89613864,"statfs":{"total":95995035648,"available":91764596736,"internally_reserved":0,"allocated":3666817024,"data_stored":3664720349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":563608005},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3621519360,"data_stored":3621519360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2267348992,"data_stored":2267348992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1249378304,"data_stored":1249378304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2861498368,"data_stored":2861498368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3570733056,"data_stored":3570728984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1911033856,"data_stored":1911029784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":2727739392,"data_stored":2727739392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":2214854656,"data_stored":2214854656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":2114129920,"data_stored":2114125848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T14:26:52.963 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:52.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:52.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:26:53.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:52 smithi032 ceph-mon[20004]: osdmap e471: 12 total, 12 up, 9 in 2024-08-19T14:26:53.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:52 smithi045 ceph-mon[20143]: osdmap e471: 12 total, 12 up, 9 in 2024-08-19T14:26:53.065 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:26:53.096 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:53.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:53.097 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:26:53.193 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:26:53.194 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:26:53.198 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:26:53.229 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:53.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:53.231 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:26:53.332 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:26:53.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:53.353 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:53.353 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:26:53.454 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:26:53.476 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:53.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:53.477 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:26:53.578 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:26:53.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:53.609 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:53.609 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:26:53.710 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:26:53.744 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:53.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:53.745 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:26:53.803 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:53 smithi031 ceph-mon[16336]: pgmap v1080: 57 pgs: 57 active+clean; 5.1 GiB data, 22 GiB used, 783 GiB / 805 GiB avail; 0 B/s wr, 1.47k op/s 2024-08-19T14:26:53.847 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:26:53.875 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:53.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:53.876 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:26:53.936 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:26:53.977 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:26:54.005 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:54.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:54.006 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:26:54.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:53 smithi032 ceph-mon[20004]: pgmap v1080: 57 pgs: 57 active+clean; 5.1 GiB data, 22 GiB used, 783 GiB / 805 GiB avail; 0 B/s wr, 1.47k op/s 2024-08-19T14:26:54.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:53 smithi045 ceph-mon[20143]: pgmap v1080: 57 pgs: 57 active+clean; 5.1 GiB data, 22 GiB used, 783 GiB / 805 GiB avail; 0 B/s wr, 1.47k op/s 2024-08-19T14:26:54.107 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:26:54.146 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:54.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:54.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:26:54.248 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:26:54.279 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:54.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:54.280 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:26:54.381 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:26:54.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:54.415 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:54.415 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:26:54.516 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:26:54.546 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:54.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:54.547 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:26:54.580 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-08-19T14:26:54.588 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is unset 2024-08-19T14:26:54.647 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:26:54.678 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:54.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:54.679 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:26:54.780 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:26:54.813 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:54.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:54.814 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:26:54.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:54 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/615783297' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:26:54.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:54 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:26:54.916 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:26:54.948 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:54.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:54.950 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:26:55.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:54 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/615783297' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:26:55.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:54 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:26:55.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:54 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/615783297' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:26:55.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:54 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:26:55.051 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:26:55.078 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:55.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:55.079 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:26:55.182 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:26:55.220 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:55.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:55.221 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:26:55.322 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:26:55.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:55.393 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:55.393 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:26:55.397 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:26:55.398 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:26:55.423 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:26:55.423 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:26:55.447 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:26:55.448 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:26:55.474 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:26:55.475 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:26:55.494 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:26:55.502 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:26:55.502 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:26:55.536 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:55.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:55.537 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:26:55.538 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:26:55.538 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:26:55.566 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:26:55.567 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:26:55.598 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:26:55.598 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:26:55.626 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:26:55.627 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:26:55.638 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:26:55.655 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:26:55.655 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:26:55.670 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:55.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:55.671 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:26:55.681 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:26:55.682 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:26:55.709 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:26:55.709 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:26:55.755 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:55 smithi031 ceph-mon[16336]: pgmap v1081: 57 pgs: 57 active+clean; 4.9 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 0 B/s wr, 1.62k op/s 2024-08-19T14:26:55.755 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:55 smithi031 ceph-mon[16336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:26:55.755 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:55 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:26:55.755 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:55 smithi031 ceph-mon[16336]: osdmap e472: 12 total, 12 up, 9 in 2024-08-19T14:26:55.756 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:55 smithi031 ceph-mon[16336]: 5.12 scrub starts 2024-08-19T14:26:55.756 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:55 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1739602074' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:26:55.756 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:55 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:26:55.756 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:55 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:26:55.772 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:26:55.806 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:55.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:55.807 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:26:55.908 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:26:55.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:55.935 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:55.935 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:26:55.947 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:55 smithi045 ceph-mon[20143]: pgmap v1081: 57 pgs: 57 active+clean; 4.9 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 0 B/s wr, 1.62k op/s 2024-08-19T14:26:55.947 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:55 smithi045 ceph-mon[20143]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:26:55.947 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:55 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:26:55.947 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:55 smithi045 ceph-mon[20143]: osdmap e472: 12 total, 12 up, 9 in 2024-08-19T14:26:55.947 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:55 smithi045 ceph-mon[20143]: 5.12 scrub starts 2024-08-19T14:26:55.947 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:55 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1739602074' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:26:55.947 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:55 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:26:55.947 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:55 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:26:56.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:55 smithi032 ceph-mon[20004]: pgmap v1081: 57 pgs: 57 active+clean; 4.9 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 0 B/s wr, 1.62k op/s 2024-08-19T14:26:56.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:55 smithi032 ceph-mon[20004]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:26:56.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:55 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:26:56.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:55 smithi032 ceph-mon[20004]: osdmap e472: 12 total, 12 up, 9 in 2024-08-19T14:26:56.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:55 smithi032 ceph-mon[20004]: 5.12 scrub starts 2024-08-19T14:26:56.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:55 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1739602074' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:26:56.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:55 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:26:56.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:55 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:26:56.037 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:26:56.066 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:56.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:56.067 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:26:56.168 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:26:56.197 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:56.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:56.198 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:26:56.299 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:26:56.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:56.332 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:56.332 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:26:56.433 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:26:56.461 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:56.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:56.462 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:26:56.563 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:26:56.595 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:56.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:56.596 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:26:56.635 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:26:56.697 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:26:56.727 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:56.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:56.728 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:26:56.829 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:26:56.864 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:56.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:56.865 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:26:56.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:56 smithi031 ceph-mon[16336]: 1.19 scrub starts 2024-08-19T14:26:56.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:56 smithi031 ceph-mon[16336]: 1.19 scrub ok 2024-08-19T14:26:56.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:56 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:26:56.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:56 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:26:56.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:56 smithi031 ceph-mon[16336]: 5.16 scrub starts 2024-08-19T14:26:56.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:56 smithi031 ceph-mon[16336]: 5.12 scrub ok 2024-08-19T14:26:56.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:56 smithi031 ceph-mon[16336]: osdmap e473: 12 total, 12 up, 9 in 2024-08-19T14:26:56.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:56 smithi031 ceph-mon[16336]: 1.4 scrub starts 2024-08-19T14:26:56.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:56 smithi031 ceph-mon[16336]: 1.4 scrub ok 2024-08-19T14:26:56.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:56 smithi031 ceph-mon[16336]: 5.13 scrub starts 2024-08-19T14:26:56.966 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:26:56.991 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:56.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:56.992 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:26:57.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:56 smithi032 ceph-mon[20004]: 1.19 scrub starts 2024-08-19T14:26:57.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:56 smithi032 ceph-mon[20004]: 1.19 scrub ok 2024-08-19T14:26:57.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:56 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:26:57.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:56 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:26:57.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:56 smithi032 ceph-mon[20004]: 5.16 scrub starts 2024-08-19T14:26:57.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:56 smithi032 ceph-mon[20004]: 5.12 scrub ok 2024-08-19T14:26:57.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:56 smithi032 ceph-mon[20004]: osdmap e473: 12 total, 12 up, 9 in 2024-08-19T14:26:57.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:56 smithi032 ceph-mon[20004]: 1.4 scrub starts 2024-08-19T14:26:57.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:56 smithi032 ceph-mon[20004]: 1.4 scrub ok 2024-08-19T14:26:57.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:56 smithi032 ceph-mon[20004]: 5.13 scrub starts 2024-08-19T14:26:57.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:56 smithi045 ceph-mon[20143]: 1.19 scrub starts 2024-08-19T14:26:57.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:56 smithi045 ceph-mon[20143]: 1.19 scrub ok 2024-08-19T14:26:57.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:56 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:26:57.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:56 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:26:57.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:56 smithi045 ceph-mon[20143]: 5.16 scrub starts 2024-08-19T14:26:57.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:56 smithi045 ceph-mon[20143]: 5.12 scrub ok 2024-08-19T14:26:57.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:56 smithi045 ceph-mon[20143]: osdmap e473: 12 total, 12 up, 9 in 2024-08-19T14:26:57.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:56 smithi045 ceph-mon[20143]: 1.4 scrub starts 2024-08-19T14:26:57.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:56 smithi045 ceph-mon[20143]: 1.4 scrub ok 2024-08-19T14:26:57.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:56 smithi045 ceph-mon[20143]: 5.13 scrub starts 2024-08-19T14:26:57.093 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:26:57.125 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:57.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:57.126 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:26:57.213 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:26:57.213 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-08-19T14:26:57.213 INFO:teuthology.orchestra.run.smithi031.stdout: "complaint_time": 30, 2024-08-19T14:26:57.213 INFO:teuthology.orchestra.run.smithi031.stdout: "num_blocked_ops": 0 2024-08-19T14:26:57.213 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:26:57.227 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:26:57.258 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:57.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:57.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:26:57.360 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:26:57.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:57.391 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:57.391 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:26:57.492 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:26:57.524 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:57.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:57.526 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:26:57.627 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:26:57.657 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:57.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:57.659 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:26:57.759 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:26:57.790 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:57.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:57.792 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:26:57.797 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-19T14:26:57.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:57 smithi031 ceph-mon[16336]: 5.16 scrub ok 2024-08-19T14:26:57.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:57 smithi031 ceph-mon[16336]: pgmap v1084: 57 pgs: 1 active+clean+scrubbing, 56 active+clean; 4.8 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 0 B/s wr, 1.29k op/s 2024-08-19T14:26:57.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:57 smithi031 ceph-mon[16336]: 5.2 scrub starts 2024-08-19T14:26:57.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:57 smithi031 ceph-mon[16336]: 5.15 scrub starts 2024-08-19T14:26:57.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:57 smithi031 ceph-mon[16336]: 5.13 scrub ok 2024-08-19T14:26:57.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:57 smithi031 ceph-mon[16336]: 5.2 scrub ok 2024-08-19T14:26:57.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:57 smithi031 ceph-mon[16336]: 1.5 scrub starts 2024-08-19T14:26:57.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:57 smithi031 ceph-mon[16336]: 1.5 scrub ok 2024-08-19T14:26:57.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:57 smithi031 ceph-mon[16336]: 1.15 scrub starts 2024-08-19T14:26:57.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:57 smithi031 ceph-mon[16336]: 1.15 scrub ok 2024-08-19T14:26:57.892 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:26:57.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:57.924 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:57.925 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:26:57.951 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 7, 8, 9, 11] out_osds: [10, 6, 2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-08-19T14:26:57.951 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-19T14:26:57.952 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-08-19T14:26:57.952 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-19T14:26:58.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:57 smithi032 ceph-mon[20004]: 5.16 scrub ok 2024-08-19T14:26:58.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:57 smithi032 ceph-mon[20004]: pgmap v1084: 57 pgs: 1 active+clean+scrubbing, 56 active+clean; 4.8 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 0 B/s wr, 1.29k op/s 2024-08-19T14:26:58.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:57 smithi032 ceph-mon[20004]: 5.2 scrub starts 2024-08-19T14:26:58.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:57 smithi032 ceph-mon[20004]: 5.15 scrub starts 2024-08-19T14:26:58.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:57 smithi032 ceph-mon[20004]: 5.13 scrub ok 2024-08-19T14:26:58.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:57 smithi032 ceph-mon[20004]: 5.2 scrub ok 2024-08-19T14:26:58.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:57 smithi032 ceph-mon[20004]: 1.5 scrub starts 2024-08-19T14:26:58.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:57 smithi032 ceph-mon[20004]: 1.5 scrub ok 2024-08-19T14:26:58.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:57 smithi032 ceph-mon[20004]: 1.15 scrub starts 2024-08-19T14:26:58.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:57 smithi032 ceph-mon[20004]: 1.15 scrub ok 2024-08-19T14:26:58.025 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:26:58.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:57 smithi045 ceph-mon[20143]: 5.16 scrub ok 2024-08-19T14:26:58.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:57 smithi045 ceph-mon[20143]: pgmap v1084: 57 pgs: 1 active+clean+scrubbing, 56 active+clean; 4.8 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 0 B/s wr, 1.29k op/s 2024-08-19T14:26:58.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:57 smithi045 ceph-mon[20143]: 5.2 scrub starts 2024-08-19T14:26:58.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:57 smithi045 ceph-mon[20143]: 5.15 scrub starts 2024-08-19T14:26:58.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:57 smithi045 ceph-mon[20143]: 5.13 scrub ok 2024-08-19T14:26:58.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:57 smithi045 ceph-mon[20143]: 5.2 scrub ok 2024-08-19T14:26:58.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:57 smithi045 ceph-mon[20143]: 1.5 scrub starts 2024-08-19T14:26:58.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:57 smithi045 ceph-mon[20143]: 1.5 scrub ok 2024-08-19T14:26:58.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:57 smithi045 ceph-mon[20143]: 1.15 scrub starts 2024-08-19T14:26:58.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:57 smithi045 ceph-mon[20143]: 1.15 scrub ok 2024-08-19T14:26:58.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:58.064 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:58.065 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:26:58.142 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-08-19T14:26:58.165 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:26:58.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:58.199 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:58.199 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:26:58.210 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:26:58.299 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:26:58.328 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:58.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:58.329 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:26:58.430 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:26:58.457 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:58.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:58.458 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:26:58.559 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:26:58.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:58.606 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:58.606 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:26:58.631 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:26:58.708 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:26:58.738 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:58.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:58.739 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:26:58.839 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:26:58.863 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:58.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:58.864 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:26:58.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:58 smithi031 ceph-mon[16336]: 5.15 scrub ok 2024-08-19T14:26:58.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:58 smithi031 ceph-mon[16336]: 5.14 scrub starts 2024-08-19T14:26:58.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:58 smithi031 ceph-mon[16336]: 5.14 scrub ok 2024-08-19T14:26:58.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:58 smithi031 ceph-mon[16336]: 1.2 scrub starts 2024-08-19T14:26:58.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:58 smithi031 ceph-mon[16336]: 1.2 scrub ok 2024-08-19T14:26:58.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:58 smithi031 ceph-mon[16336]: 1.1c scrub starts 2024-08-19T14:26:58.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:58 smithi031 ceph-mon[16336]: 1.1c scrub ok 2024-08-19T14:26:58.964 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:26:58.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:58.992 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:58.992 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:26:59.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:58 smithi032 ceph-mon[20004]: 5.15 scrub ok 2024-08-19T14:26:59.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:58 smithi032 ceph-mon[20004]: 5.14 scrub starts 2024-08-19T14:26:59.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:58 smithi032 ceph-mon[20004]: 5.14 scrub ok 2024-08-19T14:26:59.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:58 smithi032 ceph-mon[20004]: 1.2 scrub starts 2024-08-19T14:26:59.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:58 smithi032 ceph-mon[20004]: 1.2 scrub ok 2024-08-19T14:26:59.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:58 smithi032 ceph-mon[20004]: 1.1c scrub starts 2024-08-19T14:26:59.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:58 smithi032 ceph-mon[20004]: 1.1c scrub ok 2024-08-19T14:26:59.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:58 smithi045 ceph-mon[20143]: 5.15 scrub ok 2024-08-19T14:26:59.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:58 smithi045 ceph-mon[20143]: 5.14 scrub starts 2024-08-19T14:26:59.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:58 smithi045 ceph-mon[20143]: 5.14 scrub ok 2024-08-19T14:26:59.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:58 smithi045 ceph-mon[20143]: 1.2 scrub starts 2024-08-19T14:26:59.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:58 smithi045 ceph-mon[20143]: 1.2 scrub ok 2024-08-19T14:26:59.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:58 smithi045 ceph-mon[20143]: 1.1c scrub starts 2024-08-19T14:26:59.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:58 smithi045 ceph-mon[20143]: 1.1c scrub ok 2024-08-19T14:26:59.055 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:26:59.093 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:26:59.094 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:26:59.113 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:26:59.122 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:59.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:59.123 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:26:59.224 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:26:59.257 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:59.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:59.259 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:26:59.360 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:26:59.386 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:26:59.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:59.387 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:26:59.488 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:26:59.524 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:59.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:59.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:26:59.619 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is unset 2024-08-19T14:26:59.626 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:26:59.661 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:59.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:59.663 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:26:59.764 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:26:59.797 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:26:59.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:59.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:26:59.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:59 smithi031 ceph-mon[16336]: pgmap v1085: 57 pgs: 1 active+clean+scrubbing, 56 active+clean; 4.6 GiB data, 20 GiB used, 784 GiB / 805 GiB avail 2024-08-19T14:26:59.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:59 smithi031 ceph-mon[16336]: 1.f scrub starts 2024-08-19T14:26:59.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:59 smithi031 ceph-mon[16336]: 1.f scrub ok 2024-08-19T14:26:59.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:59 smithi031 ceph-mon[16336]: 5.18 scrub starts 2024-08-19T14:26:59.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:59 smithi031 ceph-mon[16336]: 5.10 scrub starts 2024-08-19T14:26:59.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:59 smithi031 ceph-mon[16336]: 5.19 scrub starts 2024-08-19T14:26:59.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:59 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/194624444' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:26:59.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:26:59 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:26:59.899 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:26:59.928 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:26:59.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:26:59.929 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:27:00.010 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:59 smithi032 ceph-mon[20004]: pgmap v1085: 57 pgs: 1 active+clean+scrubbing, 56 active+clean; 4.6 GiB data, 20 GiB used, 784 GiB / 805 GiB avail 2024-08-19T14:27:00.010 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:59 smithi032 ceph-mon[20004]: 1.f scrub starts 2024-08-19T14:27:00.010 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:59 smithi032 ceph-mon[20004]: 1.f scrub ok 2024-08-19T14:27:00.010 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:59 smithi032 ceph-mon[20004]: 5.18 scrub starts 2024-08-19T14:27:00.011 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:59 smithi032 ceph-mon[20004]: 5.10 scrub starts 2024-08-19T14:27:00.011 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:59 smithi032 ceph-mon[20004]: 5.19 scrub starts 2024-08-19T14:27:00.011 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:59 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/194624444' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:27:00.011 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:26:59 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:27:00.030 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:27:00.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:59 smithi045 ceph-mon[20143]: pgmap v1085: 57 pgs: 1 active+clean+scrubbing, 56 active+clean; 4.6 GiB data, 20 GiB used, 784 GiB / 805 GiB avail 2024-08-19T14:27:00.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:59 smithi045 ceph-mon[20143]: 1.f scrub starts 2024-08-19T14:27:00.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:59 smithi045 ceph-mon[20143]: 1.f scrub ok 2024-08-19T14:27:00.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:59 smithi045 ceph-mon[20143]: 5.18 scrub starts 2024-08-19T14:27:00.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:59 smithi045 ceph-mon[20143]: 5.10 scrub starts 2024-08-19T14:27:00.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:59 smithi045 ceph-mon[20143]: 5.19 scrub starts 2024-08-19T14:27:00.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:59 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/194624444' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:27:00.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:26:59 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:27:00.060 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:00.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:00.061 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:27:00.066 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:27:00.067 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:27:00.085 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:27:00.085 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:27:00.109 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:27:00.110 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:27:00.133 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:27:00.133 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:27:00.161 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:27:00.164 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:27:00.164 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:27:00.192 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:00.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:00.193 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:27:00.194 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:27:00.194 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:27:00.224 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:27:00.225 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:27:00.253 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:27:00.253 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:27:00.279 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:27:00.279 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:27:00.295 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:27:00.310 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:27:00.310 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:27:00.324 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:00.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:00.325 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:27:00.334 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:27:00.334 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:27:00.368 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:27:00.369 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:27:00.426 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:27:00.454 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:00.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:00.455 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:27:00.556 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:27:00.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:00.587 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:00.587 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:27:00.688 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:27:00.720 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:00.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:00.721 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:27:00.822 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:27:00.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:00.851 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:00.851 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:27:00.951 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:27:00.984 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:00.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:00.985 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:27:01.000 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:00 smithi031 ceph-mon[16336]: 1.11 scrub starts 2024-08-19T14:27:01.000 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:00 smithi031 ceph-mon[16336]: 1.11 scrub ok 2024-08-19T14:27:01.000 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:00 smithi031 ceph-mon[16336]: 5.18 scrub ok 2024-08-19T14:27:01.000 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:00 smithi031 ceph-mon[16336]: 5.17 scrub starts 2024-08-19T14:27:01.000 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:00 smithi031 ceph-mon[16336]: 5.10 scrub ok 2024-08-19T14:27:01.000 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:00 smithi031 ceph-mon[16336]: 5.6 scrub starts 2024-08-19T14:27:01.000 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:00 smithi031 ceph-mon[16336]: 5.19 scrub ok 2024-08-19T14:27:01.000 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:00 smithi031 ceph-mon[16336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:27:01.000 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:00 smithi031 ceph-mon[16336]: Cluster is now healthy 2024-08-19T14:27:01.000 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:00 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:27:01.000 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:00 smithi031 ceph-mon[16336]: 5.11 scrub starts 2024-08-19T14:27:01.000 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:00 smithi031 ceph-mon[16336]: osdmap e474: 12 total, 12 up, 9 in 2024-08-19T14:27:01.000 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:00 smithi031 ceph-mon[16336]: 5.17 scrub ok 2024-08-19T14:27:01.001 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:00 smithi031 ceph-mon[16336]: 1.1d scrub starts 2024-08-19T14:27:01.001 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:00 smithi031 ceph-mon[16336]: 1.1d scrub ok 2024-08-19T14:27:01.001 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:00 smithi031 ceph-mon[16336]: 1.1b scrub starts 2024-08-19T14:27:01.001 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:00 smithi031 ceph-mon[16336]: 1.1b scrub ok 2024-08-19T14:27:01.001 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:00 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1170144493' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:27:01.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:00 smithi032 ceph-mon[20004]: 1.11 scrub starts 2024-08-19T14:27:01.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:00 smithi032 ceph-mon[20004]: 1.11 scrub ok 2024-08-19T14:27:01.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:00 smithi032 ceph-mon[20004]: 5.18 scrub ok 2024-08-19T14:27:01.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:00 smithi032 ceph-mon[20004]: 5.17 scrub starts 2024-08-19T14:27:01.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:00 smithi032 ceph-mon[20004]: 5.10 scrub ok 2024-08-19T14:27:01.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:00 smithi032 ceph-mon[20004]: 5.6 scrub starts 2024-08-19T14:27:01.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:00 smithi032 ceph-mon[20004]: 5.19 scrub ok 2024-08-19T14:27:01.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:00 smithi032 ceph-mon[20004]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:27:01.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:00 smithi032 ceph-mon[20004]: Cluster is now healthy 2024-08-19T14:27:01.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:00 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:27:01.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:00 smithi032 ceph-mon[20004]: 5.11 scrub starts 2024-08-19T14:27:01.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:00 smithi032 ceph-mon[20004]: osdmap e474: 12 total, 12 up, 9 in 2024-08-19T14:27:01.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:00 smithi032 ceph-mon[20004]: 5.17 scrub ok 2024-08-19T14:27:01.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:00 smithi032 ceph-mon[20004]: 1.1d scrub starts 2024-08-19T14:27:01.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:00 smithi032 ceph-mon[20004]: 1.1d scrub ok 2024-08-19T14:27:01.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:00 smithi032 ceph-mon[20004]: 1.1b scrub starts 2024-08-19T14:27:01.009 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:00 smithi032 ceph-mon[20004]: 1.1b scrub ok 2024-08-19T14:27:01.009 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:00 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1170144493' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:27:01.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:00 smithi045 ceph-mon[20143]: 1.11 scrub starts 2024-08-19T14:27:01.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:00 smithi045 ceph-mon[20143]: 1.11 scrub ok 2024-08-19T14:27:01.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:00 smithi045 ceph-mon[20143]: 5.18 scrub ok 2024-08-19T14:27:01.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:00 smithi045 ceph-mon[20143]: 5.17 scrub starts 2024-08-19T14:27:01.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:00 smithi045 ceph-mon[20143]: 5.10 scrub ok 2024-08-19T14:27:01.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:00 smithi045 ceph-mon[20143]: 5.6 scrub starts 2024-08-19T14:27:01.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:00 smithi045 ceph-mon[20143]: 5.19 scrub ok 2024-08-19T14:27:01.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:00 smithi045 ceph-mon[20143]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:27:01.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:00 smithi045 ceph-mon[20143]: Cluster is now healthy 2024-08-19T14:27:01.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:00 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:27:01.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:00 smithi045 ceph-mon[20143]: 5.11 scrub starts 2024-08-19T14:27:01.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:00 smithi045 ceph-mon[20143]: osdmap e474: 12 total, 12 up, 9 in 2024-08-19T14:27:01.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:00 smithi045 ceph-mon[20143]: 5.17 scrub ok 2024-08-19T14:27:01.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:00 smithi045 ceph-mon[20143]: 1.1d scrub starts 2024-08-19T14:27:01.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:00 smithi045 ceph-mon[20143]: 1.1d scrub ok 2024-08-19T14:27:01.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:00 smithi045 ceph-mon[20143]: 1.1b scrub starts 2024-08-19T14:27:01.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:00 smithi045 ceph-mon[20143]: 1.1b scrub ok 2024-08-19T14:27:01.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:00 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1170144493' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:27:01.086 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:27:01.119 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:01.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:01.121 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:27:01.222 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:27:01.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:01.252 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:01.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:27:01.353 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:27:01.389 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:01.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:01.390 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:27:01.396 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:27:01.397 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-08-19T14:27:01.490 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:27:01.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:01.518 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:01.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:27:01.620 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:27:01.656 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:01.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:01.657 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:27:01.757 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:27:01.783 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:01.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:01.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:27:01.787 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:01 smithi031 ceph-mon[16336]: 5.6 scrub ok 2024-08-19T14:27:01.787 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:01 smithi031 ceph-mon[16336]: pgmap v1087: 57 pgs: 1 active+clean+scrubbing, 56 active+clean; 4.5 GiB data, 20 GiB used, 785 GiB / 805 GiB avail 2024-08-19T14:27:01.788 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:01 smithi031 ceph-mon[16336]: 5.11 scrub ok 2024-08-19T14:27:01.788 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:01 smithi031 ceph-mon[16336]: 1.c scrub starts 2024-08-19T14:27:01.788 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:01 smithi031 ceph-mon[16336]: 1.c scrub ok 2024-08-19T14:27:01.788 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:01 smithi031 ceph-mon[16336]: 1.12 scrub starts 2024-08-19T14:27:01.788 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:01 smithi031 ceph-mon[16336]: 1.12 scrub ok 2024-08-19T14:27:01.788 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:01 smithi031 ceph-mon[16336]: 1.1a scrub starts 2024-08-19T14:27:01.788 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:01 smithi031 ceph-mon[16336]: 1.1a scrub ok 2024-08-19T14:27:01.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:01 smithi045 ceph-mon[20143]: 5.6 scrub ok 2024-08-19T14:27:01.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:01 smithi045 ceph-mon[20143]: pgmap v1087: 57 pgs: 1 active+clean+scrubbing, 56 active+clean; 4.5 GiB data, 20 GiB used, 785 GiB / 805 GiB avail 2024-08-19T14:27:01.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:01 smithi045 ceph-mon[20143]: 5.11 scrub ok 2024-08-19T14:27:01.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:01 smithi045 ceph-mon[20143]: 1.c scrub starts 2024-08-19T14:27:01.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:01 smithi045 ceph-mon[20143]: 1.c scrub ok 2024-08-19T14:27:01.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:01 smithi045 ceph-mon[20143]: 1.12 scrub starts 2024-08-19T14:27:01.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:01 smithi045 ceph-mon[20143]: 1.12 scrub ok 2024-08-19T14:27:01.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:01 smithi045 ceph-mon[20143]: 1.1a scrub starts 2024-08-19T14:27:01.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:01 smithi045 ceph-mon[20143]: 1.1a scrub ok 2024-08-19T14:27:01.885 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:27:01.911 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:01.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:01.912 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:27:02.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:01 smithi032 ceph-mon[20004]: 5.6 scrub ok 2024-08-19T14:27:02.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:01 smithi032 ceph-mon[20004]: pgmap v1087: 57 pgs: 1 active+clean+scrubbing, 56 active+clean; 4.5 GiB data, 20 GiB used, 785 GiB / 805 GiB avail 2024-08-19T14:27:02.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:01 smithi032 ceph-mon[20004]: 5.11 scrub ok 2024-08-19T14:27:02.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:01 smithi032 ceph-mon[20004]: 1.c scrub starts 2024-08-19T14:27:02.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:01 smithi032 ceph-mon[20004]: 1.c scrub ok 2024-08-19T14:27:02.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:01 smithi032 ceph-mon[20004]: 1.12 scrub starts 2024-08-19T14:27:02.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:01 smithi032 ceph-mon[20004]: 1.12 scrub ok 2024-08-19T14:27:02.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:01 smithi032 ceph-mon[20004]: 1.1a scrub starts 2024-08-19T14:27:02.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:01 smithi032 ceph-mon[20004]: 1.1a scrub ok 2024-08-19T14:27:02.012 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:27:02.026 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:27:02.026 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:27:02.040 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:02.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:02.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:27:02.142 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:27:02.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:02.178 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:02.178 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:27:02.279 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:27:02.306 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:02.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:02.307 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:27:02.408 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:27:02.439 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:02.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:02.440 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:27:02.540 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:27:02.569 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:02.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:02.570 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:27:02.671 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:27:02.702 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:02.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:02.704 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:27:02.805 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:27:02.828 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:02.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:02.829 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:27:02.931 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:27:02.958 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:02.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:02.959 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:27:03.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:02 smithi032 ceph-mon[20004]: from='client.16353 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:27:03.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:02 smithi032 ceph-mon[20004]: 5.b scrub ok 2024-08-19T14:27:03.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:02 smithi032 ceph-mon[20004]: 5.d deep-scrub starts 2024-08-19T14:27:03.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:02 smithi032 ceph-mon[20004]: 1.d scrub starts 2024-08-19T14:27:03.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:02 smithi032 ceph-mon[20004]: 1.d scrub ok 2024-08-19T14:27:03.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:02 smithi045 ceph-mon[20143]: from='client.16353 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:27:03.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:02 smithi045 ceph-mon[20143]: 5.b scrub ok 2024-08-19T14:27:03.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:02 smithi045 ceph-mon[20143]: 5.d deep-scrub starts 2024-08-19T14:27:03.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:02 smithi045 ceph-mon[20143]: 1.d scrub starts 2024-08-19T14:27:03.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:02 smithi045 ceph-mon[20143]: 1.d scrub ok 2024-08-19T14:27:03.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:02 smithi031 ceph-mon[16336]: from='client.16353 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:27:03.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:02 smithi031 ceph-mon[16336]: 5.b scrub ok 2024-08-19T14:27:03.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:02 smithi031 ceph-mon[16336]: 5.d deep-scrub starts 2024-08-19T14:27:03.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:02 smithi031 ceph-mon[16336]: 1.d scrub starts 2024-08-19T14:27:03.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:02 smithi031 ceph-mon[16336]: 1.d scrub ok 2024-08-19T14:27:03.061 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:27:03.087 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:03.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:03.087 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:27:03.096 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":1087,"stamp":"2024-08-19T14:26:59.992109+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4809130552,"num_objects":73377,"num_object_clones":0,"num_object_copies":220131,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73377,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":124254,"num_write_kb":5317217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19441,"num_bytes_recovered":1274085376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":223356,"ondisk_log_size":223356,"up":171,"acting":171,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":178,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":20697168,"kb_used_data":15213232,"kb_used_omap":14,"kb_used_meta":5483825,"kb_avail":823009200,"statfs":{"total":863955320832,"available":842761420800,"internally_reserved":0,"allocated":15578349568,"data_stored":15559088695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14353,"internal_metadata":5615437807},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,4,3,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-452296704,"num_objects":-6901,"num_object_clones":0,"num_object_copies":-20703,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6901,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-13641,"num_write_kb":-953056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8375,"num_bytes_recovered":-548864000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001369"},"pg_stats":[{"pgid":"5.4","version":"473'7897","reported_seq":8060,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-08-19T14:26:56.862111+0000","last_change":"2024-08-19T14:25:31.443869+0000","last_active":"2024-08-19T14:26:56.862111+0000","last_peered":"2024-08-19T14:26:56.862111+0000","last_clean":"2024-08-19T14:26:56.862111+0000","last_became_active":"2024-08-19T14:25:29.845148+0000","last_became_peered":"2024-08-19T14:25:29.845148+0000","last_unstale":"2024-08-19T14:26:56.862111+0000","last_undegraded":"2024-08-19T14:26:56.862111+0000","last_fullsized":"2024-08-19T14:26:56.862111+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'5445","last_scrub_stamp":"2024-08-19T14:25:31.443758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:25:31.443758+0000","objects_scrubbed":2768,"log_size":7897,"log_dups_size":0,"ondisk_log_size":7897,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:58.360199+0000","scrub_duration":277,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154140672,"num_objects":2352,"num_object_clones":0,"num_object_copies":7056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2452,"num_write_kb":65152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,0],"acting":[5,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"70'139","reported_seq":1181,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-08-19T14:26:55.595496+0000","last_change":"2024-08-19T14:24:48.191443+0000","last_active":"2024-08-19T14:26:55.595496+0000","last_peered":"2024-08-19T14:26:55.595496+0000","last_clean":"2024-08-19T14:26:55.595496+0000","last_became_active":"2024-08-19T14:18:13.807133+0000","last_became_peered":"2024-08-19T14:18:13.807133+0000","last_unstale":"2024-08-19T14:26:55.595496+0000","last_undegraded":"2024-08-19T14:26:55.595496+0000","last_fullsized":"2024-08-19T14:26:55.595496+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:48.191339+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:19:58.072083+0000","last_clean_scrub_stamp":"2024-08-19T14:24:48.191339+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:55.927001+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,9],"acting":[7,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"473'7999","reported_seq":8162,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-08-19T14:26:56.898030+0000","last_change":"2024-08-19T14:25:32.405310+0000","last_active":"2024-08-19T14:26:56.898030+0000","last_peered":"2024-08-19T14:26:56.898030+0000","last_clean":"2024-08-19T14:26:56.898030+0000","last_became_active":"2024-08-19T14:25:29.845782+0000","last_became_peered":"2024-08-19T14:25:29.845782+0000","last_unstale":"2024-08-19T14:26:56.898030+0000","last_undegraded":"2024-08-19T14:26:56.898030+0000","last_fullsized":"2024-08-19T14:26:56.898030+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'5561","last_scrub_stamp":"2024-08-19T14:25:32.405201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:25:32.405201+0000","objects_scrubbed":2780,"log_size":7999,"log_dups_size":0,"ondisk_log_size":7999,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:59.012193+0000","scrub_duration":268,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":151126016,"num_objects":2306,"num_object_clones":0,"num_object_copies":6918,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2438,"num_write_kb":62848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.3","version":"70'139","reported_seq":1079,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-08-19T14:26:55.595240+0000","last_change":"2024-08-19T14:24:42.697899+0000","last_active":"2024-08-19T14:26:55.595240+0000","last_peered":"2024-08-19T14:26:55.595240+0000","last_clean":"2024-08-19T14:26:55.595240+0000","last_became_active":"2024-08-19T14:18:13.804721+0000","last_became_peered":"2024-08-19T14:18:13.804721+0000","last_unstale":"2024-08-19T14:26:55.595240+0000","last_undegraded":"2024-08-19T14:26:55.595240+0000","last_fullsized":"2024-08-19T14:26:55.595240+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:42.697801+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:24:42.697801+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:57.830750+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"473'8022","reported_seq":8150,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-08-19T14:26:56.917651+0000","last_change":"2024-08-19T14:25:29.831288+0000","last_active":"2024-08-19T14:26:56.917651+0000","last_peered":"2024-08-19T14:26:56.917651+0000","last_clean":"2024-08-19T14:26:56.917651+0000","last_became_active":"2024-08-19T14:25:29.824222+0000","last_became_peered":"2024-08-19T14:25:29.824222+0000","last_unstale":"2024-08-19T14:26:56.917651+0000","last_undegraded":"2024-08-19T14:26:56.917651+0000","last_fullsized":"2024-08-19T14:26:56.917651+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:24:43.574783+0000","objects_scrubbed":0,"log_size":8022,"log_dups_size":0,"ondisk_log_size":8022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:59.393882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":152436736,"num_objects":2326,"num_object_clones":0,"num_object_copies":6978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5240,"num_write_kb":242112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,8],"acting":[5,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"70'139","reported_seq":1035,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-08-19T14:26:57.876390+0000","last_change":"2024-08-19T14:26:57.876424+0000","last_active":"2024-08-19T14:26:57.876390+0000","last_peered":"2024-08-19T14:26:57.876390+0000","last_clean":"2024-08-19T14:26:57.876390+0000","last_became_active":"2024-08-19T14:20:50.742354+0000","last_became_peered":"2024-08-19T14:20:50.742354+0000","last_unstale":"2024-08-19T14:26:57.876390+0000","last_undegraded":"2024-08-19T14:26:57.876390+0000","last_fullsized":"2024-08-19T14:26:57.876390+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:26:57.876316+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:19:59.302931+0000","last_clean_scrub_stamp":"2024-08-19T14:26:57.876316+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:00.325583+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.5","version":"473'7731","reported_seq":7893,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-08-19T14:26:56.933786+0000","last_change":"2024-08-19T14:25:31.371771+0000","last_active":"2024-08-19T14:26:56.933786+0000","last_peered":"2024-08-19T14:26:56.933786+0000","last_clean":"2024-08-19T14:26:56.933786+0000","last_became_active":"2024-08-19T14:25:29.826953+0000","last_became_peered":"2024-08-19T14:25:29.826953+0000","last_unstale":"2024-08-19T14:26:56.933786+0000","last_undegraded":"2024-08-19T14:26:56.933786+0000","last_fullsized":"2024-08-19T14:26:56.933786+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":0,"last_scrub":"440'5369","last_scrub_stamp":"2024-08-19T14:25:31.371653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:25:31.371653+0000","objects_scrubbed":2676,"log_size":7731,"log_dups_size":0,"ondisk_log_size":7731,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:58.060691+0000","scrub_duration":281,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":147718144,"num_objects":2254,"num_object_clones":0,"num_object_copies":6762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2362,"num_write_kb":62080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"70'139","reported_seq":1076,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-08-19T14:26:55.592202+0000","last_change":"2024-08-19T14:24:48.179646+0000","last_active":"2024-08-19T14:26:55.592202+0000","last_peered":"2024-08-19T14:26:55.592202+0000","last_clean":"2024-08-19T14:26:55.592202+0000","last_became_active":"2024-08-19T14:18:13.808505+0000","last_became_peered":"2024-08-19T14:18:13.808505+0000","last_unstale":"2024-08-19T14:26:55.592202+0000","last_undegraded":"2024-08-19T14:26:55.592202+0000","last_fullsized":"2024-08-19T14:26:55.592202+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:48.179558+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:24:48.179558+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:56.070802+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,11],"acting":[9,0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.3","version":"473'7735","reported_seq":7897,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-08-19T14:26:56.976787+0000","last_change":"2024-08-19T14:25:32.404120+0000","last_active":"2024-08-19T14:26:56.976787+0000","last_peered":"2024-08-19T14:26:56.976787+0000","last_clean":"2024-08-19T14:26:56.976787+0000","last_became_active":"2024-08-19T14:25:29.851182+0000","last_became_peered":"2024-08-19T14:25:29.851182+0000","last_unstale":"2024-08-19T14:26:56.976787+0000","last_undegraded":"2024-08-19T14:26:56.976787+0000","last_fullsized":"2024-08-19T14:26:56.976787+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":0,"last_scrub":"440'5410","last_scrub_stamp":"2024-08-19T14:25:32.403944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:25:32.403944+0000","objects_scrubbed":2699,"log_size":7735,"log_dups_size":0,"ondisk_log_size":7735,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":152567808,"num_objects":2328,"num_object_clones":0,"num_object_copies":6984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2325,"num_write_kb":62528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"70'139","reported_seq":728,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-08-19T14:26:55.589507+0000","last_change":"2024-08-19T14:24:47.412400+0000","last_active":"2024-08-19T14:26:55.589507+0000","last_peered":"2024-08-19T14:26:55.589507+0000","last_clean":"2024-08-19T14:26:55.589507+0000","last_became_active":"2024-08-19T14:20:28.204384+0000","last_became_peered":"2024-08-19T14:20:28.204384+0000","last_unstale":"2024-08-19T14:26:55.589507+0000","last_undegraded":"2024-08-19T14:26:55.589507+0000","last_fullsized":"2024-08-19T14:26:55.589507+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:47.412291+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:21:46.057624+0000","last_clean_scrub_stamp":"2024-08-19T14:24:47.412291+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:58.263010+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"473'7857","reported_seq":8018,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-08-19T14:26:56.939955+0000","last_change":"2024-08-19T14:26:56.652316+0000","last_active":"2024-08-19T14:26:56.939955+0000","last_peered":"2024-08-19T14:26:56.939955+0000","last_clean":"2024-08-19T14:26:56.939955+0000","last_became_active":"2024-08-19T14:25:29.831300+0000","last_became_peered":"2024-08-19T14:25:29.831300+0000","last_unstale":"2024-08-19T14:26:56.939955+0000","last_undegraded":"2024-08-19T14:26:56.939955+0000","last_fullsized":"2024-08-19T14:26:56.939955+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'7842","last_scrub_stamp":"2024-08-19T14:26:56.652182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:26:56.652182+0000","objects_scrubbed":2335,"log_size":7857,"log_dups_size":0,"ondisk_log_size":7857,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:21.680001+0000","scrub_duration":607,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":151584768,"num_objects":2313,"num_object_clones":0,"num_object_copies":6939,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,8],"acting":[0,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"70'139","reported_seq":1080,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-08-19T14:26:55.594789+0000","last_change":"2024-08-19T14:24:46.222297+0000","last_active":"2024-08-19T14:26:55.594789+0000","last_peered":"2024-08-19T14:26:55.594789+0000","last_clean":"2024-08-19T14:26:55.594789+0000","last_became_active":"2024-08-19T14:18:26.874902+0000","last_became_peered":"2024-08-19T14:18:26.874902+0000","last_unstale":"2024-08-19T14:26:55.594789+0000","last_undegraded":"2024-08-19T14:26:55.594789+0000","last_fullsized":"2024-08-19T14:26:55.594789+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:46.222166+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:27.848870+0000","last_clean_scrub_stamp":"2024-08-19T14:24:46.222166+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:56.070873+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,4],"acting":[9,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.1","version":"473'8116","reported_seq":8280,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-08-19T14:26:57.839224+0000","last_change":"2024-08-19T14:25:32.082493+0000","last_active":"2024-08-19T14:26:57.839224+0000","last_peered":"2024-08-19T14:26:57.839224+0000","last_clean":"2024-08-19T14:26:57.839224+0000","last_became_active":"2024-08-19T14:25:29.820527+0000","last_became_peered":"2024-08-19T14:25:29.820527+0000","last_unstale":"2024-08-19T14:26:57.839224+0000","last_undegraded":"2024-08-19T14:26:57.839224+0000","last_fullsized":"2024-08-19T14:26:57.839224+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'5651","last_scrub_stamp":"2024-08-19T14:25:32.082374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:25:32.082374+0000","objects_scrubbed":2854,"log_size":8116,"log_dups_size":0,"ondisk_log_size":8116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":280,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150929408,"num_objects":2303,"num_object_clones":0,"num_object_copies":6909,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2303,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2465,"num_write_kb":61248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"70'139","reported_seq":899,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-08-19T14:26:54.594076+0000","last_change":"2024-08-19T14:24:46.629045+0000","last_active":"2024-08-19T14:26:54.594076+0000","last_peered":"2024-08-19T14:26:54.594076+0000","last_clean":"2024-08-19T14:26:54.594076+0000","last_became_active":"2024-08-19T14:18:26.881435+0000","last_became_peered":"2024-08-19T14:18:26.881435+0000","last_unstale":"2024-08-19T14:26:54.594076+0000","last_undegraded":"2024-08-19T14:26:54.594076+0000","last_fullsized":"2024-08-19T14:26:54.594076+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:46.628960+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:24:46.628960+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,4],"acting":[11,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.0","version":"473'8149","reported_seq":8312,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-08-19T14:26:58.655139+0000","last_change":"2024-08-19T14:25:32.103509+0000","last_active":"2024-08-19T14:26:58.655139+0000","last_peered":"2024-08-19T14:26:58.655139+0000","last_clean":"2024-08-19T14:26:58.655139+0000","last_became_active":"2024-08-19T14:25:29.850301+0000","last_became_peered":"2024-08-19T14:25:29.850301+0000","last_unstale":"2024-08-19T14:26:58.655139+0000","last_undegraded":"2024-08-19T14:26:58.655139+0000","last_fullsized":"2024-08-19T14:26:58.655139+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'5592","last_scrub_stamp":"2024-08-19T14:25:32.103399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:25:32.103399+0000","objects_scrubbed":2856,"log_size":8149,"log_dups_size":0,"ondisk_log_size":8149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:59.127154+0000","scrub_duration":317,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":147652608,"num_objects":2253,"num_object_clones":0,"num_object_copies":6759,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2253,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2557,"num_write_kb":62528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"70'139","reported_seq":1124,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-08-19T14:26:54.590688+0000","last_change":"2024-08-19T14:24:11.345984+0000","last_active":"2024-08-19T14:26:54.590688+0000","last_peered":"2024-08-19T14:26:54.590688+0000","last_clean":"2024-08-19T14:26:54.590688+0000","last_became_active":"2024-08-19T14:18:13.806013+0000","last_became_peered":"2024-08-19T14:18:13.806013+0000","last_unstale":"2024-08-19T14:26:54.590688+0000","last_undegraded":"2024-08-19T14:26:54.590688+0000","last_fullsized":"2024-08-19T14:26:54.590688+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:11.345873+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:24:11.345873+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.e","version":"473'10486","reported_seq":10607,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-08-19T14:26:58.170870+0000","last_change":"2024-08-19T14:24:44.595518+0000","last_active":"2024-08-19T14:26:58.170870+0000","last_peered":"2024-08-19T14:26:58.170870+0000","last_clean":"2024-08-19T14:26:58.170870+0000","last_became_active":"2024-08-19T14:24:44.595166+0000","last_became_peered":"2024-08-19T14:24:44.595166+0000","last_unstale":"2024-08-19T14:26:58.170870+0000","last_undegraded":"2024-08-19T14:26:58.170870+0000","last_fullsized":"2024-08-19T14:26:58.170870+0000","mapping_epoch":419,"log_start":"424'400","ondisk_log_start":"424'400","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:24:43.574783+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":300023808,"num_objects":4578,"num_object_clones":0,"num_object_copies":13734,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10486,"num_write_kb":482048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,0],"acting":[9,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.a","version":"70'139","reported_seq":1088,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-08-19T14:26:55.591819+0000","last_change":"2024-08-19T14:24:50.196984+0000","last_active":"2024-08-19T14:26:55.591819+0000","last_peered":"2024-08-19T14:26:55.591819+0000","last_clean":"2024-08-19T14:26:55.591819+0000","last_became_active":"2024-08-19T14:18:26.875006+0000","last_became_peered":"2024-08-19T14:18:26.875006+0000","last_unstale":"2024-08-19T14:26:55.591819+0000","last_undegraded":"2024-08-19T14:26:55.591819+0000","last_fullsized":"2024-08-19T14:26:55.591819+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:50.196885+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:28.864870+0000","last_clean_scrub_stamp":"2024-08-19T14:24:50.196885+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:56.070910+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,0],"acting":[9,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.d","version":"472'10450","reported_seq":10570,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-08-19T14:26:55.504951+0000","last_change":"2024-08-19T14:24:44.592424+0000","last_active":"2024-08-19T14:26:55.504951+0000","last_peered":"2024-08-19T14:26:55.504951+0000","last_clean":"2024-08-19T14:26:55.504951+0000","last_became_active":"2024-08-19T14:24:44.592098+0000","last_became_peered":"2024-08-19T14:24:44.592098+0000","last_unstale":"2024-08-19T14:26:55.504951+0000","last_undegraded":"2024-08-19T14:26:55.504951+0000","last_fullsized":"2024-08-19T14:26:55.504951+0000","mapping_epoch":419,"log_start":"424'400","ondisk_log_start":"424'400","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:24:43.574783+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320536600,"num_objects":4892,"num_object_clones":0,"num_object_copies":14676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4892,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10450,"num_write_kb":490881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7],"acting":[11,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.9","version":"70'139","reported_seq":1203,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-08-19T14:26:55.596965+0000","last_change":"2024-08-19T14:25:04.127400+0000","last_active":"2024-08-19T14:26:55.596965+0000","last_peered":"2024-08-19T14:26:55.596965+0000","last_clean":"2024-08-19T14:26:55.596965+0000","last_became_active":"2024-08-19T14:18:26.887657+0000","last_became_peered":"2024-08-19T14:18:26.887657+0000","last_unstale":"2024-08-19T14:26:55.596965+0000","last_undegraded":"2024-08-19T14:26:55.596965+0000","last_fullsized":"2024-08-19T14:26:55.596965+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:25:04.127296+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:25:04.127296+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:55.927032+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"473'10641","reported_seq":10760,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-08-19T14:26:58.660345+0000","last_change":"2024-08-19T14:24:44.595877+0000","last_active":"2024-08-19T14:26:58.660345+0000","last_peered":"2024-08-19T14:26:58.660345+0000","last_clean":"2024-08-19T14:26:58.660345+0000","last_became_active":"2024-08-19T14:24:44.595538+0000","last_became_peered":"2024-08-19T14:24:44.595538+0000","last_unstale":"2024-08-19T14:26:58.660345+0000","last_undegraded":"2024-08-19T14:26:58.660345+0000","last_fullsized":"2024-08-19T14:26:58.660345+0000","mapping_epoch":419,"log_start":"425'600","ondisk_log_start":"425'600","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:24:43.574783+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":293142528,"num_objects":4473,"num_object_clones":0,"num_object_copies":13419,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4473,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10641,"num_write_kb":483648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"70'139","reported_seq":1100,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-08-19T14:26:55.594334+0000","last_change":"2024-08-19T14:24:45.176469+0000","last_active":"2024-08-19T14:26:55.594334+0000","last_peered":"2024-08-19T14:26:55.594334+0000","last_clean":"2024-08-19T14:26:55.594334+0000","last_became_active":"2024-08-19T14:18:26.887277+0000","last_became_peered":"2024-08-19T14:18:26.887277+0000","last_unstale":"2024-08-19T14:26:55.594334+0000","last_undegraded":"2024-08-19T14:26:55.594334+0000","last_fullsized":"2024-08-19T14:26:55.594334+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:45.176400+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:24:45.176400+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:58.360165+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,8],"acting":[5,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.f","version":"473'10248","reported_seq":10369,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-08-19T14:26:56.908129+0000","last_change":"2024-08-19T14:24:44.593761+0000","last_active":"2024-08-19T14:26:56.908129+0000","last_peered":"2024-08-19T14:26:56.908129+0000","last_clean":"2024-08-19T14:26:56.908129+0000","last_became_active":"2024-08-19T14:24:44.593498+0000","last_became_peered":"2024-08-19T14:24:44.593498+0000","last_unstale":"2024-08-19T14:26:56.908129+0000","last_undegraded":"2024-08-19T14:26:56.908129+0000","last_fullsized":"2024-08-19T14:26:56.908129+0000","mapping_epoch":419,"log_start":"423'200","ondisk_log_start":"423'200","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:24:43.574783+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:58.360132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":294649856,"num_objects":4496,"num_object_clones":0,"num_object_copies":13488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4496,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10248,"num_write_kb":471808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,11],"acting":[5,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"70'139","reported_seq":1099,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-08-19T14:26:54.590825+0000","last_change":"2024-08-19T14:24:45.610330+0000","last_active":"2024-08-19T14:26:54.590825+0000","last_peered":"2024-08-19T14:26:54.590825+0000","last_clean":"2024-08-19T14:26:54.590825+0000","last_became_active":"2024-08-19T14:18:26.875179+0000","last_became_peered":"2024-08-19T14:18:26.875179+0000","last_unstale":"2024-08-19T14:26:54.590825+0000","last_undegraded":"2024-08-19T14:26:54.590825+0000","last_fullsized":"2024-08-19T14:26:54.590825+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:45.610239+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:24:45.610239+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.a","version":"472'10298","reported_seq":10416,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-08-19T14:26:55.056194+0000","last_change":"2024-08-19T14:24:44.592347+0000","last_active":"2024-08-19T14:26:55.056194+0000","last_peered":"2024-08-19T14:26:55.056194+0000","last_clean":"2024-08-19T14:26:55.056194+0000","last_became_active":"2024-08-19T14:24:44.592024+0000","last_became_peered":"2024-08-19T14:24:44.592024+0000","last_unstale":"2024-08-19T14:26:55.056194+0000","last_undegraded":"2024-08-19T14:26:55.056194+0000","last_fullsized":"2024-08-19T14:26:55.056194+0000","mapping_epoch":419,"log_start":"423'200","ondisk_log_start":"423'200","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:24:43.574783+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":311558144,"num_objects":4754,"num_object_clones":0,"num_object_copies":14262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4754,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10298,"num_write_kb":481664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,11],"acting":[8,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.e","version":"70'139","reported_seq":755,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-08-19T14:26:55.599309+0000","last_change":"2024-08-19T14:25:04.737969+0000","last_active":"2024-08-19T14:26:55.599309+0000","last_peered":"2024-08-19T14:26:55.599309+0000","last_clean":"2024-08-19T14:26:55.599309+0000","last_became_active":"2024-08-19T14:18:26.884105+0000","last_became_peered":"2024-08-19T14:18:26.884105+0000","last_unstale":"2024-08-19T14:26:55.599309+0000","last_undegraded":"2024-08-19T14:26:55.599309+0000","last_fullsized":"2024-08-19T14:26:55.599309+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:25:04.737897+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:20:15.321945+0000","last_clean_scrub_stamp":"2024-08-19T14:25:04.737897+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.b","version":"473'10772","reported_seq":10893,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-08-19T14:26:58.179234+0000","last_change":"2024-08-19T14:24:44.595612+0000","last_active":"2024-08-19T14:26:58.179234+0000","last_peered":"2024-08-19T14:26:58.179234+0000","last_clean":"2024-08-19T14:26:58.179234+0000","last_became_active":"2024-08-19T14:24:44.595175+0000","last_became_peered":"2024-08-19T14:24:44.595175+0000","last_unstale":"2024-08-19T14:26:58.179234+0000","last_undegraded":"2024-08-19T14:26:58.179234+0000","last_fullsized":"2024-08-19T14:26:58.179234+0000","mapping_epoch":419,"log_start":"426'700","ondisk_log_start":"426'700","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:24:43.574783+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":297926656,"num_objects":4546,"num_object_clones":0,"num_object_copies":13638,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4546,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10772,"num_write_kb":490176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,5],"acting":[9,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.f","version":"70'139","reported_seq":999,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-08-19T14:26:55.604625+0000","last_change":"2024-08-19T14:24:16.076239+0000","last_active":"2024-08-19T14:26:55.604625+0000","last_peered":"2024-08-19T14:26:55.604625+0000","last_clean":"2024-08-19T14:26:55.604625+0000","last_became_active":"2024-08-19T14:18:18.711708+0000","last_became_peered":"2024-08-19T14:18:18.711708+0000","last_unstale":"2024-08-19T14:26:55.604625+0000","last_undegraded":"2024-08-19T14:26:55.604625+0000","last_fullsized":"2024-08-19T14:26:55.604625+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:16.076143+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:24:16.076143+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:58.360077+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.8","version":"472'7789","reported_seq":7950,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-08-19T14:26:55.478815+0000","last_change":"2024-08-19T14:25:32.033723+0000","last_active":"2024-08-19T14:26:55.478815+0000","last_peered":"2024-08-19T14:26:55.478815+0000","last_clean":"2024-08-19T14:26:55.478815+0000","last_became_active":"2024-08-19T14:25:29.849790+0000","last_became_peered":"2024-08-19T14:25:29.849790+0000","last_unstale":"2024-08-19T14:26:55.478815+0000","last_undegraded":"2024-08-19T14:26:55.478815+0000","last_fullsized":"2024-08-19T14:26:55.478815+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":0,"last_scrub":"440'5444","last_scrub_stamp":"2024-08-19T14:25:32.033618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:25:32.033618+0000","objects_scrubbed":2745,"log_size":7789,"log_dups_size":0,"ondisk_log_size":7789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":152043520,"num_objects":2320,"num_object_clones":0,"num_object_copies":6960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2345,"num_write_kb":61440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.c","version":"70'139","reported_seq":860,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-08-19T14:26:55.589356+0000","last_change":"2024-08-19T14:24:13.465173+0000","last_active":"2024-08-19T14:26:55.589356+0000","last_peered":"2024-08-19T14:26:55.589356+0000","last_clean":"2024-08-19T14:26:55.589356+0000","last_became_active":"2024-08-19T14:18:26.872942+0000","last_became_peered":"2024-08-19T14:18:26.872942+0000","last_unstale":"2024-08-19T14:26:55.589356+0000","last_undegraded":"2024-08-19T14:26:55.589356+0000","last_fullsized":"2024-08-19T14:26:55.589356+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:13.463716+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:22:52.164586+0000","last_clean_scrub_stamp":"2024-08-19T14:24:13.463716+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:58.262976+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,9],"acting":[3,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"473'8005","reported_seq":8165,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-08-19T14:26:58.630017+0000","last_change":"2024-08-19T14:25:31.300738+0000","last_active":"2024-08-19T14:26:58.630017+0000","last_peered":"2024-08-19T14:26:58.630017+0000","last_clean":"2024-08-19T14:26:58.630017+0000","last_became_active":"2024-08-19T14:25:29.821239+0000","last_became_peered":"2024-08-19T14:25:29.821239+0000","last_unstale":"2024-08-19T14:26:58.630017+0000","last_undegraded":"2024-08-19T14:26:58.630017+0000","last_fullsized":"2024-08-19T14:26:58.630017+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'5530","last_scrub_stamp":"2024-08-19T14:25:31.300597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:25:31.300597+0000","objects_scrubbed":2843,"log_size":8005,"log_dups_size":0,"ondisk_log_size":8005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":284,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":148111360,"num_objects":2260,"num_object_clones":0,"num_object_copies":6780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2260,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2475,"num_write_kb":60544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,9],"acting":[7,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"70'139","reported_seq":775,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-08-19T14:26:55.597368+0000","last_change":"2024-08-19T14:24:50.276533+0000","last_active":"2024-08-19T14:26:55.597368+0000","last_peered":"2024-08-19T14:26:55.597368+0000","last_clean":"2024-08-19T14:26:55.597368+0000","last_became_active":"2024-08-19T14:18:26.882002+0000","last_became_peered":"2024-08-19T14:18:26.882002+0000","last_unstale":"2024-08-19T14:26:55.597368+0000","last_undegraded":"2024-08-19T14:26:55.597368+0000","last_fullsized":"2024-08-19T14:26:55.597368+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:50.276408+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:24:50.276408+0000","last_clean_scrub_stamp":"2024-08-19T14:24:50.276408+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:55.926959+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"473'8031","reported_seq":16816,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-08-19T14:26:56.900905+0000","last_change":"2024-08-19T14:26:55.980993+0000","last_active":"2024-08-19T14:26:56.900905+0000","last_peered":"2024-08-19T14:26:56.900905+0000","last_clean":"2024-08-19T14:26:56.900905+0000","last_became_active":"2024-08-19T14:26:22.378003+0000","last_became_peered":"2024-08-19T14:26:22.378003+0000","last_unstale":"2024-08-19T14:26:56.900905+0000","last_undegraded":"2024-08-19T14:26:56.900905+0000","last_fullsized":"2024-08-19T14:26:56.900905+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":439,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":5,"last_scrub":"473'8001","last_scrub_stamp":"2024-08-19T14:26:55.980803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:26:55.980803+0000","objects_scrubbed":2365,"log_size":8031,"log_dups_size":0,"ondisk_log_size":8031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:15.747238+0000","scrub_duration":563,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":151977984,"num_objects":2319,"num_object_clones":0,"num_object_copies":6957,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":46,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.12","version":"70'139","reported_seq":1034,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-08-19T14:26:55.594206+0000","last_change":"2024-08-19T14:24:14.035778+0000","last_active":"2024-08-19T14:26:55.594206+0000","last_peered":"2024-08-19T14:26:55.594206+0000","last_clean":"2024-08-19T14:26:55.594206+0000","last_became_active":"2024-08-19T14:18:18.703594+0000","last_became_peered":"2024-08-19T14:18:18.703594+0000","last_unstale":"2024-08-19T14:26:55.594206+0000","last_undegraded":"2024-08-19T14:26:55.594206+0000","last_fullsized":"2024-08-19T14:26:55.594206+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:14.035665+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:24:14.035665+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:58.360114+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.17","version":"473'8126","reported_seq":19165,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-08-19T14:26:58.795797+0000","last_change":"2024-08-19T14:26:36.454840+0000","last_active":"2024-08-19T14:26:58.795797+0000","last_peered":"2024-08-19T14:26:58.795797+0000","last_clean":"2024-08-19T14:26:58.795797+0000","last_became_active":"2024-08-19T14:26:36.445968+0000","last_became_peered":"2024-08-19T14:26:36.445968+0000","last_unstale":"2024-08-19T14:26:58.795797+0000","last_undegraded":"2024-08-19T14:26:58.795797+0000","last_fullsized":"2024-08-19T14:26:58.795797+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":439,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:24:43.574783+0000","objects_scrubbed":0,"log_size":8126,"log_dups_size":0,"ondisk_log_size":8126,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":143261696,"num_objects":2186,"num_object_clones":0,"num_object_copies":6558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5376,"num_write_kb":241984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2726,"num_bytes_recovered":178651136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8,11],"acting":[3,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.13","version":"70'139","reported_seq":1086,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-08-19T14:26:55.595129+0000","last_change":"2024-08-19T14:25:04.742939+0000","last_active":"2024-08-19T14:26:55.595129+0000","last_peered":"2024-08-19T14:26:55.595129+0000","last_clean":"2024-08-19T14:26:55.595129+0000","last_became_active":"2024-08-19T14:18:18.698334+0000","last_became_peered":"2024-08-19T14:18:18.698334+0000","last_unstale":"2024-08-19T14:26:55.595129+0000","last_undegraded":"2024-08-19T14:26:55.595129+0000","last_fullsized":"2024-08-19T14:26:55.595129+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:25:04.742869+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:25:04.742869+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:57.830841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.14","version":"473'7900","reported_seq":16289,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-08-19T14:26:58.832761+0000","last_change":"2024-08-19T14:26:57.705333+0000","last_active":"2024-08-19T14:26:58.832761+0000","last_peered":"2024-08-19T14:26:58.832761+0000","last_clean":"2024-08-19T14:26:58.832761+0000","last_became_active":"2024-08-19T14:26:34.444635+0000","last_became_peered":"2024-08-19T14:26:34.444635+0000","last_unstale":"2024-08-19T14:26:58.832761+0000","last_undegraded":"2024-08-19T14:26:58.832761+0000","last_fullsized":"2024-08-19T14:26:58.832761+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":439,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":5,"last_scrub":"473'7847","last_scrub_stamp":"2024-08-19T14:26:57.705140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:26:57.705140+0000","objects_scrubbed":2241,"log_size":7900,"log_dups_size":0,"ondisk_log_size":7900,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:24.476329+0000","scrub_duration":460,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":142671872,"num_objects":2177,"num_object_clones":0,"num_object_copies":6531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":64,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8,4],"acting":[3,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.10","version":"70'139","reported_seq":1197,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-08-19T14:26:55.594249+0000","last_change":"2024-08-19T14:25:06.143718+0000","last_active":"2024-08-19T14:26:55.594249+0000","last_peered":"2024-08-19T14:26:55.594249+0000","last_clean":"2024-08-19T14:26:55.594249+0000","last_became_active":"2024-08-19T14:18:18.704562+0000","last_became_peered":"2024-08-19T14:18:18.704562+0000","last_unstale":"2024-08-19T14:26:55.594249+0000","last_undegraded":"2024-08-19T14:26:55.594249+0000","last_fullsized":"2024-08-19T14:26:55.594249+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:25:06.143606+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:25:06.143606+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:58.360182+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,8],"acting":[5,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"473'7772","reported_seq":16229,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-08-19T14:26:56.904048+0000","last_change":"2024-08-19T14:26:56.904328+0000","last_active":"2024-08-19T14:26:56.904048+0000","last_peered":"2024-08-19T14:26:56.904048+0000","last_clean":"2024-08-19T14:26:56.904048+0000","last_became_active":"2024-08-19T14:26:33.443677+0000","last_became_peered":"2024-08-19T14:26:33.443677+0000","last_unstale":"2024-08-19T14:26:56.904048+0000","last_undegraded":"2024-08-19T14:26:56.904048+0000","last_fullsized":"2024-08-19T14:26:56.904048+0000","mapping_epoch":464,"log_start":"0'0","ondisk_log_start":"0'0","created":439,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":5,"last_scrub":"473'7772","last_scrub_stamp":"2024-08-19T14:26:56.903963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:26:56.903963+0000","objects_scrubbed":2299,"log_size":7772,"log_dups_size":0,"ondisk_log_size":7772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:13.538703+0000","scrub_duration":532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150142976,"num_objects":2291,"num_object_clones":0,"num_object_copies":6873,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2291,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,8],"acting":[5,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"70'139","reported_seq":746,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-08-19T14:26:55.595953+0000","last_change":"2024-08-19T14:24:50.547549+0000","last_active":"2024-08-19T14:26:55.595953+0000","last_peered":"2024-08-19T14:26:55.595953+0000","last_clean":"2024-08-19T14:26:55.595953+0000","last_became_active":"2024-08-19T14:18:18.697894+0000","last_became_peered":"2024-08-19T14:18:18.697894+0000","last_unstale":"2024-08-19T14:26:55.595953+0000","last_undegraded":"2024-08-19T14:26:55.595953+0000","last_fullsized":"2024-08-19T14:26:55.595953+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:50.547428+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:24:50.547428+0000","last_clean_scrub_stamp":"2024-08-19T14:24:50.547428+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"472'7820","reported_seq":16270,"reported_epoch":472,"state":"active+clean+scrubbing","last_fresh":"2024-08-19T14:26:55.075301+0000","last_change":"2024-08-19T14:26:54.940571+0000","last_active":"2024-08-19T14:26:55.075301+0000","last_peered":"2024-08-19T14:26:55.075301+0000","last_clean":"2024-08-19T14:26:55.075301+0000","last_became_active":"2024-08-19T14:26:16.340194+0000","last_became_peered":"2024-08-19T14:26:16.340194+0000","last_unstale":"2024-08-19T14:26:55.075301+0000","last_undegraded":"2024-08-19T14:26:55.075301+0000","last_fullsized":"2024-08-19T14:26:55.075301+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":439,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:24:43.574783+0000","objects_scrubbed":599,"log_size":7820,"log_dups_size":0,"ondisk_log_size":7820,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155549696,"num_objects":2374,"num_object_clones":0,"num_object_copies":7122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2374,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5114,"num_write_kb":239584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2769,"num_bytes_recovered":181469184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,3],"acting":[8,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"70'139","reported_seq":715,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-08-19T14:26:55.595512+0000","last_change":"2024-08-19T14:24:41.729459+0000","last_active":"2024-08-19T14:26:55.595512+0000","last_peered":"2024-08-19T14:26:55.595512+0000","last_clean":"2024-08-19T14:26:55.595512+0000","last_became_active":"2024-08-19T14:20:28.204724+0000","last_became_peered":"2024-08-19T14:20:28.204724+0000","last_unstale":"2024-08-19T14:26:55.595512+0000","last_undegraded":"2024-08-19T14:26:55.595512+0000","last_fullsized":"2024-08-19T14:26:55.595512+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:41.729377+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:24:41.729377+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:57.830779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"472'7742","reported_seq":11545,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-08-19T14:26:55.042393+0000","last_change":"2024-08-19T14:26:07.300253+0000","last_active":"2024-08-19T14:26:55.042393+0000","last_peered":"2024-08-19T14:26:55.042393+0000","last_clean":"2024-08-19T14:26:55.042393+0000","last_became_active":"2024-08-19T14:26:07.295750+0000","last_became_peered":"2024-08-19T14:26:07.295750+0000","last_unstale":"2024-08-19T14:26:55.042393+0000","last_undegraded":"2024-08-19T14:26:55.042393+0000","last_fullsized":"2024-08-19T14:26:55.042393+0000","mapping_epoch":454,"log_start":"0'0","ondisk_log_start":"0'0","created":439,"last_epoch_clean":455,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:24:43.574783+0000","objects_scrubbed":0,"log_size":7742,"log_dups_size":0,"ondisk_log_size":7742,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:55.903439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":152436736,"num_objects":2326,"num_object_clones":0,"num_object_copies":6978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5066,"num_write_kb":236544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2780,"num_bytes_recovered":182190080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,5],"acting":[8,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.17","version":"70'139","reported_seq":841,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-08-19T14:26:54.593837+0000","last_change":"2024-08-19T14:24:48.428636+0000","last_active":"2024-08-19T14:26:54.593837+0000","last_peered":"2024-08-19T14:26:54.593837+0000","last_clean":"2024-08-19T14:26:54.593837+0000","last_became_active":"2024-08-19T14:18:31.901556+0000","last_became_peered":"2024-08-19T14:18:31.901556+0000","last_unstale":"2024-08-19T14:26:54.593837+0000","last_undegraded":"2024-08-19T14:26:54.593837+0000","last_fullsized":"2024-08-19T14:26:54.593837+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:48.428515+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:24:48.428515+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,0],"acting":[8,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"5.10","version":"473'8094","reported_seq":16619,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-08-19T14:26:58.705088+0000","last_change":"2024-08-19T14:26:35.452358+0000","last_active":"2024-08-19T14:26:58.705088+0000","last_peered":"2024-08-19T14:26:58.705088+0000","last_clean":"2024-08-19T14:26:58.705088+0000","last_became_active":"2024-08-19T14:26:35.443770+0000","last_became_peered":"2024-08-19T14:26:35.443770+0000","last_unstale":"2024-08-19T14:26:58.705088+0000","last_undegraded":"2024-08-19T14:26:58.705088+0000","last_fullsized":"2024-08-19T14:26:58.705088+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":439,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:24:43.574783+0000","objects_scrubbed":0,"log_size":8094,"log_dups_size":0,"ondisk_log_size":8094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":147357696,"num_objects":2249,"num_object_clones":0,"num_object_copies":6747,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2249,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5331,"num_write_kb":242528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2797,"num_bytes_recovered":183304192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,0],"acting":[7,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"70'139","reported_seq":852,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-08-19T14:26:55.595505+0000","last_change":"2024-08-19T14:25:31.977479+0000","last_active":"2024-08-19T14:26:55.595505+0000","last_peered":"2024-08-19T14:26:55.595505+0000","last_clean":"2024-08-19T14:26:55.595505+0000","last_became_active":"2024-08-19T14:18:18.703926+0000","last_became_peered":"2024-08-19T14:18:18.703926+0000","last_unstale":"2024-08-19T14:26:55.595505+0000","last_undegraded":"2024-08-19T14:26:55.595505+0000","last_fullsized":"2024-08-19T14:26:55.595505+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:25:31.977358+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:25:31.977358+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:55.927060+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,1],"acting":[7,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"473'8139","reported_seq":13996,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-08-19T14:26:57.857734+0000","last_change":"2024-08-19T14:26:00.211612+0000","last_active":"2024-08-19T14:26:57.857734+0000","last_peered":"2024-08-19T14:26:57.857734+0000","last_clean":"2024-08-19T14:26:57.857734+0000","last_became_active":"2024-08-19T14:26:00.207101+0000","last_became_peered":"2024-08-19T14:26:00.207101+0000","last_unstale":"2024-08-19T14:26:57.857734+0000","last_undegraded":"2024-08-19T14:26:57.857734+0000","last_fullsized":"2024-08-19T14:26:57.857734+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":439,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:24:43.574783+0000","objects_scrubbed":0,"log_size":8139,"log_dups_size":0,"ondisk_log_size":8139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:59.560360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":148439040,"num_objects":2265,"num_object_clones":0,"num_object_copies":6795,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5353,"num_write_kb":243776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2860,"num_bytes_recovered":187432960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7],"acting":[4,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.15","version":"70'139","reported_seq":739,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-08-19T14:26:56.872640+0000","last_change":"2024-08-19T14:26:56.872678+0000","last_active":"2024-08-19T14:26:56.872640+0000","last_peered":"2024-08-19T14:26:56.872640+0000","last_clean":"2024-08-19T14:26:56.872640+0000","last_became_active":"2024-08-19T14:18:31.912496+0000","last_became_peered":"2024-08-19T14:18:31.912496+0000","last_unstale":"2024-08-19T14:26:56.872640+0000","last_undegraded":"2024-08-19T14:26:56.872640+0000","last_fullsized":"2024-08-19T14:26:56.872640+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:26:56.872544+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:20:09.180923+0000","last_clean_scrub_stamp":"2024-08-19T14:26:56.872544+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:08.677502+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"70'139","reported_seq":717,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-08-19T14:26:55.600456+0000","last_change":"2024-08-19T14:24:44.775890+0000","last_active":"2024-08-19T14:26:55.600456+0000","last_peered":"2024-08-19T14:26:55.600456+0000","last_clean":"2024-08-19T14:26:55.600456+0000","last_became_active":"2024-08-19T14:18:31.901229+0000","last_became_peered":"2024-08-19T14:18:31.901229+0000","last_unstale":"2024-08-19T14:26:55.600456+0000","last_undegraded":"2024-08-19T14:26:55.600456+0000","last_fullsized":"2024-08-19T14:26:55.600456+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:44.775817+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:24:44.775817+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1b","version":"70'139","reported_seq":1067,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-08-19T14:26:54.592109+0000","last_change":"2024-08-19T14:24:47.410923+0000","last_active":"2024-08-19T14:26:54.592109+0000","last_peered":"2024-08-19T14:26:54.592109+0000","last_clean":"2024-08-19T14:26:54.592109+0000","last_became_active":"2024-08-19T14:18:31.901152+0000","last_became_peered":"2024-08-19T14:18:31.901152+0000","last_unstale":"2024-08-19T14:26:54.592109+0000","last_undegraded":"2024-08-19T14:26:54.592109+0000","last_fullsized":"2024-08-19T14:26:54.592109+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:47.410831+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:24:47.410831+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.18","version":"70'139","reported_seq":1113,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-08-19T14:26:55.594146+0000","last_change":"2024-08-19T14:24:40.113681+0000","last_active":"2024-08-19T14:26:55.594146+0000","last_peered":"2024-08-19T14:26:55.594146+0000","last_clean":"2024-08-19T14:26:55.594146+0000","last_became_active":"2024-08-19T14:18:31.902735+0000","last_became_peered":"2024-08-19T14:18:31.902735+0000","last_unstale":"2024-08-19T14:26:55.594146+0000","last_undegraded":"2024-08-19T14:26:55.594146+0000","last_fullsized":"2024-08-19T14:26:55.594146+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:40.113600+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:24:40.113600+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:58.360149+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.19","version":"70'139","reported_seq":1069,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-08-19T14:26:55.595834+0000","last_change":"2024-08-19T14:26:55.039720+0000","last_active":"2024-08-19T14:26:55.595834+0000","last_peered":"2024-08-19T14:26:55.595834+0000","last_clean":"2024-08-19T14:26:55.595834+0000","last_became_active":"2024-08-19T14:18:31.902547+0000","last_became_peered":"2024-08-19T14:18:31.902547+0000","last_unstale":"2024-08-19T14:26:55.595834+0000","last_undegraded":"2024-08-19T14:26:55.595834+0000","last_fullsized":"2024-08-19T14:26:55.595834+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:26:55.039598+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:26:55.039598+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:27:57.212315+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1e","version":"70'139","reported_seq":718,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-08-19T14:26:54.591139+0000","last_change":"2024-08-19T14:24:44.651398+0000","last_active":"2024-08-19T14:26:54.591139+0000","last_peered":"2024-08-19T14:26:54.591139+0000","last_clean":"2024-08-19T14:26:54.591139+0000","last_became_active":"2024-08-19T14:18:31.902546+0000","last_became_peered":"2024-08-19T14:18:31.902546+0000","last_unstale":"2024-08-19T14:26:54.591139+0000","last_undegraded":"2024-08-19T14:26:54.591139+0000","last_fullsized":"2024-08-19T14:26:54.591139+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:44.651299+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:24:44.651299+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.18","version":"473'7779","reported_seq":16168,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-08-19T14:26:57.850157+0000","last_change":"2024-08-19T14:26:36.456444+0000","last_active":"2024-08-19T14:26:57.850157+0000","last_peered":"2024-08-19T14:26:57.850157+0000","last_clean":"2024-08-19T14:26:57.850157+0000","last_became_active":"2024-08-19T14:26:36.445937+0000","last_became_peered":"2024-08-19T14:26:36.445937+0000","last_unstale":"2024-08-19T14:26:57.850157+0000","last_undegraded":"2024-08-19T14:26:57.850157+0000","last_fullsized":"2024-08-19T14:26:57.850157+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":439,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:24:43.574783+0000","objects_scrubbed":0,"log_size":7779,"log_dups_size":0,"ondisk_log_size":7779,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":141983744,"num_objects":2167,"num_object_clones":0,"num_object_copies":6501,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2167,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5091,"num_write_kb":232224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2756,"num_bytes_recovered":180617216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,1],"acting":[4,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1c","version":"70'139","reported_seq":825,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-08-19T14:26:54.592230+0000","last_change":"2024-08-19T14:24:14.461312+0000","last_active":"2024-08-19T14:26:54.592230+0000","last_peered":"2024-08-19T14:26:54.592230+0000","last_clean":"2024-08-19T14:26:54.592230+0000","last_became_active":"2024-08-19T14:18:31.902494+0000","last_became_peered":"2024-08-19T14:18:31.902494+0000","last_unstale":"2024-08-19T14:26:54.592230+0000","last_undegraded":"2024-08-19T14:26:54.592230+0000","last_fullsized":"2024-08-19T14:26:54.592230+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:14.461176+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:24:14.461176+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,11],"acting":[8,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"5.19","version":"473'7949","reported_seq":16340,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-08-19T14:26:58.614828+0000","last_change":"2024-08-19T14:26:15.347698+0000","last_active":"2024-08-19T14:26:58.614828+0000","last_peered":"2024-08-19T14:26:58.614828+0000","last_clean":"2024-08-19T14:26:58.614828+0000","last_became_active":"2024-08-19T14:26:15.340594+0000","last_became_peered":"2024-08-19T14:26:15.340594+0000","last_unstale":"2024-08-19T14:26:58.614828+0000","last_undegraded":"2024-08-19T14:26:58.614828+0000","last_fullsized":"2024-08-19T14:26:58.614828+0000","mapping_epoch":457,"log_start":"0'0","ondisk_log_start":"0'0","created":439,"last_epoch_clean":458,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:24:43.574783+0000","objects_scrubbed":0,"log_size":7949,"log_dups_size":0,"ondisk_log_size":7949,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:58.853123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":148570112,"num_objects":2267,"num_object_clones":0,"num_object_copies":6801,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2267,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5229,"num_write_kb":239872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2753,"num_bytes_recovered":180420608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1d","version":"70'139","reported_seq":741,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-08-19T14:26:55.595444+0000","last_change":"2024-08-19T14:24:22.588391+0000","last_active":"2024-08-19T14:26:55.595444+0000","last_peered":"2024-08-19T14:26:55.595444+0000","last_clean":"2024-08-19T14:26:55.595444+0000","last_became_active":"2024-08-19T14:18:31.901183+0000","last_became_peered":"2024-08-19T14:18:31.901183+0000","last_unstale":"2024-08-19T14:26:55.595444+0000","last_undegraded":"2024-08-19T14:26:55.595444+0000","last_fullsized":"2024-08-19T14:26:55.595444+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:24:22.588266+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:24:22.588266+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:26:57.830714+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,5],"acting":[1,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":26,"stat_sum":{"num_bytes":4808540184,"num_objects":73375,"num_object_clones":0,"num_object_copies":220125,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73375,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":124254,"num_write_kb":5317217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19441,"num_bytes_recovered":1274085376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15372398592,"data_stored":15372386376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":219047,"ondisk_log_size":219047,"up":78,"acting":78,"num_store_stats":9},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1802240,"data_stored":1788032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187643,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2119880,"kb_used_data":1538940,"kb_used_omap":1,"kb_used_meta":580926,"kb_avail":91625272,"statfs":{"total":95995035648,"available":93824278528,"internally_reserved":0,"allocated":1575874560,"data_stored":1573736164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":594868676},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":343597383873,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":74,"seq":317827580101,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2171084,"kb_used_data":1536004,"kb_used_omap":1,"kb_used_meta":635070,"kb_avail":91574068,"statfs":{"total":95995035648,"available":93771845632,"internally_reserved":0,"allocated":1572868096,"data_stored":1570729617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":650312129},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874442,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2522372,"kb_used_data":1995576,"kb_used_omap":1,"kb_used_meta":526782,"kb_avail":91222780,"statfs":{"total":95995035648,"available":93412126720,"internally_reserved":0,"allocated":2043469824,"data_stored":2041335500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":539425221},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":57,"seq":244813136078,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1993560,"kb_used_data":1326544,"kb_used_omap":1,"kb_used_meta":667006,"kb_avail":91751592,"statfs":{"total":95995035648,"available":93953630208,"internally_reserved":0,"allocated":1358381056,"data_stored":1356234953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":683014602},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":206158430417,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692053,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3052572,"kb_used_data":2525776,"kb_used_omap":1,"kb_used_meta":526782,"kb_avail":90692580,"statfs":{"total":95995035648,"available":92869201920,"internally_reserved":0,"allocated":2586394624,"data_stored":2584248521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":539425221},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986394,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2673356,"kb_used_data":2006340,"kb_used_omap":1,"kb_used_meta":667006,"kb_avail":91071796,"statfs":{"total":95995035648,"available":93257519104,"internally_reserved":0,"allocated":2054492160,"data_stored":2052353681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":683014597},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149920,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1477772,"kb_used_data":869060,"kb_used_omap":1,"kb_used_meta":608702,"kb_avail":92267380,"statfs":{"total":95995035648,"available":94481797120,"internally_reserved":0,"allocated":889917440,"data_stored":887778961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":623311303},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411557,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575081,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2000472,"kb_used_data":1301516,"kb_used_omap":1,"kb_used_meta":698942,"kb_avail":91744680,"statfs":{"total":95995035648,"available":93946552320,"internally_reserved":0,"allocated":1332752384,"data_stored":1330610353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":715717061},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738605,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2686100,"kb_used_data":2113476,"kb_used_omap":1,"kb_used_meta":572606,"kb_avail":91059052,"statfs":{"total":95995035648,"available":93244469248,"internally_reserved":0,"allocated":2164199424,"data_stored":2162060945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":586348997},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2122448896,"data_stored":2122448896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1307246592,"data_stored":1307246592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":872546304,"data_stored":872546304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2037710848,"data_stored":2037710848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2557087744,"data_stored":2557083672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1329401856,"data_stored":1329397784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":2032599040,"data_stored":2032599040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1564213248,"data_stored":1564213248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1549144064,"data_stored":1549139992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T14:27:03.099 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-08-19T14:27:03.099 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd pool set .mgr pg_num 61 2024-08-19T14:27:03.188 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:27:03.217 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:03.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:03.218 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:27:03.319 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:27:03.352 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:03.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:03.353 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:27:03.454 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:27:03.484 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:03.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:03.485 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:27:03.586 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:27:03.624 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:03.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:03.625 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:27:03.726 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:27:03.759 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-08-19T14:27:03.760 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:03.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:03.761 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:27:03.812 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:27:03.830 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:27:03.862 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:27:03.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:03.906 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:03.906 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:27:03.945 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:03 smithi031 ceph-mon[16336]: pgmap v1088: 57 pgs: 57 active+clean; 4.3 GiB data, 19 GiB used, 785 GiB / 805 GiB avail 2024-08-19T14:27:03.945 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:03 smithi031 ceph-mon[16336]: 1.1 scrub starts 2024-08-19T14:27:03.945 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:03 smithi031 ceph-mon[16336]: 1.1 scrub ok 2024-08-19T14:27:03.945 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {auth=30,config=2,health=12,kv=56} crc {auth=1205602389,config=2267314355,health=3897446751,kv=4017770840}) 2024-08-19T14:27:03.945 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=2161662259}) 2024-08-19T14:27:03.945 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=519748276}) 2024-08-19T14:27:03.946 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {kv=32,logm=68} crc {kv=3185740101,logm=1184266953}) 2024-08-19T14:27:03.946 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2853547193}) 2024-08-19T14:27:03.946 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3868211782}) 2024-08-19T14:27:03.946 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1639334040}) 2024-08-19T14:27:03.946 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3360795500}) 2024-08-19T14:27:03.946 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3263542888}) 2024-08-19T14:27:03.946 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2951912651}) 2024-08-19T14:27:03.946 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4089840294}) 2024-08-19T14:27:03.946 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=13034349}) 2024-08-19T14:27:03.946 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3283083369}) 2024-08-19T14:27:03.946 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3367908668}) 2024-08-19T14:27:03.946 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3295747179}) 2024-08-19T14:27:03.947 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=26759101}) 2024-08-19T14:27:03.947 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3125916620}) 2024-08-19T14:27:03.947 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3142839186}) 2024-08-19T14:27:03.947 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1296576885}) 2024-08-19T14:27:03.947 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3936105032}) 2024-08-19T14:27:03.947 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2608645930}) 2024-08-19T14:27:03.947 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=833702215}) 2024-08-19T14:27:03.947 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=962397548}) 2024-08-19T14:27:03.947 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3005091088}) 2024-08-19T14:27:03.947 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2758060111}) 2024-08-19T14:27:03.947 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=180344754}) 2024-08-19T14:27:03.947 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2245077880}) 2024-08-19T14:27:03.947 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4256513820}) 2024-08-19T14:27:03.948 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3096440792}) 2024-08-19T14:27:03.948 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=553760775}) 2024-08-19T14:27:03.948 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1976210172}) 2024-08-19T14:27:03.948 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1366921125}) 2024-08-19T14:27:03.948 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2850989307}) 2024-08-19T14:27:03.948 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2000495325}) 2024-08-19T14:27:03.948 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4162635364}) 2024-08-19T14:27:03.948 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3042341206}) 2024-08-19T14:27:03.948 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=867751675}) 2024-08-19T14:27:03.948 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3303051014}) 2024-08-19T14:27:03.948 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1995673585}) 2024-08-19T14:27:03.948 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3500615200}) 2024-08-19T14:27:03.948 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2777995676}) 2024-08-19T14:27:03.948 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=640992681}) 2024-08-19T14:27:03.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2179424205}) 2024-08-19T14:27:03.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1644423568}) 2024-08-19T14:27:03.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4077989558}) 2024-08-19T14:27:03.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=422495003}) 2024-08-19T14:27:03.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=685310709}) 2024-08-19T14:27:03.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=997429390}) 2024-08-19T14:27:03.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=749076766}) 2024-08-19T14:27:03.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1372846600}) 2024-08-19T14:27:03.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1798196142}) 2024-08-19T14:27:03.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=3,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=73} crc {logm=2636660164,mdsmap=2474562478,mgr=2924679688,mgr_command_descs=1300230762,mgr_metadata=482233419,mgrstat=2234033157}) 2024-08-19T14:27:03.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=39,mon_config_key=61} crc {mgrstat=2580231020,mon_config_key=2411234863}) 2024-08-19T14:27:03.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3978470777}) 2024-08-19T14:27:03.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1966259998}) 2024-08-19T14:27:03.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=15,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=67} crc {mon_config_key=1607266857,monmap=3193798142,osd_metadata=3369745613,osd_pg_creating=1962461481,osdmap=3912527381}) 2024-08-19T14:27:03.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=596181280}) 2024-08-19T14:27:03.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=47897565}) 2024-08-19T14:27:03.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2351329221}) 2024-08-19T14:27:03.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2484791984}) 2024-08-19T14:27:03.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1655569532}) 2024-08-19T14:27:03.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3234073969}) 2024-08-19T14:27:03.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3210536547}) 2024-08-19T14:27:03.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2689123211}) 2024-08-19T14:27:03.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=84} crc {osdmap=3052285442}) 2024-08-19T14:27:04.007 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:27:04.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:03 smithi032 ceph-mon[20004]: pgmap v1088: 57 pgs: 57 active+clean; 4.3 GiB data, 19 GiB used, 785 GiB / 805 GiB avail 2024-08-19T14:27:04.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:03 smithi032 ceph-mon[20004]: 1.1 scrub starts 2024-08-19T14:27:04.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:03 smithi032 ceph-mon[20004]: 1.1 scrub ok 2024-08-19T14:27:04.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {auth=30,config=2,health=12,kv=56} crc {auth=1205602389,config=2267314355,health=3897446751,kv=4017770840}) 2024-08-19T14:27:04.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=2161662259}) 2024-08-19T14:27:04.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=519748276}) 2024-08-19T14:27:04.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {kv=32,logm=68} crc {kv=3185740101,logm=1184266953}) 2024-08-19T14:27:04.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2853547193}) 2024-08-19T14:27:04.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3868211782}) 2024-08-19T14:27:04.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1639334040}) 2024-08-19T14:27:04.009 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3360795500}) 2024-08-19T14:27:04.009 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3263542888}) 2024-08-19T14:27:04.009 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2951912651}) 2024-08-19T14:27:04.009 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4089840294}) 2024-08-19T14:27:04.009 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=13034349}) 2024-08-19T14:27:04.009 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3283083369}) 2024-08-19T14:27:04.009 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3367908668}) 2024-08-19T14:27:04.009 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3295747179}) 2024-08-19T14:27:04.009 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=26759101}) 2024-08-19T14:27:04.009 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3125916620}) 2024-08-19T14:27:04.009 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3142839186}) 2024-08-19T14:27:04.009 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1296576885}) 2024-08-19T14:27:04.009 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3936105032}) 2024-08-19T14:27:04.009 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2608645930}) 2024-08-19T14:27:04.010 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=833702215}) 2024-08-19T14:27:04.010 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=962397548}) 2024-08-19T14:27:04.010 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3005091088}) 2024-08-19T14:27:04.010 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2758060111}) 2024-08-19T14:27:04.010 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=180344754}) 2024-08-19T14:27:04.010 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2245077880}) 2024-08-19T14:27:04.010 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4256513820}) 2024-08-19T14:27:04.010 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3096440792}) 2024-08-19T14:27:04.010 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=553760775}) 2024-08-19T14:27:04.010 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1976210172}) 2024-08-19T14:27:04.010 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1366921125}) 2024-08-19T14:27:04.010 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2850989307}) 2024-08-19T14:27:04.011 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2000495325}) 2024-08-19T14:27:04.011 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4162635364}) 2024-08-19T14:27:04.011 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3042341206}) 2024-08-19T14:27:04.011 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=867751675}) 2024-08-19T14:27:04.011 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3303051014}) 2024-08-19T14:27:04.011 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1995673585}) 2024-08-19T14:27:04.011 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3500615200}) 2024-08-19T14:27:04.012 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2777995676}) 2024-08-19T14:27:04.012 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=640992681}) 2024-08-19T14:27:04.012 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2179424205}) 2024-08-19T14:27:04.012 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1644423568}) 2024-08-19T14:27:04.012 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4077989558}) 2024-08-19T14:27:04.012 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=422495003}) 2024-08-19T14:27:04.012 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=685310709}) 2024-08-19T14:27:04.012 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=997429390}) 2024-08-19T14:27:04.012 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=749076766}) 2024-08-19T14:27:04.012 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1372846600}) 2024-08-19T14:27:04.012 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1798196142}) 2024-08-19T14:27:04.012 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=3,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=73} crc {logm=2636660164,mdsmap=2474562478,mgr=2924679688,mgr_command_descs=1300230762,mgr_metadata=482233419,mgrstat=2234033157}) 2024-08-19T14:27:04.012 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=39,mon_config_key=61} crc {mgrstat=2580231020,mon_config_key=2411234863}) 2024-08-19T14:27:04.012 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3978470777}) 2024-08-19T14:27:04.013 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1966259998}) 2024-08-19T14:27:04.013 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=15,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=67} crc {mon_config_key=1607266857,monmap=3193798142,osd_metadata=3369745613,osd_pg_creating=1962461481,osdmap=3912527381}) 2024-08-19T14:27:04.013 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=596181280}) 2024-08-19T14:27:04.013 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=47897565}) 2024-08-19T14:27:04.013 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2351329221}) 2024-08-19T14:27:04.013 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2484791984}) 2024-08-19T14:27:04.013 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1655569532}) 2024-08-19T14:27:04.013 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3234073969}) 2024-08-19T14:27:04.013 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3210536547}) 2024-08-19T14:27:04.013 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2689123211}) 2024-08-19T14:27:04.013 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=84} crc {osdmap=3052285442}) 2024-08-19T14:27:04.040 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:04.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:04.041 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:27:04.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:03 smithi045 ceph-mon[20143]: pgmap v1088: 57 pgs: 57 active+clean; 4.3 GiB data, 19 GiB used, 785 GiB / 805 GiB avail 2024-08-19T14:27:04.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:03 smithi045 ceph-mon[20143]: 1.1 scrub starts 2024-08-19T14:27:04.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:03 smithi045 ceph-mon[20143]: 1.1 scrub ok 2024-08-19T14:27:04.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {auth=30,config=2,health=12,kv=56} crc {auth=1205602389,config=2267314355,health=3897446751,kv=4017770840}) 2024-08-19T14:27:04.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=2161662259}) 2024-08-19T14:27:04.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=519748276}) 2024-08-19T14:27:04.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {kv=32,logm=68} crc {kv=3185740101,logm=1184266953}) 2024-08-19T14:27:04.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2853547193}) 2024-08-19T14:27:04.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3868211782}) 2024-08-19T14:27:04.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1639334040}) 2024-08-19T14:27:04.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3360795500}) 2024-08-19T14:27:04.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3263542888}) 2024-08-19T14:27:04.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2951912651}) 2024-08-19T14:27:04.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4089840294}) 2024-08-19T14:27:04.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=13034349}) 2024-08-19T14:27:04.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3283083369}) 2024-08-19T14:27:04.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3367908668}) 2024-08-19T14:27:04.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3295747179}) 2024-08-19T14:27:04.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=26759101}) 2024-08-19T14:27:04.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3125916620}) 2024-08-19T14:27:04.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3142839186}) 2024-08-19T14:27:04.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1296576885}) 2024-08-19T14:27:04.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3936105032}) 2024-08-19T14:27:04.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2608645930}) 2024-08-19T14:27:04.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=833702215}) 2024-08-19T14:27:04.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=962397548}) 2024-08-19T14:27:04.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3005091088}) 2024-08-19T14:27:04.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2758060111}) 2024-08-19T14:27:04.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=180344754}) 2024-08-19T14:27:04.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2245077880}) 2024-08-19T14:27:04.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4256513820}) 2024-08-19T14:27:04.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3096440792}) 2024-08-19T14:27:04.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=553760775}) 2024-08-19T14:27:04.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1976210172}) 2024-08-19T14:27:04.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1366921125}) 2024-08-19T14:27:04.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2850989307}) 2024-08-19T14:27:04.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2000495325}) 2024-08-19T14:27:04.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4162635364}) 2024-08-19T14:27:04.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3042341206}) 2024-08-19T14:27:04.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=867751675}) 2024-08-19T14:27:04.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3303051014}) 2024-08-19T14:27:04.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1995673585}) 2024-08-19T14:27:04.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3500615200}) 2024-08-19T14:27:04.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2777995676}) 2024-08-19T14:27:04.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=640992681}) 2024-08-19T14:27:04.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2179424205}) 2024-08-19T14:27:04.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1644423568}) 2024-08-19T14:27:04.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4077989558}) 2024-08-19T14:27:04.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=422495003}) 2024-08-19T14:27:04.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=685310709}) 2024-08-19T14:27:04.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=997429390}) 2024-08-19T14:27:04.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=749076766}) 2024-08-19T14:27:04.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1372846600}) 2024-08-19T14:27:04.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1798196142}) 2024-08-19T14:27:04.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=3,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=73} crc {logm=2636660164,mdsmap=2474562478,mgr=2924679688,mgr_command_descs=1300230762,mgr_metadata=482233419,mgrstat=2234033157}) 2024-08-19T14:27:04.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=39,mon_config_key=61} crc {mgrstat=2580231020,mon_config_key=2411234863}) 2024-08-19T14:27:04.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3978470777}) 2024-08-19T14:27:04.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1966259998}) 2024-08-19T14:27:04.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=15,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=67} crc {mon_config_key=1607266857,monmap=3193798142,osd_metadata=3369745613,osd_pg_creating=1962461481,osdmap=3912527381}) 2024-08-19T14:27:04.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=596181280}) 2024-08-19T14:27:04.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=47897565}) 2024-08-19T14:27:04.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2351329221}) 2024-08-19T14:27:04.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2484791984}) 2024-08-19T14:27:04.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1655569532}) 2024-08-19T14:27:04.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3234073969}) 2024-08-19T14:27:04.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3210536547}) 2024-08-19T14:27:04.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2689123211}) 2024-08-19T14:27:04.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=84} crc {osdmap=3052285442}) 2024-08-19T14:27:04.143 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:27:04.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:04.173 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:04.173 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:27:04.274 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:27:04.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:04.301 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:04.301 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:27:04.402 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:27:04.408 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:27:04.410 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:27:04.427 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:04.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:04.428 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:27:04.529 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:27:04.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:04.565 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:04.565 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:27:04.596 INFO:teuthology.orchestra.run.smithi031.stderr:Error EINVAL: specified pg_num 61 < pg_num_max 32 2024-08-19T14:27:04.666 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:27:04.688 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-19T14:27:04.699 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:04.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:04.700 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:27:04.801 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:27:04.831 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:04.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:04.832 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:27:04.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:04 smithi031 ceph-mon[16336]: 5.3 scrub starts 2024-08-19T14:27:04.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:04 smithi031 ceph-mon[16336]: 5.e scrub starts 2024-08-19T14:27:04.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:04 smithi031 ceph-mon[16336]: 1.7 scrub starts 2024-08-19T14:27:04.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:04 smithi031 ceph-mon[16336]: 1.7 scrub ok 2024-08-19T14:27:04.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:04 smithi031 ceph-mon[16336]: 5.3 scrub ok 2024-08-19T14:27:04.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:04 smithi031 ceph-mon[16336]: 1.e scrub starts 2024-08-19T14:27:04.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:04 smithi031 ceph-mon[16336]: 1.e scrub ok 2024-08-19T14:27:04.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:04 smithi031 ceph-mon[16336]: 5.a deep-scrub starts 2024-08-19T14:27:04.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:04 smithi031 ceph-mon[16336]: 1.0 scrub starts 2024-08-19T14:27:04.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:04 smithi031 ceph-mon[16336]: 1.0 scrub ok 2024-08-19T14:27:04.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:04 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1714973117' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "61"}]: dispatch 2024-08-19T14:27:04.933 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:27:04.956 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:04.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:04.957 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:27:05.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:04 smithi032 ceph-mon[20004]: 5.3 scrub starts 2024-08-19T14:27:05.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:04 smithi032 ceph-mon[20004]: 5.e scrub starts 2024-08-19T14:27:05.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:04 smithi032 ceph-mon[20004]: 1.7 scrub starts 2024-08-19T14:27:05.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:04 smithi032 ceph-mon[20004]: 1.7 scrub ok 2024-08-19T14:27:05.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:04 smithi032 ceph-mon[20004]: 5.3 scrub ok 2024-08-19T14:27:05.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:04 smithi032 ceph-mon[20004]: 1.e scrub starts 2024-08-19T14:27:05.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:04 smithi032 ceph-mon[20004]: 1.e scrub ok 2024-08-19T14:27:05.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:04 smithi032 ceph-mon[20004]: 5.a deep-scrub starts 2024-08-19T14:27:05.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:04 smithi032 ceph-mon[20004]: 1.0 scrub starts 2024-08-19T14:27:05.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:04 smithi032 ceph-mon[20004]: 1.0 scrub ok 2024-08-19T14:27:05.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:04 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1714973117' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "61"}]: dispatch 2024-08-19T14:27:05.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:04 smithi045 ceph-mon[20143]: 5.3 scrub starts 2024-08-19T14:27:05.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:04 smithi045 ceph-mon[20143]: 5.e scrub starts 2024-08-19T14:27:05.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:04 smithi045 ceph-mon[20143]: 1.7 scrub starts 2024-08-19T14:27:05.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:04 smithi045 ceph-mon[20143]: 1.7 scrub ok 2024-08-19T14:27:05.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:04 smithi045 ceph-mon[20143]: 5.3 scrub ok 2024-08-19T14:27:05.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:04 smithi045 ceph-mon[20143]: 1.e scrub starts 2024-08-19T14:27:05.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:04 smithi045 ceph-mon[20143]: 1.e scrub ok 2024-08-19T14:27:05.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:04 smithi045 ceph-mon[20143]: 5.a deep-scrub starts 2024-08-19T14:27:05.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:04 smithi045 ceph-mon[20143]: 1.0 scrub starts 2024-08-19T14:27:05.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:04 smithi045 ceph-mon[20143]: 1.0 scrub ok 2024-08-19T14:27:05.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:04 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1714973117' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "61"}]: dispatch 2024-08-19T14:27:05.058 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:27:05.083 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:05.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:05.084 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:27:05.185 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:27:05.210 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:05.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:05.212 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:27:05.313 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:27:05.314 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-19T14:27:05.351 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:05.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:05.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:27:05.453 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:27:05.482 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:05.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:05.483 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:27:05.584 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:27:05.611 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:05.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:05.612 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:27:05.713 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:27:05.743 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:05.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:05.744 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:27:05.845 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:27:05.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:05 smithi031 ceph-mon[16336]: 5.5 scrub starts 2024-08-19T14:27:05.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:05 smithi031 ceph-mon[16336]: pgmap v1089: 57 pgs: 1 active+clean+scrubbing, 56 active+clean; 4.1 GiB data, 18 GiB used, 786 GiB / 805 GiB avail 2024-08-19T14:27:05.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:05 smithi031 ceph-mon[16336]: 5.e scrub ok 2024-08-19T14:27:05.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:05 smithi031 ceph-mon[16336]: 5.5 scrub ok 2024-08-19T14:27:05.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:05 smithi031 ceph-mon[16336]: 1.1e deep-scrub starts 2024-08-19T14:27:05.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:05 smithi031 ceph-mon[16336]: 1.1e deep-scrub ok 2024-08-19T14:27:05.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:05 smithi031 ceph-mon[16336]: 1.3 scrub starts 2024-08-19T14:27:05.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:05 smithi031 ceph-mon[16336]: 1.3 scrub ok 2024-08-19T14:27:05.872 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:05.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:05.873 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:27:05.974 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:27:05.999 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:05.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:06.000 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:27:06.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:05 smithi032 ceph-mon[20004]: 5.5 scrub starts 2024-08-19T14:27:06.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:05 smithi032 ceph-mon[20004]: pgmap v1089: 57 pgs: 1 active+clean+scrubbing, 56 active+clean; 4.1 GiB data, 18 GiB used, 786 GiB / 805 GiB avail 2024-08-19T14:27:06.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:05 smithi032 ceph-mon[20004]: 5.e scrub ok 2024-08-19T14:27:06.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:05 smithi032 ceph-mon[20004]: 5.5 scrub ok 2024-08-19T14:27:06.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:05 smithi032 ceph-mon[20004]: 1.1e deep-scrub starts 2024-08-19T14:27:06.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:05 smithi032 ceph-mon[20004]: 1.1e deep-scrub ok 2024-08-19T14:27:06.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:05 smithi032 ceph-mon[20004]: 1.3 scrub starts 2024-08-19T14:27:06.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:05 smithi032 ceph-mon[20004]: 1.3 scrub ok 2024-08-19T14:27:06.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:05 smithi045 ceph-mon[20143]: 5.5 scrub starts 2024-08-19T14:27:06.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:05 smithi045 ceph-mon[20143]: pgmap v1089: 57 pgs: 1 active+clean+scrubbing, 56 active+clean; 4.1 GiB data, 18 GiB used, 786 GiB / 805 GiB avail 2024-08-19T14:27:06.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:05 smithi045 ceph-mon[20143]: 5.e scrub ok 2024-08-19T14:27:06.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:05 smithi045 ceph-mon[20143]: 5.5 scrub ok 2024-08-19T14:27:06.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:05 smithi045 ceph-mon[20143]: 1.1e deep-scrub starts 2024-08-19T14:27:06.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:05 smithi045 ceph-mon[20143]: 1.1e deep-scrub ok 2024-08-19T14:27:06.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:05 smithi045 ceph-mon[20143]: 1.3 scrub starts 2024-08-19T14:27:06.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:05 smithi045 ceph-mon[20143]: 1.3 scrub ok 2024-08-19T14:27:06.101 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:27:06.125 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:06.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:06.127 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:27:06.228 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:27:06.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:06.255 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:06.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:27:06.356 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:27:06.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:06.391 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:06.391 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:27:06.492 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:27:06.534 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:06.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:06.535 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:27:06.636 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:27:06.664 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:06.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:06.665 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:27:06.766 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:27:06.788 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:06.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:06.789 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:27:06.890 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:27:06.926 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:06.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:06.927 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:27:07.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:06 smithi032 ceph-mon[20004]: 1.6 scrub starts 2024-08-19T14:27:07.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:06 smithi032 ceph-mon[20004]: 1.6 scrub ok 2024-08-19T14:27:07.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:06 smithi032 ceph-mon[20004]: 5.1 deep-scrub starts 2024-08-19T14:27:07.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:06 smithi032 ceph-mon[20004]: 1.9 scrub starts 2024-08-19T14:27:07.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:06 smithi032 ceph-mon[20004]: 1.9 scrub ok 2024-08-19T14:27:07.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:06 smithi032 ceph-mon[20004]: 1.b scrub starts 2024-08-19T14:27:07.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:06 smithi032 ceph-mon[20004]: 1.b scrub ok 2024-08-19T14:27:07.028 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:27:07.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:06 smithi045 ceph-mon[20143]: 1.6 scrub starts 2024-08-19T14:27:07.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:06 smithi045 ceph-mon[20143]: 1.6 scrub ok 2024-08-19T14:27:07.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:06 smithi045 ceph-mon[20143]: 5.1 deep-scrub starts 2024-08-19T14:27:07.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:06 smithi045 ceph-mon[20143]: 1.9 scrub starts 2024-08-19T14:27:07.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:06 smithi045 ceph-mon[20143]: 1.9 scrub ok 2024-08-19T14:27:07.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:06 smithi045 ceph-mon[20143]: 1.b scrub starts 2024-08-19T14:27:07.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:06 smithi045 ceph-mon[20143]: 1.b scrub ok 2024-08-19T14:27:07.061 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:07.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:07.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:27:07.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:06 smithi031 ceph-mon[16336]: 1.6 scrub starts 2024-08-19T14:27:07.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:06 smithi031 ceph-mon[16336]: 1.6 scrub ok 2024-08-19T14:27:07.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:06 smithi031 ceph-mon[16336]: 5.1 deep-scrub starts 2024-08-19T14:27:07.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:06 smithi031 ceph-mon[16336]: 1.9 scrub starts 2024-08-19T14:27:07.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:06 smithi031 ceph-mon[16336]: 1.9 scrub ok 2024-08-19T14:27:07.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:06 smithi031 ceph-mon[16336]: 1.b scrub starts 2024-08-19T14:27:07.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:06 smithi031 ceph-mon[16336]: 1.b scrub ok 2024-08-19T14:27:07.162 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:27:07.198 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:07.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:07.199 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:27:07.300 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:27:07.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:07.338 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:07.338 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:27:07.439 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:27:07.466 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:07.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:07.467 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:27:07.568 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:27:07.592 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:07.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:07.593 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:27:07.694 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:27:07.719 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:07.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:07.721 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:27:07.822 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:27:07.865 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:07.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:07.866 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:27:07.966 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:27:07.988 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:07.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:07.989 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:27:08.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:07 smithi032 ceph-mon[20004]: pgmap v1090: 57 pgs: 2 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 54 active+clean; 4.0 GiB data, 18 GiB used, 786 GiB / 805 GiB avail 2024-08-19T14:27:08.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:07 smithi032 ceph-mon[20004]: 1.a scrub starts 2024-08-19T14:27:08.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:07 smithi032 ceph-mon[20004]: 1.a scrub ok 2024-08-19T14:27:08.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:07 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:27:08.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:07 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:27:08.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:07 smithi032 ceph-mon[20004]: 5.f scrub starts 2024-08-19T14:27:08.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:07 smithi032 ceph-mon[20004]: osdmap e475: 12 total, 12 up, 9 in 2024-08-19T14:27:08.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:07 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1600699635' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:27:08.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:07 smithi045 ceph-mon[20143]: pgmap v1090: 57 pgs: 2 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 54 active+clean; 4.0 GiB data, 18 GiB used, 786 GiB / 805 GiB avail 2024-08-19T14:27:08.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:07 smithi045 ceph-mon[20143]: 1.a scrub starts 2024-08-19T14:27:08.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:07 smithi045 ceph-mon[20143]: 1.a scrub ok 2024-08-19T14:27:08.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:07 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:27:08.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:07 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:27:08.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:07 smithi045 ceph-mon[20143]: 5.f scrub starts 2024-08-19T14:27:08.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:07 smithi045 ceph-mon[20143]: osdmap e475: 12 total, 12 up, 9 in 2024-08-19T14:27:08.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:07 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1600699635' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:27:08.090 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:27:08.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:07 smithi031 ceph-mon[16336]: pgmap v1090: 57 pgs: 2 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 54 active+clean; 4.0 GiB data, 18 GiB used, 786 GiB / 805 GiB avail 2024-08-19T14:27:08.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:07 smithi031 ceph-mon[16336]: 1.a scrub starts 2024-08-19T14:27:08.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:07 smithi031 ceph-mon[16336]: 1.a scrub ok 2024-08-19T14:27:08.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:07 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:27:08.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:07 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:27:08.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:07 smithi031 ceph-mon[16336]: 5.f scrub starts 2024-08-19T14:27:08.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:07 smithi031 ceph-mon[16336]: osdmap e475: 12 total, 12 up, 9 in 2024-08-19T14:27:08.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:07 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1600699635' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:27:08.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:08.131 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:08.131 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:27:08.194 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:27:08.194 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:27:08.194 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:27:08.194 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:27:08.195 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:27:08.195 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:27:08.195 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:27:08.195 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:27:08.195 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:27:08.195 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:27:08.195 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:27:08.196 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:27:08.196 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:27:08.196 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:27:08.196 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:27:08.196 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:27:08.196 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:27:08.196 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:27:08.196 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:27:08.196 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:27:08.196 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:27:08.196 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:27:08.196 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:27:08.197 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:27:08.232 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:27:08.256 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:08.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:08.257 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:27:08.358 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:27:08.380 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:08.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:08.381 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:27:08.482 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:27:08.512 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:08.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:08.513 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:27:08.614 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:27:08.642 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:08.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:08.643 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:27:08.744 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:27:08.770 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:08.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:08.771 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:27:08.872 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:27:08.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:08.895 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:08.895 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:27:08.996 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:27:09.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:08 smithi032 ceph-mon[20004]: 5.f scrub ok 2024-08-19T14:27:09.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:08 smithi032 ceph-mon[20004]: 5.8 deep-scrub starts 2024-08-19T14:27:09.019 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:09.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:09.020 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:27:09.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:08 smithi045 ceph-mon[20143]: 5.f scrub ok 2024-08-19T14:27:09.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:08 smithi045 ceph-mon[20143]: 5.8 deep-scrub starts 2024-08-19T14:27:09.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:08 smithi031 ceph-mon[16336]: 5.f scrub ok 2024-08-19T14:27:09.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:08 smithi031 ceph-mon[16336]: 5.8 deep-scrub starts 2024-08-19T14:27:09.120 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:27:09.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:09.154 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:09.154 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:27:09.255 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:27:09.279 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:09.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:09.280 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:27:09.381 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:27:09.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:09.411 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:09.411 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:27:09.512 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:27:09.543 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:09.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:09.544 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:27:09.549 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:27:09.549 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [ 2024-08-19T14:27:09.549 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:27:09.549 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(osd.4.0:98654 5.1 e475/439 5:837cbff7:::benchmark_data_smithi045_85309_object115833:head v 475'9322, mlcod=475'9306)", 2024-08-19T14:27:09.549 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:27:09.522604+0000", 2024-08-19T14:27:09.549 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.025535068000000001, 2024-08-19T14:27:09.550 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.025567678999999999, 2024-08-19T14:27:09.550 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:27:09.550 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:27:09.550 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-08-19T14:27:09.550 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:27:09.550 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:27:09.550 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:27:09.550 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:27:09.522604+0000", 2024-08-19T14:27:09.550 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:27:09.550 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:27:09.550 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:27:09.550 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:27:09.551 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:27:09.522604+0000", 2024-08-19T14:27:09.551 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:27:09.551 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:27:09.551 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:27:09.551 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:27:09.551 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:27:09.522604+0000", 2024-08-19T14:27:09.551 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.09e-07 2024-08-19T14:27:09.551 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:27:09.551 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:27:09.551 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:27:09.551 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:27:09.522611+0000", 2024-08-19T14:27:09.551 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.686e-06 2024-08-19T14:27:09.551 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:27:09.552 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:27:09.552 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:27:09.552 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:27:09.522624+0000", 2024-08-19T14:27:09.552 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2884000000000001e-05 2024-08-19T14:27:09.552 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:27:09.552 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:27:09.552 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:27:09.552 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:27:09.522631+0000", 2024-08-19T14:27:09.552 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.4050000000000003e-06 2024-08-19T14:27:09.552 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:27:09.552 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:27:09.552 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:27:09.552 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:27:09.522730+0000", 2024-08-19T14:27:09.553 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.8270999999999995e-05 2024-08-19T14:27:09.553 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:27:09.553 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:27:09.553 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:27:09.553 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:27:09.522748+0000", 2024-08-19T14:27:09.553 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.8600000000000001e-05 2024-08-19T14:27:09.553 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:27:09.553 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:27:09.553 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:27:09.553 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:27:09.553 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:27:09.553 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(osd.4.0:98655 5.1 e475/439 5:837dcad2:::benchmark_data_smithi045_85309_object119643:head v 475'9323, mlcod=475'9306)", 2024-08-19T14:27:09.553 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:27:09.524584+0000", 2024-08-19T14:27:09.554 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.023554327999999999, 2024-08-19T14:27:09.554 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.023649645, 2024-08-19T14:27:09.554 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:27:09.554 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:27:09.554 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-08-19T14:27:09.554 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:27:09.554 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:27:09.554 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:27:09.554 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:27:09.524584+0000", 2024-08-19T14:27:09.554 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:27:09.554 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:27:09.554 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:27:09.555 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:27:09.555 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:27:09.524584+0000", 2024-08-19T14:27:09.555 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:27:09.555 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:27:09.555 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:27:09.555 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:27:09.555 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:27:09.524585+0000", 2024-08-19T14:27:09.555 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.026e-06 2024-08-19T14:27:09.555 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:27:09.555 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:27:09.555 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:27:09.555 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:27:09.524593+0000", 2024-08-19T14:27:09.555 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.6799999999999993e-06 2024-08-19T14:27:09.556 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:27:09.556 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:27:09.556 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:27:09.556 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:27:09.524611+0000", 2024-08-19T14:27:09.556 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.8461000000000002e-05 2024-08-19T14:27:09.556 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:27:09.556 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:27:09.556 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:27:09.556 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:27:09.524620+0000", 2024-08-19T14:27:09.556 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.4749999999999993e-06 2024-08-19T14:27:09.556 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:27:09.556 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:27:09.556 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:27:09.557 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:27:09.526660+0000", 2024-08-19T14:27:09.557 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0020396680000000001 2024-08-19T14:27:09.557 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:27:09.557 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:27:09.557 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:27:09.557 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:27:09.526678+0000", 2024-08-19T14:27:09.557 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.8647e-05 2024-08-19T14:27:09.557 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:27:09.557 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:27:09.557 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:27:09.557 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:27:09.557 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:27:09.557 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(osd.4.0:98656 5.1 e475/439 5:837ded08:::benchmark_data_smithi045_85309_object103579:head v 475'9324, mlcod=475'9306)", 2024-08-19T14:27:09.557 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:27:09.528474+0000", 2024-08-19T14:27:09.558 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.019664427000000002, 2024-08-19T14:27:09.558 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.019818334999999999, 2024-08-19T14:27:09.558 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:27:09.558 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:27:09.558 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-08-19T14:27:09.558 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:27:09.558 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:27:09.558 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:27:09.558 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:27:09.528474+0000", 2024-08-19T14:27:09.558 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:27:09.558 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:27:09.558 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:27:09.558 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:27:09.559 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:27:09.528474+0000", 2024-08-19T14:27:09.559 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:27:09.559 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:27:09.559 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:27:09.559 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:27:09.559 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:27:09.528475+0000", 2024-08-19T14:27:09.559 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.95e-07 2024-08-19T14:27:09.559 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:27:09.559 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:27:09.559 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:27:09.559 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:27:09.528483+0000", 2024-08-19T14:27:09.559 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.1219999999999995e-06 2024-08-19T14:27:09.559 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:27:09.559 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:27:09.560 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:27:09.560 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:27:09.528500+0000", 2024-08-19T14:27:09.560 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.643e-05 2024-08-19T14:27:09.560 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:27:09.560 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:27:09.560 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:27:09.560 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:27:09.528508+0000", 2024-08-19T14:27:09.560 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.4460000000000001e-06 2024-08-19T14:27:09.560 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:27:09.560 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:27:09.560 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:27:09.560 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:27:09.529873+0000", 2024-08-19T14:27:09.560 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.001365254 2024-08-19T14:27:09.560 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:27:09.561 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:27:09.561 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:27:09.561 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:27:09.529895+0000", 2024-08-19T14:27:09.561 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.1773e-05 2024-08-19T14:27:09.561 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:27:09.561 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:27:09.561 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:27:09.561 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:27:09.561 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:27:09.561 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(osd.4.0:98657 5.1 e475/439 5:837e3606:::benchmark_data_smithi045_85309_object107481:head v 475'9325, mlcod=475'9306)", 2024-08-19T14:27:09.561 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:27:09.531849+0000", 2024-08-19T14:27:09.561 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.016289673000000001, 2024-08-19T14:27:09.561 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.016498631, 2024-08-19T14:27:09.561 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:27:09.562 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:27:09.562 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-08-19T14:27:09.562 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:27:09.562 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:27:09.562 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:27:09.562 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:27:09.531849+0000", 2024-08-19T14:27:09.562 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:27:09.562 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:27:09.562 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:27:09.562 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:27:09.562 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:27:09.531849+0000", 2024-08-19T14:27:09.562 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:27:09.562 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:27:09.562 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:27:09.563 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:27:09.563 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:27:09.531850+0000", 2024-08-19T14:27:09.563 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3319999999999999e-06 2024-08-19T14:27:09.563 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:27:09.563 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:27:09.563 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:27:09.563 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:27:09.531885+0000", 2024-08-19T14:27:09.563 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.4316000000000001e-05 2024-08-19T14:27:09.563 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:27:09.563 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:27:09.563 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:27:09.563 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:27:09.531903+0000", 2024-08-19T14:27:09.563 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.8070000000000001e-05 2024-08-19T14:27:09.563 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:27:09.564 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:27:09.564 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:27:09.564 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:27:09.531912+0000", 2024-08-19T14:27:09.564 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.3230000000000004e-06 2024-08-19T14:27:09.564 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:27:09.564 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:27:09.564 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:27:09.565 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:27:09.533153+0000", 2024-08-19T14:27:09.565 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0012413109999999999 2024-08-19T14:27:09.565 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:27:09.565 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:27:09.565 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:27:09.565 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:27:09.533177+0000", 2024-08-19T14:27:09.565 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.3952999999999999e-05 2024-08-19T14:27:09.565 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:27:09.565 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:27:09.566 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:27:09.566 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:27:09.566 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:27:09.566 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(osd.4.0:98658 5.1 e475/439 5:837fed25:::benchmark_data_smithi045_85309_object113461:head v 475'9326, mlcod=475'9306)", 2024-08-19T14:27:09.566 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:27:09.534879+0000", 2024-08-19T14:27:09.566 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.013260103000000001, 2024-08-19T14:27:09.566 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.013525056000000001, 2024-08-19T14:27:09.566 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:27:09.566 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:27:09.566 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-08-19T14:27:09.566 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:27:09.566 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:27:09.567 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:27:09.567 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:27:09.534879+0000", 2024-08-19T14:27:09.567 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:27:09.567 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:27:09.567 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:27:09.567 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:27:09.567 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:27:09.534879+0000", 2024-08-19T14:27:09.567 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:27:09.567 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:27:09.567 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:27:09.567 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:27:09.567 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:27:09.534880+0000", 2024-08-19T14:27:09.567 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.22e-06 2024-08-19T14:27:09.567 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:27:09.568 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:27:09.568 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:27:09.568 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:27:09.534892+0000", 2024-08-19T14:27:09.568 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2561000000000001e-05 2024-08-19T14:27:09.568 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:27:09.568 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:27:09.568 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:27:09.568 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:27:09.534909+0000", 2024-08-19T14:27:09.568 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6784999999999999e-05 2024-08-19T14:27:09.568 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:27:09.568 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:27:09.568 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:27:09.568 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:27:09.534917+0000", 2024-08-19T14:27:09.569 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.055e-06 2024-08-19T14:27:09.569 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:27:09.569 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:27:09.569 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:27:09.569 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:27:09.535775+0000", 2024-08-19T14:27:09.569 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00085738800000000005 2024-08-19T14:27:09.569 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:27:09.569 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:27:09.569 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:27:09.569 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:27:09.535794+0000", 2024-08-19T14:27:09.569 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.9448000000000001e-05 2024-08-19T14:27:09.569 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:27:09.569 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:27:09.569 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:27:09.570 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:27:09.570 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:27:09.570 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(osd.4.0:98659 5.1 e475/439 5:8382e460:::benchmark_data_smithi045_85309_object59929:head v 475'9327, mlcod=475'9306)", 2024-08-19T14:27:09.570 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:27:09.538256+0000", 2024-08-19T14:27:09.570 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.0098826090000000005, 2024-08-19T14:27:09.570 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.010201972, 2024-08-19T14:27:09.570 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:27:09.570 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:27:09.570 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-08-19T14:27:09.570 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:27:09.571 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:27:09.571 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:27:09.571 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:27:09.538256+0000", 2024-08-19T14:27:09.571 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:27:09.571 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:27:09.571 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:27:09.571 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:27:09.571 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:27:09.538256+0000", 2024-08-19T14:27:09.572 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:27:09.572 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:27:09.572 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:27:09.572 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:27:09.572 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:27:09.538258+0000", 2024-08-19T14:27:09.572 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.8780000000000001e-06 2024-08-19T14:27:09.572 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:27:09.572 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:27:09.572 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:27:09.573 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:27:09.538271+0000", 2024-08-19T14:27:09.573 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3193000000000001e-05 2024-08-19T14:27:09.573 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:27:09.573 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:27:09.573 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:27:09.573 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:27:09.538298+0000", 2024-08-19T14:27:09.573 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.7236000000000001e-05 2024-08-19T14:27:09.573 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:27:09.573 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:27:09.573 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:27:09.573 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:27:09.538311+0000", 2024-08-19T14:27:09.573 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3062e-05 2024-08-19T14:27:09.574 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:27:09.574 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:27:09.574 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:27:09.574 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:27:09.538799+0000", 2024-08-19T14:27:09.574 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00048716900000000002 2024-08-19T14:27:09.574 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:27:09.574 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:27:09.574 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:27:09.574 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:27:09.538820+0000", 2024-08-19T14:27:09.574 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.1518999999999998e-05 2024-08-19T14:27:09.574 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:27:09.574 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:27:09.575 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:27:09.575 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:27:09.575 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:27:09.575 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(osd.4.0:98660 5.1 e475/439 5:838503b5:::benchmark_data_smithi045_85309_object94867:head v 475'9328, mlcod=475'9306)", 2024-08-19T14:27:09.575 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:27:09.541053+0000", 2024-08-19T14:27:09.575 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.0070856479999999999, 2024-08-19T14:27:09.575 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0074595959999999998, 2024-08-19T14:27:09.575 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:27:09.575 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:27:09.575 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-08-19T14:27:09.575 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:27:09.575 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:27:09.575 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:27:09.576 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:27:09.541053+0000", 2024-08-19T14:27:09.576 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:27:09.576 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:27:09.576 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:27:09.576 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:27:09.576 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:27:09.541053+0000", 2024-08-19T14:27:09.576 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:27:09.576 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:27:09.576 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:27:09.576 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:27:09.576 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:27:09.541054+0000", 2024-08-19T14:27:09.576 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1090000000000001e-06 2024-08-19T14:27:09.576 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:27:09.577 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:27:09.577 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:27:09.577 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:27:09.541063+0000", 2024-08-19T14:27:09.577 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.9900000000000003e-06 2024-08-19T14:27:09.577 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:27:09.577 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:27:09.577 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:27:09.577 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:27:09.541076+0000", 2024-08-19T14:27:09.577 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2486999999999999e-05 2024-08-19T14:27:09.577 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:27:09.577 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:27:09.577 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:27:09.577 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:27:09.541085+0000", 2024-08-19T14:27:09.577 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.9299999999999992e-06 2024-08-19T14:27:09.578 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:27:09.578 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:27:09.578 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:27:09.578 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:27:09.545606+0000", 2024-08-19T14:27:09.578 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0045218760000000002 2024-08-19T14:27:09.578 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:27:09.578 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:27:09.578 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:27:09.578 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:27:09.545622+0000", 2024-08-19T14:27:09.578 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5189e-05 2024-08-19T14:27:09.578 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:27:09.578 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:27:09.578 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:27:09.579 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:27:09.579 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:27:09.579 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(osd.4.0:98661 5.1 e475/439)", 2024-08-19T14:27:09.579 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:27:09.543595+0000", 2024-08-19T14:27:09.579 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.0045440089999999999, 2024-08-19T14:27:09.579 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0049689310000000002, 2024-08-19T14:27:09.579 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:27:09.579 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:27:09.579 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:27:09.579 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:27:09.579 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:27:09.579 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:27:09.579 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:27:09.543595+0000", 2024-08-19T14:27:09.579 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:27:09.580 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:27:09.580 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:27:09.580 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:27:09.580 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:27:09.543595+0000", 2024-08-19T14:27:09.580 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:27:09.580 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:27:09.580 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:27:09.580 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:27:09.580 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:27:09.543597+0000", 2024-08-19T14:27:09.580 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.883e-06 2024-08-19T14:27:09.580 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:27:09.580 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:27:09.580 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:27:09.581 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:27:09.543607+0000", 2024-08-19T14:27:09.581 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0957e-05 2024-08-19T14:27:09.581 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:27:09.581 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:27:09.581 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:27:09.581 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:27:09.543629+0000", 2024-08-19T14:27:09.581 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.1780000000000002e-05 2024-08-19T14:27:09.581 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:27:09.581 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:27:09.581 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:27:09.581 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:27:09.543641+0000", 2024-08-19T14:27:09.581 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1780000000000001e-05 2024-08-19T14:27:09.581 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:27:09.582 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:27:09.582 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:27:09.582 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:27:09.548552+0000", 2024-08-19T14:27:09.582 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0049110350000000002 2024-08-19T14:27:09.582 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:27:09.582 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:27:09.582 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:27:09.582 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:27:09.548567+0000", 2024-08-19T14:27:09.582 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4771e-05 2024-08-19T14:27:09.582 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:27:09.582 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:27:09.582 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:27:09.582 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:27:09.583 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:27:09.583 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(osd.4.0:98662 5.1 e475/439)", 2024-08-19T14:27:09.583 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:27:09.545945+0000", 2024-08-19T14:27:09.583 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.00219404, 2024-08-19T14:27:09.583 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.002672813, 2024-08-19T14:27:09.583 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:27:09.583 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:27:09.583 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "queued for pg", 2024-08-19T14:27:09.583 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:27:09.583 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:27:09.583 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:27:09.583 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:27:09.545945+0000", 2024-08-19T14:27:09.583 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:27:09.584 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:27:09.584 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:27:09.584 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:27:09.584 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:27:09.545945+0000", 2024-08-19T14:27:09.584 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:27:09.584 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:27:09.584 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:27:09.584 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:27:09.584 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:27:09.545945+0000", 2024-08-19T14:27:09.584 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.8400000000000003e-07 2024-08-19T14:27:09.584 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:27:09.584 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:27:09.584 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:27:09.584 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:27:09.545951+0000", 2024-08-19T14:27:09.585 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.6860000000000003e-06 2024-08-19T14:27:09.585 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:27:09.585 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:27:09.585 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:27:09.585 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:27:09.545962+0000", 2024-08-19T14:27:09.585 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1287000000000001e-05 2024-08-19T14:27:09.585 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:27:09.585 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:27:09.585 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:27:09.585 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:27:09.545969+0000", 2024-08-19T14:27:09.585 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.5869999999999998e-06 2024-08-19T14:27:09.585 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:27:09.585 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:27:09.586 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:27:09.586 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:27:09.586 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-08-19T14:27:09.586 INFO:teuthology.orchestra.run.smithi031.stdout: "num_ops": 9 2024-08-19T14:27:09.586 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:27:09.645 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:27:09.677 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:09.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:09.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:27:09.779 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:27:09.808 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:09.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:09.809 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:27:09.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:09 smithi031 ceph-mon[16336]: pgmap v1092: 57 pgs: 3 active+clean+scrubbing+deep, 2 active+clean+scrubbing, 52 active+clean; 3.9 GiB data, 18 GiB used, 787 GiB / 805 GiB avail 2024-08-19T14:27:09.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:09 smithi031 ceph-mon[16336]: 1.18 scrub starts 2024-08-19T14:27:09.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:09 smithi031 ceph-mon[16336]: 1.18 scrub ok 2024-08-19T14:27:09.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:09 smithi031 ceph-mon[16336]: 1.16 scrub starts 2024-08-19T14:27:09.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:09 smithi031 ceph-mon[16336]: 1.16 scrub ok 2024-08-19T14:27:09.910 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:27:09.934 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:09.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:09.935 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:27:10.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:09 smithi032 ceph-mon[20004]: pgmap v1092: 57 pgs: 3 active+clean+scrubbing+deep, 2 active+clean+scrubbing, 52 active+clean; 3.9 GiB data, 18 GiB used, 787 GiB / 805 GiB avail 2024-08-19T14:27:10.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:09 smithi032 ceph-mon[20004]: 1.18 scrub starts 2024-08-19T14:27:10.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:09 smithi032 ceph-mon[20004]: 1.18 scrub ok 2024-08-19T14:27:10.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:09 smithi032 ceph-mon[20004]: 1.16 scrub starts 2024-08-19T14:27:10.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:09 smithi032 ceph-mon[20004]: 1.16 scrub ok 2024-08-19T14:27:10.036 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:27:10.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:09 smithi045 ceph-mon[20143]: pgmap v1092: 57 pgs: 3 active+clean+scrubbing+deep, 2 active+clean+scrubbing, 52 active+clean; 3.9 GiB data, 18 GiB used, 787 GiB / 805 GiB avail 2024-08-19T14:27:10.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:09 smithi045 ceph-mon[20143]: 1.18 scrub starts 2024-08-19T14:27:10.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:09 smithi045 ceph-mon[20143]: 1.18 scrub ok 2024-08-19T14:27:10.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:09 smithi045 ceph-mon[20143]: 1.16 scrub starts 2024-08-19T14:27:10.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:09 smithi045 ceph-mon[20143]: 1.16 scrub ok 2024-08-19T14:27:10.066 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:10.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:10.067 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:27:10.168 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:27:10.204 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:10.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:10.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:27:10.306 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:27:10.315 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 7, 8, 9, 11] out_osds: [10, 6, 2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-08-19T14:27:10.315 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-19T14:27:10.315 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-08-19T14:27:10.316 INFO:tasks.thrashosds.thrasher:Setting osd 9 primary_affinity to 0.000000 2024-08-19T14:27:10.316 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd primary-affinity 9 0 2024-08-19T14:27:10.341 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:10.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:10.343 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:27:10.444 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:27:10.468 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:10.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:10.469 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:27:10.570 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:27:10.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:10.603 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:10.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:27:10.704 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:27:10.735 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:10.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:10.736 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:27:10.837 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:27:10.860 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:10.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:10.860 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:27:10.935 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:27:10.957 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:10 smithi031 ceph-mon[16336]: 1.8 scrub starts 2024-08-19T14:27:10.957 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:10 smithi031 ceph-mon[16336]: 1.8 scrub ok 2024-08-19T14:27:10.958 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-08-19T14:27:10.962 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:27:11.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:10 smithi032 ceph-mon[20004]: 1.8 scrub starts 2024-08-19T14:27:11.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:10 smithi032 ceph-mon[20004]: 1.8 scrub ok 2024-08-19T14:27:11.036 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:11.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:11.037 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:27:11.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:10 smithi045 ceph-mon[20143]: 1.8 scrub starts 2024-08-19T14:27:11.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:10 smithi045 ceph-mon[20143]: 1.8 scrub ok 2024-08-19T14:27:11.138 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:27:11.173 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:11.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:11.176 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:27:11.276 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:27:11.307 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:11.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:11.308 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:27:11.409 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:27:11.440 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:11.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:11.441 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:27:11.542 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:27:11.568 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:11.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:11.569 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:27:11.616 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:27:11.628 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:27:11.669 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:27:11.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:11.713 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:11.713 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:27:11.767 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:11 smithi031 ceph-mon[16336]: pgmap v1093: 57 pgs: 3 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 53 active+clean; 3.8 GiB data, 18 GiB used, 787 GiB / 805 GiB avail 2024-08-19T14:27:11.767 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:11 smithi031 ceph-mon[16336]: 1.10 scrub starts 2024-08-19T14:27:11.767 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:11 smithi031 ceph-mon[16336]: 1.10 scrub ok 2024-08-19T14:27:11.767 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:11 smithi031 ceph-mon[16336]: 5.c scrub starts 2024-08-19T14:27:11.814 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:27:11.844 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:11.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:11.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:27:11.946 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:27:11.971 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:11.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:11.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:27:12.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:11 smithi032 ceph-mon[20004]: pgmap v1093: 57 pgs: 3 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 53 active+clean; 3.8 GiB data, 18 GiB used, 787 GiB / 805 GiB avail 2024-08-19T14:27:12.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:11 smithi032 ceph-mon[20004]: 1.10 scrub starts 2024-08-19T14:27:12.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:11 smithi032 ceph-mon[20004]: 1.10 scrub ok 2024-08-19T14:27:12.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:11 smithi032 ceph-mon[20004]: 5.c scrub starts 2024-08-19T14:27:12.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:11 smithi045 ceph-mon[20143]: pgmap v1093: 57 pgs: 3 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 53 active+clean; 3.8 GiB data, 18 GiB used, 787 GiB / 805 GiB avail 2024-08-19T14:27:12.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:11 smithi045 ceph-mon[20143]: 1.10 scrub starts 2024-08-19T14:27:12.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:11 smithi045 ceph-mon[20143]: 1.10 scrub ok 2024-08-19T14:27:12.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:11 smithi045 ceph-mon[20143]: 5.c scrub starts 2024-08-19T14:27:12.073 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:27:12.101 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:12.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:12.102 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:27:12.159 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:27:12.203 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:27:12.229 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:12.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:12.231 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:27:12.332 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:27:12.371 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:12.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:12.372 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:27:12.473 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:27:12.503 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:12.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:12.504 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:27:12.605 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:27:12.632 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:12.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:12.633 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:27:12.688 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is set 2024-08-19T14:27:12.734 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:27:12.761 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:12.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:12.762 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:27:12.862 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:27:12.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:12 smithi031 ceph-mon[16336]: 5.c scrub ok 2024-08-19T14:27:12.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:12 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3506983481' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:27:12.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:12 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:27:12.894 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:12.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:12.895 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:27:12.996 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:27:13.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:12 smithi032 ceph-mon[20004]: 5.c scrub ok 2024-08-19T14:27:13.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:12 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3506983481' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:27:13.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:12 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:27:13.022 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:13.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:13.023 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:27:13.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:12 smithi045 ceph-mon[20143]: 5.c scrub ok 2024-08-19T14:27:13.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:12 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3506983481' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:27:13.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:12 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:27:13.124 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:27:13.153 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:13.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:13.154 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:27:13.255 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:27:13.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:13.285 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:13.285 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:27:13.386 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:27:13.412 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:13.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:13.413 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:27:13.514 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:27:13.543 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:13.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:13.544 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:27:13.645 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:27:13.672 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:13.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:13.674 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:27:13.775 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:27:13.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:13.805 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:13.805 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:27:13.839 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:13 smithi031 ceph-mon[16336]: pgmap v1094: 57 pgs: 4 active+clean+scrubbing+deep, 53 active+clean; 3.8 GiB data, 17 GiB used, 787 GiB / 805 GiB avail 2024-08-19T14:27:13.839 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:13 smithi031 ceph-mon[16336]: 5.4 scrub starts 2024-08-19T14:27:13.839 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:13 smithi031 ceph-mon[16336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:27:13.839 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:13 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:27:13.840 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:13 smithi031 ceph-mon[16336]: 5.4 scrub ok 2024-08-19T14:27:13.840 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:13 smithi031 ceph-mon[16336]: osdmap e476: 12 total, 12 up, 9 in 2024-08-19T14:27:13.852 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:27:13.852 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-08-19T14:27:13.853 INFO:teuthology.orchestra.run.smithi031.stdout: "complaint_time": 30, 2024-08-19T14:27:13.853 INFO:teuthology.orchestra.run.smithi031.stdout: "num_blocked_ops": 0 2024-08-19T14:27:13.853 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:27:13.905 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:27:13.937 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:13.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:13.938 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:27:14.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:13 smithi032 ceph-mon[20004]: pgmap v1094: 57 pgs: 4 active+clean+scrubbing+deep, 53 active+clean; 3.8 GiB data, 17 GiB used, 787 GiB / 805 GiB avail 2024-08-19T14:27:14.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:13 smithi032 ceph-mon[20004]: 5.4 scrub starts 2024-08-19T14:27:14.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:13 smithi032 ceph-mon[20004]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:27:14.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:13 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:27:14.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:13 smithi032 ceph-mon[20004]: 5.4 scrub ok 2024-08-19T14:27:14.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:13 smithi032 ceph-mon[20004]: osdmap e476: 12 total, 12 up, 9 in 2024-08-19T14:27:14.039 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:27:14.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:13 smithi045 ceph-mon[20143]: pgmap v1094: 57 pgs: 4 active+clean+scrubbing+deep, 53 active+clean; 3.8 GiB data, 17 GiB used, 787 GiB / 805 GiB avail 2024-08-19T14:27:14.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:13 smithi045 ceph-mon[20143]: 5.4 scrub starts 2024-08-19T14:27:14.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:13 smithi045 ceph-mon[20143]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:27:14.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:13 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:27:14.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:13 smithi045 ceph-mon[20143]: 5.4 scrub ok 2024-08-19T14:27:14.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:13 smithi045 ceph-mon[20143]: osdmap e476: 12 total, 12 up, 9 in 2024-08-19T14:27:14.063 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:14.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:14.064 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:27:14.165 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:27:14.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:14.193 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:14.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:27:14.294 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:27:14.331 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:14.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:14.332 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:27:14.433 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:27:14.469 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:14.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:14.470 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:27:14.571 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:27:14.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:14.601 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:14.601 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:27:14.627 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:27:14.681 INFO:teuthology.orchestra.run.smithi031.stderr:set osd.9 primary-affinity to 0 (802) 2024-08-19T14:27:14.701 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:27:14.728 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:14.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:14.729 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:27:14.831 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:27:14.853 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:14.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:14.854 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:27:14.955 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:27:14.976 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:14.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:14.977 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:27:15.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:14 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/883786815' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 9, "weight": 0.0}]: dispatch 2024-08-19T14:27:15.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:14 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/883786815' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 9, "weight": 0.0}]: dispatch 2024-08-19T14:27:15.054 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:14 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/883786815' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 9, "weight": 0.0}]: dispatch 2024-08-19T14:27:15.078 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:27:15.080 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-08-19T14:27:15.104 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:15.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:15.105 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:27:15.206 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:27:15.229 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:15.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:15.230 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:27:15.331 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:27:15.360 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:15.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:15.361 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:27:15.462 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:27:15.485 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:15.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:15.486 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:27:15.587 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:27:15.608 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:15.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:15.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:27:15.710 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:27:15.733 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:15.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:15.735 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:27:15.765 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:27:15.765 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:27:15.797 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:27:15.797 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:27:15.822 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:27:15.822 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:27:15.836 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:27:15.841 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:27:15.841 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:27:15.858 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:27:15.859 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:27:15.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:15.859 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:15.860 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:27:15.877 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:27:15.878 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:27:15.894 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:15 smithi031 ceph-mon[16336]: pgmap v1096: 57 pgs: 4 active+clean+scrubbing+deep, 53 active+clean; 3.7 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 700 op/s 2024-08-19T14:27:15.894 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:15 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/883786815' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 9, "weight": 0.0}]': finished 2024-08-19T14:27:15.894 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:15 smithi031 ceph-mon[16336]: osdmap e477: 12 total, 12 up, 9 in 2024-08-19T14:27:15.895 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:15 smithi031 ceph-mon[16336]: 5.1 deep-scrub ok 2024-08-19T14:27:15.899 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:27:15.900 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:27:15.924 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:27:15.924 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:27:15.955 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:27:15.955 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:27:15.960 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:27:15.976 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:27:15.976 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:27:15.985 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:15.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:15.986 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:27:15.999 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:27:15.999 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:27:16.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:15 smithi032 ceph-mon[20004]: pgmap v1096: 57 pgs: 4 active+clean+scrubbing+deep, 53 active+clean; 3.7 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 700 op/s 2024-08-19T14:27:16.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:15 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/883786815' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 9, "weight": 0.0}]': finished 2024-08-19T14:27:16.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:15 smithi032 ceph-mon[20004]: osdmap e477: 12 total, 12 up, 9 in 2024-08-19T14:27:16.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:15 smithi032 ceph-mon[20004]: 5.1 deep-scrub ok 2024-08-19T14:27:16.021 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:27:16.022 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:27:16.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:15 smithi045 ceph-mon[20143]: pgmap v1096: 57 pgs: 4 active+clean+scrubbing+deep, 53 active+clean; 3.7 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 700 op/s 2024-08-19T14:27:16.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:15 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/883786815' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 9, "weight": 0.0}]': finished 2024-08-19T14:27:16.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:15 smithi045 ceph-mon[20143]: osdmap e477: 12 total, 12 up, 9 in 2024-08-19T14:27:16.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:15 smithi045 ceph-mon[20143]: 5.1 deep-scrub ok 2024-08-19T14:27:16.088 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:27:16.111 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:16.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:16.112 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:27:16.213 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:27:16.235 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:16.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:16.237 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:27:16.337 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:27:16.365 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:16.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:16.366 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:27:16.432 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-19T14:27:16.466 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:27:16.491 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:16.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:16.492 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:27:16.593 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:27:16.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:16.627 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:16.627 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:27:16.728 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:27:16.755 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:16.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:16.756 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:27:16.857 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:27:16.883 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:16.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:16.884 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:27:16.984 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:16 smithi031 ceph-mon[16336]: osdmap e478: 12 total, 12 up, 9 in 2024-08-19T14:27:16.984 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:16 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1401550745' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:27:16.984 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:16 smithi031 ceph-mon[16336]: 5.8 deep-scrub ok 2024-08-19T14:27:16.985 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:27:17.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:16 smithi032 ceph-mon[20004]: osdmap e478: 12 total, 12 up, 9 in 2024-08-19T14:27:17.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:16 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1401550745' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:27:17.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:16 smithi032 ceph-mon[20004]: 5.8 deep-scrub ok 2024-08-19T14:27:17.014 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:17.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:17.015 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:27:17.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:16 smithi045 ceph-mon[20143]: osdmap e478: 12 total, 12 up, 9 in 2024-08-19T14:27:17.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:16 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1401550745' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:27:17.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:16 smithi045 ceph-mon[20143]: 5.8 deep-scrub ok 2024-08-19T14:27:17.116 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:27:17.136 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:27:17.140 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:17.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:17.141 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:27:17.171 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:27:17.242 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:27:17.275 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:17.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:17.276 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:27:17.377 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:27:17.406 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:17.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:17.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:27:17.508 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:27:17.531 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:17.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:17.532 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:27:17.633 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:27:17.666 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:17.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:17.667 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:27:17.768 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:27:17.798 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:17.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:17.799 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:27:17.832 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:17 smithi031 ceph-mon[16336]: pgmap v1099: 57 pgs: 2 peering, 4 active+clean+scrubbing+deep, 51 active+clean; 3.6 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 341 B/s rd, 11 KiB/s wr, 561 op/s; 7.8 KiB/s, 0 objects/s recovering 2024-08-19T14:27:17.832 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:17 smithi031 ceph-mon[16336]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-08-19T14:27:17.900 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:27:17.927 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:17.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:17.928 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:27:17.953 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:27:17.953 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:27:18.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:17 smithi032 ceph-mon[20004]: pgmap v1099: 57 pgs: 2 peering, 4 active+clean+scrubbing+deep, 51 active+clean; 3.6 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 341 B/s rd, 11 KiB/s wr, 561 op/s; 7.8 KiB/s, 0 objects/s recovering 2024-08-19T14:27:18.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:17 smithi032 ceph-mon[20004]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-08-19T14:27:18.029 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:27:18.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:17 smithi045 ceph-mon[20143]: pgmap v1099: 57 pgs: 2 peering, 4 active+clean+scrubbing+deep, 51 active+clean; 3.6 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 341 B/s rd, 11 KiB/s wr, 561 op/s; 7.8 KiB/s, 0 objects/s recovering 2024-08-19T14:27:18.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:17 smithi045 ceph-mon[20143]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-08-19T14:27:18.057 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:18.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:18.058 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:27:18.159 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:27:18.183 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:18.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:18.184 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:27:18.285 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:27:18.317 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:18.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:18.319 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:27:18.420 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:27:18.446 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:18.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:18.448 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:27:18.548 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:27:18.574 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:18.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:18.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:27:18.676 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:27:18.701 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is set 2024-08-19T14:27:18.705 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:18.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:18.706 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:27:18.807 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:27:18.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:18.839 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:18.839 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:27:18.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:18 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3547386124' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:27:18.940 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:27:18.969 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:18.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:18.970 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:27:19.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:18 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3547386124' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:27:19.027 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:27:19.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:18 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3547386124' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:27:19.074 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:27:19.085 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-08-19T14:27:19.108 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:19.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:19.109 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:27:19.209 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:27:19.237 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:19.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:19.238 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:27:19.339 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:27:19.366 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:19.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:19.368 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:27:19.468 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:27:19.499 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:19.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:19.500 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:27:19.544 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:27:19.601 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:27:19.631 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:19.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:19.632 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:27:19.733 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:27:19.757 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:19.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:19.758 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:27:19.800 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:27:19.859 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:27:19.882 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:19.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:19.883 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:27:19.951 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:19 smithi031 ceph-mon[16336]: pgmap v1100: 57 pgs: 2 peering, 3 active+clean+scrubbing+deep, 52 active+clean; 3.6 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 2.0 KiB/s rd, 55 KiB/s wr, 533 op/s; 39 KiB/s, 0 objects/s recovering 2024-08-19T14:27:19.952 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:19 smithi031 ceph-mon[16336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:27:19.952 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:19 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3547386124' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:27:19.952 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:19 smithi031 ceph-mon[16336]: osdmap e479: 12 total, 12 up, 9 in 2024-08-19T14:27:19.983 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:27:20.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:19 smithi032 ceph-mon[20004]: pgmap v1100: 57 pgs: 2 peering, 3 active+clean+scrubbing+deep, 52 active+clean; 3.6 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 2.0 KiB/s rd, 55 KiB/s wr, 533 op/s; 39 KiB/s, 0 objects/s recovering 2024-08-19T14:27:20.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:19 smithi032 ceph-mon[20004]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:27:20.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:19 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3547386124' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:27:20.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:19 smithi032 ceph-mon[20004]: osdmap e479: 12 total, 12 up, 9 in 2024-08-19T14:27:20.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:20.032 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:20.032 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:27:20.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:19 smithi045 ceph-mon[20143]: pgmap v1100: 57 pgs: 2 peering, 3 active+clean+scrubbing+deep, 52 active+clean; 3.6 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 2.0 KiB/s rd, 55 KiB/s wr, 533 op/s; 39 KiB/s, 0 objects/s recovering 2024-08-19T14:27:20.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:19 smithi045 ceph-mon[20143]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:27:20.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:19 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3547386124' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:27:20.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:19 smithi045 ceph-mon[20143]: osdmap e479: 12 total, 12 up, 9 in 2024-08-19T14:27:20.133 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:27:20.159 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:20.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:20.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:27:20.190 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:27:20.191 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:27:20.261 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:27:20.298 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:20.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:20.299 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:27:20.400 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:27:20.401 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:27:20.401 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:27:20.428 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:20.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:20.430 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:27:20.430 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:27:20.430 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:27:20.460 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:27:20.461 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:27:20.489 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:27:20.489 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:27:20.517 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:27:20.518 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:27:20.530 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:27:20.546 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:27:20.546 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:27:20.561 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:20.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:20.562 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:27:20.578 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:27:20.578 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:27:20.608 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:27:20.609 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:27:20.640 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:27:20.641 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:27:20.663 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:27:20.672 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:27:20.672 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:27:20.690 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:20.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:20.691 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:27:20.702 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:27:20.702 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:27:20.732 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:27:20.732 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:27:20.746 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:20 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/4278506471' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:27:20.792 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:27:20.822 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:20.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:20.823 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:27:20.924 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:27:20.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:20.953 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:20.953 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:27:21.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:20 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/4278506471' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:27:21.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:20 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/4278506471' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:27:21.054 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:27:21.084 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:21.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:21.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:27:21.176 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-08-19T14:27:21.186 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:27:21.213 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:21.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:21.214 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:27:21.315 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:27:21.341 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:21.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:21.342 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:27:21.380 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 7, 8, 9, 11] out_osds: [10, 6, 2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-08-19T14:27:21.380 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-19T14:27:21.380 INFO:tasks.thrashosds.thrasher:inject_pause on osd.10 2024-08-19T14:27:21.380 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-08-19T14:27:21.381 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-08-19T14:27:21.381 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.10 config set filestore_inject_stall 3 2024-08-19T14:27:21.442 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:27:21.476 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:21.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:21.477 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:27:21.578 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:27:21.607 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:21.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:21.608 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:27:21.625 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:27:21.709 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:27:21.738 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:21.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:21.739 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:27:21.743 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:27:21.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:21 smithi032 ceph-mon[20004]: pgmap v1102: 57 pgs: 2 peering, 3 active+clean+scrubbing+deep, 52 active+clean; 3.5 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 2.0 KiB/s rd, 55 KiB/s wr, 402 op/s; 39 KiB/s, 0 objects/s recovering 2024-08-19T14:27:21.758 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-19T14:27:21.783 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:21 smithi031 ceph-mon[16336]: pgmap v1102: 57 pgs: 2 peering, 3 active+clean+scrubbing+deep, 52 active+clean; 3.5 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 2.0 KiB/s rd, 55 KiB/s wr, 402 op/s; 39 KiB/s, 0 objects/s recovering 2024-08-19T14:27:21.840 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:27:21.862 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:21.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:21.863 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:27:21.964 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:27:21.995 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:21.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:21.996 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:27:22.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:21 smithi045 ceph-mon[20143]: pgmap v1102: 57 pgs: 2 peering, 3 active+clean+scrubbing+deep, 52 active+clean; 3.5 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 2.0 KiB/s rd, 55 KiB/s wr, 402 op/s; 39 KiB/s, 0 objects/s recovering 2024-08-19T14:27:22.078 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:27:22.096 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:27:22.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:22.135 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:22.135 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:27:22.236 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:27:22.262 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:22.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:22.263 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:27:22.330 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:27:22.330 INFO:teuthology.orchestra.run.smithi045.stdout: "success": "" 2024-08-19T14:27:22.330 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:27:22.364 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:27:22.390 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:27:22.390 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-08-19T14:27:22.390 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-08-19T14:27:22.390 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-08-19T14:27:22.390 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:27:22.393 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:22.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:22.394 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:27:22.495 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:27:22.524 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:22.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:22.525 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:27:22.626 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:27:22.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:22.663 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:22.663 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:27:22.764 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:27:22.801 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:22.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:22.803 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:27:22.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:22 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:27:22.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:22 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:27:22.903 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:27:22.944 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:22.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:22.945 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:27:23.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:22 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:27:23.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:22 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:27:23.046 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:27:23.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:22 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:27:23.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:22 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:27:23.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:23.085 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:23.085 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:27:23.186 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:27:23.226 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:23.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:23.227 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:27:23.235 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-08-19T14:27:23.328 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:27:23.358 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:23.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:23.360 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:27:23.461 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:27:23.493 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:23.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:23.494 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:27:23.549 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:27:23.595 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:27:23.619 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:23.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:23.620 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:27:23.686 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:27:23.718 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is unset 2024-08-19T14:27:23.721 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:27:23.744 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:23.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:23.745 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:27:23.846 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:27:23.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:23 smithi031 ceph-mon[16336]: pgmap v1103: 57 pgs: 57 active+clean; 3.4 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 1.6 KiB/s rd, 46 KiB/s wr, 638 op/s; 32 KiB/s, 0 objects/s recovering 2024-08-19T14:27:23.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:23 smithi031 ceph-mon[16336]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-08-19T14:27:23.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:23 smithi031 ceph-mon[16336]: osdmap e480: 12 total, 12 up, 9 in 2024-08-19T14:27:23.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:23 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/218058437' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:27:23.872 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:23.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:23.874 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:27:23.975 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:27:23.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:23 smithi032 ceph-mon[20004]: pgmap v1103: 57 pgs: 57 active+clean; 3.4 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 1.6 KiB/s rd, 46 KiB/s wr, 638 op/s; 32 KiB/s, 0 objects/s recovering 2024-08-19T14:27:23.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:23 smithi032 ceph-mon[20004]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-08-19T14:27:23.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:23 smithi032 ceph-mon[20004]: osdmap e480: 12 total, 12 up, 9 in 2024-08-19T14:27:23.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:23 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/218058437' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:27:23.998 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:23.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:23.999 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:27:24.015 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:23 smithi045 ceph-mon[20143]: pgmap v1103: 57 pgs: 57 active+clean; 3.4 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 1.6 KiB/s rd, 46 KiB/s wr, 638 op/s; 32 KiB/s, 0 objects/s recovering 2024-08-19T14:27:24.015 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:23 smithi045 ceph-mon[20143]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-08-19T14:27:24.015 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:23 smithi045 ceph-mon[20143]: osdmap e480: 12 total, 12 up, 9 in 2024-08-19T14:27:24.015 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:23 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/218058437' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:27:24.100 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:27:24.131 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:24.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:24.132 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:27:24.234 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:27:24.281 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:24.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:24.282 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:27:24.297 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:27:24.298 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-08-19T14:27:24.298 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-08-19T14:27:24.298 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-08-19T14:27:24.298 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.298 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25057.0:63305 3.1a 3:594f84d9:::benchmark_data_smithi045_59973_object63304:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e244)", 2024-08-19T14:27:24.298 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:17:33.888645+0000", 2024-08-19T14:27:24.298 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 590.40421721500002, 2024-08-19T14:27:24.298 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0982461939999999, 2024-08-19T14:27:24.298 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:27:24.298 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:27:24.298 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:27:24.299 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:27:24.299 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25057", 2024-08-19T14:27:24.299 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:27:24.299 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 63305 2024-08-19T14:27:24.299 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.299 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:27:24.299 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.299 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:27:24.299 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:33.888645+0000", 2024-08-19T14:27:24.299 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:24.299 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.300 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.300 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:27:24.300 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:33.888645+0000", 2024-08-19T14:27:24.300 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:24.300 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.300 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.300 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:27:24.300 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:33.888646+0000", 2024-08-19T14:27:24.300 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2550000000000001e-06 2024-08-19T14:27:24.300 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.300 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.301 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:27:24.301 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:33.888765+0000", 2024-08-19T14:27:24.301 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00011885 2024-08-19T14:27:24.301 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.301 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.301 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:27:24.301 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:33.888786+0000", 2024-08-19T14:27:24.301 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0828000000000001e-05 2024-08-19T14:27:24.301 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.301 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.301 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:27:24.301 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:33.888795+0000", 2024-08-19T14:27:24.302 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.1649999999999995e-06 2024-08-19T14:27:24.302 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.302 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.302 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:27:24.302 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:33.996820+0000", 2024-08-19T14:27:24.302 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.108024672 2024-08-19T14:27:24.302 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.302 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.302 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:27:24.302 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:33.996825+0000", 2024-08-19T14:27:24.302 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8149999999999996e-06 2024-08-19T14:27:24.302 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.303 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.303 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:27:24.303 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:34.986807+0000", 2024-08-19T14:27:24.303 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.98998204300000003 2024-08-19T14:27:24.303 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.303 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.303 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:27:24.303 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:34.986891+0000", 2024-08-19T14:27:24.303 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.4566000000000003e-05 2024-08-19T14:27:24.303 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:24.303 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:27:24.303 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:24.303 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.304 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.304 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25057.0:63311 3.1a 3:5f4435a5:::benchmark_data_smithi045_59973_object63310:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e244)", 2024-08-19T14:27:24.304 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:17:33.888830+0000", 2024-08-19T14:27:24.304 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 590.40403218699998, 2024-08-19T14:27:24.304 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.098523232, 2024-08-19T14:27:24.304 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:27:24.304 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:27:24.304 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:27:24.304 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:27:24.304 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25057", 2024-08-19T14:27:24.304 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:27:24.304 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 63311 2024-08-19T14:27:24.305 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.305 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:27:24.305 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.305 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:27:24.305 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:33.888830+0000", 2024-08-19T14:27:24.305 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:24.305 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.305 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.305 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:27:24.305 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:33.888830+0000", 2024-08-19T14:27:24.305 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:24.305 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.305 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.305 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:27:24.306 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:33.888831+0000", 2024-08-19T14:27:24.306 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.99e-07 2024-08-19T14:27:24.306 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.306 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.306 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:27:24.306 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:33.888901+0000", 2024-08-19T14:27:24.306 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9948000000000002e-05 2024-08-19T14:27:24.306 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.306 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.306 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:27:24.306 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:33.888911+0000", 2024-08-19T14:27:24.306 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0679000000000001e-05 2024-08-19T14:27:24.307 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.307 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.307 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:27:24.307 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:33.888918+0000", 2024-08-19T14:27:24.307 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.5880000000000003e-06 2024-08-19T14:27:24.307 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.307 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.307 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:27:24.307 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:33.997358+0000", 2024-08-19T14:27:24.307 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.108439887 2024-08-19T14:27:24.307 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.307 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.308 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:27:24.308 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:33.997362+0000", 2024-08-19T14:27:24.308 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1219999999999997e-06 2024-08-19T14:27:24.308 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.308 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.308 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:27:24.308 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:34.987287+0000", 2024-08-19T14:27:24.308 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.989925481 2024-08-19T14:27:24.308 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.309 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.309 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:27:24.309 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:34.987353+0000", 2024-08-19T14:27:24.309 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.5827999999999994e-05 2024-08-19T14:27:24.309 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:24.309 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:27:24.309 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:24.309 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.309 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.309 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25057.0:64606 3.1b 3.702ddf7b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e246)", 2024-08-19T14:27:24.309 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:17:35.969630+0000", 2024-08-19T14:27:24.309 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 588.32323248, 2024-08-19T14:27:24.310 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.99329972499999997, 2024-08-19T14:27:24.310 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:27:24.310 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:27:24.310 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:27:24.310 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:27:24.310 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25057", 2024-08-19T14:27:24.310 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:27:24.310 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 64606 2024-08-19T14:27:24.310 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.310 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:27:24.310 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.310 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:27:24.311 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.969630+0000", 2024-08-19T14:27:24.311 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:24.311 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.311 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.311 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:27:24.311 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.969630+0000", 2024-08-19T14:27:24.311 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:24.311 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.311 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.311 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:27:24.311 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.969632+0000", 2024-08-19T14:27:24.311 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9069999999999999e-06 2024-08-19T14:27:24.312 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.312 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.312 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:27:24.312 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.969667+0000", 2024-08-19T14:27:24.312 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5187e-05 2024-08-19T14:27:24.312 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.312 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.312 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:27:24.312 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.969678+0000", 2024-08-19T14:27:24.312 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0849000000000001e-05 2024-08-19T14:27:24.312 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.312 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.312 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:27:24.313 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.969685+0000", 2024-08-19T14:27:24.313 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5039999999999996e-06 2024-08-19T14:27:24.313 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.313 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.313 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:27:24.313 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.969734+0000", 2024-08-19T14:27:24.313 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8690000000000003e-05 2024-08-19T14:27:24.313 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.313 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.313 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:27:24.313 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.969744+0000", 2024-08-19T14:27:24.313 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0183999999999999e-05 2024-08-19T14:27:24.313 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.313 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.314 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:27:24.314 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:36.962899+0000", 2024-08-19T14:27:24.314 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.99315520700000004 2024-08-19T14:27:24.314 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.314 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.314 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:27:24.314 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:36.962929+0000", 2024-08-19T14:27:24.314 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0196999999999999e-05 2024-08-19T14:27:24.314 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:24.314 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:27:24.314 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:24.314 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.314 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.315 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25057.0:64610 3.1b 3.810f60bb (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e246)", 2024-08-19T14:27:24.315 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:17:35.977398+0000", 2024-08-19T14:27:24.315 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 588.31546375599999, 2024-08-19T14:27:24.315 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.98556503799999995, 2024-08-19T14:27:24.315 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:27:24.315 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:27:24.315 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:27:24.315 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:27:24.315 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25057", 2024-08-19T14:27:24.315 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:27:24.315 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 64610 2024-08-19T14:27:24.315 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.315 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:27:24.316 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.316 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:27:24.316 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.977398+0000", 2024-08-19T14:27:24.316 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:24.316 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.316 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.316 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:27:24.316 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.977398+0000", 2024-08-19T14:27:24.316 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:24.316 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.316 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.316 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:27:24.316 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.977400+0000", 2024-08-19T14:27:24.316 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.469e-06 2024-08-19T14:27:24.317 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.317 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.317 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:27:24.317 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.977457+0000", 2024-08-19T14:27:24.317 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7043999999999998e-05 2024-08-19T14:27:24.317 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.317 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.317 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:27:24.317 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.977465+0000", 2024-08-19T14:27:24.317 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.2320000000000001e-06 2024-08-19T14:27:24.317 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.317 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.317 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:27:24.317 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.977470+0000", 2024-08-19T14:27:24.318 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5759999999999999e-06 2024-08-19T14:27:24.318 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.318 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.318 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:27:24.318 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.978418+0000", 2024-08-19T14:27:24.318 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00094818599999999995 2024-08-19T14:27:24.318 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.318 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.318 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:27:24.318 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.978424+0000", 2024-08-19T14:27:24.318 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2979999999999997e-06 2024-08-19T14:27:24.318 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.318 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.319 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:27:24.319 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:36.962956+0000", 2024-08-19T14:27:24.319 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.98453159300000004 2024-08-19T14:27:24.319 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.319 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.319 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:27:24.319 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:36.962963+0000", 2024-08-19T14:27:24.319 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6399999999999997e-06 2024-08-19T14:27:24.319 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:24.319 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:27:24.319 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:24.319 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.319 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.319 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25057.0:64616 3.1b 3.cab7ee3b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e246)", 2024-08-19T14:27:24.320 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:17:35.982530+0000", 2024-08-19T14:27:24.320 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 588.31033251300005, 2024-08-19T14:27:24.320 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.98047576599999997, 2024-08-19T14:27:24.320 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:27:24.320 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:27:24.320 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:27:24.320 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:27:24.320 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25057", 2024-08-19T14:27:24.320 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:27:24.320 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 64616 2024-08-19T14:27:24.320 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.320 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:27:24.320 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.320 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:27:24.320 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.982530+0000", 2024-08-19T14:27:24.321 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:24.321 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.321 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.321 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:27:24.321 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.982530+0000", 2024-08-19T14:27:24.321 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:24.321 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.321 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.321 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:27:24.321 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.982531+0000", 2024-08-19T14:27:24.321 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.626e-06 2024-08-19T14:27:24.321 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.321 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.321 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:27:24.322 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.982589+0000", 2024-08-19T14:27:24.322 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8013000000000002e-05 2024-08-19T14:27:24.322 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.322 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.322 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:27:24.322 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.982603+0000", 2024-08-19T14:27:24.322 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.384e-05 2024-08-19T14:27:24.322 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.322 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.322 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:27:24.322 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.982611+0000", 2024-08-19T14:27:24.322 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6329999999999996e-06 2024-08-19T14:27:24.323 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.323 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.323 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:27:24.323 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.982674+0000", 2024-08-19T14:27:24.323 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3575e-05 2024-08-19T14:27:24.323 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.323 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.323 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:27:24.323 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.982686+0000", 2024-08-19T14:27:24.323 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1929999999999999e-05 2024-08-19T14:27:24.323 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.323 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.323 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:27:24.323 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:36.962998+0000", 2024-08-19T14:27:24.324 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.98031178900000004 2024-08-19T14:27:24.324 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.324 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.324 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:27:24.324 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:36.963005+0000", 2024-08-19T14:27:24.324 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3599999999999998e-06 2024-08-19T14:27:24.324 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:24.324 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:27:24.324 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:24.324 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.324 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.324 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25057.0:188779 3.2a 3:54ce2294:::benchmark_data_smithi045_59973_object57600:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-08-19T14:27:24.324 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:19:02.351958+0000", 2024-08-19T14:27:24.325 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 501.940904284, 2024-08-19T14:27:24.325 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.86379963100000001, 2024-08-19T14:27:24.325 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:27:24.325 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:27:24.325 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:27:24.325 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:27:24.325 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25057", 2024-08-19T14:27:24.325 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:27:24.325 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 188779 2024-08-19T14:27:24.325 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.325 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:27:24.325 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.325 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:27:24.325 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:19:02.351958+0000", 2024-08-19T14:27:24.326 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:24.326 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.326 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.326 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:27:24.326 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:19:02.351958+0000", 2024-08-19T14:27:24.326 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:24.326 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.326 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.326 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:27:24.326 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:19:02.351959+0000", 2024-08-19T14:27:24.326 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.3200000000000004e-07 2024-08-19T14:27:24.326 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.326 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.327 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:27:24.327 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:19:02.351962+0000", 2024-08-19T14:27:24.327 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.32e-06 2024-08-19T14:27:24.327 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.327 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.327 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:27:24.327 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:19:02.351970+0000", 2024-08-19T14:27:24.327 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6790000000000005e-06 2024-08-19T14:27:24.327 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.327 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.327 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:27:24.327 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:19:02.351974+0000", 2024-08-19T14:27:24.327 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5979999999999999e-06 2024-08-19T14:27:24.327 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.328 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.328 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:27:24.328 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:19:02.352047+0000", 2024-08-19T14:27:24.328 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2187999999999994e-05 2024-08-19T14:27:24.328 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.329 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.329 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:27:24.329 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:19:02.352057+0000", 2024-08-19T14:27:24.329 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0642000000000001e-05 2024-08-19T14:27:24.329 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.329 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.329 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:27:24.329 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:19:03.210075+0000", 2024-08-19T14:27:24.329 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.85801831699999997 2024-08-19T14:27:24.330 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.330 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.330 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:27:24.330 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:19:03.210203+0000", 2024-08-19T14:27:24.330 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012764300000000001 2024-08-19T14:27:24.330 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.330 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.330 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 1,8", 2024-08-19T14:27:24.330 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:19:03.210318+0000", 2024-08-19T14:27:24.330 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000114954 2024-08-19T14:27:24.330 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.330 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.330 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:24.331 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:19:03.212413+0000", 2024-08-19T14:27:24.331 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0020953249999999999 2024-08-19T14:27:24.331 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.331 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.331 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:24.331 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:19:03.212766+0000", 2024-08-19T14:27:24.331 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00035248899999999999 2024-08-19T14:27:24.331 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.331 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.331 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:27:24.331 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:19:03.215663+0000", 2024-08-19T14:27:24.331 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0028968959999999999 2024-08-19T14:27:24.331 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.332 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.332 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:27:24.332 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:19:03.215714+0000", 2024-08-19T14:27:24.332 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0719e-05 2024-08-19T14:27:24.332 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.332 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.332 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:27:24.332 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:19:03.215758+0000", 2024-08-19T14:27:24.332 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4029000000000001e-05 2024-08-19T14:27:24.332 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:24.332 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:27:24.332 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:24.332 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.333 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.333 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25333.0:149070 4.e 4:77cd6059:::benchmark_data_smithi045_71562_object17895:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e378)", 2024-08-19T14:27:24.333 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:22:24.358621+0000", 2024-08-19T14:27:24.333 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 299.934240778, 2024-08-19T14:27:24.333 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.223820359, 2024-08-19T14:27:24.333 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:27:24.333 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:27:24.333 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:27:24.333 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:27:24.333 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25333", 2024-08-19T14:27:24.333 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:27:24.333 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 149070 2024-08-19T14:27:24.333 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.333 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:27:24.334 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.334 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:27:24.334 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:22:24.358621+0000", 2024-08-19T14:27:24.334 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:24.334 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.334 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.334 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:27:24.334 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:22:24.358621+0000", 2024-08-19T14:27:24.334 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:24.334 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.334 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.334 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:27:24.334 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:22:24.358622+0000", 2024-08-19T14:27:24.335 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.0999999999999997e-07 2024-08-19T14:27:24.335 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.335 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.335 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:27:24.335 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:22:24.358627+0000", 2024-08-19T14:27:24.335 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3760000000000001e-06 2024-08-19T14:27:24.335 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.335 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.335 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:27:24.335 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:22:24.358642+0000", 2024-08-19T14:27:24.335 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5582000000000001e-05 2024-08-19T14:27:24.335 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.336 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.336 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:27:24.336 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:22:24.358650+0000", 2024-08-19T14:27:24.336 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5079999999999999e-06 2024-08-19T14:27:24.336 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.336 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.336 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:27:24.336 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:22:24.362772+0000", 2024-08-19T14:27:24.336 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0041220220000000004 2024-08-19T14:27:24.336 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.336 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.336 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:27:24.336 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:22:24.362918+0000", 2024-08-19T14:27:24.337 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014636200000000001 2024-08-19T14:27:24.337 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.337 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.337 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 1,3", 2024-08-19T14:27:24.337 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:22:24.363057+0000", 2024-08-19T14:27:24.337 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013936800000000001 2024-08-19T14:27:24.337 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.337 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.337 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:27:24.337 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:22:24.367511+0000", 2024-08-19T14:27:24.337 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0044538939999999999 2024-08-19T14:27:24.337 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.337 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.338 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:24.338 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:22:24.379873+0000", 2024-08-19T14:27:24.338 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.01236145 2024-08-19T14:27:24.338 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.338 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.338 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:24.338 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:22:24.582321+0000", 2024-08-19T14:27:24.338 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.202448131 2024-08-19T14:27:24.338 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.338 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.338 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:27:24.338 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:22:24.582389+0000", 2024-08-19T14:27:24.339 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7778000000000001e-05 2024-08-19T14:27:24.339 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.339 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.339 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:27:24.339 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:22:24.582442+0000", 2024-08-19T14:27:24.339 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3078000000000002e-05 2024-08-19T14:27:24.339 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:24.339 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:27:24.339 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:24.339 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.339 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.339 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25333.0:228230 4.e 4:7f46c467:::benchmark_data_smithi045_71562_object97055:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e400)", 2024-08-19T14:27:24.339 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:23:52.892067+0000", 2024-08-19T14:27:24.340 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 211.40079495099999, 2024-08-19T14:27:24.340 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.223615182, 2024-08-19T14:27:24.340 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:27:24.340 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:27:24.340 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:27:24.340 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:27:24.340 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25333", 2024-08-19T14:27:24.340 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:27:24.340 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 228230 2024-08-19T14:27:24.340 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.340 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:27:24.340 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.340 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:27:24.341 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:52.892067+0000", 2024-08-19T14:27:24.341 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:24.341 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.341 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.341 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:27:24.341 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:52.892067+0000", 2024-08-19T14:27:24.341 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:24.341 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.341 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.341 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:27:24.341 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:52.892069+0000", 2024-08-19T14:27:24.341 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2750000000000001e-06 2024-08-19T14:27:24.341 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.341 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.342 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:27:24.342 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:52.892073+0000", 2024-08-19T14:27:24.342 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3880000000000002e-06 2024-08-19T14:27:24.342 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.342 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.342 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:27:24.342 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:52.892083+0000", 2024-08-19T14:27:24.342 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.798e-06 2024-08-19T14:27:24.342 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.342 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.342 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:27:24.342 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:52.892090+0000", 2024-08-19T14:27:24.342 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9650000000000002e-06 2024-08-19T14:27:24.343 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.343 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.343 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:27:24.343 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:52.892145+0000", 2024-08-19T14:27:24.343 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4994999999999998e-05 2024-08-19T14:27:24.343 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.343 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.343 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:27:24.343 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:52.892270+0000", 2024-08-19T14:27:24.343 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012498000000000001 2024-08-19T14:27:24.343 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.343 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.343 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 1,3", 2024-08-19T14:27:24.343 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:52.892388+0000", 2024-08-19T14:27:24.344 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00011871600000000001 2024-08-19T14:27:24.344 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.344 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.344 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:24.344 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:52.898094+0000", 2024-08-19T14:27:24.344 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0057056449999999996 2024-08-19T14:27:24.344 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.344 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.344 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:27:24.344 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:52.898531+0000", 2024-08-19T14:27:24.344 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00043701699999999999 2024-08-19T14:27:24.344 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.344 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.344 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:24.344 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:53.115594+0000", 2024-08-19T14:27:24.345 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.21706310500000001 2024-08-19T14:27:24.345 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.345 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.345 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:27:24.345 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:53.115650+0000", 2024-08-19T14:27:24.345 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5398000000000001e-05 2024-08-19T14:27:24.345 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.345 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.345 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:27:24.345 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:53.115682+0000", 2024-08-19T14:27:24.345 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.29e-05 2024-08-19T14:27:24.345 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:24.345 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:27:24.345 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:24.345 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.346 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.346 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25333.0:228231 4.d 4:b8b5ade8:::benchmark_data_smithi045_71562_object97056:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e400)", 2024-08-19T14:27:24.346 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:23:52.892113+0000", 2024-08-19T14:27:24.346 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 211.40074941200001, 2024-08-19T14:27:24.346 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.22368043300000001, 2024-08-19T14:27:24.346 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:27:24.346 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:27:24.346 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:27:24.346 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:27:24.346 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25333", 2024-08-19T14:27:24.346 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:27:24.346 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 228231 2024-08-19T14:27:24.346 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.346 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:27:24.347 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.347 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:27:24.347 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:52.892113+0000", 2024-08-19T14:27:24.347 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:24.347 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.347 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.347 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:27:24.347 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:52.892113+0000", 2024-08-19T14:27:24.347 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:24.347 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.347 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.347 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:27:24.347 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:52.892113+0000", 2024-08-19T14:27:24.348 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.44e-07 2024-08-19T14:27:24.348 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.348 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.348 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:27:24.348 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:52.892116+0000", 2024-08-19T14:27:24.348 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8289999999999999e-06 2024-08-19T14:27:24.348 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.348 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.348 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:27:24.348 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:52.892124+0000", 2024-08-19T14:27:24.348 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.0520000000000002e-06 2024-08-19T14:27:24.348 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.348 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.349 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:27:24.349 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:52.892131+0000", 2024-08-19T14:27:24.349 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.387e-06 2024-08-19T14:27:24.349 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.349 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.349 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:27:24.349 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:52.892234+0000", 2024-08-19T14:27:24.349 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000103662 2024-08-19T14:27:24.349 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.349 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.349 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:27:24.349 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:52.892337+0000", 2024-08-19T14:27:24.349 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000103099 2024-08-19T14:27:24.350 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.350 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.350 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 1,9", 2024-08-19T14:27:24.350 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:52.892445+0000", 2024-08-19T14:27:24.350 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000107695 2024-08-19T14:27:24.350 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.350 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.350 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:24.350 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:52.897668+0000", 2024-08-19T14:27:24.350 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0052227059999999997 2024-08-19T14:27:24.350 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.350 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.350 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:27:24.351 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:52.898542+0000", 2024-08-19T14:27:24.351 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00087437400000000003 2024-08-19T14:27:24.351 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.351 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.351 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:24.351 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:53.115682+0000", 2024-08-19T14:27:24.351 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.217139837 2024-08-19T14:27:24.351 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.351 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.351 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:27:24.351 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:53.115748+0000", 2024-08-19T14:27:24.351 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.5968000000000003e-05 2024-08-19T14:27:24.351 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.352 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.352 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:27:24.352 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:23:53.115793+0000", 2024-08-19T14:27:24.352 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.528e-05 2024-08-19T14:27:24.352 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:24.352 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:27:24.352 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:24.352 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.352 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.352 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:15526 5.1 5:81aa3e11:::benchmark_data_smithi045_85309_object15525:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e426)", 2024-08-19T14:27:24.352 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:24:52.832831+0000", 2024-08-19T14:27:24.352 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 151.46003121699999, 2024-08-19T14:27:24.352 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.265005194, 2024-08-19T14:27:24.352 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:27:24.353 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:27:24.353 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:27:24.353 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:27:24.353 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:27:24.353 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:27:24.353 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 15526 2024-08-19T14:27:24.353 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.353 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:27:24.353 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.353 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:27:24.353 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.832831+0000", 2024-08-19T14:27:24.353 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:24.353 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.353 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.354 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:27:24.354 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.832831+0000", 2024-08-19T14:27:24.354 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:24.354 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.354 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.354 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:27:24.354 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.832832+0000", 2024-08-19T14:27:24.354 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.499e-06 2024-08-19T14:27:24.354 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.354 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.354 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:27:24.354 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.832894+0000", 2024-08-19T14:27:24.354 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1909999999999995e-05 2024-08-19T14:27:24.355 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.355 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.355 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:27:24.355 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.832910+0000", 2024-08-19T14:27:24.355 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5906e-05 2024-08-19T14:27:24.355 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.355 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.355 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:27:24.355 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.832918+0000", 2024-08-19T14:27:24.355 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2400000000000001e-06 2024-08-19T14:27:24.355 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.355 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.355 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:27:24.356 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.832964+0000", 2024-08-19T14:27:24.356 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6697999999999999e-05 2024-08-19T14:27:24.356 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.356 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.356 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:27:24.356 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.833093+0000", 2024-08-19T14:27:24.356 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012924599999999999 2024-08-19T14:27:24.356 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.356 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.356 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 3,7", 2024-08-19T14:27:24.356 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.833228+0000", 2024-08-19T14:27:24.356 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000134621 2024-08-19T14:27:24.356 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.356 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.357 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:27:24.357 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.834341+0000", 2024-08-19T14:27:24.357 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001112701 2024-08-19T14:27:24.357 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.357 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.357 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:24.357 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.835208+0000", 2024-08-19T14:27:24.357 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00086687000000000005 2024-08-19T14:27:24.357 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.357 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.357 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:24.357 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:53.097747+0000", 2024-08-19T14:27:24.357 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.26253938399999999 2024-08-19T14:27:24.357 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.358 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.358 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:27:24.358 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:53.097796+0000", 2024-08-19T14:27:24.358 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8965999999999998e-05 2024-08-19T14:27:24.358 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.358 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.358 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:27:24.358 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:53.097836+0000", 2024-08-19T14:27:24.358 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0153e-05 2024-08-19T14:27:24.358 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:24.358 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:27:24.358 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:24.358 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.359 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.359 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:15538 5.1 5:88fc82fa:::benchmark_data_smithi045_85309_object15537:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e426)", 2024-08-19T14:27:24.359 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:24:52.836081+0000", 2024-08-19T14:27:24.359 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 151.45678139099999, 2024-08-19T14:27:24.359 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.26185454899999999, 2024-08-19T14:27:24.359 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:27:24.359 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:27:24.359 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:27:24.359 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:27:24.359 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:27:24.359 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:27:24.359 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 15538 2024-08-19T14:27:24.359 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.360 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:27:24.360 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.360 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:27:24.360 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.836081+0000", 2024-08-19T14:27:24.360 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:24.360 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.360 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.360 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:27:24.360 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.836081+0000", 2024-08-19T14:27:24.360 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:24.360 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.360 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.360 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:27:24.361 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.836082+0000", 2024-08-19T14:27:24.361 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4309999999999999e-06 2024-08-19T14:27:24.361 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.361 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.361 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:27:24.361 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.836111+0000", 2024-08-19T14:27:24.361 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8274000000000001e-05 2024-08-19T14:27:24.361 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.361 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.361 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:27:24.361 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.836127+0000", 2024-08-19T14:27:24.361 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6427999999999999e-05 2024-08-19T14:27:24.361 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.361 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.362 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:27:24.362 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.836135+0000", 2024-08-19T14:27:24.362 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.4589999999999998e-06 2024-08-19T14:27:24.362 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.362 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.362 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:27:24.362 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.836291+0000", 2024-08-19T14:27:24.362 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00015594299999999999 2024-08-19T14:27:24.362 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.362 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.362 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:27:24.362 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.836589+0000", 2024-08-19T14:27:24.362 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00029754899999999998 2024-08-19T14:27:24.363 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.363 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.363 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 3,7", 2024-08-19T14:27:24.363 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.836726+0000", 2024-08-19T14:27:24.363 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000136737 2024-08-19T14:27:24.363 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.363 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.363 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:27:24.363 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.838701+0000", 2024-08-19T14:27:24.363 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0019752480000000002 2024-08-19T14:27:24.363 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.363 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.363 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:24.364 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.844308+0000", 2024-08-19T14:27:24.364 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0056073670000000003 2024-08-19T14:27:24.364 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.364 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.364 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:24.364 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:53.097873+0000", 2024-08-19T14:27:24.364 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.25356439400000003 2024-08-19T14:27:24.364 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.364 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.364 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:27:24.364 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:53.097909+0000", 2024-08-19T14:27:24.364 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6303000000000001e-05 2024-08-19T14:27:24.364 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.364 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.365 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:27:24.365 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:53.097935+0000", 2024-08-19T14:27:24.365 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6415999999999999e-05 2024-08-19T14:27:24.365 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:24.365 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:27:24.365 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:24.365 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.365 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.365 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:15554 5.1 5:8ece79d1:::benchmark_data_smithi045_85309_object15553:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e426)", 2024-08-19T14:27:24.365 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:24:52.841409+0000", 2024-08-19T14:27:24.365 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 151.45145367699999, 2024-08-19T14:27:24.365 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.25662066100000003, 2024-08-19T14:27:24.365 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:27:24.366 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:27:24.366 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:27:24.366 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:27:24.366 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:27:24.366 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:27:24.366 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 15554 2024-08-19T14:27:24.366 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.366 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:27:24.366 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.366 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:27:24.366 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.841409+0000", 2024-08-19T14:27:24.366 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:24.366 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.367 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.367 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:27:24.367 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.841409+0000", 2024-08-19T14:27:24.367 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:24.367 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.367 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.367 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:27:24.367 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.841410+0000", 2024-08-19T14:27:24.367 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3239999999999999e-06 2024-08-19T14:27:24.367 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.367 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.367 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:27:24.367 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.841462+0000", 2024-08-19T14:27:24.367 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1799000000000003e-05 2024-08-19T14:27:24.368 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.368 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.368 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:27:24.368 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.841476+0000", 2024-08-19T14:27:24.368 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4608e-05 2024-08-19T14:27:24.368 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.368 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.368 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:27:24.368 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.841485+0000", 2024-08-19T14:27:24.368 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6090000000000001e-06 2024-08-19T14:27:24.368 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.368 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.368 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:27:24.369 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.841546+0000", 2024-08-19T14:27:24.369 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1168999999999997e-05 2024-08-19T14:27:24.369 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.369 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.369 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:27:24.369 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.841682+0000", 2024-08-19T14:27:24.369 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013579900000000001 2024-08-19T14:27:24.369 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.369 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.369 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 3,7", 2024-08-19T14:27:24.369 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.841765+0000", 2024-08-19T14:27:24.369 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.3494999999999999e-05 2024-08-19T14:27:24.369 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.370 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.370 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:27:24.370 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.842729+0000", 2024-08-19T14:27:24.370 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00096402900000000004 2024-08-19T14:27:24.370 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.370 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.370 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:24.370 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.846426+0000", 2024-08-19T14:27:24.370 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0036964699999999999 2024-08-19T14:27:24.370 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.370 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.370 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:24.370 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:53.097964+0000", 2024-08-19T14:27:24.371 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.25153829300000002 2024-08-19T14:27:24.371 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.371 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.371 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:27:24.371 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:53.098003+0000", 2024-08-19T14:27:24.371 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9193999999999998e-05 2024-08-19T14:27:24.371 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.371 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.371 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:27:24.371 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:53.098029+0000", 2024-08-19T14:27:24.371 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5871999999999999e-05 2024-08-19T14:27:24.371 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:24.371 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:27:24.372 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:24.372 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.372 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.373 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:15557 5.1 5:894b28fd:::benchmark_data_smithi045_85309_object15556:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e426)", 2024-08-19T14:27:24.373 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:24:52.843093+0000", 2024-08-19T14:27:24.373 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 151.449768772, 2024-08-19T14:27:24.373 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.25539605599999998, 2024-08-19T14:27:24.373 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:27:24.373 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:27:24.373 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:27:24.373 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:27:24.373 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:27:24.373 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:27:24.373 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 15557 2024-08-19T14:27:24.373 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.374 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:27:24.374 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.374 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:27:24.374 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.843093+0000", 2024-08-19T14:27:24.374 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:24.374 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.374 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.374 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:27:24.374 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.843093+0000", 2024-08-19T14:27:24.374 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:24.374 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.374 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.375 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:27:24.378 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.843095+0000", 2024-08-19T14:27:24.378 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.246e-06 2024-08-19T14:27:24.378 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.378 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.378 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:27:24.378 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.843147+0000", 2024-08-19T14:27:24.378 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2379000000000001e-05 2024-08-19T14:27:24.378 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.378 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.378 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:27:24.378 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.843158+0000", 2024-08-19T14:27:24.378 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1238e-05 2024-08-19T14:27:24.379 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.379 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.379 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:27:24.379 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.843165+0000", 2024-08-19T14:27:24.379 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1609999999999997e-06 2024-08-19T14:27:24.379 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.379 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.379 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:27:24.379 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.843217+0000", 2024-08-19T14:27:24.379 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1898999999999999e-05 2024-08-19T14:27:24.379 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.379 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.379 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:27:24.380 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.843352+0000", 2024-08-19T14:27:24.380 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000135017 2024-08-19T14:27:24.380 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.380 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.380 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 3,7", 2024-08-19T14:27:24.380 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.843506+0000", 2024-08-19T14:27:24.380 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00015404200000000001 2024-08-19T14:27:24.380 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.380 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.380 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:27:24.380 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.844660+0000", 2024-08-19T14:27:24.380 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001153987 2024-08-19T14:27:24.380 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.380 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.381 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:24.381 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.846459+0000", 2024-08-19T14:27:24.381 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0017983700000000001 2024-08-19T14:27:24.381 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.381 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.381 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:24.381 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:53.098357+0000", 2024-08-19T14:27:24.381 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.25189793100000002 2024-08-19T14:27:24.381 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.381 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.381 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:27:24.381 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:53.098411+0000", 2024-08-19T14:27:24.381 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4796999999999997e-05 2024-08-19T14:27:24.381 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.382 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.382 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:27:24.382 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:53.098489+0000", 2024-08-19T14:27:24.382 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7989000000000002e-05 2024-08-19T14:27:24.382 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:24.382 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:27:24.382 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:24.382 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.382 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.382 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:15580 5.1 5:8e704e69:::benchmark_data_smithi045_85309_object15579:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e426)", 2024-08-19T14:27:24.382 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:24:52.850956+0000", 2024-08-19T14:27:24.382 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 151.441906139, 2024-08-19T14:27:24.382 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.24772282200000001, 2024-08-19T14:27:24.383 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:27:24.383 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:27:24.383 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:27:24.383 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:27:24.383 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:27:24.383 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:27:24.383 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 15580 2024-08-19T14:27:24.383 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.383 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:27:24.383 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.383 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:27:24.383 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.850956+0000", 2024-08-19T14:27:24.383 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:24.384 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.384 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.384 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:27:24.384 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.850956+0000", 2024-08-19T14:27:24.384 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:24.384 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.384 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.384 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:27:24.384 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.850959+0000", 2024-08-19T14:27:24.384 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8720000000000002e-06 2024-08-19T14:27:24.384 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.384 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.384 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:27:24.384 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.850994+0000", 2024-08-19T14:27:24.384 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5323e-05 2024-08-19T14:27:24.385 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.385 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.385 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:27:24.385 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.851010+0000", 2024-08-19T14:27:24.385 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5885000000000001e-05 2024-08-19T14:27:24.385 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.385 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.385 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:27:24.385 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.851018+0000", 2024-08-19T14:27:24.385 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1000000000000004e-06 2024-08-19T14:27:24.385 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.385 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.385 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:27:24.385 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.851085+0000", 2024-08-19T14:27:24.386 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6692000000000005e-05 2024-08-19T14:27:24.386 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.386 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.386 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:27:24.386 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.851218+0000", 2024-08-19T14:27:24.386 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013344400000000001 2024-08-19T14:27:24.386 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.386 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.386 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 3,7", 2024-08-19T14:27:24.386 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.851382+0000", 2024-08-19T14:27:24.386 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000163907 2024-08-19T14:27:24.386 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.386 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.387 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:27:24.387 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.852814+0000", 2024-08-19T14:27:24.387 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001431779 2024-08-19T14:27:24.387 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.387 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.387 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:24.387 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.853191+0000", 2024-08-19T14:27:24.387 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00037734900000000002 2024-08-19T14:27:24.387 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.387 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.387 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:24.387 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:53.098547+0000", 2024-08-19T14:27:24.387 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.24535599799999999 2024-08-19T14:27:24.387 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.388 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.388 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:27:24.388 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:53.098640+0000", 2024-08-19T14:27:24.388 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.2829000000000002e-05 2024-08-19T14:27:24.388 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.388 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.388 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:27:24.388 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:53.098679+0000", 2024-08-19T14:27:24.388 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8643999999999998e-05 2024-08-19T14:27:24.388 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:24.388 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:27:24.388 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:24.388 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.389 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.389 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:15592 5.1 5:892e880d:::benchmark_data_smithi045_85309_object15591:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e426)", 2024-08-19T14:27:24.389 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:24:52.856807+0000", 2024-08-19T14:27:24.389 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 151.43605474899999, 2024-08-19T14:27:24.389 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.241978421, 2024-08-19T14:27:24.389 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:27:24.389 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:27:24.389 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:27:24.389 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:27:24.389 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:27:24.389 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:27:24.389 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 15592 2024-08-19T14:27:24.389 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.390 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:27:24.390 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.390 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:27:24.390 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.856807+0000", 2024-08-19T14:27:24.390 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:24.390 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.390 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.390 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:27:24.390 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.856807+0000", 2024-08-19T14:27:24.390 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:24.390 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.390 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.390 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:27:24.391 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.856809+0000", 2024-08-19T14:27:24.391 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.609e-06 2024-08-19T14:27:24.391 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.391 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.391 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:27:24.391 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.856899+0000", 2024-08-19T14:27:24.391 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.0403999999999997e-05 2024-08-19T14:27:24.391 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.391 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.391 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:27:24.391 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.856911+0000", 2024-08-19T14:27:24.391 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1284000000000001e-05 2024-08-19T14:27:24.391 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.392 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.392 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:27:24.392 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.856918+0000", 2024-08-19T14:27:24.392 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2699999999999999e-06 2024-08-19T14:27:24.392 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.392 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.392 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:27:24.392 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.856977+0000", 2024-08-19T14:27:24.392 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9429999999999999e-05 2024-08-19T14:27:24.392 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.392 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.392 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:27:24.392 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.857118+0000", 2024-08-19T14:27:24.393 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014077299999999999 2024-08-19T14:27:24.393 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.393 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.393 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 3,7", 2024-08-19T14:27:24.393 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.857264+0000", 2024-08-19T14:27:24.393 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000145389 2024-08-19T14:27:24.393 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.393 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.393 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:27:24.393 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.858844+0000", 2024-08-19T14:27:24.393 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0015806399999999999 2024-08-19T14:27:24.393 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.393 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.393 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:24.394 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.859418+0000", 2024-08-19T14:27:24.394 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00057360899999999999 2024-08-19T14:27:24.394 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.394 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.394 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:24.394 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:53.098731+0000", 2024-08-19T14:27:24.394 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.23931316799999999 2024-08-19T14:27:24.394 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.394 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.394 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:27:24.394 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:53.098764+0000", 2024-08-19T14:27:24.394 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3074999999999998e-05 2024-08-19T14:27:24.395 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.395 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.395 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:27:24.395 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:53.098786+0000", 2024-08-19T14:27:24.395 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.177e-05 2024-08-19T14:27:24.395 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:24.395 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:27:24.395 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:24.395 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.395 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.395 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:15594 5.1 5:8280d572:::benchmark_data_smithi045_85309_object15593:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e426)", 2024-08-19T14:27:24.395 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:24:52.857436+0000", 2024-08-19T14:27:24.395 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 151.435426259, 2024-08-19T14:27:24.396 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.24148493300000001, 2024-08-19T14:27:24.396 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:27:24.396 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:27:24.396 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:27:24.396 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:27:24.396 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:27:24.396 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:27:24.396 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 15594 2024-08-19T14:27:24.396 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.396 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:27:24.396 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.396 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:27:24.396 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.857436+0000", 2024-08-19T14:27:24.396 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:24.397 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.397 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.397 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:27:24.397 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.857436+0000", 2024-08-19T14:27:24.397 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:24.397 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.397 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.397 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:27:24.397 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.857437+0000", 2024-08-19T14:27:24.397 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6700000000000003e-07 2024-08-19T14:27:24.397 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.397 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.397 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:27:24.398 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.857458+0000", 2024-08-19T14:27:24.398 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1036999999999999e-05 2024-08-19T14:27:24.398 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.398 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.398 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:27:24.398 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.857469+0000", 2024-08-19T14:27:24.398 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1171e-05 2024-08-19T14:27:24.398 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.398 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.398 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:27:24.398 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.857476+0000", 2024-08-19T14:27:24.398 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0729999999999999e-06 2024-08-19T14:27:24.398 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.399 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.399 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:27:24.399 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.858055+0000", 2024-08-19T14:27:24.399 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00057897700000000005 2024-08-19T14:27:24.399 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.399 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.399 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:27:24.399 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.858172+0000", 2024-08-19T14:27:24.399 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00011683700000000001 2024-08-19T14:27:24.399 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.399 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.399 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 3,7", 2024-08-19T14:27:24.399 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.858265+0000", 2024-08-19T14:27:24.399 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3407999999999997e-05 2024-08-19T14:27:24.399 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.400 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.400 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:27:24.400 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.859246+0000", 2024-08-19T14:27:24.400 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0009809319999999999 2024-08-19T14:27:24.400 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.400 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.400 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:24.400 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.859597+0000", 2024-08-19T14:27:24.400 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00035098200000000001 2024-08-19T14:27:24.400 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.400 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.400 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:24.400 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:53.098831+0000", 2024-08-19T14:27:24.400 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.23923408700000001 2024-08-19T14:27:24.401 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.401 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.401 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:27:24.401 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:53.098887+0000", 2024-08-19T14:27:24.401 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6087e-05 2024-08-19T14:27:24.401 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.401 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.401 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:27:24.401 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:53.098921+0000", 2024-08-19T14:27:24.401 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3575000000000003e-05 2024-08-19T14:27:24.401 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:24.401 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:27:24.401 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:24.401 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.402 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.402 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:15599 5.1 5:8315f30c:::benchmark_data_smithi045_85309_object15598:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e426)", 2024-08-19T14:27:24.402 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:24:52.860545+0000", 2024-08-19T14:27:24.402 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 151.432316805, 2024-08-19T14:27:24.402 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.23852358100000001, 2024-08-19T14:27:24.402 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:27:24.402 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:27:24.402 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:27:24.402 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:27:24.402 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:27:24.402 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:27:24.402 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 15599 2024-08-19T14:27:24.402 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.402 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:27:24.403 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.403 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:27:24.403 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.860545+0000", 2024-08-19T14:27:24.403 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:24.403 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.403 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.403 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:27:24.403 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.860545+0000", 2024-08-19T14:27:24.403 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:24.403 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.403 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.403 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:27:24.403 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.860547+0000", 2024-08-19T14:27:24.404 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.646e-06 2024-08-19T14:27:24.404 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.404 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.404 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:27:24.404 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.860608+0000", 2024-08-19T14:27:24.404 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1289000000000002e-05 2024-08-19T14:27:24.404 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.404 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.404 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:27:24.404 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.860620+0000", 2024-08-19T14:27:24.404 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1508000000000001e-05 2024-08-19T14:27:24.404 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.404 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.404 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:27:24.405 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.860627+0000", 2024-08-19T14:27:24.405 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0010000000000004e-06 2024-08-19T14:27:24.405 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.405 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.405 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:27:24.405 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.860680+0000", 2024-08-19T14:27:24.405 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2757000000000003e-05 2024-08-19T14:27:24.405 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.405 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.405 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:27:24.405 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.860826+0000", 2024-08-19T14:27:24.405 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014667799999999999 2024-08-19T14:27:24.405 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.405 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.406 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 3,7", 2024-08-19T14:27:24.406 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.860967+0000", 2024-08-19T14:27:24.406 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000140288 2024-08-19T14:27:24.406 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.406 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.406 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:24.406 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.862920+0000", 2024-08-19T14:27:24.406 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001953074 2024-08-19T14:27:24.406 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.406 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.406 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:27:24.406 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.862933+0000", 2024-08-19T14:27:24.406 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3461e-05 2024-08-19T14:27:24.407 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.407 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.407 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:24.407 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:53.098976+0000", 2024-08-19T14:27:24.407 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.23604261300000001 2024-08-19T14:27:24.407 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.407 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.407 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:27:24.407 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:53.099032+0000", 2024-08-19T14:27:24.407 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6483000000000001e-05 2024-08-19T14:27:24.407 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.407 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.407 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:27:24.407 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:53.099069+0000", 2024-08-19T14:27:24.408 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6782999999999997e-05 2024-08-19T14:27:24.408 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:24.408 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:27:24.408 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:24.408 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.408 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.408 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:15600 5.1 5:89abf743:::benchmark_data_smithi045_85309_object15599:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e426)", 2024-08-19T14:27:24.408 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:24:52.861028+0000", 2024-08-19T14:27:24.408 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 151.43183379600001, 2024-08-19T14:27:24.408 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.23827572399999999, 2024-08-19T14:27:24.408 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:27:24.408 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:27:24.408 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:27:24.409 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:27:24.409 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:27:24.409 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:27:24.409 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 15600 2024-08-19T14:27:24.409 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.409 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:27:24.409 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.409 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:27:24.409 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.861028+0000", 2024-08-19T14:27:24.409 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:24.409 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.409 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.409 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:27:24.409 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.861028+0000", 2024-08-19T14:27:24.410 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:24.410 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.410 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.410 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:27:24.410 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.861029+0000", 2024-08-19T14:27:24.410 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.8999999999999995e-07 2024-08-19T14:27:24.410 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.410 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.410 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:27:24.410 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.861062+0000", 2024-08-19T14:27:24.410 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2947999999999997e-05 2024-08-19T14:27:24.410 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.410 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.411 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:27:24.411 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.861071+0000", 2024-08-19T14:27:24.411 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.1379999999999996e-06 2024-08-19T14:27:24.411 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.411 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.411 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:27:24.411 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.861078+0000", 2024-08-19T14:27:24.411 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8040000000000004e-06 2024-08-19T14:27:24.411 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.411 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.411 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:27:24.411 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.861949+0000", 2024-08-19T14:27:24.411 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00087082499999999998 2024-08-19T14:27:24.411 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.412 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.412 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:27:24.412 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.862099+0000", 2024-08-19T14:27:24.412 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000150548 2024-08-19T14:27:24.412 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.412 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.412 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 3,7", 2024-08-19T14:27:24.412 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.862254+0000", 2024-08-19T14:27:24.412 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000154831 2024-08-19T14:27:24.412 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.412 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.412 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:27:24.412 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.863056+0000", 2024-08-19T14:27:24.412 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00080124899999999997 2024-08-19T14:27:24.413 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.413 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.413 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:24.413 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.863289+0000", 2024-08-19T14:27:24.413 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000233699 2024-08-19T14:27:24.413 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.413 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.413 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:24.413 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:53.099207+0000", 2024-08-19T14:27:24.413 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.23591731299999999 2024-08-19T14:27:24.413 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.413 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.413 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:27:24.413 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:53.099263+0000", 2024-08-19T14:27:24.414 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6025e-05 2024-08-19T14:27:24.414 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.414 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.414 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:27:24.414 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:53.099304+0000", 2024-08-19T14:27:24.414 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1554000000000002e-05 2024-08-19T14:27:24.414 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:24.414 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:27:24.414 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:24.414 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.414 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.414 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:188300 5.1 5:8424352e:::benchmark_data_smithi045_85309_object68047:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e479)", 2024-08-19T14:27:24.414 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:27:22.314257+0000", 2024-08-19T14:27:24.415 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 1.9786047180000002, 2024-08-19T14:27:24.415 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.038767797, 2024-08-19T14:27:24.415 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:27:24.415 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:27:24.415 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:27:24.415 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:27:24.415 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:27:24.415 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:27:24.415 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 188300 2024-08-19T14:27:24.415 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.415 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:27:24.415 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.415 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:27:24.415 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:22.314257+0000", 2024-08-19T14:27:24.416 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:24.416 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.416 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.416 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:27:24.416 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:22.314257+0000", 2024-08-19T14:27:24.416 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:24.416 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.416 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.416 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:27:24.416 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:22.314258+0000", 2024-08-19T14:27:24.416 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9200000000000002e-07 2024-08-19T14:27:24.416 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.416 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.416 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:27:24.417 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:22.314260+0000", 2024-08-19T14:27:24.417 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3379999999999999e-06 2024-08-19T14:27:24.417 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.417 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.417 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:27:24.417 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:22.314266+0000", 2024-08-19T14:27:24.417 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6060000000000002e-06 2024-08-19T14:27:24.417 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.417 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.417 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:27:24.417 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:27:24.418 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:22.314270+0000", 2024-08-19T14:27:24.418 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4159999999999997e-06 2024-08-19T14:27:24.418 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.419 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.419 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:27:24.419 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:22.331493+0000", 2024-08-19T14:27:24.419 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.017223236999999999 2024-08-19T14:27:24.419 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.419 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.419 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:27:24.419 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:22.331615+0000", 2024-08-19T14:27:24.419 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000121821 2024-08-19T14:27:24.419 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.419 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.419 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 3,7", 2024-08-19T14:27:24.419 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:22.332279+0000", 2024-08-19T14:27:24.419 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00066394900000000005 2024-08-19T14:27:24.420 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.420 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.420 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:24.420 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:22.337970+0000", 2024-08-19T14:27:24.420 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0056911970000000003 2024-08-19T14:27:24.420 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.420 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.420 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:27:24.420 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:22.338039+0000", 2024-08-19T14:27:24.420 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8361000000000005e-05 2024-08-19T14:27:24.420 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.420 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.420 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:24.421 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:22.352885+0000", 2024-08-19T14:27:24.421 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.014846276 2024-08-19T14:27:24.421 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.421 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.421 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:27:24.421 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:22.352974+0000", 2024-08-19T14:27:24.421 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.8910000000000001e-05 2024-08-19T14:27:24.421 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.421 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.421 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:27:24.421 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:22.353025+0000", 2024-08-19T14:27:24.421 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1294000000000001e-05 2024-08-19T14:27:24.421 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:24.421 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:27:24.422 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:24.422 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.422 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.422 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:189524 5.18 5:1fb13a25:::benchmark_data_smithi045_85309_object69271:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e480)", 2024-08-19T14:27:24.422 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:27:23.298819+0000", 2024-08-19T14:27:24.422 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.99404369000000004, 2024-08-19T14:27:24.422 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.039843992000000002, 2024-08-19T14:27:24.422 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:27:24.422 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:27:24.422 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:27:24.422 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:27:24.422 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:27:24.423 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:27:24.423 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 189524 2024-08-19T14:27:24.423 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.423 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:27:24.423 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.423 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:27:24.423 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:23.298819+0000", 2024-08-19T14:27:24.423 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:24.423 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.423 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.423 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:27:24.423 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:23.298819+0000", 2024-08-19T14:27:24.423 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:24.423 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.424 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.424 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:27:24.424 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:23.298819+0000", 2024-08-19T14:27:24.424 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3200000000000004e-07 2024-08-19T14:27:24.424 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.424 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.424 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:27:24.424 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:23.298824+0000", 2024-08-19T14:27:24.424 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3409999999999996e-06 2024-08-19T14:27:24.424 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.424 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.424 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:27:24.424 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:23.298836+0000", 2024-08-19T14:27:24.424 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2187999999999999e-05 2024-08-19T14:27:24.425 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.425 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.425 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:27:24.425 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:23.298843+0000", 2024-08-19T14:27:24.425 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9890000000000003e-06 2024-08-19T14:27:24.425 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.425 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.425 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:27:24.425 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:23.327566+0000", 2024-08-19T14:27:24.425 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.028723392 2024-08-19T14:27:24.425 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.425 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.425 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:27:24.425 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:23.327696+0000", 2024-08-19T14:27:24.426 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012999800000000001 2024-08-19T14:27:24.426 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.426 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.426 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 1,11", 2024-08-19T14:27:24.426 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:23.327804+0000", 2024-08-19T14:27:24.426 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000107422 2024-08-19T14:27:24.426 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.426 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.426 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:24.426 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:23.336784+0000", 2024-08-19T14:27:24.426 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0089801859999999994 2024-08-19T14:27:24.426 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.426 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.427 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:27:24.427 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:23.337161+0000", 2024-08-19T14:27:24.427 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00037691100000000001 2024-08-19T14:27:24.427 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.427 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.427 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:24.427 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:23.338555+0000", 2024-08-19T14:27:24.427 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0013940160000000001 2024-08-19T14:27:24.427 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.427 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.427 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:27:24.427 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:23.338619+0000", 2024-08-19T14:27:24.427 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3856999999999995e-05 2024-08-19T14:27:24.428 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:24.428 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:24.428 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:27:24.428 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:23.338662+0000", 2024-08-19T14:27:24.428 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3959999999999999e-05 2024-08-19T14:27:24.428 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:24.428 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:27:24.428 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:24.428 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:24.428 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:27:24.428 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:27:24.473 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:24.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:24.474 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:27:24.575 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:27:24.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:24.608 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:24.608 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:27:24.709 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:27:24.734 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:27:24.748 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:24.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:24.749 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:27:24.850 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:27:24.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:24 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:27:24.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:24 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:27:24.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:24 smithi031 ceph-mon[16336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:27:24.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:24 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/218058437' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:27:24.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:24 smithi031 ceph-mon[16336]: osdmap e481: 12 total, 12 up, 9 in 2024-08-19T14:27:24.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:24 smithi031 ceph-mon[16336]: 1.13 scrub starts 2024-08-19T14:27:24.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:24 smithi031 ceph-mon[16336]: 1.13 scrub ok 2024-08-19T14:27:24.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:24 smithi031 ceph-mon[16336]: 5.a scrub starts 2024-08-19T14:27:24.883 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:24.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:24.884 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:27:24.985 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:27:25.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:24 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:27:25.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:24 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:27:25.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:24 smithi032 ceph-mon[20004]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:27:25.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:24 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/218058437' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:27:25.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:24 smithi032 ceph-mon[20004]: osdmap e481: 12 total, 12 up, 9 in 2024-08-19T14:27:25.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:24 smithi032 ceph-mon[20004]: 1.13 scrub starts 2024-08-19T14:27:25.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:24 smithi032 ceph-mon[20004]: 1.13 scrub ok 2024-08-19T14:27:25.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:24 smithi032 ceph-mon[20004]: 5.a scrub starts 2024-08-19T14:27:25.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:25.017 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:25.017 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:27:25.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:24 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:27:25.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:24 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:27:25.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:24 smithi045 ceph-mon[20143]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:27:25.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:24 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/218058437' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:27:25.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:24 smithi045 ceph-mon[20143]: osdmap e481: 12 total, 12 up, 9 in 2024-08-19T14:27:25.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:24 smithi045 ceph-mon[20143]: 1.13 scrub starts 2024-08-19T14:27:25.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:24 smithi045 ceph-mon[20143]: 1.13 scrub ok 2024-08-19T14:27:25.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:24 smithi045 ceph-mon[20143]: 5.a scrub starts 2024-08-19T14:27:25.117 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:27:25.150 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:25.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:25.151 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:27:25.252 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:27:25.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:25.283 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:25.283 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:27:25.360 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:27:25.360 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:27:25.384 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:27:25.389 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:27:25.389 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:27:25.410 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:25.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:25.411 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:27:25.418 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:27:25.418 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:27:25.428 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-08-19T14:27:25.443 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:27:25.443 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:27:25.474 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:27:25.474 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:27:25.500 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:27:25.500 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:27:25.513 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:27:25.526 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:27:25.526 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:27:25.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:25.545 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:25.546 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:27:25.558 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:27:25.558 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:27:25.583 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:27:25.584 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:27:25.646 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:27:25.673 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:25.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:25.674 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:27:25.721 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:25 smithi032 ceph-mon[20004]: pgmap v1106: 57 pgs: 57 active+clean; 3.2 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 992 op/s 2024-08-19T14:27:25.721 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:25 smithi032 ceph-mon[20004]: 5.a scrub ok 2024-08-19T14:27:25.721 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:25 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/798286690' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:27:25.775 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:27:25.803 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:25.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:25.804 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:27:25.811 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:27:25.811 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:27:25.812 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:25 smithi031 ceph-mon[16336]: pgmap v1106: 57 pgs: 57 active+clean; 3.2 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 992 op/s 2024-08-19T14:27:25.812 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:25 smithi031 ceph-mon[16336]: 5.a scrub ok 2024-08-19T14:27:25.812 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:25 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/798286690' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:27:25.834 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:27:25.834 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:27:25.847 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:27:25.860 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:27:25.860 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:27:25.905 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:27:25.941 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:25.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:25.942 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:27:26.043 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:27:26.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:25 smithi045 ceph-mon[20143]: pgmap v1106: 57 pgs: 57 active+clean; 3.2 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 992 op/s 2024-08-19T14:27:26.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:25 smithi045 ceph-mon[20143]: 5.a scrub ok 2024-08-19T14:27:26.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:25 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/798286690' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:27:26.069 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:26.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:26.109 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:27:26.210 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:27:26.237 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:26.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:26.238 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:27:26.339 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:27:26.372 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:26.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:26.373 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:27:26.474 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:27:26.506 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:26.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:26.507 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:27:26.607 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:27:26.634 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-19T14:27:26.638 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:26.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:26.639 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:27:26.739 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:27:26.745 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:27:26.745 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:27:26.769 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:26.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:26.770 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:27:26.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:26 smithi031 ceph-mon[16336]: 1.17 scrub starts 2024-08-19T14:27:26.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:26 smithi031 ceph-mon[16336]: 1.17 scrub ok 2024-08-19T14:27:26.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:26 smithi031 ceph-mon[16336]: 1.14 scrub starts 2024-08-19T14:27:26.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:26 smithi031 ceph-mon[16336]: 1.14 scrub ok 2024-08-19T14:27:26.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:26 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:27:26.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:26 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:27:26.871 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:27:26.903 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:26.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:26.904 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:27:26.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:26 smithi032 ceph-mon[20004]: 1.17 scrub starts 2024-08-19T14:27:26.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:26 smithi032 ceph-mon[20004]: 1.17 scrub ok 2024-08-19T14:27:26.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:26 smithi032 ceph-mon[20004]: 1.14 scrub starts 2024-08-19T14:27:26.987 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:26 smithi032 ceph-mon[20004]: 1.14 scrub ok 2024-08-19T14:27:26.987 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:26 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:27:26.987 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:26 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:27:27.001 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:27:27.004 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:27:27.035 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:27.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:27.036 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:27:27.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:26 smithi045 ceph-mon[20143]: 1.17 scrub starts 2024-08-19T14:27:27.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:26 smithi045 ceph-mon[20143]: 1.17 scrub ok 2024-08-19T14:27:27.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:26 smithi045 ceph-mon[20143]: 1.14 scrub starts 2024-08-19T14:27:27.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:26 smithi045 ceph-mon[20143]: 1.14 scrub ok 2024-08-19T14:27:27.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:26 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:27:27.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:26 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:27:27.137 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:27:27.159 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:27.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:27.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:27:27.261 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:27:27.299 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:27.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:27.300 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:27:27.401 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:27:27.429 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:27.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:27.430 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:27:27.450 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-08-19T14:27:27.531 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:27:27.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:27.563 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:27.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:27:27.664 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:27:27.693 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:27.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:27.694 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:27:27.795 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:27:27.852 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:27.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:27.853 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:27:27.902 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:27:27.954 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:27:27.978 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:27.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:27.979 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:27:28.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:27 smithi032 ceph-mon[20004]: 1.a scrub starts 2024-08-19T14:27:28.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:27 smithi032 ceph-mon[20004]: 1.a scrub ok 2024-08-19T14:27:28.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:27 smithi032 ceph-mon[20004]: pgmap v1107: 57 pgs: 57 active+clean; 3.1 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 1.00k op/s 2024-08-19T14:27:28.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:27 smithi032 ceph-mon[20004]: 5.7 scrub starts 2024-08-19T14:27:28.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:27 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:27:28.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:27 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:27:28.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:27 smithi032 ceph-mon[20004]: osdmap e482: 12 total, 12 up, 9 in 2024-08-19T14:27:28.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:27 smithi032 ceph-mon[20004]: 5.7 scrub ok 2024-08-19T14:27:28.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:27 smithi032 ceph-mon[20004]: 1.6 scrub starts 2024-08-19T14:27:28.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:27 smithi032 ceph-mon[20004]: 1.6 scrub ok 2024-08-19T14:27:28.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:27 smithi032 ceph-mon[20004]: 5.9 scrub starts 2024-08-19T14:27:28.009 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:27 smithi032 ceph-mon[20004]: 1.1 scrub starts 2024-08-19T14:27:28.009 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:27 smithi032 ceph-mon[20004]: 1.1 scrub ok 2024-08-19T14:27:28.009 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:27 smithi032 ceph-mon[20004]: 5.9 scrub ok 2024-08-19T14:27:28.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:27 smithi045 ceph-mon[20143]: 1.a scrub starts 2024-08-19T14:27:28.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:27 smithi045 ceph-mon[20143]: 1.a scrub ok 2024-08-19T14:27:28.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:27 smithi045 ceph-mon[20143]: pgmap v1107: 57 pgs: 57 active+clean; 3.1 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 1.00k op/s 2024-08-19T14:27:28.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:27 smithi045 ceph-mon[20143]: 5.7 scrub starts 2024-08-19T14:27:28.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:27 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:27:28.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:27 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:27:28.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:27 smithi045 ceph-mon[20143]: osdmap e482: 12 total, 12 up, 9 in 2024-08-19T14:27:28.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:27 smithi045 ceph-mon[20143]: 5.7 scrub ok 2024-08-19T14:27:28.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:27 smithi045 ceph-mon[20143]: 1.6 scrub starts 2024-08-19T14:27:28.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:27 smithi045 ceph-mon[20143]: 1.6 scrub ok 2024-08-19T14:27:28.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:27 smithi045 ceph-mon[20143]: 5.9 scrub starts 2024-08-19T14:27:28.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:27 smithi045 ceph-mon[20143]: 1.1 scrub starts 2024-08-19T14:27:28.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:27 smithi045 ceph-mon[20143]: 1.1 scrub ok 2024-08-19T14:27:28.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:27 smithi045 ceph-mon[20143]: 5.9 scrub ok 2024-08-19T14:27:28.080 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:27:28.109 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:28.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:28.110 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:27:28.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:27 smithi031 ceph-mon[16336]: 1.a scrub starts 2024-08-19T14:27:28.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:27 smithi031 ceph-mon[16336]: 1.a scrub ok 2024-08-19T14:27:28.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:27 smithi031 ceph-mon[16336]: pgmap v1107: 57 pgs: 57 active+clean; 3.1 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 1.00k op/s 2024-08-19T14:27:28.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:27 smithi031 ceph-mon[16336]: 5.7 scrub starts 2024-08-19T14:27:28.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:27 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:27:28.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:27 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:27:28.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:27 smithi031 ceph-mon[16336]: osdmap e482: 12 total, 12 up, 9 in 2024-08-19T14:27:28.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:27 smithi031 ceph-mon[16336]: 5.7 scrub ok 2024-08-19T14:27:28.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:27 smithi031 ceph-mon[16336]: 1.6 scrub starts 2024-08-19T14:27:28.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:27 smithi031 ceph-mon[16336]: 1.6 scrub ok 2024-08-19T14:27:28.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:27 smithi031 ceph-mon[16336]: 5.9 scrub starts 2024-08-19T14:27:28.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:27 smithi031 ceph-mon[16336]: 1.1 scrub starts 2024-08-19T14:27:28.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:27 smithi031 ceph-mon[16336]: 1.1 scrub ok 2024-08-19T14:27:28.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:27 smithi031 ceph-mon[16336]: 5.9 scrub ok 2024-08-19T14:27:28.211 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:27:28.233 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 7, 8, 9, 11] out_osds: [10, 6, 2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-08-19T14:27:28.233 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-19T14:27:28.233 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-08-19T14:27:28.233 INFO:tasks.thrashosds.thrasher:Setting osd 5 primary_affinity to 0.000000 2024-08-19T14:27:28.233 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd primary-affinity 5 0 2024-08-19T14:27:28.240 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:28.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:28.241 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:27:28.342 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:27:28.375 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:28.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:28.376 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:27:28.477 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:27:28.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:28.513 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:28.513 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:27:28.535 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:27:28.585 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:27:28.614 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:27:28.640 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:28.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:28.641 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:27:28.742 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:27:28.766 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:28.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:28.767 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:27:28.830 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is unset 2024-08-19T14:27:28.868 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:27:28.895 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:28.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:28.896 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:27:28.996 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:27:29.029 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:29.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:29.030 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:27:29.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:28 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2556300435' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:27:29.130 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:27:29.160 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:29.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:29.161 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:27:29.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:28 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2556300435' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:27:29.261 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:27:29.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:28 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2556300435' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:27:29.300 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:29.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:29.301 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:27:29.401 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:27:29.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:29.436 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:29.437 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:27:29.537 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:27:29.577 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:29.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:29.578 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:27:29.679 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:27:29.706 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:29.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:29.707 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:27:29.808 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:27:29.840 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:29.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:29.841 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:27:29.942 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:27:29.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:29.976 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:29.976 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:27:30.023 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:27:30.077 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:27:30.104 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:30.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:30.105 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:27:30.118 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:29 smithi031 ceph-mon[16336]: pgmap v1109: 57 pgs: 57 active+clean; 2.9 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 1.27k op/s 2024-08-19T14:27:30.118 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:29 smithi031 ceph-mon[16336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:27:30.118 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:29 smithi031 ceph-mon[16336]: Cluster is now healthy 2024-08-19T14:27:30.118 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:29 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2556300435' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:27:30.119 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:29 smithi031 ceph-mon[16336]: osdmap e483: 12 total, 12 up, 9 in 2024-08-19T14:27:30.119 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:29 smithi031 ceph-mon[16336]: 5.d scrub starts 2024-08-19T14:27:30.205 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:27:30.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:30.235 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:30.236 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:27:30.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:29 smithi032 ceph-mon[20004]: pgmap v1109: 57 pgs: 57 active+clean; 2.9 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 1.27k op/s 2024-08-19T14:27:30.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:29 smithi032 ceph-mon[20004]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:27:30.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:29 smithi032 ceph-mon[20004]: Cluster is now healthy 2024-08-19T14:27:30.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:29 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2556300435' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:27:30.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:29 smithi032 ceph-mon[20004]: osdmap e483: 12 total, 12 up, 9 in 2024-08-19T14:27:30.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:29 smithi032 ceph-mon[20004]: 5.d scrub starts 2024-08-19T14:27:30.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:29 smithi045 ceph-mon[20143]: pgmap v1109: 57 pgs: 57 active+clean; 2.9 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 1.27k op/s 2024-08-19T14:27:30.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:29 smithi045 ceph-mon[20143]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:27:30.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:29 smithi045 ceph-mon[20143]: Cluster is now healthy 2024-08-19T14:27:30.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:29 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2556300435' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:27:30.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:29 smithi045 ceph-mon[20143]: osdmap e483: 12 total, 12 up, 9 in 2024-08-19T14:27:30.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:29 smithi045 ceph-mon[20143]: 5.d scrub starts 2024-08-19T14:27:30.336 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:27:30.370 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:30.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:30.371 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:27:30.472 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:27:30.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:30.496 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:30.496 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:27:30.597 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:27:30.627 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:30.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:30.628 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:27:30.660 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:27:30.660 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:27:30.692 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:27:30.692 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:27:30.715 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:27:30.716 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:27:30.730 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:27:30.740 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:27:30.740 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:27:30.764 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:30.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:30.766 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:27:30.767 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:27:30.767 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:27:30.797 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:27:30.797 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:27:30.825 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:27:30.825 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:27:30.827 INFO:teuthology.orchestra.run.smithi031.stderr:set osd.5 primary-affinity to 0 (802) 2024-08-19T14:27:30.852 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:27:30.852 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:27:30.866 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:27:30.875 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:27:30.875 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:27:30.892 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:30.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:30.893 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:27:30.897 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:27:30.897 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:27:30.914 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:27:30.914 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-08-19T14:27:30.914 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-08-19T14:27:30.914 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-08-19T14:27:30.914 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:27:30.915 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:27:30.915 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:27:30.935 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:27:30.936 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:27:30.994 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:27:31.015 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:31.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:31.016 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:27:31.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:30 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/536543223' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.0}]: dispatch 2024-08-19T14:27:31.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:30 smithi031 ceph-mon[16336]: 5.d scrub ok 2024-08-19T14:27:31.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:30 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3621371242' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:27:31.116 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:27:31.138 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:31.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:31.139 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:27:31.240 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:27:31.250 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:30 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/536543223' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.0}]: dispatch 2024-08-19T14:27:31.250 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:30 smithi032 ceph-mon[20004]: 5.d scrub ok 2024-08-19T14:27:31.250 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:30 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3621371242' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:27:31.263 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:31.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:31.264 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:27:31.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:30 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/536543223' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.0}]: dispatch 2024-08-19T14:27:31.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:30 smithi045 ceph-mon[20143]: 5.d scrub ok 2024-08-19T14:27:31.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:30 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3621371242' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:27:31.364 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:27:31.386 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:31.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:31.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:27:31.488 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:27:31.511 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:31.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:31.512 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:27:31.590 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-08-19T14:27:31.613 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:27:31.638 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:31.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:31.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:27:31.740 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:27:31.764 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:31.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:31.765 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:27:31.770 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-19T14:27:31.851 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:27:31.867 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:27:31.895 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:31.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:31.896 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:27:31.997 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:27:32.030 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:32.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:32.031 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:27:32.073 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:31 smithi031 ceph-mon[16336]: pgmap v1111: 57 pgs: 57 active+clean; 2.8 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 1.10k op/s 2024-08-19T14:27:32.074 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:31 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/536543223' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.0}]': finished 2024-08-19T14:27:32.074 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:31 smithi031 ceph-mon[16336]: osdmap e484: 12 total, 12 up, 9 in 2024-08-19T14:27:32.074 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:31 smithi031 ceph-mon[16336]: osdmap e485: 12 total, 12 up, 9 in 2024-08-19T14:27:32.131 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:27:32.138 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:31 smithi032 ceph-mon[20004]: pgmap v1111: 57 pgs: 57 active+clean; 2.8 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 1.10k op/s 2024-08-19T14:27:32.138 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:31 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/536543223' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.0}]': finished 2024-08-19T14:27:32.138 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:31 smithi032 ceph-mon[20004]: osdmap e484: 12 total, 12 up, 9 in 2024-08-19T14:27:32.138 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:31 smithi032 ceph-mon[20004]: osdmap e485: 12 total, 12 up, 9 in 2024-08-19T14:27:32.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:32.163 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:32.164 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:27:32.264 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:27:32.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:31 smithi045 ceph-mon[20143]: pgmap v1111: 57 pgs: 57 active+clean; 2.8 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 1.10k op/s 2024-08-19T14:27:32.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:31 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/536543223' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.0}]': finished 2024-08-19T14:27:32.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:31 smithi045 ceph-mon[20143]: osdmap e484: 12 total, 12 up, 9 in 2024-08-19T14:27:32.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:31 smithi045 ceph-mon[20143]: osdmap e485: 12 total, 12 up, 9 in 2024-08-19T14:27:32.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:32.302 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:32.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:27:32.403 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:27:32.438 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:32.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:32.439 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:27:32.461 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:27:32.462 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-08-19T14:27:32.462 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-08-19T14:27:32.462 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-08-19T14:27:32.462 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.462 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25057.0:64611 3.1c 3.ea59d81c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e246)", 2024-08-19T14:27:32.462 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:17:35.978148+0000", 2024-08-19T14:27:32.462 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 596.47926684200002, 2024-08-19T14:27:32.462 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.98523712200000002, 2024-08-19T14:27:32.462 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:27:32.462 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:27:32.462 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:27:32.462 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:27:32.463 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25057", 2024-08-19T14:27:32.463 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:27:32.463 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 64611 2024-08-19T14:27:32.463 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.463 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:27:32.463 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.463 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:27:32.463 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.978148+0000", 2024-08-19T14:27:32.463 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:32.463 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.463 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.464 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:27:32.464 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.978148+0000", 2024-08-19T14:27:32.464 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:32.464 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.464 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.464 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:27:32.464 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.978150+0000", 2024-08-19T14:27:32.464 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.903e-06 2024-08-19T14:27:32.464 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.464 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.464 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:27:32.464 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.978235+0000", 2024-08-19T14:27:32.464 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.4945999999999996e-05 2024-08-19T14:27:32.465 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.465 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.465 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:27:32.465 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.978255+0000", 2024-08-19T14:27:32.465 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9361e-05 2024-08-19T14:27:32.465 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.465 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.465 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:27:32.465 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.978263+0000", 2024-08-19T14:27:32.465 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6770000000000001e-06 2024-08-19T14:27:32.465 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.465 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.465 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:27:32.466 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.978330+0000", 2024-08-19T14:27:32.466 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7089999999999996e-05 2024-08-19T14:27:32.466 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.466 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.466 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:27:32.466 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:35.978341+0000", 2024-08-19T14:27:32.466 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0559e-05 2024-08-19T14:27:32.466 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.466 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.466 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:27:32.466 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:36.963362+0000", 2024-08-19T14:27:32.466 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.98502080999999997 2024-08-19T14:27:32.467 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.467 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.467 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:27:32.467 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:36.963385+0000", 2024-08-19T14:27:32.467 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3776000000000001e-05 2024-08-19T14:27:32.467 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:32.467 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:27:32.467 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:32.467 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.467 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.467 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25057.0:64634 3.1c 3.60566bc (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e246)", 2024-08-19T14:27:32.467 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:17:36.183428+0000", 2024-08-19T14:27:32.467 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 596.27398729399999, 2024-08-19T14:27:32.468 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.78006474000000003, 2024-08-19T14:27:32.468 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:27:32.468 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:27:32.468 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:27:32.468 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:27:32.468 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25057", 2024-08-19T14:27:32.468 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:27:32.468 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 64634 2024-08-19T14:27:32.468 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.468 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:27:32.468 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.468 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:27:32.468 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:36.183428+0000", 2024-08-19T14:27:32.469 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:32.469 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.469 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.469 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:27:32.469 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:36.183428+0000", 2024-08-19T14:27:32.469 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:32.469 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.469 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.469 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:27:32.469 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:36.183430+0000", 2024-08-19T14:27:32.469 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8160000000000001e-06 2024-08-19T14:27:32.469 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.470 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.470 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:27:32.470 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:36.183463+0000", 2024-08-19T14:27:32.470 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3254999999999999e-05 2024-08-19T14:27:32.470 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.470 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.470 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:27:32.470 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:36.183482+0000", 2024-08-19T14:27:32.470 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8595000000000001e-05 2024-08-19T14:27:32.470 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.470 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.470 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:27:32.470 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:36.183491+0000", 2024-08-19T14:27:32.470 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.8840000000000006e-06 2024-08-19T14:27:32.471 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.471 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.471 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:27:32.471 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:36.183557+0000", 2024-08-19T14:27:32.471 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.5791999999999996e-05 2024-08-19T14:27:32.471 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.471 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.471 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:27:32.471 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:36.183566+0000", 2024-08-19T14:27:32.471 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.9919999999999996e-06 2024-08-19T14:27:32.471 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.471 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.471 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:27:32.471 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:36.963484+0000", 2024-08-19T14:27:32.472 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.77991771499999996 2024-08-19T14:27:32.472 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.472 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.472 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:27:32.472 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:36.963493+0000", 2024-08-19T14:27:32.472 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6910000000000003e-06 2024-08-19T14:27:32.472 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:32.472 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:27:32.472 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:32.472 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.472 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.472 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25057.0:68717 3.1d 3:b8ee1297:::benchmark_data_smithi045_59973_object68716:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e251)", 2024-08-19T14:27:32.472 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:17:43.596001+0000", 2024-08-19T14:27:32.472 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 588.86141417299996, 2024-08-19T14:27:32.472 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.22569884800000001, 2024-08-19T14:27:32.473 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:27:32.473 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:27:32.473 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:27:32.473 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:27:32.473 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25057", 2024-08-19T14:27:32.473 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:27:32.473 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 68717 2024-08-19T14:27:32.473 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.473 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:27:32.473 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.473 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:27:32.473 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:43.596001+0000", 2024-08-19T14:27:32.473 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:32.474 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.474 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.474 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:27:32.474 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:43.596001+0000", 2024-08-19T14:27:32.474 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:32.474 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.474 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.474 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:27:32.474 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:43.596003+0000", 2024-08-19T14:27:32.474 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0540000000000002e-06 2024-08-19T14:27:32.474 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.474 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.474 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:27:32.475 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:43.596035+0000", 2024-08-19T14:27:32.475 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2276000000000001e-05 2024-08-19T14:27:32.475 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.475 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.475 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:27:32.475 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:43.596919+0000", 2024-08-19T14:27:32.475 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00088367600000000001 2024-08-19T14:27:32.475 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.475 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.475 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:27:32.475 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:43.596940+0000", 2024-08-19T14:27:32.475 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1239000000000001e-05 2024-08-19T14:27:32.475 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.475 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.476 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:27:32.476 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:43.598926+0000", 2024-08-19T14:27:32.476 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0019856399999999999 2024-08-19T14:27:32.476 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.476 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.476 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:27:32.476 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:43.599089+0000", 2024-08-19T14:27:32.476 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00016281999999999999 2024-08-19T14:27:32.476 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.476 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.476 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4,8", 2024-08-19T14:27:32.476 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:43.599257+0000", 2024-08-19T14:27:32.476 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000168114 2024-08-19T14:27:32.477 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.477 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.477 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:27:32.477 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:43.604920+0000", 2024-08-19T14:27:32.477 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0056633059999999999 2024-08-19T14:27:32.477 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.477 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.477 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:32.477 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:43.607476+0000", 2024-08-19T14:27:32.477 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0025555230000000001 2024-08-19T14:27:32.477 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.477 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.477 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:32.478 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:43.820484+0000", 2024-08-19T14:27:32.478 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.21300864999999999 2024-08-19T14:27:32.478 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.478 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.478 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:27:32.478 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:43.820556+0000", 2024-08-19T14:27:32.478 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1452999999999995e-05 2024-08-19T14:27:32.478 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.478 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.478 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:27:32.478 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:17:43.821700+0000", 2024-08-19T14:27:32.478 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001144097 2024-08-19T14:27:32.478 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:32.478 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:27:32.479 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:32.479 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.479 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.479 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:22993 5.7 5:e5194b18:::benchmark_data_smithi045_85309_object22992:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e426)", 2024-08-19T14:27:32.479 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:24:55.243091+0000", 2024-08-19T14:27:32.479 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 157.21432440000001, 2024-08-19T14:27:32.479 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20799368400000001, 2024-08-19T14:27:32.479 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:27:32.479 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:27:32.479 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:27:32.479 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:27:32.479 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:27:32.479 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:27:32.480 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 22993 2024-08-19T14:27:32.480 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.480 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:27:32.480 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.480 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:27:32.480 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.243091+0000", 2024-08-19T14:27:32.480 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:32.480 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.480 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.480 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:27:32.480 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.243091+0000", 2024-08-19T14:27:32.480 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:32.480 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.480 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.481 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:27:32.481 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.243092+0000", 2024-08-19T14:27:32.481 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.9400000000000004e-07 2024-08-19T14:27:32.481 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.481 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.481 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:27:32.481 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.243122+0000", 2024-08-19T14:27:32.481 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0105000000000001e-05 2024-08-19T14:27:32.481 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.481 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.481 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:27:32.481 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.243131+0000", 2024-08-19T14:27:32.481 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.6889999999999999e-06 2024-08-19T14:27:32.482 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.482 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.482 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:27:32.482 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.243138+0000", 2024-08-19T14:27:32.482 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0339999999999999e-06 2024-08-19T14:27:32.482 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.482 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.482 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:27:32.482 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.243187+0000", 2024-08-19T14:27:32.482 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8764999999999998e-05 2024-08-19T14:27:32.482 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.482 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.482 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:27:32.483 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.243294+0000", 2024-08-19T14:27:32.483 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000106908 2024-08-19T14:27:32.483 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.483 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.483 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 1,9", 2024-08-19T14:27:32.483 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.243424+0000", 2024-08-19T14:27:32.483 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000129441 2024-08-19T14:27:32.483 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.483 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.483 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:27:32.483 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.245731+0000", 2024-08-19T14:27:32.483 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0023073210000000002 2024-08-19T14:27:32.483 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.483 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.484 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:32.484 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.246134+0000", 2024-08-19T14:27:32.484 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00040348800000000002 2024-08-19T14:27:32.484 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.484 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.484 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:32.484 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.450957+0000", 2024-08-19T14:27:32.484 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20482256099999999 2024-08-19T14:27:32.484 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.484 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.484 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:27:32.484 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.451032+0000", 2024-08-19T14:27:32.484 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4701000000000002e-05 2024-08-19T14:27:32.484 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.485 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.485 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:27:32.485 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.451084+0000", 2024-08-19T14:27:32.485 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2777e-05 2024-08-19T14:27:32.485 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:32.485 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:27:32.485 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:32.485 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.485 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.485 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:23013 5.7 5:ee7b3505:::benchmark_data_smithi045_85309_object23012:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e426)", 2024-08-19T14:27:32.485 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:24:55.249101+0000", 2024-08-19T14:27:32.485 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 157.208313863, 2024-08-19T14:27:32.485 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20208954800000001, 2024-08-19T14:27:32.485 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:27:32.486 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:27:32.486 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:27:32.486 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:27:32.486 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:27:32.486 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:27:32.486 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 23013 2024-08-19T14:27:32.486 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.486 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:27:32.486 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.486 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:27:32.486 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.249101+0000", 2024-08-19T14:27:32.486 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:32.486 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.486 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.487 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:27:32.487 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.249101+0000", 2024-08-19T14:27:32.487 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:32.487 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.487 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.487 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:27:32.487 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.249103+0000", 2024-08-19T14:27:32.487 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.271e-06 2024-08-19T14:27:32.487 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.487 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.487 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:27:32.487 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.249300+0000", 2024-08-19T14:27:32.487 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00019775200000000001 2024-08-19T14:27:32.487 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.488 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.488 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:27:32.488 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.249313+0000", 2024-08-19T14:27:32.488 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.312e-05 2024-08-19T14:27:32.488 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.488 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.488 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:27:32.488 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.249321+0000", 2024-08-19T14:27:32.488 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.43e-06 2024-08-19T14:27:32.488 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.488 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.488 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:27:32.488 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.249451+0000", 2024-08-19T14:27:32.488 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000129982 2024-08-19T14:27:32.489 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.489 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.489 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:27:32.489 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.249576+0000", 2024-08-19T14:27:32.489 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012470299999999999 2024-08-19T14:27:32.489 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.489 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.489 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 1,9", 2024-08-19T14:27:32.489 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.249703+0000", 2024-08-19T14:27:32.489 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000127315 2024-08-19T14:27:32.489 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.489 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.489 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:27:32.489 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.253189+0000", 2024-08-19T14:27:32.490 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0034865310000000002 2024-08-19T14:27:32.490 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.490 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.490 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:32.490 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.254256+0000", 2024-08-19T14:27:32.490 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0010667630000000001 2024-08-19T14:27:32.490 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.490 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.490 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:32.491 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.451121+0000", 2024-08-19T14:27:32.491 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.19686435199999999 2024-08-19T14:27:32.491 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.492 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.492 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:27:32.492 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.451155+0000", 2024-08-19T14:27:32.492 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.4785e-05 2024-08-19T14:27:32.492 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.492 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.492 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:27:32.492 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.451191+0000", 2024-08-19T14:27:32.492 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5543999999999997e-05 2024-08-19T14:27:32.492 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:32.492 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:27:32.492 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:32.492 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.493 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.493 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:23018 5.7 5:eee02748:::benchmark_data_smithi045_85309_object23017:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e426)", 2024-08-19T14:27:32.493 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:24:55.251860+0000", 2024-08-19T14:27:32.493 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 157.205554737, 2024-08-19T14:27:32.493 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20029677200000001, 2024-08-19T14:27:32.493 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:27:32.493 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:27:32.493 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:27:32.493 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:27:32.493 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:27:32.493 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:27:32.493 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 23018 2024-08-19T14:27:32.493 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.494 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:27:32.494 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.494 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:27:32.494 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.251860+0000", 2024-08-19T14:27:32.494 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:32.494 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.494 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.494 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:27:32.494 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.251860+0000", 2024-08-19T14:27:32.494 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:32.494 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.494 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.494 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:27:32.495 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.251861+0000", 2024-08-19T14:27:32.495 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.7399999999999991e-07 2024-08-19T14:27:32.495 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.495 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.495 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:27:32.495 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.251893+0000", 2024-08-19T14:27:32.495 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1671000000000002e-05 2024-08-19T14:27:32.495 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.495 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.495 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:27:32.495 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.251903+0000", 2024-08-19T14:27:32.495 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.8360000000000004e-06 2024-08-19T14:27:32.495 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.495 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.496 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:27:32.496 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.251910+0000", 2024-08-19T14:27:32.496 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1760000000000004e-06 2024-08-19T14:27:32.496 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.496 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.496 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:27:32.496 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.251991+0000", 2024-08-19T14:27:32.496 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.0811999999999998e-05 2024-08-19T14:27:32.496 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.496 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.496 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:27:32.496 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.252123+0000", 2024-08-19T14:27:32.496 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013224100000000001 2024-08-19T14:27:32.496 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.497 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.497 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 1,9", 2024-08-19T14:27:32.497 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.252250+0000", 2024-08-19T14:27:32.497 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012669800000000001 2024-08-19T14:27:32.497 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.497 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.497 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:27:32.497 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.254218+0000", 2024-08-19T14:27:32.497 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001968114 2024-08-19T14:27:32.497 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.497 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.497 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:32.497 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.254934+0000", 2024-08-19T14:27:32.498 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00071644199999999997 2024-08-19T14:27:32.498 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.498 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.498 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:32.498 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.452060+0000", 2024-08-19T14:27:32.498 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.19712607600000001 2024-08-19T14:27:32.498 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.498 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.498 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:27:32.498 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.452114+0000", 2024-08-19T14:27:32.498 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3983999999999997e-05 2024-08-19T14:27:32.498 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.498 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.499 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:27:32.499 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.452157+0000", 2024-08-19T14:27:32.499 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2747999999999998e-05 2024-08-19T14:27:32.499 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:32.499 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:27:32.499 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:32.499 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.499 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.499 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:55755 5.6 5:699d86b9:::benchmark_data_smithi045_85309_object55754:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e431)", 2024-08-19T14:27:32.499 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:25:09.597312+0000", 2024-08-19T14:27:32.499 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 142.86010302599999, 2024-08-19T14:27:32.499 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.209956798, 2024-08-19T14:27:32.499 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:27:32.500 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:27:32.500 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:27:32.500 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:27:32.500 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:27:32.500 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:27:32.500 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 55755 2024-08-19T14:27:32.500 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.500 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:27:32.500 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.500 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:27:32.500 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.597312+0000", 2024-08-19T14:27:32.500 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:32.500 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.500 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.501 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:27:32.501 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.597312+0000", 2024-08-19T14:27:32.501 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:32.501 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.501 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.501 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:27:32.501 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.597313+0000", 2024-08-19T14:27:32.501 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.13e-06 2024-08-19T14:27:32.501 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.501 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.501 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:27:32.501 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.597346+0000", 2024-08-19T14:27:32.501 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3019999999999999e-05 2024-08-19T14:27:32.502 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.502 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.502 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:27:32.502 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.597355+0000", 2024-08-19T14:27:32.502 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.0399999999999998e-06 2024-08-19T14:27:32.502 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.502 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.502 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:27:32.502 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.597362+0000", 2024-08-19T14:27:32.502 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2659999999999998e-06 2024-08-19T14:27:32.502 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.502 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.502 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:27:32.503 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.597410+0000", 2024-08-19T14:27:32.503 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8013000000000003e-05 2024-08-19T14:27:32.503 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.503 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.503 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:27:32.503 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.597518+0000", 2024-08-19T14:27:32.503 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00010813 2024-08-19T14:27:32.503 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.503 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.503 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 7,8", 2024-08-19T14:27:32.503 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.597639+0000", 2024-08-19T14:27:32.503 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000121205 2024-08-19T14:27:32.503 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.504 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.504 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:27:32.504 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.600786+0000", 2024-08-19T14:27:32.504 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.003147558 2024-08-19T14:27:32.504 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.504 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.504 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:32.504 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.601648+0000", 2024-08-19T14:27:32.504 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00086196199999999997 2024-08-19T14:27:32.504 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.504 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.504 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:32.504 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.807181+0000", 2024-08-19T14:27:32.504 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20553236799999999 2024-08-19T14:27:32.505 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.505 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.505 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:27:32.505 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.807220+0000", 2024-08-19T14:27:32.505 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9119999999999998e-05 2024-08-19T14:27:32.505 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.505 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.505 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:27:32.505 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.807269+0000", 2024-08-19T14:27:32.505 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8986000000000002e-05 2024-08-19T14:27:32.505 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:32.505 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:27:32.505 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:32.505 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.506 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.506 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:55761 5.4 5:21702bb2:::benchmark_data_smithi045_85309_object55760:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e431)", 2024-08-19T14:27:32.506 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:25:09.599621+0000", 2024-08-19T14:27:32.506 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 142.85779422499999, 2024-08-19T14:27:32.506 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20994792700000001, 2024-08-19T14:27:32.506 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:27:32.506 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:27:32.506 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:27:32.506 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:27:32.506 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:27:32.506 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:27:32.506 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 55761 2024-08-19T14:27:32.506 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.507 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:27:32.507 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.507 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:27:32.507 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.599621+0000", 2024-08-19T14:27:32.507 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:32.507 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.507 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.507 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:27:32.507 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.599621+0000", 2024-08-19T14:27:32.507 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:32.507 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.507 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.507 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:27:32.507 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.599622+0000", 2024-08-19T14:27:32.508 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1689999999999999e-06 2024-08-19T14:27:32.508 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.508 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.508 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:27:32.508 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.599684+0000", 2024-08-19T14:27:32.508 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2228000000000001e-05 2024-08-19T14:27:32.508 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.508 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.508 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:27:32.508 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.599698+0000", 2024-08-19T14:27:32.508 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4031e-05 2024-08-19T14:27:32.508 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.508 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.509 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:27:32.509 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.599705+0000", 2024-08-19T14:27:32.509 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0099999999999998e-06 2024-08-19T14:27:32.509 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.509 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.509 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:27:32.509 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.599804+0000", 2024-08-19T14:27:32.509 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.8288000000000005e-05 2024-08-19T14:27:32.509 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.509 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.509 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:27:32.509 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.600054+0000", 2024-08-19T14:27:32.509 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00025056999999999998 2024-08-19T14:27:32.509 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.509 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.510 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0,8", 2024-08-19T14:27:32.510 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.600199+0000", 2024-08-19T14:27:32.510 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014504200000000001 2024-08-19T14:27:32.510 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.510 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.510 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:32.510 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.604843+0000", 2024-08-19T14:27:32.510 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0046438290000000004 2024-08-19T14:27:32.510 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.510 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.510 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:27:32.510 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.604854+0000", 2024-08-19T14:27:32.510 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0971e-05 2024-08-19T14:27:32.511 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.511 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.511 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:32.511 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.809503+0000", 2024-08-19T14:27:32.511 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20464875799999999 2024-08-19T14:27:32.511 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.511 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.511 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:27:32.511 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.809534+0000", 2024-08-19T14:27:32.511 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1217999999999998e-05 2024-08-19T14:27:32.511 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.511 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.511 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:27:32.511 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.809569+0000", 2024-08-19T14:27:32.512 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.4813e-05 2024-08-19T14:27:32.512 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:32.512 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:27:32.512 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:32.512 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.512 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.512 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:55763 5.4 5:22d4b04e:::benchmark_data_smithi045_85309_object55762:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e431)", 2024-08-19T14:27:32.512 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:25:09.599725+0000", 2024-08-19T14:27:32.512 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 142.85769014300001, 2024-08-19T14:27:32.512 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20996778999999999, 2024-08-19T14:27:32.512 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:27:32.512 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:27:32.512 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:27:32.512 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:27:32.513 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:27:32.513 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:27:32.513 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 55763 2024-08-19T14:27:32.513 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.513 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:27:32.513 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.513 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:27:32.513 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.599725+0000", 2024-08-19T14:27:32.513 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:32.513 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.513 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.513 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:27:32.513 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.599725+0000", 2024-08-19T14:27:32.514 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:32.514 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.514 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.514 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:27:32.514 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.599726+0000", 2024-08-19T14:27:32.514 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2699999999999999e-07 2024-08-19T14:27:32.514 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.514 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.514 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:27:32.514 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.599802+0000", 2024-08-19T14:27:32.514 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6167000000000004e-05 2024-08-19T14:27:32.514 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.514 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.514 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:27:32.515 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.599811+0000", 2024-08-19T14:27:32.515 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.6220000000000004e-06 2024-08-19T14:27:32.515 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.515 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.515 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:27:32.515 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.599818+0000", 2024-08-19T14:27:32.515 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4420000000000003e-06 2024-08-19T14:27:32.515 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.515 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.515 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:27:32.515 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.602462+0000", 2024-08-19T14:27:32.515 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0026439990000000002 2024-08-19T14:27:32.515 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.515 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.515 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:27:32.516 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.602619+0000", 2024-08-19T14:27:32.516 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00015674800000000001 2024-08-19T14:27:32.516 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.516 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.516 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0,8", 2024-08-19T14:27:32.516 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.602748+0000", 2024-08-19T14:27:32.516 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012939900000000001 2024-08-19T14:27:32.516 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.516 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.516 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:32.516 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.609052+0000", 2024-08-19T14:27:32.516 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.006304128 2024-08-19T14:27:32.516 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.516 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.517 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:27:32.517 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.611165+0000", 2024-08-19T14:27:32.517 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.002113008 2024-08-19T14:27:32.517 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.517 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.517 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:32.517 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.809609+0000", 2024-08-19T14:27:32.517 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.19844395500000001 2024-08-19T14:27:32.517 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.517 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.517 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:27:32.517 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.809648+0000", 2024-08-19T14:27:32.517 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9081000000000001e-05 2024-08-19T14:27:32.517 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.517 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.518 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:27:32.518 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.809693+0000", 2024-08-19T14:27:32.518 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4613999999999997e-05 2024-08-19T14:27:32.518 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:32.518 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:27:32.518 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:32.518 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.518 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.518 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:55767 5.4 5:200c3bc0:::benchmark_data_smithi045_85309_object55766:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e431)", 2024-08-19T14:27:32.518 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:25:09.601839+0000", 2024-08-19T14:27:32.518 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 142.85557611499999, 2024-08-19T14:27:32.518 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20836729700000001, 2024-08-19T14:27:32.518 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:27:32.518 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:27:32.519 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:27:32.519 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:27:32.519 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:27:32.519 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:27:32.519 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 55767 2024-08-19T14:27:32.519 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.519 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:27:32.519 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.519 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:27:32.519 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.601839+0000", 2024-08-19T14:27:32.519 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:32.519 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.519 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.519 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:27:32.520 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.601839+0000", 2024-08-19T14:27:32.520 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:32.520 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.520 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.520 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:27:32.520 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.601840+0000", 2024-08-19T14:27:32.520 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.049e-06 2024-08-19T14:27:32.520 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.520 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.520 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:27:32.520 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.601863+0000", 2024-08-19T14:27:32.521 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2748999999999999e-05 2024-08-19T14:27:32.521 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.521 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.521 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:27:32.521 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.601874+0000", 2024-08-19T14:27:32.521 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1219999999999999e-05 2024-08-19T14:27:32.521 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.521 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.521 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:27:32.521 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.601881+0000", 2024-08-19T14:27:32.521 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.562e-06 2024-08-19T14:27:32.521 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.521 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.521 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:27:32.522 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.604918+0000", 2024-08-19T14:27:32.522 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.003037629 2024-08-19T14:27:32.522 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.522 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.522 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:27:32.522 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.605062+0000", 2024-08-19T14:27:32.522 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000143821 2024-08-19T14:27:32.522 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.522 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.522 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0,8", 2024-08-19T14:27:32.522 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.605192+0000", 2024-08-19T14:27:32.522 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013037300000000001 2024-08-19T14:27:32.522 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.523 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.523 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:32.523 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.611145+0000", 2024-08-19T14:27:32.523 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0059520939999999998 2024-08-19T14:27:32.523 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.523 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.523 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:27:32.523 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.611171+0000", 2024-08-19T14:27:32.523 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6618000000000002e-05 2024-08-19T14:27:32.523 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.523 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.523 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:32.523 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.810142+0000", 2024-08-19T14:27:32.524 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.19897035199999999 2024-08-19T14:27:32.524 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.524 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.524 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:27:32.524 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.810178+0000", 2024-08-19T14:27:32.524 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6950999999999999e-05 2024-08-19T14:27:32.524 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.524 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.524 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:27:32.524 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.810206+0000", 2024-08-19T14:27:32.524 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7878999999999999e-05 2024-08-19T14:27:32.524 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:32.524 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:27:32.524 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:32.524 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.525 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.525 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:55775 5.4 5:2da10789:::benchmark_data_smithi045_85309_object55774:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e431)", 2024-08-19T14:27:32.525 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:25:09.605530+0000", 2024-08-19T14:27:32.525 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 142.851885568, 2024-08-19T14:27:32.525 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20478083, 2024-08-19T14:27:32.525 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:27:32.525 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:27:32.525 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:27:32.525 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:27:32.525 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:27:32.525 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:27:32.525 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 55775 2024-08-19T14:27:32.525 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.526 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:27:32.526 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.526 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:27:32.526 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.605530+0000", 2024-08-19T14:27:32.526 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:32.526 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.526 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.526 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:27:32.526 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.605530+0000", 2024-08-19T14:27:32.526 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:32.526 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.526 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.526 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:27:32.526 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.605531+0000", 2024-08-19T14:27:32.527 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3039999999999999e-06 2024-08-19T14:27:32.527 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.527 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.527 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:27:32.527 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.605633+0000", 2024-08-19T14:27:32.527 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000102602 2024-08-19T14:27:32.527 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.527 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.527 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:27:32.527 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.605643+0000", 2024-08-19T14:27:32.527 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.4399999999999994e-06 2024-08-19T14:27:32.527 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.527 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.527 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:27:32.528 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.605649+0000", 2024-08-19T14:27:32.528 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2169999999999999e-06 2024-08-19T14:27:32.528 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.528 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.528 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:27:32.528 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.607082+0000", 2024-08-19T14:27:32.528 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0014326289999999999 2024-08-19T14:27:32.528 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.528 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.528 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:27:32.528 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.607190+0000", 2024-08-19T14:27:32.528 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000108428 2024-08-19T14:27:32.528 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.528 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.529 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0,8", 2024-08-19T14:27:32.529 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.607312+0000", 2024-08-19T14:27:32.529 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000121927 2024-08-19T14:27:32.529 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.529 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.529 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:27:32.529 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.611175+0000", 2024-08-19T14:27:32.529 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0038631889999999999 2024-08-19T14:27:32.529 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.529 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.529 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:32.529 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.614394+0000", 2024-08-19T14:27:32.529 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.003218505 2024-08-19T14:27:32.529 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.530 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.530 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:32.530 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.810243+0000", 2024-08-19T14:27:32.530 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.19584892000000001 2024-08-19T14:27:32.530 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.530 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.530 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:27:32.530 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.810269+0000", 2024-08-19T14:27:32.530 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6027e-05 2024-08-19T14:27:32.530 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.530 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.530 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:27:32.530 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.810310+0000", 2024-08-19T14:27:32.530 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1641999999999999e-05 2024-08-19T14:27:32.531 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:32.531 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:27:32.531 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:32.531 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.531 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.531 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:55779 5.4 5:28c62a63:::benchmark_data_smithi045_85309_object55778:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e431)", 2024-08-19T14:27:32.531 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:25:09.607121+0000", 2024-08-19T14:27:32.531 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 142.850294238, 2024-08-19T14:27:32.531 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20333711199999999, 2024-08-19T14:27:32.531 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:27:32.531 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:27:32.531 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:27:32.532 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:27:32.532 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:27:32.532 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:27:32.532 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 55779 2024-08-19T14:27:32.532 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.532 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:27:32.532 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.532 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:27:32.532 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.607121+0000", 2024-08-19T14:27:32.532 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:32.532 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.532 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.532 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:27:32.532 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.607121+0000", 2024-08-19T14:27:32.532 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:32.533 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.533 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.533 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:27:32.534 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.607122+0000", 2024-08-19T14:27:32.534 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1820000000000001e-06 2024-08-19T14:27:32.534 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.534 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.534 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:27:32.534 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.607157+0000", 2024-08-19T14:27:32.534 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.4443999999999998e-05 2024-08-19T14:27:32.534 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.535 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.535 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:27:32.535 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.607169+0000", 2024-08-19T14:27:32.535 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2531000000000001e-05 2024-08-19T14:27:32.535 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.535 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.535 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:27:32.535 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.607176+0000", 2024-08-19T14:27:32.535 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1199999999999996e-06 2024-08-19T14:27:32.535 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.535 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.535 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:27:32.535 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.609081+0000", 2024-08-19T14:27:32.535 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001904705 2024-08-19T14:27:32.535 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.536 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.536 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:27:32.536 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.609202+0000", 2024-08-19T14:27:32.536 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000121138 2024-08-19T14:27:32.536 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.536 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.536 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0,8", 2024-08-19T14:27:32.536 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.609328+0000", 2024-08-19T14:27:32.536 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012569100000000001 2024-08-19T14:27:32.536 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.536 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.536 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:27:32.536 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.611568+0000", 2024-08-19T14:27:32.537 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0022404769999999998 2024-08-19T14:27:32.537 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.537 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.537 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:32.537 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.616823+0000", 2024-08-19T14:27:32.537 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0052546889999999999 2024-08-19T14:27:32.537 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.537 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.537 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:32.537 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.810361+0000", 2024-08-19T14:27:32.537 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.193538245 2024-08-19T14:27:32.537 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.537 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.537 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:27:32.538 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.810410+0000", 2024-08-19T14:27:32.538 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8819000000000001e-05 2024-08-19T14:27:32.538 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.538 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.538 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:27:32.538 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.810458+0000", 2024-08-19T14:27:32.538 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8071000000000001e-05 2024-08-19T14:27:32.538 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:32.538 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:27:32.538 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:32.538 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.538 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.539 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:81628 5.7 5:e12ddb93:::benchmark_data_smithi045_85309_object81627:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e437)", 2024-08-19T14:27:32.539 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:25:25.047926+0000", 2024-08-19T14:27:32.539 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 127.40948927399999, 2024-08-19T14:27:32.539 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.207246069, 2024-08-19T14:27:32.539 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:27:32.539 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:27:32.539 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:27:32.539 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:27:32.539 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:27:32.539 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:27:32.539 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 81628 2024-08-19T14:27:32.539 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.539 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:27:32.539 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.540 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:27:32.540 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.047926+0000", 2024-08-19T14:27:32.540 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:32.540 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.540 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.540 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:27:32.540 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.047926+0000", 2024-08-19T14:27:32.540 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:32.540 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.540 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.540 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:27:32.541 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.047927+0000", 2024-08-19T14:27:32.541 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0219999999999999e-06 2024-08-19T14:27:32.541 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.541 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.541 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:27:32.541 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.047977+0000", 2024-08-19T14:27:32.541 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0170000000000002e-05 2024-08-19T14:27:32.541 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.541 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.541 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:27:32.541 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.047993+0000", 2024-08-19T14:27:32.541 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5835999999999999e-05 2024-08-19T14:27:32.541 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.542 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.542 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:27:32.542 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.047998+0000", 2024-08-19T14:27:32.542 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8500000000000002e-06 2024-08-19T14:27:32.542 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.542 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.542 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:27:32.542 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.048045+0000", 2024-08-19T14:27:32.542 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6828e-05 2024-08-19T14:27:32.542 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.542 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.542 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:27:32.542 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.048139+0000", 2024-08-19T14:27:32.543 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.4199999999999999e-05 2024-08-19T14:27:32.543 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.543 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.543 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 1,9", 2024-08-19T14:27:32.543 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.048232+0000", 2024-08-19T14:27:32.543 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3393999999999994e-05 2024-08-19T14:27:32.543 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.543 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.543 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:27:32.543 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.050672+0000", 2024-08-19T14:27:32.543 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0024403189999999998 2024-08-19T14:27:32.543 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.543 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.543 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:32.543 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.051858+0000", 2024-08-19T14:27:32.544 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0011859489999999999 2024-08-19T14:27:32.544 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.544 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.544 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:32.544 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.255101+0000", 2024-08-19T14:27:32.544 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20324223 2024-08-19T14:27:32.544 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.544 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.544 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:27:32.544 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.255142+0000", 2024-08-19T14:27:32.544 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0915999999999999e-05 2024-08-19T14:27:32.544 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.544 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.544 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:27:32.545 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.255172+0000", 2024-08-19T14:27:32.545 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0355e-05 2024-08-19T14:27:32.545 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:32.545 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:27:32.545 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:32.545 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.545 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.545 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:81639 5.7 5:ef0c5b8b:::benchmark_data_smithi045_85309_object81638:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e437)", 2024-08-19T14:27:32.545 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:25:25.053488+0000", 2024-08-19T14:27:32.545 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 127.40392739399999, 2024-08-19T14:27:32.545 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20480580300000001, 2024-08-19T14:27:32.545 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:27:32.545 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:27:32.545 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:27:32.546 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:27:32.546 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:27:32.546 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:27:32.546 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 81639 2024-08-19T14:27:32.546 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.546 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:27:32.546 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.546 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:27:32.546 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.053488+0000", 2024-08-19T14:27:32.546 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:32.546 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.546 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.546 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:27:32.546 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.053488+0000", 2024-08-19T14:27:32.547 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:32.547 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.547 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.547 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:27:32.547 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.053489+0000", 2024-08-19T14:27:32.547 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.4200000000000005e-07 2024-08-19T14:27:32.547 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.547 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.547 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:27:32.547 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.053559+0000", 2024-08-19T14:27:32.547 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0240000000000005e-05 2024-08-19T14:27:32.547 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.547 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.547 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:27:32.548 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.053568+0000", 2024-08-19T14:27:32.548 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.9989999999999997e-06 2024-08-19T14:27:32.548 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.548 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.548 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:27:32.548 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.053575+0000", 2024-08-19T14:27:32.548 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.092e-06 2024-08-19T14:27:32.548 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.548 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.548 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:27:32.548 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.053624+0000", 2024-08-19T14:27:32.548 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9209000000000003e-05 2024-08-19T14:27:32.548 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.549 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.549 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:27:32.549 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.053826+0000", 2024-08-19T14:27:32.549 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000201573 2024-08-19T14:27:32.549 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.549 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.549 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 1,9", 2024-08-19T14:27:32.549 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.054893+0000", 2024-08-19T14:27:32.549 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0010669849999999999 2024-08-19T14:27:32.549 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.549 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.549 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:32.549 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.058523+0000", 2024-08-19T14:27:32.550 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0036301929999999999 2024-08-19T14:27:32.550 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.550 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.550 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:27:32.550 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.060958+0000", 2024-08-19T14:27:32.550 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0024348579999999998 2024-08-19T14:27:32.550 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.550 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.550 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:32.550 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.258229+0000", 2024-08-19T14:27:32.550 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.19727160199999999 2024-08-19T14:27:32.550 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.550 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.551 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:27:32.551 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.258265+0000", 2024-08-19T14:27:32.551 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5717999999999999e-05 2024-08-19T14:27:32.551 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.551 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.551 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:27:32.551 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.258294+0000", 2024-08-19T14:27:32.551 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8492000000000001e-05 2024-08-19T14:27:32.551 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:32.551 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:27:32.551 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:32.551 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.551 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.552 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:81655 5.7 5:e1e2b816:::benchmark_data_smithi045_85309_object81654:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e437)", 2024-08-19T14:27:32.552 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:25:25.061831+0000", 2024-08-19T14:27:32.552 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 127.395584391, 2024-08-19T14:27:32.552 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.19841341400000001, 2024-08-19T14:27:32.552 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:27:32.552 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:27:32.552 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:27:32.552 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:27:32.552 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:27:32.552 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:27:32.552 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 81655 2024-08-19T14:27:32.552 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.552 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:27:32.553 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.553 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:27:32.553 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.061831+0000", 2024-08-19T14:27:32.553 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:32.553 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.553 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.553 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:27:32.553 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.061831+0000", 2024-08-19T14:27:32.553 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:32.553 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.553 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.553 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:27:32.553 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.061831+0000", 2024-08-19T14:27:32.553 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4099999999999998e-07 2024-08-19T14:27:32.554 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.554 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.554 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:27:32.554 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.061902+0000", 2024-08-19T14:27:32.554 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0333999999999994e-05 2024-08-19T14:27:32.554 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.554 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.554 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:27:32.554 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.061912+0000", 2024-08-19T14:27:32.554 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0125e-05 2024-08-19T14:27:32.554 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.554 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.554 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:27:32.554 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.061918+0000", 2024-08-19T14:27:32.555 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.879e-06 2024-08-19T14:27:32.555 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.555 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.555 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:27:32.555 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.061972+0000", 2024-08-19T14:27:32.555 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4756000000000002e-05 2024-08-19T14:27:32.555 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.555 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.555 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:27:32.555 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.062117+0000", 2024-08-19T14:27:32.555 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000144103 2024-08-19T14:27:32.555 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.555 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.555 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 1,9", 2024-08-19T14:27:32.556 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.062257+0000", 2024-08-19T14:27:32.556 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000140271 2024-08-19T14:27:32.556 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.556 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.556 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:27:32.556 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.065634+0000", 2024-08-19T14:27:32.556 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0033769440000000002 2024-08-19T14:27:32.556 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.556 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.556 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:32.556 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.066411+0000", 2024-08-19T14:27:32.556 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00077719599999999996 2024-08-19T14:27:32.556 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.556 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.557 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:32.557 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.260128+0000", 2024-08-19T14:27:32.557 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.19371739800000001 2024-08-19T14:27:32.557 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.557 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.557 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:27:32.557 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.260203+0000", 2024-08-19T14:27:32.557 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4293000000000002e-05 2024-08-19T14:27:32.557 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.557 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.557 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:27:32.557 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.260244+0000", 2024-08-19T14:27:32.557 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1474000000000003e-05 2024-08-19T14:27:32.557 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:32.558 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:27:32.558 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:32.558 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.558 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.558 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:92831 5.f 5:fd6d6df7:::benchmark_data_smithi045_85309_object92830:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e445)", 2024-08-19T14:27:32.558 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:25:37.231829+0000", 2024-08-19T14:27:32.558 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 115.225586423, 2024-08-19T14:27:32.558 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.194037033, 2024-08-19T14:27:32.558 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:27:32.558 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:27:32.558 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:27:32.558 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:27:32.558 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:27:32.558 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:27:32.559 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 92831 2024-08-19T14:27:32.559 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.559 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:27:32.559 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.559 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:27:32.559 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.231829+0000", 2024-08-19T14:27:32.559 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:32.559 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.559 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.559 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:27:32.559 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.231829+0000", 2024-08-19T14:27:32.559 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:32.559 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.559 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.560 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:27:32.560 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.231831+0000", 2024-08-19T14:27:32.560 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8670000000000001e-06 2024-08-19T14:27:32.560 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.560 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.560 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:27:32.560 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.231959+0000", 2024-08-19T14:27:32.560 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000128181 2024-08-19T14:27:32.560 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.560 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.560 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:27:32.560 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.231972+0000", 2024-08-19T14:27:32.561 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3689999999999999e-05 2024-08-19T14:27:32.561 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.561 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.561 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:27:32.561 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.231988+0000", 2024-08-19T14:27:32.561 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.562e-05 2024-08-19T14:27:32.561 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.561 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.561 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:27:32.561 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.232043+0000", 2024-08-19T14:27:32.561 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4694999999999997e-05 2024-08-19T14:27:32.561 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.561 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.561 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:27:32.562 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.232161+0000", 2024-08-19T14:27:32.562 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00011810499999999999 2024-08-19T14:27:32.562 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.562 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.562 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4,11", 2024-08-19T14:27:32.562 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.232284+0000", 2024-08-19T14:27:32.562 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000123452 2024-08-19T14:27:32.562 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.562 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.562 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:32.562 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.235655+0000", 2024-08-19T14:27:32.562 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.003370428 2024-08-19T14:27:32.562 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.563 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.563 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:32.563 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.243630+0000", 2024-08-19T14:27:32.563 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0079749609999999992 2024-08-19T14:27:32.563 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.563 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.563 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:27:32.563 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.425783+0000", 2024-08-19T14:27:32.563 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.18215342500000001 2024-08-19T14:27:32.563 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.563 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.563 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:27:32.563 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.425829+0000", 2024-08-19T14:27:32.563 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5809999999999997e-05 2024-08-19T14:27:32.564 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.564 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.564 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:27:32.564 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.425866+0000", 2024-08-19T14:27:32.564 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6798999999999998e-05 2024-08-19T14:27:32.564 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:32.564 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:27:32.564 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:32.564 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.564 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.564 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:92839 5.f 5:f7605d59:::benchmark_data_smithi045_85309_object92838:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e445)", 2024-08-19T14:27:32.564 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:25:37.241907+0000", 2024-08-19T14:27:32.564 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 115.215508575, 2024-08-19T14:27:32.565 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20647278499999999, 2024-08-19T14:27:32.565 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:27:32.565 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:27:32.565 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:27:32.565 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:27:32.565 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:27:32.565 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:27:32.565 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 92839 2024-08-19T14:27:32.565 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.565 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:27:32.565 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.565 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:27:32.565 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.241907+0000", 2024-08-19T14:27:32.565 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:32.566 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.566 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.566 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:27:32.566 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.241907+0000", 2024-08-19T14:27:32.566 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:32.566 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.566 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.566 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:27:32.566 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.241908+0000", 2024-08-19T14:27:32.566 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4050000000000001e-06 2024-08-19T14:27:32.566 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.566 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.566 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:27:32.566 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.241947+0000", 2024-08-19T14:27:32.567 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8847999999999998e-05 2024-08-19T14:27:32.567 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.567 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.567 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:27:32.567 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.241956+0000", 2024-08-19T14:27:32.567 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.5550000000000008e-06 2024-08-19T14:27:32.567 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.567 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.567 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:27:32.567 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.241962+0000", 2024-08-19T14:27:32.567 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2490000000000001e-06 2024-08-19T14:27:32.567 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.567 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.567 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:27:32.568 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.243721+0000", 2024-08-19T14:27:32.568 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0017588910000000001 2024-08-19T14:27:32.568 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.568 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.568 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:27:32.568 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.422533+0000", 2024-08-19T14:27:32.568 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.178812148 2024-08-19T14:27:32.568 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.568 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.568 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4,11", 2024-08-19T14:27:32.568 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.422686+0000", 2024-08-19T14:27:32.568 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00015375499999999999 2024-08-19T14:27:32.568 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.568 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.569 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:27:32.569 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.426020+0000", 2024-08-19T14:27:32.569 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.003333698 2024-08-19T14:27:32.569 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.569 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.569 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:32.569 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.430495+0000", 2024-08-19T14:27:32.569 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0044751740000000002 2024-08-19T14:27:32.569 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.569 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.569 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:32.569 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.435394+0000", 2024-08-19T14:27:32.569 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0048991900000000003 2024-08-19T14:27:32.570 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.570 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.570 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:27:32.570 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.435458+0000", 2024-08-19T14:27:32.570 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3068e-05 2024-08-19T14:27:32.570 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.570 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.570 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:27:32.570 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.448379+0000", 2024-08-19T14:27:32.570 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.012921804 2024-08-19T14:27:32.570 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:32.570 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:27:32.570 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:32.571 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.571 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.571 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.25813.0:92826 5.15 e445/442 5:a965286d:::benchmark_data_smithi045_85309_object92825:head v 445'5490, mlcod=423'262)", 2024-08-19T14:27:32.571 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:25:37.245988+0000", 2024-08-19T14:27:32.571 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 115.21142689, 2024-08-19T14:27:32.571 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20615012599999999, 2024-08-19T14:27:32.571 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:27:32.571 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:27:32.571 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:27:32.571 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:27:32.571 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.571 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:27:32.571 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.245988+0000", 2024-08-19T14:27:32.571 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:32.572 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.572 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.572 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:27:32.572 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.245988+0000", 2024-08-19T14:27:32.572 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:32.572 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.572 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.572 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:27:32.572 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.245989+0000", 2024-08-19T14:27:32.572 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0210000000000001e-06 2024-08-19T14:27:32.572 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.572 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.572 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:27:32.572 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.246023+0000", 2024-08-19T14:27:32.572 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.4044999999999997e-05 2024-08-19T14:27:32.573 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.573 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.573 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:27:32.573 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.422398+0000", 2024-08-19T14:27:32.573 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.17637508399999999 2024-08-19T14:27:32.573 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.573 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.573 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:27:32.573 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.422438+0000", 2024-08-19T14:27:32.573 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9362e-05 2024-08-19T14:27:32.573 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.573 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.573 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:27:32.573 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.426229+0000", 2024-08-19T14:27:32.574 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.003791141 2024-08-19T14:27:32.574 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.574 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.574 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:27:32.574 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.426253+0000", 2024-08-19T14:27:32.574 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4159e-05 2024-08-19T14:27:32.574 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.574 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.574 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:27:32.574 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.452061+0000", 2024-08-19T14:27:32.574 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.025807672 2024-08-19T14:27:32.574 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.574 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.575 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:27:32.575 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.452138+0000", 2024-08-19T14:27:32.575 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7642e-05 2024-08-19T14:27:32.575 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:32.575 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:27:32.575 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:32.575 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.575 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.575 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:137353 5.4 5:21a8a66c:::benchmark_data_smithi045_85309_object17100:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e465)", 2024-08-19T14:27:32.575 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:26:33.514959+0000", 2024-08-19T14:27:32.575 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 58.942456399999998, 2024-08-19T14:27:32.575 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.163544353, 2024-08-19T14:27:32.575 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:27:32.575 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:27:32.575 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:27:32.576 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:27:32.576 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:27:32.576 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:27:32.577 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:27:32.577 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 137353 2024-08-19T14:27:32.577 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.577 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:27:32.577 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.577 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:27:32.577 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:26:33.514959+0000", 2024-08-19T14:27:32.577 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:32.577 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.577 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.577 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:27:32.577 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:26:33.514959+0000", 2024-08-19T14:27:32.577 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:32.578 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.578 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.578 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:27:32.578 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:26:33.514959+0000", 2024-08-19T14:27:32.578 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7000000000000004e-07 2024-08-19T14:27:32.578 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.578 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.578 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:27:32.578 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:26:33.514963+0000", 2024-08-19T14:27:32.578 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.4860000000000002e-06 2024-08-19T14:27:32.578 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.578 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.578 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:27:32.578 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:26:33.514972+0000", 2024-08-19T14:27:32.579 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6710000000000005e-06 2024-08-19T14:27:32.579 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.579 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.579 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:27:32.579 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:26:33.514976+0000", 2024-08-19T14:27:32.579 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5050000000000001e-06 2024-08-19T14:27:32.579 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.579 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.579 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:27:32.579 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:26:33.515031+0000", 2024-08-19T14:27:32.579 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5306000000000002e-05 2024-08-19T14:27:32.579 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.579 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.579 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:27:32.580 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:26:33.515112+0000", 2024-08-19T14:27:32.580 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.0171000000000002e-05 2024-08-19T14:27:32.580 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.580 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.580 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0,8", 2024-08-19T14:27:32.580 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:26:33.515178+0000", 2024-08-19T14:27:32.580 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.5963000000000005e-05 2024-08-19T14:27:32.580 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.580 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.580 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:27:32.580 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:26:33.517095+0000", 2024-08-19T14:27:32.580 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0019177510000000001 2024-08-19T14:27:32.581 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.581 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.581 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:32.581 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:26:33.517719+0000", 2024-08-19T14:27:32.581 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00062383399999999998 2024-08-19T14:27:32.581 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.581 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.581 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:32.581 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:26:33.678433+0000", 2024-08-19T14:27:32.581 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.16071344200000001 2024-08-19T14:27:32.581 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.581 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.581 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:27:32.582 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:26:33.678470+0000", 2024-08-19T14:27:32.582 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7592999999999997e-05 2024-08-19T14:27:32.582 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.582 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.582 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:27:32.582 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:26:33.678503+0000", 2024-08-19T14:27:32.582 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2960999999999998e-05 2024-08-19T14:27:32.582 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:32.582 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:27:32.582 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:32.582 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.582 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.582 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:178106 5.15 5:afb34903:::benchmark_data_smithi045_85309_object57853:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e475)", 2024-08-19T14:27:32.583 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:27:08.194118+0000", 2024-08-19T14:27:32.583 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 24.263296927999999, 2024-08-19T14:27:32.583 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.143674514, 2024-08-19T14:27:32.583 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:27:32.583 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:27:32.583 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:27:32.583 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:27:32.583 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:27:32.583 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:27:32.583 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 178106 2024-08-19T14:27:32.583 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.583 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:27:32.584 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.584 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:27:32.584 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:08.194118+0000", 2024-08-19T14:27:32.584 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:32.584 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.584 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.584 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:27:32.584 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:08.194118+0000", 2024-08-19T14:27:32.584 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:32.584 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.584 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.584 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:27:32.584 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:08.194120+0000", 2024-08-19T14:27:32.584 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.505e-06 2024-08-19T14:27:32.584 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.585 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.585 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:27:32.585 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:08.194125+0000", 2024-08-19T14:27:32.585 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6350000000000003e-06 2024-08-19T14:27:32.585 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.585 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.585 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:27:32.585 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:08.194140+0000", 2024-08-19T14:27:32.585 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4493e-05 2024-08-19T14:27:32.585 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.585 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.585 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:27:32.585 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:08.194147+0000", 2024-08-19T14:27:32.585 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8770000000000004e-06 2024-08-19T14:27:32.585 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.586 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.586 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:27:32.586 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:08.326719+0000", 2024-08-19T14:27:32.586 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.13257237799999999 2024-08-19T14:27:32.586 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.586 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.586 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:27:32.586 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:08.326847+0000", 2024-08-19T14:27:32.586 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012791199999999999 2024-08-19T14:27:32.586 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.586 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.586 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 1,8", 2024-08-19T14:27:32.586 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:08.326954+0000", 2024-08-19T14:27:32.587 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00010745 2024-08-19T14:27:32.587 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.587 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.587 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:32.587 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:08.332374+0000", 2024-08-19T14:27:32.587 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0054192959999999997 2024-08-19T14:27:32.587 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.587 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.587 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:32.587 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:08.337456+0000", 2024-08-19T14:27:32.587 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0050822100000000002 2024-08-19T14:27:32.587 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.587 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.587 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:27:32.588 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:08.337600+0000", 2024-08-19T14:27:32.588 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014429699999999999 2024-08-19T14:27:32.588 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.588 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.588 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:27:32.588 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:08.337744+0000", 2024-08-19T14:27:32.588 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000143824 2024-08-19T14:27:32.588 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:32.588 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:32.588 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:27:32.588 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:08.337793+0000", 2024-08-19T14:27:32.588 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8637000000000002e-05 2024-08-19T14:27:32.588 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:32.588 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:27:32.589 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:32.589 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:32.589 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:27:32.589 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:27:32.631 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:32.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:32.632 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:27:32.733 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:27:32.760 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:32.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:32.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:27:32.861 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:27:32.895 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:32.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:32.896 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:27:32.996 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:27:33.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:32 smithi032 ceph-mon[20004]: 1.12 scrub starts 2024-08-19T14:27:33.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:32 smithi032 ceph-mon[20004]: 1.12 scrub ok 2024-08-19T14:27:33.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:32 smithi032 ceph-mon[20004]: 1.10 scrub starts 2024-08-19T14:27:33.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:32 smithi032 ceph-mon[20004]: 1.10 scrub ok 2024-08-19T14:27:33.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:32 smithi032 ceph-mon[20004]: 1.8 scrub starts 2024-08-19T14:27:33.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:32 smithi032 ceph-mon[20004]: 1.8 scrub ok 2024-08-19T14:27:33.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:33.029 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:33.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:27:33.102 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:27:33.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:32 smithi031 ceph-mon[16336]: 1.12 scrub starts 2024-08-19T14:27:33.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:32 smithi031 ceph-mon[16336]: 1.12 scrub ok 2024-08-19T14:27:33.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:32 smithi031 ceph-mon[16336]: 1.10 scrub starts 2024-08-19T14:27:33.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:32 smithi031 ceph-mon[16336]: 1.10 scrub ok 2024-08-19T14:27:33.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:32 smithi031 ceph-mon[16336]: 1.8 scrub starts 2024-08-19T14:27:33.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:32 smithi031 ceph-mon[16336]: 1.8 scrub ok 2024-08-19T14:27:33.130 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:27:33.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:33.168 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:33.168 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:27:33.269 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:27:33.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:32 smithi045 ceph-mon[20143]: 1.12 scrub starts 2024-08-19T14:27:33.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:32 smithi045 ceph-mon[20143]: 1.12 scrub ok 2024-08-19T14:27:33.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:32 smithi045 ceph-mon[20143]: 1.10 scrub starts 2024-08-19T14:27:33.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:32 smithi045 ceph-mon[20143]: 1.10 scrub ok 2024-08-19T14:27:33.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:32 smithi045 ceph-mon[20143]: 1.8 scrub starts 2024-08-19T14:27:33.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:32 smithi045 ceph-mon[20143]: 1.8 scrub ok 2024-08-19T14:27:33.307 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:33.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:33.308 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:27:33.409 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:27:33.439 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:33.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:33.441 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:27:33.542 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:27:33.566 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:33.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:33.567 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:27:33.668 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:27:33.700 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:33.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:33.701 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:27:33.802 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:27:33.835 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:33.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:33.836 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:27:33.937 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:27:33.967 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:33.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:33.968 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:27:34.069 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:27:34.095 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:34.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:34.096 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:27:34.108 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:33 smithi045 ceph-mon[20143]: pgmap v1114: 57 pgs: 57 active+clean; 2.7 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 1.17k op/s 2024-08-19T14:27:34.108 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:33 smithi045 ceph-mon[20143]: 1.f scrub starts 2024-08-19T14:27:34.108 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:33 smithi045 ceph-mon[20143]: 1.f scrub ok 2024-08-19T14:27:34.108 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:33 smithi045 ceph-mon[20143]: 1.18 deep-scrub starts 2024-08-19T14:27:34.108 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:33 smithi045 ceph-mon[20143]: 1.18 deep-scrub ok 2024-08-19T14:27:34.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:33 smithi031 ceph-mon[16336]: pgmap v1114: 57 pgs: 57 active+clean; 2.7 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 1.17k op/s 2024-08-19T14:27:34.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:33 smithi031 ceph-mon[16336]: 1.f scrub starts 2024-08-19T14:27:34.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:33 smithi031 ceph-mon[16336]: 1.f scrub ok 2024-08-19T14:27:34.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:33 smithi031 ceph-mon[16336]: 1.18 deep-scrub starts 2024-08-19T14:27:34.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:33 smithi031 ceph-mon[16336]: 1.18 deep-scrub ok 2024-08-19T14:27:34.197 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:27:34.237 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:34.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:34.238 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:27:34.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:33 smithi032 ceph-mon[20004]: pgmap v1114: 57 pgs: 57 active+clean; 2.7 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 1.17k op/s 2024-08-19T14:27:34.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:33 smithi032 ceph-mon[20004]: 1.f scrub starts 2024-08-19T14:27:34.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:33 smithi032 ceph-mon[20004]: 1.f scrub ok 2024-08-19T14:27:34.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:33 smithi032 ceph-mon[20004]: 1.18 deep-scrub starts 2024-08-19T14:27:34.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:33 smithi032 ceph-mon[20004]: 1.18 deep-scrub ok 2024-08-19T14:27:34.339 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:27:34.368 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:34.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:34.369 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:27:34.389 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-08-19T14:27:34.470 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:27:34.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:34.509 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:34.509 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:27:34.610 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:27:34.641 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:34.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:34.642 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:27:34.743 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:27:34.769 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:34.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:34.770 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:27:34.838 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:27:34.858 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is set 2024-08-19T14:27:34.871 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:27:34.896 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:34.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:34.897 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:27:34.990 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:27:34.998 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:27:35.020 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:35.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:35.021 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:27:35.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:34 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3463499597' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:27:35.121 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:27:35.137 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:34 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3463499597' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:27:35.165 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:35.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:35.166 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:27:35.267 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:27:35.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:34 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3463499597' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:27:35.306 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:35.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:35.307 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:27:35.408 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:27:35.437 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:35.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:35.438 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:27:35.507 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:27:35.507 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-08-19T14:27:35.507 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-08-19T14:27:35.507 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:27:35.539 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:27:35.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:35.580 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:35.580 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:27:35.681 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:27:35.712 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:35.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:35.713 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:27:35.763 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:27:35.814 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:27:35.849 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:35.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:35.850 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:27:35.951 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:27:35.984 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:35.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:35.986 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:27:36.087 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:27:36.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:35 smithi031 ceph-mon[16336]: pgmap v1115: 57 pgs: 57 active+clean; 2.5 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 1.17k op/s; 0 B/s, 0 objects/s recovering 2024-08-19T14:27:36.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:35 smithi031 ceph-mon[16336]: 5.0 scrub starts 2024-08-19T14:27:36.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:35 smithi031 ceph-mon[16336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:27:36.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:35 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3463499597' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:27:36.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:35 smithi031 ceph-mon[16336]: osdmap e486: 12 total, 12 up, 9 in 2024-08-19T14:27:36.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:36.120 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:36.121 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:27:36.222 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:27:36.252 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:36.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:36.253 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:27:36.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:35 smithi032 ceph-mon[20004]: pgmap v1115: 57 pgs: 57 active+clean; 2.5 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 1.17k op/s; 0 B/s, 0 objects/s recovering 2024-08-19T14:27:36.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:35 smithi032 ceph-mon[20004]: 5.0 scrub starts 2024-08-19T14:27:36.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:35 smithi032 ceph-mon[20004]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:27:36.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:35 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3463499597' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:27:36.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:35 smithi032 ceph-mon[20004]: osdmap e486: 12 total, 12 up, 9 in 2024-08-19T14:27:36.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:35 smithi045 ceph-mon[20143]: pgmap v1115: 57 pgs: 57 active+clean; 2.5 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 1.17k op/s; 0 B/s, 0 objects/s recovering 2024-08-19T14:27:36.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:35 smithi045 ceph-mon[20143]: 5.0 scrub starts 2024-08-19T14:27:36.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:35 smithi045 ceph-mon[20143]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:27:36.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:35 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3463499597' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:27:36.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:35 smithi045 ceph-mon[20143]: osdmap e486: 12 total, 12 up, 9 in 2024-08-19T14:27:36.354 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:27:36.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:36.397 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:36.398 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:27:36.490 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:27:36.491 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:27:36.491 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:27:36.491 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:27:36.491 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:27:36.491 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:27:36.499 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:27:36.512 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-08-19T14:27:36.526 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:36.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:36.527 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:27:36.628 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:27:36.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:36.659 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:36.659 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:27:36.712 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:27:36.713 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:27:36.745 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:27:36.745 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:27:36.759 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:27:36.779 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:27:36.780 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:27:36.791 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:36.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:36.793 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:27:36.807 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:27:36.808 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:27:36.850 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:27:36.873 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:27:36.873 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:27:36.894 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:27:36.901 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:27:36.901 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:27:36.927 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:27:36.928 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:27:36.928 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:27:36.951 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:36.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:36.952 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:27:36.953 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:27:36.954 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:27:36.969 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:36 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3038228690' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:27:37.053 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:27:37.074 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:37.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:37.075 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:27:37.176 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:27:37.207 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:37.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:37.208 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:27:37.231 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:36 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3038228690' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:27:37.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:36 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3038228690' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:27:37.308 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:27:37.348 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:37.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:37.349 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:27:37.450 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:27:37.482 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:37.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:37.483 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:27:37.534 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 7, 8, 9, 11] out_osds: [10, 6, 2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-08-19T14:27:37.534 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-19T14:27:37.535 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-08-19T14:27:37.535 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-19T14:27:37.542 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:27:37.542 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:27:37.583 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:27:37.608 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:37.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:37.609 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:27:37.636 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-19T14:27:37.710 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:27:37.746 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:37.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:37.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:27:37.848 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:27:37.876 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:37.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:37.877 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:27:37.978 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:27:38.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:38.012 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:38.012 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:27:38.113 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:27:38.142 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:38.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:38.143 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:27:38.196 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:37 smithi032 ceph-mon[20004]: pgmap v1117: 57 pgs: 57 active+clean; 2.4 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 1.20k op/s; 0 B/s, 0 objects/s recovering 2024-08-19T14:27:38.240 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-08-19T14:27:38.244 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:27:38.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:37 smithi045 ceph-mon[20143]: pgmap v1117: 57 pgs: 57 active+clean; 2.4 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 1.20k op/s; 0 B/s, 0 objects/s recovering 2024-08-19T14:27:38.321 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:38.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:38.322 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:27:38.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:37 smithi031 ceph-mon[16336]: pgmap v1117: 57 pgs: 57 active+clean; 2.4 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 1.20k op/s; 0 B/s, 0 objects/s recovering 2024-08-19T14:27:38.422 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:27:38.446 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:38.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:38.447 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:27:38.548 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:27:38.577 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:38.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:38.578 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:27:38.620 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:27:38.679 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:27:38.700 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:38.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:38.701 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:27:38.802 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:27:38.830 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:38.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:38.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:27:38.932 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:27:38.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:38.979 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:38.979 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:27:39.080 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:27:39.115 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:39.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:39.116 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:27:39.217 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:27:39.252 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:39.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:39.253 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:27:39.262 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:27:39.262 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:27:39.354 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:27:39.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:39.391 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:39.391 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:27:39.492 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:27:39.526 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:39.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:39.527 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:27:39.628 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:27:39.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:39.650 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:39.650 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:27:39.751 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:27:39.779 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:39.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:39.780 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:27:39.881 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:27:39.911 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:39.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:39.912 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:27:40.012 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:27:40.034 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:40.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:40.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:27:40.136 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:27:40.157 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:40.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:40.158 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:27:40.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:39 smithi032 ceph-mon[20004]: pgmap v1118: 57 pgs: 57 active+clean; 2.3 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 1.11k op/s; 0 B/s, 0 objects/s recovering 2024-08-19T14:27:40.259 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:27:40.292 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:40.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:40.293 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:27:40.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:39 smithi045 ceph-mon[20143]: pgmap v1118: 57 pgs: 57 active+clean; 2.3 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 1.11k op/s; 0 B/s, 0 objects/s recovering 2024-08-19T14:27:40.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:39 smithi031 ceph-mon[16336]: pgmap v1118: 57 pgs: 57 active+clean; 2.3 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 1.11k op/s; 0 B/s, 0 objects/s recovering 2024-08-19T14:27:40.394 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:27:40.421 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:40.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:40.422 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:27:40.523 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:27:40.553 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:40.557 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:27:40.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:40.557 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:27:40.574 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:27:40.658 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:27:40.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:40.695 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:40.695 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:27:40.716 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-08-19T14:27:40.796 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:27:40.821 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:40.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:40.822 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:27:40.924 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:27:40.948 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:40 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:27:40.948 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:40 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:27:40.952 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:40.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:40.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:27:41.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:40 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:27:41.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:40 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:27:41.054 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:27:41.079 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:41.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:41.080 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:27:41.161 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:27:41.181 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:27:41.208 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:41.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:41.209 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:27:41.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:40 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:27:41.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:40 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:27:41.309 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:27:41.338 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:41.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:41.339 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:27:41.440 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:27:41.473 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:41.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:41.474 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:27:41.575 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:27:41.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:41.607 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:41.607 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:27:41.708 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:27:41.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:41.747 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:41.748 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:27:41.849 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:27:41.877 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:41.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:41.879 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:27:41.892 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:27:41.892 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:27:41.897 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is set 2024-08-19T14:27:41.980 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:27:41.981 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-08-19T14:27:41.983 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:27:42.015 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:42.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:42.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:27:42.117 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:27:42.147 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:42.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:42.148 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:27:42.249 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:27:42.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:41 smithi032 ceph-mon[20004]: pgmap v1119: 57 pgs: 57 active+clean; 2.1 GiB data, 12 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 1.18k op/s; 0 B/s, 0 objects/s recovering 2024-08-19T14:27:42.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:41 smithi032 ceph-mon[20004]: osdmap e487: 12 total, 12 up, 9 in 2024-08-19T14:27:42.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:41 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3801705618' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:27:42.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:41 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:27:42.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:42.282 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:42.282 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:27:42.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:41 smithi045 ceph-mon[20143]: pgmap v1119: 57 pgs: 57 active+clean; 2.1 GiB data, 12 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 1.18k op/s; 0 B/s, 0 objects/s recovering 2024-08-19T14:27:42.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:41 smithi045 ceph-mon[20143]: osdmap e487: 12 total, 12 up, 9 in 2024-08-19T14:27:42.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:41 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3801705618' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:27:42.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:41 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:27:42.301 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:41 smithi031 ceph-mon[16336]: pgmap v1119: 57 pgs: 57 active+clean; 2.1 GiB data, 12 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 1.18k op/s; 0 B/s, 0 objects/s recovering 2024-08-19T14:27:42.301 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:41 smithi031 ceph-mon[16336]: osdmap e487: 12 total, 12 up, 9 in 2024-08-19T14:27:42.301 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:41 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3801705618' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:27:42.301 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:41 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:27:42.383 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:27:42.415 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:42.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:42.416 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:27:42.486 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:27:42.516 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:27:42.549 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:42.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:42.550 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:27:42.640 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-08-19T14:27:42.652 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:27:42.680 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:42.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:42.682 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:27:42.782 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:27:42.810 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:42.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:42.811 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:27:42.912 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:27:42.941 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:42.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:42.942 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:27:43.042 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:27:43.053 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:42 smithi031 ceph-mon[16336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:27:43.053 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:42 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:27:43.053 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:42 smithi031 ceph-mon[16336]: osdmap e488: 12 total, 12 up, 9 in 2024-08-19T14:27:43.054 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:42 smithi031 ceph-mon[16336]: from='client.26141 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:27:43.059 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:27:43.076 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:43.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:43.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:27:43.178 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:27:43.197 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:42 smithi032 ceph-mon[20004]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:27:43.197 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:42 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:27:43.198 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:42 smithi032 ceph-mon[20004]: osdmap e488: 12 total, 12 up, 9 in 2024-08-19T14:27:43.198 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:42 smithi032 ceph-mon[20004]: from='client.26141 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:27:43.211 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:43.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:43.212 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:27:43.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:42 smithi045 ceph-mon[20143]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:27:43.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:42 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:27:43.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:42 smithi045 ceph-mon[20143]: osdmap e488: 12 total, 12 up, 9 in 2024-08-19T14:27:43.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:42 smithi045 ceph-mon[20143]: from='client.26141 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:27:43.313 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:27:43.350 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:43.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:43.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:27:43.452 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:27:43.463 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":1121,"stamp":"2024-08-19T14:27:41.891915+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2174157368,"num_objects":33169,"num_object_clones":0,"num_object_copies":99507,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33169,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":122357,"num_write_kb":3398593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":251772,"ondisk_log_size":251772,"up":171,"acting":171,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":178,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":12724048,"kb_used_data":6993028,"kb_used_omap":14,"kb_used_meta":5730929,"kb_avail":830982320,"statfs":{"total":863955320832,"available":850925895680,"internally_reserved":0,"allocated":7160860672,"data_stored":7141096624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14353,"internal_metadata":5868472303},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,4,4,3,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-500301824,"num_objects":-7634,"num_object_clones":0,"num_object_copies":-22902,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-7634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7634,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.890920"},"pg_stats":[{"pgid":"5.4","version":"486'9182","reported_seq":9404,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-08-19T14:27:39.908928+0000","last_change":"2024-08-19T14:27:31.858648+0000","last_active":"2024-08-19T14:27:39.908928+0000","last_peered":"2024-08-19T14:27:39.908928+0000","last_clean":"2024-08-19T14:27:39.908928+0000","last_became_active":"2024-08-19T14:27:31.836653+0000","last_became_peered":"2024-08-19T14:27:31.836653+0000","last_unstale":"2024-08-19T14:27:39.908928+0000","last_undegraded":"2024-08-19T14:27:39.908928+0000","last_fullsized":"2024-08-19T14:27:39.908928+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'8317","last_scrub_stamp":"2024-08-19T14:27:12.684467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:27:12.684467+0000","objects_scrubbed":1938,"log_size":9182,"log_dups_size":0,"ondisk_log_size":9182,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:22.704448+0000","scrub_duration":382,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69926912,"num_objects":1067,"num_object_clones":0,"num_object_copies":3201,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1067,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3737,"num_write_kb":65152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,0],"acting":[8,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.0","version":"70'139","reported_seq":1215,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-08-19T14:27:34.871706+0000","last_change":"2024-08-19T14:27:03.929122+0000","last_active":"2024-08-19T14:27:34.871706+0000","last_peered":"2024-08-19T14:27:34.871706+0000","last_clean":"2024-08-19T14:27:34.871706+0000","last_became_active":"2024-08-19T14:18:13.807133+0000","last_became_peered":"2024-08-19T14:18:13.807133+0000","last_unstale":"2024-08-19T14:27:34.871706+0000","last_undegraded":"2024-08-19T14:27:34.871706+0000","last_fullsized":"2024-08-19T14:27:34.871706+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:27:03.929012+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:19:58.072083+0000","last_clean_scrub_stamp":"2024-08-19T14:27:03.929012+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:27.998267+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,9],"acting":[7,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"486'9253","reported_seq":9471,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-08-19T14:27:38.642929+0000","last_change":"2024-08-19T14:27:31.862872+0000","last_active":"2024-08-19T14:27:38.642929+0000","last_peered":"2024-08-19T14:27:38.642929+0000","last_clean":"2024-08-19T14:27:38.642929+0000","last_became_active":"2024-08-19T14:27:31.839708+0000","last_became_peered":"2024-08-19T14:27:31.839708+0000","last_unstale":"2024-08-19T14:27:38.642929+0000","last_undegraded":"2024-08-19T14:27:38.642929+0000","last_fullsized":"2024-08-19T14:27:38.642929+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"482'8807","last_scrub_stamp":"2024-08-19T14:27:26.742962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:27:26.742962+0000","objects_scrubbed":1505,"log_size":9253,"log_dups_size":0,"ondisk_log_size":9253,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:32.743963+0000","scrub_duration":429,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68943872,"num_objects":1052,"num_object_clones":0,"num_object_copies":3156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1052,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3692,"num_write_kb":62848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,9],"acting":[1,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"70'139","reported_seq":1113,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-08-19T14:27:34.866971+0000","last_change":"2024-08-19T14:27:04.891618+0000","last_active":"2024-08-19T14:27:34.866971+0000","last_peered":"2024-08-19T14:27:34.866971+0000","last_clean":"2024-08-19T14:27:34.866971+0000","last_became_active":"2024-08-19T14:18:13.804721+0000","last_became_peered":"2024-08-19T14:18:13.804721+0000","last_unstale":"2024-08-19T14:27:34.866971+0000","last_undegraded":"2024-08-19T14:27:34.866971+0000","last_fullsized":"2024-08-19T14:27:34.866971+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:27:04.891524+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:27:04.891524+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:28.108665+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"486'9314","reported_seq":9504,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-08-19T14:27:38.796627+0000","last_change":"2024-08-19T14:27:31.857176+0000","last_active":"2024-08-19T14:27:38.796627+0000","last_peered":"2024-08-19T14:27:38.796627+0000","last_clean":"2024-08-19T14:27:38.796627+0000","last_became_active":"2024-08-19T14:27:31.839054+0000","last_became_peered":"2024-08-19T14:27:31.839054+0000","last_unstale":"2024-08-19T14:27:38.796627+0000","last_undegraded":"2024-08-19T14:27:38.796627+0000","last_fullsized":"2024-08-19T14:27:38.796627+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"474'8126","last_scrub_stamp":"2024-08-19T14:26:59.955550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:26:59.955550+0000","objects_scrubbed":2208,"log_size":9314,"log_dups_size":0,"ondisk_log_size":9314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:09.277471+0000","scrub_duration":558,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66322432,"num_objects":1012,"num_object_clones":0,"num_object_copies":3036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1012,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1196,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"70'139","reported_seq":1062,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-08-19T14:27:34.874733+0000","last_change":"2024-08-19T14:26:57.876424+0000","last_active":"2024-08-19T14:27:34.874733+0000","last_peered":"2024-08-19T14:27:34.874733+0000","last_clean":"2024-08-19T14:27:34.874733+0000","last_became_active":"2024-08-19T14:20:50.742354+0000","last_became_peered":"2024-08-19T14:20:50.742354+0000","last_unstale":"2024-08-19T14:27:34.874733+0000","last_undegraded":"2024-08-19T14:27:34.874733+0000","last_fullsized":"2024-08-19T14:27:34.874733+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:26:57.876316+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:19:59.302931+0000","last_clean_scrub_stamp":"2024-08-19T14:26:57.876316+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:00.325583+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.5","version":"486'8898","reported_seq":9114,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-08-19T14:27:37.010124+0000","last_change":"2024-08-19T14:27:04.288663+0000","last_active":"2024-08-19T14:27:37.010124+0000","last_peered":"2024-08-19T14:27:37.010124+0000","last_clean":"2024-08-19T14:27:37.010124+0000","last_became_active":"2024-08-19T14:25:29.826953+0000","last_became_peered":"2024-08-19T14:25:29.826953+0000","last_unstale":"2024-08-19T14:27:37.010124+0000","last_undegraded":"2024-08-19T14:27:37.010124+0000","last_fullsized":"2024-08-19T14:27:37.010124+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":0,"last_scrub":"474'7988","last_scrub_stamp":"2024-08-19T14:27:04.288536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:27:04.288536+0000","objects_scrubbed":2000,"log_size":8898,"log_dups_size":0,"ondisk_log_size":8898,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:06.060755+0000","scrub_duration":326,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71237632,"num_objects":1087,"num_object_clones":0,"num_object_copies":3261,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1087,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3529,"num_write_kb":62080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"70'139","reported_seq":307,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-08-19T14:27:34.871104+0000","last_change":"2024-08-19T14:27:27.108572+0000","last_active":"2024-08-19T14:27:34.871104+0000","last_peered":"2024-08-19T14:27:34.871104+0000","last_clean":"2024-08-19T14:27:34.871104+0000","last_became_active":"2024-08-19T14:27:15.696041+0000","last_became_peered":"2024-08-19T14:27:15.696041+0000","last_unstale":"2024-08-19T14:27:34.871104+0000","last_undegraded":"2024-08-19T14:27:34.871104+0000","last_fullsized":"2024-08-19T14:27:34.871104+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:27:27.108504+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:27:27.108504+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:44.170107+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,11],"acting":[0,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"486'8955","reported_seq":9172,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-08-19T14:27:37.046644+0000","last_change":"2024-08-19T14:27:03.375077+0000","last_active":"2024-08-19T14:27:37.046644+0000","last_peered":"2024-08-19T14:27:37.046644+0000","last_clean":"2024-08-19T14:27:37.046644+0000","last_became_active":"2024-08-19T14:25:29.851182+0000","last_became_peered":"2024-08-19T14:25:29.851182+0000","last_unstale":"2024-08-19T14:27:37.046644+0000","last_undegraded":"2024-08-19T14:27:37.046644+0000","last_fullsized":"2024-08-19T14:27:37.046644+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":0,"last_scrub":"474'8002","last_scrub_stamp":"2024-08-19T14:27:03.374935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:27:03.374935+0000","objects_scrubbed":2068,"log_size":8955,"log_dups_size":0,"ondisk_log_size":8955,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:04.049022+0000","scrub_duration":396,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72613888,"num_objects":1108,"num_object_clones":0,"num_object_copies":3324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3545,"num_write_kb":62528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"70'139","reported_seq":762,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-08-19T14:27:34.867377+0000","last_change":"2024-08-19T14:27:03.234463+0000","last_active":"2024-08-19T14:27:34.867377+0000","last_peered":"2024-08-19T14:27:34.867377+0000","last_clean":"2024-08-19T14:27:34.867377+0000","last_became_active":"2024-08-19T14:20:28.204384+0000","last_became_peered":"2024-08-19T14:20:28.204384+0000","last_unstale":"2024-08-19T14:27:34.867377+0000","last_undegraded":"2024-08-19T14:27:34.867377+0000","last_fullsized":"2024-08-19T14:27:34.867377+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:27:03.234352+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:21:46.057624+0000","last_clean_scrub_stamp":"2024-08-19T14:27:03.234352+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:26.281527+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"486'9071","reported_seq":9258,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-08-19T14:27:37.156388+0000","last_change":"2024-08-19T14:26:56.652316+0000","last_active":"2024-08-19T14:27:37.156388+0000","last_peered":"2024-08-19T14:27:37.156388+0000","last_clean":"2024-08-19T14:27:37.156388+0000","last_became_active":"2024-08-19T14:25:29.831300+0000","last_became_peered":"2024-08-19T14:25:29.831300+0000","last_unstale":"2024-08-19T14:27:37.156388+0000","last_undegraded":"2024-08-19T14:27:37.156388+0000","last_fullsized":"2024-08-19T14:27:37.156388+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'7842","last_scrub_stamp":"2024-08-19T14:26:56.652182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:26:56.652182+0000","objects_scrubbed":2335,"log_size":9071,"log_dups_size":0,"ondisk_log_size":9071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:21.680001+0000","scrub_duration":607,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72024064,"num_objects":1099,"num_object_clones":0,"num_object_copies":3297,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1099,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1236,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,8],"acting":[0,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"70'139","reported_seq":307,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-08-19T14:27:34.867527+0000","last_change":"2024-08-19T14:27:26.969984+0000","last_active":"2024-08-19T14:27:34.867527+0000","last_peered":"2024-08-19T14:27:34.867527+0000","last_clean":"2024-08-19T14:27:34.867527+0000","last_became_active":"2024-08-19T14:27:15.697224+0000","last_became_peered":"2024-08-19T14:27:15.697224+0000","last_unstale":"2024-08-19T14:27:34.867527+0000","last_undegraded":"2024-08-19T14:27:34.867527+0000","last_fullsized":"2024-08-19T14:27:34.867527+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:27:26.969901+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:27.848870+0000","last_clean_scrub_stamp":"2024-08-19T14:27:26.969901+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:52.544619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"5.1","version":"486'11282","reported_seq":11560,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-08-19T14:27:37.955611+0000","last_change":"2024-08-19T14:27:14.776017+0000","last_active":"2024-08-19T14:27:37.955611+0000","last_peered":"2024-08-19T14:27:37.955611+0000","last_clean":"2024-08-19T14:27:37.955611+0000","last_became_active":"2024-08-19T14:25:29.820527+0000","last_became_peered":"2024-08-19T14:25:29.820527+0000","last_unstale":"2024-08-19T14:27:37.955611+0000","last_undegraded":"2024-08-19T14:27:37.955611+0000","last_fullsized":"2024-08-19T14:27:37.955611+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":0,"last_scrub":"477'10484","last_scrub_stamp":"2024-08-19T14:27:14.775877+0000","last_deep_scrub":"477'10484","last_deep_scrub_stamp":"2024-08-19T14:27:14.775877+0000","last_clean_scrub_stamp":"2024-08-19T14:27:14.775877+0000","objects_scrubbed":1945,"log_size":11282,"log_dups_size":0,"ondisk_log_size":11282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":10,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:22.349281+0000","scrub_duration":9182,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70909952,"num_objects":1082,"num_object_clones":0,"num_object_copies":3246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1082,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3686,"num_write_kb":61248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"70'139","reported_seq":938,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-08-19T14:27:40.883519+0000","last_change":"2024-08-19T14:26:56.775790+0000","last_active":"2024-08-19T14:27:40.883519+0000","last_peered":"2024-08-19T14:27:40.883519+0000","last_clean":"2024-08-19T14:27:40.883519+0000","last_became_active":"2024-08-19T14:18:26.881435+0000","last_became_peered":"2024-08-19T14:18:26.881435+0000","last_unstale":"2024-08-19T14:27:40.883519+0000","last_undegraded":"2024-08-19T14:27:40.883519+0000","last_fullsized":"2024-08-19T14:27:40.883519+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:26:56.775676+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:26:56.775676+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:04.024892+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,4],"acting":[11,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.0","version":"486'9345","reported_seq":9538,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-08-19T14:27:38.502182+0000","last_change":"2024-08-19T14:27:34.897610+0000","last_active":"2024-08-19T14:27:38.502182+0000","last_peered":"2024-08-19T14:27:38.502182+0000","last_clean":"2024-08-19T14:27:38.502182+0000","last_became_active":"2024-08-19T14:25:29.850301+0000","last_became_peered":"2024-08-19T14:25:29.850301+0000","last_unstale":"2024-08-19T14:27:38.502182+0000","last_undegraded":"2024-08-19T14:27:38.502182+0000","last_fullsized":"2024-08-19T14:27:38.502182+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'5592","last_scrub_stamp":"2024-08-19T14:25:32.103399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:25:32.103399+0000","objects_scrubbed":99,"log_size":9345,"log_dups_size":0,"ondisk_log_size":9345,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:27:39.869362+0000","scrub_duration":317,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69271552,"num_objects":1057,"num_object_clones":0,"num_object_copies":3171,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1057,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3753,"num_write_kb":62528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"70'139","reported_seq":1163,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-08-19T14:27:40.883663+0000","last_change":"2024-08-19T14:26:55.764218+0000","last_active":"2024-08-19T14:27:40.883663+0000","last_peered":"2024-08-19T14:27:40.883663+0000","last_clean":"2024-08-19T14:27:40.883663+0000","last_became_active":"2024-08-19T14:18:13.806013+0000","last_became_peered":"2024-08-19T14:18:13.806013+0000","last_unstale":"2024-08-19T14:27:40.883663+0000","last_undegraded":"2024-08-19T14:27:40.883663+0000","last_fullsized":"2024-08-19T14:27:40.883663+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:26:55.764116+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:26:55.764116+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:24.552401+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.e","version":"486'13017","reported_seq":13220,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-08-19T14:27:40.018924+0000","last_change":"2024-08-19T14:27:15.741425+0000","last_active":"2024-08-19T14:27:40.018924+0000","last_peered":"2024-08-19T14:27:40.018924+0000","last_clean":"2024-08-19T14:27:40.018924+0000","last_became_active":"2024-08-19T14:27:15.697672+0000","last_became_peered":"2024-08-19T14:27:15.697672+0000","last_unstale":"2024-08-19T14:27:40.018924+0000","last_undegraded":"2024-08-19T14:27:40.018924+0000","last_fullsized":"2024-08-19T14:27:40.018924+0000","mapping_epoch":477,"log_start":"430'3000","ondisk_log_start":"430'3000","created":419,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":0,"last_scrub":"474'10948","last_scrub_stamp":"2024-08-19T14:27:04.056561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:27:04.056561+0000","objects_scrubbed":4147,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:24.468871+0000","scrub_duration":1025,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":134152192,"num_objects":2047,"num_object_clones":0,"num_object_copies":6141,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2047,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13017,"num_write_kb":482048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.a","version":"70'139","reported_seq":307,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-08-19T14:27:34.868250+0000","last_change":"2024-08-19T14:27:25.888737+0000","last_active":"2024-08-19T14:27:34.868250+0000","last_peered":"2024-08-19T14:27:34.868250+0000","last_clean":"2024-08-19T14:27:34.868250+0000","last_became_active":"2024-08-19T14:27:15.698792+0000","last_became_peered":"2024-08-19T14:27:15.698792+0000","last_unstale":"2024-08-19T14:27:34.868250+0000","last_undegraded":"2024-08-19T14:27:34.868250+0000","last_fullsized":"2024-08-19T14:27:34.868250+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:27:25.888493+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:28.864870+0000","last_clean_scrub_stamp":"2024-08-19T14:27:25.888493+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:27.170057+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,0],"acting":[4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"487'15909","reported_seq":16204,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-08-19T14:27:40.980104+0000","last_change":"2024-08-19T14:27:30.378381+0000","last_active":"2024-08-19T14:27:40.980104+0000","last_peered":"2024-08-19T14:27:40.980104+0000","last_clean":"2024-08-19T14:27:40.980104+0000","last_became_active":"2024-08-19T14:24:44.592098+0000","last_became_peered":"2024-08-19T14:24:44.592098+0000","last_unstale":"2024-08-19T14:27:40.980104+0000","last_undegraded":"2024-08-19T14:27:40.980104+0000","last_fullsized":"2024-08-19T14:27:40.980104+0000","mapping_epoch":419,"log_start":"444'5900","ondisk_log_start":"444'5900","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"483'15174","last_scrub_stamp":"2024-08-19T14:27:30.378045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:27:30.378045+0000","objects_scrubbed":2739,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:35.644617+0000","scrub_duration":1017,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":129761304,"num_objects":1981,"num_object_clones":0,"num_object_copies":5943,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1981,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13361,"num_write_kb":490881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7],"acting":[11,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.9","version":"70'139","reported_seq":1237,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-08-19T14:27:34.877159+0000","last_change":"2024-08-19T14:27:05.872064+0000","last_active":"2024-08-19T14:27:34.877159+0000","last_peered":"2024-08-19T14:27:34.877159+0000","last_clean":"2024-08-19T14:27:34.877159+0000","last_became_active":"2024-08-19T14:18:26.887657+0000","last_became_peered":"2024-08-19T14:18:26.887657+0000","last_unstale":"2024-08-19T14:27:34.877159+0000","last_undegraded":"2024-08-19T14:27:34.877159+0000","last_fullsized":"2024-08-19T14:27:34.877159+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:27:05.871950+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:27:05.871950+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:32.663051+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"486'13033","reported_seq":13226,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-08-19T14:27:38.662360+0000","last_change":"2024-08-19T14:27:11.671223+0000","last_active":"2024-08-19T14:27:38.662360+0000","last_peered":"2024-08-19T14:27:38.662360+0000","last_clean":"2024-08-19T14:27:38.662360+0000","last_became_active":"2024-08-19T14:24:44.595538+0000","last_became_peered":"2024-08-19T14:24:44.595538+0000","last_unstale":"2024-08-19T14:27:38.662360+0000","last_undegraded":"2024-08-19T14:27:38.662360+0000","last_fullsized":"2024-08-19T14:27:38.662360+0000","mapping_epoch":419,"log_start":"430'3000","ondisk_log_start":"430'3000","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'11344","last_scrub_stamp":"2024-08-19T14:27:11.670737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:27:11.670737+0000","objects_scrubbed":3780,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:29.589644+0000","scrub_duration":775,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":136380416,"num_objects":2081,"num_object_clones":0,"num_object_copies":6243,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2081,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13033,"num_write_kb":483648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"70'139","reported_seq":293,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-08-19T14:27:34.867203+0000","last_change":"2024-08-19T14:27:32.221695+0000","last_active":"2024-08-19T14:27:34.867203+0000","last_peered":"2024-08-19T14:27:34.867203+0000","last_clean":"2024-08-19T14:27:34.867203+0000","last_became_active":"2024-08-19T14:27:31.834018+0000","last_became_peered":"2024-08-19T14:27:31.834018+0000","last_unstale":"2024-08-19T14:27:34.867203+0000","last_undegraded":"2024-08-19T14:27:34.867203+0000","last_fullsized":"2024-08-19T14:27:34.867203+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:27:32.221578+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:27:32.221578+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:59.258809+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,8],"acting":[3,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"486'12669","reported_seq":12875,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-08-19T14:27:37.998053+0000","last_change":"2024-08-19T14:27:31.957791+0000","last_active":"2024-08-19T14:27:37.998053+0000","last_peered":"2024-08-19T14:27:37.998053+0000","last_clean":"2024-08-19T14:27:37.998053+0000","last_became_active":"2024-08-19T14:27:31.855047+0000","last_became_peered":"2024-08-19T14:27:31.855047+0000","last_unstale":"2024-08-19T14:27:37.998053+0000","last_undegraded":"2024-08-19T14:27:37.998053+0000","last_fullsized":"2024-08-19T14:27:37.998053+0000","mapping_epoch":484,"log_start":"429'2600","ondisk_log_start":"429'2600","created":419,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'10913","last_scrub_stamp":"2024-08-19T14:27:07.305888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:27:07.305888+0000","objects_scrubbed":3846,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:17.348716+0000","scrub_duration":914,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":135987200,"num_objects":2075,"num_object_clones":0,"num_object_copies":6225,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2075,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12669,"num_write_kb":471808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,11],"acting":[4,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"70'139","reported_seq":1138,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-08-19T14:27:40.883437+0000","last_change":"2024-08-19T14:27:05.891327+0000","last_active":"2024-08-19T14:27:40.883437+0000","last_peered":"2024-08-19T14:27:40.883437+0000","last_clean":"2024-08-19T14:27:40.883437+0000","last_became_active":"2024-08-19T14:18:26.875179+0000","last_became_peered":"2024-08-19T14:18:26.875179+0000","last_unstale":"2024-08-19T14:27:40.883437+0000","last_undegraded":"2024-08-19T14:27:40.883437+0000","last_fullsized":"2024-08-19T14:27:40.883437+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:27:05.891237+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:27:05.891237+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:29.101160+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.a","version":"486'15336","reported_seq":15618,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-08-19T14:27:40.015586+0000","last_change":"2024-08-19T14:27:24.956372+0000","last_active":"2024-08-19T14:27:40.015586+0000","last_peered":"2024-08-19T14:27:40.015586+0000","last_clean":"2024-08-19T14:27:40.015586+0000","last_became_active":"2024-08-19T14:24:44.592024+0000","last_became_peered":"2024-08-19T14:24:44.592024+0000","last_unstale":"2024-08-19T14:27:40.015586+0000","last_undegraded":"2024-08-19T14:27:40.015586+0000","last_fullsized":"2024-08-19T14:27:40.015586+0000","mapping_epoch":419,"log_start":"437'5302","ondisk_log_start":"437'5302","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"481'14256","last_scrub_stamp":"2024-08-19T14:27:24.956176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:27:24.956176+0000","objects_scrubbed":3085,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:32.574796+0000","scrub_duration":1048,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":129564672,"num_objects":1977,"num_object_clones":0,"num_object_copies":5931,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13075,"num_write_kb":481664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,11],"acting":[8,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.e","version":"70'139","reported_seq":790,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-08-19T14:27:34.867966+0000","last_change":"2024-08-19T14:27:03.631853+0000","last_active":"2024-08-19T14:27:34.867966+0000","last_peered":"2024-08-19T14:27:34.867966+0000","last_clean":"2024-08-19T14:27:34.867966+0000","last_became_active":"2024-08-19T14:18:26.884105+0000","last_became_peered":"2024-08-19T14:18:26.884105+0000","last_unstale":"2024-08-19T14:27:34.867966+0000","last_undegraded":"2024-08-19T14:27:34.867966+0000","last_fullsized":"2024-08-19T14:27:34.867966+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:27:03.631729+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:20:15.321945+0000","last_clean_scrub_stamp":"2024-08-19T14:27:03.631729+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:24.374451+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.b","version":"486'13114","reported_seq":13319,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-08-19T14:27:37.043813+0000","last_change":"2024-08-19T14:27:15.727136+0000","last_active":"2024-08-19T14:27:37.043813+0000","last_peered":"2024-08-19T14:27:37.043813+0000","last_clean":"2024-08-19T14:27:37.043813+0000","last_became_active":"2024-08-19T14:27:15.695445+0000","last_became_peered":"2024-08-19T14:27:15.695445+0000","last_unstale":"2024-08-19T14:27:37.043813+0000","last_undegraded":"2024-08-19T14:27:37.043813+0000","last_fullsized":"2024-08-19T14:27:37.043813+0000","mapping_epoch":477,"log_start":"430'3100","ondisk_log_start":"430'3100","created":419,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":0,"last_scrub":"474'11027","last_scrub_stamp":"2024-08-19T14:27:01.445200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:27:01.445200+0000","objects_scrubbed":4362,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:23.766180+0000","scrub_duration":1433,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":144441344,"num_objects":2204,"num_object_clones":0,"num_object_copies":6612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13114,"num_write_kb":490176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,5],"acting":[0,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"70'139","reported_seq":558,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-08-19T14:27:34.871372+0000","last_change":"2024-08-19T14:27:32.825905+0000","last_active":"2024-08-19T14:27:34.871372+0000","last_peered":"2024-08-19T14:27:34.871372+0000","last_clean":"2024-08-19T14:27:34.871372+0000","last_became_active":"2024-08-19T14:27:31.837754+0000","last_became_peered":"2024-08-19T14:27:31.837754+0000","last_unstale":"2024-08-19T14:27:34.871372+0000","last_undegraded":"2024-08-19T14:27:34.871372+0000","last_fullsized":"2024-08-19T14:27:34.871372+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:27:32.825795+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:27:32.825795+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:49.111004+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"487'10996","reported_seq":11269,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-08-19T14:27:40.985750+0000","last_change":"2024-08-19T14:27:16.182678+0000","last_active":"2024-08-19T14:27:40.985750+0000","last_peered":"2024-08-19T14:27:40.985750+0000","last_clean":"2024-08-19T14:27:40.985750+0000","last_became_active":"2024-08-19T14:25:29.849790+0000","last_became_peered":"2024-08-19T14:25:29.849790+0000","last_unstale":"2024-08-19T14:27:40.985750+0000","last_undegraded":"2024-08-19T14:27:40.985750+0000","last_fullsized":"2024-08-19T14:27:40.985750+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":0,"last_scrub":"478'10163","last_scrub_stamp":"2024-08-19T14:27:16.182488+0000","last_deep_scrub":"478'10163","last_deep_scrub_stamp":"2024-08-19T14:27:16.182488+0000","last_clean_scrub_stamp":"2024-08-19T14:27:16.182488+0000","objects_scrubbed":1849,"log_size":10996,"log_dups_size":0,"ondisk_log_size":10996,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":9,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:43.664659+0000","scrub_duration":8297,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63045632,"num_objects":962,"num_object_clones":0,"num_object_copies":2886,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":962,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3703,"num_write_kb":61440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.c","version":"70'139","reported_seq":895,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-08-19T14:27:34.867267+0000","last_change":"2024-08-19T14:27:00.232219+0000","last_active":"2024-08-19T14:27:34.867267+0000","last_peered":"2024-08-19T14:27:34.867267+0000","last_clean":"2024-08-19T14:27:34.867267+0000","last_became_active":"2024-08-19T14:18:26.872942+0000","last_became_peered":"2024-08-19T14:18:26.872942+0000","last_unstale":"2024-08-19T14:27:34.867267+0000","last_undegraded":"2024-08-19T14:27:34.867267+0000","last_fullsized":"2024-08-19T14:27:34.867267+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:27:00.232092+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:22:52.164586+0000","last_clean_scrub_stamp":"2024-08-19T14:27:00.232092+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:04.916962+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,9],"acting":[3,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"486'9217","reported_seq":9425,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-08-19T14:27:38.599969+0000","last_change":"2024-08-19T14:27:27.280756+0000","last_active":"2024-08-19T14:27:38.599969+0000","last_peered":"2024-08-19T14:27:38.599969+0000","last_clean":"2024-08-19T14:27:38.599969+0000","last_became_active":"2024-08-19T14:25:29.821239+0000","last_became_peered":"2024-08-19T14:25:29.821239+0000","last_unstale":"2024-08-19T14:27:38.599969+0000","last_undegraded":"2024-08-19T14:27:38.599969+0000","last_fullsized":"2024-08-19T14:27:38.599969+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":0,"last_scrub":"482'8776","last_scrub_stamp":"2024-08-19T14:27:27.280495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:27:27.280495+0000","objects_scrubbed":1490,"log_size":9217,"log_dups_size":0,"ondisk_log_size":9217,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:45.484395+0000","scrub_duration":199,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68681728,"num_objects":1048,"num_object_clones":0,"num_object_copies":3144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1048,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3687,"num_write_kb":60544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,9],"acting":[7,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"70'139","reported_seq":810,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-08-19T14:27:34.872005+0000","last_change":"2024-08-19T14:27:01.958255+0000","last_active":"2024-08-19T14:27:34.872005+0000","last_peered":"2024-08-19T14:27:34.872005+0000","last_clean":"2024-08-19T14:27:34.872005+0000","last_became_active":"2024-08-19T14:18:26.882002+0000","last_became_peered":"2024-08-19T14:18:26.882002+0000","last_unstale":"2024-08-19T14:27:34.872005+0000","last_undegraded":"2024-08-19T14:27:34.872005+0000","last_fullsized":"2024-08-19T14:27:34.872005+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:27:01.958142+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:24:50.276408+0000","last_clean_scrub_stamp":"2024-08-19T14:27:01.958142+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:07.061913+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"486'9350","reported_seq":18167,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-08-19T14:27:38.827671+0000","last_change":"2024-08-19T14:27:31.860278+0000","last_active":"2024-08-19T14:27:38.827671+0000","last_peered":"2024-08-19T14:27:38.827671+0000","last_clean":"2024-08-19T14:27:38.827671+0000","last_became_active":"2024-08-19T14:27:31.835536+0000","last_became_peered":"2024-08-19T14:27:31.835536+0000","last_unstale":"2024-08-19T14:27:38.827671+0000","last_undegraded":"2024-08-19T14:27:38.827671+0000","last_fullsized":"2024-08-19T14:27:38.827671+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":439,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":5,"last_scrub":"473'8001","last_scrub_stamp":"2024-08-19T14:26:55.980803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:26:55.980803+0000","objects_scrubbed":2365,"log_size":9350,"log_dups_size":0,"ondisk_log_size":9350,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:21.890801+0000","scrub_duration":563,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536000,"num_objects":1000,"num_object_clones":0,"num_object_copies":3000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1000,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1365,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.12","version":"70'139","reported_seq":597,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-08-19T14:27:34.866499+0000","last_change":"2024-08-19T14:27:31.855657+0000","last_active":"2024-08-19T14:27:34.866499+0000","last_peered":"2024-08-19T14:27:34.866499+0000","last_clean":"2024-08-19T14:27:34.866499+0000","last_became_active":"2024-08-19T14:27:31.838968+0000","last_became_peered":"2024-08-19T14:27:31.838968+0000","last_unstale":"2024-08-19T14:27:34.866499+0000","last_undegraded":"2024-08-19T14:27:34.866499+0000","last_fullsized":"2024-08-19T14:27:34.866499+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:27:31.855524+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:27:31.855524+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:32.492630+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"486'9303","reported_seq":20398,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-08-19T14:27:38.817803+0000","last_change":"2024-08-19T14:26:59.683109+0000","last_active":"2024-08-19T14:27:38.817803+0000","last_peered":"2024-08-19T14:27:38.817803+0000","last_clean":"2024-08-19T14:27:38.817803+0000","last_became_active":"2024-08-19T14:26:36.445968+0000","last_became_peered":"2024-08-19T14:26:36.445968+0000","last_unstale":"2024-08-19T14:27:38.817803+0000","last_undegraded":"2024-08-19T14:27:38.817803+0000","last_fullsized":"2024-08-19T14:27:38.817803+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":439,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":5,"last_scrub":"474'8148","last_scrub_stamp":"2024-08-19T14:26:59.682873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:26:59.682873+0000","objects_scrubbed":2199,"log_size":9303,"log_dups_size":0,"ondisk_log_size":9303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:25.733561+0000","scrub_duration":440,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68157440,"num_objects":1040,"num_object_clones":0,"num_object_copies":3120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1040,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1159,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8,11],"acting":[3,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.13","version":"70'139","reported_seq":1121,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-08-19T14:27:34.866977+0000","last_change":"2024-08-19T14:27:23.812536+0000","last_active":"2024-08-19T14:27:34.866977+0000","last_peered":"2024-08-19T14:27:34.866977+0000","last_clean":"2024-08-19T14:27:34.866977+0000","last_became_active":"2024-08-19T14:18:18.698334+0000","last_became_peered":"2024-08-19T14:18:18.698334+0000","last_unstale":"2024-08-19T14:27:34.866977+0000","last_undegraded":"2024-08-19T14:27:34.866977+0000","last_fullsized":"2024-08-19T14:27:34.866977+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:27:23.812426+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:27:23.812426+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:30.068468+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.14","version":"486'9052","reported_seq":17467,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-08-19T14:27:38.615753+0000","last_change":"2024-08-19T14:26:57.705333+0000","last_active":"2024-08-19T14:27:38.615753+0000","last_peered":"2024-08-19T14:27:38.615753+0000","last_clean":"2024-08-19T14:27:38.615753+0000","last_became_active":"2024-08-19T14:26:34.444635+0000","last_became_peered":"2024-08-19T14:26:34.444635+0000","last_unstale":"2024-08-19T14:27:38.615753+0000","last_undegraded":"2024-08-19T14:27:38.615753+0000","last_fullsized":"2024-08-19T14:27:38.615753+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":439,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":5,"last_scrub":"473'7847","last_scrub_stamp":"2024-08-19T14:26:57.705140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:26:57.705140+0000","objects_scrubbed":2241,"log_size":9052,"log_dups_size":0,"ondisk_log_size":9052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:24.476329+0000","scrub_duration":460,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67174400,"num_objects":1025,"num_object_clones":0,"num_object_copies":3075,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1025,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8,4],"acting":[3,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.10","version":"70'139","reported_seq":759,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-08-19T14:27:34.871604+0000","last_change":"2024-08-19T14:27:32.129561+0000","last_active":"2024-08-19T14:27:34.871604+0000","last_peered":"2024-08-19T14:27:34.871604+0000","last_clean":"2024-08-19T14:27:34.871604+0000","last_became_active":"2024-08-19T14:27:31.839611+0000","last_became_peered":"2024-08-19T14:27:31.839611+0000","last_unstale":"2024-08-19T14:27:34.871604+0000","last_undegraded":"2024-08-19T14:27:34.871604+0000","last_fullsized":"2024-08-19T14:27:34.871604+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:27:32.129449+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:27:32.129449+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:56.975541+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"486'9033","reported_seq":17526,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-08-19T14:27:38.598279+0000","last_change":"2024-08-19T14:27:31.867569+0000","last_active":"2024-08-19T14:27:38.598279+0000","last_peered":"2024-08-19T14:27:38.598279+0000","last_clean":"2024-08-19T14:27:38.598279+0000","last_became_active":"2024-08-19T14:27:31.839752+0000","last_became_peered":"2024-08-19T14:27:31.839752+0000","last_unstale":"2024-08-19T14:27:38.598279+0000","last_undegraded":"2024-08-19T14:27:38.598279+0000","last_fullsized":"2024-08-19T14:27:38.598279+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":439,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":5,"last_scrub":"473'7772","last_scrub_stamp":"2024-08-19T14:26:56.903963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:26:56.903963+0000","objects_scrubbed":2299,"log_size":9033,"log_dups_size":0,"ondisk_log_size":9033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:20.150424+0000","scrub_duration":532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67502080,"num_objects":1030,"num_object_clones":0,"num_object_copies":3090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1269,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,8],"acting":[1,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.11","version":"70'139","reported_seq":781,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-08-19T14:27:34.871045+0000","last_change":"2024-08-19T14:26:59.004413+0000","last_active":"2024-08-19T14:27:34.871045+0000","last_peered":"2024-08-19T14:27:34.871045+0000","last_clean":"2024-08-19T14:27:34.871045+0000","last_became_active":"2024-08-19T14:18:18.697894+0000","last_became_peered":"2024-08-19T14:18:18.697894+0000","last_unstale":"2024-08-19T14:27:34.871045+0000","last_undegraded":"2024-08-19T14:27:34.871045+0000","last_fullsized":"2024-08-19T14:27:34.871045+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:26:59.004304+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:24:50.547428+0000","last_clean_scrub_stamp":"2024-08-19T14:26:59.004304+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:19.347065+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"486'9186","reported_seq":17687,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-08-19T14:27:39.908772+0000","last_change":"2024-08-19T14:26:55.482856+0000","last_active":"2024-08-19T14:27:39.908772+0000","last_peered":"2024-08-19T14:27:39.908772+0000","last_clean":"2024-08-19T14:27:39.908772+0000","last_became_active":"2024-08-19T14:26:16.340194+0000","last_became_peered":"2024-08-19T14:26:16.340194+0000","last_unstale":"2024-08-19T14:27:39.908772+0000","last_undegraded":"2024-08-19T14:27:39.908772+0000","last_fullsized":"2024-08-19T14:27:39.908772+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":439,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":5,"last_scrub":"472'7834","last_scrub_stamp":"2024-08-19T14:26:55.482598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:26:55.482598+0000","objects_scrubbed":2372,"log_size":9186,"log_dups_size":0,"ondisk_log_size":9186,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:20.501632+0000","scrub_duration":541,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66256896,"num_objects":1011,"num_object_clones":0,"num_object_copies":3033,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1011,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1361,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,3],"acting":[8,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"70'139","reported_seq":749,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-08-19T14:27:34.866768+0000","last_change":"2024-08-19T14:27:08.949559+0000","last_active":"2024-08-19T14:27:34.866768+0000","last_peered":"2024-08-19T14:27:34.866768+0000","last_clean":"2024-08-19T14:27:34.866768+0000","last_became_active":"2024-08-19T14:20:28.204724+0000","last_became_peered":"2024-08-19T14:20:28.204724+0000","last_unstale":"2024-08-19T14:27:34.866768+0000","last_undegraded":"2024-08-19T14:27:34.866768+0000","last_fullsized":"2024-08-19T14:27:34.866768+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:27:08.949445+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:27:08.949445+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:28.588653+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"486'9118","reported_seq":12979,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-08-19T14:27:39.994166+0000","last_change":"2024-08-19T14:26:56.478334+0000","last_active":"2024-08-19T14:27:39.994166+0000","last_peered":"2024-08-19T14:27:39.994166+0000","last_clean":"2024-08-19T14:27:39.994166+0000","last_became_active":"2024-08-19T14:26:07.295750+0000","last_became_peered":"2024-08-19T14:26:07.295750+0000","last_unstale":"2024-08-19T14:27:39.994166+0000","last_undegraded":"2024-08-19T14:27:39.994166+0000","last_fullsized":"2024-08-19T14:27:39.994166+0000","mapping_epoch":454,"log_start":"0'0","ondisk_log_start":"0'0","created":439,"last_epoch_clean":455,"parent":"0.0","parent_split_bits":5,"last_scrub":"473'7804","last_scrub_stamp":"2024-08-19T14:26:56.478096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:26:56.478096+0000","objects_scrubbed":2308,"log_size":9118,"log_dups_size":0,"ondisk_log_size":9118,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:19.761013+0000","scrub_duration":545,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64552960,"num_objects":985,"num_object_clones":0,"num_object_copies":2955,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1323,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,5],"acting":[8,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.17","version":"70'139","reported_seq":877,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-08-19T14:27:34.867531+0000","last_change":"2024-08-19T14:27:25.944264+0000","last_active":"2024-08-19T14:27:34.867531+0000","last_peered":"2024-08-19T14:27:34.867531+0000","last_clean":"2024-08-19T14:27:34.867531+0000","last_became_active":"2024-08-19T14:18:31.901556+0000","last_became_peered":"2024-08-19T14:18:31.901556+0000","last_unstale":"2024-08-19T14:27:34.867531+0000","last_undegraded":"2024-08-19T14:27:34.867531+0000","last_fullsized":"2024-08-19T14:27:34.867531+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:27:25.944188+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:27:25.944188+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:26.208810+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,0],"acting":[8,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"5.10","version":"486'9278","reported_seq":17860,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-08-19T14:27:38.677617+0000","last_change":"2024-08-19T14:26:59.390121+0000","last_active":"2024-08-19T14:27:38.677617+0000","last_peered":"2024-08-19T14:27:38.677617+0000","last_clean":"2024-08-19T14:27:38.677617+0000","last_became_active":"2024-08-19T14:26:35.443770+0000","last_became_peered":"2024-08-19T14:26:35.443770+0000","last_unstale":"2024-08-19T14:27:38.677617+0000","last_undegraded":"2024-08-19T14:27:38.677617+0000","last_fullsized":"2024-08-19T14:27:38.677617+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":439,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":5,"last_scrub":"473'8115","last_scrub_stamp":"2024-08-19T14:26:59.389786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:26:59.389786+0000","objects_scrubbed":2208,"log_size":9278,"log_dups_size":0,"ondisk_log_size":9278,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:27.336579+0000","scrub_duration":521,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67960832,"num_objects":1037,"num_object_clones":0,"num_object_copies":3111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1171,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,0],"acting":[7,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"70'139","reported_seq":887,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-08-19T14:27:34.871898+0000","last_change":"2024-08-19T14:27:26.038394+0000","last_active":"2024-08-19T14:27:34.871898+0000","last_peered":"2024-08-19T14:27:34.871898+0000","last_clean":"2024-08-19T14:27:34.871898+0000","last_became_active":"2024-08-19T14:18:18.703926+0000","last_became_peered":"2024-08-19T14:18:18.703926+0000","last_unstale":"2024-08-19T14:27:34.871898+0000","last_undegraded":"2024-08-19T14:27:34.871898+0000","last_fullsized":"2024-08-19T14:27:34.871898+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:27:26.038293+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:27:26.038293+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:36.114804+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,1],"acting":[7,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"486'9338","reported_seq":15250,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-08-19T14:27:37.931674+0000","last_change":"2024-08-19T14:27:00.100949+0000","last_active":"2024-08-19T14:27:37.931674+0000","last_peered":"2024-08-19T14:27:37.931674+0000","last_clean":"2024-08-19T14:27:37.931674+0000","last_became_active":"2024-08-19T14:26:00.207101+0000","last_became_peered":"2024-08-19T14:26:00.207101+0000","last_unstale":"2024-08-19T14:27:37.931674+0000","last_undegraded":"2024-08-19T14:27:37.931674+0000","last_fullsized":"2024-08-19T14:27:37.931674+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":439,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":5,"last_scrub":"474'8221","last_scrub_stamp":"2024-08-19T14:27:00.100580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:27:00.100580+0000","objects_scrubbed":2198,"log_size":9338,"log_dups_size":0,"ondisk_log_size":9338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:15.940877+0000","scrub_duration":491,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70582272,"num_objects":1077,"num_object_clones":0,"num_object_copies":3231,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1121,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7],"acting":[4,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.15","version":"70'139","reported_seq":765,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-08-19T14:27:34.871891+0000","last_change":"2024-08-19T14:26:56.872678+0000","last_active":"2024-08-19T14:27:34.871891+0000","last_peered":"2024-08-19T14:27:34.871891+0000","last_clean":"2024-08-19T14:27:34.871891+0000","last_became_active":"2024-08-19T14:18:31.912496+0000","last_became_peered":"2024-08-19T14:18:31.912496+0000","last_unstale":"2024-08-19T14:27:34.871891+0000","last_undegraded":"2024-08-19T14:27:34.871891+0000","last_fullsized":"2024-08-19T14:27:34.871891+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:26:56.872544+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:20:09.180923+0000","last_clean_scrub_stamp":"2024-08-19T14:26:56.872544+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:08.677502+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"70'139","reported_seq":751,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-08-19T14:27:34.872828+0000","last_change":"2024-08-19T14:27:00.651838+0000","last_active":"2024-08-19T14:27:34.872828+0000","last_peered":"2024-08-19T14:27:34.872828+0000","last_clean":"2024-08-19T14:27:34.872828+0000","last_became_active":"2024-08-19T14:18:31.901229+0000","last_became_peered":"2024-08-19T14:18:31.901229+0000","last_unstale":"2024-08-19T14:27:34.872828+0000","last_undegraded":"2024-08-19T14:27:34.872828+0000","last_fullsized":"2024-08-19T14:27:34.872828+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:27:00.651729+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:27:00.651729+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:30.091916+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1b","version":"70'139","reported_seq":1103,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-08-19T14:27:34.867224+0000","last_change":"2024-08-19T14:26:59.962358+0000","last_active":"2024-08-19T14:27:34.867224+0000","last_peered":"2024-08-19T14:27:34.867224+0000","last_clean":"2024-08-19T14:27:34.867224+0000","last_became_active":"2024-08-19T14:18:31.901152+0000","last_became_peered":"2024-08-19T14:18:31.901152+0000","last_unstale":"2024-08-19T14:27:34.867224+0000","last_undegraded":"2024-08-19T14:27:34.867224+0000","last_fullsized":"2024-08-19T14:27:34.867224+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:26:59.962262+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:26:59.962262+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:08.672325+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.18","version":"70'139","reported_seq":688,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-08-19T14:27:34.871410+0000","last_change":"2024-08-19T14:27:33.084267+0000","last_active":"2024-08-19T14:27:34.871410+0000","last_peered":"2024-08-19T14:27:34.871410+0000","last_clean":"2024-08-19T14:27:34.871410+0000","last_became_active":"2024-08-19T14:27:31.841421+0000","last_became_peered":"2024-08-19T14:27:31.841421+0000","last_unstale":"2024-08-19T14:27:34.871410+0000","last_undegraded":"2024-08-19T14:27:34.871410+0000","last_fullsized":"2024-08-19T14:27:34.871410+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:27:33.084161+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:27:33.084161+0000","last_clean_scrub_stamp":"2024-08-19T14:27:33.084161+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:29:01.689805+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.19","version":"70'139","reported_seq":1095,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-08-19T14:27:34.870898+0000","last_change":"2024-08-19T14:26:55.039720+0000","last_active":"2024-08-19T14:27:34.870898+0000","last_peered":"2024-08-19T14:27:34.870898+0000","last_clean":"2024-08-19T14:27:34.870898+0000","last_became_active":"2024-08-19T14:18:31.902547+0000","last_became_peered":"2024-08-19T14:18:31.902547+0000","last_unstale":"2024-08-19T14:27:34.870898+0000","last_undegraded":"2024-08-19T14:27:34.870898+0000","last_fullsized":"2024-08-19T14:27:34.870898+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:26:55.039598+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:26:55.039598+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:27:57.212315+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1e","version":"70'139","reported_seq":756,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-08-19T14:27:40.883470+0000","last_change":"2024-08-19T14:27:04.781012+0000","last_active":"2024-08-19T14:27:40.883470+0000","last_peered":"2024-08-19T14:27:40.883470+0000","last_clean":"2024-08-19T14:27:40.883470+0000","last_became_active":"2024-08-19T14:18:31.902546+0000","last_became_peered":"2024-08-19T14:18:31.902546+0000","last_unstale":"2024-08-19T14:27:40.883470+0000","last_undegraded":"2024-08-19T14:27:40.883470+0000","last_fullsized":"2024-08-19T14:27:40.883470+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:27:04.780939+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:27:04.780939+0000","last_clean_scrub_stamp":"2024-08-19T14:27:04.780939+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:20.233676+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.18","version":"486'8974","reported_seq":17418,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-08-19T14:27:37.898683+0000","last_change":"2024-08-19T14:26:59.043461+0000","last_active":"2024-08-19T14:27:37.898683+0000","last_peered":"2024-08-19T14:27:37.898683+0000","last_clean":"2024-08-19T14:27:37.898683+0000","last_became_active":"2024-08-19T14:26:36.445937+0000","last_became_peered":"2024-08-19T14:26:36.445937+0000","last_unstale":"2024-08-19T14:27:37.898683+0000","last_undegraded":"2024-08-19T14:27:37.898683+0000","last_fullsized":"2024-08-19T14:27:37.898683+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":439,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":5,"last_scrub":"473'7817","last_scrub_stamp":"2024-08-19T14:26:59.043047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:26:59.043047+0000","objects_scrubbed":2147,"log_size":8974,"log_dups_size":0,"ondisk_log_size":8974,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:27.776983+0000","scrub_duration":473,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64356352,"num_objects":982,"num_object_clones":0,"num_object_copies":2946,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":982,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1165,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,1],"acting":[4,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1c","version":"70'139","reported_seq":861,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-08-19T14:27:34.867414+0000","last_change":"2024-08-19T14:26:57.940698+0000","last_active":"2024-08-19T14:27:34.867414+0000","last_peered":"2024-08-19T14:27:34.867414+0000","last_clean":"2024-08-19T14:27:34.867414+0000","last_became_active":"2024-08-19T14:18:31.902494+0000","last_became_peered":"2024-08-19T14:18:31.902494+0000","last_unstale":"2024-08-19T14:27:34.867414+0000","last_undegraded":"2024-08-19T14:27:34.867414+0000","last_fullsized":"2024-08-19T14:27:34.867414+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:26:57.940466+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:26:57.940466+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:24.547300+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,11],"acting":[8,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"5.19","version":"486'9142","reported_seq":17589,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-08-19T14:27:38.604759+0000","last_change":"2024-08-19T14:26:59.531135+0000","last_active":"2024-08-19T14:27:38.604759+0000","last_peered":"2024-08-19T14:27:38.604759+0000","last_clean":"2024-08-19T14:27:38.604759+0000","last_became_active":"2024-08-19T14:26:15.340594+0000","last_became_peered":"2024-08-19T14:26:15.340594+0000","last_unstale":"2024-08-19T14:27:38.604759+0000","last_undegraded":"2024-08-19T14:27:38.604759+0000","last_fullsized":"2024-08-19T14:27:38.604759+0000","mapping_epoch":457,"log_start":"0'0","ondisk_log_start":"0'0","created":439,"last_epoch_clean":458,"parent":"0.0","parent_split_bits":5,"last_scrub":"473'7983","last_scrub_stamp":"2024-08-19T14:26:59.530991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:26:59.530991+0000","objects_scrubbed":2215,"log_size":9142,"log_dups_size":0,"ondisk_log_size":9142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:18.755230+0000","scrub_duration":640,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68222976,"num_objects":1041,"num_object_clones":0,"num_object_copies":3123,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1174,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1d","version":"70'139","reported_seq":775,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-08-19T14:27:34.866739+0000","last_change":"2024-08-19T14:26:59.878357+0000","last_active":"2024-08-19T14:27:34.866739+0000","last_peered":"2024-08-19T14:27:34.866739+0000","last_clean":"2024-08-19T14:27:34.866739+0000","last_became_active":"2024-08-19T14:18:31.901183+0000","last_became_peered":"2024-08-19T14:18:31.901183+0000","last_unstale":"2024-08-19T14:27:34.866739+0000","last_undegraded":"2024-08-19T14:27:34.866739+0000","last_fullsized":"2024-08-19T14:27:34.866739+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:26:59.878236+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:26:59.878236+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:21.153098+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,5],"acting":[1,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":26,"stat_sum":{"num_bytes":2173567000,"num_objects":33167,"num_object_clones":0,"num_object_copies":99501,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33167,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":122357,"num_write_kb":3398593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6976516096,"data_stored":6976503880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":247463,"ondisk_log_size":247463,"up":78,"acting":78,"num_store_stats":9},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1802240,"data_stored":1788032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187652,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1246764,"kb_used_data":629732,"kb_used_omap":1,"kb_used_meta":617022,"kb_avail":92498388,"statfs":{"total":95995035648,"available":94718349312,"internally_reserved":0,"allocated":644845568,"data_stored":642648032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":631830980},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":343597383881,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":74,"seq":317827580109,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1367528,"kb_used_data":696352,"kb_used_omap":1,"kb_used_meta":671166,"kb_avail":92377624,"statfs":{"total":95995035648,"available":94594686976,"internally_reserved":0,"allocated":713064448,"data_stored":710870984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":687274433},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874451,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1400040,"kb_used_data":841312,"kb_used_omap":1,"kb_used_meta":558718,"kb_avail":92345112,"statfs":{"total":95995035648,"available":94561394688,"internally_reserved":0,"allocated":861503488,"data_stored":859310024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":572127685},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":57,"seq":244813136086,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1283700,"kb_used_data":620844,"kb_used_omap":1,"kb_used_meta":662846,"kb_avail":92461452,"statfs":{"total":95995035648,"available":94680526848,"internally_reserved":0,"allocated":635744256,"data_stored":633543168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":678754762},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,1,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":206158430425,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692061,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1773752,"kb_used_data":1215020,"kb_used_omap":1,"kb_used_meta":558718,"kb_avail":91971400,"statfs":{"total":95995035648,"available":94178713600,"internally_reserved":0,"allocated":1244180480,"data_stored":1241979392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":572127685},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986402,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1639720,"kb_used_data":932448,"kb_used_omap":1,"kb_used_meta":707262,"kb_avail":92105432,"statfs":{"total":95995035648,"available":94315962368,"internally_reserved":0,"allocated":954826752,"data_stored":952633288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":724236741},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149928,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1022632,"kb_used_data":409760,"kb_used_omap":1,"kb_used_meta":612862,"kb_avail":92722520,"statfs":{"total":95995035648,"available":94947860480,"internally_reserved":0,"allocated":419594240,"data_stored":417400776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":627571143},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411566,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575089,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1359216,"kb_used_data":615848,"kb_used_omap":1,"kb_used_meta":743358,"kb_avail":92385936,"statfs":{"total":95995035648,"available":94603198464,"internally_reserved":0,"allocated":630628352,"data_stored":628431336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":761199045},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738613,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1630696,"kb_used_data":1031712,"kb_used_omap":1,"kb_used_meta":598974,"kb_avail":92114456,"statfs":{"total":95995035648,"available":94325202944,"internally_reserved":0,"allocated":1056473088,"data_stored":1054279624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":613349829},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1016987648,"data_stored":1016987648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":606666752,"data_stored":606666752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":405929984,"data_stored":405929984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":941359104,"data_stored":941359104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1211764736,"data_stored":1211760664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":615321600,"data_stored":615317528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":852885504,"data_stored":852885504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":701956096,"data_stored":701956096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":623644672,"data_stored":623640600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T14:27:43.466 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-08-19T14:27:43.466 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd pool set .mgr pg_num 71 2024-08-19T14:27:43.483 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:27:43.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:43.491 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:43.491 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:27:43.592 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:27:43.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:43.638 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:43.638 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:27:43.667 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:27:43.667 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:27:43.739 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:27:43.766 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:43.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:43.767 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:27:43.868 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:27:43.902 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:43.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:43.903 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:27:44.004 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:27:44.026 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:43 smithi031 ceph-mon[16336]: pgmap v1122: 57 pgs: 57 active+clean; 2.0 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 1.07k op/s 2024-08-19T14:27:44.040 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:44.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:44.041 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:27:44.082 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:27:44.086 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:27:44.086 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:27:44.124 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:27:44.124 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:27:44.142 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:27:44.152 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:27:44.152 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:27:44.173 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:44.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:44.174 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:27:44.175 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:27:44.175 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:27:44.204 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:27:44.205 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:27:44.231 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:27:44.231 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:27:44.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:43 smithi032 ceph-mon[20004]: pgmap v1122: 57 pgs: 57 active+clean; 2.0 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 1.07k op/s 2024-08-19T14:27:44.260 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:27:44.260 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:27:44.276 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:27:44.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:43 smithi045 ceph-mon[20143]: pgmap v1122: 57 pgs: 57 active+clean; 2.0 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 1.07k op/s 2024-08-19T14:27:44.297 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:27:44.297 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:27:44.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:44.312 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:44.313 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:27:44.328 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:27:44.328 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:27:44.367 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:27:44.367 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:27:44.373 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-08-19T14:27:44.397 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:27:44.397 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:27:44.413 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:27:44.421 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:27:44.422 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:27:44.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:44.442 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:44.442 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:27:44.543 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:27:44.585 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:44.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:44.587 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:27:44.688 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:27:44.723 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:44.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:44.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:27:44.773 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-19T14:27:44.803 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:27:44.825 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:27:44.857 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:44.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:44.858 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:27:44.939 INFO:teuthology.orchestra.run.smithi031.stderr:Error EINVAL: specified pg_num 71 < pg_num_max 32 2024-08-19T14:27:44.959 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:27:45.010 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:45.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:45.011 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:27:45.112 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:27:45.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:44 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1653323271' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:27:45.137 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:45.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:45.138 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:27:45.161 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:44 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1653323271' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:27:45.240 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:27:45.278 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:45.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:45.279 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:27:45.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:44 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1653323271' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:27:45.380 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:27:45.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:45.417 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:45.417 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:27:45.430 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:27:45.430 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-08-19T14:27:45.430 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-08-19T14:27:45.430 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-08-19T14:27:45.430 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.430 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25057.0:130072 3.f 3:f7f93fc4:::benchmark_data_smithi045_59973_object130071:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e281)", 2024-08-19T14:27:45.430 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:18:31.609517+0000", 2024-08-19T14:27:45.431 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 553.81356849199994, 2024-08-19T14:27:45.431 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.21221680200000001, 2024-08-19T14:27:45.431 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:27:45.431 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:27:45.431 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:27:45.431 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:27:45.431 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25057", 2024-08-19T14:27:45.431 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:27:45.431 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 130072 2024-08-19T14:27:45.431 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.431 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:27:45.431 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.431 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:27:45.431 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.609517+0000", 2024-08-19T14:27:45.432 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:45.432 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.432 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.432 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:27:45.432 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.609517+0000", 2024-08-19T14:27:45.432 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:45.432 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.432 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.432 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:27:45.432 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.609517+0000", 2024-08-19T14:27:45.432 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.7899999999999997e-07 2024-08-19T14:27:45.432 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.432 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.432 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:27:45.433 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.609574+0000", 2024-08-19T14:27:45.433 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6422000000000003e-05 2024-08-19T14:27:45.433 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.433 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.433 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:27:45.433 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.609588+0000", 2024-08-19T14:27:45.433 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.403e-05 2024-08-19T14:27:45.433 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.433 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.433 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:27:45.433 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.609595+0000", 2024-08-19T14:27:45.433 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8970000000000002e-06 2024-08-19T14:27:45.433 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.434 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.434 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:27:45.434 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.610985+0000", 2024-08-19T14:27:45.434 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0013905949999999999 2024-08-19T14:27:45.434 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.434 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.434 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:27:45.434 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.611111+0000", 2024-08-19T14:27:45.434 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012530700000000001 2024-08-19T14:27:45.435 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.435 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.435 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 8,11", 2024-08-19T14:27:45.435 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.611234+0000", 2024-08-19T14:27:45.435 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012355100000000001 2024-08-19T14:27:45.435 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.435 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.435 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:45.435 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.616670+0000", 2024-08-19T14:27:45.435 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0054355339999999997 2024-08-19T14:27:45.435 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.435 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.435 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:27:45.435 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.616703+0000", 2024-08-19T14:27:45.436 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3281999999999997e-05 2024-08-19T14:27:45.436 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.436 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.436 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:45.436 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.821464+0000", 2024-08-19T14:27:45.436 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20476105 2024-08-19T14:27:45.436 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.436 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.436 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:27:45.436 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.821687+0000", 2024-08-19T14:27:45.436 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000223241 2024-08-19T14:27:45.436 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.436 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.436 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:27:45.437 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.821733+0000", 2024-08-19T14:27:45.437 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6013999999999997e-05 2024-08-19T14:27:45.437 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:45.437 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:27:45.437 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:45.437 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.437 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.437 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25057.0:130075 3.f 3:f32a0229:::benchmark_data_smithi045_59973_object130074:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e281)", 2024-08-19T14:27:45.437 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:18:31.611127+0000", 2024-08-19T14:27:45.437 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 553.81195825199995, 2024-08-19T14:27:45.437 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.21073734099999999, 2024-08-19T14:27:45.437 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:27:45.437 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:27:45.438 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:27:45.438 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:27:45.438 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25057", 2024-08-19T14:27:45.438 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:27:45.438 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 130075 2024-08-19T14:27:45.438 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.438 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:27:45.438 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.438 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:27:45.438 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.611127+0000", 2024-08-19T14:27:45.438 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:45.438 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.439 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.439 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:27:45.439 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.611127+0000", 2024-08-19T14:27:45.439 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:45.439 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.439 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.439 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:27:45.439 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.611128+0000", 2024-08-19T14:27:45.439 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0640000000000001e-06 2024-08-19T14:27:45.439 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.439 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.439 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:27:45.439 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.611271+0000", 2024-08-19T14:27:45.440 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000143027 2024-08-19T14:27:45.440 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.440 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.440 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:27:45.440 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.611284+0000", 2024-08-19T14:27:45.440 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2979000000000001e-05 2024-08-19T14:27:45.440 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.440 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.440 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:27:45.440 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.611291+0000", 2024-08-19T14:27:45.440 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2810000000000003e-06 2024-08-19T14:27:45.440 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.440 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.441 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:27:45.441 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.614035+0000", 2024-08-19T14:27:45.441 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0027435020000000001 2024-08-19T14:27:45.441 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.441 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.441 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:27:45.441 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.614198+0000", 2024-08-19T14:27:45.441 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00016282100000000001 2024-08-19T14:27:45.441 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.441 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.441 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 8,11", 2024-08-19T14:27:45.441 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.614337+0000", 2024-08-19T14:27:45.442 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000139266 2024-08-19T14:27:45.442 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.442 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.442 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:27:45.442 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.617108+0000", 2024-08-19T14:27:45.442 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0027711960000000001 2024-08-19T14:27:45.442 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.442 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.442 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:45.442 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.617630+0000", 2024-08-19T14:27:45.442 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00052161000000000004 2024-08-19T14:27:45.442 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.442 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.443 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:45.443 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.821780+0000", 2024-08-19T14:27:45.443 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.204150741 2024-08-19T14:27:45.443 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.443 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.443 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:27:45.443 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.821812+0000", 2024-08-19T14:27:45.443 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1785999999999997e-05 2024-08-19T14:27:45.443 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.443 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.443 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:27:45.443 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.821864+0000", 2024-08-19T14:27:45.443 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2067999999999997e-05 2024-08-19T14:27:45.444 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:45.444 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:27:45.444 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:45.444 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.444 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.444 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25057.0:130097 3.f 3:f04c1e35:::benchmark_data_smithi045_59973_object130096:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e281)", 2024-08-19T14:27:45.444 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:18:31.621483+0000", 2024-08-19T14:27:45.444 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 553.80160257499995, 2024-08-19T14:27:45.444 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.200549171, 2024-08-19T14:27:45.445 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:27:45.445 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:27:45.445 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:27:45.445 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:27:45.445 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25057", 2024-08-19T14:27:45.445 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:27:45.445 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 130097 2024-08-19T14:27:45.445 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.445 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:27:45.445 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.445 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:27:45.445 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.621483+0000", 2024-08-19T14:27:45.445 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:45.445 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.446 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.446 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:27:45.446 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.621483+0000", 2024-08-19T14:27:45.446 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:45.446 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.446 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.446 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:27:45.446 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.621484+0000", 2024-08-19T14:27:45.446 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3200000000000001e-06 2024-08-19T14:27:45.446 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.446 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.446 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:27:45.446 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.621516+0000", 2024-08-19T14:27:45.447 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2369000000000001e-05 2024-08-19T14:27:45.447 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.447 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.447 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:27:45.447 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.621533+0000", 2024-08-19T14:27:45.447 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6511000000000002e-05 2024-08-19T14:27:45.447 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.447 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.447 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:27:45.447 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.621540+0000", 2024-08-19T14:27:45.447 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4580000000000004e-06 2024-08-19T14:27:45.447 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.447 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.447 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:27:45.448 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.621617+0000", 2024-08-19T14:27:45.448 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6584999999999999e-05 2024-08-19T14:27:45.448 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.448 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.448 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:27:45.448 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.621760+0000", 2024-08-19T14:27:45.448 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014331399999999999 2024-08-19T14:27:45.448 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.448 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.448 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 8,11", 2024-08-19T14:27:45.448 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.621903+0000", 2024-08-19T14:27:45.448 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000142927 2024-08-19T14:27:45.448 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.449 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.449 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:45.449 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.626175+0000", 2024-08-19T14:27:45.449 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0042722639999999996 2024-08-19T14:27:45.449 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.449 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.449 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:27:45.449 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.626353+0000", 2024-08-19T14:27:45.449 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000178152 2024-08-19T14:27:45.449 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.449 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.449 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:45.449 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.821907+0000", 2024-08-19T14:27:45.449 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.195553372 2024-08-19T14:27:45.450 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.450 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.450 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:27:45.450 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.821996+0000", 2024-08-19T14:27:45.450 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.8832000000000006e-05 2024-08-19T14:27:45.450 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.450 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.450 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:27:45.450 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.822032+0000", 2024-08-19T14:27:45.450 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6066999999999999e-05 2024-08-19T14:27:45.450 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:45.450 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:27:45.450 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:45.451 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.451 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.451 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25057.0:130120 3.f 3:f699ea53:::benchmark_data_smithi045_59973_object130119:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e281)", 2024-08-19T14:27:45.451 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:18:31.629426+0000", 2024-08-19T14:27:45.451 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 553.79365927799995, 2024-08-19T14:27:45.451 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.19275684200000001, 2024-08-19T14:27:45.451 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:27:45.451 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:27:45.451 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:27:45.451 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:27:45.451 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25057", 2024-08-19T14:27:45.451 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:27:45.451 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 130120 2024-08-19T14:27:45.452 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.452 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:27:45.452 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.452 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:27:45.452 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.629426+0000", 2024-08-19T14:27:45.452 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:45.452 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.452 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.452 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:27:45.452 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.629426+0000", 2024-08-19T14:27:45.452 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:45.452 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.452 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.452 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:27:45.453 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.629427+0000", 2024-08-19T14:27:45.453 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.274e-06 2024-08-19T14:27:45.453 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.453 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.453 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:27:45.453 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.629476+0000", 2024-08-19T14:27:45.453 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8590000000000001e-05 2024-08-19T14:27:45.453 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.454 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.454 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:27:45.454 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.629487+0000", 2024-08-19T14:27:45.454 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1508999999999999e-05 2024-08-19T14:27:45.454 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.454 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.454 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:27:45.454 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.629493+0000", 2024-08-19T14:27:45.454 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.677e-06 2024-08-19T14:27:45.454 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.454 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.454 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:27:45.455 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.629543+0000", 2024-08-19T14:27:45.455 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0630000000000001e-05 2024-08-19T14:27:45.455 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.455 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.455 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:27:45.455 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.629643+0000", 2024-08-19T14:27:45.455 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.9978999999999999e-05 2024-08-19T14:27:45.455 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.455 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.455 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 8,11", 2024-08-19T14:27:45.455 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.629728+0000", 2024-08-19T14:27:45.455 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.4127000000000003e-05 2024-08-19T14:27:45.456 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.456 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.456 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:27:45.456 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.631709+0000", 2024-08-19T14:27:45.456 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0019811009999999999 2024-08-19T14:27:45.456 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.456 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.456 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:45.456 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.632189+0000", 2024-08-19T14:27:45.456 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00047994299999999998 2024-08-19T14:27:45.456 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.457 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.457 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:45.457 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.822073+0000", 2024-08-19T14:27:45.457 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.18988412800000001 2024-08-19T14:27:45.457 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.457 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.457 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:27:45.457 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.822143+0000", 2024-08-19T14:27:45.457 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0588999999999998e-05 2024-08-19T14:27:45.457 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.457 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.457 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:27:45.458 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.822183+0000", 2024-08-19T14:27:45.458 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9295000000000002e-05 2024-08-19T14:27:45.458 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:45.458 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:27:45.458 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:45.458 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.458 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.458 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25057.0:130174 3.f 3:f33afdbe:::benchmark_data_smithi045_59973_object130173:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e281)", 2024-08-19T14:27:45.458 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:18:31.660169+0000", 2024-08-19T14:27:45.458 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 553.76291586599996, 2024-08-19T14:27:45.458 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.164501495, 2024-08-19T14:27:45.458 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:27:45.459 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:27:45.459 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:27:45.459 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:27:45.459 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25057", 2024-08-19T14:27:45.459 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:27:45.459 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 130174 2024-08-19T14:27:45.460 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.460 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:27:45.460 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.460 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:27:45.460 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.660169+0000", 2024-08-19T14:27:45.460 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:45.460 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.460 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.460 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:27:45.460 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.660169+0000", 2024-08-19T14:27:45.460 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:45.460 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.460 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.461 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:27:45.461 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.660170+0000", 2024-08-19T14:27:45.461 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.119e-06 2024-08-19T14:27:45.461 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.461 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.461 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:27:45.461 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.660216+0000", 2024-08-19T14:27:45.461 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5192999999999999e-05 2024-08-19T14:27:45.461 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.461 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.461 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:27:45.461 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.660227+0000", 2024-08-19T14:27:45.461 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1496e-05 2024-08-19T14:27:45.461 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.462 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.462 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:27:45.462 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.660232+0000", 2024-08-19T14:27:45.462 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1919999999999996e-06 2024-08-19T14:27:45.462 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.462 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.462 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:27:45.462 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.660278+0000", 2024-08-19T14:27:45.462 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.566e-05 2024-08-19T14:27:45.462 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.462 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.462 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:27:45.462 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.660368+0000", 2024-08-19T14:27:45.463 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.0274999999999999e-05 2024-08-19T14:27:45.463 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.463 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.463 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 8,11", 2024-08-19T14:27:45.463 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.660458+0000", 2024-08-19T14:27:45.463 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.9733000000000003e-05 2024-08-19T14:27:45.463 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.463 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.463 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:27:45.463 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.662221+0000", 2024-08-19T14:27:45.463 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0017634409999999999 2024-08-19T14:27:45.463 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.463 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.463 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:45.464 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.662766+0000", 2024-08-19T14:27:45.464 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00054489399999999998 2024-08-19T14:27:45.464 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.464 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.464 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:45.464 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.822238+0000", 2024-08-19T14:27:45.464 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.159471785 2024-08-19T14:27:45.464 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.464 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.464 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:27:45.464 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.822286+0000", 2024-08-19T14:27:45.464 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7633999999999998e-05 2024-08-19T14:27:45.464 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.464 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.464 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:27:45.465 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:18:31.824671+0000", 2024-08-19T14:27:45.465 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0023850730000000001 2024-08-19T14:27:45.466 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:45.466 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:27:45.466 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:45.466 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.466 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.466 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:89465 5.19 5:9a835940:::benchmark_data_smithi045_85309_object89464:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e443)", 2024-08-19T14:27:45.466 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:25:32.916576+0000", 2024-08-19T14:27:45.466 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 132.50650864400001, 2024-08-19T14:27:45.466 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.096745901999999995, 2024-08-19T14:27:45.466 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:27:45.466 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:27:45.466 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:27:45.466 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:27:45.467 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:27:45.467 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:27:45.467 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 89465 2024-08-19T14:27:45.467 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.467 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:27:45.467 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.467 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:27:45.467 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:32.916576+0000", 2024-08-19T14:27:45.467 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:45.467 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.467 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.468 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:27:45.468 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:32.916576+0000", 2024-08-19T14:27:45.468 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:45.468 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.468 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.468 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:27:45.468 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:32.916579+0000", 2024-08-19T14:27:45.468 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.458e-06 2024-08-19T14:27:45.468 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.468 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.468 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:27:45.468 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:32.916620+0000", 2024-08-19T14:27:45.468 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0713000000000001e-05 2024-08-19T14:27:45.469 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.469 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.469 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:27:45.469 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:32.916644+0000", 2024-08-19T14:27:45.469 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4335999999999999e-05 2024-08-19T14:27:45.469 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.469 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.469 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:27:45.469 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:32.916657+0000", 2024-08-19T14:27:45.469 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2792000000000001e-05 2024-08-19T14:27:45.469 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.469 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.469 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:27:45.470 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:32.918924+0000", 2024-08-19T14:27:45.470 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0022671179999999998 2024-08-19T14:27:45.470 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.470 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.470 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:27:45.470 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:32.918936+0000", 2024-08-19T14:27:45.470 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2079000000000001e-05 2024-08-19T14:27:45.470 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.470 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.470 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:27:45.470 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:32.996376+0000", 2024-08-19T14:27:45.471 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.077440264999999994 2024-08-19T14:27:45.471 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.471 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.471 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:27:45.471 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:32.996520+0000", 2024-08-19T14:27:45.471 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000144096 2024-08-19T14:27:45.471 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.471 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.471 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 1,5", 2024-08-19T14:27:45.471 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:32.996634+0000", 2024-08-19T14:27:45.471 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000113603 2024-08-19T14:27:45.471 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.471 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.472 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:27:45.472 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:33.001785+0000", 2024-08-19T14:27:45.472 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0051515679999999996 2024-08-19T14:27:45.472 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.472 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.472 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:45.472 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:33.001831+0000", 2024-08-19T14:27:45.472 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5834000000000002e-05 2024-08-19T14:27:45.472 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.472 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.472 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:45.472 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:33.011472+0000", 2024-08-19T14:27:45.472 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0096403749999999996 2024-08-19T14:27:45.472 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.472 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.473 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:27:45.473 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:33.011515+0000", 2024-08-19T14:27:45.473 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3211999999999999e-05 2024-08-19T14:27:45.473 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.473 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.473 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:27:45.473 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:33.013322+0000", 2024-08-19T14:27:45.473 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0018074530000000001 2024-08-19T14:27:45.473 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:45.473 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:27:45.473 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:45.473 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.473 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.473 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:89468 5.19 5:9e4b8327:::benchmark_data_smithi045_85309_object89467:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e443)", 2024-08-19T14:27:45.474 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:25:32.916683+0000", 2024-08-19T14:27:45.474 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 132.50640197999999, 2024-08-19T14:27:45.474 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.099019897999999995, 2024-08-19T14:27:45.474 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:27:45.474 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:27:45.474 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:27:45.474 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:27:45.474 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:27:45.474 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:27:45.474 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 89468 2024-08-19T14:27:45.474 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.474 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:27:45.475 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.475 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:27:45.475 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:32.916683+0000", 2024-08-19T14:27:45.475 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:45.475 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.475 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.475 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:27:45.475 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:32.916683+0000", 2024-08-19T14:27:45.475 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:45.475 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.475 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.475 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:27:45.476 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:32.916684+0000", 2024-08-19T14:27:45.476 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.51e-07 2024-08-19T14:27:45.477 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.477 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.477 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:27:45.477 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:32.916711+0000", 2024-08-19T14:27:45.477 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7039e-05 2024-08-19T14:27:45.477 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.477 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.477 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:27:45.477 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:32.916717+0000", 2024-08-19T14:27:45.477 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4910000000000002e-06 2024-08-19T14:27:45.477 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.477 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.477 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:27:45.478 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:32.916722+0000", 2024-08-19T14:27:45.478 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8439999999999997e-06 2024-08-19T14:27:45.478 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.478 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.478 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:27:45.478 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:32.918964+0000", 2024-08-19T14:27:45.478 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0022423360000000002 2024-08-19T14:27:45.478 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.478 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.478 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:27:45.478 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:32.918972+0000", 2024-08-19T14:27:45.478 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5889999999999997e-06 2024-08-19T14:27:45.478 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.479 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.479 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:27:45.479 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:33.000266+0000", 2024-08-19T14:27:45.479 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.081293821000000002 2024-08-19T14:27:45.479 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.479 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.479 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:27:45.479 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:33.000398+0000", 2024-08-19T14:27:45.479 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013238400000000001 2024-08-19T14:27:45.479 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.479 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.480 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 1,5", 2024-08-19T14:27:45.480 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:33.000511+0000", 2024-08-19T14:27:45.480 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00011237499999999999 2024-08-19T14:27:45.480 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.480 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.480 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:27:45.480 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:33.003797+0000", 2024-08-19T14:27:45.480 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0032867439999999999 2024-08-19T14:27:45.480 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.480 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.480 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:45.480 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:33.004129+0000", 2024-08-19T14:27:45.481 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00033177899999999998 2024-08-19T14:27:45.481 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.481 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.481 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:45.481 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:33.013377+0000", 2024-08-19T14:27:45.481 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0092478950000000008 2024-08-19T14:27:45.481 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.481 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.481 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:27:45.481 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:33.013437+0000", 2024-08-19T14:27:45.481 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9800999999999999e-05 2024-08-19T14:27:45.481 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.481 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.481 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:27:45.482 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:33.015703+0000", 2024-08-19T14:27:45.482 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0022662490000000001 2024-08-19T14:27:45.482 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:45.482 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:27:45.482 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:45.482 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.482 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.482 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:89500 5.19 5:9e4f5b0d:::benchmark_data_smithi045_85309_object89499:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e443)", 2024-08-19T14:27:45.482 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:25:32.916729+0000", 2024-08-19T14:27:45.482 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 132.50635559, 2024-08-19T14:27:45.482 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.10171635599999999, 2024-08-19T14:27:45.482 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:27:45.482 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:27:45.482 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:27:45.483 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:27:45.483 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:27:45.483 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:27:45.483 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 89500 2024-08-19T14:27:45.483 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.483 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:27:45.483 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.483 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:27:45.483 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:32.916729+0000", 2024-08-19T14:27:45.483 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:45.483 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.483 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.483 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:27:45.484 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:32.916729+0000", 2024-08-19T14:27:45.484 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:45.484 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.484 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.484 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:27:45.484 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:32.916730+0000", 2024-08-19T14:27:45.484 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.51e-07 2024-08-19T14:27:45.484 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.484 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.484 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:27:45.484 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:32.916756+0000", 2024-08-19T14:27:45.484 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6288999999999999e-05 2024-08-19T14:27:45.484 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.484 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.485 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:27:45.485 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:32.916762+0000", 2024-08-19T14:27:45.485 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9200000000000001e-06 2024-08-19T14:27:45.485 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.485 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.485 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:27:45.485 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:32.916767+0000", 2024-08-19T14:27:45.485 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3810000000000001e-06 2024-08-19T14:27:45.485 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.485 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.485 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:27:45.485 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:32.919008+0000", 2024-08-19T14:27:45.485 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.002241472 2024-08-19T14:27:45.486 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.486 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.486 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:27:45.486 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:32.919016+0000", 2024-08-19T14:27:45.486 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4780000000000002e-06 2024-08-19T14:27:45.486 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.486 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.486 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:27:45.486 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:33.002039+0000", 2024-08-19T14:27:45.486 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.083022928999999995 2024-08-19T14:27:45.486 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.486 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.486 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:27:45.486 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:33.002207+0000", 2024-08-19T14:27:45.487 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000168042 2024-08-19T14:27:45.487 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.487 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.487 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 1,5", 2024-08-19T14:27:45.487 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:33.002380+0000", 2024-08-19T14:27:45.487 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000173158 2024-08-19T14:27:45.487 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.487 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.487 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:27:45.487 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:33.003997+0000", 2024-08-19T14:27:45.487 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0016177069999999999 2024-08-19T14:27:45.487 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.487 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.488 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:45.488 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:33.007588+0000", 2024-08-19T14:27:45.488 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0035907690000000002 2024-08-19T14:27:45.488 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.488 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.488 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:45.488 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:33.015879+0000", 2024-08-19T14:27:45.488 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0082904199999999997 2024-08-19T14:27:45.488 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.488 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.488 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:27:45.488 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:33.015972+0000", 2024-08-19T14:27:45.488 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.2960000000000004e-05 2024-08-19T14:27:45.489 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.489 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.489 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:27:45.489 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:33.018446+0000", 2024-08-19T14:27:45.489 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0024742800000000001 2024-08-19T14:27:45.489 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:45.489 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:27:45.489 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:45.489 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.489 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.489 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:92561 5.19 5:9dc7b92a:::benchmark_data_smithi045_85309_object92560:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e444)", 2024-08-19T14:27:45.489 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:25:36.840181+0000", 2024-08-19T14:27:45.489 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 128.582904576, 2024-08-19T14:27:45.489 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.099885679000000005, 2024-08-19T14:27:45.490 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:27:45.490 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:27:45.490 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:27:45.490 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:27:45.490 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:27:45.490 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:27:45.490 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 92561 2024-08-19T14:27:45.490 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.490 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:27:45.490 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.490 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:27:45.490 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:36.840181+0000", 2024-08-19T14:27:45.490 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:45.491 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.491 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.491 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:27:45.491 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:36.840181+0000", 2024-08-19T14:27:45.491 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:45.491 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.491 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.491 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:27:45.491 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:36.840183+0000", 2024-08-19T14:27:45.491 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0219999999999999e-06 2024-08-19T14:27:45.491 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.491 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.491 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:27:45.492 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:36.840215+0000", 2024-08-19T14:27:45.492 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2224000000000002e-05 2024-08-19T14:27:45.492 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.492 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.492 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:27:45.492 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:36.840229+0000", 2024-08-19T14:27:45.492 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4246e-05 2024-08-19T14:27:45.492 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.492 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.492 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:27:45.493 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:36.840238+0000", 2024-08-19T14:27:45.493 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.8599999999999999e-06 2024-08-19T14:27:45.493 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.493 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.493 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:27:45.493 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:36.842963+0000", 2024-08-19T14:27:45.493 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0027252829999999998 2024-08-19T14:27:45.493 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.493 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.493 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:27:45.494 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:36.843158+0000", 2024-08-19T14:27:45.494 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00019457400000000001 2024-08-19T14:27:45.494 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.494 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.494 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 1,5", 2024-08-19T14:27:45.494 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:36.843342+0000", 2024-08-19T14:27:45.494 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000184661 2024-08-19T14:27:45.494 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.494 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.494 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:45.494 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:36.856641+0000", 2024-08-19T14:27:45.494 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.013298140999999999 2024-08-19T14:27:45.494 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.495 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.495 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:45.495 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:36.856697+0000", 2024-08-19T14:27:45.495 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6005999999999999e-05 2024-08-19T14:27:45.495 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.495 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.495 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:27:45.495 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:36.856710+0000", 2024-08-19T14:27:45.495 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3407999999999999e-05 2024-08-19T14:27:45.495 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.495 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.495 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:27:45.495 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:36.856766+0000", 2024-08-19T14:27:45.495 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6520000000000001e-05 2024-08-19T14:27:45.496 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.496 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.496 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:27:45.496 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:36.940066+0000", 2024-08-19T14:27:45.496 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.083299734 2024-08-19T14:27:45.496 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:45.496 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:27:45.496 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:45.497 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.497 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.497 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:92574 5.9 5:954c195b:::benchmark_data_smithi045_85309_object92573:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e444)", 2024-08-19T14:27:45.497 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:25:36.851423+0000", 2024-08-19T14:27:45.497 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 128.57166160200001, 2024-08-19T14:27:45.497 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.096954398999999997, 2024-08-19T14:27:45.497 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:27:45.497 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:27:45.498 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:27:45.498 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:27:45.498 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:27:45.498 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:27:45.498 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 92574 2024-08-19T14:27:45.498 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.498 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:27:45.498 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.498 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:27:45.498 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:36.851423+0000", 2024-08-19T14:27:45.498 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:45.498 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.498 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.498 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:27:45.499 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:36.851423+0000", 2024-08-19T14:27:45.499 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:45.499 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.499 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.499 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:27:45.499 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:36.851425+0000", 2024-08-19T14:27:45.499 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2410000000000001e-06 2024-08-19T14:27:45.499 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.499 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.500 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:27:45.500 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:36.851480+0000", 2024-08-19T14:27:45.500 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5155000000000003e-05 2024-08-19T14:27:45.500 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.500 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.500 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:27:45.500 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:36.851489+0000", 2024-08-19T14:27:45.500 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.7390000000000005e-06 2024-08-19T14:27:45.500 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.500 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.500 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:27:45.500 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:36.851494+0000", 2024-08-19T14:27:45.500 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3329999999999996e-06 2024-08-19T14:27:45.500 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.501 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.501 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:27:45.501 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:36.856668+0000", 2024-08-19T14:27:45.501 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0051744829999999997 2024-08-19T14:27:45.501 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.501 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.501 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:27:45.501 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:36.856772+0000", 2024-08-19T14:27:45.501 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000103173 2024-08-19T14:27:45.501 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.501 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.501 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 9,11", 2024-08-19T14:27:45.501 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:36.856893+0000", 2024-08-19T14:27:45.502 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000121455 2024-08-19T14:27:45.502 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.502 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.502 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:27:45.502 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:36.940129+0000", 2024-08-19T14:27:45.502 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.083235839000000006 2024-08-19T14:27:45.502 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.502 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.502 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:45.502 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:36.940290+0000", 2024-08-19T14:27:45.502 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00016116200000000001 2024-08-19T14:27:45.502 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.502 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.503 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:45.503 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:36.948279+0000", 2024-08-19T14:27:45.503 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0079892629999999999 2024-08-19T14:27:45.503 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.503 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.503 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:27:45.503 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:36.948339+0000", 2024-08-19T14:27:45.503 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9823e-05 2024-08-19T14:27:45.503 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.503 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.503 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:27:45.503 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:36.948378+0000", 2024-08-19T14:27:45.503 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8733000000000003e-05 2024-08-19T14:27:45.504 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:45.504 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:27:45.504 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:45.504 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.504 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.504 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDPGPushReply(5.19 444/442 [PushReplyOp(5:99e915ee:::benchmark_data_smithi045_85309_object4538:head)])", 2024-08-19T14:27:45.504 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:25:36.860235+0000", 2024-08-19T14:27:45.504 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 128.56285023000001, 2024-08-19T14:27:45.504 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.10405706300000001, 2024-08-19T14:27:45.504 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:27:45.504 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:27:45.504 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:27:45.504 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:27:45.504 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.505 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:27:45.505 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:36.860235+0000", 2024-08-19T14:27:45.505 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:45.505 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.505 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.505 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:27:45.505 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:36.860235+0000", 2024-08-19T14:27:45.505 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:45.505 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.505 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.505 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:27:45.505 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:36.860236+0000", 2024-08-19T14:27:45.505 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.5400000000000001e-07 2024-08-19T14:27:45.505 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.505 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.506 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:27:45.506 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:36.860249+0000", 2024-08-19T14:27:45.506 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3607e-05 2024-08-19T14:27:45.506 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.506 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.506 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:27:45.506 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:36.943383+0000", 2024-08-19T14:27:45.506 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.083134068000000005 2024-08-19T14:27:45.506 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.506 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.506 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:27:45.506 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:36.943397+0000", 2024-08-19T14:27:45.506 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3988999999999999e-05 2024-08-19T14:27:45.507 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.507 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.507 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:27:45.507 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:36.961780+0000", 2024-08-19T14:27:45.507 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.018382586999999999 2024-08-19T14:27:45.507 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.507 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.507 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:27:45.507 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:36.964292+0000", 2024-08-19T14:27:45.507 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.002512158 2024-08-19T14:27:45.507 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:45.507 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:27:45.507 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:45.507 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.508 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.508 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:92844 5.19 5:9cd3cd78:::benchmark_data_smithi045_85309_object92843:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e445)", 2024-08-19T14:27:45.508 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:25:37.246536+0000", 2024-08-19T14:27:45.508 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 128.17654938300001, 2024-08-19T14:27:45.508 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.188362328, 2024-08-19T14:27:45.508 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:27:45.508 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:27:45.508 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:27:45.508 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:27:45.508 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:27:45.508 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:27:45.508 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 92844 2024-08-19T14:27:45.508 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.509 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:27:45.509 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.509 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:27:45.509 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.246536+0000", 2024-08-19T14:27:45.509 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:45.509 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.509 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.509 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:27:45.509 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.246536+0000", 2024-08-19T14:27:45.509 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:45.509 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.509 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.509 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:27:45.509 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.246537+0000", 2024-08-19T14:27:45.510 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2899999999999999e-06 2024-08-19T14:27:45.510 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.510 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.510 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:27:45.510 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.246579+0000", 2024-08-19T14:27:45.510 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2042000000000002e-05 2024-08-19T14:27:45.510 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.510 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.510 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:27:45.510 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.246589+0000", 2024-08-19T14:27:45.510 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0033e-05 2024-08-19T14:27:45.510 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.510 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.510 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:27:45.511 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.246603+0000", 2024-08-19T14:27:45.511 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4054e-05 2024-08-19T14:27:45.511 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.511 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.511 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:27:45.511 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.252793+0000", 2024-08-19T14:27:45.511 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0061897319999999999 2024-08-19T14:27:45.511 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.511 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.511 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:27:45.511 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.252929+0000", 2024-08-19T14:27:45.512 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013627800000000001 2024-08-19T14:27:45.512 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.512 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.512 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 1,5", 2024-08-19T14:27:45.512 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.253056+0000", 2024-08-19T14:27:45.512 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000126389 2024-08-19T14:27:45.512 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.512 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.512 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:27:45.512 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.254339+0000", 2024-08-19T14:27:45.512 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001283588 2024-08-19T14:27:45.513 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.513 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.513 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:45.513 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.260841+0000", 2024-08-19T14:27:45.513 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.006501522 2024-08-19T14:27:45.513 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.513 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.513 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:45.513 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.432465+0000", 2024-08-19T14:27:45.513 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.17162460500000001 2024-08-19T14:27:45.513 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.513 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.513 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:27:45.513 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.432578+0000", 2024-08-19T14:27:45.514 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000113151 2024-08-19T14:27:45.514 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.514 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.514 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:27:45.514 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.434898+0000", 2024-08-19T14:27:45.514 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.002319644 2024-08-19T14:27:45.514 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:45.514 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:27:45.514 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:45.514 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.514 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.514 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:140418 5.10 5:0c2afd7d:::benchmark_data_smithi045_85309_object20165:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e468)", 2024-08-19T14:27:45.514 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:26:38.020272+0000", 2024-08-19T14:27:45.515 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 67.402813304999995, 2024-08-19T14:27:45.515 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.121873517, 2024-08-19T14:27:45.515 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:27:45.515 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:27:45.515 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:27:45.515 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:27:45.515 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:27:45.515 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:27:45.515 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 140418 2024-08-19T14:27:45.515 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.515 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:27:45.515 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.515 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:27:45.515 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:26:38.020272+0000", 2024-08-19T14:27:45.516 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:45.516 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.516 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.516 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:27:45.516 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:26:38.020272+0000", 2024-08-19T14:27:45.516 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:45.516 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.516 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.516 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:27:45.516 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:26:38.020273+0000", 2024-08-19T14:27:45.516 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.1999999999999998e-07 2024-08-19T14:27:45.516 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.516 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.516 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:27:45.516 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:26:38.020277+0000", 2024-08-19T14:27:45.517 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5159999999999996e-06 2024-08-19T14:27:45.517 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.517 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.517 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:27:45.517 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:26:38.020289+0000", 2024-08-19T14:27:45.517 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2279000000000001e-05 2024-08-19T14:27:45.517 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.517 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.517 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:27:45.517 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:26:38.020296+0000", 2024-08-19T14:27:45.517 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8170000000000001e-06 2024-08-19T14:27:45.517 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.517 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.517 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:27:45.518 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:26:38.020352+0000", 2024-08-19T14:27:45.518 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5266000000000002e-05 2024-08-19T14:27:45.518 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.518 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.518 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:27:45.518 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:26:38.020452+0000", 2024-08-19T14:27:45.519 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000100794 2024-08-19T14:27:45.519 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.519 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.519 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0,8", 2024-08-19T14:27:45.519 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:26:38.020556+0000", 2024-08-19T14:27:45.519 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00010367899999999999 2024-08-19T14:27:45.519 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.520 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.520 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:27:45.520 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:26:38.023320+0000", 2024-08-19T14:27:45.520 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.002763485 2024-08-19T14:27:45.520 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.520 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.520 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:45.520 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:26:38.025085+0000", 2024-08-19T14:27:45.520 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0017659469999999999 2024-08-19T14:27:45.520 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.520 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.520 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:45.520 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:26:38.142052+0000", 2024-08-19T14:27:45.521 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.11696635599999999 2024-08-19T14:27:45.521 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.521 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.521 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:27:45.521 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:26:38.142112+0000", 2024-08-19T14:27:45.521 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0482999999999997e-05 2024-08-19T14:27:45.521 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.521 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.521 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:27:45.521 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:26:38.142145+0000", 2024-08-19T14:27:45.521 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2975000000000002e-05 2024-08-19T14:27:45.521 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:45.521 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:27:45.521 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:45.522 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.522 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.522 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:27:45.523 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:140442 5.10 5:0829aa6a:::benchmark_data_smithi045_85309_object20189:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e468)", 2024-08-19T14:27:45.523 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:26:38.038407+0000", 2024-08-19T14:27:45.523 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 67.384678167999994, 2024-08-19T14:27:45.523 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.121998258, 2024-08-19T14:27:45.523 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:27:45.524 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:27:45.524 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:27:45.524 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:27:45.524 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:27:45.524 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:27:45.524 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 140442 2024-08-19T14:27:45.524 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.524 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:27:45.524 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.524 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:27:45.524 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:26:38.038407+0000", 2024-08-19T14:27:45.524 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:45.524 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.524 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.525 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:27:45.525 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:26:38.038407+0000", 2024-08-19T14:27:45.525 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:45.525 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.525 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.525 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:27:45.525 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:26:38.038409+0000", 2024-08-19T14:27:45.525 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.61e-06 2024-08-19T14:27:45.525 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.525 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.525 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:27:45.525 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:26:38.038414+0000", 2024-08-19T14:27:45.525 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7300000000000002e-06 2024-08-19T14:27:45.525 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.526 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.526 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:27:45.526 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:26:38.038430+0000", 2024-08-19T14:27:45.526 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5410999999999999e-05 2024-08-19T14:27:45.526 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.526 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.526 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:27:45.526 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:26:38.038438+0000", 2024-08-19T14:27:45.526 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1270000000000003e-06 2024-08-19T14:27:45.526 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.526 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.526 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:27:45.526 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:26:38.038500+0000", 2024-08-19T14:27:45.527 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2601999999999995e-05 2024-08-19T14:27:45.527 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.527 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.527 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:27:45.527 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:26:38.038634+0000", 2024-08-19T14:27:45.527 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013324499999999999 2024-08-19T14:27:45.527 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.527 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.527 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0,8", 2024-08-19T14:27:45.527 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:26:38.038754+0000", 2024-08-19T14:27:45.527 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012026 2024-08-19T14:27:45.527 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.527 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.527 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:27:45.528 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:26:38.040820+0000", 2024-08-19T14:27:45.528 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0020659630000000001 2024-08-19T14:27:45.528 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.528 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.528 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:45.528 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:26:38.041110+0000", 2024-08-19T14:27:45.528 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00029045900000000002 2024-08-19T14:27:45.528 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.528 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.528 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:45.528 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:26:38.160331+0000", 2024-08-19T14:27:45.528 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.119220961 2024-08-19T14:27:45.528 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.528 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.529 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:27:45.529 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:26:38.160364+0000", 2024-08-19T14:27:45.529 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2747999999999999e-05 2024-08-19T14:27:45.529 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.529 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.529 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:27:45.529 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:26:38.160405+0000", 2024-08-19T14:27:45.529 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1142e-05 2024-08-19T14:27:45.529 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:45.529 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:27:45.529 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:45.529 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.529 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.529 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:178021 5.10 5:0f12eb93:::benchmark_data_smithi045_85309_object57768:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e475)", 2024-08-19T14:27:45.530 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:27:07.982386+0000", 2024-08-19T14:27:45.530 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 37.440698625000003, 2024-08-19T14:27:45.530 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.12361873299999999, 2024-08-19T14:27:45.530 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:27:45.530 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:27:45.530 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:27:45.530 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:27:45.530 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:27:45.530 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:27:45.530 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 178021 2024-08-19T14:27:45.530 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.530 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:27:45.530 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.531 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:27:45.531 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:07.982386+0000", 2024-08-19T14:27:45.531 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:45.531 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.531 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.531 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:27:45.531 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:07.982386+0000", 2024-08-19T14:27:45.531 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:45.531 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.531 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.531 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:27:45.531 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:07.982388+0000", 2024-08-19T14:27:45.531 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.505e-06 2024-08-19T14:27:45.531 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.532 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.532 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:27:45.532 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:07.982394+0000", 2024-08-19T14:27:45.532 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8150000000000002e-06 2024-08-19T14:27:45.532 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.532 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.532 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:27:45.532 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:07.982410+0000", 2024-08-19T14:27:45.532 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6688e-05 2024-08-19T14:27:45.532 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.532 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.532 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:27:45.532 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:07.982419+0000", 2024-08-19T14:27:45.532 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.2689999999999999e-06 2024-08-19T14:27:45.533 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.533 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.533 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:27:45.533 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:07.982483+0000", 2024-08-19T14:27:45.533 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4616000000000006e-05 2024-08-19T14:27:45.533 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.533 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.533 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:27:45.533 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:07.982593+0000", 2024-08-19T14:27:45.533 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00010999500000000001 2024-08-19T14:27:45.533 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.533 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.533 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0,8", 2024-08-19T14:27:45.533 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:07.982692+0000", 2024-08-19T14:27:45.534 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.8523000000000005e-05 2024-08-19T14:27:45.534 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.534 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.534 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:27:45.534 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:07.985725+0000", 2024-08-19T14:27:45.534 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0030332760000000001 2024-08-19T14:27:45.534 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.534 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.534 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:45.534 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:07.987629+0000", 2024-08-19T14:27:45.534 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0019042309999999999 2024-08-19T14:27:45.534 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.534 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.534 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:45.535 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:08.105905+0000", 2024-08-19T14:27:45.535 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.11827531400000001 2024-08-19T14:27:45.535 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.535 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.535 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:27:45.535 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:08.105959+0000", 2024-08-19T14:27:45.535 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4123999999999999e-05 2024-08-19T14:27:45.535 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.535 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.535 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:27:45.535 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:08.106005+0000", 2024-08-19T14:27:45.535 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6377e-05 2024-08-19T14:27:45.535 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:45.535 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:27:45.536 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:45.536 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.536 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.536 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:178026 5.10 5:0e5302d6:::benchmark_data_smithi045_85309_object57773:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e475)", 2024-08-19T14:27:45.536 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:27:07.994616+0000", 2024-08-19T14:27:45.536 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 37.428468614000003, 2024-08-19T14:27:45.536 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.111501924, 2024-08-19T14:27:45.536 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:27:45.536 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:27:45.536 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:27:45.536 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:27:45.536 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:27:45.536 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:27:45.537 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 178026 2024-08-19T14:27:45.537 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.537 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:27:45.537 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.537 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:27:45.537 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:07.994616+0000", 2024-08-19T14:27:45.537 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:45.537 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.537 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.537 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:27:45.537 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:07.994616+0000", 2024-08-19T14:27:45.537 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:45.537 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.537 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.538 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:27:45.538 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:07.994617+0000", 2024-08-19T14:27:45.538 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.7600000000000006e-07 2024-08-19T14:27:45.538 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.538 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.538 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:27:45.538 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:07.994620+0000", 2024-08-19T14:27:45.538 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9960000000000002e-06 2024-08-19T14:27:45.538 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.538 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.538 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:27:45.538 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:07.994631+0000", 2024-08-19T14:27:45.538 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0555e-05 2024-08-19T14:27:45.539 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.539 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.539 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:27:45.539 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:07.994635+0000", 2024-08-19T14:27:45.539 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8199999999999998e-06 2024-08-19T14:27:45.539 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.539 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.539 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:27:45.539 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:07.994713+0000", 2024-08-19T14:27:45.539 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7987999999999999e-05 2024-08-19T14:27:45.539 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.539 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.539 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:27:45.539 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:07.994843+0000", 2024-08-19T14:27:45.540 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012988400000000001 2024-08-19T14:27:45.540 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.540 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.540 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0,8", 2024-08-19T14:27:45.540 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:07.994949+0000", 2024-08-19T14:27:45.540 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000106258 2024-08-19T14:27:45.540 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.540 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.540 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:27:45.540 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:07.997707+0000", 2024-08-19T14:27:45.540 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0027578490000000002 2024-08-19T14:27:45.540 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.540 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.540 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:45.541 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:07.997923+0000", 2024-08-19T14:27:45.541 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00021638699999999999 2024-08-19T14:27:45.541 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.541 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.541 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:45.541 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:08.106047+0000", 2024-08-19T14:27:45.541 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.108123764 2024-08-19T14:27:45.541 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.541 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.541 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:27:45.541 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:08.106088+0000", 2024-08-19T14:27:45.541 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0809999999999997e-05 2024-08-19T14:27:45.541 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.542 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.542 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:27:45.542 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:08.106118+0000", 2024-08-19T14:27:45.542 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0636999999999998e-05 2024-08-19T14:27:45.542 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:45.542 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:27:45.542 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:45.542 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.542 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.542 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:181713 5.9 5:93e540c3:::benchmark_data_smithi045_85309_object61460:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e477)", 2024-08-19T14:27:45.542 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:27:15.012008+0000", 2024-08-19T14:27:45.542 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 30.411077021000001, 2024-08-19T14:27:45.542 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.13837481099999999, 2024-08-19T14:27:45.542 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:27:45.543 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:27:45.543 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:27:45.543 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:27:45.543 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:27:45.543 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:27:45.543 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 181713 2024-08-19T14:27:45.543 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.543 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:27:45.543 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.543 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:27:45.543 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:15.012008+0000", 2024-08-19T14:27:45.543 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:45.543 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.544 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.544 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:27:45.544 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:15.012008+0000", 2024-08-19T14:27:45.544 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:45.544 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.544 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.544 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:27:45.544 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:15.012010+0000", 2024-08-19T14:27:45.544 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7969999999999999e-06 2024-08-19T14:27:45.544 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.544 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.544 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:27:45.544 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:15.012018+0000", 2024-08-19T14:27:45.545 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.123e-06 2024-08-19T14:27:45.545 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.545 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.545 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:27:45.545 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:15.012039+0000", 2024-08-19T14:27:45.545 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0533000000000001e-05 2024-08-19T14:27:45.545 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.545 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.545 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:27:45.545 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:15.012048+0000", 2024-08-19T14:27:45.545 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.2920000000000001e-06 2024-08-19T14:27:45.545 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.545 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.546 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:27:45.546 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:15.012117+0000", 2024-08-19T14:27:45.546 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9443999999999994e-05 2024-08-19T14:27:45.546 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.546 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.546 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:27:45.546 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:15.012242+0000", 2024-08-19T14:27:45.546 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012462199999999999 2024-08-19T14:27:45.546 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.546 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.546 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 9,11", 2024-08-19T14:27:45.546 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:15.012346+0000", 2024-08-19T14:27:45.546 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00010425399999999999 2024-08-19T14:27:45.546 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.547 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.547 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:27:45.547 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:15.015227+0000", 2024-08-19T14:27:45.547 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0028807619999999998 2024-08-19T14:27:45.547 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.547 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.547 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:45.547 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:15.015530+0000", 2024-08-19T14:27:45.547 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0003032 2024-08-19T14:27:45.547 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.547 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.547 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:45.547 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:15.150281+0000", 2024-08-19T14:27:45.547 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.134750539 2024-08-19T14:27:45.548 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.548 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.548 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:27:45.548 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:15.150336+0000", 2024-08-19T14:27:45.548 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5535999999999998e-05 2024-08-19T14:27:45.548 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.548 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.548 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:27:45.548 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:15.150383+0000", 2024-08-19T14:27:45.548 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6709000000000003e-05 2024-08-19T14:27:45.548 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:45.548 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:27:45.548 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:45.548 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.549 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.549 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:181723 5.9 5:94591254:::benchmark_data_smithi045_85309_object61470:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e477)", 2024-08-19T14:27:45.549 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:27:15.024387+0000", 2024-08-19T14:27:45.549 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 30.398698039999999, 2024-08-19T14:27:45.549 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.12613291600000001, 2024-08-19T14:27:45.549 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:27:45.549 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:27:45.549 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:27:45.549 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:27:45.549 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:27:45.549 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:27:45.549 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 181723 2024-08-19T14:27:45.549 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.549 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:27:45.550 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.550 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:27:45.550 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:15.024387+0000", 2024-08-19T14:27:45.550 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:45.550 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.550 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.550 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:27:45.550 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:15.024387+0000", 2024-08-19T14:27:45.550 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:45.550 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.550 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.550 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:27:45.550 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:15.024388+0000", 2024-08-19T14:27:45.550 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.37e-06 2024-08-19T14:27:45.551 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.551 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.551 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:27:45.551 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:15.024393+0000", 2024-08-19T14:27:45.551 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.028e-06 2024-08-19T14:27:45.551 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.551 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.551 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:27:45.551 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:15.024405+0000", 2024-08-19T14:27:45.551 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1172000000000001e-05 2024-08-19T14:27:45.551 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.551 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.551 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:27:45.551 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:15.024424+0000", 2024-08-19T14:27:45.552 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9103999999999999e-05 2024-08-19T14:27:45.552 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.552 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.552 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:27:45.552 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:15.024488+0000", 2024-08-19T14:27:45.552 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4175000000000001e-05 2024-08-19T14:27:45.552 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.552 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.552 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:27:45.552 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:15.024618+0000", 2024-08-19T14:27:45.552 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013007000000000001 2024-08-19T14:27:45.552 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.552 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.552 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 9,11", 2024-08-19T14:27:45.553 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:15.024739+0000", 2024-08-19T14:27:45.553 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000121075 2024-08-19T14:27:45.553 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.553 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.553 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:27:45.553 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:15.027644+0000", 2024-08-19T14:27:45.553 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0029049810000000001 2024-08-19T14:27:45.553 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.553 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.553 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:45.553 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:15.027788+0000", 2024-08-19T14:27:45.553 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000143988 2024-08-19T14:27:45.553 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.554 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.554 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:45.554 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:15.150429+0000", 2024-08-19T14:27:45.554 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.12264072199999999 2024-08-19T14:27:45.554 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.554 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.554 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:27:45.554 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:15.150481+0000", 2024-08-19T14:27:45.554 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2553999999999999e-05 2024-08-19T14:27:45.554 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.554 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.554 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:27:45.554 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:15.150520+0000", 2024-08-19T14:27:45.555 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8677000000000003e-05 2024-08-19T14:27:45.555 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:45.555 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:27:45.555 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:45.555 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.555 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.555 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.25813.0:201740 5.d e485/419 5:b391c56d:::benchmark_data_smithi045_85309_object81487:head v 485'15393, mlcod=485'15389)", 2024-08-19T14:27:45.555 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:27:34.054436+0000", 2024-08-19T14:27:45.555 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 11.368649485000001, 2024-08-19T14:27:45.555 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.063495234999999997, 2024-08-19T14:27:45.555 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:27:45.555 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:27:45.555 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:27:45.555 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:27:45.556 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.556 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:27:45.556 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:34.054436+0000", 2024-08-19T14:27:45.556 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:45.556 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.556 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.556 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:27:45.556 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:34.054436+0000", 2024-08-19T14:27:45.556 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:45.556 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.556 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.556 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:27:45.556 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:34.054436+0000", 2024-08-19T14:27:45.556 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.9299999999999996e-07 2024-08-19T14:27:45.557 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.557 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.557 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:27:45.557 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:34.054443+0000", 2024-08-19T14:27:45.557 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.5189999999999998e-06 2024-08-19T14:27:45.557 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.557 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.557 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:27:45.557 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:34.054453+0000", 2024-08-19T14:27:45.557 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0203999999999999e-05 2024-08-19T14:27:45.557 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.557 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.557 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:27:45.558 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:34.054460+0000", 2024-08-19T14:27:45.558 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4740000000000002e-06 2024-08-19T14:27:45.558 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.558 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.558 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:27:45.558 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:34.054519+0000", 2024-08-19T14:27:45.558 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9626999999999997e-05 2024-08-19T14:27:45.558 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.558 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.558 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:27:45.558 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:34.054537+0000", 2024-08-19T14:27:45.558 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7541999999999999e-05 2024-08-19T14:27:45.558 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.558 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.559 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:27:45.559 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:34.117877+0000", 2024-08-19T14:27:45.559 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.063340281999999998 2024-08-19T14:27:45.559 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.559 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.559 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:27:45.559 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:34.117931+0000", 2024-08-19T14:27:45.559 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3693999999999998e-05 2024-08-19T14:27:45.559 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:45.559 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:27:45.559 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:45.559 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.559 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.560 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.25813.0:201742 5.d e485/419 5:bd2d6d46:::benchmark_data_smithi045_85309_object81489:head v 485'15394, mlcod=485'15390)", 2024-08-19T14:27:45.560 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:27:34.061004+0000", 2024-08-19T14:27:45.560 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 11.362081452, 2024-08-19T14:27:45.560 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.071699502999999998, 2024-08-19T14:27:45.560 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:27:45.560 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:27:45.560 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:27:45.560 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:27:45.560 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.560 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:27:45.560 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:34.061004+0000", 2024-08-19T14:27:45.560 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:45.560 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.560 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.561 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:27:45.561 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:34.061004+0000", 2024-08-19T14:27:45.561 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:27:45.561 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.561 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.561 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:27:45.561 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:34.061004+0000", 2024-08-19T14:27:45.561 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0900000000000001e-07 2024-08-19T14:27:45.561 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.561 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.561 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:27:45.561 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:34.061010+0000", 2024-08-19T14:27:45.561 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0229999999999998e-06 2024-08-19T14:27:45.562 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.562 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.562 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:27:45.562 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:34.099472+0000", 2024-08-19T14:27:45.562 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.038461869000000003 2024-08-19T14:27:45.562 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.562 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.562 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:27:45.562 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:34.099478+0000", 2024-08-19T14:27:45.562 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7239999999999997e-06 2024-08-19T14:27:45.562 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.562 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.562 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:27:45.562 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:34.099587+0000", 2024-08-19T14:27:45.563 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000108868 2024-08-19T14:27:45.563 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.563 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.563 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:27:45.563 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:34.099607+0000", 2024-08-19T14:27:45.563 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0078e-05 2024-08-19T14:27:45.563 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.563 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.563 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:27:45.563 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:34.132662+0000", 2024-08-19T14:27:45.563 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.033054808999999997 2024-08-19T14:27:45.563 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:27:45.563 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:27:45.563 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:27:45.564 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:34.132703+0000", 2024-08-19T14:27:45.564 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1423e-05 2024-08-19T14:27:45.564 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:45.564 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:27:45.564 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:45.564 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:27:45.564 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:27:45.564 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:27:45.596 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:45.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:45.597 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:27:45.698 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:27:45.727 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:45.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:45.728 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:27:45.829 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:27:45.862 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:45.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:45.863 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:27:45.964 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:27:45.989 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:45.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:45.990 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:27:46.091 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:27:46.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:46.126 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:46.126 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:27:46.180 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:45 smithi031 ceph-mon[16336]: pgmap v1123: 57 pgs: 57 active+clean; 1.8 GiB data, 11 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 1.22k op/s 2024-08-19T14:27:46.180 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:45 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/4135610934' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "71"}]: dispatch 2024-08-19T14:27:46.227 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:27:46.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:45 smithi032 ceph-mon[20004]: pgmap v1123: 57 pgs: 57 active+clean; 1.8 GiB data, 11 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 1.22k op/s 2024-08-19T14:27:46.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:45 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/4135610934' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "71"}]: dispatch 2024-08-19T14:27:46.259 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:46.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:46.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:27:46.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:45 smithi045 ceph-mon[20143]: pgmap v1123: 57 pgs: 57 active+clean; 1.8 GiB data, 11 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 1.22k op/s 2024-08-19T14:27:46.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:45 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/4135610934' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "71"}]: dispatch 2024-08-19T14:27:46.362 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:27:46.418 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:46.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:46.419 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:27:46.520 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:27:46.549 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:46.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:46.550 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:27:46.651 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:27:46.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:46.683 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:46.684 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:27:46.785 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:27:46.816 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:46.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:46.817 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:27:46.848 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:27:46.852 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-19T14:27:46.917 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:27:46.947 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:46.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:46.948 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:27:47.049 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:27:47.083 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:47.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:47.085 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:27:47.186 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:27:47.207 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:47.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:47.208 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:27:47.309 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:27:47.343 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:47.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:47.344 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:27:47.424 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-08-19T14:27:47.445 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:27:47.475 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:47.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:47.476 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:27:47.577 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:27:47.603 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:47.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:47.604 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:27:47.705 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:27:47.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:47.744 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:47.744 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:27:47.822 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:27:47.844 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:27:47.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:47.873 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:47.873 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:27:47.909 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is unset 2024-08-19T14:27:47.974 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:27:48.001 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:48.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:48.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:27:48.103 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:27:48.129 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:27:48.131 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:48.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:48.132 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:27:48.163 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:47 smithi031 ceph-mon[16336]: pgmap v1124: 57 pgs: 57 active+clean; 1.7 GiB data, 11 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 1.16k op/s 2024-08-19T14:27:48.163 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:47 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1066660897' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:27:48.222 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:47 smithi045 ceph-mon[20143]: pgmap v1124: 57 pgs: 57 active+clean; 1.7 GiB data, 11 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 1.16k op/s 2024-08-19T14:27:48.222 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:47 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1066660897' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:27:48.233 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:27:48.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:47 smithi032 ceph-mon[20004]: pgmap v1124: 57 pgs: 57 active+clean; 1.7 GiB data, 11 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 1.16k op/s 2024-08-19T14:27:48.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:47 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1066660897' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:27:48.268 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:48.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:48.269 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:27:48.370 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:27:48.375 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:27:48.375 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [ 2024-08-19T14:27:48.375 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:48.375 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25813.0:218630 5.e 5:71f2a037:::benchmark_data_smithi045_85309_object98377:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e489)", 2024-08-19T14:27:48.376 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:27:48.366415+0000", 2024-08-19T14:27:48.376 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.0065059250000000001, 2024-08-19T14:27:48.376 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0065376080000000003, 2024-08-19T14:27:48.376 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:27:48.376 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:27:48.376 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "waiting for sub ops", 2024-08-19T14:27:48.376 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:27:48.376 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25813", 2024-08-19T14:27:48.376 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:27:48.376 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 218630 2024-08-19T14:27:48.376 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:48.376 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:27:48.376 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:48.377 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:27:48.377 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:48.366415+0000", 2024-08-19T14:27:48.377 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:27:48.377 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:48.377 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:48.377 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:27:48.377 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:48.366415+0000", 2024-08-19T14:27:48.377 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:27:48.377 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:48.377 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:48.377 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:27:48.377 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:48.366416+0000", 2024-08-19T14:27:48.377 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.5700000000000001e-07 2024-08-19T14:27:48.377 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:48.378 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:48.378 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:27:48.378 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:48.366420+0000", 2024-08-19T14:27:48.378 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.206e-06 2024-08-19T14:27:48.378 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:48.378 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:48.378 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:27:48.378 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:48.366430+0000", 2024-08-19T14:27:48.378 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.6719999999999999e-06 2024-08-19T14:27:48.378 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:48.379 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:48.379 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:27:48.379 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:48.366434+0000", 2024-08-19T14:27:48.379 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.6380000000000003e-06 2024-08-19T14:27:48.379 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:48.379 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:48.379 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:27:48.379 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:48.366480+0000", 2024-08-19T14:27:48.379 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.5954000000000001e-05 2024-08-19T14:27:48.379 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:48.379 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:48.379 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:27:48.379 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:48.366606+0000", 2024-08-19T14:27:48.379 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000125604 2024-08-19T14:27:48.380 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:48.380 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:48.380 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,9", 2024-08-19T14:27:48.380 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:48.366734+0000", 2024-08-19T14:27:48.380 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012815999999999999 2024-08-19T14:27:48.380 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:27:48.380 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:27:48.380 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:27:48.380 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:48.380 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:48.380 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop_reply(client.25813.0:218617 5.e e489/477)", 2024-08-19T14:27:48.380 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:27:48.369307+0000", 2024-08-19T14:27:48.380 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.0036140059999999999, 2024-08-19T14:27:48.381 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0037216770000000001, 2024-08-19T14:27:48.381 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:27:48.381 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:27:48.381 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "queued for pg", 2024-08-19T14:27:48.381 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:27:48.381 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:48.381 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:27:48.381 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:48.369307+0000", 2024-08-19T14:27:48.381 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:27:48.381 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:48.381 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:48.381 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:27:48.381 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:48.369307+0000", 2024-08-19T14:27:48.381 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:27:48.382 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:48.382 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:48.382 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:27:48.382 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:48.369308+0000", 2024-08-19T14:27:48.382 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.2600000000000002e-07 2024-08-19T14:27:48.382 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:48.382 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:48.382 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:27:48.382 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:48.369313+0000", 2024-08-19T14:27:48.382 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.3199999999999999e-06 2024-08-19T14:27:48.382 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:48.382 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:48.382 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:27:48.383 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:48.369323+0000", 2024-08-19T14:27:48.383 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.944e-06 2024-08-19T14:27:48.383 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:48.383 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:48.383 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:27:48.383 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:48.369329+0000", 2024-08-19T14:27:48.383 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.4949999999999996e-06 2024-08-19T14:27:48.383 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:27:48.383 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:27:48.383 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:27:48.383 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:48.383 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:48.383 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25813.0:218633 5.e 5.a30c82fe (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e489)", 2024-08-19T14:27:48.383 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:27:48.369668+0000", 2024-08-19T14:27:48.384 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.0032529590000000001, 2024-08-19T14:27:48.384 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0034100630000000001, 2024-08-19T14:27:48.384 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:27:48.384 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:27:48.384 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "queued for pg", 2024-08-19T14:27:48.384 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:27:48.384 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25813", 2024-08-19T14:27:48.384 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:27:48.384 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 218633 2024-08-19T14:27:48.384 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:48.384 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:27:48.384 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:48.384 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:27:48.385 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:48.369668+0000", 2024-08-19T14:27:48.385 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:27:48.385 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:48.385 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:48.385 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:27:48.385 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:48.369668+0000", 2024-08-19T14:27:48.385 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:27:48.385 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:48.385 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:48.385 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:27:48.385 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:48.369669+0000", 2024-08-19T14:27:48.385 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.037e-06 2024-08-19T14:27:48.386 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:48.386 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:48.386 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:27:48.386 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:48.369674+0000", 2024-08-19T14:27:48.386 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0560000000000004e-06 2024-08-19T14:27:48.386 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:48.386 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:48.386 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:27:48.386 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:48.369684+0000", 2024-08-19T14:27:48.386 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0427e-05 2024-08-19T14:27:48.386 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:48.386 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:48.387 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:27:48.387 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:48.369692+0000", 2024-08-19T14:27:48.387 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.3889999999999999e-06 2024-08-19T14:27:48.387 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:27:48.387 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:27:48.387 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:27:48.387 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:48.387 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:48.387 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.25813.0:218632 5.10 e489/466 5:092c0ba2:::benchmark_data_smithi045_85309_object98379:head v 489'9623, mlcod=489'9620)", 2024-08-19T14:27:48.387 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:27:48.370077+0000", 2024-08-19T14:27:48.387 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.0028436609999999999, 2024-08-19T14:27:48.387 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.003055912, 2024-08-19T14:27:48.388 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:27:48.388 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:27:48.388 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-08-19T14:27:48.388 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:27:48.388 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:48.389 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:27:48.389 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:48.370077+0000", 2024-08-19T14:27:48.389 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:27:48.389 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:48.389 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:48.389 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:27:48.389 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:48.370077+0000", 2024-08-19T14:27:48.390 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:27:48.390 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:48.390 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:48.390 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:27:48.390 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:48.370078+0000", 2024-08-19T14:27:48.390 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7700000000000004e-07 2024-08-19T14:27:48.390 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:48.390 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:48.390 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:27:48.390 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:48.370083+0000", 2024-08-19T14:27:48.391 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4929999999999998e-06 2024-08-19T14:27:48.392 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:48.392 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:48.392 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:27:48.392 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:48.370094+0000", 2024-08-19T14:27:48.392 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0832000000000001e-05 2024-08-19T14:27:48.392 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:48.392 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:48.392 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:27:48.392 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:48.370100+0000", 2024-08-19T14:27:48.392 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.0379999999999996e-06 2024-08-19T14:27:48.392 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:48.392 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:48.392 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:27:48.393 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:48.370159+0000", 2024-08-19T14:27:48.393 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.9252000000000001e-05 2024-08-19T14:27:48.393 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:48.393 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:48.393 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:27:48.393 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:48.370179+0000", 2024-08-19T14:27:48.393 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9252000000000001e-05 2024-08-19T14:27:48.393 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:27:48.393 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:27:48.393 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:27:48.393 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:48.394 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:48.394 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.25813.0:218634 5.15 e489/484 5:aa9f8269:::benchmark_data_smithi045_85309_object98381:head v 489'9389, mlcod=489'9387)", 2024-08-19T14:27:48.394 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:27:48.370186+0000", 2024-08-19T14:27:48.394 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.002734612, 2024-08-19T14:27:48.394 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0030047889999999999, 2024-08-19T14:27:48.394 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:27:48.394 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:27:48.394 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-08-19T14:27:48.394 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:27:48.394 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:48.394 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:27:48.394 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:48.370186+0000", 2024-08-19T14:27:48.394 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:27:48.394 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:48.395 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:48.395 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:27:48.395 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:48.370186+0000", 2024-08-19T14:27:48.395 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:27:48.395 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:48.395 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:48.395 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:27:48.395 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:48.370187+0000", 2024-08-19T14:27:48.395 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.3300000000000002e-07 2024-08-19T14:27:48.395 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:48.395 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:48.396 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:27:48.396 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:48.370191+0000", 2024-08-19T14:27:48.396 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.6410000000000001e-06 2024-08-19T14:27:48.396 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:48.396 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:48.396 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:27:48.396 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:48.370200+0000", 2024-08-19T14:27:48.396 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.259e-06 2024-08-19T14:27:48.396 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:48.397 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:48.397 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:27:48.397 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:48.370206+0000", 2024-08-19T14:27:48.397 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.7109999999999998e-06 2024-08-19T14:27:48.397 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:48.397 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:48.397 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:27:48.397 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:48.370269+0000", 2024-08-19T14:27:48.397 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.2147e-05 2024-08-19T14:27:48.397 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:48.397 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:48.397 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:27:48.397 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:48.370281+0000", 2024-08-19T14:27:48.397 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2808e-05 2024-08-19T14:27:48.398 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:27:48.398 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:27:48.398 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:27:48.398 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:48.398 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:48.398 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.25813.0:218631 5.2 e489/439 5:41d25664:::benchmark_data_smithi045_85309_object98378:head v 489'9515, mlcod=489'9513)", 2024-08-19T14:27:48.398 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:27:48.370771+0000", 2024-08-19T14:27:48.398 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.002150276, 2024-08-19T14:27:48.398 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0024798789999999999, 2024-08-19T14:27:48.398 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:27:48.398 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:27:48.398 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-08-19T14:27:48.398 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:27:48.399 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:48.399 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:27:48.399 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:48.370771+0000", 2024-08-19T14:27:48.399 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:27:48.399 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:48.399 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:48.400 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:27:48.400 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:48.370771+0000", 2024-08-19T14:27:48.400 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:27:48.400 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:48.400 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:48.400 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:27:48.400 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:48.370771+0000", 2024-08-19T14:27:48.400 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.9999999999999999e-07 2024-08-19T14:27:48.401 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:48.401 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:48.401 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:27:48.401 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:48.370775+0000", 2024-08-19T14:27:48.401 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.2699999999999998e-06 2024-08-19T14:27:48.401 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:48.401 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:48.401 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:27:48.401 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:48.370781+0000", 2024-08-19T14:27:48.401 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.0190000000000003e-06 2024-08-19T14:27:48.401 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:48.402 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:48.402 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:27:48.402 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:48.370785+0000", 2024-08-19T14:27:48.402 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.2390000000000004e-06 2024-08-19T14:27:48.402 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:48.402 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:48.402 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:27:48.402 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:48.370829+0000", 2024-08-19T14:27:48.403 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.3992000000000002e-05 2024-08-19T14:27:48.403 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:48.403 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:48.403 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:27:48.403 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:48.370847+0000", 2024-08-19T14:27:48.403 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7144000000000001e-05 2024-08-19T14:27:48.403 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:27:48.403 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:27:48.403 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:27:48.403 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:48.404 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:48.404 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25813.0:218635 5.13 5:ce01fc9b:::benchmark_data_smithi045_85309_object98382:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e489)", 2024-08-19T14:27:48.404 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:27:48.371277+0000", 2024-08-19T14:27:48.404 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.001644087, 2024-08-19T14:27:48.404 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0020333450000000002, 2024-08-19T14:27:48.404 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:27:48.404 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:27:48.404 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "waiting for sub ops", 2024-08-19T14:27:48.404 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:27:48.404 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25813", 2024-08-19T14:27:48.405 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:27:48.405 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 218635 2024-08-19T14:27:48.405 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:48.405 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:27:48.405 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:48.405 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:27:48.405 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:48.371277+0000", 2024-08-19T14:27:48.405 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:27:48.405 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:48.405 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:48.406 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:27:48.406 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:48.371277+0000", 2024-08-19T14:27:48.406 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:27:48.406 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:48.406 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:48.406 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:27:48.406 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:48.371278+0000", 2024-08-19T14:27:48.406 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.9500000000000001e-07 2024-08-19T14:27:48.406 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:48.406 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:48.406 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:27:48.407 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:48.371281+0000", 2024-08-19T14:27:48.407 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.8759999999999998e-06 2024-08-19T14:27:48.407 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:48.407 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:48.407 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:27:48.407 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:48.371295+0000", 2024-08-19T14:27:48.407 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3709999999999999e-05 2024-08-19T14:27:48.407 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:48.407 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:48.407 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:27:48.407 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:48.371302+0000", 2024-08-19T14:27:48.407 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.272e-06 2024-08-19T14:27:48.407 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:48.407 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:48.408 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:27:48.408 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:48.371356+0000", 2024-08-19T14:27:48.408 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.3445000000000001e-05 2024-08-19T14:27:48.408 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:48.408 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:48.408 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:27:48.408 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:48.371479+0000", 2024-08-19T14:27:48.408 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012308100000000001 2024-08-19T14:27:48.408 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:48.408 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:48.408 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,5", 2024-08-19T14:27:48.408 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:48.371587+0000", 2024-08-19T14:27:48.408 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000107911 2024-08-19T14:27:48.408 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:27:48.408 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:27:48.409 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:27:48.409 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:48.409 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:48.409 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25813.0:218636 5.12 5:4b4bc288:::benchmark_data_smithi045_85309_object98383:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e489)", 2024-08-19T14:27:48.409 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:27:48.371729+0000", 2024-08-19T14:27:48.409 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.001192287, 2024-08-19T14:27:48.409 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001651376, 2024-08-19T14:27:48.409 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:27:48.409 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:27:48.409 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "waiting for sub ops", 2024-08-19T14:27:48.409 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:27:48.409 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25813", 2024-08-19T14:27:48.409 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:27:48.409 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 218636 2024-08-19T14:27:48.410 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:48.410 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:27:48.410 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:48.410 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:27:48.410 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:48.371729+0000", 2024-08-19T14:27:48.410 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:27:48.410 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:48.410 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:48.410 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:27:48.411 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:48.371729+0000", 2024-08-19T14:27:48.411 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:27:48.411 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:48.411 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:48.411 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:27:48.411 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:48.371729+0000", 2024-08-19T14:27:48.411 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.1500000000000003e-07 2024-08-19T14:27:48.411 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:48.411 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:48.411 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:27:48.412 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:48.371734+0000", 2024-08-19T14:27:48.412 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.6330000000000004e-06 2024-08-19T14:27:48.412 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:48.412 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:48.412 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:27:48.412 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:48.371748+0000", 2024-08-19T14:27:48.412 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4144999999999999e-05 2024-08-19T14:27:48.412 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:48.412 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:48.412 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:27:48.412 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:48.371755+0000", 2024-08-19T14:27:48.412 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.216e-06 2024-08-19T14:27:48.413 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:48.413 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:48.413 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:27:48.413 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:48.371820+0000", 2024-08-19T14:27:48.413 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.4913999999999995e-05 2024-08-19T14:27:48.413 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:48.413 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:48.413 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:27:48.413 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:48.371932+0000", 2024-08-19T14:27:48.413 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00011121899999999999 2024-08-19T14:27:48.413 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:48.413 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:48.413 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 3,5", 2024-08-19T14:27:48.414 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:48.372025+0000", 2024-08-19T14:27:48.414 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.3733000000000005e-05 2024-08-19T14:27:48.414 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:27:48.414 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:27:48.414 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:27:48.414 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:48.414 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:48.414 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop_reply(client.25813.0:218630 5.e e489/477)", 2024-08-19T14:27:48.414 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:27:48.373043+0000", 2024-08-19T14:27:48.414 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 4294967295.9998775, 2024-08-19T14:27:48.414 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000406582, 2024-08-19T14:27:48.414 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:27:48.414 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:27:48.415 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "queued for pg", 2024-08-19T14:27:48.415 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:27:48.415 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:48.415 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:27:48.415 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:48.373043+0000", 2024-08-19T14:27:48.415 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:27:48.415 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:48.415 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:48.415 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:27:48.415 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:48.373043+0000", 2024-08-19T14:27:48.415 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:27:48.415 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:48.415 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:48.415 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:27:48.416 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:48.373044+0000", 2024-08-19T14:27:48.416 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.7999999999999995e-07 2024-08-19T14:27:48.416 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:48.416 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:48.416 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:27:48.416 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:48.373048+0000", 2024-08-19T14:27:48.416 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.36e-06 2024-08-19T14:27:48.416 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:48.416 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:48.416 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:27:48.416 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:48.373137+0000", 2024-08-19T14:27:48.416 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.9594999999999999e-05 2024-08-19T14:27:48.416 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:48.417 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:48.417 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:27:48.417 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:48.373145+0000", 2024-08-19T14:27:48.417 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.5830000000000001e-06 2024-08-19T14:27:48.417 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:27:48.417 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:27:48.417 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:27:48.417 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:48.417 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:48.417 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25813.0:218617 5.e 5:7a8e5142:::benchmark_data_smithi045_85309_object98364:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e489)", 2024-08-19T14:27:48.417 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:27:48.358823+0000", 2024-08-19T14:27:48.417 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.014097464000000001, 2024-08-19T14:27:48.417 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.014675647999999999, 2024-08-19T14:27:48.417 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:27:48.418 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:27:48.418 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "waiting for sub ops", 2024-08-19T14:27:48.418 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:27:48.418 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25813", 2024-08-19T14:27:48.418 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:27:48.418 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 218617 2024-08-19T14:27:48.418 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:48.418 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:27:48.418 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:48.418 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:27:48.418 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:48.358823+0000", 2024-08-19T14:27:48.418 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:27:48.419 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:48.419 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:48.419 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:27:48.419 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:48.358823+0000", 2024-08-19T14:27:48.419 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:27:48.419 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:48.419 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:48.419 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:27:48.419 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:48.358825+0000", 2024-08-19T14:27:48.419 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.271e-06 2024-08-19T14:27:48.419 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:48.419 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:48.419 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:27:48.419 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:48.358830+0000", 2024-08-19T14:27:48.419 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.9289999999999997e-06 2024-08-19T14:27:48.420 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:48.420 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:48.420 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:27:48.420 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:48.358845+0000", 2024-08-19T14:27:48.420 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5098e-05 2024-08-19T14:27:48.420 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:48.420 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:48.420 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:27:48.420 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:48.358852+0000", 2024-08-19T14:27:48.420 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.798e-06 2024-08-19T14:27:48.420 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:48.420 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:48.420 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:27:48.420 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:48.359007+0000", 2024-08-19T14:27:48.421 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015538999999999999 2024-08-19T14:27:48.421 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:48.421 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:48.421 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:27:48.421 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:48.359147+0000", 2024-08-19T14:27:48.421 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000139787 2024-08-19T14:27:48.421 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:48.421 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:48.421 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,9", 2024-08-19T14:27:48.421 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:48.359293+0000", 2024-08-19T14:27:48.421 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014641900000000001 2024-08-19T14:27:48.421 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:48.421 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:48.421 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:27:48.422 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:48.364727+0000", 2024-08-19T14:27:48.422 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0054341449999999996 2024-08-19T14:27:48.422 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:48.422 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:48.422 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:27:48.422 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:48.365096+0000", 2024-08-19T14:27:48.422 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00036842800000000002 2024-08-19T14:27:48.422 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:27:48.422 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:27:48.422 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:27:48.423 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:27:48.423 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-08-19T14:27:48.423 INFO:teuthology.orchestra.run.smithi045.stdout: "num_ops": 10 2024-08-19T14:27:48.423 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:27:48.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:48.424 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:48.424 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:27:48.525 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:27:48.555 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:48.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:48.556 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:27:48.657 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:27:48.694 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:48.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:48.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:27:48.796 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:27:48.831 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:48.832 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:27:48.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:48.833 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:27:48.923 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:48 smithi031 ceph-mon[16336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:27:48.924 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:48 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1066660897' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:27:48.924 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:48 smithi031 ceph-mon[16336]: osdmap e489: 12 total, 12 up, 9 in 2024-08-19T14:27:48.933 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:27:48.961 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:48.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:48.963 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:27:49.063 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:27:49.096 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:49.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:49.098 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:27:49.198 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:27:49.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:49.230 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:49.230 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:27:49.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:48 smithi032 ceph-mon[20004]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:27:49.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:48 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1066660897' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:27:49.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:48 smithi032 ceph-mon[20004]: osdmap e489: 12 total, 12 up, 9 in 2024-08-19T14:27:49.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:48 smithi045 ceph-mon[20143]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:27:49.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:48 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1066660897' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:27:49.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:48 smithi045 ceph-mon[20143]: osdmap e489: 12 total, 12 up, 9 in 2024-08-19T14:27:49.332 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:27:49.358 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-08-19T14:27:49.362 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:49.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:49.363 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:27:49.405 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:27:49.405 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:27:49.429 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:27:49.429 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:27:49.455 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:27:49.455 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:27:49.464 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:27:49.480 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:27:49.480 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:27:49.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:49.496 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:49.497 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:27:49.510 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:27:49.510 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:27:49.541 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:27:49.541 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:27:49.572 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:27:49.573 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:27:49.597 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:27:49.602 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:27:49.603 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:27:49.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:49.627 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:49.627 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:27:49.629 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:27:49.630 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:27:49.663 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:27:49.663 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:27:49.691 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:27:49.691 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:27:49.717 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:27:49.717 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:27:49.729 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:27:49.744 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:27:49.768 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:49.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:49.769 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:27:49.870 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:27:49.902 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:49.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:49.903 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:27:50.004 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:27:50.026 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:50.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:50.027 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:27:50.128 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:27:50.132 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:49 smithi045 ceph-mon[20143]: pgmap v1126: 57 pgs: 57 active+clean; 1.6 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 1.04k op/s 2024-08-19T14:27:50.132 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:49 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/840189515' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:27:50.154 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:50.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:50.155 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:27:50.256 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:27:50.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:49 smithi032 ceph-mon[20004]: pgmap v1126: 57 pgs: 57 active+clean; 1.6 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 1.04k op/s 2024-08-19T14:27:50.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:49 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/840189515' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:27:50.293 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:27:50.293 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:27:50.294 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:50.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:50.295 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:27:50.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:49 smithi031 ceph-mon[16336]: pgmap v1126: 57 pgs: 57 active+clean; 1.6 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 1.04k op/s 2024-08-19T14:27:50.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:49 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/840189515' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:27:50.395 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:27:50.429 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:50.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:50.430 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:27:50.530 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:27:50.579 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:50.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:50.580 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:27:50.681 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:27:50.705 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-19T14:27:50.709 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:50.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:50.710 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:27:50.811 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:27:50.851 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:50.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:50.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:27:50.954 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:27:50.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:50.999 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:50.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:27:51.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:50 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:27:51.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:50 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:27:51.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:50 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:27:51.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:50 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:27:51.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:50 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:27:51.089 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:27:51.100 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:27:51.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:50 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:27:51.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:50 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:27:51.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:50 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:27:51.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:50 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:27:51.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:50 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:27:51.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:51.143 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:51.143 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:27:51.170 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-08-19T14:27:51.225 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:50 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:27:51.225 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:50 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:27:51.225 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:50 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:27:51.225 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:50 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:27:51.225 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:50 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:27:51.244 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:27:51.274 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:51.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:51.276 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:27:51.377 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:27:51.414 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:51.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:51.416 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:27:51.516 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:27:51.541 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:51.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:51.542 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:27:51.559 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:27:51.643 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:27:51.677 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:51.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:51.678 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:27:51.779 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:27:51.809 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:51.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:51.810 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:27:51.854 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 7, 8, 9, 11] out_osds: [10, 6, 2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-08-19T14:27:51.854 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-19T14:27:51.854 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_3 2024-08-19T14:27:51.854 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-19T14:27:51.911 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:27:51.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:51.945 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:51.946 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:27:52.046 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:27:52.082 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:52.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:52.083 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:27:52.139 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:51 smithi032 ceph-mon[20004]: pgmap v1127: 57 pgs: 57 active+clean; 1.4 GiB data, 10 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 1.24k op/s 2024-08-19T14:27:52.139 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:51 smithi032 ceph-mon[20004]: 5.0 scrub starts 2024-08-19T14:27:52.139 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:51 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:27:52.139 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:51 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:27:52.139 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:51 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:27:52.164 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:27:52.164 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:27:52.184 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:27:52.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:52.223 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:52.223 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:27:52.229 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:27:52.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:51 smithi045 ceph-mon[20143]: pgmap v1127: 57 pgs: 57 active+clean; 1.4 GiB data, 10 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 1.24k op/s 2024-08-19T14:27:52.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:51 smithi045 ceph-mon[20143]: 5.0 scrub starts 2024-08-19T14:27:52.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:51 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:27:52.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:51 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:27:52.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:51 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:27:52.323 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:27:52.352 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:51 smithi031 ceph-mon[16336]: pgmap v1127: 57 pgs: 57 active+clean; 1.4 GiB data, 10 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 1.24k op/s 2024-08-19T14:27:52.353 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:51 smithi031 ceph-mon[16336]: 5.0 scrub starts 2024-08-19T14:27:52.353 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:51 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:27:52.353 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:51 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:27:52.353 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:51 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:27:52.353 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is unset 2024-08-19T14:27:52.356 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:52.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:52.357 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:27:52.458 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:27:52.485 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:27:52.489 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:52.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:52.491 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:27:52.592 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:27:52.618 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:52.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:52.619 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:27:52.720 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:27:52.761 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:52.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:52.762 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:27:52.863 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:27:52.891 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:52.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:52.892 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:27:52.961 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-08-19T14:27:52.983 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:52 smithi045 ceph-mon[20143]: 5.0 scrub ok 2024-08-19T14:27:52.983 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:52 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:27:52.983 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:52 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:27:52.983 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:52 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/4250039191' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:27:52.983 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:52 smithi045 ceph-mon[20143]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:27:52.983 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:52 smithi045 ceph-mon[20143]: Cluster is now healthy 2024-08-19T14:27:52.983 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:52 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/4250039191' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:27:52.983 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:52 smithi045 ceph-mon[20143]: osdmap e490: 12 total, 12 up, 9 in 2024-08-19T14:27:52.992 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:27:53.022 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:53.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:53.023 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:27:53.109 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:52 smithi031 ceph-mon[16336]: 5.0 scrub ok 2024-08-19T14:27:53.109 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:52 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:27:53.109 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:52 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:27:53.109 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:52 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/4250039191' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:27:53.109 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:52 smithi031 ceph-mon[16336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:27:53.109 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:52 smithi031 ceph-mon[16336]: Cluster is now healthy 2024-08-19T14:27:53.109 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:52 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/4250039191' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:27:53.109 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:52 smithi031 ceph-mon[16336]: osdmap e490: 12 total, 12 up, 9 in 2024-08-19T14:27:53.124 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:27:53.144 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:53.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:53.145 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:27:53.233 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:27:53.246 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:27:53.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:52 smithi032 ceph-mon[20004]: 5.0 scrub ok 2024-08-19T14:27:53.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:52 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:27:53.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:52 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:27:53.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:52 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/4250039191' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:27:53.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:52 smithi032 ceph-mon[20004]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:27:53.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:52 smithi032 ceph-mon[20004]: Cluster is now healthy 2024-08-19T14:27:53.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:52 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/4250039191' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:27:53.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:52 smithi032 ceph-mon[20004]: osdmap e490: 12 total, 12 up, 9 in 2024-08-19T14:27:53.267 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:53.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:53.268 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:27:53.369 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:27:53.403 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:53.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:53.404 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:27:53.446 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:27:53.447 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-08-19T14:27:53.504 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:27:53.536 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:53.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:53.537 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:27:53.638 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:27:53.671 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:53.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:53.672 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:27:53.773 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:27:53.787 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:27:53.787 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:27:53.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:53.811 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:53.811 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:27:53.912 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:27:53.933 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:53.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:53.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:27:54.035 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:27:54.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:54.068 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:54.068 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:27:54.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:53 smithi031 ceph-mon[16336]: pgmap v1128: 57 pgs: 57 active+clean; 1.3 GiB data, 9.9 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 1.20k op/s 2024-08-19T14:27:54.169 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:27:54.196 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:54.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:54.197 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:27:54.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:53 smithi032 ceph-mon[20004]: pgmap v1128: 57 pgs: 57 active+clean; 1.3 GiB data, 9.9 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 1.20k op/s 2024-08-19T14:27:54.266 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:27:54.294 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":1129,"stamp":"2024-08-19T14:27:52.352667+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1331495480,"num_objects":20311,"num_object_clones":0,"num_object_copies":60933,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20311,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":135215,"num_write_kb":3398593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":260030,"ondisk_log_size":260030,"up":171,"acting":171,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":178,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":9989636,"kb_used_data":4202736,"kb_used_omap":14,"kb_used_meta":5786481,"kb_avail":833716732,"statfs":{"total":863955320832,"available":853725933568,"internally_reserved":0,"allocated":4303601664,"data_stored":4283731691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14353,"internal_metadata":5925357551},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-592052224,"num_objects":-9034,"num_object_clones":0,"num_object_copies":-27102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-9034,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9034,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.348882"},"pg_stats":[{"pgid":"5.4","version":"489'9591","reported_seq":9819,"reported_epoch":489,"state":"active+clean","last_fresh":"2024-08-19T14:27:49.986492+0000","last_change":"2024-08-19T14:27:31.858648+0000","last_active":"2024-08-19T14:27:49.986492+0000","last_peered":"2024-08-19T14:27:49.986492+0000","last_clean":"2024-08-19T14:27:49.986492+0000","last_became_active":"2024-08-19T14:27:31.836653+0000","last_became_peered":"2024-08-19T14:27:31.836653+0000","last_unstale":"2024-08-19T14:27:49.986492+0000","last_undegraded":"2024-08-19T14:27:49.986492+0000","last_fullsized":"2024-08-19T14:27:49.986492+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'8317","last_scrub_stamp":"2024-08-19T14:27:12.684467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:27:12.684467+0000","objects_scrubbed":1938,"log_size":9591,"log_dups_size":0,"ondisk_log_size":9591,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:22.704448+0000","scrub_duration":382,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43122688,"num_objects":658,"num_object_clones":0,"num_object_copies":1974,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":658,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4146,"num_write_kb":65152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,0],"acting":[8,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.0","version":"70'139","reported_seq":1221,"reported_epoch":489,"state":"active+clean","last_fresh":"2024-08-19T14:27:47.919388+0000","last_change":"2024-08-19T14:27:03.929122+0000","last_active":"2024-08-19T14:27:47.919388+0000","last_peered":"2024-08-19T14:27:47.919388+0000","last_clean":"2024-08-19T14:27:47.919388+0000","last_became_active":"2024-08-19T14:18:13.807133+0000","last_became_peered":"2024-08-19T14:18:13.807133+0000","last_unstale":"2024-08-19T14:27:47.919388+0000","last_undegraded":"2024-08-19T14:27:47.919388+0000","last_fullsized":"2024-08-19T14:27:47.919388+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:27:03.929012+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:19:58.072083+0000","last_clean_scrub_stamp":"2024-08-19T14:27:03.929012+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:27.998267+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,9],"acting":[7,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"489'9600","reported_seq":9824,"reported_epoch":489,"state":"active+clean","last_fresh":"2024-08-19T14:27:48.703272+0000","last_change":"2024-08-19T14:27:31.862872+0000","last_active":"2024-08-19T14:27:48.703272+0000","last_peered":"2024-08-19T14:27:48.703272+0000","last_clean":"2024-08-19T14:27:48.703272+0000","last_became_active":"2024-08-19T14:27:31.839708+0000","last_became_peered":"2024-08-19T14:27:31.839708+0000","last_unstale":"2024-08-19T14:27:48.703272+0000","last_undegraded":"2024-08-19T14:27:48.703272+0000","last_fullsized":"2024-08-19T14:27:48.703272+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"482'8807","last_scrub_stamp":"2024-08-19T14:27:26.742962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:27:26.742962+0000","objects_scrubbed":1505,"log_size":9600,"log_dups_size":0,"ondisk_log_size":9600,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:32.743963+0000","scrub_duration":429,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46202880,"num_objects":705,"num_object_clones":0,"num_object_copies":2115,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4039,"num_write_kb":62848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,9],"acting":[1,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"70'139","reported_seq":1120,"reported_epoch":489,"state":"active+clean","last_fresh":"2024-08-19T14:27:47.917987+0000","last_change":"2024-08-19T14:27:04.891618+0000","last_active":"2024-08-19T14:27:47.917987+0000","last_peered":"2024-08-19T14:27:47.917987+0000","last_clean":"2024-08-19T14:27:47.917987+0000","last_became_active":"2024-08-19T14:18:13.804721+0000","last_became_peered":"2024-08-19T14:18:13.804721+0000","last_unstale":"2024-08-19T14:27:47.917987+0000","last_undegraded":"2024-08-19T14:27:47.917987+0000","last_fullsized":"2024-08-19T14:27:47.917987+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:27:04.891524+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:27:04.891524+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:28.108665+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"489'9685","reported_seq":9881,"reported_epoch":489,"state":"active+clean","last_fresh":"2024-08-19T14:27:48.764464+0000","last_change":"2024-08-19T14:27:31.857176+0000","last_active":"2024-08-19T14:27:48.764464+0000","last_peered":"2024-08-19T14:27:48.764464+0000","last_clean":"2024-08-19T14:27:48.764464+0000","last_became_active":"2024-08-19T14:27:31.839054+0000","last_became_peered":"2024-08-19T14:27:31.839054+0000","last_unstale":"2024-08-19T14:27:48.764464+0000","last_undegraded":"2024-08-19T14:27:48.764464+0000","last_fullsized":"2024-08-19T14:27:48.764464+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"474'8126","last_scrub_stamp":"2024-08-19T14:26:59.955550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:26:59.955550+0000","objects_scrubbed":2208,"log_size":9685,"log_dups_size":0,"ondisk_log_size":9685,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:09.277471+0000","scrub_duration":558,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42008576,"num_objects":641,"num_object_clones":0,"num_object_copies":1923,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":641,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1567,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"70'139","reported_seq":1068,"reported_epoch":489,"state":"active+clean","last_fresh":"2024-08-19T14:27:47.919782+0000","last_change":"2024-08-19T14:26:57.876424+0000","last_active":"2024-08-19T14:27:47.919782+0000","last_peered":"2024-08-19T14:27:47.919782+0000","last_clean":"2024-08-19T14:27:47.919782+0000","last_became_active":"2024-08-19T14:20:50.742354+0000","last_became_peered":"2024-08-19T14:20:50.742354+0000","last_unstale":"2024-08-19T14:27:47.919782+0000","last_undegraded":"2024-08-19T14:27:47.919782+0000","last_fullsized":"2024-08-19T14:27:47.919782+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:26:57.876316+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:19:59.302931+0000","last_clean_scrub_stamp":"2024-08-19T14:26:57.876316+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:00.325583+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.5","version":"489'9451","reported_seq":9673,"reported_epoch":489,"state":"active+clean","last_fresh":"2024-08-19T14:27:52.127166+0000","last_change":"2024-08-19T14:27:04.288663+0000","last_active":"2024-08-19T14:27:52.127166+0000","last_peered":"2024-08-19T14:27:52.127166+0000","last_clean":"2024-08-19T14:27:52.127166+0000","last_became_active":"2024-08-19T14:25:29.826953+0000","last_became_peered":"2024-08-19T14:25:29.826953+0000","last_unstale":"2024-08-19T14:27:52.127166+0000","last_undegraded":"2024-08-19T14:27:52.127166+0000","last_fullsized":"2024-08-19T14:27:52.127166+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":0,"last_scrub":"474'7988","last_scrub_stamp":"2024-08-19T14:27:04.288536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:27:04.288536+0000","objects_scrubbed":2000,"log_size":9451,"log_dups_size":0,"ondisk_log_size":9451,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:06.060755+0000","scrub_duration":326,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34996224,"num_objects":534,"num_object_clones":0,"num_object_copies":1602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":534,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4082,"num_write_kb":62080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"70'139","reported_seq":313,"reported_epoch":489,"state":"active+clean","last_fresh":"2024-08-19T14:27:47.919124+0000","last_change":"2024-08-19T14:27:27.108572+0000","last_active":"2024-08-19T14:27:47.919124+0000","last_peered":"2024-08-19T14:27:47.919124+0000","last_clean":"2024-08-19T14:27:47.919124+0000","last_became_active":"2024-08-19T14:27:15.696041+0000","last_became_peered":"2024-08-19T14:27:15.696041+0000","last_unstale":"2024-08-19T14:27:47.919124+0000","last_undegraded":"2024-08-19T14:27:47.919124+0000","last_fullsized":"2024-08-19T14:27:47.919124+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:27:27.108504+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:27:27.108504+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:44.170107+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,11],"acting":[0,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"489'9518","reported_seq":9741,"reported_epoch":489,"state":"active+clean","last_fresh":"2024-08-19T14:27:52.146525+0000","last_change":"2024-08-19T14:27:03.375077+0000","last_active":"2024-08-19T14:27:52.146525+0000","last_peered":"2024-08-19T14:27:52.146525+0000","last_clean":"2024-08-19T14:27:52.146525+0000","last_became_active":"2024-08-19T14:25:29.851182+0000","last_became_peered":"2024-08-19T14:25:29.851182+0000","last_unstale":"2024-08-19T14:27:52.146525+0000","last_undegraded":"2024-08-19T14:27:52.146525+0000","last_fullsized":"2024-08-19T14:27:52.146525+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":0,"last_scrub":"474'8002","last_scrub_stamp":"2024-08-19T14:27:03.374935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:27:03.374935+0000","objects_scrubbed":2068,"log_size":9518,"log_dups_size":0,"ondisk_log_size":9518,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:04.049022+0000","scrub_duration":396,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35717120,"num_objects":545,"num_object_clones":0,"num_object_copies":1635,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":545,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4108,"num_write_kb":62528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"70'139","reported_seq":768,"reported_epoch":489,"state":"active+clean","last_fresh":"2024-08-19T14:27:47.926517+0000","last_change":"2024-08-19T14:27:03.234463+0000","last_active":"2024-08-19T14:27:47.926517+0000","last_peered":"2024-08-19T14:27:47.926517+0000","last_clean":"2024-08-19T14:27:47.926517+0000","last_became_active":"2024-08-19T14:20:28.204384+0000","last_became_peered":"2024-08-19T14:20:28.204384+0000","last_unstale":"2024-08-19T14:27:47.926517+0000","last_undegraded":"2024-08-19T14:27:47.926517+0000","last_fullsized":"2024-08-19T14:27:47.926517+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:27:03.234352+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:21:46.057624+0000","last_clean_scrub_stamp":"2024-08-19T14:27:03.234352+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:26.281527+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"489'9639","reported_seq":9832,"reported_epoch":489,"state":"active+clean","last_fresh":"2024-08-19T14:27:52.051692+0000","last_change":"2024-08-19T14:26:56.652316+0000","last_active":"2024-08-19T14:27:52.051692+0000","last_peered":"2024-08-19T14:27:52.051692+0000","last_clean":"2024-08-19T14:27:52.051692+0000","last_became_active":"2024-08-19T14:25:29.831300+0000","last_became_peered":"2024-08-19T14:25:29.831300+0000","last_unstale":"2024-08-19T14:27:52.051692+0000","last_undegraded":"2024-08-19T14:27:52.051692+0000","last_fullsized":"2024-08-19T14:27:52.051692+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'7842","last_scrub_stamp":"2024-08-19T14:26:56.652182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:26:56.652182+0000","objects_scrubbed":2335,"log_size":9639,"log_dups_size":0,"ondisk_log_size":9639,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:21.680001+0000","scrub_duration":607,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34799616,"num_objects":531,"num_object_clones":0,"num_object_copies":1593,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1804,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,8],"acting":[0,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"70'139","reported_seq":313,"reported_epoch":489,"state":"active+clean","last_fresh":"2024-08-19T14:27:47.916375+0000","last_change":"2024-08-19T14:27:26.969984+0000","last_active":"2024-08-19T14:27:47.916375+0000","last_peered":"2024-08-19T14:27:47.916375+0000","last_clean":"2024-08-19T14:27:47.916375+0000","last_became_active":"2024-08-19T14:27:15.697224+0000","last_became_peered":"2024-08-19T14:27:15.697224+0000","last_unstale":"2024-08-19T14:27:47.916375+0000","last_undegraded":"2024-08-19T14:27:47.916375+0000","last_fullsized":"2024-08-19T14:27:47.916375+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:27:26.969901+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:27.848870+0000","last_clean_scrub_stamp":"2024-08-19T14:27:26.969901+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:52.544619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"5.1","version":"489'11642","reported_seq":11926,"reported_epoch":489,"state":"active+clean","last_fresh":"2024-08-19T14:27:47.992863+0000","last_change":"2024-08-19T14:27:14.776017+0000","last_active":"2024-08-19T14:27:47.992863+0000","last_peered":"2024-08-19T14:27:47.992863+0000","last_clean":"2024-08-19T14:27:47.992863+0000","last_became_active":"2024-08-19T14:25:29.820527+0000","last_became_peered":"2024-08-19T14:25:29.820527+0000","last_unstale":"2024-08-19T14:27:47.992863+0000","last_undegraded":"2024-08-19T14:27:47.992863+0000","last_fullsized":"2024-08-19T14:27:47.992863+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":0,"last_scrub":"477'10484","last_scrub_stamp":"2024-08-19T14:27:14.775877+0000","last_deep_scrub":"477'10484","last_deep_scrub_stamp":"2024-08-19T14:27:14.775877+0000","last_clean_scrub_stamp":"2024-08-19T14:27:14.775877+0000","objects_scrubbed":1945,"log_size":11642,"log_dups_size":0,"ondisk_log_size":11642,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":10,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:22.349281+0000","scrub_duration":9182,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47316992,"num_objects":722,"num_object_clones":0,"num_object_copies":2166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":722,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4046,"num_write_kb":61248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"70'139","reported_seq":942,"reported_epoch":489,"state":"active+clean","last_fresh":"2024-08-19T14:27:47.915301+0000","last_change":"2024-08-19T14:26:56.775790+0000","last_active":"2024-08-19T14:27:47.915301+0000","last_peered":"2024-08-19T14:27:47.915301+0000","last_clean":"2024-08-19T14:27:47.915301+0000","last_became_active":"2024-08-19T14:18:26.881435+0000","last_became_peered":"2024-08-19T14:18:26.881435+0000","last_unstale":"2024-08-19T14:27:47.915301+0000","last_undegraded":"2024-08-19T14:27:47.915301+0000","last_fullsized":"2024-08-19T14:27:47.915301+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:26:56.775676+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:26:56.775676+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:04.024892+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,4],"acting":[11,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.0","version":"489'9724","reported_seq":9923,"reported_epoch":489,"state":"active+clean","last_fresh":"2024-08-19T14:27:48.671102+0000","last_change":"2024-08-19T14:27:34.897610+0000","last_active":"2024-08-19T14:27:48.671102+0000","last_peered":"2024-08-19T14:27:48.671102+0000","last_clean":"2024-08-19T14:27:48.671102+0000","last_became_active":"2024-08-19T14:25:29.850301+0000","last_became_peered":"2024-08-19T14:25:29.850301+0000","last_unstale":"2024-08-19T14:27:48.671102+0000","last_undegraded":"2024-08-19T14:27:48.671102+0000","last_fullsized":"2024-08-19T14:27:48.671102+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'5592","last_scrub_stamp":"2024-08-19T14:25:32.103399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:25:32.103399+0000","objects_scrubbed":99,"log_size":9724,"log_dups_size":0,"ondisk_log_size":9724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":317,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44433408,"num_objects":678,"num_object_clones":0,"num_object_copies":2034,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":678,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4132,"num_write_kb":62528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"70'139","reported_seq":1167,"reported_epoch":489,"state":"active+clean","last_fresh":"2024-08-19T14:27:47.914858+0000","last_change":"2024-08-19T14:26:55.764218+0000","last_active":"2024-08-19T14:27:47.914858+0000","last_peered":"2024-08-19T14:27:47.914858+0000","last_clean":"2024-08-19T14:27:47.914858+0000","last_became_active":"2024-08-19T14:18:13.806013+0000","last_became_peered":"2024-08-19T14:18:13.806013+0000","last_unstale":"2024-08-19T14:27:47.914858+0000","last_undegraded":"2024-08-19T14:27:47.914858+0000","last_fullsized":"2024-08-19T14:27:47.914858+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:26:55.764116+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:26:55.764116+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:24.552401+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.e","version":"489'13777","reported_seq":13986,"reported_epoch":489,"state":"active+clean","last_fresh":"2024-08-19T14:27:49.922195+0000","last_change":"2024-08-19T14:27:15.741425+0000","last_active":"2024-08-19T14:27:49.922195+0000","last_peered":"2024-08-19T14:27:49.922195+0000","last_clean":"2024-08-19T14:27:49.922195+0000","last_became_active":"2024-08-19T14:27:15.697672+0000","last_became_peered":"2024-08-19T14:27:15.697672+0000","last_unstale":"2024-08-19T14:27:49.922195+0000","last_undegraded":"2024-08-19T14:27:49.922195+0000","last_fullsized":"2024-08-19T14:27:49.922195+0000","mapping_epoch":477,"log_start":"433'3700","ondisk_log_start":"433'3700","created":419,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":0,"last_scrub":"474'10948","last_scrub_stamp":"2024-08-19T14:27:04.056561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:27:04.056561+0000","objects_scrubbed":4147,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:24.468871+0000","scrub_duration":1025,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84344832,"num_objects":1287,"num_object_clones":0,"num_object_copies":3861,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1287,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13777,"num_write_kb":482048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.a","version":"70'139","reported_seq":313,"reported_epoch":489,"state":"active+clean","last_fresh":"2024-08-19T14:27:47.916233+0000","last_change":"2024-08-19T14:27:25.888737+0000","last_active":"2024-08-19T14:27:47.916233+0000","last_peered":"2024-08-19T14:27:47.916233+0000","last_clean":"2024-08-19T14:27:47.916233+0000","last_became_active":"2024-08-19T14:27:15.698792+0000","last_became_peered":"2024-08-19T14:27:15.698792+0000","last_unstale":"2024-08-19T14:27:47.916233+0000","last_undegraded":"2024-08-19T14:27:47.916233+0000","last_fullsized":"2024-08-19T14:27:47.916233+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:27:25.888493+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:28.864870+0000","last_clean_scrub_stamp":"2024-08-19T14:27:25.888493+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:27.170057+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,0],"acting":[4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"489'16687","reported_seq":16986,"reported_epoch":489,"state":"active+clean","last_fresh":"2024-08-19T14:27:50.999373+0000","last_change":"2024-08-19T14:27:30.378381+0000","last_active":"2024-08-19T14:27:50.999373+0000","last_peered":"2024-08-19T14:27:50.999373+0000","last_clean":"2024-08-19T14:27:50.999373+0000","last_became_active":"2024-08-19T14:24:44.592098+0000","last_became_peered":"2024-08-19T14:24:44.592098+0000","last_unstale":"2024-08-19T14:27:50.999373+0000","last_undegraded":"2024-08-19T14:27:50.999373+0000","last_fullsized":"2024-08-19T14:27:50.999373+0000","mapping_epoch":419,"log_start":"449'6600","ondisk_log_start":"449'6600","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"483'15174","last_scrub_stamp":"2024-08-19T14:27:30.378045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:27:30.378045+0000","objects_scrubbed":2739,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:35.644617+0000","scrub_duration":1017,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78774296,"num_objects":1203,"num_object_clones":0,"num_object_copies":3609,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1203,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14139,"num_write_kb":490881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7],"acting":[11,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.9","version":"70'139","reported_seq":1243,"reported_epoch":489,"state":"active+clean","last_fresh":"2024-08-19T14:27:47.922155+0000","last_change":"2024-08-19T14:27:05.872064+0000","last_active":"2024-08-19T14:27:47.922155+0000","last_peered":"2024-08-19T14:27:47.922155+0000","last_clean":"2024-08-19T14:27:47.922155+0000","last_became_active":"2024-08-19T14:18:26.887657+0000","last_became_peered":"2024-08-19T14:18:26.887657+0000","last_unstale":"2024-08-19T14:27:47.922155+0000","last_undegraded":"2024-08-19T14:27:47.922155+0000","last_fullsized":"2024-08-19T14:27:47.922155+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:27:05.871950+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:27:05.871950+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:32.663051+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"489'13761","reported_seq":13960,"reported_epoch":489,"state":"active+clean","last_fresh":"2024-08-19T14:27:48.656650+0000","last_change":"2024-08-19T14:27:11.671223+0000","last_active":"2024-08-19T14:27:48.656650+0000","last_peered":"2024-08-19T14:27:48.656650+0000","last_clean":"2024-08-19T14:27:48.656650+0000","last_became_active":"2024-08-19T14:24:44.595538+0000","last_became_peered":"2024-08-19T14:24:44.595538+0000","last_unstale":"2024-08-19T14:27:48.656650+0000","last_undegraded":"2024-08-19T14:27:48.656650+0000","last_fullsized":"2024-08-19T14:27:48.656650+0000","mapping_epoch":419,"log_start":"432'3700","ondisk_log_start":"432'3700","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'11344","last_scrub_stamp":"2024-08-19T14:27:11.670737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:27:11.670737+0000","objects_scrubbed":3780,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:29.589644+0000","scrub_duration":775,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88670208,"num_objects":1353,"num_object_clones":0,"num_object_copies":4059,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1353,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13761,"num_write_kb":483648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"70'139","reported_seq":299,"reported_epoch":489,"state":"active+clean","last_fresh":"2024-08-19T14:27:47.926337+0000","last_change":"2024-08-19T14:27:32.221695+0000","last_active":"2024-08-19T14:27:47.926337+0000","last_peered":"2024-08-19T14:27:47.926337+0000","last_clean":"2024-08-19T14:27:47.926337+0000","last_became_active":"2024-08-19T14:27:31.834018+0000","last_became_peered":"2024-08-19T14:27:31.834018+0000","last_unstale":"2024-08-19T14:27:47.926337+0000","last_undegraded":"2024-08-19T14:27:47.926337+0000","last_fullsized":"2024-08-19T14:27:47.926337+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:27:32.221578+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:27:32.221578+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:59.258809+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,8],"acting":[3,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"489'13419","reported_seq":13631,"reported_epoch":489,"state":"active+clean","last_fresh":"2024-08-19T14:27:47.983385+0000","last_change":"2024-08-19T14:27:31.957791+0000","last_active":"2024-08-19T14:27:47.983385+0000","last_peered":"2024-08-19T14:27:47.983385+0000","last_clean":"2024-08-19T14:27:47.983385+0000","last_became_active":"2024-08-19T14:27:31.855047+0000","last_became_peered":"2024-08-19T14:27:31.855047+0000","last_unstale":"2024-08-19T14:27:47.983385+0000","last_undegraded":"2024-08-19T14:27:47.983385+0000","last_fullsized":"2024-08-19T14:27:47.983385+0000","mapping_epoch":484,"log_start":"431'3400","ondisk_log_start":"431'3400","created":419,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'10913","last_scrub_stamp":"2024-08-19T14:27:07.305888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:27:07.305888+0000","objects_scrubbed":3846,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:17.348716+0000","scrub_duration":914,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86835200,"num_objects":1325,"num_object_clones":0,"num_object_copies":3975,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1325,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13419,"num_write_kb":471808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,11],"acting":[4,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"70'139","reported_seq":1142,"reported_epoch":489,"state":"active+clean","last_fresh":"2024-08-19T14:27:47.914705+0000","last_change":"2024-08-19T14:27:05.891327+0000","last_active":"2024-08-19T14:27:47.914705+0000","last_peered":"2024-08-19T14:27:47.914705+0000","last_clean":"2024-08-19T14:27:47.914705+0000","last_became_active":"2024-08-19T14:18:26.875179+0000","last_became_peered":"2024-08-19T14:18:26.875179+0000","last_unstale":"2024-08-19T14:27:47.914705+0000","last_undegraded":"2024-08-19T14:27:47.914705+0000","last_fullsized":"2024-08-19T14:27:47.914705+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:27:05.891237+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:27:05.891237+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:29.101160+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.a","version":"489'16100","reported_seq":16388,"reported_epoch":489,"state":"active+clean","last_fresh":"2024-08-19T14:27:49.998157+0000","last_change":"2024-08-19T14:27:24.956372+0000","last_active":"2024-08-19T14:27:49.998157+0000","last_peered":"2024-08-19T14:27:49.998157+0000","last_clean":"2024-08-19T14:27:49.998157+0000","last_became_active":"2024-08-19T14:24:44.592024+0000","last_became_peered":"2024-08-19T14:24:44.592024+0000","last_unstale":"2024-08-19T14:27:49.998157+0000","last_undegraded":"2024-08-19T14:27:49.998157+0000","last_fullsized":"2024-08-19T14:27:49.998157+0000","mapping_epoch":419,"log_start":"446'6002","ondisk_log_start":"446'6002","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"481'14256","last_scrub_stamp":"2024-08-19T14:27:24.956176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:27:24.956176+0000","objects_scrubbed":3085,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:32.574796+0000","scrub_duration":1048,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79495168,"num_objects":1213,"num_object_clones":0,"num_object_copies":3639,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13839,"num_write_kb":481664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,11],"acting":[8,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.e","version":"70'139","reported_seq":796,"reported_epoch":489,"state":"active+clean","last_fresh":"2024-08-19T14:27:47.916144+0000","last_change":"2024-08-19T14:27:03.631853+0000","last_active":"2024-08-19T14:27:47.916144+0000","last_peered":"2024-08-19T14:27:47.916144+0000","last_clean":"2024-08-19T14:27:47.916144+0000","last_became_active":"2024-08-19T14:18:26.884105+0000","last_became_peered":"2024-08-19T14:18:26.884105+0000","last_unstale":"2024-08-19T14:27:47.916144+0000","last_undegraded":"2024-08-19T14:27:47.916144+0000","last_fullsized":"2024-08-19T14:27:47.916144+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:27:03.631729+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:20:15.321945+0000","last_clean_scrub_stamp":"2024-08-19T14:27:03.631729+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:24.374451+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.b","version":"489'14175","reported_seq":14386,"reported_epoch":489,"state":"active+clean","last_fresh":"2024-08-19T14:27:52.168121+0000","last_change":"2024-08-19T14:27:15.727136+0000","last_active":"2024-08-19T14:27:52.168121+0000","last_peered":"2024-08-19T14:27:52.168121+0000","last_clean":"2024-08-19T14:27:52.168121+0000","last_became_active":"2024-08-19T14:27:15.695445+0000","last_became_peered":"2024-08-19T14:27:15.695445+0000","last_unstale":"2024-08-19T14:27:52.168121+0000","last_undegraded":"2024-08-19T14:27:52.168121+0000","last_fullsized":"2024-08-19T14:27:52.168121+0000","mapping_epoch":477,"log_start":"433'4100","ondisk_log_start":"433'4100","created":419,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":0,"last_scrub":"474'11027","last_scrub_stamp":"2024-08-19T14:27:01.445200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:27:01.445200+0000","objects_scrubbed":4362,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:23.766180+0000","scrub_duration":1433,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":74907648,"num_objects":1143,"num_object_clones":0,"num_object_copies":3429,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1143,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14175,"num_write_kb":490176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,5],"acting":[0,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"70'139","reported_seq":564,"reported_epoch":489,"state":"active+clean","last_fresh":"2024-08-19T14:27:47.916484+0000","last_change":"2024-08-19T14:27:32.825905+0000","last_active":"2024-08-19T14:27:47.916484+0000","last_peered":"2024-08-19T14:27:47.916484+0000","last_clean":"2024-08-19T14:27:47.916484+0000","last_became_active":"2024-08-19T14:27:31.837754+0000","last_became_peered":"2024-08-19T14:27:31.837754+0000","last_unstale":"2024-08-19T14:27:47.916484+0000","last_undegraded":"2024-08-19T14:27:47.916484+0000","last_fullsized":"2024-08-19T14:27:47.916484+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:27:32.825795+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:27:32.825795+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:49.111004+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"489'11369","reported_seq":11646,"reported_epoch":489,"state":"active+clean","last_fresh":"2024-08-19T14:27:51.012078+0000","last_change":"2024-08-19T14:27:16.182678+0000","last_active":"2024-08-19T14:27:51.012078+0000","last_peered":"2024-08-19T14:27:51.012078+0000","last_clean":"2024-08-19T14:27:51.012078+0000","last_became_active":"2024-08-19T14:25:29.849790+0000","last_became_peered":"2024-08-19T14:25:29.849790+0000","last_unstale":"2024-08-19T14:27:51.012078+0000","last_undegraded":"2024-08-19T14:27:51.012078+0000","last_fullsized":"2024-08-19T14:27:51.012078+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":0,"last_scrub":"478'10163","last_scrub_stamp":"2024-08-19T14:27:16.182488+0000","last_deep_scrub":"478'10163","last_deep_scrub_stamp":"2024-08-19T14:27:16.182488+0000","last_clean_scrub_stamp":"2024-08-19T14:27:16.182488+0000","objects_scrubbed":1849,"log_size":11369,"log_dups_size":0,"ondisk_log_size":11369,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":9,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:43.664659+0000","scrub_duration":8297,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38600704,"num_objects":589,"num_object_clones":0,"num_object_copies":1767,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4076,"num_write_kb":61440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.c","version":"70'139","reported_seq":901,"reported_epoch":489,"state":"active+clean","last_fresh":"2024-08-19T14:27:47.926426+0000","last_change":"2024-08-19T14:27:00.232219+0000","last_active":"2024-08-19T14:27:47.926426+0000","last_peered":"2024-08-19T14:27:47.926426+0000","last_clean":"2024-08-19T14:27:47.926426+0000","last_became_active":"2024-08-19T14:18:26.872942+0000","last_became_peered":"2024-08-19T14:18:26.872942+0000","last_unstale":"2024-08-19T14:27:47.926426+0000","last_undegraded":"2024-08-19T14:27:47.926426+0000","last_fullsized":"2024-08-19T14:27:47.926426+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:27:00.232092+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:22:52.164586+0000","last_clean_scrub_stamp":"2024-08-19T14:27:00.232092+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:04.916962+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,9],"acting":[3,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"489'9607","reported_seq":9821,"reported_epoch":489,"state":"active+clean","last_fresh":"2024-08-19T14:27:48.796392+0000","last_change":"2024-08-19T14:27:27.280756+0000","last_active":"2024-08-19T14:27:48.796392+0000","last_peered":"2024-08-19T14:27:48.796392+0000","last_clean":"2024-08-19T14:27:48.796392+0000","last_became_active":"2024-08-19T14:25:29.821239+0000","last_became_peered":"2024-08-19T14:25:29.821239+0000","last_unstale":"2024-08-19T14:27:48.796392+0000","last_undegraded":"2024-08-19T14:27:48.796392+0000","last_fullsized":"2024-08-19T14:27:48.796392+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":0,"last_scrub":"482'8776","last_scrub_stamp":"2024-08-19T14:27:27.280495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:27:27.280495+0000","objects_scrubbed":1490,"log_size":9607,"log_dups_size":0,"ondisk_log_size":9607,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:45.484395+0000","scrub_duration":199,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43122688,"num_objects":658,"num_object_clones":0,"num_object_copies":1974,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":658,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4077,"num_write_kb":60544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,9],"acting":[7,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"70'139","reported_seq":816,"reported_epoch":489,"state":"active+clean","last_fresh":"2024-08-19T14:27:47.922654+0000","last_change":"2024-08-19T14:27:01.958255+0000","last_active":"2024-08-19T14:27:47.922654+0000","last_peered":"2024-08-19T14:27:47.922654+0000","last_clean":"2024-08-19T14:27:47.922654+0000","last_became_active":"2024-08-19T14:18:26.882002+0000","last_became_peered":"2024-08-19T14:18:26.882002+0000","last_unstale":"2024-08-19T14:27:47.922654+0000","last_undegraded":"2024-08-19T14:27:47.922654+0000","last_fullsized":"2024-08-19T14:27:47.922654+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:27:01.958142+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:24:50.276408+0000","last_clean_scrub_stamp":"2024-08-19T14:27:01.958142+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:07.061913+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"489'9704","reported_seq":18527,"reported_epoch":489,"state":"active+clean","last_fresh":"2024-08-19T14:27:48.820797+0000","last_change":"2024-08-19T14:27:31.860278+0000","last_active":"2024-08-19T14:27:48.820797+0000","last_peered":"2024-08-19T14:27:48.820797+0000","last_clean":"2024-08-19T14:27:48.820797+0000","last_became_active":"2024-08-19T14:27:31.835536+0000","last_became_peered":"2024-08-19T14:27:31.835536+0000","last_unstale":"2024-08-19T14:27:48.820797+0000","last_undegraded":"2024-08-19T14:27:48.820797+0000","last_fullsized":"2024-08-19T14:27:48.820797+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":439,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":5,"last_scrub":"473'8001","last_scrub_stamp":"2024-08-19T14:26:55.980803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:26:55.980803+0000","objects_scrubbed":2365,"log_size":9704,"log_dups_size":0,"ondisk_log_size":9704,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:21.890801+0000","scrub_duration":563,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42336256,"num_objects":646,"num_object_clones":0,"num_object_copies":1938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1719,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.12","version":"70'139","reported_seq":603,"reported_epoch":489,"state":"active+clean","last_fresh":"2024-08-19T14:27:47.917761+0000","last_change":"2024-08-19T14:27:31.855657+0000","last_active":"2024-08-19T14:27:47.917761+0000","last_peered":"2024-08-19T14:27:47.917761+0000","last_clean":"2024-08-19T14:27:47.917761+0000","last_became_active":"2024-08-19T14:27:31.838968+0000","last_became_peered":"2024-08-19T14:27:31.838968+0000","last_unstale":"2024-08-19T14:27:47.917761+0000","last_undegraded":"2024-08-19T14:27:47.917761+0000","last_fullsized":"2024-08-19T14:27:47.917761+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:27:31.855524+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:27:31.855524+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:32.492630+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"489'9699","reported_seq":20800,"reported_epoch":489,"state":"active+clean","last_fresh":"2024-08-19T14:27:48.845674+0000","last_change":"2024-08-19T14:26:59.683109+0000","last_active":"2024-08-19T14:27:48.845674+0000","last_peered":"2024-08-19T14:27:48.845674+0000","last_clean":"2024-08-19T14:27:48.845674+0000","last_became_active":"2024-08-19T14:26:36.445968+0000","last_became_peered":"2024-08-19T14:26:36.445968+0000","last_unstale":"2024-08-19T14:27:48.845674+0000","last_undegraded":"2024-08-19T14:27:48.845674+0000","last_fullsized":"2024-08-19T14:27:48.845674+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":439,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":5,"last_scrub":"474'8148","last_scrub_stamp":"2024-08-19T14:26:59.682873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:26:59.682873+0000","objects_scrubbed":2199,"log_size":9699,"log_dups_size":0,"ondisk_log_size":9699,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:25.733561+0000","scrub_duration":440,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42205184,"num_objects":644,"num_object_clones":0,"num_object_copies":1932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1555,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8,11],"acting":[3,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.13","version":"70'139","reported_seq":1127,"reported_epoch":489,"state":"active+clean","last_fresh":"2024-08-19T14:27:47.917891+0000","last_change":"2024-08-19T14:27:23.812536+0000","last_active":"2024-08-19T14:27:47.917891+0000","last_peered":"2024-08-19T14:27:47.917891+0000","last_clean":"2024-08-19T14:27:47.917891+0000","last_became_active":"2024-08-19T14:18:18.698334+0000","last_became_peered":"2024-08-19T14:18:18.698334+0000","last_unstale":"2024-08-19T14:27:47.917891+0000","last_undegraded":"2024-08-19T14:27:47.917891+0000","last_fullsized":"2024-08-19T14:27:47.917891+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:27:23.812426+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:27:23.812426+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:30.068468+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.14","version":"489'9427","reported_seq":17847,"reported_epoch":489,"state":"active+clean","last_fresh":"2024-08-19T14:27:48.836955+0000","last_change":"2024-08-19T14:26:57.705333+0000","last_active":"2024-08-19T14:27:48.836955+0000","last_peered":"2024-08-19T14:27:48.836955+0000","last_clean":"2024-08-19T14:27:48.836955+0000","last_became_active":"2024-08-19T14:26:34.444635+0000","last_became_peered":"2024-08-19T14:26:34.444635+0000","last_unstale":"2024-08-19T14:27:48.836955+0000","last_undegraded":"2024-08-19T14:27:48.836955+0000","last_fullsized":"2024-08-19T14:27:48.836955+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":439,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":5,"last_scrub":"473'7847","last_scrub_stamp":"2024-08-19T14:26:57.705140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:26:57.705140+0000","objects_scrubbed":2241,"log_size":9427,"log_dups_size":0,"ondisk_log_size":9427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:24.476329+0000","scrub_duration":460,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42598400,"num_objects":650,"num_object_clones":0,"num_object_copies":1950,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":650,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1591,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8,4],"acting":[3,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.10","version":"70'139","reported_seq":765,"reported_epoch":489,"state":"active+clean","last_fresh":"2024-08-19T14:27:47.919299+0000","last_change":"2024-08-19T14:27:32.129561+0000","last_active":"2024-08-19T14:27:47.919299+0000","last_peered":"2024-08-19T14:27:47.919299+0000","last_clean":"2024-08-19T14:27:47.919299+0000","last_became_active":"2024-08-19T14:27:31.839611+0000","last_became_peered":"2024-08-19T14:27:31.839611+0000","last_unstale":"2024-08-19T14:27:47.919299+0000","last_undegraded":"2024-08-19T14:27:47.919299+0000","last_fullsized":"2024-08-19T14:27:47.919299+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:27:32.129449+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:27:32.129449+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:56.975541+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"489'9400","reported_seq":17899,"reported_epoch":489,"state":"active+clean","last_fresh":"2024-08-19T14:27:48.697697+0000","last_change":"2024-08-19T14:27:31.867569+0000","last_active":"2024-08-19T14:27:48.697697+0000","last_peered":"2024-08-19T14:27:48.697697+0000","last_clean":"2024-08-19T14:27:48.697697+0000","last_became_active":"2024-08-19T14:27:31.839752+0000","last_became_peered":"2024-08-19T14:27:31.839752+0000","last_unstale":"2024-08-19T14:27:48.697697+0000","last_undegraded":"2024-08-19T14:27:48.697697+0000","last_fullsized":"2024-08-19T14:27:48.697697+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":439,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":5,"last_scrub":"473'7772","last_scrub_stamp":"2024-08-19T14:26:56.903963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:26:56.903963+0000","objects_scrubbed":2299,"log_size":9400,"log_dups_size":0,"ondisk_log_size":9400,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:20.150424+0000","scrub_duration":532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43450368,"num_objects":663,"num_object_clones":0,"num_object_copies":1989,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1636,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,8],"acting":[1,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.11","version":"70'139","reported_seq":787,"reported_epoch":489,"state":"active+clean","last_fresh":"2024-08-19T14:27:47.919341+0000","last_change":"2024-08-19T14:26:59.004413+0000","last_active":"2024-08-19T14:27:47.919341+0000","last_peered":"2024-08-19T14:27:47.919341+0000","last_clean":"2024-08-19T14:27:47.919341+0000","last_became_active":"2024-08-19T14:18:18.697894+0000","last_became_peered":"2024-08-19T14:18:18.697894+0000","last_unstale":"2024-08-19T14:27:47.919341+0000","last_undegraded":"2024-08-19T14:27:47.919341+0000","last_fullsized":"2024-08-19T14:27:47.919341+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:26:59.004304+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:24:50.547428+0000","last_clean_scrub_stamp":"2024-08-19T14:26:59.004304+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:19.347065+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"489'9575","reported_seq":18082,"reported_epoch":489,"state":"active+clean","last_fresh":"2024-08-19T14:27:49.980808+0000","last_change":"2024-08-19T14:26:55.482856+0000","last_active":"2024-08-19T14:27:49.980808+0000","last_peered":"2024-08-19T14:27:49.980808+0000","last_clean":"2024-08-19T14:27:49.980808+0000","last_became_active":"2024-08-19T14:26:16.340194+0000","last_became_peered":"2024-08-19T14:26:16.340194+0000","last_unstale":"2024-08-19T14:27:49.980808+0000","last_undegraded":"2024-08-19T14:27:49.980808+0000","last_fullsized":"2024-08-19T14:27:49.980808+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":439,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":5,"last_scrub":"472'7834","last_scrub_stamp":"2024-08-19T14:26:55.482598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:26:55.482598+0000","objects_scrubbed":2372,"log_size":9575,"log_dups_size":0,"ondisk_log_size":9575,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:20.501632+0000","scrub_duration":541,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40763392,"num_objects":622,"num_object_clones":0,"num_object_copies":1866,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1750,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,3],"acting":[8,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"70'139","reported_seq":755,"reported_epoch":489,"state":"active+clean","last_fresh":"2024-08-19T14:27:47.918407+0000","last_change":"2024-08-19T14:27:08.949559+0000","last_active":"2024-08-19T14:27:47.918407+0000","last_peered":"2024-08-19T14:27:47.918407+0000","last_clean":"2024-08-19T14:27:47.918407+0000","last_became_active":"2024-08-19T14:20:28.204724+0000","last_became_peered":"2024-08-19T14:20:28.204724+0000","last_unstale":"2024-08-19T14:27:47.918407+0000","last_undegraded":"2024-08-19T14:27:47.918407+0000","last_fullsized":"2024-08-19T14:27:47.918407+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:27:08.949445+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:27:08.949445+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:28.588653+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"489'9491","reported_seq":13359,"reported_epoch":489,"state":"active+clean","last_fresh":"2024-08-19T14:27:49.964992+0000","last_change":"2024-08-19T14:26:56.478334+0000","last_active":"2024-08-19T14:27:49.964992+0000","last_peered":"2024-08-19T14:27:49.964992+0000","last_clean":"2024-08-19T14:27:49.964992+0000","last_became_active":"2024-08-19T14:26:07.295750+0000","last_became_peered":"2024-08-19T14:26:07.295750+0000","last_unstale":"2024-08-19T14:27:49.964992+0000","last_undegraded":"2024-08-19T14:27:49.964992+0000","last_fullsized":"2024-08-19T14:27:49.964992+0000","mapping_epoch":454,"log_start":"0'0","ondisk_log_start":"0'0","created":439,"last_epoch_clean":455,"parent":"0.0","parent_split_bits":5,"last_scrub":"473'7804","last_scrub_stamp":"2024-08-19T14:26:56.478096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:26:56.478096+0000","objects_scrubbed":2308,"log_size":9491,"log_dups_size":0,"ondisk_log_size":9491,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:19.761013+0000","scrub_duration":545,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40108032,"num_objects":612,"num_object_clones":0,"num_object_copies":1836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":612,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1696,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,5],"acting":[8,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.17","version":"70'139","reported_seq":883,"reported_epoch":489,"state":"active+clean","last_fresh":"2024-08-19T14:27:47.916304+0000","last_change":"2024-08-19T14:27:25.944264+0000","last_active":"2024-08-19T14:27:47.916304+0000","last_peered":"2024-08-19T14:27:47.916304+0000","last_clean":"2024-08-19T14:27:47.916304+0000","last_became_active":"2024-08-19T14:18:31.901556+0000","last_became_peered":"2024-08-19T14:18:31.901556+0000","last_unstale":"2024-08-19T14:27:47.916304+0000","last_undegraded":"2024-08-19T14:27:47.916304+0000","last_fullsized":"2024-08-19T14:27:47.916304+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:27:25.944188+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:27:25.944188+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:26.208810+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,0],"acting":[8,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"5.10","version":"489'9652","reported_seq":18240,"reported_epoch":489,"state":"active+clean","last_fresh":"2024-08-19T14:27:48.786291+0000","last_change":"2024-08-19T14:26:59.390121+0000","last_active":"2024-08-19T14:27:48.786291+0000","last_peered":"2024-08-19T14:27:48.786291+0000","last_clean":"2024-08-19T14:27:48.786291+0000","last_became_active":"2024-08-19T14:26:35.443770+0000","last_became_peered":"2024-08-19T14:26:35.443770+0000","last_unstale":"2024-08-19T14:27:48.786291+0000","last_undegraded":"2024-08-19T14:27:48.786291+0000","last_fullsized":"2024-08-19T14:27:48.786291+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":439,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":5,"last_scrub":"473'8115","last_scrub_stamp":"2024-08-19T14:26:59.389786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:26:59.389786+0000","objects_scrubbed":2208,"log_size":9652,"log_dups_size":0,"ondisk_log_size":9652,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:27.336579+0000","scrub_duration":521,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43450368,"num_objects":663,"num_object_clones":0,"num_object_copies":1989,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1545,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,0],"acting":[7,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"70'139","reported_seq":893,"reported_epoch":489,"state":"active+clean","last_fresh":"2024-08-19T14:27:47.919078+0000","last_change":"2024-08-19T14:27:26.038394+0000","last_active":"2024-08-19T14:27:47.919078+0000","last_peered":"2024-08-19T14:27:47.919078+0000","last_clean":"2024-08-19T14:27:47.919078+0000","last_became_active":"2024-08-19T14:18:18.703926+0000","last_became_peered":"2024-08-19T14:18:18.703926+0000","last_unstale":"2024-08-19T14:27:47.919078+0000","last_undegraded":"2024-08-19T14:27:47.919078+0000","last_fullsized":"2024-08-19T14:27:47.919078+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:27:26.038293+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:27:26.038293+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:36.114804+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,1],"acting":[7,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"489'9701","reported_seq":15619,"reported_epoch":489,"state":"active+clean","last_fresh":"2024-08-19T14:27:47.983572+0000","last_change":"2024-08-19T14:27:00.100949+0000","last_active":"2024-08-19T14:27:47.983572+0000","last_peered":"2024-08-19T14:27:47.983572+0000","last_clean":"2024-08-19T14:27:47.983572+0000","last_became_active":"2024-08-19T14:26:00.207101+0000","last_became_peered":"2024-08-19T14:26:00.207101+0000","last_unstale":"2024-08-19T14:27:47.983572+0000","last_undegraded":"2024-08-19T14:27:47.983572+0000","last_fullsized":"2024-08-19T14:27:47.983572+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":439,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":5,"last_scrub":"474'8221","last_scrub_stamp":"2024-08-19T14:27:00.100580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:27:00.100580+0000","objects_scrubbed":2198,"log_size":9701,"log_dups_size":0,"ondisk_log_size":9701,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:15.940877+0000","scrub_duration":491,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46792704,"num_objects":714,"num_object_clones":0,"num_object_copies":2142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":714,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1484,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7],"acting":[4,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.15","version":"70'139","reported_seq":771,"reported_epoch":489,"state":"active+clean","last_fresh":"2024-08-19T14:27:47.922557+0000","last_change":"2024-08-19T14:26:56.872678+0000","last_active":"2024-08-19T14:27:47.922557+0000","last_peered":"2024-08-19T14:27:47.922557+0000","last_clean":"2024-08-19T14:27:47.922557+0000","last_became_active":"2024-08-19T14:18:31.912496+0000","last_became_peered":"2024-08-19T14:18:31.912496+0000","last_unstale":"2024-08-19T14:27:47.922557+0000","last_undegraded":"2024-08-19T14:27:47.922557+0000","last_fullsized":"2024-08-19T14:27:47.922557+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:26:56.872544+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:20:09.180923+0000","last_clean_scrub_stamp":"2024-08-19T14:26:56.872544+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:08.677502+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"70'139","reported_seq":757,"reported_epoch":489,"state":"active+clean","last_fresh":"2024-08-19T14:27:47.916042+0000","last_change":"2024-08-19T14:27:00.651838+0000","last_active":"2024-08-19T14:27:47.916042+0000","last_peered":"2024-08-19T14:27:47.916042+0000","last_clean":"2024-08-19T14:27:47.916042+0000","last_became_active":"2024-08-19T14:18:31.901229+0000","last_became_peered":"2024-08-19T14:18:31.901229+0000","last_unstale":"2024-08-19T14:27:47.916042+0000","last_undegraded":"2024-08-19T14:27:47.916042+0000","last_fullsized":"2024-08-19T14:27:47.916042+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:27:00.651729+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:27:00.651729+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:30.091916+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1b","version":"70'139","reported_seq":1109,"reported_epoch":489,"state":"active+clean","last_fresh":"2024-08-19T14:27:47.920881+0000","last_change":"2024-08-19T14:26:59.962358+0000","last_active":"2024-08-19T14:27:47.920881+0000","last_peered":"2024-08-19T14:27:47.920881+0000","last_clean":"2024-08-19T14:27:47.920881+0000","last_became_active":"2024-08-19T14:18:31.901152+0000","last_became_peered":"2024-08-19T14:18:31.901152+0000","last_unstale":"2024-08-19T14:27:47.920881+0000","last_undegraded":"2024-08-19T14:27:47.920881+0000","last_fullsized":"2024-08-19T14:27:47.920881+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:26:59.962262+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:26:59.962262+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:08.672325+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.18","version":"70'139","reported_seq":694,"reported_epoch":489,"state":"active+clean","last_fresh":"2024-08-19T14:27:47.919209+0000","last_change":"2024-08-19T14:27:33.084267+0000","last_active":"2024-08-19T14:27:47.919209+0000","last_peered":"2024-08-19T14:27:47.919209+0000","last_clean":"2024-08-19T14:27:47.919209+0000","last_became_active":"2024-08-19T14:27:31.841421+0000","last_became_peered":"2024-08-19T14:27:31.841421+0000","last_unstale":"2024-08-19T14:27:47.919209+0000","last_undegraded":"2024-08-19T14:27:47.919209+0000","last_fullsized":"2024-08-19T14:27:47.919209+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:27:33.084161+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:27:33.084161+0000","last_clean_scrub_stamp":"2024-08-19T14:27:33.084161+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:29:01.689805+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.19","version":"70'139","reported_seq":1101,"reported_epoch":489,"state":"active+clean","last_fresh":"2024-08-19T14:27:47.919014+0000","last_change":"2024-08-19T14:26:55.039720+0000","last_active":"2024-08-19T14:27:47.919014+0000","last_peered":"2024-08-19T14:27:47.919014+0000","last_clean":"2024-08-19T14:27:47.919014+0000","last_became_active":"2024-08-19T14:18:31.902547+0000","last_became_peered":"2024-08-19T14:18:31.902547+0000","last_unstale":"2024-08-19T14:27:47.919014+0000","last_undegraded":"2024-08-19T14:27:47.919014+0000","last_fullsized":"2024-08-19T14:27:47.919014+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:26:55.039598+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:26:55.039598+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:27:57.212315+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1e","version":"70'139","reported_seq":760,"reported_epoch":489,"state":"active+clean","last_fresh":"2024-08-19T14:27:47.914965+0000","last_change":"2024-08-19T14:27:04.781012+0000","last_active":"2024-08-19T14:27:47.914965+0000","last_peered":"2024-08-19T14:27:47.914965+0000","last_clean":"2024-08-19T14:27:47.914965+0000","last_became_active":"2024-08-19T14:18:31.902546+0000","last_became_peered":"2024-08-19T14:18:31.902546+0000","last_unstale":"2024-08-19T14:27:47.914965+0000","last_undegraded":"2024-08-19T14:27:47.914965+0000","last_fullsized":"2024-08-19T14:27:47.914965+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:27:04.780939+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:27:04.780939+0000","last_clean_scrub_stamp":"2024-08-19T14:27:04.780939+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:20.233676+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.18","version":"489'9308","reported_seq":17758,"reported_epoch":489,"state":"active+clean","last_fresh":"2024-08-19T14:27:47.983388+0000","last_change":"2024-08-19T14:26:59.043461+0000","last_active":"2024-08-19T14:27:47.983388+0000","last_peered":"2024-08-19T14:27:47.983388+0000","last_clean":"2024-08-19T14:27:47.983388+0000","last_became_active":"2024-08-19T14:26:36.445937+0000","last_became_peered":"2024-08-19T14:26:36.445937+0000","last_unstale":"2024-08-19T14:27:47.983388+0000","last_undegraded":"2024-08-19T14:27:47.983388+0000","last_fullsized":"2024-08-19T14:27:47.983388+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":439,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":5,"last_scrub":"473'7817","last_scrub_stamp":"2024-08-19T14:26:59.043047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:26:59.043047+0000","objects_scrubbed":2147,"log_size":9308,"log_dups_size":0,"ondisk_log_size":9308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:27.776983+0000","scrub_duration":473,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42467328,"num_objects":648,"num_object_clones":0,"num_object_copies":1944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":648,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1499,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,1],"acting":[4,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1c","version":"70'139","reported_seq":867,"reported_epoch":489,"state":"active+clean","last_fresh":"2024-08-19T14:27:47.916261+0000","last_change":"2024-08-19T14:26:57.940698+0000","last_active":"2024-08-19T14:27:47.916261+0000","last_peered":"2024-08-19T14:27:47.916261+0000","last_clean":"2024-08-19T14:27:47.916261+0000","last_became_active":"2024-08-19T14:18:31.902494+0000","last_became_peered":"2024-08-19T14:18:31.902494+0000","last_unstale":"2024-08-19T14:27:47.916261+0000","last_undegraded":"2024-08-19T14:27:47.916261+0000","last_fullsized":"2024-08-19T14:27:47.916261+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:26:57.940466+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:26:57.940466+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:24.547300+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,11],"acting":[8,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"5.19","version":"489'9521","reported_seq":17974,"reported_epoch":489,"state":"active+clean","last_fresh":"2024-08-19T14:27:48.701968+0000","last_change":"2024-08-19T14:26:59.531135+0000","last_active":"2024-08-19T14:27:48.701968+0000","last_peered":"2024-08-19T14:27:48.701968+0000","last_clean":"2024-08-19T14:27:48.701968+0000","last_became_active":"2024-08-19T14:26:15.340594+0000","last_became_peered":"2024-08-19T14:26:15.340594+0000","last_unstale":"2024-08-19T14:27:48.701968+0000","last_undegraded":"2024-08-19T14:27:48.701968+0000","last_fullsized":"2024-08-19T14:27:48.701968+0000","mapping_epoch":457,"log_start":"0'0","ondisk_log_start":"0'0","created":439,"last_epoch_clean":458,"parent":"0.0","parent_split_bits":5,"last_scrub":"473'7983","last_scrub_stamp":"2024-08-19T14:26:59.530991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:24:43.574783+0000","last_clean_scrub_stamp":"2024-08-19T14:26:59.530991+0000","objects_scrubbed":2215,"log_size":9521,"log_dups_size":0,"ondisk_log_size":9521,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:18.755230+0000","scrub_duration":640,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43384832,"num_objects":662,"num_object_clones":0,"num_object_copies":1986,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1553,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1d","version":"70'139","reported_seq":781,"reported_epoch":489,"state":"active+clean","last_fresh":"2024-08-19T14:27:47.918137+0000","last_change":"2024-08-19T14:26:59.878357+0000","last_active":"2024-08-19T14:27:47.918137+0000","last_peered":"2024-08-19T14:27:47.918137+0000","last_clean":"2024-08-19T14:27:47.918137+0000","last_became_active":"2024-08-19T14:18:31.901183+0000","last_became_peered":"2024-08-19T14:18:31.901183+0000","last_unstale":"2024-08-19T14:27:47.918137+0000","last_undegraded":"2024-08-19T14:27:47.918137+0000","last_fullsized":"2024-08-19T14:27:47.918137+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:26:59.878236+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:26:59.878236+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:21.153098+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,5],"acting":[1,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":26,"stat_sum":{"num_bytes":1330905112,"num_objects":20309,"num_object_clones":0,"num_object_copies":60927,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20309,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":135215,"num_write_kb":3398593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4145426432,"data_stored":4145414216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":255721,"ondisk_log_size":255721,"up":78,"acting":78,"num_store_stats":9},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1802240,"data_stored":1788032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187654,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1000208,"kb_used_data":387272,"kb_used_omap":1,"kb_used_meta":612862,"kb_avail":92744944,"statfs":{"total":95995035648,"available":94970822656,"internally_reserved":0,"allocated":396566528,"data_stored":394356277,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":627571140},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":343597383883,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":74,"seq":317827580111,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1121680,"kb_used_data":458820,"kb_used_omap":1,"kb_used_meta":662846,"kb_avail":92623472,"statfs":{"total":95995035648,"available":94846435328,"internally_reserved":0,"allocated":469831680,"data_stored":467625501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":678754753},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874453,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1078476,"kb_used_data":519748,"kb_used_omap":1,"kb_used_meta":558718,"kb_avail":92666676,"statfs":{"total":95995035648,"available":94890676224,"internally_reserved":0,"allocated":532221952,"data_stored":530015773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":572127685},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":57,"seq":244813136088,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1106700,"kb_used_data":413316,"kb_used_omap":1,"kb_used_meta":693374,"kb_avail":92638452,"statfs":{"total":95995035648,"available":94861774848,"internally_reserved":0,"allocated":423235584,"data_stored":421026036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":710015434},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":206158430427,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692064,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1217560,"kb_used_data":626896,"kb_used_omap":1,"kb_used_meta":590654,"kb_avail":92527592,"statfs":{"total":95995035648,"available":94748254208,"internally_reserved":0,"allocated":641941504,"data_stored":639727701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":604830149},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986404,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1322112,"kb_used_data":614840,"kb_used_omap":1,"kb_used_meta":707262,"kb_avail":92423040,"statfs":{"total":95995035648,"available":94641192960,"internally_reserved":0,"allocated":629596160,"data_stored":627394236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":724236741},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149930,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":870864,"kb_used_data":266308,"kb_used_omap":1,"kb_used_meta":604542,"kb_avail":92874288,"statfs":{"total":95995035648,"available":95103270912,"internally_reserved":0,"allocated":272699392,"data_stored":270493213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":619051463},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411568,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575091,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1147288,"kb_used_data":407884,"kb_used_omap":1,"kb_used_meta":739198,"kb_avail":92597864,"statfs":{"total":95995035648,"available":94820212736,"internally_reserved":0,"allocated":417673216,"data_stored":415463485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":756939205},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738616,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1124748,"kb_used_data":507652,"kb_used_omap":1,"kb_used_meta":617022,"kb_avail":92620404,"statfs":{"total":95995035648,"available":94843293696,"internally_reserved":0,"allocated":519835648,"data_stored":517629469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":631830981},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":512098304,"data_stored":512098304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":392429568,"data_stored":392429568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":259457024,"data_stored":259457024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":613810176,"data_stored":613810176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":614731776,"data_stored":614727704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":393154560,"data_stored":393150488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":525271040,"data_stored":525271040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":457637888,"data_stored":457637888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":376836096,"data_stored":376832024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T14:27:54.298 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd pool set unique_pool_3 pgp_num 26 2024-08-19T14:27:54.298 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:27:54.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:53 smithi045 ceph-mon[20143]: pgmap v1128: 57 pgs: 57 active+clean; 1.3 GiB data, 9.9 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 1.20k op/s 2024-08-19T14:27:54.325 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:54.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:54.326 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:27:54.427 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:27:54.463 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:54.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:54.464 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:27:54.484 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-08-19T14:27:54.565 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:27:54.590 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:54.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:54.591 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:27:54.692 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:27:54.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:54.734 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:54.734 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:27:54.835 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:27:54.876 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:54.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:54.877 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:27:54.943 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:27:54.978 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:27:55.006 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:55.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:55.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:27:55.108 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:27:55.142 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:55.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:55.143 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:27:55.244 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:27:55.246 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:54 smithi045 ceph-mon[20143]: from='client.16476 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:27:55.248 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:54 smithi031 ceph-mon[16336]: from='client.16476 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:27:55.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:54 smithi032 ceph-mon[20004]: from='client.16476 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:27:55.279 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:55.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:55.280 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:27:55.285 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:27:55.286 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:27:55.317 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:27:55.317 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:27:55.347 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:27:55.347 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:27:55.350 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:27:55.376 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:27:55.376 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:27:55.380 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:27:55.405 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:27:55.405 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:27:55.410 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:55.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:55.411 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:27:55.429 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:27:55.429 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:27:55.446 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:27:55.446 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:27:55.468 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:27:55.468 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:27:55.492 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:27:55.493 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:27:55.512 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:27:55.526 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:27:55.527 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:27:55.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:55.545 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:55.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:27:55.550 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:27:55.550 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [], 2024-08-19T14:27:55.550 INFO:teuthology.orchestra.run.smithi045.stdout: "complaint_time": 30, 2024-08-19T14:27:55.550 INFO:teuthology.orchestra.run.smithi045.stdout: "num_blocked_ops": 0 2024-08-19T14:27:55.550 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:27:55.561 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:27:55.561 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:27:55.590 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:27:55.590 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:27:55.646 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:27:55.673 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:55.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:55.674 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:27:55.775 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:27:55.812 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:55.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:55.813 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:27:55.914 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:27:55.947 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:55.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:55.948 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:27:56.049 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:27:56.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:56.078 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:56.078 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:27:56.115 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:55 smithi045 ceph-mon[20143]: pgmap v1130: 57 pgs: 57 active+clean; 1.1 GiB data, 9.0 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 1.31k op/s 2024-08-19T14:27:56.115 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:55 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2992304317' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:27:56.178 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:27:56.209 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:56.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:56.210 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:27:56.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:55 smithi032 ceph-mon[20004]: pgmap v1130: 57 pgs: 57 active+clean; 1.1 GiB data, 9.0 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 1.31k op/s 2024-08-19T14:27:56.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:55 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2992304317' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:27:56.274 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-08-19T14:27:56.311 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:27:56.341 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:56.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:56.342 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:27:56.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:55 smithi031 ceph-mon[16336]: pgmap v1130: 57 pgs: 57 active+clean; 1.1 GiB data, 9.0 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 1.31k op/s 2024-08-19T14:27:56.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:55 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2992304317' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:27:56.443 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:27:56.486 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:56.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:56.487 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:27:56.589 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:27:56.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:56.619 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:56.620 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:27:56.687 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:27:56.720 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:27:56.749 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:56.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:56.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:27:56.851 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:27:56.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:56.899 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:56.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:27:56.959 INFO:teuthology.orchestra.run.smithi031.stderr:set pool 5 pgp_num to 26 2024-08-19T14:27:56.969 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-19T14:27:56.972 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:56 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1277207249' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num", "val": "26"}]: dispatch 2024-08-19T14:27:56.999 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:27:57.026 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:57.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:57.027 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:27:57.128 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:27:57.154 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:56 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1277207249' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num", "val": "26"}]: dispatch 2024-08-19T14:27:57.163 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:57.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:57.164 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:27:57.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:56 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1277207249' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num", "val": "26"}]: dispatch 2024-08-19T14:27:57.265 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:27:57.291 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:57.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:57.293 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:27:57.394 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:27:57.421 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:57.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:57.422 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:27:57.523 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:27:57.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:57.556 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:57.556 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:27:57.587 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:27:57.588 INFO:teuthology.orchestra.run.smithi045.stdout: "size": 20, 2024-08-19T14:27:57.588 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 600, 2024-08-19T14:27:57.588 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [ 2024-08-19T14:27:57.588 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.588 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:66232 4.6 4.fa550576 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e353)", 2024-08-19T14:27:57.588 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:21:04.816011+0000", 2024-08-19T14:27:57.588 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 412.76795591899997, 2024-08-19T14:27:57.588 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.98858874100000005, 2024-08-19T14:27:57.588 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:27:57.589 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:27:57.589 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-08-19T14:27:57.589 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:27:57.589 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:27:57.589 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:27:57.589 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 66232 2024-08-19T14:27:57.589 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.589 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:27:57.589 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.589 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:27:57.589 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.816011+0000", 2024-08-19T14:27:57.589 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:27:57.589 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.589 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.590 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:27:57.590 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.816011+0000", 2024-08-19T14:27:57.590 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:27:57.590 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.590 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.590 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:27:57.590 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.816013+0000", 2024-08-19T14:27:57.590 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.35e-06 2024-08-19T14:27:57.590 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.590 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.590 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:27:57.590 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.816032+0000", 2024-08-19T14:27:57.590 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9745000000000002e-05 2024-08-19T14:27:57.591 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.591 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.591 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:27:57.591 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.816046+0000", 2024-08-19T14:27:57.591 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3472000000000001e-05 2024-08-19T14:27:57.591 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.591 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.591 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:27:57.591 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.816052+0000", 2024-08-19T14:27:57.591 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.1929999999999998e-06 2024-08-19T14:27:57.591 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.591 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.591 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:27:57.592 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.816105+0000", 2024-08-19T14:27:57.592 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2914000000000002e-05 2024-08-19T14:27:57.592 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.592 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.592 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for peered", 2024-08-19T14:27:57.593 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.816116+0000", 2024-08-19T14:27:57.593 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0772e-05 2024-08-19T14:27:57.593 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.593 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.593 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:27:57.593 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.804584+0000", 2024-08-19T14:27:57.593 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.98846780999999995 2024-08-19T14:27:57.593 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.594 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.594 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:27:57.594 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.804600+0000", 2024-08-19T14:27:57.594 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6484999999999999e-05 2024-08-19T14:27:57.594 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:27:57.594 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:27:57.594 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:27:57.594 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.594 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.594 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:66234 4.6 4.4176ed76 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e353)", 2024-08-19T14:27:57.594 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:21:04.818467+0000", 2024-08-19T14:27:57.594 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 412.765500605, 2024-08-19T14:27:57.594 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.98617420499999997, 2024-08-19T14:27:57.594 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:27:57.595 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:27:57.595 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-08-19T14:27:57.595 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:27:57.595 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:27:57.595 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:27:57.595 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 66234 2024-08-19T14:27:57.595 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.595 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:27:57.595 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.595 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:27:57.595 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.818467+0000", 2024-08-19T14:27:57.595 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:27:57.595 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.595 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.596 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:27:57.596 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.818467+0000", 2024-08-19T14:27:57.596 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:27:57.596 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.596 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.596 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:27:57.596 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.818468+0000", 2024-08-19T14:27:57.596 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.0699999999999996e-07 2024-08-19T14:27:57.596 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.596 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.596 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:27:57.596 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.818487+0000", 2024-08-19T14:27:57.596 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8958999999999999e-05 2024-08-19T14:27:57.596 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.597 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.597 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:27:57.597 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.818500+0000", 2024-08-19T14:27:57.597 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3098999999999999e-05 2024-08-19T14:27:57.597 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.597 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.597 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:27:57.597 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.818507+0000", 2024-08-19T14:27:57.597 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.8369999999999997e-06 2024-08-19T14:27:57.597 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.597 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.597 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:27:57.597 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.818561+0000", 2024-08-19T14:27:57.597 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.3165999999999999e-05 2024-08-19T14:27:57.598 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.598 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.598 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for peered", 2024-08-19T14:27:57.598 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.818568+0000", 2024-08-19T14:27:57.598 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.8599999999999993e-06 2024-08-19T14:27:57.598 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.598 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.598 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:27:57.599 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.804628+0000", 2024-08-19T14:27:57.599 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.98605925000000005 2024-08-19T14:27:57.599 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.599 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.599 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:27:57.599 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.804641+0000", 2024-08-19T14:27:57.599 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3127e-05 2024-08-19T14:27:57.599 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:27:57.600 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:27:57.600 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:27:57.600 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.600 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.600 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:66252 4.6 4.4ffe49a6 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e353)", 2024-08-19T14:27:57.600 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:21:04.825959+0000", 2024-08-19T14:27:57.600 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 412.75800847699998, 2024-08-19T14:27:57.600 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.97870372400000005, 2024-08-19T14:27:57.600 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:27:57.600 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:27:57.600 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-08-19T14:27:57.600 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:27:57.600 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:27:57.600 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:27:57.601 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 66252 2024-08-19T14:27:57.601 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.601 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:27:57.601 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.601 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:27:57.601 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.825959+0000", 2024-08-19T14:27:57.601 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:27:57.601 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.601 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.601 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:27:57.601 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.825959+0000", 2024-08-19T14:27:57.601 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:27:57.602 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.602 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.602 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:27:57.602 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.825960+0000", 2024-08-19T14:27:57.602 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.13e-06 2024-08-19T14:27:57.602 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.602 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.602 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:27:57.602 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.825979+0000", 2024-08-19T14:27:57.602 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9284999999999999e-05 2024-08-19T14:27:57.602 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.602 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.603 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:27:57.603 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.825991+0000", 2024-08-19T14:27:57.603 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1894e-05 2024-08-19T14:27:57.603 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.603 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.603 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:27:57.603 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.825998+0000", 2024-08-19T14:27:57.603 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.1080000000000004e-06 2024-08-19T14:27:57.603 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.603 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.604 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:27:57.604 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.826050+0000", 2024-08-19T14:27:57.604 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2061000000000002e-05 2024-08-19T14:27:57.604 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.604 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.604 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for peered", 2024-08-19T14:27:57.604 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.826057+0000", 2024-08-19T14:27:57.604 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.4389999999999997e-06 2024-08-19T14:27:57.604 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.604 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.604 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:27:57.604 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.804657+0000", 2024-08-19T14:27:57.604 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.97859989999999997 2024-08-19T14:27:57.604 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.604 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.605 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:27:57.605 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.804663+0000", 2024-08-19T14:27:57.605 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.9070000000000004e-06 2024-08-19T14:27:57.605 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:27:57.605 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:27:57.605 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:27:57.605 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.605 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.605 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:66266 4.6 4.2a59efa6 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e353)", 2024-08-19T14:27:57.605 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:21:04.834727+0000", 2024-08-19T14:27:57.605 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 412.74923984600002, 2024-08-19T14:27:57.605 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.96997657800000003, 2024-08-19T14:27:57.605 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:27:57.605 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:27:57.606 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-08-19T14:27:57.606 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:27:57.606 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:27:57.606 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:27:57.606 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 66266 2024-08-19T14:27:57.606 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.606 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:27:57.606 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.606 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:27:57.607 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.834727+0000", 2024-08-19T14:27:57.607 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:27:57.607 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.607 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.607 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:27:57.607 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.834727+0000", 2024-08-19T14:27:57.607 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:27:57.607 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.607 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.607 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:27:57.607 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.834729+0000", 2024-08-19T14:27:57.607 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5579999999999999e-06 2024-08-19T14:27:57.608 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.608 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.608 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:27:57.608 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.834755+0000", 2024-08-19T14:27:57.608 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.6080000000000001e-05 2024-08-19T14:27:57.608 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.608 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.608 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:27:57.608 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.834781+0000", 2024-08-19T14:27:57.608 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.5605999999999998e-05 2024-08-19T14:27:57.608 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.608 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.608 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:27:57.609 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.834812+0000", 2024-08-19T14:27:57.609 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.1229999999999997e-05 2024-08-19T14:27:57.609 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.609 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.609 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:27:57.609 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.834865+0000", 2024-08-19T14:27:57.610 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2611000000000002e-05 2024-08-19T14:27:57.610 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.610 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.610 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for peered", 2024-08-19T14:27:57.610 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.834870+0000", 2024-08-19T14:27:57.610 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.5169999999999999e-06 2024-08-19T14:27:57.610 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.610 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.610 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:27:57.610 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.804695+0000", 2024-08-19T14:27:57.610 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.96982486499999998 2024-08-19T14:27:57.610 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.610 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.611 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:27:57.611 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.804704+0000", 2024-08-19T14:27:57.611 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.1109999999999997e-06 2024-08-19T14:27:57.611 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:27:57.611 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:27:57.611 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:27:57.611 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.611 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.611 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:66269 4.6 4.f7812866 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e353)", 2024-08-19T14:27:57.611 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:21:04.835697+0000", 2024-08-19T14:27:57.611 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 412.74826993599999, 2024-08-19T14:27:57.611 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.96905406199999999, 2024-08-19T14:27:57.611 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:27:57.611 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:27:57.611 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-08-19T14:27:57.611 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:27:57.612 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:27:57.612 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:27:57.612 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 66269 2024-08-19T14:27:57.612 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.612 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:27:57.612 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.612 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:27:57.612 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.835697+0000", 2024-08-19T14:27:57.612 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:27:57.612 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.612 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.612 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:27:57.612 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.835697+0000", 2024-08-19T14:27:57.613 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:27:57.613 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.613 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.613 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:27:57.613 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.835698+0000", 2024-08-19T14:27:57.613 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.5600000000000004e-07 2024-08-19T14:27:57.613 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.613 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.613 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:27:57.614 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.835719+0000", 2024-08-19T14:27:57.614 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0975000000000002e-05 2024-08-19T14:27:57.614 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.614 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.614 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:27:57.614 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.835732+0000", 2024-08-19T14:27:57.614 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2690999999999999e-05 2024-08-19T14:27:57.614 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.614 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.614 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:27:57.615 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.835739+0000", 2024-08-19T14:27:57.615 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.2069999999999998e-06 2024-08-19T14:27:57.615 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.615 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.615 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:27:57.615 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.835781+0000", 2024-08-19T14:27:57.615 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.1743999999999999e-05 2024-08-19T14:27:57.615 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.615 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.615 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for peered", 2024-08-19T14:27:57.615 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.835785+0000", 2024-08-19T14:27:57.615 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.4320000000000001e-06 2024-08-19T14:27:57.615 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.615 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.615 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:27:57.615 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.804742+0000", 2024-08-19T14:27:57.616 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.96895661200000005 2024-08-19T14:27:57.616 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.616 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.616 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:27:57.616 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.804751+0000", 2024-08-19T14:27:57.616 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.4450000000000002e-06 2024-08-19T14:27:57.616 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:27:57.616 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:27:57.616 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:27:57.616 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.616 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.616 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:66273 4.6 4.a206dc36 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e353)", 2024-08-19T14:27:57.616 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:21:04.837306+0000", 2024-08-19T14:27:57.616 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 412.74666151500003, 2024-08-19T14:27:57.616 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.96748620299999999, 2024-08-19T14:27:57.617 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:27:57.617 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:27:57.617 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-08-19T14:27:57.617 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:27:57.617 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:27:57.617 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:27:57.617 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 66273 2024-08-19T14:27:57.617 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.617 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:27:57.617 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.617 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:27:57.617 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.837306+0000", 2024-08-19T14:27:57.617 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:27:57.617 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.617 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.618 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:27:57.618 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.837306+0000", 2024-08-19T14:27:57.618 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:27:57.618 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.618 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.618 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:27:57.618 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.837307+0000", 2024-08-19T14:27:57.618 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.226e-06 2024-08-19T14:27:57.618 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.618 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.618 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:27:57.618 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.837330+0000", 2024-08-19T14:27:57.619 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.3516999999999999e-05 2024-08-19T14:27:57.619 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.619 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.619 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:27:57.619 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.837344+0000", 2024-08-19T14:27:57.619 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3548e-05 2024-08-19T14:27:57.619 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.619 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.619 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:27:57.619 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.837351+0000", 2024-08-19T14:27:57.619 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.3799999999999996e-06 2024-08-19T14:27:57.619 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.619 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.619 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:27:57.619 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.837396+0000", 2024-08-19T14:27:57.620 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.4087e-05 2024-08-19T14:27:57.620 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.620 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.620 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for peered", 2024-08-19T14:27:57.620 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.837400+0000", 2024-08-19T14:27:57.620 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.9389999999999996e-06 2024-08-19T14:27:57.620 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.620 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.620 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:27:57.621 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.804780+0000", 2024-08-19T14:27:57.621 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.96737990600000001 2024-08-19T14:27:57.621 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.621 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.621 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:27:57.621 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.804792+0000", 2024-08-19T14:27:57.621 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1600000000000001e-05 2024-08-19T14:27:57.622 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:27:57.622 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:27:57.622 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:27:57.622 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.622 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.622 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:66344 4.6 4.4505c966 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e353)", 2024-08-19T14:27:57.622 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:21:04.878110+0000", 2024-08-19T14:27:57.622 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 412.70585677899999, 2024-08-19T14:27:57.622 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.92671215799999995, 2024-08-19T14:27:57.622 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:27:57.622 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:27:57.622 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-08-19T14:27:57.622 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:27:57.622 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:27:57.623 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:27:57.623 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 66344 2024-08-19T14:27:57.623 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.623 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:27:57.623 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.623 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:27:57.623 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.878110+0000", 2024-08-19T14:27:57.623 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:27:57.623 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.624 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.624 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:27:57.624 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.878110+0000", 2024-08-19T14:27:57.624 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:27:57.624 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.624 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.624 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:27:57.624 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.878112+0000", 2024-08-19T14:27:57.624 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3689999999999999e-06 2024-08-19T14:27:57.624 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.625 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.625 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:27:57.625 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.878135+0000", 2024-08-19T14:27:57.625 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2932999999999999e-05 2024-08-19T14:27:57.625 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.625 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.625 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:27:57.625 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.878149+0000", 2024-08-19T14:27:57.625 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4239999999999999e-05 2024-08-19T14:27:57.625 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.625 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.625 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:27:57.625 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.878156+0000", 2024-08-19T14:27:57.625 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.1350000000000003e-06 2024-08-19T14:27:57.625 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.626 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.626 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:27:57.626 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.878212+0000", 2024-08-19T14:27:57.626 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.6112000000000001e-05 2024-08-19T14:27:57.626 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.626 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.626 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for peered", 2024-08-19T14:27:57.626 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.878222+0000", 2024-08-19T14:27:57.626 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.2939999999999994e-06 2024-08-19T14:27:57.626 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.627 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.627 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:27:57.627 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.804814+0000", 2024-08-19T14:27:57.627 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.92659212300000005 2024-08-19T14:27:57.627 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.627 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.627 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:27:57.627 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.804823+0000", 2024-08-19T14:27:57.627 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.952e-06 2024-08-19T14:27:57.627 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:27:57.627 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:27:57.627 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:27:57.627 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.628 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.628 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:66352 4.6 4.84354a56 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e353)", 2024-08-19T14:27:57.628 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:21:04.884189+0000", 2024-08-19T14:27:57.628 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 412.69977857499998, 2024-08-19T14:27:57.628 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.92066624200000002, 2024-08-19T14:27:57.628 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:27:57.628 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:27:57.628 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-08-19T14:27:57.628 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:27:57.628 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:27:57.628 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:27:57.628 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 66352 2024-08-19T14:27:57.629 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.629 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:27:57.629 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.629 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:27:57.629 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.884189+0000", 2024-08-19T14:27:57.629 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:27:57.629 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.629 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.629 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:27:57.629 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.884189+0000", 2024-08-19T14:27:57.629 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:27:57.629 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.630 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.630 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:27:57.630 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.884190+0000", 2024-08-19T14:27:57.630 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2729999999999999e-06 2024-08-19T14:27:57.630 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.630 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.630 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:27:57.630 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.884209+0000", 2024-08-19T14:27:57.630 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.946e-05 2024-08-19T14:27:57.630 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.631 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.631 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:27:57.631 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.884229+0000", 2024-08-19T14:27:57.631 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9086e-05 2024-08-19T14:27:57.631 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.631 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.631 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:27:57.631 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.884237+0000", 2024-08-19T14:27:57.631 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.5760000000000006e-06 2024-08-19T14:27:57.631 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.631 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.631 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:27:57.632 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.884321+0000", 2024-08-19T14:27:57.632 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.3695000000000004e-05 2024-08-19T14:27:57.632 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.632 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.632 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for peered", 2024-08-19T14:27:57.632 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.884329+0000", 2024-08-19T14:27:57.632 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.6100000000000006e-06 2024-08-19T14:27:57.632 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.632 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.632 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:27:57.632 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.804845+0000", 2024-08-19T14:27:57.632 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.92051592900000001 2024-08-19T14:27:57.632 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.633 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.633 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:27:57.633 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.804855+0000", 2024-08-19T14:27:57.633 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.6129999999999993e-06 2024-08-19T14:27:57.633 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:27:57.633 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:27:57.633 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:27:57.633 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.633 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.633 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:66355 4.6 4.67c06196 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e353)", 2024-08-19T14:27:57.634 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:21:04.884802+0000", 2024-08-19T14:27:57.634 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 412.699165107, 2024-08-19T14:27:57.634 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.92009072300000005, 2024-08-19T14:27:57.634 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:27:57.634 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:27:57.634 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-08-19T14:27:57.634 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:27:57.634 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:27:57.634 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:27:57.634 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 66355 2024-08-19T14:27:57.634 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.634 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:27:57.635 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.635 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:27:57.635 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.884802+0000", 2024-08-19T14:27:57.635 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:27:57.635 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.635 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.635 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:27:57.635 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.884802+0000", 2024-08-19T14:27:57.635 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:27:57.635 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.635 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.636 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:27:57.636 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.884803+0000", 2024-08-19T14:27:57.636 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.6599999999999994e-07 2024-08-19T14:27:57.636 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.636 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.636 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:27:57.636 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.884827+0000", 2024-08-19T14:27:57.636 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.4334000000000001e-05 2024-08-19T14:27:57.636 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.636 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.636 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:27:57.636 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.884844+0000", 2024-08-19T14:27:57.636 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6330000000000001e-05 2024-08-19T14:27:57.637 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.637 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.637 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:27:57.637 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.884852+0000", 2024-08-19T14:27:57.638 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.9149999999999996e-06 2024-08-19T14:27:57.638 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.638 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.638 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:27:57.638 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.884910+0000", 2024-08-19T14:27:57.638 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7828000000000003e-05 2024-08-19T14:27:57.638 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.638 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.638 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for peered", 2024-08-19T14:27:57.638 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.884918+0000", 2024-08-19T14:27:57.639 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.8349999999999993e-06 2024-08-19T14:27:57.639 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.639 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.639 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:27:57.639 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.804883+0000", 2024-08-19T14:27:57.639 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.91996511400000003 2024-08-19T14:27:57.639 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.639 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.639 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:27:57.639 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.804893+0000", 2024-08-19T14:27:57.639 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.4010000000000003e-06 2024-08-19T14:27:57.639 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:27:57.639 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:27:57.639 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:27:57.639 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.640 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.640 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:66439 4.6 4.e8f25516 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e353)", 2024-08-19T14:27:57.640 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:21:04.937010+0000", 2024-08-19T14:27:57.640 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 412.64695765900001, 2024-08-19T14:27:57.640 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.86793240199999999, 2024-08-19T14:27:57.640 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:27:57.640 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:27:57.640 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-08-19T14:27:57.640 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:27:57.640 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:27:57.640 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:27:57.640 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 66439 2024-08-19T14:27:57.640 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.640 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:27:57.641 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.641 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:27:57.641 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.937010+0000", 2024-08-19T14:27:57.641 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:27:57.641 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.641 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.641 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:27:57.641 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.937010+0000", 2024-08-19T14:27:57.641 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:27:57.641 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.641 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.641 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:27:57.641 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.937011+0000", 2024-08-19T14:27:57.642 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5120000000000001e-06 2024-08-19T14:27:57.642 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.642 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.642 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:27:57.642 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.937029+0000", 2024-08-19T14:27:57.642 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7706e-05 2024-08-19T14:27:57.642 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.642 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.642 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:27:57.643 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.937043+0000", 2024-08-19T14:27:57.643 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4214e-05 2024-08-19T14:27:57.643 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.643 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.643 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:27:57.643 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.937050+0000", 2024-08-19T14:27:57.643 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4139999999999997e-06 2024-08-19T14:27:57.643 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.644 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.644 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:27:57.644 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.937095+0000", 2024-08-19T14:27:57.644 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.4833999999999998e-05 2024-08-19T14:27:57.644 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.644 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.644 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for peered", 2024-08-19T14:27:57.644 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.937102+0000", 2024-08-19T14:27:57.644 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.3740000000000003e-06 2024-08-19T14:27:57.644 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.644 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.644 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:27:57.644 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.804932+0000", 2024-08-19T14:27:57.644 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.86783048900000004 2024-08-19T14:27:57.645 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.645 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.645 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:27:57.645 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.804942+0000", 2024-08-19T14:27:57.645 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.859e-06 2024-08-19T14:27:57.645 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:27:57.645 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:27:57.645 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:27:57.645 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.645 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.645 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:66443 4.6 4.31b8f586 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e353)", 2024-08-19T14:27:57.645 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:21:04.938019+0000", 2024-08-19T14:27:57.645 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 412.64594851499999, 2024-08-19T14:27:57.645 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.86697104899999999, 2024-08-19T14:27:57.646 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:27:57.646 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:27:57.646 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-08-19T14:27:57.646 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:27:57.646 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:27:57.646 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:27:57.646 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 66443 2024-08-19T14:27:57.646 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.646 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:27:57.646 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.646 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:27:57.646 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.938019+0000", 2024-08-19T14:27:57.646 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:27:57.647 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.647 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.647 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:27:57.647 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.938019+0000", 2024-08-19T14:27:57.647 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:27:57.647 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.647 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.647 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:27:57.647 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.938020+0000", 2024-08-19T14:27:57.647 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.09e-07 2024-08-19T14:27:57.647 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.647 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.647 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:27:57.648 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.938059+0000", 2024-08-19T14:27:57.648 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.9134000000000002e-05 2024-08-19T14:27:57.648 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.648 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.648 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:27:57.648 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.938076+0000", 2024-08-19T14:27:57.648 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.766e-05 2024-08-19T14:27:57.648 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.648 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.648 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:27:57.648 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.938081+0000", 2024-08-19T14:27:57.648 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0969999999999996e-06 2024-08-19T14:27:57.648 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.649 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.649 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:27:57.649 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.938126+0000", 2024-08-19T14:27:57.649 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.4736e-05 2024-08-19T14:27:57.649 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.649 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.649 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for peered", 2024-08-19T14:27:57.649 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.938134+0000", 2024-08-19T14:27:57.649 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.464e-06 2024-08-19T14:27:57.649 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.649 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.649 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:27:57.649 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.804980+0000", 2024-08-19T14:27:57.649 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.86684594400000003 2024-08-19T14:27:57.649 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.650 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.650 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:27:57.650 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.804990+0000", 2024-08-19T14:27:57.650 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0205e-05 2024-08-19T14:27:57.650 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:27:57.650 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:27:57.650 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:27:57.650 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.650 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.650 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:66472 4.6 4.b9e97686 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e353)", 2024-08-19T14:27:57.650 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:21:04.958071+0000", 2024-08-19T14:27:57.650 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 412.62589597200002, 2024-08-19T14:27:57.650 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.84697957599999996, 2024-08-19T14:27:57.650 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:27:57.651 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:27:57.651 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-08-19T14:27:57.651 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:27:57.651 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:27:57.651 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:27:57.651 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 66472 2024-08-19T14:27:57.651 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.651 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:27:57.651 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.651 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:27:57.651 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.958071+0000", 2024-08-19T14:27:57.651 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:27:57.651 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.651 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.651 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:27:57.652 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.958071+0000", 2024-08-19T14:27:57.652 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:27:57.652 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.652 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.652 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:27:57.652 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.958073+0000", 2024-08-19T14:27:57.652 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.533e-06 2024-08-19T14:27:57.652 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.652 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.652 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:27:57.652 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.958126+0000", 2024-08-19T14:27:57.652 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.3430999999999997e-05 2024-08-19T14:27:57.652 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.652 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.652 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:27:57.653 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.958142+0000", 2024-08-19T14:27:57.653 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5259000000000001e-05 2024-08-19T14:27:57.653 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.653 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.653 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:27:57.653 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.958148+0000", 2024-08-19T14:27:57.653 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.7109999999999998e-06 2024-08-19T14:27:57.653 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.653 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.653 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:27:57.653 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.958204+0000", 2024-08-19T14:27:57.653 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.5282999999999999e-05 2024-08-19T14:27:57.653 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.653 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.654 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for peered", 2024-08-19T14:27:57.654 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.958211+0000", 2024-08-19T14:27:57.654 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.5950000000000001e-06 2024-08-19T14:27:57.654 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.654 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.654 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:27:57.654 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.805037+0000", 2024-08-19T14:27:57.654 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.84682633500000004 2024-08-19T14:27:57.654 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.654 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.654 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:27:57.654 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.805051+0000", 2024-08-19T14:27:57.654 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3429e-05 2024-08-19T14:27:57.655 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:27:57.655 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:27:57.655 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:27:57.655 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.655 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.655 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:66507 4.6 4.d2914886 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e353)", 2024-08-19T14:27:57.655 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:21:04.985293+0000", 2024-08-19T14:27:57.655 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 412.598674593, 2024-08-19T14:27:57.655 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.81979336000000003, 2024-08-19T14:27:57.655 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:27:57.655 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:27:57.655 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-08-19T14:27:57.655 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:27:57.655 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:27:57.656 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:27:57.656 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 66507 2024-08-19T14:27:57.656 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.656 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:27:57.656 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.656 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:27:57.656 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.985293+0000", 2024-08-19T14:27:57.656 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:27:57.656 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.656 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.656 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:27:57.657 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.985293+0000", 2024-08-19T14:27:57.657 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:27:57.657 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.657 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.657 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:27:57.657 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.985294+0000", 2024-08-19T14:27:57.657 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.06e-06 2024-08-19T14:27:57.657 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.657 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.657 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:27:57.658 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.985312+0000", 2024-08-19T14:27:57.658 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8604999999999999e-05 2024-08-19T14:27:57.658 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.658 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.658 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:27:57.658 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.985324+0000", 2024-08-19T14:27:57.658 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1511e-05 2024-08-19T14:27:57.658 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.659 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.659 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:27:57.659 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.985330+0000", 2024-08-19T14:27:57.659 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.2230000000000004e-06 2024-08-19T14:27:57.659 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.659 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.659 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:27:57.659 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.985375+0000", 2024-08-19T14:27:57.659 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.4808999999999997e-05 2024-08-19T14:27:57.659 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.659 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.659 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for peered", 2024-08-19T14:27:57.660 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.985380+0000", 2024-08-19T14:27:57.660 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4480000000000002e-06 2024-08-19T14:27:57.660 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.660 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.660 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:27:57.660 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.805076+0000", 2024-08-19T14:27:57.660 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.81969612000000003 2024-08-19T14:27:57.660 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.660 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.660 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:27:57.660 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.805086+0000", 2024-08-19T14:27:57.660 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.5840000000000001e-06 2024-08-19T14:27:57.660 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:27:57.660 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:27:57.660 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:27:57.661 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.661 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.661 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:66512 4.6 4.e5cfc156 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e353)", 2024-08-19T14:27:57.661 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:21:04.992563+0000", 2024-08-19T14:27:57.661 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 412.59140391199998, 2024-08-19T14:27:57.661 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.812551829, 2024-08-19T14:27:57.661 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:27:57.661 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:27:57.661 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-08-19T14:27:57.661 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:27:57.661 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:27:57.661 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:27:57.661 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 66512 2024-08-19T14:27:57.661 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.662 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:27:57.662 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.662 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:27:57.662 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.992563+0000", 2024-08-19T14:27:57.662 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:27:57.662 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.662 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.662 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:27:57.662 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.992563+0000", 2024-08-19T14:27:57.662 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:27:57.662 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.662 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.662 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:27:57.662 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.992564+0000", 2024-08-19T14:27:57.663 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.7600000000000006e-07 2024-08-19T14:27:57.663 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.663 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.663 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:27:57.663 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.992582+0000", 2024-08-19T14:27:57.663 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7362000000000001e-05 2024-08-19T14:27:57.663 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.663 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.663 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:27:57.663 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.992594+0000", 2024-08-19T14:27:57.663 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1956999999999999e-05 2024-08-19T14:27:57.663 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.664 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.664 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:27:57.664 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.992601+0000", 2024-08-19T14:27:57.664 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.2599999999999999e-06 2024-08-19T14:27:57.664 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.664 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.664 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:27:57.664 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.992644+0000", 2024-08-19T14:27:57.664 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.3111000000000001e-05 2024-08-19T14:27:57.664 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.664 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.664 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for peered", 2024-08-19T14:27:57.664 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:04.992649+0000", 2024-08-19T14:27:57.664 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.9960000000000001e-06 2024-08-19T14:27:57.664 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.665 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.665 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:27:57.665 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.805107+0000", 2024-08-19T14:27:57.665 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.81245753099999996 2024-08-19T14:27:57.665 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.665 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.665 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:27:57.665 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.805115+0000", 2024-08-19T14:27:57.665 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.636e-06 2024-08-19T14:27:57.665 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:27:57.665 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:27:57.665 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:27:57.666 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.666 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.666 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:66538 4.6 4.94521946 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e353)", 2024-08-19T14:27:57.666 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:21:05.030374+0000", 2024-08-19T14:27:57.666 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 412.55359291399998, 2024-08-19T14:27:57.666 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.77477269400000004, 2024-08-19T14:27:57.667 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:27:57.667 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:27:57.667 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-08-19T14:27:57.667 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:27:57.667 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:27:57.667 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:27:57.667 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 66538 2024-08-19T14:27:57.667 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.667 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:27:57.668 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.668 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:27:57.668 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.030374+0000", 2024-08-19T14:27:57.668 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:27:57.668 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.668 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.668 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:27:57.668 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.030374+0000", 2024-08-19T14:27:57.668 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:27:57.668 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.668 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.668 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:27:57.668 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.030376+0000", 2024-08-19T14:27:57.668 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1850000000000001e-06 2024-08-19T14:27:57.669 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.669 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.669 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:27:57.669 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.030394+0000", 2024-08-19T14:27:57.669 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8159999999999999e-05 2024-08-19T14:27:57.669 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.669 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.669 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:27:57.669 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.030406+0000", 2024-08-19T14:27:57.669 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2004000000000001e-05 2024-08-19T14:27:57.669 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.669 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.669 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:27:57.669 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.030411+0000", 2024-08-19T14:27:57.670 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2000000000000002e-06 2024-08-19T14:27:57.670 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.670 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.670 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:27:57.670 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:27:57.671 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.030463+0000", 2024-08-19T14:27:57.671 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2227e-05 2024-08-19T14:27:57.671 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.672 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.672 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for peered", 2024-08-19T14:27:57.672 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.030473+0000", 2024-08-19T14:27:57.672 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0353000000000001e-05 2024-08-19T14:27:57.672 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.672 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.672 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:27:57.672 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.805138+0000", 2024-08-19T14:27:57.672 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.77466448700000001 2024-08-19T14:27:57.672 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.672 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.672 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:27:57.672 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.805147+0000", 2024-08-19T14:27:57.672 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.0780000000000002e-06 2024-08-19T14:27:57.673 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:27:57.673 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:27:57.673 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:27:57.673 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.673 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.673 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:66550 4.6 4.8249c486 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e353)", 2024-08-19T14:27:57.673 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:21:05.061787+0000", 2024-08-19T14:27:57.673 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 412.52218026200001, 2024-08-19T14:27:57.673 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.74339024399999998, 2024-08-19T14:27:57.673 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:27:57.673 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:27:57.673 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-08-19T14:27:57.673 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:27:57.673 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:27:57.674 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:27:57.674 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 66550 2024-08-19T14:27:57.674 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.674 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:27:57.674 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.674 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:27:57.674 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.061787+0000", 2024-08-19T14:27:57.674 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:27:57.674 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.674 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.674 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:27:57.674 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.061787+0000", 2024-08-19T14:27:57.674 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:27:57.674 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.675 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.675 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:27:57.675 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.061788+0000", 2024-08-19T14:27:57.675 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.263e-06 2024-08-19T14:27:57.675 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.675 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.675 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:27:57.675 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.061829+0000", 2024-08-19T14:27:57.675 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.0988000000000001e-05 2024-08-19T14:27:57.675 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.675 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.675 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:27:57.675 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.061840+0000", 2024-08-19T14:27:57.676 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.112e-05 2024-08-19T14:27:57.676 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.676 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.676 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:27:57.676 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.061846+0000", 2024-08-19T14:27:57.676 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.3299999999999998e-06 2024-08-19T14:27:57.676 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.676 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.676 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:27:57.677 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.061902+0000", 2024-08-19T14:27:57.677 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.6368e-05 2024-08-19T14:27:57.677 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.677 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.677 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for peered", 2024-08-19T14:27:57.677 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.061911+0000", 2024-08-19T14:27:57.677 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.3910000000000004e-06 2024-08-19T14:27:57.677 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.677 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.677 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:27:57.677 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.805169+0000", 2024-08-19T14:27:57.677 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.74325783099999998 2024-08-19T14:27:57.677 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.677 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.678 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:27:57.678 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:05.805177+0000", 2024-08-19T14:27:57.678 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.9529999999999999e-06 2024-08-19T14:27:57.678 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:27:57.678 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:27:57.678 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:27:57.678 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.678 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.678 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.25813.0:190861 5.e e481/477 5:75e2f3e5:::benchmark_data_smithi045_85309_object70608:head v 481'11903, mlcod=481'11899)", 2024-08-19T14:27:57.678 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:27:24.466007+0000", 2024-08-19T14:27:57.678 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 33.117960468, 2024-08-19T14:27:57.678 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.028908151999999999, 2024-08-19T14:27:57.678 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:27:57.679 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:27:57.679 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:27:57.679 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:27:57.679 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.679 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:27:57.679 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:24.466007+0000", 2024-08-19T14:27:57.679 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:27:57.679 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.679 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.679 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:27:57.679 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:24.466007+0000", 2024-08-19T14:27:57.679 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:27:57.679 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.680 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.680 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:27:57.680 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:24.466007+0000", 2024-08-19T14:27:57.680 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.8400000000000004e-07 2024-08-19T14:27:57.680 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.680 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.680 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:27:57.680 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:24.466015+0000", 2024-08-19T14:27:57.680 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.1810000000000004e-06 2024-08-19T14:27:57.680 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.680 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.680 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:27:57.680 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:24.466030+0000", 2024-08-19T14:27:57.681 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5647000000000001e-05 2024-08-19T14:27:57.681 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.681 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.681 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:27:57.681 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:24.466038+0000", 2024-08-19T14:27:57.681 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4660000000000002e-06 2024-08-19T14:27:57.681 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.681 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.681 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:27:57.681 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:24.470082+0000", 2024-08-19T14:27:57.682 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0040439849999999999 2024-08-19T14:27:57.682 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.682 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.682 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:27:57.682 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:24.470119+0000", 2024-08-19T14:27:57.682 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.6885999999999999e-05 2024-08-19T14:27:57.682 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.682 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.682 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:27:57.682 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:24.494866+0000", 2024-08-19T14:27:57.682 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.02474734 2024-08-19T14:27:57.682 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.683 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.683 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:27:57.683 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:24.494915+0000", 2024-08-19T14:27:57.683 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.8962999999999999e-05 2024-08-19T14:27:57.683 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:27:57.683 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:27:57.683 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:27:57.683 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.683 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.683 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.25813.0:217785 5.b e488/477 5:dc92cfe9:::benchmark_data_smithi045_85309_object97532:head v 488'13839, mlcod=488'13834)", 2024-08-19T14:27:57.683 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:27:47.695225+0000", 2024-08-19T14:27:57.683 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 9.8887419059999999, 2024-08-19T14:27:57.683 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.032146156000000002, 2024-08-19T14:27:57.684 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:27:57.684 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:27:57.684 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:27:57.684 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:27:57.684 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.684 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:27:57.684 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:47.695225+0000", 2024-08-19T14:27:57.684 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:27:57.684 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.684 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.684 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:27:57.685 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:47.695225+0000", 2024-08-19T14:27:57.685 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:27:57.685 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.685 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.685 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:27:57.685 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:47.695226+0000", 2024-08-19T14:27:57.685 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.1500000000000003e-07 2024-08-19T14:27:57.685 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.685 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.685 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:27:57.685 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:47.695233+0000", 2024-08-19T14:27:57.685 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.1149999999999996e-06 2024-08-19T14:27:57.686 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.686 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.686 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:27:57.686 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:47.695248+0000", 2024-08-19T14:27:57.686 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4582e-05 2024-08-19T14:27:57.686 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.686 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.686 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:27:57.686 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:47.695256+0000", 2024-08-19T14:27:57.687 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.5989999999999996e-06 2024-08-19T14:27:57.687 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.687 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.687 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:27:57.687 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:47.700093+0000", 2024-08-19T14:27:57.687 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0048376959999999998 2024-08-19T14:27:57.687 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.687 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.687 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:27:57.688 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:47.700146+0000", 2024-08-19T14:27:57.688 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2874999999999998e-05 2024-08-19T14:27:57.688 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.688 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.688 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:27:57.688 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:47.727189+0000", 2024-08-19T14:27:57.688 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.027043306999999999 2024-08-19T14:27:57.688 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.688 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.688 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:27:57.688 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:47.727372+0000", 2024-08-19T14:27:57.689 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000182067 2024-08-19T14:27:57.689 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:27:57.689 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:27:57.689 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:27:57.689 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.689 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.689 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.25813.0:220135 5.b e489/477 5:d32b43da:::benchmark_data_smithi045_85309_object99882:head v 489'13980, mlcod=489'13977)", 2024-08-19T14:27:57.689 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:27:49.555811+0000", 2024-08-19T14:27:57.689 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 8.0281558850000003, 2024-08-19T14:27:57.689 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.024294257, 2024-08-19T14:27:57.689 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:27:57.689 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:27:57.689 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:27:57.690 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:27:57.690 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.690 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:27:57.690 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:49.555811+0000", 2024-08-19T14:27:57.690 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:27:57.690 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.690 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.690 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:27:57.690 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:49.555811+0000", 2024-08-19T14:27:57.690 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:27:57.690 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.691 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.691 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:27:57.691 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:49.555812+0000", 2024-08-19T14:27:57.691 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.3900000000000005e-07 2024-08-19T14:27:57.691 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.691 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.691 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:27:57.691 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:49.555817+0000", 2024-08-19T14:27:57.691 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.8590000000000004e-06 2024-08-19T14:27:57.691 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.691 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.691 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:27:57.691 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:49.555825+0000", 2024-08-19T14:27:57.691 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.038e-06 2024-08-19T14:27:57.691 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.692 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.692 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:27:57.692 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:49.555831+0000", 2024-08-19T14:27:57.692 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.2439999999999998e-06 2024-08-19T14:27:57.692 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.692 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.692 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:27:57.692 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:49.555905+0000", 2024-08-19T14:27:57.692 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.3854000000000001e-05 2024-08-19T14:27:57.692 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.692 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.693 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:27:57.693 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:49.555928+0000", 2024-08-19T14:27:57.693 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2799999999999999e-05 2024-08-19T14:27:57.693 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.693 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.693 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:27:57.693 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:49.580078+0000", 2024-08-19T14:27:57.693 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.024149886999999998 2024-08-19T14:27:57.693 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.693 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.694 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:27:57.694 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:49.580106+0000", 2024-08-19T14:27:57.694 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.8036000000000001e-05 2024-08-19T14:27:57.694 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:27:57.694 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:27:57.694 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:27:57.694 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.694 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.694 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.25813.0:228246 5.b e491/477 5:dd52a1a4:::benchmark_data_smithi045_85309_object107993:head v 491'14522, mlcod=491'14518)", 2024-08-19T14:27:57.694 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:27:57.286721+0000", 2024-08-19T14:27:57.694 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.297246595, 2024-08-19T14:27:57.694 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.026552952000000001, 2024-08-19T14:27:57.694 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:27:57.695 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:27:57.695 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:27:57.695 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:27:57.695 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.695 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:27:57.695 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:57.286721+0000", 2024-08-19T14:27:57.695 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:27:57.695 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.695 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.695 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:27:57.695 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:57.286721+0000", 2024-08-19T14:27:57.695 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:27:57.695 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.696 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.696 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:27:57.696 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:57.286721+0000", 2024-08-19T14:27:57.696 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.3200000000000004e-07 2024-08-19T14:27:57.696 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.696 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.696 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:27:57.696 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:57.286728+0000", 2024-08-19T14:27:57.696 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.4189999999999999e-06 2024-08-19T14:27:57.696 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.696 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.696 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:27:57.696 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:57.286741+0000", 2024-08-19T14:27:57.697 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3438000000000001e-05 2024-08-19T14:27:57.697 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.697 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.697 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:27:57.697 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:57.286748+0000", 2024-08-19T14:27:57.697 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.0149999999999998e-06 2024-08-19T14:27:57.697 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.697 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.697 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:27:57.697 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:57.289440+0000", 2024-08-19T14:27:57.697 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0026919190000000001 2024-08-19T14:27:57.697 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.697 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.697 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:27:57.698 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:57.289458+0000", 2024-08-19T14:27:57.698 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.755e-05 2024-08-19T14:27:57.698 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.698 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.698 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:27:57.698 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:57.313178+0000", 2024-08-19T14:27:57.698 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.023719994000000001 2024-08-19T14:27:57.698 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:27:57.698 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:27:57.698 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:27:57.698 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:57.313274+0000", 2024-08-19T14:27:57.698 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.5884999999999994e-05 2024-08-19T14:27:57.698 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:27:57.698 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:27:57.699 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:27:57.699 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:27:57.699 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:27:57.699 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:27:57.764 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:27:57.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:57.780 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:57.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:27:57.882 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:27:57.916 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:57.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:57.917 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:27:58.018 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:27:58.045 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:58.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:58.046 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:27:58.147 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:27:58.180 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:58.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:58.181 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:27:58.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:57 smithi032 ceph-mon[20004]: pgmap v1131: 57 pgs: 57 active+clean; 1020 MiB data, 8.9 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 1.24k op/s 2024-08-19T14:27:58.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:57 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1277207249' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num", "val": "26"}]': finished 2024-08-19T14:27:58.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:57 smithi032 ceph-mon[20004]: osdmap e491: 12 total, 12 up, 9 in 2024-08-19T14:27:58.282 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:27:58.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:57 smithi045 ceph-mon[20143]: pgmap v1131: 57 pgs: 57 active+clean; 1020 MiB data, 8.9 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 1.24k op/s 2024-08-19T14:27:58.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:57 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1277207249' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num", "val": "26"}]': finished 2024-08-19T14:27:58.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:57 smithi045 ceph-mon[20143]: osdmap e491: 12 total, 12 up, 9 in 2024-08-19T14:27:58.299 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:57 smithi031 ceph-mon[16336]: pgmap v1131: 57 pgs: 57 active+clean; 1020 MiB data, 8.9 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 1.24k op/s 2024-08-19T14:27:58.299 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:57 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1277207249' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num", "val": "26"}]': finished 2024-08-19T14:27:58.299 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:57 smithi031 ceph-mon[16336]: osdmap e491: 12 total, 12 up, 9 in 2024-08-19T14:27:58.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:58.308 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:58.309 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:27:58.410 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:27:58.441 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:58.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:58.442 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:27:58.469 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:27:58.543 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:27:58.571 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:58.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:58.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:27:58.592 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-08-19T14:27:58.673 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:27:58.723 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:58.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:58.724 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:27:58.825 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:27:58.853 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:58.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:58.854 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:27:58.951 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:27:58.954 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:27:58.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:58.983 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:58.984 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:27:59.085 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:27:59.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:59.116 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:59.116 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:27:59.164 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is set 2024-08-19T14:27:59.217 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:27:59.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:59.249 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:59.249 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:27:59.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:58 smithi032 ceph-mon[20004]: 5.12 scrub starts 2024-08-19T14:27:59.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:58 smithi032 ceph-mon[20004]: 5.12 scrub ok 2024-08-19T14:27:59.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:58 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/726790067' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:27:59.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:58 smithi045 ceph-mon[20143]: 5.12 scrub starts 2024-08-19T14:27:59.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:58 smithi045 ceph-mon[20143]: 5.12 scrub ok 2024-08-19T14:27:59.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:58 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/726790067' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:27:59.349 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:27:59.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:58 smithi031 ceph-mon[16336]: 5.12 scrub starts 2024-08-19T14:27:59.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:59 smithi031 ceph-mon[16336]: 5.12 scrub ok 2024-08-19T14:27:59.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:59 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/726790067' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:27:59.379 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:59.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:59.380 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:27:59.481 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:27:59.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:59.511 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:59.511 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:27:59.575 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:27:59.575 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [], 2024-08-19T14:27:59.575 INFO:teuthology.orchestra.run.smithi045.stdout: "num_ops": 0 2024-08-19T14:27:59.575 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:27:59.612 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:27:59.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:59.644 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:27:59.645 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:27:59.745 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:27:59.780 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:27:59.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:59.781 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:27:59.882 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:27:59.909 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:27:59.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:27:59.910 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:28:00.010 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:28:00.044 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:00.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:00.045 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:28:00.146 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:28:00.178 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:00.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:00.179 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:28:00.221 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:59 smithi045 ceph-mon[20143]: pgmap v1133: 57 pgs: 57 active+clean; 940 MiB data, 8.5 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 1.00k op/s 2024-08-19T14:28:00.221 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:59 smithi045 ceph-mon[20143]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:28:00.221 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:59 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/726790067' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:28:00.221 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:27:59 smithi045 ceph-mon[20143]: osdmap e492: 12 total, 12 up, 9 in 2024-08-19T14:28:00.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:59 smithi032 ceph-mon[20004]: pgmap v1133: 57 pgs: 57 active+clean; 940 MiB data, 8.5 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 1.00k op/s 2024-08-19T14:28:00.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:59 smithi032 ceph-mon[20004]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:28:00.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:59 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/726790067' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:28:00.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:27:59 smithi032 ceph-mon[20004]: osdmap e492: 12 total, 12 up, 9 in 2024-08-19T14:28:00.264 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-08-19T14:28:00.280 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:28:00.310 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:00.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:00.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:28:00.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:59 smithi031 ceph-mon[16336]: pgmap v1133: 57 pgs: 57 active+clean; 940 MiB data, 8.5 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 1.00k op/s 2024-08-19T14:28:00.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:59 smithi031 ceph-mon[16336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:28:00.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:59 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/726790067' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:28:00.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:27:59 smithi031 ceph-mon[16336]: osdmap e492: 12 total, 12 up, 9 in 2024-08-19T14:28:00.412 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:28:00.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:00.441 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:00.441 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:28:00.542 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:28:00.581 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:00.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:00.582 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:28:00.619 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:28:00.666 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:28:00.683 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:28:00.714 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:00.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:00.715 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:28:00.816 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:28:00.851 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:00.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:00.852 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:28:00.953 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:28:01.007 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:01.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:01.008 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:28:01.109 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:28:01.134 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:01.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:01.135 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:28:01.220 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:28:01.220 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:28:01.236 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:28:01.251 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:28:01.251 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [], 2024-08-19T14:28:01.251 INFO:teuthology.orchestra.run.smithi045.stdout: "complaint_time": 30, 2024-08-19T14:28:01.251 INFO:teuthology.orchestra.run.smithi045.stdout: "num_blocked_ops": 0 2024-08-19T14:28:01.251 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:28:01.255 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:28:01.255 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:28:01.269 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:01.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:01.270 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:28:01.282 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:28:01.282 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:28:01.303 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:28:01.303 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:28:01.328 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:28:01.328 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:28:01.355 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:28:01.356 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:28:01.371 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:28:01.386 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:28:01.386 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:28:01.403 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:01.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:01.404 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:28:01.423 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:28:01.423 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:28:01.460 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:28:01.460 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:28:01.491 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:28:01.491 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:28:01.505 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:28:01.523 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:28:01.524 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:28:01.538 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:01.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:01.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:28:01.556 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:28:01.556 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:28:01.640 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:28:01.673 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:01.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:01.674 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:28:01.775 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:28:01.800 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:01.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:01.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:28:01.903 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:28:01.927 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-08-19T14:28:01.947 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:01.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:01.948 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:28:02.050 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:28:02.076 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:02.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:02.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:28:02.178 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:28:02.204 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:02.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:02.205 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:28:02.237 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:01 smithi031 ceph-mon[16336]: pgmap v1135: 57 pgs: 57 active+clean; 757 MiB data, 8.0 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 1.07k op/s 2024-08-19T14:28:02.238 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:01 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3707804229' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:28:02.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:01 smithi032 ceph-mon[20004]: pgmap v1135: 57 pgs: 57 active+clean; 757 MiB data, 8.0 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 1.07k op/s 2024-08-19T14:28:02.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:01 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3707804229' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:28:02.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:01 smithi045 ceph-mon[20143]: pgmap v1135: 57 pgs: 57 active+clean; 757 MiB data, 8.0 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 1.07k op/s 2024-08-19T14:28:02.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:01 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3707804229' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:28:02.305 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:28:02.323 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:28:02.343 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:02.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:02.344 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:28:02.445 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:28:02.472 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:02.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:02.473 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:28:02.574 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:28:02.599 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:02.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:02.600 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:28:02.663 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-19T14:28:02.700 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 7, 8, 9, 11] out_osds: [10, 6, 2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-08-19T14:28:02.700 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-19T14:28:02.700 INFO:tasks.thrashosds.thrasher:Removing osd 9, in_osds are: [0, 1, 3, 4, 5, 7, 8, 9, 11] 2024-08-19T14:28:02.701 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd out 9 2024-08-19T14:28:02.701 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:28:02.739 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:02.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:02.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:28:02.841 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:28:02.870 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:02.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:02.872 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:28:02.972 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:28:03.001 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:03.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:03.002 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:28:03.099 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:28:03.102 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:28:03.151 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:03.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:03.152 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:28:03.156 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:28:03.253 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:28:03.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:03.282 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:03.283 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:28:03.383 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:28:03.411 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:03.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:03.412 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:28:03.513 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:28:03.537 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:03.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:03.539 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:28:03.640 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:28:03.666 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:03.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:03.667 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:28:03.768 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:28:03.801 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:03.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:03.802 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:28:03.903 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:28:03.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:03.936 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:03.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:28:04.037 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:28:04.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:04.066 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:04.067 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:28:04.168 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:28:04.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:04.201 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:04.201 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:28:04.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:03 smithi032 ceph-mon[20004]: pgmap v1136: 57 pgs: 57 active+clean; 640 MiB data, 7.8 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 899 op/s 2024-08-19T14:28:04.281 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:28:04.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:03 smithi045 ceph-mon[20143]: pgmap v1136: 57 pgs: 57 active+clean; 640 MiB data, 7.8 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 899 op/s 2024-08-19T14:28:04.302 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:28:04.315 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:03 smithi031 ceph-mon[16336]: pgmap v1136: 57 pgs: 57 active+clean; 640 MiB data, 7.8 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 899 op/s 2024-08-19T14:28:04.335 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:04.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:04.336 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:28:04.437 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:28:04.469 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:04.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:04.470 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:28:04.471 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:28:04.471 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:28:04.571 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:28:04.602 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:04.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:04.603 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:28:04.704 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:28:04.736 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:04.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:04.737 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:28:04.829 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:04 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mon-a[16332]: 2024-08-19T14:28:04.570+0000 7f2ff503e640 -1 mon.a@0(leader).osd e492 definitely_dead 0 2024-08-19T14:28:04.838 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:28:04.872 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:04.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:04.873 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:28:04.933 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:28:04.974 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:28:05.004 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:05.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:05.005 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:28:05.070 INFO:teuthology.orchestra.run.smithi031.stderr:marked out osd.9. 2024-08-19T14:28:05.071 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is set 2024-08-19T14:28:05.092 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:04 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1378998993' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:28:05.093 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:04 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:28:05.093 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:04 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2264892967' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["9"]}]: dispatch 2024-08-19T14:28:05.093 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:04 smithi031 ceph-mon[16336]: Client client.admin marked osd.9 out, while it was still marked up 2024-08-19T14:28:05.106 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:28:05.133 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:05.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:05.134 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:28:05.235 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:28:05.255 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:05.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:05.256 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:28:05.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:04 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1378998993' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:28:05.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:05 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:28:05.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:05 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2264892967' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["9"]}]: dispatch 2024-08-19T14:28:05.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:05 smithi032 ceph-mon[20004]: Client client.admin marked osd.9 out, while it was still marked up 2024-08-19T14:28:05.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:04 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1378998993' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:28:05.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:04 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:28:05.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:04 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2264892967' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["9"]}]: dispatch 2024-08-19T14:28:05.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:04 smithi045 ceph-mon[20143]: Client client.admin marked osd.9 out, while it was still marked up 2024-08-19T14:28:05.358 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:28:05.379 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:05.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:05.380 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:28:05.475 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-08-19T14:28:05.481 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:28:05.538 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:05.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:05.539 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:28:05.640 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:28:05.662 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:05.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:05.663 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:28:05.722 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:28:05.763 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:28:05.785 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:05.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:05.786 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:28:05.887 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:28:05.908 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:05.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:05.909 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:28:06.010 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:28:06.031 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:06.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:06.032 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:28:06.111 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:05 smithi045 ceph-mon[20143]: pgmap v1137: 57 pgs: 57 active+clean; 381 MiB data, 6.9 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 1.28k op/s 2024-08-19T14:28:06.111 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:05 smithi045 ceph-mon[20143]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:28:06.111 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:05 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:28:06.111 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:05 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2264892967' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["9"]}]': finished 2024-08-19T14:28:06.111 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:05 smithi045 ceph-mon[20143]: osdmap e493: 12 total, 12 up, 8 in 2024-08-19T14:28:06.132 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:28:06.158 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:06.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:06.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:28:06.211 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:28:06.211 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:28:06.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:05 smithi032 ceph-mon[20004]: pgmap v1137: 57 pgs: 57 active+clean; 381 MiB data, 6.9 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 1.28k op/s 2024-08-19T14:28:06.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:05 smithi032 ceph-mon[20004]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:28:06.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:05 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:28:06.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:05 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2264892967' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["9"]}]': finished 2024-08-19T14:28:06.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:05 smithi032 ceph-mon[20004]: osdmap e493: 12 total, 12 up, 8 in 2024-08-19T14:28:06.260 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:28:06.271 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:05 smithi031 ceph-mon[16336]: pgmap v1137: 57 pgs: 57 active+clean; 381 MiB data, 6.9 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 1.28k op/s 2024-08-19T14:28:06.271 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:05 smithi031 ceph-mon[16336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:28:06.271 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:05 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:28:06.271 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:05 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2264892967' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["9"]}]': finished 2024-08-19T14:28:06.271 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:05 smithi031 ceph-mon[16336]: osdmap e493: 12 total, 12 up, 8 in 2024-08-19T14:28:06.281 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:06.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:06.282 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:28:06.383 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:28:06.404 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:06.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:06.405 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:28:06.507 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:28:06.532 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:06.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:06.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:28:06.633 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:28:06.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:06.655 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:06.655 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:28:06.756 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:28:06.778 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:06.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:06.779 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:28:06.842 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:28:06.842 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:28:06.858 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:28:06.859 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:28:06.864 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-08-19T14:28:06.875 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:28:06.875 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:28:06.880 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:28:06.891 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:28:06.891 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:28:06.905 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:06.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:06.906 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:28:06.908 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:28:06.908 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:28:06.925 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:28:06.925 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:28:06.942 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:28:06.942 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:28:06.959 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:28:06.960 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:28:06.978 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:28:06.978 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:28:06.995 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:28:06.995 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:28:07.007 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:28:07.013 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:28:07.013 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:28:07.028 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:07.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:07.029 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:28:07.030 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:28:07.030 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:28:07.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:07 smithi031 ceph-mon[16336]: pgmap v1139: 57 pgs: 3 peering, 54 active+clean; 296 MiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 1.29k op/s 2024-08-19T14:28:07.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:07 smithi031 ceph-mon[16336]: osdmap e494: 12 total, 12 up, 8 in 2024-08-19T14:28:07.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:07 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2747576424' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:28:07.130 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:28:07.137 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:28:07.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:07.162 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:07.162 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:28:07.249 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:07 smithi045 ceph-mon[20143]: pgmap v1139: 57 pgs: 3 peering, 54 active+clean; 296 MiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 1.29k op/s 2024-08-19T14:28:07.250 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:07 smithi045 ceph-mon[20143]: osdmap e494: 12 total, 12 up, 8 in 2024-08-19T14:28:07.250 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:07 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2747576424' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:28:07.263 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:28:07.297 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:07.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:07.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:28:07.399 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:28:07.455 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:07.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:07.456 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:28:07.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:07 smithi032 ceph-mon[20004]: pgmap v1139: 57 pgs: 3 peering, 54 active+clean; 296 MiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 1.29k op/s 2024-08-19T14:28:07.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:07 smithi032 ceph-mon[20004]: osdmap e494: 12 total, 12 up, 8 in 2024-08-19T14:28:07.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:07 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2747576424' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:28:07.557 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:28:07.585 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:07.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:07.586 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:28:07.687 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:28:07.711 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:07.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:07.712 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:28:07.754 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:28:07.754 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [], 2024-08-19T14:28:07.754 INFO:teuthology.orchestra.run.smithi045.stdout: "complaint_time": 30, 2024-08-19T14:28:07.754 INFO:teuthology.orchestra.run.smithi045.stdout: "num_blocked_ops": 0 2024-08-19T14:28:07.754 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:28:07.813 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:28:07.838 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:07.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:07.839 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:28:07.940 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:28:07.967 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:07.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:07.967 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:28:08.068 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:28:08.100 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:08.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:08.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:28:08.202 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:28:08.227 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:08.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:08.228 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:28:08.329 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:28:08.338 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:08 smithi031 ceph-mon[16336]: osdmap e495: 12 total, 12 up, 8 in 2024-08-19T14:28:08.338 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:08 smithi031 ceph-mon[16336]: 5.e starting backfill to osd.4 from (0'0,0'0] MAX to 492'14887 2024-08-19T14:28:08.338 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:08 smithi031 ceph-mon[16336]: 5.b starting backfill to osd.7 from (0'0,0'0] MAX to 492'15127 2024-08-19T14:28:08.338 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:08 smithi031 ceph-mon[16336]: osdmap e496: 12 total, 12 up, 8 in 2024-08-19T14:28:08.350 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:08.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:08.351 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:28:08.434 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:08 smithi045 ceph-mon[20143]: osdmap e495: 12 total, 12 up, 8 in 2024-08-19T14:28:08.434 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:08 smithi045 ceph-mon[20143]: 5.e starting backfill to osd.4 from (0'0,0'0] MAX to 492'14887 2024-08-19T14:28:08.434 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:08 smithi045 ceph-mon[20143]: 5.b starting backfill to osd.7 from (0'0,0'0] MAX to 492'15127 2024-08-19T14:28:08.434 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:08 smithi045 ceph-mon[20143]: osdmap e496: 12 total, 12 up, 8 in 2024-08-19T14:28:08.452 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:28:08.463 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-08-19T14:28:08.474 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:08.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:08.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:28:08.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:08 smithi032 ceph-mon[20004]: osdmap e495: 12 total, 12 up, 8 in 2024-08-19T14:28:08.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:08 smithi032 ceph-mon[20004]: 5.e starting backfill to osd.4 from (0'0,0'0] MAX to 492'14887 2024-08-19T14:28:08.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:08 smithi032 ceph-mon[20004]: 5.b starting backfill to osd.7 from (0'0,0'0] MAX to 492'15127 2024-08-19T14:28:08.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:08 smithi032 ceph-mon[20004]: osdmap e496: 12 total, 12 up, 8 in 2024-08-19T14:28:08.559 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-19T14:28:08.576 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:28:08.597 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:08.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:08.598 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:28:08.700 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:28:08.722 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:28:08.722 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:08.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:08.723 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:28:08.824 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:28:08.851 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:28:08.854 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:08.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:08.856 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:28:08.957 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:28:08.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:08.999 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:08.999 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:28:09.100 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:28:09.127 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:09.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:09.128 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:28:09.229 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:28:09.256 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:09.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:09.257 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:28:09.266 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:28:09.266 INFO:teuthology.orchestra.run.smithi045.stdout: "size": 20, 2024-08-19T14:28:09.266 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 600, 2024-08-19T14:28:09.266 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [ 2024-08-19T14:28:09.266 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.266 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:22707 4.5 4:aa5c4d32:::benchmark_data_smithi045_71562_object22706:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-08-19T14:28:09.266 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:20:41.355284+0000", 2024-08-19T14:28:09.266 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 447.908739932, 2024-08-19T14:28:09.266 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.32540933900000002, 2024-08-19T14:28:09.266 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:28:09.266 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:28:09.267 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:28:09.267 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:28:09.267 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:28:09.267 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:28:09.267 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 22707 2024-08-19T14:28:09.267 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.267 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:28:09.267 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.267 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:28:09.267 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.355284+0000", 2024-08-19T14:28:09.267 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:28:09.267 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.267 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.267 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:28:09.268 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.355284+0000", 2024-08-19T14:28:09.268 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:28:09.268 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.268 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.268 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:28:09.268 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.355285+0000", 2024-08-19T14:28:09.268 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.2699999999999998e-07 2024-08-19T14:28:09.268 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.268 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.268 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:28:09.268 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.355304+0000", 2024-08-19T14:28:09.268 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8834e-05 2024-08-19T14:28:09.269 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.269 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.269 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:28:09.269 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.355315+0000", 2024-08-19T14:28:09.269 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1388e-05 2024-08-19T14:28:09.269 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.269 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.269 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:28:09.269 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.355326+0000", 2024-08-19T14:28:09.269 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0101999999999999e-05 2024-08-19T14:28:09.269 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.269 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.270 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:28:09.270 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.355381+0000", 2024-08-19T14:28:09.270 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.5114999999999997e-05 2024-08-19T14:28:09.270 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.270 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.270 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:28:09.270 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.355481+0000", 2024-08-19T14:28:09.270 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000100448 2024-08-19T14:28:09.270 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.270 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.270 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,7", 2024-08-19T14:28:09.270 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.355563+0000", 2024-08-19T14:28:09.270 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.1891999999999995e-05 2024-08-19T14:28:09.270 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.271 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.271 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:28:09.271 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.356429+0000", 2024-08-19T14:28:09.271 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00086571700000000001 2024-08-19T14:28:09.271 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.271 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.271 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:09.271 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.356898+0000", 2024-08-19T14:28:09.271 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000469171 2024-08-19T14:28:09.271 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.271 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.271 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:09.271 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.680624+0000", 2024-08-19T14:28:09.272 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.32372564599999998 2024-08-19T14:28:09.272 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.272 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.272 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:28:09.272 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.680661+0000", 2024-08-19T14:28:09.272 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.7589999999999998e-05 2024-08-19T14:28:09.272 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.272 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.272 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:28:09.272 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.680694+0000", 2024-08-19T14:28:09.272 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.2509000000000003e-05 2024-08-19T14:28:09.272 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:28:09.272 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:28:09.272 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:28:09.273 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.273 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.273 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:22723 4.5 4:a03b0b8c:::benchmark_data_smithi045_71562_object22722:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-08-19T14:28:09.273 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:20:41.360250+0000", 2024-08-19T14:28:09.273 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 447.90377459899997, 2024-08-19T14:28:09.273 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.32180027, 2024-08-19T14:28:09.273 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:28:09.273 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:28:09.273 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:28:09.273 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:28:09.273 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:28:09.273 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:28:09.273 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 22723 2024-08-19T14:28:09.273 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.274 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:28:09.274 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.274 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:28:09.274 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.360250+0000", 2024-08-19T14:28:09.274 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:28:09.274 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.274 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.274 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:28:09.274 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.360250+0000", 2024-08-19T14:28:09.274 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:28:09.274 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.274 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.274 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:28:09.274 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.360251+0000", 2024-08-19T14:28:09.275 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7740000000000001e-06 2024-08-19T14:28:09.275 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.275 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.275 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:28:09.275 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.360271+0000", 2024-08-19T14:28:09.275 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9755999999999999e-05 2024-08-19T14:28:09.275 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.275 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.275 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:28:09.275 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.360280+0000", 2024-08-19T14:28:09.275 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.268e-06 2024-08-19T14:28:09.275 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.275 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.275 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:28:09.276 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.360286+0000", 2024-08-19T14:28:09.276 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2700000000000004e-06 2024-08-19T14:28:09.276 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.276 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.276 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:28:09.276 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.360339+0000", 2024-08-19T14:28:09.276 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.3644999999999999e-05 2024-08-19T14:28:09.276 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.276 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.276 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:28:09.276 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.360424+0000", 2024-08-19T14:28:09.276 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.4493999999999994e-05 2024-08-19T14:28:09.276 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.276 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.277 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,7", 2024-08-19T14:28:09.277 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.360529+0000", 2024-08-19T14:28:09.277 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00010503200000000001 2024-08-19T14:28:09.277 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.277 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.277 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:28:09.277 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.362147+0000", 2024-08-19T14:28:09.277 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001617718 2024-08-19T14:28:09.277 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.277 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.277 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:09.277 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.362188+0000", 2024-08-19T14:28:09.277 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.1553e-05 2024-08-19T14:28:09.277 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.278 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.278 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:09.278 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.681950+0000", 2024-08-19T14:28:09.278 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.31976217299999998 2024-08-19T14:28:09.278 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.278 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.278 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:28:09.278 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.682008+0000", 2024-08-19T14:28:09.278 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7494000000000002e-05 2024-08-19T14:28:09.278 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.278 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.278 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:28:09.278 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.682050+0000", 2024-08-19T14:28:09.278 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.2092999999999998e-05 2024-08-19T14:28:09.279 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:28:09.279 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:28:09.279 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:28:09.279 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.279 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.279 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:22725 4.5 4:adb3fee7:::benchmark_data_smithi045_71562_object22724:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-08-19T14:28:09.279 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:20:41.361334+0000", 2024-08-19T14:28:09.279 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 447.90269062499999, 2024-08-19T14:28:09.279 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.32179169200000002, 2024-08-19T14:28:09.279 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:28:09.279 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:28:09.279 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:28:09.279 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:28:09.280 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:28:09.280 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:28:09.280 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 22725 2024-08-19T14:28:09.280 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.280 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:28:09.280 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.281 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:28:09.281 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.361334+0000", 2024-08-19T14:28:09.281 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:28:09.281 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.281 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.281 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:28:09.281 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.361334+0000", 2024-08-19T14:28:09.281 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:28:09.281 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.281 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.281 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:28:09.281 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.361334+0000", 2024-08-19T14:28:09.281 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.61e-07 2024-08-19T14:28:09.281 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.282 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.282 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:28:09.282 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.361353+0000", 2024-08-19T14:28:09.282 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9074999999999999e-05 2024-08-19T14:28:09.282 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.282 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.282 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:28:09.282 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.361365+0000", 2024-08-19T14:28:09.282 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1498000000000001e-05 2024-08-19T14:28:09.282 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.282 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.282 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:28:09.283 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.361371+0000", 2024-08-19T14:28:09.283 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.1290000000000001e-06 2024-08-19T14:28:09.283 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.283 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.283 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:28:09.283 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.361415+0000", 2024-08-19T14:28:09.283 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.3445999999999997e-05 2024-08-19T14:28:09.283 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.283 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.283 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:28:09.283 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.361542+0000", 2024-08-19T14:28:09.283 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012710400000000001 2024-08-19T14:28:09.283 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.283 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.284 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,7", 2024-08-19T14:28:09.284 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.361632+0000", 2024-08-19T14:28:09.284 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.9963999999999994e-05 2024-08-19T14:28:09.284 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.284 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.284 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:28:09.284 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.362288+0000", 2024-08-19T14:28:09.284 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00065625500000000001 2024-08-19T14:28:09.284 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.284 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.284 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:09.284 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.362699+0000", 2024-08-19T14:28:09.284 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00041120600000000001 2024-08-19T14:28:09.284 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.285 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.285 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:09.285 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.683027+0000", 2024-08-19T14:28:09.285 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.32032799699999998 2024-08-19T14:28:09.285 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.285 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.285 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:28:09.285 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.683084+0000", 2024-08-19T14:28:09.285 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7054e-05 2024-08-19T14:28:09.285 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.285 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.285 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:28:09.285 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.683125+0000", 2024-08-19T14:28:09.286 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.1202999999999998e-05 2024-08-19T14:28:09.286 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:28:09.286 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:28:09.286 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:28:09.286 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.286 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.286 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:116285 4.8 4:15b9fca8:::benchmark_data_smithi045_71562_object116284:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-08-19T14:28:09.286 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:21:48.600383+0000", 2024-08-19T14:28:09.286 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 380.66364138400002, 2024-08-19T14:28:09.286 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.25002218100000001, 2024-08-19T14:28:09.286 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:28:09.286 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:28:09.286 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:28:09.287 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:28:09.287 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:28:09.287 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:28:09.287 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 116285 2024-08-19T14:28:09.287 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.287 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:28:09.287 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.287 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:28:09.287 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.600383+0000", 2024-08-19T14:28:09.287 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:28:09.287 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.287 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.287 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:28:09.287 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.600383+0000", 2024-08-19T14:28:09.288 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:28:09.288 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.288 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.288 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:28:09.288 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.600384+0000", 2024-08-19T14:28:09.288 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.3200000000000003e-07 2024-08-19T14:28:09.288 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.288 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.288 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:28:09.288 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.600403+0000", 2024-08-19T14:28:09.288 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9326000000000001e-05 2024-08-19T14:28:09.288 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.288 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.289 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:28:09.289 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.600414+0000", 2024-08-19T14:28:09.289 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1129e-05 2024-08-19T14:28:09.289 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.289 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.289 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:28:09.289 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.600421+0000", 2024-08-19T14:28:09.289 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.8589999999999999e-06 2024-08-19T14:28:09.289 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.289 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.289 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:28:09.289 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.600478+0000", 2024-08-19T14:28:09.289 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.6502000000000002e-05 2024-08-19T14:28:09.289 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.289 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.290 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:28:09.290 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.600573+0000", 2024-08-19T14:28:09.290 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.5165000000000001e-05 2024-08-19T14:28:09.290 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.290 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.290 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 7,9", 2024-08-19T14:28:09.290 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.600654+0000", 2024-08-19T14:28:09.290 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.1002999999999997e-05 2024-08-19T14:28:09.290 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.290 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.290 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:28:09.290 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.607545+0000", 2024-08-19T14:28:09.290 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0068915039999999997 2024-08-19T14:28:09.291 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.291 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.291 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:09.291 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.610300+0000", 2024-08-19T14:28:09.291 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0027545199999999999 2024-08-19T14:28:09.291 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.291 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.291 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:09.291 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.850322+0000", 2024-08-19T14:28:09.291 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.24002195200000001 2024-08-19T14:28:09.291 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.291 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.291 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:28:09.291 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.850364+0000", 2024-08-19T14:28:09.292 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.2582e-05 2024-08-19T14:28:09.292 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.292 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.292 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:28:09.292 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.850405+0000", 2024-08-19T14:28:09.292 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.0707000000000002e-05 2024-08-19T14:28:09.292 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:28:09.292 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:28:09.292 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:28:09.292 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.292 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.292 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:116286 4.8 4:198dc556:::benchmark_data_smithi045_71562_object116285:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-08-19T14:28:09.292 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:21:48.600439+0000", 2024-08-19T14:28:09.293 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 380.66358535699999, 2024-08-19T14:28:09.293 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.25012546899999999, 2024-08-19T14:28:09.293 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:28:09.293 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:28:09.293 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:28:09.293 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:28:09.293 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:28:09.293 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:28:09.293 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 116286 2024-08-19T14:28:09.293 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.293 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:28:09.293 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.293 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:28:09.293 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.600439+0000", 2024-08-19T14:28:09.293 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:28:09.294 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.294 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.294 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:28:09.294 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.600439+0000", 2024-08-19T14:28:09.294 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:28:09.294 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.294 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.294 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:28:09.294 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.600439+0000", 2024-08-19T14:28:09.294 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2900000000000004e-07 2024-08-19T14:28:09.294 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.294 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.295 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:28:09.295 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.600458+0000", 2024-08-19T14:28:09.295 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8729999999999999e-05 2024-08-19T14:28:09.295 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.295 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.295 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:28:09.295 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.600466+0000", 2024-08-19T14:28:09.295 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.1030000000000002e-06 2024-08-19T14:28:09.295 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.295 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.295 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:28:09.295 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.600472+0000", 2024-08-19T14:28:09.295 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4380000000000003e-06 2024-08-19T14:28:09.295 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.295 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.296 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:28:09.296 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.603745+0000", 2024-08-19T14:28:09.296 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0032732350000000002 2024-08-19T14:28:09.296 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.296 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.296 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:28:09.296 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.603905+0000", 2024-08-19T14:28:09.296 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015985500000000001 2024-08-19T14:28:09.296 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.296 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.296 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 7,9", 2024-08-19T14:28:09.296 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.604060+0000", 2024-08-19T14:28:09.296 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015485799999999999 2024-08-19T14:28:09.296 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.297 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.297 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:28:09.297 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.607871+0000", 2024-08-19T14:28:09.297 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0038114540000000001 2024-08-19T14:28:09.297 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.297 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.297 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:09.297 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.610365+0000", 2024-08-19T14:28:09.297 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0024943449999999998 2024-08-19T14:28:09.297 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.297 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.297 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:09.297 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.850477+0000", 2024-08-19T14:28:09.297 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.24011110099999999 2024-08-19T14:28:09.297 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.298 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.298 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:28:09.298 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.850528+0000", 2024-08-19T14:28:09.298 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1684999999999997e-05 2024-08-19T14:28:09.298 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.298 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.298 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:28:09.298 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.850564+0000", 2024-08-19T14:28:09.298 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.6136000000000001e-05 2024-08-19T14:28:09.298 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:28:09.298 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:28:09.298 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:28:09.299 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.299 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.299 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:116297 4.8 4:17724d94:::benchmark_data_smithi045_71562_object116296:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-08-19T14:28:09.299 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:21:48.612374+0000", 2024-08-19T14:28:09.299 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 380.65165028000001, 2024-08-19T14:28:09.299 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.28029534299999997, 2024-08-19T14:28:09.299 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:28:09.299 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:28:09.299 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:28:09.299 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:28:09.299 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:28:09.299 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:28:09.299 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 116297 2024-08-19T14:28:09.299 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.299 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:28:09.300 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.300 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:28:09.300 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.612374+0000", 2024-08-19T14:28:09.300 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:28:09.300 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.300 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.300 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:28:09.300 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.612374+0000", 2024-08-19T14:28:09.300 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:28:09.300 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.300 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.300 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:28:09.300 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.612375+0000", 2024-08-19T14:28:09.301 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2869999999999999e-06 2024-08-19T14:28:09.301 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.301 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.301 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:28:09.301 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.612393+0000", 2024-08-19T14:28:09.301 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7864e-05 2024-08-19T14:28:09.301 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.301 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.301 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:28:09.301 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.612404+0000", 2024-08-19T14:28:09.301 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1377e-05 2024-08-19T14:28:09.301 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.301 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.302 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:28:09.302 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.612410+0000", 2024-08-19T14:28:09.302 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.3730000000000001e-06 2024-08-19T14:28:09.302 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.302 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.302 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:28:09.302 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.612460+0000", 2024-08-19T14:28:09.302 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.9675999999999997e-05 2024-08-19T14:28:09.302 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.302 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.302 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:28:09.302 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.612561+0000", 2024-08-19T14:28:09.302 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00010157099999999999 2024-08-19T14:28:09.302 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.303 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.303 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 7,9", 2024-08-19T14:28:09.303 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.612676+0000", 2024-08-19T14:28:09.303 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00011491 2024-08-19T14:28:09.303 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.303 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.303 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:09.303 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.616157+0000", 2024-08-19T14:28:09.303 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00348073 2024-08-19T14:28:09.303 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.303 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.303 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:28:09.303 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.616249+0000", 2024-08-19T14:28:09.303 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.2085999999999999e-05 2024-08-19T14:28:09.304 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.304 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.304 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:09.304 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.850604+0000", 2024-08-19T14:28:09.304 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.23435562900000001 2024-08-19T14:28:09.304 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.304 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.304 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:28:09.304 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.892613+0000", 2024-08-19T14:28:09.304 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.042008839999999999 2024-08-19T14:28:09.304 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.304 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.304 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:28:09.305 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.892669+0000", 2024-08-19T14:28:09.305 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.5999999999999999e-05 2024-08-19T14:28:09.305 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:28:09.305 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:28:09.305 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:28:09.305 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.305 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.305 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:116299 4.3 4:c0da230c:::benchmark_data_smithi045_71562_object116298:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-08-19T14:28:09.305 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:21:48.612978+0000", 2024-08-19T14:28:09.305 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 380.65104599099999, 2024-08-19T14:28:09.305 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.23536219899999999, 2024-08-19T14:28:09.305 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:28:09.305 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:28:09.305 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:28:09.306 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:28:09.306 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:28:09.306 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:28:09.306 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 116299 2024-08-19T14:28:09.306 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.306 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:28:09.306 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.306 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:28:09.306 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.612978+0000", 2024-08-19T14:28:09.306 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:28:09.306 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.306 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.306 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:28:09.307 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.612978+0000", 2024-08-19T14:28:09.307 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:28:09.307 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.307 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.307 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:28:09.307 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.612979+0000", 2024-08-19T14:28:09.307 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.7599999999999996e-07 2024-08-19T14:28:09.307 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.307 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.307 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:28:09.307 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.612998+0000", 2024-08-19T14:28:09.307 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8527000000000001e-05 2024-08-19T14:28:09.307 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.307 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.308 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:28:09.308 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.613008+0000", 2024-08-19T14:28:09.308 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.079e-05 2024-08-19T14:28:09.308 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.308 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.308 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:28:09.308 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.613014+0000", 2024-08-19T14:28:09.308 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4210000000000003e-06 2024-08-19T14:28:09.308 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.308 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.308 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:28:09.308 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.613062+0000", 2024-08-19T14:28:09.308 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.8674999999999997e-05 2024-08-19T14:28:09.308 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.309 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.309 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:28:09.309 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.613200+0000", 2024-08-19T14:28:09.309 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013715400000000001 2024-08-19T14:28:09.309 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.309 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.309 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,7", 2024-08-19T14:28:09.309 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.613317+0000", 2024-08-19T14:28:09.309 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00011734399999999999 2024-08-19T14:28:09.309 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.309 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.309 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:28:09.309 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.616326+0000", 2024-08-19T14:28:09.309 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0030090450000000001 2024-08-19T14:28:09.310 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.310 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.310 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:09.310 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.621117+0000", 2024-08-19T14:28:09.310 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0047912160000000001 2024-08-19T14:28:09.310 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.310 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.310 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:09.310 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.848233+0000", 2024-08-19T14:28:09.310 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.22711604099999999 2024-08-19T14:28:09.310 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.310 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.310 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:28:09.311 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.848288+0000", 2024-08-19T14:28:09.311 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4972999999999997e-05 2024-08-19T14:28:09.311 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.311 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.311 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:28:09.311 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.848340+0000", 2024-08-19T14:28:09.311 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2237000000000001e-05 2024-08-19T14:28:09.311 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:28:09.311 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:28:09.311 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:28:09.311 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.311 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.311 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:116311 4.8 4:16b96ac4:::benchmark_data_smithi045_71562_object116310:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-08-19T14:28:09.311 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:21:48.627253+0000", 2024-08-19T14:28:09.312 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 380.63677104200002, 2024-08-19T14:28:09.312 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.26561561299999997, 2024-08-19T14:28:09.312 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:28:09.312 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:28:09.312 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:28:09.312 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:28:09.312 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:28:09.312 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:28:09.312 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 116311 2024-08-19T14:28:09.312 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.312 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:28:09.312 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.312 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:28:09.312 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.627253+0000", 2024-08-19T14:28:09.313 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:28:09.313 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.313 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.313 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:28:09.314 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.627253+0000", 2024-08-19T14:28:09.314 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:28:09.314 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.314 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.314 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:28:09.314 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.627254+0000", 2024-08-19T14:28:09.314 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4000000000000001e-07 2024-08-19T14:28:09.314 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.314 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.314 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:28:09.314 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.627293+0000", 2024-08-19T14:28:09.314 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.8742000000000003e-05 2024-08-19T14:28:09.315 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.315 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.315 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:28:09.315 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.627302+0000", 2024-08-19T14:28:09.315 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.2639999999999997e-06 2024-08-19T14:28:09.315 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.315 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.315 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:28:09.315 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.627307+0000", 2024-08-19T14:28:09.315 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0209999999999999e-06 2024-08-19T14:28:09.315 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.315 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.315 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:28:09.315 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.627378+0000", 2024-08-19T14:28:09.316 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.0529000000000002e-05 2024-08-19T14:28:09.316 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.316 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.316 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:28:09.316 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.627458+0000", 2024-08-19T14:28:09.316 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.0259999999999994e-05 2024-08-19T14:28:09.316 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.316 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.316 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 7,9", 2024-08-19T14:28:09.316 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.627624+0000", 2024-08-19T14:28:09.316 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00016623300000000001 2024-08-19T14:28:09.316 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.316 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.316 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:09.317 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.631383+0000", 2024-08-19T14:28:09.317 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0037587599999999999 2024-08-19T14:28:09.317 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.317 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.317 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:28:09.317 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.631513+0000", 2024-08-19T14:28:09.317 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013022199999999999 2024-08-19T14:28:09.317 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.317 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.317 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:09.317 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.892784+0000", 2024-08-19T14:28:09.317 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.26127100399999997 2024-08-19T14:28:09.317 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.317 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.318 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:28:09.318 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.892832+0000", 2024-08-19T14:28:09.318 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.7970999999999999e-05 2024-08-19T14:28:09.318 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.318 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.318 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:28:09.318 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.892869+0000", 2024-08-19T14:28:09.318 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.6866999999999998e-05 2024-08-19T14:28:09.318 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:28:09.318 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:28:09.318 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:28:09.318 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.318 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.318 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:116334 4.8 4:1f777a70:::benchmark_data_smithi045_71562_object116333:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-08-19T14:28:09.319 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:21:48.650934+0000", 2024-08-19T14:28:09.319 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 380.61309033499998, 2024-08-19T14:28:09.319 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.242080929, 2024-08-19T14:28:09.319 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:28:09.319 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:28:09.319 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:28:09.319 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:28:09.319 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:28:09.319 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:28:09.319 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 116334 2024-08-19T14:28:09.319 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.319 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:28:09.319 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.319 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:28:09.320 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.650934+0000", 2024-08-19T14:28:09.320 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:28:09.320 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.320 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.320 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:28:09.320 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.650934+0000", 2024-08-19T14:28:09.320 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:28:09.320 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.320 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.320 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:28:09.320 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.650935+0000", 2024-08-19T14:28:09.320 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.26e-06 2024-08-19T14:28:09.320 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.320 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.321 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:28:09.321 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.650954+0000", 2024-08-19T14:28:09.321 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9000000000000001e-05 2024-08-19T14:28:09.321 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.321 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.321 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:28:09.321 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.650964+0000", 2024-08-19T14:28:09.321 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0253e-05 2024-08-19T14:28:09.321 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.321 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.321 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:28:09.321 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.650970+0000", 2024-08-19T14:28:09.321 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1499999999999998e-06 2024-08-19T14:28:09.321 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.322 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.322 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:28:09.322 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.651014+0000", 2024-08-19T14:28:09.322 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.4391000000000002e-05 2024-08-19T14:28:09.322 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.322 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.323 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:28:09.323 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.651134+0000", 2024-08-19T14:28:09.323 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000119535 2024-08-19T14:28:09.323 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.323 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.324 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 7,9", 2024-08-19T14:28:09.324 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.651262+0000", 2024-08-19T14:28:09.324 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012826100000000001 2024-08-19T14:28:09.324 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.324 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.324 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:28:09.324 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.654276+0000", 2024-08-19T14:28:09.324 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0030142839999999999 2024-08-19T14:28:09.324 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.324 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.324 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:09.324 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.655021+0000", 2024-08-19T14:28:09.324 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00074536299999999997 2024-08-19T14:28:09.324 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.325 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.325 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:09.325 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.892914+0000", 2024-08-19T14:28:09.325 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.23789228400000001 2024-08-19T14:28:09.325 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.325 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.325 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:28:09.325 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.892980+0000", 2024-08-19T14:28:09.325 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.6193999999999997e-05 2024-08-19T14:28:09.325 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.325 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.325 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:28:09.325 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.893015+0000", 2024-08-19T14:28:09.325 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.4953999999999998e-05 2024-08-19T14:28:09.326 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:28:09.326 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:28:09.326 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:28:09.326 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.326 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.326 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:165636 4.3 4:c997d720:::benchmark_data_smithi045_71562_object34461:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-08-19T14:28:09.326 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:22:42.884498+0000", 2024-08-19T14:28:09.326 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 326.37952640899999, 2024-08-19T14:28:09.326 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.22208514900000001, 2024-08-19T14:28:09.326 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:28:09.326 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:28:09.326 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:28:09.326 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:28:09.327 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:28:09.327 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:28:09.327 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 165636 2024-08-19T14:28:09.327 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.327 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:28:09.327 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.327 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:28:09.327 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.884498+0000", 2024-08-19T14:28:09.327 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:28:09.327 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.327 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.327 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:28:09.327 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.884498+0000", 2024-08-19T14:28:09.328 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:28:09.328 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.328 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.328 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:28:09.328 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.884499+0000", 2024-08-19T14:28:09.328 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.238e-06 2024-08-19T14:28:09.328 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.328 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.328 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:28:09.328 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.884504+0000", 2024-08-19T14:28:09.328 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2959999999999998e-06 2024-08-19T14:28:09.328 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.328 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.328 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:28:09.329 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.884515+0000", 2024-08-19T14:28:09.329 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0995999999999999e-05 2024-08-19T14:28:09.329 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.329 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.329 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:28:09.329 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.884523+0000", 2024-08-19T14:28:09.329 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.294e-06 2024-08-19T14:28:09.329 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.329 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.329 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:28:09.329 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.884580+0000", 2024-08-19T14:28:09.329 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7224e-05 2024-08-19T14:28:09.329 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.329 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.330 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:28:09.330 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.884704+0000", 2024-08-19T14:28:09.330 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000124046 2024-08-19T14:28:09.330 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.330 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.330 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,7", 2024-08-19T14:28:09.330 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.884836+0000", 2024-08-19T14:28:09.330 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000132478 2024-08-19T14:28:09.330 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.330 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.330 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:28:09.330 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.888660+0000", 2024-08-19T14:28:09.330 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0038240399999999999 2024-08-19T14:28:09.331 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.331 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.331 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:09.331 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.889616+0000", 2024-08-19T14:28:09.331 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00095516899999999996 2024-08-19T14:28:09.331 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.331 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.331 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:09.331 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:43.106482+0000", 2024-08-19T14:28:09.331 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.21686655299999999 2024-08-19T14:28:09.331 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.331 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.331 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:28:09.331 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:43.106539+0000", 2024-08-19T14:28:09.332 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.6366999999999998e-05 2024-08-19T14:28:09.332 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.332 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.332 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:28:09.332 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:43.106583+0000", 2024-08-19T14:28:09.332 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.4447999999999999e-05 2024-08-19T14:28:09.332 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:28:09.332 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:28:09.332 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:28:09.332 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.332 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.332 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:165645 4.3 4:c4ef6944:::benchmark_data_smithi045_71562_object34470:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-08-19T14:28:09.332 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:22:42.892140+0000", 2024-08-19T14:28:09.332 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 326.37188383400002, 2024-08-19T14:28:09.333 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.223849192, 2024-08-19T14:28:09.333 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:28:09.333 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:28:09.333 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:28:09.333 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:28:09.333 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:28:09.333 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:28:09.333 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 165645 2024-08-19T14:28:09.333 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.333 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:28:09.333 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.333 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:28:09.333 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.892140+0000", 2024-08-19T14:28:09.333 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:28:09.334 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.334 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.334 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:28:09.334 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.892140+0000", 2024-08-19T14:28:09.334 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:28:09.334 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.334 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.334 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:28:09.334 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.892142+0000", 2024-08-19T14:28:09.334 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.127e-06 2024-08-19T14:28:09.334 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.334 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.334 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:28:09.334 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.892146+0000", 2024-08-19T14:28:09.335 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.7949999999999998e-06 2024-08-19T14:28:09.335 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.335 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.335 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:28:09.335 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.892157+0000", 2024-08-19T14:28:09.335 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0387000000000001e-05 2024-08-19T14:28:09.335 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.335 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.335 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:28:09.335 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.892162+0000", 2024-08-19T14:28:09.336 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.6269999999999997e-06 2024-08-19T14:28:09.336 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.336 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.336 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:28:09.336 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.892203+0000", 2024-08-19T14:28:09.336 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.0238999999999999e-05 2024-08-19T14:28:09.336 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.336 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.336 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:28:09.336 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.892284+0000", 2024-08-19T14:28:09.336 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.1198000000000004e-05 2024-08-19T14:28:09.336 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.337 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.337 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,7", 2024-08-19T14:28:09.337 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.892359+0000", 2024-08-19T14:28:09.337 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4783000000000006e-05 2024-08-19T14:28:09.337 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.337 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.337 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:28:09.337 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.895461+0000", 2024-08-19T14:28:09.337 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0031028869999999999 2024-08-19T14:28:09.338 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.338 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.338 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:09.338 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.897428+0000", 2024-08-19T14:28:09.338 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0019669069999999999 2024-08-19T14:28:09.338 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.338 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.338 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:09.338 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:43.115899+0000", 2024-08-19T14:28:09.338 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.21847051100000001 2024-08-19T14:28:09.338 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.338 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.338 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:28:09.339 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:43.115943+0000", 2024-08-19T14:28:09.339 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.4181000000000003e-05 2024-08-19T14:28:09.339 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.339 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.339 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:28:09.339 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:43.115990+0000", 2024-08-19T14:28:09.339 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.655e-05 2024-08-19T14:28:09.339 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:28:09.339 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:28:09.339 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:28:09.339 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.339 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.339 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25813.0:115483 5.8 5:118541ec:::benchmark_data_smithi045_85309_object115482:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e455)", 2024-08-19T14:28:09.340 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:26:12.254294+0000", 2024-08-19T14:28:09.340 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 117.00973056799999, 2024-08-19T14:28:09.340 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.41025914400000002, 2024-08-19T14:28:09.340 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:28:09.340 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:28:09.340 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:28:09.340 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:28:09.340 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25813", 2024-08-19T14:28:09.340 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:28:09.340 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 115483 2024-08-19T14:28:09.340 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.340 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:28:09.340 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.340 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:28:09.340 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.254294+0000", 2024-08-19T14:28:09.341 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:28:09.341 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.341 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.341 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:28:09.341 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.254294+0000", 2024-08-19T14:28:09.341 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:28:09.341 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.341 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.341 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:28:09.341 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.254295+0000", 2024-08-19T14:28:09.341 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0049999999999999e-06 2024-08-19T14:28:09.341 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.341 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.341 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:28:09.342 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.254351+0000", 2024-08-19T14:28:09.342 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.6385999999999999e-05 2024-08-19T14:28:09.342 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.342 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.342 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:28:09.342 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.254365+0000", 2024-08-19T14:28:09.342 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4425e-05 2024-08-19T14:28:09.342 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.342 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.342 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:28:09.342 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.254373+0000", 2024-08-19T14:28:09.342 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.1620000000000002e-06 2024-08-19T14:28:09.342 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.342 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.343 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:28:09.343 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.254438+0000", 2024-08-19T14:28:09.343 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.5295000000000004e-05 2024-08-19T14:28:09.343 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.343 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.343 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:28:09.343 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.254556+0000", 2024-08-19T14:28:09.343 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000117555 2024-08-19T14:28:09.343 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.343 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.343 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,9", 2024-08-19T14:28:09.343 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.254698+0000", 2024-08-19T14:28:09.343 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000142003 2024-08-19T14:28:09.344 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.344 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.344 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:28:09.344 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.256435+0000", 2024-08-19T14:28:09.344 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0017378999999999999 2024-08-19T14:28:09.344 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.344 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.344 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:09.344 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.265174+0000", 2024-08-19T14:28:09.344 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0087388980000000002 2024-08-19T14:28:09.344 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.344 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.344 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:09.344 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.664449+0000", 2024-08-19T14:28:09.345 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.39927473800000002 2024-08-19T14:28:09.345 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.345 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.345 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:28:09.345 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.664502+0000", 2024-08-19T14:28:09.345 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.3066000000000003e-05 2024-08-19T14:28:09.345 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.345 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.345 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:28:09.345 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.664553+0000", 2024-08-19T14:28:09.345 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0711000000000003e-05 2024-08-19T14:28:09.345 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:28:09.345 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:28:09.345 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:28:09.346 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.346 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.346 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25813.0:115542 5.8 5:126f9ca1:::benchmark_data_smithi045_85309_object115541:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e455)", 2024-08-19T14:28:09.346 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:26:12.310153+0000", 2024-08-19T14:28:09.346 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 116.95387092, 2024-08-19T14:28:09.346 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.35453124699999999, 2024-08-19T14:28:09.346 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:28:09.346 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:28:09.346 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:28:09.346 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:28:09.346 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25813", 2024-08-19T14:28:09.346 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:28:09.346 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 115542 2024-08-19T14:28:09.347 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.347 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:28:09.347 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.347 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:28:09.347 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.310153+0000", 2024-08-19T14:28:09.347 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:28:09.347 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.347 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.347 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:28:09.347 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.310153+0000", 2024-08-19T14:28:09.347 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:28:09.347 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.347 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.348 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:28:09.348 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.310155+0000", 2024-08-19T14:28:09.348 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.279e-06 2024-08-19T14:28:09.348 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.348 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.348 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:28:09.348 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.310186+0000", 2024-08-19T14:28:09.348 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.1291999999999998e-05 2024-08-19T14:28:09.348 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.348 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.348 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:28:09.348 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.310199+0000", 2024-08-19T14:28:09.348 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2824e-05 2024-08-19T14:28:09.348 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.349 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.349 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:28:09.349 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.310206+0000", 2024-08-19T14:28:09.349 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.3189999999999997e-06 2024-08-19T14:28:09.349 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.349 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.349 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:28:09.349 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.311624+0000", 2024-08-19T14:28:09.349 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0014176740000000001 2024-08-19T14:28:09.349 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.349 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.350 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:28:09.350 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.311764+0000", 2024-08-19T14:28:09.350 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014065900000000001 2024-08-19T14:28:09.350 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.350 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.350 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,9", 2024-08-19T14:28:09.350 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.311923+0000", 2024-08-19T14:28:09.350 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015905800000000001 2024-08-19T14:28:09.350 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.350 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.350 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:28:09.350 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.313965+0000", 2024-08-19T14:28:09.350 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0020413430000000002 2024-08-19T14:28:09.350 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.351 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.351 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:09.351 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.320836+0000", 2024-08-19T14:28:09.351 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0068708529999999997 2024-08-19T14:28:09.351 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.351 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.351 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:09.351 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.664598+0000", 2024-08-19T14:28:09.351 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.34376222299999998 2024-08-19T14:28:09.351 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.351 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.351 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:28:09.351 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.664648+0000", 2024-08-19T14:28:09.351 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0489999999999999e-05 2024-08-19T14:28:09.352 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.352 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.352 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:28:09.352 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.664685+0000", 2024-08-19T14:28:09.352 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.6232999999999997e-05 2024-08-19T14:28:09.352 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:28:09.352 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:28:09.352 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:28:09.352 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.352 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.352 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25813.0:115641 5.8 5:1358c592:::benchmark_data_smithi045_85309_object115640:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e455)", 2024-08-19T14:28:09.352 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:26:12.409980+0000", 2024-08-19T14:28:09.352 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 116.85404388400001, 2024-08-19T14:28:09.353 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.25595528499999998, 2024-08-19T14:28:09.353 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:28:09.353 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:28:09.353 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:28:09.353 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:28:09.353 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25813", 2024-08-19T14:28:09.353 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:28:09.353 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 115641 2024-08-19T14:28:09.353 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.353 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:28:09.353 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.353 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:28:09.353 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.409980+0000", 2024-08-19T14:28:09.354 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:28:09.354 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.354 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.354 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:28:09.354 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.409980+0000", 2024-08-19T14:28:09.354 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:28:09.354 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.354 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.354 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:28:09.354 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.409982+0000", 2024-08-19T14:28:09.354 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2950000000000001e-06 2024-08-19T14:28:09.354 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.354 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.354 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:28:09.355 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.410007+0000", 2024-08-19T14:28:09.355 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.5198000000000002e-05 2024-08-19T14:28:09.355 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.355 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.355 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:28:09.355 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.410022+0000", 2024-08-19T14:28:09.355 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4678000000000001e-05 2024-08-19T14:28:09.355 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.355 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.355 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:28:09.355 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.410029+0000", 2024-08-19T14:28:09.355 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4590000000000001e-06 2024-08-19T14:28:09.355 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.355 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.356 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:28:09.356 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.410088+0000", 2024-08-19T14:28:09.356 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.9169999999999998e-05 2024-08-19T14:28:09.356 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.356 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.356 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:28:09.356 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.410232+0000", 2024-08-19T14:28:09.356 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014414699999999999 2024-08-19T14:28:09.356 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.356 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.356 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,9", 2024-08-19T14:28:09.356 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.410377+0000", 2024-08-19T14:28:09.356 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000144566 2024-08-19T14:28:09.356 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.356 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.357 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:28:09.357 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.412287+0000", 2024-08-19T14:28:09.357 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001909628 2024-08-19T14:28:09.357 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.357 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.357 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:09.357 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.449686+0000", 2024-08-19T14:28:09.357 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.037399549999999997 2024-08-19T14:28:09.357 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.357 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.357 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:09.357 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.665809+0000", 2024-08-19T14:28:09.357 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.21612313899999999 2024-08-19T14:28:09.357 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.358 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.358 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:28:09.358 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.665895+0000", 2024-08-19T14:28:09.358 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.5433999999999996e-05 2024-08-19T14:28:09.358 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.358 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.358 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:28:09.358 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.665936+0000", 2024-08-19T14:28:09.358 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.1020999999999999e-05 2024-08-19T14:28:09.358 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:28:09.358 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:28:09.358 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:28:09.358 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.358 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.359 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25813.0:115648 5.8 5:15d8ce6a:::benchmark_data_smithi045_85309_object115647:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e455)", 2024-08-19T14:28:09.359 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:26:12.416281+0000", 2024-08-19T14:28:09.359 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 116.847743572, 2024-08-19T14:28:09.359 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.24979046499999999, 2024-08-19T14:28:09.359 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:28:09.359 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:28:09.359 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:28:09.359 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:28:09.359 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25813", 2024-08-19T14:28:09.359 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:28:09.359 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 115648 2024-08-19T14:28:09.359 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.359 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:28:09.359 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.360 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:28:09.360 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.416281+0000", 2024-08-19T14:28:09.360 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:28:09.360 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.360 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.360 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:28:09.360 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.416281+0000", 2024-08-19T14:28:09.360 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:28:09.360 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.360 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.360 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:28:09.360 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.416282+0000", 2024-08-19T14:28:09.360 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5740000000000001e-06 2024-08-19T14:28:09.361 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.361 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.361 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:28:09.361 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.416304+0000", 2024-08-19T14:28:09.361 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1698000000000001e-05 2024-08-19T14:28:09.361 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.361 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.361 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:28:09.361 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.416317+0000", 2024-08-19T14:28:09.361 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3084e-05 2024-08-19T14:28:09.361 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.361 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.361 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:28:09.361 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.416324+0000", 2024-08-19T14:28:09.361 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.0570000000000004e-06 2024-08-19T14:28:09.362 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.362 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.362 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:28:09.362 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.416383+0000", 2024-08-19T14:28:09.362 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.9261000000000001e-05 2024-08-19T14:28:09.362 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.362 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.362 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:28:09.362 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.416500+0000", 2024-08-19T14:28:09.362 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000116523 2024-08-19T14:28:09.362 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.362 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.362 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,9", 2024-08-19T14:28:09.362 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.416646+0000", 2024-08-19T14:28:09.363 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014645500000000001 2024-08-19T14:28:09.363 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.363 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.363 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:28:09.363 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.418689+0000", 2024-08-19T14:28:09.363 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0020425019999999999 2024-08-19T14:28:09.363 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.363 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.363 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:09.363 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.449715+0000", 2024-08-19T14:28:09.363 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.031026194 2024-08-19T14:28:09.363 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.363 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.363 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:09.364 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.665979+0000", 2024-08-19T14:28:09.364 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.21626399099999999 2024-08-19T14:28:09.364 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.364 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.364 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:28:09.364 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.666032+0000", 2024-08-19T14:28:09.364 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.3140000000000003e-05 2024-08-19T14:28:09.364 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.364 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.364 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:28:09.364 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.666071+0000", 2024-08-19T14:28:09.364 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.8986000000000002e-05 2024-08-19T14:28:09.364 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:28:09.364 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:28:09.364 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:28:09.365 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.365 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.365 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25813.0:180218 5.d 5:bd4207f5:::benchmark_data_smithi045_85309_object59965:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e475)", 2024-08-19T14:28:09.365 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:27:11.940920+0000", 2024-08-19T14:28:09.365 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 57.323104350000001, 2024-08-19T14:28:09.365 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.23667333700000001, 2024-08-19T14:28:09.365 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:28:09.365 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:28:09.365 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:28:09.365 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:28:09.365 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25813", 2024-08-19T14:28:09.365 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:28:09.365 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 180218 2024-08-19T14:28:09.365 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.366 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:28:09.366 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.366 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:28:09.366 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:11.940920+0000", 2024-08-19T14:28:09.366 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:28:09.366 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.366 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.366 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:28:09.366 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:11.940920+0000", 2024-08-19T14:28:09.366 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:28:09.366 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.366 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.366 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:28:09.366 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:11.940921+0000", 2024-08-19T14:28:09.367 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.561e-06 2024-08-19T14:28:09.367 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.367 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.367 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:28:09.367 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:11.940927+0000", 2024-08-19T14:28:09.367 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7300000000000002e-06 2024-08-19T14:28:09.367 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.367 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.367 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:28:09.367 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:11.940938+0000", 2024-08-19T14:28:09.367 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0958000000000001e-05 2024-08-19T14:28:09.367 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.367 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.367 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:28:09.368 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:11.940945+0000", 2024-08-19T14:28:09.368 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.6579999999999996e-06 2024-08-19T14:28:09.368 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.368 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.368 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:28:09.368 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.031816+0000", 2024-08-19T14:28:09.368 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.090871065000000001 2024-08-19T14:28:09.368 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.368 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.368 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:28:09.368 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.031914+0000", 2024-08-19T14:28:09.368 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.7686e-05 2024-08-19T14:28:09.368 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.369 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.369 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 5,7", 2024-08-19T14:28:09.369 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.032000+0000", 2024-08-19T14:28:09.369 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.6169999999999997e-05 2024-08-19T14:28:09.369 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.369 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.369 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:28:09.369 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.045113+0000", 2024-08-19T14:28:09.369 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.013113028000000001 2024-08-19T14:28:09.369 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.369 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.369 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:09.369 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.047147+0000", 2024-08-19T14:28:09.369 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0020338050000000001 2024-08-19T14:28:09.370 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.370 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.370 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:09.370 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.177520+0000", 2024-08-19T14:28:09.370 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.130373085 2024-08-19T14:28:09.370 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.370 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.370 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:28:09.370 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.177556+0000", 2024-08-19T14:28:09.370 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.6403999999999999e-05 2024-08-19T14:28:09.370 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.370 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.370 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:28:09.371 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.177593+0000", 2024-08-19T14:28:09.371 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.7187000000000002e-05 2024-08-19T14:28:09.371 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:28:09.371 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:28:09.371 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:28:09.371 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.371 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.371 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25813.0:180682 5.8 5:1602eddf:::benchmark_data_smithi045_85309_object60429:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e476)", 2024-08-19T14:28:09.371 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:27:12.846408+0000", 2024-08-19T14:28:09.371 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 56.417616103999997, 2024-08-19T14:28:09.371 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.27902613199999998, 2024-08-19T14:28:09.371 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:28:09.371 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:28:09.371 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:28:09.372 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:28:09.372 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25813", 2024-08-19T14:28:09.372 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:28:09.372 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 180682 2024-08-19T14:28:09.372 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.372 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:28:09.372 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.372 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:28:09.372 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.846408+0000", 2024-08-19T14:28:09.372 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:28:09.372 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.372 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.372 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:28:09.372 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.846408+0000", 2024-08-19T14:28:09.373 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:28:09.373 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.373 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.373 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:28:09.373 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.846410+0000", 2024-08-19T14:28:09.373 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9980000000000002e-06 2024-08-19T14:28:09.373 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.373 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.373 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:28:09.373 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.846415+0000", 2024-08-19T14:28:09.373 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1120000000000004e-06 2024-08-19T14:28:09.373 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.373 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.373 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:28:09.374 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.846431+0000", 2024-08-19T14:28:09.374 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5786000000000001e-05 2024-08-19T14:28:09.374 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.374 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.374 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:28:09.374 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.846438+0000", 2024-08-19T14:28:09.374 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.3699999999999997e-06 2024-08-19T14:28:09.374 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.374 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.374 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:28:09.374 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.915048+0000", 2024-08-19T14:28:09.374 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.068609724999999996 2024-08-19T14:28:09.374 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.375 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.375 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:28:09.375 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.915174+0000", 2024-08-19T14:28:09.375 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012629899999999999 2024-08-19T14:28:09.375 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.375 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.375 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,9", 2024-08-19T14:28:09.375 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.915288+0000", 2024-08-19T14:28:09.375 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000113544 2024-08-19T14:28:09.375 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.375 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.375 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:09.375 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.921139+0000", 2024-08-19T14:28:09.375 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0058512420000000004 2024-08-19T14:28:09.375 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.376 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.376 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:28:09.376 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.923745+0000", 2024-08-19T14:28:09.376 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0026053589999999998 2024-08-19T14:28:09.376 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.376 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.376 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:09.376 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:13.125373+0000", 2024-08-19T14:28:09.376 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.201628589 2024-08-19T14:28:09.376 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.376 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.376 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:28:09.376 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:13.125405+0000", 2024-08-19T14:28:09.376 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.2224999999999997e-05 2024-08-19T14:28:09.376 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.377 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.377 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:28:09.377 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:13.125434+0000", 2024-08-19T14:28:09.377 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.8883000000000001e-05 2024-08-19T14:28:09.377 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:28:09.377 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:28:09.377 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:28:09.377 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.377 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.377 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25813.0:180713 5.8 5:13e650ea:::benchmark_data_smithi045_85309_object60460:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e476)", 2024-08-19T14:28:09.377 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:27:12.880045+0000", 2024-08-19T14:28:09.377 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 56.383979171999997, 2024-08-19T14:28:09.377 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.24953703699999999, 2024-08-19T14:28:09.377 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:28:09.378 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:28:09.378 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:28:09.378 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:28:09.378 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25813", 2024-08-19T14:28:09.378 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:28:09.378 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 180713 2024-08-19T14:28:09.378 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.378 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:28:09.378 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.378 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:28:09.378 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.880045+0000", 2024-08-19T14:28:09.378 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:28:09.378 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.378 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.379 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:28:09.379 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.880045+0000", 2024-08-19T14:28:09.379 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:28:09.379 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.379 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.379 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:28:09.379 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.880047+0000", 2024-08-19T14:28:09.379 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5510000000000001e-06 2024-08-19T14:28:09.379 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.379 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.379 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:28:09.379 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.880052+0000", 2024-08-19T14:28:09.379 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.5709999999999997e-06 2024-08-19T14:28:09.379 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.380 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.380 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:28:09.380 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.880068+0000", 2024-08-19T14:28:09.380 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6087e-05 2024-08-19T14:28:09.380 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.380 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.380 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:28:09.380 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.880076+0000", 2024-08-19T14:28:09.380 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4039999999999998e-06 2024-08-19T14:28:09.380 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.380 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.380 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:28:09.380 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.918429+0000", 2024-08-19T14:28:09.381 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.038353230000000002 2024-08-19T14:28:09.381 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.381 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.381 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:28:09.381 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.918538+0000", 2024-08-19T14:28:09.381 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00010878999999999999 2024-08-19T14:28:09.381 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.381 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.381 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,9", 2024-08-19T14:28:09.381 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.918613+0000", 2024-08-19T14:28:09.381 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.5405000000000001e-05 2024-08-19T14:28:09.381 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.381 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.381 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:28:09.381 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.923754+0000", 2024-08-19T14:28:09.382 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0051403999999999998 2024-08-19T14:28:09.382 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.382 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.382 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:09.382 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.924797+0000", 2024-08-19T14:28:09.382 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0010431150000000001 2024-08-19T14:28:09.382 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.382 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.382 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:09.382 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:13.129461+0000", 2024-08-19T14:28:09.382 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20466435799999999 2024-08-19T14:28:09.382 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.382 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.383 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:28:09.383 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:13.129522+0000", 2024-08-19T14:28:09.383 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.1054000000000002e-05 2024-08-19T14:28:09.383 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.383 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.383 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:28:09.383 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:13.129582+0000", 2024-08-19T14:28:09.383 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.0071999999999997e-05 2024-08-19T14:28:09.383 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:28:09.383 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:28:09.383 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:28:09.383 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.383 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.384 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25813.0:180719 5.8 5:15c77316:::benchmark_data_smithi045_85309_object60466:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e476)", 2024-08-19T14:28:09.384 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:27:12.886662+0000", 2024-08-19T14:28:09.384 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 56.377362668000004, 2024-08-19T14:28:09.384 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.24306710300000001, 2024-08-19T14:28:09.384 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:28:09.384 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:28:09.384 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:28:09.384 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:28:09.384 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25813", 2024-08-19T14:28:09.384 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:28:09.384 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 180719 2024-08-19T14:28:09.384 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.384 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:28:09.384 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.385 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:28:09.385 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.886662+0000", 2024-08-19T14:28:09.385 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:28:09.385 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.385 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.385 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:28:09.385 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.886662+0000", 2024-08-19T14:28:09.385 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:28:09.385 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.385 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.385 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:28:09.385 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.886663+0000", 2024-08-19T14:28:09.385 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.184e-06 2024-08-19T14:28:09.385 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.386 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.386 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:28:09.386 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.886667+0000", 2024-08-19T14:28:09.386 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.907e-06 2024-08-19T14:28:09.386 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.386 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.386 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:28:09.386 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.886679+0000", 2024-08-19T14:28:09.386 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1915e-05 2024-08-19T14:28:09.386 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.386 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.386 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:28:09.386 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.886686+0000", 2024-08-19T14:28:09.387 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.2049999999999996e-06 2024-08-19T14:28:09.387 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.387 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.387 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:28:09.387 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.921183+0000", 2024-08-19T14:28:09.387 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.034497556999999998 2024-08-19T14:28:09.387 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.387 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.387 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:28:09.387 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.921260+0000", 2024-08-19T14:28:09.387 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.6388000000000001e-05 2024-08-19T14:28:09.387 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.387 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.387 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,9", 2024-08-19T14:28:09.388 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.921322+0000", 2024-08-19T14:28:09.388 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.2611000000000001e-05 2024-08-19T14:28:09.388 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.388 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.388 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:28:09.388 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.923834+0000", 2024-08-19T14:28:09.388 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.002511924 2024-08-19T14:28:09.388 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.388 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.388 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:09.388 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.924841+0000", 2024-08-19T14:28:09.388 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0010065129999999999 2024-08-19T14:28:09.388 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.388 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.389 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:09.389 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:13.129629+0000", 2024-08-19T14:28:09.389 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20478792500000001 2024-08-19T14:28:09.389 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.389 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.389 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:28:09.389 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:13.129683+0000", 2024-08-19T14:28:09.389 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4293999999999999e-05 2024-08-19T14:28:09.389 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.389 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.389 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:28:09.389 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:13.129729+0000", 2024-08-19T14:28:09.389 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.5680000000000003e-05 2024-08-19T14:28:09.389 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:28:09.390 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:28:09.390 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:28:09.390 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.390 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.390 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25813.0:237770 5.8 5:11831c02:::benchmark_data_smithi045_85309_object117517:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e495)", 2024-08-19T14:28:09.390 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:28:07.106685+0000", 2024-08-19T14:28:09.390 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 2.1573395799999999, 2024-08-19T14:28:09.390 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.22493542599999999, 2024-08-19T14:28:09.390 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:28:09.390 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:28:09.390 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:28:09.390 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:28:09.390 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25813", 2024-08-19T14:28:09.390 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:28:09.391 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 237770 2024-08-19T14:28:09.391 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.391 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:28:09.391 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.391 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:28:09.391 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:28:07.106685+0000", 2024-08-19T14:28:09.391 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:28:09.391 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.391 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.391 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:28:09.391 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:28:07.106685+0000", 2024-08-19T14:28:09.391 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:28:09.391 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.391 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.392 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:28:09.392 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:28:07.106685+0000", 2024-08-19T14:28:09.392 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.6199999999999998e-07 2024-08-19T14:28:09.392 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.392 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.392 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:28:09.392 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:28:07.106688+0000", 2024-08-19T14:28:09.392 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.0510000000000001e-06 2024-08-19T14:28:09.392 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.392 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.392 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:28:09.392 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:28:07.106696+0000", 2024-08-19T14:28:09.392 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.9899999999999997e-06 2024-08-19T14:28:09.392 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.393 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.393 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:28:09.393 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:28:07.106702+0000", 2024-08-19T14:28:09.393 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7810000000000002e-06 2024-08-19T14:28:09.393 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.393 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.393 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:28:09.393 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:28:07.106846+0000", 2024-08-19T14:28:09.393 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014427600000000001 2024-08-19T14:28:09.393 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.393 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.393 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for peered", 2024-08-19T14:28:09.393 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:28:07.106854+0000", 2024-08-19T14:28:09.393 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.3309999999999998e-06 2024-08-19T14:28:09.394 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.394 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.394 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:28:09.394 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:28:07.317267+0000", 2024-08-19T14:28:09.394 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.21041380400000001 2024-08-19T14:28:09.394 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.394 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.394 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:28:09.394 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:28:07.317422+0000", 2024-08-19T14:28:09.394 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015415200000000001 2024-08-19T14:28:09.394 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.394 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.394 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,5", 2024-08-19T14:28:09.395 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:28:07.318149+0000", 2024-08-19T14:28:09.395 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00072762499999999997 2024-08-19T14:28:09.395 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.395 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.395 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:28:09.395 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:28:07.322380+0000", 2024-08-19T14:28:09.395 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0042309590000000003 2024-08-19T14:28:09.395 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.395 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.395 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:09.395 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:28:07.323678+0000", 2024-08-19T14:28:09.395 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001298366 2024-08-19T14:28:09.395 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.396 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.396 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:09.396 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:28:07.331373+0000", 2024-08-19T14:28:09.396 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0076946790000000003 2024-08-19T14:28:09.396 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.396 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.396 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:28:09.396 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:28:07.331429+0000", 2024-08-19T14:28:09.396 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.5517999999999999e-05 2024-08-19T14:28:09.396 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:28:09.396 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:28:09.396 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:28:09.396 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:28:07.331620+0000", 2024-08-19T14:28:09.396 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00019143200000000001 2024-08-19T14:28:09.397 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:28:09.397 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:28:09.397 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:28:09.397 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:28:09.397 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:28:09.397 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:28:09.397 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:28:09.419 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:09.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:09.420 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:28:09.521 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:28:09.545 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:09.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:09.546 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:28:09.647 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:28:09.671 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:09.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:09.672 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:28:09.773 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:28:09.796 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:09.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:09.798 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:28:09.844 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is unset 2024-08-19T14:28:09.898 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:28:09.927 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:09.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:09.928 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:28:10.029 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:28:10.055 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:10.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:10.057 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:28:10.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:09 smithi031 ceph-mon[16336]: pgmap v1143: 57 pgs: 1 active+remapped+backfill_wait, 2 active+undersized+remapped, 4 peering, 50 active+clean; 232 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 596 op/s; 190/11112 objects misplaced (1.710%) 2024-08-19T14:28:10.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:09 smithi031 ceph-mon[16336]: osdmap e497: 12 total, 12 up, 8 in 2024-08-19T14:28:10.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:09 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:28:10.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:09 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:28:10.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:09 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/127542232' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:28:10.157 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:28:10.197 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:10.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:10.198 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:28:10.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:09 smithi032 ceph-mon[20004]: pgmap v1143: 57 pgs: 1 active+remapped+backfill_wait, 2 active+undersized+remapped, 4 peering, 50 active+clean; 232 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 596 op/s; 190/11112 objects misplaced (1.710%) 2024-08-19T14:28:10.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:09 smithi032 ceph-mon[20004]: osdmap e497: 12 total, 12 up, 8 in 2024-08-19T14:28:10.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:09 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:28:10.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:09 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:28:10.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:09 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/127542232' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:28:10.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:09 smithi045 ceph-mon[20143]: pgmap v1143: 57 pgs: 1 active+remapped+backfill_wait, 2 active+undersized+remapped, 4 peering, 50 active+clean; 232 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 596 op/s; 190/11112 objects misplaced (1.710%) 2024-08-19T14:28:10.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:09 smithi045 ceph-mon[20143]: osdmap e497: 12 total, 12 up, 8 in 2024-08-19T14:28:10.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:09 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:28:10.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:09 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:28:10.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:09 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/127542232' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:28:10.299 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:28:10.324 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:10.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:10.325 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:28:10.375 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:28:10.408 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-08-19T14:28:10.426 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:28:10.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:10.519 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:10.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:28:10.621 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:28:10.647 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:10.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:10.648 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:28:10.749 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:28:10.774 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:10.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:10.775 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:28:10.875 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:28:10.900 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:10.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:10.901 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:28:10.972 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 7, 8, 11] out_osds: [10, 6, 2, 9] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-08-19T14:28:10.972 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-19T14:28:10.972 INFO:tasks.thrashosds.thrasher:Removing osd 11, in_osds are: [0, 1, 3, 4, 5, 7, 8, 11] 2024-08-19T14:28:10.973 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd out 11 2024-08-19T14:28:11.002 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:28:11.038 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:11.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:11.039 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:28:11.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:10 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/127542232' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:28:11.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:10 smithi031 ceph-mon[16336]: osdmap e498: 12 total, 12 up, 8 in 2024-08-19T14:28:11.139 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:28:11.173 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:11.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:11.174 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:28:11.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:10 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/127542232' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:28:11.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:10 smithi032 ceph-mon[20004]: osdmap e498: 12 total, 12 up, 8 in 2024-08-19T14:28:11.275 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:28:11.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:10 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/127542232' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:28:11.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:10 smithi045 ceph-mon[20143]: osdmap e498: 12 total, 12 up, 8 in 2024-08-19T14:28:11.304 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:11.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:11.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:28:11.406 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:28:11.441 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:11.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:11.443 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:28:11.544 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:28:11.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:11.583 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:11.583 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:28:11.684 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:28:11.709 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:11.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:11.710 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:28:11.811 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:28:11.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:11.847 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:11.847 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:28:11.939 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Removed 120250 objects 2024-08-19T14:28:11.939 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Clean up completed and total clean up time :113.607 2024-08-19T14:28:11.948 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:28:12.009 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:12.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:12.010 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:28:12.111 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:28:12.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:11 smithi031 ceph-mon[16336]: pgmap v1146: 57 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 2 active+undersized+remapped, 8 peering, 43 active+clean; 153 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 461 op/s; 9/7314 objects degraded (0.123%); 231/7314 objects misplaced (3.158%); 3.5 MiB/s, 83 objects/s recovering 2024-08-19T14:28:12.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:11 smithi031 ceph-mon[16336]: Health check failed: Degraded data redundancy: 9/7314 objects degraded (0.123%), 2 pgs degraded (PG_DEGRADED) 2024-08-19T14:28:12.134 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:12.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:12.135 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:28:12.236 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:28:12.259 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:11 smithi032 ceph-mon[20004]: pgmap v1146: 57 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 2 active+undersized+remapped, 8 peering, 43 active+clean; 153 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 461 op/s; 9/7314 objects degraded (0.123%); 231/7314 objects misplaced (3.158%); 3.5 MiB/s, 83 objects/s recovering 2024-08-19T14:28:12.259 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:11 smithi032 ceph-mon[20004]: Health check failed: Degraded data redundancy: 9/7314 objects degraded (0.123%), 2 pgs degraded (PG_DEGRADED) 2024-08-19T14:28:12.266 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:12.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:12.267 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:28:12.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:11 smithi045 ceph-mon[20143]: pgmap v1146: 57 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 2 active+undersized+remapped, 8 peering, 43 active+clean; 153 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 461 op/s; 9/7314 objects degraded (0.123%); 231/7314 objects misplaced (3.158%); 3.5 MiB/s, 83 objects/s recovering 2024-08-19T14:28:12.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:11 smithi045 ceph-mon[20143]: Health check failed: Degraded data redundancy: 9/7314 objects degraded (0.123%), 2 pgs degraded (PG_DEGRADED) 2024-08-19T14:28:12.368 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:28:12.392 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:12.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:12.393 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:28:12.494 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:28:12.516 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:12.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:12.517 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:28:12.618 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:28:12.642 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:12.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:12.643 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:28:12.744 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:28:12.766 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:12.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:12.768 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:28:12.869 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:28:12.891 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:12.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:12.892 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:28:12.993 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:28:13.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:13.017 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:13.017 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:28:13.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:12 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:28:13.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:12 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:28:13.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:12 smithi031 ceph-mon[16336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:28:13.118 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:28:13.139 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:13.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:13.140 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:28:13.241 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:28:13.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:12 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:28:13.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:12 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:28:13.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:12 smithi032 ceph-mon[20004]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:28:13.266 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:13.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:13.267 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:28:13.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:12 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:28:13.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:12 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:28:13.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:12 smithi045 ceph-mon[20143]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:28:13.368 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:28:13.389 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:13.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:13.390 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:28:13.491 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:28:13.515 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:13.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:13.516 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:28:13.617 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:28:13.640 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:13.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:13.642 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:28:13.743 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:28:13.767 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:13.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:13.768 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:28:13.869 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:28:13.892 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:13.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:13.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:28:13.994 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:28:14.020 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:14.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:14.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:28:14.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:13 smithi031 ceph-mon[16336]: pgmap v1147: 57 pgs: 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 2 active+undersized+remapped, 4 peering, 46 active+clean; 126 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 453 op/s; 66/6048 objects degraded (1.091%); 251/6048 objects misplaced (4.150%); 4.9 MiB/s, 193 objects/s recovering 2024-08-19T14:28:14.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:13 smithi031 ceph-mon[16336]: osdmap e499: 12 total, 12 up, 8 in 2024-08-19T14:28:14.121 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:28:14.145 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:14.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:14.146 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:28:14.247 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:28:14.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:13 smithi032 ceph-mon[20004]: pgmap v1147: 57 pgs: 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 2 active+undersized+remapped, 4 peering, 46 active+clean; 126 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 453 op/s; 66/6048 objects degraded (1.091%); 251/6048 objects misplaced (4.150%); 4.9 MiB/s, 193 objects/s recovering 2024-08-19T14:28:14.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:13 smithi032 ceph-mon[20004]: osdmap e499: 12 total, 12 up, 8 in 2024-08-19T14:28:14.269 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:14.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:14.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:28:14.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:13 smithi045 ceph-mon[20143]: pgmap v1147: 57 pgs: 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 2 active+undersized+remapped, 4 peering, 46 active+clean; 126 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 453 op/s; 66/6048 objects degraded (1.091%); 251/6048 objects misplaced (4.150%); 4.9 MiB/s, 193 objects/s recovering 2024-08-19T14:28:14.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:13 smithi045 ceph-mon[20143]: osdmap e499: 12 total, 12 up, 8 in 2024-08-19T14:28:14.371 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:28:14.394 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:14.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:14.395 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:28:14.496 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:28:14.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:14.523 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:14.523 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:28:14.624 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:28:14.650 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:14.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:14.651 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:28:14.752 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:28:14.776 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:14.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:14.777 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:28:14.878 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:28:14.902 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:14.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:14.903 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:28:15.004 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:28:15.029 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:15.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:15.030 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:28:15.131 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:28:15.151 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:15.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:15.152 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:28:15.253 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:28:15.274 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:15.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:15.275 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:28:15.376 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:28:15.384 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:28:15.392 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:28:15.400 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:15.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:15.402 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:28:15.503 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:28:15.529 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:15.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:15.530 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:28:15.631 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:28:15.657 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:15.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:15.658 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:28:15.710 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_3 2024-08-19T14:28:15.710 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd pool rm unique_pool_3 unique_pool_3 --yes-i-really-really-mean-it 2024-08-19T14:28:15.759 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:28:15.789 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:15.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:15.791 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:28:15.891 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:28:15.912 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:15.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:15.913 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:28:16.014 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:28:16.036 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:16.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:16.037 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:28:16.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:15 smithi031 ceph-mon[16336]: pgmap v1149: 57 pgs: 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 53 active+clean; 29 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 541 op/s; 66/1359 objects degraded (4.857%); 20/1359 objects misplaced (1.472%); 6.3 MiB/s, 487 objects/s recovering 2024-08-19T14:28:16.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:15 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:28:16.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:15 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:28:16.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:15 smithi031 ceph-mon[16336]: osdmap e500: 12 total, 12 up, 8 in 2024-08-19T14:28:16.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:15 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:28:16.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:15 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:28:16.138 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:28:16.159 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:16.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:16.160 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:28:16.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:15 smithi032 ceph-mon[20004]: pgmap v1149: 57 pgs: 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 53 active+clean; 29 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 541 op/s; 66/1359 objects degraded (4.857%); 20/1359 objects misplaced (1.472%); 6.3 MiB/s, 487 objects/s recovering 2024-08-19T14:28:16.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:15 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:28:16.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:15 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:28:16.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:15 smithi032 ceph-mon[20004]: osdmap e500: 12 total, 12 up, 8 in 2024-08-19T14:28:16.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:15 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:28:16.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:15 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:28:16.261 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:28:16.261 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:28:16.262 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-08-19T14:28:16.262 INFO:teuthology.orchestra.run.smithi031.stdout: "num_ops": 0 2024-08-19T14:28:16.262 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:28:16.283 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:16.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:16.284 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:28:16.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:15 smithi045 ceph-mon[20143]: pgmap v1149: 57 pgs: 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 53 active+clean; 29 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 541 op/s; 66/1359 objects degraded (4.857%); 20/1359 objects misplaced (1.472%); 6.3 MiB/s, 487 objects/s recovering 2024-08-19T14:28:16.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:15 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:28:16.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:15 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:28:16.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:15 smithi045 ceph-mon[20143]: osdmap e500: 12 total, 12 up, 8 in 2024-08-19T14:28:16.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:15 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:28:16.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:15 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:28:16.384 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:28:16.415 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:16.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:16.416 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:28:16.439 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:28:16.517 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:28:16.538 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:16.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:16.539 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:28:16.640 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:28:16.667 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:16.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:16.668 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:28:16.769 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:28:16.794 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:16.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:16.795 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:28:16.882 INFO:teuthology.orchestra.run.smithi031.stderr:marked out osd.11. 2024-08-19T14:28:16.896 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:28:16.917 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:16.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:16.918 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:28:16.960 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:16 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mon-a[16332]: 2024-08-19T14:28:16.734+0000 7f2ff503e640 -1 mon.a@0(leader).osd e501 definitely_dead 0 2024-08-19T14:28:16.960 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:16 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:28:16.960 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:16 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:28:16.960 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:16 smithi031 ceph-mon[16336]: osdmap e501: 12 total, 12 up, 8 in 2024-08-19T14:28:16.960 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:16 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2969917156' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["11"]}]: dispatch 2024-08-19T14:28:16.960 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:16 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["11"]}]: dispatch 2024-08-19T14:28:16.960 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:16 smithi031 ceph-mon[16336]: Client client.admin marked osd.11 out, while it was still marked up 2024-08-19T14:28:17.019 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:28:17.045 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:17.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:17.047 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:28:17.148 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:28:17.173 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:17.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:17.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:28:17.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:16 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:28:17.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:16 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:28:17.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:16 smithi032 ceph-mon[20004]: osdmap e501: 12 total, 12 up, 8 in 2024-08-19T14:28:17.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:16 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2969917156' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["11"]}]: dispatch 2024-08-19T14:28:17.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:16 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["11"]}]: dispatch 2024-08-19T14:28:17.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:16 smithi032 ceph-mon[20004]: Client client.admin marked osd.11 out, while it was still marked up 2024-08-19T14:28:17.275 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:28:17.287 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-08-19T14:28:17.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:16 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:28:17.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:16 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:28:17.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:16 smithi045 ceph-mon[20143]: osdmap e501: 12 total, 12 up, 8 in 2024-08-19T14:28:17.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:16 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2969917156' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["11"]}]: dispatch 2024-08-19T14:28:17.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:16 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["11"]}]: dispatch 2024-08-19T14:28:17.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:16 smithi045 ceph-mon[20143]: Client client.admin marked osd.11 out, while it was still marked up 2024-08-19T14:28:17.297 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-19T14:28:17.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:17.299 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:17.299 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:28:17.400 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:28:17.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:17.423 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:17.423 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:28:17.524 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:28:17.546 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:17.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:17.547 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:28:17.648 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:28:17.673 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:17.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:17.674 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:28:17.775 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:28:17.796 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:17.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:17.797 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:28:17.827 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:28:17.827 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:28:17.857 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:28:17.857 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:28:17.886 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:28:17.886 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:28:17.898 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:28:17.914 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:28:17.915 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:28:17.921 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:17.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:17.923 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:28:17.938 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:28:17.938 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:28:17.963 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:28:17.964 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:28:17.983 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:28:17.983 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:28:18.002 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:28:18.002 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:28:18.023 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:28:18.024 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:28:18.024 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:28:18.045 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:18.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:18.046 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:28:18.046 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:28:18.047 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:28:18.068 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:28:18.068 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:28:18.088 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:28:18.088 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:28:18.089 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:17 smithi031 ceph-mon[16336]: pgmap v1152: 57 pgs: 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 53 active+clean; 6.5 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 389 op/s; 66/294 objects degraded (22.449%); 20/294 objects misplaced (6.803%); 4.0 MiB/s, 432 objects/s recovering 2024-08-19T14:28:18.089 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:17 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 66/294 objects degraded (22.449%), 3 pgs degraded (PG_DEGRADED) 2024-08-19T14:28:18.089 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:17 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["11"]}]': finished 2024-08-19T14:28:18.089 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:17 smithi031 ceph-mon[16336]: osdmap e502: 12 total, 12 up, 7 in 2024-08-19T14:28:18.089 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:17 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2275776892' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:28:18.089 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:17 smithi031 ceph-mon[16336]: osdmap e503: 12 total, 12 up, 7 in 2024-08-19T14:28:18.147 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:28:18.171 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:18.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:18.172 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:28:18.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:17 smithi032 ceph-mon[20004]: pgmap v1152: 57 pgs: 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 53 active+clean; 6.5 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 389 op/s; 66/294 objects degraded (22.449%); 20/294 objects misplaced (6.803%); 4.0 MiB/s, 432 objects/s recovering 2024-08-19T14:28:18.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:17 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 66/294 objects degraded (22.449%), 3 pgs degraded (PG_DEGRADED) 2024-08-19T14:28:18.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:17 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["11"]}]': finished 2024-08-19T14:28:18.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:17 smithi032 ceph-mon[20004]: osdmap e502: 12 total, 12 up, 7 in 2024-08-19T14:28:18.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:17 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2275776892' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:28:18.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:17 smithi032 ceph-mon[20004]: osdmap e503: 12 total, 12 up, 7 in 2024-08-19T14:28:18.272 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:28:18.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:17 smithi045 ceph-mon[20143]: pgmap v1152: 57 pgs: 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 53 active+clean; 6.5 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 389 op/s; 66/294 objects degraded (22.449%); 20/294 objects misplaced (6.803%); 4.0 MiB/s, 432 objects/s recovering 2024-08-19T14:28:18.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:17 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 66/294 objects degraded (22.449%), 3 pgs degraded (PG_DEGRADED) 2024-08-19T14:28:18.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:17 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["11"]}]': finished 2024-08-19T14:28:18.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:17 smithi045 ceph-mon[20143]: osdmap e502: 12 total, 12 up, 7 in 2024-08-19T14:28:18.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:17 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2275776892' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:28:18.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:17 smithi045 ceph-mon[20143]: osdmap e503: 12 total, 12 up, 7 in 2024-08-19T14:28:18.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:18.298 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:18.298 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:28:18.399 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:28:18.425 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:18.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:18.426 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:28:18.527 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:28:18.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:18.549 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:18.550 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:28:18.651 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:28:18.674 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:18.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:18.675 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:28:18.776 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:28:18.797 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:18.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:18.798 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:28:18.899 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:28:18.923 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:18.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:18.924 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:28:19.025 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:28:19.048 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:19.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:19.049 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:28:19.150 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:28:19.178 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:19.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:19.179 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:28:19.280 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:28:19.305 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:19.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:19.306 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:28:19.342 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:28:19.407 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:28:19.432 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:19.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:19.434 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:28:19.535 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:28:19.561 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:19.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:19.562 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:28:19.663 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:28:19.689 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:19.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:19.690 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:28:19.790 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:28:19.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:19.817 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:19.817 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:28:19.918 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:28:19.929 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:28:19.945 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:19.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:19.946 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:28:19.973 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:28:20.047 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:28:20.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:20.074 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:20.074 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:28:20.094 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:19 smithi031 ceph-mon[16336]: pgmap v1155: 57 pgs: 2 unknown, 1 peering, 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 50 active+clean; 590 KiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 511 B/s rd, 16 KiB/s wr, 114 op/s; 86/6 objects degraded (1433.333%); 12 KiB/s, 352 objects/s recovering 2024-08-19T14:28:20.094 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:19 smithi031 ceph-mon[16336]: osdmap e504: 12 total, 12 up, 7 in 2024-08-19T14:28:20.094 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:19 smithi031 ceph-mon[16336]: 5.d starting backfill to osd.0 from (0'0,0'0] MAX to 498'17890 2024-08-19T14:28:20.094 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:19 smithi031 ceph-mon[16336]: 5.a starting backfill to osd.7 from (0'0,0'0] MAX to 498'17313 2024-08-19T14:28:20.094 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:19 smithi031 ceph-mon[16336]: Health check failed: Reduced data availability: 2 pgs inactive (PG_AVAILABILITY) 2024-08-19T14:28:20.175 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:28:20.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:20.202 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:20.202 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:28:20.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:19 smithi032 ceph-mon[20004]: pgmap v1155: 57 pgs: 2 unknown, 1 peering, 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 50 active+clean; 590 KiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 511 B/s rd, 16 KiB/s wr, 114 op/s; 86/6 objects degraded (1433.333%); 12 KiB/s, 352 objects/s recovering 2024-08-19T14:28:20.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:19 smithi032 ceph-mon[20004]: osdmap e504: 12 total, 12 up, 7 in 2024-08-19T14:28:20.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:19 smithi032 ceph-mon[20004]: 5.d starting backfill to osd.0 from (0'0,0'0] MAX to 498'17890 2024-08-19T14:28:20.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:19 smithi032 ceph-mon[20004]: 5.a starting backfill to osd.7 from (0'0,0'0] MAX to 498'17313 2024-08-19T14:28:20.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:19 smithi032 ceph-mon[20004]: Health check failed: Reduced data availability: 2 pgs inactive (PG_AVAILABILITY) 2024-08-19T14:28:20.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:19 smithi045 ceph-mon[20143]: pgmap v1155: 57 pgs: 2 unknown, 1 peering, 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 50 active+clean; 590 KiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 511 B/s rd, 16 KiB/s wr, 114 op/s; 86/6 objects degraded (1433.333%); 12 KiB/s, 352 objects/s recovering 2024-08-19T14:28:20.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:19 smithi045 ceph-mon[20143]: osdmap e504: 12 total, 12 up, 7 in 2024-08-19T14:28:20.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:19 smithi045 ceph-mon[20143]: 5.d starting backfill to osd.0 from (0'0,0'0] MAX to 498'17890 2024-08-19T14:28:20.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:19 smithi045 ceph-mon[20143]: 5.a starting backfill to osd.7 from (0'0,0'0] MAX to 498'17313 2024-08-19T14:28:20.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:19 smithi045 ceph-mon[20143]: Health check failed: Reduced data availability: 2 pgs inactive (PG_AVAILABILITY) 2024-08-19T14:28:20.303 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:28:20.328 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:20.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:20.329 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:28:20.430 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:28:20.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:20.467 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:20.467 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:28:20.568 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:28:20.595 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:20.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:20.596 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:28:20.677 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:28:20.677 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:28:20.697 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:28:20.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:20.724 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:20.724 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:28:20.826 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:28:20.857 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:20.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:20.858 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:28:20.959 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:28:20.987 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:20.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:20.988 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:28:21.089 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:28:21.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:20 smithi031 ceph-mon[16336]: 5.f starting backfill to osd.3 from (0'0,0'0] MAX to 498'14744 2024-08-19T14:28:21.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:20 smithi031 ceph-mon[16336]: osdmap e505: 12 total, 12 up, 7 in 2024-08-19T14:28:21.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:21.117 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:21.117 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:28:21.158 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:20 smithi032 ceph-mon[20004]: 5.f starting backfill to osd.3 from (0'0,0'0] MAX to 498'14744 2024-08-19T14:28:21.158 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:20 smithi032 ceph-mon[20004]: osdmap e505: 12 total, 12 up, 7 in 2024-08-19T14:28:21.218 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:28:21.246 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:21.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:21.247 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:28:21.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:20 smithi045 ceph-mon[20143]: 5.f starting backfill to osd.3 from (0'0,0'0] MAX to 498'14744 2024-08-19T14:28:21.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:20 smithi045 ceph-mon[20143]: osdmap e505: 12 total, 12 up, 7 in 2024-08-19T14:28:21.348 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:28:21.373 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:21.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:21.374 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:28:21.475 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:28:21.499 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:21.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:21.500 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:28:21.601 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:28:21.630 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:21.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:21.631 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:28:21.733 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:28:21.758 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:21.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:21.760 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:28:21.860 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'unique_pool_3' removed 2024-08-19T14:28:21.861 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:28:21.871 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:28:21.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:21.891 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:21.891 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:28:21.992 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:28:22.026 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:22.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:22.027 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:28:22.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:21 smithi031 ceph-mon[16336]: pgmap v1158: 57 pgs: 1 activating+undersized+degraded+remapped, 1 activating+undersized+remapped, 1 active+undersized+remapped, 1 active+recovering+undersized+remapped, 1 activating+remapped, 4 remapped+peering, 2 unknown, 1 peering, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 590 KiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 511 B/s rd, 16 KiB/s wr, 25 op/s; 9854/6 objects degraded (164233.333%); 12 KiB/s, 1.15k objects/s recovering 2024-08-19T14:28:22.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:21 smithi031 ceph-mon[16336]: osdmap e506: 12 total, 12 up, 7 in 2024-08-19T14:28:22.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:21 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3956423777' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-08-19T14:28:22.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:21 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-08-19T14:28:22.128 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:28:22.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:22.150 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:22.150 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:28:22.251 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:28:22.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:21 smithi032 ceph-mon[20004]: pgmap v1158: 57 pgs: 1 activating+undersized+degraded+remapped, 1 activating+undersized+remapped, 1 active+undersized+remapped, 1 active+recovering+undersized+remapped, 1 activating+remapped, 4 remapped+peering, 2 unknown, 1 peering, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 590 KiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 511 B/s rd, 16 KiB/s wr, 25 op/s; 9854/6 objects degraded (164233.333%); 12 KiB/s, 1.15k objects/s recovering 2024-08-19T14:28:22.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:21 smithi032 ceph-mon[20004]: osdmap e506: 12 total, 12 up, 7 in 2024-08-19T14:28:22.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:21 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3956423777' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-08-19T14:28:22.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:21 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-08-19T14:28:22.274 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:22.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:22.275 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:28:22.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:21 smithi045 ceph-mon[20143]: pgmap v1158: 57 pgs: 1 activating+undersized+degraded+remapped, 1 activating+undersized+remapped, 1 active+undersized+remapped, 1 active+recovering+undersized+remapped, 1 activating+remapped, 4 remapped+peering, 2 unknown, 1 peering, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 590 KiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 511 B/s rd, 16 KiB/s wr, 25 op/s; 9854/6 objects degraded (164233.333%); 12 KiB/s, 1.15k objects/s recovering 2024-08-19T14:28:22.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:21 smithi045 ceph-mon[20143]: osdmap e506: 12 total, 12 up, 7 in 2024-08-19T14:28:22.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:21 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3956423777' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-08-19T14:28:22.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:21 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-08-19T14:28:22.376 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:28:22.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:22.401 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:22.401 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:28:22.503 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:28:22.527 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:22.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:22.528 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:28:22.630 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:28:22.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:22.656 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:22.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:28:22.757 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:28:22.782 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:22.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:22.783 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:28:22.884 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:28:22.909 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:22.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:22.910 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:28:23.011 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:28:23.037 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:23.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:23.038 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:28:23.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:22 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-08-19T14:28:23.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:22 smithi031 ceph-mon[16336]: osdmap e507: 12 total, 12 up, 7 in 2024-08-19T14:28:23.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:22 smithi031 ceph-mon[16336]: 1.19 scrub starts 2024-08-19T14:28:23.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:22 smithi031 ceph-mon[16336]: 1.19 scrub ok 2024-08-19T14:28:23.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:22 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 9854/6 objects degraded (164233.333%), 3 pgs degraded (PG_DEGRADED) 2024-08-19T14:28:23.139 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:28:23.164 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:23.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:23.165 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:28:23.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:22 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-08-19T14:28:23.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:22 smithi032 ceph-mon[20004]: osdmap e507: 12 total, 12 up, 7 in 2024-08-19T14:28:23.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:22 smithi032 ceph-mon[20004]: 1.19 scrub starts 2024-08-19T14:28:23.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:22 smithi032 ceph-mon[20004]: 1.19 scrub ok 2024-08-19T14:28:23.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:22 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 9854/6 objects degraded (164233.333%), 3 pgs degraded (PG_DEGRADED) 2024-08-19T14:28:23.266 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:28:23.291 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:23.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:23.292 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:28:23.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:22 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-08-19T14:28:23.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:22 smithi045 ceph-mon[20143]: osdmap e507: 12 total, 12 up, 7 in 2024-08-19T14:28:23.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:22 smithi045 ceph-mon[20143]: 1.19 scrub starts 2024-08-19T14:28:23.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:22 smithi045 ceph-mon[20143]: 1.19 scrub ok 2024-08-19T14:28:23.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:22 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 9854/6 objects degraded (164233.333%), 3 pgs degraded (PG_DEGRADED) 2024-08-19T14:28:23.393 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:28:23.419 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:23.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:23.420 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:28:23.521 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:28:23.547 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:23.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:23.549 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:28:23.650 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:28:23.675 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:23.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:23.676 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:28:23.776 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:28:23.802 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:23.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:23.803 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:28:23.876 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:23 smithi031 ceph-mon[16336]: pgmap v1161: 31 pgs: 1 active+undersized+remapped, 2 remapped+peering, 2 unknown, 4 peering, 22 active+clean; 620 KiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-08-19T14:28:23.877 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:23 smithi031 ceph-mon[16336]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 9854/6 objects degraded (164233.333%), 3 pgs degraded) 2024-08-19T14:28:23.904 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:28:23.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:23.931 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:23.931 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:28:24.032 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:28:24.056 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:24.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:24.057 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:28:24.158 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:28:24.165 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 7, 8] out_osds: [10, 6, 2, 9, 11] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-08-19T14:28:24.165 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-19T14:28:24.182 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:24.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:24.183 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:28:24.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:23 smithi032 ceph-mon[20004]: pgmap v1161: 31 pgs: 1 active+undersized+remapped, 2 remapped+peering, 2 unknown, 4 peering, 22 active+clean; 620 KiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-08-19T14:28:24.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:23 smithi032 ceph-mon[20004]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 9854/6 objects degraded (164233.333%), 3 pgs degraded) 2024-08-19T14:28:24.284 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:28:24.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:23 smithi045 ceph-mon[20143]: pgmap v1161: 31 pgs: 1 active+undersized+remapped, 2 remapped+peering, 2 unknown, 4 peering, 22 active+clean; 620 KiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-08-19T14:28:24.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:23 smithi045 ceph-mon[20143]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 9854/6 objects degraded (164233.333%), 3 pgs degraded) 2024-08-19T14:28:24.310 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:24.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:24.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:28:24.412 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:28:24.437 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:24.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:24.439 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:28:24.542 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:28:24.567 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:24.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:24.569 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:28:24.670 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:28:24.695 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:24.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:24.696 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:28:24.797 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:28:24.823 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:24.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:24.824 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:28:24.925 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:28:24.954 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:24.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:24.955 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:28:25.055 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:28:25.080 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:25.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:25.081 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:28:25.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:24 smithi031 ceph-mon[16336]: 1.1b scrub starts 2024-08-19T14:28:25.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:24 smithi031 ceph-mon[16336]: 1.1b scrub ok 2024-08-19T14:28:25.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:24 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:28:25.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:24 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:28:25.136 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-08-19T14:28:25.182 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:28:25.196 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:24 smithi045 ceph-mon[20143]: 1.1b scrub starts 2024-08-19T14:28:25.197 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:24 smithi045 ceph-mon[20143]: 1.1b scrub ok 2024-08-19T14:28:25.197 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:24 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:28:25.197 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:24 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:28:25.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:25.208 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:25.208 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:28:25.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:24 smithi032 ceph-mon[20004]: 1.1b scrub starts 2024-08-19T14:28:25.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:24 smithi032 ceph-mon[20004]: 1.1b scrub ok 2024-08-19T14:28:25.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:24 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:28:25.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:24 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:28:25.309 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:28:25.331 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:25.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:25.332 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:28:25.433 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:28:25.454 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:25.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:25.456 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:28:25.557 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:28:25.580 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:25.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:25.581 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:28:25.588 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-08-19T14:28:25.589 INFO:tasks.radosbench:Beginning radosbench... 2024-08-19T14:28:25.589 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_4 2024-08-19T14:28:25.590 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd pool create unique_pool_4 16 2024-08-19T14:28:25.682 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:28:25.707 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:25.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:25.709 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:28:25.810 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:28:25.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:25.836 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:25.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:28:25.938 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:28:25.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:25.964 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:25.964 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:28:26.065 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:28:26.086 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:26.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:26.087 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:28:26.103 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:28:26.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:25 smithi031 ceph-mon[16336]: pgmap v1162: 31 pgs: 3 peering, 28 active+clean; 620 KiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 988 B/s rd, 1 op/s 2024-08-19T14:28:26.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:25 smithi031 ceph-mon[16336]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive) 2024-08-19T14:28:26.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:25 smithi031 ceph-mon[16336]: osdmap e508: 12 total, 12 up, 7 in 2024-08-19T14:28:26.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:25 smithi031 ceph-mon[16336]: 1.1c scrub starts 2024-08-19T14:28:26.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:25 smithi031 ceph-mon[16336]: 1.1c scrub ok 2024-08-19T14:28:26.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:25 smithi031 ceph-mon[16336]: 1.11 scrub starts 2024-08-19T14:28:26.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:25 smithi031 ceph-mon[16336]: 1.11 scrub ok 2024-08-19T14:28:26.142 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:28:26.188 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:28:26.213 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:26.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:26.214 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:28:26.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:25 smithi032 ceph-mon[20004]: pgmap v1162: 31 pgs: 3 peering, 28 active+clean; 620 KiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 988 B/s rd, 1 op/s 2024-08-19T14:28:26.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:25 smithi032 ceph-mon[20004]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive) 2024-08-19T14:28:26.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:25 smithi032 ceph-mon[20004]: osdmap e508: 12 total, 12 up, 7 in 2024-08-19T14:28:26.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:25 smithi032 ceph-mon[20004]: 1.1c scrub starts 2024-08-19T14:28:26.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:25 smithi032 ceph-mon[20004]: 1.1c scrub ok 2024-08-19T14:28:26.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:25 smithi032 ceph-mon[20004]: 1.11 scrub starts 2024-08-19T14:28:26.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:25 smithi032 ceph-mon[20004]: 1.11 scrub ok 2024-08-19T14:28:26.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:25 smithi045 ceph-mon[20143]: pgmap v1162: 31 pgs: 3 peering, 28 active+clean; 620 KiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 988 B/s rd, 1 op/s 2024-08-19T14:28:26.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:25 smithi045 ceph-mon[20143]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive) 2024-08-19T14:28:26.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:25 smithi045 ceph-mon[20143]: osdmap e508: 12 total, 12 up, 7 in 2024-08-19T14:28:26.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:25 smithi045 ceph-mon[20143]: 1.1c scrub starts 2024-08-19T14:28:26.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:25 smithi045 ceph-mon[20143]: 1.1c scrub ok 2024-08-19T14:28:26.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:25 smithi045 ceph-mon[20143]: 1.11 scrub starts 2024-08-19T14:28:26.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:25 smithi045 ceph-mon[20143]: 1.11 scrub ok 2024-08-19T14:28:26.315 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:28:26.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:26.339 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:26.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:28:26.374 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:28:26.440 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:28:26.461 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:26.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:26.462 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:28:26.563 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:28:26.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:26.591 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:26.591 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:28:26.693 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:28:26.713 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:26.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:26.714 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:28:26.815 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:28:26.828 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:28:26.828 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:28:26.838 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:26.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:26.839 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:28:26.845 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:28:26.846 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:28:26.862 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:28:26.862 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:28:26.871 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is unset 2024-08-19T14:28:26.880 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:28:26.880 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:28:26.898 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:28:26.898 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:28:26.916 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:28:26.916 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:28:26.933 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:28:26.933 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:28:26.940 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:28:26.951 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:28:26.951 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:28:26.961 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:26.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:26.962 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:28:26.970 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:28:26.970 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:28:26.987 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:28:26.987 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:28:27.004 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:28:27.004 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:28:27.022 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:28:27.022 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:28:27.063 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:28:27.084 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:27.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:27.085 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:28:27.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:26 smithi031 ceph-mon[16336]: 1.2 scrub starts 2024-08-19T14:28:27.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:26 smithi031 ceph-mon[16336]: 1.2 scrub ok 2024-08-19T14:28:27.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:26 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2586759846' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:28:27.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:26 smithi031 ceph-mon[16336]: 1.b scrub starts 2024-08-19T14:28:27.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:26 smithi031 ceph-mon[16336]: 1.b scrub ok 2024-08-19T14:28:27.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:26 smithi031 ceph-mon[16336]: 1.c scrub starts 2024-08-19T14:28:27.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:26 smithi031 ceph-mon[16336]: 1.c scrub ok 2024-08-19T14:28:27.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:26 smithi031 ceph-mon[16336]: 1.e scrub starts 2024-08-19T14:28:27.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:26 smithi031 ceph-mon[16336]: 1.e scrub ok 2024-08-19T14:28:27.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:26 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3513566535' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:28:27.186 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:28:27.212 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:27.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:27.213 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:28:27.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:26 smithi032 ceph-mon[20004]: 1.2 scrub starts 2024-08-19T14:28:27.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:26 smithi032 ceph-mon[20004]: 1.2 scrub ok 2024-08-19T14:28:27.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:26 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2586759846' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:28:27.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:26 smithi032 ceph-mon[20004]: 1.b scrub starts 2024-08-19T14:28:27.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:26 smithi032 ceph-mon[20004]: 1.b scrub ok 2024-08-19T14:28:27.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:26 smithi032 ceph-mon[20004]: 1.c scrub starts 2024-08-19T14:28:27.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:26 smithi032 ceph-mon[20004]: 1.c scrub ok 2024-08-19T14:28:27.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:26 smithi032 ceph-mon[20004]: 1.e scrub starts 2024-08-19T14:28:27.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:26 smithi032 ceph-mon[20004]: 1.e scrub ok 2024-08-19T14:28:27.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:26 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3513566535' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:28:27.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:26 smithi045 ceph-mon[20143]: 1.2 scrub starts 2024-08-19T14:28:27.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:26 smithi045 ceph-mon[20143]: 1.2 scrub ok 2024-08-19T14:28:27.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:26 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2586759846' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:28:27.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:26 smithi045 ceph-mon[20143]: 1.b scrub starts 2024-08-19T14:28:27.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:26 smithi045 ceph-mon[20143]: 1.b scrub ok 2024-08-19T14:28:27.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:26 smithi045 ceph-mon[20143]: 1.c scrub starts 2024-08-19T14:28:27.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:26 smithi045 ceph-mon[20143]: 1.c scrub ok 2024-08-19T14:28:27.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:26 smithi045 ceph-mon[20143]: 1.e scrub starts 2024-08-19T14:28:27.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:26 smithi045 ceph-mon[20143]: 1.e scrub ok 2024-08-19T14:28:27.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:26 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3513566535' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:28:27.314 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:28:27.340 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:27.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:27.341 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:28:27.442 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:28:27.464 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:27.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:27.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:28:27.566 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:28:27.593 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:27.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:27.594 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:28:27.695 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:28:27.719 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:27.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:27.720 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:28:27.821 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:28:27.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:27.848 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:27.849 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:28:27.950 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:28:27.976 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:27.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:27.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:28:28.078 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:28:28.104 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:28.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:28.105 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:28:28.206 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:28:28.232 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:28.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:28.234 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:28:28.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:27 smithi032 ceph-mon[20004]: pgmap v1164: 31 pgs: 31 active+clean; 620 KiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-08-19T14:28:28.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:27 smithi032 ceph-mon[20004]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:28:28.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:27 smithi032 ceph-mon[20004]: Cluster is now healthy 2024-08-19T14:28:28.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:27 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2586759846' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:28:28.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:27 smithi032 ceph-mon[20004]: osdmap e509: 12 total, 12 up, 7 in 2024-08-19T14:28:28.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:27 smithi032 ceph-mon[20004]: 1.1e scrub starts 2024-08-19T14:28:28.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:27 smithi032 ceph-mon[20004]: 1.1e scrub ok 2024-08-19T14:28:28.259 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:27 smithi032 ceph-mon[20004]: 1.5 scrub starts 2024-08-19T14:28:28.259 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:27 smithi032 ceph-mon[20004]: 1.5 scrub ok 2024-08-19T14:28:28.259 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:27 smithi032 ceph-mon[20004]: 1.7 scrub starts 2024-08-19T14:28:28.259 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:27 smithi032 ceph-mon[20004]: 1.7 scrub ok 2024-08-19T14:28:28.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:27 smithi045 ceph-mon[20143]: pgmap v1164: 31 pgs: 31 active+clean; 620 KiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-08-19T14:28:28.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:27 smithi045 ceph-mon[20143]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:28:28.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:27 smithi045 ceph-mon[20143]: Cluster is now healthy 2024-08-19T14:28:28.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:27 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2586759846' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:28:28.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:27 smithi045 ceph-mon[20143]: osdmap e509: 12 total, 12 up, 7 in 2024-08-19T14:28:28.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:27 smithi045 ceph-mon[20143]: 1.1e scrub starts 2024-08-19T14:28:28.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:27 smithi045 ceph-mon[20143]: 1.1e scrub ok 2024-08-19T14:28:28.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:27 smithi045 ceph-mon[20143]: 1.5 scrub starts 2024-08-19T14:28:28.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:27 smithi045 ceph-mon[20143]: 1.5 scrub ok 2024-08-19T14:28:28.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:27 smithi045 ceph-mon[20143]: 1.7 scrub starts 2024-08-19T14:28:28.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:27 smithi045 ceph-mon[20143]: 1.7 scrub ok 2024-08-19T14:28:28.335 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:28:28.360 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:28.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:28.361 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:28:28.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:27 smithi031 ceph-mon[16336]: pgmap v1164: 31 pgs: 31 active+clean; 620 KiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-08-19T14:28:28.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:27 smithi031 ceph-mon[16336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:28:28.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:27 smithi031 ceph-mon[16336]: Cluster is now healthy 2024-08-19T14:28:28.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:27 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2586759846' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:28:28.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:27 smithi031 ceph-mon[16336]: osdmap e509: 12 total, 12 up, 7 in 2024-08-19T14:28:28.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:27 smithi031 ceph-mon[16336]: 1.1e scrub starts 2024-08-19T14:28:28.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:27 smithi031 ceph-mon[16336]: 1.1e scrub ok 2024-08-19T14:28:28.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:27 smithi031 ceph-mon[16336]: 1.5 scrub starts 2024-08-19T14:28:28.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:27 smithi031 ceph-mon[16336]: 1.5 scrub ok 2024-08-19T14:28:28.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:27 smithi031 ceph-mon[16336]: 1.7 scrub starts 2024-08-19T14:28:28.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:27 smithi031 ceph-mon[16336]: 1.7 scrub ok 2024-08-19T14:28:28.461 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:28:28.484 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:28.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:28.485 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:28:28.586 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:28:28.612 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:28.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:28.613 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:28:28.714 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:28:28.739 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:28.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:28.740 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:28:28.841 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:28:28.866 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:28.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:28.867 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:28:28.968 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:28:28.993 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:28.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:28.994 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:28:29.095 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:28:29.121 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:29.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:29.122 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:28:29.223 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:28:29.249 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:29.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:29.250 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:28:29.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:28 smithi032 ceph-mon[20004]: 1.4 scrub starts 2024-08-19T14:28:29.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:28 smithi032 ceph-mon[20004]: 1.4 scrub ok 2024-08-19T14:28:29.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:28 smithi032 ceph-mon[20004]: 1.1 scrub starts 2024-08-19T14:28:29.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:28 smithi032 ceph-mon[20004]: 1.1 scrub ok 2024-08-19T14:28:29.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:28 smithi045 ceph-mon[20143]: 1.4 scrub starts 2024-08-19T14:28:29.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:28 smithi045 ceph-mon[20143]: 1.4 scrub ok 2024-08-19T14:28:29.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:28 smithi045 ceph-mon[20143]: 1.1 scrub starts 2024-08-19T14:28:29.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:28 smithi045 ceph-mon[20143]: 1.1 scrub ok 2024-08-19T14:28:29.351 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:28:29.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:28 smithi031 ceph-mon[16336]: 1.4 scrub starts 2024-08-19T14:28:29.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:28 smithi031 ceph-mon[16336]: 1.4 scrub ok 2024-08-19T14:28:29.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:28 smithi031 ceph-mon[16336]: 1.1 scrub starts 2024-08-19T14:28:29.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:28 smithi031 ceph-mon[16336]: 1.1 scrub ok 2024-08-19T14:28:29.376 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:29.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:29.377 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:28:29.478 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:28:29.503 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:29.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:29.504 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:28:29.605 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:28:29.631 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:29.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:29.632 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:28:29.733 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:28:29.759 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:29.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:29.760 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:28:29.861 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:28:29.884 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:29.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:29.885 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:28:29.986 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:28:30.012 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:30.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:30.013 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:28:30.114 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:28:30.116 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:28:30.116 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 20, 2024-08-19T14:28:30.116 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 600, 2024-08-19T14:28:30.116 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [ 2024-08-19T14:28:30.116 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.116 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25057.0:188741 3.26 3:65f5d80b:::benchmark_data_smithi045_59973_object57562:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e296)", 2024-08-19T14:28:30.116 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:19:02.154496+0000", 2024-08-19T14:28:30.116 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 567.40946825200001, 2024-08-19T14:28:30.117 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.047062189, 2024-08-19T14:28:30.117 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:28:30.117 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:28:30.117 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:28:30.117 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:28:30.117 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25057", 2024-08-19T14:28:30.117 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:28:30.117 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 188741 2024-08-19T14:28:30.117 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.117 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:28:30.117 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.117 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:28:30.117 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.154496+0000", 2024-08-19T14:28:30.118 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:28:30.118 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.118 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.118 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:28:30.118 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.154496+0000", 2024-08-19T14:28:30.118 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:28:30.118 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.118 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.118 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:28:30.118 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.154497+0000", 2024-08-19T14:28:30.118 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.5000000000000001e-07 2024-08-19T14:28:30.118 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.118 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.119 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:28:30.119 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.154502+0000", 2024-08-19T14:28:30.119 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0690000000000001e-06 2024-08-19T14:28:30.119 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.119 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.119 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:28:30.119 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.154514+0000", 2024-08-19T14:28:30.119 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1922e-05 2024-08-19T14:28:30.119 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.119 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.119 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:28:30.119 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.154522+0000", 2024-08-19T14:28:30.119 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.4730000000000003e-06 2024-08-19T14:28:30.120 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.120 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.120 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:28:30.120 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.275482+0000", 2024-08-19T14:28:30.120 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.12095984899999999 2024-08-19T14:28:30.120 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.120 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.120 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:28:30.120 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.275489+0000", 2024-08-19T14:28:30.120 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.8949999999999998e-06 2024-08-19T14:28:30.120 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.120 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.120 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:28:30.121 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:03.201475+0000", 2024-08-19T14:28:30.121 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.92598516200000003 2024-08-19T14:28:30.121 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.121 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.121 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:28:30.121 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:03.201559+0000", 2024-08-19T14:28:30.121 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.3968999999999995e-05 2024-08-19T14:28:30.121 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:28:30.121 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:28:30.121 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:28:30.121 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.121 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.121 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25057.0:188765 3.26 3:6524f2fe:::benchmark_data_smithi045_59973_object57586:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-08-19T14:28:30.121 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:19:02.192404+0000", 2024-08-19T14:28:30.122 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 567.37156115300002, 2024-08-19T14:28:30.122 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0202571499999999, 2024-08-19T14:28:30.122 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:28:30.122 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:28:30.122 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:28:30.122 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:28:30.122 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25057", 2024-08-19T14:28:30.122 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:28:30.122 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 188765 2024-08-19T14:28:30.122 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.122 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:28:30.122 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.123 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:28:30.123 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.192404+0000", 2024-08-19T14:28:30.123 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:28:30.123 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.124 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.124 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:28:30.124 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.192404+0000", 2024-08-19T14:28:30.124 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:28:30.124 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.124 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.124 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:28:30.124 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.192406+0000", 2024-08-19T14:28:30.124 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.3140000000000002e-06 2024-08-19T14:28:30.124 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.124 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.124 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:28:30.124 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.192417+0000", 2024-08-19T14:28:30.125 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0839000000000001e-05 2024-08-19T14:28:30.125 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.125 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.125 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:28:30.125 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.192440+0000", 2024-08-19T14:28:30.125 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.2911000000000001e-05 2024-08-19T14:28:30.125 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.125 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.125 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:28:30.125 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.192450+0000", 2024-08-19T14:28:30.125 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0292000000000001e-05 2024-08-19T14:28:30.125 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.125 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.126 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:28:30.126 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.275507+0000", 2024-08-19T14:28:30.126 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.083056873000000003 2024-08-19T14:28:30.126 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.126 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.126 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:28:30.126 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.275512+0000", 2024-08-19T14:28:30.126 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.8799999999999999e-06 2024-08-19T14:28:30.126 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.126 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.126 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:28:30.126 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:03.201920+0000", 2024-08-19T14:28:30.126 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.92640795799999998 2024-08-19T14:28:30.126 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.127 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.127 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:28:30.127 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:03.202036+0000", 2024-08-19T14:28:30.127 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000116364 2024-08-19T14:28:30.127 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.127 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.127 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 1,4", 2024-08-19T14:28:30.127 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:03.202142+0000", 2024-08-19T14:28:30.127 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000106115 2024-08-19T14:28:30.127 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.127 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.127 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:28:30.127 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:03.206756+0000", 2024-08-19T14:28:30.127 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0046137319999999997 2024-08-19T14:28:30.128 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.128 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.128 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:30.128 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:03.206778+0000", 2024-08-19T14:28:30.128 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.2056999999999999e-05 2024-08-19T14:28:30.128 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.128 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.128 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:30.128 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:03.212547+0000", 2024-08-19T14:28:30.128 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0057695740000000004 2024-08-19T14:28:30.128 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.128 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.128 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:28:30.128 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:03.212603+0000", 2024-08-19T14:28:30.128 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.5189999999999998e-05 2024-08-19T14:28:30.129 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.129 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.129 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:28:30.129 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:03.212661+0000", 2024-08-19T14:28:30.129 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.8050999999999997e-05 2024-08-19T14:28:30.129 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:28:30.129 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:28:30.129 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:28:30.129 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.129 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.129 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25333.0:22685 4.b 4:d7f4804b:::benchmark_data_smithi045_71562_object22684:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-08-19T14:28:30.129 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:20:41.347757+0000", 2024-08-19T14:28:30.129 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 468.21620740600002, 2024-08-19T14:28:30.129 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.33048444500000002, 2024-08-19T14:28:30.129 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:28:30.130 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:28:30.130 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:28:30.130 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:28:30.130 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25333", 2024-08-19T14:28:30.130 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:28:30.130 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 22685 2024-08-19T14:28:30.130 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.130 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:28:30.130 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.130 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:28:30.130 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.347757+0000", 2024-08-19T14:28:30.130 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:28:30.130 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.131 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.131 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:28:30.131 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.347757+0000", 2024-08-19T14:28:30.131 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:28:30.131 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.131 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.131 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:28:30.131 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.347759+0000", 2024-08-19T14:28:30.131 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4410000000000001e-06 2024-08-19T14:28:30.131 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.131 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.131 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:28:30.132 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.347940+0000", 2024-08-19T14:28:30.132 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00018171800000000001 2024-08-19T14:28:30.132 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.132 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.132 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:28:30.132 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.347953+0000", 2024-08-19T14:28:30.132 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2724e-05 2024-08-19T14:28:30.132 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.132 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.132 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:28:30.132 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.347960+0000", 2024-08-19T14:28:30.132 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.3320000000000003e-06 2024-08-19T14:28:30.132 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.132 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.132 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:28:30.133 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.348009+0000", 2024-08-19T14:28:30.133 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.8882999999999999e-05 2024-08-19T14:28:30.133 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.133 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.133 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:28:30.133 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.348129+0000", 2024-08-19T14:28:30.133 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012009 2024-08-19T14:28:30.133 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.133 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.133 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 1,4", 2024-08-19T14:28:30.133 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.348257+0000", 2024-08-19T14:28:30.133 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012718399999999999 2024-08-19T14:28:30.133 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.133 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.134 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:28:30.134 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.349748+0000", 2024-08-19T14:28:30.134 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.001491579 2024-08-19T14:28:30.134 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.134 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.134 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:30.134 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.350166+0000", 2024-08-19T14:28:30.134 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00041812000000000002 2024-08-19T14:28:30.134 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.134 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.134 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:30.134 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.351389+0000", 2024-08-19T14:28:30.134 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0012227119999999999 2024-08-19T14:28:30.135 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.135 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.135 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:28:30.135 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.678185+0000", 2024-08-19T14:28:30.135 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.32679576199999999 2024-08-19T14:28:30.135 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.135 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.135 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:28:30.135 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.678242+0000", 2024-08-19T14:28:30.135 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.6900000000000001e-05 2024-08-19T14:28:30.135 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:28:30.135 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:28:30.135 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:28:30.135 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.136 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.136 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop_reply(client.25333.0:22685 4.b e341/335 ondisk, result = 0)", 2024-08-19T14:28:30.136 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:20:41.350582+0000", 2024-08-19T14:28:30.136 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 468.21338298799998, 2024-08-19T14:28:30.136 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.32767402200000001, 2024-08-19T14:28:30.136 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:28:30.136 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:28:30.136 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-08-19T14:28:30.136 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:28:30.136 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.136 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:28:30.136 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.350582+0000", 2024-08-19T14:28:30.136 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:28:30.136 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.136 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.136 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:28:30.137 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.350582+0000", 2024-08-19T14:28:30.137 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:28:30.137 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.137 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.137 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:28:30.137 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.350582+0000", 2024-08-19T14:28:30.137 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.1999999999999999e-07 2024-08-19T14:28:30.137 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.137 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.137 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:28:30.137 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.350587+0000", 2024-08-19T14:28:30.137 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.6870000000000002e-06 2024-08-19T14:28:30.137 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.137 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.137 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:28:30.138 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.350598+0000", 2024-08-19T14:28:30.138 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0781999999999999e-05 2024-08-19T14:28:30.138 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.138 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.138 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:28:30.138 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.350605+0000", 2024-08-19T14:28:30.138 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.0419999999999997e-06 2024-08-19T14:28:30.138 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.138 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.138 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:28:30.138 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.351377+0000", 2024-08-19T14:28:30.138 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00077201899999999998 2024-08-19T14:28:30.138 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.139 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.139 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:28:30.139 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.351384+0000", 2024-08-19T14:28:30.139 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.917e-06 2024-08-19T14:28:30.139 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.139 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.139 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:28:30.139 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.678256+0000", 2024-08-19T14:28:30.139 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.32687185499999999 2024-08-19T14:28:30.139 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:28:30.139 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:28:30.139 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:28:30.139 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.139 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.139 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.25333.0:22687 4.7 e341/335 4:efd17a0f:::benchmark_data_smithi045_71562_object22686:head v 341'1401, mlcod=341'1397)", 2024-08-19T14:28:30.140 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:20:41.351262+0000", 2024-08-19T14:28:30.140 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 468.21270287700003, 2024-08-19T14:28:30.140 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.32996297600000002, 2024-08-19T14:28:30.140 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:28:30.140 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:28:30.140 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:28:30.140 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:28:30.140 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.140 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:28:30.140 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.351262+0000", 2024-08-19T14:28:30.140 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:28:30.140 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.140 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.140 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:28:30.141 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.351262+0000", 2024-08-19T14:28:30.141 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:28:30.141 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.141 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.141 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:28:30.141 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.351262+0000", 2024-08-19T14:28:30.141 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.0900000000000002e-07 2024-08-19T14:28:30.141 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.141 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.141 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:28:30.141 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.351289+0000", 2024-08-19T14:28:30.141 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.7036999999999999e-05 2024-08-19T14:28:30.141 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.142 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.142 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:28:30.142 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.351299+0000", 2024-08-19T14:28:30.142 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.9299999999999998e-06 2024-08-19T14:28:30.142 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.142 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.142 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:28:30.142 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.351305+0000", 2024-08-19T14:28:30.142 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.9229999999999999e-06 2024-08-19T14:28:30.142 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.142 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.142 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:28:30.142 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.351361+0000", 2024-08-19T14:28:30.142 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.6311999999999999e-05 2024-08-19T14:28:30.143 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.143 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.143 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:28:30.143 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.351380+0000", 2024-08-19T14:28:30.143 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.861e-05 2024-08-19T14:28:30.143 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.143 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.143 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:28:30.143 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.681181+0000", 2024-08-19T14:28:30.143 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.32980058099999998 2024-08-19T14:28:30.143 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.143 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.144 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:28:30.144 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.681225+0000", 2024-08-19T14:28:30.144 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4174000000000001e-05 2024-08-19T14:28:30.144 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:28:30.144 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:28:30.144 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:28:30.144 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.144 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.144 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.25333.0:22691 4.7 e341/335 4:e60184b4:::benchmark_data_smithi045_71562_object22690:head v 341'1402, mlcod=341'1397)", 2024-08-19T14:28:30.144 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:20:41.352142+0000", 2024-08-19T14:28:30.144 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 468.21182266, 2024-08-19T14:28:30.145 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.32972854099999999, 2024-08-19T14:28:30.145 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:28:30.145 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:28:30.145 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:28:30.145 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:28:30.145 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.145 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:28:30.145 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.352142+0000", 2024-08-19T14:28:30.145 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:28:30.145 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.145 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.145 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:28:30.145 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.352142+0000", 2024-08-19T14:28:30.146 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:28:30.146 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.146 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.146 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:28:30.146 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.352143+0000", 2024-08-19T14:28:30.146 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.5199999999999996e-07 2024-08-19T14:28:30.146 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.146 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.146 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:28:30.146 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.352196+0000", 2024-08-19T14:28:30.146 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.3468999999999999e-05 2024-08-19T14:28:30.146 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.146 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.146 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:28:30.147 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.678332+0000", 2024-08-19T14:28:30.147 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.32613542000000001 2024-08-19T14:28:30.147 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.147 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.147 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:28:30.147 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.678342+0000", 2024-08-19T14:28:30.147 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0696999999999999e-05 2024-08-19T14:28:30.147 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.147 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.147 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:28:30.147 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.679002+0000", 2024-08-19T14:28:30.147 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000659945 2024-08-19T14:28:30.147 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.147 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.148 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:28:30.148 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.679024+0000", 2024-08-19T14:28:30.148 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.1693999999999999e-05 2024-08-19T14:28:30.148 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.148 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.148 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:28:30.148 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.681816+0000", 2024-08-19T14:28:30.148 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0027922590000000001 2024-08-19T14:28:30.148 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.148 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.148 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:28:30.148 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.681871+0000", 2024-08-19T14:28:30.148 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.4305000000000003e-05 2024-08-19T14:28:30.148 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:28:30.148 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:28:30.149 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:28:30.149 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.149 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.149 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25333.0:22700 4.b 4:d65b8e70:::benchmark_data_smithi045_71562_object22699:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-08-19T14:28:30.149 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:20:41.353478+0000", 2024-08-19T14:28:30.149 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 468.21048644000001, 2024-08-19T14:28:30.149 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.32867489300000002, 2024-08-19T14:28:30.149 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:28:30.149 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:28:30.149 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:28:30.149 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:28:30.149 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25333", 2024-08-19T14:28:30.149 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:28:30.149 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 22700 2024-08-19T14:28:30.150 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.150 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:28:30.150 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.150 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:28:30.150 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.353478+0000", 2024-08-19T14:28:30.150 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:28:30.150 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.150 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.150 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:28:30.150 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.353478+0000", 2024-08-19T14:28:30.150 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:28:30.150 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.150 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.150 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:28:30.151 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.353480+0000", 2024-08-19T14:28:30.151 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6980000000000001e-06 2024-08-19T14:28:30.151 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.151 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.151 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:28:30.151 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.353537+0000", 2024-08-19T14:28:30.151 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.6724000000000001e-05 2024-08-19T14:28:30.151 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.151 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.151 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:28:30.151 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.677867+0000", 2024-08-19T14:28:30.151 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.32433008200000002 2024-08-19T14:28:30.151 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.151 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.152 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:28:30.152 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.677885+0000", 2024-08-19T14:28:30.152 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.8405000000000001e-05 2024-08-19T14:28:30.152 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.152 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.152 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:28:30.152 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.678314+0000", 2024-08-19T14:28:30.152 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00042872500000000003 2024-08-19T14:28:30.152 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.152 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.152 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:28:30.152 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.678522+0000", 2024-08-19T14:28:30.152 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00020833599999999999 2024-08-19T14:28:30.152 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.153 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.153 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 1,4", 2024-08-19T14:28:30.153 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.678691+0000", 2024-08-19T14:28:30.153 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00016871200000000001 2024-08-19T14:28:30.153 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.153 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.153 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:28:30.153 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.681852+0000", 2024-08-19T14:28:30.153 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0031611849999999999 2024-08-19T14:28:30.153 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.153 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.153 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:30.153 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.681921+0000", 2024-08-19T14:28:30.154 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.9191999999999998e-05 2024-08-19T14:28:30.154 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.154 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.154 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:30.154 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.682059+0000", 2024-08-19T14:28:30.154 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013730400000000001 2024-08-19T14:28:30.154 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.154 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.154 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:28:30.154 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.682113+0000", 2024-08-19T14:28:30.154 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.4840999999999999e-05 2024-08-19T14:28:30.154 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.155 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.155 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:28:30.155 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.682153+0000", 2024-08-19T14:28:30.155 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.9688999999999999e-05 2024-08-19T14:28:30.155 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:28:30.155 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:28:30.155 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:28:30.155 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.155 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.155 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25813.0:15525 5.3 5:c601536c:::benchmark_data_smithi045_85309_object15524:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e426)", 2024-08-19T14:28:30.155 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:24:52.832423+0000", 2024-08-19T14:28:30.155 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 216.73154117000001, 2024-08-19T14:28:30.155 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.26183261899999999, 2024-08-19T14:28:30.155 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:28:30.155 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:28:30.156 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:28:30.156 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:28:30.156 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25813", 2024-08-19T14:28:30.156 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:28:30.156 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 15525 2024-08-19T14:28:30.156 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.156 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:28:30.156 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.156 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:28:30.156 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:52.832423+0000", 2024-08-19T14:28:30.156 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:28:30.156 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.156 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.156 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:28:30.156 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:52.832423+0000", 2024-08-19T14:28:30.157 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:28:30.157 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.157 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.157 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:28:30.157 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:52.832424+0000", 2024-08-19T14:28:30.157 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.5899999999999995e-07 2024-08-19T14:28:30.157 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.157 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.157 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:28:30.157 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:52.832480+0000", 2024-08-19T14:28:30.157 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.5377999999999997e-05 2024-08-19T14:28:30.157 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.157 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.157 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:28:30.158 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:52.832489+0000", 2024-08-19T14:28:30.158 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.6169999999999996e-06 2024-08-19T14:28:30.158 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.158 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.158 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:28:30.158 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:52.832494+0000", 2024-08-19T14:28:30.158 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.6720000000000003e-06 2024-08-19T14:28:30.158 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.158 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.158 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:28:30.158 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:52.832531+0000", 2024-08-19T14:28:30.158 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7120999999999999e-05 2024-08-19T14:28:30.158 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.158 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.159 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:28:30.159 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:52.832617+0000", 2024-08-19T14:28:30.159 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.5587000000000006e-05 2024-08-19T14:28:30.159 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.159 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.159 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,5", 2024-08-19T14:28:30.159 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:52.832694+0000", 2024-08-19T14:28:30.159 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.7669000000000005e-05 2024-08-19T14:28:30.159 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.159 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.159 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:28:30.159 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:52.834430+0000", 2024-08-19T14:28:30.159 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.001735225 2024-08-19T14:28:30.159 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.160 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.160 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:30.160 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:52.836161+0000", 2024-08-19T14:28:30.160 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0017313960000000001 2024-08-19T14:28:30.160 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.160 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.160 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:30.160 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:53.094158+0000", 2024-08-19T14:28:30.160 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.257997003 2024-08-19T14:28:30.160 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.160 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.160 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:28:30.160 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:53.094209+0000", 2024-08-19T14:28:30.160 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0763000000000002e-05 2024-08-19T14:28:30.161 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.161 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.161 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:28:30.161 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:53.094256+0000", 2024-08-19T14:28:30.161 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7429000000000003e-05 2024-08-19T14:28:30.161 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:28:30.161 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:28:30.161 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:28:30.161 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.161 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.161 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25813.0:15529 5.3 5:c6799b39:::benchmark_data_smithi045_85309_object15528:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e426)", 2024-08-19T14:28:30.161 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:24:52.833224+0000", 2024-08-19T14:28:30.161 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 216.73074026099999, 2024-08-19T14:28:30.161 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.26288265900000002, 2024-08-19T14:28:30.162 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:28:30.162 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:28:30.162 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:28:30.162 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:28:30.162 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25813", 2024-08-19T14:28:30.162 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:28:30.162 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 15529 2024-08-19T14:28:30.162 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.162 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:28:30.162 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.162 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:28:30.162 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:52.833224+0000", 2024-08-19T14:28:30.162 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:28:30.163 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.163 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.163 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:28:30.163 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:52.833224+0000", 2024-08-19T14:28:30.163 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:28:30.163 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.163 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.163 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:28:30.163 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:52.833225+0000", 2024-08-19T14:28:30.163 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.3000000000000001e-07 2024-08-19T14:28:30.163 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.163 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.163 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:28:30.164 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:52.833253+0000", 2024-08-19T14:28:30.164 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.7852999999999999e-05 2024-08-19T14:28:30.164 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.164 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.164 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:28:30.164 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:52.833260+0000", 2024-08-19T14:28:30.164 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.61e-06 2024-08-19T14:28:30.164 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.164 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.164 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:28:30.164 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:52.833266+0000", 2024-08-19T14:28:30.164 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.614e-06 2024-08-19T14:28:30.164 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.164 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.165 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:28:30.165 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:52.833376+0000", 2024-08-19T14:28:30.165 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000110365 2024-08-19T14:28:30.165 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.165 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.165 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:28:30.165 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:52.833512+0000", 2024-08-19T14:28:30.165 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013533800000000001 2024-08-19T14:28:30.165 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.165 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.165 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,5", 2024-08-19T14:28:30.165 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:52.833748+0000", 2024-08-19T14:28:30.165 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00023644700000000001 2024-08-19T14:28:30.166 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.166 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.166 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:28:30.166 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:52.834588+0000", 2024-08-19T14:28:30.166 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00083990900000000001 2024-08-19T14:28:30.166 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.166 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.166 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:30.166 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:52.836202+0000", 2024-08-19T14:28:30.166 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0016136270000000001 2024-08-19T14:28:30.166 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.166 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.166 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:30.167 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:53.096008+0000", 2024-08-19T14:28:30.167 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.25980599500000001 2024-08-19T14:28:30.167 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.167 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.167 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:28:30.167 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:53.096076+0000", 2024-08-19T14:28:30.167 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.8532e-05 2024-08-19T14:28:30.167 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.167 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.167 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:28:30.167 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:53.096107+0000", 2024-08-19T14:28:30.167 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.0839000000000001e-05 2024-08-19T14:28:30.167 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:28:30.168 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:28:30.168 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:28:30.168 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.168 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.168 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25813.0:15536 5.3 5:cba1709d:::benchmark_data_smithi045_85309_object15535:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e426)", 2024-08-19T14:28:30.168 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:24:52.835227+0000", 2024-08-19T14:28:30.168 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 216.72873805399999, 2024-08-19T14:28:30.168 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.26103143200000001, 2024-08-19T14:28:30.168 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:28:30.168 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:28:30.168 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:28:30.168 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:28:30.168 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25813", 2024-08-19T14:28:30.168 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:28:30.169 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 15536 2024-08-19T14:28:30.169 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.169 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:28:30.169 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.169 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:28:30.169 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:52.835227+0000", 2024-08-19T14:28:30.169 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:28:30.169 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.169 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.169 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:28:30.169 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:52.835227+0000", 2024-08-19T14:28:30.169 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:28:30.169 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.170 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.170 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:28:30.170 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:52.835227+0000", 2024-08-19T14:28:30.170 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.54e-07 2024-08-19T14:28:30.170 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.170 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.170 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:28:30.170 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:52.835266+0000", 2024-08-19T14:28:30.170 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.8816000000000002e-05 2024-08-19T14:28:30.170 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.170 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.170 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:28:30.170 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:52.835279+0000", 2024-08-19T14:28:30.170 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.242e-05 2024-08-19T14:28:30.171 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.171 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.171 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:28:30.171 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:52.835285+0000", 2024-08-19T14:28:30.171 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.7039999999999997e-06 2024-08-19T14:28:30.171 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.171 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.171 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:28:30.171 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:52.835329+0000", 2024-08-19T14:28:30.171 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.3108000000000001e-05 2024-08-19T14:28:30.171 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.171 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.171 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:28:30.171 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:52.835399+0000", 2024-08-19T14:28:30.172 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.0006999999999996e-05 2024-08-19T14:28:30.172 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.172 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.172 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,5", 2024-08-19T14:28:30.172 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:52.835576+0000", 2024-08-19T14:28:30.172 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000176995 2024-08-19T14:28:30.172 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.172 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.172 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:28:30.172 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:52.836602+0000", 2024-08-19T14:28:30.172 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0010267399999999999 2024-08-19T14:28:30.172 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.172 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.173 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:30.173 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:52.837148+0000", 2024-08-19T14:28:30.173 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00054603900000000001 2024-08-19T14:28:30.173 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.173 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.173 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:30.173 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:53.096162+0000", 2024-08-19T14:28:30.173 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.25901379099999999 2024-08-19T14:28:30.173 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.173 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.173 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:28:30.173 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:53.096217+0000", 2024-08-19T14:28:30.173 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.4599999999999999e-05 2024-08-19T14:28:30.173 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.174 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.174 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:28:30.174 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:53.096258+0000", 2024-08-19T14:28:30.174 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.1357999999999999e-05 2024-08-19T14:28:30.174 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:28:30.174 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:28:30.174 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:28:30.174 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.174 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.174 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25813.0:237746 5.b 5:dfc85b88:::benchmark_data_smithi045_85309_object117493:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e494)", 2024-08-19T14:28:30.174 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:28:07.094689+0000", 2024-08-19T14:28:30.174 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 22.469276043000001, 2024-08-19T14:28:30.174 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.21855882500000001, 2024-08-19T14:28:30.174 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:28:30.175 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:28:30.175 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:28:30.175 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:28:30.175 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25813", 2024-08-19T14:28:30.175 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:28:30.175 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 237746 2024-08-19T14:28:30.175 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.175 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:28:30.175 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.175 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:28:30.175 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.094689+0000", 2024-08-19T14:28:30.175 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:28:30.175 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.175 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.176 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:28:30.176 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.094689+0000", 2024-08-19T14:28:30.176 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:28:30.176 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.176 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.176 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:28:30.176 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.094690+0000", 2024-08-19T14:28:30.176 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.801e-06 2024-08-19T14:28:30.176 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.176 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.176 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:28:30.176 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.094699+0000", 2024-08-19T14:28:30.176 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.5250000000000005e-06 2024-08-19T14:28:30.176 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.177 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.177 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:28:30.177 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.094713+0000", 2024-08-19T14:28:30.177 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4173e-05 2024-08-19T14:28:30.177 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.177 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.177 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:28:30.177 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.094724+0000", 2024-08-19T14:28:30.177 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1276e-05 2024-08-19T14:28:30.177 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.177 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.177 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:28:30.177 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.094795+0000", 2024-08-19T14:28:30.178 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.0828999999999996e-05 2024-08-19T14:28:30.178 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.178 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.178 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:28:30.178 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.094809+0000", 2024-08-19T14:28:30.178 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3835999999999999e-05 2024-08-19T14:28:30.178 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.178 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.178 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:28:30.178 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.297164+0000", 2024-08-19T14:28:30.178 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.20235447000000001 2024-08-19T14:28:30.178 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.178 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.178 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:28:30.179 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.297325+0000", 2024-08-19T14:28:30.179 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00016111200000000001 2024-08-19T14:28:30.179 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.179 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.179 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 5,7,9", 2024-08-19T14:28:30.179 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.297479+0000", 2024-08-19T14:28:30.179 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000154597 2024-08-19T14:28:30.179 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.179 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.179 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:30.180 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.302943+0000", 2024-08-19T14:28:30.180 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0054640610000000001 2024-08-19T14:28:30.180 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.180 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.180 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:28:30.180 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.302960+0000", 2024-08-19T14:28:30.180 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6339999999999999e-05 2024-08-19T14:28:30.180 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.180 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.180 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:30.180 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.313009+0000", 2024-08-19T14:28:30.180 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.010049677 2024-08-19T14:28:30.180 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.181 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.181 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:30.181 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.313072+0000", 2024-08-19T14:28:30.181 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.2273000000000005e-05 2024-08-19T14:28:30.181 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.181 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.181 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:28:30.181 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.313133+0000", 2024-08-19T14:28:30.181 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.1699999999999995e-05 2024-08-19T14:28:30.181 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.181 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.181 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:28:30.181 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.313247+0000", 2024-08-19T14:28:30.181 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00011415499999999999 2024-08-19T14:28:30.181 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:28:30.182 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:28:30.182 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:28:30.182 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.182 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.182 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25813.0:237766 5.b 5:d29ec4b5:::benchmark_data_smithi045_85309_object117513:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e494)", 2024-08-19T14:28:30.182 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:28:07.094746+0000", 2024-08-19T14:28:30.182 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 22.469218741999999, 2024-08-19T14:28:30.182 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.21897851500000001, 2024-08-19T14:28:30.182 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:28:30.182 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:28:30.182 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:28:30.182 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:28:30.182 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25813", 2024-08-19T14:28:30.182 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:28:30.183 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 237766 2024-08-19T14:28:30.183 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.183 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:28:30.183 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.183 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:28:30.183 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.094746+0000", 2024-08-19T14:28:30.183 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:28:30.183 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.183 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.183 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:28:30.183 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.094746+0000", 2024-08-19T14:28:30.183 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:28:30.183 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.183 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.184 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:28:30.184 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.094746+0000", 2024-08-19T14:28:30.184 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1500000000000005e-07 2024-08-19T14:28:30.184 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.184 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.184 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:28:30.184 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.094750+0000", 2024-08-19T14:28:30.184 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.1489999999999998e-06 2024-08-19T14:28:30.184 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.184 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.184 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:28:30.184 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.094758+0000", 2024-08-19T14:28:30.184 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.2090000000000005e-06 2024-08-19T14:28:30.184 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.184 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.185 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:28:30.185 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.094764+0000", 2024-08-19T14:28:30.185 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.0859999999999998e-06 2024-08-19T14:28:30.185 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.185 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.185 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:28:30.185 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.094842+0000", 2024-08-19T14:28:30.185 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.7929000000000006e-05 2024-08-19T14:28:30.185 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.185 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.185 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:28:30.185 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.094847+0000", 2024-08-19T14:28:30.185 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.3580000000000002e-06 2024-08-19T14:28:30.185 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.186 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.186 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:28:30.186 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.303001+0000", 2024-08-19T14:28:30.186 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.20815415500000001 2024-08-19T14:28:30.186 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.186 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.186 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:28:30.186 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.303144+0000", 2024-08-19T14:28:30.186 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00014260999999999999 2024-08-19T14:28:30.186 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.186 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.186 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 5,7,9", 2024-08-19T14:28:30.186 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.303304+0000", 2024-08-19T14:28:30.186 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000160342 2024-08-19T14:28:30.187 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.187 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.187 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:30.187 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.312836+0000", 2024-08-19T14:28:30.187 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0095319470000000007 2024-08-19T14:28:30.187 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.187 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.187 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:28:30.187 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.313028+0000", 2024-08-19T14:28:30.187 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000192165 2024-08-19T14:28:30.187 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.187 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.187 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:30.188 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.313296+0000", 2024-08-19T14:28:30.188 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00026788199999999999 2024-08-19T14:28:30.188 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.188 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.188 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:30.188 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.313571+0000", 2024-08-19T14:28:30.188 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00027463299999999998 2024-08-19T14:28:30.188 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.188 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.188 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:28:30.188 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.313624+0000", 2024-08-19T14:28:30.188 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.3044000000000002e-05 2024-08-19T14:28:30.188 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.189 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.189 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:28:30.189 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.313724+0000", 2024-08-19T14:28:30.189 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000100491 2024-08-19T14:28:30.189 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:28:30.189 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:28:30.189 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:28:30.189 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.189 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.189 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25813.0:237773 5.b 5:d874f9fd:::benchmark_data_smithi045_85309_object117520:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e494)", 2024-08-19T14:28:30.189 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:28:07.094775+0000", 2024-08-19T14:28:30.189 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 22.469189750999998, 2024-08-19T14:28:30.189 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.21923524999999999, 2024-08-19T14:28:30.189 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:28:30.189 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:28:30.189 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:28:30.190 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:28:30.190 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25813", 2024-08-19T14:28:30.190 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:28:30.190 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 237773 2024-08-19T14:28:30.190 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.190 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:28:30.190 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.190 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:28:30.190 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.094775+0000", 2024-08-19T14:28:30.190 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:28:30.190 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.190 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.190 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:28:30.191 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.094775+0000", 2024-08-19T14:28:30.191 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:28:30.191 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.191 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.191 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:28:30.191 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.094775+0000", 2024-08-19T14:28:30.191 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.0900000000000002e-07 2024-08-19T14:28:30.191 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.191 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.191 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:28:30.191 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.094778+0000", 2024-08-19T14:28:30.191 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.6809999999999998e-06 2024-08-19T14:28:30.191 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.191 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.191 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:28:30.192 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.094785+0000", 2024-08-19T14:28:30.192 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.9070000000000001e-06 2024-08-19T14:28:30.192 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.192 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.192 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:28:30.192 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.094791+0000", 2024-08-19T14:28:30.192 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.7459999999999997e-06 2024-08-19T14:28:30.192 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.192 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.192 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:28:30.192 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.094882+0000", 2024-08-19T14:28:30.192 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.1512000000000001e-05 2024-08-19T14:28:30.192 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.192 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.193 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:28:30.193 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.094890+0000", 2024-08-19T14:28:30.193 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.3849999999999996e-06 2024-08-19T14:28:30.193 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.193 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.193 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:28:30.193 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.308208+0000", 2024-08-19T14:28:30.193 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.21331883400000001 2024-08-19T14:28:30.193 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.193 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.193 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:28:30.193 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.308354+0000", 2024-08-19T14:28:30.193 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000145978 2024-08-19T14:28:30.193 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.194 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.194 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 5,7,9", 2024-08-19T14:28:30.194 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.308506+0000", 2024-08-19T14:28:30.194 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00015196099999999999 2024-08-19T14:28:30.194 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.194 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.194 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:28:30.194 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.313259+0000", 2024-08-19T14:28:30.194 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0047530309999999996 2024-08-19T14:28:30.194 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.194 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.194 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:30.194 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.313771+0000", 2024-08-19T14:28:30.194 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00051169099999999999 2024-08-19T14:28:30.195 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.195 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.195 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:30.195 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.313809+0000", 2024-08-19T14:28:30.195 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.8024000000000001e-05 2024-08-19T14:28:30.195 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.195 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.195 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:30.195 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.313844+0000", 2024-08-19T14:28:30.195 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.5036999999999997e-05 2024-08-19T14:28:30.195 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.195 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.195 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:28:30.195 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.313902+0000", 2024-08-19T14:28:30.196 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.8109999999999998e-05 2024-08-19T14:28:30.196 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.196 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.196 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:28:30.196 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.314010+0000", 2024-08-19T14:28:30.196 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000107944 2024-08-19T14:28:30.196 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:28:30.196 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:28:30.196 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:28:30.196 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.196 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.196 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25813.0:238547 5.b 5:d56f9220:::benchmark_data_smithi045_85309_object118294:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e498)", 2024-08-19T14:28:30.196 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:28:09.849892+0000", 2024-08-19T14:28:30.196 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 19.714072523999999, 2024-08-19T14:28:30.197 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.13644729799999999, 2024-08-19T14:28:30.197 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:28:30.197 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:28:30.197 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:28:30.197 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:28:30.197 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25813", 2024-08-19T14:28:30.197 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:28:30.197 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 238547 2024-08-19T14:28:30.197 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.197 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:28:30.197 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.197 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:28:30.197 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.849892+0000", 2024-08-19T14:28:30.197 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:28:30.197 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.198 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.198 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:28:30.198 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.849892+0000", 2024-08-19T14:28:30.198 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:28:30.198 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.198 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.198 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:28:30.198 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.849892+0000", 2024-08-19T14:28:30.198 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.05e-07 2024-08-19T14:28:30.198 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.198 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.198 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:28:30.198 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.849894+0000", 2024-08-19T14:28:30.198 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4890000000000001e-06 2024-08-19T14:28:30.199 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.199 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.199 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:28:30.199 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.849898+0000", 2024-08-19T14:28:30.199 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.617e-06 2024-08-19T14:28:30.199 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.199 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.199 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:28:30.199 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.849902+0000", 2024-08-19T14:28:30.199 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7529999999999999e-06 2024-08-19T14:28:30.199 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.199 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.199 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:28:30.200 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.850730+0000", 2024-08-19T14:28:30.200 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00082774599999999997 2024-08-19T14:28:30.200 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.200 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.200 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:28:30.200 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.850735+0000", 2024-08-19T14:28:30.200 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0320000000000003e-06 2024-08-19T14:28:30.200 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.200 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.200 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:28:30.200 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.948239+0000", 2024-08-19T14:28:30.200 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.097503653999999995 2024-08-19T14:28:30.200 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.200 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.201 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:28:30.201 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.948362+0000", 2024-08-19T14:28:30.201 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012290000000000001 2024-08-19T14:28:30.201 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.201 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.201 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 5,7", 2024-08-19T14:28:30.201 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.948506+0000", 2024-08-19T14:28:30.201 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00014444600000000001 2024-08-19T14:28:30.201 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.201 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.201 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:30.201 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.959986+0000", 2024-08-19T14:28:30.201 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.011480321999999999 2024-08-19T14:28:30.201 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.202 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.202 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:28:30.202 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.959998+0000", 2024-08-19T14:28:30.202 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2021000000000001e-05 2024-08-19T14:28:30.202 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.202 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.202 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:30.202 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.986262+0000", 2024-08-19T14:28:30.202 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.026263686000000001 2024-08-19T14:28:30.202 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.202 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.202 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:28:30.202 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.986293+0000", 2024-08-19T14:28:30.202 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.0672e-05 2024-08-19T14:28:30.203 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.203 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.203 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:28:30.203 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.986339+0000", 2024-08-19T14:28:30.203 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.6755000000000002e-05 2024-08-19T14:28:30.203 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:28:30.203 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:28:30.203 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:28:30.203 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.203 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.203 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25813.0:238577 5.b 5:d3853414:::benchmark_data_smithi045_85309_object118324:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e498)", 2024-08-19T14:28:30.203 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:28:09.849914+0000", 2024-08-19T14:28:30.203 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 19.714050562000001, 2024-08-19T14:28:30.203 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.136581801, 2024-08-19T14:28:30.204 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:28:30.204 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:28:30.204 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:28:30.204 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:28:30.204 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25813", 2024-08-19T14:28:30.204 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:28:30.204 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 238577 2024-08-19T14:28:30.204 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.204 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:28:30.204 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.204 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:28:30.204 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.849914+0000", 2024-08-19T14:28:30.204 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:28:30.204 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.205 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.205 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:28:30.205 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.849914+0000", 2024-08-19T14:28:30.205 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:28:30.205 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.205 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.205 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:28:30.205 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.849914+0000", 2024-08-19T14:28:30.205 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.5600000000000002e-07 2024-08-19T14:28:30.205 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.205 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.205 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:28:30.205 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.849916+0000", 2024-08-19T14:28:30.205 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.7940000000000001e-06 2024-08-19T14:28:30.205 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.206 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.206 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:28:30.206 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.849920+0000", 2024-08-19T14:28:30.206 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2320000000000003e-06 2024-08-19T14:28:30.206 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.206 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.206 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:28:30.206 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.849924+0000", 2024-08-19T14:28:30.206 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.5990000000000002e-06 2024-08-19T14:28:30.206 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.206 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.206 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:28:30.206 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.850911+0000", 2024-08-19T14:28:30.206 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00098736700000000006 2024-08-19T14:28:30.207 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.207 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.207 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:28:30.207 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.850918+0000", 2024-08-19T14:28:30.207 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.7560000000000002e-06 2024-08-19T14:28:30.207 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.207 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.207 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:28:30.207 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.953723+0000", 2024-08-19T14:28:30.207 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.10280465499999999 2024-08-19T14:28:30.207 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.207 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.207 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:28:30.207 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.953905+0000", 2024-08-19T14:28:30.208 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000181981 2024-08-19T14:28:30.208 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.208 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.208 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 5,7", 2024-08-19T14:28:30.208 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.954067+0000", 2024-08-19T14:28:30.208 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00016210700000000001 2024-08-19T14:28:30.208 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.208 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.208 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:30.208 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.971940+0000", 2024-08-19T14:28:30.208 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.017872938000000001 2024-08-19T14:28:30.208 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.208 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.208 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:28:30.208 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.971954+0000", 2024-08-19T14:28:30.209 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3869000000000001e-05 2024-08-19T14:28:30.209 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.209 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.209 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:30.209 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.986405+0000", 2024-08-19T14:28:30.209 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.014451606000000001 2024-08-19T14:28:30.209 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.209 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.209 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:28:30.209 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.986457+0000", 2024-08-19T14:28:30.209 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1310000000000002e-05 2024-08-19T14:28:30.209 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.209 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.209 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:28:30.210 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.986496+0000", 2024-08-19T14:28:30.210 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.9331e-05 2024-08-19T14:28:30.210 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:28:30.210 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:28:30.210 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:28:30.210 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.210 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.210 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25813.0:238610 5.b 5:d7a5e7d6:::benchmark_data_smithi045_85309_object118357:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e498)", 2024-08-19T14:28:30.210 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:28:09.849928+0000", 2024-08-19T14:28:30.210 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 19.714036580999998, 2024-08-19T14:28:30.210 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.13690683400000001, 2024-08-19T14:28:30.210 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:28:30.210 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:28:30.210 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:28:30.211 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:28:30.211 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25813", 2024-08-19T14:28:30.211 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:28:30.211 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 238610 2024-08-19T14:28:30.211 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.211 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:28:30.211 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.211 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:28:30.211 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.849928+0000", 2024-08-19T14:28:30.211 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:28:30.211 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.211 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.211 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:28:30.211 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.849928+0000", 2024-08-19T14:28:30.211 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:28:30.212 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.212 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.212 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:28:30.212 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.849928+0000", 2024-08-19T14:28:30.212 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.17e-07 2024-08-19T14:28:30.212 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.212 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.212 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:28:30.212 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.849930+0000", 2024-08-19T14:28:30.212 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.469e-06 2024-08-19T14:28:30.212 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.212 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.212 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:28:30.213 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.849933+0000", 2024-08-19T14:28:30.213 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.1200000000000002e-06 2024-08-19T14:28:30.213 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.213 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.213 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:28:30.213 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.849935+0000", 2024-08-19T14:28:30.213 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.4329999999999998e-06 2024-08-19T14:28:30.213 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.213 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.213 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:28:30.213 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.850958+0000", 2024-08-19T14:28:30.213 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0010230180000000001 2024-08-19T14:28:30.213 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.213 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.214 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:28:30.214 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.850965+0000", 2024-08-19T14:28:30.214 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.0190000000000001e-06 2024-08-19T14:28:30.214 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.214 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.214 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:28:30.214 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.960218+0000", 2024-08-19T14:28:30.214 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.109252668 2024-08-19T14:28:30.214 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.214 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.214 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:28:30.214 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.960328+0000", 2024-08-19T14:28:30.214 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00010982899999999999 2024-08-19T14:28:30.214 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.215 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.215 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 5,7", 2024-08-19T14:28:30.215 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.960851+0000", 2024-08-19T14:28:30.215 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00052273100000000002 2024-08-19T14:28:30.215 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.215 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.215 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:28:30.215 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.972242+0000", 2024-08-19T14:28:30.215 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.011391814 2024-08-19T14:28:30.215 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.215 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.215 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:30.215 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.972297+0000", 2024-08-19T14:28:30.215 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.4901999999999997e-05 2024-08-19T14:28:30.216 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.216 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.216 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:30.216 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.986720+0000", 2024-08-19T14:28:30.216 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.01442267 2024-08-19T14:28:30.216 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.216 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.216 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:28:30.216 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.986797+0000", 2024-08-19T14:28:30.216 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.7232999999999997e-05 2024-08-19T14:28:30.216 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.216 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.216 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:28:30.216 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.986835+0000", 2024-08-19T14:28:30.217 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7710999999999999e-05 2024-08-19T14:28:30.217 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:28:30.217 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:28:30.217 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:28:30.217 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.217 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.217 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25813.0:238620 5.b 5:de13d110:::benchmark_data_smithi045_85309_object118367:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e498)", 2024-08-19T14:28:30.217 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:28:09.849939+0000", 2024-08-19T14:28:30.217 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 19.714026099000002, 2024-08-19T14:28:30.217 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.14434017499999999, 2024-08-19T14:28:30.217 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:28:30.217 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:28:30.217 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:28:30.217 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:28:30.217 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25813", 2024-08-19T14:28:30.218 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:28:30.218 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 238620 2024-08-19T14:28:30.218 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.218 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:28:30.218 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.218 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:28:30.218 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.849939+0000", 2024-08-19T14:28:30.218 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:28:30.218 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.218 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.218 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:28:30.218 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.849939+0000", 2024-08-19T14:28:30.218 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:28:30.219 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.219 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.219 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:28:30.219 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.849939+0000", 2024-08-19T14:28:30.219 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.2600000000000001e-07 2024-08-19T14:28:30.219 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.219 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.219 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:28:30.219 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.849949+0000", 2024-08-19T14:28:30.219 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.7799999999999995e-06 2024-08-19T14:28:30.219 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.219 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.219 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:28:30.219 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.849956+0000", 2024-08-19T14:28:30.219 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.0169999999999999e-06 2024-08-19T14:28:30.220 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.220 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.220 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:28:30.220 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.849959+0000", 2024-08-19T14:28:30.220 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.45e-06 2024-08-19T14:28:30.220 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.220 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.220 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:28:30.220 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.851033+0000", 2024-08-19T14:28:30.220 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0010738360000000001 2024-08-19T14:28:30.220 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.220 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.220 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:28:30.220 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.851039+0000", 2024-08-19T14:28:30.221 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.173e-06 2024-08-19T14:28:30.221 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.221 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.221 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:28:30.221 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.967436+0000", 2024-08-19T14:28:30.221 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.116397002 2024-08-19T14:28:30.221 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.221 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.221 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:28:30.221 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.967564+0000", 2024-08-19T14:28:30.221 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012765000000000001 2024-08-19T14:28:30.221 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.221 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.221 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 5,7", 2024-08-19T14:28:30.222 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.967646+0000", 2024-08-19T14:28:30.222 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.2244999999999996e-05 2024-08-19T14:28:30.222 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.222 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.222 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:30.222 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.972334+0000", 2024-08-19T14:28:30.222 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0046884650000000002 2024-08-19T14:28:30.222 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.222 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.222 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:28:30.222 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.976677+0000", 2024-08-19T14:28:30.222 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.004342996 2024-08-19T14:28:30.222 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.222 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.223 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:30.223 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.994184+0000", 2024-08-19T14:28:30.223 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.017507035000000001 2024-08-19T14:28:30.223 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.223 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.223 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:28:30.223 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.994236+0000", 2024-08-19T14:28:30.223 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1666999999999998e-05 2024-08-19T14:28:30.223 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.223 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.223 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:28:30.223 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.994279+0000", 2024-08-19T14:28:30.223 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2633000000000003e-05 2024-08-19T14:28:30.223 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:28:30.223 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:28:30.224 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:28:30.224 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.224 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.224 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25813.0:238635 5.b 5:dfa08c61:::benchmark_data_smithi045_85309_object118382:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e498)", 2024-08-19T14:28:30.224 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:28:09.849964+0000", 2024-08-19T14:28:30.224 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 19.714001021000001, 2024-08-19T14:28:30.224 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.14442117300000001, 2024-08-19T14:28:30.224 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:28:30.224 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:28:30.224 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:28:30.224 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:28:30.224 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25813", 2024-08-19T14:28:30.224 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:28:30.224 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 238635 2024-08-19T14:28:30.225 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.225 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:28:30.225 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.225 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:28:30.225 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.849964+0000", 2024-08-19T14:28:30.225 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:28:30.225 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.225 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.225 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:28:30.225 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.849964+0000", 2024-08-19T14:28:30.225 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:28:30.225 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.225 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.225 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:28:30.225 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.849964+0000", 2024-08-19T14:28:30.226 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.36e-07 2024-08-19T14:28:30.226 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.226 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.226 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:28:30.226 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.849966+0000", 2024-08-19T14:28:30.226 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.649e-06 2024-08-19T14:28:30.226 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.226 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.226 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:28:30.226 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.849971+0000", 2024-08-19T14:28:30.226 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.3430000000000003e-06 2024-08-19T14:28:30.226 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.226 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.226 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:28:30.227 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.849976+0000", 2024-08-19T14:28:30.227 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.673e-06 2024-08-19T14:28:30.227 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.227 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.227 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:28:30.227 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.851084+0000", 2024-08-19T14:28:30.227 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.001108038 2024-08-19T14:28:30.227 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.227 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.227 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:28:30.227 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.851094+0000", 2024-08-19T14:28:30.227 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0756e-05 2024-08-19T14:28:30.228 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.228 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.228 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:28:30.228 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.972363+0000", 2024-08-19T14:28:30.228 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.121268689 2024-08-19T14:28:30.228 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.228 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.228 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:28:30.228 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.972486+0000", 2024-08-19T14:28:30.228 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000122908 2024-08-19T14:28:30.228 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.228 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.228 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 5,7", 2024-08-19T14:28:30.228 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.972615+0000", 2024-08-19T14:28:30.229 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012890300000000001 2024-08-19T14:28:30.229 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.229 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.229 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:30.229 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.986239+0000", 2024-08-19T14:28:30.229 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.013623764 2024-08-19T14:28:30.229 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.229 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.229 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:28:30.229 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.986509+0000", 2024-08-19T14:28:30.229 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00027078699999999998 2024-08-19T14:28:30.229 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.229 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.229 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:30.230 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.994314+0000", 2024-08-19T14:28:30.230 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0078044810000000003 2024-08-19T14:28:30.230 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.230 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.230 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:28:30.230 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.994354+0000", 2024-08-19T14:28:30.230 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.9761000000000001e-05 2024-08-19T14:28:30.230 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.230 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.230 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:28:30.230 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.994385+0000", 2024-08-19T14:28:30.230 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.1185e-05 2024-08-19T14:28:30.230 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:28:30.230 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:28:30.231 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:28:30.231 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.231 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.231 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25813.0:238651 5.b 5:d8cfb65b:::benchmark_data_smithi045_85309_object118398:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e498)", 2024-08-19T14:28:30.231 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:28:09.849994+0000", 2024-08-19T14:28:30.231 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 19.713970922000001, 2024-08-19T14:28:30.231 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.155591969, 2024-08-19T14:28:30.231 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:28:30.231 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:28:30.231 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:28:30.231 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:28:30.231 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25813", 2024-08-19T14:28:30.231 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:28:30.231 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 238651 2024-08-19T14:28:30.232 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.232 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:28:30.232 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.232 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:28:30.232 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.849994+0000", 2024-08-19T14:28:30.232 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:28:30.232 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.232 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.232 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:28:30.232 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.849994+0000", 2024-08-19T14:28:30.232 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:28:30.232 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.232 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.232 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:28:30.233 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.849994+0000", 2024-08-19T14:28:30.233 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7e-07 2024-08-19T14:28:30.233 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.233 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.233 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:28:30.233 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.849997+0000", 2024-08-19T14:28:30.233 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.6350000000000002e-06 2024-08-19T14:28:30.233 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.233 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.233 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:28:30.233 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.850004+0000", 2024-08-19T14:28:30.233 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.9029999999999998e-06 2024-08-19T14:28:30.233 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.233 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.234 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:28:30.234 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.850009+0000", 2024-08-19T14:28:30.234 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.3669999999999996e-06 2024-08-19T14:28:30.234 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.234 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.234 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:28:30.234 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.851147+0000", 2024-08-19T14:28:30.234 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.001138004 2024-08-19T14:28:30.234 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.234 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.234 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:28:30.234 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.851155+0000", 2024-08-19T14:28:30.234 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.2670000000000006e-06 2024-08-19T14:28:30.234 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.235 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.235 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:28:30.235 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.976882+0000", 2024-08-19T14:28:30.235 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.12572666399999999 2024-08-19T14:28:30.235 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.235 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.235 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:28:30.235 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.976965+0000", 2024-08-19T14:28:30.235 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.2959000000000003e-05 2024-08-19T14:28:30.235 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.235 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.235 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 5,7", 2024-08-19T14:28:30.235 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.977048+0000", 2024-08-19T14:28:30.235 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.2614999999999994e-05 2024-08-19T14:28:30.236 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.236 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.236 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:28:30.236 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.986514+0000", 2024-08-19T14:28:30.236 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0094668259999999994 2024-08-19T14:28:30.236 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.236 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.236 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:30.236 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.986630+0000", 2024-08-19T14:28:30.236 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00011531699999999999 2024-08-19T14:28:30.236 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.236 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.236 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:30.236 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:10.005520+0000", 2024-08-19T14:28:30.236 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.018890529 2024-08-19T14:28:30.237 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.237 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.237 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:28:30.237 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:10.005555+0000", 2024-08-19T14:28:30.237 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.5095999999999997e-05 2024-08-19T14:28:30.237 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.237 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.237 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:28:30.237 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:10.005586+0000", 2024-08-19T14:28:30.237 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.0317000000000001e-05 2024-08-19T14:28:30.237 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:28:30.237 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:28:30.237 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:28:30.237 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.237 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.238 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25813.0:238667 5.b 5:d4711926:::benchmark_data_smithi045_85309_object118414:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e498)", 2024-08-19T14:28:30.238 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:28:09.850027+0000", 2024-08-19T14:28:30.238 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 19.713937250000001, 2024-08-19T14:28:30.238 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.16378791000000001, 2024-08-19T14:28:30.238 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:28:30.238 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:28:30.238 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:28:30.238 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:28:30.238 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25813", 2024-08-19T14:28:30.238 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:28:30.238 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 238667 2024-08-19T14:28:30.238 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.238 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:28:30.238 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.239 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:28:30.239 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.850027+0000", 2024-08-19T14:28:30.239 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:28:30.239 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.239 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.239 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:28:30.239 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.850027+0000", 2024-08-19T14:28:30.239 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:28:30.239 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.239 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.239 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:28:30.239 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.850028+0000", 2024-08-19T14:28:30.239 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.7700000000000001e-07 2024-08-19T14:28:30.239 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.240 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.240 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:28:30.240 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.850030+0000", 2024-08-19T14:28:30.240 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.886e-06 2024-08-19T14:28:30.240 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.240 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.240 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:28:30.240 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.850037+0000", 2024-08-19T14:28:30.240 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.3370000000000002e-06 2024-08-19T14:28:30.240 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.240 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.240 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:28:30.240 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.850040+0000", 2024-08-19T14:28:30.240 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.2569999999999999e-06 2024-08-19T14:28:30.240 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.241 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.241 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:28:30.241 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.851179+0000", 2024-08-19T14:28:30.241 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.001138436 2024-08-19T14:28:30.241 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.241 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.241 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:28:30.241 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.851184+0000", 2024-08-19T14:28:30.241 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1599999999999997e-06 2024-08-19T14:28:30.241 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.241 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.241 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:28:30.241 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.981742+0000", 2024-08-19T14:28:30.241 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.130558701 2024-08-19T14:28:30.242 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.242 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.242 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:28:30.242 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.981886+0000", 2024-08-19T14:28:30.242 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00014362000000000001 2024-08-19T14:28:30.242 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.242 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.242 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 5,7", 2024-08-19T14:28:30.242 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.982015+0000", 2024-08-19T14:28:30.242 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012872200000000001 2024-08-19T14:28:30.242 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.242 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.242 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:28:30.242 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.986845+0000", 2024-08-19T14:28:30.243 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0048297169999999999 2024-08-19T14:28:30.243 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.243 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.243 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:30.243 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:09.994416+0000", 2024-08-19T14:28:30.243 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0075716489999999997 2024-08-19T14:28:30.243 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.243 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.243 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:30.243 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:10.013659+0000", 2024-08-19T14:28:30.243 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.019243023000000001 2024-08-19T14:28:30.243 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.243 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.243 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:28:30.243 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:10.013762+0000", 2024-08-19T14:28:30.244 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000103185 2024-08-19T14:28:30.244 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:30.244 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:30.244 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:28:30.244 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:10.013815+0000", 2024-08-19T14:28:30.244 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.2939999999999998e-05 2024-08-19T14:28:30.244 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:28:30.244 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:28:30.244 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:28:30.244 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:28:30.244 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:28:30.244 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:28:30.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:29 smithi032 ceph-mon[20004]: pgmap v1166: 31 pgs: 31 active+clean; 620 KiB data, 4.5 GiB used, 621 GiB / 626 GiB avail 2024-08-19T14:28:30.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:29 smithi032 ceph-mon[20004]: 1.d scrub starts 2024-08-19T14:28:30.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:29 smithi032 ceph-mon[20004]: 1.d scrub ok 2024-08-19T14:28:30.270 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:30.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:30.271 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:28:30.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:29 smithi045 ceph-mon[20143]: pgmap v1166: 31 pgs: 31 active+clean; 620 KiB data, 4.5 GiB used, 621 GiB / 626 GiB avail 2024-08-19T14:28:30.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:29 smithi045 ceph-mon[20143]: 1.d scrub starts 2024-08-19T14:28:30.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:29 smithi045 ceph-mon[20143]: 1.d scrub ok 2024-08-19T14:28:30.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:29 smithi031 ceph-mon[16336]: pgmap v1166: 31 pgs: 31 active+clean; 620 KiB data, 4.5 GiB used, 621 GiB / 626 GiB avail 2024-08-19T14:28:30.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:29 smithi031 ceph-mon[16336]: 1.d scrub starts 2024-08-19T14:28:30.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:29 smithi031 ceph-mon[16336]: 1.d scrub ok 2024-08-19T14:28:30.372 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:28:30.397 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:30.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:30.398 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:28:30.499 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:28:30.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:30.521 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:30.521 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:28:30.622 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:28:30.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:30.649 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:30.649 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:28:30.750 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:28:30.776 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:30.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:30.777 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:28:30.878 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:28:30.903 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:30.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:30.904 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:28:31.005 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:28:31.030 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:31.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:31.032 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:28:31.068 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:30 smithi031 ceph-mon[16336]: 1.1d scrub starts 2024-08-19T14:28:31.068 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:30 smithi031 ceph-mon[16336]: 1.1d scrub ok 2024-08-19T14:28:31.068 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:30 smithi031 ceph-mon[16336]: 1.17 scrub starts 2024-08-19T14:28:31.069 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:30 smithi031 ceph-mon[16336]: 1.17 scrub ok 2024-08-19T14:28:31.069 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:30 smithi031 ceph-mon[16336]: 1.1a scrub starts 2024-08-19T14:28:31.069 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:30 smithi031 ceph-mon[16336]: 1.1a scrub ok 2024-08-19T14:28:31.132 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:28:31.157 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:31.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:31.158 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:28:31.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:30 smithi032 ceph-mon[20004]: 1.1d scrub starts 2024-08-19T14:28:31.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:30 smithi032 ceph-mon[20004]: 1.1d scrub ok 2024-08-19T14:28:31.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:30 smithi032 ceph-mon[20004]: 1.17 scrub starts 2024-08-19T14:28:31.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:30 smithi032 ceph-mon[20004]: 1.17 scrub ok 2024-08-19T14:28:31.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:30 smithi032 ceph-mon[20004]: 1.1a scrub starts 2024-08-19T14:28:31.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:30 smithi032 ceph-mon[20004]: 1.1a scrub ok 2024-08-19T14:28:31.259 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:28:31.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:31.287 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:31.288 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:28:31.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:30 smithi045 ceph-mon[20143]: 1.1d scrub starts 2024-08-19T14:28:31.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:30 smithi045 ceph-mon[20143]: 1.1d scrub ok 2024-08-19T14:28:31.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:30 smithi045 ceph-mon[20143]: 1.17 scrub starts 2024-08-19T14:28:31.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:30 smithi045 ceph-mon[20143]: 1.17 scrub ok 2024-08-19T14:28:31.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:30 smithi045 ceph-mon[20143]: 1.1a scrub starts 2024-08-19T14:28:31.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:30 smithi045 ceph-mon[20143]: 1.1a scrub ok 2024-08-19T14:28:31.388 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:28:31.414 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:31.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:31.415 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:28:31.516 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:28:31.541 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:31.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:31.542 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:28:31.643 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:28:31.665 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:31.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:31.666 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:28:31.767 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:28:31.788 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:31.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:31.789 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:28:31.890 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:28:31.916 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:31.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:31.917 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:28:32.018 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:28:32.038 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:32.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:32.039 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:28:32.140 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:28:32.141 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'unique_pool_4' created 2024-08-19T14:28:32.166 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:32.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:32.167 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:28:32.268 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:28:32.290 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:32.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:32.291 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:28:32.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:32 smithi031 ceph-mon[16336]: pgmap v1167: 31 pgs: 31 active+clean; 620 KiB data, 4.5 GiB used, 621 GiB / 626 GiB avail 2024-08-19T14:28:32.392 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:28:32.417 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:32.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:32.418 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:28:32.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:32 smithi032 ceph-mon[20004]: pgmap v1167: 31 pgs: 31 active+clean; 620 KiB data, 4.5 GiB used, 621 GiB / 626 GiB avail 2024-08-19T14:28:32.519 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:28:32.544 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:32.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:32.545 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:28:32.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:32 smithi045 ceph-mon[20143]: pgmap v1167: 31 pgs: 31 active+clean; 620 KiB data, 4.5 GiB used, 621 GiB / 626 GiB avail 2024-08-19T14:28:32.603 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-08-19T14:28:32.646 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:28:32.668 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:32.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:32.670 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:28:32.771 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:28:32.792 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:32.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:32.793 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:28:32.894 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:28:32.920 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:32.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:32.921 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:28:33.022 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:28:33.048 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:33.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:33.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:28:33.150 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:28:33.171 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:33.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:33.172 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:28:33.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:33 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2195801674' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-08-19T14:28:33.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:33 smithi031 ceph-mon[16336]: 1.15 scrub starts 2024-08-19T14:28:33.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:33 smithi031 ceph-mon[16336]: 1.15 scrub ok 2024-08-19T14:28:33.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:33 smithi031 ceph-mon[16336]: pgmap v1168: 31 pgs: 31 active+clean; 590 KiB data, 4.5 GiB used, 621 GiB / 626 GiB avail 2024-08-19T14:28:33.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:33 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2195801674' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-08-19T14:28:33.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:33 smithi031 ceph-mon[16336]: osdmap e510: 12 total, 12 up, 7 in 2024-08-19T14:28:33.213 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd pool application enable unique_pool_4 rados --yes-i-really-mean-it || true 2024-08-19T14:28:33.273 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:28:33.287 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:28:33.299 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:33.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:33.300 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:28:33.400 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:28:33.422 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:33.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:33.423 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:28:33.461 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-19T14:28:33.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:33 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2195801674' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-08-19T14:28:33.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:33 smithi032 ceph-mon[20004]: 1.15 scrub starts 2024-08-19T14:28:33.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:33 smithi032 ceph-mon[20004]: 1.15 scrub ok 2024-08-19T14:28:33.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:33 smithi032 ceph-mon[20004]: pgmap v1168: 31 pgs: 31 active+clean; 590 KiB data, 4.5 GiB used, 621 GiB / 626 GiB avail 2024-08-19T14:28:33.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:33 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2195801674' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-08-19T14:28:33.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:33 smithi032 ceph-mon[20004]: osdmap e510: 12 total, 12 up, 7 in 2024-08-19T14:28:33.524 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:28:33.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:33 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2195801674' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-08-19T14:28:33.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:33 smithi045 ceph-mon[20143]: 1.15 scrub starts 2024-08-19T14:28:33.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:33 smithi045 ceph-mon[20143]: 1.15 scrub ok 2024-08-19T14:28:33.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:33 smithi045 ceph-mon[20143]: pgmap v1168: 31 pgs: 31 active+clean; 590 KiB data, 4.5 GiB used, 621 GiB / 626 GiB avail 2024-08-19T14:28:33.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:33 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2195801674' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-08-19T14:28:33.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:33 smithi045 ceph-mon[20143]: osdmap e510: 12 total, 12 up, 7 in 2024-08-19T14:28:33.549 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:33.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:33.550 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:28:33.651 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:28:33.672 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:33.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:33.673 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:28:33.774 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:28:33.800 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:33.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:33.801 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:28:33.818 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:28:33.818 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-08-19T14:28:33.818 INFO:teuthology.orchestra.run.smithi031.stdout: "num_ops": 0 2024-08-19T14:28:33.818 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:28:33.827 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:28:33.902 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:28:33.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:33.924 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:33.924 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:28:34.025 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:28:34.050 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:34.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:34.051 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:28:34.118 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:28:34.152 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:28:34.177 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:34.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:34.178 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:28:34.279 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:28:34.304 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:34.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:34.305 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:28:34.406 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:28:34.420 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:34 smithi031 ceph-mon[16336]: osdmap e511: 12 total, 12 up, 7 in 2024-08-19T14:28:34.420 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:34 smithi031 ceph-mon[16336]: 1.3 scrub starts 2024-08-19T14:28:34.420 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:34 smithi031 ceph-mon[16336]: 1.3 scrub ok 2024-08-19T14:28:34.424 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:28:34.432 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:34.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:34.433 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:28:34.453 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-08-19T14:28:34.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:34 smithi032 ceph-mon[20004]: osdmap e511: 12 total, 12 up, 7 in 2024-08-19T14:28:34.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:34 smithi032 ceph-mon[20004]: 1.3 scrub starts 2024-08-19T14:28:34.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:34 smithi032 ceph-mon[20004]: 1.3 scrub ok 2024-08-19T14:28:34.534 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:28:34.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:34 smithi045 ceph-mon[20143]: osdmap e511: 12 total, 12 up, 7 in 2024-08-19T14:28:34.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:34 smithi045 ceph-mon[20143]: 1.3 scrub starts 2024-08-19T14:28:34.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:34 smithi045 ceph-mon[20143]: 1.3 scrub ok 2024-08-19T14:28:34.559 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:34.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:34.560 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:28:34.661 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:28:34.687 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:34.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:34.688 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:28:34.789 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:28:34.815 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:34.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:34.816 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:28:34.916 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:28:34.937 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:34.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:34.938 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:28:35.039 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:28:35.065 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:35.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:35.066 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:28:35.167 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:28:35.193 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:35.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:35.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:28:35.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:35 smithi031 ceph-mon[16336]: pgmap v1171: 47 pgs: 6 creating+peering, 1 unknown, 40 active+clean; 577 KiB data, 4.5 GiB used, 621 GiB / 626 GiB avail 2024-08-19T14:28:35.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:35 smithi031 ceph-mon[16336]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-19T14:28:35.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:35 smithi031 ceph-mon[16336]: osdmap e512: 12 total, 12 up, 7 in 2024-08-19T14:28:35.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:35 smithi031 ceph-mon[16336]: 1.16 deep-scrub starts 2024-08-19T14:28:35.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:35 smithi031 ceph-mon[16336]: 1.16 deep-scrub ok 2024-08-19T14:28:35.278 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:28:35.295 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:28:35.316 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:35.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:35.317 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:28:35.418 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:28:35.444 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:35.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:35.445 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:28:35.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:35 smithi032 ceph-mon[20004]: pgmap v1171: 47 pgs: 6 creating+peering, 1 unknown, 40 active+clean; 577 KiB data, 4.5 GiB used, 621 GiB / 626 GiB avail 2024-08-19T14:28:35.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:35 smithi032 ceph-mon[20004]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-19T14:28:35.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:35 smithi032 ceph-mon[20004]: osdmap e512: 12 total, 12 up, 7 in 2024-08-19T14:28:35.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:35 smithi032 ceph-mon[20004]: 1.16 deep-scrub starts 2024-08-19T14:28:35.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:35 smithi032 ceph-mon[20004]: 1.16 deep-scrub ok 2024-08-19T14:28:35.546 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:28:35.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:35 smithi045 ceph-mon[20143]: pgmap v1171: 47 pgs: 6 creating+peering, 1 unknown, 40 active+clean; 577 KiB data, 4.5 GiB used, 621 GiB / 626 GiB avail 2024-08-19T14:28:35.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:35 smithi045 ceph-mon[20143]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-19T14:28:35.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:35 smithi045 ceph-mon[20143]: osdmap e512: 12 total, 12 up, 7 in 2024-08-19T14:28:35.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:35 smithi045 ceph-mon[20143]: 1.16 deep-scrub starts 2024-08-19T14:28:35.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:35 smithi045 ceph-mon[20143]: 1.16 deep-scrub ok 2024-08-19T14:28:35.567 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:35.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:35.568 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:28:35.669 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:28:35.690 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:35.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:35.691 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:28:35.763 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:28:35.763 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-08-19T14:28:35.763 INFO:teuthology.orchestra.run.smithi031.stdout: "complaint_time": 30, 2024-08-19T14:28:35.763 INFO:teuthology.orchestra.run.smithi031.stdout: "num_blocked_ops": 0 2024-08-19T14:28:35.763 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:28:35.792 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:28:35.817 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:35.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:35.818 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:28:35.919 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:28:35.942 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:35.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:35.943 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:28:36.044 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:28:36.065 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:36.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:36.066 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:28:36.154 INFO:teuthology.orchestra.run.smithi031.stderr:enabled application 'rados' on pool 'unique_pool_4' 2024-08-19T14:28:36.154 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is set 2024-08-19T14:28:36.166 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:28:36.190 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:36.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:36.191 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:28:36.292 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:28:36.317 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:36.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:36.318 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:28:36.419 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:28:36.425 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:36 smithi031 ceph-mon[16336]: 1.12 deep-scrub starts 2024-08-19T14:28:36.425 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:36 smithi031 ceph-mon[16336]: 1.12 deep-scrub ok 2024-08-19T14:28:36.425 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:36 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1603703635' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:28:36.425 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:36 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2111568911' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-08-19T14:28:36.444 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:36.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:36.445 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:28:36.466 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:36 smithi045 ceph-mon[20143]: 1.12 deep-scrub starts 2024-08-19T14:28:36.466 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:36 smithi045 ceph-mon[20143]: 1.12 deep-scrub ok 2024-08-19T14:28:36.466 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:36 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1603703635' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:28:36.466 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:36 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2111568911' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-08-19T14:28:36.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:36 smithi032 ceph-mon[20004]: 1.12 deep-scrub starts 2024-08-19T14:28:36.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:36 smithi032 ceph-mon[20004]: 1.12 deep-scrub ok 2024-08-19T14:28:36.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:36 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1603703635' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:28:36.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:36 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2111568911' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-08-19T14:28:36.510 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-08-19T14:28:36.545 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:28:36.568 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:36.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:36.569 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:28:36.670 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:28:36.696 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:36.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:36.697 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:28:36.798 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:28:36.820 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:36.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:36.822 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:28:36.923 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:28:36.948 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:36.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:36.949 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:28:37.050 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:28:37.075 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:37.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:37.076 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:28:37.177 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:28:37.199 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:37.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:37.200 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:28:37.301 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:28:37.327 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:37.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:37.328 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:28:37.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:37 smithi031 ceph-mon[16336]: pgmap v1173: 47 pgs: 6 creating+peering, 41 active+clean; 577 KiB data, 4.5 GiB used, 621 GiB / 626 GiB avail 2024-08-19T14:28:37.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:37 smithi031 ceph-mon[16336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:28:37.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:37 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1603703635' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:28:37.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:37 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2111568911' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-08-19T14:28:37.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:37 smithi031 ceph-mon[16336]: osdmap e513: 12 total, 12 up, 7 in 2024-08-19T14:28:37.429 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:28:37.451 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:37.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:37.452 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:28:37.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:37 smithi032 ceph-mon[20004]: pgmap v1173: 47 pgs: 6 creating+peering, 41 active+clean; 577 KiB data, 4.5 GiB used, 621 GiB / 626 GiB avail 2024-08-19T14:28:37.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:37 smithi032 ceph-mon[20004]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:28:37.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:37 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1603703635' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:28:37.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:37 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2111568911' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-08-19T14:28:37.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:37 smithi032 ceph-mon[20004]: osdmap e513: 12 total, 12 up, 7 in 2024-08-19T14:28:37.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:37 smithi045 ceph-mon[20143]: pgmap v1173: 47 pgs: 6 creating+peering, 41 active+clean; 577 KiB data, 4.5 GiB used, 621 GiB / 626 GiB avail 2024-08-19T14:28:37.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:37 smithi045 ceph-mon[20143]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:28:37.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:37 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1603703635' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:28:37.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:37 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2111568911' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-08-19T14:28:37.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:37 smithi045 ceph-mon[20143]: osdmap e513: 12 total, 12 up, 7 in 2024-08-19T14:28:37.553 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:28:37.578 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:37.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:37.579 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:28:37.680 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:28:37.705 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:37.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:37.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:28:37.780 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:28:37.780 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:28:37.807 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:28:37.829 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:37.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:37.830 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:28:37.931 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:28:37.953 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:37.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:37.954 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:28:38.055 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:28:38.080 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:38.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:38.081 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:28:38.182 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:28:38.208 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:38.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:38.210 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:28:38.306 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:28:38.307 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 20, 2024-08-19T14:28:38.307 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 600, 2024-08-19T14:28:38.307 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [ 2024-08-19T14:28:38.307 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.307 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25057.0:188768 3.24 3:25c22bc2:::benchmark_data_smithi045_59973_object57589:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-08-19T14:28:38.307 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:19:02.299148+0000", 2024-08-19T14:28:38.307 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 576.00537693499996, 2024-08-19T14:28:38.307 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.92656360999999998, 2024-08-19T14:28:38.307 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:28:38.307 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:28:38.307 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:28:38.308 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:28:38.308 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25057", 2024-08-19T14:28:38.308 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:28:38.308 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 188768 2024-08-19T14:28:38.308 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.308 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:28:38.308 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.308 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:28:38.308 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.299148+0000", 2024-08-19T14:28:38.308 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:28:38.308 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.308 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.308 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:28:38.308 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.299148+0000", 2024-08-19T14:28:38.308 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:28:38.309 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.309 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.309 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:28:38.309 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.299150+0000", 2024-08-19T14:28:38.309 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.869e-06 2024-08-19T14:28:38.309 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.309 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.309 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:28:38.309 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.299157+0000", 2024-08-19T14:28:38.309 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.2699999999999999e-06 2024-08-19T14:28:38.309 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.309 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.309 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:28:38.310 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.299173+0000", 2024-08-19T14:28:38.310 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6099e-05 2024-08-19T14:28:38.310 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.310 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.310 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:28:38.310 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.299184+0000", 2024-08-19T14:28:38.311 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0523000000000001e-05 2024-08-19T14:28:38.311 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.311 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.311 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:28:38.311 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.299253+0000", 2024-08-19T14:28:38.311 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.8752999999999997e-05 2024-08-19T14:28:38.311 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.311 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.311 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:28:38.311 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:02.299265+0000", 2024-08-19T14:28:38.311 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2295e-05 2024-08-19T14:28:38.311 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.311 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.312 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:28:38.312 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:03.218622+0000", 2024-08-19T14:28:38.312 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.91935708900000002 2024-08-19T14:28:38.312 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.312 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.312 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:28:38.312 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:03.218724+0000", 2024-08-19T14:28:38.312 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00010233700000000001 2024-08-19T14:28:38.312 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.312 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.312 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 5,11", 2024-08-19T14:28:38.312 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:03.218822+0000", 2024-08-19T14:28:38.312 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.7435999999999993e-05 2024-08-19T14:28:38.313 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.313 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.313 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:28:38.313 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:03.220396+0000", 2024-08-19T14:28:38.313 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0015744450000000001 2024-08-19T14:28:38.313 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.313 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.313 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:38.313 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:03.223542+0000", 2024-08-19T14:28:38.313 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0031455879999999999 2024-08-19T14:28:38.313 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.313 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.313 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:38.313 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:03.225592+0000", 2024-08-19T14:28:38.314 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0020502789999999999 2024-08-19T14:28:38.314 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.314 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.314 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:28:38.314 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:03.225650+0000", 2024-08-19T14:28:38.314 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.7472000000000001e-05 2024-08-19T14:28:38.314 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.314 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.314 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:28:38.314 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:03.225712+0000", 2024-08-19T14:28:38.314 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.2155000000000004e-05 2024-08-19T14:28:38.314 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:28:38.314 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:28:38.315 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:28:38.315 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.315 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.315 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25333.0:22687 4.7 4:efd17a0f:::benchmark_data_smithi045_71562_object22686:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-08-19T14:28:38.315 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:20:41.348541+0000", 2024-08-19T14:28:38.315 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 476.95598464699998, 2024-08-19T14:28:38.315 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.33315307799999999, 2024-08-19T14:28:38.315 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:28:38.315 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:28:38.315 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:28:38.315 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:28:38.315 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25333", 2024-08-19T14:28:38.315 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:28:38.316 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 22687 2024-08-19T14:28:38.316 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.316 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:28:38.316 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.316 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:28:38.316 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.348541+0000", 2024-08-19T14:28:38.316 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:28:38.316 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.316 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.316 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:28:38.316 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.348541+0000", 2024-08-19T14:28:38.316 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:28:38.316 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.316 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.316 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:28:38.317 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.348541+0000", 2024-08-19T14:28:38.317 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.3e-07 2024-08-19T14:28:38.317 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.317 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.317 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:28:38.317 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.348571+0000", 2024-08-19T14:28:38.317 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.0145e-05 2024-08-19T14:28:38.317 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.317 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.317 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:28:38.317 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.348580+0000", 2024-08-19T14:28:38.317 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.3240000000000003e-06 2024-08-19T14:28:38.317 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.317 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.317 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:28:38.318 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.348586+0000", 2024-08-19T14:28:38.318 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.4500000000000001e-06 2024-08-19T14:28:38.318 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.318 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.318 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:28:38.318 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.349519+0000", 2024-08-19T14:28:38.318 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00093283700000000005 2024-08-19T14:28:38.318 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.318 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.318 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:28:38.318 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.349639+0000", 2024-08-19T14:28:38.318 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000119703 2024-08-19T14:28:38.318 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.318 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.319 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,7", 2024-08-19T14:28:38.319 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.350370+0000", 2024-08-19T14:28:38.319 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00073159899999999999 2024-08-19T14:28:38.319 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.319 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.319 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:38.319 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.353458+0000", 2024-08-19T14:28:38.319 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0030877280000000001 2024-08-19T14:28:38.319 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.319 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.319 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:28:38.319 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.354430+0000", 2024-08-19T14:28:38.319 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00097207399999999996 2024-08-19T14:28:38.319 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.320 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.320 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:38.320 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.681572+0000", 2024-08-19T14:28:38.320 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.32714184499999999 2024-08-19T14:28:38.320 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.320 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.320 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:28:38.320 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.681644+0000", 2024-08-19T14:28:38.320 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.2311000000000006e-05 2024-08-19T14:28:38.320 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.320 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.320 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:28:38.320 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.681694+0000", 2024-08-19T14:28:38.320 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.9332000000000001e-05 2024-08-19T14:28:38.320 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:28:38.321 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:28:38.321 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:28:38.321 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.321 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.321 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25333.0:22691 4.7 4:e60184b4:::benchmark_data_smithi045_71562_object22690:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-08-19T14:28:38.321 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:20:41.350039+0000", 2024-08-19T14:28:38.321 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 476.95448572399999, 2024-08-19T14:28:38.321 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.33211775999999998, 2024-08-19T14:28:38.321 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:28:38.321 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:28:38.321 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:28:38.321 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:28:38.322 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25333", 2024-08-19T14:28:38.322 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:28:38.322 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 22691 2024-08-19T14:28:38.322 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.322 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:28:38.322 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.322 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:28:38.322 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.350039+0000", 2024-08-19T14:28:38.322 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:28:38.322 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.322 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.322 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:28:38.323 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.350039+0000", 2024-08-19T14:28:38.323 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:28:38.323 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.323 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.323 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:28:38.323 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.350040+0000", 2024-08-19T14:28:38.323 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.7599999999999996e-07 2024-08-19T14:28:38.323 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.323 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.323 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:28:38.323 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.350097+0000", 2024-08-19T14:28:38.323 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.7108999999999998e-05 2024-08-19T14:28:38.323 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.323 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.323 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:28:38.323 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.350109+0000", 2024-08-19T14:28:38.324 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1926999999999999e-05 2024-08-19T14:28:38.324 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.324 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.324 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:28:38.324 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.350117+0000", 2024-08-19T14:28:38.324 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.6429999999999995e-06 2024-08-19T14:28:38.324 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.324 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.324 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:28:38.324 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.351286+0000", 2024-08-19T14:28:38.324 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0011685179999999999 2024-08-19T14:28:38.324 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.324 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.324 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:28:38.324 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.351545+0000", 2024-08-19T14:28:38.325 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00025918799999999999 2024-08-19T14:28:38.325 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.325 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.325 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,7", 2024-08-19T14:28:38.325 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.351671+0000", 2024-08-19T14:28:38.325 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012657099999999999 2024-08-19T14:28:38.325 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.325 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.325 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:38.325 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.354827+0000", 2024-08-19T14:28:38.325 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.003155808 2024-08-19T14:28:38.325 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.325 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.325 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:28:38.326 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.354902+0000", 2024-08-19T14:28:38.326 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.4691999999999996e-05 2024-08-19T14:28:38.326 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.326 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.326 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:38.326 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.682051+0000", 2024-08-19T14:28:38.326 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.32714955600000001 2024-08-19T14:28:38.326 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.326 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.326 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:28:38.326 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.682110+0000", 2024-08-19T14:28:38.326 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.8746999999999998e-05 2024-08-19T14:28:38.326 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.326 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.327 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:28:38.327 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.682157+0000", 2024-08-19T14:28:38.327 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7125e-05 2024-08-19T14:28:38.327 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:28:38.327 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:28:38.327 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:28:38.327 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.327 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.327 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25333.0:22704 4.7 4:e3ca772f:::benchmark_data_smithi045_71562_object22703:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-08-19T14:28:38.327 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:20:41.354754+0000", 2024-08-19T14:28:38.327 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 476.94977099900001, 2024-08-19T14:28:38.327 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.32755525600000002, 2024-08-19T14:28:38.327 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:28:38.327 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:28:38.328 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:28:38.328 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:28:38.328 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25333", 2024-08-19T14:28:38.328 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:28:38.328 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 22704 2024-08-19T14:28:38.328 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.328 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:28:38.328 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.328 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:28:38.328 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.354754+0000", 2024-08-19T14:28:38.328 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:28:38.328 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.328 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.328 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:28:38.328 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.354754+0000", 2024-08-19T14:28:38.329 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:28:38.329 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.329 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.329 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:28:38.329 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.354755+0000", 2024-08-19T14:28:38.329 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0589999999999999e-06 2024-08-19T14:28:38.329 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.329 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.329 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:28:38.329 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.354883+0000", 2024-08-19T14:28:38.329 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000127515 2024-08-19T14:28:38.330 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.330 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.330 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:28:38.330 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.354893+0000", 2024-08-19T14:28:38.330 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0291e-05 2024-08-19T14:28:38.330 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.330 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.330 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:28:38.330 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.354901+0000", 2024-08-19T14:28:38.330 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.6760000000000007e-06 2024-08-19T14:28:38.330 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.330 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.330 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:28:38.331 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.354944+0000", 2024-08-19T14:28:38.331 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.3155999999999998e-05 2024-08-19T14:28:38.331 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.331 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.331 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:28:38.331 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.355068+0000", 2024-08-19T14:28:38.331 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000124521 2024-08-19T14:28:38.331 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.331 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.331 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,7", 2024-08-19T14:28:38.331 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.355175+0000", 2024-08-19T14:28:38.331 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000106984 2024-08-19T14:28:38.331 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.332 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.332 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:38.332 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.357801+0000", 2024-08-19T14:28:38.332 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0026251920000000002 2024-08-19T14:28:38.332 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.332 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.332 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:28:38.332 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.357813+0000", 2024-08-19T14:28:38.332 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2734e-05 2024-08-19T14:28:38.332 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.332 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.332 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:38.332 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.682208+0000", 2024-08-19T14:28:38.332 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.324394395 2024-08-19T14:28:38.333 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.333 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.333 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:28:38.333 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.682268+0000", 2024-08-19T14:28:38.333 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.0228e-05 2024-08-19T14:28:38.333 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.333 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.333 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:28:38.333 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.682309+0000", 2024-08-19T14:28:38.333 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.1505000000000003e-05 2024-08-19T14:28:38.333 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:28:38.333 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:28:38.333 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:28:38.333 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.334 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.334 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25333.0:22709 4.7 4:e2a7b456:::benchmark_data_smithi045_71562_object22708:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-08-19T14:28:38.334 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:20:41.355501+0000", 2024-08-19T14:28:38.334 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 476.94902435900002, 2024-08-19T14:28:38.334 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.32904467700000001, 2024-08-19T14:28:38.334 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:28:38.334 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:28:38.334 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:28:38.334 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:28:38.334 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25333", 2024-08-19T14:28:38.334 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:28:38.334 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 22709 2024-08-19T14:28:38.334 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.334 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:28:38.335 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.335 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:28:38.336 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:28:38.336 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.355501+0000", 2024-08-19T14:28:38.336 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:28:38.336 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.337 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.337 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:28:38.337 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.355501+0000", 2024-08-19T14:28:38.337 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:28:38.337 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.337 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.337 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:28:38.337 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.355501+0000", 2024-08-19T14:28:38.337 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1099999999999996e-07 2024-08-19T14:28:38.337 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.337 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.337 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:28:38.338 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.355554+0000", 2024-08-19T14:28:38.338 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.2317000000000001e-05 2024-08-19T14:28:38.338 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.338 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.338 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:28:38.338 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.355561+0000", 2024-08-19T14:28:38.338 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.6190000000000002e-06 2024-08-19T14:28:38.338 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.338 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.338 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:28:38.338 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.355567+0000", 2024-08-19T14:28:38.338 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.3090000000000003e-06 2024-08-19T14:28:38.338 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.338 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.339 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:28:38.339 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.356458+0000", 2024-08-19T14:28:38.339 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00089112600000000001 2024-08-19T14:28:38.339 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.339 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.339 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:28:38.339 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.356702+0000", 2024-08-19T14:28:38.339 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00024424000000000001 2024-08-19T14:28:38.339 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.339 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.339 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,7", 2024-08-19T14:28:38.339 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.356835+0000", 2024-08-19T14:28:38.339 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000132571 2024-08-19T14:28:38.340 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.340 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.340 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:28:38.340 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.358857+0000", 2024-08-19T14:28:38.340 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0020221280000000002 2024-08-19T14:28:38.340 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.340 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.340 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:38.340 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.358903+0000", 2024-08-19T14:28:38.340 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.6088999999999999e-05 2024-08-19T14:28:38.340 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.340 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.340 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:38.340 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.684455+0000", 2024-08-19T14:28:38.341 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.32555214500000001 2024-08-19T14:28:38.341 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.341 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.341 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:28:38.341 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.684505+0000", 2024-08-19T14:28:38.341 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.9835e-05 2024-08-19T14:28:38.341 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.341 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.341 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:28:38.341 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.684545+0000", 2024-08-19T14:28:38.341 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.0787000000000001e-05 2024-08-19T14:28:38.341 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:28:38.341 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:28:38.342 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:28:38.342 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.342 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.342 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25333.0:22715 4.7 4:ee462d5b:::benchmark_data_smithi045_71562_object22714:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-08-19T14:28:38.342 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:20:41.357321+0000", 2024-08-19T14:28:38.342 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 476.94720373799998, 2024-08-19T14:28:38.342 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.32736539599999998, 2024-08-19T14:28:38.342 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:28:38.342 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:28:38.342 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:28:38.342 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:28:38.342 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25333", 2024-08-19T14:28:38.343 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:28:38.343 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 22715 2024-08-19T14:28:38.343 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.343 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:28:38.343 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.343 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:28:38.343 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.357321+0000", 2024-08-19T14:28:38.343 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:28:38.343 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.344 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.344 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:28:38.344 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.357321+0000", 2024-08-19T14:28:38.344 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:28:38.344 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.344 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.344 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:28:38.344 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.357322+0000", 2024-08-19T14:28:38.344 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.3099999999999996e-07 2024-08-19T14:28:38.344 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.344 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.344 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:28:38.344 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.357399+0000", 2024-08-19T14:28:38.345 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.6248000000000006e-05 2024-08-19T14:28:38.345 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.345 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.345 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:28:38.345 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.357412+0000", 2024-08-19T14:28:38.345 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3199e-05 2024-08-19T14:28:38.345 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.345 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.345 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:28:38.345 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.357419+0000", 2024-08-19T14:28:38.345 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.7429999999999996e-06 2024-08-19T14:28:38.345 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.345 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.345 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:28:38.346 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.357883+0000", 2024-08-19T14:28:38.346 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00046407799999999998 2024-08-19T14:28:38.346 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.346 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.346 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:28:38.346 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.358001+0000", 2024-08-19T14:28:38.346 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000118293 2024-08-19T14:28:38.346 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.346 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.346 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,7", 2024-08-19T14:28:38.346 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.358145+0000", 2024-08-19T14:28:38.346 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00014417400000000001 2024-08-19T14:28:38.346 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.347 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.347 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:28:38.347 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.359078+0000", 2024-08-19T14:28:38.347 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00093282800000000002 2024-08-19T14:28:38.347 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.347 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.347 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:38.347 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.359246+0000", 2024-08-19T14:28:38.347 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00016784699999999999 2024-08-19T14:28:38.347 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.347 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.347 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:38.347 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.684596+0000", 2024-08-19T14:28:38.347 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.32535034800000001 2024-08-19T14:28:38.348 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.348 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.348 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:28:38.348 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.684650+0000", 2024-08-19T14:28:38.348 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.3631999999999997e-05 2024-08-19T14:28:38.348 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.348 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.348 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:28:38.348 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.684687+0000", 2024-08-19T14:28:38.348 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7075e-05 2024-08-19T14:28:38.348 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:28:38.348 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:28:38.348 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:28:38.348 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.348 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.349 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25333.0:22726 4.7 4:eca380cd:::benchmark_data_smithi045_71562_object22725:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-08-19T14:28:38.349 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:20:41.361609+0000", 2024-08-19T14:28:38.349 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 476.942916169, 2024-08-19T14:28:38.349 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.32322025399999998, 2024-08-19T14:28:38.349 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:28:38.349 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:28:38.349 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:28:38.349 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:28:38.349 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25333", 2024-08-19T14:28:38.349 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:28:38.349 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 22726 2024-08-19T14:28:38.349 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.349 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:28:38.349 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.350 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:28:38.350 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.361609+0000", 2024-08-19T14:28:38.350 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:28:38.350 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.350 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.350 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:28:38.350 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.361609+0000", 2024-08-19T14:28:38.350 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:28:38.350 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.350 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.350 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:28:38.350 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.361610+0000", 2024-08-19T14:28:38.350 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1060000000000001e-06 2024-08-19T14:28:38.350 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.351 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.351 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:28:38.351 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.361662+0000", 2024-08-19T14:28:38.351 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1965000000000001e-05 2024-08-19T14:28:38.351 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.351 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.351 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:28:38.351 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.361673+0000", 2024-08-19T14:28:38.351 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0577999999999999e-05 2024-08-19T14:28:38.351 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.351 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.351 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:28:38.351 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.361679+0000", 2024-08-19T14:28:38.351 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.2809999999999997e-06 2024-08-19T14:28:38.352 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.352 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.352 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:28:38.352 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.361734+0000", 2024-08-19T14:28:38.352 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.5056999999999998e-05 2024-08-19T14:28:38.352 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.352 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.352 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:28:38.352 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.361872+0000", 2024-08-19T14:28:38.352 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013795999999999999 2024-08-19T14:28:38.352 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.352 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.352 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,7", 2024-08-19T14:28:38.352 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.361987+0000", 2024-08-19T14:28:38.353 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000114725 2024-08-19T14:28:38.353 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.353 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.353 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:38.353 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.363128+0000", 2024-08-19T14:28:38.353 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.001141818 2024-08-19T14:28:38.353 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.353 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.353 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:28:38.353 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.363443+0000", 2024-08-19T14:28:38.353 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00031427400000000002 2024-08-19T14:28:38.353 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.353 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.353 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:38.354 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.684736+0000", 2024-08-19T14:28:38.354 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.32129368000000003 2024-08-19T14:28:38.354 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.354 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.354 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:28:38.354 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.684791+0000", 2024-08-19T14:28:38.354 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.4759000000000002e-05 2024-08-19T14:28:38.354 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.354 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.355 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:28:38.355 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.684829+0000", 2024-08-19T14:28:38.355 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.8050999999999999e-05 2024-08-19T14:28:38.355 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:28:38.355 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:28:38.355 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:28:38.355 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.355 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.355 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25333.0:22730 4.7 4:e3e08cfc:::benchmark_data_smithi045_71562_object22729:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-08-19T14:28:38.355 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:20:41.363527+0000", 2024-08-19T14:28:38.355 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 476.94099795400001, 2024-08-19T14:28:38.355 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.32152788399999999, 2024-08-19T14:28:38.355 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:28:38.356 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:28:38.356 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:28:38.356 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:28:38.356 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25333", 2024-08-19T14:28:38.356 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:28:38.356 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 22730 2024-08-19T14:28:38.356 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.356 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:28:38.356 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.356 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:28:38.356 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.363527+0000", 2024-08-19T14:28:38.356 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:28:38.356 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.357 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.357 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:28:38.357 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.363527+0000", 2024-08-19T14:28:38.357 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:28:38.357 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.357 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.357 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:28:38.357 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.363528+0000", 2024-08-19T14:28:38.357 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.2699999999999999e-07 2024-08-19T14:28:38.357 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.357 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.357 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:28:38.357 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.363625+0000", 2024-08-19T14:28:38.357 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.6584999999999998e-05 2024-08-19T14:28:38.358 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.358 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.358 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:28:38.358 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.363636+0000", 2024-08-19T14:28:38.358 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1361e-05 2024-08-19T14:28:38.358 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.358 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.358 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:28:38.358 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.363641+0000", 2024-08-19T14:28:38.358 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.8389999999999997e-06 2024-08-19T14:28:38.358 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.358 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.358 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:28:38.358 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.363681+0000", 2024-08-19T14:28:38.359 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.0339999999999997e-05 2024-08-19T14:28:38.359 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.359 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.359 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:28:38.359 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.363804+0000", 2024-08-19T14:28:38.359 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012319800000000001 2024-08-19T14:28:38.359 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.359 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.359 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,7", 2024-08-19T14:28:38.359 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.363954+0000", 2024-08-19T14:28:38.359 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000149309 2024-08-19T14:28:38.359 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.359 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.359 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:28:38.360 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.364732+0000", 2024-08-19T14:28:38.360 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00077795000000000002 2024-08-19T14:28:38.360 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.360 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.360 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:38.360 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.364924+0000", 2024-08-19T14:28:38.360 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00019224699999999999 2024-08-19T14:28:38.360 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.360 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.360 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:38.360 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.684963+0000", 2024-08-19T14:28:38.360 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.32003914700000002 2024-08-19T14:28:38.360 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.361 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.361 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:28:38.361 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.685016+0000", 2024-08-19T14:28:38.361 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.2636000000000002e-05 2024-08-19T14:28:38.361 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.361 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.361 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:28:38.361 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.685055+0000", 2024-08-19T14:28:38.361 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.9545000000000002e-05 2024-08-19T14:28:38.361 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:28:38.361 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:28:38.361 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:28:38.361 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.361 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.361 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25813.0:132390 5.17 5:eadc7f5f:::benchmark_data_smithi045_85309_object12137:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e462)", 2024-08-19T14:28:38.362 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:26:28.790357+0000", 2024-08-19T14:28:38.362 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 129.514168391, 2024-08-19T14:28:38.362 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.61116161199999997, 2024-08-19T14:28:38.362 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:28:38.362 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:28:38.362 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:28:38.362 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:28:38.362 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25813", 2024-08-19T14:28:38.362 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:28:38.362 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 132390 2024-08-19T14:28:38.362 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.362 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:28:38.362 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.363 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:28:38.363 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.790357+0000", 2024-08-19T14:28:38.363 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:28:38.363 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.363 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.363 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:28:38.363 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.790357+0000", 2024-08-19T14:28:38.363 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:28:38.363 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.363 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.363 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:28:38.363 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.790358+0000", 2024-08-19T14:28:38.363 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2500000000000001e-06 2024-08-19T14:28:38.363 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.364 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.364 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:28:38.364 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.790363+0000", 2024-08-19T14:28:38.364 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4800000000000003e-06 2024-08-19T14:28:38.364 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.364 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.364 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:28:38.364 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.790373+0000", 2024-08-19T14:28:38.364 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0117e-05 2024-08-19T14:28:38.364 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.364 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.364 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:28:38.364 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.790378+0000", 2024-08-19T14:28:38.364 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.6280000000000002e-06 2024-08-19T14:28:38.365 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.365 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.365 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:28:38.365 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.791871+0000", 2024-08-19T14:28:38.365 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0014923829999999999 2024-08-19T14:28:38.365 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.365 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.365 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:28:38.365 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.791957+0000", 2024-08-19T14:28:38.365 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.6532000000000004e-05 2024-08-19T14:28:38.365 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.365 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.365 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,8,11", 2024-08-19T14:28:38.365 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.792037+0000", 2024-08-19T14:28:38.366 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.9797999999999997e-05 2024-08-19T14:28:38.366 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.366 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.366 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:28:38.366 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.793676+0000", 2024-08-19T14:28:38.366 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.001639176 2024-08-19T14:28:38.366 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.366 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.366 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:38.366 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.796271+0000", 2024-08-19T14:28:38.366 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0025944750000000002 2024-08-19T14:28:38.366 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.366 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.367 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:38.367 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.796315+0000", 2024-08-19T14:28:38.367 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.3995999999999997e-05 2024-08-19T14:28:38.367 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.367 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.367 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:38.367 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.400666+0000", 2024-08-19T14:28:38.367 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.60435100500000005 2024-08-19T14:28:38.367 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.367 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.367 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:28:38.367 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.400710+0000", 2024-08-19T14:28:38.367 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4027999999999999e-05 2024-08-19T14:28:38.368 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.368 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.368 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:28:38.368 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.401518+0000", 2024-08-19T14:28:38.368 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00080874400000000002 2024-08-19T14:28:38.368 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:28:38.368 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:28:38.368 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:28:38.368 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.368 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.368 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25813.0:132430 5.17 5:eeb9c3c7:::benchmark_data_smithi045_85309_object12177:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e462)", 2024-08-19T14:28:38.368 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:26:28.885232+0000", 2024-08-19T14:28:38.368 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 129.419293028, 2024-08-19T14:28:38.368 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.51860473299999998, 2024-08-19T14:28:38.369 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:28:38.369 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:28:38.369 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:28:38.369 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:28:38.369 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25813", 2024-08-19T14:28:38.369 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:28:38.369 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 132430 2024-08-19T14:28:38.369 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.369 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:28:38.369 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.369 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:28:38.369 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.885232+0000", 2024-08-19T14:28:38.369 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:28:38.369 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.370 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.370 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:28:38.370 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.885232+0000", 2024-08-19T14:28:38.370 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:28:38.370 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.370 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.370 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:28:38.370 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.885234+0000", 2024-08-19T14:28:38.370 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.7600000000000001e-06 2024-08-19T14:28:38.370 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.370 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.370 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:28:38.370 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.885240+0000", 2024-08-19T14:28:38.371 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.7579999999999998e-06 2024-08-19T14:28:38.371 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.371 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.371 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:28:38.371 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.885257+0000", 2024-08-19T14:28:38.371 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.7309999999999999e-05 2024-08-19T14:28:38.371 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.371 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.371 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:28:38.371 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.885265+0000", 2024-08-19T14:28:38.371 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.9209999999999992e-06 2024-08-19T14:28:38.371 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.371 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.371 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:28:38.372 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.885328+0000", 2024-08-19T14:28:38.372 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.2842000000000006e-05 2024-08-19T14:28:38.372 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.372 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.372 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:28:38.372 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.885477+0000", 2024-08-19T14:28:38.372 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00014877799999999999 2024-08-19T14:28:38.372 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.372 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.372 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,8,11", 2024-08-19T14:28:38.372 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.885711+0000", 2024-08-19T14:28:38.372 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000234074 2024-08-19T14:28:38.372 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.372 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.373 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:28:38.373 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.890476+0000", 2024-08-19T14:28:38.373 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0047655429999999997 2024-08-19T14:28:38.373 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.373 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.373 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:38.373 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.892622+0000", 2024-08-19T14:28:38.373 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0021459930000000001 2024-08-19T14:28:38.373 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.373 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.373 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:38.373 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.893155+0000", 2024-08-19T14:28:38.373 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00053261900000000002 2024-08-19T14:28:38.373 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.373 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.374 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:38.374 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.403023+0000", 2024-08-19T14:28:38.374 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.50986806399999995 2024-08-19T14:28:38.374 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.374 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.374 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:28:38.374 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.403099+0000", 2024-08-19T14:28:38.374 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.6025000000000005e-05 2024-08-19T14:28:38.374 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.374 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.374 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:28:38.374 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.403837+0000", 2024-08-19T14:28:38.374 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00073804600000000001 2024-08-19T14:28:38.375 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:28:38.375 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:28:38.375 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:28:38.375 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.375 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.375 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25813.0:132434 5.17 5:e9c564fa:::benchmark_data_smithi045_85309_object12181:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e462)", 2024-08-19T14:28:38.375 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:26:28.887497+0000", 2024-08-19T14:28:38.375 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 129.41702861300001, 2024-08-19T14:28:38.375 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.51694072000000002, 2024-08-19T14:28:38.375 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:28:38.375 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:28:38.375 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:28:38.375 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:28:38.376 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25813", 2024-08-19T14:28:38.376 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:28:38.376 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 132434 2024-08-19T14:28:38.376 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.376 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:28:38.376 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.376 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:28:38.376 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.887497+0000", 2024-08-19T14:28:38.376 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:28:38.376 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.376 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.376 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:28:38.376 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.887497+0000", 2024-08-19T14:28:38.376 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:28:38.377 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.377 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.377 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:28:38.377 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.887498+0000", 2024-08-19T14:28:38.377 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3599999999999999e-06 2024-08-19T14:28:38.377 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.377 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.377 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:28:38.377 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.887503+0000", 2024-08-19T14:28:38.377 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.8829999999999997e-06 2024-08-19T14:28:38.377 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.377 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.377 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:28:38.378 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.887519+0000", 2024-08-19T14:28:38.378 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6169000000000001e-05 2024-08-19T14:28:38.378 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.378 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.378 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:28:38.378 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.887526+0000", 2024-08-19T14:28:38.378 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.3379999999999999e-06 2024-08-19T14:28:38.378 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.378 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.378 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:28:38.378 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.887857+0000", 2024-08-19T14:28:38.378 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00033069799999999998 2024-08-19T14:28:38.378 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.379 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:38 smithi031 ceph-mon[16336]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-19T14:28:38.380 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.380 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:28:38.380 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.888014+0000", 2024-08-19T14:28:38.380 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00015727800000000001 2024-08-19T14:28:38.380 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.380 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.380 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,8,11", 2024-08-19T14:28:38.380 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.888227+0000", 2024-08-19T14:28:38.380 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000212874 2024-08-19T14:28:38.380 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.380 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.380 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:28:38.381 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.890486+0000", 2024-08-19T14:28:38.381 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0022583909999999998 2024-08-19T14:28:38.381 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.381 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.381 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:38.381 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.892678+0000", 2024-08-19T14:28:38.381 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0021919600000000002 2024-08-19T14:28:38.381 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.381 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.381 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:38.381 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.893220+0000", 2024-08-19T14:28:38.381 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00054226800000000005 2024-08-19T14:28:38.381 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.381 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.382 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:38.382 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.403898+0000", 2024-08-19T14:28:38.382 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.51067800399999996 2024-08-19T14:28:38.382 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.382 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.382 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:28:38.382 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.403952+0000", 2024-08-19T14:28:38.382 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.4141999999999998e-05 2024-08-19T14:28:38.382 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.382 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.382 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:28:38.382 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.404437+0000", 2024-08-19T14:28:38.382 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00048535499999999998 2024-08-19T14:28:38.383 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:28:38.383 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:28:38.383 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:28:38.383 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.383 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.383 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25813.0:132587 5.17 5:e858db2d:::benchmark_data_smithi045_85309_object12334:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e462)", 2024-08-19T14:28:38.383 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:26:28.996803+0000", 2024-08-19T14:28:38.383 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 129.30772170099999, 2024-08-19T14:28:38.383 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.41584004200000002, 2024-08-19T14:28:38.383 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:28:38.383 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:28:38.383 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:28:38.383 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:28:38.383 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25813", 2024-08-19T14:28:38.384 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:28:38.384 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 132587 2024-08-19T14:28:38.384 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.384 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:28:38.384 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.384 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:28:38.384 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.996803+0000", 2024-08-19T14:28:38.384 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:28:38.384 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.384 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.384 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:28:38.384 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.996803+0000", 2024-08-19T14:28:38.384 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:28:38.384 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.385 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.385 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:28:38.385 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.996805+0000", 2024-08-19T14:28:38.385 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.049e-06 2024-08-19T14:28:38.385 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.385 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.385 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:28:38.385 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.996809+0000", 2024-08-19T14:28:38.385 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.3699999999999997e-06 2024-08-19T14:28:38.385 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.385 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.385 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:28:38.385 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.996821+0000", 2024-08-19T14:28:38.386 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2041e-05 2024-08-19T14:28:38.386 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.386 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.386 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:28:38.386 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.996828+0000", 2024-08-19T14:28:38.386 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.832e-06 2024-08-19T14:28:38.386 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.386 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.386 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:28:38.386 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.996895+0000", 2024-08-19T14:28:38.386 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.7636000000000001e-05 2024-08-19T14:28:38.386 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.386 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.386 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:28:38.387 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.997058+0000", 2024-08-19T14:28:38.387 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00016274799999999999 2024-08-19T14:28:38.387 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.387 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.387 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,8,11", 2024-08-19T14:28:38.387 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.997215+0000", 2024-08-19T14:28:38.387 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00015702899999999999 2024-08-19T14:28:38.387 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.387 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.387 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:38.387 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.000366+0000", 2024-08-19T14:28:38.387 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.003151208 2024-08-19T14:28:38.387 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.387 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.388 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:28:38.388 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.000550+0000", 2024-08-19T14:28:38.388 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00018371699999999999 2024-08-19T14:28:38.388 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.388 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.388 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:38.388 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.001943+0000", 2024-08-19T14:28:38.388 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.001392732 2024-08-19T14:28:38.388 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.388 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.388 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:38.388 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.411104+0000", 2024-08-19T14:28:38.388 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.40916105800000002 2024-08-19T14:28:38.389 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.389 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.389 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:28:38.389 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.411180+0000", 2024-08-19T14:28:38.389 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.5617999999999993e-05 2024-08-19T14:28:38.389 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.389 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.389 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:28:38.389 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.412644+0000", 2024-08-19T14:28:38.389 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0014640040000000001 2024-08-19T14:28:38.389 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:28:38.389 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:28:38.389 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:28:38.389 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.390 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.390 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25813.0:132638 5.17 5:eec760cc:::benchmark_data_smithi045_85309_object12385:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e462)", 2024-08-19T14:28:38.390 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:26:29.033888+0000", 2024-08-19T14:28:38.390 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 129.270637599, 2024-08-19T14:28:38.390 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.38830038300000003, 2024-08-19T14:28:38.390 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:28:38.390 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:28:38.390 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:28:38.390 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:28:38.390 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25813", 2024-08-19T14:28:38.390 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:28:38.390 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 132638 2024-08-19T14:28:38.390 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.391 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:28:38.391 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.391 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:28:38.391 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.033888+0000", 2024-08-19T14:28:38.391 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:28:38.391 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.391 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.391 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:28:38.391 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.033888+0000", 2024-08-19T14:28:38.391 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:28:38.391 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.391 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.391 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:28:38.391 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.033889+0000", 2024-08-19T14:28:38.392 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2950000000000001e-06 2024-08-19T14:28:38.392 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.392 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.392 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:28:38.392 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.033898+0000", 2024-08-19T14:28:38.392 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.7660000000000004e-06 2024-08-19T14:28:38.392 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.392 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.392 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:28:38.392 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.033910+0000", 2024-08-19T14:28:38.392 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1987000000000001e-05 2024-08-19T14:28:38.392 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.392 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.392 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:28:38.393 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.033918+0000", 2024-08-19T14:28:38.393 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.3089999999999996e-06 2024-08-19T14:28:38.393 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.393 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.393 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:28:38.393 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.033993+0000", 2024-08-19T14:28:38.393 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.4912000000000004e-05 2024-08-19T14:28:38.393 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.393 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.393 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:28:38.393 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.034154+0000", 2024-08-19T14:28:38.393 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000161423 2024-08-19T14:28:38.393 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.393 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.394 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,8,11", 2024-08-19T14:28:38.394 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.034316+0000", 2024-08-19T14:28:38.394 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00016205000000000001 2024-08-19T14:28:38.395 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.395 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.395 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:28:38.395 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.036209+0000", 2024-08-19T14:28:38.395 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.001892473 2024-08-19T14:28:38.396 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.396 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.396 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:38.396 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.037758+0000", 2024-08-19T14:28:38.396 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.001549351 2024-08-19T14:28:38.396 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.396 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.396 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:38.396 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.037977+0000", 2024-08-19T14:28:38.396 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00021854399999999999 2024-08-19T14:28:38.396 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.396 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.396 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:38.397 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.421288+0000", 2024-08-19T14:28:38.397 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.383311179 2024-08-19T14:28:38.397 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.397 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.397 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:28:38.397 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.421387+0000", 2024-08-19T14:28:38.397 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.9487999999999993e-05 2024-08-19T14:28:38.397 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.397 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.397 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:28:38.397 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.422188+0000", 2024-08-19T14:28:38.397 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00080060600000000004 2024-08-19T14:28:38.397 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:28:38.398 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:28:38.398 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:28:38.398 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.398 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.398 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25813.0:132698 5.17 5:e9ffa124:::benchmark_data_smithi045_85309_object12445:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e462)", 2024-08-19T14:28:38.398 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:26:29.076721+0000", 2024-08-19T14:28:38.398 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 129.227803986, 2024-08-19T14:28:38.398 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.346118178, 2024-08-19T14:28:38.398 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:28:38.398 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:28:38.399 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:28:38.399 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:28:38.399 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25813", 2024-08-19T14:28:38.399 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:28:38.399 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 132698 2024-08-19T14:28:38.399 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.399 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:28:38.399 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.399 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:28:38.399 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.076721+0000", 2024-08-19T14:28:38.399 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:28:38.399 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.399 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.400 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:28:38.400 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.076721+0000", 2024-08-19T14:28:38.400 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:28:38.400 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.400 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.400 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:28:38.400 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.076723+0000", 2024-08-19T14:28:38.400 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.519e-06 2024-08-19T14:28:38.400 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.400 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.400 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:28:38.400 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.076728+0000", 2024-08-19T14:28:38.400 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.011e-06 2024-08-19T14:28:38.401 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.401 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.401 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:28:38.401 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.076743+0000", 2024-08-19T14:28:38.401 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4952e-05 2024-08-19T14:28:38.401 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.401 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.401 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:28:38.401 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.076751+0000", 2024-08-19T14:28:38.401 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.5199999999999997e-06 2024-08-19T14:28:38.401 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.401 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.401 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:28:38.402 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.076826+0000", 2024-08-19T14:28:38.402 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.4789999999999994e-05 2024-08-19T14:28:38.402 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.402 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.402 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:28:38.402 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.076991+0000", 2024-08-19T14:28:38.402 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00016521600000000001 2024-08-19T14:28:38.402 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.402 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.402 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,8,11", 2024-08-19T14:28:38.402 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.077122+0000", 2024-08-19T14:28:38.402 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000130588 2024-08-19T14:28:38.402 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.403 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.403 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:28:38.403 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.079196+0000", 2024-08-19T14:28:38.403 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0020745350000000002 2024-08-19T14:28:38.403 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.403 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.403 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:38.403 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.080180+0000", 2024-08-19T14:28:38.403 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00098345099999999999 2024-08-19T14:28:38.403 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.403 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.403 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:38.403 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.080323+0000", 2024-08-19T14:28:38.403 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000143583 2024-08-19T14:28:38.404 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.404 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.404 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:38.404 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.422252+0000", 2024-08-19T14:28:38.404 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.34192826399999998 2024-08-19T14:28:38.404 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.404 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.404 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:28:38.404 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.422327+0000", 2024-08-19T14:28:38.404 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.5099999999999996e-05 2024-08-19T14:28:38.404 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.404 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.404 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:28:38.405 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.422839+0000", 2024-08-19T14:28:38.405 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000512649 2024-08-19T14:28:38.405 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:28:38.405 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:28:38.405 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:28:38.405 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.405 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.405 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25813.0:132733 5.17 5:e9c717cb:::benchmark_data_smithi045_85309_object12480:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e462)", 2024-08-19T14:28:38.405 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:26:29.101389+0000", 2024-08-19T14:28:38.405 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 129.203136003, 2024-08-19T14:28:38.405 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.32198057000000002, 2024-08-19T14:28:38.405 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:28:38.405 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:28:38.406 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:28:38.406 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:28:38.406 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25813", 2024-08-19T14:28:38.406 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:28:38.406 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 132733 2024-08-19T14:28:38.406 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.406 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:28:38.406 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.406 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:28:38.406 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.101389+0000", 2024-08-19T14:28:38.406 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:28:38.406 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.407 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.407 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:28:38.407 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.101389+0000", 2024-08-19T14:28:38.407 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:28:38.407 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.407 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.407 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:28:38.407 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.101391+0000", 2024-08-19T14:28:38.407 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.401e-06 2024-08-19T14:28:38.407 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.408 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.408 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:28:38.408 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.101396+0000", 2024-08-19T14:28:38.408 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.6810000000000003e-06 2024-08-19T14:28:38.408 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.408 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.408 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:28:38.408 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.101414+0000", 2024-08-19T14:28:38.409 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.7323999999999999e-05 2024-08-19T14:28:38.409 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.409 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.409 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:28:38.410 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.101424+0000", 2024-08-19T14:28:38.410 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0479000000000001e-05 2024-08-19T14:28:38.410 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.410 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.410 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:28:38.410 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.101497+0000", 2024-08-19T14:28:38.410 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.3008000000000003e-05 2024-08-19T14:28:38.410 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.410 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.410 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:28:38.410 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.101672+0000", 2024-08-19T14:28:38.410 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00017511600000000001 2024-08-19T14:28:38.411 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.411 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.411 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,8,11", 2024-08-19T14:28:38.411 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.101846+0000", 2024-08-19T14:28:38.411 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000174179 2024-08-19T14:28:38.411 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.411 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.411 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:28:38.411 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.104105+0000", 2024-08-19T14:28:38.411 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.002259063 2024-08-19T14:28:38.411 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.411 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.411 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:38.411 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.104876+0000", 2024-08-19T14:28:38.412 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00077034 2024-08-19T14:28:38.412 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.412 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.412 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:38.412 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.107300+0000", 2024-08-19T14:28:38.412 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0024242370000000001 2024-08-19T14:28:38.412 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.412 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.412 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:38.412 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.422905+0000", 2024-08-19T14:28:38.412 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.31560517500000002 2024-08-19T14:28:38.412 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.412 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.413 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:28:38.413 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.422964+0000", 2024-08-19T14:28:38.413 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.8692e-05 2024-08-19T14:28:38.413 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.413 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.413 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:28:38.413 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.423370+0000", 2024-08-19T14:28:38.413 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00040587499999999999 2024-08-19T14:28:38.413 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:28:38.413 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:28:38.413 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:28:38.413 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.413 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.413 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25813.0:237705 5.7 5:e5695e68:::benchmark_data_smithi045_85309_object117452:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e495)", 2024-08-19T14:28:38.414 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:28:07.101072+0000", 2024-08-19T14:28:38.414 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 31.203453044, 2024-08-19T14:28:38.414 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.231283661, 2024-08-19T14:28:38.414 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:28:38.414 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:28:38.414 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:28:38.414 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:28:38.414 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25813", 2024-08-19T14:28:38.414 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:28:38.414 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 237705 2024-08-19T14:28:38.414 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.414 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:28:38.414 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.415 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:28:38.415 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101072+0000", 2024-08-19T14:28:38.415 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:28:38.415 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.415 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.415 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:28:38.415 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101072+0000", 2024-08-19T14:28:38.415 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:28:38.415 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.416 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.416 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:28:38.416 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101074+0000", 2024-08-19T14:28:38.416 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.751e-06 2024-08-19T14:28:38.416 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.416 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.416 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:28:38.416 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101080+0000", 2024-08-19T14:28:38.416 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.9050000000000002e-06 2024-08-19T14:28:38.416 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.416 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.416 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:28:38.416 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101096+0000", 2024-08-19T14:28:38.417 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6364000000000001e-05 2024-08-19T14:28:38.417 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.417 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.417 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:28:38.417 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101107+0000", 2024-08-19T14:28:38.417 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0572e-05 2024-08-19T14:28:38.417 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.417 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.417 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:28:38.417 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101166+0000", 2024-08-19T14:28:38.417 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.8981999999999999e-05 2024-08-19T14:28:38.417 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.417 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.417 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:28:38.418 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101181+0000", 2024-08-19T14:28:38.418 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5364000000000001e-05 2024-08-19T14:28:38.418 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.418 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.418 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:28:38.418 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.315274+0000", 2024-08-19T14:28:38.418 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.21409246900000001 2024-08-19T14:28:38.418 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.418 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.418 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for missing object", 2024-08-19T14:28:38.419 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.315413+0000", 2024-08-19T14:28:38.419 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000139782 2024-08-19T14:28:38.419 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.419 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.419 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:28:38.419 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.332277+0000", 2024-08-19T14:28:38.419 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.016863488999999999 2024-08-19T14:28:38.419 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.419 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.419 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:28:38.419 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.332356+0000", 2024-08-19T14:28:38.419 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.8982999999999999e-05 2024-08-19T14:28:38.419 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:28:38.420 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:28:38.420 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:28:38.420 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.420 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.420 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25813.0:237714 5.7 5:e252da68:::benchmark_data_smithi045_85309_object117461:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e495)", 2024-08-19T14:28:38.420 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:28:07.101128+0000", 2024-08-19T14:28:38.420 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 31.20339757, 2024-08-19T14:28:38.420 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.24052662399999999, 2024-08-19T14:28:38.420 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:28:38.420 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:28:38.420 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:28:38.420 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:28:38.420 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25813", 2024-08-19T14:28:38.421 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:28:38.421 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 237714 2024-08-19T14:28:38.421 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.421 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:28:38.421 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.421 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:28:38.421 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101128+0000", 2024-08-19T14:28:38.421 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:28:38.421 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.421 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.421 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:28:38.421 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101128+0000", 2024-08-19T14:28:38.421 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:28:38.421 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.422 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.422 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:28:38.422 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101128+0000", 2024-08-19T14:28:38.422 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.8500000000000002e-07 2024-08-19T14:28:38.422 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.422 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.422 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:28:38.422 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101131+0000", 2024-08-19T14:28:38.422 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.6910000000000002e-06 2024-08-19T14:28:38.422 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.422 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.422 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:28:38.422 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101139+0000", 2024-08-19T14:28:38.423 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.1799999999999996e-06 2024-08-19T14:28:38.423 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.423 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.423 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:28:38.423 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101145+0000", 2024-08-19T14:28:38.423 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.7359999999999998e-06 2024-08-19T14:28:38.423 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.423 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.423 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:28:38.423 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101214+0000", 2024-08-19T14:28:38.423 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.9695000000000003e-05 2024-08-19T14:28:38.423 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.423 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.423 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:28:38.424 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101224+0000", 2024-08-19T14:28:38.424 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0043e-05 2024-08-19T14:28:38.424 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.424 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.424 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:28:38.424 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.315795+0000", 2024-08-19T14:28:38.424 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.21457066599999999 2024-08-19T14:28:38.424 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.424 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.424 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:28:38.424 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.315966+0000", 2024-08-19T14:28:38.424 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00017090700000000001 2024-08-19T14:28:38.424 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.425 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.425 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,5", 2024-08-19T14:28:38.425 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.316585+0000", 2024-08-19T14:28:38.425 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00061851899999999999 2024-08-19T14:28:38.425 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.425 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.425 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:28:38.425 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.332349+0000", 2024-08-19T14:28:38.425 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.015764158 2024-08-19T14:28:38.425 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.425 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.425 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:38.425 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.332428+0000", 2024-08-19T14:28:38.425 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.9651e-05 2024-08-19T14:28:38.426 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.426 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.426 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:38.426 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.341243+0000", 2024-08-19T14:28:38.426 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0088151610000000002 2024-08-19T14:28:38.426 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.426 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.426 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:28:38.426 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.341320+0000", 2024-08-19T14:28:38.426 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.6304000000000007e-05 2024-08-19T14:28:38.426 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.426 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.426 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:28:38.427 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.341654+0000", 2024-08-19T14:28:38.427 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000334428 2024-08-19T14:28:38.427 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:28:38.427 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:28:38.427 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:28:38.427 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.427 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.427 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25813.0:237731 5.7 5:e54e0cab:::benchmark_data_smithi045_85309_object117478:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e495)", 2024-08-19T14:28:38.427 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:28:07.101155+0000", 2024-08-19T14:28:38.427 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 31.203369909999999, 2024-08-19T14:28:38.427 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.240822699, 2024-08-19T14:28:38.427 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:28:38.427 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:28:38.428 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:28:38.428 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:28:38.428 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25813", 2024-08-19T14:28:38.428 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:28:38.428 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 237731 2024-08-19T14:28:38.428 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.428 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:28:38.428 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.428 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:28:38.428 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101155+0000", 2024-08-19T14:28:38.428 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:28:38.428 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.428 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.429 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:28:38.429 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101155+0000", 2024-08-19T14:28:38.429 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:28:38.429 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.429 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.429 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:28:38.429 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101156+0000", 2024-08-19T14:28:38.429 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.5900000000000003e-07 2024-08-19T14:28:38.429 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.429 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.429 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:28:38.430 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101158+0000", 2024-08-19T14:28:38.430 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.0420000000000001e-06 2024-08-19T14:28:38.430 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.430 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.430 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:28:38.430 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101164+0000", 2024-08-19T14:28:38.430 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.7769999999999997e-06 2024-08-19T14:28:38.430 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.430 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.430 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:28:38.430 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101184+0000", 2024-08-19T14:28:38.430 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.9989000000000001e-05 2024-08-19T14:28:38.430 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.431 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.431 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:28:38.431 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101265+0000", 2024-08-19T14:28:38.431 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.0782e-05 2024-08-19T14:28:38.431 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.431 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.431 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:28:38.431 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101274+0000", 2024-08-19T14:28:38.431 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.0780000000000002e-06 2024-08-19T14:28:38.431 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.431 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.431 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:28:38.432 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.320042+0000", 2024-08-19T14:28:38.432 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.21876728300000001 2024-08-19T14:28:38.432 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.432 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.432 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:28:38.432 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.320209+0000", 2024-08-19T14:28:38.432 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00016716800000000001 2024-08-19T14:28:38.432 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.432 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.432 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,5", 2024-08-19T14:28:38.432 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.320830+0000", 2024-08-19T14:28:38.432 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00062140999999999997 2024-08-19T14:28:38.433 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.433 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.433 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:28:38.433 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.332354+0000", 2024-08-19T14:28:38.433 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.011524123000000001 2024-08-19T14:28:38.433 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.433 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.433 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:38.433 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.341182+0000", 2024-08-19T14:28:38.433 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0088281499999999999 2024-08-19T14:28:38.433 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.433 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.433 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:38.434 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.341703+0000", 2024-08-19T14:28:38.434 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000521022 2024-08-19T14:28:38.434 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.434 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.434 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:28:38.434 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.341740+0000", 2024-08-19T14:28:38.434 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.6885999999999999e-05 2024-08-19T14:28:38.434 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.434 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.434 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:28:38.434 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.341978+0000", 2024-08-19T14:28:38.434 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00023763000000000001 2024-08-19T14:28:38.434 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:28:38.435 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:28:38.435 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:28:38.435 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.435 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.435 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25813.0:237758 5.7 5:e57a2935:::benchmark_data_smithi045_85309_object117505:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e495)", 2024-08-19T14:28:38.435 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:28:07.101204+0000", 2024-08-19T14:28:38.435 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 31.203320881, 2024-08-19T14:28:38.435 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.24687310400000001, 2024-08-19T14:28:38.435 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:28:38.435 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:28:38.435 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:28:38.435 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:28:38.436 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25813", 2024-08-19T14:28:38.436 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:28:38.436 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 237758 2024-08-19T14:28:38.436 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.436 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:28:38.436 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.436 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:28:38.436 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101204+0000", 2024-08-19T14:28:38.436 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:28:38.436 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.436 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.436 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:28:38.436 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101204+0000", 2024-08-19T14:28:38.437 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:28:38.437 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.437 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.437 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:28:38.437 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101205+0000", 2024-08-19T14:28:38.437 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2800000000000002e-07 2024-08-19T14:28:38.437 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.437 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.437 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:28:38.437 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101207+0000", 2024-08-19T14:28:38.437 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.768e-06 2024-08-19T14:28:38.437 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.437 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.438 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:28:38.438 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101215+0000", 2024-08-19T14:28:38.438 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.3900000000000004e-06 2024-08-19T14:28:38.438 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.438 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.438 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:28:38.438 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101220+0000", 2024-08-19T14:28:38.438 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.3040000000000004e-06 2024-08-19T14:28:38.438 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.438 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.438 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:28:38.438 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101303+0000", 2024-08-19T14:28:38.438 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.2443999999999999e-05 2024-08-19T14:28:38.439 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.439 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.439 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:28:38.439 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101311+0000", 2024-08-19T14:28:38.439 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.3389999999999993e-06 2024-08-19T14:28:38.439 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.439 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.439 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:28:38.439 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.324142+0000", 2024-08-19T14:28:38.439 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.222830844 2024-08-19T14:28:38.440 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.440 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.440 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:28:38.440 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.324274+0000", 2024-08-19T14:28:38.440 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013226199999999999 2024-08-19T14:28:38.440 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.440 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.440 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,5", 2024-08-19T14:28:38.440 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.324842+0000", 2024-08-19T14:28:38.440 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00056831899999999996 2024-08-19T14:28:38.440 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.440 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.440 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:28:38.441 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.332438+0000", 2024-08-19T14:28:38.441 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0075956060000000004 2024-08-19T14:28:38.441 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.441 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.441 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:38.441 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.342013+0000", 2024-08-19T14:28:38.441 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0095747650000000007 2024-08-19T14:28:38.441 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.441 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.441 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:38.441 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.347519+0000", 2024-08-19T14:28:38.441 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0055063330000000004 2024-08-19T14:28:38.441 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.442 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.442 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:28:38.442 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.347576+0000", 2024-08-19T14:28:38.442 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.6610000000000002e-05 2024-08-19T14:28:38.442 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.442 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.442 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:28:38.442 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.348077+0000", 2024-08-19T14:28:38.442 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00050169199999999998 2024-08-19T14:28:38.442 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:28:38.442 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:28:38.442 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:28:38.443 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.443 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.443 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25813.0:239500 5.7 5:e628e8a7:::benchmark_data_smithi045_85309_object119247:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e498)", 2024-08-19T14:28:38.443 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:28:10.817288+0000", 2024-08-19T14:28:38.443 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 27.487236939999999, 2024-08-19T14:28:38.443 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.16529574399999999, 2024-08-19T14:28:38.443 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:28:38.443 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:28:38.443 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:28:38.443 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:28:38.443 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25813", 2024-08-19T14:28:38.443 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:28:38.443 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 239500 2024-08-19T14:28:38.443 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.444 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:28:38.444 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.444 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:28:38.444 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:10.817288+0000", 2024-08-19T14:28:38.444 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:28:38.444 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.444 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.444 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:28:38.444 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:10.817288+0000", 2024-08-19T14:28:38.445 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:28:38.445 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.445 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.445 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:28:38.445 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:10.817290+0000", 2024-08-19T14:28:38.445 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3799999999999999e-06 2024-08-19T14:28:38.445 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.445 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.445 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:28:38.445 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:10.817295+0000", 2024-08-19T14:28:38.445 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.2680000000000002e-06 2024-08-19T14:28:38.445 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.445 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.445 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:28:38.445 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:10.817308+0000", 2024-08-19T14:28:38.446 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2738999999999999e-05 2024-08-19T14:28:38.446 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.446 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.446 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:28:38.446 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:10.817315+0000", 2024-08-19T14:28:38.446 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.0650000000000001e-06 2024-08-19T14:28:38.446 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.446 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.446 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:28:38.446 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:10.817375+0000", 2024-08-19T14:28:38.446 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.0174999999999999e-05 2024-08-19T14:28:38.446 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.446 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.447 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:28:38.447 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:10.817518+0000", 2024-08-19T14:28:38.447 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00014345300000000001 2024-08-19T14:28:38.447 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.447 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.447 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,5", 2024-08-19T14:28:38.447 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:10.818121+0000", 2024-08-19T14:28:38.447 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00060233800000000001 2024-08-19T14:28:38.447 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.448 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.448 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:28:38.448 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:10.821146+0000", 2024-08-19T14:28:38.448 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0030256469999999998 2024-08-19T14:28:38.448 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.448 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.448 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:38.448 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:10.826024+0000", 2024-08-19T14:28:38.448 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.004878009 2024-08-19T14:28:38.448 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.448 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.448 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:28:38.448 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:10.982143+0000", 2024-08-19T14:28:38.448 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.156118857 2024-08-19T14:28:38.448 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.449 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.449 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:28:38.449 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:10.982195+0000", 2024-08-19T14:28:38.449 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1990999999999997e-05 2024-08-19T14:28:38.449 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:28:38.449 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:28:38.449 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:28:38.449 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:10.982584+0000", 2024-08-19T14:28:38.449 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00038882199999999998 2024-08-19T14:28:38.449 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:28:38.450 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:28:38.450 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:28:38.450 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:28:38.450 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:28:38.450 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:28:38.450 DEBUG:teuthology.orchestra.run.smithi045:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_4 bench 90 write' 2024-08-19T14:28:38.451 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-08-19T14:28:38.451 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-19T14:28:38.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:38.453 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:38.453 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:28:38.454 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-08-19T14:28:38.454 INFO:teuthology.orchestra.run:waiting for 3000 2024-08-19T14:28:38.505 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:hints = 1 2024-08-19T14:28:38.506 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-08-19T14:28:38.506 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Object prefix: benchmark_data_smithi045_97937 2024-08-19T14:28:38.506 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-19T14:28:38.506 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 0 0 0 0 0 0 - 0 2024-08-19T14:28:38.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:38 smithi032 ceph-mon[20004]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-19T14:28:38.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:38 smithi045 ceph-mon[20143]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-19T14:28:38.554 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:28:38.593 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:38.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:38.594 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:28:38.695 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:28:38.727 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:38.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:38.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:28:38.829 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:28:38.866 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:38.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:38.867 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:28:38.968 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:28:39.008 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:39.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:39.009 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:28:39.110 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:28:39.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:39.139 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:39.139 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:28:39.240 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:28:39.266 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:39.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:39.267 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:28:39.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:39 smithi031 ceph-mon[16336]: pgmap v1175: 47 pgs: 3 creating+peering, 44 active+clean; 577 KiB data, 4.5 GiB used, 621 GiB / 626 GiB avail 2024-08-19T14:28:39.367 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:28:39.385 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:28:39.386 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:28:39.406 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:39.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:39.407 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:28:39.423 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:28:39.423 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:28:39.462 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:28:39.462 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:28:39.494 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:28:39.494 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:28:39.506 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 1 16 3981 3965 247.784 247.812 0.00168122 0.00394982 2024-08-19T14:28:39.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:39 smithi032 ceph-mon[20004]: pgmap v1175: 47 pgs: 3 creating+peering, 44 active+clean; 577 KiB data, 4.5 GiB used, 621 GiB / 626 GiB avail 2024-08-19T14:28:39.508 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:28:39.511 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:28:39.511 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:28:39.528 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:28:39.528 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:28:39.529 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:39.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:39.530 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:28:39.544 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:28:39.544 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:28:39.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:39 smithi045 ceph-mon[20143]: pgmap v1175: 47 pgs: 3 creating+peering, 44 active+clean; 577 KiB data, 4.5 GiB used, 621 GiB / 626 GiB avail 2024-08-19T14:28:39.562 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:28:39.562 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:28:39.579 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:28:39.579 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:28:39.596 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:28:39.596 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:28:39.613 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:28:39.613 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:28:39.630 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:28:39.630 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:28:39.630 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:28:39.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:39.652 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:39.652 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:28:39.738 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-19T14:28:39.752 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:28:39.816 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:39.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:39.818 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:28:39.907 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-08-19T14:28:39.919 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:28:39.944 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:39.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:39.945 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:28:40.046 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:28:40.070 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:40.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:40.071 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:28:40.172 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:28:40.190 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:28:40.195 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:40.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:40.196 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:28:40.297 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:28:40.323 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:40.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:40.324 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:28:40.425 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:28:40.449 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:40.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:40.450 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:28:40.506 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 2 15 7240 7225 225.758 203.75 0.00478818 0.00442236 2024-08-19T14:28:40.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:40 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/843206946' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:28:40.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:40 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/843206946' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:28:40.551 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:28:40.577 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:40.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:40.578 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:28:40.611 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:40 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/843206946' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:28:40.678 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:28:40.702 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:40.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:40.703 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:28:40.804 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:28:40.840 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:40.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:40.841 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:28:40.916 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:28:40.942 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:28:40.965 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:40.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:40.966 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:28:41.067 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:28:41.093 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:41.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:41.094 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:28:41.194 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:28:41.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:41.226 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:41.227 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:28:41.263 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:28:41.329 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:28:41.364 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:41.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:41.365 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:28:41.466 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:28:41.501 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:41.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:41.502 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:28:41.506 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 3 16 11003 10987 228.874 235.125 0.00567117 0.0043628 2024-08-19T14:28:41.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:41 smithi032 ceph-mon[20004]: pgmap v1176: 47 pgs: 47 active+clean; 53 MiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 7.6 MiB/s wr, 120 op/s 2024-08-19T14:28:41.543 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:41 smithi031 ceph-mon[16336]: pgmap v1176: 47 pgs: 47 active+clean; 53 MiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 7.6 MiB/s wr, 120 op/s 2024-08-19T14:28:41.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:41 smithi045 ceph-mon[20143]: pgmap v1176: 47 pgs: 47 active+clean; 53 MiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 7.6 MiB/s wr, 120 op/s 2024-08-19T14:28:41.603 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:28:41.606 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:28:41.607 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-08-19T14:28:41.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:41.630 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:41.630 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:28:41.731 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:28:41.756 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:41.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:41.758 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:28:41.859 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:28:41.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:41.900 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:41.901 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:28:42.001 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:28:42.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:42.036 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:42.037 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:28:42.137 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:28:42.172 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:42.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:42.173 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:28:42.274 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:28:42.311 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:42.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:42.312 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:28:42.413 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:28:42.447 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:42.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:42.448 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:28:42.506 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 4 16 14586 14570 227.636 223.938 0.00288976 0.00438733 2024-08-19T14:28:42.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:42 smithi032 ceph-mon[20004]: from='client.26203 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:28:42.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:42 smithi045 ceph-mon[20143]: from='client.26203 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:28:42.549 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:28:42.570 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:42.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:42.571 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:28:42.584 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:42 smithi031 ceph-mon[16336]: from='client.26203 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:28:42.610 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":1176,"stamp":"2024-08-19T14:28:40.016543+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":55050784,"num_objects":833,"num_object_clones":0,"num_object_copies":2499,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":833,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":831,"num_write_kb":53184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5140,"ondisk_log_size":5140,"up":141,"acting":141,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":136,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":4733084,"kb_used_data":48636,"kb_used_omap":10,"kb_used_meta":4684341,"kb_avail":651482980,"statfs":{"total":671965249536,"available":667118571520,"internally_reserved":0,"allocated":49803264,"data_stored":33550526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11158,"internal_metadata":4796765290},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":54460416,"num_objects":831,"num_object_clones":0,"num_object_copies":2493,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":831,"num_write_kb":53184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.873235"},"pg_stats":[{"pgid":"6.7","version":"513'91","reported_seq":107,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-08-19T14:28:38.859079+0000","last_change":"2024-08-19T14:28:33.150392+0000","last_active":"2024-08-19T14:28:38.859079+0000","last_peered":"2024-08-19T14:28:38.859079+0000","last_clean":"2024-08-19T14:28:38.859079+0000","last_became_active":"2024-08-19T14:28:33.150057+0000","last_became_peered":"2024-08-19T14:28:33.150057+0000","last_unstale":"2024-08-19T14:28:38.859079+0000","last_undegraded":"2024-08-19T14:28:38.859079+0000","last_fullsized":"2024-08-19T14:28:38.859079+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":510,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:28:32.139424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:28:32.139424+0000","last_clean_scrub_stamp":"2024-08-19T14:28:32.139424+0000","objects_scrubbed":0,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:29:48.570044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5963776,"num_objects":91,"num_object_clones":0,"num_object_copies":273,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":91,"num_write_kb":5824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"70'139","reported_seq":1287,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-08-19T14:28:36.854116+0000","last_change":"2024-08-19T14:28:20.867081+0000","last_active":"2024-08-19T14:28:36.854116+0000","last_peered":"2024-08-19T14:28:36.854116+0000","last_clean":"2024-08-19T14:28:36.854116+0000","last_became_active":"2024-08-19T14:28:20.866740+0000","last_became_peered":"2024-08-19T14:28:20.866740+0000","last_unstale":"2024-08-19T14:28:36.854116+0000","last_undegraded":"2024-08-19T14:28:36.854116+0000","last_fullsized":"2024-08-19T14:28:36.854116+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:27:03.929012+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:19:58.072083+0000","last_clean_scrub_stamp":"2024-08-19T14:27:03.929012+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,4],"acting":[7,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"513'60","reported_seq":76,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-08-19T14:28:38.745337+0000","last_change":"2024-08-19T14:28:33.151848+0000","last_active":"2024-08-19T14:28:38.745337+0000","last_peered":"2024-08-19T14:28:38.745337+0000","last_clean":"2024-08-19T14:28:38.745337+0000","last_became_active":"2024-08-19T14:28:33.150851+0000","last_became_peered":"2024-08-19T14:28:33.150851+0000","last_unstale":"2024-08-19T14:28:38.745337+0000","last_undegraded":"2024-08-19T14:28:38.745337+0000","last_fullsized":"2024-08-19T14:28:38.745337+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":510,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:28:32.139424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:28:32.139424+0000","last_clean_scrub_stamp":"2024-08-19T14:28:32.139424+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:29:36.912715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3932160,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":60,"num_write_kb":3840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"70'139","reported_seq":1176,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-08-19T14:28:36.854185+0000","last_change":"2024-08-19T14:28:33.188989+0000","last_active":"2024-08-19T14:28:36.854185+0000","last_peered":"2024-08-19T14:28:36.854185+0000","last_clean":"2024-08-19T14:28:36.854185+0000","last_became_active":"2024-08-19T14:18:13.804721+0000","last_became_peered":"2024-08-19T14:18:13.804721+0000","last_unstale":"2024-08-19T14:28:36.854185+0000","last_undegraded":"2024-08-19T14:28:36.854185+0000","last_fullsized":"2024-08-19T14:28:36.854185+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:33.188905+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:28:33.188905+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:29:33.790602+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"513'109","reported_seq":125,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-08-19T14:28:38.954369+0000","last_change":"2024-08-19T14:28:33.153384+0000","last_active":"2024-08-19T14:28:38.954369+0000","last_peered":"2024-08-19T14:28:38.954369+0000","last_clean":"2024-08-19T14:28:38.954369+0000","last_became_active":"2024-08-19T14:28:33.152697+0000","last_became_peered":"2024-08-19T14:28:33.152697+0000","last_unstale":"2024-08-19T14:28:38.954369+0000","last_undegraded":"2024-08-19T14:28:38.954369+0000","last_fullsized":"2024-08-19T14:28:38.954369+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":510,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:28:32.139424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:28:32.139424+0000","last_clean_scrub_stamp":"2024-08-19T14:28:32.139424+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:29:33.284654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"70'139","reported_seq":1124,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-08-19T14:28:36.853936+0000","last_change":"2024-08-19T14:28:26.020419+0000","last_active":"2024-08-19T14:28:36.853936+0000","last_peered":"2024-08-19T14:28:36.853936+0000","last_clean":"2024-08-19T14:28:36.853936+0000","last_became_active":"2024-08-19T14:20:50.742354+0000","last_became_peered":"2024-08-19T14:20:50.742354+0000","last_unstale":"2024-08-19T14:28:36.853936+0000","last_undegraded":"2024-08-19T14:28:36.853936+0000","last_fullsized":"2024-08-19T14:28:36.853936+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:26.020327+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:19:59.302931+0000","last_clean_scrub_stamp":"2024-08-19T14:28:26.020327+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:29:30.428255+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.6","version":"513'86","reported_seq":101,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-08-19T14:28:38.871018+0000","last_change":"2024-08-19T14:28:33.151142+0000","last_active":"2024-08-19T14:28:38.871018+0000","last_peered":"2024-08-19T14:28:38.871018+0000","last_clean":"2024-08-19T14:28:38.871018+0000","last_became_active":"2024-08-19T14:28:33.150830+0000","last_became_peered":"2024-08-19T14:28:33.150830+0000","last_unstale":"2024-08-19T14:28:38.871018+0000","last_undegraded":"2024-08-19T14:28:38.871018+0000","last_fullsized":"2024-08-19T14:28:38.871018+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":510,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:28:32.139424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:28:32.139424+0000","last_clean_scrub_stamp":"2024-08-19T14:28:32.139424+0000","objects_scrubbed":0,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:29:49.490797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5636096,"num_objects":86,"num_object_clones":0,"num_object_copies":258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":86,"num_write_kb":5504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"70'139","reported_seq":387,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-08-19T14:28:36.854033+0000","last_change":"2024-08-19T14:28:28.323888+0000","last_active":"2024-08-19T14:28:36.854033+0000","last_peered":"2024-08-19T14:28:36.854033+0000","last_clean":"2024-08-19T14:28:36.854033+0000","last_became_active":"2024-08-19T14:28:20.860080+0000","last_became_peered":"2024-08-19T14:28:20.860080+0000","last_unstale":"2024-08-19T14:28:36.854033+0000","last_undegraded":"2024-08-19T14:28:36.854033+0000","last_fullsized":"2024-08-19T14:28:36.854033+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:28.323773+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:28:28.323773+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:29:51.382221+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":16,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-08-19T14:28:36.853245+0000","last_change":"2024-08-19T14:28:33.150140+0000","last_active":"2024-08-19T14:28:36.853245+0000","last_peered":"2024-08-19T14:28:36.853245+0000","last_clean":"2024-08-19T14:28:36.853245+0000","last_became_active":"2024-08-19T14:28:33.149519+0000","last_became_peered":"2024-08-19T14:28:33.149519+0000","last_unstale":"2024-08-19T14:28:36.853245+0000","last_undegraded":"2024-08-19T14:28:36.853245+0000","last_fullsized":"2024-08-19T14:28:36.853245+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":510,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:28:32.139424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:28:32.139424+0000","last_clean_scrub_stamp":"2024-08-19T14:28:32.139424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:29:35.138434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"70'139","reported_seq":824,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-08-19T14:28:36.853616+0000","last_change":"2024-08-19T14:28:27.537007+0000","last_active":"2024-08-19T14:28:36.853616+0000","last_peered":"2024-08-19T14:28:36.853616+0000","last_clean":"2024-08-19T14:28:36.853616+0000","last_became_active":"2024-08-19T14:20:28.204384+0000","last_became_peered":"2024-08-19T14:20:28.204384+0000","last_unstale":"2024-08-19T14:28:36.853616+0000","last_undegraded":"2024-08-19T14:28:36.853616+0000","last_fullsized":"2024-08-19T14:28:36.853616+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:27.536928+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:21:46.057624+0000","last_clean_scrub_stamp":"2024-08-19T14:28:27.536928+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:29:28.630006+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":13,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-08-19T14:28:34.535747+0000","last_change":"2024-08-19T14:28:33.151118+0000","last_active":"2024-08-19T14:28:34.535747+0000","last_peered":"2024-08-19T14:28:34.535747+0000","last_clean":"2024-08-19T14:28:34.535747+0000","last_became_active":"2024-08-19T14:28:33.150732+0000","last_became_peered":"2024-08-19T14:28:33.150732+0000","last_unstale":"2024-08-19T14:28:34.535747+0000","last_undegraded":"2024-08-19T14:28:34.535747+0000","last_fullsized":"2024-08-19T14:28:34.535747+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":510,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:28:32.139424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:28:32.139424+0000","last_clean_scrub_stamp":"2024-08-19T14:28:32.139424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:29:43.711132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,5],"acting":[8,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.6","version":"70'139","reported_seq":368,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-08-19T14:28:34.535856+0000","last_change":"2024-08-19T14:28:08.847324+0000","last_active":"2024-08-19T14:28:34.535856+0000","last_peered":"2024-08-19T14:28:34.535856+0000","last_clean":"2024-08-19T14:28:34.535856+0000","last_became_active":"2024-08-19T14:28:08.847096+0000","last_became_peered":"2024-08-19T14:28:08.847096+0000","last_unstale":"2024-08-19T14:28:34.535856+0000","last_undegraded":"2024-08-19T14:28:34.535856+0000","last_fullsized":"2024-08-19T14:28:34.535856+0000","mapping_epoch":496,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":497,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:27:26.969901+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:27.848870+0000","last_clean_scrub_stamp":"2024-08-19T14:27:26.969901+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:56.346017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,3],"acting":[8,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":16,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-08-19T14:28:36.155656+0000","last_change":"2024-08-19T14:28:33.151777+0000","last_active":"2024-08-19T14:28:36.155656+0000","last_peered":"2024-08-19T14:28:36.155656+0000","last_clean":"2024-08-19T14:28:36.155656+0000","last_became_active":"2024-08-19T14:28:33.151451+0000","last_became_peered":"2024-08-19T14:28:33.151451+0000","last_unstale":"2024-08-19T14:28:36.155656+0000","last_undegraded":"2024-08-19T14:28:36.155656+0000","last_fullsized":"2024-08-19T14:28:36.155656+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":510,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:28:32.139424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:28:32.139424+0000","last_clean_scrub_stamp":"2024-08-19T14:28:32.139424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:29:43.319838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,8],"acting":[4,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"70'139","reported_seq":343,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-08-19T14:28:36.854170+0000","last_change":"2024-08-19T14:28:27.331120+0000","last_active":"2024-08-19T14:28:36.854170+0000","last_peered":"2024-08-19T14:28:36.854170+0000","last_clean":"2024-08-19T14:28:36.854170+0000","last_became_active":"2024-08-19T14:28:20.915453+0000","last_became_peered":"2024-08-19T14:28:20.915453+0000","last_unstale":"2024-08-19T14:28:36.854170+0000","last_undegraded":"2024-08-19T14:28:36.854170+0000","last_fullsized":"2024-08-19T14:28:36.854170+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:27.331041+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:28:27.331041+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:29:34.418382+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"513'129","reported_seq":145,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-08-19T14:28:38.944123+0000","last_change":"2024-08-19T14:28:33.151000+0000","last_active":"2024-08-19T14:28:38.944123+0000","last_peered":"2024-08-19T14:28:38.944123+0000","last_clean":"2024-08-19T14:28:38.944123+0000","last_became_active":"2024-08-19T14:28:33.150652+0000","last_became_peered":"2024-08-19T14:28:33.150652+0000","last_unstale":"2024-08-19T14:28:38.944123+0000","last_undegraded":"2024-08-19T14:28:38.944123+0000","last_fullsized":"2024-08-19T14:28:38.944123+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":510,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:28:32.139424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:28:32.139424+0000","last_clean_scrub_stamp":"2024-08-19T14:28:32.139424+0000","objects_scrubbed":0,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:29:45.878306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8454144,"num_objects":129,"num_object_clones":0,"num_object_copies":387,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":129,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":8256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,0],"acting":[7,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"70'139","reported_seq":350,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-08-19T14:28:34.535813+0000","last_change":"2024-08-19T14:28:28.122304+0000","last_active":"2024-08-19T14:28:34.535813+0000","last_peered":"2024-08-19T14:28:34.535813+0000","last_clean":"2024-08-19T14:28:34.535813+0000","last_became_active":"2024-08-19T14:28:20.859362+0000","last_became_peered":"2024-08-19T14:28:20.859362+0000","last_unstale":"2024-08-19T14:28:34.535813+0000","last_undegraded":"2024-08-19T14:28:34.535813+0000","last_fullsized":"2024-08-19T14:28:34.535813+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:28.122172+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:28:28.122172+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:29:30.892036+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,4],"acting":[8,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"6.d","version":"513'64","reported_seq":80,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-08-19T14:28:38.745431+0000","last_change":"2024-08-19T14:28:33.151442+0000","last_active":"2024-08-19T14:28:38.745431+0000","last_peered":"2024-08-19T14:28:38.745431+0000","last_clean":"2024-08-19T14:28:38.745431+0000","last_became_active":"2024-08-19T14:28:33.151135+0000","last_became_peered":"2024-08-19T14:28:33.151135+0000","last_unstale":"2024-08-19T14:28:38.745431+0000","last_undegraded":"2024-08-19T14:28:38.745431+0000","last_fullsized":"2024-08-19T14:28:38.745431+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":510,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:28:32.139424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:28:32.139424+0000","last_clean_scrub_stamp":"2024-08-19T14:28:32.139424+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:29:52.030299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":64,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"70'139","reported_seq":370,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-08-19T14:28:36.155903+0000","last_change":"2024-08-19T14:28:08.847900+0000","last_active":"2024-08-19T14:28:36.155903+0000","last_peered":"2024-08-19T14:28:36.155903+0000","last_clean":"2024-08-19T14:28:36.155903+0000","last_became_active":"2024-08-19T14:28:08.847539+0000","last_became_peered":"2024-08-19T14:28:08.847539+0000","last_unstale":"2024-08-19T14:28:36.155903+0000","last_undegraded":"2024-08-19T14:28:36.155903+0000","last_fullsized":"2024-08-19T14:28:36.155903+0000","mapping_epoch":496,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":497,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:27:25.888493+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:28.864870+0000","last_clean_scrub_stamp":"2024-08-19T14:27:25.888493+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":16,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-08-19T14:28:36.155830+0000","last_change":"2024-08-19T14:28:33.154483+0000","last_active":"2024-08-19T14:28:36.155830+0000","last_peered":"2024-08-19T14:28:36.155830+0000","last_clean":"2024-08-19T14:28:36.155830+0000","last_became_active":"2024-08-19T14:28:33.154031+0000","last_became_peered":"2024-08-19T14:28:33.154031+0000","last_unstale":"2024-08-19T14:28:36.155830+0000","last_undegraded":"2024-08-19T14:28:36.155830+0000","last_fullsized":"2024-08-19T14:28:36.155830+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":510,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:28:32.139424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:28:32.139424+0000","last_clean_scrub_stamp":"2024-08-19T14:28:32.139424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:29:41.323804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"70'139","reported_seq":1301,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-08-19T14:28:36.853934+0000","last_change":"2024-08-19T14:28:08.848316+0000","last_active":"2024-08-19T14:28:36.853934+0000","last_peered":"2024-08-19T14:28:36.853934+0000","last_clean":"2024-08-19T14:28:36.853934+0000","last_became_active":"2024-08-19T14:28:08.847950+0000","last_became_peered":"2024-08-19T14:28:08.847950+0000","last_unstale":"2024-08-19T14:28:36.853934+0000","last_undegraded":"2024-08-19T14:28:36.853934+0000","last_fullsized":"2024-08-19T14:28:36.853934+0000","mapping_epoch":496,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":497,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:27:05.871950+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:27:05.871950+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":16,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-08-19T14:28:36.155957+0000","last_change":"2024-08-19T14:28:33.153338+0000","last_active":"2024-08-19T14:28:36.155957+0000","last_peered":"2024-08-19T14:28:36.155957+0000","last_clean":"2024-08-19T14:28:36.155957+0000","last_became_active":"2024-08-19T14:28:33.153016+0000","last_became_peered":"2024-08-19T14:28:33.153016+0000","last_unstale":"2024-08-19T14:28:36.155957+0000","last_undegraded":"2024-08-19T14:28:36.155957+0000","last_fullsized":"2024-08-19T14:28:36.155957+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":510,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:28:32.139424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:28:32.139424+0000","last_clean_scrub_stamp":"2024-08-19T14:28:32.139424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:29:38.573372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"70'139","reported_seq":347,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-08-19T14:28:36.853329+0000","last_change":"2024-08-19T14:27:32.221695+0000","last_active":"2024-08-19T14:28:36.853329+0000","last_peered":"2024-08-19T14:28:36.853329+0000","last_clean":"2024-08-19T14:28:36.853329+0000","last_became_active":"2024-08-19T14:27:31.834018+0000","last_became_peered":"2024-08-19T14:27:31.834018+0000","last_unstale":"2024-08-19T14:28:36.853329+0000","last_undegraded":"2024-08-19T14:28:36.853329+0000","last_fullsized":"2024-08-19T14:28:36.853329+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:27:32.221578+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:27:32.221578+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:59.258809+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,8],"acting":[3,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"513'84","reported_seq":100,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-08-19T14:28:38.869775+0000","last_change":"2024-08-19T14:28:33.150427+0000","last_active":"2024-08-19T14:28:38.869775+0000","last_peered":"2024-08-19T14:28:38.869775+0000","last_clean":"2024-08-19T14:28:38.869775+0000","last_became_active":"2024-08-19T14:28:33.150105+0000","last_became_peered":"2024-08-19T14:28:33.150105+0000","last_unstale":"2024-08-19T14:28:38.869775+0000","last_undegraded":"2024-08-19T14:28:38.869775+0000","last_fullsized":"2024-08-19T14:28:38.869775+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":510,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:28:32.139424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:28:32.139424+0000","last_clean_scrub_stamp":"2024-08-19T14:28:32.139424+0000","objects_scrubbed":0,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:29:41.906694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5505024,"num_objects":84,"num_object_clones":0,"num_object_copies":252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":84,"num_write_kb":5376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,8],"acting":[3,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"70'139","reported_seq":673,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-08-19T14:28:36.853845+0000","last_change":"2024-08-19T14:28:26.372113+0000","last_active":"2024-08-19T14:28:36.853845+0000","last_peered":"2024-08-19T14:28:36.853845+0000","last_clean":"2024-08-19T14:28:36.853845+0000","last_became_active":"2024-08-19T14:28:20.859513+0000","last_became_peered":"2024-08-19T14:28:20.859513+0000","last_unstale":"2024-08-19T14:28:36.853845+0000","last_undegraded":"2024-08-19T14:28:36.853845+0000","last_fullsized":"2024-08-19T14:28:36.853845+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:26.371992+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:28:26.371992+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:29:47.315095+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":16,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-08-19T14:28:36.853885+0000","last_change":"2024-08-19T14:28:33.148923+0000","last_active":"2024-08-19T14:28:36.853885+0000","last_peered":"2024-08-19T14:28:36.853885+0000","last_clean":"2024-08-19T14:28:36.853885+0000","last_became_active":"2024-08-19T14:28:33.148336+0000","last_became_peered":"2024-08-19T14:28:33.148336+0000","last_unstale":"2024-08-19T14:28:36.853885+0000","last_undegraded":"2024-08-19T14:28:36.853885+0000","last_fullsized":"2024-08-19T14:28:36.853885+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":510,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:28:32.139424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:28:32.139424+0000","last_clean_scrub_stamp":"2024-08-19T14:28:32.139424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:29:43.386602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,1],"acting":[0,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"70'139","reported_seq":852,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-08-19T14:28:36.155966+0000","last_change":"2024-08-19T14:28:26.742527+0000","last_active":"2024-08-19T14:28:36.155966+0000","last_peered":"2024-08-19T14:28:36.155966+0000","last_clean":"2024-08-19T14:28:36.155966+0000","last_became_active":"2024-08-19T14:18:26.884105+0000","last_became_peered":"2024-08-19T14:18:26.884105+0000","last_unstale":"2024-08-19T14:28:36.155966+0000","last_undegraded":"2024-08-19T14:28:36.155966+0000","last_fullsized":"2024-08-19T14:28:36.155966+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:26.742385+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:20:15.321945+0000","last_clean_scrub_stamp":"2024-08-19T14:28:26.742385+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:29:31.989457+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"513'116","reported_seq":132,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-08-19T14:28:38.954006+0000","last_change":"2024-08-19T14:28:33.154021+0000","last_active":"2024-08-19T14:28:38.954006+0000","last_peered":"2024-08-19T14:28:38.954006+0000","last_clean":"2024-08-19T14:28:38.954006+0000","last_became_active":"2024-08-19T14:28:33.153670+0000","last_became_peered":"2024-08-19T14:28:33.153670+0000","last_unstale":"2024-08-19T14:28:38.954006+0000","last_undegraded":"2024-08-19T14:28:38.954006+0000","last_fullsized":"2024-08-19T14:28:38.954006+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":510,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:28:32.139424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:28:32.139424+0000","last_clean_scrub_stamp":"2024-08-19T14:28:32.139424+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:29:59.471284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":116,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"70'139","reported_seq":612,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-08-19T14:28:36.156004+0000","last_change":"2024-08-19T14:27:32.825905+0000","last_active":"2024-08-19T14:28:36.156004+0000","last_peered":"2024-08-19T14:28:36.156004+0000","last_clean":"2024-08-19T14:28:36.156004+0000","last_became_active":"2024-08-19T14:27:31.837754+0000","last_became_peered":"2024-08-19T14:27:31.837754+0000","last_unstale":"2024-08-19T14:28:36.156004+0000","last_undegraded":"2024-08-19T14:28:36.156004+0000","last_fullsized":"2024-08-19T14:28:36.156004+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:27:32.825795+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:27:32.825795+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:49.111004+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"513'92","reported_seq":106,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-08-19T14:28:38.865256+0000","last_change":"2024-08-19T14:28:33.151083+0000","last_active":"2024-08-19T14:28:38.865256+0000","last_peered":"2024-08-19T14:28:38.865256+0000","last_clean":"2024-08-19T14:28:38.865256+0000","last_became_active":"2024-08-19T14:28:33.150788+0000","last_became_peered":"2024-08-19T14:28:33.150788+0000","last_unstale":"2024-08-19T14:28:38.865256+0000","last_undegraded":"2024-08-19T14:28:38.865256+0000","last_fullsized":"2024-08-19T14:28:38.865256+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":510,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:28:32.139424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:28:32.139424+0000","last_clean_scrub_stamp":"2024-08-19T14:28:32.139424+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:29:45.153470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6029312,"num_objects":92,"num_object_clones":0,"num_object_copies":276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":92,"num_write_kb":5888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"70'139","reported_seq":970,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-08-19T14:28:36.853624+0000","last_change":"2024-08-19T14:28:26.508582+0000","last_active":"2024-08-19T14:28:36.853624+0000","last_peered":"2024-08-19T14:28:36.853624+0000","last_clean":"2024-08-19T14:28:36.853624+0000","last_became_active":"2024-08-19T14:28:08.849293+0000","last_became_peered":"2024-08-19T14:28:08.849293+0000","last_unstale":"2024-08-19T14:28:36.853624+0000","last_undegraded":"2024-08-19T14:28:36.853624+0000","last_fullsized":"2024-08-19T14:28:36.853624+0000","mapping_epoch":496,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":497,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:26.508506+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:22:52.164586+0000","last_clean_scrub_stamp":"2024-08-19T14:28:26.508506+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:29:32.339585+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":16,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-08-19T14:28:36.853480+0000","last_change":"2024-08-19T14:28:33.148679+0000","last_active":"2024-08-19T14:28:36.853480+0000","last_peered":"2024-08-19T14:28:36.853480+0000","last_clean":"2024-08-19T14:28:36.853480+0000","last_became_active":"2024-08-19T14:28:33.148373+0000","last_became_peered":"2024-08-19T14:28:33.148373+0000","last_unstale":"2024-08-19T14:28:36.853480+0000","last_undegraded":"2024-08-19T14:28:36.853480+0000","last_fullsized":"2024-08-19T14:28:36.853480+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":510,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:28:32.139424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:28:32.139424+0000","last_clean_scrub_stamp":"2024-08-19T14:28:32.139424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:29:43.824602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"70'139","reported_seq":872,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-08-19T14:28:36.854509+0000","last_change":"2024-08-19T14:28:29.007026+0000","last_active":"2024-08-19T14:28:36.854509+0000","last_peered":"2024-08-19T14:28:36.854509+0000","last_clean":"2024-08-19T14:28:36.854509+0000","last_became_active":"2024-08-19T14:18:26.882002+0000","last_became_peered":"2024-08-19T14:18:26.882002+0000","last_unstale":"2024-08-19T14:28:36.854509+0000","last_undegraded":"2024-08-19T14:28:36.854509+0000","last_fullsized":"2024-08-19T14:28:36.854509+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:29.006921+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:24:50.276408+0000","last_clean_scrub_stamp":"2024-08-19T14:28:29.006921+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:29:46.383674+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"70'139","reported_seq":659,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-08-19T14:28:36.853592+0000","last_change":"2024-08-19T14:28:35.206191+0000","last_active":"2024-08-19T14:28:36.853592+0000","last_peered":"2024-08-19T14:28:36.853592+0000","last_clean":"2024-08-19T14:28:36.853592+0000","last_became_active":"2024-08-19T14:27:31.838968+0000","last_became_peered":"2024-08-19T14:27:31.838968+0000","last_unstale":"2024-08-19T14:28:36.853592+0000","last_undegraded":"2024-08-19T14:28:36.853592+0000","last_fullsized":"2024-08-19T14:28:36.853592+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:35.206115+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:28:35.206115+0000","last_clean_scrub_stamp":"2024-08-19T14:28:35.206115+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:29:48.838987+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.13","version":"70'139","reported_seq":1184,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-08-19T14:28:36.854153+0000","last_change":"2024-08-19T14:28:20.861936+0000","last_active":"2024-08-19T14:28:36.854153+0000","last_peered":"2024-08-19T14:28:36.854153+0000","last_clean":"2024-08-19T14:28:36.854153+0000","last_became_active":"2024-08-19T14:28:20.861463+0000","last_became_peered":"2024-08-19T14:28:20.861463+0000","last_unstale":"2024-08-19T14:28:36.854153+0000","last_undegraded":"2024-08-19T14:28:36.854153+0000","last_fullsized":"2024-08-19T14:28:36.854153+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:27:23.812426+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:27:23.812426+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:45.989931+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.10","version":"70'139","reported_seq":813,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-08-19T14:28:36.853759+0000","last_change":"2024-08-19T14:27:32.129561+0000","last_active":"2024-08-19T14:28:36.853759+0000","last_peered":"2024-08-19T14:28:36.853759+0000","last_clean":"2024-08-19T14:28:36.853759+0000","last_became_active":"2024-08-19T14:27:31.839611+0000","last_became_peered":"2024-08-19T14:27:31.839611+0000","last_unstale":"2024-08-19T14:28:36.853759+0000","last_undegraded":"2024-08-19T14:28:36.853759+0000","last_fullsized":"2024-08-19T14:28:36.853759+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:27:32.129449+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:27:32.129449+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:56.975541+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.11","version":"70'139","reported_seq":852,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-08-19T14:28:36.853983+0000","last_change":"2024-08-19T14:28:25.362916+0000","last_active":"2024-08-19T14:28:36.853983+0000","last_peered":"2024-08-19T14:28:36.853983+0000","last_clean":"2024-08-19T14:28:36.853983+0000","last_became_active":"2024-08-19T14:28:08.848199+0000","last_became_peered":"2024-08-19T14:28:08.848199+0000","last_unstale":"2024-08-19T14:28:36.853983+0000","last_undegraded":"2024-08-19T14:28:36.853983+0000","last_fullsized":"2024-08-19T14:28:36.853983+0000","mapping_epoch":496,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":497,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:25.362836+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:24:50.547428+0000","last_clean_scrub_stamp":"2024-08-19T14:28:25.362836+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:29:52.931208+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,3],"acting":[0,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"70'139","reported_seq":812,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-08-19T14:28:36.854301+0000","last_change":"2024-08-19T14:28:34.216440+0000","last_active":"2024-08-19T14:28:36.854301+0000","last_peered":"2024-08-19T14:28:36.854301+0000","last_clean":"2024-08-19T14:28:36.854301+0000","last_became_active":"2024-08-19T14:20:28.204724+0000","last_became_peered":"2024-08-19T14:20:28.204724+0000","last_unstale":"2024-08-19T14:28:36.854301+0000","last_undegraded":"2024-08-19T14:28:36.854301+0000","last_fullsized":"2024-08-19T14:28:36.854301+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:34.216370+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:28:34.216370+0000","last_clean_scrub_stamp":"2024-08-19T14:28:34.216370+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:29:59.113928+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"70'139","reported_seq":938,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-08-19T14:28:34.535917+0000","last_change":"2024-08-19T14:28:30.136536+0000","last_active":"2024-08-19T14:28:34.535917+0000","last_peered":"2024-08-19T14:28:34.535917+0000","last_clean":"2024-08-19T14:28:34.535917+0000","last_became_active":"2024-08-19T14:18:31.901556+0000","last_became_peered":"2024-08-19T14:18:31.901556+0000","last_unstale":"2024-08-19T14:28:34.535917+0000","last_undegraded":"2024-08-19T14:28:34.535917+0000","last_fullsized":"2024-08-19T14:28:34.535917+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:30.136404+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:28:30.136404+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:29:50.233865+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,0],"acting":[8,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.14","version":"70'139","reported_seq":950,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-08-19T14:28:36.854251+0000","last_change":"2024-08-19T14:28:08.847104+0000","last_active":"2024-08-19T14:28:36.854251+0000","last_peered":"2024-08-19T14:28:36.854251+0000","last_clean":"2024-08-19T14:28:36.854251+0000","last_became_active":"2024-08-19T14:28:08.846736+0000","last_became_peered":"2024-08-19T14:28:08.846736+0000","last_unstale":"2024-08-19T14:28:36.854251+0000","last_undegraded":"2024-08-19T14:28:36.854251+0000","last_fullsized":"2024-08-19T14:28:36.854251+0000","mapping_epoch":496,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":497,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:27:26.038293+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:27:26.038293+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:28:52.202656+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,8],"acting":[7,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.15","version":"70'139","reported_seq":828,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-08-19T14:28:36.854451+0000","last_change":"2024-08-19T14:28:31.999297+0000","last_active":"2024-08-19T14:28:36.854451+0000","last_peered":"2024-08-19T14:28:36.854451+0000","last_clean":"2024-08-19T14:28:36.854451+0000","last_became_active":"2024-08-19T14:18:31.912496+0000","last_became_peered":"2024-08-19T14:18:31.912496+0000","last_unstale":"2024-08-19T14:28:36.854451+0000","last_undegraded":"2024-08-19T14:28:36.854451+0000","last_fullsized":"2024-08-19T14:28:36.854451+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:31.999202+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:20:09.180923+0000","last_clean_scrub_stamp":"2024-08-19T14:28:31.999202+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:29:45.270948+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"70'139","reported_seq":814,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-08-19T14:28:36.155646+0000","last_change":"2024-08-19T14:28:30.643104+0000","last_active":"2024-08-19T14:28:36.155646+0000","last_peered":"2024-08-19T14:28:36.155646+0000","last_clean":"2024-08-19T14:28:36.155646+0000","last_became_active":"2024-08-19T14:18:31.901229+0000","last_became_peered":"2024-08-19T14:18:31.901229+0000","last_unstale":"2024-08-19T14:28:36.155646+0000","last_undegraded":"2024-08-19T14:28:36.155646+0000","last_fullsized":"2024-08-19T14:28:36.155646+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:30.643007+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:28:30.643007+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:29:45.601878+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1b","version":"70'139","reported_seq":1173,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-08-19T14:28:34.535853+0000","last_change":"2024-08-19T14:28:24.065455+0000","last_active":"2024-08-19T14:28:34.535853+0000","last_peered":"2024-08-19T14:28:34.535853+0000","last_clean":"2024-08-19T14:28:34.535853+0000","last_became_active":"2024-08-19T14:28:08.847496+0000","last_became_peered":"2024-08-19T14:28:08.847496+0000","last_unstale":"2024-08-19T14:28:34.535853+0000","last_undegraded":"2024-08-19T14:28:34.535853+0000","last_fullsized":"2024-08-19T14:28:34.535853+0000","mapping_epoch":496,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":497,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:24.065359+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:28:24.065359+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:29:38.744124+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,7],"acting":[8,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.18","version":"70'139","reported_seq":742,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-08-19T14:28:36.853698+0000","last_change":"2024-08-19T14:27:33.084267+0000","last_active":"2024-08-19T14:28:36.853698+0000","last_peered":"2024-08-19T14:28:36.853698+0000","last_clean":"2024-08-19T14:28:36.853698+0000","last_became_active":"2024-08-19T14:27:31.841421+0000","last_became_peered":"2024-08-19T14:27:31.841421+0000","last_unstale":"2024-08-19T14:28:36.853698+0000","last_undegraded":"2024-08-19T14:28:36.853698+0000","last_fullsized":"2024-08-19T14:28:36.853698+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:27:33.084161+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:27:33.084161+0000","last_clean_scrub_stamp":"2024-08-19T14:27:33.084161+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:29:01.689805+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.19","version":"70'139","reported_seq":1158,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-08-19T14:28:36.853919+0000","last_change":"2024-08-19T14:28:22.353720+0000","last_active":"2024-08-19T14:28:36.853919+0000","last_peered":"2024-08-19T14:28:36.853919+0000","last_clean":"2024-08-19T14:28:36.853919+0000","last_became_active":"2024-08-19T14:18:31.902547+0000","last_became_peered":"2024-08-19T14:18:31.902547+0000","last_unstale":"2024-08-19T14:28:36.853919+0000","last_undegraded":"2024-08-19T14:28:36.853919+0000","last_fullsized":"2024-08-19T14:28:36.853919+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:22.353632+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:28:22.353632+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:29:28.351658+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1e","version":"70'139","reported_seq":322,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-08-19T14:28:34.535754+0000","last_change":"2024-08-19T14:28:27.136259+0000","last_active":"2024-08-19T14:28:34.535754+0000","last_peered":"2024-08-19T14:28:34.535754+0000","last_clean":"2024-08-19T14:28:34.535754+0000","last_became_active":"2024-08-19T14:28:20.859275+0000","last_became_peered":"2024-08-19T14:28:20.859275+0000","last_unstale":"2024-08-19T14:28:34.535754+0000","last_undegraded":"2024-08-19T14:28:34.535754+0000","last_fullsized":"2024-08-19T14:28:34.535754+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:27.136174+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:27:04.780939+0000","last_clean_scrub_stamp":"2024-08-19T14:28:27.136174+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:29:56.964361+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,4],"acting":[8,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1c","version":"70'139","reported_seq":931,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-08-19T14:28:34.535846+0000","last_change":"2024-08-19T14:28:25.098266+0000","last_active":"2024-08-19T14:28:34.535846+0000","last_peered":"2024-08-19T14:28:34.535846+0000","last_clean":"2024-08-19T14:28:34.535846+0000","last_became_active":"2024-08-19T14:28:20.856125+0000","last_became_peered":"2024-08-19T14:28:20.856125+0000","last_unstale":"2024-08-19T14:28:34.535846+0000","last_undegraded":"2024-08-19T14:28:34.535846+0000","last_fullsized":"2024-08-19T14:28:34.535846+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:25.098173+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:28:25.098173+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:29:52.218046+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1d","version":"70'139","reported_seq":847,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-08-19T14:28:36.853957+0000","last_change":"2024-08-19T14:28:29.194721+0000","last_active":"2024-08-19T14:28:36.853957+0000","last_peered":"2024-08-19T14:28:36.853957+0000","last_clean":"2024-08-19T14:28:36.853957+0000","last_became_active":"2024-08-19T14:28:20.856669+0000","last_became_peered":"2024-08-19T14:28:20.856669+0000","last_unstale":"2024-08-19T14:28:36.853957+0000","last_undegraded":"2024-08-19T14:28:36.853957+0000","last_fullsized":"2024-08-19T14:28:36.853957+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:29.194639+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:28:29.194639+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:29:53.504962+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":54460416,"num_objects":831,"num_object_clones":0,"num_object_copies":2493,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":831,"num_write_kb":53184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":130875392,"data_stored":130875392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":831,"ondisk_log_size":831,"up":48,"acting":48,"num_store_stats":7},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2129920,"data_stored":2115712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":0,"seq":369367187663,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":343597383893,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":0,"seq":317827580121,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874462,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":564784,"kb_used_data":6048,"kb_used_omap":1,"kb_used_meta":558718,"kb_avail":93180368,"statfs":{"total":95995035648,"available":95416696832,"internally_reserved":0,"allocated":6193152,"data_stored":3879029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":572127685},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":57,"seq":244813136098,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":731596,"kb_used_data":6272,"kb_used_omap":1,"kb_used_meta":725310,"kb_avail":93013556,"statfs":{"total":95995035648,"available":95245881344,"internally_reserved":0,"allocated":6422528,"data_stored":4096997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":742717898},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":206158430437,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692073,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":637512,"kb_used_data":6580,"kb_used_omap":1,"kb_used_meta":630910,"kb_avail":93107640,"statfs":{"total":95995035648,"available":95342223360,"internally_reserved":0,"allocated":6737920,"data_stored":4416304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":646052293},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986414,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":741128,"kb_used_data":6072,"kb_used_omap":1,"kb_used_meta":735038,"kb_avail":93004024,"statfs":{"total":95995035648,"available":95236120576,"internally_reserved":0,"allocated":6217728,"data_stored":3895749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":752679365},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149940,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":655808,"kb_used_data":11000,"kb_used_omap":1,"kb_used_meta":644798,"kb_avail":93089344,"statfs":{"total":95995035648,"available":95323488256,"internally_reserved":0,"allocated":11264000,"data_stored":8942021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":660273607},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411577,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575101,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":755212,"kb_used_data":6272,"kb_used_omap":1,"kb_used_meta":748926,"kb_avail":92989940,"statfs":{"total":95995035648,"available":95221698560,"internally_reserved":0,"allocated":6422528,"data_stored":4096997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":766900677},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738625,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":647044,"kb_used_data":6392,"kb_used_omap":1,"kb_used_meta":640638,"kb_avail":93098108,"statfs":{"total":95995035648,"available":95332462592,"internally_reserved":0,"allocated":6545408,"data_stored":4223429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":656013765},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":30212096,"data_stored":30212096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":48234496,"data_stored":48234496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":52428800,"data_stored":52428800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T14:28:42.613 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd pool set .mgr pgp_num 71 2024-08-19T14:28:42.664 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:28:42.672 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:28:42.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:42.749 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:42.749 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:28:42.850 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:28:42.874 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:42.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:42.875 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:28:42.897 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:28:42.897 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:28:42.976 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:28:43.028 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:43.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:43.030 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:28:43.131 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:28:43.165 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:43.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:43.167 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:28:43.268 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:28:43.292 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:43.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:43.293 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:28:43.391 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:28:43.395 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:28:43.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:43.416 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:43.416 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:28:43.506 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 5 16 16863 16847 210.569 142.312 0.00186149 0.00457613 2024-08-19T14:28:43.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:43 smithi032 ceph-mon[20004]: pgmap v1177: 47 pgs: 47 active+clean; 75 MiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 9.2 MiB/s wr, 147 op/s 2024-08-19T14:28:43.517 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:28:43.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:43 smithi045 ceph-mon[20143]: pgmap v1177: 47 pgs: 47 active+clean; 75 MiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 9.2 MiB/s wr, 147 op/s 2024-08-19T14:28:43.554 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:43.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:43.555 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:28:43.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:43 smithi031 ceph-mon[16336]: pgmap v1177: 47 pgs: 47 active+clean; 75 MiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 9.2 MiB/s wr, 147 op/s 2024-08-19T14:28:43.655 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:28:43.681 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:43.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:43.682 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:28:43.783 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:28:43.806 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:43.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:43.807 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:28:43.813 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-08-19T14:28:43.909 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:28:43.943 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:43.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:43.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:28:44.045 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:28:44.068 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:44.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:44.069 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:28:44.170 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:28:44.213 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:44.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:44.214 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:28:44.242 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is set 2024-08-19T14:28:44.316 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:28:44.339 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:44.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:44.340 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:28:44.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:44 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3831082319' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:28:44.441 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:28:44.476 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:44.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:44.477 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:28:44.506 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 6 16 19762 19746 205.67 181.188 0.00501688 0.00485712 2024-08-19T14:28:44.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:44 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3831082319' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:28:44.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:44 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3831082319' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:28:44.577 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:28:44.598 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:28:44.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:44.603 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:44.603 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:28:44.703 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:28:44.741 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:44.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:44.743 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:28:44.771 INFO:teuthology.orchestra.run.smithi031.stderr:Error EINVAL: specified pgp_num 71 > pg_num 31 2024-08-19T14:28:44.844 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:28:44.868 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:44.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:44.869 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:28:44.970 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:28:45.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:45.004 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:45.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:28:45.106 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:28:45.127 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:45.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:45.128 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:28:45.228 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:28:45.264 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:45.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:45.265 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:28:45.366 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:28:45.390 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:45.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:45.391 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:28:45.493 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:28:45.506 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 7 15 22329 22314 199.215 160.5 0.00370415 0.00501488 2024-08-19T14:28:45.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:45 smithi032 ceph-mon[20004]: pgmap v1178: 47 pgs: 47 active+clean; 1004 MiB data, 6.9 GiB used, 619 GiB / 626 GiB avail; 102 MiB/s wr, 1.63k op/s 2024-08-19T14:28:45.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:45 smithi032 ceph-mon[20004]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:28:45.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:45 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3831082319' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:28:45.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:45 smithi032 ceph-mon[20004]: osdmap e514: 12 total, 12 up, 7 in 2024-08-19T14:28:45.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:45 smithi032 ceph-mon[20004]: from='client.16605 172.21.15.31:0/369154111' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "71"}]: dispatch 2024-08-19T14:28:45.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:45 smithi032 ceph-mon[20004]: from='client.16605 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "71"}]: dispatch 2024-08-19T14:28:45.517 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:45.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:45.518 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:28:45.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:45 smithi045 ceph-mon[20143]: pgmap v1178: 47 pgs: 47 active+clean; 1004 MiB data, 6.9 GiB used, 619 GiB / 626 GiB avail; 102 MiB/s wr, 1.63k op/s 2024-08-19T14:28:45.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:45 smithi045 ceph-mon[20143]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:28:45.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:45 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3831082319' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:28:45.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:45 smithi045 ceph-mon[20143]: osdmap e514: 12 total, 12 up, 7 in 2024-08-19T14:28:45.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:45 smithi045 ceph-mon[20143]: from='client.16605 172.21.15.31:0/369154111' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "71"}]: dispatch 2024-08-19T14:28:45.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:45 smithi045 ceph-mon[20143]: from='client.16605 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "71"}]: dispatch 2024-08-19T14:28:45.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:45 smithi031 ceph-mon[16336]: pgmap v1178: 47 pgs: 47 active+clean; 1004 MiB data, 6.9 GiB used, 619 GiB / 626 GiB avail; 102 MiB/s wr, 1.63k op/s 2024-08-19T14:28:45.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:45 smithi031 ceph-mon[16336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:28:45.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:45 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3831082319' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:28:45.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:45 smithi031 ceph-mon[16336]: osdmap e514: 12 total, 12 up, 7 in 2024-08-19T14:28:45.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:45 smithi031 ceph-mon[16336]: from='client.16605 172.21.15.31:0/369154111' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "71"}]: dispatch 2024-08-19T14:28:45.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:45 smithi031 ceph-mon[16336]: from='client.16605 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "71"}]: dispatch 2024-08-19T14:28:45.619 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:28:45.643 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:45.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:45.644 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:28:45.744 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:28:45.779 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:45.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:45.780 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:28:45.881 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:28:45.902 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:45.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:45.903 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:28:46.003 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:28:46.026 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:46.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:46.028 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:28:46.128 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:28:46.156 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:46.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:46.157 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:28:46.258 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:28:46.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:46.293 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:46.293 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:28:46.394 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:28:46.427 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:46.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:46.428 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:28:46.507 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 8 16 25217 25201 196.866 180.438 0.002584 0.00507451 2024-08-19T14:28:46.529 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:28:46.552 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:46.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:46.553 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:28:46.594 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:28:46.636 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-19T14:28:46.654 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:28:46.687 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:46.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:46.688 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:28:46.788 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:28:46.813 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:46.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:46.814 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:28:46.915 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:28:46.950 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:46.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:46.951 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:28:47.052 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:28:47.087 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:47.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:47.087 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:28:47.162 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-19T14:28:47.188 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:28:47.211 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:47.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:47.212 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:28:47.313 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:28:47.363 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:47.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:47.364 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:28:47.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:47 smithi031 ceph-mon[16336]: pgmap v1180: 47 pgs: 47 active+clean; 1.0 GiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 108 MiB/s wr, 1.73k op/s 2024-08-19T14:28:47.430 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:28:47.431 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:28:47.465 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:28:47.470 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:28:47.471 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:28:47.500 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:47.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:47.501 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:28:47.504 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:28:47.504 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:28:47.507 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 9 15 27838 27823 193.199 163.875 0.0136256 0.00517094 2024-08-19T14:28:47.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:47 smithi032 ceph-mon[20004]: pgmap v1180: 47 pgs: 47 active+clean; 1.0 GiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 108 MiB/s wr, 1.73k op/s 2024-08-19T14:28:47.539 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:28:47.540 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:28:47.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:47 smithi045 ceph-mon[20143]: pgmap v1180: 47 pgs: 47 active+clean; 1.0 GiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 108 MiB/s wr, 1.73k op/s 2024-08-19T14:28:47.575 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:28:47.575 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:28:47.602 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:28:47.611 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:28:47.611 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:28:47.642 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:28:47.642 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:28:47.651 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:47.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:47.652 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:28:47.657 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:28:47.657 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-08-19T14:28:47.657 INFO:teuthology.orchestra.run.smithi031.stdout: "complaint_time": 30, 2024-08-19T14:28:47.657 INFO:teuthology.orchestra.run.smithi031.stdout: "num_blocked_ops": 0 2024-08-19T14:28:47.658 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:28:47.659 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:28:47.701 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:28:47.701 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:28:47.733 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:28:47.733 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:28:47.753 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:28:47.767 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:28:47.768 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:28:47.791 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:47.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:47.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:28:47.802 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:28:47.802 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:28:47.835 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:28:47.835 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:28:47.893 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:28:47.929 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:47.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:47.930 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:28:48.031 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:28:48.064 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:48.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:48.065 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:28:48.165 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:28:48.199 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:48.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:48.200 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:28:48.301 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:28:48.326 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:48.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:48.327 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:28:48.428 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:28:48.461 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:48.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:48.462 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:28:48.507 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 10 16 30448 30432 190.184 163.062 0.0047514 0.00525428 2024-08-19T14:28:48.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:48 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3641941922' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:28:48.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:48 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3641941922' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:28:48.563 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:28:48.573 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:48 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3641941922' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:28:48.605 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:48.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:48.606 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:28:48.626 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-08-19T14:28:48.707 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:28:48.750 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:48.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:48.751 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:28:48.852 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:28:48.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:48.891 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:48.891 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:28:48.992 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:28:49.025 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:49.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:49.026 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:28:49.127 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:28:49.150 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:49.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:49.151 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:28:49.252 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:28:49.273 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:49.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:49.274 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:28:49.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:49 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:28:49.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:49 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:28:49.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:49 smithi031 ceph-mon[16336]: pgmap v1181: 47 pgs: 47 active+clean; 1.2 GiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 122 MiB/s wr, 1.95k op/s 2024-08-19T14:28:49.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:49 smithi031 ceph-mon[16336]: osdmap e515: 12 total, 12 up, 7 in 2024-08-19T14:28:49.375 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:28:49.396 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:49.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:49.397 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:28:49.498 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:28:49.507 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 11 16 32513 32497 184.627 129.062 0.00595511 0.00541205 2024-08-19T14:28:49.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:49 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:28:49.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:49 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:28:49.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:49 smithi032 ceph-mon[20004]: pgmap v1181: 47 pgs: 47 active+clean; 1.2 GiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 122 MiB/s wr, 1.95k op/s 2024-08-19T14:28:49.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:49 smithi032 ceph-mon[20004]: osdmap e515: 12 total, 12 up, 7 in 2024-08-19T14:28:49.522 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:49.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:49.523 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:28:49.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:49 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:28:49.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:49 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:28:49.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:49 smithi045 ceph-mon[20143]: pgmap v1181: 47 pgs: 47 active+clean; 1.2 GiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 122 MiB/s wr, 1.95k op/s 2024-08-19T14:28:49.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:49 smithi045 ceph-mon[20143]: osdmap e515: 12 total, 12 up, 7 in 2024-08-19T14:28:49.624 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:28:49.624 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:28:49.677 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:49.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:49.678 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:28:49.779 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:28:49.802 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:49.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:49.803 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:28:49.904 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:28:49.935 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:49.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:49.936 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:28:50.037 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:28:50.075 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:50.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:50.076 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:28:50.177 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:28:50.201 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:50.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:50.203 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:28:50.235 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:28:50.235 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 20, 2024-08-19T14:28:50.235 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 600, 2024-08-19T14:28:50.235 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [] 2024-08-19T14:28:50.235 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:28:50.258 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is unset 2024-08-19T14:28:50.303 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:28:50.326 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:50.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:50.327 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:28:50.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:50 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/160548889' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:28:50.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:50 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:28:50.428 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:28:50.451 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:50.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:50.452 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:28:50.507 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 12 16 34937 34921 181.865 151.5 0.00993483 0.00549437 2024-08-19T14:28:50.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:50 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/160548889' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:28:50.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:50 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:28:50.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:50 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/160548889' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:28:50.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:50 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:28:50.553 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:28:50.589 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:50.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:50.591 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:28:50.692 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:28:50.715 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:50.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:50.716 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:28:50.817 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:28:50.841 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:50.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:50.842 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:28:50.943 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:28:50.968 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:50.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:50.969 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:28:51.070 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:28:51.077 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:28:51.101 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:51.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:51.103 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:28:51.204 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:28:51.233 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:51.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:51.234 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:28:51.335 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:28:51.373 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:51.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:51.374 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:28:51.475 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:28:51.507 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 13 16 37317 37301 179.317 148.75 0.00718533 0.0055724 2024-08-19T14:28:51.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:51.510 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:51.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:28:51.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:51 smithi045 ceph-mon[20143]: pgmap v1183: 47 pgs: 47 active+clean; 1.8 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 222 MiB/s wr, 3.55k op/s 2024-08-19T14:28:51.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:51 smithi045 ceph-mon[20143]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:28:51.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:51 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:28:51.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:51 smithi045 ceph-mon[20143]: osdmap e516: 12 total, 12 up, 7 in 2024-08-19T14:28:51.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:51 smithi045 ceph-mon[20143]: 1.a scrub starts 2024-08-19T14:28:51.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:51 smithi045 ceph-mon[20143]: 1.a scrub ok 2024-08-19T14:28:51.611 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:28:51.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:51 smithi031 ceph-mon[16336]: pgmap v1183: 47 pgs: 47 active+clean; 1.8 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 222 MiB/s wr, 3.55k op/s 2024-08-19T14:28:51.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:51 smithi031 ceph-mon[16336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:28:51.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:51 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:28:51.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:51 smithi031 ceph-mon[16336]: osdmap e516: 12 total, 12 up, 7 in 2024-08-19T14:28:51.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:51 smithi031 ceph-mon[16336]: 1.a scrub starts 2024-08-19T14:28:51.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:51 smithi031 ceph-mon[16336]: 1.a scrub ok 2024-08-19T14:28:51.634 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:51.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:51.635 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:28:51.638 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 7, 8] out_osds: [10, 6, 2, 9, 11] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-08-19T14:28:51.638 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-19T14:28:51.638 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-08-19T14:28:51.638 INFO:tasks.thrashosds.thrasher:Setting osd 0 primary_affinity to 1.000000 2024-08-19T14:28:51.638 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd primary-affinity 0 1 2024-08-19T14:28:51.736 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:28:51.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:51 smithi032 ceph-mon[20004]: pgmap v1183: 47 pgs: 47 active+clean; 1.8 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 222 MiB/s wr, 3.55k op/s 2024-08-19T14:28:51.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:51 smithi032 ceph-mon[20004]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:28:51.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:51 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:28:51.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:51 smithi032 ceph-mon[20004]: osdmap e516: 12 total, 12 up, 7 in 2024-08-19T14:28:51.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:51 smithi032 ceph-mon[20004]: 1.a scrub starts 2024-08-19T14:28:51.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:51 smithi032 ceph-mon[20004]: 1.a scrub ok 2024-08-19T14:28:51.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:51.759 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:51.759 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:28:51.860 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:28:51.885 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:51.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:51.886 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:28:51.986 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:28:52.027 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:52.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:52.028 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:28:52.129 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:28:52.150 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:52.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:52.151 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:28:52.253 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:28:52.290 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:52.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:52.291 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:28:52.392 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:28:52.416 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:52.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:52.418 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:28:52.507 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 14 16 39046 39030 174.227 108.062 0.00343564 0.00573538 2024-08-19T14:28:52.519 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:28:52.540 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:52.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:52.541 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:28:52.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:52 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:28:52.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:52 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:28:52.642 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:28:52.678 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:52.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:52.680 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:28:52.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:52 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:28:52.780 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:28:52.818 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:52.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:52.819 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:28:52.920 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:28:52.941 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:52.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:52.942 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:28:53.043 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:28:53.067 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:53.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:53.069 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:28:53.170 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:28:53.193 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:53.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:53.194 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:28:53.295 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:28:53.332 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:53.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:53.333 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:28:53.434 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:28:53.457 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:53.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:53.458 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:28:53.507 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 15 16 41046 41030 170.945 125 0.00641658 0.00584546 2024-08-19T14:28:53.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:53 smithi045 ceph-mon[20143]: pgmap v1185: 47 pgs: 47 active+clean; 1.9 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 115 MiB/s wr, 1.84k op/s 2024-08-19T14:28:53.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:53 smithi045 ceph-mon[20143]: 1.f scrub starts 2024-08-19T14:28:53.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:53 smithi045 ceph-mon[20143]: 1.f scrub ok 2024-08-19T14:28:53.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:53 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:28:53.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:53 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:28:53.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:53 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:28:53.559 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:28:53.582 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:53.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:53.584 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:28:53.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:53 smithi031 ceph-mon[16336]: pgmap v1185: 47 pgs: 47 active+clean; 1.9 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 115 MiB/s wr, 1.84k op/s 2024-08-19T14:28:53.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:53 smithi031 ceph-mon[16336]: 1.f scrub starts 2024-08-19T14:28:53.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:53 smithi031 ceph-mon[16336]: 1.f scrub ok 2024-08-19T14:28:53.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:53 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:28:53.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:53 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:28:53.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:53 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:28:53.684 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:28:53.716 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:53.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:53.717 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:28:53.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:53 smithi032 ceph-mon[20004]: pgmap v1185: 47 pgs: 47 active+clean; 1.9 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 115 MiB/s wr, 1.84k op/s 2024-08-19T14:28:53.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:53 smithi032 ceph-mon[20004]: 1.f scrub starts 2024-08-19T14:28:53.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:53 smithi032 ceph-mon[20004]: 1.f scrub ok 2024-08-19T14:28:53.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:53 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:28:53.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:53 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:28:53.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:53 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:28:53.818 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:28:53.839 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:53.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:53.840 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:28:53.941 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:28:53.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:53.977 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:53.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:28:54.078 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:28:54.111 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:54.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:54.112 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:28:54.213 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:28:54.254 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:54.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:54.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:28:54.356 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:28:54.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:54 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:28:54.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:54 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:28:54.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:54 smithi031 ceph-mon[16336]: 1.13 scrub starts 2024-08-19T14:28:54.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:54 smithi031 ceph-mon[16336]: 1.13 scrub ok 2024-08-19T14:28:54.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:54 smithi031 ceph-mon[16336]: 1.9 scrub starts 2024-08-19T14:28:54.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:54 smithi031 ceph-mon[16336]: 1.9 scrub ok 2024-08-19T14:28:54.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:54 smithi031 ceph-mon[16336]: osdmap e517: 12 total, 12 up, 7 in 2024-08-19T14:28:54.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:54 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:28:54.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:54 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:28:54.377 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:54.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:54.378 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:28:54.479 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:28:54.501 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:54.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:54.502 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:28:54.507 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 16 16 42630 42614 166.448 99 0.199973 0.00597263 2024-08-19T14:28:54.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:54 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:28:54.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:54 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:28:54.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:54 smithi045 ceph-mon[20143]: 1.13 scrub starts 2024-08-19T14:28:54.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:54 smithi045 ceph-mon[20143]: 1.13 scrub ok 2024-08-19T14:28:54.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:54 smithi045 ceph-mon[20143]: 1.9 scrub starts 2024-08-19T14:28:54.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:54 smithi045 ceph-mon[20143]: 1.9 scrub ok 2024-08-19T14:28:54.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:54 smithi045 ceph-mon[20143]: osdmap e517: 12 total, 12 up, 7 in 2024-08-19T14:28:54.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:54 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:28:54.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:54 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:28:54.602 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:28:54.637 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:54.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:54.638 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:28:54.739 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:28:54.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:54 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:28:54.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:54 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:28:54.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:54 smithi032 ceph-mon[20004]: 1.13 scrub starts 2024-08-19T14:28:54.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:54 smithi032 ceph-mon[20004]: 1.13 scrub ok 2024-08-19T14:28:54.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:54 smithi032 ceph-mon[20004]: 1.9 scrub starts 2024-08-19T14:28:54.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:54 smithi032 ceph-mon[20004]: 1.9 scrub ok 2024-08-19T14:28:54.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:54 smithi032 ceph-mon[20004]: osdmap e517: 12 total, 12 up, 7 in 2024-08-19T14:28:54.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:54 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:28:54.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:54 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:28:54.775 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:54.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:54.776 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:28:54.877 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:28:54.913 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:54.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:54.915 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:28:55.015 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:28:55.046 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:55.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:55.047 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:28:55.148 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:28:55.170 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:55.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:55.171 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:28:55.272 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:28:55.295 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:55.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:55.297 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:28:55.398 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:28:55.418 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:55.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:55.419 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:28:55.507 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 17 16 44537 44521 163.667 119.188 0.0088831 0.00610577 2024-08-19T14:28:55.520 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:28:55.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:55.558 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:55.558 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:28:55.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:55 smithi031 ceph-mon[16336]: pgmap v1187: 47 pgs: 47 active+clean; 2.5 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 217 MiB/s wr, 3.47k op/s 2024-08-19T14:28:55.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:55 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:28:55.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:55 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:28:55.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:55 smithi031 ceph-mon[16336]: osdmap e518: 12 total, 12 up, 7 in 2024-08-19T14:28:55.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:55 smithi031 ceph-mon[16336]: osdmap e519: 12 total, 12 up, 7 in 2024-08-19T14:28:55.659 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:28:55.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:55.692 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:55.692 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:28:55.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:55 smithi032 ceph-mon[20004]: pgmap v1187: 47 pgs: 47 active+clean; 2.5 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 217 MiB/s wr, 3.47k op/s 2024-08-19T14:28:55.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:55 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:28:55.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:55 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:28:55.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:55 smithi032 ceph-mon[20004]: osdmap e518: 12 total, 12 up, 7 in 2024-08-19T14:28:55.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:55 smithi032 ceph-mon[20004]: osdmap e519: 12 total, 12 up, 7 in 2024-08-19T14:28:55.793 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:28:55.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:55 smithi045 ceph-mon[20143]: pgmap v1187: 47 pgs: 47 active+clean; 2.5 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 217 MiB/s wr, 3.47k op/s 2024-08-19T14:28:55.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:55 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:28:55.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:55 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:28:55.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:55 smithi045 ceph-mon[20143]: osdmap e518: 12 total, 12 up, 7 in 2024-08-19T14:28:55.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:55 smithi045 ceph-mon[20143]: osdmap e519: 12 total, 12 up, 7 in 2024-08-19T14:28:55.830 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:55.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:55.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:28:55.932 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:28:55.953 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:55.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:55.954 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:28:56.055 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:28:56.090 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:56.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:56.092 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:28:56.193 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:28:56.236 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:56.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:56.237 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:28:56.337 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:28:56.371 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:56.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:56.373 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:28:56.474 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:28:56.507 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 18 15 46525 46510 161.48 124.312 0.00581175 0.00618844 2024-08-19T14:28:56.510 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:56.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:56.511 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:28:56.613 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:28:56.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:56 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:28:56.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:56 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:28:56.635 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:56.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:56.636 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:28:56.737 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:28:56.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:56 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:28:56.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:56 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:28:56.769 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:56.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:56.770 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:28:56.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:56 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:28:56.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:56 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:28:56.870 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:28:56.892 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:56.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:56.893 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:28:56.994 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:28:57.016 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:57.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:57.018 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:28:57.119 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:28:57.151 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:57.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:57.152 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:28:57.253 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:28:57.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:57.290 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:57.291 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:28:57.392 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:28:57.426 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:57.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:57.427 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:28:57.508 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 19 16 48468 48452 159.369 121.375 0.00380895 0.00626779 2024-08-19T14:28:57.527 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:28:57.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:57.549 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:57.550 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:28:57.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:57 smithi031 ceph-mon[16336]: pgmap v1190: 47 pgs: 47 active+clean; 2.5 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 115 MiB/s wr, 1.84k op/s 2024-08-19T14:28:57.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:57 smithi031 ceph-mon[16336]: osdmap e520: 12 total, 12 up, 7 in 2024-08-19T14:28:57.651 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:28:57.672 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:57.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:57.673 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:28:57.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:57 smithi032 ceph-mon[20004]: pgmap v1190: 47 pgs: 47 active+clean; 2.5 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 115 MiB/s wr, 1.84k op/s 2024-08-19T14:28:57.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:57 smithi032 ceph-mon[20004]: osdmap e520: 12 total, 12 up, 7 in 2024-08-19T14:28:57.775 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:28:57.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:57 smithi045 ceph-mon[20143]: pgmap v1190: 47 pgs: 47 active+clean; 2.5 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 115 MiB/s wr, 1.84k op/s 2024-08-19T14:28:57.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:57 smithi045 ceph-mon[20143]: osdmap e520: 12 total, 12 up, 7 in 2024-08-19T14:28:57.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:57.814 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:57.814 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:28:57.915 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:28:57.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:57.938 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:57.939 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:28:58.040 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:28:58.069 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:58.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:58.070 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:28:58.171 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:28:58.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:58.259 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:58.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:28:58.360 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:28:58.382 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:58.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:58.383 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:28:58.484 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:28:58.508 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:2024-08-19T14:28:58.508415+0000 min lat: 0.00119513 max lat: 0.230278 avg lat: 0.00639149 2024-08-19T14:28:58.508 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-19T14:28:58.508 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 20 16 50057 50041 156.365 99.3125 0.0086658 0.00639149 2024-08-19T14:28:58.518 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:58.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:58.519 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:28:58.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:58 smithi031 ceph-mon[16336]: 1.0 scrub starts 2024-08-19T14:28:58.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:58 smithi031 ceph-mon[16336]: 1.0 scrub ok 2024-08-19T14:28:58.620 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:28:58.655 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:58.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:58.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:28:58.757 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:28:58.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:58 smithi032 ceph-mon[20004]: 1.0 scrub starts 2024-08-19T14:28:58.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:58 smithi032 ceph-mon[20004]: 1.0 scrub ok 2024-08-19T14:28:58.786 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:58.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:58.787 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:28:58.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:58 smithi045 ceph-mon[20143]: 1.0 scrub starts 2024-08-19T14:28:58.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:58 smithi045 ceph-mon[20143]: 1.0 scrub ok 2024-08-19T14:28:58.888 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:28:58.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:58.923 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:58.923 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:28:59.024 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:28:59.045 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:59.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:59.046 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:28:59.147 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:28:59.181 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:59.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:59.182 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:28:59.283 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:28:59.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:59.327 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:28:59.329 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:28:59.431 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:28:59.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:59.464 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:59.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:28:59.508 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 21 16 51747 51731 153.949 105.625 0.00793083 0.0064918 2024-08-19T14:28:59.565 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:28:59.586 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:59.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:59.587 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:28:59.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:59 smithi031 ceph-mon[16336]: pgmap v1192: 47 pgs: 47 active+clean; 2.6 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 109 MiB/s wr, 1.74k op/s 2024-08-19T14:28:59.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:59 smithi031 ceph-mon[16336]: 1.14 scrub starts 2024-08-19T14:28:59.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:59 smithi031 ceph-mon[16336]: 1.14 scrub ok 2024-08-19T14:28:59.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:59 smithi031 ceph-mon[16336]: 1.6 scrub starts 2024-08-19T14:28:59.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:28:59 smithi031 ceph-mon[16336]: 1.6 scrub ok 2024-08-19T14:28:59.688 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:28:59.709 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:59.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:59.710 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:28:59.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:59 smithi032 ceph-mon[20004]: pgmap v1192: 47 pgs: 47 active+clean; 2.6 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 109 MiB/s wr, 1.74k op/s 2024-08-19T14:28:59.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:59 smithi032 ceph-mon[20004]: 1.14 scrub starts 2024-08-19T14:28:59.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:59 smithi032 ceph-mon[20004]: 1.14 scrub ok 2024-08-19T14:28:59.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:59 smithi032 ceph-mon[20004]: 1.6 scrub starts 2024-08-19T14:28:59.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:28:59 smithi032 ceph-mon[20004]: 1.6 scrub ok 2024-08-19T14:28:59.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:59 smithi045 ceph-mon[20143]: pgmap v1192: 47 pgs: 47 active+clean; 2.6 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 109 MiB/s wr, 1.74k op/s 2024-08-19T14:28:59.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:59 smithi045 ceph-mon[20143]: 1.14 scrub starts 2024-08-19T14:28:59.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:59 smithi045 ceph-mon[20143]: 1.14 scrub ok 2024-08-19T14:28:59.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:59 smithi045 ceph-mon[20143]: 1.6 scrub starts 2024-08-19T14:28:59.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:28:59 smithi045 ceph-mon[20143]: 1.6 scrub ok 2024-08-19T14:28:59.811 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:28:59.834 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:28:59.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:59.835 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:28:59.937 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:28:59.973 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:28:59.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:28:59.974 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:29:00.076 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:29:00.100 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:00.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:00.101 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:29:00.202 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:29:00.224 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:00.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:00.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:29:00.326 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:29:00.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:00.363 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:00.363 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:29:00.464 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:29:00.493 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:00.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:00.494 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:29:00.508 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 22 15 53199 53184 151.078 90.8125 0.03772 0.00661021 2024-08-19T14:29:00.595 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:29:00.628 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:00.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:00.629 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:29:00.730 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:29:00.766 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:00.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:00.767 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:29:00.868 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:29:00.908 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:00.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:00.909 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:29:01.008 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-08-19T14:29:01.009 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:29:01.022 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:29:01.040 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:29:01.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:01.077 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:01.078 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:29:01.179 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:29:01.214 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:01.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:01.215 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:29:01.317 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:29:01.325 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-19T14:29:01.339 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:01.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:01.340 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:29:01.441 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:29:01.485 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:01.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:01.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:29:01.508 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 23 16 55058 55042 149.558 116.125 0.00653379 0.00668246 2024-08-19T14:29:01.587 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:29:01.609 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:01.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:01.610 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:29:01.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:01 smithi031 ceph-mon[16336]: 1.8 scrub starts 2024-08-19T14:29:01.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:01 smithi031 ceph-mon[16336]: 1.8 scrub ok 2024-08-19T14:29:01.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:01 smithi031 ceph-mon[16336]: pgmap v1193: 47 pgs: 47 active+clean; 3.0 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-08-19T14:29:01.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:01 smithi031 ceph-mon[16336]: 1.10 scrub starts 2024-08-19T14:29:01.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:01 smithi031 ceph-mon[16336]: 1.10 scrub ok 2024-08-19T14:29:01.711 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:29:01.747 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:01.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:01.748 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:29:01.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:01 smithi032 ceph-mon[20004]: 1.8 scrub starts 2024-08-19T14:29:01.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:01 smithi032 ceph-mon[20004]: 1.8 scrub ok 2024-08-19T14:29:01.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:01 smithi032 ceph-mon[20004]: pgmap v1193: 47 pgs: 47 active+clean; 3.0 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-08-19T14:29:01.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:01 smithi032 ceph-mon[20004]: 1.10 scrub starts 2024-08-19T14:29:01.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:01 smithi032 ceph-mon[20004]: 1.10 scrub ok 2024-08-19T14:29:01.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:01 smithi045 ceph-mon[20143]: 1.8 scrub starts 2024-08-19T14:29:01.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:01 smithi045 ceph-mon[20143]: 1.8 scrub ok 2024-08-19T14:29:01.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:01 smithi045 ceph-mon[20143]: pgmap v1193: 47 pgs: 47 active+clean; 3.0 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-08-19T14:29:01.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:01 smithi045 ceph-mon[20143]: 1.10 scrub starts 2024-08-19T14:29:01.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:01 smithi045 ceph-mon[20143]: 1.10 scrub ok 2024-08-19T14:29:01.849 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:29:01.887 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:01.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:01.888 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:29:01.989 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:29:02.012 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:02.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:02.013 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:29:02.114 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:29:02.138 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:02.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:02.139 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:29:02.240 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:29:02.261 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:02.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:02.262 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:29:02.363 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:29:02.403 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:02.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:02.404 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:29:02.505 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:29:02.508 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 24 16 56956 56940 148.269 118.625 0.00715124 0.0067396 2024-08-19T14:29:02.532 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:02.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:02.533 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:29:02.598 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:29:02.610 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:29:02.610 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:29:02.634 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:29:02.642 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:29:02.646 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:29:02.646 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:29:02.654 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:02.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:02.655 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:29:02.673 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:29:02.674 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:29:02.692 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:29:02.692 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:29:02.756 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:29:02.778 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:02.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:02.779 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:29:02.880 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:29:02.902 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:02.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:02.903 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:29:02.932 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:29:02.932 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:29:02.965 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:29:02.965 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:29:02.997 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:29:02.997 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:29:03.004 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:29:03.033 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:29:03.033 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:29:03.038 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:03.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:03.039 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:29:03.066 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:29:03.067 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:29:03.095 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:29:03.095 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:29:03.124 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:29:03.125 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:29:03.140 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:29:03.160 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:29:03.160 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:29:03.186 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:03.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:03.187 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:29:03.288 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:29:03.319 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:03.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:03.321 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:29:03.329 INFO:teuthology.orchestra.run.smithi031.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-08-19T14:29:03.422 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:29:03.458 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:03.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:03.459 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:29:03.483 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:29:03.483 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:29:03.483 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:03 smithi031 ceph-mon[16336]: pgmap v1194: 47 pgs: 47 active+clean; 3.1 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-08-19T14:29:03.483 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:03 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1845375553' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:29:03.483 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:03 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1188172926' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-08-19T14:29:03.483 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:03 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-08-19T14:29:03.508 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 25 16 58500 58484 146.198 96.5 0.011348 0.00683594 2024-08-19T14:29:03.560 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:29:03.582 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:03.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:03.583 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:29:03.684 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:29:03.706 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:03.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:03.708 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:29:03.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:03 smithi032 ceph-mon[20004]: pgmap v1194: 47 pgs: 47 active+clean; 3.1 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-08-19T14:29:03.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:03 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1845375553' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:29:03.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:03 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1188172926' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-08-19T14:29:03.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:03 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-08-19T14:29:03.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:03 smithi045 ceph-mon[20143]: pgmap v1194: 47 pgs: 47 active+clean; 3.1 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-08-19T14:29:03.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:03 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1845375553' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:29:03.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:03 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1188172926' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-08-19T14:29:03.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:03 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-08-19T14:29:03.809 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:29:03.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:03.847 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:03.847 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:29:03.948 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:29:03.969 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:03.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:03.970 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:29:04.071 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:29:04.108 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:04.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:04.109 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:29:04.210 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:29:04.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:04.233 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:04.233 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:29:04.334 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:29:04.347 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is unset 2024-08-19T14:29:04.365 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:04.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:04.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:29:04.468 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:29:04.500 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:04.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:04.501 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:29:04.508 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 26 16 60351 60335 145.024 115.688 0.00532088 0.00689128 2024-08-19T14:29:04.600 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:04 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-08-19T14:29:04.600 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:04 smithi031 ceph-mon[16336]: osdmap e521: 12 total, 12 up, 7 in 2024-08-19T14:29:04.600 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:04 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1286847910' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:29:04.600 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:04 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:29:04.602 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:29:04.625 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:04.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:04.627 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:29:04.727 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:29:04.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:04 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-08-19T14:29:04.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:04 smithi032 ceph-mon[20004]: osdmap e521: 12 total, 12 up, 7 in 2024-08-19T14:29:04.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:04 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1286847910' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:29:04.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:04 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:29:04.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:04.760 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:04.760 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:29:04.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:04 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-08-19T14:29:04.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:04 smithi045 ceph-mon[20143]: osdmap e521: 12 total, 12 up, 7 in 2024-08-19T14:29:04.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:04 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1286847910' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:29:04.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:04 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:29:04.861 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:29:04.894 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:04.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:04.895 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:29:04.996 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:29:05.019 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:05.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:05.021 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:29:05.122 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:29:05.161 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:05.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:05.162 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:29:05.263 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:29:05.286 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:05.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:05.287 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:29:05.388 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:29:05.424 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:05.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:05.425 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:29:05.508 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 27 16 62245 62229 144.037 118.375 0.015019 0.00693865 2024-08-19T14:29:05.526 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:29:05.560 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:05.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:05.561 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:29:05.609 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:05 smithi031 ceph-mon[16336]: pgmap v1196: 47 pgs: 47 active+clean; 3.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 133 MiB/s wr, 2.13k op/s 2024-08-19T14:29:05.609 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:05 smithi031 ceph-mon[16336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:29:05.609 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:05 smithi031 ceph-mon[16336]: Cluster is now healthy 2024-08-19T14:29:05.610 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:05 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:29:05.610 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:05 smithi031 ceph-mon[16336]: osdmap e522: 12 total, 12 up, 7 in 2024-08-19T14:29:05.610 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:05 smithi031 ceph-mon[16336]: 1.18 scrub starts 2024-08-19T14:29:05.610 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:05 smithi031 ceph-mon[16336]: 1.18 scrub ok 2024-08-19T14:29:05.662 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:29:05.686 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:05.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:05.688 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:29:05.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:05 smithi032 ceph-mon[20004]: pgmap v1196: 47 pgs: 47 active+clean; 3.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 133 MiB/s wr, 2.13k op/s 2024-08-19T14:29:05.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:05 smithi032 ceph-mon[20004]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:29:05.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:05 smithi032 ceph-mon[20004]: Cluster is now healthy 2024-08-19T14:29:05.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:05 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:29:05.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:05 smithi032 ceph-mon[20004]: osdmap e522: 12 total, 12 up, 7 in 2024-08-19T14:29:05.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:05 smithi032 ceph-mon[20004]: 1.18 scrub starts 2024-08-19T14:29:05.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:05 smithi032 ceph-mon[20004]: 1.18 scrub ok 2024-08-19T14:29:05.788 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:29:05.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:05 smithi045 ceph-mon[20143]: pgmap v1196: 47 pgs: 47 active+clean; 3.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 133 MiB/s wr, 2.13k op/s 2024-08-19T14:29:05.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:05 smithi045 ceph-mon[20143]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:29:05.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:05 smithi045 ceph-mon[20143]: Cluster is now healthy 2024-08-19T14:29:05.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:05 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:29:05.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:05 smithi045 ceph-mon[20143]: osdmap e522: 12 total, 12 up, 7 in 2024-08-19T14:29:05.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:05 smithi045 ceph-mon[20143]: 1.18 scrub starts 2024-08-19T14:29:05.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:05 smithi045 ceph-mon[20143]: 1.18 scrub ok 2024-08-19T14:29:05.821 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:05.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:05.822 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:29:05.923 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:29:05.944 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:05.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:05.946 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:29:06.047 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:29:06.077 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:06.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:06.078 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:29:06.179 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:29:06.216 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:06.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:06.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:29:06.318 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:29:06.353 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:06.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:06.354 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:29:06.455 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:29:06.490 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-08-19T14:29:06.504 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:06.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:06.505 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:29:06.508 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 28 16 63774 63758 142.305 95.5625 0.00616508 0.00702284 2024-08-19T14:29:06.606 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:29:06.644 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:06.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:06.645 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:29:06.647 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:29:06.746 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:29:06.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:06.787 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:06.788 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:29:06.889 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:29:06.910 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:06.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:06.911 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:29:06.961 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:29:07.013 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:29:07.052 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:07.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:07.053 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:29:07.154 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:29:07.190 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:07.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:07.191 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:29:07.292 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:29:07.309 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-19T14:29:07.315 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:07.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:07.316 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:29:07.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:07 smithi031 ceph-mon[16336]: pgmap v1198: 47 pgs: 47 active+clean; 3.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 124 MiB/s wr, 1.98k op/s 2024-08-19T14:29:07.416 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:29:07.453 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:07.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:07.454 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:29:07.510 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 29 16 65353 65337 140.801 98.6875 0.00573647 0.00709751 2024-08-19T14:29:07.554 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:29:07.581 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:07.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:07.582 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:29:07.683 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:29:07.706 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:07.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:07.707 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:29:07.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:07 smithi032 ceph-mon[20004]: pgmap v1198: 47 pgs: 47 active+clean; 3.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 124 MiB/s wr, 1.98k op/s 2024-08-19T14:29:07.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:07 smithi045 ceph-mon[20143]: pgmap v1198: 47 pgs: 47 active+clean; 3.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 124 MiB/s wr, 1.98k op/s 2024-08-19T14:29:07.807 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:29:07.829 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:07.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:07.831 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:29:07.932 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:29:07.953 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:07.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:07.954 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:29:08.055 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:29:08.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:08.078 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:08.078 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:29:08.179 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:29:08.213 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:08.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:08.215 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:29:08.316 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:29:08.338 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:08.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:08.339 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:29:08.440 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:29:08.463 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:08.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:08.464 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:29:08.509 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 30 16 66928 66912 139.389 98.4375 0.0112949 0.00717078 2024-08-19T14:29:08.565 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:29:08.587 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:08.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:08.588 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:29:08.689 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:29:08.729 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:08.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:08.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:29:08.831 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:29:08.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:08.871 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:08.871 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:29:08.972 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:29:08.995 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:08.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:08.996 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:29:09.097 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:29:09.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:09.131 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:09.131 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:29:09.232 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:29:09.267 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:09.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:09.268 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:29:09.369 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:29:09.390 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:09.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:09.391 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:29:09.492 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:29:09.509 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 31 16 68164 68148 137.384 77.25 0.00988419 0.00726514 2024-08-19T14:29:09.514 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:09.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:09.515 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:29:09.616 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:29:09.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:09 smithi031 ceph-mon[16336]: pgmap v1199: 47 pgs: 47 active+clean; 3.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 84 MiB/s wr, 1.34k op/s 2024-08-19T14:29:09.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:09.647 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:09.647 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:29:09.748 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:29:09.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:09 smithi032 ceph-mon[20004]: pgmap v1199: 47 pgs: 47 active+clean; 3.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 84 MiB/s wr, 1.34k op/s 2024-08-19T14:29:09.770 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:09.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:09.774 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:29:09.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:09 smithi045 ceph-mon[20143]: pgmap v1199: 47 pgs: 47 active+clean; 3.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 84 MiB/s wr, 1.34k op/s 2024-08-19T14:29:09.875 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:29:09.898 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:09.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:09.899 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:29:09.999 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:29:10.021 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:10.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:10.022 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:29:10.123 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:29:10.151 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:10.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:10.152 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:29:10.172 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:29:10.172 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:29:10.187 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:29:10.252 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:29:10.292 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:10.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:10.294 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:29:10.395 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:29:10.427 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:10.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:10.428 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:29:10.509 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 32 16 69679 69663 136.05 94.6875 0.00837554 0.00734553 2024-08-19T14:29:10.528 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:29:10.550 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:10.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:10.552 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:29:10.652 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:29:10.688 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:10.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:10.689 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:29:10.790 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:29:10.815 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:10.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:10.816 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:29:10.917 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:29:10.936 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 7, 8] out_osds: [10, 6, 2, 9, 11] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-08-19T14:29:10.936 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-19T14:29:10.936 INFO:tasks.thrashosds.thrasher:Adding osd 2 2024-08-19T14:29:10.936 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd in 2 2024-08-19T14:29:10.952 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:10.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:10.953 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:29:10.962 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-08-19T14:29:11.054 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:29:11.094 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:11.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:11.096 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:29:11.197 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:29:11.218 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:11.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:11.219 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:29:11.309 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:29:11.320 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:29:11.348 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:11.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:11.349 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:29:11.450 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:29:11.474 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:11.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:11.475 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:29:11.478 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:11 smithi031 ceph-mon[16336]: pgmap v1200: 47 pgs: 47 active+clean; 4.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 128 MiB/s wr, 2.05k op/s 2024-08-19T14:29:11.509 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 33 16 71224 71208 134.853 96.5625 0.0132581 0.00741176 2024-08-19T14:29:11.575 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:29:11.598 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:11.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:11.599 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:29:11.661 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:29:11.661 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:29:11.694 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:29:11.694 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:29:11.699 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:29:11.726 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:29:11.726 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:29:11.733 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:11.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:11.735 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:29:11.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:11 smithi032 ceph-mon[20004]: pgmap v1200: 47 pgs: 47 active+clean; 4.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 128 MiB/s wr, 2.05k op/s 2024-08-19T14:29:11.757 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:29:11.758 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:29:11.796 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:29:11.796 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:29:11.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:11 smithi045 ceph-mon[20143]: pgmap v1200: 47 pgs: 47 active+clean; 4.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 128 MiB/s wr, 2.05k op/s 2024-08-19T14:29:11.833 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:29:11.834 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:29:11.835 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:29:11.865 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:29:11.866 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:29:11.866 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:11.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:11.867 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:29:11.899 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:29:11.899 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:29:11.933 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:29:11.933 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:29:11.963 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:29:11.964 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:29:11.968 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:29:11.989 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:29:11.990 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:29:11.991 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:11.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:11.992 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:29:12.014 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:29:12.015 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:29:12.093 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:29:12.126 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:12.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:12.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:29:12.229 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:29:12.264 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:12.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:12.265 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:29:12.366 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:29:12.404 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:12.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:12.405 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:29:12.506 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:29:12.509 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 34 16 72576 72560 133.371 84.5 0.00298955 0.00747161 2024-08-19T14:29:12.527 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:12.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:12.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:29:12.630 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:29:12.652 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:12.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:12.653 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:29:12.754 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:29:12.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:12 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/4244722886' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:29:12.776 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:12.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:12.777 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:29:12.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:12 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/4244722886' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:29:12.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:12 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/4244722886' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:29:12.878 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:29:12.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:12.912 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:12.912 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:29:13.013 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:29:13.053 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:13.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:13.054 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:29:13.156 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:29:13.178 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:13.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:13.180 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:29:13.281 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:29:13.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:13.321 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:13.321 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:29:13.381 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:29:13.422 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:29:13.465 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:13.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:13.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:29:13.509 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 35 16 73789 73773 131.727 75.8125 0.00227703 0.00757954 2024-08-19T14:29:13.567 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:29:13.588 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:13.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:13.589 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:29:13.690 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:29:13.712 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:13.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:13.713 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:29:13.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:13 smithi032 ceph-mon[20004]: pgmap v1201: 47 pgs: 47 active+clean; 4.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 98 MiB/s wr, 1.57k op/s 2024-08-19T14:29:13.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:13 smithi045 ceph-mon[20143]: pgmap v1201: 47 pgs: 47 active+clean; 4.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 98 MiB/s wr, 1.57k op/s 2024-08-19T14:29:13.814 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:29:13.847 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:13 smithi031 ceph-mon[16336]: pgmap v1201: 47 pgs: 47 active+clean; 4.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 98 MiB/s wr, 1.57k op/s 2024-08-19T14:29:13.854 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:13.854 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:29:13.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:13.855 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:29:13.956 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:29:13.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:13.981 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:13.981 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:29:14.082 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:29:14.108 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:14.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:14.109 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:29:14.210 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:29:14.251 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:14.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:14.253 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:29:14.354 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:29:14.392 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:14.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:14.393 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:29:14.396 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is set 2024-08-19T14:29:14.494 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:29:14.509 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 36 16 74939 74923 130.064 71.875 0.00660605 0.00768531 2024-08-19T14:29:14.528 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:14.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:14.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:29:14.630 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:29:14.652 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:14.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:14.653 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:29:14.730 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:14 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/4215332135' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:29:14.730 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:14 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:29:14.754 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:29:14.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:14 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/4215332135' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:29:14.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:14 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:29:14.776 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:14.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:14.777 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:29:14.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:14 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/4215332135' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:29:14.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:14 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:29:14.878 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:29:14.909 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:14.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:14.910 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:29:14.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-08-19T14:29:14.975+0000 7f764b7fe640 0 --2- 172.21.15.31:0/2369894176 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f764c109560 0x7f764c1a6bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-08-19T14:29:15.011 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:29:15.034 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:15.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:15.035 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:29:15.137 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:29:15.177 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:15.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:15.178 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:29:15.279 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:29:15.318 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:15.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:15.319 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:29:15.399 INFO:teuthology.orchestra.run.smithi031.stderr:marked in osd.2. 2024-08-19T14:29:15.420 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:29:15.443 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:15.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:15.445 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:29:15.509 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 37 16 76326 76310 128.891 86.6875 0.00524366 0.00773563 2024-08-19T14:29:15.546 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:29:15.567 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:15.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:15.568 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:29:15.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:15 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mon-a[16332]: 2024-08-19T14:29:15.330+0000 7f2ff503e640 -1 mon.a@0(leader).osd e523 definitely_dead 0 2024-08-19T14:29:15.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:15 smithi031 ceph-mon[16336]: pgmap v1202: 47 pgs: 47 active+clean; 4.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 96 MiB/s wr, 1.54k op/s 2024-08-19T14:29:15.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:15 smithi031 ceph-mon[16336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:29:15.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:15 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:29:15.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:15 smithi031 ceph-mon[16336]: osdmap e523: 12 total, 12 up, 7 in 2024-08-19T14:29:15.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:15 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2369894176' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2024-08-19T14:29:15.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:15 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2024-08-19T14:29:15.669 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:29:15.691 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:15.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:15.692 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:29:15.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:15 smithi032 ceph-mon[20004]: pgmap v1202: 47 pgs: 47 active+clean; 4.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 96 MiB/s wr, 1.54k op/s 2024-08-19T14:29:15.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:15 smithi032 ceph-mon[20004]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:29:15.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:15 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:29:15.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:15 smithi032 ceph-mon[20004]: osdmap e523: 12 total, 12 up, 7 in 2024-08-19T14:29:15.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:15 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2369894176' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2024-08-19T14:29:15.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:15 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2024-08-19T14:29:15.792 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:29:15.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:15 smithi045 ceph-mon[20143]: pgmap v1202: 47 pgs: 47 active+clean; 4.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 96 MiB/s wr, 1.54k op/s 2024-08-19T14:29:15.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:15 smithi045 ceph-mon[20143]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:29:15.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:15 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:29:15.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:15 smithi045 ceph-mon[20143]: osdmap e523: 12 total, 12 up, 7 in 2024-08-19T14:29:15.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:15 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2369894176' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2024-08-19T14:29:15.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:15 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2024-08-19T14:29:15.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:15.814 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:15.814 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:29:15.907 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:29:15.916 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:29:15.971 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:15.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:15.972 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:29:16.073 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:29:16.095 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:16.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:16.096 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:29:16.198 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:29:16.216 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:29:16.217 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 20, 2024-08-19T14:29:16.217 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 600, 2024-08-19T14:29:16.217 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [ 2024-08-19T14:29:16.217 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.217 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25057.0:216925 3.25 3:a76b9d69:::benchmark_data_smithi045_59973_object85746:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e307)", 2024-08-19T14:29:16.217 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:19:34.561921+0000", 2024-08-19T14:29:16.217 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 581.65272543200001, 2024-08-19T14:29:16.217 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.208080195, 2024-08-19T14:29:16.217 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:29:16.217 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:29:16.217 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:29:16.217 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:29:16.217 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25057", 2024-08-19T14:29:16.218 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2616186731", 2024-08-19T14:29:16.218 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 216925 2024-08-19T14:29:16.218 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.218 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:29:16.218 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.218 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:29:16.218 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:34.561921+0000", 2024-08-19T14:29:16.218 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:29:16.218 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.218 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.218 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:29:16.218 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:34.561921+0000", 2024-08-19T14:29:16.219 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:29:16.219 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.219 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.219 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:29:16.219 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:34.561922+0000", 2024-08-19T14:29:16.219 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4589999999999999e-06 2024-08-19T14:29:16.219 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.219 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.219 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:29:16.219 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:34.561926+0000", 2024-08-19T14:29:16.219 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.799e-06 2024-08-19T14:29:16.219 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.219 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.219 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:29:16.220 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:34.561935+0000", 2024-08-19T14:29:16.220 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.217e-06 2024-08-19T14:29:16.220 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.220 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.220 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:29:16.220 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:34.561941+0000", 2024-08-19T14:29:16.220 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.8130000000000001e-06 2024-08-19T14:29:16.220 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.220 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.220 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:29:16.220 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:34.721674+0000", 2024-08-19T14:29:16.220 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.15973267599999999 2024-08-19T14:29:16.220 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.221 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.221 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:29:16.221 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:34.721844+0000", 2024-08-19T14:29:16.221 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000170271 2024-08-19T14:29:16.221 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.221 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.221 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 8,9,11", 2024-08-19T14:29:16.221 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:34.722139+0000", 2024-08-19T14:29:16.221 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00029475399999999999 2024-08-19T14:29:16.221 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.221 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.221 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:16.221 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:34.726021+0000", 2024-08-19T14:29:16.221 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.003881865 2024-08-19T14:29:16.222 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.222 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.222 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:16.222 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:34.734915+0000", 2024-08-19T14:29:16.222 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0088943110000000002 2024-08-19T14:29:16.222 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.222 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.222 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:16.222 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:34.734993+0000", 2024-08-19T14:29:16.222 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.7733999999999998e-05 2024-08-19T14:29:16.223 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.223 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.223 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:29:16.223 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:34.769714+0000", 2024-08-19T14:29:16.223 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.034721476000000001 2024-08-19T14:29:16.223 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.223 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.223 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:29:16.223 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:34.769792+0000", 2024-08-19T14:29:16.223 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.7800999999999996e-05 2024-08-19T14:29:16.223 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.223 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.223 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:29:16.224 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:19:34.770001+0000", 2024-08-19T14:29:16.224 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00020901900000000001 2024-08-19T14:29:16.224 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:29:16.224 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:29:16.224 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:29:16.224 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.224 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.224 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25333.0:22719 4.0 4:0fc4203e:::benchmark_data_smithi045_71562_object22718:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-08-19T14:29:16.224 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:20:41.358778+0000", 2024-08-19T14:29:16.224 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 514.85586819900004, 2024-08-19T14:29:16.224 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.32258211599999997, 2024-08-19T14:29:16.224 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:29:16.224 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:29:16.224 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:29:16.225 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:29:16.225 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25333", 2024-08-19T14:29:16.225 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:29:16.225 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 22719 2024-08-19T14:29:16.225 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.225 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:29:16.225 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.225 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:29:16.225 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.358778+0000", 2024-08-19T14:29:16.225 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:29:16.225 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.225 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.225 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:29:16.225 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.358778+0000", 2024-08-19T14:29:16.226 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:29:16.226 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.226 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.226 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:29:16.226 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.358789+0000", 2024-08-19T14:29:16.226 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0446e-05 2024-08-19T14:29:16.226 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.226 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.226 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:29:16.226 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.358854+0000", 2024-08-19T14:29:16.226 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.5469000000000006e-05 2024-08-19T14:29:16.226 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.226 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.227 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:29:16.227 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.358877+0000", 2024-08-19T14:29:16.227 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.3285000000000002e-05 2024-08-19T14:29:16.227 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.227 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.227 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:29:16.227 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.358884+0000", 2024-08-19T14:29:16.227 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.229e-06 2024-08-19T14:29:16.227 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.227 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.227 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:29:16.227 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.358934+0000", 2024-08-19T14:29:16.227 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0878999999999999e-05 2024-08-19T14:29:16.227 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.228 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.228 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:29:16.228 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.359078+0000", 2024-08-19T14:29:16.228 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000143805 2024-08-19T14:29:16.228 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.228 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.228 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,7", 2024-08-19T14:29:16.228 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.359265+0000", 2024-08-19T14:29:16.228 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00018657799999999999 2024-08-19T14:29:16.228 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.228 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.228 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:16.228 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.360662+0000", 2024-08-19T14:29:16.228 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.001397162 2024-08-19T14:29:16.229 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.229 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.229 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:29:16.229 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.360902+0000", 2024-08-19T14:29:16.229 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000239614 2024-08-19T14:29:16.229 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.229 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.229 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:16.229 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.681249+0000", 2024-08-19T14:29:16.229 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.32034695800000001 2024-08-19T14:29:16.229 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.229 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.229 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:29:16.229 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.681311+0000", 2024-08-19T14:29:16.230 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.2346000000000002e-05 2024-08-19T14:29:16.230 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.230 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.230 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:29:16.230 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:20:41.681360+0000", 2024-08-19T14:29:16.230 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.9345000000000003e-05 2024-08-19T14:29:16.230 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:29:16.230 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:29:16.230 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:29:16.230 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.230 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.230 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25333.0:59928 4.0 4:01eb36bc:::benchmark_data_smithi045_71562_object59927:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-08-19T14:29:16.230 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:21:00.987826+0000", 2024-08-19T14:29:16.231 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 495.22681994599998, 2024-08-19T14:29:16.231 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.20708318000000001, 2024-08-19T14:29:16.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:16.232 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:29:16.232 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:29:16.232 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:29:16.232 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:29:16.233 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25333", 2024-08-19T14:29:16.233 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:29:16.233 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 59928 2024-08-19T14:29:16.233 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.233 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:29:16.233 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.233 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:29:16.233 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.987826+0000", 2024-08-19T14:29:16.233 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:29:16.233 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.233 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.233 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:29:16.233 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.987826+0000", 2024-08-19T14:29:16.234 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:29:16.234 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.234 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.234 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:29:16.234 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.987835+0000", 2024-08-19T14:29:16.234 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.7809999999999994e-06 2024-08-19T14:29:16.234 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.234 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.234 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:29:16.234 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.987889+0000", 2024-08-19T14:29:16.234 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.3498000000000002e-05 2024-08-19T14:29:16.234 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.234 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.234 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:29:16.235 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.987901+0000", 2024-08-19T14:29:16.235 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2661e-05 2024-08-19T14:29:16.235 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.235 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.235 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:29:16.235 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.987909+0000", 2024-08-19T14:29:16.235 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.3019999999999997e-06 2024-08-19T14:29:16.235 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.235 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.235 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:29:16.235 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.987976+0000", 2024-08-19T14:29:16.235 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.7602999999999996e-05 2024-08-19T14:29:16.235 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.235 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.235 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:29:16.236 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.988101+0000", 2024-08-19T14:29:16.236 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000124902 2024-08-19T14:29:16.236 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.236 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.236 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,7", 2024-08-19T14:29:16.236 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.988524+0000", 2024-08-19T14:29:16.236 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000422922 2024-08-19T14:29:16.236 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.236 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.236 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:16.237 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.997402+0000", 2024-08-19T14:29:16.237 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0088783839999999996 2024-08-19T14:29:16.237 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.237 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.237 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:29:16.237 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.997415+0000", 2024-08-19T14:29:16.237 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2125e-05 2024-08-19T14:29:16.237 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.237 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.237 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:16.237 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.194820+0000", 2024-08-19T14:29:16.237 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.19740545000000001 2024-08-19T14:29:16.237 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.237 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.238 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:29:16.238 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.194855+0000", 2024-08-19T14:29:16.238 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.4551000000000002e-05 2024-08-19T14:29:16.238 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.238 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.238 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:29:16.238 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.194910+0000", 2024-08-19T14:29:16.238 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.5000999999999997e-05 2024-08-19T14:29:16.238 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:29:16.238 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:29:16.238 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:29:16.238 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.238 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.239 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25333.0:59930 4.0 4:053c7972:::benchmark_data_smithi045_71562_object59929:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-08-19T14:29:16.239 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:21:00.988067+0000", 2024-08-19T14:29:16.239 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 495.22657913, 2024-08-19T14:29:16.239 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.20703280199999999, 2024-08-19T14:29:16.239 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:29:16.239 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:29:16.239 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:29:16.239 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:29:16.239 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25333", 2024-08-19T14:29:16.239 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:29:16.239 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 59930 2024-08-19T14:29:16.239 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.239 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:29:16.239 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.240 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:29:16.240 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.988067+0000", 2024-08-19T14:29:16.240 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:29:16.240 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.240 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.240 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:29:16.240 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.988067+0000", 2024-08-19T14:29:16.240 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:29:16.240 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.240 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.240 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:29:16.240 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.988068+0000", 2024-08-19T14:29:16.240 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.6700000000000002e-07 2024-08-19T14:29:16.240 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.240 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.241 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:29:16.241 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.988128+0000", 2024-08-19T14:29:16.241 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.0041000000000003e-05 2024-08-19T14:29:16.241 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.241 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.241 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:29:16.241 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.988150+0000", 2024-08-19T14:29:16.241 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.2262000000000001e-05 2024-08-19T14:29:16.241 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.241 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.241 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:29:16.241 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.988157+0000", 2024-08-19T14:29:16.241 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.567e-06 2024-08-19T14:29:16.241 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.241 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.242 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:29:16.242 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.990871+0000", 2024-08-19T14:29:16.242 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.002714475 2024-08-19T14:29:16.242 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.242 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.242 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:29:16.242 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.990996+0000", 2024-08-19T14:29:16.242 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012425700000000001 2024-08-19T14:29:16.242 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.242 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.242 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,7", 2024-08-19T14:29:16.242 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.991126+0000", 2024-08-19T14:29:16.242 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000130548 2024-08-19T14:29:16.242 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.243 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.243 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:29:16.243 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.997419+0000", 2024-08-19T14:29:16.243 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0062931339999999997 2024-08-19T14:29:16.243 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.243 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.243 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:16.243 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.997461+0000", 2024-08-19T14:29:16.243 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2061999999999998e-05 2024-08-19T14:29:16.243 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.243 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.243 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:16.243 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.194959+0000", 2024-08-19T14:29:16.243 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.197497439 2024-08-19T14:29:16.243 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.244 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.244 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:29:16.244 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.195017+0000", 2024-08-19T14:29:16.244 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.8272e-05 2024-08-19T14:29:16.244 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.244 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.244 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:29:16.244 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.195100+0000", 2024-08-19T14:29:16.244 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.2878000000000002e-05 2024-08-19T14:29:16.244 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:29:16.244 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:29:16.244 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:29:16.244 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.244 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.245 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25333.0:59932 4.0 4:0ae6f13c:::benchmark_data_smithi045_71562_object59931:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-08-19T14:29:16.245 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:21:00.988175+0000", 2024-08-19T14:29:16.245 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 495.22647100199998, 2024-08-19T14:29:16.245 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.21384358000000001, 2024-08-19T14:29:16.245 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:29:16.245 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:29:16.245 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:29:16.245 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:29:16.245 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25333", 2024-08-19T14:29:16.245 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:29:16.245 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 59932 2024-08-19T14:29:16.245 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.245 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:29:16.245 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.245 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:29:16.246 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.988175+0000", 2024-08-19T14:29:16.246 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:29:16.246 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.246 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.246 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:29:16.246 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.988175+0000", 2024-08-19T14:29:16.246 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:29:16.246 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.246 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.246 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:29:16.246 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.988176+0000", 2024-08-19T14:29:16.246 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.8999999999999997e-07 2024-08-19T14:29:16.246 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.247 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.247 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:29:16.247 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.988205+0000", 2024-08-19T14:29:16.247 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.9047000000000002e-05 2024-08-19T14:29:16.247 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.247 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.247 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:29:16.247 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.988212+0000", 2024-08-19T14:29:16.247 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.2250000000000003e-06 2024-08-19T14:29:16.247 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.247 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.247 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:29:16.247 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.988218+0000", 2024-08-19T14:29:16.248 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.4670000000000003e-06 2024-08-19T14:29:16.248 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.248 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.248 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:29:16.248 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.993116+0000", 2024-08-19T14:29:16.248 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0048980830000000001 2024-08-19T14:29:16.248 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.248 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.248 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:29:16.248 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.993256+0000", 2024-08-19T14:29:16.248 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000140426 2024-08-19T14:29:16.248 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.249 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.249 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,7", 2024-08-19T14:29:16.249 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.993350+0000", 2024-08-19T14:29:16.249 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.4026999999999999e-05 2024-08-19T14:29:16.249 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.249 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.249 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:29:16.249 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.997424+0000", 2024-08-19T14:29:16.249 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0040743929999999999 2024-08-19T14:29:16.249 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.249 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.249 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:16.249 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.002435+0000", 2024-08-19T14:29:16.249 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0050106919999999998 2024-08-19T14:29:16.250 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.250 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.250 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:16.250 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.201944+0000", 2024-08-19T14:29:16.250 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.19950880200000001 2024-08-19T14:29:16.250 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.250 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.250 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:29:16.250 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.201978+0000", 2024-08-19T14:29:16.250 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.4379999999999999e-05 2024-08-19T14:29:16.250 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.250 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.250 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:29:16.250 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.202019+0000", 2024-08-19T14:29:16.251 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.0547999999999999e-05 2024-08-19T14:29:16.251 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:29:16.251 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:29:16.251 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:29:16.251 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.251 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.251 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25333.0:59939 4.0 4:022bf8ca:::benchmark_data_smithi045_71562_object59938:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-08-19T14:29:16.251 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:21:00.992948+0000", 2024-08-19T14:29:16.251 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 495.22169796999998, 2024-08-19T14:29:16.251 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.20916093399999999, 2024-08-19T14:29:16.251 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:29:16.251 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:29:16.251 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:29:16.251 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:29:16.252 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25333", 2024-08-19T14:29:16.252 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:29:16.252 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 59939 2024-08-19T14:29:16.252 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.252 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:29:16.252 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.252 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:29:16.252 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.992948+0000", 2024-08-19T14:29:16.252 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:29:16.252 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.252 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.252 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:29:16.252 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.992948+0000", 2024-08-19T14:29:16.252 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:29:16.253 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.253 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.253 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:29:16.253 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.992950+0000", 2024-08-19T14:29:16.253 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.668e-06 2024-08-19T14:29:16.253 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.253 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.253 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:29:16.253 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.992997+0000", 2024-08-19T14:29:16.253 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7491999999999999e-05 2024-08-19T14:29:16.253 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.253 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.253 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:29:16.253 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.993017+0000", 2024-08-19T14:29:16.254 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.9837999999999999e-05 2024-08-19T14:29:16.254 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.254 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.254 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:29:16.254 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.993025+0000", 2024-08-19T14:29:16.254 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.2710000000000003e-06 2024-08-19T14:29:16.254 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.254 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.254 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:29:16.254 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.995120+0000", 2024-08-19T14:29:16.254 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.002095436 2024-08-19T14:29:16.254 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.254 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.255 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:29:16.255 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.995232+0000", 2024-08-19T14:29:16.255 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000111691 2024-08-19T14:29:16.255 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.255 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.255 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,7", 2024-08-19T14:29:16.255 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.995370+0000", 2024-08-19T14:29:16.255 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013791000000000001 2024-08-19T14:29:16.255 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.255 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.255 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:29:16.255 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.001953+0000", 2024-08-19T14:29:16.255 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0065829340000000004 2024-08-19T14:29:16.255 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.256 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.256 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:16.256 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.002472+0000", 2024-08-19T14:29:16.256 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00051963200000000001 2024-08-19T14:29:16.256 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.256 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.256 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:16.256 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.202049+0000", 2024-08-19T14:29:16.256 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.19957635800000001 2024-08-19T14:29:16.256 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.256 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.256 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:29:16.256 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.202076+0000", 2024-08-19T14:29:16.256 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.7314999999999999e-05 2024-08-19T14:29:16.256 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.257 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.257 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:29:16.257 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.202109+0000", 2024-08-19T14:29:16.257 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.3389000000000002e-05 2024-08-19T14:29:16.257 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:29:16.257 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:29:16.257 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:29:16.257 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.257 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.257 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25333.0:59946 4.0 4:0b0e7328:::benchmark_data_smithi045_71562_object59945:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-08-19T14:29:16.257 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:21:00.995937+0000", 2024-08-19T14:29:16.257 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 495.21870917299998, 2024-08-19T14:29:16.257 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.20888220900000001, 2024-08-19T14:29:16.257 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:29:16.258 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:29:16.258 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:29:16.258 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:29:16.258 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25333", 2024-08-19T14:29:16.258 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:29:16.258 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 59946 2024-08-19T14:29:16.258 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.258 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:29:16.258 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.258 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:29:16.258 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.995937+0000", 2024-08-19T14:29:16.258 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:29:16.258 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.258 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.259 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:29:16.259 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.995937+0000", 2024-08-19T14:29:16.259 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:29:16.259 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.259 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.259 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:29:16.259 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.995939+0000", 2024-08-19T14:29:16.259 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5510000000000001e-06 2024-08-19T14:29:16.259 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.259 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.259 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:29:16.259 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.996006+0000", 2024-08-19T14:29:16.259 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.6914000000000003e-05 2024-08-19T14:29:16.259 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.260 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.260 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:29:16.260 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.996018+0000", 2024-08-19T14:29:16.260 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2361000000000001e-05 2024-08-19T14:29:16.260 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.260 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.260 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:29:16.260 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.996025+0000", 2024-08-19T14:29:16.260 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.199e-06 2024-08-19T14:29:16.260 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.260 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.260 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:29:16.260 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.997492+0000", 2024-08-19T14:29:16.261 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.001466572 2024-08-19T14:29:16.261 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.261 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.261 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:29:16.261 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.997618+0000", 2024-08-19T14:29:16.261 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012644699999999999 2024-08-19T14:29:16.261 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.261 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.261 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,7", 2024-08-19T14:29:16.261 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.997835+0000", 2024-08-19T14:29:16.261 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000216357 2024-08-19T14:29:16.261 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.261 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.261 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:29:16.261 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.001957+0000", 2024-08-19T14:29:16.262 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0041229559999999997 2024-08-19T14:29:16.262 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.262 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.262 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:16.262 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.005478+0000", 2024-08-19T14:29:16.262 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.003520535 2024-08-19T14:29:16.262 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.262 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.262 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:16.262 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.204708+0000", 2024-08-19T14:29:16.262 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.199229884 2024-08-19T14:29:16.262 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.262 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.262 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:29:16.263 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.204761+0000", 2024-08-19T14:29:16.263 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.3443999999999999e-05 2024-08-19T14:29:16.263 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.263 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.263 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:29:16.263 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.204819+0000", 2024-08-19T14:29:16.263 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.7989000000000003e-05 2024-08-19T14:29:16.263 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:29:16.263 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:29:16.263 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:29:16.263 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.263 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.263 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25333.0:59952 4.0 4:0d22fdcc:::benchmark_data_smithi045_71562_object59951:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-08-19T14:29:16.263 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:21:00.997380+0000", 2024-08-19T14:29:16.264 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 495.21726675799999, 2024-08-19T14:29:16.264 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.207624002, 2024-08-19T14:29:16.264 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:29:16.264 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:29:16.264 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:29:16.264 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:29:16.264 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25333", 2024-08-19T14:29:16.264 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:29:16.264 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 59952 2024-08-19T14:29:16.264 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.264 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:29:16.264 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.264 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:29:16.264 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.997380+0000", 2024-08-19T14:29:16.264 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:29:16.265 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.265 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.265 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:29:16.265 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.997380+0000", 2024-08-19T14:29:16.265 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:29:16.265 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.265 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.265 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:29:16.265 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.997381+0000", 2024-08-19T14:29:16.265 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1310000000000001e-06 2024-08-19T14:29:16.265 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.265 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.265 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:29:16.265 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.997419+0000", 2024-08-19T14:29:16.266 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.8781e-05 2024-08-19T14:29:16.266 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.266 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.266 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:29:16.266 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.997433+0000", 2024-08-19T14:29:16.266 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3301e-05 2024-08-19T14:29:16.266 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.266 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.266 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:29:16.266 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.997440+0000", 2024-08-19T14:29:16.266 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.289e-06 2024-08-19T14:29:16.266 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.266 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.266 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:29:16.267 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.999803+0000", 2024-08-19T14:29:16.267 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.002363066 2024-08-19T14:29:16.267 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.267 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.267 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:29:16.267 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:00.999943+0000", 2024-08-19T14:29:16.267 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013990000000000001 2024-08-19T14:29:16.267 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.267 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.267 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,7", 2024-08-19T14:29:16.267 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.000083+0000", 2024-08-19T14:29:16.267 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000139963 2024-08-19T14:29:16.267 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.267 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.268 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:29:16.268 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.002202+0000", 2024-08-19T14:29:16.268 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0021193430000000001 2024-08-19T14:29:16.268 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.268 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.268 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:16.268 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.006637+0000", 2024-08-19T14:29:16.268 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0044344629999999996 2024-08-19T14:29:16.268 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.268 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.268 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:16.268 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.204885+0000", 2024-08-19T14:29:16.268 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.198248705 2024-08-19T14:29:16.268 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.268 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.269 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:29:16.269 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.204945+0000", 2024-08-19T14:29:16.269 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.9762000000000001e-05 2024-08-19T14:29:16.269 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.269 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.269 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:29:16.269 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:21:01.205004+0000", 2024-08-19T14:29:16.269 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.8298000000000003e-05 2024-08-19T14:29:16.269 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:29:16.269 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:29:16.269 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:29:16.269 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.269 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.269 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.25333.0:149067 4.e e378/335 4:7e247c1a:::benchmark_data_smithi045_71562_object17892:head v 378'9178, mlcod=378'9176)", 2024-08-19T14:29:16.269 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:22:24.358485+0000", 2024-08-19T14:29:16.270 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 411.856161654, 2024-08-19T14:29:16.270 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.21993733300000001, 2024-08-19T14:29:16.270 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:29:16.270 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:29:16.270 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:29:16.270 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:29:16.270 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.270 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:29:16.270 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:24.358485+0000", 2024-08-19T14:29:16.270 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:29:16.270 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.270 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.270 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:29:16.270 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:24.358485+0000", 2024-08-19T14:29:16.271 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:29:16.271 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.271 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.271 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:29:16.271 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:24.358485+0000", 2024-08-19T14:29:16.271 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.9500000000000001e-07 2024-08-19T14:29:16.271 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.271 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.271 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:29:16.271 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:24.358493+0000", 2024-08-19T14:29:16.271 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.9470000000000003e-06 2024-08-19T14:29:16.271 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.271 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.271 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:29:16.272 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:24.358505+0000", 2024-08-19T14:29:16.272 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2082000000000001e-05 2024-08-19T14:29:16.272 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.272 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.272 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:29:16.272 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:24.358513+0000", 2024-08-19T14:29:16.272 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.7489999999999998e-06 2024-08-19T14:29:16.272 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.272 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.272 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:29:16.272 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:24.358590+0000", 2024-08-19T14:29:16.272 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.7158999999999998e-05 2024-08-19T14:29:16.272 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.272 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.272 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:29:16.273 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:24.358610+0000", 2024-08-19T14:29:16.273 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.9456000000000001e-05 2024-08-19T14:29:16.273 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.273 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.273 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:29:16.273 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:24.578385+0000", 2024-08-19T14:29:16.273 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.21977474999999999 2024-08-19T14:29:16.273 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.273 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.273 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:29:16.273 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:24.578422+0000", 2024-08-19T14:29:16.273 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7394999999999997e-05 2024-08-19T14:29:16.273 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:29:16.273 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:29:16.274 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:29:16.274 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.274 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.274 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.25333.0:149070 4.e e378/335 4:77cd6059:::benchmark_data_smithi045_71562_object17895:head v 378'9179, mlcod=378'9176)", 2024-08-19T14:29:16.274 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:22:24.363184+0000", 2024-08-19T14:29:16.274 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 411.85146265600002, 2024-08-19T14:29:16.274 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.21893797600000001, 2024-08-19T14:29:16.274 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:29:16.274 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:29:16.274 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:29:16.274 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:29:16.274 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.274 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:29:16.274 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:24.363184+0000", 2024-08-19T14:29:16.275 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:29:16.275 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.275 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.275 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:29:16.275 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:24.363184+0000", 2024-08-19T14:29:16.275 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:29:16.275 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.275 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.275 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:29:16.275 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:24.363185+0000", 2024-08-19T14:29:16.275 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.9400000000000004e-07 2024-08-19T14:29:16.275 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.275 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.275 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:29:16.276 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:24.363192+0000", 2024-08-19T14:29:16.276 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.34e-06 2024-08-19T14:29:16.276 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.276 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.276 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:29:16.276 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:24.363204+0000", 2024-08-19T14:29:16.276 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1934e-05 2024-08-19T14:29:16.276 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.276 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.276 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:29:16.276 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:24.363211+0000", 2024-08-19T14:29:16.276 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.7109999999999998e-06 2024-08-19T14:29:16.276 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.276 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.276 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:29:16.277 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:24.366560+0000", 2024-08-19T14:29:16.277 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.003349552 2024-08-19T14:29:16.277 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.277 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.277 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:29:16.277 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:24.366586+0000", 2024-08-19T14:29:16.277 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.6336999999999999e-05 2024-08-19T14:29:16.277 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.277 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.277 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:29:16.277 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:24.582085+0000", 2024-08-19T14:29:16.277 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.21549889 2024-08-19T14:29:16.277 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.277 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.278 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:29:16.278 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:24.582122+0000", 2024-08-19T14:29:16.278 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.6318e-05 2024-08-19T14:29:16.278 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:29:16.278 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:29:16.278 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:29:16.278 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.278 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.278 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.25333.0:149073 4.f e378/335 4:fc7f52b3:::benchmark_data_smithi045_71562_object17898:head v 378'9244, mlcod=378'9241)", 2024-08-19T14:29:16.278 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:22:24.367356+0000", 2024-08-19T14:29:16.278 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 411.84728996199999, 2024-08-19T14:29:16.278 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.20626061600000001, 2024-08-19T14:29:16.278 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:29:16.278 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:29:16.279 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:29:16.279 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:29:16.279 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.279 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:29:16.279 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:24.367356+0000", 2024-08-19T14:29:16.279 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:29:16.279 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.279 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.279 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:29:16.279 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:24.367356+0000", 2024-08-19T14:29:16.279 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:29:16.279 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.279 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.279 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:29:16.280 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:24.367357+0000", 2024-08-19T14:29:16.280 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.4300000000000001e-07 2024-08-19T14:29:16.280 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.280 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.280 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:29:16.280 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:24.367363+0000", 2024-08-19T14:29:16.280 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.2389999999999999e-06 2024-08-19T14:29:16.280 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.280 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.280 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:29:16.280 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:24.565241+0000", 2024-08-19T14:29:16.280 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.19787778 2024-08-19T14:29:16.280 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.280 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.280 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:29:16.281 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:24.565247+0000", 2024-08-19T14:29:16.281 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.3499999999999996e-06 2024-08-19T14:29:16.281 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.281 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.281 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:29:16.281 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:24.565560+0000", 2024-08-19T14:29:16.281 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00031315899999999997 2024-08-19T14:29:16.281 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.281 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.281 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:29:16.281 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:24.565581+0000", 2024-08-19T14:29:16.281 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.1067999999999999e-05 2024-08-19T14:29:16.281 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.281 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.282 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:29:16.282 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:24.573490+0000", 2024-08-19T14:29:16.282 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0079087540000000005 2024-08-19T14:29:16.282 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.282 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.282 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:29:16.282 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:24.573617+0000", 2024-08-19T14:29:16.282 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012732299999999999 2024-08-19T14:29:16.282 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:29:16.282 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:29:16.282 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:29:16.282 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.282 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.282 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.25333.0:149082 4.e e378/335 4:766b7f6c:::benchmark_data_smithi045_71562_object17907:head v 378'9180, mlcod=378'9176)", 2024-08-19T14:29:16.283 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:22:24.374332+0000", 2024-08-19T14:29:16.283 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 411.84031424199998, 2024-08-19T14:29:16.283 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.20781767900000001, 2024-08-19T14:29:16.283 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:29:16.283 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:29:16.283 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:29:16.283 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:29:16.283 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.283 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:29:16.283 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:24.374332+0000", 2024-08-19T14:29:16.283 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:29:16.283 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.283 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.283 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:29:16.284 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:24.374332+0000", 2024-08-19T14:29:16.284 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:29:16.284 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.284 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.284 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:29:16.284 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:24.374333+0000", 2024-08-19T14:29:16.284 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.105e-06 2024-08-19T14:29:16.284 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.284 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.284 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:29:16.284 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:24.374353+0000", 2024-08-19T14:29:16.284 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.9695e-05 2024-08-19T14:29:16.284 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.284 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.284 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:29:16.285 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:24.565211+0000", 2024-08-19T14:29:16.285 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.19085808200000001 2024-08-19T14:29:16.285 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.285 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.285 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:29:16.285 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:24.565224+0000", 2024-08-19T14:29:16.285 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3305999999999999e-05 2024-08-19T14:29:16.285 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.285 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.285 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:29:16.285 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:24.569252+0000", 2024-08-19T14:29:16.285 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.004027976 2024-08-19T14:29:16.285 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.285 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.286 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:29:16.286 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:24.569277+0000", 2024-08-19T14:29:16.286 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.4561000000000001e-05 2024-08-19T14:29:16.286 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.286 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.286 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:29:16.286 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:24.582125+0000", 2024-08-19T14:29:16.286 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.012848613 2024-08-19T14:29:16.286 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.286 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.286 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:29:16.286 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:22:24.582150+0000", 2024-08-19T14:29:16.286 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.4341e-05 2024-08-19T14:29:16.286 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:29:16.287 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:29:16.287 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:29:16.287 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.287 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.287 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:16.287 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:29:16.288 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25333.0:212509 4.0 4:06a04a03:::benchmark_data_smithi045_71562_object81334:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e396)", 2024-08-19T14:29:16.288 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:23:35.781886+0000", 2024-08-19T14:29:16.288 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 340.43276021899999, 2024-08-19T14:29:16.288 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.21969265399999999, 2024-08-19T14:29:16.289 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:29:16.289 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:29:16.289 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:29:16.289 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:29:16.289 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25333", 2024-08-19T14:29:16.289 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:29:16.289 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 212509 2024-08-19T14:29:16.289 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.289 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:29:16.289 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.289 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:29:16.289 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:23:35.781886+0000", 2024-08-19T14:29:16.289 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:29:16.289 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.290 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.290 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:29:16.290 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:23:35.781886+0000", 2024-08-19T14:29:16.290 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:29:16.290 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.290 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.290 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:29:16.290 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:23:35.781887+0000", 2024-08-19T14:29:16.290 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3799999999999999e-06 2024-08-19T14:29:16.290 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.290 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.290 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:29:16.290 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:23:35.781893+0000", 2024-08-19T14:29:16.290 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1399999999999999e-06 2024-08-19T14:29:16.291 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.291 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.291 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:29:16.291 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:23:35.781903+0000", 2024-08-19T14:29:16.291 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0902e-05 2024-08-19T14:29:16.291 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.291 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.291 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:29:16.291 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:23:35.781911+0000", 2024-08-19T14:29:16.291 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.6680000000000001e-06 2024-08-19T14:29:16.291 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.291 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.291 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:29:16.292 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:23:35.782005+0000", 2024-08-19T14:29:16.292 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.3586999999999997e-05 2024-08-19T14:29:16.292 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.292 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.292 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:29:16.292 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:23:35.782128+0000", 2024-08-19T14:29:16.292 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012343900000000001 2024-08-19T14:29:16.292 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.292 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.292 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,7", 2024-08-19T14:29:16.292 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:23:35.782260+0000", 2024-08-19T14:29:16.292 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013141100000000001 2024-08-19T14:29:16.292 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.292 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.292 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:29:16.293 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:23:35.790786+0000", 2024-08-19T14:29:16.293 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0085262970000000004 2024-08-19T14:29:16.293 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.293 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.293 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:16.293 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:23:35.808252+0000", 2024-08-19T14:29:16.293 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.01746578 2024-08-19T14:29:16.293 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.293 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.293 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:16.293 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:23:36.001465+0000", 2024-08-19T14:29:16.293 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.19321337599999999 2024-08-19T14:29:16.293 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.293 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.294 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:29:16.294 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:23:36.001529+0000", 2024-08-19T14:29:16.294 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.3825000000000006e-05 2024-08-19T14:29:16.294 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.294 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.294 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:29:16.294 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:23:36.001579+0000", 2024-08-19T14:29:16.294 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.9849000000000003e-05 2024-08-19T14:29:16.294 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:29:16.294 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:29:16.294 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:29:16.294 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.294 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.294 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.25813.0:15525 5.3 e426/419 5:c601536c:::benchmark_data_smithi045_85309_object15524:head v 426'920, mlcod=426'918)", 2024-08-19T14:29:16.295 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:24:52.832761+0000", 2024-08-19T14:29:16.295 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 263.38188575300001, 2024-08-19T14:29:16.295 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.26098515500000002, 2024-08-19T14:29:16.295 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:29:16.295 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:29:16.295 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:29:16.295 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:29:16.295 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.295 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:29:16.295 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:52.832761+0000", 2024-08-19T14:29:16.295 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:29:16.295 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.295 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.295 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:29:16.296 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:52.832761+0000", 2024-08-19T14:29:16.296 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:29:16.296 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.296 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.296 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:29:16.296 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:52.832761+0000", 2024-08-19T14:29:16.296 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.4600000000000004e-07 2024-08-19T14:29:16.296 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.296 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.296 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:29:16.296 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:52.832786+0000", 2024-08-19T14:29:16.296 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.4635e-05 2024-08-19T14:29:16.296 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.296 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.297 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:29:16.297 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:52.832797+0000", 2024-08-19T14:29:16.297 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1031e-05 2024-08-19T14:29:16.297 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.297 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.297 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:29:16.297 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:52.832804+0000", 2024-08-19T14:29:16.297 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.8530000000000003e-06 2024-08-19T14:29:16.297 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.297 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.297 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:29:16.297 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:52.832857+0000", 2024-08-19T14:29:16.297 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.2985000000000002e-05 2024-08-19T14:29:16.297 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.297 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.298 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:29:16.298 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:52.832887+0000", 2024-08-19T14:29:16.298 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.9935000000000001e-05 2024-08-19T14:29:16.298 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.298 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.298 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:29:16.298 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:53.093697+0000", 2024-08-19T14:29:16.298 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.26081047800000001 2024-08-19T14:29:16.298 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.298 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.298 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:29:16.298 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:53.093746+0000", 2024-08-19T14:29:16.298 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.8492000000000003e-05 2024-08-19T14:29:16.298 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:29:16.299 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:29:16.299 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:29:16.299 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.299 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.299 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.25813.0:15526 5.1 e426/419 5:81aa3e11:::benchmark_data_smithi045_85309_object15525:head v 426'981, mlcod=426'978)", 2024-08-19T14:29:16.299 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:24:52.833461+0000", 2024-08-19T14:29:16.299 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 263.38118491900002, 2024-08-19T14:29:16.299 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.26242400999999999, 2024-08-19T14:29:16.299 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:29:16.299 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:29:16.299 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:29:16.299 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:29:16.299 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.299 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:29:16.300 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:52.833461+0000", 2024-08-19T14:29:16.300 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:29:16.300 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.300 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.300 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:29:16.300 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:52.833461+0000", 2024-08-19T14:29:16.300 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:29:16.300 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.300 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.300 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:29:16.300 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:52.833462+0000", 2024-08-19T14:29:16.300 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.1999999999999999e-07 2024-08-19T14:29:16.300 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.300 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.300 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:29:16.301 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:52.833503+0000", 2024-08-19T14:29:16.301 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.0778999999999997e-05 2024-08-19T14:29:16.301 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.301 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.301 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:29:16.301 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:53.093690+0000", 2024-08-19T14:29:16.301 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.26018666499999998 2024-08-19T14:29:16.301 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.301 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.301 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:29:16.301 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:53.093699+0000", 2024-08-19T14:29:16.301 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.9059999999999997e-06 2024-08-19T14:29:16.301 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.301 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.302 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:29:16.302 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:53.094140+0000", 2024-08-19T14:29:16.302 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00044045500000000002 2024-08-19T14:29:16.302 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.302 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.302 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:29:16.302 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:53.094155+0000", 2024-08-19T14:29:16.302 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5457000000000001e-05 2024-08-19T14:29:16.302 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.302 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.302 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:29:16.302 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:53.095828+0000", 2024-08-19T14:29:16.302 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.001672535 2024-08-19T14:29:16.303 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.303 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.303 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:29:16.303 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:53.095885+0000", 2024-08-19T14:29:16.303 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.7493e-05 2024-08-19T14:29:16.303 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:29:16.303 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:29:16.303 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:29:16.303 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.303 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.303 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.25813.0:15529 5.3 e426/419 5:c6799b39:::benchmark_data_smithi045_85309_object15528:head v 426'921, mlcod=426'918)", 2024-08-19T14:29:16.303 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:24:52.833916+0000", 2024-08-19T14:29:16.303 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 263.38073024800002, 2024-08-19T14:29:16.303 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.26193699199999998, 2024-08-19T14:29:16.303 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:29:16.304 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:29:16.304 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:29:16.304 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:29:16.304 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.304 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:29:16.304 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:52.833916+0000", 2024-08-19T14:29:16.304 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:29:16.304 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.304 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.304 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:29:16.304 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:52.833916+0000", 2024-08-19T14:29:16.304 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:29:16.304 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.305 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.305 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:29:16.305 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:52.833917+0000", 2024-08-19T14:29:16.305 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.2399999999999998e-07 2024-08-19T14:29:16.305 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.305 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.305 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:29:16.305 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:52.833962+0000", 2024-08-19T14:29:16.305 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.5729999999999998e-05 2024-08-19T14:29:16.305 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.305 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.305 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:29:16.305 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:53.093651+0000", 2024-08-19T14:29:16.305 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.25968860900000001 2024-08-19T14:29:16.305 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.306 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.306 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:29:16.306 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:53.093669+0000", 2024-08-19T14:29:16.306 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.8397e-05 2024-08-19T14:29:16.306 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.306 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.306 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:29:16.306 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:53.093785+0000", 2024-08-19T14:29:16.306 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000115746 2024-08-19T14:29:16.306 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.306 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.306 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:29:16.306 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:53.093812+0000", 2024-08-19T14:29:16.307 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.6451000000000001e-05 2024-08-19T14:29:16.307 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.307 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.307 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:29:16.307 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:53.095820+0000", 2024-08-19T14:29:16.307 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0020083359999999999 2024-08-19T14:29:16.307 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.307 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.307 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:29:16.307 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:24:53.095853+0000", 2024-08-19T14:29:16.307 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.3099000000000003e-05 2024-08-19T14:29:16.307 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:29:16.307 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:29:16.307 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:29:16.307 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.308 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.308 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:42536 6.6 6:613b1bb4:::benchmark_data_smithi045_97937_object42535:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e518)", 2024-08-19T14:29:16.308 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:28:54.302040+0000", 2024-08-19T14:29:16.308 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 21.912605974000002, 2024-08-19T14:29:16.308 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.20558112200000001, 2024-08-19T14:29:16.308 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:29:16.308 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:29:16.308 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:29:16.308 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:29:16.308 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:29:16.308 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:29:16.308 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 42536 2024-08-19T14:29:16.308 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.308 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:29:16.309 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.309 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:29:16.309 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:54.302040+0000", 2024-08-19T14:29:16.309 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:29:16.309 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.309 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.309 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:29:16.309 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:54.302040+0000", 2024-08-19T14:29:16.309 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:29:16.309 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.309 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.309 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:29:16.309 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:54.302042+0000", 2024-08-19T14:29:16.309 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4100000000000001e-06 2024-08-19T14:29:16.310 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.310 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.310 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:29:16.310 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:54.302071+0000", 2024-08-19T14:29:16.310 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.8858999999999999e-05 2024-08-19T14:29:16.310 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.310 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.310 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:29:16.310 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:54.302084+0000", 2024-08-19T14:29:16.310 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2995e-05 2024-08-19T14:29:16.310 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.310 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.310 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:29:16.311 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:54.302091+0000", 2024-08-19T14:29:16.311 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.0570000000000004e-06 2024-08-19T14:29:16.311 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.311 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.311 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:29:16.311 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:54.302151+0000", 2024-08-19T14:29:16.311 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.9963000000000002e-05 2024-08-19T14:29:16.311 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.311 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.311 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:29:16.311 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:54.302279+0000", 2024-08-19T14:29:16.311 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000128448 2024-08-19T14:29:16.312 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.312 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.312 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 4,7", 2024-08-19T14:29:16.312 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:54.302484+0000", 2024-08-19T14:29:16.312 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00020510299999999999 2024-08-19T14:29:16.312 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.312 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.312 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:16.312 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:54.306784+0000", 2024-08-19T14:29:16.312 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0042997399999999998 2024-08-19T14:29:16.312 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.312 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.312 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:29:16.312 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:54.306797+0000", 2024-08-19T14:29:16.313 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3577000000000001e-05 2024-08-19T14:29:16.313 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.313 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.313 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:16.313 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:54.507559+0000", 2024-08-19T14:29:16.313 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.20076169599999999 2024-08-19T14:29:16.313 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.313 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.313 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:29:16.313 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:54.507594+0000", 2024-08-19T14:29:16.313 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.4805999999999998e-05 2024-08-19T14:29:16.313 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.313 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.313 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:29:16.313 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:54.507621+0000", 2024-08-19T14:29:16.314 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.7467999999999999e-05 2024-08-19T14:29:16.314 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:29:16.314 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:29:16.314 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:29:16.314 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.314 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.314 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:42539 6.6 6:6f7a4d35:::benchmark_data_smithi045_97937_object42538:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e518)", 2024-08-19T14:29:16.314 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:28:54.303101+0000", 2024-08-19T14:29:16.314 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 21.911545755999999, 2024-08-19T14:29:16.314 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.20461691300000001, 2024-08-19T14:29:16.314 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:29:16.314 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:29:16.314 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:29:16.314 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:29:16.315 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:29:16.315 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:29:16.315 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 42539 2024-08-19T14:29:16.315 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.315 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:29:16.315 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.315 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:29:16.315 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:54.303101+0000", 2024-08-19T14:29:16.315 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:29:16.315 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.315 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.315 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:29:16.315 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:54.303101+0000", 2024-08-19T14:29:16.315 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:29:16.316 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.316 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.316 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:29:16.316 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:54.303102+0000", 2024-08-19T14:29:16.316 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.381e-06 2024-08-19T14:29:16.316 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.316 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.316 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:29:16.316 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:54.303131+0000", 2024-08-19T14:29:16.316 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.9091999999999999e-05 2024-08-19T14:29:16.316 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.316 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.316 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:29:16.316 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:54.303142+0000", 2024-08-19T14:29:16.317 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1212999999999999e-05 2024-08-19T14:29:16.317 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.317 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.317 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:29:16.317 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:54.303148+0000", 2024-08-19T14:29:16.317 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.1500000000000004e-06 2024-08-19T14:29:16.317 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.317 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.317 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:29:16.317 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:54.304064+0000", 2024-08-19T14:29:16.317 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00091552199999999999 2024-08-19T14:29:16.317 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.317 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.317 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:29:16.318 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:54.304187+0000", 2024-08-19T14:29:16.318 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012346500000000001 2024-08-19T14:29:16.318 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.318 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.318 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 4,7", 2024-08-19T14:29:16.318 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:54.304317+0000", 2024-08-19T14:29:16.318 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000130123 2024-08-19T14:29:16.318 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.318 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.318 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:29:16.318 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:54.306802+0000", 2024-08-19T14:29:16.318 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.002484375 2024-08-19T14:29:16.318 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.318 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.319 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:16.319 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:54.308810+0000", 2024-08-19T14:29:16.319 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.002007979 2024-08-19T14:29:16.319 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.319 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.319 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:16.319 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:54.507658+0000", 2024-08-19T14:29:16.319 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.19884776400000001 2024-08-19T14:29:16.319 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.319 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.319 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:29:16.319 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:54.507693+0000", 2024-08-19T14:29:16.319 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.5015000000000003e-05 2024-08-19T14:29:16.319 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.319 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.320 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:29:16.320 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:54.507717+0000", 2024-08-19T14:29:16.320 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.4834e-05 2024-08-19T14:29:16.320 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:29:16.320 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:29:16.320 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:29:16.320 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.320 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.320 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:74161 6.b 6:d4d69f9d:::benchmark_data_smithi045_97937_object74160:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e522)", 2024-08-19T14:29:16.320 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:29:13.871397+0000", 2024-08-19T14:29:16.320 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 2.3432488830000002, 2024-08-19T14:29:16.320 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.21687408, 2024-08-19T14:29:16.320 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:29:16.320 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:29:16.321 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:29:16.321 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:29:16.321 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:29:16.321 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:29:16.321 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 74161 2024-08-19T14:29:16.321 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.321 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:29:16.321 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.321 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:29:16.321 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:13.871397+0000", 2024-08-19T14:29:16.321 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:29:16.321 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.321 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.321 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:29:16.322 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:13.871397+0000", 2024-08-19T14:29:16.322 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:29:16.322 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.322 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.322 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:29:16.322 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:13.871399+0000", 2024-08-19T14:29:16.322 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.477e-06 2024-08-19T14:29:16.322 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.322 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.322 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:29:16.322 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:13.871475+0000", 2024-08-19T14:29:16.322 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.5794e-05 2024-08-19T14:29:16.322 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.323 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.323 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:29:16.323 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:13.871486+0000", 2024-08-19T14:29:16.323 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1613999999999999e-05 2024-08-19T14:29:16.323 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.323 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.323 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:29:16.323 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:13.871493+0000", 2024-08-19T14:29:16.323 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.0990000000000001e-06 2024-08-19T14:29:16.323 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.323 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.323 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:29:16.323 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:13.871702+0000", 2024-08-19T14:29:16.323 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00020854200000000001 2024-08-19T14:29:16.323 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.324 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.324 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:29:16.324 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:13.871842+0000", 2024-08-19T14:29:16.324 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00014012399999999999 2024-08-19T14:29:16.324 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.324 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.324 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 1,7", 2024-08-19T14:29:16.324 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:13.871969+0000", 2024-08-19T14:29:16.324 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012744300000000001 2024-08-19T14:29:16.324 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.324 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.324 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:29:16.324 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:13.877733+0000", 2024-08-19T14:29:16.324 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.005763677 2024-08-19T14:29:16.325 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.325 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.325 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:16.325 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:13.892127+0000", 2024-08-19T14:29:16.325 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.014394006000000001 2024-08-19T14:29:16.325 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.325 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.325 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:16.325 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:14.088167+0000", 2024-08-19T14:29:16.325 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.196040195 2024-08-19T14:29:16.325 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.325 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.325 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:29:16.325 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:14.088225+0000", 2024-08-19T14:29:16.326 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.7516999999999998e-05 2024-08-19T14:29:16.326 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.326 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.326 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:29:16.326 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:14.088271+0000", 2024-08-19T14:29:16.326 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.6591999999999997e-05 2024-08-19T14:29:16.326 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:29:16.326 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:29:16.326 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:29:16.326 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.326 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.326 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:74171 6.b 6:d6ff64ce:::benchmark_data_smithi045_97937_object74170:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e522)", 2024-08-19T14:29:16.326 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:29:13.881718+0000", 2024-08-19T14:29:16.326 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 2.33292834, 2024-08-19T14:29:16.327 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.21282888899999999, 2024-08-19T14:29:16.327 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:29:16.327 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:29:16.327 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:29:16.327 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:29:16.327 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:29:16.327 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:29:16.327 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 74171 2024-08-19T14:29:16.327 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.327 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:29:16.327 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.327 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:29:16.327 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:13.881718+0000", 2024-08-19T14:29:16.327 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:29:16.328 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.328 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.328 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:29:16.328 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:13.881718+0000", 2024-08-19T14:29:16.328 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:29:16.328 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.328 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.328 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:29:16.328 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:13.881719+0000", 2024-08-19T14:29:16.328 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5209999999999999e-06 2024-08-19T14:29:16.328 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.328 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.328 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:29:16.328 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:13.881744+0000", 2024-08-19T14:29:16.328 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.4562e-05 2024-08-19T14:29:16.329 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.329 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.329 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:29:16.329 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:13.881758+0000", 2024-08-19T14:29:16.329 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3920000000000001e-05 2024-08-19T14:29:16.329 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.329 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.329 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:29:16.329 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:13.881765+0000", 2024-08-19T14:29:16.329 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.2409999999999998e-06 2024-08-19T14:29:16.329 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.329 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.329 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:29:16.329 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:13.881826+0000", 2024-08-19T14:29:16.330 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.0534e-05 2024-08-19T14:29:16.330 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.330 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.330 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:29:16.330 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:13.882098+0000", 2024-08-19T14:29:16.330 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00027196700000000002 2024-08-19T14:29:16.330 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.330 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.330 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 1,7", 2024-08-19T14:29:16.330 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:13.882242+0000", 2024-08-19T14:29:16.330 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000144332 2024-08-19T14:29:16.330 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.330 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.330 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:29:16.330 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:13.885283+0000", 2024-08-19T14:29:16.331 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0030411050000000001 2024-08-19T14:29:16.331 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.331 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.331 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:16.331 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:13.893809+0000", 2024-08-19T14:29:16.331 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0085259470000000007 2024-08-19T14:29:16.331 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.331 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.331 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:16.331 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:14.094443+0000", 2024-08-19T14:29:16.331 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.20063367100000001 2024-08-19T14:29:16.331 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.331 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.331 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:29:16.332 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:14.094502+0000", 2024-08-19T14:29:16.332 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.9038e-05 2024-08-19T14:29:16.332 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:29:16.332 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:29:16.332 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:29:16.332 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:14.094547+0000", 2024-08-19T14:29:16.332 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.5050999999999999e-05 2024-08-19T14:29:16.332 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:29:16.332 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:29:16.332 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:29:16.332 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:29:16.332 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:29:16.332 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:29:16.341 INFO:tasks.thrashosds.thrasher:Added osd 2 2024-08-19T14:29:16.388 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:29:16.413 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:16.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:16.414 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:29:16.509 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 38 16 76326 76310 125.499 0 - 0.00773563 2024-08-19T14:29:16.515 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:29:16.537 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:16.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:16.538 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:29:16.639 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:29:16.665 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:16.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:16.666 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:29:16.746 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:16 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2024-08-19T14:29:16.746 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:16 smithi031 ceph-mon[16336]: osdmap e524: 12 total, 12 up, 8 in 2024-08-19T14:29:16.746 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:16 smithi031 ceph-mon[16336]: osdmap e525: 12 total, 12 up, 8 in 2024-08-19T14:29:16.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:16 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2024-08-19T14:29:16.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:16 smithi032 ceph-mon[20004]: osdmap e524: 12 total, 12 up, 8 in 2024-08-19T14:29:16.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:16 smithi032 ceph-mon[20004]: osdmap e525: 12 total, 12 up, 8 in 2024-08-19T14:29:16.766 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:29:16.792 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:16.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:16.793 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:29:16.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:16 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2024-08-19T14:29:16.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:16 smithi045 ceph-mon[20143]: osdmap e524: 12 total, 12 up, 8 in 2024-08-19T14:29:16.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:16 smithi045 ceph-mon[20143]: osdmap e525: 12 total, 12 up, 8 in 2024-08-19T14:29:16.895 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:29:16.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:16.918 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:16.918 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:29:17.019 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:29:17.045 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:17.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:17.046 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:29:17.147 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:29:17.173 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:17.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:17.174 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:29:17.275 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:29:17.301 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:17.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:17.302 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:29:17.323 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-08-19T14:29:17.347 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:29:17.403 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:29:17.429 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:17.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:17.430 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:29:17.483 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-19T14:29:17.509 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 39 16 76326 76310 122.281 0 - 0.00773563 2024-08-19T14:29:17.531 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:29:17.575 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:17.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:17.576 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:29:17.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:17 smithi031 ceph-mon[16336]: pgmap v1205: 47 pgs: 47 active+clean; 4.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 106 MiB/s wr, 1.70k op/s 2024-08-19T14:29:17.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:17 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:29:17.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:17 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:29:17.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:17 smithi031 ceph-mon[16336]: osdmap e526: 12 total, 12 up, 8 in 2024-08-19T14:29:17.626 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:29:17.677 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:29:17.699 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:17.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:17.700 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:29:17.748 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:17 smithi032 ceph-mon[20004]: pgmap v1205: 47 pgs: 47 active+clean; 4.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 106 MiB/s wr, 1.70k op/s 2024-08-19T14:29:17.748 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:17 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:29:17.748 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:17 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:29:17.748 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:17 smithi032 ceph-mon[20004]: osdmap e526: 12 total, 12 up, 8 in 2024-08-19T14:29:17.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:17 smithi045 ceph-mon[20143]: pgmap v1205: 47 pgs: 47 active+clean; 4.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 106 MiB/s wr, 1.70k op/s 2024-08-19T14:29:17.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:17 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:29:17.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:17 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:29:17.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:17 smithi045 ceph-mon[20143]: osdmap e526: 12 total, 12 up, 8 in 2024-08-19T14:29:17.802 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:29:17.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:17.827 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:17.827 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:29:17.928 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:29:17.949 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:17.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:17.951 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:29:18.052 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:29:18.073 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:18.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:18.074 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:29:18.175 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:29:18.200 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:18.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:18.201 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:29:18.302 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:29:18.324 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:18.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:18.325 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:29:18.407 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:29:18.408 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-08-19T14:29:18.408 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:18.408 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26191.0:76514 6.2 6.a1620b52 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e527)", 2024-08-19T14:29:18.408 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:29:18.400719+0000", 2024-08-19T14:29:18.408 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.0042245429999999999, 2024-08-19T14:29:18.408 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0042496799999999996, 2024-08-19T14:29:18.408 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:29:18.408 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:29:18.408 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "queued for pg", 2024-08-19T14:29:18.408 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:29:18.409 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26191", 2024-08-19T14:29:18.409 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:29:18.409 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 76514 2024-08-19T14:29:18.409 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:18.409 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:29:18.409 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:18.409 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:29:18.409 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.400719+0000", 2024-08-19T14:29:18.409 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:18.410 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:18.410 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:18.410 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:29:18.410 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.400719+0000", 2024-08-19T14:29:18.410 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:18.410 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:18.410 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:18.410 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:29:18.410 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.400721+0000", 2024-08-19T14:29:18.410 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.663e-06 2024-08-19T14:29:18.410 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:18.410 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:18.410 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:29:18.411 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.400760+0000", 2024-08-19T14:29:18.411 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9937000000000001e-05 2024-08-19T14:29:18.411 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:18.411 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:18.411 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:29:18.411 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.400775+0000", 2024-08-19T14:29:18.411 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4746000000000001e-05 2024-08-19T14:29:18.411 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:18.411 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:18.411 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:29:18.411 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.400784+0000", 2024-08-19T14:29:18.411 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6179999999999995e-06 2024-08-19T14:29:18.411 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:18.412 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:29:18.412 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:18.412 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:18.412 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:18.412 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26191.0:76516 6.f 6:f19a8995:::benchmark_data_smithi045_97937_object76515:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e527)", 2024-08-19T14:29:18.412 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:29:18.401632+0000", 2024-08-19T14:29:18.412 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.0033117810000000002, 2024-08-19T14:29:18.412 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0034076850000000001, 2024-08-19T14:29:18.412 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:29:18.412 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:29:18.412 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "waiting for sub ops", 2024-08-19T14:29:18.412 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:29:18.412 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26191", 2024-08-19T14:29:18.413 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:29:18.413 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 76516 2024-08-19T14:29:18.413 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:18.413 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:29:18.413 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:18.413 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:29:18.413 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.401632+0000", 2024-08-19T14:29:18.413 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:18.413 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:18.413 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:18.413 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:29:18.413 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.401632+0000", 2024-08-19T14:29:18.414 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:18.414 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:18.414 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:18.414 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:29:18.414 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.401633+0000", 2024-08-19T14:29:18.414 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2249999999999999e-06 2024-08-19T14:29:18.414 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:18.414 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:18.414 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:29:18.414 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.401678+0000", 2024-08-19T14:29:18.414 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5634999999999999e-05 2024-08-19T14:29:18.414 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:18.414 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:18.415 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:29:18.415 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.401693+0000", 2024-08-19T14:29:18.415 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.429e-05 2024-08-19T14:29:18.415 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:18.415 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:18.415 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:29:18.415 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.401701+0000", 2024-08-19T14:29:18.415 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1839999999999999e-06 2024-08-19T14:29:18.415 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:18.415 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:18.415 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:29:18.415 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.403572+0000", 2024-08-19T14:29:18.415 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0018707820000000001 2024-08-19T14:29:18.416 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:18.416 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:18.416 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:29:18.416 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.403725+0000", 2024-08-19T14:29:18.416 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000153256 2024-08-19T14:29:18.416 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:18.416 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:18.416 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,5", 2024-08-19T14:29:18.416 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.404615+0000", 2024-08-19T14:29:18.416 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00089037199999999995 2024-08-19T14:29:18.416 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:18.416 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:29:18.416 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:18.417 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:18.417 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:18.417 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop_reply(client.26191.0:76503 6.f e527/525)", 2024-08-19T14:29:18.417 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:29:18.403281+0000", 2024-08-19T14:29:18.417 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.001661919, 2024-08-19T14:29:18.417 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0018219779999999999, 2024-08-19T14:29:18.417 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:29:18.417 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:29:18.417 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "queued for pg", 2024-08-19T14:29:18.417 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:29:18.417 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:18.417 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:29:18.417 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.403281+0000", 2024-08-19T14:29:18.418 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:18.418 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:18.418 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:18.418 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:29:18.418 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.403281+0000", 2024-08-19T14:29:18.418 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:18.418 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:18.418 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:18.418 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:29:18.418 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.403282+0000", 2024-08-19T14:29:18.418 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.5499999999999997e-07 2024-08-19T14:29:18.418 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:18.418 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:18.419 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:29:18.419 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.403288+0000", 2024-08-19T14:29:18.419 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4040000000000003e-06 2024-08-19T14:29:18.419 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:18.419 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:18.419 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:29:18.419 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.403299+0000", 2024-08-19T14:29:18.419 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0946e-05 2024-08-19T14:29:18.419 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:18.419 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:18.419 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:29:18.419 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.403306+0000", 2024-08-19T14:29:18.419 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2679999999999997e-06 2024-08-19T14:29:18.420 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:18.420 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:29:18.420 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:18.420 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:18.420 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:18.420 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop_reply(client.26191.0:76497 6.2 e527/525)", 2024-08-19T14:29:18.420 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:29:18.403848+0000", 2024-08-19T14:29:18.420 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.0010953709999999999, 2024-08-19T14:29:18.420 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001296583, 2024-08-19T14:29:18.420 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:29:18.420 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:29:18.420 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "queued for pg", 2024-08-19T14:29:18.420 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:29:18.420 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:18.421 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:29:18.421 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.403848+0000", 2024-08-19T14:29:18.421 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:18.421 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:18.421 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:18.421 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:29:18.421 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.403848+0000", 2024-08-19T14:29:18.421 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:18.421 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:18.421 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:18.421 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:29:18.421 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.403849+0000", 2024-08-19T14:29:18.422 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6799999999999999e-07 2024-08-19T14:29:18.422 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:18.422 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:18.422 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:29:18.422 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.403878+0000", 2024-08-19T14:29:18.422 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9241999999999999e-05 2024-08-19T14:29:18.422 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:18.422 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:18.422 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:29:18.422 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.403889+0000", 2024-08-19T14:29:18.422 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0538e-05 2024-08-19T14:29:18.422 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:18.422 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:18.423 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:29:18.423 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.403897+0000", 2024-08-19T14:29:18.423 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.9969999999999998e-06 2024-08-19T14:29:18.423 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:18.423 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:29:18.423 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:18.423 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:18.423 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:18.424 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26191.0:76497 6.2 6:4829a6cb:::benchmark_data_smithi045_97937_object76496:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e527)", 2024-08-19T14:29:18.424 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:29:18.384154+0000", 2024-08-19T14:29:18.424 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.020789808, 2024-08-19T14:29:18.424 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.021044624000000001, 2024-08-19T14:29:18.424 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:29:18.424 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:29:18.424 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "waiting for sub ops", 2024-08-19T14:29:18.424 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:29:18.424 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26191", 2024-08-19T14:29:18.424 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:29:18.424 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 76497 2024-08-19T14:29:18.425 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:18.425 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:29:18.425 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:18.425 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:29:18.425 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.384154+0000", 2024-08-19T14:29:18.425 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:18.425 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:18.425 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:18.425 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:29:18.425 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.384154+0000", 2024-08-19T14:29:18.425 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:18.425 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:18.425 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:18.426 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:29:18.426 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.384155+0000", 2024-08-19T14:29:18.426 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3740000000000001e-06 2024-08-19T14:29:18.426 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:18.426 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:18.426 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:29:18.426 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.384194+0000", 2024-08-19T14:29:18.426 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9514999999999997e-05 2024-08-19T14:29:18.426 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:18.426 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:18.426 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:29:18.426 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.384210+0000", 2024-08-19T14:29:18.426 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5260999999999998e-05 2024-08-19T14:29:18.427 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:18.427 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:18.427 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:29:18.427 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.384219+0000", 2024-08-19T14:29:18.427 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.8689999999999993e-06 2024-08-19T14:29:18.427 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:18.427 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:18.427 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:29:18.427 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.390476+0000", 2024-08-19T14:29:18.427 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0062570050000000004 2024-08-19T14:29:18.427 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:18.427 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:18.427 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:29:18.427 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.390635+0000", 2024-08-19T14:29:18.427 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00015976099999999999 2024-08-19T14:29:18.428 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:18.428 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:18.428 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,3", 2024-08-19T14:29:18.428 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.390802+0000", 2024-08-19T14:29:18.428 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000166801 2024-08-19T14:29:18.428 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:18.428 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:29:18.428 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:18.428 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:18.428 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:18.428 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26191.0:76503 6.f 6:f6c22118:::benchmark_data_smithi045_97937_object76502:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e527)", 2024-08-19T14:29:18.428 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:29:18.388883+0000", 2024-08-19T14:29:18.428 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.016060695, 2024-08-19T14:29:18.429 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.016385885999999999, 2024-08-19T14:29:18.429 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:29:18.429 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:29:18.429 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "waiting for sub ops", 2024-08-19T14:29:18.429 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:29:18.429 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26191", 2024-08-19T14:29:18.429 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:29:18.429 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 76503 2024-08-19T14:29:18.429 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:18.429 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:29:18.429 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:18.429 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:29:18.429 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.388883+0000", 2024-08-19T14:29:18.430 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:18.430 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:18.430 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:18.430 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:29:18.430 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.388883+0000", 2024-08-19T14:29:18.430 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:18.430 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:18.430 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:18.430 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:29:18.430 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.388884+0000", 2024-08-19T14:29:18.430 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.677e-06 2024-08-19T14:29:18.430 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:18.430 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:18.431 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:29:18.431 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.388922+0000", 2024-08-19T14:29:18.431 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7851000000000001e-05 2024-08-19T14:29:18.431 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:18.431 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:18.431 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:29:18.431 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.388939+0000", 2024-08-19T14:29:18.431 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.66e-05 2024-08-19T14:29:18.431 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:18.431 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:18.431 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:29:18.431 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.388948+0000", 2024-08-19T14:29:18.431 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.8040000000000007e-06 2024-08-19T14:29:18.432 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:18.432 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:18.432 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:29:18.432 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.398962+0000", 2024-08-19T14:29:18.432 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010014779999999999 2024-08-19T14:29:18.432 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:18.432 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:18.432 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:29:18.432 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.399902+0000", 2024-08-19T14:29:18.432 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00093971899999999999 2024-08-19T14:29:18.432 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:18.432 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:18.432 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,5", 2024-08-19T14:29:18.432 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.400118+0000", 2024-08-19T14:29:18.433 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000216116 2024-08-19T14:29:18.433 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:18.433 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:29:18.433 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:18.433 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:18.433 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:18.433 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26191.0:76508 6.e 6:74b89b17:::benchmark_data_smithi045_97937_object76507:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e527)", 2024-08-19T14:29:18.433 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:29:18.394011+0000", 2024-08-19T14:29:18.433 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.010932839, 2024-08-19T14:29:18.433 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011326911, 2024-08-19T14:29:18.433 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:29:18.433 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:29:18.433 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "waiting for sub ops", 2024-08-19T14:29:18.433 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:29:18.434 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26191", 2024-08-19T14:29:18.434 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:29:18.434 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 76508 2024-08-19T14:29:18.434 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:18.434 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:29:18.434 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:18.434 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:29:18.434 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.394011+0000", 2024-08-19T14:29:18.434 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:18.434 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:18.434 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:18.434 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:29:18.434 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.394011+0000", 2024-08-19T14:29:18.434 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:18.435 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:18.435 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:18.435 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:29:18.435 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.394012+0000", 2024-08-19T14:29:18.435 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4899999999999999e-06 2024-08-19T14:29:18.435 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:18.435 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:18.435 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:29:18.435 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.394068+0000", 2024-08-19T14:29:18.435 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5534000000000001e-05 2024-08-19T14:29:18.435 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:18.435 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:18.435 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:29:18.435 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.394084+0000", 2024-08-19T14:29:18.436 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6073999999999999e-05 2024-08-19T14:29:18.436 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:18.436 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:18.436 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:29:18.436 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.394090+0000", 2024-08-19T14:29:18.436 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.63e-06 2024-08-19T14:29:18.436 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:18.436 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:18.436 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:29:18.436 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.404795+0000", 2024-08-19T14:29:18.436 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.01070486 2024-08-19T14:29:18.436 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:18.436 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:18.436 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:29:18.437 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.404927+0000", 2024-08-19T14:29:18.437 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013168199999999999 2024-08-19T14:29:18.437 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:18.437 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:18.437 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 1,2", 2024-08-19T14:29:18.437 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.405080+0000", 2024-08-19T14:29:18.437 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000153213 2024-08-19T14:29:18.437 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:18.437 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:29:18.437 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:18.437 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:18.437 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:18.437 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26191.0:76510 6.2 6:4a97f4b0:::benchmark_data_smithi045_97937_object76509:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e527)", 2024-08-19T14:29:18.437 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:29:18.395202+0000", 2024-08-19T14:29:18.438 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.0097416780000000001, 2024-08-19T14:29:18.438 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010206026999999999, 2024-08-19T14:29:18.438 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:29:18.438 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:29:18.438 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "waiting for sub ops", 2024-08-19T14:29:18.438 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:29:18.438 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26191", 2024-08-19T14:29:18.438 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:29:18.438 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 76510 2024-08-19T14:29:18.438 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:18.439 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:29:18.439 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:18.439 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:29:18.439 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.395202+0000", 2024-08-19T14:29:18.439 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:18.439 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:18.439 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:18.439 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:29:18.439 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.395202+0000", 2024-08-19T14:29:18.439 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:18.439 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:18.439 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:18.439 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:29:18.439 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.395203+0000", 2024-08-19T14:29:18.440 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.9999999999999996e-07 2024-08-19T14:29:18.440 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:18.440 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:18.440 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:29:18.440 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.395266+0000", 2024-08-19T14:29:18.440 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3736000000000001e-05 2024-08-19T14:29:18.440 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:18.440 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:18.440 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:29:18.440 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.395280+0000", 2024-08-19T14:29:18.440 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3424000000000001e-05 2024-08-19T14:29:18.440 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:18.440 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:18.440 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:29:18.440 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.395287+0000", 2024-08-19T14:29:18.441 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9759999999999998e-06 2024-08-19T14:29:18.441 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:18.441 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:18.441 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:29:18.441 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.403454+0000", 2024-08-19T14:29:18.441 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0081675150000000002 2024-08-19T14:29:18.441 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:18.441 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:18.441 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:29:18.441 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.403625+0000", 2024-08-19T14:29:18.441 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000170213 2024-08-19T14:29:18.441 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:18.441 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:18.441 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,3", 2024-08-19T14:29:18.442 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.403790+0000", 2024-08-19T14:29:18.442 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000165437 2024-08-19T14:29:18.442 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:18.442 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:29:18.442 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:18.442 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:18.442 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-08-19T14:29:18.442 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 8 2024-08-19T14:29:18.442 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:29:18.442 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:29:18.481 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:18.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:18.482 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:29:18.509 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:2024-08-19T14:29:18.510216+0000 min lat: 0.00119513 max lat: 2.46333 avg lat: 0.00835267 2024-08-19T14:29:18.510 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-19T14:29:18.510 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 40 16 76585 76569 119.629 5.39583 0.00665185 0.00835267 2024-08-19T14:29:18.583 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:29:18.605 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:18.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:18.606 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:29:18.706 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:29:18.728 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:18.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:18.730 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:29:18.752 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:29:18.761 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:29:18.762 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:29:18.796 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:29:18.797 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:29:18.830 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:29:18.853 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:18.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:18.855 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:29:18.856 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:29:18.856 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:29:18.888 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:29:18.888 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:29:18.927 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:29:18.927 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:29:18.956 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:29:18.964 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:29:18.964 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:29:19.008 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:29:19.008 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:29:19.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:19.016 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:19.016 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:29:19.041 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:29:19.041 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:29:19.073 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:29:19.074 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:29:19.107 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:29:19.107 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:18 smithi031 ceph-mon[16336]: osdmap e527: 12 total, 12 up, 8 in 2024-08-19T14:29:19.107 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:18 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/855984781' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:29:19.108 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:29:19.117 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:29:19.142 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:29:19.143 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:29:19.145 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:19.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:19.146 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:29:19.181 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:29:19.181 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:29:19.248 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:29:19.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:18 smithi032 ceph-mon[20004]: osdmap e527: 12 total, 12 up, 8 in 2024-08-19T14:29:19.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:18 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/855984781' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:29:19.269 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:19.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:19.270 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:29:19.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:18 smithi045 ceph-mon[20143]: osdmap e527: 12 total, 12 up, 8 in 2024-08-19T14:29:19.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:18 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/855984781' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:29:19.371 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:29:19.391 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-08-19T14:29:19.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:19.394 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:19.394 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:29:19.495 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:29:19.509 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 41 15 77197 77182 117.645 38.3125 0.0245893 0.00849468 2024-08-19T14:29:19.517 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:19.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:19.518 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:29:19.619 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:29:19.652 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:19.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:19.653 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:29:19.754 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:29:19.796 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:19.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:19.797 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:29:19.806 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:29:19.867 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is set 2024-08-19T14:29:19.898 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:29:19.919 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:19.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:19.920 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:29:19.995 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:19 smithi032 ceph-mon[20004]: pgmap v1209: 47 pgs: 6 remapped+peering, 41 active+clean; 4.5 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 19 MiB/s wr, 302 op/s 2024-08-19T14:29:19.995 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:19 smithi032 ceph-mon[20004]: osdmap e528: 12 total, 12 up, 8 in 2024-08-19T14:29:19.995 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:19 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3039153980' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:29:20.020 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:29:20.052 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:20.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:20.053 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:29:20.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:19 smithi031 ceph-mon[16336]: pgmap v1209: 47 pgs: 6 remapped+peering, 41 active+clean; 4.5 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 19 MiB/s wr, 302 op/s 2024-08-19T14:29:20.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:19 smithi031 ceph-mon[16336]: osdmap e528: 12 total, 12 up, 8 in 2024-08-19T14:29:20.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:19 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3039153980' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:29:20.154 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:29:20.186 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:20.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:20.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:29:20.288 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:29:20.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:19 smithi045 ceph-mon[20143]: pgmap v1209: 47 pgs: 6 remapped+peering, 41 active+clean; 4.5 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 19 MiB/s wr, 302 op/s 2024-08-19T14:29:20.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:19 smithi045 ceph-mon[20143]: osdmap e528: 12 total, 12 up, 8 in 2024-08-19T14:29:20.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:19 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3039153980' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:29:20.310 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:20.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:20.311 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:29:20.413 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:29:20.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:20.450 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:20.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:29:20.510 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 42 16 77863 77847 115.834 41.5625 0.0134911 0.00862798 2024-08-19T14:29:20.521 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:29:20.521 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:29:20.551 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:29:20.590 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:20.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:20.591 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:29:20.692 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:29:20.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:20.728 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:20.728 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:29:20.830 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:29:20.854 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:20.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:20.855 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:29:20.957 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:29:20.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:20.980 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:20.980 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:29:21.081 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:29:21.102 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:21.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:21.103 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:29:21.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:20 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:29:21.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:20 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:29:21.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:20 smithi031 ceph-mon[16336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:29:21.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:20 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3039153980' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:29:21.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:20 smithi031 ceph-mon[16336]: osdmap e529: 12 total, 12 up, 8 in 2024-08-19T14:29:21.204 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:29:21.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:21.234 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:21.234 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:29:21.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:20 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:29:21.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:20 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:29:21.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:20 smithi032 ceph-mon[20004]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:29:21.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:20 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3039153980' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:29:21.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:20 smithi032 ceph-mon[20004]: osdmap e529: 12 total, 12 up, 8 in 2024-08-19T14:29:21.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:20 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:29:21.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:20 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:29:21.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:20 smithi045 ceph-mon[20143]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:29:21.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:20 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3039153980' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:29:21.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:20 smithi045 ceph-mon[20143]: osdmap e529: 12 total, 12 up, 8 in 2024-08-19T14:29:21.335 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:29:21.342 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 7, 8, 2] out_osds: [10, 6, 9, 11] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-08-19T14:29:21.342 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-19T14:29:21.342 INFO:tasks.thrashosds.thrasher:Killing osd 10, live_osds are [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-08-19T14:29:21.342 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-08-19T14:29:21.343 DEBUG:teuthology.orchestra.run.smithi045:> sudo systemctl stop ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968@osd.10 2024-08-19T14:29:21.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:21.359 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:21.359 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:29:21.459 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:29:21.494 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:21.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:21.495 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:29:21.509 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 43 16 78485 78469 114.044 38.875 0.062562 0.0087613 2024-08-19T14:29:21.596 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:29:21.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:21.632 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:21.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:29:21.733 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:29:21.761 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:21.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:21.762 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:29:21.796 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:29:21 smithi045 systemd[1]: Stopping Ceph osd.10 for ed7d1c6e-5e33-11ef-bccf-c7b262605968... 2024-08-19T14:29:21.863 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:29:21.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:21.893 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:21.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:29:21.994 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:29:22.033 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:22.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:22.034 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:29:22.135 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:29:22.158 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:21 smithi032 ceph-mon[20004]: pgmap v1212: 47 pgs: 4 active+recovering+undersized+remapped, 2 peering, 6 remapped+peering, 35 active+clean; 4.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 48 MiB/s wr, 770 op/s; 18960/229800 objects misplaced (8.251%); 1.9 MiB/s, 29 objects/s recovering 2024-08-19T14:29:22.167 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:22.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:22.168 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:29:22.204 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:29:22.252 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:21 smithi031 ceph-mon[16336]: pgmap v1212: 47 pgs: 4 active+recovering+undersized+remapped, 2 peering, 6 remapped+peering, 35 active+clean; 4.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 48 MiB/s wr, 770 op/s; 18960/229800 objects misplaced (8.251%); 1.9 MiB/s, 29 objects/s recovering 2024-08-19T14:29:22.268 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:29:22.292 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:22.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:22.293 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:29:22.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:21 smithi045 ceph-mon[20143]: pgmap v1212: 47 pgs: 4 active+recovering+undersized+remapped, 2 peering, 6 remapped+peering, 35 active+clean; 4.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 48 MiB/s wr, 770 op/s; 18960/229800 objects misplaced (8.251%); 1.9 MiB/s, 29 objects/s recovering 2024-08-19T14:29:22.296 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:29:21 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[35889]: 2024-08-19T14:29:21.870+0000 7fb75e552640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-19T14:29:22.296 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:29:21 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[35889]: 2024-08-19T14:29:21.870+0000 7fb75e552640 -1 osd.10 529 *** Got signal Terminated *** 2024-08-19T14:29:22.296 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:29:21 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[35889]: 2024-08-19T14:29:21.870+0000 7fb75e552640 -1 osd.10 529 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-08-19T14:29:22.394 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:29:22.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:22.457 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:22.457 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:29:22.510 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 44 16 79171 79155 112.426 42.875 0.0161825 0.0088893 2024-08-19T14:29:22.557 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:29:22.590 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:22.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:22.591 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:29:22.686 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:29:22.692 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:29:22.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:22.747 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:22.747 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:29:22.848 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:29:22.889 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:22.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:22.890 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:29:22.895 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-19T14:29:22.991 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:29:23.040 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:23.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:23.041 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:29:23.102 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:22 smithi031 ceph-mon[16336]: osd.10 marked itself down and dead 2024-08-19T14:29:23.142 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:29:23.165 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:22 smithi032 ceph-mon[20004]: osd.10 marked itself down and dead 2024-08-19T14:29:23.171 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:23.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:23.172 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:29:23.273 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:29:23.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:22 smithi045 ceph-mon[20143]: osd.10 marked itself down and dead 2024-08-19T14:29:23.299 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:23.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:23.300 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:29:23.401 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:29:23.427 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:23.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:23.428 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:29:23.510 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 45 16 79795 79779 110.795 39 0.0737209 0.00901921 2024-08-19T14:29:23.529 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:29:23.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:23.581 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:23.581 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:29:23.682 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:29:23.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:23.713 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:23.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:29:23.796 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:29:23 smithi045 podman[100175]: 2024-08-19 14:29:23.474745823 +0000 UTC m=+1.890678993 container died 2d01c8b69bf18d0c6b8f198274c25c35dd2637fb90004a0d6c88edc4e3eb2939 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-de09dd0, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-08-19T14:29:23.815 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:29:23.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:23.858 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:23.858 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:29:23.960 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:29:23.998 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:23.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:24.000 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:29:24.100 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:29:24.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:23 smithi031 ceph-mon[16336]: pgmap v1213: 47 pgs: 5 active+recovering+undersized+remapped, 2 peering, 6 remapped+peering, 34 active+clean; 4.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 29 MiB/s wr, 470 op/s; 23492/230043 objects misplaced (10.212%); 3.0 MiB/s, 47 objects/s recovering 2024-08-19T14:29:24.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:23 smithi031 ceph-mon[16336]: osdmap e530: 12 total, 11 up, 8 in 2024-08-19T14:29:24.122 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:24.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:24.124 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:29:24.224 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:29:24.229 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:23 smithi045 ceph-mon[20143]: pgmap v1213: 47 pgs: 5 active+recovering+undersized+remapped, 2 peering, 6 remapped+peering, 34 active+clean; 4.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 29 MiB/s wr, 470 op/s; 23492/230043 objects misplaced (10.212%); 3.0 MiB/s, 47 objects/s recovering 2024-08-19T14:29:24.229 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:23 smithi045 ceph-mon[20143]: osdmap e530: 12 total, 11 up, 8 in 2024-08-19T14:29:24.247 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:24.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:24.248 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:29:24.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:23 smithi032 ceph-mon[20004]: pgmap v1213: 47 pgs: 5 active+recovering+undersized+remapped, 2 peering, 6 remapped+peering, 34 active+clean; 4.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 29 MiB/s wr, 470 op/s; 23492/230043 objects misplaced (10.212%); 3.0 MiB/s, 47 objects/s recovering 2024-08-19T14:29:24.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:23 smithi032 ceph-mon[20004]: osdmap e530: 12 total, 11 up, 8 in 2024-08-19T14:29:24.349 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:29:24.371 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:24.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:24.372 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:29:24.473 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:29:24.495 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:24.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:24.496 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:29:24.510 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 46 16 80461 80445 109.291 41.625 0.0170526 0.00914354 2024-08-19T14:29:24.546 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:29:24 smithi045 podman[100175]: 2024-08-19 14:29:24.229655871 +0000 UTC m=+2.645589033 container cleanup 2d01c8b69bf18d0c6b8f198274c25c35dd2637fb90004a0d6c88edc4e3eb2939 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-de09dd0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-08-19T14:29:24.546 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:29:24 smithi045 bash[100175]: ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10 2024-08-19T14:29:24.597 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:29:24.618 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:24.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:24.619 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:29:24.720 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:29:24.744 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:24.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:24.745 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:29:24.846 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:29:24.868 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:24.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:24.869 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:29:24.926 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:29:24 smithi045 podman[100208]: 2024-08-19 14:29:24.590170514 +0000 UTC m=+1.117003472 container remove 2d01c8b69bf18d0c6b8f198274c25c35dd2637fb90004a0d6c88edc4e3eb2939 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=squid-de09dd0, org.label-schema.vendor=CentOS) 2024-08-19T14:29:24.970 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:29:24.998 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:24.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:24.999 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:29:25.100 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:29:25.140 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:25.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:25.141 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:29:25.194 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:24 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:29:25.194 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:24 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:29:25.194 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:24 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:29:25.194 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:24 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:29:25.195 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:29:24 smithi045 podman[100382]: 2024-08-19 14:29:24.829346818 +0000 UTC m=+0.018302023 image pull ea7a5189f30bad3aca9524788e2965d2368d7ce786df09d4edcd8c3edf9cefba quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9 2024-08-19T14:29:25.195 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:29:24 smithi045 podman[100382]: 2024-08-19 14:29:24.949444747 +0000 UTC m=+0.138399942 container create d19d7cb8f0e8fd425bb3be0790cd49f691ada859d02d80c8836333e8fa08c855 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10-deactivate, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=squid-de09dd0) 2024-08-19T14:29:25.205 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:24 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:29:25.205 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:24 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:29:25.205 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:24 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:29:25.205 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:24 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:29:25.206 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-08-19T14:29:25.242 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:29:25.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:25.286 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:25.286 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:29:25.332 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:24 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:29:25.332 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:24 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:29:25.333 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:24 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:29:25.333 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:24 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:29:25.387 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:29:25.411 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:25.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:25.412 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:29:25.478 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:29:25 smithi045 podman[100382]: 2024-08-19 14:29:25.194443348 +0000 UTC m=+0.383398544 container init d19d7cb8f0e8fd425bb3be0790cd49f691ada859d02d80c8836333e8fa08c855 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=squid-de09dd0, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-08-19T14:29:25.478 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:29:25 smithi045 podman[100382]: 2024-08-19 14:29:25.199353508 +0000 UTC m=+0.388308703 container start d19d7cb8f0e8fd425bb3be0790cd49f691ada859d02d80c8836333e8fa08c855 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10-deactivate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, RELEASE=squid-de09dd0, GIT_CLEAN=True) 2024-08-19T14:29:25.478 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:29:25 smithi045 podman[100382]: 2024-08-19 14:29:25.302956283 +0000 UTC m=+0.491911491 container attach d19d7cb8f0e8fd425bb3be0790cd49f691ada859d02d80c8836333e8fa08c855 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10-deactivate, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=squid-de09dd0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731) 2024-08-19T14:29:25.478 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:29:25 smithi045 podman[100382]: 2024-08-19 14:29:25.428944335 +0000 UTC m=+0.617899540 container died d19d7cb8f0e8fd425bb3be0790cd49f691ada859d02d80c8836333e8fa08c855 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, RELEASE=squid-de09dd0, org.label-schema.build-date=20240731, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-08-19T14:29:25.510 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 47 16 81085 81069 107.795 39 0.0325908 0.00927116 2024-08-19T14:29:25.514 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:29:25.536 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:25.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:25.537 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:29:25.578 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:29:25.639 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:29:25.662 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:25.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:25.663 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:29:25.764 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:29:25.792 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:25.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:25.793 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:29:25.894 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:29:25.934 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:25.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:25.936 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:29:26.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:25 smithi032 ceph-mon[20004]: pgmap v1215: 47 pgs: 8 active+recovering+undersized+remapped, 39 active+clean; 4.9 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 53 MiB/s wr, 853 op/s; 36924/238890 objects misplaced (15.456%); 12 MiB/s, 193 objects/s recovering 2024-08-19T14:29:26.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:25 smithi032 ceph-mon[20004]: osdmap e531: 12 total, 11 up, 8 in 2024-08-19T14:29:26.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:25 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:29:26.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:25 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:29:26.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:25 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3147446502' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:29:26.037 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:29:26.067 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:26.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:26.068 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:29:26.108 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:29:26.108 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:29:26.108 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:29:26.108 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:29:26.108 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:29:26.108 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:29:26.108 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:29:26.108 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:29:26.108 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:29:26.108 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:29:26.108 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:29:26.109 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:29:26.109 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:29:26.109 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:29:26.109 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:29:26.109 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:29:26.109 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:29:26.109 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:29:26.109 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:29:26.109 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:29:26.109 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:29:26.109 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-19T14:29:26.109 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:29:26.169 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:29:26.197 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:26.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:26.198 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:29:26.283 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:25 smithi045 ceph-mon[20143]: pgmap v1215: 47 pgs: 8 active+recovering+undersized+remapped, 39 active+clean; 4.9 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 53 MiB/s wr, 853 op/s; 36924/238890 objects misplaced (15.456%); 12 MiB/s, 193 objects/s recovering 2024-08-19T14:29:26.283 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:25 smithi045 ceph-mon[20143]: osdmap e531: 12 total, 11 up, 8 in 2024-08-19T14:29:26.283 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:25 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:29:26.283 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:25 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:29:26.283 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:25 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3147446502' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:29:26.299 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:29:26.322 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:26.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:26.323 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:29:26.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:25 smithi031 ceph-mon[16336]: pgmap v1215: 47 pgs: 8 active+recovering+undersized+remapped, 39 active+clean; 4.9 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 53 MiB/s wr, 853 op/s; 36924/238890 objects misplaced (15.456%); 12 MiB/s, 193 objects/s recovering 2024-08-19T14:29:26.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:25 smithi031 ceph-mon[16336]: osdmap e531: 12 total, 11 up, 8 in 2024-08-19T14:29:26.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:25 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:29:26.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:25 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:29:26.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:25 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3147446502' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:29:26.424 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:29:26.474 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:26.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:26.475 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:29:26.510 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 48 16 81635 81619 106.266 34.375 0.0118186 0.00940512 2024-08-19T14:29:26.576 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:29:26.599 DEBUG:teuthology.orchestra.run.smithi045:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968@osd.10.service' 2024-08-19T14:29:26.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:26.600 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:26.600 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:29:26.638 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:29:26.639 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-08-19T14:29:26.639 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-08-19T14:29:26.639 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-08-19T14:29:26.639 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.639 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:15526 5.1 5:81aa3e11:::benchmark_data_smithi045_85309_object15525:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e426)", 2024-08-19T14:29:26.639 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:24:52.832831+0000", 2024-08-19T14:29:26.639 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 273.80107901000002, 2024-08-19T14:29:26.639 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.265005194, 2024-08-19T14:29:26.639 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:29:26.639 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:29:26.639 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:29:26.639 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:29:26.639 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:29:26.640 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:29:26.640 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 15526 2024-08-19T14:29:26.640 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.640 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:29:26.640 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.640 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:29:26.640 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.832831+0000", 2024-08-19T14:29:26.640 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:26.640 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.640 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.640 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:29:26.640 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.832831+0000", 2024-08-19T14:29:26.640 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:26.641 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.641 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.641 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:29:26.641 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.832832+0000", 2024-08-19T14:29:26.641 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.499e-06 2024-08-19T14:29:26.641 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.641 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.641 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:29:26.641 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.832894+0000", 2024-08-19T14:29:26.641 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1909999999999995e-05 2024-08-19T14:29:26.641 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.641 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.641 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:29:26.641 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.832910+0000", 2024-08-19T14:29:26.642 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5906e-05 2024-08-19T14:29:26.642 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.642 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.642 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:29:26.642 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.832918+0000", 2024-08-19T14:29:26.642 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2400000000000001e-06 2024-08-19T14:29:26.642 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.642 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.642 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:29:26.642 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.832964+0000", 2024-08-19T14:29:26.642 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6697999999999999e-05 2024-08-19T14:29:26.642 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.642 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.642 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:29:26.643 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.833093+0000", 2024-08-19T14:29:26.643 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012924599999999999 2024-08-19T14:29:26.643 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.643 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.643 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 3,7", 2024-08-19T14:29:26.643 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.833228+0000", 2024-08-19T14:29:26.643 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000134621 2024-08-19T14:29:26.643 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.643 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.643 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:29:26.643 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.834341+0000", 2024-08-19T14:29:26.643 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001112701 2024-08-19T14:29:26.643 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.644 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.644 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:26.644 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.835208+0000", 2024-08-19T14:29:26.644 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00086687000000000005 2024-08-19T14:29:26.644 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.644 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.644 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:26.644 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:53.097747+0000", 2024-08-19T14:29:26.644 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.26253938399999999 2024-08-19T14:29:26.644 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.644 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.644 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:29:26.644 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:53.097796+0000", 2024-08-19T14:29:26.644 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8965999999999998e-05 2024-08-19T14:29:26.644 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.644 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.645 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:29:26.645 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:53.097836+0000", 2024-08-19T14:29:26.645 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0153e-05 2024-08-19T14:29:26.645 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:26.645 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:29:26.645 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:26.645 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.645 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.645 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:15538 5.1 5:88fc82fa:::benchmark_data_smithi045_85309_object15537:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e426)", 2024-08-19T14:29:26.645 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:24:52.836081+0000", 2024-08-19T14:29:26.645 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 273.79782918400002, 2024-08-19T14:29:26.645 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.26185454899999999, 2024-08-19T14:29:26.645 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:29:26.646 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:29:26.646 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:29:26.646 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:29:26.646 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:29:26.646 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:29:26.646 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 15538 2024-08-19T14:29:26.646 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.646 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:29:26.646 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.646 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:29:26.646 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.836081+0000", 2024-08-19T14:29:26.646 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:26.647 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.647 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.647 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:29:26.647 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.836081+0000", 2024-08-19T14:29:26.647 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:26.647 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.647 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.647 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:29:26.647 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.836082+0000", 2024-08-19T14:29:26.647 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4309999999999999e-06 2024-08-19T14:29:26.647 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.647 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.647 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:29:26.647 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.836111+0000", 2024-08-19T14:29:26.648 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8274000000000001e-05 2024-08-19T14:29:26.648 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.648 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.648 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:29:26.648 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.836127+0000", 2024-08-19T14:29:26.648 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6427999999999999e-05 2024-08-19T14:29:26.648 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.648 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.648 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:29:26.648 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.836135+0000", 2024-08-19T14:29:26.648 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.4589999999999998e-06 2024-08-19T14:29:26.648 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.648 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.648 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:29:26.649 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.836291+0000", 2024-08-19T14:29:26.649 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00015594299999999999 2024-08-19T14:29:26.649 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.649 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.649 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:29:26.649 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.836589+0000", 2024-08-19T14:29:26.649 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00029754899999999998 2024-08-19T14:29:26.649 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.649 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.649 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 3,7", 2024-08-19T14:29:26.649 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.836726+0000", 2024-08-19T14:29:26.649 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000136737 2024-08-19T14:29:26.649 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.649 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.650 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:29:26.650 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.838701+0000", 2024-08-19T14:29:26.650 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0019752480000000002 2024-08-19T14:29:26.650 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.650 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.650 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:26.650 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.844308+0000", 2024-08-19T14:29:26.650 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0056073670000000003 2024-08-19T14:29:26.650 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.650 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.650 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:26.650 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:53.097873+0000", 2024-08-19T14:29:26.650 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.25356439400000003 2024-08-19T14:29:26.651 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.651 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.651 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:29:26.651 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:53.097909+0000", 2024-08-19T14:29:26.651 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6303000000000001e-05 2024-08-19T14:29:26.651 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.651 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.651 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:29:26.651 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:53.097935+0000", 2024-08-19T14:29:26.651 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6415999999999999e-05 2024-08-19T14:29:26.651 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:26.651 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:29:26.651 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:26.652 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.652 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.652 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:15554 5.1 5:8ece79d1:::benchmark_data_smithi045_85309_object15553:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e426)", 2024-08-19T14:29:26.652 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:24:52.841409+0000", 2024-08-19T14:29:26.652 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 273.79250146999999, 2024-08-19T14:29:26.652 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.25662066100000003, 2024-08-19T14:29:26.652 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:29:26.652 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:29:26.652 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:29:26.652 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:29:26.652 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:29:26.652 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:29:26.652 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 15554 2024-08-19T14:29:26.652 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.653 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:29:26.653 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.653 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:29:26.653 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.841409+0000", 2024-08-19T14:29:26.653 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:26.653 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.653 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.653 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:29:26.653 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.841409+0000", 2024-08-19T14:29:26.653 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:26.653 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.653 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.653 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:29:26.654 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.841410+0000", 2024-08-19T14:29:26.654 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3239999999999999e-06 2024-08-19T14:29:26.654 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.654 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.654 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:29:26.654 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.841462+0000", 2024-08-19T14:29:26.654 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1799000000000003e-05 2024-08-19T14:29:26.654 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.654 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.654 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:29:26.654 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.841476+0000", 2024-08-19T14:29:26.654 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4608e-05 2024-08-19T14:29:26.654 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.655 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.655 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:29:26.655 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.841485+0000", 2024-08-19T14:29:26.655 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6090000000000001e-06 2024-08-19T14:29:26.655 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.655 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.655 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:29:26.655 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.841546+0000", 2024-08-19T14:29:26.655 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1168999999999997e-05 2024-08-19T14:29:26.655 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.655 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.655 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:29:26.655 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.841682+0000", 2024-08-19T14:29:26.655 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013579900000000001 2024-08-19T14:29:26.655 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.655 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.656 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 3,7", 2024-08-19T14:29:26.656 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.841765+0000", 2024-08-19T14:29:26.656 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.3494999999999999e-05 2024-08-19T14:29:26.656 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.656 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.656 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:29:26.656 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.842729+0000", 2024-08-19T14:29:26.656 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00096402900000000004 2024-08-19T14:29:26.656 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.656 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.656 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:26.656 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.846426+0000", 2024-08-19T14:29:26.656 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0036964699999999999 2024-08-19T14:29:26.657 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.657 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.657 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:26.657 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:53.097964+0000", 2024-08-19T14:29:26.657 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.25153829300000002 2024-08-19T14:29:26.657 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.657 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.657 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:29:26.657 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:53.098003+0000", 2024-08-19T14:29:26.657 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9193999999999998e-05 2024-08-19T14:29:26.657 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.657 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.657 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:29:26.657 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:53.098029+0000", 2024-08-19T14:29:26.658 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5871999999999999e-05 2024-08-19T14:29:26.658 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:26.658 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:29:26.658 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:26.658 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.658 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.658 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:15557 5.1 5:894b28fd:::benchmark_data_smithi045_85309_object15556:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e426)", 2024-08-19T14:29:26.658 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:24:52.843093+0000", 2024-08-19T14:29:26.658 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 273.790816565, 2024-08-19T14:29:26.658 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.25539605599999998, 2024-08-19T14:29:26.658 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:29:26.658 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:29:26.658 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:29:26.658 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:29:26.659 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:29:26.659 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:29:26.659 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 15557 2024-08-19T14:29:26.659 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.659 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:29:26.659 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.659 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:29:26.659 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.843093+0000", 2024-08-19T14:29:26.659 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:26.659 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.659 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.659 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:29:26.659 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.843093+0000", 2024-08-19T14:29:26.660 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:26.660 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.660 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.660 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:29:26.660 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.843095+0000", 2024-08-19T14:29:26.660 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.246e-06 2024-08-19T14:29:26.660 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.660 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.660 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:29:26.660 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.843147+0000", 2024-08-19T14:29:26.660 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2379000000000001e-05 2024-08-19T14:29:26.660 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.660 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.661 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:29:26.661 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.843158+0000", 2024-08-19T14:29:26.661 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1238e-05 2024-08-19T14:29:26.661 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.661 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.661 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:29:26.661 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.843165+0000", 2024-08-19T14:29:26.661 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1609999999999997e-06 2024-08-19T14:29:26.661 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.661 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.661 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:29:26.661 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.843217+0000", 2024-08-19T14:29:26.661 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1898999999999999e-05 2024-08-19T14:29:26.661 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.662 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.662 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:29:26.662 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.843352+0000", 2024-08-19T14:29:26.662 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000135017 2024-08-19T14:29:26.662 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.662 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.662 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 3,7", 2024-08-19T14:29:26.662 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.843506+0000", 2024-08-19T14:29:26.662 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00015404200000000001 2024-08-19T14:29:26.662 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.662 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.662 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:29:26.662 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.844660+0000", 2024-08-19T14:29:26.662 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001153987 2024-08-19T14:29:26.663 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.663 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.663 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:26.663 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.846459+0000", 2024-08-19T14:29:26.663 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0017983700000000001 2024-08-19T14:29:26.663 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.663 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.663 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:26.663 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:53.098357+0000", 2024-08-19T14:29:26.663 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.25189793100000002 2024-08-19T14:29:26.663 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.663 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.663 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:29:26.663 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:53.098411+0000", 2024-08-19T14:29:26.664 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4796999999999997e-05 2024-08-19T14:29:26.664 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.664 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.664 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:29:26.664 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:53.098489+0000", 2024-08-19T14:29:26.664 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7989000000000002e-05 2024-08-19T14:29:26.664 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:26.664 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:29:26.664 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:26.664 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.664 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.664 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:15580 5.1 5:8e704e69:::benchmark_data_smithi045_85309_object15579:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e426)", 2024-08-19T14:29:26.664 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:24:52.850956+0000", 2024-08-19T14:29:26.664 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 273.782953932, 2024-08-19T14:29:26.665 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.24772282200000001, 2024-08-19T14:29:26.665 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:29:26.665 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:29:26.665 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:29:26.665 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:29:26.665 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:29:26.665 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:29:26.665 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 15580 2024-08-19T14:29:26.665 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.665 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:29:26.665 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.665 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:29:26.665 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.850956+0000", 2024-08-19T14:29:26.665 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:26.666 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.666 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.666 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:29:26.666 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.850956+0000", 2024-08-19T14:29:26.666 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:26.666 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.667 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.667 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:29:26.667 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.850959+0000", 2024-08-19T14:29:26.668 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8720000000000002e-06 2024-08-19T14:29:26.668 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.668 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.668 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:29:26.668 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.850994+0000", 2024-08-19T14:29:26.668 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5323e-05 2024-08-19T14:29:26.668 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.668 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.668 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:29:26.668 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.851010+0000", 2024-08-19T14:29:26.668 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5885000000000001e-05 2024-08-19T14:29:26.668 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.668 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.669 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:29:26.669 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.851018+0000", 2024-08-19T14:29:26.669 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1000000000000004e-06 2024-08-19T14:29:26.669 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.669 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.669 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:29:26.669 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.851085+0000", 2024-08-19T14:29:26.669 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6692000000000005e-05 2024-08-19T14:29:26.669 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.669 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.669 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:29:26.669 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.851218+0000", 2024-08-19T14:29:26.669 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013344400000000001 2024-08-19T14:29:26.669 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.670 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.670 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 3,7", 2024-08-19T14:29:26.670 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.851382+0000", 2024-08-19T14:29:26.670 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000163907 2024-08-19T14:29:26.670 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.670 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.670 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:29:26.670 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.852814+0000", 2024-08-19T14:29:26.670 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001431779 2024-08-19T14:29:26.670 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.670 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.670 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:26.670 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.853191+0000", 2024-08-19T14:29:26.671 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00037734900000000002 2024-08-19T14:29:26.671 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.671 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.671 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:26.671 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:53.098547+0000", 2024-08-19T14:29:26.671 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.24535599799999999 2024-08-19T14:29:26.671 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.671 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.671 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:29:26.671 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:53.098640+0000", 2024-08-19T14:29:26.671 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.2829000000000002e-05 2024-08-19T14:29:26.671 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.671 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.671 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:29:26.672 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:53.098679+0000", 2024-08-19T14:29:26.672 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8643999999999998e-05 2024-08-19T14:29:26.672 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:26.672 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:29:26.672 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:26.672 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.672 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.672 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:237733 5.11 5:8ce96d21:::benchmark_data_smithi045_85309_object117480:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e495)", 2024-08-19T14:29:26.672 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:28:07.104832+0000", 2024-08-19T14:29:26.672 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 79.529078303999995, 2024-08-19T14:29:26.672 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.249395372, 2024-08-19T14:29:26.672 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:29:26.672 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:29:26.672 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:29:26.673 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:29:26.673 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:29:26.673 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:29:26.673 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 237733 2024-08-19T14:29:26.673 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.673 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:29:26.673 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.673 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:29:26.673 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:28:07.104832+0000", 2024-08-19T14:29:26.673 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:26.673 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.673 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.673 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:29:26.674 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:28:07.104832+0000", 2024-08-19T14:29:26.674 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:26.674 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.674 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.674 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:29:26.674 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:28:07.104833+0000", 2024-08-19T14:29:26.674 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.505e-06 2024-08-19T14:29:26.674 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.674 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.674 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:29:26.674 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:28:07.104839+0000", 2024-08-19T14:29:26.674 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4689999999999996e-06 2024-08-19T14:29:26.674 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.674 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.674 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:29:26.675 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:28:07.104879+0000", 2024-08-19T14:29:26.675 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0061000000000002e-05 2024-08-19T14:29:26.675 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.675 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.675 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:29:26.675 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:28:07.104888+0000", 2024-08-19T14:29:26.675 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.9479999999999997e-06 2024-08-19T14:29:26.675 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.675 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.675 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:29:26.675 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:28:07.105668+0000", 2024-08-19T14:29:26.675 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00078030099999999995 2024-08-19T14:29:26.675 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.676 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.676 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:29:26.676 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:28:07.105681+0000", 2024-08-19T14:29:26.676 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.325e-05 2024-08-19T14:29:26.676 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.676 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.676 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:29:26.676 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:28:07.311371+0000", 2024-08-19T14:29:26.676 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20568988599999999 2024-08-19T14:29:26.676 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.676 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.676 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:29:26.676 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:28:07.311556+0000", 2024-08-19T14:29:26.676 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000184847 2024-08-19T14:29:26.677 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.677 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.677 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0,7", 2024-08-19T14:29:26.677 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:28:07.312176+0000", 2024-08-19T14:29:26.677 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00062027000000000002 2024-08-19T14:29:26.677 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.677 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.677 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:29:26.677 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:28:07.316296+0000", 2024-08-19T14:29:26.677 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.004120014 2024-08-19T14:29:26.677 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.677 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.677 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:26.677 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:28:07.323513+0000", 2024-08-19T14:29:26.677 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0072170070000000001 2024-08-19T14:29:26.678 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.678 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.678 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:26.678 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:28:07.353960+0000", 2024-08-19T14:29:26.678 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.030446371 2024-08-19T14:29:26.678 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.678 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.678 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:29:26.678 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:28:07.354014+0000", 2024-08-19T14:29:26.678 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4528999999999999e-05 2024-08-19T14:29:26.678 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.678 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.678 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:29:26.678 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:28:07.354227+0000", 2024-08-19T14:29:26.679 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000212914 2024-08-19T14:29:26.679 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:26.679 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:29:26.679 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:26.679 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.679 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.679 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26191.0:76311 6.2 6.4a652072 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e524)", 2024-08-19T14:29:26.679 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:29:15.416180+0000", 2024-08-19T14:29:26.679 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 11.217729787, 2024-08-19T14:29:26.679 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.98615755400000005, 2024-08-19T14:29:26.679 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:29:26.679 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:29:26.679 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:29:26.679 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:29:26.680 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26191", 2024-08-19T14:29:26.680 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:29:26.680 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 76311 2024-08-19T14:29:26.680 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.680 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:29:26.680 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.680 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:29:26.680 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:15.416180+0000", 2024-08-19T14:29:26.680 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:26.680 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.680 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.680 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:29:26.681 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:15.416180+0000", 2024-08-19T14:29:26.681 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:26.681 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.681 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.681 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:29:26.681 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:15.416182+0000", 2024-08-19T14:29:26.681 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.341e-06 2024-08-19T14:29:26.681 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.681 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.681 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:29:26.681 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:15.416221+0000", 2024-08-19T14:29:26.682 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9122999999999998e-05 2024-08-19T14:29:26.682 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.682 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.682 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:29:26.682 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:15.416231+0000", 2024-08-19T14:29:26.682 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.076e-05 2024-08-19T14:29:26.682 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.682 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.682 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:29:26.682 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:15.416238+0000", 2024-08-19T14:29:26.682 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1160000000000004e-06 2024-08-19T14:29:26.682 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.682 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.683 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:29:26.683 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:15.416292+0000", 2024-08-19T14:29:26.683 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4039000000000002e-05 2024-08-19T14:29:26.683 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.683 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.683 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:29:26.683 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:15.416304+0000", 2024-08-19T14:29:26.683 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2388000000000001e-05 2024-08-19T14:29:26.683 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.683 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.683 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:29:26.683 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:16.402265+0000", 2024-08-19T14:29:26.683 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.98596146799999995 2024-08-19T14:29:26.683 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.684 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.684 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:29:26.684 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:16.402338+0000", 2024-08-19T14:29:26.684 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2318999999999997e-05 2024-08-19T14:29:26.684 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:26.684 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:29:26.684 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:26.684 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.684 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.684 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26191.0:76321 6.2 6.2bd467a2 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e524)", 2024-08-19T14:29:26.684 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:29:15.422552+0000", 2024-08-19T14:29:26.684 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 11.211358326999999, 2024-08-19T14:29:26.684 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.98004182100000004, 2024-08-19T14:29:26.685 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:29:26.685 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:29:26.685 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:29:26.685 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:29:26.685 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26191", 2024-08-19T14:29:26.685 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:29:26.685 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 76321 2024-08-19T14:29:26.685 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.685 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:29:26.685 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.685 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:29:26.685 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:15.422552+0000", 2024-08-19T14:29:26.685 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:26.685 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.686 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.686 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:29:26.686 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:15.422552+0000", 2024-08-19T14:29:26.686 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:26.686 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.686 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.686 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:29:26.686 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:15.422553+0000", 2024-08-19T14:29:26.686 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7090000000000001e-06 2024-08-19T14:29:26.686 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.686 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.686 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:29:26.686 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:15.422588+0000", 2024-08-19T14:29:26.686 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.4136e-05 2024-08-19T14:29:26.687 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.687 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.687 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:29:26.687 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:15.422598+0000", 2024-08-19T14:29:26.687 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0560999999999999e-05 2024-08-19T14:29:26.687 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.687 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.687 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:29:26.687 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:15.422603+0000", 2024-08-19T14:29:26.687 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3879999999999999e-06 2024-08-19T14:29:26.687 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.687 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.687 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:29:26.687 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:15.422653+0000", 2024-08-19T14:29:26.688 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9595000000000002e-05 2024-08-19T14:29:26.688 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.688 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.688 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:29:26.688 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:15.422662+0000", 2024-08-19T14:29:26.688 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.0899999999999994e-06 2024-08-19T14:29:26.688 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.688 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.688 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:29:26.688 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:16.402581+0000", 2024-08-19T14:29:26.688 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.97991880200000003 2024-08-19T14:29:26.688 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.688 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.688 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:29:26.689 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:16.402593+0000", 2024-08-19T14:29:26.689 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.254e-05 2024-08-19T14:29:26.689 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:26.689 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:29:26.689 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:26.689 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.689 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.689 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26191.0:76326 6.e 6.dc7cf32e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e524)", 2024-08-19T14:29:26.689 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:29:15.428428+0000", 2024-08-19T14:29:26.689 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 11.205481555, 2024-08-19T14:29:26.689 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.97310281799999998, 2024-08-19T14:29:26.689 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:29:26.689 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:29:26.689 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:29:26.690 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:29:26.690 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26191", 2024-08-19T14:29:26.690 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:29:26.690 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 76326 2024-08-19T14:29:26.690 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.690 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:29:26.690 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.690 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:29:26.690 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:15.428428+0000", 2024-08-19T14:29:26.690 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:26.690 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.690 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.690 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:29:26.691 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:15.428428+0000", 2024-08-19T14:29:26.691 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:26.691 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.691 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.691 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:29:26.691 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:15.428429+0000", 2024-08-19T14:29:26.691 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.7999999999999993e-07 2024-08-19T14:29:26.691 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.691 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.691 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:29:26.691 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:15.428462+0000", 2024-08-19T14:29:26.691 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2212999999999998e-05 2024-08-19T14:29:26.691 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.691 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.691 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:29:26.692 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:15.428472+0000", 2024-08-19T14:29:26.692 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0208e-05 2024-08-19T14:29:26.692 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.692 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.692 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:29:26.692 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:15.428477+0000", 2024-08-19T14:29:26.692 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1919999999999996e-06 2024-08-19T14:29:26.692 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.692 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.692 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:29:26.692 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:15.428524+0000", 2024-08-19T14:29:26.692 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6946000000000001e-05 2024-08-19T14:29:26.692 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.692 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.693 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:29:26.693 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:15.428532+0000", 2024-08-19T14:29:26.693 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.0879999999999995e-06 2024-08-19T14:29:26.693 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.693 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.693 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:29:26.693 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:16.401497+0000", 2024-08-19T14:29:26.693 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.97296510999999997 2024-08-19T14:29:26.693 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.693 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.693 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:29:26.693 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:16.401531+0000", 2024-08-19T14:29:26.693 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.4081000000000001e-05 2024-08-19T14:29:26.693 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:26.694 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:29:26.694 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:26.694 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.694 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.694 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26191.0:76326 6.e 6:74cf3e3b:::benchmark_data_smithi045_97937_object76325:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e526)", 2024-08-19T14:29:26.694 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:29:17.439806+0000", 2024-08-19T14:29:26.694 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 9.1941036290000007, 2024-08-19T14:29:26.694 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.31982314000000001, 2024-08-19T14:29:26.694 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:29:26.694 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:29:26.694 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:29:26.694 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:29:26.694 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26191", 2024-08-19T14:29:26.694 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:29:26.695 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 76326 2024-08-19T14:29:26.695 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.695 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:29:26.695 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.695 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:29:26.695 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.439806+0000", 2024-08-19T14:29:26.695 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:26.695 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.695 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.695 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:29:26.695 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.439806+0000", 2024-08-19T14:29:26.695 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:26.695 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.695 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.695 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:29:26.696 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.439808+0000", 2024-08-19T14:29:26.696 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8899999999999999e-06 2024-08-19T14:29:26.696 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.696 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.696 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:29:26.696 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.439913+0000", 2024-08-19T14:29:26.696 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00010467300000000001 2024-08-19T14:29:26.696 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.696 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.696 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:29:26.696 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.439930+0000", 2024-08-19T14:29:26.696 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6891000000000001e-05 2024-08-19T14:29:26.696 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.696 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.697 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:29:26.697 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.439941+0000", 2024-08-19T14:29:26.697 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1482e-05 2024-08-19T14:29:26.697 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.697 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.697 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:29:26.697 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.442395+0000", 2024-08-19T14:29:26.697 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0024540460000000001 2024-08-19T14:29:26.697 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.697 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.697 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:29:26.697 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.442404+0000", 2024-08-19T14:29:26.697 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.337e-06 2024-08-19T14:29:26.697 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.698 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.698 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:29:26.698 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.727910+0000", 2024-08-19T14:29:26.698 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.28550632999999997 2024-08-19T14:29:26.698 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.698 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.698 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:29:26.698 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.728089+0000", 2024-08-19T14:29:26.698 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000178696 2024-08-19T14:29:26.698 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.698 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.698 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 1,2", 2024-08-19T14:29:26.698 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.728278+0000", 2024-08-19T14:29:26.699 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000189249 2024-08-19T14:29:26.699 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.699 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.699 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:26.699 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.736958+0000", 2024-08-19T14:29:26.699 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0086800530000000001 2024-08-19T14:29:26.699 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.699 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.699 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:29:26.699 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.736975+0000", 2024-08-19T14:29:26.699 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7269999999999999e-05 2024-08-19T14:29:26.699 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.699 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.699 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:26.700 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.756064+0000", 2024-08-19T14:29:26.700 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.019088357 2024-08-19T14:29:26.700 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.700 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.700 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:29:26.700 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.756105+0000", 2024-08-19T14:29:26.700 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1349e-05 2024-08-19T14:29:26.700 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.700 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.700 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:29:26.700 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.759629+0000", 2024-08-19T14:29:26.700 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.003524517 2024-08-19T14:29:26.700 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:26.700 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:29:26.700 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:26.701 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.701 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.701 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26191.0:76311 6.2 6:4e04a652:::benchmark_data_smithi045_97937_object76310:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e526)", 2024-08-19T14:29:26.701 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:29:17.440956+0000", 2024-08-19T14:29:26.701 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 9.1929540159999998, 2024-08-19T14:29:26.701 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.30557601400000001, 2024-08-19T14:29:26.701 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:29:26.701 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:29:26.701 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:29:26.701 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:29:26.701 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26191", 2024-08-19T14:29:26.701 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:29:26.701 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 76311 2024-08-19T14:29:26.701 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.702 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:29:26.702 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.702 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:29:26.702 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.440956+0000", 2024-08-19T14:29:26.702 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:26.702 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.702 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.702 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:29:26.702 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.440956+0000", 2024-08-19T14:29:26.702 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:26.702 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.702 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.702 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:29:26.702 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.440957+0000", 2024-08-19T14:29:26.703 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0330000000000001e-06 2024-08-19T14:29:26.703 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.703 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.703 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:29:26.703 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.441017+0000", 2024-08-19T14:29:26.703 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0331000000000002e-05 2024-08-19T14:29:26.703 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.703 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.703 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:29:26.703 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.441031+0000", 2024-08-19T14:29:26.703 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4096e-05 2024-08-19T14:29:26.703 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.703 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.704 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:29:26.704 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.441042+0000", 2024-08-19T14:29:26.704 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0315e-05 2024-08-19T14:29:26.704 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.704 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.704 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:29:26.704 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.444349+0000", 2024-08-19T14:29:26.704 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0033070479999999999 2024-08-19T14:29:26.704 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.704 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.704 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:29:26.704 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.444361+0000", 2024-08-19T14:29:26.704 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2007e-05 2024-08-19T14:29:26.704 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.705 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.705 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:29:26.705 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.709039+0000", 2024-08-19T14:29:26.705 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.26467776300000001 2024-08-19T14:29:26.705 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.705 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.705 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:29:26.705 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.709150+0000", 2024-08-19T14:29:26.705 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000111773 2024-08-19T14:29:26.705 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.705 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.705 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,3", 2024-08-19T14:29:26.705 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.709263+0000", 2024-08-19T14:29:26.705 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000112893 2024-08-19T14:29:26.706 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.706 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.706 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:29:26.706 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.717202+0000", 2024-08-19T14:29:26.706 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0079389950000000008 2024-08-19T14:29:26.706 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.706 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.706 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:26.706 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.717300+0000", 2024-08-19T14:29:26.706 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.8022000000000004e-05 2024-08-19T14:29:26.706 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.706 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.706 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:26.706 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.742903+0000", 2024-08-19T14:29:26.707 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.025602771999999999 2024-08-19T14:29:26.707 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.707 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.707 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:29:26.707 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.742968+0000", 2024-08-19T14:29:26.707 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4818000000000002e-05 2024-08-19T14:29:26.707 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.707 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.707 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:29:26.707 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.746532+0000", 2024-08-19T14:29:26.707 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0035641480000000001 2024-08-19T14:29:26.707 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:26.707 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:29:26.707 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:26.708 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.708 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.708 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26191.0:76321 6.2 6:45e62bd4:::benchmark_data_smithi045_97937_object76320:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e526)", 2024-08-19T14:29:26.708 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:29:17.441060+0000", 2024-08-19T14:29:26.708 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 9.1928502240000007, 2024-08-19T14:29:26.708 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.30905421399999999, 2024-08-19T14:29:26.708 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:29:26.708 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:29:26.708 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:29:26.708 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:29:26.708 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26191", 2024-08-19T14:29:26.708 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:29:26.708 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 76321 2024-08-19T14:29:26.708 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.709 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:29:26.709 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.709 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:29:26.709 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.441060+0000", 2024-08-19T14:29:26.709 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:26.709 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.709 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.709 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:29:26.710 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:29:26.710 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.441060+0000", 2024-08-19T14:29:26.710 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:26.711 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.711 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.711 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:29:26.711 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.441060+0000", 2024-08-19T14:29:26.711 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2900000000000004e-07 2024-08-19T14:29:26.711 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.711 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.711 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:29:26.711 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.441093+0000", 2024-08-19T14:29:26.711 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2753999999999998e-05 2024-08-19T14:29:26.711 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.711 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.711 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:29:26.711 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.441102+0000", 2024-08-19T14:29:26.712 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.1069999999999994e-06 2024-08-19T14:29:26.712 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.712 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.712 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:29:26.712 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.441109+0000", 2024-08-19T14:29:26.712 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4699999999999999e-06 2024-08-19T14:29:26.712 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.712 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.712 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:29:26.712 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.444384+0000", 2024-08-19T14:29:26.712 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.003274927 2024-08-19T14:29:26.712 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.712 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.712 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:29:26.713 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.444390+0000", 2024-08-19T14:29:26.713 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4799999999999998e-06 2024-08-19T14:29:26.713 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.713 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.713 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:29:26.713 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.714684+0000", 2024-08-19T14:29:26.713 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.27029436899999998 2024-08-19T14:29:26.713 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.713 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.713 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:29:26.713 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.714834+0000", 2024-08-19T14:29:26.713 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014928900000000001 2024-08-19T14:29:26.713 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.713 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.714 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,3", 2024-08-19T14:29:26.714 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.715028+0000", 2024-08-19T14:29:26.714 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00019421999999999999 2024-08-19T14:29:26.714 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.714 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.714 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:29:26.714 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.717424+0000", 2024-08-19T14:29:26.714 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0023956350000000001 2024-08-19T14:29:26.714 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.714 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.714 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:26.714 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.718122+0000", 2024-08-19T14:29:26.715 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00069822999999999999 2024-08-19T14:29:26.715 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.715 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.715 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:26.715 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.746605+0000", 2024-08-19T14:29:26.715 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.028483366999999999 2024-08-19T14:29:26.715 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.715 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.715 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:29:26.715 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.746677+0000", 2024-08-19T14:29:26.715 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1638999999999996e-05 2024-08-19T14:29:26.715 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.715 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.715 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:29:26.715 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.750114+0000", 2024-08-19T14:29:26.716 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0034371979999999998 2024-08-19T14:29:26.716 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:26.716 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:29:26.716 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:26.716 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.716 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.716 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26191.0:76305 6.f 6:fe56d33d:::benchmark_data_smithi045_97937_object76304:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e526)", 2024-08-19T14:29:26.716 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:29:17.446200+0000", 2024-08-19T14:29:26.716 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 9.1877099849999997, 2024-08-19T14:29:26.716 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.26858252100000002, 2024-08-19T14:29:26.716 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:29:26.716 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:29:26.716 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:29:26.716 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:29:26.717 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26191", 2024-08-19T14:29:26.717 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:29:26.717 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 76305 2024-08-19T14:29:26.717 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.717 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:29:26.717 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.717 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:29:26.717 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.446200+0000", 2024-08-19T14:29:26.717 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:26.717 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.717 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.717 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:29:26.717 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.446200+0000", 2024-08-19T14:29:26.717 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:26.717 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.718 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.718 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:29:26.718 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.446202+0000", 2024-08-19T14:29:26.718 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9819999999999998e-06 2024-08-19T14:29:26.718 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.718 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.718 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:29:26.718 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.446270+0000", 2024-08-19T14:29:26.718 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8035999999999997e-05 2024-08-19T14:29:26.718 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.718 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.718 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:29:26.718 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.446290+0000", 2024-08-19T14:29:26.718 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9882999999999999e-05 2024-08-19T14:29:26.719 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.719 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.719 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:29:26.719 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.446303+0000", 2024-08-19T14:29:26.719 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3233e-05 2024-08-19T14:29:26.719 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.719 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.719 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:29:26.719 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.446901+0000", 2024-08-19T14:29:26.719 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00059783899999999997 2024-08-19T14:29:26.719 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.719 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.719 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:29:26.719 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.446912+0000", 2024-08-19T14:29:26.719 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0791000000000001e-05 2024-08-19T14:29:26.719 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.720 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.720 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:29:26.720 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.688663+0000", 2024-08-19T14:29:26.720 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.24175074299999999 2024-08-19T14:29:26.720 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.720 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.720 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:29:26.720 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.688828+0000", 2024-08-19T14:29:26.720 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00016598200000000001 2024-08-19T14:29:26.720 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.720 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.720 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,5", 2024-08-19T14:29:26.720 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.689014+0000", 2024-08-19T14:29:26.720 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000185704 2024-08-19T14:29:26.721 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.721 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.721 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:26.721 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.695917+0000", 2024-08-19T14:29:26.721 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.006903111 2024-08-19T14:29:26.721 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.721 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.721 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:29:26.721 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.695928+0000", 2024-08-19T14:29:26.721 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0241000000000001e-05 2024-08-19T14:29:26.721 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.721 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.721 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:26.722 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.711710+0000", 2024-08-19T14:29:26.722 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.015782348000000002 2024-08-19T14:29:26.722 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.722 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.722 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:29:26.722 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.711759+0000", 2024-08-19T14:29:26.722 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8758000000000003e-05 2024-08-19T14:29:26.722 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.722 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.722 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:29:26.722 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.714783+0000", 2024-08-19T14:29:26.722 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0030238700000000001 2024-08-19T14:29:26.722 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:26.722 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:29:26.723 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:26.723 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.723 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.723 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26191.0:76312 6.f 6:f7fbbdfd:::benchmark_data_smithi045_97937_object76311:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e526)", 2024-08-19T14:29:26.723 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:29:17.446323+0000", 2024-08-19T14:29:26.723 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 9.1875866750000004, 2024-08-19T14:29:26.723 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.27083775100000002, 2024-08-19T14:29:26.723 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:29:26.723 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:29:26.723 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:29:26.723 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:29:26.723 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26191", 2024-08-19T14:29:26.723 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:29:26.724 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 76312 2024-08-19T14:29:26.724 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.724 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:29:26.724 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.724 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:29:26.724 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.446323+0000", 2024-08-19T14:29:26.724 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:26.724 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.724 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.724 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:29:26.724 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.446323+0000", 2024-08-19T14:29:26.724 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:26.724 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.724 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.724 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:29:26.725 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.446324+0000", 2024-08-19T14:29:26.725 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4899999999999995e-07 2024-08-19T14:29:26.725 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.725 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.725 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:29:26.725 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.446356+0000", 2024-08-19T14:29:26.725 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1934999999999998e-05 2024-08-19T14:29:26.725 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.725 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.725 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:29:26.725 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.446368+0000", 2024-08-19T14:29:26.725 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1874e-05 2024-08-19T14:29:26.725 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.725 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.726 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:29:26.726 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.446374+0000", 2024-08-19T14:29:26.726 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8880000000000002e-06 2024-08-19T14:29:26.726 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.726 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.726 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:29:26.726 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.446934+0000", 2024-08-19T14:29:26.726 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000560237 2024-08-19T14:29:26.726 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.726 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.726 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:29:26.726 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.446942+0000", 2024-08-19T14:29:26.726 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.4509999999999992e-06 2024-08-19T14:29:26.727 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.727 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.727 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:29:26.727 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.693544+0000", 2024-08-19T14:29:26.727 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.246602191 2024-08-19T14:29:26.727 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.727 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.727 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:29:26.727 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.693660+0000", 2024-08-19T14:29:26.727 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00011593 2024-08-19T14:29:26.727 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.727 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.727 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,5", 2024-08-19T14:29:26.727 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.693774+0000", 2024-08-19T14:29:26.727 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00011357999999999999 2024-08-19T14:29:26.728 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.728 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.728 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:29:26.728 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.696063+0000", 2024-08-19T14:29:26.728 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0022885800000000001 2024-08-19T14:29:26.728 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.728 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.728 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:26.728 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.696305+0000", 2024-08-19T14:29:26.728 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00024199200000000001 2024-08-19T14:29:26.728 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.728 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.728 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:26.728 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.714831+0000", 2024-08-19T14:29:26.728 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.018526885999999999 2024-08-19T14:29:26.729 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.729 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.729 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:29:26.729 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.714883+0000", 2024-08-19T14:29:26.729 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1433e-05 2024-08-19T14:29:26.729 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.729 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.729 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:29:26.729 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.717161+0000", 2024-08-19T14:29:26.729 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.002278225 2024-08-19T14:29:26.729 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:26.729 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:29:26.729 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:26.729 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.730 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.730 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26191.0:76378 6.e 6:78c77635:::benchmark_data_smithi045_97937_object76377:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e527)", 2024-08-19T14:29:26.730 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:29:17.871362+0000", 2024-08-19T14:29:26.730 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 8.7625477899999993, 2024-08-19T14:29:26.730 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.45114312600000001, 2024-08-19T14:29:26.730 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:29:26.730 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:29:26.730 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:29:26.730 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:29:26.730 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26191", 2024-08-19T14:29:26.730 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:29:26.730 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 76378 2024-08-19T14:29:26.731 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.731 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:29:26.731 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.731 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:29:26.731 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.871362+0000", 2024-08-19T14:29:26.731 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:26.731 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.731 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.731 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:29:26.731 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.871362+0000", 2024-08-19T14:29:26.732 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:26.732 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.732 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.732 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:29:26.732 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.871364+0000", 2024-08-19T14:29:26.732 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.891e-06 2024-08-19T14:29:26.732 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.732 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.732 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:29:26.732 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.871416+0000", 2024-08-19T14:29:26.732 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1847e-05 2024-08-19T14:29:26.732 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.732 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.732 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:29:26.733 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.871431+0000", 2024-08-19T14:29:26.733 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.465e-05 2024-08-19T14:29:26.733 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.733 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.733 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:29:26.733 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.871438+0000", 2024-08-19T14:29:26.733 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3760000000000002e-06 2024-08-19T14:29:26.733 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.733 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.733 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:29:26.733 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.880941+0000", 2024-08-19T14:29:26.733 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0095025909999999995 2024-08-19T14:29:26.733 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.733 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.733 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:29:26.734 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.881111+0000", 2024-08-19T14:29:26.734 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00017021000000000001 2024-08-19T14:29:26.734 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.734 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.734 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 1,2", 2024-08-19T14:29:26.734 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.881293+0000", 2024-08-19T14:29:26.734 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00018233600000000001 2024-08-19T14:29:26.734 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.734 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.734 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:26.734 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.893200+0000", 2024-08-19T14:29:26.734 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011907188000000001 2024-08-19T14:29:26.734 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.734 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.735 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:29:26.735 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.895909+0000", 2024-08-19T14:29:26.735 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.002708462 2024-08-19T14:29:26.735 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.735 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.735 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:26.735 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.318493+0000", 2024-08-19T14:29:26.735 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.422583977 2024-08-19T14:29:26.735 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.735 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.735 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:29:26.735 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.318556+0000", 2024-08-19T14:29:26.735 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3541999999999996e-05 2024-08-19T14:29:26.735 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.736 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.736 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:29:26.736 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.322505+0000", 2024-08-19T14:29:26.736 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0039490560000000003 2024-08-19T14:29:26.736 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:26.736 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:29:26.736 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:26.736 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.736 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.736 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26191.0:76387 6.e 6:73030306:::benchmark_data_smithi045_97937_object76386:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e527)", 2024-08-19T14:29:26.736 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:29:17.889174+0000", 2024-08-19T14:29:26.736 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 8.7447359700000007, 2024-08-19T14:29:26.736 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.43627335299999997, 2024-08-19T14:29:26.736 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:29:26.736 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:29:26.737 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:29:26.737 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:29:26.737 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26191", 2024-08-19T14:29:26.737 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:29:26.737 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 76387 2024-08-19T14:29:26.737 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.737 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:29:26.737 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.737 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:29:26.737 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.889174+0000", 2024-08-19T14:29:26.737 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:26.737 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.737 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.738 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:29:26.738 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.889174+0000", 2024-08-19T14:29:26.738 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:26.738 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.738 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.738 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:29:26.738 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.889176+0000", 2024-08-19T14:29:26.738 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.533e-06 2024-08-19T14:29:26.738 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.738 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.738 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:29:26.738 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.889227+0000", 2024-08-19T14:29:26.738 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1677e-05 2024-08-19T14:29:26.738 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.739 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.739 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:29:26.739 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.889239+0000", 2024-08-19T14:29:26.739 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.223e-05 2024-08-19T14:29:26.739 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.739 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.739 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:29:26.739 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.889247+0000", 2024-08-19T14:29:26.739 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2980000000000003e-06 2024-08-19T14:29:26.739 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.739 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.739 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:29:26.739 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.893233+0000", 2024-08-19T14:29:26.739 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0039860779999999997 2024-08-19T14:29:26.740 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.740 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.740 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:29:26.740 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.893388+0000", 2024-08-19T14:29:26.740 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000155501 2024-08-19T14:29:26.740 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.740 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.740 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 1,2", 2024-08-19T14:29:26.740 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.893560+0000", 2024-08-19T14:29:26.740 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00017121700000000001 2024-08-19T14:29:26.740 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.740 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.740 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:29:26.741 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.896062+0000", 2024-08-19T14:29:26.741 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0025023749999999998 2024-08-19T14:29:26.741 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.741 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.741 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:26.741 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.912814+0000", 2024-08-19T14:29:26.741 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.016751920999999999 2024-08-19T14:29:26.741 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.741 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.741 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:26.741 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.322546+0000", 2024-08-19T14:29:26.741 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.40973219 2024-08-19T14:29:26.741 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.741 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.741 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:29:26.742 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.322586+0000", 2024-08-19T14:29:26.742 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0370999999999997e-05 2024-08-19T14:29:26.742 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.742 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.742 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:29:26.742 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.325447+0000", 2024-08-19T14:29:26.742 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0028609619999999999 2024-08-19T14:29:26.742 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:26.742 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:29:26.742 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:26.742 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.742 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.742 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26191.0:76389 6.e 6:771c5bb8:::benchmark_data_smithi045_97937_object76388:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e527)", 2024-08-19T14:29:26.743 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:29:17.898506+0000", 2024-08-19T14:29:26.743 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 8.7354037489999996, 2024-08-19T14:29:26.743 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.43093812399999998, 2024-08-19T14:29:26.743 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:29:26.743 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:29:26.743 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:29:26.743 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:29:26.743 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26191", 2024-08-19T14:29:26.743 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:29:26.743 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 76389 2024-08-19T14:29:26.743 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.743 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:29:26.743 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.743 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:29:26.744 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.898506+0000", 2024-08-19T14:29:26.744 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:26.744 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.744 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.744 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:29:26.744 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.898506+0000", 2024-08-19T14:29:26.744 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:26.744 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.744 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.744 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:29:26.744 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.898507+0000", 2024-08-19T14:29:26.744 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.251e-06 2024-08-19T14:29:26.744 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.744 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.745 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:29:26.745 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.898538+0000", 2024-08-19T14:29:26.745 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0233999999999999e-05 2024-08-19T14:29:26.745 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.745 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.745 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:29:26.745 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.898549+0000", 2024-08-19T14:29:26.745 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1674999999999999e-05 2024-08-19T14:29:26.745 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.745 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.745 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:29:26.745 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.898555+0000", 2024-08-19T14:29:26.745 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6160000000000001e-06 2024-08-19T14:29:26.745 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.745 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.746 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:29:26.746 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.898599+0000", 2024-08-19T14:29:26.746 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3967000000000001e-05 2024-08-19T14:29:26.746 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.746 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.746 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:29:26.746 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.898713+0000", 2024-08-19T14:29:26.746 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000114004 2024-08-19T14:29:26.746 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.746 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.746 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 1,2", 2024-08-19T14:29:26.746 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.898895+0000", 2024-08-19T14:29:26.746 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000182237 2024-08-19T14:29:26.746 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.747 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.747 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:29:26.747 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.901611+0000", 2024-08-19T14:29:26.747 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0027155510000000001 2024-08-19T14:29:26.747 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.747 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.747 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:26.747 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.912879+0000", 2024-08-19T14:29:26.747 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011268174000000001 2024-08-19T14:29:26.747 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.747 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.747 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:26.747 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.325505+0000", 2024-08-19T14:29:26.747 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.41262576299999998 2024-08-19T14:29:26.748 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.748 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.748 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:29:26.748 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.325568+0000", 2024-08-19T14:29:26.748 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3264999999999998e-05 2024-08-19T14:29:26.748 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.748 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.748 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:29:26.748 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.329444+0000", 2024-08-19T14:29:26.748 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0038763869999999998 2024-08-19T14:29:26.748 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:26.748 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:29:26.748 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:26.748 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.749 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.749 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26191.0:76394 6.e 6:7e16e1b0:::benchmark_data_smithi045_97937_object76393:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e527)", 2024-08-19T14:29:26.749 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:29:17.910225+0000", 2024-08-19T14:29:26.749 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 8.7236849840000001, 2024-08-19T14:29:26.749 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.42283432900000001, 2024-08-19T14:29:26.749 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:29:26.749 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:29:26.749 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:29:26.749 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:29:26.749 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26191", 2024-08-19T14:29:26.749 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:29:26.749 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 76394 2024-08-19T14:29:26.749 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.749 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:29:26.750 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.750 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:29:26.750 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.910225+0000", 2024-08-19T14:29:26.750 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:26.750 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.750 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.750 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:29:26.750 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.910225+0000", 2024-08-19T14:29:26.750 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:26.750 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.750 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.750 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:29:26.750 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.910230+0000", 2024-08-19T14:29:26.751 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7589999999999997e-06 2024-08-19T14:29:26.751 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.751 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.751 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:29:26.751 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.910271+0000", 2024-08-19T14:29:26.752 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1482e-05 2024-08-19T14:29:26.752 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.752 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.752 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:29:26.752 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.910290+0000", 2024-08-19T14:29:26.752 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8986000000000001e-05 2024-08-19T14:29:26.753 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.753 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.753 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:29:26.753 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.910298+0000", 2024-08-19T14:29:26.753 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5979999999999999e-06 2024-08-19T14:29:26.753 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.753 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.753 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:29:26.753 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.912937+0000", 2024-08-19T14:29:26.753 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.002639078 2024-08-19T14:29:26.753 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.753 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.753 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:29:26.753 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.913081+0000", 2024-08-19T14:29:26.754 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014420399999999999 2024-08-19T14:29:26.754 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.754 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.754 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 1,2", 2024-08-19T14:29:26.754 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.913237+0000", 2024-08-19T14:29:26.754 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00015604599999999999 2024-08-19T14:29:26.754 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.754 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.754 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:29:26.754 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.924898+0000", 2024-08-19T14:29:26.754 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011660831 2024-08-19T14:29:26.754 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.754 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.755 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:26.755 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.924947+0000", 2024-08-19T14:29:26.755 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9063000000000001e-05 2024-08-19T14:29:26.755 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.755 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.755 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:26.755 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.329507+0000", 2024-08-19T14:29:26.755 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.40456007399999999 2024-08-19T14:29:26.755 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.755 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.755 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:29:26.755 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.329568+0000", 2024-08-19T14:29:26.755 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1334000000000006e-05 2024-08-19T14:29:26.756 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.756 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.756 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:29:26.756 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.333059+0000", 2024-08-19T14:29:26.756 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0034908740000000001 2024-08-19T14:29:26.756 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:26.756 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:29:26.756 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:26.756 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.756 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.756 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26191.0:76413 6.e 6:7e0cb892:::benchmark_data_smithi045_97937_object76412:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e527)", 2024-08-19T14:29:26.756 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:29:17.945624+0000", 2024-08-19T14:29:26.756 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 8.6882863179999994, 2024-08-19T14:29:26.756 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.39115324499999998, 2024-08-19T14:29:26.756 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:29:26.757 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:29:26.757 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:29:26.757 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:29:26.757 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26191", 2024-08-19T14:29:26.757 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:29:26.757 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 76413 2024-08-19T14:29:26.757 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.757 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:29:26.757 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.757 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:29:26.757 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.945624+0000", 2024-08-19T14:29:26.757 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:26.757 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.757 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.758 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:29:26.758 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.945624+0000", 2024-08-19T14:29:26.758 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:26.758 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.758 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.758 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:29:26.758 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.945625+0000", 2024-08-19T14:29:26.758 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7710000000000001e-06 2024-08-19T14:29:26.758 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.758 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.758 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:29:26.758 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.945669+0000", 2024-08-19T14:29:26.758 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3237e-05 2024-08-19T14:29:26.758 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.759 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.759 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:29:26.759 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.945682+0000", 2024-08-19T14:29:26.759 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3033999999999999e-05 2024-08-19T14:29:26.759 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.759 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.759 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:29:26.759 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.945690+0000", 2024-08-19T14:29:26.759 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.5769999999999994e-06 2024-08-19T14:29:26.759 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.759 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.759 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:29:26.759 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.945808+0000", 2024-08-19T14:29:26.759 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000117803 2024-08-19T14:29:26.760 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.760 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.760 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:29:26.760 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.946009+0000", 2024-08-19T14:29:26.760 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00020067899999999999 2024-08-19T14:29:26.760 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.760 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.760 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 1,2", 2024-08-19T14:29:26.760 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.946188+0000", 2024-08-19T14:29:26.760 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000179657 2024-08-19T14:29:26.760 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.760 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.760 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:29:26.760 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.949682+0000", 2024-08-19T14:29:26.761 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0034936250000000002 2024-08-19T14:29:26.761 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.761 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.761 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:26.761 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.955654+0000", 2024-08-19T14:29:26.761 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0059714590000000001 2024-08-19T14:29:26.761 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.761 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.761 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:26.761 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.333114+0000", 2024-08-19T14:29:26.761 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.37746024700000003 2024-08-19T14:29:26.761 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.761 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.761 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:29:26.762 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.333155+0000", 2024-08-19T14:29:26.762 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1365000000000001e-05 2024-08-19T14:29:26.762 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.762 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.762 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:29:26.762 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.336777+0000", 2024-08-19T14:29:26.762 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0036217910000000001 2024-08-19T14:29:26.762 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:26.762 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:29:26.762 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:26.762 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.762 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.762 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26191.0:76434 6.e 6:7ca6e209:::benchmark_data_smithi045_97937_object76433:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e527)", 2024-08-19T14:29:26.762 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:29:18.006573+0000", 2024-08-19T14:29:26.763 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 8.6273365829999999, 2024-08-19T14:29:26.763 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.33460235999999999, 2024-08-19T14:29:26.763 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:29:26.763 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:29:26.763 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:29:26.763 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:29:26.763 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26191", 2024-08-19T14:29:26.763 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:29:26.763 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 76434 2024-08-19T14:29:26.763 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.763 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:29:26.763 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.763 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:29:26.763 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.006573+0000", 2024-08-19T14:29:26.763 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:26.763 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.764 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.764 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:29:26.764 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.006573+0000", 2024-08-19T14:29:26.764 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:26.764 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.764 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.764 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:29:26.764 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.006575+0000", 2024-08-19T14:29:26.764 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2529999999999999e-06 2024-08-19T14:29:26.764 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.764 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.764 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:29:26.764 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.006633+0000", 2024-08-19T14:29:26.764 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8835000000000002e-05 2024-08-19T14:29:26.765 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.765 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.765 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:29:26.765 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.006644+0000", 2024-08-19T14:29:26.765 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0866999999999999e-05 2024-08-19T14:29:26.765 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.765 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.765 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:29:26.765 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.006650+0000", 2024-08-19T14:29:26.765 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1819999999999997e-06 2024-08-19T14:29:26.765 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.765 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.765 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:29:26.765 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.006705+0000", 2024-08-19T14:29:26.766 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5167000000000002e-05 2024-08-19T14:29:26.766 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.766 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.766 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:29:26.766 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.006871+0000", 2024-08-19T14:29:26.766 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000166663 2024-08-19T14:29:26.766 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.766 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.766 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 1,2", 2024-08-19T14:29:26.766 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.007065+0000", 2024-08-19T14:29:26.766 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00019317299999999999 2024-08-19T14:29:26.766 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.766 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.767 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:29:26.767 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.009738+0000", 2024-08-19T14:29:26.767 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0026729639999999999 2024-08-19T14:29:26.767 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.767 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.767 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:26.767 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.018726+0000", 2024-08-19T14:29:26.767 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0089886069999999992 2024-08-19T14:29:26.767 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.767 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.767 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:26.767 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.337011+0000", 2024-08-19T14:29:26.767 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.31828483499999999 2024-08-19T14:29:26.768 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.768 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.768 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:29:26.768 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.337109+0000", 2024-08-19T14:29:26.768 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.8092999999999997e-05 2024-08-19T14:29:26.768 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:26.768 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:26.768 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:29:26.768 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.341176+0000", 2024-08-19T14:29:26.768 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0040667209999999997 2024-08-19T14:29:26.768 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:26.768 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:29:26.768 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:26.768 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:26.769 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:29:26.769 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:29:26.769 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-19T14:29:26.770 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-08-19T14:29:26.770 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.10 2024-08-19T14:29:26.770 INFO:tasks.thrashosds.thrasher:remote for osd 10 is ubuntu@smithi045.front.sepia.ceph.com 2024-08-19T14:29:26.770 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell -n osd.10 --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-10 --journal-path /var/lib/ceph/osd/ceph-10/journal --op list-pgs 2024-08-19T14:29:26.773 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:26.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:26.774 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:29:26.876 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:29:26.906 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:26.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:26.908 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:29:26.998 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/osd.10/config 2024-08-19T14:29:27.008 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:29:27.053 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:27.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:27.054 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:29:27.156 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:29:27.185 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:27.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:27.186 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:29:27.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:26 smithi032 ceph-mon[20004]: osdmap e532: 12 total, 11 up, 8 in 2024-08-19T14:29:27.269 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:26 smithi045 ceph-mon[20143]: osdmap e532: 12 total, 11 up, 8 in 2024-08-19T14:29:27.287 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:29:27.310 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:27.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:27.312 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:29:27.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:26 smithi031 ceph-mon[16336]: osdmap e532: 12 total, 11 up, 8 in 2024-08-19T14:29:27.412 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:29:27.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:27.446 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:27.446 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:29:27.510 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 49 16 82307 82291 104.954 42 0.0174467 0.00952217 2024-08-19T14:29:27.548 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:29:27.582 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:27.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:27.583 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:29:27.684 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:29:27.713 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:27.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:27.714 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:29:27.815 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:29:27.852 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:27.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:27.853 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:29:27.939 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:29:27.954 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:29:28.011 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:28.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:28.012 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:29:28.113 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:29:28.140 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:28.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:28.141 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:29:28.242 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:29:28.246 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:27 smithi045 ceph-mon[20143]: pgmap v1218: 47 pgs: 8 active+recovering+undersized+remapped, 39 active+clean; 4.9 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 33 MiB/s wr, 534 op/s; 36708/239427 objects misplaced (15.332%); 13 MiB/s, 211 objects/s recovering 2024-08-19T14:29:28.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:27 smithi032 ceph-mon[20004]: pgmap v1218: 47 pgs: 8 active+recovering+undersized+remapped, 39 active+clean; 4.9 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 33 MiB/s wr, 534 op/s; 36708/239427 objects misplaced (15.332%); 13 MiB/s, 211 objects/s recovering 2024-08-19T14:29:28.265 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:28.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:28.266 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:29:28.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:27 smithi031 ceph-mon[16336]: pgmap v1218: 47 pgs: 8 active+recovering+undersized+remapped, 39 active+clean; 4.9 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 33 MiB/s wr, 534 op/s; 36708/239427 objects misplaced (15.332%); 13 MiB/s, 211 objects/s recovering 2024-08-19T14:29:28.366 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:29:28.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:28.391 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:28.392 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:29:28.493 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:29:28.510 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 50 16 82969 82953 103.683 41.375 0.0484843 0.00963711 2024-08-19T14:29:28.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:28.515 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:28.515 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:29:28.616 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:29:28.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:28.650 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:28.650 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:29:28.751 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:29:28.781 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:28.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:28.782 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:29:28.883 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:29:28.904 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:28.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:28.905 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:29:29.006 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:29:29.028 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:29.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:29.029 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:29:29.095 INFO:teuthology.orchestra.run.smithi045.stderr:Creating an OSD daemon form without an OSD FSID value 2024-08-19T14:29:29.129 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:29:29.152 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:29.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:29.153 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:29:29.254 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:29:29.283 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:29.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:29.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:29:29.386 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:29:29.415 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:29.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:29.417 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:29:29.510 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 51 16 83639 83623 102.471 41.875 0.0515063 0.00975243 2024-08-19T14:29:29.513 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-19T14:29:29.513 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-19T14:29:29.517 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:29:29.539 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:29.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:29.540 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:29:29.641 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:29:29.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:29.683 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:29.683 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:29:29.784 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:29:29.827 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:29.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:29.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:29:29.929 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:29:29.956 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:29.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:29.957 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:29:30.058 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:29:30.080 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:30.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:30.081 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:29:30.182 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:29:30.204 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:30.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:30.205 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:29:30.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:29 smithi032 ceph-mon[20004]: pgmap v1219: 47 pgs: 8 active+recovering+undersized+remapped, 39 active+clean; 4.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 39 MiB/s wr, 620 op/s; 36386/241227 objects misplaced (15.084%); 15 MiB/s, 246 objects/s recovering 2024-08-19T14:29:30.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:29 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:29:30.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:29 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:29:30.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:29 smithi045 ceph-mon[20143]: pgmap v1219: 47 pgs: 8 active+recovering+undersized+remapped, 39 active+clean; 4.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 39 MiB/s wr, 620 op/s; 36386/241227 objects misplaced (15.084%); 15 MiB/s, 246 objects/s recovering 2024-08-19T14:29:30.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:29 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:29:30.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:29 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:29:30.306 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:29:30.335 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:30.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:30.336 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:29:30.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:29 smithi031 ceph-mon[16336]: pgmap v1219: 47 pgs: 8 active+recovering+undersized+remapped, 39 active+clean; 4.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 39 MiB/s wr, 620 op/s; 36386/241227 objects misplaced (15.084%); 15 MiB/s, 246 objects/s recovering 2024-08-19T14:29:30.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:29 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:29:30.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:29 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:29:30.437 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:29:30.469 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:30.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:30.470 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:29:30.510 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 52 15 84279 84264 101.27 40.0625 0.0234079 0.00986947 2024-08-19T14:29:30.571 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:29:30.594 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:30.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:30.595 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:29:30.687 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-08-19T14:29:30.695 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:29:30.734 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:30.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:30.735 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:29:30.836 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:29:30.878 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:30.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:30.879 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:29:30.980 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:29:31.025 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is unset 2024-08-19T14:29:31.031 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:31.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:31.032 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:29:31.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:31 smithi045 ceph-mon[20143]: osdmap e533: 12 total, 11 up, 8 in 2024-08-19T14:29:31.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:31 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3619757225' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:29:31.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:31 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:29:31.083 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:29:31.133 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:29:31.143 INFO:tasks.thrashosds.thrasher:No PGs found for osd.10 2024-08-19T14:29:31.171 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:31.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:31.172 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:29:31.273 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:29:31.323 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:31.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:31.324 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:29:31.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:31 smithi031 ceph-mon[16336]: osdmap e533: 12 total, 11 up, 8 in 2024-08-19T14:29:31.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:31 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3619757225' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:29:31.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:31 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:29:31.368 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:31 smithi032 ceph-mon[20004]: osdmap e533: 12 total, 11 up, 8 in 2024-08-19T14:29:31.368 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:31 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3619757225' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:29:31.368 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:31 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:29:31.425 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:29:31.448 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:31.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:31.449 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:29:31.510 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 53 16 84934 84918 100.131 40.875 0.0164295 0.00997899 2024-08-19T14:29:31.514 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:29:31.550 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:29:31.600 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:31.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:31.601 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:29:31.702 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:29:31.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:31.726 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:31.727 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:29:31.828 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:29:31.849 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:31.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:31.850 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:29:31.950 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:29:31.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:31.984 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:31.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:29:31.996 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:29:32.085 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:29:32.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:32 smithi031 ceph-mon[16336]: pgmap v1221: 47 pgs: 8 active+recovering+undersized+remapped, 39 active+clean; 5.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 33 MiB/s wr, 529 op/s; 35770/248427 objects misplaced (14.399%); 12 MiB/s, 192 objects/s recovering 2024-08-19T14:29:32.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:32 smithi031 ceph-mon[16336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:29:32.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:32 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:29:32.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:32 smithi031 ceph-mon[16336]: osdmap e534: 12 total, 11 up, 8 in 2024-08-19T14:29:32.133 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:32.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:32.134 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:29:32.235 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:29:32.279 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:32.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:32.280 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:29:32.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:32 smithi045 ceph-mon[20143]: pgmap v1221: 47 pgs: 8 active+recovering+undersized+remapped, 39 active+clean; 5.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 33 MiB/s wr, 529 op/s; 35770/248427 objects misplaced (14.399%); 12 MiB/s, 192 objects/s recovering 2024-08-19T14:29:32.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:32 smithi045 ceph-mon[20143]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:29:32.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:32 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:29:32.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:32 smithi045 ceph-mon[20143]: osdmap e534: 12 total, 11 up, 8 in 2024-08-19T14:29:32.381 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:29:32.403 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:32.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:32.404 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:29:32.452 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:32 smithi032 ceph-mon[20004]: pgmap v1221: 47 pgs: 8 active+recovering+undersized+remapped, 39 active+clean; 5.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 33 MiB/s wr, 529 op/s; 35770/248427 objects misplaced (14.399%); 12 MiB/s, 192 objects/s recovering 2024-08-19T14:29:32.452 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:32 smithi032 ceph-mon[20004]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:29:32.453 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:32 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:29:32.453 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:32 smithi032 ceph-mon[20004]: osdmap e534: 12 total, 11 up, 8 in 2024-08-19T14:29:32.504 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:29:32.510 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 54 16 85542 85526 98.9802 38 0.0101433 0.0100936 2024-08-19T14:29:32.547 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:32.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:32.548 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:29:32.649 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:29:32.671 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:32.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:32.672 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:29:32.773 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:29:32.794 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:32.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:32.795 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:29:32.897 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:29:32.918 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:32.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:32.919 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:29:33.020 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:29:33.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:33.043 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:33.044 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:29:33.145 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:29:33.180 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:33.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:33.181 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:29:33.261 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:29:33.261 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-08-19T14:29:33.261 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-08-19T14:29:33.261 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:29:33.281 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:29:33.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:33 smithi045 ceph-mon[20143]: pgmap v1223: 47 pgs: 8 active+recovering+undersized+remapped, 39 active+clean; 5.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 33 MiB/s wr, 524 op/s; 35568/249027 objects misplaced (14.283%); 12 MiB/s, 186 objects/s recovering 2024-08-19T14:29:33.304 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:33.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:33.305 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:29:33.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:33 smithi031 ceph-mon[16336]: pgmap v1223: 47 pgs: 8 active+recovering+undersized+remapped, 39 active+clean; 5.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 33 MiB/s wr, 524 op/s; 35568/249027 objects misplaced (14.283%); 12 MiB/s, 186 objects/s recovering 2024-08-19T14:29:33.406 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:29:33.443 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:33.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:33.444 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:29:33.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:33 smithi032 ceph-mon[20004]: pgmap v1223: 47 pgs: 8 active+recovering+undersized+remapped, 39 active+clean; 5.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 33 MiB/s wr, 524 op/s; 35568/249027 objects misplaced (14.283%); 12 MiB/s, 186 objects/s recovering 2024-08-19T14:29:33.510 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 55 16 86218 86202 97.9486 42.25 0.00487121 0.0102042 2024-08-19T14:29:33.545 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:29:33.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:33.578 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:33.578 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:29:33.679 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:29:33.701 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:33.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:33.702 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:29:33.803 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:29:33.830 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:29:33.831 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:29:33.837 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:33.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:33.839 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:29:33.853 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-19T14:29:33.864 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:29:33.864 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:29:33.911 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:29:33.911 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:29:33.940 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:29:33.961 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:29:33.961 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:29:33.965 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:33.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:33.966 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:29:34.030 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:29:34.031 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:29:34.066 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:29:34.067 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:29:34.067 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:29:34.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:34.092 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:34.092 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:29:34.101 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:29:34.101 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:29:34.137 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:29:34.137 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:29:34.179 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:29:34.179 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:29:34.179 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:34 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2602373256' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:29:34.193 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:29:34.211 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:29:34.212 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:29:34.217 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-19T14:29:34.219 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:34.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:34.221 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:29:34.252 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:29:34.252 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:29:34.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:34 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2602373256' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:29:34.321 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:29:34.362 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:34.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:34.364 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:29:34.465 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:29:34.486 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:34.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:34.487 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:29:34.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:34 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2602373256' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:29:34.511 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 56 16 86813 86797 96.8636 37.1875 0.00384094 0.0103163 2024-08-19T14:29:34.588 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:29:34.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:34.634 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:34.634 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:29:34.736 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:29:34.757 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:34.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:34.758 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:29:34.859 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:29:34.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:34.899 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:34.900 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:29:35.001 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:29:35.036 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:35.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:35.037 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:29:35.086 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-19T14:29:35.087 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-19T14:29:35.099 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:29:35.138 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:29:35.163 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:35.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:35.165 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:29:35.266 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:29:35.288 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:35.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:35.289 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:29:35.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:35 smithi031 ceph-mon[16336]: pgmap v1224: 47 pgs: 8 active+recovering+undersized+remapped, 39 active+clean; 5.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 49 MiB/s wr, 778 op/s; 34661/258114 objects misplaced (13.429%); 16 MiB/s, 255 objects/s recovering 2024-08-19T14:29:35.389 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:29:35.417 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:35.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:35.419 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:29:35.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:35 smithi032 ceph-mon[20004]: pgmap v1224: 47 pgs: 8 active+recovering+undersized+remapped, 39 active+clean; 5.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 49 MiB/s wr, 778 op/s; 34661/258114 objects misplaced (13.429%); 16 MiB/s, 255 objects/s recovering 2024-08-19T14:29:35.511 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 57 16 87381 87365 95.787 35.5 0.0227494 0.0104345 2024-08-19T14:29:35.520 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:29:35.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:35 smithi045 ceph-mon[20143]: pgmap v1224: 47 pgs: 8 active+recovering+undersized+remapped, 39 active+clean; 5.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 49 MiB/s wr, 778 op/s; 34661/258114 objects misplaced (13.429%); 16 MiB/s, 255 objects/s recovering 2024-08-19T14:29:35.549 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:35.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:35.550 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:29:35.651 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:29:35.692 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:35.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:35.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:29:35.794 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:29:35.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:35.824 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:35.824 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:29:35.926 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:29:35.957 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:35.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:35.958 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:29:36.059 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:29:36.060 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is unset 2024-08-19T14:29:36.088 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:36.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:36.089 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:29:36.144 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 7, 8, 2] out_osds: [10, 6, 9, 11] dead_osds: [10] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11] 2024-08-19T14:29:36.145 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-19T14:29:36.145 INFO:tasks.thrashosds.thrasher:inject_pause on osd.9 2024-08-19T14:29:36.145 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-08-19T14:29:36.145 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-08-19T14:29:36.145 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.9 config set filestore_inject_stall 3 2024-08-19T14:29:36.189 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:29:36.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:36.220 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:36.220 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:29:36.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:36 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3305713784' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:29:36.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:36 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3305713784' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:29:36.321 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:29:36.351 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:36.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:36.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:29:36.378 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:36 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3305713784' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:29:36.424 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:29:36.453 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:29:36.494 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:36.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:36.495 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:29:36.511 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 58 16 87943 87927 94.741 35.125 0.0363919 0.0105496 2024-08-19T14:29:36.597 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:29:36.629 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:36.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:36.630 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:29:36.731 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:29:36.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:36.762 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:36.762 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:29:36.863 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:29:36.887 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:36.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:36.888 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:29:36.989 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:29:37.021 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:37.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:37.022 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:29:37.059 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:29:37.059 INFO:teuthology.orchestra.run.smithi045.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-08-19T14:29:37.059 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:29:37.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:37 smithi045 ceph-mon[20143]: pgmap v1225: 47 pgs: 8 active+recovering+undersized+remapped, 39 active+clean; 5.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 46 MiB/s wr, 730 op/s; 34449/258759 objects misplaced (13.313%); 15 MiB/s, 242 objects/s recovering 2024-08-19T14:29:37.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:37 smithi045 ceph-mon[20143]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:29:37.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:37 smithi045 ceph-mon[20143]: Cluster is now healthy 2024-08-19T14:29:37.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:37 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3305713784' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:29:37.059 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:37 smithi045 ceph-mon[20143]: osdmap e535: 12 total, 11 up, 8 in 2024-08-19T14:29:37.088 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:29:37.122 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:29:37.148 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:37.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:37.149 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:29:37.250 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:29:37.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:37 smithi032 ceph-mon[20004]: pgmap v1225: 47 pgs: 8 active+recovering+undersized+remapped, 39 active+clean; 5.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 46 MiB/s wr, 730 op/s; 34449/258759 objects misplaced (13.313%); 15 MiB/s, 242 objects/s recovering 2024-08-19T14:29:37.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:37 smithi032 ceph-mon[20004]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:29:37.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:37 smithi032 ceph-mon[20004]: Cluster is now healthy 2024-08-19T14:29:37.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:37 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3305713784' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:29:37.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:37 smithi032 ceph-mon[20004]: osdmap e535: 12 total, 11 up, 8 in 2024-08-19T14:29:37.273 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:37.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:37.274 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:29:37.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:37 smithi031 ceph-mon[16336]: pgmap v1225: 47 pgs: 8 active+recovering+undersized+remapped, 39 active+clean; 5.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 46 MiB/s wr, 730 op/s; 34449/258759 objects misplaced (13.313%); 15 MiB/s, 242 objects/s recovering 2024-08-19T14:29:37.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:37 smithi031 ceph-mon[16336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:29:37.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:37 smithi031 ceph-mon[16336]: Cluster is now healthy 2024-08-19T14:29:37.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:37 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3305713784' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:29:37.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:37 smithi031 ceph-mon[16336]: osdmap e535: 12 total, 11 up, 8 in 2024-08-19T14:29:37.375 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:29:37.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:37.421 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:37.422 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:29:37.511 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 59 16 88531 88515 93.7581 36.75 0.0148333 0.0106514 2024-08-19T14:29:37.523 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:29:37.546 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:37.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:37.548 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:29:37.649 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:29:37.682 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:37.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:37.683 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:29:37.784 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:29:37.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:37.809 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:37.809 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:29:37.911 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:29:37.932 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:37.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:37.933 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:29:38.034 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:29:38.055 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:38.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:38.056 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:29:38.157 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:29:38.188 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:38.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:38.189 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:29:38.290 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:29:38.316 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:38.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:38.317 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:29:38.418 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:29:38.462 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:38.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:38.464 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:29:38.511 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:2024-08-19T14:29:38.511728+0000 min lat: 0.00119513 max lat: 2.46333 avg lat: 0.0107709 2024-08-19T14:29:38.511 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-19T14:29:38.511 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 60 16 89112 89096 92.8007 36.3125 0.0185834 0.0107709 2024-08-19T14:29:38.564 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:29:38.586 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:38.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:38.587 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:29:38.688 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:29:38.729 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:38.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:38.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:29:38.831 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:29:38.853 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:38.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:38.854 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:29:38.955 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:29:38.993 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:38.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:38.994 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:29:39.095 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:29:39.130 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:39.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:39.131 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:29:39.232 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:29:39.276 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:39.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:39.277 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:29:39.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:39 smithi031 ceph-mon[16336]: pgmap v1227: 47 pgs: 8 active+recovering+undersized+remapped, 39 active+clean; 5.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 31 MiB/s wr, 499 op/s; 34092/260427 objects misplaced (13.091%); 13 MiB/s, 209 objects/s recovering 2024-08-19T14:29:39.378 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:29:39.400 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:39.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:39.401 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:29:39.502 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:29:39.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:39 smithi032 ceph-mon[20004]: pgmap v1227: 47 pgs: 8 active+recovering+undersized+remapped, 39 active+clean; 5.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 31 MiB/s wr, 499 op/s; 34092/260427 objects misplaced (13.091%); 13 MiB/s, 209 objects/s recovering 2024-08-19T14:29:39.511 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 61 16 89780 89764 91.9637 41.75 0.0386354 0.0108674 2024-08-19T14:29:39.529 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:39.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:39.530 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:29:39.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:39 smithi045 ceph-mon[20143]: pgmap v1227: 47 pgs: 8 active+recovering+undersized+remapped, 39 active+clean; 5.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 31 MiB/s wr, 499 op/s; 34092/260427 objects misplaced (13.091%); 13 MiB/s, 209 objects/s recovering 2024-08-19T14:29:39.631 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:29:39.663 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:39.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:39.665 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:29:39.765 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:29:39.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:39.788 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:39.788 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:29:39.889 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:29:39.920 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:39.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:39.921 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:29:40.022 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:29:40.043 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:40.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:40.045 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:29:40.146 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:29:40.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:40.177 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:40.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:29:40.239 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-08-19T14:29:40.278 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:29:40.299 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:40.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:40.300 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:29:40.401 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:29:40.427 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:40.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:40.428 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:29:40.511 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 62 16 90357 90341 91.062 36.0625 0.0183553 0.0109454 2024-08-19T14:29:40.529 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:29:40.550 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:40.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:40.551 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:29:40.591 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:29:40.652 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:29:40.674 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:40.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:40.675 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:29:40.776 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:29:40.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:40.797 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:40.798 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:29:40.899 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:29:40.943 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:40.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:40.944 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:29:41.045 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:29:41.084 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:41.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:41.086 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:29:41.160 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:29:41.160 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:29:41.186 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:29:41.208 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:41.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:41.209 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:29:41.310 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:29:41.358 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:41.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:41.359 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:29:41.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:41 smithi031 ceph-mon[16336]: pgmap v1228: 47 pgs: 8 active+recovering+undersized+remapped, 39 active+clean; 5.4 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 42 MiB/s wr, 669 op/s; 33497/267135 objects misplaced (12.539%); 14 MiB/s, 229 objects/s recovering 2024-08-19T14:29:41.461 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:29:41.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:41.499 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:41.499 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:29:41.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:41 smithi032 ceph-mon[20004]: pgmap v1228: 47 pgs: 8 active+recovering+undersized+remapped, 39 active+clean; 5.4 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 42 MiB/s wr, 669 op/s; 33497/267135 objects misplaced (12.539%); 14 MiB/s, 229 objects/s recovering 2024-08-19T14:29:41.511 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 63 16 91010 90994 90.2643 40.8125 0.0380342 0.0110716 2024-08-19T14:29:41.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:41 smithi045 ceph-mon[20143]: pgmap v1228: 47 pgs: 8 active+recovering+undersized+remapped, 39 active+clean; 5.4 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 42 MiB/s wr, 669 op/s; 33497/267135 objects misplaced (12.539%); 14 MiB/s, 229 objects/s recovering 2024-08-19T14:29:41.600 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:29:41.632 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:41.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:41.633 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:29:41.735 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:29:41.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:41.757 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:41.757 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:29:41.859 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:29:41.901 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:41.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:41.902 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:29:42.003 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:29:42.016 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:29:42.029 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:42.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:42.030 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:29:42.131 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:29:42.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:42.166 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:42.166 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:29:42.267 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:29:42.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:42.299 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:42.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:29:42.401 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:29:42.431 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:42.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:42.433 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:29:42.511 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 64 16 91649 91633 89.4779 39.9375 0.0431429 0.0111707 2024-08-19T14:29:42.533 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:29:42.561 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:42.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:42.562 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:29:42.663 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:29:42.706 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:42.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:42.707 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:29:42.742 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 7, 8, 2] out_osds: [10, 6, 9, 11] dead_osds: [10] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11] 2024-08-19T14:29:42.743 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-19T14:29:42.743 INFO:tasks.thrashosds.thrasher:Reviving osd 10 2024-08-19T14:29:42.743 INFO:tasks.cephadm.osd.10:Restarting osd.10 (starting--it wasn't running)... 2024-08-19T14:29:42.743 DEBUG:teuthology.orchestra.run.smithi045:osd.10> sudo journalctl -f -n 0 -u ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968@osd.10.service 2024-08-19T14:29:42.746 DEBUG:teuthology.orchestra.run.smithi045:> sudo systemctl start ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968@osd.10 2024-08-19T14:29:42.747 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:29:42.747 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:29:42.783 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:29:42.783 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:29:42.808 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:29:42.817 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:29:42.817 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:29:42.851 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:29:42.852 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:29:42.862 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:42.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:42.863 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:29:42.889 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:29:42.890 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:29:42.931 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:29:42.931 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:29:42.962 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:29:42.963 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:29:42.964 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:29:43.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:43.005 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:43.005 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:29:43.007 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:29:43.008 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:29:43.039 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:29:43.039 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:29:43.046 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:29:42 smithi045 systemd[1]: Starting Ceph osd.10 for ed7d1c6e-5e33-11ef-bccf-c7b262605968... 2024-08-19T14:29:43.074 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:29:43.075 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:29:43.075 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-19T14:29:43.106 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:29:43.112 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:29:43.112 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:29:43.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:43.148 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:43.148 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:29:43.158 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-08-19T14:29:43.249 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:29:43.307 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:43.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:43.308 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:29:43.351 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:43 smithi031 ceph-mon[16336]: pgmap v1229: 47 pgs: 8 active+recovering+undersized+remapped, 39 active+clean; 5.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 39 MiB/s wr, 621 op/s; 33257/267693 objects misplaced (12.424%); 14 MiB/s, 230 objects/s recovering 2024-08-19T14:29:43.352 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:43 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3172768779' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:29:43.359 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:43 smithi045 ceph-mon[20143]: pgmap v1229: 47 pgs: 8 active+recovering+undersized+remapped, 39 active+clean; 5.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 39 MiB/s wr, 621 op/s; 33257/267693 objects misplaced (12.424%); 14 MiB/s, 230 objects/s recovering 2024-08-19T14:29:43.359 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:43 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3172768779' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:29:43.359 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:29:43 smithi045 podman[101898]: 2024-08-19 14:29:42.973342847 +0000 UTC m=+0.017733121 image pull ea7a5189f30bad3aca9524788e2965d2368d7ce786df09d4edcd8c3edf9cefba quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9 2024-08-19T14:29:43.360 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:29:43 smithi045 podman[101898]: 2024-08-19 14:29:43.097738799 +0000 UTC m=+0.142129046 container create 9e801c2f0d3ac440920cb35459d83d1242da864c0294e0f597e936396454bb19 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10-activate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=squid-de09dd0, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731) 2024-08-19T14:29:43.360 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:29:43 smithi045 podman[101898]: 2024-08-19 14:29:43.359302453 +0000 UTC m=+0.403692701 container init 9e801c2f0d3ac440920cb35459d83d1242da864c0294e0f597e936396454bb19 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10-activate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, RELEASE=squid-de09dd0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, org.label-schema.license=GPLv2) 2024-08-19T14:29:43.408 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:29:43.458 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:43.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:43.459 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:29:43.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:43 smithi032 ceph-mon[20004]: pgmap v1229: 47 pgs: 8 active+recovering+undersized+remapped, 39 active+clean; 5.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 39 MiB/s wr, 621 op/s; 33257/267693 objects misplaced (12.424%); 14 MiB/s, 230 objects/s recovering 2024-08-19T14:29:43.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:43 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3172768779' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:29:43.511 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 65 16 92273 92257 88.7013 39 0.0156459 0.0112678 2024-08-19T14:29:43.526 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:29:43.560 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:29:43.603 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:43.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:43.604 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:29:43.705 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:29:43.720 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:29:43 smithi045 podman[101898]: 2024-08-19 14:29:43.363228198 +0000 UTC m=+0.407618446 container start 9e801c2f0d3ac440920cb35459d83d1242da864c0294e0f597e936396454bb19 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10-activate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, RELEASE=squid-de09dd0) 2024-08-19T14:29:43.720 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:29:43 smithi045 podman[101898]: 2024-08-19 14:29:43.484479757 +0000 UTC m=+0.528870020 container attach 9e801c2f0d3ac440920cb35459d83d1242da864c0294e0f597e936396454bb19 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10-activate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=squid-de09dd0) 2024-08-19T14:29:43.720 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:29:43 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10-activate[101913]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-19T14:29:43.720 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:29:43 smithi045 bash[101898]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-19T14:29:43.720 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:29:43 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10-activate[101913]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-19T14:29:43.720 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:29:43 smithi045 bash[101898]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-19T14:29:43.720 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:29:43 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10-activate[101913]: --> Failed to activate via raw: activate() takes 1 positional argument but 5 were given 2024-08-19T14:29:43.720 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:29:43 smithi045 bash[101898]: --> Failed to activate via raw: activate() takes 1 positional argument but 5 were given 2024-08-19T14:29:43.720 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:29:43 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10-activate[101913]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-19T14:29:43.720 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:29:43 smithi045 bash[101898]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-19T14:29:43.721 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:29:43 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10-activate[101913]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-19T14:29:43.721 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:29:43 smithi045 bash[101898]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-19T14:29:43.721 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:29:43 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10-activate[101913]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-10 2024-08-19T14:29:43.721 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:29:43 smithi045 bash[101898]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-10 2024-08-19T14:29:43.751 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:43.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:43.753 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:29:43.854 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:29:43.887 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:43.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:43.888 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:29:43.926 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-19T14:29:43.932 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-19T14:29:43.932 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-19T14:29:43.989 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:29:44.021 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:44.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:44.022 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:29:44.046 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:29:43 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10-activate[101913]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_2 --path /var/lib/ceph/osd/ceph-10 --no-mon-config 2024-08-19T14:29:44.046 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:29:43 smithi045 bash[101898]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_2 --path /var/lib/ceph/osd/ceph-10 --no-mon-config 2024-08-19T14:29:44.046 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:29:43 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10-activate[101913]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-10/block 2024-08-19T14:29:44.046 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:29:43 smithi045 bash[101898]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-10/block 2024-08-19T14:29:44.047 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:29:43 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10-activate[101913]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-10/block 2024-08-19T14:29:44.047 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:29:43 smithi045 bash[101898]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-10/block 2024-08-19T14:29:44.047 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:29:43 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10-activate[101913]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-08-19T14:29:44.047 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:29:43 smithi045 bash[101898]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-08-19T14:29:44.047 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:29:43 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10-activate[101913]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-10 2024-08-19T14:29:44.047 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:29:43 smithi045 bash[101898]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-10 2024-08-19T14:29:44.047 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:29:43 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10-activate[101913]: --> ceph-volume lvm activate successful for osd ID: 10 2024-08-19T14:29:44.047 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:29:43 smithi045 bash[101898]: --> ceph-volume lvm activate successful for osd ID: 10 2024-08-19T14:29:44.047 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:29:43 smithi045 podman[101898]: 2024-08-19 14:29:43.776330124 +0000 UTC m=+0.820720390 container died 9e801c2f0d3ac440920cb35459d83d1242da864c0294e0f597e936396454bb19 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10-activate, RELEASE=squid-de09dd0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD) 2024-08-19T14:29:44.107 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:29:44.107 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-08-19T14:29:44.107 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-08-19T14:29:44.107 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-08-19T14:29:44.107 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.107 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:22993 5.7 5:e5194b18:::benchmark_data_smithi045_85309_object22992:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e426)", 2024-08-19T14:29:44.108 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:24:55.243091+0000", 2024-08-19T14:29:44.108 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 288.86053984599999, 2024-08-19T14:29:44.108 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20799368400000001, 2024-08-19T14:29:44.108 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:29:44.108 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:29:44.108 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:29:44.108 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:29:44.108 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:29:44.108 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:29:44.108 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 22993 2024-08-19T14:29:44.109 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.109 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:29:44.109 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.109 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:29:44.109 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.243091+0000", 2024-08-19T14:29:44.109 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:44.109 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.109 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.109 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:29:44.109 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.243091+0000", 2024-08-19T14:29:44.109 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:44.110 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.110 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.110 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:29:44.110 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.243092+0000", 2024-08-19T14:29:44.110 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.9400000000000004e-07 2024-08-19T14:29:44.110 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.110 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.110 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:29:44.110 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.243122+0000", 2024-08-19T14:29:44.110 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0105000000000001e-05 2024-08-19T14:29:44.111 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.111 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.111 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:29:44.111 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.243131+0000", 2024-08-19T14:29:44.111 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.6889999999999999e-06 2024-08-19T14:29:44.111 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.111 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.111 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:29:44.111 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.243138+0000", 2024-08-19T14:29:44.111 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0339999999999999e-06 2024-08-19T14:29:44.111 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.112 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.112 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:29:44.112 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.243187+0000", 2024-08-19T14:29:44.112 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8764999999999998e-05 2024-08-19T14:29:44.112 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.112 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.112 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:29:44.112 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.243294+0000", 2024-08-19T14:29:44.112 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000106908 2024-08-19T14:29:44.112 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.113 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.113 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 1,9", 2024-08-19T14:29:44.113 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.243424+0000", 2024-08-19T14:29:44.113 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000129441 2024-08-19T14:29:44.113 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.113 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.113 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:29:44.113 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.245731+0000", 2024-08-19T14:29:44.113 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0023073210000000002 2024-08-19T14:29:44.113 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.113 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.114 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:44.114 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.246134+0000", 2024-08-19T14:29:44.114 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00040348800000000002 2024-08-19T14:29:44.114 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.114 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.114 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:44.114 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.450957+0000", 2024-08-19T14:29:44.114 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20482256099999999 2024-08-19T14:29:44.114 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.114 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.114 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:29:44.115 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.451032+0000", 2024-08-19T14:29:44.115 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4701000000000002e-05 2024-08-19T14:29:44.115 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.115 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.115 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:29:44.115 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.451084+0000", 2024-08-19T14:29:44.115 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2777e-05 2024-08-19T14:29:44.115 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:44.115 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:29:44.115 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:44.115 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.115 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.115 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:23013 5.7 5:ee7b3505:::benchmark_data_smithi045_85309_object23012:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e426)", 2024-08-19T14:29:44.116 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:24:55.249101+0000", 2024-08-19T14:29:44.116 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 288.85452930899999, 2024-08-19T14:29:44.116 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20208954800000001, 2024-08-19T14:29:44.116 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:29:44.116 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:29:44.116 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:29:44.116 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:29:44.116 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:29:44.116 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:29:44.116 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 23013 2024-08-19T14:29:44.116 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.117 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:29:44.117 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.117 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:29:44.117 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.249101+0000", 2024-08-19T14:29:44.117 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:44.117 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.117 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.117 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:29:44.117 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.249101+0000", 2024-08-19T14:29:44.117 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:44.117 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.117 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.118 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:29:44.118 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.249103+0000", 2024-08-19T14:29:44.118 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.271e-06 2024-08-19T14:29:44.118 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.118 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.118 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:29:44.118 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.249300+0000", 2024-08-19T14:29:44.118 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00019775200000000001 2024-08-19T14:29:44.118 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.118 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.118 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:29:44.119 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.249313+0000", 2024-08-19T14:29:44.119 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.312e-05 2024-08-19T14:29:44.119 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.119 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.119 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:29:44.119 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.249321+0000", 2024-08-19T14:29:44.119 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.43e-06 2024-08-19T14:29:44.119 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.119 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.119 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:29:44.119 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.249451+0000", 2024-08-19T14:29:44.119 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000129982 2024-08-19T14:29:44.120 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.120 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.120 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:29:44.120 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.249576+0000", 2024-08-19T14:29:44.121 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012470299999999999 2024-08-19T14:29:44.121 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.121 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.121 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 1,9", 2024-08-19T14:29:44.121 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.249703+0000", 2024-08-19T14:29:44.121 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000127315 2024-08-19T14:29:44.121 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.121 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.121 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:29:44.122 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.253189+0000", 2024-08-19T14:29:44.122 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0034865310000000002 2024-08-19T14:29:44.122 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.122 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.122 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:44.122 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.254256+0000", 2024-08-19T14:29:44.122 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0010667630000000001 2024-08-19T14:29:44.122 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.122 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.122 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:44.122 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.451121+0000", 2024-08-19T14:29:44.122 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.19686435199999999 2024-08-19T14:29:44.122 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.123 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.123 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:29:44.123 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.451155+0000", 2024-08-19T14:29:44.123 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.4785e-05 2024-08-19T14:29:44.123 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.123 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.123 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:29:44.123 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.451191+0000", 2024-08-19T14:29:44.123 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5543999999999997e-05 2024-08-19T14:29:44.123 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:44.123 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:29:44.123 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:44.123 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.124 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.124 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:23018 5.7 5:eee02748:::benchmark_data_smithi045_85309_object23017:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e426)", 2024-08-19T14:29:44.124 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:24:55.251860+0000", 2024-08-19T14:29:44.124 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 288.85177018299999, 2024-08-19T14:29:44.124 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20029677200000001, 2024-08-19T14:29:44.124 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:29:44.124 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:29:44.124 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:29:44.124 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:29:44.124 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:29:44.124 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:29:44.124 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 23018 2024-08-19T14:29:44.124 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.125 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:29:44.125 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.125 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:29:44.125 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.251860+0000", 2024-08-19T14:29:44.125 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:44.125 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.125 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.125 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:29:44.125 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.251860+0000", 2024-08-19T14:29:44.125 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:44.125 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.125 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.125 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:29:44.125 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.251861+0000", 2024-08-19T14:29:44.126 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.7399999999999991e-07 2024-08-19T14:29:44.126 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.126 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.126 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:29:44.126 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.251893+0000", 2024-08-19T14:29:44.126 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1671000000000002e-05 2024-08-19T14:29:44.126 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.126 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.126 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:29:44.126 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.251903+0000", 2024-08-19T14:29:44.126 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.8360000000000004e-06 2024-08-19T14:29:44.127 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.127 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.127 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:29:44.127 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.251910+0000", 2024-08-19T14:29:44.127 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1760000000000004e-06 2024-08-19T14:29:44.127 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.127 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.127 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:29:44.127 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.251991+0000", 2024-08-19T14:29:44.127 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.0811999999999998e-05 2024-08-19T14:29:44.128 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.128 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.128 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:29:44.128 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.252123+0000", 2024-08-19T14:29:44.128 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013224100000000001 2024-08-19T14:29:44.128 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.128 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.128 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 1,9", 2024-08-19T14:29:44.128 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.252250+0000", 2024-08-19T14:29:44.128 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012669800000000001 2024-08-19T14:29:44.128 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.128 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.129 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:29:44.129 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.254218+0000", 2024-08-19T14:29:44.129 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001968114 2024-08-19T14:29:44.129 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.129 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.129 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:44.129 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.254934+0000", 2024-08-19T14:29:44.129 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00071644199999999997 2024-08-19T14:29:44.129 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.129 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.129 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:44.129 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.452060+0000", 2024-08-19T14:29:44.130 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.19712607600000001 2024-08-19T14:29:44.130 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.130 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.130 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:29:44.130 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.452114+0000", 2024-08-19T14:29:44.130 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3983999999999997e-05 2024-08-19T14:29:44.130 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.130 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.130 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:29:44.130 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.452157+0000", 2024-08-19T14:29:44.130 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2747999999999998e-05 2024-08-19T14:29:44.130 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:44.131 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:29:44.131 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:44.131 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.131 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.131 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:55755 5.6 5:699d86b9:::benchmark_data_smithi045_85309_object55754:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e431)", 2024-08-19T14:29:44.131 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:25:09.597312+0000", 2024-08-19T14:29:44.131 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 274.50631847199998, 2024-08-19T14:29:44.131 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.209956798, 2024-08-19T14:29:44.131 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:29:44.131 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:29:44.131 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:29:44.131 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:29:44.132 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:29:44.132 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:29:44.132 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 55755 2024-08-19T14:29:44.132 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.132 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:29:44.132 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.132 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:29:44.132 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.597312+0000", 2024-08-19T14:29:44.132 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:44.133 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.133 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.133 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:29:44.133 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.597312+0000", 2024-08-19T14:29:44.133 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:44.133 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.133 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.133 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:29:44.133 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.597313+0000", 2024-08-19T14:29:44.133 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.13e-06 2024-08-19T14:29:44.133 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.133 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.133 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:29:44.134 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.597346+0000", 2024-08-19T14:29:44.134 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3019999999999999e-05 2024-08-19T14:29:44.134 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.134 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.134 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:29:44.134 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.597355+0000", 2024-08-19T14:29:44.134 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.0399999999999998e-06 2024-08-19T14:29:44.134 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.134 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.134 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:29:44.134 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.597362+0000", 2024-08-19T14:29:44.134 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2659999999999998e-06 2024-08-19T14:29:44.135 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.135 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.135 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:29:44.135 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.597410+0000", 2024-08-19T14:29:44.135 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8013000000000003e-05 2024-08-19T14:29:44.135 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.135 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.135 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:29:44.135 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.597518+0000", 2024-08-19T14:29:44.135 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00010813 2024-08-19T14:29:44.135 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.135 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.136 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 7,8", 2024-08-19T14:29:44.136 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.597639+0000", 2024-08-19T14:29:44.136 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000121205 2024-08-19T14:29:44.136 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.136 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.136 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:29:44.136 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.600786+0000", 2024-08-19T14:29:44.136 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.003147558 2024-08-19T14:29:44.136 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.136 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.136 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:44.136 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.601648+0000", 2024-08-19T14:29:44.136 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00086196199999999997 2024-08-19T14:29:44.137 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.137 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.137 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:44.137 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.807181+0000", 2024-08-19T14:29:44.137 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20553236799999999 2024-08-19T14:29:44.137 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.137 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.137 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:29:44.137 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.807220+0000", 2024-08-19T14:29:44.137 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9119999999999998e-05 2024-08-19T14:29:44.137 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.137 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.137 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:29:44.137 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.807269+0000", 2024-08-19T14:29:44.138 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8986000000000002e-05 2024-08-19T14:29:44.138 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:44.138 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:29:44.138 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:44.138 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.138 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.138 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:55761 5.4 5:21702bb2:::benchmark_data_smithi045_85309_object55760:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e431)", 2024-08-19T14:29:44.138 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:25:09.599621+0000", 2024-08-19T14:29:44.138 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 274.50400967100001, 2024-08-19T14:29:44.138 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20994792700000001, 2024-08-19T14:29:44.138 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:29:44.138 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:29:44.138 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:29:44.139 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:29:44.139 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:29:44.139 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:29:44.141 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 55761 2024-08-19T14:29:44.141 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.141 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:29:44.142 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.142 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:29:44.142 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.599621+0000", 2024-08-19T14:29:44.142 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:44.142 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.142 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.142 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:29:44.142 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.599621+0000", 2024-08-19T14:29:44.142 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:44.142 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.142 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.143 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:29:44.144 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:29:44.144 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.599622+0000", 2024-08-19T14:29:44.144 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1689999999999999e-06 2024-08-19T14:29:44.144 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.145 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.145 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:29:44.145 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.599684+0000", 2024-08-19T14:29:44.145 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2228000000000001e-05 2024-08-19T14:29:44.145 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.145 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.145 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:29:44.145 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.599698+0000", 2024-08-19T14:29:44.145 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4031e-05 2024-08-19T14:29:44.145 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.145 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.145 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:29:44.145 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.599705+0000", 2024-08-19T14:29:44.146 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0099999999999998e-06 2024-08-19T14:29:44.146 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.146 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.146 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:29:44.146 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.599804+0000", 2024-08-19T14:29:44.146 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.8288000000000005e-05 2024-08-19T14:29:44.146 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.146 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.146 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:29:44.146 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.600054+0000", 2024-08-19T14:29:44.146 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00025056999999999998 2024-08-19T14:29:44.146 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.147 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.147 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0,8", 2024-08-19T14:29:44.147 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.600199+0000", 2024-08-19T14:29:44.147 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014504200000000001 2024-08-19T14:29:44.147 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.147 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.147 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:44.147 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.604843+0000", 2024-08-19T14:29:44.147 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0046438290000000004 2024-08-19T14:29:44.147 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.147 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.147 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:29:44.147 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.604854+0000", 2024-08-19T14:29:44.147 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0971e-05 2024-08-19T14:29:44.148 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.148 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.148 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:44.148 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.809503+0000", 2024-08-19T14:29:44.148 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20464875799999999 2024-08-19T14:29:44.148 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.148 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.148 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:29:44.148 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.809534+0000", 2024-08-19T14:29:44.148 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1217999999999998e-05 2024-08-19T14:29:44.148 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.148 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.148 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:29:44.148 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.809569+0000", 2024-08-19T14:29:44.149 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.4813e-05 2024-08-19T14:29:44.149 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:44.149 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:29:44.149 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:44.149 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.149 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.149 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:55763 5.4 5:22d4b04e:::benchmark_data_smithi045_85309_object55762:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e431)", 2024-08-19T14:29:44.149 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:25:09.599725+0000", 2024-08-19T14:29:44.149 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 274.503905589, 2024-08-19T14:29:44.149 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20996778999999999, 2024-08-19T14:29:44.149 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:29:44.149 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:29:44.149 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:29:44.150 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:29:44.150 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:29:44.150 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:29:44.150 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 55763 2024-08-19T14:29:44.150 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.150 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:29:44.150 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.150 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:29:44.150 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.599725+0000", 2024-08-19T14:29:44.150 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:44.150 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.150 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.150 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:29:44.150 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.599725+0000", 2024-08-19T14:29:44.151 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:44.151 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.151 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.151 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:29:44.151 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.599726+0000", 2024-08-19T14:29:44.151 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2699999999999999e-07 2024-08-19T14:29:44.151 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.151 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.151 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:29:44.151 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.599802+0000", 2024-08-19T14:29:44.151 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6167000000000004e-05 2024-08-19T14:29:44.152 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.152 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.152 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:29:44.152 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.599811+0000", 2024-08-19T14:29:44.152 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.6220000000000004e-06 2024-08-19T14:29:44.152 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.152 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.152 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:29:44.152 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.599818+0000", 2024-08-19T14:29:44.152 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4420000000000003e-06 2024-08-19T14:29:44.152 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.152 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.152 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:29:44.152 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.602462+0000", 2024-08-19T14:29:44.153 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0026439990000000002 2024-08-19T14:29:44.153 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.153 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.153 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:29:44.153 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.602619+0000", 2024-08-19T14:29:44.153 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00015674800000000001 2024-08-19T14:29:44.153 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.153 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.153 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0,8", 2024-08-19T14:29:44.153 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.602748+0000", 2024-08-19T14:29:44.153 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012939900000000001 2024-08-19T14:29:44.153 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.153 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.154 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:44.154 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.609052+0000", 2024-08-19T14:29:44.154 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.006304128 2024-08-19T14:29:44.154 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.154 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.154 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:29:44.154 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.611165+0000", 2024-08-19T14:29:44.154 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.002113008 2024-08-19T14:29:44.154 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.154 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.154 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:44.154 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.809609+0000", 2024-08-19T14:29:44.154 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.19844395500000001 2024-08-19T14:29:44.154 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.155 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.155 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:29:44.155 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.809648+0000", 2024-08-19T14:29:44.155 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9081000000000001e-05 2024-08-19T14:29:44.155 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.155 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.155 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:29:44.155 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.809693+0000", 2024-08-19T14:29:44.155 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4613999999999997e-05 2024-08-19T14:29:44.155 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:44.155 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:29:44.155 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:44.155 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.155 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.156 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:55767 5.4 5:200c3bc0:::benchmark_data_smithi045_85309_object55766:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e431)", 2024-08-19T14:29:44.156 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:25:09.601839+0000", 2024-08-19T14:29:44.156 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 274.501791561, 2024-08-19T14:29:44.156 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20836729700000001, 2024-08-19T14:29:44.156 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:29:44.156 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:29:44.156 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:29:44.156 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:29:44.156 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:29:44.156 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:29:44.156 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 55767 2024-08-19T14:29:44.156 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.156 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:29:44.157 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.157 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:29:44.157 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.601839+0000", 2024-08-19T14:29:44.157 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:44.157 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.157 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.157 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:29:44.157 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.601839+0000", 2024-08-19T14:29:44.157 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:44.157 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.157 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.158 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:29:44.158 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.601840+0000", 2024-08-19T14:29:44.158 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.049e-06 2024-08-19T14:29:44.158 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.158 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.158 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:29:44.158 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.601863+0000", 2024-08-19T14:29:44.158 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2748999999999999e-05 2024-08-19T14:29:44.158 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.158 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.158 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:29:44.159 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.601874+0000", 2024-08-19T14:29:44.159 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1219999999999999e-05 2024-08-19T14:29:44.159 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.159 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.159 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:29:44.159 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.601881+0000", 2024-08-19T14:29:44.159 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.562e-06 2024-08-19T14:29:44.159 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.159 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.160 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:29:44.160 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.604918+0000", 2024-08-19T14:29:44.160 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.003037629 2024-08-19T14:29:44.160 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.160 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.160 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:29:44.160 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.605062+0000", 2024-08-19T14:29:44.160 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000143821 2024-08-19T14:29:44.160 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.160 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.160 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0,8", 2024-08-19T14:29:44.160 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.605192+0000", 2024-08-19T14:29:44.160 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013037300000000001 2024-08-19T14:29:44.161 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.161 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.161 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:44.161 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.611145+0000", 2024-08-19T14:29:44.161 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0059520939999999998 2024-08-19T14:29:44.161 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.161 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.161 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:29:44.161 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.611171+0000", 2024-08-19T14:29:44.161 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6618000000000002e-05 2024-08-19T14:29:44.161 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.161 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.162 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:44.162 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.810142+0000", 2024-08-19T14:29:44.162 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.19897035199999999 2024-08-19T14:29:44.162 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.162 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.162 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:29:44.162 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.810178+0000", 2024-08-19T14:29:44.162 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6950999999999999e-05 2024-08-19T14:29:44.162 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.162 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.162 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:29:44.162 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.810206+0000", 2024-08-19T14:29:44.162 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7878999999999999e-05 2024-08-19T14:29:44.163 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:44.163 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:29:44.163 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:44.163 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.163 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.163 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:55775 5.4 5:2da10789:::benchmark_data_smithi045_85309_object55774:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e431)", 2024-08-19T14:29:44.163 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:25:09.605530+0000", 2024-08-19T14:29:44.163 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 274.49810101399999, 2024-08-19T14:29:44.163 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20478083, 2024-08-19T14:29:44.163 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:29:44.163 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:29:44.163 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:29:44.163 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:29:44.163 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:29:44.164 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:29:44.164 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 55775 2024-08-19T14:29:44.164 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.164 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:29:44.164 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.164 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:29:44.164 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.605530+0000", 2024-08-19T14:29:44.164 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:44.164 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.164 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.164 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:29:44.164 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.605530+0000", 2024-08-19T14:29:44.164 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:44.164 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.165 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.165 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:29:44.165 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.605531+0000", 2024-08-19T14:29:44.165 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3039999999999999e-06 2024-08-19T14:29:44.165 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.165 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.165 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:29:44.165 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.605633+0000", 2024-08-19T14:29:44.165 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000102602 2024-08-19T14:29:44.165 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.165 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.165 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:29:44.165 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.605643+0000", 2024-08-19T14:29:44.166 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.4399999999999994e-06 2024-08-19T14:29:44.166 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.166 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.166 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:29:44.166 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.605649+0000", 2024-08-19T14:29:44.166 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2169999999999999e-06 2024-08-19T14:29:44.166 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.166 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.166 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:29:44.166 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.607082+0000", 2024-08-19T14:29:44.166 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0014326289999999999 2024-08-19T14:29:44.166 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.166 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.166 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:29:44.167 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.607190+0000", 2024-08-19T14:29:44.167 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000108428 2024-08-19T14:29:44.167 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.167 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.167 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0,8", 2024-08-19T14:29:44.167 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.607312+0000", 2024-08-19T14:29:44.167 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000121927 2024-08-19T14:29:44.167 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.167 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.167 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:29:44.167 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.611175+0000", 2024-08-19T14:29:44.167 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0038631889999999999 2024-08-19T14:29:44.167 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.167 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.168 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:44.168 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.614394+0000", 2024-08-19T14:29:44.168 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.003218505 2024-08-19T14:29:44.168 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.168 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.168 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:44.168 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.810243+0000", 2024-08-19T14:29:44.168 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.19584892000000001 2024-08-19T14:29:44.168 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.168 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.168 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:29:44.168 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.810269+0000", 2024-08-19T14:29:44.168 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6027e-05 2024-08-19T14:29:44.168 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.169 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.169 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:29:44.169 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.810310+0000", 2024-08-19T14:29:44.169 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1641999999999999e-05 2024-08-19T14:29:44.169 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:44.169 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:29:44.169 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:44.169 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.169 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.169 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:55779 5.4 5:28c62a63:::benchmark_data_smithi045_85309_object55778:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e431)", 2024-08-19T14:29:44.169 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:25:09.607121+0000", 2024-08-19T14:29:44.169 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 274.49650968399999, 2024-08-19T14:29:44.169 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20333711199999999, 2024-08-19T14:29:44.169 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:29:44.170 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:29:44.170 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:29:44.170 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:29:44.170 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:29:44.170 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:29:44.170 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 55779 2024-08-19T14:29:44.170 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.170 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:29:44.170 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.170 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:29:44.170 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.607121+0000", 2024-08-19T14:29:44.170 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:44.170 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.170 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.170 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:29:44.171 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.607121+0000", 2024-08-19T14:29:44.171 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:44.171 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.171 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.171 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:29:44.171 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.607122+0000", 2024-08-19T14:29:44.171 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1820000000000001e-06 2024-08-19T14:29:44.171 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.171 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.171 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:29:44.171 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.607157+0000", 2024-08-19T14:29:44.171 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.4443999999999998e-05 2024-08-19T14:29:44.171 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.171 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.172 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:29:44.172 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.607169+0000", 2024-08-19T14:29:44.172 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2531000000000001e-05 2024-08-19T14:29:44.172 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.172 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.172 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:29:44.172 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.607176+0000", 2024-08-19T14:29:44.172 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1199999999999996e-06 2024-08-19T14:29:44.172 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.172 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.172 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:29:44.172 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.609081+0000", 2024-08-19T14:29:44.172 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001904705 2024-08-19T14:29:44.172 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.173 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.173 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:29:44.173 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.609202+0000", 2024-08-19T14:29:44.173 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000121138 2024-08-19T14:29:44.173 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.173 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.173 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0,8", 2024-08-19T14:29:44.173 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.609328+0000", 2024-08-19T14:29:44.173 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012569100000000001 2024-08-19T14:29:44.173 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.173 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.173 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:29:44.173 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.611568+0000", 2024-08-19T14:29:44.173 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0022404769999999998 2024-08-19T14:29:44.174 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.174 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.174 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:44.174 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.616823+0000", 2024-08-19T14:29:44.174 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0052546889999999999 2024-08-19T14:29:44.174 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.174 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.174 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:44.174 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.810361+0000", 2024-08-19T14:29:44.174 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.193538245 2024-08-19T14:29:44.174 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.174 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.174 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:29:44.174 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.810410+0000", 2024-08-19T14:29:44.175 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8819000000000001e-05 2024-08-19T14:29:44.175 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.175 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.175 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:29:44.175 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.810458+0000", 2024-08-19T14:29:44.175 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8071000000000001e-05 2024-08-19T14:29:44.175 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:44.175 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:29:44.175 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:44.175 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.175 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.175 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:81628 5.7 5:e12ddb93:::benchmark_data_smithi045_85309_object81627:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e437)", 2024-08-19T14:29:44.175 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:25:25.047926+0000", 2024-08-19T14:29:44.175 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 259.05570471999999, 2024-08-19T14:29:44.176 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.207246069, 2024-08-19T14:29:44.176 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:29:44.176 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:29:44.176 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:29:44.176 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:29:44.176 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:29:44.176 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:29:44.176 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 81628 2024-08-19T14:29:44.176 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.176 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:29:44.176 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.176 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:29:44.176 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.047926+0000", 2024-08-19T14:29:44.176 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:44.176 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.177 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.177 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:29:44.177 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.047926+0000", 2024-08-19T14:29:44.177 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:44.177 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.177 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.177 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:29:44.177 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.047927+0000", 2024-08-19T14:29:44.177 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0219999999999999e-06 2024-08-19T14:29:44.177 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.177 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.177 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:29:44.177 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.047977+0000", 2024-08-19T14:29:44.177 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0170000000000002e-05 2024-08-19T14:29:44.178 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.178 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.178 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:29:44.178 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.047993+0000", 2024-08-19T14:29:44.178 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5835999999999999e-05 2024-08-19T14:29:44.178 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.178 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.178 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:29:44.178 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.047998+0000", 2024-08-19T14:29:44.178 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8500000000000002e-06 2024-08-19T14:29:44.178 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.178 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.178 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:29:44.179 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.048045+0000", 2024-08-19T14:29:44.179 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6828e-05 2024-08-19T14:29:44.179 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.179 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.179 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:29:44.179 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.048139+0000", 2024-08-19T14:29:44.179 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.4199999999999999e-05 2024-08-19T14:29:44.179 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.179 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.179 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 1,9", 2024-08-19T14:29:44.179 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.048232+0000", 2024-08-19T14:29:44.179 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3393999999999994e-05 2024-08-19T14:29:44.179 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.179 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.180 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:29:44.180 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.050672+0000", 2024-08-19T14:29:44.180 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0024403189999999998 2024-08-19T14:29:44.180 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.180 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.180 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:44.180 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.051858+0000", 2024-08-19T14:29:44.180 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0011859489999999999 2024-08-19T14:29:44.180 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.180 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.180 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:44.180 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.255101+0000", 2024-08-19T14:29:44.180 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20324223 2024-08-19T14:29:44.180 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.181 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.181 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:29:44.181 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.255142+0000", 2024-08-19T14:29:44.181 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0915999999999999e-05 2024-08-19T14:29:44.181 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.181 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.181 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:29:44.181 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.255172+0000", 2024-08-19T14:29:44.181 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0355e-05 2024-08-19T14:29:44.181 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:44.181 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:29:44.181 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:44.181 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.181 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.182 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:81639 5.7 5:ef0c5b8b:::benchmark_data_smithi045_85309_object81638:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e437)", 2024-08-19T14:29:44.182 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:25:25.053488+0000", 2024-08-19T14:29:44.182 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 259.05014283999998, 2024-08-19T14:29:44.182 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20480580300000001, 2024-08-19T14:29:44.182 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:29:44.182 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:29:44.182 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:29:44.182 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:29:44.182 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:29:44.182 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:29:44.182 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 81639 2024-08-19T14:29:44.182 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.183 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:29:44.183 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.183 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:29:44.183 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.053488+0000", 2024-08-19T14:29:44.183 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:44.183 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.183 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.183 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:29:44.183 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.053488+0000", 2024-08-19T14:29:44.183 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:44.183 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.183 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.183 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:29:44.184 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.053489+0000", 2024-08-19T14:29:44.184 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.4200000000000005e-07 2024-08-19T14:29:44.184 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.184 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.184 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:29:44.184 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.053559+0000", 2024-08-19T14:29:44.184 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0240000000000005e-05 2024-08-19T14:29:44.184 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.184 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.184 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:29:44.184 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.053568+0000", 2024-08-19T14:29:44.184 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.9989999999999997e-06 2024-08-19T14:29:44.185 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.185 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.185 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:29:44.185 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.053575+0000", 2024-08-19T14:29:44.185 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.092e-06 2024-08-19T14:29:44.185 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.185 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.185 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:29:44.185 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.053624+0000", 2024-08-19T14:29:44.185 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9209000000000003e-05 2024-08-19T14:29:44.185 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.185 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.185 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:29:44.186 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.053826+0000", 2024-08-19T14:29:44.186 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000201573 2024-08-19T14:29:44.186 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.186 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.186 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 1,9", 2024-08-19T14:29:44.186 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.054893+0000", 2024-08-19T14:29:44.186 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0010669849999999999 2024-08-19T14:29:44.186 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.186 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.186 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:44.186 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.058523+0000", 2024-08-19T14:29:44.186 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0036301929999999999 2024-08-19T14:29:44.186 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.187 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.187 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:29:44.187 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.060958+0000", 2024-08-19T14:29:44.187 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0024348579999999998 2024-08-19T14:29:44.187 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.187 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.187 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:44.187 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.258229+0000", 2024-08-19T14:29:44.187 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.19727160199999999 2024-08-19T14:29:44.187 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.187 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.187 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:29:44.187 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.258265+0000", 2024-08-19T14:29:44.188 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5717999999999999e-05 2024-08-19T14:29:44.188 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.188 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.188 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:29:44.188 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.258294+0000", 2024-08-19T14:29:44.188 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8492000000000001e-05 2024-08-19T14:29:44.188 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:44.188 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:29:44.188 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:44.188 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.188 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.188 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:81655 5.7 5:e1e2b816:::benchmark_data_smithi045_85309_object81654:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e437)", 2024-08-19T14:29:44.189 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:25:25.061831+0000", 2024-08-19T14:29:44.189 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 259.04179983699999, 2024-08-19T14:29:44.189 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.19841341400000001, 2024-08-19T14:29:44.189 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:29:44.189 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:29:44.189 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:29:44.189 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:29:44.190 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:29:44.190 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:29:44.190 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 81655 2024-08-19T14:29:44.190 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.190 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:29:44.190 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.190 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:29:44.190 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.061831+0000", 2024-08-19T14:29:44.190 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:44.191 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.191 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.191 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:29:44.191 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.061831+0000", 2024-08-19T14:29:44.191 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:44.191 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.191 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.191 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:29:44.191 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.061831+0000", 2024-08-19T14:29:44.191 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4099999999999998e-07 2024-08-19T14:29:44.191 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.191 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.191 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:29:44.192 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.061902+0000", 2024-08-19T14:29:44.192 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0333999999999994e-05 2024-08-19T14:29:44.192 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.192 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.192 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:29:44.192 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.061912+0000", 2024-08-19T14:29:44.192 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0125e-05 2024-08-19T14:29:44.192 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.192 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.192 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:29:44.192 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.061918+0000", 2024-08-19T14:29:44.192 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.879e-06 2024-08-19T14:29:44.192 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.192 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.193 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:29:44.193 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.061972+0000", 2024-08-19T14:29:44.193 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4756000000000002e-05 2024-08-19T14:29:44.193 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.193 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.193 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:29:44.193 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.062117+0000", 2024-08-19T14:29:44.193 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000144103 2024-08-19T14:29:44.193 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.193 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.193 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 1,9", 2024-08-19T14:29:44.193 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.062257+0000", 2024-08-19T14:29:44.193 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000140271 2024-08-19T14:29:44.193 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.194 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.194 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:29:44.194 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.065634+0000", 2024-08-19T14:29:44.194 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0033769440000000002 2024-08-19T14:29:44.194 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.194 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.194 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:44.194 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.066411+0000", 2024-08-19T14:29:44.194 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00077719599999999996 2024-08-19T14:29:44.194 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.194 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.194 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:44.194 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.260128+0000", 2024-08-19T14:29:44.194 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.19371739800000001 2024-08-19T14:29:44.195 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.195 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.195 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:29:44.195 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.260203+0000", 2024-08-19T14:29:44.195 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4293000000000002e-05 2024-08-19T14:29:44.195 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.195 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.195 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:29:44.195 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.260244+0000", 2024-08-19T14:29:44.195 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1474000000000003e-05 2024-08-19T14:29:44.195 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:44.195 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:29:44.195 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:44.195 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.196 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.196 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:92831 5.f 5:fd6d6df7:::benchmark_data_smithi045_85309_object92830:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e445)", 2024-08-19T14:29:44.196 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:25:37.231829+0000", 2024-08-19T14:29:44.196 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 246.871801869, 2024-08-19T14:29:44.196 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.194037033, 2024-08-19T14:29:44.196 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:29:44.196 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:29:44.196 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:29:44.196 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:29:44.196 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:29:44.196 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:29:44.196 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 92831 2024-08-19T14:29:44.196 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.197 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:29:44.197 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.197 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:29:44.197 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.231829+0000", 2024-08-19T14:29:44.197 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:44.197 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.197 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.197 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:29:44.197 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.231829+0000", 2024-08-19T14:29:44.197 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:44.197 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.198 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.198 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:29:44.198 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.231831+0000", 2024-08-19T14:29:44.198 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8670000000000001e-06 2024-08-19T14:29:44.198 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.198 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.198 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:29:44.198 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.231959+0000", 2024-08-19T14:29:44.198 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000128181 2024-08-19T14:29:44.198 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.198 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.198 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:29:44.198 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.231972+0000", 2024-08-19T14:29:44.199 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3689999999999999e-05 2024-08-19T14:29:44.199 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.199 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.199 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:29:44.199 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.231988+0000", 2024-08-19T14:29:44.199 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.562e-05 2024-08-19T14:29:44.199 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.199 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.199 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:29:44.199 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.232043+0000", 2024-08-19T14:29:44.199 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4694999999999997e-05 2024-08-19T14:29:44.199 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.199 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.199 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:29:44.200 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.232161+0000", 2024-08-19T14:29:44.200 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00011810499999999999 2024-08-19T14:29:44.200 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.200 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.200 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4,11", 2024-08-19T14:29:44.200 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.232284+0000", 2024-08-19T14:29:44.200 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000123452 2024-08-19T14:29:44.200 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.200 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.200 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:44.200 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.235655+0000", 2024-08-19T14:29:44.200 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.003370428 2024-08-19T14:29:44.200 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.200 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.201 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:44.201 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.243630+0000", 2024-08-19T14:29:44.201 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0079749609999999992 2024-08-19T14:29:44.201 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.201 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.201 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:29:44.201 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.425783+0000", 2024-08-19T14:29:44.201 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.18215342500000001 2024-08-19T14:29:44.201 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.201 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.201 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:29:44.201 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.425829+0000", 2024-08-19T14:29:44.201 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5809999999999997e-05 2024-08-19T14:29:44.201 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.201 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.202 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:29:44.202 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.425866+0000", 2024-08-19T14:29:44.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:44.204 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6798999999999998e-05 2024-08-19T14:29:44.204 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:44.204 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:29:44.204 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:44.204 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.204 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.204 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:92839 5.f 5:f7605d59:::benchmark_data_smithi045_85309_object92838:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e445)", 2024-08-19T14:29:44.204 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:25:37.241907+0000", 2024-08-19T14:29:44.205 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 246.86172402099999, 2024-08-19T14:29:44.205 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20647278499999999, 2024-08-19T14:29:44.205 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:29:44.205 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:29:44.205 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:29:44.205 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:29:44.205 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:29:44.205 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:29:44.205 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 92839 2024-08-19T14:29:44.205 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.205 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:29:44.205 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.205 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:29:44.206 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.241907+0000", 2024-08-19T14:29:44.206 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:44.206 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.206 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.206 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:29:44.206 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.241907+0000", 2024-08-19T14:29:44.206 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:44.206 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.206 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.206 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:29:44.206 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.241908+0000", 2024-08-19T14:29:44.206 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4050000000000001e-06 2024-08-19T14:29:44.206 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.207 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.207 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:29:44.207 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.241947+0000", 2024-08-19T14:29:44.207 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8847999999999998e-05 2024-08-19T14:29:44.207 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.207 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.207 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:29:44.207 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.241956+0000", 2024-08-19T14:29:44.207 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.5550000000000008e-06 2024-08-19T14:29:44.207 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.207 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.207 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:29:44.207 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.241962+0000", 2024-08-19T14:29:44.207 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2490000000000001e-06 2024-08-19T14:29:44.208 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.208 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.208 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:29:44.208 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.243721+0000", 2024-08-19T14:29:44.208 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0017588910000000001 2024-08-19T14:29:44.208 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.208 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.208 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:29:44.208 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.422533+0000", 2024-08-19T14:29:44.208 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.178812148 2024-08-19T14:29:44.208 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.208 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.208 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4,11", 2024-08-19T14:29:44.208 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.422686+0000", 2024-08-19T14:29:44.208 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00015375499999999999 2024-08-19T14:29:44.209 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.209 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.209 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:29:44.209 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.426020+0000", 2024-08-19T14:29:44.209 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.003333698 2024-08-19T14:29:44.209 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.209 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.209 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:44.209 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.430495+0000", 2024-08-19T14:29:44.209 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0044751740000000002 2024-08-19T14:29:44.209 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.209 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.209 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:44.209 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.435394+0000", 2024-08-19T14:29:44.210 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0048991900000000003 2024-08-19T14:29:44.210 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.210 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.210 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:29:44.210 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.435458+0000", 2024-08-19T14:29:44.210 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3068e-05 2024-08-19T14:29:44.210 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.210 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.210 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:29:44.210 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.448379+0000", 2024-08-19T14:29:44.210 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.012921804 2024-08-19T14:29:44.210 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:44.210 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:29:44.211 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:44.211 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.211 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.211 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.25813.0:92826 5.15 e445/442 5:a965286d:::benchmark_data_smithi045_85309_object92825:head v 445'5490, mlcod=423'262)", 2024-08-19T14:29:44.211 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:25:37.245988+0000", 2024-08-19T14:29:44.211 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 246.857642336, 2024-08-19T14:29:44.211 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20615012599999999, 2024-08-19T14:29:44.211 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:29:44.211 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:29:44.211 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:29:44.212 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:29:44.212 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.212 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:29:44.212 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.245988+0000", 2024-08-19T14:29:44.212 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:44.212 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.212 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.212 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:29:44.212 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.245988+0000", 2024-08-19T14:29:44.212 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:44.212 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.212 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.212 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:29:44.212 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.245989+0000", 2024-08-19T14:29:44.213 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0210000000000001e-06 2024-08-19T14:29:44.213 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.213 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.213 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:29:44.213 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.246023+0000", 2024-08-19T14:29:44.213 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.4044999999999997e-05 2024-08-19T14:29:44.213 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.213 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.213 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:29:44.213 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.422398+0000", 2024-08-19T14:29:44.213 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.17637508399999999 2024-08-19T14:29:44.213 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.213 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.213 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:29:44.213 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.422438+0000", 2024-08-19T14:29:44.214 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9362e-05 2024-08-19T14:29:44.214 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.214 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.214 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:29:44.214 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.426229+0000", 2024-08-19T14:29:44.214 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.003791141 2024-08-19T14:29:44.214 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.214 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.214 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:29:44.214 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.426253+0000", 2024-08-19T14:29:44.214 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4159e-05 2024-08-19T14:29:44.214 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.214 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.214 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:29:44.215 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.452061+0000", 2024-08-19T14:29:44.215 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.025807672 2024-08-19T14:29:44.215 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.215 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.215 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:29:44.215 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.452138+0000", 2024-08-19T14:29:44.215 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7642e-05 2024-08-19T14:29:44.215 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:44.215 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:29:44.215 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:44.215 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.215 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.215 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:137353 5.4 5:21a8a66c:::benchmark_data_smithi045_85309_object17100:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e465)", 2024-08-19T14:29:44.215 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:26:33.514959+0000", 2024-08-19T14:29:44.216 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 190.58867184600001, 2024-08-19T14:29:44.216 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.163544353, 2024-08-19T14:29:44.216 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:29:44.216 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:29:44.216 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:29:44.216 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:29:44.216 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:29:44.216 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:29:44.216 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 137353 2024-08-19T14:29:44.216 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.216 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:29:44.216 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.216 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:29:44.216 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:26:33.514959+0000", 2024-08-19T14:29:44.216 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:44.217 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.217 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.217 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:29:44.217 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:26:33.514959+0000", 2024-08-19T14:29:44.217 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:44.217 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.217 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.217 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:29:44.217 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:26:33.514959+0000", 2024-08-19T14:29:44.217 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7000000000000004e-07 2024-08-19T14:29:44.217 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.217 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.217 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:29:44.217 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:26:33.514963+0000", 2024-08-19T14:29:44.218 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.4860000000000002e-06 2024-08-19T14:29:44.218 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.218 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.218 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:29:44.218 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:26:33.514972+0000", 2024-08-19T14:29:44.218 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6710000000000005e-06 2024-08-19T14:29:44.218 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.218 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.218 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:29:44.218 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:26:33.514976+0000", 2024-08-19T14:29:44.218 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5050000000000001e-06 2024-08-19T14:29:44.218 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.219 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.219 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:29:44.219 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:26:33.515031+0000", 2024-08-19T14:29:44.219 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5306000000000002e-05 2024-08-19T14:29:44.219 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.219 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.219 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:29:44.219 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:26:33.515112+0000", 2024-08-19T14:29:44.219 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.0171000000000002e-05 2024-08-19T14:29:44.219 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.219 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.220 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0,8", 2024-08-19T14:29:44.220 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:26:33.515178+0000", 2024-08-19T14:29:44.220 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.5963000000000005e-05 2024-08-19T14:29:44.220 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.220 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.220 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:29:44.220 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:26:33.517095+0000", 2024-08-19T14:29:44.220 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0019177510000000001 2024-08-19T14:29:44.220 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.220 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.220 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:44.220 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:26:33.517719+0000", 2024-08-19T14:29:44.220 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00062383399999999998 2024-08-19T14:29:44.220 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.221 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.221 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:44.221 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:26:33.678433+0000", 2024-08-19T14:29:44.221 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.16071344200000001 2024-08-19T14:29:44.221 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.221 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.221 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:29:44.221 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:26:33.678470+0000", 2024-08-19T14:29:44.222 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7592999999999997e-05 2024-08-19T14:29:44.222 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.222 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.223 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:29:44.223 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:26:33.678503+0000", 2024-08-19T14:29:44.223 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2960999999999998e-05 2024-08-19T14:29:44.223 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:44.223 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:29:44.223 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:44.223 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.223 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.223 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:178106 5.15 5:afb34903:::benchmark_data_smithi045_85309_object57853:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e475)", 2024-08-19T14:29:44.223 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:27:08.194118+0000", 2024-08-19T14:29:44.223 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 155.909512374, 2024-08-19T14:29:44.223 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.143674514, 2024-08-19T14:29:44.223 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:29:44.224 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:29:44.224 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:29:44.224 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:29:44.224 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:29:44.224 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:29:44.224 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 178106 2024-08-19T14:29:44.224 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.224 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:29:44.224 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.224 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:29:44.224 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:08.194118+0000", 2024-08-19T14:29:44.224 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:44.224 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.225 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.225 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:29:44.225 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:08.194118+0000", 2024-08-19T14:29:44.225 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:44.225 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.225 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.225 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:29:44.225 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:08.194120+0000", 2024-08-19T14:29:44.225 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.505e-06 2024-08-19T14:29:44.225 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.225 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.225 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:29:44.225 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:08.194125+0000", 2024-08-19T14:29:44.225 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6350000000000003e-06 2024-08-19T14:29:44.226 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.226 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.226 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:29:44.226 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:08.194140+0000", 2024-08-19T14:29:44.226 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4493e-05 2024-08-19T14:29:44.226 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.226 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.226 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:29:44.226 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:08.194147+0000", 2024-08-19T14:29:44.226 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8770000000000004e-06 2024-08-19T14:29:44.226 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.226 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.226 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:29:44.227 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:08.326719+0000", 2024-08-19T14:29:44.227 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.13257237799999999 2024-08-19T14:29:44.227 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.227 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.227 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:29:44.227 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:08.326847+0000", 2024-08-19T14:29:44.227 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012791199999999999 2024-08-19T14:29:44.227 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.227 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.227 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 1,8", 2024-08-19T14:29:44.227 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:08.326954+0000", 2024-08-19T14:29:44.227 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00010745 2024-08-19T14:29:44.227 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.228 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.228 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:44.228 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:08.332374+0000", 2024-08-19T14:29:44.228 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0054192959999999997 2024-08-19T14:29:44.228 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.228 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.228 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:44.228 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:08.337456+0000", 2024-08-19T14:29:44.228 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0050822100000000002 2024-08-19T14:29:44.228 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.228 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.228 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:29:44.228 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:08.337600+0000", 2024-08-19T14:29:44.228 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014429699999999999 2024-08-19T14:29:44.229 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.229 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.229 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:29:44.229 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:08.337744+0000", 2024-08-19T14:29:44.229 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000143824 2024-08-19T14:29:44.229 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.229 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.229 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:29:44.229 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:08.337793+0000", 2024-08-19T14:29:44.229 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8637000000000002e-05 2024-08-19T14:29:44.229 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:44.229 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:29:44.229 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:44.229 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.230 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.230 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.25813.0:214166 5.b e488/477 5:db5fbde7:::benchmark_data_smithi045_85309_object93913:head v 488'13616, mlcod=488'13613)", 2024-08-19T14:29:44.230 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:27:44.732108+0000", 2024-08-19T14:29:44.230 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 119.371522099, 2024-08-19T14:29:44.230 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.041127666, 2024-08-19T14:29:44.230 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:29:44.230 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:29:44.230 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:29:44.230 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:29:44.230 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.230 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:29:44.230 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:44.732108+0000", 2024-08-19T14:29:44.230 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:44.230 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.231 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.231 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:29:44.231 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:44.732108+0000", 2024-08-19T14:29:44.231 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:44.231 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.231 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.231 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:29:44.231 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:44.732109+0000", 2024-08-19T14:29:44.231 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.58e-07 2024-08-19T14:29:44.232 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.232 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.232 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:29:44.232 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:44.732114+0000", 2024-08-19T14:29:44.232 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.69e-06 2024-08-19T14:29:44.232 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.232 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.232 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:29:44.232 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:44.732122+0000", 2024-08-19T14:29:44.232 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.994e-06 2024-08-19T14:29:44.232 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.232 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.232 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:29:44.233 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:44.732128+0000", 2024-08-19T14:29:44.233 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4300000000000003e-06 2024-08-19T14:29:44.233 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.233 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.233 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:29:44.233 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:44.734116+0000", 2024-08-19T14:29:44.233 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0019877110000000001 2024-08-19T14:29:44.233 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.233 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.233 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:29:44.233 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:44.734164+0000", 2024-08-19T14:29:44.233 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8151e-05 2024-08-19T14:29:44.233 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.233 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.234 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:29:44.234 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:44.773129+0000", 2024-08-19T14:29:44.234 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.038965059000000003 2024-08-19T14:29:44.234 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.234 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.234 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:29:44.234 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:44.773236+0000", 2024-08-19T14:29:44.234 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000107173 2024-08-19T14:29:44.234 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:44.234 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:29:44.234 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:44.234 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.234 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.234 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.25813.0:226878 5.d e490/419 5:b7d0760d:::benchmark_data_smithi045_85309_object106625:head v 490'17002, mlcod=490'16999)", 2024-08-19T14:29:44.235 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:27:56.020064+0000", 2024-08-19T14:29:44.235 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 108.08356659499999, 2024-08-19T14:29:44.235 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.038232558999999999, 2024-08-19T14:29:44.235 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:29:44.235 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:29:44.235 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:29:44.235 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:29:44.235 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.235 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:29:44.235 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:56.020064+0000", 2024-08-19T14:29:44.235 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:44.235 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.235 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.236 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:29:44.236 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:56.020064+0000", 2024-08-19T14:29:44.236 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:44.236 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.236 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.236 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:29:44.236 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:56.020064+0000", 2024-08-19T14:29:44.236 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6199999999999998e-07 2024-08-19T14:29:44.236 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.236 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.236 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:29:44.236 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:56.020068+0000", 2024-08-19T14:29:44.236 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6440000000000002e-06 2024-08-19T14:29:44.236 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.237 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.237 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:29:44.237 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:56.020075+0000", 2024-08-19T14:29:44.237 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1500000000000002e-06 2024-08-19T14:29:44.237 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.237 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.237 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:29:44.237 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:56.020081+0000", 2024-08-19T14:29:44.237 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3009999999999997e-06 2024-08-19T14:29:44.237 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.237 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.237 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:29:44.237 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:56.020135+0000", 2024-08-19T14:29:44.237 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4169999999999998e-05 2024-08-19T14:29:44.238 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.238 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.238 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:29:44.238 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:56.020154+0000", 2024-08-19T14:29:44.238 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8902999999999998e-05 2024-08-19T14:29:44.238 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.238 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.238 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:29:44.238 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:56.058254+0000", 2024-08-19T14:29:44.238 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.038100137999999999 2024-08-19T14:29:44.238 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.238 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.238 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:29:44.238 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:56.058297+0000", 2024-08-19T14:29:44.239 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2790999999999997e-05 2024-08-19T14:29:44.239 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:44.239 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:29:44.239 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:44.239 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.239 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.239 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.25813.0:238811 5.13 e498/454 5:c8e9800f:::benchmark_data_smithi045_85309_object118558:head v 498'10045, mlcod=498'10043)", 2024-08-19T14:29:44.239 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:28:10.087006+0000", 2024-08-19T14:29:44.239 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 94.016624725, 2024-08-19T14:29:44.239 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.037579071999999998, 2024-08-19T14:29:44.239 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:29:44.240 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:29:44.240 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:29:44.240 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:29:44.240 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.240 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:29:44.240 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:28:10.087006+0000", 2024-08-19T14:29:44.240 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:44.240 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.240 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.240 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:29:44.240 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:28:10.087006+0000", 2024-08-19T14:29:44.240 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:44.240 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.241 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.241 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:29:44.241 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:28:10.087006+0000", 2024-08-19T14:29:44.241 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1799999999999995e-07 2024-08-19T14:29:44.241 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.241 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.241 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:29:44.241 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:28:10.087011+0000", 2024-08-19T14:29:44.241 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9130000000000002e-06 2024-08-19T14:29:44.241 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.241 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.241 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:29:44.241 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:28:10.087021+0000", 2024-08-19T14:29:44.241 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.5829999999999996e-06 2024-08-19T14:29:44.242 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.242 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.242 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:29:44.242 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:28:10.087027+0000", 2024-08-19T14:29:44.242 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2349999999999996e-06 2024-08-19T14:29:44.242 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.242 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.242 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:29:44.242 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:28:10.092635+0000", 2024-08-19T14:29:44.242 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.005607852 2024-08-19T14:29:44.242 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.242 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.242 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:29:44.243 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:28:10.092652+0000", 2024-08-19T14:29:44.243 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7263999999999999e-05 2024-08-19T14:29:44.243 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.243 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.243 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:29:44.243 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:28:10.124451+0000", 2024-08-19T14:29:44.243 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.031799132000000001 2024-08-19T14:29:44.243 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:44.243 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:44.243 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:29:44.243 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:28:10.124585+0000", 2024-08-19T14:29:44.243 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013347500000000001 2024-08-19T14:29:44.243 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:44.243 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:29:44.244 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:44.244 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:44.244 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:29:44.244 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:29:44.244 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:44.244 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:29:44.345 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:29:44.401 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:29:44.411 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:44.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:44.412 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:29:44.511 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 66 16 92916 92900 87.9662 40.1875 0.0206011 0.0113636 2024-08-19T14:29:44.513 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:29:44.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:44.566 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:44.566 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:29:44.667 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:29:44.704 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:44.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:44.705 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:29:44.796 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:29:44 smithi045 podman[102043]: 2024-08-19 14:29:44.539723424 +0000 UTC m=+0.752177308 container remove 9e801c2f0d3ac440920cb35459d83d1242da864c0294e0f597e936396454bb19 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10-activate, CEPH_POINT_RELEASE=, RELEASE=squid-de09dd0, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-08-19T14:29:44.796 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:29:44 smithi045 podman[102100]: 2024-08-19 14:29:44.662168611 +0000 UTC m=+0.018362954 image pull ea7a5189f30bad3aca9524788e2965d2368d7ce786df09d4edcd8c3edf9cefba quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9 2024-08-19T14:29:44.806 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:29:44.829 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:44.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:44.831 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:29:44.932 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:29:44.983 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:44.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:44.984 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:29:45.061 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:29:44 smithi045 podman[102100]: 2024-08-19 14:29:44.799203499 +0000 UTC m=+0.155397872 container create cf08abdd92bff21b8971545e5c2328af695bc08a4dd9b8117d9789853dd5dcae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=squid-de09dd0, org.label-schema.license=GPLv2) 2024-08-19T14:29:45.062 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:29:45 smithi045 podman[102100]: 2024-08-19 14:29:45.035634595 +0000 UTC m=+0.391828934 container init cf08abdd92bff21b8971545e5c2328af695bc08a4dd9b8117d9789853dd5dcae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10, GIT_CLEAN=True, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-de09dd0, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2) 2024-08-19T14:29:45.062 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:29:45 smithi045 podman[102100]: 2024-08-19 14:29:45.039395235 +0000 UTC m=+0.395589567 container start cf08abdd92bff21b8971545e5c2328af695bc08a4dd9b8117d9789853dd5dcae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=squid-de09dd0, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.36.0, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-08-19T14:29:45.084 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:29:45.107 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:45.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:45.108 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:29:45.128 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-08-19T14:29:45.142 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-08-19T14:29:45.208 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:29:45.237 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:45.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:45.238 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:29:45.339 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:29:45.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:45 smithi031 ceph-mon[16336]: pgmap v1230: 47 pgs: 8 active+recovering+undersized+remapped, 39 active+clean; 5.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 38 MiB/s wr, 613 op/s; 32204/276531 objects misplaced (11.646%); 15 MiB/s, 245 objects/s recovering 2024-08-19T14:29:45.371 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:45.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:45.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:29:45.473 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:29:45.481 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:29:45.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:45.507 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:45.507 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:29:45.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:45 smithi032 ceph-mon[20004]: pgmap v1230: 47 pgs: 8 active+recovering+undersized+remapped, 39 active+clean; 5.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 38 MiB/s wr, 613 op/s; 32204/276531 objects misplaced (11.646%); 15 MiB/s, 245 objects/s recovering 2024-08-19T14:29:45.511 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 67 16 93552 93536 87.2465 39.75 0.0179699 0.0114577 2024-08-19T14:29:45.520 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:29:45.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:45 smithi045 ceph-mon[20143]: pgmap v1230: 47 pgs: 8 active+recovering+undersized+remapped, 39 active+clean; 5.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 38 MiB/s wr, 613 op/s; 32204/276531 objects misplaced (11.646%); 15 MiB/s, 245 objects/s recovering 2024-08-19T14:29:45.546 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:29:45 smithi045 ceph-osd[102135]: -- 172.21.15.45:0/2058241107 <== mon.1 v2:172.21.15.45:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560b87583a40 con 0x560b88346400 2024-08-19T14:29:45.546 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:29:45 smithi045 bash[102100]: cf08abdd92bff21b8971545e5c2328af695bc08a4dd9b8117d9789853dd5dcae 2024-08-19T14:29:45.546 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:29:45 smithi045 systemd[1]: Started Ceph osd.10 for ed7d1c6e-5e33-11ef-bccf-c7b262605968. 2024-08-19T14:29:45.608 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:29:45.636 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:45.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:45.637 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:29:45.737 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:29:45.774 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:45.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:45.778 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:29:45.879 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:29:45.912 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:29:45 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:29:45.578+0000 7fbc19db4640 -1 Falling back to public interface 2024-08-19T14:29:45.919 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:45.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:45.920 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:29:45.933 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:29:46.021 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:29:46.058 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:46.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:46.059 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:29:46.091 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:29:46.091 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-08-19T14:29:46.091 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-08-19T14:29:46.091 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:29:46.104 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is set 2024-08-19T14:29:46.160 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:29:46.188 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:46.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:46.189 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:29:46.191 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:46 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:29:46.191 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:46 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:29:46.191 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:46 smithi045 ceph-mon[20143]: osdmap e536: 12 total, 11 up, 8 in 2024-08-19T14:29:46.191 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:46 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2363638933' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:29:46.191 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:46 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:29:46.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:46 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:29:46.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:46 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:29:46.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:46 smithi032 ceph-mon[20004]: osdmap e536: 12 total, 11 up, 8 in 2024-08-19T14:29:46.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:46 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2363638933' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:29:46.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:46 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:29:46.290 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:29:46.291 INFO:teuthology.orchestra.run.smithi045.stderr:no valid command found; 10 closest matches: 2024-08-19T14:29:46.291 INFO:teuthology.orchestra.run.smithi045.stderr:0 2024-08-19T14:29:46.291 INFO:teuthology.orchestra.run.smithi045.stderr:1 2024-08-19T14:29:46.291 INFO:teuthology.orchestra.run.smithi045.stderr:2 2024-08-19T14:29:46.291 INFO:teuthology.orchestra.run.smithi045.stderr:abort 2024-08-19T14:29:46.291 INFO:teuthology.orchestra.run.smithi045.stderr:assert 2024-08-19T14:29:46.291 INFO:teuthology.orchestra.run.smithi045.stderr:config diff 2024-08-19T14:29:46.291 INFO:teuthology.orchestra.run.smithi045.stderr:config diff get 2024-08-19T14:29:46.292 INFO:teuthology.orchestra.run.smithi045.stderr:config get 2024-08-19T14:29:46.292 INFO:teuthology.orchestra.run.smithi045.stderr:config help [] 2024-08-19T14:29:46.292 INFO:teuthology.orchestra.run.smithi045.stderr:config set ... 2024-08-19T14:29:46.292 INFO:teuthology.orchestra.run.smithi045.stderr:admin_socket: invalid command 2024-08-19T14:29:46.297 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:46 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:29:46.297 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:46 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:29:46.298 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:46 smithi031 ceph-mon[16336]: osdmap e536: 12 total, 11 up, 8 in 2024-08-19T14:29:46.298 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:46 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2363638933' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:29:46.298 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:46 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:29:46.352 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:46.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:46.353 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:29:46.454 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:29:46.479 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:46.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:46.480 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:29:46.512 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 68 16 94200 94184 86.5591 40.5 0.0204599 0.0115476 2024-08-19T14:29:46.581 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:29:46.623 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:46.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:46.624 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:29:46.725 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:29:46.749 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:46.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:46.750 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:29:46.851 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:29:46.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:46.893 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:46.894 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:29:46.911 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-08-19T14:29:46.996 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:29:47.033 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-19T14:29:47.033 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-10, ['dump_ops_in_flight'] 2024-08-19T14:29:47.042 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:47.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:47.044 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:29:47.145 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:29:47.177 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:47.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:47.178 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:29:47.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:47 smithi032 ceph-mon[20004]: pgmap v1232: 47 pgs: 8 active+recovering+undersized+remapped, 39 active+clean; 5.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 38 MiB/s wr, 610 op/s; 31975/277044 objects misplaced (11.541%); 15 MiB/s, 247 objects/s recovering 2024-08-19T14:29:47.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:47 smithi032 ceph-mon[20004]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:29:47.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:47 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:29:47.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:47 smithi032 ceph-mon[20004]: osdmap e537: 12 total, 11 up, 8 in 2024-08-19T14:29:47.279 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:29:47.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:47 smithi045 ceph-mon[20143]: pgmap v1232: 47 pgs: 8 active+recovering+undersized+remapped, 39 active+clean; 5.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 38 MiB/s wr, 610 op/s; 31975/277044 objects misplaced (11.541%); 15 MiB/s, 247 objects/s recovering 2024-08-19T14:29:47.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:47 smithi045 ceph-mon[20143]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:29:47.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:47 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:29:47.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:47 smithi045 ceph-mon[20143]: osdmap e537: 12 total, 11 up, 8 in 2024-08-19T14:29:47.320 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:47.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:47.321 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:29:47.338 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:29:47.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:47 smithi031 ceph-mon[16336]: pgmap v1232: 47 pgs: 8 active+recovering+undersized+remapped, 39 active+clean; 5.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 38 MiB/s wr, 610 op/s; 31975/277044 objects misplaced (11.541%); 15 MiB/s, 247 objects/s recovering 2024-08-19T14:29:47.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:47 smithi031 ceph-mon[16336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:29:47.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:47 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:29:47.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:47 smithi031 ceph-mon[16336]: osdmap e537: 12 total, 11 up, 8 in 2024-08-19T14:29:47.422 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:29:47.457 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:47.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:47.458 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:29:47.512 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 69 16 94896 94880 85.935 43.5 0.013596 0.0116307 2024-08-19T14:29:47.558 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:29:47.585 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:47.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:47.586 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:29:47.687 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:29:47.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:47.719 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:47.719 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:29:47.820 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:29:47.854 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:47.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:47.855 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:29:47.956 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:29:47.978 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:47.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:47.979 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:29:48.080 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:29:48.109 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:48.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:48.111 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:29:48.212 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:29:48.242 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:48.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:48.243 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:29:48.344 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:29:48.366 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:48.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:48.367 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:29:48.468 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:29:48.509 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:48.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:48.511 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:29:48.512 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 70 16 95560 95544 85.3001 41.5 0.0361258 0.0117172 2024-08-19T14:29:48.546 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:29:48 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:29:48.274+0000 7fbc19db4640 -1 osd.10 529 log_to_monitors true 2024-08-19T14:29:48.612 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:29:48.645 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:48.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:48.647 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:29:48.747 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:29:48.777 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:48.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:48.778 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:29:48.879 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:29:48.900 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:48.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:48.901 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:29:49.002 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:29:49.023 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:49.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:49.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:29:49.125 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:29:49.167 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:49.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:49.168 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:29:49.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:49 smithi032 ceph-mon[20004]: pgmap v1234: 47 pgs: 8 active+recovering+undersized+remapped, 39 active+clean; 5.7 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 30 MiB/s wr, 486 op/s; 31668/278823 objects misplaced (11.358%); 14 MiB/s, 228 objects/s recovering 2024-08-19T14:29:49.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:49 smithi032 ceph-mon[20004]: from='osd.10 [v2:172.21.15.45:6816/1069743893,v1:172.21.15.45:6817/1069743893]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-08-19T14:29:49.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:49 smithi032 ceph-mon[20004]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-08-19T14:29:49.269 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:29:49.292 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:49.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:49.293 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:29:49.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:49 smithi031 ceph-mon[16336]: pgmap v1234: 47 pgs: 8 active+recovering+undersized+remapped, 39 active+clean; 5.7 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 30 MiB/s wr, 486 op/s; 31668/278823 objects misplaced (11.358%); 14 MiB/s, 228 objects/s recovering 2024-08-19T14:29:49.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:49 smithi031 ceph-mon[16336]: from='osd.10 [v2:172.21.15.45:6816/1069743893,v1:172.21.15.45:6817/1069743893]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-08-19T14:29:49.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:49 smithi031 ceph-mon[16336]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-08-19T14:29:49.395 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:29:49.427 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:49.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:49.428 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:29:49.512 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 71 16 96120 96104 84.5916 35 0.0125127 0.011796 2024-08-19T14:29:49.529 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:29:49.546 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:29:49 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:29:49.115+0000 7fbc0db57640 -1 osd.10 529 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-19T14:29:49.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:49 smithi045 ceph-mon[20143]: pgmap v1234: 47 pgs: 8 active+recovering+undersized+remapped, 39 active+clean; 5.7 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 30 MiB/s wr, 486 op/s; 31668/278823 objects misplaced (11.358%); 14 MiB/s, 228 objects/s recovering 2024-08-19T14:29:49.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:49 smithi045 ceph-mon[20143]: from='osd.10 [v2:172.21.15.45:6816/1069743893,v1:172.21.15.45:6817/1069743893]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-08-19T14:29:49.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:49 smithi045 ceph-mon[20143]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-08-19T14:29:49.568 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:49.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:49.569 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:29:49.643 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:29:49.670 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:29:49.698 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:49.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:49.699 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:29:49.800 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:29:49.822 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:49.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:49.823 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:29:49.924 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:29:49.953 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:49.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:49.954 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:29:50.055 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:29:50.091 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:50.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:50.092 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:29:50.193 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:29:50.220 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:50.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:50.221 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:29:50.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:50 smithi032 ceph-mon[20004]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-08-19T14:29:50.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:50 smithi032 ceph-mon[20004]: from='osd.10 [v2:172.21.15.45:6816/1069743893,v1:172.21.15.45:6817/1069743893]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-08-19T14:29:50.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:50 smithi032 ceph-mon[20004]: osdmap e538: 12 total, 11 up, 8 in 2024-08-19T14:29:50.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:50 smithi032 ceph-mon[20004]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-08-19T14:29:50.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:50 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-08-19T14:29:50.302 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:29:50.302 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:29:50.322 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:29:50.337 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:29:50.337 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:29:50.347 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:50.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:50.348 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:29:50.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:50 smithi031 ceph-mon[16336]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-08-19T14:29:50.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:50 smithi031 ceph-mon[16336]: from='osd.10 [v2:172.21.15.45:6816/1069743893,v1:172.21.15.45:6817/1069743893]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-08-19T14:29:50.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:50 smithi031 ceph-mon[16336]: osdmap e538: 12 total, 11 up, 8 in 2024-08-19T14:29:50.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:50 smithi031 ceph-mon[16336]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-08-19T14:29:50.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:50 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-08-19T14:29:50.373 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:29:50.373 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:29:50.414 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:29:50.414 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:29:50.446 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:29:50.446 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:29:50.449 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:29:50.485 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:29:50.485 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:29:50.485 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:50.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:50.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:29:50.512 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 72 16 96717 96701 83.9349 37.3125 0.0348093 0.0119091 2024-08-19T14:29:50.525 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:29:50.526 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:29:50.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:50 smithi045 ceph-mon[20143]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-08-19T14:29:50.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:50 smithi045 ceph-mon[20143]: from='osd.10 [v2:172.21.15.45:6816/1069743893,v1:172.21.15.45:6817/1069743893]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-08-19T14:29:50.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:50 smithi045 ceph-mon[20143]: osdmap e538: 12 total, 11 up, 8 in 2024-08-19T14:29:50.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:50 smithi045 ceph-mon[20143]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-08-19T14:29:50.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:50 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-08-19T14:29:50.562 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:29:50.562 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:29:50.588 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:29:50.594 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:29:50.595 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:29:50.620 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:50.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:50.622 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:29:50.623 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:29:50.623 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:29:50.646 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:29:50.646 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_enable_op_tracker = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2024-08-19T14:29:50.669 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:29:50.669 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:29:50.723 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:29:50.747 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:50.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:50.748 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:29:50.849 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:29:50.880 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:50.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:50.881 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:29:50.982 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:29:51.011 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:51.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:51.012 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:29:51.113 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:29:51.150 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:51.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:51.151 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:29:51.252 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:29:51.293 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:51.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:51.294 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:29:51.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:51 smithi031 ceph-mon[16336]: pgmap v1236: 47 pgs: 8 active+recovering+undersized+remapped, 39 active+clean; 5.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 34 MiB/s wr, 551 op/s; 31028/286452 objects misplaced (10.832%); 12 MiB/s, 195 objects/s recovering 2024-08-19T14:29:51.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:51 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-08-19T14:29:51.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:51 smithi031 ceph-mon[16336]: osd.10 [v2:172.21.15.45:6816/1069743893,v1:172.21.15.45:6817/1069743893] boot 2024-08-19T14:29:51.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:51 smithi031 ceph-mon[16336]: osdmap e539: 12 total, 12 up, 8 in 2024-08-19T14:29:51.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:51 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1208083269' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:29:51.395 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:29:51.432 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:51.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:51.433 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:29:51.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:51 smithi032 ceph-mon[20004]: pgmap v1236: 47 pgs: 8 active+recovering+undersized+remapped, 39 active+clean; 5.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 34 MiB/s wr, 551 op/s; 31028/286452 objects misplaced (10.832%); 12 MiB/s, 195 objects/s recovering 2024-08-19T14:29:51.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:51 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-08-19T14:29:51.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:51 smithi032 ceph-mon[20004]: osd.10 [v2:172.21.15.45:6816/1069743893,v1:172.21.15.45:6817/1069743893] boot 2024-08-19T14:29:51.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:51 smithi032 ceph-mon[20004]: osdmap e539: 12 total, 12 up, 8 in 2024-08-19T14:29:51.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:51 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1208083269' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:29:51.514 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-19T14:29:51.514 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 73 16 97245 97229 83.2372 33 0.0162141 0.0120078 2024-08-19T14:29:51.534 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:29:51.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:51 smithi045 ceph-mon[20143]: pgmap v1236: 47 pgs: 8 active+recovering+undersized+remapped, 39 active+clean; 5.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 34 MiB/s wr, 551 op/s; 31028/286452 objects misplaced (10.832%); 12 MiB/s, 195 objects/s recovering 2024-08-19T14:29:51.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:51 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-08-19T14:29:51.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:51 smithi045 ceph-mon[20143]: osd.10 [v2:172.21.15.45:6816/1069743893,v1:172.21.15.45:6817/1069743893] boot 2024-08-19T14:29:51.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:51 smithi045 ceph-mon[20143]: osdmap e539: 12 total, 12 up, 8 in 2024-08-19T14:29:51.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:51 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1208083269' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:29:51.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:51.596 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:51.597 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:29:51.698 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:29:51.745 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:51.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:51.746 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:29:51.846 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:29:51.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:51.928 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:51.928 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:29:51.973 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:29:52.029 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:29:52.034 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-08-19T14:29:52.051 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:52.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:52.052 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:29:52.152 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:29:52.174 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:29:52.174 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:29:52.191 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:52.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:52.192 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:29:52.293 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:29:52.322 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:52.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:52.323 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:29:52.403 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:29:52.424 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:29:52.458 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:52.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:52.460 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:29:52.512 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 74 16 97820 97804 82.5979 35.9375 0.0165448 0.0121004 2024-08-19T14:29:52.560 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:29:52.582 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:52.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:52.583 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:29:52.684 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:29:52.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:52.723 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:52.723 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:29:52.824 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:29:52.852 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:52.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:52.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:29:52.908 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:29:52.909 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:29:52.954 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:29:52.979 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:52.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:52.980 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:29:53.081 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:29:53.108 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:53.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:53.109 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:29:53.143 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is set 2024-08-19T14:29:53.210 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:29:53.238 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:53.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:53.239 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:29:53.340 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:29:53.383 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:53.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:53.384 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:29:53.485 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:29:53.496 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:29:53.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:53 smithi032 ceph-mon[20004]: pgmap v1238: 47 pgs: 8 active+recovering+undersized+remapped, 39 active+clean; 5.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 35 MiB/s wr, 555 op/s; 30781/287043 objects misplaced (10.723%); 12 MiB/s, 198 objects/s recovering 2024-08-19T14:29:53.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:53 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/692319216' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:29:53.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:53.508 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:53.508 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:29:53.512 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 75 16 98435 98419 82.0091 38.4375 0.0154333 0.0121876 2024-08-19T14:29:53.537 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:53 smithi031 ceph-mon[16336]: pgmap v1238: 47 pgs: 8 active+recovering+undersized+remapped, 39 active+clean; 5.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 35 MiB/s wr, 555 op/s; 30781/287043 objects misplaced (10.723%); 12 MiB/s, 198 objects/s recovering 2024-08-19T14:29:53.537 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:53 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/692319216' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:29:53.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:53 smithi045 ceph-mon[20143]: pgmap v1238: 47 pgs: 8 active+recovering+undersized+remapped, 39 active+clean; 5.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 35 MiB/s wr, 555 op/s; 30781/287043 objects misplaced (10.723%); 12 MiB/s, 198 objects/s recovering 2024-08-19T14:29:53.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:53 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/692319216' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:29:53.609 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:29:53.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:53.693 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:53.693 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:29:53.795 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:29:53.817 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:53.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:53.818 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:29:53.920 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:29:53.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:53.976 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:53.977 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:29:54.078 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:29:54.100 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:54.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:54.101 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:29:54.202 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:29:54.232 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:54.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:54.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:29:54.334 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:29:54.355 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:54.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:54.356 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:29:54.457 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:29:54.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:54.487 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:54.487 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:29:54.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:54 smithi032 ceph-mon[20004]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:29:54.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:54 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/692319216' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:29:54.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:54 smithi032 ceph-mon[20004]: osdmap e540: 12 total, 12 up, 8 in 2024-08-19T14:29:54.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:54 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:29:54.512 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 76 16 98970 98954 81.37 33.4375 0.0058764 0.012285 2024-08-19T14:29:54.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:54 smithi045 ceph-mon[20143]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:29:54.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:54 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/692319216' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:29:54.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:54 smithi045 ceph-mon[20143]: osdmap e540: 12 total, 12 up, 8 in 2024-08-19T14:29:54.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:54 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:29:54.590 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:29:54.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:54 smithi031 ceph-mon[16336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:29:54.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:54 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/692319216' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:29:54.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:54 smithi031 ceph-mon[16336]: osdmap e540: 12 total, 12 up, 8 in 2024-08-19T14:29:54.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:54 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:29:54.633 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:54.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:54.634 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:29:54.736 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:29:54.757 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:54.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:54.758 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:29:54.859 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:29:54.881 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:54.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:54.882 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:29:54.983 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:29:55.004 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:55.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:55.005 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:29:55.106 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:29:55.127 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:55.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:55.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:29:55.229 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:29:55.251 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:55.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:55.252 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:29:55.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:55 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:29:55.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:55 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:29:55.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:55 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:29:55.353 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:29:55.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:55.376 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:55.376 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:29:55.446 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:55 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:29:55.447 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:55 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:29:55.447 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:55 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:29:55.477 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:29:55.504 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:55.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:55.506 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:29:55.512 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 77 16 99453 99437 80.7053 30.1875 0.0204571 0.0123863 2024-08-19T14:29:55.606 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:29:55.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:55 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:29:55.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:55 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:29:55.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:55 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:29:55.617 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-08-19T14:29:55.630 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:55.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:55.630 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:29:55.731 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:29:55.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:55.768 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:55.768 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:29:55.869 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:29:55.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:55.914 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:55.914 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:29:55.993 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:29:56.015 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:29:56.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:56.038 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:56.039 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:29:56.140 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:29:56.165 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:56.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:56.166 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:29:56.267 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:29:56.304 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:56.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:56.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:29:56.406 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:29:56.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:56.455 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:56.455 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:29:56.512 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 78 16 100057 100041 80.1545 37.75 0.0217709 0.0124711 2024-08-19T14:29:56.556 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:29:56.577 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:56.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:56.578 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:29:56.679 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:29:56.705 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:56.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:56.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:29:56.807 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:29:56.830 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:56.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:56.831 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:29:56.932 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:29:56.958 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:56.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:56.959 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:29:57.060 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:29:57.105 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:57.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:57.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:29:57.207 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:29:57.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:57.231 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:57.231 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:29:57.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:57 smithi045 ceph-mon[20143]: pgmap v1241: 47 pgs: 8 active+recovering+undersized+remapped, 39 active+clean; 6.0 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 27 MiB/s wr, 438 op/s; 29478/295599 objects misplaced (9.972%); 14 MiB/s, 223 objects/s recovering 2024-08-19T14:29:57.332 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:29:57.366 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:57.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:57.367 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:29:57.469 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:29:57.490 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:57.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:57.491 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:29:57.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:57 smithi032 ceph-mon[20004]: pgmap v1241: 47 pgs: 8 active+recovering+undersized+remapped, 39 active+clean; 6.0 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 27 MiB/s wr, 438 op/s; 29478/295599 objects misplaced (9.972%); 14 MiB/s, 223 objects/s recovering 2024-08-19T14:29:57.512 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 79 16 100698 100682 79.647 40.0625 0.0357623 0.0125495 2024-08-19T14:29:57.592 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:29:57.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:57 smithi031 ceph-mon[16336]: pgmap v1241: 47 pgs: 8 active+recovering+undersized+remapped, 39 active+clean; 6.0 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 27 MiB/s wr, 438 op/s; 29478/295599 objects misplaced (9.972%); 14 MiB/s, 223 objects/s recovering 2024-08-19T14:29:57.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:57.629 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:57.629 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:29:57.730 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:29:57.779 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:57.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:57.780 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:29:57.799 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:29:57.799 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:29:57.881 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:29:57.908 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:57.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:57.909 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:29:58.010 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:29:58.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:58.053 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:58.053 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:29:58.154 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:29:58.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:58.199 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:58.199 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:29:58.272 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:29:58.300 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:29:58.369 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:58.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:58.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:29:58.471 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:29:58.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:58.495 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:58.495 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:29:58.513 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:2024-08-19T14:29:58.513283+0000 min lat: 0.00119513 max lat: 2.46333 avg lat: 0.0126389 2024-08-19T14:29:58.513 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-19T14:29:58.513 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 80 16 101262 101246 79.092 35.25 0.00834518 0.0126389 2024-08-19T14:29:58.562 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-08-19T14:29:58.596 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:29:58.629 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 7, 8, 2] out_osds: [10, 6, 9, 11] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 10] 2024-08-19T14:29:58.629 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-19T14:29:58.629 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-08-19T14:29:58.629 INFO:tasks.thrashosds.thrasher:Setting osd 8 primary_affinity to 1.000000 2024-08-19T14:29:58.629 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd primary-affinity 8 1 2024-08-19T14:29:58.646 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:58.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:58.647 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:29:58.748 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:29:58.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:58.779 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:58.779 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:29:58.880 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:29:58.926 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:29:58.938 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:58.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:58.939 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:29:59.003 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:29:59.003 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:29:59.040 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:29:59.044 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:29:59.045 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:29:59.070 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:59.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:59.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:29:59.086 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:29:59.087 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:29:59.124 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:29:59.124 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:29:59.161 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:29:59.161 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:29:59.172 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:29:59.193 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:59.194 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:29:59.194 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:29:59.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:59.195 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:29:59.228 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:29:59.229 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:29:59.262 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:29:59.263 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:29:59.295 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:29:59.298 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:29:59.299 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:29:59.299 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:59 smithi031 ceph-mon[16336]: pgmap v1242: 47 pgs: 8 active+recovering+undersized+remapped, 39 active+clean; 6.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 28 MiB/s wr, 450 op/s; 29210/297258 objects misplaced (9.826%); 14 MiB/s, 227 objects/s recovering 2024-08-19T14:29:59.299 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:29:59 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3352386374' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:29:59.317 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:59.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:59.318 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:29:59.330 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:29:59.331 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:29:59.367 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:29:59.367 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:29:59.389 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:29:59.393 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:59 smithi032 ceph-mon[20004]: pgmap v1242: 47 pgs: 8 active+recovering+undersized+remapped, 39 active+clean; 6.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 28 MiB/s wr, 450 op/s; 29210/297258 objects misplaced (9.826%); 14 MiB/s, 227 objects/s recovering 2024-08-19T14:29:59.393 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:29:59 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3352386374' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:29:59.403 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:29:59.403 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:29:59.419 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:29:59.455 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:59.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:59.456 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:29:59.513 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 81 16 101803 101787 78.533 33.8125 0.00925109 0.0127281 2024-08-19T14:29:59.539 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:29:59.540 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-08-19T14:29:59.540 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:59.540 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26191.0:101803 6.5 6:ab096ce0:::benchmark_data_smithi045_97937_object101802:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e540)", 2024-08-19T14:29:59.540 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:29:59.514103+0000", 2024-08-19T14:29:59.540 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.024863330999999999, 2024-08-19T14:29:59.540 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.024905884, 2024-08-19T14:29:59.540 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:29:59.540 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:29:59.540 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "waiting for sub ops", 2024-08-19T14:29:59.540 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:29:59.540 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26191", 2024-08-19T14:29:59.541 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:29:59.541 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 101803 2024-08-19T14:29:59.541 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:59.541 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:29:59.541 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:59.541 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:29:59.541 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:59.514103+0000", 2024-08-19T14:29:59.541 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:59.541 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:59.541 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:59.541 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:29:59.541 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:59.514103+0000", 2024-08-19T14:29:59.541 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:59.542 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:59.542 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:59.542 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:29:59.542 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:59.514104+0000", 2024-08-19T14:29:59.542 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8190000000000001e-06 2024-08-19T14:29:59.542 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:59.542 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:59.542 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:29:59.542 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:59.514139+0000", 2024-08-19T14:29:59.542 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5027999999999997e-05 2024-08-19T14:29:59.543 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:59.543 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:59.543 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:29:59.543 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:59.514160+0000", 2024-08-19T14:29:59.543 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0556000000000001e-05 2024-08-19T14:29:59.543 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:59.543 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:59.543 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:29:59.543 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:59.514169+0000", 2024-08-19T14:29:59.543 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.9979999999999992e-06 2024-08-19T14:29:59.543 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:59.543 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:59.544 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:29:59.544 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:59.518578+0000", 2024-08-19T14:29:59.544 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0044088240000000004 2024-08-19T14:29:59.544 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:59.544 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:59.544 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:29:59.544 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:59.518752+0000", 2024-08-19T14:29:59.544 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000174365 2024-08-19T14:29:59.544 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:59.544 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:59.544 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,5", 2024-08-19T14:29:59.544 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:59.518936+0000", 2024-08-19T14:29:59.545 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00018434300000000001 2024-08-19T14:29:59.545 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:59.545 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:59.545 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:29:59.545 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:59.522588+0000", 2024-08-19T14:29:59.545 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0036513370000000002 2024-08-19T14:29:59.545 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:59.545 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:59.545 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:59.545 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:59.528103+0000", 2024-08-19T14:29:59.545 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0055151949999999996 2024-08-19T14:29:59.546 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:59.546 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:29:59.546 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:59.546 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:59.546 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:59.546 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26191.0:101805 6.5 6:a723bc8f:::benchmark_data_smithi045_97937_object101804:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e540)", 2024-08-19T14:29:59.546 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:29:59.516733+0000", 2024-08-19T14:29:59.546 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.022232450000000001, 2024-08-19T14:29:59.546 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.022332818000000001, 2024-08-19T14:29:59.546 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:29:59.546 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:29:59.546 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "waiting for sub ops", 2024-08-19T14:29:59.546 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:29:59.547 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26191", 2024-08-19T14:29:59.547 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:29:59.547 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 101805 2024-08-19T14:29:59.547 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:59.547 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:29:59.547 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:59.547 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:29:59.547 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:59.516733+0000", 2024-08-19T14:29:59.547 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:59.547 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:59.547 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:59.547 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:29:59.547 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:59.516733+0000", 2024-08-19T14:29:59.548 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:29:59.548 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:59.548 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:59.548 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:29:59.548 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:59.516735+0000", 2024-08-19T14:29:59.548 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0329999999999998e-06 2024-08-19T14:29:59.548 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:59.548 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:59.548 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:29:59.548 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:59.516770+0000", 2024-08-19T14:29:59.548 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.4517999999999997e-05 2024-08-19T14:29:59.548 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:59.548 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:59.549 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:29:59.549 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:59.516790+0000", 2024-08-19T14:29:59.549 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9758e-05 2024-08-19T14:29:59.549 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:59.549 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:59.549 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:29:59.549 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:59.516801+0000", 2024-08-19T14:29:59.549 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1062e-05 2024-08-19T14:29:59.549 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:59.549 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:59.549 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:29:59.549 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:59.524542+0000", 2024-08-19T14:29:59.549 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0077413050000000004 2024-08-19T14:29:59.550 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:59.550 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:59.550 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:29:59.550 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:59.524694+0000", 2024-08-19T14:29:59.550 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000152292 2024-08-19T14:29:59.550 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:59.550 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:59.550 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,5", 2024-08-19T14:29:59.550 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:59.524846+0000", 2024-08-19T14:29:59.550 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000151972 2024-08-19T14:29:59.550 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:59.550 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:59.551 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:29:59.551 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:59.528226+0000", 2024-08-19T14:29:59.551 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.003379426 2024-08-19T14:29:59.551 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:29:59.551 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:29:59.551 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:29:59.551 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:59.528815+0000", 2024-08-19T14:29:59.551 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00058882900000000002 2024-08-19T14:29:59.551 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:59.551 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:29:59.551 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:59.551 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:29:59.551 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-08-19T14:29:59.552 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 2 2024-08-19T14:29:59.552 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:29:59.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:59 smithi045 ceph-mon[20143]: pgmap v1242: 47 pgs: 8 active+recovering+undersized+remapped, 39 active+clean; 6.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 28 MiB/s wr, 450 op/s; 29210/297258 objects misplaced (9.826%); 14 MiB/s, 227 objects/s recovering 2024-08-19T14:29:59.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:29:59 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3352386374' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:29:59.558 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:29:59.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:59.592 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:29:59.592 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:29:59.694 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:29:59.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:59.760 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:29:59.760 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:29:59.861 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:29:59.882 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:29:59.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:29:59.883 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:29:59.984 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:30:00.014 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:00.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:00.016 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:30:00.117 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:30:00.140 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:00.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:00.141 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:30:00.242 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:30:00.251 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-19T14:30:00.263 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:00.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:00.264 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:30:00.319 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-08-19T14:30:00.365 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:30:00.387 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:00.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:00.388 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:30:00.438 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:00 smithi031 ceph-mon[16336]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set 2024-08-19T14:30:00.438 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:00 smithi031 ceph-mon[16336]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-08-19T14:30:00.488 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:30:00.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:00 smithi032 ceph-mon[20004]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set 2024-08-19T14:30:00.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:00 smithi032 ceph-mon[20004]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-08-19T14:30:00.514 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 82 16 102445 102429 78.0645 40.125 0.0445802 0.0128048 2024-08-19T14:30:00.516 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:00.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:00.517 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:30:00.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:00 smithi045 ceph-mon[20143]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set 2024-08-19T14:30:00.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:00 smithi045 ceph-mon[20143]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-08-19T14:30:00.618 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:30:00.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:00.670 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:00.670 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:30:00.683 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:30:00.771 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:30:00.799 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:00.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:00.800 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:30:00.901 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:30:00.930 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:00.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:00.931 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:30:01.032 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:30:01.054 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:01.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:01.055 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:30:01.156 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:30:01.180 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:01.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:01.181 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:30:01.282 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:30:01.304 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:01.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:01.305 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:30:01.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:01 smithi031 ceph-mon[16336]: pgmap v1243: 47 pgs: 8 active+recovering+undersized+remapped, 39 active+clean; 6.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 36 MiB/s wr, 574 op/s; 28551/303582 objects misplaced (9.405%); 16 MiB/s, 249 objects/s recovering 2024-08-19T14:30:01.388 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:30:01.406 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:30:01.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:01.436 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:01.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:30:01.503 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:01 smithi032 ceph-mon[20004]: pgmap v1243: 47 pgs: 8 active+recovering+undersized+remapped, 39 active+clean; 6.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 36 MiB/s wr, 574 op/s; 28551/303582 objects misplaced (9.405%); 16 MiB/s, 249 objects/s recovering 2024-08-19T14:30:01.513 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 83 16 103039 103023 77.5712 37.125 0.0531161 0.012886 2024-08-19T14:30:01.520 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:30:01.520 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-08-19T14:30:01.520 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-08-19T14:30:01.520 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-08-19T14:30:01.521 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:30:01.537 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:30:01.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:01 smithi045 ceph-mon[20143]: pgmap v1243: 47 pgs: 8 active+recovering+undersized+remapped, 39 active+clean; 6.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 36 MiB/s wr, 574 op/s; 28551/303582 objects misplaced (9.405%); 16 MiB/s, 249 objects/s recovering 2024-08-19T14:30:01.592 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:01.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:01.593 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:30:01.694 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:30:01.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:01.760 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:01.760 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:30:01.861 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:30:01.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:01.884 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:01.884 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:30:01.985 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:30:02.007 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:02.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:02.008 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:30:02.109 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:30:02.138 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:02.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:02.139 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:30:02.183 INFO:teuthology.orchestra.run.smithi031.stderr:set osd.8 primary-affinity to 1 (8655362) 2024-08-19T14:30:02.241 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:30:02.295 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:02.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:02.296 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:30:02.316 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:30:02.397 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:30:02.418 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:02.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:02.419 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:30:02.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:02 smithi032 ceph-mon[20004]: from='client.26351 172.21.15.31:0/4145086053' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]: dispatch 2024-08-19T14:30:02.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:02 smithi032 ceph-mon[20004]: from='client.26351 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]: dispatch 2024-08-19T14:30:02.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:02 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:30:02.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:02 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:30:02.513 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 84 16 103574 103558 77.0457 33.4375 0.0151813 0.0129737 2024-08-19T14:30:02.520 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:30:02.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:02 smithi045 ceph-mon[20143]: from='client.26351 172.21.15.31:0/4145086053' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]: dispatch 2024-08-19T14:30:02.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:02 smithi045 ceph-mon[20143]: from='client.26351 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]: dispatch 2024-08-19T14:30:02.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:02 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:30:02.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:02 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:30:02.548 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:02.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:02.550 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:30:02.571 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-08-19T14:30:02.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:02 smithi031 ceph-mon[16336]: from='client.26351 172.21.15.31:0/4145086053' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]: dispatch 2024-08-19T14:30:02.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:02 smithi031 ceph-mon[16336]: from='client.26351 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]: dispatch 2024-08-19T14:30:02.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:02 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:30:02.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:02 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:30:02.650 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:30:02.691 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:02.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:02.692 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:30:02.793 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:30:02.825 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:02.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:02.826 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:30:02.927 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:30:02.972 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:02.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:02.973 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:30:02.982 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:30:03.074 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:30:03.110 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:03.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:03.111 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:30:03.192 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is unset 2024-08-19T14:30:03.212 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:30:03.234 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:03.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:03.235 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:30:03.336 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:30:03.366 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:03.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:03.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:30:03.376 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:03 smithi032 ceph-mon[20004]: pgmap v1244: 47 pgs: 8 active+recovering+undersized+remapped, 39 active+clean; 6.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 36 MiB/s wr, 568 op/s; 28334/304119 objects misplaced (9.317%); 15 MiB/s, 244 objects/s recovering 2024-08-19T14:30:03.376 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:03 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:30:03.376 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:03 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:30:03.376 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:03 smithi032 ceph-mon[20004]: from='client.26351 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]': finished 2024-08-19T14:30:03.376 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:03 smithi032 ceph-mon[20004]: osdmap e541: 12 total, 12 up, 8 in 2024-08-19T14:30:03.376 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:03 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/83315867' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:30:03.376 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:03 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:30:03.469 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:30:03.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:03.502 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:03.502 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:30:03.513 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 85 16 104170 104154 76.5775 37.25 0.0290317 0.0130536 2024-08-19T14:30:03.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:03 smithi045 ceph-mon[20143]: pgmap v1244: 47 pgs: 8 active+recovering+undersized+remapped, 39 active+clean; 6.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 36 MiB/s wr, 568 op/s; 28334/304119 objects misplaced (9.317%); 15 MiB/s, 244 objects/s recovering 2024-08-19T14:30:03.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:03 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:30:03.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:03 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:30:03.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:03 smithi045 ceph-mon[20143]: from='client.26351 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]': finished 2024-08-19T14:30:03.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:03 smithi045 ceph-mon[20143]: osdmap e541: 12 total, 12 up, 8 in 2024-08-19T14:30:03.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:03 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/83315867' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:30:03.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:03 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:30:03.603 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:30:03.618 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:03 smithi031 ceph-mon[16336]: pgmap v1244: 47 pgs: 8 active+recovering+undersized+remapped, 39 active+clean; 6.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 36 MiB/s wr, 568 op/s; 28334/304119 objects misplaced (9.317%); 15 MiB/s, 244 objects/s recovering 2024-08-19T14:30:03.619 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:03 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:30:03.619 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:03 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:30:03.619 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:03 smithi031 ceph-mon[16336]: from='client.26351 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]': finished 2024-08-19T14:30:03.619 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:03 smithi031 ceph-mon[16336]: osdmap e541: 12 total, 12 up, 8 in 2024-08-19T14:30:03.619 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:03 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/83315867' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:30:03.619 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:03 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:30:03.647 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:03.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:03.648 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:30:03.749 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:30:03.778 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:03.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:03.779 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:30:03.880 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:30:03.921 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:03.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:03.922 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:30:04.022 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:30:04.058 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:04.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:04.059 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:30:04.160 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:30:04.204 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:04.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:04.205 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:30:04.306 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:30:04.337 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:04.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:04.338 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:30:04.439 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:30:04.462 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:04.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:04.463 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:30:04.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:04 smithi032 ceph-mon[20004]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:30:04.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:04 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:30:04.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:04 smithi032 ceph-mon[20004]: osdmap e542: 12 total, 12 up, 8 in 2024-08-19T14:30:04.513 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 86 16 104760 104744 76.1158 36.875 0.090951 0.013128 2024-08-19T14:30:04.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:04 smithi045 ceph-mon[20143]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:30:04.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:04 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:30:04.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:04 smithi045 ceph-mon[20143]: osdmap e542: 12 total, 12 up, 8 in 2024-08-19T14:30:04.552 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:30:04.552 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-08-19T14:30:04.552 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-08-19T14:30:04.552 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-08-19T14:30:04.552 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.552 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:92844 5.19 5:9cd3cd78:::benchmark_data_smithi045_85309_object92843:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e445)", 2024-08-19T14:30:04.552 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:25:37.246536+0000", 2024-08-19T14:30:04.553 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 267.30271528899999, 2024-08-19T14:30:04.553 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.188362328, 2024-08-19T14:30:04.553 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:30:04.553 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:30:04.553 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:30:04.553 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:30:04.553 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:30:04.553 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:30:04.553 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 92844 2024-08-19T14:30:04.553 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.553 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:30:04.553 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.553 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:30:04.553 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.246536+0000", 2024-08-19T14:30:04.554 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:30:04.554 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.554 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.554 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:30:04.554 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.246536+0000", 2024-08-19T14:30:04.554 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:30:04.554 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.554 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.554 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:30:04.554 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.246537+0000", 2024-08-19T14:30:04.554 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2899999999999999e-06 2024-08-19T14:30:04.554 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.554 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.554 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:30:04.555 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.246579+0000", 2024-08-19T14:30:04.555 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2042000000000002e-05 2024-08-19T14:30:04.555 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.555 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.555 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:30:04.555 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.246589+0000", 2024-08-19T14:30:04.555 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0033e-05 2024-08-19T14:30:04.555 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.555 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.555 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:30:04.555 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.246603+0000", 2024-08-19T14:30:04.555 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4054e-05 2024-08-19T14:30:04.555 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.556 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.556 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:30:04.556 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.252793+0000", 2024-08-19T14:30:04.556 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0061897319999999999 2024-08-19T14:30:04.556 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.556 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.556 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:30:04.556 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.252929+0000", 2024-08-19T14:30:04.556 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013627800000000001 2024-08-19T14:30:04.556 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.556 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.556 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 1,5", 2024-08-19T14:30:04.556 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.253056+0000", 2024-08-19T14:30:04.556 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000126389 2024-08-19T14:30:04.556 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.557 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.557 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:30:04.557 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.254339+0000", 2024-08-19T14:30:04.557 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001283588 2024-08-19T14:30:04.557 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.557 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.557 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:30:04.557 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.260841+0000", 2024-08-19T14:30:04.557 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.006501522 2024-08-19T14:30:04.557 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.557 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.557 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:30:04.557 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.432465+0000", 2024-08-19T14:30:04.558 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.17162460500000001 2024-08-19T14:30:04.558 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.558 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.558 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:30:04.558 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.432578+0000", 2024-08-19T14:30:04.558 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000113151 2024-08-19T14:30:04.558 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.558 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.558 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:30:04.558 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.434898+0000", 2024-08-19T14:30:04.558 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.002319644 2024-08-19T14:30:04.558 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:30:04.558 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:30:04.558 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:30:04.559 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.559 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.559 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:181713 5.9 5:93e540c3:::benchmark_data_smithi045_85309_object61460:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e477)", 2024-08-19T14:30:04.559 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:27:15.012008+0000", 2024-08-19T14:30:04.559 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 169.53724292699999, 2024-08-19T14:30:04.559 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.13837481099999999, 2024-08-19T14:30:04.559 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:30:04.559 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:30:04.559 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:30:04.559 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:30:04.559 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:30:04.559 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:30:04.559 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 181713 2024-08-19T14:30:04.559 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.560 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:30:04.560 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.560 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:30:04.560 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:15.012008+0000", 2024-08-19T14:30:04.560 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:30:04.560 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.560 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.560 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:30:04.560 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:15.012008+0000", 2024-08-19T14:30:04.560 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:30:04.560 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.560 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.560 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:30:04.560 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:15.012010+0000", 2024-08-19T14:30:04.561 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7969999999999999e-06 2024-08-19T14:30:04.561 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.561 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.561 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:30:04.561 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:15.012018+0000", 2024-08-19T14:30:04.561 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.123e-06 2024-08-19T14:30:04.561 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.561 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.561 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:30:04.561 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:15.012039+0000", 2024-08-19T14:30:04.561 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0533000000000001e-05 2024-08-19T14:30:04.561 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.561 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.561 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:30:04.561 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:15.012048+0000", 2024-08-19T14:30:04.562 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.2920000000000001e-06 2024-08-19T14:30:04.562 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.562 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.562 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:30:04.562 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:15.012117+0000", 2024-08-19T14:30:04.562 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9443999999999994e-05 2024-08-19T14:30:04.562 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.562 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.562 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:30:04.562 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:15.012242+0000", 2024-08-19T14:30:04.562 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012462199999999999 2024-08-19T14:30:04.562 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.562 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.562 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 9,11", 2024-08-19T14:30:04.563 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:15.012346+0000", 2024-08-19T14:30:04.563 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00010425399999999999 2024-08-19T14:30:04.563 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.563 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.563 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:30:04.563 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:15.015227+0000", 2024-08-19T14:30:04.563 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0028807619999999998 2024-08-19T14:30:04.563 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.563 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.563 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:30:04.563 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:15.015530+0000", 2024-08-19T14:30:04.563 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0003032 2024-08-19T14:30:04.563 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.563 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.564 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:30:04.564 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:15.150281+0000", 2024-08-19T14:30:04.564 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.134750539 2024-08-19T14:30:04.564 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.564 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.564 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:30:04.564 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:15.150336+0000", 2024-08-19T14:30:04.564 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5535999999999998e-05 2024-08-19T14:30:04.564 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.564 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.564 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:30:04.564 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:15.150383+0000", 2024-08-19T14:30:04.564 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6709000000000003e-05 2024-08-19T14:30:04.564 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:30:04.564 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:30:04.565 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:30:04.565 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.565 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.565 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:237782 5.9 5:97a86776:::benchmark_data_smithi045_85309_object117529:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e495)", 2024-08-19T14:30:04.565 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:28:07.103392+0000", 2024-08-19T14:30:04.565 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 117.445858642, 2024-08-19T14:30:04.565 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.24805988200000001, 2024-08-19T14:30:04.565 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:30:04.565 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:30:04.565 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:30:04.565 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:30:04.565 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:30:04.565 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:30:04.566 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:30:04.566 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 237782 2024-08-19T14:30:04.567 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.567 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:30:04.567 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.568 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:30:04.568 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:28:07.103392+0000", 2024-08-19T14:30:04.568 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:30:04.568 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.568 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.568 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:30:04.568 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:28:07.103392+0000", 2024-08-19T14:30:04.568 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:30:04.568 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.568 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.568 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:30:04.569 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:28:07.103395+0000", 2024-08-19T14:30:04.569 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3180000000000001e-06 2024-08-19T14:30:04.569 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.569 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.569 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:30:04.569 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:28:07.103402+0000", 2024-08-19T14:30:04.569 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2089999999999999e-06 2024-08-19T14:30:04.569 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.569 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.569 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:30:04.569 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:28:07.103421+0000", 2024-08-19T14:30:04.569 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9570000000000001e-05 2024-08-19T14:30:04.569 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.570 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.570 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:30:04.570 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:28:07.103436+0000", 2024-08-19T14:30:04.570 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4103e-05 2024-08-19T14:30:04.570 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.570 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.570 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:30:04.570 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:28:07.103702+0000", 2024-08-19T14:30:04.570 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000266171 2024-08-19T14:30:04.570 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.570 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.570 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:30:04.570 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:28:07.103714+0000", 2024-08-19T14:30:04.571 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2150000000000001e-05 2024-08-19T14:30:04.571 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.571 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.571 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:30:04.571 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:28:07.335533+0000", 2024-08-19T14:30:04.571 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.23181948099999999 2024-08-19T14:30:04.571 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.571 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.571 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:30:04.571 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:28:07.335686+0000", 2024-08-19T14:30:04.571 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00015228 2024-08-19T14:30:04.571 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.571 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.571 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 1,11", 2024-08-19T14:30:04.572 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:28:07.336407+0000", 2024-08-19T14:30:04.572 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00072105900000000002 2024-08-19T14:30:04.572 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.572 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.572 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:30:04.572 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:28:07.343530+0000", 2024-08-19T14:30:04.572 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0071228680000000001 2024-08-19T14:30:04.572 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.572 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.573 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:30:04.573 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:28:07.350656+0000", 2024-08-19T14:30:04.573 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0071266419999999999 2024-08-19T14:30:04.573 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.573 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.573 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:30:04.573 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:28:07.350710+0000", 2024-08-19T14:30:04.573 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4143e-05 2024-08-19T14:30:04.573 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.573 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.573 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:30:04.573 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:28:07.350813+0000", 2024-08-19T14:30:04.573 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000102601 2024-08-19T14:30:04.574 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.574 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.574 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:30:04.574 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:28:07.351452+0000", 2024-08-19T14:30:04.574 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00063928699999999999 2024-08-19T14:30:04.574 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:30:04.574 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:30:04.574 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:30:04.574 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.574 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.574 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:237871 5.10 5:099edba3:::benchmark_data_smithi045_85309_object117618:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e495)", 2024-08-19T14:30:04.574 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:28:07.555218+0000", 2024-08-19T14:30:04.574 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 116.994032774, 2024-08-19T14:30:04.575 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.145422414, 2024-08-19T14:30:04.575 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:30:04.575 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:30:04.575 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:30:04.575 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:30:04.575 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:30:04.575 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:30:04.575 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 237871 2024-08-19T14:30:04.575 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.575 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:30:04.575 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.575 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:30:04.575 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:28:07.555218+0000", 2024-08-19T14:30:04.575 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:30:04.575 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.576 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.576 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:30:04.576 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:28:07.555218+0000", 2024-08-19T14:30:04.576 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:30:04.576 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.576 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.576 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:30:04.577 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:28:07.555220+0000", 2024-08-19T14:30:04.577 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.711e-06 2024-08-19T14:30:04.577 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.577 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.577 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:30:04.577 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:28:07.555226+0000", 2024-08-19T14:30:04.577 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3439999999999997e-06 2024-08-19T14:30:04.577 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.577 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.577 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:30:04.577 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:28:07.555239+0000", 2024-08-19T14:30:04.577 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2574e-05 2024-08-19T14:30:04.577 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.577 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.578 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:30:04.578 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:28:07.555246+0000", 2024-08-19T14:30:04.578 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3039999999999999e-06 2024-08-19T14:30:04.578 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.578 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.578 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:30:04.578 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:28:07.555312+0000", 2024-08-19T14:30:04.578 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.5568000000000006e-05 2024-08-19T14:30:04.578 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.579 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.579 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:30:04.579 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:28:07.555464+0000", 2024-08-19T14:30:04.579 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00015208600000000001 2024-08-19T14:30:04.579 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.579 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.579 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0,8", 2024-08-19T14:30:04.580 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:28:07.556078+0000", 2024-08-19T14:30:04.580 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00061452399999999997 2024-08-19T14:30:04.580 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.580 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.580 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:30:04.580 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:28:07.563613+0000", 2024-08-19T14:30:04.580 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0075351330000000003 2024-08-19T14:30:04.580 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.580 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.580 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:30:04.580 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:28:07.563670+0000", 2024-08-19T14:30:04.580 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6419000000000003e-05 2024-08-19T14:30:04.580 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.581 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.581 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:30:04.581 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:28:07.700575+0000", 2024-08-19T14:30:04.581 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.13690512899999999 2024-08-19T14:30:04.581 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.581 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.581 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:30:04.581 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:28:07.700610+0000", 2024-08-19T14:30:04.581 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.4801000000000001e-05 2024-08-19T14:30:04.581 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.581 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.581 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:30:04.581 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:28:07.700641+0000", 2024-08-19T14:30:04.581 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0821000000000002e-05 2024-08-19T14:30:04.582 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:30:04.582 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:30:04.582 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:30:04.582 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.582 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.582 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26191.0:52646 6.3 6:ccd54a02:::benchmark_data_smithi045_97937_object52645:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e520)", 2024-08-19T14:30:04.582 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:29:00.094100+0000", 2024-08-19T14:30:04.582 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 64.455151336, 2024-08-19T14:30:04.582 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.15301577999999999, 2024-08-19T14:30:04.582 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:30:04.582 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:30:04.582 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:30:04.582 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:30:04.583 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26191", 2024-08-19T14:30:04.583 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:30:04.583 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 52646 2024-08-19T14:30:04.583 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.583 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:30:04.583 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.583 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:30:04.583 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:00.094100+0000", 2024-08-19T14:30:04.583 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:30:04.583 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.584 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.584 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:30:04.584 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:00.094100+0000", 2024-08-19T14:30:04.584 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:30:04.584 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.584 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.584 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:30:04.584 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:00.094101+0000", 2024-08-19T14:30:04.585 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.212e-06 2024-08-19T14:30:04.585 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.585 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.585 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:30:04.585 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:00.094149+0000", 2024-08-19T14:30:04.585 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7981e-05 2024-08-19T14:30:04.585 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.585 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.585 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:30:04.585 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:00.094159+0000", 2024-08-19T14:30:04.586 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.8179999999999999e-06 2024-08-19T14:30:04.586 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.586 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.586 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:30:04.586 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:00.094175+0000", 2024-08-19T14:30:04.586 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6161e-05 2024-08-19T14:30:04.586 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.586 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.586 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:30:04.586 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:00.094233+0000", 2024-08-19T14:30:04.586 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8556e-05 2024-08-19T14:30:04.586 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.586 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.587 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:30:04.587 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:00.094396+0000", 2024-08-19T14:30:04.587 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000162905 2024-08-19T14:30:04.587 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.587 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.587 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0,8", 2024-08-19T14:30:04.587 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:00.094532+0000", 2024-08-19T14:30:04.587 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013584400000000001 2024-08-19T14:30:04.587 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.587 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.587 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:30:04.587 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:00.096492+0000", 2024-08-19T14:30:04.587 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0019602999999999999 2024-08-19T14:30:04.588 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.588 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.588 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:30:04.589 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:00.097942+0000", 2024-08-19T14:30:04.589 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001449105 2024-08-19T14:30:04.589 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.589 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.589 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:30:04.589 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:00.247008+0000", 2024-08-19T14:30:04.589 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.14906687399999999 2024-08-19T14:30:04.589 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.589 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.589 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:30:04.590 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:00.247068+0000", 2024-08-19T14:30:04.590 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9988999999999998e-05 2024-08-19T14:30:04.590 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.590 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.590 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:30:04.590 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:00.247115+0000", 2024-08-19T14:30:04.590 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7034999999999999e-05 2024-08-19T14:30:04.590 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:30:04.590 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:30:04.590 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:30:04.590 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.590 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.591 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26191.0:52655 6.3 6:c20f72c8:::benchmark_data_smithi045_97937_object52654:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e520)", 2024-08-19T14:30:04.591 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:29:00.098157+0000", 2024-08-19T14:30:04.591 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 64.451094157, 2024-08-19T14:30:04.591 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.15304041299999999, 2024-08-19T14:30:04.591 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:30:04.591 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:30:04.591 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:30:04.591 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:30:04.591 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26191", 2024-08-19T14:30:04.591 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:30:04.591 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 52655 2024-08-19T14:30:04.591 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.591 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:30:04.591 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.592 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:30:04.592 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:00.098157+0000", 2024-08-19T14:30:04.592 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:30:04.592 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.592 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.592 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:30:04.592 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:00.098157+0000", 2024-08-19T14:30:04.592 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:30:04.592 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.592 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.592 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:30:04.592 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:00.098158+0000", 2024-08-19T14:30:04.592 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.5199999999999995e-07 2024-08-19T14:30:04.592 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.593 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.593 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:30:04.593 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:00.098200+0000", 2024-08-19T14:30:04.593 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2585e-05 2024-08-19T14:30:04.593 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.593 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.593 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:30:04.593 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:00.098209+0000", 2024-08-19T14:30:04.593 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.7329999999999992e-06 2024-08-19T14:30:04.593 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.593 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.593 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:30:04.593 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:00.098214+0000", 2024-08-19T14:30:04.593 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7770000000000002e-06 2024-08-19T14:30:04.594 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.594 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.594 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:30:04.594 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:00.099510+0000", 2024-08-19T14:30:04.594 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001295909 2024-08-19T14:30:04.594 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.594 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.594 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:30:04.594 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:00.099642+0000", 2024-08-19T14:30:04.594 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013237000000000001 2024-08-19T14:30:04.594 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.594 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.594 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0,8", 2024-08-19T14:30:04.594 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:00.099762+0000", 2024-08-19T14:30:04.595 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000119552 2024-08-19T14:30:04.595 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.595 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.595 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:30:04.595 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:00.103620+0000", 2024-08-19T14:30:04.595 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0038587529999999999 2024-08-19T14:30:04.595 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.595 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.595 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:30:04.595 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:00.103630+0000", 2024-08-19T14:30:04.595 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.7289999999999995e-06 2024-08-19T14:30:04.595 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.595 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.595 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:30:04.596 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:00.251130+0000", 2024-08-19T14:30:04.596 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.14749949200000001 2024-08-19T14:30:04.596 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.596 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.596 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:30:04.596 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:00.251170+0000", 2024-08-19T14:30:04.596 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0200999999999997e-05 2024-08-19T14:30:04.596 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.596 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.596 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:30:04.596 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:00.251197+0000", 2024-08-19T14:30:04.596 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7359999999999999e-05 2024-08-19T14:30:04.596 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:30:04.596 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:30:04.596 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:30:04.597 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.597 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.597 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26191.0:52658 6.3 6:cc7d1f3b:::benchmark_data_smithi045_97937_object52657:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e520)", 2024-08-19T14:30:04.597 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:29:00.100007+0000", 2024-08-19T14:30:04.597 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 64.449243526000004, 2024-08-19T14:30:04.597 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.15134437000000001, 2024-08-19T14:30:04.597 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:30:04.597 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:30:04.597 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:30:04.597 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:30:04.597 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26191", 2024-08-19T14:30:04.597 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:30:04.597 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 52658 2024-08-19T14:30:04.597 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.598 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:30:04.598 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.598 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:30:04.598 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:00.100007+0000", 2024-08-19T14:30:04.598 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:30:04.598 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.598 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.598 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:30:04.598 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:00.100007+0000", 2024-08-19T14:30:04.598 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:30:04.598 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.598 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.598 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:30:04.599 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:00.100008+0000", 2024-08-19T14:30:04.602 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.1299999999999998e-07 2024-08-19T14:30:04.602 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.602 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.603 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:30:04.603 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:00.100085+0000", 2024-08-19T14:30:04.603 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6274999999999997e-05 2024-08-19T14:30:04.603 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.605 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.605 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:30:04.605 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:00.100096+0000", 2024-08-19T14:30:04.605 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1199999999999999e-05 2024-08-19T14:30:04.605 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.605 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.605 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:30:04.605 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:00.100104+0000", 2024-08-19T14:30:04.605 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6669999999999996e-06 2024-08-19T14:30:04.606 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.606 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.606 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:30:04.606 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:00.101482+0000", 2024-08-19T14:30:04.606 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0013786460000000001 2024-08-19T14:30:04.606 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.606 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.606 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:30:04.606 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:00.101704+0000", 2024-08-19T14:30:04.606 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00022199599999999999 2024-08-19T14:30:04.606 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.606 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.606 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0,8", 2024-08-19T14:30:04.607 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:00.101834+0000", 2024-08-19T14:30:04.607 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000129438 2024-08-19T14:30:04.607 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.607 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.607 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:30:04.607 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:00.103768+0000", 2024-08-19T14:30:04.607 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001934776 2024-08-19T14:30:04.607 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.607 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.607 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:30:04.607 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:00.104739+0000", 2024-08-19T14:30:04.607 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00097063399999999995 2024-08-19T14:30:04.607 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.607 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.607 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:30:04.608 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:00.251253+0000", 2024-08-19T14:30:04.608 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.146513633 2024-08-19T14:30:04.608 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.608 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.608 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:30:04.608 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:00.251310+0000", 2024-08-19T14:30:04.608 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7448000000000003e-05 2024-08-19T14:30:04.608 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.608 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.608 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:30:04.609 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:00.251352+0000", 2024-08-19T14:30:04.609 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1743999999999999e-05 2024-08-19T14:30:04.609 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:30:04.609 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:30:04.609 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:30:04.609 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.609 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.609 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26191.0:72559 6.3 6:c071ee9c:::benchmark_data_smithi045_97937_object72558:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e522)", 2024-08-19T14:30:04.609 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:29:12.401220+0000", 2024-08-19T14:30:04.609 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 52.148030511000002, 2024-08-19T14:30:04.610 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.15635301500000001, 2024-08-19T14:30:04.610 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:30:04.610 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:30:04.610 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:30:04.610 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:30:04.610 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26191", 2024-08-19T14:30:04.610 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:30:04.610 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 72559 2024-08-19T14:30:04.610 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.611 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:30:04.611 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.611 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:30:04.611 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:12.401220+0000", 2024-08-19T14:30:04.611 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:30:04.611 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.611 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.611 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:30:04.611 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:12.401220+0000", 2024-08-19T14:30:04.611 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:30:04.611 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.611 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.611 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:30:04.612 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:12.401222+0000", 2024-08-19T14:30:04.612 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.61e-06 2024-08-19T14:30:04.612 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.612 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.612 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:30:04.612 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:12.401289+0000", 2024-08-19T14:30:04.612 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6544999999999994e-05 2024-08-19T14:30:04.612 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.612 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.612 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:30:04.612 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:12.401300+0000", 2024-08-19T14:30:04.613 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1411e-05 2024-08-19T14:30:04.613 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.613 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.613 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:30:04.613 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:12.401307+0000", 2024-08-19T14:30:04.613 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2710000000000003e-06 2024-08-19T14:30:04.613 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.613 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.613 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:30:04.613 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:12.402328+0000", 2024-08-19T14:30:04.613 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0010201870000000001 2024-08-19T14:30:04.613 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.613 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.614 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:30:04.614 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:12.402546+0000", 2024-08-19T14:30:04.614 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00021844900000000001 2024-08-19T14:30:04.614 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.614 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.614 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0,8", 2024-08-19T14:30:04.614 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:12.402737+0000", 2024-08-19T14:30:04.614 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00019096799999999999 2024-08-19T14:30:04.614 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.614 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.614 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:30:04.614 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:12.405087+0000", 2024-08-19T14:30:04.614 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0023497489999999999 2024-08-19T14:30:04.615 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.615 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.615 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:30:04.615 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:12.405217+0000", 2024-08-19T14:30:04.615 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013065599999999999 2024-08-19T14:30:04.615 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.615 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.615 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:30:04.615 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:12.557488+0000", 2024-08-19T14:30:04.615 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.15227074400000001 2024-08-19T14:30:04.615 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.615 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.615 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:30:04.615 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:12.557533+0000", 2024-08-19T14:30:04.615 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5411999999999998e-05 2024-08-19T14:30:04.616 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.616 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.616 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:30:04.616 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:12.557573+0000", 2024-08-19T14:30:04.616 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0012999999999998e-05 2024-08-19T14:30:04.616 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:30:04.616 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:30:04.616 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:30:04.616 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.616 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.616 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26191.0:72572 6.3 6:ce1a127c:::benchmark_data_smithi045_97937_object72571:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e522)", 2024-08-19T14:30:04.617 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:29:12.411055+0000", 2024-08-19T14:30:04.617 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 52.138195734, 2024-08-19T14:30:04.617 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.146646837, 2024-08-19T14:30:04.617 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:30:04.617 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:30:04.617 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:30:04.617 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:30:04.617 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26191", 2024-08-19T14:30:04.617 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:30:04.617 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 72572 2024-08-19T14:30:04.617 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.617 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:30:04.618 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.618 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:30:04.618 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:12.411055+0000", 2024-08-19T14:30:04.618 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:30:04.618 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.618 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.618 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:30:04.618 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:12.411055+0000", 2024-08-19T14:30:04.618 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:30:04.619 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.619 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.619 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:30:04.619 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:12.411056+0000", 2024-08-19T14:30:04.619 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1340000000000001e-06 2024-08-19T14:30:04.619 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.619 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.619 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:30:04.619 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:12.411099+0000", 2024-08-19T14:30:04.619 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2385999999999997e-05 2024-08-19T14:30:04.619 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.619 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.619 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:30:04.619 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:12.411111+0000", 2024-08-19T14:30:04.620 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1919000000000001e-05 2024-08-19T14:30:04.620 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.620 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.620 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:30:04.620 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:12.411116+0000", 2024-08-19T14:30:04.620 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5249999999999996e-06 2024-08-19T14:30:04.620 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.620 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.620 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:30:04.620 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:12.411197+0000", 2024-08-19T14:30:04.620 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.0771000000000003e-05 2024-08-19T14:30:04.620 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.620 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.621 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:30:04.621 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:12.411320+0000", 2024-08-19T14:30:04.621 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012275599999999999 2024-08-19T14:30:04.621 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.621 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.621 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0,8", 2024-08-19T14:30:04.621 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:12.411428+0000", 2024-08-19T14:30:04.621 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000108539 2024-08-19T14:30:04.621 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.621 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.621 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:30:04.621 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:12.413466+0000", 2024-08-19T14:30:04.621 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0020379830000000002 2024-08-19T14:30:04.622 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.622 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.622 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:30:04.622 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:12.413622+0000", 2024-08-19T14:30:04.622 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000155558 2024-08-19T14:30:04.622 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.622 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.622 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:30:04.622 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:12.557628+0000", 2024-08-19T14:30:04.622 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.14400574499999999 2024-08-19T14:30:04.622 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.622 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.622 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:30:04.623 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:12.557665+0000", 2024-08-19T14:30:04.623 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7672000000000001e-05 2024-08-19T14:30:04.623 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.623 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.623 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:30:04.623 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:12.557702+0000", 2024-08-19T14:30:04.623 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6848999999999999e-05 2024-08-19T14:30:04.623 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:30:04.623 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:30:04.623 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:30:04.623 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.623 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.623 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26191.0:72576 6.3 6:cfe264a5:::benchmark_data_smithi045_97937_object72575:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e522)", 2024-08-19T14:30:04.623 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:29:12.414121+0000", 2024-08-19T14:30:04.624 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 52.135129532000001, 2024-08-19T14:30:04.624 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.14366614599999999, 2024-08-19T14:30:04.624 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:30:04.624 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:30:04.624 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:30:04.624 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:30:04.624 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26191", 2024-08-19T14:30:04.624 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:30:04.624 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 72576 2024-08-19T14:30:04.624 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.624 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:30:04.624 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.624 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:30:04.624 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:12.414121+0000", 2024-08-19T14:30:04.625 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:30:04.625 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.625 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.625 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:30:04.625 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:12.414121+0000", 2024-08-19T14:30:04.625 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:30:04.625 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.625 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.625 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:30:04.625 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:12.414123+0000", 2024-08-19T14:30:04.625 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.277e-06 2024-08-19T14:30:04.625 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.625 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.625 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:30:04.625 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:12.414166+0000", 2024-08-19T14:30:04.626 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2888e-05 2024-08-19T14:30:04.626 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.626 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.626 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:30:04.626 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:12.414176+0000", 2024-08-19T14:30:04.626 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0494e-05 2024-08-19T14:30:04.626 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.626 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.626 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:30:04.626 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:12.414181+0000", 2024-08-19T14:30:04.626 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8600000000000001e-06 2024-08-19T14:30:04.626 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.626 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.626 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:30:04.627 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:12.414220+0000", 2024-08-19T14:30:04.627 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9304000000000002e-05 2024-08-19T14:30:04.627 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.627 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.627 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:30:04.627 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:12.414313+0000", 2024-08-19T14:30:04.627 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3138000000000002e-05 2024-08-19T14:30:04.627 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.627 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.627 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0,8", 2024-08-19T14:30:04.627 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:12.414399+0000", 2024-08-19T14:30:04.627 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.5370000000000004e-05 2024-08-19T14:30:04.627 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.628 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.628 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:30:04.628 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:12.415946+0000", 2024-08-19T14:30:04.628 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0015471840000000001 2024-08-19T14:30:04.628 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.628 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.628 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:30:04.628 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:12.416075+0000", 2024-08-19T14:30:04.628 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012893400000000001 2024-08-19T14:30:04.628 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.628 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.628 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:30:04.628 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:12.557735+0000", 2024-08-19T14:30:04.628 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.14165973200000001 2024-08-19T14:30:04.629 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.629 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.629 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:30:04.629 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:12.557764+0000", 2024-08-19T14:30:04.629 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9652e-05 2024-08-19T14:30:04.629 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.629 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.629 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:30:04.629 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:12.557788+0000", 2024-08-19T14:30:04.629 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3312999999999999e-05 2024-08-19T14:30:04.629 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:30:04.629 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:30:04.629 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:30:04.629 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.630 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.630 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26191.0:76303 6.5 6:a62f8e5b:::benchmark_data_smithi045_97937_object76302:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e526)", 2024-08-19T14:30:04.630 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:29:17.435178+0000", 2024-08-19T14:30:04.630 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 47.114073363999999, 2024-08-19T14:30:04.630 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.27830824199999998, 2024-08-19T14:30:04.630 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:30:04.630 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:30:04.630 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:30:04.630 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:30:04.630 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26191", 2024-08-19T14:30:04.630 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:30:04.630 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 76303 2024-08-19T14:30:04.630 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.631 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:30:04.631 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.631 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:30:04.631 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.435178+0000", 2024-08-19T14:30:04.631 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:30:04.631 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.631 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.631 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:30:04.631 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.435178+0000", 2024-08-19T14:30:04.631 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:30:04.631 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.631 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.631 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:30:04.631 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.435179+0000", 2024-08-19T14:30:04.632 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.874e-06 2024-08-19T14:30:04.632 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.632 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.632 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:30:04.632 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.435304+0000", 2024-08-19T14:30:04.632 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012452399999999999 2024-08-19T14:30:04.632 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.632 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.632 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:30:04.632 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.435325+0000", 2024-08-19T14:30:04.632 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0639999999999999e-05 2024-08-19T14:30:04.632 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.632 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.633 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:30:04.633 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.435337+0000", 2024-08-19T14:30:04.633 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2544000000000001e-05 2024-08-19T14:30:04.633 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.633 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.633 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:30:04.633 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.437771+0000", 2024-08-19T14:30:04.633 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0024334729999999998 2024-08-19T14:30:04.633 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.633 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.633 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:30:04.633 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.437779+0000", 2024-08-19T14:30:04.633 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.0989999999999999e-06 2024-08-19T14:30:04.633 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.634 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.634 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:30:04.634 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.672561+0000", 2024-08-19T14:30:04.634 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.23478202500000001 2024-08-19T14:30:04.634 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.634 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.634 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:30:04.634 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.672692+0000", 2024-08-19T14:30:04.634 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000130844 2024-08-19T14:30:04.634 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.634 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.634 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,5", 2024-08-19T14:30:04.634 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.672799+0000", 2024-08-19T14:30:04.634 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000106873 2024-08-19T14:30:04.635 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.635 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.635 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:30:04.635 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.678796+0000", 2024-08-19T14:30:04.635 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0059978000000000002 2024-08-19T14:30:04.635 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.635 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.635 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:30:04.635 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.678812+0000", 2024-08-19T14:30:04.635 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6143000000000001e-05 2024-08-19T14:30:04.635 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.635 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.635 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:30:04.635 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.710311+0000", 2024-08-19T14:30:04.636 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.031498783000000002 2024-08-19T14:30:04.636 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.636 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.636 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:30:04.636 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.710382+0000", 2024-08-19T14:30:04.636 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0769e-05 2024-08-19T14:30:04.636 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.636 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.636 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:30:04.636 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.713486+0000", 2024-08-19T14:30:04.636 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0031038509999999999 2024-08-19T14:30:04.636 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:30:04.636 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:30:04.636 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:30:04.636 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.637 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.637 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26191.0:76317 6.5 6:ad41793c:::benchmark_data_smithi045_97937_object76316:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e526)", 2024-08-19T14:30:04.637 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:29:17.435362+0000", 2024-08-19T14:30:04.637 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 47.113889462000003, 2024-08-19T14:30:04.637 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.28174579900000002, 2024-08-19T14:30:04.637 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:30:04.637 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:30:04.637 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:30:04.637 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:30:04.637 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26191", 2024-08-19T14:30:04.637 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:30:04.637 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 76317 2024-08-19T14:30:04.637 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.638 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:30:04.638 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.638 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:30:04.638 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.435362+0000", 2024-08-19T14:30:04.638 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:30:04.638 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.638 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.638 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:30:04.638 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.435362+0000", 2024-08-19T14:30:04.638 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:30:04.638 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.638 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.638 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:30:04.638 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.435362+0000", 2024-08-19T14:30:04.639 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9999999999999997e-07 2024-08-19T14:30:04.639 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.639 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.639 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:30:04.639 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.435397+0000", 2024-08-19T14:30:04.639 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.4538e-05 2024-08-19T14:30:04.639 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.639 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.639 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:30:04.639 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.435405+0000", 2024-08-19T14:30:04.639 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.32e-06 2024-08-19T14:30:04.639 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.639 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.639 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:30:04.640 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.435411+0000", 2024-08-19T14:30:04.640 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0090000000000004e-06 2024-08-19T14:30:04.640 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.640 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.640 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:30:04.640 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.437792+0000", 2024-08-19T14:30:04.640 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0023814999999999999 2024-08-19T14:30:04.640 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.640 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.640 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:30:04.640 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.437796+0000", 2024-08-19T14:30:04.640 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7610000000000001e-06 2024-08-19T14:30:04.640 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.640 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.641 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:30:04.641 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.676699+0000", 2024-08-19T14:30:04.641 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.23890259699999999 2024-08-19T14:30:04.641 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.641 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.641 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:30:04.641 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.676815+0000", 2024-08-19T14:30:04.641 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00011618800000000001 2024-08-19T14:30:04.641 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.641 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.641 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,5", 2024-08-19T14:30:04.641 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.676929+0000", 2024-08-19T14:30:04.642 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00011372899999999999 2024-08-19T14:30:04.642 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.642 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.642 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:30:04.642 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.678907+0000", 2024-08-19T14:30:04.642 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0019780900000000001 2024-08-19T14:30:04.642 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.642 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.642 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:30:04.642 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.679369+0000", 2024-08-19T14:30:04.642 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00046200599999999999 2024-08-19T14:30:04.642 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.642 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.643 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:30:04.643 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.713578+0000", 2024-08-19T14:30:04.643 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.034208648000000001 2024-08-19T14:30:04.643 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.643 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.643 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:30:04.643 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.713646+0000", 2024-08-19T14:30:04.643 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8518999999999999e-05 2024-08-19T14:30:04.643 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.643 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.643 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:30:04.643 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.717107+0000", 2024-08-19T14:30:04.643 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0034612940000000002 2024-08-19T14:30:04.643 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:30:04.644 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:30:04.644 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:30:04.644 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.644 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.644 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26191.0:82701 6.5 6:a3bdb82a:::benchmark_data_smithi045_97937_object82700:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e532)", 2024-08-19T14:30:04.644 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:29:28.094522+0000", 2024-08-19T14:30:04.644 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 36.454729131000001, 2024-08-19T14:30:04.644 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.22778499199999999, 2024-08-19T14:30:04.644 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:30:04.644 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:30:04.644 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:30:04.644 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:30:04.644 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26191", 2024-08-19T14:30:04.644 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:30:04.645 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 82701 2024-08-19T14:30:04.645 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.645 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:30:04.645 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.645 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:30:04.645 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.094522+0000", 2024-08-19T14:30:04.645 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:30:04.645 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.645 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.645 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:30:04.645 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.094522+0000", 2024-08-19T14:30:04.645 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:30:04.645 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.645 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.645 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:30:04.646 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.094523+0000", 2024-08-19T14:30:04.646 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4729999999999999e-06 2024-08-19T14:30:04.646 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.646 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.646 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:30:04.646 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.094590+0000", 2024-08-19T14:30:04.646 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7082000000000006e-05 2024-08-19T14:30:04.646 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.646 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.646 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:30:04.646 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.094603+0000", 2024-08-19T14:30:04.646 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2892e-05 2024-08-19T14:30:04.646 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.646 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.647 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:30:04.647 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.094612+0000", 2024-08-19T14:30:04.647 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.3729999999999993e-06 2024-08-19T14:30:04.647 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.647 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.647 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:30:04.647 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.101069+0000", 2024-08-19T14:30:04.647 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0064572550000000003 2024-08-19T14:30:04.647 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.647 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.647 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:30:04.647 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.101185+0000", 2024-08-19T14:30:04.647 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000115888 2024-08-19T14:30:04.647 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.647 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.648 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,5", 2024-08-19T14:30:04.648 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.101311+0000", 2024-08-19T14:30:04.648 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012650500000000001 2024-08-19T14:30:04.648 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.648 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.648 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:30:04.648 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.106572+0000", 2024-08-19T14:30:04.648 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00526115 2024-08-19T14:30:04.648 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.648 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.648 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:30:04.648 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.119157+0000", 2024-08-19T14:30:04.648 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.01258454 2024-08-19T14:30:04.649 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.649 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.649 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:30:04.649 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.319119+0000", 2024-08-19T14:30:04.649 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.19996164799999999 2024-08-19T14:30:04.649 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.649 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.649 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:30:04.649 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.319194+0000", 2024-08-19T14:30:04.649 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5297999999999996e-05 2024-08-19T14:30:04.649 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.649 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.649 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:30:04.649 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.322307+0000", 2024-08-19T14:30:04.649 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0031128879999999999 2024-08-19T14:30:04.650 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:30:04.650 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:30:04.650 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:30:04.650 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.650 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.650 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26191.0:82705 6.5 6:acb592fa:::benchmark_data_smithi045_97937_object82704:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e532)", 2024-08-19T14:30:04.650 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:29:28.097021+0000", 2024-08-19T14:30:04.650 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 36.452229975999998, 2024-08-19T14:30:04.650 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.22856966400000001, 2024-08-19T14:30:04.650 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:30:04.650 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:30:04.650 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:30:04.650 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:30:04.651 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26191", 2024-08-19T14:30:04.651 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:30:04.651 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 82705 2024-08-19T14:30:04.651 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.651 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:30:04.651 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.651 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:30:04.651 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.097021+0000", 2024-08-19T14:30:04.651 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:30:04.651 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.651 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.651 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:30:04.651 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.097021+0000", 2024-08-19T14:30:04.652 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:30:04.652 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.652 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.652 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:30:04.652 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.097023+0000", 2024-08-19T14:30:04.652 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.548e-06 2024-08-19T14:30:04.652 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.652 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.652 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:30:04.652 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.097054+0000", 2024-08-19T14:30:04.652 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1301999999999999e-05 2024-08-19T14:30:04.652 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.652 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.652 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:30:04.653 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.097065+0000", 2024-08-19T14:30:04.653 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.132e-05 2024-08-19T14:30:04.653 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.653 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.653 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:30:04.653 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.097072+0000", 2024-08-19T14:30:04.653 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9789999999999996e-06 2024-08-19T14:30:04.653 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.653 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.653 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:30:04.653 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.103771+0000", 2024-08-19T14:30:04.653 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0066992500000000003 2024-08-19T14:30:04.653 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.654 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.654 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:30:04.654 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.103924+0000", 2024-08-19T14:30:04.654 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00015227 2024-08-19T14:30:04.654 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.654 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.654 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,5", 2024-08-19T14:30:04.654 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.104085+0000", 2024-08-19T14:30:04.654 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00016106399999999999 2024-08-19T14:30:04.654 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.654 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.654 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:30:04.654 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.111674+0000", 2024-08-19T14:30:04.654 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0075892770000000002 2024-08-19T14:30:04.655 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.655 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.655 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:30:04.655 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.119225+0000", 2024-08-19T14:30:04.655 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.007551065 2024-08-19T14:30:04.655 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.655 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.655 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:30:04.655 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.322372+0000", 2024-08-19T14:30:04.655 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20314706299999999 2024-08-19T14:30:04.655 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.655 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.655 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:30:04.655 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.322557+0000", 2024-08-19T14:30:04.656 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000185298 2024-08-19T14:30:04.656 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.656 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.656 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:30:04.656 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.325591+0000", 2024-08-19T14:30:04.656 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0030332279999999998 2024-08-19T14:30:04.656 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:30:04.656 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:30:04.656 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:30:04.656 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.656 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.656 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26191.0:82728 6.5 6:a26136df:::benchmark_data_smithi045_97937_object82727:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e532)", 2024-08-19T14:30:04.656 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:29:28.129123+0000", 2024-08-19T14:30:04.656 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 36.420128255999998, 2024-08-19T14:30:04.656 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20736944900000001, 2024-08-19T14:30:04.657 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:30:04.657 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:30:04.657 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:30:04.657 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:30:04.657 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26191", 2024-08-19T14:30:04.657 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:30:04.657 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 82728 2024-08-19T14:30:04.657 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.657 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:30:04.657 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.657 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:30:04.657 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.129123+0000", 2024-08-19T14:30:04.657 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:30:04.657 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.658 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.658 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:30:04.658 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.129123+0000", 2024-08-19T14:30:04.658 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:30:04.658 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.658 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.658 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:30:04.658 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.129124+0000", 2024-08-19T14:30:04.658 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.525e-06 2024-08-19T14:30:04.658 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.658 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.658 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:30:04.658 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.129179+0000", 2024-08-19T14:30:04.658 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4682000000000003e-05 2024-08-19T14:30:04.659 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.659 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.659 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:30:04.659 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.129195+0000", 2024-08-19T14:30:04.659 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6291e-05 2024-08-19T14:30:04.659 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.659 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.659 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:30:04.659 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.129203+0000", 2024-08-19T14:30:04.659 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4479999999999997e-06 2024-08-19T14:30:04.659 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.659 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.659 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:30:04.659 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.135082+0000", 2024-08-19T14:30:04.660 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0058794379999999999 2024-08-19T14:30:04.660 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.660 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.660 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:30:04.660 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.135273+0000", 2024-08-19T14:30:04.660 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000190657 2024-08-19T14:30:04.660 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.660 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.660 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,5", 2024-08-19T14:30:04.660 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.135453+0000", 2024-08-19T14:30:04.660 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000179774 2024-08-19T14:30:04.660 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.660 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.660 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:30:04.661 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.147427+0000", 2024-08-19T14:30:04.661 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011974182 2024-08-19T14:30:04.661 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.661 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.661 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:30:04.661 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.147485+0000", 2024-08-19T14:30:04.661 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8545999999999998e-05 2024-08-19T14:30:04.661 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.661 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.661 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:30:04.661 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.333947+0000", 2024-08-19T14:30:04.661 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.186461296 2024-08-19T14:30:04.661 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.661 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.662 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:30:04.662 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.333997+0000", 2024-08-19T14:30:04.662 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0553999999999997e-05 2024-08-19T14:30:04.662 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.662 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.662 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:30:04.662 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.336492+0000", 2024-08-19T14:30:04.662 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0024950559999999998 2024-08-19T14:30:04.662 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:30:04.662 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:30:04.662 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:30:04.662 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.662 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.663 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26191.0:82743 6.5 6:ab19624b:::benchmark_data_smithi045_97937_object82742:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e532)", 2024-08-19T14:30:04.663 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:29:28.154818+0000", 2024-08-19T14:30:04.663 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 36.394432678000001, 2024-08-19T14:30:04.663 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.184198685, 2024-08-19T14:30:04.663 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:30:04.663 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:30:04.663 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:30:04.663 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:30:04.663 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26191", 2024-08-19T14:30:04.663 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:30:04.663 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 82743 2024-08-19T14:30:04.663 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.663 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:30:04.663 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.663 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:30:04.664 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.154818+0000", 2024-08-19T14:30:04.664 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:30:04.664 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.664 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.664 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:30:04.664 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.154818+0000", 2024-08-19T14:30:04.664 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:30:04.664 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.664 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.664 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:30:04.664 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.154820+0000", 2024-08-19T14:30:04.664 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2929999999999999e-06 2024-08-19T14:30:04.664 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.664 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.665 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:30:04.665 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.154899+0000", 2024-08-19T14:30:04.665 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.9339999999999996e-05 2024-08-19T14:30:04.665 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.665 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.665 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:30:04.665 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.154915+0000", 2024-08-19T14:30:04.665 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.615e-05 2024-08-19T14:30:04.665 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.665 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.665 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:30:04.665 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.154923+0000", 2024-08-19T14:30:04.665 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.9740000000000002e-06 2024-08-19T14:30:04.665 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.666 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.666 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:30:04.666 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.157717+0000", 2024-08-19T14:30:04.666 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0027936440000000001 2024-08-19T14:30:04.666 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.666 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.666 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:30:04.666 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.157828+0000", 2024-08-19T14:30:04.666 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000110875 2024-08-19T14:30:04.666 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.666 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.666 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,5", 2024-08-19T14:30:04.666 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.157933+0000", 2024-08-19T14:30:04.666 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000105556 2024-08-19T14:30:04.667 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.667 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.667 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:30:04.667 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.168960+0000", 2024-08-19T14:30:04.667 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011026919 2024-08-19T14:30:04.667 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.667 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.667 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:30:04.667 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.169011+0000", 2024-08-19T14:30:04.667 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1350000000000001e-05 2024-08-19T14:30:04.667 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.667 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.667 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:30:04.667 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.336704+0000", 2024-08-19T14:30:04.667 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.16769240499999999 2024-08-19T14:30:04.668 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.668 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.668 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:30:04.668 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.336776+0000", 2024-08-19T14:30:04.668 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2135e-05 2024-08-19T14:30:04.668 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.668 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.668 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:30:04.668 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.339017+0000", 2024-08-19T14:30:04.668 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0022410440000000002 2024-08-19T14:30:04.668 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:30:04.668 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:30:04.668 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:30:04.668 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.669 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.669 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26191.0:82751 6.5 6:a9d39f54:::benchmark_data_smithi045_97937_object82750:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e532)", 2024-08-19T14:30:04.669 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:29:28.162278+0000", 2024-08-19T14:30:04.669 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 36.386972778999997, 2024-08-19T14:30:04.669 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.17968440399999999, 2024-08-19T14:30:04.669 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:30:04.669 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:30:04.669 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:30:04.669 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:30:04.669 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26191", 2024-08-19T14:30:04.669 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:30:04.669 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 82751 2024-08-19T14:30:04.669 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.669 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:30:04.669 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.670 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:30:04.670 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.162278+0000", 2024-08-19T14:30:04.670 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:30:04.670 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.670 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.670 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:30:04.670 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.162278+0000", 2024-08-19T14:30:04.670 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:30:04.670 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.670 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.670 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:30:04.670 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.162280+0000", 2024-08-19T14:30:04.670 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.75e-06 2024-08-19T14:30:04.671 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.671 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.671 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:30:04.671 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.162339+0000", 2024-08-19T14:30:04.671 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8564999999999999e-05 2024-08-19T14:30:04.671 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.671 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.671 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:30:04.671 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.162355+0000", 2024-08-19T14:30:04.671 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6053e-05 2024-08-19T14:30:04.671 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.671 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.671 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:30:04.671 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.162363+0000", 2024-08-19T14:30:04.671 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.4610000000000008e-06 2024-08-19T14:30:04.672 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.672 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.672 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:30:04.672 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.169037+0000", 2024-08-19T14:30:04.672 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0066734730000000001 2024-08-19T14:30:04.672 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.672 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.672 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:30:04.672 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.169151+0000", 2024-08-19T14:30:04.672 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000114096 2024-08-19T14:30:04.672 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.672 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.672 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,5", 2024-08-19T14:30:04.672 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.169269+0000", 2024-08-19T14:30:04.672 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000118194 2024-08-19T14:30:04.673 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.673 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.673 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:30:04.673 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.174245+0000", 2024-08-19T14:30:04.673 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0049762929999999997 2024-08-19T14:30:04.673 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.673 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.673 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:30:04.673 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.192778+0000", 2024-08-19T14:30:04.673 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.018533279999999999 2024-08-19T14:30:04.673 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.673 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.673 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:30:04.673 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.339090+0000", 2024-08-19T14:30:04.673 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.14631174799999999 2024-08-19T14:30:04.674 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.674 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.674 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:30:04.674 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.339159+0000", 2024-08-19T14:30:04.674 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8649e-05 2024-08-19T14:30:04.674 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.674 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.674 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:30:04.674 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.341963+0000", 2024-08-19T14:30:04.674 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.002803842 2024-08-19T14:30:04.674 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:30:04.674 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:30:04.674 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:30:04.674 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.675 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.676 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:04 smithi031 ceph-mon[16336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:30:04.676 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:04 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:30:04.676 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:04 smithi031 ceph-mon[16336]: osdmap e542: 12 total, 12 up, 8 in 2024-08-19T14:30:04.676 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26191.0:82756 6.5 6:a9427b19:::benchmark_data_smithi045_97937_object82755:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e532)", 2024-08-19T14:30:04.677 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:29:28.170891+0000", 2024-08-19T14:30:04.677 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 36.378359625000002, 2024-08-19T14:30:04.677 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.17433968899999999, 2024-08-19T14:30:04.677 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:30:04.677 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:30:04.677 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:30:04.677 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:30:04.677 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26191", 2024-08-19T14:30:04.677 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:30:04.677 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 82756 2024-08-19T14:30:04.677 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.677 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:30:04.677 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.678 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:30:04.678 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.170891+0000", 2024-08-19T14:30:04.678 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:30:04.678 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.678 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.678 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:30:04.678 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.170891+0000", 2024-08-19T14:30:04.678 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:30:04.678 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.678 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.678 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:30:04.679 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.170892+0000", 2024-08-19T14:30:04.679 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.1200000000000001e-07 2024-08-19T14:30:04.679 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.679 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.679 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:30:04.679 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.170955+0000", 2024-08-19T14:30:04.679 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2665999999999999e-05 2024-08-19T14:30:04.679 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.679 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.679 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:30:04.679 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.170971+0000", 2024-08-19T14:30:04.680 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5790999999999998e-05 2024-08-19T14:30:04.680 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.680 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.680 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:30:04.680 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.170983+0000", 2024-08-19T14:30:04.680 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2507000000000001e-05 2024-08-19T14:30:04.680 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.680 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.680 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:30:04.680 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.171479+0000", 2024-08-19T14:30:04.680 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00049534500000000001 2024-08-19T14:30:04.680 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.680 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.681 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:30:04.681 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.171646+0000", 2024-08-19T14:30:04.681 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00016748800000000001 2024-08-19T14:30:04.681 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.681 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.681 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,5", 2024-08-19T14:30:04.681 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.171783+0000", 2024-08-19T14:30:04.681 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013677900000000001 2024-08-19T14:30:04.681 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.681 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.681 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:30:04.681 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.174465+0000", 2024-08-19T14:30:04.681 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0026819930000000001 2024-08-19T14:30:04.681 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.682 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.682 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:30:04.682 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.192834+0000", 2024-08-19T14:30:04.682 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.018369449 2024-08-19T14:30:04.682 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.682 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.682 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:30:04.682 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.342040+0000", 2024-08-19T14:30:04.682 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.14920577800000001 2024-08-19T14:30:04.682 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.682 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.682 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:30:04.682 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.342109+0000", 2024-08-19T14:30:04.682 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8851999999999997e-05 2024-08-19T14:30:04.683 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.683 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.683 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:30:04.683 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.345231+0000", 2024-08-19T14:30:04.683 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.003122129 2024-08-19T14:30:04.683 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:30:04.683 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:30:04.683 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:30:04.683 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.683 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.683 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26191.0:82772 6.5 6:ab0eb172:::benchmark_data_smithi045_97937_object82771:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e532)", 2024-08-19T14:30:04.683 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:29:28.197443+0000", 2024-08-19T14:30:04.683 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 36.351807817999997, 2024-08-19T14:30:04.683 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.15060845, 2024-08-19T14:30:04.684 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:30:04.684 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:30:04.684 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:30:04.684 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:30:04.684 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26191", 2024-08-19T14:30:04.684 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:30:04.684 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 82772 2024-08-19T14:30:04.684 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.684 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:30:04.684 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.684 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:30:04.684 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.197443+0000", 2024-08-19T14:30:04.684 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:30:04.684 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.685 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.685 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:30:04.685 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.197443+0000", 2024-08-19T14:30:04.685 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:30:04.685 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.685 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.685 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:30:04.685 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.197445+0000", 2024-08-19T14:30:04.685 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.81e-06 2024-08-19T14:30:04.685 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.685 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.685 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:30:04.685 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.197509+0000", 2024-08-19T14:30:04.685 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4320999999999996e-05 2024-08-19T14:30:04.686 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.686 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.686 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:30:04.686 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.197527+0000", 2024-08-19T14:30:04.686 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7631000000000001e-05 2024-08-19T14:30:04.686 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.686 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.686 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:30:04.686 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.197536+0000", 2024-08-19T14:30:04.686 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.8969999999999997e-06 2024-08-19T14:30:04.686 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.686 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.686 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:30:04.686 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.201180+0000", 2024-08-19T14:30:04.687 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0036445649999999998 2024-08-19T14:30:04.687 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.687 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.687 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:30:04.687 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.201330+0000", 2024-08-19T14:30:04.687 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000149495 2024-08-19T14:30:04.687 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.687 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.687 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,5", 2024-08-19T14:30:04.687 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.201487+0000", 2024-08-19T14:30:04.687 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000157391 2024-08-19T14:30:04.687 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.687 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.687 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:30:04.688 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.204533+0000", 2024-08-19T14:30:04.688 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0030456929999999999 2024-08-19T14:30:04.688 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.688 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.688 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:30:04.688 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.219485+0000", 2024-08-19T14:30:04.688 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.014951991 2024-08-19T14:30:04.688 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.689 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.689 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:30:04.689 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.345299+0000", 2024-08-19T14:30:04.689 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.1258136 2024-08-19T14:30:04.689 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.689 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.689 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:30:04.689 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.345357+0000", 2024-08-19T14:30:04.689 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8585999999999998e-05 2024-08-19T14:30:04.689 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.689 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.690 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:30:04.690 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.348052+0000", 2024-08-19T14:30:04.690 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00269447 2024-08-19T14:30:04.690 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:30:04.690 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:30:04.690 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:30:04.690 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.690 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.690 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26191.0:82784 6.5 6:a60b68b0:::benchmark_data_smithi045_97937_object82783:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e532)", 2024-08-19T14:30:04.690 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:29:28.210322+0000", 2024-08-19T14:30:04.690 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 36.338929358999998, 2024-08-19T14:30:04.690 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.14111590099999999, 2024-08-19T14:30:04.691 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:30:04.691 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:30:04.691 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:30:04.691 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:30:04.691 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26191", 2024-08-19T14:30:04.691 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:30:04.691 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 82784 2024-08-19T14:30:04.691 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.691 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:30:04.691 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.691 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:30:04.691 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.210322+0000", 2024-08-19T14:30:04.692 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:30:04.692 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.692 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.692 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:30:04.692 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.210322+0000", 2024-08-19T14:30:04.692 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:30:04.692 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.692 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.692 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:30:04.692 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.210323+0000", 2024-08-19T14:30:04.692 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3799999999999999e-06 2024-08-19T14:30:04.692 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.692 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.692 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:30:04.693 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.210372+0000", 2024-08-19T14:30:04.693 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8711000000000002e-05 2024-08-19T14:30:04.693 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.693 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.693 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:30:04.693 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.210382+0000", 2024-08-19T14:30:04.693 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.065e-05 2024-08-19T14:30:04.693 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.693 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.693 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:30:04.693 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.210390+0000", 2024-08-19T14:30:04.693 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1520000000000003e-06 2024-08-19T14:30:04.693 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.693 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.693 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:30:04.694 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.216577+0000", 2024-08-19T14:30:04.694 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0061870440000000001 2024-08-19T14:30:04.694 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.694 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.694 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:30:04.694 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.216755+0000", 2024-08-19T14:30:04.694 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000178402 2024-08-19T14:30:04.694 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.694 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.694 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,5", 2024-08-19T14:30:04.694 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.216933+0000", 2024-08-19T14:30:04.694 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00017762799999999999 2024-08-19T14:30:04.694 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.694 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.695 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:30:04.695 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.226883+0000", 2024-08-19T14:30:04.695 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0099504680000000005 2024-08-19T14:30:04.695 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.695 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.695 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:30:04.695 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.226918+0000", 2024-08-19T14:30:04.695 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.4671000000000001e-05 2024-08-19T14:30:04.695 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.695 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.695 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:30:04.695 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.348120+0000", 2024-08-19T14:30:04.695 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.12120204 2024-08-19T14:30:04.695 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.695 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.696 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:30:04.696 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.348212+0000", 2024-08-19T14:30:04.696 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.2554999999999997e-05 2024-08-19T14:30:04.696 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:30:04.696 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:30:04.696 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:30:04.696 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:28.351438+0000", 2024-08-19T14:30:04.696 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0032252000000000001 2024-08-19T14:30:04.696 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:30:04.696 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:30:04.696 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:30:04.696 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:30:04.696 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:30:04.696 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:30:04.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:04.697 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:04.698 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:30:04.801 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:30:04.831 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:04.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:04.833 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:30:04.934 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:30:04.959 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:04.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:04.960 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:30:05.061 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:30:05.082 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:05.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:05.083 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:30:05.184 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:30:05.216 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:05.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:05.217 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:30:05.318 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:30:05.345 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:05.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:05.346 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:30:05.447 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:30:05.466 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:05 smithi032 ceph-mon[20004]: pgmap v1247: 47 pgs: 8 active+recovering+undersized+remapped, 39 active+clean; 6.3 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 39 MiB/s wr, 630 op/s; 27602/310755 objects misplaced (8.882%); 15 MiB/s, 234 objects/s recovering 2024-08-19T14:30:05.491 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:05.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:05.492 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:30:05.513 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-08-19T14:30:05.514 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 87 16 105403 105387 75.7028 40.1875 0.0180777 0.0132045 2024-08-19T14:30:05.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:05 smithi045 ceph-mon[20143]: pgmap v1247: 47 pgs: 8 active+recovering+undersized+remapped, 39 active+clean; 6.3 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 39 MiB/s wr, 630 op/s; 27602/310755 objects misplaced (8.882%); 15 MiB/s, 234 objects/s recovering 2024-08-19T14:30:05.593 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:30:05.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:05 smithi031 ceph-mon[16336]: pgmap v1247: 47 pgs: 8 active+recovering+undersized+remapped, 39 active+clean; 6.3 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 39 MiB/s wr, 630 op/s; 27602/310755 objects misplaced (8.882%); 15 MiB/s, 234 objects/s recovering 2024-08-19T14:30:05.639 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:05.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:05.640 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:30:05.741 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:30:05.763 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:05.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:05.764 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:30:05.865 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:30:05.895 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:05.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:05.896 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:30:05.947 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:30:05.997 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:30:06.019 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:06.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:06.020 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:30:06.121 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:30:06.164 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:06.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:06.165 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:30:06.266 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:30:06.296 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:06.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:06.297 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:30:06.398 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:30:06.426 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:06.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:06.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:30:06.514 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 88 16 106045 106029 75.2985 40.125 0.0212486 0.0132753 2024-08-19T14:30:06.524 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:30:06.524 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [ 2024-08-19T14:30:06.524 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:06.524 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.26191.0:106041 6.1 6:8f4e2751:::benchmark_data_smithi045_97937_object106040:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e542)", 2024-08-19T14:30:06.524 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:30:06.501542+0000", 2024-08-19T14:30:06.524 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.021531326999999999, 2024-08-19T14:30:06.524 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.021555667000000001, 2024-08-19T14:30:06.524 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:30:06.524 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:30:06.524 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "waiting for sub ops", 2024-08-19T14:30:06.524 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:30:06.525 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.26191", 2024-08-19T14:30:06.525 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:30:06.525 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 106041 2024-08-19T14:30:06.525 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:06.525 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:30:06.525 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:06.525 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:30:06.525 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:30:06.501542+0000", 2024-08-19T14:30:06.525 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:30:06.525 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:06.525 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:06.525 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:30:06.525 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:30:06.501542+0000", 2024-08-19T14:30:06.526 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:30:06.526 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:06.526 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:06.526 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:30:06.526 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:30:06.501544+0000", 2024-08-19T14:30:06.526 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9190000000000002e-06 2024-08-19T14:30:06.526 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:06.526 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:06.526 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:30:06.526 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:30:06.501573+0000", 2024-08-19T14:30:06.526 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.9862999999999999e-05 2024-08-19T14:30:06.526 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:06.526 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:06.527 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:30:06.527 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:30:06.501593+0000", 2024-08-19T14:30:06.527 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9089999999999998e-05 2024-08-19T14:30:06.527 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:06.527 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:06.527 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:30:06.527 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:30:06.501604+0000", 2024-08-19T14:30:06.527 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0944999999999999e-05 2024-08-19T14:30:06.527 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:06.527 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:06.527 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:30:06.527 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:30:06.502840+0000", 2024-08-19T14:30:06.527 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001236233 2024-08-19T14:30:06.527 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:06.528 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:06.528 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:30:06.528 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:30:06.502960+0000", 2024-08-19T14:30:06.528 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000119975 2024-08-19T14:30:06.528 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:06.528 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:06.528 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2,7", 2024-08-19T14:30:06.528 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:30:06.503075+0000", 2024-08-19T14:30:06.528 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000115232 2024-08-19T14:30:06.528 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:06.528 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:06.528 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:30:06.528 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:30:06.505891+0000", 2024-08-19T14:30:06.529 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0028158319999999999 2024-08-19T14:30:06.529 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:06.529 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:06.529 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:30:06.529 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:30:06.506910+0000", 2024-08-19T14:30:06.529 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0010196199999999999 2024-08-19T14:30:06.529 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:30:06.529 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:30:06.529 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:30:06.529 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:06.529 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:06.529 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.26191.0:106047 6.1 6:8179a656:::benchmark_data_smithi045_97937_object106046:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e542)", 2024-08-19T14:30:06.529 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:30:06.516559+0000", 2024-08-19T14:30:06.529 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.0065140709999999998, 2024-08-19T14:30:06.530 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0065930650000000004, 2024-08-19T14:30:06.530 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:30:06.530 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:30:06.530 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "waiting for sub ops", 2024-08-19T14:30:06.530 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:30:06.530 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.26191", 2024-08-19T14:30:06.530 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:30:06.530 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 106047 2024-08-19T14:30:06.530 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:06.530 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:30:06.530 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:06.530 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:30:06.530 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:30:06.516559+0000", 2024-08-19T14:30:06.530 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:30:06.531 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:06.531 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:06.531 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:30:06.531 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:30:06.516559+0000", 2024-08-19T14:30:06.531 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:30:06.531 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:06.531 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:06.531 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:30:06.531 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:30:06.516561+0000", 2024-08-19T14:30:06.531 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6640000000000001e-06 2024-08-19T14:30:06.531 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:06.531 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:06.531 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:30:06.531 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:30:06.516591+0000", 2024-08-19T14:30:06.532 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.9964e-05 2024-08-19T14:30:06.532 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:06.532 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:06.532 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:30:06.532 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:30:06.516604+0000", 2024-08-19T14:30:06.532 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3050999999999999e-05 2024-08-19T14:30:06.532 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:06.532 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:06.532 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:30:06.532 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:30:06.516612+0000", 2024-08-19T14:30:06.532 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.4279999999999996e-06 2024-08-19T14:30:06.532 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:06.532 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:06.533 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:30:06.533 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:30:06.516682+0000", 2024-08-19T14:30:06.533 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.9486000000000005e-05 2024-08-19T14:30:06.533 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:06.533 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:06.533 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:30:06.533 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:30:06.516867+0000", 2024-08-19T14:30:06.533 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00018496300000000001 2024-08-19T14:30:06.533 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:06.533 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:06.533 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2,7", 2024-08-19T14:30:06.533 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:30:06.517084+0000", 2024-08-19T14:30:06.533 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00021740099999999999 2024-08-19T14:30:06.533 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:06.534 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:06.534 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:30:06.534 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:30:06.520518+0000", 2024-08-19T14:30:06.534 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0034339209999999999 2024-08-19T14:30:06.534 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:06.534 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:06.534 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:30:06.534 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:30:06.520647+0000", 2024-08-19T14:30:06.534 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012914099999999999 2024-08-19T14:30:06.534 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:30:06.534 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:30:06.534 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:30:06.534 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:30:06.535 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-08-19T14:30:06.535 INFO:teuthology.orchestra.run.smithi045.stdout: "num_ops": 2 2024-08-19T14:30:06.535 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:30:06.535 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:30:06.570 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:06.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:06.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:30:06.671 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:30:06.696 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:06.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:06.697 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:30:06.798 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:30:06.800 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:30:06.820 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:06.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:06.821 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:30:06.922 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:30:06.973 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:06.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:06.974 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:30:07.075 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:30:07.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:07.129 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:07.129 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:30:07.230 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:30:07.253 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:07.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:07.254 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:30:07.356 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:30:07.380 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:07.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:07.381 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:30:07.397 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:07 smithi045 ceph-mon[20143]: pgmap v1248: 47 pgs: 8 active+recovering+undersized+remapped, 39 active+clean; 6.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 41 MiB/s wr, 656 op/s; 27178/313044 objects misplaced (8.682%); 16 MiB/s, 253 objects/s recovering 2024-08-19T14:30:07.397 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:07 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:30:07.398 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:07 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:30:07.398 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:07 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:30:07.398 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:07 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:30:07.398 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:07 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:30:07.398 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:07 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:30:07.398 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:07 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:30:07.399 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-08-19T14:30:07.482 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:30:07.498 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:30:07.499 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:30:07.499 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:07 smithi031 ceph-mon[16336]: pgmap v1248: 47 pgs: 8 active+recovering+undersized+remapped, 39 active+clean; 6.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 41 MiB/s wr, 656 op/s; 27178/313044 objects misplaced (8.682%); 16 MiB/s, 253 objects/s recovering 2024-08-19T14:30:07.499 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:07 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:30:07.499 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:07 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:30:07.499 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:07 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:30:07.499 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:07 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:30:07.499 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:07 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:30:07.499 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:07 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:30:07.500 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:07 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:30:07.505 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:07.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:07.506 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:30:07.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:07 smithi032 ceph-mon[20004]: pgmap v1248: 47 pgs: 8 active+recovering+undersized+remapped, 39 active+clean; 6.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 41 MiB/s wr, 656 op/s; 27178/313044 objects misplaced (8.682%); 16 MiB/s, 253 objects/s recovering 2024-08-19T14:30:07.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:07 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:30:07.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:07 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:30:07.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:07 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:30:07.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:07 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:30:07.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:07 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:30:07.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:07 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:30:07.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:07 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:30:07.513 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 89 16 106631 106615 74.8639 36.625 0.0187148 0.0133511 2024-08-19T14:30:07.536 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:30:07.536 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:30:07.576 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:30:07.577 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:30:07.607 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:30:07.615 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:30:07.615 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:30:07.636 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:07.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:07.637 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:30:07.648 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:30:07.649 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:30:07.685 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:30:07.685 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:30:07.707 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:30:07.725 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:30:07.725 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:30:07.738 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:30:07.758 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:07.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:07.760 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:30:07.763 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:30:07.764 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:30:07.799 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:30:07.799 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:30:07.829 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:30:07.829 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:30:07.860 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:30:07.865 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:30:07.865 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:30:07.882 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:07.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:07.883 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:30:07.901 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:30:07.901 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:30:07.984 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:30:08.019 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:08.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:08.020 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:30:08.121 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:30:08.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:08.151 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:08.151 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:30:08.215 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:30:08.215 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:30:08.252 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:30:08.280 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:08.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:08.281 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:30:08.382 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:30:08.426 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:08.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:08.427 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:30:08.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:08 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3395860967' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:30:08.513 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 90 9 107270 107261 74.4807 40.375 0.0444251 0.0134209 2024-08-19T14:30:08.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:08 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3395860967' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:30:08.528 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:30:08.551 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:08.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:08.552 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:30:08.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:08 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3395860967' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:30:08.654 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:30:08.657 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-19T14:30:08.675 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:08.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:08.676 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:30:08.777 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:30:08.802 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:08.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:08.803 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:30:08.901 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-08-19T14:30:08.904 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:30:08.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:08.929 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:08.929 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:30:09.001 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:30:09.030 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:30:09.053 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:09.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:09.054 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:30:09.154 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:30:09.169 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:30:09.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:09.183 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:09.184 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:30:09.284 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:30:09.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:09.307 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:09.307 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:30:09.407 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:30:09.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:09.430 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:09.430 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:30:09.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:09 smithi032 ceph-mon[20004]: pgmap v1249: 47 pgs: 8 active+recovering+undersized+remapped, 39 active+clean; 6.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 29 MiB/s wr, 460 op/s; 26852/314643 objects misplaced (8.534%); 13 MiB/s, 212 objects/s recovering 2024-08-19T14:30:09.513 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Total time run: 90.0177 2024-08-19T14:30:09.514 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Total writes made: 107270 2024-08-19T14:30:09.514 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Write size: 65536 2024-08-19T14:30:09.514 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Object size: 65536 2024-08-19T14:30:09.514 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Bandwidth (MB/sec): 74.4784 2024-08-19T14:30:09.514 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Stddev Bandwidth: 54.5325 2024-08-19T14:30:09.514 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Max bandwidth (MB/sec): 247.812 2024-08-19T14:30:09.514 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Min bandwidth (MB/sec): 0 2024-08-19T14:30:09.514 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Average IOPS: 1191 2024-08-19T14:30:09.517 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Stddev IOPS: 872.525 2024-08-19T14:30:09.518 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Max IOPS: 3965 2024-08-19T14:30:09.518 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Min IOPS: 0 2024-08-19T14:30:09.518 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Average Latency(s): 0.0134227 2024-08-19T14:30:09.518 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Stddev Latency(s): 0.0339033 2024-08-19T14:30:09.518 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Max latency(s): 2.46333 2024-08-19T14:30:09.518 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Min latency(s): 0.00119513 2024-08-19T14:30:09.520 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Cleaning up (deleting benchmark objects) 2024-08-19T14:30:09.531 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:30:09.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:09 smithi045 ceph-mon[20143]: pgmap v1249: 47 pgs: 8 active+recovering+undersized+remapped, 39 active+clean; 6.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 29 MiB/s wr, 460 op/s; 26852/314643 objects misplaced (8.534%); 13 MiB/s, 212 objects/s recovering 2024-08-19T14:30:09.558 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:09.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:09.559 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:30:09.581 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:09 smithi031 ceph-mon[16336]: pgmap v1249: 47 pgs: 8 active+recovering+undersized+remapped, 39 active+clean; 6.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 29 MiB/s wr, 460 op/s; 26852/314643 objects misplaced (8.534%); 13 MiB/s, 212 objects/s recovering 2024-08-19T14:30:09.659 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:30:09.703 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:09.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:09.704 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:30:09.805 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:30:09.827 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:09.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:09.828 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:30:09.929 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:30:09.954 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:30:09.954 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:30:09.968 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:09.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:09.969 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:30:10.070 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:30:10.100 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:10.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:10.101 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:30:10.202 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:30:10.223 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:10.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:10.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:30:10.325 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:30:10.349 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:10.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:10.350 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:30:10.451 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:30:10.489 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:10.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:10.490 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:30:10.591 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:30:10.640 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:10.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:10.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:30:10.676 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-08-19T14:30:10.742 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:30:10.776 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:10.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:10.777 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:30:10.809 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:30:10.878 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:30:10.919 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:10.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:10.920 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:30:10.937 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:30:10.955 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 7, 8, 2] out_osds: [10, 6, 9, 11] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 10] 2024-08-19T14:30:10.955 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-19T14:30:10.956 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-08-19T14:30:10.956 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd dump -f json-pretty 2024-08-19T14:30:11.022 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:30:11.043 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:11.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:11.044 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:30:11.145 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:30:11.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:11.230 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:11.230 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:30:11.331 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:30:11.341 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:11 smithi045 ceph-mon[20143]: pgmap v1250: 47 pgs: 8 active+recovering+undersized+remapped, 39 active+clean; 6.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 44 MiB/s wr, 710 op/s; 26141/321183 objects misplaced (8.139%); 17 MiB/s, 274 objects/s recovering 2024-08-19T14:30:11.341 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:11 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:30:11.341 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:11 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:30:11.354 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:11.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:11.355 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:30:11.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:11 smithi031 ceph-mon[16336]: pgmap v1250: 47 pgs: 8 active+recovering+undersized+remapped, 39 active+clean; 6.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 44 MiB/s wr, 710 op/s; 26141/321183 objects misplaced (8.139%); 17 MiB/s, 274 objects/s recovering 2024-08-19T14:30:11.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:11 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:30:11.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:11 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:30:11.449 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:30:11.449 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:30:11.456 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:30:11.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:11.482 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:11.482 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:30:11.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:11 smithi032 ceph-mon[20004]: pgmap v1250: 47 pgs: 8 active+recovering+undersized+remapped, 39 active+clean; 6.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 44 MiB/s wr, 710 op/s; 26141/321183 objects misplaced (8.139%); 17 MiB/s, 274 objects/s recovering 2024-08-19T14:30:11.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:11 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:30:11.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:11 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:30:11.583 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:30:11.616 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:11.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:11.617 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:30:11.718 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:30:11.757 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:11.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:11.760 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:30:11.861 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:30:11.883 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:11.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:11.884 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:30:11.985 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:30:12.006 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:12.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:12.007 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:30:12.108 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:30:12.133 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:12.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:12.134 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:30:12.217 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-08-19T14:30:12.235 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:30:12.270 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:12.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:12.271 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:30:12.372 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:30:12.393 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:12.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:12.394 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:30:12.489 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:30:12.495 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:30:12.516 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:12.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:12.517 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:30:12.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:12 smithi045 ceph-mon[20143]: osdmap e543: 12 total, 12 up, 8 in 2024-08-19T14:30:12.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:12 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1921736319' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:30:12.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:12 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:30:12.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:12 smithi031 ceph-mon[16336]: osdmap e543: 12 total, 12 up, 8 in 2024-08-19T14:30:12.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:12 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1921736319' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:30:12.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:12 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:30:12.618 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:30:12.643 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:12.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:12.644 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:30:12.745 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:30:12.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:12 smithi032 ceph-mon[20004]: osdmap e543: 12 total, 12 up, 8 in 2024-08-19T14:30:12.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:12 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1921736319' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:30:12.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:12 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:30:12.770 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:12.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:12.771 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:30:12.860 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is unset 2024-08-19T14:30:12.872 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:30:12.910 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:12.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:12.911 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:30:13.002 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:30:13.003 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:30:13.012 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:30:13.036 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:13.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:13.037 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:30:13.138 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:30:13.160 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:13.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:13.161 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:30:13.262 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:30:13.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:13.285 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:13.285 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:30:13.386 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:30:13.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:13.459 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:13.459 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:30:13.515 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:13 smithi031 ceph-mon[16336]: pgmap v1252: 47 pgs: 8 active+recovering+undersized+remapped, 39 active+clean; 6.5 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 37 MiB/s wr, 597 op/s; 25864/321525 objects misplaced (8.044%); 16 MiB/s, 249 objects/s recovering 2024-08-19T14:30:13.515 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:13 smithi031 ceph-mon[16336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:30:13.515 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:13 smithi031 ceph-mon[16336]: Cluster is now healthy 2024-08-19T14:30:13.515 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:13 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:30:13.515 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:13 smithi031 ceph-mon[16336]: osdmap e544: 12 total, 12 up, 8 in 2024-08-19T14:30:13.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:13 smithi045 ceph-mon[20143]: pgmap v1252: 47 pgs: 8 active+recovering+undersized+remapped, 39 active+clean; 6.5 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 37 MiB/s wr, 597 op/s; 25864/321525 objects misplaced (8.044%); 16 MiB/s, 249 objects/s recovering 2024-08-19T14:30:13.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:13 smithi045 ceph-mon[20143]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:30:13.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:13 smithi045 ceph-mon[20143]: Cluster is now healthy 2024-08-19T14:30:13.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:13 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:30:13.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:13 smithi045 ceph-mon[20143]: osdmap e544: 12 total, 12 up, 8 in 2024-08-19T14:30:13.560 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:30:13.582 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:13.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:13.583 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:30:13.684 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:30:13.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:13.725 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:13.725 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:30:13.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:13 smithi032 ceph-mon[20004]: pgmap v1252: 47 pgs: 8 active+recovering+undersized+remapped, 39 active+clean; 6.5 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 37 MiB/s wr, 597 op/s; 25864/321525 objects misplaced (8.044%); 16 MiB/s, 249 objects/s recovering 2024-08-19T14:30:13.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:13 smithi032 ceph-mon[20004]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:30:13.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:13 smithi032 ceph-mon[20004]: Cluster is now healthy 2024-08-19T14:30:13.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:13 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:30:13.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:13 smithi032 ceph-mon[20004]: osdmap e544: 12 total, 12 up, 8 in 2024-08-19T14:30:13.778 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-08-19T14:30:13.826 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:30:13.867 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:13.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:13.868 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:30:13.970 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:30:13.994 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:13.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:13.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:30:14.044 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:30:14.083 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:30:14.096 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:30:14.102 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:30:14.118 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:14.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:14.119 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:30:14.220 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:30:14.248 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:14.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:14.249 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:30:14.350 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:30:14.401 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:14.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:14.403 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:30:14.504 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:30:14.532 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:14.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:14.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:30:14.634 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:30:14.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:14.672 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:14.672 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:30:14.773 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:30:14.795 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:14.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:14.797 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:30:14.877 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:30:14.878 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:30:14.878 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:30:14.879 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:30:14.897 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:30:14.915 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:30:14.915 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:30:14.920 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:14.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:14.921 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:30:14.959 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:30:14.959 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:30:14.991 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:30:14.992 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:30:15.021 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:30:15.028 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:30:15.029 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:30:15.061 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:30:15.062 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:30:15.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:15.074 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:15.074 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:30:15.103 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:30:15.104 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:30:15.137 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:30:15.138 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:30:15.175 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:30:15.182 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:30:15.182 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:30:15.199 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:15.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:15.200 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:30:15.232 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:30:15.232 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:30:15.279 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:30:15.280 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:30:15.301 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:30:15.319 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:30:15.320 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:30:15.320 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:15 smithi031 ceph-mon[16336]: pgmap v1254: 47 pgs: 8 active+recovering+undersized+remapped, 39 active+clean; 6.4 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 127 B/s rd, 23 MiB/s wr, 585 op/s; 24740/316533 objects misplaced (7.816%); 19 MiB/s, 304 objects/s recovering 2024-08-19T14:30:15.320 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:15 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3872476177' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:30:15.349 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:15.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:15.350 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:30:15.451 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:30:15.487 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:15.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:15.489 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:30:15.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:15 smithi045 ceph-mon[20143]: pgmap v1254: 47 pgs: 8 active+recovering+undersized+remapped, 39 active+clean; 6.4 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 127 B/s rd, 23 MiB/s wr, 585 op/s; 24740/316533 objects misplaced (7.816%); 19 MiB/s, 304 objects/s recovering 2024-08-19T14:30:15.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:15 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3872476177' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:30:15.590 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:30:15.630 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:30:15.631 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:30:15.631 INFO:teuthology.orchestra.run.smithi031.stdout: "epoch": 544, 2024-08-19T14:30:15.631 INFO:teuthology.orchestra.run.smithi031.stdout: "fsid": "ed7d1c6e-5e33-11ef-bccf-c7b262605968", 2024-08-19T14:30:15.631 INFO:teuthology.orchestra.run.smithi031.stdout: "created": "2024-08-19T14:05:22.842169+0000", 2024-08-19T14:30:15.631 INFO:teuthology.orchestra.run.smithi031.stdout: "modified": "2024-08-19T14:30:12.258268+0000", 2024-08-19T14:30:15.631 INFO:teuthology.orchestra.run.smithi031.stdout: "last_up_change": "2024-08-19T14:29:50.110723+0000", 2024-08-19T14:30:15.632 INFO:teuthology.orchestra.run.smithi031.stdout: "last_in_change": "2024-08-19T14:29:15.393401+0000", 2024-08-19T14:30:15.632 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-08-19T14:30:15.632 INFO:teuthology.orchestra.run.smithi031.stdout: "flags_num": 5799936, 2024-08-19T14:30:15.632 INFO:teuthology.orchestra.run.smithi031.stdout: "flags_set": [ 2024-08-19T14:30:15.632 INFO:teuthology.orchestra.run.smithi031.stdout: "pglog_hardlimit", 2024-08-19T14:30:15.632 INFO:teuthology.orchestra.run.smithi031.stdout: "purged_snapdirs", 2024-08-19T14:30:15.632 INFO:teuthology.orchestra.run.smithi031.stdout: "recovery_deletes", 2024-08-19T14:30:15.632 INFO:teuthology.orchestra.run.smithi031.stdout: "sortbitwise" 2024-08-19T14:30:15.632 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-08-19T14:30:15.633 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_version": 28, 2024-08-19T14:30:15.633 INFO:teuthology.orchestra.run.smithi031.stdout: "full_ratio": 0.94999998807907104, 2024-08-19T14:30:15.633 INFO:teuthology.orchestra.run.smithi031.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-08-19T14:30:15.633 INFO:teuthology.orchestra.run.smithi031.stdout: "nearfull_ratio": 0.85000002384185791, 2024-08-19T14:30:15.633 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_snapshot": "", 2024-08-19T14:30:15.633 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_max": 6, 2024-08-19T14:30:15.633 INFO:teuthology.orchestra.run.smithi031.stdout: "max_osd": 12, 2024-08-19T14:30:15.633 INFO:teuthology.orchestra.run.smithi031.stdout: "require_min_compat_client": "luminous", 2024-08-19T14:30:15.633 INFO:teuthology.orchestra.run.smithi031.stdout: "min_compat_client": "luminous", 2024-08-19T14:30:15.633 INFO:teuthology.orchestra.run.smithi031.stdout: "require_osd_release": "squid", 2024-08-19T14:30:15.633 INFO:teuthology.orchestra.run.smithi031.stdout: "allow_crimson": false, 2024-08-19T14:30:15.634 INFO:teuthology.orchestra.run.smithi031.stdout: "pools": [ 2024-08-19T14:30:15.634 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.634 INFO:teuthology.orchestra.run.smithi031.stdout: "pool": 1, 2024-08-19T14:30:15.634 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_name": ".mgr", 2024-08-19T14:30:15.634 INFO:teuthology.orchestra.run.smithi031.stdout: "create_time": "2024-08-19T14:08:15.966221+0000", 2024-08-19T14:30:15.634 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-08-19T14:30:15.634 INFO:teuthology.orchestra.run.smithi031.stdout: "flags_names": "hashpspool", 2024-08-19T14:30:15.634 INFO:teuthology.orchestra.run.smithi031.stdout: "type": 1, 2024-08-19T14:30:15.634 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 3, 2024-08-19T14:30:15.634 INFO:teuthology.orchestra.run.smithi031.stdout: "min_size": 2, 2024-08-19T14:30:15.634 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_rule": 0, 2024-08-19T14:30:15.634 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_count": 0, 2024-08-19T14:30:15.635 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_target": 0, 2024-08-19T14:30:15.635 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_barrier": 0, 2024-08-19T14:30:15.635 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-08-19T14:30:15.635 INFO:teuthology.orchestra.run.smithi031.stdout: "object_hash": 2, 2024-08-19T14:30:15.635 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_autoscale_mode": "off", 2024-08-19T14:30:15.635 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num": 31, 2024-08-19T14:30:15.635 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_placement_num": 31, 2024-08-19T14:30:15.635 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_placement_num_target": 31, 2024-08-19T14:30:15.635 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num_target": 31, 2024-08-19T14:30:15.635 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num_pending": 31, 2024-08-19T14:30:15.636 INFO:teuthology.orchestra.run.smithi031.stdout: "last_pg_merge_meta": { 2024-08-19T14:30:15.636 INFO:teuthology.orchestra.run.smithi031.stdout: "source_pgid": "0.0", 2024-08-19T14:30:15.636 INFO:teuthology.orchestra.run.smithi031.stdout: "ready_epoch": 0, 2024-08-19T14:30:15.636 INFO:teuthology.orchestra.run.smithi031.stdout: "last_epoch_started": 0, 2024-08-19T14:30:15.636 INFO:teuthology.orchestra.run.smithi031.stdout: "last_epoch_clean": 0, 2024-08-19T14:30:15.636 INFO:teuthology.orchestra.run.smithi031.stdout: "source_version": "0'0", 2024-08-19T14:30:15.636 INFO:teuthology.orchestra.run.smithi031.stdout: "target_version": "0'0" 2024-08-19T14:30:15.636 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.636 INFO:teuthology.orchestra.run.smithi031.stdout: "last_change": "278", 2024-08-19T14:30:15.636 INFO:teuthology.orchestra.run.smithi031.stdout: "last_force_op_resend": "0", 2024-08-19T14:30:15.636 INFO:teuthology.orchestra.run.smithi031.stdout: "last_force_op_resend_prenautilus": "0", 2024-08-19T14:30:15.636 INFO:teuthology.orchestra.run.smithi031.stdout: "last_force_op_resend_preluminous": "276", 2024-08-19T14:30:15.636 INFO:teuthology.orchestra.run.smithi031.stdout: "auid": 0, 2024-08-19T14:30:15.637 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_mode": "selfmanaged", 2024-08-19T14:30:15.637 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_seq": 0, 2024-08-19T14:30:15.637 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_epoch": 0, 2024-08-19T14:30:15.637 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_snaps": [], 2024-08-19T14:30:15.637 INFO:teuthology.orchestra.run.smithi031.stdout: "removed_snaps": "[]", 2024-08-19T14:30:15.637 INFO:teuthology.orchestra.run.smithi031.stdout: "quota_max_bytes": 0, 2024-08-19T14:30:15.637 INFO:teuthology.orchestra.run.smithi031.stdout: "quota_max_objects": 0, 2024-08-19T14:30:15.637 INFO:teuthology.orchestra.run.smithi031.stdout: "tiers": [], 2024-08-19T14:30:15.637 INFO:teuthology.orchestra.run.smithi031.stdout: "tier_of": -1, 2024-08-19T14:30:15.637 INFO:teuthology.orchestra.run.smithi031.stdout: "read_tier": -1, 2024-08-19T14:30:15.637 INFO:teuthology.orchestra.run.smithi031.stdout: "write_tier": -1, 2024-08-19T14:30:15.637 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_mode": "none", 2024-08-19T14:30:15.638 INFO:teuthology.orchestra.run.smithi031.stdout: "target_max_bytes": 0, 2024-08-19T14:30:15.638 INFO:teuthology.orchestra.run.smithi031.stdout: "target_max_objects": 0, 2024-08-19T14:30:15.638 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-08-19T14:30:15.638 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-08-19T14:30:15.638 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_target_full_ratio_micro": 800000, 2024-08-19T14:30:15.638 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_min_flush_age": 0, 2024-08-19T14:30:15.638 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_min_evict_age": 0, 2024-08-19T14:30:15.638 INFO:teuthology.orchestra.run.smithi031.stdout: "erasure_code_profile": "", 2024-08-19T14:30:15.638 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_params": { 2024-08-19T14:30:15.638 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "none" 2024-08-19T14:30:15.638 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.638 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_period": 0, 2024-08-19T14:30:15.639 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_count": 0, 2024-08-19T14:30:15.639 INFO:teuthology.orchestra.run.smithi031.stdout: "use_gmt_hitset": true, 2024-08-19T14:30:15.639 INFO:teuthology.orchestra.run.smithi031.stdout: "min_read_recency_for_promote": 0, 2024-08-19T14:30:15.639 INFO:teuthology.orchestra.run.smithi031.stdout: "min_write_recency_for_promote": 0, 2024-08-19T14:30:15.639 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_grade_decay_rate": 0, 2024-08-19T14:30:15.639 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_search_last_n": 0, 2024-08-19T14:30:15.639 INFO:teuthology.orchestra.run.smithi031.stdout: "grade_table": [], 2024-08-19T14:30:15.639 INFO:teuthology.orchestra.run.smithi031.stdout: "stripe_width": 0, 2024-08-19T14:30:15.639 INFO:teuthology.orchestra.run.smithi031.stdout: "expected_num_objects": 0, 2024-08-19T14:30:15.639 INFO:teuthology.orchestra.run.smithi031.stdout: "fast_read": false, 2024-08-19T14:30:15.639 INFO:teuthology.orchestra.run.smithi031.stdout: "options": { 2024-08-19T14:30:15.639 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num_max": 32, 2024-08-19T14:30:15.639 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num_min": 1 2024-08-19T14:30:15.640 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.640 INFO:teuthology.orchestra.run.smithi031.stdout: "application_metadata": { 2024-08-19T14:30:15.640 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": {} 2024-08-19T14:30:15.640 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.640 INFO:teuthology.orchestra.run.smithi031.stdout: "read_balance": { 2024-08-19T14:30:15.640 INFO:teuthology.orchestra.run.smithi031.stdout: "score_type": "Fair distribution", 2024-08-19T14:30:15.640 INFO:teuthology.orchestra.run.smithi031.stdout: "score_acting": 2.5699999332427979, 2024-08-19T14:30:15.640 INFO:teuthology.orchestra.run.smithi031.stdout: "score_stable": 2.5699999332427979, 2024-08-19T14:30:15.640 INFO:teuthology.orchestra.run.smithi031.stdout: "optimal_score": 0.80000001192092896, 2024-08-19T14:30:15.640 INFO:teuthology.orchestra.run.smithi031.stdout: "raw_score_acting": 2.059999942779541, 2024-08-19T14:30:15.640 INFO:teuthology.orchestra.run.smithi031.stdout: "raw_score_stable": 2.059999942779541, 2024-08-19T14:30:15.640 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity_weighted": 0.75614923238754272, 2024-08-19T14:30:15.640 INFO:teuthology.orchestra.run.smithi031.stdout: "average_primary_affinity": 0.9100000262260437, 2024-08-19T14:30:15.641 INFO:teuthology.orchestra.run.smithi031.stdout: "average_primary_affinity_weighted": 0.75999999046325684 2024-08-19T14:30:15.641 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:30:15.641 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.641 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.641 INFO:teuthology.orchestra.run.smithi031.stdout: "pool": 6, 2024-08-19T14:30:15.641 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_name": "unique_pool_4", 2024-08-19T14:30:15.641 INFO:teuthology.orchestra.run.smithi031.stdout: "create_time": "2024-08-19T14:28:32.089179+0000", 2024-08-19T14:30:15.641 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-08-19T14:30:15.641 INFO:teuthology.orchestra.run.smithi031.stdout: "flags_names": "hashpspool", 2024-08-19T14:30:15.641 INFO:teuthology.orchestra.run.smithi031.stdout: "type": 1, 2024-08-19T14:30:15.641 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 3, 2024-08-19T14:30:15.641 INFO:teuthology.orchestra.run.smithi031.stdout: "min_size": 2, 2024-08-19T14:30:15.642 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_rule": 0, 2024-08-19T14:30:15.642 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_count": 0, 2024-08-19T14:30:15.642 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_target": 0, 2024-08-19T14:30:15.642 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_barrier": 0, 2024-08-19T14:30:15.642 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-08-19T14:30:15.642 INFO:teuthology.orchestra.run.smithi031.stdout: "object_hash": 2, 2024-08-19T14:30:15.642 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_autoscale_mode": "off", 2024-08-19T14:30:15.642 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num": 16, 2024-08-19T14:30:15.642 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_placement_num": 16, 2024-08-19T14:30:15.642 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_placement_num_target": 16, 2024-08-19T14:30:15.642 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num_target": 16, 2024-08-19T14:30:15.642 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num_pending": 16, 2024-08-19T14:30:15.642 INFO:teuthology.orchestra.run.smithi031.stdout: "last_pg_merge_meta": { 2024-08-19T14:30:15.643 INFO:teuthology.orchestra.run.smithi031.stdout: "source_pgid": "0.0", 2024-08-19T14:30:15.643 INFO:teuthology.orchestra.run.smithi031.stdout: "ready_epoch": 0, 2024-08-19T14:30:15.643 INFO:teuthology.orchestra.run.smithi031.stdout: "last_epoch_started": 0, 2024-08-19T14:30:15.643 INFO:teuthology.orchestra.run.smithi031.stdout: "last_epoch_clean": 0, 2024-08-19T14:30:15.643 INFO:teuthology.orchestra.run.smithi031.stdout: "source_version": "0'0", 2024-08-19T14:30:15.644 INFO:teuthology.orchestra.run.smithi031.stdout: "target_version": "0'0" 2024-08-19T14:30:15.644 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.644 INFO:teuthology.orchestra.run.smithi031.stdout: "last_change": "513", 2024-08-19T14:30:15.644 INFO:teuthology.orchestra.run.smithi031.stdout: "last_force_op_resend": "0", 2024-08-19T14:30:15.644 INFO:teuthology.orchestra.run.smithi031.stdout: "last_force_op_resend_prenautilus": "0", 2024-08-19T14:30:15.644 INFO:teuthology.orchestra.run.smithi031.stdout: "last_force_op_resend_preluminous": "0", 2024-08-19T14:30:15.644 INFO:teuthology.orchestra.run.smithi031.stdout: "auid": 0, 2024-08-19T14:30:15.644 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_mode": "selfmanaged", 2024-08-19T14:30:15.644 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_seq": 0, 2024-08-19T14:30:15.644 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_epoch": 0, 2024-08-19T14:30:15.644 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_snaps": [], 2024-08-19T14:30:15.644 INFO:teuthology.orchestra.run.smithi031.stdout: "removed_snaps": "[]", 2024-08-19T14:30:15.644 INFO:teuthology.orchestra.run.smithi031.stdout: "quota_max_bytes": 0, 2024-08-19T14:30:15.644 INFO:teuthology.orchestra.run.smithi031.stdout: "quota_max_objects": 0, 2024-08-19T14:30:15.645 INFO:teuthology.orchestra.run.smithi031.stdout: "tiers": [], 2024-08-19T14:30:15.645 INFO:teuthology.orchestra.run.smithi031.stdout: "tier_of": -1, 2024-08-19T14:30:15.645 INFO:teuthology.orchestra.run.smithi031.stdout: "read_tier": -1, 2024-08-19T14:30:15.645 INFO:teuthology.orchestra.run.smithi031.stdout: "write_tier": -1, 2024-08-19T14:30:15.645 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_mode": "none", 2024-08-19T14:30:15.645 INFO:teuthology.orchestra.run.smithi031.stdout: "target_max_bytes": 0, 2024-08-19T14:30:15.645 INFO:teuthology.orchestra.run.smithi031.stdout: "target_max_objects": 0, 2024-08-19T14:30:15.645 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-08-19T14:30:15.645 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-08-19T14:30:15.645 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_target_full_ratio_micro": 800000, 2024-08-19T14:30:15.645 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_min_flush_age": 0, 2024-08-19T14:30:15.646 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_min_evict_age": 0, 2024-08-19T14:30:15.646 INFO:teuthology.orchestra.run.smithi031.stdout: "erasure_code_profile": "", 2024-08-19T14:30:15.646 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_params": { 2024-08-19T14:30:15.646 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "none" 2024-08-19T14:30:15.646 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.646 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_period": 0, 2024-08-19T14:30:15.646 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_count": 0, 2024-08-19T14:30:15.646 INFO:teuthology.orchestra.run.smithi031.stdout: "use_gmt_hitset": true, 2024-08-19T14:30:15.646 INFO:teuthology.orchestra.run.smithi031.stdout: "min_read_recency_for_promote": 0, 2024-08-19T14:30:15.646 INFO:teuthology.orchestra.run.smithi031.stdout: "min_write_recency_for_promote": 0, 2024-08-19T14:30:15.646 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_grade_decay_rate": 0, 2024-08-19T14:30:15.646 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_search_last_n": 0, 2024-08-19T14:30:15.646 INFO:teuthology.orchestra.run.smithi031.stdout: "grade_table": [], 2024-08-19T14:30:15.646 INFO:teuthology.orchestra.run.smithi031.stdout: "stripe_width": 0, 2024-08-19T14:30:15.647 INFO:teuthology.orchestra.run.smithi031.stdout: "expected_num_objects": 0, 2024-08-19T14:30:15.647 INFO:teuthology.orchestra.run.smithi031.stdout: "fast_read": false, 2024-08-19T14:30:15.647 INFO:teuthology.orchestra.run.smithi031.stdout: "options": {}, 2024-08-19T14:30:15.647 INFO:teuthology.orchestra.run.smithi031.stdout: "application_metadata": { 2024-08-19T14:30:15.647 INFO:teuthology.orchestra.run.smithi031.stdout: "rados": {} 2024-08-19T14:30:15.647 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.647 INFO:teuthology.orchestra.run.smithi031.stdout: "read_balance": { 2024-08-19T14:30:15.647 INFO:teuthology.orchestra.run.smithi031.stdout: "score_type": "Fair distribution", 2024-08-19T14:30:15.647 INFO:teuthology.orchestra.run.smithi031.stdout: "score_acting": 2.5, 2024-08-19T14:30:15.647 INFO:teuthology.orchestra.run.smithi031.stdout: "score_stable": 2.5, 2024-08-19T14:30:15.648 INFO:teuthology.orchestra.run.smithi031.stdout: "optimal_score": 0.80000001192092896, 2024-08-19T14:30:15.648 INFO:teuthology.orchestra.run.smithi031.stdout: "raw_score_acting": 2, 2024-08-19T14:30:15.648 INFO:teuthology.orchestra.run.smithi031.stdout: "raw_score_stable": 2, 2024-08-19T14:30:15.648 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity_weighted": 0.75614923238754272, 2024-08-19T14:30:15.648 INFO:teuthology.orchestra.run.smithi031.stdout: "average_primary_affinity": 0.9100000262260437, 2024-08-19T14:30:15.648 INFO:teuthology.orchestra.run.smithi031.stdout: "average_primary_affinity_weighted": 0.75999999046325684 2024-08-19T14:30:15.648 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:30:15.648 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:30:15.648 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-08-19T14:30:15.648 INFO:teuthology.orchestra.run.smithi031.stdout: "osds": [ 2024-08-19T14:30:15.648 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.648 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 0, 2024-08-19T14:30:15.649 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "d7a0fff3-23c6-4ee3-a275-a844862aabfb", 2024-08-19T14:30:15.649 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-08-19T14:30:15.649 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-08-19T14:30:15.649 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-08-19T14:30:15.649 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-08-19T14:30:15.649 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-08-19T14:30:15.649 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-08-19T14:30:15.649 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 8, 2024-08-19T14:30:15.649 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 527, 2024-08-19T14:30:15.649 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-08-19T14:30:15.649 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-08-19T14:30:15.649 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-08-19T14:30:15.649 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:30:15.649 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.650 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:30:15.650 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6802", 2024-08-19T14:30:15.650 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1108945097 2024-08-19T14:30:15.650 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.650 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.650 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:30:15.650 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6803", 2024-08-19T14:30:15.650 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1108945097 2024-08-19T14:30:15.650 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:30:15.650 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:30:15.650 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.650 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-08-19T14:30:15.651 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:30:15.651 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.651 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:30:15.651 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6804", 2024-08-19T14:30:15.651 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1108945097 2024-08-19T14:30:15.651 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.651 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.652 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:30:15.652 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6805", 2024-08-19T14:30:15.652 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1108945097 2024-08-19T14:30:15.652 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:30:15.652 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:30:15.652 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.652 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-08-19T14:30:15.652 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:30:15.652 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.652 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:30:15.652 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6808", 2024-08-19T14:30:15.652 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1108945097 2024-08-19T14:30:15.652 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.652 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.653 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:30:15.653 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6809", 2024-08-19T14:30:15.653 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1108945097 2024-08-19T14:30:15.653 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:30:15.653 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:30:15.653 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.653 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-08-19T14:30:15.653 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:30:15.653 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.653 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:30:15.654 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6806", 2024-08-19T14:30:15.654 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1108945097 2024-08-19T14:30:15.654 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.654 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.654 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:30:15.654 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6807", 2024-08-19T14:30:15.654 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1108945097 2024-08-19T14:30:15.654 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:30:15.654 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:30:15.654 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.654 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.31:6803/1108945097", 2024-08-19T14:30:15.655 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.31:6805/1108945097", 2024-08-19T14:30:15.655 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.31:6809/1108945097", 2024-08-19T14:30:15.655 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.31:6807/1108945097", 2024-08-19T14:30:15.655 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-08-19T14:30:15.655 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-08-19T14:30:15.655 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-08-19T14:30:15.655 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:30:15.655 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.655 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.655 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 1, 2024-08-19T14:30:15.655 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "145d5bb4-74dd-440f-bcc9-7e9ebda9c5d6", 2024-08-19T14:30:15.655 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-08-19T14:30:15.655 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-08-19T14:30:15.656 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-08-19T14:30:15.656 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-08-19T14:30:15.656 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-08-19T14:30:15.656 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-08-19T14:30:15.656 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 13, 2024-08-19T14:30:15.656 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 527, 2024-08-19T14:30:15.657 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-08-19T14:30:15.657 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-08-19T14:30:15.657 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-08-19T14:30:15.657 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:30:15.657 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.657 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:30:15.657 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6810", 2024-08-19T14:30:15.657 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3236186806 2024-08-19T14:30:15.657 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.657 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.657 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:30:15.657 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6811", 2024-08-19T14:30:15.657 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3236186806 2024-08-19T14:30:15.658 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:30:15.658 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:30:15.658 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.658 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-08-19T14:30:15.658 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:30:15.658 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.658 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:30:15.658 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6812", 2024-08-19T14:30:15.658 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3236186806 2024-08-19T14:30:15.658 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.658 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.658 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:30:15.658 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6813", 2024-08-19T14:30:15.658 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3236186806 2024-08-19T14:30:15.659 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:30:15.659 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:30:15.659 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.659 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-08-19T14:30:15.659 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:30:15.659 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.659 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:30:15.659 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6816", 2024-08-19T14:30:15.659 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3236186806 2024-08-19T14:30:15.659 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.659 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.659 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:30:15.660 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6817", 2024-08-19T14:30:15.660 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3236186806 2024-08-19T14:30:15.660 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:30:15.660 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:30:15.660 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.660 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-08-19T14:30:15.660 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:30:15.660 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.660 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:30:15.660 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6814", 2024-08-19T14:30:15.660 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3236186806 2024-08-19T14:30:15.660 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.660 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.660 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:30:15.661 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6815", 2024-08-19T14:30:15.661 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3236186806 2024-08-19T14:30:15.661 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:30:15.661 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:30:15.661 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.661 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.31:6811/3236186806", 2024-08-19T14:30:15.661 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.31:6813/3236186806", 2024-08-19T14:30:15.661 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.31:6817/3236186806", 2024-08-19T14:30:15.661 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.31:6815/3236186806", 2024-08-19T14:30:15.661 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-08-19T14:30:15.661 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-08-19T14:30:15.661 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-08-19T14:30:15.662 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:30:15.662 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.662 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.662 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 2, 2024-08-19T14:30:15.662 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "376c18d0-4768-46a0-bb60-e79c88ca909c", 2024-08-19T14:30:15.662 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-08-19T14:30:15.662 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-08-19T14:30:15.662 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-08-19T14:30:15.662 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 0.07379150390625, 2024-08-19T14:30:15.662 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-08-19T14:30:15.662 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-08-19T14:30:15.662 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 18, 2024-08-19T14:30:15.663 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 204, 2024-08-19T14:30:15.663 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-08-19T14:30:15.663 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-08-19T14:30:15.663 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-08-19T14:30:15.663 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:30:15.663 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.663 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:30:15.663 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6818", 2024-08-19T14:30:15.663 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 4045553387 2024-08-19T14:30:15.663 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.663 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.663 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:30:15.663 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6819", 2024-08-19T14:30:15.664 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 4045553387 2024-08-19T14:30:15.664 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:30:15.664 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:30:15.664 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.664 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-08-19T14:30:15.664 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:30:15.664 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.664 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:30:15.664 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6820", 2024-08-19T14:30:15.664 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 4045553387 2024-08-19T14:30:15.664 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.664 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.664 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:30:15.664 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6821", 2024-08-19T14:30:15.665 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 4045553387 2024-08-19T14:30:15.665 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:30:15.665 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:30:15.665 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.665 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-08-19T14:30:15.665 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:30:15.665 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.665 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:30:15.665 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6824", 2024-08-19T14:30:15.665 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 4045553387 2024-08-19T14:30:15.665 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.665 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.665 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:30:15.666 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6825", 2024-08-19T14:30:15.666 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 4045553387 2024-08-19T14:30:15.666 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:30:15.666 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:30:15.666 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.666 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-08-19T14:30:15.666 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:30:15.666 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.666 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:30:15.666 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6822", 2024-08-19T14:30:15.666 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 4045553387 2024-08-19T14:30:15.666 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.666 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.667 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:30:15.667 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6823", 2024-08-19T14:30:15.667 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 4045553387 2024-08-19T14:30:15.667 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:30:15.667 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:30:15.667 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.667 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.31:6819/4045553387", 2024-08-19T14:30:15.667 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.31:6821/4045553387", 2024-08-19T14:30:15.667 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.31:6825/4045553387", 2024-08-19T14:30:15.667 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.31:6823/4045553387", 2024-08-19T14:30:15.667 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-08-19T14:30:15.667 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-08-19T14:30:15.667 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-08-19T14:30:15.667 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:30:15.668 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.668 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.668 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 3, 2024-08-19T14:30:15.668 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "a787ea9f-b9ba-450f-b388-614cfba90634", 2024-08-19T14:30:15.668 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-08-19T14:30:15.668 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-08-19T14:30:15.668 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-08-19T14:30:15.668 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-08-19T14:30:15.668 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-08-19T14:30:15.668 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-08-19T14:30:15.668 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 26, 2024-08-19T14:30:15.668 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 527, 2024-08-19T14:30:15.668 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-08-19T14:30:15.669 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-08-19T14:30:15.669 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-08-19T14:30:15.669 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:30:15.669 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.669 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:30:15.669 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6826", 2024-08-19T14:30:15.669 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2288082716 2024-08-19T14:30:15.669 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.669 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.669 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:30:15.669 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6827", 2024-08-19T14:30:15.669 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2288082716 2024-08-19T14:30:15.669 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:30:15.669 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:30:15.670 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.670 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-08-19T14:30:15.670 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:30:15.670 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.670 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:30:15.670 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6828", 2024-08-19T14:30:15.670 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2288082716 2024-08-19T14:30:15.670 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.670 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.670 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:30:15.670 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6829", 2024-08-19T14:30:15.670 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2288082716 2024-08-19T14:30:15.670 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:30:15.671 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:30:15.671 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.671 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-08-19T14:30:15.671 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:30:15.671 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.671 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:30:15.671 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6832", 2024-08-19T14:30:15.671 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2288082716 2024-08-19T14:30:15.671 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:15.673 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:15.673 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:30:15.673 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.673 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:30:15.673 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6833", 2024-08-19T14:30:15.674 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2288082716 2024-08-19T14:30:15.674 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:30:15.674 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:30:15.674 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.674 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-08-19T14:30:15.674 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:30:15.674 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.674 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:30:15.674 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6830", 2024-08-19T14:30:15.674 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2288082716 2024-08-19T14:30:15.674 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.674 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.675 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:30:15.675 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6831", 2024-08-19T14:30:15.675 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2288082716 2024-08-19T14:30:15.675 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:30:15.675 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:30:15.675 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.675 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.31:6827/2288082716", 2024-08-19T14:30:15.675 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.31:6829/2288082716", 2024-08-19T14:30:15.675 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.31:6833/2288082716", 2024-08-19T14:30:15.675 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.31:6831/2288082716", 2024-08-19T14:30:15.675 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-08-19T14:30:15.675 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-08-19T14:30:15.675 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-08-19T14:30:15.676 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:30:15.676 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.676 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.676 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 4, 2024-08-19T14:30:15.676 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "7859751a-12f5-4a57-a332-435360479d53", 2024-08-19T14:30:15.676 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-08-19T14:30:15.676 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-08-19T14:30:15.676 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-08-19T14:30:15.676 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-08-19T14:30:15.676 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-08-19T14:30:15.676 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-08-19T14:30:15.676 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 31, 2024-08-19T14:30:15.676 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 527, 2024-08-19T14:30:15.676 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-08-19T14:30:15.677 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-08-19T14:30:15.677 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-08-19T14:30:15.677 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:30:15.677 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.677 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:30:15.677 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6800", 2024-08-19T14:30:15.677 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3934739984 2024-08-19T14:30:15.677 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.677 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.678 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:30:15.678 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6801", 2024-08-19T14:30:15.678 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3934739984 2024-08-19T14:30:15.678 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:30:15.678 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:30:15.678 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.678 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-08-19T14:30:15.678 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:30:15.678 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.678 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:30:15.678 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6802", 2024-08-19T14:30:15.678 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3934739984 2024-08-19T14:30:15.679 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.679 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.679 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:30:15.679 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6803", 2024-08-19T14:30:15.679 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3934739984 2024-08-19T14:30:15.679 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:30:15.679 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:30:15.679 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.679 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-08-19T14:30:15.679 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:30:15.679 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.679 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:30:15.679 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6806", 2024-08-19T14:30:15.679 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3934739984 2024-08-19T14:30:15.680 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.680 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.680 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:30:15.680 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6807", 2024-08-19T14:30:15.680 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3934739984 2024-08-19T14:30:15.680 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:30:15.680 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:30:15.680 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.680 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-08-19T14:30:15.680 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:30:15.680 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.680 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:30:15.680 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6804", 2024-08-19T14:30:15.681 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3934739984 2024-08-19T14:30:15.681 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.681 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.681 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:30:15.681 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6805", 2024-08-19T14:30:15.681 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3934739984 2024-08-19T14:30:15.681 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:30:15.681 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:30:15.681 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.681 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.32:6801/3934739984", 2024-08-19T14:30:15.681 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.32:6803/3934739984", 2024-08-19T14:30:15.681 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.32:6807/3934739984", 2024-08-19T14:30:15.681 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.32:6805/3934739984", 2024-08-19T14:30:15.682 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-08-19T14:30:15.682 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-08-19T14:30:15.682 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-08-19T14:30:15.682 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:30:15.682 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.682 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.682 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 5, 2024-08-19T14:30:15.682 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "017f3eef-ac60-4b82-8350-919017cb4ccd", 2024-08-19T14:30:15.682 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-08-19T14:30:15.682 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-08-19T14:30:15.682 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-08-19T14:30:15.682 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 0, 2024-08-19T14:30:15.682 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-08-19T14:30:15.683 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-08-19T14:30:15.683 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 40, 2024-08-19T14:30:15.683 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 464, 2024-08-19T14:30:15.683 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-08-19T14:30:15.683 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-08-19T14:30:15.683 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-08-19T14:30:15.683 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:30:15.683 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.683 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:30:15.683 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6808", 2024-08-19T14:30:15.683 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 526388985 2024-08-19T14:30:15.683 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.683 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.683 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:30:15.684 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6809", 2024-08-19T14:30:15.684 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 526388985 2024-08-19T14:30:15.684 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:30:15.684 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:30:15.684 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.684 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-08-19T14:30:15.684 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:30:15.684 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.684 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:30:15.684 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6810", 2024-08-19T14:30:15.684 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 526388985 2024-08-19T14:30:15.684 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.684 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.685 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:30:15.685 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6811", 2024-08-19T14:30:15.685 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 526388985 2024-08-19T14:30:15.685 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:30:15.685 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:30:15.685 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.685 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-08-19T14:30:15.685 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:30:15.685 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.685 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:30:15.685 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6814", 2024-08-19T14:30:15.685 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 526388985 2024-08-19T14:30:15.685 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.685 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.686 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:30:15.686 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6815", 2024-08-19T14:30:15.686 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 526388985 2024-08-19T14:30:15.686 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:30:15.686 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:30:15.686 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.686 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-08-19T14:30:15.686 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:30:15.686 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.686 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:30:15.686 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6812", 2024-08-19T14:30:15.686 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 526388985 2024-08-19T14:30:15.686 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.686 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.687 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:30:15.687 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6813", 2024-08-19T14:30:15.687 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 526388985 2024-08-19T14:30:15.687 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:30:15.687 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:30:15.687 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.687 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.32:6809/526388985", 2024-08-19T14:30:15.687 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.32:6811/526388985", 2024-08-19T14:30:15.687 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.32:6815/526388985", 2024-08-19T14:30:15.687 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.32:6813/526388985", 2024-08-19T14:30:15.687 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-08-19T14:30:15.687 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-08-19T14:30:15.687 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-08-19T14:30:15.687 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:30:15.688 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.688 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.688 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 6, 2024-08-19T14:30:15.688 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "44aa8229-0268-49d7-bd57-3bf1f384e5e6", 2024-08-19T14:30:15.688 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-08-19T14:30:15.688 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 0, 2024-08-19T14:30:15.688 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 0, 2024-08-19T14:30:15.688 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-08-19T14:30:15.688 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-08-19T14:30:15.688 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-08-19T14:30:15.688 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 48, 2024-08-19T14:30:15.688 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 126, 2024-08-19T14:30:15.688 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-08-19T14:30:15.689 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-08-19T14:30:15.689 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-08-19T14:30:15.689 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:30:15.689 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.689 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:30:15.689 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6816", 2024-08-19T14:30:15.689 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1131035844 2024-08-19T14:30:15.689 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.689 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.689 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:30:15.689 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6817", 2024-08-19T14:30:15.689 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1131035844 2024-08-19T14:30:15.690 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:30:15.690 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:30:15.690 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.690 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-08-19T14:30:15.690 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:30:15.690 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.690 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:30:15.690 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6818", 2024-08-19T14:30:15.690 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1131035844 2024-08-19T14:30:15.690 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.690 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.690 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:30:15.690 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6819", 2024-08-19T14:30:15.690 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1131035844 2024-08-19T14:30:15.691 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:30:15.691 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:30:15.691 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.691 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-08-19T14:30:15.691 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:30:15.691 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.691 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:30:15.691 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6822", 2024-08-19T14:30:15.691 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1131035844 2024-08-19T14:30:15.691 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.691 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.691 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:30:15.691 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6823", 2024-08-19T14:30:15.692 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1131035844 2024-08-19T14:30:15.692 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:30:15.692 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:30:15.692 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.692 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-08-19T14:30:15.692 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:30:15.692 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.692 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:30:15.692 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6820", 2024-08-19T14:30:15.692 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1131035844 2024-08-19T14:30:15.692 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.692 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.692 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:30:15.692 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6821", 2024-08-19T14:30:15.693 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1131035844 2024-08-19T14:30:15.693 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:30:15.693 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:30:15.693 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.693 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.32:6817/1131035844", 2024-08-19T14:30:15.693 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.32:6819/1131035844", 2024-08-19T14:30:15.693 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.32:6823/1131035844", 2024-08-19T14:30:15.693 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.32:6821/1131035844", 2024-08-19T14:30:15.693 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-08-19T14:30:15.693 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-08-19T14:30:15.693 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-08-19T14:30:15.693 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:30:15.693 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.694 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.694 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 7, 2024-08-19T14:30:15.694 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "263433e5-70b6-4192-86ff-dc2ad95927eb", 2024-08-19T14:30:15.694 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-08-19T14:30:15.694 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-08-19T14:30:15.694 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-08-19T14:30:15.694 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-08-19T14:30:15.694 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-08-19T14:30:15.694 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-08-19T14:30:15.694 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 57, 2024-08-19T14:30:15.694 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 527, 2024-08-19T14:30:15.694 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-08-19T14:30:15.694 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-08-19T14:30:15.695 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-08-19T14:30:15.695 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:30:15.695 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.695 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:30:15.695 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6824", 2024-08-19T14:30:15.695 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2014939947 2024-08-19T14:30:15.695 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.695 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.695 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:30:15.695 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6825", 2024-08-19T14:30:15.695 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2014939947 2024-08-19T14:30:15.695 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:30:15.695 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:30:15.695 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.696 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-08-19T14:30:15.696 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:30:15.696 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.696 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:30:15.696 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6826", 2024-08-19T14:30:15.696 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2014939947 2024-08-19T14:30:15.696 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.696 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.696 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:30:15.696 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6827", 2024-08-19T14:30:15.696 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2014939947 2024-08-19T14:30:15.696 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:30:15.696 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:30:15.696 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.697 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-08-19T14:30:15.697 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:30:15.697 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.697 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:30:15.697 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6830", 2024-08-19T14:30:15.697 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2014939947 2024-08-19T14:30:15.697 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.697 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.697 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:30:15.697 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6831", 2024-08-19T14:30:15.697 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2014939947 2024-08-19T14:30:15.697 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:30:15.697 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:30:15.698 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.698 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-08-19T14:30:15.698 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:30:15.698 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.698 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:30:15.698 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6828", 2024-08-19T14:30:15.698 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2014939947 2024-08-19T14:30:15.698 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.698 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.698 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:30:15.698 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6829", 2024-08-19T14:30:15.698 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2014939947 2024-08-19T14:30:15.698 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:30:15.699 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:30:15.699 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.699 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.32:6825/2014939947", 2024-08-19T14:30:15.699 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.32:6827/2014939947", 2024-08-19T14:30:15.699 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.32:6831/2014939947", 2024-08-19T14:30:15.699 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.32:6829/2014939947", 2024-08-19T14:30:15.699 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-08-19T14:30:15.699 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-08-19T14:30:15.699 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-08-19T14:30:15.699 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:30:15.699 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.699 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.699 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 8, 2024-08-19T14:30:15.699 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "2bbcad36-0c74-4528-9fb9-a8812afa1662", 2024-08-19T14:30:15.700 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-08-19T14:30:15.700 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-08-19T14:30:15.700 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-08-19T14:30:15.700 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-08-19T14:30:15.700 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-08-19T14:30:15.700 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-08-19T14:30:15.700 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 65, 2024-08-19T14:30:15.700 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 527, 2024-08-19T14:30:15.700 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-08-19T14:30:15.700 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-08-19T14:30:15.700 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-08-19T14:30:15.700 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:30:15.701 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.701 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:30:15.701 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6800", 2024-08-19T14:30:15.701 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1445140892 2024-08-19T14:30:15.701 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.701 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.701 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:30:15.701 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6801", 2024-08-19T14:30:15.701 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1445140892 2024-08-19T14:30:15.701 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:30:15.701 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:30:15.701 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.701 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-08-19T14:30:15.702 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:30:15.702 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.702 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:30:15.702 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6802", 2024-08-19T14:30:15.702 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1445140892 2024-08-19T14:30:15.702 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.702 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.702 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:30:15.702 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6803", 2024-08-19T14:30:15.702 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1445140892 2024-08-19T14:30:15.702 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:30:15.702 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:30:15.702 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.703 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-08-19T14:30:15.703 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:30:15.703 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.703 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:30:15.703 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6806", 2024-08-19T14:30:15.703 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1445140892 2024-08-19T14:30:15.703 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.703 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.703 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:30:15.703 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6807", 2024-08-19T14:30:15.703 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1445140892 2024-08-19T14:30:15.703 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:30:15.703 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:30:15.703 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.704 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-08-19T14:30:15.704 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:30:15.704 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.704 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:30:15.704 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6804", 2024-08-19T14:30:15.704 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1445140892 2024-08-19T14:30:15.704 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.704 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.704 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:30:15.704 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6805", 2024-08-19T14:30:15.704 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1445140892 2024-08-19T14:30:15.704 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:30:15.704 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:30:15.705 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.705 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.45:6801/1445140892", 2024-08-19T14:30:15.705 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.45:6803/1445140892", 2024-08-19T14:30:15.705 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.45:6807/1445140892", 2024-08-19T14:30:15.705 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.45:6805/1445140892", 2024-08-19T14:30:15.705 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-08-19T14:30:15.705 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-08-19T14:30:15.705 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-08-19T14:30:15.705 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:30:15.705 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.705 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.705 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 9, 2024-08-19T14:30:15.705 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "c8ee4277-3bf9-46bf-9cc0-152c1c4b9508", 2024-08-19T14:30:15.706 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-08-19T14:30:15.706 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 0, 2024-08-19T14:30:15.706 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 0, 2024-08-19T14:30:15.706 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 0, 2024-08-19T14:30:15.706 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-08-19T14:30:15.706 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-08-19T14:30:15.706 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 74, 2024-08-19T14:30:15.706 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 419, 2024-08-19T14:30:15.706 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-08-19T14:30:15.706 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-08-19T14:30:15.706 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-08-19T14:30:15.706 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:30:15.706 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.707 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:30:15.707 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6808", 2024-08-19T14:30:15.707 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1213787169 2024-08-19T14:30:15.707 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.707 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.707 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:30:15.707 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6809", 2024-08-19T14:30:15.707 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1213787169 2024-08-19T14:30:15.707 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:30:15.707 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:30:15.707 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.707 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-08-19T14:30:15.707 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:30:15.708 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.708 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:30:15.708 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6810", 2024-08-19T14:30:15.708 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1213787169 2024-08-19T14:30:15.708 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.708 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.708 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:30:15.708 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6811", 2024-08-19T14:30:15.708 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1213787169 2024-08-19T14:30:15.708 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:30:15.708 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:30:15.708 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.708 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-08-19T14:30:15.708 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:30:15.709 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.709 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:30:15.709 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6814", 2024-08-19T14:30:15.709 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1213787169 2024-08-19T14:30:15.709 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.709 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.709 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:30:15.709 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6815", 2024-08-19T14:30:15.709 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1213787169 2024-08-19T14:30:15.709 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:30:15.709 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:30:15.709 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.709 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-08-19T14:30:15.709 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:30:15.710 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.710 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:30:15.710 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6812", 2024-08-19T14:30:15.710 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1213787169 2024-08-19T14:30:15.710 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.710 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.710 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:30:15.710 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6813", 2024-08-19T14:30:15.710 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1213787169 2024-08-19T14:30:15.710 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:30:15.710 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:30:15.710 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.710 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.45:6809/1213787169", 2024-08-19T14:30:15.710 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.45:6811/1213787169", 2024-08-19T14:30:15.711 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.45:6815/1213787169", 2024-08-19T14:30:15.711 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.45:6813/1213787169", 2024-08-19T14:30:15.711 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-08-19T14:30:15.711 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-08-19T14:30:15.711 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-08-19T14:30:15.711 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:30:15.711 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.711 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.711 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 10, 2024-08-19T14:30:15.711 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "29031395-5f5c-40f8-ac90-9f6ac48e5c9f", 2024-08-19T14:30:15.711 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-08-19T14:30:15.711 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 0, 2024-08-19T14:30:15.712 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 0, 2024-08-19T14:30:15.712 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-08-19T14:30:15.712 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 80, 2024-08-19T14:30:15.712 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 529, 2024-08-19T14:30:15.712 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 539, 2024-08-19T14:30:15.713 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 115, 2024-08-19T14:30:15.713 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 530, 2024-08-19T14:30:15.713 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-08-19T14:30:15.713 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-08-19T14:30:15.713 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:30:15.713 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.713 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:30:15.713 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6816", 2024-08-19T14:30:15.713 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1069743893 2024-08-19T14:30:15.713 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.713 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.713 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:30:15.713 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6817", 2024-08-19T14:30:15.714 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1069743893 2024-08-19T14:30:15.714 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:30:15.714 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:30:15.714 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.714 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-08-19T14:30:15.714 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:30:15.714 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.714 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:30:15.714 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6818", 2024-08-19T14:30:15.714 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1069743893 2024-08-19T14:30:15.714 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.714 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.714 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:30:15.715 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6819", 2024-08-19T14:30:15.715 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1069743893 2024-08-19T14:30:15.715 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:30:15.715 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:30:15.715 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.715 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-08-19T14:30:15.715 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:30:15.715 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.715 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:30:15.715 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6822", 2024-08-19T14:30:15.715 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1069743893 2024-08-19T14:30:15.715 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.715 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.716 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:30:15.716 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6823", 2024-08-19T14:30:15.716 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1069743893 2024-08-19T14:30:15.716 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:30:15.716 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:30:15.716 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.716 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-08-19T14:30:15.716 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:30:15.716 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.716 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:30:15.716 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6820", 2024-08-19T14:30:15.716 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1069743893 2024-08-19T14:30:15.716 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.716 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.717 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:30:15.717 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6821", 2024-08-19T14:30:15.717 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1069743893 2024-08-19T14:30:15.717 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:30:15.717 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:30:15.717 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.717 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.45:6817/1069743893", 2024-08-19T14:30:15.717 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.45:6819/1069743893", 2024-08-19T14:30:15.717 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.45:6823/1069743893", 2024-08-19T14:30:15.717 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.45:6821/1069743893", 2024-08-19T14:30:15.717 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-08-19T14:30:15.717 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-08-19T14:30:15.717 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-08-19T14:30:15.718 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:30:15.718 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.718 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.718 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 11, 2024-08-19T14:30:15.718 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "5ef0db4b-0fef-4ea0-a19b-d3677e082c29", 2024-08-19T14:30:15.718 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-08-19T14:30:15.718 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 0, 2024-08-19T14:30:15.718 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 0, 2024-08-19T14:30:15.718 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-08-19T14:30:15.718 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-08-19T14:30:15.718 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-08-19T14:30:15.718 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 86, 2024-08-19T14:30:15.718 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 502, 2024-08-19T14:30:15.719 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-08-19T14:30:15.719 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-08-19T14:30:15.719 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-08-19T14:30:15.719 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:30:15.719 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.719 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:30:15.719 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6824", 2024-08-19T14:30:15.719 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2438396298 2024-08-19T14:30:15.719 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.719 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.719 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:30:15.719 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6825", 2024-08-19T14:30:15.719 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2438396298 2024-08-19T14:30:15.719 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:30:15.720 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:30:15.720 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.720 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-08-19T14:30:15.720 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:30:15.720 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.720 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:30:15.720 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6826", 2024-08-19T14:30:15.720 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2438396298 2024-08-19T14:30:15.720 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.720 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.720 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:30:15.720 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6827", 2024-08-19T14:30:15.720 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2438396298 2024-08-19T14:30:15.720 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:30:15.721 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:30:15.721 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.721 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-08-19T14:30:15.721 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:30:15.721 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.721 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:30:15.721 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6830", 2024-08-19T14:30:15.721 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2438396298 2024-08-19T14:30:15.721 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.721 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.721 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:30:15.721 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6831", 2024-08-19T14:30:15.721 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2438396298 2024-08-19T14:30:15.721 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:30:15.722 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:30:15.722 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.722 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-08-19T14:30:15.722 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:30:15.722 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.722 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:30:15.722 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6828", 2024-08-19T14:30:15.722 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2438396298 2024-08-19T14:30:15.722 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.722 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.722 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:30:15.722 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6829", 2024-08-19T14:30:15.722 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2438396298 2024-08-19T14:30:15.723 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:30:15.723 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:30:15.723 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.723 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.45:6825/2438396298", 2024-08-19T14:30:15.723 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.45:6827/2438396298", 2024-08-19T14:30:15.723 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.45:6831/2438396298", 2024-08-19T14:30:15.723 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.45:6829/2438396298", 2024-08-19T14:30:15.723 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-08-19T14:30:15.723 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-08-19T14:30:15.723 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-08-19T14:30:15.723 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:30:15.723 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:30:15.723 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-08-19T14:30:15.723 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_xinfo": [ 2024-08-19T14:30:15.724 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.724 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 0, 2024-08-19T14:30:15.724 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-08-19T14:30:15.724 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-08-19T14:30:15.724 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-08-19T14:30:15.724 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-08-19T14:30:15.724 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-08-19T14:30:15.724 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-08-19T14:28:53.698008+0000", 2024-08-19T14:30:15.724 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-08-19T14:30:15.724 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.724 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.724 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 1, 2024-08-19T14:30:15.724 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-08-19T14:30:15.724 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-08-19T14:30:15.724 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-08-19T14:30:15.725 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-08-19T14:30:15.725 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-08-19T14:30:15.725 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-08-19T14:29:24.156962+0000", 2024-08-19T14:30:15.725 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-08-19T14:30:15.725 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.725 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.725 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 2, 2024-08-19T14:30:15.725 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-08-19T14:30:15.725 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-08-19T14:30:15.725 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-08-19T14:30:15.725 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-08-19T14:30:15.725 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-08-19T14:30:15.725 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-08-19T14:28:56.229352+0000", 2024-08-19T14:30:15.725 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-08-19T14:30:15.726 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.726 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.726 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 3, 2024-08-19T14:30:15.726 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-08-19T14:30:15.726 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-08-19T14:30:15.726 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-08-19T14:30:15.726 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-08-19T14:30:15.726 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-08-19T14:30:15.726 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-08-19T14:29:29.083982+0000", 2024-08-19T14:30:15.726 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-08-19T14:30:15.726 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.726 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.726 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 4, 2024-08-19T14:30:15.726 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-08-19T14:30:15.727 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-08-19T14:30:15.727 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-08-19T14:30:15.727 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-08-19T14:30:15.727 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-08-19T14:30:15.727 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-08-19T14:29:24.360330+0000", 2024-08-19T14:30:15.727 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-08-19T14:30:15.727 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.727 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.727 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 5, 2024-08-19T14:30:15.727 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-08-19T14:30:15.727 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-08-19T14:30:15.727 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-08-19T14:30:15.727 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-08-19T14:30:15.728 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-08-19T14:30:15.728 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-08-19T14:30:10.593498+0000", 2024-08-19T14:30:15.728 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-08-19T14:30:15.728 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.728 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.728 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 6, 2024-08-19T14:30:15.728 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-08-19T14:30:15.728 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-08-19T14:30:15.728 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-08-19T14:30:15.728 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-08-19T14:30:15.728 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 65536, 2024-08-19T14:30:15.728 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-08-19T14:28:52.737952+0000", 2024-08-19T14:30:15.728 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-08-19T14:30:15.728 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.728 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.729 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 7, 2024-08-19T14:30:15.729 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-08-19T14:30:15.729 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-08-19T14:30:15.729 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-08-19T14:30:15.729 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-08-19T14:30:15.729 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-08-19T14:30:15.729 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-08-19T14:29:44.514874+0000", 2024-08-19T14:30:15.729 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-08-19T14:30:15.729 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.729 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.729 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 8, 2024-08-19T14:30:15.729 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-08-19T14:30:15.729 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-08-19T14:30:15.730 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-08-19T14:30:15.730 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-08-19T14:30:15.730 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-08-19T14:30:15.730 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-08-19T14:29:19.593495+0000", 2024-08-19T14:30:15.730 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-08-19T14:30:15.730 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.730 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.730 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 9, 2024-08-19T14:30:15.730 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-08-19T14:30:15.730 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-08-19T14:30:15.730 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-08-19T14:30:15.730 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-08-19T14:30:15.730 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 65536, 2024-08-19T14:30:15.730 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-08-19T14:30:02.098728+0000", 2024-08-19T14:30:15.731 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-08-19T14:30:15.731 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.731 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.731 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 10, 2024-08-19T14:30:15.731 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-08-19T14:30:15.731 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-08-19T14:30:15.731 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-08-19T14:30:15.731 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-08-19T14:30:15.731 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 65536, 2024-08-19T14:30:15.731 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-08-19T14:29:16.601743+0000", 2024-08-19T14:30:15.731 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 529 2024-08-19T14:30:15.731 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.731 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.732 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 11, 2024-08-19T14:30:15.732 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-08-19T14:30:15.732 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-08-19T14:30:15.732 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-08-19T14:30:15.732 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-08-19T14:30:15.732 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 65536, 2024-08-19T14:30:15.732 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-08-19T14:29:25.160334+0000", 2024-08-19T14:30:15.732 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-08-19T14:30:15.732 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:30:15.732 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-08-19T14:30:15.732 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_upmap": [ 2024-08-19T14:30:15.732 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.732 INFO:teuthology.orchestra.run.smithi031.stdout: "pgid": "1.0", 2024-08-19T14:30:15.732 INFO:teuthology.orchestra.run.smithi031.stdout: "osds": [ 2024-08-19T14:30:15.733 INFO:teuthology.orchestra.run.smithi031.stdout: 2, 2024-08-19T14:30:15.733 INFO:teuthology.orchestra.run.smithi031.stdout: 9, 2024-08-19T14:30:15.733 INFO:teuthology.orchestra.run.smithi031.stdout: 0 2024-08-19T14:30:15.733 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:30:15.733 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:30:15.733 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-08-19T14:30:15.733 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_upmap_items": [], 2024-08-19T14:30:15.733 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_upmap_primaries": [], 2024-08-19T14:30:15.733 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_temp": [ 2024-08-19T14:30:15.733 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.733 INFO:teuthology.orchestra.run.smithi031.stdout: "pgid": "6.0", 2024-08-19T14:30:15.733 INFO:teuthology.orchestra.run.smithi031.stdout: "osds": [ 2024-08-19T14:30:15.733 INFO:teuthology.orchestra.run.smithi031.stdout: 0, 2024-08-19T14:30:15.733 INFO:teuthology.orchestra.run.smithi031.stdout: 3 2024-08-19T14:30:15.734 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:30:15.734 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.734 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.734 INFO:teuthology.orchestra.run.smithi031.stdout: "pgid": "6.1", 2024-08-19T14:30:15.734 INFO:teuthology.orchestra.run.smithi031.stdout: "osds": [ 2024-08-19T14:30:15.734 INFO:teuthology.orchestra.run.smithi031.stdout: 8, 2024-08-19T14:30:15.734 INFO:teuthology.orchestra.run.smithi031.stdout: 7 2024-08-19T14:30:15.734 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:30:15.734 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.734 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.734 INFO:teuthology.orchestra.run.smithi031.stdout: "pgid": "6.2", 2024-08-19T14:30:15.734 INFO:teuthology.orchestra.run.smithi031.stdout: "osds": [ 2024-08-19T14:30:15.734 INFO:teuthology.orchestra.run.smithi031.stdout: 4, 2024-08-19T14:30:15.735 INFO:teuthology.orchestra.run.smithi031.stdout: 3 2024-08-19T14:30:15.735 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:30:15.735 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.735 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.735 INFO:teuthology.orchestra.run.smithi031.stdout: "pgid": "6.5", 2024-08-19T14:30:15.735 INFO:teuthology.orchestra.run.smithi031.stdout: "osds": [ 2024-08-19T14:30:15.735 INFO:teuthology.orchestra.run.smithi031.stdout: 7, 2024-08-19T14:30:15.735 INFO:teuthology.orchestra.run.smithi031.stdout: 5 2024-08-19T14:30:15.735 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:30:15.735 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.735 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.735 INFO:teuthology.orchestra.run.smithi031.stdout: "pgid": "6.9", 2024-08-19T14:30:15.735 INFO:teuthology.orchestra.run.smithi031.stdout: "osds": [ 2024-08-19T14:30:15.735 INFO:teuthology.orchestra.run.smithi031.stdout: 0, 2024-08-19T14:30:15.735 INFO:teuthology.orchestra.run.smithi031.stdout: 8 2024-08-19T14:30:15.736 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:30:15.736 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.736 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.736 INFO:teuthology.orchestra.run.smithi031.stdout: "pgid": "6.a", 2024-08-19T14:30:15.736 INFO:teuthology.orchestra.run.smithi031.stdout: "osds": [ 2024-08-19T14:30:15.736 INFO:teuthology.orchestra.run.smithi031.stdout: 0, 2024-08-19T14:30:15.736 INFO:teuthology.orchestra.run.smithi031.stdout: 5 2024-08-19T14:30:15.736 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:30:15.736 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.736 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.736 INFO:teuthology.orchestra.run.smithi031.stdout: "pgid": "6.e", 2024-08-19T14:30:15.736 INFO:teuthology.orchestra.run.smithi031.stdout: "osds": [ 2024-08-19T14:30:15.736 INFO:teuthology.orchestra.run.smithi031.stdout: 4, 2024-08-19T14:30:15.737 INFO:teuthology.orchestra.run.smithi031.stdout: 1 2024-08-19T14:30:15.737 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:30:15.737 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.737 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:15.737 INFO:teuthology.orchestra.run.smithi031.stdout: "pgid": "6.f", 2024-08-19T14:30:15.737 INFO:teuthology.orchestra.run.smithi031.stdout: "osds": [ 2024-08-19T14:30:15.737 INFO:teuthology.orchestra.run.smithi031.stdout: 4, 2024-08-19T14:30:15.737 INFO:teuthology.orchestra.run.smithi031.stdout: 5 2024-08-19T14:30:15.737 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:30:15.737 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:30:15.737 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-08-19T14:30:15.737 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_temp": [], 2024-08-19T14:30:15.737 INFO:teuthology.orchestra.run.smithi031.stdout: "blocklist": { 2024-08-19T14:30:15.737 INFO:teuthology.orchestra.run.smithi031.stdout: "172.21.15.31:0/1451745969": "2024-08-20T14:06:13.660657+0000", 2024-08-19T14:30:15.738 INFO:teuthology.orchestra.run.smithi031.stdout: "172.21.15.31:0/1515698918": "2024-08-20T14:06:13.660657+0000", 2024-08-19T14:30:15.738 INFO:teuthology.orchestra.run.smithi031.stdout: "172.21.15.31:6801/3297539242": "2024-08-20T14:06:13.660657+0000", 2024-08-19T14:30:15.738 INFO:teuthology.orchestra.run.smithi031.stdout: "172.21.15.31:6800/3297539242": "2024-08-20T14:06:13.660657+0000", 2024-08-19T14:30:15.738 INFO:teuthology.orchestra.run.smithi031.stdout: "172.21.15.31:0/2217423818": "2024-08-20T14:05:48.827576+0000", 2024-08-19T14:30:15.738 INFO:teuthology.orchestra.run.smithi031.stdout: "172.21.15.31:0/3353498281": "2024-08-20T14:06:13.660657+0000", 2024-08-19T14:30:15.738 INFO:teuthology.orchestra.run.smithi031.stdout: "172.21.15.31:0/4013594220": "2024-08-20T14:05:48.827576+0000", 2024-08-19T14:30:15.738 INFO:teuthology.orchestra.run.smithi031.stdout: "172.21.15.31:0/1654518065": "2024-08-20T14:05:48.827576+0000", 2024-08-19T14:30:15.738 INFO:teuthology.orchestra.run.smithi031.stdout: "172.21.15.31:6801/2368821258": "2024-08-20T14:05:48.827576+0000", 2024-08-19T14:30:15.738 INFO:teuthology.orchestra.run.smithi031.stdout: "172.21.15.31:6800/2368821258": "2024-08-20T14:05:48.827576+0000" 2024-08-19T14:30:15.738 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.738 INFO:teuthology.orchestra.run.smithi031.stdout: "range_blocklist": {}, 2024-08-19T14:30:15.738 INFO:teuthology.orchestra.run.smithi031.stdout: "erasure_code_profiles": { 2024-08-19T14:30:15.738 INFO:teuthology.orchestra.run.smithi031.stdout: "default": { 2024-08-19T14:30:15.738 INFO:teuthology.orchestra.run.smithi031.stdout: "crush-failure-domain": "osd", 2024-08-19T14:30:15.739 INFO:teuthology.orchestra.run.smithi031.stdout: "k": "2", 2024-08-19T14:30:15.739 INFO:teuthology.orchestra.run.smithi031.stdout: "m": "1", 2024-08-19T14:30:15.739 INFO:teuthology.orchestra.run.smithi031.stdout: "plugin": "jerasure", 2024-08-19T14:30:15.739 INFO:teuthology.orchestra.run.smithi031.stdout: "technique": "reed_sol_van" 2024-08-19T14:30:15.739 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:30:15.739 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:15.739 INFO:teuthology.orchestra.run.smithi031.stdout: "removed_snaps_queue": [], 2024-08-19T14:30:15.739 INFO:teuthology.orchestra.run.smithi031.stdout: "new_removed_snaps": [], 2024-08-19T14:30:15.739 INFO:teuthology.orchestra.run.smithi031.stdout: "new_purged_snaps": [], 2024-08-19T14:30:15.739 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_node_flags": {}, 2024-08-19T14:30:15.739 INFO:teuthology.orchestra.run.smithi031.stdout: "device_class_flags": {}, 2024-08-19T14:30:15.739 INFO:teuthology.orchestra.run.smithi031.stdout: "stretch_mode": { 2024-08-19T14:30:15.739 INFO:teuthology.orchestra.run.smithi031.stdout: "stretch_mode_enabled": false, 2024-08-19T14:30:15.739 INFO:teuthology.orchestra.run.smithi031.stdout: "stretch_bucket_count": 0, 2024-08-19T14:30:15.740 INFO:teuthology.orchestra.run.smithi031.stdout: "degraded_stretch_mode": 0, 2024-08-19T14:30:15.740 INFO:teuthology.orchestra.run.smithi031.stdout: "recovering_stretch_mode": 0, 2024-08-19T14:30:15.740 INFO:teuthology.orchestra.run.smithi031.stdout: "stretch_mode_bucket": 0 2024-08-19T14:30:15.740 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:30:15.740 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:30:15.741 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-08-19T14:30:15.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:15 smithi032 ceph-mon[20004]: pgmap v1254: 47 pgs: 8 active+recovering+undersized+remapped, 39 active+clean; 6.4 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 127 B/s rd, 23 MiB/s wr, 585 op/s; 24740/316533 objects misplaced (7.816%); 19 MiB/s, 304 objects/s recovering 2024-08-19T14:30:15.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:15 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3872476177' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:30:15.774 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:30:15.818 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:15.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:15.819 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:30:15.920 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:30:15.943 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:15.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:15.944 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:30:15.978 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-19T14:30:16.013 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:30:16.044 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:30:16.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:16.102 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:16.102 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:30:16.203 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:30:16.224 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:16.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:16.226 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:30:16.326 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:30:16.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:16.353 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:16.353 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:30:16.454 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:30:16.470 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:16 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2462425490' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-08-19T14:30:16.488 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:16.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:16.489 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:30:16.577 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:30:16.577 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [], 2024-08-19T14:30:16.577 INFO:teuthology.orchestra.run.smithi045.stdout: "num_ops": 0 2024-08-19T14:30:16.577 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:30:16.590 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:30:16.615 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:16.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:16.616 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:30:16.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:16 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2462425490' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-08-19T14:30:16.717 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:30:16.753 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:16.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:16.754 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:30:16.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:16 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2462425490' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-08-19T14:30:16.855 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:30:16.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:16.899 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:16.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:30:17.000 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:30:17.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:17.041 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:17.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:30:17.142 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:30:17.170 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:17.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:17.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:30:17.200 INFO:tasks.thrashosds.thrasher:j is {'epoch': 544, 'fsid': 'ed7d1c6e-5e33-11ef-bccf-c7b262605968', 'created': '2024-08-19T14:05:22.842169+0000', 'modified': '2024-08-19T14:30:12.258268+0000', 'last_up_change': '2024-08-19T14:29:50.110723+0000', 'last_in_change': '2024-08-19T14:29:15.393401+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 28, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 6, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-08-19T14:08:15.966221+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 31, 'pg_placement_num': 31, 'pg_placement_num_target': 31, 'pg_num_target': 31, 'pg_num_pending': 31, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '278', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '276', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.569999933242798, 'score_stable': 2.569999933242798, 'optimal_score': 0.800000011920929, 'raw_score_acting': 2.059999942779541, 'raw_score_stable': 2.059999942779541, 'primary_affinity_weighted': 0.7561492323875427, 'average_primary_affinity': 0.9100000262260437, 'average_primary_affinity_weighted': 0.7599999904632568}}, {'pool': 6, 'pool_name': 'unique_pool_4', 'create_time': '2024-08-19T14:28:32.089179+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '513', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 0.800000011920929, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.7561492323875427, 'average_primary_affinity': 0.9100000262260437, 'average_primary_affinity_weighted': 0.7599999904632568}}], 'osds': [{'osd': 0, 'uuid': 'd7a0fff3-23c6-4ee3-a275-a844862aabfb', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 527, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6802', 'nonce': 1108945097}, {'type': 'v1', 'addr': '172.21.15.31:6803', 'nonce': 1108945097}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6804', 'nonce': 1108945097}, {'type': 'v1', 'addr': '172.21.15.31:6805', 'nonce': 1108945097}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6808', 'nonce': 1108945097}, {'type': 'v1', 'addr': '172.21.15.31:6809', 'nonce': 1108945097}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6806', 'nonce': 1108945097}, {'type': 'v1', 'addr': '172.21.15.31:6807', 'nonce': 1108945097}]}, 'public_addr': '172.21.15.31:6803/1108945097', 'cluster_addr': '172.21.15.31:6805/1108945097', 'heartbeat_back_addr': '172.21.15.31:6809/1108945097', 'heartbeat_front_addr': '172.21.15.31:6807/1108945097', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '145d5bb4-74dd-440f-bcc9-7e9ebda9c5d6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 527, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6810', 'nonce': 3236186806}, {'type': 'v1', 'addr': '172.21.15.31:6811', 'nonce': 3236186806}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6812', 'nonce': 3236186806}, {'type': 'v1', 'addr': '172.21.15.31:6813', 'nonce': 3236186806}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6816', 'nonce': 3236186806}, {'type': 'v1', 'addr': '172.21.15.31:6817', 'nonce': 3236186806}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6814', 'nonce': 3236186806}, {'type': 'v1', 'addr': '172.21.15.31:6815', 'nonce': 3236186806}]}, 'public_addr': '172.21.15.31:6811/3236186806', 'cluster_addr': '172.21.15.31:6813/3236186806', 'heartbeat_back_addr': '172.21.15.31:6817/3236186806', 'heartbeat_front_addr': '172.21.15.31:6815/3236186806', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '376c18d0-4768-46a0-bb60-e79c88ca909c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.07379150390625, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 204, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6818', 'nonce': 4045553387}, {'type': 'v1', 'addr': '172.21.15.31:6819', 'nonce': 4045553387}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6820', 'nonce': 4045553387}, {'type': 'v1', 'addr': '172.21.15.31:6821', 'nonce': 4045553387}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6824', 'nonce': 4045553387}, {'type': 'v1', 'addr': '172.21.15.31:6825', 'nonce': 4045553387}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6822', 'nonce': 4045553387}, {'type': 'v1', 'addr': '172.21.15.31:6823', 'nonce': 4045553387}]}, 'public_addr': '172.21.15.31:6819/4045553387', 'cluster_addr': '172.21.15.31:6821/4045553387', 'heartbeat_back_addr': '172.21.15.31:6825/4045553387', 'heartbeat_front_addr': '172.21.15.31:6823/4045553387', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'a787ea9f-b9ba-450f-b388-614cfba90634', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 527, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6826', 'nonce': 2288082716}, {'type': 'v1', 'addr': '172.21.15.31:6827', 'nonce': 2288082716}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6828', 'nonce': 2288082716}, {'type': 'v1', 'addr': '172.21.15.31:6829', 'nonce': 2288082716}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6832', 'nonce': 2288082716}, {'type': 'v1', 'addr': '172.21.15.31:6833', 'nonce': 2288082716}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6830', 'nonce': 2288082716}, {'type': 'v1', 'addr': '172.21.15.31:6831', 'nonce': 2288082716}]}, 'public_addr': '172.21.15.31:6827/2288082716', 'cluster_addr': '172.21.15.31:6829/2288082716', 'heartbeat_back_addr': '172.21.15.31:6833/2288082716', 'heartbeat_front_addr': '172.21.15.31:6831/2288082716', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '7859751a-12f5-4a57-a332-435360479d53', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 527, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6800', 'nonce': 3934739984}, {'type': 'v1', 'addr': '172.21.15.32:6801', 'nonce': 3934739984}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6802', 'nonce': 3934739984}, {'type': 'v1', 'addr': '172.21.15.32:6803', 'nonce': 3934739984}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6806', 'nonce': 3934739984}, {'type': 'v1', 'addr': '172.21.15.32:6807', 'nonce': 3934739984}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6804', 'nonce': 3934739984}, {'type': 'v1', 'addr': '172.21.15.32:6805', 'nonce': 3934739984}]}, 'public_addr': '172.21.15.32:6801/3934739984', 'cluster_addr': '172.21.15.32:6803/3934739984', 'heartbeat_back_addr': '172.21.15.32:6807/3934739984', 'heartbeat_front_addr': '172.21.15.32:6805/3934739984', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '017f3eef-ac60-4b82-8350-919017cb4ccd', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 40, 'up_thru': 464, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6808', 'nonce': 526388985}, {'type': 'v1', 'addr': '172.21.15.32:6809', 'nonce': 526388985}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6810', 'nonce': 526388985}, {'type': 'v1', 'addr': '172.21.15.32:6811', 'nonce': 526388985}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6814', 'nonce': 526388985}, {'type': 'v1', 'addr': '172.21.15.32:6815', 'nonce': 526388985}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6812', 'nonce': 526388985}, {'type': 'v1', 'addr': '172.21.15.32:6813', 'nonce': 526388985}]}, 'public_addr': '172.21.15.32:6809/526388985', 'cluster_addr': '172.21.15.32:6811/526388985', 'heartbeat_back_addr': '172.21.15.32:6815/526388985', 'heartbeat_front_addr': '172.21.15.32:6813/526388985', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '44aa8229-0268-49d7-bd57-3bf1f384e5e6', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 48, 'up_thru': 126, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6816', 'nonce': 1131035844}, {'type': 'v1', 'addr': '172.21.15.32:6817', 'nonce': 1131035844}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6818', 'nonce': 1131035844}, {'type': 'v1', 'addr': '172.21.15.32:6819', 'nonce': 1131035844}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6822', 'nonce': 1131035844}, {'type': 'v1', 'addr': '172.21.15.32:6823', 'nonce': 1131035844}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6820', 'nonce': 1131035844}, {'type': 'v1', 'addr': '172.21.15.32:6821', 'nonce': 1131035844}]}, 'public_addr': '172.21.15.32:6817/1131035844', 'cluster_addr': '172.21.15.32:6819/1131035844', 'heartbeat_back_addr': '172.21.15.32:6823/1131035844', 'heartbeat_front_addr': '172.21.15.32:6821/1131035844', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '263433e5-70b6-4192-86ff-dc2ad95927eb', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 57, 'up_thru': 527, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6824', 'nonce': 2014939947}, {'type': 'v1', 'addr': '172.21.15.32:6825', 'nonce': 2014939947}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6826', 'nonce': 2014939947}, {'type': 'v1', 'addr': '172.21.15.32:6827', 'nonce': 2014939947}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6830', 'nonce': 2014939947}, {'type': 'v1', 'addr': '172.21.15.32:6831', 'nonce': 2014939947}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6828', 'nonce': 2014939947}, {'type': 'v1', 'addr': '172.21.15.32:6829', 'nonce': 2014939947}]}, 'public_addr': '172.21.15.32:6825/2014939947', 'cluster_addr': '172.21.15.32:6827/2014939947', 'heartbeat_back_addr': '172.21.15.32:6831/2014939947', 'heartbeat_front_addr': '172.21.15.32:6829/2014939947', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': '2bbcad36-0c74-4528-9fb9-a8812afa1662', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 65, 'up_thru': 527, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6800', 'nonce': 1445140892}, {'type': 'v1', 'addr': '172.21.15.45:6801', 'nonce': 1445140892}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6802', 'nonce': 1445140892}, {'type': 'v1', 'addr': '172.21.15.45:6803', 'nonce': 1445140892}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6806', 'nonce': 1445140892}, {'type': 'v1', 'addr': '172.21.15.45:6807', 'nonce': 1445140892}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6804', 'nonce': 1445140892}, {'type': 'v1', 'addr': '172.21.15.45:6805', 'nonce': 1445140892}]}, 'public_addr': '172.21.15.45:6801/1445140892', 'cluster_addr': '172.21.15.45:6803/1445140892', 'heartbeat_back_addr': '172.21.15.45:6807/1445140892', 'heartbeat_front_addr': '172.21.15.45:6805/1445140892', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': 'c8ee4277-3bf9-46bf-9cc0-152c1c4b9508', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 74, 'up_thru': 419, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6808', 'nonce': 1213787169}, {'type': 'v1', 'addr': '172.21.15.45:6809', 'nonce': 1213787169}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6810', 'nonce': 1213787169}, {'type': 'v1', 'addr': '172.21.15.45:6811', 'nonce': 1213787169}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6814', 'nonce': 1213787169}, {'type': 'v1', 'addr': '172.21.15.45:6815', 'nonce': 1213787169}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6812', 'nonce': 1213787169}, {'type': 'v1', 'addr': '172.21.15.45:6813', 'nonce': 1213787169}]}, 'public_addr': '172.21.15.45:6809/1213787169', 'cluster_addr': '172.21.15.45:6811/1213787169', 'heartbeat_back_addr': '172.21.15.45:6815/1213787169', 'heartbeat_front_addr': '172.21.15.45:6813/1213787169', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '29031395-5f5c-40f8-ac90-9f6ac48e5c9f', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 80, 'last_clean_end': 529, 'up_from': 539, 'up_thru': 115, 'down_at': 530, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6816', 'nonce': 1069743893}, {'type': 'v1', 'addr': '172.21.15.45:6817', 'nonce': 1069743893}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6818', 'nonce': 1069743893}, {'type': 'v1', 'addr': '172.21.15.45:6819', 'nonce': 1069743893}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6822', 'nonce': 1069743893}, {'type': 'v1', 'addr': '172.21.15.45:6823', 'nonce': 1069743893}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6820', 'nonce': 1069743893}, {'type': 'v1', 'addr': '172.21.15.45:6821', 'nonce': 1069743893}]}, 'public_addr': '172.21.15.45:6817/1069743893', 'cluster_addr': '172.21.15.45:6819/1069743893', 'heartbeat_back_addr': '172.21.15.45:6823/1069743893', 'heartbeat_front_addr': '172.21.15.45:6821/1069743893', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': '5ef0db4b-0fef-4ea0-a19b-d3677e082c29', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 86, 'up_thru': 502, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6824', 'nonce': 2438396298}, {'type': 'v1', 'addr': '172.21.15.45:6825', 'nonce': 2438396298}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6826', 'nonce': 2438396298}, {'type': 'v1', 'addr': '172.21.15.45:6827', 'nonce': 2438396298}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6830', 'nonce': 2438396298}, {'type': 'v1', 'addr': '172.21.15.45:6831', 'nonce': 2438396298}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6828', 'nonce': 2438396298}, {'type': 'v1', 'addr': '172.21.15.45:6829', 'nonce': 2438396298}]}, 'public_addr': '172.21.15.45:6825/2438396298', 'cluster_addr': '172.21.15.45:6827/2438396298', 'heartbeat_back_addr': '172.21.15.45:6831/2438396298', 'heartbeat_front_addr': '172.21.15.45:6829/2438396298', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-19T14:28:53.698008+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-19T14:29:24.156962+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-19T14:28:56.229352+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-19T14:29:29.083982+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-19T14:29:24.360330+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-19T14:30:10.593498+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-08-19T14:28:52.737952+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-19T14:29:44.514874+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-19T14:29:19.593495+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-08-19T14:30:02.098728+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-08-19T14:29:16.601743+0000', 'dead_epoch': 529}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-08-19T14:29:25.160334+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '1.0', 'osds': [2, 9, 0]}], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '6.0', 'osds': [0, 3]}, {'pgid': '6.1', 'osds': [8, 7]}, {'pgid': '6.2', 'osds': [4, 3]}, {'pgid': '6.5', 'osds': [7, 5]}, {'pgid': '6.9', 'osds': [0, 8]}, {'pgid': '6.a', 'osds': [0, 5]}, {'pgid': '6.e', 'osds': [4, 1]}, {'pgid': '6.f', 'osds': [4, 5]}], 'primary_temp': [], 'blocklist': {'172.21.15.31:0/1451745969': '2024-08-20T14:06:13.660657+0000', '172.21.15.31:0/1515698918': '2024-08-20T14:06:13.660657+0000', '172.21.15.31:6801/3297539242': '2024-08-20T14:06:13.660657+0000', '172.21.15.31:6800/3297539242': '2024-08-20T14:06:13.660657+0000', '172.21.15.31:0/2217423818': '2024-08-20T14:05:48.827576+0000', '172.21.15.31:0/3353498281': '2024-08-20T14:06:13.660657+0000', '172.21.15.31:0/4013594220': '2024-08-20T14:05:48.827576+0000', '172.21.15.31:0/1654518065': '2024-08-20T14:05:48.827576+0000', '172.21.15.31:6801/2368821258': '2024-08-20T14:05:48.827576+0000', '172.21.15.31:6800/2368821258': '2024-08-20T14:05:48.827576+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-08-19T14:30:17.201 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-19T14:30:17.273 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:30:17.295 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:17.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:17.296 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:30:17.319 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:30:17.371 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-08-19T14:30:17.397 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:30:17.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:17.454 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:17.454 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:30:17.481 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:17 smithi031 ceph-mon[16336]: pgmap v1255: 47 pgs: 8 active+recovering+undersized+remapped, 39 active+clean; 6.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 127 B/s rd, 19 MiB/s wr, 600 op/s; 24263/314574 objects misplaced (7.713%); 20 MiB/s, 323 objects/s recovering 2024-08-19T14:30:17.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:17 smithi045 ceph-mon[20143]: pgmap v1255: 47 pgs: 8 active+recovering+undersized+remapped, 39 active+clean; 6.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 127 B/s rd, 19 MiB/s wr, 600 op/s; 24263/314574 objects misplaced (7.713%); 20 MiB/s, 323 objects/s recovering 2024-08-19T14:30:17.555 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:30:17.579 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:17.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:17.580 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:30:17.634 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:30:17.681 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:30:17.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:17.703 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:17.704 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:30:17.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:17 smithi032 ceph-mon[20004]: pgmap v1255: 47 pgs: 8 active+recovering+undersized+remapped, 39 active+clean; 6.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 127 B/s rd, 19 MiB/s wr, 600 op/s; 24263/314574 objects misplaced (7.713%); 20 MiB/s, 323 objects/s recovering 2024-08-19T14:30:17.804 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:30:17.825 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:17.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:17.827 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:30:17.927 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:30:17.961 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:17.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:17.961 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:30:18.063 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:30:18.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:18.086 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:18.086 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:30:18.122 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:30:18.122 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [], 2024-08-19T14:30:18.122 INFO:teuthology.orchestra.run.smithi045.stdout: "complaint_time": 30, 2024-08-19T14:30:18.122 INFO:teuthology.orchestra.run.smithi045.stdout: "num_blocked_ops": 0 2024-08-19T14:30:18.122 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:30:18.162 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:30:18.187 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:30:18.214 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:18.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:18.215 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:30:18.253 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:30:18.316 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:30:18.333 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:18 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:30:18.333 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:18 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:30:18.333 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:18 smithi031 ceph-mon[16336]: osdmap e545: 12 total, 12 up, 8 in 2024-08-19T14:30:18.344 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:18.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:18.345 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:30:18.445 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:30:18.467 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:18.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:18.468 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:30:18.569 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:30:18.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:18.625 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:18.626 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:30:18.727 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:30:18.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:18 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:30:18.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:18 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:30:18.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:18 smithi032 ceph-mon[20004]: osdmap e545: 12 total, 12 up, 8 in 2024-08-19T14:30:18.779 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:18.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:18.780 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:30:18.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:18 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:30:18.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:18 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:30:18.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:18 smithi045 ceph-mon[20143]: osdmap e545: 12 total, 12 up, 8 in 2024-08-19T14:30:18.880 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:30:18.885 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:30:18.919 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-08-19T14:30:18.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:18.939 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:18.939 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:30:19.040 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:30:19.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:19.063 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:19.063 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:30:19.153 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:30:19.155 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-08-19T14:30:19.164 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:30:19.184 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:30:19.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:19.206 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:19.206 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:30:19.307 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:30:19.317 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is set 2024-08-19T14:30:19.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:19.344 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:19.344 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:30:19.444 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:30:19.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:19.489 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:19.489 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:30:19.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:19 smithi045 ceph-mon[20143]: pgmap v1257: 47 pgs: 8 active+recovering+undersized+remapped, 39 active+clean; 6.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 150 B/s rd, 687 KiB/s wr, 362 op/s; 24263/314574 objects misplaced (7.713%); 15 MiB/s, 235 objects/s recovering 2024-08-19T14:30:19.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:19 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:30:19.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:19 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:30:19.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:19 smithi045 ceph-mon[20143]: Health check failed: Degraded data redundancy: 8 pgs undersized (PG_DEGRADED) 2024-08-19T14:30:19.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:19 smithi045 ceph-mon[20143]: osdmap e546: 12 total, 12 up, 8 in 2024-08-19T14:30:19.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:19 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3051016702' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:30:19.590 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:30:19.596 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:19 smithi031 ceph-mon[16336]: pgmap v1257: 47 pgs: 8 active+recovering+undersized+remapped, 39 active+clean; 6.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 150 B/s rd, 687 KiB/s wr, 362 op/s; 24263/314574 objects misplaced (7.713%); 15 MiB/s, 235 objects/s recovering 2024-08-19T14:30:19.596 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:19 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:30:19.596 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:19 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:30:19.596 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:19 smithi031 ceph-mon[16336]: Health check failed: Degraded data redundancy: 8 pgs undersized (PG_DEGRADED) 2024-08-19T14:30:19.596 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:19 smithi031 ceph-mon[16336]: osdmap e546: 12 total, 12 up, 8 in 2024-08-19T14:30:19.596 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:19 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3051016702' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:30:19.612 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:19.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:19.613 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:30:19.685 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:30:19.685 INFO:teuthology.orchestra.run.smithi045.stdout: "size": 20, 2024-08-19T14:30:19.685 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 600, 2024-08-19T14:30:19.685 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [ 2024-08-19T14:30:19.685 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.685 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:22707 4.5 4:aa5c4d32:::benchmark_data_smithi045_71562_object22706:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-08-19T14:30:19.685 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:20:41.355284+0000", 2024-08-19T14:30:19.685 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 578.32743707899999, 2024-08-19T14:30:19.686 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.32540933900000002, 2024-08-19T14:30:19.686 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:30:19.686 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:30:19.686 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:30:19.686 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:30:19.686 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:30:19.686 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:30:19.686 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 22707 2024-08-19T14:30:19.686 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.686 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:30:19.686 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.686 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:30:19.686 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.355284+0000", 2024-08-19T14:30:19.687 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:30:19.687 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.687 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.687 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:30:19.687 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.355284+0000", 2024-08-19T14:30:19.687 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:30:19.687 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.687 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.687 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:30:19.687 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.355285+0000", 2024-08-19T14:30:19.687 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.2699999999999998e-07 2024-08-19T14:30:19.688 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.688 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.688 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:30:19.688 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.355304+0000", 2024-08-19T14:30:19.688 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8834e-05 2024-08-19T14:30:19.688 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.689 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.689 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:30:19.689 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.355315+0000", 2024-08-19T14:30:19.689 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1388e-05 2024-08-19T14:30:19.689 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.689 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.689 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:30:19.690 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.355326+0000", 2024-08-19T14:30:19.690 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0101999999999999e-05 2024-08-19T14:30:19.690 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.690 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.690 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:30:19.690 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.355381+0000", 2024-08-19T14:30:19.690 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.5114999999999997e-05 2024-08-19T14:30:19.690 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.690 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.691 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:30:19.691 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.355481+0000", 2024-08-19T14:30:19.691 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000100448 2024-08-19T14:30:19.691 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.691 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.691 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,7", 2024-08-19T14:30:19.691 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.355563+0000", 2024-08-19T14:30:19.691 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.1891999999999995e-05 2024-08-19T14:30:19.691 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.691 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.691 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:30:19.691 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.356429+0000", 2024-08-19T14:30:19.691 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00086571700000000001 2024-08-19T14:30:19.692 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.692 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.692 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:30:19.692 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.356898+0000", 2024-08-19T14:30:19.692 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000469171 2024-08-19T14:30:19.692 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.692 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.692 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:30:19.692 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.680624+0000", 2024-08-19T14:30:19.692 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.32372564599999998 2024-08-19T14:30:19.692 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.692 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.693 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:30:19.693 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.680661+0000", 2024-08-19T14:30:19.693 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.7589999999999998e-05 2024-08-19T14:30:19.693 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.693 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.693 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:30:19.693 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.680694+0000", 2024-08-19T14:30:19.693 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.2509000000000003e-05 2024-08-19T14:30:19.693 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:30:19.693 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:30:19.693 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:30:19.693 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.693 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.694 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:22723 4.5 4:a03b0b8c:::benchmark_data_smithi045_71562_object22722:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-08-19T14:30:19.694 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:20:41.360250+0000", 2024-08-19T14:30:19.694 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 578.32247174600002, 2024-08-19T14:30:19.694 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.32180027, 2024-08-19T14:30:19.694 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:30:19.694 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:30:19.694 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:30:19.694 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:30:19.694 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:30:19.695 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:30:19.695 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 22723 2024-08-19T14:30:19.695 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.695 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:30:19.695 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.695 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:30:19.695 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.360250+0000", 2024-08-19T14:30:19.695 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:30:19.695 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.696 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.696 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:30:19.696 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.360250+0000", 2024-08-19T14:30:19.696 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:30:19.696 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.696 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.696 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:30:19.696 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.360251+0000", 2024-08-19T14:30:19.696 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7740000000000001e-06 2024-08-19T14:30:19.696 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.696 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.696 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:30:19.696 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.360271+0000", 2024-08-19T14:30:19.696 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9755999999999999e-05 2024-08-19T14:30:19.697 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.697 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.697 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:30:19.697 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.360280+0000", 2024-08-19T14:30:19.697 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.268e-06 2024-08-19T14:30:19.697 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.697 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.697 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:30:19.697 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.360286+0000", 2024-08-19T14:30:19.697 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2700000000000004e-06 2024-08-19T14:30:19.697 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.697 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.697 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:30:19.697 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.360339+0000", 2024-08-19T14:30:19.698 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.3644999999999999e-05 2024-08-19T14:30:19.698 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.698 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.698 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:30:19.698 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.360424+0000", 2024-08-19T14:30:19.698 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.4493999999999994e-05 2024-08-19T14:30:19.698 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.698 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.698 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,7", 2024-08-19T14:30:19.698 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.360529+0000", 2024-08-19T14:30:19.698 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00010503200000000001 2024-08-19T14:30:19.698 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.699 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.699 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:30:19.699 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.362147+0000", 2024-08-19T14:30:19.699 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001617718 2024-08-19T14:30:19.699 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.699 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.699 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:30:19.699 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.362188+0000", 2024-08-19T14:30:19.699 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.1553e-05 2024-08-19T14:30:19.699 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.699 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.700 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:30:19.700 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.681950+0000", 2024-08-19T14:30:19.700 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.31976217299999998 2024-08-19T14:30:19.700 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.700 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.700 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:30:19.700 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.682008+0000", 2024-08-19T14:30:19.700 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7494000000000002e-05 2024-08-19T14:30:19.700 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.700 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.700 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:30:19.700 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.682050+0000", 2024-08-19T14:30:19.700 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.2092999999999998e-05 2024-08-19T14:30:19.700 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:30:19.701 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:30:19.701 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:30:19.701 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.701 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.701 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:22725 4.5 4:adb3fee7:::benchmark_data_smithi045_71562_object22724:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-08-19T14:30:19.701 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:20:41.361334+0000", 2024-08-19T14:30:19.701 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 578.32138777199998, 2024-08-19T14:30:19.701 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.32179169200000002, 2024-08-19T14:30:19.701 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:30:19.701 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:30:19.702 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:30:19.702 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:30:19.702 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:30:19.702 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:30:19.702 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 22725 2024-08-19T14:30:19.703 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.703 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:30:19.703 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.703 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:30:19.703 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.361334+0000", 2024-08-19T14:30:19.704 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:30:19.704 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.704 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.704 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:30:19.705 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.361334+0000", 2024-08-19T14:30:19.705 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:30:19.705 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.705 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.705 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:30:19.706 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.361334+0000", 2024-08-19T14:30:19.706 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.61e-07 2024-08-19T14:30:19.706 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.706 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.706 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:30:19.706 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.361353+0000", 2024-08-19T14:30:19.706 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9074999999999999e-05 2024-08-19T14:30:19.707 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.707 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.707 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:30:19.707 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.361365+0000", 2024-08-19T14:30:19.707 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1498000000000001e-05 2024-08-19T14:30:19.707 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.708 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.708 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:30:19.708 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.361371+0000", 2024-08-19T14:30:19.708 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.1290000000000001e-06 2024-08-19T14:30:19.708 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.708 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.709 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:30:19.709 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.361415+0000", 2024-08-19T14:30:19.709 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.3445999999999997e-05 2024-08-19T14:30:19.709 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.709 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.709 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:30:19.710 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.361542+0000", 2024-08-19T14:30:19.710 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012710400000000001 2024-08-19T14:30:19.710 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.710 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.710 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,7", 2024-08-19T14:30:19.710 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.361632+0000", 2024-08-19T14:30:19.710 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.9963999999999994e-05 2024-08-19T14:30:19.710 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.711 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.711 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:30:19.711 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.362288+0000", 2024-08-19T14:30:19.711 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00065625500000000001 2024-08-19T14:30:19.711 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.711 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.711 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:30:19.711 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.362699+0000", 2024-08-19T14:30:19.711 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00041120600000000001 2024-08-19T14:30:19.711 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.711 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.711 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:30:19.711 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.683027+0000", 2024-08-19T14:30:19.712 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.32032799699999998 2024-08-19T14:30:19.712 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.712 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.712 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:30:19.712 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.683084+0000", 2024-08-19T14:30:19.712 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7054e-05 2024-08-19T14:30:19.712 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.712 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.712 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:30:19.712 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:20:41.683125+0000", 2024-08-19T14:30:19.712 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.1202999999999998e-05 2024-08-19T14:30:19.712 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:30:19.712 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:30:19.712 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:30:19.713 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.713 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.713 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:116285 4.8 4:15b9fca8:::benchmark_data_smithi045_71562_object116284:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-08-19T14:30:19.713 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:21:48.600383+0000", 2024-08-19T14:30:19.713 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 511.082338531, 2024-08-19T14:30:19.713 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.25002218100000001, 2024-08-19T14:30:19.713 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:30:19.713 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:30:19.713 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:30:19.713 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:30:19.713 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:30:19.713 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:30:19.713 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 116285 2024-08-19T14:30:19.713 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.714 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:30:19.714 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.714 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:30:19.714 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.600383+0000", 2024-08-19T14:30:19.714 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:30:19.714 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.714 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.714 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:30:19.714 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.600383+0000", 2024-08-19T14:30:19.714 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:30:19.714 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.714 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.714 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:30:19.715 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.600384+0000", 2024-08-19T14:30:19.715 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.3200000000000003e-07 2024-08-19T14:30:19.715 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.715 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.715 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:30:19.715 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.600403+0000", 2024-08-19T14:30:19.715 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9326000000000001e-05 2024-08-19T14:30:19.715 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.715 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.715 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:30:19.715 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.600414+0000", 2024-08-19T14:30:19.715 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1129e-05 2024-08-19T14:30:19.715 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.716 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.716 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:30:19.716 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.600421+0000", 2024-08-19T14:30:19.716 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.8589999999999999e-06 2024-08-19T14:30:19.716 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.716 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.716 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:30:19.716 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.600478+0000", 2024-08-19T14:30:19.716 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.6502000000000002e-05 2024-08-19T14:30:19.716 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.716 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.716 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:30:19.717 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.600573+0000", 2024-08-19T14:30:19.717 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.5165000000000001e-05 2024-08-19T14:30:19.717 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.717 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.717 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 7,9", 2024-08-19T14:30:19.717 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.600654+0000", 2024-08-19T14:30:19.717 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.1002999999999997e-05 2024-08-19T14:30:19.717 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.717 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.717 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:30:19.717 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.607545+0000", 2024-08-19T14:30:19.717 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0068915039999999997 2024-08-19T14:30:19.717 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.717 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.718 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:30:19.718 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.610300+0000", 2024-08-19T14:30:19.718 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0027545199999999999 2024-08-19T14:30:19.718 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.718 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.718 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:30:19.718 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.850322+0000", 2024-08-19T14:30:19.718 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.24002195200000001 2024-08-19T14:30:19.718 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.718 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.718 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:30:19.719 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.850364+0000", 2024-08-19T14:30:19.719 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.2582e-05 2024-08-19T14:30:19.719 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.719 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.719 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:30:19.719 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.850405+0000", 2024-08-19T14:30:19.719 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.0707000000000002e-05 2024-08-19T14:30:19.719 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:30:19.719 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:30:19.719 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:30:19.719 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.719 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.719 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:116286 4.8 4:198dc556:::benchmark_data_smithi045_71562_object116285:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-08-19T14:30:19.719 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:21:48.600439+0000", 2024-08-19T14:30:19.720 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 511.08228250399998, 2024-08-19T14:30:19.720 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.25012546899999999, 2024-08-19T14:30:19.720 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:30:19.720 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:30:19.720 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:30:19.720 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:30:19.720 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:30:19.720 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:30:19.720 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 116286 2024-08-19T14:30:19.720 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.720 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:30:19.720 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.720 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:30:19.721 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.600439+0000", 2024-08-19T14:30:19.721 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:30:19.721 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.721 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.721 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:30:19.721 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.600439+0000", 2024-08-19T14:30:19.721 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:30:19.721 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:30:19.723 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.723 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.723 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:30:19.723 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.600439+0000", 2024-08-19T14:30:19.723 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2900000000000004e-07 2024-08-19T14:30:19.723 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.724 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.724 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:30:19.724 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.600458+0000", 2024-08-19T14:30:19.724 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8729999999999999e-05 2024-08-19T14:30:19.724 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.724 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.724 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:30:19.724 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.600466+0000", 2024-08-19T14:30:19.724 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.1030000000000002e-06 2024-08-19T14:30:19.724 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.724 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.724 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:30:19.725 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.600472+0000", 2024-08-19T14:30:19.725 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4380000000000003e-06 2024-08-19T14:30:19.725 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.725 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.725 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:30:19.725 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.603745+0000", 2024-08-19T14:30:19.725 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0032732350000000002 2024-08-19T14:30:19.725 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.725 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.725 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:30:19.725 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.603905+0000", 2024-08-19T14:30:19.725 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015985500000000001 2024-08-19T14:30:19.725 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.726 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.726 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 7,9", 2024-08-19T14:30:19.726 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.604060+0000", 2024-08-19T14:30:19.726 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015485799999999999 2024-08-19T14:30:19.726 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.726 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.726 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:30:19.726 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.607871+0000", 2024-08-19T14:30:19.726 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0038114540000000001 2024-08-19T14:30:19.726 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.726 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.726 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:30:19.726 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.610365+0000", 2024-08-19T14:30:19.727 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0024943449999999998 2024-08-19T14:30:19.727 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.727 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.727 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:30:19.727 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.850477+0000", 2024-08-19T14:30:19.727 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.24011110099999999 2024-08-19T14:30:19.727 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.727 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.727 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:30:19.728 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.850528+0000", 2024-08-19T14:30:19.728 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1684999999999997e-05 2024-08-19T14:30:19.728 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.728 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.728 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:30:19.728 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.850564+0000", 2024-08-19T14:30:19.729 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.6136000000000001e-05 2024-08-19T14:30:19.729 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:30:19.729 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:30:19.729 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:30:19.729 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.730 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.730 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:116297 4.8 4:17724d94:::benchmark_data_smithi045_71562_object116296:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-08-19T14:30:19.730 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:21:48.612374+0000", 2024-08-19T14:30:19.730 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 511.070347427, 2024-08-19T14:30:19.730 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.28029534299999997, 2024-08-19T14:30:19.730 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:30:19.731 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:30:19.731 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:30:19.731 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:30:19.731 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:30:19.731 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:30:19.731 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 116297 2024-08-19T14:30:19.732 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.732 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:30:19.732 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.732 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:30:19.732 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.612374+0000", 2024-08-19T14:30:19.732 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:30:19.733 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.733 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.733 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:30:19.733 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.612374+0000", 2024-08-19T14:30:19.733 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:30:19.733 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.733 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.734 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:30:19.734 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.612375+0000", 2024-08-19T14:30:19.734 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2869999999999999e-06 2024-08-19T14:30:19.734 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.734 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.735 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:30:19.735 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.612393+0000", 2024-08-19T14:30:19.735 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7864e-05 2024-08-19T14:30:19.735 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.735 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.735 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:30:19.735 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.612404+0000", 2024-08-19T14:30:19.736 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1377e-05 2024-08-19T14:30:19.736 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.736 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.736 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:30:19.736 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.612410+0000", 2024-08-19T14:30:19.737 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.3730000000000001e-06 2024-08-19T14:30:19.737 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.737 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.737 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:30:19.737 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.612460+0000", 2024-08-19T14:30:19.738 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.9675999999999997e-05 2024-08-19T14:30:19.738 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.738 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.738 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:30:19.738 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.612561+0000", 2024-08-19T14:30:19.738 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00010157099999999999 2024-08-19T14:30:19.738 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.739 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.739 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 7,9", 2024-08-19T14:30:19.739 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.612676+0000", 2024-08-19T14:30:19.739 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00011491 2024-08-19T14:30:19.739 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.739 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.740 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:30:19.740 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.616157+0000", 2024-08-19T14:30:19.740 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00348073 2024-08-19T14:30:19.740 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.740 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.741 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:30:19.741 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.616249+0000", 2024-08-19T14:30:19.741 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.2085999999999999e-05 2024-08-19T14:30:19.741 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.741 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.741 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:30:19.741 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.850604+0000", 2024-08-19T14:30:19.741 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.23435562900000001 2024-08-19T14:30:19.742 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.742 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.742 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:30:19.742 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.892613+0000", 2024-08-19T14:30:19.742 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.042008839999999999 2024-08-19T14:30:19.742 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.742 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.742 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:30:19.742 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.892669+0000", 2024-08-19T14:30:19.742 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.5999999999999999e-05 2024-08-19T14:30:19.742 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:30:19.743 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:30:19.743 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:30:19.743 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.743 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.743 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:116299 4.3 4:c0da230c:::benchmark_data_smithi045_71562_object116298:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-08-19T14:30:19.743 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:21:48.612978+0000", 2024-08-19T14:30:19.743 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 511.06974313799998, 2024-08-19T14:30:19.744 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.23536219899999999, 2024-08-19T14:30:19.744 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:30:19.744 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:30:19.744 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:30:19.744 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:30:19.745 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:30:19.745 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:30:19.745 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 116299 2024-08-19T14:30:19.745 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.745 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:30:19.745 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.745 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:30:19.746 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.612978+0000", 2024-08-19T14:30:19.746 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:30:19.746 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.746 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.746 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:30:19.746 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.612978+0000", 2024-08-19T14:30:19.746 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:30:19.747 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.747 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.747 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:30:19.747 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.612979+0000", 2024-08-19T14:30:19.748 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.7599999999999996e-07 2024-08-19T14:30:19.748 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.748 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.748 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:30:19.748 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.612998+0000", 2024-08-19T14:30:19.748 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8527000000000001e-05 2024-08-19T14:30:19.749 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.749 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.749 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:30:19.749 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.613008+0000", 2024-08-19T14:30:19.749 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.079e-05 2024-08-19T14:30:19.749 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.750 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.750 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:30:19.750 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.613014+0000", 2024-08-19T14:30:19.750 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4210000000000003e-06 2024-08-19T14:30:19.750 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.750 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.751 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:30:19.751 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.613062+0000", 2024-08-19T14:30:19.751 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.8674999999999997e-05 2024-08-19T14:30:19.751 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.751 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.752 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:30:19.752 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.613200+0000", 2024-08-19T14:30:19.752 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013715400000000001 2024-08-19T14:30:19.752 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.752 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.752 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,7", 2024-08-19T14:30:19.752 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.613317+0000", 2024-08-19T14:30:19.752 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00011734399999999999 2024-08-19T14:30:19.752 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.752 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.752 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:30:19.753 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.616326+0000", 2024-08-19T14:30:19.753 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0030090450000000001 2024-08-19T14:30:19.753 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.753 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.753 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:30:19.753 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.621117+0000", 2024-08-19T14:30:19.753 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0047912160000000001 2024-08-19T14:30:19.753 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.753 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.753 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:30:19.753 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.848233+0000", 2024-08-19T14:30:19.753 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.22711604099999999 2024-08-19T14:30:19.753 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.753 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.753 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:30:19.754 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.848288+0000", 2024-08-19T14:30:19.754 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4972999999999997e-05 2024-08-19T14:30:19.754 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.754 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.754 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:30:19.754 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.848340+0000", 2024-08-19T14:30:19.754 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2237000000000001e-05 2024-08-19T14:30:19.754 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:30:19.754 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:30:19.754 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:30:19.754 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.754 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.755 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:116311 4.8 4:16b96ac4:::benchmark_data_smithi045_71562_object116310:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-08-19T14:30:19.755 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:21:48.627253+0000", 2024-08-19T14:30:19.755 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 511.05546818900001, 2024-08-19T14:30:19.755 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.26561561299999997, 2024-08-19T14:30:19.755 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:30:19.755 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:30:19.755 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:30:19.755 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:30:19.755 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:30:19.755 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:30:19.755 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 116311 2024-08-19T14:30:19.755 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.755 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:30:19.755 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.756 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:30:19.756 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.627253+0000", 2024-08-19T14:30:19.756 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:30:19.756 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.756 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.756 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:30:19.756 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.627253+0000", 2024-08-19T14:30:19.756 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:30:19.756 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.756 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.756 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:30:19.757 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.627254+0000", 2024-08-19T14:30:19.757 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4000000000000001e-07 2024-08-19T14:30:19.757 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.757 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.757 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:30:19.757 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.627293+0000", 2024-08-19T14:30:19.757 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.8742000000000003e-05 2024-08-19T14:30:19.757 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.757 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.757 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:30:19.757 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.627302+0000", 2024-08-19T14:30:19.757 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.2639999999999997e-06 2024-08-19T14:30:19.758 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.758 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.758 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:30:19.758 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.627307+0000", 2024-08-19T14:30:19.758 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0209999999999999e-06 2024-08-19T14:30:19.758 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.758 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.758 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:30:19.758 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.627378+0000", 2024-08-19T14:30:19.758 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.0529000000000002e-05 2024-08-19T14:30:19.758 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.759 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.759 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:30:19.759 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.627458+0000", 2024-08-19T14:30:19.759 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.0259999999999994e-05 2024-08-19T14:30:19.759 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.759 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.759 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 7,9", 2024-08-19T14:30:19.759 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.627624+0000", 2024-08-19T14:30:19.759 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00016623300000000001 2024-08-19T14:30:19.759 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.759 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.759 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:30:19.759 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.631383+0000", 2024-08-19T14:30:19.759 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0037587599999999999 2024-08-19T14:30:19.760 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.760 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.760 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:30:19.760 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.631513+0000", 2024-08-19T14:30:19.760 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013022199999999999 2024-08-19T14:30:19.760 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.760 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.760 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:30:19.760 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.892784+0000", 2024-08-19T14:30:19.760 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.26127100399999997 2024-08-19T14:30:19.760 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.760 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.761 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:30:19.761 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.892832+0000", 2024-08-19T14:30:19.761 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.7970999999999999e-05 2024-08-19T14:30:19.761 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.761 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.761 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:30:19.761 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.892869+0000", 2024-08-19T14:30:19.761 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.6866999999999998e-05 2024-08-19T14:30:19.761 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:30:19.761 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:30:19.761 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:30:19.761 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.761 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.761 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:116334 4.8 4:1f777a70:::benchmark_data_smithi045_71562_object116333:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-08-19T14:30:19.761 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:21:48.650934+0000", 2024-08-19T14:30:19.762 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 511.03178748200003, 2024-08-19T14:30:19.762 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.242080929, 2024-08-19T14:30:19.762 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:30:19.762 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:30:19.762 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:30:19.762 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:30:19.762 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:30:19.762 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:30:19.762 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 116334 2024-08-19T14:30:19.762 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.762 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:30:19.762 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.763 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:30:19.763 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.650934+0000", 2024-08-19T14:30:19.763 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:30:19.763 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.763 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.763 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:30:19.763 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.650934+0000", 2024-08-19T14:30:19.763 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:30:19.763 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.763 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.763 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:30:19.763 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.650935+0000", 2024-08-19T14:30:19.763 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.26e-06 2024-08-19T14:30:19.763 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.764 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.764 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:30:19.764 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.650954+0000", 2024-08-19T14:30:19.764 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9000000000000001e-05 2024-08-19T14:30:19.764 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.764 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.764 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:30:19.764 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.650964+0000", 2024-08-19T14:30:19.764 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0253e-05 2024-08-19T14:30:19.764 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.764 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.764 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:30:19.764 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.650970+0000", 2024-08-19T14:30:19.764 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1499999999999998e-06 2024-08-19T14:30:19.765 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.765 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.765 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:30:19.765 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.651014+0000", 2024-08-19T14:30:19.765 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.4391000000000002e-05 2024-08-19T14:30:19.765 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.765 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.765 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:30:19.765 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.651134+0000", 2024-08-19T14:30:19.765 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000119535 2024-08-19T14:30:19.765 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.765 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.765 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 7,9", 2024-08-19T14:30:19.765 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.651262+0000", 2024-08-19T14:30:19.766 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012826100000000001 2024-08-19T14:30:19.766 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.766 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.766 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:30:19.766 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.654276+0000", 2024-08-19T14:30:19.766 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0030142839999999999 2024-08-19T14:30:19.766 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.766 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.766 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:30:19.767 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.655021+0000", 2024-08-19T14:30:19.767 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00074536299999999997 2024-08-19T14:30:19.767 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.767 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.767 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:30:19.767 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.892914+0000", 2024-08-19T14:30:19.767 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.23789228400000001 2024-08-19T14:30:19.767 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.767 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.767 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:30:19.767 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.892980+0000", 2024-08-19T14:30:19.767 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.6193999999999997e-05 2024-08-19T14:30:19.767 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.768 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.768 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:30:19.768 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:21:48.893015+0000", 2024-08-19T14:30:19.768 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.4953999999999998e-05 2024-08-19T14:30:19.768 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:30:19.768 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:30:19.768 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:30:19.768 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.768 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.768 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:165636 4.3 4:c997d720:::benchmark_data_smithi045_71562_object34461:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-08-19T14:30:19.768 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:22:42.884498+0000", 2024-08-19T14:30:19.768 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 456.79822355599998, 2024-08-19T14:30:19.768 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.22208514900000001, 2024-08-19T14:30:19.769 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:30:19.769 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:30:19.769 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:30:19.769 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:30:19.769 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:30:19.769 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:30:19.769 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 165636 2024-08-19T14:30:19.769 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.769 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:30:19.769 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.769 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:30:19.769 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.884498+0000", 2024-08-19T14:30:19.769 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:30:19.769 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.770 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.770 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:30:19.770 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.884498+0000", 2024-08-19T14:30:19.770 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:30:19.770 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.770 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.770 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:30:19.770 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.884499+0000", 2024-08-19T14:30:19.770 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.238e-06 2024-08-19T14:30:19.770 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.770 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.770 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:30:19.770 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.884504+0000", 2024-08-19T14:30:19.771 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2959999999999998e-06 2024-08-19T14:30:19.771 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.771 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.771 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:30:19.771 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.884515+0000", 2024-08-19T14:30:19.771 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0995999999999999e-05 2024-08-19T14:30:19.771 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.771 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.771 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:30:19.771 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.884523+0000", 2024-08-19T14:30:19.771 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.294e-06 2024-08-19T14:30:19.771 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.771 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.771 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:30:19.772 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.884580+0000", 2024-08-19T14:30:19.772 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7224e-05 2024-08-19T14:30:19.772 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.772 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.772 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:30:19.772 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.884704+0000", 2024-08-19T14:30:19.772 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000124046 2024-08-19T14:30:19.772 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.772 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.772 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,7", 2024-08-19T14:30:19.772 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.884836+0000", 2024-08-19T14:30:19.772 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000132478 2024-08-19T14:30:19.772 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.773 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.773 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:30:19.773 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.888660+0000", 2024-08-19T14:30:19.773 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0038240399999999999 2024-08-19T14:30:19.773 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.773 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.773 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:30:19.773 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.889616+0000", 2024-08-19T14:30:19.773 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00095516899999999996 2024-08-19T14:30:19.773 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.773 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.773 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:30:19.773 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:43.106482+0000", 2024-08-19T14:30:19.773 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.21686655299999999 2024-08-19T14:30:19.774 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.774 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.774 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:30:19.774 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:43.106539+0000", 2024-08-19T14:30:19.774 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.6366999999999998e-05 2024-08-19T14:30:19.774 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.774 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.774 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:30:19.774 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:43.106583+0000", 2024-08-19T14:30:19.774 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.4447999999999999e-05 2024-08-19T14:30:19.774 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:30:19.774 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:30:19.774 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:30:19.775 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.775 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.775 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25333.0:165645 4.3 4:c4ef6944:::benchmark_data_smithi045_71562_object34470:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-08-19T14:30:19.775 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:22:42.892140+0000", 2024-08-19T14:30:19.775 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 456.79058098100001, 2024-08-19T14:30:19.775 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.223849192, 2024-08-19T14:30:19.775 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:30:19.775 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:30:19.775 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:30:19.775 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:30:19.775 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25333", 2024-08-19T14:30:19.775 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2872021341", 2024-08-19T14:30:19.775 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 165645 2024-08-19T14:30:19.776 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.776 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:30:19.776 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.776 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:30:19.776 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.892140+0000", 2024-08-19T14:30:19.776 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:30:19.776 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.776 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.776 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:30:19.776 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.892140+0000", 2024-08-19T14:30:19.776 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:30:19.776 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.776 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.776 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:30:19.777 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.892142+0000", 2024-08-19T14:30:19.777 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.127e-06 2024-08-19T14:30:19.777 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.777 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.777 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:30:19.777 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.892146+0000", 2024-08-19T14:30:19.777 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.7949999999999998e-06 2024-08-19T14:30:19.777 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.777 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.777 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:30:19.778 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.892157+0000", 2024-08-19T14:30:19.778 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0387000000000001e-05 2024-08-19T14:30:19.778 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.778 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.778 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:30:19.778 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.892162+0000", 2024-08-19T14:30:19.778 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.6269999999999997e-06 2024-08-19T14:30:19.778 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.778 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.778 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:30:19.778 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.892203+0000", 2024-08-19T14:30:19.778 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.0238999999999999e-05 2024-08-19T14:30:19.778 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.779 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.779 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:30:19.779 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.892284+0000", 2024-08-19T14:30:19.779 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.1198000000000004e-05 2024-08-19T14:30:19.779 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.779 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.779 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,7", 2024-08-19T14:30:19.779 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.892359+0000", 2024-08-19T14:30:19.779 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4783000000000006e-05 2024-08-19T14:30:19.779 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.779 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.779 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:30:19.779 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.895461+0000", 2024-08-19T14:30:19.779 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0031028869999999999 2024-08-19T14:30:19.780 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.780 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.780 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:30:19.780 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:42.897428+0000", 2024-08-19T14:30:19.780 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0019669069999999999 2024-08-19T14:30:19.780 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.780 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.780 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:30:19.780 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:43.115899+0000", 2024-08-19T14:30:19.780 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.21847051100000001 2024-08-19T14:30:19.780 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.780 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.780 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:30:19.781 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:43.115943+0000", 2024-08-19T14:30:19.781 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.4181000000000003e-05 2024-08-19T14:30:19.781 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.781 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.781 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:30:19.781 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:22:43.115990+0000", 2024-08-19T14:30:19.781 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.655e-05 2024-08-19T14:30:19.781 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:30:19.781 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:30:19.781 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:30:19.781 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.781 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.781 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25813.0:115483 5.8 5:118541ec:::benchmark_data_smithi045_85309_object115482:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e455)", 2024-08-19T14:30:19.781 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:26:12.254294+0000", 2024-08-19T14:30:19.782 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 247.428427715, 2024-08-19T14:30:19.782 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.41025914400000002, 2024-08-19T14:30:19.782 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:30:19.782 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:30:19.782 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:30:19.782 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:30:19.782 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25813", 2024-08-19T14:30:19.782 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:30:19.782 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 115483 2024-08-19T14:30:19.782 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.782 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:30:19.782 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.782 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:30:19.783 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.254294+0000", 2024-08-19T14:30:19.783 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:30:19.783 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.783 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.783 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:30:19.783 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.254294+0000", 2024-08-19T14:30:19.783 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:30:19.783 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.783 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.783 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:30:19.783 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.254295+0000", 2024-08-19T14:30:19.783 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0049999999999999e-06 2024-08-19T14:30:19.783 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.784 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.784 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:30:19.784 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.254351+0000", 2024-08-19T14:30:19.784 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.6385999999999999e-05 2024-08-19T14:30:19.784 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.784 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.784 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:30:19.784 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.254365+0000", 2024-08-19T14:30:19.784 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4425e-05 2024-08-19T14:30:19.784 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.784 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.784 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:30:19.784 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.254373+0000", 2024-08-19T14:30:19.784 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.1620000000000002e-06 2024-08-19T14:30:19.785 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.785 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.785 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:30:19.785 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.254438+0000", 2024-08-19T14:30:19.785 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.5295000000000004e-05 2024-08-19T14:30:19.785 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.785 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.785 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:30:19.785 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.254556+0000", 2024-08-19T14:30:19.785 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000117555 2024-08-19T14:30:19.785 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.785 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.786 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,9", 2024-08-19T14:30:19.786 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.254698+0000", 2024-08-19T14:30:19.786 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000142003 2024-08-19T14:30:19.786 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.786 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.786 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:30:19.786 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.256435+0000", 2024-08-19T14:30:19.786 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0017378999999999999 2024-08-19T14:30:19.786 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.786 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.786 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:30:19.786 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.265174+0000", 2024-08-19T14:30:19.787 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0087388980000000002 2024-08-19T14:30:19.787 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.787 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.787 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:30:19.787 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.664449+0000", 2024-08-19T14:30:19.787 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.39927473800000002 2024-08-19T14:30:19.787 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.788 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.788 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:30:19.788 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.664502+0000", 2024-08-19T14:30:19.788 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.3066000000000003e-05 2024-08-19T14:30:19.788 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.788 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.789 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:30:19.789 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.664553+0000", 2024-08-19T14:30:19.789 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0711000000000003e-05 2024-08-19T14:30:19.789 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:30:19.789 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:30:19.789 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:30:19.790 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.790 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.790 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25813.0:115542 5.8 5:126f9ca1:::benchmark_data_smithi045_85309_object115541:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e455)", 2024-08-19T14:30:19.790 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:26:12.310153+0000", 2024-08-19T14:30:19.790 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 247.372568067, 2024-08-19T14:30:19.790 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.35453124699999999, 2024-08-19T14:30:19.790 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:30:19.791 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:30:19.791 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:30:19.791 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:30:19.791 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25813", 2024-08-19T14:30:19.791 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:30:19.791 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 115542 2024-08-19T14:30:19.791 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.792 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:30:19.792 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.792 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:30:19.792 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.310153+0000", 2024-08-19T14:30:19.792 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:30:19.792 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.792 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.792 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:30:19.793 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.310153+0000", 2024-08-19T14:30:19.793 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:30:19.793 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.793 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.793 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:30:19.793 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.310155+0000", 2024-08-19T14:30:19.794 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.279e-06 2024-08-19T14:30:19.794 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.794 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.794 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:30:19.794 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.310186+0000", 2024-08-19T14:30:19.794 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.1291999999999998e-05 2024-08-19T14:30:19.795 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.795 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.795 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:30:19.795 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.310199+0000", 2024-08-19T14:30:19.795 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2824e-05 2024-08-19T14:30:19.795 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.796 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.796 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:30:19.796 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.310206+0000", 2024-08-19T14:30:19.796 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.3189999999999997e-06 2024-08-19T14:30:19.796 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.796 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.796 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:30:19.797 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.311624+0000", 2024-08-19T14:30:19.797 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0014176740000000001 2024-08-19T14:30:19.797 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.797 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.797 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:30:19.797 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.311764+0000", 2024-08-19T14:30:19.797 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014065900000000001 2024-08-19T14:30:19.797 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.797 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.797 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,9", 2024-08-19T14:30:19.797 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.311923+0000", 2024-08-19T14:30:19.797 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015905800000000001 2024-08-19T14:30:19.798 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.798 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.798 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:30:19.798 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.313965+0000", 2024-08-19T14:30:19.798 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0020413430000000002 2024-08-19T14:30:19.798 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.798 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.798 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:30:19.798 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.320836+0000", 2024-08-19T14:30:19.798 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0068708529999999997 2024-08-19T14:30:19.798 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.798 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.799 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:30:19.799 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.664598+0000", 2024-08-19T14:30:19.799 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.34376222299999998 2024-08-19T14:30:19.799 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.799 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.799 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:30:19.799 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.664648+0000", 2024-08-19T14:30:19.799 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0489999999999999e-05 2024-08-19T14:30:19.799 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.799 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.799 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:30:19.799 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.664685+0000", 2024-08-19T14:30:19.799 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.6232999999999997e-05 2024-08-19T14:30:19.800 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:30:19.800 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:30:19.800 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:30:19.800 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.800 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.800 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25813.0:115641 5.8 5:1358c592:::benchmark_data_smithi045_85309_object115640:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e455)", 2024-08-19T14:30:19.800 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:26:12.409980+0000", 2024-08-19T14:30:19.800 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 247.27274103100001, 2024-08-19T14:30:19.800 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.25595528499999998, 2024-08-19T14:30:19.800 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:30:19.800 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:30:19.800 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:30:19.800 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:30:19.800 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25813", 2024-08-19T14:30:19.801 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:30:19.801 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 115641 2024-08-19T14:30:19.801 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.801 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:30:19.801 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.801 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:30:19.801 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.409980+0000", 2024-08-19T14:30:19.801 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:30:19.801 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.801 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.801 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:30:19.801 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.409980+0000", 2024-08-19T14:30:19.801 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:30:19.802 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.802 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.802 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:30:19.802 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.409982+0000", 2024-08-19T14:30:19.802 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2950000000000001e-06 2024-08-19T14:30:19.802 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.802 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.802 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:30:19.802 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.410007+0000", 2024-08-19T14:30:19.802 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.5198000000000002e-05 2024-08-19T14:30:19.802 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.802 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.802 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:30:19.803 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.410022+0000", 2024-08-19T14:30:19.803 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4678000000000001e-05 2024-08-19T14:30:19.803 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.803 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.803 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:30:19.803 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.410029+0000", 2024-08-19T14:30:19.803 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4590000000000001e-06 2024-08-19T14:30:19.803 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.803 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.803 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:30:19.803 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.410088+0000", 2024-08-19T14:30:19.803 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.9169999999999998e-05 2024-08-19T14:30:19.803 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.804 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.804 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:30:19.804 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.410232+0000", 2024-08-19T14:30:19.804 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014414699999999999 2024-08-19T14:30:19.804 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.804 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.804 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,9", 2024-08-19T14:30:19.804 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.410377+0000", 2024-08-19T14:30:19.804 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000144566 2024-08-19T14:30:19.804 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.804 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.804 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:30:19.804 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.412287+0000", 2024-08-19T14:30:19.805 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001909628 2024-08-19T14:30:19.805 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.805 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.805 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:30:19.805 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.449686+0000", 2024-08-19T14:30:19.805 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.037399549999999997 2024-08-19T14:30:19.805 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.805 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.805 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:30:19.805 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.665809+0000", 2024-08-19T14:30:19.805 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.21612313899999999 2024-08-19T14:30:19.806 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.806 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.806 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:30:19.806 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.665895+0000", 2024-08-19T14:30:19.806 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.5433999999999996e-05 2024-08-19T14:30:19.806 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.806 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.806 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:30:19.806 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.665936+0000", 2024-08-19T14:30:19.806 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.1020999999999999e-05 2024-08-19T14:30:19.806 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:30:19.807 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:30:19.807 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:30:19.807 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.807 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.807 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25813.0:115648 5.8 5:15d8ce6a:::benchmark_data_smithi045_85309_object115647:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e455)", 2024-08-19T14:30:19.807 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:26:12.416281+0000", 2024-08-19T14:30:19.807 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 247.266440719, 2024-08-19T14:30:19.807 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.24979046499999999, 2024-08-19T14:30:19.807 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:30:19.807 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:30:19.807 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:30:19.808 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:30:19.808 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25813", 2024-08-19T14:30:19.808 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:30:19.808 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 115648 2024-08-19T14:30:19.808 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.808 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:30:19.808 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.808 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:30:19.808 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.416281+0000", 2024-08-19T14:30:19.808 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:30:19.808 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.808 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.808 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:30:19.808 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.416281+0000", 2024-08-19T14:30:19.809 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:30:19.809 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.809 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.809 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:30:19.809 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.416282+0000", 2024-08-19T14:30:19.809 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5740000000000001e-06 2024-08-19T14:30:19.809 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.809 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.809 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:30:19.809 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.416304+0000", 2024-08-19T14:30:19.809 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1698000000000001e-05 2024-08-19T14:30:19.809 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.809 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.810 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:30:19.810 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.416317+0000", 2024-08-19T14:30:19.810 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3084e-05 2024-08-19T14:30:19.810 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.810 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.810 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:30:19.810 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.416324+0000", 2024-08-19T14:30:19.810 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.0570000000000004e-06 2024-08-19T14:30:19.810 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.810 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.810 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:30:19.810 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.416383+0000", 2024-08-19T14:30:19.810 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.9261000000000001e-05 2024-08-19T14:30:19.811 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.811 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.811 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:30:19.811 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.416500+0000", 2024-08-19T14:30:19.811 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000116523 2024-08-19T14:30:19.811 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.811 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.811 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,9", 2024-08-19T14:30:19.811 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.416646+0000", 2024-08-19T14:30:19.811 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014645500000000001 2024-08-19T14:30:19.811 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.811 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.811 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:30:19.811 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.418689+0000", 2024-08-19T14:30:19.812 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0020425019999999999 2024-08-19T14:30:19.812 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.812 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.812 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:30:19.812 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.449715+0000", 2024-08-19T14:30:19.812 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.031026194 2024-08-19T14:30:19.812 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.812 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.812 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:30:19.812 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.665979+0000", 2024-08-19T14:30:19.812 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.21626399099999999 2024-08-19T14:30:19.812 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.812 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.812 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:30:19.813 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.666032+0000", 2024-08-19T14:30:19.813 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.3140000000000003e-05 2024-08-19T14:30:19.813 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.813 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.813 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:30:19.813 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.666071+0000", 2024-08-19T14:30:19.813 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.8986000000000002e-05 2024-08-19T14:30:19.813 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:30:19.813 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:30:19.813 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:30:19.813 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.813 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.813 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25813.0:180218 5.d 5:bd4207f5:::benchmark_data_smithi045_85309_object59965:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e475)", 2024-08-19T14:30:19.814 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:27:11.940920+0000", 2024-08-19T14:30:19.814 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 187.74180149700001, 2024-08-19T14:30:19.814 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.23667333700000001, 2024-08-19T14:30:19.814 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:30:19.814 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:30:19.814 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:30:19.814 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:30:19.814 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25813", 2024-08-19T14:30:19.815 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:30:19.815 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 180218 2024-08-19T14:30:19.815 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.815 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:30:19.815 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.815 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:30:19.815 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:11.940920+0000", 2024-08-19T14:30:19.815 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:30:19.815 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.815 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.815 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:30:19.815 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:11.940920+0000", 2024-08-19T14:30:19.815 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:30:19.816 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.816 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.816 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:30:19.816 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:11.940921+0000", 2024-08-19T14:30:19.816 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.561e-06 2024-08-19T14:30:19.816 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.816 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.816 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:30:19.816 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:11.940927+0000", 2024-08-19T14:30:19.816 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7300000000000002e-06 2024-08-19T14:30:19.816 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.816 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.816 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:30:19.816 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:11.940938+0000", 2024-08-19T14:30:19.817 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0958000000000001e-05 2024-08-19T14:30:19.817 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.817 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.817 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:30:19.817 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:11.940945+0000", 2024-08-19T14:30:19.817 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.6579999999999996e-06 2024-08-19T14:30:19.817 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.817 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.817 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:30:19.817 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.031816+0000", 2024-08-19T14:30:19.817 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.090871065000000001 2024-08-19T14:30:19.817 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.817 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.817 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:30:19.818 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.031914+0000", 2024-08-19T14:30:19.818 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.7686e-05 2024-08-19T14:30:19.818 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.818 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.818 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 5,7", 2024-08-19T14:30:19.818 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.032000+0000", 2024-08-19T14:30:19.818 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.6169999999999997e-05 2024-08-19T14:30:19.818 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.818 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.818 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:30:19.818 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.045113+0000", 2024-08-19T14:30:19.818 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.013113028000000001 2024-08-19T14:30:19.818 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.819 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.819 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:30:19.819 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.047147+0000", 2024-08-19T14:30:19.819 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0020338050000000001 2024-08-19T14:30:19.819 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.819 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.819 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:30:19.819 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.177520+0000", 2024-08-19T14:30:19.819 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.130373085 2024-08-19T14:30:19.819 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.819 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.819 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:30:19.819 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.177556+0000", 2024-08-19T14:30:19.819 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.6403999999999999e-05 2024-08-19T14:30:19.819 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.820 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.820 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:30:19.820 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.177593+0000", 2024-08-19T14:30:19.820 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.7187000000000002e-05 2024-08-19T14:30:19.820 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:30:19.820 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:30:19.820 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:30:19.820 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.820 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.820 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25813.0:180682 5.8 5:1602eddf:::benchmark_data_smithi045_85309_object60429:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e476)", 2024-08-19T14:30:19.820 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:27:12.846408+0000", 2024-08-19T14:30:19.820 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 186.83631325100001, 2024-08-19T14:30:19.820 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.27902613199999998, 2024-08-19T14:30:19.821 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:30:19.821 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:30:19.821 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:30:19.821 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:30:19.821 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25813", 2024-08-19T14:30:19.821 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:30:19.821 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 180682 2024-08-19T14:30:19.821 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.821 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:30:19.821 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.821 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:30:19.821 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.846408+0000", 2024-08-19T14:30:19.821 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:30:19.822 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.822 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.822 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:30:19.822 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.846408+0000", 2024-08-19T14:30:19.822 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:30:19.822 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.822 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.822 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:30:19.822 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.846410+0000", 2024-08-19T14:30:19.822 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9980000000000002e-06 2024-08-19T14:30:19.822 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.822 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.823 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:30:19.823 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.846415+0000", 2024-08-19T14:30:19.823 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1120000000000004e-06 2024-08-19T14:30:19.823 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.823 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.823 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:30:19.823 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.846431+0000", 2024-08-19T14:30:19.823 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5786000000000001e-05 2024-08-19T14:30:19.823 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.823 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.823 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:30:19.823 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.846438+0000", 2024-08-19T14:30:19.823 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.3699999999999997e-06 2024-08-19T14:30:19.823 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.824 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.824 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:30:19.824 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.915048+0000", 2024-08-19T14:30:19.824 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.068609724999999996 2024-08-19T14:30:19.824 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.824 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.824 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:30:19.824 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.915174+0000", 2024-08-19T14:30:19.824 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012629899999999999 2024-08-19T14:30:19.824 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.824 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.824 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,9", 2024-08-19T14:30:19.825 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.915288+0000", 2024-08-19T14:30:19.825 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000113544 2024-08-19T14:30:19.825 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.825 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.825 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:30:19.825 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.921139+0000", 2024-08-19T14:30:19.825 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0058512420000000004 2024-08-19T14:30:19.825 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.825 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.825 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:30:19.825 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.923745+0000", 2024-08-19T14:30:19.825 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0026053589999999998 2024-08-19T14:30:19.826 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.826 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.826 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:30:19.826 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:13.125373+0000", 2024-08-19T14:30:19.826 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.201628589 2024-08-19T14:30:19.826 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.826 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.826 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:30:19.826 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:13.125405+0000", 2024-08-19T14:30:19.826 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.2224999999999997e-05 2024-08-19T14:30:19.826 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.826 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.827 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:30:19.827 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:13.125434+0000", 2024-08-19T14:30:19.827 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.8883000000000001e-05 2024-08-19T14:30:19.827 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:30:19.827 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:30:19.827 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:30:19.827 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.827 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.827 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25813.0:180713 5.8 5:13e650ea:::benchmark_data_smithi045_85309_object60460:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e476)", 2024-08-19T14:30:19.827 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:27:12.880045+0000", 2024-08-19T14:30:19.827 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 186.802676319, 2024-08-19T14:30:19.827 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.24953703699999999, 2024-08-19T14:30:19.827 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:30:19.827 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:30:19.828 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:30:19.828 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:30:19.828 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25813", 2024-08-19T14:30:19.828 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:30:19.828 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 180713 2024-08-19T14:30:19.828 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.828 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:30:19.828 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.828 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:30:19.829 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.880045+0000", 2024-08-19T14:30:19.829 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:30:19.829 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.829 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.829 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:30:19.829 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.880045+0000", 2024-08-19T14:30:19.829 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:30:19.829 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.829 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.830 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:30:19.831 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:19 smithi032 ceph-mon[20004]: pgmap v1257: 47 pgs: 8 active+recovering+undersized+remapped, 39 active+clean; 6.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 150 B/s rd, 687 KiB/s wr, 362 op/s; 24263/314574 objects misplaced (7.713%); 15 MiB/s, 235 objects/s recovering 2024-08-19T14:30:19.831 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:19 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:30:19.831 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:19 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:30:19.831 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:19 smithi032 ceph-mon[20004]: Health check failed: Degraded data redundancy: 8 pgs undersized (PG_DEGRADED) 2024-08-19T14:30:19.831 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:19 smithi032 ceph-mon[20004]: osdmap e546: 12 total, 12 up, 8 in 2024-08-19T14:30:19.831 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:19 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3051016702' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:30:19.831 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:30:19.832 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:30:19.832 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.880047+0000", 2024-08-19T14:30:19.832 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5510000000000001e-06 2024-08-19T14:30:19.832 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.832 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.832 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:30:19.832 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.880052+0000", 2024-08-19T14:30:19.832 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.5709999999999997e-06 2024-08-19T14:30:19.832 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.832 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.832 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:30:19.832 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.880068+0000", 2024-08-19T14:30:19.832 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6087e-05 2024-08-19T14:30:19.833 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.833 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.833 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:30:19.833 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.880076+0000", 2024-08-19T14:30:19.833 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4039999999999998e-06 2024-08-19T14:30:19.833 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.833 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.833 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:30:19.833 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.918429+0000", 2024-08-19T14:30:19.833 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.038353230000000002 2024-08-19T14:30:19.833 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.833 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.833 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:30:19.833 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.918538+0000", 2024-08-19T14:30:19.834 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00010878999999999999 2024-08-19T14:30:19.834 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.834 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.834 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,9", 2024-08-19T14:30:19.834 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.918613+0000", 2024-08-19T14:30:19.834 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.5405000000000001e-05 2024-08-19T14:30:19.834 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.834 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.834 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:30:19.834 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.923754+0000", 2024-08-19T14:30:19.834 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0051403999999999998 2024-08-19T14:30:19.834 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.835 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.835 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:30:19.835 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.924797+0000", 2024-08-19T14:30:19.835 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0010431150000000001 2024-08-19T14:30:19.835 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.835 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.835 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:30:19.835 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:13.129461+0000", 2024-08-19T14:30:19.835 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20466435799999999 2024-08-19T14:30:19.835 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.835 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.835 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:30:19.835 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:13.129522+0000", 2024-08-19T14:30:19.836 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.1054000000000002e-05 2024-08-19T14:30:19.836 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.836 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.836 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:30:19.836 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:13.129582+0000", 2024-08-19T14:30:19.836 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.0071999999999997e-05 2024-08-19T14:30:19.836 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:30:19.836 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:30:19.836 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:30:19.836 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.836 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.836 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25813.0:180719 5.8 5:15c77316:::benchmark_data_smithi045_85309_object60466:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e476)", 2024-08-19T14:30:19.836 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:27:12.886662+0000", 2024-08-19T14:30:19.837 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 186.79605981500001, 2024-08-19T14:30:19.837 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.24306710300000001, 2024-08-19T14:30:19.837 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:30:19.837 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:30:19.837 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:30:19.837 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:30:19.837 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25813", 2024-08-19T14:30:19.837 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:30:19.837 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 180719 2024-08-19T14:30:19.837 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.837 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:30:19.837 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.838 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:30:19.838 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.886662+0000", 2024-08-19T14:30:19.838 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:30:19.838 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.838 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.838 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:30:19.838 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.886662+0000", 2024-08-19T14:30:19.838 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:30:19.838 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.838 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.838 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:30:19.838 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.886663+0000", 2024-08-19T14:30:19.839 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.184e-06 2024-08-19T14:30:19.839 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.839 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.839 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:30:19.839 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.886667+0000", 2024-08-19T14:30:19.839 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.907e-06 2024-08-19T14:30:19.839 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.839 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.839 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:30:19.839 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.886679+0000", 2024-08-19T14:30:19.840 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1915e-05 2024-08-19T14:30:19.840 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.840 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.840 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:30:19.840 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.886686+0000", 2024-08-19T14:30:19.840 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.2049999999999996e-06 2024-08-19T14:30:19.840 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.840 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.840 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:30:19.840 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.921183+0000", 2024-08-19T14:30:19.840 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.034497556999999998 2024-08-19T14:30:19.840 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.840 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.840 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:30:19.841 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.921260+0000", 2024-08-19T14:30:19.841 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.6388000000000001e-05 2024-08-19T14:30:19.841 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.841 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.841 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,9", 2024-08-19T14:30:19.841 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.921322+0000", 2024-08-19T14:30:19.841 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.2611000000000001e-05 2024-08-19T14:30:19.841 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.841 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.841 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:30:19.841 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.923834+0000", 2024-08-19T14:30:19.841 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.002511924 2024-08-19T14:30:19.841 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.842 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.842 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:30:19.842 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.924841+0000", 2024-08-19T14:30:19.842 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0010065129999999999 2024-08-19T14:30:19.842 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.842 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.842 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:30:19.842 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:13.129629+0000", 2024-08-19T14:30:19.842 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20478792500000001 2024-08-19T14:30:19.842 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.842 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.842 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:30:19.842 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:13.129683+0000", 2024-08-19T14:30:19.843 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4293999999999999e-05 2024-08-19T14:30:19.843 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.843 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.843 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:30:19.843 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:13.129729+0000", 2024-08-19T14:30:19.843 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.5680000000000003e-05 2024-08-19T14:30:19.843 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:30:19.843 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:30:19.843 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:30:19.843 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.843 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.844 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25813.0:237770 5.8 5:11831c02:::benchmark_data_smithi045_85309_object117517:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e495)", 2024-08-19T14:30:19.844 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:28:07.106685+0000", 2024-08-19T14:30:19.844 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 132.576036727, 2024-08-19T14:30:19.844 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.22493542599999999, 2024-08-19T14:30:19.844 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:30:19.844 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:30:19.844 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:30:19.844 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:30:19.844 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25813", 2024-08-19T14:30:19.844 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:30:19.844 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 237770 2024-08-19T14:30:19.845 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.845 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:30:19.845 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.845 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:30:19.845 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:28:07.106685+0000", 2024-08-19T14:30:19.845 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:30:19.845 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.845 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.845 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:30:19.845 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:28:07.106685+0000", 2024-08-19T14:30:19.845 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:30:19.846 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.846 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.846 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:30:19.846 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:28:07.106685+0000", 2024-08-19T14:30:19.846 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.6199999999999998e-07 2024-08-19T14:30:19.846 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.846 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.846 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:30:19.846 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:28:07.106688+0000", 2024-08-19T14:30:19.846 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.0510000000000001e-06 2024-08-19T14:30:19.846 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.846 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.846 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:30:19.847 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:28:07.106696+0000", 2024-08-19T14:30:19.847 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.9899999999999997e-06 2024-08-19T14:30:19.847 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.847 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.847 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:30:19.847 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:28:07.106702+0000", 2024-08-19T14:30:19.847 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7810000000000002e-06 2024-08-19T14:30:19.847 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.847 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.847 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:30:19.847 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:28:07.106846+0000", 2024-08-19T14:30:19.847 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014427600000000001 2024-08-19T14:30:19.847 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.848 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.848 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for peered", 2024-08-19T14:30:19.848 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:28:07.106854+0000", 2024-08-19T14:30:19.848 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.3309999999999998e-06 2024-08-19T14:30:19.848 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.848 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.848 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:30:19.848 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:28:07.317267+0000", 2024-08-19T14:30:19.848 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.21041380400000001 2024-08-19T14:30:19.848 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.848 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.849 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:30:19.849 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:28:07.317422+0000", 2024-08-19T14:30:19.849 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015415200000000001 2024-08-19T14:30:19.849 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.849 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.849 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,5", 2024-08-19T14:30:19.849 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:28:07.318149+0000", 2024-08-19T14:30:19.850 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00072762499999999997 2024-08-19T14:30:19.850 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.850 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.850 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:30:19.850 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:28:07.322380+0000", 2024-08-19T14:30:19.850 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0042309590000000003 2024-08-19T14:30:19.850 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.850 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.850 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:30:19.850 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:28:07.323678+0000", 2024-08-19T14:30:19.851 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001298366 2024-08-19T14:30:19.851 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.851 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.851 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:30:19.851 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:28:07.331373+0000", 2024-08-19T14:30:19.851 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0076946790000000003 2024-08-19T14:30:19.851 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.851 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.851 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:30:19.851 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:28:07.331429+0000", 2024-08-19T14:30:19.851 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.5517999999999999e-05 2024-08-19T14:30:19.851 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:30:19.851 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:30:19.851 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:30:19.852 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:28:07.331620+0000", 2024-08-19T14:30:19.852 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00019143200000000001 2024-08-19T14:30:19.852 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:30:19.852 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:30:19.852 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:30:19.852 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:30:19.852 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:30:19.852 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:30:19.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:19.854 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:19.854 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:30:19.854 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:30:19.855 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:30:19.884 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:30:19.884 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:30:19.917 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:30:19.917 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:30:19.955 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:30:19.956 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:30:19.956 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:30:19.988 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:30:19.989 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:30:19.997 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:19.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:19.998 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:30:20.032 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:30:20.032 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:30:20.066 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:30:20.066 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:30:20.099 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:30:20.104 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:30:20.104 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:30:20.138 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:30:20.138 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:30:20.140 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:20.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:20.141 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:30:20.170 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:30:20.171 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:30:20.206 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:30:20.206 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:30:20.243 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:30:20.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:20.265 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:20.265 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:30:20.366 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:30:20.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:20 smithi031 ceph-mon[16336]: from='client.26381 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:30:20.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:20 smithi031 ceph-mon[16336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:30:20.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:20 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3051016702' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:30:20.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:20 smithi031 ceph-mon[16336]: osdmap e547: 12 total, 12 up, 8 in 2024-08-19T14:30:20.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:20 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2901460495' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:30:20.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:20.394 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:20.395 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:30:20.496 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:30:20.520 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:20.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:20.521 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:30:20.622 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:30:20.643 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:20.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:20.644 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:30:20.745 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:30:20.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:20 smithi032 ceph-mon[20004]: from='client.26381 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:30:20.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:20 smithi032 ceph-mon[20004]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:30:20.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:20 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3051016702' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:30:20.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:20 smithi032 ceph-mon[20004]: osdmap e547: 12 total, 12 up, 8 in 2024-08-19T14:30:20.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:20 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2901460495' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:30:20.766 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:20.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:20.767 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:30:20.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:20 smithi045 ceph-mon[20143]: from='client.26381 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:30:20.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:20 smithi045 ceph-mon[20143]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:30:20.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:20 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3051016702' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:30:20.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:20 smithi045 ceph-mon[20143]: osdmap e547: 12 total, 12 up, 8 in 2024-08-19T14:30:20.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:20 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2901460495' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:30:20.868 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:30:20.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:20.892 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:20.892 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:30:20.960 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":1258,"stamp":"2024-08-19T14:30:18.310083+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6834946616,"num_objects":104287,"num_object_clones":0,"num_object_copies":312861,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":23959,"num_objects_unfound":0,"num_objects_dirty":104287,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":110257,"num_write_kb":6865281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14127,"num_bytes_recovered":926224928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114566,"ondisk_log_size":114566,"up":141,"acting":133,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":149,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":26530916,"kb_used_data":21005776,"kb_used_omap":12,"kb_used_meta":5524851,"kb_avail":723430300,"statfs":{"total":767960285184,"available":740792627200,"internally_reserved":0,"allocated":21509914624,"data_stored":21490265787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12753,"internal_metadata":5657447983},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,4,2,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-189333480,"num_objects":-2888,"num_object_clones":0,"num_object_copies":-8664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1905,"num_objects_unfound":0,"num_objects_dirty":-2888,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3036,"num_write_kb":4673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1905,"num_bytes_recovered":124846080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.248020"},"pg_stats":[{"pgid":"6.7","version":"544'6740","reported_seq":6818,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-08-19T14:30:13.974012+0000","last_change":"2024-08-19T14:28:33.150392+0000","last_active":"2024-08-19T14:30:13.974012+0000","last_peered":"2024-08-19T14:30:13.974012+0000","last_clean":"2024-08-19T14:30:13.974012+0000","last_became_active":"2024-08-19T14:28:33.150057+0000","last_became_peered":"2024-08-19T14:28:33.150057+0000","last_unstale":"2024-08-19T14:30:13.974012+0000","last_undegraded":"2024-08-19T14:30:13.974012+0000","last_fullsized":"2024-08-19T14:30:13.974012+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":510,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:28:32.139424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:28:32.139424+0000","last_clean_scrub_stamp":"2024-08-19T14:28:32.139424+0000","objects_scrubbed":0,"log_size":6740,"log_dups_size":0,"ondisk_log_size":6740,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":422313984,"num_objects":6444,"num_object_clones":0,"num_object_copies":19332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6444,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6740,"num_write_kb":421888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"70'139","reported_seq":1358,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-08-19T14:30:12.272059+0000","last_change":"2024-08-19T14:28:57.210696+0000","last_active":"2024-08-19T14:30:12.272059+0000","last_peered":"2024-08-19T14:30:12.272059+0000","last_clean":"2024-08-19T14:30:12.272059+0000","last_became_active":"2024-08-19T14:28:20.866740+0000","last_became_peered":"2024-08-19T14:28:20.866740+0000","last_unstale":"2024-08-19T14:30:12.272059+0000","last_undegraded":"2024-08-19T14:30:12.272059+0000","last_fullsized":"2024-08-19T14:30:12.272059+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:57.210582+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:19:58.072083+0000","last_clean_scrub_stamp":"2024-08-19T14:28:57.210582+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:30:22.228862+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,4],"acting":[7,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"544'6900","reported_seq":6978,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-08-19T14:30:13.767728+0000","last_change":"2024-08-19T14:28:33.151848+0000","last_active":"2024-08-19T14:30:13.767728+0000","last_peered":"2024-08-19T14:30:13.767728+0000","last_clean":"2024-08-19T14:30:13.767728+0000","last_became_active":"2024-08-19T14:28:33.150851+0000","last_became_peered":"2024-08-19T14:28:33.150851+0000","last_unstale":"2024-08-19T14:30:13.767728+0000","last_undegraded":"2024-08-19T14:30:13.767728+0000","last_fullsized":"2024-08-19T14:30:13.767728+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":510,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:28:32.139424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:28:32.139424+0000","last_clean_scrub_stamp":"2024-08-19T14:28:32.139424+0000","objects_scrubbed":0,"log_size":6900,"log_dups_size":0,"ondisk_log_size":6900,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":432537600,"num_objects":6600,"num_object_clones":0,"num_object_copies":19800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6600,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6900,"num_write_kb":432000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"70'139","reported_seq":1239,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-08-19T14:30:12.300315+0000","last_change":"2024-08-19T14:28:33.188989+0000","last_active":"2024-08-19T14:30:12.300315+0000","last_peered":"2024-08-19T14:30:12.300315+0000","last_clean":"2024-08-19T14:30:12.300315+0000","last_became_active":"2024-08-19T14:18:13.804721+0000","last_became_peered":"2024-08-19T14:18:13.804721+0000","last_unstale":"2024-08-19T14:30:12.300315+0000","last_undegraded":"2024-08-19T14:30:12.300315+0000","last_fullsized":"2024-08-19T14:30:12.300315+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:33.188905+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:28:33.188905+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"544'6872","reported_seq":11260,"reported_epoch":544,"state":"active+recovering+undersized+remapped","last_fresh":"2024-08-19T14:30:14.068531+0000","last_change":"2024-08-19T14:29:17.710384+0000","last_active":"2024-08-19T14:30:14.068531+0000","last_peered":"2024-08-19T14:30:14.068531+0000","last_clean":"2024-08-19T14:29:15.397635+0000","last_became_active":"2024-08-19T14:29:17.667411+0000","last_became_peered":"2024-08-19T14:29:17.667411+0000","last_unstale":"2024-08-19T14:30:14.068531+0000","last_undegraded":"2024-08-19T14:30:14.068531+0000","last_fullsized":"2024-08-19T14:29:17.402564+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":510,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:28:32.139424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:28:32.139424+0000","last_clean_scrub_stamp":"2024-08-19T14:28:32.139424+0000","objects_scrubbed":0,"log_size":6872,"log_dups_size":0,"ondisk_log_size":6872,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":429129728,"num_objects":6548,"num_object_clones":0,"num_object_copies":19644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2606,"num_objects_unfound":0,"num_objects_dirty":6548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6872,"num_write_kb":429440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2153,"num_bytes_recovered":141099008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,5],"avail_no_missing":["7","1","5"],"object_location_counts":[{"shards":"1,5,7","objects":2606},{"shards":"5,7","objects":3942}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"70'139","reported_seq":1187,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-08-19T14:30:12.271882+0000","last_change":"2024-08-19T14:28:26.020419+0000","last_active":"2024-08-19T14:30:12.271882+0000","last_peered":"2024-08-19T14:30:12.271882+0000","last_clean":"2024-08-19T14:30:12.271882+0000","last_became_active":"2024-08-19T14:20:50.742354+0000","last_became_peered":"2024-08-19T14:20:50.742354+0000","last_unstale":"2024-08-19T14:30:12.271882+0000","last_undegraded":"2024-08-19T14:30:12.271882+0000","last_fullsized":"2024-08-19T14:30:12.271882+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:26.020327+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:19:59.302931+0000","last_clean_scrub_stamp":"2024-08-19T14:28:26.020327+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.6","version":"544'6868","reported_seq":6946,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-08-19T14:30:14.017188+0000","last_change":"2024-08-19T14:28:33.151142+0000","last_active":"2024-08-19T14:30:14.017188+0000","last_peered":"2024-08-19T14:30:14.017188+0000","last_clean":"2024-08-19T14:30:14.017188+0000","last_became_active":"2024-08-19T14:28:33.150830+0000","last_became_peered":"2024-08-19T14:28:33.150830+0000","last_unstale":"2024-08-19T14:30:14.017188+0000","last_undegraded":"2024-08-19T14:30:14.017188+0000","last_fullsized":"2024-08-19T14:30:14.017188+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":510,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:28:32.139424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:28:32.139424+0000","last_clean_scrub_stamp":"2024-08-19T14:28:32.139424+0000","objects_scrubbed":0,"log_size":6868,"log_dups_size":0,"ondisk_log_size":6868,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":430047232,"num_objects":6562,"num_object_clones":0,"num_object_copies":19686,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6562,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6868,"num_write_kb":429760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"70'139","reported_seq":461,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-08-19T14:30:17.298661+0000","last_change":"2024-08-19T14:29:18.889951+0000","last_active":"2024-08-19T14:30:17.298661+0000","last_peered":"2024-08-19T14:30:17.298661+0000","last_clean":"2024-08-19T14:30:17.298661+0000","last_became_active":"2024-08-19T14:29:18.889593+0000","last_became_peered":"2024-08-19T14:29:18.889593+0000","last_unstale":"2024-08-19T14:30:17.298661+0000","last_undegraded":"2024-08-19T14:30:17.298661+0000","last_fullsized":"2024-08-19T14:30:17.298661+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:28.323773+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:28:28.323773+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"545'7016","reported_seq":10036,"reported_epoch":545,"state":"active+recovering+undersized+remapped","last_fresh":"2024-08-19T14:30:18.071017+0000","last_change":"2024-08-19T14:29:17.791162+0000","last_active":"2024-08-19T14:30:18.071017+0000","last_peered":"2024-08-19T14:30:18.071017+0000","last_clean":"2024-08-19T14:29:15.399141+0000","last_became_active":"2024-08-19T14:29:17.721465+0000","last_became_peered":"2024-08-19T14:29:17.721465+0000","last_unstale":"2024-08-19T14:30:18.071017+0000","last_undegraded":"2024-08-19T14:30:18.071017+0000","last_fullsized":"2024-08-19T14:29:17.401297+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":510,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:28:32.139424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:28:32.139424+0000","last_clean_scrub_stamp":"2024-08-19T14:28:32.139424+0000","objects_scrubbed":0,"log_size":7016,"log_dups_size":0,"ondisk_log_size":7016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":418381824,"num_objects":6384,"num_object_clones":0,"num_object_copies":19152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3326,"num_objects_unfound":0,"num_objects_dirty":6384,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7016,"num_write_kb":428800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1469,"num_bytes_recovered":96272384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3],"avail_no_missing":["0","3","5"],"object_location_counts":[{"shards":"0,3","objects":3058},{"shards":"0,3,5","objects":3326}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"70'139","reported_seq":896,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-08-19T14:30:12.275239+0000","last_change":"2024-08-19T14:29:18.872571+0000","last_active":"2024-08-19T14:30:12.275239+0000","last_peered":"2024-08-19T14:30:12.275239+0000","last_clean":"2024-08-19T14:30:12.275239+0000","last_became_active":"2024-08-19T14:29:18.872252+0000","last_became_peered":"2024-08-19T14:29:18.872252+0000","last_unstale":"2024-08-19T14:30:12.275239+0000","last_undegraded":"2024-08-19T14:30:12.275239+0000","last_fullsized":"2024-08-19T14:30:12.275239+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:27.536928+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:21:46.057624+0000","last_clean_scrub_stamp":"2024-08-19T14:28:27.536928+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1","version":"544'6721","reported_seq":12053,"reported_epoch":544,"state":"active+recovering+undersized+remapped","last_fresh":"2024-08-19T14:30:15.165608+0000","last_change":"2024-08-19T14:29:17.849372+0000","last_active":"2024-08-19T14:30:15.165608+0000","last_peered":"2024-08-19T14:30:15.165608+0000","last_clean":"2024-08-19T14:29:15.373255+0000","last_became_active":"2024-08-19T14:29:17.784763+0000","last_became_peered":"2024-08-19T14:29:17.784763+0000","last_unstale":"2024-08-19T14:30:15.165608+0000","last_undegraded":"2024-08-19T14:30:15.165608+0000","last_fullsized":"2024-08-19T14:29:17.402400+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":510,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:28:32.139424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:28:32.139424+0000","last_clean_scrub_stamp":"2024-08-19T14:28:32.139424+0000","objects_scrubbed":0,"log_size":6721,"log_dups_size":0,"ondisk_log_size":6721,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":415432704,"num_objects":6339,"num_object_clones":0,"num_object_copies":19017,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2020,"num_objects_unfound":0,"num_objects_dirty":6339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6721,"num_write_kb":417920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2625,"num_bytes_recovered":172032000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,2],"acting":[8,7],"avail_no_missing":["8","5","7"],"object_location_counts":[{"shards":"5,7,8","objects":2020},{"shards":"7,8","objects":4319}],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.6","version":"70'139","reported_seq":441,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-08-19T14:30:12.277637+0000","last_change":"2024-08-19T14:28:59.173001+0000","last_active":"2024-08-19T14:30:12.277637+0000","last_peered":"2024-08-19T14:30:12.277637+0000","last_clean":"2024-08-19T14:30:12.277637+0000","last_became_active":"2024-08-19T14:28:08.847096+0000","last_became_peered":"2024-08-19T14:28:08.847096+0000","last_unstale":"2024-08-19T14:30:12.277637+0000","last_undegraded":"2024-08-19T14:30:12.277637+0000","last_fullsized":"2024-08-19T14:30:12.277637+0000","mapping_epoch":496,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":497,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:59.172931+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:27.848870+0000","last_clean_scrub_stamp":"2024-08-19T14:28:59.172931+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:30:23.212613+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,3],"acting":[8,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"6.2","version":"544'6901","reported_seq":9887,"reported_epoch":544,"state":"active+recovering+undersized+remapped","last_fresh":"2024-08-19T14:30:13.331608+0000","last_change":"2024-08-19T14:29:17.742970+0000","last_active":"2024-08-19T14:30:13.331608+0000","last_peered":"2024-08-19T14:30:13.331608+0000","last_clean":"2024-08-19T14:29:15.390795+0000","last_became_active":"2024-08-19T14:29:17.702334+0000","last_became_peered":"2024-08-19T14:29:17.702334+0000","last_unstale":"2024-08-19T14:30:13.331608+0000","last_undegraded":"2024-08-19T14:30:13.331608+0000","last_fullsized":"2024-08-19T14:29:17.400096+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":510,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:28:32.139424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:28:32.139424+0000","last_clean_scrub_stamp":"2024-08-19T14:28:32.139424+0000","objects_scrubbed":0,"log_size":6901,"log_dups_size":0,"ondisk_log_size":6901,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":434044928,"num_objects":6623,"num_object_clones":0,"num_object_copies":19869,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3273,"num_objects_unfound":0,"num_objects_dirty":6623,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6901,"num_write_kb":432768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1453,"num_bytes_recovered":95223808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3],"avail_no_missing":["4","3","8"],"object_location_counts":[{"shards":"3,4","objects":3350},{"shards":"3,4,8","objects":3273}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"70'139","reported_seq":416,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-08-19T14:30:17.294223+0000","last_change":"2024-08-19T14:29:18.880926+0000","last_active":"2024-08-19T14:30:17.294223+0000","last_peered":"2024-08-19T14:30:17.294223+0000","last_clean":"2024-08-19T14:30:17.294223+0000","last_became_active":"2024-08-19T14:29:18.880523+0000","last_became_peered":"2024-08-19T14:29:18.880523+0000","last_unstale":"2024-08-19T14:30:17.294223+0000","last_undegraded":"2024-08-19T14:30:17.294223+0000","last_fullsized":"2024-08-19T14:30:17.294223+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:27.331041+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:28:27.331041+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"544'6900","reported_seq":6978,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-08-19T14:30:14.039480+0000","last_change":"2024-08-19T14:28:33.151000+0000","last_active":"2024-08-19T14:30:14.039480+0000","last_peered":"2024-08-19T14:30:14.039480+0000","last_clean":"2024-08-19T14:30:14.039480+0000","last_became_active":"2024-08-19T14:28:33.150652+0000","last_became_peered":"2024-08-19T14:28:33.150652+0000","last_unstale":"2024-08-19T14:30:14.039480+0000","last_undegraded":"2024-08-19T14:30:14.039480+0000","last_fullsized":"2024-08-19T14:30:14.039480+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":510,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:28:32.139424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:28:32.139424+0000","last_clean_scrub_stamp":"2024-08-19T14:28:32.139424+0000","objects_scrubbed":0,"log_size":6900,"log_dups_size":0,"ondisk_log_size":6900,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":430178304,"num_objects":6564,"num_object_clones":0,"num_object_copies":19692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6900,"num_write_kb":430848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,0],"acting":[7,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"70'139","reported_seq":414,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-08-19T14:30:12.277611+0000","last_change":"2024-08-19T14:28:28.122304+0000","last_active":"2024-08-19T14:30:12.277611+0000","last_peered":"2024-08-19T14:30:12.277611+0000","last_clean":"2024-08-19T14:30:12.277611+0000","last_became_active":"2024-08-19T14:28:20.859362+0000","last_became_peered":"2024-08-19T14:28:20.859362+0000","last_unstale":"2024-08-19T14:30:12.277611+0000","last_undegraded":"2024-08-19T14:30:12.277611+0000","last_fullsized":"2024-08-19T14:30:12.277611+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:28.122172+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:28:28.122172+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,4],"acting":[8,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"6.d","version":"544'6868","reported_seq":6947,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-08-19T14:30:13.693354+0000","last_change":"2024-08-19T14:28:33.151442+0000","last_active":"2024-08-19T14:30:13.693354+0000","last_peered":"2024-08-19T14:30:13.693354+0000","last_clean":"2024-08-19T14:30:13.693354+0000","last_became_active":"2024-08-19T14:28:33.151135+0000","last_became_peered":"2024-08-19T14:28:33.151135+0000","last_unstale":"2024-08-19T14:30:13.693354+0000","last_undegraded":"2024-08-19T14:30:13.693354+0000","last_fullsized":"2024-08-19T14:30:13.693354+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":510,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:28:32.139424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:28:32.139424+0000","last_clean_scrub_stamp":"2024-08-19T14:28:32.139424+0000","objects_scrubbed":0,"log_size":6868,"log_dups_size":0,"ondisk_log_size":6868,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":428146712,"num_objects":6534,"num_object_clones":0,"num_object_copies":19602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6534,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6868,"num_write_kb":428801,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"70'139","reported_seq":450,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-08-19T14:30:12.268606+0000","last_change":"2024-08-19T14:29:18.887671+0000","last_active":"2024-08-19T14:30:12.268606+0000","last_peered":"2024-08-19T14:30:12.268606+0000","last_clean":"2024-08-19T14:30:12.268606+0000","last_became_active":"2024-08-19T14:29:18.887032+0000","last_became_peered":"2024-08-19T14:29:18.887032+0000","last_unstale":"2024-08-19T14:30:12.268606+0000","last_undegraded":"2024-08-19T14:30:12.268606+0000","last_fullsized":"2024-08-19T14:30:12.268606+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:50.681699+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:28.864870+0000","last_clean_scrub_stamp":"2024-08-19T14:28:50.681699+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"544'6951","reported_seq":10547,"reported_epoch":544,"state":"active+recovering+undersized+remapped","last_fresh":"2024-08-19T14:30:13.324868+0000","last_change":"2024-08-19T14:29:17.766952+0000","last_active":"2024-08-19T14:30:13.324868+0000","last_peered":"2024-08-19T14:30:13.324868+0000","last_clean":"2024-08-19T14:29:15.399942+0000","last_became_active":"2024-08-19T14:29:17.717582+0000","last_became_peered":"2024-08-19T14:29:17.717582+0000","last_unstale":"2024-08-19T14:30:13.324868+0000","last_undegraded":"2024-08-19T14:30:13.324868+0000","last_fullsized":"2024-08-19T14:29:17.399900+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":510,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:28:32.139424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:28:32.139424+0000","last_clean_scrub_stamp":"2024-08-19T14:28:32.139424+0000","objects_scrubbed":0,"log_size":6951,"log_dups_size":0,"ondisk_log_size":6951,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436928512,"num_objects":6667,"num_object_clones":0,"num_object_copies":20001,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3109,"num_objects_unfound":0,"num_objects_dirty":6667,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6951,"num_write_kb":435776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1757,"num_bytes_recovered":115146752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1],"avail_no_missing":["4","1","5"],"object_location_counts":[{"shards":"1,4","objects":3558},{"shards":"1,4,5","objects":3109}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"70'139","reported_seq":1380,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-08-19T14:30:12.272010+0000","last_change":"2024-08-19T14:29:18.881074+0000","last_active":"2024-08-19T14:30:12.272010+0000","last_peered":"2024-08-19T14:30:12.272010+0000","last_clean":"2024-08-19T14:30:12.272010+0000","last_became_active":"2024-08-19T14:29:18.880645+0000","last_became_peered":"2024-08-19T14:29:18.880645+0000","last_unstale":"2024-08-19T14:30:12.272010+0000","last_undegraded":"2024-08-19T14:30:12.272010+0000","last_fullsized":"2024-08-19T14:30:12.272010+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:53.273092+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:28:53.273092+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:30:17.484617+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.f","version":"544'6712","reported_seq":11154,"reported_epoch":544,"state":"active+recovering+undersized+remapped","last_fresh":"2024-08-19T14:30:13.334650+0000","last_change":"2024-08-19T14:29:17.711760+0000","last_active":"2024-08-19T14:30:13.334650+0000","last_peered":"2024-08-19T14:30:13.334650+0000","last_clean":"2024-08-19T14:29:15.392366+0000","last_became_active":"2024-08-19T14:29:17.682206+0000","last_became_peered":"2024-08-19T14:29:17.682206+0000","last_unstale":"2024-08-19T14:30:13.334650+0000","last_undegraded":"2024-08-19T14:30:13.334650+0000","last_fullsized":"2024-08-19T14:29:17.400464+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":510,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:28:32.139424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:28:32.139424+0000","last_clean_scrub_stamp":"2024-08-19T14:28:32.139424+0000","objects_scrubbed":0,"log_size":6712,"log_dups_size":0,"ondisk_log_size":6712,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":422969344,"num_objects":6454,"num_object_clones":0,"num_object_copies":19362,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2534,"num_objects_unfound":0,"num_objects_dirty":6454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6712,"num_write_kb":421312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2180,"num_bytes_recovered":142868480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,5],"avail_no_missing":["4","5","7"],"object_location_counts":[{"shards":"4,5","objects":3920},{"shards":"4,5,7","objects":2534}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"70'139","reported_seq":427,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-08-19T14:30:12.278724+0000","last_change":"2024-08-19T14:29:18.890961+0000","last_active":"2024-08-19T14:30:12.278724+0000","last_peered":"2024-08-19T14:30:12.278724+0000","last_clean":"2024-08-19T14:30:12.278724+0000","last_became_active":"2024-08-19T14:29:18.890423+0000","last_became_peered":"2024-08-19T14:29:18.890423+0000","last_unstale":"2024-08-19T14:30:12.278724+0000","last_undegraded":"2024-08-19T14:30:12.278724+0000","last_fullsized":"2024-08-19T14:30:12.278724+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:59.829567+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:28:59.829567+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"544'6910","reported_seq":6988,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-08-19T14:30:14.045189+0000","last_change":"2024-08-19T14:28:33.150427+0000","last_active":"2024-08-19T14:30:14.045189+0000","last_peered":"2024-08-19T14:30:14.045189+0000","last_clean":"2024-08-19T14:30:14.045189+0000","last_became_active":"2024-08-19T14:28:33.150105+0000","last_became_peered":"2024-08-19T14:28:33.150105+0000","last_unstale":"2024-08-19T14:30:14.045189+0000","last_undegraded":"2024-08-19T14:30:14.045189+0000","last_fullsized":"2024-08-19T14:30:14.045189+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":510,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:28:32.139424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:28:32.139424+0000","last_clean_scrub_stamp":"2024-08-19T14:28:32.139424+0000","objects_scrubbed":0,"log_size":6910,"log_dups_size":0,"ondisk_log_size":6910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":433324032,"num_objects":6612,"num_object_clones":0,"num_object_copies":19836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6612,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6910,"num_write_kb":432704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,8],"acting":[3,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"70'139","reported_seq":746,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-08-19T14:30:17.293779+0000","last_change":"2024-08-19T14:29:18.872722+0000","last_active":"2024-08-19T14:30:17.293779+0000","last_peered":"2024-08-19T14:30:17.293779+0000","last_clean":"2024-08-19T14:30:17.293779+0000","last_became_active":"2024-08-19T14:29:18.872333+0000","last_became_peered":"2024-08-19T14:29:18.872333+0000","last_unstale":"2024-08-19T14:30:17.293779+0000","last_undegraded":"2024-08-19T14:30:17.293779+0000","last_fullsized":"2024-08-19T14:30:17.293779+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:26.371992+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:28:26.371992+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.9","version":"545'7029","reported_seq":9489,"reported_epoch":545,"state":"active+recovering+undersized+remapped","last_fresh":"2024-08-19T14:30:18.117619+0000","last_change":"2024-08-19T14:29:17.884357+0000","last_active":"2024-08-19T14:30:18.117619+0000","last_peered":"2024-08-19T14:30:18.117619+0000","last_clean":"2024-08-19T14:29:15.389803+0000","last_became_active":"2024-08-19T14:29:17.790859+0000","last_became_peered":"2024-08-19T14:29:17.790859+0000","last_unstale":"2024-08-19T14:30:18.117619+0000","last_undegraded":"2024-08-19T14:30:18.117619+0000","last_fullsized":"2024-08-19T14:29:17.401770+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":510,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:28:32.139424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:28:32.139424+0000","last_clean_scrub_stamp":"2024-08-19T14:28:32.139424+0000","objects_scrubbed":0,"log_size":7029,"log_dups_size":0,"ondisk_log_size":7029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":417792000,"num_objects":6375,"num_object_clones":0,"num_object_copies":19125,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3576,"num_objects_unfound":0,"num_objects_dirty":6375,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7029,"num_write_kb":428928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1188,"num_bytes_recovered":77856768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,2],"acting":[0,8],"avail_no_missing":["0","1","8"],"object_location_counts":[{"shards":"0,1,8","objects":3576},{"shards":"0,8","objects":2799}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"70'139","reported_seq":915,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-08-19T14:30:12.268768+0000","last_change":"2024-08-19T14:28:26.742527+0000","last_active":"2024-08-19T14:30:12.268768+0000","last_peered":"2024-08-19T14:30:12.268768+0000","last_clean":"2024-08-19T14:30:12.268768+0000","last_became_active":"2024-08-19T14:18:26.884105+0000","last_became_peered":"2024-08-19T14:18:26.884105+0000","last_unstale":"2024-08-19T14:30:12.268768+0000","last_undegraded":"2024-08-19T14:30:12.268768+0000","last_fullsized":"2024-08-19T14:30:12.268768+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:26.742385+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:20:15.321945+0000","last_clean_scrub_stamp":"2024-08-19T14:28:26.742385+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"544'6941","reported_seq":7019,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-08-19T14:30:14.059686+0000","last_change":"2024-08-19T14:28:33.154021+0000","last_active":"2024-08-19T14:30:14.059686+0000","last_peered":"2024-08-19T14:30:14.059686+0000","last_clean":"2024-08-19T14:30:14.059686+0000","last_became_active":"2024-08-19T14:28:33.153670+0000","last_became_peered":"2024-08-19T14:28:33.153670+0000","last_unstale":"2024-08-19T14:30:14.059686+0000","last_undegraded":"2024-08-19T14:30:14.059686+0000","last_fullsized":"2024-08-19T14:30:14.059686+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":510,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:28:32.139424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:28:32.139424+0000","last_clean_scrub_stamp":"2024-08-19T14:28:32.139424+0000","objects_scrubbed":0,"log_size":6941,"log_dups_size":0,"ondisk_log_size":6941,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":434438144,"num_objects":6629,"num_object_clones":0,"num_object_copies":19887,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6941,"num_write_kb":434240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"70'139","reported_seq":683,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-08-19T14:30:12.269627+0000","last_change":"2024-08-19T14:28:52.777147+0000","last_active":"2024-08-19T14:30:12.269627+0000","last_peered":"2024-08-19T14:30:12.269627+0000","last_clean":"2024-08-19T14:30:12.269627+0000","last_became_active":"2024-08-19T14:27:31.837754+0000","last_became_peered":"2024-08-19T14:27:31.837754+0000","last_unstale":"2024-08-19T14:30:12.269627+0000","last_undegraded":"2024-08-19T14:30:12.269627+0000","last_fullsized":"2024-08-19T14:30:12.269627+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:52.777044+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:28:52.777044+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"544'6924","reported_seq":7002,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-08-19T14:30:13.974030+0000","last_change":"2024-08-19T14:28:33.151083+0000","last_active":"2024-08-19T14:30:13.974030+0000","last_peered":"2024-08-19T14:30:13.974030+0000","last_clean":"2024-08-19T14:30:13.974030+0000","last_became_active":"2024-08-19T14:28:33.150788+0000","last_became_peered":"2024-08-19T14:28:33.150788+0000","last_unstale":"2024-08-19T14:30:13.974030+0000","last_undegraded":"2024-08-19T14:30:13.974030+0000","last_fullsized":"2024-08-19T14:30:13.974030+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":510,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:28:32.139424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:28:32.139424+0000","last_clean_scrub_stamp":"2024-08-19T14:28:32.139424+0000","objects_scrubbed":0,"log_size":6924,"log_dups_size":0,"ondisk_log_size":6924,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":432537600,"num_objects":6600,"num_object_clones":0,"num_object_copies":19800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6600,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6924,"num_write_kb":432768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"70'139","reported_seq":1033,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-08-19T14:30:12.280376+0000","last_change":"2024-08-19T14:28:26.508582+0000","last_active":"2024-08-19T14:30:12.280376+0000","last_peered":"2024-08-19T14:30:12.280376+0000","last_clean":"2024-08-19T14:30:12.280376+0000","last_became_active":"2024-08-19T14:28:08.849293+0000","last_became_peered":"2024-08-19T14:28:08.849293+0000","last_unstale":"2024-08-19T14:30:12.280376+0000","last_undegraded":"2024-08-19T14:30:12.280376+0000","last_fullsized":"2024-08-19T14:30:12.280376+0000","mapping_epoch":496,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":497,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:26.508506+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:22:52.164586+0000","last_clean_scrub_stamp":"2024-08-19T14:28:26.508506+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"545'7004","reported_seq":9688,"reported_epoch":545,"state":"active+recovering+undersized+remapped","last_fresh":"2024-08-19T14:30:18.140520+0000","last_change":"2024-08-19T14:29:17.782083+0000","last_active":"2024-08-19T14:30:18.140520+0000","last_peered":"2024-08-19T14:30:18.140520+0000","last_clean":"2024-08-19T14:29:15.395974+0000","last_became_active":"2024-08-19T14:29:17.703574+0000","last_became_peered":"2024-08-19T14:29:17.703574+0000","last_unstale":"2024-08-19T14:30:18.140520+0000","last_undegraded":"2024-08-19T14:30:18.140520+0000","last_fullsized":"2024-08-19T14:29:17.401163+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":510,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:28:32.139424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:28:32.139424+0000","last_clean_scrub_stamp":"2024-08-19T14:28:32.139424+0000","objects_scrubbed":0,"log_size":7004,"log_dups_size":0,"ondisk_log_size":7004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":416153600,"num_objects":6350,"num_object_clones":0,"num_object_copies":19050,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3515,"num_objects_unfound":0,"num_objects_dirty":6350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7004,"num_write_kb":427328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1300,"num_bytes_recovered":85196800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5],"avail_no_missing":["0","1","5"],"object_location_counts":[{"shards":"0,1,5","objects":3515},{"shards":"0,5","objects":2835}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"70'139","reported_seq":935,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-08-19T14:30:12.275926+0000","last_change":"2024-08-19T14:28:29.007026+0000","last_active":"2024-08-19T14:30:12.275926+0000","last_peered":"2024-08-19T14:30:12.275926+0000","last_clean":"2024-08-19T14:30:12.275926+0000","last_became_active":"2024-08-19T14:18:26.882002+0000","last_became_peered":"2024-08-19T14:18:26.882002+0000","last_unstale":"2024-08-19T14:30:12.275926+0000","last_undegraded":"2024-08-19T14:30:12.275926+0000","last_fullsized":"2024-08-19T14:30:12.275926+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:29.006921+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:24:50.276408+0000","last_clean_scrub_stamp":"2024-08-19T14:28:29.006921+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"70'139","reported_seq":722,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-08-19T14:30:12.290851+0000","last_change":"2024-08-19T14:28:35.206191+0000","last_active":"2024-08-19T14:30:12.290851+0000","last_peered":"2024-08-19T14:30:12.290851+0000","last_clean":"2024-08-19T14:30:12.290851+0000","last_became_active":"2024-08-19T14:27:31.838968+0000","last_became_peered":"2024-08-19T14:27:31.838968+0000","last_unstale":"2024-08-19T14:30:12.290851+0000","last_undegraded":"2024-08-19T14:30:12.290851+0000","last_fullsized":"2024-08-19T14:30:12.290851+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:35.206115+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:28:35.206115+0000","last_clean_scrub_stamp":"2024-08-19T14:28:35.206115+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.13","version":"70'139","reported_seq":1255,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-08-19T14:30:12.299725+0000","last_change":"2024-08-19T14:28:53.095477+0000","last_active":"2024-08-19T14:30:12.299725+0000","last_peered":"2024-08-19T14:30:12.299725+0000","last_clean":"2024-08-19T14:30:12.299725+0000","last_became_active":"2024-08-19T14:28:20.861463+0000","last_became_peered":"2024-08-19T14:28:20.861463+0000","last_unstale":"2024-08-19T14:30:12.299725+0000","last_undegraded":"2024-08-19T14:30:12.299725+0000","last_fullsized":"2024-08-19T14:30:12.299725+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:53.095364+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:28:53.095364+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.10","version":"70'139","reported_seq":884,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-08-19T14:30:12.271817+0000","last_change":"2024-08-19T14:29:00.244077+0000","last_active":"2024-08-19T14:30:12.271817+0000","last_peered":"2024-08-19T14:30:12.271817+0000","last_clean":"2024-08-19T14:30:12.271817+0000","last_became_active":"2024-08-19T14:27:31.839611+0000","last_became_peered":"2024-08-19T14:27:31.839611+0000","last_unstale":"2024-08-19T14:30:12.271817+0000","last_undegraded":"2024-08-19T14:30:12.271817+0000","last_fullsized":"2024-08-19T14:30:12.271817+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:29:00.244002+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:29:00.244002+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.11","version":"70'139","reported_seq":917,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-08-19T14:30:17.297557+0000","last_change":"2024-08-19T14:28:25.362916+0000","last_active":"2024-08-19T14:30:17.297557+0000","last_peered":"2024-08-19T14:30:17.297557+0000","last_clean":"2024-08-19T14:30:17.297557+0000","last_became_active":"2024-08-19T14:28:08.848199+0000","last_became_peered":"2024-08-19T14:28:08.848199+0000","last_unstale":"2024-08-19T14:30:17.297557+0000","last_undegraded":"2024-08-19T14:30:17.297557+0000","last_fullsized":"2024-08-19T14:30:17.297557+0000","mapping_epoch":496,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":497,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:25.362836+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:24:50.547428+0000","last_clean_scrub_stamp":"2024-08-19T14:28:25.362836+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,3],"acting":[0,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"70'139","reported_seq":874,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-08-19T14:30:12.291124+0000","last_change":"2024-08-19T14:28:34.216440+0000","last_active":"2024-08-19T14:30:12.291124+0000","last_peered":"2024-08-19T14:30:12.291124+0000","last_clean":"2024-08-19T14:30:12.291124+0000","last_became_active":"2024-08-19T14:20:28.204724+0000","last_became_peered":"2024-08-19T14:20:28.204724+0000","last_unstale":"2024-08-19T14:30:12.291124+0000","last_undegraded":"2024-08-19T14:30:12.291124+0000","last_fullsized":"2024-08-19T14:30:12.291124+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:34.216370+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:28:34.216370+0000","last_clean_scrub_stamp":"2024-08-19T14:28:34.216370+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"70'139","reported_seq":1002,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-08-19T14:30:12.277501+0000","last_change":"2024-08-19T14:28:30.136536+0000","last_active":"2024-08-19T14:30:12.277501+0000","last_peered":"2024-08-19T14:30:12.277501+0000","last_clean":"2024-08-19T14:30:12.277501+0000","last_became_active":"2024-08-19T14:18:31.901556+0000","last_became_peered":"2024-08-19T14:18:31.901556+0000","last_unstale":"2024-08-19T14:30:12.277501+0000","last_undegraded":"2024-08-19T14:30:12.277501+0000","last_fullsized":"2024-08-19T14:30:12.277501+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:30.136404+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:28:30.136404+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,0],"acting":[8,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.14","version":"70'139","reported_seq":1030,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-08-19T14:30:12.272009+0000","last_change":"2024-08-19T14:29:18.872109+0000","last_active":"2024-08-19T14:30:12.272009+0000","last_peered":"2024-08-19T14:30:12.272009+0000","last_clean":"2024-08-19T14:30:12.272009+0000","last_became_active":"2024-08-19T14:29:18.871735+0000","last_became_peered":"2024-08-19T14:29:18.871735+0000","last_unstale":"2024-08-19T14:30:12.272009+0000","last_undegraded":"2024-08-19T14:30:12.272009+0000","last_fullsized":"2024-08-19T14:30:12.272009+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:58.211358+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:28:58.211358+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.15","version":"70'139","reported_seq":890,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-08-19T14:30:12.275828+0000","last_change":"2024-08-19T14:28:31.999297+0000","last_active":"2024-08-19T14:30:12.275828+0000","last_peered":"2024-08-19T14:30:12.275828+0000","last_clean":"2024-08-19T14:30:12.275828+0000","last_became_active":"2024-08-19T14:18:31.912496+0000","last_became_peered":"2024-08-19T14:18:31.912496+0000","last_unstale":"2024-08-19T14:30:12.275828+0000","last_undegraded":"2024-08-19T14:30:12.275828+0000","last_fullsized":"2024-08-19T14:30:12.275828+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:31.999202+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:20:09.180923+0000","last_clean_scrub_stamp":"2024-08-19T14:28:31.999202+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"70'139","reported_seq":876,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-08-19T14:30:12.268435+0000","last_change":"2024-08-19T14:28:30.643104+0000","last_active":"2024-08-19T14:30:12.268435+0000","last_peered":"2024-08-19T14:30:12.268435+0000","last_clean":"2024-08-19T14:30:12.268435+0000","last_became_active":"2024-08-19T14:18:31.901229+0000","last_became_peered":"2024-08-19T14:18:31.901229+0000","last_unstale":"2024-08-19T14:30:12.268435+0000","last_undegraded":"2024-08-19T14:30:12.268435+0000","last_fullsized":"2024-08-19T14:30:12.268435+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:30.643007+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:28:30.643007+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1b","version":"70'139","reported_seq":1246,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-08-19T14:30:12.279212+0000","last_change":"2024-08-19T14:29:18.871775+0000","last_active":"2024-08-19T14:30:12.279212+0000","last_peered":"2024-08-19T14:30:12.279212+0000","last_clean":"2024-08-19T14:30:12.279212+0000","last_became_active":"2024-08-19T14:29:18.871524+0000","last_became_peered":"2024-08-19T14:29:18.871524+0000","last_unstale":"2024-08-19T14:30:12.279212+0000","last_undegraded":"2024-08-19T14:30:12.279212+0000","last_fullsized":"2024-08-19T14:30:12.279212+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:24.065359+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:28:24.065359+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.18","version":"70'139","reported_seq":826,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-08-19T14:30:12.271815+0000","last_change":"2024-08-19T14:29:18.887071+0000","last_active":"2024-08-19T14:30:12.271815+0000","last_peered":"2024-08-19T14:30:12.271815+0000","last_clean":"2024-08-19T14:30:12.271815+0000","last_became_active":"2024-08-19T14:29:18.886630+0000","last_became_peered":"2024-08-19T14:29:18.886630+0000","last_unstale":"2024-08-19T14:30:12.271815+0000","last_undegraded":"2024-08-19T14:30:12.271815+0000","last_fullsized":"2024-08-19T14:30:12.271815+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:29:05.336156+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:27:33.084161+0000","last_clean_scrub_stamp":"2024-08-19T14:29:05.336156+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:30:23.881542+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.19","version":"70'139","reported_seq":1222,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-08-19T14:30:17.297159+0000","last_change":"2024-08-19T14:28:22.353720+0000","last_active":"2024-08-19T14:30:17.297159+0000","last_peered":"2024-08-19T14:30:17.297159+0000","last_clean":"2024-08-19T14:30:17.297159+0000","last_became_active":"2024-08-19T14:18:31.902547+0000","last_became_peered":"2024-08-19T14:18:31.902547+0000","last_unstale":"2024-08-19T14:30:17.297159+0000","last_undegraded":"2024-08-19T14:30:17.297159+0000","last_fullsized":"2024-08-19T14:30:17.297159+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:22.353632+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:28:22.353632+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1e","version":"70'139","reported_seq":386,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-08-19T14:30:12.277541+0000","last_change":"2024-08-19T14:28:27.136259+0000","last_active":"2024-08-19T14:30:12.277541+0000","last_peered":"2024-08-19T14:30:12.277541+0000","last_clean":"2024-08-19T14:30:12.277541+0000","last_became_active":"2024-08-19T14:28:20.859275+0000","last_became_peered":"2024-08-19T14:28:20.859275+0000","last_unstale":"2024-08-19T14:30:12.277541+0000","last_undegraded":"2024-08-19T14:30:12.277541+0000","last_fullsized":"2024-08-19T14:30:12.277541+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:27.136174+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:27:04.780939+0000","last_clean_scrub_stamp":"2024-08-19T14:28:27.136174+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,4],"acting":[8,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1c","version":"70'139","reported_seq":995,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-08-19T14:30:12.277505+0000","last_change":"2024-08-19T14:28:25.098266+0000","last_active":"2024-08-19T14:30:12.277505+0000","last_peered":"2024-08-19T14:30:12.277505+0000","last_clean":"2024-08-19T14:30:12.277505+0000","last_became_active":"2024-08-19T14:28:20.856125+0000","last_became_peered":"2024-08-19T14:28:20.856125+0000","last_unstale":"2024-08-19T14:30:12.277505+0000","last_undegraded":"2024-08-19T14:30:12.277505+0000","last_fullsized":"2024-08-19T14:30:12.277505+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:25.098173+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:28:25.098173+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1d","version":"70'139","reported_seq":918,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-08-19T14:30:12.291245+0000","last_change":"2024-08-19T14:29:18.875372+0000","last_active":"2024-08-19T14:30:12.291245+0000","last_peered":"2024-08-19T14:30:12.291245+0000","last_clean":"2024-08-19T14:30:12.291245+0000","last_became_active":"2024-08-19T14:29:18.874819+0000","last_became_peered":"2024-08-19T14:29:18.874819+0000","last_unstale":"2024-08-19T14:30:12.291245+0000","last_undegraded":"2024-08-19T14:30:12.291245+0000","last_fullsized":"2024-08-19T14:30:12.291245+0000","mapping_epoch":527,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:29.194639+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:28:29.194639+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":6834356248,"num_objects":104285,"num_object_clones":0,"num_object_copies":312855,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":23959,"num_objects_unfound":0,"num_objects_dirty":104285,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":110257,"num_write_kb":6865281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14125,"num_bytes_recovered":925696000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":21381918720,"data_stored":21381906504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110257,"ondisk_log_size":110257,"up":48,"acting":40,"num_store_stats":8},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1925120,"data_stored":1914464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":0,"seq":369367187683,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":2314987372550,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":0,"seq":317827580141,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874482,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2624252,"kb_used_data":1975284,"kb_used_omap":1,"kb_used_meta":648958,"kb_avail":91120900,"statfs":{"total":95995035648,"available":93307801600,"internally_reserved":0,"allocated":2022690816,"data_stored":2020238067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":664533445},"hb_peers":[0,1,2,3,4,5,6,7,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":57,"seq":244813136117,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3650244,"kb_used_data":2834684,"kb_used_omap":1,"kb_used_meta":815550,"kb_avail":90094908,"statfs":{"total":95995035648,"available":92257185792,"internally_reserved":0,"allocated":2902716416,"data_stored":2900260115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":835123658},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":206158430456,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692093,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4520456,"kb_used_data":3807616,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":89224696,"statfs":{"total":95995035648,"available":91366088704,"internally_reserved":0,"allocated":3898998784,"data_stored":3896538411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986433,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3347204,"kb_used_data":2535796,"kb_used_omap":1,"kb_used_meta":811390,"kb_avail":90397948,"statfs":{"total":95995035648,"available":92567498752,"internally_reserved":0,"allocated":2596655104,"data_stored":2594202355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":830863813},"hb_peers":[0,1,2,3,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149959,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4109312,"kb_used_data":3388148,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":89635840,"statfs":{"total":95995035648,"available":91787100160,"internally_reserved":0,"allocated":3469463552,"data_stored":3467010803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":738458055},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411597,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2045252,"kb_used_data":1789116,"kb_used_omap":1,"kb_used_meta":256062,"kb_avail":91699900,"statfs":{"total":95995035648,"available":93900697600,"internally_reserved":0,"allocated":1832054784,"data_stored":1829598483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":262207941},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575120,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3472580,"kb_used_data":2620792,"kb_used_omap":1,"kb_used_meta":851646,"kb_avail":90272572,"statfs":{"total":95995035648,"available":92439113728,"internally_reserved":0,"allocated":2683691008,"data_stored":2681234187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":872085957},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738645,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2761616,"kb_used_data":2054340,"kb_used_omap":1,"kb_used_meta":707262,"kb_avail":90983536,"statfs":{"total":95995035648,"available":93167140864,"internally_reserved":0,"allocated":2103644160,"data_stored":2101183366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":724236741},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,2,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2090012672,"data_stored":2090008600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2669285376,"data_stored":2669281304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1823277056,"data_stored":1823277056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3445227520,"data_stored":3445227520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2586247168,"data_stored":2586247168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3875475456,"data_stored":3875471384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2883846144,"data_stored":2883846144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":2008547328,"data_stored":2008547328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T14:30:20.963 INFO:tasks.thrashosds.thrasher:Setting 6.6 to [5, 1, 7] 2024-08-19T14:30:20.963 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '6.6', '5', '1', '7'] 2024-08-19T14:30:20.963 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd pg-upmap 6.6 5 1 7 2024-08-19T14:30:20.993 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:30:21.023 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:21.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:21.024 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:30:21.125 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:30:21.150 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:21.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:21.151 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:30:21.252 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:30:21.274 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:21.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:21.275 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:30:21.376 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:30:21.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:21.416 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:21.416 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:30:21.517 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:30:21.557 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:21.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:21.558 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:30:21.562 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:21 smithi031 ceph-mon[16336]: pgmap v1260: 47 pgs: 8 active+recovering+undersized+remapped, 39 active+clean; 6.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 582 op/s; 23233/306045 objects misplaced (7.591%); 16 MiB/s, 251 objects/s recovering 2024-08-19T14:30:21.563 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:21 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:30:21.563 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:21 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:30:21.659 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:30:21.681 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:21.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:21.682 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:30:21.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:21 smithi032 ceph-mon[20004]: pgmap v1260: 47 pgs: 8 active+recovering+undersized+remapped, 39 active+clean; 6.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 582 op/s; 23233/306045 objects misplaced (7.591%); 16 MiB/s, 251 objects/s recovering 2024-08-19T14:30:21.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:21 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:30:21.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:21 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:30:21.765 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:30:21.782 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:30:21.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:21 smithi045 ceph-mon[20143]: pgmap v1260: 47 pgs: 8 active+recovering+undersized+remapped, 39 active+clean; 6.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 582 op/s; 23233/306045 objects misplaced (7.591%); 16 MiB/s, 251 objects/s recovering 2024-08-19T14:30:21.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:21 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:30:21.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:21 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:30:21.814 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:21.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:21.816 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:30:21.917 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:30:21.947 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:21.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:21.948 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:30:22.049 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:30:22.075 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:22.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:22.076 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:30:22.143 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-08-19T14:30:22.177 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:30:22.219 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:22.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:22.220 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:30:22.321 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:30:22.359 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:22.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:22.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:30:22.412 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:30:22.461 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:30:22.482 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:22.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:22.483 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:30:22.532 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-19T14:30:22.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:22 smithi045 ceph-mon[20143]: osdmap e548: 12 total, 12 up, 8 in 2024-08-19T14:30:22.573 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:22 smithi031 ceph-mon[16336]: osdmap e548: 12 total, 12 up, 8 in 2024-08-19T14:30:22.584 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:30:22.605 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:22.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:22.606 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:30:22.707 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:30:22.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:22 smithi032 ceph-mon[20004]: osdmap e548: 12 total, 12 up, 8 in 2024-08-19T14:30:22.760 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:22.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:22.761 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:30:22.862 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:30:22.885 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:22.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:22.886 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:30:22.945 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:30:22.946 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:30:22.988 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:30:23.009 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:23.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:23.010 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:30:23.016 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:30:23.111 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:30:23.133 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:23.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:23.134 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:30:23.236 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:30:23.276 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:23.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:23.277 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:30:23.354 INFO:teuthology.orchestra.run.smithi031.stderr:set 6.6 pg_upmap mapping to [5,1,7] 2024-08-19T14:30:23.378 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:30:23.422 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:23.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:23.423 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:30:23.478 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:23 smithi031 ceph-mon[16336]: pgmap v1262: 47 pgs: 8 active+recovering+undersized+remapped, 39 active+clean; 6.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 637 op/s; 23051/305457 objects misplaced (7.546%); 16 MiB/s, 254 objects/s recovering 2024-08-19T14:30:23.478 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:23 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1502375816' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.6", "id": [5, 1, 7]}]: dispatch 2024-08-19T14:30:23.478 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:23 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.6", "id": [5, 1, 7]}]: dispatch 2024-08-19T14:30:23.524 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:30:23.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:23.562 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:23.562 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:30:23.615 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:30:23.652 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:23 smithi045 ceph-mon[20143]: pgmap v1262: 47 pgs: 8 active+recovering+undersized+remapped, 39 active+clean; 6.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 637 op/s; 23051/305457 objects misplaced (7.546%); 16 MiB/s, 254 objects/s recovering 2024-08-19T14:30:23.652 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:23 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1502375816' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.6", "id": [5, 1, 7]}]: dispatch 2024-08-19T14:30:23.652 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:23 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.6", "id": [5, 1, 7]}]: dispatch 2024-08-19T14:30:23.663 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:30:23.679 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-08-19T14:30:23.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:23.707 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:23.707 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:30:23.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:23 smithi032 ceph-mon[20004]: pgmap v1262: 47 pgs: 8 active+recovering+undersized+remapped, 39 active+clean; 6.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 637 op/s; 23051/305457 objects misplaced (7.546%); 16 MiB/s, 254 objects/s recovering 2024-08-19T14:30:23.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:23 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1502375816' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.6", "id": [5, 1, 7]}]: dispatch 2024-08-19T14:30:23.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:23 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.6", "id": [5, 1, 7]}]: dispatch 2024-08-19T14:30:23.808 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:30:23.829 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:23.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:23.830 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:30:23.931 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:30:23.954 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:30:23.954 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:23.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:23.955 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:30:24.056 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:30:24.078 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:24.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:24.079 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:30:24.179 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:30:24.201 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:24.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:24.202 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:30:24.303 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:30:24.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:24.326 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:24.326 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:30:24.352 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is set 2024-08-19T14:30:24.428 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:30:24.457 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:24.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:24.459 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:30:24.481 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:30:24.482 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:30:24.560 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:30:24.585 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:24.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:24.586 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:30:24.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:24 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "6.6", "id": [5, 1, 7]}]': finished 2024-08-19T14:30:24.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:24 smithi031 ceph-mon[16336]: osdmap e549: 12 total, 12 up, 8 in 2024-08-19T14:30:24.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:24 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/4182883719' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:30:24.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:24 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:30:24.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:24 smithi031 ceph-mon[16336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:30:24.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:24 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:30:24.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:24 smithi031 ceph-mon[16336]: osdmap e550: 12 total, 12 up, 8 in 2024-08-19T14:30:24.686 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:30:24.716 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:24.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:24.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:30:24.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:24 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "6.6", "id": [5, 1, 7]}]': finished 2024-08-19T14:30:24.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:24 smithi032 ceph-mon[20004]: osdmap e549: 12 total, 12 up, 8 in 2024-08-19T14:30:24.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:24 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/4182883719' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:30:24.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:24 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:30:24.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:24 smithi032 ceph-mon[20004]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:30:24.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:24 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:30:24.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:24 smithi032 ceph-mon[20004]: osdmap e550: 12 total, 12 up, 8 in 2024-08-19T14:30:24.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:24 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "6.6", "id": [5, 1, 7]}]': finished 2024-08-19T14:30:24.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:24 smithi045 ceph-mon[20143]: osdmap e549: 12 total, 12 up, 8 in 2024-08-19T14:30:24.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:24 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/4182883719' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:30:24.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:24 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:30:24.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:24 smithi045 ceph-mon[20143]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:30:24.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:24 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:30:24.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:24 smithi045 ceph-mon[20143]: osdmap e550: 12 total, 12 up, 8 in 2024-08-19T14:30:24.818 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:30:24.845 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:24.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:24.847 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:30:24.948 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:30:24.972 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:24.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:24.973 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:30:25.074 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:30:25.101 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:25.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:25.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:30:25.199 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-08-19T14:30:25.203 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:30:25.228 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:25.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:25.230 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:30:25.331 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:30:25.359 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:25.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:25.360 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:30:25.461 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:30:25.464 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:30:25.484 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:25.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:25.485 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:30:25.497 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:25 smithi031 ceph-mon[16336]: pgmap v1264: 47 pgs: 1 unknown, 8 active+recovering+undersized+remapped, 38 active+clean; 5.7 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 22253/280203 objects misplaced (7.942%); 19 MiB/s, 296 objects/s recovering 2024-08-19T14:30:25.497 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:25 smithi031 ceph-mon[16336]: osdmap e551: 12 total, 12 up, 8 in 2024-08-19T14:30:25.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:25 smithi045 ceph-mon[20143]: pgmap v1264: 47 pgs: 1 unknown, 8 active+recovering+undersized+remapped, 38 active+clean; 5.7 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 22253/280203 objects misplaced (7.942%); 19 MiB/s, 296 objects/s recovering 2024-08-19T14:30:25.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:25 smithi045 ceph-mon[20143]: osdmap e551: 12 total, 12 up, 8 in 2024-08-19T14:30:25.586 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:30:25.612 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:25.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:25.613 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:30:25.714 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:30:25.739 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:25.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:25.740 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:30:25.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:25 smithi032 ceph-mon[20004]: pgmap v1264: 47 pgs: 1 unknown, 8 active+recovering+undersized+remapped, 38 active+clean; 5.7 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 22253/280203 objects misplaced (7.942%); 19 MiB/s, 296 objects/s recovering 2024-08-19T14:30:25.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:25 smithi032 ceph-mon[20004]: osdmap e551: 12 total, 12 up, 8 in 2024-08-19T14:30:25.841 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:30:25.876 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:25.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:25.877 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:30:25.978 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:30:26.002 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:26.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:26.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:30:26.103 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:30:26.124 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:26.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:26.125 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:30:26.226 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:30:26.248 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:26.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:26.249 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:30:26.350 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:30:26.401 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:26.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:26.402 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:30:26.498 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:30:26.498 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:30:26.502 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:30:26.525 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:26.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:26.526 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:30:26.626 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:30:26.662 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:26.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:26.663 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:30:26.764 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:30:26.768 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:30:26.821 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:26.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:26.823 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:30:26.923 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:30:26.965 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:26.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:26.967 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:30:27.067 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:30:27.094 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:27.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:27.095 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:30:27.196 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:30:27.234 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-08-19T14:30:27.240 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:27.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:27.241 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:30:27.342 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:30:27.364 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:27.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:27.365 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:30:27.466 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:30:27.492 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:30:27.492 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:30:27.504 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:27.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:27.505 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:30:27.524 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:30:27.524 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:30:27.524 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:27 smithi031 ceph-mon[16336]: pgmap v1267: 47 pgs: 1 unknown, 8 active+recovering+undersized+remapped, 38 active+clean; 5.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 21606/278301 objects misplaced (7.764%); 19 MiB/s, 304 objects/s recovering 2024-08-19T14:30:27.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:27 smithi045 ceph-mon[20143]: pgmap v1267: 47 pgs: 1 unknown, 8 active+recovering+undersized+remapped, 38 active+clean; 5.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 21606/278301 objects misplaced (7.764%); 19 MiB/s, 304 objects/s recovering 2024-08-19T14:30:27.555 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:30:27.555 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:30:27.607 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:30:27.611 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:30:27.612 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:30:27.633 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:27.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:27.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:30:27.643 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:30:27.643 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:30:27.680 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:30:27.680 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:30:27.712 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:30:27.712 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:30:27.717 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:30:27.734 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:30:27.755 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:30:27.755 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:30:27.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:27 smithi032 ceph-mon[20004]: pgmap v1267: 47 pgs: 1 unknown, 8 active+recovering+undersized+remapped, 38 active+clean; 5.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 21606/278301 objects misplaced (7.764%); 19 MiB/s, 304 objects/s recovering 2024-08-19T14:30:27.792 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:27.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:27.793 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:30:27.795 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:30:27.795 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:30:27.826 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:30:27.827 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:30:27.859 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:30:27.859 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:30:27.892 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:30:27.893 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:30:27.894 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:30:27.916 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:27.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:27.917 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:30:28.018 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:30:28.039 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:28.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:28.040 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:30:28.142 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:30:28.163 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:28.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:28.164 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:30:28.265 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:30:28.287 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:28.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:28.288 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:30:28.389 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:30:28.438 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:28.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:28.439 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:30:28.540 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:30:28.562 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:28.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:28.563 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:30:28.627 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-19T14:30:28.663 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:30:28.676 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:28 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/92449676' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:30:28.676 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:28 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:30:28.676 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:28 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:30:28.688 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:28.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:28.689 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:30:28.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:28 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/92449676' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:30:28.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:28 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:30:28.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:28 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:30:28.790 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:30:28.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:28 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/92449676' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:30:28.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:28 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:30:28.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:28 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:30:28.812 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:28.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:28.813 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:30:28.914 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:30:28.936 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:28.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:28.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:30:29.038 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:30:29.084 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:29.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:29.085 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:30:29.186 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:30:29.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:29.214 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:29.214 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:30:29.315 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:30:29.351 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:29.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:29.352 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:30:29.453 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:30:29.479 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:29.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:29.480 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:30:29.582 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:30:29.612 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:29.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:29.613 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:30:29.714 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:30:29.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:29 smithi032 ceph-mon[20004]: pgmap v1268: 47 pgs: 1 unknown, 8 active+recovering+undersized+remapped, 38 active+clean; 5.7 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 21606/278301 objects misplaced (7.764%); 15 MiB/s, 240 objects/s recovering 2024-08-19T14:30:29.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:29 smithi032 ceph-mon[20004]: osdmap e552: 12 total, 12 up, 8 in 2024-08-19T14:30:29.760 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:29.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:29.761 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:30:29.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:29 smithi045 ceph-mon[20143]: pgmap v1268: 47 pgs: 1 unknown, 8 active+recovering+undersized+remapped, 38 active+clean; 5.7 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 21606/278301 objects misplaced (7.764%); 15 MiB/s, 240 objects/s recovering 2024-08-19T14:30:29.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:29 smithi045 ceph-mon[20143]: osdmap e552: 12 total, 12 up, 8 in 2024-08-19T14:30:29.862 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:30:29.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:29 smithi031 ceph-mon[16336]: pgmap v1268: 47 pgs: 1 unknown, 8 active+recovering+undersized+remapped, 38 active+clean; 5.7 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 21606/278301 objects misplaced (7.764%); 15 MiB/s, 240 objects/s recovering 2024-08-19T14:30:29.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:29 smithi031 ceph-mon[16336]: osdmap e552: 12 total, 12 up, 8 in 2024-08-19T14:30:29.888 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:29.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:29.890 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:30:29.991 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:30:30.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:30.043 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:30.043 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:30:30.144 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:30:30.167 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:30.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:30.169 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:30:30.179 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 7, 8, 2] out_osds: [10, 6, 9, 11] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 10] 2024-08-19T14:30:30.179 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-19T14:30:30.180 INFO:tasks.thrashosds.thrasher:Adding osd 11 2024-08-19T14:30:30.180 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd in 11 2024-08-19T14:30:30.269 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:30:30.316 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:30.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:30.317 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:30:30.418 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:30:30.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:30.440 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:30.440 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:30:30.542 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:30:30.586 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:30.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:30.588 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:30:30.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:30 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 78/292938 objects degraded (0.027%), 1 pg degraded, 8 pgs undersized (PG_DEGRADED) 2024-08-19T14:30:30.688 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:30:30.715 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:30.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:30.716 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:30:30.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:30 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 78/292938 objects degraded (0.027%), 1 pg degraded, 8 pgs undersized (PG_DEGRADED) 2024-08-19T14:30:30.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:30 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 78/292938 objects degraded (0.027%), 1 pg degraded, 8 pgs undersized (PG_DEGRADED) 2024-08-19T14:30:30.816 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:30:30.838 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:30.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:30.839 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:30:30.940 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:30:30.966 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:30.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:30.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:30:31.068 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:30:31.089 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:31.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:31.090 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:30:31.191 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:30:31.234 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:31.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:31.235 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:30:31.336 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:30:31.389 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:31.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:31.391 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:30:31.492 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:30:31.543 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:31.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:31.544 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:30:31.645 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:30:31.666 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:31.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:31.667 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:30:31.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:31 smithi032 ceph-mon[20004]: pgmap v1270: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 38 active+clean; 6.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 70 MiB/s wr, 1.53k op/s; 78/292938 objects degraded (0.027%); 31799/292938 objects misplaced (10.855%); 28 MiB/s, 453 objects/s recovering 2024-08-19T14:30:31.768 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:30:31.794 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:31.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:31.796 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:30:31.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:31 smithi045 ceph-mon[20143]: pgmap v1270: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 38 active+clean; 6.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 70 MiB/s wr, 1.53k op/s; 78/292938 objects degraded (0.027%); 31799/292938 objects misplaced (10.855%); 28 MiB/s, 453 objects/s recovering 2024-08-19T14:30:31.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:31 smithi031 ceph-mon[16336]: pgmap v1270: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 38 active+clean; 6.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 70 MiB/s wr, 1.53k op/s; 78/292938 objects degraded (0.027%); 31799/292938 objects misplaced (10.855%); 28 MiB/s, 453 objects/s recovering 2024-08-19T14:30:31.896 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:30:31.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:31.941 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:31.942 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:30:32.043 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:30:32.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:32.064 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:32.064 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:30:32.165 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:30:32.186 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:32.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:32.187 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:30:32.288 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:30:32.316 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:32.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:32.318 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:30:32.419 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:30:32.445 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:32.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:32.447 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:30:32.548 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:30:32.596 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:32.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:32.597 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:30:32.698 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:30:32.719 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:32.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:32.721 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:30:32.822 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:30:32.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:32.843 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:32.845 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:30:32.946 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:30:32.973 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:32.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:32.974 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:30:33.075 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:30:33.097 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:33.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:33.098 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:30:33.199 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:30:33.220 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:33.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:33.221 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:30:33.323 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:30:33.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:33.364 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:33.364 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:30:33.465 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:30:33.486 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:33.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:33.488 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:30:33.589 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:30:33.615 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:33.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:33.616 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:30:33.717 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:30:33.746 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:33.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:33.747 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:30:33.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:33 smithi032 ceph-mon[20004]: pgmap v1271: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 38 active+clean; 6.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 54 MiB/s wr, 1.14k op/s; 78/292527 objects degraded (0.027%); 31468/292527 objects misplaced (10.757%); 23 MiB/s, 361 objects/s recovering 2024-08-19T14:30:33.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:33 smithi045 ceph-mon[20143]: pgmap v1271: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 38 active+clean; 6.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 54 MiB/s wr, 1.14k op/s; 78/292527 objects degraded (0.027%); 31468/292527 objects misplaced (10.757%); 23 MiB/s, 361 objects/s recovering 2024-08-19T14:30:33.847 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:30:33.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:33 smithi031 ceph-mon[16336]: pgmap v1271: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 38 active+clean; 6.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 54 MiB/s wr, 1.14k op/s; 78/292527 objects degraded (0.027%); 31468/292527 objects misplaced (10.757%); 23 MiB/s, 361 objects/s recovering 2024-08-19T14:30:33.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:33.889 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:33.889 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:30:33.990 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:30:34.011 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:34.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:34.012 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:30:34.113 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:30:34.158 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:34.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:34.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:30:34.260 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:30:34.282 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:34.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:34.285 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:30:34.386 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:30:34.414 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:34.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:34.415 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:30:34.516 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:30:34.562 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:34.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:34.564 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:30:34.665 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:30:34.708 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:34.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:34.709 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:30:34.810 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:30:34.831 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:34.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:34.832 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:30:34.933 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:30:34.982 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:34.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:34.984 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:30:35.085 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:30:35.128 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:35.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:35.129 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:30:35.230 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:30:35.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:35.279 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:35.279 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:30:35.380 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:30:35.401 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:35.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:35.402 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:30:35.503 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:30:35.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:35.526 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:35.527 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:30:35.628 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:30:35.649 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:35.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:35.650 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:30:35.751 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:30:35.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:35 smithi032 ceph-mon[20004]: pgmap v1272: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 38 active+clean; 5.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 48 MiB/s wr, 1.14k op/s; 78/288909 objects degraded (0.027%); 30552/288909 objects misplaced (10.575%); 24 MiB/s, 381 objects/s recovering 2024-08-19T14:30:35.792 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:35.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:35.793 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:30:35.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:35 smithi045 ceph-mon[20143]: pgmap v1272: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 38 active+clean; 5.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 48 MiB/s wr, 1.14k op/s; 78/288909 objects degraded (0.027%); 30552/288909 objects misplaced (10.575%); 24 MiB/s, 381 objects/s recovering 2024-08-19T14:30:35.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:35 smithi031 ceph-mon[16336]: pgmap v1272: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 38 active+clean; 5.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 48 MiB/s wr, 1.14k op/s; 78/288909 objects degraded (0.027%); 30552/288909 objects misplaced (10.575%); 24 MiB/s, 381 objects/s recovering 2024-08-19T14:30:35.893 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:30:35.938 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:35.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:35.939 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:30:36.039 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:30:36.084 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:36.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:36.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:30:36.186 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:30:36.213 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:36.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:36.214 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:30:36.315 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:30:36.368 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:36.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:36.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:30:36.471 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:30:36.495 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:36.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:36.496 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:30:36.597 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:30:36.624 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:36.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:36.625 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:30:36.726 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:30:36.751 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:36.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:36.752 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:30:36.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:36 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 178/285072 objects degraded (0.062%), 1 pg degraded, 8 pgs undersized (PG_DEGRADED) 2024-08-19T14:30:36.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:36 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 178/285072 objects degraded (0.062%), 1 pg degraded, 8 pgs undersized (PG_DEGRADED) 2024-08-19T14:30:36.853 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:30:36.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:36 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 178/285072 objects degraded (0.062%), 1 pg degraded, 8 pgs undersized (PG_DEGRADED) 2024-08-19T14:30:36.884 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:36.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:36.885 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:30:36.987 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:30:37.009 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:37.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:37.010 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:30:37.111 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:30:37.145 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:37.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:37.146 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:30:37.148 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:30:37.148 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [ 2024-08-19T14:30:37.148 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:37.148 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:120858 6.0 6:075e3448:::benchmark_data_smithi045_97937_object13585:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e552)", 2024-08-19T14:30:37.148 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:30:36.374886+0000", 2024-08-19T14:30:37.148 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.034632297999999999, 2024-08-19T14:30:37.148 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.034663869, 2024-08-19T14:30:37.148 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:30:37.148 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:30:37.148 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "waiting for sub ops", 2024-08-19T14:30:37.148 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:30:37.148 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:30:37.149 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:30:37.149 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 120858 2024-08-19T14:30:37.149 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:37.149 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:30:37.149 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:37.149 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:30:37.149 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:36.374886+0000", 2024-08-19T14:30:37.149 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:30:37.149 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:37.149 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:37.149 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:30:37.149 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:36.374886+0000", 2024-08-19T14:30:37.149 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:30:37.150 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:37.150 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:37.150 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:30:37.150 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:36.374888+0000", 2024-08-19T14:30:37.150 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.905e-06 2024-08-19T14:30:37.151 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:37.151 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:37.151 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:30:37.151 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:36.374899+0000", 2024-08-19T14:30:37.151 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.066e-05 2024-08-19T14:30:37.151 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:37.151 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:37.151 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:30:37.151 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:36.374913+0000", 2024-08-19T14:30:37.151 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3551e-05 2024-08-19T14:30:37.152 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:37.152 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:37.152 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:30:37.152 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:36.374922+0000", 2024-08-19T14:30:37.152 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.1179999999999998e-06 2024-08-19T14:30:37.152 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:37.152 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:37.152 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:30:37.152 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:36.391194+0000", 2024-08-19T14:30:37.152 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.016272003 2024-08-19T14:30:37.152 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:37.152 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:37.152 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:30:37.152 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:36.391513+0000", 2024-08-19T14:30:37.153 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00031883299999999997 2024-08-19T14:30:37.153 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:37.153 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:37.153 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 2,3", 2024-08-19T14:30:37.153 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:36.391867+0000", 2024-08-19T14:30:37.153 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00035455399999999998 2024-08-19T14:30:37.153 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:37.153 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:37.153 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:30:37.153 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:36.403916+0000", 2024-08-19T14:30:37.153 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.012048497 2024-08-19T14:30:37.153 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:30:37.153 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:30:37.154 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:30:37.154 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:37.154 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:37.154 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:120859 6.9 6:916842e1:::benchmark_data_smithi045_97937_object13586:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e552)", 2024-08-19T14:30:37.154 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:30:36.374942+0000", 2024-08-19T14:30:37.154 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.034576702000000001, 2024-08-19T14:30:37.154 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.034690662999999997, 2024-08-19T14:30:37.154 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:30:37.154 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:30:37.154 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "waiting for sub ops", 2024-08-19T14:30:37.154 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:30:37.154 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:30:37.155 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:30:37.155 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 120859 2024-08-19T14:30:37.155 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:37.155 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:30:37.155 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:37.155 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:30:37.155 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:36.374942+0000", 2024-08-19T14:30:37.155 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:30:37.155 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:37.155 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:37.155 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:30:37.155 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:36.374942+0000", 2024-08-19T14:30:37.155 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:30:37.156 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:37.156 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:37.156 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:30:37.156 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:36.374943+0000", 2024-08-19T14:30:37.156 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7599999999999997e-07 2024-08-19T14:30:37.156 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:37.156 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:37.156 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:30:37.156 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:36.374946+0000", 2024-08-19T14:30:37.156 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.0649999999999999e-06 2024-08-19T14:30:37.156 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:37.156 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:37.156 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:30:37.156 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:36.374952+0000", 2024-08-19T14:30:37.156 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.8310000000000003e-06 2024-08-19T14:30:37.157 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:37.157 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:37.157 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:30:37.157 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:36.374958+0000", 2024-08-19T14:30:37.157 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.5269999999999998e-06 2024-08-19T14:30:37.157 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:37.157 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:37.157 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:30:37.157 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:36.377045+0000", 2024-08-19T14:30:37.157 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.002086662 2024-08-19T14:30:37.157 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:37.157 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:37.157 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:30:37.157 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:36.377372+0000", 2024-08-19T14:30:37.158 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000327824 2024-08-19T14:30:37.158 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:37.158 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:37.158 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 2,8", 2024-08-19T14:30:37.158 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:36.377733+0000", 2024-08-19T14:30:37.158 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00036007200000000002 2024-08-19T14:30:37.158 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:37.158 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:37.158 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:30:37.158 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:36.382123+0000", 2024-08-19T14:30:37.158 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.004390494 2024-08-19T14:30:37.158 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:37.158 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:37.159 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:30:37.159 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:36.385097+0000", 2024-08-19T14:30:37.159 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0029742739999999998 2024-08-19T14:30:37.159 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:30:37.159 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:30:37.159 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:30:37.159 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:37.159 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:37.159 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "MOSDPGPushReply(6.a 552/525 [PushReplyOp(6:59e18e0d:::benchmark_data_smithi045_97937_object27733:head)])", 2024-08-19T14:30:37.159 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:30:36.388573+0000", 2024-08-19T14:30:37.159 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.020946244999999999, 2024-08-19T14:30:37.159 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.021135503, 2024-08-19T14:30:37.159 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:30:37.160 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:30:37.160 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:30:37.160 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:30:37.160 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:37.160 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:30:37.160 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:36.388573+0000", 2024-08-19T14:30:37.160 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:30:37.160 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:37.160 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:37.160 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:30:37.160 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:36.388573+0000", 2024-08-19T14:30:37.160 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:30:37.160 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:37.160 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:37.161 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:30:37.161 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:36.388573+0000", 2024-08-19T14:30:37.161 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3799999999999999e-07 2024-08-19T14:30:37.161 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:37.161 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:37.161 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:30:37.161 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:36.388576+0000", 2024-08-19T14:30:37.161 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.681e-06 2024-08-19T14:30:37.161 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:37.161 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:37.161 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:30:37.161 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:36.388582+0000", 2024-08-19T14:30:37.161 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.7710000000000003e-06 2024-08-19T14:30:37.161 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:37.162 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:37.162 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:30:37.162 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:36.388587+0000", 2024-08-19T14:30:37.162 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1189999999999996e-06 2024-08-19T14:30:37.162 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:37.162 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:37.162 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:30:37.162 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:36.408137+0000", 2024-08-19T14:30:37.162 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.019550088 2024-08-19T14:30:37.162 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:30:37.162 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:30:37.162 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:30:37.162 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:37.163 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:37.163 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.26191.0:120866 6.3 e552/510 6:c8429b34:::benchmark_data_smithi045_97937_object13593:head v 552'7595, mlcod=552'7592)", 2024-08-19T14:30:37.163 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:30:36.397363+0000", 2024-08-19T14:30:37.163 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.012155987, 2024-08-19T14:30:37.163 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.012395244999999999, 2024-08-19T14:30:37.163 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:30:37.163 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:30:37.163 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:30:37.163 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:30:37.163 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:37.163 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:30:37.163 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:36.397363+0000", 2024-08-19T14:30:37.163 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:30:37.164 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:37.164 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:37.164 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:30:37.164 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:36.397363+0000", 2024-08-19T14:30:37.164 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:30:37.164 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:37.164 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:37.164 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:30:37.164 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:36.397364+0000", 2024-08-19T14:30:37.164 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.4e-07 2024-08-19T14:30:37.164 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:37.164 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:37.164 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:30:37.164 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:36.397371+0000", 2024-08-19T14:30:37.165 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.2660000000000004e-06 2024-08-19T14:30:37.165 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:37.165 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:37.165 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:30:37.165 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:36.397384+0000", 2024-08-19T14:30:37.165 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3475999999999999e-05 2024-08-19T14:30:37.165 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:37.165 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:37.165 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:30:37.165 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:36.397392+0000", 2024-08-19T14:30:37.165 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.4780000000000002e-06 2024-08-19T14:30:37.165 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:37.165 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:37.166 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:30:37.166 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:36.403538+0000", 2024-08-19T14:30:37.166 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0061465199999999999 2024-08-19T14:30:37.166 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:37.166 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:37.166 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:30:37.166 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:36.403561+0000", 2024-08-19T14:30:37.166 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.2146000000000001e-05 2024-08-19T14:30:37.166 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:37.166 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:37.166 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:30:37.166 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:36.409717+0000", 2024-08-19T14:30:37.166 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0061565889999999996 2024-08-19T14:30:37.167 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:37.167 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:37.167 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:30:37.167 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:36.409763+0000", 2024-08-19T14:30:37.167 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.5401000000000001e-05 2024-08-19T14:30:37.167 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:30:37.167 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:30:37.167 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:30:37.167 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:37.167 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:37.167 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:120869 6.a 6.b85e1d4a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e552)", 2024-08-19T14:30:37.167 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:30:36.409473+0000", 2024-08-19T14:30:37.167 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 4.5895e-05, 2024-08-19T14:30:37.168 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00035559300000000002, 2024-08-19T14:30:37.168 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:30:37.168 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:30:37.168 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "queued for pg", 2024-08-19T14:30:37.168 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:30:37.168 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:30:37.168 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:30:37.168 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 120869 2024-08-19T14:30:37.168 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:37.168 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:30:37.168 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:37.168 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:30:37.169 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:36.409473+0000", 2024-08-19T14:30:37.169 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:30:37.169 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:37.169 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:37.169 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:30:37.169 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:36.409473+0000", 2024-08-19T14:30:37.169 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:30:37.169 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:37.169 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:37.169 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:30:37.169 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:36.409474+0000", 2024-08-19T14:30:37.169 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.099e-06 2024-08-19T14:30:37.169 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:37.170 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:37.170 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:30:37.170 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:36.409478+0000", 2024-08-19T14:30:37.170 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.6160000000000002e-06 2024-08-19T14:30:37.170 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:37.170 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:37.170 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:30:37.170 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:36.409489+0000", 2024-08-19T14:30:37.170 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1735e-05 2024-08-19T14:30:37.170 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:30:37.170 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:30:37.170 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:30:37.170 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:36.409496+0000", 2024-08-19T14:30:37.171 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.2480000000000001e-06 2024-08-19T14:30:37.171 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:30:37.171 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:30:37.171 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:30:37.171 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:30:37.171 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-08-19T14:30:37.171 INFO:teuthology.orchestra.run.smithi031.stdout: "num_ops": 5 2024-08-19T14:30:37.171 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:30:37.247 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:30:37.268 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:37.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:37.269 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:30:37.370 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:30:37.415 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:37.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:37.416 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:30:37.517 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:30:37.551 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:37.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:37.552 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:30:37.653 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:30:37.674 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:37.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:37.675 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:30:37.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:37 smithi032 ceph-mon[20004]: pgmap v1273: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 38 active+clean; 5.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 42 MiB/s wr, 1.12k op/s; 178/285072 objects degraded (0.062%); 29806/285072 objects misplaced (10.456%); 24 MiB/s, 391 objects/s recovering 2024-08-19T14:30:37.776 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:30:37.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:37 smithi045 ceph-mon[20143]: pgmap v1273: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 38 active+clean; 5.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 42 MiB/s wr, 1.12k op/s; 178/285072 objects degraded (0.062%); 29806/285072 objects misplaced (10.456%); 24 MiB/s, 391 objects/s recovering 2024-08-19T14:30:37.800 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:37.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:37.801 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:30:37.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:37 smithi031 ceph-mon[16336]: pgmap v1273: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 38 active+clean; 5.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 42 MiB/s wr, 1.12k op/s; 178/285072 objects degraded (0.062%); 29806/285072 objects misplaced (10.456%); 24 MiB/s, 391 objects/s recovering 2024-08-19T14:30:37.902 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:30:37.944 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:37.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:37.945 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:30:38.046 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:30:38.066 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:38.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:38.067 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:30:38.168 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:30:38.189 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:38.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:38.191 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:30:38.292 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:30:38.329 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:38.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:38.330 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:30:38.431 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:30:38.476 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:38.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:38.477 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:30:38.578 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:30:38.605 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:38.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:38.606 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:30:38.707 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:30:38.736 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:38.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:38.737 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:30:38.837 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:30:38.861 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:38.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:38.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:30:38.963 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:30:38.986 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:38.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:38.987 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:30:39.088 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:30:39.109 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:39.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:39.110 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:30:39.211 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:30:39.232 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:39.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:39.233 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:30:39.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:38 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:30:39.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:38 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:30:39.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:38 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:30:39.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:38 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:30:39.334 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:30:39.360 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:39.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:39.361 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:30:39.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:38 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:30:39.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:38 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:30:39.462 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:30:39.483 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:39.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:39.484 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:30:39.585 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:30:39.630 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:39.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:39.631 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:30:39.732 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:30:39.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:39.754 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:39.754 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:30:39.855 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:30:39.882 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:39.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:39.883 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:30:39.984 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:30:40.005 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:40.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:40.006 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:30:40.107 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:30:40.176 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:40.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:40.177 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:30:40.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:39 smithi032 ceph-mon[20004]: pgmap v1274: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 38 active+clean; 5.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 42 MiB/s wr, 1.12k op/s; 178/285072 objects degraded (0.062%); 29806/285072 objects misplaced (10.456%); 24 MiB/s, 391 objects/s recovering 2024-08-19T14:30:40.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:39 smithi032 ceph-mon[20004]: osdmap e553: 12 total, 12 up, 8 in 2024-08-19T14:30:40.278 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:30:40.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:39 smithi045 ceph-mon[20143]: pgmap v1274: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 38 active+clean; 5.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 42 MiB/s wr, 1.12k op/s; 178/285072 objects degraded (0.062%); 29806/285072 objects misplaced (10.456%); 24 MiB/s, 391 objects/s recovering 2024-08-19T14:30:40.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:39 smithi045 ceph-mon[20143]: osdmap e553: 12 total, 12 up, 8 in 2024-08-19T14:30:40.298 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:40.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:40.299 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:30:40.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:39 smithi031 ceph-mon[16336]: pgmap v1274: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 38 active+clean; 5.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 42 MiB/s wr, 1.12k op/s; 178/285072 objects degraded (0.062%); 29806/285072 objects misplaced (10.456%); 24 MiB/s, 391 objects/s recovering 2024-08-19T14:30:40.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:39 smithi031 ceph-mon[16336]: osdmap e553: 12 total, 12 up, 8 in 2024-08-19T14:30:40.401 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:30:40.422 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:40.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:40.423 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:30:40.524 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:30:40.551 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:40.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:40.552 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:30:40.652 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:30:40.680 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:40.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:40.682 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:30:40.783 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:30:40.805 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:40.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:40.806 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:30:40.907 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:30:40.931 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:40.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:40.932 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:30:41.033 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:30:41.060 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:41.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:41.061 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:30:41.162 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:30:41.184 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:41.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:41.185 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:30:41.286 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:30:41.307 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:41.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:41.308 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:30:41.409 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:30:41.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:41.451 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:41.451 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:30:41.552 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:30:41.578 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:41.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:41.579 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:30:41.680 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:30:41.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:41.726 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:41.727 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:30:41.828 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:30:41.854 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:41.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:41.855 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:30:41.956 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:30:41.982 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:41.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:41.983 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:30:42.083 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:30:42.130 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-08-19T14:30:42.165 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:30:42.167 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:42.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:42.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:30:42.180 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:30:42.241 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:41 smithi031 ceph-mon[16336]: pgmap v1276: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 38 active+clean; 5.7 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 497 op/s; 265/278013 objects degraded (0.095%); 28362/278013 objects misplaced (10.202%); 20 MiB/s, 313 objects/s recovering 2024-08-19T14:30:42.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:41 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:30:42.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:41 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:30:42.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:41 smithi031 ceph-mon[16336]: osdmap e554: 12 total, 12 up, 8 in 2024-08-19T14:30:42.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:41 smithi032 ceph-mon[20004]: pgmap v1276: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 38 active+clean; 5.7 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 497 op/s; 265/278013 objects degraded (0.095%); 28362/278013 objects misplaced (10.202%); 20 MiB/s, 313 objects/s recovering 2024-08-19T14:30:42.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:41 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:30:42.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:41 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:30:42.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:41 smithi032 ceph-mon[20004]: osdmap e554: 12 total, 12 up, 8 in 2024-08-19T14:30:42.275 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:30:42.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:41 smithi045 ceph-mon[20143]: pgmap v1276: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 38 active+clean; 5.7 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 497 op/s; 265/278013 objects degraded (0.095%); 28362/278013 objects misplaced (10.202%); 20 MiB/s, 313 objects/s recovering 2024-08-19T14:30:42.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:41 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:30:42.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:41 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:30:42.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:41 smithi045 ceph-mon[20143]: osdmap e554: 12 total, 12 up, 8 in 2024-08-19T14:30:42.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:42.341 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:42.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:30:42.442 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:30:42.465 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:42.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:42.466 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:30:42.567 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:30:42.593 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:42.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:42.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:30:42.695 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:30:42.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:42.735 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:42.735 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:30:42.836 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:30:42.880 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:42.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:42.881 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:30:42.982 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:30:43.037 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:43.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:43.038 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:30:43.087 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:42 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 265/278013 objects degraded (0.095%), 1 pg degraded, 8 pgs undersized (PG_DEGRADED) 2024-08-19T14:30:43.140 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:30:43.159 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:30:43.163 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:43.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:43.164 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:30:43.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:42 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 265/278013 objects degraded (0.095%), 1 pg degraded, 8 pgs undersized (PG_DEGRADED) 2024-08-19T14:30:43.265 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:30:43.289 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:43.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:43.290 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:30:43.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:42 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 265/278013 objects degraded (0.095%), 1 pg degraded, 8 pgs undersized (PG_DEGRADED) 2024-08-19T14:30:43.391 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:30:43.412 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:43.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:43.413 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:30:43.514 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:30:43.536 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:43.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:43.537 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:30:43.561 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:30:43.638 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:30:43.689 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:43.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:43.690 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:30:43.791 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:30:43.819 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:43.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:43.820 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:30:43.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:43 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mon-a[16332]: 2024-08-19T14:30:43.594+0000 7f2ff503e640 -1 mon.a@0(leader).osd e554 definitely_dead 0 2024-08-19T14:30:43.899 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is unset 2024-08-19T14:30:43.904 INFO:teuthology.orchestra.run.smithi031.stderr:marked in osd.11. 2024-08-19T14:30:43.921 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:30:43.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:43.991 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:43.991 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:30:44.092 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:30:44.133 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:44.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:44.134 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:30:44.216 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:43 smithi031 ceph-mon[16336]: pgmap v1278: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 38 active+clean; 5.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 468 op/s; 265/277653 objects degraded (0.095%); 28087/277653 objects misplaced (10.116%); 17 MiB/s, 270 objects/s recovering 2024-08-19T14:30:44.217 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:43 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:30:44.217 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:43 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:30:44.217 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:43 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/4000910551' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["11"]}]: dispatch 2024-08-19T14:30:44.217 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:43 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["11"]}]: dispatch 2024-08-19T14:30:44.217 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:43 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/928522482' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:30:44.235 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:30:44.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:44.256 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:44.256 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:30:44.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:43 smithi032 ceph-mon[20004]: pgmap v1278: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 38 active+clean; 5.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 468 op/s; 265/277653 objects degraded (0.095%); 28087/277653 objects misplaced (10.116%); 17 MiB/s, 270 objects/s recovering 2024-08-19T14:30:44.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:43 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:30:44.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:43 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:30:44.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:43 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/4000910551' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["11"]}]: dispatch 2024-08-19T14:30:44.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:43 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["11"]}]: dispatch 2024-08-19T14:30:44.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:43 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/928522482' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:30:44.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:43 smithi045 ceph-mon[20143]: pgmap v1278: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 38 active+clean; 5.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 468 op/s; 265/277653 objects degraded (0.095%); 28087/277653 objects misplaced (10.116%); 17 MiB/s, 270 objects/s recovering 2024-08-19T14:30:44.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:43 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:30:44.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:43 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:30:44.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:43 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/4000910551' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["11"]}]: dispatch 2024-08-19T14:30:44.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:43 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["11"]}]: dispatch 2024-08-19T14:30:44.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:43 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/928522482' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:30:44.358 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:30:44.378 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:44.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:44.379 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:30:44.480 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:30:44.513 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:44.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:44.514 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:30:44.615 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:30:44.636 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:44.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:44.637 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:30:44.738 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:30:44.759 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:44.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:44.760 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:30:44.861 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:30:44.882 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:44.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:44.883 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:30:44.984 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:30:45.017 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:45.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:45.018 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:30:45.119 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:30:45.144 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:45.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:45.145 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:30:45.246 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:30:45.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:44 smithi032 ceph-mon[20004]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:30:45.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:44 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["11"]}]': finished 2024-08-19T14:30:45.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:44 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/928522482' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:30:45.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:44 smithi032 ceph-mon[20004]: osdmap e555: 12 total, 12 up, 9 in 2024-08-19T14:30:45.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:44 smithi032 ceph-mon[20004]: osdmap e556: 12 total, 12 up, 9 in 2024-08-19T14:30:45.276 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:45.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:45.277 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:30:45.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:44 smithi045 ceph-mon[20143]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:30:45.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:44 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["11"]}]': finished 2024-08-19T14:30:45.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:44 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/928522482' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:30:45.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:44 smithi045 ceph-mon[20143]: osdmap e555: 12 total, 12 up, 9 in 2024-08-19T14:30:45.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:44 smithi045 ceph-mon[20143]: osdmap e556: 12 total, 12 up, 9 in 2024-08-19T14:30:45.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:44 smithi031 ceph-mon[16336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:30:45.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:44 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["11"]}]': finished 2024-08-19T14:30:45.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:44 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/928522482' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:30:45.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:44 smithi031 ceph-mon[16336]: osdmap e555: 12 total, 12 up, 9 in 2024-08-19T14:30:45.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:44 smithi031 ceph-mon[16336]: osdmap e556: 12 total, 12 up, 9 in 2024-08-19T14:30:45.378 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:30:45.400 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:45.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:45.401 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:30:45.502 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:30:45.534 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:45.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:45.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:30:45.636 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:30:45.657 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:45.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:45.658 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:30:45.758 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:30:45.780 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:45.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:45.781 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:30:45.882 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:30:45.909 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:45.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:45.910 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:30:46.010 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:30:46.032 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:46.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:46.034 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:30:46.135 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:30:46.157 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:46.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:46.158 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:30:46.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:45 smithi032 ceph-mon[20004]: pgmap v1280: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 38 active+clean; 5.6 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 624 op/s; 265/273819 objects degraded (0.097%); 27114/273819 objects misplaced (9.902%); 26 MiB/s, 423 objects/s recovering 2024-08-19T14:30:46.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:45 smithi032 ceph-mon[20004]: osdmap e557: 12 total, 12 up, 9 in 2024-08-19T14:30:46.259 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:30:46.292 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:46.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:46.293 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:30:46.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:45 smithi045 ceph-mon[20143]: pgmap v1280: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 38 active+clean; 5.6 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 624 op/s; 265/273819 objects degraded (0.097%); 27114/273819 objects misplaced (9.902%); 26 MiB/s, 423 objects/s recovering 2024-08-19T14:30:46.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:45 smithi045 ceph-mon[20143]: osdmap e557: 12 total, 12 up, 9 in 2024-08-19T14:30:46.344 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:30:46.344 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-08-19T14:30:46.344 INFO:teuthology.orchestra.run.smithi031.stdout: "complaint_time": 30, 2024-08-19T14:30:46.344 INFO:teuthology.orchestra.run.smithi031.stdout: "num_blocked_ops": 0 2024-08-19T14:30:46.344 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:30:46.346 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:45 smithi031 ceph-mon[16336]: pgmap v1280: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 38 active+clean; 5.6 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 624 op/s; 265/273819 objects degraded (0.097%); 27114/273819 objects misplaced (9.902%); 26 MiB/s, 423 objects/s recovering 2024-08-19T14:30:46.346 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:45 smithi031 ceph-mon[16336]: osdmap e557: 12 total, 12 up, 9 in 2024-08-19T14:30:46.394 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:30:46.434 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:46.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:46.435 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:30:46.536 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:30:46.558 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:46.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:46.559 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:30:46.660 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:30:46.681 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:46.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:46.682 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:30:46.783 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:30:46.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:46.827 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:46.827 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:30:46.928 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:30:46.949 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:46.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:46.950 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:30:47.051 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:30:47.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:47.082 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:47.082 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:30:47.183 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:30:47.220 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:47.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:47.221 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:30:47.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:46 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:30:47.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:46 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:30:47.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:46 smithi032 ceph-mon[20004]: osdmap e558: 12 total, 12 up, 9 in 2024-08-19T14:30:47.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:46 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:30:47.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:46 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:30:47.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:46 smithi045 ceph-mon[20143]: osdmap e558: 12 total, 12 up, 9 in 2024-08-19T14:30:47.322 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:30:47.349 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:47.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:47.350 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:30:47.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:46 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:30:47.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:46 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:30:47.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:46 smithi031 ceph-mon[16336]: osdmap e558: 12 total, 12 up, 9 in 2024-08-19T14:30:47.451 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:30:47.472 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:47.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:47.473 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:30:47.574 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:30:47.604 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:47.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:47.605 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:30:47.706 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:30:47.732 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:47.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:47.733 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:30:47.834 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:30:47.871 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:47.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:47.872 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:30:47.973 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:30:48.004 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:48.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:48.005 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:30:48.106 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:30:48.133 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:48.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:48.134 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:30:48.235 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:30:48.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:47 smithi032 ceph-mon[20004]: pgmap v1283: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 1 remapped+peering, 2 unknown, 8 active+recovering+undersized+remapped, 34 active+clean; 5.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 483 op/s; 352/270153 objects degraded (0.130%); 26272/270153 objects misplaced (9.725%); 20 MiB/s, 323 objects/s recovering 2024-08-19T14:30:48.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:47 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 352/270153 objects degraded (0.130%), 1 pg degraded, 8 pgs undersized (PG_DEGRADED) 2024-08-19T14:30:48.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:47 smithi032 ceph-mon[20004]: osdmap e559: 12 total, 12 up, 9 in 2024-08-19T14:30:48.270 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:48.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:48.272 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:30:48.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:47 smithi045 ceph-mon[20143]: pgmap v1283: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 1 remapped+peering, 2 unknown, 8 active+recovering+undersized+remapped, 34 active+clean; 5.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 483 op/s; 352/270153 objects degraded (0.130%); 26272/270153 objects misplaced (9.725%); 20 MiB/s, 323 objects/s recovering 2024-08-19T14:30:48.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:47 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 352/270153 objects degraded (0.130%), 1 pg degraded, 8 pgs undersized (PG_DEGRADED) 2024-08-19T14:30:48.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:47 smithi045 ceph-mon[20143]: osdmap e559: 12 total, 12 up, 9 in 2024-08-19T14:30:48.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:47 smithi031 ceph-mon[16336]: pgmap v1283: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 1 remapped+peering, 2 unknown, 8 active+recovering+undersized+remapped, 34 active+clean; 5.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 483 op/s; 352/270153 objects degraded (0.130%); 26272/270153 objects misplaced (9.725%); 20 MiB/s, 323 objects/s recovering 2024-08-19T14:30:48.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:47 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 352/270153 objects degraded (0.130%), 1 pg degraded, 8 pgs undersized (PG_DEGRADED) 2024-08-19T14:30:48.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:47 smithi031 ceph-mon[16336]: osdmap e559: 12 total, 12 up, 9 in 2024-08-19T14:30:48.372 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:30:48.404 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:48.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:48.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:30:48.506 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:30:48.547 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:48.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:48.548 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:30:48.649 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:30:48.671 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:48.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:48.671 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:30:48.772 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:30:48.800 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:48.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:48.801 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:30:48.901 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:30:48.943 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:48.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:48.944 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:30:49.045 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:30:49.067 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:49.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:49.068 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:30:49.168 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:30:49.216 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:49.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:49.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:30:49.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:48 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:30:49.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:48 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:30:49.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:48 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:30:49.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:48 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:30:49.317 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:30:49.355 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:49.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:49.356 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:30:49.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:48 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:30:49.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:48 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:30:49.456 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:30:49.496 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:49.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:49.497 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:30:49.598 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:30:49.620 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:49.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:49.621 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:30:49.721 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:30:49.744 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:49.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:49.745 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:30:49.846 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:30:49.895 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:49.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:49.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:30:49.996 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:30:50.018 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:50.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:50.019 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:30:50.120 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:30:50.163 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:50.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:50.164 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:30:50.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:49 smithi032 ceph-mon[20004]: pgmap v1286: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 1 remapped+peering, 2 unknown, 8 active+recovering+undersized+remapped, 34 active+clean; 5.5 GiB data, 24 GiB used, 781 GiB / 805 GiB avail; 0 B/s wr, 305 op/s; 352/270153 objects degraded (0.130%); 26272/270153 objects misplaced (9.725%); 11 MiB/s, 175 objects/s recovering 2024-08-19T14:30:50.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:49 smithi032 ceph-mon[20004]: osdmap e560: 12 total, 12 up, 9 in 2024-08-19T14:30:50.265 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:30:50.295 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:50.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:50.296 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:30:50.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:49 smithi045 ceph-mon[20143]: pgmap v1286: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 1 remapped+peering, 2 unknown, 8 active+recovering+undersized+remapped, 34 active+clean; 5.5 GiB data, 24 GiB used, 781 GiB / 805 GiB avail; 0 B/s wr, 305 op/s; 352/270153 objects degraded (0.130%); 26272/270153 objects misplaced (9.725%); 11 MiB/s, 175 objects/s recovering 2024-08-19T14:30:50.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:49 smithi045 ceph-mon[20143]: osdmap e560: 12 total, 12 up, 9 in 2024-08-19T14:30:50.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:49 smithi031 ceph-mon[16336]: pgmap v1286: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 1 remapped+peering, 2 unknown, 8 active+recovering+undersized+remapped, 34 active+clean; 5.5 GiB data, 24 GiB used, 781 GiB / 805 GiB avail; 0 B/s wr, 305 op/s; 352/270153 objects degraded (0.130%); 26272/270153 objects misplaced (9.725%); 11 MiB/s, 175 objects/s recovering 2024-08-19T14:30:50.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:49 smithi031 ceph-mon[16336]: osdmap e560: 12 total, 12 up, 9 in 2024-08-19T14:30:50.396 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:30:50.437 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:30:50.441 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:50.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:50.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:30:50.542 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:30:50.594 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:50.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:50.595 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:30:50.696 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:30:50.718 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:50.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:50.719 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:30:50.820 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:30:50.853 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:50.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:50.854 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:30:50.955 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:30:50.997 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:50.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:50.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:30:51.100 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:30:51.122 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:51.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:51.123 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:30:51.197 INFO:tasks.thrashosds.thrasher:Added osd 11 2024-08-19T14:30:51.223 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:30:51.267 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:51.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:51.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:30:51.369 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:30:51.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:51.408 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:51.408 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:30:51.509 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:30:51.540 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:51.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:51.541 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:30:51.643 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:30:51.654 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-08-19T14:30:51.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:51.692 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:51.693 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:30:51.794 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:30:51.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:51.869 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:51.870 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:30:51.971 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:30:51.976 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:51 smithi031 ceph-mon[16336]: pgmap v1288: 47 pgs: 3 active+recovering+undersized+degraded+remapped, 1 remapped+peering, 8 active+recovering+undersized+remapped, 35 active+clean; 5.4 GiB data, 24 GiB used, 781 GiB / 805 GiB avail; 0 B/s wr, 287 op/s; 389/266550 objects degraded (0.146%); 34827/266550 objects misplaced (13.066%); 40 MiB/s, 634 objects/s recovering 2024-08-19T14:30:52.059 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:52.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:52.060 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:30:52.161 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:30:52.184 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:52.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:52.185 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:30:52.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:51 smithi032 ceph-mon[20004]: pgmap v1288: 47 pgs: 3 active+recovering+undersized+degraded+remapped, 1 remapped+peering, 8 active+recovering+undersized+remapped, 35 active+clean; 5.4 GiB data, 24 GiB used, 781 GiB / 805 GiB avail; 0 B/s wr, 287 op/s; 389/266550 objects degraded (0.146%); 34827/266550 objects misplaced (13.066%); 40 MiB/s, 634 objects/s recovering 2024-08-19T14:30:52.286 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:30:52.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:51 smithi045 ceph-mon[20143]: pgmap v1288: 47 pgs: 3 active+recovering+undersized+degraded+remapped, 1 remapped+peering, 8 active+recovering+undersized+remapped, 35 active+clean; 5.4 GiB data, 24 GiB used, 781 GiB / 805 GiB avail; 0 B/s wr, 287 op/s; 389/266550 objects degraded (0.146%); 34827/266550 objects misplaced (13.066%); 40 MiB/s, 634 objects/s recovering 2024-08-19T14:30:52.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:52.329 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:52.329 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:30:52.430 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:30:52.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:52.455 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:52.455 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:30:52.556 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:30:52.594 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:52.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:52.595 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:30:52.690 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:30:52.696 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:30:52.755 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:52.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:52.756 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:30:52.857 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:30:52.889 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:52.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:52.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:30:52.980 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:30:52.980 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:30:52.991 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:30:53.017 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:53.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:53.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:30:53.019 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:30:53.019 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:30:53.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:52 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1559672917' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:30:53.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:52 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 389/266550 objects degraded (0.146%), 3 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-08-19T14:30:53.089 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:30:53.089 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:30:53.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:52 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1559672917' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:30:53.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:52 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 389/266550 objects degraded (0.146%), 3 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-08-19T14:30:53.118 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:30:53.128 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:30:53.128 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:30:53.163 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:30:53.164 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:30:53.164 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:53.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:53.165 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:30:53.196 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:30:53.197 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:30:53.239 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:30:53.240 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:30:53.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:52 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1559672917' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:30:53.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:52 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 389/266550 objects degraded (0.146%), 3 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-08-19T14:30:53.266 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:30:53.285 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:30:53.286 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:30:53.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:53.304 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:53.304 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:30:53.339 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:30:53.339 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:30:53.379 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:30:53.380 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:30:53.381 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:30:53.381 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:30:53.405 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:30:53.428 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:30:53.428 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:30:53.449 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:53.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:53.450 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:30:53.470 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:30:53.470 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:30:53.551 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:30:53.576 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:53.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:53.577 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:30:53.679 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:30:53.706 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:53.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:53.707 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:30:53.808 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:30:53.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:53.839 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:53.840 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:30:53.941 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:30:53.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:53.981 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:53.981 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:30:54.029 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-19T14:30:54.082 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:30:54.086 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:53 smithi031 ceph-mon[16336]: pgmap v1289: 47 pgs: 3 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 37 active+clean; 5.4 GiB data, 24 GiB used, 781 GiB / 805 GiB avail; 1023 B/s rd, 26 KiB/s wr, 218 op/s; 389/266259 objects degraded (0.146%); 34636/266259 objects misplaced (13.008%); 30 MiB/s, 473 objects/s recovering 2024-08-19T14:30:54.105 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:54.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:54.106 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:30:54.207 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:30:54.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:53 smithi032 ceph-mon[20004]: pgmap v1289: 47 pgs: 3 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 37 active+clean; 5.4 GiB data, 24 GiB used, 781 GiB / 805 GiB avail; 1023 B/s rd, 26 KiB/s wr, 218 op/s; 389/266259 objects degraded (0.146%); 34636/266259 objects misplaced (13.008%); 30 MiB/s, 473 objects/s recovering 2024-08-19T14:30:54.291 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:54.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:54.292 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:30:54.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:53 smithi045 ceph-mon[20143]: pgmap v1289: 47 pgs: 3 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 37 active+clean; 5.4 GiB data, 24 GiB used, 781 GiB / 805 GiB avail; 1023 B/s rd, 26 KiB/s wr, 218 op/s; 389/266259 objects degraded (0.146%); 34636/266259 objects misplaced (13.008%); 30 MiB/s, 473 objects/s recovering 2024-08-19T14:30:54.393 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:30:54.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:54.457 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:54.457 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:30:54.558 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:30:54.599 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:54.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:54.600 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:30:54.701 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:30:54.737 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:54.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:54.738 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:30:54.839 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:30:54.868 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:54.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:54.869 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:30:54.941 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-08-19T14:30:54.971 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:30:55.069 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:55.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:55.070 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:30:55.171 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:30:55.206 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:55.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:55.207 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:30:55.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:55 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:30:55.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:55 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:30:55.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:55 smithi032 ceph-mon[20004]: osdmap e561: 12 total, 12 up, 9 in 2024-08-19T14:30:55.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:55 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:30:55.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:55 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:30:55.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:55 smithi045 ceph-mon[20143]: osdmap e561: 12 total, 12 up, 9 in 2024-08-19T14:30:55.308 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:30:55.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:55.360 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:55.360 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:30:55.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:55 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:30:55.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:55 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:30:55.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:55 smithi031 ceph-mon[16336]: osdmap e561: 12 total, 12 up, 9 in 2024-08-19T14:30:55.461 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:30:55.488 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:55.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:55.489 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:30:55.590 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:30:55.617 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:55.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:55.618 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:30:55.719 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:30:55.745 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:55.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:55.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:30:55.848 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:30:55.876 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:55.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:55.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:30:55.978 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:30:56.007 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:56.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:56.008 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:30:56.109 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:30:56.132 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:56.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:56.133 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:30:56.198 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 7, 8, 2, 11] out_osds: [10, 6, 9] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 10] 2024-08-19T14:30:56.198 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-19T14:30:56.198 INFO:tasks.thrashosds.thrasher:Killing osd 2, live_osds are [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 10] 2024-08-19T14:30:56.198 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-08-19T14:30:56.199 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968@osd.2 2024-08-19T14:30:56.235 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:30:56.256 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:56.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:56.257 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:30:56.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:56 smithi032 ceph-mon[20004]: pgmap v1291: 47 pgs: 3 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 37 active+clean; 5.3 GiB data, 24 GiB used, 781 GiB / 805 GiB avail; 995 B/s rd, 25 KiB/s wr, 418 op/s; 389/262464 objects degraded (0.148%); 33545/262464 objects misplaced (12.781%); 40 MiB/s, 637 objects/s recovering 2024-08-19T14:30:56.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:56 smithi032 ceph-mon[20004]: osdmap e562: 12 total, 12 up, 9 in 2024-08-19T14:30:56.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:56 smithi045 ceph-mon[20143]: pgmap v1291: 47 pgs: 3 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 37 active+clean; 5.3 GiB data, 24 GiB used, 781 GiB / 805 GiB avail; 995 B/s rd, 25 KiB/s wr, 418 op/s; 389/262464 objects degraded (0.148%); 33545/262464 objects misplaced (12.781%); 40 MiB/s, 637 objects/s recovering 2024-08-19T14:30:56.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:56 smithi045 ceph-mon[20143]: osdmap e562: 12 total, 12 up, 9 in 2024-08-19T14:30:56.358 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:30:56.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:56 smithi031 ceph-mon[16336]: pgmap v1291: 47 pgs: 3 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 37 active+clean; 5.3 GiB data, 24 GiB used, 781 GiB / 805 GiB avail; 995 B/s rd, 25 KiB/s wr, 418 op/s; 389/262464 objects degraded (0.148%); 33545/262464 objects misplaced (12.781%); 40 MiB/s, 637 objects/s recovering 2024-08-19T14:30:56.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:56 smithi031 ceph-mon[16336]: osdmap e562: 12 total, 12 up, 9 in 2024-08-19T14:30:56.367 INFO:journalctl@ceph.osd.2.smithi031.stdout:Aug 19 14:30:56 smithi031 systemd[1]: Stopping Ceph osd.2 for ed7d1c6e-5e33-11ef-bccf-c7b262605968... 2024-08-19T14:30:56.393 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:56.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:56.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:30:56.495 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:30:56.517 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:56.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:56.518 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:30:56.619 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:30:56.648 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:56.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:56.650 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:30:56.751 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:30:56.771 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:56.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:56.772 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:30:56.873 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:30:56.895 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:56.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:56.896 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:30:56.998 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:30:57.019 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:57.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:57.020 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:30:57.121 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:30:57.145 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:57.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:57.146 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:30:57.247 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:30:57.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:57 smithi032 ceph-mon[20004]: osdmap e563: 12 total, 12 up, 9 in 2024-08-19T14:30:57.269 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:57.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:57.271 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:30:57.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:57 smithi045 ceph-mon[20143]: osdmap e563: 12 total, 12 up, 9 in 2024-08-19T14:30:57.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:57 smithi031 ceph-mon[16336]: osdmap e563: 12 total, 12 up, 9 in 2024-08-19T14:30:57.371 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:30:57.410 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:57.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:57.411 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:30:57.512 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:30:57.533 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:57.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:57.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:30:57.635 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:30:57.666 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:57.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:57.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:30:57.768 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:30:57.810 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:57.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:57.815 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:30:57.916 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:30:57.938 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:57.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:57.939 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:30:58.040 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:30:58.063 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:58.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:58.064 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:30:58.165 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:30:58.192 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:58.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:58.193 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:30:58.294 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:30:58.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:58 smithi045 ceph-mon[20143]: pgmap v1294: 47 pgs: 3 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 37 active+clean; 5.3 GiB data, 24 GiB used, 781 GiB / 805 GiB avail; 1023 B/s rd, 26 KiB/s wr, 448 op/s; 484/258516 objects degraded (0.187%); 32722/258516 objects misplaced (12.658%); 20 MiB/s, 325 objects/s recovering 2024-08-19T14:30:58.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:58 smithi045 ceph-mon[20143]: osdmap e564: 12 total, 12 up, 9 in 2024-08-19T14:30:58.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:30:58 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 484/258516 objects degraded (0.187%), 3 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-19T14:30:58.315 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:58.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:58.316 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:30:58.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:58 smithi031 ceph-mon[16336]: pgmap v1294: 47 pgs: 3 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 37 active+clean; 5.3 GiB data, 24 GiB used, 781 GiB / 805 GiB avail; 1023 B/s rd, 26 KiB/s wr, 448 op/s; 484/258516 objects degraded (0.187%); 32722/258516 objects misplaced (12.658%); 20 MiB/s, 325 objects/s recovering 2024-08-19T14:30:58.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:58 smithi031 ceph-mon[16336]: osdmap e564: 12 total, 12 up, 9 in 2024-08-19T14:30:58.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:30:58 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 484/258516 objects degraded (0.187%), 3 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-19T14:30:58.417 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:30:58.449 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:58.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:58.450 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:30:58.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:58 smithi032 ceph-mon[20004]: pgmap v1294: 47 pgs: 3 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 37 active+clean; 5.3 GiB data, 24 GiB used, 781 GiB / 805 GiB avail; 1023 B/s rd, 26 KiB/s wr, 448 op/s; 484/258516 objects degraded (0.187%); 32722/258516 objects misplaced (12.658%); 20 MiB/s, 325 objects/s recovering 2024-08-19T14:30:58.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:58 smithi032 ceph-mon[20004]: osdmap e564: 12 total, 12 up, 9 in 2024-08-19T14:30:58.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:30:58 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 484/258516 objects degraded (0.187%), 3 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-19T14:30:58.551 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:30:58.572 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:58.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:58.573 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:30:58.674 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:30:58.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:58.698 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:58.698 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:30:58.799 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:30:58.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:58.822 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:58.823 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:30:58.923 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:30:58.984 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:58.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:58.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:30:59.086 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:30:59.115 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:59.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:59.116 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:30:59.217 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:30:59.239 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:30:59.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:59.241 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:30:59.342 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:30:59.363 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:59.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:59.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:30:59.466 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:30:59.502 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:59.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:59.504 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:30:59.605 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:30:59.649 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:30:59.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:59.650 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:30:59.751 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:30:59.780 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:59.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:59.781 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:30:59.882 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:30:59.909 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:30:59.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:30:59.910 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:31:00.011 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:31:00.051 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:00.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:00.052 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:31:00.153 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:31:00.175 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:00.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:00.176 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:31:00.277 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:31:00.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:00 smithi045 ceph-mon[20143]: pgmap v1296: 47 pgs: 3 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 37 active+clean; 5.3 GiB data, 24 GiB used, 781 GiB / 805 GiB avail; 0 B/s wr, 321 op/s; 484/258516 objects degraded (0.187%); 32722/258516 objects misplaced (12.658%); 10 MiB/s, 163 objects/s recovering 2024-08-19T14:31:00.313 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:00.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:00.314 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:31:00.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:00 smithi031 ceph-mon[16336]: pgmap v1296: 47 pgs: 3 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 37 active+clean; 5.3 GiB data, 24 GiB used, 781 GiB / 805 GiB avail; 0 B/s wr, 321 op/s; 484/258516 objects degraded (0.187%); 32722/258516 objects misplaced (12.658%); 10 MiB/s, 163 objects/s recovering 2024-08-19T14:31:00.415 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:31:00.453 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:00.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:00.454 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:31:00.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:00 smithi032 ceph-mon[20004]: pgmap v1296: 47 pgs: 3 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 37 active+clean; 5.3 GiB data, 24 GiB used, 781 GiB / 805 GiB avail; 0 B/s wr, 321 op/s; 484/258516 objects degraded (0.187%); 32722/258516 objects misplaced (12.658%); 10 MiB/s, 163 objects/s recovering 2024-08-19T14:31:00.554 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:31:00.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:00.590 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:00.590 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:31:00.691 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:31:00.726 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:00.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:00.727 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:31:00.828 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:31:00.856 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:00.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:00.857 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:31:00.959 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:31:00.982 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:00.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:00.983 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:31:01.084 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:31:01.106 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:01.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:01.107 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:31:01.208 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:31:01.236 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:01.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:01.237 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:31:01.338 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:31:01.360 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:01.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:01.361 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:31:01.461 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:31:01.484 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:01.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:01.485 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:31:01.488 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:31:01.517 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:31:01.585 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:31:01.606 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:01.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:01.608 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:31:01.710 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:31:01.756 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:01.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:01.757 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:31:01.858 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:31:01.881 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:01.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:01.882 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:31:01.917 INFO:journalctl@ceph.osd.2.smithi031.stdout:Aug 19 14:31:01 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-2[41419]: 2024-08-19T14:31:01.637+0000 7f9f498ff640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-19T14:31:01.918 INFO:journalctl@ceph.osd.2.smithi031.stdout:Aug 19 14:31:01 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-2[41419]: 2024-08-19T14:31:01.640+0000 7f9f498ff640 -1 osd.2 564 *** Got signal Terminated *** 2024-08-19T14:31:01.918 INFO:journalctl@ceph.osd.2.smithi031.stdout:Aug 19 14:31:01 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-2[41419]: 2024-08-19T14:31:01.640+0000 7f9f498ff640 -1 osd.2 564 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-08-19T14:31:01.983 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:31:02.004 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:02.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:02.005 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:31:02.106 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:31:02.127 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:02.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:02.128 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:31:02.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:02 smithi031 ceph-mon[16336]: pgmap v1297: 47 pgs: 3 active+recovering+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 39 active+clean; 5.1 GiB data, 23 GiB used, 782 GiB / 805 GiB avail; 0 B/s wr, 533 op/s; 548/252861 objects degraded (0.217%); 30672/252861 objects misplaced (12.130%); 27 MiB/s, 430 objects/s recovering 2024-08-19T14:31:02.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:02 smithi031 ceph-mon[16336]: osd.2 marked itself down and dead 2024-08-19T14:31:02.229 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:31:02.250 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:02.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:02.251 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:31:02.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:02 smithi045 ceph-mon[20143]: pgmap v1297: 47 pgs: 3 active+recovering+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 39 active+clean; 5.1 GiB data, 23 GiB used, 782 GiB / 805 GiB avail; 0 B/s wr, 533 op/s; 548/252861 objects degraded (0.217%); 30672/252861 objects misplaced (12.130%); 27 MiB/s, 430 objects/s recovering 2024-08-19T14:31:02.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:02 smithi045 ceph-mon[20143]: osd.2 marked itself down and dead 2024-08-19T14:31:02.352 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:31:02.376 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:02.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:02.377 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:31:02.478 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:31:02.501 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:02.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:02.502 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:31:02.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:02 smithi032 ceph-mon[20004]: pgmap v1297: 47 pgs: 3 active+recovering+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 39 active+clean; 5.1 GiB data, 23 GiB used, 782 GiB / 805 GiB avail; 0 B/s wr, 533 op/s; 548/252861 objects degraded (0.217%); 30672/252861 objects misplaced (12.130%); 27 MiB/s, 430 objects/s recovering 2024-08-19T14:31:02.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:02 smithi032 ceph-mon[20004]: osd.2 marked itself down and dead 2024-08-19T14:31:02.603 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:31:02.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:02.626 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:02.626 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:31:02.730 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:31:02.751 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:02.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:02.752 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:31:02.853 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:31:02.870 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is unset 2024-08-19T14:31:02.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:02.882 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:02.882 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:31:02.918 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:31:02.918 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:31:02.983 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:31:03.006 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:03.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:03.007 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:31:03.108 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:31:03.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:03 smithi031 ceph-mon[16336]: Health check failed: 1 osds down (OSD_DOWN) 2024-08-19T14:31:03.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:03 smithi031 ceph-mon[16336]: osdmap e565: 12 total, 11 up, 9 in 2024-08-19T14:31:03.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:03 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1998254298' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:31:03.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:03 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 548/252861 objects degraded (0.217%), 3 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-19T14:31:03.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:03 smithi031 ceph-mon[16336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:31:03.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:03 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1998254298' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:31:03.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:03 smithi031 ceph-mon[16336]: osdmap e566: 12 total, 11 up, 9 in 2024-08-19T14:31:03.129 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:03.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:03.130 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:31:03.231 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:31:03.253 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:03.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:03.254 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:31:03.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:03 smithi045 ceph-mon[20143]: Health check failed: 1 osds down (OSD_DOWN) 2024-08-19T14:31:03.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:03 smithi045 ceph-mon[20143]: osdmap e565: 12 total, 11 up, 9 in 2024-08-19T14:31:03.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:03 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1998254298' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:31:03.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:03 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 548/252861 objects degraded (0.217%), 3 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-19T14:31:03.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:03 smithi045 ceph-mon[20143]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:31:03.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:03 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1998254298' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:31:03.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:03 smithi045 ceph-mon[20143]: osdmap e566: 12 total, 11 up, 9 in 2024-08-19T14:31:03.355 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:31:03.376 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:03.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:03.377 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:31:03.400 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:31:03.440 INFO:journalctl@ceph.osd.2.smithi031.stdout:Aug 19 14:31:03 smithi031 podman[161981]: 2024-08-19 14:31:03.12899328 +0000 UTC m=+1.650627519 container died d2aa136987f7f265c7e349cba529a52b7696fcb2eeba6db234a3d9d746d413b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-2, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, RELEASE=squid-de09dd0, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-08-19T14:31:03.479 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:31:03.501 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:03.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:03.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:31:03.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:03 smithi032 ceph-mon[20004]: Health check failed: 1 osds down (OSD_DOWN) 2024-08-19T14:31:03.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:03 smithi032 ceph-mon[20004]: osdmap e565: 12 total, 11 up, 9 in 2024-08-19T14:31:03.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:03 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1998254298' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:31:03.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:03 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 548/252861 objects degraded (0.217%), 3 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-19T14:31:03.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:03 smithi032 ceph-mon[20004]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:31:03.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:03 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1998254298' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:31:03.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:03 smithi032 ceph-mon[20004]: osdmap e566: 12 total, 11 up, 9 in 2024-08-19T14:31:03.604 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:31:03.628 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:03.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:03.629 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:31:03.730 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:31:03.752 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:03.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:03.753 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:31:03.854 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:31:03.866 INFO:journalctl@ceph.osd.2.smithi031.stdout:Aug 19 14:31:03 smithi031 podman[161981]: 2024-08-19 14:31:03.659289896 +0000 UTC m=+2.180924140 container cleanup d2aa136987f7f265c7e349cba529a52b7696fcb2eeba6db234a3d9d746d413b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-de09dd0) 2024-08-19T14:31:03.866 INFO:journalctl@ceph.osd.2.smithi031.stdout:Aug 19 14:31:03 smithi031 bash[161981]: ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-2 2024-08-19T14:31:03.880 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:03.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:03.882 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:31:03.983 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:31:04.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:04.020 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:04.021 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:31:04.121 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:31:04.159 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:04.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:04.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:31:04.262 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:31:04.285 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:04.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:04.286 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:31:04.290 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:04 smithi031 ceph-mon[16336]: pgmap v1299: 47 pgs: 3 active+recovering+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 39 active+clean; 5.1 GiB data, 23 GiB used, 782 GiB / 805 GiB avail; 0 B/s wr, 329 op/s; 548/252516 objects degraded (0.217%); 30672/252516 objects misplaced (12.147%); 20 MiB/s, 314 objects/s recovering 2024-08-19T14:31:04.290 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:04 smithi031 ceph-mon[16336]: 1.19 scrub starts 2024-08-19T14:31:04.290 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:04 smithi031 ceph-mon[16336]: 1.19 scrub ok 2024-08-19T14:31:04.290 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:04 smithi031 ceph-mon[16336]: osdmap e567: 12 total, 11 up, 9 in 2024-08-19T14:31:04.290 INFO:journalctl@ceph.osd.2.smithi031.stdout:Aug 19 14:31:03 smithi031 podman[162071]: 2024-08-19 14:31:03.894126902 +0000 UTC m=+1.164584219 container remove d2aa136987f7f265c7e349cba529a52b7696fcb2eeba6db234a3d9d746d413b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-2, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.license=GPLv2, RELEASE=squid-de09dd0, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731) 2024-08-19T14:31:04.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:04 smithi045 ceph-mon[20143]: pgmap v1299: 47 pgs: 3 active+recovering+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 39 active+clean; 5.1 GiB data, 23 GiB used, 782 GiB / 805 GiB avail; 0 B/s wr, 329 op/s; 548/252516 objects degraded (0.217%); 30672/252516 objects misplaced (12.147%); 20 MiB/s, 314 objects/s recovering 2024-08-19T14:31:04.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:04 smithi045 ceph-mon[20143]: 1.19 scrub starts 2024-08-19T14:31:04.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:04 smithi045 ceph-mon[20143]: 1.19 scrub ok 2024-08-19T14:31:04.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:04 smithi045 ceph-mon[20143]: osdmap e567: 12 total, 11 up, 9 in 2024-08-19T14:31:04.387 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:31:04.419 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:04.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:04.420 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:31:04.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:04 smithi032 ceph-mon[20004]: pgmap v1299: 47 pgs: 3 active+recovering+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 39 active+clean; 5.1 GiB data, 23 GiB used, 782 GiB / 805 GiB avail; 0 B/s wr, 329 op/s; 548/252516 objects degraded (0.217%); 30672/252516 objects misplaced (12.147%); 20 MiB/s, 314 objects/s recovering 2024-08-19T14:31:04.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:04 smithi032 ceph-mon[20004]: 1.19 scrub starts 2024-08-19T14:31:04.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:04 smithi032 ceph-mon[20004]: 1.19 scrub ok 2024-08-19T14:31:04.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:04 smithi032 ceph-mon[20004]: osdmap e567: 12 total, 11 up, 9 in 2024-08-19T14:31:04.521 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:31:04.551 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:04.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:04.552 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:31:04.653 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:31:04.684 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:04.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:04.686 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:31:04.787 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:31:04.811 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:04.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:04.812 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:31:04.913 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:31:04.936 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:04.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:04.937 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:31:05.038 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:31:05.078 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-08-19T14:31:05.090 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:05.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:05.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:31:05.115 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:31:05.191 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:31:05.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:05.224 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:05.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:31:05.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:05 smithi045 ceph-mon[20143]: 1.e scrub starts 2024-08-19T14:31:05.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:05 smithi045 ceph-mon[20143]: 1.e scrub ok 2024-08-19T14:31:05.302 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:05 smithi031 ceph-mon[16336]: 1.e scrub starts 2024-08-19T14:31:05.302 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:05 smithi031 ceph-mon[16336]: 1.e scrub ok 2024-08-19T14:31:05.324 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:31:05.357 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:05.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:05.358 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:31:05.459 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:31:05.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:05 smithi032 ceph-mon[20004]: 1.e scrub starts 2024-08-19T14:31:05.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:05 smithi032 ceph-mon[20004]: 1.e scrub ok 2024-08-19T14:31:05.520 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:05.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:05.521 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:31:05.622 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:31:05.650 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:05.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:05.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:31:05.752 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:31:05.774 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:05.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:05.775 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:31:05.828 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:31:05.829 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:31:05.836 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:31:05.875 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:31:05.881 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:31:05.881 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:31:05.887 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-08-19T14:31:05.927 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:31:05.927 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:31:05.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:05.932 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:05.932 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:31:05.965 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:31:05.965 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:31:06.002 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:31:06.003 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:31:06.033 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:31:06.035 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:31:06.035 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:31:06.063 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:06.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:06.064 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:31:06.074 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:31:06.074 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:31:06.106 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:31:06.106 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:31:06.137 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:31:06.138 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:31:06.165 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:31:06.170 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:31:06.170 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:31:06.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:06 smithi031 ceph-mon[16336]: pgmap v1302: 47 pgs: 5 remapped+peering, 3 active+undersized, 3 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded, 35 active+clean; 5.1 GiB data, 23 GiB used, 782 GiB / 805 GiB avail; 0 B/s wr, 475 op/s; 5582/249954 objects degraded (2.233%); 21445/249954 objects misplaced (8.580%); 27 MiB/s, 435 objects/s recovering 2024-08-19T14:31:06.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:06 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2614862926' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:31:06.198 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:06.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:06.203 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:31:06.210 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:31:06.210 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:31:06.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:06 smithi045 ceph-mon[20143]: pgmap v1302: 47 pgs: 5 remapped+peering, 3 active+undersized, 3 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded, 35 active+clean; 5.1 GiB data, 23 GiB used, 782 GiB / 805 GiB avail; 0 B/s wr, 475 op/s; 5582/249954 objects degraded (2.233%); 21445/249954 objects misplaced (8.580%); 27 MiB/s, 435 objects/s recovering 2024-08-19T14:31:06.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:06 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2614862926' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:31:06.303 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:31:06.325 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:06.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:06.326 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:31:06.426 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:31:06.447 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:06.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:06.448 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:31:06.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:06 smithi032 ceph-mon[20004]: pgmap v1302: 47 pgs: 5 remapped+peering, 3 active+undersized, 3 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded, 35 active+clean; 5.1 GiB data, 23 GiB used, 782 GiB / 805 GiB avail; 0 B/s wr, 475 op/s; 5582/249954 objects degraded (2.233%); 21445/249954 objects misplaced (8.580%); 27 MiB/s, 435 objects/s recovering 2024-08-19T14:31:06.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:06 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2614862926' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:31:06.549 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:31:06.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:06.584 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:06.584 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:31:06.685 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:31:06.708 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:06.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:06.709 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:31:06.765 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-19T14:31:06.810 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:31:06.818 INFO:journalctl@ceph.osd.2.smithi031.stdout:Aug 19 14:31:06 smithi031 podman[162504]: 2024-08-19 14:31:06.470807121 +0000 UTC m=+0.267486458 image pull ea7a5189f30bad3aca9524788e2965d2368d7ce786df09d4edcd8c3edf9cefba quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9 2024-08-19T14:31:06.818 INFO:journalctl@ceph.osd.2.smithi031.stdout:Aug 19 14:31:06 smithi031 podman[162504]: 2024-08-19 14:31:06.678775191 +0000 UTC m=+0.475454519 container create d72372c417aecb52ef77f9e076e82f7eafa38f2a859ac7e53095c94a2499d010 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-2-deactivate, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-de09dd0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-08-19T14:31:06.839 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:06.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:06.840 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:31:06.941 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:31:06.982 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:06.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:06.983 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:31:07.084 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:31:07.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:07 smithi031 ceph-mon[16336]: pgmap v1303: 47 pgs: 5 remapped+peering, 8 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 27 active+clean; 5.0 GiB data, 23 GiB used, 782 GiB / 805 GiB avail; 0 B/s wr, 302 op/s; 15838/247413 objects degraded (6.401%); 20724/247413 objects misplaced (8.376%); 13 MiB/s, 208 objects/s recovering 2024-08-19T14:31:07.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:07 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:31:07.127 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:07.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:07.128 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:31:07.230 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:31:07.253 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:07.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:07.254 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:31:07.351 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-19T14:31:07.351 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-19T14:31:07.355 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:31:07.393 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:07.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:07.394 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:31:07.451 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:31:07.451 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-08-19T14:31:07.452 INFO:teuthology.orchestra.run.smithi031.stdout: "complaint_time": 30, 2024-08-19T14:31:07.452 INFO:teuthology.orchestra.run.smithi031.stdout: "num_blocked_ops": 0 2024-08-19T14:31:07.452 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:31:07.494 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:31:07.510 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:07 smithi032 ceph-mon[20004]: pgmap v1303: 47 pgs: 5 remapped+peering, 8 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 27 active+clean; 5.0 GiB data, 23 GiB used, 782 GiB / 805 GiB avail; 0 B/s wr, 302 op/s; 15838/247413 objects degraded (6.401%); 20724/247413 objects misplaced (8.376%); 13 MiB/s, 208 objects/s recovering 2024-08-19T14:31:07.510 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:07 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:31:07.532 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:07.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:07.533 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:31:07.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:07 smithi045 ceph-mon[20143]: pgmap v1303: 47 pgs: 5 remapped+peering, 8 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 27 active+clean; 5.0 GiB data, 23 GiB used, 782 GiB / 805 GiB avail; 0 B/s wr, 302 op/s; 15838/247413 objects degraded (6.401%); 20724/247413 objects misplaced (8.376%); 13 MiB/s, 208 objects/s recovering 2024-08-19T14:31:07.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:07 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:31:07.633 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:31:07.655 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:07.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:07.656 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:31:07.757 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:31:07.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:07.811 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:07.811 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:31:07.912 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:31:07.961 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:07.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:07.962 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:31:08.063 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:31:08.086 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:08.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:08.087 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:31:08.138 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-08-19T14:31:08.188 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:31:08.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:08.231 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:08.231 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:31:08.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:08 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 15838/247413 objects degraded (6.401%), 7 pgs degraded (PG_DEGRADED) 2024-08-19T14:31:08.333 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:31:08.364 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:08.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:08.365 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:31:08.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:08 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 15838/247413 objects degraded (6.401%), 7 pgs degraded (PG_DEGRADED) 2024-08-19T14:31:08.466 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:31:08.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:08.514 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:08.514 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:31:08.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:08 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 15838/247413 objects degraded (6.401%), 7 pgs degraded (PG_DEGRADED) 2024-08-19T14:31:08.615 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:31:08.617 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:31:08.636 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:08.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:08.637 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:31:08.677 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:31:08.738 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:31:08.762 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:08.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:08.763 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:31:08.774 INFO:journalctl@ceph.osd.2.smithi031.stdout:Aug 19 14:31:08 smithi031 podman[162504]: 2024-08-19 14:31:08.406401358 +0000 UTC m=+2.203080693 container init d72372c417aecb52ef77f9e076e82f7eafa38f2a859ac7e53095c94a2499d010 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-2-deactivate, ceph=True, RELEASE=squid-de09dd0, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-08-19T14:31:08.774 INFO:journalctl@ceph.osd.2.smithi031.stdout:Aug 19 14:31:08 smithi031 podman[162504]: 2024-08-19 14:31:08.415632772 +0000 UTC m=+2.212312102 container start d72372c417aecb52ef77f9e076e82f7eafa38f2a859ac7e53095c94a2499d010 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-2-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-de09dd0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux ) 2024-08-19T14:31:08.774 INFO:journalctl@ceph.osd.2.smithi031.stdout:Aug 19 14:31:08 smithi031 podman[162504]: 2024-08-19 14:31:08.52284124 +0000 UTC m=+2.319520566 container attach d72372c417aecb52ef77f9e076e82f7eafa38f2a859ac7e53095c94a2499d010 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-2-deactivate, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, RELEASE=squid-de09dd0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-08-19T14:31:08.864 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:31:08.905 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:08.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:08.906 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:31:09.007 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:31:09.028 INFO:journalctl@ceph.osd.2.smithi031.stdout:Aug 19 14:31:09 smithi031 podman[162504]: 2024-08-19 14:31:09.006574013 +0000 UTC m=+2.803253341 container died d72372c417aecb52ef77f9e076e82f7eafa38f2a859ac7e53095c94a2499d010 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-2-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, RELEASE=squid-de09dd0, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, org.label-schema.build-date=20240731) 2024-08-19T14:31:09.029 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:09.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:09.030 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:31:09.131 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:31:09.174 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:09.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:09.175 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:31:09.276 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:31:09.297 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:09.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:09.298 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:31:09.352 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:31:09.399 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:31:09.421 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:09.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:09.422 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:31:09.523 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:31:09.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:09.559 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:09.559 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:31:09.635 INFO:journalctl@ceph.osd.2.smithi031.stdout:Aug 19 14:31:09 smithi031 podman[162821]: 2024-08-19 14:31:09.61305269 +0000 UTC m=+0.584024832 container remove d72372c417aecb52ef77f9e076e82f7eafa38f2a859ac7e53095c94a2499d010 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-2-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=squid-de09dd0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240731) 2024-08-19T14:31:09.639 DEBUG:teuthology.orchestra.run.smithi031:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968@osd.2.service' 2024-08-19T14:31:09.660 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:31:09.723 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:09.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:09.724 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:31:09.738 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-19T14:31:09.738 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-08-19T14:31:09.738 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.2 2024-08-19T14:31:09.738 INFO:tasks.thrashosds.thrasher:remote for osd 2 is ubuntu@smithi031.front.sepia.ceph.com 2024-08-19T14:31:09.738 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell -n osd.2 --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --op list-pgs 2024-08-19T14:31:09.825 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:31:09.920 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:09 smithi031 ceph-mon[16336]: pgmap v1304: 47 pgs: 5 remapped+peering, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 5.0 GiB data, 23 GiB used, 782 GiB / 805 GiB avail; 0 B/s wr, 281 op/s; 15838/247413 objects degraded (6.401%); 20724/247413 objects misplaced (8.376%); 13 MiB/s, 206 objects/s recovering 2024-08-19T14:31:09.920 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:09 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:31:09.920 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:09 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:31:09.920 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:09 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:31:09.920 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:09 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:31:09.920 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:09 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:31:09.926 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:09.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:09.927 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:31:10.028 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:31:10.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:10.060 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:10.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:31:10.161 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:31:10.182 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:10.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:10.183 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:31:10.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:09 smithi032 ceph-mon[20004]: pgmap v1304: 47 pgs: 5 remapped+peering, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 5.0 GiB data, 23 GiB used, 782 GiB / 805 GiB avail; 0 B/s wr, 281 op/s; 15838/247413 objects degraded (6.401%); 20724/247413 objects misplaced (8.376%); 13 MiB/s, 206 objects/s recovering 2024-08-19T14:31:10.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:09 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:31:10.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:09 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:31:10.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:09 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:31:10.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:09 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:31:10.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:09 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:31:10.284 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:31:10.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:09 smithi045 ceph-mon[20143]: pgmap v1304: 47 pgs: 5 remapped+peering, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 5.0 GiB data, 23 GiB used, 782 GiB / 805 GiB avail; 0 B/s wr, 281 op/s; 15838/247413 objects degraded (6.401%); 20724/247413 objects misplaced (8.376%); 13 MiB/s, 206 objects/s recovering 2024-08-19T14:31:10.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:09 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:31:10.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:09 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:31:10.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:09 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:31:10.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:09 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:31:10.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:09 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:31:10.306 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:10.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:10.307 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:31:10.408 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:31:10.430 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:10.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:10.431 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:31:10.532 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:31:10.571 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:10.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:10.572 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:31:10.673 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:31:10.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:10.714 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:10.714 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:31:10.815 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:31:10.856 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:10.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:10.857 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:31:10.893 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:10 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2927415013' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:31:10.893 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:10 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:31:10.957 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:31:10.991 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:10.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:10.992 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:31:11.093 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:31:11.127 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:11.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:11.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:31:11.151 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is set 2024-08-19T14:31:11.229 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:31:11.254 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:11.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:11.256 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:31:11.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:10 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2927415013' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:31:11.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:10 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:31:11.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:10 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2927415013' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:31:11.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:10 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:31:11.357 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:31:11.381 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:11.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:11.381 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:31:11.393 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/osd.2/config 2024-08-19T14:31:11.482 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:31:11.505 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:11.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:11.506 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:31:11.606 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:31:11.627 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:11.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:11.628 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:31:11.729 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:31:11.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:11.753 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:11.753 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:31:11.854 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:31:11.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:11.890 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:11.890 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:31:11.992 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:31:12.030 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:12.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:12.031 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:31:12.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:11 smithi031 ceph-mon[16336]: pgmap v1305: 47 pgs: 5 active+recovering+degraded+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 4.9 GiB data, 22 GiB used, 782 GiB / 805 GiB avail; 0 B/s wr, 531 op/s; 40682/239763 objects degraded (16.968%); 20297/239763 objects misplaced (8.465%); 19 MiB/s, 307 objects/s recovering 2024-08-19T14:31:12.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:11 smithi031 ceph-mon[16336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:31:12.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:11 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:31:12.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:11 smithi031 ceph-mon[16336]: osdmap e568: 12 total, 11 up, 9 in 2024-08-19T14:31:12.134 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:31:12.156 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:12.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:12.157 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:31:12.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:11 smithi032 ceph-mon[20004]: pgmap v1305: 47 pgs: 5 active+recovering+degraded+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 4.9 GiB data, 22 GiB used, 782 GiB / 805 GiB avail; 0 B/s wr, 531 op/s; 40682/239763 objects degraded (16.968%); 20297/239763 objects misplaced (8.465%); 19 MiB/s, 307 objects/s recovering 2024-08-19T14:31:12.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:11 smithi032 ceph-mon[20004]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:31:12.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:11 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:31:12.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:11 smithi032 ceph-mon[20004]: osdmap e568: 12 total, 11 up, 9 in 2024-08-19T14:31:12.258 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:31:12.279 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:12.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:12.280 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:31:12.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:11 smithi045 ceph-mon[20143]: pgmap v1305: 47 pgs: 5 active+recovering+degraded+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 4.9 GiB data, 22 GiB used, 782 GiB / 805 GiB avail; 0 B/s wr, 531 op/s; 40682/239763 objects degraded (16.968%); 20297/239763 objects misplaced (8.465%); 19 MiB/s, 307 objects/s recovering 2024-08-19T14:31:12.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:11 smithi045 ceph-mon[20143]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:31:12.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:11 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:31:12.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:11 smithi045 ceph-mon[20143]: osdmap e568: 12 total, 11 up, 9 in 2024-08-19T14:31:12.381 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:31:12.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:12.419 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:12.419 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:31:12.519 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:31:12.542 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:12.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:12.546 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:31:12.647 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:31:12.671 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:12.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:12.673 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:31:12.774 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:31:12.797 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:12.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:12.799 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:31:12.900 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:31:12.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:12.933 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:12.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:31:12.970 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:31:12.970 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 20, 2024-08-19T14:31:12.970 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 600, 2024-08-19T14:31:12.970 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [ 2024-08-19T14:31:12.970 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:12.970 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25813.0:132390 5.17 5:eadc7f5f:::benchmark_data_smithi045_85309_object12137:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e462)", 2024-08-19T14:31:12.970 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:26:28.790357+0000", 2024-08-19T14:31:12.970 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 284.17484072000002, 2024-08-19T14:31:12.971 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.61116161199999997, 2024-08-19T14:31:12.971 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:31:12.971 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:31:12.971 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:31:12.971 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:31:12.971 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25813", 2024-08-19T14:31:12.971 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:31:12.971 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 132390 2024-08-19T14:31:12.971 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:12.971 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:31:12.971 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:12.971 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:31:12.971 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.790357+0000", 2024-08-19T14:31:12.972 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:31:12.972 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:12.972 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:12.972 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:31:12.972 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.790357+0000", 2024-08-19T14:31:12.972 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:31:12.972 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:12.972 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:12.972 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:31:12.972 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.790358+0000", 2024-08-19T14:31:12.972 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2500000000000001e-06 2024-08-19T14:31:12.972 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:12.973 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:12.973 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:31:12.973 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.790363+0000", 2024-08-19T14:31:12.973 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4800000000000003e-06 2024-08-19T14:31:12.973 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:12.973 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:12.973 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:31:12.973 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.790373+0000", 2024-08-19T14:31:12.973 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0117e-05 2024-08-19T14:31:12.973 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:12.973 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:12.973 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:31:12.973 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.790378+0000", 2024-08-19T14:31:12.974 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.6280000000000002e-06 2024-08-19T14:31:12.974 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:12.974 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:12.974 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:31:12.974 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.791871+0000", 2024-08-19T14:31:12.974 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0014923829999999999 2024-08-19T14:31:12.974 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:12.974 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:12.974 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:31:12.974 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.791957+0000", 2024-08-19T14:31:12.974 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.6532000000000004e-05 2024-08-19T14:31:12.974 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:12.975 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:12.975 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,8,11", 2024-08-19T14:31:12.975 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.792037+0000", 2024-08-19T14:31:12.975 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.9797999999999997e-05 2024-08-19T14:31:12.975 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:12.975 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:12.975 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:31:12.975 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.793676+0000", 2024-08-19T14:31:12.975 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.001639176 2024-08-19T14:31:12.975 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:12.975 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:12.975 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:31:12.976 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.796271+0000", 2024-08-19T14:31:12.976 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0025944750000000002 2024-08-19T14:31:12.976 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:12.976 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:12.976 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:31:12.976 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.796315+0000", 2024-08-19T14:31:12.976 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.3995999999999997e-05 2024-08-19T14:31:12.976 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:12.976 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:12.976 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:31:12.976 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.400666+0000", 2024-08-19T14:31:12.976 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.60435100500000005 2024-08-19T14:31:12.976 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:12.977 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:12.977 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:31:12.977 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.400710+0000", 2024-08-19T14:31:12.977 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4027999999999999e-05 2024-08-19T14:31:12.977 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:12.977 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:12.977 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:31:12.977 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.401518+0000", 2024-08-19T14:31:12.977 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00080874400000000002 2024-08-19T14:31:12.977 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:31:12.977 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:31:12.977 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:31:12.977 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:12.978 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:12.978 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25813.0:132430 5.17 5:eeb9c3c7:::benchmark_data_smithi045_85309_object12177:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e462)", 2024-08-19T14:31:12.978 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:26:28.885232+0000", 2024-08-19T14:31:12.978 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 284.07996535699999, 2024-08-19T14:31:12.978 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.51860473299999998, 2024-08-19T14:31:12.978 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:31:12.978 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:31:12.978 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:31:12.978 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:31:12.978 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25813", 2024-08-19T14:31:12.978 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:31:12.978 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 132430 2024-08-19T14:31:12.979 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:12.979 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:31:12.979 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:12.979 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:31:12.979 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.885232+0000", 2024-08-19T14:31:12.979 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:31:12.979 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:12.979 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:12.979 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:31:12.979 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.885232+0000", 2024-08-19T14:31:12.979 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:31:12.979 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:12.979 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:12.979 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:31:12.980 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.885234+0000", 2024-08-19T14:31:12.980 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.7600000000000001e-06 2024-08-19T14:31:12.980 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:12.980 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:12.980 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:31:12.980 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.885240+0000", 2024-08-19T14:31:12.980 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.7579999999999998e-06 2024-08-19T14:31:12.980 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:12.980 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:12.980 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:31:12.980 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.885257+0000", 2024-08-19T14:31:12.980 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.7309999999999999e-05 2024-08-19T14:31:12.980 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:12.981 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:12.981 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:31:12.981 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.885265+0000", 2024-08-19T14:31:12.981 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.9209999999999992e-06 2024-08-19T14:31:12.981 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:12.981 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:12.981 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:31:12.981 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.885328+0000", 2024-08-19T14:31:12.981 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.2842000000000006e-05 2024-08-19T14:31:12.981 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:12.981 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:12.981 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:31:12.981 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.885477+0000", 2024-08-19T14:31:12.981 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00014877799999999999 2024-08-19T14:31:12.981 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:12.982 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:12.982 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,8,11", 2024-08-19T14:31:12.982 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.885711+0000", 2024-08-19T14:31:12.982 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000234074 2024-08-19T14:31:12.982 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:12.982 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:12.982 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:31:12.982 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.890476+0000", 2024-08-19T14:31:12.982 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0047655429999999997 2024-08-19T14:31:12.982 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:12.982 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:12.982 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:31:12.982 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.892622+0000", 2024-08-19T14:31:12.983 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0021459930000000001 2024-08-19T14:31:12.983 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:12.983 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:12.983 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:31:12.983 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.893155+0000", 2024-08-19T14:31:12.983 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00053261900000000002 2024-08-19T14:31:12.983 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:12.983 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:12.983 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:31:12.983 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.403023+0000", 2024-08-19T14:31:12.983 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.50986806399999995 2024-08-19T14:31:12.983 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:12.983 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:12.983 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:31:12.984 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.403099+0000", 2024-08-19T14:31:12.984 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.6025000000000005e-05 2024-08-19T14:31:12.984 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:12.984 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:12.984 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:31:12.984 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.403837+0000", 2024-08-19T14:31:12.984 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00073804600000000001 2024-08-19T14:31:12.984 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:31:12.984 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:31:12.985 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:31:12.985 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:12.985 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:12.985 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25813.0:132434 5.17 5:e9c564fa:::benchmark_data_smithi045_85309_object12181:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e462)", 2024-08-19T14:31:12.985 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:26:28.887497+0000", 2024-08-19T14:31:12.985 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 284.07770094199998, 2024-08-19T14:31:12.985 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.51694072000000002, 2024-08-19T14:31:12.985 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:31:12.985 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:31:12.985 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:31:12.985 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:31:12.985 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25813", 2024-08-19T14:31:12.986 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:31:12.986 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 132434 2024-08-19T14:31:12.986 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:12.986 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:31:12.986 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:12.986 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:31:12.986 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.887497+0000", 2024-08-19T14:31:12.986 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:31:12.986 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:12.986 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:12.986 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:31:12.986 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.887497+0000", 2024-08-19T14:31:12.986 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:31:12.987 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:12.987 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:12.987 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:31:12.987 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.887498+0000", 2024-08-19T14:31:12.987 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3599999999999999e-06 2024-08-19T14:31:12.987 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:12.987 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:12.987 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:31:12.987 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.887503+0000", 2024-08-19T14:31:12.987 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.8829999999999997e-06 2024-08-19T14:31:12.987 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:12.987 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:12.987 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:31:12.987 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.887519+0000", 2024-08-19T14:31:12.988 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6169000000000001e-05 2024-08-19T14:31:12.988 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:12.988 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:12.988 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:31:12.988 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.887526+0000", 2024-08-19T14:31:12.988 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.3379999999999999e-06 2024-08-19T14:31:12.988 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:12.988 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:12.988 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:31:12.988 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.887857+0000", 2024-08-19T14:31:12.988 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00033069799999999998 2024-08-19T14:31:12.988 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:12.989 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:12.989 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:31:12.989 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.888014+0000", 2024-08-19T14:31:12.989 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00015727800000000001 2024-08-19T14:31:12.989 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:12.989 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:12.989 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,8,11", 2024-08-19T14:31:12.989 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.888227+0000", 2024-08-19T14:31:12.989 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000212874 2024-08-19T14:31:12.989 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:12.989 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:12.989 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:31:12.989 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.890486+0000", 2024-08-19T14:31:12.990 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0022583909999999998 2024-08-19T14:31:12.990 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:12.990 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:12.990 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:31:12.990 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.892678+0000", 2024-08-19T14:31:12.990 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0021919600000000002 2024-08-19T14:31:12.990 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:12.990 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:12.990 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:31:12.990 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.893220+0000", 2024-08-19T14:31:12.990 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00054226800000000005 2024-08-19T14:31:12.990 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:12.990 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:12.991 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:31:12.991 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.403898+0000", 2024-08-19T14:31:12.991 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.51067800399999996 2024-08-19T14:31:12.991 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:12.991 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:12.991 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:31:12.991 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.403952+0000", 2024-08-19T14:31:12.991 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.4141999999999998e-05 2024-08-19T14:31:12.991 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:12.991 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:12.991 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:31:12.991 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.404437+0000", 2024-08-19T14:31:12.991 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00048535499999999998 2024-08-19T14:31:12.992 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:31:12.992 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:31:12.992 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:31:12.992 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:12.992 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:12.992 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25813.0:132587 5.17 5:e858db2d:::benchmark_data_smithi045_85309_object12334:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e462)", 2024-08-19T14:31:12.992 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:26:28.996803+0000", 2024-08-19T14:31:12.992 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 283.96839403000001, 2024-08-19T14:31:12.992 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.41584004200000002, 2024-08-19T14:31:12.992 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:31:12.992 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:31:12.992 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:31:12.992 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:31:12.992 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25813", 2024-08-19T14:31:12.993 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:31:12.993 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 132587 2024-08-19T14:31:12.993 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:12.993 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:31:12.993 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:12.993 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:31:12.993 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.996803+0000", 2024-08-19T14:31:12.993 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:31:12.993 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:12.993 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:12.993 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:31:12.993 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.996803+0000", 2024-08-19T14:31:12.993 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:31:12.993 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:12.994 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:12.994 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:31:12.994 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.996805+0000", 2024-08-19T14:31:12.994 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.049e-06 2024-08-19T14:31:12.994 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:12.994 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:12.994 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:31:12.994 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.996809+0000", 2024-08-19T14:31:12.994 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.3699999999999997e-06 2024-08-19T14:31:12.994 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:12.994 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:12.994 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:31:12.995 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.996821+0000", 2024-08-19T14:31:12.995 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2041e-05 2024-08-19T14:31:12.995 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:12.995 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:12.995 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:31:12.995 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.996828+0000", 2024-08-19T14:31:12.995 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.832e-06 2024-08-19T14:31:12.995 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:12.995 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:12.995 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:31:12.995 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.996895+0000", 2024-08-19T14:31:12.995 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.7636000000000001e-05 2024-08-19T14:31:12.995 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:12.996 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:12.996 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:31:12.996 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.997058+0000", 2024-08-19T14:31:12.996 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00016274799999999999 2024-08-19T14:31:12.996 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:12.996 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:12.996 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,8,11", 2024-08-19T14:31:12.996 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.997215+0000", 2024-08-19T14:31:12.996 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00015702899999999999 2024-08-19T14:31:12.997 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:12.997 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:12.997 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:31:12.997 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.000366+0000", 2024-08-19T14:31:12.997 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.003151208 2024-08-19T14:31:12.997 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:12.997 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:12.997 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:31:12.997 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.000550+0000", 2024-08-19T14:31:12.997 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00018371699999999999 2024-08-19T14:31:12.997 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:12.998 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:12.998 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:31:12.998 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.001943+0000", 2024-08-19T14:31:12.998 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.001392732 2024-08-19T14:31:12.998 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:12.998 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:12.998 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:31:12.998 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.411104+0000", 2024-08-19T14:31:12.999 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.40916105800000002 2024-08-19T14:31:12.999 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:12.999 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:12.999 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:31:12.999 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.411180+0000", 2024-08-19T14:31:13.000 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.5617999999999993e-05 2024-08-19T14:31:13.000 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.000 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.000 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:31:13.000 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.412644+0000", 2024-08-19T14:31:13.000 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0014640040000000001 2024-08-19T14:31:13.000 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:31:13.001 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:31:13.001 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:31:13.001 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.001 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.001 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25813.0:132638 5.17 5:eec760cc:::benchmark_data_smithi045_85309_object12385:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e462)", 2024-08-19T14:31:13.001 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:26:29.033888+0000", 2024-08-19T14:31:13.001 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 283.93130992800002, 2024-08-19T14:31:13.001 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.38830038300000003, 2024-08-19T14:31:13.001 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:31:13.001 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:31:13.002 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:31:13.003 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:31:13.003 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25813", 2024-08-19T14:31:13.003 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:31:13.003 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 132638 2024-08-19T14:31:13.003 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.003 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:31:13.003 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.003 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:31:13.003 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.033888+0000", 2024-08-19T14:31:13.003 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:31:13.003 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.003 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.003 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:31:13.004 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.033888+0000", 2024-08-19T14:31:13.004 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:31:13.004 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.004 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.004 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:31:13.004 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.033889+0000", 2024-08-19T14:31:13.004 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2950000000000001e-06 2024-08-19T14:31:13.004 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.004 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.004 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:31:13.004 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.033898+0000", 2024-08-19T14:31:13.004 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.7660000000000004e-06 2024-08-19T14:31:13.004 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.004 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.005 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:31:13.005 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.033910+0000", 2024-08-19T14:31:13.005 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1987000000000001e-05 2024-08-19T14:31:13.005 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.005 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.005 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:31:13.005 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.033918+0000", 2024-08-19T14:31:13.005 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.3089999999999996e-06 2024-08-19T14:31:13.005 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.005 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.005 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:31:13.006 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.033993+0000", 2024-08-19T14:31:13.006 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.4912000000000004e-05 2024-08-19T14:31:13.006 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.006 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.006 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:31:13.006 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.034154+0000", 2024-08-19T14:31:13.006 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000161423 2024-08-19T14:31:13.006 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.006 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.006 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,8,11", 2024-08-19T14:31:13.006 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.034316+0000", 2024-08-19T14:31:13.006 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00016205000000000001 2024-08-19T14:31:13.006 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.006 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.007 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:31:13.007 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.036209+0000", 2024-08-19T14:31:13.007 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.001892473 2024-08-19T14:31:13.007 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.007 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.007 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:31:13.007 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.037758+0000", 2024-08-19T14:31:13.007 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.001549351 2024-08-19T14:31:13.007 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.007 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.007 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:31:13.007 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.037977+0000", 2024-08-19T14:31:13.007 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00021854399999999999 2024-08-19T14:31:13.007 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.008 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.008 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:31:13.008 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.421288+0000", 2024-08-19T14:31:13.008 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.383311179 2024-08-19T14:31:13.008 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.008 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.008 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:31:13.008 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.421387+0000", 2024-08-19T14:31:13.008 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.9487999999999993e-05 2024-08-19T14:31:13.008 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.008 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.008 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:31:13.008 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.422188+0000", 2024-08-19T14:31:13.008 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00080060600000000004 2024-08-19T14:31:13.009 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:31:13.009 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:31:13.009 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:31:13.009 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.009 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.009 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25813.0:132698 5.17 5:e9ffa124:::benchmark_data_smithi045_85309_object12445:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e462)", 2024-08-19T14:31:13.009 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:26:29.076721+0000", 2024-08-19T14:31:13.009 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 283.88847631499999, 2024-08-19T14:31:13.009 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.346118178, 2024-08-19T14:31:13.009 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:31:13.009 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:31:13.009 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:31:13.009 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:31:13.009 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25813", 2024-08-19T14:31:13.010 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:31:13.010 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 132698 2024-08-19T14:31:13.010 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.010 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:31:13.011 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.011 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:31:13.011 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.076721+0000", 2024-08-19T14:31:13.011 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:31:13.011 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.011 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.011 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:31:13.011 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.076721+0000", 2024-08-19T14:31:13.011 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:31:13.011 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.011 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.012 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:31:13.012 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.076723+0000", 2024-08-19T14:31:13.012 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.519e-06 2024-08-19T14:31:13.012 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.012 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.012 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:31:13.012 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.076728+0000", 2024-08-19T14:31:13.012 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.011e-06 2024-08-19T14:31:13.012 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.012 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.013 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:31:13.013 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.076743+0000", 2024-08-19T14:31:13.013 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4952e-05 2024-08-19T14:31:13.013 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.013 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.013 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:31:13.013 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.076751+0000", 2024-08-19T14:31:13.013 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.5199999999999997e-06 2024-08-19T14:31:13.013 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.013 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.013 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:31:13.013 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.076826+0000", 2024-08-19T14:31:13.013 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.4789999999999994e-05 2024-08-19T14:31:13.013 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.014 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.014 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:31:13.014 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.076991+0000", 2024-08-19T14:31:13.014 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00016521600000000001 2024-08-19T14:31:13.014 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.014 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.014 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,8,11", 2024-08-19T14:31:13.014 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.077122+0000", 2024-08-19T14:31:13.014 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000130588 2024-08-19T14:31:13.014 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.015 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.015 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:31:13.015 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.079196+0000", 2024-08-19T14:31:13.015 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0020745350000000002 2024-08-19T14:31:13.015 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.015 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.015 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:31:13.015 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.080180+0000", 2024-08-19T14:31:13.015 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00098345099999999999 2024-08-19T14:31:13.015 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.015 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.015 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:31:13.015 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.080323+0000", 2024-08-19T14:31:13.015 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000143583 2024-08-19T14:31:13.016 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.016 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.016 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:31:13.016 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.422252+0000", 2024-08-19T14:31:13.016 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.34192826399999998 2024-08-19T14:31:13.016 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.016 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.016 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:31:13.016 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.422327+0000", 2024-08-19T14:31:13.016 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.5099999999999996e-05 2024-08-19T14:31:13.016 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.016 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.016 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:31:13.016 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.422839+0000", 2024-08-19T14:31:13.016 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000512649 2024-08-19T14:31:13.017 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:31:13.017 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:31:13.017 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:31:13.017 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.017 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.017 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25813.0:132733 5.17 5:e9c717cb:::benchmark_data_smithi045_85309_object12480:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e462)", 2024-08-19T14:31:13.017 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:26:29.101389+0000", 2024-08-19T14:31:13.017 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 283.86380833200002, 2024-08-19T14:31:13.017 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.32198057000000002, 2024-08-19T14:31:13.017 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:31:13.017 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:31:13.017 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:31:13.018 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:31:13.018 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25813", 2024-08-19T14:31:13.018 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:31:13.018 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 132733 2024-08-19T14:31:13.018 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.018 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:31:13.018 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.018 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:31:13.018 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.101389+0000", 2024-08-19T14:31:13.018 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:31:13.018 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.019 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.019 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:31:13.019 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.101389+0000", 2024-08-19T14:31:13.019 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:31:13.019 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.019 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.019 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:31:13.019 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.101391+0000", 2024-08-19T14:31:13.019 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.401e-06 2024-08-19T14:31:13.019 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.019 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.019 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:31:13.019 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.101396+0000", 2024-08-19T14:31:13.019 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.6810000000000003e-06 2024-08-19T14:31:13.020 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.020 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.020 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:31:13.020 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.101414+0000", 2024-08-19T14:31:13.020 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.7323999999999999e-05 2024-08-19T14:31:13.020 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.020 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.020 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:31:13.020 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.101424+0000", 2024-08-19T14:31:13.020 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0479000000000001e-05 2024-08-19T14:31:13.021 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.021 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.021 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:31:13.021 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.101497+0000", 2024-08-19T14:31:13.021 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.3008000000000003e-05 2024-08-19T14:31:13.021 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.021 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.021 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:31:13.021 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.101672+0000", 2024-08-19T14:31:13.021 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00017511600000000001 2024-08-19T14:31:13.021 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.021 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.022 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,8,11", 2024-08-19T14:31:13.022 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.101846+0000", 2024-08-19T14:31:13.022 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000174179 2024-08-19T14:31:13.022 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.022 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.022 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:31:13.022 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.104105+0000", 2024-08-19T14:31:13.022 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.002259063 2024-08-19T14:31:13.022 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.022 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.022 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:31:13.022 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.104876+0000", 2024-08-19T14:31:13.023 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00077034 2024-08-19T14:31:13.023 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.023 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.023 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:31:13.023 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.107300+0000", 2024-08-19T14:31:13.023 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0024242370000000001 2024-08-19T14:31:13.023 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.023 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.023 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:31:13.023 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.422905+0000", 2024-08-19T14:31:13.023 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.31560517500000002 2024-08-19T14:31:13.023 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.023 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.023 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:31:13.023 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.422964+0000", 2024-08-19T14:31:13.024 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.8692e-05 2024-08-19T14:31:13.024 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.024 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.024 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:31:13.024 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.423370+0000", 2024-08-19T14:31:13.024 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00040587499999999999 2024-08-19T14:31:13.024 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:31:13.024 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:31:13.024 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:31:13.024 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.024 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.024 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25813.0:237705 5.7 5:e5695e68:::benchmark_data_smithi045_85309_object117452:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e495)", 2024-08-19T14:31:13.024 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:28:07.101072+0000", 2024-08-19T14:31:13.024 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 185.86412537300001, 2024-08-19T14:31:13.025 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.231283661, 2024-08-19T14:31:13.025 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:31:13.025 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:31:13.025 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:31:13.025 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:31:13.025 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25813", 2024-08-19T14:31:13.025 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:31:13.025 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 237705 2024-08-19T14:31:13.025 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.026 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:31:13.026 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.026 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:31:13.026 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101072+0000", 2024-08-19T14:31:13.026 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:31:13.026 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.026 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.026 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:31:13.026 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101072+0000", 2024-08-19T14:31:13.026 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:31:13.026 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.026 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.027 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:31:13.027 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101074+0000", 2024-08-19T14:31:13.027 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.751e-06 2024-08-19T14:31:13.027 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.027 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.027 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:31:13.027 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101080+0000", 2024-08-19T14:31:13.027 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.9050000000000002e-06 2024-08-19T14:31:13.027 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.027 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.027 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:31:13.027 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101096+0000", 2024-08-19T14:31:13.027 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6364000000000001e-05 2024-08-19T14:31:13.027 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.028 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.028 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:31:13.028 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101107+0000", 2024-08-19T14:31:13.028 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0572e-05 2024-08-19T14:31:13.028 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.028 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.028 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:31:13.028 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101166+0000", 2024-08-19T14:31:13.028 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.8981999999999999e-05 2024-08-19T14:31:13.028 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.028 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.028 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:31:13.029 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101181+0000", 2024-08-19T14:31:13.029 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5364000000000001e-05 2024-08-19T14:31:13.029 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.029 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.029 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:31:13.029 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.315274+0000", 2024-08-19T14:31:13.030 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.21409246900000001 2024-08-19T14:31:13.030 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.030 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.030 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for missing object", 2024-08-19T14:31:13.030 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.315413+0000", 2024-08-19T14:31:13.031 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000139782 2024-08-19T14:31:13.031 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.031 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.031 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:31:13.031 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.332277+0000", 2024-08-19T14:31:13.032 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.016863488999999999 2024-08-19T14:31:13.032 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.032 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.032 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:31:13.032 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.332356+0000", 2024-08-19T14:31:13.032 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.8982999999999999e-05 2024-08-19T14:31:13.032 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:31:13.032 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:31:13.032 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:31:13.032 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.033 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.033 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25813.0:237714 5.7 5:e252da68:::benchmark_data_smithi045_85309_object117461:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e495)", 2024-08-19T14:31:13.033 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:28:07.101128+0000", 2024-08-19T14:31:13.033 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 185.86406989899999, 2024-08-19T14:31:13.033 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.24052662399999999, 2024-08-19T14:31:13.033 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:31:13.033 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:31:13.033 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:31:13.033 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:31:13.033 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25813", 2024-08-19T14:31:13.033 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:31:13.034 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 237714 2024-08-19T14:31:13.034 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.034 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:31:13.034 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.034 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:31:13.034 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101128+0000", 2024-08-19T14:31:13.034 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:31:13.034 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.034 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.034 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:31:13.034 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101128+0000", 2024-08-19T14:31:13.034 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:31:13.034 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.035 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.035 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:31:13.035 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101128+0000", 2024-08-19T14:31:13.035 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.8500000000000002e-07 2024-08-19T14:31:13.035 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.035 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.035 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:31:13.035 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101131+0000", 2024-08-19T14:31:13.035 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.6910000000000002e-06 2024-08-19T14:31:13.035 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.035 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.035 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:31:13.035 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101139+0000", 2024-08-19T14:31:13.035 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.1799999999999996e-06 2024-08-19T14:31:13.036 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.036 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.036 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:31:13.036 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101145+0000", 2024-08-19T14:31:13.036 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.7359999999999998e-06 2024-08-19T14:31:13.036 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.036 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.036 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:31:13.036 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101214+0000", 2024-08-19T14:31:13.036 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.9695000000000003e-05 2024-08-19T14:31:13.036 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.036 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.036 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:31:13.036 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101224+0000", 2024-08-19T14:31:13.037 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0043e-05 2024-08-19T14:31:13.037 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.037 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.037 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:31:13.037 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.315795+0000", 2024-08-19T14:31:13.037 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.21457066599999999 2024-08-19T14:31:13.037 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.037 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.037 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:31:13.037 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.315966+0000", 2024-08-19T14:31:13.037 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00017090700000000001 2024-08-19T14:31:13.037 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.037 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.037 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,5", 2024-08-19T14:31:13.038 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.316585+0000", 2024-08-19T14:31:13.038 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00061851899999999999 2024-08-19T14:31:13.038 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.038 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.038 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:31:13.038 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.332349+0000", 2024-08-19T14:31:13.038 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.015764158 2024-08-19T14:31:13.038 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.038 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.038 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:31:13.038 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.332428+0000", 2024-08-19T14:31:13.038 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.9651e-05 2024-08-19T14:31:13.039 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.039 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.039 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:31:13.039 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.341243+0000", 2024-08-19T14:31:13.039 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0088151610000000002 2024-08-19T14:31:13.039 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.039 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.039 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:31:13.039 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.341320+0000", 2024-08-19T14:31:13.039 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.6304000000000007e-05 2024-08-19T14:31:13.039 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.039 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.040 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:31:13.040 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.341654+0000", 2024-08-19T14:31:13.040 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000334428 2024-08-19T14:31:13.040 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:31:13.040 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:31:13.040 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:31:13.040 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.040 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.040 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25813.0:237731 5.7 5:e54e0cab:::benchmark_data_smithi045_85309_object117478:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e495)", 2024-08-19T14:31:13.040 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:28:07.101155+0000", 2024-08-19T14:31:13.040 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 185.86404223900001, 2024-08-19T14:31:13.040 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.240822699, 2024-08-19T14:31:13.040 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:31:13.040 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:31:13.040 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:31:13.041 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:31:13.041 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25813", 2024-08-19T14:31:13.041 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:31:13.041 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 237731 2024-08-19T14:31:13.041 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.041 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:31:13.041 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.041 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:31:13.041 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101155+0000", 2024-08-19T14:31:13.041 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:31:13.041 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.041 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.041 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:31:13.041 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101155+0000", 2024-08-19T14:31:13.042 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:31:13.042 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.042 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.042 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:31:13.042 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101156+0000", 2024-08-19T14:31:13.042 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.5900000000000003e-07 2024-08-19T14:31:13.042 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.042 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.042 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:31:13.042 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101158+0000", 2024-08-19T14:31:13.042 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.0420000000000001e-06 2024-08-19T14:31:13.042 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.042 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.043 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:31:13.043 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101164+0000", 2024-08-19T14:31:13.043 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.7769999999999997e-06 2024-08-19T14:31:13.043 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.043 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.043 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:31:13.043 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101184+0000", 2024-08-19T14:31:13.043 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.9989000000000001e-05 2024-08-19T14:31:13.043 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.043 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.043 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:31:13.043 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101265+0000", 2024-08-19T14:31:13.043 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.0782e-05 2024-08-19T14:31:13.043 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.044 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.044 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:31:13.044 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101274+0000", 2024-08-19T14:31:13.044 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.0780000000000002e-06 2024-08-19T14:31:13.044 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.044 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.044 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:31:13.044 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.320042+0000", 2024-08-19T14:31:13.044 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.21876728300000001 2024-08-19T14:31:13.044 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.044 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.044 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:31:13.044 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.320209+0000", 2024-08-19T14:31:13.044 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00016716800000000001 2024-08-19T14:31:13.045 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.045 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.045 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,5", 2024-08-19T14:31:13.045 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.320830+0000", 2024-08-19T14:31:13.045 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00062140999999999997 2024-08-19T14:31:13.045 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.045 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.045 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:31:13.045 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.332354+0000", 2024-08-19T14:31:13.045 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.011524123000000001 2024-08-19T14:31:13.045 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.045 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.045 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:31:13.045 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.341182+0000", 2024-08-19T14:31:13.045 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0088281499999999999 2024-08-19T14:31:13.046 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.046 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.046 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:31:13.046 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.341703+0000", 2024-08-19T14:31:13.046 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000521022 2024-08-19T14:31:13.046 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.046 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.046 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:31:13.046 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.341740+0000", 2024-08-19T14:31:13.046 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.6885999999999999e-05 2024-08-19T14:31:13.046 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.046 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.046 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:31:13.047 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.341978+0000", 2024-08-19T14:31:13.047 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00023763000000000001 2024-08-19T14:31:13.047 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:31:13.047 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:31:13.047 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:31:13.047 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.047 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.047 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25813.0:237758 5.7 5:e57a2935:::benchmark_data_smithi045_85309_object117505:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e495)", 2024-08-19T14:31:13.047 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:28:07.101204+0000", 2024-08-19T14:31:13.047 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 185.86399320999999, 2024-08-19T14:31:13.047 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.24687310400000001, 2024-08-19T14:31:13.047 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:31:13.047 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:31:13.047 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:31:13.047 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:31:13.048 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25813", 2024-08-19T14:31:13.048 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:31:13.048 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 237758 2024-08-19T14:31:13.048 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.048 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:31:13.048 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.048 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:31:13.048 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101204+0000", 2024-08-19T14:31:13.048 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:31:13.048 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.048 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.048 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:31:13.048 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101204+0000", 2024-08-19T14:31:13.048 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:31:13.049 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.049 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.049 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:31:13.049 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101205+0000", 2024-08-19T14:31:13.049 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2800000000000002e-07 2024-08-19T14:31:13.049 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.049 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.049 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:31:13.049 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101207+0000", 2024-08-19T14:31:13.049 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:31:13.051 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.768e-06 2024-08-19T14:31:13.051 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.051 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.051 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:31:13.051 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101215+0000", 2024-08-19T14:31:13.051 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.3900000000000004e-06 2024-08-19T14:31:13.051 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.051 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.051 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:31:13.052 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101220+0000", 2024-08-19T14:31:13.052 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.3040000000000004e-06 2024-08-19T14:31:13.052 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.052 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.052 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:31:13.052 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101303+0000", 2024-08-19T14:31:13.052 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.2443999999999999e-05 2024-08-19T14:31:13.052 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.052 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.052 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:31:13.052 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101311+0000", 2024-08-19T14:31:13.052 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.3389999999999993e-06 2024-08-19T14:31:13.052 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.052 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.053 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:31:13.053 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.324142+0000", 2024-08-19T14:31:13.053 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.222830844 2024-08-19T14:31:13.053 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.053 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.053 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:31:13.053 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.324274+0000", 2024-08-19T14:31:13.053 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013226199999999999 2024-08-19T14:31:13.053 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.053 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.053 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,5", 2024-08-19T14:31:13.053 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.324842+0000", 2024-08-19T14:31:13.053 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00056831899999999996 2024-08-19T14:31:13.053 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.053 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.054 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:31:13.054 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.332438+0000", 2024-08-19T14:31:13.054 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0075956060000000004 2024-08-19T14:31:13.054 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.054 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.054 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:31:13.054 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.342013+0000", 2024-08-19T14:31:13.054 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0095747650000000007 2024-08-19T14:31:13.054 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.054 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.054 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:31:13.054 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.347519+0000", 2024-08-19T14:31:13.054 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0055063330000000004 2024-08-19T14:31:13.055 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.055 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.055 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:31:13.055 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.347576+0000", 2024-08-19T14:31:13.055 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.6610000000000002e-05 2024-08-19T14:31:13.055 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.055 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.055 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:31:13.055 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.348077+0000", 2024-08-19T14:31:13.055 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00050169199999999998 2024-08-19T14:31:13.055 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:31:13.056 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:31:13.056 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:31:13.056 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.056 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.056 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:74181 6.4 6:249a6cd4:::benchmark_data_smithi045_97937_object74180:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e522)", 2024-08-19T14:31:13.056 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:29:13.888445+0000", 2024-08-19T14:31:13.056 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 119.07675219399999, 2024-08-19T14:31:13.056 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.20806031899999999, 2024-08-19T14:31:13.056 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:31:13.056 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:31:13.056 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:31:13.057 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:31:13.057 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:31:13.057 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:31:13.057 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 74181 2024-08-19T14:31:13.057 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.057 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:31:13.057 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.057 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:31:13.057 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:13.888445+0000", 2024-08-19T14:31:13.057 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:31:13.057 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.057 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.057 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:31:13.057 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:13.888445+0000", 2024-08-19T14:31:13.058 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:31:13.058 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.058 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.058 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:31:13.058 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:13.888447+0000", 2024-08-19T14:31:13.058 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.601e-06 2024-08-19T14:31:13.058 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.058 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.058 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:31:13.058 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:13.888520+0000", 2024-08-19T14:31:13.058 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.3189999999999996e-05 2024-08-19T14:31:13.058 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.058 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.059 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:31:13.059 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:13.888532+0000", 2024-08-19T14:31:13.059 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1953999999999999e-05 2024-08-19T14:31:13.059 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.059 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.059 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:31:13.059 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:13.888541+0000", 2024-08-19T14:31:13.059 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.67e-06 2024-08-19T14:31:13.059 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.059 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.059 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:31:13.059 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:13.888603+0000", 2024-08-19T14:31:13.060 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.1906e-05 2024-08-19T14:31:13.060 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.060 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.060 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:31:13.060 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:13.888753+0000", 2024-08-19T14:31:13.060 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000150049 2024-08-19T14:31:13.060 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.060 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.060 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,5", 2024-08-19T14:31:13.060 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:13.888949+0000", 2024-08-19T14:31:13.060 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00019627800000000001 2024-08-19T14:31:13.060 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.060 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.060 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:31:13.060 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:13.891830+0000", 2024-08-19T14:31:13.061 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0028813250000000001 2024-08-19T14:31:13.061 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.061 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.061 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:31:13.061 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:13.892208+0000", 2024-08-19T14:31:13.061 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00037749499999999999 2024-08-19T14:31:13.061 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.061 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.061 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:31:13.061 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:14.096388+0000", 2024-08-19T14:31:13.061 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.20418057100000001 2024-08-19T14:31:13.061 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.061 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.061 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:31:13.062 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:14.096458+0000", 2024-08-19T14:31:13.062 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.9890999999999999e-05 2024-08-19T14:31:13.062 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.062 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.062 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:31:13.062 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:14.096506+0000", 2024-08-19T14:31:13.062 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7389000000000003e-05 2024-08-19T14:31:13.062 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:31:13.062 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:31:13.062 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:31:13.062 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.063 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.063 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:76374 6.d 6:bf7b5a72:::benchmark_data_smithi045_97937_object76373:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e526)", 2024-08-19T14:31:13.063 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:29:17.862978+0000", 2024-08-19T14:31:13.063 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 115.102219753, 2024-08-19T14:31:13.063 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.44319341499999998, 2024-08-19T14:31:13.063 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:31:13.063 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:31:13.063 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:31:13.063 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:31:13.063 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:31:13.063 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:31:13.063 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 76374 2024-08-19T14:31:13.063 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.063 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:31:13.064 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.064 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:31:13.064 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.862978+0000", 2024-08-19T14:31:13.064 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:31:13.064 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.064 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.064 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:31:13.064 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.862978+0000", 2024-08-19T14:31:13.064 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:31:13.064 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.064 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.064 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:31:13.064 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.862980+0000", 2024-08-19T14:31:13.064 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.39e-06 2024-08-19T14:31:13.064 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.065 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.065 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:31:13.065 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.863020+0000", 2024-08-19T14:31:13.065 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.9978999999999998e-05 2024-08-19T14:31:13.065 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.065 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.065 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:31:13.065 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:18.285688+0000", 2024-08-19T14:31:13.066 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.42266810500000002 2024-08-19T14:31:13.066 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.066 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.066 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:31:13.066 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:18.285699+0000", 2024-08-19T14:31:13.066 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1187e-05 2024-08-19T14:31:13.066 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.066 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.066 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:31:13.066 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:18.288107+0000", 2024-08-19T14:31:13.067 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0024075279999999999 2024-08-19T14:31:13.067 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.067 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.067 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:31:13.067 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:18.288250+0000", 2024-08-19T14:31:13.067 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00014349400000000001 2024-08-19T14:31:13.067 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.067 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.067 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,5", 2024-08-19T14:31:13.067 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:18.288380+0000", 2024-08-19T14:31:13.067 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013004100000000001 2024-08-19T14:31:13.067 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.067 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.067 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:31:13.068 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:18.294529+0000", 2024-08-19T14:31:13.068 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0061484909999999999 2024-08-19T14:31:13.068 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.068 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.068 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:31:13.068 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:18.300842+0000", 2024-08-19T14:31:13.068 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0063133299999999998 2024-08-19T14:31:13.068 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.068 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.068 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:31:13.068 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:18.305954+0000", 2024-08-19T14:31:13.068 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.005111979 2024-08-19T14:31:13.068 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.068 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.069 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:31:13.069 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:18.306116+0000", 2024-08-19T14:31:13.069 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000161531 2024-08-19T14:31:13.069 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.069 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.069 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:31:13.069 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:18.306171+0000", 2024-08-19T14:31:13.069 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.5359999999999999e-05 2024-08-19T14:31:13.069 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:31:13.069 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:31:13.069 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:31:13.070 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.070 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.070 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:125474 6.d 6:bf3ca66b:::benchmark_data_smithi045_97937_object18201:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e559)", 2024-08-19T14:31:13.070 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:30:48.230541+0000", 2024-08-19T14:31:13.070 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 24.734656151999999, 2024-08-19T14:31:13.070 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.090725502999999999, 2024-08-19T14:31:13.070 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:31:13.070 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:31:13.070 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:31:13.070 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:31:13.070 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:31:13.070 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:31:13.070 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 125474 2024-08-19T14:31:13.071 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.071 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:31:13.071 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.071 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:31:13.071 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:48.230541+0000", 2024-08-19T14:31:13.071 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:31:13.071 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.071 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.071 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:31:13.071 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:48.230541+0000", 2024-08-19T14:31:13.071 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:31:13.071 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.071 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.071 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:31:13.071 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:48.230543+0000", 2024-08-19T14:31:13.072 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2100000000000001e-06 2024-08-19T14:31:13.072 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.072 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.072 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:31:13.072 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:48.230548+0000", 2024-08-19T14:31:13.072 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.9559999999999996e-06 2024-08-19T14:31:13.072 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.072 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.072 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:31:13.072 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:48.230556+0000", 2024-08-19T14:31:13.072 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.7739999999999993e-06 2024-08-19T14:31:13.072 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.073 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.073 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:31:13.073 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:48.230563+0000", 2024-08-19T14:31:13.073 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.3659999999999997e-06 2024-08-19T14:31:13.073 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.073 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.073 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:31:13.073 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:48.235239+0000", 2024-08-19T14:31:13.073 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0046765590000000003 2024-08-19T14:31:13.073 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.074 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.074 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:31:13.074 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:48.235374+0000", 2024-08-19T14:31:13.074 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013480799999999999 2024-08-19T14:31:13.074 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.075 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.075 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,5", 2024-08-19T14:31:13.075 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:48.235490+0000", 2024-08-19T14:31:13.075 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000116422 2024-08-19T14:31:13.075 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.075 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.076 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:31:13.076 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:48.247554+0000", 2024-08-19T14:31:13.076 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.012064001 2024-08-19T14:31:13.076 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.076 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.076 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:31:13.076 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:48.247766+0000", 2024-08-19T14:31:13.076 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000212041 2024-08-19T14:31:13.076 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.076 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.077 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:31:13.077 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:48.320947+0000", 2024-08-19T14:31:13.077 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.073180484000000004 2024-08-19T14:31:13.077 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.077 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.077 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:31:13.077 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:48.321131+0000", 2024-08-19T14:31:13.077 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000184298 2024-08-19T14:31:13.077 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.077 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.078 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:31:13.078 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:48.321267+0000", 2024-08-19T14:31:13.078 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013558400000000001 2024-08-19T14:31:13.078 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:31:13.078 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:31:13.078 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:31:13.078 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.078 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.079 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:125481 6.d 6:b9ad39dd:::benchmark_data_smithi045_97937_object18208:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e559)", 2024-08-19T14:31:13.079 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:30:48.247075+0000", 2024-08-19T14:31:13.079 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 24.718122294, 2024-08-19T14:31:13.079 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.075285060000000001, 2024-08-19T14:31:13.079 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:31:13.079 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:31:13.079 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:31:13.079 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:31:13.079 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:31:13.079 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:31:13.079 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 125481 2024-08-19T14:31:13.079 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.079 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:31:13.079 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.079 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:31:13.080 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:48.247075+0000", 2024-08-19T14:31:13.080 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:31:13.080 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.080 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.080 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:31:13.080 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:48.247075+0000", 2024-08-19T14:31:13.080 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:31:13.080 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.080 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.080 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:31:13.080 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:48.247077+0000", 2024-08-19T14:31:13.080 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.2249999999999999e-06 2024-08-19T14:31:13.080 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.081 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.081 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:31:13.081 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:48.247084+0000", 2024-08-19T14:31:13.081 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.9029999999999998e-06 2024-08-19T14:31:13.081 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.081 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.081 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:31:13.081 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:48.247096+0000", 2024-08-19T14:31:13.081 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1897e-05 2024-08-19T14:31:13.081 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.081 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.081 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:31:13.081 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:48.247103+0000", 2024-08-19T14:31:13.081 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.255e-06 2024-08-19T14:31:13.081 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.082 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.082 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:31:13.082 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:48.247793+0000", 2024-08-19T14:31:13.082 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00068912800000000001 2024-08-19T14:31:13.082 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.082 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.082 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:31:13.082 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:48.247902+0000", 2024-08-19T14:31:13.082 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000109708 2024-08-19T14:31:13.082 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.082 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.083 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,5", 2024-08-19T14:31:13.083 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:48.247997+0000", 2024-08-19T14:31:13.083 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.4648000000000006e-05 2024-08-19T14:31:13.083 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.083 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.083 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:31:13.083 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:48.321514+0000", 2024-08-19T14:31:13.083 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.073516775000000006 2024-08-19T14:31:13.083 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.083 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.083 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:31:13.083 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:48.321652+0000", 2024-08-19T14:31:13.083 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013787700000000001 2024-08-19T14:31:13.084 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.084 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.084 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:31:13.084 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:48.322001+0000", 2024-08-19T14:31:13.084 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00034889699999999998 2024-08-19T14:31:13.084 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.084 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.084 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:31:13.084 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:48.322219+0000", 2024-08-19T14:31:13.084 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00021890700000000001 2024-08-19T14:31:13.084 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.084 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.084 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:31:13.084 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:48.322360+0000", 2024-08-19T14:31:13.084 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00014083999999999999 2024-08-19T14:31:13.085 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:31:13.085 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:31:13.085 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:31:13.085 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.085 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.086 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:125489 6.d 6:b594b41f:::benchmark_data_smithi045_97937_object18216:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e559)", 2024-08-19T14:31:13.086 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:30:48.251229+0000", 2024-08-19T14:31:13.086 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 24.713968765000001, 2024-08-19T14:31:13.086 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.082237971000000007, 2024-08-19T14:31:13.086 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:31:13.086 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:31:13.086 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:31:13.086 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:31:13.086 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:31:13.086 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:31:13.087 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 125489 2024-08-19T14:31:13.087 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.087 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:31:13.087 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.087 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:31:13.087 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:48.251229+0000", 2024-08-19T14:31:13.087 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:31:13.087 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.087 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.087 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:31:13.087 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:48.251229+0000", 2024-08-19T14:31:13.087 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:31:13.087 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.087 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.088 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:31:13.088 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:48.251230+0000", 2024-08-19T14:31:13.088 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6419999999999999e-06 2024-08-19T14:31:13.088 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.088 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.088 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:31:13.088 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:48.251236+0000", 2024-08-19T14:31:13.088 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.9599999999999997e-06 2024-08-19T14:31:13.088 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.088 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.088 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:31:13.088 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:48.251249+0000", 2024-08-19T14:31:13.088 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3077e-05 2024-08-19T14:31:13.088 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.088 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.089 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:31:13.089 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:48.251258+0000", 2024-08-19T14:31:13.089 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.1599999999999998e-06 2024-08-19T14:31:13.089 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.089 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.089 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:31:13.089 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:48.314260+0000", 2024-08-19T14:31:13.089 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.063002760000000005 2024-08-19T14:31:13.089 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.089 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.089 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:31:13.089 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:48.314386+0000", 2024-08-19T14:31:13.089 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012545000000000001 2024-08-19T14:31:13.089 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.090 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.090 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,5", 2024-08-19T14:31:13.090 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:48.314502+0000", 2024-08-19T14:31:13.090 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00011599600000000001 2024-08-19T14:31:13.090 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.090 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.090 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:31:13.090 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:48.321657+0000", 2024-08-19T14:31:13.090 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0071555639999999997 2024-08-19T14:31:13.090 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.090 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.090 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:31:13.090 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:48.322561+0000", 2024-08-19T14:31:13.091 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00090345599999999996 2024-08-19T14:31:13.091 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.091 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.091 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:31:13.091 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:48.333338+0000", 2024-08-19T14:31:13.091 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.010776809 2024-08-19T14:31:13.091 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.091 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.091 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:31:13.091 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:48.333408+0000", 2024-08-19T14:31:13.091 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.0508999999999999e-05 2024-08-19T14:31:13.091 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.091 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.091 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:31:13.092 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:48.333467+0000", 2024-08-19T14:31:13.092 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.8588000000000002e-05 2024-08-19T14:31:13.092 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:31:13.092 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:31:13.092 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:31:13.092 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.092 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.092 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.26191.0:125515 6.6 e559/550 6:6c3a22d7:::benchmark_data_smithi045_97937_object18242:head v 559'7838, mlcod=513'1152)", 2024-08-19T14:31:13.092 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:30:48.338348+0000", 2024-08-19T14:31:13.092 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 24.626849566000001, 2024-08-19T14:31:13.092 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.089309664999999996, 2024-08-19T14:31:13.092 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:31:13.092 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:31:13.093 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:31:13.093 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:31:13.093 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.093 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:31:13.093 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:48.338348+0000", 2024-08-19T14:31:13.093 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:31:13.093 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.093 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.093 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:31:13.093 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:48.338348+0000", 2024-08-19T14:31:13.093 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:31:13.093 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.093 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.093 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:31:13.093 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:48.338349+0000", 2024-08-19T14:31:13.094 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.8299999999999995e-07 2024-08-19T14:31:13.094 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.094 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.094 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:31:13.094 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:48.338357+0000", 2024-08-19T14:31:13.094 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.3040000000000005e-06 2024-08-19T14:31:13.094 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.094 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.094 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:31:13.094 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:48.338373+0000", 2024-08-19T14:31:13.094 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5753e-05 2024-08-19T14:31:13.094 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.094 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.095 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:31:13.095 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:48.338381+0000", 2024-08-19T14:31:13.095 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.6869999999999994e-06 2024-08-19T14:31:13.095 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.095 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.095 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:31:13.095 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:48.373322+0000", 2024-08-19T14:31:13.095 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.034941102000000002 2024-08-19T14:31:13.095 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.095 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.095 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:31:13.096 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:48.373347+0000", 2024-08-19T14:31:13.096 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.5154e-05 2024-08-19T14:31:13.096 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.096 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.096 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:31:13.096 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:48.427607+0000", 2024-08-19T14:31:13.096 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.054260475000000002 2024-08-19T14:31:13.096 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.096 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.096 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:31:13.096 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:48.427658+0000", 2024-08-19T14:31:13.096 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0207000000000002e-05 2024-08-19T14:31:13.097 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:31:13.097 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:31:13.097 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:31:13.097 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.097 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.097 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.26191.0:125517 6.6 e559/550 6:67cfb602:::benchmark_data_smithi045_97937_object18244:head v 559'7839, mlcod=514'1169)", 2024-08-19T14:31:13.097 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:30:48.351218+0000", 2024-08-19T14:31:13.097 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 24.613979023999999, 2024-08-19T14:31:13.097 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.076466614000000002, 2024-08-19T14:31:13.097 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:31:13.097 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:31:13.097 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:31:13.097 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:31:13.097 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.098 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:31:13.098 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:48.351218+0000", 2024-08-19T14:31:13.098 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:31:13.098 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.098 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.098 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:31:13.098 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:48.351218+0000", 2024-08-19T14:31:13.098 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:31:13.098 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.098 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.098 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:31:13.098 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:48.351220+0000", 2024-08-19T14:31:13.099 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0300000000000001e-06 2024-08-19T14:31:13.099 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.099 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.099 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:31:13.099 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:48.351226+0000", 2024-08-19T14:31:13.099 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.9e-06 2024-08-19T14:31:13.099 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.099 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.099 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:31:13.099 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:48.351243+0000", 2024-08-19T14:31:13.099 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6718000000000002e-05 2024-08-19T14:31:13.100 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.100 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.100 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:31:13.100 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:48.351251+0000", 2024-08-19T14:31:13.100 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.9489999999999996e-06 2024-08-19T14:31:13.100 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.101 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.101 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:31:13.101 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:48.381991+0000", 2024-08-19T14:31:13.101 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.030739439 2024-08-19T14:31:13.101 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.101 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.101 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:31:13.101 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:48.382017+0000", 2024-08-19T14:31:13.101 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.6112999999999999e-05 2024-08-19T14:31:13.101 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.101 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.102 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:31:13.102 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:48.427662+0000", 2024-08-19T14:31:13.102 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.045645446999999999 2024-08-19T14:31:13.102 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.102 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.102 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:31:13.102 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:48.427685+0000", 2024-08-19T14:31:13.102 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.3017999999999999e-05 2024-08-19T14:31:13.102 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:31:13.102 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:31:13.102 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:31:13.102 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.102 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.103 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:135417 6.d 6:b70bc664:::benchmark_data_smithi045_97937_object28144:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e567)", 2024-08-19T14:31:13.103 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:31:10.145917+0000", 2024-08-19T14:31:13.103 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 2.8192806150000003, 2024-08-19T14:31:13.103 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.10515998999999999, 2024-08-19T14:31:13.103 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:31:13.103 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:31:13.103 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:31:13.103 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:31:13.103 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:31:13.103 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:31:13.103 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 135417 2024-08-19T14:31:13.103 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.103 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:31:13.103 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.104 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:31:13.104 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:31:10.145917+0000", 2024-08-19T14:31:13.104 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:31:13.104 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.104 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.104 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:31:13.104 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:31:10.145917+0000", 2024-08-19T14:31:13.104 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:31:13.104 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.104 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.104 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:31:13.104 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:31:10.145918+0000", 2024-08-19T14:31:13.104 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4389999999999999e-06 2024-08-19T14:31:13.104 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.104 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.105 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:31:13.105 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:31:10.145924+0000", 2024-08-19T14:31:13.105 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.7910000000000001e-06 2024-08-19T14:31:13.105 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.105 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.105 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:31:13.105 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:31:10.145937+0000", 2024-08-19T14:31:13.105 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3229e-05 2024-08-19T14:31:13.105 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.105 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.105 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:31:13.105 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:31:10.145945+0000", 2024-08-19T14:31:13.105 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.6830000000000008e-06 2024-08-19T14:31:13.105 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.106 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.106 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:31:13.106 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:31:10.146012+0000", 2024-08-19T14:31:13.106 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.7464000000000003e-05 2024-08-19T14:31:13.106 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.106 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.106 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:31:13.106 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:31:10.146113+0000", 2024-08-19T14:31:13.106 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000100123 2024-08-19T14:31:13.106 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.106 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.106 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,5", 2024-08-19T14:31:13.106 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:31:10.146207+0000", 2024-08-19T14:31:13.107 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.4396999999999998e-05 2024-08-19T14:31:13.107 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.107 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.107 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:31:13.107 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:31:10.154769+0000", 2024-08-19T14:31:13.107 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0085617000000000002 2024-08-19T14:31:13.107 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.107 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.107 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:31:13.107 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:31:10.154789+0000", 2024-08-19T14:31:13.107 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.0667000000000001e-05 2024-08-19T14:31:13.108 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.108 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.108 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:31:13.108 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:31:10.250983+0000", 2024-08-19T14:31:13.108 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.096193789000000002 2024-08-19T14:31:13.108 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.108 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.108 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:31:13.108 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:31:10.251035+0000", 2024-08-19T14:31:13.108 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.2324000000000003e-05 2024-08-19T14:31:13.108 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.109 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.109 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:31:13.109 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:31:10.251077+0000", 2024-08-19T14:31:13.109 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.1384000000000002e-05 2024-08-19T14:31:13.109 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:31:13.109 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:31:13.110 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:31:13.110 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.110 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.110 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:135420 6.4 6:22a6efbb:::benchmark_data_smithi045_97937_object28147:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e567)", 2024-08-19T14:31:13.110 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:31:10.154754+0000", 2024-08-19T14:31:13.111 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 2.8104437469999999, 2024-08-19T14:31:13.111 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.108985684, 2024-08-19T14:31:13.111 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:31:13.111 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:31:13.111 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:31:13.111 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:31:13.112 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:31:13.112 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:31:13.112 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 135420 2024-08-19T14:31:13.112 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.112 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:31:13.112 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.112 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:31:13.112 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:31:10.154754+0000", 2024-08-19T14:31:13.112 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:31:13.112 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.112 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.112 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:31:13.112 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:31:10.154754+0000", 2024-08-19T14:31:13.113 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:31:13.113 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.113 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.113 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:31:13.113 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:31:10.154755+0000", 2024-08-19T14:31:13.113 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3680000000000001e-06 2024-08-19T14:31:13.113 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.113 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.113 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:31:13.113 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:31:10.154760+0000", 2024-08-19T14:31:13.113 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4560000000000002e-06 2024-08-19T14:31:13.113 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.113 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.114 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:31:13.114 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:31:10.154770+0000", 2024-08-19T14:31:13.114 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0079e-05 2024-08-19T14:31:13.114 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.114 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.114 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:31:13.114 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:31:10.154777+0000", 2024-08-19T14:31:13.114 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.216e-06 2024-08-19T14:31:13.114 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.114 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.114 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:31:13.114 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:31:10.154844+0000", 2024-08-19T14:31:13.114 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.7262999999999996e-05 2024-08-19T14:31:13.115 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.115 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.115 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:31:13.115 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:31:10.154997+0000", 2024-08-19T14:31:13.115 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00015307 2024-08-19T14:31:13.115 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.115 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.115 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,5", 2024-08-19T14:31:13.115 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:31:10.155125+0000", 2024-08-19T14:31:13.115 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012732800000000001 2024-08-19T14:31:13.115 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.116 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.116 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:31:13.116 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:31:10.159637+0000", 2024-08-19T14:31:13.116 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0045125039999999996 2024-08-19T14:31:13.116 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.116 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.116 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:31:13.116 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:31:10.160091+0000", 2024-08-19T14:31:13.116 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00045373100000000002 2024-08-19T14:31:13.116 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.116 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.116 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:31:13.116 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:31:10.263634+0000", 2024-08-19T14:31:13.116 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.103543388 2024-08-19T14:31:13.117 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.117 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.117 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:31:13.117 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:31:10.263690+0000", 2024-08-19T14:31:13.117 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.5928999999999999e-05 2024-08-19T14:31:13.117 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:31:13.117 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:31:13.117 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:31:13.117 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:31:10.263739+0000", 2024-08-19T14:31:13.117 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.9351999999999998e-05 2024-08-19T14:31:13.117 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:31:13.117 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:31:13.117 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:31:13.117 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:31:13.118 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:31:13.118 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:31:13.118 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:12 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:31:13.118 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:12 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:31:13.139 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:13.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:13.140 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:31:13.241 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:31:13.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:12 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:31:13.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:12 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:31:13.276 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:13.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:13.277 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:31:13.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:12 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:31:13.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:12 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:31:13.378 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:31:13.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:13.418 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:13.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:31:13.519 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:31:13.558 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:13.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:13.559 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:31:13.660 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:31:13.691 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:13.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:13.692 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:31:13.695 INFO:teuthology.orchestra.run.smithi031.stderr:Creating an OSD daemon form without an OSD FSID value 2024-08-19T14:31:13.793 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:31:13.828 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:13.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:13.829 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:31:13.883 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-19T14:31:13.920 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:13 smithi031 ceph-mon[16336]: pgmap v1307: 47 pgs: 5 active+recovering+degraded+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 4.9 GiB data, 22 GiB used, 782 GiB / 805 GiB avail; 0 B/s wr, 436 op/s; 40499/239214 objects degraded (16.930%); 20297/239214 objects misplaced (8.485%); 13 MiB/s, 213 objects/s recovering 2024-08-19T14:31:13.921 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:13 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 40499/239214 objects degraded (16.930%), 12 pgs degraded (PG_DEGRADED) 2024-08-19T14:31:13.921 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:13 smithi031 ceph-mon[16336]: osdmap e569: 12 total, 11 up, 9 in 2024-08-19T14:31:13.930 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:31:13.962 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:13.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:13.963 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:31:14.026 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:31:14.063 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:31:14.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:14.096 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:14.096 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:31:14.197 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:31:14.219 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:14.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:14.220 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:31:14.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:13 smithi032 ceph-mon[20004]: pgmap v1307: 47 pgs: 5 active+recovering+degraded+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 4.9 GiB data, 22 GiB used, 782 GiB / 805 GiB avail; 0 B/s wr, 436 op/s; 40499/239214 objects degraded (16.930%); 20297/239214 objects misplaced (8.485%); 13 MiB/s, 213 objects/s recovering 2024-08-19T14:31:14.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:13 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 40499/239214 objects degraded (16.930%), 12 pgs degraded (PG_DEGRADED) 2024-08-19T14:31:14.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:13 smithi032 ceph-mon[20004]: osdmap e569: 12 total, 11 up, 9 in 2024-08-19T14:31:14.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:13 smithi045 ceph-mon[20143]: pgmap v1307: 47 pgs: 5 active+recovering+degraded+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 4.9 GiB data, 22 GiB used, 782 GiB / 805 GiB avail; 0 B/s wr, 436 op/s; 40499/239214 objects degraded (16.930%); 20297/239214 objects misplaced (8.485%); 13 MiB/s, 213 objects/s recovering 2024-08-19T14:31:14.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:13 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 40499/239214 objects degraded (16.930%), 12 pgs degraded (PG_DEGRADED) 2024-08-19T14:31:14.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:13 smithi045 ceph-mon[20143]: osdmap e569: 12 total, 11 up, 9 in 2024-08-19T14:31:14.321 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:31:14.355 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:14.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:14.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:31:14.457 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:31:14.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:14.486 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:14.486 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:31:14.587 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:31:14.611 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:14.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:14.612 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:31:14.713 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:31:14.735 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:14.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:14.736 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:31:14.837 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:31:14.875 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:14.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:14.876 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:31:14.977 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:31:15.010 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:15.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:15.011 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:31:15.112 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:31:15.148 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:15.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:15.150 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:31:15.250 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:31:15.290 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:15.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:15.291 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:31:15.392 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:31:15.414 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:15.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:15.415 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:31:15.516 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:31:15.547 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:15.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:15.548 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:31:15.649 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:31:15.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:15.695 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:15.695 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:31:15.796 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:31:15.819 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:15.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:15.820 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:31:15.921 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:31:15.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:15.960 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:15.961 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:31:16.062 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:31:16.101 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:16.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:16.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:31:16.127 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:15 smithi031 ceph-mon[16336]: pgmap v1309: 47 pgs: 5 active+recovering+degraded+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 4.8 GiB data, 22 GiB used, 783 GiB / 805 GiB avail; 0 B/s wr, 532 op/s; 39342/234621 objects degraded (16.768%); 20297/234621 objects misplaced (8.651%); 17 MiB/s, 277 objects/s recovering 2024-08-19T14:31:16.203 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:31:16.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:16.227 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:16.227 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:31:16.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:15 smithi032 ceph-mon[20004]: pgmap v1309: 47 pgs: 5 active+recovering+degraded+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 4.8 GiB data, 22 GiB used, 783 GiB / 805 GiB avail; 0 B/s wr, 532 op/s; 39342/234621 objects degraded (16.768%); 20297/234621 objects misplaced (8.651%); 17 MiB/s, 277 objects/s recovering 2024-08-19T14:31:16.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:15 smithi045 ceph-mon[20143]: pgmap v1309: 47 pgs: 5 active+recovering+degraded+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 4.8 GiB data, 22 GiB used, 783 GiB / 805 GiB avail; 0 B/s wr, 532 op/s; 39342/234621 objects degraded (16.768%); 20297/234621 objects misplaced (8.651%); 17 MiB/s, 277 objects/s recovering 2024-08-19T14:31:16.327 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:31:16.361 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:16.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:16.362 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:31:16.463 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:31:16.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:16.505 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:16.505 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:31:16.606 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:31:16.629 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:16.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:16.630 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:31:16.731 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:31:16.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:16.771 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:16.771 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:31:16.872 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:31:16.895 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:16.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:16.896 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:31:16.998 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:31:17.030 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:17.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:17.031 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:31:17.132 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:31:17.156 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:17.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:17.157 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:31:17.258 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:31:17.280 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:17.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:17.281 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:31:17.383 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:31:17.418 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:17.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:17.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:31:17.520 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:31:17.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:17.553 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:17.553 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:31:17.654 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:31:17.690 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:17.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:17.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:31:17.792 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:31:17.816 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:17.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:17.817 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:31:17.918 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:31:17.970 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:17.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:17.972 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:31:18.072 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:31:18.094 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:18.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:18.095 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:31:18.196 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:31:18.220 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:18.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:18.221 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:31:18.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:17 smithi032 ceph-mon[20004]: pgmap v1310: 47 pgs: 5 active+recovering+degraded+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 4.7 GiB data, 22 GiB used, 783 GiB / 805 GiB avail; 0 B/s wr, 723 op/s; 39072/230043 objects degraded (16.985%); 19759/230043 objects misplaced (8.589%); 20 MiB/s, 319 objects/s recovering 2024-08-19T14:31:18.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:17 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 39072/230043 objects degraded (16.985%), 12 pgs degraded (PG_DEGRADED) 2024-08-19T14:31:18.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:17 smithi045 ceph-mon[20143]: pgmap v1310: 47 pgs: 5 active+recovering+degraded+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 4.7 GiB data, 22 GiB used, 783 GiB / 805 GiB avail; 0 B/s wr, 723 op/s; 39072/230043 objects degraded (16.985%); 19759/230043 objects misplaced (8.589%); 20 MiB/s, 319 objects/s recovering 2024-08-19T14:31:18.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:17 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 39072/230043 objects degraded (16.985%), 12 pgs degraded (PG_DEGRADED) 2024-08-19T14:31:18.322 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:31:18.348 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:18.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:18.349 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:31:18.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:17 smithi031 ceph-mon[16336]: pgmap v1310: 47 pgs: 5 active+recovering+degraded+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 4.7 GiB data, 22 GiB used, 783 GiB / 805 GiB avail; 0 B/s wr, 723 op/s; 39072/230043 objects degraded (16.985%); 19759/230043 objects misplaced (8.589%); 20 MiB/s, 319 objects/s recovering 2024-08-19T14:31:18.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:17 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 39072/230043 objects degraded (16.985%), 12 pgs degraded (PG_DEGRADED) 2024-08-19T14:31:18.449 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:31:18.488 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:18.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:18.489 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:31:18.590 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:31:18.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:18.634 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:18.634 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:31:18.734 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:31:18.768 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:18.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:18.769 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:31:18.870 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:31:18.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:18.904 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:18.904 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:31:19.005 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:31:19.043 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:19.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:19.045 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:31:19.146 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:31:19.177 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:19.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:19.179 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:31:19.280 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:31:19.301 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:19.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:19.302 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:31:19.403 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:31:19.436 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:19.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:19.437 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:31:19.538 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:31:19.561 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:19.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:19.562 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:31:19.663 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:31:19.698 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:19.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:19.700 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:31:19.801 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:31:19.824 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:19.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:19.825 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:31:19.926 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:31:19.948 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:19.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:19.949 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:31:20.050 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:31:20.073 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:20.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:20.075 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:31:20.177 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:31:20.199 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:20.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:20.200 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:31:20.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:19 smithi032 ceph-mon[20004]: pgmap v1311: 47 pgs: 5 active+recovering+degraded+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 4.7 GiB data, 22 GiB used, 783 GiB / 805 GiB avail; 0 B/s wr, 404 op/s; 39072/230043 objects degraded (16.985%); 19759/230043 objects misplaced (8.589%); 11 MiB/s, 168 objects/s recovering 2024-08-19T14:31:20.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:19 smithi045 ceph-mon[20143]: pgmap v1311: 47 pgs: 5 active+recovering+degraded+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 4.7 GiB data, 22 GiB used, 783 GiB / 805 GiB avail; 0 B/s wr, 404 op/s; 39072/230043 objects degraded (16.985%); 19759/230043 objects misplaced (8.589%); 11 MiB/s, 168 objects/s recovering 2024-08-19T14:31:20.301 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:31:20.337 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:20.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:20.338 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:31:20.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:19 smithi031 ceph-mon[16336]: pgmap v1311: 47 pgs: 5 active+recovering+degraded+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 4.7 GiB data, 22 GiB used, 783 GiB / 805 GiB avail; 0 B/s wr, 404 op/s; 39072/230043 objects degraded (16.985%); 19759/230043 objects misplaced (8.589%); 11 MiB/s, 168 objects/s recovering 2024-08-19T14:31:20.439 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:31:20.461 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:20.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:20.462 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:31:20.563 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:31:20.584 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:20.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:20.585 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:31:20.687 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:31:20.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:20.726 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:20.727 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:31:20.828 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:31:20.852 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:20.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:20.853 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:31:20.954 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:31:20.993 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:20.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:20.995 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:31:21.095 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:31:21.127 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:21.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:21.128 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:31:21.229 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:31:21.268 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:21.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:21.269 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:31:21.370 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:31:21.392 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:21.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:21.393 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:31:21.494 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:31:21.525 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:21.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:21.526 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:31:21.627 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:31:21.666 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:21.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:21.667 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:31:21.768 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:31:21.789 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:21.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:21.790 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:31:21.891 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:31:21.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:21.932 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:21.932 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:31:22.033 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:31:22.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:22.065 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:22.065 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:31:22.166 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:31:22.191 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:22.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:22.192 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:31:22.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:21 smithi032 ceph-mon[20004]: pgmap v1312: 47 pgs: 5 active+recovering+degraded+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 4.5 GiB data, 21 GiB used, 783 GiB / 805 GiB avail; 0 B/s wr, 638 op/s; 37880/221574 objects degraded (17.096%); 19238/221574 objects misplaced (8.682%); 19 MiB/s, 306 objects/s recovering 2024-08-19T14:31:22.284 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:21 smithi031 ceph-mon[16336]: pgmap v1312: 47 pgs: 5 active+recovering+degraded+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 4.5 GiB data, 21 GiB used, 783 GiB / 805 GiB avail; 0 B/s wr, 638 op/s; 37880/221574 objects degraded (17.096%); 19238/221574 objects misplaced (8.682%); 19 MiB/s, 306 objects/s recovering 2024-08-19T14:31:22.293 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:31:22.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:21 smithi045 ceph-mon[20143]: pgmap v1312: 47 pgs: 5 active+recovering+degraded+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 4.5 GiB data, 21 GiB used, 783 GiB / 805 GiB avail; 0 B/s wr, 638 op/s; 37880/221574 objects degraded (17.096%); 19238/221574 objects misplaced (8.682%); 19 MiB/s, 306 objects/s recovering 2024-08-19T14:31:22.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:22.329 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:22.329 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:31:22.430 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:31:22.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:22.471 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:22.471 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:31:22.572 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:31:22.610 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:22.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:22.612 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:31:22.713 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:31:22.735 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:22.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:22.736 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:31:22.837 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:31:22.866 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:22.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:22.867 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:31:22.968 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:31:22.998 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:22.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:22.999 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:31:23.100 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:31:23.122 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:23.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:23.124 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:31:23.224 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:31:23.239 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:22 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1968106424' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:31:23.239 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:22 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 37880/221574 objects degraded (17.096%), 12 pgs degraded (PG_DEGRADED) 2024-08-19T14:31:23.252 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:23.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:23.253 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:31:23.279 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:31:23.279 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:31:23.279 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:31:23.279 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:31:23.279 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:31:23.279 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:31:23.279 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:31:23.279 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:31:23.280 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:31:23.280 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:31:23.280 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:31:23.280 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:31:23.280 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:31:23.280 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-08-19T14:31:23.280 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:31:23.280 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:31:23.281 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:31:23.281 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:31:23.281 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:31:23.281 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:31:23.281 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:31:23.281 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:31:23.281 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:31:23.281 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:22 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1968106424' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:31:23.281 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:22 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 37880/221574 objects degraded (17.096%), 12 pgs degraded (PG_DEGRADED) 2024-08-19T14:31:23.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:22 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1968106424' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:31:23.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:22 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 37880/221574 objects degraded (17.096%), 12 pgs degraded (PG_DEGRADED) 2024-08-19T14:31:23.354 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:31:23.386 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:23.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:23.387 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:31:23.489 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:31:23.528 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:23.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:23.529 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:31:23.630 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:31:23.671 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:23.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:23.672 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:31:23.772 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:31:23.796 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:23.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:23.797 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:31:23.898 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:31:23.935 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:23.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:23.936 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:31:24.037 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:31:24.077 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:24.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:24.078 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:31:24.179 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:31:24.201 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:24.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:24.202 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:31:24.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:23 smithi032 ceph-mon[20004]: pgmap v1313: 47 pgs: 5 active+recovering+degraded+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 4.5 GiB data, 21 GiB used, 783 GiB / 805 GiB avail; 0 B/s wr, 605 op/s; 37705/221049 objects degraded (17.057%); 19238/221049 objects misplaced (8.703%); 18 MiB/s, 282 objects/s recovering 2024-08-19T14:31:24.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:23 smithi045 ceph-mon[20143]: pgmap v1313: 47 pgs: 5 active+recovering+degraded+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 4.5 GiB data, 21 GiB used, 783 GiB / 805 GiB avail; 0 B/s wr, 605 op/s; 37705/221049 objects degraded (17.057%); 19238/221049 objects misplaced (8.703%); 18 MiB/s, 282 objects/s recovering 2024-08-19T14:31:24.302 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:31:24.339 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:24.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:24.340 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:31:24.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:23 smithi031 ceph-mon[16336]: pgmap v1313: 47 pgs: 5 active+recovering+degraded+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 4.5 GiB data, 21 GiB used, 783 GiB / 805 GiB avail; 0 B/s wr, 605 op/s; 37705/221049 objects degraded (17.057%); 19238/221049 objects misplaced (8.703%); 18 MiB/s, 282 objects/s recovering 2024-08-19T14:31:24.441 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:31:24.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:24.476 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:24.476 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:31:24.577 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:31:24.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:24.623 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:24.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:31:24.724 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:31:24.761 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:24.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:24.762 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:31:24.863 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:31:24.886 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:24.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:24.887 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:31:24.988 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:31:25.011 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:25.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:25.012 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:31:25.113 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:31:25.134 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:25.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:25.136 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:31:25.237 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:31:25.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:25.265 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:25.265 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:31:25.366 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:31:25.388 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:25.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:25.389 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:31:25.490 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:31:25.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:25.529 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:25.529 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:31:25.630 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:31:25.666 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:25.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:25.668 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:31:25.769 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:31:25.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:25.807 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:25.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:31:25.909 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:31:25.932 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:25.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:25.933 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:31:26.034 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:31:26.078 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:26.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:26.079 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:31:26.180 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:31:26.216 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:26.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:26.218 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:31:26.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:25 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:31:26.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:25 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:31:26.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:25 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:31:26.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:25 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:31:26.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:25 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:31:26.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:25 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:31:26.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:25 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:31:26.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:25 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:31:26.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:25 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:31:26.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:25 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:31:26.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:25 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:31:26.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:25 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:31:26.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:25 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:31:26.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:25 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:31:26.318 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:31:26.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:26.342 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:26.342 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:31:26.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:25 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:31:26.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:25 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:31:26.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:25 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:31:26.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:25 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:31:26.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:25 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:31:26.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:25 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:31:26.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:25 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:31:26.443 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:31:26.465 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:26.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:26.467 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:31:26.567 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:31:26.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:26.616 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:26.617 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:31:26.717 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:31:26.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:26.754 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:26.754 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:31:26.855 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:31:26.877 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:26.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:26.878 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:31:26.979 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:31:27.001 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:27.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:27.002 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:31:27.066 INFO:teuthology.orchestra.run.smithi031.stdout:6.f 2024-08-19T14:31:27.066 INFO:teuthology.orchestra.run.smithi031.stdout:6.e 2024-08-19T14:31:27.066 INFO:teuthology.orchestra.run.smithi031.stdout:6.9 2024-08-19T14:31:27.066 INFO:teuthology.orchestra.run.smithi031.stdout:6.5 2024-08-19T14:31:27.066 INFO:teuthology.orchestra.run.smithi031.stdout:1.1b 2024-08-19T14:31:27.066 INFO:teuthology.orchestra.run.smithi031.stdout:1.18 2024-08-19T14:31:27.066 INFO:teuthology.orchestra.run.smithi031.stdout:1.1d 2024-08-19T14:31:27.066 INFO:teuthology.orchestra.run.smithi031.stdout:1.1 2024-08-19T14:31:27.066 INFO:teuthology.orchestra.run.smithi031.stdout:6.0 2024-08-19T14:31:27.066 INFO:teuthology.orchestra.run.smithi031.stdout:1.5 2024-08-19T14:31:27.067 INFO:teuthology.orchestra.run.smithi031.stdout:1.7 2024-08-19T14:31:27.067 INFO:teuthology.orchestra.run.smithi031.stdout:6.a 2024-08-19T14:31:27.067 INFO:teuthology.orchestra.run.smithi031.stdout:1.8 2024-08-19T14:31:27.067 INFO:teuthology.orchestra.run.smithi031.stdout:1.a 2024-08-19T14:31:27.067 INFO:teuthology.orchestra.run.smithi031.stdout:1.14 2024-08-19T14:31:27.067 INFO:teuthology.orchestra.run.smithi031.stdout:6.1 2024-08-19T14:31:27.067 INFO:teuthology.orchestra.run.smithi031.stdout:1.9 2024-08-19T14:31:27.067 INFO:teuthology.orchestra.run.smithi031.stdout:6.2 2024-08-19T14:31:27.103 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:31:27.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:27.147 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:27.148 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:31:27.249 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:31:27.279 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:27.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:27.280 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:31:27.381 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:31:27.417 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:27.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:27.418 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:31:27.519 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:31:27.554 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:27.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:27.555 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:31:27.656 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:31:27.679 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:27.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:27.680 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:31:27.782 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:31:27.803 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:27.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:27.804 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:31:27.905 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:31:27.939 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:27.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:27.940 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:31:28.041 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:31:28.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:28.078 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:28.079 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:31:28.180 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:31:28.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:28.223 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:28.223 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:31:28.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:27 smithi032 ceph-mon[20004]: pgmap v1315: 47 pgs: 5 active+recovering+degraded+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 4.3 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 0 B/s wr, 610 op/s; 36495/212637 objects degraded (17.163%); 18647/212637 objects misplaced (8.769%); 17 MiB/s, 275 objects/s recovering 2024-08-19T14:31:28.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:27 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 36495/212637 objects degraded (17.163%), 12 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-19T14:31:28.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:27 smithi045 ceph-mon[20143]: pgmap v1315: 47 pgs: 5 active+recovering+degraded+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 4.3 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 0 B/s wr, 610 op/s; 36495/212637 objects degraded (17.163%); 18647/212637 objects misplaced (8.769%); 17 MiB/s, 275 objects/s recovering 2024-08-19T14:31:28.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:27 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 36495/212637 objects degraded (17.163%), 12 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-19T14:31:28.324 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:31:28.345 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:27 smithi031 ceph-mon[16336]: pgmap v1315: 47 pgs: 5 active+recovering+degraded+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 4.3 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 0 B/s wr, 610 op/s; 36495/212637 objects degraded (17.163%); 18647/212637 objects misplaced (8.769%); 17 MiB/s, 275 objects/s recovering 2024-08-19T14:31:28.345 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:27 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 36495/212637 objects degraded (17.163%), 12 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-19T14:31:28.358 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:28.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:28.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:31:28.461 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:31:28.494 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:28.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:28.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:31:28.597 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:31:28.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:28.633 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:28.633 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:31:28.735 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:31:28.779 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:28.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:28.780 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:31:28.882 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:31:28.919 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:28.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:28.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:31:29.022 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:31:29.062 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:29.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:29.064 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:31:29.165 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:31:29.188 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:29.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:29.189 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:31:29.290 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:31:29.327 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:29.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:29.328 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:31:29.429 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:31:29.460 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:29.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:29.462 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:31:29.563 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:31:29.586 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:29.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:29.587 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:31:29.688 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:31:29.718 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:29.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:29.719 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:31:29.821 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:31:29.843 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:29.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:29.844 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:31:29.945 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:31:29.970 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:29.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:29.971 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:31:30.072 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:31:30.093 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:30.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:30.094 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:31:30.195 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:31:30.233 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:30.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:30.234 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:31:30.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:29 smithi032 ceph-mon[20004]: pgmap v1316: 47 pgs: 5 active+recovering+degraded+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 4.3 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 0 B/s wr, 483 op/s; 36495/212637 objects degraded (17.163%); 18647/212637 objects misplaced (8.769%); 15 MiB/s, 247 objects/s recovering 2024-08-19T14:31:30.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:29 smithi045 ceph-mon[20143]: pgmap v1316: 47 pgs: 5 active+recovering+degraded+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 4.3 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 0 B/s wr, 483 op/s; 36495/212637 objects degraded (17.163%); 18647/212637 objects misplaced (8.769%); 15 MiB/s, 247 objects/s recovering 2024-08-19T14:31:30.335 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:31:30.358 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:30.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:30.359 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:31:30.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:29 smithi031 ceph-mon[16336]: pgmap v1316: 47 pgs: 5 active+recovering+degraded+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 4.3 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 0 B/s wr, 483 op/s; 36495/212637 objects degraded (17.163%); 18647/212637 objects misplaced (8.769%); 15 MiB/s, 247 objects/s recovering 2024-08-19T14:31:30.460 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:31:30.497 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:30.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:30.498 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:31:30.599 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:31:30.622 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:30.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:30.623 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:31:30.724 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:31:30.756 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:30.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:30.757 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:31:30.858 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:31:30.886 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:30.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:30.887 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:31:30.988 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:31:31.017 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:31.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:31.019 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:31:31.120 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:31:31.164 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:31.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:31.165 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:31:31.266 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:31:31.288 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:31.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:31.289 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:31:31.390 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:31:31.412 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:31.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:31.413 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:31:31.514 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:31:31.549 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:31.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:31.550 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:31:31.651 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:31:31.684 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-08-19T14:31:31.694 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:31.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:31.695 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:31:31.796 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:31:31.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:31.847 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:31.847 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:31:31.948 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:31:31.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:31.981 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:31.981 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:31:32.082 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:31:32.108 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:32.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:32.109 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:31:32.210 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:31:32.232 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:32.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:32.234 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:31:32.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:31 smithi032 ceph-mon[20004]: pgmap v1317: 47 pgs: 5 active+recovering+degraded+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 4.2 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 0 B/s wr, 718 op/s; 35385/204168 objects degraded (17.331%); 18029/204168 objects misplaced (8.830%); 23 MiB/s, 372 objects/s recovering 2024-08-19T14:31:32.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:31 smithi045 ceph-mon[20143]: pgmap v1317: 47 pgs: 5 active+recovering+degraded+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 4.2 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 0 B/s wr, 718 op/s; 35385/204168 objects degraded (17.331%); 18029/204168 objects misplaced (8.830%); 23 MiB/s, 372 objects/s recovering 2024-08-19T14:31:32.334 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:31:32.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:31 smithi031 ceph-mon[16336]: pgmap v1317: 47 pgs: 5 active+recovering+degraded+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 4.2 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 0 B/s wr, 718 op/s; 35385/204168 objects degraded (17.331%); 18029/204168 objects misplaced (8.830%); 23 MiB/s, 372 objects/s recovering 2024-08-19T14:31:32.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:32.367 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:32.368 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:31:32.469 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:31:32.491 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:32.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:32.492 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:31:32.593 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:31:32.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:32.617 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:32.617 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:31:32.718 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:31:32.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:32.762 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:32.762 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:31:32.863 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:31:32.885 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:32.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:32.886 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:31:32.987 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:31:33.022 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:33.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:33.023 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:31:33.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:32 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 35385/204168 objects degraded (17.331%), 12 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-19T14:31:33.123 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:31:33.158 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:33.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:33.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:31:33.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:32 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 35385/204168 objects degraded (17.331%), 12 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-19T14:31:33.261 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:31:33.284 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:33.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:33.285 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:31:33.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:32 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 35385/204168 objects degraded (17.331%), 12 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-19T14:31:33.387 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:31:33.408 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:33.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:33.409 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:31:33.510 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:31:33.534 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:33.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:33.536 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:31:33.637 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:31:33.659 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:33.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:33.660 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:31:33.761 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:31:33.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:33.806 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:33.806 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:31:33.907 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:31:33.951 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:33.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:33.952 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:31:34.053 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:31:34.077 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:34.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:34.078 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:31:34.179 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:31:34.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:34.221 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:34.221 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:31:34.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:33 smithi032 ceph-mon[20004]: pgmap v1318: 47 pgs: 5 active+recovering+degraded+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 4.1 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 0 B/s wr, 499 op/s; 35195/203598 objects degraded (17.287%); 18029/203598 objects misplaced (8.855%); 16 MiB/s, 250 objects/s recovering 2024-08-19T14:31:34.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:33 smithi045 ceph-mon[20143]: pgmap v1318: 47 pgs: 5 active+recovering+degraded+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 4.1 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 0 B/s wr, 499 op/s; 35195/203598 objects degraded (17.287%); 18029/203598 objects misplaced (8.855%); 16 MiB/s, 250 objects/s recovering 2024-08-19T14:31:34.323 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:31:34.355 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:34.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:34.357 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:31:34.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:33 smithi031 ceph-mon[16336]: pgmap v1318: 47 pgs: 5 active+recovering+degraded+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 4.1 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 0 B/s wr, 499 op/s; 35195/203598 objects degraded (17.287%); 18029/203598 objects misplaced (8.855%); 16 MiB/s, 250 objects/s recovering 2024-08-19T14:31:34.458 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:31:34.482 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:34.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:34.483 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:31:34.584 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:31:34.606 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:34.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:34.607 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:31:34.708 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:31:34.732 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:34.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:34.733 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:31:34.835 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:31:34.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:34.871 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:34.871 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:31:34.973 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:31:35.016 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:35.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:35.017 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:31:35.118 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:31:35.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:35.154 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:35.154 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:31:35.256 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:31:35.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:35.286 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:35.286 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:31:35.387 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:31:35.410 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:35.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:35.411 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:31:35.512 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:31:35.536 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:35.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:35.537 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:31:35.638 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:31:35.660 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:35.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:35.661 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:31:35.762 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:31:35.795 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:35.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:35.797 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:31:35.898 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:31:35.919 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:35.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:35.920 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:31:36.021 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:31:36.044 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:36.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:36.045 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:31:36.146 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:31:36.178 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:36.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:36.179 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:31:36.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:35 smithi032 ceph-mon[20004]: pgmap v1319: 47 pgs: 5 active+recovering+degraded+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 4.1 GiB data, 20 GiB used, 784 GiB / 805 GiB avail; 0 B/s wr, 596 op/s; 34169/199584 objects degraded (17.120%); 18029/199584 objects misplaced (9.033%); 23 MiB/s, 365 objects/s recovering 2024-08-19T14:31:36.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:35 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:31:36.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:35 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:31:36.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:35 smithi032 ceph-mon[20004]: osdmap e570: 12 total, 11 up, 9 in 2024-08-19T14:31:36.280 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:31:36.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:35 smithi045 ceph-mon[20143]: pgmap v1319: 47 pgs: 5 active+recovering+degraded+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 4.1 GiB data, 20 GiB used, 784 GiB / 805 GiB avail; 0 B/s wr, 596 op/s; 34169/199584 objects degraded (17.120%); 18029/199584 objects misplaced (9.033%); 23 MiB/s, 365 objects/s recovering 2024-08-19T14:31:36.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:35 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:31:36.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:35 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:31:36.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:35 smithi045 ceph-mon[20143]: osdmap e570: 12 total, 11 up, 9 in 2024-08-19T14:31:36.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:36.312 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:36.313 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:31:36.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:35 smithi031 ceph-mon[16336]: pgmap v1319: 47 pgs: 5 active+recovering+degraded+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 4.1 GiB data, 20 GiB used, 784 GiB / 805 GiB avail; 0 B/s wr, 596 op/s; 34169/199584 objects degraded (17.120%); 18029/199584 objects misplaced (9.033%); 23 MiB/s, 365 objects/s recovering 2024-08-19T14:31:36.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:35 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:31:36.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:35 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:31:36.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:35 smithi031 ceph-mon[16336]: osdmap e570: 12 total, 11 up, 9 in 2024-08-19T14:31:36.413 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:31:36.442 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:36.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:36.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:31:36.544 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:31:36.581 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:36.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:36.582 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:31:36.683 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:31:36.724 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:36.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:36.725 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:31:36.826 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:31:36.860 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:36.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:36.861 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:31:36.962 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:31:36.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:36.998 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:36.998 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:31:37.099 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:31:37.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:37.134 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:37.134 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:31:37.236 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:31:37.270 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:37.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:37.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:31:37.372 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:31:37.394 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:37.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:37.395 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:31:37.496 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:31:37.518 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:37.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:37.519 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:31:37.620 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:31:37.659 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:37.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:37.660 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:31:37.763 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:31:37.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:37.796 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:37.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:31:37.897 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:31:37.933 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:37.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:37.934 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:31:38.035 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:31:38.063 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:38.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:38.064 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:31:38.165 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:31:38.202 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:38.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:38.203 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:31:38.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:37 smithi032 ceph-mon[20004]: pgmap v1321: 47 pgs: 5 active+recovering+degraded+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 4.0 GiB data, 20 GiB used, 784 GiB / 805 GiB avail; 0 B/s wr, 577 op/s; 34016/195309 objects degraded (17.417%); 17389/195309 objects misplaced (8.903%); 20 MiB/s, 327 objects/s recovering 2024-08-19T14:31:38.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:37 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 34016/195309 objects degraded (17.417%), 12 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-19T14:31:38.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:37 smithi045 ceph-mon[20143]: pgmap v1321: 47 pgs: 5 active+recovering+degraded+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 4.0 GiB data, 20 GiB used, 784 GiB / 805 GiB avail; 0 B/s wr, 577 op/s; 34016/195309 objects degraded (17.417%); 17389/195309 objects misplaced (8.903%); 20 MiB/s, 327 objects/s recovering 2024-08-19T14:31:38.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:37 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 34016/195309 objects degraded (17.417%), 12 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-19T14:31:38.304 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:31:38.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:38.344 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:38.345 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:31:38.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:37 smithi031 ceph-mon[16336]: pgmap v1321: 47 pgs: 5 active+recovering+degraded+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 4.0 GiB data, 20 GiB used, 784 GiB / 805 GiB avail; 0 B/s wr, 577 op/s; 34016/195309 objects degraded (17.417%); 17389/195309 objects misplaced (8.903%); 20 MiB/s, 327 objects/s recovering 2024-08-19T14:31:38.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:37 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 34016/195309 objects degraded (17.417%), 12 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-19T14:31:38.445 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:31:38.475 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:38.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:38.476 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:31:38.577 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:31:38.616 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:38.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:38.616 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:31:38.717 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:31:38.752 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:38.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:38.753 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:31:38.854 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:31:38.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:38.875 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:38.876 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:31:38.977 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:31:38.998 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:38.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:38.999 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:31:39.100 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:31:39.135 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:39.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:39.138 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:31:39.183 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-19T14:31:39.183 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-19T14:31:39.240 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:31:39.271 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:39.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:39.272 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:31:39.373 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:31:39.412 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:39.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:39.413 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:31:39.514 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:31:39.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:39.555 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:39.556 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:31:39.657 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:31:39.679 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:39.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:39.680 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:31:39.781 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:31:39.812 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:39.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:39.813 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:31:39.914 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:31:39.936 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:39.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:39.937 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:31:40.038 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:31:40.059 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:40.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:40.060 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:31:40.161 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:31:40.195 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:40.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:40.196 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:31:40.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:39 smithi032 ceph-mon[20004]: pgmap v1322: 47 pgs: 5 active+recovering+degraded+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 4.0 GiB data, 20 GiB used, 784 GiB / 805 GiB avail; 0 B/s wr, 577 op/s; 34016/195309 objects degraded (17.417%); 17389/195309 objects misplaced (8.903%); 20 MiB/s, 327 objects/s recovering 2024-08-19T14:31:40.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:39 smithi045 ceph-mon[20143]: pgmap v1322: 47 pgs: 5 active+recovering+degraded+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 4.0 GiB data, 20 GiB used, 784 GiB / 805 GiB avail; 0 B/s wr, 577 op/s; 34016/195309 objects degraded (17.417%); 17389/195309 objects misplaced (8.903%); 20 MiB/s, 327 objects/s recovering 2024-08-19T14:31:40.296 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:31:40.332 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:40.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:40.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:31:40.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:39 smithi031 ceph-mon[16336]: pgmap v1322: 47 pgs: 5 active+recovering+degraded+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 4.0 GiB data, 20 GiB used, 784 GiB / 805 GiB avail; 0 B/s wr, 577 op/s; 34016/195309 objects degraded (17.417%); 17389/195309 objects misplaced (8.903%); 20 MiB/s, 327 objects/s recovering 2024-08-19T14:31:40.411 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:31:40.434 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:31:40.467 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell -n osd.2 --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --op export --pgid 1.5 --file /var/log/ceph/exp.1.5.2 2024-08-19T14:31:40.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:40.469 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:40.469 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:31:40.570 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:31:40.602 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:40.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:40.603 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:31:40.704 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:31:40.739 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:40.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:40.740 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:31:40.841 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:31:40.867 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:40.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:40.869 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:31:40.970 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:31:41.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:41.011 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:41.011 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:31:41.112 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:31:41.134 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:41.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:41.135 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:31:41.185 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:31:41.236 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:31:41.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:41.273 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:41.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:31:41.374 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:31:41.398 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:41.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:41.399 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:31:41.499 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:31:41.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:41.534 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:41.535 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:31:41.635 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:31:41.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:41.674 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:41.675 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:31:41.776 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:31:41.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:41.815 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:41.815 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:31:41.917 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:31:41.949 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:41.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:41.950 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:31:42.052 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:31:42.076 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:42.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:42.077 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:31:42.178 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:31:42.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:42.211 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:42.211 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:31:42.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:41 smithi032 ceph-mon[20004]: pgmap v1323: 47 pgs: 5 active+recovering+degraded+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 3.8 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 580 op/s; 32911/186738 objects degraded (17.624%); 16744/186738 objects misplaced (8.967%); 21 MiB/s, 357 objects/s recovering 2024-08-19T14:31:42.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:41 smithi045 ceph-mon[20143]: pgmap v1323: 47 pgs: 5 active+recovering+degraded+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 3.8 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 580 op/s; 32911/186738 objects degraded (17.624%); 16744/186738 objects misplaced (8.967%); 21 MiB/s, 357 objects/s recovering 2024-08-19T14:31:42.312 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:31:42.346 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:42.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:42.347 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:31:42.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:41 smithi031 ceph-mon[16336]: pgmap v1323: 47 pgs: 5 active+recovering+degraded+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 3.8 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 580 op/s; 32911/186738 objects degraded (17.624%); 16744/186738 objects misplaced (8.967%); 21 MiB/s, 357 objects/s recovering 2024-08-19T14:31:42.448 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:31:42.486 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:42.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:42.487 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:31:42.524 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/osd.2/config 2024-08-19T14:31:42.537 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:31:42.588 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:31:42.628 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:42.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:42.629 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:31:42.730 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:31:42.753 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:42.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:42.754 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:31:42.855 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:31:42.891 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:42.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:42.892 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:31:42.993 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:31:43.020 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:43.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:43.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:31:43.122 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:31:43.147 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:43.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:43.148 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:31:43.249 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:31:43.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:42 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 32911/186738 objects degraded (17.624%), 12 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-19T14:31:43.274 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:43.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:43.275 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:31:43.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:42 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 32911/186738 objects degraded (17.624%), 12 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-19T14:31:43.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:42 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 32911/186738 objects degraded (17.624%), 12 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-19T14:31:43.375 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:31:43.407 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:43.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:43.408 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:31:43.509 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:31:43.553 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:43.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:43.555 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:31:43.656 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:31:43.688 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:43.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:43.689 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:31:43.789 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:31:43.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:43.823 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:43.823 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:31:43.925 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:31:43.965 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:43.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:43.966 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:31:44.067 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:31:44.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:44.097 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:44.098 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:31:44.200 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:31:44.232 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:44.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:44.233 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:31:44.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:43 smithi032 ceph-mon[20004]: pgmap v1324: 47 pgs: 5 active+recovering+degraded+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 3.8 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 581 op/s; 32718/186159 objects degraded (17.575%); 16744/186159 objects misplaced (8.994%); 21 MiB/s, 357 objects/s recovering 2024-08-19T14:31:44.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:43 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:31:44.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:43 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:31:44.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:43 smithi032 ceph-mon[20004]: osdmap e571: 12 total, 11 up, 9 in 2024-08-19T14:31:44.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:43 smithi045 ceph-mon[20143]: pgmap v1324: 47 pgs: 5 active+recovering+degraded+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 3.8 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 581 op/s; 32718/186159 objects degraded (17.575%); 16744/186159 objects misplaced (8.994%); 21 MiB/s, 357 objects/s recovering 2024-08-19T14:31:44.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:43 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:31:44.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:43 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:31:44.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:43 smithi045 ceph-mon[20143]: osdmap e571: 12 total, 11 up, 9 in 2024-08-19T14:31:44.334 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:31:44.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:43 smithi031 ceph-mon[16336]: pgmap v1324: 47 pgs: 5 active+recovering+degraded+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 3.8 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 581 op/s; 32718/186159 objects degraded (17.575%); 16744/186159 objects misplaced (8.994%); 21 MiB/s, 357 objects/s recovering 2024-08-19T14:31:44.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:43 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:31:44.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:43 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:31:44.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:43 smithi031 ceph-mon[16336]: osdmap e571: 12 total, 11 up, 9 in 2024-08-19T14:31:44.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:44.368 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:44.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:31:44.469 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:31:44.493 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:44.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:44.494 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:31:44.595 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:31:44.631 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:44.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:44.632 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:31:44.637 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:31:44.733 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:31:44.770 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:44.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:44.771 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:31:44.872 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:31:44.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:44.908 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:44.908 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:31:44.985 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is set 2024-08-19T14:31:45.009 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:31:45.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:45.062 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:45.062 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:31:45.163 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:31:45.196 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:45.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:45.197 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:31:45.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:44 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3839563241' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:31:45.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:44 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3839563241' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:31:45.298 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:31:45.338 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:45.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:45.340 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:31:45.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:44 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3839563241' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:31:45.441 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:31:45.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:45.478 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:45.478 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:31:45.579 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:31:45.613 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:45.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:45.614 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:31:45.715 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:31:45.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:45.766 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:45.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:31:45.867 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:31:45.891 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:45.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:45.892 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:31:45.993 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:31:46.033 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:46.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:46.034 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:31:46.135 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:31:46.172 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:46.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:46.173 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:31:46.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:45 smithi032 ceph-mon[20004]: pgmap v1326: 47 pgs: 5 active+recovering+degraded+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 3.7 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 510 op/s; 31628/181767 objects degraded (17.400%); 16744/181767 objects misplaced (9.212%); 19 MiB/s, 380 objects/s recovering 2024-08-19T14:31:46.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:45 smithi032 ceph-mon[20004]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:31:46.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:45 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3839563241' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:31:46.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:45 smithi032 ceph-mon[20004]: osdmap e572: 12 total, 11 up, 9 in 2024-08-19T14:31:46.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:45 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:31:46.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:45 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:31:46.273 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:31:46.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:45 smithi045 ceph-mon[20143]: pgmap v1326: 47 pgs: 5 active+recovering+degraded+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 3.7 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 510 op/s; 31628/181767 objects degraded (17.400%); 16744/181767 objects misplaced (9.212%); 19 MiB/s, 380 objects/s recovering 2024-08-19T14:31:46.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:45 smithi045 ceph-mon[20143]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:31:46.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:45 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3839563241' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:31:46.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:45 smithi045 ceph-mon[20143]: osdmap e572: 12 total, 11 up, 9 in 2024-08-19T14:31:46.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:45 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:31:46.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:45 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:31:46.297 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:46.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:46.298 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:31:46.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:45 smithi031 ceph-mon[16336]: pgmap v1326: 47 pgs: 5 active+recovering+degraded+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 3.7 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 510 op/s; 31628/181767 objects degraded (17.400%); 16744/181767 objects misplaced (9.212%); 19 MiB/s, 380 objects/s recovering 2024-08-19T14:31:46.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:45 smithi031 ceph-mon[16336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:31:46.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:45 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3839563241' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:31:46.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:45 smithi031 ceph-mon[16336]: osdmap e572: 12 total, 11 up, 9 in 2024-08-19T14:31:46.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:45 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:31:46.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:45 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:31:46.399 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:31:46.440 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:46.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:46.441 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:31:46.542 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:31:46.576 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:46.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:46.577 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:31:46.678 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:31:46.708 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:46.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:46.709 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:31:46.810 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:31:46.841 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:46.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:46.842 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:31:46.943 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:31:46.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:46.987 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:46.987 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:31:47.088 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:31:47.125 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:47.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:47.126 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:31:47.227 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:31:47.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:47 smithi032 ceph-mon[20004]: osdmap e573: 12 total, 11 up, 9 in 2024-08-19T14:31:47.258 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:47.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:47.259 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:31:47.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:47 smithi045 ceph-mon[20143]: osdmap e573: 12 total, 11 up, 9 in 2024-08-19T14:31:47.360 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:31:47.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:47 smithi031 ceph-mon[16336]: osdmap e573: 12 total, 11 up, 9 in 2024-08-19T14:31:47.383 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:47.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:47.384 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:31:47.485 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:31:47.523 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:47.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:47.524 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:31:47.625 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:31:47.663 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:47.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:47.664 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:31:47.765 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:31:47.795 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:47.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:47.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:31:47.897 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:31:47.930 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:47.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:47.931 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:31:48.032 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:31:48.054 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:48.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:48.055 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:31:48.156 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:31:48.188 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:48.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:48.189 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:31:48.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:48 smithi032 ceph-mon[20004]: pgmap v1329: 47 pgs: 5 active+recovering+degraded+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 3.6 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 524 op/s; 31451/177288 objects degraded (17.740%); 16089/177288 objects misplaced (9.075%); 15 MiB/s, 339 objects/s recovering 2024-08-19T14:31:48.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:48 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:31:48.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:48 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:31:48.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:48 smithi032 ceph-mon[20004]: osdmap e574: 12 total, 11 up, 9 in 2024-08-19T14:31:48.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:48 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 31451/177288 objects degraded (17.740%), 12 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-19T14:31:48.290 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:31:48.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:48 smithi045 ceph-mon[20143]: pgmap v1329: 47 pgs: 5 active+recovering+degraded+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 3.6 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 524 op/s; 31451/177288 objects degraded (17.740%); 16089/177288 objects misplaced (9.075%); 15 MiB/s, 339 objects/s recovering 2024-08-19T14:31:48.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:48 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:31:48.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:48 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:31:48.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:48 smithi045 ceph-mon[20143]: osdmap e574: 12 total, 11 up, 9 in 2024-08-19T14:31:48.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:48 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 31451/177288 objects degraded (17.740%), 12 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-19T14:31:48.325 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:48.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:48.327 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:31:48.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:48 smithi031 ceph-mon[16336]: pgmap v1329: 47 pgs: 5 active+recovering+degraded+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 3.6 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 524 op/s; 31451/177288 objects degraded (17.740%); 16089/177288 objects misplaced (9.075%); 15 MiB/s, 339 objects/s recovering 2024-08-19T14:31:48.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:48 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:31:48.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:48 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:31:48.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:48 smithi031 ceph-mon[16336]: osdmap e574: 12 total, 11 up, 9 in 2024-08-19T14:31:48.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:48 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 31451/177288 objects degraded (17.740%), 12 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-19T14:31:48.428 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:31:48.450 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:48.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:48.451 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:31:48.552 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:31:48.588 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:48.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:48.590 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:31:48.691 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:31:48.714 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:48.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:48.715 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:31:48.816 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:31:48.847 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:48.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:48.848 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:31:48.949 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:31:48.991 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:48.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:48.992 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:31:49.093 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:31:49.117 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:49.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:49.119 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:31:49.219 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:31:49.252 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:49.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:49.253 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:31:49.354 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:31:49.378 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:49.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:49.379 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:31:49.480 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:31:49.514 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:49.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:49.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:31:49.616 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:31:49.639 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:49.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:49.640 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:31:49.741 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:31:49.778 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:49.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:49.779 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:31:49.880 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:31:49.914 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:49.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:49.915 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:31:50.016 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:31:50.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:50.058 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:50.058 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:31:50.159 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:31:50.182 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:50.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:50.183 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:31:50.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:50 smithi032 ceph-mon[20004]: pgmap v1331: 47 pgs: 5 active+recovering+degraded+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 3.6 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 577 op/s; 31451/177288 objects degraded (17.740%); 16089/177288 objects misplaced (9.075%); 18 MiB/s, 397 objects/s recovering 2024-08-19T14:31:50.283 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:31:50.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:50 smithi045 ceph-mon[20143]: pgmap v1331: 47 pgs: 5 active+recovering+degraded+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 3.6 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 577 op/s; 31451/177288 objects degraded (17.740%); 16089/177288 objects misplaced (9.075%); 18 MiB/s, 397 objects/s recovering 2024-08-19T14:31:50.305 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:50.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:50.306 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:31:50.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:50 smithi031 ceph-mon[16336]: pgmap v1331: 47 pgs: 5 active+recovering+degraded+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 3.6 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 577 op/s; 31451/177288 objects degraded (17.740%); 16089/177288 objects misplaced (9.075%); 18 MiB/s, 397 objects/s recovering 2024-08-19T14:31:50.407 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:31:50.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:50.430 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:50.430 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:31:50.531 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:31:50.553 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:50.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:50.554 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:31:50.655 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:31:50.700 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:50.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:50.701 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:31:50.802 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:31:50.834 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:50.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:50.835 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:31:50.936 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:31:50.960 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:50.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:50.961 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:31:51.062 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:31:51.082 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:51.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:51.083 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:31:51.184 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:31:51.206 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:51.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:51.208 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:31:51.309 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:31:51.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:51.347 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:51.347 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:31:51.448 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:31:51.480 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:51.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:51.482 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:31:51.582 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:31:51.604 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:51.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:51.605 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:31:51.706 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:31:51.735 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:51.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:51.737 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:31:51.838 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:31:51.872 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:51.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:51.873 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:31:51.974 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:31:51.996 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:51.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:51.997 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:31:52.098 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:31:52.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:52.132 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:52.132 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:31:52.233 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:31:52.267 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:52.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:52.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:31:52.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:52 smithi045 ceph-mon[20143]: pgmap v1332: 47 pgs: 5 active+recovering+degraded+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 3.4 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 722 op/s; 30331/168750 objects degraded (17.974%); 15442/168750 objects misplaced (9.151%); 13 MiB/s, 452 objects/s recovering 2024-08-19T14:31:52.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:52 smithi031 ceph-mon[16336]: pgmap v1332: 47 pgs: 5 active+recovering+degraded+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 3.4 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 722 op/s; 30331/168750 objects degraded (17.974%); 15442/168750 objects misplaced (9.151%); 13 MiB/s, 452 objects/s recovering 2024-08-19T14:31:52.369 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:31:52.399 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:52.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:52.401 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:31:52.502 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:31:52.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:52 smithi032 ceph-mon[20004]: pgmap v1332: 47 pgs: 5 active+recovering+degraded+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 3.4 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 722 op/s; 30331/168750 objects degraded (17.974%); 15442/168750 objects misplaced (9.151%); 13 MiB/s, 452 objects/s recovering 2024-08-19T14:31:52.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:52.538 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:52.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:31:52.640 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:31:52.674 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:52.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:52.676 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:31:52.777 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:31:52.811 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:52.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:52.812 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:31:52.913 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:31:52.948 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:52.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:52.949 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:31:53.050 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:31:53.095 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:53.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:53.096 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:31:53.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:53 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 30331/168750 objects degraded (17.974%), 12 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-19T14:31:53.196 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:31:53.234 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:53.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:53.235 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:31:53.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:53 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 30331/168750 objects degraded (17.974%), 12 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-19T14:31:53.336 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:31:53.368 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:53.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:53.369 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:31:53.470 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:31:53.493 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:53.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:53.494 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:31:53.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:53 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 30331/168750 objects degraded (17.974%), 12 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-19T14:31:53.596 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:31:53.617 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:53.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:53.618 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:31:53.720 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:31:53.759 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:53.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:53.760 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:31:53.861 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:31:53.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:53.901 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:53.901 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:31:54.002 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:31:54.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:54.038 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:54.038 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:31:54.140 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:31:54.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:54.180 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:54.181 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:31:54.282 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:31:54.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:54 smithi045 ceph-mon[20143]: pgmap v1333: 47 pgs: 5 active+recovering+degraded+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 3.4 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 452 op/s; 30152/168213 objects degraded (17.925%); 15442/168213 objects misplaced (9.180%); 7.9 MiB/s, 331 objects/s recovering 2024-08-19T14:31:54.316 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:54.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:54.317 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:31:54.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:54 smithi031 ceph-mon[16336]: pgmap v1333: 47 pgs: 5 active+recovering+degraded+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 3.4 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 452 op/s; 30152/168213 objects degraded (17.925%); 15442/168213 objects misplaced (9.180%); 7.9 MiB/s, 331 objects/s recovering 2024-08-19T14:31:54.418 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:31:54.439 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:54.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:54.440 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:31:54.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:54 smithi032 ceph-mon[20004]: pgmap v1333: 47 pgs: 5 active+recovering+degraded+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 3.4 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 452 op/s; 30152/168213 objects degraded (17.925%); 15442/168213 objects misplaced (9.180%); 7.9 MiB/s, 331 objects/s recovering 2024-08-19T14:31:54.541 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:31:54.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:54.579 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:54.579 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:31:54.681 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:31:54.702 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:54.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:54.703 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:31:54.805 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:31:54.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:54.854 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:54.854 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:31:54.955 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:31:54.977 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:54.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:54.978 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:31:55.079 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:31:55.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:55.101 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:55.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:31:55.202 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:31:55.223 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:55.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:55.224 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:31:55.325 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:31:55.372 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:55.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:55.373 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:31:55.474 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:31:55.495 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:55.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:55.496 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:31:55.597 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:31:55.638 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:55.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:55.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:31:55.740 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:31:55.762 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:55.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:55.763 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:31:55.865 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:31:55.886 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:55.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:55.887 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:31:55.988 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:31:56.011 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:56.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:56.012 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:31:56.113 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:31:56.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:56.151 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:56.151 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:31:56.252 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:31:56.288 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:56.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:56.289 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:31:56.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:56 smithi045 ceph-mon[20143]: pgmap v1334: 47 pgs: 1 active+clean+remapped, 4 active+recovering+degraded+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 3.3 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 577 op/s; 25555/163236 objects degraded (15.655%); 18801/163236 objects misplaced (11.518%); 7.0 MiB/s, 467 objects/s recovering 2024-08-19T14:31:56.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:56 smithi031 ceph-mon[16336]: pgmap v1334: 47 pgs: 1 active+clean+remapped, 4 active+recovering+degraded+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 3.3 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 577 op/s; 25555/163236 objects degraded (15.655%); 18801/163236 objects misplaced (11.518%); 7.0 MiB/s, 467 objects/s recovering 2024-08-19T14:31:56.390 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:31:56.413 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:56.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:56.414 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:31:56.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:56 smithi032 ceph-mon[20004]: pgmap v1334: 47 pgs: 1 active+clean+remapped, 4 active+recovering+degraded+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 3.3 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 577 op/s; 25555/163236 objects degraded (15.655%); 18801/163236 objects misplaced (11.518%); 7.0 MiB/s, 467 objects/s recovering 2024-08-19T14:31:56.515 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:31:56.562 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:56.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:56.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:31:56.664 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:31:56.686 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:56.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:56.687 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:31:56.787 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:31:56.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:56.822 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:56.822 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:31:56.923 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:31:56.957 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:56.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:56.958 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:31:57.059 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:31:57.093 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:57.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:57.094 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:31:57.195 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:31:57.217 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:57.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:57.218 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:31:57.319 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:31:57.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:57.356 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:57.356 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:31:57.457 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:31:57.502 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:57.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:57.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:31:57.604 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:31:57.640 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:57.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:57.642 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:31:57.743 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:31:57.779 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:57.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:57.780 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:31:57.881 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:31:57.906 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:57.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:57.907 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:31:58.008 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:31:58.040 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:58.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:58.041 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:31:58.142 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:31:58.165 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:58.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:58.166 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:31:58.267 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:31:58.288 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:58.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:58.289 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:31:58.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:58 smithi045 ceph-mon[20143]: pgmap v1335: 47 pgs: 1 active+clean+remapped, 4 active+recovering+degraded+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 3.2 GiB data, 18 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 639 op/s; 25379/158100 objects degraded (16.052%); 18108/158100 objects misplaced (11.454%); 8.0 MiB/s, 416 objects/s recovering 2024-08-19T14:31:58.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:58 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:31:58.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:58 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:31:58.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:58 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 25379/158100 objects degraded (16.052%), 11 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-19T14:31:58.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:58 smithi031 ceph-mon[16336]: pgmap v1335: 47 pgs: 1 active+clean+remapped, 4 active+recovering+degraded+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 3.2 GiB data, 18 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 639 op/s; 25379/158100 objects degraded (16.052%); 18108/158100 objects misplaced (11.454%); 8.0 MiB/s, 416 objects/s recovering 2024-08-19T14:31:58.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:58 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:31:58.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:58 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:31:58.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:58 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 25379/158100 objects degraded (16.052%), 11 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-19T14:31:58.391 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:31:58.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:58.432 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:58.432 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:31:58.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:58 smithi032 ceph-mon[20004]: pgmap v1335: 47 pgs: 1 active+clean+remapped, 4 active+recovering+degraded+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 3.2 GiB data, 18 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 639 op/s; 25379/158100 objects degraded (16.052%); 18108/158100 objects misplaced (11.454%); 8.0 MiB/s, 416 objects/s recovering 2024-08-19T14:31:58.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:58 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:31:58.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:58 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:31:58.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:58 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 25379/158100 objects degraded (16.052%), 11 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-19T14:31:58.533 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:31:58.566 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:58.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:58.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:31:58.668 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:31:58.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:58.691 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:58.691 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:31:58.792 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:31:58.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:58.831 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:58.832 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:31:58.933 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:31:58.969 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:58.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:58.970 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:31:59.071 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:31:59.100 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:31:59.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:59.101 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:31:59.202 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:31:59.225 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:59.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:59.226 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:31:59.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:31:59 smithi045 ceph-mon[20143]: osdmap e575: 12 total, 11 up, 9 in 2024-08-19T14:31:59.327 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:31:59.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:59.367 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:59.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:31:59.368 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:31:59 smithi031 ceph-mon[16336]: osdmap e575: 12 total, 11 up, 9 in 2024-08-19T14:31:59.468 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:31:59.502 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:59.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:59.503 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:31:59.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:31:59 smithi032 ceph-mon[20004]: osdmap e575: 12 total, 11 up, 9 in 2024-08-19T14:31:59.604 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:31:59.626 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:59.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:59.627 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:31:59.728 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:31:59.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:59.763 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:31:59.763 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:31:59.864 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:31:59.886 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:31:59.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:31:59.887 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:31:59.988 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:32:00.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:00.027 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:00.027 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:32:00.128 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:32:00.163 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:00.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:00.164 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:32:00.265 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:32:00.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:00 smithi045 ceph-mon[20143]: pgmap v1337: 47 pgs: 1 active+clean+remapped, 4 active+recovering+degraded+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 3.2 GiB data, 18 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 639 op/s; 25379/158100 objects degraded (16.052%); 18108/158100 objects misplaced (11.454%); 8.0 MiB/s, 416 objects/s recovering 2024-08-19T14:32:00.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:00 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:32:00.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:00 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:32:00.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:00 smithi045 ceph-mon[20143]: osdmap e576: 12 total, 11 up, 9 in 2024-08-19T14:32:00.305 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:00.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:00.306 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:32:00.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:00 smithi031 ceph-mon[16336]: pgmap v1337: 47 pgs: 1 active+clean+remapped, 4 active+recovering+degraded+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 3.2 GiB data, 18 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 639 op/s; 25379/158100 objects degraded (16.052%); 18108/158100 objects misplaced (11.454%); 8.0 MiB/s, 416 objects/s recovering 2024-08-19T14:32:00.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:00 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:32:00.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:00 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:32:00.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:00 smithi031 ceph-mon[16336]: osdmap e576: 12 total, 11 up, 9 in 2024-08-19T14:32:00.407 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:32:00.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:00.440 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:00.441 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:32:00.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:00 smithi032 ceph-mon[20004]: pgmap v1337: 47 pgs: 1 active+clean+remapped, 4 active+recovering+degraded+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 3.2 GiB data, 18 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 639 op/s; 25379/158100 objects degraded (16.052%); 18108/158100 objects misplaced (11.454%); 8.0 MiB/s, 416 objects/s recovering 2024-08-19T14:32:00.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:00 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:32:00.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:00 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:32:00.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:00 smithi032 ceph-mon[20004]: osdmap e576: 12 total, 11 up, 9 in 2024-08-19T14:32:00.542 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:32:00.578 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:00.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:00.579 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:32:00.680 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:32:00.703 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:00.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:00.704 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:32:00.805 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:32:00.827 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:00.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:00.829 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:32:00.930 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:32:00.953 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:00.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:00.954 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:32:01.055 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:32:01.095 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:01.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:01.099 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:32:01.200 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:32:01.233 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:01.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:01.234 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:32:01.335 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:32:01.363 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:01.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:01.364 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:32:01.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:01 smithi031 ceph-mon[16336]: pgmap v1339: 47 pgs: 1 active+clean+remapped, 4 active+recovering+degraded+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 3.0 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 815 op/s; 24278/148632 objects degraded (16.334%); 17212/148632 objects misplaced (11.580%); 5.7 MiB/s, 450 objects/s recovering 2024-08-19T14:32:01.465 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:32:01.502 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:01.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:01.503 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:32:01.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:01 smithi032 ceph-mon[20004]: pgmap v1339: 47 pgs: 1 active+clean+remapped, 4 active+recovering+degraded+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 3.0 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 815 op/s; 24278/148632 objects degraded (16.334%); 17212/148632 objects misplaced (11.580%); 5.7 MiB/s, 450 objects/s recovering 2024-08-19T14:32:01.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:01 smithi045 ceph-mon[20143]: pgmap v1339: 47 pgs: 1 active+clean+remapped, 4 active+recovering+degraded+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 3.0 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 815 op/s; 24278/148632 objects degraded (16.334%); 17212/148632 objects misplaced (11.580%); 5.7 MiB/s, 450 objects/s recovering 2024-08-19T14:32:01.604 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:32:01.626 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:01.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:01.627 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:32:01.728 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:32:01.763 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:01.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:01.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:32:01.865 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:32:01.906 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:01.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:01.907 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:32:02.008 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:32:02.047 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:02.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:02.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:32:02.148 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:32:02.190 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:02.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:02.191 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:32:02.292 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:32:02.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:02.328 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:02.328 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:32:02.429 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:32:02.461 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:02.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:02.462 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:32:02.563 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:32:02.635 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:02.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:02.637 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:32:02.739 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:32:02.760 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:02.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:02.761 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:32:02.862 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:32:02.899 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:02.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:02.900 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:32:03.001 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:32:03.025 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:03.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:03.026 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:32:03.127 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:32:03.151 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:03.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:03.152 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:32:03.253 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:32:03.283 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:03 smithi031 ceph-mon[16336]: pgmap v1340: 47 pgs: 1 active+clean+remapped, 4 active+recovering+degraded+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 3.0 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 632 op/s; 24085/148053 objects degraded (16.268%); 17212/148053 objects misplaced (11.626%); 5.7 MiB/s, 270 objects/s recovering 2024-08-19T14:32:03.283 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:03 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 24085/148053 objects degraded (16.268%), 11 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-19T14:32:03.283 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:03 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3610087645' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:32:03.283 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {auth=32,config=2,health=12,kv=54} crc {auth=218401245,config=2267314355,health=1468064291,kv=2832160264}) 2024-08-19T14:32:03.283 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=3489745205}) 2024-08-19T14:32:03.283 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=3070578288}) 2024-08-19T14:32:03.283 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {kv=48,logm=52} crc {kv=2334889970,logm=234087443}) 2024-08-19T14:32:03.283 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2879071695}) 2024-08-19T14:32:03.283 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3473792255}) 2024-08-19T14:32:03.283 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=222693128}) 2024-08-19T14:32:03.283 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=845423194}) 2024-08-19T14:32:03.284 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=27918551}) 2024-08-19T14:32:03.284 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4290934247}) 2024-08-19T14:32:03.284 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3840548831}) 2024-08-19T14:32:03.284 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=505232905}) 2024-08-19T14:32:03.284 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=286712508}) 2024-08-19T14:32:03.284 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1543795485}) 2024-08-19T14:32:03.284 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1319764447}) 2024-08-19T14:32:03.284 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1168181955}) 2024-08-19T14:32:03.284 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3967512161}) 2024-08-19T14:32:03.284 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3210932549}) 2024-08-19T14:32:03.284 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=773971113}) 2024-08-19T14:32:03.284 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1114173028}) 2024-08-19T14:32:03.285 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2971746184}) 2024-08-19T14:32:03.285 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4103574727}) 2024-08-19T14:32:03.285 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=258450147}) 2024-08-19T14:32:03.285 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1408261252}) 2024-08-19T14:32:03.285 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2297225835}) 2024-08-19T14:32:03.285 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2477496122}) 2024-08-19T14:32:03.285 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4027765653}) 2024-08-19T14:32:03.285 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=339215316}) 2024-08-19T14:32:03.285 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3889628903}) 2024-08-19T14:32:03.285 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2962922828}) 2024-08-19T14:32:03.285 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2951093685}) 2024-08-19T14:32:03.286 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1586911583}) 2024-08-19T14:32:03.286 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=592864079}) 2024-08-19T14:32:03.286 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3081125197}) 2024-08-19T14:32:03.286 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2380059923}) 2024-08-19T14:32:03.286 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=113615402}) 2024-08-19T14:32:03.286 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=546582214}) 2024-08-19T14:32:03.286 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=665216842}) 2024-08-19T14:32:03.286 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2501624066}) 2024-08-19T14:32:03.286 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=33668620}) 2024-08-19T14:32:03.286 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1702469355}) 2024-08-19T14:32:03.286 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3267211740}) 2024-08-19T14:32:03.287 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3601678667}) 2024-08-19T14:32:03.287 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1281040670}) 2024-08-19T14:32:03.287 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=907429787}) 2024-08-19T14:32:03.287 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1224356290}) 2024-08-19T14:32:03.287 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1663115848}) 2024-08-19T14:32:03.287 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3394328553}) 2024-08-19T14:32:03.287 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2086061696}) 2024-08-19T14:32:03.287 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3595160342}) 2024-08-19T14:32:03.287 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2785819872}) 2024-08-19T14:32:03.287 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3383912192}) 2024-08-19T14:32:03.287 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1821934333}) 2024-08-19T14:32:03.287 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=931154433}) 2024-08-19T14:32:03.287 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1422866155}) 2024-08-19T14:32:03.287 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1603833002}) 2024-08-19T14:32:03.288 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=145016258}) 2024-08-19T14:32:03.288 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3851361580}) 2024-08-19T14:32:03.288 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=229312402}) 2024-08-19T14:32:03.288 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=32,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=11,mon_config_key=33} crc {logm=3996198492,mdsmap=2474562478,mgr=2924679688,mgr_command_descs=1300230762,mgr_metadata=482233419,mgrstat=1640475107,mon_config_key=2171626434}) 2024-08-19T14:32:03.288 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2543886440}) 2024-08-19T14:32:03.288 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=4212897187}) 2024-08-19T14:32:03.288 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=43,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=39} crc {mon_config_key=448982093,monmap=3193798142,osd_metadata=2015703563,osd_pg_creating=1878909250,osdmap=265225965}) 2024-08-19T14:32:03.288 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1776839714}) 2024-08-19T14:32:03.288 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=813833144}) 2024-08-19T14:32:03.288 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2822093294}) 2024-08-19T14:32:03.288 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2614990451}) 2024-08-19T14:32:03.288 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3326533406}) 2024-08-19T14:32:03.288 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1145689043}) 2024-08-19T14:32:03.288 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2613221977}) 2024-08-19T14:32:03.288 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3610531805}) 2024-08-19T14:32:03.289 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2810223649}) 2024-08-19T14:32:03.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:03.290 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:03.290 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:32:03.391 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:32:03.416 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:03.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:03.417 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:32:03.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:03 smithi032 ceph-mon[20004]: pgmap v1340: 47 pgs: 1 active+clean+remapped, 4 active+recovering+degraded+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 3.0 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 632 op/s; 24085/148053 objects degraded (16.268%); 17212/148053 objects misplaced (11.626%); 5.7 MiB/s, 270 objects/s recovering 2024-08-19T14:32:03.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:03 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 24085/148053 objects degraded (16.268%), 11 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-19T14:32:03.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:03 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3610087645' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:32:03.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {auth=32,config=2,health=12,kv=54} crc {auth=218401245,config=2267314355,health=1468064291,kv=2832160264}) 2024-08-19T14:32:03.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=3489745205}) 2024-08-19T14:32:03.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=3070578288}) 2024-08-19T14:32:03.509 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {kv=48,logm=52} crc {kv=2334889970,logm=234087443}) 2024-08-19T14:32:03.509 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2879071695}) 2024-08-19T14:32:03.509 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3473792255}) 2024-08-19T14:32:03.509 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=222693128}) 2024-08-19T14:32:03.509 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=845423194}) 2024-08-19T14:32:03.509 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=27918551}) 2024-08-19T14:32:03.509 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4290934247}) 2024-08-19T14:32:03.509 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3840548831}) 2024-08-19T14:32:03.509 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=505232905}) 2024-08-19T14:32:03.509 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=286712508}) 2024-08-19T14:32:03.509 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1543795485}) 2024-08-19T14:32:03.509 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1319764447}) 2024-08-19T14:32:03.509 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1168181955}) 2024-08-19T14:32:03.510 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3967512161}) 2024-08-19T14:32:03.510 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3210932549}) 2024-08-19T14:32:03.510 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=773971113}) 2024-08-19T14:32:03.510 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1114173028}) 2024-08-19T14:32:03.510 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2971746184}) 2024-08-19T14:32:03.510 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4103574727}) 2024-08-19T14:32:03.510 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=258450147}) 2024-08-19T14:32:03.510 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1408261252}) 2024-08-19T14:32:03.510 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2297225835}) 2024-08-19T14:32:03.510 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2477496122}) 2024-08-19T14:32:03.510 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4027765653}) 2024-08-19T14:32:03.511 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=339215316}) 2024-08-19T14:32:03.511 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3889628903}) 2024-08-19T14:32:03.511 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2962922828}) 2024-08-19T14:32:03.511 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2951093685}) 2024-08-19T14:32:03.511 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1586911583}) 2024-08-19T14:32:03.511 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=592864079}) 2024-08-19T14:32:03.512 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3081125197}) 2024-08-19T14:32:03.512 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2380059923}) 2024-08-19T14:32:03.512 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=113615402}) 2024-08-19T14:32:03.512 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=546582214}) 2024-08-19T14:32:03.512 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=665216842}) 2024-08-19T14:32:03.512 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2501624066}) 2024-08-19T14:32:03.512 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=33668620}) 2024-08-19T14:32:03.512 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1702469355}) 2024-08-19T14:32:03.512 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3267211740}) 2024-08-19T14:32:03.512 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3601678667}) 2024-08-19T14:32:03.512 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1281040670}) 2024-08-19T14:32:03.512 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=907429787}) 2024-08-19T14:32:03.512 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1224356290}) 2024-08-19T14:32:03.513 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1663115848}) 2024-08-19T14:32:03.513 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3394328553}) 2024-08-19T14:32:03.513 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2086061696}) 2024-08-19T14:32:03.513 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3595160342}) 2024-08-19T14:32:03.513 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2785819872}) 2024-08-19T14:32:03.513 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3383912192}) 2024-08-19T14:32:03.513 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1821934333}) 2024-08-19T14:32:03.513 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=931154433}) 2024-08-19T14:32:03.513 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1422866155}) 2024-08-19T14:32:03.513 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1603833002}) 2024-08-19T14:32:03.513 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=145016258}) 2024-08-19T14:32:03.513 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3851361580}) 2024-08-19T14:32:03.513 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=229312402}) 2024-08-19T14:32:03.513 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=32,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=11,mon_config_key=33} crc {logm=3996198492,mdsmap=2474562478,mgr=2924679688,mgr_command_descs=1300230762,mgr_metadata=482233419,mgrstat=1640475107,mon_config_key=2171626434}) 2024-08-19T14:32:03.514 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2543886440}) 2024-08-19T14:32:03.514 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=4212897187}) 2024-08-19T14:32:03.514 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=43,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=39} crc {mon_config_key=448982093,monmap=3193798142,osd_metadata=2015703563,osd_pg_creating=1878909250,osdmap=265225965}) 2024-08-19T14:32:03.514 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1776839714}) 2024-08-19T14:32:03.514 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=813833144}) 2024-08-19T14:32:03.514 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2822093294}) 2024-08-19T14:32:03.514 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2614990451}) 2024-08-19T14:32:03.514 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3326533406}) 2024-08-19T14:32:03.514 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1145689043}) 2024-08-19T14:32:03.514 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2613221977}) 2024-08-19T14:32:03.514 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3610531805}) 2024-08-19T14:32:03.514 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2810223649}) 2024-08-19T14:32:03.518 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:32:03.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:03 smithi045 ceph-mon[20143]: pgmap v1340: 47 pgs: 1 active+clean+remapped, 4 active+recovering+degraded+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 3.0 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 632 op/s; 24085/148053 objects degraded (16.268%); 17212/148053 objects misplaced (11.626%); 5.7 MiB/s, 270 objects/s recovering 2024-08-19T14:32:03.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:03 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 24085/148053 objects degraded (16.268%), 11 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-19T14:32:03.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:03 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3610087645' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:32:03.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {auth=32,config=2,health=12,kv=54} crc {auth=218401245,config=2267314355,health=1468064291,kv=2832160264}) 2024-08-19T14:32:03.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=3489745205}) 2024-08-19T14:32:03.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=3070578288}) 2024-08-19T14:32:03.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {kv=48,logm=52} crc {kv=2334889970,logm=234087443}) 2024-08-19T14:32:03.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2879071695}) 2024-08-19T14:32:03.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3473792255}) 2024-08-19T14:32:03.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=222693128}) 2024-08-19T14:32:03.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=845423194}) 2024-08-19T14:32:03.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=27918551}) 2024-08-19T14:32:03.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4290934247}) 2024-08-19T14:32:03.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3840548831}) 2024-08-19T14:32:03.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=505232905}) 2024-08-19T14:32:03.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=286712508}) 2024-08-19T14:32:03.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1543795485}) 2024-08-19T14:32:03.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1319764447}) 2024-08-19T14:32:03.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1168181955}) 2024-08-19T14:32:03.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3967512161}) 2024-08-19T14:32:03.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3210932549}) 2024-08-19T14:32:03.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=773971113}) 2024-08-19T14:32:03.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1114173028}) 2024-08-19T14:32:03.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2971746184}) 2024-08-19T14:32:03.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4103574727}) 2024-08-19T14:32:03.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=258450147}) 2024-08-19T14:32:03.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1408261252}) 2024-08-19T14:32:03.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2297225835}) 2024-08-19T14:32:03.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2477496122}) 2024-08-19T14:32:03.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4027765653}) 2024-08-19T14:32:03.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=339215316}) 2024-08-19T14:32:03.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3889628903}) 2024-08-19T14:32:03.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2962922828}) 2024-08-19T14:32:03.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2951093685}) 2024-08-19T14:32:03.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1586911583}) 2024-08-19T14:32:03.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=592864079}) 2024-08-19T14:32:03.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3081125197}) 2024-08-19T14:32:03.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2380059923}) 2024-08-19T14:32:03.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=113615402}) 2024-08-19T14:32:03.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=546582214}) 2024-08-19T14:32:03.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=665216842}) 2024-08-19T14:32:03.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2501624066}) 2024-08-19T14:32:03.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=33668620}) 2024-08-19T14:32:03.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1702469355}) 2024-08-19T14:32:03.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3267211740}) 2024-08-19T14:32:03.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3601678667}) 2024-08-19T14:32:03.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1281040670}) 2024-08-19T14:32:03.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=907429787}) 2024-08-19T14:32:03.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1224356290}) 2024-08-19T14:32:03.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1663115848}) 2024-08-19T14:32:03.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3394328553}) 2024-08-19T14:32:03.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2086061696}) 2024-08-19T14:32:03.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3595160342}) 2024-08-19T14:32:03.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2785819872}) 2024-08-19T14:32:03.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3383912192}) 2024-08-19T14:32:03.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1821934333}) 2024-08-19T14:32:03.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=931154433}) 2024-08-19T14:32:03.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1422866155}) 2024-08-19T14:32:03.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1603833002}) 2024-08-19T14:32:03.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=145016258}) 2024-08-19T14:32:03.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3851361580}) 2024-08-19T14:32:03.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=229312402}) 2024-08-19T14:32:03.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=32,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=11,mon_config_key=33} crc {logm=3996198492,mdsmap=2474562478,mgr=2924679688,mgr_command_descs=1300230762,mgr_metadata=482233419,mgrstat=1640475107,mon_config_key=2171626434}) 2024-08-19T14:32:03.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2543886440}) 2024-08-19T14:32:03.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=4212897187}) 2024-08-19T14:32:03.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=43,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=39} crc {mon_config_key=448982093,monmap=3193798142,osd_metadata=2015703563,osd_pg_creating=1878909250,osdmap=265225965}) 2024-08-19T14:32:03.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1776839714}) 2024-08-19T14:32:03.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=813833144}) 2024-08-19T14:32:03.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2822093294}) 2024-08-19T14:32:03.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2614990451}) 2024-08-19T14:32:03.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3326533406}) 2024-08-19T14:32:03.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1145689043}) 2024-08-19T14:32:03.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2613221977}) 2024-08-19T14:32:03.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3610531805}) 2024-08-19T14:32:03.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2810223649}) 2024-08-19T14:32:03.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:03.559 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:03.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:32:03.660 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:32:03.681 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:03.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:03.682 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:32:03.784 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:32:03.813 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:03.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:03.814 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:32:03.915 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:32:03.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:03.948 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:03.948 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:32:03.967 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:32:03.967 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:32:03.967 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:32:03.967 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:32:03.967 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:32:03.968 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:32:03.968 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:32:03.968 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-08-19T14:32:03.968 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:32:03.968 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:32:03.968 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:32:03.968 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:32:03.968 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:32:03.968 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:32:03.969 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:32:03.969 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:32:03.969 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:32:03.969 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:32:03.969 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:32:03.969 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:32:03.969 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:32:03.969 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:32:03.969 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:32:04.049 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:32:04.071 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:04.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:04.072 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:32:04.172 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:32:04.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:04.207 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:04.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:32:04.308 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:32:04.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:04.356 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:04.357 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:32:04.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:04 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3347177795}) 2024-08-19T14:32:04.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:04 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3344479576}) 2024-08-19T14:32:04.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:04 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=16} crc {osdmap=1607694944}) 2024-08-19T14:32:04.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:04 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:32:04.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:04 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:32:04.457 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:32:04.489 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:04.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:04.490 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:32:04.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:04 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3347177795}) 2024-08-19T14:32:04.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:04 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3344479576}) 2024-08-19T14:32:04.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:04 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=16} crc {osdmap=1607694944}) 2024-08-19T14:32:04.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:04 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:32:04.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:04 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:32:04.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:04 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3347177795}) 2024-08-19T14:32:04.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:04 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3344479576}) 2024-08-19T14:32:04.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:04 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=16} crc {osdmap=1607694944}) 2024-08-19T14:32:04.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:04 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:32:04.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:04 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:32:04.590 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:32:04.612 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:04.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:04.614 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:32:04.715 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:32:04.737 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:04.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:04.738 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:32:04.839 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:32:04.864 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:04.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:04.865 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:32:04.967 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:32:05.001 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:05.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:05.002 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:32:05.106 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:32:05.146 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:05.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:05.147 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:32:05.248 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:32:05.270 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:05.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:05.271 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:32:05.372 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:32:05.407 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:05.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:05.408 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:32:05.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:05 smithi032 ceph-mon[20004]: pgmap v1341: 47 pgs: 1 active+clean+remapped, 4 active+recovering+degraded+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 2.9 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 623 op/s; 23068/143121 objects degraded (16.118%); 17002/143121 objects misplaced (11.879%); 2.8 MiB/s, 417 objects/s recovering 2024-08-19T14:32:05.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:05 smithi032 ceph-mon[20004]: osdmap e577: 12 total, 11 up, 9 in 2024-08-19T14:32:05.509 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:32:05.530 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:05.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:05.531 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:32:05.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:05 smithi045 ceph-mon[20143]: pgmap v1341: 47 pgs: 1 active+clean+remapped, 4 active+recovering+degraded+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 2.9 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 623 op/s; 23068/143121 objects degraded (16.118%); 17002/143121 objects misplaced (11.879%); 2.8 MiB/s, 417 objects/s recovering 2024-08-19T14:32:05.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:05 smithi045 ceph-mon[20143]: osdmap e577: 12 total, 11 up, 9 in 2024-08-19T14:32:05.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:05 smithi031 ceph-mon[16336]: pgmap v1341: 47 pgs: 1 active+clean+remapped, 4 active+recovering+degraded+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 2.9 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 623 op/s; 23068/143121 objects degraded (16.118%); 17002/143121 objects misplaced (11.879%); 2.8 MiB/s, 417 objects/s recovering 2024-08-19T14:32:05.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:05 smithi031 ceph-mon[16336]: osdmap e577: 12 total, 11 up, 9 in 2024-08-19T14:32:05.632 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:32:05.660 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:05.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:05.661 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:32:05.762 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:32:05.792 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:05.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:05.793 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:32:05.894 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:32:05.917 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:05.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:05.918 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:32:06.019 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:32:06.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:06.042 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:06.042 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:32:06.143 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:32:06.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:06.175 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:06.175 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:32:06.204 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:32:06.204 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:32:06.276 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:32:06.300 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:06.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:06.301 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:32:06.402 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:32:06.424 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:06.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:06.425 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:32:06.526 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:32:06.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:06.551 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:06.551 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:32:06.652 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:32:06.680 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:06.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:06.681 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:32:06.685 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-19T14:32:06.686 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-19T14:32:06.782 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:32:06.815 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:06.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:06.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:32:06.917 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:32:06.946 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:06.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:06.947 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:32:07.048 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:32:07.073 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:07.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:07.074 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:32:07.175 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:32:07.198 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:07.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:07.199 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:32:07.220 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:07 smithi031 ceph-mon[16336]: pgmap v1343: 47 pgs: 1 active+clean+remapped, 4 active+recovering+degraded+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 2.8 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 824 op/s; 22824/138318 objects degraded (16.501%); 16326/138318 objects misplaced (11.803%); 6.0 MiB/s, 469 objects/s recovering 2024-08-19T14:32:07.275 INFO:teuthology.orchestra.run.smithi031.stderr:Creating an OSD daemon form without an OSD FSID value 2024-08-19T14:32:07.300 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:32:07.331 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:07.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:07.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:32:07.433 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:32:07.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:07.471 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:07.471 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:32:07.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:07 smithi032 ceph-mon[20004]: pgmap v1343: 47 pgs: 1 active+clean+remapped, 4 active+recovering+degraded+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 2.8 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 824 op/s; 22824/138318 objects degraded (16.501%); 16326/138318 objects misplaced (11.803%); 6.0 MiB/s, 469 objects/s recovering 2024-08-19T14:32:07.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:07 smithi045 ceph-mon[20143]: pgmap v1343: 47 pgs: 1 active+clean+remapped, 4 active+recovering+degraded+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 2.8 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 824 op/s; 22824/138318 objects degraded (16.501%); 16326/138318 objects misplaced (11.803%); 6.0 MiB/s, 469 objects/s recovering 2024-08-19T14:32:07.573 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:32:07.606 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:07.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:07.607 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:32:07.708 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:32:07.729 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:07.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:07.730 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:32:07.831 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:32:07.867 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:07.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:07.868 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:32:07.906 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-08-19T14:32:07.969 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:32:07.990 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:07.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:07.991 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:32:08.092 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:32:08.129 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:08.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:08.131 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:32:08.231 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:32:08.264 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:08.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:08.265 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:32:08.366 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:32:08.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:08.399 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:08.399 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:32:08.454 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:08 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 22824/138318 objects degraded (16.501%), 11 pgs degraded, 16 pgs undersized (PG_DEGRADED) 2024-08-19T14:32:08.500 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:32:08.501 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:32:08.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:08 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 22824/138318 objects degraded (16.501%), 11 pgs degraded, 16 pgs undersized (PG_DEGRADED) 2024-08-19T14:32:08.535 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:08.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:08.536 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:32:08.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:08 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 22824/138318 objects degraded (16.501%), 11 pgs degraded, 16 pgs undersized (PG_DEGRADED) 2024-08-19T14:32:08.638 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:32:08.661 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:08.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:08.662 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:32:08.686 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:32:08.697 INFO:teuthology.orchestra.run.smithi031.stderr:Exporting 1.5 info 1.5( v 70'139 (0'0,70'139] local-lis/les=558/559 n=0 ec=110/20 lis/c=558/558 les/c/f=559/559/0 sis=558) 2024-08-19T14:32:08.729 INFO:teuthology.orchestra.run.smithi031.stderr:Export successful 2024-08-19T14:32:08.762 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:32:08.784 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:08.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:08.785 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:32:08.787 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-19T14:32:08.886 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:32:08.908 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:08.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:08.909 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:32:09.010 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:32:09.045 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:09.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:09.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:32:09.148 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:32:09.182 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:09.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:09.183 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:32:09.284 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:32:09.307 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:09.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:09.308 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:32:09.408 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:32:09.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:09.451 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:09.452 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:32:09.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:09 smithi032 ceph-mon[20004]: pgmap v1344: 47 pgs: 1 active+clean+remapped, 4 active+recovering+degraded+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 2.8 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 536 op/s; 22824/138318 objects degraded (16.501%); 16326/138318 objects misplaced (11.803%); 5.3 MiB/s, 257 objects/s recovering 2024-08-19T14:32:09.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:09 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:32:09.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:09 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:32:09.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:09 smithi045 ceph-mon[20143]: pgmap v1344: 47 pgs: 1 active+clean+remapped, 4 active+recovering+degraded+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 2.8 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 536 op/s; 22824/138318 objects degraded (16.501%); 16326/138318 objects misplaced (11.803%); 5.3 MiB/s, 257 objects/s recovering 2024-08-19T14:32:09.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:09 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:32:09.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:09 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:32:09.553 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:32:09.583 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:09.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:09.585 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:32:09.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:09 smithi031 ceph-mon[16336]: pgmap v1344: 47 pgs: 1 active+clean+remapped, 4 active+recovering+degraded+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 2.8 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 536 op/s; 22824/138318 objects degraded (16.501%); 16326/138318 objects misplaced (11.803%); 5.3 MiB/s, 257 objects/s recovering 2024-08-19T14:32:09.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:09 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:32:09.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:09 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:32:09.686 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:32:09.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:09.718 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:09.718 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:32:09.818 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:32:09.853 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:09.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:09.854 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:32:09.955 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:32:10.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:10.007 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:10.007 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:32:10.108 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:32:10.131 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:10.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:10.132 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:32:10.233 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:32:10.267 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:10.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:10.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:32:10.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:10 smithi031 ceph-mon[16336]: osdmap e578: 12 total, 11 up, 9 in 2024-08-19T14:32:10.369 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:32:10.401 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:10.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:10.402 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:32:10.503 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:32:10.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:10 smithi032 ceph-mon[20004]: osdmap e578: 12 total, 11 up, 9 in 2024-08-19T14:32:10.532 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:10.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:10.533 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:32:10.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:10 smithi045 ceph-mon[20143]: osdmap e578: 12 total, 11 up, 9 in 2024-08-19T14:32:10.635 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:32:10.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:10.673 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:10.673 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:32:10.775 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:32:10.812 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:10.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:10.813 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:32:10.914 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:32:10.945 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:10.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:10.946 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:32:11.047 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:32:11.068 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:11.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:11.069 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:32:11.078 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:32:11.078 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-08-19T14:32:11.078 INFO:teuthology.orchestra.run.smithi031.stdout: "complaint_time": 30, 2024-08-19T14:32:11.078 INFO:teuthology.orchestra.run.smithi031.stdout: "num_blocked_ops": 0 2024-08-19T14:32:11.078 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:32:11.170 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:32:11.198 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:11.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:11.199 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:32:11.300 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:32:11.326 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:11.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:11.327 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:32:11.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:11 smithi031 ceph-mon[16336]: pgmap v1346: 47 pgs: 1 active+recovering+degraded+remapped, 4 active+clean+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 2.6 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 788 op/s; 13743/129135 objects degraded (10.642%); 23451/129135 objects misplaced (18.160%); 6.6 MiB/s, 396 objects/s recovering 2024-08-19T14:32:11.429 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:32:11.465 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:11.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:11.466 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:32:11.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:11 smithi032 ceph-mon[20004]: pgmap v1346: 47 pgs: 1 active+recovering+degraded+remapped, 4 active+clean+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 2.6 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 788 op/s; 13743/129135 objects degraded (10.642%); 23451/129135 objects misplaced (18.160%); 6.6 MiB/s, 396 objects/s recovering 2024-08-19T14:32:11.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:11 smithi045 ceph-mon[20143]: pgmap v1346: 47 pgs: 1 active+recovering+degraded+remapped, 4 active+clean+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 2.6 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 788 op/s; 13743/129135 objects degraded (10.642%); 23451/129135 objects misplaced (18.160%); 6.6 MiB/s, 396 objects/s recovering 2024-08-19T14:32:11.566 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:32:11.591 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:11.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:11.592 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:32:11.693 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:32:11.716 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:11.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:11.717 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:32:11.818 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-08-19T14:32:11.818 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:32:11.840 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:11.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:11.842 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:32:11.942 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:32:11.964 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:11.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:11.965 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:32:12.066 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:32:12.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:12.105 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:12.105 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:32:12.206 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:32:12.230 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:12.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:12.231 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:32:12.332 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:32:12.355 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:12.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:12.356 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:32:12.457 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:32:12.481 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:12.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:12.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:32:12.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:12 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:32:12.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:12 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:32:12.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:12 smithi032 ceph-mon[20004]: osdmap e579: 12 total, 11 up, 9 in 2024-08-19T14:32:12.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:12 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:32:12.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:12 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:32:12.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:12 smithi045 ceph-mon[20143]: osdmap e579: 12 total, 11 up, 9 in 2024-08-19T14:32:12.583 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:32:12.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:12.605 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:12.605 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:32:12.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:12 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:32:12.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:12 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:32:12.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:12 smithi031 ceph-mon[16336]: osdmap e579: 12 total, 11 up, 9 in 2024-08-19T14:32:12.706 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:32:12.727 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:12.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:12.728 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:32:12.829 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:32:12.852 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:12.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:12.853 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:32:12.954 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:32:12.986 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:12.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:12.987 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:32:13.088 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:32:13.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:13.111 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:13.111 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:32:13.213 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:32:13.272 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell -n osd.2 --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --force --op remove --pgid 1.5 2024-08-19T14:32:13.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:13.290 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:13.290 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:32:13.316 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:32:13.327 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:32:13.350 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:32:13.391 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:32:13.413 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:13.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:13.414 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:32:13.473 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:13 smithi031 ceph-mon[16336]: pgmap v1348: 47 pgs: 1 active+recovering+degraded+remapped, 4 active+clean+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 2.6 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 614 op/s; 13528/128490 objects degraded (10.528%); 23451/128490 objects misplaced (18.251%); 6.7 MiB/s, 204 objects/s recovering 2024-08-19T14:32:13.473 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:13 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:32:13.473 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:13 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:32:13.473 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:13 smithi031 ceph-mon[16336]: osdmap e580: 12 total, 11 up, 9 in 2024-08-19T14:32:13.473 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:13 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 13528/128490 objects degraded (10.528%), 8 pgs degraded, 16 pgs undersized (PG_DEGRADED) 2024-08-19T14:32:13.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:13 smithi032 ceph-mon[20004]: pgmap v1348: 47 pgs: 1 active+recovering+degraded+remapped, 4 active+clean+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 2.6 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 614 op/s; 13528/128490 objects degraded (10.528%); 23451/128490 objects misplaced (18.251%); 6.7 MiB/s, 204 objects/s recovering 2024-08-19T14:32:13.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:13 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:32:13.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:13 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:32:13.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:13 smithi032 ceph-mon[20004]: osdmap e580: 12 total, 11 up, 9 in 2024-08-19T14:32:13.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:13 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 13528/128490 objects degraded (10.528%), 8 pgs degraded, 16 pgs undersized (PG_DEGRADED) 2024-08-19T14:32:13.514 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:32:13.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:13 smithi045 ceph-mon[20143]: pgmap v1348: 47 pgs: 1 active+recovering+degraded+remapped, 4 active+clean+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 2.6 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 614 op/s; 13528/128490 objects degraded (10.528%); 23451/128490 objects misplaced (18.251%); 6.7 MiB/s, 204 objects/s recovering 2024-08-19T14:32:13.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:13 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:32:13.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:13 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:32:13.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:13 smithi045 ceph-mon[20143]: osdmap e580: 12 total, 11 up, 9 in 2024-08-19T14:32:13.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:13 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 13528/128490 objects degraded (10.528%), 8 pgs degraded, 16 pgs undersized (PG_DEGRADED) 2024-08-19T14:32:13.559 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:13.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:13.560 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:32:13.601 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/osd.2/config 2024-08-19T14:32:13.661 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:32:13.682 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:13.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:13.683 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:32:13.784 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:32:13.812 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:13.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:13.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:32:13.915 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:32:13.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:13.939 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:13.939 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:32:14.040 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:32:14.088 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:14.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:14.089 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:32:14.190 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:32:14.233 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:14.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:14.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:32:14.335 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:32:14.357 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:14.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:14.358 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:32:14.459 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:32:14.481 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:14.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:14.482 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:32:14.583 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:32:14.606 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:14.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:14.607 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:32:14.708 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:32:14.752 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:14.752 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:32:14.752 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:32:14.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:14.753 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:32:14.812 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:32:14.813 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:32:14.813 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-08-19T14:32:14.827 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:32:14.827 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 20, 2024-08-19T14:32:14.827 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 600, 2024-08-19T14:32:14.827 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [ 2024-08-19T14:32:14.828 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.828 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:76430 6.b 6:dcffab99:::benchmark_data_smithi045_97937_object76429:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e527)", 2024-08-19T14:32:14.828 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:29:17.985925+0000", 2024-08-19T14:32:14.828 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 176.80649900099999, 2024-08-19T14:32:14.828 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.31489470200000003, 2024-08-19T14:32:14.828 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:32:14.828 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:32:14.828 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:32:14.828 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:32:14.828 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:32:14.828 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:32:14.828 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 76430 2024-08-19T14:32:14.828 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.829 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:32:14.829 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.829 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:32:14.829 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.985925+0000", 2024-08-19T14:32:14.829 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:32:14.829 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.829 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.829 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:32:14.829 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.985925+0000", 2024-08-19T14:32:14.829 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:32:14.829 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.829 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.829 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:32:14.830 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.985926+0000", 2024-08-19T14:32:14.830 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5600000000000001e-06 2024-08-19T14:32:14.830 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.830 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.830 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:32:14.830 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.985980+0000", 2024-08-19T14:32:14.830 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.3575999999999997e-05 2024-08-19T14:32:14.830 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.830 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.830 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:32:14.830 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.985992+0000", 2024-08-19T14:32:14.830 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2568999999999999e-05 2024-08-19T14:32:14.831 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.831 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.831 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:32:14.831 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.985998+0000", 2024-08-19T14:32:14.831 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.7899999999999996e-06 2024-08-19T14:32:14.831 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.831 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.831 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:32:14.831 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.986046+0000", 2024-08-19T14:32:14.831 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7429000000000003e-05 2024-08-19T14:32:14.831 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.831 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.831 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:32:14.831 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.986157+0000", 2024-08-19T14:32:14.832 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000111094 2024-08-19T14:32:14.832 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.832 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.832 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 1,7", 2024-08-19T14:32:14.832 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.986254+0000", 2024-08-19T14:32:14.832 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.7152999999999996e-05 2024-08-19T14:32:14.832 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.832 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.832 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:32:14.832 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.988872+0000", 2024-08-19T14:32:14.832 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.002618507 2024-08-19T14:32:14.832 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.833 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.833 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:32:14.833 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.989328+0000", 2024-08-19T14:32:14.833 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00045559000000000002 2024-08-19T14:32:14.833 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.833 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.833 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:32:14.833 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:18.300657+0000", 2024-08-19T14:32:14.833 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.311329358 2024-08-19T14:32:14.833 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.833 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.833 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:32:14.833 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:18.300758+0000", 2024-08-19T14:32:14.833 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000100963 2024-08-19T14:32:14.834 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.834 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.834 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:32:14.834 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:18.300819+0000", 2024-08-19T14:32:14.834 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.1112999999999996e-05 2024-08-19T14:32:14.834 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:32:14.834 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:32:14.834 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:32:14.834 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.834 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.834 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:76432 6.b 6:da44d1ad:::benchmark_data_smithi045_97937_object76431:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e527)", 2024-08-19T14:32:14.834 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:29:17.991983+0000", 2024-08-19T14:32:14.834 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 176.80044061500001, 2024-08-19T14:32:14.835 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.30901316000000001, 2024-08-19T14:32:14.835 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:32:14.835 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:32:14.835 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:32:14.835 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:32:14.835 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:32:14.835 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:32:14.835 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 76432 2024-08-19T14:32:14.835 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.835 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:32:14.835 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.835 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:32:14.835 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.991983+0000", 2024-08-19T14:32:14.836 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:32:14.836 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.836 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.836 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:32:14.836 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.991983+0000", 2024-08-19T14:32:14.836 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:32:14.836 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.836 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.836 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:32:14.836 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.991985+0000", 2024-08-19T14:32:14.836 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.097e-06 2024-08-19T14:32:14.836 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.836 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.836 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:32:14.837 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.992034+0000", 2024-08-19T14:32:14.837 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.8433000000000002e-05 2024-08-19T14:32:14.837 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.837 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.837 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:32:14.837 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.992056+0000", 2024-08-19T14:32:14.837 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.2425999999999998e-05 2024-08-19T14:32:14.837 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.837 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.837 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:32:14.837 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.992066+0000", 2024-08-19T14:32:14.837 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0161e-05 2024-08-19T14:32:14.837 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.837 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.838 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:32:14.838 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.992141+0000", 2024-08-19T14:32:14.838 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.4301999999999994e-05 2024-08-19T14:32:14.838 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.838 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.838 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:32:14.838 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.992315+0000", 2024-08-19T14:32:14.838 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00017417100000000001 2024-08-19T14:32:14.838 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.838 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.838 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 1,7", 2024-08-19T14:32:14.838 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.992491+0000", 2024-08-19T14:32:14.838 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00017612500000000001 2024-08-19T14:32:14.839 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.839 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.839 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:32:14.839 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.994990+0000", 2024-08-19T14:32:14.839 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.002498718 2024-08-19T14:32:14.839 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.839 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.839 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:32:14.839 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:18.011302+0000", 2024-08-19T14:32:14.839 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.016312621999999999 2024-08-19T14:32:14.839 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.839 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.839 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:32:14.839 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:18.300874+0000", 2024-08-19T14:32:14.840 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.28957173200000003 2024-08-19T14:32:14.840 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.840 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.840 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:32:14.840 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:18.300959+0000", 2024-08-19T14:32:14.840 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.4843000000000001e-05 2024-08-19T14:32:14.840 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.840 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.840 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:32:14.840 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:18.300996+0000", 2024-08-19T14:32:14.840 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7530000000000002e-05 2024-08-19T14:32:14.840 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:32:14.840 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:32:14.841 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:32:14.841 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.841 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.841 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:118807 6.6 6:6cd55873:::benchmark_data_smithi045_97937_object11534:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e552)", 2024-08-19T14:32:14.841 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:30:32.268038+0000", 2024-08-19T14:32:14.841 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 102.524385931, 2024-08-19T14:32:14.841 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.30788272799999999, 2024-08-19T14:32:14.841 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:32:14.841 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:32:14.841 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:32:14.841 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:32:14.841 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:32:14.841 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:32:14.841 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 118807 2024-08-19T14:32:14.842 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.842 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:32:14.842 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.842 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:32:14.842 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:32.268038+0000", 2024-08-19T14:32:14.842 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:32:14.842 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.842 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.842 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:32:14.842 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:32.268038+0000", 2024-08-19T14:32:14.842 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:32:14.842 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.842 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.843 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:32:14.843 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:32.268038+0000", 2024-08-19T14:32:14.843 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2100000000000002e-07 2024-08-19T14:32:14.843 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.843 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.843 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:32:14.843 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:32.268041+0000", 2024-08-19T14:32:14.843 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.8770000000000002e-06 2024-08-19T14:32:14.843 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.843 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.843 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:32:14.843 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:32.268048+0000", 2024-08-19T14:32:14.843 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.5379999999999999e-06 2024-08-19T14:32:14.843 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.844 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.844 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:32:14.844 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:32.268052+0000", 2024-08-19T14:32:14.844 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7790000000000003e-06 2024-08-19T14:32:14.844 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.844 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.844 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:32:14.844 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:32.280586+0000", 2024-08-19T14:32:14.844 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.012533624 2024-08-19T14:32:14.844 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.844 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.844 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for rw locks", 2024-08-19T14:32:14.844 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:32.280707+0000", 2024-08-19T14:32:14.845 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000121295 2024-08-19T14:32:14.845 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.845 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.845 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:32:14.845 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:32.518381+0000", 2024-08-19T14:32:14.845 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.23767379899999999 2024-08-19T14:32:14.845 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.845 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.845 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:32:14.845 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:32.518479+0000", 2024-08-19T14:32:14.845 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.8368000000000004e-05 2024-08-19T14:32:14.845 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.845 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.846 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 1,5,7", 2024-08-19T14:32:14.846 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:32.518614+0000", 2024-08-19T14:32:14.846 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000134151 2024-08-19T14:32:14.846 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.846 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.846 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:32:14.846 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:32.534031+0000", 2024-08-19T14:32:14.846 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.015417597999999999 2024-08-19T14:32:14.846 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.846 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.846 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:32:14.846 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:32.534072+0000", 2024-08-19T14:32:14.846 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.0818999999999997e-05 2024-08-19T14:32:14.847 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.847 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.847 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:32:14.847 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:32.570929+0000", 2024-08-19T14:32:14.847 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.036857433000000002 2024-08-19T14:32:14.847 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.847 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.847 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:32:14.847 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:32.571042+0000", 2024-08-19T14:32:14.847 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00011267700000000001 2024-08-19T14:32:14.847 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.847 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.847 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:32:14.848 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:32.571104+0000", 2024-08-19T14:32:14.848 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.1948999999999999e-05 2024-08-19T14:32:14.848 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.848 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.848 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:32:14.848 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:32.575921+0000", 2024-08-19T14:32:14.848 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0048164000000000002 2024-08-19T14:32:14.848 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:32:14.848 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:32:14.848 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:32:14.848 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.848 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.848 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:118930 6.6 6:6e7e027f:::benchmark_data_smithi045_97937_object11657:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e552)", 2024-08-19T14:32:14.848 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:30:32.618449+0000", 2024-08-19T14:32:14.849 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 102.173974671, 2024-08-19T14:32:14.849 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.34887296600000001, 2024-08-19T14:32:14.849 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:32:14.849 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:32:14.849 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:32:14.849 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:32:14.849 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:32:14.849 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:32:14.849 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 118930 2024-08-19T14:32:14.849 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.849 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:32:14.849 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.849 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:32:14.850 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:32.618449+0000", 2024-08-19T14:32:14.850 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:32:14.850 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.850 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.850 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:32:14.850 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:32.618449+0000", 2024-08-19T14:32:14.850 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:32:14.850 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.850 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.850 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:32:14.850 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:32.618451+0000", 2024-08-19T14:32:14.850 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.9099999999999999e-06 2024-08-19T14:32:14.851 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.851 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.851 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:32:14.851 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:32.618456+0000", 2024-08-19T14:32:14.851 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.2619999999999998e-06 2024-08-19T14:32:14.851 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.851 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.851 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:32:14.851 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:32.618470+0000", 2024-08-19T14:32:14.851 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4038e-05 2024-08-19T14:32:14.851 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.851 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.851 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:32:14.851 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:32.618476+0000", 2024-08-19T14:32:14.852 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.0610000000000001e-06 2024-08-19T14:32:14.852 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.852 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.852 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:32:14.852 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:32.648776+0000", 2024-08-19T14:32:14.852 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.030299403999999999 2024-08-19T14:32:14.852 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.852 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.852 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for rw locks", 2024-08-19T14:32:14.852 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:32.648894+0000", 2024-08-19T14:32:14.852 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00011780100000000001 2024-08-19T14:32:14.852 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.852 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.853 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:32:14.853 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:32.789354+0000", 2024-08-19T14:32:14.853 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.14046091999999999 2024-08-19T14:32:14.853 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.853 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.853 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:32:14.853 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:32.789450+0000", 2024-08-19T14:32:14.853 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.5760000000000005e-05 2024-08-19T14:32:14.853 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.853 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.853 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 1,5,7", 2024-08-19T14:32:14.853 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:32.789588+0000", 2024-08-19T14:32:14.853 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000137653 2024-08-19T14:32:14.854 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.854 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.854 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:32:14.854 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:32.913096+0000", 2024-08-19T14:32:14.854 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.123507748 2024-08-19T14:32:14.854 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.854 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.854 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:32:14.854 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:32.913250+0000", 2024-08-19T14:32:14.854 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00015415599999999999 2024-08-19T14:32:14.854 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.854 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.855 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:32:14.855 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:32.913306+0000", 2024-08-19T14:32:14.855 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.5954e-05 2024-08-19T14:32:14.855 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.855 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.855 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:32:14.855 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:32.962881+0000", 2024-08-19T14:32:14.855 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.049575385 2024-08-19T14:32:14.855 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.855 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.855 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:32:14.855 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:32.962955+0000", 2024-08-19T14:32:14.855 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.4126000000000001e-05 2024-08-19T14:32:14.855 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.856 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.856 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:32:14.856 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:32.967322+0000", 2024-08-19T14:32:14.856 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0043667879999999999 2024-08-19T14:32:14.856 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:32:14.856 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:32:14.856 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:32:14.856 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.856 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.856 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:120225 6.6 6:68f0d154:::benchmark_data_smithi045_97937_object12952:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e552)", 2024-08-19T14:32:14.856 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:30:35.204256+0000", 2024-08-19T14:32:14.856 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 99.588167659999996, 2024-08-19T14:32:14.856 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1367397340000001, 2024-08-19T14:32:14.857 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:32:14.857 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:32:14.857 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:32:14.857 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:32:14.857 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:32:14.857 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:32:14.857 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 120225 2024-08-19T14:32:14.857 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.857 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:32:14.857 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.857 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:32:14.857 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:35.204256+0000", 2024-08-19T14:32:14.857 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:32:14.858 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.858 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.858 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:32:14.858 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:35.204256+0000", 2024-08-19T14:32:14.858 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:32:14.858 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.858 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.858 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:32:14.858 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:35.204256+0000", 2024-08-19T14:32:14.858 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.7700000000000001e-07 2024-08-19T14:32:14.858 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.858 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.859 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:32:14.859 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:35.204258+0000", 2024-08-19T14:32:14.859 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.1229999999999998e-06 2024-08-19T14:32:14.859 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.859 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.859 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:32:14.859 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:35.204263+0000", 2024-08-19T14:32:14.859 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.9080000000000003e-06 2024-08-19T14:32:14.859 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.859 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.859 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:32:14.859 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:35.204268+0000", 2024-08-19T14:32:14.859 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.8169999999999998e-06 2024-08-19T14:32:14.859 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.860 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.860 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:32:14.860 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:35.214570+0000", 2024-08-19T14:32:14.860 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.010301805000000001 2024-08-19T14:32:14.860 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.860 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.860 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for missing object", 2024-08-19T14:32:14.860 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:35.214727+0000", 2024-08-19T14:32:14.860 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00015671700000000001 2024-08-19T14:32:14.860 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.860 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.860 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:32:14.860 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:36.340980+0000", 2024-08-19T14:32:14.861 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1262536810000001 2024-08-19T14:32:14.861 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.861 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.861 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:32:14.861 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:36.340996+0000", 2024-08-19T14:32:14.861 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5406000000000001e-05 2024-08-19T14:32:14.861 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:32:14.861 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:32:14.861 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:32:14.861 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.861 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.861 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:120229 6.6 6:6672e843:::benchmark_data_smithi045_97937_object12956:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e552)", 2024-08-19T14:32:14.861 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:30:35.204275+0000", 2024-08-19T14:32:14.862 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 99.588148822999997, 2024-08-19T14:32:14.862 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.151004988, 2024-08-19T14:32:14.862 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:32:14.862 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:32:14.862 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:32:14.862 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:32:14.862 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:32:14.862 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:32:14.862 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 120229 2024-08-19T14:32:14.862 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.862 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:32:14.862 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.862 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:32:14.863 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:35.204275+0000", 2024-08-19T14:32:14.863 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:32:14.863 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.863 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.863 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:32:14.863 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:35.204275+0000", 2024-08-19T14:32:14.863 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:32:14.863 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.863 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.863 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:32:14.863 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:35.204275+0000", 2024-08-19T14:32:14.863 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.9299999999999999e-07 2024-08-19T14:32:14.863 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.863 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.863 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:32:14.864 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:35.204278+0000", 2024-08-19T14:32:14.864 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.3199999999999998e-06 2024-08-19T14:32:14.864 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.864 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.864 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:32:14.864 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:35.204283+0000", 2024-08-19T14:32:14.864 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.7780000000000004e-06 2024-08-19T14:32:14.864 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.864 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.864 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:32:14.864 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:35.204289+0000", 2024-08-19T14:32:14.864 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.3800000000000002e-06 2024-08-19T14:32:14.865 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.865 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.865 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:32:14.865 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:35.214760+0000", 2024-08-19T14:32:14.865 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.010471044000000001 2024-08-19T14:32:14.865 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.865 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.865 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for missing object", 2024-08-19T14:32:14.865 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:35.215051+0000", 2024-08-19T14:32:14.865 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00029112499999999999 2024-08-19T14:32:14.865 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.865 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.865 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:32:14.865 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:36.355266+0000", 2024-08-19T14:32:14.866 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1402151220000001 2024-08-19T14:32:14.866 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.866 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.866 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:32:14.866 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:36.355280+0000", 2024-08-19T14:32:14.866 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3825999999999999e-05 2024-08-19T14:32:14.866 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:32:14.866 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:32:14.866 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:32:14.866 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.866 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.866 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:120799 6.6 6:6d4cbd5c:::benchmark_data_smithi045_97937_object13526:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e552)", 2024-08-19T14:32:14.866 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:30:36.258857+0000", 2024-08-19T14:32:14.867 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 98.533567133000005, 2024-08-19T14:32:14.867 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2809968280000001, 2024-08-19T14:32:14.867 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:32:14.867 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:32:14.867 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:32:14.867 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:32:14.867 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:32:14.867 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:32:14.867 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 120799 2024-08-19T14:32:14.867 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.867 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:32:14.867 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.867 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:32:14.868 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:36.258857+0000", 2024-08-19T14:32:14.868 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:32:14.868 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.868 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.868 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:32:14.868 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:36.258857+0000", 2024-08-19T14:32:14.868 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:32:14.868 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.868 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.868 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:32:14.868 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:36.258857+0000", 2024-08-19T14:32:14.868 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.01e-07 2024-08-19T14:32:14.868 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.869 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.869 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:32:14.869 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:36.258863+0000", 2024-08-19T14:32:14.869 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.1419999999999998e-06 2024-08-19T14:32:14.869 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.869 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.869 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:32:14.869 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:36.258870+0000", 2024-08-19T14:32:14.869 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.4699999999999999e-06 2024-08-19T14:32:14.869 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.869 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.869 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:32:14.869 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:36.258875+0000", 2024-08-19T14:32:14.869 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.3580000000000002e-06 2024-08-19T14:32:14.870 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.870 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.870 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:32:14.870 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:36.261800+0000", 2024-08-19T14:32:14.870 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.002925422 2024-08-19T14:32:14.870 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.870 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.870 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for missing object", 2024-08-19T14:32:14.870 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:36.261888+0000", 2024-08-19T14:32:14.870 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.7714999999999996e-05 2024-08-19T14:32:14.870 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.870 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.870 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:32:14.871 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:40.539815+0000", 2024-08-19T14:32:14.871 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2779266820000004 2024-08-19T14:32:14.871 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.871 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.871 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:32:14.871 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:40.539853+0000", 2024-08-19T14:32:14.871 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.8637999999999998e-05 2024-08-19T14:32:14.871 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:32:14.871 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:32:14.871 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:32:14.871 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.871 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.871 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:122769 6.6 6:6e814bcb:::benchmark_data_smithi045_97937_object15496:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e553)", 2024-08-19T14:32:14.871 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:30:40.459976+0000", 2024-08-19T14:32:14.872 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 94.332447537999997, 2024-08-19T14:32:14.872 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.52570701900000005, 2024-08-19T14:32:14.872 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:32:14.872 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:32:14.872 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:32:14.872 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:32:14.872 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:32:14.872 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:32:14.872 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 122769 2024-08-19T14:32:14.872 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.872 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:32:14.872 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.872 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:32:14.872 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:40.459976+0000", 2024-08-19T14:32:14.873 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:32:14.873 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.873 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.873 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:32:14.873 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:40.459976+0000", 2024-08-19T14:32:14.873 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:32:14.873 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.873 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.873 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:32:14.873 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:40.459976+0000", 2024-08-19T14:32:14.873 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.2899999999999999e-07 2024-08-19T14:32:14.873 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.873 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.873 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:32:14.874 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:40.459979+0000", 2024-08-19T14:32:14.874 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.0660000000000002e-06 2024-08-19T14:32:14.874 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.874 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.874 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:32:14.874 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:40.459984+0000", 2024-08-19T14:32:14.874 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.2989999999999996e-06 2024-08-19T14:32:14.874 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.874 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.874 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:32:14.874 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:40.459989+0000", 2024-08-19T14:32:14.874 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.763e-06 2024-08-19T14:32:14.875 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.875 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.875 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:32:14.875 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:40.484365+0000", 2024-08-19T14:32:14.875 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.024376687000000001 2024-08-19T14:32:14.875 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.875 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.875 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for missing object", 2024-08-19T14:32:14.875 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:40.484465+0000", 2024-08-19T14:32:14.875 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.9951999999999994e-05 2024-08-19T14:32:14.875 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.875 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.875 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:32:14.875 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:40.985670+0000", 2024-08-19T14:32:14.875 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.50120491599999994 2024-08-19T14:32:14.876 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.876 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.876 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:32:14.876 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:40.985683+0000", 2024-08-19T14:32:14.876 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3006999999999999e-05 2024-08-19T14:32:14.876 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:32:14.876 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:32:14.876 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:32:14.876 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.876 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.876 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:122792 6.6 6:6a0a9fd1:::benchmark_data_smithi045_97937_object15519:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e553)", 2024-08-19T14:32:14.876 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:30:40.459994+0000", 2024-08-19T14:32:14.876 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 94.332429559000005, 2024-08-19T14:32:14.876 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.55843794999999996, 2024-08-19T14:32:14.877 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:32:14.877 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:32:14.877 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:32:14.877 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:32:14.877 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:32:14.877 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:32:14.877 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 122792 2024-08-19T14:32:14.877 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.877 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:32:14.877 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.877 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:32:14.877 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:40.459994+0000", 2024-08-19T14:32:14.877 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:32:14.878 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.878 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.878 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:32:14.878 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:40.459994+0000", 2024-08-19T14:32:14.878 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:32:14.878 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.878 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.878 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:32:14.878 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:40.459994+0000", 2024-08-19T14:32:14.878 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.0699999999999998e-07 2024-08-19T14:32:14.878 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.878 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.878 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:32:14.879 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:40.459996+0000", 2024-08-19T14:32:14.879 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.86e-06 2024-08-19T14:32:14.879 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.879 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.879 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:32:14.879 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:40.460002+0000", 2024-08-19T14:32:14.879 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.2599999999999996e-06 2024-08-19T14:32:14.879 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.879 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.879 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:32:14.879 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:40.460006+0000", 2024-08-19T14:32:14.879 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.3780000000000003e-06 2024-08-19T14:32:14.879 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.879 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.880 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:32:14.880 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:40.484501+0000", 2024-08-19T14:32:14.880 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.024495052 2024-08-19T14:32:14.880 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.880 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.880 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for missing object", 2024-08-19T14:32:14.880 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:40.484606+0000", 2024-08-19T14:32:14.880 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000104708 2024-08-19T14:32:14.880 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.880 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.880 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:32:14.880 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:41.018418+0000", 2024-08-19T14:32:14.880 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.53381277000000005 2024-08-19T14:32:14.880 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.881 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.881 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:32:14.881 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:41.018432+0000", 2024-08-19T14:32:14.881 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3614999999999999e-05 2024-08-19T14:32:14.881 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:32:14.881 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:32:14.881 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:32:14.881 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.881 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.881 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:126903 6.6 6:67cd8495:::benchmark_data_smithi045_97937_object19630:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e560)", 2024-08-19T14:32:14.881 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:30:50.979743+0000", 2024-08-19T14:32:14.881 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 83.812680853000003, 2024-08-19T14:32:14.881 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.39269494900000002, 2024-08-19T14:32:14.881 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:32:14.881 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:32:14.882 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:32:14.882 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:32:14.882 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:32:14.882 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:32:14.882 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 126903 2024-08-19T14:32:14.882 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.882 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:32:14.882 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.882 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:32:14.882 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:50.979743+0000", 2024-08-19T14:32:14.882 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:32:14.882 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.882 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.883 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:32:14.883 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:50.979743+0000", 2024-08-19T14:32:14.883 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:32:14.883 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.883 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.883 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:32:14.883 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:50.979743+0000", 2024-08-19T14:32:14.883 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.3700000000000001e-07 2024-08-19T14:32:14.883 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.883 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.883 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:32:14.883 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:50.979746+0000", 2024-08-19T14:32:14.883 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.6000000000000001e-06 2024-08-19T14:32:14.884 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.884 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.884 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:32:14.884 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:50.979752+0000", 2024-08-19T14:32:14.884 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.1879999999999999e-06 2024-08-19T14:32:14.884 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.884 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.884 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:32:14.884 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:50.979757+0000", 2024-08-19T14:32:14.884 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.8420000000000004e-06 2024-08-19T14:32:14.884 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.884 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.884 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:32:14.885 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:50.993486+0000", 2024-08-19T14:32:14.885 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.013729005000000001 2024-08-19T14:32:14.885 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.885 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.885 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for missing object", 2024-08-19T14:32:14.885 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:50.993581+0000", 2024-08-19T14:32:14.885 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.5191000000000004e-05 2024-08-19T14:32:14.885 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.885 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.885 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:32:14.885 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:51.372426+0000", 2024-08-19T14:32:14.885 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.378844652 2024-08-19T14:32:14.885 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.885 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.886 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:32:14.886 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:51.372438+0000", 2024-08-19T14:32:14.886 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2034e-05 2024-08-19T14:32:14.886 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:32:14.886 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:32:14.886 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:32:14.886 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.886 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.886 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:137589 6.6 6:61484f3f:::benchmark_data_smithi045_97937_object30316:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e569)", 2024-08-19T14:32:14.886 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:31:13.695526+0000", 2024-08-19T14:32:14.886 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 61.096897693999999, 2024-08-19T14:32:14.886 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.8720014850000002, 2024-08-19T14:32:14.887 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:32:14.887 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:32:14.887 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:32:14.887 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:32:14.887 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:32:14.887 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:32:14.887 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 137589 2024-08-19T14:32:14.887 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.887 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:32:14.887 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.887 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:32:14.887 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:31:13.695526+0000", 2024-08-19T14:32:14.887 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:32:14.887 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.888 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.888 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:32:14.888 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:31:13.695526+0000", 2024-08-19T14:32:14.888 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:32:14.888 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.888 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.888 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:32:14.888 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:31:13.695526+0000", 2024-08-19T14:32:14.888 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.6100000000000001e-07 2024-08-19T14:32:14.888 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.888 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.888 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:32:14.888 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:31:13.695530+0000", 2024-08-19T14:32:14.889 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.0979999999999998e-06 2024-08-19T14:32:14.889 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.889 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.889 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:32:14.889 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:31:13.695537+0000", 2024-08-19T14:32:14.889 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.7109999999999995e-06 2024-08-19T14:32:14.889 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.889 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.889 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:32:14.889 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:31:13.695543+0000", 2024-08-19T14:32:14.889 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.5500000000000002e-06 2024-08-19T14:32:14.889 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.889 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.889 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:32:14.890 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:31:13.695732+0000", 2024-08-19T14:32:14.890 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000188801 2024-08-19T14:32:14.890 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.890 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.890 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for missing object", 2024-08-19T14:32:14.890 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:31:13.695811+0000", 2024-08-19T14:32:14.890 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.9010000000000004e-05 2024-08-19T14:32:14.890 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.890 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.890 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:32:14.890 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:31:16.567509+0000", 2024-08-19T14:32:14.890 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.8716988030000001 2024-08-19T14:32:14.890 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.890 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.891 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:32:14.891 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:31:16.567527+0000", 2024-08-19T14:32:14.891 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.8051e-05 2024-08-19T14:32:14.891 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:32:14.891 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:32:14.891 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:32:14.891 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.891 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.891 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:168001 6.6 6:607c6180:::benchmark_data_smithi045_97937_object60728:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e576)", 2024-08-19T14:32:14.891 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:32:03.729180+0000", 2024-08-19T14:32:14.891 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 11.063243257, 2024-08-19T14:32:14.891 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.1240902739999998, 2024-08-19T14:32:14.891 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:32:14.891 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:32:14.892 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:32:14.892 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:32:14.892 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:32:14.892 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:32:14.892 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 168001 2024-08-19T14:32:14.892 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.892 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:32:14.892 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.892 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:32:14.892 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:03.729180+0000", 2024-08-19T14:32:14.892 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:32:14.892 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.892 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.892 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:32:14.892 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:03.729180+0000", 2024-08-19T14:32:14.893 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:32:14.893 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.893 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.893 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:32:14.893 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:03.729181+0000", 2024-08-19T14:32:14.893 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.1199999999999999e-07 2024-08-19T14:32:14.893 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.893 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.893 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:32:14.893 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:03.729182+0000", 2024-08-19T14:32:14.893 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.319e-06 2024-08-19T14:32:14.893 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.893 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.893 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:32:14.894 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:03.729185+0000", 2024-08-19T14:32:14.894 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.0180000000000002e-06 2024-08-19T14:32:14.894 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.894 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.894 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:32:14.894 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:03.729188+0000", 2024-08-19T14:32:14.894 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.5859999999999999e-06 2024-08-19T14:32:14.894 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.894 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.894 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:32:14.894 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:03.734604+0000", 2024-08-19T14:32:14.894 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0054159550000000001 2024-08-19T14:32:14.894 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.895 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.895 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for missing object", 2024-08-19T14:32:14.895 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:03.743722+0000", 2024-08-19T14:32:14.895 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0091187130000000005 2024-08-19T14:32:14.895 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.895 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.895 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:32:14.895 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:05.853261+0000", 2024-08-19T14:32:14.895 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.1095391330000002 2024-08-19T14:32:14.895 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.895 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.895 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:32:14.895 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:05.853271+0000", 2024-08-19T14:32:14.896 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.3379999999999994e-06 2024-08-19T14:32:14.896 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:32:14.896 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:32:14.896 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:32:14.896 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.896 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.896 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:169537 6.6 6:6db58647:::benchmark_data_smithi045_97937_object62264:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e577)", 2024-08-19T14:32:14.896 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:32:06.053447+0000", 2024-08-19T14:32:14.896 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 8.7389770989999995, 2024-08-19T14:32:14.896 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.54333740200000002, 2024-08-19T14:32:14.896 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:32:14.896 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:32:14.896 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:32:14.896 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:32:14.897 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:32:14.897 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:32:14.897 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 169537 2024-08-19T14:32:14.897 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.897 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:32:14.897 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.897 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:32:14.897 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:06.053447+0000", 2024-08-19T14:32:14.897 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:32:14.897 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.897 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.897 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:32:14.897 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:06.053447+0000", 2024-08-19T14:32:14.897 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:32:14.898 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.898 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.898 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:32:14.898 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:06.053447+0000", 2024-08-19T14:32:14.898 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.8400000000000004e-07 2024-08-19T14:32:14.898 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.898 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.898 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:32:14.898 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:06.053451+0000", 2024-08-19T14:32:14.898 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.4209999999999999e-06 2024-08-19T14:32:14.898 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.898 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.898 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:32:14.899 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:32:14.900 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:06.053458+0000", 2024-08-19T14:32:14.900 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.6090000000000003e-06 2024-08-19T14:32:14.900 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.900 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.900 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:32:14.900 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:06.053465+0000", 2024-08-19T14:32:14.900 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.8109999999999997e-06 2024-08-19T14:32:14.900 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.900 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.901 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:32:14.901 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:06.053636+0000", 2024-08-19T14:32:14.901 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00017099699999999999 2024-08-19T14:32:14.901 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.901 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.901 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for missing object", 2024-08-19T14:32:14.901 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:06.053727+0000", 2024-08-19T14:32:14.901 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.1371000000000003e-05 2024-08-19T14:32:14.901 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.901 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.901 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:32:14.901 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:06.596774+0000", 2024-08-19T14:32:14.901 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.54304690899999997 2024-08-19T14:32:14.902 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.902 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.902 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:32:14.902 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:06.596784+0000", 2024-08-19T14:32:14.902 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.7000000000000003e-06 2024-08-19T14:32:14.902 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:32:14.902 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:32:14.902 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:32:14.902 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.902 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.902 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:169556 6.6 6:67d2a184:::benchmark_data_smithi045_97937_object62283:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e577)", 2024-08-19T14:32:14.902 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:32:06.053476+0000", 2024-08-19T14:32:14.903 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 8.7389472510000008, 2024-08-19T14:32:14.903 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.54729812700000002, 2024-08-19T14:32:14.903 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:32:14.903 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:32:14.903 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:32:14.903 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:32:14.903 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:32:14.903 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:32:14.903 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 169556 2024-08-19T14:32:14.903 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.903 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:32:14.903 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.903 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:32:14.903 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:06.053476+0000", 2024-08-19T14:32:14.904 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:32:14.904 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.904 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.904 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:32:14.904 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:06.053476+0000", 2024-08-19T14:32:14.904 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:32:14.904 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.904 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.904 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:32:14.904 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:06.053477+0000", 2024-08-19T14:32:14.904 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2599999999999998e-07 2024-08-19T14:32:14.904 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.904 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.905 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:32:14.905 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:06.053480+0000", 2024-08-19T14:32:14.905 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.7690000000000001e-06 2024-08-19T14:32:14.905 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.905 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.905 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:32:14.905 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:06.053486+0000", 2024-08-19T14:32:14.905 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.6610000000000003e-06 2024-08-19T14:32:14.905 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.905 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.905 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:32:14.905 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:06.053492+0000", 2024-08-19T14:32:14.905 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.8930000000000002e-06 2024-08-19T14:32:14.906 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.906 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.906 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:32:14.906 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:06.053757+0000", 2024-08-19T14:32:14.906 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00026490800000000001 2024-08-19T14:32:14.906 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.906 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.906 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for missing object", 2024-08-19T14:32:14.906 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:06.053833+0000", 2024-08-19T14:32:14.906 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.6384000000000006e-05 2024-08-19T14:32:14.906 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.906 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.906 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:32:14.907 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:06.600760+0000", 2024-08-19T14:32:14.907 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.54692678900000002 2024-08-19T14:32:14.907 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.907 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.907 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:32:14.907 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:06.600775+0000", 2024-08-19T14:32:14.907 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4297000000000001e-05 2024-08-19T14:32:14.907 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:32:14.907 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:32:14.907 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:32:14.907 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.907 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.907 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:169562 6.6 6:66b440fd:::benchmark_data_smithi045_97937_object62289:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e577)", 2024-08-19T14:32:14.908 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:32:06.053500+0000", 2024-08-19T14:32:14.908 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 8.7389232339999996, 2024-08-19T14:32:14.908 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.55661340299999995, 2024-08-19T14:32:14.908 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:32:14.908 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:32:14.908 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:32:14.908 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:32:14.908 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:32:14.908 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:32:14.908 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 169562 2024-08-19T14:32:14.908 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.908 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:32:14.908 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.908 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:32:14.909 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:06.053500+0000", 2024-08-19T14:32:14.909 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:32:14.909 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.909 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.909 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:32:14.909 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:06.053500+0000", 2024-08-19T14:32:14.909 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:32:14.909 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.909 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.909 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:32:14.909 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:06.053501+0000", 2024-08-19T14:32:14.909 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.6300000000000001e-07 2024-08-19T14:32:14.909 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.910 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.910 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:32:14.910 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:06.053503+0000", 2024-08-19T14:32:14.910 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.4499999999999998e-06 2024-08-19T14:32:14.910 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.910 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.910 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:32:14.910 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:06.053507+0000", 2024-08-19T14:32:14.910 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.9570000000000004e-06 2024-08-19T14:32:14.910 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.910 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.910 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:32:14.910 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:06.053511+0000", 2024-08-19T14:32:14.911 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.258e-06 2024-08-19T14:32:14.911 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.911 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.911 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:32:14.911 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:06.053869+0000", 2024-08-19T14:32:14.911 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00035858599999999999 2024-08-19T14:32:14.911 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.911 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.911 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for missing object", 2024-08-19T14:32:14.911 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:06.053980+0000", 2024-08-19T14:32:14.911 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000110888 2024-08-19T14:32:14.911 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.911 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.911 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:32:14.912 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:06.610097+0000", 2024-08-19T14:32:14.912 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.556116791 2024-08-19T14:32:14.912 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.912 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.912 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:32:14.912 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:06.610114+0000", 2024-08-19T14:32:14.912 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.7110000000000001e-05 2024-08-19T14:32:14.912 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:32:14.912 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:32:14.912 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:32:14.912 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.912 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.912 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:169564 6.6 6:6fdc4a6a:::benchmark_data_smithi045_97937_object62291:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e577)", 2024-08-19T14:32:14.912 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:32:06.053514+0000", 2024-08-19T14:32:14.913 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 8.7389093449999997, 2024-08-19T14:32:14.913 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.56045669499999995, 2024-08-19T14:32:14.913 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:32:14.913 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:32:14.913 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:32:14.913 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:32:14.913 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:32:14.913 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:32:14.913 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 169564 2024-08-19T14:32:14.913 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.913 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:32:14.913 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.913 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:32:14.913 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:06.053514+0000", 2024-08-19T14:32:14.913 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:32:14.914 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.914 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.914 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:32:14.914 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:06.053514+0000", 2024-08-19T14:32:14.914 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:32:14.914 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.914 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.914 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:32:14.914 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:06.053515+0000", 2024-08-19T14:32:14.914 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.11e-07 2024-08-19T14:32:14.914 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.914 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.914 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:32:14.915 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:06.053520+0000", 2024-08-19T14:32:14.915 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.057e-06 2024-08-19T14:32:14.915 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.915 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.915 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:32:14.915 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:06.053523+0000", 2024-08-19T14:32:14.915 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.371e-06 2024-08-19T14:32:14.915 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.915 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.915 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:32:14.915 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:06.053526+0000", 2024-08-19T14:32:14.915 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.473e-06 2024-08-19T14:32:14.915 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.915 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.916 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:32:14.916 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:06.054069+0000", 2024-08-19T14:32:14.916 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00054250200000000004 2024-08-19T14:32:14.916 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.916 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.916 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for missing object", 2024-08-19T14:32:14.916 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:06.054145+0000", 2024-08-19T14:32:14.916 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.6328999999999994e-05 2024-08-19T14:32:14.916 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.916 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.916 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:32:14.916 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:06.613955+0000", 2024-08-19T14:32:14.916 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.55980960199999996 2024-08-19T14:32:14.917 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.917 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.917 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:32:14.917 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:06.613971+0000", 2024-08-19T14:32:14.917 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.615e-05 2024-08-19T14:32:14.917 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:32:14.917 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:32:14.917 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:32:14.917 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.917 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.917 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:170739 6.6 6:67c34ebd:::benchmark_data_smithi045_97937_object63466:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e577)", 2024-08-19T14:32:14.917 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:32:07.961026+0000", 2024-08-19T14:32:14.917 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 6.8313974960000001, 2024-08-19T14:32:14.917 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.82947012499999995, 2024-08-19T14:32:14.918 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:32:14.918 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:32:14.918 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:32:14.918 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:32:14.918 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:32:14.918 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:32:14.918 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 170739 2024-08-19T14:32:14.918 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.918 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:32:14.918 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.918 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:32:14.918 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:07.961026+0000", 2024-08-19T14:32:14.918 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:32:14.919 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.919 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.919 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:32:14.919 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:07.961026+0000", 2024-08-19T14:32:14.919 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:32:14.919 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.919 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.919 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:32:14.919 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:07.961026+0000", 2024-08-19T14:32:14.919 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.6399999999999998e-07 2024-08-19T14:32:14.919 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.919 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.919 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:32:14.919 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:07.961028+0000", 2024-08-19T14:32:14.919 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.73e-06 2024-08-19T14:32:14.920 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.920 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.920 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:32:14.920 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:07.961032+0000", 2024-08-19T14:32:14.920 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7759999999999999e-06 2024-08-19T14:32:14.920 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.920 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.920 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:32:14.920 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:07.961035+0000", 2024-08-19T14:32:14.920 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.0690000000000002e-06 2024-08-19T14:32:14.920 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.920 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.920 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:32:14.920 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:07.965622+0000", 2024-08-19T14:32:14.921 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0045865539999999996 2024-08-19T14:32:14.921 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.921 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.921 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for missing object", 2024-08-19T14:32:14.921 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:07.965678+0000", 2024-08-19T14:32:14.921 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.6433e-05 2024-08-19T14:32:14.921 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.921 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.921 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:32:14.921 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:08.790485+0000", 2024-08-19T14:32:14.921 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.82480673199999999 2024-08-19T14:32:14.921 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.921 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.921 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:32:14.922 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:08.790496+0000", 2024-08-19T14:32:14.922 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1567e-05 2024-08-19T14:32:14.922 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:32:14.922 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:32:14.922 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:32:14.922 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.922 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.922 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:170742 6.6 6:6d8d58bc:::benchmark_data_smithi045_97937_object63469:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e577)", 2024-08-19T14:32:14.922 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:32:07.961040+0000", 2024-08-19T14:32:14.922 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 6.8313840560000001, 2024-08-19T14:32:14.922 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.83643209600000001, 2024-08-19T14:32:14.922 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:32:14.922 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:32:14.923 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:32:14.923 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:32:14.923 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:32:14.923 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:32:14.923 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 170742 2024-08-19T14:32:14.923 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.923 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:32:14.923 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.923 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:32:14.923 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:07.961040+0000", 2024-08-19T14:32:14.923 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:32:14.923 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.923 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.923 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:32:14.923 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:07.961040+0000", 2024-08-19T14:32:14.924 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:32:14.924 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.924 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.924 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:32:14.924 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:07.961040+0000", 2024-08-19T14:32:14.924 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.2600000000000001e-07 2024-08-19T14:32:14.924 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.924 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.924 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:32:14.924 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:07.961041+0000", 2024-08-19T14:32:14.924 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.482e-06 2024-08-19T14:32:14.924 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.924 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.925 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:32:14.925 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:07.961045+0000", 2024-08-19T14:32:14.925 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.23e-06 2024-08-19T14:32:14.925 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.925 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.925 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:32:14.925 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:07.961047+0000", 2024-08-19T14:32:14.925 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.8200000000000001e-06 2024-08-19T14:32:14.925 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.925 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.925 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:32:14.925 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:07.965714+0000", 2024-08-19T14:32:14.925 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0046665930000000001 2024-08-19T14:32:14.925 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.925 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.926 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for missing object", 2024-08-19T14:32:14.926 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:07.965762+0000", 2024-08-19T14:32:14.926 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7778000000000002e-05 2024-08-19T14:32:14.926 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.926 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.926 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:32:14.926 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:08.797458+0000", 2024-08-19T14:32:14.926 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.83169643000000004 2024-08-19T14:32:14.926 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.926 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.926 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:32:14.926 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:08.797472+0000", 2024-08-19T14:32:14.926 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3536999999999999e-05 2024-08-19T14:32:14.926 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:32:14.927 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:32:14.927 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:32:14.927 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.927 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.927 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:170753 6.6 6:6932d32f:::benchmark_data_smithi045_97937_object63480:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e577)", 2024-08-19T14:32:14.927 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:32:07.961051+0000", 2024-08-19T14:32:14.927 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 6.83137259, 2024-08-19T14:32:14.927 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.845350775, 2024-08-19T14:32:14.927 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:32:14.927 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:32:14.927 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:32:14.927 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:32:14.927 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:32:14.928 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:32:14.928 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 170753 2024-08-19T14:32:14.928 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.928 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:32:14.928 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.928 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:32:14.928 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:07.961051+0000", 2024-08-19T14:32:14.928 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:32:14.929 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:32:14.929 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:32:14.929 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.929 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.929 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:32:14.929 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:07.961051+0000", 2024-08-19T14:32:14.929 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:32:14.929 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.929 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.929 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:32:14.930 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:07.961051+0000", 2024-08-19T14:32:14.930 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.28e-07 2024-08-19T14:32:14.930 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.930 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.930 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:32:14.930 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:07.961053+0000", 2024-08-19T14:32:14.930 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3510000000000001e-06 2024-08-19T14:32:14.930 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.930 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.930 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:32:14.930 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:07.961056+0000", 2024-08-19T14:32:14.930 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.1599999999999998e-06 2024-08-19T14:32:14.930 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.931 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.931 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:32:14.931 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:07.961059+0000", 2024-08-19T14:32:14.931 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.773e-06 2024-08-19T14:32:14.931 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.931 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.931 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:32:14.931 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:07.965780+0000", 2024-08-19T14:32:14.931 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0047209560000000001 2024-08-19T14:32:14.931 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.931 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.931 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for missing object", 2024-08-19T14:32:14.931 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:07.965822+0000", 2024-08-19T14:32:14.931 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2515999999999997e-05 2024-08-19T14:32:14.931 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.932 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.932 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:32:14.932 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:08.806390+0000", 2024-08-19T14:32:14.932 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.840567446 2024-08-19T14:32:14.932 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.932 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.932 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:32:14.932 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:08.806402+0000", 2024-08-19T14:32:14.932 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2345e-05 2024-08-19T14:32:14.932 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:32:14.932 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:32:14.932 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:32:14.932 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.932 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.932 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:171565 6.6 6:60529a06:::benchmark_data_smithi045_97937_object64292:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e578)", 2024-08-19T14:32:14.933 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:32:09.184196+0000", 2024-08-19T14:32:14.933 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 5.6082273019999995, 2024-08-19T14:32:14.933 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.84338733600000004, 2024-08-19T14:32:14.933 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:32:14.933 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:32:14.933 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:32:14.933 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:32:14.933 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:32:14.933 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:32:14.933 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 171565 2024-08-19T14:32:14.933 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.933 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:32:14.933 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.933 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:32:14.934 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:09.184196+0000", 2024-08-19T14:32:14.934 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:32:14.934 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.934 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.934 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:32:14.934 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:09.184196+0000", 2024-08-19T14:32:14.934 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:32:14.934 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.934 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.934 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:32:14.934 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:09.184197+0000", 2024-08-19T14:32:14.934 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.77e-07 2024-08-19T14:32:14.934 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.935 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.935 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:32:14.935 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:09.184202+0000", 2024-08-19T14:32:14.935 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.4689999999999996e-06 2024-08-19T14:32:14.935 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.935 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.935 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:32:14.935 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:09.184208+0000", 2024-08-19T14:32:14.935 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.5550000000000002e-06 2024-08-19T14:32:14.935 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.935 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.935 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:32:14.935 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:09.184212+0000", 2024-08-19T14:32:14.935 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.3980000000000001e-06 2024-08-19T14:32:14.936 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.936 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.936 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:32:14.936 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:09.186929+0000", 2024-08-19T14:32:14.936 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0027163069999999998 2024-08-19T14:32:14.936 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.936 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.936 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for missing object", 2024-08-19T14:32:14.936 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:09.187000+0000", 2024-08-19T14:32:14.936 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.1049000000000004e-05 2024-08-19T14:32:14.936 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.936 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.936 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:32:14.936 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:10.027568+0000", 2024-08-19T14:32:14.937 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.84056829300000002 2024-08-19T14:32:14.937 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:32:14.937 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:32:14.937 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:32:14.937 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:10.027584+0000", 2024-08-19T14:32:14.937 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5888000000000001e-05 2024-08-19T14:32:14.937 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:32:14.937 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:32:14.937 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:32:14.937 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:32:14.937 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:32:14.937 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:32:14.938 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:32:14.938 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:32:14.960 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:32:14.960 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:32:14.964 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:14.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:14.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:32:14.993 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:32:14.993 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:32:15.057 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:32:15.058 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:32:15.065 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:32:15.080 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:32:15.080 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:32:15.091 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:15.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:15.091 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:32:15.110 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:32:15.110 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:32:15.147 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:32:15.147 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:32:15.179 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:32:15.179 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:32:15.193 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:32:15.230 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is unset 2024-08-19T14:32:15.232 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:15.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:15.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:32:15.334 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:32:15.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:15 smithi031 ceph-mon[16336]: pgmap v1350: 47 pgs: 5 active+clean+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 2.5 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 832 op/s; 10556/123327 objects degraded (8.559%); 25124/123327 objects misplaced (20.372%); 4.5 MiB/s, 261 objects/s recovering 2024-08-19T14:32:15.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:15 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3680486580' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:32:15.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:15 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3657455669' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:32:15.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:15 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:32:15.369 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:15.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:15.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:32:15.471 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:32:15.497 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:15.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:15.498 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:32:15.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:15 smithi032 ceph-mon[20004]: pgmap v1350: 47 pgs: 5 active+clean+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 2.5 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 832 op/s; 10556/123327 objects degraded (8.559%); 25124/123327 objects misplaced (20.372%); 4.5 MiB/s, 261 objects/s recovering 2024-08-19T14:32:15.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:15 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3680486580' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:32:15.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:15 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3657455669' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:32:15.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:15 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:32:15.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:15 smithi045 ceph-mon[20143]: pgmap v1350: 47 pgs: 5 active+clean+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 2.5 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 832 op/s; 10556/123327 objects degraded (8.559%); 25124/123327 objects misplaced (20.372%); 4.5 MiB/s, 261 objects/s recovering 2024-08-19T14:32:15.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:15 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3680486580' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:32:15.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:15 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3657455669' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:32:15.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:15 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:32:15.599 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:32:15.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:15.629 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:15.629 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:32:15.730 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:32:15.752 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:15.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:15.753 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:32:15.854 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:32:15.890 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:15.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:15.891 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:32:15.992 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:32:16.013 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:16.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:16.014 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:32:16.114 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:32:16.150 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:16.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:16.151 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:32:16.252 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:32:16.280 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:16.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:16.281 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:32:16.297 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-08-19T14:32:16.382 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:32:16.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:16.426 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:16.427 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:32:16.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:16 smithi032 ceph-mon[20004]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:32:16.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:16 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:32:16.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:16 smithi032 ceph-mon[20004]: osdmap e581: 12 total, 11 up, 9 in 2024-08-19T14:32:16.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:16 smithi032 ceph-mon[20004]: 6.0 scrub starts 2024-08-19T14:32:16.527 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:32:16.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:16 smithi045 ceph-mon[20143]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:32:16.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:16 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:32:16.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:16 smithi045 ceph-mon[20143]: osdmap e581: 12 total, 11 up, 9 in 2024-08-19T14:32:16.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:16 smithi045 ceph-mon[20143]: 6.0 scrub starts 2024-08-19T14:32:16.566 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:16 smithi031 ceph-mon[16336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:32:16.566 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:16 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:32:16.567 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:16 smithi031 ceph-mon[16336]: osdmap e581: 12 total, 11 up, 9 in 2024-08-19T14:32:16.567 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:16 smithi031 ceph-mon[16336]: 6.0 scrub starts 2024-08-19T14:32:16.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:16.574 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:16.574 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:32:16.676 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:32:16.698 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:16.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:16.700 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:32:16.741 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:32:16.800 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:32:16.826 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:16.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:16.828 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:32:16.929 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:32:16.952 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:16.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:16.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:32:17.054 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:32:17.079 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:17.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:17.080 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:32:17.181 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:32:17.206 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:17.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:17.208 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:32:17.308 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:32:17.325 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:32:17.325 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:32:17.333 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:17.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:17.334 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:32:17.436 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:32:17.457 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:17.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:17.458 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:32:17.504 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:17 smithi031 ceph-mon[16336]: pgmap v1352: 47 pgs: 5 active+clean+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 2.4 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 600 op/s; 10350/118317 objects degraded (8.748%); 24332/118317 objects misplaced (20.565%); 5.1 MiB/s, 142 objects/s recovering 2024-08-19T14:32:17.504 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:17 smithi031 ceph-mon[16336]: 6.0 scrub ok 2024-08-19T14:32:17.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:17 smithi032 ceph-mon[20004]: pgmap v1352: 47 pgs: 5 active+clean+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 2.4 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 600 op/s; 10350/118317 objects degraded (8.748%); 24332/118317 objects misplaced (20.565%); 5.1 MiB/s, 142 objects/s recovering 2024-08-19T14:32:17.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:17 smithi032 ceph-mon[20004]: 6.0 scrub ok 2024-08-19T14:32:17.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:17 smithi045 ceph-mon[20143]: pgmap v1352: 47 pgs: 5 active+clean+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 2.4 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 600 op/s; 10350/118317 objects degraded (8.748%); 24332/118317 objects misplaced (20.565%); 5.1 MiB/s, 142 objects/s recovering 2024-08-19T14:32:17.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:17 smithi045 ceph-mon[20143]: 6.0 scrub ok 2024-08-19T14:32:17.559 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:32:17.583 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:17.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:17.584 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:32:17.685 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:32:17.707 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:17.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:17.708 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:32:17.809 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:32:17.835 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:17.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:17.836 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:32:17.937 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:32:17.972 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:17.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:17.973 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:32:18.074 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:32:18.096 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:18.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:18.097 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:32:18.198 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:32:18.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:18.239 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:18.239 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:32:18.273 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-08-19T14:32:18.339 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:32:18.362 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:18.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:18.363 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:32:18.464 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:32:18.487 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:18.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:18.488 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:32:18.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:18 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 10350/118317 objects degraded (8.748%), 7 pgs degraded, 16 pgs undersized (PG_DEGRADED) 2024-08-19T14:32:18.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:18 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 10350/118317 objects degraded (8.748%), 7 pgs degraded, 16 pgs undersized (PG_DEGRADED) 2024-08-19T14:32:18.589 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:32:18.613 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:18.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:18.614 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:32:18.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:18 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 10350/118317 objects degraded (8.748%), 7 pgs degraded, 16 pgs undersized (PG_DEGRADED) 2024-08-19T14:32:18.715 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:32:18.736 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:18.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:18.737 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:32:18.772 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:32:18.838 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:32:18.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:18.861 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:18.861 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:32:18.962 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:32:18.998 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:18.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:18.999 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:32:19.100 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:32:19.128 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:19.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:19.129 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:32:19.231 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:32:19.252 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:19.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:19.254 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:32:19.355 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:32:19.376 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:19.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:19.378 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:32:19.433 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:32:19.433 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:32:19.479 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:32:19.515 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:19.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:19.516 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:32:19.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:19 smithi045 ceph-mon[20143]: pgmap v1353: 47 pgs: 5 active+clean+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 2.4 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 491 op/s; 10350/118317 objects degraded (8.748%); 24332/118317 objects misplaced (20.565%); 4.5 MiB/s, 124 objects/s recovering 2024-08-19T14:32:19.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:19 smithi045 ceph-mon[20143]: 1.1a scrub starts 2024-08-19T14:32:19.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:19 smithi045 ceph-mon[20143]: 1.1a scrub ok 2024-08-19T14:32:19.551 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-19T14:32:19.582 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:19 smithi031 ceph-mon[16336]: pgmap v1353: 47 pgs: 5 active+clean+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 2.4 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 491 op/s; 10350/118317 objects degraded (8.748%); 24332/118317 objects misplaced (20.565%); 4.5 MiB/s, 124 objects/s recovering 2024-08-19T14:32:19.583 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:19 smithi031 ceph-mon[16336]: 1.1a scrub starts 2024-08-19T14:32:19.583 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:19 smithi031 ceph-mon[16336]: 1.1a scrub ok 2024-08-19T14:32:19.617 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:32:19.641 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:19.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:19.642 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:32:19.743 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:32:19.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:19 smithi032 ceph-mon[20004]: pgmap v1353: 47 pgs: 5 active+clean+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 2.4 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 491 op/s; 10350/118317 objects degraded (8.748%); 24332/118317 objects misplaced (20.565%); 4.5 MiB/s, 124 objects/s recovering 2024-08-19T14:32:19.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:19 smithi032 ceph-mon[20004]: 1.1a scrub starts 2024-08-19T14:32:19.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:19 smithi032 ceph-mon[20004]: 1.1a scrub ok 2024-08-19T14:32:19.778 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:19.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:19.779 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:32:19.880 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:32:19.902 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:19.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:19.903 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:32:20.004 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:32:20.032 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:20.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:20.033 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:32:20.135 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:32:20.156 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:20.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:20.157 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:32:20.178 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-08-19T14:32:20.257 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:32:20.290 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:20.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:20.291 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:32:20.392 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:32:20.422 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:20.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:20.423 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:32:20.524 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:32:20.545 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:20.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:20.546 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:32:20.646 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:32:20.647 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:32:20.669 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:20.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:20.670 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:32:20.771 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:32:20.806 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:20.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:20.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:32:20.909 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:32:20.931 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:20.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:20.932 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:32:21.034 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:32:21.076 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:21.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:21.077 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:32:21.179 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:32:21.200 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:21.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:21.201 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:32:21.302 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:32:21.316 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:32:21.317 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:32:21.342 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:21.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:21.343 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:32:21.443 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:32:21.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:21.469 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:21.469 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:32:21.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:21 smithi032 ceph-mon[20004]: pgmap v1354: 47 pgs: 5 active+clean+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 2.2 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 832 op/s; 10016/108501 objects degraded (9.231%); 22561/108501 objects misplaced (20.793%); 7.4 MiB/s, 164 objects/s recovering 2024-08-19T14:32:21.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:21 smithi032 ceph-mon[20004]: 6.2 scrub starts 2024-08-19T14:32:21.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:21 smithi045 ceph-mon[20143]: pgmap v1354: 47 pgs: 5 active+clean+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 2.2 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 832 op/s; 10016/108501 objects degraded (9.231%); 22561/108501 objects misplaced (20.793%); 7.4 MiB/s, 164 objects/s recovering 2024-08-19T14:32:21.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:21 smithi045 ceph-mon[20143]: 6.2 scrub starts 2024-08-19T14:32:21.570 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:32:21.575 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:21 smithi031 ceph-mon[16336]: pgmap v1354: 47 pgs: 5 active+clean+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 2.2 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 832 op/s; 10016/108501 objects degraded (9.231%); 22561/108501 objects misplaced (20.793%); 7.4 MiB/s, 164 objects/s recovering 2024-08-19T14:32:21.575 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:21 smithi031 ceph-mon[16336]: 6.2 scrub starts 2024-08-19T14:32:21.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:21.603 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:21.603 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:32:21.704 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:32:21.735 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:21.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:21.736 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:32:21.837 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:32:21.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:21.876 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:21.876 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:32:21.977 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:32:22.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:22.013 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:22.013 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:32:22.114 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:32:22.144 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:22.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:22.145 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:32:22.218 INFO:teuthology.orchestra.run.smithi031.stderr:Creating an OSD daemon form without an OSD FSID value 2024-08-19T14:32:22.246 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:32:22.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:22.300 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:22.300 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:32:22.313 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-08-19T14:32:22.402 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:32:22.439 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:22.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:22.441 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:32:22.517 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:22 smithi031 ceph-mon[16336]: 6.2 scrub ok 2024-08-19T14:32:22.541 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:32:22.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:22 smithi045 ceph-mon[20143]: 6.2 scrub ok 2024-08-19T14:32:22.567 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-19T14:32:22.568 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-19T14:32:22.572 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:32:22.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:22.583 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:22.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:32:22.685 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:32:22.721 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:22.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:22.722 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:32:22.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:22 smithi032 ceph-mon[20004]: 6.2 scrub ok 2024-08-19T14:32:22.771 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:32:22.823 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:32:22.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:22.846 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:22.847 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:32:22.948 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:32:22.979 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:22.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:22.980 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:32:23.081 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:32:23.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:23.115 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:23.115 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:32:23.217 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:32:23.255 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:23.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:23.256 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:32:23.357 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:32:23.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:23 smithi031 ceph-mon[16336]: pgmap v1355: 47 pgs: 5 active+clean+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 2.2 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 694 op/s; 9804/107865 objects degraded (9.089%); 22561/107865 objects misplaced (20.916%); 6.0 MiB/s, 133 objects/s recovering 2024-08-19T14:32:23.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:23 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 9804/107865 objects degraded (9.089%), 7 pgs degraded, 16 pgs undersized (PG_DEGRADED) 2024-08-19T14:32:23.389 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:23.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:23.390 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:32:23.491 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:32:23.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:23 smithi032 ceph-mon[20004]: pgmap v1355: 47 pgs: 5 active+clean+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 2.2 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 694 op/s; 9804/107865 objects degraded (9.089%); 22561/107865 objects misplaced (20.916%); 6.0 MiB/s, 133 objects/s recovering 2024-08-19T14:32:23.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:23 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 9804/107865 objects degraded (9.089%), 7 pgs degraded, 16 pgs undersized (PG_DEGRADED) 2024-08-19T14:32:23.522 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:23.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:23.523 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:32:23.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:23 smithi045 ceph-mon[20143]: pgmap v1355: 47 pgs: 5 active+clean+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 2.2 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 694 op/s; 9804/107865 objects degraded (9.089%); 22561/107865 objects misplaced (20.916%); 6.0 MiB/s, 133 objects/s recovering 2024-08-19T14:32:23.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:23 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 9804/107865 objects degraded (9.089%), 7 pgs degraded, 16 pgs undersized (PG_DEGRADED) 2024-08-19T14:32:23.581 INFO:teuthology.orchestra.run.smithi031.stdout: marking collection for removal 2024-08-19T14:32:23.581 INFO:teuthology.orchestra.run.smithi031.stdout:setting '_remove' omap key 2024-08-19T14:32:23.581 INFO:teuthology.orchestra.run.smithi031.stdout:finish_remove_pgs 1.5_head removing 1.5 2024-08-19T14:32:23.598 INFO:teuthology.orchestra.run.smithi031.stdout:Remove successful 2024-08-19T14:32:23.624 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:32:23.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:23.654 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:23.654 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:32:23.755 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:32:23.776 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:23.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:23.777 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:32:23.878 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:32:23.912 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:23.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:23.913 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:32:24.015 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:32:24.027 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:32:24.027 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:32:24.036 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:24.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:24.037 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:32:24.138 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:32:24.161 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:24.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:24.162 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:32:24.263 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:32:24.286 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is unset 2024-08-19T14:32:24.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:24.287 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:24.287 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:32:24.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:24 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3736946696' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:32:24.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:24 smithi031 ceph-mon[16336]: 6.9 scrub starts 2024-08-19T14:32:24.389 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:32:24.411 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:24.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:24.413 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:32:24.514 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:32:24.536 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:24.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:24.537 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:32:24.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:24 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3736946696' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:32:24.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:24 smithi045 ceph-mon[20143]: 6.9 scrub starts 2024-08-19T14:32:24.569 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:32:24.638 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:32:24.661 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:24.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:24.662 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:32:24.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:24 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3736946696' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:32:24.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:24 smithi032 ceph-mon[20004]: 6.9 scrub starts 2024-08-19T14:32:24.763 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:32:24.807 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:24.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:24.808 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:32:24.909 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:32:24.932 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:24.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:24.933 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:32:25.034 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:32:25.070 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:25.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:25.071 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:32:25.172 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:32:25.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:25.208 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:25.209 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:32:25.310 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:32:25.361 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:25.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:25.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:32:25.367 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell -n osd.2 --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.1.5.2 2024-08-19T14:32:25.463 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:32:25.486 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:25.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:25.487 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:32:25.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:25 smithi045 ceph-mon[20143]: pgmap v1356: 47 pgs: 5 active+clean+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 2.1 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 689 op/s; 9573/102621 objects degraded (9.329%); 21501/102621 objects misplaced (20.952%); 6.0 MiB/s, 96 objects/s recovering 2024-08-19T14:32:25.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:25 smithi045 ceph-mon[20143]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:32:25.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:25 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3736946696' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:32:25.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:25 smithi045 ceph-mon[20143]: osdmap e582: 12 total, 11 up, 9 in 2024-08-19T14:32:25.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:25 smithi045 ceph-mon[20143]: 6.9 scrub ok 2024-08-19T14:32:25.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:25 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:32:25.588 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:32:25.604 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:25 smithi032 ceph-mon[20004]: pgmap v1356: 47 pgs: 5 active+clean+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 2.1 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 689 op/s; 9573/102621 objects degraded (9.329%); 21501/102621 objects misplaced (20.952%); 6.0 MiB/s, 96 objects/s recovering 2024-08-19T14:32:25.604 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:25 smithi032 ceph-mon[20004]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:32:25.604 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:25 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3736946696' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:32:25.604 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:25 smithi032 ceph-mon[20004]: osdmap e582: 12 total, 11 up, 9 in 2024-08-19T14:32:25.604 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:25 smithi032 ceph-mon[20004]: 6.9 scrub ok 2024-08-19T14:32:25.604 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:25 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:32:25.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:25.611 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:25.611 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:32:25.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:25 smithi031 ceph-mon[16336]: pgmap v1356: 47 pgs: 5 active+clean+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 2.1 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 689 op/s; 9573/102621 objects degraded (9.329%); 21501/102621 objects misplaced (20.952%); 6.0 MiB/s, 96 objects/s recovering 2024-08-19T14:32:25.618 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:25 smithi031 ceph-mon[16336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:32:25.618 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:25 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3736946696' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:32:25.618 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:25 smithi031 ceph-mon[16336]: osdmap e582: 12 total, 11 up, 9 in 2024-08-19T14:32:25.618 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:25 smithi031 ceph-mon[16336]: 6.9 scrub ok 2024-08-19T14:32:25.618 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:25 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:32:25.651 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-08-19T14:32:25.712 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:32:25.792 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:25.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:25.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:32:25.895 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:32:25.909 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/osd.2/config 2024-08-19T14:32:25.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:25.956 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:25.957 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:32:26.058 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:32:26.073 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:32:26.079 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:26.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:26.080 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:32:26.181 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:32:26.205 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:26.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:26.206 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:32:26.307 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:32:26.350 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:26.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:26.351 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:32:26.399 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:26 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:32:26.399 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:26 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:32:26.400 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:26 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:32:26.452 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:32:26.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:26.491 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:26.491 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:32:26.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:26 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:32:26.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:26 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:32:26.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:26 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:32:26.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:26 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:32:26.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:26 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:32:26.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:26 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:32:26.592 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:32:26.615 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:26.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:26.616 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:32:26.715 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:32:26.715 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:32:26.717 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:32:26.739 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:26.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:26.740 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:32:26.840 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:32:26.877 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:26.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:26.878 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:32:26.979 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:32:27.001 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:27.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:27.003 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:32:27.104 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:32:27.136 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:27.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:27.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:32:27.238 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:32:27.242 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:32:27.281 INFO:teuthology.orchestra.run.smithi031.stderr:Creating an OSD daemon form without an OSD FSID value 2024-08-19T14:32:27.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:27.287 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:27.287 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:32:27.388 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:32:27.410 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:27.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:27.411 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:32:27.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:27 smithi032 ceph-mon[20004]: pgmap v1358: 47 pgs: 5 active+clean+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 2.0 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 701 op/s; 9391/97263 objects degraded (9.655%); 20713/97263 objects misplaced (21.296%); 5.7 MiB/s, 91 objects/s recovering 2024-08-19T14:32:27.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:27 smithi032 ceph-mon[20004]: 6.e scrub starts 2024-08-19T14:32:27.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:27 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:32:27.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:27 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:32:27.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:27 smithi032 ceph-mon[20004]: 6.a deep-scrub starts 2024-08-19T14:32:27.512 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:32:27.528 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:27 smithi031 ceph-mon[16336]: pgmap v1358: 47 pgs: 5 active+clean+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 2.0 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 701 op/s; 9391/97263 objects degraded (9.655%); 20713/97263 objects misplaced (21.296%); 5.7 MiB/s, 91 objects/s recovering 2024-08-19T14:32:27.528 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:27 smithi031 ceph-mon[16336]: 6.e scrub starts 2024-08-19T14:32:27.528 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:27 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:32:27.528 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:27 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:32:27.528 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:27 smithi031 ceph-mon[16336]: 6.a deep-scrub starts 2024-08-19T14:32:27.534 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:27.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:27.535 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:32:27.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:27 smithi045 ceph-mon[20143]: pgmap v1358: 47 pgs: 5 active+clean+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 2.0 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 701 op/s; 9391/97263 objects degraded (9.655%); 20713/97263 objects misplaced (21.296%); 5.7 MiB/s, 91 objects/s recovering 2024-08-19T14:32:27.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:27 smithi045 ceph-mon[20143]: 6.e scrub starts 2024-08-19T14:32:27.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:27 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:32:27.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:27 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:32:27.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:27 smithi045 ceph-mon[20143]: 6.a deep-scrub starts 2024-08-19T14:32:27.636 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:32:27.658 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:27.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:27.659 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:32:27.714 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-08-19T14:32:27.760 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:32:27.781 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:27.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:27.782 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:32:27.883 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:32:27.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:27.925 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:27.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:32:28.026 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:32:28.058 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:28.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:28.059 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:32:28.064 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-19T14:32:28.113 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:32:28.113 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:32:28.146 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:32:28.147 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:32:28.147 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-08-19T14:32:28.160 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:32:28.179 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:32:28.180 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:32:28.182 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:32:28.201 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:28.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:28.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:32:28.217 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:32:28.217 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:32:28.249 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:32:28.250 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:32:28.286 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:32:28.286 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:32:28.303 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:32:28.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:28.326 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:28.326 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:32:28.328 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:32:28.328 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:32:28.363 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:32:28.363 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:32:28.395 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:32:28.395 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:32:28.427 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:32:28.431 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:32:28.432 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:32:28.450 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:28.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:28.451 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:32:28.462 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:32:28.462 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:32:28.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:28 smithi045 ceph-mon[20143]: 6.e scrub ok 2024-08-19T14:32:28.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:28 smithi045 ceph-mon[20143]: osdmap e583: 12 total, 11 up, 9 in 2024-08-19T14:32:28.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:28 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 9391/97263 objects degraded (9.655%), 7 pgs degraded, 16 pgs undersized (PG_DEGRADED) 2024-08-19T14:32:28.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:28 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/4130251197' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:32:28.552 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:32:28.606 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:28.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:28.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:32:28.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:28 smithi031 ceph-mon[16336]: 6.e scrub ok 2024-08-19T14:32:28.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:28 smithi031 ceph-mon[16336]: osdmap e583: 12 total, 11 up, 9 in 2024-08-19T14:32:28.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:28 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 9391/97263 objects degraded (9.655%), 7 pgs degraded, 16 pgs undersized (PG_DEGRADED) 2024-08-19T14:32:28.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:28 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/4130251197' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:32:28.618 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:28 smithi032 ceph-mon[20004]: 6.e scrub ok 2024-08-19T14:32:28.618 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:28 smithi032 ceph-mon[20004]: osdmap e583: 12 total, 11 up, 9 in 2024-08-19T14:32:28.618 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:28 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 9391/97263 objects degraded (9.655%), 7 pgs degraded, 16 pgs undersized (PG_DEGRADED) 2024-08-19T14:32:28.618 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:28 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/4130251197' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:32:28.709 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:32:28.730 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:28.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:28.731 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:32:28.795 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:32:28.795 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-08-19T14:32:28.795 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-08-19T14:32:28.795 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-08-19T14:32:28.795 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.795 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:22993 5.7 5:e5194b18:::benchmark_data_smithi045_85309_object22992:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e426)", 2024-08-19T14:32:28.796 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:24:55.243091+0000", 2024-08-19T14:32:28.796 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 453.53875849100001, 2024-08-19T14:32:28.796 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20799368400000001, 2024-08-19T14:32:28.796 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:32:28.796 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:32:28.796 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:32:28.796 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:32:28.796 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:32:28.796 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:32:28.796 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 22993 2024-08-19T14:32:28.796 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.796 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:32:28.797 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.797 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:32:28.797 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.243091+0000", 2024-08-19T14:32:28.797 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:32:28.797 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.797 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.797 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:32:28.797 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.243091+0000", 2024-08-19T14:32:28.797 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:32:28.797 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.797 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.797 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:32:28.798 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.243092+0000", 2024-08-19T14:32:28.798 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.9400000000000004e-07 2024-08-19T14:32:28.798 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.798 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.798 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:32:28.798 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.243122+0000", 2024-08-19T14:32:28.798 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0105000000000001e-05 2024-08-19T14:32:28.798 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.798 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.798 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:32:28.798 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.243131+0000", 2024-08-19T14:32:28.798 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.6889999999999999e-06 2024-08-19T14:32:28.799 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.799 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.799 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:32:28.799 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.243138+0000", 2024-08-19T14:32:28.799 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0339999999999999e-06 2024-08-19T14:32:28.799 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.799 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.799 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:32:28.799 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.243187+0000", 2024-08-19T14:32:28.799 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8764999999999998e-05 2024-08-19T14:32:28.799 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.799 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.800 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:32:28.800 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.243294+0000", 2024-08-19T14:32:28.800 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000106908 2024-08-19T14:32:28.800 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.800 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.800 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 1,9", 2024-08-19T14:32:28.800 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.243424+0000", 2024-08-19T14:32:28.800 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000129441 2024-08-19T14:32:28.800 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.800 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.800 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:32:28.801 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.245731+0000", 2024-08-19T14:32:28.801 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0023073210000000002 2024-08-19T14:32:28.801 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.801 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.801 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:32:28.801 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.246134+0000", 2024-08-19T14:32:28.801 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00040348800000000002 2024-08-19T14:32:28.801 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.801 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.801 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:32:28.801 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.450957+0000", 2024-08-19T14:32:28.801 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20482256099999999 2024-08-19T14:32:28.802 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.802 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.802 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:32:28.802 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.451032+0000", 2024-08-19T14:32:28.802 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4701000000000002e-05 2024-08-19T14:32:28.802 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.802 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.802 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:32:28.802 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.451084+0000", 2024-08-19T14:32:28.802 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2777e-05 2024-08-19T14:32:28.802 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:32:28.802 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:32:28.802 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:32:28.803 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.803 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.803 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:23013 5.7 5:ee7b3505:::benchmark_data_smithi045_85309_object23012:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e426)", 2024-08-19T14:32:28.803 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:24:55.249101+0000", 2024-08-19T14:32:28.803 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 453.532747954, 2024-08-19T14:32:28.803 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20208954800000001, 2024-08-19T14:32:28.803 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:32:28.803 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:32:28.803 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:32:28.803 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:32:28.803 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:32:28.803 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:32:28.803 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 23013 2024-08-19T14:32:28.804 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.804 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:32:28.804 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.804 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:32:28.804 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.249101+0000", 2024-08-19T14:32:28.804 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:32:28.804 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.804 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.804 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:32:28.804 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.249101+0000", 2024-08-19T14:32:28.804 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:32:28.804 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.804 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.804 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:32:28.805 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.249103+0000", 2024-08-19T14:32:28.805 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.271e-06 2024-08-19T14:32:28.805 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.805 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.805 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:32:28.805 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.249300+0000", 2024-08-19T14:32:28.805 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00019775200000000001 2024-08-19T14:32:28.805 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.805 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.805 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:32:28.805 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.249313+0000", 2024-08-19T14:32:28.805 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.312e-05 2024-08-19T14:32:28.805 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.805 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.806 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:32:28.806 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.249321+0000", 2024-08-19T14:32:28.806 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.43e-06 2024-08-19T14:32:28.806 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.806 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.806 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:32:28.806 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.249451+0000", 2024-08-19T14:32:28.806 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000129982 2024-08-19T14:32:28.806 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.806 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.806 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:32:28.806 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.249576+0000", 2024-08-19T14:32:28.806 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012470299999999999 2024-08-19T14:32:28.806 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.807 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.807 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 1,9", 2024-08-19T14:32:28.807 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.249703+0000", 2024-08-19T14:32:28.807 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000127315 2024-08-19T14:32:28.807 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.807 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.807 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:32:28.807 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.253189+0000", 2024-08-19T14:32:28.807 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0034865310000000002 2024-08-19T14:32:28.807 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.807 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.807 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:32:28.808 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.254256+0000", 2024-08-19T14:32:28.808 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0010667630000000001 2024-08-19T14:32:28.808 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.808 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.808 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:32:28.808 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.451121+0000", 2024-08-19T14:32:28.808 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.19686435199999999 2024-08-19T14:32:28.808 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.808 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.808 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:32:28.808 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.451155+0000", 2024-08-19T14:32:28.808 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.4785e-05 2024-08-19T14:32:28.808 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.808 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.808 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:32:28.809 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.451191+0000", 2024-08-19T14:32:28.809 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5543999999999997e-05 2024-08-19T14:32:28.809 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:32:28.809 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:32:28.809 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:32:28.809 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.809 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.809 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:23018 5.7 5:eee02748:::benchmark_data_smithi045_85309_object23017:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e426)", 2024-08-19T14:32:28.809 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:24:55.251860+0000", 2024-08-19T14:32:28.809 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 453.529988828, 2024-08-19T14:32:28.809 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20029677200000001, 2024-08-19T14:32:28.809 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:32:28.809 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:32:28.809 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:32:28.809 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:32:28.810 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:32:28.810 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:32:28.810 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 23018 2024-08-19T14:32:28.810 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.810 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:32:28.810 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.810 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:32:28.810 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.251860+0000", 2024-08-19T14:32:28.810 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:32:28.810 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.810 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.810 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:32:28.811 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.251860+0000", 2024-08-19T14:32:28.811 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:32:28.811 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.811 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.811 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:32:28.811 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.251861+0000", 2024-08-19T14:32:28.811 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.7399999999999991e-07 2024-08-19T14:32:28.811 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.811 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.811 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:32:28.811 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.251893+0000", 2024-08-19T14:32:28.811 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1671000000000002e-05 2024-08-19T14:32:28.812 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.812 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.812 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:32:28.812 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.251903+0000", 2024-08-19T14:32:28.812 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.8360000000000004e-06 2024-08-19T14:32:28.812 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.812 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.812 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:32:28.812 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.251910+0000", 2024-08-19T14:32:28.812 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1760000000000004e-06 2024-08-19T14:32:28.812 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.812 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.813 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:32:28.813 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.251991+0000", 2024-08-19T14:32:28.813 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.0811999999999998e-05 2024-08-19T14:32:28.813 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.813 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.813 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:32:28.813 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.252123+0000", 2024-08-19T14:32:28.813 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013224100000000001 2024-08-19T14:32:28.813 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.813 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.813 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 1,9", 2024-08-19T14:32:28.813 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.252250+0000", 2024-08-19T14:32:28.813 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012669800000000001 2024-08-19T14:32:28.814 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.814 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.814 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:32:28.814 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.254218+0000", 2024-08-19T14:32:28.814 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001968114 2024-08-19T14:32:28.814 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.814 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.814 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:32:28.814 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.254934+0000", 2024-08-19T14:32:28.814 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00071644199999999997 2024-08-19T14:32:28.814 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.814 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.815 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:32:28.815 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.452060+0000", 2024-08-19T14:32:28.815 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.19712607600000001 2024-08-19T14:32:28.815 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.815 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.815 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:32:28.815 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.452114+0000", 2024-08-19T14:32:28.815 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3983999999999997e-05 2024-08-19T14:32:28.815 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.815 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.815 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:32:28.815 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.452157+0000", 2024-08-19T14:32:28.815 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2747999999999998e-05 2024-08-19T14:32:28.816 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:32:28.816 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:32:28.816 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:32:28.816 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.816 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.816 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:55755 5.6 5:699d86b9:::benchmark_data_smithi045_85309_object55754:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e431)", 2024-08-19T14:32:28.816 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:25:09.597312+0000", 2024-08-19T14:32:28.816 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 439.18453711699999, 2024-08-19T14:32:28.816 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.209956798, 2024-08-19T14:32:28.816 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:32:28.816 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:32:28.816 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:32:28.816 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:32:28.817 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:32:28.817 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:32:28.817 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 55755 2024-08-19T14:32:28.817 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.817 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:32:28.817 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.817 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:32:28.817 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.597312+0000", 2024-08-19T14:32:28.817 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:32:28.817 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.817 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.817 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:32:28.817 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.597312+0000", 2024-08-19T14:32:28.817 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:32:28.818 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.818 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.818 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:32:28.818 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.597313+0000", 2024-08-19T14:32:28.818 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.13e-06 2024-08-19T14:32:28.818 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.818 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.818 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:32:28.818 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.597346+0000", 2024-08-19T14:32:28.818 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3019999999999999e-05 2024-08-19T14:32:28.818 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.818 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.819 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:32:28.819 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.597355+0000", 2024-08-19T14:32:28.819 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.0399999999999998e-06 2024-08-19T14:32:28.819 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.819 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.819 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:32:28.819 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.597362+0000", 2024-08-19T14:32:28.819 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2659999999999998e-06 2024-08-19T14:32:28.819 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.819 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.819 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:32:28.819 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.597410+0000", 2024-08-19T14:32:28.819 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8013000000000003e-05 2024-08-19T14:32:28.820 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.820 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.820 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:32:28.820 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.597518+0000", 2024-08-19T14:32:28.820 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00010813 2024-08-19T14:32:28.820 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.820 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.820 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 7,8", 2024-08-19T14:32:28.820 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.597639+0000", 2024-08-19T14:32:28.820 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000121205 2024-08-19T14:32:28.820 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.820 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.821 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:32:28.821 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.600786+0000", 2024-08-19T14:32:28.821 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.003147558 2024-08-19T14:32:28.821 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.821 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.821 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:32:28.821 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.601648+0000", 2024-08-19T14:32:28.821 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00086196199999999997 2024-08-19T14:32:28.821 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.821 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.821 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:32:28.821 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.807181+0000", 2024-08-19T14:32:28.822 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20553236799999999 2024-08-19T14:32:28.822 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.822 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.822 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:32:28.822 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.807220+0000", 2024-08-19T14:32:28.822 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9119999999999998e-05 2024-08-19T14:32:28.822 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.822 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.822 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:32:28.822 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.807269+0000", 2024-08-19T14:32:28.823 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8986000000000002e-05 2024-08-19T14:32:28.823 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:32:28.823 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:32:28.823 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:32:28.823 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.823 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.823 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:55761 5.4 5:21702bb2:::benchmark_data_smithi045_85309_object55760:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e431)", 2024-08-19T14:32:28.823 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:25:09.599621+0000", 2024-08-19T14:32:28.823 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 439.18222831600002, 2024-08-19T14:32:28.823 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20994792700000001, 2024-08-19T14:32:28.823 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:32:28.823 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:32:28.824 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:32:28.824 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:32:28.824 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:32:28.824 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:32:28.824 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 55761 2024-08-19T14:32:28.824 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.824 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:32:28.824 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.824 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:32:28.824 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.599621+0000", 2024-08-19T14:32:28.824 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:32:28.824 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.824 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.824 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:32:28.825 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.599621+0000", 2024-08-19T14:32:28.825 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:32:28.825 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.825 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.825 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:32:28.825 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.599622+0000", 2024-08-19T14:32:28.825 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1689999999999999e-06 2024-08-19T14:32:28.825 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.825 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.825 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:32:28.825 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.599684+0000", 2024-08-19T14:32:28.825 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2228000000000001e-05 2024-08-19T14:32:28.825 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.826 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.826 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:32:28.826 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.599698+0000", 2024-08-19T14:32:28.826 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4031e-05 2024-08-19T14:32:28.826 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.826 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.826 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:32:28.826 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.599705+0000", 2024-08-19T14:32:28.826 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0099999999999998e-06 2024-08-19T14:32:28.826 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.826 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.827 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:32:28.827 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.599804+0000", 2024-08-19T14:32:28.827 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.8288000000000005e-05 2024-08-19T14:32:28.827 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.827 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.827 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:32:28.827 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.600054+0000", 2024-08-19T14:32:28.827 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00025056999999999998 2024-08-19T14:32:28.827 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.827 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.827 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0,8", 2024-08-19T14:32:28.827 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.600199+0000", 2024-08-19T14:32:28.827 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014504200000000001 2024-08-19T14:32:28.828 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.828 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.828 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:32:28.828 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.604843+0000", 2024-08-19T14:32:28.828 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0046438290000000004 2024-08-19T14:32:28.828 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.828 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.828 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:32:28.828 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.604854+0000", 2024-08-19T14:32:28.828 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0971e-05 2024-08-19T14:32:28.828 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.828 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.829 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:32:28.829 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.809503+0000", 2024-08-19T14:32:28.829 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20464875799999999 2024-08-19T14:32:28.829 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.829 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.829 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:32:28.829 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.809534+0000", 2024-08-19T14:32:28.829 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1217999999999998e-05 2024-08-19T14:32:28.829 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.829 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.829 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:32:28.829 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.809569+0000", 2024-08-19T14:32:28.829 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.4813e-05 2024-08-19T14:32:28.830 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:32:28.830 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:32:28.830 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:32:28.830 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.830 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.830 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:55763 5.4 5:22d4b04e:::benchmark_data_smithi045_85309_object55762:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e431)", 2024-08-19T14:32:28.830 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:25:09.599725+0000", 2024-08-19T14:32:28.830 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 439.18212423400001, 2024-08-19T14:32:28.830 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20996778999999999, 2024-08-19T14:32:28.830 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:32:28.830 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:32:28.830 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:32:28.830 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:32:28.830 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:32:28.831 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:32:28.831 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 55763 2024-08-19T14:32:28.831 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.831 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:32:28.831 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.831 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:32:28.831 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.599725+0000", 2024-08-19T14:32:28.831 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:32:28.831 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.831 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.831 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:32:28.831 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.599725+0000", 2024-08-19T14:32:28.831 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:32:28.832 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.832 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.832 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:32:28.832 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.599726+0000", 2024-08-19T14:32:28.832 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2699999999999999e-07 2024-08-19T14:32:28.832 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.832 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.832 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:32:28.832 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.599802+0000", 2024-08-19T14:32:28.832 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6167000000000004e-05 2024-08-19T14:32:28.832 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.832 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.832 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:32:28.832 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.599811+0000", 2024-08-19T14:32:28.833 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.6220000000000004e-06 2024-08-19T14:32:28.833 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.833 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.833 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:32:28.833 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.599818+0000", 2024-08-19T14:32:28.833 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4420000000000003e-06 2024-08-19T14:32:28.833 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.833 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.833 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:32:28.833 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.602462+0000", 2024-08-19T14:32:28.833 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0026439990000000002 2024-08-19T14:32:28.833 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.833 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.834 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:32:28.834 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.602619+0000", 2024-08-19T14:32:28.834 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00015674800000000001 2024-08-19T14:32:28.834 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.834 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.834 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0,8", 2024-08-19T14:32:28.834 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.602748+0000", 2024-08-19T14:32:28.834 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012939900000000001 2024-08-19T14:32:28.834 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.834 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.834 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:32:28.834 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.609052+0000", 2024-08-19T14:32:28.834 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.006304128 2024-08-19T14:32:28.835 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.835 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.835 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:32:28.835 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.611165+0000", 2024-08-19T14:32:28.835 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.002113008 2024-08-19T14:32:28.835 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.835 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.835 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:32:28.835 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.809609+0000", 2024-08-19T14:32:28.835 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.19844395500000001 2024-08-19T14:32:28.835 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.835 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.835 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:32:28.835 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.809648+0000", 2024-08-19T14:32:28.835 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9081000000000001e-05 2024-08-19T14:32:28.836 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.836 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.836 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:32:28.836 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.809693+0000", 2024-08-19T14:32:28.836 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4613999999999997e-05 2024-08-19T14:32:28.836 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:32:28.836 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:32:28.836 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:32:28.836 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.836 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.836 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:55767 5.4 5:200c3bc0:::benchmark_data_smithi045_85309_object55766:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e431)", 2024-08-19T14:32:28.836 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:25:09.601839+0000", 2024-08-19T14:32:28.836 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 439.18001020600002, 2024-08-19T14:32:28.837 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20836729700000001, 2024-08-19T14:32:28.837 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:32:28.837 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:32:28.837 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:32:28.837 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:32:28.837 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:32:28.837 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:32:28.837 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 55767 2024-08-19T14:32:28.837 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.837 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:32:28.837 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.837 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:32:28.837 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.601839+0000", 2024-08-19T14:32:28.837 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:32:28.838 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.838 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.838 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:32:28.838 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.601839+0000", 2024-08-19T14:32:28.838 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:32:28.838 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.838 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.838 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:32:28.838 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.601840+0000", 2024-08-19T14:32:28.838 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.049e-06 2024-08-19T14:32:28.838 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.838 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.838 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:32:28.839 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.601863+0000", 2024-08-19T14:32:28.839 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2748999999999999e-05 2024-08-19T14:32:28.839 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.839 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.839 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:32:28.839 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.601874+0000", 2024-08-19T14:32:28.839 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1219999999999999e-05 2024-08-19T14:32:28.839 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.839 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:32:28.841 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.841 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:32:28.841 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.601881+0000", 2024-08-19T14:32:28.841 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.562e-06 2024-08-19T14:32:28.841 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.841 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.841 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:32:28.841 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.604918+0000", 2024-08-19T14:32:28.842 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.003037629 2024-08-19T14:32:28.842 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.842 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.842 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:32:28.842 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.605062+0000", 2024-08-19T14:32:28.842 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000143821 2024-08-19T14:32:28.842 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.842 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.842 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0,8", 2024-08-19T14:32:28.842 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.605192+0000", 2024-08-19T14:32:28.842 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013037300000000001 2024-08-19T14:32:28.842 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.842 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.843 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:32:28.843 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.611145+0000", 2024-08-19T14:32:28.843 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0059520939999999998 2024-08-19T14:32:28.843 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.843 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.843 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:32:28.843 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.611171+0000", 2024-08-19T14:32:28.843 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6618000000000002e-05 2024-08-19T14:32:28.843 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.843 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.843 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:32:28.843 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.810142+0000", 2024-08-19T14:32:28.843 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.19897035199999999 2024-08-19T14:32:28.844 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.844 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.844 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:32:28.844 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.810178+0000", 2024-08-19T14:32:28.844 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6950999999999999e-05 2024-08-19T14:32:28.844 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.844 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.844 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:32:28.844 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.810206+0000", 2024-08-19T14:32:28.844 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7878999999999999e-05 2024-08-19T14:32:28.844 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:32:28.844 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:32:28.844 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:32:28.844 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.845 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.845 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:55775 5.4 5:2da10789:::benchmark_data_smithi045_85309_object55774:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e431)", 2024-08-19T14:32:28.845 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:25:09.605530+0000", 2024-08-19T14:32:28.845 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 439.176319659, 2024-08-19T14:32:28.845 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20478083, 2024-08-19T14:32:28.845 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:32:28.845 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:32:28.845 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:32:28.845 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:32:28.845 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:32:28.845 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:32:28.845 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 55775 2024-08-19T14:32:28.846 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.846 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:32:28.846 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.846 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:32:28.846 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.605530+0000", 2024-08-19T14:32:28.846 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:32:28.846 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.846 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.846 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:32:28.846 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.605530+0000", 2024-08-19T14:32:28.846 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:32:28.846 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.846 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.847 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:32:28.847 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.605531+0000", 2024-08-19T14:32:28.847 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3039999999999999e-06 2024-08-19T14:32:28.847 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.847 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.847 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:32:28.847 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.605633+0000", 2024-08-19T14:32:28.847 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000102602 2024-08-19T14:32:28.847 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.847 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.847 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:32:28.847 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.605643+0000", 2024-08-19T14:32:28.848 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.4399999999999994e-06 2024-08-19T14:32:28.848 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.848 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.848 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:32:28.848 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.605649+0000", 2024-08-19T14:32:28.848 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2169999999999999e-06 2024-08-19T14:32:28.848 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.848 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.848 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:32:28.848 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.607082+0000", 2024-08-19T14:32:28.848 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0014326289999999999 2024-08-19T14:32:28.848 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.848 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.849 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:32:28.849 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.607190+0000", 2024-08-19T14:32:28.849 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000108428 2024-08-19T14:32:28.849 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.849 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.849 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0,8", 2024-08-19T14:32:28.849 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.607312+0000", 2024-08-19T14:32:28.849 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000121927 2024-08-19T14:32:28.849 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.849 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.849 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:32:28.849 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.611175+0000", 2024-08-19T14:32:28.849 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0038631889999999999 2024-08-19T14:32:28.849 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.850 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.850 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:32:28.850 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.614394+0000", 2024-08-19T14:32:28.850 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.003218505 2024-08-19T14:32:28.850 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.850 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.850 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:32:28.850 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.810243+0000", 2024-08-19T14:32:28.850 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.19584892000000001 2024-08-19T14:32:28.850 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.850 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.850 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:32:28.850 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.810269+0000", 2024-08-19T14:32:28.851 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6027e-05 2024-08-19T14:32:28.851 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.851 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.851 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:32:28.851 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.810310+0000", 2024-08-19T14:32:28.851 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1641999999999999e-05 2024-08-19T14:32:28.851 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:32:28.851 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:32:28.851 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:32:28.851 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.851 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.851 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:55779 5.4 5:28c62a63:::benchmark_data_smithi045_85309_object55778:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e431)", 2024-08-19T14:32:28.851 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:25:09.607121+0000", 2024-08-19T14:32:28.852 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 439.174728329, 2024-08-19T14:32:28.852 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20333711199999999, 2024-08-19T14:32:28.852 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:32:28.852 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:32:28.852 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:32:28.852 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:32:28.852 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:32:28.852 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:32:28.852 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 55779 2024-08-19T14:32:28.852 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.852 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:32:28.852 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.852 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:32:28.853 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.607121+0000", 2024-08-19T14:32:28.853 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:32:28.853 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.853 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.853 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:32:28.853 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.607121+0000", 2024-08-19T14:32:28.853 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:32:28.853 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.853 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.853 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:32:28.853 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.607122+0000", 2024-08-19T14:32:28.853 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1820000000000001e-06 2024-08-19T14:32:28.853 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.853 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.854 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:32:28.854 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.607157+0000", 2024-08-19T14:32:28.854 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.4443999999999998e-05 2024-08-19T14:32:28.854 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.854 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.854 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:32:28.854 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.607169+0000", 2024-08-19T14:32:28.854 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2531000000000001e-05 2024-08-19T14:32:28.854 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.854 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.854 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:32:28.854 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.607176+0000", 2024-08-19T14:32:28.854 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1199999999999996e-06 2024-08-19T14:32:28.855 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.855 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.855 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:32:28.855 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.609081+0000", 2024-08-19T14:32:28.855 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001904705 2024-08-19T14:32:28.855 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.855 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.855 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:32:28.855 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.609202+0000", 2024-08-19T14:32:28.855 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000121138 2024-08-19T14:32:28.855 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.855 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.855 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0,8", 2024-08-19T14:32:28.856 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.609328+0000", 2024-08-19T14:32:28.856 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012569100000000001 2024-08-19T14:32:28.856 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.856 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.856 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:32:28.856 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.611568+0000", 2024-08-19T14:32:28.856 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0022404769999999998 2024-08-19T14:32:28.856 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.856 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.856 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:32:28.856 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.616823+0000", 2024-08-19T14:32:28.856 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0052546889999999999 2024-08-19T14:32:28.856 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.856 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.857 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:32:28.857 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.810361+0000", 2024-08-19T14:32:28.857 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.193538245 2024-08-19T14:32:28.857 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.857 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.857 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:32:28.857 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.810410+0000", 2024-08-19T14:32:28.857 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8819000000000001e-05 2024-08-19T14:32:28.857 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.857 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.857 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:32:28.857 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.810458+0000", 2024-08-19T14:32:28.857 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8071000000000001e-05 2024-08-19T14:32:28.857 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:32:28.858 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:32:28.858 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:32:28.858 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.858 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.858 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:81628 5.7 5:e12ddb93:::benchmark_data_smithi045_85309_object81627:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e437)", 2024-08-19T14:32:28.858 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:25:25.047926+0000", 2024-08-19T14:32:28.858 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 423.73392336500001, 2024-08-19T14:32:28.858 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.207246069, 2024-08-19T14:32:28.858 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:32:28.858 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:32:28.858 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:32:28.858 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:32:28.858 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:32:28.858 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:32:28.859 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 81628 2024-08-19T14:32:28.859 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.859 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:32:28.859 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.859 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:32:28.859 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.047926+0000", 2024-08-19T14:32:28.859 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:32:28.859 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.859 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.859 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:32:28.859 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.047926+0000", 2024-08-19T14:32:28.859 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:32:28.859 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.860 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.860 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:32:28.860 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.047927+0000", 2024-08-19T14:32:28.860 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0219999999999999e-06 2024-08-19T14:32:28.860 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.860 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.860 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:32:28.860 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.047977+0000", 2024-08-19T14:32:28.860 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0170000000000002e-05 2024-08-19T14:32:28.860 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.861 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.861 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:32:28.861 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.047993+0000", 2024-08-19T14:32:28.861 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5835999999999999e-05 2024-08-19T14:32:28.861 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.861 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.861 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:32:28.861 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.047998+0000", 2024-08-19T14:32:28.861 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8500000000000002e-06 2024-08-19T14:32:28.861 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.861 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.861 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:32:28.862 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.048045+0000", 2024-08-19T14:32:28.862 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6828e-05 2024-08-19T14:32:28.862 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.862 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.862 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:32:28.862 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.048139+0000", 2024-08-19T14:32:28.862 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.4199999999999999e-05 2024-08-19T14:32:28.862 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.862 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.862 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 1,9", 2024-08-19T14:32:28.862 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.048232+0000", 2024-08-19T14:32:28.863 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3393999999999994e-05 2024-08-19T14:32:28.863 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.863 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.863 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:32:28.863 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.050672+0000", 2024-08-19T14:32:28.863 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0024403189999999998 2024-08-19T14:32:28.863 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.863 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.863 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:32:28.863 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.051858+0000", 2024-08-19T14:32:28.863 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0011859489999999999 2024-08-19T14:32:28.863 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.863 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.864 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:32:28.864 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.255101+0000", 2024-08-19T14:32:28.864 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20324223 2024-08-19T14:32:28.864 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.864 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.864 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:32:28.864 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.255142+0000", 2024-08-19T14:32:28.864 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0915999999999999e-05 2024-08-19T14:32:28.864 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.864 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.864 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:32:28.864 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.255172+0000", 2024-08-19T14:32:28.865 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0355e-05 2024-08-19T14:32:28.865 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:32:28.865 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:32:28.865 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:32:28.865 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.865 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.865 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:81639 5.7 5:ef0c5b8b:::benchmark_data_smithi045_85309_object81638:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e437)", 2024-08-19T14:32:28.865 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:25:25.053488+0000", 2024-08-19T14:32:28.865 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 423.72836148499999, 2024-08-19T14:32:28.865 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20480580300000001, 2024-08-19T14:32:28.865 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:32:28.865 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:32:28.865 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:32:28.865 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:32:28.866 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:32:28.866 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:32:28.866 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 81639 2024-08-19T14:32:28.866 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.866 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:32:28.866 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.866 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:32:28.866 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.053488+0000", 2024-08-19T14:32:28.866 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:32:28.866 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.866 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.866 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:32:28.866 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.053488+0000", 2024-08-19T14:32:28.867 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:32:28.867 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.867 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.867 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:32:28.867 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.053489+0000", 2024-08-19T14:32:28.867 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.4200000000000005e-07 2024-08-19T14:32:28.867 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.867 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.867 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:32:28.867 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.053559+0000", 2024-08-19T14:32:28.867 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0240000000000005e-05 2024-08-19T14:32:28.867 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.867 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.868 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:32:28.868 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.053568+0000", 2024-08-19T14:32:28.868 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.9989999999999997e-06 2024-08-19T14:32:28.868 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.868 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.868 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:32:28.868 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.053575+0000", 2024-08-19T14:32:28.868 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.092e-06 2024-08-19T14:32:28.868 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.868 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.868 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:32:28.868 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.053624+0000", 2024-08-19T14:32:28.868 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9209000000000003e-05 2024-08-19T14:32:28.869 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.869 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.869 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:32:28.869 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.053826+0000", 2024-08-19T14:32:28.869 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000201573 2024-08-19T14:32:28.869 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.869 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.869 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 1,9", 2024-08-19T14:32:28.869 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.054893+0000", 2024-08-19T14:32:28.869 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0010669849999999999 2024-08-19T14:32:28.869 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.869 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.869 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:32:28.870 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.058523+0000", 2024-08-19T14:32:28.870 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0036301929999999999 2024-08-19T14:32:28.870 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.870 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.870 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:32:28.870 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.060958+0000", 2024-08-19T14:32:28.870 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0024348579999999998 2024-08-19T14:32:28.870 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.870 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.870 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:32:28.870 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.258229+0000", 2024-08-19T14:32:28.870 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.19727160199999999 2024-08-19T14:32:28.871 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.871 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.871 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:32:28.871 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.258265+0000", 2024-08-19T14:32:28.871 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5717999999999999e-05 2024-08-19T14:32:28.871 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.871 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.871 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:32:28.871 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.258294+0000", 2024-08-19T14:32:28.871 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8492000000000001e-05 2024-08-19T14:32:28.871 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:32:28.871 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:32:28.871 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:32:28.871 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.872 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.872 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:81655 5.7 5:e1e2b816:::benchmark_data_smithi045_85309_object81654:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e437)", 2024-08-19T14:32:28.872 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:25:25.061831+0000", 2024-08-19T14:32:28.872 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 423.720018482, 2024-08-19T14:32:28.872 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.19841341400000001, 2024-08-19T14:32:28.872 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:32:28.872 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:32:28.872 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:32:28.872 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:32:28.872 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:32:28.872 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:32:28.872 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 81655 2024-08-19T14:32:28.872 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.873 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:32:28.873 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.873 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:32:28.873 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.061831+0000", 2024-08-19T14:32:28.873 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:32:28.873 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.873 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.873 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:32:28.873 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.061831+0000", 2024-08-19T14:32:28.873 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:32:28.873 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.873 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.873 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:32:28.874 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.061831+0000", 2024-08-19T14:32:28.874 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4099999999999998e-07 2024-08-19T14:32:28.874 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.874 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.874 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:32:28.874 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.061902+0000", 2024-08-19T14:32:28.874 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0333999999999994e-05 2024-08-19T14:32:28.874 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.874 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.874 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:32:28.874 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.061912+0000", 2024-08-19T14:32:28.874 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0125e-05 2024-08-19T14:32:28.874 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.875 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.875 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:32:28.875 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.061918+0000", 2024-08-19T14:32:28.875 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.879e-06 2024-08-19T14:32:28.875 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.875 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.875 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:32:28.875 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.061972+0000", 2024-08-19T14:32:28.875 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4756000000000002e-05 2024-08-19T14:32:28.875 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.875 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.875 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:32:28.875 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.062117+0000", 2024-08-19T14:32:28.876 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000144103 2024-08-19T14:32:28.876 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.876 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.876 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 1,9", 2024-08-19T14:32:28.876 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.062257+0000", 2024-08-19T14:32:28.876 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000140271 2024-08-19T14:32:28.876 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.876 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.876 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:32:28.876 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.065634+0000", 2024-08-19T14:32:28.876 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0033769440000000002 2024-08-19T14:32:28.876 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.876 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.876 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:32:28.877 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.066411+0000", 2024-08-19T14:32:28.877 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00077719599999999996 2024-08-19T14:32:28.877 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.877 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.877 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:32:28.877 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.260128+0000", 2024-08-19T14:32:28.877 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.19371739800000001 2024-08-19T14:32:28.877 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.877 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.877 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:32:28.877 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.260203+0000", 2024-08-19T14:32:28.877 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4293000000000002e-05 2024-08-19T14:32:28.877 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.878 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.878 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:32:28.878 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.260244+0000", 2024-08-19T14:32:28.878 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1474000000000003e-05 2024-08-19T14:32:28.878 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:32:28.878 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:32:28.878 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:32:28.878 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.878 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.878 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:92831 5.f 5:fd6d6df7:::benchmark_data_smithi045_85309_object92830:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e445)", 2024-08-19T14:32:28.878 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:25:37.231829+0000", 2024-08-19T14:32:28.878 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 411.55002051399998, 2024-08-19T14:32:28.878 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.194037033, 2024-08-19T14:32:28.879 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:32:28.879 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:32:28.879 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:32:28.879 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:32:28.879 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:32:28.879 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:32:28.879 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 92831 2024-08-19T14:32:28.879 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.879 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:32:28.879 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.879 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:32:28.879 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.231829+0000", 2024-08-19T14:32:28.879 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:32:28.880 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.880 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.880 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:32:28.880 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.231829+0000", 2024-08-19T14:32:28.880 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:32:28.880 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.880 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.880 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:32:28.880 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.231831+0000", 2024-08-19T14:32:28.880 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8670000000000001e-06 2024-08-19T14:32:28.880 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.880 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.880 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:32:28.881 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.231959+0000", 2024-08-19T14:32:28.881 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000128181 2024-08-19T14:32:28.881 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.881 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.881 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:32:28.881 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.231972+0000", 2024-08-19T14:32:28.881 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3689999999999999e-05 2024-08-19T14:32:28.881 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.881 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.881 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:32:28.881 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.231988+0000", 2024-08-19T14:32:28.881 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.562e-05 2024-08-19T14:32:28.882 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.882 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.882 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:32:28.882 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.232043+0000", 2024-08-19T14:32:28.882 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4694999999999997e-05 2024-08-19T14:32:28.882 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.882 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.882 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:32:28.882 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.232161+0000", 2024-08-19T14:32:28.882 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00011810499999999999 2024-08-19T14:32:28.882 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.882 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.882 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4,11", 2024-08-19T14:32:28.882 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.232284+0000", 2024-08-19T14:32:28.883 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000123452 2024-08-19T14:32:28.883 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.883 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.883 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:32:28.883 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.235655+0000", 2024-08-19T14:32:28.883 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.003370428 2024-08-19T14:32:28.883 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.883 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.883 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:32:28.883 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.243630+0000", 2024-08-19T14:32:28.883 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0079749609999999992 2024-08-19T14:32:28.883 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.884 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.884 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:32:28.884 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.425783+0000", 2024-08-19T14:32:28.884 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.18215342500000001 2024-08-19T14:32:28.884 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.884 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.884 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:32:28.884 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.425829+0000", 2024-08-19T14:32:28.884 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5809999999999997e-05 2024-08-19T14:32:28.884 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.884 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.884 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:32:28.884 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.425866+0000", 2024-08-19T14:32:28.885 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6798999999999998e-05 2024-08-19T14:32:28.885 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:32:28.885 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:32:28.885 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:32:28.885 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.885 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.885 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:92839 5.f 5:f7605d59:::benchmark_data_smithi045_85309_object92838:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e445)", 2024-08-19T14:32:28.885 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:25:37.241907+0000", 2024-08-19T14:32:28.885 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 411.539942666, 2024-08-19T14:32:28.885 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20647278499999999, 2024-08-19T14:32:28.885 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:32:28.885 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:32:28.885 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:32:28.886 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:32:28.886 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:32:28.886 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:32:28.886 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 92839 2024-08-19T14:32:28.886 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.886 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:32:28.886 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.886 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:32:28.886 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.241907+0000", 2024-08-19T14:32:28.886 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:32:28.886 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.886 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.886 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:32:28.887 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.241907+0000", 2024-08-19T14:32:28.887 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:32:28.887 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.887 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.887 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:32:28.887 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.241908+0000", 2024-08-19T14:32:28.887 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4050000000000001e-06 2024-08-19T14:32:28.887 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.887 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.887 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:32:28.887 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.241947+0000", 2024-08-19T14:32:28.887 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8847999999999998e-05 2024-08-19T14:32:28.887 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.888 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.888 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:32:28.888 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.241956+0000", 2024-08-19T14:32:28.888 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.5550000000000008e-06 2024-08-19T14:32:28.888 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.888 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.888 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:32:28.888 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.241962+0000", 2024-08-19T14:32:28.888 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2490000000000001e-06 2024-08-19T14:32:28.888 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.888 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.888 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:32:28.889 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.243721+0000", 2024-08-19T14:32:28.889 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0017588910000000001 2024-08-19T14:32:28.889 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.889 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.889 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:32:28.889 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.422533+0000", 2024-08-19T14:32:28.889 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.178812148 2024-08-19T14:32:28.889 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.889 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.889 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4,11", 2024-08-19T14:32:28.889 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.422686+0000", 2024-08-19T14:32:28.889 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00015375499999999999 2024-08-19T14:32:28.889 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.890 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.890 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:32:28.890 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.426020+0000", 2024-08-19T14:32:28.890 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.003333698 2024-08-19T14:32:28.890 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.890 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.890 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:32:28.890 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.430495+0000", 2024-08-19T14:32:28.890 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0044751740000000002 2024-08-19T14:32:28.890 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.890 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.890 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:32:28.890 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.435394+0000", 2024-08-19T14:32:28.891 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0048991900000000003 2024-08-19T14:32:28.891 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.891 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.891 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:32:28.891 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.435458+0000", 2024-08-19T14:32:28.891 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3068e-05 2024-08-19T14:32:28.891 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.891 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.891 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:32:28.891 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.448379+0000", 2024-08-19T14:32:28.891 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.012921804 2024-08-19T14:32:28.891 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:32:28.891 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:32:28.892 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:32:28.892 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.892 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.892 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.25813.0:92826 5.15 e445/442 5:a965286d:::benchmark_data_smithi045_85309_object92825:head v 445'5490, mlcod=423'262)", 2024-08-19T14:32:28.892 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:25:37.245988+0000", 2024-08-19T14:32:28.892 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 411.53586098099998, 2024-08-19T14:32:28.892 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20615012599999999, 2024-08-19T14:32:28.892 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:32:28.892 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:32:28.892 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:32:28.892 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:32:28.892 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.893 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:32:28.893 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.245988+0000", 2024-08-19T14:32:28.893 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:32:28.893 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.893 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.893 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:32:28.893 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.245988+0000", 2024-08-19T14:32:28.893 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:32:28.893 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.893 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.893 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:32:28.893 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.245989+0000", 2024-08-19T14:32:28.894 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0210000000000001e-06 2024-08-19T14:32:28.894 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.894 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.894 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:32:28.894 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.246023+0000", 2024-08-19T14:32:28.894 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.4044999999999997e-05 2024-08-19T14:32:28.894 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.894 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.894 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:32:28.894 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.422398+0000", 2024-08-19T14:32:28.894 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.17637508399999999 2024-08-19T14:32:28.894 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.895 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.895 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:32:28.895 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.422438+0000", 2024-08-19T14:32:28.895 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9362e-05 2024-08-19T14:32:28.895 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.895 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.895 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:32:28.895 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.426229+0000", 2024-08-19T14:32:28.895 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.003791141 2024-08-19T14:32:28.895 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.895 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.895 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:32:28.895 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.426253+0000", 2024-08-19T14:32:28.896 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4159e-05 2024-08-19T14:32:28.896 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.896 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.896 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:32:28.896 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.452061+0000", 2024-08-19T14:32:28.896 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.025807672 2024-08-19T14:32:28.896 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.896 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.896 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:32:28.896 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.452138+0000", 2024-08-19T14:32:28.896 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7642e-05 2024-08-19T14:32:28.896 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:32:28.896 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:32:28.897 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:32:28.897 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.897 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.897 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:137353 5.4 5:21a8a66c:::benchmark_data_smithi045_85309_object17100:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e465)", 2024-08-19T14:32:28.897 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:26:33.514959+0000", 2024-08-19T14:32:28.897 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 355.26689049100003, 2024-08-19T14:32:28.897 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.163544353, 2024-08-19T14:32:28.897 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:32:28.897 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:32:28.897 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:32:28.897 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:32:28.897 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:32:28.897 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:32:28.898 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 137353 2024-08-19T14:32:28.898 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.898 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:32:28.898 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.898 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:32:28.898 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:26:33.514959+0000", 2024-08-19T14:32:28.898 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:32:28.898 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.898 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.898 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:32:28.898 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:26:33.514959+0000", 2024-08-19T14:32:28.898 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:32:28.898 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.898 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.899 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:32:28.899 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:26:33.514959+0000", 2024-08-19T14:32:28.899 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7000000000000004e-07 2024-08-19T14:32:28.899 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.899 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.899 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:32:28.899 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:26:33.514963+0000", 2024-08-19T14:32:28.899 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.4860000000000002e-06 2024-08-19T14:32:28.899 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.902 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.902 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:32:28.902 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:26:33.514972+0000", 2024-08-19T14:32:28.902 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6710000000000005e-06 2024-08-19T14:32:28.903 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.903 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.903 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:32:28.904 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:26:33.514976+0000", 2024-08-19T14:32:28.904 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5050000000000001e-06 2024-08-19T14:32:28.904 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.904 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.904 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:32:28.904 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:26:33.515031+0000", 2024-08-19T14:32:28.904 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5306000000000002e-05 2024-08-19T14:32:28.904 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.904 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.904 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:32:28.904 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:26:33.515112+0000", 2024-08-19T14:32:28.904 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.0171000000000002e-05 2024-08-19T14:32:28.905 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.905 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.905 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0,8", 2024-08-19T14:32:28.905 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:26:33.515178+0000", 2024-08-19T14:32:28.905 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.5963000000000005e-05 2024-08-19T14:32:28.905 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.905 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.905 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:32:28.905 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:26:33.517095+0000", 2024-08-19T14:32:28.905 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0019177510000000001 2024-08-19T14:32:28.905 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.906 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.906 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:32:28.906 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:26:33.517719+0000", 2024-08-19T14:32:28.906 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00062383399999999998 2024-08-19T14:32:28.906 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.906 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.906 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:32:28.906 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:26:33.678433+0000", 2024-08-19T14:32:28.906 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.16071344200000001 2024-08-19T14:32:28.906 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.906 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.906 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:32:28.907 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:26:33.678470+0000", 2024-08-19T14:32:28.907 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7592999999999997e-05 2024-08-19T14:32:28.907 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.907 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.907 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:32:28.907 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:26:33.678503+0000", 2024-08-19T14:32:28.907 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2960999999999998e-05 2024-08-19T14:32:28.907 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:32:28.907 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:32:28.907 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:32:28.907 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.907 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.907 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:178106 5.15 5:afb34903:::benchmark_data_smithi045_85309_object57853:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e475)", 2024-08-19T14:32:28.908 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:27:08.194118+0000", 2024-08-19T14:32:28.908 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 320.58773101899999, 2024-08-19T14:32:28.908 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.143674514, 2024-08-19T14:32:28.908 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:32:28.908 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:32:28.908 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:32:28.908 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:32:28.908 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:32:28.908 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:32:28.908 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 178106 2024-08-19T14:32:28.908 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.908 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:32:28.908 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.908 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:32:28.909 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:08.194118+0000", 2024-08-19T14:32:28.909 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:32:28.909 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.909 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.909 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:32:28.909 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:08.194118+0000", 2024-08-19T14:32:28.909 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:32:28.909 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.909 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.909 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:32:28.909 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:08.194120+0000", 2024-08-19T14:32:28.909 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.505e-06 2024-08-19T14:32:28.909 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.910 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.910 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:32:28.910 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:08.194125+0000", 2024-08-19T14:32:28.910 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6350000000000003e-06 2024-08-19T14:32:28.910 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.910 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.910 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:32:28.910 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:08.194140+0000", 2024-08-19T14:32:28.910 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4493e-05 2024-08-19T14:32:28.910 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.910 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.910 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:32:28.910 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:08.194147+0000", 2024-08-19T14:32:28.911 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8770000000000004e-06 2024-08-19T14:32:28.911 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.911 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.911 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:32:28.911 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:08.326719+0000", 2024-08-19T14:32:28.911 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.13257237799999999 2024-08-19T14:32:28.911 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.911 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.911 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:32:28.911 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:08.326847+0000", 2024-08-19T14:32:28.911 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012791199999999999 2024-08-19T14:32:28.911 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.911 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.912 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 1,8", 2024-08-19T14:32:28.912 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:08.326954+0000", 2024-08-19T14:32:28.912 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00010745 2024-08-19T14:32:28.912 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.912 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.912 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:32:28.912 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:08.332374+0000", 2024-08-19T14:32:28.912 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0054192959999999997 2024-08-19T14:32:28.912 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.912 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.912 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:32:28.912 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:08.337456+0000", 2024-08-19T14:32:28.912 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0050822100000000002 2024-08-19T14:32:28.912 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.913 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.913 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:32:28.913 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:08.337600+0000", 2024-08-19T14:32:28.913 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014429699999999999 2024-08-19T14:32:28.913 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.913 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.913 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:32:28.913 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:08.337744+0000", 2024-08-19T14:32:28.913 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000143824 2024-08-19T14:32:28.913 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.913 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.913 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:32:28.914 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:27:08.337793+0000", 2024-08-19T14:32:28.914 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8637000000000002e-05 2024-08-19T14:32:28.914 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:32:28.914 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:32:28.914 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:32:28.914 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.914 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.914 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDPGRecoveryDelete(6.e e577,566 6:7b81fd8f:::benchmark_data_smithi045_97937_object14871:head,553'7742)", 2024-08-19T14:32:28.914 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:32:04.320049+0000", 2024-08-19T14:32:28.914 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 24.461800293, 2024-08-19T14:32:28.914 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.093924232999999996, 2024-08-19T14:32:28.914 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:32:28.914 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:32:28.915 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-08-19T14:32:28.915 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:32:28.915 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.915 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:32:28.915 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:04.320049+0000", 2024-08-19T14:32:28.915 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:32:28.915 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.915 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.915 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:32:28.915 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:04.320049+0000", 2024-08-19T14:32:28.915 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:32:28.915 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.915 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.915 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:32:28.916 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:04.320050+0000", 2024-08-19T14:32:28.916 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4600000000000004e-07 2024-08-19T14:32:28.916 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.916 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.916 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:32:28.916 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:04.320054+0000", 2024-08-19T14:32:28.916 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8729999999999998e-06 2024-08-19T14:32:28.916 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.916 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.916 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:32:28.916 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:04.320066+0000", 2024-08-19T14:32:28.916 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1523e-05 2024-08-19T14:32:28.916 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.917 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.917 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:32:28.917 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:04.320074+0000", 2024-08-19T14:32:28.917 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1110000000000008e-06 2024-08-19T14:32:28.917 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.917 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.917 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:32:28.917 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:04.413789+0000", 2024-08-19T14:32:28.917 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.093715074999999995 2024-08-19T14:32:28.917 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.917 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.917 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:32:28.917 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:04.413797+0000", 2024-08-19T14:32:28.918 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.3019999999999995e-06 2024-08-19T14:32:28.918 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.918 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.918 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:32:28.918 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:04.413973+0000", 2024-08-19T14:32:28.918 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00017570300000000001 2024-08-19T14:32:28.918 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:32:28.918 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:32:28.918 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:32:28.918 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.918 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.918 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.26191.0:168397 6.6 e577/550 6:60f858b8:::benchmark_data_smithi045_97937_object61124:head v 577'10483, mlcod=522'3765)", 2024-08-19T14:32:28.918 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:32:04.320084+0000", 2024-08-19T14:32:28.919 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 24.461765411999998, 2024-08-19T14:32:28.919 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.089859497999999996, 2024-08-19T14:32:28.919 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:32:28.919 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:32:28.919 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:32:28.919 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:32:28.919 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.919 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:32:28.919 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:04.320084+0000", 2024-08-19T14:32:28.919 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:32:28.919 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.919 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.919 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:32:28.919 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:04.320084+0000", 2024-08-19T14:32:28.920 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:32:28.920 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.920 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.920 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:32:28.920 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:04.320084+0000", 2024-08-19T14:32:28.920 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6100000000000001e-07 2024-08-19T14:32:28.920 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.920 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.920 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:32:28.920 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:04.320089+0000", 2024-08-19T14:32:28.920 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7090000000000001e-06 2024-08-19T14:32:28.920 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.920 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.921 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:32:28.921 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:04.320097+0000", 2024-08-19T14:32:28.921 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.3110000000000006e-06 2024-08-19T14:32:28.921 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.921 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.921 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:32:28.921 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:04.320103+0000", 2024-08-19T14:32:28.921 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3129999999999998e-06 2024-08-19T14:32:28.921 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.921 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.921 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:32:28.922 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:04.320861+0000", 2024-08-19T14:32:28.922 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00075795099999999998 2024-08-19T14:32:28.922 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.922 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.922 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:32:28.922 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:04.320887+0000", 2024-08-19T14:32:28.922 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6092e-05 2024-08-19T14:32:28.922 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.922 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.922 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:32:28.922 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:04.409904+0000", 2024-08-19T14:32:28.922 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.089017642999999994 2024-08-19T14:32:28.922 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.923 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.923 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:32:28.923 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:04.409943+0000", 2024-08-19T14:32:28.923 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9017999999999998e-05 2024-08-19T14:32:28.923 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:32:28.923 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:32:28.923 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:32:28.923 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.923 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.923 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.26191.0:168399 6.6 e577/550 6:60e60f67:::benchmark_data_smithi045_97937_object61126:head v 577'10484, mlcod=522'3765)", 2024-08-19T14:32:28.923 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:32:04.320112+0000", 2024-08-19T14:32:28.923 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 24.461737723999999, 2024-08-19T14:32:28.923 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.089961263, 2024-08-19T14:32:28.924 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:32:28.924 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:32:28.924 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:32:28.924 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:32:28.924 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.924 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:32:28.924 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:04.320112+0000", 2024-08-19T14:32:28.924 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:32:28.924 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.924 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.924 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:32:28.924 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:04.320112+0000", 2024-08-19T14:32:28.924 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:32:28.925 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.925 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.925 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:32:28.925 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:04.320112+0000", 2024-08-19T14:32:28.925 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3200000000000001e-07 2024-08-19T14:32:28.925 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.925 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.925 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:32:28.925 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:04.320121+0000", 2024-08-19T14:32:28.925 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.8400000000000001e-06 2024-08-19T14:32:28.925 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.925 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.925 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:32:28.925 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:04.320127+0000", 2024-08-19T14:32:28.926 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4409999999999998e-06 2024-08-19T14:32:28.926 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.926 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.926 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:32:28.926 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:04.320132+0000", 2024-08-19T14:32:28.926 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5900000000000001e-06 2024-08-19T14:32:28.926 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.926 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.926 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:32:28.926 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:04.332223+0000", 2024-08-19T14:32:28.926 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.012091601 2024-08-19T14:32:28.926 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.927 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.927 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:32:28.927 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:04.332248+0000", 2024-08-19T14:32:28.927 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4351000000000001e-05 2024-08-19T14:32:28.927 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.927 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.927 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:32:28.927 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:04.409947+0000", 2024-08-19T14:32:28.927 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.077699123999999994 2024-08-19T14:32:28.927 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:28.927 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:28.927 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:32:28.927 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:04.410073+0000", 2024-08-19T14:32:28.927 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012608399999999999 2024-08-19T14:32:28.928 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:32:28.928 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:32:28.928 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:32:28.928 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:32:28.928 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:32:28.928 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:32:28.942 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:28.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:28.943 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:32:29.044 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:32:29.067 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:29.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:29.068 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:32:29.169 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:32:29.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:29.204 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:29.204 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:32:29.255 INFO:teuthology.orchestra.run.smithi031.stdout:get_pg_num_history pg_num_history pg_num_history(e564 pg_nums {1={20=1,110=11,267=21,276=31},2={102=16,219=26},3={222=16,234=26,244=36,296=46},4={335=16},5={419=16,439=26},6={510=16}} deleted_pools 221,2,330,3,418,4,507,5) 2024-08-19T14:32:29.255 INFO:teuthology.orchestra.run.smithi031.stdout:Importing pgid 1.5 2024-08-19T14:32:29.255 INFO:teuthology.orchestra.run.smithi031.stdout:write_pg epoch 559 info 1.5( v 70'139 (0'0,70'139] local-lis/les=558/559 n=0 ec=110/20 lis/c=558/558 les/c/f=559/559/0 sis=558) 2024-08-19T14:32:29.256 INFO:teuthology.orchestra.run.smithi031.stdout:Import successful 2024-08-19T14:32:29.305 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:32:29.336 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:29.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:29.337 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:32:29.438 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:32:29.459 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:29.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:29.460 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:32:29.561 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:32:29.584 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:29.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:29.585 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:32:29.686 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:32:29.707 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:29.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:29.708 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:32:29.809 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:32:29.834 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:29.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:29.835 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:32:29.937 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:32:29.962 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:29.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:29.963 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:32:30.064 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:32:30.087 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:30.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:30.088 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:32:30.189 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:32:30.213 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:30.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:30.214 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:32:30.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:29 smithi032 ceph-mon[20004]: 1.f scrub starts 2024-08-19T14:32:30.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:29 smithi032 ceph-mon[20004]: 1.f scrub ok 2024-08-19T14:32:30.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:29 smithi032 ceph-mon[20004]: pgmap v1360: 47 pgs: 5 active+clean+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 2.0 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 468 op/s; 9391/97263 objects degraded (9.655%); 20713/97263 objects misplaced (21.296%); 3.5 MiB/s, 56 objects/s recovering 2024-08-19T14:32:30.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:29 smithi045 ceph-mon[20143]: 1.f scrub starts 2024-08-19T14:32:30.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:29 smithi045 ceph-mon[20143]: 1.f scrub ok 2024-08-19T14:32:30.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:29 smithi045 ceph-mon[20143]: pgmap v1360: 47 pgs: 5 active+clean+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 2.0 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 468 op/s; 9391/97263 objects degraded (9.655%); 20713/97263 objects misplaced (21.296%); 3.5 MiB/s, 56 objects/s recovering 2024-08-19T14:32:30.315 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:32:30.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:30.350 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:30.351 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:32:30.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:29 smithi031 ceph-mon[16336]: 1.f scrub starts 2024-08-19T14:32:30.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:29 smithi031 ceph-mon[16336]: 1.f scrub ok 2024-08-19T14:32:30.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:29 smithi031 ceph-mon[16336]: pgmap v1360: 47 pgs: 5 active+clean+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 2.0 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 468 op/s; 9391/97263 objects degraded (9.655%); 20713/97263 objects misplaced (21.296%); 3.5 MiB/s, 56 objects/s recovering 2024-08-19T14:32:30.452 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:32:30.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:30.477 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:30.477 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:32:30.578 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:32:30.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:30.613 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:30.613 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:32:30.714 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:32:30.740 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:30.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:30.741 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:32:30.842 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:32:30.875 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:30.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:30.876 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:32:30.977 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:32:31.002 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:31.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:31.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:32:31.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:30 smithi032 ceph-mon[20004]: 1.11 scrub starts 2024-08-19T14:32:31.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:30 smithi032 ceph-mon[20004]: 1.11 scrub ok 2024-08-19T14:32:31.105 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:32:31.129 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:31.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:31.130 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:32:31.231 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:32:31.267 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:31.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:31.268 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:32:31.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:30 smithi045 ceph-mon[20143]: 1.11 scrub starts 2024-08-19T14:32:31.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:30 smithi045 ceph-mon[20143]: 1.11 scrub ok 2024-08-19T14:32:31.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:30 smithi031 ceph-mon[16336]: 1.11 scrub starts 2024-08-19T14:32:31.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:30 smithi031 ceph-mon[16336]: 1.11 scrub ok 2024-08-19T14:32:31.368 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:32:31.391 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:31.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:31.392 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:32:31.493 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:32:31.524 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:31.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:31.525 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:32:31.626 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:32:31.654 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:31.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:31.655 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:32:31.756 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:32:31.778 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:31.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:31.779 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:32:31.880 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:32:31.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:31.915 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:31.915 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:32:32.016 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:32:32.048 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:32.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:32.049 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:32:32.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:31 smithi031 ceph-mon[16336]: pgmap v1361: 47 pgs: 1 active+clean+remapped+scrubbing+deep, 4 active+clean+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 1.8 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 831 op/s; 9152/87894 objects degraded (10.413%); 18869/87894 objects misplaced (21.468%); 7.4 MiB/s, 120 objects/s recovering 2024-08-19T14:32:32.150 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:32:32.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:32.190 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:32.190 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:32:32.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:31 smithi032 ceph-mon[20004]: pgmap v1361: 47 pgs: 1 active+clean+remapped+scrubbing+deep, 4 active+clean+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 1.8 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 831 op/s; 9152/87894 objects degraded (10.413%); 18869/87894 objects misplaced (21.468%); 7.4 MiB/s, 120 objects/s recovering 2024-08-19T14:32:32.291 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:32:32.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:31 smithi045 ceph-mon[20143]: pgmap v1361: 47 pgs: 1 active+clean+remapped+scrubbing+deep, 4 active+clean+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 1.8 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 831 op/s; 9152/87894 objects degraded (10.413%); 18869/87894 objects misplaced (21.468%); 7.4 MiB/s, 120 objects/s recovering 2024-08-19T14:32:32.319 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -f /var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/exp.1.5.2 2024-08-19T14:32:32.321 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:32.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:32.322 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:32:32.423 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:32:32.445 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:32.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:32.446 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:32:32.547 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:32:32.573 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:32.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:32.574 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:32:32.675 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:32:32.696 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:32.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:32.697 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:32:32.798 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:32:32.809 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-19T14:32:32.809 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-19T14:32:32.821 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:32.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:32.822 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:32:32.857 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:32:32.923 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:32:32.945 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:32.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:32.946 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:32:33.019 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:32 smithi031 ceph-mon[16336]: 1.e deep-scrub starts 2024-08-19T14:32:33.019 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:32 smithi031 ceph-mon[16336]: 1.e deep-scrub ok 2024-08-19T14:32:33.019 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:32 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 9152/87894 objects degraded (10.413%), 7 pgs degraded, 16 pgs undersized (PG_DEGRADED) 2024-08-19T14:32:33.046 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:32:33.090 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:33.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:33.091 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:32:33.192 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:32:33.214 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:33.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:33.215 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:32:33.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:32 smithi032 ceph-mon[20004]: 1.e deep-scrub starts 2024-08-19T14:32:33.309 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:32 smithi032 ceph-mon[20004]: 1.e deep-scrub ok 2024-08-19T14:32:33.309 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:32 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 9152/87894 objects degraded (10.413%), 7 pgs degraded, 16 pgs undersized (PG_DEGRADED) 2024-08-19T14:32:33.310 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:32 smithi045 ceph-mon[20143]: 1.e deep-scrub starts 2024-08-19T14:32:33.310 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:32 smithi045 ceph-mon[20143]: 1.e deep-scrub ok 2024-08-19T14:32:33.310 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:32 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 9152/87894 objects degraded (10.413%), 7 pgs degraded, 16 pgs undersized (PG_DEGRADED) 2024-08-19T14:32:33.316 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:32:33.347 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:33.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:33.348 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:32:33.449 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:32:33.480 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:33.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:33.481 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:32:33.582 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:32:33.603 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:33.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:33.604 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:32:33.705 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:32:33.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:33.749 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:33.749 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:32:33.850 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:32:33.876 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:33.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:33.877 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:32:33.933 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is set 2024-08-19T14:32:33.978 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:32:34.026 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:34.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:34.027 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:32:34.128 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:32:34.149 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:34.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:34.150 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:32:34.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:33 smithi031 ceph-mon[16336]: pgmap v1362: 47 pgs: 1 active+clean+remapped+scrubbing+deep, 4 active+clean+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 1.8 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 635 op/s; 8977/87369 objects degraded (10.275%); 18869/87369 objects misplaced (21.597%); 7.4 MiB/s, 120 objects/s recovering 2024-08-19T14:32:34.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:33 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3427063946' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:32:34.251 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:32:34.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:33 smithi032 ceph-mon[20004]: pgmap v1362: 47 pgs: 1 active+clean+remapped+scrubbing+deep, 4 active+clean+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 1.8 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 635 op/s; 8977/87369 objects degraded (10.275%); 18869/87369 objects misplaced (21.597%); 7.4 MiB/s, 120 objects/s recovering 2024-08-19T14:32:34.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:33 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3427063946' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:32:34.274 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:34.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:34.275 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:32:34.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:33 smithi045 ceph-mon[20143]: pgmap v1362: 47 pgs: 1 active+clean+remapped+scrubbing+deep, 4 active+clean+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 1.8 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 635 op/s; 8977/87369 objects degraded (10.275%); 18869/87369 objects misplaced (21.597%); 7.4 MiB/s, 120 objects/s recovering 2024-08-19T14:32:34.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:33 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3427063946' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:32:34.354 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-08-19T14:32:34.376 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:32:34.446 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:34.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:34.448 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:32:34.548 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:32:34.585 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:34.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:34.586 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:32:34.687 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:32:34.710 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:34.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:34.711 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:32:34.785 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:32:34.810 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:32:34.811 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:32:34.834 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:34.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:34.836 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:32:34.937 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:32:34.958 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:34.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:34.959 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:32:35.060 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:32:35.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:35.099 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:35.099 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:32:35.200 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:32:35.216 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:32:35.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:35.224 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:35.224 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:32:35.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:34 smithi032 ceph-mon[20004]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:32:35.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:34 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3427063946' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:32:35.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:34 smithi032 ceph-mon[20004]: osdmap e584: 12 total, 11 up, 9 in 2024-08-19T14:32:35.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:34 smithi045 ceph-mon[20143]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:32:35.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:34 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3427063946' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:32:35.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:34 smithi045 ceph-mon[20143]: osdmap e584: 12 total, 11 up, 9 in 2024-08-19T14:32:35.325 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:32:35.355 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:34 smithi031 ceph-mon[16336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:32:35.355 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:34 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3427063946' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:32:35.355 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:34 smithi031 ceph-mon[16336]: osdmap e584: 12 total, 11 up, 9 in 2024-08-19T14:32:35.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:35.362 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:35.362 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:32:35.463 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:32:35.501 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:35.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:35.502 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:32:35.603 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:32:35.624 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:35.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:35.625 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:32:35.726 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:32:35.747 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:35.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:35.748 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:32:35.849 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:32:35.873 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:35.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:35.874 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:32:35.874 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:32:35.875 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:32:35.875 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:32:35.875 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:32:35.876 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-08-19T14:32:35.890 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:32:35.891 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:32:35.919 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:32:35.919 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:32:35.954 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:35 smithi031 ceph-mon[16336]: pgmap v1364: 47 pgs: 1 active+clean+remapped+scrubbing+deep, 4 active+clean+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 1.7 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 614 op/s; 8794/82506 objects degraded (10.659%); 17822/82506 objects misplaced (21.601%); 3.9 MiB/s, 63 objects/s recovering 2024-08-19T14:32:35.954 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:35 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2880720800' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:32:35.954 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:32:35.954 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:32:35.975 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:32:35.984 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:32:35.984 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:32:36.009 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:36.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:36.010 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:32:36.056 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:32:36.057 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:32:36.088 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:32:36.088 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:32:36.110 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:32:36.118 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:32:36.119 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:32:36.141 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:36.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:36.142 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:32:36.151 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:32:36.152 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:32:36.184 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:32:36.185 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:32:36.243 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:32:36.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:35 smithi032 ceph-mon[20004]: pgmap v1364: 47 pgs: 1 active+clean+remapped+scrubbing+deep, 4 active+clean+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 1.7 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 614 op/s; 8794/82506 objects degraded (10.659%); 17822/82506 objects misplaced (21.601%); 3.9 MiB/s, 63 objects/s recovering 2024-08-19T14:32:36.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:35 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2880720800' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:32:36.273 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:36.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:36.274 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:32:36.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:35 smithi045 ceph-mon[20143]: pgmap v1364: 47 pgs: 1 active+clean+remapped+scrubbing+deep, 4 active+clean+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 1.7 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 614 op/s; 8794/82506 objects degraded (10.659%); 17822/82506 objects misplaced (21.601%); 3.9 MiB/s, 63 objects/s recovering 2024-08-19T14:32:36.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:35 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2880720800' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:32:36.375 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:32:36.403 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:36.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:36.404 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:32:36.506 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:32:36.528 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:36.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:36.529 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:32:36.630 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:32:36.669 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:36.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:36.671 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:32:36.722 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-19T14:32:36.772 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:32:36.800 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:36.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:36.802 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:32:36.903 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:32:36.931 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:36.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:36.932 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:32:37.033 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:32:37.054 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:37.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:37.055 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:32:37.156 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:32:37.185 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:37.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:37.186 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:32:37.287 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:32:37.319 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:37.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:37.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:32:37.361 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 7, 8, 2, 11] out_osds: [10, 6, 9] dead_osds: [2] live_osds: [0, 1, 3, 4, 5, 6, 7, 8, 9, 11, 10] 2024-08-19T14:32:37.362 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-19T14:32:37.362 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-08-19T14:32:37.362 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-19T14:32:37.421 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:32:37.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:37.468 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:37.469 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:32:37.569 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:32:37.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:37.611 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:37.612 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:32:37.713 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:32:37.734 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:37.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:37.736 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:32:37.837 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:32:37.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:37.868 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:37.868 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:32:37.969 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:32:38.020 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:38.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:38.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:32:38.122 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:32:38.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:38.145 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:38.145 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:32:38.246 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:32:38.260 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:37 smithi032 ceph-mon[20004]: pgmap v1365: 47 pgs: 1 active+clean+remapped+scrubbing+deep, 4 active+clean+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 1.6 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 733 op/s; 8476/77883 objects degraded (10.883%); 17062/77883 objects misplaced (21.907%); 7.7 MiB/s, 123 objects/s recovering 2024-08-19T14:32:38.260 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:37 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 8476/77883 objects degraded (10.883%), 7 pgs degraded, 16 pgs undersized (PG_DEGRADED) 2024-08-19T14:32:38.281 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:38.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:38.282 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:32:38.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:37 smithi045 ceph-mon[20143]: pgmap v1365: 47 pgs: 1 active+clean+remapped+scrubbing+deep, 4 active+clean+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 1.6 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 733 op/s; 8476/77883 objects degraded (10.883%); 17062/77883 objects misplaced (21.907%); 7.7 MiB/s, 123 objects/s recovering 2024-08-19T14:32:38.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:37 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 8476/77883 objects degraded (10.883%), 7 pgs degraded, 16 pgs undersized (PG_DEGRADED) 2024-08-19T14:32:38.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:37 smithi031 ceph-mon[16336]: pgmap v1365: 47 pgs: 1 active+clean+remapped+scrubbing+deep, 4 active+clean+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 1.6 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 733 op/s; 8476/77883 objects degraded (10.883%); 17062/77883 objects misplaced (21.907%); 7.7 MiB/s, 123 objects/s recovering 2024-08-19T14:32:38.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:37 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 8476/77883 objects degraded (10.883%), 7 pgs degraded, 16 pgs undersized (PG_DEGRADED) 2024-08-19T14:32:38.383 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:32:38.407 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:38.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:38.408 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:32:38.439 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-19T14:32:38.440 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-19T14:32:38.480 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:32:38.509 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:32:38.551 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:32:38.559 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:38.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:38.560 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:32:38.661 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:32:38.697 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:38.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:38.698 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:32:38.799 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:32:38.830 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:32:38.830 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:32:38.834 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:38.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:38.836 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:32:38.936 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:32:38.958 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:38.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:38.959 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:32:39.060 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:32:39.095 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:39.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:39.096 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:32:39.197 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:32:39.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:39.235 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:39.235 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:32:39.336 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:32:39.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:39.372 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:39.372 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:32:39.386 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:32:39.387 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-08-19T14:32:39.473 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:32:39.502 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:39.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:39.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:32:39.604 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:32:39.644 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:39.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:39.645 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:32:39.746 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:32:39.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:39.781 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:39.781 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:32:39.882 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:32:39.903 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:39.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:39.905 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:32:40.006 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:32:40.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:40.035 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:40.035 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:32:40.136 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:32:40.159 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:40.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:40.161 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:32:40.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:39 smithi032 ceph-mon[20004]: pgmap v1366: 47 pgs: 1 active+clean+remapped+scrubbing+deep, 4 active+clean+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 1.6 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 645 op/s; 8476/77883 objects degraded (10.883%); 17062/77883 objects misplaced (21.907%); 6.8 MiB/s, 108 objects/s recovering 2024-08-19T14:32:40.261 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:32:40.292 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:40.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:40.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:32:40.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:39 smithi045 ceph-mon[20143]: pgmap v1366: 47 pgs: 1 active+clean+remapped+scrubbing+deep, 4 active+clean+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 1.6 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 645 op/s; 8476/77883 objects degraded (10.883%); 17062/77883 objects misplaced (21.907%); 6.8 MiB/s, 108 objects/s recovering 2024-08-19T14:32:40.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:39 smithi031 ceph-mon[16336]: pgmap v1366: 47 pgs: 1 active+clean+remapped+scrubbing+deep, 4 active+clean+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 1.6 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 645 op/s; 8476/77883 objects degraded (10.883%); 17062/77883 objects misplaced (21.907%); 6.8 MiB/s, 108 objects/s recovering 2024-08-19T14:32:40.395 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:32:40.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:40.427 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:40.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:32:40.441 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:32:40.528 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:32:40.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:40.579 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:40.579 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:32:40.680 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:32:40.733 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:40.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:40.734 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:32:40.835 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:32:40.847 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":1366,"stamp":"2024-08-19T14:32:38.104226+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1701805188,"num_objects":25961,"num_object_clones":0,"num_object_copies":77883,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8476,"num_objects_misplaced":17062,"num_objects_unfound":0,"num_objects_dirty":25961,"num_whiteouts":0,"num_read":8,"num_read_kb":7,"num_write":188592,"num_write_kb":6865437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56040,"num_bytes_recovered":3161180261,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":165128,"ondisk_log_size":165128,"up":123,"acting":125,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":151,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":14503008,"kb_used_data":7877644,"kb_used_omap":14,"kb_used_meta":6625009,"kb_avail":829203360,"statfs":{"total":863955320832,"available":849104240640,"internally_reserved":0,"allocated":8066707456,"data_stored":8043227366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14349,"internal_metadata":6784010227},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,6,6,6,10,1,6,2,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-423362560,"num_objects":-6460,"num_object_clones":0,"num_object_copies":-19380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-915,"num_objects_misplaced":-3651,"num_objects_unfound":0,"num_objects_dirty":-6460,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6460,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1089,"num_bytes_recovered":71172096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003056"},"pg_stats":[{"pgid":"6.7","version":"584'11586","reported_seq":11744,"reported_epoch":584,"state":"active+clean","last_fresh":"2024-08-19T14:32:35.032496+0000","last_change":"2024-08-19T14:28:33.150392+0000","last_active":"2024-08-19T14:32:35.032496+0000","last_peered":"2024-08-19T14:32:35.032496+0000","last_clean":"2024-08-19T14:32:35.032496+0000","last_became_active":"2024-08-19T14:28:33.150057+0000","last_became_peered":"2024-08-19T14:28:33.150057+0000","last_unstale":"2024-08-19T14:32:35.032496+0000","last_undegraded":"2024-08-19T14:32:35.032496+0000","last_fullsized":"2024-08-19T14:32:35.032496+0000","mapping_epoch":510,"log_start":"514'1500","ondisk_log_start":"514'1500","created":510,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:28:32.139424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:28:32.139424+0000","last_clean_scrub_stamp":"2024-08-19T14:28:32.139424+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104726528,"num_objects":1598,"num_object_clones":0,"num_object_copies":4794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11586,"num_write_kb":421888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"70'139","reported_seq":1448,"reported_epoch":584,"state":"active+clean","last_fresh":"2024-08-19T14:32:33.953071+0000","last_change":"2024-08-19T14:30:47.916736+0000","last_active":"2024-08-19T14:32:33.953071+0000","last_peered":"2024-08-19T14:32:33.953071+0000","last_clean":"2024-08-19T14:32:33.953071+0000","last_became_active":"2024-08-19T14:30:47.916450+0000","last_became_peered":"2024-08-19T14:30:47.916450+0000","last_unstale":"2024-08-19T14:32:33.953071+0000","last_undegraded":"2024-08-19T14:32:33.953071+0000","last_fullsized":"2024-08-19T14:32:33.953071+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:57.210582+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:19:58.072083+0000","last_clean_scrub_stamp":"2024-08-19T14:28:57.210582+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"584'11820","reported_seq":11979,"reported_epoch":584,"state":"active+clean","last_fresh":"2024-08-19T14:32:33.938640+0000","last_change":"2024-08-19T14:28:33.151848+0000","last_active":"2024-08-19T14:32:33.938640+0000","last_peered":"2024-08-19T14:32:33.938640+0000","last_clean":"2024-08-19T14:32:33.938640+0000","last_became_active":"2024-08-19T14:28:33.150851+0000","last_became_peered":"2024-08-19T14:28:33.150851+0000","last_unstale":"2024-08-19T14:32:33.938640+0000","last_undegraded":"2024-08-19T14:32:33.938640+0000","last_fullsized":"2024-08-19T14:32:33.938640+0000","mapping_epoch":510,"log_start":"514'1800","ondisk_log_start":"514'1800","created":510,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:28:32.139424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:28:32.139424+0000","last_clean_scrub_stamp":"2024-08-19T14:28:32.139424+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110100480,"num_objects":1680,"num_object_clones":0,"num_object_copies":5040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1680,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11820,"num_write_kb":432000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"70'139","reported_seq":1320,"reported_epoch":584,"state":"active+clean","last_fresh":"2024-08-19T14:32:33.939735+0000","last_change":"2024-08-19T14:28:33.188989+0000","last_active":"2024-08-19T14:32:33.939735+0000","last_peered":"2024-08-19T14:32:33.939735+0000","last_clean":"2024-08-19T14:32:33.939735+0000","last_became_active":"2024-08-19T14:18:13.804721+0000","last_became_peered":"2024-08-19T14:18:13.804721+0000","last_unstale":"2024-08-19T14:32:33.939735+0000","last_undegraded":"2024-08-19T14:32:33.939735+0000","last_fullsized":"2024-08-19T14:32:33.939735+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:33.188905+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:28:33.188905+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"584'11795","reported_seq":21486,"reported_epoch":584,"state":"active+undersized+degraded","last_fresh":"2024-08-19T14:32:35.092361+0000","last_change":"2024-08-19T14:31:02.876731+0000","last_active":"2024-08-19T14:32:35.092361+0000","last_peered":"2024-08-19T14:32:35.092361+0000","last_clean":"2024-08-19T14:31:01.953843+0000","last_became_active":"2024-08-19T14:31:02.876731+0000","last_became_peered":"2024-08-19T14:31:02.876731+0000","last_unstale":"2024-08-19T14:32:35.092361+0000","last_undegraded":"2024-08-19T14:31:02.872907+0000","last_fullsized":"2024-08-19T14:31:02.872525+0000","mapping_epoch":565,"log_start":"514'1700","ondisk_log_start":"514'1700","created":510,"last_epoch_clean":564,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:28:32.139424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:28:32.139424+0000","last_clean_scrub_stamp":"2024-08-19T14:28:32.139424+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106496000,"num_objects":1625,"num_object_clones":0,"num_object_copies":4875,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1625,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1625,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11795,"num_write_kb":429440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4759,"num_bytes_recovered":311885824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[{"shards":"5,7","objects":1625}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"70'139","reported_seq":1268,"reported_epoch":584,"state":"active+clean","last_fresh":"2024-08-19T14:32:33.944671+0000","last_change":"2024-08-19T14:28:26.020419+0000","last_active":"2024-08-19T14:32:33.944671+0000","last_peered":"2024-08-19T14:32:33.944671+0000","last_clean":"2024-08-19T14:32:33.944671+0000","last_became_active":"2024-08-19T14:20:50.742354+0000","last_became_peered":"2024-08-19T14:20:50.742354+0000","last_unstale":"2024-08-19T14:32:33.944671+0000","last_undegraded":"2024-08-19T14:32:33.944671+0000","last_fullsized":"2024-08-19T14:32:33.944671+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:26.020327+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:19:59.302931+0000","last_clean_scrub_stamp":"2024-08-19T14:28:26.020327+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.6","version":"584'11766","reported_seq":18103,"reported_epoch":584,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-19T14:32:35.026257+0000","last_change":"2024-08-19T14:30:26.078622+0000","last_active":"2024-08-19T14:32:35.026257+0000","last_peered":"2024-08-19T14:32:35.026257+0000","last_clean":"2024-08-19T14:30:23.347611+0000","last_became_active":"2024-08-19T14:30:26.005124+0000","last_became_peered":"2024-08-19T14:30:26.005124+0000","last_unstale":"2024-08-19T14:32:35.026257+0000","last_undegraded":"2024-08-19T14:30:25.614704+0000","last_fullsized":"2024-08-19T14:30:25.358551+0000","mapping_epoch":550,"log_start":"514'1700","ondisk_log_start":"514'1700","created":510,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:28:32.139424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:28:32.139424+0000","last_clean_scrub_stamp":"2024-08-19T14:28:32.139424+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":1664,"num_object_clones":0,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2468,"num_objects_misplaced":5790,"num_objects_unfound":0,"num_objects_dirty":1664,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11766,"num_write_kb":429760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2058,"num_bytes_recovered":129630208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[3,7],"avail_no_missing":["3","4","7"],"object_location_counts":[{"shards":"3,4,7","objects":1661},{"shards":"3,7","objects":-2465}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"70'139","reported_seq":550,"reported_epoch":583,"state":"active+undersized","last_fresh":"2024-08-19T14:32:27.300809+0000","last_change":"2024-08-19T14:31:02.876535+0000","last_active":"2024-08-19T14:32:27.300809+0000","last_peered":"2024-08-19T14:32:27.300809+0000","last_clean":"2024-08-19T14:30:57.030374+0000","last_became_active":"2024-08-19T14:31:02.876535+0000","last_became_peered":"2024-08-19T14:31:02.876535+0000","last_unstale":"2024-08-19T14:32:27.300809+0000","last_undegraded":"2024-08-19T14:32:27.300809+0000","last_fullsized":"2024-08-19T14:31:02.872426+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:28.323773+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:28:28.323773+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:32:29.003161+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11],"acting":[0,11],"avail_no_missing":["0","11"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"583'11815","reported_seq":24659,"reported_epoch":583,"state":"active+clean+remapped","last_fresh":"2024-08-19T14:32:33.856196+0000","last_change":"2024-08-19T14:32:16.819215+0000","last_active":"2024-08-19T14:32:33.856196+0000","last_peered":"2024-08-19T14:32:33.856196+0000","last_clean":"2024-08-19T14:32:33.856196+0000","last_became_active":"2024-08-19T14:31:03.996500+0000","last_became_peered":"2024-08-19T14:31:03.996500+0000","last_unstale":"2024-08-19T14:32:33.856196+0000","last_undegraded":"2024-08-19T14:32:33.856196+0000","last_fullsized":"2024-08-19T14:32:33.856196+0000","mapping_epoch":566,"log_start":"514'1800","ondisk_log_start":"514'1800","created":510,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"581'11071","last_scrub_stamp":"2024-08-19T14:32:16.818924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:28:32.139424+0000","last_clean_scrub_stamp":"2024-08-19T14:32:16.818924+0000","objects_scrubbed":2350,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:33:17.247075+0000","scrub_duration":798,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103874560,"num_objects":1585,"num_object_clones":0,"num_object_copies":4755,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1585,"num_objects_unfound":0,"num_objects_dirty":1585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11815,"num_write_kb":428800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6322,"num_bytes_recovered":314900480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3,5],"avail_no_missing":["0","3","5"],"object_location_counts":[{"shards":"0,3,5","objects":1585}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"70'139","reported_seq":980,"reported_epoch":584,"state":"active+undersized","last_fresh":"2024-08-19T14:32:33.939803+0000","last_change":"2024-08-19T14:31:02.880951+0000","last_active":"2024-08-19T14:32:33.939803+0000","last_peered":"2024-08-19T14:32:33.939803+0000","last_clean":"2024-08-19T14:30:57.035896+0000","last_became_active":"2024-08-19T14:31:02.880951+0000","last_became_peered":"2024-08-19T14:31:02.880951+0000","last_unstale":"2024-08-19T14:32:33.939803+0000","last_undegraded":"2024-08-19T14:32:33.939803+0000","last_fullsized":"2024-08-19T14:31:02.878172+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:27.536928+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:21:46.057624+0000","last_clean_scrub_stamp":"2024-08-19T14:28:27.536928+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1","version":"584'11491","reported_seq":20952,"reported_epoch":584,"state":"active+undersized+degraded","last_fresh":"2024-08-19T14:32:35.202469+0000","last_change":"2024-08-19T14:31:02.875748+0000","last_active":"2024-08-19T14:32:35.202469+0000","last_peered":"2024-08-19T14:32:35.202469+0000","last_clean":"2024-08-19T14:31:02.007145+0000","last_became_active":"2024-08-19T14:31:02.875748+0000","last_became_peered":"2024-08-19T14:31:02.875748+0000","last_unstale":"2024-08-19T14:32:35.202469+0000","last_undegraded":"2024-08-19T14:31:02.871217+0000","last_fullsized":"2024-08-19T14:31:02.871094+0000","mapping_epoch":565,"log_start":"514'1400","ondisk_log_start":"514'1400","created":510,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:28:32.139424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:28:32.139424+0000","last_clean_scrub_stamp":"2024-08-19T14:28:32.139424+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102825984,"num_objects":1569,"num_object_clones":0,"num_object_copies":4707,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1569,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1569,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11491,"num_write_kb":417920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4645,"num_bytes_recovered":304414720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":["8","5","7"],"object_location_counts":[{"shards":"7,8","objects":1569}],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.6","version":"70'139","reported_seq":522,"reported_epoch":584,"state":"active+clean","last_fresh":"2024-08-19T14:32:33.941602+0000","last_change":"2024-08-19T14:28:59.173001+0000","last_active":"2024-08-19T14:32:33.941602+0000","last_peered":"2024-08-19T14:32:33.941602+0000","last_clean":"2024-08-19T14:32:33.941602+0000","last_became_active":"2024-08-19T14:28:08.847096+0000","last_became_peered":"2024-08-19T14:28:08.847096+0000","last_unstale":"2024-08-19T14:32:33.941602+0000","last_undegraded":"2024-08-19T14:32:33.941602+0000","last_fullsized":"2024-08-19T14:32:33.941602+0000","mapping_epoch":496,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":497,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:59.172931+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:27.848870+0000","last_clean_scrub_stamp":"2024-08-19T14:28:59.172931+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,3],"acting":[8,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"6.2","version":"583'11817","reported_seq":25971,"reported_epoch":583,"state":"active+clean+remapped","last_fresh":"2024-08-19T14:32:33.578863+0000","last_change":"2024-08-19T14:32:20.973031+0000","last_active":"2024-08-19T14:32:33.578863+0000","last_peered":"2024-08-19T14:32:33.578863+0000","last_clean":"2024-08-19T14:32:33.578863+0000","last_became_active":"2024-08-19T14:31:03.974413+0000","last_became_peered":"2024-08-19T14:31:03.974413+0000","last_unstale":"2024-08-19T14:32:33.578863+0000","last_undegraded":"2024-08-19T14:32:33.578863+0000","last_fullsized":"2024-08-19T14:32:33.578863+0000","mapping_epoch":566,"log_start":"514'1800","ondisk_log_start":"514'1800","created":510,"last_epoch_clean":574,"parent":"0.0","parent_split_bits":0,"last_scrub":"581'11243","last_scrub_stamp":"2024-08-19T14:32:20.972201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:28:32.139424+0000","last_clean_scrub_stamp":"2024-08-19T14:32:20.972201+0000","objects_scrubbed":2299,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:33:30.872941+0000","scrub_duration":866,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":111869952,"num_objects":1707,"num_object_clones":0,"num_object_copies":5121,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1707,"num_objects_unfound":0,"num_objects_dirty":1707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11817,"num_write_kb":432768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6977,"num_bytes_recovered":354680832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3,8],"avail_no_missing":["4","3","8"],"object_location_counts":[{"shards":"3,4,8","objects":1707}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"70'139","reported_seq":502,"reported_epoch":584,"state":"active+undersized","last_fresh":"2024-08-19T14:32:33.973893+0000","last_change":"2024-08-19T14:31:02.876003+0000","last_active":"2024-08-19T14:32:33.973893+0000","last_peered":"2024-08-19T14:32:33.973893+0000","last_clean":"2024-08-19T14:30:57.017098+0000","last_became_active":"2024-08-19T14:31:02.876003+0000","last_became_peered":"2024-08-19T14:31:02.876003+0000","last_unstale":"2024-08-19T14:32:33.973893+0000","last_undegraded":"2024-08-19T14:32:33.973893+0000","last_fullsized":"2024-08-19T14:31:02.871830+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:27.331041+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:28:27.331041+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":["11","0"],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.3","version":"584'11845","reported_seq":18901,"reported_epoch":584,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-19T14:32:35.108162+0000","last_change":"2024-08-19T14:30:46.222600+0000","last_active":"2024-08-19T14:32:35.108162+0000","last_peered":"2024-08-19T14:32:35.108162+0000","last_clean":"2024-08-19T14:30:43.878513+0000","last_became_active":"2024-08-19T14:30:46.149753+0000","last_became_peered":"2024-08-19T14:30:46.149753+0000","last_unstale":"2024-08-19T14:32:35.108162+0000","last_undegraded":"2024-08-19T14:30:45.947690+0000","last_fullsized":"2024-08-19T14:30:45.904871+0000","mapping_epoch":556,"log_start":"514'1800","ondisk_log_start":"514'1800","created":510,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:28:32.139424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:28:32.139424+0000","last_clean_scrub_stamp":"2024-08-19T14:28:32.139424+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106102784,"num_objects":1619,"num_object_clones":0,"num_object_copies":4857,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":622,"num_objects_misplaced":1575,"num_objects_unfound":0,"num_objects_dirty":1619,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11845,"num_write_kb":430848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3447,"num_bytes_recovered":225247232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,8],"avail_no_missing":["7","0","8"],"object_location_counts":[{"shards":"0,7,8","objects":1575},{"shards":"7,8","objects":-578}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"70'139","reported_seq":377,"reported_epoch":584,"state":"active+clean","last_fresh":"2024-08-19T14:32:33.973863+0000","last_change":"2024-08-19T14:30:47.909434+0000","last_active":"2024-08-19T14:32:33.973863+0000","last_peered":"2024-08-19T14:32:33.973863+0000","last_clean":"2024-08-19T14:32:33.973863+0000","last_became_active":"2024-08-19T14:30:47.909067+0000","last_became_peered":"2024-08-19T14:30:47.909067+0000","last_unstale":"2024-08-19T14:32:33.973863+0000","last_undegraded":"2024-08-19T14:32:33.973863+0000","last_fullsized":"2024-08-19T14:32:33.973863+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:28.122172+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:28:28.122172+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:32:38.532122+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17890,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":5,"num_write_kb":89,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":64112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.d","version":"584'11770","reported_seq":11930,"reported_epoch":584,"state":"active+clean","last_fresh":"2024-08-19T14:32:33.939162+0000","last_change":"2024-08-19T14:28:33.151442+0000","last_active":"2024-08-19T14:32:33.939162+0000","last_peered":"2024-08-19T14:32:33.939162+0000","last_clean":"2024-08-19T14:32:33.939162+0000","last_became_active":"2024-08-19T14:28:33.151135+0000","last_became_peered":"2024-08-19T14:28:33.151135+0000","last_unstale":"2024-08-19T14:32:33.939162+0000","last_undegraded":"2024-08-19T14:32:33.939162+0000","last_fullsized":"2024-08-19T14:32:33.939162+0000","mapping_epoch":510,"log_start":"514'1700","ondisk_log_start":"514'1700","created":510,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:28:32.139424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:28:32.139424+0000","last_clean_scrub_stamp":"2024-08-19T14:28:32.139424+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106889240,"num_objects":1632,"num_object_clones":0,"num_object_copies":4896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1632,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11770,"num_write_kb":428801,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"70'139","reported_seq":532,"reported_epoch":583,"state":"active+undersized","last_fresh":"2024-08-19T14:32:27.296037+0000","last_change":"2024-08-19T14:31:02.878486+0000","last_active":"2024-08-19T14:32:27.296037+0000","last_peered":"2024-08-19T14:32:27.296037+0000","last_clean":"2024-08-19T14:30:57.026214+0000","last_became_active":"2024-08-19T14:31:02.878486+0000","last_became_peered":"2024-08-19T14:31:02.878486+0000","last_unstale":"2024-08-19T14:32:27.296037+0000","last_undegraded":"2024-08-19T14:32:27.296037+0000","last_fullsized":"2024-08-19T14:31:02.874265+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:50.681699+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:28.864870+0000","last_clean_scrub_stamp":"2024-08-19T14:28:50.681699+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:32:31.104680+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"583'11990","reported_seq":27801,"reported_epoch":583,"state":"active+clean+remapped","last_fresh":"2024-08-19T14:32:33.578814+0000","last_change":"2024-08-19T14:32:27.069535+0000","last_active":"2024-08-19T14:32:33.578814+0000","last_peered":"2024-08-19T14:32:33.578814+0000","last_clean":"2024-08-19T14:32:33.578814+0000","last_became_active":"2024-08-19T14:31:03.982740+0000","last_became_peered":"2024-08-19T14:31:03.982740+0000","last_unstale":"2024-08-19T14:32:33.578814+0000","last_undegraded":"2024-08-19T14:32:33.578814+0000","last_fullsized":"2024-08-19T14:32:33.578814+0000","mapping_epoch":566,"log_start":"515'1900","ondisk_log_start":"515'1900","created":510,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":0,"last_scrub":"582'11709","last_scrub_stamp":"2024-08-19T14:32:27.069170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:28:32.139424+0000","last_clean_scrub_stamp":"2024-08-19T14:32:27.069170+0000","objects_scrubbed":1942,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:33:43.680276+0000","scrub_duration":962,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106692608,"num_objects":1628,"num_object_clones":0,"num_object_copies":4884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1628,"num_objects_unfound":0,"num_objects_dirty":1628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11990,"num_write_kb":435776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7807,"num_bytes_recovered":410976256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1,5],"avail_no_missing":["4","1","5"],"object_location_counts":[{"shards":"1,4,5","objects":1628}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"70'139","reported_seq":1465,"reported_epoch":584,"state":"active+undersized","last_fresh":"2024-08-19T14:32:33.945522+0000","last_change":"2024-08-19T14:31:02.877070+0000","last_active":"2024-08-19T14:32:33.945522+0000","last_peered":"2024-08-19T14:32:33.945522+0000","last_clean":"2024-08-19T14:30:57.018732+0000","last_became_active":"2024-08-19T14:31:02.877070+0000","last_became_peered":"2024-08-19T14:31:02.877070+0000","last_unstale":"2024-08-19T14:32:33.945522+0000","last_undegraded":"2024-08-19T14:32:33.945522+0000","last_fullsized":"2024-08-19T14:31:02.872568+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:53.273092+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:28:53.273092+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.f","version":"583'11553","reported_seq":21151,"reported_epoch":583,"state":"active+undersized+degraded","last_fresh":"2024-08-19T14:32:33.609650+0000","last_change":"2024-08-19T14:31:02.876732+0000","last_active":"2024-08-19T14:32:33.609650+0000","last_peered":"2024-08-19T14:32:33.609650+0000","last_clean":"2024-08-19T14:31:02.016969+0000","last_became_active":"2024-08-19T14:31:02.876732+0000","last_became_peered":"2024-08-19T14:31:02.876732+0000","last_unstale":"2024-08-19T14:32:33.609650+0000","last_undegraded":"2024-08-19T14:31:02.875124+0000","last_fullsized":"2024-08-19T14:31:02.874904+0000","mapping_epoch":565,"log_start":"514'1500","ondisk_log_start":"514'1500","created":510,"last_epoch_clean":563,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:28:32.139424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:28:32.139424+0000","last_clean_scrub_stamp":"2024-08-19T14:28:32.139424+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:32:36.192278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":105709568,"num_objects":1613,"num_object_clones":0,"num_object_copies":4839,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1613,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11553,"num_write_kb":421312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4714,"num_bytes_recovered":308936704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[{"shards":"4,5","objects":1613}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"70'139","reported_seq":511,"reported_epoch":584,"state":"active+undersized","last_fresh":"2024-08-19T14:32:33.940083+0000","last_change":"2024-08-19T14:31:02.882810+0000","last_active":"2024-08-19T14:32:33.940083+0000","last_peered":"2024-08-19T14:32:33.940083+0000","last_clean":"2024-08-19T14:30:57.035787+0000","last_became_active":"2024-08-19T14:31:02.882810+0000","last_became_peered":"2024-08-19T14:31:02.882810+0000","last_unstale":"2024-08-19T14:32:33.940083+0000","last_undegraded":"2024-08-19T14:32:33.940083+0000","last_fullsized":"2024-08-19T14:31:02.877995+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:59.829567+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:28:59.829567+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"584'11943","reported_seq":12102,"reported_epoch":584,"state":"active+clean","last_fresh":"2024-08-19T14:32:35.014197+0000","last_change":"2024-08-19T14:28:33.150427+0000","last_active":"2024-08-19T14:32:35.014197+0000","last_peered":"2024-08-19T14:32:35.014197+0000","last_clean":"2024-08-19T14:32:35.014197+0000","last_became_active":"2024-08-19T14:28:33.150105+0000","last_became_peered":"2024-08-19T14:28:33.150105+0000","last_unstale":"2024-08-19T14:32:35.014197+0000","last_undegraded":"2024-08-19T14:32:35.014197+0000","last_fullsized":"2024-08-19T14:32:35.014197+0000","mapping_epoch":510,"log_start":"514'1900","ondisk_log_start":"514'1900","created":510,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:28:32.139424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:28:32.139424+0000","last_clean_scrub_stamp":"2024-08-19T14:28:32.139424+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103481344,"num_objects":1579,"num_object_clones":0,"num_object_copies":4737,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11943,"num_write_kb":432704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,8],"acting":[3,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"70'139","reported_seq":829,"reported_epoch":584,"state":"active+clean","last_fresh":"2024-08-19T14:32:33.977067+0000","last_change":"2024-08-19T14:30:47.941523+0000","last_active":"2024-08-19T14:32:33.977067+0000","last_peered":"2024-08-19T14:32:33.977067+0000","last_clean":"2024-08-19T14:32:33.977067+0000","last_became_active":"2024-08-19T14:30:47.941200+0000","last_became_peered":"2024-08-19T14:30:47.941200+0000","last_unstale":"2024-08-19T14:32:33.977067+0000","last_undegraded":"2024-08-19T14:32:33.977067+0000","last_fullsized":"2024-08-19T14:32:33.977067+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:26.371992+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:28:26.371992+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.9","version":"583'11750","reported_seq":23595,"reported_epoch":583,"state":"active+clean+remapped","last_fresh":"2024-08-19T14:32:33.823336+0000","last_change":"2024-08-19T14:32:24.559081+0000","last_active":"2024-08-19T14:32:33.823336+0000","last_peered":"2024-08-19T14:32:33.823336+0000","last_clean":"2024-08-19T14:32:33.823336+0000","last_became_active":"2024-08-19T14:31:04.005765+0000","last_became_peered":"2024-08-19T14:31:04.005765+0000","last_unstale":"2024-08-19T14:32:33.823336+0000","last_undegraded":"2024-08-19T14:32:33.823336+0000","last_fullsized":"2024-08-19T14:32:33.823336+0000","mapping_epoch":566,"log_start":"514'1700","ondisk_log_start":"514'1700","created":510,"last_epoch_clean":576,"parent":"0.0","parent_split_bits":0,"last_scrub":"582'11388","last_scrub_stamp":"2024-08-19T14:32:24.558810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:28:32.139424+0000","last_clean_scrub_stamp":"2024-08-19T14:32:24.558810+0000","objects_scrubbed":2031,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:33:37.409089+0000","scrub_duration":554,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108396544,"num_objects":1654,"num_object_clones":0,"num_object_copies":4962,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1654,"num_objects_unfound":0,"num_objects_dirty":1654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11750,"num_write_kb":428928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5824,"num_bytes_recovered":281673728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8,1],"avail_no_missing":["0","1","8"],"object_location_counts":[{"shards":"0,1,8","objects":1654}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"70'139","reported_seq":1010,"reported_epoch":583,"state":"active+clean","last_fresh":"2024-08-19T14:32:31.196634+0000","last_change":"2024-08-19T14:32:31.196784+0000","last_active":"2024-08-19T14:32:31.196634+0000","last_peered":"2024-08-19T14:32:31.196634+0000","last_clean":"2024-08-19T14:32:31.196634+0000","last_became_active":"2024-08-19T14:18:26.884105+0000","last_became_peered":"2024-08-19T14:18:26.884105+0000","last_unstale":"2024-08-19T14:32:31.196634+0000","last_undegraded":"2024-08-19T14:32:31.196634+0000","last_fullsized":"2024-08-19T14:32:31.196634+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:32:31.196288+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:32:31.196288+0000","last_clean_scrub_stamp":"2024-08-19T14:32:31.196288+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:33:33.184265+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"584'11956","reported_seq":19270,"reported_epoch":584,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-19T14:32:35.082245+0000","last_change":"2024-08-19T14:30:46.177059+0000","last_active":"2024-08-19T14:32:35.082245+0000","last_peered":"2024-08-19T14:32:35.082245+0000","last_clean":"2024-08-19T14:30:43.872795+0000","last_became_active":"2024-08-19T14:30:46.119312+0000","last_became_peered":"2024-08-19T14:30:46.119312+0000","last_unstale":"2024-08-19T14:32:35.082245+0000","last_undegraded":"2024-08-19T14:30:45.951116+0000","last_fullsized":"2024-08-19T14:30:45.905392+0000","mapping_epoch":556,"log_start":"514'1900","ondisk_log_start":"514'1900","created":510,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:28:32.139424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:28:32.139424+0000","last_clean_scrub_stamp":"2024-08-19T14:28:32.139424+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":105775104,"num_objects":1614,"num_object_clones":0,"num_object_copies":4842,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":578,"num_objects_misplaced":1541,"num_objects_unfound":0,"num_objects_dirty":1614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11956,"num_write_kb":434240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3576,"num_bytes_recovered":233832448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,3],"acting":[7,3],"avail_no_missing":["7","3","4"],"object_location_counts":[{"shards":"3,4,7","objects":1541},{"shards":"3,7","objects":-505}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"70'139","reported_seq":770,"reported_epoch":583,"state":"active+clean","last_fresh":"2024-08-19T14:32:28.096789+0000","last_change":"2024-08-19T14:32:28.096831+0000","last_active":"2024-08-19T14:32:28.096789+0000","last_peered":"2024-08-19T14:32:28.096789+0000","last_clean":"2024-08-19T14:32:28.096789+0000","last_became_active":"2024-08-19T14:27:31.837754+0000","last_became_peered":"2024-08-19T14:27:31.837754+0000","last_unstale":"2024-08-19T14:32:28.096789+0000","last_undegraded":"2024-08-19T14:32:28.096789+0000","last_fullsized":"2024-08-19T14:32:28.096789+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:32:28.096695+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:32:28.096695+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:33:49.162058+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"584'11914","reported_seq":12073,"reported_epoch":584,"state":"active+clean","last_fresh":"2024-08-19T14:32:35.044840+0000","last_change":"2024-08-19T14:28:33.151083+0000","last_active":"2024-08-19T14:32:35.044840+0000","last_peered":"2024-08-19T14:32:35.044840+0000","last_clean":"2024-08-19T14:32:35.044840+0000","last_became_active":"2024-08-19T14:28:33.150788+0000","last_became_peered":"2024-08-19T14:28:33.150788+0000","last_unstale":"2024-08-19T14:32:35.044840+0000","last_undegraded":"2024-08-19T14:32:35.044840+0000","last_fullsized":"2024-08-19T14:32:35.044840+0000","mapping_epoch":510,"log_start":"514'1900","ondisk_log_start":"514'1900","created":510,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:28:32.139424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:28:32.139424+0000","last_clean_scrub_stamp":"2024-08-19T14:28:32.139424+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":105512960,"num_objects":1610,"num_object_clones":0,"num_object_copies":4830,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1610,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11914,"num_write_kb":432768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"70'139","reported_seq":1114,"reported_epoch":584,"state":"active+clean","last_fresh":"2024-08-19T14:32:33.940320+0000","last_change":"2024-08-19T14:28:26.508582+0000","last_active":"2024-08-19T14:32:33.940320+0000","last_peered":"2024-08-19T14:32:33.940320+0000","last_clean":"2024-08-19T14:32:33.940320+0000","last_became_active":"2024-08-19T14:28:08.849293+0000","last_became_peered":"2024-08-19T14:28:08.849293+0000","last_unstale":"2024-08-19T14:32:33.940320+0000","last_undegraded":"2024-08-19T14:32:33.940320+0000","last_fullsized":"2024-08-19T14:32:33.940320+0000","mapping_epoch":496,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":497,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:26.508506+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:22:52.164586+0000","last_clean_scrub_stamp":"2024-08-19T14:28:26.508506+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"583'12508","reported_seq":24526,"reported_epoch":583,"state":"active+clean+remapped+scrubbing+deep","last_fresh":"2024-08-19T14:32:33.857121+0000","last_change":"2024-08-19T14:32:27.028458+0000","last_active":"2024-08-19T14:32:33.857121+0000","last_peered":"2024-08-19T14:32:33.857121+0000","last_clean":"2024-08-19T14:32:33.857121+0000","last_became_active":"2024-08-19T14:31:04.010296+0000","last_became_peered":"2024-08-19T14:31:04.010296+0000","last_unstale":"2024-08-19T14:32:33.857121+0000","last_undegraded":"2024-08-19T14:32:33.857121+0000","last_fullsized":"2024-08-19T14:32:33.857121+0000","mapping_epoch":566,"log_start":"516'2500","ondisk_log_start":"516'2500","created":510,"last_epoch_clean":576,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:28:32.139424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:28:32.139424+0000","last_clean_scrub_stamp":"2024-08-19T14:28:32.139424+0000","objects_scrubbed":736,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 6s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103677952,"num_objects":1582,"num_object_clones":0,"num_object_copies":4746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1582,"num_objects_unfound":0,"num_objects_dirty":1582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11772,"num_write_kb":427328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5908,"num_bytes_recovered":284360704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5,1],"avail_no_missing":["0","1","5"],"object_location_counts":[{"shards":"0,1,5","objects":1582}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"70'139","reported_seq":1016,"reported_epoch":584,"state":"active+clean","last_fresh":"2024-08-19T14:32:33.945875+0000","last_change":"2024-08-19T14:28:29.007026+0000","last_active":"2024-08-19T14:32:33.945875+0000","last_peered":"2024-08-19T14:32:33.945875+0000","last_clean":"2024-08-19T14:32:33.945875+0000","last_became_active":"2024-08-19T14:18:26.882002+0000","last_became_peered":"2024-08-19T14:18:26.882002+0000","last_unstale":"2024-08-19T14:32:33.945875+0000","last_undegraded":"2024-08-19T14:32:33.945875+0000","last_fullsized":"2024-08-19T14:32:33.945875+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:29.006921+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:24:50.276408+0000","last_clean_scrub_stamp":"2024-08-19T14:28:29.006921+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"70'139","reported_seq":803,"reported_epoch":584,"state":"active+clean","last_fresh":"2024-08-19T14:32:33.940097+0000","last_change":"2024-08-19T14:28:35.206191+0000","last_active":"2024-08-19T14:32:33.940097+0000","last_peered":"2024-08-19T14:32:33.940097+0000","last_clean":"2024-08-19T14:32:33.940097+0000","last_became_active":"2024-08-19T14:27:31.838968+0000","last_became_peered":"2024-08-19T14:27:31.838968+0000","last_unstale":"2024-08-19T14:32:33.940097+0000","last_undegraded":"2024-08-19T14:32:33.940097+0000","last_fullsized":"2024-08-19T14:32:33.940097+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:35.206115+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:28:35.206115+0000","last_clean_scrub_stamp":"2024-08-19T14:28:35.206115+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.13","version":"70'139","reported_seq":1345,"reported_epoch":584,"state":"active+clean","last_fresh":"2024-08-19T14:32:33.939644+0000","last_change":"2024-08-19T14:30:47.940339+0000","last_active":"2024-08-19T14:32:33.939644+0000","last_peered":"2024-08-19T14:32:33.939644+0000","last_clean":"2024-08-19T14:32:33.939644+0000","last_became_active":"2024-08-19T14:30:47.939849+0000","last_became_peered":"2024-08-19T14:30:47.939849+0000","last_unstale":"2024-08-19T14:32:33.939644+0000","last_undegraded":"2024-08-19T14:32:33.939644+0000","last_fullsized":"2024-08-19T14:32:33.939644+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:53.095364+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:28:53.095364+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.10","version":"70'139","reported_seq":965,"reported_epoch":584,"state":"active+clean","last_fresh":"2024-08-19T14:32:33.946977+0000","last_change":"2024-08-19T14:29:00.244077+0000","last_active":"2024-08-19T14:32:33.946977+0000","last_peered":"2024-08-19T14:32:33.946977+0000","last_clean":"2024-08-19T14:32:33.946977+0000","last_became_active":"2024-08-19T14:27:31.839611+0000","last_became_peered":"2024-08-19T14:27:31.839611+0000","last_unstale":"2024-08-19T14:32:33.946977+0000","last_undegraded":"2024-08-19T14:32:33.946977+0000","last_fullsized":"2024-08-19T14:32:33.946977+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:29:00.244002+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:29:00.244002+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.11","version":"70'139","reported_seq":1002,"reported_epoch":583,"state":"active+clean","last_fresh":"2024-08-19T14:32:30.127772+0000","last_change":"2024-08-19T14:32:30.127838+0000","last_active":"2024-08-19T14:32:30.127772+0000","last_peered":"2024-08-19T14:32:30.127772+0000","last_clean":"2024-08-19T14:32:30.127772+0000","last_became_active":"2024-08-19T14:28:08.848199+0000","last_became_peered":"2024-08-19T14:28:08.848199+0000","last_unstale":"2024-08-19T14:32:30.127772+0000","last_undegraded":"2024-08-19T14:32:30.127772+0000","last_fullsized":"2024-08-19T14:32:30.127772+0000","mapping_epoch":496,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":497,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:32:30.127626+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:24:50.547428+0000","last_clean_scrub_stamp":"2024-08-19T14:32:30.127626+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:33:43.631680+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,3],"acting":[0,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"70'139","reported_seq":955,"reported_epoch":584,"state":"active+clean","last_fresh":"2024-08-19T14:32:33.942711+0000","last_change":"2024-08-19T14:28:34.216440+0000","last_active":"2024-08-19T14:32:33.942711+0000","last_peered":"2024-08-19T14:32:33.942711+0000","last_clean":"2024-08-19T14:32:33.942711+0000","last_became_active":"2024-08-19T14:20:28.204724+0000","last_became_peered":"2024-08-19T14:20:28.204724+0000","last_unstale":"2024-08-19T14:32:33.942711+0000","last_undegraded":"2024-08-19T14:32:33.942711+0000","last_fullsized":"2024-08-19T14:32:33.942711+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:34.216370+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:28:34.216370+0000","last_clean_scrub_stamp":"2024-08-19T14:28:34.216370+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"70'139","reported_seq":1083,"reported_epoch":584,"state":"active+clean","last_fresh":"2024-08-19T14:32:33.941531+0000","last_change":"2024-08-19T14:28:30.136536+0000","last_active":"2024-08-19T14:32:33.941531+0000","last_peered":"2024-08-19T14:32:33.941531+0000","last_clean":"2024-08-19T14:32:33.941531+0000","last_became_active":"2024-08-19T14:18:31.901556+0000","last_became_peered":"2024-08-19T14:18:31.901556+0000","last_unstale":"2024-08-19T14:32:33.941531+0000","last_undegraded":"2024-08-19T14:32:33.941531+0000","last_fullsized":"2024-08-19T14:32:33.941531+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:30.136404+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:28:30.136404+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,0],"acting":[8,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.14","version":"70'139","reported_seq":1114,"reported_epoch":584,"state":"active+undersized","last_fresh":"2024-08-19T14:32:33.944853+0000","last_change":"2024-08-19T14:31:02.878909+0000","last_active":"2024-08-19T14:32:33.944853+0000","last_peered":"2024-08-19T14:32:33.944853+0000","last_clean":"2024-08-19T14:30:57.018886+0000","last_became_active":"2024-08-19T14:31:02.878909+0000","last_became_peered":"2024-08-19T14:31:02.878909+0000","last_unstale":"2024-08-19T14:32:33.944853+0000","last_undegraded":"2024-08-19T14:32:33.944853+0000","last_fullsized":"2024-08-19T14:31:02.872535+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:58.211358+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:28:58.211358+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.15","version":"70'139","reported_seq":971,"reported_epoch":584,"state":"active+clean","last_fresh":"2024-08-19T14:32:33.945763+0000","last_change":"2024-08-19T14:28:31.999297+0000","last_active":"2024-08-19T14:32:33.945763+0000","last_peered":"2024-08-19T14:32:33.945763+0000","last_clean":"2024-08-19T14:32:33.945763+0000","last_became_active":"2024-08-19T14:18:31.912496+0000","last_became_peered":"2024-08-19T14:18:31.912496+0000","last_unstale":"2024-08-19T14:32:33.945763+0000","last_undegraded":"2024-08-19T14:32:33.945763+0000","last_fullsized":"2024-08-19T14:32:33.945763+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:31.999202+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:20:09.180923+0000","last_clean_scrub_stamp":"2024-08-19T14:28:31.999202+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"70'139","reported_seq":963,"reported_epoch":583,"state":"active+clean","last_fresh":"2024-08-19T14:32:27.295894+0000","last_change":"2024-08-19T14:32:18.138447+0000","last_active":"2024-08-19T14:32:27.295894+0000","last_peered":"2024-08-19T14:32:27.295894+0000","last_clean":"2024-08-19T14:32:27.295894+0000","last_became_active":"2024-08-19T14:18:31.901229+0000","last_became_peered":"2024-08-19T14:18:31.901229+0000","last_unstale":"2024-08-19T14:32:27.295894+0000","last_undegraded":"2024-08-19T14:32:27.295894+0000","last_fullsized":"2024-08-19T14:32:27.295894+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:32:18.138345+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:32:18.138345+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:33:33.570724+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1b","version":"70'139","reported_seq":1330,"reported_epoch":584,"state":"active+undersized","last_fresh":"2024-08-19T14:32:33.941574+0000","last_change":"2024-08-19T14:31:02.875562+0000","last_active":"2024-08-19T14:32:33.941574+0000","last_peered":"2024-08-19T14:32:33.941574+0000","last_clean":"2024-08-19T14:30:57.063847+0000","last_became_active":"2024-08-19T14:31:02.875562+0000","last_became_peered":"2024-08-19T14:31:02.875562+0000","last_unstale":"2024-08-19T14:32:33.941574+0000","last_undegraded":"2024-08-19T14:32:33.941574+0000","last_fullsized":"2024-08-19T14:31:02.871236+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:24.065359+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:28:24.065359+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":["8","0"],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.18","version":"70'139","reported_seq":910,"reported_epoch":584,"state":"active+undersized+degraded","last_fresh":"2024-08-19T14:32:33.946789+0000","last_change":"2024-08-19T14:31:02.878454+0000","last_active":"2024-08-19T14:32:33.946789+0000","last_peered":"2024-08-19T14:32:33.946789+0000","last_clean":"2024-08-19T14:30:57.018623+0000","last_became_active":"2024-08-19T14:31:02.878454+0000","last_became_peered":"2024-08-19T14:31:02.878454+0000","last_unstale":"2024-08-19T14:32:33.946789+0000","last_undegraded":"2024-08-19T14:31:02.872398+0000","last_fullsized":"2024-08-19T14:31:02.872290+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:29:05.336156+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:27:33.084161+0000","last_clean_scrub_stamp":"2024-08-19T14:29:05.336156+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[{"shards":"5,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.19","version":"70'139","reported_seq":1307,"reported_epoch":583,"state":"active+clean","last_fresh":"2024-08-19T14:32:27.300594+0000","last_change":"2024-08-19T14:31:02.849476+0000","last_active":"2024-08-19T14:32:27.300594+0000","last_peered":"2024-08-19T14:32:27.300594+0000","last_clean":"2024-08-19T14:32:27.300594+0000","last_became_active":"2024-08-19T14:18:31.902547+0000","last_became_peered":"2024-08-19T14:18:31.902547+0000","last_unstale":"2024-08-19T14:32:27.300594+0000","last_undegraded":"2024-08-19T14:32:27.300594+0000","last_fullsized":"2024-08-19T14:32:27.300594+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:31:02.849371+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:31:02.849371+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1e","version":"70'139","reported_seq":349,"reported_epoch":584,"state":"active+clean","last_fresh":"2024-08-19T14:32:33.973896+0000","last_change":"2024-08-19T14:30:47.910267+0000","last_active":"2024-08-19T14:32:33.973896+0000","last_peered":"2024-08-19T14:32:33.973896+0000","last_clean":"2024-08-19T14:32:33.973896+0000","last_became_active":"2024-08-19T14:30:47.909974+0000","last_became_peered":"2024-08-19T14:30:47.909974+0000","last_unstale":"2024-08-19T14:32:33.973896+0000","last_undegraded":"2024-08-19T14:32:33.973896+0000","last_fullsized":"2024-08-19T14:32:33.973896+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:27.136174+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:27:04.780939+0000","last_clean_scrub_stamp":"2024-08-19T14:28:27.136174+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:32:38.532145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13418,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":48085,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1c","version":"70'139","reported_seq":1085,"reported_epoch":584,"state":"active+clean","last_fresh":"2024-08-19T14:32:33.941446+0000","last_change":"2024-08-19T14:30:47.909239+0000","last_active":"2024-08-19T14:32:33.941446+0000","last_peered":"2024-08-19T14:32:33.941446+0000","last_clean":"2024-08-19T14:32:33.941446+0000","last_became_active":"2024-08-19T14:30:47.908866+0000","last_became_peered":"2024-08-19T14:30:47.908866+0000","last_unstale":"2024-08-19T14:32:33.941446+0000","last_undegraded":"2024-08-19T14:32:33.941446+0000","last_fullsized":"2024-08-19T14:32:33.941446+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:25.098173+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:28:25.098173+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,11],"acting":[8,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1d","version":"70'139","reported_seq":1011,"reported_epoch":584,"state":"active+undersized","last_fresh":"2024-08-19T14:32:33.938642+0000","last_change":"2024-08-19T14:31:02.878718+0000","last_active":"2024-08-19T14:32:33.938642+0000","last_peered":"2024-08-19T14:32:33.938642+0000","last_clean":"2024-08-19T14:30:57.053277+0000","last_became_active":"2024-08-19T14:31:02.878718+0000","last_became_peered":"2024-08-19T14:31:02.878718+0000","last_unstale":"2024-08-19T14:32:33.938642+0000","last_undegraded":"2024-08-19T14:32:33.938642+0000","last_fullsized":"2024-08-19T14:31:02.875751+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:29.194639+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:28:29.194639+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":["1","11"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":1701183512,"num_objects":25959,"num_object_clones":0,"num_object_copies":77877,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8475,"num_objects_misplaced":17062,"num_objects_unfound":0,"num_objects_dirty":25959,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":188583,"num_write_kb":6865281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56037,"num_bytes_recovered":3160539136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":7959425024,"data_stored":7959412808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160819,"ondisk_log_size":160819,"up":40,"acting":42,"num_store_stats":9},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":621676,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":9,"num_write_kb":156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":641125,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1925120,"data_stored":1914464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":83,"acting":83,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187711,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":686884,"kb_used_data":15708,"kb_used_omap":1,"kb_used_meta":671166,"kb_avail":93058268,"statfs":{"total":95995035648,"available":95291666432,"internally_reserved":0,"allocated":16084992,"data_stored":13468612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":687274436},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":2314987372578,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":0,"seq":317827580168,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874511,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1202472,"kb_used_data":517404,"kb_used_omap":1,"kb_used_meta":685054,"kb_avail":92542680,"statfs":{"total":95995035648,"available":94763704320,"internally_reserved":0,"allocated":529821696,"data_stored":527205316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":701495749},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":57,"seq":244813136146,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1484656,"kb_used_data":632996,"kb_used_omap":1,"kb_used_meta":851646,"kb_avail":92260496,"statfs":{"total":95995035648,"available":94474747904,"internally_reserved":0,"allocated":648187904,"data_stored":645567972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":872085962},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,0,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":206158430485,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692121,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1945072,"kb_used_data":1147560,"kb_used_omap":1,"kb_used_meta":797502,"kb_avail":91800080,"statfs":{"total":95995035648,"available":94003281920,"internally_reserved":0,"allocated":1175101440,"data_stored":1172477436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":816642501},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986461,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2041752,"kb_used_data":1194256,"kb_used_omap":1,"kb_used_meta":847486,"kb_avail":91703400,"statfs":{"total":95995035648,"available":93904281600,"internally_reserved":0,"allocated":1222918144,"data_stored":1220305734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":867826117},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149987,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1668776,"kb_used_data":839324,"kb_used_omap":1,"kb_used_meta":829438,"kb_avail":92076376,"statfs":{"total":95995035648,"available":94286209024,"internally_reserved":0,"allocated":859467776,"data_stored":856851396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":849344967},"hb_peers":[0,1,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,0,4,4,0,0,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411606,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2413884,"kb_used_data":2103476,"kb_used_omap":1,"kb_used_meta":310206,"kb_avail":91331268,"statfs":{"total":95995035648,"available":93523218432,"internally_reserved":0,"allocated":2153959424,"data_stored":2151417784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":317651397},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575148,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1549084,"kb_used_data":637716,"kb_used_omap":1,"kb_used_meta":911358,"kb_avail":92196068,"statfs":{"total":95995035648,"available":94408773632,"internally_reserved":0,"allocated":653021184,"data_stored":650404702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":933231045},"hb_peers":[0,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738672,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1510428,"kb_used_data":789204,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":92234724,"statfs":{"total":95995035648,"available":94448357376,"internally_reserved":0,"allocated":808144896,"data_stored":805528414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1,1,6,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":792989696,"data_stored":792985624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":641667072,"data_stored":641663000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2144075776,"data_stored":2144075776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":844365824,"data_stored":844365824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1210974208,"data_stored":1210974208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1158418432,"data_stored":1158414360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":635764736,"data_stored":635764736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":522518528,"data_stored":522518528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8650752,"data_stored":8650752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T14:32:40.850 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-08-19T14:32:40.850 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd pool set .mgr pg_num 81 2024-08-19T14:32:40.858 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:40.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:40.859 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:32:40.930 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:32:40.954 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is set 2024-08-19T14:32:40.959 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:32:40.986 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:40.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:40.987 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:32:41.077 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:40 smithi031 ceph-mon[16336]: from='client.26483 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:32:41.078 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:40 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/4191726991' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:32:41.078 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:40 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:32:41.088 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:32:41.110 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:41.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:41.112 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:32:41.213 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:32:41.236 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:41.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:41.237 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:32:41.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:40 smithi032 ceph-mon[20004]: from='client.26483 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:32:41.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:40 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/4191726991' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:32:41.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:40 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:32:41.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:40 smithi045 ceph-mon[20143]: from='client.26483 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:32:41.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:40 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/4191726991' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:32:41.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:40 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:32:41.338 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:32:41.362 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:41.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:41.363 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:32:41.464 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:32:41.492 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:41.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:41.493 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:32:41.593 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:32:41.628 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:41.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:41.629 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:32:41.730 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:32:41.752 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:41.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:41.753 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:32:41.854 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:32:41.885 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:41.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:41.887 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:32:41.987 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:32:42.018 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:42.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:42.019 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:32:42.120 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:32:42.158 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:42.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:42.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:32:42.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:41 smithi032 ceph-mon[20004]: pgmap v1367: 47 pgs: 1 active+clean+remapped+scrubbing+deep, 4 active+clean+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 1.4 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 561 op/s; 8389/71046 objects degraded (11.808%); 15818/71046 objects misplaced (22.264%); 4.3 MiB/s, 68 objects/s recovering 2024-08-19T14:32:42.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:41 smithi032 ceph-mon[20004]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:32:42.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:41 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:32:42.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:41 smithi032 ceph-mon[20004]: osdmap e585: 12 total, 11 up, 9 in 2024-08-19T14:32:42.260 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:32:42.271 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:32:42.293 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:42.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:42.294 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:32:42.295 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:41 smithi031 ceph-mon[16336]: pgmap v1367: 47 pgs: 1 active+clean+remapped+scrubbing+deep, 4 active+clean+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 1.4 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 561 op/s; 8389/71046 objects degraded (11.808%); 15818/71046 objects misplaced (22.264%); 4.3 MiB/s, 68 objects/s recovering 2024-08-19T14:32:42.295 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:41 smithi031 ceph-mon[16336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:32:42.295 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:41 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:32:42.295 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:41 smithi031 ceph-mon[16336]: osdmap e585: 12 total, 11 up, 9 in 2024-08-19T14:32:42.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:41 smithi045 ceph-mon[20143]: pgmap v1367: 47 pgs: 1 active+clean+remapped+scrubbing+deep, 4 active+clean+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 1.4 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 561 op/s; 8389/71046 objects degraded (11.808%); 15818/71046 objects misplaced (22.264%); 4.3 MiB/s, 68 objects/s recovering 2024-08-19T14:32:42.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:41 smithi045 ceph-mon[20143]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:32:42.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:41 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:32:42.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:41 smithi045 ceph-mon[20143]: osdmap e585: 12 total, 11 up, 9 in 2024-08-19T14:32:42.297 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:32:42.297 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:32:42.331 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:32:42.331 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:32:42.331 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-08-19T14:32:42.362 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:32:42.362 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:32:42.390 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:32:42.390 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:32:42.395 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:32:42.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:42.434 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:42.434 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:32:42.444 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:32:42.445 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:32:42.476 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:32:42.477 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:32:42.503 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:32:42.504 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:32:42.532 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:32:42.533 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:32:42.535 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:32:42.557 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:42.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:42.558 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:32:42.566 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:32:42.566 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:32:42.573 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-08-19T14:32:42.601 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:32:42.602 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:32:42.639 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:32:42.639 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:32:42.658 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:32:42.698 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:42.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:42.699 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:32:42.800 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:32:42.824 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:42.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:42.825 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:32:42.926 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:32:42.947 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:42.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:42.948 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:32:43.049 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:32:43.070 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:32:43.072 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:43.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:43.073 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:32:43.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:42 smithi031 ceph-mon[16336]: from='client.26495 172.21.15.31:0/3303640133' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:32:43.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:42 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 8389/71046 objects degraded (11.808%), 7 pgs degraded, 16 pgs undersized (PG_DEGRADED) 2024-08-19T14:32:43.174 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:32:43.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:43.207 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:43.207 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:32:43.223 INFO:teuthology.orchestra.run.smithi031.stderr:Error EINVAL: specified pg_num 81 < pg_num_max 32 2024-08-19T14:32:43.246 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:42 smithi032 ceph-mon[20004]: from='client.26495 172.21.15.31:0/3303640133' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:32:43.246 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:42 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 8389/71046 objects degraded (11.808%), 7 pgs degraded, 16 pgs undersized (PG_DEGRADED) 2024-08-19T14:32:43.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:42 smithi045 ceph-mon[20143]: from='client.26495 172.21.15.31:0/3303640133' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:32:43.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:42 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 8389/71046 objects degraded (11.808%), 7 pgs degraded, 16 pgs undersized (PG_DEGRADED) 2024-08-19T14:32:43.307 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:32:43.329 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:43.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:43.330 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:32:43.431 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:32:43.464 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:43.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:43.465 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:32:43.566 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:32:43.587 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-19T14:32:43.587 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-19T14:32:43.589 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:43.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:43.590 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:32:43.684 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:32:43.684 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-08-19T14:32:43.684 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-08-19T14:32:43.684 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-08-19T14:32:43.684 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:32:43.690 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:32:43.715 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:43.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:43.715 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:32:43.755 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-19T14:32:43.816 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:32:43.839 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:43.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:43.840 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:32:43.941 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:32:43.974 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:43.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:43.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:32:44.076 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:32:44.112 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:44.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:44.113 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:32:44.214 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:32:44.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:44.250 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:44.250 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:32:44.252 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:43 smithi032 ceph-mon[20004]: pgmap v1369: 47 pgs: 1 active+clean+remapped+scrubbing+deep, 4 active+clean+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 1.4 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 606 op/s; 8009/68811 objects degraded (11.639%); 15320/68811 objects misplaced (22.264%); 9.0 MiB/s, 144 objects/s recovering 2024-08-19T14:32:44.253 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:43 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/188249036' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "81"}]: dispatch 2024-08-19T14:32:44.253 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:43 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "81"}]: dispatch 2024-08-19T14:32:44.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:43 smithi045 ceph-mon[20143]: pgmap v1369: 47 pgs: 1 active+clean+remapped+scrubbing+deep, 4 active+clean+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 1.4 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 606 op/s; 8009/68811 objects degraded (11.639%); 15320/68811 objects misplaced (22.264%); 9.0 MiB/s, 144 objects/s recovering 2024-08-19T14:32:44.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:43 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/188249036' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "81"}]: dispatch 2024-08-19T14:32:44.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:43 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "81"}]: dispatch 2024-08-19T14:32:44.326 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:43 smithi031 ceph-mon[16336]: pgmap v1369: 47 pgs: 1 active+clean+remapped+scrubbing+deep, 4 active+clean+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 1.4 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 606 op/s; 8009/68811 objects degraded (11.639%); 15320/68811 objects misplaced (22.264%); 9.0 MiB/s, 144 objects/s recovering 2024-08-19T14:32:44.326 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:43 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/188249036' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "81"}]: dispatch 2024-08-19T14:32:44.326 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:43 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "81"}]: dispatch 2024-08-19T14:32:44.351 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:32:44.375 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-19T14:32:44.395 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:44.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:44.396 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:32:44.497 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:32:44.508 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:32:44.522 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:44.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:44.523 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:32:44.556 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-08-19T14:32:44.624 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:32:44.646 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:44.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:44.647 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:32:44.748 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:32:44.771 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:44.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:44.772 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:32:44.873 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:32:44.919 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:44.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:44.921 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:32:45.020 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:32:45.022 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:32:45.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:45.059 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:45.059 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:32:45.160 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:32:45.183 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:45.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:45.184 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:32:45.285 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:32:45.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:45.319 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:45.319 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:32:45.420 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:32:45.452 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:45.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:45.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:32:45.554 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:32:45.575 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:45.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:45.576 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:32:45.588 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:32:45.614 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:32:45.614 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-08-19T14:32:45.614 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-08-19T14:32:45.614 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [] 2024-08-19T14:32:45.614 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:32:45.677 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:32:45.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:45.707 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:45.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:32:45.808 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:32:45.847 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:45.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:45.848 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:32:45.949 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:32:45.971 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:45.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:45.972 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:32:46.009 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is unset 2024-08-19T14:32:46.074 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:32:46.096 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:46.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:46.097 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:32:46.198 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:32:46.227 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:46.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:46.228 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:32:46.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:45 smithi032 ceph-mon[20004]: pgmap v1370: 47 pgs: 5 active+clean+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 1.3 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 632 op/s; 7793/63516 objects degraded (12.269%); 14189/63516 objects misplaced (22.339%); 7.4 MiB/s, 117 objects/s recovering 2024-08-19T14:32:46.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:45 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:32:46.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:45 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:32:46.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:45 smithi032 ceph-mon[20004]: osdmap e586: 12 total, 11 up, 9 in 2024-08-19T14:32:46.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:45 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/825607238' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:32:46.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:45 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:32:46.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:45 smithi045 ceph-mon[20143]: pgmap v1370: 47 pgs: 5 active+clean+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 1.3 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 632 op/s; 7793/63516 objects degraded (12.269%); 14189/63516 objects misplaced (22.339%); 7.4 MiB/s, 117 objects/s recovering 2024-08-19T14:32:46.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:45 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:32:46.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:45 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:32:46.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:45 smithi045 ceph-mon[20143]: osdmap e586: 12 total, 11 up, 9 in 2024-08-19T14:32:46.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:45 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/825607238' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:32:46.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:45 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:32:46.330 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:32:46.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:45 smithi031 ceph-mon[16336]: pgmap v1370: 47 pgs: 5 active+clean+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 1.3 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 632 op/s; 7793/63516 objects degraded (12.269%); 14189/63516 objects misplaced (22.339%); 7.4 MiB/s, 117 objects/s recovering 2024-08-19T14:32:46.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:45 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:32:46.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:45 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:32:46.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:45 smithi031 ceph-mon[16336]: osdmap e586: 12 total, 11 up, 9 in 2024-08-19T14:32:46.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:45 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/825607238' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:32:46.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:45 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:32:46.368 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:46.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:46.369 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:32:46.391 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-08-19T14:32:46.470 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:32:46.508 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:46.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:46.509 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:32:46.610 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:32:46.641 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:46.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:46.643 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:32:46.744 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:32:46.777 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:46.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:46.778 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:32:46.827 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:32:46.879 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:32:46.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:46.915 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:46.915 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:32:47.017 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:32:47.056 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:47.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:47.057 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:32:47.158 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:32:47.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:47.199 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:47.199 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:32:47.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:46 smithi032 ceph-mon[20004]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:32:47.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:46 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:32:47.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:46 smithi032 ceph-mon[20004]: osdmap e587: 12 total, 11 up, 9 in 2024-08-19T14:32:47.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:46 smithi045 ceph-mon[20143]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:32:47.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:46 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:32:47.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:46 smithi045 ceph-mon[20143]: osdmap e587: 12 total, 11 up, 9 in 2024-08-19T14:32:47.299 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:32:47.331 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:47.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:47.332 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:32:47.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:46 smithi031 ceph-mon[16336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:32:47.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:46 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:32:47.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:46 smithi031 ceph-mon[16336]: osdmap e587: 12 total, 11 up, 9 in 2024-08-19T14:32:47.432 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:32:47.466 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:47.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:47.468 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:32:47.569 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:32:47.591 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:47.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:47.592 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:32:47.693 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:32:47.716 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:47.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:47.717 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:32:47.818 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:32:47.839 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:47.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:47.840 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:32:47.941 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:32:47.963 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:47.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:47.964 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:32:47.982 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:32:48.065 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:32:48.087 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:48.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:48.088 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:32:48.189 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:32:48.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:48.222 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:48.223 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:32:48.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:48 smithi032 ceph-mon[20004]: pgmap v1373: 47 pgs: 5 active+clean+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 1.2 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 724 op/s; 7482/57993 objects degraded (12.902%); 13328/57993 objects misplaced (22.982%); 11 MiB/s, 183 objects/s recovering 2024-08-19T14:32:48.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:48 smithi032 ceph-mon[20004]: 6.a scrub starts 2024-08-19T14:32:48.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:48 smithi032 ceph-mon[20004]: 6.a scrub ok 2024-08-19T14:32:48.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:48 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 7482/57993 objects degraded (12.902%), 7 pgs degraded, 16 pgs undersized (PG_DEGRADED) 2024-08-19T14:32:48.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:48 smithi045 ceph-mon[20143]: pgmap v1373: 47 pgs: 5 active+clean+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 1.2 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 724 op/s; 7482/57993 objects degraded (12.902%); 13328/57993 objects misplaced (22.982%); 11 MiB/s, 183 objects/s recovering 2024-08-19T14:32:48.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:48 smithi045 ceph-mon[20143]: 6.a scrub starts 2024-08-19T14:32:48.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:48 smithi045 ceph-mon[20143]: 6.a scrub ok 2024-08-19T14:32:48.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:48 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 7482/57993 objects degraded (12.902%), 7 pgs degraded, 16 pgs undersized (PG_DEGRADED) 2024-08-19T14:32:48.324 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:32:48.346 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:48.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:48.347 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:32:48.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:48 smithi031 ceph-mon[16336]: pgmap v1373: 47 pgs: 5 active+clean+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 1.2 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 724 op/s; 7482/57993 objects degraded (12.902%); 13328/57993 objects misplaced (22.982%); 11 MiB/s, 183 objects/s recovering 2024-08-19T14:32:48.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:48 smithi031 ceph-mon[16336]: 6.a scrub starts 2024-08-19T14:32:48.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:48 smithi031 ceph-mon[16336]: 6.a scrub ok 2024-08-19T14:32:48.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:48 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 7482/57993 objects degraded (12.902%), 7 pgs degraded, 16 pgs undersized (PG_DEGRADED) 2024-08-19T14:32:48.449 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:32:48.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:48.478 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:48.478 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:32:48.579 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:32:48.618 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:48.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:48.620 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:32:48.720 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:32:48.754 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:48.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:48.755 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:32:48.857 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:32:48.892 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:48.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:48.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:32:48.994 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:32:49.015 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:49.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:49.016 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:32:49.117 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:32:49.147 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:49.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:49.148 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:32:49.249 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:32:49.271 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:49.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:49.272 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:32:49.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:49 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:32:49.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:49 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:32:49.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:49 smithi045 ceph-mon[20143]: osdmap e588: 12 total, 11 up, 9 in 2024-08-19T14:32:49.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:49 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:32:49.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:49 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:32:49.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:49 smithi031 ceph-mon[16336]: osdmap e588: 12 total, 11 up, 9 in 2024-08-19T14:32:49.373 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:32:49.377 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 7, 8, 2, 11] out_osds: [10, 6, 9] dead_osds: [2] live_osds: [0, 1, 3, 4, 5, 6, 7, 8, 9, 11, 10] 2024-08-19T14:32:49.377 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-19T14:32:49.377 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-08-19T14:32:49.377 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-08-19T14:32:49.377 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-08-19T14:32:49.377 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.5 config set filestore_inject_stall 3 2024-08-19T14:32:49.380 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:49 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:32:49.380 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:49 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:32:49.380 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:49 smithi032 ceph-mon[20004]: osdmap e588: 12 total, 11 up, 9 in 2024-08-19T14:32:49.424 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:49.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:49.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:32:49.526 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:32:49.561 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:49.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:49.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:32:49.663 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:32:49.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:49.700 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:49.700 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:32:49.800 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:32:49.833 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:49.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:49.834 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:32:49.935 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:32:49.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:49.961 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:49.961 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:32:50.062 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:32:50.087 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:50.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:50.088 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:32:50.115 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:32:50.115 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-08-19T14:32:50.115 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.115 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDPGPushReply(6.3 588/556 [PushReplyOp(6:c92dd1af:::benchmark_data_smithi045_97937_object93729:head)])", 2024-08-19T14:32:50.115 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:32:49.354891+0000", 2024-08-19T14:32:50.115 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.023364592, 2024-08-19T14:32:50.115 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.023389777, 2024-08-19T14:32:50.115 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:32:50.115 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:32:50.115 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "queued for pg", 2024-08-19T14:32:50.115 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:32:50.115 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.116 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:32:50.116 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.354891+0000", 2024-08-19T14:32:50.116 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:32:50.116 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.116 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.116 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:32:50.116 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.354891+0000", 2024-08-19T14:32:50.116 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:32:50.116 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.116 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.116 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:32:50.116 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.354891+0000", 2024-08-19T14:32:50.116 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1600000000000002e-07 2024-08-19T14:32:50.117 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.117 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.117 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:32:50.117 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.354896+0000", 2024-08-19T14:32:50.117 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8550000000000001e-06 2024-08-19T14:32:50.117 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.117 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.117 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:32:50.117 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.354904+0000", 2024-08-19T14:32:50.117 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.0690000000000002e-06 2024-08-19T14:32:50.117 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.117 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.118 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:32:50.118 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.354910+0000", 2024-08-19T14:32:50.118 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.772e-06 2024-08-19T14:32:50.118 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:32:50.118 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:32:50.118 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:32:50.118 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.118 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.118 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26191.0:198681 6.3 6:c61f03ad:::benchmark_data_smithi045_97937_object91408:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e588)", 2024-08-19T14:32:50.118 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:32:49.365095+0000", 2024-08-19T14:32:50.118 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.013160760000000001, 2024-08-19T14:32:50.118 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.013244282, 2024-08-19T14:32:50.118 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:32:50.119 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:32:50.119 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "waiting for sub ops", 2024-08-19T14:32:50.119 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:32:50.119 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26191", 2024-08-19T14:32:50.119 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:32:50.119 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 198681 2024-08-19T14:32:50.119 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.119 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:32:50.119 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.119 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:32:50.119 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.365095+0000", 2024-08-19T14:32:50.119 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:32:50.119 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.119 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.120 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:32:50.120 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.365095+0000", 2024-08-19T14:32:50.120 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:32:50.120 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.120 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.120 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:32:50.120 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.365097+0000", 2024-08-19T14:32:50.120 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7570000000000001e-06 2024-08-19T14:32:50.120 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.120 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.120 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:32:50.120 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.365103+0000", 2024-08-19T14:32:50.120 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9199999999999998e-06 2024-08-19T14:32:50.120 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.120 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.121 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:32:50.121 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.365118+0000", 2024-08-19T14:32:50.121 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4594e-05 2024-08-19T14:32:50.121 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.121 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.121 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:32:50.121 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.365127+0000", 2024-08-19T14:32:50.121 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6829999999999997e-06 2024-08-19T14:32:50.121 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.121 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.121 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:32:50.121 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.368916+0000", 2024-08-19T14:32:50.121 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0037896470000000002 2024-08-19T14:32:50.122 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.122 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.122 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:32:50.122 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.369057+0000", 2024-08-19T14:32:50.122 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014079500000000001 2024-08-19T14:32:50.122 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.122 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.122 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 8,11", 2024-08-19T14:32:50.122 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.369190+0000", 2024-08-19T14:32:50.122 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013242700000000001 2024-08-19T14:32:50.122 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:32:50.122 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:32:50.122 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:32:50.122 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.123 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.123 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.26191.0:198676 6.6 e588/550 6:67166ac2:::benchmark_data_smithi045_97937_object91403:head v 588'12409, mlcod=535'5689)", 2024-08-19T14:32:50.123 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:32:49.366305+0000", 2024-08-19T14:32:50.123 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.011950643, 2024-08-19T14:32:50.123 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.012109052, 2024-08-19T14:32:50.123 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:32:50.123 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:32:50.123 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-08-19T14:32:50.123 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:32:50.123 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.123 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:32:50.123 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.366305+0000", 2024-08-19T14:32:50.123 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:32:50.123 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.123 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.123 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:32:50.124 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.366305+0000", 2024-08-19T14:32:50.124 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:32:50.124 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.124 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.124 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:32:50.124 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.366306+0000", 2024-08-19T14:32:50.124 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.78e-07 2024-08-19T14:32:50.124 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.124 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.124 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:32:50.125 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.366313+0000", 2024-08-19T14:32:50.125 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4100000000000002e-06 2024-08-19T14:32:50.125 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.125 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.125 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:32:50.125 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.366323+0000", 2024-08-19T14:32:50.125 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.9539999999999999e-06 2024-08-19T14:32:50.125 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.125 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.125 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:32:50.125 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.366330+0000", 2024-08-19T14:32:50.125 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3169999999999998e-06 2024-08-19T14:32:50.125 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.126 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.126 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:32:50.126 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.366390+0000", 2024-08-19T14:32:50.126 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.037e-05 2024-08-19T14:32:50.126 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.126 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.126 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:32:50.126 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.366415+0000", 2024-08-19T14:32:50.126 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5459000000000001e-05 2024-08-19T14:32:50.126 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:32:50.126 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:32:50.126 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:32:50.126 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.126 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.126 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.26191.0:198679 6.6 e588/550 6:638dee3a:::benchmark_data_smithi045_97937_object91406:head v 588'12410, mlcod=535'5693)", 2024-08-19T14:32:50.127 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:32:49.372739+0000", 2024-08-19T14:32:50.127 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.0055164999999999997, 2024-08-19T14:32:50.127 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.005731346, 2024-08-19T14:32:50.127 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:32:50.127 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:32:50.127 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-08-19T14:32:50.127 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:32:50.127 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.127 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:32:50.127 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.372739+0000", 2024-08-19T14:32:50.127 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:32:50.127 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.127 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.127 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:32:50.128 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.372739+0000", 2024-08-19T14:32:50.128 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:32:50.128 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.128 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.128 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:32:50.128 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.372740+0000", 2024-08-19T14:32:50.128 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.1399999999999995e-07 2024-08-19T14:32:50.128 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.128 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.128 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:32:50.128 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.372746+0000", 2024-08-19T14:32:50.128 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2439999999999998e-06 2024-08-19T14:32:50.128 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.128 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.129 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:32:50.129 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.372758+0000", 2024-08-19T14:32:50.129 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1545e-05 2024-08-19T14:32:50.129 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.129 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.129 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:32:50.129 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.372765+0000", 2024-08-19T14:32:50.129 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8759999999999999e-06 2024-08-19T14:32:50.129 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.129 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.129 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:32:50.129 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.372842+0000", 2024-08-19T14:32:50.129 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7088000000000005e-05 2024-08-19T14:32:50.129 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.130 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.130 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:32:50.130 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.372865+0000", 2024-08-19T14:32:50.130 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3391e-05 2024-08-19T14:32:50.130 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:32:50.130 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:32:50.130 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:32:50.130 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.130 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.130 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26191.0:198686 6.5 6:ac51656e:::benchmark_data_smithi045_97937_object91413:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e588)", 2024-08-19T14:32:50.130 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:32:49.373375+0000", 2024-08-19T14:32:50.130 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.0048807099999999999, 2024-08-19T14:32:50.130 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0051558189999999999, 2024-08-19T14:32:50.131 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:32:50.131 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:32:50.131 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "waiting for sub ops", 2024-08-19T14:32:50.131 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:32:50.131 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26191", 2024-08-19T14:32:50.131 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:32:50.131 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 198686 2024-08-19T14:32:50.131 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.131 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:32:50.131 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.131 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:32:50.131 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.373375+0000", 2024-08-19T14:32:50.131 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:32:50.132 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.132 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.132 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:32:50.132 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.373375+0000", 2024-08-19T14:32:50.132 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:32:50.132 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.132 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.132 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:32:50.132 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.373376+0000", 2024-08-19T14:32:50.132 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.536e-06 2024-08-19T14:32:50.132 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.132 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.132 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:32:50.132 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.373383+0000", 2024-08-19T14:32:50.133 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1249999999999998e-06 2024-08-19T14:32:50.133 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.133 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.133 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:32:50.133 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.373396+0000", 2024-08-19T14:32:50.133 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3111e-05 2024-08-19T14:32:50.133 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.133 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.133 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:32:50.133 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.373403+0000", 2024-08-19T14:32:50.133 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7249999999999997e-06 2024-08-19T14:32:50.133 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.134 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.134 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:32:50.134 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.376766+0000", 2024-08-19T14:32:50.134 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0033630449999999998 2024-08-19T14:32:50.134 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.134 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.134 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:32:50.134 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.376895+0000", 2024-08-19T14:32:50.134 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000128644 2024-08-19T14:32:50.134 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.135 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.135 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 5", 2024-08-19T14:32:50.135 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.377780+0000", 2024-08-19T14:32:50.135 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000885431 2024-08-19T14:32:50.135 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:32:50.135 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:32:50.135 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:32:50.135 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.135 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.135 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.26191.0:198687 6.1 e588/565 6:8c8b64b1:::benchmark_data_smithi045_97937_object91414:head v 588'12098, mlcod=588'12096)", 2024-08-19T14:32:50.135 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:32:49.373608+0000", 2024-08-19T14:32:50.135 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.0046478040000000002, 2024-08-19T14:32:50.135 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0049951989999999996, 2024-08-19T14:32:50.136 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:32:50.136 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:32:50.136 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-08-19T14:32:50.136 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:32:50.136 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.136 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:32:50.136 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.373608+0000", 2024-08-19T14:32:50.136 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:32:50.136 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.136 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.136 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:32:50.136 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.373608+0000", 2024-08-19T14:32:50.136 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:32:50.137 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.137 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.137 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:32:50.137 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.373608+0000", 2024-08-19T14:32:50.137 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3799999999999997e-07 2024-08-19T14:32:50.137 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.137 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.137 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:32:50.137 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.373614+0000", 2024-08-19T14:32:50.137 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8619999999999999e-06 2024-08-19T14:32:50.137 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.137 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.137 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:32:50.138 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.373624+0000", 2024-08-19T14:32:50.138 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0018e-05 2024-08-19T14:32:50.138 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.138 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.138 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:32:50.138 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.373631+0000", 2024-08-19T14:32:50.138 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4450000000000001e-06 2024-08-19T14:32:50.138 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.138 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.138 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:32:50.138 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.373684+0000", 2024-08-19T14:32:50.138 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3749000000000003e-05 2024-08-19T14:32:50.138 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.138 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.139 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:32:50.139 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.373704+0000", 2024-08-19T14:32:50.139 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9947999999999999e-05 2024-08-19T14:32:50.139 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:32:50.139 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:32:50.139 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:32:50.139 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.139 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.139 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop_reply(client.26191.0:198681 6.3 e588/556)", 2024-08-19T14:32:50.139 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:32:49.377865+0000", 2024-08-19T14:32:50.139 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.00039071700000000001, 2024-08-19T14:32:50.139 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00079113299999999996, 2024-08-19T14:32:50.139 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:32:50.139 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:32:50.139 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "queued for pg", 2024-08-19T14:32:50.140 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:32:50.140 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.140 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:32:50.140 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.377865+0000", 2024-08-19T14:32:50.140 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:32:50.140 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.140 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.140 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:32:50.140 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.377865+0000", 2024-08-19T14:32:50.140 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:32:50.140 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.140 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.140 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:32:50.140 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.377866+0000", 2024-08-19T14:32:50.141 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.16e-07 2024-08-19T14:32:50.141 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.141 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.141 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:32:50.141 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.377872+0000", 2024-08-19T14:32:50.141 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0669999999999997e-06 2024-08-19T14:32:50.141 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.141 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.141 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:32:50.141 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.377883+0000", 2024-08-19T14:32:50.141 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.115e-05 2024-08-19T14:32:50.141 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.141 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.141 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:32:50.142 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.377891+0000", 2024-08-19T14:32:50.142 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.8310000000000001e-06 2024-08-19T14:32:50.142 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:32:50.142 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:32:50.142 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:32:50.142 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.142 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.142 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.26191.0:198663 6.b e588/510 6:d47c0201:::benchmark_data_smithi045_97937_object91390:head v 588'12526, mlcod=588'12522)", 2024-08-19T14:32:50.142 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:32:49.353896+0000", 2024-08-19T14:32:50.142 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.024359568000000002, 2024-08-19T14:32:50.142 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.024807333000000001, 2024-08-19T14:32:50.142 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:32:50.142 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:32:50.142 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-08-19T14:32:50.143 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:32:50.143 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.143 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:32:50.143 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.353896+0000", 2024-08-19T14:32:50.143 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:32:50.143 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.143 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.143 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:32:50.143 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.353896+0000", 2024-08-19T14:32:50.143 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:32:50.143 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.143 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.143 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:32:50.144 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.353897+0000", 2024-08-19T14:32:50.144 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3900000000000004e-07 2024-08-19T14:32:50.144 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.144 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.144 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:32:50.144 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.353903+0000", 2024-08-19T14:32:50.144 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2809999999999997e-06 2024-08-19T14:32:50.144 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.144 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.144 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:32:50.144 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.353910+0000", 2024-08-19T14:32:50.144 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5209999999999997e-06 2024-08-19T14:32:50.144 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.145 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.145 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:32:50.145 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.353915+0000", 2024-08-19T14:32:50.145 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7740000000000004e-06 2024-08-19T14:32:50.145 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.145 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.145 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:32:50.145 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.360365+0000", 2024-08-19T14:32:50.145 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0064492949999999999 2024-08-19T14:32:50.145 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.145 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.145 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:32:50.145 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.360385+0000", 2024-08-19T14:32:50.145 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.073e-05 2024-08-19T14:32:50.145 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:32:50.146 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:32:50.146 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:32:50.146 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.146 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.146 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26191.0:198666 6.5 6:a754379a:::benchmark_data_smithi045_97937_object91393:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e588)", 2024-08-19T14:32:50.146 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:32:49.354030+0000", 2024-08-19T14:32:50.146 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.024225838999999999, 2024-08-19T14:32:50.146 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.024733489000000001, 2024-08-19T14:32:50.146 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:32:50.146 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:32:50.146 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "waiting for sub ops", 2024-08-19T14:32:50.146 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:32:50.146 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26191", 2024-08-19T14:32:50.146 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:32:50.147 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 198666 2024-08-19T14:32:50.147 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.147 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:32:50.147 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.147 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:32:50.147 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.354030+0000", 2024-08-19T14:32:50.147 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:32:50.147 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.147 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.147 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:32:50.147 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.354030+0000", 2024-08-19T14:32:50.147 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:32:50.147 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.147 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.148 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:32:50.148 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.354031+0000", 2024-08-19T14:32:50.148 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3340000000000001e-06 2024-08-19T14:32:50.148 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.148 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.148 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:32:50.148 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.354035+0000", 2024-08-19T14:32:50.148 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5020000000000001e-06 2024-08-19T14:32:50.148 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.148 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.148 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:32:50.148 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.354043+0000", 2024-08-19T14:32:50.148 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.4619999999999996e-06 2024-08-19T14:32:50.148 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.149 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.149 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:32:50.149 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.354050+0000", 2024-08-19T14:32:50.149 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6479999999999997e-06 2024-08-19T14:32:50.149 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.149 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.149 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:32:50.149 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.356181+0000", 2024-08-19T14:32:50.149 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0021314039999999999 2024-08-19T14:32:50.149 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.149 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.149 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:32:50.149 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.356292+0000", 2024-08-19T14:32:50.149 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000110463 2024-08-19T14:32:50.150 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.150 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.150 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 5", 2024-08-19T14:32:50.150 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.356399+0000", 2024-08-19T14:32:50.150 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00010732 2024-08-19T14:32:50.150 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.150 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.150 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:32:50.150 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.371739+0000", 2024-08-19T14:32:50.150 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.015339689 2024-08-19T14:32:50.150 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:32:50.150 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:32:50.150 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:32:50.151 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.151 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.151 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26191.0:198671 6.5 6:acfc60d2:::benchmark_data_smithi045_97937_object91398:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e588)", 2024-08-19T14:32:50.151 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:32:49.354099+0000", 2024-08-19T14:32:50.151 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.024156243000000001, 2024-08-19T14:32:50.151 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.024739523999999999, 2024-08-19T14:32:50.151 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:32:50.151 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:32:50.151 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "waiting for sub ops", 2024-08-19T14:32:50.151 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:32:50.151 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26191", 2024-08-19T14:32:50.151 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:32:50.151 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 198671 2024-08-19T14:32:50.151 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.152 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:32:50.152 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.152 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:32:50.152 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.354099+0000", 2024-08-19T14:32:50.152 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:32:50.152 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.152 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.152 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:32:50.152 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.354099+0000", 2024-08-19T14:32:50.152 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:32:50.152 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.152 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.152 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:32:50.153 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.354100+0000", 2024-08-19T14:32:50.153 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3799999999999998e-07 2024-08-19T14:32:50.153 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.153 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.153 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:32:50.153 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.354102+0000", 2024-08-19T14:32:50.153 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2840000000000001e-06 2024-08-19T14:32:50.153 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.153 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.153 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:32:50.153 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.354108+0000", 2024-08-19T14:32:50.153 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2280000000000003e-06 2024-08-19T14:32:50.153 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.153 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.154 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:32:50.154 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.354112+0000", 2024-08-19T14:32:50.154 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.189e-06 2024-08-19T14:32:50.154 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.154 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.154 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:32:50.154 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.361071+0000", 2024-08-19T14:32:50.154 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0069585350000000001 2024-08-19T14:32:50.154 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.154 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.154 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:32:50.154 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.361182+0000", 2024-08-19T14:32:50.154 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00011089 2024-08-19T14:32:50.155 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.155 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.155 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 5", 2024-08-19T14:32:50.155 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.361280+0000", 2024-08-19T14:32:50.155 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.8604000000000007e-05 2024-08-19T14:32:50.155 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:32:50.155 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:32:50.155 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:32:50.155 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.155 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.155 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26191.0:198673 6.5 6:a97002a0:::benchmark_data_smithi045_97937_object91400:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e588)", 2024-08-19T14:32:50.155 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:32:49.354118+0000", 2024-08-19T14:32:50.155 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.024137494999999998, 2024-08-19T14:32:50.155 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.024789934999999999, 2024-08-19T14:32:50.156 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:32:50.156 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:32:50.156 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "waiting for sub ops", 2024-08-19T14:32:50.156 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:32:50.156 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26191", 2024-08-19T14:32:50.156 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:32:50.156 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 198673 2024-08-19T14:32:50.156 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.156 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:32:50.156 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.156 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:32:50.156 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.354118+0000", 2024-08-19T14:32:50.156 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:32:50.156 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.157 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.157 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:32:50.157 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.354118+0000", 2024-08-19T14:32:50.157 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:32:50.157 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.157 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.157 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:32:50.157 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.354118+0000", 2024-08-19T14:32:50.157 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2300000000000002e-07 2024-08-19T14:32:50.157 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.157 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.157 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:32:50.157 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.354121+0000", 2024-08-19T14:32:50.157 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1090000000000001e-06 2024-08-19T14:32:50.157 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.157 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.158 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:32:50.158 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.354126+0000", 2024-08-19T14:32:50.158 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1150000000000002e-06 2024-08-19T14:32:50.158 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.158 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.158 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:32:50.158 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.354130+0000", 2024-08-19T14:32:50.158 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2760000000000002e-06 2024-08-19T14:32:50.158 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.158 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.158 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:32:50.159 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.366012+0000", 2024-08-19T14:32:50.159 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011882403 2024-08-19T14:32:50.159 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.159 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.159 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:32:50.159 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.366138+0000", 2024-08-19T14:32:50.159 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000125339 2024-08-19T14:32:50.159 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.159 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.159 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 5", 2024-08-19T14:32:50.159 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.366262+0000", 2024-08-19T14:32:50.159 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012443 2024-08-19T14:32:50.160 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:32:50.160 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:32:50.160 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:32:50.160 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.160 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.160 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26191.0:198677 6.5 6:a03d18ce:::benchmark_data_smithi045_97937_object91404:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e588)", 2024-08-19T14:32:50.160 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:32:49.354733+0000", 2024-08-19T14:32:50.160 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.023522978999999999, 2024-08-19T14:32:50.160 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.024244056, 2024-08-19T14:32:50.160 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:32:50.160 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:32:50.160 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "waiting for sub ops", 2024-08-19T14:32:50.160 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:32:50.160 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26191", 2024-08-19T14:32:50.161 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:32:50.161 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 198677 2024-08-19T14:32:50.161 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.161 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:32:50.161 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.161 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:32:50.161 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.354733+0000", 2024-08-19T14:32:50.161 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:32:50.161 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.161 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.161 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:32:50.161 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.354733+0000", 2024-08-19T14:32:50.161 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:32:50.161 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.161 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.162 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:32:50.162 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.354733+0000", 2024-08-19T14:32:50.162 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6099999999999999e-07 2024-08-19T14:32:50.162 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.162 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.162 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:32:50.162 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.354737+0000", 2024-08-19T14:32:50.162 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9400000000000004e-06 2024-08-19T14:32:50.162 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.162 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.162 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:32:50.162 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.354750+0000", 2024-08-19T14:32:50.162 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2833999999999999e-05 2024-08-19T14:32:50.163 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.163 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.163 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:32:50.163 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.354758+0000", 2024-08-19T14:32:50.163 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.9899999999999997e-06 2024-08-19T14:32:50.163 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.163 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.163 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:32:50.163 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.371769+0000", 2024-08-19T14:32:50.163 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.017010549 2024-08-19T14:32:50.163 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.163 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.163 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:32:50.163 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.371883+0000", 2024-08-19T14:32:50.163 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000113892 2024-08-19T14:32:50.163 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.163 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.164 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 5", 2024-08-19T14:32:50.164 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.372017+0000", 2024-08-19T14:32:50.164 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013451399999999999 2024-08-19T14:32:50.164 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:32:50.164 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:32:50.164 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:32:50.164 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.164 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.164 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.26191.0:198675 6.b e588/510 6:dd0a4292:::benchmark_data_smithi045_97937_object91402:head v 588'12527, mlcod=588'12523)", 2024-08-19T14:32:50.164 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:32:49.354779+0000", 2024-08-19T14:32:50.164 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.023476324, 2024-08-19T14:32:50.164 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.024275718000000002, 2024-08-19T14:32:50.164 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:32:50.164 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:32:50.164 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-08-19T14:32:50.164 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:32:50.165 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.165 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:32:50.165 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.354779+0000", 2024-08-19T14:32:50.165 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:32:50.165 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.165 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.165 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:32:50.165 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.354779+0000", 2024-08-19T14:32:50.165 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:32:50.165 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.165 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.165 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:32:50.165 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.354780+0000", 2024-08-19T14:32:50.165 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1800000000000002e-07 2024-08-19T14:32:50.166 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.166 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.166 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:32:50.166 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.354784+0000", 2024-08-19T14:32:50.166 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2910000000000001e-06 2024-08-19T14:32:50.166 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.166 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.166 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:32:50.166 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.354791+0000", 2024-08-19T14:32:50.166 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2919999999999998e-06 2024-08-19T14:32:50.166 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.166 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.166 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:32:50.166 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.354805+0000", 2024-08-19T14:32:50.166 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3379e-05 2024-08-19T14:32:50.167 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.167 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.167 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:32:50.167 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.369279+0000", 2024-08-19T14:32:50.167 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.014474226 2024-08-19T14:32:50.167 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:32:50.167 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:32:50.167 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:32:50.167 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:49.369297+0000", 2024-08-19T14:32:50.167 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7918999999999999e-05 2024-08-19T14:32:50.167 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:32:50.167 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:32:50.167 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:32:50.167 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:32:50.167 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-08-19T14:32:50.167 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 13 2024-08-19T14:32:50.168 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:32:50.189 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:32:50.212 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:50.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:50.213 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:32:50.227 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:50 smithi031 ceph-mon[16336]: pgmap v1375: 47 pgs: 5 active+clean+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 1.2 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 600 op/s; 7482/57993 objects degraded (12.902%); 13328/57993 objects misplaced (22.982%); 6.3 MiB/s, 100 objects/s recovering 2024-08-19T14:32:50.290 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:32:50.290 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:32:50.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:50 smithi045 ceph-mon[20143]: pgmap v1375: 47 pgs: 5 active+clean+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 1.2 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 600 op/s; 7482/57993 objects degraded (12.902%); 13328/57993 objects misplaced (22.982%); 6.3 MiB/s, 100 objects/s recovering 2024-08-19T14:32:50.313 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:32:50.325 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:32:50.326 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:32:50.326 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-08-19T14:32:50.347 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:50.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:50.348 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:32:50.358 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:32:50.358 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:32:50.398 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:32:50.398 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:32:50.437 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:32:50.437 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:32:50.448 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:32:50.466 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:32:50.466 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:32:50.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:50.471 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:50.471 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:32:50.497 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:32:50.498 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:32:50.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:50 smithi032 ceph-mon[20004]: pgmap v1375: 47 pgs: 5 active+clean+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 1.2 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 600 op/s; 7482/57993 objects degraded (12.902%); 13328/57993 objects misplaced (22.982%); 6.3 MiB/s, 100 objects/s recovering 2024-08-19T14:32:50.526 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:32:50.526 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:32:50.555 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:32:50.556 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:32:50.572 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:32:50.586 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:32:50.586 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:32:50.608 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:50.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:50.609 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:32:50.617 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:32:50.617 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:32:50.710 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:32:50.739 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:50.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:50.741 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:32:50.842 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:32:50.881 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:50.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:50.882 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:32:50.983 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:32:51.006 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:51.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:51.007 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:32:51.108 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:32:51.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:51.142 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:51.142 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:32:51.243 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:32:51.279 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:51.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:51.280 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:32:51.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:51 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3075510495' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:32:51.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:51 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3075510495' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:32:51.380 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:32:51.417 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:51.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:51.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:32:51.477 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-19T14:32:51.477 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-19T14:32:51.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:51 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3075510495' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:32:51.520 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:32:51.555 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:51.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:51.556 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:32:51.657 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:32:51.660 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-19T14:32:51.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:51.690 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:51.690 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:32:51.791 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:32:51.813 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:51.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:51.814 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:32:51.915 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:32:51.948 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:51.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:51.949 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:32:52.050 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:32:52.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:52.079 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:52.080 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:32:52.147 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:32:52.180 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:32:52.212 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:52.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:52.213 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:32:52.289 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:52 smithi032 ceph-mon[20004]: pgmap v1376: 47 pgs: 5 active+clean+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 1.1 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 599 op/s; 7263/52728 objects degraded (13.774%); 12225/52728 objects misplaced (23.185%); 6.3 MiB/s, 100 objects/s recovering 2024-08-19T14:32:52.295 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:52 smithi031 ceph-mon[16336]: pgmap v1376: 47 pgs: 5 active+clean+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 1.1 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 599 op/s; 7263/52728 objects degraded (13.774%); 12225/52728 objects misplaced (23.185%); 6.3 MiB/s, 100 objects/s recovering 2024-08-19T14:32:52.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:52 smithi045 ceph-mon[20143]: pgmap v1376: 47 pgs: 5 active+clean+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 1.1 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 599 op/s; 7263/52728 objects degraded (13.774%); 12225/52728 objects misplaced (23.185%); 6.3 MiB/s, 100 objects/s recovering 2024-08-19T14:32:52.314 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:32:52.349 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:52.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:52.350 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:32:52.450 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:32:52.481 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:52.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:52.482 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:32:52.583 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:32:52.605 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:52.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:52.606 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:32:52.707 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:32:52.732 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:52.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:52.733 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:32:52.834 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:32:52.856 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:52.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:52.857 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:32:52.958 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:32:52.980 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:52.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:52.982 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:32:53.082 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:32:53.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:53.119 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:53.120 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:32:53.221 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:32:53.242 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:53.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:53.243 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:32:53.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:53 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 7263/52728 objects degraded (13.774%), 7 pgs degraded, 16 pgs undersized (PG_DEGRADED) 2024-08-19T14:32:53.344 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:32:53.366 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:53.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:53 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 7263/52728 objects degraded (13.774%), 7 pgs degraded, 16 pgs undersized (PG_DEGRADED) 2024-08-19T14:32:53.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:53.367 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:32:53.468 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:32:53.478 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:32:53.489 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:53.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:53.490 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:32:53.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:53 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 7263/52728 objects degraded (13.774%), 7 pgs degraded, 16 pgs undersized (PG_DEGRADED) 2024-08-19T14:32:53.591 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:32:53.613 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:53.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:53.614 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:32:53.715 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:32:53.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:53.757 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:53.757 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:32:53.853 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:32:53.858 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:32:53.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:53.896 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:53.896 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:32:53.997 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:32:54.019 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:54.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:54.020 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:32:54.057 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is unset 2024-08-19T14:32:54.121 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:32:54.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:54.147 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:54.147 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:32:54.248 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:32:54.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:54.279 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:54.279 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:32:54.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:54 smithi045 ceph-mon[20143]: pgmap v1377: 47 pgs: 5 active+clean+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 985 MiB data, 12 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 760 op/s; 6962/47280 objects degraded (14.725%); 11396/47280 objects misplaced (24.103%); 11 MiB/s, 169 objects/s recovering 2024-08-19T14:32:54.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:54 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3370034422' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:32:54.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:54 smithi031 ceph-mon[16336]: pgmap v1377: 47 pgs: 5 active+clean+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 985 MiB data, 12 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 760 op/s; 6962/47280 objects degraded (14.725%); 11396/47280 objects misplaced (24.103%); 11 MiB/s, 169 objects/s recovering 2024-08-19T14:32:54.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:54 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3370034422' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:32:54.380 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:32:54.413 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:54.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:54.414 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:32:54.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:54 smithi032 ceph-mon[20004]: pgmap v1377: 47 pgs: 5 active+clean+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 985 MiB data, 12 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 760 op/s; 6962/47280 objects degraded (14.725%); 11396/47280 objects misplaced (24.103%); 11 MiB/s, 169 objects/s recovering 2024-08-19T14:32:54.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:54 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3370034422' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:32:54.515 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:32:54.537 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:54.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:54.538 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:32:54.625 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:32:54.625 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:32:54.630 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-08-19T14:32:54.639 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:32:54.658 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:32:54.658 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:32:54.659 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-08-19T14:32:54.670 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:54.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:54.671 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:32:54.690 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:32:54.690 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:32:54.716 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:32:54.716 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:32:54.733 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:32:54.742 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:32:54.742 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:32:54.764 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:32:54.764 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:32:54.772 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:32:54.782 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:32:54.782 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:32:54.793 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:54.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:54.794 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:32:54.799 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:32:54.799 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:32:54.818 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:32:54.819 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:32:54.846 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:32:54.846 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:32:54.867 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:32:54.868 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:32:54.895 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:32:54.927 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:54.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:54.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:32:55.030 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:32:55.052 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:55.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:55.053 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:32:55.154 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:32:55.177 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:55.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:55.178 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:32:55.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:55 smithi032 ceph-mon[20004]: 1.19 scrub starts 2024-08-19T14:32:55.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:55 smithi032 ceph-mon[20004]: 1.19 scrub ok 2024-08-19T14:32:55.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:55 smithi032 ceph-mon[20004]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:32:55.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:55 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3370034422' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:32:55.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:55 smithi032 ceph-mon[20004]: osdmap e589: 12 total, 11 up, 9 in 2024-08-19T14:32:55.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:55 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3108607189' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:32:55.278 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:32:55.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:55 smithi045 ceph-mon[20143]: 1.19 scrub starts 2024-08-19T14:32:55.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:55 smithi045 ceph-mon[20143]: 1.19 scrub ok 2024-08-19T14:32:55.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:55 smithi045 ceph-mon[20143]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:32:55.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:55 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3370034422' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:32:55.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:55 smithi045 ceph-mon[20143]: osdmap e589: 12 total, 11 up, 9 in 2024-08-19T14:32:55.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:55 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3108607189' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:32:55.300 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:55.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:55.301 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:32:55.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:55 smithi031 ceph-mon[16336]: 1.19 scrub starts 2024-08-19T14:32:55.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:55 smithi031 ceph-mon[16336]: 1.19 scrub ok 2024-08-19T14:32:55.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:55 smithi031 ceph-mon[16336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:32:55.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:55 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3370034422' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:32:55.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:55 smithi031 ceph-mon[16336]: osdmap e589: 12 total, 11 up, 9 in 2024-08-19T14:32:55.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:55 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3108607189' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:32:55.401 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:32:55.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:55.425 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:55.425 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:32:55.526 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:32:55.555 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:55.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:55.556 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:32:55.657 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:32:55.679 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:55.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:55.680 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:32:55.781 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:32:55.803 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:55.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:55.804 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:32:55.905 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:32:55.926 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:55.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:55.927 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:32:56.028 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:32:56.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:56.065 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:56.065 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:32:56.166 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:32:56.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:56.202 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:56.202 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:32:56.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:56 smithi032 ceph-mon[20004]: pgmap v1379: 47 pgs: 5 active+clean+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 899 MiB data, 12 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 618 op/s; 6720/43149 objects degraded (15.574%); 10261/43149 objects misplaced (23.780%); 4.7 MiB/s, 75 objects/s recovering 2024-08-19T14:32:56.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:56 smithi045 ceph-mon[20143]: pgmap v1379: 47 pgs: 5 active+clean+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 899 MiB data, 12 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 618 op/s; 6720/43149 objects degraded (15.574%); 10261/43149 objects misplaced (23.780%); 4.7 MiB/s, 75 objects/s recovering 2024-08-19T14:32:56.303 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:32:56.336 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:56.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:56.337 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:32:56.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:56 smithi031 ceph-mon[16336]: pgmap v1379: 47 pgs: 5 active+clean+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 899 MiB data, 12 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 618 op/s; 6720/43149 objects degraded (15.574%); 10261/43149 objects misplaced (23.780%); 4.7 MiB/s, 75 objects/s recovering 2024-08-19T14:32:56.437 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:32:56.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:56.467 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:56.468 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:32:56.569 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:32:56.623 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:56.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:56.624 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:32:56.724 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:32:56.754 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:56.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:56.755 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:32:56.856 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:32:56.879 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:56.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:56.881 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:32:56.982 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:32:57.004 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:57.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:57.005 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:32:57.106 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:32:57.128 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:57.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:57.129 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:32:57.230 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:32:57.263 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:57.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:57.265 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:32:57.365 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:32:57.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:57 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:32:57.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:57 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:32:57.391 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:57.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:57.392 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:32:57.455 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:57 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:32:57.456 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:57 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:32:57.493 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:32:57.526 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:57.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:57.527 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:32:57.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:57 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:32:57.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:57 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:32:57.628 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:32:57.663 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:57.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:57.664 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:32:57.765 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:32:57.790 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:57.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:57.791 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:32:57.892 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:32:57.905 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:32:57.905 INFO:teuthology.orchestra.run.smithi032.stdout: "success": "" 2024-08-19T14:32:57.905 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:32:57.927 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:57.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:57.928 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:32:58.029 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:32:58.031 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-19T14:32:58.031 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-19T14:32:58.053 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:58.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:58.054 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:32:58.158 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:32:58.181 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:58.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:58.182 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:32:58.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:58 smithi032 ceph-mon[20004]: pgmap v1380: 47 pgs: 5 active+clean+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 748 MiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 909 op/s; 6399/35907 objects degraded (17.821%); 9320/35907 objects misplaced (25.956%); 9.9 MiB/s, 160 objects/s recovering 2024-08-19T14:32:58.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:58 smithi032 ceph-mon[20004]: osdmap e590: 12 total, 11 up, 9 in 2024-08-19T14:32:58.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:58 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 6399/35907 objects degraded (17.821%), 7 pgs degraded, 16 pgs undersized (PG_DEGRADED) 2024-08-19T14:32:58.283 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:32:58.313 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:58.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:58.314 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:32:58.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:58 smithi031 ceph-mon[16336]: pgmap v1380: 47 pgs: 5 active+clean+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 748 MiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 909 op/s; 6399/35907 objects degraded (17.821%); 9320/35907 objects misplaced (25.956%); 9.9 MiB/s, 160 objects/s recovering 2024-08-19T14:32:58.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:58 smithi031 ceph-mon[16336]: osdmap e590: 12 total, 11 up, 9 in 2024-08-19T14:32:58.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:58 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 6399/35907 objects degraded (17.821%), 7 pgs degraded, 16 pgs undersized (PG_DEGRADED) 2024-08-19T14:32:58.416 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:32:58.453 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:58.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:58.455 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:32:58.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:58 smithi045 ceph-mon[20143]: pgmap v1380: 47 pgs: 5 active+clean+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 748 MiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 909 op/s; 6399/35907 objects degraded (17.821%); 9320/35907 objects misplaced (25.956%); 9.9 MiB/s, 160 objects/s recovering 2024-08-19T14:32:58.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:58 smithi045 ceph-mon[20143]: osdmap e590: 12 total, 11 up, 9 in 2024-08-19T14:32:58.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:58 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 6399/35907 objects degraded (17.821%), 7 pgs degraded, 16 pgs undersized (PG_DEGRADED) 2024-08-19T14:32:58.556 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:32:58.586 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:32:58.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:58.587 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:32:58.688 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:32:58.711 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:58.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:58.712 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:32:58.813 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:32:58.842 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:58.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:58.843 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:32:58.944 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:32:58.967 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:58.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:58.968 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:32:59.069 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:32:59.108 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:59.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:59.109 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:32:59.210 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:32:59.233 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:59.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:59.234 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:32:59.297 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-19T14:32:59.323 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:32:59 smithi031 ceph-mon[16336]: pgmap v1382: 47 pgs: 5 active+clean+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 748 MiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 700 op/s; 6399/35907 objects degraded (17.821%); 9320/35907 objects misplaced (25.956%); 10 MiB/s, 161 objects/s recovering 2024-08-19T14:32:59.334 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:32:59.356 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:59.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:59.357 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:32:59.379 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:32:59 smithi032 ceph-mon[20004]: pgmap v1382: 47 pgs: 5 active+clean+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 748 MiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 700 op/s; 6399/35907 objects degraded (17.821%); 9320/35907 objects misplaced (25.956%); 10 MiB/s, 161 objects/s recovering 2024-08-19T14:32:59.458 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:32:59.481 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:59.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:59.482 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:32:59.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:32:59 smithi045 ceph-mon[20143]: pgmap v1382: 47 pgs: 5 active+clean+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 748 MiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 700 op/s; 6399/35907 objects degraded (17.821%); 9320/35907 objects misplaced (25.956%); 10 MiB/s, 161 objects/s recovering 2024-08-19T14:32:59.583 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:32:59.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:59.613 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:59.613 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:32:59.689 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:32:59.713 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:32:59.747 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:32:59.756 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:32:59.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:59.757 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:32:59.858 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:32:59.880 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:32:59.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:32:59.881 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:32:59.982 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:33:00.004 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:00.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:00.005 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:33:00.032 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:33:00.106 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:33:00.136 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:00.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:00.137 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:33:00.238 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:33:00.260 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:00.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:00.261 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:33:00.360 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:33:00.360 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-08-19T14:33:00.360 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-08-19T14:33:00.360 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-08-19T14:33:00.361 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:33:00.362 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:33:00.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:00.404 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:00.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:33:00.454 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:33:00.505 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:33:00.535 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:00.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:00.536 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:33:00.637 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:33:00.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:00.670 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:00.670 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:33:00.771 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:33:00.801 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:00.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:00.802 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:33:00.903 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:33:00.924 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:00.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:00.925 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:33:01.026 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:33:01.057 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:01.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:01.058 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:33:01.159 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:33:01.179 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is set 2024-08-19T14:33:01.181 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:01.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:01.182 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:33:01.281 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-08-19T14:33:01.283 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:33:01.306 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:01.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:01.307 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:33:01.408 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:33:01.439 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:01.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:01.440 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:33:01.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:01 smithi032 ceph-mon[20004]: pgmap v1383: 47 pgs: 5 active+clean+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 638 MiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 695 op/s; 6185/30594 objects degraded (20.216%); 8239/30594 objects misplaced (26.930%); 5.3 MiB/s, 86 objects/s recovering 2024-08-19T14:33:01.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:01 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1135606665' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:33:01.541 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:33:01.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:01 smithi045 ceph-mon[20143]: pgmap v1383: 47 pgs: 5 active+clean+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 638 MiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 695 op/s; 6185/30594 objects degraded (20.216%); 8239/30594 objects misplaced (26.930%); 5.3 MiB/s, 86 objects/s recovering 2024-08-19T14:33:01.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:01 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1135606665' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:33:01.563 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:01.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:01.564 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:33:01.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:01 smithi031 ceph-mon[16336]: pgmap v1383: 47 pgs: 5 active+clean+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 638 MiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 695 op/s; 6185/30594 objects degraded (20.216%); 8239/30594 objects misplaced (26.930%); 5.3 MiB/s, 86 objects/s recovering 2024-08-19T14:33:01.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:01 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1135606665' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:33:01.664 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:33:01.686 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:01.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:01.687 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:33:01.706 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:33:01.788 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:33:01.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:01.830 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:01.830 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:33:01.931 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:33:01.964 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:01.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:01.965 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:33:02.066 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:33:02.119 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:02.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:02.120 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:33:02.221 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:33:02.244 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:02.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:02.245 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:33:02.305 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:33:02.305 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-08-19T14:33:02.305 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-08-19T14:33:02.305 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-08-19T14:33:02.305 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.305 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26191.0:76303 6.5 6:a62f8e5b:::benchmark_data_smithi045_97937_object76302:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e526)", 2024-08-19T14:33:02.305 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:29:17.435178+0000", 2024-08-19T14:33:02.305 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 224.86490569700001, 2024-08-19T14:33:02.305 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.27830824199999998, 2024-08-19T14:33:02.306 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:33:02.306 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:33:02.306 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:33:02.306 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:33:02.306 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26191", 2024-08-19T14:33:02.306 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:33:02.306 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 76303 2024-08-19T14:33:02.306 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.306 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:33:02.306 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.306 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:33:02.306 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.435178+0000", 2024-08-19T14:33:02.307 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:33:02.307 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.307 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.307 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:33:02.307 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.435178+0000", 2024-08-19T14:33:02.307 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:33:02.307 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.307 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.307 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:33:02.307 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.435179+0000", 2024-08-19T14:33:02.307 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.874e-06 2024-08-19T14:33:02.307 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.307 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.308 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:33:02.308 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.435304+0000", 2024-08-19T14:33:02.308 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012452399999999999 2024-08-19T14:33:02.308 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.308 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.308 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:33:02.308 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.435325+0000", 2024-08-19T14:33:02.308 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0639999999999999e-05 2024-08-19T14:33:02.308 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.308 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.308 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:33:02.308 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.435337+0000", 2024-08-19T14:33:02.308 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2544000000000001e-05 2024-08-19T14:33:02.309 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.309 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.309 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:33:02.309 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.437771+0000", 2024-08-19T14:33:02.309 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0024334729999999998 2024-08-19T14:33:02.309 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.309 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.309 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:33:02.309 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.437779+0000", 2024-08-19T14:33:02.309 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.0989999999999999e-06 2024-08-19T14:33:02.309 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.309 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.310 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:33:02.310 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.672561+0000", 2024-08-19T14:33:02.310 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.23478202500000001 2024-08-19T14:33:02.310 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.310 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.310 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:33:02.310 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.672692+0000", 2024-08-19T14:33:02.310 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000130844 2024-08-19T14:33:02.310 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.310 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.310 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,5", 2024-08-19T14:33:02.310 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.672799+0000", 2024-08-19T14:33:02.310 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000106873 2024-08-19T14:33:02.311 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.311 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.311 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:02.311 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.678796+0000", 2024-08-19T14:33:02.311 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0059978000000000002 2024-08-19T14:33:02.311 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.311 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.311 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:33:02.311 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.678812+0000", 2024-08-19T14:33:02.311 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6143000000000001e-05 2024-08-19T14:33:02.311 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.311 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.311 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:02.311 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.710311+0000", 2024-08-19T14:33:02.312 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.031498783000000002 2024-08-19T14:33:02.312 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.312 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.312 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:33:02.312 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.710382+0000", 2024-08-19T14:33:02.312 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0769e-05 2024-08-19T14:33:02.312 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.312 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.312 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:33:02.312 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.713486+0000", 2024-08-19T14:33:02.312 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0031038509999999999 2024-08-19T14:33:02.312 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:33:02.312 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:33:02.312 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:33:02.312 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.312 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.313 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26191.0:76317 6.5 6:ad41793c:::benchmark_data_smithi045_97937_object76316:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e526)", 2024-08-19T14:33:02.313 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:29:17.435362+0000", 2024-08-19T14:33:02.313 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 224.86472179500001, 2024-08-19T14:33:02.313 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.28174579900000002, 2024-08-19T14:33:02.313 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:33:02.313 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:33:02.313 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:33:02.313 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:33:02.313 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26191", 2024-08-19T14:33:02.313 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:33:02.313 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 76317 2024-08-19T14:33:02.313 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.313 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:33:02.313 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.314 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:33:02.314 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.435362+0000", 2024-08-19T14:33:02.314 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:33:02.314 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.314 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.314 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:33:02.314 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.435362+0000", 2024-08-19T14:33:02.314 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:33:02.314 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.314 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.314 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:33:02.314 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.435362+0000", 2024-08-19T14:33:02.315 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9999999999999997e-07 2024-08-19T14:33:02.315 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.315 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.315 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:33:02.315 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.435397+0000", 2024-08-19T14:33:02.315 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.4538e-05 2024-08-19T14:33:02.315 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.315 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.315 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:33:02.315 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.435405+0000", 2024-08-19T14:33:02.315 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.32e-06 2024-08-19T14:33:02.315 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.315 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.315 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:33:02.316 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.435411+0000", 2024-08-19T14:33:02.316 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0090000000000004e-06 2024-08-19T14:33:02.316 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.316 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.316 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:33:02.316 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.437792+0000", 2024-08-19T14:33:02.316 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0023814999999999999 2024-08-19T14:33:02.316 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.316 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.316 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:33:02.316 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.437796+0000", 2024-08-19T14:33:02.316 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7610000000000001e-06 2024-08-19T14:33:02.316 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.316 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.317 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:33:02.317 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.676699+0000", 2024-08-19T14:33:02.317 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.23890259699999999 2024-08-19T14:33:02.317 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.317 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.317 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:33:02.317 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.676815+0000", 2024-08-19T14:33:02.317 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00011618800000000001 2024-08-19T14:33:02.317 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.317 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.317 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,5", 2024-08-19T14:33:02.317 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.676929+0000", 2024-08-19T14:33:02.317 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00011372899999999999 2024-08-19T14:33:02.318 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.318 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.318 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:33:02.318 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.678907+0000", 2024-08-19T14:33:02.318 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0019780900000000001 2024-08-19T14:33:02.318 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.318 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.318 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:02.318 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.679369+0000", 2024-08-19T14:33:02.318 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00046200599999999999 2024-08-19T14:33:02.318 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.318 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.318 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:02.318 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.713578+0000", 2024-08-19T14:33:02.318 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.034208648000000001 2024-08-19T14:33:02.319 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.319 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.319 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:33:02.319 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.713646+0000", 2024-08-19T14:33:02.319 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8518999999999999e-05 2024-08-19T14:33:02.319 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.319 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.319 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:33:02.319 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.717107+0000", 2024-08-19T14:33:02.319 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0034612940000000002 2024-08-19T14:33:02.319 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:33:02.319 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:33:02.319 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:33:02.319 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.320 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.320 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26191.0:124554 6.8 6.b3d98118 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e555)", 2024-08-19T14:33:02.320 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:30:43.948816+0000", 2024-08-19T14:33:02.320 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 138.3512672, 2024-08-19T14:33:02.320 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.95529832699999995, 2024-08-19T14:33:02.320 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:33:02.320 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:33:02.320 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:33:02.320 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:33:02.320 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26191", 2024-08-19T14:33:02.320 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:33:02.320 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 124554 2024-08-19T14:33:02.320 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.321 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:33:02.321 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.321 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:33:02.321 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:43.948816+0000", 2024-08-19T14:33:02.321 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:33:02.321 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.321 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.321 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:33:02.321 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:43.948816+0000", 2024-08-19T14:33:02.321 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:33:02.321 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.321 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.321 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:33:02.321 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:43.948817+0000", 2024-08-19T14:33:02.321 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.252e-06 2024-08-19T14:33:02.322 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.322 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.322 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:33:02.322 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:43.948823+0000", 2024-08-19T14:33:02.322 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1309999999999997e-06 2024-08-19T14:33:02.322 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.322 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.322 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:33:02.322 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:43.948833+0000", 2024-08-19T14:33:02.322 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0886e-05 2024-08-19T14:33:02.322 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.322 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.322 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:33:02.322 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:43.948841+0000", 2024-08-19T14:33:02.323 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.8320000000000006e-06 2024-08-19T14:33:02.323 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.323 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.323 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:33:02.323 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:43.948915+0000", 2024-08-19T14:33:02.323 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3396e-05 2024-08-19T14:33:02.323 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.323 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.323 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:33:02.323 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:43.948924+0000", 2024-08-19T14:33:02.323 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.8740000000000007e-06 2024-08-19T14:33:02.323 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.323 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.324 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:33:02.324 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:44.904085+0000", 2024-08-19T14:33:02.324 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.95516005900000001 2024-08-19T14:33:02.324 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.324 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.324 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:33:02.324 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:44.904114+0000", 2024-08-19T14:33:02.324 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9896999999999999e-05 2024-08-19T14:33:02.324 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:33:02.324 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:33:02.324 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:33:02.324 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.324 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.324 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26191.0:124571 6.8 6.e5f0dfd8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e555)", 2024-08-19T14:33:02.325 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:30:43.990212+0000", 2024-08-19T14:33:02.325 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 138.309870897, 2024-08-19T14:33:02.325 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.91398457099999997, 2024-08-19T14:33:02.325 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:33:02.325 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:33:02.325 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:33:02.325 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:33:02.325 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26191", 2024-08-19T14:33:02.325 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:33:02.325 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 124571 2024-08-19T14:33:02.325 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.325 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:33:02.325 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.325 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:33:02.326 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:43.990212+0000", 2024-08-19T14:33:02.326 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:33:02.326 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.326 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.326 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:33:02.326 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:43.990212+0000", 2024-08-19T14:33:02.326 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:33:02.326 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.326 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.326 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:33:02.326 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:43.990214+0000", 2024-08-19T14:33:02.326 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.508e-06 2024-08-19T14:33:02.326 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.327 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.327 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:33:02.327 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:43.990221+0000", 2024-08-19T14:33:02.327 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9149999999999999e-06 2024-08-19T14:33:02.327 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.327 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.327 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:33:02.327 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:43.990236+0000", 2024-08-19T14:33:02.327 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4978e-05 2024-08-19T14:33:02.327 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.327 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.327 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:33:02.327 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:43.990244+0000", 2024-08-19T14:33:02.327 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.4479999999999994e-06 2024-08-19T14:33:02.328 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.328 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.328 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:33:02.328 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:43.990304+0000", 2024-08-19T14:33:02.328 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9757999999999999e-05 2024-08-19T14:33:02.328 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.328 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.328 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:33:02.328 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:43.990313+0000", 2024-08-19T14:33:02.328 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3710000000000006e-06 2024-08-19T14:33:02.328 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.328 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.328 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:33:02.328 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:44.904189+0000", 2024-08-19T14:33:02.329 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.91387590100000005 2024-08-19T14:33:02.329 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.329 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.329 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:33:02.329 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:44.904197+0000", 2024-08-19T14:33:02.329 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6920000000000002e-06 2024-08-19T14:33:02.329 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:33:02.329 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:33:02.329 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:33:02.329 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.329 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.329 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26191.0:124580 6.8 6.2fef8638 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e555)", 2024-08-19T14:33:02.329 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:30:44.009826+0000", 2024-08-19T14:33:02.329 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 138.290256838, 2024-08-19T14:33:02.330 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.89440363899999997, 2024-08-19T14:33:02.330 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:33:02.330 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:33:02.330 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:33:02.330 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:33:02.330 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26191", 2024-08-19T14:33:02.330 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:33:02.330 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 124580 2024-08-19T14:33:02.330 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.330 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:33:02.330 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.330 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:33:02.330 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:44.009826+0000", 2024-08-19T14:33:02.330 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:33:02.331 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.331 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.331 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:33:02.331 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:44.009826+0000", 2024-08-19T14:33:02.331 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:33:02.331 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.331 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.331 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:33:02.331 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:44.009828+0000", 2024-08-19T14:33:02.331 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6279999999999999e-06 2024-08-19T14:33:02.331 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.331 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.331 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:33:02.331 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:44.009839+0000", 2024-08-19T14:33:02.332 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1113e-05 2024-08-19T14:33:02.332 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.332 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.332 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:33:02.332 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:44.009856+0000", 2024-08-19T14:33:02.332 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6752000000000002e-05 2024-08-19T14:33:02.332 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.332 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.332 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:33:02.332 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:44.009865+0000", 2024-08-19T14:33:02.332 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.0869999999999996e-06 2024-08-19T14:33:02.332 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.332 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.333 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:33:02.333 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:44.009926+0000", 2024-08-19T14:33:02.333 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0501999999999998e-05 2024-08-19T14:33:02.333 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.333 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.333 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:33:02.333 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:44.009935+0000", 2024-08-19T14:33:02.334 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.9609999999999994e-06 2024-08-19T14:33:02.334 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.334 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.334 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:33:02.334 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:44.904224+0000", 2024-08-19T14:33:02.334 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.89428911799999999 2024-08-19T14:33:02.334 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.334 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.334 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:33:02.334 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:44.904230+0000", 2024-08-19T14:33:02.335 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4779999999999996e-06 2024-08-19T14:33:02.335 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:33:02.335 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:33:02.335 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:33:02.335 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.335 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.335 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26191.0:124594 6.8 6.3e87bdf8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e555)", 2024-08-19T14:33:02.335 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:30:44.038535+0000", 2024-08-19T14:33:02.335 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 138.261548175, 2024-08-19T14:33:02.335 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.86572874600000005, 2024-08-19T14:33:02.335 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:33:02.335 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:33:02.335 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:33:02.335 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:33:02.336 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26191", 2024-08-19T14:33:02.336 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:33:02.336 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 124594 2024-08-19T14:33:02.336 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.336 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:33:02.336 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.336 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:33:02.336 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:44.038535+0000", 2024-08-19T14:33:02.336 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:33:02.336 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.336 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.336 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:33:02.336 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:44.038535+0000", 2024-08-19T14:33:02.336 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:33:02.337 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.337 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.337 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:33:02.337 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:44.038537+0000", 2024-08-19T14:33:02.337 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4079999999999999e-06 2024-08-19T14:33:02.337 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.337 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.337 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:33:02.337 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:44.038541+0000", 2024-08-19T14:33:02.337 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.882e-06 2024-08-19T14:33:02.337 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.337 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.337 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:33:02.337 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:44.038553+0000", 2024-08-19T14:33:02.338 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2063e-05 2024-08-19T14:33:02.338 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.338 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.338 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:33:02.338 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:44.038561+0000", 2024-08-19T14:33:02.338 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.9359999999999999e-06 2024-08-19T14:33:02.338 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.338 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.338 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:33:02.338 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:44.038623+0000", 2024-08-19T14:33:02.338 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1766000000000005e-05 2024-08-19T14:33:02.338 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.338 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.339 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:33:02.339 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:44.038637+0000", 2024-08-19T14:33:02.339 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3651e-05 2024-08-19T14:33:02.339 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.339 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.339 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:33:02.339 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:44.904257+0000", 2024-08-19T14:33:02.339 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.86562061499999998 2024-08-19T14:33:02.339 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.339 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.339 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:33:02.339 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:44.904264+0000", 2024-08-19T14:33:02.339 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4250000000000003e-06 2024-08-19T14:33:02.340 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:33:02.340 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:33:02.340 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:33:02.340 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.340 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.340 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26191.0:124606 6.8 6.a7971958 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e555)", 2024-08-19T14:33:02.340 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:30:44.059903+0000", 2024-08-19T14:33:02.340 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 138.24018065600001, 2024-08-19T14:33:02.340 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.84439685900000006, 2024-08-19T14:33:02.340 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:33:02.340 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:33:02.340 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:33:02.340 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:33:02.340 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26191", 2024-08-19T14:33:02.341 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:33:02.341 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 124606 2024-08-19T14:33:02.341 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.341 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:33:02.341 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.341 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:33:02.341 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:44.059903+0000", 2024-08-19T14:33:02.341 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:33:02.341 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.342 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.342 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:33:02.342 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:44.059903+0000", 2024-08-19T14:33:02.342 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:33:02.342 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.342 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.342 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:33:02.342 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:44.059904+0000", 2024-08-19T14:33:02.342 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.488e-06 2024-08-19T14:33:02.342 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.342 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.342 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:33:02.343 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:44.059910+0000", 2024-08-19T14:33:02.343 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.947e-06 2024-08-19T14:33:02.343 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.343 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.343 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:33:02.343 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:44.059922+0000", 2024-08-19T14:33:02.343 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2381999999999999e-05 2024-08-19T14:33:02.343 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.343 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.343 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:33:02.343 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:44.059930+0000", 2024-08-19T14:33:02.343 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5089999999999996e-06 2024-08-19T14:33:02.343 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.343 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.344 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:33:02.344 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:44.060000+0000", 2024-08-19T14:33:02.344 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9993000000000006e-05 2024-08-19T14:33:02.344 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.344 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.344 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:33:02.344 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:44.060011+0000", 2024-08-19T14:33:02.344 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1469e-05 2024-08-19T14:33:02.344 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.344 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.344 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:33:02.344 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:44.904294+0000", 2024-08-19T14:33:02.344 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.84428247099999998 2024-08-19T14:33:02.345 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.345 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.345 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:33:02.345 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:44.904300+0000", 2024-08-19T14:33:02.345 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5999999999999997e-06 2024-08-19T14:33:02.345 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:33:02.345 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:33:02.345 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:33:02.345 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.345 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.345 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26191.0:124625 6.8 6.c84f22e8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e555)", 2024-08-19T14:33:02.345 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:30:44.166898+0000", 2024-08-19T14:33:02.345 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 138.13318558099999, 2024-08-19T14:33:02.345 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.738722553, 2024-08-19T14:33:02.346 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:33:02.346 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:33:02.346 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:33:02.346 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:33:02.346 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26191", 2024-08-19T14:33:02.346 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:33:02.346 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 124625 2024-08-19T14:33:02.346 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.346 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:33:02.346 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.346 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:33:02.346 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:44.166898+0000", 2024-08-19T14:33:02.346 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:33:02.347 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.347 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.347 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:33:02.347 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:44.166898+0000", 2024-08-19T14:33:02.347 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:33:02.347 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.347 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.347 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:33:02.347 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:44.166899+0000", 2024-08-19T14:33:02.347 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.536e-06 2024-08-19T14:33:02.347 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.347 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.347 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:33:02.347 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:44.166905+0000", 2024-08-19T14:33:02.348 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8900000000000004e-06 2024-08-19T14:33:02.348 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.348 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.348 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:33:02.348 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:44.166918+0000", 2024-08-19T14:33:02.348 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2658e-05 2024-08-19T14:33:02.348 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.348 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.348 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:33:02.348 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:44.166925+0000", 2024-08-19T14:33:02.348 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6199999999999999e-06 2024-08-19T14:33:02.348 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.348 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.348 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:33:02.348 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:44.166991+0000", 2024-08-19T14:33:02.349 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.5537000000000006e-05 2024-08-19T14:33:02.349 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.349 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.349 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:33:02.349 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:44.166999+0000", 2024-08-19T14:33:02.349 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.4999999999999999e-06 2024-08-19T14:33:02.349 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.349 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.349 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:33:02.349 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:44.905612+0000", 2024-08-19T14:33:02.349 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.73861242199999999 2024-08-19T14:33:02.349 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.349 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.349 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:33:02.350 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:44.905620+0000", 2024-08-19T14:33:02.350 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.3899999999999993e-06 2024-08-19T14:33:02.350 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:33:02.350 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:33:02.350 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:33:02.350 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.350 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.350 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26191.0:124553 6.3 6:c717ed58:::benchmark_data_smithi045_97937_object17280:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e556)", 2024-08-19T14:33:02.350 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:30:45.947971+0000", 2024-08-19T14:33:02.350 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 136.35211231400001, 2024-08-19T14:33:02.350 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.26965875099999997, 2024-08-19T14:33:02.350 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:33:02.350 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:33:02.351 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:33:02.351 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:33:02.351 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26191", 2024-08-19T14:33:02.351 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:33:02.351 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 124553 2024-08-19T14:33:02.351 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.351 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:33:02.351 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.351 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:33:02.351 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.947971+0000", 2024-08-19T14:33:02.351 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:33:02.351 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.351 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.352 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:33:02.352 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.947971+0000", 2024-08-19T14:33:02.352 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:33:02.352 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.352 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.352 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:33:02.352 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.947971+0000", 2024-08-19T14:33:02.352 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6300000000000001e-07 2024-08-19T14:33:02.352 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.352 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.352 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:33:02.352 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.947973+0000", 2024-08-19T14:33:02.352 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9819999999999998e-06 2024-08-19T14:33:02.352 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.352 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.353 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:33:02.353 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.947997+0000", 2024-08-19T14:33:02.353 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3612e-05 2024-08-19T14:33:02.353 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.353 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.353 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:33:02.353 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.948003+0000", 2024-08-19T14:33:02.353 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8409999999999996e-06 2024-08-19T14:33:02.353 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.353 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.353 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:33:02.353 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.951341+0000", 2024-08-19T14:33:02.353 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0033386259999999999 2024-08-19T14:33:02.353 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.354 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.354 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:33:02.354 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.951357+0000", 2024-08-19T14:33:02.354 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5523e-05 2024-08-19T14:33:02.354 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.354 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.354 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:33:02.354 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.181324+0000", 2024-08-19T14:33:02.354 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.22996752500000001 2024-08-19T14:33:02.354 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.354 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.354 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:33:02.355 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.181489+0000", 2024-08-19T14:33:02.355 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00016493700000000001 2024-08-19T14:33:02.355 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.355 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.355 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 8,11", 2024-08-19T14:33:02.355 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.181653+0000", 2024-08-19T14:33:02.355 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000163153 2024-08-19T14:33:02.355 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.355 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.355 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:02.355 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.188324+0000", 2024-08-19T14:33:02.355 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0066717970000000001 2024-08-19T14:33:02.355 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.355 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.355 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:33:02.356 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.188421+0000", 2024-08-19T14:33:02.356 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.6439000000000003e-05 2024-08-19T14:33:02.356 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.356 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.356 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:02.356 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.213274+0000", 2024-08-19T14:33:02.356 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.024852868 2024-08-19T14:33:02.356 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.356 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.356 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:33:02.356 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.213313+0000", 2024-08-19T14:33:02.356 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8840000000000001e-05 2024-08-19T14:33:02.356 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.356 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.357 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:33:02.357 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.217630+0000", 2024-08-19T14:33:02.357 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0043172449999999999 2024-08-19T14:33:02.357 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:33:02.357 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:33:02.357 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:33:02.357 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.357 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.357 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26191.0:124557 6.3 6:c9b3fc27:::benchmark_data_smithi045_97937_object17284:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e556)", 2024-08-19T14:33:02.357 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:30:45.948017+0000", 2024-08-19T14:33:02.357 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 136.352066251, 2024-08-19T14:33:02.357 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.27443830000000002, 2024-08-19T14:33:02.357 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:33:02.358 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:33:02.358 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:33:02.358 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:33:02.358 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26191", 2024-08-19T14:33:02.358 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:33:02.358 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 124557 2024-08-19T14:33:02.358 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.358 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:33:02.358 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.358 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:33:02.358 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.948017+0000", 2024-08-19T14:33:02.358 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:33:02.358 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.358 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.359 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:33:02.359 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.948017+0000", 2024-08-19T14:33:02.359 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:33:02.359 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.359 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.359 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:33:02.359 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.948017+0000", 2024-08-19T14:33:02.359 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1300000000000001e-07 2024-08-19T14:33:02.359 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.359 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.359 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:33:02.359 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.948020+0000", 2024-08-19T14:33:02.359 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1830000000000001e-06 2024-08-19T14:33:02.360 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.360 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.360 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:33:02.360 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.948024+0000", 2024-08-19T14:33:02.360 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9289999999999997e-06 2024-08-19T14:33:02.360 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.360 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.360 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:33:02.360 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.948028+0000", 2024-08-19T14:33:02.360 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.27e-06 2024-08-19T14:33:02.360 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.360 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.360 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:33:02.361 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.951394+0000", 2024-08-19T14:33:02.361 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0033660439999999999 2024-08-19T14:33:02.361 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.361 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.361 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:33:02.361 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.951400+0000", 2024-08-19T14:33:02.361 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1340000000000001e-06 2024-08-19T14:33:02.361 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.361 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.361 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:33:02.361 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.184815+0000", 2024-08-19T14:33:02.361 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.233415396 2024-08-19T14:33:02.361 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.362 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.362 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:33:02.362 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.184962+0000", 2024-08-19T14:33:02.362 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014632100000000001 2024-08-19T14:33:02.362 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.362 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.362 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 8,11", 2024-08-19T14:33:02.362 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.185077+0000", 2024-08-19T14:33:02.362 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00011516899999999999 2024-08-19T14:33:02.362 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.362 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.362 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:02.362 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.188403+0000", 2024-08-19T14:33:02.362 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0033262650000000001 2024-08-19T14:33:02.363 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.363 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.363 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:33:02.363 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.201815+0000", 2024-08-19T14:33:02.363 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.013412145 2024-08-19T14:33:02.363 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.363 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.363 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:02.363 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.217734+0000", 2024-08-19T14:33:02.363 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.015918993999999999 2024-08-19T14:33:02.363 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.363 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.363 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:33:02.364 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.217786+0000", 2024-08-19T14:33:02.364 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2135000000000001e-05 2024-08-19T14:33:02.364 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.364 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.364 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:33:02.364 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.222455+0000", 2024-08-19T14:33:02.364 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0046690020000000002 2024-08-19T14:33:02.364 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:33:02.364 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:33:02.364 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:33:02.364 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.364 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.364 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26191.0:124569 6.3 6:cd06ef03:::benchmark_data_smithi045_97937_object17296:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e556)", 2024-08-19T14:33:02.365 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:30:45.948032+0000", 2024-08-19T14:33:02.365 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 136.352051348, 2024-08-19T14:33:02.365 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.30062119799999998, 2024-08-19T14:33:02.365 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:33:02.365 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:33:02.365 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:33:02.365 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:33:02.365 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26191", 2024-08-19T14:33:02.365 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:33:02.365 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 124569 2024-08-19T14:33:02.365 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.365 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:33:02.366 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.366 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:33:02.366 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.948032+0000", 2024-08-19T14:33:02.366 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:33:02.366 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.366 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.366 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:33:02.366 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.948032+0000", 2024-08-19T14:33:02.366 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:33:02.366 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.366 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.366 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:33:02.366 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.948032+0000", 2024-08-19T14:33:02.367 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.48e-07 2024-08-19T14:33:02.367 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.367 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.367 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:33:02.367 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.948034+0000", 2024-08-19T14:33:02.367 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.81e-06 2024-08-19T14:33:02.367 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.367 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.367 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:33:02.367 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.948039+0000", 2024-08-19T14:33:02.367 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7609999999999998e-06 2024-08-19T14:33:02.367 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.367 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.368 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:33:02.368 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.948041+0000", 2024-08-19T14:33:02.368 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.649e-06 2024-08-19T14:33:02.368 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.368 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.368 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:33:02.368 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.951441+0000", 2024-08-19T14:33:02.368 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0033990600000000002 2024-08-19T14:33:02.368 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.368 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.368 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:33:02.368 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.951448+0000", 2024-08-19T14:33:02.369 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2779999999999996e-06 2024-08-19T14:33:02.369 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.369 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.369 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:33:02.369 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.188462+0000", 2024-08-19T14:33:02.369 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.23701393500000001 2024-08-19T14:33:02.369 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.369 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.369 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:33:02.369 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.188628+0000", 2024-08-19T14:33:02.369 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00016646099999999999 2024-08-19T14:33:02.369 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.369 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.369 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 8,11", 2024-08-19T14:33:02.370 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.188795+0000", 2024-08-19T14:33:02.370 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000166551 2024-08-19T14:33:02.370 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.370 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.370 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:02.370 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.197612+0000", 2024-08-19T14:33:02.370 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0088169370000000004 2024-08-19T14:33:02.370 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.370 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.370 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:33:02.370 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.208975+0000", 2024-08-19T14:33:02.370 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011363836 2024-08-19T14:33:02.370 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.371 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.371 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:02.371 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.240706+0000", 2024-08-19T14:33:02.371 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.031730534999999997 2024-08-19T14:33:02.371 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.371 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.371 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:33:02.371 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.240742+0000", 2024-08-19T14:33:02.371 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5945000000000002e-05 2024-08-19T14:33:02.371 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.371 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.371 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:33:02.371 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.248653+0000", 2024-08-19T14:33:02.371 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0079111919999999992 2024-08-19T14:33:02.372 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:33:02.372 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:33:02.372 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:33:02.372 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.372 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.372 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26191.0:124584 6.3 6:c2228c7c:::benchmark_data_smithi045_97937_object17311:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e556)", 2024-08-19T14:33:02.372 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:30:45.948045+0000", 2024-08-19T14:33:02.372 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 136.35203871799999, 2024-08-19T14:33:02.372 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.305624848, 2024-08-19T14:33:02.372 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:33:02.372 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:33:02.372 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:33:02.372 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:33:02.373 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26191", 2024-08-19T14:33:02.373 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:33:02.373 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 124584 2024-08-19T14:33:02.373 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.373 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:33:02.373 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.373 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:33:02.373 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.948045+0000", 2024-08-19T14:33:02.373 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:33:02.373 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.373 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.373 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:33:02.373 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.948045+0000", 2024-08-19T14:33:02.373 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:33:02.374 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.374 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.374 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:33:02.374 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.948045+0000", 2024-08-19T14:33:02.374 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1e-07 2024-08-19T14:33:02.374 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.374 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.374 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:33:02.374 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.948046+0000", 2024-08-19T14:33:02.374 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.248e-06 2024-08-19T14:33:02.374 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.374 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.374 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:33:02.375 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.948052+0000", 2024-08-19T14:33:02.375 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2809999999999997e-06 2024-08-19T14:33:02.375 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.375 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.375 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:33:02.375 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.948057+0000", 2024-08-19T14:33:02.375 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3270000000000002e-06 2024-08-19T14:33:02.375 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.375 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.375 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:33:02.375 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.951478+0000", 2024-08-19T14:33:02.375 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0034215280000000001 2024-08-19T14:33:02.375 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.375 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.375 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:33:02.376 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.951484+0000", 2024-08-19T14:33:02.376 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8839999999999999e-06 2024-08-19T14:33:02.376 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.376 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.376 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:33:02.376 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.192069+0000", 2024-08-19T14:33:02.376 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.24058475700000001 2024-08-19T14:33:02.376 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.376 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.376 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:33:02.376 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.192250+0000", 2024-08-19T14:33:02.376 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00018143200000000001 2024-08-19T14:33:02.376 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.376 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.377 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 8,11", 2024-08-19T14:33:02.377 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.192415+0000", 2024-08-19T14:33:02.377 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00016437299999999999 2024-08-19T14:33:02.377 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.377 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.377 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:02.377 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.197682+0000", 2024-08-19T14:33:02.377 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:33:02.379 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.005266967 2024-08-19T14:33:02.379 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.379 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.379 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:33:02.379 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.209047+0000", 2024-08-19T14:33:02.379 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011365488 2024-08-19T14:33:02.379 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.379 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.380 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:02.380 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.248780+0000", 2024-08-19T14:33:02.380 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.039732827999999998 2024-08-19T14:33:02.380 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.380 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.380 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:33:02.380 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.248935+0000", 2024-08-19T14:33:02.380 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00015519000000000001 2024-08-19T14:33:02.380 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.380 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.380 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:33:02.380 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.253669+0000", 2024-08-19T14:33:02.380 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0047343350000000001 2024-08-19T14:33:02.381 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:33:02.381 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:33:02.381 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:33:02.381 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.381 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.381 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26191.0:124595 6.3 6:c8af054e:::benchmark_data_smithi045_97937_object17322:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e556)", 2024-08-19T14:33:02.381 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:30:45.948063+0000", 2024-08-19T14:33:02.381 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 136.35202033799999, 2024-08-19T14:33:02.381 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.310528945, 2024-08-19T14:33:02.381 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:33:02.381 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:33:02.381 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:33:02.382 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:33:02.382 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26191", 2024-08-19T14:33:02.382 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:33:02.382 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 124595 2024-08-19T14:33:02.382 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.382 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:33:02.382 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.382 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:33:02.382 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.948063+0000", 2024-08-19T14:33:02.382 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:33:02.382 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.382 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.383 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:33:02.383 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.948063+0000", 2024-08-19T14:33:02.383 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:33:02.383 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.383 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.383 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:33:02.383 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.948063+0000", 2024-08-19T14:33:02.383 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.77e-07 2024-08-19T14:33:02.383 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.383 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.383 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:33:02.383 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.948066+0000", 2024-08-19T14:33:02.383 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2029999999999999e-06 2024-08-19T14:33:02.383 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.383 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.384 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:33:02.384 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.948073+0000", 2024-08-19T14:33:02.384 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3270000000000003e-06 2024-08-19T14:33:02.384 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.384 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.384 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:33:02.384 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.948078+0000", 2024-08-19T14:33:02.384 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.95e-06 2024-08-19T14:33:02.384 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.384 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.384 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:33:02.384 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.951524+0000", 2024-08-19T14:33:02.384 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0034457720000000002 2024-08-19T14:33:02.385 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.385 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.385 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:33:02.385 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.951531+0000", 2024-08-19T14:33:02.385 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4880000000000001e-06 2024-08-19T14:33:02.385 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.385 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.385 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:33:02.385 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.197745+0000", 2024-08-19T14:33:02.385 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.246213662 2024-08-19T14:33:02.385 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.385 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.385 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:33:02.385 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.197918+0000", 2024-08-19T14:33:02.386 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00017316099999999999 2024-08-19T14:33:02.386 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.386 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.386 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 8,11", 2024-08-19T14:33:02.386 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.198094+0000", 2024-08-19T14:33:02.386 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000176379 2024-08-19T14:33:02.386 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.386 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.386 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:02.386 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.209024+0000", 2024-08-19T14:33:02.386 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010929578000000001 2024-08-19T14:33:02.386 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.386 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.387 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:33:02.387 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.213164+0000", 2024-08-19T14:33:02.387 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0041403109999999998 2024-08-19T14:33:02.387 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.387 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.387 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:02.387 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.253794+0000", 2024-08-19T14:33:02.387 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.040630219000000002 2024-08-19T14:33:02.387 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.387 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.387 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:33:02.387 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.253832+0000", 2024-08-19T14:33:02.387 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7249999999999997e-05 2024-08-19T14:33:02.387 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.387 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.388 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:33:02.388 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.258592+0000", 2024-08-19T14:33:02.388 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0047602679999999998 2024-08-19T14:33:02.388 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:33:02.388 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:33:02.388 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:33:02.388 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.388 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.388 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26191.0:124601 6.3 6:ccebfe95:::benchmark_data_smithi045_97937_object17328:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e556)", 2024-08-19T14:33:02.388 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:30:45.948088+0000", 2024-08-19T14:33:02.388 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 136.35199525600001, 2024-08-19T14:33:02.388 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.31743455700000001, 2024-08-19T14:33:02.388 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:33:02.389 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:33:02.389 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:33:02.389 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:33:02.389 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26191", 2024-08-19T14:33:02.389 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:33:02.389 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 124601 2024-08-19T14:33:02.389 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.389 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:33:02.389 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.389 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:33:02.389 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.948088+0000", 2024-08-19T14:33:02.389 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:33:02.389 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.389 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.390 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:33:02.390 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.948088+0000", 2024-08-19T14:33:02.390 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:33:02.390 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.390 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.390 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:33:02.390 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.948089+0000", 2024-08-19T14:33:02.390 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3500000000000002e-07 2024-08-19T14:33:02.390 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.390 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.390 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:33:02.390 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.948092+0000", 2024-08-19T14:33:02.390 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.202e-06 2024-08-19T14:33:02.390 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.391 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.391 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:33:02.391 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.948100+0000", 2024-08-19T14:33:02.391 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.3389999999999993e-06 2024-08-19T14:33:02.391 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.391 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.391 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:33:02.391 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.948106+0000", 2024-08-19T14:33:02.391 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4879999999999998e-06 2024-08-19T14:33:02.391 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.391 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.391 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:33:02.391 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.951597+0000", 2024-08-19T14:33:02.391 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.003491671 2024-08-19T14:33:02.392 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.392 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.392 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:33:02.392 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.951605+0000", 2024-08-19T14:33:02.392 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3000000000000004e-06 2024-08-19T14:33:02.392 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.392 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.392 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:33:02.392 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.201868+0000", 2024-08-19T14:33:02.392 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.25026306300000001 2024-08-19T14:33:02.392 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.392 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.392 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:33:02.392 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.202011+0000", 2024-08-19T14:33:02.393 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014307999999999999 2024-08-19T14:33:02.393 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.393 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.393 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 8,11", 2024-08-19T14:33:02.393 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.202106+0000", 2024-08-19T14:33:02.393 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.5429999999999999e-05 2024-08-19T14:33:02.393 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.393 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.393 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:02.393 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.209098+0000", 2024-08-19T14:33:02.393 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0069915450000000001 2024-08-19T14:33:02.393 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.393 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.394 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:33:02.394 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.213170+0000", 2024-08-19T14:33:02.394 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0040719069999999996 2024-08-19T14:33:02.394 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.394 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.394 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:02.394 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.258696+0000", 2024-08-19T14:33:02.394 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.045526509999999999 2024-08-19T14:33:02.394 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.394 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.394 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:33:02.394 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.258794+0000", 2024-08-19T14:33:02.394 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.7588000000000001e-05 2024-08-19T14:33:02.395 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.395 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.395 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:33:02.395 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.265523+0000", 2024-08-19T14:33:02.395 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0067289990000000003 2024-08-19T14:33:02.395 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:33:02.395 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:33:02.395 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:33:02.395 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.395 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.395 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26191.0:124603 6.3 6:cbb85d3f:::benchmark_data_smithi045_97937_object17330:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e556)", 2024-08-19T14:33:02.395 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:30:45.948113+0000", 2024-08-19T14:33:02.395 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 136.35197055899999, 2024-08-19T14:33:02.395 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.32293990299999997, 2024-08-19T14:33:02.396 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:33:02.396 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:33:02.396 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:33:02.396 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:33:02.396 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26191", 2024-08-19T14:33:02.396 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:33:02.396 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 124603 2024-08-19T14:33:02.396 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.396 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:33:02.396 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.396 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:33:02.396 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.948113+0000", 2024-08-19T14:33:02.396 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:33:02.396 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.397 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.397 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:33:02.397 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.948113+0000", 2024-08-19T14:33:02.397 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:33:02.397 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.397 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.397 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:33:02.397 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.948113+0000", 2024-08-19T14:33:02.397 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2e-07 2024-08-19T14:33:02.397 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.397 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.397 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:33:02.397 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.948116+0000", 2024-08-19T14:33:02.397 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4980000000000001e-06 2024-08-19T14:33:02.398 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.398 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.398 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:33:02.398 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.948122+0000", 2024-08-19T14:33:02.398 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.297e-06 2024-08-19T14:33:02.398 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.398 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.398 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:33:02.398 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.948127+0000", 2024-08-19T14:33:02.398 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1499999999999998e-06 2024-08-19T14:33:02.398 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.398 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.399 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:33:02.399 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.951637+0000", 2024-08-19T14:33:02.399 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0035102319999999998 2024-08-19T14:33:02.399 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.399 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.399 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:33:02.399 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.951645+0000", 2024-08-19T14:33:02.399 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.165e-06 2024-08-19T14:33:02.399 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.399 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.399 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:33:02.399 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.205514+0000", 2024-08-19T14:33:02.399 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.25386919899999999 2024-08-19T14:33:02.399 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.399 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.400 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:33:02.400 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.205684+0000", 2024-08-19T14:33:02.400 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00017037399999999999 2024-08-19T14:33:02.400 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.400 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.400 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 8,11", 2024-08-19T14:33:02.400 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.205849+0000", 2024-08-19T14:33:02.400 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00016521400000000001 2024-08-19T14:33:02.400 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.400 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.400 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:02.400 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.213228+0000", 2024-08-19T14:33:02.400 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0073782439999999999 2024-08-19T14:33:02.401 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.401 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.401 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:33:02.401 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.217650+0000", 2024-08-19T14:33:02.401 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0044225009999999997 2024-08-19T14:33:02.401 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.401 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.401 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:02.401 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.265586+0000", 2024-08-19T14:33:02.401 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.047936421 2024-08-19T14:33:02.401 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.401 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.401 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:33:02.402 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.265693+0000", 2024-08-19T14:33:02.402 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00010662099999999999 2024-08-19T14:33:02.402 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.402 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.402 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:33:02.402 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.271053+0000", 2024-08-19T14:33:02.402 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.005359567 2024-08-19T14:33:02.402 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:33:02.402 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:33:02.402 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:33:02.402 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.402 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.402 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26191.0:124622 6.3 6:cf0d0516:::benchmark_data_smithi045_97937_object17349:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e556)", 2024-08-19T14:33:02.403 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:30:45.948133+0000", 2024-08-19T14:33:02.403 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 136.351950217, 2024-08-19T14:33:02.403 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.327898566, 2024-08-19T14:33:02.403 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:33:02.403 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:33:02.403 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:33:02.403 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:33:02.403 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26191", 2024-08-19T14:33:02.403 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:33:02.403 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 124622 2024-08-19T14:33:02.403 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.403 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:33:02.403 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.403 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:33:02.404 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.948133+0000", 2024-08-19T14:33:02.404 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:33:02.404 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.404 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.404 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:33:02.404 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.948133+0000", 2024-08-19T14:33:02.404 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:33:02.404 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.404 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.404 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:33:02.404 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.948133+0000", 2024-08-19T14:33:02.404 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.84e-07 2024-08-19T14:33:02.404 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.404 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.405 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:33:02.405 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.948136+0000", 2024-08-19T14:33:02.405 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3140000000000002e-06 2024-08-19T14:33:02.405 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.405 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.405 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:33:02.405 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.948142+0000", 2024-08-19T14:33:02.405 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9279999999999998e-06 2024-08-19T14:33:02.405 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.405 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.405 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:33:02.405 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.948148+0000", 2024-08-19T14:33:02.405 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8109999999999999e-06 2024-08-19T14:33:02.405 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.406 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.406 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:33:02.406 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.951687+0000", 2024-08-19T14:33:02.406 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0035393669999999999 2024-08-19T14:33:02.406 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.406 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.406 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:33:02.406 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.951697+0000", 2024-08-19T14:33:02.406 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0471e-05 2024-08-19T14:33:02.406 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.406 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.406 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:33:02.406 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.209143+0000", 2024-08-19T14:33:02.407 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.257445596 2024-08-19T14:33:02.407 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.407 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.407 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:33:02.407 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.209302+0000", 2024-08-19T14:33:02.407 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00015897799999999999 2024-08-19T14:33:02.407 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.407 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.407 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 8,11", 2024-08-19T14:33:02.407 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.209465+0000", 2024-08-19T14:33:02.407 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00016353400000000001 2024-08-19T14:33:02.407 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.407 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.408 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:02.408 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.222533+0000", 2024-08-19T14:33:02.408 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.013067466999999999 2024-08-19T14:33:02.408 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.408 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.408 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:33:02.408 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.222549+0000", 2024-08-19T14:33:02.408 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6079999999999999e-05 2024-08-19T14:33:02.408 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.408 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.408 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:02.408 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.271113+0000", 2024-08-19T14:33:02.408 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.048563518999999999 2024-08-19T14:33:02.408 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.408 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.409 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:33:02.409 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.271178+0000", 2024-08-19T14:33:02.409 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.5464999999999997e-05 2024-08-19T14:33:02.409 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.409 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.409 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:33:02.409 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.276032+0000", 2024-08-19T14:33:02.409 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.004853652 2024-08-19T14:33:02.409 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:33:02.409 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:33:02.409 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:33:02.409 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.409 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.409 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26191.0:124632 6.3 6:c97ec9d6:::benchmark_data_smithi045_97937_object17359:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e556)", 2024-08-19T14:33:02.410 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:30:45.948155+0000", 2024-08-19T14:33:02.410 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 136.35192867999999, 2024-08-19T14:33:02.410 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.33379947399999998, 2024-08-19T14:33:02.410 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:33:02.410 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:33:02.410 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:33:02.410 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:33:02.410 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26191", 2024-08-19T14:33:02.410 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:33:02.410 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 124632 2024-08-19T14:33:02.410 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.410 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:33:02.410 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.411 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:33:02.411 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.948155+0000", 2024-08-19T14:33:02.411 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:33:02.411 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.411 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.411 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:33:02.411 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.948155+0000", 2024-08-19T14:33:02.411 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:33:02.411 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.411 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.411 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:33:02.411 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.948155+0000", 2024-08-19T14:33:02.411 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9799999999999999e-07 2024-08-19T14:33:02.412 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.412 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.412 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:33:02.412 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.948157+0000", 2024-08-19T14:33:02.412 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2730000000000001e-06 2024-08-19T14:33:02.412 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.412 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.412 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:33:02.412 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.948164+0000", 2024-08-19T14:33:02.412 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3300000000000004e-06 2024-08-19T14:33:02.412 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.412 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.412 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:33:02.412 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.948168+0000", 2024-08-19T14:33:02.413 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8099999999999997e-06 2024-08-19T14:33:02.413 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.413 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.413 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:33:02.413 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.951734+0000", 2024-08-19T14:33:02.413 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.003565085 2024-08-19T14:33:02.413 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.413 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.413 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:33:02.413 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.951740+0000", 2024-08-19T14:33:02.413 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1770000000000003e-06 2024-08-19T14:33:02.413 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.413 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.414 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:33:02.414 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.227097+0000", 2024-08-19T14:33:02.414 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.27535768399999999 2024-08-19T14:33:02.414 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.414 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.414 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:33:02.414 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.227257+0000", 2024-08-19T14:33:02.414 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00015980399999999999 2024-08-19T14:33:02.414 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.414 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.414 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 8,11", 2024-08-19T14:33:02.414 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.227420+0000", 2024-08-19T14:33:02.414 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00016232599999999999 2024-08-19T14:33:02.415 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.415 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.415 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:02.415 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.253743+0000", 2024-08-19T14:33:02.415 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.026323053999999999 2024-08-19T14:33:02.415 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.415 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.415 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:33:02.415 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.253755+0000", 2024-08-19T14:33:02.415 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2296000000000001e-05 2024-08-19T14:33:02.415 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.415 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.415 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:02.416 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.276102+0000", 2024-08-19T14:33:02.416 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.022346774999999999 2024-08-19T14:33:02.416 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.416 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.416 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:33:02.416 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.276358+0000", 2024-08-19T14:33:02.416 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000256245 2024-08-19T14:33:02.416 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.416 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.416 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:33:02.416 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.281954+0000", 2024-08-19T14:33:02.416 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0055962169999999997 2024-08-19T14:33:02.416 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:33:02.416 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:33:02.417 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:33:02.417 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.417 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.417 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26191.0:124625 6.8 6:1744f213:::benchmark_data_smithi045_97937_object17352:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e556)", 2024-08-19T14:33:02.417 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:30:45.951475+0000", 2024-08-19T14:33:02.417 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 136.34860804300001, 2024-08-19T14:33:02.417 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.25573605399999999, 2024-08-19T14:33:02.417 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:33:02.417 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:33:02.417 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:33:02.417 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:33:02.417 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26191", 2024-08-19T14:33:02.417 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:33:02.418 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 124625 2024-08-19T14:33:02.418 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.418 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:33:02.418 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.418 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:33:02.418 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.951475+0000", 2024-08-19T14:33:02.418 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:33:02.418 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.418 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.418 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:33:02.418 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.951475+0000", 2024-08-19T14:33:02.418 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:33:02.418 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.418 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.419 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:33:02.419 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.951476+0000", 2024-08-19T14:33:02.419 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3299999999999998e-07 2024-08-19T14:33:02.419 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.419 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.419 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:33:02.419 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.951477+0000", 2024-08-19T14:33:02.419 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8810000000000001e-06 2024-08-19T14:33:02.419 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.419 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.419 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:33:02.419 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.951483+0000", 2024-08-19T14:33:02.419 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.959e-06 2024-08-19T14:33:02.419 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.420 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.420 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:33:02.420 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.951488+0000", 2024-08-19T14:33:02.420 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2390000000000004e-06 2024-08-19T14:33:02.420 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.420 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.420 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:33:02.420 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.956183+0000", 2024-08-19T14:33:02.420 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0046949189999999997 2024-08-19T14:33:02.420 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.420 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.420 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:33:02.420 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.956189+0000", 2024-08-19T14:33:02.420 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.5590000000000002e-06 2024-08-19T14:33:02.421 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.421 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.421 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:33:02.421 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.172962+0000", 2024-08-19T14:33:02.421 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.216772927 2024-08-19T14:33:02.421 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.421 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.421 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:33:02.422 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.173125+0000", 2024-08-19T14:33:02.422 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00016332400000000001 2024-08-19T14:33:02.422 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.422 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.422 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 3,11", 2024-08-19T14:33:02.423 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.173285+0000", 2024-08-19T14:33:02.423 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00015976799999999999 2024-08-19T14:33:02.423 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.423 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.423 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:33:02.423 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.176919+0000", 2024-08-19T14:33:02.423 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0036333310000000001 2024-08-19T14:33:02.423 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.423 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.423 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:02.423 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.194170+0000", 2024-08-19T14:33:02.423 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.017251635000000001 2024-08-19T14:33:02.424 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.424 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.424 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:02.424 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.203256+0000", 2024-08-19T14:33:02.424 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.009086261 2024-08-19T14:33:02.424 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.424 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.424 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:33:02.424 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.203294+0000", 2024-08-19T14:33:02.424 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7592999999999997e-05 2024-08-19T14:33:02.424 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.424 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.425 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:33:02.425 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.207211+0000", 2024-08-19T14:33:02.425 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0039173250000000001 2024-08-19T14:33:02.425 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:33:02.425 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:33:02.425 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:33:02.425 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.425 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.425 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26191.0:168503 6.8 6:10e97d8a:::benchmark_data_smithi045_97937_object61230:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e577)", 2024-08-19T14:33:02.425 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:32:04.492777+0000", 2024-08-19T14:33:02.425 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 57.807305990000003, 2024-08-19T14:33:02.425 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1376310749999998, 2024-08-19T14:33:02.425 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:33:02.426 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:33:02.426 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:33:02.426 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:33:02.426 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26191", 2024-08-19T14:33:02.426 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:33:02.426 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 168503 2024-08-19T14:33:02.426 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.426 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:33:02.426 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.426 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:33:02.426 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:04.492777+0000", 2024-08-19T14:33:02.426 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:33:02.426 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.426 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.427 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:33:02.427 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:04.492777+0000", 2024-08-19T14:33:02.427 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:33:02.427 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.427 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.427 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:33:02.427 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:04.492778+0000", 2024-08-19T14:33:02.427 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7300000000000002e-07 2024-08-19T14:33:02.427 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.427 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.427 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:33:02.427 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:04.492780+0000", 2024-08-19T14:33:02.427 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9199999999999998e-06 2024-08-19T14:33:02.427 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.428 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.428 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:33:02.428 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:04.492786+0000", 2024-08-19T14:33:02.428 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.302e-06 2024-08-19T14:33:02.428 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.428 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.428 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:33:02.428 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:04.492791+0000", 2024-08-19T14:33:02.428 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.69e-06 2024-08-19T14:33:02.428 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.428 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.428 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:33:02.428 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:04.494902+0000", 2024-08-19T14:33:02.429 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0021112470000000001 2024-08-19T14:33:02.429 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.429 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.429 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for missing object", 2024-08-19T14:33:02.429 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:04.494953+0000", 2024-08-19T14:33:02.429 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1008999999999999e-05 2024-08-19T14:33:02.429 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.429 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.429 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:33:02.429 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:07.630397+0000", 2024-08-19T14:33:02.429 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1354437609999999 2024-08-19T14:33:02.429 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.429 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.429 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:33:02.430 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:07.630408+0000", 2024-08-19T14:33:02.430 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1873e-05 2024-08-19T14:33:02.430 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:33:02.430 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:33:02.430 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:33:02.430 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.430 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.430 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26191.0:168508 6.8 6:164fb7a4:::benchmark_data_smithi045_97937_object61235:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e577)", 2024-08-19T14:33:02.430 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:32:04.492815+0000", 2024-08-19T14:33:02.430 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 57.807268524999998, 2024-08-19T14:33:02.430 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1427530990000001, 2024-08-19T14:33:02.430 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:33:02.430 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:33:02.430 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:33:02.431 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:33:02.431 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26191", 2024-08-19T14:33:02.431 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:33:02.431 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 168508 2024-08-19T14:33:02.431 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.431 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:33:02.431 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.431 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:33:02.431 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:04.492815+0000", 2024-08-19T14:33:02.431 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:33:02.431 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.431 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.431 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:33:02.431 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:04.492815+0000", 2024-08-19T14:33:02.432 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:33:02.432 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.432 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.432 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:33:02.432 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:04.492815+0000", 2024-08-19T14:33:02.432 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9799999999999999e-07 2024-08-19T14:33:02.432 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.432 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.432 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:33:02.432 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:04.492820+0000", 2024-08-19T14:33:02.432 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6750000000000001e-06 2024-08-19T14:33:02.432 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.432 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.432 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:33:02.433 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:04.492826+0000", 2024-08-19T14:33:02.433 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3380000000000001e-06 2024-08-19T14:33:02.433 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.433 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.433 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:33:02.433 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:04.492831+0000", 2024-08-19T14:33:02.433 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4619999999999998e-06 2024-08-19T14:33:02.433 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.433 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.433 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:33:02.433 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:04.495042+0000", 2024-08-19T14:33:02.433 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0022109550000000001 2024-08-19T14:33:02.433 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.433 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.434 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for missing object", 2024-08-19T14:33:02.434 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:04.495088+0000", 2024-08-19T14:33:02.434 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6928999999999997e-05 2024-08-19T14:33:02.434 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.434 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.434 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:33:02.434 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:07.635556+0000", 2024-08-19T14:33:02.434 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1404670929999998 2024-08-19T14:33:02.434 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:33:02.434 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:33:02.434 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:33:02.434 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:07.635568+0000", 2024-08-19T14:33:02.435 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2349e-05 2024-08-19T14:33:02.435 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:33:02.435 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:33:02.435 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:33:02.435 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:33:02.435 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:33:02.435 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:33:02.436 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:02 smithi032 ceph-mon[20004]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:33:02.436 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:02 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1135606665' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:33:02.436 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:02 smithi032 ceph-mon[20004]: osdmap e591: 12 total, 11 up, 9 in 2024-08-19T14:33:02.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:02.437 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:02.437 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:33:02.538 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:33:02.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:02 smithi045 ceph-mon[20143]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:33:02.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:02 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1135606665' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:33:02.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:02 smithi045 ceph-mon[20143]: osdmap e591: 12 total, 11 up, 9 in 2024-08-19T14:33:02.560 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:02.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:02.561 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:33:02.593 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:02 smithi031 ceph-mon[16336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:33:02.594 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:02 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1135606665' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:33:02.594 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:02 smithi031 ceph-mon[16336]: osdmap e591: 12 total, 11 up, 9 in 2024-08-19T14:33:02.661 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:33:02.695 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:02.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:02.696 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:33:02.797 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:33:02.831 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:02.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:02.832 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:33:02.933 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:33:02.963 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:02.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:02.964 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:33:03.065 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:33:03.097 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:03.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:03.098 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:33:03.199 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:33:03.244 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:03.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:03.246 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:33:03.270 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:33:03.270 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:33:03.300 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:33:03.300 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:33:03.300 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-08-19T14:33:03.333 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:33:03.333 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:33:03.346 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:33:03.361 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:33:03.361 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:33:03.361 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:03 smithi031 ceph-mon[16336]: pgmap v1385: 47 pgs: 5 active+clean+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 525 MiB data, 11 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 747 op/s; 5862/25197 objects degraded (23.265%); 7218/25197 objects misplaced (28.646%); 11 MiB/s, 185 objects/s recovering 2024-08-19T14:33:03.362 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:03 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:33:03.362 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:03 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:33:03.362 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:03 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 5862/25197 objects degraded (23.265%), 7 pgs degraded, 16 pgs undersized (PG_DEGRADED) 2024-08-19T14:33:03.374 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:03.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:03.375 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:33:03.393 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:33:03.393 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:33:03.429 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:33:03.429 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:33:03.434 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:03 smithi032 ceph-mon[20004]: pgmap v1385: 47 pgs: 5 active+clean+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 525 MiB data, 11 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 747 op/s; 5862/25197 objects degraded (23.265%); 7218/25197 objects misplaced (28.646%); 11 MiB/s, 185 objects/s recovering 2024-08-19T14:33:03.434 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:03 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:33:03.434 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:03 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:33:03.434 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:03 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 5862/25197 objects degraded (23.265%), 7 pgs degraded, 16 pgs undersized (PG_DEGRADED) 2024-08-19T14:33:03.456 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:33:03.456 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:33:03.475 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:33:03.476 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-08-19T14:33:03.483 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:33:03.484 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:33:03.501 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:03.502 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:03 smithi045 ceph-mon[20143]: pgmap v1385: 47 pgs: 5 active+clean+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 525 MiB data, 11 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 747 op/s; 5862/25197 objects degraded (23.265%); 7218/25197 objects misplaced (28.646%); 11 MiB/s, 185 objects/s recovering 2024-08-19T14:33:03.502 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:03 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:33:03.502 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:03 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:33:03.502 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:03 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 5862/25197 objects degraded (23.265%), 7 pgs degraded, 16 pgs undersized (PG_DEGRADED) 2024-08-19T14:33:03.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:03.503 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:33:03.513 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:33:03.513 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:33:03.544 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:33:03.545 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:33:03.580 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:33:03.581 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:33:03.604 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:33:03.641 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:03.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:03.642 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:33:03.743 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:33:03.771 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:03.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:03.772 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:33:03.780 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:33:03.873 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:33:03.894 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:03.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:03.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:33:03.996 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:33:04.019 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:04.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:04.020 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:33:04.121 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:33:04.157 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:04.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:04.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:33:04.260 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:33:04.294 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:04.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:04.295 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:33:04.353 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:04 smithi031 ceph-mon[16336]: osdmap e592: 12 total, 11 up, 9 in 2024-08-19T14:33:04.353 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:04 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2693766011' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:33:04.396 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:33:04.397 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-19T14:33:04.398 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-19T14:33:04.420 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:04.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:04.421 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:33:04.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:04 smithi032 ceph-mon[20004]: osdmap e592: 12 total, 11 up, 9 in 2024-08-19T14:33:04.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:04 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2693766011' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:33:04.522 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:33:04.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:04 smithi045 ceph-mon[20143]: osdmap e592: 12 total, 11 up, 9 in 2024-08-19T14:33:04.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:04 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2693766011' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:33:04.555 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:04.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:04.556 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:33:04.558 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:33:04.558 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:33:04.657 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:33:04.683 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:04.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:04.685 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:33:04.745 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 7, 8, 2, 11] out_osds: [10, 6, 9] dead_osds: [2] live_osds: [0, 1, 3, 4, 5, 6, 7, 8, 9, 11, 10] 2024-08-19T14:33:04.745 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-19T14:33:04.745 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-08-19T14:33:04.745 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph tell osd.0 'trim stale osdmaps' 2024-08-19T14:33:04.786 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:33:04.814 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:04.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:04.816 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:33:04.916 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:33:04.958 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:04.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:04.959 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:33:05.060 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:33:05.096 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:05.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:05.097 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:33:05.181 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:33:05.198 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:33:05.225 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:05.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:05.227 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:33:05.328 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:33:05.363 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:05.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:05.364 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:33:05.465 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:33:05.491 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:05.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:05.492 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:33:05.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:05 smithi032 ceph-mon[20004]: pgmap v1387: 47 pgs: 5 active+clean+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 440 MiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 700 op/s; 5643/21078 objects degraded (26.772%); 6064/21078 objects misplaced (28.769%); 6.9 MiB/s, 111 objects/s recovering 2024-08-19T14:33:05.507 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:05 smithi031 ceph-mon[16336]: pgmap v1387: 47 pgs: 5 active+clean+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 440 MiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 700 op/s; 5643/21078 objects degraded (26.772%); 6064/21078 objects misplaced (28.769%); 6.9 MiB/s, 111 objects/s recovering 2024-08-19T14:33:05.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:05 smithi045 ceph-mon[20143]: pgmap v1387: 47 pgs: 5 active+clean+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 440 MiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 700 op/s; 5643/21078 objects degraded (26.772%); 6064/21078 objects misplaced (28.769%); 6.9 MiB/s, 111 objects/s recovering 2024-08-19T14:33:05.593 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:33:05.608 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-08-19T14:33:05.628 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:05.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:05.629 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:33:05.730 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:33:05.756 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:05.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:05.757 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:33:05.852 INFO:teuthology.orchestra.run.smithi031.stderr:no valid command found; 10 closest matches: 2024-08-19T14:33:05.852 INFO:teuthology.orchestra.run.smithi031.stderr:0 2024-08-19T14:33:05.852 INFO:teuthology.orchestra.run.smithi031.stderr:1 2024-08-19T14:33:05.852 INFO:teuthology.orchestra.run.smithi031.stderr:2 2024-08-19T14:33:05.852 INFO:teuthology.orchestra.run.smithi031.stderr:abort 2024-08-19T14:33:05.852 INFO:teuthology.orchestra.run.smithi031.stderr:assert 2024-08-19T14:33:05.852 INFO:teuthology.orchestra.run.smithi031.stderr:bench [] [] [] [] 2024-08-19T14:33:05.852 INFO:teuthology.orchestra.run.smithi031.stderr:bluefs debug_inject_read_zeros 2024-08-19T14:33:05.853 INFO:teuthology.orchestra.run.smithi031.stderr:bluefs files list 2024-08-19T14:33:05.853 INFO:teuthology.orchestra.run.smithi031.stderr:bluefs stats 2024-08-19T14:33:05.853 INFO:teuthology.orchestra.run.smithi031.stderr:bluestore allocator dump block 2024-08-19T14:33:05.853 INFO:teuthology.orchestra.run.smithi031.stderr:Error EINVAL: invalid command 2024-08-19T14:33:05.858 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:33:05.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:05.890 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:05.890 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:33:05.940 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:33:05.989 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-19T14:33:05.992 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:33:06.026 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:06.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:06.027 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:33:06.129 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:33:06.165 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:06.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:06.166 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:33:06.267 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:33:06.307 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:06.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:06.308 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:33:06.398 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:33:06.409 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:33:06.439 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:06.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:06.440 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:33:06.541 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:33:06.568 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:06.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:06.569 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:33:06.596 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:33:06.596 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:33:06.671 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:33:06.704 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:06.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:06.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:33:06.807 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:33:06.832 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:06.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:06.833 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:33:06.934 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:33:06.964 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:06.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:06.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:33:07.066 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:33:07.089 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:07.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:07.091 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:33:07.191 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:33:07.214 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:07.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:07.215 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:33:07.274 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-19T14:33:07.274 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-19T14:33:07.274 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph tell osd.1 'trim stale osdmaps' 2024-08-19T14:33:07.316 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:33:07.367 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-08-19T14:33:07.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:07.384 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:07.384 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:33:07.400 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:33:07.409 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:33:07.485 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:33:07.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:07 smithi032 ceph-mon[20004]: pgmap v1388: 47 pgs: 5 active+clean+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 292 MiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 912 op/s; 5285/14001 objects degraded (37.747%); 5344/14001 objects misplaced (38.169%); 9.7 MiB/s, 214 objects/s recovering 2024-08-19T14:33:07.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:07.522 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:07.523 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:33:07.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:07 smithi045 ceph-mon[20143]: pgmap v1388: 47 pgs: 5 active+clean+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 292 MiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 912 op/s; 5285/14001 objects degraded (37.747%); 5344/14001 objects misplaced (38.169%); 9.7 MiB/s, 214 objects/s recovering 2024-08-19T14:33:07.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:07 smithi031 ceph-mon[16336]: pgmap v1388: 47 pgs: 5 active+clean+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 292 MiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 912 op/s; 5285/14001 objects degraded (37.747%); 5344/14001 objects misplaced (38.169%); 9.7 MiB/s, 214 objects/s recovering 2024-08-19T14:33:07.623 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:33:07.660 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:07.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:07.661 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:33:07.675 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:33:07.762 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:33:07.798 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:07.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:07.800 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:33:07.901 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:33:07.949 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:07.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:07.950 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:33:08.051 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:33:08.079 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:08.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:08.080 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:33:08.181 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:33:08.208 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:08.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:08.209 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:33:08.268 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:33:08.269 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:33:08.300 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:33:08.300 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:33:08.308 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:33:08.308 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:33:08.308 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-08-19T14:33:08.310 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:33:08.336 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:08.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:08.337 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:33:08.342 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:08 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 5285/14001 objects degraded (37.747%), 7 pgs degraded, 16 pgs undersized (PG_DEGRADED) 2024-08-19T14:33:08.344 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:33:08.344 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:33:08.378 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:33:08.378 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:33:08.406 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:33:08.407 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:33:08.432 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:33:08.432 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:33:08.438 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:33:08.461 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:08.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:08.462 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:33:08.471 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:33:08.471 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:33:08.500 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:33:08.501 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:33:08.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:08 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 5285/14001 objects degraded (37.747%), 7 pgs degraded, 16 pgs undersized (PG_DEGRADED) 2024-08-19T14:33:08.517 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:33:08.545 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:33:08.545 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:33:08.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:08 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 5285/14001 objects degraded (37.747%), 7 pgs degraded, 16 pgs undersized (PG_DEGRADED) 2024-08-19T14:33:08.563 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:33:08.587 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:33:08.587 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:33:08.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:08.588 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:08.588 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:33:08.626 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:33:08.626 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:33:08.689 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:33:08.733 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:08.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:08.735 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:33:08.836 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:33:08.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:08.878 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:08.878 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:33:08.959 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-08-19T14:33:08.979 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:33:09.028 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:09.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:09.029 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:33:09.130 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:33:09.158 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:09.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:09.159 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:33:09.221 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is set 2024-08-19T14:33:09.260 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:33:09.273 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:33:09.301 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:09.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:09.302 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:33:09.346 INFO:teuthology.orchestra.run.smithi031.stderr:no valid command found; 10 closest matches: 2024-08-19T14:33:09.346 INFO:teuthology.orchestra.run.smithi031.stderr:0 2024-08-19T14:33:09.347 INFO:teuthology.orchestra.run.smithi031.stderr:1 2024-08-19T14:33:09.347 INFO:teuthology.orchestra.run.smithi031.stderr:2 2024-08-19T14:33:09.347 INFO:teuthology.orchestra.run.smithi031.stderr:abort 2024-08-19T14:33:09.347 INFO:teuthology.orchestra.run.smithi031.stderr:assert 2024-08-19T14:33:09.347 INFO:teuthology.orchestra.run.smithi031.stderr:bench [] [] [] [] 2024-08-19T14:33:09.347 INFO:teuthology.orchestra.run.smithi031.stderr:bluefs debug_inject_read_zeros 2024-08-19T14:33:09.347 INFO:teuthology.orchestra.run.smithi031.stderr:bluefs files list 2024-08-19T14:33:09.347 INFO:teuthology.orchestra.run.smithi031.stderr:bluefs stats 2024-08-19T14:33:09.347 INFO:teuthology.orchestra.run.smithi031.stderr:bluestore allocator dump block 2024-08-19T14:33:09.347 INFO:teuthology.orchestra.run.smithi031.stderr:Error EINVAL: invalid command 2024-08-19T14:33:09.403 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:33:09.433 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:09.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:09.434 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:33:09.454 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:09 smithi031 ceph-mon[16336]: pgmap v1389: 47 pgs: 5 active+clean+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 292 MiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 691 op/s; 5285/14001 objects degraded (37.747%); 5344/14001 objects misplaced (38.169%); 9.7 MiB/s, 214 objects/s recovering 2024-08-19T14:33:09.454 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:09 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3357538490' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:33:09.454 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:09 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/537105103' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:33:09.455 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:09 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:33:09.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:09 smithi032 ceph-mon[20004]: pgmap v1389: 47 pgs: 5 active+clean+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 292 MiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 691 op/s; 5285/14001 objects degraded (37.747%); 5344/14001 objects misplaced (38.169%); 9.7 MiB/s, 214 objects/s recovering 2024-08-19T14:33:09.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:09 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3357538490' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:33:09.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:09 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/537105103' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:33:09.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:09 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:33:09.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:09 smithi045 ceph-mon[20143]: pgmap v1389: 47 pgs: 5 active+clean+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 292 MiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 691 op/s; 5285/14001 objects degraded (37.747%); 5344/14001 objects misplaced (38.169%); 9.7 MiB/s, 214 objects/s recovering 2024-08-19T14:33:09.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:09 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3357538490' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:33:09.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:09 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/537105103' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:33:09.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:09 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:33:09.535 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:33:09.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:09.573 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:09.573 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:33:09.674 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:33:09.699 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:09.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:09.700 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:33:09.780 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:33:09.780 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [], 2024-08-19T14:33:09.780 INFO:teuthology.orchestra.run.smithi045.stdout: "num_ops": 0 2024-08-19T14:33:09.781 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:33:09.801 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:33:09.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:09.827 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:09.827 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:33:09.928 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:33:09.966 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:09.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:09.967 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:33:10.068 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:33:10.100 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:10.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:10.101 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:33:10.203 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:33:10.227 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:10.227 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-19T14:33:10.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:10.228 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-19T14:33:10.228 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph tell osd.3 'trim stale osdmaps' 2024-08-19T14:33:10.229 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:33:10.246 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Removed 107270 objects 2024-08-19T14:33:10.246 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Clean up completed and total clean up time :180.726 2024-08-19T14:33:10.330 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:33:10.352 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:10.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:10.353 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:33:10.455 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:33:10.476 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:10.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:10.477 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:33:10.502 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:10 smithi045 ceph-mon[20143]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:33:10.502 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:10 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:33:10.502 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:10 smithi045 ceph-mon[20143]: osdmap e593: 12 total, 11 up, 9 in 2024-08-19T14:33:10.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:10 smithi032 ceph-mon[20004]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:33:10.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:10 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:33:10.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:10 smithi032 ceph-mon[20004]: osdmap e593: 12 total, 11 up, 9 in 2024-08-19T14:33:10.536 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-08-19T14:33:10.577 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:33:10.600 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:10.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:10.601 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:33:10.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:10 smithi031 ceph-mon[16336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:33:10.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:10 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:33:10.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:10 smithi031 ceph-mon[16336]: osdmap e593: 12 total, 11 up, 9 in 2024-08-19T14:33:10.702 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:33:10.724 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:10.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:10.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:33:10.826 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:33:10.850 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:10.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:10.854 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:33:10.876 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:33:10.955 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:33:10.977 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:10.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:10.978 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:33:11.079 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:33:11.100 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:11.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:11.101 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:33:11.181 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-19T14:33:11.182 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-19T14:33:11.201 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:33:11.224 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:11.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:11.225 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:33:11.255 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:33:11.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:11 smithi045 ceph-mon[20143]: pgmap v1391: 47 pgs: 5 active+clean+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 166 MiB data, 10 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 717 op/s; 5042/7962 objects degraded (63.326%); 4119/7962 objects misplaced (51.733%); 3.5 MiB/s, 116 objects/s recovering 2024-08-19T14:33:11.326 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:33:11.348 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:11.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:11.349 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:33:11.388 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:33:11.388 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [], 2024-08-19T14:33:11.388 INFO:teuthology.orchestra.run.smithi045.stdout: "complaint_time": 30, 2024-08-19T14:33:11.389 INFO:teuthology.orchestra.run.smithi045.stdout: "num_blocked_ops": 0 2024-08-19T14:33:11.389 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:33:11.449 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:33:11.472 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:11.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:11.473 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:33:11.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:11 smithi032 ceph-mon[20004]: pgmap v1391: 47 pgs: 5 active+clean+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 166 MiB data, 10 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 717 op/s; 5042/7962 objects degraded (63.326%); 4119/7962 objects misplaced (51.733%); 3.5 MiB/s, 116 objects/s recovering 2024-08-19T14:33:11.525 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:11 smithi031 ceph-mon[16336]: pgmap v1391: 47 pgs: 5 active+clean+remapped, 9 active+undersized, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 26 active+clean; 166 MiB data, 10 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 717 op/s; 5042/7962 objects degraded (63.326%); 4119/7962 objects misplaced (51.733%); 3.5 MiB/s, 116 objects/s recovering 2024-08-19T14:33:11.574 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:33:11.596 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:11.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:11.597 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:33:11.698 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:33:11.719 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:11.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:11.720 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:33:11.821 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:33:11.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:11.845 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:11.845 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:33:11.946 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:33:11.969 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:11.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:11.970 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:33:12.071 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:33:12.093 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:12.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:12.094 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:33:12.115 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-08-19T14:33:12.194 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:33:12.217 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:12.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:12.218 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:33:12.319 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:33:12.341 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:12.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:12.342 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:33:12.404 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:33:12.443 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:33:12.466 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:12.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:12.467 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:33:12.568 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:33:12.590 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:12.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:12.591 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:33:12.638 INFO:teuthology.orchestra.run.smithi031.stderr:no valid command found; 10 closest matches: 2024-08-19T14:33:12.638 INFO:teuthology.orchestra.run.smithi031.stderr:0 2024-08-19T14:33:12.639 INFO:teuthology.orchestra.run.smithi031.stderr:1 2024-08-19T14:33:12.639 INFO:teuthology.orchestra.run.smithi031.stderr:2 2024-08-19T14:33:12.639 INFO:teuthology.orchestra.run.smithi031.stderr:abort 2024-08-19T14:33:12.639 INFO:teuthology.orchestra.run.smithi031.stderr:assert 2024-08-19T14:33:12.639 INFO:teuthology.orchestra.run.smithi031.stderr:bench [] [] [] [] 2024-08-19T14:33:12.639 INFO:teuthology.orchestra.run.smithi031.stderr:bluefs debug_inject_read_zeros 2024-08-19T14:33:12.639 INFO:teuthology.orchestra.run.smithi031.stderr:bluefs files list 2024-08-19T14:33:12.639 INFO:teuthology.orchestra.run.smithi031.stderr:bluefs stats 2024-08-19T14:33:12.639 INFO:teuthology.orchestra.run.smithi031.stderr:bluestore allocator dump block 2024-08-19T14:33:12.639 INFO:teuthology.orchestra.run.smithi031.stderr:Error EINVAL: invalid command 2024-08-19T14:33:12.694 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:33:12.704 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-19T14:33:12.718 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:12.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:12.719 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:33:12.820 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:33:12.844 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:12.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:12.845 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:33:12.946 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:33:12.948 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:33:12.948 INFO:teuthology.orchestra.run.smithi045.stdout: "size": 20, 2024-08-19T14:33:12.948 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 600, 2024-08-19T14:33:12.948 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [ 2024-08-19T14:33:12.948 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:12.948 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.25813.0:190861 5.e e481/477 5:75e2f3e5:::benchmark_data_smithi045_85309_object70608:head v 481'11903, mlcod=481'11899)", 2024-08-19T14:33:12.948 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:27:24.466007+0000", 2024-08-19T14:33:12.948 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 348.48005180299998, 2024-08-19T14:33:12.948 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.028908151999999999, 2024-08-19T14:33:12.948 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:33:12.948 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:33:12.949 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:33:12.949 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:33:12.949 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:12.949 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:33:12.949 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:24.466007+0000", 2024-08-19T14:33:12.949 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:33:12.949 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:12.949 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:12.949 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:33:12.949 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:24.466007+0000", 2024-08-19T14:33:12.949 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:33:12.949 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:12.949 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:12.949 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:33:12.950 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:24.466007+0000", 2024-08-19T14:33:12.950 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.8400000000000004e-07 2024-08-19T14:33:12.950 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:12.950 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:12.950 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:33:12.950 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:24.466015+0000", 2024-08-19T14:33:12.950 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.1810000000000004e-06 2024-08-19T14:33:12.950 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:12.950 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:12.950 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:33:12.950 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:24.466030+0000", 2024-08-19T14:33:12.950 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5647000000000001e-05 2024-08-19T14:33:12.950 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:12.951 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:12.951 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:33:12.951 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:24.466038+0000", 2024-08-19T14:33:12.951 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4660000000000002e-06 2024-08-19T14:33:12.951 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:12.951 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:12.951 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:33:12.951 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:24.470082+0000", 2024-08-19T14:33:12.951 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0040439849999999999 2024-08-19T14:33:12.951 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:12.951 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:12.951 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:33:12.951 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:24.470119+0000", 2024-08-19T14:33:12.951 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.6885999999999999e-05 2024-08-19T14:33:12.951 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:12.952 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:12.952 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:33:12.952 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:24.494866+0000", 2024-08-19T14:33:12.952 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.02474734 2024-08-19T14:33:12.952 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:12.952 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:12.952 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:33:12.952 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:24.494915+0000", 2024-08-19T14:33:12.952 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.8962999999999999e-05 2024-08-19T14:33:12.952 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:33:12.952 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:33:12.952 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:33:12.952 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:12.952 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:12.953 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.25813.0:217785 5.b e488/477 5:dc92cfe9:::benchmark_data_smithi045_85309_object97532:head v 488'13839, mlcod=488'13834)", 2024-08-19T14:33:12.953 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:27:47.695225+0000", 2024-08-19T14:33:12.953 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 325.25083324100001, 2024-08-19T14:33:12.953 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.032146156000000002, 2024-08-19T14:33:12.953 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:33:12.953 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:33:12.953 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:33:12.953 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:33:12.953 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:12.953 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:33:12.953 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:47.695225+0000", 2024-08-19T14:33:12.953 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:33:12.953 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:12.953 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:12.954 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:33:12.954 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:47.695225+0000", 2024-08-19T14:33:12.954 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:33:12.954 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:12.954 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:12.954 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:33:12.954 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:47.695226+0000", 2024-08-19T14:33:12.954 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.1500000000000003e-07 2024-08-19T14:33:12.954 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:12.954 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:12.954 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:33:12.954 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:47.695233+0000", 2024-08-19T14:33:12.954 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.1149999999999996e-06 2024-08-19T14:33:12.954 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:12.955 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:12.955 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:33:12.955 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:47.695248+0000", 2024-08-19T14:33:12.955 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4582e-05 2024-08-19T14:33:12.955 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:12.955 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:12.955 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:33:12.955 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:47.695256+0000", 2024-08-19T14:33:12.955 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.5989999999999996e-06 2024-08-19T14:33:12.955 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:12.955 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:12.955 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:33:12.955 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:47.700093+0000", 2024-08-19T14:33:12.955 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0048376959999999998 2024-08-19T14:33:12.956 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:12.956 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:12.956 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:33:12.956 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:47.700146+0000", 2024-08-19T14:33:12.956 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2874999999999998e-05 2024-08-19T14:33:12.956 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:12.956 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:12.956 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:33:12.956 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:47.727189+0000", 2024-08-19T14:33:12.956 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.027043306999999999 2024-08-19T14:33:12.956 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:12.956 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:12.956 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:33:12.956 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:47.727372+0000", 2024-08-19T14:33:12.957 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000182067 2024-08-19T14:33:12.957 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:33:12.957 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:33:12.957 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:33:12.957 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:12.957 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:12.957 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.25813.0:229596 5.b e491/477 5:d6edeaf5:::benchmark_data_smithi045_85309_object109343:head v 491'14603, mlcod=491'14601)", 2024-08-19T14:33:12.957 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:27:58.363548+0000", 2024-08-19T14:33:12.957 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 314.582510931, 2024-08-19T14:33:12.957 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.033372664000000003, 2024-08-19T14:33:12.957 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:33:12.957 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:33:12.957 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:33:12.957 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:33:12.957 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:12.958 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:33:12.958 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:58.363548+0000", 2024-08-19T14:33:12.958 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:33:12.958 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:12.958 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:12.958 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:33:12.958 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:58.363548+0000", 2024-08-19T14:33:12.958 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:33:12.958 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:12.958 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:12.958 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:33:12.958 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:58.363549+0000", 2024-08-19T14:33:12.958 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.2200000000000002e-07 2024-08-19T14:33:12.958 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:12.959 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:12.959 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:33:12.959 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:58.363556+0000", 2024-08-19T14:33:12.959 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4109999999999999e-06 2024-08-19T14:33:12.959 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:12.959 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:12.959 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:33:12.959 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:58.363568+0000", 2024-08-19T14:33:12.959 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1545e-05 2024-08-19T14:33:12.959 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:12.959 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:12.959 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:33:12.959 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:58.363575+0000", 2024-08-19T14:33:12.959 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.0010000000000004e-06 2024-08-19T14:33:12.959 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:12.960 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:12.960 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:33:12.960 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:58.365369+0000", 2024-08-19T14:33:12.960 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0017943049999999999 2024-08-19T14:33:12.960 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:12.960 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:12.960 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:33:12.960 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:58.365387+0000", 2024-08-19T14:33:12.960 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.844e-05 2024-08-19T14:33:12.960 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:12.960 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:12.960 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:33:12.960 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:58.396862+0000", 2024-08-19T14:33:12.960 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.031475086999999999 2024-08-19T14:33:12.960 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:12.961 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:12.961 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:33:12.961 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:58.396920+0000", 2024-08-19T14:33:12.961 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7952999999999999e-05 2024-08-19T14:33:12.961 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:33:12.961 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:33:12.961 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:33:12.961 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:12.961 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:12.961 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.25813.0:229600 5.b e491/477 5:d239d9b9:::benchmark_data_smithi045_85309_object109347:head v 491'14604, mlcod=491'14601)", 2024-08-19T14:33:12.961 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:27:58.367870+0000", 2024-08-19T14:33:12.961 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 314.57818847700003, 2024-08-19T14:33:12.961 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.029084236999999999, 2024-08-19T14:33:12.961 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:33:12.962 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:33:12.962 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:33:12.962 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:33:12.962 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:12.962 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:33:12.962 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:58.367870+0000", 2024-08-19T14:33:12.962 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:33:12.962 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:12.962 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:12.962 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:33:12.962 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:58.367870+0000", 2024-08-19T14:33:12.962 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:33:12.962 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:12.962 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:12.963 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:33:12.963 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:58.367871+0000", 2024-08-19T14:33:12.963 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.088e-06 2024-08-19T14:33:12.963 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:12.963 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:12.963 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:33:12.963 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:58.367882+0000", 2024-08-19T14:33:12.963 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1009000000000001e-05 2024-08-19T14:33:12.963 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:12.963 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:12.963 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:33:12.963 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:58.367892+0000", 2024-08-19T14:33:12.963 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0200999999999999e-05 2024-08-19T14:33:12.963 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:12.963 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:12.964 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:33:12.964 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:58.367901+0000", 2024-08-19T14:33:12.964 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.8089999999999999e-06 2024-08-19T14:33:12.964 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:12.964 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:12.964 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:33:12.964 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:58.370387+0000", 2024-08-19T14:33:12.964 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0024857059999999999 2024-08-19T14:33:12.964 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:12.964 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:12.964 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:33:12.964 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:58.370435+0000", 2024-08-19T14:33:12.964 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.7731000000000001e-05 2024-08-19T14:33:12.964 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:12.964 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:12.965 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:33:12.965 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:58.396926+0000", 2024-08-19T14:33:12.965 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.026491114999999999 2024-08-19T14:33:12.965 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:12.965 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:12.965 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:33:12.965 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:58.396954+0000", 2024-08-19T14:33:12.965 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.8578e-05 2024-08-19T14:33:12.965 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:33:12.965 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:33:12.965 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:33:12.965 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:33:12.965 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:33:12.965 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:33:12.968 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:12.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:12.969 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:33:13.016 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_4 2024-08-19T14:33:13.017 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd pool rm unique_pool_4 unique_pool_4 --yes-i-really-really-mean-it 2024-08-19T14:33:13.070 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:33:13.091 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:13.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:13.092 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:33:13.183 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:33:13.193 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:33:13.217 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:13.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:13.218 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:33:13.319 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:33:13.334 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:13 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:33:13.335 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:13 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:33:13.335 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:13 smithi031 ceph-mon[16336]: osdmap e594: 12 total, 11 up, 9 in 2024-08-19T14:33:13.335 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:13 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:33:13.335 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:13 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:33:13.335 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:13 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:33:13.335 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:13 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:33:13.335 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:13 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 4660/1722 objects degraded (270.616%), 6 pgs degraded, 16 pgs undersized (PG_DEGRADED) 2024-08-19T14:33:13.340 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:13.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:13.341 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:33:13.378 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-19T14:33:13.378 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-19T14:33:13.379 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph tell osd.4 'trim stale osdmaps' 2024-08-19T14:33:13.379 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:33:13.408 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:33:13.442 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:33:13.463 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:13.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:13.464 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:33:13.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:13 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:33:13.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:13 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:33:13.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:13 smithi032 ceph-mon[20004]: osdmap e594: 12 total, 11 up, 9 in 2024-08-19T14:33:13.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:13 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:33:13.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:13 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:33:13.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:13 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:33:13.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:13 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:33:13.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:13 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 4660/1722 objects degraded (270.616%), 6 pgs degraded, 16 pgs undersized (PG_DEGRADED) 2024-08-19T14:33:13.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:13 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:33:13.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:13 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:33:13.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:13 smithi045 ceph-mon[20143]: osdmap e594: 12 total, 11 up, 9 in 2024-08-19T14:33:13.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:13 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:33:13.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:13 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:33:13.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:13 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:33:13.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:13 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:33:13.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:13 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 4660/1722 objects degraded (270.616%), 6 pgs degraded, 16 pgs undersized (PG_DEGRADED) 2024-08-19T14:33:13.565 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:33:13.586 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:13.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:13.587 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:33:13.688 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:33:13.710 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:13.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:13.715 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:33:13.773 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-08-19T14:33:13.815 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:33:13.836 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:13.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:13.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:33:13.938 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:33:13.960 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:13.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:13.961 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:33:14.062 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:33:14.084 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:14.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:14.085 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:33:14.186 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:33:14.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:14.208 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:14.208 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:33:14.252 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is unset 2024-08-19T14:33:14.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:14 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3954490559' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:33:14.309 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:33:14.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:14.339 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:14.339 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:33:14.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:14 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3954490559' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:33:14.440 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:33:14.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:14.463 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:14.463 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:33:14.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:14 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3954490559' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:33:14.554 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:33:14.564 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:33:14.588 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:14.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:14.589 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:33:14.690 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:33:14.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:14.713 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:14.713 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:33:14.814 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:33:14.836 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:14.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:14.837 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:33:14.938 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:33:14.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:14.960 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:14.960 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:33:15.061 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:33:15.082 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:15.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:15.084 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:33:15.097 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:33:15.097 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [], 2024-08-19T14:33:15.097 INFO:teuthology.orchestra.run.smithi045.stdout: "num_ops": 0 2024-08-19T14:33:15.097 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:33:15.185 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:33:15.206 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:15.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:15.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:33:15.308 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:33:15.329 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:15.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:15.330 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:33:15.431 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:33:15.452 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:15.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:15.453 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:33:15.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:15 smithi032 ceph-mon[20004]: pgmap v1394: 47 pgs: 5 active+clean+remapped, 11 active+undersized, 3 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 26 active+clean; 7.9 MiB data, 9.9 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 568 op/s; 4579/360 objects degraded (1271.944%); 3378/360 objects misplaced (938.333%); 848 KiB/s, 133 objects/s recovering 2024-08-19T14:33:15.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:15 smithi032 ceph-mon[20004]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:33:15.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:15 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3954490559' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:33:15.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:15 smithi032 ceph-mon[20004]: osdmap e595: 12 total, 11 up, 9 in 2024-08-19T14:33:15.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:15 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:33:15.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:15 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:33:15.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:15 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:33:15.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:15 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:33:15.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:15 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:33:15.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:15 smithi045 ceph-mon[20143]: pgmap v1394: 47 pgs: 5 active+clean+remapped, 11 active+undersized, 3 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 26 active+clean; 7.9 MiB data, 9.9 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 568 op/s; 4579/360 objects degraded (1271.944%); 3378/360 objects misplaced (938.333%); 848 KiB/s, 133 objects/s recovering 2024-08-19T14:33:15.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:15 smithi045 ceph-mon[20143]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:33:15.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:15 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3954490559' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:33:15.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:15 smithi045 ceph-mon[20143]: osdmap e595: 12 total, 11 up, 9 in 2024-08-19T14:33:15.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:15 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:33:15.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:15 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:33:15.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:15 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:33:15.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:15 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:33:15.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:15 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:33:15.555 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:33:15.576 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:15.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:15.577 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:33:15.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:15 smithi031 ceph-mon[16336]: pgmap v1394: 47 pgs: 5 active+clean+remapped, 11 active+undersized, 3 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 26 active+clean; 7.9 MiB data, 9.9 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 568 op/s; 4579/360 objects degraded (1271.944%); 3378/360 objects misplaced (938.333%); 848 KiB/s, 133 objects/s recovering 2024-08-19T14:33:15.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:15 smithi031 ceph-mon[16336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:33:15.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:15 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3954490559' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:33:15.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:15 smithi031 ceph-mon[16336]: osdmap e595: 12 total, 11 up, 9 in 2024-08-19T14:33:15.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:15 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:33:15.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:15 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:33:15.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:15 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:33:15.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:15 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:33:15.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:15 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:33:15.678 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-08-19T14:33:15.679 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:33:15.700 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:15.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:15.701 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:33:15.802 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:33:15.823 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:15.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:15.824 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:33:15.925 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:33:15.929 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:33:15.947 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:15.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:15.948 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:33:16.049 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:33:16.072 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:16.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:16.073 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:33:16.175 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:33:16.197 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:16.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:16.197 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:33:16.265 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'unique_pool_4' removed 2024-08-19T14:33:16.298 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:33:16.337 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:16 smithi045 ceph-mon[20143]: osdmap e596: 12 total, 11 up, 9 in 2024-08-19T14:33:16.337 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:16 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1993352183' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-08-19T14:33:16.351 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:16.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:16.352 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:33:16.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:16 smithi031 ceph-mon[16336]: osdmap e596: 12 total, 11 up, 9 in 2024-08-19T14:33:16.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:16 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1993352183' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-08-19T14:33:16.453 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:33:16.474 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:16.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:16.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:33:16.570 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:33:16.570 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [], 2024-08-19T14:33:16.570 INFO:teuthology.orchestra.run.smithi045.stdout: "complaint_time": 30, 2024-08-19T14:33:16.570 INFO:teuthology.orchestra.run.smithi045.stdout: "num_blocked_ops": 0 2024-08-19T14:33:16.570 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:33:16.575 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:33:16.596 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:16.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:16.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:33:16.698 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:33:16.720 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:16.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:16.721 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:33:16.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:16 smithi032 ceph-mon[20004]: osdmap e596: 12 total, 11 up, 9 in 2024-08-19T14:33:16.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:16 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1993352183' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-08-19T14:33:16.821 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:33:16.842 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:16.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:16.843 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:33:16.944 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:33:16.967 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:16.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:16.968 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:33:17.069 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:33:17.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:17.092 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:17.092 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:33:17.193 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:33:17.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:17.219 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:17.219 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:33:17.228 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-08-19T14:33:17.319 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:33:17.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:17.345 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:17.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:33:17.446 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:33:17.468 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:17.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:17.469 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:33:17.478 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:33:17.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:17 smithi045 ceph-mon[20143]: 1.b scrub starts 2024-08-19T14:33:17.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:17 smithi045 ceph-mon[20143]: 1.b scrub ok 2024-08-19T14:33:17.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:17 smithi045 ceph-mon[20143]: pgmap v1397: 47 pgs: 1 peering, 1 active+undersized+remapped, 5 active+clean+remapped, 12 active+undersized, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded, 26 active+clean; 607 KiB data, 9.8 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 441 op/s; 3379/6 objects degraded (56316.667%); 3378/6 objects misplaced (56300.000%); 1.1 MiB/s, 571 objects/s recovering 2024-08-19T14:33:17.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:17 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1993352183' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-08-19T14:33:17.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:17 smithi045 ceph-mon[20143]: osdmap e597: 12 total, 11 up, 9 in 2024-08-19T14:33:17.570 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:33:17.591 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:17.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:17.592 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:33:17.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:17 smithi031 ceph-mon[16336]: 1.b scrub starts 2024-08-19T14:33:17.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:17 smithi031 ceph-mon[16336]: 1.b scrub ok 2024-08-19T14:33:17.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:17 smithi031 ceph-mon[16336]: pgmap v1397: 47 pgs: 1 peering, 1 active+undersized+remapped, 5 active+clean+remapped, 12 active+undersized, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded, 26 active+clean; 607 KiB data, 9.8 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 441 op/s; 3379/6 objects degraded (56316.667%); 3378/6 objects misplaced (56300.000%); 1.1 MiB/s, 571 objects/s recovering 2024-08-19T14:33:17.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:17 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1993352183' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-08-19T14:33:17.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:17 smithi031 ceph-mon[16336]: osdmap e597: 12 total, 11 up, 9 in 2024-08-19T14:33:17.693 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:33:17.716 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:17.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:17.717 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:33:17.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:17 smithi032 ceph-mon[20004]: 1.b scrub starts 2024-08-19T14:33:17.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:17 smithi032 ceph-mon[20004]: 1.b scrub ok 2024-08-19T14:33:17.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:17 smithi032 ceph-mon[20004]: pgmap v1397: 47 pgs: 1 peering, 1 active+undersized+remapped, 5 active+clean+remapped, 12 active+undersized, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded, 26 active+clean; 607 KiB data, 9.8 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 441 op/s; 3379/6 objects degraded (56316.667%); 3378/6 objects misplaced (56300.000%); 1.1 MiB/s, 571 objects/s recovering 2024-08-19T14:33:17.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:17 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1993352183' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-08-19T14:33:17.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:17 smithi032 ceph-mon[20004]: osdmap e597: 12 total, 11 up, 9 in 2024-08-19T14:33:17.817 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:33:17.843 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:17.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:17.844 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:33:17.945 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:33:17.949 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:33:17.949 INFO:teuthology.orchestra.run.smithi045.stdout: "size": 20, 2024-08-19T14:33:17.949 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 600, 2024-08-19T14:33:17.949 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [ 2024-08-19T14:33:17.949 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:17.949 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25813.0:115483 5.8 5:118541ec:::benchmark_data_smithi045_85309_object115482:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e455)", 2024-08-19T14:33:17.950 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:26:12.254294+0000", 2024-08-19T14:33:17.950 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 425.69271636500002, 2024-08-19T14:33:17.950 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.41025914400000002, 2024-08-19T14:33:17.950 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:33:17.950 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:33:17.950 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:33:17.950 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:33:17.950 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25813", 2024-08-19T14:33:17.951 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:33:17.951 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 115483 2024-08-19T14:33:17.951 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:17.951 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:33:17.951 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:17.951 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:33:17.951 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.254294+0000", 2024-08-19T14:33:17.951 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:33:17.951 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:17.951 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:17.951 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:33:17.951 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.254294+0000", 2024-08-19T14:33:17.952 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:33:17.952 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:17.952 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:17.952 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:33:17.952 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.254295+0000", 2024-08-19T14:33:17.952 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0049999999999999e-06 2024-08-19T14:33:17.952 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:17.952 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:17.952 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:33:17.952 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.254351+0000", 2024-08-19T14:33:17.952 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.6385999999999999e-05 2024-08-19T14:33:17.952 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:17.953 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:17.953 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:33:17.953 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.254365+0000", 2024-08-19T14:33:17.953 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4425e-05 2024-08-19T14:33:17.953 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:17.953 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:17.953 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:33:17.953 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.254373+0000", 2024-08-19T14:33:17.953 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.1620000000000002e-06 2024-08-19T14:33:17.953 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:17.953 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:17.954 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:33:17.954 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.254438+0000", 2024-08-19T14:33:17.954 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.5295000000000004e-05 2024-08-19T14:33:17.954 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:17.954 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:17.954 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:33:17.955 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.254556+0000", 2024-08-19T14:33:17.955 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000117555 2024-08-19T14:33:17.955 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:17.955 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:17.955 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,9", 2024-08-19T14:33:17.955 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.254698+0000", 2024-08-19T14:33:17.955 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000142003 2024-08-19T14:33:17.955 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:17.956 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:17.956 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:33:17.956 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.256435+0000", 2024-08-19T14:33:17.956 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0017378999999999999 2024-08-19T14:33:17.956 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:17.956 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:17.956 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:17.956 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.265174+0000", 2024-08-19T14:33:17.956 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0087388980000000002 2024-08-19T14:33:17.956 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:17.956 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:17.957 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:17.957 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.664449+0000", 2024-08-19T14:33:17.957 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.39927473800000002 2024-08-19T14:33:17.957 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:17.957 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:17.957 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:33:17.957 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.664502+0000", 2024-08-19T14:33:17.957 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.3066000000000003e-05 2024-08-19T14:33:17.957 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:17.957 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:17.958 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:33:17.958 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.664553+0000", 2024-08-19T14:33:17.958 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0711000000000003e-05 2024-08-19T14:33:17.958 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:33:17.958 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:33:17.958 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:33:17.958 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:17.958 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:17.958 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25813.0:115542 5.8 5:126f9ca1:::benchmark_data_smithi045_85309_object115541:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e455)", 2024-08-19T14:33:17.958 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:26:12.310153+0000", 2024-08-19T14:33:17.959 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 425.636856717, 2024-08-19T14:33:17.959 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.35453124699999999, 2024-08-19T14:33:17.959 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:33:17.959 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:33:17.959 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:33:17.959 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:33:17.959 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25813", 2024-08-19T14:33:17.959 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:33:17.959 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 115542 2024-08-19T14:33:17.959 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:17.959 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:33:17.960 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:17.960 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:33:17.960 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.310153+0000", 2024-08-19T14:33:17.960 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:33:17.960 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:17.960 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:17.960 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:33:17.960 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.310153+0000", 2024-08-19T14:33:17.960 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:33:17.960 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:17.961 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:17.961 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:33:17.961 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.310155+0000", 2024-08-19T14:33:17.961 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.279e-06 2024-08-19T14:33:17.961 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:17.961 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:17.961 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:33:17.961 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.310186+0000", 2024-08-19T14:33:17.961 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.1291999999999998e-05 2024-08-19T14:33:17.961 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:17.961 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:17.961 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:33:17.962 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.310199+0000", 2024-08-19T14:33:17.962 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2824e-05 2024-08-19T14:33:17.962 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:17.962 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:17.962 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:33:17.962 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.310206+0000", 2024-08-19T14:33:17.962 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.3189999999999997e-06 2024-08-19T14:33:17.962 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:17.962 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:17.963 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:33:17.963 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.311624+0000", 2024-08-19T14:33:17.963 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0014176740000000001 2024-08-19T14:33:17.963 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:17.963 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:17.963 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:33:17.963 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.311764+0000", 2024-08-19T14:33:17.963 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014065900000000001 2024-08-19T14:33:17.963 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:17.963 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:17.963 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,9", 2024-08-19T14:33:17.963 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.311923+0000", 2024-08-19T14:33:17.964 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015905800000000001 2024-08-19T14:33:17.964 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:17.964 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:17.964 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:33:17.964 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.313965+0000", 2024-08-19T14:33:17.964 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0020413430000000002 2024-08-19T14:33:17.964 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:17.964 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:17.964 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:17.964 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.320836+0000", 2024-08-19T14:33:17.964 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0068708529999999997 2024-08-19T14:33:17.965 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:17.965 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:17.965 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:17.965 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.664598+0000", 2024-08-19T14:33:17.965 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.34376222299999998 2024-08-19T14:33:17.965 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:17.965 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:17.965 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:33:17.965 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.664648+0000", 2024-08-19T14:33:17.965 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0489999999999999e-05 2024-08-19T14:33:17.965 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:17.965 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:17.965 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:33:17.966 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.664685+0000", 2024-08-19T14:33:17.966 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.6232999999999997e-05 2024-08-19T14:33:17.966 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:33:17.966 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:33:17.966 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:33:17.966 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:17.966 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:17.966 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25813.0:115641 5.8 5:1358c592:::benchmark_data_smithi045_85309_object115640:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e455)", 2024-08-19T14:33:17.966 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:26:12.409980+0000", 2024-08-19T14:33:17.966 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 425.53702968099998, 2024-08-19T14:33:17.966 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.25595528499999998, 2024-08-19T14:33:17.967 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:33:17.967 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:33:17.967 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:33:17.967 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:33:17.967 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25813", 2024-08-19T14:33:17.967 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:33:17.967 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 115641 2024-08-19T14:33:17.967 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:17.967 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:33:17.967 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:17.967 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:33:17.968 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.409980+0000", 2024-08-19T14:33:17.968 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:33:17.968 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:17.968 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:17.968 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:33:17.968 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.409980+0000", 2024-08-19T14:33:17.968 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:33:17.968 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:17.968 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:17.968 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:33:17.968 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.409982+0000", 2024-08-19T14:33:17.968 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2950000000000001e-06 2024-08-19T14:33:17.969 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:17.969 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:17.969 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:33:17.969 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.410007+0000", 2024-08-19T14:33:17.969 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.5198000000000002e-05 2024-08-19T14:33:17.969 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:17.969 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:17.969 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:33:17.969 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.410022+0000", 2024-08-19T14:33:17.969 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4678000000000001e-05 2024-08-19T14:33:17.969 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:17.969 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:17.970 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:33:17.970 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.410029+0000", 2024-08-19T14:33:17.970 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4590000000000001e-06 2024-08-19T14:33:17.970 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:17.970 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:17.970 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:33:17.970 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.410088+0000", 2024-08-19T14:33:17.970 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.9169999999999998e-05 2024-08-19T14:33:17.970 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:17.970 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:17.970 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:33:17.971 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.410232+0000", 2024-08-19T14:33:17.971 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014414699999999999 2024-08-19T14:33:17.971 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:17.971 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:17.971 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,9", 2024-08-19T14:33:17.971 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.410377+0000", 2024-08-19T14:33:17.971 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000144566 2024-08-19T14:33:17.971 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:17.971 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:17.971 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:33:17.971 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.412287+0000", 2024-08-19T14:33:17.972 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001909628 2024-08-19T14:33:17.972 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:17.972 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:17.972 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:17.972 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.449686+0000", 2024-08-19T14:33:17.972 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.037399549999999997 2024-08-19T14:33:17.972 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:17.972 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:17.972 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:17.972 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.665809+0000", 2024-08-19T14:33:17.972 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.21612313899999999 2024-08-19T14:33:17.972 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:17.973 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:17.973 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:33:17.973 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.665895+0000", 2024-08-19T14:33:17.973 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.5433999999999996e-05 2024-08-19T14:33:17.973 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:17.973 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:17.973 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:33:17.973 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.665936+0000", 2024-08-19T14:33:17.973 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.1020999999999999e-05 2024-08-19T14:33:17.973 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:33:17.973 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:33:17.973 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:33:17.974 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:17.974 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:17.974 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25813.0:115648 5.8 5:15d8ce6a:::benchmark_data_smithi045_85309_object115647:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e455)", 2024-08-19T14:33:17.974 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:26:12.416281+0000", 2024-08-19T14:33:17.974 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 425.53072936900003, 2024-08-19T14:33:17.974 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.24979046499999999, 2024-08-19T14:33:17.974 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:33:17.974 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:33:17.974 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:33:17.974 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:33:17.975 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25813", 2024-08-19T14:33:17.975 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:33:17.975 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 115648 2024-08-19T14:33:17.975 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:17.975 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:33:17.975 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:17.975 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:33:17.975 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.416281+0000", 2024-08-19T14:33:17.975 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:33:17.975 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:17.975 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:17.975 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:33:17.976 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.416281+0000", 2024-08-19T14:33:17.976 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:33:17.976 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:17.976 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:17.976 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:33:17.976 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.416282+0000", 2024-08-19T14:33:17.976 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5740000000000001e-06 2024-08-19T14:33:17.976 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:17.976 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:17.976 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:33:17.977 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.416304+0000", 2024-08-19T14:33:17.977 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1698000000000001e-05 2024-08-19T14:33:17.977 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:17.977 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:17.977 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:33:17.977 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.416317+0000", 2024-08-19T14:33:17.977 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3084e-05 2024-08-19T14:33:17.977 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:17.977 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:17.977 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:33:17.977 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.416324+0000", 2024-08-19T14:33:17.978 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.0570000000000004e-06 2024-08-19T14:33:17.978 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:17.978 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:17.978 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:33:17.978 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.416383+0000", 2024-08-19T14:33:17.978 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.9261000000000001e-05 2024-08-19T14:33:17.978 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:17.978 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:17.978 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:33:17.978 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.416500+0000", 2024-08-19T14:33:17.978 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000116523 2024-08-19T14:33:17.979 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:17.979 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:17.979 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,9", 2024-08-19T14:33:17.979 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.416646+0000", 2024-08-19T14:33:17.979 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014645500000000001 2024-08-19T14:33:17.979 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:17.979 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:17.979 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:33:17.979 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.418689+0000", 2024-08-19T14:33:17.979 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0020425019999999999 2024-08-19T14:33:17.979 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:17.980 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:17.980 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:17.980 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.449715+0000", 2024-08-19T14:33:17.980 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.031026194 2024-08-19T14:33:17.980 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:17.980 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:17.980 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:17.980 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.665979+0000", 2024-08-19T14:33:17.980 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.21626399099999999 2024-08-19T14:33:17.980 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:17.980 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:17.981 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:33:17.981 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.666032+0000", 2024-08-19T14:33:17.981 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.3140000000000003e-05 2024-08-19T14:33:17.981 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:17.981 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:17.981 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:33:17.981 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.666071+0000", 2024-08-19T14:33:17.981 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.8986000000000002e-05 2024-08-19T14:33:17.981 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:33:17.981 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:33:17.981 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:33:17.981 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:17.982 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:17.982 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25813.0:180218 5.d 5:bd4207f5:::benchmark_data_smithi045_85309_object59965:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e475)", 2024-08-19T14:33:17.982 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:27:11.940920+0000", 2024-08-19T14:33:17.982 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 366.00609014700001, 2024-08-19T14:33:17.982 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.23667333700000001, 2024-08-19T14:33:17.982 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:33:17.982 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:33:17.982 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:33:17.982 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:33:17.982 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25813", 2024-08-19T14:33:17.982 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:33:17.982 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 180218 2024-08-19T14:33:17.983 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:17.983 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:33:17.983 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:17.983 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:33:17.983 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:11.940920+0000", 2024-08-19T14:33:17.983 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:33:17.983 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:17.983 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:17.983 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:33:17.983 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:11.940920+0000", 2024-08-19T14:33:17.983 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:33:17.983 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:17.984 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:17.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:17.986 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:33:17.986 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:11.940921+0000", 2024-08-19T14:33:17.986 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.561e-06 2024-08-19T14:33:17.986 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:17.986 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:17.987 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:33:17.987 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:11.940927+0000", 2024-08-19T14:33:17.987 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7300000000000002e-06 2024-08-19T14:33:17.987 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:17.987 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:17.987 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:33:17.987 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:11.940938+0000", 2024-08-19T14:33:17.987 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0958000000000001e-05 2024-08-19T14:33:17.987 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:17.987 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:17.987 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:33:17.988 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:11.940945+0000", 2024-08-19T14:33:17.988 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.6579999999999996e-06 2024-08-19T14:33:17.988 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:17.988 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:17.988 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:33:17.988 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.031816+0000", 2024-08-19T14:33:17.988 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.090871065000000001 2024-08-19T14:33:17.988 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:17.988 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:17.988 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:33:17.988 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.031914+0000", 2024-08-19T14:33:17.988 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.7686e-05 2024-08-19T14:33:17.989 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:17.989 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:17.989 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 5,7", 2024-08-19T14:33:17.989 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.032000+0000", 2024-08-19T14:33:17.989 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.6169999999999997e-05 2024-08-19T14:33:17.989 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:17.989 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:17.989 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:33:17.989 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.045113+0000", 2024-08-19T14:33:17.989 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.013113028000000001 2024-08-19T14:33:17.989 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:17.989 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:17.990 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:17.990 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.047147+0000", 2024-08-19T14:33:17.990 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0020338050000000001 2024-08-19T14:33:17.990 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:17.990 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:17.990 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:17.990 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.177520+0000", 2024-08-19T14:33:17.990 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.130373085 2024-08-19T14:33:17.990 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:17.990 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:17.990 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:33:17.991 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.177556+0000", 2024-08-19T14:33:17.991 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.6403999999999999e-05 2024-08-19T14:33:17.991 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:17.991 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:17.991 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:33:17.991 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.177593+0000", 2024-08-19T14:33:17.991 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.7187000000000002e-05 2024-08-19T14:33:17.991 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:33:17.991 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:33:17.991 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:33:17.992 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:17.992 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:17.992 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25813.0:180682 5.8 5:1602eddf:::benchmark_data_smithi045_85309_object60429:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e476)", 2024-08-19T14:33:17.992 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:27:12.846408+0000", 2024-08-19T14:33:17.992 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 365.100601901, 2024-08-19T14:33:17.992 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.27902613199999998, 2024-08-19T14:33:17.992 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:33:17.992 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:33:17.992 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:33:17.992 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:33:17.992 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25813", 2024-08-19T14:33:17.992 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:33:17.993 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 180682 2024-08-19T14:33:17.993 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:17.993 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:33:17.993 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:17.993 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:33:17.993 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.846408+0000", 2024-08-19T14:33:17.993 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:33:17.993 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:17.994 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:17.994 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:33:17.994 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.846408+0000", 2024-08-19T14:33:17.994 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:33:17.994 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:17.994 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:17.994 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:33:17.994 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.846410+0000", 2024-08-19T14:33:17.994 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9980000000000002e-06 2024-08-19T14:33:17.995 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:17.995 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:17.995 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:33:17.995 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.846415+0000", 2024-08-19T14:33:17.995 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1120000000000004e-06 2024-08-19T14:33:17.995 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:17.995 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:17.995 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:33:17.995 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.846431+0000", 2024-08-19T14:33:17.995 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5786000000000001e-05 2024-08-19T14:33:17.995 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:17.995 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:17.996 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:33:17.996 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.846438+0000", 2024-08-19T14:33:17.996 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.3699999999999997e-06 2024-08-19T14:33:17.996 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:17.996 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:17.996 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:33:17.996 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.915048+0000", 2024-08-19T14:33:17.996 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.068609724999999996 2024-08-19T14:33:17.996 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:17.996 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:17.996 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:33:17.996 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.915174+0000", 2024-08-19T14:33:17.997 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012629899999999999 2024-08-19T14:33:17.997 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:17.997 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:17.997 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,9", 2024-08-19T14:33:17.997 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.915288+0000", 2024-08-19T14:33:17.997 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000113544 2024-08-19T14:33:17.997 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:17.997 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:17.997 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:17.997 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.921139+0000", 2024-08-19T14:33:17.997 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0058512420000000004 2024-08-19T14:33:17.997 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:17.998 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:17.998 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:33:17.998 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.923745+0000", 2024-08-19T14:33:17.998 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0026053589999999998 2024-08-19T14:33:17.998 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:17.998 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:17.998 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:17.998 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:13.125373+0000", 2024-08-19T14:33:17.998 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.201628589 2024-08-19T14:33:17.998 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:17.998 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:17.998 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:33:17.999 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:13.125405+0000", 2024-08-19T14:33:17.999 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.2224999999999997e-05 2024-08-19T14:33:17.999 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:17.999 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:17.999 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:33:17.999 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:13.125434+0000", 2024-08-19T14:33:17.999 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.8883000000000001e-05 2024-08-19T14:33:17.999 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:33:17.999 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:33:17.999 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:33:17.999 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:17.999 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:17.999 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25813.0:180713 5.8 5:13e650ea:::benchmark_data_smithi045_85309_object60460:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e476)", 2024-08-19T14:33:18.000 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:27:12.880045+0000", 2024-08-19T14:33:18.000 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 365.06696496900003, 2024-08-19T14:33:18.000 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.24953703699999999, 2024-08-19T14:33:18.000 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:33:18.000 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:33:18.000 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:33:18.000 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:33:18.000 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25813", 2024-08-19T14:33:18.000 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:33:18.000 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 180713 2024-08-19T14:33:18.000 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.001 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:33:18.001 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.001 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:33:18.001 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.880045+0000", 2024-08-19T14:33:18.001 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:33:18.001 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.001 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.001 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:33:18.001 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.880045+0000", 2024-08-19T14:33:18.001 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:33:18.001 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.001 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.002 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:33:18.002 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.880047+0000", 2024-08-19T14:33:18.002 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5510000000000001e-06 2024-08-19T14:33:18.002 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.002 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.002 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:33:18.002 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.880052+0000", 2024-08-19T14:33:18.002 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.5709999999999997e-06 2024-08-19T14:33:18.002 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.002 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.002 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:33:18.002 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.880068+0000", 2024-08-19T14:33:18.003 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6087e-05 2024-08-19T14:33:18.003 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.003 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.003 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:33:18.003 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.880076+0000", 2024-08-19T14:33:18.003 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4039999999999998e-06 2024-08-19T14:33:18.003 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.003 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.003 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:33:18.003 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.918429+0000", 2024-08-19T14:33:18.003 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.038353230000000002 2024-08-19T14:33:18.004 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.004 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.004 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:33:18.004 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.918538+0000", 2024-08-19T14:33:18.004 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00010878999999999999 2024-08-19T14:33:18.004 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.004 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.004 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,9", 2024-08-19T14:33:18.004 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.918613+0000", 2024-08-19T14:33:18.004 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.5405000000000001e-05 2024-08-19T14:33:18.004 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.004 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.005 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:33:18.005 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.923754+0000", 2024-08-19T14:33:18.005 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0051403999999999998 2024-08-19T14:33:18.005 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.005 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.005 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:18.005 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.924797+0000", 2024-08-19T14:33:18.005 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0010431150000000001 2024-08-19T14:33:18.005 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.005 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.005 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:18.006 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:13.129461+0000", 2024-08-19T14:33:18.006 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20466435799999999 2024-08-19T14:33:18.006 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.006 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.006 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:33:18.006 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:13.129522+0000", 2024-08-19T14:33:18.006 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.1054000000000002e-05 2024-08-19T14:33:18.006 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.006 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.006 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:33:18.006 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:13.129582+0000", 2024-08-19T14:33:18.006 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.0071999999999997e-05 2024-08-19T14:33:18.007 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:33:18.007 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:33:18.007 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:33:18.007 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.007 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.007 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25813.0:180719 5.8 5:15c77316:::benchmark_data_smithi045_85309_object60466:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e476)", 2024-08-19T14:33:18.007 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:27:12.886662+0000", 2024-08-19T14:33:18.007 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 365.060348465, 2024-08-19T14:33:18.007 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.24306710300000001, 2024-08-19T14:33:18.007 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:33:18.007 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:33:18.008 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:33:18.008 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:33:18.008 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25813", 2024-08-19T14:33:18.008 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:33:18.008 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 180719 2024-08-19T14:33:18.008 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.008 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:33:18.008 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.008 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:33:18.008 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.886662+0000", 2024-08-19T14:33:18.008 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:33:18.008 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.009 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.009 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:33:18.009 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.886662+0000", 2024-08-19T14:33:18.009 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:33:18.009 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.009 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.009 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:33:18.009 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.886663+0000", 2024-08-19T14:33:18.009 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.184e-06 2024-08-19T14:33:18.009 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.009 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.009 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:33:18.010 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.886667+0000", 2024-08-19T14:33:18.010 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.907e-06 2024-08-19T14:33:18.010 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.010 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.010 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:33:18.010 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.886679+0000", 2024-08-19T14:33:18.010 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1915e-05 2024-08-19T14:33:18.010 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.010 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.010 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:33:18.010 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.886686+0000", 2024-08-19T14:33:18.010 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.2049999999999996e-06 2024-08-19T14:33:18.011 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.011 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.011 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:33:18.011 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.921183+0000", 2024-08-19T14:33:18.011 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.034497556999999998 2024-08-19T14:33:18.011 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.011 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.011 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:33:18.011 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.921260+0000", 2024-08-19T14:33:18.011 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.6388000000000001e-05 2024-08-19T14:33:18.011 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.011 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.012 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,9", 2024-08-19T14:33:18.012 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.921322+0000", 2024-08-19T14:33:18.012 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.2611000000000001e-05 2024-08-19T14:33:18.012 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.012 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.012 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:33:18.012 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.923834+0000", 2024-08-19T14:33:18.012 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.002511924 2024-08-19T14:33:18.012 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.012 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.012 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:18.012 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.924841+0000", 2024-08-19T14:33:18.013 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0010065129999999999 2024-08-19T14:33:18.013 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.013 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.013 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:18.013 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:13.129629+0000", 2024-08-19T14:33:18.013 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20478792500000001 2024-08-19T14:33:18.013 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.013 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.013 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:33:18.013 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:13.129683+0000", 2024-08-19T14:33:18.013 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4293999999999999e-05 2024-08-19T14:33:18.013 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.014 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.014 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:33:18.014 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:13.129729+0000", 2024-08-19T14:33:18.014 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.5680000000000003e-05 2024-08-19T14:33:18.014 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:33:18.014 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:33:18.014 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:33:18.014 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.014 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.014 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25813.0:237770 5.8 5:11831c02:::benchmark_data_smithi045_85309_object117517:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e495)", 2024-08-19T14:33:18.014 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:28:07.106685+0000", 2024-08-19T14:33:18.014 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 310.840325377, 2024-08-19T14:33:18.015 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.22493542599999999, 2024-08-19T14:33:18.015 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:33:18.015 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:33:18.015 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:33:18.015 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:33:18.015 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25813", 2024-08-19T14:33:18.015 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:33:18.015 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 237770 2024-08-19T14:33:18.015 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.015 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:33:18.015 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.015 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:33:18.016 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:28:07.106685+0000", 2024-08-19T14:33:18.016 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:33:18.016 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.016 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.016 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:33:18.016 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:28:07.106685+0000", 2024-08-19T14:33:18.016 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:33:18.016 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.016 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.016 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:33:18.016 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:28:07.106685+0000", 2024-08-19T14:33:18.016 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.6199999999999998e-07 2024-08-19T14:33:18.017 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.017 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.017 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:33:18.017 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:28:07.106688+0000", 2024-08-19T14:33:18.017 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.0510000000000001e-06 2024-08-19T14:33:18.017 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.017 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.017 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:33:18.017 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:28:07.106696+0000", 2024-08-19T14:33:18.017 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.9899999999999997e-06 2024-08-19T14:33:18.017 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.017 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.018 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:33:18.018 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:28:07.106702+0000", 2024-08-19T14:33:18.018 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7810000000000002e-06 2024-08-19T14:33:18.018 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.018 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.018 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:33:18.018 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:28:07.106846+0000", 2024-08-19T14:33:18.018 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014427600000000001 2024-08-19T14:33:18.018 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.018 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.018 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for peered", 2024-08-19T14:33:18.019 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:28:07.106854+0000", 2024-08-19T14:33:18.019 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.3309999999999998e-06 2024-08-19T14:33:18.019 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.019 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.019 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:33:18.019 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:28:07.317267+0000", 2024-08-19T14:33:18.019 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.21041380400000001 2024-08-19T14:33:18.019 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.019 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.019 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:33:18.019 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:28:07.317422+0000", 2024-08-19T14:33:18.020 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015415200000000001 2024-08-19T14:33:18.020 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.020 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.020 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,5", 2024-08-19T14:33:18.020 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:28:07.318149+0000", 2024-08-19T14:33:18.020 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00072762499999999997 2024-08-19T14:33:18.020 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.020 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.020 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:33:18.020 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:28:07.322380+0000", 2024-08-19T14:33:18.020 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0042309590000000003 2024-08-19T14:33:18.021 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.021 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.021 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:18.021 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:28:07.323678+0000", 2024-08-19T14:33:18.021 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001298366 2024-08-19T14:33:18.021 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.021 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.021 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:18.021 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:28:07.331373+0000", 2024-08-19T14:33:18.021 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0076946790000000003 2024-08-19T14:33:18.021 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.022 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.022 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:33:18.022 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:28:07.331429+0000", 2024-08-19T14:33:18.022 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.5517999999999999e-05 2024-08-19T14:33:18.022 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.022 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.022 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:33:18.022 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:28:07.331620+0000", 2024-08-19T14:33:18.022 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00019143200000000001 2024-08-19T14:33:18.022 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:33:18.023 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:33:18.023 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:33:18.023 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.023 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.023 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.26191.0:139041 6.8 e569/556 6:1ccc8732:::benchmark_data_smithi045_97937_object31768:head v 569'8799, mlcod=515'2087)", 2024-08-19T14:33:18.023 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:31:16.006537+0000", 2024-08-19T14:33:18.023 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 121.940473277, 2024-08-19T14:33:18.023 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.046317775999999998, 2024-08-19T14:33:18.023 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:33:18.023 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:33:18.023 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:33:18.024 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:33:18.024 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.024 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:33:18.024 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:31:16.006537+0000", 2024-08-19T14:33:18.024 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:33:18.024 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.024 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.024 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:33:18.024 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:31:16.006537+0000", 2024-08-19T14:33:18.024 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:33:18.024 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.025 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.025 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:33:18.025 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:31:16.006537+0000", 2024-08-19T14:33:18.025 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0900000000000002e-07 2024-08-19T14:33:18.025 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.025 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.025 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:33:18.025 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:31:16.006542+0000", 2024-08-19T14:33:18.025 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0980000000000001e-06 2024-08-19T14:33:18.025 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.026 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.026 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:33:18.026 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:31:16.006551+0000", 2024-08-19T14:33:18.026 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.2560000000000002e-06 2024-08-19T14:33:18.026 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.026 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.026 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:33:18.026 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:31:16.006556+0000", 2024-08-19T14:33:18.026 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.6119999999999998e-06 2024-08-19T14:33:18.026 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.026 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.026 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:33:18.027 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:31:16.010399+0000", 2024-08-19T14:33:18.027 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.003843249 2024-08-19T14:33:18.027 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.027 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.027 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:33:18.027 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:31:16.010417+0000", 2024-08-19T14:33:18.027 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7521e-05 2024-08-19T14:33:18.027 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.027 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.027 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:33:18.027 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:31:16.052817+0000", 2024-08-19T14:33:18.028 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.042399629000000001 2024-08-19T14:33:18.028 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.028 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.028 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:33:18.028 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:31:16.052855+0000", 2024-08-19T14:33:18.028 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.7901999999999997e-05 2024-08-19T14:33:18.028 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:33:18.028 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:33:18.029 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:33:18.029 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.029 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.029 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.26191.0:168503 6.8 e577/556 6:10e97d8a:::benchmark_data_smithi045_97937_object61230:head v 577'10676, mlcod=522'3890)", 2024-08-19T14:33:18.029 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:32:04.485005+0000", 2024-08-19T14:33:18.029 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 73.462004938999996, 2024-08-19T14:33:18.029 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.049155524999999999, 2024-08-19T14:33:18.029 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:33:18.029 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:33:18.029 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:33:18.030 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:33:18.030 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.030 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:33:18.030 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:04.485005+0000", 2024-08-19T14:33:18.030 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:33:18.030 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.030 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.030 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:33:18.030 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:04.485005+0000", 2024-08-19T14:33:18.030 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:33:18.030 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.031 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.031 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:33:18.031 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:04.485006+0000", 2024-08-19T14:33:18.031 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.0399999999999996e-07 2024-08-19T14:33:18.031 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.031 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.031 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:33:18.031 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:04.485011+0000", 2024-08-19T14:33:18.031 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1240000000000004e-06 2024-08-19T14:33:18.031 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.031 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.032 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:33:18.032 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:04.485021+0000", 2024-08-19T14:33:18.032 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.8220000000000002e-06 2024-08-19T14:33:18.032 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.032 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.032 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:33:18.032 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:04.485027+0000", 2024-08-19T14:33:18.032 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.3890000000000002e-06 2024-08-19T14:33:18.032 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.032 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.032 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:33:18.033 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:04.490410+0000", 2024-08-19T14:33:18.033 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0053833359999999999 2024-08-19T14:33:18.033 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.033 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.033 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:33:18.033 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:04.490431+0000", 2024-08-19T14:33:18.033 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0605e-05 2024-08-19T14:33:18.033 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.033 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.033 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:33:18.033 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:04.534123+0000", 2024-08-19T14:33:18.034 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.043692166999999997 2024-08-19T14:33:18.034 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.034 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.034 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:33:18.034 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:04.534161+0000", 2024-08-19T14:33:18.034 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.7478000000000003e-05 2024-08-19T14:33:18.034 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:33:18.034 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:33:18.034 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:33:18.034 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.035 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.035 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.26191.0:168508 6.8 e577/556 6:164fb7a4:::benchmark_data_smithi045_97937_object61235:head v 577'10677, mlcod=522'3890)", 2024-08-19T14:33:18.035 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:32:04.490323+0000", 2024-08-19T14:33:18.035 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 73.456687436999999, 2024-08-19T14:33:18.035 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.043866335999999999, 2024-08-19T14:33:18.035 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:33:18.035 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:33:18.035 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:33:18.035 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:33:18.035 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.035 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:33:18.036 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:04.490323+0000", 2024-08-19T14:33:18.036 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:33:18.036 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.036 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.036 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:33:18.036 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:04.490323+0000", 2024-08-19T14:33:18.036 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:33:18.036 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.036 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.036 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:33:18.036 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:04.490324+0000", 2024-08-19T14:33:18.037 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0729999999999999e-06 2024-08-19T14:33:18.037 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.037 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.037 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:33:18.037 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:04.490332+0000", 2024-08-19T14:33:18.037 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.9529999999999999e-06 2024-08-19T14:33:18.037 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.037 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.037 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:33:18.037 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:04.490344+0000", 2024-08-19T14:33:18.037 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.26e-05 2024-08-19T14:33:18.038 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.038 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.038 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:33:18.038 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:04.490351+0000", 2024-08-19T14:33:18.038 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.8580000000000002e-06 2024-08-19T14:33:18.038 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.038 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.038 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:33:18.038 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:04.498033+0000", 2024-08-19T14:33:18.038 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0076819660000000001 2024-08-19T14:33:18.039 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.039 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.039 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:33:18.039 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:04.498052+0000", 2024-08-19T14:33:18.039 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9273999999999999e-05 2024-08-19T14:33:18.039 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.039 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.039 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:33:18.039 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:04.534163+0000", 2024-08-19T14:33:18.039 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.036110999999999997 2024-08-19T14:33:18.039 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.040 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.040 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:33:18.040 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:04.534189+0000", 2024-08-19T14:33:18.040 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.5612000000000001e-05 2024-08-19T14:33:18.040 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:33:18.040 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:33:18.040 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:33:18.040 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.040 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.040 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.26191.0:189354 6.3 e584/556 6:c0ccbda7:::benchmark_data_smithi045_97937_object82081:head v 584'11859, mlcod=532'5183)", 2024-08-19T14:33:18.041 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:32:35.820018+0000", 2024-08-19T14:33:18.041 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 42.126991594000003, 2024-08-19T14:33:18.041 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.043216037999999998, 2024-08-19T14:33:18.041 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:33:18.041 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:33:18.041 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:33:18.041 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:33:18.041 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.041 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:33:18.041 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:35.820018+0000", 2024-08-19T14:33:18.041 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:33:18.042 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.042 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.042 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:33:18.042 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:35.820018+0000", 2024-08-19T14:33:18.042 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:33:18.042 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.042 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.042 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:33:18.042 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:35.820019+0000", 2024-08-19T14:33:18.042 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.14e-07 2024-08-19T14:33:18.042 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.043 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.043 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:33:18.043 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:35.820026+0000", 2024-08-19T14:33:18.043 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.6739999999999997e-06 2024-08-19T14:33:18.043 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.043 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.043 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:33:18.043 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:35.820031+0000", 2024-08-19T14:33:18.043 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.7060000000000003e-06 2024-08-19T14:33:18.043 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.043 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.044 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:33:18.044 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:35.820033+0000", 2024-08-19T14:33:18.044 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.5440000000000001e-06 2024-08-19T14:33:18.044 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.044 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.044 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:33:18.044 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:35.842250+0000", 2024-08-19T14:33:18.044 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.022216799999999998 2024-08-19T14:33:18.044 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.044 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.044 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:33:18.045 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:35.842268+0000", 2024-08-19T14:33:18.045 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7645000000000001e-05 2024-08-19T14:33:18.045 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.045 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.045 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:33:18.045 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:35.863198+0000", 2024-08-19T14:33:18.045 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.020929627999999999 2024-08-19T14:33:18.045 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.045 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.045 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:33:18.045 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:35.863234+0000", 2024-08-19T14:33:18.045 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.6927000000000001e-05 2024-08-19T14:33:18.046 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:33:18.046 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:33:18.046 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:33:18.046 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.046 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.046 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.26191.0:189375 6.8 e584/556 6:194e17a4:::benchmark_data_smithi045_97937_object82102:head v 584'11985, mlcod=532'5266)", 2024-08-19T14:33:18.046 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:32:35.820059+0000", 2024-08-19T14:33:18.046 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 42.126951388999998, 2024-08-19T14:33:18.046 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.046071972000000003, 2024-08-19T14:33:18.046 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:33:18.046 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:33:18.047 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:33:18.047 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:33:18.047 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.047 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:33:18.047 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:35.820059+0000", 2024-08-19T14:33:18.047 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:33:18.047 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.047 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.047 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:33:18.047 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:35.820059+0000", 2024-08-19T14:33:18.047 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:33:18.047 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.048 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.048 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:33:18.048 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:35.820059+0000", 2024-08-19T14:33:18.048 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1300000000000001e-07 2024-08-19T14:33:18.048 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.048 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.048 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:33:18.048 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:35.820065+0000", 2024-08-19T14:33:18.048 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.4819999999999999e-06 2024-08-19T14:33:18.048 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.048 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.048 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:33:18.049 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:35.820069+0000", 2024-08-19T14:33:18.049 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.478e-06 2024-08-19T14:33:18.049 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.049 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.049 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:33:18.049 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:35.820071+0000", 2024-08-19T14:33:18.049 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.5210000000000001e-06 2024-08-19T14:33:18.049 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.049 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.049 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:33:18.049 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:35.847650+0000", 2024-08-19T14:33:18.049 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.027578458 2024-08-19T14:33:18.050 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.050 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.050 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:33:18.050 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:35.847666+0000", 2024-08-19T14:33:18.050 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.624e-05 2024-08-19T14:33:18.050 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.050 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.050 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:33:18.050 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:35.866075+0000", 2024-08-19T14:33:18.050 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.018409149 2024-08-19T14:33:18.050 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.051 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.051 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:33:18.051 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:35.866131+0000", 2024-08-19T14:33:18.051 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.5531000000000001e-05 2024-08-19T14:33:18.051 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:33:18.051 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:33:18.051 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:33:18.051 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.051 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.051 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.26191.0:189393 6.8 e584/556 6:1c05cfe2:::benchmark_data_smithi045_97937_object82120:head v 584'11986, mlcod=532'5266)", 2024-08-19T14:33:18.051 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:32:35.820075+0000", 2024-08-19T14:33:18.051 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 42.126934769999998, 2024-08-19T14:33:18.052 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.046094296999999999, 2024-08-19T14:33:18.052 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:33:18.052 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:33:18.052 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:33:18.052 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:33:18.052 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.052 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:33:18.052 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:35.820075+0000", 2024-08-19T14:33:18.052 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:33:18.052 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.052 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.052 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:33:18.053 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:35.820075+0000", 2024-08-19T14:33:18.053 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:33:18.053 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.053 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.053 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:33:18.053 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:35.820075+0000", 2024-08-19T14:33:18.053 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4000000000000001e-07 2024-08-19T14:33:18.053 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.053 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.053 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:33:18.053 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:35.820078+0000", 2024-08-19T14:33:18.053 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1229999999999998e-06 2024-08-19T14:33:18.054 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.054 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.054 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:33:18.054 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:35.820080+0000", 2024-08-19T14:33:18.054 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.9359999999999999e-06 2024-08-19T14:33:18.054 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.054 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.054 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:33:18.054 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:35.820083+0000", 2024-08-19T14:33:18.054 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2249999999999999e-06 2024-08-19T14:33:18.054 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.055 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.055 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:33:18.055 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:35.853086+0000", 2024-08-19T14:33:18.055 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.033003757000000002 2024-08-19T14:33:18.055 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.055 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.055 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:33:18.055 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:35.853118+0000", 2024-08-19T14:33:18.055 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.1463e-05 2024-08-19T14:33:18.055 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.055 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.056 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:33:18.056 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:35.866136+0000", 2024-08-19T14:33:18.056 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.013018333999999999 2024-08-19T14:33:18.056 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.056 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.056 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:33:18.056 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:35.866170+0000", 2024-08-19T14:33:18.056 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.3318999999999997e-05 2024-08-19T14:33:18.056 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:33:18.056 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:33:18.056 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:33:18.057 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.057 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.057 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.26191.0:189394 6.8 e584/556 6:172de5ab:::benchmark_data_smithi045_97937_object82121:head v 584'11987, mlcod=532'5266)", 2024-08-19T14:33:18.057 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:32:35.820086+0000", 2024-08-19T14:33:18.057 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 42.126924097, 2024-08-19T14:33:18.057 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.057390129999999998, 2024-08-19T14:33:18.057 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:33:18.057 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:33:18.057 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:33:18.057 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:33:18.057 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.057 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:33:18.058 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:35.820086+0000", 2024-08-19T14:33:18.058 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:33:18.058 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.058 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.058 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:33:18.058 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:35.820086+0000", 2024-08-19T14:33:18.058 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:33:18.058 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.058 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.058 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:33:18.058 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:35.820086+0000", 2024-08-19T14:33:18.058 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1600000000000001e-07 2024-08-19T14:33:18.059 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.059 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.059 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:33:18.059 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:35.820093+0000", 2024-08-19T14:33:18.059 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.9439999999999999e-06 2024-08-19T14:33:18.059 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.059 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.059 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:33:18.059 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:35.820097+0000", 2024-08-19T14:33:18.059 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.687e-06 2024-08-19T14:33:18.059 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.059 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.060 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:33:18.060 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:35.820099+0000", 2024-08-19T14:33:18.060 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2989999999999999e-06 2024-08-19T14:33:18.060 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.060 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.060 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:33:18.060 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:35.859262+0000", 2024-08-19T14:33:18.060 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.039163146000000003 2024-08-19T14:33:18.060 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.060 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.060 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:33:18.061 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:35.859287+0000", 2024-08-19T14:33:18.061 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.5320000000000002e-05 2024-08-19T14:33:18.061 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.061 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.061 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:33:18.061 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:35.877442+0000", 2024-08-19T14:33:18.061 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.018154724000000001 2024-08-19T14:33:18.061 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.061 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.061 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:33:18.061 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:35.877476+0000", 2024-08-19T14:33:18.061 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.3893999999999998e-05 2024-08-19T14:33:18.061 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:33:18.062 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:33:18.062 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:33:18.062 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.062 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.062 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.26191.0:189395 6.8 e584/556 6:159adabe:::benchmark_data_smithi045_97937_object82122:head v 584'11988, mlcod=532'5266)", 2024-08-19T14:33:18.062 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:32:35.820103+0000", 2024-08-19T14:33:18.062 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 42.12690705, 2024-08-19T14:33:18.062 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.057401354000000002, 2024-08-19T14:33:18.062 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:33:18.062 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:33:18.062 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:33:18.063 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:33:18.063 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.063 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:33:18.063 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:35.820103+0000", 2024-08-19T14:33:18.063 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:33:18.063 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.063 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.063 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:33:18.063 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:35.820103+0000", 2024-08-19T14:33:18.063 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:33:18.063 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.064 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.064 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:33:18.064 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:35.820103+0000", 2024-08-19T14:33:18.064 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4399999999999999e-07 2024-08-19T14:33:18.064 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.064 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.064 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:33:18.064 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:35.820105+0000", 2024-08-19T14:33:18.064 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9750000000000001e-06 2024-08-19T14:33:18.064 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.064 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.064 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:33:18.064 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:35.820111+0000", 2024-08-19T14:33:18.065 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.8719999999999999e-06 2024-08-19T14:33:18.065 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.065 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.065 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:33:18.065 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:35.820113+0000", 2024-08-19T14:33:18.065 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.317e-06 2024-08-19T14:33:18.065 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.065 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.065 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:33:18.065 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:35.866218+0000", 2024-08-19T14:33:18.065 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.046105184 2024-08-19T14:33:18.066 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.066 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.066 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:33:18.066 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:35.866247+0000", 2024-08-19T14:33:18.066 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.8759999999999999e-05 2024-08-19T14:33:18.066 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.066 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.066 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:33:18.066 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:35.877483+0000", 2024-08-19T14:33:18.066 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.011235642000000001 2024-08-19T14:33:18.066 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.067 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.067 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:33:18.067 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:35.877504+0000", 2024-08-19T14:33:18.067 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1460000000000001e-05 2024-08-19T14:33:18.067 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:33:18.067 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:33:18.067 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:33:18.067 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.067 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.067 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDPGPush(6.8 584/556 [PushOp(6:160cc344:::benchmark_data_smithi045_97937_object83204:head, version: 532'5268, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:160cc344:::benchmark_data_smithi045_97937_object83204:head@532'5268, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-19T14:33:18.067 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:32:35.820128+0000", 2024-08-19T14:33:18.067 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 42.126882262000002, 2024-08-19T14:33:18.068 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.060743908999999999, 2024-08-19T14:33:18.068 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:33:18.068 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:33:18.068 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-08-19T14:33:18.068 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:33:18.068 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.068 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:33:18.068 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:35.820128+0000", 2024-08-19T14:33:18.068 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:33:18.068 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.068 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.068 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:33:18.069 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:35.820128+0000", 2024-08-19T14:33:18.069 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:33:18.069 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.069 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.069 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:33:18.069 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:35.820128+0000", 2024-08-19T14:33:18.069 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5599999999999999e-07 2024-08-19T14:33:18.069 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.069 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.069 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:33:18.069 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:35.820173+0000", 2024-08-19T14:33:18.069 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.4617000000000003e-05 2024-08-19T14:33:18.070 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.070 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.070 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:33:18.070 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:35.820187+0000", 2024-08-19T14:33:18.070 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4929000000000001e-05 2024-08-19T14:33:18.070 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.070 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.070 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:33:18.070 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:35.820201+0000", 2024-08-19T14:33:18.070 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3380999999999999e-05 2024-08-19T14:33:18.071 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.071 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.071 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:33:18.071 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:35.877565+0000", 2024-08-19T14:33:18.071 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.057364149000000003 2024-08-19T14:33:18.071 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.071 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.072 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:33:18.072 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:35.877574+0000", 2024-08-19T14:33:18.072 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.9209999999999998e-06 2024-08-19T14:33:18.073 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.073 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.073 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:33:18.073 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:35.880872+0000", 2024-08-19T14:33:18.073 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0032977560000000002 2024-08-19T14:33:18.073 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:33:18.073 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:33:18.073 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:33:18.073 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.073 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.073 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.26191.0:203708 6.3 e589/556 6:cea2b389:::benchmark_data_smithi045_97937_object96435:head v 589'12781, mlcod=540'6234)", 2024-08-19T14:33:18.074 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:32:56.374257+0000", 2024-08-19T14:33:18.074 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 21.572752584, 2024-08-19T14:33:18.074 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.025874665000000002, 2024-08-19T14:33:18.074 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:33:18.074 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:33:18.074 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:33:18.074 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:33:18.074 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.075 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:33:18.075 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:56.374257+0000", 2024-08-19T14:33:18.075 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:33:18.075 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.075 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.075 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:33:18.075 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:56.374257+0000", 2024-08-19T14:33:18.075 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:33:18.075 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.075 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.075 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:33:18.076 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:56.374258+0000", 2024-08-19T14:33:18.076 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.3099999999999997e-07 2024-08-19T14:33:18.076 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.076 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.076 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:33:18.076 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:56.374265+0000", 2024-08-19T14:33:18.076 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.3980000000000004e-06 2024-08-19T14:33:18.076 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.076 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.076 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:33:18.077 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:56.374276+0000", 2024-08-19T14:33:18.077 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1333e-05 2024-08-19T14:33:18.077 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.077 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.077 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:33:18.077 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:56.374282+0000", 2024-08-19T14:33:18.077 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.4679999999999997e-06 2024-08-19T14:33:18.077 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.077 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.077 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:33:18.077 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:56.376815+0000", 2024-08-19T14:33:18.077 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0025327610000000001 2024-08-19T14:33:18.078 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.078 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.078 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:33:18.078 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:56.376841+0000", 2024-08-19T14:33:18.078 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.6074000000000001e-05 2024-08-19T14:33:18.078 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.078 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.078 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:33:18.078 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:56.400092+0000", 2024-08-19T14:33:18.078 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.023251244000000001 2024-08-19T14:33:18.078 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.078 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.079 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:33:18.079 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:56.400132+0000", 2024-08-19T14:33:18.079 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.9656000000000001e-05 2024-08-19T14:33:18.079 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:33:18.079 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:33:18.079 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:33:18.079 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.079 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.079 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.26191.0:205690 6.3 e590/556 6:c45243e7:::benchmark_data_smithi045_97937_object98417:head v 590'12899, mlcod=540'6373)", 2024-08-19T14:33:18.079 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:32:59.001184+0000", 2024-08-19T14:33:18.079 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 18.945826295, 2024-08-19T14:33:18.079 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.029390666999999999, 2024-08-19T14:33:18.080 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:33:18.080 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:33:18.080 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:33:18.080 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:33:18.080 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.080 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:33:18.080 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:59.001184+0000", 2024-08-19T14:33:18.080 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:33:18.080 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.080 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.080 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:33:18.080 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:59.001184+0000", 2024-08-19T14:33:18.080 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:33:18.081 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.081 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.081 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:33:18.081 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:59.001184+0000", 2024-08-19T14:33:18.081 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.9999999999999998e-07 2024-08-19T14:33:18.081 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.081 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.081 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:33:18.081 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:59.001189+0000", 2024-08-19T14:33:18.081 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.7859999999999996e-06 2024-08-19T14:33:18.081 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.081 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.082 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:33:18.082 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:59.001198+0000", 2024-08-19T14:33:18.082 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.6379999999999993e-06 2024-08-19T14:33:18.082 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.082 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.082 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:33:18.082 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:59.001204+0000", 2024-08-19T14:33:18.082 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.1940000000000003e-06 2024-08-19T14:33:18.082 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.082 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.082 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:33:18.083 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:59.003170+0000", 2024-08-19T14:33:18.083 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0019666000000000002 2024-08-19T14:33:18.083 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.083 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.083 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:33:18.083 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:59.003195+0000", 2024-08-19T14:33:18.083 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.4156e-05 2024-08-19T14:33:18.083 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.083 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.083 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:33:18.083 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:59.030537+0000", 2024-08-19T14:33:18.083 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.027342862999999999 2024-08-19T14:33:18.083 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:33:18.084 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:33:18.084 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:33:18.084 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:32:59.030574+0000", 2024-08-19T14:33:18.084 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.693e-05 2024-08-19T14:33:18.084 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:33:18.084 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:33:18.084 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:33:18.084 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:33:18.084 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:33:18.084 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:33:18.085 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:18.085 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:33:18.186 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:33:18.212 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:18.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:18.213 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:33:18.314 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:33:18.335 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:18.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:18.337 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:33:18.438 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:33:18.459 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:18.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:18.460 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:33:18.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:18 smithi045 ceph-mon[20143]: 1.4 scrub starts 2024-08-19T14:33:18.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:18 smithi045 ceph-mon[20143]: 1.4 scrub ok 2024-08-19T14:33:18.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:18 smithi045 ceph-mon[20143]: 1.2 scrub starts 2024-08-19T14:33:18.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:18 smithi045 ceph-mon[20143]: 1.2 scrub ok 2024-08-19T14:33:18.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:18 smithi045 ceph-mon[20143]: 1.c scrub starts 2024-08-19T14:33:18.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:18 smithi045 ceph-mon[20143]: 1.c scrub ok 2024-08-19T14:33:18.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:18 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 3379/6 objects degraded (56316.667%), 2 pgs degraded, 15 pgs undersized (PG_DEGRADED) 2024-08-19T14:33:18.561 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:33:18.582 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:18.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:18.583 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:33:18.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:18 smithi031 ceph-mon[16336]: 1.4 scrub starts 2024-08-19T14:33:18.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:18 smithi031 ceph-mon[16336]: 1.4 scrub ok 2024-08-19T14:33:18.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:18 smithi031 ceph-mon[16336]: 1.2 scrub starts 2024-08-19T14:33:18.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:18 smithi031 ceph-mon[16336]: 1.2 scrub ok 2024-08-19T14:33:18.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:18 smithi031 ceph-mon[16336]: 1.c scrub starts 2024-08-19T14:33:18.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:18 smithi031 ceph-mon[16336]: 1.c scrub ok 2024-08-19T14:33:18.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:18 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 3379/6 objects degraded (56316.667%), 2 pgs degraded, 15 pgs undersized (PG_DEGRADED) 2024-08-19T14:33:18.683 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:33:18.709 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-08-19T14:33:18.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:18.710 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:18.710 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:33:18.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:18 smithi032 ceph-mon[20004]: 1.4 scrub starts 2024-08-19T14:33:18.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:18 smithi032 ceph-mon[20004]: 1.4 scrub ok 2024-08-19T14:33:18.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:18 smithi032 ceph-mon[20004]: 1.2 scrub starts 2024-08-19T14:33:18.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:18 smithi032 ceph-mon[20004]: 1.2 scrub ok 2024-08-19T14:33:18.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:18 smithi032 ceph-mon[20004]: 1.c scrub starts 2024-08-19T14:33:18.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:18 smithi032 ceph-mon[20004]: 1.c scrub ok 2024-08-19T14:33:18.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:18 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 3379/6 objects degraded (56316.667%), 2 pgs degraded, 15 pgs undersized (PG_DEGRADED) 2024-08-19T14:33:18.811 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:33:18.837 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:18.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:18.838 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:33:18.939 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:33:18.959 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:33:18.964 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:18.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:18.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:33:19.066 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:33:19.091 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:19.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:19.092 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:33:19.193 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:33:19.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:19.217 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:19.218 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:33:19.319 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:33:19.345 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:19.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:19.346 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:33:19.399 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:19 smithi045 ceph-mon[20143]: 1.15 scrub starts 2024-08-19T14:33:19.400 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:19 smithi045 ceph-mon[20143]: 1.1e scrub starts 2024-08-19T14:33:19.400 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:19 smithi045 ceph-mon[20143]: 1.15 scrub ok 2024-08-19T14:33:19.400 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:19 smithi045 ceph-mon[20143]: 1.1e scrub ok 2024-08-19T14:33:19.400 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:19 smithi045 ceph-mon[20143]: pgmap v1399: 31 pgs: 9 active+undersized, 1 active+undersized+degraded, 21 active+clean; 607 KiB data, 9.8 GiB used, 795 GiB / 805 GiB avail; 1/6 objects degraded (16.667%) 2024-08-19T14:33:19.400 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:19 smithi045 ceph-mon[20143]: 1.1c scrub starts 2024-08-19T14:33:19.400 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:19 smithi045 ceph-mon[20143]: 1.1c scrub ok 2024-08-19T14:33:19.425 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:33:19.425 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [], 2024-08-19T14:33:19.425 INFO:teuthology.orchestra.run.smithi045.stdout: "num_ops": 0 2024-08-19T14:33:19.425 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:33:19.447 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:33:19.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:19.469 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:19.470 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:33:19.570 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:33:19.596 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:19.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:19.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:33:19.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:19 smithi031 ceph-mon[16336]: 1.15 scrub starts 2024-08-19T14:33:19.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:19 smithi031 ceph-mon[16336]: 1.1e scrub starts 2024-08-19T14:33:19.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:19 smithi031 ceph-mon[16336]: 1.15 scrub ok 2024-08-19T14:33:19.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:19 smithi031 ceph-mon[16336]: 1.1e scrub ok 2024-08-19T14:33:19.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:19 smithi031 ceph-mon[16336]: pgmap v1399: 31 pgs: 9 active+undersized, 1 active+undersized+degraded, 21 active+clean; 607 KiB data, 9.8 GiB used, 795 GiB / 805 GiB avail; 1/6 objects degraded (16.667%) 2024-08-19T14:33:19.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:19 smithi031 ceph-mon[16336]: 1.1c scrub starts 2024-08-19T14:33:19.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:19 smithi031 ceph-mon[16336]: 1.1c scrub ok 2024-08-19T14:33:19.698 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:33:19.720 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:19.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:19.721 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:33:19.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:19 smithi032 ceph-mon[20004]: 1.15 scrub starts 2024-08-19T14:33:19.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:19 smithi032 ceph-mon[20004]: 1.1e scrub starts 2024-08-19T14:33:19.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:19 smithi032 ceph-mon[20004]: 1.15 scrub ok 2024-08-19T14:33:19.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:19 smithi032 ceph-mon[20004]: 1.1e scrub ok 2024-08-19T14:33:19.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:19 smithi032 ceph-mon[20004]: pgmap v1399: 31 pgs: 9 active+undersized, 1 active+undersized+degraded, 21 active+clean; 607 KiB data, 9.8 GiB used, 795 GiB / 805 GiB avail; 1/6 objects degraded (16.667%) 2024-08-19T14:33:19.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:19 smithi032 ceph-mon[20004]: 1.1c scrub starts 2024-08-19T14:33:19.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:19 smithi032 ceph-mon[20004]: 1.1c scrub ok 2024-08-19T14:33:19.822 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:33:19.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:19.846 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:19.846 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:33:19.947 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:33:19.969 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:19.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:19.970 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:33:20.050 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-08-19T14:33:20.071 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:33:20.096 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:20.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:20.097 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:33:20.198 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:33:20.224 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:20.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:20.225 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:33:20.300 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:33:20.326 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:33:20.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:20.352 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:20.352 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:33:20.453 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:33:20.477 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:20.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:20.478 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:33:20.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:20 smithi045 ceph-mon[20143]: 1.d scrub starts 2024-08-19T14:33:20.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:20 smithi045 ceph-mon[20143]: 1.d scrub ok 2024-08-19T14:33:20.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:20 smithi045 ceph-mon[20143]: 1.3 scrub starts 2024-08-19T14:33:20.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:20 smithi045 ceph-mon[20143]: 1.3 scrub ok 2024-08-19T14:33:20.579 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:33:20.604 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:20.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:20.606 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:33:20.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:20 smithi031 ceph-mon[16336]: 1.d scrub starts 2024-08-19T14:33:20.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:20 smithi031 ceph-mon[16336]: 1.d scrub ok 2024-08-19T14:33:20.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:20 smithi031 ceph-mon[16336]: 1.3 scrub starts 2024-08-19T14:33:20.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:20 smithi031 ceph-mon[16336]: 1.3 scrub ok 2024-08-19T14:33:20.707 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:33:20.732 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:20.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:20.733 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:33:20.739 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:33:20.739 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [], 2024-08-19T14:33:20.739 INFO:teuthology.orchestra.run.smithi045.stdout: "complaint_time": 30, 2024-08-19T14:33:20.740 INFO:teuthology.orchestra.run.smithi045.stdout: "num_blocked_ops": 0 2024-08-19T14:33:20.740 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:33:20.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:20 smithi032 ceph-mon[20004]: 1.d scrub starts 2024-08-19T14:33:20.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:20 smithi032 ceph-mon[20004]: 1.d scrub ok 2024-08-19T14:33:20.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:20 smithi032 ceph-mon[20004]: 1.3 scrub starts 2024-08-19T14:33:20.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:20 smithi032 ceph-mon[20004]: 1.3 scrub ok 2024-08-19T14:33:20.760 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:33:20.760 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:33:20.835 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:33:20.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:20.860 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:20.860 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:33:20.961 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:33:20.987 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:20.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:20.988 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:33:21.089 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:33:21.115 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:21.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:21.117 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:33:21.218 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:33:21.244 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:21.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:21.245 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:33:21.252 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-08-19T14:33:21.253 INFO:tasks.radosbench:Beginning radosbench... 2024-08-19T14:33:21.253 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_5 2024-08-19T14:33:21.253 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd pool create unique_pool_5 16 2024-08-19T14:33:21.346 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:33:21.371 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:21.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:21.372 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:33:21.417 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:21 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:33:21.417 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:21 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:33:21.417 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:21 smithi031 ceph-mon[16336]: 1.10 scrub starts 2024-08-19T14:33:21.417 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:21 smithi031 ceph-mon[16336]: 1.10 scrub ok 2024-08-19T14:33:21.417 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:21 smithi031 ceph-mon[16336]: 1.12 scrub starts 2024-08-19T14:33:21.417 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:21 smithi031 ceph-mon[16336]: 1.12 scrub ok 2024-08-19T14:33:21.417 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:21 smithi031 ceph-mon[16336]: pgmap v1400: 31 pgs: 9 active+undersized, 1 active+undersized+degraded, 21 active+clean; 607 KiB data, 9.5 GiB used, 795 GiB / 805 GiB avail; 1/6 objects degraded (16.667%) 2024-08-19T14:33:21.417 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:21 smithi031 ceph-mon[16336]: 1.17 scrub starts 2024-08-19T14:33:21.417 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:21 smithi031 ceph-mon[16336]: 1.17 scrub ok 2024-08-19T14:33:21.417 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:21 smithi031 ceph-mon[16336]: osdmap e598: 12 total, 11 up, 9 in 2024-08-19T14:33:21.446 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-08-19T14:33:21.473 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:33:21.510 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:21.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:21.511 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:33:21.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:21 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:33:21.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:21 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:33:21.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:21 smithi045 ceph-mon[20143]: 1.10 scrub starts 2024-08-19T14:33:21.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:21 smithi045 ceph-mon[20143]: 1.10 scrub ok 2024-08-19T14:33:21.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:21 smithi045 ceph-mon[20143]: 1.12 scrub starts 2024-08-19T14:33:21.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:21 smithi045 ceph-mon[20143]: 1.12 scrub ok 2024-08-19T14:33:21.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:21 smithi045 ceph-mon[20143]: pgmap v1400: 31 pgs: 9 active+undersized, 1 active+undersized+degraded, 21 active+clean; 607 KiB data, 9.5 GiB used, 795 GiB / 805 GiB avail; 1/6 objects degraded (16.667%) 2024-08-19T14:33:21.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:21 smithi045 ceph-mon[20143]: 1.17 scrub starts 2024-08-19T14:33:21.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:21 smithi045 ceph-mon[20143]: 1.17 scrub ok 2024-08-19T14:33:21.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:21 smithi045 ceph-mon[20143]: osdmap e598: 12 total, 11 up, 9 in 2024-08-19T14:33:21.612 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:33:21.637 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:21.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:21.638 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:33:21.699 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:33:21.739 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:33:21.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:21 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:33:21.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:21 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:33:21.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:21 smithi032 ceph-mon[20004]: 1.10 scrub starts 2024-08-19T14:33:21.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:21 smithi032 ceph-mon[20004]: 1.10 scrub ok 2024-08-19T14:33:21.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:21 smithi032 ceph-mon[20004]: 1.12 scrub starts 2024-08-19T14:33:21.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:21 smithi032 ceph-mon[20004]: 1.12 scrub ok 2024-08-19T14:33:21.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:21 smithi032 ceph-mon[20004]: pgmap v1400: 31 pgs: 9 active+undersized, 1 active+undersized+degraded, 21 active+clean; 607 KiB data, 9.5 GiB used, 795 GiB / 805 GiB avail; 1/6 objects degraded (16.667%) 2024-08-19T14:33:21.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:21 smithi032 ceph-mon[20004]: 1.17 scrub starts 2024-08-19T14:33:21.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:21 smithi032 ceph-mon[20004]: 1.17 scrub ok 2024-08-19T14:33:21.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:21 smithi032 ceph-mon[20004]: osdmap e598: 12 total, 11 up, 9 in 2024-08-19T14:33:21.761 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:21.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:21.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:33:21.862 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:33:21.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:21.887 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:21.887 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:33:21.988 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:33:22.014 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:22.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:22.015 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:33:22.116 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:33:22.139 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:22.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:22.140 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:33:22.179 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:33:22.179 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:33:22.197 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:33:22.197 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:33:22.197 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-08-19T14:33:22.213 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:33:22.214 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:33:22.231 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:33:22.231 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:33:22.233 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:33:22.233 INFO:teuthology.orchestra.run.smithi045.stdout: "size": 20, 2024-08-19T14:33:22.233 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 600, 2024-08-19T14:33:22.233 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [] 2024-08-19T14:33:22.233 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:33:22.242 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:33:22.268 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:22.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:22.269 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:33:22.361 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:22 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2944195398' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:33:22.370 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:33:22.391 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:22.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:22.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:33:22.421 INFO:teuthology.orchestra.run.smithi031.stderr:no valid command found; 10 closest matches: 2024-08-19T14:33:22.421 INFO:teuthology.orchestra.run.smithi031.stderr:0 2024-08-19T14:33:22.422 INFO:teuthology.orchestra.run.smithi031.stderr:1 2024-08-19T14:33:22.422 INFO:teuthology.orchestra.run.smithi031.stderr:2 2024-08-19T14:33:22.422 INFO:teuthology.orchestra.run.smithi031.stderr:abort 2024-08-19T14:33:22.422 INFO:teuthology.orchestra.run.smithi031.stderr:assert 2024-08-19T14:33:22.422 INFO:teuthology.orchestra.run.smithi031.stderr:bench [] [] [] [] 2024-08-19T14:33:22.422 INFO:teuthology.orchestra.run.smithi031.stderr:bluefs debug_inject_read_zeros 2024-08-19T14:33:22.422 INFO:teuthology.orchestra.run.smithi031.stderr:bluefs files list 2024-08-19T14:33:22.422 INFO:teuthology.orchestra.run.smithi031.stderr:bluefs stats 2024-08-19T14:33:22.422 INFO:teuthology.orchestra.run.smithi031.stderr:bluestore allocator dump block 2024-08-19T14:33:22.422 INFO:teuthology.orchestra.run.smithi031.stderr:Error EINVAL: invalid command 2024-08-19T14:33:22.451 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:33:22.452 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:33:22.468 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:33:22.469 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:33:22.485 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:33:22.486 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:33:22.486 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:22 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2944195398' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:33:22.493 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:33:22.503 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:33:22.503 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:33:22.515 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:22.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:22.516 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:33:22.520 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:33:22.520 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:33:22.536 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-19T14:33:22.538 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:33:22.538 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:33:22.555 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:33:22.555 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:33:22.617 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:33:22.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:22.644 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:22.645 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:33:22.659 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:22 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2944195398' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:33:22.745 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:33:22.766 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:22.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:22.767 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:33:22.861 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-08-19T14:33:22.868 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:33:22.926 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:22.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:22.927 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:33:23.028 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:33:23.054 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:23.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:23.056 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:33:23.152 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-19T14:33:23.152 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-19T14:33:23.153 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph tell osd.5 'trim stale osdmaps' 2024-08-19T14:33:23.156 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:33:23.180 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:23.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:23.181 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:33:23.282 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:33:23.303 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:23.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:23.304 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:33:23.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:23 smithi031 ceph-mon[16336]: pgmap v1402: 31 pgs: 9 active+undersized, 1 active+undersized+degraded, 21 active+clean; 577 KiB data, 9.5 GiB used, 795 GiB / 805 GiB avail; 1/6 objects degraded (16.667%) 2024-08-19T14:33:23.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:23 smithi031 ceph-mon[16336]: 1.6 scrub starts 2024-08-19T14:33:23.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:23 smithi031 ceph-mon[16336]: 1.6 scrub ok 2024-08-19T14:33:23.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:23 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 1/6 objects degraded (16.667%), 1 pg degraded, 10 pgs undersized (PG_DEGRADED) 2024-08-19T14:33:23.405 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:33:23.426 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:23.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:23.428 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:33:23.428 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:23 smithi032 ceph-mon[20004]: pgmap v1402: 31 pgs: 9 active+undersized, 1 active+undersized+degraded, 21 active+clean; 577 KiB data, 9.5 GiB used, 795 GiB / 805 GiB avail; 1/6 objects degraded (16.667%) 2024-08-19T14:33:23.428 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:23 smithi032 ceph-mon[20004]: 1.6 scrub starts 2024-08-19T14:33:23.428 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:23 smithi032 ceph-mon[20004]: 1.6 scrub ok 2024-08-19T14:33:23.428 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:23 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 1/6 objects degraded (16.667%), 1 pg degraded, 10 pgs undersized (PG_DEGRADED) 2024-08-19T14:33:23.528 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:33:23.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:23 smithi045 ceph-mon[20143]: pgmap v1402: 31 pgs: 9 active+undersized, 1 active+undersized+degraded, 21 active+clean; 577 KiB data, 9.5 GiB used, 795 GiB / 805 GiB avail; 1/6 objects degraded (16.667%) 2024-08-19T14:33:23.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:23 smithi045 ceph-mon[20143]: 1.6 scrub starts 2024-08-19T14:33:23.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:23 smithi045 ceph-mon[20143]: 1.6 scrub ok 2024-08-19T14:33:23.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:23 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 1/6 objects degraded (16.667%), 1 pg degraded, 10 pgs undersized (PG_DEGRADED) 2024-08-19T14:33:23.552 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:23.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:23.553 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:33:23.653 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:33:23.675 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:23.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:23.676 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:33:23.777 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:33:23.799 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:23.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:23.800 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:33:23.886 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-19T14:33:23.887 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-19T14:33:23.901 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:33:23.923 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:23.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:23.924 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:33:23.926 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:33:23.973 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:33:23.974 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:33:24.025 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:33:24.045 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:24.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:24.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:33:24.148 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:33:24.169 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:24.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:24.170 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:33:24.271 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:33:24.297 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:24.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:24.298 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:33:24.399 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:33:24.420 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:24.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:24.421 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:33:24.458 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:24 smithi031 ceph-mon[16336]: 1.0 scrub starts 2024-08-19T14:33:24.458 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:24 smithi031 ceph-mon[16336]: 1.0 scrub ok 2024-08-19T14:33:24.521 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:33:24.542 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:24.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:24.543 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:33:24.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:24 smithi045 ceph-mon[20143]: 1.0 scrub starts 2024-08-19T14:33:24.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:24 smithi045 ceph-mon[20143]: 1.0 scrub ok 2024-08-19T14:33:24.644 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:33:24.667 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:24.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:24.668 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:33:24.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:24 smithi032 ceph-mon[20004]: 1.0 scrub starts 2024-08-19T14:33:24.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:24 smithi032 ceph-mon[20004]: 1.0 scrub ok 2024-08-19T14:33:24.769 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:33:24.791 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:24.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:24.792 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:33:24.893 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:33:24.915 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:24.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:24.916 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:33:24.996 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:33:24.996 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:33:25.016 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:33:25.039 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:25.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:25.040 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:33:25.141 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:33:25.163 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:25.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:25.164 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:33:25.265 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:33:25.291 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:25.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:25.292 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:33:25.393 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:33:25.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:25.420 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:25.420 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:33:25.522 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:33:25.547 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:25.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:25.548 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:33:25.597 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:25 smithi031 ceph-mon[16336]: pgmap v1403: 31 pgs: 9 active+undersized, 1 active+undersized+degraded, 21 active+clean; 577 KiB data, 9.4 GiB used, 795 GiB / 805 GiB avail; 1/6 objects degraded (16.667%) 2024-08-19T14:33:25.597 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:25 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3206308903' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:33:25.625 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-08-19T14:33:25.649 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:33:25.674 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:25.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:25.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:33:25.688 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is unset 2024-08-19T14:33:25.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:25 smithi032 ceph-mon[20004]: pgmap v1403: 31 pgs: 9 active+undersized, 1 active+undersized+degraded, 21 active+clean; 577 KiB data, 9.4 GiB used, 795 GiB / 805 GiB avail; 1/6 objects degraded (16.667%) 2024-08-19T14:33:25.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:25 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3206308903' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:33:25.776 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:33:25.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:25 smithi045 ceph-mon[20143]: pgmap v1403: 31 pgs: 9 active+undersized, 1 active+undersized+degraded, 21 active+clean; 577 KiB data, 9.4 GiB used, 795 GiB / 805 GiB avail; 1/6 objects degraded (16.667%) 2024-08-19T14:33:25.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:25 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3206308903' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:33:25.802 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:25.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:25.803 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:33:25.888 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:33:25.903 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:33:25.929 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:25.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:25.930 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:33:26.031 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:33:26.057 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:26.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:26.058 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:33:26.159 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:33:26.184 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:26.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:26.185 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:33:26.201 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:33:26.285 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:33:26.311 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:26.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:26.312 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:33:26.412 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:33:26.438 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:26.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:26.439 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:33:26.540 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:33:26.562 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:26.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:26.563 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:33:26.568 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:26 smithi031 ceph-mon[16336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:33:26.568 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:26 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3206308903' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:33:26.568 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:26 smithi031 ceph-mon[16336]: osdmap e599: 12 total, 11 up, 9 in 2024-08-19T14:33:26.663 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:33:26.689 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:26.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:26.690 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:33:26.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:26 smithi032 ceph-mon[20004]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:33:26.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:26 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3206308903' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:33:26.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:26 smithi032 ceph-mon[20004]: osdmap e599: 12 total, 11 up, 9 in 2024-08-19T14:33:26.791 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:33:26.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:26 smithi045 ceph-mon[20143]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:33:26.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:26 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3206308903' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:33:26.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:26 smithi045 ceph-mon[20143]: osdmap e599: 12 total, 11 up, 9 in 2024-08-19T14:33:26.817 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:26.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:26.818 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:33:26.919 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:33:26.940 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:26.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:26.941 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:33:27.041 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:33:27.067 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:27.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:27.068 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:33:27.135 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:33:27.150 INFO:teuthology.orchestra.run.smithi031.stderr:no valid command found; 10 closest matches: 2024-08-19T14:33:27.150 INFO:teuthology.orchestra.run.smithi031.stderr:0 2024-08-19T14:33:27.150 INFO:teuthology.orchestra.run.smithi031.stderr:1 2024-08-19T14:33:27.150 INFO:teuthology.orchestra.run.smithi031.stderr:2 2024-08-19T14:33:27.151 INFO:teuthology.orchestra.run.smithi031.stderr:abort 2024-08-19T14:33:27.151 INFO:teuthology.orchestra.run.smithi031.stderr:assert 2024-08-19T14:33:27.151 INFO:teuthology.orchestra.run.smithi031.stderr:bench [] [] [] [] 2024-08-19T14:33:27.151 INFO:teuthology.orchestra.run.smithi031.stderr:bluefs debug_inject_read_zeros 2024-08-19T14:33:27.151 INFO:teuthology.orchestra.run.smithi031.stderr:bluefs files list 2024-08-19T14:33:27.151 INFO:teuthology.orchestra.run.smithi031.stderr:bluefs stats 2024-08-19T14:33:27.151 INFO:teuthology.orchestra.run.smithi031.stderr:bluestore allocator dump block 2024-08-19T14:33:27.151 INFO:teuthology.orchestra.run.smithi031.stderr:Error EINVAL: invalid command 2024-08-19T14:33:27.169 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:33:27.194 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:27.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:27.195 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:33:27.296 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:33:27.320 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'unique_pool_5' created 2024-08-19T14:33:27.322 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:27.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:27.324 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:33:27.425 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:33:27.447 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:27.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:27.448 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:33:27.535 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:33:27.549 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:33:27.570 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:27.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:27.571 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:33:27.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:27 smithi031 ceph-mon[16336]: pgmap v1405: 31 pgs: 9 active+undersized, 1 active+undersized+degraded, 21 active+clean; 577 KiB data, 9.4 GiB used, 795 GiB / 805 GiB avail; 1/6 objects degraded (16.667%) 2024-08-19T14:33:27.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:27 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2170145011' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]: dispatch 2024-08-19T14:33:27.672 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:33:27.697 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:27.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:27.698 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:33:27.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:27 smithi032 ceph-mon[20004]: pgmap v1405: 31 pgs: 9 active+undersized, 1 active+undersized+degraded, 21 active+clean; 577 KiB data, 9.4 GiB used, 795 GiB / 805 GiB avail; 1/6 objects degraded (16.667%) 2024-08-19T14:33:27.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:27 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2170145011' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]: dispatch 2024-08-19T14:33:27.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:27 smithi045 ceph-mon[20143]: pgmap v1405: 31 pgs: 9 active+undersized, 1 active+undersized+degraded, 21 active+clean; 577 KiB data, 9.4 GiB used, 795 GiB / 805 GiB avail; 1/6 objects degraded (16.667%) 2024-08-19T14:33:27.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:27 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2170145011' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]: dispatch 2024-08-19T14:33:27.799 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:33:27.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:27.826 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:27.826 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:33:27.927 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:33:27.946 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd pool application enable unique_pool_5 rados --yes-i-really-mean-it || true 2024-08-19T14:33:27.953 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:27.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:27.954 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:33:28.055 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:33:28.080 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:28.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:28.081 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:33:28.182 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:33:28.207 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:28.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:28.208 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:33:28.309 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:33:28.335 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:28.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:28.336 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:33:28.437 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:33:28.459 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:28.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:28.460 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:33:28.483 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:33:28.483 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:33:28.483 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:28 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2170145011' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]': finished 2024-08-19T14:33:28.483 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:28 smithi031 ceph-mon[16336]: osdmap e600: 12 total, 11 up, 9 in 2024-08-19T14:33:28.560 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:33:28.586 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:28.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:28.587 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:33:28.600 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-19T14:33:28.688 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:33:28.714 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:28.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:28.715 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:33:28.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:28 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2170145011' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]': finished 2024-08-19T14:33:28.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:28 smithi032 ceph-mon[20004]: osdmap e600: 12 total, 11 up, 9 in 2024-08-19T14:33:28.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:28 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2170145011' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]': finished 2024-08-19T14:33:28.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:28 smithi045 ceph-mon[20143]: osdmap e600: 12 total, 11 up, 9 in 2024-08-19T14:33:28.816 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:33:28.842 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:28.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:28.843 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:33:28.944 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:33:28.970 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:28.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:28.971 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:33:29.072 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:33:29.094 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:29.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:29.095 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:33:29.132 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-08-19T14:33:29.196 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:33:29.236 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:29.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:29.237 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:33:29.338 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:33:29.364 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:29.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:29.365 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:33:29.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:29 smithi031 ceph-mon[16336]: 1.16 scrub starts 2024-08-19T14:33:29.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:29 smithi031 ceph-mon[16336]: 1.16 scrub ok 2024-08-19T14:33:29.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:29 smithi031 ceph-mon[16336]: pgmap v1407: 47 pgs: 16 unknown, 9 active+undersized, 1 active+undersized+degraded, 21 active+clean; 577 KiB data, 9.4 GiB used, 795 GiB / 805 GiB avail; 1/6 objects degraded (16.667%) 2024-08-19T14:33:29.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:29 smithi031 ceph-mon[16336]: osdmap e601: 12 total, 11 up, 9 in 2024-08-19T14:33:29.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:29 smithi031 ceph-mon[16336]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-19T14:33:29.466 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:33:29.491 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:29.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:29.492 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:33:29.541 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:33:29.576 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-19T14:33:29.577 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-19T14:33:29.577 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph tell osd.7 'trim stale osdmaps' 2024-08-19T14:33:29.593 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:33:29.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:29.642 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:29.643 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:33:29.744 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:33:29.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:29 smithi032 ceph-mon[20004]: 1.16 scrub starts 2024-08-19T14:33:29.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:29 smithi032 ceph-mon[20004]: 1.16 scrub ok 2024-08-19T14:33:29.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:29 smithi032 ceph-mon[20004]: pgmap v1407: 47 pgs: 16 unknown, 9 active+undersized, 1 active+undersized+degraded, 21 active+clean; 577 KiB data, 9.4 GiB used, 795 GiB / 805 GiB avail; 1/6 objects degraded (16.667%) 2024-08-19T14:33:29.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:29 smithi032 ceph-mon[20004]: osdmap e601: 12 total, 11 up, 9 in 2024-08-19T14:33:29.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:29 smithi032 ceph-mon[20004]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-19T14:33:29.770 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:29.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:29.771 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:33:29.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:29 smithi045 ceph-mon[20143]: 1.16 scrub starts 2024-08-19T14:33:29.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:29 smithi045 ceph-mon[20143]: 1.16 scrub ok 2024-08-19T14:33:29.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:29 smithi045 ceph-mon[20143]: pgmap v1407: 47 pgs: 16 unknown, 9 active+undersized, 1 active+undersized+degraded, 21 active+clean; 577 KiB data, 9.4 GiB used, 795 GiB / 805 GiB avail; 1/6 objects degraded (16.667%) 2024-08-19T14:33:29.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:29 smithi045 ceph-mon[20143]: osdmap e601: 12 total, 11 up, 9 in 2024-08-19T14:33:29.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:29 smithi045 ceph-mon[20143]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-19T14:33:29.872 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:33:29.897 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:29.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:29.898 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:33:29.999 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:33:30.025 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:30.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:30.026 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:33:30.127 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:33:30.152 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:30.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:30.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:33:30.254 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:33:30.279 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:30.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:30.280 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:33:30.317 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:33:30.318 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:33:30.318 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:33:30.318 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:33:30.318 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:33:30.318 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:33:30.318 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:33:30.318 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-08-19T14:33:30.318 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:33:30.319 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:33:30.319 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:33:30.319 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:33:30.319 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:33:30.319 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:33:30.319 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:33:30.319 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:33:30.319 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:33:30.319 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:33:30.320 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:33:30.320 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:33:30.320 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:33:30.320 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:33:30.320 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:33:30.382 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:33:30.408 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:30.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:30.409 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:33:30.510 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:33:30.536 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:30.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:30.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:33:30.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:30 smithi031 ceph-mon[16336]: osdmap e602: 12 total, 11 up, 9 in 2024-08-19T14:33:30.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:30 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/531410676' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:33:30.638 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:33:30.661 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:30.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:30.662 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:33:30.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:30 smithi032 ceph-mon[20004]: osdmap e602: 12 total, 11 up, 9 in 2024-08-19T14:33:30.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:30 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/531410676' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:33:30.763 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:33:30.788 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:30.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:30.789 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:33:30.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:30 smithi045 ceph-mon[20143]: osdmap e602: 12 total, 11 up, 9 in 2024-08-19T14:33:30.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:30 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/531410676' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:33:30.890 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:33:30.916 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:30.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:30.917 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:33:31.018 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:33:31.044 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:31.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:31.045 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:33:31.146 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:33:31.167 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:31.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:31.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:33:31.269 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:33:31.279 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:33:31.280 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:33:31.295 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:31.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:31.296 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:33:31.308 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:33:31.317 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-19T14:33:31.317 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-19T14:33:31.339 INFO:teuthology.orchestra.run.smithi031.stderr:enabled application 'rados' on pool 'unique_pool_5' 2024-08-19T14:33:31.396 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:33:31.422 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:31.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:31.423 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:33:31.524 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:33:31.545 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:31.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:31.546 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:33:31.581 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:31 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:33:31.581 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:31 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:33:31.581 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:31 smithi031 ceph-mon[16336]: pgmap v1410: 47 pgs: 8 unknown, 10 active+undersized, 1 active+undersized+degraded, 28 active+clean; 577 KiB data, 9.4 GiB used, 795 GiB / 805 GiB avail; 1/6 objects degraded (16.667%) 2024-08-19T14:33:31.581 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:31 smithi031 ceph-mon[16336]: osdmap e603: 12 total, 11 up, 9 in 2024-08-19T14:33:31.581 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:31 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/945008273' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-08-19T14:33:31.648 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:33:31.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:31.674 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:31.674 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:33:31.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:31 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:33:31.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:31 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:33:31.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:31 smithi032 ceph-mon[20004]: pgmap v1410: 47 pgs: 8 unknown, 10 active+undersized, 1 active+undersized+degraded, 28 active+clean; 577 KiB data, 9.4 GiB used, 795 GiB / 805 GiB avail; 1/6 objects degraded (16.667%) 2024-08-19T14:33:31.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:31 smithi032 ceph-mon[20004]: osdmap e603: 12 total, 11 up, 9 in 2024-08-19T14:33:31.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:31 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/945008273' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-08-19T14:33:31.775 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:33:31.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:31 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:33:31.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:31 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:33:31.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:31 smithi045 ceph-mon[20143]: pgmap v1410: 47 pgs: 8 unknown, 10 active+undersized, 1 active+undersized+degraded, 28 active+clean; 577 KiB data, 9.4 GiB used, 795 GiB / 805 GiB avail; 1/6 objects degraded (16.667%) 2024-08-19T14:33:31.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:31 smithi045 ceph-mon[20143]: osdmap e603: 12 total, 11 up, 9 in 2024-08-19T14:33:31.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:31 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/945008273' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-08-19T14:33:31.801 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:31.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:31.802 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:33:31.903 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:33:31.928 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:31.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:31.929 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:33:32.030 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:33:32.056 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:32.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:32.057 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:33:32.158 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:33:32.183 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:32.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:32.184 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:33:32.285 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:33:32.310 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:32.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:32.311 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:33:32.412 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:33:32.437 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:32.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:32.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:33:32.540 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:33:32.543 INFO:teuthology.orchestra.run.smithi031.stderr:no valid command found; 10 closest matches: 2024-08-19T14:33:32.543 INFO:teuthology.orchestra.run.smithi031.stderr:0 2024-08-19T14:33:32.543 INFO:teuthology.orchestra.run.smithi031.stderr:1 2024-08-19T14:33:32.543 INFO:teuthology.orchestra.run.smithi031.stderr:2 2024-08-19T14:33:32.543 INFO:teuthology.orchestra.run.smithi031.stderr:abort 2024-08-19T14:33:32.543 INFO:teuthology.orchestra.run.smithi031.stderr:assert 2024-08-19T14:33:32.544 INFO:teuthology.orchestra.run.smithi031.stderr:bench [] [] [] [] 2024-08-19T14:33:32.544 INFO:teuthology.orchestra.run.smithi031.stderr:bluefs debug_inject_read_zeros 2024-08-19T14:33:32.544 INFO:teuthology.orchestra.run.smithi031.stderr:bluefs files list 2024-08-19T14:33:32.544 INFO:teuthology.orchestra.run.smithi031.stderr:bluefs stats 2024-08-19T14:33:32.544 INFO:teuthology.orchestra.run.smithi031.stderr:bluestore allocator dump block 2024-08-19T14:33:32.544 INFO:teuthology.orchestra.run.smithi031.stderr:Error EINVAL: invalid command 2024-08-19T14:33:32.564 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:32.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:32.565 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:33:32.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:32 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/945008273' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-08-19T14:33:32.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:32 smithi031 ceph-mon[16336]: osdmap e604: 12 total, 11 up, 9 in 2024-08-19T14:33:32.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:32 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:33:32.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:32 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:33:32.666 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:33:32.691 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:32.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:32.692 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:33:32.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:32 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/945008273' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-08-19T14:33:32.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:32 smithi032 ceph-mon[20004]: osdmap e604: 12 total, 11 up, 9 in 2024-08-19T14:33:32.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:32 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:33:32.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:32 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:33:32.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:32 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/945008273' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-08-19T14:33:32.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:32 smithi045 ceph-mon[20143]: osdmap e604: 12 total, 11 up, 9 in 2024-08-19T14:33:32.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:32 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:33:32.791 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:32 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:33:32.792 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:33:32.815 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:32.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:32.817 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:33:32.917 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:33:32.941 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:32.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:32.942 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:33:33.043 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:33:33.068 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:33.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:33.069 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:33:33.170 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:33:33.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:33.192 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:33.193 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:33:33.198 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:33:33.198 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 20, 2024-08-19T14:33:33.199 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 600, 2024-08-19T14:33:33.199 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [ 2024-08-19T14:33:33.199 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.199 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:76306 6.9 6:93e0e6a2:::benchmark_data_smithi045_97937_object76305:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e525)", 2024-08-19T14:33:33.199 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:29:17.429480+0000", 2024-08-19T14:33:33.199 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 255.76701493600001, 2024-08-19T14:33:33.199 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.43747081700000001, 2024-08-19T14:33:33.199 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:33:33.199 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:33:33.199 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:33:33.199 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:33:33.199 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:33:33.199 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:33:33.200 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 76306 2024-08-19T14:33:33.200 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.200 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:33:33.200 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.200 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:33:33.200 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.429480+0000", 2024-08-19T14:33:33.200 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:33.200 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.200 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.200 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:33:33.200 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.429480+0000", 2024-08-19T14:33:33.200 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:33.200 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.200 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.201 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:33:33.201 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.429483+0000", 2024-08-19T14:33:33.201 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.5450000000000002e-06 2024-08-19T14:33:33.201 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.201 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.201 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:33:33.201 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.429662+0000", 2024-08-19T14:33:33.201 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000178944 2024-08-19T14:33:33.201 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.201 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.201 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:33:33.201 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.429683+0000", 2024-08-19T14:33:33.201 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.1327000000000001e-05 2024-08-19T14:33:33.201 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.202 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.202 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:33:33.202 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.429696+0000", 2024-08-19T14:33:33.202 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3359e-05 2024-08-19T14:33:33.202 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.202 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.203 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:33:33.203 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.433475+0000", 2024-08-19T14:33:33.203 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.003778989 2024-08-19T14:33:33.203 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.203 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.203 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:33:33.203 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.433491+0000", 2024-08-19T14:33:33.203 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5894999999999999e-05 2024-08-19T14:33:33.203 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.203 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.203 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:33:33.203 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.805712+0000", 2024-08-19T14:33:33.203 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.37222085500000002 2024-08-19T14:33:33.204 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.204 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.204 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:33:33.204 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.805869+0000", 2024-08-19T14:33:33.204 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00015659800000000001 2024-08-19T14:33:33.204 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.204 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.204 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 2,8", 2024-08-19T14:33:33.204 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.806040+0000", 2024-08-19T14:33:33.204 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00017071799999999999 2024-08-19T14:33:33.204 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.204 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.204 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:33.204 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.822089+0000", 2024-08-19T14:33:33.205 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.016049562 2024-08-19T14:33:33.205 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.205 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.205 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:33:33.205 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.828981+0000", 2024-08-19T14:33:33.205 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0068919150000000002 2024-08-19T14:33:33.205 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.205 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.205 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:33.205 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.862487+0000", 2024-08-19T14:33:33.205 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.033505842000000001 2024-08-19T14:33:33.205 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.205 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.206 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:33:33.206 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.862563+0000", 2024-08-19T14:33:33.206 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.6458000000000006e-05 2024-08-19T14:33:33.206 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.206 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.206 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:33:33.206 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.866951+0000", 2024-08-19T14:33:33.206 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0043878099999999998 2024-08-19T14:33:33.206 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:33.206 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:33:33.206 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:33.206 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.206 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.207 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:76308 6.9 6:9d9a3d15:::benchmark_data_smithi045_97937_object76307:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e525)", 2024-08-19T14:33:33.207 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:29:17.429724+0000", 2024-08-19T14:33:33.207 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 255.76677097000001, 2024-08-19T14:33:33.207 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.44172447399999998, 2024-08-19T14:33:33.207 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:33:33.207 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:33:33.207 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:33:33.207 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:33:33.207 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:33:33.207 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:33:33.207 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 76308 2024-08-19T14:33:33.207 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.207 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:33:33.208 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.208 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:33:33.208 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.429724+0000", 2024-08-19T14:33:33.208 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:33.208 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.208 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.208 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:33:33.208 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.429724+0000", 2024-08-19T14:33:33.208 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:33.208 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.208 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.208 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:33:33.208 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.429725+0000", 2024-08-19T14:33:33.208 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.99e-07 2024-08-19T14:33:33.208 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.209 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.209 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:33:33.209 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.429754+0000", 2024-08-19T14:33:33.209 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.9033000000000001e-05 2024-08-19T14:33:33.209 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.209 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.209 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:33:33.209 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.429763+0000", 2024-08-19T14:33:33.209 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.4569999999999994e-06 2024-08-19T14:33:33.209 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.210 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.210 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:33:33.210 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.429770+0000", 2024-08-19T14:33:33.210 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.229e-06 2024-08-19T14:33:33.210 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.210 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.210 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:33:33.210 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.433520+0000", 2024-08-19T14:33:33.210 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.003750761 2024-08-19T14:33:33.210 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.210 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.211 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:33:33.211 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.433528+0000", 2024-08-19T14:33:33.211 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.8800000000000008e-06 2024-08-19T14:33:33.211 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.211 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.211 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:33:33.211 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.813660+0000", 2024-08-19T14:33:33.211 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.380131846 2024-08-19T14:33:33.211 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.211 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.211 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:33:33.211 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.813821+0000", 2024-08-19T14:33:33.211 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000161237 2024-08-19T14:33:33.211 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.212 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.212 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 2,8", 2024-08-19T14:33:33.212 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.814009+0000", 2024-08-19T14:33:33.212 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000187622 2024-08-19T14:33:33.212 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.212 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.212 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:33.212 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.822117+0000", 2024-08-19T14:33:33.212 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0081084610000000008 2024-08-19T14:33:33.212 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.212 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.212 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:33:33.212 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.828993+0000", 2024-08-19T14:33:33.213 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0068753349999999998 2024-08-19T14:33:33.213 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.213 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.213 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:33.213 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.867012+0000", 2024-08-19T14:33:33.213 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.038018926000000001 2024-08-19T14:33:33.213 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.214 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.214 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:33:33.214 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.867114+0000", 2024-08-19T14:33:33.215 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000102434 2024-08-19T14:33:33.215 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.215 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.215 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:33:33.215 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.871449+0000", 2024-08-19T14:33:33.215 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0043346540000000003 2024-08-19T14:33:33.215 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:33.215 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:33:33.215 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:33.215 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.215 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.215 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:76295 6.0 6:09c9db57:::benchmark_data_smithi045_97937_object76294:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e526)", 2024-08-19T14:33:33.215 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:29:17.442554+0000", 2024-08-19T14:33:33.216 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 255.753941174, 2024-08-19T14:33:33.216 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.29366227700000003, 2024-08-19T14:33:33.216 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:33:33.216 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:33:33.216 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:33:33.216 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:33:33.216 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:33:33.216 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:33:33.216 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 76295 2024-08-19T14:33:33.216 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.216 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:33:33.216 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.216 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:33:33.216 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.442554+0000", 2024-08-19T14:33:33.217 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:33.217 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.217 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.217 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:33:33.217 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.442554+0000", 2024-08-19T14:33:33.217 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:33.217 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.217 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.217 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:33:33.217 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.442556+0000", 2024-08-19T14:33:33.217 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.9630000000000001e-06 2024-08-19T14:33:33.217 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.217 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.217 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:33:33.218 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.442602+0000", 2024-08-19T14:33:33.218 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.5590999999999998e-05 2024-08-19T14:33:33.218 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.218 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.218 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:33:33.218 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.442620+0000", 2024-08-19T14:33:33.218 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.8179e-05 2024-08-19T14:33:33.218 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.218 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.218 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:33:33.218 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.442631+0000", 2024-08-19T14:33:33.218 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1467e-05 2024-08-19T14:33:33.218 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.218 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.219 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:33:33.219 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.446010+0000", 2024-08-19T14:33:33.219 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.003378551 2024-08-19T14:33:33.219 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.219 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.219 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:33:33.219 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.446021+0000", 2024-08-19T14:33:33.219 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1204e-05 2024-08-19T14:33:33.219 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.219 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.219 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:33:33.219 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.735842+0000", 2024-08-19T14:33:33.219 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.28982053699999999 2024-08-19T14:33:33.219 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.220 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.220 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:33:33.220 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.736216+0000", 2024-08-19T14:33:33.220 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00037478499999999998 2024-08-19T14:33:33.220 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:33.220 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:33:33.220 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:33.220 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.220 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.220 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:76320 6.0 6:0c4767d7:::benchmark_data_smithi045_97937_object76319:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e526)", 2024-08-19T14:33:33.220 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:29:17.442650+0000", 2024-08-19T14:33:33.220 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 255.75384491099999, 2024-08-19T14:33:33.220 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.327560244, 2024-08-19T14:33:33.221 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:33:33.221 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:33:33.221 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:33:33.221 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:33:33.221 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:33:33.221 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:33:33.221 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 76320 2024-08-19T14:33:33.221 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.221 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:33:33.221 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.221 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:33:33.221 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.442650+0000", 2024-08-19T14:33:33.221 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:33.221 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.221 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.222 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:33:33.222 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.442650+0000", 2024-08-19T14:33:33.222 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:33.222 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.222 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.222 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:33:33.222 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.442651+0000", 2024-08-19T14:33:33.222 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.8699999999999995e-07 2024-08-19T14:33:33.222 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.222 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.222 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:33:33.222 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.442716+0000", 2024-08-19T14:33:33.222 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.5117999999999996e-05 2024-08-19T14:33:33.222 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.223 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.223 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:33:33.223 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.442725+0000", 2024-08-19T14:33:33.223 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.704e-06 2024-08-19T14:33:33.223 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.223 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.223 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:33:33.223 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.442731+0000", 2024-08-19T14:33:33.223 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.5810000000000002e-06 2024-08-19T14:33:33.223 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.223 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.223 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:33:33.223 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.446043+0000", 2024-08-19T14:33:33.224 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0033121180000000002 2024-08-19T14:33:33.224 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.224 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.224 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:33:33.224 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.446050+0000", 2024-08-19T14:33:33.224 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.2920000000000001e-06 2024-08-19T14:33:33.224 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.224 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.224 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:33:33.224 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.741896+0000", 2024-08-19T14:33:33.224 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.29584602100000001 2024-08-19T14:33:33.224 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.224 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.224 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:33:33.224 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.742326+0000", 2024-08-19T14:33:33.225 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00043030599999999998 2024-08-19T14:33:33.225 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.225 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.225 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 2,3", 2024-08-19T14:33:33.225 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.742898+0000", 2024-08-19T14:33:33.225 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00057183400000000001 2024-08-19T14:33:33.225 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.225 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.225 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:33.225 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.753358+0000", 2024-08-19T14:33:33.225 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.010460265999999999 2024-08-19T14:33:33.225 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.225 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.225 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:33:33.226 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.764669+0000", 2024-08-19T14:33:33.226 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.011310659000000001 2024-08-19T14:33:33.226 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.226 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.226 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:33.226 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.764724+0000", 2024-08-19T14:33:33.226 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.5470000000000003e-05 2024-08-19T14:33:33.226 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.226 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.226 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:33:33.226 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.764865+0000", 2024-08-19T14:33:33.226 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00014082400000000001 2024-08-19T14:33:33.226 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.227 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.227 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:33:33.227 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.770211+0000", 2024-08-19T14:33:33.227 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0053455639999999997 2024-08-19T14:33:33.227 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:33.227 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:33:33.227 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:33.227 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.227 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.227 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:76322 6.0 6:0c6cfef1:::benchmark_data_smithi045_97937_object76321:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e526)", 2024-08-19T14:33:33.227 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:29:17.442741+0000", 2024-08-19T14:33:33.227 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 255.75375462400001, 2024-08-19T14:33:33.227 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.35300911099999999, 2024-08-19T14:33:33.227 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:33:33.228 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:33:33.228 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:33:33.228 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:33:33.228 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:33:33.228 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:33:33.228 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 76322 2024-08-19T14:33:33.228 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.228 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:33:33.228 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.228 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:33:33.228 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.442741+0000", 2024-08-19T14:33:33.228 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:33.228 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.228 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.229 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:33:33.229 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.442741+0000", 2024-08-19T14:33:33.229 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:33.229 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.229 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.229 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:33:33.229 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.442741+0000", 2024-08-19T14:33:33.229 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4900000000000001e-07 2024-08-19T14:33:33.229 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.229 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.229 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:33:33.229 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.442792+0000", 2024-08-19T14:33:33.229 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0696999999999999e-05 2024-08-19T14:33:33.229 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.230 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.230 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:33:33.230 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.442800+0000", 2024-08-19T14:33:33.230 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.7740000000000004e-06 2024-08-19T14:33:33.230 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.230 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.230 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:33:33.230 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.442805+0000", 2024-08-19T14:33:33.230 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.9699999999999996e-06 2024-08-19T14:33:33.230 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.230 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.230 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:33:33.230 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.446071+0000", 2024-08-19T14:33:33.231 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0032653069999999998 2024-08-19T14:33:33.231 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.231 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.231 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:33:33.231 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.446077+0000", 2024-08-19T14:33:33.231 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.2389999999999999e-06 2024-08-19T14:33:33.231 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.231 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.231 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:33:33.231 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.748182+0000", 2024-08-19T14:33:33.231 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.30210518400000003 2024-08-19T14:33:33.231 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.231 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.231 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:33:33.232 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.748342+0000", 2024-08-19T14:33:33.232 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00015941800000000001 2024-08-19T14:33:33.232 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.232 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.232 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 2,3", 2024-08-19T14:33:33.232 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.748506+0000", 2024-08-19T14:33:33.232 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00016431700000000001 2024-08-19T14:33:33.232 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.232 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.232 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:33.232 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.753468+0000", 2024-08-19T14:33:33.232 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0049616770000000003 2024-08-19T14:33:33.232 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.232 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.233 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:33:33.233 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.770233+0000", 2024-08-19T14:33:33.233 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.016765174000000001 2024-08-19T14:33:33.233 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.233 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.233 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:33.233 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.791094+0000", 2024-08-19T14:33:33.233 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.020861570999999999 2024-08-19T14:33:33.233 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.233 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.233 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:33:33.233 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.791161+0000", 2024-08-19T14:33:33.234 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.6308000000000003e-05 2024-08-19T14:33:33.234 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.234 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.234 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:33:33.234 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.795750+0000", 2024-08-19T14:33:33.234 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0045890260000000004 2024-08-19T14:33:33.234 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:33.234 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:33:33.234 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:33.235 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.235 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.235 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:76325 6.0 6:086941de:::benchmark_data_smithi045_97937_object76324:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e526)", 2024-08-19T14:33:33.235 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:29:17.442814+0000", 2024-08-19T14:33:33.235 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 255.75368138100001, 2024-08-19T14:33:33.235 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.35774626300000001, 2024-08-19T14:33:33.235 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:33:33.235 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:33:33.235 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:33:33.235 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:33:33.235 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:33:33.235 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:33:33.235 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 76325 2024-08-19T14:33:33.235 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.236 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:33:33.236 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.236 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:33:33.236 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.442814+0000", 2024-08-19T14:33:33.236 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:33.236 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.236 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.236 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:33:33.236 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.442814+0000", 2024-08-19T14:33:33.236 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:33.236 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.237 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.237 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:33:33.237 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.442814+0000", 2024-08-19T14:33:33.237 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.1300000000000001e-07 2024-08-19T14:33:33.237 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.237 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.237 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:33:33.237 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.442870+0000", 2024-08-19T14:33:33.237 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.5402000000000003e-05 2024-08-19T14:33:33.237 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.237 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.237 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:33:33.238 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.442878+0000", 2024-08-19T14:33:33.238 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.9910000000000002e-06 2024-08-19T14:33:33.238 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.238 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.238 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:33:33.238 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.442883+0000", 2024-08-19T14:33:33.238 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.6670000000000001e-06 2024-08-19T14:33:33.238 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.238 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.238 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:33:33.238 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.446099+0000", 2024-08-19T14:33:33.238 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0032153910000000002 2024-08-19T14:33:33.238 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.239 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.239 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:33:33.239 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.446105+0000", 2024-08-19T14:33:33.239 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.1700000000000002e-06 2024-08-19T14:33:33.239 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.239 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.239 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:33:33.239 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.753503+0000", 2024-08-19T14:33:33.239 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.30739764600000002 2024-08-19T14:33:33.239 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.240 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.240 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:33:33.240 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.753662+0000", 2024-08-19T14:33:33.240 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00015969199999999999 2024-08-19T14:33:33.240 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.240 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.240 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 2,3", 2024-08-19T14:33:33.240 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.753833+0000", 2024-08-19T14:33:33.240 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000170311 2024-08-19T14:33:33.240 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.240 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.240 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:33:33.241 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.770238+0000", 2024-08-19T14:33:33.241 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.016405685 2024-08-19T14:33:33.241 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.241 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.241 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:33.241 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.772478+0000", 2024-08-19T14:33:33.241 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0022397609999999998 2024-08-19T14:33:33.241 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.241 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.241 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:33.241 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.795837+0000", 2024-08-19T14:33:33.242 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.023359081 2024-08-19T14:33:33.242 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.242 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.242 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:33:33.242 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.795923+0000", 2024-08-19T14:33:33.242 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.6125999999999995e-05 2024-08-19T14:33:33.242 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.242 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.242 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:33:33.242 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.800560+0000", 2024-08-19T14:33:33.243 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0046369269999999999 2024-08-19T14:33:33.243 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:33.243 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:33:33.243 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:33.243 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.243 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.243 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:76309 6.a 6:5713778d:::benchmark_data_smithi045_97937_object76308:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e526)", 2024-08-19T14:33:33.243 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:29:17.447616+0000", 2024-08-19T14:33:33.243 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 255.748878843, 2024-08-19T14:33:33.243 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.30109721699999997, 2024-08-19T14:33:33.243 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:33:33.244 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:33:33.244 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:33:33.244 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:33:33.244 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:33:33.244 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:33:33.244 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 76309 2024-08-19T14:33:33.244 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.244 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:33:33.244 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.245 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:33:33.245 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.447616+0000", 2024-08-19T14:33:33.245 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:33.245 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.245 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.245 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:33:33.245 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.447616+0000", 2024-08-19T14:33:33.245 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:33.245 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.246 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.246 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:33:33.246 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.447618+0000", 2024-08-19T14:33:33.246 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.739e-06 2024-08-19T14:33:33.246 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.246 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.246 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:33:33.246 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.447688+0000", 2024-08-19T14:33:33.246 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.0035000000000003e-05 2024-08-19T14:33:33.246 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.247 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.247 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:33:33.247 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.447702+0000", 2024-08-19T14:33:33.247 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3608000000000001e-05 2024-08-19T14:33:33.247 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.247 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.247 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:33:33.247 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.447712+0000", 2024-08-19T14:33:33.247 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0689999999999999e-05 2024-08-19T14:33:33.247 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.247 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.247 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:33:33.247 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.451794+0000", 2024-08-19T14:33:33.247 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0040818909999999998 2024-08-19T14:33:33.248 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.248 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.248 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:33:33.248 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.451806+0000", 2024-08-19T14:33:33.248 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2021000000000001e-05 2024-08-19T14:33:33.248 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.248 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.248 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:33:33.248 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.716205+0000", 2024-08-19T14:33:33.248 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.26439825099999997 2024-08-19T14:33:33.248 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.248 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.248 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:33:33.249 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.716381+0000", 2024-08-19T14:33:33.249 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00017665899999999999 2024-08-19T14:33:33.249 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.249 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.249 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 2,5", 2024-08-19T14:33:33.249 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.716552+0000", 2024-08-19T14:33:33.249 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00017030499999999999 2024-08-19T14:33:33.249 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.249 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.249 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:33.249 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.725424+0000", 2024-08-19T14:33:33.249 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0088728460000000002 2024-08-19T14:33:33.249 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.249 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.250 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:33:33.250 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.728499+0000", 2024-08-19T14:33:33.250 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.003074406 2024-08-19T14:33:33.250 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.250 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.250 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:33.250 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.745239+0000", 2024-08-19T14:33:33.250 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.016740488000000001 2024-08-19T14:33:33.250 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.250 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.250 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:33:33.250 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.745315+0000", 2024-08-19T14:33:33.250 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.5198999999999996e-05 2024-08-19T14:33:33.250 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.251 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.251 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:33:33.251 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.748714+0000", 2024-08-19T14:33:33.251 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0033990790000000002 2024-08-19T14:33:33.251 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:33.251 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:33:33.251 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:33.251 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.251 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.251 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:76319 6.a 6:526b7568:::benchmark_data_smithi045_97937_object76318:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e526)", 2024-08-19T14:33:33.251 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:29:17.447736+0000", 2024-08-19T14:33:33.251 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 255.74875887100001, 2024-08-19T14:33:33.251 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.30434869799999997, 2024-08-19T14:33:33.251 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:33:33.252 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:33:33.252 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:33:33.252 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:33:33.252 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:33:33.252 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:33:33.252 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 76319 2024-08-19T14:33:33.252 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.252 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:33:33.252 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.252 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:33:33.252 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.447736+0000", 2024-08-19T14:33:33.252 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:33.252 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.252 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.253 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:33:33.253 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.447736+0000", 2024-08-19T14:33:33.253 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:33.253 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.253 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.253 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:33:33.253 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.447737+0000", 2024-08-19T14:33:33.253 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.44e-07 2024-08-19T14:33:33.253 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.253 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.253 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:33:33.253 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.447791+0000", 2024-08-19T14:33:33.253 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.3844000000000002e-05 2024-08-19T14:33:33.253 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.254 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.254 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:33:33.254 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.447804+0000", 2024-08-19T14:33:33.254 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2948999999999999e-05 2024-08-19T14:33:33.254 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.254 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.254 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:33:33.254 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.447810+0000", 2024-08-19T14:33:33.254 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.6699999999999997e-06 2024-08-19T14:33:33.254 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.254 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.254 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:33:33.254 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.451832+0000", 2024-08-19T14:33:33.254 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0040218060000000002 2024-08-19T14:33:33.255 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.255 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.255 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:33:33.255 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.451840+0000", 2024-08-19T14:33:33.255 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.9370000000000004e-06 2024-08-19T14:33:33.255 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.255 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.255 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:33:33.255 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.722373+0000", 2024-08-19T14:33:33.255 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.270533148 2024-08-19T14:33:33.255 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.255 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.255 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:33:33.255 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.722632+0000", 2024-08-19T14:33:33.256 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00025911400000000001 2024-08-19T14:33:33.256 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.256 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.256 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 2,5", 2024-08-19T14:33:33.256 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.722801+0000", 2024-08-19T14:33:33.256 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00016822899999999999 2024-08-19T14:33:33.256 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.256 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.256 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:33:33.256 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.728511+0000", 2024-08-19T14:33:33.256 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0057099280000000004 2024-08-19T14:33:33.256 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.256 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.256 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:33.256 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.728918+0000", 2024-08-19T14:33:33.257 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00040767000000000001 2024-08-19T14:33:33.257 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.257 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.257 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:33.257 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.748769+0000", 2024-08-19T14:33:33.257 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.019850390999999998 2024-08-19T14:33:33.257 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.257 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.257 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:33:33.257 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.748822+0000", 2024-08-19T14:33:33.257 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.2957000000000001e-05 2024-08-19T14:33:33.257 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.257 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.257 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:33:33.258 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.752085+0000", 2024-08-19T14:33:33.258 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0032634109999999999 2024-08-19T14:33:33.258 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:33.258 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:33:33.258 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:33.258 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.258 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.258 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:174023 6.9 6:9fb9b151:::benchmark_data_smithi045_97937_object66750:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e580)", 2024-08-19T14:33:33.258 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:32:12.445756+0000", 2024-08-19T14:33:33.258 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 80.750739503000005, 2024-08-19T14:33:33.258 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.292813779, 2024-08-19T14:33:33.258 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:33:33.258 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:33:33.259 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:33:33.259 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:33:33.259 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:33:33.259 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:33:33.259 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 174023 2024-08-19T14:33:33.259 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.259 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:33:33.259 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.259 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:33:33.259 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.445756+0000", 2024-08-19T14:33:33.259 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:33.259 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.259 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.259 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:33:33.260 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.445756+0000", 2024-08-19T14:33:33.260 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:33.260 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.260 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.260 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:33:33.260 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.445757+0000", 2024-08-19T14:33:33.260 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2130000000000001e-06 2024-08-19T14:33:33.260 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.260 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.260 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:33:33.260 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.445762+0000", 2024-08-19T14:33:33.260 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.5839999999999996e-06 2024-08-19T14:33:33.260 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.260 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.261 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:33:33.261 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.445771+0000", 2024-08-19T14:33:33.261 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.0089999999999996e-06 2024-08-19T14:33:33.261 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.261 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.261 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:33:33.261 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.445775+0000", 2024-08-19T14:33:33.261 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7219999999999999e-06 2024-08-19T14:33:33.261 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.261 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.261 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:33:33.261 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.445819+0000", 2024-08-19T14:33:33.261 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.3497e-05 2024-08-19T14:33:33.261 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.262 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.262 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:33:33.262 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.445903+0000", 2024-08-19T14:33:33.262 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.4499999999999994e-05 2024-08-19T14:33:33.262 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.262 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.262 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 1,8", 2024-08-19T14:33:33.262 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.445973+0000", 2024-08-19T14:33:33.262 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.0239000000000003e-05 2024-08-19T14:33:33.262 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.262 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.262 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:33.262 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.449815+0000", 2024-08-19T14:33:33.263 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.003841461 2024-08-19T14:33:33.263 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.263 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.263 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:33:33.263 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.449950+0000", 2024-08-19T14:33:33.263 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000135377 2024-08-19T14:33:33.263 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.263 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.263 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:33.263 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.450730+0000", 2024-08-19T14:33:33.263 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00077977600000000004 2024-08-19T14:33:33.263 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.263 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.264 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:33:33.264 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.450763+0000", 2024-08-19T14:33:33.264 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.3161000000000003e-05 2024-08-19T14:33:33.264 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.264 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.264 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:33:33.264 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.738570+0000", 2024-08-19T14:33:33.264 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.28780623999999999 2024-08-19T14:33:33.264 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:33.264 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:33:33.264 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:33.264 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.264 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.264 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop_reply(client.26191.0:174023 6.9 e580/566 ondisk, result = 0)", 2024-08-19T14:33:33.265 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:32:12.450667+0000", 2024-08-19T14:33:33.265 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 80.745828610999993, 2024-08-19T14:33:33.265 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.28791044700000001, 2024-08-19T14:33:33.265 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:33:33.265 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:33:33.265 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-08-19T14:33:33.265 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:33:33.265 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.265 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:33:33.265 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.450667+0000", 2024-08-19T14:33:33.265 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:33.265 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.265 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.265 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:33:33.266 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.450667+0000", 2024-08-19T14:33:33.266 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:33.266 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.266 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.266 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:33:33.266 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.450667+0000", 2024-08-19T14:33:33.266 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.6100000000000001e-07 2024-08-19T14:33:33.266 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.266 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.266 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:33:33.266 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.450671+0000", 2024-08-19T14:33:33.266 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.4769999999999999e-06 2024-08-19T14:33:33.266 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.267 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.267 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:33:33.267 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.450682+0000", 2024-08-19T14:33:33.267 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1151e-05 2024-08-19T14:33:33.267 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.267 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.267 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:33:33.267 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.450687+0000", 2024-08-19T14:33:33.267 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7779999999999998e-06 2024-08-19T14:33:33.267 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.267 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.267 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:33:33.267 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.450722+0000", 2024-08-19T14:33:33.267 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.5212000000000001e-05 2024-08-19T14:33:33.267 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.268 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.268 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:33:33.268 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.450727+0000", 2024-08-19T14:33:33.268 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.8749999999999999e-06 2024-08-19T14:33:33.268 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.268 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.268 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:33:33.268 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.738577+0000", 2024-08-19T14:33:33.268 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.28785039299999998 2024-08-19T14:33:33.268 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:33.268 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:33:33.268 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:33.268 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.268 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.268 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.26191.0:174030 6.d e580/510 6:b7de3b0b:::benchmark_data_smithi045_97937_object66757:head v 580'10851, mlcod=580'10849)", 2024-08-19T14:33:33.269 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:32:12.457359+0000", 2024-08-19T14:33:33.269 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 80.739136176000002, 2024-08-19T14:33:33.269 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.31230491700000002, 2024-08-19T14:33:33.269 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:33:33.269 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:33:33.269 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:33:33.269 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:33:33.269 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.269 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:33:33.269 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.457359+0000", 2024-08-19T14:33:33.269 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:33.269 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.269 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.269 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:33:33.269 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.457359+0000", 2024-08-19T14:33:33.270 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:33.270 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.270 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.270 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:33:33.270 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.457361+0000", 2024-08-19T14:33:33.270 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.528e-06 2024-08-19T14:33:33.270 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.270 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.270 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:33:33.270 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.457370+0000", 2024-08-19T14:33:33.270 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.2180000000000005e-06 2024-08-19T14:33:33.270 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.270 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.270 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:33:33.271 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.738511+0000", 2024-08-19T14:33:33.271 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.28114076599999999 2024-08-19T14:33:33.271 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.271 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.271 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:33:33.271 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.738520+0000", 2024-08-19T14:33:33.271 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.5400000000000001e-06 2024-08-19T14:33:33.271 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.271 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.271 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:33:33.271 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.738582+0000", 2024-08-19T14:33:33.271 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.1537000000000004e-05 2024-08-19T14:33:33.271 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.271 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.272 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:33:33.272 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.738606+0000", 2024-08-19T14:33:33.272 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.4678e-05 2024-08-19T14:33:33.272 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.272 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.273 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:33:33.273 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.769625+0000", 2024-08-19T14:33:33.273 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.031018782000000002 2024-08-19T14:33:33.273 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.273 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.273 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:33:33.273 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.769664+0000", 2024-08-19T14:33:33.274 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.8868000000000001e-05 2024-08-19T14:33:33.274 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:33.274 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:33:33.274 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:33.274 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.274 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.274 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:174037 6.a 6:572c0c1a:::benchmark_data_smithi045_97937_object66764:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e580)", 2024-08-19T14:33:33.274 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:32:12.467095+0000", 2024-08-19T14:33:33.274 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 80.729400584000004, 2024-08-19T14:33:33.274 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.28614824, 2024-08-19T14:33:33.274 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:33:33.274 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:33:33.274 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:33:33.275 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:33:33.275 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:33:33.275 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:33:33.275 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 174037 2024-08-19T14:33:33.275 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.275 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:33:33.275 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.275 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:33:33.275 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.467095+0000", 2024-08-19T14:33:33.275 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:33.275 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.275 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.275 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:33:33.275 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.467095+0000", 2024-08-19T14:33:33.276 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:33.276 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.276 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.276 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:33:33.276 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.467096+0000", 2024-08-19T14:33:33.276 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.773e-06 2024-08-19T14:33:33.276 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.276 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.276 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:33:33.277 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.467104+0000", 2024-08-19T14:33:33.277 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.5569999999999998e-06 2024-08-19T14:33:33.277 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.277 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.277 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:33:33.277 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.738490+0000", 2024-08-19T14:33:33.277 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.27138640000000003 2024-08-19T14:33:33.277 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.277 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.277 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:33:33.277 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.738508+0000", 2024-08-19T14:33:33.277 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.7618e-05 2024-08-19T14:33:33.278 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.278 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.278 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:33:33.278 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.738578+0000", 2024-08-19T14:33:33.278 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.9646000000000004e-05 2024-08-19T14:33:33.278 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.278 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.278 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:33:33.278 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.738709+0000", 2024-08-19T14:33:33.278 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013147699999999999 2024-08-19T14:33:33.278 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.279 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.279 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 1,5", 2024-08-19T14:33:33.279 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.738832+0000", 2024-08-19T14:33:33.279 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000123168 2024-08-19T14:33:33.279 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.279 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.279 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:33.279 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.748897+0000", 2024-08-19T14:33:33.279 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.010065203999999999 2024-08-19T14:33:33.279 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.279 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.279 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:33.280 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.753055+0000", 2024-08-19T14:33:33.280 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0041572880000000003 2024-08-19T14:33:33.280 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.280 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.280 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:33:33.280 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.753071+0000", 2024-08-19T14:33:33.280 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6103000000000002e-05 2024-08-19T14:33:33.281 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.281 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.281 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:33:33.281 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.753106+0000", 2024-08-19T14:33:33.281 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.5447999999999997e-05 2024-08-19T14:33:33.281 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.281 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.281 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:33:33.281 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.753243+0000", 2024-08-19T14:33:33.282 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000136558 2024-08-19T14:33:33.282 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:33.282 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:33:33.282 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:33.282 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.282 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.282 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:186725 6.a 6:5538fe55:::benchmark_data_smithi045_97937_object79452:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e583)", 2024-08-19T14:33:33.282 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:32:31.435725+0000", 2024-08-19T14:33:33.282 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 61.760769967999998, 2024-08-19T14:33:33.282 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.28468506500000001, 2024-08-19T14:33:33.283 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:33:33.283 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:33:33.283 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:33:33.283 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:33:33.283 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:33:33.283 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:33:33.283 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 186725 2024-08-19T14:33:33.283 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.283 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:33:33.284 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.284 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:33:33.284 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.435725+0000", 2024-08-19T14:33:33.284 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:33.284 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.284 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.284 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:33:33.284 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.435725+0000", 2024-08-19T14:33:33.284 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:33.284 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.285 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.285 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:33:33.285 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.435726+0000", 2024-08-19T14:33:33.285 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.7199999999999998e-07 2024-08-19T14:33:33.285 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.285 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.285 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:33:33.285 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.435730+0000", 2024-08-19T14:33:33.285 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.8530000000000002e-06 2024-08-19T14:33:33.285 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.285 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.285 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:33:33.285 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.435746+0000", 2024-08-19T14:33:33.285 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6078000000000001e-05 2024-08-19T14:33:33.286 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.286 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.286 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:33:33.286 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.435753+0000", 2024-08-19T14:33:33.286 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.2860000000000002e-06 2024-08-19T14:33:33.286 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.286 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.286 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:33:33.286 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.542354+0000", 2024-08-19T14:33:33.286 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.106601 2024-08-19T14:33:33.286 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.286 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.286 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:33:33.286 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.542464+0000", 2024-08-19T14:33:33.287 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00010945 2024-08-19T14:33:33.287 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.287 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.287 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 1,5", 2024-08-19T14:33:33.287 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.542572+0000", 2024-08-19T14:33:33.287 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00010788 2024-08-19T14:33:33.287 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.287 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.287 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:33:33.287 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.553201+0000", 2024-08-19T14:33:33.287 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.010629868000000001 2024-08-19T14:33:33.287 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.287 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.288 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:33.288 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.553326+0000", 2024-08-19T14:33:33.288 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012446700000000001 2024-08-19T14:33:33.288 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.288 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.288 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:33.288 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.720241+0000", 2024-08-19T14:33:33.288 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.16691509299999999 2024-08-19T14:33:33.288 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.288 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.288 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:33:33.288 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.720298+0000", 2024-08-19T14:33:33.288 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.6777000000000002e-05 2024-08-19T14:33:33.288 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.289 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.289 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:33:33.289 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.720410+0000", 2024-08-19T14:33:33.289 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00011254100000000001 2024-08-19T14:33:33.289 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:33.289 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:33:33.289 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:33.289 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.289 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.289 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:186728 6.a 6:52d9dad4:::benchmark_data_smithi045_97937_object79455:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e583)", 2024-08-19T14:33:33.289 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:32:31.436585+0000", 2024-08-19T14:33:33.289 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 61.759910697000002, 2024-08-19T14:33:33.289 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.28396986200000002, 2024-08-19T14:33:33.289 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:33:33.290 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:33:33.290 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:33:33.290 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:33:33.290 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:33:33.290 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:33:33.290 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 186728 2024-08-19T14:33:33.290 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.290 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:33:33.290 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.290 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:33:33.290 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.436585+0000", 2024-08-19T14:33:33.290 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:33.290 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.291 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.291 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:33:33.291 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.436585+0000", 2024-08-19T14:33:33.291 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:33.291 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.291 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.291 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:33:33.291 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.436585+0000", 2024-08-19T14:33:33.291 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.9100000000000003e-07 2024-08-19T14:33:33.291 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.291 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.291 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:33:33.292 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.436590+0000", 2024-08-19T14:33:33.292 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.5210000000000004e-06 2024-08-19T14:33:33.292 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.292 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.292 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:33:33.292 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.436603+0000", 2024-08-19T14:33:33.292 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3254999999999999e-05 2024-08-19T14:33:33.292 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.292 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.292 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:33:33.292 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.436610+0000", 2024-08-19T14:33:33.292 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.8090000000000004e-06 2024-08-19T14:33:33.292 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.293 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.293 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:33:33.293 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.548046+0000", 2024-08-19T14:33:33.293 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.111436379 2024-08-19T14:33:33.293 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.293 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.293 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:33:33.293 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.548154+0000", 2024-08-19T14:33:33.293 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000107911 2024-08-19T14:33:33.293 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.293 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.293 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 1,5", 2024-08-19T14:33:33.293 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.548291+0000", 2024-08-19T14:33:33.294 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013718700000000001 2024-08-19T14:33:33.294 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.294 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.294 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:33:33.294 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.563156+0000", 2024-08-19T14:33:33.294 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.014865094000000001 2024-08-19T14:33:33.294 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.294 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.294 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:33.294 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.563191+0000", 2024-08-19T14:33:33.294 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.4768000000000003e-05 2024-08-19T14:33:33.294 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.294 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.295 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:33.295 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.720452+0000", 2024-08-19T14:33:33.295 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.15726029499999999 2024-08-19T14:33:33.295 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.295 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.295 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:33:33.295 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.720486+0000", 2024-08-19T14:33:33.295 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.4805999999999998e-05 2024-08-19T14:33:33.295 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.295 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.295 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:33:33.295 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.720554+0000", 2024-08-19T14:33:33.295 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.8046000000000005e-05 2024-08-19T14:33:33.296 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:33.296 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:33:33.296 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:33.296 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.296 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.296 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:186730 6.a 6:53c81287:::benchmark_data_smithi045_97937_object79457:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e583)", 2024-08-19T14:33:33.296 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:32:31.440667+0000", 2024-08-19T14:33:33.296 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 61.755828520000001, 2024-08-19T14:33:33.296 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.28015182900000002, 2024-08-19T14:33:33.296 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:33:33.296 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:33:33.296 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:33:33.296 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:33:33.296 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:33:33.297 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:33:33.297 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 186730 2024-08-19T14:33:33.297 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.297 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:33:33.297 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.297 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:33:33.297 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.440667+0000", 2024-08-19T14:33:33.297 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:33.297 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.297 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.297 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:33:33.297 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.440667+0000", 2024-08-19T14:33:33.297 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:33.298 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.298 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.298 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:33:33.298 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.440668+0000", 2024-08-19T14:33:33.298 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.395e-06 2024-08-19T14:33:33.298 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.298 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.298 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:33:33.298 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.440673+0000", 2024-08-19T14:33:33.298 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7520000000000004e-06 2024-08-19T14:33:33.298 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.298 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.298 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:33:33.299 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.440681+0000", 2024-08-19T14:33:33.299 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.4999999999999999e-06 2024-08-19T14:33:33.299 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.299 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.299 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:33:33.299 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.440688+0000", 2024-08-19T14:33:33.299 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.3099999999999997e-06 2024-08-19T14:33:33.299 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.299 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.299 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:33:33.299 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.553367+0000", 2024-08-19T14:33:33.299 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.112679032 2024-08-19T14:33:33.299 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.299 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.300 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:33:33.300 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.553507+0000", 2024-08-19T14:33:33.300 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00014019600000000001 2024-08-19T14:33:33.300 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.300 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.300 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 1,5", 2024-08-19T14:33:33.300 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.553643+0000", 2024-08-19T14:33:33.300 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013585800000000001 2024-08-19T14:33:33.300 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.300 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.300 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:33:33.300 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.563160+0000", 2024-08-19T14:33:33.300 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0095170180000000004 2024-08-19T14:33:33.301 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.301 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.301 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:33.301 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.563322+0000", 2024-08-19T14:33:33.301 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00016175299999999999 2024-08-19T14:33:33.301 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.301 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.301 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:33.301 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.720659+0000", 2024-08-19T14:33:33.301 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.15733791599999999 2024-08-19T14:33:33.301 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.301 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.301 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:33:33.301 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.720710+0000", 2024-08-19T14:33:33.302 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0176999999999997e-05 2024-08-19T14:33:33.302 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.302 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.302 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:33:33.302 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.720819+0000", 2024-08-19T14:33:33.302 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000108922 2024-08-19T14:33:33.302 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:33.302 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:33:33.302 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:33.302 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.302 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.302 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:186734 6.a 6:55f30169:::benchmark_data_smithi045_97937_object79461:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e583)", 2024-08-19T14:33:33.302 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:32:31.443487+0000", 2024-08-19T14:33:33.303 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 61.753008397000002, 2024-08-19T14:33:33.303 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.27757852900000002, 2024-08-19T14:33:33.303 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:33:33.303 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:33:33.303 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:33:33.303 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:33:33.303 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:33:33.303 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:33:33.303 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 186734 2024-08-19T14:33:33.303 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.303 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:33:33.303 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.303 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:33:33.303 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.443487+0000", 2024-08-19T14:33:33.304 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:33.304 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.304 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.304 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:33:33.304 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.443487+0000", 2024-08-19T14:33:33.304 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:33.304 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.304 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.304 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:33:33.304 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.443488+0000", 2024-08-19T14:33:33.304 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1710000000000001e-06 2024-08-19T14:33:33.305 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.305 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.305 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:33:33.305 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.443493+0000", 2024-08-19T14:33:33.305 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.4920000000000001e-06 2024-08-19T14:33:33.305 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.305 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.305 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:33:33.305 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.443509+0000", 2024-08-19T14:33:33.305 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5824999999999998e-05 2024-08-19T14:33:33.305 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.306 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.306 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:33:33.306 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.443516+0000", 2024-08-19T14:33:33.306 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.5119999999999997e-06 2024-08-19T14:33:33.306 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.306 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.306 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:33:33.306 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.557757+0000", 2024-08-19T14:33:33.306 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.114241205 2024-08-19T14:33:33.306 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.306 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.306 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:33:33.307 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.557880+0000", 2024-08-19T14:33:33.307 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012296799999999999 2024-08-19T14:33:33.307 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.307 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.307 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 1,5", 2024-08-19T14:33:33.307 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.558022+0000", 2024-08-19T14:33:33.307 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00014233299999999999 2024-08-19T14:33:33.307 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.307 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.308 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:33:33.308 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.563206+0000", 2024-08-19T14:33:33.308 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0051838509999999997 2024-08-19T14:33:33.308 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.308 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.308 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:33.308 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.567804+0000", 2024-08-19T14:33:33.308 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0045981900000000003 2024-08-19T14:33:33.308 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.308 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.308 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:33.308 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.720909+0000", 2024-08-19T14:33:33.308 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.15310435999999999 2024-08-19T14:33:33.308 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.309 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.309 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:33:33.309 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.720963+0000", 2024-08-19T14:33:33.309 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.3999000000000003e-05 2024-08-19T14:33:33.309 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.309 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.309 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:33:33.309 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.721065+0000", 2024-08-19T14:33:33.309 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000102623 2024-08-19T14:33:33.309 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:33.310 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:33:33.310 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:33.310 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.310 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.310 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:186742 6.a 6:57b0fc9a:::benchmark_data_smithi045_97937_object79469:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e583)", 2024-08-19T14:33:33.310 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:32:31.455602+0000", 2024-08-19T14:33:33.310 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 61.740893264, 2024-08-19T14:33:33.310 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.265680048, 2024-08-19T14:33:33.310 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:33:33.310 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:33:33.310 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:33:33.310 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:33:33.311 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:33:33.311 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:33:33.311 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 186742 2024-08-19T14:33:33.311 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.311 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:33:33.311 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.311 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:33:33.311 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.455602+0000", 2024-08-19T14:33:33.311 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:33.311 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.312 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.312 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:33:33.312 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.455602+0000", 2024-08-19T14:33:33.312 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:33.312 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.312 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.312 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:33:33.312 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.455603+0000", 2024-08-19T14:33:33.312 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4449999999999999e-06 2024-08-19T14:33:33.312 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.312 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.312 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:33:33.313 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.455608+0000", 2024-08-19T14:33:33.313 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.6419999999999998e-06 2024-08-19T14:33:33.313 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.313 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.313 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:33:33.313 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.455621+0000", 2024-08-19T14:33:33.313 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2839e-05 2024-08-19T14:33:33.313 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.313 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.313 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:33:33.314 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.455628+0000", 2024-08-19T14:33:33.314 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.8730000000000001e-06 2024-08-19T14:33:33.314 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.314 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.314 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:33:33.314 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.563374+0000", 2024-08-19T14:33:33.314 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.107745823 2024-08-19T14:33:33.314 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.314 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.315 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:33:33.315 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.563468+0000", 2024-08-19T14:33:33.315 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.4765000000000005e-05 2024-08-19T14:33:33.315 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.315 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.315 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 1,5", 2024-08-19T14:33:33.315 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.563536+0000", 2024-08-19T14:33:33.315 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.8077999999999994e-05 2024-08-19T14:33:33.315 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.315 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.316 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:33:33.316 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.568259+0000", 2024-08-19T14:33:33.316 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0047226550000000001 2024-08-19T14:33:33.316 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.316 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.316 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:33.316 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.568693+0000", 2024-08-19T14:33:33.316 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00043382700000000002 2024-08-19T14:33:33.316 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.316 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.316 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:33.317 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.721128+0000", 2024-08-19T14:33:33.317 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.152435293 2024-08-19T14:33:33.317 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.317 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.317 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:33:33.317 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.721179+0000", 2024-08-19T14:33:33.317 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1029000000000002e-05 2024-08-19T14:33:33.317 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.317 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.317 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:33:33.317 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.721282+0000", 2024-08-19T14:33:33.318 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000102779 2024-08-19T14:33:33.318 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:33.318 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:33:33.318 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:33.318 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.318 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.318 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:187356 6.a 6:575cb98d:::benchmark_data_smithi045_97937_object80083:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e583)", 2024-08-19T14:33:33.318 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:32:32.420297+0000", 2024-08-19T14:33:33.318 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 60.776198125000001, 2024-08-19T14:33:33.318 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.34771512199999999, 2024-08-19T14:33:33.318 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:33:33.318 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:33:33.318 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:33:33.318 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:33:33.319 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:33:33.319 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:33:33.319 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 187356 2024-08-19T14:33:33.319 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.319 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:33:33.319 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.319 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:33:33.319 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.420297+0000", 2024-08-19T14:33:33.319 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:33.319 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.319 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.319 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:33:33.319 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.420297+0000", 2024-08-19T14:33:33.319 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:33.319 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.320 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.320 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:33:33.320 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.420298+0000", 2024-08-19T14:33:33.320 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.8100000000000001e-07 2024-08-19T14:33:33.320 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.320 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.320 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:33:33.320 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.420302+0000", 2024-08-19T14:33:33.320 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.374e-06 2024-08-19T14:33:33.320 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.320 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.320 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:33:33.320 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.420317+0000", 2024-08-19T14:33:33.320 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4683e-05 2024-08-19T14:33:33.321 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.321 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.321 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:33:33.321 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.420324+0000", 2024-08-19T14:33:33.321 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.3869999999999997e-06 2024-08-19T14:33:33.321 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.321 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.321 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:33:33.321 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.551342+0000", 2024-08-19T14:33:33.321 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.13101795199999999 2024-08-19T14:33:33.321 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.321 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.322 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:33:33.322 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.551468+0000", 2024-08-19T14:33:33.322 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012597899999999999 2024-08-19T14:33:33.322 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.322 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.322 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 1,5", 2024-08-19T14:33:33.322 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.551599+0000", 2024-08-19T14:33:33.322 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000130369 2024-08-19T14:33:33.322 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.322 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.322 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:33:33.322 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.557051+0000", 2024-08-19T14:33:33.323 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0054523949999999996 2024-08-19T14:33:33.323 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.323 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.323 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:33.323 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.560585+0000", 2024-08-19T14:33:33.323 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0035343240000000001 2024-08-19T14:33:33.323 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.323 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.323 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:33.323 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.767809+0000", 2024-08-19T14:33:33.323 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.20722385400000001 2024-08-19T14:33:33.323 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.323 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.323 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:33:33.324 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.767907+0000", 2024-08-19T14:33:33.324 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.7996000000000002e-05 2024-08-19T14:33:33.324 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.324 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.324 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:33:33.324 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.768012+0000", 2024-08-19T14:33:33.324 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000104928 2024-08-19T14:33:33.324 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:33.324 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:33:33.324 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:33.324 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.324 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.324 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:187395 6.a 6:57126a4f:::benchmark_data_smithi045_97937_object80122:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e583)", 2024-08-19T14:33:33.324 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:32:32.474580+0000", 2024-08-19T14:33:33.324 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 60.721915625999998, 2024-08-19T14:33:33.324 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.293779544, 2024-08-19T14:33:33.325 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:33:33.325 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:33:33.325 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:33:33.325 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:33:33.325 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:33:33.325 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:33:33.325 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 187395 2024-08-19T14:33:33.325 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.325 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:33:33.325 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.325 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:33:33.325 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.474580+0000", 2024-08-19T14:33:33.325 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:33.325 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.326 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.326 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:33:33.326 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.474580+0000", 2024-08-19T14:33:33.326 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:33.326 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.326 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.326 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:33:33.326 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.474581+0000", 2024-08-19T14:33:33.326 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.268e-06 2024-08-19T14:33:33.326 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.326 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.326 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:33:33.326 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.474585+0000", 2024-08-19T14:33:33.326 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.3329999999999998e-06 2024-08-19T14:33:33.327 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.327 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.327 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:33:33.327 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.474600+0000", 2024-08-19T14:33:33.327 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5274999999999999e-05 2024-08-19T14:33:33.327 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.327 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.327 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:33:33.327 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.474609+0000", 2024-08-19T14:33:33.327 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.1950000000000003e-06 2024-08-19T14:33:33.327 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.327 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.327 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:33:33.327 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.560680+0000", 2024-08-19T14:33:33.328 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.086071163000000006 2024-08-19T14:33:33.328 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.328 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.328 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:33:33.328 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.560822+0000", 2024-08-19T14:33:33.328 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000141917 2024-08-19T14:33:33.328 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.328 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.328 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 1,5", 2024-08-19T14:33:33.328 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.560968+0000", 2024-08-19T14:33:33.328 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00014591300000000001 2024-08-19T14:33:33.328 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.328 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.329 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:33.329 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.576675+0000", 2024-08-19T14:33:33.329 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.015707216999999999 2024-08-19T14:33:33.329 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.329 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.329 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:33:33.329 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.576890+0000", 2024-08-19T14:33:33.329 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000214959 2024-08-19T14:33:33.329 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.329 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.329 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:33.329 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.768143+0000", 2024-08-19T14:33:33.329 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.191252961 2024-08-19T14:33:33.330 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.330 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.330 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:33:33.330 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.768243+0000", 2024-08-19T14:33:33.330 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000100589 2024-08-19T14:33:33.330 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.330 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.330 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:33:33.330 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.768359+0000", 2024-08-19T14:33:33.330 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000115754 2024-08-19T14:33:33.330 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:33.330 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:33:33.330 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:33.330 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.331 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.331 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:187415 6.a 6:512e1698:::benchmark_data_smithi045_97937_object80142:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e583)", 2024-08-19T14:33:33.331 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:32:32.505253+0000", 2024-08-19T14:33:33.331 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 60.691242197999998, 2024-08-19T14:33:33.331 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.26330838699999998, 2024-08-19T14:33:33.331 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:33:33.331 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:33:33.331 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:33:33.331 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:33:33.331 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:33:33.331 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:33:33.331 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 187415 2024-08-19T14:33:33.331 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.331 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:33:33.332 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.332 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:33:33.333 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.505253+0000", 2024-08-19T14:33:33.333 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:33.333 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.333 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.333 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:33:33.333 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.505253+0000", 2024-08-19T14:33:33.334 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:33.334 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.334 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.334 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:33:33.334 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.505254+0000", 2024-08-19T14:33:33.334 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2979999999999999e-06 2024-08-19T14:33:33.334 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.334 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.334 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:33:33.334 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.505259+0000", 2024-08-19T14:33:33.334 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1540000000000001e-06 2024-08-19T14:33:33.334 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.334 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.334 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:33:33.335 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.505275+0000", 2024-08-19T14:33:33.335 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5625999999999999e-05 2024-08-19T14:33:33.335 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.335 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.335 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:33:33.335 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.505282+0000", 2024-08-19T14:33:33.335 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.097e-06 2024-08-19T14:33:33.335 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.335 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.335 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:33:33.335 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.565866+0000", 2024-08-19T14:33:33.335 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.060583692000000001 2024-08-19T14:33:33.335 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.335 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.336 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:33:33.336 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.566565+0000", 2024-08-19T14:33:33.336 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00069894499999999997 2024-08-19T14:33:33.336 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.336 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.336 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 1,5", 2024-08-19T14:33:33.336 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.566749+0000", 2024-08-19T14:33:33.336 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00018427999999999999 2024-08-19T14:33:33.336 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.336 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.336 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:33.336 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.576876+0000", 2024-08-19T14:33:33.337 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.010126476000000001 2024-08-19T14:33:33.337 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.337 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.337 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:33:33.337 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.576898+0000", 2024-08-19T14:33:33.337 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.2382999999999999e-05 2024-08-19T14:33:33.337 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.337 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.337 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:33.337 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.768415+0000", 2024-08-19T14:33:33.337 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.19151680400000001 2024-08-19T14:33:33.337 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.337 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.337 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:33:33.338 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.768467+0000", 2024-08-19T14:33:33.338 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1916000000000002e-05 2024-08-19T14:33:33.338 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:33.338 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:33.338 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:33:33.338 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.768561+0000", 2024-08-19T14:33:33.338 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.4716000000000006e-05 2024-08-19T14:33:33.338 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:33.338 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:33:33.338 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:33.338 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:33.338 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:33:33.338 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:33:33.339 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:33:33.339 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:33:33.340 DEBUG:teuthology.orchestra.run.smithi045:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_5 bench 90 write' 2024-08-19T14:33:33.343 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-08-19T14:33:33.343 INFO:teuthology.orchestra.run:waiting for 3000 2024-08-19T14:33:33.365 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:33.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:33.366 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:33:33.440 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:hints = 1 2024-08-19T14:33:33.440 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-08-19T14:33:33.440 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Object prefix: benchmark_data_smithi045_115842 2024-08-19T14:33:33.440 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-19T14:33:33.440 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 0 0 0 0 0 0 - 0 2024-08-19T14:33:33.467 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:33:33.495 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:33.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:33.497 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:33:33.598 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:33:33.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:33 smithi031 ceph-mon[16336]: pgmap v1413: 47 pgs: 15 active+undersized, 1 active+undersized+degraded, 31 active+clean; 577 KiB data, 9.4 GiB used, 795 GiB / 805 GiB avail; 1/6 objects degraded (16.667%) 2024-08-19T14:33:33.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:33 smithi031 ceph-mon[16336]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-19T14:33:33.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:33 smithi031 ceph-mon[16336]: osdmap e605: 12 total, 11 up, 9 in 2024-08-19T14:33:33.622 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:33.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:33.623 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:33:33.724 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:33:33.751 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:33.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:33.752 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:33:33.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:33 smithi032 ceph-mon[20004]: pgmap v1413: 47 pgs: 15 active+undersized, 1 active+undersized+degraded, 31 active+clean; 577 KiB data, 9.4 GiB used, 795 GiB / 805 GiB avail; 1/6 objects degraded (16.667%) 2024-08-19T14:33:33.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:33 smithi032 ceph-mon[20004]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-19T14:33:33.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:33 smithi032 ceph-mon[20004]: osdmap e605: 12 total, 11 up, 9 in 2024-08-19T14:33:33.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:33 smithi045 ceph-mon[20143]: pgmap v1413: 47 pgs: 15 active+undersized, 1 active+undersized+degraded, 31 active+clean; 577 KiB data, 9.4 GiB used, 795 GiB / 805 GiB avail; 1/6 objects degraded (16.667%) 2024-08-19T14:33:33.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:33 smithi045 ceph-mon[20143]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-19T14:33:33.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:33 smithi045 ceph-mon[20143]: osdmap e605: 12 total, 11 up, 9 in 2024-08-19T14:33:33.853 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:33:33.881 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:33.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:33.883 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:33:33.984 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:33:34.026 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:34.027 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-19T14:33:34.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:34.027 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-19T14:33:34.027 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph tell osd.8 'trim stale osdmaps' 2024-08-19T14:33:34.028 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:33:34.129 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:33:34.172 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:34.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:34.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:33:34.274 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:33:34.299 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:34.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:34.300 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:33:34.401 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:33:34.422 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:34.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:34.423 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:33:34.440 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 1 16 3396 3380 211.226 211.25 0.00143345 0.00400876 2024-08-19T14:33:34.524 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:33:34.551 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:34.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:34.552 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:33:34.653 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:33:34.682 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:34.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:34.683 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:33:34.746 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:34 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 1/1692 objects degraded (0.059%), 1 pg degraded, 10 pgs undersized (PG_DEGRADED) 2024-08-19T14:33:34.775 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:34 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 1/1692 objects degraded (0.059%), 1 pg degraded, 10 pgs undersized (PG_DEGRADED) 2024-08-19T14:33:34.784 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:33:34.785 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-08-19T14:33:34.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:34 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 1/1692 objects degraded (0.059%), 1 pg degraded, 10 pgs undersized (PG_DEGRADED) 2024-08-19T14:33:34.864 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:34.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:34.865 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:33:34.966 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:33:35.009 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:35.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:35.010 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:33:35.111 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:33:35.136 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:35.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:35.137 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:33:35.238 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:33:35.263 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:35.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:35.264 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:33:35.365 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:33:35.390 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:35.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:35.391 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:33:35.440 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 2 16 6937 6921 216.259 221.312 0.00164662 0.00459442 2024-08-19T14:33:35.492 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:33:35.513 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:35.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:35.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:33:35.615 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:33:35.639 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:35.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:35.640 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:33:35.703 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:33:35.741 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:33:35.754 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:33:35.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:35 smithi032 ceph-mon[20004]: pgmap v1415: 47 pgs: 15 active+undersized, 1 active+undersized+degraded, 31 active+clean; 36 MiB data, 9.4 GiB used, 795 GiB / 805 GiB avail; 7.3 MiB/s wr, 117 op/s; 1/1692 objects degraded (0.059%) 2024-08-19T14:33:35.764 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:35.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:35.765 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:33:35.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:35 smithi045 ceph-mon[20143]: pgmap v1415: 47 pgs: 15 active+undersized, 1 active+undersized+degraded, 31 active+clean; 36 MiB data, 9.4 GiB used, 795 GiB / 805 GiB avail; 7.3 MiB/s wr, 117 op/s; 1/1692 objects degraded (0.059%) 2024-08-19T14:33:35.848 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:35 smithi031 ceph-mon[16336]: pgmap v1415: 47 pgs: 15 active+undersized, 1 active+undersized+degraded, 31 active+clean; 36 MiB data, 9.4 GiB used, 795 GiB / 805 GiB avail; 7.3 MiB/s wr, 117 op/s; 1/1692 objects degraded (0.059%) 2024-08-19T14:33:35.865 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:33:35.891 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:35.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:35.892 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:33:35.993 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:33:36.014 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:36.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:36.015 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:33:36.116 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:33:36.137 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:36.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:36.138 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:33:36.239 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:33:36.266 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:36.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:36.267 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:33:36.368 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:33:36.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:36.409 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:36.409 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:33:36.413 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is set 2024-08-19T14:33:36.440 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 3 16 8931 8915 185.712 124.625 0.00438839 0.00537566 2024-08-19T14:33:36.509 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:33:36.547 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:36.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:36.548 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:33:36.650 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:33:36.676 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:36.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:36.677 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:33:36.688 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:36 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/390744925' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:33:36.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:36 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/390744925' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:33:36.778 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:33:36.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:36 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/390744925' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:33:36.802 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:36.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:36.803 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:33:36.904 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:33:36.927 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:36.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:36.928 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:33:36.992 INFO:teuthology.orchestra.run.smithi031.stderr:no valid command found; 10 closest matches: 2024-08-19T14:33:36.992 INFO:teuthology.orchestra.run.smithi031.stderr:0 2024-08-19T14:33:36.992 INFO:teuthology.orchestra.run.smithi031.stderr:1 2024-08-19T14:33:36.992 INFO:teuthology.orchestra.run.smithi031.stderr:2 2024-08-19T14:33:36.993 INFO:teuthology.orchestra.run.smithi031.stderr:abort 2024-08-19T14:33:36.993 INFO:teuthology.orchestra.run.smithi031.stderr:assert 2024-08-19T14:33:36.993 INFO:teuthology.orchestra.run.smithi031.stderr:bench [] [] [] [] 2024-08-19T14:33:36.993 INFO:teuthology.orchestra.run.smithi031.stderr:bluefs debug_inject_read_zeros 2024-08-19T14:33:36.993 INFO:teuthology.orchestra.run.smithi031.stderr:bluefs files list 2024-08-19T14:33:36.993 INFO:teuthology.orchestra.run.smithi031.stderr:bluefs stats 2024-08-19T14:33:36.993 INFO:teuthology.orchestra.run.smithi031.stderr:bluestore allocator dump block 2024-08-19T14:33:36.993 INFO:teuthology.orchestra.run.smithi031.stderr:Error EINVAL: invalid command 2024-08-19T14:33:37.029 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:33:37.050 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:37.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:37.051 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:33:37.152 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:33:37.180 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:37.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:37.181 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:33:37.282 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:33:37.307 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:37.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:37.308 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:33:37.409 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:33:37.440 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 4 16 11780 11764 183.797 178.062 0.00715901 0.00543274 2024-08-19T14:33:37.452 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:37.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:37.453 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:33:37.554 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:33:37.575 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:37.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:37.576 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:33:37.677 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:33:37.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:37.714 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:37.714 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:33:37.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:37 smithi032 ceph-mon[20004]: 1.13 deep-scrub starts 2024-08-19T14:33:37.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:37 smithi032 ceph-mon[20004]: 1.13 deep-scrub ok 2024-08-19T14:33:37.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:37 smithi032 ceph-mon[20004]: pgmap v1416: 47 pgs: 10 active+undersized, 6 active+undersized+degraded, 31 active+clean; 212 MiB data, 9.7 GiB used, 795 GiB / 805 GiB avail; 35 MiB/s wr, 564 op/s; 1725/10170 objects degraded (16.962%) 2024-08-19T14:33:37.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:37 smithi032 ceph-mon[20004]: 1.e scrub starts 2024-08-19T14:33:37.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:37 smithi032 ceph-mon[20004]: 1.e scrub ok 2024-08-19T14:33:37.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:37 smithi032 ceph-mon[20004]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:33:37.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:37 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/390744925' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:33:37.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:37 smithi032 ceph-mon[20004]: osdmap e606: 12 total, 11 up, 9 in 2024-08-19T14:33:37.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:37 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:33:37.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:37 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:33:37.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:37 smithi045 ceph-mon[20143]: 1.13 deep-scrub starts 2024-08-19T14:33:37.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:37 smithi045 ceph-mon[20143]: 1.13 deep-scrub ok 2024-08-19T14:33:37.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:37 smithi045 ceph-mon[20143]: pgmap v1416: 47 pgs: 10 active+undersized, 6 active+undersized+degraded, 31 active+clean; 212 MiB data, 9.7 GiB used, 795 GiB / 805 GiB avail; 35 MiB/s wr, 564 op/s; 1725/10170 objects degraded (16.962%) 2024-08-19T14:33:37.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:37 smithi045 ceph-mon[20143]: 1.e scrub starts 2024-08-19T14:33:37.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:37 smithi045 ceph-mon[20143]: 1.e scrub ok 2024-08-19T14:33:37.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:37 smithi045 ceph-mon[20143]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:33:37.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:37 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/390744925' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:33:37.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:37 smithi045 ceph-mon[20143]: osdmap e606: 12 total, 11 up, 9 in 2024-08-19T14:33:37.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:37 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:33:37.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:37 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:33:37.814 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:33:37.839 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:37.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:37.840 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:33:37.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:37 smithi031 ceph-mon[16336]: 1.13 deep-scrub starts 2024-08-19T14:33:37.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:37 smithi031 ceph-mon[16336]: 1.13 deep-scrub ok 2024-08-19T14:33:37.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:37 smithi031 ceph-mon[16336]: pgmap v1416: 47 pgs: 10 active+undersized, 6 active+undersized+degraded, 31 active+clean; 212 MiB data, 9.7 GiB used, 795 GiB / 805 GiB avail; 35 MiB/s wr, 564 op/s; 1725/10170 objects degraded (16.962%) 2024-08-19T14:33:37.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:37 smithi031 ceph-mon[16336]: 1.e scrub starts 2024-08-19T14:33:37.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:37 smithi031 ceph-mon[16336]: 1.e scrub ok 2024-08-19T14:33:37.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:37 smithi031 ceph-mon[16336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:33:37.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:37 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/390744925' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:33:37.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:37 smithi031 ceph-mon[16336]: osdmap e606: 12 total, 11 up, 9 in 2024-08-19T14:33:37.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:37 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:33:37.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:37 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:33:37.940 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:33:37.980 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:37.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:37.981 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:33:38.082 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:33:38.108 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:38.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:38.110 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:33:38.210 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:33:38.235 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:38.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:38.236 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:33:38.337 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:33:38.374 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:38.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:38.375 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:33:38.440 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 5 15 14804 14789 184.848 189.062 0.00366248 0.00540409 2024-08-19T14:33:38.476 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:33:38.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:38.501 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:38.502 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:33:38.603 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:33:38.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:38 smithi031 ceph-mon[16336]: osdmap e607: 12 total, 11 up, 9 in 2024-08-19T14:33:38.629 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:38.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:38.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:33:38.732 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:33:38.757 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:38.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:38 smithi032 ceph-mon[20004]: osdmap e607: 12 total, 11 up, 9 in 2024-08-19T14:33:38.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:38.758 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:33:38.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:38 smithi045 ceph-mon[20143]: osdmap e607: 12 total, 11 up, 9 in 2024-08-19T14:33:38.859 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:33:38.898 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:38.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:38.900 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:33:39.001 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:33:39.024 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:39.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:39.025 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:33:39.126 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:33:39.153 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:39.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:39.154 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:33:39.255 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:33:39.290 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:39.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:39.291 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:33:39.392 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:33:39.428 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:39.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:39.430 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:33:39.440 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 6 15 18065 18050 188.006 203.812 0.00386205 0.00531331 2024-08-19T14:33:39.530 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:33:39.556 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:39.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:39.557 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:33:39.657 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:33:39.683 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:39.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:39.684 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:33:39.701 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:39 smithi031 ceph-mon[16336]: pgmap v1419: 47 pgs: 9 active+undersized, 7 active+undersized+degraded, 31 active+clean; 278 MiB data, 10 GiB used, 795 GiB / 805 GiB avail; 46 MiB/s wr, 740 op/s; 2253/13338 objects degraded (16.892%) 2024-08-19T14:33:39.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:39 smithi032 ceph-mon[20004]: pgmap v1419: 47 pgs: 9 active+undersized, 7 active+undersized+degraded, 31 active+clean; 278 MiB data, 10 GiB used, 795 GiB / 805 GiB avail; 46 MiB/s wr, 740 op/s; 2253/13338 objects degraded (16.892%) 2024-08-19T14:33:39.785 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:33:39.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:39 smithi045 ceph-mon[20143]: pgmap v1419: 47 pgs: 9 active+undersized, 7 active+undersized+degraded, 31 active+clean; 278 MiB data, 10 GiB used, 795 GiB / 805 GiB avail; 46 MiB/s wr, 740 op/s; 2253/13338 objects degraded (16.892%) 2024-08-19T14:33:39.810 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:39.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:39.811 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:33:39.912 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:33:39.937 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:39.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:39.938 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:33:40.039 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:33:40.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:40.061 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:40.061 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:33:40.162 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:33:40.189 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:40.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:40.190 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:33:40.291 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:33:40.329 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:40.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:40.331 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:33:40.432 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:33:40.440 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 7 16 20826 20810 185.788 172.5 0.00880266 0.00537688 2024-08-19T14:33:40.461 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:40.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:40.462 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:33:40.563 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:33:40.586 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:40.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:40.587 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:33:40.688 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:33:40.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:40.724 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:40.724 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:33:40.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:40 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:33:40.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:40 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:33:40.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:40 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:33:40.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:40 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:33:40.825 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:33:40.856 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:40.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:40.857 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:33:40.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:40 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:33:40.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:40 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:33:40.958 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:33:40.979 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:40.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:40.980 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:33:41.081 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:33:41.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:41.117 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:41.118 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:33:41.219 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:33:41.246 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:41.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:41.247 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:33:41.348 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:33:41.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:41.376 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:41.376 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:33:41.441 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 8 16 23159 23143 180.79 145.812 0.00646078 0.00552509 2024-08-19T14:33:41.477 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:33:41.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:41.499 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:41.499 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:33:41.542 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-19T14:33:41.600 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:33:41.624 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:41.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:41.625 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:33:41.726 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:33:41.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:41 smithi032 ceph-mon[20004]: pgmap v1420: 47 pgs: 9 active+undersized, 7 active+undersized+degraded, 31 active+clean; 737 MiB data, 11 GiB used, 794 GiB / 805 GiB avail; 95 MiB/s wr, 1.51k op/s; 3209/35334 objects degraded (9.082%) 2024-08-19T14:33:41.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:41 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 3209/35334 objects degraded (9.082%), 7 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-08-19T14:33:41.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:41 smithi032 ceph-mon[20004]: osdmap e608: 12 total, 11 up, 9 in 2024-08-19T14:33:41.762 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:41.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:41.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:33:41.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:41 smithi045 ceph-mon[20143]: pgmap v1420: 47 pgs: 9 active+undersized, 7 active+undersized+degraded, 31 active+clean; 737 MiB data, 11 GiB used, 794 GiB / 805 GiB avail; 95 MiB/s wr, 1.51k op/s; 3209/35334 objects degraded (9.082%) 2024-08-19T14:33:41.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:41 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 3209/35334 objects degraded (9.082%), 7 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-08-19T14:33:41.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:41 smithi045 ceph-mon[20143]: osdmap e608: 12 total, 11 up, 9 in 2024-08-19T14:33:41.864 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:33:41.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:41 smithi031 ceph-mon[16336]: pgmap v1420: 47 pgs: 9 active+undersized, 7 active+undersized+degraded, 31 active+clean; 737 MiB data, 11 GiB used, 794 GiB / 805 GiB avail; 95 MiB/s wr, 1.51k op/s; 3209/35334 objects degraded (9.082%) 2024-08-19T14:33:41.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:41 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 3209/35334 objects degraded (9.082%), 7 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-08-19T14:33:41.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:41 smithi031 ceph-mon[16336]: osdmap e608: 12 total, 11 up, 9 in 2024-08-19T14:33:41.890 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:41.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:41.891 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:33:41.992 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:33:42.018 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:42.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:42.019 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:33:42.120 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:33:42.146 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:42.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:42.147 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:33:42.248 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:33:42.269 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:42.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:42.270 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:33:42.295 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:33:42.296 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [ 2024-08-19T14:33:42.296 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:42.296 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.16962.0:19049 7.1 e607/600 7:80fd47eb:::benchmark_data_smithi045_115842_object19048:head v 607'1147, mlcod=607'1144)", 2024-08-19T14:33:42.296 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:33:39.769133+0000", 2024-08-19T14:33:42.296 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.0033989379999999998, 2024-08-19T14:33:42.296 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.003430734, 2024-08-19T14:33:42.296 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:33:42.296 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:33:42.296 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-08-19T14:33:42.296 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:33:42.296 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:42.296 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:33:42.296 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:33:39.769133+0000", 2024-08-19T14:33:42.297 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:42.297 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:42.297 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:42.297 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:33:42.297 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:33:39.769133+0000", 2024-08-19T14:33:42.297 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:42.297 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:42.297 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:42.297 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:33:42.297 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:33:39.769134+0000", 2024-08-19T14:33:42.297 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.0699999999999997e-07 2024-08-19T14:33:42.297 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:42.297 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:42.298 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:33:42.298 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:33:39.769196+0000", 2024-08-19T14:33:42.298 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.1735000000000004e-05 2024-08-19T14:33:42.298 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:42.298 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:42.298 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:33:42.298 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:33:39.769209+0000", 2024-08-19T14:33:42.298 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3764000000000001e-05 2024-08-19T14:33:42.298 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:42.298 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:42.298 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:33:42.298 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:33:39.769216+0000", 2024-08-19T14:33:42.298 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.8480000000000003e-06 2024-08-19T14:33:42.299 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:42.299 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:42.299 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:33:42.299 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:33:39.769383+0000", 2024-08-19T14:33:42.299 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00016639500000000001 2024-08-19T14:33:42.299 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:42.299 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:42.299 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:33:42.299 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:33:39.769401+0000", 2024-08-19T14:33:42.299 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.7767000000000001e-05 2024-08-19T14:33:42.299 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:42.299 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:33:42.299 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:42.300 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:42.300 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:42.300 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:19048 7.b 7:d80acc61:::benchmark_data_smithi045_115842_object19047:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e607)", 2024-08-19T14:33:42.300 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:33:39.768606+0000", 2024-08-19T14:33:42.300 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.0039258690000000002, 2024-08-19T14:33:42.300 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0040350710000000003, 2024-08-19T14:33:42.300 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:33:42.300 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:33:42.300 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "waiting for sub ops", 2024-08-19T14:33:42.300 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:33:42.300 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:33:42.300 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:33:42.300 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 19048 2024-08-19T14:33:42.300 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:42.301 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:33:42.301 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:42.301 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:33:42.301 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:33:39.768606+0000", 2024-08-19T14:33:42.301 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:42.301 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:42.301 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:42.301 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:33:42.301 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:33:39.768606+0000", 2024-08-19T14:33:42.302 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:42.302 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:42.302 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:42.302 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:33:42.302 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:33:39.768608+0000", 2024-08-19T14:33:42.302 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6509999999999999e-06 2024-08-19T14:33:42.302 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:42.302 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:42.302 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:33:42.302 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:33:39.769200+0000", 2024-08-19T14:33:42.302 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00059148899999999995 2024-08-19T14:33:42.302 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:42.302 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:42.303 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:33:42.303 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:33:39.769213+0000", 2024-08-19T14:33:42.303 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3361e-05 2024-08-19T14:33:42.303 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:42.303 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:42.303 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:33:42.303 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:33:39.769221+0000", 2024-08-19T14:33:42.303 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.6469999999999998e-06 2024-08-19T14:33:42.303 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:42.303 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:42.303 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:33:42.304 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:33:39.769290+0000", 2024-08-19T14:33:42.304 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.9770000000000005e-05 2024-08-19T14:33:42.304 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:42.304 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:42.304 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:33:42.304 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:33:39.769446+0000", 2024-08-19T14:33:42.304 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00015543499999999999 2024-08-19T14:33:42.304 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:42.304 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:42.305 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 5,11", 2024-08-19T14:33:42.305 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:33:39.769574+0000", 2024-08-19T14:33:42.305 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012857100000000001 2024-08-19T14:33:42.305 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:42.305 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:42.305 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:42.305 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:33:39.772543+0000", 2024-08-19T14:33:42.305 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.002968928 2024-08-19T14:33:42.305 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:42.305 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:42.305 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:42.305 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:33:39.772586+0000", 2024-08-19T14:33:42.306 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2506000000000003e-05 2024-08-19T14:33:42.306 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:42.306 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:33:42.306 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:42.306 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:42.306 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:42.306 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:19050 7.d 7:b480f012:::benchmark_data_smithi045_115842_object19049:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e607)", 2024-08-19T14:33:42.306 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:33:39.769244+0000", 2024-08-19T14:33:42.306 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.0032878590000000002, 2024-08-19T14:33:42.307 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0034938170000000002, 2024-08-19T14:33:42.307 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:33:42.307 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:33:42.307 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "waiting for sub ops", 2024-08-19T14:33:42.307 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:33:42.307 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:33:42.307 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:33:42.307 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 19050 2024-08-19T14:33:42.307 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:42.307 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:33:42.307 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:42.308 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:33:42.308 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:33:39.769244+0000", 2024-08-19T14:33:42.308 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:42.308 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:42.308 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:42.308 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:33:42.308 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:33:39.769244+0000", 2024-08-19T14:33:42.308 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:42.308 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:42.308 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:42.309 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:33:42.309 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:33:39.769245+0000", 2024-08-19T14:33:42.309 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.1100000000000005e-07 2024-08-19T14:33:42.309 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:42.309 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:42.309 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:33:42.309 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:33:39.769278+0000", 2024-08-19T14:33:42.309 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.2607999999999997e-05 2024-08-19T14:33:42.309 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:42.309 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:42.309 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:33:42.310 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:33:39.769287+0000", 2024-08-19T14:33:42.310 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.0999999999999993e-06 2024-08-19T14:33:42.310 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:42.310 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:42.310 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:33:42.310 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:33:39.769293+0000", 2024-08-19T14:33:42.310 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.2400000000000004e-06 2024-08-19T14:33:42.310 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:42.310 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:42.310 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:33:42.311 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:33:39.769448+0000", 2024-08-19T14:33:42.311 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000155036 2024-08-19T14:33:42.311 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:42.311 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:42.311 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:33:42.311 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:33:39.769587+0000", 2024-08-19T14:33:42.311 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000139066 2024-08-19T14:33:42.311 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:42.311 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:42.311 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,3", 2024-08-19T14:33:42.311 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:33:39.769734+0000", 2024-08-19T14:33:42.312 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000147078 2024-08-19T14:33:42.312 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:42.312 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:42.312 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:42.312 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:33:39.772576+0000", 2024-08-19T14:33:42.312 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00284111 2024-08-19T14:33:42.312 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:42.312 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:33:42.312 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:42.312 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:42.313 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:42.313 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.16962.0:19036 7.4 e607/600 7:25d80529:::benchmark_data_smithi045_115842_object19035:head v 607'1226, mlcod=607'1224)", 2024-08-19T14:33:42.313 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:33:39.769317+0000", 2024-08-19T14:33:42.313 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.0032154240000000001, 2024-08-19T14:33:42.313 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0034925999999999998, 2024-08-19T14:33:42.313 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:33:42.313 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:33:42.313 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-08-19T14:33:42.313 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:33:42.313 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:42.313 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:33:42.313 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:33:39.769317+0000", 2024-08-19T14:33:42.313 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:42.313 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:42.313 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:42.314 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:33:42.314 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:33:39.769317+0000", 2024-08-19T14:33:42.314 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:42.314 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:42.314 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:42.314 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:33:42.314 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:33:39.769317+0000", 2024-08-19T14:33:42.314 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7500000000000001e-07 2024-08-19T14:33:42.314 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:42.314 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:42.314 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:33:42.314 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:33:39.769341+0000", 2024-08-19T14:33:42.315 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.3691000000000001e-05 2024-08-19T14:33:42.315 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:42.315 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:42.315 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:33:42.315 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:33:39.769350+0000", 2024-08-19T14:33:42.315 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.6670000000000002e-06 2024-08-19T14:33:42.315 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:42.315 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:42.315 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:33:42.315 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:33:39.769356+0000", 2024-08-19T14:33:42.315 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.0100000000000001e-06 2024-08-19T14:33:42.315 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:42.315 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:42.316 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:33:42.316 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:33:39.769648+0000", 2024-08-19T14:33:42.316 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000292551 2024-08-19T14:33:42.316 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:42.316 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:42.316 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:33:42.316 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:33:39.769665+0000", 2024-08-19T14:33:42.316 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6753999999999999e-05 2024-08-19T14:33:42.316 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:42.317 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:33:42.317 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:42.317 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:42.317 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:42.317 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.16962.0:19046 7.0 e607/600 7:0e7f9604:::benchmark_data_smithi045_115842_object19045:head v 607'1118, mlcod=607'1116)", 2024-08-19T14:33:42.317 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:33:39.770196+0000", 2024-08-19T14:33:42.317 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.0023367420000000002, 2024-08-19T14:33:42.317 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.002677114, 2024-08-19T14:33:42.318 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:33:42.318 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:33:42.318 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-08-19T14:33:42.318 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:33:42.318 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:42.318 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:33:42.318 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:33:39.770196+0000", 2024-08-19T14:33:42.318 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:42.318 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:42.318 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:42.318 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:33:42.318 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:33:39.770196+0000", 2024-08-19T14:33:42.319 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:42.319 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:42.319 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:42.319 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:33:42.319 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:33:39.770196+0000", 2024-08-19T14:33:42.319 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.91e-07 2024-08-19T14:33:42.319 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:42.319 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:42.319 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:33:42.319 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:33:39.770217+0000", 2024-08-19T14:33:42.319 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.0809e-05 2024-08-19T14:33:42.319 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:42.319 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:42.319 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:33:42.319 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:33:39.770224+0000", 2024-08-19T14:33:42.320 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.3300000000000001e-06 2024-08-19T14:33:42.320 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:42.320 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:42.320 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:33:42.320 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:33:39.770229+0000", 2024-08-19T14:33:42.320 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0499999999999999e-06 2024-08-19T14:33:42.320 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:42.320 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:42.320 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:33:42.320 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:33:39.770278+0000", 2024-08-19T14:33:42.320 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.8686000000000001e-05 2024-08-19T14:33:42.320 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:42.320 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:42.320 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:33:42.321 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:33:39.770297+0000", 2024-08-19T14:33:42.321 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.9202e-05 2024-08-19T14:33:42.321 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:42.321 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:33:42.321 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:42.321 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:42.321 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:42.321 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.16962.0:19051 7.7 e607/600 7:ea9e2d88:::benchmark_data_smithi045_115842_object19050:head v 607'1080, mlcod=607'1076)", 2024-08-19T14:33:42.321 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:33:39.770253+0000", 2024-08-19T14:33:42.321 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.0022795440000000001, 2024-08-19T14:33:42.321 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.002681091, 2024-08-19T14:33:42.322 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:33:42.322 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:33:42.322 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-08-19T14:33:42.322 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:33:42.322 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:42.322 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:33:42.322 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:33:39.770253+0000", 2024-08-19T14:33:42.322 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:42.322 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:42.322 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:42.322 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:33:42.322 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:33:39.770253+0000", 2024-08-19T14:33:42.323 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:42.323 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:42.323 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:42.323 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:33:42.323 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:33:39.770253+0000", 2024-08-19T14:33:42.323 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.9999999999999999e-07 2024-08-19T14:33:42.323 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:42.323 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:42.323 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:33:42.323 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:33:39.770276+0000", 2024-08-19T14:33:42.323 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.2721000000000002e-05 2024-08-19T14:33:42.323 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:42.324 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:42.324 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:33:42.324 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:33:39.770283+0000", 2024-08-19T14:33:42.324 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.8009999999999998e-06 2024-08-19T14:33:42.324 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:42.324 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:42.324 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:33:42.324 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:33:39.770288+0000", 2024-08-19T14:33:42.324 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0089999999999998e-06 2024-08-19T14:33:42.324 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:42.324 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:42.324 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:33:42.324 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:33:39.771008+0000", 2024-08-19T14:33:42.324 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00072016099999999998 2024-08-19T14:33:42.324 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:42.325 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:42.325 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:33:42.325 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:33:39.771034+0000", 2024-08-19T14:33:42.325 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.6366000000000001e-05 2024-08-19T14:33:42.325 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:42.325 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:33:42.325 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:42.325 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:42.325 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:42.325 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.16962.0:19040 7.4 e607/600 7:2dced9c1:::benchmark_data_smithi045_115842_object19039:head v 607'1227, mlcod=607'1224)", 2024-08-19T14:33:42.325 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:33:39.770841+0000", 2024-08-19T14:33:42.325 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.001691166, 2024-08-19T14:33:42.325 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.002152995, 2024-08-19T14:33:42.326 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:33:42.326 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:33:42.326 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-08-19T14:33:42.326 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:33:42.326 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:42.326 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:33:42.326 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:33:39.770841+0000", 2024-08-19T14:33:42.326 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:42.326 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:42.326 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:42.326 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:33:42.326 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:33:39.770841+0000", 2024-08-19T14:33:42.326 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:42.327 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:42.327 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:42.327 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:33:42.327 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:33:39.770842+0000", 2024-08-19T14:33:42.327 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.0100000000000005e-07 2024-08-19T14:33:42.327 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:42.327 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:42.327 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:33:42.327 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:33:39.770894+0000", 2024-08-19T14:33:42.327 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.2255e-05 2024-08-19T14:33:42.327 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:42.327 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:42.327 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:33:42.327 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:33:39.770905+0000", 2024-08-19T14:33:42.328 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0623e-05 2024-08-19T14:33:42.328 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:42.328 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:42.328 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:33:42.328 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:33:39.770912+0000", 2024-08-19T14:33:42.328 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.4719999999999998e-06 2024-08-19T14:33:42.328 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:42.328 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:42.328 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:33:42.328 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:33:39.771354+0000", 2024-08-19T14:33:42.328 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00044181100000000002 2024-08-19T14:33:42.328 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:42.328 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:42.328 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:33:42.329 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:33:39.771373+0000", 2024-08-19T14:33:42.329 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.8740999999999999e-05 2024-08-19T14:33:42.329 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:42.329 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:33:42.329 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:42.329 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:42.329 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:42.329 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.16962.0:19047 7.0 e607/600 7:088c013d:::benchmark_data_smithi045_115842_object19046:head v 607'1119, mlcod=607'1116)", 2024-08-19T14:33:42.329 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:33:39.772154+0000", 2024-08-19T14:33:42.329 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.00037782799999999997, 2024-08-19T14:33:42.329 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00090291299999999998, 2024-08-19T14:33:42.329 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:33:42.330 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:33:42.330 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-08-19T14:33:42.330 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:33:42.330 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:42.330 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:33:42.330 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:33:39.772154+0000", 2024-08-19T14:33:42.330 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:42.330 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:42.330 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:42.330 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:33:42.330 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:33:39.772154+0000", 2024-08-19T14:33:42.330 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:42.330 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:42.331 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:42.331 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:33:42.331 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:33:39.772155+0000", 2024-08-19T14:33:42.331 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.3799999999999996e-07 2024-08-19T14:33:42.331 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:42.331 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:42.331 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:33:42.331 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:33:39.772180+0000", 2024-08-19T14:33:42.331 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.4907999999999999e-05 2024-08-19T14:33:42.331 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:42.331 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:42.331 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:33:42.331 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:33:39.772191+0000", 2024-08-19T14:33:42.331 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.066e-05 2024-08-19T14:33:42.331 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:42.332 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:42.332 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:33:42.332 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:33:39.772198+0000", 2024-08-19T14:33:42.332 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.9009999999999997e-06 2024-08-19T14:33:42.332 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:42.332 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:42.332 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:33:42.332 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:33:39.772242+0000", 2024-08-19T14:33:42.332 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4657000000000003e-05 2024-08-19T14:33:42.332 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:42.332 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:42.333 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:33:42.333 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:33:39.772272+0000", 2024-08-19T14:33:42.333 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.9087000000000001e-05 2024-08-19T14:33:42.333 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:42.333 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:33:42.333 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:42.333 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:42.333 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:42.333 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.16962.0:19042 7.4 e607/600 7:29cd9c05:::benchmark_data_smithi045_115842_object19041:head v 607'1228, mlcod=607'1224)", 2024-08-19T14:33:42.333 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:33:39.772481+0000", 2024-08-19T14:33:42.333 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 5.0924999999999998e-05, 2024-08-19T14:33:42.334 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00063638200000000001, 2024-08-19T14:33:42.334 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:33:42.334 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:33:42.334 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-08-19T14:33:42.334 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:33:42.334 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:42.334 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:33:42.334 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:33:39.772481+0000", 2024-08-19T14:33:42.334 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:42.334 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:42.334 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:42.334 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:33:42.334 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:33:39.772481+0000", 2024-08-19T14:33:42.334 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:42.335 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:42.335 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:42.335 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:33:42.335 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:33:39.772482+0000", 2024-08-19T14:33:42.335 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.4700000000000002e-07 2024-08-19T14:33:42.335 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:42.335 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:42.335 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:33:42.335 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:33:39.772508+0000", 2024-08-19T14:33:42.335 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.6550000000000002e-05 2024-08-19T14:33:42.335 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:42.336 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:42.336 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:33:42.336 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:33:39.772521+0000", 2024-08-19T14:33:42.336 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2828e-05 2024-08-19T14:33:42.336 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:42.336 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:42.336 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:33:42.336 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:33:39.772527+0000", 2024-08-19T14:33:42.336 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.3330000000000002e-06 2024-08-19T14:33:42.336 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:42.336 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:42.336 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:33:42.336 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:33:39.772649+0000", 2024-08-19T14:33:42.337 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000121927 2024-08-19T14:33:42.337 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:42.337 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:42.337 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:33:42.337 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:33:39.772666+0000", 2024-08-19T14:33:42.337 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6229e-05 2024-08-19T14:33:42.337 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:42.337 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:33:42.337 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:42.337 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:42.337 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-08-19T14:33:42.337 INFO:teuthology.orchestra.run.smithi031.stdout: "num_ops": 9 2024-08-19T14:33:42.337 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:33:42.371 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:33:42.392 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:42.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:42.393 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:33:42.441 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 9 16 24976 24960 173.319 113.562 0.0870606 0.00576566 2024-08-19T14:33:42.495 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:33:42.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:42.531 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:42.531 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:33:42.632 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:33:42.653 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:42.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:42.654 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:33:42.755 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:33:42.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:42.792 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:42.792 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:33:42.894 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:33:42.915 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:42.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:42.916 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:33:43.017 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:33:43.042 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:43.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:43.043 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:33:43.144 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:33:43.179 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:43.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:43.181 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:33:43.282 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:33:43.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:43.310 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:43.311 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:33:43.412 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:33:43.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:43.438 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:43.438 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:33:43.441 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 10 16 27038 27022 168.874 128.875 0.0204538 0.00590983 2024-08-19T14:33:43.540 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:33:43.565 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:43.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:43.567 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:33:43.668 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:33:43.693 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:43.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:43.694 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:33:43.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:43 smithi032 ceph-mon[20004]: pgmap v1422: 47 pgs: 9 active+undersized, 7 active+undersized+degraded, 31 active+clean; 1.1 GiB data, 12 GiB used, 792 GiB / 805 GiB avail; 160 MiB/s wr, 2.56k op/s; 7667/56283 objects degraded (13.622%) 2024-08-19T14:33:43.795 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:33:43.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:43 smithi045 ceph-mon[20143]: pgmap v1422: 47 pgs: 9 active+undersized, 7 active+undersized+degraded, 31 active+clean; 1.1 GiB data, 12 GiB used, 792 GiB / 805 GiB avail; 160 MiB/s wr, 2.56k op/s; 7667/56283 objects degraded (13.622%) 2024-08-19T14:33:43.817 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:43.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:43.818 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:33:43.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:43 smithi031 ceph-mon[16336]: pgmap v1422: 47 pgs: 9 active+undersized, 7 active+undersized+degraded, 31 active+clean; 1.1 GiB data, 12 GiB used, 792 GiB / 805 GiB avail; 160 MiB/s wr, 2.56k op/s; 7667/56283 objects degraded (13.622%) 2024-08-19T14:33:43.918 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:33:43.944 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:43.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:43.946 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:33:44.048 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:33:44.072 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:44.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:44.073 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:33:44.174 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:33:44.211 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:44.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:44.212 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:33:44.313 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:33:44.335 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:44.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:44.336 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:33:44.437 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:33:44.441 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 11 16 29130 29114 165.407 130.75 0.00264741 0.00600541 2024-08-19T14:33:44.464 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:44.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:44.465 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:33:44.566 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:33:44.604 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:44.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:44.606 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:33:44.706 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:33:44.732 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:44.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:44.733 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:33:44.834 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:33:44.867 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:44.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:44.868 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:33:44.969 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:33:44.990 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:44.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:44.991 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:33:45.092 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:33:45.113 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:45.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:45.114 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:33:45.215 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:33:45.241 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:45.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:45.242 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:33:45.342 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:33:45.370 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:45.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:45.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:33:45.441 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 12 16 31246 31230 162.643 132.25 0.00608935 0.00614361 2024-08-19T14:33:45.472 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:33:45.498 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:45.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:45.499 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:33:45.519 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:33:45.568 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-19T14:33:45.568 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-19T14:33:45.568 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph tell osd.2 'trim stale osdmaps' 2024-08-19T14:33:45.600 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:33:45.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:45.659 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:45.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:33:45.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:45 smithi032 ceph-mon[20004]: pgmap v1423: 47 pgs: 9 active+undersized, 7 active+undersized+degraded, 31 active+clean; 1.4 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 146 MiB/s wr, 2.33k op/s; 7667/67188 objects degraded (11.411%) 2024-08-19T14:33:45.760 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:33:45.784 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:45.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:45.785 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:33:45.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:45 smithi045 ceph-mon[20143]: pgmap v1423: 47 pgs: 9 active+undersized, 7 active+undersized+degraded, 31 active+clean; 1.4 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 146 MiB/s wr, 2.33k op/s; 7667/67188 objects degraded (11.411%) 2024-08-19T14:33:45.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:45 smithi031 ceph-mon[16336]: pgmap v1423: 47 pgs: 9 active+undersized, 7 active+undersized+degraded, 31 active+clean; 1.4 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 146 MiB/s wr, 2.33k op/s; 7667/67188 objects degraded (11.411%) 2024-08-19T14:33:45.885 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:33:45.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:45.920 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:45.920 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:33:46.021 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:33:46.047 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:46.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:46.048 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:33:46.149 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:33:46.175 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:46.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:46.176 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:33:46.277 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:33:46.303 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:46.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:46.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:33:46.405 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:33:46.428 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:46.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:46.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:33:46.441 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 13 16 33524 33508 161.083 142.375 0.00183677 0.00618889 2024-08-19T14:33:46.530 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:33:46.545 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-08-19T14:33:46.545 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:33:46.551 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:46.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:46.552 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:33:46.653 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:33:46.680 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:46.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:46.681 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:33:46.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:46 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 11040/85485 objects degraded (12.915%), 7 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-08-19T14:33:46.783 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:33:46.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:46 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 11040/85485 objects degraded (12.915%), 7 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-08-19T14:33:46.804 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:46.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:46.805 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:33:46.836 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:46 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 11040/85485 objects degraded (12.915%), 7 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-08-19T14:33:46.905 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:33:46.930 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:46.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:46.931 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:33:47.032 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:33:47.059 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:47.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:47.060 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:33:47.161 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:33:47.186 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:47.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:47.187 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:33:47.288 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:33:47.314 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:47.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:47.316 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:33:47.417 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:33:47.428 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:33:47.429 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:33:47.440 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:47.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:47.440 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:33:47.441 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 14 16 35394 35378 157.925 116.875 0.0124391 0.00632746 2024-08-19T14:33:47.464 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:33:47.465 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:33:47.467 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-08-19T14:33:47.499 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:33:47.499 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:33:47.523 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:33:47.523 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:33:47.540 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:33:47.541 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:33:47.541 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:33:47.557 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:33:47.558 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:33:47.562 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:47.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:47.563 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:33:47.579 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:33:47.580 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:33:47.597 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:33:47.598 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:33:47.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:47 smithi031 ceph-mon[16336]: pgmap v1424: 47 pgs: 9 active+undersized, 7 active+undersized+degraded, 31 active+clean; 1.7 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 173 MiB/s wr, 2.76k op/s; 11040/85485 objects degraded (12.915%) 2024-08-19T14:33:47.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:47 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3800341049' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:33:47.618 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:33:47.618 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:33:47.618 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:33:47.655 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:33:47.655 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:33:47.664 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:33:47.668 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:33:47.687 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:47.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:47.688 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:33:47.695 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:33:47.695 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:33:47.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:47 smithi032 ceph-mon[20004]: pgmap v1424: 47 pgs: 9 active+undersized, 7 active+undersized+degraded, 31 active+clean; 1.7 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 173 MiB/s wr, 2.76k op/s; 11040/85485 objects degraded (12.915%) 2024-08-19T14:33:47.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:47 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3800341049' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:33:47.789 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:33:47.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:47 smithi045 ceph-mon[20143]: pgmap v1424: 47 pgs: 9 active+undersized, 7 active+undersized+degraded, 31 active+clean; 1.7 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 173 MiB/s wr, 2.76k op/s; 11040/85485 objects degraded (12.915%) 2024-08-19T14:33:47.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:47 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3800341049' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:33:47.827 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:47.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:47.828 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:33:47.929 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:33:47.954 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:47.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:47.955 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:33:48.056 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:33:48.082 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:48.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:48.083 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:33:48.184 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:33:48.212 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:48.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:48.213 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:33:48.314 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:33:48.339 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:48.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:48.340 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:33:48.441 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:33:48.442 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 15 16 37294 37278 155.313 118.75 0.00911277 0.00643429 2024-08-19T14:33:48.450 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:33:48.450 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:33:48.465 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:48.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:48.466 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:33:48.517 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is set 2024-08-19T14:33:48.540 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:48 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:33:48.540 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:48 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:33:48.540 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:48 smithi031 ceph-mon[16336]: osdmap e609: 12 total, 11 up, 9 in 2024-08-19T14:33:48.540 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:48 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/471883245' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:33:48.567 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:33:48.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:48.604 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:48.605 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:33:48.706 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:33:48.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:48.731 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:48.731 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:33:48.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:48 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:33:48.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:48 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:33:48.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:48 smithi032 ceph-mon[20004]: osdmap e609: 12 total, 11 up, 9 in 2024-08-19T14:33:48.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:48 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/471883245' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:33:48.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:48 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:33:48.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:48 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:33:48.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:48 smithi045 ceph-mon[20143]: osdmap e609: 12 total, 11 up, 9 in 2024-08-19T14:33:48.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:48 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/471883245' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:33:48.832 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:33:48.858 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:48.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:48.859 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:33:48.960 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:33:48.983 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:48.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:48.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:33:49.004 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-19T14:33:49.004 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-19T14:33:49.085 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:33:49.111 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:49.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:49.112 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:33:49.213 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:33:49.240 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:49.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:49.241 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:33:49.342 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:33:49.366 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:49.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:49.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:33:49.441 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 16 16 39667 39651 154.874 148.312 0.00643125 0.00645253 2024-08-19T14:33:49.468 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:33:49.493 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:49.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:49.494 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:33:49.595 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:33:49.619 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:49.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:49.620 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:33:49.721 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:33:49.764 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:49.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:49.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:33:49.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:49 smithi045 ceph-mon[20143]: pgmap v1426: 47 pgs: 9 active+undersized, 7 active+undersized+degraded, 31 active+clean; 1.8 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 140 MiB/s wr, 2.25k op/s; 11664/89301 objects degraded (13.061%) 2024-08-19T14:33:49.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:49 smithi045 ceph-mon[20143]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:33:49.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:49 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/471883245' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:33:49.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:49 smithi045 ceph-mon[20143]: osdmap e610: 12 total, 11 up, 9 in 2024-08-19T14:33:49.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:49 smithi031 ceph-mon[16336]: pgmap v1426: 47 pgs: 9 active+undersized, 7 active+undersized+degraded, 31 active+clean; 1.8 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 140 MiB/s wr, 2.25k op/s; 11664/89301 objects degraded (13.061%) 2024-08-19T14:33:49.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:49 smithi031 ceph-mon[16336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:33:49.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:49 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/471883245' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:33:49.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:49 smithi031 ceph-mon[16336]: osdmap e610: 12 total, 11 up, 9 in 2024-08-19T14:33:49.867 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:33:49.893 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:49.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:49.894 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:33:49.995 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:33:50.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:49 smithi032 ceph-mon[20004]: pgmap v1426: 47 pgs: 9 active+undersized, 7 active+undersized+degraded, 31 active+clean; 1.8 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 140 MiB/s wr, 2.25k op/s; 11664/89301 objects degraded (13.061%) 2024-08-19T14:33:50.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:49 smithi032 ceph-mon[20004]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:33:50.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:49 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/471883245' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:33:50.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:49 smithi032 ceph-mon[20004]: osdmap e610: 12 total, 11 up, 9 in 2024-08-19T14:33:50.021 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:50.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:50.022 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:33:50.123 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:33:50.147 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:50.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:50.148 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:33:50.201 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-08-19T14:33:50.249 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:33:50.273 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:50.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:50.274 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:33:50.375 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:33:50.399 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:50.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:50.400 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:33:50.441 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 17 16 41904 41888 153.988 139.812 0.013824 0.0064892 2024-08-19T14:33:50.481 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-08-19T14:33:50.500 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:33:50.524 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:50.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:50.526 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:33:50.627 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:33:50.649 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:50.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:50.650 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:33:50.751 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:33:50.774 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:50.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:50.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:33:50.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:50 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:33:50.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:50 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:33:50.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:50 smithi045 ceph-mon[20143]: osdmap e611: 12 total, 11 up, 9 in 2024-08-19T14:33:50.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:50 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:33:50.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:50 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:33:50.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:50 smithi031 ceph-mon[16336]: osdmap e611: 12 total, 11 up, 9 in 2024-08-19T14:33:50.876 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:33:50.900 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:50.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:50.901 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:33:51.002 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:33:51.005 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:33:51.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:50 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:33:51.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:50 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:33:51.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:50 smithi032 ceph-mon[20004]: osdmap e611: 12 total, 11 up, 9 in 2024-08-19T14:33:51.026 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:51.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:51.027 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:33:51.128 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:33:51.153 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:51.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:51.154 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:33:51.255 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:33:51.277 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:51.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:51.278 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:33:51.379 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:33:51.416 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:51.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:51.418 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:33:51.441 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 18 15 43686 43671 151.624 111.438 0.00723412 0.00659063 2024-08-19T14:33:51.518 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:33:51.543 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:51.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:51.544 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:33:51.645 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:33:51.656 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:51 smithi031 ceph-mon[16336]: pgmap v1429: 47 pgs: 9 active+undersized, 7 active+undersized+degraded, 31 active+clean; 2.1 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 131 MiB/s wr, 2.10k op/s; 12339/104997 objects degraded (11.752%) 2024-08-19T14:33:51.667 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:51.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:51.668 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:33:51.769 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:33:51.793 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:51.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:51.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:33:51.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:51 smithi045 ceph-mon[20143]: pgmap v1429: 47 pgs: 9 active+undersized, 7 active+undersized+degraded, 31 active+clean; 2.1 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 131 MiB/s wr, 2.10k op/s; 12339/104997 objects degraded (11.752%) 2024-08-19T14:33:51.895 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:33:51.921 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:51.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:51.922 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:33:52.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:51 smithi032 ceph-mon[20004]: pgmap v1429: 47 pgs: 9 active+undersized, 7 active+undersized+degraded, 31 active+clean; 2.1 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 131 MiB/s wr, 2.10k op/s; 12339/104997 objects degraded (11.752%) 2024-08-19T14:33:52.023 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:33:52.047 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:52.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:52.048 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:33:52.149 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:33:52.188 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:52.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:52.189 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:33:52.290 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:33:52.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:52.315 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:52.315 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:33:52.416 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:33:52.440 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:52.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:52.442 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:33:52.442 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 19 16 45851 45835 150.761 135.25 0.00218187 0.0066276 2024-08-19T14:33:52.543 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:33:52.568 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:52.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:52.569 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:33:52.610 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-19T14:33:52.614 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-19T14:33:52.615 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph tell osd.11 'trim stale osdmaps' 2024-08-19T14:33:52.669 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:33:52.678 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:33:52.679 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:33:52.692 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:52.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:52.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:33:52.794 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:33:52.819 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:52.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:52.820 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:33:52.821 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:52 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 15697/120939 objects degraded (12.979%), 7 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-08-19T14:33:52.908 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:52 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 15697/120939 objects degraded (12.979%), 7 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-08-19T14:33:52.921 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:33:52.947 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:52.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:52.948 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:33:53.007 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:52 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 15697/120939 objects degraded (12.979%), 7 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-08-19T14:33:53.049 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:33:53.076 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:53.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:53.077 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:33:53.178 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:33:53.219 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:53.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:53.221 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:33:53.321 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:33:53.346 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:53.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:53.348 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:33:53.441 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:2024-08-19T14:33:53.442110+0000 min lat: 0.00106356 max lat: 0.285252 avg lat: 0.00674434 2024-08-19T14:33:53.442 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-19T14:33:53.442 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 20 16 47432 47416 148.163 98.8125 0.0173312 0.00674434 2024-08-19T14:33:53.449 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:33:53.474 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:53.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:53.475 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:33:53.576 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:33:53.579 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:33:53.580 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:33:53.580 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:33:53.580 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-08-19T14:33:53.580 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:33:53.580 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:33:53.581 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:33:53.581 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:33:53.581 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:33:53.581 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:33:53.581 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:33:53.581 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:33:53.581 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:33:53.598 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:33:53.598 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:33:53.599 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:53.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:53.600 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:33:53.636 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:53 smithi031 ceph-mon[16336]: pgmap v1430: 47 pgs: 9 active+undersized, 7 active+undersized+degraded, 31 active+clean; 2.5 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 123 MiB/s wr, 1.97k op/s; 15697/120939 objects degraded (12.979%) 2024-08-19T14:33:53.636 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:53 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/4057479953' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:33:53.636 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:33:53.637 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:33:53.670 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:33:53.671 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:33:53.701 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:33:53.707 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:33:53.707 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:33:53.726 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:53.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:53.727 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:33:53.748 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:33:53.749 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:33:53.827 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:33:53.851 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:53.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:53.852 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:33:53.876 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:33:53.876 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 20, 2024-08-19T14:33:53.876 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 600, 2024-08-19T14:33:53.876 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [ 2024-08-19T14:33:53.876 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.876 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25813.0:132390 5.17 5:eadc7f5f:::benchmark_data_smithi045_85309_object12137:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e462)", 2024-08-19T14:33:53.876 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:26:28.790357+0000", 2024-08-19T14:33:53.876 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 445.08372047199998, 2024-08-19T14:33:53.876 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.61116161199999997, 2024-08-19T14:33:53.876 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:33:53.876 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:33:53.877 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:33:53.877 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:33:53.877 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25813", 2024-08-19T14:33:53.877 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:33:53.877 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 132390 2024-08-19T14:33:53.877 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.877 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:33:53.877 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.877 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:33:53.877 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.790357+0000", 2024-08-19T14:33:53.877 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:53.878 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.878 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.878 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:33:53.878 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.790357+0000", 2024-08-19T14:33:53.878 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:53.878 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.878 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.878 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:33:53.878 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.790358+0000", 2024-08-19T14:33:53.878 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2500000000000001e-06 2024-08-19T14:33:53.878 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.878 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.878 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:33:53.879 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.790363+0000", 2024-08-19T14:33:53.879 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4800000000000003e-06 2024-08-19T14:33:53.879 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.879 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.879 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:33:53.879 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.790373+0000", 2024-08-19T14:33:53.879 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0117e-05 2024-08-19T14:33:53.879 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.879 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.879 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:33:53.879 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.790378+0000", 2024-08-19T14:33:53.879 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.6280000000000002e-06 2024-08-19T14:33:53.880 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.880 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.880 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:33:53.880 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.791871+0000", 2024-08-19T14:33:53.880 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0014923829999999999 2024-08-19T14:33:53.880 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.880 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.880 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:33:53.880 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.791957+0000", 2024-08-19T14:33:53.880 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.6532000000000004e-05 2024-08-19T14:33:53.880 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.880 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.881 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,8,11", 2024-08-19T14:33:53.881 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.792037+0000", 2024-08-19T14:33:53.881 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.9797999999999997e-05 2024-08-19T14:33:53.881 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.881 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.881 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:33:53.881 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.793676+0000", 2024-08-19T14:33:53.881 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.001639176 2024-08-19T14:33:53.881 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.881 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.881 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:53.881 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.796271+0000", 2024-08-19T14:33:53.881 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0025944750000000002 2024-08-19T14:33:53.881 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.882 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.882 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:53.882 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.796315+0000", 2024-08-19T14:33:53.882 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.3995999999999997e-05 2024-08-19T14:33:53.882 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.882 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.882 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:53.882 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.400666+0000", 2024-08-19T14:33:53.882 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.60435100500000005 2024-08-19T14:33:53.882 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.882 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.882 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:33:53.882 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.400710+0000", 2024-08-19T14:33:53.883 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4027999999999999e-05 2024-08-19T14:33:53.883 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.883 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.883 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:33:53.883 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.401518+0000", 2024-08-19T14:33:53.883 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00080874400000000002 2024-08-19T14:33:53.883 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:53.883 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:33:53.883 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:53.883 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.883 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.883 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25813.0:132430 5.17 5:eeb9c3c7:::benchmark_data_smithi045_85309_object12177:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e462)", 2024-08-19T14:33:53.883 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:26:28.885232+0000", 2024-08-19T14:33:53.884 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 444.98884510900001, 2024-08-19T14:33:53.884 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.51860473299999998, 2024-08-19T14:33:53.884 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:33:53.884 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:33:53.884 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:33:53.884 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:33:53.884 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25813", 2024-08-19T14:33:53.884 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:33:53.884 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 132430 2024-08-19T14:33:53.884 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.884 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:33:53.884 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.884 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:33:53.885 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.885232+0000", 2024-08-19T14:33:53.885 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:53.885 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.885 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.885 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:33:53.885 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.885232+0000", 2024-08-19T14:33:53.885 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:53.885 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.885 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.885 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:33:53.885 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.885234+0000", 2024-08-19T14:33:53.885 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.7600000000000001e-06 2024-08-19T14:33:53.885 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.886 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.886 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:33:53.886 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.885240+0000", 2024-08-19T14:33:53.886 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.7579999999999998e-06 2024-08-19T14:33:53.886 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.886 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.886 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:33:53.886 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.885257+0000", 2024-08-19T14:33:53.886 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.7309999999999999e-05 2024-08-19T14:33:53.886 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.886 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.886 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:33:53.886 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.885265+0000", 2024-08-19T14:33:53.886 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.9209999999999992e-06 2024-08-19T14:33:53.887 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.887 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.887 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:33:53.887 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.885328+0000", 2024-08-19T14:33:53.887 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.2842000000000006e-05 2024-08-19T14:33:53.887 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.887 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.887 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:33:53.887 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.885477+0000", 2024-08-19T14:33:53.887 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00014877799999999999 2024-08-19T14:33:53.887 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.887 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.887 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,8,11", 2024-08-19T14:33:53.887 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.885711+0000", 2024-08-19T14:33:53.888 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000234074 2024-08-19T14:33:53.888 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.888 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.888 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:33:53.888 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.890476+0000", 2024-08-19T14:33:53.888 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0047655429999999997 2024-08-19T14:33:53.888 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.888 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.888 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:53.888 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.892622+0000", 2024-08-19T14:33:53.888 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0021459930000000001 2024-08-19T14:33:53.888 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.888 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.888 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:53.889 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.893155+0000", 2024-08-19T14:33:53.889 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00053261900000000002 2024-08-19T14:33:53.889 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.889 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.889 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:53.889 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.403023+0000", 2024-08-19T14:33:53.889 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.50986806399999995 2024-08-19T14:33:53.889 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.889 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.889 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:33:53.889 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.403099+0000", 2024-08-19T14:33:53.889 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.6025000000000005e-05 2024-08-19T14:33:53.889 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.889 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.889 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:33:53.890 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.403837+0000", 2024-08-19T14:33:53.890 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00073804600000000001 2024-08-19T14:33:53.890 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:53.890 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:33:53.890 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:53.890 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.890 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.891 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25813.0:132434 5.17 5:e9c564fa:::benchmark_data_smithi045_85309_object12181:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e462)", 2024-08-19T14:33:53.891 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:26:28.887497+0000", 2024-08-19T14:33:53.891 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 444.986580694, 2024-08-19T14:33:53.891 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.51694072000000002, 2024-08-19T14:33:53.891 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:33:53.891 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:33:53.891 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:33:53.891 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:33:53.891 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25813", 2024-08-19T14:33:53.891 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:33:53.892 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 132434 2024-08-19T14:33:53.892 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.892 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:33:53.892 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.892 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:33:53.892 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.887497+0000", 2024-08-19T14:33:53.892 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:53.892 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.892 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.892 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:33:53.893 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.887497+0000", 2024-08-19T14:33:53.893 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:53.893 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.893 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.893 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:33:53.893 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.887498+0000", 2024-08-19T14:33:53.893 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3599999999999999e-06 2024-08-19T14:33:53.893 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.893 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.893 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:33:53.893 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.887503+0000", 2024-08-19T14:33:53.893 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.8829999999999997e-06 2024-08-19T14:33:53.893 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.893 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.893 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:33:53.894 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.887519+0000", 2024-08-19T14:33:53.894 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6169000000000001e-05 2024-08-19T14:33:53.894 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.894 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.894 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:33:53.894 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.887526+0000", 2024-08-19T14:33:53.894 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.3379999999999999e-06 2024-08-19T14:33:53.894 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.894 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.894 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:33:53.894 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.887857+0000", 2024-08-19T14:33:53.894 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00033069799999999998 2024-08-19T14:33:53.894 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.895 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.895 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:33:53.895 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.888014+0000", 2024-08-19T14:33:53.895 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00015727800000000001 2024-08-19T14:33:53.895 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.895 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.895 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,8,11", 2024-08-19T14:33:53.895 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.888227+0000", 2024-08-19T14:33:53.895 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000212874 2024-08-19T14:33:53.895 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.895 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.895 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:33:53.895 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.890486+0000", 2024-08-19T14:33:53.896 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0022583909999999998 2024-08-19T14:33:53.896 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.896 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.896 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:53.896 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.892678+0000", 2024-08-19T14:33:53.896 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0021919600000000002 2024-08-19T14:33:53.896 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.896 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.896 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:53.896 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.893220+0000", 2024-08-19T14:33:53.896 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00054226800000000005 2024-08-19T14:33:53.896 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.896 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.896 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:53.897 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.403898+0000", 2024-08-19T14:33:53.897 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.51067800399999996 2024-08-19T14:33:53.897 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.897 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.897 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:33:53.897 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.403952+0000", 2024-08-19T14:33:53.897 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.4141999999999998e-05 2024-08-19T14:33:53.897 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.897 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.897 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:33:53.897 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.404437+0000", 2024-08-19T14:33:53.897 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00048535499999999998 2024-08-19T14:33:53.898 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:53.898 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:33:53.898 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:53.898 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.898 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.898 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25813.0:132587 5.17 5:e858db2d:::benchmark_data_smithi045_85309_object12334:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e462)", 2024-08-19T14:33:53.898 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:26:28.996803+0000", 2024-08-19T14:33:53.898 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 444.87727378199997, 2024-08-19T14:33:53.898 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.41584004200000002, 2024-08-19T14:33:53.898 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:33:53.898 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:33:53.898 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:33:53.898 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:33:53.898 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25813", 2024-08-19T14:33:53.899 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:33:53.899 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 132587 2024-08-19T14:33:53.899 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.899 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:33:53.899 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.899 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:33:53.899 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.996803+0000", 2024-08-19T14:33:53.899 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:53.899 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.899 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.899 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:33:53.899 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.996803+0000", 2024-08-19T14:33:53.899 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:53.900 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.900 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.900 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:33:53.900 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.996805+0000", 2024-08-19T14:33:53.900 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.049e-06 2024-08-19T14:33:53.900 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.900 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.900 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:33:53.900 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.996809+0000", 2024-08-19T14:33:53.900 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.3699999999999997e-06 2024-08-19T14:33:53.900 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.900 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.900 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:33:53.901 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.996821+0000", 2024-08-19T14:33:53.901 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2041e-05 2024-08-19T14:33:53.901 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.901 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.901 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:33:53.901 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.996828+0000", 2024-08-19T14:33:53.901 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.832e-06 2024-08-19T14:33:53.901 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.901 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.901 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:33:53.901 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.996895+0000", 2024-08-19T14:33:53.901 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.7636000000000001e-05 2024-08-19T14:33:53.901 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.901 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.902 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:33:53.902 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.997058+0000", 2024-08-19T14:33:53.902 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00016274799999999999 2024-08-19T14:33:53.902 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.902 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.902 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,8,11", 2024-08-19T14:33:53.902 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:28.997215+0000", 2024-08-19T14:33:53.902 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00015702899999999999 2024-08-19T14:33:53.902 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.902 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.902 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:53.902 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.000366+0000", 2024-08-19T14:33:53.902 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.003151208 2024-08-19T14:33:53.903 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.903 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.903 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:33:53.903 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.000550+0000", 2024-08-19T14:33:53.903 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00018371699999999999 2024-08-19T14:33:53.903 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.903 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.903 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:53.903 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.001943+0000", 2024-08-19T14:33:53.903 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.001392732 2024-08-19T14:33:53.903 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.903 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.903 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:53.903 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.411104+0000", 2024-08-19T14:33:53.904 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.40916105800000002 2024-08-19T14:33:53.904 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.904 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.904 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:33:53.904 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.411180+0000", 2024-08-19T14:33:53.904 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.5617999999999993e-05 2024-08-19T14:33:53.904 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.904 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.904 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:33:53.904 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.412644+0000", 2024-08-19T14:33:53.904 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0014640040000000001 2024-08-19T14:33:53.904 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:53.904 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:33:53.904 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:53.905 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.905 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.905 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25813.0:132638 5.17 5:eec760cc:::benchmark_data_smithi045_85309_object12385:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e462)", 2024-08-19T14:33:53.905 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:26:29.033888+0000", 2024-08-19T14:33:53.905 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 444.84018967999998, 2024-08-19T14:33:53.905 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.38830038300000003, 2024-08-19T14:33:53.905 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:33:53.905 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:33:53.905 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:33:53.905 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:33:53.905 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25813", 2024-08-19T14:33:53.905 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:33:53.906 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 132638 2024-08-19T14:33:53.906 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.906 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:33:53.906 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.906 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:33:53.906 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.033888+0000", 2024-08-19T14:33:53.906 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:53.906 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.906 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.906 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:33:53.906 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.033888+0000", 2024-08-19T14:33:53.906 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:53.907 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.907 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.907 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:33:53.907 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.033889+0000", 2024-08-19T14:33:53.907 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2950000000000001e-06 2024-08-19T14:33:53.907 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.907 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.907 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:33:53.907 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.033898+0000", 2024-08-19T14:33:53.907 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.7660000000000004e-06 2024-08-19T14:33:53.907 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.907 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.907 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:33:53.907 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.033910+0000", 2024-08-19T14:33:53.908 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1987000000000001e-05 2024-08-19T14:33:53.908 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.908 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.908 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:33:53.908 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.033918+0000", 2024-08-19T14:33:53.908 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.3089999999999996e-06 2024-08-19T14:33:53.908 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.908 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.908 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:33:53.908 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.033993+0000", 2024-08-19T14:33:53.908 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.4912000000000004e-05 2024-08-19T14:33:53.908 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.909 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.909 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:33:53.909 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.034154+0000", 2024-08-19T14:33:53.909 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000161423 2024-08-19T14:33:53.909 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.909 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.909 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,8,11", 2024-08-19T14:33:53.909 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.034316+0000", 2024-08-19T14:33:53.909 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00016205000000000001 2024-08-19T14:33:53.909 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.909 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.909 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:33:53.909 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.036209+0000", 2024-08-19T14:33:53.909 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.001892473 2024-08-19T14:33:53.910 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.910 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.910 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:53.910 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.037758+0000", 2024-08-19T14:33:53.910 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.001549351 2024-08-19T14:33:53.910 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.910 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.910 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:53.910 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.037977+0000", 2024-08-19T14:33:53.910 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00021854399999999999 2024-08-19T14:33:53.910 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.910 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.911 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:53.911 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.421288+0000", 2024-08-19T14:33:53.911 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.383311179 2024-08-19T14:33:53.911 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.911 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.911 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:33:53.911 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.421387+0000", 2024-08-19T14:33:53.911 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.9487999999999993e-05 2024-08-19T14:33:53.911 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.911 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.911 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:33:53.911 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.422188+0000", 2024-08-19T14:33:53.911 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00080060600000000004 2024-08-19T14:33:53.912 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:53.912 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:33:53.912 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:53.912 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.912 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.912 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25813.0:132698 5.17 5:e9ffa124:::benchmark_data_smithi045_85309_object12445:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e462)", 2024-08-19T14:33:53.912 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:26:29.076721+0000", 2024-08-19T14:33:53.912 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 444.79735606700001, 2024-08-19T14:33:53.912 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.346118178, 2024-08-19T14:33:53.912 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:33:53.912 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:33:53.912 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:33:53.912 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:33:53.913 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25813", 2024-08-19T14:33:53.913 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:33:53.913 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 132698 2024-08-19T14:33:53.913 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.913 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:33:53.913 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.913 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:33:53.913 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.076721+0000", 2024-08-19T14:33:53.913 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:53.913 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.913 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.913 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:33:53.913 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.076721+0000", 2024-08-19T14:33:53.914 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:53.914 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.914 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.914 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:33:53.914 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.076723+0000", 2024-08-19T14:33:53.914 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.519e-06 2024-08-19T14:33:53.914 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.914 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.914 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:33:53.914 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.076728+0000", 2024-08-19T14:33:53.914 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.011e-06 2024-08-19T14:33:53.914 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.914 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.915 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:33:53.915 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.076743+0000", 2024-08-19T14:33:53.915 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4952e-05 2024-08-19T14:33:53.915 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.915 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.915 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:33:53.915 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.076751+0000", 2024-08-19T14:33:53.915 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.5199999999999997e-06 2024-08-19T14:33:53.915 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.915 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.915 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:33:53.915 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.076826+0000", 2024-08-19T14:33:53.915 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.4789999999999994e-05 2024-08-19T14:33:53.915 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.916 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.916 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:33:53.916 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.076991+0000", 2024-08-19T14:33:53.916 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00016521600000000001 2024-08-19T14:33:53.916 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.916 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.916 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,8,11", 2024-08-19T14:33:53.916 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.077122+0000", 2024-08-19T14:33:53.916 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000130588 2024-08-19T14:33:53.916 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.916 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.916 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:33:53.916 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.079196+0000", 2024-08-19T14:33:53.916 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0020745350000000002 2024-08-19T14:33:53.917 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.917 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.917 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:53.917 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.080180+0000", 2024-08-19T14:33:53.917 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00098345099999999999 2024-08-19T14:33:53.917 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.917 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.917 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:53.917 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.080323+0000", 2024-08-19T14:33:53.917 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000143583 2024-08-19T14:33:53.917 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.917 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.917 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:53.917 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.422252+0000", 2024-08-19T14:33:53.918 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.34192826399999998 2024-08-19T14:33:53.918 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.918 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.918 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:33:53.918 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.422327+0000", 2024-08-19T14:33:53.918 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.5099999999999996e-05 2024-08-19T14:33:53.918 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.918 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.918 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:33:53.918 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.422839+0000", 2024-08-19T14:33:53.918 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000512649 2024-08-19T14:33:53.918 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:53.918 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:33:53.919 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:53.919 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.919 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.919 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25813.0:132733 5.17 5:e9c717cb:::benchmark_data_smithi045_85309_object12480:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e462)", 2024-08-19T14:33:53.919 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:26:29.101389+0000", 2024-08-19T14:33:53.919 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 444.77268808399998, 2024-08-19T14:33:53.919 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.32198057000000002, 2024-08-19T14:33:53.919 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:33:53.919 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:33:53.919 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:33:53.919 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:33:53.919 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25813", 2024-08-19T14:33:53.919 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:33:53.920 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 132733 2024-08-19T14:33:53.920 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.920 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:33:53.920 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.920 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:33:53.920 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.101389+0000", 2024-08-19T14:33:53.920 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:53.920 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.920 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.920 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:33:53.920 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.101389+0000", 2024-08-19T14:33:53.920 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:53.921 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.921 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.921 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:33:53.921 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.101391+0000", 2024-08-19T14:33:53.921 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.401e-06 2024-08-19T14:33:53.921 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.921 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.921 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:33:53.921 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.101396+0000", 2024-08-19T14:33:53.921 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.6810000000000003e-06 2024-08-19T14:33:53.921 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.921 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.921 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:33:53.921 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.101414+0000", 2024-08-19T14:33:53.922 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.7323999999999999e-05 2024-08-19T14:33:53.922 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.922 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.922 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:33:53.922 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.101424+0000", 2024-08-19T14:33:53.922 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0479000000000001e-05 2024-08-19T14:33:53.922 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.922 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.922 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:33:53.922 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.101497+0000", 2024-08-19T14:33:53.922 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.3008000000000003e-05 2024-08-19T14:33:53.922 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.922 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.923 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:33:53.923 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.101672+0000", 2024-08-19T14:33:53.923 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00017511600000000001 2024-08-19T14:33:53.923 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.923 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.923 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,8,11", 2024-08-19T14:33:53.923 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.101846+0000", 2024-08-19T14:33:53.923 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000174179 2024-08-19T14:33:53.923 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.923 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.923 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:33:53.923 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.104105+0000", 2024-08-19T14:33:53.923 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.002259063 2024-08-19T14:33:53.923 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.924 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.924 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:53.924 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.104876+0000", 2024-08-19T14:33:53.924 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00077034 2024-08-19T14:33:53.924 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.924 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.924 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:53.924 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.107300+0000", 2024-08-19T14:33:53.924 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0024242370000000001 2024-08-19T14:33:53.924 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.924 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.924 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:53.924 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.422905+0000", 2024-08-19T14:33:53.925 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.31560517500000002 2024-08-19T14:33:53.925 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.925 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.925 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:33:53.925 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.422964+0000", 2024-08-19T14:33:53.925 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.8692e-05 2024-08-19T14:33:53.925 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.925 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.925 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:33:53.925 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:26:29.423370+0000", 2024-08-19T14:33:53.925 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00040587499999999999 2024-08-19T14:33:53.925 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:53.925 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:33:53.926 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:53.926 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.926 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.926 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25813.0:237705 5.7 5:e5695e68:::benchmark_data_smithi045_85309_object117452:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e495)", 2024-08-19T14:33:53.926 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:28:07.101072+0000", 2024-08-19T14:33:53.926 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 346.773005125, 2024-08-19T14:33:53.926 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.231283661, 2024-08-19T14:33:53.926 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:33:53.926 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:33:53.926 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:33:53.926 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:33:53.926 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25813", 2024-08-19T14:33:53.926 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:33:53.926 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 237705 2024-08-19T14:33:53.927 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.927 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:33:53.927 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.927 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:33:53.927 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101072+0000", 2024-08-19T14:33:53.927 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:53.927 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.927 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.927 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:33:53.927 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101072+0000", 2024-08-19T14:33:53.927 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:53.927 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.927 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.927 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:33:53.928 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101074+0000", 2024-08-19T14:33:53.928 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.751e-06 2024-08-19T14:33:53.928 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.928 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.928 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:33:53.928 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101080+0000", 2024-08-19T14:33:53.928 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.9050000000000002e-06 2024-08-19T14:33:53.928 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.928 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.928 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:33:53.928 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101096+0000", 2024-08-19T14:33:53.928 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6364000000000001e-05 2024-08-19T14:33:53.928 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.928 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.929 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:33:53.929 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101107+0000", 2024-08-19T14:33:53.929 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0572e-05 2024-08-19T14:33:53.929 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.929 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.929 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:33:53.929 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101166+0000", 2024-08-19T14:33:53.929 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.8981999999999999e-05 2024-08-19T14:33:53.929 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.929 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.929 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:33:53.929 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101181+0000", 2024-08-19T14:33:53.930 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5364000000000001e-05 2024-08-19T14:33:53.930 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.930 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.930 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:33:53.930 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.315274+0000", 2024-08-19T14:33:53.930 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.21409246900000001 2024-08-19T14:33:53.930 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.930 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.930 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for missing object", 2024-08-19T14:33:53.930 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.315413+0000", 2024-08-19T14:33:53.930 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000139782 2024-08-19T14:33:53.930 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.930 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.930 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:33:53.931 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.332277+0000", 2024-08-19T14:33:53.931 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.016863488999999999 2024-08-19T14:33:53.931 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.931 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.931 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:33:53.931 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.332356+0000", 2024-08-19T14:33:53.931 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.8982999999999999e-05 2024-08-19T14:33:53.931 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:53.931 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:33:53.931 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:53.931 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.931 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.931 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25813.0:237714 5.7 5:e252da68:::benchmark_data_smithi045_85309_object117461:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e495)", 2024-08-19T14:33:53.932 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:28:07.101128+0000", 2024-08-19T14:33:53.932 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 346.77294965099998, 2024-08-19T14:33:53.932 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.24052662399999999, 2024-08-19T14:33:53.932 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:33:53.932 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:33:53.932 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:33:53.932 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:33:53.932 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25813", 2024-08-19T14:33:53.932 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:33:53.932 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 237714 2024-08-19T14:33:53.932 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.932 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:33:53.932 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.932 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:33:53.933 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101128+0000", 2024-08-19T14:33:53.933 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:53.933 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.933 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.933 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:33:53.933 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101128+0000", 2024-08-19T14:33:53.933 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:53.933 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.933 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.933 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:33:53.933 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101128+0000", 2024-08-19T14:33:53.933 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.8500000000000002e-07 2024-08-19T14:33:53.933 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.933 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.933 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:33:53.934 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101131+0000", 2024-08-19T14:33:53.934 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.6910000000000002e-06 2024-08-19T14:33:53.934 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.934 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.934 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:33:53.934 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101139+0000", 2024-08-19T14:33:53.935 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.1799999999999996e-06 2024-08-19T14:33:53.935 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.935 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.935 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:33:53.935 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101145+0000", 2024-08-19T14:33:53.935 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.7359999999999998e-06 2024-08-19T14:33:53.935 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.936 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.936 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:33:53.936 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101214+0000", 2024-08-19T14:33:53.936 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.9695000000000003e-05 2024-08-19T14:33:53.936 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.936 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.936 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:33:53.936 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101224+0000", 2024-08-19T14:33:53.936 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0043e-05 2024-08-19T14:33:53.936 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.936 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.936 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:33:53.936 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.315795+0000", 2024-08-19T14:33:53.937 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.21457066599999999 2024-08-19T14:33:53.937 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.937 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.937 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:33:53.937 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.315966+0000", 2024-08-19T14:33:53.937 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00017090700000000001 2024-08-19T14:33:53.937 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.937 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.937 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,5", 2024-08-19T14:33:53.937 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.316585+0000", 2024-08-19T14:33:53.937 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00061851899999999999 2024-08-19T14:33:53.937 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.937 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.937 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:33:53.938 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.332349+0000", 2024-08-19T14:33:53.938 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.015764158 2024-08-19T14:33:53.938 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.938 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.938 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:53.938 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.332428+0000", 2024-08-19T14:33:53.938 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.9651e-05 2024-08-19T14:33:53.938 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.938 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.938 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:53.938 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.341243+0000", 2024-08-19T14:33:53.938 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0088151610000000002 2024-08-19T14:33:53.938 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.939 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.939 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:33:53.939 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.341320+0000", 2024-08-19T14:33:53.939 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.6304000000000007e-05 2024-08-19T14:33:53.939 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.939 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.939 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:33:53.939 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.341654+0000", 2024-08-19T14:33:53.939 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000334428 2024-08-19T14:33:53.939 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:53.939 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:33:53.939 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:53.939 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.940 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.940 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25813.0:237731 5.7 5:e54e0cab:::benchmark_data_smithi045_85309_object117478:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e495)", 2024-08-19T14:33:53.940 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:28:07.101155+0000", 2024-08-19T14:33:53.940 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 346.77292199099998, 2024-08-19T14:33:53.940 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.240822699, 2024-08-19T14:33:53.940 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:33:53.940 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:33:53.940 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:33:53.940 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:33:53.940 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25813", 2024-08-19T14:33:53.940 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:33:53.940 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 237731 2024-08-19T14:33:53.940 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.941 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:33:53.941 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.941 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:33:53.941 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101155+0000", 2024-08-19T14:33:53.941 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:53.941 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.941 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.941 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:33:53.941 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101155+0000", 2024-08-19T14:33:53.941 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:53.941 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.941 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.941 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:33:53.941 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101156+0000", 2024-08-19T14:33:53.942 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.5900000000000003e-07 2024-08-19T14:33:53.942 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.942 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.942 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:33:53.942 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101158+0000", 2024-08-19T14:33:53.942 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.0420000000000001e-06 2024-08-19T14:33:53.942 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.942 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.942 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:33:53.942 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101164+0000", 2024-08-19T14:33:53.942 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.7769999999999997e-06 2024-08-19T14:33:53.942 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.942 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.943 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:33:53.943 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101184+0000", 2024-08-19T14:33:53.943 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.9989000000000001e-05 2024-08-19T14:33:53.943 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.943 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.943 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:33:53.943 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101265+0000", 2024-08-19T14:33:53.943 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.0782e-05 2024-08-19T14:33:53.943 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.943 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.943 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:33:53.943 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101274+0000", 2024-08-19T14:33:53.943 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.0780000000000002e-06 2024-08-19T14:33:53.943 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.944 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.944 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:33:53.944 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.320042+0000", 2024-08-19T14:33:53.944 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.21876728300000001 2024-08-19T14:33:53.944 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.944 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.944 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:33:53.944 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.320209+0000", 2024-08-19T14:33:53.944 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00016716800000000001 2024-08-19T14:33:53.944 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.944 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.944 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,5", 2024-08-19T14:33:53.944 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.320830+0000", 2024-08-19T14:33:53.944 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00062140999999999997 2024-08-19T14:33:53.945 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.945 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.945 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:33:53.945 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.332354+0000", 2024-08-19T14:33:53.945 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.011524123000000001 2024-08-19T14:33:53.945 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.945 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.945 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:53.945 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.341182+0000", 2024-08-19T14:33:53.945 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0088281499999999999 2024-08-19T14:33:53.945 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.945 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.945 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:53.945 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.341703+0000", 2024-08-19T14:33:53.946 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000521022 2024-08-19T14:33:53.946 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.946 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.946 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:33:53.946 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.341740+0000", 2024-08-19T14:33:53.946 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.6885999999999999e-05 2024-08-19T14:33:53.946 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.946 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.946 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:33:53.946 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.341978+0000", 2024-08-19T14:33:53.946 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00023763000000000001 2024-08-19T14:33:53.946 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:53.946 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:33:53.947 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:53.947 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.947 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.947 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25813.0:237758 5.7 5:e57a2935:::benchmark_data_smithi045_85309_object117505:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e495)", 2024-08-19T14:33:53.947 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:28:07.101204+0000", 2024-08-19T14:33:53.947 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 346.77287296200001, 2024-08-19T14:33:53.947 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.24687310400000001, 2024-08-19T14:33:53.947 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:33:53.947 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:33:53.947 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:33:53.947 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:33:53.947 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25813", 2024-08-19T14:33:53.947 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:33:53.947 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 237758 2024-08-19T14:33:53.947 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.948 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:33:53.948 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.948 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:33:53.948 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101204+0000", 2024-08-19T14:33:53.948 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:53.948 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.948 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.948 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:33:53.948 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101204+0000", 2024-08-19T14:33:53.948 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:53.948 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.948 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.948 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:33:53.949 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101205+0000", 2024-08-19T14:33:53.949 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2800000000000002e-07 2024-08-19T14:33:53.949 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.949 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.949 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:33:53.949 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101207+0000", 2024-08-19T14:33:53.949 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.768e-06 2024-08-19T14:33:53.949 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.949 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.949 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:33:53.949 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101215+0000", 2024-08-19T14:33:53.949 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.3900000000000004e-06 2024-08-19T14:33:53.949 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.950 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.950 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:33:53.950 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101220+0000", 2024-08-19T14:33:53.950 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.3040000000000004e-06 2024-08-19T14:33:53.950 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.950 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.950 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:33:53.950 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101303+0000", 2024-08-19T14:33:53.950 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.2443999999999999e-05 2024-08-19T14:33:53.950 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.950 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.950 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:33:53.950 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101311+0000", 2024-08-19T14:33:53.951 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.3389999999999993e-06 2024-08-19T14:33:53.951 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.951 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.951 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:33:53.951 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.324142+0000", 2024-08-19T14:33:53.951 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.222830844 2024-08-19T14:33:53.951 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.951 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.951 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:33:53.951 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.324274+0000", 2024-08-19T14:33:53.951 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013226199999999999 2024-08-19T14:33:53.951 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.951 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.951 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,5", 2024-08-19T14:33:53.952 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.324842+0000", 2024-08-19T14:33:53.952 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00056831899999999996 2024-08-19T14:33:53.952 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.952 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.952 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:33:53.952 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.332438+0000", 2024-08-19T14:33:53.952 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0075956060000000004 2024-08-19T14:33:53.952 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.952 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.952 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:53.952 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.342013+0000", 2024-08-19T14:33:53.952 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0095747650000000007 2024-08-19T14:33:53.953 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.953 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.953 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:53.953 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.347519+0000", 2024-08-19T14:33:53.953 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0055063330000000004 2024-08-19T14:33:53.953 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.953 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.953 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:33:53.953 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.347576+0000", 2024-08-19T14:33:53.953 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.6610000000000002e-05 2024-08-19T14:33:53.953 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.953 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.953 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:33:53.954 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.348077+0000", 2024-08-19T14:33:53.954 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00050169199999999998 2024-08-19T14:33:53.954 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:53.954 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:33:53.954 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:53.954 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.954 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.954 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:76374 6.d 6:bf7b5a72:::benchmark_data_smithi045_97937_object76373:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e526)", 2024-08-19T14:33:53.954 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:29:17.862978+0000", 2024-08-19T14:33:53.954 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 276.011099505, 2024-08-19T14:33:53.954 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.44319341499999998, 2024-08-19T14:33:53.954 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:33:53.955 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:33:53.955 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:33:53.955 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:33:53.955 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:33:53.955 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:33:53.955 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 76374 2024-08-19T14:33:53.955 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.955 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:33:53.955 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.955 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:33:53.955 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.862978+0000", 2024-08-19T14:33:53.955 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:53.955 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.955 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.956 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:33:53.956 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.862978+0000", 2024-08-19T14:33:53.956 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:53.956 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.956 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.956 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:33:53.956 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.862980+0000", 2024-08-19T14:33:53.956 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.39e-06 2024-08-19T14:33:53.956 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.956 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.956 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:33:53.956 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.863020+0000", 2024-08-19T14:33:53.956 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.9978999999999998e-05 2024-08-19T14:33:53.957 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.957 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.957 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:33:53.957 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:18.285688+0000", 2024-08-19T14:33:53.957 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.42266810500000002 2024-08-19T14:33:53.957 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.957 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.957 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:33:53.957 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:18.285699+0000", 2024-08-19T14:33:53.957 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1187e-05 2024-08-19T14:33:53.957 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.957 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.957 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:33:53.958 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:18.288107+0000", 2024-08-19T14:33:53.958 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0024075279999999999 2024-08-19T14:33:53.958 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.958 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.958 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:33:53.958 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:18.288250+0000", 2024-08-19T14:33:53.958 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00014349400000000001 2024-08-19T14:33:53.958 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.958 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.958 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,5", 2024-08-19T14:33:53.958 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:18.288380+0000", 2024-08-19T14:33:53.958 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013004100000000001 2024-08-19T14:33:53.958 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.959 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.959 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:53.959 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:18.294529+0000", 2024-08-19T14:33:53.959 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0061484909999999999 2024-08-19T14:33:53.959 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.959 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.959 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:33:53.959 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:18.300842+0000", 2024-08-19T14:33:53.959 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0063133299999999998 2024-08-19T14:33:53.960 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.960 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.960 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:53.960 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:18.305954+0000", 2024-08-19T14:33:53.960 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.005111979 2024-08-19T14:33:53.960 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.960 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.960 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:33:53.960 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:18.306116+0000", 2024-08-19T14:33:53.960 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000161531 2024-08-19T14:33:53.960 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.960 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.960 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:33:53.961 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:18.306171+0000", 2024-08-19T14:33:53.961 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.5359999999999999e-05 2024-08-19T14:33:53.961 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:53.961 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:33:53.961 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:53.961 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.961 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.961 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:174030 6.d 6:b7de3b0b:::benchmark_data_smithi045_97937_object66757:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e580)", 2024-08-19T14:33:53.961 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:32:12.456928+0000", 2024-08-19T14:33:53.961 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 101.417149322, 2024-08-19T14:33:53.961 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.31321180500000001, 2024-08-19T14:33:53.961 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:33:53.961 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:33:53.961 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:33:53.962 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:33:53.962 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:33:53.962 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:33:53.962 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 174030 2024-08-19T14:33:53.962 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.962 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:33:53.962 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.962 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:33:53.962 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.456928+0000", 2024-08-19T14:33:53.962 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:53.962 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.962 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.962 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:33:53.963 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.456928+0000", 2024-08-19T14:33:53.963 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:53.963 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.963 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.963 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:33:53.963 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.456929+0000", 2024-08-19T14:33:53.963 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5600000000000001e-06 2024-08-19T14:33:53.963 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.963 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.963 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:33:53.963 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.456935+0000", 2024-08-19T14:33:53.963 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.8599999999999998e-06 2024-08-19T14:33:53.963 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.963 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.963 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:33:53.964 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.456950+0000", 2024-08-19T14:33:53.964 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4413000000000001e-05 2024-08-19T14:33:53.964 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.964 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.964 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:33:53.964 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.456957+0000", 2024-08-19T14:33:53.964 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.3819999999999998e-06 2024-08-19T14:33:53.964 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.964 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.964 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:33:53.964 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.457017+0000", 2024-08-19T14:33:53.964 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.9490999999999997e-05 2024-08-19T14:33:53.964 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.965 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.965 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:33:53.965 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.457136+0000", 2024-08-19T14:33:53.965 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000119491 2024-08-19T14:33:53.965 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.965 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.965 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,5", 2024-08-19T14:33:53.965 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.457262+0000", 2024-08-19T14:33:53.965 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012542200000000001 2024-08-19T14:33:53.965 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.965 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.965 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:33:53.965 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.466456+0000", 2024-08-19T14:33:53.966 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0091943749999999994 2024-08-19T14:33:53.966 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.966 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.966 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:53.966 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.466505+0000", 2024-08-19T14:33:53.966 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.8785000000000001e-05 2024-08-19T14:33:53.966 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.966 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.966 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:53.966 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.770050+0000", 2024-08-19T14:33:53.966 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.30354529000000002 2024-08-19T14:33:53.966 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.966 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.967 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:33:53.967 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.770096+0000", 2024-08-19T14:33:53.967 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.6480000000000002e-05 2024-08-19T14:33:53.967 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.967 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.967 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:33:53.967 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.770140+0000", 2024-08-19T14:33:53.967 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.3256000000000001e-05 2024-08-19T14:33:53.967 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:53.967 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:33:53.967 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:53.967 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.967 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.968 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:174032 6.d 6:b8a5c293:::benchmark_data_smithi045_97937_object66759:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e580)", 2024-08-19T14:33:53.968 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:32:12.458962+0000", 2024-08-19T14:33:53.968 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 101.415115063, 2024-08-19T14:33:53.968 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.31142265299999999, 2024-08-19T14:33:53.968 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:33:53.968 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:33:53.968 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:33:53.968 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:33:53.968 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:33:53.968 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:33:53.968 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 174032 2024-08-19T14:33:53.968 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.968 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:33:53.968 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.969 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:33:53.969 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.458962+0000", 2024-08-19T14:33:53.969 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:53.969 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.969 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.969 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:33:53.969 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.458962+0000", 2024-08-19T14:33:53.969 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:53.969 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.969 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.969 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:33:53.969 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.458963+0000", 2024-08-19T14:33:53.969 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1289999999999999e-06 2024-08-19T14:33:53.970 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.970 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.970 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:33:53.970 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.458968+0000", 2024-08-19T14:33:53.970 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4830000000000001e-06 2024-08-19T14:33:53.970 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.970 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.970 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:33:53.970 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.458980+0000", 2024-08-19T14:33:53.970 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2476999999999999e-05 2024-08-19T14:33:53.970 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.970 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.970 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:33:53.970 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.458987+0000", 2024-08-19T14:33:53.971 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.3729999999999998e-06 2024-08-19T14:33:53.971 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.971 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.971 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:33:53.971 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.461794+0000", 2024-08-19T14:33:53.971 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0028076809999999998 2024-08-19T14:33:53.971 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.971 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.971 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:33:53.971 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.461948+0000", 2024-08-19T14:33:53.971 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00015364299999999999 2024-08-19T14:33:53.971 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.971 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.972 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,5", 2024-08-19T14:33:53.972 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.462046+0000", 2024-08-19T14:33:53.972 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.7576000000000002e-05 2024-08-19T14:33:53.972 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.972 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.972 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:33:53.972 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.466596+0000", 2024-08-19T14:33:53.972 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0045504179999999997 2024-08-19T14:33:53.972 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.972 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.972 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:53.972 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.470406+0000", 2024-08-19T14:33:53.972 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.003809977 2024-08-19T14:33:53.973 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.973 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.973 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:53.973 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.770183+0000", 2024-08-19T14:33:53.973 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.29977723699999997 2024-08-19T14:33:53.973 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.973 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.973 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:33:53.973 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.770239+0000", 2024-08-19T14:33:53.973 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.5961999999999997e-05 2024-08-19T14:33:53.973 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.973 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.973 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:33:53.974 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.770385+0000", 2024-08-19T14:33:53.974 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000145697 2024-08-19T14:33:53.974 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:53.974 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:33:53.974 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:53.974 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.974 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.974 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:174039 6.d 6:b7cfbf20:::benchmark_data_smithi045_97937_object66766:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e580)", 2024-08-19T14:33:53.974 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:32:12.470750+0000", 2024-08-19T14:33:53.974 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 101.403327092, 2024-08-19T14:33:53.974 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.299785941, 2024-08-19T14:33:53.974 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:33:53.974 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:33:53.975 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:33:53.975 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:33:53.975 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:33:53.975 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:33:53.975 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 174039 2024-08-19T14:33:53.975 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.975 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:33:53.975 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.975 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:33:53.975 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.470750+0000", 2024-08-19T14:33:53.975 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:53.975 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.975 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.975 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:33:53.976 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.470750+0000", 2024-08-19T14:33:53.976 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:53.976 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.976 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.976 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:33:53.976 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.470752+0000", 2024-08-19T14:33:53.976 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.739e-06 2024-08-19T14:33:53.976 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.976 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.976 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:33:53.976 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.470759+0000", 2024-08-19T14:33:53.976 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.754e-06 2024-08-19T14:33:53.976 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.977 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.977 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:33:53.977 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.470773+0000", 2024-08-19T14:33:53.977 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4365000000000001e-05 2024-08-19T14:33:53.977 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.977 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.977 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:33:53.977 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.470780+0000", 2024-08-19T14:33:53.977 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.4220000000000003e-06 2024-08-19T14:33:53.977 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.977 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.977 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:33:53.977 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.470837+0000", 2024-08-19T14:33:53.977 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.6246999999999999e-05 2024-08-19T14:33:53.977 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.978 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.978 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:33:53.978 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.471823+0000", 2024-08-19T14:33:53.978 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00098676800000000006 2024-08-19T14:33:53.978 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.978 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.978 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,5", 2024-08-19T14:33:53.978 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.471942+0000", 2024-08-19T14:33:53.978 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00011807399999999999 2024-08-19T14:33:53.978 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.978 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.978 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:33:53.978 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.476422+0000", 2024-08-19T14:33:53.979 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0044803660000000004 2024-08-19T14:33:53.979 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.979 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.979 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:53.979 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.477208+0000", 2024-08-19T14:33:53.979 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00078598299999999997 2024-08-19T14:33:53.979 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.979 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.979 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:53.979 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.770437+0000", 2024-08-19T14:33:53.979 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.293229036 2024-08-19T14:33:53.979 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.979 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.980 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:33:53.980 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.770497+0000", 2024-08-19T14:33:53.980 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.9998999999999999e-05 2024-08-19T14:33:53.980 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.980 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.980 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:33:53.980 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.770536+0000", 2024-08-19T14:33:53.980 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.9187999999999998e-05 2024-08-19T14:33:53.980 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:53.980 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:33:53.980 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:53.980 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.980 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.980 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:174060 6.d 6:bd715b6c:::benchmark_data_smithi045_97937_object66787:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e580)", 2024-08-19T14:33:53.981 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:32:12.500214+0000", 2024-08-19T14:33:53.981 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 101.373863192, 2024-08-19T14:33:53.981 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.27045459700000002, 2024-08-19T14:33:53.981 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:33:53.981 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:33:53.981 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:33:53.981 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:33:53.981 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:33:53.981 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:33:53.981 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 174060 2024-08-19T14:33:53.981 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.981 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:33:53.982 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.982 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:33:53.982 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.500214+0000", 2024-08-19T14:33:53.982 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:53.982 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.982 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.982 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:33:53.982 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.500214+0000", 2024-08-19T14:33:53.982 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:53.982 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.982 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.982 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:33:53.982 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.500215+0000", 2024-08-19T14:33:53.983 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.091e-06 2024-08-19T14:33:53.983 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.983 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.983 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:33:53.983 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.500219+0000", 2024-08-19T14:33:53.983 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2150000000000003e-06 2024-08-19T14:33:53.983 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.983 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.983 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:33:53.983 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.500229+0000", 2024-08-19T14:33:53.983 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.8789999999999998e-06 2024-08-19T14:33:53.983 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.983 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.984 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:33:53.984 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.500234+0000", 2024-08-19T14:33:53.984 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0590000000000002e-06 2024-08-19T14:33:53.984 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.984 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.984 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:33:53.984 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.500290+0000", 2024-08-19T14:33:53.984 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.6144999999999999e-05 2024-08-19T14:33:53.984 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.984 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.984 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:33:53.984 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.500418+0000", 2024-08-19T14:33:53.984 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012713199999999999 2024-08-19T14:33:53.985 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.985 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.985 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,5", 2024-08-19T14:33:53.985 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.500544+0000", 2024-08-19T14:33:53.985 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012666899999999999 2024-08-19T14:33:53.985 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.985 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.985 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:33:53.985 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.504930+0000", 2024-08-19T14:33:53.985 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0043855669999999999 2024-08-19T14:33:53.985 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.985 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.985 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:53.985 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.506635+0000", 2024-08-19T14:33:53.986 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.001705266 2024-08-19T14:33:53.986 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.986 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.986 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:53.986 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.770578+0000", 2024-08-19T14:33:53.986 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.26394303899999999 2024-08-19T14:33:53.986 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.986 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.986 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:33:53.986 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.770629+0000", 2024-08-19T14:33:53.986 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1159000000000003e-05 2024-08-19T14:33:53.986 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.987 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.987 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:33:53.987 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.770669+0000", 2024-08-19T14:33:53.987 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.9375999999999997e-05 2024-08-19T14:33:53.987 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:53.987 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:33:53.987 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:53.987 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.987 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.987 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:174071 6.d 6:b8039fbe:::benchmark_data_smithi045_97937_object66798:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e580)", 2024-08-19T14:33:53.987 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:32:12.518669+0000", 2024-08-19T14:33:53.987 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 101.355408258, 2024-08-19T14:33:53.987 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.25216244300000001, 2024-08-19T14:33:53.987 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:33:53.988 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:33:53.988 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:33:53.988 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:33:53.988 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:33:53.988 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:33:53.988 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 174071 2024-08-19T14:33:53.988 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.988 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:33:53.988 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.988 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:33:53.988 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.518669+0000", 2024-08-19T14:33:53.988 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:53.988 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.989 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.989 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:33:53.989 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.518669+0000", 2024-08-19T14:33:53.989 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:53.989 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.989 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.989 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:33:53.989 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.518670+0000", 2024-08-19T14:33:53.989 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3009999999999999e-06 2024-08-19T14:33:53.989 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.989 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.989 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:33:53.989 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.518676+0000", 2024-08-19T14:33:53.989 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.8039999999999998e-06 2024-08-19T14:33:53.990 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.990 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.990 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:33:53.990 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.518689+0000", 2024-08-19T14:33:53.990 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2546e-05 2024-08-19T14:33:53.990 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.990 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.990 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:33:53.990 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.518694+0000", 2024-08-19T14:33:53.990 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.4750000000000001e-06 2024-08-19T14:33:53.990 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.990 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.991 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:33:53.991 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.518742+0000", 2024-08-19T14:33:53.991 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.8078999999999998e-05 2024-08-19T14:33:53.991 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.991 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.991 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:33:53.991 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.518849+0000", 2024-08-19T14:33:53.991 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00010722600000000001 2024-08-19T14:33:53.991 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.991 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.991 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,5", 2024-08-19T14:33:53.991 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.518980+0000", 2024-08-19T14:33:53.991 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013033599999999999 2024-08-19T14:33:53.992 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.992 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.992 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:33:53.992 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.523029+0000", 2024-08-19T14:33:53.992 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0040492619999999997 2024-08-19T14:33:53.992 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.992 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.992 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:53.992 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.523872+0000", 2024-08-19T14:33:53.992 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00084340800000000003 2024-08-19T14:33:53.992 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.992 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.993 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:33:53.994 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:53.994 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.770717+0000", 2024-08-19T14:33:53.994 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.24684499800000001 2024-08-19T14:33:53.994 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.994 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.994 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:33:53.994 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.770768+0000", 2024-08-19T14:33:53.994 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0438000000000001e-05 2024-08-19T14:33:53.994 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.994 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.995 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:33:53.995 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.770831+0000", 2024-08-19T14:33:53.995 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.3570000000000003e-05 2024-08-19T14:33:53.995 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:53.995 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:33:53.995 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:53.995 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.995 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.995 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:174075 6.d 6:bbccb6e4:::benchmark_data_smithi045_97937_object66802:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e580)", 2024-08-19T14:33:53.995 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:32:12.527198+0000", 2024-08-19T14:33:53.995 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 101.34687960700001, 2024-08-19T14:33:53.995 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.24381356800000001, 2024-08-19T14:33:53.995 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:33:53.995 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:33:53.996 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:33:53.996 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:33:53.996 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:33:53.996 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:33:53.996 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 174075 2024-08-19T14:33:53.996 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.996 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:33:53.996 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.996 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:33:53.996 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.527198+0000", 2024-08-19T14:33:53.996 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:53.996 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.996 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.997 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:33:53.997 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.527198+0000", 2024-08-19T14:33:53.997 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:53.997 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.997 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.997 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:33:53.997 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.527199+0000", 2024-08-19T14:33:53.997 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.685e-06 2024-08-19T14:33:53.997 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.997 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.997 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:33:53.997 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.527206+0000", 2024-08-19T14:33:53.997 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.2400000000000004e-06 2024-08-19T14:33:53.998 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.998 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.998 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:33:53.998 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.527220+0000", 2024-08-19T14:33:53.998 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4815e-05 2024-08-19T14:33:53.998 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.998 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.998 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:33:53.998 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.527228+0000", 2024-08-19T14:33:53.998 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.1999999999999997e-06 2024-08-19T14:33:53.998 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.998 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.998 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:33:53.999 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.527298+0000", 2024-08-19T14:33:53.999 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.0264000000000004e-05 2024-08-19T14:33:53.999 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.999 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.999 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:33:53.999 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.527391+0000", 2024-08-19T14:33:53.999 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.2843000000000005e-05 2024-08-19T14:33:53.999 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:53.999 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:53.999 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,5", 2024-08-19T14:33:53.999 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.527502+0000", 2024-08-19T14:33:53.999 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000111468 2024-08-19T14:33:53.999 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:54.000 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:54.000 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:33:54.000 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.531337+0000", 2024-08-19T14:33:54.000 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00383468 2024-08-19T14:33:54.000 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:54.000 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:54.000 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:54.000 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.531933+0000", 2024-08-19T14:33:54.000 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00059663700000000001 2024-08-19T14:33:54.000 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:54.000 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:54.000 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:54.000 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.770928+0000", 2024-08-19T14:33:54.001 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.238994768 2024-08-19T14:33:54.001 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:54.001 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:54.001 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:33:54.001 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.770976+0000", 2024-08-19T14:33:54.001 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7500999999999998e-05 2024-08-19T14:33:54.001 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:54.001 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:54.001 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:33:54.001 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.771011+0000", 2024-08-19T14:33:54.001 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.5466999999999998e-05 2024-08-19T14:33:54.001 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:54.001 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:33:54.001 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:54.002 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:54.002 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:54.002 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:187409 6.d 6:b8c5c813:::benchmark_data_smithi045_97937_object80136:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e583)", 2024-08-19T14:33:54.002 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:32:32.490311+0000", 2024-08-19T14:33:54.002 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 81.383766015000006, 2024-08-19T14:33:54.002 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.20933813400000001, 2024-08-19T14:33:54.002 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:33:54.002 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:33:54.002 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:33:54.002 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:33:54.002 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:33:54.002 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:33:54.003 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 187409 2024-08-19T14:33:54.003 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:54.003 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:33:54.003 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:54.003 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:33:54.003 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.490311+0000", 2024-08-19T14:33:54.003 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:54.003 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:54.003 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:54.003 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:33:54.003 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.490311+0000", 2024-08-19T14:33:54.003 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:54.003 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:54.003 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:54.004 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:33:54.004 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.490312+0000", 2024-08-19T14:33:54.004 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.2500000000000004e-07 2024-08-19T14:33:54.004 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:54.004 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:54.004 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:33:54.004 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.490316+0000", 2024-08-19T14:33:54.004 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.1699999999999999e-06 2024-08-19T14:33:54.004 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:54.004 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:54.004 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:33:54.004 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.490328+0000", 2024-08-19T14:33:54.004 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2099000000000001e-05 2024-08-19T14:33:54.004 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:54.004 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:54.004 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:33:54.005 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.490336+0000", 2024-08-19T14:33:54.005 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.3390000000000004e-06 2024-08-19T14:33:54.005 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:54.005 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:54.005 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:33:54.005 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.490397+0000", 2024-08-19T14:33:54.005 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.1661000000000005e-05 2024-08-19T14:33:54.005 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:54.005 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:54.005 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:33:54.005 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.490500+0000", 2024-08-19T14:33:54.005 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00010226300000000001 2024-08-19T14:33:54.005 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:54.005 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:54.006 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,5", 2024-08-19T14:33:54.006 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.490697+0000", 2024-08-19T14:33:54.006 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00019770599999999999 2024-08-19T14:33:54.006 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:54.006 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:54.006 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:54.006 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.498835+0000", 2024-08-19T14:33:54.006 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0081380870000000004 2024-08-19T14:33:54.006 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:54.006 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:54.006 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:33:54.006 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.498960+0000", 2024-08-19T14:33:54.006 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012474900000000001 2024-08-19T14:33:54.006 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:54.007 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:54.007 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:54.007 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.699547+0000", 2024-08-19T14:33:54.007 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.200586546 2024-08-19T14:33:54.007 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:54.007 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:54.007 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:33:54.007 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.699603+0000", 2024-08-19T14:33:54.007 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.5849e-05 2024-08-19T14:33:54.007 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:54.007 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:54.007 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:33:54.007 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.699649+0000", 2024-08-19T14:33:54.008 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.6839999999999999e-05 2024-08-19T14:33:54.008 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:54.008 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:33:54.008 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:54.008 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:54.008 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:54.008 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:203073 6.4 6:2b68586f:::benchmark_data_smithi045_97937_object95800:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e589)", 2024-08-19T14:33:54.008 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:32:55.259652+0000", 2024-08-19T14:33:54.008 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 58.614425613999998, 2024-08-19T14:33:54.008 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.22078215300000001, 2024-08-19T14:33:54.008 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:33:54.008 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:33:54.009 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:33:54.009 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:33:54.009 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:33:54.009 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:33:54.009 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 203073 2024-08-19T14:33:54.009 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:54.009 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:33:54.009 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:54.009 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:33:54.009 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:55.259652+0000", 2024-08-19T14:33:54.009 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:54.009 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:54.009 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:54.009 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:33:54.010 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:55.259652+0000", 2024-08-19T14:33:54.010 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:33:54.010 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:54.010 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:54.010 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:33:54.010 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:55.259653+0000", 2024-08-19T14:33:54.010 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0699999999999999e-06 2024-08-19T14:33:54.010 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:54.010 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:54.010 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:33:54.010 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:55.259657+0000", 2024-08-19T14:33:54.010 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.822e-06 2024-08-19T14:33:54.011 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:54.011 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:54.011 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:33:54.011 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:55.259666+0000", 2024-08-19T14:33:54.011 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.9019999999999994e-06 2024-08-19T14:33:54.011 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:54.011 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:54.011 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:33:54.011 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:55.259672+0000", 2024-08-19T14:33:54.011 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.096e-06 2024-08-19T14:33:54.011 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:54.011 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:54.011 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:33:54.011 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:55.259719+0000", 2024-08-19T14:33:54.012 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7618999999999999e-05 2024-08-19T14:33:54.012 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:54.012 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:54.012 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:33:54.012 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:55.259795+0000", 2024-08-19T14:33:54.012 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.6328000000000005e-05 2024-08-19T14:33:54.012 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:54.012 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:54.012 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,5", 2024-08-19T14:33:54.012 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:55.259872+0000", 2024-08-19T14:33:54.012 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.6909999999999994e-05 2024-08-19T14:33:54.012 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:54.012 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:54.013 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:33:54.013 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:55.265067+0000", 2024-08-19T14:33:54.013 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0051942459999999996 2024-08-19T14:33:54.013 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:54.013 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:54.013 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:54.013 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:55.267154+0000", 2024-08-19T14:33:54.013 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.002087256 2024-08-19T14:33:54.013 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:54.013 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:54.013 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:33:54.013 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:55.480318+0000", 2024-08-19T14:33:54.013 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.213164136 2024-08-19T14:33:54.014 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:54.014 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:54.014 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:33:54.014 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:55.480378+0000", 2024-08-19T14:33:54.014 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.0195999999999998e-05 2024-08-19T14:33:54.014 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:33:54.014 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:33:54.014 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:33:54.014 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:55.480434+0000", 2024-08-19T14:33:54.014 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.5572000000000003e-05 2024-08-19T14:33:54.014 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:54.014 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:33:54.014 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:54.014 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:33:54.015 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:33:54.015 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:33:54.015 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:53 smithi032 ceph-mon[20004]: pgmap v1430: 47 pgs: 9 active+undersized, 7 active+undersized+degraded, 31 active+clean; 2.5 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 123 MiB/s wr, 1.97k op/s; 15697/120939 objects degraded (12.979%) 2024-08-19T14:33:54.015 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:53 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/4057479953' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:33:54.041 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:54.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:54.042 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:33:54.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:53 smithi045 ceph-mon[20143]: pgmap v1430: 47 pgs: 9 active+undersized, 7 active+undersized+degraded, 31 active+clean; 2.5 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 123 MiB/s wr, 1.97k op/s; 15697/120939 objects degraded (12.979%) 2024-08-19T14:33:54.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:53 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/4057479953' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:33:54.140 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-19T14:33:54.142 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:33:54.166 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:54.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:54.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:33:54.268 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:33:54.289 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:54.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:54.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:33:54.391 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:33:54.435 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:54.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:54.436 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:33:54.441 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 21 16 48955 48939 145.64 95.1875 0.00363889 0.00686188 2024-08-19T14:33:54.537 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:33:54.561 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:54.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:54.562 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:33:54.663 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:33:54.706 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:54.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:54.707 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:33:54.808 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:33:54.849 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:54.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:54.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:33:54.951 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:33:54.987 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:54.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:54.988 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:33:55.089 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:33:55.113 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:55.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:55.114 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:33:55.215 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:33:55.252 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:55.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:55.253 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:33:55.354 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:33:55.380 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:55.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:55.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:33:55.442 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 22 16 50909 50893 144.571 122.125 0.00978557 0.0069123 2024-08-19T14:33:55.482 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:33:55.520 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:55.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:55.521 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:33:55.622 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:33:55.647 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:55.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:55.648 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:33:55.749 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:33:55.770 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:55.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:55.771 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:33:55.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:55 smithi031 ceph-mon[16336]: pgmap v1431: 47 pgs: 9 active+undersized, 7 active+undersized+degraded, 31 active+clean; 2.6 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 128 MiB/s wr, 2.05k op/s; 15697/130095 objects degraded (12.066%) 2024-08-19T14:33:55.871 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:33:55.896 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:55.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:55.897 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:33:55.998 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:33:56.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:55 smithi032 ceph-mon[20004]: pgmap v1431: 47 pgs: 9 active+undersized, 7 active+undersized+degraded, 31 active+clean; 2.6 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 128 MiB/s wr, 2.05k op/s; 15697/130095 objects degraded (12.066%) 2024-08-19T14:33:56.023 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:56.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:56.024 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:33:56.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:55 smithi045 ceph-mon[20143]: pgmap v1431: 47 pgs: 9 active+undersized, 7 active+undersized+degraded, 31 active+clean; 2.6 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 128 MiB/s wr, 2.05k op/s; 15697/130095 objects degraded (12.066%) 2024-08-19T14:33:56.125 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:33:56.149 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:56.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:56.150 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:33:56.251 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:33:56.275 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:56.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:56.277 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:33:56.378 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:33:56.399 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:56.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:56.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:33:56.442 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 23 16 52527 52511 142.682 101.125 0.00263588 0.00700369 2024-08-19T14:33:56.500 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:33:56.527 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:56.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:56.528 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:33:56.629 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:33:56.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:56.666 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:56.666 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:33:56.767 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:33:56.806 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:56.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:56.807 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:33:56.908 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:33:56.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:56.935 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:56.935 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:33:57.036 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:33:57.061 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:57.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:57.062 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:33:57.163 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:33:57.184 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:57.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:57.185 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:33:57.286 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:33:57.310 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:57.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:57.311 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:33:57.412 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:33:57.438 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:57.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:57.439 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:33:57.442 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 24 16 54315 54299 141.393 111.75 0.0128234 0.00706844 2024-08-19T14:33:57.540 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:33:57.575 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:57.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:57.576 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:33:57.677 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:33:57.702 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:57.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:57.704 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:33:57.805 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:33:57.830 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:57.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:57.831 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:33:57.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:57 smithi031 ceph-mon[16336]: pgmap v1432: 47 pgs: 9 active+undersized, 7 active+undersized+degraded, 31 active+clean; 3.0 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 147 MiB/s wr, 2.35k op/s; 18604/145674 objects degraded (12.771%) 2024-08-19T14:33:57.932 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:33:57.958 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:57.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:57.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:33:58.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:57 smithi032 ceph-mon[20004]: pgmap v1432: 47 pgs: 9 active+undersized, 7 active+undersized+degraded, 31 active+clean; 3.0 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 147 MiB/s wr, 2.35k op/s; 18604/145674 objects degraded (12.771%) 2024-08-19T14:33:58.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:57 smithi045 ceph-mon[20143]: pgmap v1432: 47 pgs: 9 active+undersized, 7 active+undersized+degraded, 31 active+clean; 3.0 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 147 MiB/s wr, 2.35k op/s; 18604/145674 objects degraded (12.771%) 2024-08-19T14:33:58.060 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:33:58.085 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:58.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:58.086 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:33:58.187 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:33:58.223 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:58.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:58.224 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:33:58.325 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:33:58.348 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:58.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:58.349 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:33:58.442 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 25 16 56138 56122 140.294 113.938 0.00775879 0.00712153 2024-08-19T14:33:58.444 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-19T14:33:58.444 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-19T14:33:58.450 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:33:58.472 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:58.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:58.473 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:33:58.574 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:33:58.595 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:58.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:58.596 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:33:58.691 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:33:58.697 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:33:58.761 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:58.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:58.762 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:33:58.782 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:58 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 18604/145674 objects degraded (12.771%), 7 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-08-19T14:33:58.863 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:33:58.888 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:58.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:58.889 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:33:58.990 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:33:59.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:58 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 18604/145674 objects degraded (12.771%), 7 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-08-19T14:33:59.026 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:59.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:59.028 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:33:59.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:58 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 18604/145674 objects degraded (12.771%), 7 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-08-19T14:33:59.108 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-08-19T14:33:59.129 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:33:59.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:59.196 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:59.196 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:33:59.297 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:33:59.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:59.334 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:59.334 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:33:59.435 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:33:59.442 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 26 16 57569 57553 138.338 89.4375 0.00244567 0.00719274 2024-08-19T14:33:59.458 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:59.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:59.459 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:33:59.560 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:33:59.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:59.605 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:33:59.606 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:33:59.613 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:33:59.707 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:33:59.732 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:33:59.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:59.733 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:33:59.834 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:33:59.858 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:33:59.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:33:59.859 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:33:59.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:33:59 smithi031 ceph-mon[16336]: pgmap v1433: 47 pgs: 9 active+undersized, 7 active+undersized+degraded, 31 active+clean; 3.0 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 129 MiB/s wr, 2.07k op/s; 19143/148968 objects degraded (12.850%) 2024-08-19T14:33:59.930 INFO:teuthology.orchestra.run.smithi031.stderr:no valid command found; 10 closest matches: 2024-08-19T14:33:59.930 INFO:teuthology.orchestra.run.smithi031.stderr:0 2024-08-19T14:33:59.931 INFO:teuthology.orchestra.run.smithi031.stderr:1 2024-08-19T14:33:59.931 INFO:teuthology.orchestra.run.smithi031.stderr:2 2024-08-19T14:33:59.931 INFO:teuthology.orchestra.run.smithi031.stderr:abort 2024-08-19T14:33:59.931 INFO:teuthology.orchestra.run.smithi031.stderr:assert 2024-08-19T14:33:59.931 INFO:teuthology.orchestra.run.smithi031.stderr:bench [] [] [] [] 2024-08-19T14:33:59.931 INFO:teuthology.orchestra.run.smithi031.stderr:bluefs debug_inject_read_zeros 2024-08-19T14:33:59.931 INFO:teuthology.orchestra.run.smithi031.stderr:bluefs files list 2024-08-19T14:33:59.931 INFO:teuthology.orchestra.run.smithi031.stderr:bluefs stats 2024-08-19T14:33:59.931 INFO:teuthology.orchestra.run.smithi031.stderr:bluestore allocator dump block 2024-08-19T14:33:59.932 INFO:teuthology.orchestra.run.smithi031.stderr:Error EINVAL: invalid command 2024-08-19T14:33:59.960 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:34:00.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:33:59 smithi032 ceph-mon[20004]: pgmap v1433: 47 pgs: 9 active+undersized, 7 active+undersized+degraded, 31 active+clean; 3.0 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 129 MiB/s wr, 2.07k op/s; 19143/148968 objects degraded (12.850%) 2024-08-19T14:34:00.009 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:00.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:00.010 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:34:00.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:33:59 smithi045 ceph-mon[20143]: pgmap v1433: 47 pgs: 9 active+undersized, 7 active+undersized+degraded, 31 active+clean; 3.0 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 129 MiB/s wr, 2.07k op/s; 19143/148968 objects degraded (12.850%) 2024-08-19T14:34:00.111 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:34:00.135 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:00.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:00.136 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:34:00.237 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:34:00.262 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:00.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:00.264 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:34:00.365 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:34:00.403 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:00.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:00.404 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:34:00.442 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 27 16 59275 59259 137.163 106.625 0.0105987 0.00728619 2024-08-19T14:34:00.445 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:34:00.506 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:34:00.533 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:00.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:00.534 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:34:00.635 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:34:00.659 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:00.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:00.660 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:34:00.761 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:34:00.796 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:00.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:00.797 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:34:00.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:00 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:34:00.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:00 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:34:00.897 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:34:00.920 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:00.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:00.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:34:01.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:00 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:34:01.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:00 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:34:01.022 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:34:01.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:00 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:34:01.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:00 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:34:01.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:01.058 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:01.059 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:34:01.159 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:34:01.194 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:01.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:01.195 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:34:01.296 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:34:01.321 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:01.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:01.322 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:34:01.423 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:34:01.442 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 28 16 60900 60884 135.891 101.562 0.00501185 0.00735443 2024-08-19T14:34:01.447 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:01.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:01.448 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:34:01.549 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:34:01.572 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:01.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:01.573 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:34:01.617 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is unset 2024-08-19T14:34:01.674 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:34:01.697 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:01.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:01.698 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:34:01.799 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:34:01.823 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:01.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:01.824 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:34:01.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:01 smithi031 ceph-mon[16336]: pgmap v1434: 47 pgs: 9 active+undersized, 7 active+undersized+degraded, 31 active+clean; 3.3 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 112 MiB/s wr, 1.79k op/s; 19665/161922 objects degraded (12.145%) 2024-08-19T14:34:01.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:01 smithi031 ceph-mon[16336]: osdmap e612: 12 total, 11 up, 9 in 2024-08-19T14:34:01.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:01 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/383761793' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:34:01.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:01 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:34:01.925 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:34:01.948 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:01.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:01.950 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:34:02.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:01 smithi032 ceph-mon[20004]: pgmap v1434: 47 pgs: 9 active+undersized, 7 active+undersized+degraded, 31 active+clean; 3.3 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 112 MiB/s wr, 1.79k op/s; 19665/161922 objects degraded (12.145%) 2024-08-19T14:34:02.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:01 smithi032 ceph-mon[20004]: osdmap e612: 12 total, 11 up, 9 in 2024-08-19T14:34:02.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:01 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/383761793' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:34:02.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:01 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:34:02.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:01 smithi045 ceph-mon[20143]: pgmap v1434: 47 pgs: 9 active+undersized, 7 active+undersized+degraded, 31 active+clean; 3.3 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 112 MiB/s wr, 1.79k op/s; 19665/161922 objects degraded (12.145%) 2024-08-19T14:34:02.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:01 smithi045 ceph-mon[20143]: osdmap e612: 12 total, 11 up, 9 in 2024-08-19T14:34:02.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:01 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/383761793' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:34:02.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:01 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:34:02.051 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:34:02.088 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:02.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:02.089 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:34:02.190 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:34:02.224 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:02.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:02.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:34:02.326 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:34:02.360 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:02.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:02.361 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:34:02.442 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 29 16 62519 62503 134.694 101.188 0.0168226 0.00741805 2024-08-19T14:34:02.461 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:34:02.497 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:02.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:02.498 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:34:02.598 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:34:02.622 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:02.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:02.623 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:34:02.724 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:34:02.748 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:02.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:02.749 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:34:02.850 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:34:02.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:02 smithi031 ceph-mon[16336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:34:02.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:02 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:34:02.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:02 smithi031 ceph-mon[16336]: osdmap e613: 12 total, 11 up, 9 in 2024-08-19T14:34:02.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:02 smithi031 ceph-mon[16336]: 1.11 scrub starts 2024-08-19T14:34:02.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:02 smithi031 ceph-mon[16336]: 1.11 scrub ok 2024-08-19T14:34:02.876 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:02.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:02.877 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:34:02.978 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:34:02.999 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:02.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:03.000 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:34:03.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:02 smithi032 ceph-mon[20004]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:34:03.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:02 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:34:03.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:02 smithi032 ceph-mon[20004]: osdmap e613: 12 total, 11 up, 9 in 2024-08-19T14:34:03.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:02 smithi032 ceph-mon[20004]: 1.11 scrub starts 2024-08-19T14:34:03.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:02 smithi032 ceph-mon[20004]: 1.11 scrub ok 2024-08-19T14:34:03.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:02 smithi045 ceph-mon[20143]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:34:03.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:02 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:34:03.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:02 smithi045 ceph-mon[20143]: osdmap e613: 12 total, 11 up, 9 in 2024-08-19T14:34:03.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:02 smithi045 ceph-mon[20143]: 1.11 scrub starts 2024-08-19T14:34:03.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:02 smithi045 ceph-mon[20143]: 1.11 scrub ok 2024-08-19T14:34:03.101 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:34:03.122 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:03.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:03.123 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:34:03.223 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:34:03.244 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:03.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:03.245 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:34:03.346 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:34:03.369 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:03.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:03.369 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:34:03.442 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 30 16 63708 63692 132.681 74.3125 0.00342235 0.00753082 2024-08-19T14:34:03.470 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:34:03.493 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:03.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:03.494 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:34:03.511 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-19T14:34:03.511 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-19T14:34:03.595 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:34:03.618 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:03.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:03.619 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:34:03.698 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:34:03.720 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:34:03.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:03.782 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:03.786 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:34:03.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:03 smithi045 ceph-mon[20143]: pgmap v1437: 47 pgs: 9 active+undersized, 7 active+undersized+degraded, 31 active+clean; 3.6 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 116 MiB/s wr, 1.85k op/s; 22371/174612 objects degraded (12.812%) 2024-08-19T14:34:03.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:03 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 22371/174612 objects degraded (12.812%), 7 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-08-19T14:34:03.822 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:03 smithi031 ceph-mon[16336]: pgmap v1437: 47 pgs: 9 active+undersized, 7 active+undersized+degraded, 31 active+clean; 3.6 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 116 MiB/s wr, 1.85k op/s; 22371/174612 objects degraded (12.812%) 2024-08-19T14:34:03.822 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:03 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 22371/174612 objects degraded (12.812%), 7 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-08-19T14:34:03.888 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:34:03.913 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:03.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:03.914 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:34:04.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:03 smithi032 ceph-mon[20004]: pgmap v1437: 47 pgs: 9 active+undersized, 7 active+undersized+degraded, 31 active+clean; 3.6 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 116 MiB/s wr, 1.85k op/s; 22371/174612 objects degraded (12.812%) 2024-08-19T14:34:04.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:03 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 22371/174612 objects degraded (12.812%), 7 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-08-19T14:34:04.015 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:34:04.041 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:04.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:04.042 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:34:04.143 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:34:04.180 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:04.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:04.181 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:34:04.283 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:34:04.307 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:04.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:04.308 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:34:04.403 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:34:04.408 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:34:04.432 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:04.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:04.433 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:34:04.442 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 31 16 65410 65394 131.833 106.375 0.00710723 0.00758038 2024-08-19T14:34:04.534 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:34:04.558 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:04.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:04.559 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:34:04.660 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:34:04.684 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:04.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:04.685 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:34:04.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:04 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:34:04.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:04 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:34:04.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:04 smithi032 ceph-mon[20004]: osdmap e614: 12 total, 11 up, 9 in 2024-08-19T14:34:04.786 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:34:04.811 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:04.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:04.812 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:34:04.913 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:34:04.937 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:04 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:34:04.937 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:04 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:34:04.937 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:04 smithi031 ceph-mon[16336]: osdmap e614: 12 total, 11 up, 9 in 2024-08-19T14:34:04.937 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:04.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:04.938 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:34:05.039 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:34:05.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:04 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:34:05.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:04 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:34:05.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:04 smithi045 ceph-mon[20143]: osdmap e614: 12 total, 11 up, 9 in 2024-08-19T14:34:05.064 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:05.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:05.065 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:34:05.166 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:34:05.201 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:05.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:05.202 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:34:05.221 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:34:05.221 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [ 2024-08-19T14:34:05.222 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.222 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:66584 7.c 7:3c86191b:::benchmark_data_smithi045_115842_object66583:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e614)", 2024-08-19T14:34:05.222 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:34:05.208975+0000", 2024-08-19T14:34:05.222 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.0094136150000000002, 2024-08-19T14:34:05.222 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0094539849999999998, 2024-08-19T14:34:05.222 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:34:05.222 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:34:05.222 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "waiting for sub ops", 2024-08-19T14:34:05.222 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:34:05.222 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:34:05.223 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:34:05.223 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 66584 2024-08-19T14:34:05.223 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:05.223 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:34:05.223 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.223 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:34:05.223 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:34:05.208975+0000", 2024-08-19T14:34:05.223 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:34:05.223 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:05.223 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.223 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:34:05.223 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:34:05.208975+0000", 2024-08-19T14:34:05.223 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:34:05.223 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:05.224 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.224 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:34:05.224 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:34:05.208976+0000", 2024-08-19T14:34:05.224 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.066e-06 2024-08-19T14:34:05.224 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:05.224 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.224 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:34:05.224 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:34:05.209012+0000", 2024-08-19T14:34:05.224 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.5723999999999999e-05 2024-08-19T14:34:05.224 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:05.224 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.224 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:34:05.224 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:34:05.209024+0000", 2024-08-19T14:34:05.224 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1749000000000001e-05 2024-08-19T14:34:05.225 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:05.225 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.225 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:34:05.225 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:34:05.209030+0000", 2024-08-19T14:34:05.225 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.4880000000000004e-06 2024-08-19T14:34:05.225 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:05.225 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.225 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:34:05.225 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:34:05.209143+0000", 2024-08-19T14:34:05.225 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00011234 2024-08-19T14:34:05.225 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:05.225 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.225 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:34:05.225 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:34:05.209291+0000", 2024-08-19T14:34:05.225 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00014843500000000001 2024-08-19T14:34:05.226 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:05.226 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.226 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 11", 2024-08-19T14:34:05.226 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:34:05.209429+0000", 2024-08-19T14:34:05.226 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013809799999999999 2024-08-19T14:34:05.226 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:34:05.226 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:34:05.226 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:34:05.226 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:05.226 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.226 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:66590 7.c 7:37d42835:::benchmark_data_smithi045_115842_object66589:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e614)", 2024-08-19T14:34:05.226 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:34:05.211601+0000", 2024-08-19T14:34:05.226 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.0067874750000000003, 2024-08-19T14:34:05.227 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0069110150000000004, 2024-08-19T14:34:05.227 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:34:05.227 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:34:05.227 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "waiting for sub ops", 2024-08-19T14:34:05.227 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:34:05.227 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:34:05.227 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:34:05.227 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 66590 2024-08-19T14:34:05.227 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:05.227 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:34:05.228 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.228 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:34:05.228 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:34:05.211601+0000", 2024-08-19T14:34:05.228 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:34:05.228 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:05.228 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.228 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:34:05.228 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:34:05.211601+0000", 2024-08-19T14:34:05.228 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:34:05.228 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:05.228 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.228 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:34:05.228 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:34:05.211603+0000", 2024-08-19T14:34:05.228 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.488e-06 2024-08-19T14:34:05.229 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:05.229 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.229 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:34:05.229 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:34:05.211659+0000", 2024-08-19T14:34:05.229 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.6407999999999999e-05 2024-08-19T14:34:05.229 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:05.229 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.229 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:34:05.229 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:34:05.211670+0000", 2024-08-19T14:34:05.229 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0815999999999999e-05 2024-08-19T14:34:05.230 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:05.230 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.230 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:34:05.230 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:34:05.211679+0000", 2024-08-19T14:34:05.230 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.7080000000000003e-06 2024-08-19T14:34:05.230 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:05.230 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.230 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:34:05.230 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:34:05.214961+0000", 2024-08-19T14:34:05.230 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0032824210000000002 2024-08-19T14:34:05.231 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:05.231 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.231 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:34:05.231 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:34:05.215293+0000", 2024-08-19T14:34:05.231 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00033196000000000001 2024-08-19T14:34:05.231 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:05.231 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.231 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 11", 2024-08-19T14:34:05.231 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:34:05.215432+0000", 2024-08-19T14:34:05.231 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000138613 2024-08-19T14:34:05.231 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:34:05.231 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:34:05.231 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:34:05.231 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:05.231 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.232 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.16962.0:66591 7.1 e614/600 7:85af06d4:::benchmark_data_smithi045_115842_object66590:head v 614'4151, mlcod=614'4149)", 2024-08-19T14:34:05.232 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:34:05.212238+0000", 2024-08-19T14:34:05.232 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.0061506069999999998, 2024-08-19T14:34:05.232 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0063475370000000003, 2024-08-19T14:34:05.232 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:34:05.232 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:34:05.232 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-08-19T14:34:05.232 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:34:05.232 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.232 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:34:05.232 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:34:05.212238+0000", 2024-08-19T14:34:05.232 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:34:05.232 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:05.232 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.233 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:34:05.233 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:34:05.212238+0000", 2024-08-19T14:34:05.233 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:34:05.233 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:05.233 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.233 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:34:05.233 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:34:05.212239+0000", 2024-08-19T14:34:05.233 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.3600000000000004e-07 2024-08-19T14:34:05.233 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:05.233 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.233 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:34:05.233 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:34:05.212300+0000", 2024-08-19T14:34:05.233 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.1809000000000004e-05 2024-08-19T14:34:05.233 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:05.234 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.234 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:34:05.234 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:34:05.212313+0000", 2024-08-19T14:34:05.234 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2346e-05 2024-08-19T14:34:05.234 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:05.234 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.234 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:34:05.234 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:34:05.212320+0000", 2024-08-19T14:34:05.234 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.7519999999999999e-06 2024-08-19T14:34:05.234 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:05.234 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.234 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:34:05.234 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:34:05.212525+0000", 2024-08-19T14:34:05.234 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00020522800000000001 2024-08-19T14:34:05.235 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:05.235 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.235 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:34:05.235 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:34:05.212545+0000", 2024-08-19T14:34:05.235 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.0327999999999999e-05 2024-08-19T14:34:05.235 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:34:05.235 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:34:05.235 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:34:05.235 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:05.235 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.235 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:66593 7.c 7.660eaacc (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e614)", 2024-08-19T14:34:05.235 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:34:05.213331+0000", 2024-08-19T14:34:05.235 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.0050572990000000003, 2024-08-19T14:34:05.235 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0053135789999999997, 2024-08-19T14:34:05.235 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:34:05.236 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:34:05.236 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "queued for pg", 2024-08-19T14:34:05.236 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:34:05.236 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:34:05.236 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:34:05.236 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 66593 2024-08-19T14:34:05.236 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:05.236 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:34:05.236 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.236 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:34:05.236 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:34:05.213331+0000", 2024-08-19T14:34:05.236 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:34:05.236 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:05.236 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.237 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:34:05.237 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:34:05.213331+0000", 2024-08-19T14:34:05.237 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:34:05.237 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:05.237 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.237 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:34:05.237 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:34:05.213333+0000", 2024-08-19T14:34:05.237 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.671e-06 2024-08-19T14:34:05.237 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:05.237 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.237 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:34:05.237 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:34:05.213380+0000", 2024-08-19T14:34:05.237 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.6591000000000002e-05 2024-08-19T14:34:05.237 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:05.238 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.238 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:34:05.238 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:34:05.213395+0000", 2024-08-19T14:34:05.238 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5665999999999999e-05 2024-08-19T14:34:05.238 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:05.238 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.238 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:34:05.238 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:34:05.213403+0000", 2024-08-19T14:34:05.238 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.0229999999999992e-06 2024-08-19T14:34:05.238 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:34:05.238 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:34:05.238 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:34:05.238 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:05.238 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.239 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.16962.0:66592 7.1 e614/600 7:8d325f13:::benchmark_data_smithi045_115842_object66591:head v 614'4152, mlcod=614'4149)", 2024-08-19T14:34:05.239 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:34:05.214225+0000", 2024-08-19T14:34:05.239 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.0041636210000000002, 2024-08-19T14:34:05.239 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0044867759999999996, 2024-08-19T14:34:05.239 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:34:05.239 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:34:05.239 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-08-19T14:34:05.239 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:34:05.239 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.239 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:34:05.239 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:34:05.214225+0000", 2024-08-19T14:34:05.239 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:34:05.239 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:05.239 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.240 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:34:05.240 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:34:05.214225+0000", 2024-08-19T14:34:05.240 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:34:05.240 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:05.240 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.240 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:34:05.240 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:34:05.214226+0000", 2024-08-19T14:34:05.240 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.3099999999999996e-07 2024-08-19T14:34:05.240 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:05.240 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.240 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:34:05.240 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:34:05.214259+0000", 2024-08-19T14:34:05.240 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.3251999999999999e-05 2024-08-19T14:34:05.240 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:05.240 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.241 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:34:05.241 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:34:05.214272+0000", 2024-08-19T14:34:05.241 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2313999999999999e-05 2024-08-19T14:34:05.241 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:05.241 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.241 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:34:05.241 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:34:05.214278+0000", 2024-08-19T14:34:05.241 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.9319999999999999e-06 2024-08-19T14:34:05.241 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:05.241 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.241 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:34:05.242 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:34:05.214758+0000", 2024-08-19T14:34:05.242 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000479431 2024-08-19T14:34:05.242 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:05.242 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.243 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:34:05.243 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:34:05.214775+0000", 2024-08-19T14:34:05.243 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.7380999999999998e-05 2024-08-19T14:34:05.243 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:34:05.243 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:34:05.243 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:34:05.243 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:05.243 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.243 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.16962.0:66594 7.1 e614/600 7:818a0450:::benchmark_data_smithi045_115842_object66593:head v 614'4153, mlcod=614'4149)", 2024-08-19T14:34:05.243 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:34:05.215762+0000", 2024-08-19T14:34:05.243 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.0026271010000000002, 2024-08-19T14:34:05.243 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.002982354, 2024-08-19T14:34:05.243 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:34:05.243 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:34:05.244 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-08-19T14:34:05.244 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:34:05.244 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.244 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:34:05.244 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:34:05.215762+0000", 2024-08-19T14:34:05.244 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:34:05.244 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:05.244 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.244 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:34:05.244 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:34:05.215762+0000", 2024-08-19T14:34:05.244 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:34:05.244 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:05.244 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.244 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:34:05.245 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:34:05.215763+0000", 2024-08-19T14:34:05.245 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.47e-07 2024-08-19T14:34:05.245 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:05.245 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.245 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:34:05.245 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:34:05.215796+0000", 2024-08-19T14:34:05.245 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.3503000000000001e-05 2024-08-19T14:34:05.245 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:05.245 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.245 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:34:05.245 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:34:05.215810+0000", 2024-08-19T14:34:05.245 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3447e-05 2024-08-19T14:34:05.246 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:05.246 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.246 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:34:05.246 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:34:05.215817+0000", 2024-08-19T14:34:05.246 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.2690000000000002e-06 2024-08-19T14:34:05.246 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:05.246 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.246 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:34:05.246 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:34:05.216841+0000", 2024-08-19T14:34:05.247 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0010244150000000001 2024-08-19T14:34:05.247 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:05.247 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.247 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:34:05.247 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:34:05.216876+0000", 2024-08-19T14:34:05.247 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.4601999999999998e-05 2024-08-19T14:34:05.247 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:34:05.247 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:34:05.247 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:34:05.247 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:05.247 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.247 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:66597 7.2 7:4642c826:::benchmark_data_smithi045_115842_object66596:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e614)", 2024-08-19T14:34:05.247 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:34:05.215767+0000", 2024-08-19T14:34:05.247 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.0026218320000000002, 2024-08-19T14:34:05.248 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0030113029999999999, 2024-08-19T14:34:05.248 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:34:05.248 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:34:05.248 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "waiting for sub ops", 2024-08-19T14:34:05.248 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:34:05.248 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:34:05.248 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:34:05.248 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 66597 2024-08-19T14:34:05.248 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:05.248 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:34:05.248 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.248 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:34:05.248 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:34:05.215767+0000", 2024-08-19T14:34:05.248 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:34:05.249 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:05.249 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.249 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:34:05.249 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:34:05.215767+0000", 2024-08-19T14:34:05.249 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:34:05.249 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:05.249 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.249 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:34:05.249 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:34:05.215768+0000", 2024-08-19T14:34:05.249 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1769999999999999e-06 2024-08-19T14:34:05.249 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:05.249 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.249 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:34:05.249 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:34:05.215804+0000", 2024-08-19T14:34:05.249 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.6159999999999999e-05 2024-08-19T14:34:05.250 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:05.250 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.250 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:34:05.250 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:34:05.215817+0000", 2024-08-19T14:34:05.250 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2867999999999999e-05 2024-08-19T14:34:05.250 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:05.250 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.250 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:34:05.250 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:34:05.215824+0000", 2024-08-19T14:34:05.250 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.8929999999999999e-06 2024-08-19T14:34:05.250 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:05.250 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.250 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:34:05.250 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:34:05.215886+0000", 2024-08-19T14:34:05.251 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.1859000000000005e-05 2024-08-19T14:34:05.251 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:05.251 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.251 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:34:05.251 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:34:05.216031+0000", 2024-08-19T14:34:05.251 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00014479900000000001 2024-08-19T14:34:05.251 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:05.252 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.252 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 7", 2024-08-19T14:34:05.252 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:34:05.216166+0000", 2024-08-19T14:34:05.252 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013563599999999999 2024-08-19T14:34:05.252 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:34:05.252 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:34:05.252 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:34:05.252 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:05.252 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.252 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.16962.0:66599 7.a e614/600 7:50aa9f9d:::benchmark_data_smithi045_115842_object66598:head v 614'4135, mlcod=614'4133)", 2024-08-19T14:34:05.252 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:34:05.217097+0000", 2024-08-19T14:34:05.252 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.0012918929999999999, 2024-08-19T14:34:05.253 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00172298, 2024-08-19T14:34:05.253 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:34:05.253 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:34:05.253 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-08-19T14:34:05.253 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:34:05.253 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.253 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:34:05.253 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:34:05.217097+0000", 2024-08-19T14:34:05.253 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:34:05.253 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:05.253 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.253 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:34:05.253 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:34:05.217097+0000", 2024-08-19T14:34:05.254 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:34:05.254 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:05.254 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.254 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:34:05.254 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:34:05.217098+0000", 2024-08-19T14:34:05.254 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.3500000000000005e-07 2024-08-19T14:34:05.254 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:05.254 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.254 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:34:05.254 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:34:05.217133+0000", 2024-08-19T14:34:05.254 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.5664000000000003e-05 2024-08-19T14:34:05.254 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:05.255 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.255 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:34:05.255 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:34:05.217148+0000", 2024-08-19T14:34:05.255 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4628e-05 2024-08-19T14:34:05.255 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:05.255 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.255 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:34:05.255 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:34:05.217155+0000", 2024-08-19T14:34:05.255 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.883e-06 2024-08-19T14:34:05.255 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:05.255 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.255 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:34:05.256 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:34:05.217544+0000", 2024-08-19T14:34:05.256 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00038864800000000001 2024-08-19T14:34:05.256 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:05.256 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.256 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:34:05.256 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:34:05.217564+0000", 2024-08-19T14:34:05.256 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.0537999999999999e-05 2024-08-19T14:34:05.256 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:34:05.256 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:34:05.257 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:34:05.257 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:05.257 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.257 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.16962.0:66596 7.1 e614/600)", 2024-08-19T14:34:05.257 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:34:05.217175+0000", 2024-08-19T14:34:05.257 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.0012136320000000001, 2024-08-19T14:34:05.257 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0016734090000000001, 2024-08-19T14:34:05.257 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:34:05.257 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:34:05.257 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "queued for pg", 2024-08-19T14:34:05.257 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:34:05.257 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.258 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:34:05.258 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:34:05.217175+0000", 2024-08-19T14:34:05.258 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:34:05.258 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:05.258 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.258 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:34:05.258 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:34:05.217175+0000", 2024-08-19T14:34:05.258 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:34:05.258 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:05.258 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.258 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:34:05.258 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:34:05.217176+0000", 2024-08-19T14:34:05.259 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.9900000000000001e-07 2024-08-19T14:34:05.259 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:05.259 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.259 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:34:05.259 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:34:05.217205+0000", 2024-08-19T14:34:05.259 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.9748999999999999e-05 2024-08-19T14:34:05.259 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:05.259 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.259 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:34:05.259 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:34:05.217216+0000", 2024-08-19T14:34:05.259 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0363000000000001e-05 2024-08-19T14:34:05.259 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:05.259 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.259 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:34:05.260 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:34:05.217221+0000", 2024-08-19T14:34:05.260 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.9339999999999997e-06 2024-08-19T14:34:05.260 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:34:05.260 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:34:05.260 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:34:05.260 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:05.260 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.260 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop_reply(client.16962.0:66590 7.c e614/600 ondisk, result = 0)", 2024-08-19T14:34:05.260 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:34:05.217340+0000", 2024-08-19T14:34:05.260 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.0010492469999999999, 2024-08-19T14:34:05.260 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.001549068, 2024-08-19T14:34:05.260 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:34:05.260 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:34:05.261 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-08-19T14:34:05.261 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:34:05.261 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.261 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:34:05.261 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:34:05.217340+0000", 2024-08-19T14:34:05.261 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:34:05.261 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:05.261 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.261 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:34:05.261 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:34:05.217340+0000", 2024-08-19T14:34:05.261 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:34:05.261 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:05.261 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.261 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:34:05.261 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:34:05.217340+0000", 2024-08-19T14:34:05.262 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.91e-07 2024-08-19T14:34:05.262 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:05.262 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.262 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:34:05.262 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:34:05.217346+0000", 2024-08-19T14:34:05.262 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.1569999999999997e-06 2024-08-19T14:34:05.262 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:05.262 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.262 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:34:05.262 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:34:05.217353+0000", 2024-08-19T14:34:05.262 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.5919999999999997e-06 2024-08-19T14:34:05.262 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:05.262 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.263 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:34:05.263 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:34:05.217357+0000", 2024-08-19T14:34:05.263 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.5379999999999996e-06 2024-08-19T14:34:05.263 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:05.263 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.263 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:34:05.263 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:34:05.218536+0000", 2024-08-19T14:34:05.263 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.001178498 2024-08-19T14:34:05.263 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:05.263 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.263 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:34:05.263 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:34:05.218543+0000", 2024-08-19T14:34:05.263 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.5619999999999998e-06 2024-08-19T14:34:05.263 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:34:05.264 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:34:05.264 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:34:05.264 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:05.264 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.264 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:66600 7.c 7.21c3130c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e614)", 2024-08-19T14:34:05.264 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:34:05.218647+0000", 2024-08-19T14:34:05.264 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 4294967295.9997416, 2024-08-19T14:34:05.264 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00027277600000000002, 2024-08-19T14:34:05.264 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:34:05.264 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:34:05.264 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "queued for pg", 2024-08-19T14:34:05.264 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:34:05.264 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:34:05.265 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:34:05.265 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 66600 2024-08-19T14:34:05.265 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:05.265 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:34:05.265 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.265 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:34:05.265 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:34:05.218647+0000", 2024-08-19T14:34:05.265 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:34:05.265 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:05.265 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.265 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:34:05.265 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:34:05.218647+0000", 2024-08-19T14:34:05.265 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:34:05.266 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:05.266 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.266 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:34:05.266 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:34:05.218649+0000", 2024-08-19T14:34:05.266 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4500000000000001e-06 2024-08-19T14:34:05.266 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:05.266 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.266 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:34:05.266 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:34:05.218683+0000", 2024-08-19T14:34:05.266 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.4022000000000001e-05 2024-08-19T14:34:05.266 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:05.266 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.267 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:34:05.267 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:34:05.218696+0000", 2024-08-19T14:34:05.267 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2719e-05 2024-08-19T14:34:05.267 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:05.267 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:05.267 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:34:05.267 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:34:05.218703+0000", 2024-08-19T14:34:05.267 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.3860000000000001e-06 2024-08-19T14:34:05.267 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:34:05.267 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:34:05.267 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:34:05.267 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:34:05.267 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-08-19T14:34:05.268 INFO:teuthology.orchestra.run.smithi031.stdout: "num_ops": 11 2024-08-19T14:34:05.268 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:34:05.303 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:34:05.326 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:05.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:05.327 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:34:05.428 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:34:05.443 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 32 16 66946 66930 130.713 96 0.0116184 0.00764623 2024-08-19T14:34:05.538 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:34:05.538 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:34:05.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:05.539 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:05.539 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:34:05.547 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:34:05.547 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:34:05.549 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-08-19T14:34:05.588 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:34:05.588 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:34:05.623 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:34:05.623 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:34:05.639 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:34:05.663 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:34:05.663 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:34:05.664 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:05.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:05.664 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:34:05.696 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:34:05.697 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:34:05.732 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:34:05.732 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:34:05.765 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:34:05.766 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:34:05.767 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:34:05.767 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:05 smithi031 ceph-mon[16336]: pgmap v1439: 47 pgs: 9 active+undersized, 7 active+undersized+degraded, 31 active+clean; 3.7 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 114 MiB/s wr, 1.83k op/s; 22371/181926 objects degraded (12.297%) 2024-08-19T14:34:05.767 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:05 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3389753855' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:34:05.799 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:05.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:05.800 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:34:05.804 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:34:05.804 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:34:05.847 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:34:05.848 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:34:05.885 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:34:05.886 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:34:05.901 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:34:05.925 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:05.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:05.926 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:34:06.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:05 smithi032 ceph-mon[20004]: pgmap v1439: 47 pgs: 9 active+undersized, 7 active+undersized+degraded, 31 active+clean; 3.7 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 114 MiB/s wr, 1.83k op/s; 22371/181926 objects degraded (12.297%) 2024-08-19T14:34:06.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:05 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3389753855' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:34:06.027 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:34:06.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:05 smithi045 ceph-mon[20143]: pgmap v1439: 47 pgs: 9 active+undersized, 7 active+undersized+degraded, 31 active+clean; 3.7 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 114 MiB/s wr, 1.83k op/s; 22371/181926 objects degraded (12.297%) 2024-08-19T14:34:06.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:05 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3389753855' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:34:06.051 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:06.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:06.052 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:34:06.153 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:34:06.178 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:06.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:06.179 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:34:06.280 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:34:06.306 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:06.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:06.307 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:34:06.351 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-19T14:34:06.355 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-08-19T14:34:06.407 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:34:06.429 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:06.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:06.430 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:34:06.442 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 33 15 68564 68549 129.818 101.188 0.00577177 0.00769826 2024-08-19T14:34:06.531 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:34:06.555 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:06.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:06.557 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:34:06.657 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:34:06.681 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:06.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:06.682 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:34:06.783 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:34:06.804 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:06.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:06.805 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:34:06.906 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:34:06.929 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:06.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:06.930 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:34:07.031 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:34:07.057 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:07.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:07.059 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:34:07.159 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:34:07.185 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:07.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:07.186 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:34:07.287 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:34:07.311 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:07.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:07.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:34:07.413 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:34:07.442 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 34 16 70034 70018 128.7 91.8125 0.00301555 0.00776533 2024-08-19T14:34:07.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:07.455 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:07.455 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:34:07.556 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:34:07.577 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:07.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:07.578 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:34:07.679 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:34:07.704 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:07.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:07.705 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:34:07.806 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:34:07.830 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:07.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:07.832 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:34:07.869 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:07 smithi031 ceph-mon[16336]: pgmap v1440: 47 pgs: 9 active+undersized, 7 active+undersized+degraded, 31 active+clean; 4.0 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 114 MiB/s wr, 1.83k op/s; 24790/194841 objects degraded (12.723%) 2024-08-19T14:34:07.869 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:07 smithi031 ceph-mon[16336]: 1.1a scrub starts 2024-08-19T14:34:07.869 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:07 smithi031 ceph-mon[16336]: 1.1a scrub ok 2024-08-19T14:34:07.925 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-19T14:34:07.926 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-19T14:34:07.932 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:34:07.956 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:07.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:07.957 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:34:07.991 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:34:08.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:07 smithi032 ceph-mon[20004]: pgmap v1440: 47 pgs: 9 active+undersized, 7 active+undersized+degraded, 31 active+clean; 4.0 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 114 MiB/s wr, 1.83k op/s; 24790/194841 objects degraded (12.723%) 2024-08-19T14:34:08.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:07 smithi032 ceph-mon[20004]: 1.1a scrub starts 2024-08-19T14:34:08.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:07 smithi032 ceph-mon[20004]: 1.1a scrub ok 2024-08-19T14:34:08.025 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:34:08.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:07 smithi045 ceph-mon[20143]: pgmap v1440: 47 pgs: 9 active+undersized, 7 active+undersized+degraded, 31 active+clean; 4.0 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 114 MiB/s wr, 1.83k op/s; 24790/194841 objects degraded (12.723%) 2024-08-19T14:34:08.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:07 smithi045 ceph-mon[20143]: 1.1a scrub starts 2024-08-19T14:34:08.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:07 smithi045 ceph-mon[20143]: 1.1a scrub ok 2024-08-19T14:34:08.058 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:34:08.082 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:08.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:08.083 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:34:08.184 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:34:08.207 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:08.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:08.208 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:34:08.309 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:34:08.333 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:08.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:08.334 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:34:08.435 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:34:08.442 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 35 16 71260 71244 127.212 76.625 0.0102895 0.00785633 2024-08-19T14:34:08.461 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:08.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:08.462 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:34:08.512 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 7, 8, 2, 11] out_osds: [10, 6, 9] dead_osds: [2] live_osds: [0, 1, 3, 4, 5, 6, 7, 8, 9, 11, 10] 2024-08-19T14:34:08.512 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-19T14:34:08.513 INFO:tasks.thrashosds.thrasher:Adding osd 9 2024-08-19T14:34:08.513 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd in 9 2024-08-19T14:34:08.562 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:34:08.585 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:08.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:08.586 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:34:08.686 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:34:08.686 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:34:08.687 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:34:08.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:08.709 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:08.709 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:34:08.810 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:34:08.833 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:08.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:08.834 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:34:08.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:08 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 24790/194841 objects degraded (12.723%), 7 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-08-19T14:34:08.935 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:34:08.979 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:08.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:08.984 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:34:09.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:08 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 24790/194841 objects degraded (12.723%), 7 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-08-19T14:34:09.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:08 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 24790/194841 objects degraded (12.723%), 7 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-08-19T14:34:09.084 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:34:09.122 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:09.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:09.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:34:09.225 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:34:09.249 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:09.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:09.250 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:34:09.351 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:34:09.376 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:09.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:09.377 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:34:09.428 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:34:09.442 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 36 16 72706 72690 126.188 90.375 0.0105975 0.00791994 2024-08-19T14:34:09.477 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:34:09.500 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:09.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:09.501 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:34:09.602 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:34:09.627 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:09.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:09.628 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:34:09.730 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:34:09.754 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:09.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:09.755 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:34:09.856 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:34:09.879 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:09.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:09.880 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:34:09.897 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-08-19T14:34:09.926 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:34:09.937 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:09 smithi031 ceph-mon[16336]: pgmap v1441: 47 pgs: 9 active+undersized, 7 active+undersized+degraded, 31 active+clean; 4.0 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 72 MiB/s wr, 1.15k op/s; 25252/197595 objects degraded (12.780%) 2024-08-19T14:34:09.981 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:34:10.004 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:10.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:10.005 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:34:10.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:09 smithi032 ceph-mon[20004]: pgmap v1441: 47 pgs: 9 active+undersized, 7 active+undersized+degraded, 31 active+clean; 4.0 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 72 MiB/s wr, 1.15k op/s; 25252/197595 objects degraded (12.780%) 2024-08-19T14:34:10.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:09 smithi045 ceph-mon[20143]: pgmap v1441: 47 pgs: 9 active+undersized, 7 active+undersized+degraded, 31 active+clean; 4.0 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 72 MiB/s wr, 1.15k op/s; 25252/197595 objects degraded (12.780%) 2024-08-19T14:34:10.107 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:34:10.130 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:10.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:10.132 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:34:10.233 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:34:10.255 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:10.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:10.256 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:34:10.357 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:34:10.383 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:10.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:10.384 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:34:10.443 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 37 16 74108 74092 125.146 87.625 0.00718758 0.0079868 2024-08-19T14:34:10.485 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:34:10.509 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:10.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:10.510 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:34:10.612 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:34:10.616 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:34:10.643 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:10.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:10.645 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:34:10.668 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is unset 2024-08-19T14:34:10.745 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:34:10.759 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:10 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3858782659' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:34:10.759 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:10 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:34:10.770 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:10.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:10.771 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:34:10.872 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:34:10.897 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:10.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:10.898 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:34:10.999 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:34:11.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:10 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3858782659' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:34:11.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:10 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:34:11.022 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:11.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:11.023 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:34:11.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:10 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3858782659' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:34:11.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:10 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:34:11.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:10 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mon-a[16332]: 2024-08-19T14:34:10.903+0000 7f2ff503e640 -1 mon.a@0(leader).osd e615 definitely_dead 0 2024-08-19T14:34:11.123 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:34:11.150 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:11.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:11.151 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:34:11.203 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:34:11.252 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:34:11.277 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:11.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:11.278 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:34:11.378 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:34:11.401 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:11.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:11.402 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:34:11.443 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 38 16 75535 75519 124.199 89.1875 0.0051399 0.00804747 2024-08-19T14:34:11.503 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:34:11.528 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:11.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:11.530 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:34:11.630 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:34:11.657 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:11.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:11.658 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:34:11.692 INFO:teuthology.orchestra.run.smithi031.stderr:marked in osd.9. 2024-08-19T14:34:11.758 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:34:11.780 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:11.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:11.781 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:34:11.881 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:34:11.903 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:11.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:11.904 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:34:12.005 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:34:12.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:11 smithi032 ceph-mon[20004]: pgmap v1442: 47 pgs: 9 active+undersized, 7 active+undersized+degraded, 31 active+clean; 4.2 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 84 MiB/s wr, 1.34k op/s; 25703/208782 objects degraded (12.311%) 2024-08-19T14:34:12.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:11 smithi032 ceph-mon[20004]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:34:12.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:11 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:34:12.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:11 smithi032 ceph-mon[20004]: osdmap e615: 12 total, 11 up, 9 in 2024-08-19T14:34:12.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:11 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:34:12.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:11 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:34:12.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:11 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/925697886' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["9"]}]: dispatch 2024-08-19T14:34:12.027 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:12.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:12.028 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:34:12.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:11 smithi045 ceph-mon[20143]: pgmap v1442: 47 pgs: 9 active+undersized, 7 active+undersized+degraded, 31 active+clean; 4.2 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 84 MiB/s wr, 1.34k op/s; 25703/208782 objects degraded (12.311%) 2024-08-19T14:34:12.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:11 smithi045 ceph-mon[20143]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:34:12.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:11 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:34:12.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:11 smithi045 ceph-mon[20143]: osdmap e615: 12 total, 11 up, 9 in 2024-08-19T14:34:12.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:11 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:34:12.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:11 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:34:12.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:11 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/925697886' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["9"]}]: dispatch 2024-08-19T14:34:12.066 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:11 smithi031 ceph-mon[16336]: pgmap v1442: 47 pgs: 9 active+undersized, 7 active+undersized+degraded, 31 active+clean; 4.2 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 84 MiB/s wr, 1.34k op/s; 25703/208782 objects degraded (12.311%) 2024-08-19T14:34:12.066 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:11 smithi031 ceph-mon[16336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:34:12.066 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:11 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:34:12.066 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:11 smithi031 ceph-mon[16336]: osdmap e615: 12 total, 11 up, 9 in 2024-08-19T14:34:12.066 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:11 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:34:12.066 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:11 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:34:12.066 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:11 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/925697886' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["9"]}]: dispatch 2024-08-19T14:34:12.129 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:34:12.150 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:12.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:12.151 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:34:12.252 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:34:12.274 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:12.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:12.275 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:34:12.376 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:34:12.398 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:12.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:12.399 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:34:12.443 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 39 16 75974 75958 121.718 27.4375 0.00318249 0.00805653 2024-08-19T14:34:12.445 INFO:tasks.thrashosds.thrasher:Added osd 9 2024-08-19T14:34:12.500 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:34:12.525 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:12.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:12.526 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:34:12.627 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:34:12.628 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:34:12.628 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:34:12.651 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:12.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:12.652 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:34:12.754 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:34:12.777 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:12.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:12.778 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:34:12.879 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:34:12.905 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:12.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:12.906 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:34:13.007 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:34:13.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:12 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/925697886' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["9"]}]': finished 2024-08-19T14:34:13.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:12 smithi032 ceph-mon[20004]: osdmap e616: 12 total, 11 up, 10 in 2024-08-19T14:34:13.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:13.030 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:13.030 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:34:13.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:12 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/925697886' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["9"]}]': finished 2024-08-19T14:34:13.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:12 smithi045 ceph-mon[20143]: osdmap e616: 12 total, 11 up, 10 in 2024-08-19T14:34:13.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:12 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/925697886' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["9"]}]': finished 2024-08-19T14:34:13.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:12 smithi031 ceph-mon[16336]: osdmap e616: 12 total, 11 up, 10 in 2024-08-19T14:34:13.131 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:34:13.152 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:13.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:13.153 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:34:13.254 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:34:13.279 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:13.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:13.280 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:34:13.381 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:34:13.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:13.407 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:13.407 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:34:13.443 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:2024-08-19T14:34:13.443792+0000 min lat: 0.00106356 max lat: 0.285252 avg lat: 0.00805653 2024-08-19T14:34:13.443 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-19T14:34:13.443 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 40 16 75974 75958 118.675 0 - 0.00805653 2024-08-19T14:34:13.508 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:34:13.530 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:13.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:13.532 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:34:13.537 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-08-19T14:34:13.632 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:34:13.654 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:13.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:13.655 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:34:13.756 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:34:13.773 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-19T14:34:13.783 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:13 smithi031 ceph-mon[16336]: pgmap v1445: 47 pgs: 9 active+undersized, 7 active+undersized+degraded, 31 active+clean; 4.5 GiB data, 21 GiB used, 783 GiB / 805 GiB avail; 98 MiB/s wr, 1.57k op/s; 27953/219543 objects degraded (12.732%) 2024-08-19T14:34:13.783 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:13 smithi031 ceph-mon[16336]: 1.f scrub starts 2024-08-19T14:34:13.783 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:13 smithi031 ceph-mon[16336]: 1.f scrub ok 2024-08-19T14:34:13.783 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:13 smithi031 ceph-mon[16336]: osdmap e617: 12 total, 11 up, 10 in 2024-08-19T14:34:13.783 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:13 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 27953/219543 objects degraded (12.732%), 7 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-08-19T14:34:13.787 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:13.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:13.788 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:34:13.807 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:34:13.807 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:34:13.822 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:34:13.824 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:34:13.824 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:34:13.824 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-08-19T14:34:13.841 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:34:13.841 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:34:13.858 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:34:13.859 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:34:13.875 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:34:13.876 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:34:13.889 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:34:13.893 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:34:13.893 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:34:13.910 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:13.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:13.911 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:34:13.912 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:34:13.912 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:34:13.945 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:13 smithi032 ceph-mon[20004]: pgmap v1445: 47 pgs: 9 active+undersized, 7 active+undersized+degraded, 31 active+clean; 4.5 GiB data, 21 GiB used, 783 GiB / 805 GiB avail; 98 MiB/s wr, 1.57k op/s; 27953/219543 objects degraded (12.732%) 2024-08-19T14:34:13.945 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:13 smithi032 ceph-mon[20004]: 1.f scrub starts 2024-08-19T14:34:13.945 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:13 smithi032 ceph-mon[20004]: 1.f scrub ok 2024-08-19T14:34:13.945 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:13 smithi032 ceph-mon[20004]: osdmap e617: 12 total, 11 up, 10 in 2024-08-19T14:34:13.945 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:13 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 27953/219543 objects degraded (12.732%), 7 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-08-19T14:34:13.950 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:34:13.950 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:34:13.981 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:34:13.981 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:34:14.011 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:34:14.012 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:34:14.012 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:34:14.034 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:14.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:14.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:34:14.044 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:34:14.045 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:34:14.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:13 smithi045 ceph-mon[20143]: pgmap v1445: 47 pgs: 9 active+undersized, 7 active+undersized+degraded, 31 active+clean; 4.5 GiB data, 21 GiB used, 783 GiB / 805 GiB avail; 98 MiB/s wr, 1.57k op/s; 27953/219543 objects degraded (12.732%) 2024-08-19T14:34:14.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:13 smithi045 ceph-mon[20143]: 1.f scrub starts 2024-08-19T14:34:14.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:13 smithi045 ceph-mon[20143]: 1.f scrub ok 2024-08-19T14:34:14.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:13 smithi045 ceph-mon[20143]: osdmap e617: 12 total, 11 up, 10 in 2024-08-19T14:34:14.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:13 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 27953/219543 objects degraded (12.732%), 7 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-08-19T14:34:14.136 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:34:14.167 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:14.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:14.168 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:34:14.269 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:34:14.294 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:14.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:14.296 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:34:14.397 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:34:14.402 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:34:14.402 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-08-19T14:34:14.402 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:14.402 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.16962.0:76431 7.4 e618/600 7:2229f91a:::benchmark_data_smithi045_115842_object76430:head v 618'4852, mlcod=618'4850)", 2024-08-19T14:34:14.402 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:34:14.400260+0000", 2024-08-19T14:34:14.402 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.00060434900000000001, 2024-08-19T14:34:14.402 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00062522399999999996, 2024-08-19T14:34:14.402 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:34:14.402 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:34:14.403 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-08-19T14:34:14.403 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:34:14.403 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:14.403 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:34:14.403 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:34:14.400260+0000", 2024-08-19T14:34:14.403 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:34:14.403 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:14.403 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:14.403 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:34:14.403 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:34:14.400260+0000", 2024-08-19T14:34:14.403 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:34:14.403 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:14.404 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:14.404 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:34:14.404 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:34:14.400260+0000", 2024-08-19T14:34:14.404 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7599999999999996e-07 2024-08-19T14:34:14.404 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:14.404 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:14.404 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:34:14.404 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:34:14.400306+0000", 2024-08-19T14:34:14.404 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5543000000000001e-05 2024-08-19T14:34:14.404 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:14.404 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:14.404 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:34:14.405 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:34:14.400319+0000", 2024-08-19T14:34:14.405 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2928999999999999e-05 2024-08-19T14:34:14.405 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:14.405 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:14.405 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:34:14.405 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:34:14.400326+0000", 2024-08-19T14:34:14.405 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1520000000000003e-06 2024-08-19T14:34:14.405 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:14.405 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:14.405 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:34:14.405 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:34:14.400382+0000", 2024-08-19T14:34:14.405 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5726000000000001e-05 2024-08-19T14:34:14.406 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:14.406 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:14.406 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:34:14.406 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:34:14.400402+0000", 2024-08-19T14:34:14.406 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0690999999999999e-05 2024-08-19T14:34:14.406 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:34:14.406 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:34:14.406 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:34:14.406 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:34:14.406 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-08-19T14:34:14.406 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 1 2024-08-19T14:34:14.406 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:34:14.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:14.435 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:14.435 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:34:14.444 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 41 16 76462 76446 116.523 15.25 0.0047467 0.00857486 2024-08-19T14:34:14.535 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:34:14.573 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:14.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:14.574 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:34:14.675 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:34:14.713 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:14.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:14.714 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:34:14.815 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:34:14.838 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:14.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:14.840 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:34:14.849 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-19T14:34:14.849 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-19T14:34:14.864 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:34:14.940 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:34:14.974 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:14 smithi032 ceph-mon[20004]: osdmap e618: 12 total, 11 up, 10 in 2024-08-19T14:34:14.975 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:14 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2749048404' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:34:14.975 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:14 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:34:14.975 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:14 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:34:14.975 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:14 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:34:14.975 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:14 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:34:14.975 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:14.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:14.977 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:34:15.013 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:14 smithi031 ceph-mon[16336]: osdmap e618: 12 total, 11 up, 10 in 2024-08-19T14:34:15.013 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:14 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2749048404' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:34:15.014 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:14 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:34:15.014 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:14 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:34:15.014 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:14 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:34:15.014 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:14 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:34:15.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:14 smithi045 ceph-mon[20143]: osdmap e618: 12 total, 11 up, 10 in 2024-08-19T14:34:15.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:14 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2749048404' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:34:15.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:14 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:34:15.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:14 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:34:15.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:14 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:34:15.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:14 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:34:15.077 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-08-19T14:34:15.078 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:34:15.107 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:15.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:15.108 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:34:15.209 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:34:15.263 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:15.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:15.264 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:34:15.365 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:34:15.369 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:34:15.394 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:15.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:15.395 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:34:15.444 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 42 16 77360 77344 115.085 56.125 0.00678608 0.00868238 2024-08-19T14:34:15.496 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:34:15.528 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:15.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:15.529 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:34:15.630 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:34:15.660 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:15.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:15.661 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:34:15.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:15 smithi032 ceph-mon[20004]: osdmap e619: 12 total, 11 up, 10 in 2024-08-19T14:34:15.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:15 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:34:15.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:15 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:34:15.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:15 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:34:15.762 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:34:15.793 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:15.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:15.795 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:34:15.896 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:34:15.919 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:15.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:15.921 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:34:15.942 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:34:15.942 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-08-19T14:34:15.942 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-08-19T14:34:15.942 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-08-19T14:34:15.942 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:34:16.021 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:34:16.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:15 smithi045 ceph-mon[20143]: osdmap e619: 12 total, 11 up, 10 in 2024-08-19T14:34:16.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:15 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:34:16.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:15 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:34:16.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:15 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:34:16.055 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:16.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:16.056 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:34:16.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:15 smithi031 ceph-mon[16336]: osdmap e619: 12 total, 11 up, 10 in 2024-08-19T14:34:16.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:15 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:34:16.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:15 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:34:16.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:15 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:34:16.157 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:34:16.187 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:16.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:16.189 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:34:16.289 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:34:16.318 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:16.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:16.319 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:34:16.420 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:34:16.444 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 43 16 78378 78362 113.888 63.625 0.00455664 0.00877359 2024-08-19T14:34:16.448 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:16.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:16.449 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:34:16.550 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:34:16.582 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:16.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:16.583 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:34:16.617 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-08-19T14:34:16.683 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:34:16.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:16.720 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:16.720 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:34:16.742 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is set 2024-08-19T14:34:16.821 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:34:16.848 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:16.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:16.849 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:34:16.850 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:34:16.945 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:34:16.950 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:34:16.979 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:16.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:16.980 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:34:17.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:16 smithi032 ceph-mon[20004]: osdmap e620: 12 total, 11 up, 10 in 2024-08-19T14:34:17.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:16 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3965062329' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:34:17.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:16 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:34:17.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:16 smithi045 ceph-mon[20143]: osdmap e620: 12 total, 11 up, 10 in 2024-08-19T14:34:17.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:16 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3965062329' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:34:17.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:16 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:34:17.081 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:34:17.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:17.112 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:17.113 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:34:17.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:16 smithi031 ceph-mon[16336]: osdmap e620: 12 total, 11 up, 10 in 2024-08-19T14:34:17.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:16 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3965062329' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:34:17.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:16 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:34:17.213 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:34:17.240 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:17.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:17.241 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:34:17.343 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:34:17.366 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:17.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:17.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:34:17.444 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 44 16 79395 79379 112.744 63.5625 0.00781141 0.00886417 2024-08-19T14:34:17.445 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 7, 8, 2, 11, 9] out_osds: [10, 6] dead_osds: [2] live_osds: [0, 1, 3, 4, 5, 6, 7, 8, 9, 11, 10] 2024-08-19T14:34:17.446 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-19T14:34:17.446 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-08-19T14:34:17.446 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd dump -f json-pretty 2024-08-19T14:34:17.468 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:34:17.471 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:34:17.471 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-08-19T14:34:17.471 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-08-19T14:34:17.472 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-08-19T14:34:17.472 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.472 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:15526 5.1 5:81aa3e11:::benchmark_data_smithi045_85309_object15525:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e426)", 2024-08-19T14:34:17.472 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:24:52.832831+0000", 2024-08-19T14:34:17.472 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 564.62958267299996, 2024-08-19T14:34:17.472 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.265005194, 2024-08-19T14:34:17.472 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:34:17.472 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:34:17.472 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:34:17.472 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:34:17.472 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:34:17.472 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:34:17.472 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 15526 2024-08-19T14:34:17.473 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.473 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:34:17.473 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.473 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:34:17.473 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.832831+0000", 2024-08-19T14:34:17.473 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:34:17.473 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.473 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.473 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:34:17.473 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.832831+0000", 2024-08-19T14:34:17.473 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:34:17.473 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.473 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.473 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:34:17.473 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.832832+0000", 2024-08-19T14:34:17.474 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.499e-06 2024-08-19T14:34:17.474 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.474 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.474 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:34:17.474 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.832894+0000", 2024-08-19T14:34:17.474 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1909999999999995e-05 2024-08-19T14:34:17.474 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.474 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.474 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:34:17.474 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.832910+0000", 2024-08-19T14:34:17.474 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5906e-05 2024-08-19T14:34:17.474 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.475 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.475 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:34:17.475 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.832918+0000", 2024-08-19T14:34:17.475 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2400000000000001e-06 2024-08-19T14:34:17.475 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.475 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.475 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:34:17.475 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.832964+0000", 2024-08-19T14:34:17.475 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6697999999999999e-05 2024-08-19T14:34:17.475 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.475 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.475 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:34:17.475 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.833093+0000", 2024-08-19T14:34:17.475 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012924599999999999 2024-08-19T14:34:17.476 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.476 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.476 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 3,7", 2024-08-19T14:34:17.476 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.833228+0000", 2024-08-19T14:34:17.476 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000134621 2024-08-19T14:34:17.476 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.476 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.476 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:34:17.476 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.834341+0000", 2024-08-19T14:34:17.476 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001112701 2024-08-19T14:34:17.476 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.476 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.476 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:34:17.476 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.835208+0000", 2024-08-19T14:34:17.477 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00086687000000000005 2024-08-19T14:34:17.477 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.477 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.477 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:34:17.477 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:53.097747+0000", 2024-08-19T14:34:17.477 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.26253938399999999 2024-08-19T14:34:17.477 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.477 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.477 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:34:17.477 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:53.097796+0000", 2024-08-19T14:34:17.477 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8965999999999998e-05 2024-08-19T14:34:17.477 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.477 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.477 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:34:17.478 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:53.097836+0000", 2024-08-19T14:34:17.478 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0153e-05 2024-08-19T14:34:17.478 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:34:17.478 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:34:17.478 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:34:17.478 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.478 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.478 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:15538 5.1 5:88fc82fa:::benchmark_data_smithi045_85309_object15537:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e426)", 2024-08-19T14:34:17.478 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:24:52.836081+0000", 2024-08-19T14:34:17.478 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 564.62633284699996, 2024-08-19T14:34:17.478 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.26185454899999999, 2024-08-19T14:34:17.478 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:34:17.479 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:34:17.479 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:34:17.479 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:34:17.479 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:34:17.483 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:34:17.483 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 15538 2024-08-19T14:34:17.483 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.483 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:34:17.483 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.483 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:34:17.483 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.836081+0000", 2024-08-19T14:34:17.483 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:34:17.484 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.484 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.484 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:34:17.484 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.836081+0000", 2024-08-19T14:34:17.484 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:34:17.484 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.484 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.484 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:34:17.484 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.836082+0000", 2024-08-19T14:34:17.484 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4309999999999999e-06 2024-08-19T14:34:17.484 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.484 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.485 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:34:17.485 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.836111+0000", 2024-08-19T14:34:17.485 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8274000000000001e-05 2024-08-19T14:34:17.485 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.485 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.485 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:34:17.485 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.836127+0000", 2024-08-19T14:34:17.485 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6427999999999999e-05 2024-08-19T14:34:17.485 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.485 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.485 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:34:17.486 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.836135+0000", 2024-08-19T14:34:17.486 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.4589999999999998e-06 2024-08-19T14:34:17.486 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.486 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.486 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:34:17.486 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.836291+0000", 2024-08-19T14:34:17.486 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00015594299999999999 2024-08-19T14:34:17.486 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.486 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.486 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:34:17.486 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.836589+0000", 2024-08-19T14:34:17.486 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00029754899999999998 2024-08-19T14:34:17.486 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.487 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.487 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 3,7", 2024-08-19T14:34:17.487 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.836726+0000", 2024-08-19T14:34:17.487 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000136737 2024-08-19T14:34:17.487 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.487 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.487 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:34:17.487 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.838701+0000", 2024-08-19T14:34:17.487 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0019752480000000002 2024-08-19T14:34:17.487 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.487 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.487 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:34:17.487 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.844308+0000", 2024-08-19T14:34:17.488 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0056073670000000003 2024-08-19T14:34:17.488 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.488 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.488 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:34:17.488 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:53.097873+0000", 2024-08-19T14:34:17.488 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.25356439400000003 2024-08-19T14:34:17.488 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.488 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.488 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:34:17.488 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:53.097909+0000", 2024-08-19T14:34:17.488 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6303000000000001e-05 2024-08-19T14:34:17.488 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.489 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.489 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:34:17.489 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:53.097935+0000", 2024-08-19T14:34:17.489 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6415999999999999e-05 2024-08-19T14:34:17.489 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:34:17.489 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:34:17.489 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:34:17.489 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.489 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.489 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:15554 5.1 5:8ece79d1:::benchmark_data_smithi045_85309_object15553:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e426)", 2024-08-19T14:34:17.489 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:24:52.841409+0000", 2024-08-19T14:34:17.489 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 564.62100513300004, 2024-08-19T14:34:17.489 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.25662066100000003, 2024-08-19T14:34:17.490 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:34:17.490 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:34:17.490 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:34:17.490 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:34:17.490 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:34:17.490 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:34:17.490 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 15554 2024-08-19T14:34:17.490 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.490 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:34:17.490 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.490 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:34:17.490 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.841409+0000", 2024-08-19T14:34:17.490 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:34:17.491 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.491 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.491 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:34:17.491 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.841409+0000", 2024-08-19T14:34:17.491 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:34:17.491 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.491 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.491 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:34:17.491 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.841410+0000", 2024-08-19T14:34:17.491 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3239999999999999e-06 2024-08-19T14:34:17.491 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.491 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.491 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:34:17.492 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.841462+0000", 2024-08-19T14:34:17.492 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1799000000000003e-05 2024-08-19T14:34:17.492 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.492 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.492 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:34:17.492 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.841476+0000", 2024-08-19T14:34:17.492 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4608e-05 2024-08-19T14:34:17.492 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.492 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.492 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:34:17.492 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.841485+0000", 2024-08-19T14:34:17.492 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6090000000000001e-06 2024-08-19T14:34:17.492 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.493 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.493 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:34:17.493 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.841546+0000", 2024-08-19T14:34:17.493 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1168999999999997e-05 2024-08-19T14:34:17.493 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.493 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.493 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:34:17.493 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.841682+0000", 2024-08-19T14:34:17.493 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013579900000000001 2024-08-19T14:34:17.493 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.493 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.493 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 3,7", 2024-08-19T14:34:17.493 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.841765+0000", 2024-08-19T14:34:17.494 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.3494999999999999e-05 2024-08-19T14:34:17.494 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.494 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.494 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:34:17.494 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.842729+0000", 2024-08-19T14:34:17.494 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00096402900000000004 2024-08-19T14:34:17.494 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.494 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.494 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:34:17.494 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.846426+0000", 2024-08-19T14:34:17.494 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0036964699999999999 2024-08-19T14:34:17.494 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.494 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.495 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:34:17.495 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:53.097964+0000", 2024-08-19T14:34:17.495 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.25153829300000002 2024-08-19T14:34:17.495 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.495 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.495 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:34:17.495 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:53.098003+0000", 2024-08-19T14:34:17.495 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9193999999999998e-05 2024-08-19T14:34:17.495 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.495 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.495 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:34:17.495 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:53.098029+0000", 2024-08-19T14:34:17.495 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5871999999999999e-05 2024-08-19T14:34:17.496 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:34:17.496 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:34:17.496 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:34:17.496 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.496 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.496 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:15557 5.1 5:894b28fd:::benchmark_data_smithi045_85309_object15556:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e426)", 2024-08-19T14:34:17.496 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:24:52.843093+0000", 2024-08-19T14:34:17.496 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 564.61932022799999, 2024-08-19T14:34:17.496 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.25539605599999998, 2024-08-19T14:34:17.496 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:34:17.496 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:34:17.496 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:34:17.496 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:34:17.497 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:34:17.497 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:34:17.497 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 15557 2024-08-19T14:34:17.497 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.497 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:34:17.497 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.497 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:34:17.497 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.843093+0000", 2024-08-19T14:34:17.497 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:34:17.497 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.497 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.497 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:34:17.498 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.843093+0000", 2024-08-19T14:34:17.498 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:34:17.498 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.498 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.498 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:34:17.498 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.843095+0000", 2024-08-19T14:34:17.498 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.246e-06 2024-08-19T14:34:17.498 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.498 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.498 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:34:17.498 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.843147+0000", 2024-08-19T14:34:17.498 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2379000000000001e-05 2024-08-19T14:34:17.499 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.499 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.499 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:34:17.499 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.843158+0000", 2024-08-19T14:34:17.499 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1238e-05 2024-08-19T14:34:17.499 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.499 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.499 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:34:17.499 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.843165+0000", 2024-08-19T14:34:17.499 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1609999999999997e-06 2024-08-19T14:34:17.499 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.499 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.499 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:34:17.500 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.843217+0000", 2024-08-19T14:34:17.500 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1898999999999999e-05 2024-08-19T14:34:17.500 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.500 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.500 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:34:17.500 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.843352+0000", 2024-08-19T14:34:17.500 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000135017 2024-08-19T14:34:17.500 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.500 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.500 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 3,7", 2024-08-19T14:34:17.500 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.843506+0000", 2024-08-19T14:34:17.500 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00015404200000000001 2024-08-19T14:34:17.500 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.501 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.501 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:34:17.501 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.844660+0000", 2024-08-19T14:34:17.501 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001153987 2024-08-19T14:34:17.501 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.501 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.501 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:34:17.501 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.846459+0000", 2024-08-19T14:34:17.501 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0017983700000000001 2024-08-19T14:34:17.501 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.501 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.501 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:34:17.501 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:53.098357+0000", 2024-08-19T14:34:17.502 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.25189793100000002 2024-08-19T14:34:17.502 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.502 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.502 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:34:17.502 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:53.098411+0000", 2024-08-19T14:34:17.502 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4796999999999997e-05 2024-08-19T14:34:17.502 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.502 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.502 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:34:17.502 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:53.098489+0000", 2024-08-19T14:34:17.502 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7989000000000002e-05 2024-08-19T14:34:17.502 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:34:17.502 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:34:17.503 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:34:17.503 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.503 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.503 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:15580 5.1 5:8e704e69:::benchmark_data_smithi045_85309_object15579:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e426)", 2024-08-19T14:34:17.503 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:24:52.850956+0000", 2024-08-19T14:34:17.503 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 564.61145759500005, 2024-08-19T14:34:17.503 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.24772282200000001, 2024-08-19T14:34:17.503 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:34:17.503 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:34:17.503 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:34:17.503 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:34:17.503 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:34:17.503 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:34:17.504 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 15580 2024-08-19T14:34:17.504 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.504 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:34:17.504 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.504 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:34:17.504 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.850956+0000", 2024-08-19T14:34:17.504 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:34:17.504 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.504 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.504 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:34:17.504 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.850956+0000", 2024-08-19T14:34:17.504 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:34:17.505 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.505 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.505 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:34:17.505 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.850959+0000", 2024-08-19T14:34:17.505 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8720000000000002e-06 2024-08-19T14:34:17.505 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.505 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.505 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:34:17.506 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.850994+0000", 2024-08-19T14:34:17.506 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5323e-05 2024-08-19T14:34:17.506 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.506 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.506 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:34:17.506 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.851010+0000", 2024-08-19T14:34:17.506 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5885000000000001e-05 2024-08-19T14:34:17.506 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.506 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.506 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:34:17.506 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.851018+0000", 2024-08-19T14:34:17.506 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1000000000000004e-06 2024-08-19T14:34:17.507 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.507 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.507 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:34:17.507 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.851085+0000", 2024-08-19T14:34:17.507 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6692000000000005e-05 2024-08-19T14:34:17.507 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.507 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.507 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:34:17.507 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.851218+0000", 2024-08-19T14:34:17.507 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013344400000000001 2024-08-19T14:34:17.507 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.507 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.507 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 3,7", 2024-08-19T14:34:17.508 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.851382+0000", 2024-08-19T14:34:17.508 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000163907 2024-08-19T14:34:17.508 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.508 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.508 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:34:17.508 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.852814+0000", 2024-08-19T14:34:17.508 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001431779 2024-08-19T14:34:17.508 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.508 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.508 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:34:17.508 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:52.853191+0000", 2024-08-19T14:34:17.508 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00037734900000000002 2024-08-19T14:34:17.508 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.508 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.509 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:34:17.509 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:53.098547+0000", 2024-08-19T14:34:17.509 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.24535599799999999 2024-08-19T14:34:17.509 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.509 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.509 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:34:17.509 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:53.098640+0000", 2024-08-19T14:34:17.509 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.2829000000000002e-05 2024-08-19T14:34:17.509 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.509 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.509 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:34:17.509 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:53.098679+0000", 2024-08-19T14:34:17.510 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8643999999999998e-05 2024-08-19T14:34:17.510 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:34:17.510 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:34:17.510 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:34:17.510 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.510 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.510 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:237733 5.11 5:8ce96d21:::benchmark_data_smithi045_85309_object117480:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e495)", 2024-08-19T14:34:17.510 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:28:07.104832+0000", 2024-08-19T14:34:17.510 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 370.35758196699999, 2024-08-19T14:34:17.511 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.249395372, 2024-08-19T14:34:17.511 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:34:17.511 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:34:17.511 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:34:17.511 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:34:17.511 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:34:17.511 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:34:17.511 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 237733 2024-08-19T14:34:17.511 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.511 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:34:17.511 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.511 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:34:17.511 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:28:07.104832+0000", 2024-08-19T14:34:17.512 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:34:17.512 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.512 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.512 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:34:17.512 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:28:07.104832+0000", 2024-08-19T14:34:17.512 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:34:17.512 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.512 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.512 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:34:17.512 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:28:07.104833+0000", 2024-08-19T14:34:17.512 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.505e-06 2024-08-19T14:34:17.512 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.512 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.513 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:34:17.513 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:28:07.104839+0000", 2024-08-19T14:34:17.513 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4689999999999996e-06 2024-08-19T14:34:17.513 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.513 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.513 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:34:17.513 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:28:07.104879+0000", 2024-08-19T14:34:17.513 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0061000000000002e-05 2024-08-19T14:34:17.513 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.513 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.513 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:34:17.513 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:28:07.104888+0000", 2024-08-19T14:34:17.513 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.9479999999999997e-06 2024-08-19T14:34:17.513 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.514 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.514 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:34:17.514 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:28:07.105668+0000", 2024-08-19T14:34:17.514 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00078030099999999995 2024-08-19T14:34:17.514 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.514 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.514 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:34:17.514 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:28:07.105681+0000", 2024-08-19T14:34:17.514 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.325e-05 2024-08-19T14:34:17.514 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.514 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.514 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:34:17.514 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:28:07.311371+0000", 2024-08-19T14:34:17.515 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20568988599999999 2024-08-19T14:34:17.515 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.515 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.515 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:34:17.515 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:28:07.311556+0000", 2024-08-19T14:34:17.515 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000184847 2024-08-19T14:34:17.515 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.515 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.515 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0,7", 2024-08-19T14:34:17.515 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:28:07.312176+0000", 2024-08-19T14:34:17.515 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00062027000000000002 2024-08-19T14:34:17.515 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.515 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.515 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:34:17.516 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:28:07.316296+0000", 2024-08-19T14:34:17.516 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.004120014 2024-08-19T14:34:17.516 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.516 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.516 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:34:17.516 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:28:07.323513+0000", 2024-08-19T14:34:17.516 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0072170070000000001 2024-08-19T14:34:17.516 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.516 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.516 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:34:17.516 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:28:07.353960+0000", 2024-08-19T14:34:17.516 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.030446371 2024-08-19T14:34:17.516 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.517 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.517 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:34:17.517 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:28:07.354014+0000", 2024-08-19T14:34:17.517 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4528999999999999e-05 2024-08-19T14:34:17.517 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.517 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.517 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:34:17.517 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:28:07.354227+0000", 2024-08-19T14:34:17.517 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000212914 2024-08-19T14:34:17.517 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:34:17.517 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:34:17.517 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:34:17.517 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.517 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.518 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26191.0:76311 6.2 6.4a652072 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e524)", 2024-08-19T14:34:17.518 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:29:15.416180+0000", 2024-08-19T14:34:17.518 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 302.04623344999999, 2024-08-19T14:34:17.518 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.98615755400000005, 2024-08-19T14:34:17.518 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:34:17.518 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:34:17.518 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:34:17.518 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:34:17.518 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26191", 2024-08-19T14:34:17.518 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:34:17.518 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 76311 2024-08-19T14:34:17.518 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.518 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:34:17.519 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.519 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:34:17.519 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:15.416180+0000", 2024-08-19T14:34:17.519 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:34:17.519 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.519 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.519 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:34:17.519 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:15.416180+0000", 2024-08-19T14:34:17.519 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:34:17.519 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.519 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.519 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:34:17.519 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:15.416182+0000", 2024-08-19T14:34:17.520 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.341e-06 2024-08-19T14:34:17.520 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.520 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.520 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:34:17.520 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:15.416221+0000", 2024-08-19T14:34:17.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:17.522 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9122999999999998e-05 2024-08-19T14:34:17.522 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.522 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.523 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:34:17.523 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:15.416231+0000", 2024-08-19T14:34:17.523 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.076e-05 2024-08-19T14:34:17.523 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.523 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.523 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:34:17.523 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:15.416238+0000", 2024-08-19T14:34:17.523 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1160000000000004e-06 2024-08-19T14:34:17.523 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.523 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.523 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:34:17.523 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:15.416292+0000", 2024-08-19T14:34:17.523 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4039000000000002e-05 2024-08-19T14:34:17.524 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.524 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.524 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:34:17.524 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:15.416304+0000", 2024-08-19T14:34:17.524 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2388000000000001e-05 2024-08-19T14:34:17.524 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.524 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.524 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:34:17.524 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:16.402265+0000", 2024-08-19T14:34:17.524 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.98596146799999995 2024-08-19T14:34:17.524 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.524 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.524 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:34:17.524 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:16.402338+0000", 2024-08-19T14:34:17.525 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2318999999999997e-05 2024-08-19T14:34:17.525 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:34:17.525 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:34:17.525 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:34:17.525 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.525 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.525 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26191.0:76321 6.2 6.2bd467a2 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e524)", 2024-08-19T14:34:17.525 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:29:15.422552+0000", 2024-08-19T14:34:17.525 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 302.03986199000002, 2024-08-19T14:34:17.525 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.98004182100000004, 2024-08-19T14:34:17.525 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:34:17.525 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:34:17.525 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:34:17.526 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:34:17.526 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26191", 2024-08-19T14:34:17.526 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:34:17.526 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 76321 2024-08-19T14:34:17.526 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.526 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:34:17.526 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.526 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:34:17.526 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:15.422552+0000", 2024-08-19T14:34:17.526 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:34:17.526 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.526 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.526 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:34:17.526 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:15.422552+0000", 2024-08-19T14:34:17.527 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:34:17.527 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.527 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.527 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:34:17.527 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:15.422553+0000", 2024-08-19T14:34:17.527 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7090000000000001e-06 2024-08-19T14:34:17.527 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.527 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.527 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:34:17.527 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:15.422588+0000", 2024-08-19T14:34:17.527 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.4136e-05 2024-08-19T14:34:17.527 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.527 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.528 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:34:17.528 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:15.422598+0000", 2024-08-19T14:34:17.528 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0560999999999999e-05 2024-08-19T14:34:17.528 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.528 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.528 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:34:17.528 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:15.422603+0000", 2024-08-19T14:34:17.528 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3879999999999999e-06 2024-08-19T14:34:17.528 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.528 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.528 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:34:17.528 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:15.422653+0000", 2024-08-19T14:34:17.528 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9595000000000002e-05 2024-08-19T14:34:17.529 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.529 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.529 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:34:17.529 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:15.422662+0000", 2024-08-19T14:34:17.529 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.0899999999999994e-06 2024-08-19T14:34:17.529 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.529 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.529 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:34:17.529 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:16.402581+0000", 2024-08-19T14:34:17.529 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.97991880200000003 2024-08-19T14:34:17.529 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.529 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.529 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:34:17.530 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:16.402593+0000", 2024-08-19T14:34:17.530 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.254e-05 2024-08-19T14:34:17.530 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:34:17.530 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:34:17.530 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:34:17.530 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.530 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.530 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26191.0:76326 6.e 6.dc7cf32e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e524)", 2024-08-19T14:34:17.530 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:29:15.428428+0000", 2024-08-19T14:34:17.530 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 302.033985218, 2024-08-19T14:34:17.530 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.97310281799999998, 2024-08-19T14:34:17.530 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:34:17.530 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:34:17.531 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:34:17.531 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:34:17.531 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26191", 2024-08-19T14:34:17.531 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:34:17.531 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 76326 2024-08-19T14:34:17.531 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.531 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:34:17.531 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.531 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:34:17.531 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:15.428428+0000", 2024-08-19T14:34:17.531 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:34:17.531 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.532 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.532 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:34:17.532 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:15.428428+0000", 2024-08-19T14:34:17.532 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:34:17.532 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.532 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.532 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:34:17.532 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:15.428429+0000", 2024-08-19T14:34:17.532 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.7999999999999993e-07 2024-08-19T14:34:17.532 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.532 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.532 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:34:17.532 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:15.428462+0000", 2024-08-19T14:34:17.533 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2212999999999998e-05 2024-08-19T14:34:17.533 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.533 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.533 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:34:17.533 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:15.428472+0000", 2024-08-19T14:34:17.533 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0208e-05 2024-08-19T14:34:17.533 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.533 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.533 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:34:17.533 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:15.428477+0000", 2024-08-19T14:34:17.533 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1919999999999996e-06 2024-08-19T14:34:17.533 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.533 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.534 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:34:17.534 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:15.428524+0000", 2024-08-19T14:34:17.534 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6946000000000001e-05 2024-08-19T14:34:17.534 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.534 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.534 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:34:17.534 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:15.428532+0000", 2024-08-19T14:34:17.534 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.0879999999999995e-06 2024-08-19T14:34:17.534 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.534 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.534 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:34:17.534 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:16.401497+0000", 2024-08-19T14:34:17.534 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.97296510999999997 2024-08-19T14:34:17.535 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.535 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.535 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:34:17.535 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:16.401531+0000", 2024-08-19T14:34:17.535 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.4081000000000001e-05 2024-08-19T14:34:17.535 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:34:17.535 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:34:17.535 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:34:17.535 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.536 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.536 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26191.0:76326 6.e 6:74cf3e3b:::benchmark_data_smithi045_97937_object76325:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e526)", 2024-08-19T14:34:17.536 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:29:17.439806+0000", 2024-08-19T14:34:17.536 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 300.02260729199998, 2024-08-19T14:34:17.536 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.31982314000000001, 2024-08-19T14:34:17.536 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:34:17.536 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:34:17.536 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:34:17.536 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:34:17.536 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26191", 2024-08-19T14:34:17.536 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:34:17.536 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 76326 2024-08-19T14:34:17.536 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.537 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:34:17.537 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.537 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:34:17.537 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.439806+0000", 2024-08-19T14:34:17.537 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:34:17.537 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.537 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.537 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:34:17.537 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.439806+0000", 2024-08-19T14:34:17.537 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:34:17.537 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.537 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.537 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:34:17.537 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.439808+0000", 2024-08-19T14:34:17.538 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8899999999999999e-06 2024-08-19T14:34:17.538 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.538 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.538 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:34:17.538 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.439913+0000", 2024-08-19T14:34:17.538 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00010467300000000001 2024-08-19T14:34:17.538 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.538 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.538 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:34:17.538 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.439930+0000", 2024-08-19T14:34:17.538 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6891000000000001e-05 2024-08-19T14:34:17.538 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.538 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.539 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:34:17.539 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.439941+0000", 2024-08-19T14:34:17.539 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1482e-05 2024-08-19T14:34:17.539 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.539 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.539 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:34:17.539 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.442395+0000", 2024-08-19T14:34:17.539 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0024540460000000001 2024-08-19T14:34:17.539 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.539 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.539 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:34:17.539 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.442404+0000", 2024-08-19T14:34:17.539 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.337e-06 2024-08-19T14:34:17.540 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.540 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.540 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:34:17.540 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.727910+0000", 2024-08-19T14:34:17.540 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.28550632999999997 2024-08-19T14:34:17.540 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.540 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.540 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:34:17.540 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.728089+0000", 2024-08-19T14:34:17.540 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000178696 2024-08-19T14:34:17.540 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.540 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.540 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 1,2", 2024-08-19T14:34:17.541 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.728278+0000", 2024-08-19T14:34:17.541 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000189249 2024-08-19T14:34:17.541 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.541 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.541 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:34:17.541 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.736958+0000", 2024-08-19T14:34:17.541 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0086800530000000001 2024-08-19T14:34:17.541 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.541 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.541 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:34:17.541 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.736975+0000", 2024-08-19T14:34:17.541 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7269999999999999e-05 2024-08-19T14:34:17.541 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.542 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.542 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:34:17.542 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.756064+0000", 2024-08-19T14:34:17.542 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.019088357 2024-08-19T14:34:17.542 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.542 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.542 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:34:17.542 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.756105+0000", 2024-08-19T14:34:17.542 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1349e-05 2024-08-19T14:34:17.542 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.542 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.542 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:34:17.542 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.759629+0000", 2024-08-19T14:34:17.543 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.003524517 2024-08-19T14:34:17.543 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:34:17.543 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:34:17.543 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:34:17.543 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.543 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.543 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26191.0:76311 6.2 6:4e04a652:::benchmark_data_smithi045_97937_object76310:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e526)", 2024-08-19T14:34:17.543 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:29:17.440956+0000", 2024-08-19T14:34:17.543 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 300.02145767899998, 2024-08-19T14:34:17.543 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.30557601400000001, 2024-08-19T14:34:17.543 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:34:17.543 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:34:17.543 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:34:17.543 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:34:17.544 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26191", 2024-08-19T14:34:17.544 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:34:17.544 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 76311 2024-08-19T14:34:17.544 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.544 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:34:17.544 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.544 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:34:17.544 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.440956+0000", 2024-08-19T14:34:17.544 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:34:17.544 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.544 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.544 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:34:17.544 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.440956+0000", 2024-08-19T14:34:17.545 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:34:17.545 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.545 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.545 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:34:17.545 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.440957+0000", 2024-08-19T14:34:17.545 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0330000000000001e-06 2024-08-19T14:34:17.545 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.545 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.545 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:34:17.545 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.441017+0000", 2024-08-19T14:34:17.545 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0331000000000002e-05 2024-08-19T14:34:17.545 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.545 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.545 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:34:17.546 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.441031+0000", 2024-08-19T14:34:17.546 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4096e-05 2024-08-19T14:34:17.546 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.546 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.546 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:34:17.546 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.441042+0000", 2024-08-19T14:34:17.546 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0315e-05 2024-08-19T14:34:17.546 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.546 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.546 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:34:17.546 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.444349+0000", 2024-08-19T14:34:17.546 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0033070479999999999 2024-08-19T14:34:17.546 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.547 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.547 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:34:17.547 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.444361+0000", 2024-08-19T14:34:17.547 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2007e-05 2024-08-19T14:34:17.547 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.547 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.547 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:34:17.547 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.709039+0000", 2024-08-19T14:34:17.547 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.26467776300000001 2024-08-19T14:34:17.547 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.547 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.547 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:34:17.547 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.709150+0000", 2024-08-19T14:34:17.548 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000111773 2024-08-19T14:34:17.548 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.548 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.548 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,3", 2024-08-19T14:34:17.548 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.709263+0000", 2024-08-19T14:34:17.548 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000112893 2024-08-19T14:34:17.548 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.548 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.548 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:34:17.548 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.717202+0000", 2024-08-19T14:34:17.548 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0079389950000000008 2024-08-19T14:34:17.548 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.548 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.548 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:34:17.549 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.717300+0000", 2024-08-19T14:34:17.549 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.8022000000000004e-05 2024-08-19T14:34:17.549 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.549 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.549 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:34:17.549 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.742903+0000", 2024-08-19T14:34:17.549 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.025602771999999999 2024-08-19T14:34:17.549 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.549 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.549 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:34:17.549 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.742968+0000", 2024-08-19T14:34:17.549 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4818000000000002e-05 2024-08-19T14:34:17.549 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.549 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.550 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:34:17.550 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.746532+0000", 2024-08-19T14:34:17.550 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0035641480000000001 2024-08-19T14:34:17.550 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:34:17.550 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:34:17.550 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:34:17.550 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.550 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.550 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26191.0:76321 6.2 6:45e62bd4:::benchmark_data_smithi045_97937_object76320:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e526)", 2024-08-19T14:34:17.550 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:29:17.441060+0000", 2024-08-19T14:34:17.550 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 300.02135388699998, 2024-08-19T14:34:17.550 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.30905421399999999, 2024-08-19T14:34:17.550 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:34:17.551 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:34:17.551 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:34:17.551 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:34:17.551 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26191", 2024-08-19T14:34:17.551 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:34:17.551 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 76321 2024-08-19T14:34:17.551 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.551 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:34:17.552 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.552 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:34:17.552 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.441060+0000", 2024-08-19T14:34:17.552 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:34:17.552 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.552 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.552 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:34:17.552 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.441060+0000", 2024-08-19T14:34:17.552 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:34:17.552 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.552 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.552 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:34:17.552 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.441060+0000", 2024-08-19T14:34:17.552 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2900000000000004e-07 2024-08-19T14:34:17.553 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.553 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.553 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:34:17.553 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.441093+0000", 2024-08-19T14:34:17.553 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2753999999999998e-05 2024-08-19T14:34:17.553 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.553 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.553 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:34:17.553 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.441102+0000", 2024-08-19T14:34:17.553 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.1069999999999994e-06 2024-08-19T14:34:17.553 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.553 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.553 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:34:17.553 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.441109+0000", 2024-08-19T14:34:17.554 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4699999999999999e-06 2024-08-19T14:34:17.554 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.554 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.554 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:34:17.554 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.444384+0000", 2024-08-19T14:34:17.554 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.003274927 2024-08-19T14:34:17.554 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.554 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.554 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:34:17.554 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.444390+0000", 2024-08-19T14:34:17.554 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4799999999999998e-06 2024-08-19T14:34:17.554 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.555 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.555 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:34:17.555 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.714684+0000", 2024-08-19T14:34:17.555 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.27029436899999998 2024-08-19T14:34:17.555 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.555 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.555 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:34:17.555 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.714834+0000", 2024-08-19T14:34:17.555 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014928900000000001 2024-08-19T14:34:17.555 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.555 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.555 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,3", 2024-08-19T14:34:17.555 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.715028+0000", 2024-08-19T14:34:17.555 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00019421999999999999 2024-08-19T14:34:17.556 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.556 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.556 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:34:17.556 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.717424+0000", 2024-08-19T14:34:17.556 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0023956350000000001 2024-08-19T14:34:17.556 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.556 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.556 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:34:17.556 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.718122+0000", 2024-08-19T14:34:17.556 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00069822999999999999 2024-08-19T14:34:17.556 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.556 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.556 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:34:17.556 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.746605+0000", 2024-08-19T14:34:17.556 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.028483366999999999 2024-08-19T14:34:17.557 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.557 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.557 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:34:17.557 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.746677+0000", 2024-08-19T14:34:17.557 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1638999999999996e-05 2024-08-19T14:34:17.557 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.557 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.557 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:34:17.557 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.750114+0000", 2024-08-19T14:34:17.557 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0034371979999999998 2024-08-19T14:34:17.557 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:34:17.557 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:34:17.557 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:34:17.557 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.558 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.558 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26191.0:76305 6.f 6:fe56d33d:::benchmark_data_smithi045_97937_object76304:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e526)", 2024-08-19T14:34:17.558 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:29:17.446200+0000", 2024-08-19T14:34:17.558 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 300.01621364800002, 2024-08-19T14:34:17.558 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.26858252100000002, 2024-08-19T14:34:17.558 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:34:17.558 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:34:17.558 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:34:17.558 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:34:17.558 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26191", 2024-08-19T14:34:17.558 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:34:17.559 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 76305 2024-08-19T14:34:17.559 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.559 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:34:17.559 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.559 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:34:17.559 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.446200+0000", 2024-08-19T14:34:17.559 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:34:17.559 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.559 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.559 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:34:17.559 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.446200+0000", 2024-08-19T14:34:17.559 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:34:17.559 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.559 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.560 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:34:17.560 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.446202+0000", 2024-08-19T14:34:17.560 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9819999999999998e-06 2024-08-19T14:34:17.560 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.560 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.560 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:34:17.560 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.446270+0000", 2024-08-19T14:34:17.560 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8035999999999997e-05 2024-08-19T14:34:17.560 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.560 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.560 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:34:17.560 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.446290+0000", 2024-08-19T14:34:17.560 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9882999999999999e-05 2024-08-19T14:34:17.560 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.561 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.561 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:34:17.561 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.446303+0000", 2024-08-19T14:34:17.561 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3233e-05 2024-08-19T14:34:17.561 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.561 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.561 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:34:17.561 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.446901+0000", 2024-08-19T14:34:17.561 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00059783899999999997 2024-08-19T14:34:17.561 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.561 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.561 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:34:17.561 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.446912+0000", 2024-08-19T14:34:17.562 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0791000000000001e-05 2024-08-19T14:34:17.562 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.562 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.562 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:34:17.562 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.688663+0000", 2024-08-19T14:34:17.562 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.24175074299999999 2024-08-19T14:34:17.562 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.562 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.562 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:34:17.562 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.688828+0000", 2024-08-19T14:34:17.562 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00016598200000000001 2024-08-19T14:34:17.562 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.562 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.563 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,5", 2024-08-19T14:34:17.563 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.689014+0000", 2024-08-19T14:34:17.563 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000185704 2024-08-19T14:34:17.563 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.563 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.563 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:34:17.563 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.695917+0000", 2024-08-19T14:34:17.563 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.006903111 2024-08-19T14:34:17.563 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.563 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.563 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:34:17.563 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.695928+0000", 2024-08-19T14:34:17.563 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0241000000000001e-05 2024-08-19T14:34:17.564 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.564 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.564 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:34:17.564 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.711710+0000", 2024-08-19T14:34:17.564 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.015782348000000002 2024-08-19T14:34:17.564 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.564 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.564 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:34:17.564 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.711759+0000", 2024-08-19T14:34:17.564 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8758000000000003e-05 2024-08-19T14:34:17.564 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.564 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.564 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:34:17.564 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.714783+0000", 2024-08-19T14:34:17.565 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0030238700000000001 2024-08-19T14:34:17.565 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:34:17.565 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:34:17.565 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:34:17.565 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.565 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.565 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26191.0:76312 6.f 6:f7fbbdfd:::benchmark_data_smithi045_97937_object76311:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e526)", 2024-08-19T14:34:17.565 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:29:17.446323+0000", 2024-08-19T14:34:17.565 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 300.01609033800003, 2024-08-19T14:34:17.565 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.27083775100000002, 2024-08-19T14:34:17.565 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:34:17.565 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:34:17.565 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:34:17.566 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:34:17.566 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26191", 2024-08-19T14:34:17.566 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:34:17.566 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 76312 2024-08-19T14:34:17.566 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.566 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:34:17.566 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.566 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:34:17.566 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.446323+0000", 2024-08-19T14:34:17.566 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:34:17.566 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.566 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.566 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:34:17.567 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.446323+0000", 2024-08-19T14:34:17.567 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:34:17.567 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.567 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.567 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:34:17.567 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.446324+0000", 2024-08-19T14:34:17.567 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4899999999999995e-07 2024-08-19T14:34:17.567 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.567 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.567 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:34:17.567 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.446356+0000", 2024-08-19T14:34:17.567 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1934999999999998e-05 2024-08-19T14:34:17.567 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.567 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.568 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:34:17.568 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.446368+0000", 2024-08-19T14:34:17.568 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1874e-05 2024-08-19T14:34:17.568 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.568 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.568 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:34:17.568 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.446374+0000", 2024-08-19T14:34:17.568 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8880000000000002e-06 2024-08-19T14:34:17.568 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.568 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.568 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:34:17.568 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.446934+0000", 2024-08-19T14:34:17.568 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000560237 2024-08-19T14:34:17.569 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.569 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.569 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:34:17.569 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.446942+0000", 2024-08-19T14:34:17.569 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.4509999999999992e-06 2024-08-19T14:34:17.569 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.569 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.569 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:34:17.569 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.693544+0000", 2024-08-19T14:34:17.569 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.246602191 2024-08-19T14:34:17.569 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.569 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.569 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:34:17.570 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.693660+0000", 2024-08-19T14:34:17.570 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00011593 2024-08-19T14:34:17.570 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.570 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.570 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,5", 2024-08-19T14:34:17.570 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.693774+0000", 2024-08-19T14:34:17.570 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00011357999999999999 2024-08-19T14:34:17.570 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.570 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.570 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:34:17.570 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.696063+0000", 2024-08-19T14:34:17.571 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0022885800000000001 2024-08-19T14:34:17.571 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.571 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.571 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:34:17.571 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.696305+0000", 2024-08-19T14:34:17.571 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00024199200000000001 2024-08-19T14:34:17.571 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.571 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.571 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:34:17.571 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.714831+0000", 2024-08-19T14:34:17.571 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.018526885999999999 2024-08-19T14:34:17.571 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.571 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.571 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:34:17.572 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.714883+0000", 2024-08-19T14:34:17.572 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1433e-05 2024-08-19T14:34:17.572 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.572 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.572 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:34:17.572 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.717161+0000", 2024-08-19T14:34:17.572 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.002278225 2024-08-19T14:34:17.572 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:34:17.572 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:34:17.572 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:34:17.572 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.572 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.572 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26191.0:76378 6.e 6:78c77635:::benchmark_data_smithi045_97937_object76377:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e527)", 2024-08-19T14:34:17.572 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:29:17.871362+0000", 2024-08-19T14:34:17.573 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 299.59105145299998, 2024-08-19T14:34:17.573 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.45114312600000001, 2024-08-19T14:34:17.573 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:34:17.573 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:34:17.573 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:34:17.573 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:34:17.573 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26191", 2024-08-19T14:34:17.573 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:34:17.573 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 76378 2024-08-19T14:34:17.573 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.573 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:34:17.573 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.573 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:34:17.573 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.871362+0000", 2024-08-19T14:34:17.573 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:34:17.574 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.574 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.574 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:34:17.574 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.871362+0000", 2024-08-19T14:34:17.574 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:34:17.574 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.574 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.574 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:34:17.574 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.871364+0000", 2024-08-19T14:34:17.574 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.891e-06 2024-08-19T14:34:17.574 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.574 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.574 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:34:17.575 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.871416+0000", 2024-08-19T14:34:17.575 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1847e-05 2024-08-19T14:34:17.575 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.575 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.575 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:34:17.575 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.871431+0000", 2024-08-19T14:34:17.575 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.465e-05 2024-08-19T14:34:17.575 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.575 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.575 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:34:17.575 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.871438+0000", 2024-08-19T14:34:17.575 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3760000000000002e-06 2024-08-19T14:34:17.575 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.575 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.575 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:34:17.575 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.880941+0000", 2024-08-19T14:34:17.576 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0095025909999999995 2024-08-19T14:34:17.576 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.576 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.576 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:34:17.576 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.881111+0000", 2024-08-19T14:34:17.576 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00017021000000000001 2024-08-19T14:34:17.576 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.576 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.576 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 1,2", 2024-08-19T14:34:17.576 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.881293+0000", 2024-08-19T14:34:17.576 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00018233600000000001 2024-08-19T14:34:17.576 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.576 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.576 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:34:17.576 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.893200+0000", 2024-08-19T14:34:17.577 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011907188000000001 2024-08-19T14:34:17.577 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.577 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.577 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:34:17.577 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.895909+0000", 2024-08-19T14:34:17.577 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.002708462 2024-08-19T14:34:17.577 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.577 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.577 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:34:17.577 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.318493+0000", 2024-08-19T14:34:17.577 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.422583977 2024-08-19T14:34:17.577 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.577 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.577 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:34:17.577 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.318556+0000", 2024-08-19T14:34:17.578 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3541999999999996e-05 2024-08-19T14:34:17.578 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.578 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.578 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:34:17.578 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.322505+0000", 2024-08-19T14:34:17.578 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0039490560000000003 2024-08-19T14:34:17.578 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:34:17.578 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:34:17.578 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:34:17.578 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.578 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.578 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26191.0:76387 6.e 6:73030306:::benchmark_data_smithi045_97937_object76386:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e527)", 2024-08-19T14:34:17.578 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:29:17.889174+0000", 2024-08-19T14:34:17.578 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 299.57323963300001, 2024-08-19T14:34:17.578 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.43627335299999997, 2024-08-19T14:34:17.579 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:34:17.579 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:34:17.579 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:34:17.579 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:34:17.579 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26191", 2024-08-19T14:34:17.579 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:34:17.579 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 76387 2024-08-19T14:34:17.579 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.579 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:34:17.579 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.579 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:34:17.579 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.889174+0000", 2024-08-19T14:34:17.579 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:34:17.579 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.579 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.580 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:34:17.580 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.889174+0000", 2024-08-19T14:34:17.580 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:34:17.580 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.580 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.580 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:34:17.580 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.889176+0000", 2024-08-19T14:34:17.580 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.533e-06 2024-08-19T14:34:17.580 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.580 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.580 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:34:17.580 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.889227+0000", 2024-08-19T14:34:17.581 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1677e-05 2024-08-19T14:34:17.581 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.581 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.581 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:34:17.581 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.889239+0000", 2024-08-19T14:34:17.581 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.223e-05 2024-08-19T14:34:17.581 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.581 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.581 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:34:17.581 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.889247+0000", 2024-08-19T14:34:17.581 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2980000000000003e-06 2024-08-19T14:34:17.581 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.581 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.581 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:34:17.581 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.893233+0000", 2024-08-19T14:34:17.582 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0039860779999999997 2024-08-19T14:34:17.582 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.582 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.582 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:34:17.582 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.893388+0000", 2024-08-19T14:34:17.582 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000155501 2024-08-19T14:34:17.582 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.582 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.582 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 1,2", 2024-08-19T14:34:17.582 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.893560+0000", 2024-08-19T14:34:17.582 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00017121700000000001 2024-08-19T14:34:17.582 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.582 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.582 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:34:17.583 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.896062+0000", 2024-08-19T14:34:17.583 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0025023749999999998 2024-08-19T14:34:17.583 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.583 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.583 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:34:17.583 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.912814+0000", 2024-08-19T14:34:17.583 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.016751920999999999 2024-08-19T14:34:17.583 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.583 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.583 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:34:17.583 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.322546+0000", 2024-08-19T14:34:17.583 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.40973219 2024-08-19T14:34:17.583 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.584 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.584 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:34:17.584 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.322586+0000", 2024-08-19T14:34:17.584 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0370999999999997e-05 2024-08-19T14:34:17.584 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.584 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.584 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:34:17.584 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.325447+0000", 2024-08-19T14:34:17.584 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0028609619999999999 2024-08-19T14:34:17.584 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:34:17.584 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:34:17.584 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:34:17.584 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.584 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.585 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26191.0:76389 6.e 6:771c5bb8:::benchmark_data_smithi045_97937_object76388:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e527)", 2024-08-19T14:34:17.585 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:29:17.898506+0000", 2024-08-19T14:34:17.585 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 299.56390741199999, 2024-08-19T14:34:17.585 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.43093812399999998, 2024-08-19T14:34:17.585 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:34:17.585 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:34:17.585 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:34:17.585 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:34:17.585 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26191", 2024-08-19T14:34:17.585 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:34:17.585 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 76389 2024-08-19T14:34:17.585 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.585 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:34:17.585 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.586 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:34:17.586 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.898506+0000", 2024-08-19T14:34:17.586 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:34:17.586 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.586 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.586 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:34:17.586 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.898506+0000", 2024-08-19T14:34:17.586 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:34:17.586 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.586 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.586 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:34:17.586 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.898507+0000", 2024-08-19T14:34:17.586 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.251e-06 2024-08-19T14:34:17.586 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.587 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.587 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:34:17.587 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.898538+0000", 2024-08-19T14:34:17.587 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0233999999999999e-05 2024-08-19T14:34:17.587 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.587 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.587 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:34:17.587 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.898549+0000", 2024-08-19T14:34:17.587 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1674999999999999e-05 2024-08-19T14:34:17.587 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.587 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.587 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:34:17.587 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.898555+0000", 2024-08-19T14:34:17.587 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6160000000000001e-06 2024-08-19T14:34:17.588 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.588 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.588 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:34:17.588 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.898599+0000", 2024-08-19T14:34:17.588 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3967000000000001e-05 2024-08-19T14:34:17.588 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.588 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.588 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:34:17.588 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.898713+0000", 2024-08-19T14:34:17.588 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000114004 2024-08-19T14:34:17.588 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.588 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.588 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 1,2", 2024-08-19T14:34:17.588 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.898895+0000", 2024-08-19T14:34:17.589 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000182237 2024-08-19T14:34:17.589 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.589 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.589 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:34:17.589 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.901611+0000", 2024-08-19T14:34:17.589 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0027155510000000001 2024-08-19T14:34:17.589 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.589 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.589 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:34:17.589 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.912879+0000", 2024-08-19T14:34:17.589 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011268174000000001 2024-08-19T14:34:17.589 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.589 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.589 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:34:17.590 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.325505+0000", 2024-08-19T14:34:17.590 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.41262576299999998 2024-08-19T14:34:17.590 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.590 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.590 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:34:17.590 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.325568+0000", 2024-08-19T14:34:17.590 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3264999999999998e-05 2024-08-19T14:34:17.590 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.590 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.590 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:34:17.590 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.329444+0000", 2024-08-19T14:34:17.590 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0038763869999999998 2024-08-19T14:34:17.590 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:34:17.591 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:34:17.591 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:34:17.591 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.591 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.591 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26191.0:76394 6.e 6:7e16e1b0:::benchmark_data_smithi045_97937_object76393:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e527)", 2024-08-19T14:34:17.591 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:29:17.910225+0000", 2024-08-19T14:34:17.591 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 299.55218864699998, 2024-08-19T14:34:17.591 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.42283432900000001, 2024-08-19T14:34:17.591 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:34:17.591 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:34:17.591 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:34:17.591 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:34:17.591 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26191", 2024-08-19T14:34:17.591 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:34:17.592 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 76394 2024-08-19T14:34:17.592 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.592 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:34:17.592 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.592 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:34:17.592 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.910225+0000", 2024-08-19T14:34:17.592 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:34:17.592 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.592 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.592 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:34:17.592 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.910225+0000", 2024-08-19T14:34:17.592 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:34:17.592 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.592 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.593 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:34:17.593 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.910230+0000", 2024-08-19T14:34:17.593 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7589999999999997e-06 2024-08-19T14:34:17.593 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.593 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.593 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:34:17.593 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.910271+0000", 2024-08-19T14:34:17.593 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1482e-05 2024-08-19T14:34:17.593 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.593 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.593 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:34:17.593 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.910290+0000", 2024-08-19T14:34:17.593 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8986000000000001e-05 2024-08-19T14:34:17.593 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.594 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.594 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:34:17.594 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.910298+0000", 2024-08-19T14:34:17.594 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5979999999999999e-06 2024-08-19T14:34:17.594 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.594 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.594 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:34:17.594 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.912937+0000", 2024-08-19T14:34:17.594 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.002639078 2024-08-19T14:34:17.594 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.594 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.594 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:34:17.594 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.913081+0000", 2024-08-19T14:34:17.594 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014420399999999999 2024-08-19T14:34:17.595 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.595 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.595 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 1,2", 2024-08-19T14:34:17.595 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.913237+0000", 2024-08-19T14:34:17.595 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00015604599999999999 2024-08-19T14:34:17.595 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.595 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.595 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:34:17.595 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.924898+0000", 2024-08-19T14:34:17.595 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011660831 2024-08-19T14:34:17.595 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.595 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.595 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:34:17.595 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.924947+0000", 2024-08-19T14:34:17.596 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9063000000000001e-05 2024-08-19T14:34:17.596 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.596 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.596 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:34:17.596 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.329507+0000", 2024-08-19T14:34:17.596 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.40456007399999999 2024-08-19T14:34:17.596 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.596 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.596 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:34:17.596 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.329568+0000", 2024-08-19T14:34:17.596 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1334000000000006e-05 2024-08-19T14:34:17.596 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.596 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.596 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:34:17.597 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.333059+0000", 2024-08-19T14:34:17.597 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0034908740000000001 2024-08-19T14:34:17.597 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:34:17.597 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:34:17.597 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:34:17.597 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.597 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.597 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26191.0:76413 6.e 6:7e0cb892:::benchmark_data_smithi045_97937_object76412:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e527)", 2024-08-19T14:34:17.597 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:29:17.945624+0000", 2024-08-19T14:34:17.597 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 299.51678998099999, 2024-08-19T14:34:17.597 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.39115324499999998, 2024-08-19T14:34:17.597 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:34:17.597 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:34:17.597 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:34:17.598 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:34:17.598 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26191", 2024-08-19T14:34:17.598 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:34:17.598 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 76413 2024-08-19T14:34:17.598 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.598 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:34:17.598 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.598 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:34:17.598 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.945624+0000", 2024-08-19T14:34:17.598 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:34:17.598 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.598 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.598 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:34:17.599 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.945624+0000", 2024-08-19T14:34:17.599 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:34:17.599 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.599 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.599 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:34:17.599 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.945625+0000", 2024-08-19T14:34:17.599 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7710000000000001e-06 2024-08-19T14:34:17.599 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.599 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.599 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:34:17.599 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.945669+0000", 2024-08-19T14:34:17.599 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3237e-05 2024-08-19T14:34:17.599 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.600 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.600 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:34:17.600 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.945682+0000", 2024-08-19T14:34:17.600 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3033999999999999e-05 2024-08-19T14:34:17.600 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.600 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.600 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:34:17.600 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.945690+0000", 2024-08-19T14:34:17.600 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.5769999999999994e-06 2024-08-19T14:34:17.600 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.600 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.600 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:34:17.600 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.945808+0000", 2024-08-19T14:34:17.601 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000117803 2024-08-19T14:34:17.601 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.601 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.601 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:34:17.601 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.946009+0000", 2024-08-19T14:34:17.601 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00020067899999999999 2024-08-19T14:34:17.601 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.601 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.601 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 1,2", 2024-08-19T14:34:17.601 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.946188+0000", 2024-08-19T14:34:17.601 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000179657 2024-08-19T14:34:17.601 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.601 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.601 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:34:17.601 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.949682+0000", 2024-08-19T14:34:17.602 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0034936250000000002 2024-08-19T14:34:17.602 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.602 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.602 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:34:17.602 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:17.955654+0000", 2024-08-19T14:34:17.602 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0059714590000000001 2024-08-19T14:34:17.602 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.602 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.602 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:34:17.602 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.333114+0000", 2024-08-19T14:34:17.602 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.37746024700000003 2024-08-19T14:34:17.602 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.602 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.602 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:34:17.603 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.333155+0000", 2024-08-19T14:34:17.603 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1365000000000001e-05 2024-08-19T14:34:17.603 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.603 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.603 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:34:17.603 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.336777+0000", 2024-08-19T14:34:17.603 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0036217910000000001 2024-08-19T14:34:17.603 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:34:17.603 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:34:17.603 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:34:17.603 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.603 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.603 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26191.0:76434 6.e 6:7ca6e209:::benchmark_data_smithi045_97937_object76433:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e527)", 2024-08-19T14:34:17.603 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:29:18.006573+0000", 2024-08-19T14:34:17.603 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 299.45584024599998, 2024-08-19T14:34:17.604 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.33460235999999999, 2024-08-19T14:34:17.604 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:34:17.604 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:34:17.604 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:34:17.604 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:34:17.604 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26191", 2024-08-19T14:34:17.604 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:34:17.604 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 76434 2024-08-19T14:34:17.604 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.604 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:34:17.604 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.604 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:34:17.604 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.006573+0000", 2024-08-19T14:34:17.604 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:34:17.604 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.605 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.605 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:34:17.605 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.006573+0000", 2024-08-19T14:34:17.605 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:34:17.605 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.605 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.605 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:34:17.605 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.006575+0000", 2024-08-19T14:34:17.605 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2529999999999999e-06 2024-08-19T14:34:17.605 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.605 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.606 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:34:17.606 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.006633+0000", 2024-08-19T14:34:17.606 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8835000000000002e-05 2024-08-19T14:34:17.606 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.606 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.606 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:34:17.606 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.006644+0000", 2024-08-19T14:34:17.606 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0866999999999999e-05 2024-08-19T14:34:17.606 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.606 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.606 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:34:17.606 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.006650+0000", 2024-08-19T14:34:17.606 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1819999999999997e-06 2024-08-19T14:34:17.607 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.607 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.607 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:34:17.607 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.006705+0000", 2024-08-19T14:34:17.607 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5167000000000002e-05 2024-08-19T14:34:17.607 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.607 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.607 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:34:17.607 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.006871+0000", 2024-08-19T14:34:17.607 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000166663 2024-08-19T14:34:17.607 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.607 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.607 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 1,2", 2024-08-19T14:34:17.607 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.007065+0000", 2024-08-19T14:34:17.608 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00019317299999999999 2024-08-19T14:34:17.608 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.608 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.608 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:34:17.608 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.009738+0000", 2024-08-19T14:34:17.608 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0026729639999999999 2024-08-19T14:34:17.608 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.608 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.608 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:34:17.608 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.018726+0000", 2024-08-19T14:34:17.608 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0089886069999999992 2024-08-19T14:34:17.608 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.608 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.608 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:34:17.609 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.337011+0000", 2024-08-19T14:34:17.609 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.31828483499999999 2024-08-19T14:34:17.609 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.609 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.609 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:34:17.609 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.337109+0000", 2024-08-19T14:34:17.609 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.8092999999999997e-05 2024-08-19T14:34:17.609 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:17.609 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:17.609 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:34:17.609 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:29:18.341176+0000", 2024-08-19T14:34:17.609 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0040667209999999997 2024-08-19T14:34:17.609 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:34:17.609 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:34:17.609 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:34:17.610 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:34:17.610 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:34:17.610 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:34:17.610 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:17.610 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:34:17.711 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:34:17.747 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:17.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:17.749 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:34:17.850 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:34:17.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:17 smithi031 ceph-mon[16336]: pgmap v1451: 47 pgs: 1 active+recovering+undersized+remapped, 3 peering, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+undersized+remapped, 3 remapped+peering, 1 active+recovering+undersized+degraded+remapped, 4 active+undersized, 6 active+undersized+degraded, 26 active+clean; 4.7 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 51 MiB/s wr, 819 op/s; 33764/229374 objects degraded (14.720%); 9431/229374 objects misplaced (4.112%); 752 KiB/s, 10 objects/s recovering 2024-08-19T14:34:17.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:17 smithi031 ceph-mon[16336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:34:17.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:17 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:34:17.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:17 smithi031 ceph-mon[16336]: osdmap e621: 12 total, 11 up, 10 in 2024-08-19T14:34:17.884 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:17.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:17.885 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:34:17.986 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:34:18.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:17 smithi032 ceph-mon[20004]: pgmap v1451: 47 pgs: 1 active+recovering+undersized+remapped, 3 peering, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+undersized+remapped, 3 remapped+peering, 1 active+recovering+undersized+degraded+remapped, 4 active+undersized, 6 active+undersized+degraded, 26 active+clean; 4.7 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 51 MiB/s wr, 819 op/s; 33764/229374 objects degraded (14.720%); 9431/229374 objects misplaced (4.112%); 752 KiB/s, 10 objects/s recovering 2024-08-19T14:34:18.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:17 smithi032 ceph-mon[20004]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:34:18.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:17 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:34:18.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:17 smithi032 ceph-mon[20004]: osdmap e621: 12 total, 11 up, 10 in 2024-08-19T14:34:18.020 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:18.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:18.021 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:34:18.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:17 smithi045 ceph-mon[20143]: pgmap v1451: 47 pgs: 1 active+recovering+undersized+remapped, 3 peering, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+undersized+remapped, 3 remapped+peering, 1 active+recovering+undersized+degraded+remapped, 4 active+undersized, 6 active+undersized+degraded, 26 active+clean; 4.7 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 51 MiB/s wr, 819 op/s; 33764/229374 objects degraded (14.720%); 9431/229374 objects misplaced (4.112%); 752 KiB/s, 10 objects/s recovering 2024-08-19T14:34:18.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:17 smithi045 ceph-mon[20143]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:34:18.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:17 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:34:18.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:17 smithi045 ceph-mon[20143]: osdmap e621: 12 total, 11 up, 10 in 2024-08-19T14:34:18.123 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:34:18.145 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:18.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:18.146 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:34:18.247 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:34:18.278 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:18.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:18.279 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:34:18.380 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:34:18.402 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:18.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:18.402 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:34:18.444 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 45 16 80312 80296 111.512 57.3125 0.0635242 0.00895181 2024-08-19T14:34:18.506 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:34:18.535 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:18.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:18.536 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:34:18.637 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:34:18.646 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-08-19T14:34:18.670 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:18.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:18.671 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:34:18.772 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:34:18.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:18.796 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:18.796 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:34:18.897 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:34:18.922 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:18.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:18.923 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:34:18.929 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:34:19.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:18 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 33764/229374 objects degraded (14.720%), 8 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-19T14:34:19.024 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:34:19.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:18 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 33764/229374 objects degraded (14.720%), 8 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-19T14:34:19.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:19.047 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:19.047 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:34:19.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:18 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 33764/229374 objects degraded (14.720%), 8 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-19T14:34:19.148 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:34:19.172 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:19.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:19.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:34:19.273 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:34:19.303 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:19.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:19.304 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:34:19.405 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:34:19.439 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:19.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:19.440 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:34:19.444 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 46 16 81269 81253 110.388 59.8125 0.00763664 0.00905416 2024-08-19T14:34:19.541 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:34:19.564 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:19.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:19.565 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:34:19.590 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:34:19.590 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-08-19T14:34:19.590 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-08-19T14:34:19.590 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:34:19.666 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:34:19.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:19.700 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:19.700 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:34:19.801 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:34:19.835 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:19.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:19.836 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:34:19.937 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:34:19.967 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:19.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:19.968 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:34:20.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:19 smithi032 ceph-mon[20004]: pgmap v1453: 47 pgs: 2 active+recovering+undersized+remapped, 3 peering, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+undersized+remapped, 3 remapped+peering, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized, 5 active+undersized+degraded, 25 active+clean; 4.7 GiB data, 23 GiB used, 872 GiB / 894 GiB avail; 47 MiB/s wr, 756 op/s; 33930/230418 objects degraded (14.725%); 18617/230418 objects misplaced (8.080%); 3.0 MiB/s, 46 objects/s recovering 2024-08-19T14:34:20.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:19 smithi045 ceph-mon[20143]: pgmap v1453: 47 pgs: 2 active+recovering+undersized+remapped, 3 peering, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+undersized+remapped, 3 remapped+peering, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized, 5 active+undersized+degraded, 25 active+clean; 4.7 GiB data, 23 GiB used, 872 GiB / 894 GiB avail; 47 MiB/s wr, 756 op/s; 33930/230418 objects degraded (14.725%); 18617/230418 objects misplaced (8.080%); 3.0 MiB/s, 46 objects/s recovering 2024-08-19T14:34:20.069 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:34:20.102 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:20.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:20.104 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:34:20.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:19 smithi031 ceph-mon[16336]: pgmap v1453: 47 pgs: 2 active+recovering+undersized+remapped, 3 peering, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+undersized+remapped, 3 remapped+peering, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized, 5 active+undersized+degraded, 25 active+clean; 4.7 GiB data, 23 GiB used, 872 GiB / 894 GiB avail; 47 MiB/s wr, 756 op/s; 33930/230418 objects degraded (14.725%); 18617/230418 objects misplaced (8.080%); 3.0 MiB/s, 46 objects/s recovering 2024-08-19T14:34:20.206 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:34:20.227 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:20.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:20.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:34:20.329 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:34:20.358 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:20.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:20.360 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:34:20.364 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-08-19T14:34:20.444 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 47 16 82036 82020 109.059 47.9375 0.00684881 0.00916225 2024-08-19T14:34:20.461 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:34:20.495 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:20.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:20.496 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:34:20.597 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:34:20.629 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:20.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:20.630 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:34:20.647 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:34:20.730 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:34:20.753 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:20.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:20.754 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:34:20.855 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:34:20.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:20.894 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:20.894 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:34:20.995 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:34:21.031 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:21.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:21.034 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:34:21.135 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:34:21.165 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:21.166 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:34:21.166 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-08-19T14:34:21.166 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-08-19T14:34:21.166 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-08-19T14:34:21.166 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:34:21.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:21.167 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:34:21.268 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:34:21.300 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:21.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:21.301 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:34:21.402 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:34:21.425 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:21.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:21.426 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:34:21.444 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 48 16 82866 82850 107.868 51.875 0.00511666 0.00926512 2024-08-19T14:34:21.527 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:34:21.549 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:21.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:21.550 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:34:21.651 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:34:21.685 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:21.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:21.686 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:34:21.787 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:34:21.817 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:21.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:21.818 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:34:21.823 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-08-19T14:34:21.919 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:34:21.945 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:21.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:21.946 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:34:22.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:21 smithi032 ceph-mon[20004]: pgmap v1454: 47 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+remapped, 3 peering, 2 remapped+peering, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized, 5 active+undersized+degraded, 26 active+clean; 4.8 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 50 MiB/s wr, 793 op/s; 29526/237573 objects degraded (12.428%); 23094/237573 objects misplaced (9.721%); 4.5 MiB/s, 71 objects/s recovering 2024-08-19T14:34:22.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:21 smithi045 ceph-mon[20143]: pgmap v1454: 47 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+remapped, 3 peering, 2 remapped+peering, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized, 5 active+undersized+degraded, 26 active+clean; 4.8 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 50 MiB/s wr, 793 op/s; 29526/237573 objects degraded (12.428%); 23094/237573 objects misplaced (9.721%); 4.5 MiB/s, 71 objects/s recovering 2024-08-19T14:34:22.046 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:34:22.070 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:22.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:22.071 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:34:22.112 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:34:22.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:21 smithi031 ceph-mon[16336]: pgmap v1454: 47 pgs: 2 active+clean+remapped, 3 active+recovering+undersized+remapped, 3 peering, 2 remapped+peering, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized, 5 active+undersized+degraded, 26 active+clean; 4.8 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 50 MiB/s wr, 793 op/s; 29526/237573 objects degraded (12.428%); 23094/237573 objects misplaced (9.721%); 4.5 MiB/s, 71 objects/s recovering 2024-08-19T14:34:22.172 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:34:22.201 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:22.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:22.202 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:34:22.303 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:34:22.327 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:22.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:22.328 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:34:22.429 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:34:22.444 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 49 16 83768 83752 106.817 56.375 0.0393114 0.00935694 2024-08-19T14:34:22.453 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:22.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:22.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:34:22.555 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:34:22.580 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:22.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:22.581 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:34:22.629 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:34:22.629 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-08-19T14:34:22.629 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-08-19T14:34:22.629 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-08-19T14:34:22.629 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.629 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:22993 5.7 5:e5194b18:::benchmark_data_smithi045_85309_object22992:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e426)", 2024-08-19T14:34:22.629 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:24:55.243091+0000", 2024-08-19T14:34:22.629 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 567.38367726299998, 2024-08-19T14:34:22.629 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20799368400000001, 2024-08-19T14:34:22.629 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:34:22.629 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:34:22.630 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:34:22.630 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:34:22.630 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:34:22.630 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:34:22.630 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 22993 2024-08-19T14:34:22.630 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.630 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:34:22.630 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.630 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:34:22.630 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.243091+0000", 2024-08-19T14:34:22.630 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:34:22.630 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.630 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.631 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:34:22.631 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.243091+0000", 2024-08-19T14:34:22.631 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:34:22.631 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.631 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.631 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:34:22.631 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.243092+0000", 2024-08-19T14:34:22.631 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.9400000000000004e-07 2024-08-19T14:34:22.631 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.631 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.631 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:34:22.631 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.243122+0000", 2024-08-19T14:34:22.631 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0105000000000001e-05 2024-08-19T14:34:22.631 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.632 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.632 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:34:22.632 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.243131+0000", 2024-08-19T14:34:22.632 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.6889999999999999e-06 2024-08-19T14:34:22.632 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.632 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.632 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:34:22.632 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.243138+0000", 2024-08-19T14:34:22.632 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0339999999999999e-06 2024-08-19T14:34:22.632 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.632 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.632 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:34:22.632 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.243187+0000", 2024-08-19T14:34:22.632 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8764999999999998e-05 2024-08-19T14:34:22.632 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.633 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.633 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:34:22.633 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.243294+0000", 2024-08-19T14:34:22.633 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000106908 2024-08-19T14:34:22.633 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.633 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.633 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 1,9", 2024-08-19T14:34:22.633 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.243424+0000", 2024-08-19T14:34:22.633 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000129441 2024-08-19T14:34:22.633 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.633 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.633 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:34:22.633 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.245731+0000", 2024-08-19T14:34:22.634 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0023073210000000002 2024-08-19T14:34:22.634 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.634 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.634 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:34:22.634 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.246134+0000", 2024-08-19T14:34:22.634 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00040348800000000002 2024-08-19T14:34:22.634 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.634 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.634 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:34:22.634 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.450957+0000", 2024-08-19T14:34:22.634 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20482256099999999 2024-08-19T14:34:22.634 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.634 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.634 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:34:22.635 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.451032+0000", 2024-08-19T14:34:22.635 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4701000000000002e-05 2024-08-19T14:34:22.635 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.635 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.635 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:34:22.635 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.451084+0000", 2024-08-19T14:34:22.635 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2777e-05 2024-08-19T14:34:22.635 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:34:22.635 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:34:22.635 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:34:22.635 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.635 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.635 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:23013 5.7 5:ee7b3505:::benchmark_data_smithi045_85309_object23012:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e426)", 2024-08-19T14:34:22.636 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:24:55.249101+0000", 2024-08-19T14:34:22.636 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 567.37766672600003, 2024-08-19T14:34:22.636 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20208954800000001, 2024-08-19T14:34:22.636 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:34:22.636 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:34:22.636 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:34:22.636 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:34:22.636 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:34:22.636 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:34:22.636 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 23013 2024-08-19T14:34:22.636 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.637 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:34:22.637 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.637 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:34:22.637 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.249101+0000", 2024-08-19T14:34:22.637 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:34:22.637 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.637 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.637 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:34:22.637 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.249101+0000", 2024-08-19T14:34:22.637 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:34:22.637 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.637 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.637 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:34:22.637 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.249103+0000", 2024-08-19T14:34:22.638 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.271e-06 2024-08-19T14:34:22.638 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.638 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.638 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:34:22.638 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.249300+0000", 2024-08-19T14:34:22.638 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00019775200000000001 2024-08-19T14:34:22.638 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.638 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.638 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:34:22.638 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.249313+0000", 2024-08-19T14:34:22.638 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.312e-05 2024-08-19T14:34:22.638 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.638 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.639 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:34:22.639 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.249321+0000", 2024-08-19T14:34:22.639 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.43e-06 2024-08-19T14:34:22.639 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.639 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.639 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:34:22.639 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.249451+0000", 2024-08-19T14:34:22.639 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000129982 2024-08-19T14:34:22.639 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.639 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.639 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:34:22.639 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.249576+0000", 2024-08-19T14:34:22.639 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012470299999999999 2024-08-19T14:34:22.640 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.640 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.640 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 1,9", 2024-08-19T14:34:22.640 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.249703+0000", 2024-08-19T14:34:22.640 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000127315 2024-08-19T14:34:22.640 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.640 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.640 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:34:22.640 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.253189+0000", 2024-08-19T14:34:22.640 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0034865310000000002 2024-08-19T14:34:22.640 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.640 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.640 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:34:22.640 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.254256+0000", 2024-08-19T14:34:22.640 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0010667630000000001 2024-08-19T14:34:22.641 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.641 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.641 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:34:22.641 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.451121+0000", 2024-08-19T14:34:22.641 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.19686435199999999 2024-08-19T14:34:22.641 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.641 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.641 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:34:22.641 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.451155+0000", 2024-08-19T14:34:22.642 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.4785e-05 2024-08-19T14:34:22.642 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.642 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.642 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:34:22.642 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:24:55.451191+0000", 2024-08-19T14:34:22.642 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5543999999999997e-05 2024-08-19T14:34:22.642 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:34:22.642 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:34:22.642 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:34:22.642 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.642 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.642 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:55755 5.6 5:699d86b9:::benchmark_data_smithi045_85309_object55754:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e431)", 2024-08-19T14:34:22.642 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:25:09.597312+0000", 2024-08-19T14:34:22.643 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 553.02945588900002, 2024-08-19T14:34:22.643 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.209956798, 2024-08-19T14:34:22.643 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:34:22.643 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:34:22.643 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:34:22.643 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:34:22.643 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:34:22.643 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:34:22.644 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 55755 2024-08-19T14:34:22.644 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.644 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:34:22.644 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.644 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:34:22.644 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.597312+0000", 2024-08-19T14:34:22.644 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:34:22.644 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.644 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.645 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:34:22.645 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.597312+0000", 2024-08-19T14:34:22.645 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:34:22.645 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.645 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.645 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:34:22.645 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.597313+0000", 2024-08-19T14:34:22.645 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.13e-06 2024-08-19T14:34:22.645 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.645 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.645 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:34:22.645 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.597346+0000", 2024-08-19T14:34:22.645 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3019999999999999e-05 2024-08-19T14:34:22.645 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.646 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.646 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:34:22.646 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.597355+0000", 2024-08-19T14:34:22.646 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.0399999999999998e-06 2024-08-19T14:34:22.646 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.646 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.646 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:34:22.646 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.597362+0000", 2024-08-19T14:34:22.646 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2659999999999998e-06 2024-08-19T14:34:22.646 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.647 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.647 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:34:22.647 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.597410+0000", 2024-08-19T14:34:22.647 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8013000000000003e-05 2024-08-19T14:34:22.647 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.647 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.647 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:34:22.647 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.597518+0000", 2024-08-19T14:34:22.647 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00010813 2024-08-19T14:34:22.647 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.647 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.647 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 7,8", 2024-08-19T14:34:22.647 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.597639+0000", 2024-08-19T14:34:22.648 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000121205 2024-08-19T14:34:22.648 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.648 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.648 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:34:22.648 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.600786+0000", 2024-08-19T14:34:22.648 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.003147558 2024-08-19T14:34:22.648 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.648 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.648 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:34:22.648 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.601648+0000", 2024-08-19T14:34:22.648 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00086196199999999997 2024-08-19T14:34:22.648 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.648 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.649 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:34:22.649 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.807181+0000", 2024-08-19T14:34:22.649 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20553236799999999 2024-08-19T14:34:22.649 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.649 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.649 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:34:22.649 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.807220+0000", 2024-08-19T14:34:22.649 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9119999999999998e-05 2024-08-19T14:34:22.649 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.649 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.649 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:34:22.649 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.807269+0000", 2024-08-19T14:34:22.649 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8986000000000002e-05 2024-08-19T14:34:22.649 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:34:22.650 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:34:22.650 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:34:22.650 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.650 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.650 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:55761 5.4 5:21702bb2:::benchmark_data_smithi045_85309_object55760:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e431)", 2024-08-19T14:34:22.650 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:25:09.599621+0000", 2024-08-19T14:34:22.650 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 553.02714708799999, 2024-08-19T14:34:22.650 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20994792700000001, 2024-08-19T14:34:22.650 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:34:22.650 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:34:22.650 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:34:22.650 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:34:22.650 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:34:22.651 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:34:22.651 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 55761 2024-08-19T14:34:22.651 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.651 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:34:22.651 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.651 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:34:22.651 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.599621+0000", 2024-08-19T14:34:22.651 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:34:22.651 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.651 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.651 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:34:22.651 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.599621+0000", 2024-08-19T14:34:22.651 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:34:22.652 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.652 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.652 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:34:22.652 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.599622+0000", 2024-08-19T14:34:22.652 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1689999999999999e-06 2024-08-19T14:34:22.652 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.652 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.652 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:34:22.652 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.599684+0000", 2024-08-19T14:34:22.652 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2228000000000001e-05 2024-08-19T14:34:22.652 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.652 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.652 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:34:22.653 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.599698+0000", 2024-08-19T14:34:22.653 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4031e-05 2024-08-19T14:34:22.653 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.653 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.653 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:34:22.653 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.599705+0000", 2024-08-19T14:34:22.653 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0099999999999998e-06 2024-08-19T14:34:22.653 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.653 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.653 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:34:22.653 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.599804+0000", 2024-08-19T14:34:22.653 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.8288000000000005e-05 2024-08-19T14:34:22.653 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.654 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.654 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:34:22.654 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.600054+0000", 2024-08-19T14:34:22.654 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00025056999999999998 2024-08-19T14:34:22.654 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.654 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.654 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0,8", 2024-08-19T14:34:22.654 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.600199+0000", 2024-08-19T14:34:22.654 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014504200000000001 2024-08-19T14:34:22.654 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.654 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.654 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:34:22.655 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.604843+0000", 2024-08-19T14:34:22.655 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0046438290000000004 2024-08-19T14:34:22.655 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.655 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.655 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:34:22.655 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.604854+0000", 2024-08-19T14:34:22.655 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0971e-05 2024-08-19T14:34:22.655 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.655 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.655 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:34:22.656 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.809503+0000", 2024-08-19T14:34:22.656 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20464875799999999 2024-08-19T14:34:22.656 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.656 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.656 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:34:22.656 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.809534+0000", 2024-08-19T14:34:22.656 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1217999999999998e-05 2024-08-19T14:34:22.656 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.656 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.656 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:34:22.656 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.809569+0000", 2024-08-19T14:34:22.656 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.4813e-05 2024-08-19T14:34:22.656 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:34:22.656 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:34:22.656 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:34:22.657 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.657 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.657 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:55763 5.4 5:22d4b04e:::benchmark_data_smithi045_85309_object55762:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e431)", 2024-08-19T14:34:22.657 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:25:09.599725+0000", 2024-08-19T14:34:22.657 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 553.02704300599999, 2024-08-19T14:34:22.657 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20996778999999999, 2024-08-19T14:34:22.657 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:34:22.657 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:34:22.657 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:34:22.657 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:34:22.657 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:34:22.657 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:34:22.657 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 55763 2024-08-19T14:34:22.657 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.658 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:34:22.658 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.658 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:34:22.658 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.599725+0000", 2024-08-19T14:34:22.658 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:34:22.658 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.658 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.658 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:34:22.658 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.599725+0000", 2024-08-19T14:34:22.658 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:34:22.658 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.658 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.659 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:34:22.660 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.599726+0000", 2024-08-19T14:34:22.660 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2699999999999999e-07 2024-08-19T14:34:22.660 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.660 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.660 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:34:22.660 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.599802+0000", 2024-08-19T14:34:22.660 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6167000000000004e-05 2024-08-19T14:34:22.660 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.660 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.660 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:34:22.660 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.599811+0000", 2024-08-19T14:34:22.660 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.6220000000000004e-06 2024-08-19T14:34:22.660 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.660 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.661 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:34:22.661 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.599818+0000", 2024-08-19T14:34:22.661 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4420000000000003e-06 2024-08-19T14:34:22.661 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.661 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.661 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:34:22.661 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.602462+0000", 2024-08-19T14:34:22.661 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0026439990000000002 2024-08-19T14:34:22.661 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.661 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.661 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:34:22.661 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.602619+0000", 2024-08-19T14:34:22.661 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00015674800000000001 2024-08-19T14:34:22.661 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.661 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.661 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0,8", 2024-08-19T14:34:22.662 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.602748+0000", 2024-08-19T14:34:22.662 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012939900000000001 2024-08-19T14:34:22.662 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.662 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.662 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:34:22.662 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.609052+0000", 2024-08-19T14:34:22.662 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.006304128 2024-08-19T14:34:22.662 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.662 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.662 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:34:22.662 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.611165+0000", 2024-08-19T14:34:22.662 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.002113008 2024-08-19T14:34:22.662 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.663 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.663 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:34:22.663 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.809609+0000", 2024-08-19T14:34:22.663 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.19844395500000001 2024-08-19T14:34:22.663 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.663 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.663 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:34:22.663 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.809648+0000", 2024-08-19T14:34:22.663 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9081000000000001e-05 2024-08-19T14:34:22.663 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.663 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.663 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:34:22.663 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.809693+0000", 2024-08-19T14:34:22.663 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4613999999999997e-05 2024-08-19T14:34:22.663 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:34:22.664 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:34:22.664 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:34:22.664 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.664 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.664 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:55767 5.4 5:200c3bc0:::benchmark_data_smithi045_85309_object55766:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e431)", 2024-08-19T14:34:22.664 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:25:09.601839+0000", 2024-08-19T14:34:22.664 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 553.02492897800005, 2024-08-19T14:34:22.664 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20836729700000001, 2024-08-19T14:34:22.664 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:34:22.664 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:34:22.664 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:34:22.664 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:34:22.664 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:34:22.664 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:34:22.664 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 55767 2024-08-19T14:34:22.665 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.665 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:34:22.665 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.665 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:34:22.665 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.601839+0000", 2024-08-19T14:34:22.665 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:34:22.665 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.665 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.665 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:34:22.665 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.601839+0000", 2024-08-19T14:34:22.665 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:34:22.665 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.665 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.665 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:34:22.665 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.601840+0000", 2024-08-19T14:34:22.666 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.049e-06 2024-08-19T14:34:22.666 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.666 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.666 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:34:22.666 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.601863+0000", 2024-08-19T14:34:22.666 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2748999999999999e-05 2024-08-19T14:34:22.666 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.666 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.666 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:34:22.666 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.601874+0000", 2024-08-19T14:34:22.666 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1219999999999999e-05 2024-08-19T14:34:22.666 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.666 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.666 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:34:22.667 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.601881+0000", 2024-08-19T14:34:22.667 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.562e-06 2024-08-19T14:34:22.667 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.667 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.667 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:34:22.667 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.604918+0000", 2024-08-19T14:34:22.667 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.003037629 2024-08-19T14:34:22.667 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.668 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.668 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:34:22.668 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.605062+0000", 2024-08-19T14:34:22.668 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000143821 2024-08-19T14:34:22.668 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.668 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.668 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0,8", 2024-08-19T14:34:22.668 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.605192+0000", 2024-08-19T14:34:22.668 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013037300000000001 2024-08-19T14:34:22.668 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.668 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.668 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:34:22.668 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.611145+0000", 2024-08-19T14:34:22.669 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0059520939999999998 2024-08-19T14:34:22.669 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.669 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.669 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:34:22.669 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.611171+0000", 2024-08-19T14:34:22.669 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6618000000000002e-05 2024-08-19T14:34:22.669 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.669 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.669 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:34:22.669 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.810142+0000", 2024-08-19T14:34:22.669 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.19897035199999999 2024-08-19T14:34:22.669 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.669 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.670 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:34:22.670 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.810178+0000", 2024-08-19T14:34:22.670 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6950999999999999e-05 2024-08-19T14:34:22.670 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.670 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.670 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:34:22.670 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.810206+0000", 2024-08-19T14:34:22.670 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7878999999999999e-05 2024-08-19T14:34:22.670 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:34:22.670 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:34:22.670 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:34:22.670 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.670 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.671 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:55775 5.4 5:2da10789:::benchmark_data_smithi045_85309_object55774:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e431)", 2024-08-19T14:34:22.671 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:25:09.605530+0000", 2024-08-19T14:34:22.671 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 553.02123843100003, 2024-08-19T14:34:22.671 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20478083, 2024-08-19T14:34:22.671 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:34:22.671 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:34:22.671 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:34:22.671 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:34:22.671 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:34:22.671 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:34:22.671 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 55775 2024-08-19T14:34:22.671 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.671 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:34:22.671 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.672 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:34:22.672 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.605530+0000", 2024-08-19T14:34:22.672 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:34:22.672 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.672 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.672 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:34:22.672 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.605530+0000", 2024-08-19T14:34:22.672 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:34:22.672 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.672 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.672 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:34:22.672 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.605531+0000", 2024-08-19T14:34:22.672 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3039999999999999e-06 2024-08-19T14:34:22.672 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.672 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.673 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:34:22.673 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.605633+0000", 2024-08-19T14:34:22.673 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000102602 2024-08-19T14:34:22.673 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.673 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.673 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:34:22.673 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.605643+0000", 2024-08-19T14:34:22.673 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.4399999999999994e-06 2024-08-19T14:34:22.673 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.673 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.673 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:34:22.673 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.605649+0000", 2024-08-19T14:34:22.673 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2169999999999999e-06 2024-08-19T14:34:22.674 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.674 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.674 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:34:22.674 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.607082+0000", 2024-08-19T14:34:22.674 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0014326289999999999 2024-08-19T14:34:22.674 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.674 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.674 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:34:22.674 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.607190+0000", 2024-08-19T14:34:22.674 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000108428 2024-08-19T14:34:22.674 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.674 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.674 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0,8", 2024-08-19T14:34:22.675 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.607312+0000", 2024-08-19T14:34:22.675 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000121927 2024-08-19T14:34:22.675 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.675 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.675 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:34:22.675 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.611175+0000", 2024-08-19T14:34:22.675 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0038631889999999999 2024-08-19T14:34:22.675 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.675 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.675 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:34:22.675 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.614394+0000", 2024-08-19T14:34:22.675 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.003218505 2024-08-19T14:34:22.675 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.675 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.675 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:34:22.676 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.810243+0000", 2024-08-19T14:34:22.676 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.19584892000000001 2024-08-19T14:34:22.676 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.676 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.676 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:34:22.676 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.810269+0000", 2024-08-19T14:34:22.676 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6027e-05 2024-08-19T14:34:22.676 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.676 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.676 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:34:22.676 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.810310+0000", 2024-08-19T14:34:22.676 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1641999999999999e-05 2024-08-19T14:34:22.676 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:34:22.676 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:34:22.677 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:34:22.677 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.677 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.677 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:55779 5.4 5:28c62a63:::benchmark_data_smithi045_85309_object55778:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e431)", 2024-08-19T14:34:22.677 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:25:09.607121+0000", 2024-08-19T14:34:22.677 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 553.01964710100003, 2024-08-19T14:34:22.677 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20333711199999999, 2024-08-19T14:34:22.677 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:34:22.677 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:34:22.677 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:34:22.677 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:34:22.677 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:34:22.677 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:34:22.677 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 55779 2024-08-19T14:34:22.677 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.678 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:34:22.678 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.678 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:34:22.678 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.607121+0000", 2024-08-19T14:34:22.678 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:34:22.678 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.678 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.678 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:34:22.678 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.607121+0000", 2024-08-19T14:34:22.678 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:34:22.678 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.678 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.678 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:34:22.679 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.607122+0000", 2024-08-19T14:34:22.679 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1820000000000001e-06 2024-08-19T14:34:22.679 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.679 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.679 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:34:22.679 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.607157+0000", 2024-08-19T14:34:22.679 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.4443999999999998e-05 2024-08-19T14:34:22.679 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.679 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.679 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:34:22.679 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.607169+0000", 2024-08-19T14:34:22.679 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2531000000000001e-05 2024-08-19T14:34:22.679 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.679 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.679 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:34:22.680 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.607176+0000", 2024-08-19T14:34:22.680 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1199999999999996e-06 2024-08-19T14:34:22.680 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.680 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.680 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:34:22.680 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.609081+0000", 2024-08-19T14:34:22.680 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001904705 2024-08-19T14:34:22.680 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.680 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.680 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:34:22.680 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.609202+0000", 2024-08-19T14:34:22.680 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000121138 2024-08-19T14:34:22.680 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.680 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.680 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0,8", 2024-08-19T14:34:22.681 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.609328+0000", 2024-08-19T14:34:22.681 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012569100000000001 2024-08-19T14:34:22.681 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.681 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.681 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:34:22.681 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.611568+0000", 2024-08-19T14:34:22.681 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0022404769999999998 2024-08-19T14:34:22.681 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.681 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.681 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:34:22.681 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.616823+0000", 2024-08-19T14:34:22.681 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0052546889999999999 2024-08-19T14:34:22.681 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.681 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.682 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:34:22.682 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.810361+0000", 2024-08-19T14:34:22.682 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.193538245 2024-08-19T14:34:22.682 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.682 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.682 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:34:22.682 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.810410+0000", 2024-08-19T14:34:22.682 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8819000000000001e-05 2024-08-19T14:34:22.682 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.682 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.682 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:34:22.682 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:09.810458+0000", 2024-08-19T14:34:22.683 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8071000000000001e-05 2024-08-19T14:34:22.683 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:34:22.683 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:34:22.683 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:34:22.683 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.683 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.683 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:81628 5.7 5:e12ddb93:::benchmark_data_smithi045_85309_object81627:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e437)", 2024-08-19T14:34:22.683 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:25:25.047926+0000", 2024-08-19T14:34:22.683 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 537.57884213700004, 2024-08-19T14:34:22.683 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.207246069, 2024-08-19T14:34:22.683 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:34:22.683 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:34:22.683 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:34:22.683 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:34:22.683 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:34:22.684 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:34:22.684 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 81628 2024-08-19T14:34:22.684 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.684 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:34:22.684 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.684 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:34:22.684 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.047926+0000", 2024-08-19T14:34:22.684 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:34:22.684 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.684 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.684 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:34:22.684 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.047926+0000", 2024-08-19T14:34:22.684 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:34:22.684 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.685 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.685 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:34:22.685 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.047927+0000", 2024-08-19T14:34:22.685 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0219999999999999e-06 2024-08-19T14:34:22.685 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.685 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.685 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:34:22.685 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.047977+0000", 2024-08-19T14:34:22.685 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0170000000000002e-05 2024-08-19T14:34:22.685 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.685 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.685 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:34:22.685 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.047993+0000", 2024-08-19T14:34:22.685 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5835999999999999e-05 2024-08-19T14:34:22.685 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.686 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.686 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:34:22.686 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.047998+0000", 2024-08-19T14:34:22.686 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8500000000000002e-06 2024-08-19T14:34:22.686 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.686 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.686 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:34:22.686 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.048045+0000", 2024-08-19T14:34:22.686 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6828e-05 2024-08-19T14:34:22.686 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.686 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.686 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:34:22.686 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.048139+0000", 2024-08-19T14:34:22.687 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.4199999999999999e-05 2024-08-19T14:34:22.687 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.687 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.687 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 1,9", 2024-08-19T14:34:22.687 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.048232+0000", 2024-08-19T14:34:22.687 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3393999999999994e-05 2024-08-19T14:34:22.687 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.687 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.687 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:34:22.687 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.050672+0000", 2024-08-19T14:34:22.687 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0024403189999999998 2024-08-19T14:34:22.687 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.688 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.688 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:34:22.688 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.051858+0000", 2024-08-19T14:34:22.688 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0011859489999999999 2024-08-19T14:34:22.688 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.688 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.688 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:34:22.688 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.255101+0000", 2024-08-19T14:34:22.688 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20324223 2024-08-19T14:34:22.688 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.688 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.689 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:34:22.689 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.255142+0000", 2024-08-19T14:34:22.689 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0915999999999999e-05 2024-08-19T14:34:22.689 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.689 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.689 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:34:22.689 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.255172+0000", 2024-08-19T14:34:22.689 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0355e-05 2024-08-19T14:34:22.689 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:34:22.690 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:34:22.690 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:34:22.690 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.690 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.690 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:81639 5.7 5:ef0c5b8b:::benchmark_data_smithi045_85309_object81638:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e437)", 2024-08-19T14:34:22.690 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:25:25.053488+0000", 2024-08-19T14:34:22.690 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 537.57328025699996, 2024-08-19T14:34:22.690 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20480580300000001, 2024-08-19T14:34:22.691 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:34:22.691 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:34:22.691 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:34:22.691 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:34:22.691 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:34:22.691 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:34:22.691 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 81639 2024-08-19T14:34:22.691 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.691 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:34:22.691 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.691 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:34:22.691 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.053488+0000", 2024-08-19T14:34:22.691 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:34:22.692 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.692 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.692 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:34:22.692 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.053488+0000", 2024-08-19T14:34:22.692 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:34:22.692 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.692 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.692 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:34:22.692 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.053489+0000", 2024-08-19T14:34:22.693 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.4200000000000005e-07 2024-08-19T14:34:22.693 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.693 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.693 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:34:22.693 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.053559+0000", 2024-08-19T14:34:22.693 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0240000000000005e-05 2024-08-19T14:34:22.693 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.693 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.693 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:34:22.693 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.053568+0000", 2024-08-19T14:34:22.693 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.9989999999999997e-06 2024-08-19T14:34:22.693 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.693 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.694 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:34:22.694 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.053575+0000", 2024-08-19T14:34:22.694 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.092e-06 2024-08-19T14:34:22.694 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.694 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.694 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:34:22.694 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.053624+0000", 2024-08-19T14:34:22.694 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9209000000000003e-05 2024-08-19T14:34:22.694 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.694 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.694 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:34:22.694 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.053826+0000", 2024-08-19T14:34:22.694 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000201573 2024-08-19T14:34:22.694 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.695 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.695 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 1,9", 2024-08-19T14:34:22.695 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.054893+0000", 2024-08-19T14:34:22.695 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0010669849999999999 2024-08-19T14:34:22.695 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.695 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.695 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:34:22.695 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.058523+0000", 2024-08-19T14:34:22.695 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0036301929999999999 2024-08-19T14:34:22.695 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.695 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.695 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:34:22.695 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.060958+0000", 2024-08-19T14:34:22.695 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0024348579999999998 2024-08-19T14:34:22.696 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.696 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.696 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:34:22.696 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.258229+0000", 2024-08-19T14:34:22.696 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.19727160199999999 2024-08-19T14:34:22.696 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.696 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.696 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:34:22.696 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.258265+0000", 2024-08-19T14:34:22.696 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5717999999999999e-05 2024-08-19T14:34:22.696 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.696 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.696 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:34:22.696 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:25.258294+0000", 2024-08-19T14:34:22.697 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8492000000000001e-05 2024-08-19T14:34:22.697 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:34:22.697 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:34:22.697 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:34:22.697 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.697 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.697 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.25813.0:92839 5.f 5:f7605d59:::benchmark_data_smithi045_85309_object92838:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e445)", 2024-08-19T14:34:22.697 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:25:37.241907+0000", 2024-08-19T14:34:22.697 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 525.38486143800003, 2024-08-19T14:34:22.697 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20647278499999999, 2024-08-19T14:34:22.697 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:34:22.697 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:34:22.697 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:34:22.697 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:34:22.698 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.25813", 2024-08-19T14:34:22.698 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:34:22.698 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 92839 2024-08-19T14:34:22.698 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.698 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:34:22.698 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.698 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:34:22.698 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.241907+0000", 2024-08-19T14:34:22.698 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:34:22.698 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.698 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.698 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:34:22.698 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.241907+0000", 2024-08-19T14:34:22.699 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:34:22.699 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.699 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.699 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:34:22.699 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.241908+0000", 2024-08-19T14:34:22.699 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4050000000000001e-06 2024-08-19T14:34:22.699 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.699 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.699 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:34:22.699 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.241947+0000", 2024-08-19T14:34:22.699 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8847999999999998e-05 2024-08-19T14:34:22.699 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.699 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.700 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:34:22.700 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.241956+0000", 2024-08-19T14:34:22.700 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.5550000000000008e-06 2024-08-19T14:34:22.700 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.700 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.700 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:34:22.700 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.241962+0000", 2024-08-19T14:34:22.700 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2490000000000001e-06 2024-08-19T14:34:22.700 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.700 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.700 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:34:22.700 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.243721+0000", 2024-08-19T14:34:22.700 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0017588910000000001 2024-08-19T14:34:22.700 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.701 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.701 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:34:22.701 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.422533+0000", 2024-08-19T14:34:22.701 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.178812148 2024-08-19T14:34:22.701 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.701 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.701 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4,11", 2024-08-19T14:34:22.701 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.422686+0000", 2024-08-19T14:34:22.701 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00015375499999999999 2024-08-19T14:34:22.701 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.701 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.701 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:34:22.701 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.426020+0000", 2024-08-19T14:34:22.701 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.003333698 2024-08-19T14:34:22.702 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.702 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.702 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:34:22.702 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.430495+0000", 2024-08-19T14:34:22.702 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0044751740000000002 2024-08-19T14:34:22.702 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.702 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.702 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:34:22.702 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.435394+0000", 2024-08-19T14:34:22.702 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0048991900000000003 2024-08-19T14:34:22.702 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.702 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.703 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:34:22.703 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.435458+0000", 2024-08-19T14:34:22.703 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3068e-05 2024-08-19T14:34:22.703 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.703 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.703 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:34:22.703 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.448379+0000", 2024-08-19T14:34:22.703 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.012921804 2024-08-19T14:34:22.703 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:34:22.703 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:34:22.703 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:34:22.703 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.703 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.703 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.25813.0:92826 5.15 e445/442 5:a965286d:::benchmark_data_smithi045_85309_object92825:head v 445'5490, mlcod=423'262)", 2024-08-19T14:34:22.704 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:25:37.245988+0000", 2024-08-19T14:34:22.704 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 525.38077975299996, 2024-08-19T14:34:22.704 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20615012599999999, 2024-08-19T14:34:22.704 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:34:22.704 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:34:22.704 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:34:22.704 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:34:22.704 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.704 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:34:22.704 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.245988+0000", 2024-08-19T14:34:22.704 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:34:22.704 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.704 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.704 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:34:22.705 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.245988+0000", 2024-08-19T14:34:22.705 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:34:22.705 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.705 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.705 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:34:22.705 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.245989+0000", 2024-08-19T14:34:22.705 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0210000000000001e-06 2024-08-19T14:34:22.705 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.705 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.705 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:34:22.705 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.246023+0000", 2024-08-19T14:34:22.705 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.4044999999999997e-05 2024-08-19T14:34:22.705 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.706 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.706 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:34:22.706 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.422398+0000", 2024-08-19T14:34:22.706 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.17637508399999999 2024-08-19T14:34:22.706 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.706 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.706 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:34:22.706 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.422438+0000", 2024-08-19T14:34:22.706 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9362e-05 2024-08-19T14:34:22.706 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.706 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.706 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:34:22.706 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.426229+0000", 2024-08-19T14:34:22.707 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.003791141 2024-08-19T14:34:22.707 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.707 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.707 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:34:22.707 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.426253+0000", 2024-08-19T14:34:22.707 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4159e-05 2024-08-19T14:34:22.707 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.707 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.707 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:34:22.707 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.452061+0000", 2024-08-19T14:34:22.707 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.025807672 2024-08-19T14:34:22.707 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.707 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.708 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:34:22.708 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:25:37.452138+0000", 2024-08-19T14:34:22.708 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7642e-05 2024-08-19T14:34:22.708 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:34:22.708 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:34:22.708 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:34:22.708 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.708 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.708 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.26191.0:203064 6.e e589/566 6:77ee2f16:::benchmark_data_smithi045_97937_object95791:head v 589'12917, mlcod=589'12915)", 2024-08-19T14:34:22.708 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:32:55.253061+0000", 2024-08-19T14:34:22.708 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 87.373707284000005, 2024-08-19T14:34:22.708 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.226405574, 2024-08-19T14:34:22.708 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:34:22.709 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:34:22.709 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:34:22.709 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:34:22.709 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.709 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:34:22.709 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:55.253061+0000", 2024-08-19T14:34:22.709 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:34:22.709 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.709 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.709 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:34:22.709 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:55.253061+0000", 2024-08-19T14:34:22.709 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:34:22.709 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.709 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.710 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:34:22.710 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:55.253061+0000", 2024-08-19T14:34:22.710 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7999999999999995e-07 2024-08-19T14:34:22.710 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.710 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.710 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:34:22.710 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:55.253068+0000", 2024-08-19T14:34:22.710 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3169999999999998e-06 2024-08-19T14:34:22.710 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.710 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.710 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:34:22.710 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:55.253078+0000", 2024-08-19T14:34:22.710 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0774000000000001e-05 2024-08-19T14:34:22.711 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.711 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.711 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:34:22.711 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:55.253089+0000", 2024-08-19T14:34:22.711 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0376e-05 2024-08-19T14:34:22.711 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.711 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.711 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:34:22.711 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:55.253132+0000", 2024-08-19T14:34:22.711 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3343000000000002e-05 2024-08-19T14:34:22.711 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.711 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.711 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:34:22.712 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:55.253154+0000", 2024-08-19T14:34:22.712 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1909e-05 2024-08-19T14:34:22.712 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.712 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.712 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:34:22.712 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:55.479385+0000", 2024-08-19T14:34:22.712 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.22623082799999999 2024-08-19T14:34:22.712 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.712 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.712 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:34:22.712 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:55.479466+0000", 2024-08-19T14:34:22.712 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1347000000000006e-05 2024-08-19T14:34:22.712 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:34:22.713 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:34:22.713 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:34:22.713 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.713 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.713 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.26191.0:203033 6.6 e589/550 6:619821c3:::benchmark_data_smithi045_97937_object95760:head v 589'12695, mlcod=537'5976)", 2024-08-19T14:34:22.713 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:32:55.254199+0000", 2024-08-19T14:34:22.713 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 87.372568716999993, 2024-08-19T14:34:22.713 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.225314297, 2024-08-19T14:34:22.713 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:34:22.713 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:34:22.713 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:34:22.713 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:34:22.713 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.713 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:34:22.714 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:55.254199+0000", 2024-08-19T14:34:22.714 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:34:22.714 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.714 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.714 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:34:22.714 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:55.254199+0000", 2024-08-19T14:34:22.714 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:34:22.714 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.714 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.714 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:34:22.714 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:55.254200+0000", 2024-08-19T14:34:22.714 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0800000000000004e-07 2024-08-19T14:34:22.715 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.715 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.715 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:34:22.715 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:55.254206+0000", 2024-08-19T14:34:22.715 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5049999999999998e-06 2024-08-19T14:34:22.715 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.715 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.715 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:34:22.715 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:55.254215+0000", 2024-08-19T14:34:22.715 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3500000000000003e-06 2024-08-19T14:34:22.715 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.715 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.715 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:34:22.715 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:55.254221+0000", 2024-08-19T14:34:22.716 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8549999999999998e-06 2024-08-19T14:34:22.716 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.716 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.716 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:34:22.716 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:55.254278+0000", 2024-08-19T14:34:22.716 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7462e-05 2024-08-19T14:34:22.716 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.716 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.716 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:34:22.716 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:55.254301+0000", 2024-08-19T14:34:22.716 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2728e-05 2024-08-19T14:34:22.716 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.716 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.716 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:34:22.717 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:55.479473+0000", 2024-08-19T14:34:22.717 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.22517236600000001 2024-08-19T14:34:22.717 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.717 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.717 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:34:22.717 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:55.479514+0000", 2024-08-19T14:34:22.717 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0323e-05 2024-08-19T14:34:22.717 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:34:22.717 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:34:22.717 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:34:22.717 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.717 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.717 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.26191.0:203069 6.c e589/510 6:31ea2312:::benchmark_data_smithi045_97937_object95796:head v 589'12805, mlcod=589'12803)", 2024-08-19T14:34:22.718 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:32:55.259614+0000", 2024-08-19T14:34:22.718 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 87.367154232000004, 2024-08-19T14:34:22.718 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.22033929299999999, 2024-08-19T14:34:22.718 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:34:22.718 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:34:22.718 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:34:22.718 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:34:22.718 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.718 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:34:22.718 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:55.259614+0000", 2024-08-19T14:34:22.718 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:34:22.718 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.718 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.719 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:34:22.719 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:55.259614+0000", 2024-08-19T14:34:22.719 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:34:22.719 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.719 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.719 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:34:22.719 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:55.259615+0000", 2024-08-19T14:34:22.719 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.4900000000000004e-07 2024-08-19T14:34:22.719 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.719 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.719 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:34:22.719 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:55.259622+0000", 2024-08-19T14:34:22.719 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4710000000000001e-06 2024-08-19T14:34:22.719 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.719 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.720 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:34:22.720 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:55.259633+0000", 2024-08-19T14:34:22.720 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0995999999999999e-05 2024-08-19T14:34:22.720 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.720 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.720 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:34:22.720 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:55.259641+0000", 2024-08-19T14:34:22.720 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3889999999999999e-06 2024-08-19T14:34:22.720 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.720 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.720 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:34:22.720 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:55.259700+0000", 2024-08-19T14:34:22.720 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9704000000000003e-05 2024-08-19T14:34:22.721 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.721 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.721 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:34:22.721 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:55.259721+0000", 2024-08-19T14:34:22.721 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1070999999999999e-05 2024-08-19T14:34:22.721 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.721 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.721 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:34:22.721 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:55.479915+0000", 2024-08-19T14:34:22.721 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.22019383000000001 2024-08-19T14:34:22.721 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.721 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.721 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:34:22.721 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:55.479953+0000", 2024-08-19T14:34:22.721 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7883000000000003e-05 2024-08-19T14:34:22.722 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:34:22.722 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:34:22.722 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:34:22.722 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.722 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.722 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.26191.0:203071 6.e e589/566 6:77eae1e6:::benchmark_data_smithi045_97937_object95798:head v 589'12918, mlcod=589'12915)", 2024-08-19T14:34:22.722 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:32:55.259834+0000", 2024-08-19T14:34:22.722 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 87.366934411000003, 2024-08-19T14:34:22.722 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.22394059899999999, 2024-08-19T14:34:22.722 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:34:22.722 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:34:22.722 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:34:22.722 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:34:22.722 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.723 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:34:22.723 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:55.259834+0000", 2024-08-19T14:34:22.723 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:34:22.723 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.723 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.723 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:34:22.723 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:55.259834+0000", 2024-08-19T14:34:22.723 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:34:22.723 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.723 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.723 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:34:22.723 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:55.259834+0000", 2024-08-19T14:34:22.723 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9500000000000002e-07 2024-08-19T14:34:22.723 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.723 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.724 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:34:22.724 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:55.259841+0000", 2024-08-19T14:34:22.724 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9e-06 2024-08-19T14:34:22.724 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.724 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.724 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:34:22.724 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:55.259852+0000", 2024-08-19T14:34:22.724 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0951000000000001e-05 2024-08-19T14:34:22.724 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.724 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.724 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:34:22.724 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:55.259858+0000", 2024-08-19T14:34:22.724 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4450000000000001e-06 2024-08-19T14:34:22.724 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.725 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.725 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:34:22.725 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:55.259933+0000", 2024-08-19T14:34:22.725 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4079999999999995e-05 2024-08-19T14:34:22.725 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.725 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.725 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:34:22.725 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:55.259952+0000", 2024-08-19T14:34:22.725 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9599999999999999e-05 2024-08-19T14:34:22.725 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.725 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.725 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:34:22.725 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:55.483728+0000", 2024-08-19T14:34:22.725 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.223775469 2024-08-19T14:34:22.726 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.726 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.726 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:34:22.726 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:55.483774+0000", 2024-08-19T14:34:22.726 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6558999999999999e-05 2024-08-19T14:34:22.726 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:34:22.726 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:34:22.726 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:34:22.726 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.726 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.726 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.26191.0:203073 6.4 e589/510 6:2b68586f:::benchmark_data_smithi045_97937_object95800:head v 589'12780, mlcod=589'12778)", 2024-08-19T14:34:22.726 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:32:55.260032+0000", 2024-08-19T14:34:22.726 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 87.366736330999998, 2024-08-19T14:34:22.727 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.21997619099999999, 2024-08-19T14:34:22.727 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:34:22.727 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:34:22.727 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:34:22.727 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:34:22.727 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.727 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:34:22.727 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:55.260032+0000", 2024-08-19T14:34:22.727 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:34:22.727 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.727 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.727 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:34:22.727 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:55.260032+0000", 2024-08-19T14:34:22.727 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:34:22.727 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.728 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.728 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:34:22.728 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:55.260032+0000", 2024-08-19T14:34:22.728 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0900000000000001e-07 2024-08-19T14:34:22.728 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.728 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.728 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:34:22.728 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:55.260037+0000", 2024-08-19T14:34:22.728 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1250000000000001e-06 2024-08-19T14:34:22.728 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.728 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.728 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:34:22.728 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:55.260047+0000", 2024-08-19T14:34:22.728 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.1260000000000004e-06 2024-08-19T14:34:22.729 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.729 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.729 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:34:22.729 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:55.260052+0000", 2024-08-19T14:34:22.729 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6579999999999999e-06 2024-08-19T14:34:22.729 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.729 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.729 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:34:22.729 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:55.260106+0000", 2024-08-19T14:34:22.729 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4240000000000002e-05 2024-08-19T14:34:22.729 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.729 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.729 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:34:22.729 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:55.260125+0000", 2024-08-19T14:34:22.729 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8202e-05 2024-08-19T14:34:22.730 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.730 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.730 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:34:22.730 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:55.479956+0000", 2024-08-19T14:34:22.730 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.219830992 2024-08-19T14:34:22.730 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.730 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.730 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:34:22.730 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:55.480008+0000", 2024-08-19T14:34:22.730 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2238999999999999e-05 2024-08-19T14:34:22.730 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:34:22.731 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:34:22.732 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:34:22.732 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:34:22.732 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.732 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.732 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.26191.0:203074 6.7 e589/510 6:e6e35739:::benchmark_data_smithi045_97937_object95801:head v 589'12498, mlcod=589'12496)", 2024-08-19T14:34:22.732 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:32:55.262087+0000", 2024-08-19T14:34:22.732 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 87.364681130999998, 2024-08-19T14:34:22.732 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.22159626299999999, 2024-08-19T14:34:22.732 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:34:22.732 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:34:22.732 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:34:22.732 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:34:22.732 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.733 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:34:22.733 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:55.262087+0000", 2024-08-19T14:34:22.733 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:34:22.733 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.733 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.733 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:34:22.733 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:55.262087+0000", 2024-08-19T14:34:22.733 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:34:22.733 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.733 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.733 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:34:22.733 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:55.262088+0000", 2024-08-19T14:34:22.733 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.4e-07 2024-08-19T14:34:22.733 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.734 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.734 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:34:22.734 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:55.262093+0000", 2024-08-19T14:34:22.734 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6550000000000001e-06 2024-08-19T14:34:22.734 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.734 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.734 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:34:22.734 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:55.262102+0000", 2024-08-19T14:34:22.734 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6659999999999997e-06 2024-08-19T14:34:22.734 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.735 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.735 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:34:22.735 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:55.262107+0000", 2024-08-19T14:34:22.735 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9540000000000003e-06 2024-08-19T14:34:22.735 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.735 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.735 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:34:22.735 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:55.262151+0000", 2024-08-19T14:34:22.735 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3800000000000001e-05 2024-08-19T14:34:22.735 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.735 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.735 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:34:22.735 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:55.262164+0000", 2024-08-19T14:34:22.735 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.325e-05 2024-08-19T14:34:22.736 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.736 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.736 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:34:22.736 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:55.483646+0000", 2024-08-19T14:34:22.736 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.221482347 2024-08-19T14:34:22.736 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.736 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.736 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:34:22.736 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:55.483683+0000", 2024-08-19T14:34:22.736 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6751000000000001e-05 2024-08-19T14:34:22.736 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:34:22.736 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:34:22.736 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:34:22.736 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.736 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.737 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.26191.0:203072 6.e e589/566 6:7c289f49:::benchmark_data_smithi045_97937_object95799:head v 589'12919, mlcod=589'12915)", 2024-08-19T14:34:22.737 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:32:55.265039+0000", 2024-08-19T14:34:22.737 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 87.361729026999996, 2024-08-19T14:34:22.737 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.230644768, 2024-08-19T14:34:22.737 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:34:22.737 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:34:22.737 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:34:22.737 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:34:22.737 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.737 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:34:22.737 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:55.265039+0000", 2024-08-19T14:34:22.737 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:34:22.737 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.737 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.737 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:34:22.737 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:55.265039+0000", 2024-08-19T14:34:22.738 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:34:22.738 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.738 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.738 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:34:22.738 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:55.265040+0000", 2024-08-19T14:34:22.738 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.068e-06 2024-08-19T14:34:22.738 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.738 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.738 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:34:22.738 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:55.265047+0000", 2024-08-19T14:34:22.738 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3590000000000002e-06 2024-08-19T14:34:22.738 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.738 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.739 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:34:22.739 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:55.479365+0000", 2024-08-19T14:34:22.739 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.21431758100000001 2024-08-19T14:34:22.739 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.739 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.739 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:34:22.739 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:55.479380+0000", 2024-08-19T14:34:22.739 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5427999999999999e-05 2024-08-19T14:34:22.739 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.739 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.739 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:34:22.739 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:55.479505+0000", 2024-08-19T14:34:22.739 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012454999999999999 2024-08-19T14:34:22.739 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.739 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.739 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:34:22.740 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:55.479530+0000", 2024-08-19T14:34:22.740 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4688000000000001e-05 2024-08-19T14:34:22.740 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.740 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.740 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:34:22.740 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:55.495628+0000", 2024-08-19T14:34:22.740 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.016097920000000002 2024-08-19T14:34:22.740 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.740 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.740 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:34:22.740 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:55.495684+0000", 2024-08-19T14:34:22.740 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6174000000000001e-05 2024-08-19T14:34:22.740 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:34:22.740 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:34:22.741 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:34:22.741 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.741 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.741 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.26191.0:203076 6.a e589/566 6:5d08d9d9:::benchmark_data_smithi045_97937_object95803:head v 589'14213, mlcod=589'14211)", 2024-08-19T14:34:22.741 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:32:55.265348+0000", 2024-08-19T14:34:22.741 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 87.361419956000006, 2024-08-19T14:34:22.741 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.229606477, 2024-08-19T14:34:22.741 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:34:22.741 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:34:22.741 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:34:22.741 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:34:22.741 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.741 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:34:22.741 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:55.265348+0000", 2024-08-19T14:34:22.742 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:34:22.742 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.742 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.742 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:34:22.742 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:55.265348+0000", 2024-08-19T14:34:22.742 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:34:22.742 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.742 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.742 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:34:22.742 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:55.265349+0000", 2024-08-19T14:34:22.742 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4300000000000003e-07 2024-08-19T14:34:22.742 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.742 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.742 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:34:22.743 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:55.265354+0000", 2024-08-19T14:34:22.743 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4469999999999997e-06 2024-08-19T14:34:22.743 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.743 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.743 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:34:22.743 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:55.480038+0000", 2024-08-19T14:34:22.743 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.214683704 2024-08-19T14:34:22.743 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.743 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.743 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:34:22.743 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:55.480046+0000", 2024-08-19T14:34:22.743 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.3790000000000006e-06 2024-08-19T14:34:22.743 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.743 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.744 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:34:22.744 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:55.480096+0000", 2024-08-19T14:34:22.744 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9879000000000001e-05 2024-08-19T14:34:22.744 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.744 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.744 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:34:22.744 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:55.480117+0000", 2024-08-19T14:34:22.744 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0908999999999999e-05 2024-08-19T14:34:22.744 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.744 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.744 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:34:22.744 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:55.494868+0000", 2024-08-19T14:34:22.744 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.014751161 2024-08-19T14:34:22.744 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:34:22.744 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:34:22.745 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:34:22.745 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:55.494955+0000", 2024-08-19T14:34:22.745 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6354999999999996e-05 2024-08-19T14:34:22.745 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:34:22.745 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:34:22.745 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:34:22.745 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:34:22.745 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:34:22.745 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:34:22.770 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:22.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:22.771 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:34:22.872 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:34:22.897 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:22.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:22.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:34:22.999 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:34:23.023 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:23.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:23.023 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:34:23.124 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:34:23.147 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:23.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:23.148 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:34:23.249 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:34:23.281 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:23.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:23.282 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:34:23.383 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:34:23.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:23.410 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:23.410 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:34:23.444 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 50 15 84719 84704 105.871 59.5 0.0067369 0.00943915 2024-08-19T14:34:23.511 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:34:23.551 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:23.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:23.552 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:34:23.653 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:34:23.683 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:23.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:23.684 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:34:23.785 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:34:23.828 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:23.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:23.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:34:23.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:23 smithi031 ceph-mon[16336]: pgmap v1455: 47 pgs: 4 active+clean+remapped, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized, 5 active+undersized+degraded, 29 active+clean; 5.0 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 53 MiB/s wr, 851 op/s; 30987/244569 objects degraded (12.670%); 22560/244569 objects misplaced (9.224%); 8.1 MiB/s, 129 objects/s recovering 2024-08-19T14:34:23.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:23 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:34:23.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:23 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:34:23.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:23 smithi031 ceph-mon[16336]: osdmap e622: 12 total, 11 up, 10 in 2024-08-19T14:34:23.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:23 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 30987/244569 objects degraded (12.670%), 7 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-19T14:34:23.930 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:34:23.953 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:23.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:23.954 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:34:24.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:23 smithi032 ceph-mon[20004]: pgmap v1455: 47 pgs: 4 active+clean+remapped, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized, 5 active+undersized+degraded, 29 active+clean; 5.0 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 53 MiB/s wr, 851 op/s; 30987/244569 objects degraded (12.670%); 22560/244569 objects misplaced (9.224%); 8.1 MiB/s, 129 objects/s recovering 2024-08-19T14:34:24.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:23 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:34:24.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:23 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:34:24.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:23 smithi032 ceph-mon[20004]: osdmap e622: 12 total, 11 up, 10 in 2024-08-19T14:34:24.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:23 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 30987/244569 objects degraded (12.670%), 7 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-19T14:34:24.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:23 smithi045 ceph-mon[20143]: pgmap v1455: 47 pgs: 4 active+clean+remapped, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized, 5 active+undersized+degraded, 29 active+clean; 5.0 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 53 MiB/s wr, 851 op/s; 30987/244569 objects degraded (12.670%); 22560/244569 objects misplaced (9.224%); 8.1 MiB/s, 129 objects/s recovering 2024-08-19T14:34:24.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:23 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:34:24.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:23 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:34:24.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:23 smithi045 ceph-mon[20143]: osdmap e622: 12 total, 11 up, 10 in 2024-08-19T14:34:24.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:23 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 30987/244569 objects degraded (12.670%), 7 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-19T14:34:24.055 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:34:24.087 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:24.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:24.088 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:34:24.189 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:34:24.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:24.230 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:24.231 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:34:24.332 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:34:24.353 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:24.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:24.354 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:34:24.444 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 51 16 85486 85470 104.733 47.875 0.00650483 0.00954162 2024-08-19T14:34:24.455 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:34:24.491 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:24.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:24.491 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:34:24.592 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:34:24.613 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:24.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:24.614 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:34:24.716 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:34:24.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:24.748 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:24.748 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:34:24.849 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:34:24.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:24.887 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:24.887 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:34:24.988 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:34:25.034 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:25.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:25.036 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:34:25.039 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-08-19T14:34:25.136 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:34:25.160 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:25.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:25.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:34:25.262 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:34:25.290 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:25.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:25.291 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:34:25.346 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:34:25.392 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:34:25.416 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:34:25.416 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:25.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:25.417 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:34:25.444 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 52 16 86361 86345 103.771 54.6875 0.0434177 0.00962952 2024-08-19T14:34:25.460 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:34:25.518 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:34:25.541 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:25.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:25.542 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:34:25.643 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:34:25.664 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:25.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:25.665 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:34:25.766 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:34:25.803 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:25.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:25.804 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:34:25.905 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:34:25.939 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:25.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:25.940 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:34:26.041 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:34:26.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:25 smithi045 ceph-mon[20143]: pgmap v1457: 47 pgs: 4 active+clean+remapped, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized, 5 active+undersized+degraded, 29 active+clean; 5.1 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 50 MiB/s wr, 806 op/s; 30987/248742 objects degraded (12.457%); 22455/248742 objects misplaced (9.027%); 8.0 MiB/s, 128 objects/s recovering 2024-08-19T14:34:26.067 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:26.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:26.070 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:34:26.100 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:25 smithi032 ceph-mon[20004]: pgmap v1457: 47 pgs: 4 active+clean+remapped, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized, 5 active+undersized+degraded, 29 active+clean; 5.1 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 50 MiB/s wr, 806 op/s; 30987/248742 objects degraded (12.457%); 22455/248742 objects misplaced (9.027%); 8.0 MiB/s, 128 objects/s recovering 2024-08-19T14:34:26.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:25 smithi031 ceph-mon[16336]: pgmap v1457: 47 pgs: 4 active+clean+remapped, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized, 5 active+undersized+degraded, 29 active+clean; 5.1 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 50 MiB/s wr, 806 op/s; 30987/248742 objects degraded (12.457%); 22455/248742 objects misplaced (9.027%); 8.0 MiB/s, 128 objects/s recovering 2024-08-19T14:34:26.170 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:34:26.209 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:26.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:26.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:34:26.311 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:34:26.345 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:26.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:26.346 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:34:26.357 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:34:26.358 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-08-19T14:34:26.358 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-08-19T14:34:26.358 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:34:26.445 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 53 16 87205 87189 102.808 52.75 0.00851198 0.00972111 2024-08-19T14:34:26.447 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:34:26.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:26.479 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:26.479 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:34:26.580 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:34:26.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:26.613 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:26.613 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:34:26.714 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:34:26.757 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:26.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:26.758 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:34:26.859 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:34:26.884 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:26.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:26.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:34:26.986 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:34:27.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:27.021 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:27.021 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:34:27.122 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:34:27.149 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:27.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:27.151 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:34:27.159 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-08-19T14:34:27.251 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:34:27.285 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:27.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:27.286 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:34:27.387 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:34:27.400 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-19T14:34:27.415 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:27.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:27.416 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:34:27.445 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 54 16 88121 88105 101.964 57.25 0.00623146 0.00979982 2024-08-19T14:34:27.492 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:34:27.517 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:34:27.549 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:27.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:27.550 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:34:27.651 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:34:27.678 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:27.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:27.679 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:34:27.780 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:34:27.805 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:27.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:27.806 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:34:27.812 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:27 smithi032 ceph-mon[20004]: pgmap v1458: 47 pgs: 4 active+clean+remapped, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized, 5 active+undersized+degraded, 29 active+clean; 5.2 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 57 MiB/s wr, 909 op/s; 32368/256077 objects degraded (12.640%); 22208/256077 objects misplaced (8.672%); 7.4 MiB/s, 117 objects/s recovering 2024-08-19T14:34:27.907 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:34:27.932 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:27.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:27.933 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:34:28.034 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:34:28.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:27 smithi045 ceph-mon[20143]: pgmap v1458: 47 pgs: 4 active+clean+remapped, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized, 5 active+undersized+degraded, 29 active+clean; 5.2 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 57 MiB/s wr, 909 op/s; 32368/256077 objects degraded (12.640%); 22208/256077 objects misplaced (8.672%); 7.4 MiB/s, 117 objects/s recovering 2024-08-19T14:34:28.059 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:28.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:28.060 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:34:28.066 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:34:28.066 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-08-19T14:34:28.066 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-08-19T14:34:28.066 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-08-19T14:34:28.066 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:34:28.118 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:27 smithi031 ceph-mon[16336]: pgmap v1458: 47 pgs: 4 active+clean+remapped, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized, 5 active+undersized+degraded, 29 active+clean; 5.2 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 57 MiB/s wr, 909 op/s; 32368/256077 objects degraded (12.640%); 22208/256077 objects misplaced (8.672%); 7.4 MiB/s, 117 objects/s recovering 2024-08-19T14:34:28.160 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:34:28.182 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:28.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:28.183 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:34:28.284 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:34:28.308 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:28.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:28.309 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:34:28.410 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:34:28.445 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 55 16 88924 88908 101.023 50.1875 0.00762523 0.00989364 2024-08-19T14:34:28.447 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:28.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:28.448 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:34:28.550 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:34:28.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:28.574 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:28.574 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:34:28.675 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:34:28.712 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:28.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:28.713 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:34:28.761 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-08-19T14:34:28.814 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:34:28.847 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:28.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:28.848 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:34:28.949 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:34:28.977 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:28.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:28.978 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:34:29.009 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:28 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 32368/256077 objects degraded (12.640%), 7 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-19T14:34:29.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:28 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 32368/256077 objects degraded (12.640%), 7 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-19T14:34:29.046 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:34:29.079 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:34:29.107 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:28 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 32368/256077 objects degraded (12.640%), 7 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-19T14:34:29.113 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:29.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:29.114 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:34:29.215 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:34:29.238 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:29.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:29.239 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:34:29.286 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:34:29.287 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:34:29.324 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:34:29.324 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:34:29.324 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-08-19T14:34:29.341 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:34:29.355 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:34:29.356 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:34:29.364 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:29.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:29.365 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:34:29.383 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:34:29.383 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:34:29.421 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:34:29.422 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:34:29.445 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 56 16 89892 89876 100.299 60.5 0.0212189 0.00996504 2024-08-19T14:34:29.454 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:34:29.454 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:34:29.466 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:34:29.481 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:34:29.482 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:34:29.505 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:29.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:29.506 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:34:29.515 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:34:29.515 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:34:29.542 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:34:29.542 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:34:29.554 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:34:29.554 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:34:29.568 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:34:29.569 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:34:29.594 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:34:29.595 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:34:29.607 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:34:29.633 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:29.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:29.634 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:34:29.735 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:34:29.759 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:29.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:29.760 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:34:29.861 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:34:29.889 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:29.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:29.890 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:34:29.990 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:34:30.025 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:30.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:30.026 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:34:30.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:29 smithi045 ceph-mon[20143]: pgmap v1459: 47 pgs: 4 active+clean+remapped, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized, 5 active+undersized+degraded, 29 active+clean; 5.2 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 57 MiB/s wr, 910 op/s; 32634/257739 objects degraded (12.662%); 21774/257739 objects misplaced (8.448%); 9.6 MiB/s, 154 objects/s recovering 2024-08-19T14:34:30.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:29 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/811946427' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:34:30.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:29 smithi031 ceph-mon[16336]: pgmap v1459: 47 pgs: 4 active+clean+remapped, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized, 5 active+undersized+degraded, 29 active+clean; 5.2 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 57 MiB/s wr, 910 op/s; 32634/257739 objects degraded (12.662%); 21774/257739 objects misplaced (8.448%); 9.6 MiB/s, 154 objects/s recovering 2024-08-19T14:34:30.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:29 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/811946427' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:34:30.126 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:34:30.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:30.177 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:30.177 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:34:30.189 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:34:30.189 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:34:30.189 INFO:teuthology.orchestra.run.smithi031.stdout: "epoch": 623, 2024-08-19T14:34:30.189 INFO:teuthology.orchestra.run.smithi031.stdout: "fsid": "ed7d1c6e-5e33-11ef-bccf-c7b262605968", 2024-08-19T14:34:30.189 INFO:teuthology.orchestra.run.smithi031.stdout: "created": "2024-08-19T14:05:22.842169+0000", 2024-08-19T14:34:30.189 INFO:teuthology.orchestra.run.smithi031.stdout: "modified": "2024-08-19T14:34:29.767444+0000", 2024-08-19T14:34:30.190 INFO:teuthology.orchestra.run.smithi031.stdout: "last_up_change": "2024-08-19T14:31:02.021921+0000", 2024-08-19T14:34:30.190 INFO:teuthology.orchestra.run.smithi031.stdout: "last_in_change": "2024-08-19T14:34:11.676140+0000", 2024-08-19T14:34:30.190 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-08-19T14:34:30.190 INFO:teuthology.orchestra.run.smithi031.stdout: "flags_num": 5801984, 2024-08-19T14:34:30.190 INFO:teuthology.orchestra.run.smithi031.stdout: "flags_set": [ 2024-08-19T14:34:30.190 INFO:teuthology.orchestra.run.smithi031.stdout: "noscrub", 2024-08-19T14:34:30.190 INFO:teuthology.orchestra.run.smithi031.stdout: "pglog_hardlimit", 2024-08-19T14:34:30.190 INFO:teuthology.orchestra.run.smithi031.stdout: "purged_snapdirs", 2024-08-19T14:34:30.190 INFO:teuthology.orchestra.run.smithi031.stdout: "recovery_deletes", 2024-08-19T14:34:30.190 INFO:teuthology.orchestra.run.smithi031.stdout: "sortbitwise" 2024-08-19T14:34:30.190 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-08-19T14:34:30.190 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_version": 28, 2024-08-19T14:34:30.190 INFO:teuthology.orchestra.run.smithi031.stdout: "full_ratio": 0.94999998807907104, 2024-08-19T14:34:30.191 INFO:teuthology.orchestra.run.smithi031.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-08-19T14:34:30.191 INFO:teuthology.orchestra.run.smithi031.stdout: "nearfull_ratio": 0.85000002384185791, 2024-08-19T14:34:30.191 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_snapshot": "", 2024-08-19T14:34:30.191 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_max": 7, 2024-08-19T14:34:30.191 INFO:teuthology.orchestra.run.smithi031.stdout: "max_osd": 12, 2024-08-19T14:34:30.191 INFO:teuthology.orchestra.run.smithi031.stdout: "require_min_compat_client": "luminous", 2024-08-19T14:34:30.191 INFO:teuthology.orchestra.run.smithi031.stdout: "min_compat_client": "luminous", 2024-08-19T14:34:30.191 INFO:teuthology.orchestra.run.smithi031.stdout: "require_osd_release": "squid", 2024-08-19T14:34:30.191 INFO:teuthology.orchestra.run.smithi031.stdout: "allow_crimson": false, 2024-08-19T14:34:30.191 INFO:teuthology.orchestra.run.smithi031.stdout: "pools": [ 2024-08-19T14:34:30.191 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.191 INFO:teuthology.orchestra.run.smithi031.stdout: "pool": 1, 2024-08-19T14:34:30.191 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_name": ".mgr", 2024-08-19T14:34:30.191 INFO:teuthology.orchestra.run.smithi031.stdout: "create_time": "2024-08-19T14:08:15.966221+0000", 2024-08-19T14:34:30.192 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-08-19T14:34:30.192 INFO:teuthology.orchestra.run.smithi031.stdout: "flags_names": "hashpspool", 2024-08-19T14:34:30.192 INFO:teuthology.orchestra.run.smithi031.stdout: "type": 1, 2024-08-19T14:34:30.192 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 3, 2024-08-19T14:34:30.192 INFO:teuthology.orchestra.run.smithi031.stdout: "min_size": 2, 2024-08-19T14:34:30.192 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_rule": 0, 2024-08-19T14:34:30.192 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_count": 0, 2024-08-19T14:34:30.192 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_target": 0, 2024-08-19T14:34:30.192 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_barrier": 0, 2024-08-19T14:34:30.192 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-08-19T14:34:30.192 INFO:teuthology.orchestra.run.smithi031.stdout: "object_hash": 2, 2024-08-19T14:34:30.192 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_autoscale_mode": "off", 2024-08-19T14:34:30.192 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num": 31, 2024-08-19T14:34:30.193 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_placement_num": 31, 2024-08-19T14:34:30.193 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_placement_num_target": 31, 2024-08-19T14:34:30.193 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num_target": 31, 2024-08-19T14:34:30.193 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num_pending": 31, 2024-08-19T14:34:30.193 INFO:teuthology.orchestra.run.smithi031.stdout: "last_pg_merge_meta": { 2024-08-19T14:34:30.193 INFO:teuthology.orchestra.run.smithi031.stdout: "source_pgid": "0.0", 2024-08-19T14:34:30.193 INFO:teuthology.orchestra.run.smithi031.stdout: "ready_epoch": 0, 2024-08-19T14:34:30.193 INFO:teuthology.orchestra.run.smithi031.stdout: "last_epoch_started": 0, 2024-08-19T14:34:30.193 INFO:teuthology.orchestra.run.smithi031.stdout: "last_epoch_clean": 0, 2024-08-19T14:34:30.193 INFO:teuthology.orchestra.run.smithi031.stdout: "source_version": "0'0", 2024-08-19T14:34:30.193 INFO:teuthology.orchestra.run.smithi031.stdout: "target_version": "0'0" 2024-08-19T14:34:30.193 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.193 INFO:teuthology.orchestra.run.smithi031.stdout: "last_change": "278", 2024-08-19T14:34:30.193 INFO:teuthology.orchestra.run.smithi031.stdout: "last_force_op_resend": "0", 2024-08-19T14:34:30.193 INFO:teuthology.orchestra.run.smithi031.stdout: "last_force_op_resend_prenautilus": "0", 2024-08-19T14:34:30.194 INFO:teuthology.orchestra.run.smithi031.stdout: "last_force_op_resend_preluminous": "276", 2024-08-19T14:34:30.194 INFO:teuthology.orchestra.run.smithi031.stdout: "auid": 0, 2024-08-19T14:34:30.194 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_mode": "selfmanaged", 2024-08-19T14:34:30.194 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_seq": 0, 2024-08-19T14:34:30.194 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_epoch": 0, 2024-08-19T14:34:30.194 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_snaps": [], 2024-08-19T14:34:30.194 INFO:teuthology.orchestra.run.smithi031.stdout: "removed_snaps": "[]", 2024-08-19T14:34:30.194 INFO:teuthology.orchestra.run.smithi031.stdout: "quota_max_bytes": 0, 2024-08-19T14:34:30.194 INFO:teuthology.orchestra.run.smithi031.stdout: "quota_max_objects": 0, 2024-08-19T14:34:30.194 INFO:teuthology.orchestra.run.smithi031.stdout: "tiers": [], 2024-08-19T14:34:30.194 INFO:teuthology.orchestra.run.smithi031.stdout: "tier_of": -1, 2024-08-19T14:34:30.194 INFO:teuthology.orchestra.run.smithi031.stdout: "read_tier": -1, 2024-08-19T14:34:30.194 INFO:teuthology.orchestra.run.smithi031.stdout: "write_tier": -1, 2024-08-19T14:34:30.195 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_mode": "none", 2024-08-19T14:34:30.195 INFO:teuthology.orchestra.run.smithi031.stdout: "target_max_bytes": 0, 2024-08-19T14:34:30.195 INFO:teuthology.orchestra.run.smithi031.stdout: "target_max_objects": 0, 2024-08-19T14:34:30.195 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-08-19T14:34:30.195 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-08-19T14:34:30.195 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_target_full_ratio_micro": 800000, 2024-08-19T14:34:30.195 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_min_flush_age": 0, 2024-08-19T14:34:30.195 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_min_evict_age": 0, 2024-08-19T14:34:30.195 INFO:teuthology.orchestra.run.smithi031.stdout: "erasure_code_profile": "", 2024-08-19T14:34:30.195 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_params": { 2024-08-19T14:34:30.195 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "none" 2024-08-19T14:34:30.195 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.195 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_period": 0, 2024-08-19T14:34:30.195 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_count": 0, 2024-08-19T14:34:30.196 INFO:teuthology.orchestra.run.smithi031.stdout: "use_gmt_hitset": true, 2024-08-19T14:34:30.196 INFO:teuthology.orchestra.run.smithi031.stdout: "min_read_recency_for_promote": 0, 2024-08-19T14:34:30.196 INFO:teuthology.orchestra.run.smithi031.stdout: "min_write_recency_for_promote": 0, 2024-08-19T14:34:30.196 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_grade_decay_rate": 0, 2024-08-19T14:34:30.196 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_search_last_n": 0, 2024-08-19T14:34:30.196 INFO:teuthology.orchestra.run.smithi031.stdout: "grade_table": [], 2024-08-19T14:34:30.196 INFO:teuthology.orchestra.run.smithi031.stdout: "stripe_width": 0, 2024-08-19T14:34:30.196 INFO:teuthology.orchestra.run.smithi031.stdout: "expected_num_objects": 0, 2024-08-19T14:34:30.196 INFO:teuthology.orchestra.run.smithi031.stdout: "fast_read": false, 2024-08-19T14:34:30.196 INFO:teuthology.orchestra.run.smithi031.stdout: "options": { 2024-08-19T14:34:30.196 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num_max": 32, 2024-08-19T14:34:30.196 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num_min": 1 2024-08-19T14:34:30.196 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.196 INFO:teuthology.orchestra.run.smithi031.stdout: "application_metadata": { 2024-08-19T14:34:30.197 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": {} 2024-08-19T14:34:30.197 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.197 INFO:teuthology.orchestra.run.smithi031.stdout: "read_balance": { 2024-08-19T14:34:30.197 INFO:teuthology.orchestra.run.smithi031.stdout: "score_type": "Fair distribution", 2024-08-19T14:34:30.197 INFO:teuthology.orchestra.run.smithi031.stdout: "score_acting": 2.2599999904632568, 2024-08-19T14:34:30.197 INFO:teuthology.orchestra.run.smithi031.stdout: "score_stable": 2.2599999904632568, 2024-08-19T14:34:30.197 INFO:teuthology.orchestra.run.smithi031.stdout: "optimal_score": 0.89999997615814209, 2024-08-19T14:34:30.197 INFO:teuthology.orchestra.run.smithi031.stdout: "raw_score_acting": 2.0299999713897705, 2024-08-19T14:34:30.197 INFO:teuthology.orchestra.run.smithi031.stdout: "raw_score_stable": 2.0299999713897705, 2024-08-19T14:34:30.197 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity_weighted": 0.75614923238754272, 2024-08-19T14:34:30.197 INFO:teuthology.orchestra.run.smithi031.stdout: "average_primary_affinity": 0.9100000262260437, 2024-08-19T14:34:30.197 INFO:teuthology.orchestra.run.smithi031.stdout: "average_primary_affinity_weighted": 0.75999999046325684 2024-08-19T14:34:30.197 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:34:30.197 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.198 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.198 INFO:teuthology.orchestra.run.smithi031.stdout: "pool": 7, 2024-08-19T14:34:30.198 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_name": "unique_pool_5", 2024-08-19T14:34:30.198 INFO:teuthology.orchestra.run.smithi031.stdout: "create_time": "2024-08-19T14:33:26.611185+0000", 2024-08-19T14:34:30.198 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-08-19T14:34:30.198 INFO:teuthology.orchestra.run.smithi031.stdout: "flags_names": "hashpspool", 2024-08-19T14:34:30.198 INFO:teuthology.orchestra.run.smithi031.stdout: "type": 1, 2024-08-19T14:34:30.198 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 3, 2024-08-19T14:34:30.198 INFO:teuthology.orchestra.run.smithi031.stdout: "min_size": 2, 2024-08-19T14:34:30.198 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_rule": 0, 2024-08-19T14:34:30.198 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_count": 0, 2024-08-19T14:34:30.198 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_target": 0, 2024-08-19T14:34:30.198 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_barrier": 0, 2024-08-19T14:34:30.199 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-08-19T14:34:30.199 INFO:teuthology.orchestra.run.smithi031.stdout: "object_hash": 2, 2024-08-19T14:34:30.199 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_autoscale_mode": "off", 2024-08-19T14:34:30.199 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num": 16, 2024-08-19T14:34:30.199 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_placement_num": 16, 2024-08-19T14:34:30.199 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_placement_num_target": 16, 2024-08-19T14:34:30.199 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num_target": 16, 2024-08-19T14:34:30.199 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num_pending": 16, 2024-08-19T14:34:30.199 INFO:teuthology.orchestra.run.smithi031.stdout: "last_pg_merge_meta": { 2024-08-19T14:34:30.199 INFO:teuthology.orchestra.run.smithi031.stdout: "source_pgid": "0.0", 2024-08-19T14:34:30.199 INFO:teuthology.orchestra.run.smithi031.stdout: "ready_epoch": 0, 2024-08-19T14:34:30.199 INFO:teuthology.orchestra.run.smithi031.stdout: "last_epoch_started": 0, 2024-08-19T14:34:30.199 INFO:teuthology.orchestra.run.smithi031.stdout: "last_epoch_clean": 0, 2024-08-19T14:34:30.199 INFO:teuthology.orchestra.run.smithi031.stdout: "source_version": "0'0", 2024-08-19T14:34:30.200 INFO:teuthology.orchestra.run.smithi031.stdout: "target_version": "0'0" 2024-08-19T14:34:30.200 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.200 INFO:teuthology.orchestra.run.smithi031.stdout: "last_change": "604", 2024-08-19T14:34:30.200 INFO:teuthology.orchestra.run.smithi031.stdout: "last_force_op_resend": "0", 2024-08-19T14:34:30.200 INFO:teuthology.orchestra.run.smithi031.stdout: "last_force_op_resend_prenautilus": "0", 2024-08-19T14:34:30.200 INFO:teuthology.orchestra.run.smithi031.stdout: "last_force_op_resend_preluminous": "0", 2024-08-19T14:34:30.200 INFO:teuthology.orchestra.run.smithi031.stdout: "auid": 0, 2024-08-19T14:34:30.200 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_mode": "selfmanaged", 2024-08-19T14:34:30.200 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_seq": 0, 2024-08-19T14:34:30.200 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_epoch": 0, 2024-08-19T14:34:30.200 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_snaps": [], 2024-08-19T14:34:30.200 INFO:teuthology.orchestra.run.smithi031.stdout: "removed_snaps": "[]", 2024-08-19T14:34:30.200 INFO:teuthology.orchestra.run.smithi031.stdout: "quota_max_bytes": 0, 2024-08-19T14:34:30.200 INFO:teuthology.orchestra.run.smithi031.stdout: "quota_max_objects": 0, 2024-08-19T14:34:30.201 INFO:teuthology.orchestra.run.smithi031.stdout: "tiers": [], 2024-08-19T14:34:30.201 INFO:teuthology.orchestra.run.smithi031.stdout: "tier_of": -1, 2024-08-19T14:34:30.201 INFO:teuthology.orchestra.run.smithi031.stdout: "read_tier": -1, 2024-08-19T14:34:30.201 INFO:teuthology.orchestra.run.smithi031.stdout: "write_tier": -1, 2024-08-19T14:34:30.201 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_mode": "none", 2024-08-19T14:34:30.201 INFO:teuthology.orchestra.run.smithi031.stdout: "target_max_bytes": 0, 2024-08-19T14:34:30.201 INFO:teuthology.orchestra.run.smithi031.stdout: "target_max_objects": 0, 2024-08-19T14:34:30.201 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-08-19T14:34:30.201 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-08-19T14:34:30.201 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_target_full_ratio_micro": 800000, 2024-08-19T14:34:30.201 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_min_flush_age": 0, 2024-08-19T14:34:30.201 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_min_evict_age": 0, 2024-08-19T14:34:30.201 INFO:teuthology.orchestra.run.smithi031.stdout: "erasure_code_profile": "", 2024-08-19T14:34:30.201 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_params": { 2024-08-19T14:34:30.202 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "none" 2024-08-19T14:34:30.202 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.202 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_period": 0, 2024-08-19T14:34:30.202 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_count": 0, 2024-08-19T14:34:30.202 INFO:teuthology.orchestra.run.smithi031.stdout: "use_gmt_hitset": true, 2024-08-19T14:34:30.202 INFO:teuthology.orchestra.run.smithi031.stdout: "min_read_recency_for_promote": 0, 2024-08-19T14:34:30.202 INFO:teuthology.orchestra.run.smithi031.stdout: "min_write_recency_for_promote": 0, 2024-08-19T14:34:30.202 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_grade_decay_rate": 0, 2024-08-19T14:34:30.202 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_search_last_n": 0, 2024-08-19T14:34:30.202 INFO:teuthology.orchestra.run.smithi031.stdout: "grade_table": [], 2024-08-19T14:34:30.202 INFO:teuthology.orchestra.run.smithi031.stdout: "stripe_width": 0, 2024-08-19T14:34:30.202 INFO:teuthology.orchestra.run.smithi031.stdout: "expected_num_objects": 0, 2024-08-19T14:34:30.202 INFO:teuthology.orchestra.run.smithi031.stdout: "fast_read": false, 2024-08-19T14:34:30.202 INFO:teuthology.orchestra.run.smithi031.stdout: "options": {}, 2024-08-19T14:34:30.203 INFO:teuthology.orchestra.run.smithi031.stdout: "application_metadata": { 2024-08-19T14:34:30.203 INFO:teuthology.orchestra.run.smithi031.stdout: "rados": {} 2024-08-19T14:34:30.203 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.203 INFO:teuthology.orchestra.run.smithi031.stdout: "read_balance": { 2024-08-19T14:34:30.203 INFO:teuthology.orchestra.run.smithi031.stdout: "score_type": "Fair distribution", 2024-08-19T14:34:30.203 INFO:teuthology.orchestra.run.smithi031.stdout: "score_acting": 2.5, 2024-08-19T14:34:30.203 INFO:teuthology.orchestra.run.smithi031.stdout: "score_stable": 2.5, 2024-08-19T14:34:30.203 INFO:teuthology.orchestra.run.smithi031.stdout: "optimal_score": 0.89999997615814209, 2024-08-19T14:34:30.203 INFO:teuthology.orchestra.run.smithi031.stdout: "raw_score_acting": 2.25, 2024-08-19T14:34:30.203 INFO:teuthology.orchestra.run.smithi031.stdout: "raw_score_stable": 2.25, 2024-08-19T14:34:30.203 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity_weighted": 0.75614923238754272, 2024-08-19T14:34:30.203 INFO:teuthology.orchestra.run.smithi031.stdout: "average_primary_affinity": 0.9100000262260437, 2024-08-19T14:34:30.203 INFO:teuthology.orchestra.run.smithi031.stdout: "average_primary_affinity_weighted": 0.75999999046325684 2024-08-19T14:34:30.203 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:34:30.204 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:34:30.204 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-08-19T14:34:30.204 INFO:teuthology.orchestra.run.smithi031.stdout: "osds": [ 2024-08-19T14:34:30.204 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.204 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 0, 2024-08-19T14:34:30.204 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "d7a0fff3-23c6-4ee3-a275-a844862aabfb", 2024-08-19T14:34:30.204 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-08-19T14:34:30.204 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-08-19T14:34:30.204 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-08-19T14:34:30.204 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-08-19T14:34:30.204 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-08-19T14:34:30.204 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-08-19T14:34:30.204 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 8, 2024-08-19T14:34:30.204 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 619, 2024-08-19T14:34:30.205 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-08-19T14:34:30.205 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-08-19T14:34:30.205 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-08-19T14:34:30.205 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:34:30.205 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.205 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:34:30.205 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6802", 2024-08-19T14:34:30.205 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1108945097 2024-08-19T14:34:30.205 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.205 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.205 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:34:30.205 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6803", 2024-08-19T14:34:30.205 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1108945097 2024-08-19T14:34:30.205 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:34:30.205 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:34:30.206 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.206 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-08-19T14:34:30.206 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:34:30.206 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.206 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:34:30.206 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6804", 2024-08-19T14:34:30.206 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1108945097 2024-08-19T14:34:30.206 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.206 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.206 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:34:30.206 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6805", 2024-08-19T14:34:30.206 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1108945097 2024-08-19T14:34:30.206 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:34:30.207 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:34:30.207 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.207 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-08-19T14:34:30.207 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:34:30.207 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.207 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:34:30.207 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6808", 2024-08-19T14:34:30.207 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1108945097 2024-08-19T14:34:30.207 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.207 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.207 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:34:30.207 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6809", 2024-08-19T14:34:30.207 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1108945097 2024-08-19T14:34:30.208 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:34:30.208 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:34:30.208 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.208 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-08-19T14:34:30.208 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:34:30.208 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.208 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:34:30.208 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6806", 2024-08-19T14:34:30.208 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1108945097 2024-08-19T14:34:30.208 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.208 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.208 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:34:30.208 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6807", 2024-08-19T14:34:30.208 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1108945097 2024-08-19T14:34:30.209 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:34:30.209 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:34:30.209 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.209 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.31:6803/1108945097", 2024-08-19T14:34:30.209 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.31:6805/1108945097", 2024-08-19T14:34:30.209 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.31:6809/1108945097", 2024-08-19T14:34:30.209 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.31:6807/1108945097", 2024-08-19T14:34:30.209 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-08-19T14:34:30.209 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-08-19T14:34:30.209 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-08-19T14:34:30.209 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:34:30.209 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.209 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.210 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 1, 2024-08-19T14:34:30.210 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "145d5bb4-74dd-440f-bcc9-7e9ebda9c5d6", 2024-08-19T14:34:30.210 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-08-19T14:34:30.210 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-08-19T14:34:30.210 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-08-19T14:34:30.210 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-08-19T14:34:30.210 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-08-19T14:34:30.210 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-08-19T14:34:30.210 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 13, 2024-08-19T14:34:30.210 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 617, 2024-08-19T14:34:30.210 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-08-19T14:34:30.210 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-08-19T14:34:30.210 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-08-19T14:34:30.210 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:34:30.211 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.211 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:34:30.211 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6810", 2024-08-19T14:34:30.211 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3236186806 2024-08-19T14:34:30.211 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.211 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.211 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:34:30.211 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6811", 2024-08-19T14:34:30.211 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3236186806 2024-08-19T14:34:30.211 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:34:30.211 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:34:30.211 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.211 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-08-19T14:34:30.212 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:34:30.212 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.212 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:34:30.212 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6812", 2024-08-19T14:34:30.212 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3236186806 2024-08-19T14:34:30.212 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.212 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.212 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:34:30.212 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6813", 2024-08-19T14:34:30.212 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3236186806 2024-08-19T14:34:30.212 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:34:30.212 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:34:30.212 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.212 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-08-19T14:34:30.213 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:34:30.213 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.213 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:34:30.213 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6816", 2024-08-19T14:34:30.213 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3236186806 2024-08-19T14:34:30.213 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.213 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.213 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:34:30.213 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6817", 2024-08-19T14:34:30.213 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3236186806 2024-08-19T14:34:30.213 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:34:30.213 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:34:30.213 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.213 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-08-19T14:34:30.214 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:34:30.214 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.214 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:34:30.214 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6814", 2024-08-19T14:34:30.214 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3236186806 2024-08-19T14:34:30.214 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.214 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.214 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:34:30.214 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6815", 2024-08-19T14:34:30.214 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3236186806 2024-08-19T14:34:30.214 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:34:30.214 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:34:30.214 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.214 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.31:6811/3236186806", 2024-08-19T14:34:30.215 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.31:6813/3236186806", 2024-08-19T14:34:30.215 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.31:6817/3236186806", 2024-08-19T14:34:30.215 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.31:6815/3236186806", 2024-08-19T14:34:30.215 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-08-19T14:34:30.215 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-08-19T14:34:30.215 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-08-19T14:34:30.215 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:34:30.215 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.215 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.215 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 2, 2024-08-19T14:34:30.215 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "376c18d0-4768-46a0-bb60-e79c88ca909c", 2024-08-19T14:34:30.215 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 0, 2024-08-19T14:34:30.215 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-08-19T14:34:30.215 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-08-19T14:34:30.216 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 0.07379150390625, 2024-08-19T14:34:30.216 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-08-19T14:34:30.216 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-08-19T14:34:30.216 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 18, 2024-08-19T14:34:30.216 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 204, 2024-08-19T14:34:30.216 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 565, 2024-08-19T14:34:30.216 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-08-19T14:34:30.216 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-08-19T14:34:30.216 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:34:30.216 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.216 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:34:30.216 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6818", 2024-08-19T14:34:30.216 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 4045553387 2024-08-19T14:34:30.216 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.217 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.217 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:34:30.217 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6819", 2024-08-19T14:34:30.217 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 4045553387 2024-08-19T14:34:30.217 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:34:30.217 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:34:30.217 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.217 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-08-19T14:34:30.217 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:34:30.217 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.217 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:34:30.217 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6820", 2024-08-19T14:34:30.217 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 4045553387 2024-08-19T14:34:30.218 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.218 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.218 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:34:30.218 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6821", 2024-08-19T14:34:30.218 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 4045553387 2024-08-19T14:34:30.218 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:34:30.218 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:34:30.218 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.218 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-08-19T14:34:30.218 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:34:30.218 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.218 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:34:30.218 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6824", 2024-08-19T14:34:30.218 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 4045553387 2024-08-19T14:34:30.219 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.219 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.219 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:34:30.219 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6825", 2024-08-19T14:34:30.219 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 4045553387 2024-08-19T14:34:30.219 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:34:30.219 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:34:30.219 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.219 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-08-19T14:34:30.219 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:34:30.219 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.219 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:34:30.219 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6822", 2024-08-19T14:34:30.220 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 4045553387 2024-08-19T14:34:30.220 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.220 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.220 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:34:30.220 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6823", 2024-08-19T14:34:30.220 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 4045553387 2024-08-19T14:34:30.220 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:34:30.220 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:34:30.220 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.220 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.31:6819/4045553387", 2024-08-19T14:34:30.220 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.31:6821/4045553387", 2024-08-19T14:34:30.220 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.31:6825/4045553387", 2024-08-19T14:34:30.220 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.31:6823/4045553387", 2024-08-19T14:34:30.220 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-08-19T14:34:30.220 INFO:teuthology.orchestra.run.smithi031.stdout: "exists" 2024-08-19T14:34:30.221 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:34:30.221 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.221 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.221 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 3, 2024-08-19T14:34:30.221 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "a787ea9f-b9ba-450f-b388-614cfba90634", 2024-08-19T14:34:30.221 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-08-19T14:34:30.221 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-08-19T14:34:30.221 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-08-19T14:34:30.221 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-08-19T14:34:30.221 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-08-19T14:34:30.221 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-08-19T14:34:30.221 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 26, 2024-08-19T14:34:30.221 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 619, 2024-08-19T14:34:30.221 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-08-19T14:34:30.222 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-08-19T14:34:30.222 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-08-19T14:34:30.222 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:34:30.222 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.222 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:34:30.222 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6826", 2024-08-19T14:34:30.222 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2288082716 2024-08-19T14:34:30.222 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.222 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.222 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:34:30.222 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6827", 2024-08-19T14:34:30.222 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2288082716 2024-08-19T14:34:30.222 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:34:30.222 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:34:30.223 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.223 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-08-19T14:34:30.223 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:34:30.223 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.223 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:34:30.223 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6828", 2024-08-19T14:34:30.223 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2288082716 2024-08-19T14:34:30.223 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.223 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.223 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:34:30.223 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6829", 2024-08-19T14:34:30.223 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2288082716 2024-08-19T14:34:30.223 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:34:30.223 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:34:30.224 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.224 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-08-19T14:34:30.224 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:34:30.224 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.224 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:34:30.224 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6832", 2024-08-19T14:34:30.224 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2288082716 2024-08-19T14:34:30.224 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.225 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:29 smithi032 ceph-mon[20004]: pgmap v1459: 47 pgs: 4 active+clean+remapped, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized, 5 active+undersized+degraded, 29 active+clean; 5.2 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 57 MiB/s wr, 910 op/s; 32634/257739 objects degraded (12.662%); 21774/257739 objects misplaced (8.448%); 9.6 MiB/s, 154 objects/s recovering 2024-08-19T14:34:30.225 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:29 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/811946427' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:34:30.225 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.225 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:34:30.225 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6833", 2024-08-19T14:34:30.225 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2288082716 2024-08-19T14:34:30.225 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:34:30.225 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:34:30.226 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.226 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-08-19T14:34:30.226 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:34:30.226 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.226 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:34:30.226 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6830", 2024-08-19T14:34:30.226 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2288082716 2024-08-19T14:34:30.226 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.226 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.226 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:34:30.226 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6831", 2024-08-19T14:34:30.226 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2288082716 2024-08-19T14:34:30.226 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:34:30.227 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:34:30.227 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.227 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.31:6827/2288082716", 2024-08-19T14:34:30.227 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.31:6829/2288082716", 2024-08-19T14:34:30.227 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.31:6833/2288082716", 2024-08-19T14:34:30.227 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.31:6831/2288082716", 2024-08-19T14:34:30.227 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-08-19T14:34:30.227 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-08-19T14:34:30.227 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-08-19T14:34:30.227 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:34:30.227 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.227 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.227 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 4, 2024-08-19T14:34:30.227 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "7859751a-12f5-4a57-a332-435360479d53", 2024-08-19T14:34:30.228 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-08-19T14:34:30.228 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-08-19T14:34:30.228 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-08-19T14:34:30.228 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-08-19T14:34:30.228 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-08-19T14:34:30.228 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-08-19T14:34:30.228 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 31, 2024-08-19T14:34:30.228 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 619, 2024-08-19T14:34:30.228 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-08-19T14:34:30.228 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-08-19T14:34:30.228 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-08-19T14:34:30.228 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:34:30.228 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.228 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:34:30.229 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6800", 2024-08-19T14:34:30.229 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3934739984 2024-08-19T14:34:30.229 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.229 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.229 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:34:30.229 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6801", 2024-08-19T14:34:30.229 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3934739984 2024-08-19T14:34:30.229 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:34:30.229 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:34:30.229 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.229 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-08-19T14:34:30.229 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:34:30.229 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.229 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:34:30.230 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6802", 2024-08-19T14:34:30.230 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3934739984 2024-08-19T14:34:30.230 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.230 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.230 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:34:30.230 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6803", 2024-08-19T14:34:30.230 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3934739984 2024-08-19T14:34:30.230 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:34:30.230 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:34:30.230 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.230 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-08-19T14:34:30.230 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:34:30.230 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.231 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:34:30.231 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6806", 2024-08-19T14:34:30.231 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3934739984 2024-08-19T14:34:30.231 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.231 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.231 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:34:30.231 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6807", 2024-08-19T14:34:30.231 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3934739984 2024-08-19T14:34:30.231 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:34:30.231 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:34:30.231 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.231 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-08-19T14:34:30.231 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:34:30.232 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.232 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:34:30.232 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6804", 2024-08-19T14:34:30.232 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3934739984 2024-08-19T14:34:30.232 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.232 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.232 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:34:30.232 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6805", 2024-08-19T14:34:30.232 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3934739984 2024-08-19T14:34:30.232 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:34:30.232 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:34:30.232 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.232 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.32:6801/3934739984", 2024-08-19T14:34:30.232 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.32:6803/3934739984", 2024-08-19T14:34:30.232 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.32:6807/3934739984", 2024-08-19T14:34:30.233 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.32:6805/3934739984", 2024-08-19T14:34:30.233 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-08-19T14:34:30.233 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-08-19T14:34:30.233 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-08-19T14:34:30.233 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:34:30.233 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.233 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.233 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 5, 2024-08-19T14:34:30.233 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "017f3eef-ac60-4b82-8350-919017cb4ccd", 2024-08-19T14:34:30.233 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-08-19T14:34:30.233 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-08-19T14:34:30.233 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-08-19T14:34:30.233 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 0, 2024-08-19T14:34:30.234 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-08-19T14:34:30.234 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-08-19T14:34:30.234 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 40, 2024-08-19T14:34:30.234 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 464, 2024-08-19T14:34:30.234 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-08-19T14:34:30.234 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-08-19T14:34:30.234 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-08-19T14:34:30.234 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:34:30.234 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.234 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:34:30.234 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6808", 2024-08-19T14:34:30.234 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 526388985 2024-08-19T14:34:30.234 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.235 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.235 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:34:30.235 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6809", 2024-08-19T14:34:30.235 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 526388985 2024-08-19T14:34:30.235 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:34:30.235 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:34:30.235 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.235 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-08-19T14:34:30.235 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:34:30.235 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.235 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:34:30.235 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6810", 2024-08-19T14:34:30.235 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 526388985 2024-08-19T14:34:30.235 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.236 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.236 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:34:30.236 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6811", 2024-08-19T14:34:30.236 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 526388985 2024-08-19T14:34:30.236 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:34:30.236 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:34:30.236 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.236 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-08-19T14:34:30.236 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:34:30.236 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.236 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:34:30.236 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6814", 2024-08-19T14:34:30.236 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 526388985 2024-08-19T14:34:30.237 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.237 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.237 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:34:30.237 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6815", 2024-08-19T14:34:30.237 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 526388985 2024-08-19T14:34:30.237 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:34:30.237 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:34:30.237 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.237 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-08-19T14:34:30.237 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:34:30.237 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.237 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:34:30.237 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6812", 2024-08-19T14:34:30.237 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 526388985 2024-08-19T14:34:30.238 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.238 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.238 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:34:30.238 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6813", 2024-08-19T14:34:30.238 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 526388985 2024-08-19T14:34:30.238 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:34:30.238 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:34:30.238 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.238 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.32:6809/526388985", 2024-08-19T14:34:30.238 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.32:6811/526388985", 2024-08-19T14:34:30.238 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.32:6815/526388985", 2024-08-19T14:34:30.238 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.32:6813/526388985", 2024-08-19T14:34:30.238 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-08-19T14:34:30.238 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-08-19T14:34:30.239 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-08-19T14:34:30.239 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:34:30.239 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.239 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.239 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 6, 2024-08-19T14:34:30.239 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "44aa8229-0268-49d7-bd57-3bf1f384e5e6", 2024-08-19T14:34:30.239 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-08-19T14:34:30.239 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 0, 2024-08-19T14:34:30.239 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 0, 2024-08-19T14:34:30.239 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-08-19T14:34:30.239 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-08-19T14:34:30.239 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-08-19T14:34:30.239 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 48, 2024-08-19T14:34:30.239 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 126, 2024-08-19T14:34:30.240 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-08-19T14:34:30.240 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-08-19T14:34:30.240 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-08-19T14:34:30.240 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:34:30.240 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.240 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:34:30.240 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6816", 2024-08-19T14:34:30.240 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1131035844 2024-08-19T14:34:30.240 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.240 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.240 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:34:30.240 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6817", 2024-08-19T14:34:30.240 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1131035844 2024-08-19T14:34:30.240 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:34:30.240 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:34:30.241 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.241 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-08-19T14:34:30.241 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:34:30.241 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.241 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:34:30.241 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6818", 2024-08-19T14:34:30.241 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1131035844 2024-08-19T14:34:30.241 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.241 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.241 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:34:30.241 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6819", 2024-08-19T14:34:30.241 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1131035844 2024-08-19T14:34:30.241 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:34:30.241 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:34:30.242 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.242 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-08-19T14:34:30.242 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:34:30.242 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.242 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:34:30.242 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6822", 2024-08-19T14:34:30.242 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1131035844 2024-08-19T14:34:30.242 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.242 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.242 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:34:30.242 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6823", 2024-08-19T14:34:30.242 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1131035844 2024-08-19T14:34:30.242 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:34:30.242 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:34:30.243 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.243 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-08-19T14:34:30.243 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:34:30.243 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.243 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:34:30.243 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6820", 2024-08-19T14:34:30.243 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1131035844 2024-08-19T14:34:30.243 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.243 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.243 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:34:30.243 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6821", 2024-08-19T14:34:30.243 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1131035844 2024-08-19T14:34:30.243 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:34:30.243 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:34:30.244 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.244 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.32:6817/1131035844", 2024-08-19T14:34:30.244 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.32:6819/1131035844", 2024-08-19T14:34:30.244 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.32:6823/1131035844", 2024-08-19T14:34:30.244 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.32:6821/1131035844", 2024-08-19T14:34:30.244 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-08-19T14:34:30.244 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-08-19T14:34:30.244 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-08-19T14:34:30.244 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:34:30.244 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.244 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.244 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 7, 2024-08-19T14:34:30.244 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "263433e5-70b6-4192-86ff-dc2ad95927eb", 2024-08-19T14:34:30.244 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-08-19T14:34:30.245 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-08-19T14:34:30.245 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-08-19T14:34:30.245 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-08-19T14:34:30.245 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-08-19T14:34:30.245 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-08-19T14:34:30.245 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 57, 2024-08-19T14:34:30.245 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 619, 2024-08-19T14:34:30.245 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-08-19T14:34:30.245 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-08-19T14:34:30.245 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-08-19T14:34:30.245 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:34:30.245 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.245 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:34:30.245 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6824", 2024-08-19T14:34:30.246 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2014939947 2024-08-19T14:34:30.246 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.246 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.246 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:34:30.246 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6825", 2024-08-19T14:34:30.246 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2014939947 2024-08-19T14:34:30.246 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:34:30.246 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:34:30.246 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.246 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-08-19T14:34:30.246 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:34:30.246 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.246 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:34:30.246 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6826", 2024-08-19T14:34:30.247 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2014939947 2024-08-19T14:34:30.247 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.247 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.247 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:34:30.247 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6827", 2024-08-19T14:34:30.247 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2014939947 2024-08-19T14:34:30.247 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:34:30.247 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:34:30.247 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.247 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-08-19T14:34:30.247 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:34:30.247 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.247 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:34:30.247 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6830", 2024-08-19T14:34:30.248 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2014939947 2024-08-19T14:34:30.248 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.248 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.248 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:34:30.248 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6831", 2024-08-19T14:34:30.248 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2014939947 2024-08-19T14:34:30.248 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:34:30.248 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:34:30.248 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.248 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-08-19T14:34:30.248 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:34:30.248 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.248 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:34:30.248 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6828", 2024-08-19T14:34:30.249 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2014939947 2024-08-19T14:34:30.249 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.249 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.249 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:34:30.249 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6829", 2024-08-19T14:34:30.249 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2014939947 2024-08-19T14:34:30.249 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:34:30.249 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:34:30.249 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.249 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.32:6825/2014939947", 2024-08-19T14:34:30.249 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.32:6827/2014939947", 2024-08-19T14:34:30.249 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.32:6831/2014939947", 2024-08-19T14:34:30.249 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.32:6829/2014939947", 2024-08-19T14:34:30.249 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-08-19T14:34:30.249 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-08-19T14:34:30.250 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-08-19T14:34:30.250 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:34:30.250 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.250 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.250 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 8, 2024-08-19T14:34:30.250 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "2bbcad36-0c74-4528-9fb9-a8812afa1662", 2024-08-19T14:34:30.250 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-08-19T14:34:30.250 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-08-19T14:34:30.250 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-08-19T14:34:30.250 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-08-19T14:34:30.250 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-08-19T14:34:30.250 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-08-19T14:34:30.250 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 65, 2024-08-19T14:34:30.251 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 619, 2024-08-19T14:34:30.251 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-08-19T14:34:30.251 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-08-19T14:34:30.251 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-08-19T14:34:30.251 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:34:30.251 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.251 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:34:30.251 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6800", 2024-08-19T14:34:30.251 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1445140892 2024-08-19T14:34:30.251 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.251 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.251 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:34:30.251 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6801", 2024-08-19T14:34:30.251 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1445140892 2024-08-19T14:34:30.252 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:34:30.252 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:34:30.252 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.252 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-08-19T14:34:30.252 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:34:30.252 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.252 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:34:30.252 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6802", 2024-08-19T14:34:30.252 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1445140892 2024-08-19T14:34:30.252 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.253 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.253 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:34:30.253 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6803", 2024-08-19T14:34:30.253 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1445140892 2024-08-19T14:34:30.253 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:34:30.253 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:34:30.253 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.253 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-08-19T14:34:30.253 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:34:30.253 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.253 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:34:30.253 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6806", 2024-08-19T14:34:30.253 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1445140892 2024-08-19T14:34:30.254 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.254 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.254 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:34:30.254 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6807", 2024-08-19T14:34:30.254 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1445140892 2024-08-19T14:34:30.254 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:34:30.254 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:34:30.254 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.254 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-08-19T14:34:30.254 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:34:30.254 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.254 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:34:30.254 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6804", 2024-08-19T14:34:30.255 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1445140892 2024-08-19T14:34:30.255 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.255 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.255 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:34:30.255 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6805", 2024-08-19T14:34:30.255 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1445140892 2024-08-19T14:34:30.255 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:34:30.255 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:34:30.255 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.255 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.45:6801/1445140892", 2024-08-19T14:34:30.255 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.45:6803/1445140892", 2024-08-19T14:34:30.255 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.45:6807/1445140892", 2024-08-19T14:34:30.255 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.45:6805/1445140892", 2024-08-19T14:34:30.256 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-08-19T14:34:30.256 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-08-19T14:34:30.256 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-08-19T14:34:30.256 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:34:30.256 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.256 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.256 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 9, 2024-08-19T14:34:30.256 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "c8ee4277-3bf9-46bf-9cc0-152c1c4b9508", 2024-08-19T14:34:30.256 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-08-19T14:34:30.256 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-08-19T14:34:30.256 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-08-19T14:34:30.256 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 0, 2024-08-19T14:34:30.256 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-08-19T14:34:30.257 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-08-19T14:34:30.257 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 74, 2024-08-19T14:34:30.257 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 616, 2024-08-19T14:34:30.257 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-08-19T14:34:30.257 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-08-19T14:34:30.257 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-08-19T14:34:30.257 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:34:30.257 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.257 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:34:30.257 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6808", 2024-08-19T14:34:30.257 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1213787169 2024-08-19T14:34:30.257 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.257 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.258 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:34:30.258 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6809", 2024-08-19T14:34:30.258 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1213787169 2024-08-19T14:34:30.258 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:34:30.258 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:34:30.258 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.258 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-08-19T14:34:30.258 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:34:30.258 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.258 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:34:30.258 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6810", 2024-08-19T14:34:30.258 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1213787169 2024-08-19T14:34:30.258 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.258 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.259 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:34:30.259 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6811", 2024-08-19T14:34:30.259 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1213787169 2024-08-19T14:34:30.259 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:34:30.259 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:34:30.259 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.259 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-08-19T14:34:30.259 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:34:30.259 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.259 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:34:30.259 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6814", 2024-08-19T14:34:30.259 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1213787169 2024-08-19T14:34:30.259 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.260 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.260 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:34:30.260 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6815", 2024-08-19T14:34:30.260 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1213787169 2024-08-19T14:34:30.260 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:34:30.260 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:34:30.260 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.260 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-08-19T14:34:30.260 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:34:30.260 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.260 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:34:30.260 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6812", 2024-08-19T14:34:30.260 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1213787169 2024-08-19T14:34:30.260 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.261 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.261 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:34:30.261 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6813", 2024-08-19T14:34:30.261 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1213787169 2024-08-19T14:34:30.261 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:34:30.261 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:34:30.261 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.261 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.45:6809/1213787169", 2024-08-19T14:34:30.261 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.45:6811/1213787169", 2024-08-19T14:34:30.261 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.45:6815/1213787169", 2024-08-19T14:34:30.261 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.45:6813/1213787169", 2024-08-19T14:34:30.261 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-08-19T14:34:30.261 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-08-19T14:34:30.261 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-08-19T14:34:30.262 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:34:30.262 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.262 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.262 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 10, 2024-08-19T14:34:30.262 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "29031395-5f5c-40f8-ac90-9f6ac48e5c9f", 2024-08-19T14:34:30.262 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-08-19T14:34:30.262 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 0, 2024-08-19T14:34:30.263 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 0, 2024-08-19T14:34:30.263 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-08-19T14:34:30.263 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 80, 2024-08-19T14:34:30.263 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 529, 2024-08-19T14:34:30.263 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 539, 2024-08-19T14:34:30.264 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 115, 2024-08-19T14:34:30.264 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 530, 2024-08-19T14:34:30.264 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-08-19T14:34:30.264 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-08-19T14:34:30.264 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:34:30.264 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.264 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:34:30.264 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6816", 2024-08-19T14:34:30.264 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1069743893 2024-08-19T14:34:30.264 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.264 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.264 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:34:30.264 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6817", 2024-08-19T14:34:30.265 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1069743893 2024-08-19T14:34:30.265 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:34:30.265 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:34:30.265 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.265 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-08-19T14:34:30.265 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:34:30.265 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.265 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:34:30.265 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6818", 2024-08-19T14:34:30.265 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1069743893 2024-08-19T14:34:30.265 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.265 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.265 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:34:30.265 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6819", 2024-08-19T14:34:30.266 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1069743893 2024-08-19T14:34:30.266 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:34:30.266 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:34:30.266 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.266 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-08-19T14:34:30.266 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:34:30.266 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.266 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:34:30.266 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6822", 2024-08-19T14:34:30.266 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1069743893 2024-08-19T14:34:30.266 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.266 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.266 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:34:30.267 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6823", 2024-08-19T14:34:30.267 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1069743893 2024-08-19T14:34:30.267 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:34:30.267 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:34:30.267 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.267 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-08-19T14:34:30.267 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:34:30.267 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.267 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:34:30.267 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6820", 2024-08-19T14:34:30.267 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1069743893 2024-08-19T14:34:30.267 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.267 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.267 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:34:30.268 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6821", 2024-08-19T14:34:30.268 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1069743893 2024-08-19T14:34:30.268 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:34:30.268 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:34:30.268 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.268 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.45:6817/1069743893", 2024-08-19T14:34:30.268 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.45:6819/1069743893", 2024-08-19T14:34:30.268 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.45:6823/1069743893", 2024-08-19T14:34:30.268 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.45:6821/1069743893", 2024-08-19T14:34:30.268 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-08-19T14:34:30.268 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-08-19T14:34:30.268 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-08-19T14:34:30.268 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:34:30.268 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.269 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.269 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 11, 2024-08-19T14:34:30.269 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "5ef0db4b-0fef-4ea0-a19b-d3677e082c29", 2024-08-19T14:34:30.269 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-08-19T14:34:30.269 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-08-19T14:34:30.269 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-08-19T14:34:30.269 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-08-19T14:34:30.269 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-08-19T14:34:30.269 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-08-19T14:34:30.269 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 86, 2024-08-19T14:34:30.269 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 617, 2024-08-19T14:34:30.269 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-08-19T14:34:30.269 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-08-19T14:34:30.270 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-08-19T14:34:30.270 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:34:30.270 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.270 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:34:30.270 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6824", 2024-08-19T14:34:30.270 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2438396298 2024-08-19T14:34:30.270 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.270 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.270 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:34:30.270 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6825", 2024-08-19T14:34:30.270 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2438396298 2024-08-19T14:34:30.270 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:34:30.270 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:34:30.270 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.271 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-08-19T14:34:30.271 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:34:30.271 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.271 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:34:30.271 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6826", 2024-08-19T14:34:30.271 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2438396298 2024-08-19T14:34:30.271 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.271 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.271 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:34:30.271 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6827", 2024-08-19T14:34:30.271 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2438396298 2024-08-19T14:34:30.271 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:34:30.271 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:34:30.272 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.272 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-08-19T14:34:30.272 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:34:30.272 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.272 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:34:30.272 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6830", 2024-08-19T14:34:30.272 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2438396298 2024-08-19T14:34:30.272 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.272 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.272 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:34:30.272 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6831", 2024-08-19T14:34:30.272 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2438396298 2024-08-19T14:34:30.272 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:34:30.272 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:34:30.273 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.273 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-08-19T14:34:30.273 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:34:30.273 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.273 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:34:30.273 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6828", 2024-08-19T14:34:30.273 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2438396298 2024-08-19T14:34:30.273 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.273 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.273 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:34:30.273 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6829", 2024-08-19T14:34:30.273 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2438396298 2024-08-19T14:34:30.273 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:34:30.274 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:34:30.274 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.274 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.45:6825/2438396298", 2024-08-19T14:34:30.274 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.45:6827/2438396298", 2024-08-19T14:34:30.274 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.45:6831/2438396298", 2024-08-19T14:34:30.274 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.45:6829/2438396298", 2024-08-19T14:34:30.274 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-08-19T14:34:30.274 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-08-19T14:34:30.274 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-08-19T14:34:30.274 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:34:30.274 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:34:30.274 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-08-19T14:34:30.274 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_xinfo": [ 2024-08-19T14:34:30.274 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.275 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 0, 2024-08-19T14:34:30.275 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-08-19T14:34:30.275 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-08-19T14:34:30.275 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-08-19T14:34:30.275 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-08-19T14:34:30.275 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-08-19T14:34:30.275 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-08-19T14:33:59.806776+0000", 2024-08-19T14:34:30.275 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-08-19T14:34:30.275 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.275 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.275 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 1, 2024-08-19T14:34:30.275 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-08-19T14:34:30.275 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-08-19T14:34:30.275 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-08-19T14:34:30.276 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-08-19T14:34:30.276 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-08-19T14:34:30.276 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-08-19T14:34:29.243112+0000", 2024-08-19T14:34:30.276 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-08-19T14:34:30.276 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.276 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.276 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 2, 2024-08-19T14:34:30.276 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-08-19T14:34:30.276 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-08-19T14:34:30.276 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-08-19T14:34:30.276 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-08-19T14:34:30.276 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-08-19T14:34:30.276 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-08-19T14:30:20.319915+0000", 2024-08-19T14:34:30.276 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 564 2024-08-19T14:34:30.277 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.277 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.277 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 3, 2024-08-19T14:34:30.277 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-08-19T14:34:30.277 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-08-19T14:34:30.277 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-08-19T14:34:30.277 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-08-19T14:34:30.277 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-08-19T14:34:30.277 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-08-19T14:33:36.452367+0000", 2024-08-19T14:34:30.277 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-08-19T14:34:30.277 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.277 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.277 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 4, 2024-08-19T14:34:30.277 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-08-19T14:34:30.278 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-08-19T14:34:30.278 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-08-19T14:34:30.278 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-08-19T14:34:30.278 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-08-19T14:34:30.278 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-08-19T14:34:10.678908+0000", 2024-08-19T14:34:30.278 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-08-19T14:34:30.278 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.278 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.278 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 5, 2024-08-19T14:34:30.278 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-08-19T14:34:30.278 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-08-19T14:34:30.278 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-08-19T14:34:30.278 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-08-19T14:34:30.279 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-08-19T14:34:30.279 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-08-19T14:33:47.477474+0000", 2024-08-19T14:34:30.279 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-08-19T14:34:30.279 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.279 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.279 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 6, 2024-08-19T14:34:30.279 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-08-19T14:34:30.279 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-08-19T14:34:30.279 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-08-19T14:34:30.279 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-08-19T14:34:30.279 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 65536, 2024-08-19T14:34:30.279 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-08-19T14:34:03.193914+0000", 2024-08-19T14:34:30.279 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-08-19T14:34:30.279 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.280 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.280 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 7, 2024-08-19T14:34:30.280 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-08-19T14:34:30.280 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-08-19T14:34:30.280 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-08-19T14:34:30.280 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-08-19T14:34:30.280 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-08-19T14:34:30.280 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-08-19T14:33:31.387933+0000", 2024-08-19T14:34:30.280 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-08-19T14:34:30.280 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.280 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.280 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 8, 2024-08-19T14:34:30.280 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-08-19T14:34:30.280 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-08-19T14:34:30.280 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-08-19T14:34:30.281 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-08-19T14:34:30.281 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-08-19T14:34:30.281 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-08-19T14:33:39.921460+0000", 2024-08-19T14:34:30.281 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-08-19T14:34:30.281 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.281 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.281 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 9, 2024-08-19T14:34:30.281 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-08-19T14:34:30.281 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-08-19T14:34:30.281 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-08-19T14:34:30.281 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-08-19T14:34:30.281 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-08-19T14:34:30.282 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-08-19T14:33:48.991639+0000", 2024-08-19T14:34:30.282 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-08-19T14:34:30.282 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.282 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.282 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 10, 2024-08-19T14:34:30.282 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-08-19T14:34:30.282 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-08-19T14:34:30.282 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-08-19T14:34:30.282 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-08-19T14:34:30.282 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 65536, 2024-08-19T14:34:30.282 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-08-19T14:34:22.412011+0000", 2024-08-19T14:34:30.282 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 529 2024-08-19T14:34:30.282 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.283 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.283 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 11, 2024-08-19T14:34:30.283 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-08-19T14:34:30.283 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-08-19T14:34:30.283 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-08-19T14:34:30.283 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-08-19T14:34:30.283 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-08-19T14:34:30.283 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-08-19T14:33:29.662032+0000", 2024-08-19T14:34:30.283 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-08-19T14:34:30.283 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:34:30.283 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-08-19T14:34:30.283 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_upmap": [ 2024-08-19T14:34:30.283 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.283 INFO:teuthology.orchestra.run.smithi031.stdout: "pgid": "1.0", 2024-08-19T14:34:30.284 INFO:teuthology.orchestra.run.smithi031.stdout: "osds": [ 2024-08-19T14:34:30.284 INFO:teuthology.orchestra.run.smithi031.stdout: 2, 2024-08-19T14:34:30.284 INFO:teuthology.orchestra.run.smithi031.stdout: 9, 2024-08-19T14:34:30.284 INFO:teuthology.orchestra.run.smithi031.stdout: 0 2024-08-19T14:34:30.284 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:34:30.284 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:34:30.284 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-08-19T14:34:30.284 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_upmap_items": [], 2024-08-19T14:34:30.284 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_upmap_primaries": [], 2024-08-19T14:34:30.284 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_temp": [ 2024-08-19T14:34:30.284 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.284 INFO:teuthology.orchestra.run.smithi031.stdout: "pgid": "1.0", 2024-08-19T14:34:30.284 INFO:teuthology.orchestra.run.smithi031.stdout: "osds": [ 2024-08-19T14:34:30.285 INFO:teuthology.orchestra.run.smithi031.stdout: 0, 2024-08-19T14:34:30.285 INFO:teuthology.orchestra.run.smithi031.stdout: 9, 2024-08-19T14:34:30.285 INFO:teuthology.orchestra.run.smithi031.stdout: 7 2024-08-19T14:34:30.285 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:34:30.285 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.285 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.285 INFO:teuthology.orchestra.run.smithi031.stdout: "pgid": "1.1", 2024-08-19T14:34:30.285 INFO:teuthology.orchestra.run.smithi031.stdout: "osds": [ 2024-08-19T14:34:30.285 INFO:teuthology.orchestra.run.smithi031.stdout: 0, 2024-08-19T14:34:30.285 INFO:teuthology.orchestra.run.smithi031.stdout: 9, 2024-08-19T14:34:30.285 INFO:teuthology.orchestra.run.smithi031.stdout: 11 2024-08-19T14:34:30.285 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:34:30.285 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.285 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.286 INFO:teuthology.orchestra.run.smithi031.stdout: "pgid": "1.9", 2024-08-19T14:34:30.286 INFO:teuthology.orchestra.run.smithi031.stdout: "osds": [ 2024-08-19T14:34:30.286 INFO:teuthology.orchestra.run.smithi031.stdout: 7, 2024-08-19T14:34:30.286 INFO:teuthology.orchestra.run.smithi031.stdout: 9, 2024-08-19T14:34:30.286 INFO:teuthology.orchestra.run.smithi031.stdout: 5 2024-08-19T14:34:30.286 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:34:30.286 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.286 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.286 INFO:teuthology.orchestra.run.smithi031.stdout: "pgid": "1.14", 2024-08-19T14:34:30.286 INFO:teuthology.orchestra.run.smithi031.stdout: "osds": [ 2024-08-19T14:34:30.286 INFO:teuthology.orchestra.run.smithi031.stdout: 7, 2024-08-19T14:34:30.286 INFO:teuthology.orchestra.run.smithi031.stdout: 9, 2024-08-19T14:34:30.286 INFO:teuthology.orchestra.run.smithi031.stdout: 1 2024-08-19T14:34:30.287 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:34:30.287 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.287 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.287 INFO:teuthology.orchestra.run.smithi031.stdout: "pgid": "7.0", 2024-08-19T14:34:30.287 INFO:teuthology.orchestra.run.smithi031.stdout: "osds": [ 2024-08-19T14:34:30.287 INFO:teuthology.orchestra.run.smithi031.stdout: 3, 2024-08-19T14:34:30.287 INFO:teuthology.orchestra.run.smithi031.stdout: 5 2024-08-19T14:34:30.287 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:34:30.287 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.287 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.287 INFO:teuthology.orchestra.run.smithi031.stdout: "pgid": "7.5", 2024-08-19T14:34:30.287 INFO:teuthology.orchestra.run.smithi031.stdout: "osds": [ 2024-08-19T14:34:30.287 INFO:teuthology.orchestra.run.smithi031.stdout: 11, 2024-08-19T14:34:30.287 INFO:teuthology.orchestra.run.smithi031.stdout: 0 2024-08-19T14:34:30.288 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:34:30.288 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.288 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.288 INFO:teuthology.orchestra.run.smithi031.stdout: "pgid": "7.a", 2024-08-19T14:34:30.288 INFO:teuthology.orchestra.run.smithi031.stdout: "osds": [ 2024-08-19T14:34:30.288 INFO:teuthology.orchestra.run.smithi031.stdout: 11, 2024-08-19T14:34:30.288 INFO:teuthology.orchestra.run.smithi031.stdout: 3 2024-08-19T14:34:30.288 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:34:30.288 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.288 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.288 INFO:teuthology.orchestra.run.smithi031.stdout: "pgid": "7.e", 2024-08-19T14:34:30.288 INFO:teuthology.orchestra.run.smithi031.stdout: "osds": [ 2024-08-19T14:34:30.288 INFO:teuthology.orchestra.run.smithi031.stdout: 1, 2024-08-19T14:34:30.288 INFO:teuthology.orchestra.run.smithi031.stdout: 5 2024-08-19T14:34:30.289 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:34:30.289 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.289 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:34:30.289 INFO:teuthology.orchestra.run.smithi031.stdout: "pgid": "7.f", 2024-08-19T14:34:30.289 INFO:teuthology.orchestra.run.smithi031.stdout: "osds": [ 2024-08-19T14:34:30.289 INFO:teuthology.orchestra.run.smithi031.stdout: 0, 2024-08-19T14:34:30.289 INFO:teuthology.orchestra.run.smithi031.stdout: 8 2024-08-19T14:34:30.289 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:34:30.289 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:34:30.289 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-08-19T14:34:30.289 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_temp": [], 2024-08-19T14:34:30.289 INFO:teuthology.orchestra.run.smithi031.stdout: "blocklist": { 2024-08-19T14:34:30.289 INFO:teuthology.orchestra.run.smithi031.stdout: "172.21.15.31:0/1451745969": "2024-08-20T14:06:13.660657+0000", 2024-08-19T14:34:30.289 INFO:teuthology.orchestra.run.smithi031.stdout: "172.21.15.31:0/1515698918": "2024-08-20T14:06:13.660657+0000", 2024-08-19T14:34:30.290 INFO:teuthology.orchestra.run.smithi031.stdout: "172.21.15.31:6801/3297539242": "2024-08-20T14:06:13.660657+0000", 2024-08-19T14:34:30.290 INFO:teuthology.orchestra.run.smithi031.stdout: "172.21.15.31:6800/3297539242": "2024-08-20T14:06:13.660657+0000", 2024-08-19T14:34:30.290 INFO:teuthology.orchestra.run.smithi031.stdout: "172.21.15.31:0/2217423818": "2024-08-20T14:05:48.827576+0000", 2024-08-19T14:34:30.290 INFO:teuthology.orchestra.run.smithi031.stdout: "172.21.15.31:0/3353498281": "2024-08-20T14:06:13.660657+0000", 2024-08-19T14:34:30.290 INFO:teuthology.orchestra.run.smithi031.stdout: "172.21.15.31:0/4013594220": "2024-08-20T14:05:48.827576+0000", 2024-08-19T14:34:30.290 INFO:teuthology.orchestra.run.smithi031.stdout: "172.21.15.31:0/1654518065": "2024-08-20T14:05:48.827576+0000", 2024-08-19T14:34:30.290 INFO:teuthology.orchestra.run.smithi031.stdout: "172.21.15.31:6801/2368821258": "2024-08-20T14:05:48.827576+0000", 2024-08-19T14:34:30.290 INFO:teuthology.orchestra.run.smithi031.stdout: "172.21.15.31:6800/2368821258": "2024-08-20T14:05:48.827576+0000" 2024-08-19T14:34:30.290 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.290 INFO:teuthology.orchestra.run.smithi031.stdout: "range_blocklist": {}, 2024-08-19T14:34:30.290 INFO:teuthology.orchestra.run.smithi031.stdout: "erasure_code_profiles": { 2024-08-19T14:34:30.290 INFO:teuthology.orchestra.run.smithi031.stdout: "default": { 2024-08-19T14:34:30.290 INFO:teuthology.orchestra.run.smithi031.stdout: "crush-failure-domain": "osd", 2024-08-19T14:34:30.290 INFO:teuthology.orchestra.run.smithi031.stdout: "k": "2", 2024-08-19T14:34:30.291 INFO:teuthology.orchestra.run.smithi031.stdout: "m": "1", 2024-08-19T14:34:30.291 INFO:teuthology.orchestra.run.smithi031.stdout: "plugin": "jerasure", 2024-08-19T14:34:30.291 INFO:teuthology.orchestra.run.smithi031.stdout: "technique": "reed_sol_van" 2024-08-19T14:34:30.291 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:34:30.291 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:34:30.291 INFO:teuthology.orchestra.run.smithi031.stdout: "removed_snaps_queue": [], 2024-08-19T14:34:30.291 INFO:teuthology.orchestra.run.smithi031.stdout: "new_removed_snaps": [], 2024-08-19T14:34:30.291 INFO:teuthology.orchestra.run.smithi031.stdout: "new_purged_snaps": [], 2024-08-19T14:34:30.291 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_node_flags": {}, 2024-08-19T14:34:30.291 INFO:teuthology.orchestra.run.smithi031.stdout: "device_class_flags": {}, 2024-08-19T14:34:30.291 INFO:teuthology.orchestra.run.smithi031.stdout: "stretch_mode": { 2024-08-19T14:34:30.291 INFO:teuthology.orchestra.run.smithi031.stdout: "stretch_mode_enabled": false, 2024-08-19T14:34:30.291 INFO:teuthology.orchestra.run.smithi031.stdout: "stretch_bucket_count": 0, 2024-08-19T14:34:30.291 INFO:teuthology.orchestra.run.smithi031.stdout: "degraded_stretch_mode": 0, 2024-08-19T14:34:30.292 INFO:teuthology.orchestra.run.smithi031.stdout: "recovering_stretch_mode": 0, 2024-08-19T14:34:30.292 INFO:teuthology.orchestra.run.smithi031.stdout: "stretch_mode_bucket": 0 2024-08-19T14:34:30.292 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:34:30.292 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:34:30.292 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-08-19T14:34:30.293 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:34:30.315 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:30.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:30.316 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:34:30.416 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:34:30.440 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:30.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:30.441 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:34:30.445 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 57 16 90674 90658 99.3968 48.875 0.00696535 0.0100557 2024-08-19T14:34:30.542 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:34:30.581 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:34:30.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:30.583 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:30.583 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:34:30.683 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:34:30.705 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:30.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:30.706 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:34:30.807 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:34:30.834 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:30.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:30.835 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:34:30.936 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:34:30.958 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:30.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:30.959 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:34:31.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:30 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:34:31.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:30 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:34:31.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:30 smithi032 ceph-mon[20004]: osdmap e623: 12 total, 11 up, 10 in 2024-08-19T14:34:31.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:30 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1313120332' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-08-19T14:34:31.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:30 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:34:31.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:30 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:34:31.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:30 smithi045 ceph-mon[20143]: osdmap e623: 12 total, 11 up, 10 in 2024-08-19T14:34:31.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:30 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1313120332' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-08-19T14:34:31.060 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:34:31.093 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:31.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:31.094 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:34:31.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:30 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:34:31.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:30 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:34:31.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:30 smithi031 ceph-mon[16336]: osdmap e623: 12 total, 11 up, 10 in 2024-08-19T14:34:31.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:30 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1313120332' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-08-19T14:34:31.195 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:34:31.218 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:31.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:31.219 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:34:31.320 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:34:31.341 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:31.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:31.343 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:34:31.357 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:34:31.357 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:34:31.443 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:34:31.445 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 58 16 91411 91395 98.4772 46.0625 0.00616439 0.0101494 2024-08-19T14:34:31.483 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:31.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:31.484 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:34:31.585 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:34:31.625 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:31.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:31.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:34:31.728 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:34:31.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:31.760 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:31.760 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:34:31.861 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:34:31.893 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:31.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:31.895 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:34:31.995 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:34:32.033 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:32.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:32.034 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:34:32.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:31 smithi045 ceph-mon[20143]: pgmap v1461: 47 pgs: 4 active+clean+remapped, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized, 5 active+undersized+degraded, 29 active+clean; 5.4 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 51 MiB/s wr, 813 op/s; 32905/264093 objects degraded (12.460%); 21514/264093 objects misplaced (8.146%); 8.2 MiB/s, 130 objects/s recovering 2024-08-19T14:34:32.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:31 smithi031 ceph-mon[16336]: pgmap v1461: 47 pgs: 4 active+clean+remapped, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized, 5 active+undersized+degraded, 29 active+clean; 5.4 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 51 MiB/s wr, 813 op/s; 32905/264093 objects degraded (12.460%); 21514/264093 objects misplaced (8.146%); 8.2 MiB/s, 130 objects/s recovering 2024-08-19T14:34:32.135 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:34:32.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:32.161 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:32.161 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:34:32.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:31 smithi032 ceph-mon[20004]: pgmap v1461: 47 pgs: 4 active+clean+remapped, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized, 5 active+undersized+degraded, 29 active+clean; 5.4 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 51 MiB/s wr, 813 op/s; 32905/264093 objects degraded (12.460%); 21514/264093 objects misplaced (8.146%); 8.2 MiB/s, 130 objects/s recovering 2024-08-19T14:34:32.262 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:34:32.289 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-08-19T14:34:32.293 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:32.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:32.295 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:34:32.395 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:34:32.419 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:32.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:32.420 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:34:32.445 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 59 16 92375 92359 97.8292 60.25 0.00974945 0.0102162 2024-08-19T14:34:32.521 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:34:32.550 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:32.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:32.551 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:34:32.575 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:34:32.652 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:34:32.683 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:32.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:32.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:34:32.785 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:34:32.817 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:32.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:32.818 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:34:32.919 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:34:32.949 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:32.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:32.950 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:34:33.051 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:34:33.075 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:33.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:33.076 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:34:33.103 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:34:33.103 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:34:33.177 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:34:33.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:33.217 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:33.218 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:34:33.319 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:34:33.334 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-19T14:34:33.334 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-19T14:34:33.356 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:33.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:33.357 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:34:33.382 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:34:33.445 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:2024-08-19T14:34:33.445890+0000 min lat: 0.00106356 max lat: 2.26272 avg lat: 0.0102828 2024-08-19T14:34:33.445 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-19T14:34:33.445 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 60 16 93343 93327 97.207 60.5 0.0128629 0.0102828 2024-08-19T14:34:33.458 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:34:33.495 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:33.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:33.496 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:34:33.597 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:34:33.619 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:33.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:33.620 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:34:33.721 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:34:33.751 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:33.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:33.752 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:34:33.852 INFO:tasks.thrashosds.thrasher:j is {'epoch': 623, 'fsid': 'ed7d1c6e-5e33-11ef-bccf-c7b262605968', 'created': '2024-08-19T14:05:22.842169+0000', 'modified': '2024-08-19T14:34:29.767444+0000', 'last_up_change': '2024-08-19T14:31:02.021921+0000', 'last_in_change': '2024-08-19T14:34:11.676140+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 28, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 7, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-08-19T14:08:15.966221+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 31, 'pg_placement_num': 31, 'pg_placement_num_target': 31, 'pg_num_target': 31, 'pg_num_pending': 31, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '278', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '276', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.259999990463257, 'score_stable': 2.259999990463257, 'optimal_score': 0.8999999761581421, 'raw_score_acting': 2.0299999713897705, 'raw_score_stable': 2.0299999713897705, 'primary_affinity_weighted': 0.7561492323875427, 'average_primary_affinity': 0.9100000262260437, 'average_primary_affinity_weighted': 0.7599999904632568}}, {'pool': 7, 'pool_name': 'unique_pool_5', 'create_time': '2024-08-19T14:33:26.611185+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '604', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 0.8999999761581421, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.7561492323875427, 'average_primary_affinity': 0.9100000262260437, 'average_primary_affinity_weighted': 0.7599999904632568}}], 'osds': [{'osd': 0, 'uuid': 'd7a0fff3-23c6-4ee3-a275-a844862aabfb', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 619, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6802', 'nonce': 1108945097}, {'type': 'v1', 'addr': '172.21.15.31:6803', 'nonce': 1108945097}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6804', 'nonce': 1108945097}, {'type': 'v1', 'addr': '172.21.15.31:6805', 'nonce': 1108945097}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6808', 'nonce': 1108945097}, {'type': 'v1', 'addr': '172.21.15.31:6809', 'nonce': 1108945097}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6806', 'nonce': 1108945097}, {'type': 'v1', 'addr': '172.21.15.31:6807', 'nonce': 1108945097}]}, 'public_addr': '172.21.15.31:6803/1108945097', 'cluster_addr': '172.21.15.31:6805/1108945097', 'heartbeat_back_addr': '172.21.15.31:6809/1108945097', 'heartbeat_front_addr': '172.21.15.31:6807/1108945097', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '145d5bb4-74dd-440f-bcc9-7e9ebda9c5d6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 617, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6810', 'nonce': 3236186806}, {'type': 'v1', 'addr': '172.21.15.31:6811', 'nonce': 3236186806}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6812', 'nonce': 3236186806}, {'type': 'v1', 'addr': '172.21.15.31:6813', 'nonce': 3236186806}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6816', 'nonce': 3236186806}, {'type': 'v1', 'addr': '172.21.15.31:6817', 'nonce': 3236186806}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6814', 'nonce': 3236186806}, {'type': 'v1', 'addr': '172.21.15.31:6815', 'nonce': 3236186806}]}, 'public_addr': '172.21.15.31:6811/3236186806', 'cluster_addr': '172.21.15.31:6813/3236186806', 'heartbeat_back_addr': '172.21.15.31:6817/3236186806', 'heartbeat_front_addr': '172.21.15.31:6815/3236186806', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '376c18d0-4768-46a0-bb60-e79c88ca909c', 'up': 0, 'in': 1, 'weight': 1, 'primary_affinity': 0.07379150390625, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 204, 'down_at': 565, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6818', 'nonce': 4045553387}, {'type': 'v1', 'addr': '172.21.15.31:6819', 'nonce': 4045553387}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6820', 'nonce': 4045553387}, {'type': 'v1', 'addr': '172.21.15.31:6821', 'nonce': 4045553387}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6824', 'nonce': 4045553387}, {'type': 'v1', 'addr': '172.21.15.31:6825', 'nonce': 4045553387}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6822', 'nonce': 4045553387}, {'type': 'v1', 'addr': '172.21.15.31:6823', 'nonce': 4045553387}]}, 'public_addr': '172.21.15.31:6819/4045553387', 'cluster_addr': '172.21.15.31:6821/4045553387', 'heartbeat_back_addr': '172.21.15.31:6825/4045553387', 'heartbeat_front_addr': '172.21.15.31:6823/4045553387', 'state': ['exists']}, {'osd': 3, 'uuid': 'a787ea9f-b9ba-450f-b388-614cfba90634', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 619, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6826', 'nonce': 2288082716}, {'type': 'v1', 'addr': '172.21.15.31:6827', 'nonce': 2288082716}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6828', 'nonce': 2288082716}, {'type': 'v1', 'addr': '172.21.15.31:6829', 'nonce': 2288082716}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6832', 'nonce': 2288082716}, {'type': 'v1', 'addr': '172.21.15.31:6833', 'nonce': 2288082716}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6830', 'nonce': 2288082716}, {'type': 'v1', 'addr': '172.21.15.31:6831', 'nonce': 2288082716}]}, 'public_addr': '172.21.15.31:6827/2288082716', 'cluster_addr': '172.21.15.31:6829/2288082716', 'heartbeat_back_addr': '172.21.15.31:6833/2288082716', 'heartbeat_front_addr': '172.21.15.31:6831/2288082716', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '7859751a-12f5-4a57-a332-435360479d53', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 619, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6800', 'nonce': 3934739984}, {'type': 'v1', 'addr': '172.21.15.32:6801', 'nonce': 3934739984}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6802', 'nonce': 3934739984}, {'type': 'v1', 'addr': '172.21.15.32:6803', 'nonce': 3934739984}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6806', 'nonce': 3934739984}, {'type': 'v1', 'addr': '172.21.15.32:6807', 'nonce': 3934739984}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6804', 'nonce': 3934739984}, {'type': 'v1', 'addr': '172.21.15.32:6805', 'nonce': 3934739984}]}, 'public_addr': '172.21.15.32:6801/3934739984', 'cluster_addr': '172.21.15.32:6803/3934739984', 'heartbeat_back_addr': '172.21.15.32:6807/3934739984', 'heartbeat_front_addr': '172.21.15.32:6805/3934739984', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '017f3eef-ac60-4b82-8350-919017cb4ccd', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 40, 'up_thru': 464, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6808', 'nonce': 526388985}, {'type': 'v1', 'addr': '172.21.15.32:6809', 'nonce': 526388985}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6810', 'nonce': 526388985}, {'type': 'v1', 'addr': '172.21.15.32:6811', 'nonce': 526388985}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6814', 'nonce': 526388985}, {'type': 'v1', 'addr': '172.21.15.32:6815', 'nonce': 526388985}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6812', 'nonce': 526388985}, {'type': 'v1', 'addr': '172.21.15.32:6813', 'nonce': 526388985}]}, 'public_addr': '172.21.15.32:6809/526388985', 'cluster_addr': '172.21.15.32:6811/526388985', 'heartbeat_back_addr': '172.21.15.32:6815/526388985', 'heartbeat_front_addr': '172.21.15.32:6813/526388985', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '44aa8229-0268-49d7-bd57-3bf1f384e5e6', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 48, 'up_thru': 126, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6816', 'nonce': 1131035844}, {'type': 'v1', 'addr': '172.21.15.32:6817', 'nonce': 1131035844}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6818', 'nonce': 1131035844}, {'type': 'v1', 'addr': '172.21.15.32:6819', 'nonce': 1131035844}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6822', 'nonce': 1131035844}, {'type': 'v1', 'addr': '172.21.15.32:6823', 'nonce': 1131035844}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6820', 'nonce': 1131035844}, {'type': 'v1', 'addr': '172.21.15.32:6821', 'nonce': 1131035844}]}, 'public_addr': '172.21.15.32:6817/1131035844', 'cluster_addr': '172.21.15.32:6819/1131035844', 'heartbeat_back_addr': '172.21.15.32:6823/1131035844', 'heartbeat_front_addr': '172.21.15.32:6821/1131035844', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '263433e5-70b6-4192-86ff-dc2ad95927eb', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 57, 'up_thru': 619, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6824', 'nonce': 2014939947}, {'type': 'v1', 'addr': '172.21.15.32:6825', 'nonce': 2014939947}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6826', 'nonce': 2014939947}, {'type': 'v1', 'addr': '172.21.15.32:6827', 'nonce': 2014939947}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6830', 'nonce': 2014939947}, {'type': 'v1', 'addr': '172.21.15.32:6831', 'nonce': 2014939947}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6828', 'nonce': 2014939947}, {'type': 'v1', 'addr': '172.21.15.32:6829', 'nonce': 2014939947}]}, 'public_addr': '172.21.15.32:6825/2014939947', 'cluster_addr': '172.21.15.32:6827/2014939947', 'heartbeat_back_addr': '172.21.15.32:6831/2014939947', 'heartbeat_front_addr': '172.21.15.32:6829/2014939947', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': '2bbcad36-0c74-4528-9fb9-a8812afa1662', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 65, 'up_thru': 619, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6800', 'nonce': 1445140892}, {'type': 'v1', 'addr': '172.21.15.45:6801', 'nonce': 1445140892}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6802', 'nonce': 1445140892}, {'type': 'v1', 'addr': '172.21.15.45:6803', 'nonce': 1445140892}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6806', 'nonce': 1445140892}, {'type': 'v1', 'addr': '172.21.15.45:6807', 'nonce': 1445140892}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6804', 'nonce': 1445140892}, {'type': 'v1', 'addr': '172.21.15.45:6805', 'nonce': 1445140892}]}, 'public_addr': '172.21.15.45:6801/1445140892', 'cluster_addr': '172.21.15.45:6803/1445140892', 'heartbeat_back_addr': '172.21.15.45:6807/1445140892', 'heartbeat_front_addr': '172.21.15.45:6805/1445140892', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': 'c8ee4277-3bf9-46bf-9cc0-152c1c4b9508', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 74, 'up_thru': 616, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6808', 'nonce': 1213787169}, {'type': 'v1', 'addr': '172.21.15.45:6809', 'nonce': 1213787169}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6810', 'nonce': 1213787169}, {'type': 'v1', 'addr': '172.21.15.45:6811', 'nonce': 1213787169}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6814', 'nonce': 1213787169}, {'type': 'v1', 'addr': '172.21.15.45:6815', 'nonce': 1213787169}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6812', 'nonce': 1213787169}, {'type': 'v1', 'addr': '172.21.15.45:6813', 'nonce': 1213787169}]}, 'public_addr': '172.21.15.45:6809/1213787169', 'cluster_addr': '172.21.15.45:6811/1213787169', 'heartbeat_back_addr': '172.21.15.45:6815/1213787169', 'heartbeat_front_addr': '172.21.15.45:6813/1213787169', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '29031395-5f5c-40f8-ac90-9f6ac48e5c9f', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 80, 'last_clean_end': 529, 'up_from': 539, 'up_thru': 115, 'down_at': 530, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6816', 'nonce': 1069743893}, {'type': 'v1', 'addr': '172.21.15.45:6817', 'nonce': 1069743893}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6818', 'nonce': 1069743893}, {'type': 'v1', 'addr': '172.21.15.45:6819', 'nonce': 1069743893}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6822', 'nonce': 1069743893}, {'type': 'v1', 'addr': '172.21.15.45:6823', 'nonce': 1069743893}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6820', 'nonce': 1069743893}, {'type': 'v1', 'addr': '172.21.15.45:6821', 'nonce': 1069743893}]}, 'public_addr': '172.21.15.45:6817/1069743893', 'cluster_addr': '172.21.15.45:6819/1069743893', 'heartbeat_back_addr': '172.21.15.45:6823/1069743893', 'heartbeat_front_addr': '172.21.15.45:6821/1069743893', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': '5ef0db4b-0fef-4ea0-a19b-d3677e082c29', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 86, 'up_thru': 617, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6824', 'nonce': 2438396298}, {'type': 'v1', 'addr': '172.21.15.45:6825', 'nonce': 2438396298}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6826', 'nonce': 2438396298}, {'type': 'v1', 'addr': '172.21.15.45:6827', 'nonce': 2438396298}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6830', 'nonce': 2438396298}, {'type': 'v1', 'addr': '172.21.15.45:6831', 'nonce': 2438396298}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6828', 'nonce': 2438396298}, {'type': 'v1', 'addr': '172.21.15.45:6829', 'nonce': 2438396298}]}, 'public_addr': '172.21.15.45:6825/2438396298', 'cluster_addr': '172.21.15.45:6827/2438396298', 'heartbeat_back_addr': '172.21.15.45:6831/2438396298', 'heartbeat_front_addr': '172.21.15.45:6829/2438396298', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-19T14:33:59.806776+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-19T14:34:29.243112+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-19T14:30:20.319915+0000', 'dead_epoch': 564}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-19T14:33:36.452367+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-19T14:34:10.678908+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-19T14:33:47.477474+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-08-19T14:34:03.193914+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-19T14:33:31.387933+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-19T14:33:39.921460+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-19T14:33:48.991639+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-08-19T14:34:22.412011+0000', 'dead_epoch': 529}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-19T14:33:29.662032+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '1.0', 'osds': [2, 9, 0]}], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '1.0', 'osds': [0, 9, 7]}, {'pgid': '1.1', 'osds': [0, 9, 11]}, {'pgid': '1.9', 'osds': [7, 9, 5]}, {'pgid': '1.14', 'osds': [7, 9, 1]}, {'pgid': '7.0', 'osds': [3, 5]}, {'pgid': '7.5', 'osds': [11, 0]}, {'pgid': '7.a', 'osds': [11, 3]}, {'pgid': '7.e', 'osds': [1, 5]}, {'pgid': '7.f', 'osds': [0, 8]}], 'primary_temp': [], 'blocklist': {'172.21.15.31:0/1451745969': '2024-08-20T14:06:13.660657+0000', '172.21.15.31:0/1515698918': '2024-08-20T14:06:13.660657+0000', '172.21.15.31:6801/3297539242': '2024-08-20T14:06:13.660657+0000', '172.21.15.31:6800/3297539242': '2024-08-20T14:06:13.660657+0000', '172.21.15.31:0/2217423818': '2024-08-20T14:05:48.827576+0000', '172.21.15.31:0/3353498281': '2024-08-20T14:06:13.660657+0000', '172.21.15.31:0/4013594220': '2024-08-20T14:05:48.827576+0000', '172.21.15.31:0/1654518065': '2024-08-20T14:05:48.827576+0000', '172.21.15.31:6801/2368821258': '2024-08-20T14:05:48.827576+0000', '172.21.15.31:6800/2368821258': '2024-08-20T14:05:48.827576+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-08-19T14:34:33.852 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-19T14:34:33.853 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:34:33.885 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-08-19T14:34:33.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:33.928 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:33.928 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:34:34.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:33 smithi032 ceph-mon[20004]: pgmap v1462: 47 pgs: 4 active+clean+remapped, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized, 5 active+undersized+degraded, 29 active+clean; 5.5 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 57 MiB/s wr, 919 op/s; 34241/270450 objects degraded (12.661%); 20943/270450 objects misplaced (7.744%); 11 MiB/s, 172 objects/s recovering 2024-08-19T14:34:34.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:33 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 34241/270450 objects degraded (12.661%), 7 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-08-19T14:34:34.029 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:34:34.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:33 smithi045 ceph-mon[20143]: pgmap v1462: 47 pgs: 4 active+clean+remapped, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized, 5 active+undersized+degraded, 29 active+clean; 5.5 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 57 MiB/s wr, 919 op/s; 34241/270450 objects degraded (12.661%); 20943/270450 objects misplaced (7.744%); 11 MiB/s, 172 objects/s recovering 2024-08-19T14:34:34.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:33 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 34241/270450 objects degraded (12.661%), 7 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-08-19T14:34:34.053 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:34.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:34.054 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:34:34.097 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:33 smithi031 ceph-mon[16336]: pgmap v1462: 47 pgs: 4 active+clean+remapped, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized, 5 active+undersized+degraded, 29 active+clean; 5.5 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 57 MiB/s wr, 919 op/s; 34241/270450 objects degraded (12.661%); 20943/270450 objects misplaced (7.744%); 11 MiB/s, 172 objects/s recovering 2024-08-19T14:34:34.097 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:33 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 34241/270450 objects degraded (12.661%), 7 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-08-19T14:34:34.154 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:34:34.167 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:34:34.177 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:34.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:34.178 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:34:34.279 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:34:34.322 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:34.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:34.323 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:34:34.424 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:34:34.445 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 61 16 94252 94236 96.5447 56.8125 0.0276707 0.0103527 2024-08-19T14:34:34.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:34.465 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:34.465 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:34:34.559 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:34:34.566 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:34:34.592 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:34.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:34.594 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:34:34.680 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:34:34.680 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:34:34.695 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:34:34.725 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:34.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:34.726 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:34:34.802 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is set 2024-08-19T14:34:34.827 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:34:34.874 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:34.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:34.878 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:34:34.979 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:34:35.006 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:35.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:35.007 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:34:35.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:34 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/803450616' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:34:35.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:34 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:34:35.108 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:34:35.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:34 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/803450616' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:34:35.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:34 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:34:35.132 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:35.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:35.133 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:34:35.234 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:34:35.269 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:35.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:35.270 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:34:35.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:34 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/803450616' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:34:35.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:34 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:34:35.335 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:34:35.371 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:34:35.383 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-08-19T14:34:35.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:35.429 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:35.430 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:34:35.445 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 62 16 94924 94908 95.6649 42 0.0145589 0.0104479 2024-08-19T14:34:35.530 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:34:35.552 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:35.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:35.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:34:35.655 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:34:35.696 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:35.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:35.698 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:34:35.721 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:34:35.722 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-08-19T14:34:35.759 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:34:35.799 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:34:35.819 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:35.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:35.820 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:34:35.921 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:34:35.945 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:35.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:35.946 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:34:36.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:35 smithi032 ceph-mon[20004]: pgmap v1463: 47 pgs: 4 active+clean+remapped, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized, 5 active+undersized+degraded, 29 active+clean; 5.6 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 54 MiB/s wr, 859 op/s; 34241/274548 objects degraded (12.472%); 20809/274548 objects misplaced (7.579%); 10 MiB/s, 164 objects/s recovering 2024-08-19T14:34:36.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:35 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:34:36.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:35 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:34:36.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:35 smithi032 ceph-mon[20004]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:34:36.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:35 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:34:36.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:35 smithi032 ceph-mon[20004]: osdmap e624: 12 total, 11 up, 10 in 2024-08-19T14:34:36.046 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:34:36.069 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:36.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:36.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:34:36.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:35 smithi031 ceph-mon[16336]: pgmap v1463: 47 pgs: 4 active+clean+remapped, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized, 5 active+undersized+degraded, 29 active+clean; 5.6 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 54 MiB/s wr, 859 op/s; 34241/274548 objects degraded (12.472%); 20809/274548 objects misplaced (7.579%); 10 MiB/s, 164 objects/s recovering 2024-08-19T14:34:36.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:35 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:34:36.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:35 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:34:36.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:35 smithi031 ceph-mon[16336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:34:36.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:35 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:34:36.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:35 smithi031 ceph-mon[16336]: osdmap e624: 12 total, 11 up, 10 in 2024-08-19T14:34:36.171 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:34:36.192 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:36.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:36.193 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:34:36.294 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:34:36.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:35 smithi045 ceph-mon[20143]: pgmap v1463: 47 pgs: 4 active+clean+remapped, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized, 5 active+undersized+degraded, 29 active+clean; 5.6 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 54 MiB/s wr, 859 op/s; 34241/274548 objects degraded (12.472%); 20809/274548 objects misplaced (7.579%); 10 MiB/s, 164 objects/s recovering 2024-08-19T14:34:36.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:35 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:34:36.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:35 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:34:36.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:35 smithi045 ceph-mon[20143]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:34:36.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:35 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:34:36.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:35 smithi045 ceph-mon[20143]: osdmap e624: 12 total, 11 up, 10 in 2024-08-19T14:34:36.315 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:36.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:36.316 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:34:36.417 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:34:36.654 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 63 16 95668 95652 94.8844 46.5 0.00859131 0.0105326 2024-08-19T14:34:36.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:36.687 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:36.688 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:34:36.788 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:34:36.812 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:36.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:36.814 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:34:36.914 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:34:36.945 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:36.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:36.947 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:34:37.047 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:34:37.074 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:37.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:37.075 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:34:37.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:36 smithi031 ceph-mon[16336]: from='client.17046 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:34:37.176 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:34:37.209 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:37.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:37.210 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:34:37.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:36 smithi032 ceph-mon[20004]: from='client.17046 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:34:37.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:36 smithi045 ceph-mon[20143]: from='client.17046 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:34:37.311 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:34:37.332 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:37.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:37.333 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:34:37.434 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:34:37.445 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 64 16 96466 96450 94.1811 49.875 0.00705513 0.0106121 2024-08-19T14:34:37.470 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:37.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:37.472 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:34:37.573 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:34:37.594 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:37.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:37.595 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:34:37.696 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:34:37.730 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:37.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:37.732 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:34:37.832 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:34:37.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:37.869 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:37.870 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:34:37.971 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:34:38.003 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:38.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:38.004 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:34:38.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:37 smithi032 ceph-mon[20004]: pgmap v1465: 47 pgs: 4 active+clean+remapped, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized, 5 active+undersized+degraded, 29 active+clean; 5.7 GiB data, 26 GiB used, 869 GiB / 894 GiB avail; 63 MiB/s wr, 1.00k op/s; 35557/281847 objects degraded (12.616%); 20522/281847 objects misplaced (7.281%); 9.8 MiB/s, 156 objects/s recovering 2024-08-19T14:34:38.105 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:34:38.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:37 smithi031 ceph-mon[16336]: pgmap v1465: 47 pgs: 4 active+clean+remapped, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized, 5 active+undersized+degraded, 29 active+clean; 5.7 GiB data, 26 GiB used, 869 GiB / 894 GiB avail; 63 MiB/s wr, 1.00k op/s; 35557/281847 objects degraded (12.616%); 20522/281847 objects misplaced (7.281%); 9.8 MiB/s, 156 objects/s recovering 2024-08-19T14:34:38.127 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:38.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:38.128 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:34:38.197 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:37 smithi045 ceph-mon[20143]: pgmap v1465: 47 pgs: 4 active+clean+remapped, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized, 5 active+undersized+degraded, 29 active+clean; 5.7 GiB data, 26 GiB used, 869 GiB / 894 GiB avail; 63 MiB/s wr, 1.00k op/s; 35557/281847 objects degraded (12.616%); 20522/281847 objects misplaced (7.281%); 9.8 MiB/s, 156 objects/s recovering 2024-08-19T14:34:38.229 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:34:38.263 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:38.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:38.264 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:34:38.365 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:34:38.387 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:38.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:38.388 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:34:38.446 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 65 16 97375 97359 93.6061 56.8125 0.0392765 0.0106764 2024-08-19T14:34:38.488 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:34:38.511 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:38.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:38.512 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:34:38.613 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:34:38.650 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:38.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:38.651 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:34:38.752 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:34:38.776 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:38.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:38.777 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:34:38.877 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:34:38.899 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:38.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:38.900 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:34:39.001 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:34:39.021 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:39.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:39.022 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:34:39.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:38 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 35557/281847 objects degraded (12.616%), 7 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-08-19T14:34:39.123 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:34:39.147 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:39.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:39.148 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:34:39.249 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:34:39.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:38 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 35557/281847 objects degraded (12.616%), 7 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-08-19T14:34:39.273 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:39.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:39.274 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:34:39.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:38 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 35557/281847 objects degraded (12.616%), 7 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-08-19T14:34:39.375 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:34:39.397 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:39.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:39.399 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:34:39.446 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 66 16 98085 98069 92.8601 44.375 0.00318532 0.0107621 2024-08-19T14:34:39.499 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:34:39.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:39.533 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:39.533 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:34:39.634 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:34:39.665 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:39.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:39.666 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:34:39.767 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:34:39.789 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:39.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:39.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:34:39.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:39 smithi031 ceph-mon[16336]: pgmap v1466: 47 pgs: 4 active+clean+remapped, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized, 5 active+undersized+degraded, 29 active+clean; 5.8 GiB data, 26 GiB used, 868 GiB / 894 GiB avail; 48 MiB/s wr, 773 op/s; 35844/283524 objects degraded (12.642%); 20078/283524 objects misplaced (7.082%); 11 MiB/s, 171 objects/s recovering 2024-08-19T14:34:39.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:39 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:34:39.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:39 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:34:39.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:39 smithi031 ceph-mon[16336]: osdmap e625: 12 total, 11 up, 10 in 2024-08-19T14:34:39.891 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:34:39.914 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:39.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:39.915 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:34:40.016 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:34:40.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:40.063 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:40.063 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:34:40.164 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:34:40.189 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:40.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:40.190 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:34:40.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:39 smithi032 ceph-mon[20004]: pgmap v1466: 47 pgs: 4 active+clean+remapped, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized, 5 active+undersized+degraded, 29 active+clean; 5.8 GiB data, 26 GiB used, 868 GiB / 894 GiB avail; 48 MiB/s wr, 773 op/s; 35844/283524 objects degraded (12.642%); 20078/283524 objects misplaced (7.082%); 11 MiB/s, 171 objects/s recovering 2024-08-19T14:34:40.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:39 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:34:40.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:39 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:34:40.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:39 smithi032 ceph-mon[20004]: osdmap e625: 12 total, 11 up, 10 in 2024-08-19T14:34:40.291 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:34:40.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:39 smithi045 ceph-mon[20143]: pgmap v1466: 47 pgs: 4 active+clean+remapped, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized, 5 active+undersized+degraded, 29 active+clean; 5.8 GiB data, 26 GiB used, 868 GiB / 894 GiB avail; 48 MiB/s wr, 773 op/s; 35844/283524 objects degraded (12.642%); 20078/283524 objects misplaced (7.082%); 11 MiB/s, 171 objects/s recovering 2024-08-19T14:34:40.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:39 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:34:40.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:39 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:34:40.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:39 smithi045 ceph-mon[20143]: osdmap e625: 12 total, 11 up, 10 in 2024-08-19T14:34:40.320 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:40.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:40.321 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:34:40.422 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:34:40.442 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:40.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:40.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:34:40.446 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 67 16 98975 98959 92.3043 55.625 0.00498044 0.0108249 2024-08-19T14:34:40.544 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:34:40.571 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:40.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:40.572 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:34:40.672 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:34:40.697 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:40.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:40.698 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:34:40.799 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:34:40.823 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:34:40.823 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:34:40.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:40.844 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:40.844 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:34:40.945 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:34:40.967 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:40.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:40.968 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:34:41.069 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:34:41.116 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:41.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:41.117 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:34:41.218 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:34:41.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:41.256 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:41.256 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:34:41.357 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:34:41.379 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:41.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:41.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:34:41.446 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 68 16 99669 99653 91.5847 43.375 0.00431478 0.0109133 2024-08-19T14:34:41.481 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:34:41.503 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:41.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:41.504 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:34:41.563 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":1464,"stamp":"2024-08-19T14:34:34.799901+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6051856928,"num_objects":92337,"num_object_clones":0,"num_object_copies":277011,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34487,"num_objects_misplaced":20663,"num_objects_unfound":0,"num_objects_dirty":92337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":92335,"num_write_kb":5909440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2864,"num_bytes_recovered":188355104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96644,"ondisk_log_size":96644,"up":126,"acting":127,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":156,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":26471264,"kb_used_data":18466760,"kb_used_omap":15,"kb_used_meta":8004208,"kb_avail":910980256,"statfs":{"total":959950356480,"available":932843782144,"internally_reserved":0,"allocated":18909962240,"data_stored":18882342708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15948,"internal_metadata":8196309428},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,3,4,6,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":457310208,"num_objects":6978,"num_object_clones":0,"num_object_copies":20934,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":-1545,"num_objects_unfound":0,"num_objects_dirty":6978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6978,"num_write_kb":446592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1545,"num_bytes_recovered":101253120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.661400"},"pg_stats":[{"pgid":"7.6","version":"623'6013","reported_seq":6067,"reported_epoch":623,"state":"active+clean","last_fresh":"2024-08-19T14:34:34.001126+0000","last_change":"2024-08-19T14:33:28.334623+0000","last_active":"2024-08-19T14:34:34.001126+0000","last_peered":"2024-08-19T14:34:34.001126+0000","last_clean":"2024-08-19T14:34:34.001126+0000","last_became_active":"2024-08-19T14:33:28.334317+0000","last_became_peered":"2024-08-19T14:33:28.334317+0000","last_unstale":"2024-08-19T14:34:34.001126+0000","last_undegraded":"2024-08-19T14:34:34.001126+0000","last_fullsized":"2024-08-19T14:34:34.001126+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":600,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":6013,"log_dups_size":0,"ondisk_log_size":6013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:34:42.721427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":394067968,"num_objects":6013,"num_object_clones":0,"num_object_copies":18039,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6013,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6013,"num_write_kb":384832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"70'139","reported_seq":1538,"reported_epoch":623,"state":"active+clean+remapped","last_fresh":"2024-08-19T14:34:29.799529+0000","last_change":"2024-08-19T14:34:15.730175+0000","last_active":"2024-08-19T14:34:29.799529+0000","last_peered":"2024-08-19T14:34:29.799529+0000","last_clean":"2024-08-19T14:34:29.799529+0000","last_became_active":"2024-08-19T14:34:15.729665+0000","last_became_peered":"2024-08-19T14:34:15.729665+0000","last_unstale":"2024-08-19T14:34:29.799529+0000","last_undegraded":"2024-08-19T14:34:29.799529+0000","last_fullsized":"2024-08-19T14:34:29.799529+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:33:22.643056+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:19:58.072083+0000","last_clean_scrub_stamp":"2024-08-19T14:33:22.643056+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:34:31.767273+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9],"acting":[0,9,7],"avail_no_missing":["0","7","9"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.5","version":"623'5605","reported_seq":7051,"reported_epoch":623,"state":"active+recovering+undersized+remapped","last_fresh":"2024-08-19T14:34:31.359392+0000","last_change":"2024-08-19T14:34:13.959798+0000","last_active":"2024-08-19T14:34:31.359392+0000","last_peered":"2024-08-19T14:34:31.359392+0000","last_clean":"2024-08-19T14:34:11.695952+0000","last_became_active":"2024-08-19T14:34:13.905113+0000","last_became_peered":"2024-08-19T14:34:13.905113+0000","last_unstale":"2024-08-19T14:34:31.359392+0000","last_undegraded":"2024-08-19T14:34:31.359392+0000","last_fullsized":"2024-08-19T14:34:13.697426+0000","mapping_epoch":617,"log_start":"0'0","ondisk_log_start":"0'0","created":600,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":5605,"log_dups_size":0,"ondisk_log_size":5605,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":367329280,"num_objects":5605,"num_object_clones":0,"num_object_copies":16815,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3941,"num_objects_unfound":0,"num_objects_dirty":5605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5605,"num_write_kb":358720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":693,"num_bytes_recovered":45416448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,0],"acting":[11,0],"avail_no_missing":["11","0","1"],"object_location_counts":[{"shards":"0,1,11","objects":3941},{"shards":"0,11","objects":1664}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.3","version":"70'139","reported_seq":1407,"reported_epoch":623,"state":"active+clean","last_fresh":"2024-08-19T14:34:29.775603+0000","last_change":"2024-08-19T14:33:18.804321+0000","last_active":"2024-08-19T14:34:29.775603+0000","last_peered":"2024-08-19T14:34:29.775603+0000","last_clean":"2024-08-19T14:34:29.775603+0000","last_became_active":"2024-08-19T14:18:13.804721+0000","last_became_peered":"2024-08-19T14:18:13.804721+0000","last_unstale":"2024-08-19T14:34:29.775603+0000","last_undegraded":"2024-08-19T14:34:29.775603+0000","last_fullsized":"2024-08-19T14:34:29.775603+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:33:18.804228+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:33:18.804228+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:34:33.153519+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.4","version":"623'5921","reported_seq":5977,"reported_epoch":623,"state":"active+clean","last_fresh":"2024-08-19T14:34:34.019491+0000","last_change":"2024-08-19T14:33:28.334772+0000","last_active":"2024-08-19T14:34:34.019491+0000","last_peered":"2024-08-19T14:34:34.019491+0000","last_clean":"2024-08-19T14:34:34.019491+0000","last_became_active":"2024-08-19T14:33:28.334446+0000","last_became_peered":"2024-08-19T14:33:28.334446+0000","last_unstale":"2024-08-19T14:34:34.019491+0000","last_undegraded":"2024-08-19T14:34:34.019491+0000","last_fullsized":"2024-08-19T14:34:34.019491+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":600,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":5921,"log_dups_size":0,"ondisk_log_size":5921,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":388038656,"num_objects":5921,"num_object_clones":0,"num_object_copies":17763,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5921,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5921,"num_write_kb":378944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"70'139","reported_seq":1355,"reported_epoch":623,"state":"active+clean","last_fresh":"2024-08-19T14:34:29.778354+0000","last_change":"2024-08-19T14:33:16.809679+0000","last_active":"2024-08-19T14:34:29.778354+0000","last_peered":"2024-08-19T14:34:29.778354+0000","last_clean":"2024-08-19T14:34:29.778354+0000","last_became_active":"2024-08-19T14:20:50.742354+0000","last_became_peered":"2024-08-19T14:20:50.742354+0000","last_unstale":"2024-08-19T14:34:29.778354+0000","last_undegraded":"2024-08-19T14:34:29.778354+0000","last_fullsized":"2024-08-19T14:34:29.778354+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:33:16.809603+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:19:59.302931+0000","last_clean_scrub_stamp":"2024-08-19T14:33:16.809603+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:34:34.302113+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"623'5515","reported_seq":5571,"reported_epoch":623,"state":"active+clean","last_fresh":"2024-08-19T14:34:30.212084+0000","last_change":"2024-08-19T14:33:28.332172+0000","last_active":"2024-08-19T14:34:30.212084+0000","last_peered":"2024-08-19T14:34:30.212084+0000","last_clean":"2024-08-19T14:34:30.212084+0000","last_became_active":"2024-08-19T14:33:28.331833+0000","last_became_peered":"2024-08-19T14:33:28.331833+0000","last_unstale":"2024-08-19T14:34:30.212084+0000","last_undegraded":"2024-08-19T14:34:30.212084+0000","last_fullsized":"2024-08-19T14:34:30.212084+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":600,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":5515,"log_dups_size":0,"ondisk_log_size":5515,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:34:47.240599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":361431040,"num_objects":5515,"num_object_clones":0,"num_object_copies":16545,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5515,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5515,"num_write_kb":352960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,1],"acting":[8,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1","version":"70'139","reported_seq":640,"reported_epoch":623,"state":"active+clean+remapped","last_fresh":"2024-08-19T14:34:29.803331+0000","last_change":"2024-08-19T14:34:15.731463+0000","last_active":"2024-08-19T14:34:29.803331+0000","last_peered":"2024-08-19T14:34:29.803331+0000","last_clean":"2024-08-19T14:34:29.803331+0000","last_became_active":"2024-08-19T14:34:15.730611+0000","last_became_peered":"2024-08-19T14:34:15.730611+0000","last_unstale":"2024-08-19T14:34:29.803331+0000","last_undegraded":"2024-08-19T14:34:29.803331+0000","last_fullsized":"2024-08-19T14:34:29.803331+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:28.323773+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:28:28.323773+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9],"acting":[0,9,11],"avail_no_missing":["0","9","11"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"623'5790","reported_seq":5846,"reported_epoch":623,"state":"active+clean","last_fresh":"2024-08-19T14:34:33.664611+0000","last_change":"2024-08-19T14:33:28.333311+0000","last_active":"2024-08-19T14:34:33.664611+0000","last_peered":"2024-08-19T14:34:33.664611+0000","last_clean":"2024-08-19T14:34:33.664611+0000","last_became_active":"2024-08-19T14:33:28.332994+0000","last_became_peered":"2024-08-19T14:33:28.332994+0000","last_unstale":"2024-08-19T14:34:33.664611+0000","last_undegraded":"2024-08-19T14:34:33.664611+0000","last_fullsized":"2024-08-19T14:34:33.664611+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":600,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":5790,"log_dups_size":0,"ondisk_log_size":5790,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:34:35.125863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":379453440,"num_objects":5790,"num_object_clones":0,"num_object_copies":17370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5790,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5790,"num_write_kb":370560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"70'139","reported_seq":1059,"reported_epoch":623,"state":"active+undersized","last_fresh":"2024-08-19T14:34:29.772129+0000","last_change":"2024-08-19T14:31:02.880951+0000","last_active":"2024-08-19T14:34:29.772129+0000","last_peered":"2024-08-19T14:34:29.772129+0000","last_clean":"2024-08-19T14:30:57.035896+0000","last_became_active":"2024-08-19T14:31:02.880951+0000","last_became_peered":"2024-08-19T14:31:02.880951+0000","last_unstale":"2024-08-19T14:34:29.772129+0000","last_undegraded":"2024-08-19T14:34:29.772129+0000","last_fullsized":"2024-08-19T14:31:02.878172+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:27.536928+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:21:46.057624+0000","last_clean_scrub_stamp":"2024-08-19T14:28:27.536928+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"623'5507","reported_seq":6401,"reported_epoch":623,"state":"active+recovering+undersized+remapped","last_fresh":"2024-08-19T14:34:30.411338+0000","last_change":"2024-08-19T14:34:13.999039+0000","last_active":"2024-08-19T14:34:30.411338+0000","last_peered":"2024-08-19T14:34:30.411338+0000","last_clean":"2024-08-19T14:34:11.693578+0000","last_became_active":"2024-08-19T14:34:13.927424+0000","last_became_peered":"2024-08-19T14:34:13.927424+0000","last_unstale":"2024-08-19T14:34:30.411338+0000","last_undegraded":"2024-08-19T14:34:30.411338+0000","last_fullsized":"2024-08-19T14:34:13.697733+0000","mapping_epoch":617,"log_start":"0'0","ondisk_log_start":"0'0","created":600,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":5507,"log_dups_size":0,"ondisk_log_size":5507,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":360906752,"num_objects":5507,"num_object_clones":0,"num_object_copies":16521,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4194,"num_objects_unfound":0,"num_objects_dirty":5507,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5507,"num_write_kb":352448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":417,"num_bytes_recovered":27328512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,9],"acting":[3,5],"avail_no_missing":["3","1","5"],"object_location_counts":[{"shards":"1,3,5","objects":4194},{"shards":"3,5","objects":1313}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"70'139","reported_seq":618,"reported_epoch":623,"state":"active+clean","last_fresh":"2024-08-19T14:34:29.793924+0000","last_change":"2024-08-19T14:34:15.717448+0000","last_active":"2024-08-19T14:34:29.793924+0000","last_peered":"2024-08-19T14:34:29.793924+0000","last_clean":"2024-08-19T14:34:29.793924+0000","last_became_active":"2024-08-19T14:34:15.717054+0000","last_became_peered":"2024-08-19T14:34:15.717054+0000","last_unstale":"2024-08-19T14:34:29.793924+0000","last_undegraded":"2024-08-19T14:34:29.793924+0000","last_fullsized":"2024-08-19T14:34:29.793924+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:33:22.132750+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:27.848870+0000","last_clean_scrub_stamp":"2024-08-19T14:33:22.132750+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:34:51.143183+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"7.3","version":"623'5768","reported_seq":5821,"reported_epoch":623,"state":"active+undersized+degraded","last_fresh":"2024-08-19T14:34:30.347765+0000","last_change":"2024-08-19T14:33:33.456029+0000","last_active":"2024-08-19T14:34:30.347765+0000","last_peered":"2024-08-19T14:34:30.347765+0000","last_clean":"2024-08-19T14:33:27.312789+0000","last_became_active":"2024-08-19T14:33:28.332941+0000","last_became_peered":"2024-08-19T14:33:28.332941+0000","last_unstale":"2024-08-19T14:34:30.347765+0000","last_undegraded":"2024-08-19T14:33:33.451569+0000","last_fullsized":"2024-08-19T14:33:28.326677+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":600,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":5768,"log_dups_size":0,"ondisk_log_size":5768,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:34:38.018513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":378011648,"num_objects":5768,"num_object_clones":0,"num_object_copies":17304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5768,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5768,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5768,"num_write_kb":369152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":5768}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"70'139","reported_seq":581,"reported_epoch":623,"state":"active+undersized","last_fresh":"2024-08-19T14:34:29.786071+0000","last_change":"2024-08-19T14:31:02.876003+0000","last_active":"2024-08-19T14:34:29.786071+0000","last_peered":"2024-08-19T14:34:29.786071+0000","last_clean":"2024-08-19T14:30:57.017098+0000","last_became_active":"2024-08-19T14:31:02.876003+0000","last_became_peered":"2024-08-19T14:31:02.876003+0000","last_unstale":"2024-08-19T14:34:29.786071+0000","last_undegraded":"2024-08-19T14:34:29.786071+0000","last_fullsized":"2024-08-19T14:31:02.871830+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:27.331041+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:28:27.331041+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":["11","0"],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.2","version":"623'5699","reported_seq":5752,"reported_epoch":623,"state":"active+undersized+degraded","last_fresh":"2024-08-19T14:34:30.421658+0000","last_change":"2024-08-19T14:33:33.452955+0000","last_active":"2024-08-19T14:34:30.421658+0000","last_peered":"2024-08-19T14:34:30.421658+0000","last_clean":"2024-08-19T14:33:27.312789+0000","last_became_active":"2024-08-19T14:33:28.331836+0000","last_became_peered":"2024-08-19T14:33:28.331836+0000","last_unstale":"2024-08-19T14:34:30.421658+0000","last_undegraded":"2024-08-19T14:33:33.452783+0000","last_fullsized":"2024-08-19T14:33:28.326610+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":600,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":5699,"log_dups_size":0,"ondisk_log_size":5699,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:34:38.523026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":373489664,"num_objects":5699,"num_object_clones":0,"num_object_copies":17097,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5699,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5699,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5699,"num_write_kb":364736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":5699}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"70'139","reported_seq":464,"reported_epoch":623,"state":"active+clean","last_fresh":"2024-08-19T14:34:29.781425+0000","last_change":"2024-08-19T14:33:16.760928+0000","last_active":"2024-08-19T14:34:29.781425+0000","last_peered":"2024-08-19T14:34:29.781425+0000","last_clean":"2024-08-19T14:34:29.781425+0000","last_became_active":"2024-08-19T14:30:47.909067+0000","last_became_peered":"2024-08-19T14:30:47.909067+0000","last_unstale":"2024-08-19T14:34:29.781425+0000","last_undegraded":"2024-08-19T14:34:29.781425+0000","last_fullsized":"2024-08-19T14:34:29.781425+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:33:16.760810+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:33:16.760810+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:34:42.688716+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.c","version":"623'5594","reported_seq":5647,"reported_epoch":623,"state":"active+undersized+degraded","last_fresh":"2024-08-19T14:34:30.411511+0000","last_change":"2024-08-19T14:33:33.445104+0000","last_active":"2024-08-19T14:34:30.411511+0000","last_peered":"2024-08-19T14:34:30.411511+0000","last_clean":"2024-08-19T14:33:27.312789+0000","last_became_active":"2024-08-19T14:33:28.328685+0000","last_became_peered":"2024-08-19T14:33:28.328685+0000","last_unstale":"2024-08-19T14:34:30.411511+0000","last_undegraded":"2024-08-19T14:33:33.444906+0000","last_fullsized":"2024-08-19T14:33:28.326644+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":600,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":5594,"log_dups_size":0,"ondisk_log_size":5594,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:34:51.992967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":366608384,"num_objects":5594,"num_object_clones":0,"num_object_copies":16782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5594,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5594,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5594,"num_write_kb":358016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11],"acting":[3,11],"avail_no_missing":["3","11"],"object_location_counts":[{"shards":"3,11","objects":5594}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"70'139","reported_seq":622,"reported_epoch":623,"state":"active+clean","last_fresh":"2024-08-19T14:34:29.771162+0000","last_change":"2024-08-19T14:34:15.722633+0000","last_active":"2024-08-19T14:34:29.771162+0000","last_peered":"2024-08-19T14:34:29.771162+0000","last_clean":"2024-08-19T14:34:29.771162+0000","last_became_active":"2024-08-19T14:34:15.717472+0000","last_became_peered":"2024-08-19T14:34:15.717472+0000","last_unstale":"2024-08-19T14:34:29.771162+0000","last_undegraded":"2024-08-19T14:34:29.771162+0000","last_fullsized":"2024-08-19T14:34:29.771162+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:50.681699+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:28.864870+0000","last_clean_scrub_stamp":"2024-08-19T14:28:50.681699+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:34:34.136612+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,0],"acting":[4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"623'5812","reported_seq":6800,"reported_epoch":623,"state":"active+recovering+undersized+remapped","last_fresh":"2024-08-19T14:34:34.036846+0000","last_change":"2024-08-19T14:34:13.986629+0000","last_active":"2024-08-19T14:34:34.036846+0000","last_peered":"2024-08-19T14:34:34.036846+0000","last_clean":"2024-08-19T14:34:11.688031+0000","last_became_active":"2024-08-19T14:34:13.931523+0000","last_became_peered":"2024-08-19T14:34:13.931523+0000","last_unstale":"2024-08-19T14:34:34.036846+0000","last_undegraded":"2024-08-19T14:34:34.036846+0000","last_fullsized":"2024-08-19T14:34:13.705235+0000","mapping_epoch":617,"log_start":"0'0","ondisk_log_start":"0'0","created":600,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":5812,"log_dups_size":0,"ondisk_log_size":5812,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":380895232,"num_objects":5812,"num_object_clones":0,"num_object_copies":17436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4236,"num_objects_unfound":0,"num_objects_dirty":5812,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5812,"num_write_kb":371968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":464,"num_bytes_recovered":30408704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8],"avail_no_missing":["0","8","11"],"object_location_counts":[{"shards":"0,8","objects":1576},{"shards":"0,8,11","objects":4236}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.9","version":"70'139","reported_seq":1552,"reported_epoch":623,"state":"active+clean+remapped","last_fresh":"2024-08-19T14:34:29.776714+0000","last_change":"2024-08-19T14:34:15.719277+0000","last_active":"2024-08-19T14:34:29.776714+0000","last_peered":"2024-08-19T14:34:29.776714+0000","last_clean":"2024-08-19T14:34:29.776714+0000","last_became_active":"2024-08-19T14:34:15.718870+0000","last_became_peered":"2024-08-19T14:34:15.718870+0000","last_unstale":"2024-08-19T14:34:29.776714+0000","last_undegraded":"2024-08-19T14:34:29.776714+0000","last_fullsized":"2024-08-19T14:34:29.776714+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:53.273092+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:28:53.273092+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:34:34.302024+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9,5],"avail_no_missing":["7","5","9"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.e","version":"623'5967","reported_seq":7107,"reported_epoch":623,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-19T14:34:34.290155+0000","last_change":"2024-08-19T14:34:13.957005+0000","last_active":"2024-08-19T14:34:34.290155+0000","last_peered":"2024-08-19T14:34:34.290155+0000","last_clean":"2024-08-19T14:33:27.312789+0000","last_became_active":"2024-08-19T14:34:13.912909+0000","last_became_peered":"2024-08-19T14:34:13.912909+0000","last_unstale":"2024-08-19T14:34:34.290155+0000","last_undegraded":"2024-08-19T14:34:13.728312+0000","last_fullsized":"2024-08-19T14:34:13.697670+0000","mapping_epoch":617,"log_start":"0'0","ondisk_log_start":"0'0","created":600,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":5967,"log_dups_size":0,"ondisk_log_size":5967,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":391053312,"num_objects":5967,"num_object_clones":0,"num_object_copies":17901,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5967,"num_objects_misplaced":4319,"num_objects_unfound":0,"num_objects_dirty":5967,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5967,"num_write_kb":381888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":543,"num_bytes_recovered":35586048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":1648}],"blocked_by":[],"up_primary":9,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"70'139","reported_seq":590,"reported_epoch":623,"state":"active+undersized","last_fresh":"2024-08-19T14:34:29.788468+0000","last_change":"2024-08-19T14:31:02.882810+0000","last_active":"2024-08-19T14:34:29.788468+0000","last_peered":"2024-08-19T14:34:29.788468+0000","last_clean":"2024-08-19T14:30:57.035787+0000","last_became_active":"2024-08-19T14:31:02.882810+0000","last_became_peered":"2024-08-19T14:31:02.882810+0000","last_unstale":"2024-08-19T14:34:29.788468+0000","last_undegraded":"2024-08-19T14:34:29.788468+0000","last_fullsized":"2024-08-19T14:31:02.877995+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:59.829567+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:28:59.829567+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"623'5957","reported_seq":6013,"reported_epoch":623,"state":"active+clean","last_fresh":"2024-08-19T14:34:34.243484+0000","last_change":"2024-08-19T14:33:28.335586+0000","last_active":"2024-08-19T14:34:34.243484+0000","last_peered":"2024-08-19T14:34:34.243484+0000","last_clean":"2024-08-19T14:34:34.243484+0000","last_became_active":"2024-08-19T14:33:28.335298+0000","last_became_peered":"2024-08-19T14:33:28.335298+0000","last_unstale":"2024-08-19T14:34:34.243484+0000","last_undegraded":"2024-08-19T14:34:34.243484+0000","last_fullsized":"2024-08-19T14:34:34.243484+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":600,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":5957,"log_dups_size":0,"ondisk_log_size":5957,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:34:46.597925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":390397952,"num_objects":5957,"num_object_clones":0,"num_object_copies":17871,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5957,"num_write_kb":381248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"70'139","reported_seq":916,"reported_epoch":623,"state":"active+clean","last_fresh":"2024-08-19T14:34:29.779638+0000","last_change":"2024-08-19T14:33:15.753668+0000","last_active":"2024-08-19T14:34:29.779638+0000","last_peered":"2024-08-19T14:34:29.779638+0000","last_clean":"2024-08-19T14:34:29.779638+0000","last_became_active":"2024-08-19T14:30:47.941200+0000","last_became_peered":"2024-08-19T14:30:47.941200+0000","last_unstale":"2024-08-19T14:34:29.779638+0000","last_undegraded":"2024-08-19T14:34:29.779638+0000","last_fullsized":"2024-08-19T14:34:29.779638+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:33:15.753595+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:33:15.753595+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:34:35.515236+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.8","version":"623'5794","reported_seq":5847,"reported_epoch":623,"state":"active+undersized+degraded","last_fresh":"2024-08-19T14:34:30.243257+0000","last_change":"2024-08-19T14:33:33.448688+0000","last_active":"2024-08-19T14:34:30.243257+0000","last_peered":"2024-08-19T14:34:30.243257+0000","last_clean":"2024-08-19T14:33:27.312789+0000","last_became_active":"2024-08-19T14:33:28.331962+0000","last_became_peered":"2024-08-19T14:33:28.331962+0000","last_unstale":"2024-08-19T14:34:30.243257+0000","last_undegraded":"2024-08-19T14:33:33.446348+0000","last_fullsized":"2024-08-19T14:33:28.330096+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":600,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":5794,"log_dups_size":0,"ondisk_log_size":5794,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:34:37.043839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":379715584,"num_objects":5794,"num_object_clones":0,"num_object_copies":17382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5794,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5794,"num_write_kb":370816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[{"shards":"4,7","objects":5794}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.e","version":"70'139","reported_seq":1099,"reported_epoch":623,"state":"active+clean","last_fresh":"2024-08-19T14:34:29.771358+0000","last_change":"2024-08-19T14:33:36.162422+0000","last_active":"2024-08-19T14:34:29.771358+0000","last_peered":"2024-08-19T14:34:29.771358+0000","last_clean":"2024-08-19T14:34:29.771358+0000","last_became_active":"2024-08-19T14:18:26.884105+0000","last_became_peered":"2024-08-19T14:18:26.884105+0000","last_unstale":"2024-08-19T14:34:29.771358+0000","last_undegraded":"2024-08-19T14:34:29.771358+0000","last_fullsized":"2024-08-19T14:34:29.771358+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:33:36.162350+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:32:31.196288+0000","last_clean_scrub_stamp":"2024-08-19T14:33:36.162350+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:34:46.393050+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"623'5908","reported_seq":5963,"reported_epoch":623,"state":"active+clean","last_fresh":"2024-08-19T14:34:34.023916+0000","last_change":"2024-08-19T14:33:28.334891+0000","last_active":"2024-08-19T14:34:34.023916+0000","last_peered":"2024-08-19T14:34:34.023916+0000","last_clean":"2024-08-19T14:34:34.023916+0000","last_became_active":"2024-08-19T14:33:28.334547+0000","last_became_peered":"2024-08-19T14:33:28.334547+0000","last_unstale":"2024-08-19T14:34:34.023916+0000","last_undegraded":"2024-08-19T14:34:34.023916+0000","last_fullsized":"2024-08-19T14:34:34.023916+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":600,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":5908,"log_dups_size":0,"ondisk_log_size":5908,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:34:49.148278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":387186688,"num_objects":5908,"num_object_clones":0,"num_object_copies":17724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5908,"num_write_kb":378112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,8],"acting":[0,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"70'139","reported_seq":859,"reported_epoch":623,"state":"active+clean","last_fresh":"2024-08-19T14:34:29.771341+0000","last_change":"2024-08-19T14:34:12.199530+0000","last_active":"2024-08-19T14:34:29.771341+0000","last_peered":"2024-08-19T14:34:29.771341+0000","last_clean":"2024-08-19T14:34:29.771341+0000","last_became_active":"2024-08-19T14:27:31.837754+0000","last_became_peered":"2024-08-19T14:27:31.837754+0000","last_unstale":"2024-08-19T14:34:29.771341+0000","last_undegraded":"2024-08-19T14:34:29.771341+0000","last_fullsized":"2024-08-19T14:34:29.771341+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:34:12.199449+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:34:12.199449+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:35:21.121327+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.a","version":"623'5664","reported_seq":7203,"reported_epoch":623,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-19T14:34:31.359913+0000","last_change":"2024-08-19T14:34:13.909644+0000","last_active":"2024-08-19T14:34:31.359913+0000","last_peered":"2024-08-19T14:34:31.359913+0000","last_clean":"2024-08-19T14:33:27.312789+0000","last_became_active":"2024-08-19T14:34:13.895029+0000","last_became_peered":"2024-08-19T14:34:13.895029+0000","last_unstale":"2024-08-19T14:34:31.359913+0000","last_undegraded":"2024-08-19T14:34:13.728710+0000","last_fullsized":"2024-08-19T14:34:13.697497+0000","mapping_epoch":617,"log_start":"0'0","ondisk_log_start":"0'0","created":600,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":5664,"log_dups_size":0,"ondisk_log_size":5664,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":371195904,"num_objects":5664,"num_object_clones":0,"num_object_copies":16992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5664,"num_objects_misplaced":3973,"num_objects_unfound":0,"num_objects_dirty":5664,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5664,"num_write_kb":362496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":744,"num_bytes_recovered":48758784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,3],"avail_no_missing":["11","3"],"object_location_counts":[{"shards":"3,11","objects":1691}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.c","version":"70'139","reported_seq":1214,"reported_epoch":623,"state":"active+clean","last_fresh":"2024-08-19T14:34:29.772796+0000","last_change":"2024-08-19T14:34:15.738488+0000","last_active":"2024-08-19T14:34:29.772796+0000","last_peered":"2024-08-19T14:34:29.772796+0000","last_clean":"2024-08-19T14:34:29.772796+0000","last_became_active":"2024-08-19T14:34:15.733282+0000","last_became_peered":"2024-08-19T14:34:15.733282+0000","last_unstale":"2024-08-19T14:34:29.772796+0000","last_undegraded":"2024-08-19T14:34:29.772796+0000","last_fullsized":"2024-08-19T14:34:29.772796+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:33:17.453290+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:22:52.164586+0000","last_clean_scrub_stamp":"2024-08-19T14:33:17.453290+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:34:33.115129+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,9],"acting":[3,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"623'5821","reported_seq":5877,"reported_epoch":623,"state":"active+clean","last_fresh":"2024-08-19T14:34:34.262302+0000","last_change":"2024-08-19T14:33:28.623717+0000","last_active":"2024-08-19T14:34:34.262302+0000","last_peered":"2024-08-19T14:34:34.262302+0000","last_clean":"2024-08-19T14:34:34.262302+0000","last_became_active":"2024-08-19T14:33:28.623318+0000","last_became_peered":"2024-08-19T14:33:28.623318+0000","last_unstale":"2024-08-19T14:34:34.262302+0000","last_undegraded":"2024-08-19T14:34:34.262302+0000","last_fullsized":"2024-08-19T14:34:34.262302+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":600,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":5821,"log_dups_size":0,"ondisk_log_size":5821,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:34:44.425091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381485056,"num_objects":5821,"num_object_clones":0,"num_object_copies":17463,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5821,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5821,"num_write_kb":372544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"70'139","reported_seq":1103,"reported_epoch":623,"state":"active+clean","last_fresh":"2024-08-19T14:34:29.777006+0000","last_change":"2024-08-19T14:33:18.729718+0000","last_active":"2024-08-19T14:34:29.777006+0000","last_peered":"2024-08-19T14:34:29.777006+0000","last_clean":"2024-08-19T14:34:29.777006+0000","last_became_active":"2024-08-19T14:18:26.882002+0000","last_became_peered":"2024-08-19T14:18:26.882002+0000","last_unstale":"2024-08-19T14:34:29.777006+0000","last_undegraded":"2024-08-19T14:34:29.777006+0000","last_fullsized":"2024-08-19T14:34:29.777006+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:33:18.729624+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:24:50.276408+0000","last_clean_scrub_stamp":"2024-08-19T14:33:18.729624+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:34:47.843417+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"70'139","reported_seq":890,"reported_epoch":623,"state":"active+clean","last_fresh":"2024-08-19T14:34:29.775417+0000","last_change":"2024-08-19T14:33:19.781527+0000","last_active":"2024-08-19T14:34:29.775417+0000","last_peered":"2024-08-19T14:34:29.775417+0000","last_clean":"2024-08-19T14:34:29.775417+0000","last_became_active":"2024-08-19T14:27:31.838968+0000","last_became_peered":"2024-08-19T14:27:31.838968+0000","last_unstale":"2024-08-19T14:34:29.775417+0000","last_undegraded":"2024-08-19T14:34:29.775417+0000","last_fullsized":"2024-08-19T14:34:29.775417+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:33:19.781431+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:28:35.206115+0000","last_clean_scrub_stamp":"2024-08-19T14:33:19.781431+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:34:46.075916+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.13","version":"70'139","reported_seq":1431,"reported_epoch":623,"state":"active+clean","last_fresh":"2024-08-19T14:34:29.775499+0000","last_change":"2024-08-19T14:33:35.786236+0000","last_active":"2024-08-19T14:34:29.775499+0000","last_peered":"2024-08-19T14:34:29.775499+0000","last_clean":"2024-08-19T14:34:29.775499+0000","last_became_active":"2024-08-19T14:30:47.939849+0000","last_became_peered":"2024-08-19T14:30:47.939849+0000","last_unstale":"2024-08-19T14:34:29.775499+0000","last_undegraded":"2024-08-19T14:34:29.775499+0000","last_fullsized":"2024-08-19T14:34:29.775499+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:33:35.786105+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:33:35.786105+0000","last_clean_scrub_stamp":"2024-08-19T14:33:35.786105+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:34:52.883706+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.10","version":"70'139","reported_seq":1051,"reported_epoch":623,"state":"active+clean","last_fresh":"2024-08-19T14:34:29.776798+0000","last_change":"2024-08-19T14:33:19.700406+0000","last_active":"2024-08-19T14:34:29.776798+0000","last_peered":"2024-08-19T14:34:29.776798+0000","last_clean":"2024-08-19T14:34:29.776798+0000","last_became_active":"2024-08-19T14:27:31.839611+0000","last_became_peered":"2024-08-19T14:27:31.839611+0000","last_unstale":"2024-08-19T14:34:29.776798+0000","last_undegraded":"2024-08-19T14:34:29.776798+0000","last_fullsized":"2024-08-19T14:34:29.776798+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:33:19.700304+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:33:19.700304+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:34:45.176186+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.11","version":"70'139","reported_seq":1099,"reported_epoch":623,"state":"active+clean","last_fresh":"2024-08-19T14:34:29.799773+0000","last_change":"2024-08-19T14:34:15.731788+0000","last_active":"2024-08-19T14:34:29.799773+0000","last_peered":"2024-08-19T14:34:29.799773+0000","last_clean":"2024-08-19T14:34:29.799773+0000","last_became_active":"2024-08-19T14:34:15.731049+0000","last_became_peered":"2024-08-19T14:34:15.731049+0000","last_unstale":"2024-08-19T14:34:29.799773+0000","last_undegraded":"2024-08-19T14:34:29.799773+0000","last_fullsized":"2024-08-19T14:34:29.799773+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:34:01.957281+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:24:50.547428+0000","last_clean_scrub_stamp":"2024-08-19T14:34:01.957281+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:35:17.208355+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"70'139","reported_seq":1042,"reported_epoch":623,"state":"active+clean","last_fresh":"2024-08-19T14:34:29.779565+0000","last_change":"2024-08-19T14:33:27.737496+0000","last_active":"2024-08-19T14:34:29.779565+0000","last_peered":"2024-08-19T14:34:29.779565+0000","last_clean":"2024-08-19T14:34:29.779565+0000","last_became_active":"2024-08-19T14:20:28.204724+0000","last_became_peered":"2024-08-19T14:20:28.204724+0000","last_unstale":"2024-08-19T14:34:29.779565+0000","last_undegraded":"2024-08-19T14:34:29.779565+0000","last_fullsized":"2024-08-19T14:34:29.779565+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:33:27.737367+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:28:34.216370+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.737367+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:34:33.896638+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"70'139","reported_seq":1170,"reported_epoch":623,"state":"active+clean","last_fresh":"2024-08-19T14:34:29.793956+0000","last_change":"2024-08-19T14:33:20.155915+0000","last_active":"2024-08-19T14:34:29.793956+0000","last_peered":"2024-08-19T14:34:29.793956+0000","last_clean":"2024-08-19T14:34:29.793956+0000","last_became_active":"2024-08-19T14:18:31.901556+0000","last_became_peered":"2024-08-19T14:18:31.901556+0000","last_unstale":"2024-08-19T14:34:29.793956+0000","last_undegraded":"2024-08-19T14:34:29.793956+0000","last_fullsized":"2024-08-19T14:34:29.793956+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:33:20.155832+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:33:20.155832+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:34:41.776710+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,0],"acting":[8,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.14","version":"70'139","reported_seq":1201,"reported_epoch":623,"state":"active+clean+remapped","last_fresh":"2024-08-19T14:34:29.777004+0000","last_change":"2024-08-19T14:34:15.728836+0000","last_active":"2024-08-19T14:34:29.777004+0000","last_peered":"2024-08-19T14:34:29.777004+0000","last_clean":"2024-08-19T14:34:29.777004+0000","last_became_active":"2024-08-19T14:34:15.728412+0000","last_became_peered":"2024-08-19T14:34:15.728412+0000","last_unstale":"2024-08-19T14:34:29.777004+0000","last_undegraded":"2024-08-19T14:34:29.777004+0000","last_fullsized":"2024-08-19T14:34:29.777004+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:58.211358+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:28:58.211358+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:34:34.302063+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9,1],"avail_no_missing":["7","1","9"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.15","version":"70'139","reported_seq":1058,"reported_epoch":623,"state":"active+clean","last_fresh":"2024-08-19T14:34:29.776952+0000","last_change":"2024-08-19T14:33:17.777890+0000","last_active":"2024-08-19T14:34:29.776952+0000","last_peered":"2024-08-19T14:34:29.776952+0000","last_clean":"2024-08-19T14:34:29.776952+0000","last_became_active":"2024-08-19T14:18:31.912496+0000","last_became_peered":"2024-08-19T14:18:31.912496+0000","last_unstale":"2024-08-19T14:34:29.776952+0000","last_undegraded":"2024-08-19T14:34:29.776952+0000","last_fullsized":"2024-08-19T14:34:29.776952+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:33:17.777730+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:20:09.180923+0000","last_clean_scrub_stamp":"2024-08-19T14:33:17.777730+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:34:31.456837+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"70'139","reported_seq":1052,"reported_epoch":623,"state":"active+clean","last_fresh":"2024-08-19T14:34:29.770973+0000","last_change":"2024-08-19T14:34:06.176510+0000","last_active":"2024-08-19T14:34:29.770973+0000","last_peered":"2024-08-19T14:34:29.770973+0000","last_clean":"2024-08-19T14:34:29.770973+0000","last_became_active":"2024-08-19T14:18:31.901229+0000","last_became_peered":"2024-08-19T14:18:31.901229+0000","last_unstale":"2024-08-19T14:34:29.770973+0000","last_undegraded":"2024-08-19T14:34:29.770973+0000","last_fullsized":"2024-08-19T14:34:29.770973+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:34:06.176403+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:34:06.176403+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:35:34.700821+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1b","version":"70'139","reported_seq":1418,"reported_epoch":623,"state":"active+clean","last_fresh":"2024-08-19T14:34:29.793709+0000","last_change":"2024-08-19T14:34:15.718786+0000","last_active":"2024-08-19T14:34:29.793709+0000","last_peered":"2024-08-19T14:34:29.793709+0000","last_clean":"2024-08-19T14:34:29.793709+0000","last_became_active":"2024-08-19T14:34:15.718427+0000","last_became_peered":"2024-08-19T14:34:15.718427+0000","last_unstale":"2024-08-19T14:34:29.793709+0000","last_undegraded":"2024-08-19T14:34:29.793709+0000","last_fullsized":"2024-08-19T14:34:29.793709+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:24.065359+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:28:24.065359+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.18","version":"70'139","reported_seq":988,"reported_epoch":623,"state":"active+undersized+degraded","last_fresh":"2024-08-19T14:34:29.776646+0000","last_change":"2024-08-19T14:31:02.878454+0000","last_active":"2024-08-19T14:34:29.776646+0000","last_peered":"2024-08-19T14:34:29.776646+0000","last_clean":"2024-08-19T14:30:57.018623+0000","last_became_active":"2024-08-19T14:31:02.878454+0000","last_became_peered":"2024-08-19T14:31:02.878454+0000","last_unstale":"2024-08-19T14:34:29.776646+0000","last_undegraded":"2024-08-19T14:31:02.872398+0000","last_fullsized":"2024-08-19T14:31:02.872290+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:29:05.336156+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:27:33.084161+0000","last_clean_scrub_stamp":"2024-08-19T14:29:05.336156+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:34:34.302080+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[{"shards":"5,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.19","version":"70'139","reported_seq":1396,"reported_epoch":623,"state":"active+clean","last_fresh":"2024-08-19T14:34:29.799660+0000","last_change":"2024-08-19T14:32:53.319620+0000","last_active":"2024-08-19T14:34:29.799660+0000","last_peered":"2024-08-19T14:34:29.799660+0000","last_clean":"2024-08-19T14:34:29.799660+0000","last_became_active":"2024-08-19T14:18:31.902547+0000","last_became_peered":"2024-08-19T14:18:31.902547+0000","last_unstale":"2024-08-19T14:34:29.799660+0000","last_undegraded":"2024-08-19T14:34:29.799660+0000","last_fullsized":"2024-08-19T14:34:29.799660+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:32:53.319547+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:32:53.319547+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1e","version":"70'139","reported_seq":436,"reported_epoch":623,"state":"active+clean","last_fresh":"2024-08-19T14:34:29.779623+0000","last_change":"2024-08-19T14:33:17.778415+0000","last_active":"2024-08-19T14:34:29.779623+0000","last_peered":"2024-08-19T14:34:29.779623+0000","last_clean":"2024-08-19T14:34:29.779623+0000","last_became_active":"2024-08-19T14:30:47.909974+0000","last_became_peered":"2024-08-19T14:30:47.909974+0000","last_unstale":"2024-08-19T14:34:29.779623+0000","last_undegraded":"2024-08-19T14:34:29.779623+0000","last_fullsized":"2024-08-19T14:34:29.779623+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:33:17.778295+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:27:04.780939+0000","last_clean_scrub_stamp":"2024-08-19T14:33:17.778295+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:34:46.103360+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1c","version":"70'139","reported_seq":1172,"reported_epoch":623,"state":"active+clean","last_fresh":"2024-08-19T14:34:29.793771+0000","last_change":"2024-08-19T14:33:18.213568+0000","last_active":"2024-08-19T14:34:29.793771+0000","last_peered":"2024-08-19T14:34:29.793771+0000","last_clean":"2024-08-19T14:34:29.793771+0000","last_became_active":"2024-08-19T14:30:47.908866+0000","last_became_peered":"2024-08-19T14:30:47.908866+0000","last_unstale":"2024-08-19T14:34:29.793771+0000","last_undegraded":"2024-08-19T14:34:29.793771+0000","last_fullsized":"2024-08-19T14:34:29.793771+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:33:18.213449+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:33:18.213449+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:34:42.168441+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,11],"acting":[8,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1d","version":"70'139","reported_seq":1090,"reported_epoch":623,"state":"active+undersized","last_fresh":"2024-08-19T14:34:29.776521+0000","last_change":"2024-08-19T14:31:02.878718+0000","last_active":"2024-08-19T14:34:29.776521+0000","last_peered":"2024-08-19T14:34:29.776521+0000","last_clean":"2024-08-19T14:30:57.053277+0000","last_became_active":"2024-08-19T14:31:02.878718+0000","last_became_peered":"2024-08-19T14:31:02.878718+0000","last_unstale":"2024-08-19T14:34:29.776521+0000","last_undegraded":"2024-08-19T14:34:29.776521+0000","last_fullsized":"2024-08-19T14:31:02.875751+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:29.194639+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:28:29.194639+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":["1","11"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":6051266560,"num_objects":92335,"num_object_clones":0,"num_object_copies":277005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34486,"num_objects_misplaced":20663,"num_objects_unfound":0,"num_objects_dirty":92335,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":92335,"num_write_kb":5909440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2861,"num_bytes_recovered":187498496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16189489152,"data_stored":16189489152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92335,"ondisk_log_size":92335,"up":42,"acting":39,"num_store_stats":9},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":856608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":84,"acting":88,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187734,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2858504,"kb_used_data":2109564,"kb_used_omap":1,"kb_used_meta":748926,"kb_avail":90886648,"statfs":{"total":95995035648,"available":93067927552,"internally_reserved":0,"allocated":2160193536,"data_stored":2157407020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":766900676},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,2,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":2314987372601,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":74,"seq":317827580192,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1212804,"kb_used_data":527740,"kb_used_omap":1,"kb_used_meta":685054,"kb_avail":92532348,"statfs":{"total":95995035648,"available":94753124352,"internally_reserved":0,"allocated":540405760,"data_stored":537619244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":701495745},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874534,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1822536,"kb_used_data":1083324,"kb_used_omap":1,"kb_used_meta":739198,"kb_avail":91922616,"statfs":{"total":95995035648,"available":94128758784,"internally_reserved":0,"allocated":1109323776,"data_stored":1106537260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":756939205},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":57,"seq":244813136169,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2000448,"kb_used_data":1102968,"kb_used_omap":1,"kb_used_meta":897470,"kb_avail":91744704,"statfs":{"total":95995035648,"available":93946576896,"internally_reserved":0,"allocated":1129439232,"data_stored":1126653092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":919009738},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":206158430509,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692144,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2134220,"kb_used_data":1228420,"kb_used_omap":1,"kb_used_meta":905790,"kb_avail":91610932,"statfs":{"total":95995035648,"available":93809594368,"internally_reserved":0,"allocated":1257902080,"data_stored":1255112012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":927529413},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986485,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2920068,"kb_used_data":1978172,"kb_used_omap":1,"kb_used_meta":941886,"kb_avail":90825084,"statfs":{"total":95995035648,"available":93004886016,"internally_reserved":0,"allocated":2025648128,"data_stored":2022861612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":964491717},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150011,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4181240,"kb_used_data":3239344,"kb_used_omap":1,"kb_used_meta":941886,"kb_avail":89563912,"statfs":{"total":95995035648,"available":91713445888,"internally_reserved":0,"allocated":3317088256,"data_stored":3314305668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":964491719},"hb_peers":[0,1,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,5,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411606,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2413884,"kb_used_data":2103476,"kb_used_omap":1,"kb_used_meta":310206,"kb_avail":91331268,"statfs":{"total":95995035648,"available":93523218432,"internally_reserved":0,"allocated":2153959424,"data_stored":2151417784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":317651397},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575172,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3864452,"kb_used_data":2846204,"kb_used_omap":1,"kb_used_meta":1018238,"kb_avail":89880700,"statfs":{"total":95995035648,"available":92037836800,"internally_reserved":0,"allocated":2914512896,"data_stored":2911726380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1042676165},"hb_peers":[0,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738696,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3063108,"kb_used_data":2247548,"kb_used_omap":1,"kb_used_meta":815550,"kb_avail":90682044,"statfs":{"total":95995035648,"available":92858413056,"internally_reserved":0,"allocated":2301489152,"data_stored":2298702636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":835123653},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2318336000,"data_stored":2318336000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2915303424,"data_stored":2915303424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3331719168,"data_stored":3331719168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1549008896,"data_stored":1549008896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1128660992,"data_stored":1128660992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1131151360,"data_stored":1131151360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1103167488,"data_stored":1103167488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":550567936,"data_stored":550567936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":2161573888,"data_stored":2161573888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T14:34:41.566 INFO:tasks.thrashosds.thrasher:Setting 7.8 to [0, 6, 8] 2024-08-19T14:34:41.566 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '7.8', '0', '6', '8'] 2024-08-19T14:34:41.566 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd pg-upmap 7.8 0 6 8 2024-08-19T14:34:41.605 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:34:41.627 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:41.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:41.628 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:34:41.729 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:34:41.750 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:41.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:41.751 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:34:41.852 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:34:41.899 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:41.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:41.901 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:34:42.002 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:34:42.023 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:42.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:42.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:34:42.101 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:34:42.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:41 smithi031 ceph-mon[16336]: pgmap v1468: 47 pgs: 4 active+clean+remapped, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized, 5 active+undersized+degraded, 29 active+clean; 5.9 GiB data, 26 GiB used, 868 GiB / 894 GiB avail; 49 MiB/s wr, 785 op/s; 36085/289308 objects degraded (12.473%); 19809/289308 objects misplaced (6.847%); 8.9 MiB/s, 141 objects/s recovering 2024-08-19T14:34:42.125 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:34:42.136 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:34:42.155 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:42.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:42.156 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:34:42.257 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:34:42.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:41 smithi032 ceph-mon[20004]: pgmap v1468: 47 pgs: 4 active+clean+remapped, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized, 5 active+undersized+degraded, 29 active+clean; 5.9 GiB data, 26 GiB used, 868 GiB / 894 GiB avail; 49 MiB/s wr, 785 op/s; 36085/289308 objects degraded (12.473%); 19809/289308 objects misplaced (6.847%); 8.9 MiB/s, 141 objects/s recovering 2024-08-19T14:34:42.293 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:42.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:42.294 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:34:42.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:41 smithi045 ceph-mon[20143]: pgmap v1468: 47 pgs: 4 active+clean+remapped, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized, 5 active+undersized+degraded, 29 active+clean; 5.9 GiB data, 26 GiB used, 868 GiB / 894 GiB avail; 49 MiB/s wr, 785 op/s; 36085/289308 objects degraded (12.473%); 19809/289308 objects misplaced (6.847%); 8.9 MiB/s, 141 objects/s recovering 2024-08-19T14:34:42.395 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:34:42.416 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:42.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:42.417 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:34:42.446 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 69 16 100519 100503 91.0272 53.125 0.0267646 0.010981 2024-08-19T14:34:42.517 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:34:42.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:42.547 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:42.548 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:34:42.648 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:34:42.683 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:42.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:42.684 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:34:42.785 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:34:42.807 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:42.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:42.808 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:34:42.909 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:34:42.958 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:42.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:42.960 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:34:43.061 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:34:43.096 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:43.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:43.097 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:34:43.198 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:34:43.232 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:43.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:43.233 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:34:43.294 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:34:43.295 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:34:43.333 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:34:43.333 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:34:43.333 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-08-19T14:34:43.334 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:34:43.356 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:43.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:43.357 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:34:43.370 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:34:43.370 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:34:43.407 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:34:43.408 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:34:43.439 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:34:43.439 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:34:43.446 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 70 16 101381 101365 90.4964 53.875 0.00421081 0.0110449 2024-08-19T14:34:43.458 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:34:43.474 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:34:43.475 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:34:43.485 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:43.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:43.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:34:43.500 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:34:43.500 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:34:43.534 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:34:43.534 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:34:43.575 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:34:43.575 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:34:43.587 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:34:43.606 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:34:43.607 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:34:43.611 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:43.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:43.612 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:34:43.641 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:34:43.642 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:34:43.712 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:34:43.740 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:43.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:43.741 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:34:43.825 INFO:teuthology.orchestra.run.smithi031.stderr:set 7.8 pg_upmap mapping to [0,6,8] 2024-08-19T14:34:43.842 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:34:43.864 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:43.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:43.865 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:34:43.966 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:34:43.997 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:43.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:43.998 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:34:44.049 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-19T14:34:44.083 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:43 smithi031 ceph-mon[16336]: pgmap v1469: 47 pgs: 4 active+clean+remapped, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized, 5 active+undersized+degraded, 29 active+clean; 6.0 GiB data, 26 GiB used, 868 GiB / 894 GiB avail; 55 MiB/s wr, 872 op/s; 37386/295494 objects degraded (12.652%); 19201/295494 objects misplaced (6.498%); 13 MiB/s, 200 objects/s recovering 2024-08-19T14:34:44.083 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:43 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 37386/295494 objects degraded (12.652%), 7 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-08-19T14:34:44.084 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:43 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3361864811' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "7.8", "id": [0, 6, 8]}]: dispatch 2024-08-19T14:34:44.084 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:43 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3114367241' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:34:44.099 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:34:44.139 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:44.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:44.142 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:34:44.243 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:34:44.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:43 smithi032 ceph-mon[20004]: pgmap v1469: 47 pgs: 4 active+clean+remapped, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized, 5 active+undersized+degraded, 29 active+clean; 6.0 GiB data, 26 GiB used, 868 GiB / 894 GiB avail; 55 MiB/s wr, 872 op/s; 37386/295494 objects degraded (12.652%); 19201/295494 objects misplaced (6.498%); 13 MiB/s, 200 objects/s recovering 2024-08-19T14:34:44.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:43 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 37386/295494 objects degraded (12.652%), 7 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-08-19T14:34:44.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:43 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3361864811' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "7.8", "id": [0, 6, 8]}]: dispatch 2024-08-19T14:34:44.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:43 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3114367241' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:34:44.276 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:44.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:44.277 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:34:44.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:43 smithi045 ceph-mon[20143]: pgmap v1469: 47 pgs: 4 active+clean+remapped, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized, 5 active+undersized+degraded, 29 active+clean; 6.0 GiB data, 26 GiB used, 868 GiB / 894 GiB avail; 55 MiB/s wr, 872 op/s; 37386/295494 objects degraded (12.652%); 19201/295494 objects misplaced (6.498%); 13 MiB/s, 200 objects/s recovering 2024-08-19T14:34:44.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:43 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 37386/295494 objects degraded (12.652%), 7 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-08-19T14:34:44.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:43 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3361864811' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "7.8", "id": [0, 6, 8]}]: dispatch 2024-08-19T14:34:44.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:43 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3114367241' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:34:44.378 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:34:44.407 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:44.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:44.408 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:34:44.446 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 71 16 102170 102154 89.9163 49.3125 0.0136164 0.0111163 2024-08-19T14:34:44.509 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:34:44.545 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:44.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:44.546 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:34:44.647 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:34:44.694 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:44.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:44.695 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:34:44.796 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:34:44.830 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:44.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:44.831 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:34:44.932 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:34:44.966 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:44.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:44.967 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:34:45.068 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:34:45.105 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:45.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:45.107 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:34:45.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:44 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3361864811' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "7.8", "id": [0, 6, 8]}]': finished 2024-08-19T14:34:45.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:44 smithi031 ceph-mon[16336]: osdmap e626: 12 total, 11 up, 10 in 2024-08-19T14:34:45.208 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:34:45.214 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-19T14:34:45.215 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-19T14:34:45.229 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:45.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:45.230 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:34:45.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:44 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3361864811' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "7.8", "id": [0, 6, 8]}]': finished 2024-08-19T14:34:45.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:44 smithi032 ceph-mon[20004]: osdmap e626: 12 total, 11 up, 10 in 2024-08-19T14:34:45.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:44 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3361864811' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "7.8", "id": [0, 6, 8]}]': finished 2024-08-19T14:34:45.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:44 smithi045 ceph-mon[20143]: osdmap e626: 12 total, 11 up, 10 in 2024-08-19T14:34:45.331 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:34:45.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:45.370 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:45.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:34:45.446 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 72 16 103119 103103 89.4913 59.3125 0.00864311 0.0111701 2024-08-19T14:34:45.471 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:34:45.503 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:45.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:45.504 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:34:45.605 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:34:45.634 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:45.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:45.634 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:34:45.735 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:34:45.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:45.770 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:45.770 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:34:45.870 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:34:45.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:45.911 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:45.911 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:34:45.931 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:34:46.012 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:34:46.060 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:46.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:46.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:34:46.064 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:45 smithi031 ceph-mon[16336]: pgmap v1471: 47 pgs: 4 active+clean+remapped, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized, 5 active+undersized+degraded, 29 active+clean; 6.1 GiB data, 27 GiB used, 867 GiB / 894 GiB avail; 45 MiB/s wr, 726 op/s; 37386/299292 objects degraded (12.491%); 19059/299292 objects misplaced (6.368%); 11 MiB/s, 182 objects/s recovering 2024-08-19T14:34:46.162 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:34:46.184 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:46.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:46.185 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:34:46.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:45 smithi032 ceph-mon[20004]: pgmap v1471: 47 pgs: 4 active+clean+remapped, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized, 5 active+undersized+degraded, 29 active+clean; 6.1 GiB data, 27 GiB used, 867 GiB / 894 GiB avail; 45 MiB/s wr, 726 op/s; 37386/299292 objects degraded (12.491%); 19059/299292 objects misplaced (6.368%); 11 MiB/s, 182 objects/s recovering 2024-08-19T14:34:46.286 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:34:46.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:45 smithi045 ceph-mon[20143]: pgmap v1471: 47 pgs: 4 active+clean+remapped, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized, 5 active+undersized+degraded, 29 active+clean; 6.1 GiB data, 27 GiB used, 867 GiB / 894 GiB avail; 45 MiB/s wr, 726 op/s; 37386/299292 objects degraded (12.491%); 19059/299292 objects misplaced (6.368%); 11 MiB/s, 182 objects/s recovering 2024-08-19T14:34:46.309 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:46.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:46.310 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:34:46.412 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:34:46.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:46.437 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:46.437 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:34:46.446 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 73 16 104010 103994 89.0281 55.6875 0.0118886 0.0112235 2024-08-19T14:34:46.538 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:34:46.561 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:46.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:46.562 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:34:46.663 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:34:46.706 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:46.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:46.707 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:34:46.808 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:34:46.830 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:46.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:46.832 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:34:46.932 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:34:46.956 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:46.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:46.957 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:34:47.058 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:34:47.080 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:47.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:47.081 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:34:47.182 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:34:47.215 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:47.215 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:34:47.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:47.216 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:34:47.317 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:34:47.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:47.369 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:47.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:34:47.447 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 74 16 104944 104928 88.6138 58.375 0.00482266 0.0112803 2024-08-19T14:34:47.470 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:34:47.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:47.519 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:47.519 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:34:47.620 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:34:47.642 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:47.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:47.643 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:34:47.659 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-08-19T14:34:47.707 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:34:47.744 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:34:47.765 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:47.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:47.766 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:34:47.854 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is unset 2024-08-19T14:34:47.866 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:34:47.887 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:47.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:47.888 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:34:47.989 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:34:48.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:48.022 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:48.022 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:34:48.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:47 smithi045 ceph-mon[20143]: pgmap v1472: 47 pgs: 4 active+clean+remapped, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized, 5 active+undersized+degraded, 29 active+clean; 6.2 GiB data, 27 GiB used, 867 GiB / 894 GiB avail; 59 MiB/s wr, 946 op/s; 38676/306237 objects degraded (12.629%); 18784/306237 objects misplaced (6.134%); 10 MiB/s, 161 objects/s recovering 2024-08-19T14:34:48.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:47 smithi045 ceph-mon[20143]: from='client.17064 172.21.15.31:0/4084506839' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:34:48.059 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:34:48.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:47 smithi031 ceph-mon[16336]: pgmap v1472: 47 pgs: 4 active+clean+remapped, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized, 5 active+undersized+degraded, 29 active+clean; 6.2 GiB data, 27 GiB used, 867 GiB / 894 GiB avail; 59 MiB/s wr, 946 op/s; 38676/306237 objects degraded (12.629%); 18784/306237 objects misplaced (6.134%); 10 MiB/s, 161 objects/s recovering 2024-08-19T14:34:48.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:47 smithi031 ceph-mon[16336]: from='client.17064 172.21.15.31:0/4084506839' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:34:48.123 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:34:48.154 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:48.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:48.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:34:48.256 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:34:48.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:47 smithi032 ceph-mon[20004]: pgmap v1472: 47 pgs: 4 active+clean+remapped, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized, 5 active+undersized+degraded, 29 active+clean; 6.2 GiB data, 27 GiB used, 867 GiB / 894 GiB avail; 59 MiB/s wr, 946 op/s; 38676/306237 objects degraded (12.629%); 18784/306237 objects misplaced (6.134%); 10 MiB/s, 161 objects/s recovering 2024-08-19T14:34:48.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:47 smithi032 ceph-mon[20004]: from='client.17064 172.21.15.31:0/4084506839' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:34:48.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:48.290 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:48.290 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:34:48.391 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:34:48.431 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:48.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:48.432 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:34:48.446 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 75 16 105805 105789 88.1498 53.8125 0.00576969 0.0113384 2024-08-19T14:34:48.533 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:34:48.557 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:48.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:48.558 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:34:48.659 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:34:48.692 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:48.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:48.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:34:48.793 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:34:48.822 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:48.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:48.823 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:34:48.924 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:34:48.947 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:48.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:48.948 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:34:49.049 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:34:49.084 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:49.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:49.085 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:34:49.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:48 smithi031 ceph-mon[16336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:34:49.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:48 smithi031 ceph-mon[16336]: from='client.17064 172.21.15.31:0/4084506839' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:34:49.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:48 smithi031 ceph-mon[16336]: osdmap e627: 12 total, 11 up, 10 in 2024-08-19T14:34:49.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:48 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 38676/306237 objects degraded (12.629%), 7 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-08-19T14:34:49.185 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:34:49.224 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:48 smithi045 ceph-mon[20143]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:34:49.224 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:48 smithi045 ceph-mon[20143]: from='client.17064 172.21.15.31:0/4084506839' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:34:49.224 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:48 smithi045 ceph-mon[20143]: osdmap e627: 12 total, 11 up, 10 in 2024-08-19T14:34:49.224 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:48 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 38676/306237 objects degraded (12.629%), 7 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-08-19T14:34:49.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:49.225 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:49.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:34:49.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:48 smithi032 ceph-mon[20004]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:34:49.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:48 smithi032 ceph-mon[20004]: from='client.17064 172.21.15.31:0/4084506839' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:34:49.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:48 smithi032 ceph-mon[20004]: osdmap e627: 12 total, 11 up, 10 in 2024-08-19T14:34:49.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:48 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 38676/306237 objects degraded (12.629%), 7 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-08-19T14:34:49.326 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:34:49.372 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:49.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:49.373 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:34:49.447 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 76 16 106723 106707 87.7448 57.375 0.0420874 0.0113913 2024-08-19T14:34:49.473 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:34:49.506 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:49.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:49.507 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:34:49.608 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:34:49.634 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:49.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:49.635 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:34:49.736 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:34:49.763 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:49.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:49.764 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:34:49.865 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:34:49.904 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:49.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:49.905 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:34:50.006 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:34:50.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:50.029 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:50.029 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:34:50.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:49 smithi031 ceph-mon[16336]: pgmap v1474: 47 pgs: 4 active+clean+remapped, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized, 5 active+undersized+degraded, 29 active+clean; 6.3 GiB data, 27 GiB used, 867 GiB / 894 GiB avail; 48 MiB/s wr, 773 op/s; 38958/307890 objects degraded (12.653%); 18341/307890 objects misplaced (5.957%); 11 MiB/s, 183 objects/s recovering 2024-08-19T14:34:50.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:49 smithi031 ceph-mon[16336]: 1.d scrub starts 2024-08-19T14:34:50.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:49 smithi031 ceph-mon[16336]: 1.d scrub ok 2024-08-19T14:34:50.130 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:34:50.132 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:49 smithi045 ceph-mon[20143]: pgmap v1474: 47 pgs: 4 active+clean+remapped, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized, 5 active+undersized+degraded, 29 active+clean; 6.3 GiB data, 27 GiB used, 867 GiB / 894 GiB avail; 48 MiB/s wr, 773 op/s; 38958/307890 objects degraded (12.653%); 18341/307890 objects misplaced (5.957%); 11 MiB/s, 183 objects/s recovering 2024-08-19T14:34:50.132 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:49 smithi045 ceph-mon[20143]: 1.d scrub starts 2024-08-19T14:34:50.132 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:49 smithi045 ceph-mon[20143]: 1.d scrub ok 2024-08-19T14:34:50.152 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:50.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:50.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:34:50.254 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:34:50.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:49 smithi032 ceph-mon[20004]: pgmap v1474: 47 pgs: 4 active+clean+remapped, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized, 5 active+undersized+degraded, 29 active+clean; 6.3 GiB data, 27 GiB used, 867 GiB / 894 GiB avail; 48 MiB/s wr, 773 op/s; 38958/307890 objects degraded (12.653%); 18341/307890 objects misplaced (5.957%); 11 MiB/s, 183 objects/s recovering 2024-08-19T14:34:50.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:49 smithi032 ceph-mon[20004]: 1.d scrub starts 2024-08-19T14:34:50.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:49 smithi032 ceph-mon[20004]: 1.d scrub ok 2024-08-19T14:34:50.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:50.289 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:50.289 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:34:50.390 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:34:50.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:50.417 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:50.418 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:34:50.447 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 77 15 107487 107472 87.2261 47.8125 0.0506121 0.0114592 2024-08-19T14:34:50.518 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:34:50.553 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:50.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:50.554 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:34:50.655 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:34:50.677 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:50.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:50.678 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:34:50.779 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:34:50.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:50.815 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:50.815 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:34:50.852 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 7, 8, 2, 11, 9] out_osds: [10, 6] dead_osds: [2] live_osds: [0, 1, 3, 4, 5, 6, 7, 8, 9, 11, 10] 2024-08-19T14:34:50.852 INFO:tasks.thrashosds.thrasher:Reviving osd 2 2024-08-19T14:34:50.853 INFO:tasks.cephadm.osd.2:Restarting osd.2 (starting--it wasn't running)... 2024-08-19T14:34:50.853 DEBUG:teuthology.orchestra.run.smithi031:osd.2> sudo journalctl -f -n 0 -u ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968@osd.2.service 2024-08-19T14:34:50.859 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl start ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968@osd.2 2024-08-19T14:34:50.916 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:34:50.937 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:50.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:50.938 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:34:50.945 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-19T14:34:51.039 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:34:51.069 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:51.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:51.070 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:34:51.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:50 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2056348297' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:34:51.117 INFO:journalctl@ceph.osd.2.smithi031.stdout:Aug 19 14:34:50 smithi031 systemd[1]: Starting Ceph osd.2 for ed7d1c6e-5e33-11ef-bccf-c7b262605968... 2024-08-19T14:34:51.137 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:34:51.137 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:34:51.137 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:34:51.138 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:34:51.138 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:34:51.138 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:34:51.138 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-08-19T14:34:51.138 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:34:51.138 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:34:51.138 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:34:51.139 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:34:51.139 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:34:51.139 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:34:51.139 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:34:51.139 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:34:51.139 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:34:51.139 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:34:51.139 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:34:51.139 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:34:51.139 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:34:51.139 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:34:51.139 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:34:51.140 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:34:51.171 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:34:51.197 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:51.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:51.199 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:34:51.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:50 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2056348297' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:34:51.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:50 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2056348297' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:34:51.299 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:34:51.323 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:51.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:51.324 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:34:51.425 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:34:51.447 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 78 16 108364 108348 86.8091 54.75 0.0137638 0.0115142 2024-08-19T14:34:51.458 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:51.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:51.459 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:34:51.560 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:34:51.587 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:51.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:51.588 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:34:51.689 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:34:51.726 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:51.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:51.727 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:34:51.828 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:34:51.850 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:51.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:51.852 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:34:51.874 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:34:51.874 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:34:51.952 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:34:51.976 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:51.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:51.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:34:52.078 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:34:52.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:52.126 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:52.126 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:34:52.158 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:51 smithi032 ceph-mon[20004]: pgmap v1475: 47 pgs: 4 active+clean+remapped, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized, 5 active+undersized+degraded, 29 active+clean; 6.4 GiB data, 27 GiB used, 867 GiB / 894 GiB avail; 50 MiB/s wr, 801 op/s; 39223/314748 objects degraded (12.462%); 18076/314748 objects misplaced (5.743%); 8.8 MiB/s, 140 objects/s recovering 2024-08-19T14:34:52.158 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:51 smithi032 ceph-mon[20004]: 1.a scrub starts 2024-08-19T14:34:52.158 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:51 smithi032 ceph-mon[20004]: 1.a scrub ok 2024-08-19T14:34:52.227 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:34:52.235 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:51 smithi045 ceph-mon[20143]: pgmap v1475: 47 pgs: 4 active+clean+remapped, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized, 5 active+undersized+degraded, 29 active+clean; 6.4 GiB data, 27 GiB used, 867 GiB / 894 GiB avail; 50 MiB/s wr, 801 op/s; 39223/314748 objects degraded (12.462%); 18076/314748 objects misplaced (5.743%); 8.8 MiB/s, 140 objects/s recovering 2024-08-19T14:34:52.235 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:51 smithi045 ceph-mon[20143]: 1.a scrub starts 2024-08-19T14:34:52.235 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:51 smithi045 ceph-mon[20143]: 1.a scrub ok 2024-08-19T14:34:52.259 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:52.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:52.260 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:34:52.361 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:34:52.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:51 smithi031 ceph-mon[16336]: pgmap v1475: 47 pgs: 4 active+clean+remapped, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized, 5 active+undersized+degraded, 29 active+clean; 6.4 GiB data, 27 GiB used, 867 GiB / 894 GiB avail; 50 MiB/s wr, 801 op/s; 39223/314748 objects degraded (12.462%); 18076/314748 objects misplaced (5.743%); 8.8 MiB/s, 140 objects/s recovering 2024-08-19T14:34:52.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:51 smithi031 ceph-mon[16336]: 1.a scrub starts 2024-08-19T14:34:52.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:51 smithi031 ceph-mon[16336]: 1.a scrub ok 2024-08-19T14:34:52.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:52.383 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:52.384 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:34:52.447 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 79 16 109177 109161 86.3534 50.8125 0.119476 0.0115731 2024-08-19T14:34:52.484 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:34:52.518 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:52.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:52.520 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:34:52.621 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:34:52.644 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-08-19T14:34:52.645 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:52.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:52.646 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:34:52.747 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:34:52.772 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:52.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:52.773 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:34:52.874 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:34:52.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:52.899 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:52.900 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:34:53.001 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:34:53.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:52 smithi032 ceph-mon[20004]: 7.1 scrub starts 2024-08-19T14:34:53.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:52 smithi032 ceph-mon[20004]: 1.9 scrub starts 2024-08-19T14:34:53.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:52 smithi032 ceph-mon[20004]: 1.9 scrub ok 2024-08-19T14:34:53.026 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:53.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:53.027 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:34:53.038 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:34:53.128 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:34:53.164 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:53.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:53.165 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:34:53.173 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:52 smithi045 ceph-mon[20143]: 7.1 scrub starts 2024-08-19T14:34:53.174 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:52 smithi045 ceph-mon[20143]: 1.9 scrub starts 2024-08-19T14:34:53.174 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:52 smithi045 ceph-mon[20143]: 1.9 scrub ok 2024-08-19T14:34:53.266 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:34:53.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:53.303 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:53.303 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:34:53.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:52 smithi031 ceph-mon[16336]: 7.1 scrub starts 2024-08-19T14:34:53.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:52 smithi031 ceph-mon[16336]: 1.9 scrub starts 2024-08-19T14:34:53.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:52 smithi031 ceph-mon[16336]: 1.9 scrub ok 2024-08-19T14:34:53.404 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:34:53.439 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:53.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:53.440 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:34:53.447 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:2024-08-19T14:34:53.448074+0000 min lat: 0.00106356 max lat: 2.26272 avg lat: 0.0116412 2024-08-19T14:34:53.447 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-19T14:34:53.448 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 80 16 109926 109910 85.8591 46.8125 0.0327549 0.0116412 2024-08-19T14:34:53.541 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:34:53.579 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:53.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:53.580 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:34:53.681 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:34:53.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:53.715 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:53.716 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:34:53.816 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:34:53.852 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:53.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:53.854 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:34:53.955 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:34:53.977 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:53.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:53.978 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:34:54.029 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:34:54.029 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:34:54.079 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:34:54.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:53 smithi031 ceph-mon[16336]: pgmap v1476: 47 pgs: 4 active+clean+remapped, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized, 5 active+undersized+degraded, 29 active+clean; 6.5 GiB data, 28 GiB used, 866 GiB / 894 GiB avail; 63 MiB/s wr, 1.00k op/s; 40608/321363 objects degraded (12.636%); 17469/321363 objects misplaced (5.436%); 13 MiB/s, 207 objects/s recovering 2024-08-19T14:34:54.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:53 smithi031 ceph-mon[16336]: 7.1 scrub ok 2024-08-19T14:34:54.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:53 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 40608/321363 objects degraded (12.636%), 7 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-08-19T14:34:54.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:54.118 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:54.118 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:34:54.146 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:53 smithi045 ceph-mon[20143]: pgmap v1476: 47 pgs: 4 active+clean+remapped, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized, 5 active+undersized+degraded, 29 active+clean; 6.5 GiB data, 28 GiB used, 866 GiB / 894 GiB avail; 63 MiB/s wr, 1.00k op/s; 40608/321363 objects degraded (12.636%); 17469/321363 objects misplaced (5.436%); 13 MiB/s, 207 objects/s recovering 2024-08-19T14:34:54.146 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:53 smithi045 ceph-mon[20143]: 7.1 scrub ok 2024-08-19T14:34:54.146 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:53 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 40608/321363 objects degraded (12.636%), 7 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-08-19T14:34:54.219 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:34:54.253 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:54.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:54.254 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:34:54.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:53 smithi032 ceph-mon[20004]: pgmap v1476: 47 pgs: 4 active+clean+remapped, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized, 5 active+undersized+degraded, 29 active+clean; 6.5 GiB data, 28 GiB used, 866 GiB / 894 GiB avail; 63 MiB/s wr, 1.00k op/s; 40608/321363 objects degraded (12.636%); 17469/321363 objects misplaced (5.436%); 13 MiB/s, 207 objects/s recovering 2024-08-19T14:34:54.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:53 smithi032 ceph-mon[20004]: 7.1 scrub ok 2024-08-19T14:34:54.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:53 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 40608/321363 objects degraded (12.636%), 7 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-08-19T14:34:54.355 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:34:54.390 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:54.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:54.391 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:34:54.448 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 81 16 110756 110740 85.4395 51.875 0.0406371 0.0116992 2024-08-19T14:34:54.492 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:34:54.538 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:54.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:54.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:34:54.640 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:34:54.671 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:54.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:54.672 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:34:54.773 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:34:54.799 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:54.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:54.800 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:34:54.901 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:34:54.925 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:54.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:54.926 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:34:55.027 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:34:55.049 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:55.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:55.050 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:34:55.151 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:34:55.193 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:55.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:55.194 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:34:55.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:54 smithi032 ceph-mon[20004]: 1.14 scrub starts 2024-08-19T14:34:55.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:54 smithi032 ceph-mon[20004]: 1.14 scrub ok 2024-08-19T14:34:55.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:54 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:34:55.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:54 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:34:55.295 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:34:55.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:54 smithi045 ceph-mon[20143]: 1.14 scrub starts 2024-08-19T14:34:55.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:54 smithi045 ceph-mon[20143]: 1.14 scrub ok 2024-08-19T14:34:55.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:54 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:34:55.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:54 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:34:55.319 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:55.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:55.320 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:34:55.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:54 smithi031 ceph-mon[16336]: 1.14 scrub starts 2024-08-19T14:34:55.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:54 smithi031 ceph-mon[16336]: 1.14 scrub ok 2024-08-19T14:34:55.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:54 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:34:55.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:54 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:34:55.421 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:34:55.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:55.446 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:55.446 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:34:55.447 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 82 16 111544 111528 84.9981 49.25 0.0140835 0.0117598 2024-08-19T14:34:55.546 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:34:55.581 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:55.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:55.582 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:34:55.618 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-19T14:34:55.619 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-19T14:34:55.683 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:34:55.688 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:34:55.705 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:55.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:55.706 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:34:55.807 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:34:55.840 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:55.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:55.841 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:34:55.943 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:34:55.966 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:55.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:55.967 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:34:56.068 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:34:56.099 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:56.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:56.100 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:34:56.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:55 smithi031 ceph-mon[16336]: pgmap v1477: 47 pgs: 4 active+clean+remapped, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized, 5 active+undersized+degraded, 29 active+clean; 6.6 GiB data, 28 GiB used, 866 GiB / 894 GiB avail; 47 MiB/s wr, 759 op/s; 40608/322080 objects degraded (12.608%); 17469/322080 objects misplaced (5.424%); 9.9 MiB/s, 158 objects/s recovering 2024-08-19T14:34:56.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:55 smithi031 ceph-mon[16336]: osdmap e628: 12 total, 11 up, 10 in 2024-08-19T14:34:56.117 INFO:journalctl@ceph.osd.2.smithi031.stdout:Aug 19 14:34:55 smithi031 podman[180865]: 2024-08-19 14:34:55.750253104 +0000 UTC m=+0.038272662 image pull ea7a5189f30bad3aca9524788e2965d2368d7ce786df09d4edcd8c3edf9cefba quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9 2024-08-19T14:34:56.182 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:55 smithi045 ceph-mon[20143]: pgmap v1477: 47 pgs: 4 active+clean+remapped, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized, 5 active+undersized+degraded, 29 active+clean; 6.6 GiB data, 28 GiB used, 866 GiB / 894 GiB avail; 47 MiB/s wr, 759 op/s; 40608/322080 objects degraded (12.608%); 17469/322080 objects misplaced (5.424%); 9.9 MiB/s, 158 objects/s recovering 2024-08-19T14:34:56.182 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:55 smithi045 ceph-mon[20143]: osdmap e628: 12 total, 11 up, 10 in 2024-08-19T14:34:56.201 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:34:56.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:56.226 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:56.226 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:34:56.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:55 smithi032 ceph-mon[20004]: pgmap v1477: 47 pgs: 4 active+clean+remapped, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized, 5 active+undersized+degraded, 29 active+clean; 6.6 GiB data, 28 GiB used, 866 GiB / 894 GiB avail; 47 MiB/s wr, 759 op/s; 40608/322080 objects degraded (12.608%); 17469/322080 objects misplaced (5.424%); 9.9 MiB/s, 158 objects/s recovering 2024-08-19T14:34:56.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:55 smithi032 ceph-mon[20004]: osdmap e628: 12 total, 11 up, 10 in 2024-08-19T14:34:56.327 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:34:56.357 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:56.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:56.358 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:34:56.428 INFO:journalctl@ceph.osd.2.smithi031.stdout:Aug 19 14:34:56 smithi031 podman[180865]: 2024-08-19 14:34:56.146411598 +0000 UTC m=+0.434431128 container create c981e9ebb912e1c238fb076e9b7ceb32afa48eabb734fa28ea6cfe6517dd5b94 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-2-activate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, ceph=True, RELEASE=squid-de09dd0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-08-19T14:34:56.448 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 83 16 112329 112313 84.5651 49.0625 0.0286636 0.01182 2024-08-19T14:34:56.459 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:34:56.492 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:56.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:56.493 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:34:56.594 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:34:56.632 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:56.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:56.634 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:34:56.679 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-08-19T14:34:56.734 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:34:56.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:56.757 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:56.757 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:34:56.777 INFO:journalctl@ceph.osd.2.smithi031.stdout:Aug 19 14:34:56 smithi031 podman[180865]: 2024-08-19 14:34:56.670089369 +0000 UTC m=+0.958108897 container init c981e9ebb912e1c238fb076e9b7ceb32afa48eabb734fa28ea6cfe6517dd5b94 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-2-activate, CEPH_POINT_RELEASE=, RELEASE=squid-de09dd0, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-08-19T14:34:56.778 INFO:journalctl@ceph.osd.2.smithi031.stdout:Aug 19 14:34:56 smithi031 podman[180865]: 2024-08-19 14:34:56.675893843 +0000 UTC m=+0.963913372 container start c981e9ebb912e1c238fb076e9b7ceb32afa48eabb734fa28ea6cfe6517dd5b94 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-2-activate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, io.buildah.version=1.36.0, RELEASE=squid-de09dd0, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-08-19T14:34:56.859 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:34:56.901 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is unset 2024-08-19T14:34:56.910 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:56.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:56.912 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:34:57.013 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:34:57.035 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:57.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:57.036 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:34:57.107 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:34:57.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:56 smithi031 ceph-mon[16336]: 1.e scrub starts 2024-08-19T14:34:57.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:56 smithi031 ceph-mon[16336]: 1.e scrub ok 2024-08-19T14:34:57.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:56 smithi031 ceph-mon[16336]: 1.2 scrub starts 2024-08-19T14:34:57.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:56 smithi031 ceph-mon[16336]: 1.2 scrub ok 2024-08-19T14:34:57.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:56 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3581602723' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:34:57.117 INFO:journalctl@ceph.osd.2.smithi031.stdout:Aug 19 14:34:56 smithi031 podman[180865]: 2024-08-19 14:34:56.776674874 +0000 UTC m=+1.064694410 container attach c981e9ebb912e1c238fb076e9b7ceb32afa48eabb734fa28ea6cfe6517dd5b94 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-2-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=squid-de09dd0, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0) 2024-08-19T14:34:57.117 INFO:journalctl@ceph.osd.2.smithi031.stdout:Aug 19 14:34:56 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-2-activate[180908]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-19T14:34:57.117 INFO:journalctl@ceph.osd.2.smithi031.stdout:Aug 19 14:34:56 smithi031 bash[180865]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-19T14:34:57.117 INFO:journalctl@ceph.osd.2.smithi031.stdout:Aug 19 14:34:56 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-2-activate[180908]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-19T14:34:57.117 INFO:journalctl@ceph.osd.2.smithi031.stdout:Aug 19 14:34:56 smithi031 bash[180865]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-19T14:34:57.117 INFO:journalctl@ceph.osd.2.smithi031.stdout:Aug 19 14:34:56 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-2-activate[180908]: --> Failed to activate via raw: activate() takes 1 positional argument but 5 were given 2024-08-19T14:34:57.117 INFO:journalctl@ceph.osd.2.smithi031.stdout:Aug 19 14:34:56 smithi031 bash[180865]: --> Failed to activate via raw: activate() takes 1 positional argument but 5 were given 2024-08-19T14:34:57.117 INFO:journalctl@ceph.osd.2.smithi031.stdout:Aug 19 14:34:56 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-2-activate[180908]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-19T14:34:57.117 INFO:journalctl@ceph.osd.2.smithi031.stdout:Aug 19 14:34:56 smithi031 bash[180865]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-19T14:34:57.117 INFO:journalctl@ceph.osd.2.smithi031.stdout:Aug 19 14:34:56 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-2-activate[180908]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-19T14:34:57.117 INFO:journalctl@ceph.osd.2.smithi031.stdout:Aug 19 14:34:56 smithi031 bash[180865]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-19T14:34:57.137 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:34:57.178 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:57.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:57.179 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:34:57.246 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:56 smithi045 ceph-mon[20143]: 1.e scrub starts 2024-08-19T14:34:57.246 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:56 smithi045 ceph-mon[20143]: 1.e scrub ok 2024-08-19T14:34:57.246 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:56 smithi045 ceph-mon[20143]: 1.2 scrub starts 2024-08-19T14:34:57.246 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:56 smithi045 ceph-mon[20143]: 1.2 scrub ok 2024-08-19T14:34:57.247 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:56 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3581602723' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:34:57.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:56 smithi032 ceph-mon[20004]: 1.e scrub starts 2024-08-19T14:34:57.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:56 smithi032 ceph-mon[20004]: 1.e scrub ok 2024-08-19T14:34:57.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:56 smithi032 ceph-mon[20004]: 1.2 scrub starts 2024-08-19T14:34:57.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:56 smithi032 ceph-mon[20004]: 1.2 scrub ok 2024-08-19T14:34:57.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:56 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3581602723' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:34:57.280 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:34:57.315 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:57.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:57.316 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:34:57.417 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:34:57.439 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:57.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:57.440 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:34:57.448 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 84 16 113127 113111 84.1521 49.875 0.016002 0.0118782 2024-08-19T14:34:57.540 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:34:57.571 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:57.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:57.572 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:34:57.617 INFO:journalctl@ceph.osd.2.smithi031.stdout:Aug 19 14:34:57 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-2-activate[180908]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-08-19T14:34:57.617 INFO:journalctl@ceph.osd.2.smithi031.stdout:Aug 19 14:34:57 smithi031 bash[180865]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-08-19T14:34:57.617 INFO:journalctl@ceph.osd.2.smithi031.stdout:Aug 19 14:34:57 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-2-activate[180908]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_2 --path /var/lib/ceph/osd/ceph-2 --no-mon-config 2024-08-19T14:34:57.617 INFO:journalctl@ceph.osd.2.smithi031.stdout:Aug 19 14:34:57 smithi031 bash[180865]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_2 --path /var/lib/ceph/osd/ceph-2 --no-mon-config 2024-08-19T14:34:57.617 INFO:journalctl@ceph.osd.2.smithi031.stdout:Aug 19 14:34:57 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-2-activate[180908]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-2/block 2024-08-19T14:34:57.617 INFO:journalctl@ceph.osd.2.smithi031.stdout:Aug 19 14:34:57 smithi031 bash[180865]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-2/block 2024-08-19T14:34:57.618 INFO:journalctl@ceph.osd.2.smithi031.stdout:Aug 19 14:34:57 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-2-activate[180908]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-2/block 2024-08-19T14:34:57.618 INFO:journalctl@ceph.osd.2.smithi031.stdout:Aug 19 14:34:57 smithi031 bash[180865]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-2/block 2024-08-19T14:34:57.618 INFO:journalctl@ceph.osd.2.smithi031.stdout:Aug 19 14:34:57 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-2-activate[180908]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-08-19T14:34:57.618 INFO:journalctl@ceph.osd.2.smithi031.stdout:Aug 19 14:34:57 smithi031 bash[180865]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-08-19T14:34:57.618 INFO:journalctl@ceph.osd.2.smithi031.stdout:Aug 19 14:34:57 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-2-activate[180908]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-08-19T14:34:57.618 INFO:journalctl@ceph.osd.2.smithi031.stdout:Aug 19 14:34:57 smithi031 bash[180865]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-08-19T14:34:57.618 INFO:journalctl@ceph.osd.2.smithi031.stdout:Aug 19 14:34:57 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-2-activate[180908]: --> ceph-volume lvm activate successful for osd ID: 2 2024-08-19T14:34:57.618 INFO:journalctl@ceph.osd.2.smithi031.stdout:Aug 19 14:34:57 smithi031 bash[180865]: --> ceph-volume lvm activate successful for osd ID: 2 2024-08-19T14:34:57.620 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:34:57.673 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:34:57.697 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:57.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:57.698 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:34:57.799 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:34:57.833 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:57.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:57.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:34:57.935 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:34:57.957 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:57.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:57.958 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:34:57.979 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:57 smithi045 ceph-mon[20143]: pgmap v1479: 47 pgs: 4 active+clean+remapped, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized, 5 active+undersized+degraded, 29 active+clean; 6.8 GiB data, 29 GiB used, 866 GiB / 894 GiB avail; 60 MiB/s wr, 962 op/s; 41847/331866 objects degraded (12.610%); 17058/331866 objects misplaced (5.140%); 9.7 MiB/s, 154 objects/s recovering 2024-08-19T14:34:57.979 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:57 smithi045 ceph-mon[20143]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:34:57.979 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:57 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3581602723' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:34:57.979 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:57 smithi045 ceph-mon[20143]: osdmap e629: 12 total, 11 up, 10 in 2024-08-19T14:34:58.059 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:34:58.080 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:58.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:58.081 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:34:58.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:57 smithi031 ceph-mon[16336]: pgmap v1479: 47 pgs: 4 active+clean+remapped, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized, 5 active+undersized+degraded, 29 active+clean; 6.8 GiB data, 29 GiB used, 866 GiB / 894 GiB avail; 60 MiB/s wr, 962 op/s; 41847/331866 objects degraded (12.610%); 17058/331866 objects misplaced (5.140%); 9.7 MiB/s, 154 objects/s recovering 2024-08-19T14:34:58.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:57 smithi031 ceph-mon[16336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:34:58.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:57 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3581602723' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:34:58.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:57 smithi031 ceph-mon[16336]: osdmap e629: 12 total, 11 up, 10 in 2024-08-19T14:34:58.136 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:34:58.136 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:34:58.181 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:34:58.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:58.219 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:58.220 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:34:58.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:57 smithi032 ceph-mon[20004]: pgmap v1479: 47 pgs: 4 active+clean+remapped, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized, 5 active+undersized+degraded, 29 active+clean; 6.8 GiB data, 29 GiB used, 866 GiB / 894 GiB avail; 60 MiB/s wr, 962 op/s; 41847/331866 objects degraded (12.610%); 17058/331866 objects misplaced (5.140%); 9.7 MiB/s, 154 objects/s recovering 2024-08-19T14:34:58.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:57 smithi032 ceph-mon[20004]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:34:58.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:57 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3581602723' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:34:58.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:57 smithi032 ceph-mon[20004]: osdmap e629: 12 total, 11 up, 10 in 2024-08-19T14:34:58.320 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:34:58.350 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:58.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:58.351 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:34:58.448 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 85 16 113949 113933 83.7664 51.375 0.0105949 0.0119332 2024-08-19T14:34:58.451 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:34:58.488 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:58.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:58.489 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:34:58.590 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:34:58.611 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:58.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:58.612 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:34:58.616 INFO:journalctl@ceph.osd.2.smithi031.stdout:Aug 19 14:34:58 smithi031 podman[180865]: 2024-08-19 14:34:58.26890473 +0000 UTC m=+2.556924266 container died c981e9ebb912e1c238fb076e9b7ceb32afa48eabb734fa28ea6cfe6517dd5b94 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-2-activate, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-de09dd0) 2024-08-19T14:34:58.713 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:34:58.738 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:58.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:58.739 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:34:58.840 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:34:58.876 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:58.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:58.877 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:34:58.978 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:34:59.000 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:59.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:59.001 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:34:59.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:58 smithi045 ceph-mon[20143]: 1.15 scrub starts 2024-08-19T14:34:59.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:58 smithi045 ceph-mon[20143]: 1.15 scrub ok 2024-08-19T14:34:59.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:58 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 41847/331866 objects degraded (12.610%), 7 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-08-19T14:34:59.102 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:34:59.132 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:59.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:59.133 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:34:59.234 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:34:59.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:58 smithi032 ceph-mon[20004]: 1.15 scrub starts 2024-08-19T14:34:59.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:58 smithi032 ceph-mon[20004]: 1.15 scrub ok 2024-08-19T14:34:59.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:58 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 41847/331866 objects degraded (12.610%), 7 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-08-19T14:34:59.258 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:59.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:59.259 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:34:59.295 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-08-19T14:34:59.359 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:34:59.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:58 smithi031 ceph-mon[16336]: 1.15 scrub starts 2024-08-19T14:34:59.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:58 smithi031 ceph-mon[16336]: 1.15 scrub ok 2024-08-19T14:34:59.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:58 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 41847/331866 objects degraded (12.610%), 7 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-08-19T14:34:59.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:59.393 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:59.393 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:34:59.448 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 86 16 114808 114792 83.4166 53.6875 0.0219936 0.0119818 2024-08-19T14:34:59.494 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:34:59.515 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:59.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:59.516 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:34:59.617 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:34:59.651 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:34:59.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:59.652 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:34:59.691 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:34:59.753 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:34:59.775 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:34:59.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:59.779 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:34:59.880 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:34:59.912 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:34:59.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:34:59.913 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:35:00.014 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:35:00.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:00.052 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:00.052 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:35:00.123 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:59 smithi045 ceph-mon[20143]: pgmap v1481: 47 pgs: 4 active+clean+remapped, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized, 5 active+undersized+degraded, 29 active+clean; 6.8 GiB data, 29 GiB used, 865 GiB / 894 GiB avail; 49 MiB/s wr, 778 op/s; 42096/333429 objects degraded (12.625%); 16537/333429 objects misplaced (4.960%); 12 MiB/s, 192 objects/s recovering 2024-08-19T14:35:00.123 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:59 smithi045 ceph-mon[20143]: 1.10 deep-scrub starts 2024-08-19T14:35:00.123 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:59 smithi045 ceph-mon[20143]: 1.10 deep-scrub ok 2024-08-19T14:35:00.123 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:59 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:35:00.123 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:34:59 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:35:00.153 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:35:00.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:00.195 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:00.195 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:35:00.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:59 smithi032 ceph-mon[20004]: pgmap v1481: 47 pgs: 4 active+clean+remapped, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized, 5 active+undersized+degraded, 29 active+clean; 6.8 GiB data, 29 GiB used, 865 GiB / 894 GiB avail; 49 MiB/s wr, 778 op/s; 42096/333429 objects degraded (12.625%); 16537/333429 objects misplaced (4.960%); 12 MiB/s, 192 objects/s recovering 2024-08-19T14:35:00.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:59 smithi032 ceph-mon[20004]: 1.10 deep-scrub starts 2024-08-19T14:35:00.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:59 smithi032 ceph-mon[20004]: 1.10 deep-scrub ok 2024-08-19T14:35:00.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:59 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:35:00.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:34:59 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:35:00.286 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:35:00.286 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:35:00.296 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:35:00.310 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-19T14:35:00.329 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:00.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:00.331 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:35:00.364 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:59 smithi031 ceph-mon[16336]: pgmap v1481: 47 pgs: 4 active+clean+remapped, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized, 5 active+undersized+degraded, 29 active+clean; 6.8 GiB data, 29 GiB used, 865 GiB / 894 GiB avail; 49 MiB/s wr, 778 op/s; 42096/333429 objects degraded (12.625%); 16537/333429 objects misplaced (4.960%); 12 MiB/s, 192 objects/s recovering 2024-08-19T14:35:00.365 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:59 smithi031 ceph-mon[16336]: 1.10 deep-scrub starts 2024-08-19T14:35:00.365 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:59 smithi031 ceph-mon[16336]: 1.10 deep-scrub ok 2024-08-19T14:35:00.365 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:59 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:35:00.365 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:34:59 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:35:00.432 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:35:00.448 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 87 16 115656 115640 83.067 53 0.0123678 0.012032 2024-08-19T14:35:00.455 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:00.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:00.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:35:00.557 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:35:00.581 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:00.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:00.582 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:35:00.683 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:35:00.705 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:00.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:00.706 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:35:00.807 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:35:00.829 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:00.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:00.830 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:35:00.931 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:35:00.953 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:00.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:00.954 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:35:01.055 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:35:01.089 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:01.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:01.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:35:01.191 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:35:01.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:00 smithi032 ceph-mon[20004]: osdmap e630: 12 total, 11 up, 10 in 2024-08-19T14:35:01.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:01.258 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:01.259 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:35:01.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:00 smithi045 ceph-mon[20143]: osdmap e630: 12 total, 11 up, 10 in 2024-08-19T14:35:01.314 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:35:01.360 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:35:01.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:00 smithi031 ceph-mon[16336]: osdmap e630: 12 total, 11 up, 10 in 2024-08-19T14:35:01.366 INFO:journalctl@ceph.osd.2.smithi031.stdout:Aug 19 14:35:01 smithi031 podman[181061]: 2024-08-19 14:35:01.072282165 +0000 UTC m=+3.721415894 container remove c981e9ebb912e1c238fb076e9b7ceb32afa48eabb734fa28ea6cfe6517dd5b94 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-2-activate, io.buildah.version=1.36.0, ceph=True, RELEASE=squid-de09dd0, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux ) 2024-08-19T14:35:01.393 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:01.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:01.394 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:35:01.448 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 88 16 116552 116536 82.7593 56 0.0083408 0.0120781 2024-08-19T14:35:01.495 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:35:01.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:01.528 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:01.528 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:35:01.629 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:35:01.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:01.658 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:01.658 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:35:01.759 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:35:01.767 INFO:journalctl@ceph.osd.2.smithi031.stdout:Aug 19 14:35:01 smithi031 podman[181315]: 2024-08-19 14:35:01.345710856 +0000 UTC m=+0.039811228 image pull ea7a5189f30bad3aca9524788e2965d2368d7ce786df09d4edcd8c3edf9cefba quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9 2024-08-19T14:35:01.768 INFO:journalctl@ceph.osd.2.smithi031.stdout:Aug 19 14:35:01 smithi031 podman[181315]: 2024-08-19 14:35:01.47317422 +0000 UTC m=+0.167274583 container create 6126bfc5c5138d398119df06d2108661bc6f5f7fcd67ca072724adc90759d7e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240731, RELEASE=squid-de09dd0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=) 2024-08-19T14:35:01.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:01.808 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:01.809 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:35:01.910 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:35:01.941 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:01.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:01.943 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:35:02.044 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:35:02.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:02.073 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:02.074 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:35:02.114 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:01 smithi031 ceph-mon[16336]: pgmap v1483: 47 pgs: 4 active+clean+remapped, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized, 5 active+undersized+degraded, 29 active+clean; 6.9 GiB data, 29 GiB used, 865 GiB / 894 GiB avail; 61 MiB/s wr, 968 op/s; 42340/339519 objects degraded (12.471%); 16277/339519 objects misplaced (4.794%); 12 MiB/s, 198 objects/s recovering 2024-08-19T14:35:02.114 INFO:journalctl@ceph.osd.2.smithi031.stdout:Aug 19 14:35:02 smithi031 podman[181315]: 2024-08-19 14:35:02.11350783 +0000 UTC m=+0.807608204 container init 6126bfc5c5138d398119df06d2108661bc6f5f7fcd67ca072724adc90759d7e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, RELEASE=squid-de09dd0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-08-19T14:35:02.174 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:35:02.202 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:02.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:02.203 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:35:02.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:01 smithi032 ceph-mon[20004]: pgmap v1483: 47 pgs: 4 active+clean+remapped, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized, 5 active+undersized+degraded, 29 active+clean; 6.9 GiB data, 29 GiB used, 865 GiB / 894 GiB avail; 61 MiB/s wr, 968 op/s; 42340/339519 objects degraded (12.471%); 16277/339519 objects misplaced (4.794%); 12 MiB/s, 198 objects/s recovering 2024-08-19T14:35:02.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:01 smithi045 ceph-mon[20143]: pgmap v1483: 47 pgs: 4 active+clean+remapped, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized, 5 active+undersized+degraded, 29 active+clean; 6.9 GiB data, 29 GiB used, 865 GiB / 894 GiB avail; 61 MiB/s wr, 968 op/s; 42340/339519 objects degraded (12.471%); 16277/339519 objects misplaced (4.794%); 12 MiB/s, 198 objects/s recovering 2024-08-19T14:35:02.304 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:35:02.326 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:02.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:02.327 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:35:02.368 INFO:journalctl@ceph.osd.2.smithi031.stdout:Aug 19 14:35:02 smithi031 podman[181315]: 2024-08-19 14:35:02.12024862 +0000 UTC m=+0.814348989 container start 6126bfc5c5138d398119df06d2108661bc6f5f7fcd67ca072724adc90759d7e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=squid-de09dd0, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD) 2024-08-19T14:35:02.428 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:35:02.448 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 89 16 117435 117419 82.4495 55.1875 0.0106252 0.0121229 2024-08-19T14:35:02.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:02.474 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:02.475 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:35:02.502 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-08-19T14:35:02.576 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:35:02.620 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:02.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:02.621 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:35:02.723 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:35:02.755 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:02.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:02.756 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:35:02.857 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:35:02.901 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:02.901 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:35:02.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:02.902 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:35:03.003 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:35:03.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:03.045 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:03.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:35:03.073 INFO:journalctl@ceph.osd.2.smithi031.stdout:Aug 19 14:35:02 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-2[181347]: 2024-08-19T14:35:02.757+0000 7fcb824a2640 -1 Falling back to public interface 2024-08-19T14:35:03.074 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:02 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:35:03.074 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:02 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:35:03.074 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:02 smithi031 ceph-mon[16336]: osdmap e631: 12 total, 11 up, 10 in 2024-08-19T14:35:03.074 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:02 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:35:03.074 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:02 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:35:03.074 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:02 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2602285210' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:35:03.074 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:02 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 42340/339519 objects degraded (12.471%), 7 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-08-19T14:35:03.083 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-08-19T14:35:03.149 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:35:03.176 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:03.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:03.177 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:35:03.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:02 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:35:03.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:02 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:35:03.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:02 smithi032 ceph-mon[20004]: osdmap e631: 12 total, 11 up, 10 in 2024-08-19T14:35:03.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:02 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:35:03.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:02 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:35:03.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:02 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2602285210' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:35:03.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:02 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 42340/339519 objects degraded (12.471%), 7 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-08-19T14:35:03.278 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:35:03.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:02 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:35:03.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:02 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:35:03.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:02 smithi045 ceph-mon[20143]: osdmap e631: 12 total, 11 up, 10 in 2024-08-19T14:35:03.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:02 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:35:03.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:02 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:35:03.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:02 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2602285210' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:35:03.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:02 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 42340/339519 objects degraded (12.471%), 7 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-08-19T14:35:03.299 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:03.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:03.300 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:35:03.366 INFO:journalctl@ceph.osd.2.smithi031.stdout:Aug 19 14:35:03 smithi031 bash[181315]: 6126bfc5c5138d398119df06d2108661bc6f5f7fcd67ca072724adc90759d7e7 2024-08-19T14:35:03.366 INFO:journalctl@ceph.osd.2.smithi031.stdout:Aug 19 14:35:03 smithi031 systemd[1]: Started Ceph osd.2 for ed7d1c6e-5e33-11ef-bccf-c7b262605968. 2024-08-19T14:35:03.401 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:35:03.422 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:03.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:03.423 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:35:03.448 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 90 16 118157 118141 82.0348 45.125 0.0043877 0.0121649 2024-08-19T14:35:03.524 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:35:03.544 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:03.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:03.545 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:35:03.646 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:35:03.667 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:03.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:03.668 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:35:03.746 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:35:03.747 INFO:teuthology.orchestra.run.smithi045.stdout: "size": 20, 2024-08-19T14:35:03.747 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 600, 2024-08-19T14:35:03.747 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [ 2024-08-19T14:35:03.747 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.747 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.16962.0:76277 7.e e618/617 7:731ae8a5:::benchmark_data_smithi045_115842_object76276:head v 618'4887, mlcod=605'7)", 2024-08-19T14:35:03.747 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:34:14.255266+0000", 2024-08-19T14:35:03.747 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 49.489718144000001, 2024-08-19T14:35:03.747 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.058286588, 2024-08-19T14:35:03.747 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:35:03.747 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:35:03.747 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:35:03.748 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:35:03.748 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.748 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:35:03.748 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.255266+0000", 2024-08-19T14:35:03.748 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:35:03.748 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.748 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.748 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:35:03.748 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.255266+0000", 2024-08-19T14:35:03.748 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:35:03.748 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.748 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.749 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:35:03.749 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.255267+0000", 2024-08-19T14:35:03.749 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.8599999999999998e-07 2024-08-19T14:35:03.749 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.749 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.749 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:35:03.749 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.255324+0000", 2024-08-19T14:35:03.749 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7120000000000002e-05 2024-08-19T14:35:03.749 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.749 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.749 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:35:03.750 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.255333+0000", 2024-08-19T14:35:03.750 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.1549999999999996e-06 2024-08-19T14:35:03.750 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.750 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.750 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:35:03.750 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.255339+0000", 2024-08-19T14:35:03.750 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.6339999999999998e-06 2024-08-19T14:35:03.751 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.751 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.751 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:35:03.751 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.255389+0000", 2024-08-19T14:35:03.751 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0219999999999997e-05 2024-08-19T14:35:03.751 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.751 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.751 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:35:03.751 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.255410+0000", 2024-08-19T14:35:03.751 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1421e-05 2024-08-19T14:35:03.751 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.751 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.751 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:35:03.751 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.313471+0000", 2024-08-19T14:35:03.752 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.058061029 2024-08-19T14:35:03.752 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.752 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.752 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:35:03.752 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.313553+0000", 2024-08-19T14:35:03.752 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.1422999999999996e-05 2024-08-19T14:35:03.752 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:35:03.752 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:35:03.752 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:35:03.752 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.752 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.752 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.16962.0:76286 7.e e618/617 7:795fa467:::benchmark_data_smithi045_115842_object76285:head v 618'4888, mlcod=605'7)", 2024-08-19T14:35:03.752 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:34:14.258319+0000", 2024-08-19T14:35:03.753 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 49.486665369999997, 2024-08-19T14:35:03.753 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.055304399999999997, 2024-08-19T14:35:03.753 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:35:03.753 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:35:03.753 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:35:03.753 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:35:03.753 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.753 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:35:03.753 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.258319+0000", 2024-08-19T14:35:03.753 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:35:03.753 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.753 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.753 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:35:03.753 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.258319+0000", 2024-08-19T14:35:03.754 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:35:03.754 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.754 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.754 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:35:03.754 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.258320+0000", 2024-08-19T14:35:03.754 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7599999999999997e-07 2024-08-19T14:35:03.754 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.754 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.754 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:35:03.754 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.258367+0000", 2024-08-19T14:35:03.754 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.7895000000000001e-05 2024-08-19T14:35:03.755 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.755 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.755 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:35:03.755 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.258377+0000", 2024-08-19T14:35:03.755 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.9499999999999996e-06 2024-08-19T14:35:03.755 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.755 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.755 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:35:03.755 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.258384+0000", 2024-08-19T14:35:03.755 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.3239999999999999e-06 2024-08-19T14:35:03.755 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.755 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.755 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:35:03.756 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.261676+0000", 2024-08-19T14:35:03.756 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0032921529999999999 2024-08-19T14:35:03.756 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.756 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.756 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:35:03.756 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.261698+0000", 2024-08-19T14:35:03.756 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2260999999999999e-05 2024-08-19T14:35:03.756 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.756 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.756 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:35:03.756 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.313557+0000", 2024-08-19T14:35:03.756 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.051858429999999997 2024-08-19T14:35:03.756 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.756 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.757 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:35:03.757 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.313623+0000", 2024-08-19T14:35:03.757 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.6810999999999994e-05 2024-08-19T14:35:03.757 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:35:03.757 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:35:03.757 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:35:03.757 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.757 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.757 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.16962.0:76288 7.e e618/617 7:70295cf8:::benchmark_data_smithi045_115842_object76287:head v 618'4889, mlcod=605'7)", 2024-08-19T14:35:03.757 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:34:14.261026+0000", 2024-08-19T14:35:03.757 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 49.483958051000002, 2024-08-19T14:35:03.757 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.052634357999999999, 2024-08-19T14:35:03.757 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:35:03.758 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:35:03.758 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:35:03.758 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:35:03.758 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.758 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:35:03.758 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.261026+0000", 2024-08-19T14:35:03.758 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:35:03.758 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.758 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.758 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:35:03.758 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.261026+0000", 2024-08-19T14:35:03.758 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:35:03.758 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.759 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.759 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:35:03.759 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.261027+0000", 2024-08-19T14:35:03.759 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.5000000000000001e-07 2024-08-19T14:35:03.759 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.759 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.759 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:35:03.759 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.261065+0000", 2024-08-19T14:35:03.759 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.7907000000000001e-05 2024-08-19T14:35:03.759 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.759 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.759 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:35:03.759 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.261079+0000", 2024-08-19T14:35:03.759 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3767e-05 2024-08-19T14:35:03.760 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.760 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.760 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:35:03.760 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.261087+0000", 2024-08-19T14:35:03.760 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.8809999999999996e-06 2024-08-19T14:35:03.760 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.760 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.760 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:35:03.760 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.288780+0000", 2024-08-19T14:35:03.760 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.027693457000000001 2024-08-19T14:35:03.760 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.760 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.760 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:35:03.761 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.288804+0000", 2024-08-19T14:35:03.761 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.3886000000000001e-05 2024-08-19T14:35:03.761 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.761 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.761 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:35:03.761 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.313626+0000", 2024-08-19T14:35:03.761 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.024822331999999999 2024-08-19T14:35:03.761 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.761 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.761 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:35:03.761 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.313661+0000", 2024-08-19T14:35:03.761 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.4177999999999997e-05 2024-08-19T14:35:03.761 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:35:03.762 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:35:03.762 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:35:03.762 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.762 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.762 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.16962.0:76303 7.e e618/617 7:77649f0f:::benchmark_data_smithi045_115842_object76302:head v 618'4890, mlcod=605'7)", 2024-08-19T14:35:03.762 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:34:14.264011+0000", 2024-08-19T14:35:03.763 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 49.480973001999999, 2024-08-19T14:35:03.763 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.049686299000000003, 2024-08-19T14:35:03.763 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:35:03.763 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:35:03.763 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:35:03.763 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:35:03.763 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.763 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:35:03.763 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.264011+0000", 2024-08-19T14:35:03.764 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:35:03.764 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.764 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.764 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:35:03.764 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.264011+0000", 2024-08-19T14:35:03.764 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:35:03.764 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.764 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.764 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:35:03.764 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.264012+0000", 2024-08-19T14:35:03.764 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.16e-07 2024-08-19T14:35:03.764 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.764 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.765 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:35:03.765 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.264059+0000", 2024-08-19T14:35:03.765 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.6900000000000002e-05 2024-08-19T14:35:03.765 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.765 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.765 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:35:03.765 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.264076+0000", 2024-08-19T14:35:03.765 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7248000000000002e-05 2024-08-19T14:35:03.765 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.765 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.765 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:35:03.765 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.264083+0000", 2024-08-19T14:35:03.765 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.5200000000000003e-06 2024-08-19T14:35:03.766 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.766 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.766 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:35:03.766 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.294900+0000", 2024-08-19T14:35:03.766 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.030817157000000001 2024-08-19T14:35:03.766 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.766 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.766 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:35:03.766 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.294931+0000", 2024-08-19T14:35:03.766 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.1043000000000001e-05 2024-08-19T14:35:03.766 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.766 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.767 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:35:03.767 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.313667+0000", 2024-08-19T14:35:03.767 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.018735928999999998 2024-08-19T14:35:03.767 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.767 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.767 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:35:03.767 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.313698+0000", 2024-08-19T14:35:03.767 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.0685999999999997e-05 2024-08-19T14:35:03.767 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:35:03.767 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:35:03.767 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:35:03.767 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.767 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.768 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.16962.0:76625 7.e e618/617 7:789a4a2f:::benchmark_data_smithi045_115842_object76624:head v 618'4910, mlcod=605'15)", 2024-08-19T14:35:03.768 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:34:14.619525+0000", 2024-08-19T14:35:03.768 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 49.125459458999998, 2024-08-19T14:35:03.768 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.047508896000000002, 2024-08-19T14:35:03.768 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:35:03.768 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:35:03.768 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:35:03.768 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:35:03.768 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.768 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:35:03.768 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.619525+0000", 2024-08-19T14:35:03.768 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:35:03.768 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.769 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.769 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:35:03.769 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.619525+0000", 2024-08-19T14:35:03.769 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:35:03.769 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.769 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.769 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:35:03.769 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.619526+0000", 2024-08-19T14:35:03.769 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1310000000000001e-06 2024-08-19T14:35:03.769 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.769 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.769 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:35:03.769 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.619629+0000", 2024-08-19T14:35:03.770 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00010254700000000001 2024-08-19T14:35:03.770 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.770 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.770 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:35:03.770 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.619640+0000", 2024-08-19T14:35:03.770 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.187e-05 2024-08-19T14:35:03.770 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.770 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.770 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:35:03.770 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.619647+0000", 2024-08-19T14:35:03.770 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.5470000000000002e-06 2024-08-19T14:35:03.770 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.771 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.771 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:35:03.771 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.619828+0000", 2024-08-19T14:35:03.771 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00018073699999999999 2024-08-19T14:35:03.771 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.771 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.771 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:35:03.771 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.619894+0000", 2024-08-19T14:35:03.771 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.6544999999999994e-05 2024-08-19T14:35:03.771 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.771 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.771 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:35:03.772 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.666838+0000", 2024-08-19T14:35:03.772 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.046943529999999997 2024-08-19T14:35:03.772 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.772 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.772 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:35:03.772 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.667034+0000", 2024-08-19T14:35:03.772 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00019598900000000001 2024-08-19T14:35:03.772 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:35:03.772 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:35:03.772 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:35:03.772 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.772 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.772 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.16962.0:85089 7.f e622/617 7:fb2cc53f:::benchmark_data_smithi045_115842_object85088:head v 622'5295, mlcod=605'208)", 2024-08-19T14:35:03.772 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:34:23.914661+0000", 2024-08-19T14:35:03.773 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 39.830322959, 2024-08-19T14:35:03.773 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.051211051, 2024-08-19T14:35:03.773 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:35:03.773 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:35:03.773 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:35:03.773 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:35:03.773 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.773 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:35:03.773 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:23.914661+0000", 2024-08-19T14:35:03.773 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:35:03.773 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.773 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.773 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:35:03.774 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:23.914661+0000", 2024-08-19T14:35:03.774 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:35:03.774 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.774 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.774 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:35:03.774 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:23.914662+0000", 2024-08-19T14:35:03.774 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.6400000000000003e-07 2024-08-19T14:35:03.774 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.774 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.774 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:35:03.774 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:23.914693+0000", 2024-08-19T14:35:03.774 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.1647999999999999e-05 2024-08-19T14:35:03.775 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.775 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.775 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:35:03.775 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:23.914702+0000", 2024-08-19T14:35:03.775 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.5890000000000003e-06 2024-08-19T14:35:03.775 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.775 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.775 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:35:03.775 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:23.914708+0000", 2024-08-19T14:35:03.775 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.5029999999999997e-06 2024-08-19T14:35:03.775 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.775 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.775 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:35:03.776 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:23.924683+0000", 2024-08-19T14:35:03.776 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0099756689999999995 2024-08-19T14:35:03.776 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.776 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.776 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:35:03.776 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:23.924709+0000", 2024-08-19T14:35:03.776 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.5924000000000001e-05 2024-08-19T14:35:03.776 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.776 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.776 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:35:03.776 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:23.965829+0000", 2024-08-19T14:35:03.776 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.041120205999999999 2024-08-19T14:35:03.776 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.776 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.777 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:35:03.777 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:23.965872+0000", 2024-08-19T14:35:03.777 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.3047999999999998e-05 2024-08-19T14:35:03.777 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:35:03.777 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:35:03.777 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:35:03.777 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.777 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.777 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.16962.0:85096 7.f e622/617 7:fbe5a55f:::benchmark_data_smithi045_115842_object85095:head v 622'5296, mlcod=605'208)", 2024-08-19T14:35:03.777 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:34:23.918667+0000", 2024-08-19T14:35:03.777 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 39.826317068999998, 2024-08-19T14:35:03.777 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.047242412999999997, 2024-08-19T14:35:03.777 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:35:03.778 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:35:03.778 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:35:03.778 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:35:03.778 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.778 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:35:03.778 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:23.918667+0000", 2024-08-19T14:35:03.778 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:35:03.778 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.778 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.778 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:35:03.778 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:23.918667+0000", 2024-08-19T14:35:03.778 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:35:03.778 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.779 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.779 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:35:03.779 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:23.918668+0000", 2024-08-19T14:35:03.779 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7599999999999997e-07 2024-08-19T14:35:03.779 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.779 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.779 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:35:03.779 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:23.918700+0000", 2024-08-19T14:35:03.779 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.2258999999999997e-05 2024-08-19T14:35:03.779 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.779 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.780 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:35:03.780 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:23.918712+0000", 2024-08-19T14:35:03.780 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1620000000000001e-05 2024-08-19T14:35:03.780 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.780 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.780 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:35:03.780 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:23.918719+0000", 2024-08-19T14:35:03.780 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.8920000000000003e-06 2024-08-19T14:35:03.780 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.780 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.780 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:35:03.780 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:23.930848+0000", 2024-08-19T14:35:03.780 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.012129423 2024-08-19T14:35:03.780 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.781 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.781 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:35:03.781 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:23.930871+0000", 2024-08-19T14:35:03.781 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2719000000000001e-05 2024-08-19T14:35:03.781 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.781 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.781 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:35:03.781 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:23.965882+0000", 2024-08-19T14:35:03.781 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.035011476999999999 2024-08-19T14:35:03.781 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.781 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.781 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:35:03.781 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:23.965910+0000", 2024-08-19T14:35:03.781 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.7447e-05 2024-08-19T14:35:03.782 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:35:03.782 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:35:03.782 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:35:03.782 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.782 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.782 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.16962.0:105364 7.f e627/617 7:f24bfbcd:::benchmark_data_smithi045_115842_object105363:head v 627'6473, mlcod=607'833)", 2024-08-19T14:35:03.782 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:34:47.948278+0000", 2024-08-19T14:35:03.782 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 15.796706545999999, 2024-08-19T14:35:03.782 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.067695303999999998, 2024-08-19T14:35:03.782 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:35:03.782 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:35:03.782 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:35:03.783 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:35:03.783 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.783 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:35:03.783 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.948278+0000", 2024-08-19T14:35:03.783 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:35:03.783 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.783 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.783 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:35:03.783 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.948278+0000", 2024-08-19T14:35:03.783 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:35:03.783 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.783 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.783 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:35:03.784 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.948279+0000", 2024-08-19T14:35:03.784 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.1200000000000002e-07 2024-08-19T14:35:03.784 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.784 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.784 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:35:03.784 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.948321+0000", 2024-08-19T14:35:03.784 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.2849999999999998e-05 2024-08-19T14:35:03.784 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.784 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.784 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:35:03.784 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.948336+0000", 2024-08-19T14:35:03.784 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4107999999999999e-05 2024-08-19T14:35:03.784 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.784 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.785 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:35:03.785 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.948343+0000", 2024-08-19T14:35:03.785 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.0840000000000003e-06 2024-08-19T14:35:03.785 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.785 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.785 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:35:03.785 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.949790+0000", 2024-08-19T14:35:03.785 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001447776 2024-08-19T14:35:03.785 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.785 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.785 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:35:03.785 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.949818+0000", 2024-08-19T14:35:03.785 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.7294999999999999e-05 2024-08-19T14:35:03.785 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.786 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.786 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:35:03.786 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.015860+0000", 2024-08-19T14:35:03.786 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.066042187000000002 2024-08-19T14:35:03.786 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.786 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.786 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:35:03.786 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.015973+0000", 2024-08-19T14:35:03.786 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000113192 2024-08-19T14:35:03.786 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:35:03.786 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:35:03.786 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:35:03.787 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.787 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.787 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.16962.0:105377 7.f e627/617 7:f91fc140:::benchmark_data_smithi045_115842_object105376:head v 627'6474, mlcod=607'833)", 2024-08-19T14:35:03.787 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:34:47.951714+0000", 2024-08-19T14:35:03.787 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 15.793270473, 2024-08-19T14:35:03.787 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.064366294000000004, 2024-08-19T14:35:03.787 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:35:03.787 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:35:03.787 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:35:03.787 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:35:03.787 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.787 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:35:03.787 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.951714+0000", 2024-08-19T14:35:03.787 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:35:03.787 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.788 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.788 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:35:03.788 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.951714+0000", 2024-08-19T14:35:03.788 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:35:03.788 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.788 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.788 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:35:03.788 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.951721+0000", 2024-08-19T14:35:03.788 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.2339999999999997e-06 2024-08-19T14:35:03.788 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.788 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.788 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:35:03.788 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.951750+0000", 2024-08-19T14:35:03.788 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.8495000000000001e-05 2024-08-19T14:35:03.788 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.789 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.789 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:35:03.789 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.951765+0000", 2024-08-19T14:35:03.789 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5506e-05 2024-08-19T14:35:03.789 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.789 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.789 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:35:03.789 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.951772+0000", 2024-08-19T14:35:03.789 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.3139999999999998e-06 2024-08-19T14:35:03.789 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.789 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.789 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:35:03.789 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.956679+0000", 2024-08-19T14:35:03.789 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0049063099999999997 2024-08-19T14:35:03.790 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.790 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.790 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:35:03.790 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.956786+0000", 2024-08-19T14:35:03.790 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000107295 2024-08-19T14:35:03.790 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.790 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.790 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:35:03.790 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.015978+0000", 2024-08-19T14:35:03.790 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.059191524000000002 2024-08-19T14:35:03.790 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.790 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.790 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:35:03.791 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.016080+0000", 2024-08-19T14:35:03.791 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00010261599999999999 2024-08-19T14:35:03.791 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:35:03.791 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:35:03.791 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:35:03.791 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.791 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.791 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.16962.0:105379 7.f e627/617 7:f49d0828:::benchmark_data_smithi045_115842_object105378:head v 627'6475, mlcod=607'833)", 2024-08-19T14:35:03.791 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:34:47.956593+0000", 2024-08-19T14:35:03.791 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 15.788390875999999, 2024-08-19T14:35:03.791 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.069375785999999995, 2024-08-19T14:35:03.791 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:35:03.791 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:35:03.791 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:35:03.792 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:35:03.792 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.792 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:35:03.792 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.956593+0000", 2024-08-19T14:35:03.792 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:35:03.792 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.792 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.792 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:35:03.792 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.956593+0000", 2024-08-19T14:35:03.792 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:35:03.792 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.792 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.792 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:35:03.792 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.956595+0000", 2024-08-19T14:35:03.792 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1289999999999999e-06 2024-08-19T14:35:03.793 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.793 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.793 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:35:03.793 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.956637+0000", 2024-08-19T14:35:03.793 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.2800999999999999e-05 2024-08-19T14:35:03.793 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.793 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.793 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:35:03.793 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.956652+0000", 2024-08-19T14:35:03.793 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4129e-05 2024-08-19T14:35:03.793 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.793 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.793 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:35:03.793 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.956661+0000", 2024-08-19T14:35:03.794 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.7559999999999994e-06 2024-08-19T14:35:03.794 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.794 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.794 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:35:03.794 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.964123+0000", 2024-08-19T14:35:03.794 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0074622000000000004 2024-08-19T14:35:03.794 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.794 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.794 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:35:03.794 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.964187+0000", 2024-08-19T14:35:03.794 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.3362999999999996e-05 2024-08-19T14:35:03.794 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.794 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.795 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:35:03.795 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.025913+0000", 2024-08-19T14:35:03.795 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.061725976000000002 2024-08-19T14:35:03.795 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.795 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.795 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:35:03.795 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.025969+0000", 2024-08-19T14:35:03.795 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.6431999999999998e-05 2024-08-19T14:35:03.795 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:35:03.795 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:35:03.795 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:35:03.795 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.795 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.796 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.16962.0:105380 7.f e627/617 7:fd82eef6:::benchmark_data_smithi045_115842_object105379:head v 627'6476, mlcod=607'833)", 2024-08-19T14:35:03.796 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:34:47.958978+0000", 2024-08-19T14:35:03.796 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 15.786006055, 2024-08-19T14:35:03.796 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.067030355, 2024-08-19T14:35:03.796 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:35:03.796 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:35:03.796 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:35:03.796 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:35:03.796 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.796 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:35:03.796 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.958978+0000", 2024-08-19T14:35:03.796 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:35:03.796 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.797 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.797 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:35:03.797 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.958978+0000", 2024-08-19T14:35:03.797 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:35:03.797 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.797 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.797 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:35:03.797 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.958979+0000", 2024-08-19T14:35:03.797 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.1299999999999999e-07 2024-08-19T14:35:03.797 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.797 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.797 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:35:03.797 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.959032+0000", 2024-08-19T14:35:03.798 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.3072999999999998e-05 2024-08-19T14:35:03.798 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.798 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.798 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:35:03.798 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.959046+0000", 2024-08-19T14:35:03.798 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3702e-05 2024-08-19T14:35:03.798 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.798 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.798 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:35:03.798 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.959053+0000", 2024-08-19T14:35:03.798 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.6579999999999996e-06 2024-08-19T14:35:03.798 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.798 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.799 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:35:03.799 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.974551+0000", 2024-08-19T14:35:03.799 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.015498168 2024-08-19T14:35:03.799 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.799 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.799 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:35:03.799 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.974576+0000", 2024-08-19T14:35:03.799 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.5502000000000001e-05 2024-08-19T14:35:03.799 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.799 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.799 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:35:03.799 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.025972+0000", 2024-08-19T14:35:03.799 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.051395733999999998 2024-08-19T14:35:03.800 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.800 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.800 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:35:03.800 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.026009+0000", 2024-08-19T14:35:03.800 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.6705000000000002e-05 2024-08-19T14:35:03.800 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:35:03.800 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:35:03.800 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:35:03.800 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.800 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.800 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.16962.0:105387 7.f e627/617 7:f7f9de62:::benchmark_data_smithi045_115842_object105386:head v 627'6477, mlcod=607'833)", 2024-08-19T14:35:03.800 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:34:47.962787+0000", 2024-08-19T14:35:03.800 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 15.782197581, 2024-08-19T14:35:03.800 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.063250878999999996, 2024-08-19T14:35:03.801 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:35:03.801 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:35:03.801 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:35:03.801 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:35:03.801 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.801 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:35:03.801 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.962787+0000", 2024-08-19T14:35:03.801 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:35:03.801 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.801 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.801 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:35:03.801 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.962787+0000", 2024-08-19T14:35:03.801 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:35:03.801 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.802 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.802 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:35:03.802 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.962788+0000", 2024-08-19T14:35:03.802 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0210000000000001e-06 2024-08-19T14:35:03.802 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.802 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.802 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:35:03.802 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.962824+0000", 2024-08-19T14:35:03.802 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.6328999999999997e-05 2024-08-19T14:35:03.802 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.802 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.802 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:35:03.802 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.962841+0000", 2024-08-19T14:35:03.803 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6858e-05 2024-08-19T14:35:03.803 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.803 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.803 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:35:03.803 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.962850+0000", 2024-08-19T14:35:03.803 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.8380000000000008e-06 2024-08-19T14:35:03.803 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.803 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.803 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:35:03.803 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.980518+0000", 2024-08-19T14:35:03.803 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.017668120999999998 2024-08-19T14:35:03.803 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.803 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.803 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:35:03.804 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.980548+0000", 2024-08-19T14:35:03.804 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.9825e-05 2024-08-19T14:35:03.804 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.804 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.804 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:35:03.804 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.026011+0000", 2024-08-19T14:35:03.804 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.045463389999999999 2024-08-19T14:35:03.804 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.804 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.804 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:35:03.804 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.026038+0000", 2024-08-19T14:35:03.804 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.6497000000000001e-05 2024-08-19T14:35:03.804 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:35:03.804 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:35:03.805 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:35:03.805 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.805 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.805 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.16962.0:105389 7.f e627/617 7:fe4b891a:::benchmark_data_smithi045_115842_object105388:head v 627'6478, mlcod=607'833)", 2024-08-19T14:35:03.805 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:34:47.971841+0000", 2024-08-19T14:35:03.805 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 15.773142963, 2024-08-19T14:35:03.805 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.054228620999999998, 2024-08-19T14:35:03.805 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:35:03.805 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:35:03.805 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:35:03.805 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:35:03.805 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.805 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:35:03.805 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.971841+0000", 2024-08-19T14:35:03.805 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:35:03.806 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.806 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.806 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:35:03.806 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.971841+0000", 2024-08-19T14:35:03.806 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:35:03.806 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.806 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.806 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:35:03.806 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.971842+0000", 2024-08-19T14:35:03.806 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.2500000000000004e-07 2024-08-19T14:35:03.806 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.806 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.806 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:35:03.806 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.971882+0000", 2024-08-19T14:35:03.807 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:35:03.808 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.0009999999999998e-05 2024-08-19T14:35:03.808 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.809 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.809 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:35:03.809 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.971900+0000", 2024-08-19T14:35:03.809 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7827000000000001e-05 2024-08-19T14:35:03.809 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.809 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.809 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:35:03.809 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.971908+0000", 2024-08-19T14:35:03.809 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.2060000000000007e-06 2024-08-19T14:35:03.809 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.809 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.809 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:35:03.809 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.986839+0000", 2024-08-19T14:35:03.810 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.014930986 2024-08-19T14:35:03.810 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.810 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.810 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:35:03.810 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.986866+0000", 2024-08-19T14:35:03.810 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.6441999999999999e-05 2024-08-19T14:35:03.810 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.810 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.810 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:35:03.810 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.026040+0000", 2024-08-19T14:35:03.810 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.039174266999999999 2024-08-19T14:35:03.810 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.811 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.811 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:35:03.811 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.026070+0000", 2024-08-19T14:35:03.811 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.0057999999999999e-05 2024-08-19T14:35:03.811 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:35:03.811 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:35:03.811 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:35:03.811 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.811 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.811 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.16962.0:105392 7.f e627/617 7:ff34f05a:::benchmark_data_smithi045_115842_object105391:head v 627'6479, mlcod=607'833)", 2024-08-19T14:35:03.811 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:34:47.975534+0000", 2024-08-19T14:35:03.811 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 15.769449904, 2024-08-19T14:35:03.811 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.050567490999999999, 2024-08-19T14:35:03.812 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:35:03.812 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:35:03.812 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:35:03.812 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:35:03.812 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.812 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:35:03.812 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.975534+0000", 2024-08-19T14:35:03.812 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:35:03.812 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.812 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.812 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:35:03.812 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.975534+0000", 2024-08-19T14:35:03.812 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:35:03.812 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.812 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.813 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:35:03.813 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.975536+0000", 2024-08-19T14:35:03.813 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3799999999999999e-06 2024-08-19T14:35:03.813 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.813 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.813 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:35:03.813 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.975580+0000", 2024-08-19T14:35:03.813 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.4029000000000001e-05 2024-08-19T14:35:03.813 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.813 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.813 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:35:03.813 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.975600+0000", 2024-08-19T14:35:03.813 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0081999999999998e-05 2024-08-19T14:35:03.813 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.813 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.814 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:35:03.814 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.975609+0000", 2024-08-19T14:35:03.814 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.1290000000000002e-06 2024-08-19T14:35:03.814 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.814 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.814 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:35:03.814 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.993986+0000", 2024-08-19T14:35:03.814 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.018377370000000001 2024-08-19T14:35:03.814 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.814 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.814 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:35:03.814 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.994018+0000", 2024-08-19T14:35:03.814 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.1971000000000003e-05 2024-08-19T14:35:03.814 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.815 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.815 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:35:03.815 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.026072+0000", 2024-08-19T14:35:03.815 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.032053566999999998 2024-08-19T14:35:03.815 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.815 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.815 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:35:03.815 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.026102+0000", 2024-08-19T14:35:03.815 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.9963000000000001e-05 2024-08-19T14:35:03.815 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:35:03.815 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:35:03.815 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:35:03.815 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.815 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.816 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.16962.0:105397 7.f e627/617 7:f80187d8:::benchmark_data_smithi045_115842_object105396:head v 627'6480, mlcod=607'833)", 2024-08-19T14:35:03.816 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:34:47.978095+0000", 2024-08-19T14:35:03.816 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 15.766889357, 2024-08-19T14:35:03.816 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.048037879999999998, 2024-08-19T14:35:03.816 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:35:03.816 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:35:03.816 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:35:03.816 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:35:03.816 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.816 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:35:03.816 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.978095+0000", 2024-08-19T14:35:03.816 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:35:03.816 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.817 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.817 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:35:03.817 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.978095+0000", 2024-08-19T14:35:03.817 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:35:03.817 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.817 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.817 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:35:03.817 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.978096+0000", 2024-08-19T14:35:03.817 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0589999999999999e-06 2024-08-19T14:35:03.817 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.817 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.817 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:35:03.817 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.978132+0000", 2024-08-19T14:35:03.817 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.6171999999999999e-05 2024-08-19T14:35:03.817 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.818 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.818 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:35:03.818 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.978152+0000", 2024-08-19T14:35:03.818 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9639e-05 2024-08-19T14:35:03.818 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.818 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.818 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:35:03.818 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.978161+0000", 2024-08-19T14:35:03.818 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.1290000000000002e-06 2024-08-19T14:35:03.818 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.818 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.818 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:35:03.818 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.000277+0000", 2024-08-19T14:35:03.819 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.022116327000000002 2024-08-19T14:35:03.819 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.819 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.819 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:35:03.819 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.000308+0000", 2024-08-19T14:35:03.819 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.1078000000000003e-05 2024-08-19T14:35:03.819 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.819 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.819 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:35:03.819 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.026104+0000", 2024-08-19T14:35:03.819 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.025795241 2024-08-19T14:35:03.819 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.819 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.820 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:35:03.820 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.026133+0000", 2024-08-19T14:35:03.820 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.9235e-05 2024-08-19T14:35:03.820 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:35:03.820 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:35:03.820 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:35:03.820 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.820 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.820 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.16962.0:106074 7.a e627/617 7:59660197:::benchmark_data_smithi045_115842_object106073:head v 627'6600, mlcod=608'1547)", 2024-08-19T14:35:03.820 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:34:48.721689+0000", 2024-08-19T14:35:03.820 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 15.023295788, 2024-08-19T14:35:03.820 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.052407031999999999, 2024-08-19T14:35:03.820 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:35:03.821 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:35:03.821 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:35:03.821 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:35:03.821 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.821 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:35:03.821 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.721689+0000", 2024-08-19T14:35:03.821 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:35:03.821 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.821 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.821 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:35:03.821 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.721689+0000", 2024-08-19T14:35:03.821 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:35:03.821 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.821 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.822 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:35:03.822 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.721689+0000", 2024-08-19T14:35:03.822 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.2099999999999996e-07 2024-08-19T14:35:03.822 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.822 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.822 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:35:03.822 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.721712+0000", 2024-08-19T14:35:03.822 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2974999999999999e-05 2024-08-19T14:35:03.822 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.822 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.822 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:35:03.822 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.721721+0000", 2024-08-19T14:35:03.822 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.6379999999999993e-06 2024-08-19T14:35:03.823 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.823 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.823 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:35:03.823 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.721726+0000", 2024-08-19T14:35:03.823 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.203e-06 2024-08-19T14:35:03.823 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.823 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.823 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:35:03.823 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.721937+0000", 2024-08-19T14:35:03.823 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00021136399999999999 2024-08-19T14:35:03.823 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.823 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.823 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:35:03.824 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.721957+0000", 2024-08-19T14:35:03.824 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9609000000000001e-05 2024-08-19T14:35:03.824 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.824 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.824 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:35:03.824 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.773975+0000", 2024-08-19T14:35:03.824 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.052018350999999997 2024-08-19T14:35:03.824 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.824 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.824 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:35:03.824 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.774096+0000", 2024-08-19T14:35:03.824 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000120271 2024-08-19T14:35:03.825 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:35:03.825 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:35:03.825 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:35:03.825 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.825 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.825 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.16962.0:106077 7.a e627/617 7:5eb1cc5e:::benchmark_data_smithi045_115842_object106076:head v 627'6601, mlcod=608'1547)", 2024-08-19T14:35:03.825 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:34:48.725534+0000", 2024-08-19T14:35:03.825 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 15.019450815000001, 2024-08-19T14:35:03.825 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.052283246999999998, 2024-08-19T14:35:03.825 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:35:03.825 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:35:03.825 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:35:03.825 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:35:03.825 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.826 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:35:03.826 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.725534+0000", 2024-08-19T14:35:03.826 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:35:03.826 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.826 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.826 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:35:03.826 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.725534+0000", 2024-08-19T14:35:03.826 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:35:03.826 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.826 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.826 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:35:03.826 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.725535+0000", 2024-08-19T14:35:03.827 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.356e-06 2024-08-19T14:35:03.827 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.827 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.827 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:35:03.827 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.725563+0000", 2024-08-19T14:35:03.827 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.8135000000000001e-05 2024-08-19T14:35:03.827 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.827 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.827 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:35:03.827 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.725577+0000", 2024-08-19T14:35:03.827 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4166e-05 2024-08-19T14:35:03.827 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.827 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.827 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:35:03.828 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.725586+0000", 2024-08-19T14:35:03.828 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.1409999999999994e-06 2024-08-19T14:35:03.828 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.828 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.828 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:35:03.828 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.728894+0000", 2024-08-19T14:35:03.828 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0033076749999999999 2024-08-19T14:35:03.828 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.828 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.828 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:35:03.828 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.728919+0000", 2024-08-19T14:35:03.828 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.5488000000000001e-05 2024-08-19T14:35:03.828 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.828 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.829 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:35:03.829 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.777773+0000", 2024-08-19T14:35:03.829 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.048853079000000001 2024-08-19T14:35:03.829 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.829 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.829 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:35:03.829 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.777817+0000", 2024-08-19T14:35:03.829 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.4206999999999999e-05 2024-08-19T14:35:03.829 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:35:03.829 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:35:03.829 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:35:03.829 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.829 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.830 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.16962.0:106082 7.5 e627/617 7:a84af00f:::benchmark_data_smithi045_115842_object106081:head v 627'6560, mlcod=608'1460)", 2024-08-19T14:35:03.830 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:34:48.729996+0000", 2024-08-19T14:35:03.830 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 15.014987963999999, 2024-08-19T14:35:03.830 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.051991066000000002, 2024-08-19T14:35:03.830 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:35:03.830 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:35:03.830 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:35:03.830 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:35:03.830 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.830 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:35:03.830 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.729996+0000", 2024-08-19T14:35:03.831 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:35:03.831 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.831 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.831 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:35:03.831 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.729996+0000", 2024-08-19T14:35:03.831 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:35:03.831 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.831 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.831 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:35:03.831 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.729997+0000", 2024-08-19T14:35:03.831 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.1500000000000004e-07 2024-08-19T14:35:03.831 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.832 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.832 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:35:03.832 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.730022+0000", 2024-08-19T14:35:03.832 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.5282e-05 2024-08-19T14:35:03.832 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.832 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.832 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:35:03.832 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.730033+0000", 2024-08-19T14:35:03.832 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0370000000000001e-05 2024-08-19T14:35:03.832 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.832 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.833 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:35:03.833 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.730039+0000", 2024-08-19T14:35:03.833 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.6699999999999997e-06 2024-08-19T14:35:03.833 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.833 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.833 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:35:03.833 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.732462+0000", 2024-08-19T14:35:03.833 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0024221939999999999 2024-08-19T14:35:03.833 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.833 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.833 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:35:03.833 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.732487+0000", 2024-08-19T14:35:03.833 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.5083e-05 2024-08-19T14:35:03.834 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.834 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.834 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:35:03.834 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.781953+0000", 2024-08-19T14:35:03.834 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.049466036999999997 2024-08-19T14:35:03.834 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.834 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.834 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:35:03.834 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.781987+0000", 2024-08-19T14:35:03.834 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.4715000000000002e-05 2024-08-19T14:35:03.834 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:35:03.834 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:35:03.834 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:35:03.835 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.835 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.835 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.16962.0:106085 7.0 e627/617 7:0349f56d:::benchmark_data_smithi045_115842_object106084:head v 627'6486, mlcod=607'921)", 2024-08-19T14:35:03.835 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:34:48.734348+0000", 2024-08-19T14:35:03.835 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 15.01063647, 2024-08-19T14:35:03.835 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.061543685000000001, 2024-08-19T14:35:03.835 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:35:03.835 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:35:03.835 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:35:03.835 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:35:03.835 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.835 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:35:03.836 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.734348+0000", 2024-08-19T14:35:03.836 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:35:03.836 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.836 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.836 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:35:03.836 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.734348+0000", 2024-08-19T14:35:03.836 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:35:03.836 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.836 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.836 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:35:03.836 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.734349+0000", 2024-08-19T14:35:03.836 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.6199999999999996e-07 2024-08-19T14:35:03.836 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.836 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.837 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:35:03.837 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.734410+0000", 2024-08-19T14:35:03.837 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.1273999999999996e-05 2024-08-19T14:35:03.837 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.837 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.837 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:35:03.837 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.734420+0000", 2024-08-19T14:35:03.837 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.027e-05 2024-08-19T14:35:03.837 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.837 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.837 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:35:03.837 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.734427+0000", 2024-08-19T14:35:03.837 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.6279999999999999e-06 2024-08-19T14:35:03.837 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.838 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.838 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:35:03.838 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.734476+0000", 2024-08-19T14:35:03.838 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.9348999999999998e-05 2024-08-19T14:35:03.838 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.838 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.838 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:35:03.838 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.734499+0000", 2024-08-19T14:35:03.838 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2558e-05 2024-08-19T14:35:03.838 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.838 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.839 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:35:03.839 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.795828+0000", 2024-08-19T14:35:03.839 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.061329231999999997 2024-08-19T14:35:03.839 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.839 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.839 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:35:03.839 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.795892+0000", 2024-08-19T14:35:03.839 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.3511999999999998e-05 2024-08-19T14:35:03.839 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:35:03.839 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:35:03.839 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:35:03.839 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.839 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.839 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.16962.0:106086 7.5 e627/617 7:aecea1dc:::benchmark_data_smithi045_115842_object106085:head v 627'6561, mlcod=608'1460)", 2024-08-19T14:35:03.840 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:34:48.734484+0000", 2024-08-19T14:35:03.840 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 15.0105001, 2024-08-19T14:35:03.840 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.058300642999999999, 2024-08-19T14:35:03.840 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:35:03.840 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:35:03.840 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:35:03.840 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:35:03.840 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.840 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:35:03.840 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.734484+0000", 2024-08-19T14:35:03.840 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:35:03.840 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.840 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.841 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:35:03.841 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.734484+0000", 2024-08-19T14:35:03.841 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:35:03.841 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.841 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.841 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:35:03.841 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.734485+0000", 2024-08-19T14:35:03.841 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.5999999999999999e-07 2024-08-19T14:35:03.841 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.841 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.841 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:35:03.841 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.734507+0000", 2024-08-19T14:35:03.841 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1950000000000002e-05 2024-08-19T14:35:03.841 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.842 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.842 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:35:03.842 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.734515+0000", 2024-08-19T14:35:03.842 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.405e-06 2024-08-19T14:35:03.842 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.842 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.842 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:35:03.842 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.734520+0000", 2024-08-19T14:35:03.842 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.3449999999999996e-06 2024-08-19T14:35:03.842 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.842 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.842 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:35:03.842 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.777620+0000", 2024-08-19T14:35:03.843 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.043100029999999998 2024-08-19T14:35:03.843 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.843 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.843 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:35:03.843 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.777648+0000", 2024-08-19T14:35:03.843 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.7144e-05 2024-08-19T14:35:03.843 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.843 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.843 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:35:03.843 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.792746+0000", 2024-08-19T14:35:03.843 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.015098741000000001 2024-08-19T14:35:03.843 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:03.843 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:03.843 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:35:03.844 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.792785+0000", 2024-08-19T14:35:03.844 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.8568000000000001e-05 2024-08-19T14:35:03.844 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:35:03.844 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:35:03.844 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:35:03.844 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:35:03.844 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:35:03.844 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:35:03.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:03.846 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:03.846 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:35:03.947 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:35:03.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:03.970 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:03.970 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:35:04.071 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:35:04.094 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:04.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:04.095 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:35:04.196 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:35:04.216 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:04.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:04.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:35:04.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:03 smithi032 ceph-mon[20004]: pgmap v1485: 47 pgs: 4 active+clean+remapped, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized, 5 active+undersized+degraded, 29 active+clean; 7.0 GiB data, 29 GiB used, 865 GiB / 894 GiB avail; 48 MiB/s wr, 770 op/s; 43642/345747 objects degraded (12.623%); 15647/345747 objects misplaced (4.526%); 15 MiB/s, 235 objects/s recovering 2024-08-19T14:35:04.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:03 smithi032 ceph-mon[20004]: osdmap e632: 12 total, 11 up, 10 in 2024-08-19T14:35:04.274 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:35:04.275 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:35:04.275 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:35:04.275 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:35:04.275 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:35:04.275 INFO:teuthology.orchestra.run.smithi031.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-08-19T14:35:04.275 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:35:04.275 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:35:04.276 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:35:04.276 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:35:04.276 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:35:04.276 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:35:04.276 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:35:04.276 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:35:04.276 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:35:04.276 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:35:04.277 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:35:04.277 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:35:04.277 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:35:04.277 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:35:04.277 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:35:04.277 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:35:04.277 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:35:04.277 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:03 smithi031 ceph-mon[16336]: pgmap v1485: 47 pgs: 4 active+clean+remapped, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized, 5 active+undersized+degraded, 29 active+clean; 7.0 GiB data, 29 GiB used, 865 GiB / 894 GiB avail; 48 MiB/s wr, 770 op/s; 43642/345747 objects degraded (12.623%); 15647/345747 objects misplaced (4.526%); 15 MiB/s, 235 objects/s recovering 2024-08-19T14:35:04.277 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:03 smithi031 ceph-mon[16336]: osdmap e632: 12 total, 11 up, 10 in 2024-08-19T14:35:04.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:03 smithi045 ceph-mon[20143]: pgmap v1485: 47 pgs: 4 active+clean+remapped, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized, 5 active+undersized+degraded, 29 active+clean; 7.0 GiB data, 29 GiB used, 865 GiB / 894 GiB avail; 48 MiB/s wr, 770 op/s; 43642/345747 objects degraded (12.623%); 15647/345747 objects misplaced (4.526%); 15 MiB/s, 235 objects/s recovering 2024-08-19T14:35:04.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:03 smithi045 ceph-mon[20143]: osdmap e632: 12 total, 11 up, 10 in 2024-08-19T14:35:04.318 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:35:04.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:04.346 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:04.346 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:35:04.447 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:35:04.448 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Total time run: 90.0429 2024-08-19T14:35:04.449 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Total writes made: 118157 2024-08-19T14:35:04.449 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Write size: 65536 2024-08-19T14:35:04.449 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Object size: 65536 2024-08-19T14:35:04.449 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Bandwidth (MB/sec): 82.0143 2024-08-19T14:35:04.449 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Stddev Bandwidth: 44.331 2024-08-19T14:35:04.449 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Max bandwidth (MB/sec): 221.312 2024-08-19T14:35:04.449 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Min bandwidth (MB/sec): 0 2024-08-19T14:35:04.449 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Average IOPS: 1312 2024-08-19T14:35:04.449 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Stddev IOPS: 709.295 2024-08-19T14:35:04.449 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Max IOPS: 3541 2024-08-19T14:35:04.449 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Min IOPS: 0 2024-08-19T14:35:04.449 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Average Latency(s): 0.012188 2024-08-19T14:35:04.449 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Stddev Latency(s): 0.0305686 2024-08-19T14:35:04.450 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Max latency(s): 2.26272 2024-08-19T14:35:04.450 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Min latency(s): 0.00106356 2024-08-19T14:35:04.457 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Cleaning up (deleting benchmark objects) 2024-08-19T14:35:04.469 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:04.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:04.470 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:35:04.572 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:35:04.608 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:04.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:04.609 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:35:04.702 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-08-19T14:35:04.709 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:35:04.792 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:04.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:04.793 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:35:04.894 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:35:04.915 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:04.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:04.916 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:35:05.017 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:35:05.038 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:05.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:05.039 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:35:05.075 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:35:05.140 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:35:05.164 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:05.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:05.164 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:35:05.266 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:35:05.311 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:05.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:05.313 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:35:05.413 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:35:05.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:05.447 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:05.447 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:35:05.548 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:35:05.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:05.571 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:05.571 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:35:05.671 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:35:05.671 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [ 2024-08-19T14:35:05.671 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:05.671 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.16962.0:119231 7.a 7:5871dd23:::benchmark_data_smithi045_115842_object1071:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e632)", 2024-08-19T14:35:05.671 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:35:05.657404+0000", 2024-08-19T14:35:05.671 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.012511414, 2024-08-19T14:35:05.671 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.012533282, 2024-08-19T14:35:05.671 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:35:05.671 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:35:05.671 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "waiting for sub ops", 2024-08-19T14:35:05.671 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:35:05.672 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.16962", 2024-08-19T14:35:05.672 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:35:05.672 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 119231 2024-08-19T14:35:05.672 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:05.672 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:35:05.672 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:05.672 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:35:05.672 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:35:05.657404+0000", 2024-08-19T14:35:05.673 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:35:05.673 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:05.673 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:05.673 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:35:05.673 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:35:05.657404+0000", 2024-08-19T14:35:05.673 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:35:05.673 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:05.673 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:05.673 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:35:05.673 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:35:05.657406+0000", 2024-08-19T14:35:05.673 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8700000000000001e-06 2024-08-19T14:35:05.673 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:05.673 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:05.674 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:35:05.674 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:35:05.657413+0000", 2024-08-19T14:35:05.674 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.6030000000000001e-06 2024-08-19T14:35:05.674 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:05.674 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:05.674 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:35:05.674 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:35:05.657426+0000", 2024-08-19T14:35:05.674 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3338e-05 2024-08-19T14:35:05.674 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:05.674 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:05.674 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:35:05.674 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:35:05.657434+0000", 2024-08-19T14:35:05.674 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.0669999999999992e-06 2024-08-19T14:35:05.675 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:05.675 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:05.675 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:35:05.675 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:35:05.657499+0000", 2024-08-19T14:35:05.675 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.4572000000000005e-05 2024-08-19T14:35:05.675 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:05.675 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:05.675 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:35:05.675 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:35:05.657652+0000", 2024-08-19T14:35:05.675 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000152714 2024-08-19T14:35:05.675 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:05.675 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:05.675 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 3,9", 2024-08-19T14:35:05.675 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:35:05.657797+0000", 2024-08-19T14:35:05.676 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0001451 2024-08-19T14:35:05.676 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:05.676 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:05.676 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:35:05.676 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:35:05.669237+0000", 2024-08-19T14:35:05.676 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.011439975 2024-08-19T14:35:05.676 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:35:05.676 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:35:05.676 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:35:05.676 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:05.676 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:05.676 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.16962.0:119239 7.a 7:5952c9c9:::benchmark_data_smithi045_115842_object1079:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e632)", 2024-08-19T14:35:05.676 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:35:05.664825+0000", 2024-08-19T14:35:05.676 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.0050910010000000004, 2024-08-19T14:35:05.677 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0051664240000000002, 2024-08-19T14:35:05.677 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:35:05.677 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:35:05.677 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "waiting for sub ops", 2024-08-19T14:35:05.677 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:35:05.677 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.16962", 2024-08-19T14:35:05.677 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:35:05.677 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 119239 2024-08-19T14:35:05.677 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:05.677 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:35:05.677 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:05.677 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:35:05.677 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:35:05.664825+0000", 2024-08-19T14:35:05.678 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:35:05.678 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:05.678 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:05.678 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:35:05.678 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:35:05.664825+0000", 2024-08-19T14:35:05.678 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:35:05.678 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:05.678 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:05.678 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:35:05.678 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:35:05.664827+0000", 2024-08-19T14:35:05.678 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.779e-06 2024-08-19T14:35:05.679 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:05.679 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:05.679 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:35:05.679 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:35:05.664833+0000", 2024-08-19T14:35:05.679 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.8079999999999999e-06 2024-08-19T14:35:05.679 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:05.679 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:05.679 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:35:05.679 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:35:05.664849+0000", 2024-08-19T14:35:05.679 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5481999999999998e-05 2024-08-19T14:35:05.679 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:05.679 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:05.679 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:35:05.679 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:35:05.664858+0000", 2024-08-19T14:35:05.680 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.7390000000000005e-06 2024-08-19T14:35:05.680 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:05.680 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:05.680 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:35:05.680 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:35:05.665681+0000", 2024-08-19T14:35:05.680 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00082322899999999997 2024-08-19T14:35:05.680 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:05.680 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:05.680 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:35:05.680 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:35:05.665782+0000", 2024-08-19T14:35:05.680 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000100843 2024-08-19T14:35:05.680 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:05.680 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:05.680 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 3,9", 2024-08-19T14:35:05.681 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:35:05.665869+0000", 2024-08-19T14:35:05.681 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.6782999999999999e-05 2024-08-19T14:35:05.681 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:35:05.681 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:35:05.681 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:35:05.681 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:05.681 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:05.681 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDPGPushReply(7.a 632/617 [PushReplyOp(7:5e636c33:::benchmark_data_smithi045_115842_object40657:head)])", 2024-08-19T14:35:05.681 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:35:05.665286+0000", 2024-08-19T14:35:05.681 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.0046302080000000002, 2024-08-19T14:35:05.681 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0047450299999999999, 2024-08-19T14:35:05.681 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:35:05.681 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:35:05.682 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-08-19T14:35:05.682 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:35:05.682 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:05.682 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:35:05.682 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:35:05.665286+0000", 2024-08-19T14:35:05.682 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:35:05.682 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:05.682 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:05.682 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:35:05.683 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:35:05.665286+0000", 2024-08-19T14:35:05.683 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:35:05.683 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:05.683 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:05.683 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:35:05.683 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:35:05.665287+0000", 2024-08-19T14:35:05.683 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.2999999999999999e-07 2024-08-19T14:35:05.683 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:05.683 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:05.683 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:35:05.683 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:35:05.665292+0000", 2024-08-19T14:35:05.683 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.5389999999999998e-06 2024-08-19T14:35:05.683 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:05.683 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:05.684 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:35:05.684 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:35:05.665308+0000", 2024-08-19T14:35:05.684 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5821e-05 2024-08-19T14:35:05.684 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:05.684 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:05.684 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:35:05.684 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:35:05.665316+0000", 2024-08-19T14:35:05.684 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.7659999999999998e-06 2024-08-19T14:35:05.684 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:05.684 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:05.684 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:35:05.684 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:35:05.669269+0000", 2024-08-19T14:35:05.684 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0039532200000000003 2024-08-19T14:35:05.685 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:35:05.685 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:35:05.685 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:35:05.685 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:05.685 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:05.685 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDPGPushReply(7.5 632/617 [PushReplyOp(7:a8c29678:::benchmark_data_smithi045_115842_object38667:head)])", 2024-08-19T14:35:05.685 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:35:05.667275+0000", 2024-08-19T14:35:05.685 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.0026411059999999998, 2024-08-19T14:35:05.685 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0027834399999999999, 2024-08-19T14:35:05.685 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:35:05.685 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:35:05.685 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-08-19T14:35:05.685 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:35:05.685 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:05.686 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:35:05.686 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:35:05.667275+0000", 2024-08-19T14:35:05.686 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:35:05.686 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:05.686 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:05.686 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:35:05.686 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:35:05.667275+0000", 2024-08-19T14:35:05.686 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:35:05.686 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:05.686 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:05.686 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:35:05.686 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:35:05.667276+0000", 2024-08-19T14:35:05.686 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0419999999999999e-06 2024-08-19T14:35:05.687 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:05.687 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:05.687 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:35:05.687 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:35:05.667282+0000", 2024-08-19T14:35:05.687 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.2960000000000004e-06 2024-08-19T14:35:05.687 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:05.687 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:05.687 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:35:05.687 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:35:05.667294+0000", 2024-08-19T14:35:05.687 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1983e-05 2024-08-19T14:35:05.687 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:05.687 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:05.687 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:35:05.687 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:35:05.667302+0000", 2024-08-19T14:35:05.688 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.0539999999999995e-06 2024-08-19T14:35:05.688 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:05.688 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:05.688 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:35:05.688 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:35:05.667371+0000", 2024-08-19T14:35:05.688 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.8497999999999994e-05 2024-08-19T14:35:05.688 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:35:05.688 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:35:05.688 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:35:05.688 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:35:05.688 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-08-19T14:35:05.688 INFO:teuthology.orchestra.run.smithi045.stdout: "num_ops": 4 2024-08-19T14:35:05.688 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:35:05.689 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:35:05.715 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:05.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:05.716 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:35:05.817 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:35:05.839 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:05.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:05.841 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:35:05.942 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:35:05.982 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:05.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:05.983 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:35:06.084 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:35:06.107 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:06.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:06.107 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:35:06.209 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:35:06.231 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:06.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:06.233 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:35:06.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:05 smithi032 ceph-mon[20004]: pgmap v1487: 47 pgs: 4 active+clean+remapped, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized, 5 active+undersized+degraded, 29 active+clean; 7.0 GiB data, 30 GiB used, 864 GiB / 894 GiB avail; 45 MiB/s wr, 725 op/s; 43642/346497 objects degraded (12.595%); 15647/346497 objects misplaced (4.516%); 9.3 MiB/s, 148 objects/s recovering 2024-08-19T14:35:06.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:05 smithi045 ceph-mon[20143]: pgmap v1487: 47 pgs: 4 active+clean+remapped, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized, 5 active+undersized+degraded, 29 active+clean; 7.0 GiB data, 30 GiB used, 864 GiB / 894 GiB avail; 45 MiB/s wr, 725 op/s; 43642/346497 objects degraded (12.595%); 15647/346497 objects misplaced (4.516%); 9.3 MiB/s, 148 objects/s recovering 2024-08-19T14:35:06.334 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:35:06.357 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:06.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:06.357 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:35:06.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:05 smithi031 ceph-mon[16336]: pgmap v1487: 47 pgs: 4 active+clean+remapped, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized, 5 active+undersized+degraded, 29 active+clean; 7.0 GiB data, 30 GiB used, 864 GiB / 894 GiB avail; 45 MiB/s wr, 725 op/s; 43642/346497 objects degraded (12.595%); 15647/346497 objects misplaced (4.516%); 9.3 MiB/s, 148 objects/s recovering 2024-08-19T14:35:06.459 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:35:06.486 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:06.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:06.487 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:35:06.536 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-08-19T14:35:06.588 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:35:06.635 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:06.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:06.637 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:35:06.738 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:35:06.769 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:06.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:06.770 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:35:06.807 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:35:06.871 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:35:06.909 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:06.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:06.910 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:35:06.910 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:35:07.011 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:35:07.060 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:07.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:07.061 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:35:07.162 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:35:07.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:07.262 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:07.262 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:35:07.364 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:35:07.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:07.407 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:07.407 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:35:07.508 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:35:07.518 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:35:07.518 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [], 2024-08-19T14:35:07.518 INFO:teuthology.orchestra.run.smithi045.stdout: "complaint_time": 30, 2024-08-19T14:35:07.518 INFO:teuthology.orchestra.run.smithi045.stdout: "num_blocked_ops": 0 2024-08-19T14:35:07.518 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:35:07.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:07.544 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:07.544 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:35:07.644 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:35:07.674 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:07.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:07.675 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:35:07.776 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:35:07.798 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:07.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:07.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:35:07.900 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:35:07.931 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:07.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:07.932 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:35:08.033 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:35:08.074 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:08.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:08.075 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:35:08.158 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:07 smithi045 ceph-mon[20143]: pgmap v1488: 47 pgs: 4 active+clean+remapped, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized, 5 active+undersized+degraded, 29 active+clean; 7.2 GiB data, 30 GiB used, 864 GiB / 894 GiB avail; 48 MiB/s wr, 825 op/s; 44288/352944 objects degraded (12.548%); 15059/352944 objects misplaced (4.267%); 12 MiB/s, 195 objects/s recovering 2024-08-19T14:35:08.159 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:07 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 44288/352944 objects degraded (12.548%), 7 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-08-19T14:35:08.176 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:35:08.197 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:08.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:08.198 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:35:08.209 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-08-19T14:35:08.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:07 smithi032 ceph-mon[20004]: pgmap v1488: 47 pgs: 4 active+clean+remapped, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized, 5 active+undersized+degraded, 29 active+clean; 7.2 GiB data, 30 GiB used, 864 GiB / 894 GiB avail; 48 MiB/s wr, 825 op/s; 44288/352944 objects degraded (12.548%); 15059/352944 objects misplaced (4.267%); 12 MiB/s, 195 objects/s recovering 2024-08-19T14:35:08.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:07 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 44288/352944 objects degraded (12.548%), 7 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-08-19T14:35:08.300 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:35:08.320 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:08.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:08.321 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:35:08.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:07 smithi031 ceph-mon[16336]: pgmap v1488: 47 pgs: 4 active+clean+remapped, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized, 5 active+undersized+degraded, 29 active+clean; 7.2 GiB data, 30 GiB used, 864 GiB / 894 GiB avail; 48 MiB/s wr, 825 op/s; 44288/352944 objects degraded (12.548%); 15059/352944 objects misplaced (4.267%); 12 MiB/s, 195 objects/s recovering 2024-08-19T14:35:08.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:07 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 44288/352944 objects degraded (12.548%), 7 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-08-19T14:35:08.422 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:35:08.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:08.460 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:08.460 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:35:08.561 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:35:08.582 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:08.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:08.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:35:08.594 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:35:08.684 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:35:08.706 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:08.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:08.707 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:35:08.807 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:35:08.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:08.838 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:08.838 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:35:08.866 INFO:journalctl@ceph.osd.2.smithi031.stdout:Aug 19 14:35:08 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-2[181347]: 2024-08-19T14:35:08.506+0000 7fcb824a2640 -1 osd.2 564 log_to_monitors true 2024-08-19T14:35:08.939 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:35:08.965 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:08.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:08.966 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:35:09.022 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:08 smithi045 ceph-mon[20143]: from='osd.2 [v2:172.21.15.31:6818/656997856,v1:172.21.15.31:6819/656997856]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-08-19T14:35:09.022 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:08 smithi045 ceph-mon[20143]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-08-19T14:35:09.066 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:35:09.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:09.096 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:09.096 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:35:09.173 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:35:09.173 INFO:teuthology.orchestra.run.smithi045.stdout: "size": 20, 2024-08-19T14:35:09.173 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 600, 2024-08-19T14:35:09.173 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [ 2024-08-19T14:35:09.173 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.173 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25813.0:115483 5.8 5:118541ec:::benchmark_data_smithi045_85309_object115482:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e455)", 2024-08-19T14:35:09.173 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:26:12.254294+0000", 2024-08-19T14:35:09.173 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 536.91565024299996, 2024-08-19T14:35:09.174 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.41025914400000002, 2024-08-19T14:35:09.174 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:35:09.174 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:35:09.174 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:35:09.174 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:35:09.174 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25813", 2024-08-19T14:35:09.174 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:35:09.174 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 115483 2024-08-19T14:35:09.174 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.174 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:35:09.174 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.174 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:35:09.174 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.254294+0000", 2024-08-19T14:35:09.175 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:35:09.175 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.175 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.175 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:35:09.175 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.254294+0000", 2024-08-19T14:35:09.175 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:35:09.175 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.175 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.175 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:35:09.175 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.254295+0000", 2024-08-19T14:35:09.175 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0049999999999999e-06 2024-08-19T14:35:09.175 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.175 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.176 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:35:09.176 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.254351+0000", 2024-08-19T14:35:09.176 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.6385999999999999e-05 2024-08-19T14:35:09.176 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.176 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.176 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:35:09.176 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.254365+0000", 2024-08-19T14:35:09.176 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4425e-05 2024-08-19T14:35:09.176 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.176 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.176 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:35:09.176 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.254373+0000", 2024-08-19T14:35:09.176 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.1620000000000002e-06 2024-08-19T14:35:09.176 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.177 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.177 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:35:09.177 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.254438+0000", 2024-08-19T14:35:09.177 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.5295000000000004e-05 2024-08-19T14:35:09.177 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.177 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.177 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:35:09.177 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.254556+0000", 2024-08-19T14:35:09.177 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000117555 2024-08-19T14:35:09.177 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.177 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.177 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,9", 2024-08-19T14:35:09.177 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.254698+0000", 2024-08-19T14:35:09.178 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000142003 2024-08-19T14:35:09.178 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.178 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.178 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:35:09.178 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.256435+0000", 2024-08-19T14:35:09.178 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0017378999999999999 2024-08-19T14:35:09.178 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.178 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.178 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:35:09.178 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.265174+0000", 2024-08-19T14:35:09.178 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0087388980000000002 2024-08-19T14:35:09.178 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.178 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.179 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:35:09.179 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.664449+0000", 2024-08-19T14:35:09.179 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.39927473800000002 2024-08-19T14:35:09.179 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.179 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.179 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:35:09.179 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.664502+0000", 2024-08-19T14:35:09.179 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.3066000000000003e-05 2024-08-19T14:35:09.179 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.179 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.179 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:35:09.179 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.664553+0000", 2024-08-19T14:35:09.179 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0711000000000003e-05 2024-08-19T14:35:09.180 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:35:09.180 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:35:09.180 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:35:09.180 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.180 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.180 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25813.0:115542 5.8 5:126f9ca1:::benchmark_data_smithi045_85309_object115541:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e455)", 2024-08-19T14:35:09.180 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:26:12.310153+0000", 2024-08-19T14:35:09.180 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 536.85979059500005, 2024-08-19T14:35:09.180 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.35453124699999999, 2024-08-19T14:35:09.180 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:35:09.180 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:35:09.180 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:35:09.180 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:35:09.181 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25813", 2024-08-19T14:35:09.181 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:35:09.181 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 115542 2024-08-19T14:35:09.181 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.181 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:35:09.181 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.181 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:35:09.181 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.310153+0000", 2024-08-19T14:35:09.181 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:35:09.181 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.181 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.181 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:35:09.181 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.310153+0000", 2024-08-19T14:35:09.181 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:35:09.182 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.182 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.182 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:35:09.182 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.310155+0000", 2024-08-19T14:35:09.182 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.279e-06 2024-08-19T14:35:09.182 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.182 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.182 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:35:09.182 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.310186+0000", 2024-08-19T14:35:09.182 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.1291999999999998e-05 2024-08-19T14:35:09.182 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.182 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.182 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:35:09.183 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.310199+0000", 2024-08-19T14:35:09.183 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2824e-05 2024-08-19T14:35:09.183 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.183 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.183 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:35:09.183 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.310206+0000", 2024-08-19T14:35:09.183 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.3189999999999997e-06 2024-08-19T14:35:09.183 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.183 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.183 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:35:09.183 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.311624+0000", 2024-08-19T14:35:09.183 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0014176740000000001 2024-08-19T14:35:09.183 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.183 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.184 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:35:09.184 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.311764+0000", 2024-08-19T14:35:09.184 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014065900000000001 2024-08-19T14:35:09.184 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.184 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.184 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,9", 2024-08-19T14:35:09.184 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.311923+0000", 2024-08-19T14:35:09.184 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015905800000000001 2024-08-19T14:35:09.184 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.184 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.184 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:35:09.184 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.313965+0000", 2024-08-19T14:35:09.184 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0020413430000000002 2024-08-19T14:35:09.184 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.185 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.185 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:35:09.185 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.320836+0000", 2024-08-19T14:35:09.185 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0068708529999999997 2024-08-19T14:35:09.185 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.185 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.185 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:35:09.185 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.664598+0000", 2024-08-19T14:35:09.185 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.34376222299999998 2024-08-19T14:35:09.185 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.185 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.185 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:35:09.185 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.664648+0000", 2024-08-19T14:35:09.186 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0489999999999999e-05 2024-08-19T14:35:09.186 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.186 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.186 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:35:09.186 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.664685+0000", 2024-08-19T14:35:09.186 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.6232999999999997e-05 2024-08-19T14:35:09.186 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:35:09.186 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:35:09.186 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:35:09.186 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.186 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.186 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25813.0:115641 5.8 5:1358c592:::benchmark_data_smithi045_85309_object115640:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e455)", 2024-08-19T14:35:09.186 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:26:12.409980+0000", 2024-08-19T14:35:09.186 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 536.75996355899997, 2024-08-19T14:35:09.187 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.25595528499999998, 2024-08-19T14:35:09.187 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:35:09.187 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:35:09.187 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:35:09.187 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:35:09.187 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25813", 2024-08-19T14:35:09.187 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:35:09.187 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 115641 2024-08-19T14:35:09.187 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.187 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:35:09.187 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.187 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:35:09.187 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.409980+0000", 2024-08-19T14:35:09.188 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:35:09.188 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.188 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.188 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:35:09.188 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.409980+0000", 2024-08-19T14:35:09.188 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:35:09.188 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.188 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.188 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:35:09.188 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.409982+0000", 2024-08-19T14:35:09.188 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2950000000000001e-06 2024-08-19T14:35:09.188 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.188 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.189 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:35:09.189 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.410007+0000", 2024-08-19T14:35:09.189 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.5198000000000002e-05 2024-08-19T14:35:09.189 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.189 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.189 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:35:09.189 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.410022+0000", 2024-08-19T14:35:09.189 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4678000000000001e-05 2024-08-19T14:35:09.189 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.189 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.189 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:35:09.189 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.410029+0000", 2024-08-19T14:35:09.189 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4590000000000001e-06 2024-08-19T14:35:09.189 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.190 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.190 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:35:09.190 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.410088+0000", 2024-08-19T14:35:09.190 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.9169999999999998e-05 2024-08-19T14:35:09.190 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.190 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.190 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:35:09.190 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.410232+0000", 2024-08-19T14:35:09.190 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014414699999999999 2024-08-19T14:35:09.190 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.190 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.190 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,9", 2024-08-19T14:35:09.190 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.410377+0000", 2024-08-19T14:35:09.191 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000144566 2024-08-19T14:35:09.191 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.191 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.191 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:35:09.191 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.412287+0000", 2024-08-19T14:35:09.191 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001909628 2024-08-19T14:35:09.191 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.191 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.191 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:35:09.191 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.449686+0000", 2024-08-19T14:35:09.191 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.037399549999999997 2024-08-19T14:35:09.191 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.191 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.191 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:35:09.192 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.665809+0000", 2024-08-19T14:35:09.192 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.21612313899999999 2024-08-19T14:35:09.192 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.192 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.192 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:35:09.192 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.665895+0000", 2024-08-19T14:35:09.192 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.5433999999999996e-05 2024-08-19T14:35:09.192 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.192 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.192 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:35:09.192 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.665936+0000", 2024-08-19T14:35:09.192 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.1020999999999999e-05 2024-08-19T14:35:09.192 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:35:09.192 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:35:09.193 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:35:09.193 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.193 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.193 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25813.0:115648 5.8 5:15d8ce6a:::benchmark_data_smithi045_85309_object115647:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e455)", 2024-08-19T14:35:09.193 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:26:12.416281+0000", 2024-08-19T14:35:09.193 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 536.75366324699996, 2024-08-19T14:35:09.193 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.24979046499999999, 2024-08-19T14:35:09.193 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:35:09.193 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:35:09.193 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:35:09.193 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:35:09.193 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25813", 2024-08-19T14:35:09.193 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:35:09.193 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 115648 2024-08-19T14:35:09.194 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.194 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:35:09.194 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.194 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:35:09.194 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.416281+0000", 2024-08-19T14:35:09.194 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:35:09.194 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.194 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.194 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:35:09.194 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.416281+0000", 2024-08-19T14:35:09.194 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:35:09.194 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.194 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.194 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:35:09.195 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.416282+0000", 2024-08-19T14:35:09.195 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5740000000000001e-06 2024-08-19T14:35:09.195 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.195 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.195 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:35:09.195 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.416304+0000", 2024-08-19T14:35:09.195 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1698000000000001e-05 2024-08-19T14:35:09.195 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.195 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.195 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:35:09.195 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.416317+0000", 2024-08-19T14:35:09.195 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3084e-05 2024-08-19T14:35:09.195 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.195 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.196 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:35:09.196 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.416324+0000", 2024-08-19T14:35:09.196 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.0570000000000004e-06 2024-08-19T14:35:09.196 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.196 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.196 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:35:09.196 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.416383+0000", 2024-08-19T14:35:09.196 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.9261000000000001e-05 2024-08-19T14:35:09.196 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.196 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.196 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:35:09.196 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.416500+0000", 2024-08-19T14:35:09.196 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000116523 2024-08-19T14:35:09.197 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.197 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.197 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,9", 2024-08-19T14:35:09.197 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.416646+0000", 2024-08-19T14:35:09.197 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014645500000000001 2024-08-19T14:35:09.197 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.197 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.197 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:35:09.197 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.418689+0000", 2024-08-19T14:35:09.197 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0020425019999999999 2024-08-19T14:35:09.197 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.197 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.197 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:35:09.197 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.449715+0000", 2024-08-19T14:35:09.197 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.031026194 2024-08-19T14:35:09.198 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.198 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.198 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:35:09.198 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.665979+0000", 2024-08-19T14:35:09.198 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.21626399099999999 2024-08-19T14:35:09.198 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.198 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.198 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:35:09.198 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.666032+0000", 2024-08-19T14:35:09.198 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.3140000000000003e-05 2024-08-19T14:35:09.198 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.198 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.198 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:35:09.199 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:26:12.666071+0000", 2024-08-19T14:35:09.199 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.8986000000000002e-05 2024-08-19T14:35:09.199 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:35:09.199 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:35:09.199 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:35:09.199 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.199 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.199 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25813.0:180218 5.d 5:bd4207f5:::benchmark_data_smithi045_85309_object59965:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e475)", 2024-08-19T14:35:09.199 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:27:11.940920+0000", 2024-08-19T14:35:09.199 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 477.229024025, 2024-08-19T14:35:09.199 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.23667333700000001, 2024-08-19T14:35:09.199 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:35:09.199 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:35:09.199 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:35:09.200 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:35:09.200 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25813", 2024-08-19T14:35:09.200 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:35:09.200 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 180218 2024-08-19T14:35:09.200 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.200 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:35:09.200 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.200 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:35:09.200 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:11.940920+0000", 2024-08-19T14:35:09.200 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:35:09.200 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.200 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.200 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:35:09.200 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:11.940920+0000", 2024-08-19T14:35:09.201 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:35:09.201 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.201 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.201 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:35:09.202 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:35:09.202 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:11.940921+0000", 2024-08-19T14:35:09.202 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.561e-06 2024-08-19T14:35:09.202 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.202 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.202 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:35:09.202 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:11.940927+0000", 2024-08-19T14:35:09.202 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7300000000000002e-06 2024-08-19T14:35:09.203 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.203 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.203 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:35:09.203 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:11.940938+0000", 2024-08-19T14:35:09.203 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0958000000000001e-05 2024-08-19T14:35:09.203 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.203 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.203 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:35:09.203 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:11.940945+0000", 2024-08-19T14:35:09.203 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.6579999999999996e-06 2024-08-19T14:35:09.203 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.203 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.203 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:35:09.203 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.031816+0000", 2024-08-19T14:35:09.204 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.090871065000000001 2024-08-19T14:35:09.204 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.204 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.204 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:35:09.204 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.031914+0000", 2024-08-19T14:35:09.204 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.7686e-05 2024-08-19T14:35:09.204 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.204 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.204 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 5,7", 2024-08-19T14:35:09.204 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.032000+0000", 2024-08-19T14:35:09.204 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.6169999999999997e-05 2024-08-19T14:35:09.204 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.204 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.205 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:35:09.205 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.045113+0000", 2024-08-19T14:35:09.205 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.013113028000000001 2024-08-19T14:35:09.205 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.205 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.205 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:35:09.205 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.047147+0000", 2024-08-19T14:35:09.205 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0020338050000000001 2024-08-19T14:35:09.205 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.205 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.205 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:35:09.205 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.177520+0000", 2024-08-19T14:35:09.205 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.130373085 2024-08-19T14:35:09.205 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.206 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.206 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:35:09.206 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.177556+0000", 2024-08-19T14:35:09.206 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.6403999999999999e-05 2024-08-19T14:35:09.206 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.206 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.206 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:35:09.206 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.177593+0000", 2024-08-19T14:35:09.206 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.7187000000000002e-05 2024-08-19T14:35:09.206 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:35:09.206 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:35:09.206 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:35:09.206 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.207 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.207 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25813.0:180682 5.8 5:1602eddf:::benchmark_data_smithi045_85309_object60429:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e476)", 2024-08-19T14:35:09.207 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:27:12.846408+0000", 2024-08-19T14:35:09.207 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 476.323535779, 2024-08-19T14:35:09.207 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.27902613199999998, 2024-08-19T14:35:09.207 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:35:09.207 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:35:09.207 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:35:09.207 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:35:09.207 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25813", 2024-08-19T14:35:09.207 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:35:09.207 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 180682 2024-08-19T14:35:09.207 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.207 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:35:09.208 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.208 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:35:09.208 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.846408+0000", 2024-08-19T14:35:09.208 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:35:09.208 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.208 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.208 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:35:09.208 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.846408+0000", 2024-08-19T14:35:09.208 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:35:09.208 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.208 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.208 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:35:09.208 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.846410+0000", 2024-08-19T14:35:09.208 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9980000000000002e-06 2024-08-19T14:35:09.209 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.209 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.209 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:35:09.209 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.846415+0000", 2024-08-19T14:35:09.209 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1120000000000004e-06 2024-08-19T14:35:09.209 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.209 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.209 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:35:09.209 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.846431+0000", 2024-08-19T14:35:09.209 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5786000000000001e-05 2024-08-19T14:35:09.209 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.209 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.209 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:35:09.209 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.846438+0000", 2024-08-19T14:35:09.209 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.3699999999999997e-06 2024-08-19T14:35:09.210 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.210 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.210 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:35:09.210 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.915048+0000", 2024-08-19T14:35:09.210 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.068609724999999996 2024-08-19T14:35:09.210 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.210 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.210 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:35:09.210 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.915174+0000", 2024-08-19T14:35:09.210 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012629899999999999 2024-08-19T14:35:09.210 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.210 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.210 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,9", 2024-08-19T14:35:09.211 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.915288+0000", 2024-08-19T14:35:09.211 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000113544 2024-08-19T14:35:09.211 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.211 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.211 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:35:09.211 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.921139+0000", 2024-08-19T14:35:09.211 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0058512420000000004 2024-08-19T14:35:09.211 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.211 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.211 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:35:09.211 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.923745+0000", 2024-08-19T14:35:09.211 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0026053589999999998 2024-08-19T14:35:09.211 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.211 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.212 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:35:09.212 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:13.125373+0000", 2024-08-19T14:35:09.212 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.201628589 2024-08-19T14:35:09.212 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.212 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.212 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:35:09.212 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:13.125405+0000", 2024-08-19T14:35:09.212 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.2224999999999997e-05 2024-08-19T14:35:09.212 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.212 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.212 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:35:09.212 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:13.125434+0000", 2024-08-19T14:35:09.212 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.8883000000000001e-05 2024-08-19T14:35:09.212 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:35:09.213 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:35:09.213 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:35:09.213 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.213 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.213 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25813.0:180713 5.8 5:13e650ea:::benchmark_data_smithi045_85309_object60460:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e476)", 2024-08-19T14:35:09.213 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:27:12.880045+0000", 2024-08-19T14:35:09.213 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 476.28989884700002, 2024-08-19T14:35:09.213 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.24953703699999999, 2024-08-19T14:35:09.213 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:35:09.213 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:35:09.213 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:35:09.213 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:35:09.213 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25813", 2024-08-19T14:35:09.213 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:35:09.214 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 180713 2024-08-19T14:35:09.214 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.214 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:35:09.214 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.214 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:35:09.214 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.880045+0000", 2024-08-19T14:35:09.214 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:35:09.214 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.214 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.214 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:35:09.214 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.880045+0000", 2024-08-19T14:35:09.214 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:35:09.214 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.214 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.215 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:35:09.215 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.880047+0000", 2024-08-19T14:35:09.215 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5510000000000001e-06 2024-08-19T14:35:09.215 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.215 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.215 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:35:09.215 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.880052+0000", 2024-08-19T14:35:09.215 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.5709999999999997e-06 2024-08-19T14:35:09.215 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.215 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.215 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:35:09.215 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.880068+0000", 2024-08-19T14:35:09.215 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6087e-05 2024-08-19T14:35:09.218 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.218 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.218 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:35:09.219 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.880076+0000", 2024-08-19T14:35:09.219 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4039999999999998e-06 2024-08-19T14:35:09.219 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.219 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.219 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:35:09.219 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.918429+0000", 2024-08-19T14:35:09.219 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.038353230000000002 2024-08-19T14:35:09.219 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.219 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.219 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:35:09.219 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.918538+0000", 2024-08-19T14:35:09.219 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00010878999999999999 2024-08-19T14:35:09.219 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.220 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.220 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,9", 2024-08-19T14:35:09.220 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.918613+0000", 2024-08-19T14:35:09.220 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.5405000000000001e-05 2024-08-19T14:35:09.220 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.220 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.220 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:35:09.220 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.923754+0000", 2024-08-19T14:35:09.220 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0051403999999999998 2024-08-19T14:35:09.220 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.220 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.220 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:35:09.220 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.924797+0000", 2024-08-19T14:35:09.220 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0010431150000000001 2024-08-19T14:35:09.221 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.221 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.221 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:35:09.221 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:13.129461+0000", 2024-08-19T14:35:09.221 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20466435799999999 2024-08-19T14:35:09.221 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.221 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.221 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:35:09.221 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:13.129522+0000", 2024-08-19T14:35:09.221 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.1054000000000002e-05 2024-08-19T14:35:09.221 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.221 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.221 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:35:09.221 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:13.129582+0000", 2024-08-19T14:35:09.221 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.0071999999999997e-05 2024-08-19T14:35:09.222 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:35:09.222 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:35:09.222 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:35:09.222 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.222 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.222 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25813.0:180719 5.8 5:15c77316:::benchmark_data_smithi045_85309_object60466:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e476)", 2024-08-19T14:35:09.222 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:27:12.886662+0000", 2024-08-19T14:35:09.222 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 476.283282343, 2024-08-19T14:35:09.222 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.24306710300000001, 2024-08-19T14:35:09.222 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:35:09.222 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:35:09.222 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:35:09.222 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:35:09.223 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25813", 2024-08-19T14:35:09.223 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:35:09.223 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 180719 2024-08-19T14:35:09.223 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.223 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:35:09.223 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.223 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:35:09.223 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.886662+0000", 2024-08-19T14:35:09.223 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:35:09.223 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.223 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.223 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:35:09.223 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.886662+0000", 2024-08-19T14:35:09.223 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:35:09.224 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.224 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.224 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:35:09.224 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.886663+0000", 2024-08-19T14:35:09.224 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.184e-06 2024-08-19T14:35:09.224 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.224 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.224 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:35:09.224 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.886667+0000", 2024-08-19T14:35:09.224 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.907e-06 2024-08-19T14:35:09.224 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.224 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.224 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:35:09.224 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.886679+0000", 2024-08-19T14:35:09.225 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1915e-05 2024-08-19T14:35:09.225 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.225 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.225 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:35:09.225 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.886686+0000", 2024-08-19T14:35:09.225 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.2049999999999996e-06 2024-08-19T14:35:09.225 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.225 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.225 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:35:09.225 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.921183+0000", 2024-08-19T14:35:09.225 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.034497556999999998 2024-08-19T14:35:09.225 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.225 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.225 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:35:09.226 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.921260+0000", 2024-08-19T14:35:09.226 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.6388000000000001e-05 2024-08-19T14:35:09.226 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.226 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.226 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,9", 2024-08-19T14:35:09.226 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.921322+0000", 2024-08-19T14:35:09.226 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.2611000000000001e-05 2024-08-19T14:35:09.226 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.226 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.226 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:35:09.226 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.923834+0000", 2024-08-19T14:35:09.226 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.002511924 2024-08-19T14:35:09.226 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.226 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.227 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:35:09.227 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:12.924841+0000", 2024-08-19T14:35:09.227 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0010065129999999999 2024-08-19T14:35:09.227 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.227 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.227 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:35:09.227 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:13.129629+0000", 2024-08-19T14:35:09.227 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20478792500000001 2024-08-19T14:35:09.227 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.227 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.227 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:35:09.227 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:13.129683+0000", 2024-08-19T14:35:09.227 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4293999999999999e-05 2024-08-19T14:35:09.227 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.228 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.228 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:35:09.228 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:27:13.129729+0000", 2024-08-19T14:35:09.228 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.5680000000000003e-05 2024-08-19T14:35:09.228 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:35:09.228 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:35:09.228 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:35:09.228 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.228 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.228 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25813.0:237770 5.8 5:11831c02:::benchmark_data_smithi045_85309_object117517:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e495)", 2024-08-19T14:35:09.228 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:28:07.106685+0000", 2024-08-19T14:35:09.228 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 422.06325925499999, 2024-08-19T14:35:09.228 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.22493542599999999, 2024-08-19T14:35:09.228 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:35:09.229 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:35:09.229 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:35:09.229 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:35:09.229 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25813", 2024-08-19T14:35:09.229 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:35:09.229 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 237770 2024-08-19T14:35:09.229 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.229 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:35:09.229 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.229 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:35:09.229 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:28:07.106685+0000", 2024-08-19T14:35:09.229 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:35:09.229 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.229 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.229 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:35:09.230 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:28:07.106685+0000", 2024-08-19T14:35:09.230 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:35:09.230 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.230 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.230 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:35:09.230 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:28:07.106685+0000", 2024-08-19T14:35:09.230 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.6199999999999998e-07 2024-08-19T14:35:09.230 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.230 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.230 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:35:09.230 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:28:07.106688+0000", 2024-08-19T14:35:09.230 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.0510000000000001e-06 2024-08-19T14:35:09.230 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.231 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.231 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:35:09.231 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:28:07.106696+0000", 2024-08-19T14:35:09.231 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.9899999999999997e-06 2024-08-19T14:35:09.231 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.231 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.231 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:35:09.231 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:28:07.106702+0000", 2024-08-19T14:35:09.231 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7810000000000002e-06 2024-08-19T14:35:09.231 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.231 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.231 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:35:09.231 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:28:07.106846+0000", 2024-08-19T14:35:09.231 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014427600000000001 2024-08-19T14:35:09.231 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.232 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.232 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for peered", 2024-08-19T14:35:09.232 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:28:07.106854+0000", 2024-08-19T14:35:09.232 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.3309999999999998e-06 2024-08-19T14:35:09.232 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.232 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.232 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:35:09.232 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:28:07.317267+0000", 2024-08-19T14:35:09.232 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.21041380400000001 2024-08-19T14:35:09.232 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.232 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.232 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:35:09.232 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:28:07.317422+0000", 2024-08-19T14:35:09.232 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015415200000000001 2024-08-19T14:35:09.233 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.233 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.233 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,5", 2024-08-19T14:35:09.233 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:28:07.318149+0000", 2024-08-19T14:35:09.233 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00072762499999999997 2024-08-19T14:35:09.233 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.233 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.233 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:35:09.233 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:28:07.322380+0000", 2024-08-19T14:35:09.233 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0042309590000000003 2024-08-19T14:35:09.233 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.233 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.233 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:35:09.234 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:28:07.323678+0000", 2024-08-19T14:35:09.234 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001298366 2024-08-19T14:35:09.234 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.234 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.234 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:35:09.234 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:28:07.331373+0000", 2024-08-19T14:35:09.234 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0076946790000000003 2024-08-19T14:35:09.234 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.234 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.234 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:35:09.234 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:28:07.331429+0000", 2024-08-19T14:35:09.234 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.5517999999999999e-05 2024-08-19T14:35:09.234 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.234 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.235 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:35:09.235 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:28:07.331620+0000", 2024-08-19T14:35:09.235 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00019143200000000001 2024-08-19T14:35:09.235 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:35:09.235 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:35:09.235 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:35:09.235 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.235 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.235 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.16962.0:88508 7.a 7:5eb9d75f:::benchmark_data_smithi045_115842_object88507:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e622)", 2024-08-19T14:35:09.235 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:34:27.884193+0000", 2024-08-19T14:35:09.235 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 41.285751310999999, 2024-08-19T14:35:09.235 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.15672350099999999, 2024-08-19T14:35:09.235 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:35:09.235 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:35:09.236 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:35:09.236 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:35:09.236 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.16962", 2024-08-19T14:35:09.236 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:35:09.236 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 88508 2024-08-19T14:35:09.236 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.236 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:35:09.236 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.236 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:35:09.236 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:27.884193+0000", 2024-08-19T14:35:09.236 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:35:09.236 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.236 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.236 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:35:09.236 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:27.884193+0000", 2024-08-19T14:35:09.237 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:35:09.237 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.237 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.237 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:35:09.237 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:27.884194+0000", 2024-08-19T14:35:09.237 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.668e-06 2024-08-19T14:35:09.237 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.237 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.237 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:35:09.237 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:27.884246+0000", 2024-08-19T14:35:09.237 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1771999999999998e-05 2024-08-19T14:35:09.237 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.237 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.237 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:35:09.238 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:27.884259+0000", 2024-08-19T14:35:09.238 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2626000000000001e-05 2024-08-19T14:35:09.238 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.238 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.238 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:35:09.238 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:27.884266+0000", 2024-08-19T14:35:09.238 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4150000000000002e-06 2024-08-19T14:35:09.238 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.238 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.238 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:35:09.238 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:27.884314+0000", 2024-08-19T14:35:09.238 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.7416999999999997e-05 2024-08-19T14:35:09.238 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.238 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.239 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:35:09.239 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:27.884437+0000", 2024-08-19T14:35:09.239 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012354799999999999 2024-08-19T14:35:09.239 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.239 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.239 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 3,9", 2024-08-19T14:35:09.239 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:27.884553+0000", 2024-08-19T14:35:09.239 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000116338 2024-08-19T14:35:09.239 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.239 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.239 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:35:09.239 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:27.887748+0000", 2024-08-19T14:35:09.239 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0031948369999999999 2024-08-19T14:35:09.239 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.240 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.240 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:35:09.240 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:27.897914+0000", 2024-08-19T14:35:09.240 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.010166244 2024-08-19T14:35:09.240 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.240 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.240 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:35:09.240 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:28.039560+0000", 2024-08-19T14:35:09.240 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.14164560400000001 2024-08-19T14:35:09.240 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.240 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.240 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:35:09.240 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:28.039603+0000", 2024-08-19T14:35:09.240 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.2672000000000001e-05 2024-08-19T14:35:09.241 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.241 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.241 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:35:09.241 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:28.040916+0000", 2024-08-19T14:35:09.241 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00131336 2024-08-19T14:35:09.241 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:35:09.241 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:35:09.241 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:35:09.241 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.241 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.241 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.16962.0:101868 7.5 7:a9b55934:::benchmark_data_smithi045_115842_object101867:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e626)", 2024-08-19T14:35:09.241 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:34:44.024260+0000", 2024-08-19T14:35:09.241 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 25.145684292999999, 2024-08-19T14:35:09.241 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.22134078099999999, 2024-08-19T14:35:09.242 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:35:09.242 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:35:09.242 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:35:09.242 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:35:09.242 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.16962", 2024-08-19T14:35:09.242 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:35:09.242 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 101868 2024-08-19T14:35:09.242 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.242 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:35:09.242 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.242 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:35:09.242 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.024260+0000", 2024-08-19T14:35:09.242 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:35:09.242 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.243 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.243 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:35:09.243 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.024260+0000", 2024-08-19T14:35:09.243 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:35:09.243 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.243 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.243 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:35:09.243 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.024261+0000", 2024-08-19T14:35:09.243 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.75e-06 2024-08-19T14:35:09.243 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.243 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.243 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:35:09.243 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.024286+0000", 2024-08-19T14:35:09.243 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.5000000000000001e-05 2024-08-19T14:35:09.244 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.244 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.244 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:35:09.244 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.024300+0000", 2024-08-19T14:35:09.244 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3848e-05 2024-08-19T14:35:09.244 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.244 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.244 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:35:09.244 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.024308+0000", 2024-08-19T14:35:09.244 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.0800000000000006e-06 2024-08-19T14:35:09.244 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.244 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.244 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:35:09.244 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.028092+0000", 2024-08-19T14:35:09.245 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0037835730000000001 2024-08-19T14:35:09.245 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.245 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.245 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:35:09.245 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.028226+0000", 2024-08-19T14:35:09.245 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013406799999999999 2024-08-19T14:35:09.245 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.245 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.245 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,9", 2024-08-19T14:35:09.245 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.028410+0000", 2024-08-19T14:35:09.245 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00018418199999999999 2024-08-19T14:35:09.245 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.245 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.245 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:35:09.247 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.042589+0000", 2024-08-19T14:35:09.247 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.014178619 2024-08-19T14:35:09.247 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.247 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.247 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:35:09.247 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.042636+0000", 2024-08-19T14:35:09.247 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.7123999999999998e-05 2024-08-19T14:35:09.248 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.248 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.248 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:35:09.248 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.243010+0000", 2024-08-19T14:35:09.248 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.200373724 2024-08-19T14:35:09.248 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.248 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.248 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:35:09.248 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.243072+0000", 2024-08-19T14:35:09.248 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.2238999999999998e-05 2024-08-19T14:35:09.248 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.248 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.248 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:35:09.248 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.245600+0000", 2024-08-19T14:35:09.249 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.002528574 2024-08-19T14:35:09.249 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:35:09.249 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:35:09.249 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:35:09.249 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.249 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.249 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.16962.0:101874 7.5 7:aba2a64c:::benchmark_data_smithi045_115842_object101873:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e626)", 2024-08-19T14:35:09.249 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:34:44.030050+0000", 2024-08-19T14:35:09.249 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 25.139894399999999, 2024-08-19T14:35:09.249 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.23052292099999999, 2024-08-19T14:35:09.249 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:35:09.249 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:35:09.249 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:35:09.250 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:35:09.250 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.16962", 2024-08-19T14:35:09.250 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:35:09.250 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 101874 2024-08-19T14:35:09.250 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.250 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:35:09.250 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.250 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:35:09.250 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.030050+0000", 2024-08-19T14:35:09.250 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:35:09.250 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.250 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.250 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:35:09.250 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.030050+0000", 2024-08-19T14:35:09.251 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:35:09.251 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.251 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.251 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:35:09.251 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.030051+0000", 2024-08-19T14:35:09.251 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9099999999999999e-06 2024-08-19T14:35:09.251 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.251 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.251 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:35:09.251 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.030092+0000", 2024-08-19T14:35:09.251 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.1050999999999997e-05 2024-08-19T14:35:09.251 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.251 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.251 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:35:09.252 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.030113+0000", 2024-08-19T14:35:09.252 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0358999999999999e-05 2024-08-19T14:35:09.252 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.252 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.252 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:35:09.252 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.030123+0000", 2024-08-19T14:35:09.252 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0172e-05 2024-08-19T14:35:09.252 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.252 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.252 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:35:09.252 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.038789+0000", 2024-08-19T14:35:09.252 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0086658689999999997 2024-08-19T14:35:09.252 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.252 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.253 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:35:09.253 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.038947+0000", 2024-08-19T14:35:09.253 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015787700000000001 2024-08-19T14:35:09.253 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.253 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.253 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,9", 2024-08-19T14:35:09.253 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.039142+0000", 2024-08-19T14:35:09.253 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00019552400000000001 2024-08-19T14:35:09.253 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.253 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.253 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:35:09.253 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.044827+0000", 2024-08-19T14:35:09.253 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0056849190000000001 2024-08-19T14:35:09.253 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.254 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.254 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:35:09.254 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.054485+0000", 2024-08-19T14:35:09.254 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0096582260000000007 2024-08-19T14:35:09.254 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.254 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.254 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:35:09.254 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.257671+0000", 2024-08-19T14:35:09.254 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20318518099999999 2024-08-19T14:35:09.254 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.254 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.254 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:35:09.254 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.257731+0000", 2024-08-19T14:35:09.255 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.0825000000000001e-05 2024-08-19T14:35:09.255 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.255 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.255 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:35:09.255 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.260572+0000", 2024-08-19T14:35:09.255 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0028410079999999999 2024-08-19T14:35:09.255 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:35:09.255 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:35:09.255 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:35:09.256 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.256 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.256 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.16962.0:101888 7.5 7:a3e28936:::benchmark_data_smithi045_115842_object101887:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e626)", 2024-08-19T14:35:09.256 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:34:44.046545+0000", 2024-08-19T14:35:09.256 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 25.123398702999999, 2024-08-19T14:35:09.256 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.221824155, 2024-08-19T14:35:09.256 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:35:09.256 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:35:09.256 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:35:09.256 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:35:09.256 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.16962", 2024-08-19T14:35:09.256 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:35:09.256 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 101888 2024-08-19T14:35:09.256 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.257 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:35:09.257 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.257 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:35:09.257 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.046545+0000", 2024-08-19T14:35:09.257 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:35:09.257 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.257 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.257 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:35:09.257 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.046545+0000", 2024-08-19T14:35:09.257 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:35:09.257 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.257 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.257 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:35:09.257 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.046547+0000", 2024-08-19T14:35:09.257 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8780000000000001e-06 2024-08-19T14:35:09.258 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.258 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.258 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:35:09.258 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.046570+0000", 2024-08-19T14:35:09.258 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2459999999999998e-05 2024-08-19T14:35:09.258 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.258 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.258 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:35:09.258 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.046583+0000", 2024-08-19T14:35:09.258 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3886000000000001e-05 2024-08-19T14:35:09.258 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.258 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.258 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:35:09.259 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.046591+0000", 2024-08-19T14:35:09.259 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.6450000000000005e-06 2024-08-19T14:35:09.259 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.259 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.259 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:35:09.259 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.046648+0000", 2024-08-19T14:35:09.259 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.6694999999999998e-05 2024-08-19T14:35:09.259 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.259 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.259 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:35:09.259 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.046816+0000", 2024-08-19T14:35:09.259 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00016854899999999999 2024-08-19T14:35:09.259 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.259 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.260 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,9", 2024-08-19T14:35:09.260 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.046999+0000", 2024-08-19T14:35:09.260 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00018276100000000001 2024-08-19T14:35:09.260 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.260 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.260 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:35:09.260 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.054535+0000", 2024-08-19T14:35:09.260 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0075362509999999999 2024-08-19T14:35:09.260 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.260 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.260 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:35:09.260 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.054545+0000", 2024-08-19T14:35:09.260 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0084999999999999e-05 2024-08-19T14:35:09.260 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.261 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.261 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:35:09.261 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.266112+0000", 2024-08-19T14:35:09.261 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.211566105 2024-08-19T14:35:09.261 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.261 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.261 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:35:09.261 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.266179+0000", 2024-08-19T14:35:09.261 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.7485999999999997e-05 2024-08-19T14:35:09.261 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.262 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.262 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:35:09.262 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.268369+0000", 2024-08-19T14:35:09.262 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0021903539999999998 2024-08-19T14:35:09.262 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:35:09.262 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:35:09.262 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:35:09.262 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.262 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.262 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.16962.0:101894 7.5 7:a67a0241:::benchmark_data_smithi045_115842_object101893:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e626)", 2024-08-19T14:35:09.262 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:34:44.050292+0000", 2024-08-19T14:35:09.262 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 25.119651976, 2024-08-19T14:35:09.262 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.22549364399999999, 2024-08-19T14:35:09.263 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:35:09.263 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:35:09.263 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:35:09.263 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:35:09.263 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.16962", 2024-08-19T14:35:09.263 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:35:09.263 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 101894 2024-08-19T14:35:09.263 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.263 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:35:09.263 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.263 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:35:09.263 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.050292+0000", 2024-08-19T14:35:09.263 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:35:09.263 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.264 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.264 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:35:09.264 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.050292+0000", 2024-08-19T14:35:09.264 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:35:09.264 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.264 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.264 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:35:09.264 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.050293+0000", 2024-08-19T14:35:09.264 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.226e-06 2024-08-19T14:35:09.264 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.264 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.264 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:35:09.264 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.050314+0000", 2024-08-19T14:35:09.264 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1043000000000002e-05 2024-08-19T14:35:09.265 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.265 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.265 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:35:09.265 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.050326+0000", 2024-08-19T14:35:09.265 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1999e-05 2024-08-19T14:35:09.265 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.265 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.265 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:35:09.265 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.050332+0000", 2024-08-19T14:35:09.265 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.818e-06 2024-08-19T14:35:09.265 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.265 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.265 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:35:09.265 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.054572+0000", 2024-08-19T14:35:09.265 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0042403340000000001 2024-08-19T14:35:09.266 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.266 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.266 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:35:09.266 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.054692+0000", 2024-08-19T14:35:09.266 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000119597 2024-08-19T14:35:09.266 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.266 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.266 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,9", 2024-08-19T14:35:09.266 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.054807+0000", 2024-08-19T14:35:09.266 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00011525499999999999 2024-08-19T14:35:09.266 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.266 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.266 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:35:09.267 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.060833+0000", 2024-08-19T14:35:09.267 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0060257929999999998 2024-08-19T14:35:09.267 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.267 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.267 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:35:09.267 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.067717+0000", 2024-08-19T14:35:09.267 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0068838500000000004 2024-08-19T14:35:09.267 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.267 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.267 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:35:09.267 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.273501+0000", 2024-08-19T14:35:09.267 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.205783779 2024-08-19T14:35:09.267 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.267 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.268 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:35:09.268 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.273561+0000", 2024-08-19T14:35:09.268 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.0613999999999999e-05 2024-08-19T14:35:09.268 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.268 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.268 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:35:09.268 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.275786+0000", 2024-08-19T14:35:09.268 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0022243359999999999 2024-08-19T14:35:09.268 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:35:09.268 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:35:09.268 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:35:09.268 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.268 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.268 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.16962.0:101896 7.5 7:a83fdd11:::benchmark_data_smithi045_115842_object101895:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e626)", 2024-08-19T14:35:09.268 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:34:44.050859+0000", 2024-08-19T14:35:09.269 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 25.119084542, 2024-08-19T14:35:09.269 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.22712142900000001, 2024-08-19T14:35:09.269 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:35:09.269 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:35:09.269 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:35:09.269 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:35:09.269 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.16962", 2024-08-19T14:35:09.269 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:35:09.269 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 101896 2024-08-19T14:35:09.269 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.269 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:35:09.269 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.269 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:35:09.269 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.050859+0000", 2024-08-19T14:35:09.270 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:35:09.270 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.270 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.270 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:35:09.270 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.050859+0000", 2024-08-19T14:35:09.270 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:35:09.270 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.270 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.270 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:35:09.270 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.050860+0000", 2024-08-19T14:35:09.270 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.5900000000000005e-07 2024-08-19T14:35:09.270 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.270 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.271 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:35:09.271 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.050895+0000", 2024-08-19T14:35:09.271 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.4251000000000001e-05 2024-08-19T14:35:09.271 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.271 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.271 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:35:09.271 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.050904+0000", 2024-08-19T14:35:09.271 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.8779999999999993e-06 2024-08-19T14:35:09.271 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.271 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.271 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:35:09.271 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.050910+0000", 2024-08-19T14:35:09.271 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7370000000000003e-06 2024-08-19T14:35:09.271 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.272 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.272 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:35:09.272 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.057488+0000", 2024-08-19T14:35:09.272 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0065775440000000003 2024-08-19T14:35:09.272 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.272 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.272 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:35:09.272 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.057650+0000", 2024-08-19T14:35:09.272 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00016202 2024-08-19T14:35:09.272 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.273 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.273 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,9", 2024-08-19T14:35:09.273 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.057823+0000", 2024-08-19T14:35:09.273 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000173501 2024-08-19T14:35:09.273 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.273 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.273 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:35:09.273 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.067740+0000", 2024-08-19T14:35:09.273 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0099169500000000008 2024-08-19T14:35:09.273 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.273 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.273 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:35:09.273 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.067799+0000", 2024-08-19T14:35:09.273 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.9108999999999999e-05 2024-08-19T14:35:09.274 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.274 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.274 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:35:09.274 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.275858+0000", 2024-08-19T14:35:09.274 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20805822500000001 2024-08-19T14:35:09.274 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.274 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.274 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:35:09.274 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.275910+0000", 2024-08-19T14:35:09.274 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2253999999999998e-05 2024-08-19T14:35:09.274 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.274 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.274 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:35:09.274 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.277981+0000", 2024-08-19T14:35:09.275 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0020710009999999998 2024-08-19T14:35:09.275 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:35:09.275 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:35:09.275 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:35:09.275 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.275 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.275 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.16962.0:101904 7.5 7:ad2dd20f:::benchmark_data_smithi045_115842_object101903:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e626)", 2024-08-19T14:35:09.275 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:34:44.062127+0000", 2024-08-19T14:35:09.275 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 25.107817315999998, 2024-08-19T14:35:09.275 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.21785035, 2024-08-19T14:35:09.275 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:35:09.275 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:35:09.276 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:35:09.276 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:35:09.276 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.16962", 2024-08-19T14:35:09.276 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:35:09.276 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 101904 2024-08-19T14:35:09.276 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.276 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:35:09.276 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.276 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:35:09.276 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.062127+0000", 2024-08-19T14:35:09.276 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:35:09.276 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.276 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.276 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:35:09.277 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.062127+0000", 2024-08-19T14:35:09.277 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:35:09.277 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.277 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.277 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:35:09.277 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.062128+0000", 2024-08-19T14:35:09.277 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8780000000000001e-06 2024-08-19T14:35:09.277 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.277 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.277 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:35:09.277 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.062156+0000", 2024-08-19T14:35:09.277 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.7317e-05 2024-08-19T14:35:09.277 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.277 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.277 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:35:09.278 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.062170+0000", 2024-08-19T14:35:09.278 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4059e-05 2024-08-19T14:35:09.278 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.278 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.278 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:35:09.278 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.062177+0000", 2024-08-19T14:35:09.278 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.0360000000000001e-06 2024-08-19T14:35:09.278 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.278 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.278 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:35:09.278 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.067875+0000", 2024-08-19T14:35:09.278 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0056985459999999996 2024-08-19T14:35:09.278 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.279 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.279 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:35:09.279 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.068063+0000", 2024-08-19T14:35:09.279 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00018802399999999999 2024-08-19T14:35:09.279 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.279 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.279 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,9", 2024-08-19T14:35:09.279 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.068227+0000", 2024-08-19T14:35:09.279 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00016347900000000001 2024-08-19T14:35:09.279 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.279 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.279 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:35:09.279 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.075326+0000", 2024-08-19T14:35:09.279 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0070986160000000003 2024-08-19T14:35:09.280 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.280 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.280 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:35:09.280 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.092809+0000", 2024-08-19T14:35:09.280 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.017483207000000001 2024-08-19T14:35:09.280 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.280 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.280 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:35:09.280 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.278038+0000", 2024-08-19T14:35:09.280 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.185228804 2024-08-19T14:35:09.280 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.280 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.280 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:35:09.280 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.278077+0000", 2024-08-19T14:35:09.281 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.8949000000000003e-05 2024-08-19T14:35:09.281 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.281 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.281 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:35:09.281 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.279977+0000", 2024-08-19T14:35:09.281 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001900435 2024-08-19T14:35:09.281 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:35:09.281 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:35:09.281 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:35:09.281 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.281 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.281 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.16962.0:101907 7.5 7:a0598159:::benchmark_data_smithi045_115842_object101906:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e626)", 2024-08-19T14:35:09.281 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:34:44.064648+0000", 2024-08-19T14:35:09.281 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 25.105296319000001, 2024-08-19T14:35:09.282 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.217895121, 2024-08-19T14:35:09.282 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:35:09.282 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:35:09.282 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:35:09.282 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:35:09.282 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.16962", 2024-08-19T14:35:09.282 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:35:09.282 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 101907 2024-08-19T14:35:09.282 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.282 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:35:09.282 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.282 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:35:09.282 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.064648+0000", 2024-08-19T14:35:09.283 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:35:09.283 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.283 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.283 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:35:09.283 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.064648+0000", 2024-08-19T14:35:09.283 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:35:09.283 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.283 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.283 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:35:09.283 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.064649+0000", 2024-08-19T14:35:09.283 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5400000000000001e-06 2024-08-19T14:35:09.283 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.283 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.283 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:35:09.284 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.064676+0000", 2024-08-19T14:35:09.284 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.6812999999999999e-05 2024-08-19T14:35:09.284 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.284 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.284 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:35:09.284 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.064691+0000", 2024-08-19T14:35:09.284 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4763000000000001e-05 2024-08-19T14:35:09.284 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.284 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.284 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:35:09.284 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.064701+0000", 2024-08-19T14:35:09.284 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.9930000000000007e-06 2024-08-19T14:35:09.284 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.284 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.285 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:35:09.285 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.071510+0000", 2024-08-19T14:35:09.285 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0068096909999999997 2024-08-19T14:35:09.285 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.285 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.285 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:35:09.285 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.071680+0000", 2024-08-19T14:35:09.285 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00016987400000000001 2024-08-19T14:35:09.285 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.285 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.285 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,9", 2024-08-19T14:35:09.285 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.071848+0000", 2024-08-19T14:35:09.285 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00016773299999999999 2024-08-19T14:35:09.285 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.286 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.286 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:35:09.286 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.075335+0000", 2024-08-19T14:35:09.286 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.003486617 2024-08-19T14:35:09.286 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.286 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.286 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:35:09.286 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.092867+0000", 2024-08-19T14:35:09.286 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.017532749 2024-08-19T14:35:09.286 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.286 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.286 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:35:09.286 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.280052+0000", 2024-08-19T14:35:09.287 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.18718484899999999 2024-08-19T14:35:09.287 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.287 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.287 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:35:09.287 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.280118+0000", 2024-08-19T14:35:09.287 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.6191000000000004e-05 2024-08-19T14:35:09.287 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.287 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.287 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:35:09.287 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.282543+0000", 2024-08-19T14:35:09.287 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.002424308 2024-08-19T14:35:09.287 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:35:09.287 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:35:09.287 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:35:09.288 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.288 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.288 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.16962.0:101915 7.5 7:a194bcf3:::benchmark_data_smithi045_115842_object101914:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e626)", 2024-08-19T14:35:09.288 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:34:44.075596+0000", 2024-08-19T14:35:09.288 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 25.094347462999998, 2024-08-19T14:35:09.288 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.210082677, 2024-08-19T14:35:09.288 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:35:09.288 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:35:09.288 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:35:09.288 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:35:09.289 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.16962", 2024-08-19T14:35:09.289 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:35:09.289 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 101915 2024-08-19T14:35:09.289 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.289 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:35:09.289 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.289 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:35:09.289 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.075596+0000", 2024-08-19T14:35:09.289 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:35:09.289 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.289 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.289 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:35:09.289 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.075596+0000", 2024-08-19T14:35:09.289 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:35:09.290 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.290 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.290 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:35:09.290 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.075598+0000", 2024-08-19T14:35:09.290 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3400000000000001e-06 2024-08-19T14:35:09.290 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.290 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.290 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:35:09.290 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.075620+0000", 2024-08-19T14:35:09.290 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2259000000000002e-05 2024-08-19T14:35:09.290 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.290 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.290 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:35:09.291 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.075634+0000", 2024-08-19T14:35:09.291 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3961000000000001e-05 2024-08-19T14:35:09.291 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.291 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.291 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:35:09.291 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.075642+0000", 2024-08-19T14:35:09.291 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.2830000000000001e-06 2024-08-19T14:35:09.291 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.291 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.291 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:35:09.291 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.075703+0000", 2024-08-19T14:35:09.291 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.0565e-05 2024-08-19T14:35:09.291 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.291 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.292 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:35:09.292 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.075888+0000", 2024-08-19T14:35:09.292 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000185138 2024-08-19T14:35:09.292 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.292 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.292 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,9", 2024-08-19T14:35:09.292 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.076067+0000", 2024-08-19T14:35:09.292 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00017943000000000001 2024-08-19T14:35:09.292 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.292 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.292 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:35:09.292 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.079463+0000", 2024-08-19T14:35:09.292 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0033953920000000001 2024-08-19T14:35:09.293 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.293 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.293 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:35:09.293 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.092925+0000", 2024-08-19T14:35:09.293 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.013462399999999999 2024-08-19T14:35:09.293 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.293 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.293 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:35:09.293 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.282636+0000", 2024-08-19T14:35:09.293 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.18971106800000001 2024-08-19T14:35:09.293 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.293 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.293 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:35:09.293 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.282704+0000", 2024-08-19T14:35:09.294 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.7532000000000003e-05 2024-08-19T14:35:09.294 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.294 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.294 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:35:09.294 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.285679+0000", 2024-08-19T14:35:09.294 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0029753090000000002 2024-08-19T14:35:09.294 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:35:09.294 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:35:09.294 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:35:09.294 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.294 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.294 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.16962.0:121586 7.a 7:5f3db445:::benchmark_data_smithi045_115842_object3426:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e632)", 2024-08-19T14:35:09.294 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:35:08.828925+0000", 2024-08-19T14:35:09.295 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.341018724, 2024-08-19T14:35:09.295 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.155947213, 2024-08-19T14:35:09.295 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:35:09.295 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:35:09.295 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:35:09.295 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:35:09.295 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.16962", 2024-08-19T14:35:09.295 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:35:09.295 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 121586 2024-08-19T14:35:09.295 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.295 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:35:09.295 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.295 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:35:09.296 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:35:08.828925+0000", 2024-08-19T14:35:09.296 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:35:09.296 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.296 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.296 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:35:09.296 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:35:08.828925+0000", 2024-08-19T14:35:09.296 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:35:09.296 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.296 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.296 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:35:09.296 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:35:08.828926+0000", 2024-08-19T14:35:09.296 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2979999999999999e-06 2024-08-19T14:35:09.296 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.296 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.297 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:35:09.297 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:35:08.828932+0000", 2024-08-19T14:35:09.297 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4399999999999996e-06 2024-08-19T14:35:09.297 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.297 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.297 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:35:09.297 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:35:08.828947+0000", 2024-08-19T14:35:09.297 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5075000000000001e-05 2024-08-19T14:35:09.297 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.297 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.297 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:35:09.297 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:35:08.828955+0000", 2024-08-19T14:35:09.297 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.7479999999999993e-06 2024-08-19T14:35:09.297 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.298 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.298 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:35:09.298 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:35:08.837408+0000", 2024-08-19T14:35:09.298 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0084528169999999996 2024-08-19T14:35:09.298 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.298 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.298 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:35:09.298 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:35:08.837555+0000", 2024-08-19T14:35:09.298 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014721800000000001 2024-08-19T14:35:09.298 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.298 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.298 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 3,9", 2024-08-19T14:35:09.298 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:35:08.837735+0000", 2024-08-19T14:35:09.298 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000180075 2024-08-19T14:35:09.299 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.299 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.299 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:35:09.299 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:35:08.842973+0000", 2024-08-19T14:35:09.299 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0052380070000000003 2024-08-19T14:35:09.299 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.299 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.299 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:35:09.299 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:35:08.852450+0000", 2024-08-19T14:35:09.299 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0094771839999999996 2024-08-19T14:35:09.299 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.299 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.299 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:35:09.299 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:35:08.983752+0000", 2024-08-19T14:35:09.300 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.13130224700000001 2024-08-19T14:35:09.300 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.300 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.300 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:35:09.300 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:35:08.983794+0000", 2024-08-19T14:35:09.300 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.1545000000000003e-05 2024-08-19T14:35:09.300 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.300 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.300 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:35:09.300 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:35:08.984872+0000", 2024-08-19T14:35:09.300 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001078559 2024-08-19T14:35:09.300 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:35:09.300 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:35:09.300 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:35:09.301 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.301 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.301 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.16962.0:121598 7.a 7:5a367264:::benchmark_data_smithi045_115842_object3438:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e632)", 2024-08-19T14:35:09.301 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:35:08.841422+0000", 2024-08-19T14:35:09.301 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.32852216200000001, 2024-08-19T14:35:09.301 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.16171247799999999, 2024-08-19T14:35:09.301 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:35:09.301 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:35:09.301 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:35:09.301 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:35:09.301 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.16962", 2024-08-19T14:35:09.301 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:35:09.301 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 121598 2024-08-19T14:35:09.301 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.301 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:35:09.302 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.302 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:35:09.302 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:35:08.841422+0000", 2024-08-19T14:35:09.302 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:35:09.302 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.302 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.302 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:35:09.302 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:35:08.841422+0000", 2024-08-19T14:35:09.302 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:35:09.302 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.302 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.302 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:35:09.302 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:35:08.841424+0000", 2024-08-19T14:35:09.303 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8899999999999999e-06 2024-08-19T14:35:09.303 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.303 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.303 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:35:09.303 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:35:08.841430+0000", 2024-08-19T14:35:09.303 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.5910000000000001e-06 2024-08-19T14:35:09.303 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.303 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.303 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:35:09.303 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:35:08.841445+0000", 2024-08-19T14:35:09.303 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5047e-05 2024-08-19T14:35:09.303 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.303 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.303 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:35:09.304 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:35:08.841454+0000", 2024-08-19T14:35:09.304 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.3599999999999996e-06 2024-08-19T14:35:09.304 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.304 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.304 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:35:09.304 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:35:08.843006+0000", 2024-08-19T14:35:09.304 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001552587 2024-08-19T14:35:09.304 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.304 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.304 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:35:09.304 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:35:08.843162+0000", 2024-08-19T14:35:09.304 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000155532 2024-08-19T14:35:09.304 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.304 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.305 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 3,9", 2024-08-19T14:35:09.305 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:35:08.843310+0000", 2024-08-19T14:35:09.305 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014815 2024-08-19T14:35:09.305 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.305 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.305 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:35:09.305 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:35:08.852395+0000", 2024-08-19T14:35:09.305 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0090852659999999998 2024-08-19T14:35:09.305 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.305 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.305 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:35:09.305 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:35:08.852499+0000", 2024-08-19T14:35:09.305 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000104097 2024-08-19T14:35:09.305 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.306 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.306 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:35:09.306 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:35:09.001947+0000", 2024-08-19T14:35:09.306 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.149447358 2024-08-19T14:35:09.306 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.306 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.306 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:35:09.306 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:35:09.002012+0000", 2024-08-19T14:35:09.306 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.5358999999999995e-05 2024-08-19T14:35:09.306 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:35:09.306 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:35:09.306 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:35:09.306 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:35:09.003134+0000", 2024-08-19T14:35:09.307 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001122241 2024-08-19T14:35:09.307 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:35:09.307 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:35:09.307 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:35:09.307 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:35:09.307 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:35:09.307 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:35:09.308 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:08 smithi032 ceph-mon[20004]: from='osd.2 [v2:172.21.15.31:6818/656997856,v1:172.21.15.31:6819/656997856]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-08-19T14:35:09.308 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:08 smithi032 ceph-mon[20004]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-08-19T14:35:09.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:09.309 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:09.309 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:35:09.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:08 smithi031 ceph-mon[16336]: from='osd.2 [v2:172.21.15.31:6818/656997856,v1:172.21.15.31:6819/656997856]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-08-19T14:35:09.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:08 smithi031 ceph-mon[16336]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-08-19T14:35:09.410 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:35:09.454 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:09.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:09.455 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:35:09.556 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:35:09.611 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:09.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:09.612 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:35:09.689 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-19T14:35:09.689 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-19T14:35:09.713 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:35:09.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:09.772 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:09.772 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:35:09.866 INFO:journalctl@ceph.osd.2.smithi031.stdout:Aug 19 14:35:09 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-2[181347]: 2024-08-19T14:35:09.587+0000 7fcb76245640 -1 osd.2 633 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-19T14:35:09.873 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:35:09.894 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:09.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:09.895 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:35:09.996 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:35:10.017 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:10.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:10.019 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:35:10.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:09 smithi045 ceph-mon[20143]: pgmap v1489: 47 pgs: 4 active+clean+remapped, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized, 5 active+undersized+degraded, 29 active+clean; 7.2 GiB data, 30 GiB used, 864 GiB / 894 GiB avail; 39 MiB/s wr, 690 op/s; 44260/352854 objects degraded (12.543%); 14329/352854 objects misplaced (4.061%); 15 MiB/s, 243 objects/s recovering 2024-08-19T14:35:10.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:09 smithi045 ceph-mon[20143]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-08-19T14:35:10.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:09 smithi045 ceph-mon[20143]: from='osd.2 [v2:172.21.15.31:6818/656997856,v1:172.21.15.31:6819/656997856]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-08-19T14:35:10.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:09 smithi045 ceph-mon[20143]: osdmap e633: 12 total, 11 up, 10 in 2024-08-19T14:35:10.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:09 smithi045 ceph-mon[20143]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-08-19T14:35:10.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:09 smithi045 ceph-mon[20143]: from='osd.2 ' entity='osd.2' 2024-08-19T14:35:10.119 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:35:10.154 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:10.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:10.155 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:35:10.215 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:09 smithi031 ceph-mon[16336]: pgmap v1489: 47 pgs: 4 active+clean+remapped, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized, 5 active+undersized+degraded, 29 active+clean; 7.2 GiB data, 30 GiB used, 864 GiB / 894 GiB avail; 39 MiB/s wr, 690 op/s; 44260/352854 objects degraded (12.543%); 14329/352854 objects misplaced (4.061%); 15 MiB/s, 243 objects/s recovering 2024-08-19T14:35:10.215 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:09 smithi031 ceph-mon[16336]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-08-19T14:35:10.216 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:09 smithi031 ceph-mon[16336]: from='osd.2 [v2:172.21.15.31:6818/656997856,v1:172.21.15.31:6819/656997856]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-08-19T14:35:10.216 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:09 smithi031 ceph-mon[16336]: osdmap e633: 12 total, 11 up, 10 in 2024-08-19T14:35:10.216 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:09 smithi031 ceph-mon[16336]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-08-19T14:35:10.216 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:09 smithi031 ceph-mon[16336]: from='osd.2 ' entity='osd.2' 2024-08-19T14:35:10.255 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:35:10.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:09 smithi032 ceph-mon[20004]: pgmap v1489: 47 pgs: 4 active+clean+remapped, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized, 5 active+undersized+degraded, 29 active+clean; 7.2 GiB data, 30 GiB used, 864 GiB / 894 GiB avail; 39 MiB/s wr, 690 op/s; 44260/352854 objects degraded (12.543%); 14329/352854 objects misplaced (4.061%); 15 MiB/s, 243 objects/s recovering 2024-08-19T14:35:10.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:09 smithi032 ceph-mon[20004]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-08-19T14:35:10.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:09 smithi032 ceph-mon[20004]: from='osd.2 [v2:172.21.15.31:6818/656997856,v1:172.21.15.31:6819/656997856]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-08-19T14:35:10.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:09 smithi032 ceph-mon[20004]: osdmap e633: 12 total, 11 up, 10 in 2024-08-19T14:35:10.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:09 smithi032 ceph-mon[20004]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-08-19T14:35:10.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:09 smithi032 ceph-mon[20004]: from='osd.2 ' entity='osd.2' 2024-08-19T14:35:10.286 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:35:10.288 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:10.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:10.290 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:35:10.391 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:35:10.431 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-08-19T14:35:10.441 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:10.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:10.442 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:35:10.543 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:35:10.564 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:10.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:10.565 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:35:10.666 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:35:10.690 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:10.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:10.692 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:35:10.781 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:35:10.792 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:35:10.816 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:10.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:10.817 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:35:10.856 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:35:10.857 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-08-19T14:35:10.857 INFO:teuthology.orchestra.run.smithi031.stdout: "num_ops": 0 2024-08-19T14:35:10.857 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:35:10.917 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:35:10.940 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:10.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:10.941 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:35:11.042 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:35:11.066 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:11.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:11.068 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:35:11.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:10 smithi031 ceph-mon[16336]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-08-19T14:35:11.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:10 smithi031 ceph-mon[16336]: osd.2 [v2:172.21.15.31:6818/656997856,v1:172.21.15.31:6819/656997856] boot 2024-08-19T14:35:11.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:10 smithi031 ceph-mon[16336]: osdmap e634: 12 total, 12 up, 10 in 2024-08-19T14:35:11.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:10 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-08-19T14:35:11.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:10 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3478354108' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:35:11.169 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:35:11.191 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:11.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:11.192 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:35:11.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:10 smithi032 ceph-mon[20004]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-08-19T14:35:11.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:10 smithi032 ceph-mon[20004]: osd.2 [v2:172.21.15.31:6818/656997856,v1:172.21.15.31:6819/656997856] boot 2024-08-19T14:35:11.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:10 smithi032 ceph-mon[20004]: osdmap e634: 12 total, 12 up, 10 in 2024-08-19T14:35:11.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:10 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-08-19T14:35:11.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:10 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3478354108' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:35:11.292 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:35:11.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:10 smithi045 ceph-mon[20143]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-08-19T14:35:11.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:10 smithi045 ceph-mon[20143]: osd.2 [v2:172.21.15.31:6818/656997856,v1:172.21.15.31:6819/656997856] boot 2024-08-19T14:35:11.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:10 smithi045 ceph-mon[20143]: osdmap e634: 12 total, 12 up, 10 in 2024-08-19T14:35:11.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:10 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-08-19T14:35:11.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:10 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3478354108' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:35:11.311 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:35:11.311 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [], 2024-08-19T14:35:11.311 INFO:teuthology.orchestra.run.smithi045.stdout: "num_ops": 0 2024-08-19T14:35:11.311 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:35:11.314 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:11.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:11.315 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:35:11.416 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:35:11.439 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:11.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:11.441 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:35:11.522 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is set 2024-08-19T14:35:11.541 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:35:11.567 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:11.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:11.569 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:35:11.670 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:35:11.690 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:35:11.693 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:11.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:11.694 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:35:11.795 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:35:11.819 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:11.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:11.820 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:35:11.921 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:35:11.948 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:11.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:11.948 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:35:12.050 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:35:12.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:12.076 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:12.077 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:35:12.178 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:35:12.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:11 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:35:12.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:11 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:35:12.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:11 smithi031 ceph-mon[16336]: pgmap v1491: 47 pgs: 4 active+clean+remapped, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized, 5 active+undersized+degraded, 29 active+clean; 7.1 GiB data, 29 GiB used, 865 GiB / 894 GiB avail; 127 B/s rd, 23 MiB/s wr, 658 op/s; 44036/347409 objects degraded (12.676%); 14024/347409 objects misplaced (4.037%); 13 MiB/s, 202 objects/s recovering 2024-08-19T14:35:12.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:11 smithi031 ceph-mon[16336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:35:12.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:11 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3478354108' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:35:12.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:11 smithi031 ceph-mon[16336]: osdmap e635: 12 total, 12 up, 10 in 2024-08-19T14:35:12.200 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:12.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:12.201 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:35:12.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:11 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:35:12.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:11 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:35:12.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:11 smithi032 ceph-mon[20004]: pgmap v1491: 47 pgs: 4 active+clean+remapped, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized, 5 active+undersized+degraded, 29 active+clean; 7.1 GiB data, 29 GiB used, 865 GiB / 894 GiB avail; 127 B/s rd, 23 MiB/s wr, 658 op/s; 44036/347409 objects degraded (12.676%); 14024/347409 objects misplaced (4.037%); 13 MiB/s, 202 objects/s recovering 2024-08-19T14:35:12.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:11 smithi032 ceph-mon[20004]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:35:12.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:11 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3478354108' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:35:12.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:11 smithi032 ceph-mon[20004]: osdmap e635: 12 total, 12 up, 10 in 2024-08-19T14:35:12.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:11 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:35:12.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:11 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:35:12.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:11 smithi045 ceph-mon[20143]: pgmap v1491: 47 pgs: 4 active+clean+remapped, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized, 5 active+undersized+degraded, 29 active+clean; 7.1 GiB data, 29 GiB used, 865 GiB / 894 GiB avail; 127 B/s rd, 23 MiB/s wr, 658 op/s; 44036/347409 objects degraded (12.676%); 14024/347409 objects misplaced (4.037%); 13 MiB/s, 202 objects/s recovering 2024-08-19T14:35:12.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:11 smithi045 ceph-mon[20143]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:35:12.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:11 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3478354108' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:35:12.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:11 smithi045 ceph-mon[20143]: osdmap e635: 12 total, 12 up, 10 in 2024-08-19T14:35:12.301 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:35:12.328 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:12.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:12.329 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:35:12.380 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-08-19T14:35:12.430 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:35:12.455 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:12.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:12.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:35:12.557 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:35:12.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:12.593 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:12.593 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:35:12.682 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:35:12.694 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:35:12.722 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:12.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:12.723 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:35:12.824 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:35:12.858 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:12.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:12.859 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:35:12.960 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:35:12.996 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:12.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:12.998 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:35:13.099 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:35:13.124 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:13.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:13.125 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:35:13.226 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:35:13.253 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:35:13.253 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [], 2024-08-19T14:35:13.253 INFO:teuthology.orchestra.run.smithi045.stdout: "complaint_time": 30, 2024-08-19T14:35:13.254 INFO:teuthology.orchestra.run.smithi045.stdout: "num_blocked_ops": 0 2024-08-19T14:35:13.254 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:35:13.254 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:13.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:13.256 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:35:13.357 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:35:13.379 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:13.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:13.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:35:13.481 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:35:13.504 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:13.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:13.505 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:35:13.606 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:35:13.653 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:13.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:13.654 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:35:13.755 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:35:13.777 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:13.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:13.778 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:35:13.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:13 smithi045 ceph-mon[20143]: pgmap v1494: 47 pgs: 1 remapped+peering, 6 peering, 4 active+clean+remapped, 3 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized, 2 active+undersized+degraded, 29 active+clean; 7.0 GiB data, 29 GiB used, 865 GiB / 894 GiB avail; 170 B/s rd, 1.8 MiB/s wr, 631 op/s; 14467/342636 objects degraded (4.222%); 11329/342636 objects misplaced (3.306%); 20 MiB/s, 312 objects/s recovering 2024-08-19T14:35:13.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:13 smithi045 ceph-mon[20143]: osdmap e636: 12 total, 12 up, 10 in 2024-08-19T14:35:13.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:13 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 44036/347409 objects degraded (12.676%), 7 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-08-19T14:35:13.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:13 smithi045 ceph-mon[20143]: osdmap e637: 12 total, 12 up, 10 in 2024-08-19T14:35:13.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:13 smithi031 ceph-mon[16336]: pgmap v1494: 47 pgs: 1 remapped+peering, 6 peering, 4 active+clean+remapped, 3 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized, 2 active+undersized+degraded, 29 active+clean; 7.0 GiB data, 29 GiB used, 865 GiB / 894 GiB avail; 170 B/s rd, 1.8 MiB/s wr, 631 op/s; 14467/342636 objects degraded (4.222%); 11329/342636 objects misplaced (3.306%); 20 MiB/s, 312 objects/s recovering 2024-08-19T14:35:13.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:13 smithi031 ceph-mon[16336]: osdmap e636: 12 total, 12 up, 10 in 2024-08-19T14:35:13.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:13 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 44036/347409 objects degraded (12.676%), 7 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-08-19T14:35:13.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:13 smithi031 ceph-mon[16336]: osdmap e637: 12 total, 12 up, 10 in 2024-08-19T14:35:13.879 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:35:13.911 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:13.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:13.912 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:35:14.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:13 smithi032 ceph-mon[20004]: pgmap v1494: 47 pgs: 1 remapped+peering, 6 peering, 4 active+clean+remapped, 3 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized, 2 active+undersized+degraded, 29 active+clean; 7.0 GiB data, 29 GiB used, 865 GiB / 894 GiB avail; 170 B/s rd, 1.8 MiB/s wr, 631 op/s; 14467/342636 objects degraded (4.222%); 11329/342636 objects misplaced (3.306%); 20 MiB/s, 312 objects/s recovering 2024-08-19T14:35:14.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:13 smithi032 ceph-mon[20004]: osdmap e636: 12 total, 12 up, 10 in 2024-08-19T14:35:14.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:13 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 44036/347409 objects degraded (12.676%), 7 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-08-19T14:35:14.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:13 smithi032 ceph-mon[20004]: osdmap e637: 12 total, 12 up, 10 in 2024-08-19T14:35:14.012 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:35:14.053 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:14.054 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-08-19T14:35:14.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:14.055 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:35:14.155 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:35:14.181 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:14.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:14.182 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:35:14.283 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:35:14.313 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:14.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:14.314 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:35:14.393 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:35:14.415 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:35:14.437 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:14.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:14.438 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:35:14.510 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd reweight 0 1 2024-08-19T14:35:14.539 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:35:14.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:14.565 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:14.566 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:35:14.667 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:35:14.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:14.757 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:14.757 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:35:14.858 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:35:14.916 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:14.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:14.918 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:35:14.948 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:35:14.948 INFO:teuthology.orchestra.run.smithi045.stdout: "size": 20, 2024-08-19T14:35:14.948 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 600, 2024-08-19T14:35:14.948 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [] 2024-08-19T14:35:14.948 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:35:15.019 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:35:15.040 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:15.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:15.041 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:35:15.103 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:14 smithi031 ceph-mon[16336]: osdmap e638: 12 total, 12 up, 10 in 2024-08-19T14:35:15.142 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:35:15.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:15.220 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:15.221 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:35:15.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:14 smithi032 ceph-mon[20004]: osdmap e638: 12 total, 12 up, 10 in 2024-08-19T14:35:15.272 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:14 smithi045 ceph-mon[20143]: osdmap e638: 12 total, 12 up, 10 in 2024-08-19T14:35:15.303 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:35:15.322 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:35:15.328 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:35:15.344 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:15.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:15.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:35:15.446 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:35:15.470 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:15.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:15.471 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:35:15.572 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:35:15.600 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:15.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:15.601 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:35:15.702 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:35:15.719 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-08-19T14:35:15.731 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:15.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:15.732 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:35:15.833 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:35:15.858 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:15.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:15.859 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:35:15.960 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:35:15.984 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:15.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:15.986 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:35:16.087 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:35:16.094 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:15 smithi031 ceph-mon[16336]: pgmap v1498: 47 pgs: 1 remapped+peering, 6 peering, 4 active+clean+remapped, 3 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized, 2 active+undersized+degraded, 29 active+clean; 7.0 GiB data, 27 GiB used, 867 GiB / 894 GiB avail; 0 B/s wr, 346 op/s; 14467/342408 objects degraded (4.225%); 11329/342408 objects misplaced (3.309%); 14 MiB/s, 231 objects/s recovering 2024-08-19T14:35:16.095 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:15 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:35:16.108 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:16.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:16.109 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:35:16.210 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:35:16.236 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:16.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:16.237 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:35:16.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:15 smithi032 ceph-mon[20004]: pgmap v1498: 47 pgs: 1 remapped+peering, 6 peering, 4 active+clean+remapped, 3 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized, 2 active+undersized+degraded, 29 active+clean; 7.0 GiB data, 27 GiB used, 867 GiB / 894 GiB avail; 0 B/s wr, 346 op/s; 14467/342408 objects degraded (4.225%); 11329/342408 objects misplaced (3.309%); 14 MiB/s, 231 objects/s recovering 2024-08-19T14:35:16.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:15 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:35:16.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:15 smithi045 ceph-mon[20143]: pgmap v1498: 47 pgs: 1 remapped+peering, 6 peering, 4 active+clean+remapped, 3 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized, 2 active+undersized+degraded, 29 active+clean; 7.0 GiB data, 27 GiB used, 867 GiB / 894 GiB avail; 0 B/s wr, 346 op/s; 14467/342408 objects degraded (4.225%); 11329/342408 objects misplaced (3.309%); 14 MiB/s, 231 objects/s recovering 2024-08-19T14:35:16.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:15 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:35:16.333 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:35:16.334 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:35:16.338 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:35:16.359 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:16.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:16.360 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:35:16.407 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:35:16.407 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:35:16.459 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:35:16.460 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:35:16.461 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:35:16.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:16.483 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:16.483 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:35:16.511 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:35:16.511 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:35:16.555 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:35:16.561 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:35:16.562 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:35:16.584 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:35:16.611 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:16.612 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:35:16.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:16.613 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:35:16.613 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:35:16.654 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:35:16.654 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:35:16.696 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:35:16.696 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:35:16.713 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:35:16.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:16.737 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:16.737 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:35:16.746 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:35:16.746 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:35:16.789 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:35:16.789 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:35:16.827 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:35:16.827 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:35:16.838 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:35:16.860 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:35:16.860 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:35:16.861 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:16.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:16.862 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:35:16.962 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:35:16.985 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:16.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:16.986 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:35:17.087 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:35:17.109 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:17.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:17.110 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:35:17.160 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-19T14:35:17.208 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:16 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1874498742' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:35:17.208 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:16 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:35:17.208 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:16 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:35:17.209 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:16 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:35:17.209 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:16 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1931075385' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-08-19T14:35:17.209 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:16 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-08-19T14:35:17.211 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:35:17.235 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:17.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:17.236 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:35:17.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:16 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1874498742' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:35:17.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:16 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:35:17.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:16 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:35:17.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:16 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:35:17.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:16 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1931075385' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-08-19T14:35:17.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:16 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-08-19T14:35:17.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:16 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1874498742' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:35:17.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:16 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:35:17.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:16 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:35:17.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:16 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:35:17.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:16 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1931075385' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-08-19T14:35:17.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:16 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-08-19T14:35:17.337 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:35:17.363 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:17.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:17.364 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:35:17.423 INFO:teuthology.orchestra.run.smithi031.stderr:reweighted osd.0 to 1 (10000) 2024-08-19T14:35:17.465 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:35:17.486 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:17.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:17.487 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:35:17.510 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:35:17.510 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:35:17.589 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:35:17.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:17.646 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:17.646 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:35:17.747 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:35:17.772 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:17.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:17.773 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:35:17.874 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:35:17.895 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:17.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:17.896 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:35:17.998 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:35:18.020 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:18.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:18.021 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:35:18.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:17 smithi031 ceph-mon[16336]: pgmap v1499: 47 pgs: 1 remapped+peering, 1 peering, 3 active+recovering+undersized+remapped, 5 active+recovering+undersized+degraded+remapped, 37 active+clean; 6.9 GiB data, 27 GiB used, 867 GiB / 894 GiB avail; 0 B/s wr, 236 op/s; 31827/339345 objects degraded (9.379%); 14581/339345 objects misplaced (4.297%); 11 MiB/s, 176 objects/s recovering 2024-08-19T14:35:18.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:17 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-08-19T14:35:18.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:17 smithi031 ceph-mon[16336]: osdmap e639: 12 total, 12 up, 10 in 2024-08-19T14:35:18.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:17 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 31827/339345 objects degraded (9.379%), 5 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-19T14:35:18.122 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:35:18.143 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:18.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:18.144 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:35:18.246 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:35:18.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:17 smithi032 ceph-mon[20004]: pgmap v1499: 47 pgs: 1 remapped+peering, 1 peering, 3 active+recovering+undersized+remapped, 5 active+recovering+undersized+degraded+remapped, 37 active+clean; 6.9 GiB data, 27 GiB used, 867 GiB / 894 GiB avail; 0 B/s wr, 236 op/s; 31827/339345 objects degraded (9.379%); 14581/339345 objects misplaced (4.297%); 11 MiB/s, 176 objects/s recovering 2024-08-19T14:35:18.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:17 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-08-19T14:35:18.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:17 smithi032 ceph-mon[20004]: osdmap e639: 12 total, 12 up, 10 in 2024-08-19T14:35:18.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:17 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 31827/339345 objects degraded (9.379%), 5 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-19T14:35:18.274 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:18.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:18.275 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:35:18.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:17 smithi045 ceph-mon[20143]: pgmap v1499: 47 pgs: 1 remapped+peering, 1 peering, 3 active+recovering+undersized+remapped, 5 active+recovering+undersized+degraded+remapped, 37 active+clean; 6.9 GiB data, 27 GiB used, 867 GiB / 894 GiB avail; 0 B/s wr, 236 op/s; 31827/339345 objects degraded (9.379%); 14581/339345 objects misplaced (4.297%); 11 MiB/s, 176 objects/s recovering 2024-08-19T14:35:18.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:17 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-08-19T14:35:18.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:17 smithi045 ceph-mon[20143]: osdmap e639: 12 total, 12 up, 10 in 2024-08-19T14:35:18.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:17 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 31827/339345 objects degraded (9.379%), 5 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-19T14:35:18.376 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:35:18.400 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:18.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:18.401 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:35:18.502 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:35:18.527 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:18.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:18.528 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:35:18.629 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:35:18.653 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:18.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:18.654 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:35:18.755 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:35:18.778 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:18.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:18.780 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:35:18.881 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:35:18.904 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:18.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:18.905 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:35:19.006 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:35:19.032 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:19.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:19.033 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:35:19.134 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:35:19.158 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:19.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:19.159 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:35:19.260 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:35:19.282 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:19.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:19.283 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:35:19.384 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:35:19.407 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:19.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:19.408 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:35:19.509 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:35:19.530 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:19.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:19.531 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:35:19.632 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:35:19.673 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:19.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:19.674 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:35:19.775 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:35:19.801 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:19.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:19.802 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:35:19.903 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:35:19.927 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:19.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:19.928 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:35:20.029 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:35:20.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:20.091 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:20.091 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:35:20.192 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:35:20.214 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:20.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:20.215 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:35:20.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:19 smithi032 ceph-mon[20004]: pgmap v1501: 47 pgs: 3 active+recovering+undersized+remapped, 6 active+recovering+undersized+degraded+remapped, 38 active+clean; 6.9 GiB data, 27 GiB used, 867 GiB / 894 GiB avail; 0 B/s wr, 221 op/s; 33692/338898 objects degraded (9.942%); 20698/338898 objects misplaced (6.107%); 19 MiB/s, 301 objects/s recovering 2024-08-19T14:35:20.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:19 smithi045 ceph-mon[20143]: pgmap v1501: 47 pgs: 3 active+recovering+undersized+remapped, 6 active+recovering+undersized+degraded+remapped, 38 active+clean; 6.9 GiB data, 27 GiB used, 867 GiB / 894 GiB avail; 0 B/s wr, 221 op/s; 33692/338898 objects degraded (9.942%); 20698/338898 objects misplaced (6.107%); 19 MiB/s, 301 objects/s recovering 2024-08-19T14:35:20.316 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:35:20.343 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:20.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:20.345 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:35:20.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:19 smithi031 ceph-mon[16336]: pgmap v1501: 47 pgs: 3 active+recovering+undersized+remapped, 6 active+recovering+undersized+degraded+remapped, 38 active+clean; 6.9 GiB data, 27 GiB used, 867 GiB / 894 GiB avail; 0 B/s wr, 221 op/s; 33692/338898 objects degraded (9.942%); 20698/338898 objects misplaced (6.107%); 19 MiB/s, 301 objects/s recovering 2024-08-19T14:35:20.405 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:35:20.445 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:35:20.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:20.508 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:20.508 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:35:20.609 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:35:20.631 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:20.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:20.632 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:35:20.733 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:35:20.760 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:20.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:20.761 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:35:20.862 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:35:20.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:20.884 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:20.884 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:35:20.985 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:35:21.010 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:21.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:21.011 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:35:21.112 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:35:21.137 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:21.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:21.139 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:35:21.239 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:35:21.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:20 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:35:21.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:20 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:35:21.260 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:21.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:21.261 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:35:21.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:20 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:35:21.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:20 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:35:21.362 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:35:21.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:20 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:35:21.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:20 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:35:21.410 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:21.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:21.412 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:35:21.513 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:35:21.539 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:21.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:21.540 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:35:21.641 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:35:21.663 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:21.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:21.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:35:21.765 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:35:21.801 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:21.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:21.802 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:35:21.903 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:35:21.927 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:21.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:21.928 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:35:22.030 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:35:22.054 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:22.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:22.055 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:35:22.156 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:35:22.178 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:22.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:22.179 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:35:22.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:21 smithi032 ceph-mon[20004]: pgmap v1502: 47 pgs: 3 active+recovering+undersized+remapped, 6 active+recovering+undersized+degraded+remapped, 38 active+clean; 6.8 GiB data, 27 GiB used, 867 GiB / 894 GiB avail; 0 B/s wr, 299 op/s; 33755/336138 objects degraded (10.042%); 20483/336138 objects misplaced (6.094%); 16 MiB/s, 264 objects/s recovering 2024-08-19T14:35:22.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:21 smithi032 ceph-mon[20004]: osdmap e640: 12 total, 12 up, 10 in 2024-08-19T14:35:22.280 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:35:22.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:21 smithi045 ceph-mon[20143]: pgmap v1502: 47 pgs: 3 active+recovering+undersized+remapped, 6 active+recovering+undersized+degraded+remapped, 38 active+clean; 6.8 GiB data, 27 GiB used, 867 GiB / 894 GiB avail; 0 B/s wr, 299 op/s; 33755/336138 objects degraded (10.042%); 20483/336138 objects misplaced (6.094%); 16 MiB/s, 264 objects/s recovering 2024-08-19T14:35:22.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:21 smithi045 ceph-mon[20143]: osdmap e640: 12 total, 12 up, 10 in 2024-08-19T14:35:22.302 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:22.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:22.303 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:35:22.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:21 smithi031 ceph-mon[16336]: pgmap v1502: 47 pgs: 3 active+recovering+undersized+remapped, 6 active+recovering+undersized+degraded+remapped, 38 active+clean; 6.8 GiB data, 27 GiB used, 867 GiB / 894 GiB avail; 0 B/s wr, 299 op/s; 33755/336138 objects degraded (10.042%); 20483/336138 objects misplaced (6.094%); 16 MiB/s, 264 objects/s recovering 2024-08-19T14:35:22.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:21 smithi031 ceph-mon[16336]: osdmap e640: 12 total, 12 up, 10 in 2024-08-19T14:35:22.404 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:35:22.426 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:22.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:22.427 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:35:22.528 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:35:22.551 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:22.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:22.552 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:35:22.653 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:35:22.693 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:22.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:22.694 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:35:22.796 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:35:22.817 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:22.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:22.818 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:35:22.919 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:35:22.941 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:22.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:22.942 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:35:23.043 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:35:23.044 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:22 smithi031 ceph-mon[16336]: osdmap e641: 12 total, 12 up, 10 in 2024-08-19T14:35:23.044 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:23 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 33755/336138 objects degraded (10.042%), 6 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-19T14:35:23.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:23.067 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:23.067 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:35:23.168 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:35:23.190 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:23.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:23.191 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:35:23.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:22 smithi032 ceph-mon[20004]: osdmap e641: 12 total, 12 up, 10 in 2024-08-19T14:35:23.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:22 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 33755/336138 objects degraded (10.042%), 6 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-19T14:35:23.292 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:35:23.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:22 smithi045 ceph-mon[20143]: osdmap e641: 12 total, 12 up, 10 in 2024-08-19T14:35:23.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:22 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 33755/336138 objects degraded (10.042%), 6 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-19T14:35:23.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:23.330 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:23.330 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:35:23.431 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:35:23.466 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:23.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:23.467 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:35:23.568 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:35:23.594 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:23.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:23.595 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:35:23.696 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:35:23.717 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:23.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:23.719 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:35:23.820 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:35:23.842 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:23.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:23.843 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:35:23.944 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:35:23.966 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:23.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:23.968 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:35:24.069 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:35:24.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:24.094 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:24.094 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:35:24.196 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:35:24.219 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:24.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:24.220 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:35:24.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:23 smithi032 ceph-mon[20004]: pgmap v1505: 47 pgs: 1 active+undersized+remapped, 2 active+recovering+undersized+remapped, 6 active+recovering+undersized+degraded+remapped, 38 active+clean; 6.8 GiB data, 27 GiB used, 867 GiB / 894 GiB avail; 0 B/s wr, 331 op/s; 33459/333384 objects degraded (10.036%); 19158/333384 objects misplaced (5.747%); 27 MiB/s, 437 objects/s recovering 2024-08-19T14:35:24.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:23 smithi032 ceph-mon[20004]: osdmap e642: 12 total, 12 up, 10 in 2024-08-19T14:35:24.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:23 smithi045 ceph-mon[20143]: pgmap v1505: 47 pgs: 1 active+undersized+remapped, 2 active+recovering+undersized+remapped, 6 active+recovering+undersized+degraded+remapped, 38 active+clean; 6.8 GiB data, 27 GiB used, 867 GiB / 894 GiB avail; 0 B/s wr, 331 op/s; 33459/333384 objects degraded (10.036%); 19158/333384 objects misplaced (5.747%); 27 MiB/s, 437 objects/s recovering 2024-08-19T14:35:24.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:23 smithi045 ceph-mon[20143]: osdmap e642: 12 total, 12 up, 10 in 2024-08-19T14:35:24.321 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:35:24.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:23 smithi031 ceph-mon[16336]: pgmap v1505: 47 pgs: 1 active+undersized+remapped, 2 active+recovering+undersized+remapped, 6 active+recovering+undersized+degraded+remapped, 38 active+clean; 6.8 GiB data, 27 GiB used, 867 GiB / 894 GiB avail; 0 B/s wr, 331 op/s; 33459/333384 objects degraded (10.036%); 19158/333384 objects misplaced (5.747%); 27 MiB/s, 437 objects/s recovering 2024-08-19T14:35:24.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:23 smithi031 ceph-mon[16336]: osdmap e642: 12 total, 12 up, 10 in 2024-08-19T14:35:24.377 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:24.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:24.378 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:35:24.479 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:35:24.511 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:24.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:24.512 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:35:24.613 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:35:24.636 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:24.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:24.637 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:35:24.737 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:35:24.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:24.760 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:24.760 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:35:24.861 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:35:24.883 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:24.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:24.884 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:35:24.985 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:35:25.023 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:25.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:25.024 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:35:25.125 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:35:25.148 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:25.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:25.149 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:35:25.250 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:35:25.271 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:25.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:25.272 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:35:25.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:25 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:35:25.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:25 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:35:25.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:25 smithi045 ceph-mon[20143]: osdmap e643: 12 total, 12 up, 10 in 2024-08-19T14:35:25.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:25 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:35:25.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:25 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:35:25.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:25 smithi031 ceph-mon[16336]: osdmap e643: 12 total, 12 up, 10 in 2024-08-19T14:35:25.373 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:35:25.394 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:25.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:25.396 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:35:25.497 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:35:25.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:25 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:35:25.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:25 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:35:25.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:25 smithi032 ceph-mon[20004]: osdmap e643: 12 total, 12 up, 10 in 2024-08-19T14:35:25.518 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:25.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:25.519 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:35:25.620 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:35:25.642 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:25.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:25.643 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:35:25.744 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:35:25.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:25.788 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:25.788 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:35:25.889 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:35:25.912 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:25.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:25.913 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:35:26.014 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:35:26.060 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:26.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:26.061 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:35:26.162 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:35:26.186 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:26.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:26.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:35:26.288 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:35:26.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:26 smithi045 ceph-mon[20143]: pgmap v1508: 47 pgs: 1 active+undersized+remapped, 2 active+recovering+undersized+remapped, 6 active+recovering+undersized+degraded+remapped, 38 active+clean; 6.8 GiB data, 27 GiB used, 867 GiB / 894 GiB avail; 0 B/s wr, 252 op/s; 33459/333102 objects degraded (10.045%); 19158/333102 objects misplaced (5.751%); 24 MiB/s, 382 objects/s recovering 2024-08-19T14:35:26.328 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:26.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:26.329 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:35:26.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:26 smithi031 ceph-mon[16336]: pgmap v1508: 47 pgs: 1 active+undersized+remapped, 2 active+recovering+undersized+remapped, 6 active+recovering+undersized+degraded+remapped, 38 active+clean; 6.8 GiB data, 27 GiB used, 867 GiB / 894 GiB avail; 0 B/s wr, 252 op/s; 33459/333102 objects degraded (10.045%); 19158/333102 objects misplaced (5.751%); 24 MiB/s, 382 objects/s recovering 2024-08-19T14:35:26.430 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:35:26.452 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:26.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:26.453 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:35:26.493 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-08-19T14:35:26.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:26 smithi032 ceph-mon[20004]: pgmap v1508: 47 pgs: 1 active+undersized+remapped, 2 active+recovering+undersized+remapped, 6 active+recovering+undersized+degraded+remapped, 38 active+clean; 6.8 GiB data, 27 GiB used, 867 GiB / 894 GiB avail; 0 B/s wr, 252 op/s; 33459/333102 objects degraded (10.045%); 19158/333102 objects misplaced (5.751%); 24 MiB/s, 382 objects/s recovering 2024-08-19T14:35:26.554 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:35:26.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:26.648 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:26.648 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:35:26.749 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:35:26.771 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:26.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:26.772 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:35:26.873 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:35:26.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:26.937 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:26.937 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:35:27.038 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:35:27.061 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:27.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:27.062 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:35:27.163 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:35:27.187 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:27.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:27.188 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:35:27.289 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:35:27.311 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:27.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:27.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:35:27.414 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:35:27.436 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:27.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:27.437 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:35:27.538 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:35:27.559 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:27.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:27.560 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:35:27.661 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:35:27.686 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:27.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:27.687 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:35:27.788 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:35:27.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:27.832 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:27.832 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:35:27.933 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:35:27.956 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:27.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:27.957 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:35:28.058 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:35:28.082 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:28.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:28.084 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:35:28.185 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:35:28.209 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:28.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:28.210 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:35:28.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:28 smithi045 ceph-mon[20143]: pgmap v1509: 47 pgs: 1 active+undersized+remapped, 2 active+recovering+undersized+remapped, 6 active+recovering+undersized+degraded+remapped, 38 active+clean; 6.7 GiB data, 26 GiB used, 868 GiB / 894 GiB avail; 0 B/s wr, 332 op/s; 33103/328938 objects degraded (10.064%); 18723/328938 objects misplaced (5.692%); 23 MiB/s, 367 objects/s recovering 2024-08-19T14:35:28.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:28 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 33103/328938 objects degraded (10.064%), 6 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-19T14:35:28.311 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:35:28.333 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:28.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:28.334 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:35:28.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:28 smithi031 ceph-mon[16336]: pgmap v1509: 47 pgs: 1 active+undersized+remapped, 2 active+recovering+undersized+remapped, 6 active+recovering+undersized+degraded+remapped, 38 active+clean; 6.7 GiB data, 26 GiB used, 868 GiB / 894 GiB avail; 0 B/s wr, 332 op/s; 33103/328938 objects degraded (10.064%); 18723/328938 objects misplaced (5.692%); 23 MiB/s, 367 objects/s recovering 2024-08-19T14:35:28.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:28 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 33103/328938 objects degraded (10.064%), 6 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-19T14:35:28.435 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:35:28.458 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:28.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:28.459 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:35:28.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:28 smithi032 ceph-mon[20004]: pgmap v1509: 47 pgs: 1 active+undersized+remapped, 2 active+recovering+undersized+remapped, 6 active+recovering+undersized+degraded+remapped, 38 active+clean; 6.7 GiB data, 26 GiB used, 868 GiB / 894 GiB avail; 0 B/s wr, 332 op/s; 33103/328938 objects degraded (10.064%); 18723/328938 objects misplaced (5.692%); 23 MiB/s, 367 objects/s recovering 2024-08-19T14:35:28.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:28 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 33103/328938 objects degraded (10.064%), 6 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-19T14:35:28.560 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:35:28.581 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:28.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:28.582 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:35:28.683 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:35:28.705 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:28.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:28.706 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:35:28.807 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:35:28.829 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:28.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:28.830 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:35:28.932 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:35:28.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:28.984 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:28.985 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:35:29.086 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:35:29.109 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:29.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:29.110 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:35:29.211 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:35:29.255 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:29.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:29.256 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:35:29.357 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:35:29.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:29.382 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:29.382 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:35:29.483 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:35:29.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:29.520 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:29.521 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:35:29.621 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:35:29.656 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:29.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:29.657 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:35:29.758 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:35:29.782 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:29.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:29.783 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:35:29.884 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:35:29.915 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:29.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:29.916 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:35:30.017 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:35:30.056 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:30.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:30.057 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:35:30.158 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:35:30.179 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:30.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:30.180 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:35:30.281 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:35:30.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:30 smithi045 ceph-mon[20143]: pgmap v1510: 47 pgs: 2 active+recovering+undersized+remapped, 6 active+recovering+undersized+degraded+remapped, 39 active+clean; 6.7 GiB data, 26 GiB used, 868 GiB / 894 GiB avail; 0 B/s wr, 275 op/s; 33079/328278 objects degraded (10.077%); 18505/328278 objects misplaced (5.637%); 9.9 MiB/s, 157 objects/s recovering 2024-08-19T14:35:30.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:30 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:35:30.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:30 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:35:30.303 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:30.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:30.304 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:35:30.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:30 smithi031 ceph-mon[16336]: pgmap v1510: 47 pgs: 2 active+recovering+undersized+remapped, 6 active+recovering+undersized+degraded+remapped, 39 active+clean; 6.7 GiB data, 26 GiB used, 868 GiB / 894 GiB avail; 0 B/s wr, 275 op/s; 33079/328278 objects degraded (10.077%); 18505/328278 objects misplaced (5.637%); 9.9 MiB/s, 157 objects/s recovering 2024-08-19T14:35:30.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:30 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:35:30.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:30 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:35:30.405 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:35:30.428 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:30.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:30.429 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:35:30.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:30 smithi032 ceph-mon[20004]: pgmap v1510: 47 pgs: 2 active+recovering+undersized+remapped, 6 active+recovering+undersized+degraded+remapped, 39 active+clean; 6.7 GiB data, 26 GiB used, 868 GiB / 894 GiB avail; 0 B/s wr, 275 op/s; 33079/328278 objects degraded (10.077%); 18505/328278 objects misplaced (5.637%); 9.9 MiB/s, 157 objects/s recovering 2024-08-19T14:35:30.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:30 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:35:30.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:30 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:35:30.530 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:35:30.551 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:30.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:30.552 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:35:30.653 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:35:30.675 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:30.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:30.676 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:35:30.777 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:35:30.803 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:30.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:30.804 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:35:30.905 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:35:30.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:30.976 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:30.976 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:35:31.077 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:35:31.123 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:31.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:31.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:35:31.225 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:35:31.249 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:31.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:31.250 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:35:31.351 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:35:31.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:31 smithi031 ceph-mon[16336]: osdmap e644: 12 total, 12 up, 10 in 2024-08-19T14:35:31.372 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:31.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:31.373 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:35:31.474 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:35:31.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:31 smithi032 ceph-mon[20004]: osdmap e644: 12 total, 12 up, 10 in 2024-08-19T14:35:31.515 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:31.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:31.517 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:35:31.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:31 smithi045 ceph-mon[20143]: osdmap e644: 12 total, 12 up, 10 in 2024-08-19T14:35:31.618 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:35:31.640 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:31.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:31.641 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:35:31.742 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:35:31.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:31.763 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:31.764 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:35:31.865 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:35:31.888 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:31.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:31.889 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:35:31.990 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:35:32.012 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:32.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:32.013 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:35:32.044 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:35:32.088 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd reweight 1 1 2024-08-19T14:35:32.114 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:35:32.135 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:32.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:32.136 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:35:32.237 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:35:32.336 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:32.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:32.337 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:35:32.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:32 smithi031 ceph-mon[16336]: pgmap v1512: 47 pgs: 2 active+recovering+undersized+remapped, 6 active+recovering+undersized+degraded+remapped, 39 active+clean; 6.6 GiB data, 26 GiB used, 868 GiB / 894 GiB avail; 0 B/s wr, 373 op/s; 33101/325353 objects degraded (10.174%); 18235/325353 objects misplaced (5.605%); 11 MiB/s, 173 objects/s recovering 2024-08-19T14:35:32.438 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:35:32.459 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:32.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:32.460 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:35:32.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:32 smithi032 ceph-mon[20004]: pgmap v1512: 47 pgs: 2 active+recovering+undersized+remapped, 6 active+recovering+undersized+degraded+remapped, 39 active+clean; 6.6 GiB data, 26 GiB used, 868 GiB / 894 GiB avail; 0 B/s wr, 373 op/s; 33101/325353 objects degraded (10.174%); 18235/325353 objects misplaced (5.605%); 11 MiB/s, 173 objects/s recovering 2024-08-19T14:35:32.537 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:35:32.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:32 smithi045 ceph-mon[20143]: pgmap v1512: 47 pgs: 2 active+recovering+undersized+remapped, 6 active+recovering+undersized+degraded+remapped, 39 active+clean; 6.6 GiB data, 26 GiB used, 868 GiB / 894 GiB avail; 0 B/s wr, 373 op/s; 33101/325353 objects degraded (10.174%); 18235/325353 objects misplaced (5.605%); 11 MiB/s, 173 objects/s recovering 2024-08-19T14:35:32.561 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:35:32.583 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:32.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:32.584 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:35:32.601 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:35:32.684 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:35:32.717 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:32.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:32.718 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:35:32.819 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:35:32.841 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:32.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:32.842 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:35:32.943 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:35:32.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:32.982 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:32.983 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:35:33.083 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:35:33.108 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:33.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:33.109 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:35:33.210 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:35:33.233 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:33.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:33.234 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:35:33.335 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:35:33.364 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:33.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:33.365 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:35:33.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:33 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 33101/325353 objects degraded (10.174%), 6 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-19T14:35:33.466 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:35:33.488 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:33.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:33.489 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:35:33.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:33 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 33101/325353 objects degraded (10.174%), 6 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-19T14:35:33.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:33 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 33101/325353 objects degraded (10.174%), 6 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-19T14:35:33.591 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:35:33.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:33.629 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:33.629 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:35:33.730 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:35:33.752 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:33.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:33.754 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:35:33.854 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:35:33.876 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:33.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:33.877 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:35:33.978 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:35:34.002 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:34.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:34.003 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:35:34.104 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:35:34.127 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:34.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:34.128 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:35:34.229 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:35:34.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:34.252 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:34.252 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:35:34.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:34 smithi032 ceph-mon[20004]: pgmap v1513: 47 pgs: 2 active+recovering+undersized+remapped, 6 active+recovering+undersized+degraded+remapped, 39 active+clean; 6.6 GiB data, 26 GiB used, 868 GiB / 894 GiB avail; 0 B/s wr, 430 op/s; 32665/322566 objects degraded (10.127%); 17791/322566 objects misplaced (5.515%); 15 MiB/s, 245 objects/s recovering 2024-08-19T14:35:34.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:34 smithi045 ceph-mon[20143]: pgmap v1513: 47 pgs: 2 active+recovering+undersized+remapped, 6 active+recovering+undersized+degraded+remapped, 39 active+clean; 6.6 GiB data, 26 GiB used, 868 GiB / 894 GiB avail; 0 B/s wr, 430 op/s; 32665/322566 objects degraded (10.127%); 17791/322566 objects misplaced (5.515%); 15 MiB/s, 245 objects/s recovering 2024-08-19T14:35:34.353 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:35:34.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:34 smithi031 ceph-mon[16336]: pgmap v1513: 47 pgs: 2 active+recovering+undersized+remapped, 6 active+recovering+undersized+degraded+remapped, 39 active+clean; 6.6 GiB data, 26 GiB used, 868 GiB / 894 GiB avail; 0 B/s wr, 430 op/s; 32665/322566 objects degraded (10.127%); 17791/322566 objects misplaced (5.515%); 15 MiB/s, 245 objects/s recovering 2024-08-19T14:35:34.376 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:34.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:34.377 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:35:34.478 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:35:34.528 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:34.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:34.529 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:35:34.630 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:35:34.652 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:34.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:34.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:35:34.754 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:35:34.777 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:34.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:34.778 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:35:34.879 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:35:34.902 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:34.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:34.903 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:35:35.004 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:35:35.025 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:35.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:35.026 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:35:35.127 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:35:35.149 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:35.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:35.150 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:35:35.251 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:35:35.274 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:35.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:35.275 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:35:35.376 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:35:35.417 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:35.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:35.418 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:35:35.519 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:35:35.541 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:35.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:35.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:35:35.643 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:35:35.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:35.724 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:35.725 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:35:35.825 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:35:35.847 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:35.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:35.848 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:35:35.949 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:35:35.972 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:35.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:35.973 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:35:36.074 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:35:36.113 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:36.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:36.114 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:35:36.215 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:35:36.237 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:36.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:36.238 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:35:36.338 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:35:36.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:36 smithi031 ceph-mon[16336]: pgmap v1514: 47 pgs: 2 active+recovering+undersized+remapped, 6 active+recovering+undersized+degraded+remapped, 39 active+clean; 6.6 GiB data, 26 GiB used, 868 GiB / 894 GiB avail; 0 B/s wr, 363 op/s; 32665/322197 objects degraded (10.138%); 17791/322197 objects misplaced (5.522%); 12 MiB/s, 200 objects/s recovering 2024-08-19T14:35:36.374 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:36.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:36.375 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:35:36.476 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:35:36.498 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:36.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:36.499 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:35:36.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:36 smithi032 ceph-mon[20004]: pgmap v1514: 47 pgs: 2 active+recovering+undersized+remapped, 6 active+recovering+undersized+degraded+remapped, 39 active+clean; 6.6 GiB data, 26 GiB used, 868 GiB / 894 GiB avail; 0 B/s wr, 363 op/s; 32665/322197 objects degraded (10.138%); 17791/322197 objects misplaced (5.522%); 12 MiB/s, 200 objects/s recovering 2024-08-19T14:35:36.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:36 smithi045 ceph-mon[20143]: pgmap v1514: 47 pgs: 2 active+recovering+undersized+remapped, 6 active+recovering+undersized+degraded+remapped, 39 active+clean; 6.6 GiB data, 26 GiB used, 868 GiB / 894 GiB avail; 0 B/s wr, 363 op/s; 32665/322197 objects degraded (10.138%); 17791/322197 objects misplaced (5.522%); 12 MiB/s, 200 objects/s recovering 2024-08-19T14:35:36.600 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:35:36.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:36.636 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:36.636 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:35:36.737 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:35:36.759 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:36.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:36.760 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:35:36.861 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:35:36.885 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:36.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:36.886 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:35:36.987 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:35:37.011 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:37.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:37.012 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:35:37.113 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:35:37.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:37.156 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:37.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:35:37.258 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:35:37.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:37.313 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:37.314 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:35:37.414 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:35:37.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:37.462 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:37.462 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:35:37.563 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:35:37.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:37.615 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:37.616 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:35:37.716 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:35:37.738 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:37.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:37.739 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:35:37.840 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:35:37.897 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:37.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:37.898 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:35:37.999 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:35:38.022 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:38.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:38.023 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:35:38.124 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:35:38.146 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:38.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:38.147 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:35:38.248 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:35:38.272 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:38.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:38.273 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:35:38.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:38 smithi031 ceph-mon[16336]: pgmap v1515: 47 pgs: 2 active+recovering+undersized+remapped, 6 active+recovering+undersized+degraded+remapped, 39 active+clean; 6.5 GiB data, 26 GiB used, 868 GiB / 894 GiB avail; 0 B/s wr, 329 op/s; 32576/319041 objects degraded (10.211%); 17535/319041 objects misplaced (5.496%); 9.8 MiB/s, 157 objects/s recovering 2024-08-19T14:35:38.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:38 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:35:38.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:38 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:35:38.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:38 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 32576/319041 objects degraded (10.211%), 6 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-19T14:35:38.374 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:35:38.421 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:38.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:38.422 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:35:38.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:38 smithi032 ceph-mon[20004]: pgmap v1515: 47 pgs: 2 active+recovering+undersized+remapped, 6 active+recovering+undersized+degraded+remapped, 39 active+clean; 6.5 GiB data, 26 GiB used, 868 GiB / 894 GiB avail; 0 B/s wr, 329 op/s; 32576/319041 objects degraded (10.211%); 17535/319041 objects misplaced (5.496%); 9.8 MiB/s, 157 objects/s recovering 2024-08-19T14:35:38.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:38 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:35:38.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:38 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:35:38.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:38 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 32576/319041 objects degraded (10.211%), 6 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-19T14:35:38.523 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:35:38.546 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:38.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:38 smithi045 ceph-mon[20143]: pgmap v1515: 47 pgs: 2 active+recovering+undersized+remapped, 6 active+recovering+undersized+degraded+remapped, 39 active+clean; 6.5 GiB data, 26 GiB used, 868 GiB / 894 GiB avail; 0 B/s wr, 329 op/s; 32576/319041 objects degraded (10.211%); 17535/319041 objects misplaced (5.496%); 9.8 MiB/s, 157 objects/s recovering 2024-08-19T14:35:38.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:38 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:35:38.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:38 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:35:38.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:38 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 32576/319041 objects degraded (10.211%), 6 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-19T14:35:38.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:38.547 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:35:38.648 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:35:38.694 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:38.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:38.695 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:35:38.797 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:35:38.818 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:38.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:38.819 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:35:38.920 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:35:38.941 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:38.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:38.942 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:35:39.044 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:35:39.065 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:39.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:39.066 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:35:39.167 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:35:39.189 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:39.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:39.191 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:35:39.292 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:35:39.315 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:39.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:39.316 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:35:39.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:39 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:35:39.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:39 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:35:39.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:39 smithi031 ceph-mon[16336]: osdmap e645: 12 total, 12 up, 10 in 2024-08-19T14:35:39.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:39 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:35:39.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:39 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:35:39.416 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:35:39.443 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:39.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:39.444 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:35:39.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:39 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:35:39.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:39 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:35:39.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:39 smithi032 ceph-mon[20004]: osdmap e645: 12 total, 12 up, 10 in 2024-08-19T14:35:39.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:39 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:35:39.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:39 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:35:39.545 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:35:39.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:39 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:35:39.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:39 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:35:39.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:39 smithi045 ceph-mon[20143]: osdmap e645: 12 total, 12 up, 10 in 2024-08-19T14:35:39.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:39 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:35:39.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:39 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:35:39.568 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:39.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:39.570 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:35:39.671 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:35:39.693 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:39.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:39.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:35:39.796 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:35:39.817 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:39.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:39.818 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:35:39.919 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:35:39.942 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:39.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:39.943 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:35:40.043 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:35:40.065 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:40.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:40.066 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:35:40.136 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is set 2024-08-19T14:35:40.167 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:35:40.232 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:40.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:40.233 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:35:40.334 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:35:40.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:40.358 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:40.358 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:35:40.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:40 smithi031 ceph-mon[16336]: pgmap v1517: 47 pgs: 2 active+recovering+undersized+remapped, 6 active+recovering+undersized+degraded+remapped, 39 active+clean; 6.4 GiB data, 26 GiB used, 868 GiB / 894 GiB avail; 0 B/s wr, 348 op/s; 32365/316887 objects degraded (10.213%); 17113/316887 objects misplaced (5.400%); 13 MiB/s, 210 objects/s recovering 2024-08-19T14:35:40.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:40 smithi031 ceph-mon[16336]: osdmap e646: 12 total, 12 up, 10 in 2024-08-19T14:35:40.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:40 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3502437436' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:35:40.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:40 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:35:40.459 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:35:40.482 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:40.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:40.483 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:35:40.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:40 smithi032 ceph-mon[20004]: pgmap v1517: 47 pgs: 2 active+recovering+undersized+remapped, 6 active+recovering+undersized+degraded+remapped, 39 active+clean; 6.4 GiB data, 26 GiB used, 868 GiB / 894 GiB avail; 0 B/s wr, 348 op/s; 32365/316887 objects degraded (10.213%); 17113/316887 objects misplaced (5.400%); 13 MiB/s, 210 objects/s recovering 2024-08-19T14:35:40.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:40 smithi032 ceph-mon[20004]: osdmap e646: 12 total, 12 up, 10 in 2024-08-19T14:35:40.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:40 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3502437436' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:35:40.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:40 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:35:40.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:40 smithi045 ceph-mon[20143]: pgmap v1517: 47 pgs: 2 active+recovering+undersized+remapped, 6 active+recovering+undersized+degraded+remapped, 39 active+clean; 6.4 GiB data, 26 GiB used, 868 GiB / 894 GiB avail; 0 B/s wr, 348 op/s; 32365/316887 objects degraded (10.213%); 17113/316887 objects misplaced (5.400%); 13 MiB/s, 210 objects/s recovering 2024-08-19T14:35:40.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:40 smithi045 ceph-mon[20143]: osdmap e646: 12 total, 12 up, 10 in 2024-08-19T14:35:40.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:40 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3502437436' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:35:40.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:40 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:35:40.584 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:35:40.607 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:40.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:40.608 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:35:40.709 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:35:40.732 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:40.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:40.734 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:35:40.834 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:35:40.883 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:40.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:40.884 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:35:40.985 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:35:41.006 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:41.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:41.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:35:41.108 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:35:41.131 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:41.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:41.132 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:35:41.234 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:35:41.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:41.271 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:41.271 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:35:41.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:41 smithi031 ceph-mon[16336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:35:41.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:41 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:35:41.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:41 smithi031 ceph-mon[16336]: osdmap e647: 12 total, 12 up, 10 in 2024-08-19T14:35:41.372 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:35:41.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:41.394 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:41.394 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:35:41.495 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:35:41.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:41 smithi032 ceph-mon[20004]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:35:41.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:41 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:35:41.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:41 smithi032 ceph-mon[20004]: osdmap e647: 12 total, 12 up, 10 in 2024-08-19T14:35:41.517 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:41.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:41.518 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:35:41.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:41 smithi045 ceph-mon[20143]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:35:41.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:41 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:35:41.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:41 smithi045 ceph-mon[20143]: osdmap e647: 12 total, 12 up, 10 in 2024-08-19T14:35:41.619 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:35:41.640 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:41.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:41.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:35:41.742 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:35:41.766 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:41.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:41.768 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:35:41.868 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:35:41.890 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:41.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:41.891 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:35:41.992 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:35:42.013 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:42.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:42.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:35:42.115 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:35:42.138 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:42.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:42.139 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:35:42.240 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:35:42.263 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:42.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:42.266 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:35:42.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:42 smithi031 ceph-mon[16336]: pgmap v1520: 47 pgs: 2 active+recovering+undersized+remapped, 6 active+recovering+undersized+degraded+remapped, 39 active+clean; 6.4 GiB data, 26 GiB used, 868 GiB / 894 GiB avail; 0 B/s wr, 445 op/s; 32429/314178 objects degraded (10.322%); 16858/314178 objects misplaced (5.366%); 12 MiB/s, 200 objects/s recovering 2024-08-19T14:35:42.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:42 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3362856318' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:35:42.368 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:35:42.417 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:42.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:42.419 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:35:42.445 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:35:42.445 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:35:42.445 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:35:42.445 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:35:42.445 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:35:42.445 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:35:42.445 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:35:42.445 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:35:42.445 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:35:42.446 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:35:42.446 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:35:42.446 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:35:42.446 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:35:42.446 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:35:42.446 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:35:42.446 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:35:42.446 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:35:42.447 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:35:42.447 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:35:42.447 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:35:42.447 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:35:42.447 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:35:42.447 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:35:42.447 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:35:42.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:42 smithi032 ceph-mon[20004]: pgmap v1520: 47 pgs: 2 active+recovering+undersized+remapped, 6 active+recovering+undersized+degraded+remapped, 39 active+clean; 6.4 GiB data, 26 GiB used, 868 GiB / 894 GiB avail; 0 B/s wr, 445 op/s; 32429/314178 objects degraded (10.322%); 16858/314178 objects misplaced (5.366%); 12 MiB/s, 200 objects/s recovering 2024-08-19T14:35:42.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:42 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3362856318' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:35:42.520 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:35:42.542 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:42.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:42.543 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:35:42.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:42 smithi045 ceph-mon[20143]: pgmap v1520: 47 pgs: 2 active+recovering+undersized+remapped, 6 active+recovering+undersized+degraded+remapped, 39 active+clean; 6.4 GiB data, 26 GiB used, 868 GiB / 894 GiB avail; 0 B/s wr, 445 op/s; 32429/314178 objects degraded (10.322%); 16858/314178 objects misplaced (5.366%); 12 MiB/s, 200 objects/s recovering 2024-08-19T14:35:42.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:42 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3362856318' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:35:42.644 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:35:42.665 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:42.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:42.666 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:35:42.767 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:35:42.789 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:42.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:42.790 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:35:42.891 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:35:42.913 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:42.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:42.913 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:35:43.014 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:35:43.036 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:43.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:43.037 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:35:43.138 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:35:43.167 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:43.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:43.168 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:35:43.269 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:35:43.332 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:43.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:43.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:35:43.434 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:35:43.468 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:43.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:43.469 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:35:43.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:43 smithi032 ceph-mon[20004]: pgmap v1521: 47 pgs: 2 active+recovering+undersized+remapped, 6 active+recovering+undersized+degraded+remapped, 39 active+clean; 6.3 GiB data, 26 GiB used, 868 GiB / 894 GiB avail; 0 B/s wr, 424 op/s; 32074/311403 objects degraded (10.300%); 16445/311403 objects misplaced (5.281%); 15 MiB/s, 240 objects/s recovering 2024-08-19T14:35:43.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:43 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 32429/314178 objects degraded (10.322%), 6 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-19T14:35:43.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:43 smithi045 ceph-mon[20143]: pgmap v1521: 47 pgs: 2 active+recovering+undersized+remapped, 6 active+recovering+undersized+degraded+remapped, 39 active+clean; 6.3 GiB data, 26 GiB used, 868 GiB / 894 GiB avail; 0 B/s wr, 424 op/s; 32074/311403 objects degraded (10.300%); 16445/311403 objects misplaced (5.281%); 15 MiB/s, 240 objects/s recovering 2024-08-19T14:35:43.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:43 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 32429/314178 objects degraded (10.322%), 6 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-19T14:35:43.569 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:35:43.592 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:43.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:43.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:35:43.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:43 smithi031 ceph-mon[16336]: pgmap v1521: 47 pgs: 2 active+recovering+undersized+remapped, 6 active+recovering+undersized+degraded+remapped, 39 active+clean; 6.3 GiB data, 26 GiB used, 868 GiB / 894 GiB avail; 0 B/s wr, 424 op/s; 32074/311403 objects degraded (10.300%); 16445/311403 objects misplaced (5.281%); 15 MiB/s, 240 objects/s recovering 2024-08-19T14:35:43.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:43 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 32429/314178 objects degraded (10.322%), 6 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-19T14:35:43.694 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:35:43.716 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:43.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:43.717 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:35:43.818 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:35:43.865 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:43.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:43.866 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:35:43.967 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:35:44.019 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:44.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:44.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:35:44.123 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:35:44.145 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:44.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:44.147 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:35:44.248 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:35:44.270 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:44.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:44.271 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:35:44.372 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:35:44.396 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:44.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:44.397 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:35:44.498 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:35:44.521 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:44.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:44.522 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:35:44.623 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:35:44.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:44.669 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:44.670 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:35:44.770 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:35:44.826 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:44.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:44.827 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:35:44.928 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:35:44.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:44.950 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:44.951 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:35:45.052 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:35:45.094 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:45.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:45.095 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:35:45.196 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:35:45.219 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:45.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:45.220 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:35:45.321 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:35:45.376 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:45.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:45.377 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:35:45.478 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:35:45.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:45.504 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:45.504 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:35:45.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:45 smithi032 ceph-mon[20004]: pgmap v1522: 47 pgs: 2 active+recovering+undersized+remapped, 6 active+recovering+undersized+degraded+remapped, 39 active+clean; 6.3 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 0 B/s wr, 324 op/s; 32074/310986 objects degraded (10.314%); 16445/310986 objects misplaced (5.288%); 9.4 MiB/s, 150 objects/s recovering 2024-08-19T14:35:45.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:45 smithi045 ceph-mon[20143]: pgmap v1522: 47 pgs: 2 active+recovering+undersized+remapped, 6 active+recovering+undersized+degraded+remapped, 39 active+clean; 6.3 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 0 B/s wr, 324 op/s; 32074/310986 objects degraded (10.314%); 16445/310986 objects misplaced (5.288%); 9.4 MiB/s, 150 objects/s recovering 2024-08-19T14:35:45.606 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:35:45.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:45 smithi031 ceph-mon[16336]: pgmap v1522: 47 pgs: 2 active+recovering+undersized+remapped, 6 active+recovering+undersized+degraded+remapped, 39 active+clean; 6.3 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 0 B/s wr, 324 op/s; 32074/310986 objects degraded (10.314%); 16445/310986 objects misplaced (5.288%); 9.4 MiB/s, 150 objects/s recovering 2024-08-19T14:35:45.649 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:45.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:45.650 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:35:45.751 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:35:45.803 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:45.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:45.804 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:35:45.905 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:35:45.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:45.942 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:45.943 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:35:46.044 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:35:46.068 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:46.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:46.069 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:35:46.170 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:35:46.222 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:46.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:46.223 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:35:46.324 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:35:46.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:46.350 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:46.351 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:35:46.451 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:35:46.473 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:46.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:46.475 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:35:46.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:46 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:35:46.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:46 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:35:46.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:46 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:35:46.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:46 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:35:46.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:46 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:35:46.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:46 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:35:46.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:46 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:35:46.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:46 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:35:46.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:46 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:35:46.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:46 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:35:46.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:46 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:35:46.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:46 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:35:46.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:46 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:35:46.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:46 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:35:46.576 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:35:46.598 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:46.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:46.599 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:35:46.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:46 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:35:46.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:46 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:35:46.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:46 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:35:46.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:46 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:35:46.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:46 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:35:46.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:46 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:35:46.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:46 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:35:46.700 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:35:46.722 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:46.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:46.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:35:46.824 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:35:46.848 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:46.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:46.849 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:35:46.950 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:35:46.986 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:46.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:46.987 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:35:47.088 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:35:47.135 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:47.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:47.136 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:35:47.237 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:35:47.283 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:47.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:47.285 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:35:47.385 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:35:47.408 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:47.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:47.409 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:35:47.510 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:35:47.541 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:47.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:47.542 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:35:47.643 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:35:47.688 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:47.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:47.689 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:35:47.790 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:35:47.834 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:47.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:47.836 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:35:47.937 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:35:47.965 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:47.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:47.966 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:35:48.067 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:35:48.090 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:48.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:48.091 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:35:48.192 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:35:48.245 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:48.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:48.246 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:35:48.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:47 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 32074/310986 objects degraded (10.314%), 6 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-19T14:35:48.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:47 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 32074/310986 objects degraded (10.314%), 6 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-19T14:35:48.346 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:35:48.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:47 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 32074/310986 objects degraded (10.314%), 6 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-19T14:35:48.370 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:48.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:48.371 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:35:48.472 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:35:48.497 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:48.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:48.498 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:35:48.601 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:35:48.625 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:48.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:48.626 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:35:48.727 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:35:48.748 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:48.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:48.749 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:35:48.850 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:35:48.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:48.876 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:48.876 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:35:48.977 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:35:49.032 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:49.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:49.033 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:35:49.134 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:35:49.156 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:49.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:49.158 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:35:49.258 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:35:49.283 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:49.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:49.284 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:35:49.385 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:35:49.406 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:49.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:49.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:35:49.508 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:35:49.531 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:49.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:49.532 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:35:49.633 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:35:49.655 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:49.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:49.656 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:35:49.757 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:35:49.779 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:49.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:49.780 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:35:49.880 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:35:49.924 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:49.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:49.925 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:35:50.026 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:35:50.047 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:50.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:50.048 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:35:50.149 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:35:50.172 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:50.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:50.173 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:35:50.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:49 smithi032 ceph-mon[20004]: pgmap v1524: 47 pgs: 2 active+recovering+undersized+remapped, 6 active+recovering+undersized+degraded+remapped, 39 active+clean; 6.2 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 0 B/s wr, 430 op/s; 31800/304872 objects degraded (10.431%); 15734/304872 objects misplaced (5.161%); 13 MiB/s, 208 objects/s recovering 2024-08-19T14:35:50.274 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:35:50.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:49 smithi045 ceph-mon[20143]: pgmap v1524: 47 pgs: 2 active+recovering+undersized+remapped, 6 active+recovering+undersized+degraded+remapped, 39 active+clean; 6.2 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 0 B/s wr, 430 op/s; 31800/304872 objects degraded (10.431%); 15734/304872 objects misplaced (5.161%); 13 MiB/s, 208 objects/s recovering 2024-08-19T14:35:50.297 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:50.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:50.298 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:35:50.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:49 smithi031 ceph-mon[16336]: pgmap v1524: 47 pgs: 2 active+recovering+undersized+remapped, 6 active+recovering+undersized+degraded+remapped, 39 active+clean; 6.2 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 0 B/s wr, 430 op/s; 31800/304872 objects degraded (10.431%); 15734/304872 objects misplaced (5.161%); 13 MiB/s, 208 objects/s recovering 2024-08-19T14:35:50.398 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:35:50.421 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:50.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:50.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:35:50.523 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:35:50.564 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:50.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:50.565 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:35:50.666 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:35:50.690 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:50.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:50.691 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:35:50.792 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:35:50.832 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:50.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:50.833 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:35:50.934 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:35:50.984 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:50.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:50.986 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:35:51.087 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:35:51.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:51.150 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:51.150 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:35:51.251 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:35:51.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:51 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:35:51.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:51 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:35:51.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:51.273 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:51.274 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:35:51.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:51 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:35:51.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:51 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:35:51.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:51 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:35:51.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:51 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:35:51.375 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:35:51.397 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:51.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:51.398 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:35:51.499 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:35:51.520 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:51.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:51.521 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:35:51.622 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:35:51.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:51.671 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:51.671 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:35:51.772 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:35:51.796 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:51.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:51.797 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:35:51.898 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:35:51.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:51.943 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:51.943 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:35:52.044 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:35:52.068 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:52.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:52.069 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:35:52.170 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:35:52.212 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:52.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:52.213 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:35:52.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:52 smithi032 ceph-mon[20004]: pgmap v1525: 47 pgs: 2 active+recovering+undersized+remapped, 6 active+recovering+undersized+degraded+remapped, 39 active+clean; 6.1 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 0 B/s wr, 409 op/s; 31869/301836 objects degraded (10.558%); 15421/301836 objects misplaced (5.109%); 12 MiB/s, 193 objects/s recovering 2024-08-19T14:35:52.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:52 smithi032 ceph-mon[20004]: osdmap e648: 12 total, 12 up, 10 in 2024-08-19T14:35:52.267 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:35:52.267 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-08-19T14:35:52.267 INFO:teuthology.orchestra.run.smithi031.stdout: "complaint_time": 30, 2024-08-19T14:35:52.267 INFO:teuthology.orchestra.run.smithi031.stdout: "num_blocked_ops": 0 2024-08-19T14:35:52.267 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:35:52.268 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:52 smithi031 ceph-mon[16336]: pgmap v1525: 47 pgs: 2 active+recovering+undersized+remapped, 6 active+recovering+undersized+degraded+remapped, 39 active+clean; 6.1 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 0 B/s wr, 409 op/s; 31869/301836 objects degraded (10.558%); 15421/301836 objects misplaced (5.109%); 12 MiB/s, 193 objects/s recovering 2024-08-19T14:35:52.268 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:52 smithi031 ceph-mon[16336]: osdmap e648: 12 total, 12 up, 10 in 2024-08-19T14:35:52.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:52 smithi045 ceph-mon[20143]: pgmap v1525: 47 pgs: 2 active+recovering+undersized+remapped, 6 active+recovering+undersized+degraded+remapped, 39 active+clean; 6.1 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 0 B/s wr, 409 op/s; 31869/301836 objects degraded (10.558%); 15421/301836 objects misplaced (5.109%); 12 MiB/s, 193 objects/s recovering 2024-08-19T14:35:52.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:52 smithi045 ceph-mon[20143]: osdmap e648: 12 total, 12 up, 10 in 2024-08-19T14:35:52.313 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:35:52.361 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:52.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:52.362 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:35:52.463 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:35:52.487 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:52.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:52.488 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:35:52.589 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:35:52.635 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:52.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:52.636 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:35:52.737 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:35:52.772 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:52.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:52.773 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:35:52.874 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:35:52.896 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:52.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:52.897 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:35:52.998 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:35:53.020 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:53.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:53.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:35:53.123 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:35:53.148 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:53.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:53.149 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:35:53.250 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:35:53.272 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:53.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:53.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:35:53.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:53 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 31869/301836 objects degraded (10.558%), 6 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-19T14:35:53.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:53 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 31869/301836 objects degraded (10.558%), 6 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-19T14:35:53.373 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:35:53.395 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:53.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:53.396 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:35:53.497 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:35:53.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:53 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 31869/301836 objects degraded (10.558%), 6 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-19T14:35:53.523 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:53.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:53.524 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:35:53.625 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:35:53.646 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:53.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:53.647 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:35:53.748 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:35:53.769 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:53.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:53.770 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:35:53.871 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:35:53.893 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:53.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:53.894 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:35:53.995 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:35:54.016 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:54.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:54.017 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:35:54.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:54 smithi031 ceph-mon[16336]: pgmap v1527: 47 pgs: 2 active+recovering+undersized+remapped, 6 active+recovering+undersized+degraded+remapped, 39 active+clean; 6.1 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 0 B/s wr, 414 op/s; 31449/298950 objects degraded (10.520%); 14939/298950 objects misplaced (4.997%); 13 MiB/s, 205 objects/s recovering 2024-08-19T14:35:54.118 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:35:54.143 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:54.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:54.145 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:35:54.246 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:35:54.268 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:54.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:54.269 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:35:54.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:54 smithi045 ceph-mon[20143]: pgmap v1527: 47 pgs: 2 active+recovering+undersized+remapped, 6 active+recovering+undersized+degraded+remapped, 39 active+clean; 6.1 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 0 B/s wr, 414 op/s; 31449/298950 objects degraded (10.520%); 14939/298950 objects misplaced (4.997%); 13 MiB/s, 205 objects/s recovering 2024-08-19T14:35:54.370 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:35:54.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:54.423 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:54.423 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:35:54.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:54 smithi032 ceph-mon[20004]: pgmap v1527: 47 pgs: 2 active+recovering+undersized+remapped, 6 active+recovering+undersized+degraded+remapped, 39 active+clean; 6.1 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 0 B/s wr, 414 op/s; 31449/298950 objects degraded (10.520%); 14939/298950 objects misplaced (4.997%); 13 MiB/s, 205 objects/s recovering 2024-08-19T14:35:54.524 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:35:54.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:54.570 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:54.570 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:35:54.671 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:35:54.692 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:54.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:54.693 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:35:54.794 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:35:54.815 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:54.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:54.816 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:35:54.917 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:35:54.939 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:54.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:54.940 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:35:55.042 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:35:55.064 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:55.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:55.065 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:35:55.166 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:35:55.189 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:55.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:55.190 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:35:55.291 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:35:55.330 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:55.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:55.332 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:35:55.433 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:35:55.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:55.458 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:55.458 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:35:55.559 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:35:55.611 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:55.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:55.612 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:35:55.713 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:35:55.755 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:55.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:55.756 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:35:55.857 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:35:55.902 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:55.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:55.903 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:35:56.004 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:35:56.052 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:56.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:56.053 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:35:56.153 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:35:56.176 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:56.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:56.177 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:35:56.278 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:35:56.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:56 smithi045 ceph-mon[20143]: pgmap v1528: 47 pgs: 2 active+recovering+undersized+remapped, 6 active+recovering+undersized+degraded+remapped, 39 active+clean; 6.1 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 0 B/s wr, 412 op/s; 31449/298590 objects degraded (10.533%); 14939/298590 objects misplaced (5.003%); 13 MiB/s, 205 objects/s recovering 2024-08-19T14:35:56.324 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:56.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:56.325 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:35:56.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:56 smithi031 ceph-mon[16336]: pgmap v1528: 47 pgs: 2 active+recovering+undersized+remapped, 6 active+recovering+undersized+degraded+remapped, 39 active+clean; 6.1 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 0 B/s wr, 412 op/s; 31449/298590 objects degraded (10.533%); 14939/298590 objects misplaced (5.003%); 13 MiB/s, 205 objects/s recovering 2024-08-19T14:35:56.426 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:35:56.481 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:56.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:56.482 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:35:56.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:56 smithi032 ceph-mon[20004]: pgmap v1528: 47 pgs: 2 active+recovering+undersized+remapped, 6 active+recovering+undersized+degraded+remapped, 39 active+clean; 6.1 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 0 B/s wr, 412 op/s; 31449/298590 objects degraded (10.533%); 14939/298590 objects misplaced (5.003%); 13 MiB/s, 205 objects/s recovering 2024-08-19T14:35:56.582 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:35:56.617 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:56.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:56.618 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:35:56.719 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:35:56.743 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:56.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:56.744 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:35:56.845 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:35:56.882 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:56.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:56.883 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:35:56.984 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:35:57.023 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:57.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:57.024 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:35:57.125 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:35:57.150 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:57.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:57.151 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:35:57.252 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:35:57.301 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:57.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:57.302 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:35:57.403 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:35:57.450 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:57.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:57.451 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:35:57.552 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:35:57.573 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:57.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:57.574 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:35:57.675 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:35:57.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:57.712 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:57.712 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:35:57.813 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:35:57.861 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:57.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:57.862 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:35:57.962 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:35:57.986 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:57.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:57.987 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:35:58.088 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:35:58.132 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:58.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:58.133 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:35:58.234 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:35:58.256 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:58.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:58.257 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:35:58.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:58 smithi045 ceph-mon[20143]: pgmap v1529: 47 pgs: 2 active+recovering+undersized+remapped, 6 active+recovering+undersized+degraded+remapped, 39 active+clean; 6.0 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 0 B/s wr, 402 op/s; 31343/295275 objects degraded (10.615%); 14598/295275 objects misplaced (4.944%); 13 MiB/s, 213 objects/s recovering 2024-08-19T14:35:58.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:35:58 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 31343/295275 objects degraded (10.615%), 6 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-19T14:35:58.358 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:35:58.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:58 smithi031 ceph-mon[16336]: pgmap v1529: 47 pgs: 2 active+recovering+undersized+remapped, 6 active+recovering+undersized+degraded+remapped, 39 active+clean; 6.0 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 0 B/s wr, 402 op/s; 31343/295275 objects degraded (10.615%); 14598/295275 objects misplaced (4.944%); 13 MiB/s, 213 objects/s recovering 2024-08-19T14:35:58.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:35:58 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 31343/295275 objects degraded (10.615%), 6 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-19T14:35:58.382 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:58.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:58.383 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:35:58.484 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:35:58.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:58 smithi032 ceph-mon[20004]: pgmap v1529: 47 pgs: 2 active+recovering+undersized+remapped, 6 active+recovering+undersized+degraded+remapped, 39 active+clean; 6.0 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 0 B/s wr, 402 op/s; 31343/295275 objects degraded (10.615%); 14598/295275 objects misplaced (4.944%); 13 MiB/s, 213 objects/s recovering 2024-08-19T14:35:58.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:35:58 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 31343/295275 objects degraded (10.615%), 6 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-19T14:35:58.508 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:58.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:58.509 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:35:58.610 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:35:58.652 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:58.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:58.653 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:35:58.754 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:35:58.776 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:58.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:58.777 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:35:58.877 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:35:58.901 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:58.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:58.902 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:35:58.948 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-08-19T14:35:59.003 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:35:59.025 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:59.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:59.026 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:35:59.127 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:35:59.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:59.195 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:59.196 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:35:59.296 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:35:59.353 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:59.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:59.354 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:35:59.455 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:35:59.479 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:59.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:59.480 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:35:59.580 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:35:59.602 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:35:59.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:59.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:35:59.638 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:35:59.703 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:35:59.725 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:35:59.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:59.726 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:35:59.827 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:35:59.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:35:59.891 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:35:59.892 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:35:59.992 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:36:00.051 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:00.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:00.052 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:36:00.153 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:36:00.207 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:00.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:00.208 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:36:00.256 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:00 smithi031 ceph-mon[16336]: pgmap v1530: 47 pgs: 2 active+recovering+undersized+remapped, 6 active+recovering+undersized+degraded+remapped, 39 active+clean; 6.0 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 0 B/s wr, 394 op/s; 31144/293013 objects degraded (10.629%); 14117/293013 objects misplaced (4.818%); 13 MiB/s, 217 objects/s recovering 2024-08-19T14:36:00.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:00 smithi045 ceph-mon[20143]: pgmap v1530: 47 pgs: 2 active+recovering+undersized+remapped, 6 active+recovering+undersized+degraded+remapped, 39 active+clean; 6.0 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 0 B/s wr, 394 op/s; 31144/293013 objects degraded (10.629%); 14117/293013 objects misplaced (4.818%); 13 MiB/s, 217 objects/s recovering 2024-08-19T14:36:00.309 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:36:00.330 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:00.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:00.331 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:36:00.432 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:36:00.453 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:00.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:00.455 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:36:00.461 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:36:00.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:00 smithi032 ceph-mon[20004]: pgmap v1530: 47 pgs: 2 active+recovering+undersized+remapped, 6 active+recovering+undersized+degraded+remapped, 39 active+clean; 6.0 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 0 B/s wr, 394 op/s; 31144/293013 objects degraded (10.629%); 14117/293013 objects misplaced (4.818%); 13 MiB/s, 217 objects/s recovering 2024-08-19T14:36:00.556 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:36:00.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:00.581 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:00.581 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:36:00.682 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:36:00.704 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:00.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:00.705 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:36:00.806 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:36:00.829 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:00.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:00.830 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:36:00.931 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:36:00.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:00.953 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:00.953 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:36:01.039 INFO:teuthology.orchestra.run.smithi031.stderr:reweighted osd.1 to 1 (10000) 2024-08-19T14:36:01.053 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:36:01.076 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:01.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:01.077 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:36:01.104 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:36:01.104 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 20, 2024-08-19T14:36:01.104 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 600, 2024-08-19T14:36:01.104 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [ 2024-08-19T14:36:01.104 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.104 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "MOSDPGPush(7.2 637/635 [PushOp(7:446a5b2b:::benchmark_data_smithi045_115842_object4789:head, version: 605'282, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:446a5b2b:::benchmark_data_smithi045_115842_object4789:head@605'282, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-19T14:36:01.104 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:35:13.931437+0000", 2024-08-19T14:36:01.104 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 47.170228481999999, 2024-08-19T14:36:01.105 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.123186772, 2024-08-19T14:36:01.105 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:36:01.105 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:36:01.105 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-08-19T14:36:01.105 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:36:01.105 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.105 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:36:01.105 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:13.931437+0000", 2024-08-19T14:36:01.105 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:36:01.105 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.105 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.105 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:36:01.105 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:13.931437+0000", 2024-08-19T14:36:01.105 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:36:01.106 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.106 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.106 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:36:01.106 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:13.931438+0000", 2024-08-19T14:36:01.106 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.2699999999999999e-07 2024-08-19T14:36:01.106 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.106 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.106 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:36:01.106 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:13.931468+0000", 2024-08-19T14:36:01.106 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.0803000000000003e-05 2024-08-19T14:36:01.106 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.106 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.107 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:36:01.107 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:13.931488+0000", 2024-08-19T14:36:01.107 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.9324999999999999e-05 2024-08-19T14:36:01.107 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.107 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.107 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:36:01.107 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:13.931503+0000", 2024-08-19T14:36:01.107 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4885000000000001e-05 2024-08-19T14:36:01.107 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.107 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.107 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:36:01.107 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:14.044073+0000", 2024-08-19T14:36:01.107 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.11257072999999999 2024-08-19T14:36:01.107 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.107 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.108 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:36:01.108 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:14.044086+0000", 2024-08-19T14:36:01.108 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2699e-05 2024-08-19T14:36:01.108 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.108 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.108 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:36:01.108 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:14.054624+0000", 2024-08-19T14:36:01.108 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.010537703000000001 2024-08-19T14:36:01.108 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:36:01.108 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:36:01.108 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:36:01.108 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.108 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.108 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.16962.0:122929 7.2 e637/635 7:4e5ba6d6:::benchmark_data_smithi045_115842_object4769:head v 637'7768, mlcod=605'279)", 2024-08-19T14:36:01.109 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:35:13.939583+0000", 2024-08-19T14:36:01.109 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 47.162082042000002, 2024-08-19T14:36:01.109 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.10441437100000001, 2024-08-19T14:36:01.109 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:36:01.109 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:36:01.109 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:36:01.109 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:36:01.109 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.109 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:36:01.109 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:13.939583+0000", 2024-08-19T14:36:01.109 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:36:01.109 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.109 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.110 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:36:01.110 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:13.939583+0000", 2024-08-19T14:36:01.110 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:36:01.110 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.110 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.110 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:36:01.110 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:13.939584+0000", 2024-08-19T14:36:01.110 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.7800000000000001e-07 2024-08-19T14:36:01.110 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.110 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.110 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:36:01.110 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:13.939587+0000", 2024-08-19T14:36:01.110 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.8240000000000001e-06 2024-08-19T14:36:01.111 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.111 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.111 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:36:01.111 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:13.939596+0000", 2024-08-19T14:36:01.111 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.3659999999999992e-06 2024-08-19T14:36:01.111 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.111 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.111 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:36:01.111 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:13.939601+0000", 2024-08-19T14:36:01.111 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.3870000000000003e-06 2024-08-19T14:36:01.111 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.111 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.111 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:36:01.111 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:13.983208+0000", 2024-08-19T14:36:01.111 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.043606374000000003 2024-08-19T14:36:01.112 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.112 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.112 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:36:01.112 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:13.983237+0000", 2024-08-19T14:36:01.112 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.9037e-05 2024-08-19T14:36:01.112 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.112 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.112 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:36:01.112 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:14.043908+0000", 2024-08-19T14:36:01.112 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.060671353999999997 2024-08-19T14:36:01.112 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.112 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.112 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:36:01.112 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:14.043998+0000", 2024-08-19T14:36:01.113 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.9851000000000004e-05 2024-08-19T14:36:01.113 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:36:01.113 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:36:01.113 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:36:01.113 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.113 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.113 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.16962.0:122921 7.a e637/635 7:5780a94f:::benchmark_data_smithi045_115842_object4761:head v 637'7674, mlcod=605'302)", 2024-08-19T14:36:01.113 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:35:13.943075+0000", 2024-08-19T14:36:01.113 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 47.158590429999997, 2024-08-19T14:36:01.113 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.107604699, 2024-08-19T14:36:01.113 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:36:01.113 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:36:01.113 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:36:01.113 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:36:01.114 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.114 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:36:01.114 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:13.943075+0000", 2024-08-19T14:36:01.114 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:36:01.114 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.114 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.114 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:36:01.114 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:13.943075+0000", 2024-08-19T14:36:01.114 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:36:01.114 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.114 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.114 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:36:01.114 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:13.943076+0000", 2024-08-19T14:36:01.114 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.0500000000000002e-07 2024-08-19T14:36:01.115 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.115 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.115 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:36:01.115 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:13.943087+0000", 2024-08-19T14:36:01.115 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0984e-05 2024-08-19T14:36:01.115 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.115 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.115 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:36:01.115 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:13.943101+0000", 2024-08-19T14:36:01.115 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4598e-05 2024-08-19T14:36:01.115 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.115 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.115 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:36:01.115 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:13.943109+0000", 2024-08-19T14:36:01.116 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.8990000000000001e-06 2024-08-19T14:36:01.116 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.116 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.116 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:36:01.116 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:13.989260+0000", 2024-08-19T14:36:01.116 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.046151169999999998 2024-08-19T14:36:01.116 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.116 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.116 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:36:01.116 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:13.989282+0000", 2024-08-19T14:36:01.116 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.1409000000000001e-05 2024-08-19T14:36:01.116 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.116 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.116 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:36:01.116 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:14.050556+0000", 2024-08-19T14:36:01.117 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.061274515000000002 2024-08-19T14:36:01.117 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.117 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.117 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:36:01.117 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:14.050680+0000", 2024-08-19T14:36:01.117 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012331899999999999 2024-08-19T14:36:01.117 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:36:01.117 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:36:01.118 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:36:01.118 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.118 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.118 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.16962.0:123821 7.8 e638/635 7:1b18f7d7:::benchmark_data_smithi045_115842_object5661:head v 638'7913, mlcod=605'371)", 2024-08-19T14:36:01.118 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:35:16.295891+0000", 2024-08-19T14:36:01.118 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 44.805774425999999, 2024-08-19T14:36:01.118 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.110645012, 2024-08-19T14:36:01.118 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:36:01.118 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:36:01.118 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:36:01.118 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:36:01.118 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.119 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:36:01.119 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:16.295891+0000", 2024-08-19T14:36:01.119 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:36:01.119 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.119 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.119 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:36:01.119 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:16.295891+0000", 2024-08-19T14:36:01.119 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:36:01.119 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.119 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.119 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:36:01.119 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:16.295892+0000", 2024-08-19T14:36:01.119 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1289999999999999e-06 2024-08-19T14:36:01.120 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.120 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.120 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:36:01.120 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:16.295909+0000", 2024-08-19T14:36:01.120 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6665000000000001e-05 2024-08-19T14:36:01.120 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.120 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.120 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:36:01.120 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:16.295926+0000", 2024-08-19T14:36:01.120 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6699999999999999e-05 2024-08-19T14:36:01.120 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.120 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.120 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:36:01.120 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:16.295934+0000", 2024-08-19T14:36:01.121 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.6340000000000007e-06 2024-08-19T14:36:01.121 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.121 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.121 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:36:01.121 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:16.314309+0000", 2024-08-19T14:36:01.121 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.018374867999999999 2024-08-19T14:36:01.121 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.121 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.121 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:36:01.121 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:16.314357+0000", 2024-08-19T14:36:01.121 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.8170000000000001e-05 2024-08-19T14:36:01.121 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.122 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.122 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:36:01.122 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:16.406400+0000", 2024-08-19T14:36:01.122 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.092043038999999993 2024-08-19T14:36:01.122 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.122 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.122 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:36:01.122 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:16.406536+0000", 2024-08-19T14:36:01.122 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000135807 2024-08-19T14:36:01.122 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:36:01.122 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:36:01.122 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:36:01.122 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.122 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.123 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.16962.0:123823 7.8 e638/635 7:1e29f844:::benchmark_data_smithi045_115842_object5663:head v 638'7914, mlcod=605'371)", 2024-08-19T14:36:01.123 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:35:16.297606+0000", 2024-08-19T14:36:01.123 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 44.804059115000001, 2024-08-19T14:36:01.123 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.109066961, 2024-08-19T14:36:01.123 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:36:01.123 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:36:01.123 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:36:01.123 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:36:01.123 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.123 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:36:01.123 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:16.297606+0000", 2024-08-19T14:36:01.123 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:36:01.123 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.123 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.124 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:36:01.124 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:16.297606+0000", 2024-08-19T14:36:01.124 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:36:01.124 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.124 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.124 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:36:01.124 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:16.297608+0000", 2024-08-19T14:36:01.124 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.576e-06 2024-08-19T14:36:01.124 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.124 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.124 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:36:01.124 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:16.297618+0000", 2024-08-19T14:36:01.124 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0477e-05 2024-08-19T14:36:01.124 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.124 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.125 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:36:01.125 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:16.297638+0000", 2024-08-19T14:36:01.125 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.9259e-05 2024-08-19T14:36:01.125 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.125 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.125 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:36:01.125 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:16.297648+0000", 2024-08-19T14:36:01.125 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.8689999999999999e-06 2024-08-19T14:36:01.125 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.125 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.125 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:36:01.125 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:16.351178+0000", 2024-08-19T14:36:01.125 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.053530419000000003 2024-08-19T14:36:01.125 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.125 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.126 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:36:01.126 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:16.351206+0000", 2024-08-19T14:36:01.126 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.7699e-05 2024-08-19T14:36:01.126 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.126 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.126 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:36:01.126 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:16.406540+0000", 2024-08-19T14:36:01.126 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.055334301000000002 2024-08-19T14:36:01.126 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.126 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.126 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:36:01.126 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:16.406673+0000", 2024-08-19T14:36:01.126 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000133361 2024-08-19T14:36:01.126 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:36:01.127 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:36:01.127 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:36:01.127 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.127 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.127 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "MOSDPGPush(7.2 647/635 [PushOp(7:4ddb795e:::benchmark_data_smithi045_115842_object15961:head, version: 607'1003, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:4ddb795e:::benchmark_data_smithi045_115842_object15961:head@607'1003, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-19T14:36:01.127 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:35:43.996002+0000", 2024-08-19T14:36:01.127 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 17.105663564, 2024-08-19T14:36:01.127 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.120268605, 2024-08-19T14:36:01.127 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:36:01.127 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:36:01.128 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-08-19T14:36:01.128 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:36:01.128 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.128 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:36:01.128 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:43.996002+0000", 2024-08-19T14:36:01.128 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:36:01.128 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.128 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.128 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:36:01.128 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:43.996002+0000", 2024-08-19T14:36:01.128 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:36:01.128 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.128 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.128 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:36:01.129 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:43.996002+0000", 2024-08-19T14:36:01.129 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.8999999999999998e-07 2024-08-19T14:36:01.129 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.129 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.129 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:36:01.129 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:43.996029+0000", 2024-08-19T14:36:01.129 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.6784e-05 2024-08-19T14:36:01.129 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.129 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.129 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:36:01.129 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:43.996042+0000", 2024-08-19T14:36:01.129 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3169999999999999e-05 2024-08-19T14:36:01.129 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.129 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.130 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:36:01.130 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:43.996055+0000", 2024-08-19T14:36:01.130 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3356000000000001e-05 2024-08-19T14:36:01.130 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.130 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.130 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:36:01.130 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:44.113472+0000", 2024-08-19T14:36:01.130 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.117416246 2024-08-19T14:36:01.130 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.130 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.130 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:36:01.130 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:44.113487+0000", 2024-08-19T14:36:01.130 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5539000000000001e-05 2024-08-19T14:36:01.130 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.130 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.131 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:36:01.131 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:44.116270+0000", 2024-08-19T14:36:01.131 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0027832199999999999 2024-08-19T14:36:01.131 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:36:01.131 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:36:01.131 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:36:01.131 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.131 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.131 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.16962.0:134128 7.2 e647/635 7:4c17645c:::benchmark_data_smithi045_115842_object15968:head v 647'8494, mlcod=607'1000)", 2024-08-19T14:36:01.131 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:35:44.003351+0000", 2024-08-19T14:36:01.131 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 17.098314755000001, 2024-08-19T14:36:01.131 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.113063552, 2024-08-19T14:36:01.131 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:36:01.131 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:36:01.132 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:36:01.132 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:36:01.132 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.132 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:36:01.132 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:44.003351+0000", 2024-08-19T14:36:01.132 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:36:01.132 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.132 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.132 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:36:01.132 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:44.003351+0000", 2024-08-19T14:36:01.132 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:36:01.132 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.132 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.132 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:36:01.133 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:44.003352+0000", 2024-08-19T14:36:01.133 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.314e-06 2024-08-19T14:36:01.133 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.133 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.133 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:36:01.133 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:44.003363+0000", 2024-08-19T14:36:01.133 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0614e-05 2024-08-19T14:36:01.133 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.133 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.133 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:36:01.133 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:44.003383+0000", 2024-08-19T14:36:01.133 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.0081e-05 2024-08-19T14:36:01.133 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.133 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.134 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:36:01.134 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:44.003392+0000", 2024-08-19T14:36:01.134 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.0550000000000005e-06 2024-08-19T14:36:01.134 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.134 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.134 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:36:01.134 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:44.011252+0000", 2024-08-19T14:36:01.134 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.007859909 2024-08-19T14:36:01.134 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.134 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.134 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:36:01.134 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:44.011330+0000", 2024-08-19T14:36:01.134 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.8187000000000002e-05 2024-08-19T14:36:01.134 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.135 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.135 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:36:01.135 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:44.116379+0000", 2024-08-19T14:36:01.135 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.10504947100000001 2024-08-19T14:36:01.135 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.135 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.135 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:36:01.135 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:44.116414+0000", 2024-08-19T14:36:01.135 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.4921e-05 2024-08-19T14:36:01.135 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:36:01.135 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:36:01.135 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:36:01.136 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.136 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.136 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.16962.0:134140 7.a e647/635 7:50a1f26a:::benchmark_data_smithi045_115842_object15980:head v 647'8383, mlcod=607'1011)", 2024-08-19T14:36:01.136 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:35:44.005600+0000", 2024-08-19T14:36:01.136 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 17.096065916000001, 2024-08-19T14:36:01.136 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.11077493200000001, 2024-08-19T14:36:01.136 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:36:01.136 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:36:01.136 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:36:01.136 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:36:01.136 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.136 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:36:01.136 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:44.005600+0000", 2024-08-19T14:36:01.136 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:36:01.136 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.137 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.137 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:36:01.137 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:44.005600+0000", 2024-08-19T14:36:01.137 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:36:01.137 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.137 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.137 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:36:01.137 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:44.005601+0000", 2024-08-19T14:36:01.137 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4789999999999999e-06 2024-08-19T14:36:01.137 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.137 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.137 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:36:01.137 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:44.005611+0000", 2024-08-19T14:36:01.137 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.7999999999999993e-06 2024-08-19T14:36:01.138 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.138 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.138 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:36:01.138 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:44.005630+0000", 2024-08-19T14:36:01.138 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.8952000000000001e-05 2024-08-19T14:36:01.138 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.138 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.138 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:36:01.138 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:44.005639+0000", 2024-08-19T14:36:01.138 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.2890000000000003e-06 2024-08-19T14:36:01.138 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.138 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.138 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:36:01.138 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:44.011371+0000", 2024-08-19T14:36:01.139 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0057315680000000003 2024-08-19T14:36:01.139 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.139 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.139 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:36:01.139 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:44.011422+0000", 2024-08-19T14:36:01.139 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0930000000000002e-05 2024-08-19T14:36:01.139 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.139 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.139 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:36:01.139 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:44.116140+0000", 2024-08-19T14:36:01.139 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.104718203 2024-08-19T14:36:01.139 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.139 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.139 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:36:01.139 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:44.116374+0000", 2024-08-19T14:36:01.140 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000234711 2024-08-19T14:36:01.140 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:36:01.140 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:36:01.140 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:36:01.140 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.140 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.140 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.16962.0:136829 7.e e647/634 7:757019e4:::benchmark_data_smithi045_115842_object18669:head v 647'8630, mlcod=607'1224)", 2024-08-19T14:36:01.140 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:35:50.772630+0000", 2024-08-19T14:36:01.140 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 10.329035770999999, 2024-08-19T14:36:01.140 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.112980367, 2024-08-19T14:36:01.140 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:36:01.140 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:36:01.140 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:36:01.140 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:36:01.140 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.140 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:36:01.141 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:50.772630+0000", 2024-08-19T14:36:01.141 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:36:01.141 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.141 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.141 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:36:01.141 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:50.772630+0000", 2024-08-19T14:36:01.141 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:36:01.141 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.141 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.141 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:36:01.141 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:50.772631+0000", 2024-08-19T14:36:01.141 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.6200000000000006e-07 2024-08-19T14:36:01.141 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.141 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.141 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:36:01.142 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:50.772638+0000", 2024-08-19T14:36:01.142 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.4279999999999999e-06 2024-08-19T14:36:01.142 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.142 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.142 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:36:01.142 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:50.772653+0000", 2024-08-19T14:36:01.142 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5231e-05 2024-08-19T14:36:01.142 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.142 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.142 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:36:01.142 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:50.772661+0000", 2024-08-19T14:36:01.142 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.9119999999999998e-06 2024-08-19T14:36:01.142 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.142 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.142 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:36:01.142 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:50.784557+0000", 2024-08-19T14:36:01.143 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.011895309 2024-08-19T14:36:01.143 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.143 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.143 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:36:01.143 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:50.784580+0000", 2024-08-19T14:36:01.143 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.3896999999999998e-05 2024-08-19T14:36:01.143 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.143 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.143 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:36:01.143 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:50.885567+0000", 2024-08-19T14:36:01.143 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.100986225 2024-08-19T14:36:01.143 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.143 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.143 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:36:01.144 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:50.885610+0000", 2024-08-19T14:36:01.144 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.3402999999999998e-05 2024-08-19T14:36:01.144 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:36:01.144 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:36:01.144 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:36:01.144 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.144 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.144 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.16962.0:136841 7.e e647/634 7:71a6068f:::benchmark_data_smithi045_115842_object18681:head v 647'8631, mlcod=607'1224)", 2024-08-19T14:36:01.144 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:35:50.781139+0000", 2024-08-19T14:36:01.144 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 10.320526499, 2024-08-19T14:36:01.144 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.104497056, 2024-08-19T14:36:01.144 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:36:01.144 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:36:01.145 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:36:01.145 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:36:01.145 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.145 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:36:01.145 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:50.781139+0000", 2024-08-19T14:36:01.145 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:36:01.145 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.146 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.146 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:36:01.146 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:50.781139+0000", 2024-08-19T14:36:01.146 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:36:01.146 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.146 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.146 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:36:01.146 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:50.781140+0000", 2024-08-19T14:36:01.146 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1799999999999999e-06 2024-08-19T14:36:01.146 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.146 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.146 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:36:01.147 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:50.781151+0000", 2024-08-19T14:36:01.147 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.11e-05 2024-08-19T14:36:01.147 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.147 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.147 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:36:01.147 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:50.781173+0000", 2024-08-19T14:36:01.147 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.1401999999999999e-05 2024-08-19T14:36:01.147 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.147 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.147 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:36:01.147 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:50.781183+0000", 2024-08-19T14:36:01.147 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.9669999999999996e-06 2024-08-19T14:36:01.147 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.147 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.147 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:36:01.148 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:50.801437+0000", 2024-08-19T14:36:01.148 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.020253976 2024-08-19T14:36:01.148 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.148 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.148 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:36:01.148 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:50.801471+0000", 2024-08-19T14:36:01.148 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.4007999999999997e-05 2024-08-19T14:36:01.148 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.148 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.148 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:36:01.148 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:50.885614+0000", 2024-08-19T14:36:01.148 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.084143404000000005 2024-08-19T14:36:01.148 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.148 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.148 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:36:01.149 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:50.885636+0000", 2024-08-19T14:36:01.149 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.2019e-05 2024-08-19T14:36:01.149 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:36:01.149 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:36:01.149 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:36:01.149 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.149 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.149 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.16962.0:137063 7.a e648/635 7:59f6116d:::benchmark_data_smithi045_115842_object18903:head v 648'8557, mlcod=607'1189)", 2024-08-19T14:36:01.149 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:35:51.485122+0000", 2024-08-19T14:36:01.149 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 9.6165429719999995, 2024-08-19T14:36:01.149 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.104178857, 2024-08-19T14:36:01.149 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:36:01.149 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:36:01.149 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:36:01.150 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:36:01.150 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.150 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:36:01.150 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:51.485122+0000", 2024-08-19T14:36:01.150 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:36:01.150 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.150 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.150 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:36:01.150 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:51.485122+0000", 2024-08-19T14:36:01.150 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:36:01.150 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.150 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.151 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:36:01.151 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:51.485124+0000", 2024-08-19T14:36:01.151 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2589999999999999e-06 2024-08-19T14:36:01.151 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.151 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.151 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:36:01.151 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:51.485133+0000", 2024-08-19T14:36:01.151 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.7210000000000006e-06 2024-08-19T14:36:01.151 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.151 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.151 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:36:01.151 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:51.485144+0000", 2024-08-19T14:36:01.151 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0349999999999999e-05 2024-08-19T14:36:01.151 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.151 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.152 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:36:01.152 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:51.485153+0000", 2024-08-19T14:36:01.152 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.0299999999999999e-06 2024-08-19T14:36:01.152 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.152 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.152 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:36:01.152 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:51.498076+0000", 2024-08-19T14:36:01.152 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.012923449 2024-08-19T14:36:01.152 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.152 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.152 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:36:01.152 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:51.498101+0000", 2024-08-19T14:36:01.152 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.4592000000000001e-05 2024-08-19T14:36:01.152 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.153 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.153 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:36:01.153 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:51.589107+0000", 2024-08-19T14:36:01.153 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.091006298999999999 2024-08-19T14:36:01.153 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.153 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.153 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:36:01.153 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:51.589301+0000", 2024-08-19T14:36:01.153 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000194157 2024-08-19T14:36:01.153 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:36:01.153 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:36:01.153 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:36:01.153 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.153 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.153 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.16962.0:137064 7.a e648/635 7:53f1c25c:::benchmark_data_smithi045_115842_object18904:head v 648'8558, mlcod=607'1189)", 2024-08-19T14:36:01.153 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:35:51.488764+0000", 2024-08-19T14:36:01.154 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 9.6129018310000003, 2024-08-19T14:36:01.154 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.119202405, 2024-08-19T14:36:01.154 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:36:01.154 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:36:01.154 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:36:01.154 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:36:01.154 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.154 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:36:01.154 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:51.488764+0000", 2024-08-19T14:36:01.154 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:36:01.154 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.154 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.154 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:36:01.155 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:51.488764+0000", 2024-08-19T14:36:01.155 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:36:01.155 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.155 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.155 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:36:01.155 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:51.488764+0000", 2024-08-19T14:36:01.155 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.4600000000000004e-07 2024-08-19T14:36:01.155 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.155 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.155 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:36:01.155 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:51.488770+0000", 2024-08-19T14:36:01.155 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.1829999999999999e-06 2024-08-19T14:36:01.155 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.155 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.155 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:36:01.156 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:51.488781+0000", 2024-08-19T14:36:01.156 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0793e-05 2024-08-19T14:36:01.156 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.156 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.156 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:36:01.156 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:51.488787+0000", 2024-08-19T14:36:01.156 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.8499999999999999e-06 2024-08-19T14:36:01.156 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.156 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.156 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:36:01.156 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:51.508610+0000", 2024-08-19T14:36:01.156 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.019823090000000002 2024-08-19T14:36:01.156 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.156 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.156 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:36:01.156 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:51.508637+0000", 2024-08-19T14:36:01.157 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.6689000000000001e-05 2024-08-19T14:36:01.157 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.157 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.157 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:36:01.157 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:51.607871+0000", 2024-08-19T14:36:01.157 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.099234316000000003 2024-08-19T14:36:01.157 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.157 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.157 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:36:01.157 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:51.607966+0000", 2024-08-19T14:36:01.157 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.4838000000000002e-05 2024-08-19T14:36:01.157 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:36:01.157 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:36:01.157 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:36:01.157 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.158 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.158 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.16962.0:137055 7.2 e648/635 7:4d1c5eb2:::benchmark_data_smithi045_115842_object18895:head v 648'8675, mlcod=607'1190)", 2024-08-19T14:36:01.158 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:35:51.497890+0000", 2024-08-19T14:36:01.158 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 9.6037755409999992, 2024-08-19T14:36:01.158 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.106792869, 2024-08-19T14:36:01.158 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:36:01.158 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:36:01.158 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:36:01.158 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:36:01.158 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.158 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:36:01.158 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:51.497890+0000", 2024-08-19T14:36:01.158 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:36:01.158 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.158 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.159 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:36:01.159 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:51.497890+0000", 2024-08-19T14:36:01.159 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:36:01.159 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.159 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.159 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:36:01.159 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:51.497891+0000", 2024-08-19T14:36:01.159 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.372e-06 2024-08-19T14:36:01.159 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.159 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.159 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:36:01.159 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:51.497902+0000", 2024-08-19T14:36:01.159 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1005999999999999e-05 2024-08-19T14:36:01.159 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.159 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.160 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:36:01.160 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:51.497921+0000", 2024-08-19T14:36:01.160 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.8722000000000001e-05 2024-08-19T14:36:01.160 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.160 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.160 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:36:01.160 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:51.497931+0000", 2024-08-19T14:36:01.160 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.8300000000000008e-06 2024-08-19T14:36:01.160 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.160 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.160 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:36:01.160 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:51.508576+0000", 2024-08-19T14:36:01.160 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.010645393 2024-08-19T14:36:01.160 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.161 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.161 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:36:01.161 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:51.508603+0000", 2024-08-19T14:36:01.161 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.6359e-05 2024-08-19T14:36:01.161 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.161 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.161 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:36:01.161 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:51.604629+0000", 2024-08-19T14:36:01.161 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.096026160999999999 2024-08-19T14:36:01.161 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.161 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.161 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:36:01.161 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:51.604683+0000", 2024-08-19T14:36:01.162 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.4026000000000001e-05 2024-08-19T14:36:01.162 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:36:01.162 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:36:01.162 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:36:01.162 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.162 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.162 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.16962.0:137068 7.a e648/635 7:5729ffc0:::benchmark_data_smithi045_115842_object18908:head v 648'8559, mlcod=607'1191)", 2024-08-19T14:36:01.162 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:35:51.512114+0000", 2024-08-19T14:36:01.162 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 9.5895512780000001, 2024-08-19T14:36:01.162 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.11970992799999999, 2024-08-19T14:36:01.162 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:36:01.162 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:36:01.162 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:36:01.162 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:36:01.163 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.163 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:36:01.163 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:51.512114+0000", 2024-08-19T14:36:01.163 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:36:01.163 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.163 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.163 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:36:01.163 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:51.512114+0000", 2024-08-19T14:36:01.163 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:36:01.163 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.163 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.163 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:36:01.163 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:51.512115+0000", 2024-08-19T14:36:01.163 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.0599999999999999e-07 2024-08-19T14:36:01.164 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.164 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.164 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:36:01.164 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:51.512122+0000", 2024-08-19T14:36:01.164 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.5889999999999999e-06 2024-08-19T14:36:01.164 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.164 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.164 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:36:01.164 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:51.512140+0000", 2024-08-19T14:36:01.164 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.8400999999999999e-05 2024-08-19T14:36:01.164 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.164 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.164 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:36:01.165 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:51.512149+0000", 2024-08-19T14:36:01.165 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.6309999999999992e-06 2024-08-19T14:36:01.165 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.165 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.165 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:36:01.165 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:51.527846+0000", 2024-08-19T14:36:01.165 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.015697582000000002 2024-08-19T14:36:01.165 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.165 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.165 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:36:01.165 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:51.527885+0000", 2024-08-19T14:36:01.165 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.8705999999999998e-05 2024-08-19T14:36:01.165 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.165 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.166 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:36:01.166 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:51.631621+0000", 2024-08-19T14:36:01.166 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.103735647 2024-08-19T14:36:01.166 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.166 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.166 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:36:01.166 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:51.631824+0000", 2024-08-19T14:36:01.166 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000203566 2024-08-19T14:36:01.166 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:36:01.166 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:36:01.166 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:36:01.166 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.166 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.166 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.16962.0:137071 7.a e648/635 7:5b20c432:::benchmark_data_smithi045_115842_object18911:head v 648'8560, mlcod=607'1191)", 2024-08-19T14:36:01.167 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:35:51.515524+0000", 2024-08-19T14:36:01.167 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 9.5861411689999994, 2024-08-19T14:36:01.167 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.116498632, 2024-08-19T14:36:01.167 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:36:01.167 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:36:01.167 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:36:01.167 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:36:01.167 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.167 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:36:01.167 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:51.515524+0000", 2024-08-19T14:36:01.167 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:36:01.167 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.167 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.167 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:36:01.167 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:51.515524+0000", 2024-08-19T14:36:01.167 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:36:01.168 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.168 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.168 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:36:01.168 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:51.515526+0000", 2024-08-19T14:36:01.168 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.302e-06 2024-08-19T14:36:01.168 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.168 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.168 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:36:01.168 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:51.515540+0000", 2024-08-19T14:36:01.168 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4447000000000001e-05 2024-08-19T14:36:01.168 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.168 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.168 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:36:01.168 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:51.515558+0000", 2024-08-19T14:36:01.168 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.7951000000000001e-05 2024-08-19T14:36:01.169 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.169 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.169 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:36:01.169 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:51.515568+0000", 2024-08-19T14:36:01.169 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0453e-05 2024-08-19T14:36:01.169 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.169 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.169 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:36:01.169 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:51.540515+0000", 2024-08-19T14:36:01.169 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.024946578000000001 2024-08-19T14:36:01.169 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.169 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.169 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:36:01.169 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:51.540541+0000", 2024-08-19T14:36:01.169 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.6208e-05 2024-08-19T14:36:01.169 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.170 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.170 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:36:01.170 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:51.631828+0000", 2024-08-19T14:36:01.170 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.091286913999999997 2024-08-19T14:36:01.170 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.170 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.170 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:36:01.170 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:51.632023+0000", 2024-08-19T14:36:01.170 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00019477900000000001 2024-08-19T14:36:01.170 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:36:01.170 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:36:01.170 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:36:01.170 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.170 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.171 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.16962.0:137076 7.a e648/635 7:5b119465:::benchmark_data_smithi045_115842_object18916:head v 648'8561, mlcod=607'1191)", 2024-08-19T14:36:01.171 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:35:51.522030+0000", 2024-08-19T14:36:01.171 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 9.5796352729999992, 2024-08-19T14:36:01.171 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.110144239, 2024-08-19T14:36:01.171 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:36:01.171 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:36:01.171 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:36:01.171 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:36:01.171 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.171 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:36:01.171 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:51.522030+0000", 2024-08-19T14:36:01.171 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:36:01.171 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.171 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.171 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:36:01.171 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:51.522030+0000", 2024-08-19T14:36:01.172 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:36:01.172 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.172 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.172 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:36:01.172 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:51.522032+0000", 2024-08-19T14:36:01.172 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3969999999999999e-06 2024-08-19T14:36:01.172 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.172 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.172 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:36:01.172 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:51.522044+0000", 2024-08-19T14:36:01.172 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2279999999999999e-05 2024-08-19T14:36:01.172 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.172 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.172 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:36:01.172 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:51.522061+0000", 2024-08-19T14:36:01.173 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6909e-05 2024-08-19T14:36:01.173 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.173 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.173 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:36:01.173 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:51.522071+0000", 2024-08-19T14:36:01.173 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0244999999999999e-05 2024-08-19T14:36:01.173 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.173 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.173 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:36:01.173 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:51.550271+0000", 2024-08-19T14:36:01.173 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.028199933 2024-08-19T14:36:01.173 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.173 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.173 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:36:01.173 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:51.550301+0000", 2024-08-19T14:36:01.173 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.0150000000000001e-05 2024-08-19T14:36:01.174 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.174 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.174 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:36:01.174 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:51.632026+0000", 2024-08-19T14:36:01.174 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.081725192000000002 2024-08-19T14:36:01.174 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.174 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.174 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:36:01.174 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:51.632174+0000", 2024-08-19T14:36:01.174 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000148133 2024-08-19T14:36:01.174 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:36:01.174 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:36:01.174 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:36:01.174 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.175 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.175 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "MOSDPGPush(7.a 648/635 [PushOp(7:54af1818:::benchmark_data_smithi045_115842_object61375:head, version: 613'3830, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:54af1818:::benchmark_data_smithi045_115842_object61375:head@613'3830, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-19T14:36:01.175 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:35:58.178236+0000", 2024-08-19T14:36:01.175 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 2.9234294219999999, 2024-08-19T14:36:01.175 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.122447124, 2024-08-19T14:36:01.175 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:36:01.175 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:36:01.175 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-08-19T14:36:01.175 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:36:01.175 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.175 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:36:01.175 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:58.178236+0000", 2024-08-19T14:36:01.175 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:36:01.175 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.175 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.176 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:36:01.176 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:58.178236+0000", 2024-08-19T14:36:01.176 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:36:01.176 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.176 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.176 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:36:01.176 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:58.178237+0000", 2024-08-19T14:36:01.176 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1170000000000001e-06 2024-08-19T14:36:01.176 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.176 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.176 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:36:01.176 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:58.178292+0000", 2024-08-19T14:36:01.176 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.5192999999999998e-05 2024-08-19T14:36:01.176 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.176 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.176 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:36:01.177 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:58.178316+0000", 2024-08-19T14:36:01.177 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.3456999999999999e-05 2024-08-19T14:36:01.177 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.177 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.177 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:36:01.177 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:58.178336+0000", 2024-08-19T14:36:01.177 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.9983000000000002e-05 2024-08-19T14:36:01.177 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.177 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.177 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:36:01.177 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:58.297708+0000", 2024-08-19T14:36:01.177 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.119372505 2024-08-19T14:36:01.177 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.177 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.177 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:36:01.177 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:58.297723+0000", 2024-08-19T14:36:01.178 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5116999999999999e-05 2024-08-19T14:36:01.178 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.178 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.178 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:36:01.178 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:58.300683+0000", 2024-08-19T14:36:01.178 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.002959752 2024-08-19T14:36:01.178 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:36:01.178 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:36:01.178 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:36:01.178 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.178 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.178 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "MOSDPGPush(7.2 648/635 [PushOp(7:4c6c57d1:::benchmark_data_smithi045_115842_object21599:head, version: 608'1372, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:4c6c57d1:::benchmark_data_smithi045_115842_object21599:head@608'1372, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-19T14:36:01.178 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:35:58.220346+0000", 2024-08-19T14:36:01.179 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 2.8813191649999998, 2024-08-19T14:36:01.179 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.114320532, 2024-08-19T14:36:01.179 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:36:01.179 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:36:01.179 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-08-19T14:36:01.179 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:36:01.179 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.179 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:36:01.179 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:58.220346+0000", 2024-08-19T14:36:01.179 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:36:01.179 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.179 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.179 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:36:01.179 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:58.220346+0000", 2024-08-19T14:36:01.179 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:36:01.180 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.180 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.180 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:36:01.180 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:58.220348+0000", 2024-08-19T14:36:01.180 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.7269999999999999e-06 2024-08-19T14:36:01.180 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.180 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.180 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:36:01.180 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:58.220387+0000", 2024-08-19T14:36:01.180 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.8841999999999998e-05 2024-08-19T14:36:01.180 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.180 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.180 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:36:01.180 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:58.220418+0000", 2024-08-19T14:36:01.180 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.1155999999999998e-05 2024-08-19T14:36:01.181 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.181 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.181 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:36:01.181 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:58.220437+0000", 2024-08-19T14:36:01.181 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.8995999999999999e-05 2024-08-19T14:36:01.181 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.181 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.181 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:36:01.181 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:58.331235+0000", 2024-08-19T14:36:01.181 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.110798072 2024-08-19T14:36:01.181 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.181 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.181 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:36:01.181 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:58.331247+0000", 2024-08-19T14:36:01.181 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1958e-05 2024-08-19T14:36:01.182 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.182 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.182 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:36:01.182 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:58.334667+0000", 2024-08-19T14:36:01.182 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0034197810000000002 2024-08-19T14:36:01.182 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:36:01.182 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:36:01.182 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:36:01.182 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.182 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.182 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "MOSDPGPush(7.2 648/635 [PushOp(7:4351e647:::benchmark_data_smithi045_115842_object22022:head, version: 608'1408, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:4351e647:::benchmark_data_smithi045_115842_object22022:head@608'1408, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-19T14:36:01.182 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:36:01.183 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:35:59.380521+0000", 2024-08-19T14:36:01.183 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 1.721144644, 2024-08-19T14:36:01.183 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.104747547, 2024-08-19T14:36:01.183 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:36:01.184 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:36:01.184 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-08-19T14:36:01.184 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:36:01.184 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.184 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:36:01.184 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:59.380521+0000", 2024-08-19T14:36:01.184 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:36:01.184 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.184 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.184 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:36:01.184 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:59.380521+0000", 2024-08-19T14:36:01.184 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:36:01.184 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.184 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.184 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:36:01.185 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:59.380522+0000", 2024-08-19T14:36:01.185 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.7400000000000002e-07 2024-08-19T14:36:01.185 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.185 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.185 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:36:01.185 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:59.380557+0000", 2024-08-19T14:36:01.185 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.5021999999999998e-05 2024-08-19T14:36:01.185 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.185 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.185 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:36:01.185 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:59.380580+0000", 2024-08-19T14:36:01.185 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.3473e-05 2024-08-19T14:36:01.185 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.185 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.185 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:36:01.186 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:59.380600+0000", 2024-08-19T14:36:01.186 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.9445000000000001e-05 2024-08-19T14:36:01.186 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.186 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.186 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:36:01.186 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:59.482055+0000", 2024-08-19T14:36:01.186 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.10145488 2024-08-19T14:36:01.186 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.186 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.186 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:36:01.186 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:59.482067+0000", 2024-08-19T14:36:01.186 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2561000000000001e-05 2024-08-19T14:36:01.186 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.186 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.186 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:36:01.187 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:59.485268+0000", 2024-08-19T14:36:01.187 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0032012920000000001 2024-08-19T14:36:01.187 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:36:01.187 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:36:01.187 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:36:01.187 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.187 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.187 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "MOSDPGPush(7.a 648/635 [PushOp(7:506419e4:::benchmark_data_smithi045_115842_object61743:head, version: 613'3857, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:506419e4:::benchmark_data_smithi045_115842_object61743:head@613'3857, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-19T14:36:01.187 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:35:59.394012+0000", 2024-08-19T14:36:01.187 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 1.7076532289999999, 2024-08-19T14:36:01.187 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.12366187200000001, 2024-08-19T14:36:01.187 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:36:01.187 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:36:01.188 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-08-19T14:36:01.188 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:36:01.188 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.188 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:36:01.188 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:59.394012+0000", 2024-08-19T14:36:01.188 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:36:01.188 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.188 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.188 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:36:01.188 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:59.394012+0000", 2024-08-19T14:36:01.188 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:36:01.188 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.188 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.188 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:36:01.188 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:59.394013+0000", 2024-08-19T14:36:01.189 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2580000000000001e-06 2024-08-19T14:36:01.189 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.189 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.189 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:36:01.189 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:59.394064+0000", 2024-08-19T14:36:01.189 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0339e-05 2024-08-19T14:36:01.189 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.189 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.189 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:36:01.189 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:59.394093+0000", 2024-08-19T14:36:01.189 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.9391999999999999e-05 2024-08-19T14:36:01.189 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.189 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.189 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:36:01.189 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:59.394113+0000", 2024-08-19T14:36:01.190 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.0186999999999998e-05 2024-08-19T14:36:01.190 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.190 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.190 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:36:01.190 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:59.514967+0000", 2024-08-19T14:36:01.190 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.12085404399999999 2024-08-19T14:36:01.190 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.190 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.190 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:36:01.190 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:59.514980+0000", 2024-08-19T14:36:01.190 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2432000000000001e-05 2024-08-19T14:36:01.190 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:01.190 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:01.190 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:36:01.190 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:59.517674+0000", 2024-08-19T14:36:01.191 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0026942200000000002 2024-08-19T14:36:01.191 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:36:01.191 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:36:01.191 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:36:01.191 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:36:01.191 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:36:01.191 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:36:01.204 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:01.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:01.206 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:36:01.206 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-19T14:36:01.256 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:01 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/436082392' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-08-19T14:36:01.256 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:01 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-08-19T14:36:01.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:01 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/436082392' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-08-19T14:36:01.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:01 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-08-19T14:36:01.306 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:36:01.329 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:01.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:01.330 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:36:01.431 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:36:01.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:01.454 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:01.454 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:36:01.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:01 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/436082392' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-08-19T14:36:01.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:01 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-08-19T14:36:01.555 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:36:01.577 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:01.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:01.578 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:36:01.679 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:36:01.687 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:36:01.705 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:01.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:01.706 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:36:01.807 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:36:01.830 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:01.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:01.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:36:01.932 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:36:01.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:01.981 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:01.981 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:36:02.082 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:36:02.104 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:02.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:02.106 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:36:02.206 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:36:02.232 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:02.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:02.232 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:36:02.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:02 smithi045 ceph-mon[20143]: pgmap v1531: 47 pgs: 2 active+recovering+undersized+remapped, 6 active+recovering+undersized+degraded+remapped, 39 active+clean; 5.9 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 0 B/s wr, 390 op/s; 31201/290106 objects degraded (10.755%); 13772/290106 objects misplaced (4.747%); 14 MiB/s, 220 objects/s recovering 2024-08-19T14:36:02.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:02 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-08-19T14:36:02.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:02 smithi045 ceph-mon[20143]: osdmap e649: 12 total, 12 up, 10 in 2024-08-19T14:36:02.333 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:36:02.356 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:02.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:02.357 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:36:02.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:02 smithi031 ceph-mon[16336]: pgmap v1531: 47 pgs: 2 active+recovering+undersized+remapped, 6 active+recovering+undersized+degraded+remapped, 39 active+clean; 5.9 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 0 B/s wr, 390 op/s; 31201/290106 objects degraded (10.755%); 13772/290106 objects misplaced (4.747%); 14 MiB/s, 220 objects/s recovering 2024-08-19T14:36:02.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:02 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-08-19T14:36:02.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:02 smithi031 ceph-mon[16336]: osdmap e649: 12 total, 12 up, 10 in 2024-08-19T14:36:02.457 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:36:02.474 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-08-19T14:36:02.479 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:02.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:02.480 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:36:02.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:02 smithi032 ceph-mon[20004]: pgmap v1531: 47 pgs: 2 active+recovering+undersized+remapped, 6 active+recovering+undersized+degraded+remapped, 39 active+clean; 5.9 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 0 B/s wr, 390 op/s; 31201/290106 objects degraded (10.755%); 13772/290106 objects misplaced (4.747%); 14 MiB/s, 220 objects/s recovering 2024-08-19T14:36:02.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:02 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-08-19T14:36:02.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:02 smithi032 ceph-mon[20004]: osdmap e649: 12 total, 12 up, 10 in 2024-08-19T14:36:02.581 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:36:02.602 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:02.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:02.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:36:02.704 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:36:02.727 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:02.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:02.729 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:36:02.829 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:36:02.851 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:02.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:02.852 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:36:02.953 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:36:02.976 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:02.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:02.977 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:36:03.012 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd reweight 3 1 2024-08-19T14:36:03.078 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:36:03.099 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:03.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:03.100 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:36:03.118 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:36:03.125 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:36:03.201 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:36:03.280 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:03 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:36:03.281 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:03 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:36:03.281 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:03 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 31201/290106 objects degraded (10.755%), 6 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-19T14:36:03.285 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:03.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:03.286 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:36:03.387 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:36:03.408 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:03.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:03.409 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:36:03.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:03 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:36:03.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:03 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:36:03.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:03 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 31201/290106 objects degraded (10.755%), 6 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-19T14:36:03.510 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:36:03.533 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:03.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:03.534 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:36:03.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:03 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:36:03.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:03 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:36:03.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:03 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 31201/290106 objects degraded (10.755%), 6 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-19T14:36:03.635 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:36:03.657 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:03.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:03.658 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:36:03.759 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:36:03.782 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:03.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:03.783 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:36:03.884 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:36:03.931 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:03.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:03.931 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:36:04.032 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:36:04.055 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:04.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:04.056 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:36:04.157 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:36:04.190 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:04.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:04.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:36:04.210 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:04 smithi031 ceph-mon[16336]: pgmap v1533: 47 pgs: 2 active+recovering+undersized+remapped, 6 active+recovering+undersized+degraded+remapped, 39 active+clean; 5.9 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 0 B/s wr, 343 op/s; 31045/288639 objects degraded (10.756%); 13555/288639 objects misplaced (4.696%); 11 MiB/s, 169 objects/s recovering 2024-08-19T14:36:04.210 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:04 smithi031 ceph-mon[16336]: osdmap e650: 12 total, 12 up, 10 in 2024-08-19T14:36:04.292 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:36:04.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:04 smithi045 ceph-mon[20143]: pgmap v1533: 47 pgs: 2 active+recovering+undersized+remapped, 6 active+recovering+undersized+degraded+remapped, 39 active+clean; 5.9 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 0 B/s wr, 343 op/s; 31045/288639 objects degraded (10.756%); 13555/288639 objects misplaced (4.696%); 11 MiB/s, 169 objects/s recovering 2024-08-19T14:36:04.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:04 smithi045 ceph-mon[20143]: osdmap e650: 12 total, 12 up, 10 in 2024-08-19T14:36:04.334 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:04.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:04.335 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:36:04.436 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:36:04.484 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:04.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:04.485 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:36:04.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:04 smithi032 ceph-mon[20004]: pgmap v1533: 47 pgs: 2 active+recovering+undersized+remapped, 6 active+recovering+undersized+degraded+remapped, 39 active+clean; 5.9 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 0 B/s wr, 343 op/s; 31045/288639 objects degraded (10.756%); 13555/288639 objects misplaced (4.696%); 11 MiB/s, 169 objects/s recovering 2024-08-19T14:36:04.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:04 smithi032 ceph-mon[20004]: osdmap e650: 12 total, 12 up, 10 in 2024-08-19T14:36:04.563 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:36:04.564 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:36:04.564 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:36:04.564 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:36:04.564 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:36:04.564 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:36:04.564 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:36:04.564 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:36:04.564 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:36:04.564 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:36:04.564 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:36:04.565 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:36:04.565 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:36:04.565 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:36:04.565 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:36:04.565 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:36:04.565 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:36:04.565 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:36:04.565 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:36:04.565 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:36:04.565 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:36:04.565 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:36:04.581 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:36:04.581 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:36:04.585 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:36:04.610 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:04.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:04.611 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:36:04.711 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:36:04.733 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:04.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:04.734 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:36:04.835 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:36:04.872 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:04.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:04.873 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:36:04.974 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:36:05.009 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:05.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:05.010 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:36:05.110 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:36:05.133 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:05.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:05.135 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:36:05.236 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:36:05.260 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:05.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:05.261 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:36:05.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:05 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2734109124' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:36:05.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:05 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:36:05.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:05 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:36:05.362 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:36:05.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:05 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2734109124' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:36:05.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:05 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:36:05.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:05 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:36:05.383 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:05.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:05.384 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:36:05.485 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:36:05.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:05 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2734109124' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:36:05.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:05 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:36:05.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:05 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:36:05.509 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:05.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:05.510 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:36:05.572 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:36:05.572 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:36:05.611 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:36:05.635 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:05.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:05.637 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:36:05.738 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:36:05.777 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:05.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:05.778 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:36:05.879 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:36:05.901 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:05.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:05.902 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:36:06.003 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:36:06.025 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:06.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:06.026 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:36:06.127 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:36:06.148 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:06.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:06.149 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:36:06.250 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:36:06.272 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:06.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:06.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:36:06.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:06 smithi031 ceph-mon[16336]: pgmap v1535: 47 pgs: 2 active+recovering+undersized+remapped, 6 active+recovering+undersized+degraded+remapped, 39 active+clean; 5.8 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 0 B/s wr, 358 op/s; 30842/286662 objects degraded (10.759%); 13267/286662 objects misplaced (4.628%); 13 MiB/s, 210 objects/s recovering 2024-08-19T14:36:06.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:06 smithi031 ceph-mon[16336]: osdmap e651: 12 total, 12 up, 10 in 2024-08-19T14:36:06.374 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:36:06.396 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:06.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:06.397 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:36:06.498 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:36:06.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:06 smithi032 ceph-mon[20004]: pgmap v1535: 47 pgs: 2 active+recovering+undersized+remapped, 6 active+recovering+undersized+degraded+remapped, 39 active+clean; 5.8 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 0 B/s wr, 358 op/s; 30842/286662 objects degraded (10.759%); 13267/286662 objects misplaced (4.628%); 13 MiB/s, 210 objects/s recovering 2024-08-19T14:36:06.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:06 smithi032 ceph-mon[20004]: osdmap e651: 12 total, 12 up, 10 in 2024-08-19T14:36:06.521 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:06.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:06.522 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:36:06.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:06 smithi045 ceph-mon[20143]: pgmap v1535: 47 pgs: 2 active+recovering+undersized+remapped, 6 active+recovering+undersized+degraded+remapped, 39 active+clean; 5.8 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 0 B/s wr, 358 op/s; 30842/286662 objects degraded (10.759%); 13267/286662 objects misplaced (4.628%); 13 MiB/s, 210 objects/s recovering 2024-08-19T14:36:06.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:06 smithi045 ceph-mon[20143]: osdmap e651: 12 total, 12 up, 10 in 2024-08-19T14:36:06.623 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:36:06.646 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:06.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:06.647 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:36:06.748 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:36:06.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:06.770 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:06.770 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:36:06.871 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:36:06.893 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:06.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:06.894 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:36:06.995 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:36:07.017 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:07.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:07.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:36:07.119 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:36:07.154 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:07.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:07.155 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:36:07.256 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:36:07.278 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:07.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:07.279 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:36:07.380 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:36:07.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:07.404 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:07.404 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:36:07.504 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:36:07.527 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:07.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:07.528 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:36:07.629 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:36:07.671 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:07.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:07.672 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:36:07.773 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:36:07.797 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:07.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:07.798 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:36:07.899 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:36:07.937 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:07.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:07.938 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:36:08.039 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:36:08.061 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:08.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:08.062 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:36:08.163 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:36:08.184 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:08.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:08.185 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:36:08.286 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:36:08.308 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:08.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:08.309 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:36:08.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:08 smithi031 ceph-mon[16336]: pgmap v1537: 47 pgs: 2 active+recovering+undersized+remapped, 6 active+recovering+undersized+degraded+remapped, 39 active+clean; 5.8 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 0 B/s wr, 373 op/s; 30752/283380 objects degraded (10.852%); 12919/283380 objects misplaced (4.559%); 13 MiB/s, 210 objects/s recovering 2024-08-19T14:36:08.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:08 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 30752/283380 objects degraded (10.852%), 6 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-19T14:36:08.410 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:36:08.432 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:08.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:08.433 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:36:08.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:08 smithi032 ceph-mon[20004]: pgmap v1537: 47 pgs: 2 active+recovering+undersized+remapped, 6 active+recovering+undersized+degraded+remapped, 39 active+clean; 5.8 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 0 B/s wr, 373 op/s; 30752/283380 objects degraded (10.852%); 12919/283380 objects misplaced (4.559%); 13 MiB/s, 210 objects/s recovering 2024-08-19T14:36:08.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:08 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 30752/283380 objects degraded (10.852%), 6 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-19T14:36:08.534 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:36:08.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:08 smithi045 ceph-mon[20143]: pgmap v1537: 47 pgs: 2 active+recovering+undersized+remapped, 6 active+recovering+undersized+degraded+remapped, 39 active+clean; 5.8 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 0 B/s wr, 373 op/s; 30752/283380 objects degraded (10.852%); 12919/283380 objects misplaced (4.559%); 13 MiB/s, 210 objects/s recovering 2024-08-19T14:36:08.587 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:08 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 30752/283380 objects degraded (10.852%), 6 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-19T14:36:08.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:08.589 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:08.589 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:36:08.690 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:36:08.712 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:08.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:08.713 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:36:08.814 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:36:08.863 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:08.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:08.864 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:36:08.965 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:36:08.989 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:08.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:08.990 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:36:09.091 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:36:09.113 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:09.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:09.114 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:36:09.215 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:36:09.254 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:09.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:09.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:36:09.356 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:36:09.378 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:09.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:09.379 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:36:09.480 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:36:09.502 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:09.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:09.503 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:36:09.604 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:36:09.628 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:09.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:09.628 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:36:09.729 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:36:09.750 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:09.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:09.751 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:36:09.852 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:36:09.873 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:09.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:09.874 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:36:09.975 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:36:09.996 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:09.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:09.997 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:36:10.098 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:36:10.142 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:10.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:10.143 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:36:10.244 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:36:10.268 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:10.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:10.269 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:36:10.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:10 smithi031 ceph-mon[16336]: pgmap v1538: 47 pgs: 2 active+recovering+undersized+remapped, 6 active+recovering+undersized+degraded+remapped, 39 active+clean; 5.7 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 0 B/s wr, 391 op/s; 30562/281001 objects degraded (10.876%); 12371/281001 objects misplaced (4.402%); 15 MiB/s, 242 objects/s recovering 2024-08-19T14:36:10.369 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:36:10.428 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:10.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:10.429 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:36:10.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:10 smithi032 ceph-mon[20004]: pgmap v1538: 47 pgs: 2 active+recovering+undersized+remapped, 6 active+recovering+undersized+degraded+remapped, 39 active+clean; 5.7 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 0 B/s wr, 391 op/s; 30562/281001 objects degraded (10.876%); 12371/281001 objects misplaced (4.402%); 15 MiB/s, 242 objects/s recovering 2024-08-19T14:36:10.530 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:36:10.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:10 smithi045 ceph-mon[20143]: pgmap v1538: 47 pgs: 2 active+recovering+undersized+remapped, 6 active+recovering+undersized+degraded+remapped, 39 active+clean; 5.7 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 0 B/s wr, 391 op/s; 30562/281001 objects degraded (10.876%); 12371/281001 objects misplaced (4.402%); 15 MiB/s, 242 objects/s recovering 2024-08-19T14:36:10.553 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:10.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:10.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:36:10.655 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:36:10.690 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:10.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:10.691 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:36:10.792 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:36:10.814 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:10.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:10.815 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:36:10.916 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:36:10.938 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:10.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:10.939 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:36:11.040 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:36:11.062 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:11.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:11.063 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:36:11.164 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:36:11.187 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:11.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:11.188 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:36:11.289 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:36:11.314 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:11.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:11.315 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:36:11.415 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:36:11.457 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:11.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:11.459 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:36:11.559 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:36:11.604 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:11.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:11.605 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:36:11.706 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:36:11.730 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:11.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:11.731 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:36:11.832 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:36:11.854 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:11.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:11.855 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:36:11.955 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:36:12.000 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:12.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:12.001 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:36:12.102 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:36:12.123 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:12.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:12.124 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:36:12.225 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:36:12.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:12.283 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:12.283 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:36:12.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:12 smithi031 ceph-mon[16336]: pgmap v1539: 47 pgs: 2 active+recovering+undersized+remapped, 6 active+recovering+undersized+degraded+remapped, 39 active+clean; 5.7 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 0 B/s wr, 445 op/s; 30556/277953 objects degraded (10.993%); 11906/277953 objects misplaced (4.283%); 16 MiB/s, 261 objects/s recovering 2024-08-19T14:36:12.385 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:36:12.408 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:12.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:12.409 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:36:12.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:12 smithi032 ceph-mon[20004]: pgmap v1539: 47 pgs: 2 active+recovering+undersized+remapped, 6 active+recovering+undersized+degraded+remapped, 39 active+clean; 5.7 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 0 B/s wr, 445 op/s; 30556/277953 objects degraded (10.993%); 11906/277953 objects misplaced (4.283%); 16 MiB/s, 261 objects/s recovering 2024-08-19T14:36:12.510 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:36:12.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:12 smithi045 ceph-mon[20143]: pgmap v1539: 47 pgs: 2 active+recovering+undersized+remapped, 6 active+recovering+undersized+degraded+remapped, 39 active+clean; 5.7 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 0 B/s wr, 445 op/s; 30556/277953 objects degraded (10.993%); 11906/277953 objects misplaced (4.283%); 16 MiB/s, 261 objects/s recovering 2024-08-19T14:36:12.553 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:12.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:12.554 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:36:12.655 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:36:12.704 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:12.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:12.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:36:12.806 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:36:12.830 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:12.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:12.831 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:36:12.932 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:36:12.990 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:12.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:12.991 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:36:13.092 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:36:13.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:13.123 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:13.123 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:36:13.224 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:36:13.257 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:13.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:13.259 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:36:13.360 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:36:13.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:13 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 30556/277953 objects degraded (10.993%), 6 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-19T14:36:13.381 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:13.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:13.382 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:36:13.483 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:36:13.506 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:13.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:13.507 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:36:13.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:13 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 30556/277953 objects degraded (10.993%), 6 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-19T14:36:13.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:13 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 30556/277953 objects degraded (10.993%), 6 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-19T14:36:13.608 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:36:13.629 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:13.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:13.630 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:36:13.731 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:36:13.753 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:13.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:13.754 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:36:13.855 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:36:13.877 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:13.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:13.878 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:36:13.979 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:36:14.009 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:14.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:14.010 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:36:14.111 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:36:14.133 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:14.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:14.134 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:36:14.235 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:36:14.262 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:14.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:14.264 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:36:14.364 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:36:14.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:14 smithi031 ceph-mon[16336]: pgmap v1540: 47 pgs: 2 active+recovering+undersized+remapped, 6 active+recovering+undersized+degraded+remapped, 39 active+clean; 5.6 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 0 B/s wr, 380 op/s; 30362/276564 objects degraded (10.978%); 11678/276564 objects misplaced (4.223%); 13 MiB/s, 209 objects/s recovering 2024-08-19T14:36:14.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:14 smithi031 ceph-mon[16336]: osdmap e652: 12 total, 12 up, 10 in 2024-08-19T14:36:14.386 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:14.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:14.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:36:14.488 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:36:14.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:14 smithi032 ceph-mon[20004]: pgmap v1540: 47 pgs: 2 active+recovering+undersized+remapped, 6 active+recovering+undersized+degraded+remapped, 39 active+clean; 5.6 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 0 B/s wr, 380 op/s; 30362/276564 objects degraded (10.978%); 11678/276564 objects misplaced (4.223%); 13 MiB/s, 209 objects/s recovering 2024-08-19T14:36:14.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:14 smithi032 ceph-mon[20004]: osdmap e652: 12 total, 12 up, 10 in 2024-08-19T14:36:14.526 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:14.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:14.528 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:36:14.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:14 smithi045 ceph-mon[20143]: pgmap v1540: 47 pgs: 2 active+recovering+undersized+remapped, 6 active+recovering+undersized+degraded+remapped, 39 active+clean; 5.6 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 0 B/s wr, 380 op/s; 30362/276564 objects degraded (10.978%); 11678/276564 objects misplaced (4.223%); 13 MiB/s, 209 objects/s recovering 2024-08-19T14:36:14.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:14 smithi045 ceph-mon[20143]: osdmap e652: 12 total, 12 up, 10 in 2024-08-19T14:36:14.628 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:36:14.648 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:36:14.650 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:14.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:14.651 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:36:14.752 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:36:14.773 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:14.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:14.774 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:36:14.875 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:36:14.897 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:14.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:14.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:36:14.999 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:36:15.020 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:15.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:15.021 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:36:15.122 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:36:15.144 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:15.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:15.145 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:36:15.246 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:36:15.298 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:15.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:15.299 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:36:15.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:15 smithi031 ceph-mon[16336]: osdmap e653: 12 total, 12 up, 10 in 2024-08-19T14:36:15.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:15 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:36:15.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:15 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:36:15.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:15 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:36:15.370 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:15 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:36:15.370 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:15 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:36:15.370 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:15 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:36:15.370 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:15 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:36:15.370 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:15 smithi031 ceph-mon[16336]: osdmap e654: 12 total, 12 up, 10 in 2024-08-19T14:36:15.400 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:36:15.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:15.448 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:15.448 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:36:15.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:15 smithi032 ceph-mon[20004]: osdmap e653: 12 total, 12 up, 10 in 2024-08-19T14:36:15.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:15 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:36:15.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:15 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:36:15.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:15 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:36:15.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:15 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:36:15.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:15 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:36:15.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:15 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:36:15.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:15 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:36:15.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:15 smithi032 ceph-mon[20004]: osdmap e654: 12 total, 12 up, 10 in 2024-08-19T14:36:15.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:15 smithi045 ceph-mon[20143]: osdmap e653: 12 total, 12 up, 10 in 2024-08-19T14:36:15.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:15 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:36:15.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:15 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:36:15.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:15 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:36:15.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:15 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:36:15.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:15 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:36:15.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:15 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:36:15.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:15 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:36:15.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:15 smithi045 ceph-mon[20143]: osdmap e654: 12 total, 12 up, 10 in 2024-08-19T14:36:15.549 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:36:15.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:15.575 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:15.576 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:36:15.677 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:36:15.718 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:15.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:15.719 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:36:15.820 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:36:15.842 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:15.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:15.844 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:36:15.945 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:36:15.967 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:15.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:15.968 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:36:16.069 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:36:16.095 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:16.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:16.096 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:36:16.197 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:36:16.220 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:16.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:16.221 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:36:16.322 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:36:16.343 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:16.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:16.344 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:36:16.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:16 smithi031 ceph-mon[16336]: pgmap v1543: 47 pgs: 2 active+recovering+undersized+remapped, 6 active+recovering+undersized+degraded+remapped, 39 active+clean; 5.6 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 0 B/s wr, 359 op/s; 30150/274740 objects degraded (10.974%); 11224/274740 objects misplaced (4.085%); 16 MiB/s, 258 objects/s recovering 2024-08-19T14:36:16.446 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:36:16.468 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:16.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:16.469 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:36:16.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:16 smithi032 ceph-mon[20004]: pgmap v1543: 47 pgs: 2 active+recovering+undersized+remapped, 6 active+recovering+undersized+degraded+remapped, 39 active+clean; 5.6 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 0 B/s wr, 359 op/s; 30150/274740 objects degraded (10.974%); 11224/274740 objects misplaced (4.085%); 16 MiB/s, 258 objects/s recovering 2024-08-19T14:36:16.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:16 smithi045 ceph-mon[20143]: pgmap v1543: 47 pgs: 2 active+recovering+undersized+remapped, 6 active+recovering+undersized+degraded+remapped, 39 active+clean; 5.6 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 0 B/s wr, 359 op/s; 30150/274740 objects degraded (10.974%); 11224/274740 objects misplaced (4.085%); 16 MiB/s, 258 objects/s recovering 2024-08-19T14:36:16.570 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:36:16.592 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:16.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:16.593 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:36:16.694 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:36:16.715 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:16.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:16.716 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:36:16.817 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:36:16.840 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:16.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:16.841 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:36:16.942 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:36:16.983 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:16.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:16.984 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:36:17.085 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:36:17.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:17.116 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:17.117 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:36:17.217 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:36:17.239 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:17.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:17.240 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:36:17.341 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:36:17.363 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:17.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:17.364 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:36:17.465 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:36:17.487 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:17.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:17.488 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:36:17.589 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:36:17.611 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:17.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:17.612 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:36:17.713 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:36:17.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:17.735 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:17.736 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:36:17.836 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:36:17.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:17.861 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:17.861 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:36:17.962 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:36:17.990 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:17.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:17.991 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:36:18.092 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:36:18.141 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:18.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:18.142 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:36:18.243 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:36:18.269 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:18.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:18.270 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:36:18.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:18 smithi031 ceph-mon[16336]: pgmap v1545: 47 pgs: 1 active+recovering+undersized+remapped, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 5.5 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 0 B/s wr, 332 op/s; 29846/271968 objects degraded (10.974%); 10774/271968 objects misplaced (3.961%); 17 MiB/s, 273 objects/s recovering 2024-08-19T14:36:18.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:18 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 29846/271968 objects degraded (10.974%), 6 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-19T14:36:18.371 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:36:18.430 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:18.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:18.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:36:18.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:18 smithi032 ceph-mon[20004]: pgmap v1545: 47 pgs: 1 active+recovering+undersized+remapped, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 5.5 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 0 B/s wr, 332 op/s; 29846/271968 objects degraded (10.974%); 10774/271968 objects misplaced (3.961%); 17 MiB/s, 273 objects/s recovering 2024-08-19T14:36:18.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:18 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 29846/271968 objects degraded (10.974%), 6 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-19T14:36:18.532 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:36:18.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:18 smithi045 ceph-mon[20143]: pgmap v1545: 47 pgs: 1 active+recovering+undersized+remapped, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 5.5 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 0 B/s wr, 332 op/s; 29846/271968 objects degraded (10.974%); 10774/271968 objects misplaced (3.961%); 17 MiB/s, 273 objects/s recovering 2024-08-19T14:36:18.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:18 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 29846/271968 objects degraded (10.974%), 6 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-19T14:36:18.573 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:18.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:18.574 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:36:18.675 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:36:18.696 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:18.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:18.697 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:36:18.798 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:36:18.822 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:18.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:18.824 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:36:18.925 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:36:18.948 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:18.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:18.949 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:36:19.050 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:36:19.092 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:19.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:19.093 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:36:19.194 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:36:19.215 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:19.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:19.216 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:36:19.317 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:36:19.339 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:19.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:19.340 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:36:19.441 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:36:19.484 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:19.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:19.485 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:36:19.586 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:36:19.608 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:19.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:19.609 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:36:19.710 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:36:19.732 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:19.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:19.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:36:19.834 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:36:19.857 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:19.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:19.858 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:36:19.959 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:36:19.981 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:19.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:19.982 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:36:20.082 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:36:20.105 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:20.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:20.106 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:36:20.141 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-08-19T14:36:20.207 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:36:20.230 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:20.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:20.231 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:36:20.332 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:36:20.353 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:20.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:20.354 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:36:20.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:20 smithi031 ceph-mon[16336]: pgmap v1546: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 5.5 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 374 op/s; 29466/269814 objects degraded (10.921%); 10440/269814 objects misplaced (3.869%); 20 MiB/s, 313 objects/s recovering 2024-08-19T14:36:20.455 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:36:20.477 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:20.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:20.478 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:36:20.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:20 smithi032 ceph-mon[20004]: pgmap v1546: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 5.5 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 374 op/s; 29466/269814 objects degraded (10.921%); 10440/269814 objects misplaced (3.869%); 20 MiB/s, 313 objects/s recovering 2024-08-19T14:36:20.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:20 smithi045 ceph-mon[20143]: pgmap v1546: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 5.5 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 374 op/s; 29466/269814 objects degraded (10.921%); 10440/269814 objects misplaced (3.869%); 20 MiB/s, 313 objects/s recovering 2024-08-19T14:36:20.579 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:36:20.600 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:20.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:20.601 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:36:20.702 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:36:20.724 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:20.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:20.725 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:36:20.826 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:36:20.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:20.870 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:20.870 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:36:20.971 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:36:20.993 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:20.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:20.994 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:36:21.094 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:36:21.117 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:21.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:21.118 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:36:21.219 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:36:21.245 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:21.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:21.246 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:36:21.347 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:36:21.398 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:21.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:21.399 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:36:21.500 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:36:21.526 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:21.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:21.527 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:36:21.613 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:36:21.628 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:36:21.649 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:21.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:21.650 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:36:21.690 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:36:21.751 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:36:21.772 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:21.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:21.773 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:36:21.874 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:36:21.896 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:21.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:21.897 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:36:21.998 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:36:22.020 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:22.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:22.021 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:36:22.110 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is unset 2024-08-19T14:36:22.123 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:36:22.146 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:22.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:22.147 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:36:22.248 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:36:22.271 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:22.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:22.272 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:36:22.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:22 smithi031 ceph-mon[16336]: pgmap v1547: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 5.4 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 399 op/s; 29402/266580 objects degraded (11.029%); 10376/266580 objects misplaced (3.892%); 11 MiB/s, 180 objects/s recovering 2024-08-19T14:36:22.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:22 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:36:22.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:22 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:36:22.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:22 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2872763868' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:36:22.372 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:36:22.394 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:22.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:22.395 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:36:22.496 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:36:22.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:22 smithi032 ceph-mon[20004]: pgmap v1547: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 5.4 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 399 op/s; 29402/266580 objects degraded (11.029%); 10376/266580 objects misplaced (3.892%); 11 MiB/s, 180 objects/s recovering 2024-08-19T14:36:22.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:22 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:36:22.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:22 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:36:22.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:22 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2872763868' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:36:22.517 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:22.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:22.518 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:36:22.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:22 smithi045 ceph-mon[20143]: pgmap v1547: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 5.4 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 399 op/s; 29402/266580 objects degraded (11.029%); 10376/266580 objects misplaced (3.892%); 11 MiB/s, 180 objects/s recovering 2024-08-19T14:36:22.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:22 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:36:22.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:22 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:36:22.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:22 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2872763868' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:36:22.619 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:36:22.643 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:22.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:22.644 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:36:22.745 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:36:22.767 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:22.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:22.768 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:36:22.869 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:36:22.891 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:22.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:22.892 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:36:22.993 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:36:23.031 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:23.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:23.032 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:36:23.071 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:36:23.133 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:36:23.155 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:23.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:23.156 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:36:23.257 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:36:23.278 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:23.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:23.279 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:36:23.373 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:23 smithi031 ceph-mon[16336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:36:23.373 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:23 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2872763868' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:36:23.373 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:23 smithi031 ceph-mon[16336]: osdmap e655: 12 total, 12 up, 10 in 2024-08-19T14:36:23.373 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:23 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 29402/266580 objects degraded (11.029%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-19T14:36:23.380 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:36:23.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:23.420 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:23.420 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:36:23.446 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:23 smithi045 ceph-mon[20143]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:36:23.447 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:23 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2872763868' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:36:23.447 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:23 smithi045 ceph-mon[20143]: osdmap e655: 12 total, 12 up, 10 in 2024-08-19T14:36:23.447 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:23 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 29402/266580 objects degraded (11.029%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-19T14:36:23.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:23 smithi032 ceph-mon[20004]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:36:23.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:23 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2872763868' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:36:23.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:23 smithi032 ceph-mon[20004]: osdmap e655: 12 total, 12 up, 10 in 2024-08-19T14:36:23.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:23 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 29402/266580 objects degraded (11.029%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-19T14:36:23.521 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:36:23.544 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:23.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:23.546 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:36:23.646 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:36:23.669 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:23.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:23.670 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:36:23.771 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:36:23.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:23.818 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:23.819 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:36:23.919 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:36:23.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:23.944 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:23.945 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:36:24.046 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:36:24.069 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:24.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:24.070 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:36:24.133 INFO:teuthology.orchestra.run.smithi031.stderr:reweighted osd.3 to 1 (10000) 2024-08-19T14:36:24.170 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:36:24.177 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:24 smithi031 ceph-mon[16336]: pgmap v1549: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 5.4 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 407 op/s; 29247/264954 objects degraded (11.039%); 10115/264954 objects misplaced (3.818%); 12 MiB/s, 196 objects/s recovering 2024-08-19T14:36:24.177 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:24 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2226367077' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-08-19T14:36:24.177 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:24 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-08-19T14:36:24.222 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:24.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:24.223 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:36:24.324 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:36:24.346 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:24.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:24.347 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:36:24.448 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:36:24.471 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:24.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:24.473 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:36:24.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:24 smithi032 ceph-mon[20004]: pgmap v1549: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 5.4 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 407 op/s; 29247/264954 objects degraded (11.039%); 10115/264954 objects misplaced (3.818%); 12 MiB/s, 196 objects/s recovering 2024-08-19T14:36:24.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:24 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2226367077' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-08-19T14:36:24.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:24 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-08-19T14:36:24.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:24 smithi045 ceph-mon[20143]: pgmap v1549: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 5.4 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 407 op/s; 29247/264954 objects degraded (11.039%); 10115/264954 objects misplaced (3.818%); 12 MiB/s, 196 objects/s recovering 2024-08-19T14:36:24.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:24 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2226367077' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-08-19T14:36:24.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:24 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-08-19T14:36:24.574 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:36:24.595 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:24.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:24.596 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:36:24.688 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:36:24.688 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:36:24.697 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:36:24.719 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:24.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:24.720 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:36:24.821 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:36:24.845 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:24.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:24.846 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:36:24.947 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:36:24.984 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:24.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:24.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:36:25.086 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:36:25.108 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:25.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:25.109 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:36:25.170 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd reweight 4 1 2024-08-19T14:36:25.209 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:36:25.231 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:25.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:25.232 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:36:25.333 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:36:25.356 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:25.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:25.357 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:36:25.370 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:25 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-08-19T14:36:25.370 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:25 smithi031 ceph-mon[16336]: osdmap e656: 12 total, 12 up, 10 in 2024-08-19T14:36:25.370 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:25 smithi031 ceph-mon[16336]: pgmap v1551: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 5.3 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 376 op/s; 29018/262917 objects degraded (11.037%); 10115/262917 objects misplaced (3.847%); 8.9 MiB/s, 142 objects/s recovering 2024-08-19T14:36:25.458 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:36:25.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:25 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-08-19T14:36:25.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:25 smithi032 ceph-mon[20004]: osdmap e656: 12 total, 12 up, 10 in 2024-08-19T14:36:25.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:25 smithi032 ceph-mon[20004]: pgmap v1551: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 5.3 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 376 op/s; 29018/262917 objects degraded (11.037%); 10115/262917 objects misplaced (3.847%); 8.9 MiB/s, 142 objects/s recovering 2024-08-19T14:36:25.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:25.525 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:25.525 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:36:25.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:25 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-08-19T14:36:25.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:25 smithi045 ceph-mon[20143]: osdmap e656: 12 total, 12 up, 10 in 2024-08-19T14:36:25.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:25 smithi045 ceph-mon[20143]: pgmap v1551: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 5.3 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 376 op/s; 29018/262917 objects degraded (11.037%); 10115/262917 objects misplaced (3.847%); 8.9 MiB/s, 142 objects/s recovering 2024-08-19T14:36:25.626 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:36:25.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:25.674 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:25.674 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:36:25.775 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:36:25.821 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:36:25.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:25.825 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:25.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:36:25.857 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-08-19T14:36:25.926 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:36:25.979 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-19T14:36:26.028 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:26.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:26.029 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:36:26.130 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:36:26.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:26.153 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:26.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:36:26.254 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:36:26.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:26.329 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:26.329 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:36:26.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:26 smithi031 ceph-mon[16336]: 1.1 scrub starts 2024-08-19T14:36:26.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:26 smithi031 ceph-mon[16336]: 1.1 scrub ok 2024-08-19T14:36:26.430 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:36:26.452 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:26.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:26.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:36:26.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:26 smithi032 ceph-mon[20004]: 1.1 scrub starts 2024-08-19T14:36:26.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:26 smithi032 ceph-mon[20004]: 1.1 scrub ok 2024-08-19T14:36:26.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:26 smithi045 ceph-mon[20143]: 1.1 scrub starts 2024-08-19T14:36:26.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:26 smithi045 ceph-mon[20143]: 1.1 scrub ok 2024-08-19T14:36:26.554 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:36:26.596 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:26.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:26.597 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:36:26.698 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:36:26.719 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:26.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:26.720 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:36:26.821 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:36:26.854 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:26.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:26.855 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:36:26.956 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:36:26.979 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:26.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:26.980 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:36:27.081 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:36:27.103 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:27.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:27.104 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:36:27.205 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:36:27.227 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:27.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:27.228 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:36:27.329 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:36:27.351 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:27.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:27.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:36:27.453 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:36:27.476 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:27.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:27.477 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:36:27.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:27 smithi032 ceph-mon[20004]: pgmap v1552: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 5.3 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 431 op/s; 28800/259458 objects degraded (11.100%); 10059/259458 objects misplaced (3.877%); 6.3 MiB/s, 100 objects/s recovering 2024-08-19T14:36:27.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:27 smithi032 ceph-mon[20004]: 1.1b scrub starts 2024-08-19T14:36:27.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:27 smithi032 ceph-mon[20004]: 1.1b scrub ok 2024-08-19T14:36:27.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:27 smithi045 ceph-mon[20143]: pgmap v1552: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 5.3 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 431 op/s; 28800/259458 objects degraded (11.100%); 10059/259458 objects misplaced (3.877%); 6.3 MiB/s, 100 objects/s recovering 2024-08-19T14:36:27.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:27 smithi045 ceph-mon[20143]: 1.1b scrub starts 2024-08-19T14:36:27.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:27 smithi045 ceph-mon[20143]: 1.1b scrub ok 2024-08-19T14:36:27.578 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:36:27.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:27 smithi031 ceph-mon[16336]: pgmap v1552: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 5.3 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 431 op/s; 28800/259458 objects degraded (11.100%); 10059/259458 objects misplaced (3.877%); 6.3 MiB/s, 100 objects/s recovering 2024-08-19T14:36:27.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:27 smithi031 ceph-mon[16336]: 1.1b scrub starts 2024-08-19T14:36:27.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:27 smithi031 ceph-mon[16336]: 1.1b scrub ok 2024-08-19T14:36:27.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:27.617 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:27.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:36:27.718 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:36:27.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:27.760 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:27.760 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:36:27.861 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:36:27.884 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:27.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:27.885 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:36:27.986 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:36:28.010 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:28.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:28.011 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:36:28.112 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:36:28.137 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:28.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:28.138 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:36:28.239 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:36:28.261 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:28.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:28.262 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:36:28.362 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:36:28.384 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:28.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:28.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:36:28.486 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:36:28.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:28 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 28800/259458 objects degraded (11.100%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-19T14:36:28.510 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:28.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:28.511 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:36:28.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:28 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 28800/259458 objects degraded (11.100%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-19T14:36:28.612 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:36:28.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:28 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 28800/259458 objects degraded (11.100%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-19T14:36:28.634 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:28.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:28.635 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:36:28.736 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:36:28.762 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:28.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:28.763 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:36:28.864 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:36:28.887 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:28.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:28.888 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:36:28.989 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:36:29.011 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:29.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:29.012 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:36:29.113 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:36:29.136 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:29.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:29.138 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:36:29.239 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:36:29.263 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:29.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:29.264 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:36:29.365 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:36:29.386 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:29.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:29.388 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:36:29.489 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:36:29.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:29 smithi032 ceph-mon[20004]: pgmap v1553: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 5.2 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 391 op/s; 28579/257178 objects degraded (11.113%); 9830/257178 objects misplaced (3.822%); 8.4 MiB/s, 134 objects/s recovering 2024-08-19T14:36:29.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:29 smithi032 ceph-mon[20004]: 1.17 scrub starts 2024-08-19T14:36:29.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:29 smithi032 ceph-mon[20004]: 1.17 scrub ok 2024-08-19T14:36:29.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:29 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:36:29.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:29 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:36:29.510 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:29.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:29.511 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:36:29.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:29 smithi045 ceph-mon[20143]: pgmap v1553: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 5.2 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 391 op/s; 28579/257178 objects degraded (11.113%); 9830/257178 objects misplaced (3.822%); 8.4 MiB/s, 134 objects/s recovering 2024-08-19T14:36:29.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:29 smithi045 ceph-mon[20143]: 1.17 scrub starts 2024-08-19T14:36:29.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:29 smithi045 ceph-mon[20143]: 1.17 scrub ok 2024-08-19T14:36:29.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:29 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:36:29.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:29 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:36:29.612 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:36:29.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:29 smithi031 ceph-mon[16336]: pgmap v1553: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 5.2 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 391 op/s; 28579/257178 objects degraded (11.113%); 9830/257178 objects misplaced (3.822%); 8.4 MiB/s, 134 objects/s recovering 2024-08-19T14:36:29.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:29 smithi031 ceph-mon[16336]: 1.17 scrub starts 2024-08-19T14:36:29.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:29 smithi031 ceph-mon[16336]: 1.17 scrub ok 2024-08-19T14:36:29.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:29 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:36:29.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:29 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:36:29.634 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:29.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:29.635 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:36:29.736 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:36:29.759 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:29.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:29.760 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:36:29.861 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:36:29.884 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:29.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:29.885 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:36:29.985 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:36:30.007 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:30.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:30.008 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:36:30.109 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:36:30.146 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:30.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:30.148 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:36:30.249 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:36:30.257 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:30 smithi031 ceph-mon[16336]: osdmap e657: 12 total, 12 up, 10 in 2024-08-19T14:36:30.257 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:30 smithi031 ceph-mon[16336]: 1.19 scrub starts 2024-08-19T14:36:30.257 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:30 smithi031 ceph-mon[16336]: 1.19 scrub ok 2024-08-19T14:36:30.257 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:30 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/679340508' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:36:30.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:30.272 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:30.272 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:36:30.374 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:36:30.395 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:30.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:30.396 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:36:30.497 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:36:30.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:30 smithi032 ceph-mon[20004]: osdmap e657: 12 total, 12 up, 10 in 2024-08-19T14:36:30.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:30 smithi032 ceph-mon[20004]: 1.19 scrub starts 2024-08-19T14:36:30.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:30 smithi032 ceph-mon[20004]: 1.19 scrub ok 2024-08-19T14:36:30.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:30 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/679340508' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:36:30.520 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:30.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:30.522 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:36:30.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:30 smithi045 ceph-mon[20143]: osdmap e657: 12 total, 12 up, 10 in 2024-08-19T14:36:30.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:30 smithi045 ceph-mon[20143]: 1.19 scrub starts 2024-08-19T14:36:30.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:30 smithi045 ceph-mon[20143]: 1.19 scrub ok 2024-08-19T14:36:30.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:30 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/679340508' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:36:30.622 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:36:30.664 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:30.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:30.666 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:36:30.766 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:36:30.789 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:30.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:30.790 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:36:30.892 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:36:30.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:30.914 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:30.915 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:36:31.015 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:36:31.039 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:31.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:31.041 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:36:31.094 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:36:31.094 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:36:31.094 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:36:31.094 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:36:31.094 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:36:31.095 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:36:31.095 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:36:31.095 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:36:31.095 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:36:31.095 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:36:31.095 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:36:31.095 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:36:31.095 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:36:31.095 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:36:31.095 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:36:31.095 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:36:31.096 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:36:31.096 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:36:31.096 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:36:31.096 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:36:31.096 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:36:31.096 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:36:31.096 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:36:31.096 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:36:31.142 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:36:31.177 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:31.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:31.179 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:36:31.279 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:36:31.302 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:31.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:31.304 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:36:31.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:31 smithi031 ceph-mon[16336]: pgmap v1555: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 5.2 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 461 op/s; 28514/253878 objects degraded (11.231%); 9766/253878 objects misplaced (3.847%); 6.6 MiB/s, 105 objects/s recovering 2024-08-19T14:36:31.404 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:36:31.427 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:31.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:31.428 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:36:31.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:31 smithi032 ceph-mon[20004]: pgmap v1555: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 5.2 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 461 op/s; 28514/253878 objects degraded (11.231%); 9766/253878 objects misplaced (3.847%); 6.6 MiB/s, 105 objects/s recovering 2024-08-19T14:36:31.529 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:36:31.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:31 smithi045 ceph-mon[20143]: pgmap v1555: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 5.2 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 461 op/s; 28514/253878 objects degraded (11.231%); 9766/253878 objects misplaced (3.847%); 6.6 MiB/s, 105 objects/s recovering 2024-08-19T14:36:31.551 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:31.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:31.552 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:36:31.653 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:36:31.676 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:31.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:31.677 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:36:31.778 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:36:31.800 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:31.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:31.801 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:36:31.903 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:36:31.938 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:31.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:31.939 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:36:32.040 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:36:32.073 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:32.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:32.075 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:36:32.175 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:36:32.198 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:32.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:32.199 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:36:32.300 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:36:32.321 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:32.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:32.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:36:32.423 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:36:32.467 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:32.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:32.468 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:36:32.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:32 smithi032 ceph-mon[20004]: 1.1c scrub starts 2024-08-19T14:36:32.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:32 smithi032 ceph-mon[20004]: 1.1c scrub ok 2024-08-19T14:36:32.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:32 smithi045 ceph-mon[20143]: 1.1c scrub starts 2024-08-19T14:36:32.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:32 smithi045 ceph-mon[20143]: 1.1c scrub ok 2024-08-19T14:36:32.569 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:36:32.590 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:32.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:32.591 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:36:32.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:32 smithi031 ceph-mon[16336]: 1.1c scrub starts 2024-08-19T14:36:32.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:32 smithi031 ceph-mon[16336]: 1.1c scrub ok 2024-08-19T14:36:32.647 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:36:32.692 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:36:32.714 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:32.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:32.715 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:36:32.816 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:36:32.837 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:32.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:32.839 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:36:32.939 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:36:32.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:32.963 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:32.963 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:36:33.064 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:36:33.086 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:33.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:33.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:36:33.188 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:36:33.244 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:33.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:33.245 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:36:33.346 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:36:33.370 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:33.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:33.371 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:36:33.471 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:36:33.494 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:33.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:33.495 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:36:33.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:33 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:36:33.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:33 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:36:33.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:33 smithi032 ceph-mon[20004]: pgmap v1556: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 5.1 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 438 op/s; 28288/252312 objects degraded (11.212%); 9508/252312 objects misplaced (3.768%); 7.3 MiB/s, 117 objects/s recovering 2024-08-19T14:36:33.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:33 smithi032 ceph-mon[20004]: osdmap e658: 12 total, 12 up, 10 in 2024-08-19T14:36:33.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:33 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:36:33.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:33 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:36:33.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:33 smithi032 ceph-mon[20004]: 7.4 scrub starts 2024-08-19T14:36:33.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:33 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 28514/253878 objects degraded (11.231%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-19T14:36:33.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:33 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:36:33.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:33 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:36:33.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:33 smithi045 ceph-mon[20143]: pgmap v1556: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 5.1 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 438 op/s; 28288/252312 objects degraded (11.212%); 9508/252312 objects misplaced (3.768%); 7.3 MiB/s, 117 objects/s recovering 2024-08-19T14:36:33.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:33 smithi045 ceph-mon[20143]: osdmap e658: 12 total, 12 up, 10 in 2024-08-19T14:36:33.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:33 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:36:33.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:33 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:36:33.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:33 smithi045 ceph-mon[20143]: 7.4 scrub starts 2024-08-19T14:36:33.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:33 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 28514/253878 objects degraded (11.231%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-19T14:36:33.552 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:33 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:36:33.552 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:33 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:36:33.552 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:33 smithi031 ceph-mon[16336]: pgmap v1556: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 5.1 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 438 op/s; 28288/252312 objects degraded (11.212%); 9508/252312 objects misplaced (3.768%); 7.3 MiB/s, 117 objects/s recovering 2024-08-19T14:36:33.552 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:33 smithi031 ceph-mon[16336]: osdmap e658: 12 total, 12 up, 10 in 2024-08-19T14:36:33.552 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:33 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:36:33.552 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:33 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:36:33.552 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:33 smithi031 ceph-mon[16336]: 7.4 scrub starts 2024-08-19T14:36:33.553 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:33 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 28514/253878 objects degraded (11.231%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-19T14:36:33.596 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:36:33.618 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:33.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:33.619 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:36:33.720 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:36:33.742 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:33.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:33.743 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:36:33.844 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:36:33.855 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:36:33.865 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:33.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:33.866 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:36:33.967 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:36:33.989 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:33.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:33.990 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:36:34.091 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:36:34.114 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:34.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:34.115 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:36:34.216 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:36:34.227 INFO:teuthology.orchestra.run.smithi031.stderr:reweighted osd.4 to 1 (10000) 2024-08-19T14:36:34.277 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:34 smithi031 ceph-mon[16336]: osdmap e659: 12 total, 12 up, 10 in 2024-08-19T14:36:34.278 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:34 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3854936719' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-08-19T14:36:34.278 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:34 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-08-19T14:36:34.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:34.279 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:34.279 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:36:34.380 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:36:34.438 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:34.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:34.439 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:36:34.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:34 smithi032 ceph-mon[20004]: osdmap e659: 12 total, 12 up, 10 in 2024-08-19T14:36:34.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:34 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3854936719' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-08-19T14:36:34.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:34 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-08-19T14:36:34.540 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:36:34.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:34 smithi045 ceph-mon[20143]: osdmap e659: 12 total, 12 up, 10 in 2024-08-19T14:36:34.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:34 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3854936719' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-08-19T14:36:34.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:34 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-08-19T14:36:34.561 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:34.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:34.562 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:36:34.566 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:36:34.566 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 20, 2024-08-19T14:36:34.566 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 600, 2024-08-19T14:36:34.566 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [ 2024-08-19T14:36:34.566 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.566 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:76306 6.9 6:93e0e6a2:::benchmark_data_smithi045_97937_object76305:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e525)", 2024-08-19T14:36:34.566 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:29:17.429480+0000", 2024-08-19T14:36:34.567 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 437.133525953, 2024-08-19T14:36:34.567 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.43747081700000001, 2024-08-19T14:36:34.567 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:36:34.567 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:36:34.567 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:36:34.567 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:36:34.567 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:36:34.567 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:36:34.567 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 76306 2024-08-19T14:36:34.567 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.567 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:36:34.567 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.567 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:36:34.567 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.429480+0000", 2024-08-19T14:36:34.568 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:36:34.568 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.568 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.568 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:36:34.568 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.429480+0000", 2024-08-19T14:36:34.568 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:36:34.568 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.568 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.568 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:36:34.568 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.429483+0000", 2024-08-19T14:36:34.568 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.5450000000000002e-06 2024-08-19T14:36:34.568 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.569 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.569 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:36:34.569 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.429662+0000", 2024-08-19T14:36:34.569 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000178944 2024-08-19T14:36:34.569 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.569 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.569 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:36:34.569 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.429683+0000", 2024-08-19T14:36:34.569 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.1327000000000001e-05 2024-08-19T14:36:34.569 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.569 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.569 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:36:34.569 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.429696+0000", 2024-08-19T14:36:34.570 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3359e-05 2024-08-19T14:36:34.570 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.570 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.570 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:36:34.570 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.433475+0000", 2024-08-19T14:36:34.570 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.003778989 2024-08-19T14:36:34.570 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.570 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.570 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:36:34.570 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.433491+0000", 2024-08-19T14:36:34.570 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5894999999999999e-05 2024-08-19T14:36:34.570 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.570 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.571 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:36:34.571 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.805712+0000", 2024-08-19T14:36:34.571 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.37222085500000002 2024-08-19T14:36:34.571 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.571 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.571 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:36:34.571 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.805869+0000", 2024-08-19T14:36:34.571 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00015659800000000001 2024-08-19T14:36:34.571 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.571 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.571 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 2,8", 2024-08-19T14:36:34.571 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.806040+0000", 2024-08-19T14:36:34.571 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00017071799999999999 2024-08-19T14:36:34.572 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.572 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.572 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:36:34.572 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.822089+0000", 2024-08-19T14:36:34.572 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.016049562 2024-08-19T14:36:34.572 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.572 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.572 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:36:34.572 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.828981+0000", 2024-08-19T14:36:34.572 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0068919150000000002 2024-08-19T14:36:34.572 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.572 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.572 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:36:34.572 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.862487+0000", 2024-08-19T14:36:34.573 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.033505842000000001 2024-08-19T14:36:34.573 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.573 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.573 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:36:34.573 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.862563+0000", 2024-08-19T14:36:34.573 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.6458000000000006e-05 2024-08-19T14:36:34.573 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.573 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.573 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:36:34.573 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.866951+0000", 2024-08-19T14:36:34.573 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0043878099999999998 2024-08-19T14:36:34.573 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:36:34.573 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:36:34.574 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:36:34.574 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.574 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.574 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:76308 6.9 6:9d9a3d15:::benchmark_data_smithi045_97937_object76307:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e525)", 2024-08-19T14:36:34.574 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:29:17.429724+0000", 2024-08-19T14:36:34.574 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 437.13328198699998, 2024-08-19T14:36:34.574 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.44172447399999998, 2024-08-19T14:36:34.574 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:36:34.574 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:36:34.574 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:36:34.574 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:36:34.574 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:36:34.574 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:36:34.575 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 76308 2024-08-19T14:36:34.575 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.575 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:36:34.575 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.575 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:36:34.575 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.429724+0000", 2024-08-19T14:36:34.575 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:36:34.575 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.575 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.575 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:36:34.575 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.429724+0000", 2024-08-19T14:36:34.575 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:36:34.575 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.575 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.576 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:36:34.576 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.429725+0000", 2024-08-19T14:36:34.576 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.99e-07 2024-08-19T14:36:34.576 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.576 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.576 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:36:34.576 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.429754+0000", 2024-08-19T14:36:34.576 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.9033000000000001e-05 2024-08-19T14:36:34.576 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.576 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.576 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:36:34.576 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.429763+0000", 2024-08-19T14:36:34.576 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.4569999999999994e-06 2024-08-19T14:36:34.576 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.577 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.577 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:36:34.577 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.429770+0000", 2024-08-19T14:36:34.577 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.229e-06 2024-08-19T14:36:34.577 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.577 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.577 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:36:34.577 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.433520+0000", 2024-08-19T14:36:34.577 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.003750761 2024-08-19T14:36:34.577 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.577 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.577 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:36:34.577 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.433528+0000", 2024-08-19T14:36:34.577 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.8800000000000008e-06 2024-08-19T14:36:34.578 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.578 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.578 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:36:34.578 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.813660+0000", 2024-08-19T14:36:34.578 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.380131846 2024-08-19T14:36:34.578 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.578 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.578 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:36:34.578 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.813821+0000", 2024-08-19T14:36:34.578 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000161237 2024-08-19T14:36:34.578 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.578 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.578 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 2,8", 2024-08-19T14:36:34.578 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.814009+0000", 2024-08-19T14:36:34.579 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000187622 2024-08-19T14:36:34.579 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.579 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.579 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:36:34.579 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.822117+0000", 2024-08-19T14:36:34.579 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0081084610000000008 2024-08-19T14:36:34.579 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.579 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.579 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:36:34.579 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.828993+0000", 2024-08-19T14:36:34.579 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0068753349999999998 2024-08-19T14:36:34.579 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.579 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.580 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:36:34.580 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.867012+0000", 2024-08-19T14:36:34.580 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.038018926000000001 2024-08-19T14:36:34.580 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.580 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.580 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:36:34.580 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.867114+0000", 2024-08-19T14:36:34.581 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000102434 2024-08-19T14:36:34.581 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.581 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.581 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:36:34.581 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.871449+0000", 2024-08-19T14:36:34.581 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0043346540000000003 2024-08-19T14:36:34.581 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:36:34.581 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:36:34.581 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:36:34.581 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.581 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.581 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:76295 6.0 6:09c9db57:::benchmark_data_smithi045_97937_object76294:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e526)", 2024-08-19T14:36:34.582 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:29:17.442554+0000", 2024-08-19T14:36:34.582 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 437.12045219100003, 2024-08-19T14:36:34.582 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.29366227700000003, 2024-08-19T14:36:34.582 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:36:34.582 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:36:34.582 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:36:34.582 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:36:34.582 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:36:34.582 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:36:34.582 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 76295 2024-08-19T14:36:34.582 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.582 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:36:34.582 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.583 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:36:34.583 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.442554+0000", 2024-08-19T14:36:34.583 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:36:34.583 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.583 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.583 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:36:34.583 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.442554+0000", 2024-08-19T14:36:34.583 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:36:34.583 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.583 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.583 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:36:34.583 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.442556+0000", 2024-08-19T14:36:34.583 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.9630000000000001e-06 2024-08-19T14:36:34.583 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.584 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.584 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:36:34.584 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.442602+0000", 2024-08-19T14:36:34.584 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.5590999999999998e-05 2024-08-19T14:36:34.584 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.584 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.584 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:36:34.584 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.442620+0000", 2024-08-19T14:36:34.584 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.8179e-05 2024-08-19T14:36:34.584 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.584 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.584 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:36:34.584 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.442631+0000", 2024-08-19T14:36:34.585 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1467e-05 2024-08-19T14:36:34.585 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.585 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.585 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:36:34.585 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.446010+0000", 2024-08-19T14:36:34.585 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.003378551 2024-08-19T14:36:34.585 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.585 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.585 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:36:34.585 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.446021+0000", 2024-08-19T14:36:34.585 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1204e-05 2024-08-19T14:36:34.585 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.585 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.585 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:36:34.585 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.735842+0000", 2024-08-19T14:36:34.586 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.28982053699999999 2024-08-19T14:36:34.586 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.586 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.586 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:36:34.586 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.736216+0000", 2024-08-19T14:36:34.586 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00037478499999999998 2024-08-19T14:36:34.586 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:36:34.586 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:36:34.586 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:36:34.586 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.586 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.586 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:76320 6.0 6:0c4767d7:::benchmark_data_smithi045_97937_object76319:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e526)", 2024-08-19T14:36:34.586 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:29:17.442650+0000", 2024-08-19T14:36:34.586 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 437.12035592799998, 2024-08-19T14:36:34.587 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.327560244, 2024-08-19T14:36:34.587 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:36:34.587 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:36:34.587 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:36:34.587 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:36:34.587 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:36:34.587 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:36:34.587 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 76320 2024-08-19T14:36:34.587 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.587 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:36:34.587 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.587 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:36:34.587 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.442650+0000", 2024-08-19T14:36:34.588 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:36:34.588 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.588 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.588 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:36:34.588 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.442650+0000", 2024-08-19T14:36:34.588 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:36:34.588 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.588 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.588 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:36:34.588 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.442651+0000", 2024-08-19T14:36:34.588 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.8699999999999995e-07 2024-08-19T14:36:34.588 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.588 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.588 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:36:34.589 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.442716+0000", 2024-08-19T14:36:34.589 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.5117999999999996e-05 2024-08-19T14:36:34.589 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.589 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.589 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:36:34.589 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.442725+0000", 2024-08-19T14:36:34.589 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.704e-06 2024-08-19T14:36:34.589 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.589 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.589 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:36:34.589 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.442731+0000", 2024-08-19T14:36:34.589 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.5810000000000002e-06 2024-08-19T14:36:34.589 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.590 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.590 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:36:34.590 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.446043+0000", 2024-08-19T14:36:34.590 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0033121180000000002 2024-08-19T14:36:34.590 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.590 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.590 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:36:34.590 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.446050+0000", 2024-08-19T14:36:34.590 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.2920000000000001e-06 2024-08-19T14:36:34.590 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.590 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.590 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:36:34.590 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.741896+0000", 2024-08-19T14:36:34.591 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.29584602100000001 2024-08-19T14:36:34.591 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.591 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.591 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:36:34.591 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.742326+0000", 2024-08-19T14:36:34.591 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00043030599999999998 2024-08-19T14:36:34.591 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.591 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.591 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 2,3", 2024-08-19T14:36:34.591 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.742898+0000", 2024-08-19T14:36:34.591 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00057183400000000001 2024-08-19T14:36:34.591 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.593 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.593 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:36:34.593 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.753358+0000", 2024-08-19T14:36:34.593 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.010460265999999999 2024-08-19T14:36:34.593 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.593 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.594 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:36:34.594 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.764669+0000", 2024-08-19T14:36:34.594 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.011310659000000001 2024-08-19T14:36:34.594 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.594 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.594 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:36:34.594 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.764724+0000", 2024-08-19T14:36:34.594 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.5470000000000003e-05 2024-08-19T14:36:34.594 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.594 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.594 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:36:34.594 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.764865+0000", 2024-08-19T14:36:34.595 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00014082400000000001 2024-08-19T14:36:34.595 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.595 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.595 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:36:34.595 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.770211+0000", 2024-08-19T14:36:34.595 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0053455639999999997 2024-08-19T14:36:34.595 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:36:34.595 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:36:34.595 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:36:34.595 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.595 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.596 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:76322 6.0 6:0c6cfef1:::benchmark_data_smithi045_97937_object76321:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e526)", 2024-08-19T14:36:34.596 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:29:17.442741+0000", 2024-08-19T14:36:34.596 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 437.120265641, 2024-08-19T14:36:34.596 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.35300911099999999, 2024-08-19T14:36:34.596 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:36:34.596 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:36:34.596 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:36:34.596 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:36:34.596 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:36:34.596 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:36:34.596 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 76322 2024-08-19T14:36:34.596 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.596 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:36:34.597 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.597 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:36:34.597 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.442741+0000", 2024-08-19T14:36:34.597 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:36:34.597 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.597 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.597 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:36:34.597 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.442741+0000", 2024-08-19T14:36:34.597 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:36:34.597 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.597 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.597 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:36:34.597 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.442741+0000", 2024-08-19T14:36:34.598 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4900000000000001e-07 2024-08-19T14:36:34.598 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.598 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.598 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:36:34.598 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.442792+0000", 2024-08-19T14:36:34.598 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0696999999999999e-05 2024-08-19T14:36:34.598 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.598 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.598 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:36:34.599 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.442800+0000", 2024-08-19T14:36:34.599 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.7740000000000004e-06 2024-08-19T14:36:34.599 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.599 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.599 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:36:34.599 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.442805+0000", 2024-08-19T14:36:34.599 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.9699999999999996e-06 2024-08-19T14:36:34.599 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.599 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.599 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:36:34.599 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.446071+0000", 2024-08-19T14:36:34.599 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0032653069999999998 2024-08-19T14:36:34.599 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.600 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.600 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:36:34.600 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.446077+0000", 2024-08-19T14:36:34.600 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.2389999999999999e-06 2024-08-19T14:36:34.600 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.600 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.600 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:36:34.600 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.748182+0000", 2024-08-19T14:36:34.600 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.30210518400000003 2024-08-19T14:36:34.600 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.600 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.600 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:36:34.600 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.748342+0000", 2024-08-19T14:36:34.600 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00015941800000000001 2024-08-19T14:36:34.601 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.601 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.601 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 2,3", 2024-08-19T14:36:34.601 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.748506+0000", 2024-08-19T14:36:34.601 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00016431700000000001 2024-08-19T14:36:34.601 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.601 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.601 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:36:34.601 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.753468+0000", 2024-08-19T14:36:34.601 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0049616770000000003 2024-08-19T14:36:34.601 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.601 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.601 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:36:34.602 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.770233+0000", 2024-08-19T14:36:34.602 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.016765174000000001 2024-08-19T14:36:34.602 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.602 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.602 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:36:34.602 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.791094+0000", 2024-08-19T14:36:34.602 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.020861570999999999 2024-08-19T14:36:34.602 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.602 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.602 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:36:34.602 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.791161+0000", 2024-08-19T14:36:34.602 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.6308000000000003e-05 2024-08-19T14:36:34.602 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.603 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.603 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:36:34.603 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.795750+0000", 2024-08-19T14:36:34.603 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0045890260000000004 2024-08-19T14:36:34.603 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:36:34.603 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:36:34.603 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:36:34.603 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.603 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.603 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:76325 6.0 6:086941de:::benchmark_data_smithi045_97937_object76324:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e526)", 2024-08-19T14:36:34.603 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:29:17.442814+0000", 2024-08-19T14:36:34.603 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 437.12019239799997, 2024-08-19T14:36:34.603 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.35774626300000001, 2024-08-19T14:36:34.603 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:36:34.604 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:36:34.604 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:36:34.604 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:36:34.604 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:36:34.604 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:36:34.604 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 76325 2024-08-19T14:36:34.604 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.604 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:36:34.604 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.604 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:36:34.604 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.442814+0000", 2024-08-19T14:36:34.604 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:36:34.604 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.604 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.605 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:36:34.605 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.442814+0000", 2024-08-19T14:36:34.605 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:36:34.605 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.605 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.605 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:36:34.605 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.442814+0000", 2024-08-19T14:36:34.605 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.1300000000000001e-07 2024-08-19T14:36:34.605 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.605 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.605 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:36:34.605 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.442870+0000", 2024-08-19T14:36:34.605 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.5402000000000003e-05 2024-08-19T14:36:34.605 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.606 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.606 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:36:34.606 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.442878+0000", 2024-08-19T14:36:34.606 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.9910000000000002e-06 2024-08-19T14:36:34.606 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.606 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.606 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:36:34.606 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.442883+0000", 2024-08-19T14:36:34.606 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.6670000000000001e-06 2024-08-19T14:36:34.606 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.606 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.606 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:36:34.606 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.446099+0000", 2024-08-19T14:36:34.607 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0032153910000000002 2024-08-19T14:36:34.607 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.607 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.607 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:36:34.607 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.446105+0000", 2024-08-19T14:36:34.607 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.1700000000000002e-06 2024-08-19T14:36:34.607 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.607 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.607 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:36:34.607 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.753503+0000", 2024-08-19T14:36:34.607 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.30739764600000002 2024-08-19T14:36:34.607 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.608 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.608 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:36:34.608 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.753662+0000", 2024-08-19T14:36:34.608 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00015969199999999999 2024-08-19T14:36:34.608 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.608 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.608 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 2,3", 2024-08-19T14:36:34.608 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.753833+0000", 2024-08-19T14:36:34.608 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000170311 2024-08-19T14:36:34.608 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.608 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.608 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:36:34.608 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.770238+0000", 2024-08-19T14:36:34.608 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.016405685 2024-08-19T14:36:34.609 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.609 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.609 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:36:34.609 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.772478+0000", 2024-08-19T14:36:34.609 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0022397609999999998 2024-08-19T14:36:34.609 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.609 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.609 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:36:34.609 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.795837+0000", 2024-08-19T14:36:34.609 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.023359081 2024-08-19T14:36:34.609 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.609 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.609 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:36:34.609 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.795923+0000", 2024-08-19T14:36:34.609 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.6125999999999995e-05 2024-08-19T14:36:34.610 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.610 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.610 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:36:34.610 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.800560+0000", 2024-08-19T14:36:34.610 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0046369269999999999 2024-08-19T14:36:34.610 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:36:34.610 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:36:34.610 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:36:34.610 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.610 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.610 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:76309 6.a 6:5713778d:::benchmark_data_smithi045_97937_object76308:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e526)", 2024-08-19T14:36:34.610 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:29:17.447616+0000", 2024-08-19T14:36:34.610 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 437.11538985999999, 2024-08-19T14:36:34.611 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.30109721699999997, 2024-08-19T14:36:34.611 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:36:34.611 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:36:34.611 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:36:34.611 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:36:34.611 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:36:34.611 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:36:34.611 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 76309 2024-08-19T14:36:34.611 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.611 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:36:34.611 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.611 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:36:34.611 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.447616+0000", 2024-08-19T14:36:34.611 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:36:34.612 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.612 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.612 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:36:34.612 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.447616+0000", 2024-08-19T14:36:34.612 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:36:34.612 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.612 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.612 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:36:34.612 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.447618+0000", 2024-08-19T14:36:34.612 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.739e-06 2024-08-19T14:36:34.612 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.612 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.612 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:36:34.613 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.447688+0000", 2024-08-19T14:36:34.613 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.0035000000000003e-05 2024-08-19T14:36:34.613 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.613 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.613 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:36:34.613 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.447702+0000", 2024-08-19T14:36:34.613 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3608000000000001e-05 2024-08-19T14:36:34.613 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.613 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.613 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:36:34.613 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.447712+0000", 2024-08-19T14:36:34.613 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0689999999999999e-05 2024-08-19T14:36:34.613 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.613 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.614 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:36:34.614 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.451794+0000", 2024-08-19T14:36:34.614 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0040818909999999998 2024-08-19T14:36:34.614 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.614 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.614 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:36:34.614 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.451806+0000", 2024-08-19T14:36:34.614 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2021000000000001e-05 2024-08-19T14:36:34.614 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.614 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.614 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:36:34.614 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.716205+0000", 2024-08-19T14:36:34.614 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.26439825099999997 2024-08-19T14:36:34.615 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.615 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.615 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:36:34.615 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.716381+0000", 2024-08-19T14:36:34.615 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00017665899999999999 2024-08-19T14:36:34.615 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.615 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.615 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 2,5", 2024-08-19T14:36:34.615 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.716552+0000", 2024-08-19T14:36:34.615 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00017030499999999999 2024-08-19T14:36:34.615 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.615 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.615 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:36:34.615 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.725424+0000", 2024-08-19T14:36:34.616 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0088728460000000002 2024-08-19T14:36:34.616 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.616 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.616 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:36:34.616 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.728499+0000", 2024-08-19T14:36:34.616 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.003074406 2024-08-19T14:36:34.616 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.616 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.616 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:36:34.616 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.745239+0000", 2024-08-19T14:36:34.616 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.016740488000000001 2024-08-19T14:36:34.616 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.616 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.616 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:36:34.617 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.745315+0000", 2024-08-19T14:36:34.617 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.5198999999999996e-05 2024-08-19T14:36:34.617 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.617 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.617 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:36:34.617 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.748714+0000", 2024-08-19T14:36:34.617 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0033990790000000002 2024-08-19T14:36:34.617 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:36:34.617 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:36:34.617 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:36:34.617 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.617 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.617 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:76319 6.a 6:526b7568:::benchmark_data_smithi045_97937_object76318:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e526)", 2024-08-19T14:36:34.617 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:29:17.447736+0000", 2024-08-19T14:36:34.618 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 437.115269888, 2024-08-19T14:36:34.618 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.30434869799999997, 2024-08-19T14:36:34.618 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:36:34.618 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:36:34.618 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:36:34.618 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:36:34.618 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:36:34.618 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:36:34.618 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 76319 2024-08-19T14:36:34.618 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.618 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:36:34.618 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.618 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:36:34.619 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.447736+0000", 2024-08-19T14:36:34.619 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:36:34.619 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.619 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.619 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:36:34.619 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.447736+0000", 2024-08-19T14:36:34.619 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:36:34.619 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.619 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.620 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:36:34.620 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.447737+0000", 2024-08-19T14:36:34.620 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.44e-07 2024-08-19T14:36:34.620 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.620 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.620 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:36:34.620 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.447791+0000", 2024-08-19T14:36:34.620 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.3844000000000002e-05 2024-08-19T14:36:34.620 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.620 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.620 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:36:34.620 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.447804+0000", 2024-08-19T14:36:34.620 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2948999999999999e-05 2024-08-19T14:36:34.620 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.621 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.621 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:36:34.621 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.447810+0000", 2024-08-19T14:36:34.621 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.6699999999999997e-06 2024-08-19T14:36:34.621 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.621 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.621 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:36:34.621 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.451832+0000", 2024-08-19T14:36:34.621 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0040218060000000002 2024-08-19T14:36:34.621 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.621 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.621 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:36:34.621 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.451840+0000", 2024-08-19T14:36:34.621 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.9370000000000004e-06 2024-08-19T14:36:34.622 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.622 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.622 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:36:34.622 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.722373+0000", 2024-08-19T14:36:34.622 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.270533148 2024-08-19T14:36:34.622 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.622 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.622 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:36:34.622 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.722632+0000", 2024-08-19T14:36:34.622 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00025911400000000001 2024-08-19T14:36:34.622 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.622 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.622 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 2,5", 2024-08-19T14:36:34.622 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.722801+0000", 2024-08-19T14:36:34.623 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00016822899999999999 2024-08-19T14:36:34.623 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.623 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.623 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:36:34.623 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.728511+0000", 2024-08-19T14:36:34.623 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0057099280000000004 2024-08-19T14:36:34.623 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.623 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.623 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:36:34.623 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.728918+0000", 2024-08-19T14:36:34.623 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00040767000000000001 2024-08-19T14:36:34.623 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.623 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.624 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:36:34.624 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.748769+0000", 2024-08-19T14:36:34.624 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.019850390999999998 2024-08-19T14:36:34.624 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.624 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.624 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:36:34.624 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.748822+0000", 2024-08-19T14:36:34.624 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.2957000000000001e-05 2024-08-19T14:36:34.624 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.624 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.624 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:36:34.624 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.752085+0000", 2024-08-19T14:36:34.624 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0032634109999999999 2024-08-19T14:36:34.624 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:36:34.625 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:36:34.625 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:36:34.625 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.625 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.626 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:174023 6.9 6:9fb9b151:::benchmark_data_smithi045_97937_object66750:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e580)", 2024-08-19T14:36:34.626 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:32:12.445756+0000", 2024-08-19T14:36:34.626 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 262.11725052000003, 2024-08-19T14:36:34.626 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.292813779, 2024-08-19T14:36:34.626 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:36:34.626 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:36:34.626 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:36:34.626 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:36:34.626 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:36:34.626 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:36:34.626 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 174023 2024-08-19T14:36:34.627 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.627 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:36:34.627 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.627 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:36:34.627 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.445756+0000", 2024-08-19T14:36:34.627 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:36:34.627 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.627 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.627 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:36:34.627 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.445756+0000", 2024-08-19T14:36:34.627 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:36:34.627 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.627 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.627 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:36:34.628 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.445757+0000", 2024-08-19T14:36:34.628 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2130000000000001e-06 2024-08-19T14:36:34.628 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.628 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.628 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:36:34.628 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.445762+0000", 2024-08-19T14:36:34.628 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.5839999999999996e-06 2024-08-19T14:36:34.628 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.628 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.628 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:36:34.628 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.445771+0000", 2024-08-19T14:36:34.628 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.0089999999999996e-06 2024-08-19T14:36:34.628 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.628 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.629 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:36:34.629 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.445775+0000", 2024-08-19T14:36:34.629 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7219999999999999e-06 2024-08-19T14:36:34.629 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.629 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.629 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:36:34.629 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.445819+0000", 2024-08-19T14:36:34.629 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.3497e-05 2024-08-19T14:36:34.629 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.629 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.629 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:36:34.629 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.445903+0000", 2024-08-19T14:36:34.629 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.4499999999999994e-05 2024-08-19T14:36:34.630 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.630 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.630 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 1,8", 2024-08-19T14:36:34.630 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.445973+0000", 2024-08-19T14:36:34.630 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.0239000000000003e-05 2024-08-19T14:36:34.630 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.630 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.630 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:36:34.630 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.449815+0000", 2024-08-19T14:36:34.630 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.003841461 2024-08-19T14:36:34.630 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.630 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.631 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:36:34.631 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.449950+0000", 2024-08-19T14:36:34.631 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000135377 2024-08-19T14:36:34.631 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.631 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.631 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:36:34.631 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.450730+0000", 2024-08-19T14:36:34.631 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00077977600000000004 2024-08-19T14:36:34.631 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.631 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.631 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:36:34.631 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.450763+0000", 2024-08-19T14:36:34.631 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.3161000000000003e-05 2024-08-19T14:36:34.632 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.632 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.632 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:36:34.632 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.738570+0000", 2024-08-19T14:36:34.632 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.28780623999999999 2024-08-19T14:36:34.632 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:36:34.632 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:36:34.632 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:36:34.632 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.632 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.632 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop_reply(client.26191.0:174023 6.9 e580/566 ondisk, result = 0)", 2024-08-19T14:36:34.632 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:32:12.450667+0000", 2024-08-19T14:36:34.632 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 262.11233962799997, 2024-08-19T14:36:34.632 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.28791044700000001, 2024-08-19T14:36:34.633 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:36:34.633 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:36:34.633 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-08-19T14:36:34.633 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:36:34.633 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.633 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:36:34.633 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.450667+0000", 2024-08-19T14:36:34.633 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:36:34.633 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.633 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.633 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:36:34.633 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.450667+0000", 2024-08-19T14:36:34.633 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:36:34.634 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.634 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.634 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:36:34.634 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.450667+0000", 2024-08-19T14:36:34.634 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.6100000000000001e-07 2024-08-19T14:36:34.634 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.634 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.634 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:36:34.634 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.450671+0000", 2024-08-19T14:36:34.634 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.4769999999999999e-06 2024-08-19T14:36:34.634 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.634 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.635 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:36:34.635 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.450682+0000", 2024-08-19T14:36:34.635 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1151e-05 2024-08-19T14:36:34.635 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.635 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.635 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:36:34.635 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.450687+0000", 2024-08-19T14:36:34.635 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7779999999999998e-06 2024-08-19T14:36:34.635 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.635 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.635 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:36:34.635 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.450722+0000", 2024-08-19T14:36:34.635 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.5212000000000001e-05 2024-08-19T14:36:34.636 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.636 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.636 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:36:34.636 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.450727+0000", 2024-08-19T14:36:34.636 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.8749999999999999e-06 2024-08-19T14:36:34.636 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.636 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.636 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:36:34.636 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.738577+0000", 2024-08-19T14:36:34.636 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.28785039299999998 2024-08-19T14:36:34.636 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:36:34.636 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:36:34.637 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:36:34.637 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.637 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.637 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.26191.0:174030 6.d e580/510 6:b7de3b0b:::benchmark_data_smithi045_97937_object66757:head v 580'10851, mlcod=580'10849)", 2024-08-19T14:36:34.637 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:32:12.457359+0000", 2024-08-19T14:36:34.637 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 262.10564719299998, 2024-08-19T14:36:34.637 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.31230491700000002, 2024-08-19T14:36:34.637 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:36:34.637 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:36:34.637 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:36:34.637 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:36:34.637 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.637 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:36:34.638 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.457359+0000", 2024-08-19T14:36:34.638 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:36:34.638 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.638 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.638 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:36:34.638 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.457359+0000", 2024-08-19T14:36:34.638 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:36:34.638 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.638 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.638 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:36:34.638 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.457361+0000", 2024-08-19T14:36:34.638 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.528e-06 2024-08-19T14:36:34.638 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.639 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.639 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:36:34.639 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.457370+0000", 2024-08-19T14:36:34.639 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.2180000000000005e-06 2024-08-19T14:36:34.639 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.639 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.639 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:36:34.639 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.738511+0000", 2024-08-19T14:36:34.639 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.28114076599999999 2024-08-19T14:36:34.639 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.639 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.639 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:36:34.639 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.738520+0000", 2024-08-19T14:36:34.639 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.5400000000000001e-06 2024-08-19T14:36:34.640 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.640 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.640 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:36:34.640 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.738582+0000", 2024-08-19T14:36:34.640 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.1537000000000004e-05 2024-08-19T14:36:34.640 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.640 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.640 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:36:34.640 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.738606+0000", 2024-08-19T14:36:34.640 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.4678e-05 2024-08-19T14:36:34.640 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.640 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.640 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:36:34.640 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.769625+0000", 2024-08-19T14:36:34.641 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.031018782000000002 2024-08-19T14:36:34.641 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.641 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.641 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:36:34.641 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.769664+0000", 2024-08-19T14:36:34.641 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.8868000000000001e-05 2024-08-19T14:36:34.641 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:36:34.641 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:36:34.641 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:36:34.641 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.641 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.641 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:174037 6.a 6:572c0c1a:::benchmark_data_smithi045_97937_object66764:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e580)", 2024-08-19T14:36:34.641 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:32:12.467095+0000", 2024-08-19T14:36:34.642 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 262.09591160100001, 2024-08-19T14:36:34.642 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.28614824, 2024-08-19T14:36:34.642 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:36:34.642 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:36:34.642 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:36:34.642 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:36:34.642 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:36:34.642 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:36:34.642 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 174037 2024-08-19T14:36:34.642 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.642 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:36:34.642 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.643 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:36:34.643 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.467095+0000", 2024-08-19T14:36:34.643 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:36:34.643 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.643 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.643 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:36:34.643 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.467095+0000", 2024-08-19T14:36:34.643 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:36:34.643 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.643 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.643 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:36:34.643 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.467096+0000", 2024-08-19T14:36:34.643 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.773e-06 2024-08-19T14:36:34.643 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.643 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.644 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:36:34.644 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.467104+0000", 2024-08-19T14:36:34.644 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.5569999999999998e-06 2024-08-19T14:36:34.644 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.644 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.644 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:36:34.644 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.738490+0000", 2024-08-19T14:36:34.644 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.27138640000000003 2024-08-19T14:36:34.644 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.644 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.644 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:36:34.644 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.738508+0000", 2024-08-19T14:36:34.644 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.7618e-05 2024-08-19T14:36:34.645 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.645 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.645 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:36:34.645 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.738578+0000", 2024-08-19T14:36:34.645 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.9646000000000004e-05 2024-08-19T14:36:34.645 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.645 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.645 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:36:34.645 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.738709+0000", 2024-08-19T14:36:34.645 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013147699999999999 2024-08-19T14:36:34.645 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.645 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.645 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 1,5", 2024-08-19T14:36:34.645 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.738832+0000", 2024-08-19T14:36:34.646 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000123168 2024-08-19T14:36:34.646 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.646 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.646 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:36:34.646 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.748897+0000", 2024-08-19T14:36:34.646 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.010065203999999999 2024-08-19T14:36:34.646 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.646 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.646 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:36:34.646 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.753055+0000", 2024-08-19T14:36:34.646 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0041572880000000003 2024-08-19T14:36:34.646 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.646 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.647 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:36:34.647 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.753071+0000", 2024-08-19T14:36:34.647 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6103000000000002e-05 2024-08-19T14:36:34.647 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.647 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.647 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:36:34.647 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.753106+0000", 2024-08-19T14:36:34.647 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.5447999999999997e-05 2024-08-19T14:36:34.647 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.647 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.647 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:36:34.647 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.753243+0000", 2024-08-19T14:36:34.647 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000136558 2024-08-19T14:36:34.648 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:36:34.648 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:36:34.648 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:36:34.648 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.648 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.648 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:186725 6.a 6:5538fe55:::benchmark_data_smithi045_97937_object79452:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e583)", 2024-08-19T14:36:34.648 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:32:31.435725+0000", 2024-08-19T14:36:34.648 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 243.127280985, 2024-08-19T14:36:34.648 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.28468506500000001, 2024-08-19T14:36:34.648 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:36:34.648 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:36:34.648 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:36:34.648 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:36:34.648 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:36:34.649 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:36:34.649 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 186725 2024-08-19T14:36:34.649 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.649 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:36:34.649 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.649 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:36:34.649 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.435725+0000", 2024-08-19T14:36:34.649 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:36:34.649 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.649 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.649 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:36:34.649 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.435725+0000", 2024-08-19T14:36:34.649 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:36:34.649 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.650 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.650 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:36:34.650 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.435726+0000", 2024-08-19T14:36:34.650 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.7199999999999998e-07 2024-08-19T14:36:34.650 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.650 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.650 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:36:34.650 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.435730+0000", 2024-08-19T14:36:34.650 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.8530000000000002e-06 2024-08-19T14:36:34.650 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.650 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.650 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:36:34.650 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.435746+0000", 2024-08-19T14:36:34.651 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6078000000000001e-05 2024-08-19T14:36:34.651 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.651 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.651 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:36:34.651 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.435753+0000", 2024-08-19T14:36:34.651 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.2860000000000002e-06 2024-08-19T14:36:34.651 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.651 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.651 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:36:34.651 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.542354+0000", 2024-08-19T14:36:34.651 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.106601 2024-08-19T14:36:34.651 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.651 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.651 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:36:34.652 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.542464+0000", 2024-08-19T14:36:34.652 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00010945 2024-08-19T14:36:34.652 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.652 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.652 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 1,5", 2024-08-19T14:36:34.652 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.542572+0000", 2024-08-19T14:36:34.652 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00010788 2024-08-19T14:36:34.652 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.652 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.652 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:36:34.652 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.553201+0000", 2024-08-19T14:36:34.652 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.010629868000000001 2024-08-19T14:36:34.652 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.653 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.653 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:36:34.653 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.553326+0000", 2024-08-19T14:36:34.653 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012446700000000001 2024-08-19T14:36:34.653 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.653 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.653 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:36:34.653 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.720241+0000", 2024-08-19T14:36:34.653 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.16691509299999999 2024-08-19T14:36:34.653 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.653 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.653 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:36:34.653 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.720298+0000", 2024-08-19T14:36:34.653 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.6777000000000002e-05 2024-08-19T14:36:34.654 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.654 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.654 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:36:34.654 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.720410+0000", 2024-08-19T14:36:34.654 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00011254100000000001 2024-08-19T14:36:34.654 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:36:34.654 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:36:34.654 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:36:34.654 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.654 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.654 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:186728 6.a 6:52d9dad4:::benchmark_data_smithi045_97937_object79455:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e583)", 2024-08-19T14:36:34.654 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:32:31.436585+0000", 2024-08-19T14:36:34.654 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 243.126421714, 2024-08-19T14:36:34.655 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.28396986200000002, 2024-08-19T14:36:34.655 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:36:34.655 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:36:34.655 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:36:34.655 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:36:34.655 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:36:34.655 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:36:34.655 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 186728 2024-08-19T14:36:34.655 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.655 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:36:34.655 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.655 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:36:34.656 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.436585+0000", 2024-08-19T14:36:34.656 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:36:34.656 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.656 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.656 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:36:34.656 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.436585+0000", 2024-08-19T14:36:34.656 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:36:34.656 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.656 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.656 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:36:34.656 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.436585+0000", 2024-08-19T14:36:34.656 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.9100000000000003e-07 2024-08-19T14:36:34.656 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.656 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.657 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:36:34.657 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.436590+0000", 2024-08-19T14:36:34.657 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.5210000000000004e-06 2024-08-19T14:36:34.657 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.657 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.657 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:36:34.657 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.436603+0000", 2024-08-19T14:36:34.657 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3254999999999999e-05 2024-08-19T14:36:34.657 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.657 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.657 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:36:34.657 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.436610+0000", 2024-08-19T14:36:34.657 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.8090000000000004e-06 2024-08-19T14:36:34.657 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.658 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.658 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:36:34.658 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.548046+0000", 2024-08-19T14:36:34.658 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.111436379 2024-08-19T14:36:34.658 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.658 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.658 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:36:34.658 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.548154+0000", 2024-08-19T14:36:34.658 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000107911 2024-08-19T14:36:34.658 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.658 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.658 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 1,5", 2024-08-19T14:36:34.658 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.548291+0000", 2024-08-19T14:36:34.659 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013718700000000001 2024-08-19T14:36:34.659 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.659 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.659 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:36:34.659 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.563156+0000", 2024-08-19T14:36:34.659 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.014865094000000001 2024-08-19T14:36:34.659 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.659 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.659 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:36:34.659 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.563191+0000", 2024-08-19T14:36:34.659 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.4768000000000003e-05 2024-08-19T14:36:34.659 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.659 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.659 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:36:34.660 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.720452+0000", 2024-08-19T14:36:34.660 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.15726029499999999 2024-08-19T14:36:34.660 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.660 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.660 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:36:34.660 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.720486+0000", 2024-08-19T14:36:34.660 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.4805999999999998e-05 2024-08-19T14:36:34.660 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.660 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.660 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:36:34.660 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.720554+0000", 2024-08-19T14:36:34.660 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.8046000000000005e-05 2024-08-19T14:36:34.660 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:36:34.660 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:36:34.660 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:36:34.661 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.661 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.661 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:186730 6.a 6:53c81287:::benchmark_data_smithi045_97937_object79457:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e583)", 2024-08-19T14:36:34.661 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:32:31.440667+0000", 2024-08-19T14:36:34.661 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 243.12233953699999, 2024-08-19T14:36:34.661 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.28015182900000002, 2024-08-19T14:36:34.661 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:36:34.661 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:36:34.661 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:36:34.661 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:36:34.661 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:36:34.661 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:36:34.661 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 186730 2024-08-19T14:36:34.661 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.662 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:36:34.662 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.662 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:36:34.662 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.440667+0000", 2024-08-19T14:36:34.662 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:36:34.662 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.662 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.662 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:36:34.662 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.440667+0000", 2024-08-19T14:36:34.662 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:36:34.662 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.662 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.662 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:36:34.663 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.440668+0000", 2024-08-19T14:36:34.663 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.395e-06 2024-08-19T14:36:34.663 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.663 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.663 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:36:34.663 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.440673+0000", 2024-08-19T14:36:34.663 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7520000000000004e-06 2024-08-19T14:36:34.663 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.663 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.663 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:36:34.663 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.440681+0000", 2024-08-19T14:36:34.663 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.4999999999999999e-06 2024-08-19T14:36:34.663 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.663 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.664 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:36:34.664 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.440688+0000", 2024-08-19T14:36:34.664 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.3099999999999997e-06 2024-08-19T14:36:34.664 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.664 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.664 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:36:34.664 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.553367+0000", 2024-08-19T14:36:34.664 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.112679032 2024-08-19T14:36:34.664 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.664 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.664 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:36:34.664 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.553507+0000", 2024-08-19T14:36:34.664 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00014019600000000001 2024-08-19T14:36:34.665 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.665 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.665 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 1,5", 2024-08-19T14:36:34.665 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.553643+0000", 2024-08-19T14:36:34.665 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013585800000000001 2024-08-19T14:36:34.665 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.665 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.665 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:36:34.665 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.563160+0000", 2024-08-19T14:36:34.665 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0095170180000000004 2024-08-19T14:36:34.665 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.665 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.665 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:36:34.665 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.563322+0000", 2024-08-19T14:36:34.666 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00016175299999999999 2024-08-19T14:36:34.666 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.666 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.666 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:36:34.666 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.720659+0000", 2024-08-19T14:36:34.666 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.15733791599999999 2024-08-19T14:36:34.666 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.666 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.666 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:36:34.666 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.720710+0000", 2024-08-19T14:36:34.666 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0176999999999997e-05 2024-08-19T14:36:34.666 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.667 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.667 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:36:34.667 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.720819+0000", 2024-08-19T14:36:34.667 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000108922 2024-08-19T14:36:34.667 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:36:34.667 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:36:34.667 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:36:34.667 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.667 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.667 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:186734 6.a 6:55f30169:::benchmark_data_smithi045_97937_object79461:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e583)", 2024-08-19T14:36:34.667 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:32:31.443487+0000", 2024-08-19T14:36:34.667 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 243.119519414, 2024-08-19T14:36:34.667 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.27757852900000002, 2024-08-19T14:36:34.667 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:36:34.667 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:36:34.668 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:36:34.668 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:36:34.668 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:36:34.668 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:36:34.668 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 186734 2024-08-19T14:36:34.668 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.668 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:36:34.668 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.668 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:36:34.668 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.443487+0000", 2024-08-19T14:36:34.668 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:36:34.668 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.668 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.668 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:36:34.669 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.443487+0000", 2024-08-19T14:36:34.669 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:36:34.670 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:36:34.670 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.670 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.670 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:36:34.670 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.443488+0000", 2024-08-19T14:36:34.670 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1710000000000001e-06 2024-08-19T14:36:34.671 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.671 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.671 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:36:34.671 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.443493+0000", 2024-08-19T14:36:34.671 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.4920000000000001e-06 2024-08-19T14:36:34.671 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.671 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.671 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:36:34.671 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.443509+0000", 2024-08-19T14:36:34.671 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5824999999999998e-05 2024-08-19T14:36:34.671 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.671 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.671 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:36:34.671 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.443516+0000", 2024-08-19T14:36:34.672 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.5119999999999997e-06 2024-08-19T14:36:34.672 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.672 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.672 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:36:34.672 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.557757+0000", 2024-08-19T14:36:34.672 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.114241205 2024-08-19T14:36:34.672 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.672 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.672 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:36:34.672 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.557880+0000", 2024-08-19T14:36:34.672 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012296799999999999 2024-08-19T14:36:34.672 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.672 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.672 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 1,5", 2024-08-19T14:36:34.673 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.558022+0000", 2024-08-19T14:36:34.673 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00014233299999999999 2024-08-19T14:36:34.673 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.673 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.673 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:36:34.673 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.563206+0000", 2024-08-19T14:36:34.673 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0051838509999999997 2024-08-19T14:36:34.673 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.673 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.673 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:36:34.673 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.567804+0000", 2024-08-19T14:36:34.673 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0045981900000000003 2024-08-19T14:36:34.673 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.674 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.674 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:36:34.674 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.720909+0000", 2024-08-19T14:36:34.674 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.15310435999999999 2024-08-19T14:36:34.674 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.674 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.674 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:36:34.674 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.720963+0000", 2024-08-19T14:36:34.674 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.3999000000000003e-05 2024-08-19T14:36:34.674 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.674 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.674 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:36:34.675 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.721065+0000", 2024-08-19T14:36:34.675 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000102623 2024-08-19T14:36:34.675 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:36:34.675 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:36:34.675 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:36:34.675 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.675 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.675 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:186742 6.a 6:57b0fc9a:::benchmark_data_smithi045_97937_object79469:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e583)", 2024-08-19T14:36:34.675 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:32:31.455602+0000", 2024-08-19T14:36:34.675 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 243.10740428099999, 2024-08-19T14:36:34.675 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.265680048, 2024-08-19T14:36:34.675 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:36:34.675 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:36:34.675 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:36:34.676 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:36:34.676 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:36:34.676 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:36:34.676 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 186742 2024-08-19T14:36:34.676 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.676 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:36:34.676 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.676 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:36:34.676 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.455602+0000", 2024-08-19T14:36:34.676 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:36:34.676 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.676 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.676 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:36:34.676 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.455602+0000", 2024-08-19T14:36:34.677 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:36:34.677 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.677 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.677 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:36:34.677 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.455603+0000", 2024-08-19T14:36:34.677 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4449999999999999e-06 2024-08-19T14:36:34.677 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.677 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.677 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:36:34.677 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.455608+0000", 2024-08-19T14:36:34.677 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.6419999999999998e-06 2024-08-19T14:36:34.677 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.677 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.677 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:36:34.678 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.455621+0000", 2024-08-19T14:36:34.678 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2839e-05 2024-08-19T14:36:34.678 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.678 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.678 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:36:34.678 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.455628+0000", 2024-08-19T14:36:34.678 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.8730000000000001e-06 2024-08-19T14:36:34.678 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.678 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.678 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:36:34.678 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.563374+0000", 2024-08-19T14:36:34.678 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.107745823 2024-08-19T14:36:34.678 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.678 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.679 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:36:34.679 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.563468+0000", 2024-08-19T14:36:34.679 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.4765000000000005e-05 2024-08-19T14:36:34.679 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.679 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.679 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 1,5", 2024-08-19T14:36:34.679 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.563536+0000", 2024-08-19T14:36:34.679 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.8077999999999994e-05 2024-08-19T14:36:34.679 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.679 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.679 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:36:34.679 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.568259+0000", 2024-08-19T14:36:34.679 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0047226550000000001 2024-08-19T14:36:34.679 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.680 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.680 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:36:34.680 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.568693+0000", 2024-08-19T14:36:34.680 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00043382700000000002 2024-08-19T14:36:34.680 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.680 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.680 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:36:34.680 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.721128+0000", 2024-08-19T14:36:34.680 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.152435293 2024-08-19T14:36:34.680 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.680 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.680 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:36:34.680 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.721179+0000", 2024-08-19T14:36:34.680 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1029000000000002e-05 2024-08-19T14:36:34.680 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.681 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.681 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:36:34.681 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:31.721282+0000", 2024-08-19T14:36:34.681 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000102779 2024-08-19T14:36:34.681 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:36:34.681 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:36:34.681 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:36:34.681 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.681 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.681 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:187356 6.a 6:575cb98d:::benchmark_data_smithi045_97937_object80083:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e583)", 2024-08-19T14:36:34.681 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:32:32.420297+0000", 2024-08-19T14:36:34.681 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 242.142709142, 2024-08-19T14:36:34.681 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.34771512199999999, 2024-08-19T14:36:34.681 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:36:34.682 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:36:34.682 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:36:34.682 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:36:34.682 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:36:34.682 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:36:34.682 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 187356 2024-08-19T14:36:34.682 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.682 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:36:34.682 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.682 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:36:34.682 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.420297+0000", 2024-08-19T14:36:34.682 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:36:34.682 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.682 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.683 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:36:34.683 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.420297+0000", 2024-08-19T14:36:34.683 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:36:34.683 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.683 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.683 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:36:34.683 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.420298+0000", 2024-08-19T14:36:34.683 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.8100000000000001e-07 2024-08-19T14:36:34.683 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.683 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.683 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:36:34.683 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.420302+0000", 2024-08-19T14:36:34.683 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.374e-06 2024-08-19T14:36:34.683 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.683 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.684 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:36:34.684 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.420317+0000", 2024-08-19T14:36:34.684 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4683e-05 2024-08-19T14:36:34.684 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.684 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.684 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:36:34.684 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.420324+0000", 2024-08-19T14:36:34.684 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.3869999999999997e-06 2024-08-19T14:36:34.684 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.684 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.684 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:36:34.684 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.551342+0000", 2024-08-19T14:36:34.684 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.13101795199999999 2024-08-19T14:36:34.684 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.684 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.685 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:36:34.685 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.551468+0000", 2024-08-19T14:36:34.685 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012597899999999999 2024-08-19T14:36:34.685 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.685 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.685 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 1,5", 2024-08-19T14:36:34.685 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.551599+0000", 2024-08-19T14:36:34.685 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000130369 2024-08-19T14:36:34.685 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.685 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.685 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:36:34.685 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.557051+0000", 2024-08-19T14:36:34.685 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0054523949999999996 2024-08-19T14:36:34.685 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.685 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.685 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:36:34.686 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.560585+0000", 2024-08-19T14:36:34.686 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0035343240000000001 2024-08-19T14:36:34.686 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.686 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.686 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:36:34.686 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.767809+0000", 2024-08-19T14:36:34.686 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.20722385400000001 2024-08-19T14:36:34.686 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.686 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.686 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:36:34.686 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.767907+0000", 2024-08-19T14:36:34.686 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.7996000000000002e-05 2024-08-19T14:36:34.687 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.687 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.687 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:36:34.687 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.768012+0000", 2024-08-19T14:36:34.687 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000104928 2024-08-19T14:36:34.687 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:36:34.687 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:36:34.687 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:36:34.687 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.687 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.687 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:187395 6.a 6:57126a4f:::benchmark_data_smithi045_97937_object80122:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e583)", 2024-08-19T14:36:34.687 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:32:32.474580+0000", 2024-08-19T14:36:34.687 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 242.08842664299999, 2024-08-19T14:36:34.687 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.293779544, 2024-08-19T14:36:34.688 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:36:34.688 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:36:34.688 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:36:34.688 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:36:34.688 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:36:34.688 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:36:34.688 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 187395 2024-08-19T14:36:34.688 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.688 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:36:34.688 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.688 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:36:34.688 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.474580+0000", 2024-08-19T14:36:34.688 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:36:34.688 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.689 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.689 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:36:34.689 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.474580+0000", 2024-08-19T14:36:34.689 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:36:34.689 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.689 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.689 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:36:34.689 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.474581+0000", 2024-08-19T14:36:34.689 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.268e-06 2024-08-19T14:36:34.689 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.689 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.689 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:36:34.689 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.474585+0000", 2024-08-19T14:36:34.689 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.3329999999999998e-06 2024-08-19T14:36:34.690 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.690 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.690 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:36:34.690 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.474600+0000", 2024-08-19T14:36:34.690 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5274999999999999e-05 2024-08-19T14:36:34.690 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.690 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.690 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:36:34.690 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.474609+0000", 2024-08-19T14:36:34.690 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.1950000000000003e-06 2024-08-19T14:36:34.690 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.690 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.690 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:36:34.691 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.560680+0000", 2024-08-19T14:36:34.691 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.086071163000000006 2024-08-19T14:36:34.691 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.691 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.691 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:36:34.691 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.560822+0000", 2024-08-19T14:36:34.691 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000141917 2024-08-19T14:36:34.691 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.691 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.691 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 1,5", 2024-08-19T14:36:34.691 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.560968+0000", 2024-08-19T14:36:34.691 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00014591300000000001 2024-08-19T14:36:34.691 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.691 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.692 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:36:34.692 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.576675+0000", 2024-08-19T14:36:34.692 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.015707216999999999 2024-08-19T14:36:34.692 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.692 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.692 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:36:34.692 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.576890+0000", 2024-08-19T14:36:34.692 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000214959 2024-08-19T14:36:34.692 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.692 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.692 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:36:34.692 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.768143+0000", 2024-08-19T14:36:34.692 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.191252961 2024-08-19T14:36:34.693 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.693 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.693 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:36:34.693 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.768243+0000", 2024-08-19T14:36:34.693 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000100589 2024-08-19T14:36:34.693 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.693 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.693 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:36:34.693 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.768359+0000", 2024-08-19T14:36:34.693 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000115754 2024-08-19T14:36:34.693 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:36:34.693 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:36:34.693 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:36:34.693 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.694 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.694 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:187415 6.a 6:512e1698:::benchmark_data_smithi045_97937_object80142:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e583)", 2024-08-19T14:36:34.694 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:32:32.505253+0000", 2024-08-19T14:36:34.694 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 242.05775321499999, 2024-08-19T14:36:34.694 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.26330838699999998, 2024-08-19T14:36:34.694 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:36:34.694 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:36:34.694 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:36:34.694 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:36:34.694 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:36:34.694 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:36:34.694 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 187415 2024-08-19T14:36:34.694 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.695 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:36:34.695 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.695 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:36:34.695 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.505253+0000", 2024-08-19T14:36:34.695 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:36:34.695 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.695 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.695 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:36:34.695 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.505253+0000", 2024-08-19T14:36:34.695 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:36:34.695 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.695 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.695 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:36:34.695 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.505254+0000", 2024-08-19T14:36:34.696 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2979999999999999e-06 2024-08-19T14:36:34.696 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.696 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.696 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:36:34.696 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.505259+0000", 2024-08-19T14:36:34.696 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1540000000000001e-06 2024-08-19T14:36:34.696 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.696 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.696 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:36:34.696 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.505275+0000", 2024-08-19T14:36:34.696 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5625999999999999e-05 2024-08-19T14:36:34.696 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.696 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.696 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:36:34.697 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.505282+0000", 2024-08-19T14:36:34.697 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.097e-06 2024-08-19T14:36:34.697 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.697 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.697 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:36:34.697 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.565866+0000", 2024-08-19T14:36:34.697 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.060583692000000001 2024-08-19T14:36:34.697 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.697 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.697 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:36:34.697 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.566565+0000", 2024-08-19T14:36:34.697 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00069894499999999997 2024-08-19T14:36:34.697 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.698 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.698 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 1,5", 2024-08-19T14:36:34.698 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.566749+0000", 2024-08-19T14:36:34.698 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00018427999999999999 2024-08-19T14:36:34.698 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.698 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.698 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:36:34.698 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.576876+0000", 2024-08-19T14:36:34.698 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.010126476000000001 2024-08-19T14:36:34.698 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.698 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.698 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:36:34.699 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.576898+0000", 2024-08-19T14:36:34.699 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.2382999999999999e-05 2024-08-19T14:36:34.699 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.699 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.699 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:36:34.699 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.768415+0000", 2024-08-19T14:36:34.699 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.19151680400000001 2024-08-19T14:36:34.699 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.699 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.699 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:36:34.699 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.768467+0000", 2024-08-19T14:36:34.699 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1916000000000002e-05 2024-08-19T14:36:34.699 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:36:34.699 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:36:34.699 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:36:34.699 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.768561+0000", 2024-08-19T14:36:34.700 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.4716000000000006e-05 2024-08-19T14:36:34.700 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:36:34.700 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:36:34.700 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:36:34.700 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:36:34.700 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:36:34.700 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:36:34.745 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:34.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:34.746 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:36:34.846 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:36:34.868 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:34.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:34.869 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:36:34.970 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:36:34.993 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:34.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:34.994 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:36:35.095 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:36:35.143 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:35.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:35.144 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:36:35.245 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:36:35.269 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:35.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:35.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:36:35.371 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:36:35.393 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:35.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:35.394 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:36:35.434 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd reweight 5 1 2024-08-19T14:36:35.477 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:35 smithi031 ceph-mon[16336]: 1.0 scrub starts 2024-08-19T14:36:35.477 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:35 smithi031 ceph-mon[16336]: 1.0 scrub ok 2024-08-19T14:36:35.477 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:35 smithi031 ceph-mon[16336]: pgmap v1559: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 5.1 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 392 op/s; 28023/250107 objects degraded (11.204%); 9508/250107 objects misplaced (3.802%); 6.8 MiB/s, 110 objects/s recovering 2024-08-19T14:36:35.478 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:35 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-08-19T14:36:35.478 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:35 smithi031 ceph-mon[16336]: osdmap e660: 12 total, 12 up, 10 in 2024-08-19T14:36:35.478 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:35 smithi031 ceph-mon[16336]: 7.7 scrub starts 2024-08-19T14:36:35.495 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:36:35.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:35 smithi032 ceph-mon[20004]: 1.0 scrub starts 2024-08-19T14:36:35.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:35 smithi032 ceph-mon[20004]: 1.0 scrub ok 2024-08-19T14:36:35.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:35 smithi032 ceph-mon[20004]: pgmap v1559: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 5.1 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 392 op/s; 28023/250107 objects degraded (11.204%); 9508/250107 objects misplaced (3.802%); 6.8 MiB/s, 110 objects/s recovering 2024-08-19T14:36:35.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:35 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-08-19T14:36:35.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:35 smithi032 ceph-mon[20004]: osdmap e660: 12 total, 12 up, 10 in 2024-08-19T14:36:35.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:35 smithi032 ceph-mon[20004]: 7.7 scrub starts 2024-08-19T14:36:35.516 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:35.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:35.517 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:36:35.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:35 smithi045 ceph-mon[20143]: 1.0 scrub starts 2024-08-19T14:36:35.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:35 smithi045 ceph-mon[20143]: 1.0 scrub ok 2024-08-19T14:36:35.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:35 smithi045 ceph-mon[20143]: pgmap v1559: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 5.1 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 392 op/s; 28023/250107 objects degraded (11.204%); 9508/250107 objects misplaced (3.802%); 6.8 MiB/s, 110 objects/s recovering 2024-08-19T14:36:35.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:35 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-08-19T14:36:35.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:35 smithi045 ceph-mon[20143]: osdmap e660: 12 total, 12 up, 10 in 2024-08-19T14:36:35.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:35 smithi045 ceph-mon[20143]: 7.7 scrub starts 2024-08-19T14:36:35.607 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:36:35.618 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:36:35.639 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:35.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:35.640 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:36:35.694 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:36:35.741 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:36:35.762 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:35.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:35.763 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:36:35.864 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:36:35.907 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:35.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:35.908 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:36:36.009 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:36:36.033 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:36.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:36.034 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:36:36.135 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:36:36.160 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-08-19T14:36:36.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:36.197 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:36.198 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:36:36.298 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:36:36.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:36.344 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:36.344 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:36:36.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:36 smithi031 ceph-mon[16336]: 7.4 scrub ok 2024-08-19T14:36:36.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:36 smithi031 ceph-mon[16336]: 1.6 scrub starts 2024-08-19T14:36:36.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:36 smithi031 ceph-mon[16336]: 1.6 scrub ok 2024-08-19T14:36:36.445 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:36:36.497 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:36.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:36.502 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:36:36.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:36 smithi032 ceph-mon[20004]: 7.4 scrub ok 2024-08-19T14:36:36.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:36 smithi032 ceph-mon[20004]: 1.6 scrub starts 2024-08-19T14:36:36.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:36 smithi032 ceph-mon[20004]: 1.6 scrub ok 2024-08-19T14:36:36.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:36 smithi045 ceph-mon[20143]: 7.4 scrub ok 2024-08-19T14:36:36.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:36 smithi045 ceph-mon[20143]: 1.6 scrub starts 2024-08-19T14:36:36.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:36 smithi045 ceph-mon[20143]: 1.6 scrub ok 2024-08-19T14:36:36.603 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:36:36.625 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:36.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:36.626 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:36:36.727 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:36:36.748 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:36.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:36.749 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:36:36.850 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:36:36.872 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:36.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:36.873 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:36:36.974 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:36:36.996 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:36.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:36.997 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:36:37.098 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:36:37.141 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:37.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:37.142 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:36:37.243 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:36:37.281 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:37.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:37.282 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:36:37.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:37 smithi031 ceph-mon[16336]: 7.7 scrub ok 2024-08-19T14:36:37.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:37 smithi031 ceph-mon[16336]: pgmap v1561: 47 pgs: 2 active+clean+scrubbing, 6 active+recovering+undersized+degraded+remapped, 39 active+clean; 5.0 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 413 op/s; 27782/246432 objects degraded (11.274%); 9453/246432 objects misplaced (3.836%); 8.7 MiB/s, 140 objects/s recovering 2024-08-19T14:36:37.383 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:36:37.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:37.420 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:37.420 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:36:37.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:37 smithi032 ceph-mon[20004]: 7.7 scrub ok 2024-08-19T14:36:37.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:37 smithi032 ceph-mon[20004]: pgmap v1561: 47 pgs: 2 active+clean+scrubbing, 6 active+recovering+undersized+degraded+remapped, 39 active+clean; 5.0 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 413 op/s; 27782/246432 objects degraded (11.274%); 9453/246432 objects misplaced (3.836%); 8.7 MiB/s, 140 objects/s recovering 2024-08-19T14:36:37.521 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:36:37.542 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:37.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:37.543 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:36:37.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:37 smithi045 ceph-mon[20143]: 7.7 scrub ok 2024-08-19T14:36:37.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:37 smithi045 ceph-mon[20143]: pgmap v1561: 47 pgs: 2 active+clean+scrubbing, 6 active+recovering+undersized+degraded+remapped, 39 active+clean; 5.0 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 413 op/s; 27782/246432 objects degraded (11.274%); 9453/246432 objects misplaced (3.836%); 8.7 MiB/s, 140 objects/s recovering 2024-08-19T14:36:37.644 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:36:37.669 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:37.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:37.670 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:36:37.770 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:36:37.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:37.827 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:37.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:36:37.928 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:36:37.951 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:37.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:37.952 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:36:38.053 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:36:38.074 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:38.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:38.075 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:36:38.176 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:36:38.198 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:38.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:38.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:36:38.300 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:36:38.322 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:38.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:38.324 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:36:38.425 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:36:38.446 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:38.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:38.447 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:36:38.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:38 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/958279799' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:36:38.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:38 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 27782/246432 objects degraded (11.274%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-19T14:36:38.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:38 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/958279799' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:36:38.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:38 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 27782/246432 objects degraded (11.274%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-19T14:36:38.547 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:36:38.571 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is unset 2024-08-19T14:36:38.571 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:38 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/958279799' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:36:38.572 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:38 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 27782/246432 objects degraded (11.274%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-19T14:36:38.588 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:38.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:38.589 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:36:38.690 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:36:38.712 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:38.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:38.713 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:36:38.814 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:36:38.837 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:38.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:38.838 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:36:38.939 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:36:38.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:38.963 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:38.963 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:36:39.064 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:36:39.086 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:39.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:39.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:36:39.188 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:36:39.210 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:39.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:39.211 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:36:39.312 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:36:39.393 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:39.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:39.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:36:39.496 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:36:39.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:39 smithi032 ceph-mon[20004]: pgmap v1562: 47 pgs: 2 active+clean+scrubbing, 6 active+recovering+undersized+degraded+remapped, 39 active+clean; 5.0 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 445 op/s; 27569/244284 objects degraded (11.286%); 9281/244284 objects misplaced (3.799%); 8.7 MiB/s, 141 objects/s recovering 2024-08-19T14:36:39.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:39 smithi032 ceph-mon[20004]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:36:39.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:39 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/958279799' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:36:39.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:39 smithi032 ceph-mon[20004]: osdmap e661: 12 total, 12 up, 10 in 2024-08-19T14:36:39.534 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:39.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:39.535 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:36:39.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:39 smithi045 ceph-mon[20143]: pgmap v1562: 47 pgs: 2 active+clean+scrubbing, 6 active+recovering+undersized+degraded+remapped, 39 active+clean; 5.0 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 445 op/s; 27569/244284 objects degraded (11.286%); 9281/244284 objects misplaced (3.799%); 8.7 MiB/s, 141 objects/s recovering 2024-08-19T14:36:39.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:39 smithi045 ceph-mon[20143]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:36:39.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:39 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/958279799' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:36:39.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:39 smithi045 ceph-mon[20143]: osdmap e661: 12 total, 12 up, 10 in 2024-08-19T14:36:39.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:39 smithi031 ceph-mon[16336]: pgmap v1562: 47 pgs: 2 active+clean+scrubbing, 6 active+recovering+undersized+degraded+remapped, 39 active+clean; 5.0 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 445 op/s; 27569/244284 objects degraded (11.286%); 9281/244284 objects misplaced (3.799%); 8.7 MiB/s, 141 objects/s recovering 2024-08-19T14:36:39.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:39 smithi031 ceph-mon[16336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:36:39.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:39 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/958279799' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:36:39.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:39 smithi031 ceph-mon[16336]: osdmap e661: 12 total, 12 up, 10 in 2024-08-19T14:36:39.636 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:36:39.659 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:39.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:39.660 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:36:39.760 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:36:39.783 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:39.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:39.784 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:36:39.884 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:36:39.906 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:39.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:39.907 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:36:40.008 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:36:40.030 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:40.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:40.031 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:36:40.132 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:36:40.155 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:40.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:40.156 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:36:40.257 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:36:40.280 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:40.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:40.282 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:36:40.383 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:36:40.404 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:40.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:40.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:36:40.506 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:36:40.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:40 smithi032 ceph-mon[20004]: 7.6 scrub starts 2024-08-19T14:36:40.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:40 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:36:40.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:40 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:36:40.529 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:40.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:40.530 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:36:40.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:40 smithi045 ceph-mon[20143]: 7.6 scrub starts 2024-08-19T14:36:40.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:40 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:36:40.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:40 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:36:40.619 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:40 smithi031 ceph-mon[16336]: 7.6 scrub starts 2024-08-19T14:36:40.619 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:40 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:36:40.619 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:40 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:36:40.631 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:36:40.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:40.709 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:40.709 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:36:40.810 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:36:40.831 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:40.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:40.832 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:36:40.933 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:36:40.956 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:40.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:40.957 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:36:41.058 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:36:41.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:41.118 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:41.118 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:36:41.219 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:36:41.263 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:41.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:41.264 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:36:41.340 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:36:41.356 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:36:41.365 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:36:41.388 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:41.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:41.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:36:41.423 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:36:41.490 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:36:41.514 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:41 smithi031 ceph-mon[16336]: pgmap v1564: 47 pgs: 2 active+clean+scrubbing, 6 active+recovering+undersized+degraded+remapped, 39 active+clean; 4.9 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 458 op/s; 27508/240957 objects degraded (11.416%); 9220/240957 objects misplaced (3.826%); 5.7 MiB/s, 92 objects/s recovering 2024-08-19T14:36:41.514 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:41 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:36:41.514 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:41 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:36:41.514 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:41 smithi031 ceph-mon[16336]: osdmap e662: 12 total, 12 up, 10 in 2024-08-19T14:36:41.514 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:41 smithi031 ceph-mon[16336]: 7.6 scrub ok 2024-08-19T14:36:41.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:41 smithi045 ceph-mon[20143]: pgmap v1564: 47 pgs: 2 active+clean+scrubbing, 6 active+recovering+undersized+degraded+remapped, 39 active+clean; 4.9 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 458 op/s; 27508/240957 objects degraded (11.416%); 9220/240957 objects misplaced (3.826%); 5.7 MiB/s, 92 objects/s recovering 2024-08-19T14:36:41.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:41 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:36:41.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:41 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:36:41.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:41 smithi045 ceph-mon[20143]: osdmap e662: 12 total, 12 up, 10 in 2024-08-19T14:36:41.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:41 smithi045 ceph-mon[20143]: 7.6 scrub ok 2024-08-19T14:36:41.550 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:41.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:41.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:36:41.652 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:36:41.704 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:41.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:41.705 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:36:41.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:41 smithi032 ceph-mon[20004]: pgmap v1564: 47 pgs: 2 active+clean+scrubbing, 6 active+recovering+undersized+degraded+remapped, 39 active+clean; 4.9 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 458 op/s; 27508/240957 objects degraded (11.416%); 9220/240957 objects misplaced (3.826%); 5.7 MiB/s, 92 objects/s recovering 2024-08-19T14:36:41.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:41 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:36:41.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:41 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:36:41.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:41 smithi032 ceph-mon[20004]: osdmap e662: 12 total, 12 up, 10 in 2024-08-19T14:36:41.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:41 smithi032 ceph-mon[20004]: 7.6 scrub ok 2024-08-19T14:36:41.806 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:36:41.828 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:41.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:41.829 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:36:41.930 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:36:41.952 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:41.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:41.953 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:36:42.054 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:36:42.079 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:42.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:42.080 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:36:42.181 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:36:42.204 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:42.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:42.206 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:36:42.306 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:36:42.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:42.364 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:42.364 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:36:42.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:42 smithi031 ceph-mon[16336]: 7.f deep-scrub starts 2024-08-19T14:36:42.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:42 smithi031 ceph-mon[16336]: osdmap e663: 12 total, 12 up, 10 in 2024-08-19T14:36:42.438 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:36:42.438 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:36:42.465 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:36:42.487 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:42.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:42.488 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:36:42.499 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:36:42.499 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:36:42.543 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:36:42.544 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:36:42.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:42 smithi045 ceph-mon[20143]: 7.f deep-scrub starts 2024-08-19T14:36:42.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:42 smithi045 ceph-mon[20143]: osdmap e663: 12 total, 12 up, 10 in 2024-08-19T14:36:42.589 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:36:42.617 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:36:42.618 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:36:42.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:42.628 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:42.628 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:36:42.655 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:36:42.656 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:36:42.690 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:36:42.691 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:36:42.725 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:36:42.725 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:36:42.729 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:36:42.753 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:42.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:42.754 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:36:42.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:42 smithi032 ceph-mon[20004]: 7.f deep-scrub starts 2024-08-19T14:36:42.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:42 smithi032 ceph-mon[20004]: osdmap e663: 12 total, 12 up, 10 in 2024-08-19T14:36:42.802 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:36:42.802 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:36:42.835 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:36:42.835 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:36:42.855 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:36:42.866 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:36:42.866 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:36:42.877 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:42.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:42.878 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:36:42.903 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:36:42.904 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:36:42.952 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:36:42.952 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:36:42.978 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:36:43.002 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:43.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:43.003 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:36:43.089 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:36:43.089 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:36:43.104 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:36:43.127 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:43.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:43.127 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:36:43.228 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:36:43.254 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:43.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:43.255 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:36:43.284 INFO:teuthology.orchestra.run.smithi031.stderr:reweighted osd.5 to 1 (10000) 2024-08-19T14:36:43.355 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:36:43.358 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-19T14:36:43.413 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:43.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:43.414 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:36:43.515 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:36:43.536 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:43.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:43.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:36:43.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:43 smithi045 ceph-mon[20143]: pgmap v1567: 47 pgs: 1 active+clean+scrubbing, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 4.9 GiB data, 22 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 393 op/s; 27364/239340 objects degraded (11.433%); 9072/239340 objects misplaced (3.790%); 7.2 MiB/s, 115 objects/s recovering 2024-08-19T14:36:43.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:43 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1320190500' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:36:43.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:43 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1508963038' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-08-19T14:36:43.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:43 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 27364/239340 objects degraded (11.433%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-19T14:36:43.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:43 smithi031 ceph-mon[16336]: pgmap v1567: 47 pgs: 1 active+clean+scrubbing, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 4.9 GiB data, 22 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 393 op/s; 27364/239340 objects degraded (11.433%); 9072/239340 objects misplaced (3.790%); 7.2 MiB/s, 115 objects/s recovering 2024-08-19T14:36:43.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:43 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1320190500' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:36:43.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:43 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1508963038' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-08-19T14:36:43.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:43 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 27364/239340 objects degraded (11.433%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-19T14:36:43.637 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:36:43.661 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:43.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:43.662 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:36:43.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:43 smithi032 ceph-mon[20004]: pgmap v1567: 47 pgs: 1 active+clean+scrubbing, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 4.9 GiB data, 22 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 393 op/s; 27364/239340 objects degraded (11.433%); 9072/239340 objects misplaced (3.790%); 7.2 MiB/s, 115 objects/s recovering 2024-08-19T14:36:43.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:43 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1320190500' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:36:43.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:43 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1508963038' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-08-19T14:36:43.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:43 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 27364/239340 objects degraded (11.433%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-19T14:36:43.763 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:36:43.826 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:43.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:43.827 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:36:43.928 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:36:43.952 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:43.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:43.953 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:36:44.054 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:36:44.079 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:44.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:44.080 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:36:44.181 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:36:44.238 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:44.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:44.239 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:36:44.251 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-08-19T14:36:44.340 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:36:44.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:44.420 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:44.420 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:36:44.521 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:36:44.544 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:44.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:44.545 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:36:44.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:44 smithi045 ceph-mon[20143]: 7.9 scrub starts 2024-08-19T14:36:44.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:44 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1508963038' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-08-19T14:36:44.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:44 smithi045 ceph-mon[20143]: osdmap e664: 12 total, 12 up, 10 in 2024-08-19T14:36:44.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:44 smithi031 ceph-mon[16336]: 7.9 scrub starts 2024-08-19T14:36:44.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:44 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1508963038' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-08-19T14:36:44.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:44 smithi031 ceph-mon[16336]: osdmap e664: 12 total, 12 up, 10 in 2024-08-19T14:36:44.646 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:36:44.668 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:44.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:44.669 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:36:44.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:44 smithi032 ceph-mon[20004]: 7.9 scrub starts 2024-08-19T14:36:44.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:44 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1508963038' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-08-19T14:36:44.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:44 smithi032 ceph-mon[20004]: osdmap e664: 12 total, 12 up, 10 in 2024-08-19T14:36:44.770 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:36:44.791 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:44.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:44.792 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:36:44.893 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:36:44.916 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:44.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:44.917 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:36:45.018 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:36:45.051 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:45.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:45.052 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:36:45.153 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:36:45.176 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:45.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:45.177 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:36:45.278 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:36:45.301 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:45.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:45.302 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:36:45.403 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:36:45.426 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:45.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:45.427 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:36:45.529 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:36:45.571 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:45.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:45.572 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:36:45.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:45 smithi031 ceph-mon[16336]: 1.11 scrub starts 2024-08-19T14:36:45.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:45 smithi031 ceph-mon[16336]: 1.11 scrub ok 2024-08-19T14:36:45.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:45 smithi031 ceph-mon[16336]: pgmap v1569: 47 pgs: 1 active+clean+scrubbing, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 4.8 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 380 op/s; 27112/237498 objects degraded (11.416%); 9072/237498 objects misplaced (3.820%); 6.4 MiB/s, 101 objects/s recovering 2024-08-19T14:36:45.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:45 smithi031 ceph-mon[16336]: 7.9 scrub ok 2024-08-19T14:36:45.673 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:36:45.694 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:45.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:45.695 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:36:45.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:45 smithi032 ceph-mon[20004]: 1.11 scrub starts 2024-08-19T14:36:45.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:45 smithi032 ceph-mon[20004]: 1.11 scrub ok 2024-08-19T14:36:45.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:45 smithi032 ceph-mon[20004]: pgmap v1569: 47 pgs: 1 active+clean+scrubbing, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 4.8 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 380 op/s; 27112/237498 objects degraded (11.416%); 9072/237498 objects misplaced (3.820%); 6.4 MiB/s, 101 objects/s recovering 2024-08-19T14:36:45.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:45 smithi032 ceph-mon[20004]: 7.9 scrub ok 2024-08-19T14:36:45.796 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:36:45.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:45 smithi045 ceph-mon[20143]: 1.11 scrub starts 2024-08-19T14:36:45.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:45 smithi045 ceph-mon[20143]: 1.11 scrub ok 2024-08-19T14:36:45.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:45 smithi045 ceph-mon[20143]: pgmap v1569: 47 pgs: 1 active+clean+scrubbing, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 4.8 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 380 op/s; 27112/237498 objects degraded (11.416%); 9072/237498 objects misplaced (3.820%); 6.4 MiB/s, 101 objects/s recovering 2024-08-19T14:36:45.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:45 smithi045 ceph-mon[20143]: 7.9 scrub ok 2024-08-19T14:36:45.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:45.831 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:45.831 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:36:45.932 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:36:45.954 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:45.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:45.955 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:36:46.056 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:36:46.078 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:46.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:46.079 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:36:46.180 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:36:46.203 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:46.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:46.204 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:36:46.305 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:36:46.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:46.339 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:46.339 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:36:46.440 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:36:46.463 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:46.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:46.464 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:36:46.565 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:36:46.587 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:46.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:46.588 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:36:46.689 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:36:46.713 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:46.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:46.714 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:36:46.815 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:36:46.837 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:46.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:46.837 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:36:46.938 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:36:46.960 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:46.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:46.961 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:36:47.062 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:36:47.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:47.097 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:47.097 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:36:47.198 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:36:47.206 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:36:47.220 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:47.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:47.221 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:36:47.322 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:36:47.346 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:47.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:47.347 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:36:47.448 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:36:47.469 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:47.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:47.470 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:36:47.571 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:36:47.594 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:47.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:47.595 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:36:47.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:47 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:36:47.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:47 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:36:47.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:47 smithi031 ceph-mon[16336]: pgmap v1570: 47 pgs: 1 active+clean+scrubbing+deep, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 4.8 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 364 op/s; 26869/234384 objects degraded (11.464%); 9006/234384 objects misplaced (3.842%); 8.2 MiB/s, 130 objects/s recovering 2024-08-19T14:36:47.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:47 smithi031 ceph-mon[16336]: osdmap e665: 12 total, 12 up, 10 in 2024-08-19T14:36:47.696 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:36:47.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:47.718 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:47.719 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:36:47.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:47 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:36:47.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:47 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:36:47.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:47 smithi032 ceph-mon[20004]: pgmap v1570: 47 pgs: 1 active+clean+scrubbing+deep, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 4.8 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 364 op/s; 26869/234384 objects degraded (11.464%); 9006/234384 objects misplaced (3.842%); 8.2 MiB/s, 130 objects/s recovering 2024-08-19T14:36:47.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:47 smithi032 ceph-mon[20004]: osdmap e665: 12 total, 12 up, 10 in 2024-08-19T14:36:47.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:47 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:36:47.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:47 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:36:47.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:47 smithi045 ceph-mon[20143]: pgmap v1570: 47 pgs: 1 active+clean+scrubbing+deep, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 4.8 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 364 op/s; 26869/234384 objects degraded (11.464%); 9006/234384 objects misplaced (3.842%); 8.2 MiB/s, 130 objects/s recovering 2024-08-19T14:36:47.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:47 smithi045 ceph-mon[20143]: osdmap e665: 12 total, 12 up, 10 in 2024-08-19T14:36:47.819 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:36:47.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:47.843 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:47.844 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:36:47.944 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:36:47.966 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:47.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:47.967 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:36:48.068 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:36:48.090 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:48.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:48.091 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:36:48.192 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:36:48.217 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:48.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:48.218 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:36:48.319 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:36:48.343 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:48.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:48.344 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:36:48.445 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:36:48.470 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:48.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:48.471 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:36:48.572 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:36:48.595 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:48.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:48.597 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:36:48.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:48 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 26869/234384 objects degraded (11.464%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-19T14:36:48.698 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:36:48.744 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:48.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:48.745 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:36:48.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:48 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 26869/234384 objects degraded (11.464%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-19T14:36:48.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:48 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 26869/234384 objects degraded (11.464%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-19T14:36:48.846 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:36:48.869 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:48.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:48.870 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:36:48.971 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:36:48.992 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:48.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:48.994 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:36:49.094 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:36:49.136 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:49.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:49.137 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:36:49.238 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:36:49.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:49.275 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:49.275 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:36:49.376 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:36:49.401 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:49.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:49.402 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:36:49.503 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:36:49.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:49.552 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:49.552 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:36:49.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:49 smithi031 ceph-mon[16336]: pgmap v1572: 47 pgs: 1 active+clean+scrubbing+deep, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 4.7 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 365 op/s; 26634/232578 objects degraded (11.452%); 8874/232578 objects misplaced (3.815%); 8.7 MiB/s, 139 objects/s recovering 2024-08-19T14:36:49.654 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:36:49.680 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:49.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:49.681 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:36:49.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:49 smithi032 ceph-mon[20004]: pgmap v1572: 47 pgs: 1 active+clean+scrubbing+deep, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 4.7 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 365 op/s; 26634/232578 objects degraded (11.452%); 8874/232578 objects misplaced (3.815%); 8.7 MiB/s, 139 objects/s recovering 2024-08-19T14:36:49.782 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:36:49.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:49 smithi045 ceph-mon[20143]: pgmap v1572: 47 pgs: 1 active+clean+scrubbing+deep, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 4.7 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 365 op/s; 26634/232578 objects degraded (11.452%); 8874/232578 objects misplaced (3.815%); 8.7 MiB/s, 139 objects/s recovering 2024-08-19T14:36:49.803 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:49.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:49.804 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:36:49.905 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:36:49.928 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:49.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:49.929 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:36:50.029 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:36:50.053 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:50.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:50.054 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:36:50.155 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:36:50.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:50.191 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:50.191 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:36:50.292 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:36:50.316 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:50.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:50.317 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:36:50.418 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:36:50.440 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:50.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:50.441 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:36:50.542 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:36:50.564 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:50.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:50.565 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:36:50.666 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:36:50.711 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:50.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:50.712 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:36:50.813 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:36:50.861 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:50.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:50.863 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:36:50.964 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:36:51.006 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:51.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:51.007 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:36:51.108 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:36:51.131 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:51.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:51.133 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:36:51.234 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:36:51.255 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:51.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:51.256 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:36:51.357 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:36:51.379 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:51.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:51.380 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:36:51.481 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:36:51.503 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:51.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:51.504 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:36:51.605 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:36:51.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:51 smithi031 ceph-mon[16336]: pgmap v1573: 47 pgs: 1 active+clean+scrubbing+deep, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 4.7 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 345 op/s; 26634/231042 objects degraded (11.528%); 8874/231042 objects misplaced (3.841%); 6.7 MiB/s, 107 objects/s recovering 2024-08-19T14:36:51.643 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:51.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:51.644 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:36:51.745 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:36:51.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:51 smithi032 ceph-mon[20004]: pgmap v1573: 47 pgs: 1 active+clean+scrubbing+deep, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 4.7 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 345 op/s; 26634/231042 objects degraded (11.528%); 8874/231042 objects misplaced (3.841%); 6.7 MiB/s, 107 objects/s recovering 2024-08-19T14:36:51.767 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:51.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:51.768 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:36:51.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:51 smithi045 ceph-mon[20143]: pgmap v1573: 47 pgs: 1 active+clean+scrubbing+deep, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 4.7 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 345 op/s; 26634/231042 objects degraded (11.528%); 8874/231042 objects misplaced (3.841%); 6.7 MiB/s, 107 objects/s recovering 2024-08-19T14:36:51.869 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:36:51.894 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:51.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:51.895 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:36:51.985 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd reweight 7 1 2024-08-19T14:36:51.996 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:36:52.017 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:52.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:52.018 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:36:52.050 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:36:52.053 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:36:52.060 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:36:52.118 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:36:52.143 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:52.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:52.144 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:36:52.245 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:36:52.314 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:52.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:52.315 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:36:52.416 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:36:52.440 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:52.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:52.441 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:36:52.542 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:36:52.564 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:52.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:52.565 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:36:52.665 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:36:52.692 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:52.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:52.693 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:36:52.794 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:36:52.816 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:52.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:52.818 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:36:52.919 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:36:52.959 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:52.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:52.960 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:36:53.061 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:36:53.099 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:53.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:53.101 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:36:53.202 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:36:53.252 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:53.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:53.253 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:36:53.354 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:36:53.406 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:53.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:53.410 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:36:53.511 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:36:53.533 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:53.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:53.534 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:36:53.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:53 smithi031 ceph-mon[16336]: pgmap v1574: 47 pgs: 1 active+clean+scrubbing+deep, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 4.7 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 346 op/s; 26377/228585 objects degraded (11.539%); 8667/228585 objects misplaced (3.792%); 7.0 MiB/s, 112 objects/s recovering 2024-08-19T14:36:53.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:53 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 26377/228585 objects degraded (11.539%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-19T14:36:53.636 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:36:53.658 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:53.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:53.659 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:36:53.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:53 smithi032 ceph-mon[20004]: pgmap v1574: 47 pgs: 1 active+clean+scrubbing+deep, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 4.7 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 346 op/s; 26377/228585 objects degraded (11.539%); 8667/228585 objects misplaced (3.792%); 7.0 MiB/s, 112 objects/s recovering 2024-08-19T14:36:53.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:53 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 26377/228585 objects degraded (11.539%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-19T14:36:53.759 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:36:53.781 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:53.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:53.782 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:36:53.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:53 smithi045 ceph-mon[20143]: pgmap v1574: 47 pgs: 1 active+clean+scrubbing+deep, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 4.7 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 346 op/s; 26377/228585 objects degraded (11.539%); 8667/228585 objects misplaced (3.792%); 7.0 MiB/s, 112 objects/s recovering 2024-08-19T14:36:53.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:53 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 26377/228585 objects degraded (11.539%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-19T14:36:53.883 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:36:53.906 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:53.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:53.907 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:36:54.008 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:36:54.048 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:54.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:54.049 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:36:54.150 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:36:54.175 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:54.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:54.176 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:36:54.277 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:36:54.299 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:54.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:54.300 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:36:54.401 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:36:54.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:54.442 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:54.442 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:36:54.543 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:36:54.587 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:54.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:54.588 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:36:54.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:54 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:36:54.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:54 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:36:54.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:54 smithi031 ceph-mon[16336]: osdmap e666: 12 total, 12 up, 10 in 2024-08-19T14:36:54.689 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:36:54.712 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:54.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:54.713 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:36:54.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:54 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:36:54.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:54 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:36:54.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:54 smithi032 ceph-mon[20004]: osdmap e666: 12 total, 12 up, 10 in 2024-08-19T14:36:54.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:54 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:36:54.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:54 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:36:54.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:54 smithi045 ceph-mon[20143]: osdmap e666: 12 total, 12 up, 10 in 2024-08-19T14:36:54.814 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:36:54.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:54.838 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:54.839 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:36:54.940 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:36:54.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:54.981 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:54.982 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:36:55.083 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:36:55.105 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:55.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:55.106 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:36:55.207 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:36:55.231 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:55.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:55.233 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:36:55.334 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:36:55.356 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:55.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:55.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:36:55.458 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:36:55.479 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:55.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:55.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:36:55.581 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:36:55.604 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:55.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:55.605 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:36:55.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:55 smithi031 ceph-mon[16336]: pgmap v1576: 47 pgs: 1 active+clean+scrubbing+deep, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 4.6 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 316 op/s; 26112/226794 objects degraded (11.514%); 8667/226794 objects misplaced (3.822%); 8.0 MiB/s, 128 objects/s recovering 2024-08-19T14:36:55.706 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:36:55.731 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:55.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:55.732 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:36:55.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:55 smithi032 ceph-mon[20004]: pgmap v1576: 47 pgs: 1 active+clean+scrubbing+deep, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 4.6 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 316 op/s; 26112/226794 objects degraded (11.514%); 8667/226794 objects misplaced (3.822%); 8.0 MiB/s, 128 objects/s recovering 2024-08-19T14:36:55.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:55 smithi045 ceph-mon[20143]: pgmap v1576: 47 pgs: 1 active+clean+scrubbing+deep, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 4.6 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 316 op/s; 26112/226794 objects degraded (11.514%); 8667/226794 objects misplaced (3.822%); 8.0 MiB/s, 128 objects/s recovering 2024-08-19T14:36:55.834 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:36:55.856 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:55.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:55.857 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:36:55.958 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:36:56.023 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:56.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:56.024 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:36:56.125 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:36:56.172 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:36:56.172 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:36:56.177 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:56.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:56.178 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:36:56.223 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:36:56.223 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:36:56.263 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:36:56.263 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:36:56.278 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:36:56.310 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:36:56.311 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:36:56.332 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:56.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:56.333 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:36:56.342 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:36:56.343 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:36:56.377 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:36:56.377 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:36:56.418 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:36:56.418 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:36:56.434 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:36:56.452 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:36:56.453 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:36:56.482 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:56.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:56.483 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:36:56.492 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:36:56.493 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:36:56.493 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:56 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3204992819' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:36:56.528 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:36:56.528 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:36:56.570 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:36:56.570 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:36:56.584 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:36:56.605 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:56.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:56.606 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:36:56.609 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:36:56.609 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:36:56.707 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:36:56.728 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:56.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:56.729 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:36:56.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:56 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3204992819' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:36:56.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:56 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3204992819' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:36:56.830 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:36:56.851 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:56.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:56.852 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:36:56.953 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:36:57.022 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:57.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:57.023 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:36:57.124 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:36:57.146 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:57.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:57.147 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:36:57.248 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:36:57.269 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:57.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:57.270 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:36:57.371 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:36:57.394 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is set 2024-08-19T14:36:57.396 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:57.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:57.397 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:36:57.498 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:36:57.520 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:57.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:57.521 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:36:57.622 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:36:57.661 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:57.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:57.662 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:36:57.707 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:57 smithi031 ceph-mon[16336]: pgmap v1577: 47 pgs: 1 active+clean+scrubbing+deep, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 4.5 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 365 op/s; 25850/223533 objects degraded (11.564%); 8594/223533 objects misplaced (3.845%); 8.1 MiB/s, 130 objects/s recovering 2024-08-19T14:36:57.707 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:57 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2636215993' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:36:57.707 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:57 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:36:57.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:57 smithi032 ceph-mon[20004]: pgmap v1577: 47 pgs: 1 active+clean+scrubbing+deep, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 4.5 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 365 op/s; 25850/223533 objects degraded (11.564%); 8594/223533 objects misplaced (3.845%); 8.1 MiB/s, 130 objects/s recovering 2024-08-19T14:36:57.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:57 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2636215993' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:36:57.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:57 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:36:57.762 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:36:57.785 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:57.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:57.786 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:36:57.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:57 smithi045 ceph-mon[20143]: pgmap v1577: 47 pgs: 1 active+clean+scrubbing+deep, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 4.5 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 365 op/s; 25850/223533 objects degraded (11.564%); 8594/223533 objects misplaced (3.845%); 8.1 MiB/s, 130 objects/s recovering 2024-08-19T14:36:57.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:57 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2636215993' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:36:57.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:57 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:36:57.887 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:36:57.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:57.910 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:57.910 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:36:58.011 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:36:58.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:58.035 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:58.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:36:58.100 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:36:58.100 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-08-19T14:36:58.100 INFO:teuthology.orchestra.run.smithi031.stdout: "complaint_time": 30, 2024-08-19T14:36:58.100 INFO:teuthology.orchestra.run.smithi031.stdout: "num_blocked_ops": 0 2024-08-19T14:36:58.100 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:36:58.135 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:36:58.177 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:58.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:58.179 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:36:58.280 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:36:58.301 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:58.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:58.302 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:36:58.403 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:36:58.425 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:58.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:58.426 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:36:58.527 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:36:58.552 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:58.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:58.553 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:36:58.653 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:36:58.678 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:58.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:58.679 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:36:58.705 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:58 smithi031 ceph-mon[16336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:36:58.705 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:58 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:36:58.705 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:58 smithi031 ceph-mon[16336]: osdmap e667: 12 total, 12 up, 10 in 2024-08-19T14:36:58.705 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:58 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 25850/223533 objects degraded (11.564%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-19T14:36:58.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:58 smithi032 ceph-mon[20004]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:36:58.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:58 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:36:58.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:58 smithi032 ceph-mon[20004]: osdmap e667: 12 total, 12 up, 10 in 2024-08-19T14:36:58.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:58 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 25850/223533 objects degraded (11.564%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-19T14:36:58.761 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-08-19T14:36:58.779 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:36:58.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:58 smithi045 ceph-mon[20143]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:36:58.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:58 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:36:58.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:58 smithi045 ceph-mon[20143]: osdmap e667: 12 total, 12 up, 10 in 2024-08-19T14:36:58.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:58 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 25850/223533 objects degraded (11.564%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-19T14:36:58.803 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:58.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:58.804 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:36:58.905 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:36:58.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:58.956 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:58.957 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:36:59.058 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:36:59.107 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:59.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:59.108 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:36:59.209 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:36:59.233 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:36:59.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:59.234 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:36:59.290 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:36:59.338 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:36:59.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:59.361 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:59.362 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:36:59.365 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:36:59.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:36:59 smithi031 ceph-mon[16336]: pgmap v1579: 47 pgs: 1 active+clean+scrubbing+deep, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 4.5 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 398 op/s; 25604/221457 objects degraded (11.562%); 8452/221457 objects misplaced (3.817%); 10 MiB/s, 163 objects/s recovering 2024-08-19T14:36:59.463 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:36:59.513 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:59.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:59.514 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:36:59.615 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:36:59.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:59.637 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:36:59.638 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:36:59.739 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:36:59.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:36:59 smithi032 ceph-mon[20004]: pgmap v1579: 47 pgs: 1 active+clean+scrubbing+deep, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 4.5 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 398 op/s; 25604/221457 objects degraded (11.562%); 8452/221457 objects misplaced (3.817%); 10 MiB/s, 163 objects/s recovering 2024-08-19T14:36:59.759 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:36:59.775 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:59.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:59.776 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:36:59.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:36:59 smithi045 ceph-mon[20143]: pgmap v1579: 47 pgs: 1 active+clean+scrubbing+deep, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 4.5 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 398 op/s; 25604/221457 objects degraded (11.562%); 8452/221457 objects misplaced (3.817%); 10 MiB/s, 163 objects/s recovering 2024-08-19T14:36:59.877 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:36:59.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:36:59.918 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:36:59.918 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:37:00.018 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:37:00.041 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:00.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:00.042 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:37:00.143 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:37:00.165 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:00.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:00.166 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:37:00.267 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:37:00.331 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:00.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:00.333 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:37:00.396 INFO:teuthology.orchestra.run.smithi031.stderr:reweighted osd.7 to 1 (10000) 2024-08-19T14:37:00.433 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:37:00.455 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:00.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:00.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:37:00.557 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:37:00.578 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:00.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:00.579 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:37:00.680 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:37:00.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:00.752 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:00.753 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:37:00.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:00 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2864461165' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-08-19T14:37:00.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:00 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2864461165' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-08-19T14:37:00.826 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:00 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2864461165' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-08-19T14:37:00.854 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:37:00.876 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:00.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:00.877 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:37:00.944 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:37:00.978 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:37:01.029 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:37:01.029 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 20, 2024-08-19T14:37:01.029 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 600, 2024-08-19T14:37:01.029 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [ 2024-08-19T14:37:01.029 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.029 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25813.0:237705 5.7 5:e5695e68:::benchmark_data_smithi045_85309_object117452:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e495)", 2024-08-19T14:37:01.029 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:28:07.101072+0000", 2024-08-19T14:37:01.030 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 533.90641677799999, 2024-08-19T14:37:01.030 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.231283661, 2024-08-19T14:37:01.030 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:37:01.030 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:37:01.030 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:37:01.030 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:37:01.030 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25813", 2024-08-19T14:37:01.030 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:37:01.030 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 237705 2024-08-19T14:37:01.030 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.030 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:37:01.030 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.030 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:37:01.031 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101072+0000", 2024-08-19T14:37:01.031 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:37:01.031 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.031 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.031 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:37:01.031 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101072+0000", 2024-08-19T14:37:01.031 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:37:01.031 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.031 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.031 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:37:01.031 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101074+0000", 2024-08-19T14:37:01.031 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.751e-06 2024-08-19T14:37:01.031 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.031 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.032 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:37:01.032 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101080+0000", 2024-08-19T14:37:01.032 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.9050000000000002e-06 2024-08-19T14:37:01.032 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.032 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.032 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:37:01.032 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101096+0000", 2024-08-19T14:37:01.032 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6364000000000001e-05 2024-08-19T14:37:01.032 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.034 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.034 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:37:01.034 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101107+0000", 2024-08-19T14:37:01.034 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0572e-05 2024-08-19T14:37:01.034 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.034 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.034 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:37:01.034 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101166+0000", 2024-08-19T14:37:01.035 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.8981999999999999e-05 2024-08-19T14:37:01.035 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.035 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.035 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:37:01.035 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101181+0000", 2024-08-19T14:37:01.035 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5364000000000001e-05 2024-08-19T14:37:01.035 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.035 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.035 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:37:01.035 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.315274+0000", 2024-08-19T14:37:01.035 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.21409246900000001 2024-08-19T14:37:01.035 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.035 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.035 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for missing object", 2024-08-19T14:37:01.036 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.315413+0000", 2024-08-19T14:37:01.036 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000139782 2024-08-19T14:37:01.036 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.036 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.036 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:37:01.036 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.332277+0000", 2024-08-19T14:37:01.036 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.016863488999999999 2024-08-19T14:37:01.036 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.036 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.036 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:37:01.036 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.332356+0000", 2024-08-19T14:37:01.036 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.8982999999999999e-05 2024-08-19T14:37:01.036 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:37:01.037 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:37:01.037 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:37:01.037 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.037 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.037 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25813.0:237714 5.7 5:e252da68:::benchmark_data_smithi045_85309_object117461:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e495)", 2024-08-19T14:37:01.037 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:28:07.101128+0000", 2024-08-19T14:37:01.037 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 533.90636130400003, 2024-08-19T14:37:01.037 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.24052662399999999, 2024-08-19T14:37:01.037 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:37:01.037 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:37:01.037 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:37:01.037 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:37:01.038 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25813", 2024-08-19T14:37:01.038 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:37:01.038 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 237714 2024-08-19T14:37:01.038 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.038 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:37:01.038 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.038 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:37:01.039 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101128+0000", 2024-08-19T14:37:01.039 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:37:01.040 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.040 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.040 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:37:01.040 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101128+0000", 2024-08-19T14:37:01.040 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:37:01.040 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.040 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.040 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:37:01.040 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101128+0000", 2024-08-19T14:37:01.040 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.8500000000000002e-07 2024-08-19T14:37:01.041 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.041 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.041 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:37:01.041 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101131+0000", 2024-08-19T14:37:01.041 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.6910000000000002e-06 2024-08-19T14:37:01.041 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.041 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.041 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:37:01.041 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101139+0000", 2024-08-19T14:37:01.041 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.1799999999999996e-06 2024-08-19T14:37:01.041 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.041 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.041 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:37:01.041 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101145+0000", 2024-08-19T14:37:01.042 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.7359999999999998e-06 2024-08-19T14:37:01.042 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.042 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.042 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:37:01.042 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101214+0000", 2024-08-19T14:37:01.042 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.9695000000000003e-05 2024-08-19T14:37:01.042 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.042 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.042 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:37:01.042 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101224+0000", 2024-08-19T14:37:01.042 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0043e-05 2024-08-19T14:37:01.042 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.042 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.042 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:37:01.043 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.315795+0000", 2024-08-19T14:37:01.043 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.21457066599999999 2024-08-19T14:37:01.043 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.043 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.043 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:37:01.043 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.315966+0000", 2024-08-19T14:37:01.043 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00017090700000000001 2024-08-19T14:37:01.043 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.043 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.043 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,5", 2024-08-19T14:37:01.043 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.316585+0000", 2024-08-19T14:37:01.043 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00061851899999999999 2024-08-19T14:37:01.043 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.043 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.044 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:37:01.044 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.332349+0000", 2024-08-19T14:37:01.044 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.015764158 2024-08-19T14:37:01.044 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.044 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.044 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:37:01.044 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.332428+0000", 2024-08-19T14:37:01.044 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.9651e-05 2024-08-19T14:37:01.044 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.045 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.045 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:37:01.045 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.341243+0000", 2024-08-19T14:37:01.045 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0088151610000000002 2024-08-19T14:37:01.045 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.045 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.045 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:37:01.045 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.341320+0000", 2024-08-19T14:37:01.045 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.6304000000000007e-05 2024-08-19T14:37:01.045 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.045 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.045 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:37:01.045 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.341654+0000", 2024-08-19T14:37:01.046 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000334428 2024-08-19T14:37:01.046 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:37:01.046 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:37:01.046 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:37:01.046 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.046 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.046 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25813.0:237731 5.7 5:e54e0cab:::benchmark_data_smithi045_85309_object117478:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e495)", 2024-08-19T14:37:01.046 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:28:07.101155+0000", 2024-08-19T14:37:01.046 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 533.90633364400003, 2024-08-19T14:37:01.046 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.240822699, 2024-08-19T14:37:01.046 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:37:01.046 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:37:01.046 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:37:01.047 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:37:01.047 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25813", 2024-08-19T14:37:01.047 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:37:01.047 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 237731 2024-08-19T14:37:01.047 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.047 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:37:01.047 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.047 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:37:01.047 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101155+0000", 2024-08-19T14:37:01.047 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:37:01.047 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.047 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.047 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:37:01.047 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101155+0000", 2024-08-19T14:37:01.048 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:37:01.048 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.048 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.048 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:37:01.048 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101156+0000", 2024-08-19T14:37:01.048 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.5900000000000003e-07 2024-08-19T14:37:01.048 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.048 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.048 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:37:01.048 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101158+0000", 2024-08-19T14:37:01.048 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.0420000000000001e-06 2024-08-19T14:37:01.048 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.048 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.048 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:37:01.049 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101164+0000", 2024-08-19T14:37:01.049 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.7769999999999997e-06 2024-08-19T14:37:01.049 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.049 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.049 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:37:01.049 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101184+0000", 2024-08-19T14:37:01.049 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.9989000000000001e-05 2024-08-19T14:37:01.049 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.049 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.049 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:37:01.049 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101265+0000", 2024-08-19T14:37:01.049 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.0782e-05 2024-08-19T14:37:01.049 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.049 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.049 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:37:01.050 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101274+0000", 2024-08-19T14:37:01.050 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.0780000000000002e-06 2024-08-19T14:37:01.050 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.050 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.050 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:37:01.050 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.320042+0000", 2024-08-19T14:37:01.050 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.21876728300000001 2024-08-19T14:37:01.050 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.050 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.050 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:37:01.050 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.320209+0000", 2024-08-19T14:37:01.050 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00016716800000000001 2024-08-19T14:37:01.050 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.051 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.051 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,5", 2024-08-19T14:37:01.051 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.320830+0000", 2024-08-19T14:37:01.051 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00062140999999999997 2024-08-19T14:37:01.051 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.051 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.051 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:37:01.051 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.332354+0000", 2024-08-19T14:37:01.051 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.011524123000000001 2024-08-19T14:37:01.051 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.051 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.051 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:37:01.051 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.341182+0000", 2024-08-19T14:37:01.051 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0088281499999999999 2024-08-19T14:37:01.052 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.052 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.052 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:37:01.052 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.341703+0000", 2024-08-19T14:37:01.052 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000521022 2024-08-19T14:37:01.052 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.052 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.052 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:37:01.052 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.341740+0000", 2024-08-19T14:37:01.052 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.6885999999999999e-05 2024-08-19T14:37:01.052 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.052 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.052 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:37:01.053 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.341978+0000", 2024-08-19T14:37:01.053 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00023763000000000001 2024-08-19T14:37:01.053 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:37:01.053 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:37:01.053 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:37:01.053 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.053 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.053 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.25813.0:237758 5.7 5:e57a2935:::benchmark_data_smithi045_85309_object117505:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e495)", 2024-08-19T14:37:01.053 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:28:07.101204+0000", 2024-08-19T14:37:01.053 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 533.906284615, 2024-08-19T14:37:01.053 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.24687310400000001, 2024-08-19T14:37:01.053 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:37:01.053 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:37:01.054 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:37:01.054 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:37:01.054 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.25813", 2024-08-19T14:37:01.054 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/391543872", 2024-08-19T14:37:01.054 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 237758 2024-08-19T14:37:01.054 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.054 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:37:01.054 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.054 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:37:01.054 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101204+0000", 2024-08-19T14:37:01.054 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:37:01.054 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.055 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.055 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:37:01.055 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101204+0000", 2024-08-19T14:37:01.055 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:37:01.055 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.055 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.055 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:37:01.055 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101205+0000", 2024-08-19T14:37:01.055 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2800000000000002e-07 2024-08-19T14:37:01.055 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.055 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.055 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:37:01.055 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101207+0000", 2024-08-19T14:37:01.055 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.768e-06 2024-08-19T14:37:01.056 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.056 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.056 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:37:01.056 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101215+0000", 2024-08-19T14:37:01.056 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.3900000000000004e-06 2024-08-19T14:37:01.056 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.056 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.056 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:37:01.056 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101220+0000", 2024-08-19T14:37:01.056 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.3040000000000004e-06 2024-08-19T14:37:01.056 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.056 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.056 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:37:01.056 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101303+0000", 2024-08-19T14:37:01.057 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.2443999999999999e-05 2024-08-19T14:37:01.057 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.057 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.057 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:37:01.057 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.101311+0000", 2024-08-19T14:37:01.057 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.3389999999999993e-06 2024-08-19T14:37:01.057 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.057 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.057 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:37:01.057 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.324142+0000", 2024-08-19T14:37:01.057 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.222830844 2024-08-19T14:37:01.057 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.057 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.057 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:37:01.058 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.324274+0000", 2024-08-19T14:37:01.058 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013226199999999999 2024-08-19T14:37:01.058 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.058 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.058 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,5", 2024-08-19T14:37:01.058 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.324842+0000", 2024-08-19T14:37:01.058 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00056831899999999996 2024-08-19T14:37:01.058 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.058 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.058 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:37:01.058 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.332438+0000", 2024-08-19T14:37:01.058 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0075956060000000004 2024-08-19T14:37:01.058 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.058 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.058 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:37:01.059 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.342013+0000", 2024-08-19T14:37:01.059 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0095747650000000007 2024-08-19T14:37:01.059 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.059 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.059 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:37:01.059 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.347519+0000", 2024-08-19T14:37:01.059 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0055063330000000004 2024-08-19T14:37:01.059 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.059 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.059 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:37:01.059 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.347576+0000", 2024-08-19T14:37:01.059 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.6610000000000002e-05 2024-08-19T14:37:01.059 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.059 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.060 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:37:01.060 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:28:07.348077+0000", 2024-08-19T14:37:01.060 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00050169199999999998 2024-08-19T14:37:01.060 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:37:01.060 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:37:01.060 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:37:01.060 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.060 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.060 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:76374 6.d 6:bf7b5a72:::benchmark_data_smithi045_97937_object76373:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e526)", 2024-08-19T14:37:01.060 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:29:17.862978+0000", 2024-08-19T14:37:01.060 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 463.144511158, 2024-08-19T14:37:01.060 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.44319341499999998, 2024-08-19T14:37:01.060 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:37:01.061 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:37:01.061 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:37:01.061 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:37:01.061 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:37:01.061 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:37:01.061 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 76374 2024-08-19T14:37:01.062 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.062 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:37:01.062 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.062 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:37:01.062 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.862978+0000", 2024-08-19T14:37:01.062 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:37:01.062 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.062 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.062 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:37:01.062 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.862978+0000", 2024-08-19T14:37:01.062 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:37:01.062 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.062 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.062 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:37:01.063 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.862980+0000", 2024-08-19T14:37:01.063 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.39e-06 2024-08-19T14:37:01.064 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.064 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.064 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:37:01.064 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:17.863020+0000", 2024-08-19T14:37:01.064 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.9978999999999998e-05 2024-08-19T14:37:01.064 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.064 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.064 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:37:01.064 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:18.285688+0000", 2024-08-19T14:37:01.064 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.42266810500000002 2024-08-19T14:37:01.064 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.064 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.065 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:37:01.065 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:18.285699+0000", 2024-08-19T14:37:01.065 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1187e-05 2024-08-19T14:37:01.065 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.065 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.065 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:37:01.065 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:18.288107+0000", 2024-08-19T14:37:01.065 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0024075279999999999 2024-08-19T14:37:01.065 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.065 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.065 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:37:01.065 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:18.288250+0000", 2024-08-19T14:37:01.065 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00014349400000000001 2024-08-19T14:37:01.065 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.066 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.066 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,5", 2024-08-19T14:37:01.066 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:18.288380+0000", 2024-08-19T14:37:01.066 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013004100000000001 2024-08-19T14:37:01.066 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.066 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.066 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:37:01.066 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:18.294529+0000", 2024-08-19T14:37:01.066 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0061484909999999999 2024-08-19T14:37:01.066 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.066 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.066 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:37:01.066 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:18.300842+0000", 2024-08-19T14:37:01.067 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0063133299999999998 2024-08-19T14:37:01.067 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.067 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.067 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:37:01.067 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:18.305954+0000", 2024-08-19T14:37:01.067 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.005111979 2024-08-19T14:37:01.067 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.067 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.067 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:37:01.067 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:18.306116+0000", 2024-08-19T14:37:01.067 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000161531 2024-08-19T14:37:01.067 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.067 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.067 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:37:01.067 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:29:18.306171+0000", 2024-08-19T14:37:01.068 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.5359999999999999e-05 2024-08-19T14:37:01.068 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:37:01.068 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:37:01.068 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:37:01.068 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.068 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.068 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:174030 6.d 6:b7de3b0b:::benchmark_data_smithi045_97937_object66757:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e580)", 2024-08-19T14:37:01.068 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:32:12.456928+0000", 2024-08-19T14:37:01.068 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 288.550560975, 2024-08-19T14:37:01.068 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.31321180500000001, 2024-08-19T14:37:01.068 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:37:01.068 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:37:01.068 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:37:01.068 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:37:01.069 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:37:01.069 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:37:01.069 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 174030 2024-08-19T14:37:01.069 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.069 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:37:01.069 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.069 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:37:01.069 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.456928+0000", 2024-08-19T14:37:01.069 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:37:01.069 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.069 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.069 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:37:01.069 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.456928+0000", 2024-08-19T14:37:01.069 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:37:01.070 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.070 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.070 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:37:01.070 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.456929+0000", 2024-08-19T14:37:01.070 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5600000000000001e-06 2024-08-19T14:37:01.070 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.070 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.070 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:37:01.070 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.456935+0000", 2024-08-19T14:37:01.070 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.8599999999999998e-06 2024-08-19T14:37:01.070 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.070 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.070 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:37:01.070 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.456950+0000", 2024-08-19T14:37:01.071 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4413000000000001e-05 2024-08-19T14:37:01.071 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.071 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.071 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:37:01.071 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.456957+0000", 2024-08-19T14:37:01.071 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.3819999999999998e-06 2024-08-19T14:37:01.071 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.071 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.071 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:37:01.071 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.457017+0000", 2024-08-19T14:37:01.071 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.9490999999999997e-05 2024-08-19T14:37:01.071 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.071 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.072 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:37:01.072 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.457136+0000", 2024-08-19T14:37:01.072 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000119491 2024-08-19T14:37:01.072 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.072 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.072 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,5", 2024-08-19T14:37:01.072 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.457262+0000", 2024-08-19T14:37:01.072 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012542200000000001 2024-08-19T14:37:01.072 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.072 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.072 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:37:01.072 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.466456+0000", 2024-08-19T14:37:01.072 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0091943749999999994 2024-08-19T14:37:01.072 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.073 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.073 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:37:01.073 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.466505+0000", 2024-08-19T14:37:01.073 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.8785000000000001e-05 2024-08-19T14:37:01.073 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.073 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.073 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:37:01.073 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.770050+0000", 2024-08-19T14:37:01.073 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.30354529000000002 2024-08-19T14:37:01.073 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.073 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.073 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:37:01.073 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.770096+0000", 2024-08-19T14:37:01.073 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.6480000000000002e-05 2024-08-19T14:37:01.074 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.074 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.074 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:37:01.074 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.770140+0000", 2024-08-19T14:37:01.074 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.3256000000000001e-05 2024-08-19T14:37:01.074 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:37:01.074 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:37:01.074 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:37:01.074 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.074 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.074 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:174032 6.d 6:b8a5c293:::benchmark_data_smithi045_97937_object66759:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e580)", 2024-08-19T14:37:01.074 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:32:12.458962+0000", 2024-08-19T14:37:01.074 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 288.54852671600003, 2024-08-19T14:37:01.074 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.31142265299999999, 2024-08-19T14:37:01.075 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:37:01.075 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:37:01.075 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:37:01.075 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:37:01.075 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:37:01.075 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:37:01.075 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 174032 2024-08-19T14:37:01.075 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.075 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:37:01.075 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.075 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:37:01.075 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.458962+0000", 2024-08-19T14:37:01.075 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:37:01.075 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.075 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.076 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:37:01.076 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.458962+0000", 2024-08-19T14:37:01.076 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:37:01.076 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:01.077 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:01.077 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:37:01.078 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.078 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:37:01.078 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.458963+0000", 2024-08-19T14:37:01.078 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1289999999999999e-06 2024-08-19T14:37:01.078 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.078 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.078 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:37:01.078 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.458968+0000", 2024-08-19T14:37:01.079 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4830000000000001e-06 2024-08-19T14:37:01.079 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.079 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.079 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:37:01.079 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.458980+0000", 2024-08-19T14:37:01.079 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2476999999999999e-05 2024-08-19T14:37:01.079 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.079 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.079 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:37:01.079 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.458987+0000", 2024-08-19T14:37:01.079 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.3729999999999998e-06 2024-08-19T14:37:01.079 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.079 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.080 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:37:01.080 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.461794+0000", 2024-08-19T14:37:01.080 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0028076809999999998 2024-08-19T14:37:01.080 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.080 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.080 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:37:01.080 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.461948+0000", 2024-08-19T14:37:01.080 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00015364299999999999 2024-08-19T14:37:01.080 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.080 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.080 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,5", 2024-08-19T14:37:01.080 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.462046+0000", 2024-08-19T14:37:01.080 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.7576000000000002e-05 2024-08-19T14:37:01.080 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.081 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.081 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:37:01.081 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.466596+0000", 2024-08-19T14:37:01.081 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0045504179999999997 2024-08-19T14:37:01.081 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.081 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.081 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:37:01.081 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.470406+0000", 2024-08-19T14:37:01.081 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.003809977 2024-08-19T14:37:01.081 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.081 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.081 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:37:01.081 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.770183+0000", 2024-08-19T14:37:01.082 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.29977723699999997 2024-08-19T14:37:01.082 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.082 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.082 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:37:01.082 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.770239+0000", 2024-08-19T14:37:01.082 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.5961999999999997e-05 2024-08-19T14:37:01.082 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.082 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.082 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:37:01.082 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.770385+0000", 2024-08-19T14:37:01.082 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000145697 2024-08-19T14:37:01.082 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:37:01.082 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:37:01.083 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:37:01.083 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.083 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.083 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:174039 6.d 6:b7cfbf20:::benchmark_data_smithi045_97937_object66766:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e580)", 2024-08-19T14:37:01.083 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:32:12.470750+0000", 2024-08-19T14:37:01.083 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 288.53673874499998, 2024-08-19T14:37:01.083 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.299785941, 2024-08-19T14:37:01.083 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:37:01.083 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:37:01.083 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:37:01.083 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:37:01.083 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:37:01.084 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:37:01.084 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 174039 2024-08-19T14:37:01.084 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.084 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:37:01.084 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.084 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:37:01.084 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.470750+0000", 2024-08-19T14:37:01.084 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:37:01.084 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.084 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.084 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:37:01.084 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.470750+0000", 2024-08-19T14:37:01.085 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:37:01.085 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.085 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.085 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:37:01.085 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.470752+0000", 2024-08-19T14:37:01.085 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.739e-06 2024-08-19T14:37:01.085 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.085 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.085 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:37:01.086 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.470759+0000", 2024-08-19T14:37:01.086 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.754e-06 2024-08-19T14:37:01.086 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.086 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.086 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:37:01.087 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.470773+0000", 2024-08-19T14:37:01.087 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4365000000000001e-05 2024-08-19T14:37:01.087 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.087 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.087 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:37:01.087 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.470780+0000", 2024-08-19T14:37:01.087 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.4220000000000003e-06 2024-08-19T14:37:01.087 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.087 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.087 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:37:01.087 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.470837+0000", 2024-08-19T14:37:01.087 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.6246999999999999e-05 2024-08-19T14:37:01.087 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.088 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.088 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:37:01.088 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.471823+0000", 2024-08-19T14:37:01.088 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00098676800000000006 2024-08-19T14:37:01.088 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.088 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.088 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,5", 2024-08-19T14:37:01.088 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.471942+0000", 2024-08-19T14:37:01.088 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00011807399999999999 2024-08-19T14:37:01.088 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.088 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.088 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:37:01.088 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.476422+0000", 2024-08-19T14:37:01.088 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0044803660000000004 2024-08-19T14:37:01.089 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.089 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.089 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:37:01.089 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.477208+0000", 2024-08-19T14:37:01.089 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00078598299999999997 2024-08-19T14:37:01.089 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.089 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.089 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:37:01.089 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.770437+0000", 2024-08-19T14:37:01.089 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.293229036 2024-08-19T14:37:01.089 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.089 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.089 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:37:01.089 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.770497+0000", 2024-08-19T14:37:01.090 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.9998999999999999e-05 2024-08-19T14:37:01.090 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.090 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.090 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:37:01.090 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.770536+0000", 2024-08-19T14:37:01.090 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.9187999999999998e-05 2024-08-19T14:37:01.090 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:37:01.090 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:37:01.090 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:37:01.090 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.090 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.090 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:174060 6.d 6:bd715b6c:::benchmark_data_smithi045_97937_object66787:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e580)", 2024-08-19T14:37:01.090 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:32:12.500214+0000", 2024-08-19T14:37:01.090 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 288.50727484499998, 2024-08-19T14:37:01.091 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.27045459700000002, 2024-08-19T14:37:01.091 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:37:01.091 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:37:01.091 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:37:01.091 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:37:01.091 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:37:01.091 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:37:01.091 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 174060 2024-08-19T14:37:01.091 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.091 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:37:01.091 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.091 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:37:01.091 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.500214+0000", 2024-08-19T14:37:01.091 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:37:01.092 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.092 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.092 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:37:01.092 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.500214+0000", 2024-08-19T14:37:01.092 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:37:01.092 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.092 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.092 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:37:01.092 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.500215+0000", 2024-08-19T14:37:01.092 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.091e-06 2024-08-19T14:37:01.092 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.092 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.092 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:37:01.092 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.500219+0000", 2024-08-19T14:37:01.092 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2150000000000003e-06 2024-08-19T14:37:01.093 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.093 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.093 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:37:01.093 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.500229+0000", 2024-08-19T14:37:01.093 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.8789999999999998e-06 2024-08-19T14:37:01.093 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.093 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.093 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:37:01.093 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.500234+0000", 2024-08-19T14:37:01.093 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0590000000000002e-06 2024-08-19T14:37:01.093 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.093 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.093 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:37:01.093 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.500290+0000", 2024-08-19T14:37:01.094 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.6144999999999999e-05 2024-08-19T14:37:01.094 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.094 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.094 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:37:01.094 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.500418+0000", 2024-08-19T14:37:01.094 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012713199999999999 2024-08-19T14:37:01.094 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.094 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.094 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,5", 2024-08-19T14:37:01.094 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.500544+0000", 2024-08-19T14:37:01.094 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012666899999999999 2024-08-19T14:37:01.094 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.095 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.095 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:37:01.095 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.504930+0000", 2024-08-19T14:37:01.095 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0043855669999999999 2024-08-19T14:37:01.095 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.095 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.095 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:37:01.095 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.506635+0000", 2024-08-19T14:37:01.095 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.001705266 2024-08-19T14:37:01.095 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.095 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.095 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:37:01.096 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.770578+0000", 2024-08-19T14:37:01.096 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.26394303899999999 2024-08-19T14:37:01.096 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.096 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.096 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:37:01.096 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.770629+0000", 2024-08-19T14:37:01.096 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1159000000000003e-05 2024-08-19T14:37:01.096 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.096 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.096 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:37:01.096 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.770669+0000", 2024-08-19T14:37:01.096 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.9375999999999997e-05 2024-08-19T14:37:01.097 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:37:01.097 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:37:01.097 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:37:01.097 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.097 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.097 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:174071 6.d 6:b8039fbe:::benchmark_data_smithi045_97937_object66798:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e580)", 2024-08-19T14:37:01.097 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:32:12.518669+0000", 2024-08-19T14:37:01.097 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 288.48881991100001, 2024-08-19T14:37:01.097 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.25216244300000001, 2024-08-19T14:37:01.097 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:37:01.097 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:37:01.097 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:37:01.097 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:37:01.098 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:37:01.098 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:37:01.098 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 174071 2024-08-19T14:37:01.098 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.098 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:37:01.098 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.098 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:37:01.098 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.518669+0000", 2024-08-19T14:37:01.098 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:37:01.098 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.098 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.098 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:37:01.098 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.518669+0000", 2024-08-19T14:37:01.099 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:37:01.099 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.099 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.099 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:37:01.099 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.518670+0000", 2024-08-19T14:37:01.099 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3009999999999999e-06 2024-08-19T14:37:01.099 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.099 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.099 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:37:01.099 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.518676+0000", 2024-08-19T14:37:01.099 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.8039999999999998e-06 2024-08-19T14:37:01.099 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.099 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.099 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:37:01.100 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.518689+0000", 2024-08-19T14:37:01.100 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2546e-05 2024-08-19T14:37:01.100 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.100 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.100 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:37:01.100 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.518694+0000", 2024-08-19T14:37:01.100 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.4750000000000001e-06 2024-08-19T14:37:01.100 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.100 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.100 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:37:01.100 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.518742+0000", 2024-08-19T14:37:01.100 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.8078999999999998e-05 2024-08-19T14:37:01.101 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.101 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.101 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:37:01.101 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.518849+0000", 2024-08-19T14:37:01.101 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00010722600000000001 2024-08-19T14:37:01.101 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.101 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.101 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,5", 2024-08-19T14:37:01.101 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.518980+0000", 2024-08-19T14:37:01.101 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013033599999999999 2024-08-19T14:37:01.101 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.101 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.101 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:37:01.101 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.523029+0000", 2024-08-19T14:37:01.102 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0040492619999999997 2024-08-19T14:37:01.102 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.102 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.102 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:37:01.102 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.523872+0000", 2024-08-19T14:37:01.102 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00084340800000000003 2024-08-19T14:37:01.102 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.102 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.102 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:37:01.102 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.770717+0000", 2024-08-19T14:37:01.102 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.24684499800000001 2024-08-19T14:37:01.102 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.102 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.103 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:37:01.103 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.770768+0000", 2024-08-19T14:37:01.103 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0438000000000001e-05 2024-08-19T14:37:01.103 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.103 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.103 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:37:01.103 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.770831+0000", 2024-08-19T14:37:01.103 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.3570000000000003e-05 2024-08-19T14:37:01.103 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:37:01.103 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:37:01.103 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:37:01.103 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.103 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.104 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:174075 6.d 6:bbccb6e4:::benchmark_data_smithi045_97937_object66802:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e580)", 2024-08-19T14:37:01.104 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:32:12.527198+0000", 2024-08-19T14:37:01.104 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 288.48029126, 2024-08-19T14:37:01.104 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.24381356800000001, 2024-08-19T14:37:01.104 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:37:01.104 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:37:01.104 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:37:01.104 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:37:01.104 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:37:01.104 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:37:01.104 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 174075 2024-08-19T14:37:01.104 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.104 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:37:01.105 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.105 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:37:01.105 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.527198+0000", 2024-08-19T14:37:01.105 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:37:01.105 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.105 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.105 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:37:01.105 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.527198+0000", 2024-08-19T14:37:01.105 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:37:01.105 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.105 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.105 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:37:01.105 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.527199+0000", 2024-08-19T14:37:01.105 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.685e-06 2024-08-19T14:37:01.105 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.106 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.106 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:37:01.106 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.527206+0000", 2024-08-19T14:37:01.106 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.2400000000000004e-06 2024-08-19T14:37:01.106 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.106 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.106 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:37:01.106 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.527220+0000", 2024-08-19T14:37:01.106 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4815e-05 2024-08-19T14:37:01.106 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.106 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.106 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:37:01.106 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.527228+0000", 2024-08-19T14:37:01.106 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.1999999999999997e-06 2024-08-19T14:37:01.107 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.107 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.107 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:37:01.107 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.527298+0000", 2024-08-19T14:37:01.107 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.0264000000000004e-05 2024-08-19T14:37:01.107 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.107 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.107 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:37:01.107 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.527391+0000", 2024-08-19T14:37:01.107 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.2843000000000005e-05 2024-08-19T14:37:01.107 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.107 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.107 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,5", 2024-08-19T14:37:01.107 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.527502+0000", 2024-08-19T14:37:01.108 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000111468 2024-08-19T14:37:01.108 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.108 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.108 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:37:01.108 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.531337+0000", 2024-08-19T14:37:01.108 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00383468 2024-08-19T14:37:01.108 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.108 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.108 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:37:01.108 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.531933+0000", 2024-08-19T14:37:01.108 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00059663700000000001 2024-08-19T14:37:01.108 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.108 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.109 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:37:01.109 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.770928+0000", 2024-08-19T14:37:01.109 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.238994768 2024-08-19T14:37:01.109 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.109 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.109 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:37:01.109 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.770976+0000", 2024-08-19T14:37:01.109 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7500999999999998e-05 2024-08-19T14:37:01.109 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.109 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.109 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:37:01.109 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.771011+0000", 2024-08-19T14:37:01.109 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.5466999999999998e-05 2024-08-19T14:37:01.110 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:37:01.110 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:37:01.110 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:37:01.110 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.110 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.110 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:203073 6.4 6:2b68586f:::benchmark_data_smithi045_97937_object95800:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e589)", 2024-08-19T14:37:01.110 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:32:55.259652+0000", 2024-08-19T14:37:01.110 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 245.74783726699999, 2024-08-19T14:37:01.110 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.22078215300000001, 2024-08-19T14:37:01.110 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:37:01.110 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:37:01.110 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:37:01.110 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:37:01.111 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:37:01.111 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:37:01.111 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 203073 2024-08-19T14:37:01.111 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.111 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:37:01.111 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.111 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:37:01.111 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:55.259652+0000", 2024-08-19T14:37:01.111 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:37:01.111 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.111 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.111 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:37:01.112 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:55.259652+0000", 2024-08-19T14:37:01.112 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:37:01.112 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.112 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.112 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:37:01.112 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:55.259653+0000", 2024-08-19T14:37:01.112 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0699999999999999e-06 2024-08-19T14:37:01.112 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.112 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.112 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:37:01.112 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:55.259657+0000", 2024-08-19T14:37:01.112 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.822e-06 2024-08-19T14:37:01.113 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.113 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.113 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:37:01.113 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:55.259666+0000", 2024-08-19T14:37:01.113 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.9019999999999994e-06 2024-08-19T14:37:01.113 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.113 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.113 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:37:01.113 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:55.259672+0000", 2024-08-19T14:37:01.113 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.096e-06 2024-08-19T14:37:01.113 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.113 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.113 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:37:01.113 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:55.259719+0000", 2024-08-19T14:37:01.114 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7618999999999999e-05 2024-08-19T14:37:01.114 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.114 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.114 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:37:01.114 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:55.259795+0000", 2024-08-19T14:37:01.114 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.6328000000000005e-05 2024-08-19T14:37:01.114 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.114 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.114 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,5", 2024-08-19T14:37:01.114 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:55.259872+0000", 2024-08-19T14:37:01.114 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.6909999999999994e-05 2024-08-19T14:37:01.114 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.114 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.114 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:37:01.115 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:55.265067+0000", 2024-08-19T14:37:01.115 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0051942459999999996 2024-08-19T14:37:01.115 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.115 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.115 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:37:01.115 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:55.267154+0000", 2024-08-19T14:37:01.115 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.002087256 2024-08-19T14:37:01.115 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.115 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.115 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:37:01.115 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:55.480318+0000", 2024-08-19T14:37:01.115 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.213164136 2024-08-19T14:37:01.115 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.115 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.116 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:37:01.116 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:55.480378+0000", 2024-08-19T14:37:01.116 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.0195999999999998e-05 2024-08-19T14:37:01.116 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.116 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.116 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:37:01.116 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:55.480434+0000", 2024-08-19T14:37:01.116 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.5572000000000003e-05 2024-08-19T14:37:01.116 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:37:01.116 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:37:01.116 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:37:01.116 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.116 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.116 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:123536 7.e 7:74990b5e:::benchmark_data_smithi045_115842_object5376:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e638)", 2024-08-19T14:37:01.117 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:35:15.487240+0000", 2024-08-19T14:37:01.117 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 105.52024894199999, 2024-08-19T14:37:01.117 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.56778343899999995, 2024-08-19T14:37:01.117 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:37:01.117 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:37:01.117 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:37:01.117 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:37:01.117 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:37:01.117 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:37:01.117 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 123536 2024-08-19T14:37:01.117 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.117 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:37:01.117 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.117 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:37:01.117 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:15.487240+0000", 2024-08-19T14:37:01.118 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:37:01.118 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.118 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.118 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:37:01.118 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:15.487240+0000", 2024-08-19T14:37:01.118 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:37:01.118 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.118 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.118 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:37:01.118 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:15.487242+0000", 2024-08-19T14:37:01.118 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.61e-06 2024-08-19T14:37:01.118 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.118 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.118 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:37:01.119 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:15.487247+0000", 2024-08-19T14:37:01.119 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.3040000000000004e-06 2024-08-19T14:37:01.119 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.119 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.119 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:37:01.119 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:15.487264+0000", 2024-08-19T14:37:01.119 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.7110000000000001e-05 2024-08-19T14:37:01.119 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.119 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.119 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:37:01.119 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:15.487273+0000", 2024-08-19T14:37:01.119 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.6549999999999993e-06 2024-08-19T14:37:01.119 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.119 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.119 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:37:01.120 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:15.492302+0000", 2024-08-19T14:37:01.120 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00502908 2024-08-19T14:37:01.120 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.120 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.120 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for missing object", 2024-08-19T14:37:01.120 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:15.492390+0000", 2024-08-19T14:37:01.120 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.8712e-05 2024-08-19T14:37:01.120 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.120 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.120 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:37:01.120 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:16.055011+0000", 2024-08-19T14:37:01.120 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.56262038000000003 2024-08-19T14:37:01.120 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.120 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.120 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:37:01.121 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:16.055023+0000", 2024-08-19T14:37:01.121 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2588e-05 2024-08-19T14:37:01.121 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:37:01.121 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:37:01.121 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:37:01.121 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.121 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.121 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:152626 7.e 7:767e2359:::benchmark_data_smithi045_115842_object34466:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e657)", 2024-08-19T14:37:01.121 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:36:30.379682+0000", 2024-08-19T14:37:01.121 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 30.627807317999999, 2024-08-19T14:37:01.121 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.14086295900000001, 2024-08-19T14:37:01.121 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:37:01.122 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:37:01.123 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:37:01.123 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:37:01.123 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:37:01.123 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:37:01.123 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 152626 2024-08-19T14:37:01.123 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.123 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:37:01.123 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.123 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:37:01.123 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:30.379682+0000", 2024-08-19T14:37:01.123 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:37:01.123 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.123 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.124 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:37:01.124 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:30.379682+0000", 2024-08-19T14:37:01.124 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:37:01.124 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.124 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.124 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:37:01.124 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:30.379687+0000", 2024-08-19T14:37:01.124 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.3299999999999998e-06 2024-08-19T14:37:01.124 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.124 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.124 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:37:01.124 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:30.379697+0000", 2024-08-19T14:37:01.124 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.8260000000000005e-06 2024-08-19T14:37:01.124 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.124 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.125 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:37:01.125 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:30.379716+0000", 2024-08-19T14:37:01.125 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.9099000000000001e-05 2024-08-19T14:37:01.125 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.125 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.125 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:37:01.125 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:30.379726+0000", 2024-08-19T14:37:01.125 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0641e-05 2024-08-19T14:37:01.125 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.125 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.125 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:37:01.125 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:30.398313+0000", 2024-08-19T14:37:01.125 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.018586272000000001 2024-08-19T14:37:01.125 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.126 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.126 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:37:01.126 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:30.398626+0000", 2024-08-19T14:37:01.126 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00031284100000000001 2024-08-19T14:37:01.126 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.126 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.126 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 2,5,9", 2024-08-19T14:37:01.126 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:30.399081+0000", 2024-08-19T14:37:01.126 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00045500299999999999 2024-08-19T14:37:01.126 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.126 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.126 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:37:01.127 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:30.407623+0000", 2024-08-19T14:37:01.127 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0085422209999999991 2024-08-19T14:37:01.127 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.127 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.127 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:37:01.127 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:30.414196+0000", 2024-08-19T14:37:01.127 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0065730570000000002 2024-08-19T14:37:01.127 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.127 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.127 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:37:01.127 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:30.470360+0000", 2024-08-19T14:37:01.127 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.056164190000000003 2024-08-19T14:37:01.127 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.127 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.128 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:37:01.128 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:30.515219+0000", 2024-08-19T14:37:01.128 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.044859059999999999 2024-08-19T14:37:01.128 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.128 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.128 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:37:01.128 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:30.515340+0000", 2024-08-19T14:37:01.128 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000121347 2024-08-19T14:37:01.128 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.128 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.128 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:37:01.128 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:30.520545+0000", 2024-08-19T14:37:01.128 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0052040719999999997 2024-08-19T14:37:01.128 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:37:01.129 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:37:01.129 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:37:01.129 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.129 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.129 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:152634 7.e 7:724312e4:::benchmark_data_smithi045_115842_object34474:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e657)", 2024-08-19T14:37:01.129 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:36:30.391760+0000", 2024-08-19T14:37:01.129 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 30.615728652000001, 2024-08-19T14:37:01.129 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.14418678400000001, 2024-08-19T14:37:01.129 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:37:01.129 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:37:01.129 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:37:01.129 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:37:01.129 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:37:01.130 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:37:01.130 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 152634 2024-08-19T14:37:01.130 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.130 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:37:01.130 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.130 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:37:01.130 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:30.391760+0000", 2024-08-19T14:37:01.130 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:37:01.130 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.130 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.130 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:37:01.130 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:30.391760+0000", 2024-08-19T14:37:01.130 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:37:01.130 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.131 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.131 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:37:01.131 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:30.391762+0000", 2024-08-19T14:37:01.131 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.311e-06 2024-08-19T14:37:01.131 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.131 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.131 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:37:01.131 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:30.391767+0000", 2024-08-19T14:37:01.131 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.3669999999999996e-06 2024-08-19T14:37:01.131 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.131 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.131 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:37:01.131 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:30.391779+0000", 2024-08-19T14:37:01.132 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2520999999999999e-05 2024-08-19T14:37:01.132 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.132 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.132 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:37:01.132 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:30.391787+0000", 2024-08-19T14:37:01.132 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.6310000000000003e-06 2024-08-19T14:37:01.132 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.132 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.132 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:37:01.132 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:30.407660+0000", 2024-08-19T14:37:01.132 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.015873394999999998 2024-08-19T14:37:01.132 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.133 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.133 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:37:01.133 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:30.407827+0000", 2024-08-19T14:37:01.133 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00016693 2024-08-19T14:37:01.133 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.133 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.133 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 2,5,9", 2024-08-19T14:37:01.133 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:30.407990+0000", 2024-08-19T14:37:01.133 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00016288000000000001 2024-08-19T14:37:01.133 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.133 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.133 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:37:01.133 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:30.420175+0000", 2024-08-19T14:37:01.133 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.012184438000000001 2024-08-19T14:37:01.134 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.134 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.134 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:37:01.134 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:30.420378+0000", 2024-08-19T14:37:01.134 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00020330500000000001 2024-08-19T14:37:01.134 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.134 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.134 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:37:01.134 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:30.470404+0000", 2024-08-19T14:37:01.134 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.050025996000000003 2024-08-19T14:37:01.134 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.134 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.135 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:37:01.135 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:30.520557+0000", 2024-08-19T14:37:01.135 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.050152714000000001 2024-08-19T14:37:01.135 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.135 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.135 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:37:01.135 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:30.520663+0000", 2024-08-19T14:37:01.135 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000106158 2024-08-19T14:37:01.135 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.135 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.135 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:37:01.136 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:30.535947+0000", 2024-08-19T14:37:01.136 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.015284137999999999 2024-08-19T14:37:01.136 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:37:01.136 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:37:01.136 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:37:01.136 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.136 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.136 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:152637 7.e 7:73d68770:::benchmark_data_smithi045_115842_object34477:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e657)", 2024-08-19T14:37:01.136 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:36:30.409050+0000", 2024-08-19T14:37:01.136 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 30.598439394, 2024-08-19T14:37:01.136 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.13638571499999999, 2024-08-19T14:37:01.136 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:37:01.136 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:37:01.136 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:37:01.137 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:37:01.137 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:37:01.137 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:37:01.137 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 152637 2024-08-19T14:37:01.137 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.137 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:37:01.137 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.137 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:37:01.137 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:30.409050+0000", 2024-08-19T14:37:01.137 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:37:01.137 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.137 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.137 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:37:01.138 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:30.409050+0000", 2024-08-19T14:37:01.138 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:37:01.138 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.138 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.138 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:37:01.138 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:30.409051+0000", 2024-08-19T14:37:01.138 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4270000000000001e-06 2024-08-19T14:37:01.138 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.138 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.138 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:37:01.138 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:30.409056+0000", 2024-08-19T14:37:01.138 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1950000000000002e-06 2024-08-19T14:37:01.138 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.139 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.139 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:37:01.139 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:30.409069+0000", 2024-08-19T14:37:01.139 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2499e-05 2024-08-19T14:37:01.139 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.139 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.139 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:37:01.139 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:30.409075+0000", 2024-08-19T14:37:01.139 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.5989999999999998e-06 2024-08-19T14:37:01.139 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.139 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.139 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:37:01.139 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:30.420554+0000", 2024-08-19T14:37:01.139 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.011479161 2024-08-19T14:37:01.140 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.140 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.140 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:37:01.140 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:30.421037+0000", 2024-08-19T14:37:01.140 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00048266900000000002 2024-08-19T14:37:01.140 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.140 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.140 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 2,5,9", 2024-08-19T14:37:01.140 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:30.421421+0000", 2024-08-19T14:37:01.140 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000384058 2024-08-19T14:37:01.140 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.140 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.140 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:37:01.140 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:30.454707+0000", 2024-08-19T14:37:01.140 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.033285782 2024-08-19T14:37:01.141 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.141 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.141 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:37:01.141 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:30.454762+0000", 2024-08-19T14:37:01.141 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.5546e-05 2024-08-19T14:37:01.141 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.141 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.141 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:37:01.141 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:30.470444+0000", 2024-08-19T14:37:01.141 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.015681961000000001 2024-08-19T14:37:01.141 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.141 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.141 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:37:01.141 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:30.540586+0000", 2024-08-19T14:37:01.142 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.070141110000000007 2024-08-19T14:37:01.142 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.142 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.142 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:37:01.142 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:30.540694+0000", 2024-08-19T14:37:01.142 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000108334 2024-08-19T14:37:01.142 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.142 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.142 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:37:01.142 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:30.545435+0000", 2024-08-19T14:37:01.142 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0047413739999999996 2024-08-19T14:37:01.142 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:37:01.142 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:37:01.143 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:37:01.143 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.143 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.143 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:152779 7.e 7:774ca1cc:::benchmark_data_smithi045_115842_object34619:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e657)", 2024-08-19T14:37:01.143 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:36:30.726525+0000", 2024-08-19T14:37:01.143 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 30.280963678999999, 2024-08-19T14:37:01.143 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.133689376, 2024-08-19T14:37:01.143 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:37:01.143 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:37:01.143 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:37:01.143 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:37:01.143 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:37:01.143 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:37:01.143 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 152779 2024-08-19T14:37:01.144 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.144 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:37:01.144 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.144 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:37:01.144 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:30.726525+0000", 2024-08-19T14:37:01.144 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:37:01.144 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.144 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.144 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:37:01.144 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:30.726525+0000", 2024-08-19T14:37:01.145 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:37:01.145 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.145 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.145 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:37:01.145 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:30.726527+0000", 2024-08-19T14:37:01.145 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.533e-06 2024-08-19T14:37:01.145 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.145 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.145 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:37:01.145 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:30.726533+0000", 2024-08-19T14:37:01.145 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.8860000000000001e-06 2024-08-19T14:37:01.145 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.145 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.145 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:37:01.146 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:30.726548+0000", 2024-08-19T14:37:01.146 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5311e-05 2024-08-19T14:37:01.146 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.146 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.146 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:37:01.146 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:30.726556+0000", 2024-08-19T14:37:01.146 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.8669999999999994e-06 2024-08-19T14:37:01.146 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.146 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.146 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:37:01.146 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:30.745808+0000", 2024-08-19T14:37:01.146 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.019252592999999998 2024-08-19T14:37:01.146 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.147 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.147 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:37:01.147 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:30.745960+0000", 2024-08-19T14:37:01.147 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00015184399999999999 2024-08-19T14:37:01.147 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.147 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.147 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 2,5,9", 2024-08-19T14:37:01.147 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:30.746134+0000", 2024-08-19T14:37:01.147 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000174162 2024-08-19T14:37:01.147 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.147 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.147 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:37:01.147 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:30.775129+0000", 2024-08-19T14:37:01.147 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.028994180000000001 2024-08-19T14:37:01.148 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.148 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.148 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:37:01.148 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:30.775214+0000", 2024-08-19T14:37:01.148 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.5550999999999995e-05 2024-08-19T14:37:01.148 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.148 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.148 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:37:01.148 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:30.795209+0000", 2024-08-19T14:37:01.148 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.019994932999999999 2024-08-19T14:37:01.148 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.148 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.148 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:37:01.149 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:30.855194+0000", 2024-08-19T14:37:01.149 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.059984859000000001 2024-08-19T14:37:01.149 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.149 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.149 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:37:01.149 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:30.855267+0000", 2024-08-19T14:37:01.149 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.3433e-05 2024-08-19T14:37:01.149 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.149 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.149 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:37:01.149 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:30.860215+0000", 2024-08-19T14:37:01.149 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0049472240000000001 2024-08-19T14:37:01.149 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:37:01.150 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:37:01.150 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:37:01.150 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.150 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.150 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:155134 7.e 7:79f22201:::benchmark_data_smithi045_115842_object36974:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e660)", 2024-08-19T14:37:01.150 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:36:36.345168+0000", 2024-08-19T14:37:01.150 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 24.662321031000001, 2024-08-19T14:37:01.150 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.133941858, 2024-08-19T14:37:01.150 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:37:01.150 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:37:01.150 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:37:01.150 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:37:01.150 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:37:01.151 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:37:01.151 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 155134 2024-08-19T14:37:01.151 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.151 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:37:01.151 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.151 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:37:01.151 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:36.345168+0000", 2024-08-19T14:37:01.151 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:37:01.151 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.151 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.151 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:37:01.151 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:36.345168+0000", 2024-08-19T14:37:01.152 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:37:01.152 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.152 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.152 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:37:01.152 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:36.345170+0000", 2024-08-19T14:37:01.152 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.0250000000000001e-06 2024-08-19T14:37:01.152 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.152 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.152 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:37:01.152 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:36.345177+0000", 2024-08-19T14:37:01.152 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.1509999999999998e-06 2024-08-19T14:37:01.152 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.152 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.153 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:37:01.153 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:36.345192+0000", 2024-08-19T14:37:01.153 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4493e-05 2024-08-19T14:37:01.153 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.153 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.153 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:37:01.153 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:36.345204+0000", 2024-08-19T14:37:01.153 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2765000000000001e-05 2024-08-19T14:37:01.153 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.153 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.153 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:37:01.153 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:36.384104+0000", 2024-08-19T14:37:01.153 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.038900169999999998 2024-08-19T14:37:01.153 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.154 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.154 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:37:01.154 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:36.384280+0000", 2024-08-19T14:37:01.154 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00017582200000000001 2024-08-19T14:37:01.154 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.154 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.154 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 2,5,9", 2024-08-19T14:37:01.154 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:36.384473+0000", 2024-08-19T14:37:01.154 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000193124 2024-08-19T14:37:01.154 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.154 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.154 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:37:01.155 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:36.395838+0000", 2024-08-19T14:37:01.155 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.011365068000000001 2024-08-19T14:37:01.155 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.155 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.155 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:37:01.155 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:36.395856+0000", 2024-08-19T14:37:01.155 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.7682000000000001e-05 2024-08-19T14:37:01.155 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.155 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.155 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:37:01.155 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:36.395912+0000", 2024-08-19T14:37:01.155 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.63e-05 2024-08-19T14:37:01.156 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.156 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.156 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:37:01.156 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:36.454319+0000", 2024-08-19T14:37:01.156 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.058406228999999997 2024-08-19T14:37:01.156 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.156 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.156 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:37:01.156 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:36.454513+0000", 2024-08-19T14:37:01.156 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00019471700000000001 2024-08-19T14:37:01.156 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.156 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.157 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:37:01.157 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:36.479110+0000", 2024-08-19T14:37:01.157 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.024596311999999999 2024-08-19T14:37:01.157 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:37:01.157 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:37:01.157 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:37:01.157 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.157 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.157 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:162822 7.e 7:7c7e3d5c:::benchmark_data_smithi045_115842_object44662:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e666)", 2024-08-19T14:37:01.157 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:36:56.878150+0000", 2024-08-19T14:37:01.157 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 4.1293392720000002, 2024-08-19T14:37:01.157 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.16876550700000001, 2024-08-19T14:37:01.157 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:37:01.158 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:37:01.158 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:37:01.158 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:37:01.158 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:37:01.158 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:37:01.158 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 162822 2024-08-19T14:37:01.158 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.158 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:37:01.158 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.158 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:37:01.158 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:56.878150+0000", 2024-08-19T14:37:01.158 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:37:01.158 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.158 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.159 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:37:01.159 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:56.878150+0000", 2024-08-19T14:37:01.159 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:37:01.159 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.159 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.159 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:37:01.159 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:56.878150+0000", 2024-08-19T14:37:01.159 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.03e-07 2024-08-19T14:37:01.159 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.159 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.159 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:37:01.159 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:56.878152+0000", 2024-08-19T14:37:01.159 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.9139999999999998e-06 2024-08-19T14:37:01.159 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.160 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.160 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:37:01.160 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:56.878157+0000", 2024-08-19T14:37:01.160 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.2279999999999998e-06 2024-08-19T14:37:01.160 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.160 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.160 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:37:01.160 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:56.878161+0000", 2024-08-19T14:37:01.160 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.3270000000000002e-06 2024-08-19T14:37:01.160 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.160 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.160 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:37:01.160 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:56.879247+0000", 2024-08-19T14:37:01.161 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0010851140000000001 2024-08-19T14:37:01.161 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.161 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.161 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:37:01.161 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:56.879585+0000", 2024-08-19T14:37:01.161 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00033817899999999998 2024-08-19T14:37:01.161 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.161 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.161 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 2,5,9", 2024-08-19T14:37:01.161 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:56.880009+0000", 2024-08-19T14:37:01.161 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00042433299999999999 2024-08-19T14:37:01.161 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.161 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.162 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:37:01.162 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:56.895208+0000", 2024-08-19T14:37:01.162 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.015199185 2024-08-19T14:37:01.162 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.162 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.162 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:37:01.162 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:56.895259+0000", 2024-08-19T14:37:01.162 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0302000000000001e-05 2024-08-19T14:37:01.162 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.162 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.162 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:37:01.162 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:56.915820+0000", 2024-08-19T14:37:01.162 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.020561494999999999 2024-08-19T14:37:01.162 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.163 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.163 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:37:01.163 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:57.033428+0000", 2024-08-19T14:37:01.163 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.11760771 2024-08-19T14:37:01.163 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.163 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.163 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:37:01.163 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:57.033508+0000", 2024-08-19T14:37:01.163 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.0425000000000003e-05 2024-08-19T14:37:01.163 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.163 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.163 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:37:01.163 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:57.046915+0000", 2024-08-19T14:37:01.164 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.013406992 2024-08-19T14:37:01.164 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:37:01.164 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:37:01.164 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:37:01.164 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.164 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.164 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:162840 7.e 7:77ab8c58:::benchmark_data_smithi045_115842_object44680:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e666)", 2024-08-19T14:37:01.164 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:36:56.912820+0000", 2024-08-19T14:37:01.164 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 4.0946689330000003, 2024-08-19T14:37:01.164 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.153142532, 2024-08-19T14:37:01.164 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:37:01.164 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:37:01.164 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:37:01.165 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:37:01.165 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:37:01.165 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:37:01.165 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 162840 2024-08-19T14:37:01.165 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.165 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:37:01.165 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.165 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:37:01.165 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:56.912820+0000", 2024-08-19T14:37:01.165 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:37:01.165 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.165 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.165 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:37:01.166 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:56.912820+0000", 2024-08-19T14:37:01.166 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:37:01.166 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.166 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.166 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:37:01.166 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:56.912822+0000", 2024-08-19T14:37:01.166 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.68e-06 2024-08-19T14:37:01.166 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.166 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.166 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:37:01.166 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:56.912829+0000", 2024-08-19T14:37:01.166 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.5129999999999999e-06 2024-08-19T14:37:01.166 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.166 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.167 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:37:01.167 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:56.912846+0000", 2024-08-19T14:37:01.167 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6671e-05 2024-08-19T14:37:01.167 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.167 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.167 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:37:01.167 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:56.912856+0000", 2024-08-19T14:37:01.167 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0121e-05 2024-08-19T14:37:01.167 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.167 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.167 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:37:01.167 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:56.915883+0000", 2024-08-19T14:37:01.167 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0030268500000000002 2024-08-19T14:37:01.167 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.167 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.168 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:37:01.168 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:56.916050+0000", 2024-08-19T14:37:01.168 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00016757799999999999 2024-08-19T14:37:01.168 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.168 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.168 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 2,5,9", 2024-08-19T14:37:01.168 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:56.916219+0000", 2024-08-19T14:37:01.168 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00016865900000000001 2024-08-19T14:37:01.168 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.168 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.168 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:37:01.168 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:56.930005+0000", 2024-08-19T14:37:01.168 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.013785874 2024-08-19T14:37:01.169 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.169 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.169 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:37:01.169 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:56.936486+0000", 2024-08-19T14:37:01.169 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0064807750000000003 2024-08-19T14:37:01.169 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.169 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.169 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:37:01.169 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:56.936588+0000", 2024-08-19T14:37:01.169 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000102624 2024-08-19T14:37:01.169 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.169 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.169 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:37:01.169 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:57.047099+0000", 2024-08-19T14:37:01.169 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.110510574 2024-08-19T14:37:01.170 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.170 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.170 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:37:01.170 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:57.047299+0000", 2024-08-19T14:37:01.170 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000200495 2024-08-19T14:37:01.170 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:01.170 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:01.170 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:37:01.170 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:57.065962+0000", 2024-08-19T14:37:01.170 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.018663117999999999 2024-08-19T14:37:01.170 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:37:01.170 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:37:01.170 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:37:01.171 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:37:01.171 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:37:01.171 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:37:01.178 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:37:01.202 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:01.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:01.203 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:37:01.230 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-19T14:37:01.304 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:37:01.328 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:01.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:01.329 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:37:01.430 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:37:01.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:01.480 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:01.480 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:37:01.581 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:37:01.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:01.604 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:01.604 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:37:01.705 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:37:01.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:01.732 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:01.732 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:37:01.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:01 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:37:01.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:01 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:37:01.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:01 smithi032 ceph-mon[20004]: pgmap v1580: 47 pgs: 1 active+clean+scrubbing+deep, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 4.5 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 379 op/s; 25604/219480 objects degraded (11.666%); 8452/219480 objects misplaced (3.851%); 7.1 MiB/s, 114 objects/s recovering 2024-08-19T14:37:01.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:01 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2864461165' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-08-19T14:37:01.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:01 smithi032 ceph-mon[20004]: osdmap e668: 12 total, 12 up, 10 in 2024-08-19T14:37:01.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:01 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:37:01.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:01 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:37:01.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:01 smithi045 ceph-mon[20143]: pgmap v1580: 47 pgs: 1 active+clean+scrubbing+deep, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 4.5 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 379 op/s; 25604/219480 objects degraded (11.666%); 8452/219480 objects misplaced (3.851%); 7.1 MiB/s, 114 objects/s recovering 2024-08-19T14:37:01.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:01 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2864461165' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-08-19T14:37:01.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:01 smithi045 ceph-mon[20143]: osdmap e668: 12 total, 12 up, 10 in 2024-08-19T14:37:01.832 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:37:01.854 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:01.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:01.855 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:37:01.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:01 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:37:01.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:01 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:37:01.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:01 smithi031 ceph-mon[16336]: pgmap v1580: 47 pgs: 1 active+clean+scrubbing+deep, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 4.5 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 379 op/s; 25604/219480 objects degraded (11.666%); 8452/219480 objects misplaced (3.851%); 7.1 MiB/s, 114 objects/s recovering 2024-08-19T14:37:01.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:01 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2864461165' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-08-19T14:37:01.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:01 smithi031 ceph-mon[16336]: osdmap e668: 12 total, 12 up, 10 in 2024-08-19T14:37:01.957 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:37:01.979 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:01.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:01.980 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:37:02.081 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:37:02.102 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:02.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:02.103 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:37:02.204 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:37:02.228 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:02.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:02.229 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:37:02.330 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:37:02.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:02.366 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:02.366 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:37:02.467 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:37:02.489 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:02.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:02.490 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:37:02.591 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:37:02.637 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:02.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:02.638 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:37:02.739 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:37:02.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:02.792 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:02.792 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:37:02.893 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:37:02.943 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:02.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:02.944 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:37:03.045 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:37:03.085 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:03.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:03.086 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:37:03.187 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:37:03.211 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:03.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:03.212 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:37:03.313 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:37:03.336 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:03.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:03.337 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:37:03.418 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:03 smithi045 ceph-mon[20143]: pgmap v1582: 47 pgs: 1 active+clean+scrubbing+deep, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 4.4 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 396 op/s; 25331/217260 objects degraded (11.659%); 8251/217260 objects misplaced (3.798%); 8.2 MiB/s, 131 objects/s recovering 2024-08-19T14:37:03.418 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:03 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 25331/217260 objects degraded (11.659%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-19T14:37:03.418 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {auth=33,config=2,health=12,kv=53} crc {auth=1871003623,config=3457680788,health=614179991,kv=3633068445}) 2024-08-19T14:37:03.418 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {kv=8,logm=92} crc {kv=2206722842,logm=2821128727}) 2024-08-19T14:37:03.418 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3069204546}) 2024-08-19T14:37:03.419 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=420565585}) 2024-08-19T14:37:03.419 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3699450696}) 2024-08-19T14:37:03.419 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4149503268}) 2024-08-19T14:37:03.419 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1091657248}) 2024-08-19T14:37:03.419 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2994086572}) 2024-08-19T14:37:03.419 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=463137172}) 2024-08-19T14:37:03.419 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1383059006}) 2024-08-19T14:37:03.419 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1516906603}) 2024-08-19T14:37:03.419 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1534702555}) 2024-08-19T14:37:03.419 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=787511783}) 2024-08-19T14:37:03.419 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4286864911}) 2024-08-19T14:37:03.419 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2934414184}) 2024-08-19T14:37:03.419 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2917950998}) 2024-08-19T14:37:03.419 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4167439139}) 2024-08-19T14:37:03.420 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1864068493}) 2024-08-19T14:37:03.420 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3509697919}) 2024-08-19T14:37:03.420 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1714381435}) 2024-08-19T14:37:03.420 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2056512513}) 2024-08-19T14:37:03.420 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3530580063}) 2024-08-19T14:37:03.420 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1831373038}) 2024-08-19T14:37:03.420 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=852229581}) 2024-08-19T14:37:03.420 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2670148496}) 2024-08-19T14:37:03.420 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2819154189}) 2024-08-19T14:37:03.420 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=755988769}) 2024-08-19T14:37:03.420 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=353902407}) 2024-08-19T14:37:03.420 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3942957933}) 2024-08-19T14:37:03.420 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1743433351}) 2024-08-19T14:37:03.420 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1609057870}) 2024-08-19T14:37:03.421 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2358407984}) 2024-08-19T14:37:03.421 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3659289983}) 2024-08-19T14:37:03.421 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3852150545}) 2024-08-19T14:37:03.421 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1944677849}) 2024-08-19T14:37:03.421 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1917694053}) 2024-08-19T14:37:03.421 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2199757451}) 2024-08-19T14:37:03.421 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=326214514}) 2024-08-19T14:37:03.438 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:37:03.460 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:03.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:03.461 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:37:03.562 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:37:03.585 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:03.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:03.586 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:37:03.687 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:37:03.739 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:03.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:03.740 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:37:03.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:03 smithi032 ceph-mon[20004]: pgmap v1582: 47 pgs: 1 active+clean+scrubbing+deep, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 4.4 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 396 op/s; 25331/217260 objects degraded (11.659%); 8251/217260 objects misplaced (3.798%); 8.2 MiB/s, 131 objects/s recovering 2024-08-19T14:37:03.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:03 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 25331/217260 objects degraded (11.659%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-19T14:37:03.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {auth=33,config=2,health=12,kv=53} crc {auth=1871003623,config=3457680788,health=614179991,kv=3633068445}) 2024-08-19T14:37:03.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {kv=8,logm=92} crc {kv=2206722842,logm=2821128727}) 2024-08-19T14:37:03.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3069204546}) 2024-08-19T14:37:03.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=420565585}) 2024-08-19T14:37:03.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3699450696}) 2024-08-19T14:37:03.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4149503268}) 2024-08-19T14:37:03.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1091657248}) 2024-08-19T14:37:03.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2994086572}) 2024-08-19T14:37:03.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=463137172}) 2024-08-19T14:37:03.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1383059006}) 2024-08-19T14:37:03.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1516906603}) 2024-08-19T14:37:03.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1534702555}) 2024-08-19T14:37:03.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=787511783}) 2024-08-19T14:37:03.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4286864911}) 2024-08-19T14:37:03.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2934414184}) 2024-08-19T14:37:03.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2917950998}) 2024-08-19T14:37:03.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4167439139}) 2024-08-19T14:37:03.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1864068493}) 2024-08-19T14:37:03.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3509697919}) 2024-08-19T14:37:03.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1714381435}) 2024-08-19T14:37:03.760 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2056512513}) 2024-08-19T14:37:03.760 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3530580063}) 2024-08-19T14:37:03.760 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1831373038}) 2024-08-19T14:37:03.760 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=852229581}) 2024-08-19T14:37:03.760 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2670148496}) 2024-08-19T14:37:03.760 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2819154189}) 2024-08-19T14:37:03.760 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=755988769}) 2024-08-19T14:37:03.760 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=353902407}) 2024-08-19T14:37:03.760 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3942957933}) 2024-08-19T14:37:03.760 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1743433351}) 2024-08-19T14:37:03.761 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1609057870}) 2024-08-19T14:37:03.761 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2358407984}) 2024-08-19T14:37:03.761 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3659289983}) 2024-08-19T14:37:03.761 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3852150545}) 2024-08-19T14:37:03.761 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1944677849}) 2024-08-19T14:37:03.761 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1917694053}) 2024-08-19T14:37:03.761 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2199757451}) 2024-08-19T14:37:03.761 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=326214514}) 2024-08-19T14:37:03.762 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2902924769}) 2024-08-19T14:37:03.762 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=406267191}) 2024-08-19T14:37:03.762 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=43520539}) 2024-08-19T14:37:03.762 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=887094798}) 2024-08-19T14:37:03.762 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3768563892}) 2024-08-19T14:37:03.762 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2611490519}) 2024-08-19T14:37:03.762 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2841943618}) 2024-08-19T14:37:03.762 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1640385440}) 2024-08-19T14:37:03.762 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1656846011}) 2024-08-19T14:37:03.762 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2003884848}) 2024-08-19T14:37:03.762 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3705023499}) 2024-08-19T14:37:03.763 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1757278972}) 2024-08-19T14:37:03.763 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2064615794}) 2024-08-19T14:37:03.763 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2146051578}) 2024-08-19T14:37:03.763 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3689129839}) 2024-08-19T14:37:03.763 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3768759087}) 2024-08-19T14:37:03.763 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1146065637}) 2024-08-19T14:37:03.763 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1439914207}) 2024-08-19T14:37:03.763 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=334414885}) 2024-08-19T14:37:03.763 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2740429007}) 2024-08-19T14:37:03.763 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4260219524}) 2024-08-19T14:37:03.763 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2447165934}) 2024-08-19T14:37:03.763 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2651247019}) 2024-08-19T14:37:03.763 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2536252924}) 2024-08-19T14:37:03.764 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3136482027}) 2024-08-19T14:37:03.764 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=590208619}) 2024-08-19T14:37:03.764 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=48,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=28} crc {logm=1766965707,mdsmap=2474562478,mgr=2924679688,mgr_command_descs=1300230762,mgr_metadata=482233419,mgrstat=2061935566}) 2024-08-19T14:37:03.764 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=100} crc {mgrstat=1979500625}) 2024-08-19T14:37:03.764 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=32,mon_config_key=68} crc {mgrstat=3955565525,mon_config_key=3311414182}) 2024-08-19T14:37:03.764 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1023998681}) 2024-08-19T14:37:03.764 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=602944099}) 2024-08-19T14:37:03.764 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=11,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=71} crc {mon_config_key=1826983336,monmap=3193798142,osd_metadata=289051117,osd_pg_creating=3152913623,osdmap=782068726}) 2024-08-19T14:37:03.764 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=870242582}) 2024-08-19T14:37:03.764 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2864031445}) 2024-08-19T14:37:03.765 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1751086279}) 2024-08-19T14:37:03.765 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1864087582}) 2024-08-19T14:37:03.765 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3640043278}) 2024-08-19T14:37:03.765 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1034219318}) 2024-08-19T14:37:03.765 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=916171002}) 2024-08-19T14:37:03.765 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=349538366}) 2024-08-19T14:37:03.765 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2536927662}) 2024-08-19T14:37:03.765 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2461061247}) 2024-08-19T14:37:03.765 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2807733907}) 2024-08-19T14:37:03.766 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2681996729}) 2024-08-19T14:37:03.766 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=68} crc {osdmap=1863988254}) 2024-08-19T14:37:03.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2902924769}) 2024-08-19T14:37:03.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=406267191}) 2024-08-19T14:37:03.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=43520539}) 2024-08-19T14:37:03.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=887094798}) 2024-08-19T14:37:03.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3768563892}) 2024-08-19T14:37:03.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2611490519}) 2024-08-19T14:37:03.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2841943618}) 2024-08-19T14:37:03.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1640385440}) 2024-08-19T14:37:03.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1656846011}) 2024-08-19T14:37:03.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2003884848}) 2024-08-19T14:37:03.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3705023499}) 2024-08-19T14:37:03.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1757278972}) 2024-08-19T14:37:03.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2064615794}) 2024-08-19T14:37:03.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2146051578}) 2024-08-19T14:37:03.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3689129839}) 2024-08-19T14:37:03.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3768759087}) 2024-08-19T14:37:03.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1146065637}) 2024-08-19T14:37:03.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1439914207}) 2024-08-19T14:37:03.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=334414885}) 2024-08-19T14:37:03.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2740429007}) 2024-08-19T14:37:03.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4260219524}) 2024-08-19T14:37:03.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2447165934}) 2024-08-19T14:37:03.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2651247019}) 2024-08-19T14:37:03.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2536252924}) 2024-08-19T14:37:03.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3136482027}) 2024-08-19T14:37:03.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=590208619}) 2024-08-19T14:37:03.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=48,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=28} crc {logm=1766965707,mdsmap=2474562478,mgr=2924679688,mgr_command_descs=1300230762,mgr_metadata=482233419,mgrstat=2061935566}) 2024-08-19T14:37:03.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=100} crc {mgrstat=1979500625}) 2024-08-19T14:37:03.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=32,mon_config_key=68} crc {mgrstat=3955565525,mon_config_key=3311414182}) 2024-08-19T14:37:03.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1023998681}) 2024-08-19T14:37:03.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=602944099}) 2024-08-19T14:37:03.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=11,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=71} crc {mon_config_key=1826983336,monmap=3193798142,osd_metadata=289051117,osd_pg_creating=3152913623,osdmap=782068726}) 2024-08-19T14:37:03.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=870242582}) 2024-08-19T14:37:03.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2864031445}) 2024-08-19T14:37:03.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1751086279}) 2024-08-19T14:37:03.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1864087582}) 2024-08-19T14:37:03.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3640043278}) 2024-08-19T14:37:03.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1034219318}) 2024-08-19T14:37:03.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=916171002}) 2024-08-19T14:37:03.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=349538366}) 2024-08-19T14:37:03.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2536927662}) 2024-08-19T14:37:03.801 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2461061247}) 2024-08-19T14:37:03.801 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2807733907}) 2024-08-19T14:37:03.801 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2681996729}) 2024-08-19T14:37:03.801 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=68} crc {osdmap=1863988254}) 2024-08-19T14:37:03.841 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:37:03.869 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:03 smithi031 ceph-mon[16336]: pgmap v1582: 47 pgs: 1 active+clean+scrubbing+deep, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 4.4 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 396 op/s; 25331/217260 objects degraded (11.659%); 8251/217260 objects misplaced (3.798%); 8.2 MiB/s, 131 objects/s recovering 2024-08-19T14:37:03.869 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:03 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 25331/217260 objects degraded (11.659%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-19T14:37:03.869 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {auth=33,config=2,health=12,kv=53} crc {auth=1871003623,config=3457680788,health=614179991,kv=3633068445}) 2024-08-19T14:37:03.869 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {kv=8,logm=92} crc {kv=2206722842,logm=2821128727}) 2024-08-19T14:37:03.869 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3069204546}) 2024-08-19T14:37:03.870 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=420565585}) 2024-08-19T14:37:03.870 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3699450696}) 2024-08-19T14:37:03.870 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4149503268}) 2024-08-19T14:37:03.870 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1091657248}) 2024-08-19T14:37:03.870 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2994086572}) 2024-08-19T14:37:03.870 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=463137172}) 2024-08-19T14:37:03.870 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1383059006}) 2024-08-19T14:37:03.870 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1516906603}) 2024-08-19T14:37:03.870 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1534702555}) 2024-08-19T14:37:03.870 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=787511783}) 2024-08-19T14:37:03.870 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4286864911}) 2024-08-19T14:37:03.870 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2934414184}) 2024-08-19T14:37:03.871 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2917950998}) 2024-08-19T14:37:03.871 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4167439139}) 2024-08-19T14:37:03.871 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1864068493}) 2024-08-19T14:37:03.871 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3509697919}) 2024-08-19T14:37:03.871 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1714381435}) 2024-08-19T14:37:03.871 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2056512513}) 2024-08-19T14:37:03.871 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3530580063}) 2024-08-19T14:37:03.871 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1831373038}) 2024-08-19T14:37:03.871 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=852229581}) 2024-08-19T14:37:03.872 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2670148496}) 2024-08-19T14:37:03.872 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2819154189}) 2024-08-19T14:37:03.872 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=755988769}) 2024-08-19T14:37:03.872 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=353902407}) 2024-08-19T14:37:03.872 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3942957933}) 2024-08-19T14:37:03.872 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1743433351}) 2024-08-19T14:37:03.872 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1609057870}) 2024-08-19T14:37:03.872 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2358407984}) 2024-08-19T14:37:03.873 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3659289983}) 2024-08-19T14:37:03.873 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3852150545}) 2024-08-19T14:37:03.873 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1944677849}) 2024-08-19T14:37:03.873 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1917694053}) 2024-08-19T14:37:03.873 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2199757451}) 2024-08-19T14:37:03.873 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=326214514}) 2024-08-19T14:37:03.873 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2902924769}) 2024-08-19T14:37:03.873 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=406267191}) 2024-08-19T14:37:03.873 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=43520539}) 2024-08-19T14:37:03.873 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=887094798}) 2024-08-19T14:37:03.873 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3768563892}) 2024-08-19T14:37:03.873 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2611490519}) 2024-08-19T14:37:03.874 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2841943618}) 2024-08-19T14:37:03.874 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1640385440}) 2024-08-19T14:37:03.874 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1656846011}) 2024-08-19T14:37:03.874 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2003884848}) 2024-08-19T14:37:03.874 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3705023499}) 2024-08-19T14:37:03.874 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1757278972}) 2024-08-19T14:37:03.874 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2064615794}) 2024-08-19T14:37:03.874 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2146051578}) 2024-08-19T14:37:03.874 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3689129839}) 2024-08-19T14:37:03.874 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3768759087}) 2024-08-19T14:37:03.874 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1146065637}) 2024-08-19T14:37:03.874 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1439914207}) 2024-08-19T14:37:03.874 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=334414885}) 2024-08-19T14:37:03.875 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2740429007}) 2024-08-19T14:37:03.875 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4260219524}) 2024-08-19T14:37:03.875 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2447165934}) 2024-08-19T14:37:03.875 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2651247019}) 2024-08-19T14:37:03.875 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2536252924}) 2024-08-19T14:37:03.875 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3136482027}) 2024-08-19T14:37:03.876 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=590208619}) 2024-08-19T14:37:03.876 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=48,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=28} crc {logm=1766965707,mdsmap=2474562478,mgr=2924679688,mgr_command_descs=1300230762,mgr_metadata=482233419,mgrstat=2061935566}) 2024-08-19T14:37:03.876 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=100} crc {mgrstat=1979500625}) 2024-08-19T14:37:03.876 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=32,mon_config_key=68} crc {mgrstat=3955565525,mon_config_key=3311414182}) 2024-08-19T14:37:03.876 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1023998681}) 2024-08-19T14:37:03.877 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=602944099}) 2024-08-19T14:37:03.877 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=11,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=71} crc {mon_config_key=1826983336,monmap=3193798142,osd_metadata=289051117,osd_pg_creating=3152913623,osdmap=782068726}) 2024-08-19T14:37:03.877 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=870242582}) 2024-08-19T14:37:03.877 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2864031445}) 2024-08-19T14:37:03.878 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1751086279}) 2024-08-19T14:37:03.878 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1864087582}) 2024-08-19T14:37:03.878 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3640043278}) 2024-08-19T14:37:03.879 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1034219318}) 2024-08-19T14:37:03.879 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=916171002}) 2024-08-19T14:37:03.879 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=349538366}) 2024-08-19T14:37:03.879 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2536927662}) 2024-08-19T14:37:03.879 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2461061247}) 2024-08-19T14:37:03.879 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2807733907}) 2024-08-19T14:37:03.879 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2681996729}) 2024-08-19T14:37:03.880 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=68} crc {osdmap=1863988254}) 2024-08-19T14:37:03.889 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:03.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:03.891 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:37:03.992 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:37:04.013 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:04.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:04.014 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:37:04.115 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:37:04.138 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:04.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:04.140 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:37:04.241 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:37:04.262 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:04.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:04.263 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:37:04.364 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:37:04.388 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:04.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:04.389 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:37:04.490 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:37:04.511 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:04.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:04.512 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:37:04.613 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:37:04.647 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:04.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:04.648 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:37:04.749 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:37:04.776 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:04.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:04.777 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:37:04.878 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:37:04.902 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:04.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:04.903 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:37:05.004 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:37:05.045 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:05.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:05.046 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:37:05.147 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:37:05.169 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:05.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:05.170 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:37:05.270 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:37:05.292 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:05.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:05.293 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:37:05.394 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:37:05.415 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:05.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:05.416 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:37:05.511 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:37:05.511 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:37:05.517 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:37:05.539 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:05.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:05.540 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:37:05.569 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:37:05.569 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:37:05.569 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:05 smithi031 ceph-mon[16336]: pgmap v1583: 47 pgs: 1 active+clean+scrubbing+deep, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 4.4 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 336 op/s; 25083/215460 objects degraded (11.642%); 8251/215460 objects misplaced (3.829%); 8.1 MiB/s, 130 objects/s recovering 2024-08-19T14:37:05.603 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:37:05.603 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:37:05.641 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:37:05.651 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:37:05.652 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:37:05.690 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:37:05.690 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:37:05.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:05.694 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:05.694 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:37:05.727 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:37:05.727 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:37:05.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:05 smithi032 ceph-mon[20004]: pgmap v1583: 47 pgs: 1 active+clean+scrubbing+deep, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 4.4 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 336 op/s; 25083/215460 objects degraded (11.642%); 8251/215460 objects misplaced (3.829%); 8.1 MiB/s, 130 objects/s recovering 2024-08-19T14:37:05.760 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:37:05.760 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:37:05.795 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:37:05.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:05 smithi045 ceph-mon[20143]: pgmap v1583: 47 pgs: 1 active+clean+scrubbing+deep, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 4.4 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 336 op/s; 25083/215460 objects degraded (11.642%); 8251/215460 objects misplaced (3.829%); 8.1 MiB/s, 130 objects/s recovering 2024-08-19T14:37:05.797 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:37:05.797 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:37:05.821 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:05.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:05.822 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:37:05.832 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:37:05.832 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:37:05.872 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:37:05.872 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:37:05.905 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:37:05.905 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:37:05.923 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:37:05.942 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:37:05.943 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:37:05.969 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:05.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:05.971 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:37:06.071 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:37:06.094 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:06.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:06.095 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:37:06.178 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd reweight 8 1 2024-08-19T14:37:06.196 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:37:06.271 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:06.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:06.272 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:37:06.373 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:37:06.395 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:06.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:06.395 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:37:06.497 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:37:06.519 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:06.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:06.520 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:37:06.621 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:37:06.644 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:06.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:06.645 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:37:06.746 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:37:06.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:06 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:37:06.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:06 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:37:06.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:06 smithi032 ceph-mon[20004]: osdmap e669: 12 total, 12 up, 10 in 2024-08-19T14:37:06.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:06 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2190968904' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:37:06.792 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:06.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:06.793 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:37:06.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:06 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:37:06.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:06 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:37:06.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:06 smithi045 ceph-mon[20143]: osdmap e669: 12 total, 12 up, 10 in 2024-08-19T14:37:06.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:06 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2190968904' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:37:06.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:06 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:37:06.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:06 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:37:06.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:06 smithi031 ceph-mon[16336]: osdmap e669: 12 total, 12 up, 10 in 2024-08-19T14:37:06.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:06 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2190968904' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:37:06.895 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:37:06.916 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:06.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:06.917 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:37:07.000 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:37:07.019 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:37:07.040 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-08-19T14:37:07.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:07.042 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:07.043 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:37:07.143 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:37:07.204 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:07.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:07.216 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:37:07.317 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:37:07.339 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:07.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:07.340 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:37:07.441 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:37:07.463 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:07.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:07.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:37:07.565 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:37:07.606 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:07.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:07.607 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:37:07.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:07 smithi031 ceph-mon[16336]: pgmap v1585: 47 pgs: 1 active+clean+scrubbing+deep, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 4.3 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 390 op/s; 24811/212091 objects degraded (11.698%); 8182/212091 objects misplaced (3.858%); 7.2 MiB/s, 115 objects/s recovering 2024-08-19T14:37:07.708 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:37:07.751 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:07.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:07.752 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:37:07.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:07 smithi032 ceph-mon[20004]: pgmap v1585: 47 pgs: 1 active+clean+scrubbing+deep, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 4.3 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 390 op/s; 24811/212091 objects degraded (11.698%); 8182/212091 objects misplaced (3.858%); 7.2 MiB/s, 115 objects/s recovering 2024-08-19T14:37:07.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:07 smithi045 ceph-mon[20143]: pgmap v1585: 47 pgs: 1 active+clean+scrubbing+deep, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 4.3 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 390 op/s; 24811/212091 objects degraded (11.698%); 8182/212091 objects misplaced (3.858%); 7.2 MiB/s, 115 objects/s recovering 2024-08-19T14:37:07.853 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:37:07.877 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:07.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:07.879 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:37:07.980 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:37:08.001 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:08.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:08.003 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:37:08.103 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:37:08.150 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:08.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:08.151 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:37:08.252 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:37:08.302 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:08.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:08.303 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:37:08.403 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:37:08.429 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:08.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:08.430 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:37:08.531 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:37:08.553 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:08.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:08.554 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:37:08.655 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:37:08.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:08.702 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:08.702 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:37:08.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:08 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 24811/212091 objects degraded (11.698%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-19T14:37:08.786 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:08 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 24811/212091 objects degraded (11.698%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-19T14:37:08.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:08 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 24811/212091 objects degraded (11.698%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-19T14:37:08.803 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:37:08.824 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:08.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:08.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:37:08.926 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:37:08.967 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:08.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:08.968 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:37:09.069 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:37:09.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:09.095 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:09.095 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:37:09.196 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:37:09.233 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:09.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:09.235 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:37:09.336 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:37:09.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:09.390 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:09.390 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:37:09.491 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:37:09.513 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:09.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:09.514 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:37:09.615 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:37:09.637 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:09.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:09.638 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:37:09.739 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:37:09.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:09 smithi032 ceph-mon[20004]: pgmap v1586: 47 pgs: 1 active+clean+scrubbing+deep, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 4.3 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 337 op/s; 24811/211371 objects degraded (11.738%); 8038/211371 objects misplaced (3.803%); 8.3 MiB/s, 133 objects/s recovering 2024-08-19T14:37:09.765 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:09.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:09.766 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:37:09.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:09 smithi045 ceph-mon[20143]: pgmap v1586: 47 pgs: 1 active+clean+scrubbing+deep, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 4.3 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 337 op/s; 24811/211371 objects degraded (11.738%); 8038/211371 objects misplaced (3.803%); 8.3 MiB/s, 133 objects/s recovering 2024-08-19T14:37:09.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:09 smithi031 ceph-mon[16336]: pgmap v1586: 47 pgs: 1 active+clean+scrubbing+deep, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 4.3 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 337 op/s; 24811/211371 objects degraded (11.738%); 8038/211371 objects misplaced (3.803%); 8.3 MiB/s, 133 objects/s recovering 2024-08-19T14:37:09.868 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:37:09.889 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:09.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:09.890 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:37:09.991 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:37:10.102 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:10.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:10.103 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:37:10.204 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:37:10.226 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:10.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:10.227 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:37:10.328 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:37:10.365 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:10.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:10.366 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:37:10.467 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:37:10.489 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:10.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:10.490 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:37:10.591 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:37:10.628 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:10.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:10.629 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:37:10.730 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:37:10.769 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:10.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:10.770 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:37:10.871 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:37:10.921 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:10.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:10.922 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:37:11.023 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:37:11.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:11.063 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:11.063 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:37:11.164 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:37:11.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:11.188 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:11.188 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:37:11.289 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:37:11.311 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:11.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:11.312 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:37:11.414 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:37:11.440 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:11.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:11.441 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:37:11.542 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:37:11.581 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:11.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:11.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:37:11.686 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:37:11.708 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:11.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:11.709 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:37:11.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:11 smithi032 ceph-mon[20004]: pgmap v1587: 47 pgs: 1 active+clean+scrubbing+deep, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 4.2 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 377 op/s; 24550/208041 objects degraded (11.801%); 8038/208041 objects misplaced (3.864%); 8.4 MiB/s, 135 objects/s recovering 2024-08-19T14:37:11.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:11 smithi045 ceph-mon[20143]: pgmap v1587: 47 pgs: 1 active+clean+scrubbing+deep, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 4.2 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 377 op/s; 24550/208041 objects degraded (11.801%); 8038/208041 objects misplaced (3.864%); 8.4 MiB/s, 135 objects/s recovering 2024-08-19T14:37:11.809 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:37:11.832 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:11.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:11.834 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:37:11.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:11 smithi031 ceph-mon[16336]: pgmap v1587: 47 pgs: 1 active+clean+scrubbing+deep, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 4.2 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 377 op/s; 24550/208041 objects degraded (11.801%); 8038/208041 objects misplaced (3.864%); 8.4 MiB/s, 135 objects/s recovering 2024-08-19T14:37:11.934 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:37:11.955 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:11.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:11.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:37:12.057 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:37:12.079 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:12.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:12.081 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:37:12.181 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:37:12.205 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:12.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:12.206 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:37:12.307 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:37:12.328 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:12.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:12.329 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:37:12.430 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:37:12.453 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:12.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:12.455 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:37:12.556 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:37:12.581 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:12.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:12.583 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:37:12.684 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:37:12.707 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:12.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:12.709 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:37:12.810 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:37:12.834 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:12.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:12.835 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:37:12.936 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:37:12.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:12.959 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:12.959 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:37:13.060 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:37:13.081 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:13.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:13.082 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:37:13.183 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:37:13.204 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:13.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:13.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:37:13.306 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:37:13.327 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:13.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:13.328 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:37:13.429 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:37:13.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:13.462 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:13.462 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:37:13.564 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:37:13.586 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:13.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:13.587 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:37:13.688 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:37:13.732 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:13.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:13.734 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:37:13.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:13 smithi032 ceph-mon[20004]: pgmap v1588: 47 pgs: 1 active+clean+scrubbing+deep, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 4.2 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 392 op/s; 24258/205488 objects degraded (11.805%); 7808/205488 objects misplaced (3.800%); 8.5 MiB/s, 137 objects/s recovering 2024-08-19T14:37:13.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:13 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 24258/205488 objects degraded (11.805%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-19T14:37:13.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:13 smithi045 ceph-mon[20143]: pgmap v1588: 47 pgs: 1 active+clean+scrubbing+deep, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 4.2 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 392 op/s; 24258/205488 objects degraded (11.805%); 7808/205488 objects misplaced (3.800%); 8.5 MiB/s, 137 objects/s recovering 2024-08-19T14:37:13.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:13 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 24258/205488 objects degraded (11.805%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-19T14:37:13.829 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:13 smithi031 ceph-mon[16336]: pgmap v1588: 47 pgs: 1 active+clean+scrubbing+deep, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 4.2 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 392 op/s; 24258/205488 objects degraded (11.805%); 7808/205488 objects misplaced (3.800%); 8.5 MiB/s, 137 objects/s recovering 2024-08-19T14:37:13.829 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:13 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 24258/205488 objects degraded (11.805%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-19T14:37:13.834 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:37:13.856 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:13.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:13.857 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:37:13.958 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:37:14.006 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:14.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:14.007 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:37:14.108 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:37:14.131 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:14.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:14.132 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:37:14.233 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:37:14.255 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:14.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:14.256 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:37:14.357 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:37:14.382 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:14.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:14.383 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:37:14.484 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:37:14.509 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:14.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:14.510 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:37:14.611 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:37:14.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:14.677 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:14.677 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:37:14.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:14 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:37:14.778 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:37:14.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:14 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:37:14.799 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:14.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:14.800 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:37:14.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:14 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:37:14.901 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:37:14.928 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:14.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:14.929 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:37:15.030 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:37:15.054 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:15.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:15.055 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:37:15.156 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:37:15.193 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:15.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:15.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:37:15.295 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:37:15.316 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:15.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:15.317 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:37:15.419 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:37:15.442 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:15.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:15.443 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:37:15.544 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:37:15.566 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:15.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:15.567 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:37:15.668 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:37:15.690 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:15.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:15.692 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:37:15.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:15 smithi032 ceph-mon[20004]: pgmap v1589: 47 pgs: 1 active+clean+scrubbing+deep, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 4.1 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 394 op/s; 23939/203607 objects degraded (11.757%); 7808/203607 objects misplaced (3.835%); 8.9 MiB/s, 144 objects/s recovering 2024-08-19T14:37:15.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:15 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:37:15.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:15 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:37:15.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:15 smithi032 ceph-mon[20004]: osdmap e670: 12 total, 12 up, 10 in 2024-08-19T14:37:15.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:15 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:37:15.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:15 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:37:15.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:15 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:37:15.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:15 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:37:15.793 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:37:15.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:15 smithi045 ceph-mon[20143]: pgmap v1589: 47 pgs: 1 active+clean+scrubbing+deep, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 4.1 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 394 op/s; 23939/203607 objects degraded (11.757%); 7808/203607 objects misplaced (3.835%); 8.9 MiB/s, 144 objects/s recovering 2024-08-19T14:37:15.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:15 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:37:15.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:15 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:37:15.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:15 smithi045 ceph-mon[20143]: osdmap e670: 12 total, 12 up, 10 in 2024-08-19T14:37:15.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:15 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:37:15.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:15 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:37:15.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:15 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:37:15.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:15 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:37:15.815 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:15.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:15.816 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:37:15.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:15 smithi031 ceph-mon[16336]: pgmap v1589: 47 pgs: 1 active+clean+scrubbing+deep, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 4.1 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 394 op/s; 23939/203607 objects degraded (11.757%); 7808/203607 objects misplaced (3.835%); 8.9 MiB/s, 144 objects/s recovering 2024-08-19T14:37:15.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:15 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:37:15.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:15 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:37:15.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:15 smithi031 ceph-mon[16336]: osdmap e670: 12 total, 12 up, 10 in 2024-08-19T14:37:15.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:15 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:37:15.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:15 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:37:15.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:15 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:37:15.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:15 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:37:15.917 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:37:15.939 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:15.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:15.940 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:37:16.041 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:37:16.064 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:16.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:16.065 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:37:16.166 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:37:16.213 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:16.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:16.214 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:37:16.315 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:37:16.339 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:16.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:16.340 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:37:16.440 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:37:16.462 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:16.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:16.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:37:16.564 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:37:16.588 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:16.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:16.589 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:37:16.690 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:37:16.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:16.732 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:16.732 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:37:16.833 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:37:16.856 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:16.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:16.857 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:37:16.958 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:37:16.979 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:16.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:16.980 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:37:17.081 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:37:17.102 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:17.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:17.103 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:37:17.204 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:37:17.226 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:17.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:17.227 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:37:17.328 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:37:17.351 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:17.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:17.352 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:37:17.453 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:37:17.508 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:17.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:17.509 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:37:17.610 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:37:17.635 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:17.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:17.637 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:37:17.737 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:37:17.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:17 smithi032 ceph-mon[20004]: pgmap v1591: 47 pgs: 1 active+clean+scrubbing+deep, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 4.1 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 378 op/s; 23589/200736 objects degraded (11.751%); 7722/200736 objects misplaced (3.847%); 9.4 MiB/s, 152 objects/s recovering 2024-08-19T14:37:17.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:17 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:37:17.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:17 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:37:17.759 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:17.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:17.760 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:37:17.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:17 smithi045 ceph-mon[20143]: pgmap v1591: 47 pgs: 1 active+clean+scrubbing+deep, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 4.1 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 378 op/s; 23589/200736 objects degraded (11.751%); 7722/200736 objects misplaced (3.847%); 9.4 MiB/s, 152 objects/s recovering 2024-08-19T14:37:17.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:17 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:37:17.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:17 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:37:17.861 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:37:17.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:17 smithi031 ceph-mon[16336]: pgmap v1591: 47 pgs: 1 active+clean+scrubbing+deep, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 4.1 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 378 op/s; 23589/200736 objects degraded (11.751%); 7722/200736 objects misplaced (3.847%); 9.4 MiB/s, 152 objects/s recovering 2024-08-19T14:37:17.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:17 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:37:17.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:17 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:37:17.883 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:17.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:17.884 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:37:17.985 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:37:18.008 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:18.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:18.009 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:37:18.110 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:37:18.132 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:18.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:18.133 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:37:18.234 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:37:18.260 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:18.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:18.261 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:37:18.362 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:37:18.386 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:18.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:18.387 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:37:18.393 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:37:18.488 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:37:18.537 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:18.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:18.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:37:18.639 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:37:18.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:18.709 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:18.709 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:37:18.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:18 smithi032 ceph-mon[20004]: osdmap e671: 12 total, 12 up, 10 in 2024-08-19T14:37:18.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:18 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 23589/200736 objects degraded (11.751%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-19T14:37:18.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:18 smithi045 ceph-mon[20143]: osdmap e671: 12 total, 12 up, 10 in 2024-08-19T14:37:18.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:18 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 23589/200736 objects degraded (11.751%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-19T14:37:18.809 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:37:18.831 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:18.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:18.832 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:37:18.849 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:37:18.868 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:18 smithi031 ceph-mon[16336]: osdmap e671: 12 total, 12 up, 10 in 2024-08-19T14:37:18.869 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:18 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 23589/200736 objects degraded (11.751%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-19T14:37:18.881 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:37:18.933 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:37:18.955 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:18.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:18.956 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:37:19.057 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:37:19.078 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:19.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:19.079 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:37:19.181 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:37:19.203 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:19.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:19.204 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:37:19.305 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:37:19.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:19.377 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:19.377 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:37:19.478 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:37:19.503 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:19.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:19.504 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:37:19.508 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is set 2024-08-19T14:37:19.605 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:37:19.661 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:19.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:19.662 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:37:19.679 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:37:19.680 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:37:19.680 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:19 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3730328502' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:37:19.680 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:19 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:37:19.680 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:19 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:37:19.680 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:19 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:37:19.680 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:19 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:37:19.680 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:19 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:37:19.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:19 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:37:19.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:19 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:37:19.764 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:37:19.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:19 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3730328502' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:37:19.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:19 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:37:19.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:19 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:37:19.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:19 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:37:19.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:19 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:37:19.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:19 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:37:19.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:19 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:37:19.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:19 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:37:19.819 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:19.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:19.820 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:37:19.921 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:37:19.944 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:19.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:19.945 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:37:20.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:19 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3730328502' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:37:20.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:19 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:37:20.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:19 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:37:20.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:19 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:37:20.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:19 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:37:20.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:19 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:37:20.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:19 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:37:20.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:19 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:37:20.046 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:37:20.066 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:20.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:20.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:37:20.168 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:37:20.190 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:20.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:20.191 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:37:20.292 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:37:20.314 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:20.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:20.315 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:37:20.416 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:37:20.439 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:20.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:20.440 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:37:20.540 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:37:20.605 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:20.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:20.606 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:37:20.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:20 smithi031 ceph-mon[16336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:37:20.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:20 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:37:20.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:20 smithi031 ceph-mon[16336]: osdmap e672: 12 total, 12 up, 10 in 2024-08-19T14:37:20.707 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:37:20.750 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:20.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:20.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:37:20.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:20 smithi045 ceph-mon[20143]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:37:20.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:20 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:37:20.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:20 smithi045 ceph-mon[20143]: osdmap e672: 12 total, 12 up, 10 in 2024-08-19T14:37:20.852 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:37:20.891 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:20.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:20.892 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:37:20.993 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:37:21.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:20 smithi032 ceph-mon[20004]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:37:21.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:20 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:37:21.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:20 smithi032 ceph-mon[20004]: osdmap e672: 12 total, 12 up, 10 in 2024-08-19T14:37:21.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:21.019 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:21.019 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:37:21.120 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:37:21.142 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:21.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:21.143 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:37:21.243 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:37:21.265 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:21.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:21.266 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:37:21.367 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:37:21.410 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:21.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:21.411 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:37:21.511 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:37:21.535 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:21.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:21.537 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:37:21.637 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:37:21.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:21.700 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:21.700 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:37:21.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:21 smithi045 ceph-mon[20143]: pgmap v1595: 47 pgs: 1 active+clean+scrubbing+deep, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 4.0 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 294 op/s; 23323/198309 objects degraded (11.761%); 7533/198309 objects misplaced (3.799%); 8.4 MiB/s, 134 objects/s recovering 2024-08-19T14:37:21.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:21 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2468850469' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-08-19T14:37:21.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:21 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-08-19T14:37:21.801 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:37:21.822 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:21.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:21.823 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:37:21.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:21 smithi031 ceph-mon[16336]: pgmap v1595: 47 pgs: 1 active+clean+scrubbing+deep, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 4.0 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 294 op/s; 23323/198309 objects degraded (11.761%); 7533/198309 objects misplaced (3.799%); 8.4 MiB/s, 134 objects/s recovering 2024-08-19T14:37:21.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:21 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2468850469' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-08-19T14:37:21.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:21 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-08-19T14:37:21.924 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:37:21.970 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:21.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:21.972 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:37:22.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:21 smithi032 ceph-mon[20004]: pgmap v1595: 47 pgs: 1 active+clean+scrubbing+deep, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 4.0 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 294 op/s; 23323/198309 objects degraded (11.761%); 7533/198309 objects misplaced (3.799%); 8.4 MiB/s, 134 objects/s recovering 2024-08-19T14:37:22.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:21 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2468850469' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-08-19T14:37:22.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:21 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-08-19T14:37:22.072 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:37:22.094 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:22.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:22.095 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:37:22.196 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:37:22.219 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:22.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:22.220 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:37:22.321 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:37:22.343 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:22.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:22.344 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:37:22.445 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:37:22.467 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:22.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:22.468 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:37:22.569 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:37:22.590 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:22.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:22.591 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:37:22.650 INFO:teuthology.orchestra.run.smithi031.stderr:reweighted osd.8 to 1 (10000) 2024-08-19T14:37:22.692 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:37:22.733 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:22.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:22.734 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:37:22.835 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:37:22.857 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:22.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:22.858 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:37:22.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:22 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-08-19T14:37:22.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:22 smithi031 ceph-mon[16336]: osdmap e673: 12 total, 12 up, 10 in 2024-08-19T14:37:22.959 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:37:22.982 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:22.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:22.983 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:37:23.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:22 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-08-19T14:37:23.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:22 smithi032 ceph-mon[20004]: osdmap e673: 12 total, 12 up, 10 in 2024-08-19T14:37:23.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:22 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-08-19T14:37:23.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:22 smithi045 ceph-mon[20143]: osdmap e673: 12 total, 12 up, 10 in 2024-08-19T14:37:23.084 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:37:23.105 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:23.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:23.106 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:37:23.207 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:37:23.230 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:23.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:23.231 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:37:23.332 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:37:23.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:23.377 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:23.377 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:37:23.396 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd reweight 2 1 2024-08-19T14:37:23.479 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:37:23.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:23.542 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:23.542 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:37:23.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:23 smithi031 ceph-mon[16336]: pgmap v1597: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 4.0 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 358 op/s; 23028/194286 objects degraded (11.853%); 7322/194286 objects misplaced (3.769%); 9.2 MiB/s, 147 objects/s recovering 2024-08-19T14:37:23.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:23 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 23028/194286 objects degraded (11.853%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-19T14:37:23.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:23 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:37:23.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:23 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:37:23.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:23 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:37:23.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:23 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:37:23.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:23 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:37:23.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:23 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:37:23.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:23 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:37:23.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:23 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:37:23.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:23 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:37:23.644 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:37:23.724 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:23.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:23.725 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:37:23.826 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:37:23.848 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:23.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:23.849 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:37:23.908 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:23 smithi032 ceph-mon[20004]: pgmap v1597: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 4.0 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 358 op/s; 23028/194286 objects degraded (11.853%); 7322/194286 objects misplaced (3.769%); 9.2 MiB/s, 147 objects/s recovering 2024-08-19T14:37:23.908 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:23 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 23028/194286 objects degraded (11.853%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-19T14:37:23.908 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:23 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:37:23.908 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:23 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:37:23.908 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:23 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:37:23.908 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:23 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:37:23.908 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:23 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:37:23.909 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:23 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:37:23.909 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:23 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:37:23.909 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:23 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:37:23.909 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:23 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:37:23.912 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-08-19T14:37:23.947 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:23 smithi045 ceph-mon[20143]: pgmap v1597: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 4.0 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 358 op/s; 23028/194286 objects degraded (11.853%); 7322/194286 objects misplaced (3.769%); 9.2 MiB/s, 147 objects/s recovering 2024-08-19T14:37:23.947 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:23 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 23028/194286 objects degraded (11.853%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-19T14:37:23.947 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:23 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:37:23.947 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:23 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:37:23.947 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:23 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:37:23.947 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:23 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:37:23.947 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:23 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:37:23.947 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:23 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:37:23.947 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:23 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:37:23.948 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:23 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:37:23.948 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:23 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:37:23.950 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:37:23.953 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:37:23.972 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:23.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:23.973 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:37:24.074 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:37:24.146 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:24.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:24.147 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:37:24.249 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:37:24.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:24.336 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:24.336 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:37:24.437 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:37:24.460 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:24.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:24.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:37:24.561 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-19T14:37:24.562 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:37:24.583 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:24.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:24.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:37:24.685 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:37:24.706 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:24.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:24.707 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:37:24.713 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:37:24.757 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:37:24.757 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:37:24.788 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:37:24.808 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:37:24.824 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:37:24.824 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:37:24.831 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:24.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:24.832 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:37:24.867 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:37:24.868 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:37:24.916 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:37:24.917 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:37:24.932 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:37:24.952 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:37:24.952 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:37:24.954 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:24.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:24.955 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:37:24.970 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:37:24.970 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:37:24.987 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:37:24.988 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:37:25.006 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:37:25.007 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:37:25.031 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:37:25.032 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:37:25.056 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:37:25.068 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:37:25.068 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:37:25.079 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:25.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:25.080 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:37:25.103 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:37:25.103 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:37:25.138 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:37:25.139 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:37:25.180 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:37:25.203 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:25.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:25.204 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:37:25.305 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:37:25.348 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:25.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:25.350 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:37:25.451 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:37:25.497 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:25.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:25.498 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:37:25.599 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:37:25.621 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:25.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:25.622 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:37:25.723 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:37:25.749 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:25.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:25.750 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:37:25.851 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:37:25.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:25 smithi031 ceph-mon[16336]: pgmap v1598: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 3.9 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 383 op/s; 22787/192354 objects degraded (11.846%); 7322/192354 objects misplaced (3.807%); 8.7 MiB/s, 139 objects/s recovering 2024-08-19T14:37:25.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:25 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3446028161' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:37:25.897 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:25.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:25.899 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:37:25.999 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:37:26.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:25 smithi032 ceph-mon[20004]: pgmap v1598: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 3.9 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 383 op/s; 22787/192354 objects degraded (11.846%); 7322/192354 objects misplaced (3.807%); 8.7 MiB/s, 139 objects/s recovering 2024-08-19T14:37:26.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:25 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3446028161' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:37:26.023 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:26.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:26.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:37:26.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:25 smithi045 ceph-mon[20143]: pgmap v1598: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 3.9 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 383 op/s; 22787/192354 objects degraded (11.846%); 7322/192354 objects misplaced (3.807%); 8.7 MiB/s, 139 objects/s recovering 2024-08-19T14:37:26.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:25 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3446028161' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:37:26.125 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:37:26.146 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:26.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:26.147 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:37:26.248 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:37:26.269 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:26.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:26.270 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:37:26.371 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:37:26.421 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:26.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:26.422 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:37:26.523 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:37:26.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:26.545 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:26.545 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:37:26.646 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:37:26.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:26.669 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:26.669 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:37:26.771 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:37:26.794 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:26.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:26.795 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:37:26.895 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:37:26.919 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:26.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:26.920 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:37:27.021 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:37:27.043 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:27.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:27.044 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:37:27.145 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:37:27.168 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:27.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:27.169 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:37:27.270 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:37:27.293 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:27.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:27.295 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:37:27.396 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:37:27.422 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:27.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:27.423 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:37:27.524 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:37:27.547 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:27.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:27.547 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:37:27.648 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:37:27.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:27.713 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:27.713 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:37:27.814 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:37:27.836 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:27.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:27.837 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:37:27.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:27 smithi031 ceph-mon[16336]: pgmap v1599: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 3.8 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 467 op/s; 22499/188883 objects degraded (11.912%); 7250/188883 objects misplaced (3.838%); 9.6 MiB/s, 153 objects/s recovering 2024-08-19T14:37:27.937 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:37:27.979 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:27.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:27.981 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:37:28.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:27 smithi032 ceph-mon[20004]: pgmap v1599: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 3.8 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 467 op/s; 22499/188883 objects degraded (11.912%); 7250/188883 objects misplaced (3.838%); 9.6 MiB/s, 153 objects/s recovering 2024-08-19T14:37:28.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:27 smithi045 ceph-mon[20143]: pgmap v1599: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 3.8 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 467 op/s; 22499/188883 objects degraded (11.912%); 7250/188883 objects misplaced (3.838%); 9.6 MiB/s, 153 objects/s recovering 2024-08-19T14:37:28.081 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:37:28.108 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:28.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:28.109 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:37:28.210 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:37:28.254 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:28.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:28.255 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:37:28.356 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:37:28.378 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:28.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:28.379 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:37:28.480 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:37:28.502 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:28.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:28.503 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:37:28.604 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:37:28.626 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:28.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:28.627 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:37:28.728 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:37:28.751 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:28.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:28.752 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:37:28.853 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:37:28.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:28 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 22499/188883 objects degraded (11.912%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-19T14:37:28.874 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:28.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:28.876 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:37:28.977 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:37:28.999 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:29.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:29.000 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:37:29.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:28 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 22499/188883 objects degraded (11.912%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-19T14:37:29.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:28 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 22499/188883 objects degraded (11.912%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-19T14:37:29.102 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:37:29.123 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:29.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:29.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:37:29.225 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:37:29.247 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:29.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:29.248 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:37:29.349 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:37:29.370 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:29.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:29.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:37:29.472 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:37:29.494 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:29.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:29.494 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:37:29.596 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:37:29.618 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:29.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:29.619 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:37:29.720 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:37:29.743 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:29.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:29.744 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:37:29.845 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:37:29.879 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:29.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:29.880 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:37:29.981 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:37:30.005 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:30.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:30.006 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:37:30.107 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:37:30.149 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:30.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:30.150 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:37:30.251 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:37:30.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:29 smithi032 ceph-mon[20004]: pgmap v1600: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 3.8 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 388 op/s; 22499/188157 objects degraded (11.958%); 7108/188157 objects misplaced (3.778%); 7.9 MiB/s, 126 objects/s recovering 2024-08-19T14:37:30.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:29 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:37:30.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:29 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:37:30.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:29 smithi032 ceph-mon[20004]: osdmap e674: 12 total, 12 up, 10 in 2024-08-19T14:37:30.278 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:30.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:30.279 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:37:30.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:29 smithi045 ceph-mon[20143]: pgmap v1600: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 3.8 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 388 op/s; 22499/188157 objects degraded (11.958%); 7108/188157 objects misplaced (3.778%); 7.9 MiB/s, 126 objects/s recovering 2024-08-19T14:37:30.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:29 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:37:30.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:29 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:37:30.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:29 smithi045 ceph-mon[20143]: osdmap e674: 12 total, 12 up, 10 in 2024-08-19T14:37:30.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:29 smithi031 ceph-mon[16336]: pgmap v1600: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 3.8 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 388 op/s; 22499/188157 objects degraded (11.958%); 7108/188157 objects misplaced (3.778%); 7.9 MiB/s, 126 objects/s recovering 2024-08-19T14:37:30.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:29 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:37:30.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:29 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:37:30.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:29 smithi031 ceph-mon[16336]: osdmap e674: 12 total, 12 up, 10 in 2024-08-19T14:37:30.380 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:37:30.402 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:30.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:30.403 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:37:30.504 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:37:30.527 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:30.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:30.528 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:37:30.629 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:37:30.673 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:30.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:30.674 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:37:30.775 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:37:30.798 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:30.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:30.799 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:37:30.900 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:37:30.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:30.923 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:30.923 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:37:30.987 INFO:teuthology.orchestra.run.smithi031.stderr:reweighted osd.2 to 1 (10000) 2024-08-19T14:37:31.024 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:37:31.047 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:31.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:31.048 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:37:31.149 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:37:31.172 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:31.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:31.173 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:37:31.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:30 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3976631150' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-08-19T14:37:31.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:30 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-08-19T14:37:31.274 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:37:31.295 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:31.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:31.296 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:37:31.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:30 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3976631150' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-08-19T14:37:31.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:30 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-08-19T14:37:31.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:30 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3976631150' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-08-19T14:37:31.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:30 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-08-19T14:37:31.397 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:37:31.420 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:31.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:31.421 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:37:31.521 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:37:31.544 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:31.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:31.545 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:37:31.646 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:37:31.669 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:31.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:31.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:37:31.771 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:37:31.834 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:31.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:31.835 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:37:31.936 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:37:31.943 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:37:31.943 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-08-19T14:37:31.943 INFO:teuthology.orchestra.run.smithi031.stdout: "complaint_time": 30, 2024-08-19T14:37:31.943 INFO:teuthology.orchestra.run.smithi031.stdout: "num_blocked_ops": 0 2024-08-19T14:37:31.944 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:37:31.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:31.959 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:31.960 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:37:32.060 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:37:32.102 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:32.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:32.103 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:37:32.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:31 smithi031 ceph-mon[16336]: pgmap v1602: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 3.8 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 342 op/s; 22232/186114 objects degraded (11.945%); 7108/186114 objects misplaced (3.819%); 7.7 MiB/s, 123 objects/s recovering 2024-08-19T14:37:32.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:31 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-08-19T14:37:32.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:31 smithi031 ceph-mon[16336]: osdmap e675: 12 total, 12 up, 10 in 2024-08-19T14:37:32.204 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:37:32.225 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:32.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:32.226 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:37:32.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:31 smithi032 ceph-mon[20004]: pgmap v1602: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 3.8 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 342 op/s; 22232/186114 objects degraded (11.945%); 7108/186114 objects misplaced (3.819%); 7.7 MiB/s, 123 objects/s recovering 2024-08-19T14:37:32.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:31 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-08-19T14:37:32.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:31 smithi032 ceph-mon[20004]: osdmap e675: 12 total, 12 up, 10 in 2024-08-19T14:37:32.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:31 smithi045 ceph-mon[20143]: pgmap v1602: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 3.8 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 342 op/s; 22232/186114 objects degraded (11.945%); 7108/186114 objects misplaced (3.819%); 7.7 MiB/s, 123 objects/s recovering 2024-08-19T14:37:32.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:31 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-08-19T14:37:32.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:31 smithi045 ceph-mon[20143]: osdmap e675: 12 total, 12 up, 10 in 2024-08-19T14:37:32.327 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:37:32.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:32.375 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:32.375 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:37:32.383 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd reweight 11 1 2024-08-19T14:37:32.477 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:37:32.498 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:32.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:32.499 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:37:32.596 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:37:32.600 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:37:32.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:32.667 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:32.667 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:37:32.768 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:37:32.827 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:32.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:32.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:37:32.929 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:37:32.971 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:32.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:32.972 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:37:33.070 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-08-19T14:37:33.073 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:37:33.171 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:33.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:33.172 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:37:33.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:32 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 22232/186114 objects degraded (11.945%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-19T14:37:33.273 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:37:33.294 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:33.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:33.295 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:37:33.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:32 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 22232/186114 objects degraded (11.945%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-19T14:37:33.363 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:32 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 22232/186114 objects degraded (11.945%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-19T14:37:33.395 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:37:33.417 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:33.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:33.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:37:33.427 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:37:33.518 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:37:33.579 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:33.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:33.580 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:37:33.681 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:37:33.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:33.780 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:33.780 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:37:33.805 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:37:33.846 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:37:33.881 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:37:33.903 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:33.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:33.904 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:37:34.005 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:37:34.086 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:34.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:34.087 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:37:34.188 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:37:34.239 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:34.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:34.241 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:37:34.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:33 smithi032 ceph-mon[20004]: pgmap v1604: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 3.7 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 464 op/s; 21926/181197 objects degraded (12.101%); 6881/181197 objects misplaced (3.798%); 9.0 MiB/s, 144 objects/s recovering 2024-08-19T14:37:34.294 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:33 smithi031 ceph-mon[16336]: pgmap v1604: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 3.7 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 464 op/s; 21926/181197 objects degraded (12.101%); 6881/181197 objects misplaced (3.798%); 9.0 MiB/s, 144 objects/s recovering 2024-08-19T14:37:34.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:33 smithi045 ceph-mon[20143]: pgmap v1604: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 3.7 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 464 op/s; 21926/181197 objects degraded (12.101%); 6881/181197 objects misplaced (3.798%); 9.0 MiB/s, 144 objects/s recovering 2024-08-19T14:37:34.342 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:37:34.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:34.365 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:34.365 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:37:34.466 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:37:34.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:34.490 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:34.490 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:37:34.591 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:37:34.615 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:34.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:34.616 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:37:34.717 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:37:34.739 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:34.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:34.740 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:37:34.841 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:37:34.863 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:34.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:34.864 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:37:34.965 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:37:34.986 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:34.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:34.987 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:37:35.005 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is unset 2024-08-19T14:37:35.088 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:37:35.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:34 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2987369110' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:37:35.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:35.138 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:35.138 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:37:35.239 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:37:35.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:34 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2987369110' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:37:35.261 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:35.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:35.262 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:37:35.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:34 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2987369110' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:37:35.363 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:37:35.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:35.385 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:35.385 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:37:35.486 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:37:35.508 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:35.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:35.509 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:37:35.610 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:37:35.633 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:35.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:35.634 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:37:35.735 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:37:35.787 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:35.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:35.788 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:37:35.889 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:37:35.911 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:35.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:35.912 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:37:36.013 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:37:36.025 INFO:teuthology.orchestra.run.smithi031.stderr:reweighted osd.11 to 1 (10000) 2024-08-19T14:37:36.034 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:36.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:36.035 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:37:36.136 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:37:36.160 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:36.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:36.161 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:37:36.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:35 smithi032 ceph-mon[20004]: pgmap v1605: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 3.6 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 411 op/s; 21678/179004 objects degraded (12.110%); 6881/179004 objects misplaced (3.844%); 8.6 MiB/s, 139 objects/s recovering 2024-08-19T14:37:36.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:36 smithi032 ceph-mon[20004]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:37:36.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:36 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2987369110' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:37:36.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:36 smithi032 ceph-mon[20004]: osdmap e676: 12 total, 12 up, 10 in 2024-08-19T14:37:36.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:36 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2814523430' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-08-19T14:37:36.262 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:37:36.294 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:36.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:36.295 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:37:36.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:35 smithi045 ceph-mon[20143]: pgmap v1605: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 3.6 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 411 op/s; 21678/179004 objects degraded (12.110%); 6881/179004 objects misplaced (3.844%); 8.6 MiB/s, 139 objects/s recovering 2024-08-19T14:37:36.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:35 smithi045 ceph-mon[20143]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:37:36.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:35 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2987369110' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:37:36.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:35 smithi045 ceph-mon[20143]: osdmap e676: 12 total, 12 up, 10 in 2024-08-19T14:37:36.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:35 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2814523430' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-08-19T14:37:36.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:35 smithi031 ceph-mon[16336]: pgmap v1605: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 3.6 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 411 op/s; 21678/179004 objects degraded (12.110%); 6881/179004 objects misplaced (3.844%); 8.6 MiB/s, 139 objects/s recovering 2024-08-19T14:37:36.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:35 smithi031 ceph-mon[16336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:37:36.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:35 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2987369110' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:37:36.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:35 smithi031 ceph-mon[16336]: osdmap e676: 12 total, 12 up, 10 in 2024-08-19T14:37:36.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:35 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2814523430' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-08-19T14:37:36.396 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:37:36.418 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:36.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:36.419 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:37:36.520 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:37:36.542 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:36.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:36.543 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:37:36.644 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:37:36.667 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:36.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:36.668 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:37:36.769 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:37:36.791 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:36.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:36.792 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:37:36.893 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:37:36.939 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:36.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:36.940 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:37:37.041 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:37:37.064 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:37.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:37.065 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:37:37.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:37 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:37:37.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:37 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:37:37.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:37 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2814523430' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-08-19T14:37:37.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:37 smithi031 ceph-mon[16336]: osdmap e677: 12 total, 12 up, 10 in 2024-08-19T14:37:37.166 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:37:37.187 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:37.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:37.189 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:37:37.290 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:37:37.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:37 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:37:37.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:37 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:37:37.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:37 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2814523430' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-08-19T14:37:37.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:37 smithi045 ceph-mon[20143]: osdmap e677: 12 total, 12 up, 10 in 2024-08-19T14:37:37.312 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:37.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:37.313 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:37:37.378 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:37:37.378 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 20, 2024-08-19T14:37:37.378 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 600, 2024-08-19T14:37:37.378 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [ 2024-08-19T14:37:37.378 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.379 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:120225 6.6 6:68f0d154:::benchmark_data_smithi045_97937_object12952:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e552)", 2024-08-19T14:37:37.379 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:30:35.204256+0000", 2024-08-19T14:37:37.379 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 421.52701253499998, 2024-08-19T14:37:37.379 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1367397340000001, 2024-08-19T14:37:37.379 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:37:37.379 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:37:37.379 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:37:37.379 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:37:37.379 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:37:37.379 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:37:37.379 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 120225 2024-08-19T14:37:37.379 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.379 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:37:37.380 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.380 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:37:37.380 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:35.204256+0000", 2024-08-19T14:37:37.380 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:37:37.380 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.380 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.380 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:37:37.380 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:35.204256+0000", 2024-08-19T14:37:37.380 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:37:37.380 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.380 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.380 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:37:37.381 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:35.204256+0000", 2024-08-19T14:37:37.381 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.7700000000000001e-07 2024-08-19T14:37:37.381 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.381 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.381 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:37:37.381 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:35.204258+0000", 2024-08-19T14:37:37.381 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.1229999999999998e-06 2024-08-19T14:37:37.381 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.381 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.381 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:37:37.381 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:35.204263+0000", 2024-08-19T14:37:37.381 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.9080000000000003e-06 2024-08-19T14:37:37.382 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.382 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.382 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:37:37.382 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:35.204268+0000", 2024-08-19T14:37:37.382 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.8169999999999998e-06 2024-08-19T14:37:37.382 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.382 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.382 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:37:37.382 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:35.214570+0000", 2024-08-19T14:37:37.382 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.010301805000000001 2024-08-19T14:37:37.382 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.382 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.383 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for missing object", 2024-08-19T14:37:37.383 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:35.214727+0000", 2024-08-19T14:37:37.383 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00015671700000000001 2024-08-19T14:37:37.383 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.383 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.383 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:37:37.383 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:36.340980+0000", 2024-08-19T14:37:37.383 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1262536810000001 2024-08-19T14:37:37.383 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.383 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.383 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:37:37.383 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:36.340996+0000", 2024-08-19T14:37:37.384 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5406000000000001e-05 2024-08-19T14:37:37.384 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:37:37.384 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:37:37.384 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:37:37.384 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.384 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.384 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:120229 6.6 6:6672e843:::benchmark_data_smithi045_97937_object12956:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e552)", 2024-08-19T14:37:37.384 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:30:35.204275+0000", 2024-08-19T14:37:37.384 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 421.52699369800001, 2024-08-19T14:37:37.384 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.151004988, 2024-08-19T14:37:37.384 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:37:37.384 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:37:37.384 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:37:37.384 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:37:37.385 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:37:37.385 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:37:37.385 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 120229 2024-08-19T14:37:37.385 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.385 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:37:37.385 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.385 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:37:37.385 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:35.204275+0000", 2024-08-19T14:37:37.385 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:37:37.385 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.385 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.385 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:37:37.385 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:35.204275+0000", 2024-08-19T14:37:37.385 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:37:37.386 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.386 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.386 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:37:37.386 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:35.204275+0000", 2024-08-19T14:37:37.386 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.9299999999999999e-07 2024-08-19T14:37:37.386 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.386 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.386 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:37:37.386 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:35.204278+0000", 2024-08-19T14:37:37.386 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.3199999999999998e-06 2024-08-19T14:37:37.386 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.386 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.386 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:37:37.387 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:35.204283+0000", 2024-08-19T14:37:37.387 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.7780000000000004e-06 2024-08-19T14:37:37.387 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.387 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.387 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:37:37.387 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:35.204289+0000", 2024-08-19T14:37:37.387 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.3800000000000002e-06 2024-08-19T14:37:37.387 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.387 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.387 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:37:37.387 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:35.214760+0000", 2024-08-19T14:37:37.387 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.010471044000000001 2024-08-19T14:37:37.387 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.388 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.388 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for missing object", 2024-08-19T14:37:37.388 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:35.215051+0000", 2024-08-19T14:37:37.388 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00029112499999999999 2024-08-19T14:37:37.388 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.388 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.388 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:37:37.388 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:36.355266+0000", 2024-08-19T14:37:37.388 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1402151220000001 2024-08-19T14:37:37.388 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.388 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.388 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:37:37.388 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:36.355280+0000", 2024-08-19T14:37:37.389 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3825999999999999e-05 2024-08-19T14:37:37.389 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:37:37.389 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:37:37.389 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:37:37.389 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.389 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.389 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:120799 6.6 6:6d4cbd5c:::benchmark_data_smithi045_97937_object13526:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e552)", 2024-08-19T14:37:37.389 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:30:36.258857+0000", 2024-08-19T14:37:37.389 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 420.47241200799999, 2024-08-19T14:37:37.389 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2809968280000001, 2024-08-19T14:37:37.389 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:37:37.389 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:37:37.389 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:37:37.389 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:37:37.389 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:37:37.390 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:37:37.390 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 120799 2024-08-19T14:37:37.390 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.390 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:37:37.390 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.390 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:37:37.390 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:36.258857+0000", 2024-08-19T14:37:37.390 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:37:37.390 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.390 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.390 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:37:37.390 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:36.258857+0000", 2024-08-19T14:37:37.390 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:37:37.391 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.391 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.391 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:37:37.391 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:36.258857+0000", 2024-08-19T14:37:37.391 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.01e-07 2024-08-19T14:37:37.391 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.391 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.391 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:37:37.391 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:36.258863+0000", 2024-08-19T14:37:37.391 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.1419999999999998e-06 2024-08-19T14:37:37.391 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.391 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.391 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:37:37.392 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:36.258870+0000", 2024-08-19T14:37:37.392 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.4699999999999999e-06 2024-08-19T14:37:37.392 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.392 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.392 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:37:37.392 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:36.258875+0000", 2024-08-19T14:37:37.392 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.3580000000000002e-06 2024-08-19T14:37:37.392 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.392 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.392 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:37:37.392 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:36.261800+0000", 2024-08-19T14:37:37.392 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.002925422 2024-08-19T14:37:37.392 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.393 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.393 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for missing object", 2024-08-19T14:37:37.393 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:36.261888+0000", 2024-08-19T14:37:37.393 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.7714999999999996e-05 2024-08-19T14:37:37.393 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.393 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.393 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:37:37.393 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:40.539815+0000", 2024-08-19T14:37:37.393 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2779266820000004 2024-08-19T14:37:37.393 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.393 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.393 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:37:37.393 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:40.539853+0000", 2024-08-19T14:37:37.393 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.8637999999999998e-05 2024-08-19T14:37:37.394 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:37:37.394 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:37:37.394 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:37:37.394 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.394 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.394 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:137589 6.6 6:61484f3f:::benchmark_data_smithi045_97937_object30316:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e569)", 2024-08-19T14:37:37.394 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:31:13.695526+0000", 2024-08-19T14:37:37.394 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 383.03574256899998, 2024-08-19T14:37:37.394 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.8720014850000002, 2024-08-19T14:37:37.394 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:37:37.394 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:37:37.394 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:37:37.395 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:37:37.395 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:37:37.395 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:37:37.395 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 137589 2024-08-19T14:37:37.395 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.395 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:37:37.395 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.395 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:37:37.395 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:31:13.695526+0000", 2024-08-19T14:37:37.395 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:37:37.395 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.395 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.395 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:37:37.395 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:31:13.695526+0000", 2024-08-19T14:37:37.396 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:37:37.396 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.396 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.396 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:37:37.396 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:31:13.695526+0000", 2024-08-19T14:37:37.396 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.6100000000000001e-07 2024-08-19T14:37:37.396 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.396 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.396 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:37:37.396 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:31:13.695530+0000", 2024-08-19T14:37:37.396 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.0979999999999998e-06 2024-08-19T14:37:37.396 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.396 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.396 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:37:37.397 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:31:13.695537+0000", 2024-08-19T14:37:37.397 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.7109999999999995e-06 2024-08-19T14:37:37.397 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.397 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.397 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:37:37.397 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:31:13.695543+0000", 2024-08-19T14:37:37.397 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.5500000000000002e-06 2024-08-19T14:37:37.397 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.397 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.397 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:37:37.397 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:31:13.695732+0000", 2024-08-19T14:37:37.397 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000188801 2024-08-19T14:37:37.397 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.397 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.398 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for missing object", 2024-08-19T14:37:37.398 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:31:13.695811+0000", 2024-08-19T14:37:37.398 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.9010000000000004e-05 2024-08-19T14:37:37.398 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.398 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.398 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:37:37.398 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:31:16.567509+0000", 2024-08-19T14:37:37.398 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.8716988030000001 2024-08-19T14:37:37.398 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.398 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.398 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:37:37.398 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:31:16.567527+0000", 2024-08-19T14:37:37.398 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.8051e-05 2024-08-19T14:37:37.399 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:37:37.399 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:37:37.399 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:37:37.399 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.399 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.399 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:168001 6.6 6:607c6180:::benchmark_data_smithi045_97937_object60728:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e576)", 2024-08-19T14:37:37.399 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:32:03.729180+0000", 2024-08-19T14:37:37.399 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 333.00208813199998, 2024-08-19T14:37:37.399 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.1240902739999998, 2024-08-19T14:37:37.399 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:37:37.399 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:37:37.399 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:37:37.399 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:37:37.399 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:37:37.400 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:37:37.400 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 168001 2024-08-19T14:37:37.400 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.400 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:37:37.400 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.400 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:37:37.400 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:03.729180+0000", 2024-08-19T14:37:37.400 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:37:37.400 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.400 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.400 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:37:37.400 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:03.729180+0000", 2024-08-19T14:37:37.400 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:37:37.400 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.401 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.401 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:37:37.401 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:03.729181+0000", 2024-08-19T14:37:37.401 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.1199999999999999e-07 2024-08-19T14:37:37.401 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.401 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.401 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:37:37.401 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:03.729182+0000", 2024-08-19T14:37:37.401 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.319e-06 2024-08-19T14:37:37.401 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.401 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.401 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:37:37.402 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:03.729185+0000", 2024-08-19T14:37:37.402 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.0180000000000002e-06 2024-08-19T14:37:37.402 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.402 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.402 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:37:37.402 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:03.729188+0000", 2024-08-19T14:37:37.402 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.5859999999999999e-06 2024-08-19T14:37:37.402 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.402 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.402 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:37:37.402 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:03.734604+0000", 2024-08-19T14:37:37.402 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0054159550000000001 2024-08-19T14:37:37.402 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.402 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.403 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for missing object", 2024-08-19T14:37:37.403 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:03.743722+0000", 2024-08-19T14:37:37.403 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0091187130000000005 2024-08-19T14:37:37.403 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.403 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.403 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:37:37.403 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:05.853261+0000", 2024-08-19T14:37:37.403 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.1095391330000002 2024-08-19T14:37:37.403 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.403 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.403 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:37:37.403 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:05.853271+0000", 2024-08-19T14:37:37.403 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.3379999999999994e-06 2024-08-19T14:37:37.403 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:37:37.404 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:37:37.404 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:37:37.404 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.404 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.404 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:122558 7.3 7.e5c5e023 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e634)", 2024-08-19T14:37:37.404 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:35:10.558456+0000", 2024-08-19T14:37:37.404 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 146.17281269599999, 2024-08-19T14:37:37.404 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.96762901499999998, 2024-08-19T14:37:37.404 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:37:37.404 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:37:37.404 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:37:37.404 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:37:37.404 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:37:37.405 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:37:37.405 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 122558 2024-08-19T14:37:37.405 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.405 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:37:37.405 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.405 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:37:37.405 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:10.558456+0000", 2024-08-19T14:37:37.405 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:37:37.405 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.405 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.405 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:37:37.405 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:10.558456+0000", 2024-08-19T14:37:37.405 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:37:37.405 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.406 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.406 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:37:37.406 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:10.558457+0000", 2024-08-19T14:37:37.406 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.595e-06 2024-08-19T14:37:37.406 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.406 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.406 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:37:37.406 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:10.558464+0000", 2024-08-19T14:37:37.406 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.2060000000000004e-06 2024-08-19T14:37:37.406 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.406 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.406 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:37:37.406 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:10.558477+0000", 2024-08-19T14:37:37.406 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3553999999999999e-05 2024-08-19T14:37:37.407 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.407 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.407 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:37:37.407 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:10.558485+0000", 2024-08-19T14:37:37.407 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.926e-06 2024-08-19T14:37:37.407 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.407 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.407 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:37:37.407 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:10.558573+0000", 2024-08-19T14:37:37.407 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.8038e-05 2024-08-19T14:37:37.407 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.407 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.407 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:37:37.407 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:10.558591+0000", 2024-08-19T14:37:37.408 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.7652999999999999e-05 2024-08-19T14:37:37.408 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.408 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.408 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:37:37.408 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:11.526036+0000", 2024-08-19T14:37:37.408 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.96744472199999998 2024-08-19T14:37:37.408 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.408 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.408 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:37:37.408 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:11.526085+0000", 2024-08-19T14:37:37.408 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.9320999999999998e-05 2024-08-19T14:37:37.408 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:37:37.408 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:37:37.409 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:37:37.409 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.409 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.409 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:135117 7.2 7:4a31cbca:::benchmark_data_smithi045_115842_object16957:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e647)", 2024-08-19T14:37:37.409 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:35:46.362398+0000", 2024-08-19T14:37:37.409 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 110.368870849, 2024-08-19T14:37:37.409 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.99514260099999996, 2024-08-19T14:37:37.409 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:37:37.409 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:37:37.409 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:37:37.409 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:37:37.409 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:37:37.409 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:37:37.409 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 135117 2024-08-19T14:37:37.410 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.410 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:37:37.410 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.410 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:37:37.410 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:46.362398+0000", 2024-08-19T14:37:37.410 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:37:37.410 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.410 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.410 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:37:37.410 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:46.362398+0000", 2024-08-19T14:37:37.410 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:37:37.410 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.410 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.411 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:37:37.411 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:46.362398+0000", 2024-08-19T14:37:37.411 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4999999999999998e-07 2024-08-19T14:37:37.411 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.411 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.411 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:37:37.411 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:46.362401+0000", 2024-08-19T14:37:37.411 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.9950000000000001e-06 2024-08-19T14:37:37.411 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.411 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.411 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:37:37.411 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:46.362412+0000", 2024-08-19T14:37:37.411 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0849000000000001e-05 2024-08-19T14:37:37.411 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.412 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.412 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:37:37.412 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:46.362418+0000", 2024-08-19T14:37:37.412 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.1020000000000002e-06 2024-08-19T14:37:37.412 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.412 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.412 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:37:37.412 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:46.367475+0000", 2024-08-19T14:37:37.412 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.005057153 2024-08-19T14:37:37.412 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.412 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.412 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for missing object", 2024-08-19T14:37:37.412 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:46.367558+0000", 2024-08-19T14:37:37.412 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.2310000000000003e-05 2024-08-19T14:37:37.413 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.413 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.413 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:37:37.413 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:47.357530+0000", 2024-08-19T14:37:37.413 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.98997215599999999 2024-08-19T14:37:37.413 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.413 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.414 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:37:37.414 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:47.357540+0000", 2024-08-19T14:37:37.414 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0586e-05 2024-08-19T14:37:37.414 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:37:37.414 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:37:37.414 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:37:37.414 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.414 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.414 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:135128 7.2 7:438de927:::benchmark_data_smithi045_115842_object16968:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e647)", 2024-08-19T14:37:37.414 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:35:46.362507+0000", 2024-08-19T14:37:37.414 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 110.368761948, 2024-08-19T14:37:37.414 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.99963887799999995, 2024-08-19T14:37:37.414 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:37:37.415 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:37:37.415 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:37:37.415 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:37:37.415 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:37:37.415 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:37:37.415 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 135128 2024-08-19T14:37:37.415 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.415 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:37:37.415 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.415 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:37:37.415 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:46.362507+0000", 2024-08-19T14:37:37.415 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:37:37.415 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.415 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.416 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:37:37.416 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:46.362507+0000", 2024-08-19T14:37:37.416 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:37:37.416 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.416 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.416 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:37:37.416 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:46.362507+0000", 2024-08-19T14:37:37.416 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.8400000000000005e-07 2024-08-19T14:37:37.416 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.416 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.416 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:37:37.416 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:46.362510+0000", 2024-08-19T14:37:37.416 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.7949999999999999e-06 2024-08-19T14:37:37.416 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.417 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.417 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:37:37.417 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:46.362520+0000", 2024-08-19T14:37:37.417 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.9289999999999993e-06 2024-08-19T14:37:37.417 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.417 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.417 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:37:37.417 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:46.362525+0000", 2024-08-19T14:37:37.417 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.575e-06 2024-08-19T14:37:37.417 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.417 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.417 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:37:37.417 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:46.367599+0000", 2024-08-19T14:37:37.418 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0050739490000000003 2024-08-19T14:37:37.418 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.418 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.418 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for missing object", 2024-08-19T14:37:37.418 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:46.367668+0000", 2024-08-19T14:37:37.418 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.8748e-05 2024-08-19T14:37:37.418 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.418 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.418 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:37:37.418 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:47.362133+0000", 2024-08-19T14:37:37.418 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.99446453099999998 2024-08-19T14:37:37.418 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.418 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.419 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:37:37.419 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:47.362146+0000", 2024-08-19T14:37:37.419 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2867000000000001e-05 2024-08-19T14:37:37.419 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:37:37.419 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:37:37.419 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:37:37.419 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.419 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.419 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:135520 7.c 7:35037f76:::benchmark_data_smithi045_115842_object17360:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e647)", 2024-08-19T14:37:37.419 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:35:47.323563+0000", 2024-08-19T14:37:37.419 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 109.407705556, 2024-08-19T14:37:37.419 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.8331218590000002, 2024-08-19T14:37:37.419 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:37:37.420 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:37:37.420 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:37:37.420 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:37:37.420 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:37:37.420 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:37:37.420 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 135520 2024-08-19T14:37:37.420 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.420 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:37:37.420 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.420 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:37:37.420 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:47.323563+0000", 2024-08-19T14:37:37.420 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:37:37.420 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.420 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.421 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:37:37.421 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:47.323563+0000", 2024-08-19T14:37:37.421 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:37:37.421 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.421 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.421 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:37:37.421 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:47.323563+0000", 2024-08-19T14:37:37.421 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.8500000000000002e-07 2024-08-19T14:37:37.421 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.421 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.421 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:37:37.421 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:47.323566+0000", 2024-08-19T14:37:37.421 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.7329999999999999e-06 2024-08-19T14:37:37.421 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.422 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.422 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:37:37.422 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:47.323573+0000", 2024-08-19T14:37:37.422 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.9829999999999999e-06 2024-08-19T14:37:37.422 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.422 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.422 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:37:37.422 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:47.323579+0000", 2024-08-19T14:37:37.422 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.4999999999999999e-06 2024-08-19T14:37:37.422 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.422 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.422 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:37:37.423 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:47.323888+0000", 2024-08-19T14:37:37.423 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00030948899999999998 2024-08-19T14:37:37.423 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.423 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.423 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for missing object", 2024-08-19T14:37:37.423 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:47.323969+0000", 2024-08-19T14:37:37.423 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.0506000000000005e-05 2024-08-19T14:37:37.423 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.423 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.423 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:37:37.423 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:51.156672+0000", 2024-08-19T14:37:37.423 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.8327035739999999 2024-08-19T14:37:37.423 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.424 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.424 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:37:37.424 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:51.156685+0000", 2024-08-19T14:37:37.424 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2689e-05 2024-08-19T14:37:37.424 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:37:37.424 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:37:37.424 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:37:37.424 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.424 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.424 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:135524 7.c 7:398fc3f7:::benchmark_data_smithi045_115842_object17364:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e647)", 2024-08-19T14:37:37.424 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:35:47.323615+0000", 2024-08-19T14:37:37.424 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 109.407653716, 2024-08-19T14:37:37.424 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.9784070759999999, 2024-08-19T14:37:37.424 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:37:37.425 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:37:37.425 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:37:37.425 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:37:37.425 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:37:37.425 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:37:37.425 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 135524 2024-08-19T14:37:37.425 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.425 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:37:37.425 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.425 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:37:37.425 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:47.323615+0000", 2024-08-19T14:37:37.425 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:37:37.425 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.425 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.425 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:37:37.426 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:47.323615+0000", 2024-08-19T14:37:37.426 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:37:37.426 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.426 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.426 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:37:37.426 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:47.323615+0000", 2024-08-19T14:37:37.426 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.1199999999999998e-07 2024-08-19T14:37:37.426 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.426 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.426 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:37:37.426 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:47.323618+0000", 2024-08-19T14:37:37.426 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.6730000000000001e-06 2024-08-19T14:37:37.427 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.427 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.427 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:37:37.427 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:47.323626+0000", 2024-08-19T14:37:37.427 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.0029999999999994e-06 2024-08-19T14:37:37.427 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.427 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.427 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:37:37.427 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:47.323632+0000", 2024-08-19T14:37:37.427 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.8799999999999996e-06 2024-08-19T14:37:37.427 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.427 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.427 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:37:37.427 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:47.323996+0000", 2024-08-19T14:37:37.428 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00036417099999999999 2024-08-19T14:37:37.428 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.428 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.428 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for missing object", 2024-08-19T14:37:37.428 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:47.324073+0000", 2024-08-19T14:37:37.428 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.7356999999999998e-05 2024-08-19T14:37:37.428 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.428 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.428 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:37:37.428 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:51.301999+0000", 2024-08-19T14:37:37.428 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.977925334 2024-08-19T14:37:37.428 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.428 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.429 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:37:37.429 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:51.302022+0000", 2024-08-19T14:37:37.429 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.3246000000000001e-05 2024-08-19T14:37:37.429 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:37:37.429 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:37:37.429 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:37:37.429 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.429 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.429 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:135528 7.3 7:cb1ca295:::benchmark_data_smithi045_115842_object17368:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e647)", 2024-08-19T14:37:37.429 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:35:47.323640+0000", 2024-08-19T14:37:37.429 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 109.407628192, 2024-08-19T14:37:37.429 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7852496709999999, 2024-08-19T14:37:37.429 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:37:37.430 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:37:37.430 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:37:37.430 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:37:37.430 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:37:37.430 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:37:37.430 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 135528 2024-08-19T14:37:37.430 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.430 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:37:37.430 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.430 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:37:37.430 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:47.323640+0000", 2024-08-19T14:37:37.430 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:37:37.430 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.431 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.431 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:37:37.431 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:47.323640+0000", 2024-08-19T14:37:37.431 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:37:37.431 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.431 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.431 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:37:37.431 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:47.323641+0000", 2024-08-19T14:37:37.431 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.15e-07 2024-08-19T14:37:37.431 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.431 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.431 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:37:37.431 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:47.323643+0000", 2024-08-19T14:37:37.432 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.5100000000000001e-06 2024-08-19T14:37:37.432 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.432 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.432 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:37:37.432 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:47.323651+0000", 2024-08-19T14:37:37.432 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.204e-06 2024-08-19T14:37:37.432 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.432 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.432 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:37:37.432 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:47.323656+0000", 2024-08-19T14:37:37.432 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0540000000000002e-06 2024-08-19T14:37:37.432 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.432 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.432 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:37:37.432 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:47.323930+0000", 2024-08-19T14:37:37.433 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00027444900000000001 2024-08-19T14:37:37.433 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.433 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.433 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for missing object", 2024-08-19T14:37:37.433 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:47.324005+0000", 2024-08-19T14:37:37.433 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.4727000000000005e-05 2024-08-19T14:37:37.433 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.433 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.433 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:37:37.433 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:51.108879+0000", 2024-08-19T14:37:37.433 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7848740510000001 2024-08-19T14:37:37.433 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.433 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.433 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:37:37.433 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:51.108890+0000", 2024-08-19T14:37:37.433 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1260999999999999e-05 2024-08-19T14:37:37.434 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:37:37.434 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:37:37.434 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:37:37.434 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.434 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.434 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:137479 7.2 7:43f6534c:::benchmark_data_smithi045_115842_object19319:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e648)", 2024-08-19T14:37:37.434 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:35:52.566358+0000", 2024-08-19T14:37:37.434 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 104.164910713, 2024-08-19T14:37:37.434 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1034698629999999, 2024-08-19T14:37:37.434 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:37:37.434 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:37:37.434 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:37:37.434 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:37:37.434 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:37:37.435 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:37:37.435 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 137479 2024-08-19T14:37:37.435 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.435 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:37:37.435 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.435 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:37:37.435 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:52.566358+0000", 2024-08-19T14:37:37.435 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:37:37.435 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.435 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.435 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:37:37.435 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:52.566358+0000", 2024-08-19T14:37:37.435 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:37:37.435 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.436 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.436 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:37:37.436 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:52.566358+0000", 2024-08-19T14:37:37.436 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.9699999999999996e-07 2024-08-19T14:37:37.436 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.436 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.436 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:37:37.436 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:52.566361+0000", 2024-08-19T14:37:37.436 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.8320000000000002e-06 2024-08-19T14:37:37.436 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.436 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.436 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:37:37.436 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:52.566369+0000", 2024-08-19T14:37:37.437 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.7970000000000001e-06 2024-08-19T14:37:37.437 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.437 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.437 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:37:37.437 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:52.566375+0000", 2024-08-19T14:37:37.437 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.1550000000000004e-06 2024-08-19T14:37:37.437 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.437 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.437 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:37:37.437 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:52.567153+0000", 2024-08-19T14:37:37.437 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00077826700000000002 2024-08-19T14:37:37.437 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.437 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.437 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for missing object", 2024-08-19T14:37:37.437 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:52.567230+0000", 2024-08-19T14:37:37.438 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.6380999999999999e-05 2024-08-19T14:37:37.438 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.438 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.438 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:37:37.438 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:53.669817+0000", 2024-08-19T14:37:37.438 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1025874440000001 2024-08-19T14:37:37.438 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.438 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.438 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:37:37.438 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:53.669828+0000", 2024-08-19T14:37:37.438 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.049e-05 2024-08-19T14:37:37.438 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:37:37.438 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:37:37.438 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:37:37.439 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.439 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.439 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:137495 7.2 7:41c83363:::benchmark_data_smithi045_115842_object19335:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e648)", 2024-08-19T14:37:37.439 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:35:52.566468+0000", 2024-08-19T14:37:37.439 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 104.16480098, 2024-08-19T14:37:37.439 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1151770430000001, 2024-08-19T14:37:37.439 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:37:37.439 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:37:37.439 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:37:37.439 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:37:37.439 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:37:37.439 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:37:37.439 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 137495 2024-08-19T14:37:37.439 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.440 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:37:37.440 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.440 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:37:37.440 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:52.566468+0000", 2024-08-19T14:37:37.440 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:37:37.440 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.440 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.440 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:37:37.440 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:52.566468+0000", 2024-08-19T14:37:37.440 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:37:37.440 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.440 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.440 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:37:37.440 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:52.566468+0000", 2024-08-19T14:37:37.440 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.1199999999999999e-07 2024-08-19T14:37:37.440 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.441 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.441 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:37:37.441 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:52.566476+0000", 2024-08-19T14:37:37.441 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.4989999999999994e-06 2024-08-19T14:37:37.441 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.441 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.441 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:37:37.441 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:52.566482+0000", 2024-08-19T14:37:37.441 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.93e-06 2024-08-19T14:37:37.441 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.441 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.441 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:37:37.441 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:52.566487+0000", 2024-08-19T14:37:37.441 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.5129999999999998e-06 2024-08-19T14:37:37.442 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.442 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.442 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:37:37.442 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:52.567272+0000", 2024-08-19T14:37:37.442 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00078482900000000002 2024-08-19T14:37:37.442 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.442 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.442 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for missing object", 2024-08-19T14:37:37.442 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:52.567347+0000", 2024-08-19T14:37:37.442 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.5535000000000001e-05 2024-08-19T14:37:37.442 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.442 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.442 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:37:37.442 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:53.681634+0000", 2024-08-19T14:37:37.443 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1142871649999999 2024-08-19T14:37:37.443 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.443 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.443 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:37:37.443 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:53.681645+0000", 2024-08-19T14:37:37.443 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.026e-05 2024-08-19T14:37:37.443 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:37:37.443 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:37:37.443 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:37:37.443 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.443 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.443 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:137497 7.2 7:4c095779:::benchmark_data_smithi045_115842_object19337:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e648)", 2024-08-19T14:37:37.443 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:35:52.566494+0000", 2024-08-19T14:37:37.443 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 104.16477446, 2024-08-19T14:37:37.443 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.126120794, 2024-08-19T14:37:37.443 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:37:37.444 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:37:37.444 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:37:37.444 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:37:37.444 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:37:37.444 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:37:37.444 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 137497 2024-08-19T14:37:37.444 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.444 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:37:37.444 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.444 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:37:37.444 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:52.566494+0000", 2024-08-19T14:37:37.444 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:37:37.444 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.444 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.445 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:37:37.445 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:52.566494+0000", 2024-08-19T14:37:37.445 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:37:37.445 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.445 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.445 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:37:37.445 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:52.566494+0000", 2024-08-19T14:37:37.445 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.4900000000000001e-07 2024-08-19T14:37:37.445 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.445 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.445 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:37:37.445 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:52.566496+0000", 2024-08-19T14:37:37.445 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.0159999999999998e-06 2024-08-19T14:37:37.445 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.445 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.445 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:37:37.446 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:52.566501+0000", 2024-08-19T14:37:37.446 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.9579999999999998e-06 2024-08-19T14:37:37.446 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.446 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.446 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:37:37.446 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:52.566507+0000", 2024-08-19T14:37:37.446 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.446e-06 2024-08-19T14:37:37.446 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.446 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.446 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:37:37.446 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:52.567379+0000", 2024-08-19T14:37:37.446 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00087211500000000002 2024-08-19T14:37:37.446 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.446 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.447 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for missing object", 2024-08-19T14:37:37.447 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:52.567450+0000", 2024-08-19T14:37:37.447 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.0617999999999994e-05 2024-08-19T14:37:37.447 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.447 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.447 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:37:37.447 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:53.692578+0000", 2024-08-19T14:37:37.447 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1251279460000001 2024-08-19T14:37:37.447 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.448 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.448 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:37:37.448 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:53.692615+0000", 2024-08-19T14:37:37.448 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7345999999999998e-05 2024-08-19T14:37:37.448 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:37:37.448 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:37:37.448 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:37:37.448 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.448 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.448 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:137499 7.c 7:3eae999e:::benchmark_data_smithi045_115842_object19339:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e648)", 2024-08-19T14:37:37.448 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:35:52.566512+0000", 2024-08-19T14:37:37.448 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 104.16475625699999, 2024-08-19T14:37:37.448 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.110562754, 2024-08-19T14:37:37.448 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:37:37.448 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:37:37.449 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:37:37.449 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:37:37.449 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:37:37.449 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:37:37.449 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 137499 2024-08-19T14:37:37.449 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.449 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:37:37.449 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.449 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:37:37.449 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:52.566512+0000", 2024-08-19T14:37:37.449 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:37:37.449 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.449 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.449 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:37:37.450 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:52.566512+0000", 2024-08-19T14:37:37.450 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:37:37.450 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.450 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.450 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:37:37.450 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:52.566513+0000", 2024-08-19T14:37:37.450 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.8299999999999998e-07 2024-08-19T14:37:37.450 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.450 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.450 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:37:37.450 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:52.566514+0000", 2024-08-19T14:37:37.450 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.773e-06 2024-08-19T14:37:37.450 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.451 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.451 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:37:37.451 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:52.566519+0000", 2024-08-19T14:37:37.451 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.6700000000000002e-06 2024-08-19T14:37:37.451 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.451 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.451 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:37:37.451 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:52.566523+0000", 2024-08-19T14:37:37.451 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.0940000000000001e-06 2024-08-19T14:37:37.451 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.451 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.451 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:37:37.451 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:52.567279+0000", 2024-08-19T14:37:37.452 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00075562999999999997 2024-08-19T14:37:37.452 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.452 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.452 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for missing object", 2024-08-19T14:37:37.452 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:52.567361+0000", 2024-08-19T14:37:37.452 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.1940999999999994e-05 2024-08-19T14:37:37.452 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.452 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.452 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:37:37.452 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:53.677064+0000", 2024-08-19T14:37:37.452 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1097029140000001 2024-08-19T14:37:37.452 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.452 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.452 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:37:37.453 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:53.677075+0000", 2024-08-19T14:37:37.453 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1449e-05 2024-08-19T14:37:37.453 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:37:37.453 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:37:37.453 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:37:37.453 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.453 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.453 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:139379 7.2 7:47071399:::benchmark_data_smithi045_115842_object21219:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e648)", 2024-08-19T14:37:37.453 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:35:57.352771+0000", 2024-08-19T14:37:37.453 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 99.378497171000006, 2024-08-19T14:37:37.453 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.7060800060000001, 2024-08-19T14:37:37.453 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:37:37.453 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:37:37.454 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:37:37.454 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:37:37.454 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:37:37.454 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:37:37.454 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 139379 2024-08-19T14:37:37.454 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.454 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:37:37.454 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.454 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:37:37.454 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:57.352771+0000", 2024-08-19T14:37:37.454 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:37:37.454 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.454 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.455 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:37:37.455 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:57.352771+0000", 2024-08-19T14:37:37.455 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:37:37.455 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.455 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.455 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:37:37.455 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:57.352772+0000", 2024-08-19T14:37:37.455 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.8900000000000001e-07 2024-08-19T14:37:37.455 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.455 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.455 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:37:37.455 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:57.352774+0000", 2024-08-19T14:37:37.455 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.8929999999999999e-06 2024-08-19T14:37:37.456 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.456 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.456 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:37:37.456 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:57.352778+0000", 2024-08-19T14:37:37.456 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.6689999999999997e-06 2024-08-19T14:37:37.456 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.456 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.456 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:37:37.456 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:57.352782+0000", 2024-08-19T14:37:37.456 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.8970000000000001e-06 2024-08-19T14:37:37.456 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.456 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.456 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:37:37.456 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:57.373078+0000", 2024-08-19T14:37:37.456 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.020295853999999999 2024-08-19T14:37:37.457 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.457 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.457 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for missing object", 2024-08-19T14:37:37.457 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:57.373151+0000", 2024-08-19T14:37:37.457 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.2750999999999995e-05 2024-08-19T14:37:37.457 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.457 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.457 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:37:37.457 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:00.058840+0000", 2024-08-19T14:37:37.457 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.685688807 2024-08-19T14:37:37.457 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.457 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.457 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:37:37.457 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:00.058851+0000", 2024-08-19T14:37:37.458 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1846e-05 2024-08-19T14:37:37.458 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:37:37.458 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:37:37.458 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:37:37.458 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.458 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.458 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:139390 7.c 7:3982d552:::benchmark_data_smithi045_115842_object21230:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e648)", 2024-08-19T14:37:37.458 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:35:57.352808+0000", 2024-08-19T14:37:37.458 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 99.378460990999997, 2024-08-19T14:37:37.458 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.7041374769999997, 2024-08-19T14:37:37.458 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:37:37.458 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:37:37.458 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:37:37.458 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:37:37.459 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:37:37.459 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:37:37.459 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 139390 2024-08-19T14:37:37.459 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.459 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:37:37.459 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.459 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:37:37.459 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:57.352808+0000", 2024-08-19T14:37:37.459 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:37:37.459 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.459 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.459 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:37:37.459 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:57.352808+0000", 2024-08-19T14:37:37.459 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:37:37.459 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.460 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.460 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:37:37.460 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:57.352808+0000", 2024-08-19T14:37:37.460 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.8599999999999999e-07 2024-08-19T14:37:37.460 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.460 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.460 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:37:37.460 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:57.352810+0000", 2024-08-19T14:37:37.460 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.9410000000000001e-06 2024-08-19T14:37:37.460 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.460 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.460 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:37:37.460 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:57.352815+0000", 2024-08-19T14:37:37.460 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7720000000000002e-06 2024-08-19T14:37:37.461 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.461 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.461 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:37:37.461 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:57.352818+0000", 2024-08-19T14:37:37.461 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.9009999999999996e-06 2024-08-19T14:37:37.461 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.461 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.461 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:37:37.461 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:57.356159+0000", 2024-08-19T14:37:37.461 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0033409580000000002 2024-08-19T14:37:37.461 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.461 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.461 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for missing object", 2024-08-19T14:37:37.462 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:57.356239+0000", 2024-08-19T14:37:37.462 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.9319000000000004e-05 2024-08-19T14:37:37.462 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.462 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.462 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:37:37.462 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:00.052363+0000", 2024-08-19T14:37:37.462 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.696124121 2024-08-19T14:37:37.462 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.462 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.462 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:37:37.462 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:00.056945+0000", 2024-08-19T14:37:37.462 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0045821789999999996 2024-08-19T14:37:37.462 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:37:37.463 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:37:37.463 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:37:37.463 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.463 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.463 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:174671 7.c 7:366b45d0:::benchmark_data_smithi045_115842_object56511:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e673)", 2024-08-19T14:37:37.463 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:37:27.477093+0000", 2024-08-19T14:37:37.463 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 9.25417588, 2024-08-19T14:37:37.463 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.271195538, 2024-08-19T14:37:37.463 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:37:37.463 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:37:37.463 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:37:37.463 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:37:37.463 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:37:37.463 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:37:37.463 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 174671 2024-08-19T14:37:37.464 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.464 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:37:37.464 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.464 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:37:37.464 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:27.477093+0000", 2024-08-19T14:37:37.464 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:37:37.464 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.464 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.464 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:37:37.464 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:27.477093+0000", 2024-08-19T14:37:37.464 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:37:37.464 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.464 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.464 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:37:37.464 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:27.477093+0000", 2024-08-19T14:37:37.465 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.06e-07 2024-08-19T14:37:37.465 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.465 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.465 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:37:37.465 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:27.477096+0000", 2024-08-19T14:37:37.465 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.4660000000000002e-06 2024-08-19T14:37:37.465 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.465 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.465 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:37:37.465 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:27.477102+0000", 2024-08-19T14:37:37.465 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.0990000000000004e-06 2024-08-19T14:37:37.465 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.465 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.465 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:37:37.466 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:27.477106+0000", 2024-08-19T14:37:37.466 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.639e-06 2024-08-19T14:37:37.466 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.466 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.466 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:37:37.466 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:27.481045+0000", 2024-08-19T14:37:37.466 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0039383450000000002 2024-08-19T14:37:37.466 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.466 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.466 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for missing object", 2024-08-19T14:37:37.466 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:27.481121+0000", 2024-08-19T14:37:37.466 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.5870000000000004e-05 2024-08-19T14:37:37.466 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.467 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.467 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:37:37.467 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:28.748264+0000", 2024-08-19T14:37:37.467 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2671432440000001 2024-08-19T14:37:37.467 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.467 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.467 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:37:37.467 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:28.748288+0000", 2024-08-19T14:37:37.467 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.4369e-05 2024-08-19T14:37:37.467 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:37:37.467 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:37:37.467 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:37:37.467 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.467 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.468 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:174683 7.c 7:3c4ef1e0:::benchmark_data_smithi045_115842_object56523:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e673)", 2024-08-19T14:37:37.468 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:37:27.477189+0000", 2024-08-19T14:37:37.468 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 9.2540794900000005, 2024-08-19T14:37:37.468 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.277993326, 2024-08-19T14:37:37.468 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:37:37.468 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:37:37.468 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:37:37.468 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:37:37.468 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:37:37.468 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:37:37.468 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 174683 2024-08-19T14:37:37.468 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.468 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:37:37.468 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.468 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:37:37.469 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:27.477189+0000", 2024-08-19T14:37:37.469 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:37:37.469 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.469 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.469 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:37:37.469 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:27.477189+0000", 2024-08-19T14:37:37.469 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:37:37.469 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.469 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.469 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:37:37.469 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:27.477189+0000", 2024-08-19T14:37:37.469 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.0699999999999998e-07 2024-08-19T14:37:37.469 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.470 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.470 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:37:37.470 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:27.477192+0000", 2024-08-19T14:37:37.470 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.4530000000000001e-06 2024-08-19T14:37:37.470 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.470 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.470 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:37:37.470 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:27.477197+0000", 2024-08-19T14:37:37.470 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0370000000000002e-06 2024-08-19T14:37:37.470 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.470 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.470 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:37:37.470 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:27.477202+0000", 2024-08-19T14:37:37.471 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.9139999999999999e-06 2024-08-19T14:37:37.471 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.471 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.471 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:37:37.471 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:27.481150+0000", 2024-08-19T14:37:37.471 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0039486399999999998 2024-08-19T14:37:37.471 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.471 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.471 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for missing object", 2024-08-19T14:37:37.471 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:27.481220+0000", 2024-08-19T14:37:37.471 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.0010000000000002e-05 2024-08-19T14:37:37.471 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.471 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.472 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:37:37.472 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:28.755166+0000", 2024-08-19T14:37:37.472 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2739458350000001 2024-08-19T14:37:37.472 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.472 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.472 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:37:37.472 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:28.755182+0000", 2024-08-19T14:37:37.472 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.613e-05 2024-08-19T14:37:37.472 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:37:37.472 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:37:37.472 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:37:37.472 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.472 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.472 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:174688 7.c 7:39ff9ef2:::benchmark_data_smithi045_115842_object56528:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e673)", 2024-08-19T14:37:37.473 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:37:27.477207+0000", 2024-08-19T14:37:37.473 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 9.2540613589999996, 2024-08-19T14:37:37.473 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2846986600000001, 2024-08-19T14:37:37.473 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:37:37.473 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:37:37.473 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:37:37.473 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:37:37.473 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:37:37.473 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:37:37.473 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 174688 2024-08-19T14:37:37.473 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.473 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:37:37.473 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.474 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:37:37.474 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:27.477207+0000", 2024-08-19T14:37:37.474 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:37:37.474 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.474 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.474 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:37:37.474 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:27.477207+0000", 2024-08-19T14:37:37.474 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:37:37.474 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.474 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.474 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:37:37.474 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:27.477208+0000", 2024-08-19T14:37:37.474 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.6800000000000001e-07 2024-08-19T14:37:37.474 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.474 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.475 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:37:37.475 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:27.477217+0000", 2024-08-19T14:37:37.475 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.4620000000000002e-06 2024-08-19T14:37:37.475 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.475 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.475 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:37:37.475 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:27.477222+0000", 2024-08-19T14:37:37.475 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.3399999999999997e-06 2024-08-19T14:37:37.475 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.475 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.475 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:37:37.475 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:27.477227+0000", 2024-08-19T14:37:37.475 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4290000000000003e-06 2024-08-19T14:37:37.475 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.476 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.476 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:37:37.476 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:27.481248+0000", 2024-08-19T14:37:37.476 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0040209879999999996 2024-08-19T14:37:37.476 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.476 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.476 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for missing object", 2024-08-19T14:37:37.476 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:27.481316+0000", 2024-08-19T14:37:37.476 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.8332999999999998e-05 2024-08-19T14:37:37.476 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.476 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.476 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:37:37.476 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:28.761893+0000", 2024-08-19T14:37:37.476 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2805769979999999 2024-08-19T14:37:37.476 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:37:37.477 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:37:37.477 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:37:37.477 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:28.761906+0000", 2024-08-19T14:37:37.477 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2741999999999999e-05 2024-08-19T14:37:37.477 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:37:37.477 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:37:37.477 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:37:37.477 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:37:37.477 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:37:37.477 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:37:37.477 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:37:37.500 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:37.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:37.501 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:37:37.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:37 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:37:37.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:37 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:37:37.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:37 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2814523430' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-08-19T14:37:37.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:37 smithi032 ceph-mon[20004]: osdmap e677: 12 total, 12 up, 10 in 2024-08-19T14:37:37.601 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:37:37.625 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:37.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:37.627 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:37:37.727 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:37:37.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:37.772 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:37.772 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:37:37.873 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:37:37.922 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:37.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:37.923 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:37:38.024 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:37:38.048 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:38.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:38.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:37:38.150 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:37:38.171 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:38.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:38.172 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:37:38.273 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:37:38.296 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:38.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:38 smithi045 ceph-mon[20143]: pgmap v1608: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 3.6 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 599 op/s; 21389/175317 objects degraded (12.200%); 6810/175317 objects misplaced (3.884%); 10 MiB/s, 166 objects/s recovering 2024-08-19T14:37:38.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:38 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:37:38.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:38 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:37:38.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:38 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 21389/175317 objects degraded (12.200%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-19T14:37:38.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:38.297 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:37:38.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:38 smithi031 ceph-mon[16336]: pgmap v1608: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 3.6 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 599 op/s; 21389/175317 objects degraded (12.200%); 6810/175317 objects misplaced (3.884%); 10 MiB/s, 166 objects/s recovering 2024-08-19T14:37:38.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:38 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:37:38.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:38 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:37:38.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:38 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 21389/175317 objects degraded (12.200%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-19T14:37:38.398 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:37:38.420 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:38.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:38.421 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:37:38.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:38 smithi032 ceph-mon[20004]: pgmap v1608: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 3.6 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 599 op/s; 21389/175317 objects degraded (12.200%); 6810/175317 objects misplaced (3.884%); 10 MiB/s, 166 objects/s recovering 2024-08-19T14:37:38.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:38 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:37:38.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:38 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:37:38.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:38 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 21389/175317 objects degraded (12.200%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-19T14:37:38.522 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:37:38.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:38.577 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:38.578 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:37:38.679 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:37:38.700 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:38.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:38.701 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:37:38.802 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:37:38.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:38.829 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:38.830 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:37:38.931 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:37:38.954 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:38.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:38.955 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:37:39.056 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:37:39.103 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:39.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:39.105 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:37:39.205 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:37:39.252 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:39.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:39.253 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:37:39.354 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:37:39.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:39 smithi031 ceph-mon[16336]: osdmap e678: 12 total, 12 up, 10 in 2024-08-19T14:37:39.377 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:39.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:39.378 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:37:39.479 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:37:39.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:39 smithi032 ceph-mon[20004]: osdmap e678: 12 total, 12 up, 10 in 2024-08-19T14:37:39.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:39.528 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:39.528 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:37:39.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:39 smithi045 ceph-mon[20143]: osdmap e678: 12 total, 12 up, 10 in 2024-08-19T14:37:39.629 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:37:39.650 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:39.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:39.651 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:37:39.753 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:37:39.803 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:39.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:39.805 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:37:39.906 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:37:39.928 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:39.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:39.929 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:37:40.030 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:37:40.069 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:40.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:40.070 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:37:40.171 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:37:40.193 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:40.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:40.195 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:37:40.295 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:37:40.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:40.349 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:40.349 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:37:40.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:40 smithi031 ceph-mon[16336]: pgmap v1610: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 3.6 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 373 op/s; 21389/174480 objects degraded (12.259%); 6664/174480 objects misplaced (3.819%); 7.0 MiB/s, 113 objects/s recovering 2024-08-19T14:37:40.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:40 smithi031 ceph-mon[16336]: 1.17 scrub starts 2024-08-19T14:37:40.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:40 smithi031 ceph-mon[16336]: 1.17 scrub ok 2024-08-19T14:37:40.450 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:37:40.473 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:40.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:40.474 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:37:40.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:40 smithi032 ceph-mon[20004]: pgmap v1610: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 3.6 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 373 op/s; 21389/174480 objects degraded (12.259%); 6664/174480 objects misplaced (3.819%); 7.0 MiB/s, 113 objects/s recovering 2024-08-19T14:37:40.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:40 smithi032 ceph-mon[20004]: 1.17 scrub starts 2024-08-19T14:37:40.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:40 smithi032 ceph-mon[20004]: 1.17 scrub ok 2024-08-19T14:37:40.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:40 smithi045 ceph-mon[20143]: pgmap v1610: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 3.6 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 373 op/s; 21389/174480 objects degraded (12.259%); 6664/174480 objects misplaced (3.819%); 7.0 MiB/s, 113 objects/s recovering 2024-08-19T14:37:40.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:40 smithi045 ceph-mon[20143]: 1.17 scrub starts 2024-08-19T14:37:40.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:40 smithi045 ceph-mon[20143]: 1.17 scrub ok 2024-08-19T14:37:40.575 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:37:40.599 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:40.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:40.600 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:37:40.701 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:37:40.722 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:40.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:40.751 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:37:40.852 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:37:40.903 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:40.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:40.904 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:37:41.005 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:37:41.051 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:41.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:41.052 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:37:41.153 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:37:41.174 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:41.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:41.175 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:37:41.276 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:37:41.297 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:41.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:41.298 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:37:41.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:41 smithi031 ceph-mon[16336]: 7.f scrub starts 2024-08-19T14:37:41.398 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:37:41.421 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:41.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:41.422 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:37:41.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:41 smithi032 ceph-mon[20004]: 7.f scrub starts 2024-08-19T14:37:41.523 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:37:41.544 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:41.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:41.545 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:37:41.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:41 smithi045 ceph-mon[20143]: 7.f scrub starts 2024-08-19T14:37:41.645 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:37:41.668 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:41.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:41.669 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:37:41.770 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:37:41.825 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:41.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:41.826 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:37:41.927 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:37:41.948 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:41.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:41.950 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:37:42.050 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:37:42.098 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:42.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:42.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:37:42.200 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:37:42.239 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:42.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:42.241 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:37:42.341 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:37:42.365 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:42.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:42.366 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:37:42.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:42 smithi031 ceph-mon[16336]: pgmap v1611: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 3.5 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 362 op/s; 21072/172476 objects degraded (12.217%); 6664/172476 objects misplaced (3.864%); 7.8 MiB/s, 125 objects/s recovering 2024-08-19T14:37:42.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:42 smithi031 ceph-mon[16336]: 1.19 scrub starts 2024-08-19T14:37:42.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:42 smithi031 ceph-mon[16336]: 1.19 scrub ok 2024-08-19T14:37:42.466 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:37:42.491 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:42.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:42.492 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:37:42.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:42 smithi032 ceph-mon[20004]: pgmap v1611: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 3.5 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 362 op/s; 21072/172476 objects degraded (12.217%); 6664/172476 objects misplaced (3.864%); 7.8 MiB/s, 125 objects/s recovering 2024-08-19T14:37:42.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:42 smithi032 ceph-mon[20004]: 1.19 scrub starts 2024-08-19T14:37:42.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:42 smithi032 ceph-mon[20004]: 1.19 scrub ok 2024-08-19T14:37:42.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:42 smithi045 ceph-mon[20143]: pgmap v1611: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 3.5 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 362 op/s; 21072/172476 objects degraded (12.217%); 6664/172476 objects misplaced (3.864%); 7.8 MiB/s, 125 objects/s recovering 2024-08-19T14:37:42.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:42 smithi045 ceph-mon[20143]: 1.19 scrub starts 2024-08-19T14:37:42.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:42 smithi045 ceph-mon[20143]: 1.19 scrub ok 2024-08-19T14:37:42.592 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:37:42.639 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:42.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:42.640 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:37:42.741 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:37:42.766 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:42.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:42.767 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:37:42.868 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:37:42.912 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:42.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:42.913 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:37:43.014 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:37:43.037 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:43.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:43.038 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:37:43.139 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:37:43.168 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:43.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:43.169 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:37:43.270 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:37:43.291 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:43.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:43.293 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:37:43.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:43 smithi031 ceph-mon[16336]: 7.f scrub ok 2024-08-19T14:37:43.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:43 smithi031 ceph-mon[16336]: 7.7 scrub starts 2024-08-19T14:37:43.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:43 smithi031 ceph-mon[16336]: 1.1 scrub starts 2024-08-19T14:37:43.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:43 smithi031 ceph-mon[16336]: 1.1 scrub ok 2024-08-19T14:37:43.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:43 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 21072/172476 objects degraded (12.217%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-19T14:37:43.394 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:37:43.431 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:43.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:43.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:37:43.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:43 smithi032 ceph-mon[20004]: 7.f scrub ok 2024-08-19T14:37:43.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:43 smithi032 ceph-mon[20004]: 7.7 scrub starts 2024-08-19T14:37:43.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:43 smithi032 ceph-mon[20004]: 1.1 scrub starts 2024-08-19T14:37:43.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:43 smithi032 ceph-mon[20004]: 1.1 scrub ok 2024-08-19T14:37:43.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:43 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 21072/172476 objects degraded (12.217%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-19T14:37:43.533 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:37:43.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:43 smithi045 ceph-mon[20143]: 7.f scrub ok 2024-08-19T14:37:43.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:43 smithi045 ceph-mon[20143]: 7.7 scrub starts 2024-08-19T14:37:43.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:43 smithi045 ceph-mon[20143]: 1.1 scrub starts 2024-08-19T14:37:43.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:43 smithi045 ceph-mon[20143]: 1.1 scrub ok 2024-08-19T14:37:43.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:43 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 21072/172476 objects degraded (12.217%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-19T14:37:43.555 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:43.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:43.556 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:37:43.657 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:37:43.679 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:43.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:43.680 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:37:43.781 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:37:43.802 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:43.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:43.803 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:37:43.905 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:37:43.926 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:43.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:43.928 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:37:44.028 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:37:44.050 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:44.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:44.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:37:44.152 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:37:44.178 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:44.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:44.179 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:37:44.280 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:37:44.302 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:44.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:44.303 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:37:44.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:44 smithi031 ceph-mon[16336]: pgmap v1612: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 3.4 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 513 op/s; 20711/167883 objects degraded (12.337%); 6407/167883 objects misplaced (3.816%); 11 MiB/s, 178 objects/s recovering 2024-08-19T14:37:44.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:44 smithi031 ceph-mon[16336]: 7.7 scrub ok 2024-08-19T14:37:44.404 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:37:44.426 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:44.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:44.427 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:37:44.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:44 smithi032 ceph-mon[20004]: pgmap v1612: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 3.4 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 513 op/s; 20711/167883 objects degraded (12.337%); 6407/167883 objects misplaced (3.816%); 11 MiB/s, 178 objects/s recovering 2024-08-19T14:37:44.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:44 smithi032 ceph-mon[20004]: 7.7 scrub ok 2024-08-19T14:37:44.528 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:37:44.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:44 smithi045 ceph-mon[20143]: pgmap v1612: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 3.4 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 513 op/s; 20711/167883 objects degraded (12.337%); 6407/167883 objects misplaced (3.816%); 11 MiB/s, 178 objects/s recovering 2024-08-19T14:37:44.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:44 smithi045 ceph-mon[20143]: 7.7 scrub ok 2024-08-19T14:37:44.578 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:44.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:44.579 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:37:44.680 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:37:44.701 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:44.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:44.702 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:37:44.803 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:37:44.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:44.830 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:44.830 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:37:44.931 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:37:44.952 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:44.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:44.953 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:37:45.054 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:37:45.077 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:45.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:45.078 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:37:45.179 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:37:45.202 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:45.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:45.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:37:45.304 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:37:45.354 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:45.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:45.356 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:37:45.456 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:37:45.479 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:45.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:45.480 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:37:45.581 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:37:45.606 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:45.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:45.607 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:37:45.708 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:37:45.729 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:45.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:45.730 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:37:45.831 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:37:45.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:45.856 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:45.856 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:37:45.957 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:37:45.979 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:45.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:45.980 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:37:46.082 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:37:46.104 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:46.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:46.105 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:37:46.138 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd reweight 9 1 2024-08-19T14:37:46.206 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:37:46.227 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:46.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:46.228 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:37:46.329 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:37:46.352 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:46.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:46.353 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:37:46.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:46 smithi031 ceph-mon[16336]: pgmap v1613: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 3.4 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 319 op/s; 20711/167463 objects degraded (12.368%); 6407/167463 objects misplaced (3.826%); 7.6 MiB/s, 122 objects/s recovering 2024-08-19T14:37:46.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:46 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:37:46.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:46 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:37:46.453 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:37:46.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:46.476 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:46.476 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:37:46.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:46 smithi032 ceph-mon[20004]: pgmap v1613: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 3.4 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 319 op/s; 20711/167463 objects degraded (12.368%); 6407/167463 objects misplaced (3.826%); 7.6 MiB/s, 122 objects/s recovering 2024-08-19T14:37:46.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:46 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:37:46.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:46 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:37:46.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:46 smithi045 ceph-mon[20143]: pgmap v1613: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 3.4 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 319 op/s; 20711/167463 objects degraded (12.368%); 6407/167463 objects misplaced (3.826%); 7.6 MiB/s, 122 objects/s recovering 2024-08-19T14:37:46.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:46 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:37:46.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:46 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:37:46.577 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:37:46.599 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:46.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:46.600 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:37:46.667 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-08-19T14:37:46.701 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:37:46.734 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:46.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:46.735 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:37:46.836 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:37:46.861 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:46.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:46.862 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:37:46.959 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:37:46.963 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:37:47.001 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:47.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:47.002 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:37:47.102 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:37:47.125 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:47.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:47.125 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:37:47.146 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:37:47.226 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:37:47.230 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:37:47.295 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:47.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:47.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:37:47.319 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:47 smithi031 ceph-mon[16336]: 1.6 scrub starts 2024-08-19T14:37:47.319 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:47 smithi031 ceph-mon[16336]: 1.6 scrub ok 2024-08-19T14:37:47.319 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:47 smithi031 ceph-mon[16336]: osdmap e679: 12 total, 12 up, 10 in 2024-08-19T14:37:47.368 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:47 smithi032 ceph-mon[20004]: 1.6 scrub starts 2024-08-19T14:37:47.368 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:47 smithi032 ceph-mon[20004]: 1.6 scrub ok 2024-08-19T14:37:47.368 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:47 smithi032 ceph-mon[20004]: osdmap e679: 12 total, 12 up, 10 in 2024-08-19T14:37:47.400 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:37:47.421 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:47.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:47.422 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:37:47.486 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:37:47.487 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-08-19T14:37:47.487 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:37:47.487 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.16962.0:183604 7.1 7:8e07c7ab:::benchmark_data_smithi045_115842_object65444:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e679)", 2024-08-19T14:37:47.487 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:37:47.478008+0000", 2024-08-19T14:37:47.487 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.0074096429999999996, 2024-08-19T14:37:47.487 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0074300800000000004, 2024-08-19T14:37:47.487 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:37:47.487 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:37:47.487 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "waiting for sub ops", 2024-08-19T14:37:47.487 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:37:47.487 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.16962", 2024-08-19T14:37:47.487 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:37:47.488 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 183604 2024-08-19T14:37:47.488 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:37:47.488 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:37:47.488 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:37:47.488 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:37:47.488 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:37:47.478008+0000", 2024-08-19T14:37:47.488 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:37:47.488 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:37:47.488 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:37:47.488 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:37:47.488 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:37:47.478008+0000", 2024-08-19T14:37:47.488 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:37:47.489 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:37:47.489 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:37:47.489 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:37:47.489 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:37:47.478010+0000", 2024-08-19T14:37:47.489 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.68e-06 2024-08-19T14:37:47.489 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:37:47.489 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:37:47.489 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:37:47.489 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:37:47.478015+0000", 2024-08-19T14:37:47.489 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7610000000000004e-06 2024-08-19T14:37:47.489 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:37:47.489 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:37:47.489 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:37:47.489 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:37:47.478030+0000", 2024-08-19T14:37:47.490 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4253000000000001e-05 2024-08-19T14:37:47.490 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:37:47.490 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:37:47.490 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:37:47.490 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:37:47.478036+0000", 2024-08-19T14:37:47.490 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9489999999999999e-06 2024-08-19T14:37:47.490 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:37:47.490 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:37:47.490 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:37:47.490 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:37:47.478097+0000", 2024-08-19T14:37:47.490 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0080000000000001e-05 2024-08-19T14:37:47.490 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:37:47.490 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:37:47.491 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:37:47.491 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:37:47.478246+0000", 2024-08-19T14:37:47.491 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014936800000000001 2024-08-19T14:37:47.491 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:37:47.491 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:37:47.491 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 1,3", 2024-08-19T14:37:47.491 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:37:47.478392+0000", 2024-08-19T14:37:47.491 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000146322 2024-08-19T14:37:47.491 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:37:47.491 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:37:47.491 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:37:47.491 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:37:47.483492+0000", 2024-08-19T14:37:47.491 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.005099325 2024-08-19T14:37:47.491 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:37:47.491 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:37:47.492 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:37:47.492 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:37:47.485298+0000", 2024-08-19T14:37:47.492 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001805975 2024-08-19T14:37:47.492 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:37:47.492 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:37:47.492 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:37:47.492 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:37:47.492 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:37:47.492 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.16962.0:183603 7.4 e679/600 7:2908233e:::benchmark_data_smithi045_115842_object65443:head v 679'11629, mlcod=679'11626)", 2024-08-19T14:37:47.492 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:37:47.483208+0000", 2024-08-19T14:37:47.492 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.0022098119999999998, 2024-08-19T14:37:47.492 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0022812459999999998, 2024-08-19T14:37:47.492 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:37:47.492 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:37:47.493 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-08-19T14:37:47.493 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:37:47.493 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:37:47.493 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:37:47.493 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:37:47.483208+0000", 2024-08-19T14:37:47.493 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:37:47.493 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:37:47.493 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:37:47.493 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:37:47.493 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:37:47.483208+0000", 2024-08-19T14:37:47.493 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:37:47.493 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:37:47.493 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:37:47.494 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:37:47.494 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:37:47.483208+0000", 2024-08-19T14:37:47.494 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2200000000000004e-07 2024-08-19T14:37:47.494 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:37:47.494 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:37:47.494 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:37:47.494 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:37:47.483213+0000", 2024-08-19T14:37:47.494 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5759999999999999e-06 2024-08-19T14:37:47.494 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:37:47.494 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:37:47.494 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:37:47.494 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:37:47.483222+0000", 2024-08-19T14:37:47.494 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.7679999999999997e-06 2024-08-19T14:37:47.495 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:37:47.495 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:37:47.495 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:37:47.495 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:37:47.483227+0000", 2024-08-19T14:37:47.495 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5219999999999998e-06 2024-08-19T14:37:47.495 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:37:47.495 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:37:47.495 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:37:47.495 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:37:47.483268+0000", 2024-08-19T14:37:47.495 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0951000000000001e-05 2024-08-19T14:37:47.495 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:37:47.495 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:37:47.495 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:37:47.496 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:37:47.483281+0000", 2024-08-19T14:37:47.496 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3237e-05 2024-08-19T14:37:47.496 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:37:47.496 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:37:47.496 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:37:47.496 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:37:47.496 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-08-19T14:37:47.496 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 2 2024-08-19T14:37:47.496 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:37:47.523 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:37:47.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:47 smithi045 ceph-mon[20143]: 1.6 scrub starts 2024-08-19T14:37:47.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:47 smithi045 ceph-mon[20143]: 1.6 scrub ok 2024-08-19T14:37:47.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:47 smithi045 ceph-mon[20143]: osdmap e679: 12 total, 12 up, 10 in 2024-08-19T14:37:47.594 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:47.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:47.595 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:37:47.696 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:37:47.746 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:47.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:47.748 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:37:47.848 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:37:47.891 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:47.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:47.892 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:37:47.993 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:37:48.042 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:48.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:48.043 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:37:48.111 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:37:48.111 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:37:48.143 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:37:48.160 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:48 smithi031 ceph-mon[16336]: pgmap v1615: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 3.3 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 510 op/s; 20051/161955 objects degraded (12.381%); 6331/161955 objects misplaced (3.909%); 12 MiB/s, 185 objects/s recovering 2024-08-19T14:37:48.160 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:48 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 20051/161955 objects degraded (12.381%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-19T14:37:48.160 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:48 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/926242289' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:37:48.160 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:37:48.160 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:37:48.167 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:48.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:48.169 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:37:48.204 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:37:48.204 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:37:48.258 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:37:48.258 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:37:48.270 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:37:48.297 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:37:48.298 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:37:48.326 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:48.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:48.327 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:37:48.334 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:37:48.334 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:37:48.378 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:37:48.378 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:37:48.415 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:37:48.416 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:37:48.428 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:37:48.454 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:37:48.454 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:37:48.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:48.464 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:48.464 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:37:48.474 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:48 smithi032 ceph-mon[20004]: pgmap v1615: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 3.3 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 510 op/s; 20051/161955 objects degraded (12.381%); 6331/161955 objects misplaced (3.909%); 12 MiB/s, 185 objects/s recovering 2024-08-19T14:37:48.474 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:48 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 20051/161955 objects degraded (12.381%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-19T14:37:48.474 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:48 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/926242289' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:37:48.484 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:37:48.484 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:37:48.514 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-08-19T14:37:48.519 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:37:48.519 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:37:48.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:48 smithi045 ceph-mon[20143]: pgmap v1615: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 3.3 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 510 op/s; 20051/161955 objects degraded (12.381%); 6331/161955 objects misplaced (3.909%); 12 MiB/s, 185 objects/s recovering 2024-08-19T14:37:48.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:48 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 20051/161955 objects degraded (12.381%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-19T14:37:48.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:48 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/926242289' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:37:48.553 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:37:48.553 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:37:48.565 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:37:48.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:48.609 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:48.609 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:37:48.710 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:37:48.738 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:48.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:48.739 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:37:48.815 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:37:48.840 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:37:48.896 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:48.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:48.898 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:37:48.998 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:37:49.021 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:49.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:49.023 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:37:49.105 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-19T14:37:49.124 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:37:49.147 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:49.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:49.148 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:37:49.250 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:37:49.275 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:49.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:49.276 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:37:49.345 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:37:49.345 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:37:49.377 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:37:49.399 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:49.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:49.401 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:37:49.501 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:37:49.548 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:49.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:49.550 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:37:49.651 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:37:49.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:49.703 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:49.704 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:37:49.804 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:37:49.828 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:49.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:49.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:37:49.930 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:37:49.950 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:37:49.952 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:49.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:49.953 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:37:50.054 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:37:50.069 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-08-19T14:37:50.078 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:50.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:50.079 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:37:50.112 INFO:teuthology.orchestra.run.smithi031.stderr:reweighted osd.9 to 1 (10000) 2024-08-19T14:37:50.180 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:37:50.203 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:50.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:50.204 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:37:50.305 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:37:50.359 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:50.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:50.360 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:37:50.365 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:37:50.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:50 smithi031 ceph-mon[16336]: pgmap v1616: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 3.3 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 442 op/s; 20051/161196 objects degraded (12.439%); 6150/161196 objects misplaced (3.815%); 11 MiB/s, 169 objects/s recovering 2024-08-19T14:37:50.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:50 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:37:50.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:50 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:37:50.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:50 smithi031 ceph-mon[16336]: osdmap e680: 12 total, 12 up, 10 in 2024-08-19T14:37:50.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:50 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/629007539' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-08-19T14:37:50.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:50 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-08-19T14:37:50.461 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:37:50.491 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:50 smithi032 ceph-mon[20004]: pgmap v1616: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 3.3 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 442 op/s; 20051/161196 objects degraded (12.439%); 6150/161196 objects misplaced (3.815%); 11 MiB/s, 169 objects/s recovering 2024-08-19T14:37:50.491 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:50 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:37:50.491 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:50 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:37:50.491 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:50 smithi032 ceph-mon[20004]: osdmap e680: 12 total, 12 up, 10 in 2024-08-19T14:37:50.491 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:50 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/629007539' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-08-19T14:37:50.491 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:50 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-08-19T14:37:50.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:50.513 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:50.514 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:37:50.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:50 smithi045 ceph-mon[20143]: pgmap v1616: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 3.3 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 442 op/s; 20051/161196 objects degraded (12.439%); 6150/161196 objects misplaced (3.815%); 11 MiB/s, 169 objects/s recovering 2024-08-19T14:37:50.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:50 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:37:50.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:50 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:37:50.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:50 smithi045 ceph-mon[20143]: osdmap e680: 12 total, 12 up, 10 in 2024-08-19T14:37:50.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:50 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/629007539' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-08-19T14:37:50.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:50 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-08-19T14:37:50.614 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:37:50.656 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:50.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:50.658 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:37:50.758 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:37:50.794 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:50.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:50.795 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:37:50.896 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:37:50.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:50.921 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:50.922 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:37:51.023 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:37:51.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:51.055 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:51.057 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:37:51.075 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:37:51.075 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:37:51.108 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-08-19T14:37:51.109 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-19T14:37:51.159 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:37:51.180 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:51.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:51.181 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:37:51.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:51 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-08-19T14:37:51.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:51 smithi032 ceph-mon[20004]: osdmap e681: 12 total, 12 up, 10 in 2024-08-19T14:37:51.282 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:37:51.304 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:51.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:51.305 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:37:51.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:51 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-08-19T14:37:51.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:51 smithi031 ceph-mon[16336]: osdmap e681: 12 total, 12 up, 10 in 2024-08-19T14:37:51.406 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:37:51.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:51.428 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:51.428 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:37:51.529 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:37:51.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:51 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-08-19T14:37:51.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:51 smithi045 ceph-mon[20143]: osdmap e681: 12 total, 12 up, 10 in 2024-08-19T14:37:51.552 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:51.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:51.553 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:37:51.608 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:37:51.653 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:37:51.675 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:51.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:51.676 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:37:51.777 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:37:51.824 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:51.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:51.825 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:37:51.863 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:37:51.882 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-08-19T14:37:51.927 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:37:51.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:51.989 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:51.989 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:37:52.091 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:37:52.154 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is unset 2024-08-19T14:37:52.160 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:37:52.182 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:52.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:52.184 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:37:52.285 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:37:52.341 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:52.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:52.342 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:37:52.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:52 smithi031 ceph-mon[16336]: pgmap v1619: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 3.2 GiB data, 18 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 461 op/s; 19772/159153 objects degraded (12.423%); 6150/159153 objects misplaced (3.864%); 12 MiB/s, 186 objects/s recovering 2024-08-19T14:37:52.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:52 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2970864661' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:37:52.431 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:52 smithi032 ceph-mon[20004]: pgmap v1619: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 3.2 GiB data, 18 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 461 op/s; 19772/159153 objects degraded (12.423%); 6150/159153 objects misplaced (3.864%); 12 MiB/s, 186 objects/s recovering 2024-08-19T14:37:52.431 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:52 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2970864661' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:37:52.443 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:37:52.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:52.468 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:52.469 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:37:52.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:52 smithi045 ceph-mon[20143]: pgmap v1619: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 3.2 GiB data, 18 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 461 op/s; 19772/159153 objects degraded (12.423%); 6150/159153 objects misplaced (3.864%); 12 MiB/s, 186 objects/s recovering 2024-08-19T14:37:52.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:52 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2970864661' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:37:52.570 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:37:52.601 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:37:52.602 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-08-19T14:37:52.617 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:52.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:52.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:37:52.696 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:37:52.696 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:37:52.718 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:37:52.743 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:52.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:52.744 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:37:52.844 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:37:52.923 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:52.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:52.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:37:53.025 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:37:53.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:53.050 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:53.050 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:37:53.151 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:37:53.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:53.205 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:53.205 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:37:53.306 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:37:53.333 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:53.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:53.334 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:37:53.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:53 smithi031 ceph-mon[16336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:37:53.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:53 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2970864661' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:37:53.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:53 smithi031 ceph-mon[16336]: osdmap e682: 12 total, 12 up, 10 in 2024-08-19T14:37:53.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:53 smithi031 ceph-mon[16336]: 7.9 scrub starts 2024-08-19T14:37:53.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:53 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 19772/159153 objects degraded (12.423%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-19T14:37:53.435 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:37:53.446 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:53 smithi045 ceph-mon[20143]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:37:53.447 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:53 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2970864661' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:37:53.447 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:53 smithi045 ceph-mon[20143]: osdmap e682: 12 total, 12 up, 10 in 2024-08-19T14:37:53.447 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:53 smithi045 ceph-mon[20143]: 7.9 scrub starts 2024-08-19T14:37:53.447 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:53 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 19772/159153 objects degraded (12.423%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-19T14:37:53.458 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:53.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:53.459 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:37:53.477 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:53 smithi032 ceph-mon[20004]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:37:53.477 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:53 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2970864661' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:37:53.477 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:53 smithi032 ceph-mon[20004]: osdmap e682: 12 total, 12 up, 10 in 2024-08-19T14:37:53.477 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:53 smithi032 ceph-mon[20004]: 7.9 scrub starts 2024-08-19T14:37:53.477 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:53 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 19772/159153 objects degraded (12.423%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-19T14:37:53.507 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-08-19T14:37:53.559 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:37:53.581 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:53.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:53.582 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:37:53.604 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:37:53.608 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":1621,"stamp":"2024-08-19T14:37:52.227874+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3373924920,"num_objects":51476,"num_object_clones":0,"num_object_copies":154428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19447,"num_objects_misplaced":5934,"num_objects_unfound":0,"num_objects_dirty":51476,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":184842,"num_write_kb":7562049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38471,"num_bytes_recovered":2513965600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":164979,"ondisk_log_size":164979,"up":141,"acting":135,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":143,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":18158620,"kb_used_data":9699240,"kb_used_omap":18,"kb_used_meta":8459117,"kb_avail":919292900,"statfs":{"total":959950356480,"available":941355929600,"internally_reserved":0,"allocated":9932021760,"data_stored":9901895830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18760,"internal_metadata":8662136504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,5,4,3,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-164429824,"num_objects":-2509,"num_object_clones":0,"num_object_copies":-7527,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-604,"num_objects_misplaced":-397,"num_objects_unfound":0,"num_objects_dirty":-2509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2509,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":936,"num_bytes_recovered":61276160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002299"},"pg_stats":[{"pgid":"7.6","version":"681'11765","reported_seq":11999,"reported_epoch":681,"state":"active+clean","last_fresh":"2024-08-19T14:37:50.390049+0000","last_change":"2024-08-19T14:36:40.514235+0000","last_active":"2024-08-19T14:37:50.390049+0000","last_peered":"2024-08-19T14:37:50.390049+0000","last_clean":"2024-08-19T14:37:50.390049+0000","last_became_active":"2024-08-19T14:33:28.334317+0000","last_became_peered":"2024-08-19T14:33:28.334317+0000","last_unstale":"2024-08-19T14:37:50.390049+0000","last_undegraded":"2024-08-19T14:37:50.390049+0000","last_fullsized":"2024-08-19T14:37:50.390049+0000","mapping_epoch":600,"log_start":"608'1700","ondisk_log_start":"608'1700","created":600,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"662'10000","last_scrub_stamp":"2024-08-19T14:36:40.514040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:36:40.514040+0000","objects_scrubbed":5074,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:37:53.433562+0000","scrub_duration":1848,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":215547904,"num_objects":3289,"num_object_clones":0,"num_object_copies":9867,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11765,"num_write_kb":481728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"70'139","reported_seq":1672,"reported_epoch":681,"state":"active+clean","last_fresh":"2024-08-19T14:37:50.133764+0000","last_change":"2024-08-19T14:36:33.688668+0000","last_active":"2024-08-19T14:37:50.133764+0000","last_peered":"2024-08-19T14:37:50.133764+0000","last_clean":"2024-08-19T14:37:50.133764+0000","last_became_active":"2024-08-19T14:35:13.950224+0000","last_became_peered":"2024-08-19T14:35:13.950224+0000","last_unstale":"2024-08-19T14:37:50.133764+0000","last_undegraded":"2024-08-19T14:37:50.133764+0000","last_fullsized":"2024-08-19T14:37:50.133764+0000","mapping_epoch":637,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":638,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:36:33.688543+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:19:58.072083+0000","last_clean_scrub_stamp":"2024-08-19T14:36:33.688543+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:37:57.555609+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,9],"acting":[0,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.5","version":"681'11410","reported_seq":20863,"reported_epoch":681,"state":"active+clean","last_fresh":"2024-08-19T14:37:51.531689+0000","last_change":"2024-08-19T14:35:23.022361+0000","last_active":"2024-08-19T14:37:51.531689+0000","last_peered":"2024-08-19T14:37:51.531689+0000","last_clean":"2024-08-19T14:37:51.531689+0000","last_became_active":"2024-08-19T14:35:23.009238+0000","last_became_peered":"2024-08-19T14:35:23.009238+0000","last_unstale":"2024-08-19T14:37:51.531689+0000","last_undegraded":"2024-08-19T14:37:51.531689+0000","last_fullsized":"2024-08-19T14:37:51.531689+0000","mapping_epoch":641,"log_start":"608'1400","ondisk_log_start":"608'1400","created":600,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":214433792,"num_objects":3272,"num_object_clones":0,"num_object_copies":9816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11410,"num_write_kb":469824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4634,"num_bytes_recovered":303693824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,0],"acting":[11,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.3","version":"70'139","reported_seq":1524,"reported_epoch":681,"state":"active+clean","last_fresh":"2024-08-19T14:37:50.111901+0000","last_change":"2024-08-19T14:33:18.804321+0000","last_active":"2024-08-19T14:37:50.111901+0000","last_peered":"2024-08-19T14:37:50.111901+0000","last_clean":"2024-08-19T14:37:50.111901+0000","last_became_active":"2024-08-19T14:18:13.804721+0000","last_became_peered":"2024-08-19T14:18:13.804721+0000","last_unstale":"2024-08-19T14:37:50.111901+0000","last_undegraded":"2024-08-19T14:37:50.111901+0000","last_fullsized":"2024-08-19T14:37:50.111901+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:33:18.804228+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:33:18.804228+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.4","version":"681'11711","reported_seq":11947,"reported_epoch":681,"state":"active+clean","last_fresh":"2024-08-19T14:37:50.434940+0000","last_change":"2024-08-19T14:36:35.067670+0000","last_active":"2024-08-19T14:37:50.434940+0000","last_peered":"2024-08-19T14:37:50.434940+0000","last_clean":"2024-08-19T14:37:50.434940+0000","last_became_active":"2024-08-19T14:33:28.334446+0000","last_became_peered":"2024-08-19T14:33:28.334446+0000","last_unstale":"2024-08-19T14:37:50.434940+0000","last_undegraded":"2024-08-19T14:37:50.434940+0000","last_fullsized":"2024-08-19T14:37:50.434940+0000","mapping_epoch":600,"log_start":"608'1700","ondisk_log_start":"608'1700","created":600,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"660'9797","last_scrub_stamp":"2024-08-19T14:36:35.067331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:36:35.067331+0000","objects_scrubbed":5188,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:38:01.965392+0000","scrub_duration":2448,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":212008960,"num_objects":3235,"num_object_clones":0,"num_object_copies":9705,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3235,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11711,"num_write_kb":478272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"70'139","reported_seq":1481,"reported_epoch":681,"state":"active+clean","last_fresh":"2024-08-19T14:37:50.129437+0000","last_change":"2024-08-19T14:34:55.580212+0000","last_active":"2024-08-19T14:37:50.129437+0000","last_peered":"2024-08-19T14:37:50.129437+0000","last_clean":"2024-08-19T14:37:50.129437+0000","last_became_active":"2024-08-19T14:20:50.742354+0000","last_became_peered":"2024-08-19T14:20:50.742354+0000","last_unstale":"2024-08-19T14:37:50.129437+0000","last_undegraded":"2024-08-19T14:37:50.129437+0000","last_fullsized":"2024-08-19T14:37:50.129437+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:34:55.580141+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:19:59.302931+0000","last_clean_scrub_stamp":"2024-08-19T14:34:55.580141+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"681'11315","reported_seq":11588,"reported_epoch":681,"state":"active+clean","last_fresh":"2024-08-19T14:37:50.274293+0000","last_change":"2024-08-19T14:37:43.229201+0000","last_active":"2024-08-19T14:37:50.274293+0000","last_peered":"2024-08-19T14:37:50.274293+0000","last_clean":"2024-08-19T14:37:50.274293+0000","last_became_active":"2024-08-19T14:33:28.331833+0000","last_became_peered":"2024-08-19T14:33:28.331833+0000","last_unstale":"2024-08-19T14:37:50.274293+0000","last_undegraded":"2024-08-19T14:37:50.274293+0000","last_fullsized":"2024-08-19T14:37:50.274293+0000","mapping_epoch":600,"log_start":"608'1300","ondisk_log_start":"608'1300","created":600,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"678'11114","last_scrub_stamp":"2024-08-19T14:37:43.229072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:37:43.229072+0000","objects_scrubbed":3413,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:39:12.361681+0000","scrub_duration":910,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209911808,"num_objects":3203,"num_object_clones":0,"num_object_copies":9609,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3203,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11315,"num_write_kb":464576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,1],"acting":[8,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1","version":"70'139","reported_seq":779,"reported_epoch":681,"state":"active+clean","last_fresh":"2024-08-19T14:37:50.133968+0000","last_change":"2024-08-19T14:37:42.630047+0000","last_active":"2024-08-19T14:37:50.133968+0000","last_peered":"2024-08-19T14:37:50.133968+0000","last_clean":"2024-08-19T14:37:50.133968+0000","last_became_active":"2024-08-19T14:35:12.530566+0000","last_became_peered":"2024-08-19T14:35:12.530566+0000","last_unstale":"2024-08-19T14:37:50.133968+0000","last_undegraded":"2024-08-19T14:37:50.133968+0000","last_fullsized":"2024-08-19T14:37:50.133968+0000","mapping_epoch":635,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":636,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:37:42.629919+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:37:42.629919+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:39:05.497590+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,2],"acting":[0,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"679'11399","reported_seq":11646,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-08-19T14:37:48.671156+0000","last_change":"2024-08-19T14:34:52.869444+0000","last_active":"2024-08-19T14:37:48.671156+0000","last_peered":"2024-08-19T14:37:48.671156+0000","last_clean":"2024-08-19T14:37:48.671156+0000","last_became_active":"2024-08-19T14:33:28.332994+0000","last_became_peered":"2024-08-19T14:33:28.332994+0000","last_unstale":"2024-08-19T14:37:48.671156+0000","last_undegraded":"2024-08-19T14:37:48.671156+0000","last_fullsized":"2024-08-19T14:37:48.671156+0000","mapping_epoch":600,"log_start":"608'1300","ondisk_log_start":"608'1300","created":600,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"627'6737","last_scrub_stamp":"2024-08-19T14:34:52.869233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:34:52.869233+0000","objects_scrubbed":6705,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1588,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":208338944,"num_objects":3179,"num_object_clones":0,"num_object_copies":9537,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11399,"num_write_kb":466496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"70'139","reported_seq":1181,"reported_epoch":680,"state":"active+clean","last_fresh":"2024-08-19T14:37:49.113266+0000","last_change":"2024-08-19T14:35:11.537368+0000","last_active":"2024-08-19T14:37:49.113266+0000","last_peered":"2024-08-19T14:37:49.113266+0000","last_clean":"2024-08-19T14:37:49.113266+0000","last_became_active":"2024-08-19T14:35:11.536828+0000","last_became_peered":"2024-08-19T14:35:11.536828+0000","last_unstale":"2024-08-19T14:37:49.113266+0000","last_undegraded":"2024-08-19T14:37:49.113266+0000","last_fullsized":"2024-08-19T14:37:49.113266+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:27.536928+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:21:46.057624+0000","last_clean_scrub_stamp":"2024-08-19T14:28:27.536928+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"680'11304","reported_seq":20709,"reported_epoch":680,"state":"active+clean","last_fresh":"2024-08-19T14:37:49.132065+0000","last_change":"2024-08-19T14:36:15.120872+0000","last_active":"2024-08-19T14:37:49.132065+0000","last_peered":"2024-08-19T14:37:49.132065+0000","last_clean":"2024-08-19T14:37:49.132065+0000","last_became_active":"2024-08-19T14:36:15.096070+0000","last_became_peered":"2024-08-19T14:36:15.096070+0000","last_unstale":"2024-08-19T14:37:49.132065+0000","last_undegraded":"2024-08-19T14:37:49.132065+0000","last_fullsized":"2024-08-19T14:37:49.132065+0000","mapping_epoch":653,"log_start":"608'1300","ondisk_log_start":"608'1300","created":600,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213385216,"num_objects":3256,"num_object_clones":0,"num_object_copies":9768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3256,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11304,"num_write_kb":465920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4611,"num_bytes_recovered":302186496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,9],"acting":[3,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"70'139","reported_seq":750,"reported_epoch":681,"state":"active+clean","last_fresh":"2024-08-19T14:37:50.109242+0000","last_change":"2024-08-19T14:37:45.357396+0000","last_active":"2024-08-19T14:37:50.109242+0000","last_peered":"2024-08-19T14:37:50.109242+0000","last_clean":"2024-08-19T14:37:50.109242+0000","last_became_active":"2024-08-19T14:34:15.717054+0000","last_became_peered":"2024-08-19T14:34:15.717054+0000","last_unstale":"2024-08-19T14:37:50.109242+0000","last_undegraded":"2024-08-19T14:37:50.109242+0000","last_fullsized":"2024-08-19T14:37:50.109242+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:37:45.357293+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:27.848870+0000","last_clean_scrub_stamp":"2024-08-19T14:37:45.357293+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:38:48.964771+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"7.3","version":"680'11729","reported_seq":16967,"reported_epoch":680,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-19T14:37:49.273793+0000","last_change":"2024-08-19T14:35:13.136988+0000","last_active":"2024-08-19T14:37:49.273793+0000","last_peered":"2024-08-19T14:37:49.273793+0000","last_clean":"2024-08-19T14:33:27.312789+0000","last_became_active":"2024-08-19T14:35:13.021078+0000","last_became_peered":"2024-08-19T14:35:13.021078+0000","last_unstale":"2024-08-19T14:37:49.273793+0000","last_undegraded":"2024-08-19T14:35:12.592288+0000","last_fullsized":"2024-08-19T14:35:12.526971+0000","mapping_epoch":635,"log_start":"608'1700","ondisk_log_start":"608'1700","created":600,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217907200,"num_objects":3325,"num_object_clones":0,"num_object_copies":9975,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4725,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3325,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11729,"num_write_kb":481728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2533,"num_bytes_recovered":163971072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":-1400}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"70'139","reported_seq":705,"reported_epoch":681,"state":"active+clean","last_fresh":"2024-08-19T14:37:50.114499+0000","last_change":"2024-08-19T14:35:11.533207+0000","last_active":"2024-08-19T14:37:50.114499+0000","last_peered":"2024-08-19T14:37:50.114499+0000","last_clean":"2024-08-19T14:37:50.114499+0000","last_became_active":"2024-08-19T14:35:11.532858+0000","last_became_peered":"2024-08-19T14:35:11.532858+0000","last_unstale":"2024-08-19T14:37:50.114499+0000","last_undegraded":"2024-08-19T14:37:50.114499+0000","last_fullsized":"2024-08-19T14:37:50.114499+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:27.331041+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:28:27.331041+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,0],"acting":[11,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.2","version":"680'11660","reported_seq":16887,"reported_epoch":680,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-19T14:37:49.294623+0000","last_change":"2024-08-19T14:35:13.121204+0000","last_active":"2024-08-19T14:37:49.294623+0000","last_peered":"2024-08-19T14:37:49.294623+0000","last_clean":"2024-08-19T14:33:27.312789+0000","last_became_active":"2024-08-19T14:35:13.013347+0000","last_became_peered":"2024-08-19T14:35:13.013347+0000","last_unstale":"2024-08-19T14:37:49.294623+0000","last_undegraded":"2024-08-19T14:35:12.598259+0000","last_fullsized":"2024-08-19T14:35:12.526602+0000","mapping_epoch":635,"log_start":"608'1600","ondisk_log_start":"608'1600","created":600,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217579520,"num_objects":3320,"num_object_clones":0,"num_object_copies":9960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4708,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11660,"num_write_kb":479360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2527,"num_bytes_recovered":163577856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":-1388}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"70'139","reported_seq":582,"reported_epoch":681,"state":"active+clean","last_fresh":"2024-08-19T14:37:50.114427+0000","last_change":"2024-08-19T14:33:16.760928+0000","last_active":"2024-08-19T14:37:50.114427+0000","last_peered":"2024-08-19T14:37:50.114427+0000","last_clean":"2024-08-19T14:37:50.114427+0000","last_became_active":"2024-08-19T14:30:47.909067+0000","last_became_peered":"2024-08-19T14:30:47.909067+0000","last_unstale":"2024-08-19T14:37:50.114427+0000","last_undegraded":"2024-08-19T14:37:50.114427+0000","last_fullsized":"2024-08-19T14:37:50.114427+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:33:16.760810+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:33:16.760810+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.c","version":"680'11367","reported_seq":16569,"reported_epoch":680,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-19T14:37:49.247670+0000","last_change":"2024-08-19T14:35:13.133729+0000","last_active":"2024-08-19T14:37:49.247670+0000","last_peered":"2024-08-19T14:37:49.247670+0000","last_clean":"2024-08-19T14:33:27.312789+0000","last_became_active":"2024-08-19T14:35:13.003266+0000","last_became_peered":"2024-08-19T14:35:13.003266+0000","last_unstale":"2024-08-19T14:37:49.247670+0000","last_undegraded":"2024-08-19T14:35:12.593502+0000","last_fullsized":"2024-08-19T14:35:12.526375+0000","mapping_epoch":635,"log_start":"608'1300","ondisk_log_start":"608'1300","created":600,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":208994304,"num_objects":3189,"num_object_clones":0,"num_object_copies":9567,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4494,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11367,"num_write_kb":465792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2515,"num_bytes_recovered":163643392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,2],"acting":[3,11],"avail_no_missing":["3","11"],"object_location_counts":[{"shards":"3,11","objects":-1305}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"70'139","reported_seq":743,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-08-19T14:37:46.082779+0000","last_change":"2024-08-19T14:34:50.256878+0000","last_active":"2024-08-19T14:37:46.082779+0000","last_peered":"2024-08-19T14:37:46.082779+0000","last_clean":"2024-08-19T14:37:46.082779+0000","last_became_active":"2024-08-19T14:34:15.717472+0000","last_became_peered":"2024-08-19T14:34:15.717472+0000","last_unstale":"2024-08-19T14:37:46.082779+0000","last_undegraded":"2024-08-19T14:37:46.082779+0000","last_fullsized":"2024-08-19T14:37:46.082779+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:34:50.256808+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:28.864870+0000","last_clean_scrub_stamp":"2024-08-19T14:34:50.256808+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,0],"acting":[4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"681'15076","reported_seq":24855,"reported_epoch":681,"state":"active+clean","last_fresh":"2024-08-19T14:37:50.430962+0000","last_change":"2024-08-19T14:37:42.029576+0000","last_active":"2024-08-19T14:37:50.430962+0000","last_peered":"2024-08-19T14:37:50.430962+0000","last_clean":"2024-08-19T14:37:50.430962+0000","last_became_active":"2024-08-19T14:36:14.098166+0000","last_became_peered":"2024-08-19T14:36:14.098166+0000","last_unstale":"2024-08-19T14:37:50.430962+0000","last_undegraded":"2024-08-19T14:37:50.430962+0000","last_fullsized":"2024-08-19T14:37:50.430962+0000","mapping_epoch":652,"log_start":"621'5002","ondisk_log_start":"621'5002","created":600,"last_epoch_clean":653,"parent":"0.0","parent_split_bits":0,"last_scrub":"678'14831","last_scrub_stamp":"2024-08-19T14:37:42.029449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:37:42.029449+0000","objects_scrubbed":3383,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:38:44.946893+0000","scrub_duration":1404,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":204472320,"num_objects":3120,"num_object_clones":0,"num_object_copies":9360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11446,"num_write_kb":466112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4700,"num_bytes_recovered":308019200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.9","version":"70'139","reported_seq":1683,"reported_epoch":681,"state":"active+clean","last_fresh":"2024-08-19T14:37:50.126223+0000","last_change":"2024-08-19T14:35:12.529993+0000","last_active":"2024-08-19T14:37:50.126223+0000","last_peered":"2024-08-19T14:37:50.126223+0000","last_clean":"2024-08-19T14:37:50.126223+0000","last_became_active":"2024-08-19T14:35:12.529628+0000","last_became_peered":"2024-08-19T14:35:12.529628+0000","last_unstale":"2024-08-19T14:37:50.126223+0000","last_undegraded":"2024-08-19T14:37:50.126223+0000","last_fullsized":"2024-08-19T14:37:50.126223+0000","mapping_epoch":635,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":636,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:34:51.561893+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:34:51.561893+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.e","version":"681'11710","reported_seq":21562,"reported_epoch":681,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-19T14:37:50.870272+0000","last_change":"2024-08-19T14:35:11.863818+0000","last_active":"2024-08-19T14:37:50.870272+0000","last_peered":"2024-08-19T14:37:50.870272+0000","last_clean":"2024-08-19T14:33:27.312789+0000","last_became_active":"2024-08-19T14:35:11.806304+0000","last_became_peered":"2024-08-19T14:35:11.806304+0000","last_unstale":"2024-08-19T14:37:50.870272+0000","last_undegraded":"2024-08-19T14:35:11.779866+0000","last_fullsized":"2024-08-19T14:35:11.528207+0000","mapping_epoch":634,"log_start":"608'1700","ondisk_log_start":"608'1700","created":600,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":202244096,"num_objects":3086,"num_object_clones":0,"num_object_copies":9258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2441,"num_objects_misplaced":4977,"num_objects_unfound":0,"num_objects_dirty":3086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11710,"num_write_kb":473472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3775,"num_bytes_recovered":245432320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":-1891},{"shards":"1,5,9","objects":2536}],"blocked_by":[],"up_primary":2,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"70'139","reported_seq":711,"reported_epoch":680,"state":"active+clean","last_fresh":"2024-08-19T14:37:49.114026+0000","last_change":"2024-08-19T14:35:11.537896+0000","last_active":"2024-08-19T14:37:49.114026+0000","last_peered":"2024-08-19T14:37:49.114026+0000","last_clean":"2024-08-19T14:37:49.114026+0000","last_became_active":"2024-08-19T14:35:11.537474+0000","last_became_peered":"2024-08-19T14:35:11.537474+0000","last_unstale":"2024-08-19T14:37:49.114026+0000","last_undegraded":"2024-08-19T14:37:49.114026+0000","last_fullsized":"2024-08-19T14:37:49.114026+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:59.829567+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:28:59.829567+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"681'11632","reported_seq":11806,"reported_epoch":681,"state":"active+clean","last_fresh":"2024-08-19T14:37:50.849523+0000","last_change":"2024-08-19T14:33:28.335586+0000","last_active":"2024-08-19T14:37:50.849523+0000","last_peered":"2024-08-19T14:37:50.849523+0000","last_clean":"2024-08-19T14:37:50.849523+0000","last_became_active":"2024-08-19T14:33:28.335298+0000","last_became_peered":"2024-08-19T14:33:28.335298+0000","last_unstale":"2024-08-19T14:37:50.849523+0000","last_undegraded":"2024-08-19T14:37:50.849523+0000","last_fullsized":"2024-08-19T14:37:50.849523+0000","mapping_epoch":600,"log_start":"608'1600","ondisk_log_start":"608'1600","created":600,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":202702872,"num_objects":3094,"num_object_clones":0,"num_object_copies":9282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3094,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11632,"num_write_kb":471169,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"70'139","reported_seq":1034,"reported_epoch":681,"state":"active+clean","last_fresh":"2024-08-19T14:37:50.117802+0000","last_change":"2024-08-19T14:33:15.753668+0000","last_active":"2024-08-19T14:37:50.117802+0000","last_peered":"2024-08-19T14:37:50.117802+0000","last_clean":"2024-08-19T14:37:50.117802+0000","last_became_active":"2024-08-19T14:30:47.941200+0000","last_became_peered":"2024-08-19T14:30:47.941200+0000","last_unstale":"2024-08-19T14:37:50.117802+0000","last_undegraded":"2024-08-19T14:37:50.117802+0000","last_fullsized":"2024-08-19T14:37:50.117802+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:33:15.753595+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:33:15.753595+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.8","version":"681'11849","reported_seq":24327,"reported_epoch":681,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-19T14:37:50.413696+0000","last_change":"2024-08-19T14:35:13.081742+0000","last_active":"2024-08-19T14:37:50.413696+0000","last_peered":"2024-08-19T14:37:50.413696+0000","last_clean":"2024-08-19T14:33:27.312789+0000","last_became_active":"2024-08-19T14:35:12.998871+0000","last_became_peered":"2024-08-19T14:35:12.998871+0000","last_unstale":"2024-08-19T14:37:50.413696+0000","last_undegraded":"2024-08-19T14:35:12.561096+0000","last_fullsized":"2024-08-19T14:35:12.526115+0000","mapping_epoch":635,"log_start":"608'1800","ondisk_log_start":"608'1800","created":600,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213712896,"num_objects":3261,"num_object_clones":0,"num_object_copies":9783,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3261,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11849,"num_write_kb":483520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6152,"num_bytes_recovered":403177472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[{"shards":"4,7","objects":2136}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.e","version":"70'139","reported_seq":1220,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-08-19T14:37:46.082514+0000","last_change":"2024-08-19T14:34:55.333177+0000","last_active":"2024-08-19T14:37:46.082514+0000","last_peered":"2024-08-19T14:37:46.082514+0000","last_clean":"2024-08-19T14:37:46.082514+0000","last_became_active":"2024-08-19T14:18:26.884105+0000","last_became_peered":"2024-08-19T14:18:26.884105+0000","last_unstale":"2024-08-19T14:37:46.082514+0000","last_undegraded":"2024-08-19T14:37:46.082514+0000","last_fullsized":"2024-08-19T14:37:46.082514+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:34:55.333065+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:32:31.196288+0000","last_clean_scrub_stamp":"2024-08-19T14:34:55.333065+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"681'11632","reported_seq":11863,"reported_epoch":681,"state":"active+clean","last_fresh":"2024-08-19T14:37:50.457620+0000","last_change":"2024-08-19T14:36:44.335388+0000","last_active":"2024-08-19T14:37:50.457620+0000","last_peered":"2024-08-19T14:37:50.457620+0000","last_clean":"2024-08-19T14:37:50.457620+0000","last_became_active":"2024-08-19T14:33:28.334547+0000","last_became_peered":"2024-08-19T14:33:28.334547+0000","last_unstale":"2024-08-19T14:37:50.457620+0000","last_undegraded":"2024-08-19T14:37:50.457620+0000","last_fullsized":"2024-08-19T14:37:50.457620+0000","mapping_epoch":600,"log_start":"608'1600","ondisk_log_start":"608'1600","created":600,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"664'9921","last_scrub_stamp":"2024-08-19T14:36:44.334182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:36:44.334182+0000","objects_scrubbed":4968,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:37:51.456920+0000","scrub_duration":1563,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":212860928,"num_objects":3248,"num_object_clones":0,"num_object_copies":9744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3248,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11632,"num_write_kb":476160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,8],"acting":[0,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"70'139","reported_seq":972,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-08-19T14:37:46.082685+0000","last_change":"2024-08-19T14:34:12.199530+0000","last_active":"2024-08-19T14:37:46.082685+0000","last_peered":"2024-08-19T14:37:46.082685+0000","last_clean":"2024-08-19T14:37:46.082685+0000","last_became_active":"2024-08-19T14:27:31.837754+0000","last_became_peered":"2024-08-19T14:27:31.837754+0000","last_unstale":"2024-08-19T14:37:46.082685+0000","last_undegraded":"2024-08-19T14:37:46.082685+0000","last_fullsized":"2024-08-19T14:37:46.082685+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:34:12.199449+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:34:12.199449+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.a","version":"681'11535","reported_seq":27609,"reported_epoch":681,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-19T14:37:51.702531+0000","last_change":"2024-08-19T14:35:13.046246+0000","last_active":"2024-08-19T14:37:51.702531+0000","last_peered":"2024-08-19T14:37:51.702531+0000","last_clean":"2024-08-19T14:33:27.312789+0000","last_became_active":"2024-08-19T14:35:13.005199+0000","last_became_peered":"2024-08-19T14:35:13.005199+0000","last_unstale":"2024-08-19T14:37:51.702531+0000","last_undegraded":"2024-08-19T14:35:12.692218+0000","last_fullsized":"2024-08-19T14:35:12.524923+0000","mapping_epoch":635,"log_start":"608'1500","ondisk_log_start":"608'1500","created":600,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209125376,"num_objects":3191,"num_object_clones":0,"num_object_copies":9573,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1954,"num_objects_misplaced":957,"num_objects_unfound":0,"num_objects_dirty":3191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11535,"num_write_kb":471232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7021,"num_bytes_recovered":459407360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,2],"acting":[11,9],"avail_no_missing":["11","3","9"],"object_location_counts":[{"shards":"3,9,11","objects":957},{"shards":"9,11","objects":280}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.c","version":"70'139","reported_seq":1329,"reported_epoch":680,"state":"active+clean","last_fresh":"2024-08-19T14:37:49.134441+0000","last_change":"2024-08-19T14:34:15.738488+0000","last_active":"2024-08-19T14:37:49.134441+0000","last_peered":"2024-08-19T14:37:49.134441+0000","last_clean":"2024-08-19T14:37:49.134441+0000","last_became_active":"2024-08-19T14:34:15.733282+0000","last_became_peered":"2024-08-19T14:34:15.733282+0000","last_unstale":"2024-08-19T14:37:49.134441+0000","last_undegraded":"2024-08-19T14:37:49.134441+0000","last_fullsized":"2024-08-19T14:37:49.134441+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:33:17.453290+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:22:52.164586+0000","last_clean_scrub_stamp":"2024-08-19T14:33:17.453290+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,9],"acting":[3,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"681'11378","reported_seq":11551,"reported_epoch":681,"state":"active+clean","last_fresh":"2024-08-19T14:37:50.884989+0000","last_change":"2024-08-19T14:33:28.623717+0000","last_active":"2024-08-19T14:37:50.884989+0000","last_peered":"2024-08-19T14:37:50.884989+0000","last_clean":"2024-08-19T14:37:50.884989+0000","last_became_active":"2024-08-19T14:33:28.623318+0000","last_became_peered":"2024-08-19T14:33:28.623318+0000","last_unstale":"2024-08-19T14:37:50.884989+0000","last_undegraded":"2024-08-19T14:37:50.884989+0000","last_fullsized":"2024-08-19T14:37:50.884989+0000","mapping_epoch":600,"log_start":"608'1300","ondisk_log_start":"608'1300","created":600,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":210108416,"num_objects":3206,"num_object_clones":0,"num_object_copies":9618,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3206,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11378,"num_write_kb":466688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"70'139","reported_seq":1228,"reported_epoch":681,"state":"active+clean","last_fresh":"2024-08-19T14:37:50.126401+0000","last_change":"2024-08-19T14:34:49.476489+0000","last_active":"2024-08-19T14:37:50.126401+0000","last_peered":"2024-08-19T14:37:50.126401+0000","last_clean":"2024-08-19T14:37:50.126401+0000","last_became_active":"2024-08-19T14:18:26.882002+0000","last_became_peered":"2024-08-19T14:18:26.882002+0000","last_unstale":"2024-08-19T14:37:50.126401+0000","last_undegraded":"2024-08-19T14:37:50.126401+0000","last_fullsized":"2024-08-19T14:37:50.126401+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:34:49.476415+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:24:50.276408+0000","last_clean_scrub_stamp":"2024-08-19T14:34:49.476415+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"70'139","reported_seq":1007,"reported_epoch":681,"state":"active+clean","last_fresh":"2024-08-19T14:37:50.111386+0000","last_change":"2024-08-19T14:33:19.781527+0000","last_active":"2024-08-19T14:37:50.111386+0000","last_peered":"2024-08-19T14:37:50.111386+0000","last_clean":"2024-08-19T14:37:50.111386+0000","last_became_active":"2024-08-19T14:27:31.838968+0000","last_became_peered":"2024-08-19T14:27:31.838968+0000","last_unstale":"2024-08-19T14:37:50.111386+0000","last_undegraded":"2024-08-19T14:37:50.111386+0000","last_fullsized":"2024-08-19T14:37:50.111386+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:33:19.781431+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:28:35.206115+0000","last_clean_scrub_stamp":"2024-08-19T14:33:19.781431+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.13","version":"70'139","reported_seq":1549,"reported_epoch":681,"state":"active+clean","last_fresh":"2024-08-19T14:37:50.111784+0000","last_change":"2024-08-19T14:33:35.786236+0000","last_active":"2024-08-19T14:37:50.111784+0000","last_peered":"2024-08-19T14:37:50.111784+0000","last_clean":"2024-08-19T14:37:50.111784+0000","last_became_active":"2024-08-19T14:30:47.939849+0000","last_became_peered":"2024-08-19T14:30:47.939849+0000","last_unstale":"2024-08-19T14:37:50.111784+0000","last_undegraded":"2024-08-19T14:37:50.111784+0000","last_fullsized":"2024-08-19T14:37:50.111784+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:33:35.786105+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:33:35.786105+0000","last_clean_scrub_stamp":"2024-08-19T14:33:35.786105+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.10","version":"70'139","reported_seq":1177,"reported_epoch":681,"state":"active+clean","last_fresh":"2024-08-19T14:37:50.126175+0000","last_change":"2024-08-19T14:34:58.616722+0000","last_active":"2024-08-19T14:37:50.126175+0000","last_peered":"2024-08-19T14:37:50.126175+0000","last_clean":"2024-08-19T14:37:50.126175+0000","last_became_active":"2024-08-19T14:27:31.839611+0000","last_became_peered":"2024-08-19T14:27:31.839611+0000","last_unstale":"2024-08-19T14:37:50.126175+0000","last_undegraded":"2024-08-19T14:37:50.126175+0000","last_fullsized":"2024-08-19T14:37:50.126175+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:34:58.616646+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:34:58.616646+0000","last_clean_scrub_stamp":"2024-08-19T14:34:58.616646+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.11","version":"70'139","reported_seq":1225,"reported_epoch":681,"state":"active+clean","last_fresh":"2024-08-19T14:37:50.133922+0000","last_change":"2024-08-19T14:36:43.798545+0000","last_active":"2024-08-19T14:37:50.133922+0000","last_peered":"2024-08-19T14:37:50.133922+0000","last_clean":"2024-08-19T14:37:50.133922+0000","last_became_active":"2024-08-19T14:34:15.731049+0000","last_became_peered":"2024-08-19T14:34:15.731049+0000","last_unstale":"2024-08-19T14:37:50.133922+0000","last_undegraded":"2024-08-19T14:37:50.133922+0000","last_fullsized":"2024-08-19T14:37:50.133922+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:36:43.798385+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:24:50.547428+0000","last_clean_scrub_stamp":"2024-08-19T14:36:43.798385+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:38:10.780687+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"70'139","reported_seq":1160,"reported_epoch":681,"state":"active+clean","last_fresh":"2024-08-19T14:37:50.117932+0000","last_change":"2024-08-19T14:33:27.737496+0000","last_active":"2024-08-19T14:37:50.117932+0000","last_peered":"2024-08-19T14:37:50.117932+0000","last_clean":"2024-08-19T14:37:50.117932+0000","last_became_active":"2024-08-19T14:20:28.204724+0000","last_became_peered":"2024-08-19T14:20:28.204724+0000","last_unstale":"2024-08-19T14:37:50.117932+0000","last_undegraded":"2024-08-19T14:37:50.117932+0000","last_fullsized":"2024-08-19T14:37:50.117932+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:33:27.737367+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:28:34.216370+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.737367+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"70'139","reported_seq":1302,"reported_epoch":681,"state":"active+clean","last_fresh":"2024-08-19T14:37:50.109492+0000","last_change":"2024-08-19T14:37:38.355694+0000","last_active":"2024-08-19T14:37:50.109492+0000","last_peered":"2024-08-19T14:37:50.109492+0000","last_clean":"2024-08-19T14:37:50.109492+0000","last_became_active":"2024-08-19T14:18:31.901556+0000","last_became_peered":"2024-08-19T14:18:31.901556+0000","last_unstale":"2024-08-19T14:37:50.109492+0000","last_undegraded":"2024-08-19T14:37:50.109492+0000","last_fullsized":"2024-08-19T14:37:50.109492+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:37:38.355621+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:37:38.355621+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:38:39.528989+0000","scrub_duration":56,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,0],"acting":[8,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.14","version":"70'139","reported_seq":1332,"reported_epoch":681,"state":"active+clean","last_fresh":"2024-08-19T14:37:50.126290+0000","last_change":"2024-08-19T14:35:12.530145+0000","last_active":"2024-08-19T14:37:50.126290+0000","last_peered":"2024-08-19T14:37:50.126290+0000","last_clean":"2024-08-19T14:37:50.126290+0000","last_became_active":"2024-08-19T14:35:12.529792+0000","last_became_peered":"2024-08-19T14:35:12.529792+0000","last_unstale":"2024-08-19T14:37:50.126290+0000","last_undegraded":"2024-08-19T14:37:50.126290+0000","last_fullsized":"2024-08-19T14:37:50.126290+0000","mapping_epoch":635,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":636,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:34:53.617013+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:34:53.617013+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,9],"acting":[7,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.15","version":"70'139","reported_seq":1184,"reported_epoch":681,"state":"active+clean","last_fresh":"2024-08-19T14:37:50.126390+0000","last_change":"2024-08-19T14:34:57.640316+0000","last_active":"2024-08-19T14:37:50.126390+0000","last_peered":"2024-08-19T14:37:50.126390+0000","last_clean":"2024-08-19T14:37:50.126390+0000","last_became_active":"2024-08-19T14:18:31.912496+0000","last_became_peered":"2024-08-19T14:18:31.912496+0000","last_unstale":"2024-08-19T14:37:50.126390+0000","last_undegraded":"2024-08-19T14:37:50.126390+0000","last_fullsized":"2024-08-19T14:37:50.126390+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:34:57.640213+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:20:09.180923+0000","last_clean_scrub_stamp":"2024-08-19T14:34:57.640213+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"70'139","reported_seq":1165,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-08-19T14:37:46.082610+0000","last_change":"2024-08-19T14:34:06.176510+0000","last_active":"2024-08-19T14:37:46.082610+0000","last_peered":"2024-08-19T14:37:46.082610+0000","last_clean":"2024-08-19T14:37:46.082610+0000","last_became_active":"2024-08-19T14:18:31.901229+0000","last_became_peered":"2024-08-19T14:18:31.901229+0000","last_unstale":"2024-08-19T14:37:46.082610+0000","last_undegraded":"2024-08-19T14:37:46.082610+0000","last_fullsized":"2024-08-19T14:37:46.082610+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:34:06.176403+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:34:06.176403+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1b","version":"70'139","reported_seq":1542,"reported_epoch":681,"state":"active+clean","last_fresh":"2024-08-19T14:37:50.109116+0000","last_change":"2024-08-19T14:36:26.246458+0000","last_active":"2024-08-19T14:37:50.109116+0000","last_peered":"2024-08-19T14:37:50.109116+0000","last_clean":"2024-08-19T14:37:50.109116+0000","last_became_active":"2024-08-19T14:34:15.718427+0000","last_became_peered":"2024-08-19T14:34:15.718427+0000","last_unstale":"2024-08-19T14:37:50.109116+0000","last_undegraded":"2024-08-19T14:37:50.109116+0000","last_fullsized":"2024-08-19T14:37:50.109116+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:36:26.246332+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:36:26.246332+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:37:54.459082+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.18","version":"70'139","reported_seq":1112,"reported_epoch":681,"state":"active+clean","last_fresh":"2024-08-19T14:37:50.126073+0000","last_change":"2024-08-19T14:35:11.528963+0000","last_active":"2024-08-19T14:37:50.126073+0000","last_peered":"2024-08-19T14:37:50.126073+0000","last_clean":"2024-08-19T14:37:50.126073+0000","last_became_active":"2024-08-19T14:35:11.528530+0000","last_became_peered":"2024-08-19T14:35:11.528530+0000","last_unstale":"2024-08-19T14:37:50.126073+0000","last_undegraded":"2024-08-19T14:37:50.126073+0000","last_fullsized":"2024-08-19T14:37:50.126073+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:29:05.336156+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:27:33.084161+0000","last_clean_scrub_stamp":"2024-08-19T14:29:05.336156+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.19","version":"70'139","reported_seq":1529,"reported_epoch":681,"state":"active+clean","last_fresh":"2024-08-19T14:37:50.133818+0000","last_change":"2024-08-19T14:37:41.671836+0000","last_active":"2024-08-19T14:37:50.133818+0000","last_peered":"2024-08-19T14:37:50.133818+0000","last_clean":"2024-08-19T14:37:50.133818+0000","last_became_active":"2024-08-19T14:18:31.902547+0000","last_became_peered":"2024-08-19T14:18:31.902547+0000","last_unstale":"2024-08-19T14:37:50.133818+0000","last_undegraded":"2024-08-19T14:37:50.133818+0000","last_fullsized":"2024-08-19T14:37:50.133818+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:37:41.671731+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:37:41.671731+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:38:59.596992+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1e","version":"70'139","reported_seq":554,"reported_epoch":681,"state":"active+clean","last_fresh":"2024-08-19T14:37:50.114553+0000","last_change":"2024-08-19T14:33:17.778415+0000","last_active":"2024-08-19T14:37:50.114553+0000","last_peered":"2024-08-19T14:37:50.114553+0000","last_clean":"2024-08-19T14:37:50.114553+0000","last_became_active":"2024-08-19T14:30:47.909974+0000","last_became_peered":"2024-08-19T14:30:47.909974+0000","last_unstale":"2024-08-19T14:37:50.114553+0000","last_undegraded":"2024-08-19T14:37:50.114553+0000","last_fullsized":"2024-08-19T14:37:50.114553+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:33:17.778295+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:27:04.780939+0000","last_clean_scrub_stamp":"2024-08-19T14:33:17.778295+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1c","version":"70'139","reported_seq":1296,"reported_epoch":681,"state":"active+clean","last_fresh":"2024-08-19T14:37:50.109174+0000","last_change":"2024-08-19T14:36:31.229907+0000","last_active":"2024-08-19T14:37:50.109174+0000","last_peered":"2024-08-19T14:37:50.109174+0000","last_clean":"2024-08-19T14:37:50.109174+0000","last_became_active":"2024-08-19T14:30:47.908866+0000","last_became_peered":"2024-08-19T14:30:47.908866+0000","last_unstale":"2024-08-19T14:37:50.109174+0000","last_undegraded":"2024-08-19T14:37:50.109174+0000","last_fullsized":"2024-08-19T14:37:50.109174+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:36:31.229792+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:36:31.229792+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:37:54.715659+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,11],"acting":[8,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1d","version":"70'139","reported_seq":1213,"reported_epoch":681,"state":"active+clean","last_fresh":"2024-08-19T14:37:50.113163+0000","last_change":"2024-08-19T14:35:11.542467+0000","last_active":"2024-08-19T14:37:50.113163+0000","last_peered":"2024-08-19T14:37:50.113163+0000","last_clean":"2024-08-19T14:37:50.113163+0000","last_became_active":"2024-08-19T14:35:11.540488+0000","last_became_peered":"2024-08-19T14:35:11.540488+0000","last_unstale":"2024-08-19T14:37:50.113163+0000","last_undegraded":"2024-08-19T14:37:50.113163+0000","last_fullsized":"2024-08-19T14:37:50.113163+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:29.194639+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:28:29.194639+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":3373334552,"num_objects":51474,"num_object_clones":0,"num_object_copies":154422,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19447,"num_objects_misplaced":5934,"num_objects_unfound":0,"num_objects_dirty":51474,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":184842,"num_write_kb":7562049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38468,"num_bytes_recovered":2513108992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9795743744,"data_stored":9795731528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160670,"ondisk_log_size":160670,"up":48,"acting":42,"num_store_stats":10},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":856608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187774,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1900668,"kb_used_data":1039284,"kb_used_omap":1,"kb_used_meta":861374,"kb_avail":91844484,"statfs":{"total":95995035648,"available":94048751616,"internally_reserved":0,"allocated":1064226816,"data_stored":1061212083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":882047428},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":2314987372641,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":74,"seq":317827580231,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1831784,"kb_used_data":1010652,"kb_used_omap":1,"kb_used_meta":821118,"kb_avail":91913368,"statfs":{"total":95995035648,"available":94119288832,"internally_reserved":0,"allocated":1034907648,"data_stored":1031900953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":840825281},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874574,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1404724,"kb_used_data":625256,"kb_used_omap":1,"kb_used_meta":779454,"kb_avail":92340428,"statfs":{"total":95995035648,"available":94556598272,"internally_reserved":0,"allocated":640262144,"data_stored":637251451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":798161349},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":57,"seq":244813136209,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1579788,"kb_used_data":637884,"kb_used_omap":1,"kb_used_meta":941886,"kb_avail":92165364,"statfs":{"total":95995035648,"available":94377332736,"internally_reserved":0,"allocated":653193216,"data_stored":650174931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":964491722},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":206158430548,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692183,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1567984,"kb_used_data":638564,"kb_used_omap":1,"kb_used_meta":929406,"kb_avail":92177168,"statfs":{"total":95995035648,"available":94389420032,"internally_reserved":0,"allocated":653889536,"data_stored":650879289,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":951712197},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986524,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1828072,"kb_used_data":854236,"kb_used_omap":1,"kb_used_meta":973822,"kb_avail":91917080,"statfs":{"total":95995035648,"available":94123089920,"internally_reserved":0,"allocated":874737664,"data_stored":871730969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":997194181},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150051,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3119468,"kb_used_data":2131744,"kb_used_omap":1,"kb_used_meta":987710,"kb_avail":90625684,"statfs":{"total":95995035648,"available":92800700416,"internally_reserved":0,"allocated":2182905856,"data_stored":2179895089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1011415495},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,0,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":634,"seq":2723009265697,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":503868,"kb_used_data":287088,"kb_used_omap":4,"kb_used_meta":216699,"kb_avail":93241284,"statfs":{"total":95995035648,"available":95479074816,"internally_reserved":0,"allocated":293978112,"data_stored":290963867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4407,"internal_metadata":221900489},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575211,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2325312,"kb_used_data":1220920,"kb_used_omap":1,"kb_used_meta":1104318,"kb_avail":91419840,"statfs":{"total":95995035648,"available":93613916160,"internally_reserved":0,"allocated":1250222080,"data_stored":1247203275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1130822085},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738736,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2096952,"kb_used_data":1253612,"kb_used_omap":1,"kb_used_meta":843326,"kb_avail":91648200,"statfs":{"total":95995035648,"available":93847756800,"internally_reserved":0,"allocated":1283698688,"data_stored":1280683923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":863566277},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1265045504,"data_stored":1265041432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1239093248,"data_stored":1239089176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":285409280,"data_stored":285409280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2162823168,"data_stored":2162819096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":859766784,"data_stored":859766784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":640483328,"data_stored":640483328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":644481024,"data_stored":644481024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":627769344,"data_stored":627769344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1023541248,"data_stored":1023541248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1047330816,"data_stored":1047330816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T14:37:53.611 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-19T14:37:53.682 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:37:53.746 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:53.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:53.747 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:37:53.816 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:37:53.847 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:37:53.870 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:53.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:53.871 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:37:53.972 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:37:53.993 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:53.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:53.994 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:37:54.095 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:37:54.120 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:54.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:54.121 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:37:54.222 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:37:54.230 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:54 smithi032 ceph-mon[20004]: pgmap v1621: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 3.1 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 418 op/s; 19447/154428 objects degraded (12.593%); 5934/154428 objects misplaced (3.843%); 9.7 MiB/s, 155 objects/s recovering 2024-08-19T14:37:54.230 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:54 smithi032 ceph-mon[20004]: from='client.26776 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:37:54.230 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:54 smithi032 ceph-mon[20004]: 7.9 scrub ok 2024-08-19T14:37:54.230 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:54 smithi032 ceph-mon[20004]: 7.9 scrub starts 2024-08-19T14:37:54.277 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:54.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:54.278 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:37:54.379 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:37:54.403 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:54.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:54.404 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:37:54.479 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:37:54.479 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:37:54.505 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:37:54.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:54.530 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:54.531 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:37:54.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:54 smithi045 ceph-mon[20143]: pgmap v1621: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 3.1 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 418 op/s; 19447/154428 objects degraded (12.593%); 5934/154428 objects misplaced (3.843%); 9.7 MiB/s, 155 objects/s recovering 2024-08-19T14:37:54.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:54 smithi045 ceph-mon[20143]: from='client.26776 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:37:54.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:54 smithi045 ceph-mon[20143]: 7.9 scrub ok 2024-08-19T14:37:54.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:54 smithi045 ceph-mon[20143]: 7.9 scrub starts 2024-08-19T14:37:54.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:54 smithi031 ceph-mon[16336]: pgmap v1621: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 3.1 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 418 op/s; 19447/154428 objects degraded (12.593%); 5934/154428 objects misplaced (3.843%); 9.7 MiB/s, 155 objects/s recovering 2024-08-19T14:37:54.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:54 smithi031 ceph-mon[16336]: from='client.26776 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:37:54.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:54 smithi031 ceph-mon[16336]: 7.9 scrub ok 2024-08-19T14:37:54.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:54 smithi031 ceph-mon[16336]: 7.9 scrub starts 2024-08-19T14:37:54.632 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:37:54.653 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:54.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:54.654 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:37:54.755 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:37:54.806 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:54.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:54.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:37:54.908 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:37:54.916 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-19T14:37:54.931 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:54.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:54.932 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:37:55.033 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:37:55.055 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:55.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:55.056 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:37:55.158 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:37:55.182 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:55.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:55.183 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:37:55.184 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-08-19T14:37:55.278 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:55 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:37:55.278 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:55 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:37:55.278 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:55 smithi031 ceph-mon[16336]: osdmap e683: 12 total, 12 up, 10 in 2024-08-19T14:37:55.278 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:55 smithi031 ceph-mon[16336]: 7.6 scrub starts 2024-08-19T14:37:55.278 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:55 smithi031 ceph-mon[16336]: 7.9 scrub ok 2024-08-19T14:37:55.284 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:37:55.306 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:55.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:55.307 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:37:55.408 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:37:55.466 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:37:55.480 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:55.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:55.481 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:37:55.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:55 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:37:55.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:55 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:37:55.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:55 smithi032 ceph-mon[20004]: osdmap e683: 12 total, 12 up, 10 in 2024-08-19T14:37:55.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:55 smithi032 ceph-mon[20004]: 7.6 scrub starts 2024-08-19T14:37:55.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:55 smithi032 ceph-mon[20004]: 7.9 scrub ok 2024-08-19T14:37:55.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:55 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:37:55.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:55 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:37:55.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:55 smithi045 ceph-mon[20143]: osdmap e683: 12 total, 12 up, 10 in 2024-08-19T14:37:55.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:55 smithi045 ceph-mon[20143]: 7.6 scrub starts 2024-08-19T14:37:55.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:55 smithi045 ceph-mon[20143]: 7.9 scrub ok 2024-08-19T14:37:55.582 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:37:55.627 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:55.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:55.628 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:37:55.729 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:37:55.752 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:55.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:55.753 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:37:55.854 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:37:55.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:55.879 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:55.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:37:55.981 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:37:56.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:56.004 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:56.004 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:37:56.105 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:37:56.128 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:37:56.128 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:37:56.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:56.130 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:56.130 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:37:56.231 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:37:56.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:56.281 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:56.281 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:37:56.383 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:37:56.430 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:56.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:56.431 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:37:56.435 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:56 smithi031 ceph-mon[16336]: pgmap v1623: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 3.1 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 458 op/s; 19422/153705 objects degraded (12.636%); 5909/153705 objects misplaced (3.844%); 9.5 MiB/s, 152 objects/s recovering 2024-08-19T14:37:56.435 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:56 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:37:56.435 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:56 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:37:56.435 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:56 smithi031 ceph-mon[16336]: osdmap e684: 12 total, 12 up, 10 in 2024-08-19T14:37:56.503 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:56 smithi032 ceph-mon[20004]: pgmap v1623: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 3.1 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 458 op/s; 19422/153705 objects degraded (12.636%); 5909/153705 objects misplaced (3.844%); 9.5 MiB/s, 152 objects/s recovering 2024-08-19T14:37:56.504 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:56 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:37:56.504 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:56 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:37:56.504 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:56 smithi032 ceph-mon[20004]: osdmap e684: 12 total, 12 up, 10 in 2024-08-19T14:37:56.532 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:37:56.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:56 smithi045 ceph-mon[20143]: pgmap v1623: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 3.1 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 458 op/s; 19422/153705 objects degraded (12.636%); 5909/153705 objects misplaced (3.844%); 9.5 MiB/s, 152 objects/s recovering 2024-08-19T14:37:56.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:56 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:37:56.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:56 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:37:56.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:56 smithi045 ceph-mon[20143]: osdmap e684: 12 total, 12 up, 10 in 2024-08-19T14:37:56.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:56.581 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:56.581 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:37:56.682 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:37:56.720 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:56.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:56.721 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:37:56.822 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:37:56.844 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:56.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:56.845 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:37:56.850 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-08-19T14:37:56.946 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:37:56.969 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:56.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:56.970 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:37:57.071 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:37:57.118 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:57.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:57.119 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:37:57.129 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:37:57.220 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:37:57.241 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:57.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:57.242 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:37:57.343 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:37:57.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:57.392 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:57.392 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:37:57.445 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:57 smithi032 ceph-mon[20004]: 7.6 scrub ok 2024-08-19T14:37:57.493 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:37:57.513 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:57 smithi031 ceph-mon[16336]: 7.6 scrub ok 2024-08-19T14:37:57.517 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:57.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:57.518 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:37:57.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:57 smithi045 ceph-mon[20143]: 7.6 scrub ok 2024-08-19T14:37:57.619 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:37:57.642 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:57.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:57.643 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:37:57.679 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:37:57.679 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:37:57.743 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:37:57.783 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:57.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:57.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:37:57.885 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:37:57.930 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:57.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:57.932 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:37:58.033 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:37:58.076 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:58.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:58.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:37:58.178 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:37:58.210 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:58.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:58.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:37:58.312 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:37:58.335 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:58.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:58.337 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:37:58.362 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:37:58.362 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:37:58.363 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:37:58.363 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:37:58.363 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:37:58.363 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:37:58.363 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:37:58.363 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:37:58.364 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:37:58.364 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:37:58.364 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:37:58.364 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:37:58.365 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:37:58.365 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:37:58.365 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:37:58.365 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:37:58.366 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:37:58.366 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:37:58.366 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:37:58.366 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:37:58.366 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:37:58.366 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:37:58.367 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:37:58.367 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:37:58.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:58 smithi031 ceph-mon[16336]: pgmap v1625: 47 pgs: 1 active+clean+scrubbing, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 3.0 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 525 op/s; 18892/149682 objects degraded (12.621%); 5909/149682 objects misplaced (3.948%); 11 MiB/s, 171 objects/s recovering 2024-08-19T14:37:58.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:58 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3282915608' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:37:58.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:58 smithi031 ceph-mon[16336]: 1.0 scrub starts 2024-08-19T14:37:58.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:58 smithi031 ceph-mon[16336]: 1.0 scrub ok 2024-08-19T14:37:58.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:58 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 18892/149682 objects degraded (12.621%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-19T14:37:58.437 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:37:58.459 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:58.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:58.460 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:37:58.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:58 smithi032 ceph-mon[20004]: pgmap v1625: 47 pgs: 1 active+clean+scrubbing, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 3.0 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 525 op/s; 18892/149682 objects degraded (12.621%); 5909/149682 objects misplaced (3.948%); 11 MiB/s, 171 objects/s recovering 2024-08-19T14:37:58.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:58 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3282915608' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:37:58.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:58 smithi032 ceph-mon[20004]: 1.0 scrub starts 2024-08-19T14:37:58.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:58 smithi032 ceph-mon[20004]: 1.0 scrub ok 2024-08-19T14:37:58.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:58 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 18892/149682 objects degraded (12.621%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-19T14:37:58.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:58 smithi045 ceph-mon[20143]: pgmap v1625: 47 pgs: 1 active+clean+scrubbing, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 3.0 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 525 op/s; 18892/149682 objects degraded (12.621%); 5909/149682 objects misplaced (3.948%); 11 MiB/s, 171 objects/s recovering 2024-08-19T14:37:58.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:58 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3282915608' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:37:58.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:58 smithi045 ceph-mon[20143]: 1.0 scrub starts 2024-08-19T14:37:58.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:58 smithi045 ceph-mon[20143]: 1.0 scrub ok 2024-08-19T14:37:58.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:58 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 18892/149682 objects degraded (12.621%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-19T14:37:58.562 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:37:58.585 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:58.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:58.586 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:37:58.679 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-08-19T14:37:58.687 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:37:58.744 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:58.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:58.745 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:37:58.846 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:37:58.869 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:58.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:58.870 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:37:58.960 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:37:58.970 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:37:59.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:59.018 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:59.019 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:37:59.120 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:37:59.168 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:59.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:59.169 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:37:59.270 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:37:59.294 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:59.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:59.295 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:37:59.396 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:37:59.418 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:59.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:59.419 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:37:59.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:59 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:37:59.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:37:59 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:37:59.521 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:37:59.543 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:37:59.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:59.544 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:37:59.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:59 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:37:59.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:37:59 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:37:59.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:59 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:37:59.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:37:59 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:37:59.645 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:37:59.667 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:59.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:59.669 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:37:59.770 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:37:59.798 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:37:59.798 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-08-19T14:37:59.798 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-08-19T14:37:59.798 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-08-19T14:37:59.798 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:37:59.805 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:37:59.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:59.806 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:37:59.907 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:37:59.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:37:59.932 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:37:59.933 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:38:00.034 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:38:00.056 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:00.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:00.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:38:00.158 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:38:00.180 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:00.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:00.181 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:38:00.282 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:38:00.335 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:00.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:00.336 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:38:00.437 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:38:00.458 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:00.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:00.459 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:38:00.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:00 smithi032 ceph-mon[20004]: pgmap v1626: 47 pgs: 1 active+clean+scrubbing, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 3.0 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 368 op/s; 18821/147702 objects degraded (12.743%); 5673/147702 objects misplaced (3.841%); 8.0 MiB/s, 129 objects/s recovering 2024-08-19T14:38:00.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:00 smithi032 ceph-mon[20004]: osdmap e685: 12 total, 12 up, 10 in 2024-08-19T14:38:00.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:00 smithi045 ceph-mon[20143]: pgmap v1626: 47 pgs: 1 active+clean+scrubbing, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 3.0 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 368 op/s; 18821/147702 objects degraded (12.743%); 5673/147702 objects misplaced (3.841%); 8.0 MiB/s, 129 objects/s recovering 2024-08-19T14:38:00.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:00 smithi045 ceph-mon[20143]: osdmap e685: 12 total, 12 up, 10 in 2024-08-19T14:38:00.561 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:38:00.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:00.608 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:00.608 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:38:00.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:00 smithi031 ceph-mon[16336]: pgmap v1626: 47 pgs: 1 active+clean+scrubbing, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 3.0 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 368 op/s; 18821/147702 objects degraded (12.743%); 5673/147702 objects misplaced (3.841%); 8.0 MiB/s, 129 objects/s recovering 2024-08-19T14:38:00.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:00 smithi031 ceph-mon[16336]: osdmap e685: 12 total, 12 up, 10 in 2024-08-19T14:38:00.709 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:38:00.711 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-08-19T14:38:00.730 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:00.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:00.731 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:38:00.832 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:38:00.870 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:00.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:00.871 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:38:00.972 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:38:00.987 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:38:00.998 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:00.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:00.999 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:38:01.100 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:38:01.122 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:01.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:01.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:38:01.224 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:38:01.246 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:01.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:01.247 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:38:01.348 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:38:01.369 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:01 smithi032 ceph-mon[20004]: 1.1b deep-scrub starts 2024-08-19T14:38:01.369 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:01 smithi032 ceph-mon[20004]: 1.1b deep-scrub ok 2024-08-19T14:38:01.370 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:01 smithi032 ceph-mon[20004]: pgmap v1628: 47 pgs: 1 active+clean+scrubbing, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 3.0 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 447 op/s; 18550/145551 objects degraded (12.745%); 5673/145551 objects misplaced (3.898%); 11 MiB/s, 170 objects/s recovering 2024-08-19T14:38:01.371 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:01.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:01.372 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:38:01.473 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:38:01.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:01.499 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:01.500 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:38:01.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:01 smithi045 ceph-mon[20143]: 1.1b deep-scrub starts 2024-08-19T14:38:01.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:01 smithi045 ceph-mon[20143]: 1.1b deep-scrub ok 2024-08-19T14:38:01.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:01 smithi045 ceph-mon[20143]: pgmap v1628: 47 pgs: 1 active+clean+scrubbing, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 3.0 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 447 op/s; 18550/145551 objects degraded (12.745%); 5673/145551 objects misplaced (3.898%); 11 MiB/s, 170 objects/s recovering 2024-08-19T14:38:01.577 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:38:01.578 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-08-19T14:38:01.578 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-08-19T14:38:01.578 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [] 2024-08-19T14:38:01.578 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:38:01.600 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:38:01.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:01 smithi031 ceph-mon[16336]: 1.1b deep-scrub starts 2024-08-19T14:38:01.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:01 smithi031 ceph-mon[16336]: 1.1b deep-scrub ok 2024-08-19T14:38:01.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:01 smithi031 ceph-mon[16336]: pgmap v1628: 47 pgs: 1 active+clean+scrubbing, 6 active+recovering+undersized+degraded+remapped, 40 active+clean; 3.0 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 447 op/s; 18550/145551 objects degraded (12.745%); 5673/145551 objects misplaced (3.898%); 11 MiB/s, 170 objects/s recovering 2024-08-19T14:38:01.653 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:01.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:01.654 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:38:01.755 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:38:01.780 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:01.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:01.782 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:38:01.883 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:38:01.930 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:01.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:01.931 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:38:02.031 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:38:02.073 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:02.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:02.074 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:38:02.175 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:38:02.197 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:02.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:02.199 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:38:02.300 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:38:02.322 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:02.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:02.323 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:38:02.424 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:38:02.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:02.466 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:02.467 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:38:02.480 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-08-19T14:38:02.567 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:38:02.604 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:02.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:02.605 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:38:02.706 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:38:02.730 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:02.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:02.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:38:02.757 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:38:02.832 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:38:02.872 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:02.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:02.873 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:38:02.974 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:38:03.016 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:03.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:03.018 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:38:03.119 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:38:03.141 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:03.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:03.142 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:38:03.243 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:38:03.267 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:03.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:03.268 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:38:03.336 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:38:03.336 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-08-19T14:38:03.336 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:03.336 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.16962.0:190498 7.2 e686/635 7:47c76bef:::benchmark_data_smithi045_115842_object72338:head v 686'12072, mlcod=614'4586)", 2024-08-19T14:38:03.336 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:38:03.331120+0000", 2024-08-19T14:38:03.336 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.0044327400000000001, 2024-08-19T14:38:03.337 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.004463514, 2024-08-19T14:38:03.337 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:38:03.337 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:38:03.337 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-08-19T14:38:03.337 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:38:03.337 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:03.337 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:38:03.337 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:38:03.331120+0000", 2024-08-19T14:38:03.337 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:38:03.337 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:03.337 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:03.337 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:38:03.337 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:38:03.331120+0000", 2024-08-19T14:38:03.338 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:38:03.338 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:03.338 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:03.338 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:38:03.338 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:38:03.331120+0000", 2024-08-19T14:38:03.338 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.17e-07 2024-08-19T14:38:03.338 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:03.338 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:03.338 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:38:03.338 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:38:03.331124+0000", 2024-08-19T14:38:03.338 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5260000000000002e-06 2024-08-19T14:38:03.338 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:03.341 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:03.341 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:38:03.342 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:38:03.331135+0000", 2024-08-19T14:38:03.342 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0909e-05 2024-08-19T14:38:03.342 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:03.342 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:03.342 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:38:03.342 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:38:03.331145+0000", 2024-08-19T14:38:03.342 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0129000000000001e-05 2024-08-19T14:38:03.342 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:03.342 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:03.342 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:38:03.342 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:38:03.331203+0000", 2024-08-19T14:38:03.342 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8001000000000003e-05 2024-08-19T14:38:03.342 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:03.343 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:03.343 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:38:03.343 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:38:03.331220+0000", 2024-08-19T14:38:03.343 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6974e-05 2024-08-19T14:38:03.343 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:38:03.343 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:38:03.343 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:38:03.343 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:03.343 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:03.343 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.16962.0:190502 7.2 e686/635 7:488d53e1:::benchmark_data_smithi045_115842_object72342:head v 686'12073, mlcod=614'4586)", 2024-08-19T14:38:03.343 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:38:03.334816+0000", 2024-08-19T14:38:03.343 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.00073672699999999995, 2024-08-19T14:38:03.344 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00081849400000000001, 2024-08-19T14:38:03.344 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:38:03.344 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:38:03.344 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-08-19T14:38:03.344 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:38:03.344 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:03.344 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:38:03.344 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:38:03.334816+0000", 2024-08-19T14:38:03.345 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:38:03.345 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:03.345 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:03.345 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:38:03.345 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:38:03.334816+0000", 2024-08-19T14:38:03.345 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:38:03.345 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:03.345 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:03.345 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:38:03.345 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:38:03.334817+0000", 2024-08-19T14:38:03.345 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2799999999999995e-07 2024-08-19T14:38:03.345 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:03.345 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:03.345 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:38:03.345 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:38:03.334823+0000", 2024-08-19T14:38:03.346 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.212e-06 2024-08-19T14:38:03.346 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:03.346 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:03.346 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:38:03.346 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:38:03.334830+0000", 2024-08-19T14:38:03.346 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1690000000000003e-06 2024-08-19T14:38:03.346 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:03.346 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:03.346 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:38:03.346 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:38:03.334835+0000", 2024-08-19T14:38:03.346 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7060000000000003e-06 2024-08-19T14:38:03.346 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:03.346 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:03.347 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:38:03.347 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:38:03.334880+0000", 2024-08-19T14:38:03.347 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5303000000000003e-05 2024-08-19T14:38:03.347 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:03.347 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:03.347 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:38:03.347 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:38:03.334896+0000", 2024-08-19T14:38:03.347 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5350999999999999e-05 2024-08-19T14:38:03.347 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:38:03.347 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:38:03.347 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:38:03.347 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:38:03.347 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-08-19T14:38:03.347 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 2 2024-08-19T14:38:03.348 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:38:03.369 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:38:03.406 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:03.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:03.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:38:03.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:03 smithi032 ceph-mon[20004]: 1.1c scrub starts 2024-08-19T14:38:03.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:03 smithi032 ceph-mon[20004]: 1.1c scrub ok 2024-08-19T14:38:03.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:03 smithi032 ceph-mon[20004]: pgmap v1629: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 2.9 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 480 op/s; 18196/142176 objects degraded (12.798%); 5475/142176 objects misplaced (3.851%); 12 MiB/s, 198 objects/s recovering 2024-08-19T14:38:03.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:03 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:38:03.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:03 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:38:03.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:03 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 18196/142176 objects degraded (12.798%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-19T14:38:03.509 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:38:03.531 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:03.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:03.532 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:38:03.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:03 smithi045 ceph-mon[20143]: 1.1c scrub starts 2024-08-19T14:38:03.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:03 smithi045 ceph-mon[20143]: 1.1c scrub ok 2024-08-19T14:38:03.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:03 smithi045 ceph-mon[20143]: pgmap v1629: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 2.9 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 480 op/s; 18196/142176 objects degraded (12.798%); 5475/142176 objects misplaced (3.851%); 12 MiB/s, 198 objects/s recovering 2024-08-19T14:38:03.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:03 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:38:03.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:03 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:38:03.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:03 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 18196/142176 objects degraded (12.798%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-19T14:38:03.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:03 smithi031 ceph-mon[16336]: 1.1c scrub starts 2024-08-19T14:38:03.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:03 smithi031 ceph-mon[16336]: 1.1c scrub ok 2024-08-19T14:38:03.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:03 smithi031 ceph-mon[16336]: pgmap v1629: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 2.9 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 480 op/s; 18196/142176 objects degraded (12.798%); 5475/142176 objects misplaced (3.851%); 12 MiB/s, 198 objects/s recovering 2024-08-19T14:38:03.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:03 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:38:03.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:03 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:38:03.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:03 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 18196/142176 objects degraded (12.798%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-19T14:38:03.633 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:38:03.655 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:03.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:03.656 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:38:03.757 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:38:03.782 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:03.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:03.783 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:38:03.884 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:38:03.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:03.927 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:03.928 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:38:04.029 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:38:04.058 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:04.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:04.059 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:38:04.161 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:38:04.204 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:04.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:04.205 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:38:04.263 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-08-19T14:38:04.306 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:38:04.327 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:04.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:04.328 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:38:04.429 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:38:04.453 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:04.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:04.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:38:04.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:04 smithi032 ceph-mon[20004]: osdmap e686: 12 total, 12 up, 10 in 2024-08-19T14:38:04.555 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:38:04.556 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:38:04.579 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:04.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:04.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:38:04.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:04 smithi031 ceph-mon[16336]: osdmap e686: 12 total, 12 up, 10 in 2024-08-19T14:38:04.681 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:38:04.703 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:04.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:04.704 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:38:04.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:04 smithi045 ceph-mon[20143]: osdmap e686: 12 total, 12 up, 10 in 2024-08-19T14:38:04.805 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:38:04.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:04.857 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:04.857 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:38:04.958 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:38:05.001 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:05.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:05.002 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:38:05.103 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:38:05.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:05.128 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:05.128 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:38:05.229 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:38:05.264 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:05.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:05.265 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:38:05.366 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:38:05.406 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:05.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:05.408 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:38:05.509 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:38:05.545 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:05.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:05.546 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:38:05.599 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:38:05.606 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:05 smithi031 ceph-mon[16336]: pgmap v1631: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 2.9 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 381 op/s; 18105/140529 objects degraded (12.883%); 5385/140529 objects misplaced (3.832%); 9.0 MiB/s, 143 objects/s recovering 2024-08-19T14:38:05.647 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:38:05.670 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:05.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:05.671 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:38:05.690 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:38:05.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:05 smithi032 ceph-mon[20004]: pgmap v1631: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 2.9 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 381 op/s; 18105/140529 objects degraded (12.883%); 5385/140529 objects misplaced (3.832%); 9.0 MiB/s, 143 objects/s recovering 2024-08-19T14:38:05.772 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:38:05.794 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:05.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:05.795 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:38:05.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:05 smithi045 ceph-mon[20143]: pgmap v1631: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 2.9 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 381 op/s; 18105/140529 objects degraded (12.883%); 5385/140529 objects misplaced (3.832%); 9.0 MiB/s, 143 objects/s recovering 2024-08-19T14:38:05.895 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:38:05.952 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:05.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:05.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:38:06.054 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:38:06.075 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:06.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:06.076 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:38:06.098 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:38:06.099 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-08-19T14:38:06.099 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-08-19T14:38:06.099 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-08-19T14:38:06.099 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:38:06.177 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:38:06.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:06.233 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:06.233 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:38:06.334 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:38:06.356 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:06.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:06.357 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:38:06.458 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:38:06.472 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:06 smithi031 ceph-mon[16336]: 7.4 scrub starts 2024-08-19T14:38:06.481 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:06.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:06.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:38:06.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:06 smithi032 ceph-mon[20004]: 7.4 scrub starts 2024-08-19T14:38:06.582 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:38:06.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:06.625 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:06.625 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:38:06.726 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:38:06.749 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:06.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:06.750 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:38:06.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:06 smithi045 ceph-mon[20143]: 7.4 scrub starts 2024-08-19T14:38:06.845 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:38:06.845 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-08-19T14:38:06.851 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:38:06.874 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:06.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:06.875 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:38:06.976 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:38:07.012 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:07.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:07.013 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:38:07.114 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:38:07.156 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:07.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:07.157 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:38:07.258 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:38:07.280 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:07.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:07.281 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:38:07.331 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is set 2024-08-19T14:38:07.381 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:38:07.423 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:07.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:07.424 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:38:07.525 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:38:07.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:07.572 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:07.572 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:38:07.602 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:07 smithi031 ceph-mon[16336]: pgmap v1632: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 2.8 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 474 op/s; 17423/136317 objects degraded (12.781%); 5385/136317 objects misplaced (3.950%); 12 MiB/s, 199 objects/s recovering 2024-08-19T14:38:07.603 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:07 smithi031 ceph-mon[16336]: 7.4 scrub ok 2024-08-19T14:38:07.603 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:07 smithi031 ceph-mon[16336]: from='client.17211 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:38:07.603 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:07 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3608425054' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:38:07.603 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:07 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:38:07.607 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:38:07.673 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:38:07.674 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":1632,"stamp":"2024-08-19T14:38:06.232424+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2978284088,"num_objects":45439,"num_object_clones":0,"num_object_copies":136317,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17423,"num_objects_misplaced":5385,"num_objects_unfound":0,"num_objects_dirty":45439,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190879,"num_write_kb":7562049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40857,"num_bytes_recovered":2669548064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":165016,"ondisk_log_size":165016,"up":141,"acting":135,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":143,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":17146936,"kb_used_data":8712660,"kb_used_omap":18,"kb_used_meta":8434157,"kb_avail":920304584,"statfs":{"total":959950356480,"available":942391894016,"internally_reserved":0,"allocated":8921763840,"data_stored":8891409230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18760,"internal_metadata":8636577464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,8,2,6,6,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-248709120,"num_objects":-3795,"num_object_clones":0,"num_object_copies":-11385,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1398,"num_objects_misplaced":-288,"num_objects_unfound":0,"num_objects_dirty":-3795,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3795,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1595,"num_bytes_recovered":104333312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002189"},"pg_stats":[{"pgid":"7.6","version":"686'12184","reported_seq":12469,"reported_epoch":686,"state":"active+clean","last_fresh":"2024-08-19T14:38:05.433667+0000","last_change":"2024-08-19T14:37:56.236536+0000","last_active":"2024-08-19T14:38:05.433667+0000","last_peered":"2024-08-19T14:38:05.433667+0000","last_clean":"2024-08-19T14:38:05.433667+0000","last_became_active":"2024-08-19T14:33:28.334317+0000","last_became_peered":"2024-08-19T14:33:28.334317+0000","last_unstale":"2024-08-19T14:38:05.433667+0000","last_undegraded":"2024-08-19T14:38:05.433667+0000","last_fullsized":"2024-08-19T14:38:05.433667+0000","mapping_epoch":600,"log_start":"608'2100","ondisk_log_start":"608'2100","created":600,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"684'11923","last_scrub_stamp":"2024-08-19T14:37:56.236394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:37:56.236394+0000","objects_scrubbed":3155,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:39:12.585833+0000","scrub_duration":1586,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188088320,"num_objects":2870,"num_object_clones":0,"num_object_copies":8610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2870,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12184,"num_write_kb":481728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"70'139","reported_seq":1690,"reported_epoch":686,"state":"active+clean","last_fresh":"2024-08-19T14:38:03.302290+0000","last_change":"2024-08-19T14:37:57.701402+0000","last_active":"2024-08-19T14:38:03.302290+0000","last_peered":"2024-08-19T14:38:03.302290+0000","last_clean":"2024-08-19T14:38:03.302290+0000","last_became_active":"2024-08-19T14:35:13.950224+0000","last_became_peered":"2024-08-19T14:35:13.950224+0000","last_unstale":"2024-08-19T14:38:03.302290+0000","last_undegraded":"2024-08-19T14:38:03.302290+0000","last_fullsized":"2024-08-19T14:38:03.302290+0000","mapping_epoch":637,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":638,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:37:57.701290+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:19:58.072083+0000","last_clean_scrub_stamp":"2024-08-19T14:37:57.701290+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:39:18.431910+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,9],"acting":[0,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.5","version":"685'11683","reported_seq":21144,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-08-19T14:38:01.675513+0000","last_change":"2024-08-19T14:35:23.022361+0000","last_active":"2024-08-19T14:38:01.675513+0000","last_peered":"2024-08-19T14:38:01.675513+0000","last_clean":"2024-08-19T14:38:01.675513+0000","last_became_active":"2024-08-19T14:35:23.009238+0000","last_became_peered":"2024-08-19T14:35:23.009238+0000","last_unstale":"2024-08-19T14:38:01.675513+0000","last_undegraded":"2024-08-19T14:38:01.675513+0000","last_fullsized":"2024-08-19T14:38:01.675513+0000","mapping_epoch":641,"log_start":"608'1600","ondisk_log_start":"608'1600","created":600,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196542464,"num_objects":2999,"num_object_clones":0,"num_object_copies":8997,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11683,"num_write_kb":469824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4634,"num_bytes_recovered":303693824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,0],"acting":[11,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.3","version":"70'139","reported_seq":1532,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-08-19T14:37:59.200545+0000","last_change":"2024-08-19T14:33:18.804321+0000","last_active":"2024-08-19T14:37:59.200545+0000","last_peered":"2024-08-19T14:37:59.200545+0000","last_clean":"2024-08-19T14:37:59.200545+0000","last_became_active":"2024-08-19T14:18:13.804721+0000","last_became_peered":"2024-08-19T14:18:13.804721+0000","last_unstale":"2024-08-19T14:37:59.200545+0000","last_undegraded":"2024-08-19T14:37:59.200545+0000","last_fullsized":"2024-08-19T14:37:59.200545+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:33:18.804228+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:33:18.804228+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.4","version":"686'12137","reported_seq":12383,"reported_epoch":686,"state":"active+clean","last_fresh":"2024-08-19T14:38:05.401613+0000","last_change":"2024-08-19T14:36:35.067670+0000","last_active":"2024-08-19T14:38:05.401613+0000","last_peered":"2024-08-19T14:38:05.401613+0000","last_clean":"2024-08-19T14:38:05.401613+0000","last_became_active":"2024-08-19T14:33:28.334446+0000","last_became_peered":"2024-08-19T14:33:28.334446+0000","last_unstale":"2024-08-19T14:38:05.401613+0000","last_undegraded":"2024-08-19T14:38:05.401613+0000","last_fullsized":"2024-08-19T14:38:05.401613+0000","mapping_epoch":600,"log_start":"608'2100","ondisk_log_start":"608'2100","created":600,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"660'9797","last_scrub_stamp":"2024-08-19T14:36:35.067331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:36:35.067331+0000","objects_scrubbed":5188,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2448,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184090624,"num_objects":2809,"num_object_clones":0,"num_object_copies":8427,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2809,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12137,"num_write_kb":478272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"70'139","reported_seq":1491,"reported_epoch":686,"state":"active+clean","last_fresh":"2024-08-19T14:38:03.304785+0000","last_change":"2024-08-19T14:34:55.580212+0000","last_active":"2024-08-19T14:38:03.304785+0000","last_peered":"2024-08-19T14:38:03.304785+0000","last_clean":"2024-08-19T14:38:03.304785+0000","last_became_active":"2024-08-19T14:20:50.742354+0000","last_became_peered":"2024-08-19T14:20:50.742354+0000","last_unstale":"2024-08-19T14:38:03.304785+0000","last_undegraded":"2024-08-19T14:38:03.304785+0000","last_fullsized":"2024-08-19T14:38:03.304785+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:34:55.580141+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:19:59.302931+0000","last_clean_scrub_stamp":"2024-08-19T14:34:55.580141+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"686'11710","reported_seq":11993,"reported_epoch":686,"state":"active+clean","last_fresh":"2024-08-19T14:38:05.263044+0000","last_change":"2024-08-19T14:37:43.229201+0000","last_active":"2024-08-19T14:38:05.263044+0000","last_peered":"2024-08-19T14:38:05.263044+0000","last_clean":"2024-08-19T14:38:05.263044+0000","last_became_active":"2024-08-19T14:33:28.331833+0000","last_became_peered":"2024-08-19T14:33:28.331833+0000","last_unstale":"2024-08-19T14:38:05.263044+0000","last_undegraded":"2024-08-19T14:38:05.263044+0000","last_fullsized":"2024-08-19T14:38:05.263044+0000","mapping_epoch":600,"log_start":"608'1700","ondisk_log_start":"608'1700","created":600,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"678'11114","last_scrub_stamp":"2024-08-19T14:37:43.229072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:37:43.229072+0000","objects_scrubbed":3413,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:39:12.361681+0000","scrub_duration":910,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184025088,"num_objects":2808,"num_object_clones":0,"num_object_copies":8424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11710,"num_write_kb":464576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,1],"acting":[8,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1","version":"70'139","reported_seq":789,"reported_epoch":686,"state":"active+clean","last_fresh":"2024-08-19T14:38:03.302840+0000","last_change":"2024-08-19T14:37:42.630047+0000","last_active":"2024-08-19T14:38:03.302840+0000","last_peered":"2024-08-19T14:38:03.302840+0000","last_clean":"2024-08-19T14:38:03.302840+0000","last_became_active":"2024-08-19T14:35:12.530566+0000","last_became_peered":"2024-08-19T14:35:12.530566+0000","last_unstale":"2024-08-19T14:38:03.302840+0000","last_undegraded":"2024-08-19T14:38:03.302840+0000","last_fullsized":"2024-08-19T14:38:03.302840+0000","mapping_epoch":635,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":636,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:37:42.629919+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:37:42.629919+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:39:05.497590+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,2],"acting":[0,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"686'11817","reported_seq":12078,"reported_epoch":686,"state":"active+clean","last_fresh":"2024-08-19T14:38:03.672499+0000","last_change":"2024-08-19T14:34:52.869444+0000","last_active":"2024-08-19T14:38:03.672499+0000","last_peered":"2024-08-19T14:38:03.672499+0000","last_clean":"2024-08-19T14:38:03.672499+0000","last_became_active":"2024-08-19T14:33:28.332994+0000","last_became_peered":"2024-08-19T14:33:28.332994+0000","last_unstale":"2024-08-19T14:38:03.672499+0000","last_undegraded":"2024-08-19T14:38:03.672499+0000","last_fullsized":"2024-08-19T14:38:03.672499+0000","mapping_epoch":600,"log_start":"608'1800","ondisk_log_start":"608'1800","created":600,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"627'6737","last_scrub_stamp":"2024-08-19T14:34:52.869233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:34:52.869233+0000","objects_scrubbed":6705,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1588,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180944896,"num_objects":2761,"num_object_clones":0,"num_object_copies":8283,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11817,"num_write_kb":466496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"70'139","reported_seq":1193,"reported_epoch":686,"state":"active+clean","last_fresh":"2024-08-19T14:38:03.314094+0000","last_change":"2024-08-19T14:35:11.537368+0000","last_active":"2024-08-19T14:38:03.314094+0000","last_peered":"2024-08-19T14:38:03.314094+0000","last_clean":"2024-08-19T14:38:03.314094+0000","last_became_active":"2024-08-19T14:35:11.536828+0000","last_became_peered":"2024-08-19T14:35:11.536828+0000","last_unstale":"2024-08-19T14:38:03.314094+0000","last_undegraded":"2024-08-19T14:38:03.314094+0000","last_fullsized":"2024-08-19T14:38:03.314094+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:27.536928+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:21:46.057624+0000","last_clean_scrub_stamp":"2024-08-19T14:28:27.536928+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"686'11732","reported_seq":21149,"reported_epoch":686,"state":"active+clean","last_fresh":"2024-08-19T14:38:04.461874+0000","last_change":"2024-08-19T14:36:15.120872+0000","last_active":"2024-08-19T14:38:04.461874+0000","last_peered":"2024-08-19T14:38:04.461874+0000","last_clean":"2024-08-19T14:38:04.461874+0000","last_became_active":"2024-08-19T14:36:15.096070+0000","last_became_peered":"2024-08-19T14:36:15.096070+0000","last_unstale":"2024-08-19T14:38:04.461874+0000","last_undegraded":"2024-08-19T14:38:04.461874+0000","last_fullsized":"2024-08-19T14:38:04.461874+0000","mapping_epoch":653,"log_start":"608'1700","ondisk_log_start":"608'1700","created":600,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":185335808,"num_objects":2828,"num_object_clones":0,"num_object_copies":8484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2828,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11732,"num_write_kb":465920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4611,"num_bytes_recovered":302186496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,9],"acting":[3,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"70'139","reported_seq":761,"reported_epoch":686,"state":"active+clean","last_fresh":"2024-08-19T14:38:03.325149+0000","last_change":"2024-08-19T14:37:45.357396+0000","last_active":"2024-08-19T14:38:03.325149+0000","last_peered":"2024-08-19T14:38:03.325149+0000","last_clean":"2024-08-19T14:38:03.325149+0000","last_became_active":"2024-08-19T14:34:15.717054+0000","last_became_peered":"2024-08-19T14:34:15.717054+0000","last_unstale":"2024-08-19T14:38:03.325149+0000","last_undegraded":"2024-08-19T14:38:03.325149+0000","last_fullsized":"2024-08-19T14:38:03.325149+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:37:45.357293+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:27.848870+0000","last_clean_scrub_stamp":"2024-08-19T14:37:45.357293+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:38:48.964771+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"7.3","version":"686'12146","reported_seq":17972,"reported_epoch":686,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-19T14:38:04.535172+0000","last_change":"2024-08-19T14:35:13.136988+0000","last_active":"2024-08-19T14:38:04.535172+0000","last_peered":"2024-08-19T14:38:04.535172+0000","last_clean":"2024-08-19T14:33:27.312789+0000","last_became_active":"2024-08-19T14:35:13.021078+0000","last_became_peered":"2024-08-19T14:35:13.021078+0000","last_unstale":"2024-08-19T14:38:04.535172+0000","last_undegraded":"2024-08-19T14:35:12.592288+0000","last_fullsized":"2024-08-19T14:35:12.526971+0000","mapping_epoch":635,"log_start":"608'2100","ondisk_log_start":"608'2100","created":600,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":190578688,"num_objects":2908,"num_object_clones":0,"num_object_copies":8724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12146,"num_write_kb":481728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2821,"num_bytes_recovered":182386688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":-1529}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"70'139","reported_seq":713,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-08-19T14:37:59.187979+0000","last_change":"2024-08-19T14:35:11.533207+0000","last_active":"2024-08-19T14:37:59.187979+0000","last_peered":"2024-08-19T14:37:59.187979+0000","last_clean":"2024-08-19T14:37:59.187979+0000","last_became_active":"2024-08-19T14:35:11.532858+0000","last_became_peered":"2024-08-19T14:35:11.532858+0000","last_unstale":"2024-08-19T14:37:59.187979+0000","last_undegraded":"2024-08-19T14:37:59.187979+0000","last_fullsized":"2024-08-19T14:37:59.187979+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:27.331041+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:28:27.331041+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,0],"acting":[11,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.2","version":"686'12110","reported_seq":17877,"reported_epoch":686,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-19T14:38:04.544536+0000","last_change":"2024-08-19T14:35:13.121204+0000","last_active":"2024-08-19T14:38:04.544536+0000","last_peered":"2024-08-19T14:38:04.544536+0000","last_clean":"2024-08-19T14:33:27.312789+0000","last_became_active":"2024-08-19T14:35:13.013347+0000","last_became_peered":"2024-08-19T14:35:13.013347+0000","last_unstale":"2024-08-19T14:38:04.544536+0000","last_undegraded":"2024-08-19T14:35:12.598259+0000","last_fullsized":"2024-08-19T14:35:12.526602+0000","mapping_epoch":635,"log_start":"608'2100","ondisk_log_start":"608'2100","created":600,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188088320,"num_objects":2870,"num_object_clones":0,"num_object_copies":8610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4444,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2870,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12110,"num_write_kb":479360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2791,"num_bytes_recovered":180813824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":-1574}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"70'139","reported_seq":590,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-08-19T14:37:59.190035+0000","last_change":"2024-08-19T14:33:16.760928+0000","last_active":"2024-08-19T14:37:59.190035+0000","last_peered":"2024-08-19T14:37:59.190035+0000","last_clean":"2024-08-19T14:37:59.190035+0000","last_became_active":"2024-08-19T14:30:47.909067+0000","last_became_peered":"2024-08-19T14:30:47.909067+0000","last_unstale":"2024-08-19T14:37:59.190035+0000","last_undegraded":"2024-08-19T14:37:59.190035+0000","last_fullsized":"2024-08-19T14:37:59.190035+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:33:16.760810+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:33:16.760810+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.c","version":"686'11728","reported_seq":17536,"reported_epoch":686,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-19T14:38:04.523317+0000","last_change":"2024-08-19T14:35:13.133729+0000","last_active":"2024-08-19T14:38:04.523317+0000","last_peered":"2024-08-19T14:38:04.523317+0000","last_clean":"2024-08-19T14:33:27.312789+0000","last_became_active":"2024-08-19T14:35:13.003266+0000","last_became_peered":"2024-08-19T14:35:13.003266+0000","last_unstale":"2024-08-19T14:38:04.523317+0000","last_undegraded":"2024-08-19T14:35:12.593502+0000","last_fullsized":"2024-08-19T14:35:12.526375+0000","mapping_epoch":635,"log_start":"608'1700","ondisk_log_start":"608'1700","created":600,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":185335808,"num_objects":2828,"num_object_clones":0,"num_object_copies":8484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4197,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2828,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11728,"num_write_kb":465792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2812,"num_bytes_recovered":182910976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,2],"acting":[3,11],"avail_no_missing":["3","11"],"object_location_counts":[{"shards":"3,11","objects":-1369}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"70'139","reported_seq":757,"reported_epoch":686,"state":"active+clean","last_fresh":"2024-08-19T14:38:03.306597+0000","last_change":"2024-08-19T14:34:50.256878+0000","last_active":"2024-08-19T14:38:03.306597+0000","last_peered":"2024-08-19T14:38:03.306597+0000","last_clean":"2024-08-19T14:38:03.306597+0000","last_became_active":"2024-08-19T14:34:15.717472+0000","last_became_peered":"2024-08-19T14:34:15.717472+0000","last_unstale":"2024-08-19T14:38:03.306597+0000","last_undegraded":"2024-08-19T14:38:03.306597+0000","last_fullsized":"2024-08-19T14:38:03.306597+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:34:50.256808+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:28.864870+0000","last_clean_scrub_stamp":"2024-08-19T14:34:50.256808+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,0],"acting":[4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"686'15480","reported_seq":25269,"reported_epoch":686,"state":"active+clean","last_fresh":"2024-08-19T14:38:05.412842+0000","last_change":"2024-08-19T14:37:42.029576+0000","last_active":"2024-08-19T14:38:05.412842+0000","last_peered":"2024-08-19T14:38:05.412842+0000","last_clean":"2024-08-19T14:38:05.412842+0000","last_became_active":"2024-08-19T14:36:14.098166+0000","last_became_peered":"2024-08-19T14:36:14.098166+0000","last_unstale":"2024-08-19T14:38:05.412842+0000","last_undegraded":"2024-08-19T14:38:05.412842+0000","last_fullsized":"2024-08-19T14:38:05.412842+0000","mapping_epoch":652,"log_start":"622'5402","ondisk_log_start":"622'5402","created":600,"last_epoch_clean":653,"parent":"0.0","parent_split_bits":0,"last_scrub":"678'14831","last_scrub_stamp":"2024-08-19T14:37:42.029449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:37:42.029449+0000","objects_scrubbed":3383,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:38:44.946893+0000","scrub_duration":1404,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":177995776,"num_objects":2716,"num_object_clones":0,"num_object_copies":8148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2716,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11850,"num_write_kb":466112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4700,"num_bytes_recovered":308019200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.9","version":"70'139","reported_seq":1693,"reported_epoch":686,"state":"active+clean","last_fresh":"2024-08-19T14:38:03.304636+0000","last_change":"2024-08-19T14:35:12.529993+0000","last_active":"2024-08-19T14:38:03.304636+0000","last_peered":"2024-08-19T14:38:03.304636+0000","last_clean":"2024-08-19T14:38:03.304636+0000","last_became_active":"2024-08-19T14:35:12.529628+0000","last_became_peered":"2024-08-19T14:35:12.529628+0000","last_unstale":"2024-08-19T14:38:03.304636+0000","last_undegraded":"2024-08-19T14:38:03.304636+0000","last_fullsized":"2024-08-19T14:38:03.304636+0000","mapping_epoch":635,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":636,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:34:51.561893+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:34:51.561893+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.e","version":"685'12008","reported_seq":22428,"reported_epoch":685,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-19T14:38:02.652607+0000","last_change":"2024-08-19T14:35:11.863818+0000","last_active":"2024-08-19T14:38:02.652607+0000","last_peered":"2024-08-19T14:38:02.652607+0000","last_clean":"2024-08-19T14:33:27.312789+0000","last_became_active":"2024-08-19T14:35:11.806304+0000","last_became_peered":"2024-08-19T14:35:11.806304+0000","last_unstale":"2024-08-19T14:38:02.652607+0000","last_undegraded":"2024-08-19T14:35:11.779866+0000","last_fullsized":"2024-08-19T14:35:11.528207+0000","mapping_epoch":634,"log_start":"608'2000","ondisk_log_start":"608'2000","created":600,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":182714368,"num_objects":2788,"num_object_clones":0,"num_object_copies":8364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2254,"num_objects_misplaced":4791,"num_objects_unfound":0,"num_objects_dirty":2788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12008,"num_write_kb":473472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3961,"num_bytes_recovered":257556480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":-2003},{"shards":"1,5,9","objects":2537}],"blocked_by":[],"up_primary":2,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"70'139","reported_seq":723,"reported_epoch":686,"state":"active+clean","last_fresh":"2024-08-19T14:38:03.314373+0000","last_change":"2024-08-19T14:35:11.537896+0000","last_active":"2024-08-19T14:38:03.314373+0000","last_peered":"2024-08-19T14:38:03.314373+0000","last_clean":"2024-08-19T14:38:03.314373+0000","last_became_active":"2024-08-19T14:35:11.537474+0000","last_became_peered":"2024-08-19T14:35:11.537474+0000","last_unstale":"2024-08-19T14:38:03.314373+0000","last_undegraded":"2024-08-19T14:38:03.314373+0000","last_fullsized":"2024-08-19T14:38:03.314373+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:59.829567+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:28:59.829567+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"685'11941","reported_seq":12124,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-08-19T14:38:02.589327+0000","last_change":"2024-08-19T14:33:28.335586+0000","last_active":"2024-08-19T14:38:02.589327+0000","last_peered":"2024-08-19T14:38:02.589327+0000","last_clean":"2024-08-19T14:38:02.589327+0000","last_became_active":"2024-08-19T14:33:28.335298+0000","last_became_peered":"2024-08-19T14:33:28.335298+0000","last_unstale":"2024-08-19T14:38:02.589327+0000","last_undegraded":"2024-08-19T14:38:02.589327+0000","last_fullsized":"2024-08-19T14:38:02.589327+0000","mapping_epoch":600,"log_start":"608'1900","ondisk_log_start":"608'1900","created":600,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":182452248,"num_objects":2785,"num_object_clones":0,"num_object_copies":8355,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2785,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11941,"num_write_kb":471169,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"70'139","reported_seq":1042,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-08-19T14:37:59.188002+0000","last_change":"2024-08-19T14:33:15.753668+0000","last_active":"2024-08-19T14:37:59.188002+0000","last_peered":"2024-08-19T14:37:59.188002+0000","last_clean":"2024-08-19T14:37:59.188002+0000","last_became_active":"2024-08-19T14:30:47.941200+0000","last_became_peered":"2024-08-19T14:30:47.941200+0000","last_unstale":"2024-08-19T14:37:59.188002+0000","last_undegraded":"2024-08-19T14:37:59.188002+0000","last_fullsized":"2024-08-19T14:37:59.188002+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:33:15.753595+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:33:15.753595+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.8","version":"686'12264","reported_seq":26729,"reported_epoch":686,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-19T14:38:05.439787+0000","last_change":"2024-08-19T14:35:13.081742+0000","last_active":"2024-08-19T14:38:05.439787+0000","last_peered":"2024-08-19T14:38:05.439787+0000","last_clean":"2024-08-19T14:33:27.312789+0000","last_became_active":"2024-08-19T14:35:12.998871+0000","last_became_peered":"2024-08-19T14:35:12.998871+0000","last_unstale":"2024-08-19T14:38:05.439787+0000","last_undegraded":"2024-08-19T14:35:12.561096+0000","last_fullsized":"2024-08-19T14:35:12.526115+0000","mapping_epoch":635,"log_start":"608'2200","ondisk_log_start":"608'2200","created":600,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":186515456,"num_objects":2846,"num_object_clones":0,"num_object_copies":8538,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":137,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12264,"num_write_kb":483520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7140,"num_bytes_recovered":467927040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[{"shards":"4,7","objects":2709}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.e","version":"70'139","reported_seq":1234,"reported_epoch":686,"state":"active+clean","last_fresh":"2024-08-19T14:38:03.306625+0000","last_change":"2024-08-19T14:34:55.333177+0000","last_active":"2024-08-19T14:38:03.306625+0000","last_peered":"2024-08-19T14:38:03.306625+0000","last_clean":"2024-08-19T14:38:03.306625+0000","last_became_active":"2024-08-19T14:18:26.884105+0000","last_became_peered":"2024-08-19T14:18:26.884105+0000","last_unstale":"2024-08-19T14:38:03.306625+0000","last_undegraded":"2024-08-19T14:38:03.306625+0000","last_fullsized":"2024-08-19T14:38:03.306625+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:34:55.333065+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:32:31.196288+0000","last_clean_scrub_stamp":"2024-08-19T14:34:55.333065+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"686'12053","reported_seq":12373,"reported_epoch":686,"state":"active+clean","last_fresh":"2024-08-19T14:38:05.456657+0000","last_change":"2024-08-19T14:37:54.857799+0000","last_active":"2024-08-19T14:38:05.456657+0000","last_peered":"2024-08-19T14:38:05.456657+0000","last_clean":"2024-08-19T14:38:05.456657+0000","last_became_active":"2024-08-19T14:33:28.334547+0000","last_became_peered":"2024-08-19T14:33:28.334547+0000","last_unstale":"2024-08-19T14:38:05.456657+0000","last_undegraded":"2024-08-19T14:38:05.456657+0000","last_fullsized":"2024-08-19T14:38:05.456657+0000","mapping_epoch":600,"log_start":"608'2000","ondisk_log_start":"608'2000","created":600,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"683'11737","last_scrub_stamp":"2024-08-19T14:37:54.851356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:37:54.851356+0000","objects_scrubbed":3161,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:39:09.304335+0000","scrub_duration":1254,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":185270272,"num_objects":2827,"num_object_clones":0,"num_object_copies":8481,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2827,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12053,"num_write_kb":476160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,8],"acting":[0,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"70'139","reported_seq":986,"reported_epoch":686,"state":"active+clean","last_fresh":"2024-08-19T14:38:03.306747+0000","last_change":"2024-08-19T14:34:12.199530+0000","last_active":"2024-08-19T14:38:03.306747+0000","last_peered":"2024-08-19T14:38:03.306747+0000","last_clean":"2024-08-19T14:38:03.306747+0000","last_became_active":"2024-08-19T14:27:31.837754+0000","last_became_peered":"2024-08-19T14:27:31.837754+0000","last_unstale":"2024-08-19T14:38:03.306747+0000","last_undegraded":"2024-08-19T14:38:03.306747+0000","last_fullsized":"2024-08-19T14:38:03.306747+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:34:12.199449+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:34:12.199449+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.a","version":"685'11819","reported_seq":28629,"reported_epoch":685,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-19T14:38:01.706703+0000","last_change":"2024-08-19T14:35:13.046246+0000","last_active":"2024-08-19T14:38:01.706703+0000","last_peered":"2024-08-19T14:38:01.706703+0000","last_clean":"2024-08-19T14:33:27.312789+0000","last_became_active":"2024-08-19T14:35:13.005199+0000","last_became_peered":"2024-08-19T14:35:13.005199+0000","last_unstale":"2024-08-19T14:38:01.706703+0000","last_undegraded":"2024-08-19T14:35:12.692218+0000","last_fullsized":"2024-08-19T14:35:12.524923+0000","mapping_epoch":635,"log_start":"608'1800","ondisk_log_start":"608'1800","created":600,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":190513152,"num_objects":2907,"num_object_clones":0,"num_object_copies":8721,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1954,"num_objects_misplaced":594,"num_objects_unfound":0,"num_objects_dirty":2907,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11819,"num_write_kb":471232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7384,"num_bytes_recovered":483196928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,2],"acting":[11,9],"avail_no_missing":["11","3","9"],"object_location_counts":[{"shards":"3,9,11","objects":594},{"shards":"9,11","objects":359}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.c","version":"70'139","reported_seq":1341,"reported_epoch":686,"state":"active+clean","last_fresh":"2024-08-19T14:38:03.317914+0000","last_change":"2024-08-19T14:34:15.738488+0000","last_active":"2024-08-19T14:38:03.317914+0000","last_peered":"2024-08-19T14:38:03.317914+0000","last_clean":"2024-08-19T14:38:03.317914+0000","last_became_active":"2024-08-19T14:34:15.733282+0000","last_became_peered":"2024-08-19T14:34:15.733282+0000","last_unstale":"2024-08-19T14:38:03.317914+0000","last_undegraded":"2024-08-19T14:38:03.317914+0000","last_fullsized":"2024-08-19T14:38:03.317914+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:33:17.453290+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:22:52.164586+0000","last_clean_scrub_stamp":"2024-08-19T14:33:17.453290+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,9],"acting":[3,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"685'11697","reported_seq":11879,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-08-19T14:38:02.636303+0000","last_change":"2024-08-19T14:33:28.623717+0000","last_active":"2024-08-19T14:38:02.636303+0000","last_peered":"2024-08-19T14:38:02.636303+0000","last_clean":"2024-08-19T14:38:02.636303+0000","last_became_active":"2024-08-19T14:33:28.623318+0000","last_became_peered":"2024-08-19T14:33:28.623318+0000","last_unstale":"2024-08-19T14:38:02.636303+0000","last_undegraded":"2024-08-19T14:38:02.636303+0000","last_fullsized":"2024-08-19T14:38:02.636303+0000","mapping_epoch":600,"log_start":"608'1600","ondisk_log_start":"608'1600","created":600,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189202432,"num_objects":2887,"num_object_clones":0,"num_object_copies":8661,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11697,"num_write_kb":466688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"70'139","reported_seq":1238,"reported_epoch":686,"state":"active+clean","last_fresh":"2024-08-19T14:38:03.304960+0000","last_change":"2024-08-19T14:34:49.476489+0000","last_active":"2024-08-19T14:38:03.304960+0000","last_peered":"2024-08-19T14:38:03.304960+0000","last_clean":"2024-08-19T14:38:03.304960+0000","last_became_active":"2024-08-19T14:18:26.882002+0000","last_became_peered":"2024-08-19T14:18:26.882002+0000","last_unstale":"2024-08-19T14:38:03.304960+0000","last_undegraded":"2024-08-19T14:38:03.304960+0000","last_fullsized":"2024-08-19T14:38:03.304960+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:34:49.476415+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:24:50.276408+0000","last_clean_scrub_stamp":"2024-08-19T14:34:49.476415+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"70'139","reported_seq":1015,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-08-19T14:37:59.200154+0000","last_change":"2024-08-19T14:33:19.781527+0000","last_active":"2024-08-19T14:37:59.200154+0000","last_peered":"2024-08-19T14:37:59.200154+0000","last_clean":"2024-08-19T14:37:59.200154+0000","last_became_active":"2024-08-19T14:27:31.838968+0000","last_became_peered":"2024-08-19T14:27:31.838968+0000","last_unstale":"2024-08-19T14:37:59.200154+0000","last_undegraded":"2024-08-19T14:37:59.200154+0000","last_fullsized":"2024-08-19T14:37:59.200154+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:33:19.781431+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:28:35.206115+0000","last_clean_scrub_stamp":"2024-08-19T14:33:19.781431+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.13","version":"70'139","reported_seq":1557,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-08-19T14:37:59.200450+0000","last_change":"2024-08-19T14:33:35.786236+0000","last_active":"2024-08-19T14:37:59.200450+0000","last_peered":"2024-08-19T14:37:59.200450+0000","last_clean":"2024-08-19T14:37:59.200450+0000","last_became_active":"2024-08-19T14:30:47.939849+0000","last_became_peered":"2024-08-19T14:30:47.939849+0000","last_unstale":"2024-08-19T14:37:59.200450+0000","last_undegraded":"2024-08-19T14:37:59.200450+0000","last_fullsized":"2024-08-19T14:37:59.200450+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:33:35.786105+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:33:35.786105+0000","last_clean_scrub_stamp":"2024-08-19T14:33:35.786105+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.10","version":"70'139","reported_seq":1187,"reported_epoch":686,"state":"active+clean","last_fresh":"2024-08-19T14:38:03.304724+0000","last_change":"2024-08-19T14:34:58.616722+0000","last_active":"2024-08-19T14:38:03.304724+0000","last_peered":"2024-08-19T14:38:03.304724+0000","last_clean":"2024-08-19T14:38:03.304724+0000","last_became_active":"2024-08-19T14:27:31.839611+0000","last_became_peered":"2024-08-19T14:27:31.839611+0000","last_unstale":"2024-08-19T14:38:03.304724+0000","last_undegraded":"2024-08-19T14:38:03.304724+0000","last_fullsized":"2024-08-19T14:38:03.304724+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:34:58.616646+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:34:58.616646+0000","last_clean_scrub_stamp":"2024-08-19T14:34:58.616646+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.11","version":"70'139","reported_seq":1235,"reported_epoch":686,"state":"active+clean","last_fresh":"2024-08-19T14:38:03.302649+0000","last_change":"2024-08-19T14:36:43.798545+0000","last_active":"2024-08-19T14:38:03.302649+0000","last_peered":"2024-08-19T14:38:03.302649+0000","last_clean":"2024-08-19T14:38:03.302649+0000","last_became_active":"2024-08-19T14:34:15.731049+0000","last_became_peered":"2024-08-19T14:34:15.731049+0000","last_unstale":"2024-08-19T14:38:03.302649+0000","last_undegraded":"2024-08-19T14:38:03.302649+0000","last_fullsized":"2024-08-19T14:38:03.302649+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:36:43.798385+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:24:50.547428+0000","last_clean_scrub_stamp":"2024-08-19T14:36:43.798385+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:38:10.780687+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"70'139","reported_seq":1168,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-08-19T14:37:59.215770+0000","last_change":"2024-08-19T14:33:27.737496+0000","last_active":"2024-08-19T14:37:59.215770+0000","last_peered":"2024-08-19T14:37:59.215770+0000","last_clean":"2024-08-19T14:37:59.215770+0000","last_became_active":"2024-08-19T14:20:28.204724+0000","last_became_peered":"2024-08-19T14:20:28.204724+0000","last_unstale":"2024-08-19T14:37:59.215770+0000","last_undegraded":"2024-08-19T14:37:59.215770+0000","last_fullsized":"2024-08-19T14:37:59.215770+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:33:27.737367+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:28:34.216370+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.737367+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"70'139","reported_seq":1313,"reported_epoch":686,"state":"active+clean","last_fresh":"2024-08-19T14:38:03.329161+0000","last_change":"2024-08-19T14:37:38.355694+0000","last_active":"2024-08-19T14:38:03.329161+0000","last_peered":"2024-08-19T14:38:03.329161+0000","last_clean":"2024-08-19T14:38:03.329161+0000","last_became_active":"2024-08-19T14:18:31.901556+0000","last_became_peered":"2024-08-19T14:18:31.901556+0000","last_unstale":"2024-08-19T14:38:03.329161+0000","last_undegraded":"2024-08-19T14:38:03.329161+0000","last_fullsized":"2024-08-19T14:38:03.329161+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:37:38.355621+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:37:38.355621+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:38:39.528989+0000","scrub_duration":56,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,0],"acting":[8,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.14","version":"70'139","reported_seq":1342,"reported_epoch":686,"state":"active+clean","last_fresh":"2024-08-19T14:38:03.304815+0000","last_change":"2024-08-19T14:35:12.530145+0000","last_active":"2024-08-19T14:38:03.304815+0000","last_peered":"2024-08-19T14:38:03.304815+0000","last_clean":"2024-08-19T14:38:03.304815+0000","last_became_active":"2024-08-19T14:35:12.529792+0000","last_became_peered":"2024-08-19T14:35:12.529792+0000","last_unstale":"2024-08-19T14:38:03.304815+0000","last_undegraded":"2024-08-19T14:38:03.304815+0000","last_fullsized":"2024-08-19T14:38:03.304815+0000","mapping_epoch":635,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":636,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:34:53.617013+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:34:53.617013+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,9],"acting":[7,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.15","version":"70'139","reported_seq":1194,"reported_epoch":686,"state":"active+clean","last_fresh":"2024-08-19T14:38:03.304864+0000","last_change":"2024-08-19T14:34:57.640316+0000","last_active":"2024-08-19T14:38:03.304864+0000","last_peered":"2024-08-19T14:38:03.304864+0000","last_clean":"2024-08-19T14:38:03.304864+0000","last_became_active":"2024-08-19T14:18:31.912496+0000","last_became_peered":"2024-08-19T14:18:31.912496+0000","last_unstale":"2024-08-19T14:38:03.304864+0000","last_undegraded":"2024-08-19T14:38:03.304864+0000","last_fullsized":"2024-08-19T14:38:03.304864+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:34:57.640213+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:20:09.180923+0000","last_clean_scrub_stamp":"2024-08-19T14:34:57.640213+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"70'139","reported_seq":1179,"reported_epoch":686,"state":"active+clean","last_fresh":"2024-08-19T14:38:03.306425+0000","last_change":"2024-08-19T14:34:06.176510+0000","last_active":"2024-08-19T14:38:03.306425+0000","last_peered":"2024-08-19T14:38:03.306425+0000","last_clean":"2024-08-19T14:38:03.306425+0000","last_became_active":"2024-08-19T14:18:31.901229+0000","last_became_peered":"2024-08-19T14:18:31.901229+0000","last_unstale":"2024-08-19T14:38:03.306425+0000","last_undegraded":"2024-08-19T14:38:03.306425+0000","last_fullsized":"2024-08-19T14:38:03.306425+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:34:06.176403+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:34:06.176403+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1b","version":"70'139","reported_seq":1560,"reported_epoch":686,"state":"active+clean","last_fresh":"2024-08-19T14:38:03.324938+0000","last_change":"2024-08-19T14:37:59.510774+0000","last_active":"2024-08-19T14:38:03.324938+0000","last_peered":"2024-08-19T14:38:03.324938+0000","last_clean":"2024-08-19T14:38:03.324938+0000","last_became_active":"2024-08-19T14:34:15.718427+0000","last_became_peered":"2024-08-19T14:34:15.718427+0000","last_unstale":"2024-08-19T14:38:03.324938+0000","last_undegraded":"2024-08-19T14:38:03.324938+0000","last_fullsized":"2024-08-19T14:38:03.324938+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:37:59.510697+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:37:59.510697+0000","last_clean_scrub_stamp":"2024-08-19T14:37:59.510697+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:39:04.057695+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.18","version":"70'139","reported_seq":1122,"reported_epoch":686,"state":"active+clean","last_fresh":"2024-08-19T14:38:03.304608+0000","last_change":"2024-08-19T14:35:11.528963+0000","last_active":"2024-08-19T14:38:03.304608+0000","last_peered":"2024-08-19T14:38:03.304608+0000","last_clean":"2024-08-19T14:38:03.304608+0000","last_became_active":"2024-08-19T14:35:11.528530+0000","last_became_peered":"2024-08-19T14:35:11.528530+0000","last_unstale":"2024-08-19T14:38:03.304608+0000","last_undegraded":"2024-08-19T14:38:03.304608+0000","last_fullsized":"2024-08-19T14:38:03.304608+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:29:05.336156+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:27:33.084161+0000","last_clean_scrub_stamp":"2024-08-19T14:29:05.336156+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.19","version":"70'139","reported_seq":1540,"reported_epoch":686,"state":"active+clean","last_fresh":"2024-08-19T14:38:03.302553+0000","last_change":"2024-08-19T14:37:41.671836+0000","last_active":"2024-08-19T14:38:03.302553+0000","last_peered":"2024-08-19T14:38:03.302553+0000","last_clean":"2024-08-19T14:38:03.302553+0000","last_became_active":"2024-08-19T14:18:31.902547+0000","last_became_peered":"2024-08-19T14:18:31.902547+0000","last_unstale":"2024-08-19T14:38:03.302553+0000","last_undegraded":"2024-08-19T14:38:03.302553+0000","last_fullsized":"2024-08-19T14:38:03.302553+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:37:41.671731+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:37:41.671731+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:38:59.596992+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1e","version":"70'139","reported_seq":562,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-08-19T14:37:59.187933+0000","last_change":"2024-08-19T14:33:17.778415+0000","last_active":"2024-08-19T14:37:59.187933+0000","last_peered":"2024-08-19T14:37:59.187933+0000","last_clean":"2024-08-19T14:37:59.187933+0000","last_became_active":"2024-08-19T14:30:47.909974+0000","last_became_peered":"2024-08-19T14:30:47.909974+0000","last_unstale":"2024-08-19T14:37:59.187933+0000","last_undegraded":"2024-08-19T14:37:59.187933+0000","last_fullsized":"2024-08-19T14:37:59.187933+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:33:17.778295+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:27:04.780939+0000","last_clean_scrub_stamp":"2024-08-19T14:33:17.778295+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1c","version":"70'139","reported_seq":1314,"reported_epoch":686,"state":"active+clean","last_fresh":"2024-08-19T14:38:03.325026+0000","last_change":"2024-08-19T14:38:01.463630+0000","last_active":"2024-08-19T14:38:03.325026+0000","last_peered":"2024-08-19T14:38:03.325026+0000","last_clean":"2024-08-19T14:38:03.325026+0000","last_became_active":"2024-08-19T14:30:47.908866+0000","last_became_peered":"2024-08-19T14:30:47.908866+0000","last_unstale":"2024-08-19T14:38:03.325026+0000","last_undegraded":"2024-08-19T14:38:03.325026+0000","last_fullsized":"2024-08-19T14:38:03.325026+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:38:01.463524+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:38:01.463524+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:39:18.496929+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,11],"acting":[8,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1d","version":"70'139","reported_seq":1221,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-08-19T14:37:59.200352+0000","last_change":"2024-08-19T14:35:11.542467+0000","last_active":"2024-08-19T14:37:59.200352+0000","last_peered":"2024-08-19T14:37:59.200352+0000","last_clean":"2024-08-19T14:37:59.200352+0000","last_became_active":"2024-08-19T14:35:11.540488+0000","last_became_peered":"2024-08-19T14:35:11.540488+0000","last_unstale":"2024-08-19T14:37:59.200352+0000","last_undegraded":"2024-08-19T14:37:59.200352+0000","last_fullsized":"2024-08-19T14:37:59.200352+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:29.194639+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:28:29.194639+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":2977693720,"num_objects":45437,"num_object_clones":0,"num_object_copies":136311,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17423,"num_objects_misplaced":5385,"num_objects_unfound":0,"num_objects_dirty":45437,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190879,"num_write_kb":7562049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40854,"num_bytes_recovered":2668691456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8791535616,"data_stored":8791523400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160707,"ondisk_log_size":160707,"up":48,"acting":42,"num_store_stats":10},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":856608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187776,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1819628,"kb_used_data":954084,"kb_used_omap":1,"kb_used_meta":865534,"kb_avail":91925524,"statfs":{"total":95995035648,"available":94131736576,"internally_reserved":0,"allocated":976982016,"data_stored":973951241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":886307268},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":2314987372644,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":74,"seq":317827580234,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1734764,"kb_used_data":905316,"kb_used_omap":1,"kb_used_meta":829438,"kb_avail":92010388,"statfs":{"total":95995035648,"available":94218637312,"internally_reserved":0,"allocated":927043584,"data_stored":924012809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":849344961},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874577,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1315260,"kb_used_data":548272,"kb_used_omap":1,"kb_used_meta":766974,"kb_avail":92429892,"statfs":{"total":95995035648,"available":94648209408,"internally_reserved":0,"allocated":561430528,"data_stored":558395755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":785381829},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":57,"seq":244813136212,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1492228,"kb_used_data":562808,"kb_used_omap":1,"kb_used_meta":929406,"kb_avail":92252924,"statfs":{"total":95995035648,"available":94466994176,"internally_reserved":0,"allocated":576315392,"data_stored":573277067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":951712202},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":206158430551,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692186,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1494264,"kb_used_data":560684,"kb_used_omap":1,"kb_used_meta":933566,"kb_avail":92250888,"statfs":{"total":95995035648,"available":94464909312,"internally_reserved":0,"allocated":574140416,"data_stored":571106089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":955972037},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986527,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1706616,"kb_used_data":741104,"kb_used_omap":1,"kb_used_meta":965502,"kb_avail":92038536,"statfs":{"total":95995035648,"available":94247460864,"internally_reserved":0,"allocated":758890496,"data_stored":755855723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":988674501},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150054,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2894012,"kb_used_data":1910452,"kb_used_omap":1,"kb_used_meta":983550,"kb_avail":90851140,"statfs":{"total":95995035648,"available":93031567360,"internally_reserved":0,"allocated":1956302848,"data_stored":1953264003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1007155655},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,0,1,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":634,"seq":2723009265700,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":546756,"kb_used_data":330040,"kb_used_omap":4,"kb_used_meta":216699,"kb_avail":93198396,"statfs":{"total":95995035648,"available":95435157504,"internally_reserved":0,"allocated":337960960,"data_stored":334922635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4407,"internal_metadata":221900489},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575214,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2198256,"kb_used_data":1106408,"kb_used_omap":1,"kb_used_meta":1091838,"kb_avail":91546896,"statfs":{"total":95995035648,"available":93744021504,"internally_reserved":0,"allocated":1132961792,"data_stored":1129926945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1118042565},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,0,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738739,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1945152,"kb_used_data":1093492,"kb_used_omap":1,"kb_used_meta":851646,"kb_avail":91800000,"statfs":{"total":95995035648,"available":94003200000,"internally_reserved":0,"allocated":1119735808,"data_stored":1116696963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":872085957},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1102123008,"data_stored":1102118936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1113722880,"data_stored":1113718808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":332070912,"data_stored":332070912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1945702400,"data_stored":1945698328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":750321664,"data_stored":750321664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":560922624,"data_stored":560922624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":560529408,"data_stored":560529408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":547946496,"data_stored":547946496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":917635072,"data_stored":917635072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":960561152,"data_stored":960561152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T14:38:07.677 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph status --format=json 2024-08-19T14:38:07.699 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:07.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:07.700 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:38:07.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:07 smithi032 ceph-mon[20004]: pgmap v1632: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 2.8 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 474 op/s; 17423/136317 objects degraded (12.781%); 5385/136317 objects misplaced (3.950%); 12 MiB/s, 199 objects/s recovering 2024-08-19T14:38:07.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:07 smithi032 ceph-mon[20004]: 7.4 scrub ok 2024-08-19T14:38:07.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:07 smithi032 ceph-mon[20004]: from='client.17211 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:38:07.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:07 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3608425054' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:38:07.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:07 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:38:07.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:07 smithi045 ceph-mon[20143]: pgmap v1632: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 2.8 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 474 op/s; 17423/136317 objects degraded (12.781%); 5385/136317 objects misplaced (3.950%); 12 MiB/s, 199 objects/s recovering 2024-08-19T14:38:07.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:07 smithi045 ceph-mon[20143]: 7.4 scrub ok 2024-08-19T14:38:07.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:07 smithi045 ceph-mon[20143]: from='client.17211 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:38:07.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:07 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3608425054' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:38:07.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:07 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:38:07.802 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:38:07.871 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:07.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:07.872 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:38:07.973 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:38:07.995 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:07.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:07.996 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:38:08.097 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:38:08.119 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:08.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:08.120 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:38:08.221 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:38:08.271 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:08.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:08.272 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:38:08.374 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:38:08.398 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:08.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:08.399 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:38:08.431 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-08-19T14:38:08.500 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:38:08.536 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:08.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:08.538 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:38:08.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:08 smithi031 ceph-mon[16336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:38:08.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:08 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:38:08.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:08 smithi031 ceph-mon[16336]: osdmap e687: 12 total, 12 up, 10 in 2024-08-19T14:38:08.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:08 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 17423/136317 objects degraded (12.781%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-19T14:38:08.639 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:38:08.662 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:08.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:08.663 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:38:08.716 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:38:08.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:08 smithi032 ceph-mon[20004]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:38:08.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:08 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:38:08.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:08 smithi032 ceph-mon[20004]: osdmap e687: 12 total, 12 up, 10 in 2024-08-19T14:38:08.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:08 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 17423/136317 objects degraded (12.781%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-19T14:38:08.764 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:38:08.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:08 smithi045 ceph-mon[20143]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:38:08.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:08 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:38:08.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:08 smithi045 ceph-mon[20143]: osdmap e687: 12 total, 12 up, 10 in 2024-08-19T14:38:08.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:08 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 17423/136317 objects degraded (12.781%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-19T14:38:08.811 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:08.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:08.812 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:38:08.913 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:38:08.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:08.941 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:08.941 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:38:09.042 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:38:09.065 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:09.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:09.066 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:38:09.167 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:38:09.190 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:09.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:09.191 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:38:09.234 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:38:09.234 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-08-19T14:38:09.234 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-08-19T14:38:09.234 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-08-19T14:38:09.234 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.234 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26191.0:124554 6.8 6.b3d98118 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e555)", 2024-08-19T14:38:09.234 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:30:43.948816+0000", 2024-08-19T14:38:09.235 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 445.27553360500002, 2024-08-19T14:38:09.235 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.95529832699999995, 2024-08-19T14:38:09.235 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:38:09.235 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:38:09.235 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:38:09.235 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:38:09.235 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26191", 2024-08-19T14:38:09.235 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:38:09.235 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 124554 2024-08-19T14:38:09.235 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.235 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:38:09.235 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.235 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:38:09.236 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:43.948816+0000", 2024-08-19T14:38:09.236 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:38:09.236 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.236 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.236 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:38:09.236 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:43.948816+0000", 2024-08-19T14:38:09.236 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:38:09.236 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.236 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.236 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:38:09.236 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:43.948817+0000", 2024-08-19T14:38:09.236 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.252e-06 2024-08-19T14:38:09.236 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.236 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.237 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:38:09.237 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:43.948823+0000", 2024-08-19T14:38:09.237 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1309999999999997e-06 2024-08-19T14:38:09.237 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.237 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.237 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:38:09.237 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:43.948833+0000", 2024-08-19T14:38:09.237 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0886e-05 2024-08-19T14:38:09.237 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.237 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.237 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:38:09.237 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:43.948841+0000", 2024-08-19T14:38:09.237 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.8320000000000006e-06 2024-08-19T14:38:09.238 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.238 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.238 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:38:09.238 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:43.948915+0000", 2024-08-19T14:38:09.238 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3396e-05 2024-08-19T14:38:09.238 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.238 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.238 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:38:09.238 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:43.948924+0000", 2024-08-19T14:38:09.238 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.8740000000000007e-06 2024-08-19T14:38:09.238 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.238 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.238 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:38:09.238 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:44.904085+0000", 2024-08-19T14:38:09.239 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.95516005900000001 2024-08-19T14:38:09.239 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.239 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.239 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:38:09.239 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:44.904114+0000", 2024-08-19T14:38:09.239 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9896999999999999e-05 2024-08-19T14:38:09.239 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:38:09.239 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:38:09.239 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:38:09.239 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.239 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.239 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26191.0:124571 6.8 6.e5f0dfd8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e555)", 2024-08-19T14:38:09.239 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:30:43.990212+0000", 2024-08-19T14:38:09.239 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 445.23413730200002, 2024-08-19T14:38:09.240 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.91398457099999997, 2024-08-19T14:38:09.240 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:38:09.240 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:38:09.240 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:38:09.240 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:38:09.240 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26191", 2024-08-19T14:38:09.240 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:38:09.240 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 124571 2024-08-19T14:38:09.240 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.240 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:38:09.240 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.240 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:38:09.240 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:43.990212+0000", 2024-08-19T14:38:09.240 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:38:09.241 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.241 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.241 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:38:09.241 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:43.990212+0000", 2024-08-19T14:38:09.241 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:38:09.241 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.241 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.241 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:38:09.241 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:43.990214+0000", 2024-08-19T14:38:09.241 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.508e-06 2024-08-19T14:38:09.241 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.241 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.241 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:38:09.241 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:43.990221+0000", 2024-08-19T14:38:09.242 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9149999999999999e-06 2024-08-19T14:38:09.242 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.242 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.242 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:38:09.242 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:43.990236+0000", 2024-08-19T14:38:09.242 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4978e-05 2024-08-19T14:38:09.242 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.242 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.242 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:38:09.242 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:43.990244+0000", 2024-08-19T14:38:09.242 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.4479999999999994e-06 2024-08-19T14:38:09.242 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.242 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.242 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:38:09.243 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:43.990304+0000", 2024-08-19T14:38:09.243 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9757999999999999e-05 2024-08-19T14:38:09.243 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.243 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.243 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:38:09.243 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:43.990313+0000", 2024-08-19T14:38:09.243 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3710000000000006e-06 2024-08-19T14:38:09.243 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.243 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.243 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:38:09.243 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:44.904189+0000", 2024-08-19T14:38:09.243 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.91387590100000005 2024-08-19T14:38:09.243 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.243 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.244 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:38:09.244 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:44.904197+0000", 2024-08-19T14:38:09.244 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6920000000000002e-06 2024-08-19T14:38:09.244 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:38:09.244 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:38:09.244 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:38:09.244 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.244 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.244 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26191.0:124580 6.8 6.2fef8638 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e555)", 2024-08-19T14:38:09.244 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:30:44.009826+0000", 2024-08-19T14:38:09.244 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 445.21452324299997, 2024-08-19T14:38:09.244 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.89440363899999997, 2024-08-19T14:38:09.244 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:38:09.244 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:38:09.244 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:38:09.245 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:38:09.245 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26191", 2024-08-19T14:38:09.245 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:38:09.245 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 124580 2024-08-19T14:38:09.245 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.245 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:38:09.245 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.245 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:38:09.245 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:44.009826+0000", 2024-08-19T14:38:09.245 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:38:09.245 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.245 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.245 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:38:09.245 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:44.009826+0000", 2024-08-19T14:38:09.245 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:38:09.246 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.246 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.246 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:38:09.246 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:44.009828+0000", 2024-08-19T14:38:09.246 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6279999999999999e-06 2024-08-19T14:38:09.246 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.246 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.246 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:38:09.246 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:44.009839+0000", 2024-08-19T14:38:09.246 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1113e-05 2024-08-19T14:38:09.246 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.246 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.246 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:38:09.246 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:44.009856+0000", 2024-08-19T14:38:09.247 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6752000000000002e-05 2024-08-19T14:38:09.247 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.247 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.247 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:38:09.247 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:44.009865+0000", 2024-08-19T14:38:09.247 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.0869999999999996e-06 2024-08-19T14:38:09.247 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.247 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.247 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:38:09.247 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:44.009926+0000", 2024-08-19T14:38:09.247 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0501999999999998e-05 2024-08-19T14:38:09.247 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.247 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.247 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:38:09.248 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:44.009935+0000", 2024-08-19T14:38:09.248 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.9609999999999994e-06 2024-08-19T14:38:09.248 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.248 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.248 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:38:09.248 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:44.904224+0000", 2024-08-19T14:38:09.248 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.89428911799999999 2024-08-19T14:38:09.248 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.248 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.248 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:38:09.248 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:44.904230+0000", 2024-08-19T14:38:09.248 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4779999999999996e-06 2024-08-19T14:38:09.248 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:38:09.248 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:38:09.249 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:38:09.249 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.249 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.249 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26191.0:124594 6.8 6.3e87bdf8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e555)", 2024-08-19T14:38:09.249 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:30:44.038535+0000", 2024-08-19T14:38:09.249 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 445.18581458, 2024-08-19T14:38:09.249 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.86572874600000005, 2024-08-19T14:38:09.249 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:38:09.249 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:38:09.249 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:38:09.249 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:38:09.249 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26191", 2024-08-19T14:38:09.249 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:38:09.249 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 124594 2024-08-19T14:38:09.250 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.250 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:38:09.250 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.250 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:38:09.250 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:44.038535+0000", 2024-08-19T14:38:09.250 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:38:09.250 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.250 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.250 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:38:09.250 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:44.038535+0000", 2024-08-19T14:38:09.250 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:38:09.250 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.250 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.250 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:38:09.251 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:44.038537+0000", 2024-08-19T14:38:09.251 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4079999999999999e-06 2024-08-19T14:38:09.251 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.251 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.251 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:38:09.251 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:44.038541+0000", 2024-08-19T14:38:09.251 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.882e-06 2024-08-19T14:38:09.251 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.251 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.251 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:38:09.251 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:44.038553+0000", 2024-08-19T14:38:09.251 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2063e-05 2024-08-19T14:38:09.251 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.251 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.252 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:38:09.252 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:44.038561+0000", 2024-08-19T14:38:09.252 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.9359999999999999e-06 2024-08-19T14:38:09.252 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.252 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.252 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:38:09.252 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:44.038623+0000", 2024-08-19T14:38:09.252 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1766000000000005e-05 2024-08-19T14:38:09.252 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.252 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.252 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:38:09.252 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:44.038637+0000", 2024-08-19T14:38:09.252 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3651e-05 2024-08-19T14:38:09.252 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.253 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.253 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:38:09.253 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:44.904257+0000", 2024-08-19T14:38:09.253 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.86562061499999998 2024-08-19T14:38:09.253 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.253 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.253 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:38:09.253 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:44.904264+0000", 2024-08-19T14:38:09.253 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4250000000000003e-06 2024-08-19T14:38:09.253 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:38:09.253 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:38:09.253 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:38:09.253 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.253 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.253 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26191.0:124606 6.8 6.a7971958 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e555)", 2024-08-19T14:38:09.254 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:30:44.059903+0000", 2024-08-19T14:38:09.254 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 445.16444706099998, 2024-08-19T14:38:09.254 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.84439685900000006, 2024-08-19T14:38:09.254 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:38:09.254 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:38:09.254 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:38:09.254 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:38:09.254 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26191", 2024-08-19T14:38:09.254 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:38:09.254 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 124606 2024-08-19T14:38:09.254 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.254 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:38:09.254 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.254 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:38:09.255 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:44.059903+0000", 2024-08-19T14:38:09.255 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:38:09.255 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.255 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.255 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:38:09.255 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:44.059903+0000", 2024-08-19T14:38:09.255 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:38:09.255 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.255 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.255 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:38:09.255 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:44.059904+0000", 2024-08-19T14:38:09.255 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.488e-06 2024-08-19T14:38:09.255 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.255 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.255 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:38:09.256 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:44.059910+0000", 2024-08-19T14:38:09.256 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.947e-06 2024-08-19T14:38:09.256 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.256 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.256 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:38:09.256 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:44.059922+0000", 2024-08-19T14:38:09.256 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2381999999999999e-05 2024-08-19T14:38:09.256 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.256 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.256 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:38:09.256 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:44.059930+0000", 2024-08-19T14:38:09.256 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5089999999999996e-06 2024-08-19T14:38:09.256 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.256 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.257 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:38:09.257 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:44.060000+0000", 2024-08-19T14:38:09.257 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9993000000000006e-05 2024-08-19T14:38:09.257 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.257 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.257 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:38:09.257 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:44.060011+0000", 2024-08-19T14:38:09.257 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1469e-05 2024-08-19T14:38:09.257 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.257 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.257 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:38:09.257 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:44.904294+0000", 2024-08-19T14:38:09.257 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.84428247099999998 2024-08-19T14:38:09.257 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.257 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.258 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:38:09.258 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:44.904300+0000", 2024-08-19T14:38:09.258 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5999999999999997e-06 2024-08-19T14:38:09.258 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:38:09.258 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:38:09.258 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:38:09.258 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.258 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.258 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26191.0:124625 6.8 6.c84f22e8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e555)", 2024-08-19T14:38:09.258 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:30:44.166898+0000", 2024-08-19T14:38:09.258 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 445.05745198599999, 2024-08-19T14:38:09.258 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.738722553, 2024-08-19T14:38:09.258 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:38:09.258 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:38:09.259 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:38:09.259 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:38:09.259 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26191", 2024-08-19T14:38:09.259 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:38:09.259 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 124625 2024-08-19T14:38:09.259 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.259 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:38:09.259 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.259 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:38:09.259 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:44.166898+0000", 2024-08-19T14:38:09.259 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:38:09.259 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.259 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.259 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:38:09.259 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:44.166898+0000", 2024-08-19T14:38:09.260 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:38:09.260 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.260 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.260 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:38:09.260 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:44.166899+0000", 2024-08-19T14:38:09.260 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.536e-06 2024-08-19T14:38:09.260 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.260 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.260 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:38:09.260 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:44.166905+0000", 2024-08-19T14:38:09.260 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8900000000000004e-06 2024-08-19T14:38:09.260 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.260 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.260 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:38:09.261 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:44.166918+0000", 2024-08-19T14:38:09.261 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2658e-05 2024-08-19T14:38:09.261 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.261 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.261 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:38:09.261 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:44.166925+0000", 2024-08-19T14:38:09.261 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6199999999999999e-06 2024-08-19T14:38:09.261 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.261 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.261 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:38:09.261 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:44.166991+0000", 2024-08-19T14:38:09.261 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.5537000000000006e-05 2024-08-19T14:38:09.261 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.261 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.261 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:38:09.262 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:44.166999+0000", 2024-08-19T14:38:09.262 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.4999999999999999e-06 2024-08-19T14:38:09.262 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.262 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.262 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:38:09.262 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:44.905612+0000", 2024-08-19T14:38:09.262 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.73861242199999999 2024-08-19T14:38:09.262 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.262 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.262 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:38:09.262 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:44.905620+0000", 2024-08-19T14:38:09.262 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.3899999999999993e-06 2024-08-19T14:38:09.262 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:38:09.262 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:38:09.263 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:38:09.263 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.263 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.263 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26191.0:124603 6.3 6:cbb85d3f:::benchmark_data_smithi045_97937_object17330:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e556)", 2024-08-19T14:38:09.263 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:30:45.948113+0000", 2024-08-19T14:38:09.263 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 443.27623696400002, 2024-08-19T14:38:09.263 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.32293990299999997, 2024-08-19T14:38:09.263 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:38:09.263 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:38:09.263 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:38:09.263 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:38:09.263 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26191", 2024-08-19T14:38:09.263 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:38:09.263 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 124603 2024-08-19T14:38:09.264 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.264 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:38:09.264 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.264 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:38:09.264 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.948113+0000", 2024-08-19T14:38:09.264 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:38:09.264 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.264 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.264 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:38:09.264 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.948113+0000", 2024-08-19T14:38:09.264 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:38:09.264 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.264 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.264 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:38:09.265 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.948113+0000", 2024-08-19T14:38:09.265 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2e-07 2024-08-19T14:38:09.265 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.265 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.265 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:38:09.265 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.948116+0000", 2024-08-19T14:38:09.265 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4980000000000001e-06 2024-08-19T14:38:09.265 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.265 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.265 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:38:09.265 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.948122+0000", 2024-08-19T14:38:09.265 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.297e-06 2024-08-19T14:38:09.265 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.265 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.265 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:38:09.266 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.948127+0000", 2024-08-19T14:38:09.266 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1499999999999998e-06 2024-08-19T14:38:09.266 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.266 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.266 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:38:09.266 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.951637+0000", 2024-08-19T14:38:09.266 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0035102319999999998 2024-08-19T14:38:09.266 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.266 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.266 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:38:09.266 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.951645+0000", 2024-08-19T14:38:09.266 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.165e-06 2024-08-19T14:38:09.266 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.267 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.267 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:38:09.267 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.205514+0000", 2024-08-19T14:38:09.267 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.25386919899999999 2024-08-19T14:38:09.267 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.267 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.267 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:38:09.267 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.205684+0000", 2024-08-19T14:38:09.267 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00017037399999999999 2024-08-19T14:38:09.267 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.267 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.267 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 8,11", 2024-08-19T14:38:09.267 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.205849+0000", 2024-08-19T14:38:09.267 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00016521400000000001 2024-08-19T14:38:09.268 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.268 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.268 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:38:09.268 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.213228+0000", 2024-08-19T14:38:09.268 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0073782439999999999 2024-08-19T14:38:09.268 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.268 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.268 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:38:09.268 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.217650+0000", 2024-08-19T14:38:09.268 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0044225009999999997 2024-08-19T14:38:09.268 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.268 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.268 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:38:09.268 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.265586+0000", 2024-08-19T14:38:09.268 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.047936421 2024-08-19T14:38:09.269 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.269 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.269 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:38:09.269 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.265693+0000", 2024-08-19T14:38:09.269 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00010662099999999999 2024-08-19T14:38:09.269 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.269 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.269 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:38:09.269 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.271053+0000", 2024-08-19T14:38:09.269 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.005359567 2024-08-19T14:38:09.269 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:38:09.269 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:38:09.269 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:38:09.269 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.270 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.270 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26191.0:124622 6.3 6:cf0d0516:::benchmark_data_smithi045_97937_object17349:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e556)", 2024-08-19T14:38:09.270 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:30:45.948133+0000", 2024-08-19T14:38:09.270 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 443.27621662199999, 2024-08-19T14:38:09.270 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.327898566, 2024-08-19T14:38:09.270 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:38:09.270 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:38:09.270 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:38:09.270 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:38:09.270 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26191", 2024-08-19T14:38:09.270 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:38:09.270 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 124622 2024-08-19T14:38:09.270 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.270 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:38:09.271 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.271 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:38:09.271 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.948133+0000", 2024-08-19T14:38:09.271 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:38:09.271 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.271 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.271 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:38:09.271 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.948133+0000", 2024-08-19T14:38:09.271 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:38:09.271 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.271 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.271 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:38:09.271 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.948133+0000", 2024-08-19T14:38:09.271 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.84e-07 2024-08-19T14:38:09.271 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.272 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.272 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:38:09.272 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.948136+0000", 2024-08-19T14:38:09.272 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3140000000000002e-06 2024-08-19T14:38:09.272 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.272 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.273 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:38:09.273 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.948142+0000", 2024-08-19T14:38:09.273 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9279999999999998e-06 2024-08-19T14:38:09.274 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.274 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.274 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:38:09.274 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.948148+0000", 2024-08-19T14:38:09.274 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8109999999999999e-06 2024-08-19T14:38:09.274 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.274 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.274 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:38:09.274 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.951687+0000", 2024-08-19T14:38:09.274 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0035393669999999999 2024-08-19T14:38:09.274 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.274 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.274 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:38:09.274 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.951697+0000", 2024-08-19T14:38:09.275 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0471e-05 2024-08-19T14:38:09.275 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.275 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.275 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:38:09.275 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.209143+0000", 2024-08-19T14:38:09.275 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.257445596 2024-08-19T14:38:09.275 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.275 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.275 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:38:09.275 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.209302+0000", 2024-08-19T14:38:09.275 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00015897799999999999 2024-08-19T14:38:09.275 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.275 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.275 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 8,11", 2024-08-19T14:38:09.276 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.209465+0000", 2024-08-19T14:38:09.276 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00016353400000000001 2024-08-19T14:38:09.276 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.276 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.276 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:38:09.276 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.222533+0000", 2024-08-19T14:38:09.276 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.013067466999999999 2024-08-19T14:38:09.276 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.276 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.276 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:38:09.276 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.222549+0000", 2024-08-19T14:38:09.276 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6079999999999999e-05 2024-08-19T14:38:09.276 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.276 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.277 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:38:09.277 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.271113+0000", 2024-08-19T14:38:09.277 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.048563518999999999 2024-08-19T14:38:09.278 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.278 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.278 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:38:09.278 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.271178+0000", 2024-08-19T14:38:09.278 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.5464999999999997e-05 2024-08-19T14:38:09.278 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.279 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.279 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:38:09.279 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.276032+0000", 2024-08-19T14:38:09.279 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.004853652 2024-08-19T14:38:09.279 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:38:09.279 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:38:09.279 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:38:09.279 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.279 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.279 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26191.0:124632 6.3 6:c97ec9d6:::benchmark_data_smithi045_97937_object17359:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e556)", 2024-08-19T14:38:09.279 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:30:45.948155+0000", 2024-08-19T14:38:09.279 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 443.27619508499998, 2024-08-19T14:38:09.279 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.33379947399999998, 2024-08-19T14:38:09.279 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:38:09.280 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:38:09.280 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:38:09.280 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:38:09.280 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26191", 2024-08-19T14:38:09.280 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:38:09.280 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 124632 2024-08-19T14:38:09.280 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.280 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:38:09.280 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.280 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:38:09.280 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.948155+0000", 2024-08-19T14:38:09.280 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:38:09.280 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.280 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.281 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:38:09.281 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.948155+0000", 2024-08-19T14:38:09.281 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:38:09.281 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.281 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.281 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:38:09.281 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.948155+0000", 2024-08-19T14:38:09.281 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9799999999999999e-07 2024-08-19T14:38:09.281 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.281 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.281 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:38:09.281 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.948157+0000", 2024-08-19T14:38:09.281 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2730000000000001e-06 2024-08-19T14:38:09.281 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.281 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.281 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:38:09.282 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.948164+0000", 2024-08-19T14:38:09.282 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3300000000000004e-06 2024-08-19T14:38:09.282 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.282 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.282 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:38:09.282 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.948168+0000", 2024-08-19T14:38:09.282 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8099999999999997e-06 2024-08-19T14:38:09.282 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.282 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.282 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:38:09.282 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.951734+0000", 2024-08-19T14:38:09.282 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.003565085 2024-08-19T14:38:09.282 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.283 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.283 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:38:09.283 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:45.951740+0000", 2024-08-19T14:38:09.283 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1770000000000003e-06 2024-08-19T14:38:09.283 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.283 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.283 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:38:09.283 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.227097+0000", 2024-08-19T14:38:09.283 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.27535768399999999 2024-08-19T14:38:09.283 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.283 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.283 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:38:09.283 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.227257+0000", 2024-08-19T14:38:09.283 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00015980399999999999 2024-08-19T14:38:09.284 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.284 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.284 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 8,11", 2024-08-19T14:38:09.284 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.227420+0000", 2024-08-19T14:38:09.284 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00016232599999999999 2024-08-19T14:38:09.284 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.284 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.284 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:38:09.284 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.253743+0000", 2024-08-19T14:38:09.284 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.026323053999999999 2024-08-19T14:38:09.284 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.284 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.284 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:38:09.284 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.253755+0000", 2024-08-19T14:38:09.285 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2296000000000001e-05 2024-08-19T14:38:09.285 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.285 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.285 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:38:09.285 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.276102+0000", 2024-08-19T14:38:09.285 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.022346774999999999 2024-08-19T14:38:09.285 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.285 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.285 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:38:09.285 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.276358+0000", 2024-08-19T14:38:09.285 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000256245 2024-08-19T14:38:09.285 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.285 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.285 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:38:09.285 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:30:46.281954+0000", 2024-08-19T14:38:09.285 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0055962169999999997 2024-08-19T14:38:09.286 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:38:09.286 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:38:09.286 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:38:09.286 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.286 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.286 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26191.0:168503 6.8 6:10e97d8a:::benchmark_data_smithi045_97937_object61230:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e577)", 2024-08-19T14:38:09.286 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:32:04.492777+0000", 2024-08-19T14:38:09.286 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 364.731572395, 2024-08-19T14:38:09.286 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1376310749999998, 2024-08-19T14:38:09.286 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:38:09.286 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:38:09.286 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:38:09.286 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:38:09.286 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26191", 2024-08-19T14:38:09.287 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:38:09.287 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 168503 2024-08-19T14:38:09.287 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.287 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:38:09.287 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.287 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:38:09.287 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:04.492777+0000", 2024-08-19T14:38:09.287 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:38:09.287 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.287 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.287 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:38:09.287 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:04.492777+0000", 2024-08-19T14:38:09.287 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:38:09.287 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.288 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.288 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:38:09.288 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:04.492778+0000", 2024-08-19T14:38:09.288 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7300000000000002e-07 2024-08-19T14:38:09.288 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.288 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.288 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:38:09.288 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:04.492780+0000", 2024-08-19T14:38:09.288 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9199999999999998e-06 2024-08-19T14:38:09.288 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.288 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.288 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:38:09.288 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:04.492786+0000", 2024-08-19T14:38:09.289 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.302e-06 2024-08-19T14:38:09.289 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.289 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.289 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:38:09.289 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:04.492791+0000", 2024-08-19T14:38:09.289 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.69e-06 2024-08-19T14:38:09.289 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.289 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.289 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:38:09.289 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:04.494902+0000", 2024-08-19T14:38:09.289 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0021112470000000001 2024-08-19T14:38:09.289 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.289 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.289 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for missing object", 2024-08-19T14:38:09.289 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:04.494953+0000", 2024-08-19T14:38:09.290 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1008999999999999e-05 2024-08-19T14:38:09.290 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.290 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.290 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:38:09.290 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:07.630397+0000", 2024-08-19T14:38:09.290 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1354437609999999 2024-08-19T14:38:09.290 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.290 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.290 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:38:09.290 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:07.630408+0000", 2024-08-19T14:38:09.290 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1873e-05 2024-08-19T14:38:09.290 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:38:09.290 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:38:09.290 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:38:09.291 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.291 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.291 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26191.0:168508 6.8 6:164fb7a4:::benchmark_data_smithi045_97937_object61235:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e577)", 2024-08-19T14:38:09.291 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:32:04.492815+0000", 2024-08-19T14:38:09.291 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 364.73153493000001, 2024-08-19T14:38:09.291 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1427530990000001, 2024-08-19T14:38:09.291 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:38:09.291 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:38:09.291 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:38:09.291 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:38:09.291 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26191", 2024-08-19T14:38:09.291 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:38:09.291 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 168508 2024-08-19T14:38:09.292 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.292 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:38:09.292 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.292 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:38:09.292 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:04.492815+0000", 2024-08-19T14:38:09.292 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:38:09.292 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.292 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.292 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:38:09.292 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:04.492815+0000", 2024-08-19T14:38:09.292 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:38:09.292 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.292 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.292 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:38:09.293 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:04.492815+0000", 2024-08-19T14:38:09.293 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9799999999999999e-07 2024-08-19T14:38:09.293 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.293 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.293 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:38:09.293 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:04.492820+0000", 2024-08-19T14:38:09.293 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6750000000000001e-06 2024-08-19T14:38:09.293 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.293 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.293 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:38:09.293 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:04.492826+0000", 2024-08-19T14:38:09.293 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3380000000000001e-06 2024-08-19T14:38:09.293 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.293 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.293 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:38:09.294 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:04.492831+0000", 2024-08-19T14:38:09.294 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4619999999999998e-06 2024-08-19T14:38:09.294 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.294 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.294 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:38:09.294 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:04.495042+0000", 2024-08-19T14:38:09.294 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0022109550000000001 2024-08-19T14:38:09.294 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.294 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.294 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for missing object", 2024-08-19T14:38:09.294 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:04.495088+0000", 2024-08-19T14:38:09.294 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6928999999999997e-05 2024-08-19T14:38:09.294 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.294 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.295 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:38:09.295 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:07.635556+0000", 2024-08-19T14:38:09.295 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1404670929999998 2024-08-19T14:38:09.295 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.295 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.295 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:38:09.295 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:32:07.635568+0000", 2024-08-19T14:38:09.295 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2349e-05 2024-08-19T14:38:09.295 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:38:09.295 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:38:09.295 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:38:09.295 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.295 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.295 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.16962.0:122550 7.8 7.1966f758 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e634)", 2024-08-19T14:38:09.296 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:35:10.544722+0000", 2024-08-19T14:38:09.296 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 178.679628159, 2024-08-19T14:38:09.296 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.98226408200000004, 2024-08-19T14:38:09.296 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:38:09.296 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:38:09.296 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:38:09.296 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:38:09.296 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.16962", 2024-08-19T14:38:09.296 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:38:09.296 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 122550 2024-08-19T14:38:09.296 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.296 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:38:09.296 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.296 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:38:09.297 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:10.544722+0000", 2024-08-19T14:38:09.297 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:38:09.297 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.297 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.297 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:38:09.297 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:10.544722+0000", 2024-08-19T14:38:09.297 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:38:09.297 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.297 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.297 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:38:09.297 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:10.544724+0000", 2024-08-19T14:38:09.297 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.204e-06 2024-08-19T14:38:09.297 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.297 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.298 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:38:09.298 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:10.544742+0000", 2024-08-19T14:38:09.298 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8033000000000002e-05 2024-08-19T14:38:09.298 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.298 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.298 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:38:09.298 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:10.544752+0000", 2024-08-19T14:38:09.298 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.9059999999999997e-06 2024-08-19T14:38:09.298 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.298 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.298 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:38:09.298 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:10.544757+0000", 2024-08-19T14:38:09.298 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3310000000000003e-06 2024-08-19T14:38:09.298 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.299 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.299 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:38:09.299 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:10.544803+0000", 2024-08-19T14:38:09.299 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6122999999999999e-05 2024-08-19T14:38:09.299 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.299 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.299 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:38:09.299 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:10.544811+0000", 2024-08-19T14:38:09.299 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4909999999999999e-06 2024-08-19T14:38:09.299 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.299 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.299 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:38:09.299 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:11.526849+0000", 2024-08-19T14:38:09.299 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.982038093 2024-08-19T14:38:09.300 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.300 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.300 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:38:09.300 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:11.526986+0000", 2024-08-19T14:38:09.300 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013690100000000001 2024-08-19T14:38:09.300 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:38:09.300 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:38:09.300 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:38:09.300 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.300 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.300 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.16962.0:122554 7.8 7.cc6d1008 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e634)", 2024-08-19T14:38:09.300 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:35:10.549429+0000", 2024-08-19T14:38:09.300 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 178.674920587, 2024-08-19T14:38:09.300 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.97759647800000005, 2024-08-19T14:38:09.301 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:38:09.301 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:38:09.301 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:38:09.301 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:38:09.301 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.16962", 2024-08-19T14:38:09.301 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:38:09.301 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 122554 2024-08-19T14:38:09.301 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.301 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:38:09.301 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.301 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:38:09.301 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:10.549429+0000", 2024-08-19T14:38:09.301 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:38:09.301 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.302 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.302 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:38:09.302 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:10.549429+0000", 2024-08-19T14:38:09.302 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:38:09.302 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.302 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.302 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:38:09.302 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:10.549430+0000", 2024-08-19T14:38:09.302 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.248e-06 2024-08-19T14:38:09.302 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.302 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.302 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:38:09.302 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:10.549435+0000", 2024-08-19T14:38:09.302 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.296e-06 2024-08-19T14:38:09.303 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.303 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.303 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:38:09.303 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:10.549446+0000", 2024-08-19T14:38:09.303 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1464999999999999e-05 2024-08-19T14:38:09.303 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.303 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.303 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:38:09.303 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:10.549452+0000", 2024-08-19T14:38:09.303 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3080000000000004e-06 2024-08-19T14:38:09.303 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.303 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.303 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:38:09.303 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:10.549514+0000", 2024-08-19T14:38:09.304 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1953999999999997e-05 2024-08-19T14:38:09.304 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.304 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.304 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:38:09.304 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:10.549525+0000", 2024-08-19T14:38:09.304 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0135e-05 2024-08-19T14:38:09.304 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.304 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.304 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:38:09.304 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:11.527016+0000", 2024-08-19T14:38:09.304 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.97749134800000004 2024-08-19T14:38:09.304 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.304 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.304 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:38:09.305 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:11.527026+0000", 2024-08-19T14:38:09.305 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.7240000000000004e-06 2024-08-19T14:38:09.305 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:38:09.305 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:38:09.305 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:38:09.305 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.305 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.305 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.16962.0:122576 7.8 7.326ac1d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e634)", 2024-08-19T14:38:09.305 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:35:10.593824+0000", 2024-08-19T14:38:09.305 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 178.630526035, 2024-08-19T14:38:09.305 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.93323670000000003, 2024-08-19T14:38:09.305 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:38:09.305 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:38:09.305 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:38:09.305 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:38:09.306 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.16962", 2024-08-19T14:38:09.306 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:38:09.306 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 122576 2024-08-19T14:38:09.306 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.306 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:38:09.306 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.306 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:38:09.306 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:10.593824+0000", 2024-08-19T14:38:09.306 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:38:09.306 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.306 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.306 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:38:09.306 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:10.593824+0000", 2024-08-19T14:38:09.306 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:38:09.307 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.307 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.307 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:38:09.307 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:10.593826+0000", 2024-08-19T14:38:09.307 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3070000000000001e-06 2024-08-19T14:38:09.307 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.307 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.307 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:38:09.307 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:10.593838+0000", 2024-08-19T14:38:09.307 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1846e-05 2024-08-19T14:38:09.307 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.307 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.307 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:38:09.307 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:10.593871+0000", 2024-08-19T14:38:09.308 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3166000000000001e-05 2024-08-19T14:38:09.308 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.308 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.308 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:38:09.308 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:10.593887+0000", 2024-08-19T14:38:09.308 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5736999999999999e-05 2024-08-19T14:38:09.308 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.308 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.308 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:38:09.308 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:10.594038+0000", 2024-08-19T14:38:09.308 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00015118 2024-08-19T14:38:09.308 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.308 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.308 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:38:09.308 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:10.594059+0000", 2024-08-19T14:38:09.309 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0817000000000001e-05 2024-08-19T14:38:09.309 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.309 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.309 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:38:09.309 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:11.527053+0000", 2024-08-19T14:38:09.309 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.93299383700000005 2024-08-19T14:38:09.309 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.309 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.309 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:38:09.309 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:11.527060+0000", 2024-08-19T14:38:09.309 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.8099999999999998e-06 2024-08-19T14:38:09.309 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:38:09.309 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:38:09.309 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:38:09.310 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.310 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.310 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.16962.0:122578 7.8 7.ea8a5548 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e634)", 2024-08-19T14:38:09.310 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:35:10.610705+0000", 2024-08-19T14:38:09.310 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 178.61364438499999, 2024-08-19T14:38:09.310 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.91639136499999996, 2024-08-19T14:38:09.310 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:38:09.310 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:38:09.310 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:38:09.310 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:38:09.310 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.16962", 2024-08-19T14:38:09.310 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:38:09.310 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 122578 2024-08-19T14:38:09.310 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.311 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:38:09.311 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.311 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:38:09.311 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:10.610705+0000", 2024-08-19T14:38:09.311 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:38:09.311 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.311 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.311 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:38:09.311 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:10.610705+0000", 2024-08-19T14:38:09.311 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:38:09.311 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.311 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.311 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:38:09.311 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:10.610707+0000", 2024-08-19T14:38:09.312 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6929999999999999e-06 2024-08-19T14:38:09.312 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.312 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.312 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:38:09.312 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:10.610732+0000", 2024-08-19T14:38:09.312 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5409999999999999e-05 2024-08-19T14:38:09.312 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.312 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.312 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:38:09.312 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:10.610760+0000", 2024-08-19T14:38:09.312 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7849000000000001e-05 2024-08-19T14:38:09.312 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.312 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.312 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:38:09.313 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:10.610774+0000", 2024-08-19T14:38:09.313 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3519999999999999e-05 2024-08-19T14:38:09.313 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.313 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.313 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:38:09.313 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:10.610901+0000", 2024-08-19T14:38:09.313 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000127367 2024-08-19T14:38:09.313 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.313 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.313 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:38:09.313 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:10.610908+0000", 2024-08-19T14:38:09.313 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9e-06 2024-08-19T14:38:09.313 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.313 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.314 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:38:09.314 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:11.527089+0000", 2024-08-19T14:38:09.314 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.91618081699999998 2024-08-19T14:38:09.314 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.314 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.314 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:38:09.314 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:11.527097+0000", 2024-08-19T14:38:09.314 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.8089999999999993e-06 2024-08-19T14:38:09.314 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:38:09.314 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:38:09.314 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:38:09.314 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.314 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.314 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.16962.0:122550 7.8 7:1aef6698:::benchmark_data_smithi045_115842_object4390:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e635)", 2024-08-19T14:38:09.315 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:35:12.561381+0000", 2024-08-19T14:38:09.315 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 176.66296828099999, 2024-08-19T14:38:09.315 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.51053788600000005, 2024-08-19T14:38:09.315 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:38:09.315 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:38:09.315 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:38:09.315 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:38:09.315 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.16962", 2024-08-19T14:38:09.315 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:38:09.315 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 122550 2024-08-19T14:38:09.315 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.315 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:38:09.315 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.315 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:38:09.315 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:12.561381+0000", 2024-08-19T14:38:09.316 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:38:09.316 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.316 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.316 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:38:09.316 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:12.561381+0000", 2024-08-19T14:38:09.316 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:38:09.316 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.316 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.316 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:38:09.316 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:12.561384+0000", 2024-08-19T14:38:09.316 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0959999999999999e-06 2024-08-19T14:38:09.316 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.316 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.316 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:38:09.317 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:12.561395+0000", 2024-08-19T14:38:09.317 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1328999999999999e-05 2024-08-19T14:38:09.317 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.317 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.317 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:38:09.317 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:12.561412+0000", 2024-08-19T14:38:09.317 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6986999999999998e-05 2024-08-19T14:38:09.317 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.317 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.317 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:38:09.317 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:12.561423+0000", 2024-08-19T14:38:09.317 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1569000000000001e-05 2024-08-19T14:38:09.317 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.317 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.317 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:38:09.318 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:12.563319+0000", 2024-08-19T14:38:09.318 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0018956209999999999 2024-08-19T14:38:09.318 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.318 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.318 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:38:09.318 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:12.563336+0000", 2024-08-19T14:38:09.318 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6996000000000001e-05 2024-08-19T14:38:09.318 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.318 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.318 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:38:09.318 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:13.012754+0000", 2024-08-19T14:38:09.318 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.44941769500000001 2024-08-19T14:38:09.318 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.319 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.319 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:38:09.319 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:13.012923+0000", 2024-08-19T14:38:09.319 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00016908099999999999 2024-08-19T14:38:09.319 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.319 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.319 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,4", 2024-08-19T14:38:09.319 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:13.013046+0000", 2024-08-19T14:38:09.319 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012365600000000001 2024-08-19T14:38:09.319 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.319 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.319 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:38:09.319 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:13.024640+0000", 2024-08-19T14:38:09.319 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011593643000000001 2024-08-19T14:38:09.320 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.320 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.320 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:38:09.320 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:13.024750+0000", 2024-08-19T14:38:09.320 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000109821 2024-08-19T14:38:09.320 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.320 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.320 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:38:09.320 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:13.067849+0000", 2024-08-19T14:38:09.320 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.043099155 2024-08-19T14:38:09.320 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.320 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.320 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:38:09.320 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:13.067900+0000", 2024-08-19T14:38:09.321 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0804999999999999e-05 2024-08-19T14:38:09.321 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.321 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.321 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:38:09.321 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:13.071919+0000", 2024-08-19T14:38:09.321 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0040194319999999999 2024-08-19T14:38:09.321 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:38:09.321 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:38:09.321 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:38:09.321 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.321 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.321 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.16962.0:122554 7.8 7:1008b633:::benchmark_data_smithi045_115842_object4394:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e635)", 2024-08-19T14:38:09.321 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:35:12.561454+0000", 2024-08-19T14:38:09.321 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 176.662896088, 2024-08-19T14:38:09.322 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.51282218000000002, 2024-08-19T14:38:09.322 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:38:09.322 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:38:09.322 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:38:09.322 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:38:09.322 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.16962", 2024-08-19T14:38:09.322 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:38:09.322 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 122554 2024-08-19T14:38:09.322 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.322 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:38:09.322 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.322 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:38:09.322 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:12.561454+0000", 2024-08-19T14:38:09.323 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:38:09.323 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.323 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.323 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:38:09.323 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:12.561454+0000", 2024-08-19T14:38:09.323 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:38:09.323 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.323 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.323 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:38:09.323 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:12.561454+0000", 2024-08-19T14:38:09.323 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3200000000000005e-07 2024-08-19T14:38:09.323 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.323 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.323 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:38:09.324 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:12.561457+0000", 2024-08-19T14:38:09.324 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5600000000000001e-06 2024-08-19T14:38:09.324 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.324 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.324 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:38:09.324 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:12.561462+0000", 2024-08-19T14:38:09.324 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5849999999999999e-06 2024-08-19T14:38:09.324 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.324 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.324 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:38:09.324 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:12.561466+0000", 2024-08-19T14:38:09.324 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5899999999999999e-06 2024-08-19T14:38:09.324 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.325 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.325 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:38:09.325 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:12.563391+0000", 2024-08-19T14:38:09.325 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0019253359999999999 2024-08-19T14:38:09.325 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.325 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.325 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:38:09.325 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:12.563398+0000", 2024-08-19T14:38:09.325 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.5889999999999999e-06 2024-08-19T14:38:09.325 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.325 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.325 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:38:09.325 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:13.020921+0000", 2024-08-19T14:38:09.325 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.45752294300000002 2024-08-19T14:38:09.326 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.326 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.326 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:38:09.326 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:13.021123+0000", 2024-08-19T14:38:09.326 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00020255800000000001 2024-08-19T14:38:09.326 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.326 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.326 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,4", 2024-08-19T14:38:09.326 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:13.021289+0000", 2024-08-19T14:38:09.326 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00016585999999999999 2024-08-19T14:38:09.326 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.326 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.326 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:38:09.326 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:13.030007+0000", 2024-08-19T14:38:09.327 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0087175050000000004 2024-08-19T14:38:09.327 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.327 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.327 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:38:09.327 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:13.030020+0000", 2024-08-19T14:38:09.327 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3236e-05 2024-08-19T14:38:09.327 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.327 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.327 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:38:09.327 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:13.071963+0000", 2024-08-19T14:38:09.327 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.041942895000000001 2024-08-19T14:38:09.327 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.327 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.327 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:38:09.327 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:13.072016+0000", 2024-08-19T14:38:09.328 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2889000000000001e-05 2024-08-19T14:38:09.328 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.328 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.328 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:38:09.328 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:13.074276+0000", 2024-08-19T14:38:09.328 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0022601019999999999 2024-08-19T14:38:09.328 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:38:09.328 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:38:09.328 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:38:09.328 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.328 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.328 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.16962.0:122576 7.8 7:1b83564c:::benchmark_data_smithi045_115842_object4416:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e635)", 2024-08-19T14:38:09.328 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:35:12.561486+0000", 2024-08-19T14:38:09.328 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 176.662863289, 2024-08-19T14:38:09.329 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.51524139800000002, 2024-08-19T14:38:09.329 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:38:09.329 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:38:09.329 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:38:09.329 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:38:09.329 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.16962", 2024-08-19T14:38:09.329 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:38:09.329 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 122576 2024-08-19T14:38:09.329 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.329 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:38:09.329 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.329 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:38:09.329 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:12.561486+0000", 2024-08-19T14:38:09.329 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:38:09.330 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.330 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.330 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:38:09.330 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:12.561486+0000", 2024-08-19T14:38:09.330 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:38:09.330 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.330 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.330 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:38:09.330 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:12.561487+0000", 2024-08-19T14:38:09.330 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9200000000000002e-07 2024-08-19T14:38:09.330 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.330 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.330 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:38:09.330 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:12.561489+0000", 2024-08-19T14:38:09.330 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.888e-06 2024-08-19T14:38:09.331 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.331 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.331 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:38:09.331 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:12.561494+0000", 2024-08-19T14:38:09.331 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5509999999999999e-06 2024-08-19T14:38:09.331 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.331 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.331 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:38:09.331 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:12.561498+0000", 2024-08-19T14:38:09.331 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.405e-06 2024-08-19T14:38:09.331 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.331 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.331 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:38:09.331 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:12.563439+0000", 2024-08-19T14:38:09.332 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001941309 2024-08-19T14:38:09.332 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.332 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.332 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:38:09.332 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:12.563448+0000", 2024-08-19T14:38:09.332 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.6120000000000005e-06 2024-08-19T14:38:09.332 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.332 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.332 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:38:09.332 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:13.024863+0000", 2024-08-19T14:38:09.332 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.46141411199999999 2024-08-19T14:38:09.332 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.332 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.332 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:38:09.332 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:13.024952+0000", 2024-08-19T14:38:09.333 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.9815000000000007e-05 2024-08-19T14:38:09.333 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.333 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.333 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,4", 2024-08-19T14:38:09.333 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:13.025064+0000", 2024-08-19T14:38:09.333 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000111401 2024-08-19T14:38:09.333 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.333 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.333 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:38:09.333 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:13.032805+0000", 2024-08-19T14:38:09.333 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0077412260000000004 2024-08-19T14:38:09.333 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.333 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.333 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:38:09.334 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:13.032848+0000", 2024-08-19T14:38:09.334 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2963999999999997e-05 2024-08-19T14:38:09.334 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.334 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.334 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:38:09.334 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:13.074309+0000", 2024-08-19T14:38:09.334 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.041460637000000002 2024-08-19T14:38:09.334 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.334 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.334 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:38:09.334 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:13.074347+0000", 2024-08-19T14:38:09.334 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7858999999999998e-05 2024-08-19T14:38:09.334 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.334 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.335 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:38:09.335 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:13.076728+0000", 2024-08-19T14:38:09.335 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0023813269999999999 2024-08-19T14:38:09.335 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:38:09.335 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:38:09.335 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:38:09.335 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.335 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.335 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.16962.0:122578 7.8 7:12aa5157:::benchmark_data_smithi045_115842_object4418:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e635)", 2024-08-19T14:38:09.335 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:35:12.561503+0000", 2024-08-19T14:38:09.335 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 176.66284670799999, 2024-08-19T14:38:09.335 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.517863399, 2024-08-19T14:38:09.335 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:38:09.336 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:38:09.336 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:38:09.336 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:38:09.336 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.16962", 2024-08-19T14:38:09.336 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:38:09.336 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 122578 2024-08-19T14:38:09.336 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.336 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:38:09.336 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.336 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:38:09.336 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:12.561503+0000", 2024-08-19T14:38:09.336 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:38:09.336 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.336 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.337 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:38:09.337 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:12.561503+0000", 2024-08-19T14:38:09.337 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:38:09.337 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.337 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.337 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:38:09.337 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:12.561503+0000", 2024-08-19T14:38:09.337 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.41e-07 2024-08-19T14:38:09.337 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.337 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.337 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:38:09.337 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:12.561505+0000", 2024-08-19T14:38:09.337 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6959999999999999e-06 2024-08-19T14:38:09.337 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.338 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.338 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:38:09.338 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:12.561509+0000", 2024-08-19T14:38:09.338 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5399999999999997e-06 2024-08-19T14:38:09.338 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.338 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.338 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:38:09.338 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:12.561513+0000", 2024-08-19T14:38:09.338 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.568e-06 2024-08-19T14:38:09.338 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.338 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.338 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:38:09.338 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:12.563489+0000", 2024-08-19T14:38:09.338 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0019761850000000001 2024-08-19T14:38:09.339 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.339 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.339 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:38:09.339 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:12.563496+0000", 2024-08-19T14:38:09.339 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8170000000000001e-06 2024-08-19T14:38:09.339 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.339 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.339 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:38:09.339 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:13.027144+0000", 2024-08-19T14:38:09.339 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.463648169 2024-08-19T14:38:09.339 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.339 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.339 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:38:09.340 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:13.027255+0000", 2024-08-19T14:38:09.340 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000110849 2024-08-19T14:38:09.340 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.340 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.340 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,4", 2024-08-19T14:38:09.340 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:13.027353+0000", 2024-08-19T14:38:09.340 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.8245000000000006e-05 2024-08-19T14:38:09.340 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.340 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.340 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:38:09.340 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:13.032816+0000", 2024-08-19T14:38:09.340 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0054629570000000001 2024-08-19T14:38:09.340 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.340 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.341 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:38:09.341 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:13.034078+0000", 2024-08-19T14:38:09.341 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0012616459999999999 2024-08-19T14:38:09.341 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.341 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.341 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:38:09.341 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:13.076789+0000", 2024-08-19T14:38:09.341 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.042710654000000001 2024-08-19T14:38:09.341 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.341 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.341 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:38:09.341 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:13.076830+0000", 2024-08-19T14:38:09.341 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1519e-05 2024-08-19T14:38:09.341 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.342 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.342 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:38:09.342 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:13.079366+0000", 2024-08-19T14:38:09.342 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0025363130000000001 2024-08-19T14:38:09.342 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:38:09.342 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:38:09.342 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:38:09.342 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.342 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.342 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.16962.0:122593 7.8 7:1b7c8906:::benchmark_data_smithi045_115842_object4433:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e635)", 2024-08-19T14:38:09.342 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:35:12.561517+0000", 2024-08-19T14:38:09.342 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 176.66283276799999, 2024-08-19T14:38:09.342 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.52015626100000001, 2024-08-19T14:38:09.343 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:38:09.343 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:38:09.343 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:38:09.343 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:38:09.343 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.16962", 2024-08-19T14:38:09.343 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:38:09.343 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 122593 2024-08-19T14:38:09.343 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.343 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:38:09.343 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.343 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:38:09.343 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:12.561517+0000", 2024-08-19T14:38:09.343 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:38:09.343 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.343 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.344 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:38:09.344 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:12.561517+0000", 2024-08-19T14:38:09.344 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:38:09.344 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.344 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.344 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:38:09.344 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:12.561517+0000", 2024-08-19T14:38:09.344 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0200000000000001e-07 2024-08-19T14:38:09.344 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.344 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.344 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:38:09.344 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:12.561519+0000", 2024-08-19T14:38:09.344 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5149999999999999e-06 2024-08-19T14:38:09.344 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.344 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.345 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:38:09.345 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:12.561523+0000", 2024-08-19T14:38:09.345 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1420000000000003e-06 2024-08-19T14:38:09.345 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.345 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.345 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:38:09.345 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:12.561526+0000", 2024-08-19T14:38:09.345 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1729999999999999e-06 2024-08-19T14:38:09.345 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.345 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.345 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:38:09.345 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:12.563540+0000", 2024-08-19T14:38:09.345 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0020139279999999999 2024-08-19T14:38:09.345 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.345 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.346 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:38:09.346 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:12.563548+0000", 2024-08-19T14:38:09.346 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.5599999999999994e-06 2024-08-19T14:38:09.346 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.346 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.346 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:38:09.346 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:13.030047+0000", 2024-08-19T14:38:09.346 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.46649901900000001 2024-08-19T14:38:09.346 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.346 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.346 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:38:09.346 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:13.030152+0000", 2024-08-19T14:38:09.346 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000104202 2024-08-19T14:38:09.346 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.347 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.347 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,4", 2024-08-19T14:38:09.347 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:13.030247+0000", 2024-08-19T14:38:09.347 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.5138999999999998e-05 2024-08-19T14:38:09.347 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.347 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.347 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:38:09.347 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:13.032959+0000", 2024-08-19T14:38:09.347 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0027118950000000002 2024-08-19T14:38:09.347 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.347 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.347 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:38:09.347 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:13.034158+0000", 2024-08-19T14:38:09.347 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0011988770000000001 2024-08-19T14:38:09.347 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.348 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.348 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:38:09.348 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:13.079406+0000", 2024-08-19T14:38:09.348 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.045248267000000002 2024-08-19T14:38:09.348 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.348 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.348 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:38:09.348 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:13.079448+0000", 2024-08-19T14:38:09.348 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2076999999999997e-05 2024-08-19T14:38:09.348 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:38:09.348 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:38:09.348 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:38:09.348 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:35:13.081673+0000", 2024-08-19T14:38:09.349 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0022252650000000001 2024-08-19T14:38:09.349 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:38:09.349 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:38:09.349 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:38:09.349 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:38:09.349 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:38:09.349 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:38:09.349 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:38:09.394 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:09.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:09.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:38:09.496 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:38:09.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:09 smithi032 ceph-mon[20004]: pgmap v1634: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 2.7 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 469 op/s; 17345/134280 objects degraded (12.917%); 5113/134280 objects misplaced (3.808%); 12 MiB/s, 199 objects/s recovering 2024-08-19T14:38:09.520 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:09.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:09.521 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:38:09.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:09 smithi031 ceph-mon[16336]: pgmap v1634: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 2.7 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 469 op/s; 17345/134280 objects degraded (12.917%); 5113/134280 objects misplaced (3.808%); 12 MiB/s, 199 objects/s recovering 2024-08-19T14:38:09.621 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:38:09.646 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:09.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:09.647 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:38:09.748 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:38:09.791 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:09.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:09.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:38:09.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:09 smithi045 ceph-mon[20143]: pgmap v1634: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 2.7 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 469 op/s; 17345/134280 objects degraded (12.917%); 5113/134280 objects misplaced (3.808%); 12 MiB/s, 199 objects/s recovering 2024-08-19T14:38:09.893 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:38:09.944 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:09.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:09.947 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:38:10.047 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:38:10.052 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:38:10.055 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:38:10.069 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:10.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:10.070 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:38:10.166 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-08-19T14:38:10.171 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:38:10.232 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:10.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:10.233 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:38:10.334 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:38:10.358 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:10.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:10.359 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:38:10.460 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:38:10.462 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:38:10.485 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:10.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:10.487 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:38:10.588 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:38:10.632 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:10.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:10.633 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:38:10.734 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:38:10.758 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:10.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:10.759 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:38:10.860 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:38:10.895 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:10.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:10.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:38:10.997 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:38:11.021 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:11.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:11.022 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:38:11.106 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:38:11.106 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [], 2024-08-19T14:38:11.106 INFO:teuthology.orchestra.run.smithi045.stdout: "num_ops": 0 2024-08-19T14:38:11.106 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:38:11.123 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:38:11.149 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:11.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:11.150 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:38:11.251 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:38:11.273 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:11.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:11.274 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:38:11.375 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:38:11.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:11.420 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:11.420 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:38:11.522 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:38:11.556 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:11.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:11.557 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:38:11.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:11 smithi031 ceph-mon[16336]: pgmap v1635: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 2.7 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 415 op/s; 17066/132210 objects degraded (12.908%); 5113/132210 objects misplaced (3.867%); 10 MiB/s, 165 objects/s recovering 2024-08-19T14:38:11.657 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:38:11.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:11.702 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:11.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:38:11.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:11 smithi032 ceph-mon[20004]: pgmap v1635: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 2.7 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 415 op/s; 17066/132210 objects degraded (12.908%); 5113/132210 objects misplaced (3.867%); 10 MiB/s, 165 objects/s recovering 2024-08-19T14:38:11.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:11 smithi045 ceph-mon[20143]: pgmap v1635: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 2.7 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 415 op/s; 17066/132210 objects degraded (12.908%); 5113/132210 objects misplaced (3.867%); 10 MiB/s, 165 objects/s recovering 2024-08-19T14:38:11.803 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:38:11.850 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:11.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:11.851 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:38:11.952 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:38:11.976 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:11.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:11.978 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:38:12.011 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-19T14:38:12.078 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:38:12.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:12.103 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:12.103 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:38:12.204 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:38:12.226 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:12.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:12.228 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:38:12.329 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:38:12.351 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:12.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:12.352 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:38:12.453 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:38:12.475 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:12.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:12.476 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:38:12.577 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:38:12.600 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:12.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:12.601 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:38:12.702 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:38:12.721 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:38:12.721 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:38:12.723 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:12.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:12.724 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:38:12.761 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:38:12.761 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:38:12.793 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:38:12.793 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:38:12.823 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:38:12.823 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:38:12.825 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:38:12.846 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:12.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:12.847 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:38:12.858 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:38:12.858 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:38:12.895 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:38:12.895 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:38:12.937 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:38:12.938 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:38:12.948 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:38:12.989 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:12.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:12.990 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:38:13.090 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:38:13.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:13.122 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:13.122 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:38:13.173 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:38:13.173 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:38:13.189 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:38:13.200 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:38:13.200 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:38:13.223 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:38:13.231 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:38:13.231 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:38:13.244 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:13.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:13.246 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:38:13.262 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:38:13.262 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:38:13.295 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:38:13.295 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:38:13.347 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:38:13.377 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:13 smithi031 ceph-mon[16336]: pgmap v1636: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 2.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 447 op/s; 16934/128961 objects degraded (13.131%); 4915/128961 objects misplaced (3.811%); 11 MiB/s, 197 objects/s recovering 2024-08-19T14:38:13.377 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:13 smithi031 ceph-mon[16336]: osdmap e688: 12 total, 12 up, 10 in 2024-08-19T14:38:13.377 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:13 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1006442668' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:38:13.377 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:13 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 16934/128961 objects degraded (13.131%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-19T14:38:13.379 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:13.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:13.380 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:38:13.481 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:38:13.506 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:13.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:13.507 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:38:13.607 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:38:13.607 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"ed7d1c6e-5e33-11ef-bccf-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub flag(s) set","count":7},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 16934/128961 objects degraded (13.131%), 6 pgs degraded, 6 pgs undersized","count":12},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":1870,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":689,"num_osds":12,"num_up_osds":12,"osd_up_since":1724078110,"num_in_osds":10,"osd_in_since":1724078051,"num_remapped_pgs":5},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":41},{"state_name":"active+recovering+undersized+degraded+remapped","count":6}],"num_pgs":47,"num_pools":2,"num_objects":42987,"data_bytes":2817589816,"bytes_used":17150525440,"bytes_avail":942799831040,"bytes_total":959950356480,"degraded_objects":16934,"degraded_total":128961,"degraded_ratio":0.13131101650886703,"misplaced_objects":4915,"misplaced_total":128961,"misplaced_ratio":0.038112297516303378,"recovering_objects_per_sec":197,"recovering_bytes_per_sec":11411633,"recovering_keys_per_sec":0,"num_objects_recovered":1769,"num_bytes_recovered":101974016,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":447},"fsmap":{"epoch":1,"btime":"2024-08-19T14:05:22:841940+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.31:8443/"}},"servicemap":{"epoch":42,"modified":"2024-08-19T14:37:16.216557+0000","services":{}},"progress_events":{}} 2024-08-19T14:38:13.607 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:38:13.630 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:13.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:13.631 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:38:13.733 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:38:13.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:13 smithi032 ceph-mon[20004]: pgmap v1636: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 2.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 447 op/s; 16934/128961 objects degraded (13.131%); 4915/128961 objects misplaced (3.811%); 11 MiB/s, 197 objects/s recovering 2024-08-19T14:38:13.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:13 smithi032 ceph-mon[20004]: osdmap e688: 12 total, 12 up, 10 in 2024-08-19T14:38:13.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:13 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1006442668' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:38:13.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:13 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 16934/128961 objects degraded (13.131%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-19T14:38:13.758 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:13.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:13.759 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:38:13.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:13 smithi045 ceph-mon[20143]: pgmap v1636: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 2.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 447 op/s; 16934/128961 objects degraded (13.131%); 4915/128961 objects misplaced (3.811%); 11 MiB/s, 197 objects/s recovering 2024-08-19T14:38:13.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:13 smithi045 ceph-mon[20143]: osdmap e688: 12 total, 12 up, 10 in 2024-08-19T14:38:13.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:13 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1006442668' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:38:13.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:13 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 16934/128961 objects degraded (13.131%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-19T14:38:13.859 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:38:13.882 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:13.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:13.883 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:38:13.984 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:38:14.034 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:14.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:14.035 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:38:14.136 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:38:14.158 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:14.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:14.159 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:38:14.260 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:38:14.284 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:14.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:14.285 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:38:14.386 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:38:14.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:14.412 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:14.412 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:38:14.475 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:14 smithi031 ceph-mon[16336]: osdmap e689: 12 total, 12 up, 10 in 2024-08-19T14:38:14.475 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:14 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1510153388' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-08-19T14:38:14.513 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:38:14.561 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:14.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:14.562 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:38:14.663 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:38:14.684 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:14.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:14.685 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:38:14.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:14 smithi032 ceph-mon[20004]: osdmap e689: 12 total, 12 up, 10 in 2024-08-19T14:38:14.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:14 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1510153388' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-08-19T14:38:14.786 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:38:14.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:14 smithi045 ceph-mon[20143]: osdmap e689: 12 total, 12 up, 10 in 2024-08-19T14:38:14.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:14 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1510153388' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-08-19T14:38:14.808 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:14.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:14.809 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:38:14.910 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:38:14.931 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:14.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:14.932 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:38:15.033 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:38:15.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:15.080 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:15.080 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:38:15.157 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-08-19T14:38:15.159 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-08-19T14:38:15.159 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-19T14:38:15.181 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:38:15.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:15.264 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:15.265 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:38:15.366 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:38:15.392 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:15.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:15.393 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:38:15.393 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is set 2024-08-19T14:38:15.447 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:38:15.494 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:38:15.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:15.518 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:15.518 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:38:15.619 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:38:15.642 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:15.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:15.643 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:38:15.745 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:38:15.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:15 smithi032 ceph-mon[20004]: pgmap v1639: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 2.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 400 op/s; 16887/127215 objects degraded (13.274%); 4827/127215 objects misplaced (3.794%); 6.7 MiB/s, 142 objects/s recovering 2024-08-19T14:38:15.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:15 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2442070262' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:38:15.761 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:15 smithi031 ceph-mon[16336]: pgmap v1639: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 2.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 400 op/s; 16887/127215 objects degraded (13.274%); 4827/127215 objects misplaced (3.794%); 6.7 MiB/s, 142 objects/s recovering 2024-08-19T14:38:15.761 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:15 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2442070262' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:38:15.766 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:15.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:15.769 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:38:15.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:15 smithi045 ceph-mon[20143]: pgmap v1639: 47 pgs: 6 active+recovering+undersized+degraded+remapped, 41 active+clean; 2.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 400 op/s; 16887/127215 objects degraded (13.274%); 4827/127215 objects misplaced (3.794%); 6.7 MiB/s, 142 objects/s recovering 2024-08-19T14:38:15.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:15 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2442070262' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:38:15.870 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:38:15.895 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:15.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:15.896 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:38:15.997 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:38:16.028 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:16.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:16.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:38:16.130 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:38:16.156 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:16.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:16.157 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:38:16.216 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:38:16.233 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:38:16.233 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:38:16.258 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:38:16.282 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:16.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:16.283 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:38:16.379 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:38:16.384 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:38:16.406 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:16.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:16.407 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:38:16.508 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:38:16.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:16.584 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:16.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:38:16.684 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:16 smithi031 ceph-mon[16336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:38:16.685 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:16 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2442070262' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:38:16.685 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:16 smithi031 ceph-mon[16336]: osdmap e690: 12 total, 12 up, 10 in 2024-08-19T14:38:16.685 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:38:16.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:16 smithi032 ceph-mon[20004]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:38:16.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:16 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2442070262' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:38:16.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:16 smithi032 ceph-mon[20004]: osdmap e690: 12 total, 12 up, 10 in 2024-08-19T14:38:16.759 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:16.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:16.760 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:38:16.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:16 smithi045 ceph-mon[20143]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:38:16.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:16 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2442070262' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:38:16.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:16 smithi045 ceph-mon[20143]: osdmap e690: 12 total, 12 up, 10 in 2024-08-19T14:38:16.861 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:38:16.927 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:16.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:16.928 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:38:16.932 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:38:17.026 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-08-19T14:38:17.028 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:38:17.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:17.081 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:17.081 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:38:17.182 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:38:17.197 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:38:17.198 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-08-19T14:38:17.204 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:17.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:17.205 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:38:17.306 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:38:17.315 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:38:17.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:17.348 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:17.348 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:38:17.449 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:38:17.493 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:17.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:17.494 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:38:17.594 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:38:17.628 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:17.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:17.629 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:38:17.721 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:17 smithi045 ceph-mon[20143]: pgmap v1641: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 490 op/s; 16525/123372 objects degraded (13.394%); 4827/123372 objects misplaced (3.913%); 7.8 MiB/s, 176 objects/s recovering 2024-08-19T14:38:17.730 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:38:17.773 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:17.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:17.774 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:38:17.839 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:38:17.840 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:38:17.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:17 smithi031 ceph-mon[16336]: pgmap v1641: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 490 op/s; 16525/123372 objects degraded (13.394%); 4827/123372 objects misplaced (3.913%); 7.8 MiB/s, 176 objects/s recovering 2024-08-19T14:38:17.875 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:38:17.896 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:17.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:17.897 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:38:17.998 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:38:18.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:17 smithi032 ceph-mon[20004]: pgmap v1641: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 490 op/s; 16525/123372 objects degraded (13.394%); 4827/123372 objects misplaced (3.913%); 7.8 MiB/s, 176 objects/s recovering 2024-08-19T14:38:18.021 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:18.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:18.023 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:38:18.124 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:38:18.127 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-19T14:38:18.166 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:18.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:18.167 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:38:18.270 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:38:18.294 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:18.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:18.295 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:38:18.396 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:38:18.448 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:18.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:18.449 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:38:18.550 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:38:18.599 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:18.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:18.600 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:38:18.626 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-08-19T14:38:18.701 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:38:18.724 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:18.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:18.725 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:38:18.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:18 smithi045 ceph-mon[20143]: from='client.26803 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:38:18.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:18 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 16525/123372 objects degraded (13.394%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-19T14:38:18.826 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:38:18.848 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:18.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:18.849 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:38:18.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:18 smithi031 ceph-mon[16336]: from='client.26803 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:38:18.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:18 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 16525/123372 objects degraded (13.394%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-19T14:38:18.925 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:38:18.949 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:38:19.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:19.002 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:19.003 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:38:19.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:18 smithi032 ceph-mon[20004]: from='client.26803 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:38:19.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:18 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 16525/123372 objects degraded (13.394%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-19T14:38:19.103 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:38:19.154 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:19.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:19.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:38:19.256 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:38:19.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:19.298 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:19.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:38:19.399 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:38:19.425 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:19.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:19.426 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:38:19.440 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:38:19.440 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:38:19.527 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:38:19.579 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:19.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:19.580 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:38:19.681 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:38:19.703 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:19.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:19.704 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:38:19.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:19 smithi045 ceph-mon[20143]: pgmap v1642: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 414 op/s; 16566/121497 objects degraded (13.635%); 4524/121497 objects misplaced (3.724%); 6.4 MiB/s, 144 objects/s recovering 2024-08-19T14:38:19.805 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:38:19.829 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:19.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:19.830 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:38:19.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:19 smithi031 ceph-mon[16336]: pgmap v1642: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 414 op/s; 16566/121497 objects degraded (13.635%); 4524/121497 objects misplaced (3.724%); 6.4 MiB/s, 144 objects/s recovering 2024-08-19T14:38:19.931 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:38:19.952 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:19.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:19.953 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:38:20.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:19 smithi032 ceph-mon[20004]: pgmap v1642: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 414 op/s; 16566/121497 objects degraded (13.635%); 4524/121497 objects misplaced (3.724%); 6.4 MiB/s, 144 objects/s recovering 2024-08-19T14:38:20.054 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:38:20.075 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:20.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:20.076 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:38:20.177 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:38:20.199 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:20.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:20.200 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:38:20.301 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:38:20.357 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:20.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:20.358 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:38:20.459 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:38:20.482 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:20.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:20.483 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:38:20.584 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:38:20.606 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:20.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:20.607 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:38:20.688 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:20 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2833737136' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:38:20.708 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:38:20.730 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:20.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:20.731 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:38:20.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:20 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2833737136' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:38:20.832 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:38:20.853 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:20.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:20.854 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:38:20.956 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:38:20.979 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:20.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:20.980 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:38:21.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:20 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2833737136' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:38:21.046 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:38:21.046 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:38:21.046 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:38:21.046 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:38:21.046 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:38:21.047 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:38:21.047 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:38:21.047 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:38:21.047 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:38:21.047 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:38:21.047 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:38:21.047 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:38:21.047 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:38:21.047 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:38:21.048 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:38:21.048 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:38:21.048 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:38:21.048 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:38:21.048 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:38:21.048 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:38:21.048 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:38:21.048 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:38:21.048 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:38:21.048 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:38:21.081 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:38:21.104 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:21.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:21.105 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:38:21.122 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-08-19T14:38:21.206 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:38:21.227 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:21.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:21.228 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:38:21.330 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:38:21.383 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:21.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:21.384 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:38:21.419 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:38:21.484 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:38:21.511 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:21.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:21.512 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:38:21.613 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:38:21.634 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:21.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:21.635 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:38:21.736 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:38:21.758 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:21.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:21.760 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:38:21.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:21 smithi045 ceph-mon[20143]: pgmap v1643: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.4 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 420 op/s; 16297/119013 objects degraded (13.693%); 4524/119013 objects misplaced (3.801%); 7.0 MiB/s, 143 objects/s recovering 2024-08-19T14:38:21.860 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:38:21.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:21 smithi031 ceph-mon[16336]: pgmap v1643: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.4 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 420 op/s; 16297/119013 objects degraded (13.693%); 4524/119013 objects misplaced (3.801%); 7.0 MiB/s, 143 objects/s recovering 2024-08-19T14:38:21.882 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:21.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:21.883 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:38:21.961 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:38:21.961 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [], 2024-08-19T14:38:21.961 INFO:teuthology.orchestra.run.smithi045.stdout: "complaint_time": 30, 2024-08-19T14:38:21.961 INFO:teuthology.orchestra.run.smithi045.stdout: "num_blocked_ops": 0 2024-08-19T14:38:21.962 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:38:21.984 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:38:22.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:21 smithi032 ceph-mon[20004]: pgmap v1643: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.4 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 420 op/s; 16297/119013 objects degraded (13.693%); 4524/119013 objects misplaced (3.801%); 7.0 MiB/s, 143 objects/s recovering 2024-08-19T14:38:22.009 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:22.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:22.011 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:38:22.112 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:38:22.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:22.168 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:22.168 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:38:22.269 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:38:22.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:22.305 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:22.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:38:22.406 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:38:22.429 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:22.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:22.430 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:38:22.531 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:38:22.539 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":1641,"stamp":"2024-08-19T14:38:16.237285+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2695496248,"num_objects":41124,"num_object_clones":0,"num_object_copies":123372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16525,"num_objects_misplaced":4827,"num_objects_unfound":0,"num_objects_dirty":41124,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":195194,"num_write_kb":7562049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42466,"num_bytes_recovered":2754286112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":165031,"ondisk_log_size":165031,"up":141,"acting":136,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":143,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":16448528,"kb_used_data":7980900,"kb_used_omap":18,"kb_used_meta":8467437,"kb_avail":921002992,"statfs":{"total":959950356480,"available":943107063808,"internally_reserved":0,"allocated":8172441600,"data_stored":8141973720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18760,"internal_metadata":8670656184},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,3,3,2,0,0,0,0,2],"upper_bound":2048},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-193069056,"num_objects":-2946,"num_object_clones":0,"num_object_copies":-8838,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-541,"num_objects_misplaced":-286,"num_objects_unfound":0,"num_objects_dirty":-2946,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2946,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1058,"num_bytes_recovered":48889856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002419"},"pg_stats":[{"pgid":"7.6","version":"690'12459","reported_seq":12752,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-08-19T14:38:15.511517+0000","last_change":"2024-08-19T14:37:56.236536+0000","last_active":"2024-08-19T14:38:15.511517+0000","last_peered":"2024-08-19T14:38:15.511517+0000","last_clean":"2024-08-19T14:38:15.511517+0000","last_became_active":"2024-08-19T14:33:28.334317+0000","last_became_peered":"2024-08-19T14:33:28.334317+0000","last_unstale":"2024-08-19T14:38:15.511517+0000","last_undegraded":"2024-08-19T14:38:15.511517+0000","last_fullsized":"2024-08-19T14:38:15.511517+0000","mapping_epoch":600,"log_start":"610'2400","ondisk_log_start":"610'2400","created":600,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"684'11923","last_scrub_stamp":"2024-08-19T14:37:56.236394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:37:56.236394+0000","objects_scrubbed":3155,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:39:12.585833+0000","scrub_duration":1586,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":170065920,"num_objects":2595,"num_object_clones":0,"num_object_copies":7785,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2595,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12459,"num_write_kb":481728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"70'139","reported_seq":1698,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-08-19T14:38:15.403507+0000","last_change":"2024-08-19T14:37:57.701402+0000","last_active":"2024-08-19T14:38:15.403507+0000","last_peered":"2024-08-19T14:38:15.403507+0000","last_clean":"2024-08-19T14:38:15.403507+0000","last_became_active":"2024-08-19T14:35:13.950224+0000","last_became_peered":"2024-08-19T14:35:13.950224+0000","last_unstale":"2024-08-19T14:38:15.403507+0000","last_undegraded":"2024-08-19T14:38:15.403507+0000","last_fullsized":"2024-08-19T14:38:15.403507+0000","mapping_epoch":637,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":638,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:37:57.701290+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:19:58.072083+0000","last_clean_scrub_stamp":"2024-08-19T14:37:57.701290+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:39:18.431910+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,9],"acting":[0,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.5","version":"687'11987","reported_seq":21452,"reported_epoch":687,"state":"active+clean","last_fresh":"2024-08-19T14:38:11.577146+0000","last_change":"2024-08-19T14:35:23.022361+0000","last_active":"2024-08-19T14:38:11.577146+0000","last_peered":"2024-08-19T14:38:11.577146+0000","last_clean":"2024-08-19T14:38:11.577146+0000","last_became_active":"2024-08-19T14:35:23.009238+0000","last_became_peered":"2024-08-19T14:35:23.009238+0000","last_unstale":"2024-08-19T14:38:11.577146+0000","last_undegraded":"2024-08-19T14:38:11.577146+0000","last_fullsized":"2024-08-19T14:38:11.577146+0000","mapping_epoch":641,"log_start":"608'1900","ondisk_log_start":"608'1900","created":600,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176619520,"num_objects":2695,"num_object_clones":0,"num_object_copies":8085,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2695,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11987,"num_write_kb":469824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4634,"num_bytes_recovered":303693824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,0],"acting":[11,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.3","version":"70'139","reported_seq":1539,"reported_epoch":688,"state":"active+clean","last_fresh":"2024-08-19T14:38:12.361032+0000","last_change":"2024-08-19T14:33:18.804321+0000","last_active":"2024-08-19T14:38:12.361032+0000","last_peered":"2024-08-19T14:38:12.361032+0000","last_clean":"2024-08-19T14:38:12.361032+0000","last_became_active":"2024-08-19T14:18:13.804721+0000","last_became_peered":"2024-08-19T14:18:13.804721+0000","last_unstale":"2024-08-19T14:38:12.361032+0000","last_undegraded":"2024-08-19T14:38:12.361032+0000","last_fullsized":"2024-08-19T14:38:12.361032+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:33:18.804228+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:33:18.804228+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.4","version":"690'12379","reported_seq":12668,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-08-19T14:38:15.430669+0000","last_change":"2024-08-19T14:38:06.717577+0000","last_active":"2024-08-19T14:38:15.430669+0000","last_peered":"2024-08-19T14:38:15.430669+0000","last_clean":"2024-08-19T14:38:15.430669+0000","last_became_active":"2024-08-19T14:33:28.334446+0000","last_became_peered":"2024-08-19T14:33:28.334446+0000","last_unstale":"2024-08-19T14:38:15.430669+0000","last_undegraded":"2024-08-19T14:38:15.430669+0000","last_fullsized":"2024-08-19T14:38:15.430669+0000","mapping_epoch":600,"log_start":"609'2300","ondisk_log_start":"609'2300","created":600,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"686'12159","last_scrub_stamp":"2024-08-19T14:38:06.717004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:38:06.717004+0000","objects_scrubbed":2792,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:39:16.656979+0000","scrub_duration":1017,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168230912,"num_objects":2567,"num_object_clones":0,"num_object_copies":7701,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2567,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12379,"num_write_kb":478272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"70'139","reported_seq":1499,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-08-19T14:38:15.433893+0000","last_change":"2024-08-19T14:34:55.580212+0000","last_active":"2024-08-19T14:38:15.433893+0000","last_peered":"2024-08-19T14:38:15.433893+0000","last_clean":"2024-08-19T14:38:15.433893+0000","last_became_active":"2024-08-19T14:20:50.742354+0000","last_became_peered":"2024-08-19T14:20:50.742354+0000","last_unstale":"2024-08-19T14:38:15.433893+0000","last_undegraded":"2024-08-19T14:38:15.433893+0000","last_fullsized":"2024-08-19T14:38:15.433893+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:34:55.580141+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:19:59.302931+0000","last_clean_scrub_stamp":"2024-08-19T14:34:55.580141+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"689'11959","reported_seq":12248,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-08-19T14:38:15.275577+0000","last_change":"2024-08-19T14:37:43.229201+0000","last_active":"2024-08-19T14:38:15.275577+0000","last_peered":"2024-08-19T14:38:15.275577+0000","last_clean":"2024-08-19T14:38:15.275577+0000","last_became_active":"2024-08-19T14:33:28.331833+0000","last_became_peered":"2024-08-19T14:33:28.331833+0000","last_unstale":"2024-08-19T14:38:15.275577+0000","last_undegraded":"2024-08-19T14:38:15.275577+0000","last_fullsized":"2024-08-19T14:38:15.275577+0000","mapping_epoch":600,"log_start":"608'1900","ondisk_log_start":"608'1900","created":600,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"678'11114","last_scrub_stamp":"2024-08-19T14:37:43.229072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:37:43.229072+0000","objects_scrubbed":3413,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:39:12.361681+0000","scrub_duration":910,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167706624,"num_objects":2559,"num_object_clones":0,"num_object_copies":7677,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2559,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11959,"num_write_kb":464576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,1],"acting":[8,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1","version":"70'139","reported_seq":797,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-08-19T14:38:15.403902+0000","last_change":"2024-08-19T14:37:42.630047+0000","last_active":"2024-08-19T14:38:15.403902+0000","last_peered":"2024-08-19T14:38:15.403902+0000","last_clean":"2024-08-19T14:38:15.403902+0000","last_became_active":"2024-08-19T14:35:12.530566+0000","last_became_peered":"2024-08-19T14:35:12.530566+0000","last_unstale":"2024-08-19T14:38:15.403902+0000","last_undegraded":"2024-08-19T14:38:15.403902+0000","last_fullsized":"2024-08-19T14:38:15.403902+0000","mapping_epoch":635,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":636,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:37:42.629919+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:37:42.629919+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:39:05.497590+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,2],"acting":[0,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"689'12063","reported_seq":12330,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-08-19T14:38:13.671198+0000","last_change":"2024-08-19T14:34:52.869444+0000","last_active":"2024-08-19T14:38:13.671198+0000","last_peered":"2024-08-19T14:38:13.671198+0000","last_clean":"2024-08-19T14:38:13.671198+0000","last_became_active":"2024-08-19T14:33:28.332994+0000","last_became_peered":"2024-08-19T14:33:28.332994+0000","last_unstale":"2024-08-19T14:38:13.671198+0000","last_undegraded":"2024-08-19T14:38:13.671198+0000","last_fullsized":"2024-08-19T14:38:13.671198+0000","mapping_epoch":600,"log_start":"608'2000","ondisk_log_start":"608'2000","created":600,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"627'6737","last_scrub_stamp":"2024-08-19T14:34:52.869233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:34:52.869233+0000","objects_scrubbed":6705,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1588,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":164823040,"num_objects":2515,"num_object_clones":0,"num_object_copies":7545,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2515,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12063,"num_write_kb":466496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"70'139","reported_seq":1199,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-08-19T14:38:13.386224+0000","last_change":"2024-08-19T14:35:11.537368+0000","last_active":"2024-08-19T14:38:13.386224+0000","last_peered":"2024-08-19T14:38:13.386224+0000","last_clean":"2024-08-19T14:38:13.386224+0000","last_became_active":"2024-08-19T14:35:11.536828+0000","last_became_peered":"2024-08-19T14:35:11.536828+0000","last_unstale":"2024-08-19T14:38:13.386224+0000","last_undegraded":"2024-08-19T14:38:13.386224+0000","last_fullsized":"2024-08-19T14:38:13.386224+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:27.536928+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:21:46.057624+0000","last_clean_scrub_stamp":"2024-08-19T14:28:27.536928+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"689'11969","reported_seq":21392,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-08-19T14:38:14.583212+0000","last_change":"2024-08-19T14:36:15.120872+0000","last_active":"2024-08-19T14:38:14.583212+0000","last_peered":"2024-08-19T14:38:14.583212+0000","last_clean":"2024-08-19T14:38:14.583212+0000","last_became_active":"2024-08-19T14:36:15.096070+0000","last_became_peered":"2024-08-19T14:36:15.096070+0000","last_unstale":"2024-08-19T14:38:14.583212+0000","last_undegraded":"2024-08-19T14:38:14.583212+0000","last_fullsized":"2024-08-19T14:38:14.583212+0000","mapping_epoch":653,"log_start":"608'1900","ondisk_log_start":"608'1900","created":600,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":169803776,"num_objects":2591,"num_object_clones":0,"num_object_copies":7773,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11969,"num_write_kb":465920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4611,"num_bytes_recovered":302186496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,9],"acting":[3,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"70'139","reported_seq":767,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-08-19T14:38:13.384968+0000","last_change":"2024-08-19T14:37:45.357396+0000","last_active":"2024-08-19T14:38:13.384968+0000","last_peered":"2024-08-19T14:38:13.384968+0000","last_clean":"2024-08-19T14:38:13.384968+0000","last_became_active":"2024-08-19T14:34:15.717054+0000","last_became_peered":"2024-08-19T14:34:15.717054+0000","last_unstale":"2024-08-19T14:38:13.384968+0000","last_undegraded":"2024-08-19T14:38:13.384968+0000","last_fullsized":"2024-08-19T14:38:13.384968+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:37:45.357293+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:27.848870+0000","last_clean_scrub_stamp":"2024-08-19T14:37:45.357293+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:38:48.964771+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"7.3","version":"689'12421","reported_seq":18660,"reported_epoch":689,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-19T14:38:14.575050+0000","last_change":"2024-08-19T14:35:13.136988+0000","last_active":"2024-08-19T14:38:14.575050+0000","last_peered":"2024-08-19T14:38:14.575050+0000","last_clean":"2024-08-19T14:33:27.312789+0000","last_became_active":"2024-08-19T14:35:13.021078+0000","last_became_peered":"2024-08-19T14:35:13.021078+0000","last_unstale":"2024-08-19T14:38:14.575050+0000","last_undegraded":"2024-08-19T14:35:12.592288+0000","last_fullsized":"2024-08-19T14:35:12.526971+0000","mapping_epoch":635,"log_start":"610'2400","ondisk_log_start":"610'2400","created":600,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":172556288,"num_objects":2633,"num_object_clones":0,"num_object_copies":7899,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12421,"num_write_kb":481728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3024,"num_bytes_recovered":195690496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":-1601}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"70'139","reported_seq":717,"reported_epoch":687,"state":"active+clean","last_fresh":"2024-08-19T14:38:07.344550+0000","last_change":"2024-08-19T14:35:11.533207+0000","last_active":"2024-08-19T14:38:07.344550+0000","last_peered":"2024-08-19T14:38:07.344550+0000","last_clean":"2024-08-19T14:38:07.344550+0000","last_became_active":"2024-08-19T14:35:11.532858+0000","last_became_peered":"2024-08-19T14:35:11.532858+0000","last_unstale":"2024-08-19T14:38:07.344550+0000","last_undegraded":"2024-08-19T14:38:07.344550+0000","last_fullsized":"2024-08-19T14:38:07.344550+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:27.331041+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:28:27.331041+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,0],"acting":[11,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.2","version":"689'12382","reported_seq":18588,"reported_epoch":689,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-19T14:38:14.628738+0000","last_change":"2024-08-19T14:35:13.121204+0000","last_active":"2024-08-19T14:38:14.628738+0000","last_peered":"2024-08-19T14:38:14.628738+0000","last_clean":"2024-08-19T14:33:27.312789+0000","last_became_active":"2024-08-19T14:35:13.013347+0000","last_became_peered":"2024-08-19T14:35:13.013347+0000","last_unstale":"2024-08-19T14:38:14.628738+0000","last_undegraded":"2024-08-19T14:35:12.598259+0000","last_fullsized":"2024-08-19T14:35:12.526602+0000","mapping_epoch":635,"log_start":"609'2300","ondisk_log_start":"609'2300","created":600,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":170262528,"num_objects":2598,"num_object_clones":0,"num_object_copies":7794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4227,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12382,"num_write_kb":479360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3008,"num_bytes_recovered":194772992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":-1629}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"70'139","reported_seq":594,"reported_epoch":687,"state":"active+clean","last_fresh":"2024-08-19T14:38:07.344508+0000","last_change":"2024-08-19T14:33:16.760928+0000","last_active":"2024-08-19T14:38:07.344508+0000","last_peered":"2024-08-19T14:38:07.344508+0000","last_clean":"2024-08-19T14:38:07.344508+0000","last_became_active":"2024-08-19T14:30:47.909067+0000","last_became_peered":"2024-08-19T14:30:47.909067+0000","last_unstale":"2024-08-19T14:38:07.344508+0000","last_undegraded":"2024-08-19T14:38:07.344508+0000","last_fullsized":"2024-08-19T14:38:07.344508+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:33:16.760810+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:33:16.760810+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.c","version":"689'12006","reported_seq":18252,"reported_epoch":689,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-19T14:38:14.669802+0000","last_change":"2024-08-19T14:35:13.133729+0000","last_active":"2024-08-19T14:38:14.669802+0000","last_peered":"2024-08-19T14:38:14.669802+0000","last_clean":"2024-08-19T14:33:27.312789+0000","last_became_active":"2024-08-19T14:35:13.003266+0000","last_became_peered":"2024-08-19T14:35:13.003266+0000","last_unstale":"2024-08-19T14:38:14.669802+0000","last_undegraded":"2024-08-19T14:35:12.593502+0000","last_fullsized":"2024-08-19T14:35:12.526375+0000","mapping_epoch":635,"log_start":"608'2000","ondisk_log_start":"608'2000","created":600,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167116800,"num_objects":2550,"num_object_clones":0,"num_object_copies":7650,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2550,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12006,"num_write_kb":465792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3028,"num_bytes_recovered":197001216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,2],"acting":[3,11],"avail_no_missing":["3","11"],"object_location_counts":[{"shards":"3,11","objects":-1431}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"70'139","reported_seq":764,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-08-19T14:38:13.385501+0000","last_change":"2024-08-19T14:34:50.256878+0000","last_active":"2024-08-19T14:38:13.385501+0000","last_peered":"2024-08-19T14:38:13.385501+0000","last_clean":"2024-08-19T14:38:13.385501+0000","last_became_active":"2024-08-19T14:34:15.717472+0000","last_became_peered":"2024-08-19T14:34:15.717472+0000","last_unstale":"2024-08-19T14:38:13.385501+0000","last_undegraded":"2024-08-19T14:38:13.385501+0000","last_fullsized":"2024-08-19T14:38:13.385501+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:34:50.256808+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:28.864870+0000","last_clean_scrub_stamp":"2024-08-19T14:34:50.256808+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,0],"acting":[4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"690'15728","reported_seq":25525,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-08-19T14:38:15.445380+0000","last_change":"2024-08-19T14:37:42.029576+0000","last_active":"2024-08-19T14:38:15.445380+0000","last_peered":"2024-08-19T14:38:15.445380+0000","last_clean":"2024-08-19T14:38:15.445380+0000","last_became_active":"2024-08-19T14:36:14.098166+0000","last_became_peered":"2024-08-19T14:36:14.098166+0000","last_unstale":"2024-08-19T14:38:15.445380+0000","last_undegraded":"2024-08-19T14:38:15.445380+0000","last_fullsized":"2024-08-19T14:38:15.445380+0000","mapping_epoch":652,"log_start":"623'5702","ondisk_log_start":"623'5702","created":600,"last_epoch_clean":653,"parent":"0.0","parent_split_bits":0,"last_scrub":"678'14831","last_scrub_stamp":"2024-08-19T14:37:42.029449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:37:42.029449+0000","objects_scrubbed":3383,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:38:44.946893+0000","scrub_duration":1404,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":161742848,"num_objects":2468,"num_object_clones":0,"num_object_copies":7404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2468,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12098,"num_write_kb":466112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4700,"num_bytes_recovered":308019200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.9","version":"70'139","reported_seq":1701,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-08-19T14:38:15.433933+0000","last_change":"2024-08-19T14:35:12.529993+0000","last_active":"2024-08-19T14:38:15.433933+0000","last_peered":"2024-08-19T14:38:15.433933+0000","last_clean":"2024-08-19T14:38:15.433933+0000","last_became_active":"2024-08-19T14:35:12.529628+0000","last_became_peered":"2024-08-19T14:35:12.529628+0000","last_unstale":"2024-08-19T14:38:15.433933+0000","last_undegraded":"2024-08-19T14:38:15.433933+0000","last_fullsized":"2024-08-19T14:38:15.433933+0000","mapping_epoch":635,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":636,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:34:51.561893+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:34:51.561893+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.e","version":"688'12301","reported_seq":23227,"reported_epoch":688,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-19T14:38:12.680571+0000","last_change":"2024-08-19T14:35:11.863818+0000","last_active":"2024-08-19T14:38:12.680571+0000","last_peered":"2024-08-19T14:38:12.680571+0000","last_clean":"2024-08-19T14:33:27.312789+0000","last_became_active":"2024-08-19T14:35:11.806304+0000","last_became_peered":"2024-08-19T14:35:11.806304+0000","last_unstale":"2024-08-19T14:38:12.680571+0000","last_undegraded":"2024-08-19T14:35:11.779866+0000","last_fullsized":"2024-08-19T14:35:11.528207+0000","mapping_epoch":634,"log_start":"608'2300","ondisk_log_start":"608'2300","created":600,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163512320,"num_objects":2495,"num_object_clones":0,"num_object_copies":7485,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2129,"num_objects_misplaced":4624,"num_objects_unfound":0,"num_objects_dirty":2495,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12301,"num_write_kb":473472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4128,"num_bytes_recovered":266665984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":-2129},{"shards":"1,5,9","objects":2495}],"blocked_by":[],"up_primary":2,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"70'139","reported_seq":729,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-08-19T14:38:13.385834+0000","last_change":"2024-08-19T14:35:11.537896+0000","last_active":"2024-08-19T14:38:13.385834+0000","last_peered":"2024-08-19T14:38:13.385834+0000","last_clean":"2024-08-19T14:38:13.385834+0000","last_became_active":"2024-08-19T14:35:11.537474+0000","last_became_peered":"2024-08-19T14:35:11.537474+0000","last_unstale":"2024-08-19T14:38:13.385834+0000","last_undegraded":"2024-08-19T14:38:13.385834+0000","last_fullsized":"2024-08-19T14:38:13.385834+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:59.829567+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:28:59.829567+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"688'12210","reported_seq":12399,"reported_epoch":688,"state":"active+clean","last_fresh":"2024-08-19T14:38:12.639375+0000","last_change":"2024-08-19T14:33:28.335586+0000","last_active":"2024-08-19T14:38:12.639375+0000","last_peered":"2024-08-19T14:38:12.639375+0000","last_clean":"2024-08-19T14:38:12.639375+0000","last_became_active":"2024-08-19T14:33:28.335298+0000","last_became_peered":"2024-08-19T14:33:28.335298+0000","last_unstale":"2024-08-19T14:38:12.639375+0000","last_undegraded":"2024-08-19T14:38:12.639375+0000","last_fullsized":"2024-08-19T14:38:12.639375+0000","mapping_epoch":600,"log_start":"608'2200","ondisk_log_start":"608'2200","created":600,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":164823064,"num_objects":2516,"num_object_clones":0,"num_object_copies":7548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2516,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12210,"num_write_kb":471169,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"70'139","reported_seq":1046,"reported_epoch":687,"state":"active+clean","last_fresh":"2024-08-19T14:38:07.344412+0000","last_change":"2024-08-19T14:33:15.753668+0000","last_active":"2024-08-19T14:38:07.344412+0000","last_peered":"2024-08-19T14:38:07.344412+0000","last_clean":"2024-08-19T14:38:07.344412+0000","last_became_active":"2024-08-19T14:30:47.941200+0000","last_became_peered":"2024-08-19T14:30:47.941200+0000","last_unstale":"2024-08-19T14:38:07.344412+0000","last_undegraded":"2024-08-19T14:38:07.344412+0000","last_fullsized":"2024-08-19T14:38:07.344412+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:33:15.753595+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:33:15.753595+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.8","version":"690'12526","reported_seq":27836,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-08-19T14:38:15.433900+0000","last_change":"2024-08-19T14:38:13.418089+0000","last_active":"2024-08-19T14:38:15.433900+0000","last_peered":"2024-08-19T14:38:15.433900+0000","last_clean":"2024-08-19T14:38:15.433900+0000","last_became_active":"2024-08-19T14:38:13.389656+0000","last_became_peered":"2024-08-19T14:38:13.389656+0000","last_unstale":"2024-08-19T14:38:15.433900+0000","last_undegraded":"2024-08-19T14:38:15.433900+0000","last_fullsized":"2024-08-19T14:38:15.433900+0000","mapping_epoch":688,"log_start":"611'2500","ondisk_log_start":"611'2500","created":600,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":169345024,"num_objects":2584,"num_object_clones":0,"num_object_copies":7752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12526,"num_write_kb":483520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7555,"num_bytes_recovered":476577792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.e","version":"70'139","reported_seq":1240,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-08-19T14:38:13.385527+0000","last_change":"2024-08-19T14:34:55.333177+0000","last_active":"2024-08-19T14:38:13.385527+0000","last_peered":"2024-08-19T14:38:13.385527+0000","last_clean":"2024-08-19T14:38:13.385527+0000","last_became_active":"2024-08-19T14:18:26.884105+0000","last_became_peered":"2024-08-19T14:18:26.884105+0000","last_unstale":"2024-08-19T14:38:13.385527+0000","last_undegraded":"2024-08-19T14:38:13.385527+0000","last_fullsized":"2024-08-19T14:38:13.385527+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:34:55.333065+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:32:31.196288+0000","last_clean_scrub_stamp":"2024-08-19T14:34:55.333065+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"690'12343","reported_seq":12671,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-08-19T14:38:15.437516+0000","last_change":"2024-08-19T14:37:54.857799+0000","last_active":"2024-08-19T14:38:15.437516+0000","last_peered":"2024-08-19T14:38:15.437516+0000","last_clean":"2024-08-19T14:38:15.437516+0000","last_became_active":"2024-08-19T14:33:28.334547+0000","last_became_peered":"2024-08-19T14:33:28.334547+0000","last_unstale":"2024-08-19T14:38:15.437516+0000","last_undegraded":"2024-08-19T14:38:15.437516+0000","last_fullsized":"2024-08-19T14:38:15.437516+0000","mapping_epoch":600,"log_start":"609'2300","ondisk_log_start":"609'2300","created":600,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"683'11737","last_scrub_stamp":"2024-08-19T14:37:54.851356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:37:54.851356+0000","objects_scrubbed":3161,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:39:09.304335+0000","scrub_duration":1254,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":166264832,"num_objects":2537,"num_object_clones":0,"num_object_copies":7611,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12343,"num_write_kb":476160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,8],"acting":[0,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"70'139","reported_seq":992,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-08-19T14:38:13.385519+0000","last_change":"2024-08-19T14:34:12.199530+0000","last_active":"2024-08-19T14:38:13.385519+0000","last_peered":"2024-08-19T14:38:13.385519+0000","last_clean":"2024-08-19T14:38:13.385519+0000","last_became_active":"2024-08-19T14:27:31.837754+0000","last_became_peered":"2024-08-19T14:27:31.837754+0000","last_unstale":"2024-08-19T14:38:13.385519+0000","last_undegraded":"2024-08-19T14:38:13.385519+0000","last_fullsized":"2024-08-19T14:38:13.385519+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:34:12.199449+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:34:12.199449+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.a","version":"687'12082","reported_seq":29678,"reported_epoch":687,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-19T14:38:11.708825+0000","last_change":"2024-08-19T14:35:13.046246+0000","last_active":"2024-08-19T14:38:11.708825+0000","last_peered":"2024-08-19T14:38:11.708825+0000","last_clean":"2024-08-19T14:33:27.312789+0000","last_became_active":"2024-08-19T14:35:13.005199+0000","last_became_peered":"2024-08-19T14:35:13.005199+0000","last_unstale":"2024-08-19T14:38:11.708825+0000","last_undegraded":"2024-08-19T14:35:12.692218+0000","last_fullsized":"2024-08-19T14:35:12.524923+0000","mapping_epoch":635,"log_start":"608'2000","ondisk_log_start":"608'2000","created":600,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173277184,"num_objects":2644,"num_object_clones":0,"num_object_copies":7932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1954,"num_objects_misplaced":203,"num_objects_unfound":0,"num_objects_dirty":2644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12082,"num_write_kb":471232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7775,"num_bytes_recovered":508821504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,2],"acting":[11,9],"avail_no_missing":["11","3","9"],"object_location_counts":[{"shards":"3,9,11","objects":203},{"shards":"9,11","objects":487}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.c","version":"70'139","reported_seq":1347,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-08-19T14:38:13.391579+0000","last_change":"2024-08-19T14:34:15.738488+0000","last_active":"2024-08-19T14:38:13.391579+0000","last_peered":"2024-08-19T14:38:13.391579+0000","last_clean":"2024-08-19T14:38:13.391579+0000","last_became_active":"2024-08-19T14:34:15.733282+0000","last_became_peered":"2024-08-19T14:34:15.733282+0000","last_unstale":"2024-08-19T14:38:13.391579+0000","last_undegraded":"2024-08-19T14:38:13.391579+0000","last_fullsized":"2024-08-19T14:38:13.391579+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:33:17.453290+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:22:52.164586+0000","last_clean_scrub_stamp":"2024-08-19T14:33:17.453290+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,9],"acting":[3,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"688'12009","reported_seq":12197,"reported_epoch":688,"state":"active+clean","last_fresh":"2024-08-19T14:38:12.608590+0000","last_change":"2024-08-19T14:33:28.623717+0000","last_active":"2024-08-19T14:38:12.608590+0000","last_peered":"2024-08-19T14:38:12.608590+0000","last_clean":"2024-08-19T14:38:12.608590+0000","last_became_active":"2024-08-19T14:33:28.623318+0000","last_became_peered":"2024-08-19T14:33:28.623318+0000","last_unstale":"2024-08-19T14:38:12.608590+0000","last_undegraded":"2024-08-19T14:38:12.608590+0000","last_fullsized":"2024-08-19T14:38:12.608590+0000","mapping_epoch":600,"log_start":"608'2000","ondisk_log_start":"608'2000","created":600,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168755200,"num_objects":2575,"num_object_clones":0,"num_object_copies":7725,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2575,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12009,"num_write_kb":466688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"70'139","reported_seq":1246,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-08-19T14:38:15.434077+0000","last_change":"2024-08-19T14:34:49.476489+0000","last_active":"2024-08-19T14:38:15.434077+0000","last_peered":"2024-08-19T14:38:15.434077+0000","last_clean":"2024-08-19T14:38:15.434077+0000","last_became_active":"2024-08-19T14:18:26.882002+0000","last_became_peered":"2024-08-19T14:18:26.882002+0000","last_unstale":"2024-08-19T14:38:15.434077+0000","last_undegraded":"2024-08-19T14:38:15.434077+0000","last_fullsized":"2024-08-19T14:38:15.434077+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:34:49.476415+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:24:50.276408+0000","last_clean_scrub_stamp":"2024-08-19T14:34:49.476415+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"70'139","reported_seq":1021,"reported_epoch":688,"state":"active+clean","last_fresh":"2024-08-19T14:38:12.360620+0000","last_change":"2024-08-19T14:33:19.781527+0000","last_active":"2024-08-19T14:38:12.360620+0000","last_peered":"2024-08-19T14:38:12.360620+0000","last_clean":"2024-08-19T14:38:12.360620+0000","last_became_active":"2024-08-19T14:27:31.838968+0000","last_became_peered":"2024-08-19T14:27:31.838968+0000","last_unstale":"2024-08-19T14:38:12.360620+0000","last_undegraded":"2024-08-19T14:38:12.360620+0000","last_fullsized":"2024-08-19T14:38:12.360620+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:33:19.781431+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:28:35.206115+0000","last_clean_scrub_stamp":"2024-08-19T14:33:19.781431+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.13","version":"70'139","reported_seq":1563,"reported_epoch":688,"state":"active+clean","last_fresh":"2024-08-19T14:38:12.360854+0000","last_change":"2024-08-19T14:33:35.786236+0000","last_active":"2024-08-19T14:38:12.360854+0000","last_peered":"2024-08-19T14:38:12.360854+0000","last_clean":"2024-08-19T14:38:12.360854+0000","last_became_active":"2024-08-19T14:30:47.939849+0000","last_became_peered":"2024-08-19T14:30:47.939849+0000","last_unstale":"2024-08-19T14:38:12.360854+0000","last_undegraded":"2024-08-19T14:38:12.360854+0000","last_fullsized":"2024-08-19T14:38:12.360854+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:33:35.786105+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:33:35.786105+0000","last_clean_scrub_stamp":"2024-08-19T14:33:35.786105+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.10","version":"70'139","reported_seq":1195,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-08-19T14:38:15.434060+0000","last_change":"2024-08-19T14:34:58.616722+0000","last_active":"2024-08-19T14:38:15.434060+0000","last_peered":"2024-08-19T14:38:15.434060+0000","last_clean":"2024-08-19T14:38:15.434060+0000","last_became_active":"2024-08-19T14:27:31.839611+0000","last_became_peered":"2024-08-19T14:27:31.839611+0000","last_unstale":"2024-08-19T14:38:15.434060+0000","last_undegraded":"2024-08-19T14:38:15.434060+0000","last_fullsized":"2024-08-19T14:38:15.434060+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:34:58.616646+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:34:58.616646+0000","last_clean_scrub_stamp":"2024-08-19T14:34:58.616646+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.11","version":"70'139","reported_seq":1243,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-08-19T14:38:15.403656+0000","last_change":"2024-08-19T14:36:43.798545+0000","last_active":"2024-08-19T14:38:15.403656+0000","last_peered":"2024-08-19T14:38:15.403656+0000","last_clean":"2024-08-19T14:38:15.403656+0000","last_became_active":"2024-08-19T14:34:15.731049+0000","last_became_peered":"2024-08-19T14:34:15.731049+0000","last_unstale":"2024-08-19T14:38:15.403656+0000","last_undegraded":"2024-08-19T14:38:15.403656+0000","last_fullsized":"2024-08-19T14:38:15.403656+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:36:43.798385+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:24:50.547428+0000","last_clean_scrub_stamp":"2024-08-19T14:36:43.798385+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:38:16.708216+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"70'139","reported_seq":1174,"reported_epoch":688,"state":"active+clean","last_fresh":"2024-08-19T14:38:12.360730+0000","last_change":"2024-08-19T14:33:27.737496+0000","last_active":"2024-08-19T14:38:12.360730+0000","last_peered":"2024-08-19T14:38:12.360730+0000","last_clean":"2024-08-19T14:38:12.360730+0000","last_became_active":"2024-08-19T14:20:28.204724+0000","last_became_peered":"2024-08-19T14:20:28.204724+0000","last_unstale":"2024-08-19T14:38:12.360730+0000","last_undegraded":"2024-08-19T14:38:12.360730+0000","last_fullsized":"2024-08-19T14:38:12.360730+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:33:27.737367+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:28:34.216370+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.737367+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"70'139","reported_seq":1319,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-08-19T14:38:13.384990+0000","last_change":"2024-08-19T14:37:38.355694+0000","last_active":"2024-08-19T14:38:13.384990+0000","last_peered":"2024-08-19T14:38:13.384990+0000","last_clean":"2024-08-19T14:38:13.384990+0000","last_became_active":"2024-08-19T14:18:31.901556+0000","last_became_peered":"2024-08-19T14:18:31.901556+0000","last_unstale":"2024-08-19T14:38:13.384990+0000","last_undegraded":"2024-08-19T14:38:13.384990+0000","last_fullsized":"2024-08-19T14:38:13.384990+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:37:38.355621+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:37:38.355621+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:38:39.528989+0000","scrub_duration":56,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,0],"acting":[8,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.14","version":"70'139","reported_seq":1350,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-08-19T14:38:15.433933+0000","last_change":"2024-08-19T14:35:12.530145+0000","last_active":"2024-08-19T14:38:15.433933+0000","last_peered":"2024-08-19T14:38:15.433933+0000","last_clean":"2024-08-19T14:38:15.433933+0000","last_became_active":"2024-08-19T14:35:12.529792+0000","last_became_peered":"2024-08-19T14:35:12.529792+0000","last_unstale":"2024-08-19T14:38:15.433933+0000","last_undegraded":"2024-08-19T14:38:15.433933+0000","last_fullsized":"2024-08-19T14:38:15.433933+0000","mapping_epoch":635,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":636,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:34:53.617013+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:34:53.617013+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,9],"acting":[7,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.15","version":"70'139","reported_seq":1202,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-08-19T14:38:15.434066+0000","last_change":"2024-08-19T14:34:57.640316+0000","last_active":"2024-08-19T14:38:15.434066+0000","last_peered":"2024-08-19T14:38:15.434066+0000","last_clean":"2024-08-19T14:38:15.434066+0000","last_became_active":"2024-08-19T14:18:31.912496+0000","last_became_peered":"2024-08-19T14:18:31.912496+0000","last_unstale":"2024-08-19T14:38:15.434066+0000","last_undegraded":"2024-08-19T14:38:15.434066+0000","last_fullsized":"2024-08-19T14:38:15.434066+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:34:57.640213+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:20:09.180923+0000","last_clean_scrub_stamp":"2024-08-19T14:34:57.640213+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"70'139","reported_seq":1185,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-08-19T14:38:13.385404+0000","last_change":"2024-08-19T14:34:06.176510+0000","last_active":"2024-08-19T14:38:13.385404+0000","last_peered":"2024-08-19T14:38:13.385404+0000","last_clean":"2024-08-19T14:38:13.385404+0000","last_became_active":"2024-08-19T14:18:31.901229+0000","last_became_peered":"2024-08-19T14:18:31.901229+0000","last_unstale":"2024-08-19T14:38:13.385404+0000","last_undegraded":"2024-08-19T14:38:13.385404+0000","last_fullsized":"2024-08-19T14:38:13.385404+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:34:06.176403+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:34:06.176403+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1b","version":"70'139","reported_seq":1566,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-08-19T14:38:13.384819+0000","last_change":"2024-08-19T14:37:59.510774+0000","last_active":"2024-08-19T14:38:13.384819+0000","last_peered":"2024-08-19T14:38:13.384819+0000","last_clean":"2024-08-19T14:38:13.384819+0000","last_became_active":"2024-08-19T14:34:15.718427+0000","last_became_peered":"2024-08-19T14:34:15.718427+0000","last_unstale":"2024-08-19T14:38:13.384819+0000","last_undegraded":"2024-08-19T14:38:13.384819+0000","last_fullsized":"2024-08-19T14:38:13.384819+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:37:59.510697+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:37:59.510697+0000","last_clean_scrub_stamp":"2024-08-19T14:37:59.510697+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:39:04.057695+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.18","version":"70'139","reported_seq":1130,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-08-19T14:38:15.433851+0000","last_change":"2024-08-19T14:35:11.528963+0000","last_active":"2024-08-19T14:38:15.433851+0000","last_peered":"2024-08-19T14:38:15.433851+0000","last_clean":"2024-08-19T14:38:15.433851+0000","last_became_active":"2024-08-19T14:35:11.528530+0000","last_became_peered":"2024-08-19T14:35:11.528530+0000","last_unstale":"2024-08-19T14:38:15.433851+0000","last_undegraded":"2024-08-19T14:38:15.433851+0000","last_fullsized":"2024-08-19T14:38:15.433851+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:29:05.336156+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:27:33.084161+0000","last_clean_scrub_stamp":"2024-08-19T14:29:05.336156+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.19","version":"70'139","reported_seq":1548,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-08-19T14:38:15.403560+0000","last_change":"2024-08-19T14:37:41.671836+0000","last_active":"2024-08-19T14:38:15.403560+0000","last_peered":"2024-08-19T14:38:15.403560+0000","last_clean":"2024-08-19T14:38:15.403560+0000","last_became_active":"2024-08-19T14:18:31.902547+0000","last_became_peered":"2024-08-19T14:18:31.902547+0000","last_unstale":"2024-08-19T14:38:15.403560+0000","last_undegraded":"2024-08-19T14:38:15.403560+0000","last_fullsized":"2024-08-19T14:38:15.403560+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:37:41.671731+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:37:41.671731+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:38:59.596992+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1e","version":"70'139","reported_seq":566,"reported_epoch":687,"state":"active+clean","last_fresh":"2024-08-19T14:38:07.344433+0000","last_change":"2024-08-19T14:33:17.778415+0000","last_active":"2024-08-19T14:38:07.344433+0000","last_peered":"2024-08-19T14:38:07.344433+0000","last_clean":"2024-08-19T14:38:07.344433+0000","last_became_active":"2024-08-19T14:30:47.909974+0000","last_became_peered":"2024-08-19T14:30:47.909974+0000","last_unstale":"2024-08-19T14:38:07.344433+0000","last_undegraded":"2024-08-19T14:38:07.344433+0000","last_fullsized":"2024-08-19T14:38:07.344433+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:33:17.778295+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:27:04.780939+0000","last_clean_scrub_stamp":"2024-08-19T14:33:17.778295+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1c","version":"70'139","reported_seq":1320,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-08-19T14:38:13.384856+0000","last_change":"2024-08-19T14:38:01.463630+0000","last_active":"2024-08-19T14:38:13.384856+0000","last_peered":"2024-08-19T14:38:13.384856+0000","last_clean":"2024-08-19T14:38:13.384856+0000","last_became_active":"2024-08-19T14:30:47.908866+0000","last_became_peered":"2024-08-19T14:30:47.908866+0000","last_unstale":"2024-08-19T14:38:13.384856+0000","last_undegraded":"2024-08-19T14:38:13.384856+0000","last_fullsized":"2024-08-19T14:38:13.384856+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:38:01.463524+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:38:01.463524+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:39:18.496929+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,11],"acting":[8,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1d","version":"70'139","reported_seq":1227,"reported_epoch":688,"state":"active+clean","last_fresh":"2024-08-19T14:38:12.360724+0000","last_change":"2024-08-19T14:35:11.542467+0000","last_active":"2024-08-19T14:38:12.360724+0000","last_peered":"2024-08-19T14:38:12.360724+0000","last_clean":"2024-08-19T14:38:12.360724+0000","last_became_active":"2024-08-19T14:35:11.540488+0000","last_became_peered":"2024-08-19T14:35:11.540488+0000","last_unstale":"2024-08-19T14:38:12.360724+0000","last_undegraded":"2024-08-19T14:38:12.360724+0000","last_fullsized":"2024-08-19T14:38:12.360724+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:29.194639+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:28:29.194639+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":2694905880,"num_objects":41122,"num_object_clones":0,"num_object_copies":123366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16525,"num_objects_misplaced":4827,"num_objects_unfound":0,"num_objects_dirty":41122,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":195194,"num_write_kb":7562049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42463,"num_bytes_recovered":2753429504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8059039744,"data_stored":8059027528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160722,"ondisk_log_size":160722,"up":48,"acting":43,"num_store_stats":10},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":856608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187778,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1730692,"kb_used_data":860988,"kb_used_omap":1,"kb_used_meta":869694,"kb_avail":92014460,"statfs":{"total":95995035648,"available":94222807040,"internally_reserved":0,"allocated":881651712,"data_stored":878612891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":890567108},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":2314987372646,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":74,"seq":317827580236,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1673040,"kb_used_data":835272,"kb_used_omap":1,"kb_used_meta":837758,"kb_avail":92072112,"statfs":{"total":95995035648,"available":94281842688,"internally_reserved":0,"allocated":855318528,"data_stored":852275659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":857864641},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874579,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1264100,"kb_used_data":492948,"kb_used_omap":1,"kb_used_meta":771134,"kb_avail":92481052,"statfs":{"total":95995035648,"available":94700597248,"internally_reserved":0,"allocated":504778752,"data_stored":501731814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":789641669},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":57,"seq":244813136214,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1442412,"kb_used_data":508828,"kb_used_omap":1,"kb_used_meta":933566,"kb_avail":92302740,"statfs":{"total":95995035648,"available":94518005760,"internally_reserved":0,"allocated":521039872,"data_stored":517989382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":955972042},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":206158430553,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692188,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1441356,"kb_used_data":503620,"kb_used_omap":1,"kb_used_meta":937726,"kb_avail":92303796,"statfs":{"total":95995035648,"available":94519087104,"internally_reserved":0,"allocated":515706880,"data_stored":512664507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":960231877},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986529,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1644576,"kb_used_data":674900,"kb_used_omap":1,"kb_used_meta":969662,"kb_avail":92100576,"statfs":{"total":95995035648,"available":94310989824,"internally_reserved":0,"allocated":691097600,"data_stored":688050662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":992934341},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150056,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2769060,"kb_used_data":1777112,"kb_used_omap":1,"kb_used_meta":991870,"kb_avail":90976092,"statfs":{"total":95995035648,"available":93159518208,"internally_reserved":0,"allocated":1819762688,"data_stored":1816711678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1015675335},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":634,"seq":2723009265702,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":557156,"kb_used_data":344604,"kb_used_omap":4,"kb_used_meta":212539,"kb_avail":93187996,"statfs":{"total":95995035648,"available":95424507904,"internally_reserved":0,"allocated":352874496,"data_stored":349824006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4407,"internal_metadata":217640649},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575216,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2091672,"kb_used_data":991500,"kb_used_omap":1,"kb_used_meta":1100158,"kb_avail":91653480,"statfs":{"total":95995035648,"available":93853163520,"internally_reserved":0,"allocated":1015296000,"data_stored":1012249059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1126562245},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,0,0,0,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738741,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1834464,"kb_used_data":991128,"kb_used_omap":1,"kb_used_meta":843326,"kb_avail":91910688,"statfs":{"total":95995035648,"available":94116544512,"internally_reserved":0,"allocated":1014915072,"data_stored":1011864062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":863566277},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":998838272,"data_stored":998834200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1006505984,"data_stored":1006501912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":341639168,"data_stored":341639168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1808863232,"data_stored":1808859160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":682491904,"data_stored":682491904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":504758272,"data_stored":504758272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":508428288,"data_stored":508428288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":496041984,"data_stored":496041984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":846594048,"data_stored":846594048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":864878592,"data_stored":864878592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T14:38:22.554 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:22.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:22.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:38:22.656 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:38:22.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:22.709 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:22.709 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:38:22.723 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-08-19T14:38:22.809 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:38:22.831 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:22.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:22.832 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:38:22.933 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:38:22.955 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:22.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:22.956 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:38:22.990 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:38:23.057 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:38:23.083 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:23.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:23.084 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:38:23.185 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:38:23.208 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:23.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:23.209 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:38:23.310 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:38:23.333 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:23.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:23.334 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:38:23.435 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:38:23.469 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:23.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:23.470 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:38:23.527 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:38:23.527 INFO:teuthology.orchestra.run.smithi045.stdout: "size": 20, 2024-08-19T14:38:23.527 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 600, 2024-08-19T14:38:23.527 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [ 2024-08-19T14:38:23.527 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.527 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.16962.0:76277 7.e e618/617 7:731ae8a5:::benchmark_data_smithi045_115842_object76276:head v 618'4887, mlcod=605'7)", 2024-08-19T14:38:23.527 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:34:14.255266+0000", 2024-08-19T14:38:23.527 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 249.268411112, 2024-08-19T14:38:23.527 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.058286588, 2024-08-19T14:38:23.528 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:38:23.528 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:38:23.528 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:38:23.528 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:38:23.528 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.528 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:38:23.528 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.255266+0000", 2024-08-19T14:38:23.528 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:38:23.528 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.528 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.528 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:38:23.528 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.255266+0000", 2024-08-19T14:38:23.528 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:38:23.528 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.529 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.529 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:38:23.529 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.255267+0000", 2024-08-19T14:38:23.529 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.8599999999999998e-07 2024-08-19T14:38:23.529 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.529 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.529 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:38:23.529 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.255324+0000", 2024-08-19T14:38:23.529 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7120000000000002e-05 2024-08-19T14:38:23.529 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.529 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.529 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:38:23.529 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.255333+0000", 2024-08-19T14:38:23.530 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.1549999999999996e-06 2024-08-19T14:38:23.530 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.530 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.530 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:38:23.530 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.255339+0000", 2024-08-19T14:38:23.530 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.6339999999999998e-06 2024-08-19T14:38:23.530 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.530 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.530 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:38:23.530 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.255389+0000", 2024-08-19T14:38:23.530 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0219999999999997e-05 2024-08-19T14:38:23.530 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.531 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.531 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:38:23.531 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.255410+0000", 2024-08-19T14:38:23.531 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1421e-05 2024-08-19T14:38:23.531 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.531 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.531 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:38:23.531 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.313471+0000", 2024-08-19T14:38:23.531 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.058061029 2024-08-19T14:38:23.531 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.531 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.531 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:38:23.531 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.313553+0000", 2024-08-19T14:38:23.532 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.1422999999999996e-05 2024-08-19T14:38:23.532 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:38:23.532 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:38:23.532 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:38:23.532 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.532 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.532 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.16962.0:76286 7.e e618/617 7:795fa467:::benchmark_data_smithi045_115842_object76285:head v 618'4888, mlcod=605'7)", 2024-08-19T14:38:23.532 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:34:14.258319+0000", 2024-08-19T14:38:23.532 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 249.265358338, 2024-08-19T14:38:23.532 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.055304399999999997, 2024-08-19T14:38:23.532 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:38:23.532 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:38:23.532 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:38:23.532 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:38:23.533 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.533 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:38:23.533 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.258319+0000", 2024-08-19T14:38:23.533 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:38:23.533 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.533 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.533 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:38:23.533 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.258319+0000", 2024-08-19T14:38:23.533 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:38:23.533 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.533 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.533 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:38:23.533 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.258320+0000", 2024-08-19T14:38:23.534 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7599999999999997e-07 2024-08-19T14:38:23.534 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.534 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.534 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:38:23.534 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.258367+0000", 2024-08-19T14:38:23.534 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.7895000000000001e-05 2024-08-19T14:38:23.534 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.534 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.534 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:38:23.534 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.258377+0000", 2024-08-19T14:38:23.535 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.9499999999999996e-06 2024-08-19T14:38:23.535 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.535 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.535 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:38:23.535 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.258384+0000", 2024-08-19T14:38:23.535 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.3239999999999999e-06 2024-08-19T14:38:23.535 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.535 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.535 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:38:23.535 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.261676+0000", 2024-08-19T14:38:23.535 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0032921529999999999 2024-08-19T14:38:23.535 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.535 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.536 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:38:23.536 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.261698+0000", 2024-08-19T14:38:23.536 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2260999999999999e-05 2024-08-19T14:38:23.536 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.536 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.536 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:38:23.536 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.313557+0000", 2024-08-19T14:38:23.536 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.051858429999999997 2024-08-19T14:38:23.536 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.536 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.536 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:38:23.537 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.313623+0000", 2024-08-19T14:38:23.537 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.6810999999999994e-05 2024-08-19T14:38:23.537 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:38:23.537 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:38:23.537 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:38:23.537 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.537 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.537 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.16962.0:76288 7.e e618/617 7:70295cf8:::benchmark_data_smithi045_115842_object76287:head v 618'4889, mlcod=605'7)", 2024-08-19T14:38:23.537 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:34:14.261026+0000", 2024-08-19T14:38:23.537 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 249.262651019, 2024-08-19T14:38:23.537 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.052634357999999999, 2024-08-19T14:38:23.537 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:38:23.537 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:38:23.538 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:38:23.538 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:38:23.538 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.538 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:38:23.538 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.261026+0000", 2024-08-19T14:38:23.538 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:38:23.538 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.538 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.538 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:38:23.538 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.261026+0000", 2024-08-19T14:38:23.538 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:38:23.538 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.538 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.538 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:38:23.539 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.261027+0000", 2024-08-19T14:38:23.539 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.5000000000000001e-07 2024-08-19T14:38:23.539 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.539 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.539 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:38:23.539 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.261065+0000", 2024-08-19T14:38:23.539 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.7907000000000001e-05 2024-08-19T14:38:23.539 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.539 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.539 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:38:23.539 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.261079+0000", 2024-08-19T14:38:23.539 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3767e-05 2024-08-19T14:38:23.539 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.539 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.540 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:38:23.540 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.261087+0000", 2024-08-19T14:38:23.540 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.8809999999999996e-06 2024-08-19T14:38:23.540 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.540 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.540 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:38:23.540 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.288780+0000", 2024-08-19T14:38:23.540 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.027693457000000001 2024-08-19T14:38:23.540 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.540 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.540 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:38:23.540 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.288804+0000", 2024-08-19T14:38:23.540 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.3886000000000001e-05 2024-08-19T14:38:23.540 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.541 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.541 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:38:23.541 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.313626+0000", 2024-08-19T14:38:23.541 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.024822331999999999 2024-08-19T14:38:23.541 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.541 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.541 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:38:23.541 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.313661+0000", 2024-08-19T14:38:23.541 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.4177999999999997e-05 2024-08-19T14:38:23.541 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:38:23.541 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:38:23.541 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:38:23.541 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.541 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.542 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.16962.0:76303 7.e e618/617 7:77649f0f:::benchmark_data_smithi045_115842_object76302:head v 618'4890, mlcod=605'7)", 2024-08-19T14:38:23.542 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:34:14.264011+0000", 2024-08-19T14:38:23.542 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 249.25966596999999, 2024-08-19T14:38:23.542 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.049686299000000003, 2024-08-19T14:38:23.542 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:38:23.542 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:38:23.542 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:38:23.542 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:38:23.542 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.542 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:38:23.542 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.264011+0000", 2024-08-19T14:38:23.542 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:38:23.542 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.543 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.543 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:38:23.543 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.264011+0000", 2024-08-19T14:38:23.543 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:38:23.543 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.543 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.543 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:38:23.543 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.264012+0000", 2024-08-19T14:38:23.543 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.16e-07 2024-08-19T14:38:23.543 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.543 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.543 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:38:23.543 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.264059+0000", 2024-08-19T14:38:23.543 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.6900000000000002e-05 2024-08-19T14:38:23.544 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.544 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.544 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:38:23.544 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.264076+0000", 2024-08-19T14:38:23.544 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7248000000000002e-05 2024-08-19T14:38:23.544 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.544 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.544 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:38:23.544 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.264083+0000", 2024-08-19T14:38:23.544 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.5200000000000003e-06 2024-08-19T14:38:23.545 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.545 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.545 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:38:23.545 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.294900+0000", 2024-08-19T14:38:23.545 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.030817157000000001 2024-08-19T14:38:23.545 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.545 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.545 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:38:23.545 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.294931+0000", 2024-08-19T14:38:23.545 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.1043000000000001e-05 2024-08-19T14:38:23.545 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.545 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.545 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:38:23.546 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.313667+0000", 2024-08-19T14:38:23.546 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.018735928999999998 2024-08-19T14:38:23.546 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.546 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.546 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:38:23.546 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.313698+0000", 2024-08-19T14:38:23.546 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.0685999999999997e-05 2024-08-19T14:38:23.546 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:38:23.546 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:38:23.546 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:38:23.546 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.546 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.546 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.16962.0:76625 7.e e618/617 7:789a4a2f:::benchmark_data_smithi045_115842_object76624:head v 618'4910, mlcod=605'15)", 2024-08-19T14:38:23.547 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:34:14.619525+0000", 2024-08-19T14:38:23.547 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 248.90415242700001, 2024-08-19T14:38:23.547 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.047508896000000002, 2024-08-19T14:38:23.547 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:38:23.547 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:38:23.547 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:38:23.547 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:38:23.547 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.547 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:38:23.547 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.619525+0000", 2024-08-19T14:38:23.547 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:38:23.548 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.548 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.548 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:38:23.548 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.619525+0000", 2024-08-19T14:38:23.548 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:38:23.548 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.548 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.548 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:38:23.548 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.619526+0000", 2024-08-19T14:38:23.548 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1310000000000001e-06 2024-08-19T14:38:23.548 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.548 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.548 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:38:23.548 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.619629+0000", 2024-08-19T14:38:23.549 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00010254700000000001 2024-08-19T14:38:23.549 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.549 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.549 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:38:23.549 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.619640+0000", 2024-08-19T14:38:23.549 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.187e-05 2024-08-19T14:38:23.549 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.549 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.549 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:38:23.549 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.619647+0000", 2024-08-19T14:38:23.549 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.5470000000000002e-06 2024-08-19T14:38:23.549 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.549 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.549 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:38:23.550 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.619828+0000", 2024-08-19T14:38:23.550 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00018073699999999999 2024-08-19T14:38:23.550 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.550 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.550 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:38:23.550 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.619894+0000", 2024-08-19T14:38:23.550 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.6544999999999994e-05 2024-08-19T14:38:23.550 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.550 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.550 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:38:23.550 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.666838+0000", 2024-08-19T14:38:23.550 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.046943529999999997 2024-08-19T14:38:23.550 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.550 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.551 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:38:23.551 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:14.667034+0000", 2024-08-19T14:38:23.551 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00019598900000000001 2024-08-19T14:38:23.551 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:38:23.551 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:38:23.551 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:38:23.552 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.552 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.552 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.16962.0:85089 7.f e622/617 7:fb2cc53f:::benchmark_data_smithi045_115842_object85088:head v 622'5295, mlcod=605'208)", 2024-08-19T14:38:23.552 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:34:23.914661+0000", 2024-08-19T14:38:23.552 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 239.609015927, 2024-08-19T14:38:23.552 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.051211051, 2024-08-19T14:38:23.552 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:38:23.552 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:38:23.552 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:38:23.552 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:38:23.552 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.553 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:38:23.553 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:23.914661+0000", 2024-08-19T14:38:23.553 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:38:23.553 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.553 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.553 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:38:23.553 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:23.914661+0000", 2024-08-19T14:38:23.553 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:38:23.553 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.553 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.553 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:38:23.554 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:23.914662+0000", 2024-08-19T14:38:23.554 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.6400000000000003e-07 2024-08-19T14:38:23.554 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.554 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.554 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:38:23.554 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:23.914693+0000", 2024-08-19T14:38:23.554 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.1647999999999999e-05 2024-08-19T14:38:23.554 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.554 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.554 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:38:23.554 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:23.914702+0000", 2024-08-19T14:38:23.554 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.5890000000000003e-06 2024-08-19T14:38:23.554 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.555 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.555 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:38:23.555 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:23.914708+0000", 2024-08-19T14:38:23.555 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.5029999999999997e-06 2024-08-19T14:38:23.555 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.555 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.555 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:38:23.555 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:23.924683+0000", 2024-08-19T14:38:23.555 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0099756689999999995 2024-08-19T14:38:23.555 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.555 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.555 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:38:23.555 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:23.924709+0000", 2024-08-19T14:38:23.555 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.5924000000000001e-05 2024-08-19T14:38:23.556 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.556 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.556 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:38:23.556 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:23.965829+0000", 2024-08-19T14:38:23.556 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.041120205999999999 2024-08-19T14:38:23.556 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.556 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.556 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:38:23.556 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:23.965872+0000", 2024-08-19T14:38:23.556 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.3047999999999998e-05 2024-08-19T14:38:23.556 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:38:23.556 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:38:23.556 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:38:23.557 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.557 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.557 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.16962.0:85096 7.f e622/617 7:fbe5a55f:::benchmark_data_smithi045_115842_object85095:head v 622'5296, mlcod=605'208)", 2024-08-19T14:38:23.557 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:34:23.918667+0000", 2024-08-19T14:38:23.557 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 239.605010037, 2024-08-19T14:38:23.557 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.047242412999999997, 2024-08-19T14:38:23.557 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:38:23.557 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:38:23.557 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:38:23.557 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:38:23.557 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.557 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:38:23.558 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:23.918667+0000", 2024-08-19T14:38:23.558 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:38:23.558 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.559 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.559 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:38:23.559 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:23.918667+0000", 2024-08-19T14:38:23.559 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:38:23.559 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.559 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.559 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:38:23.559 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:23.918668+0000", 2024-08-19T14:38:23.559 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7599999999999997e-07 2024-08-19T14:38:23.559 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.559 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.559 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:38:23.559 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:23.918700+0000", 2024-08-19T14:38:23.559 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.2258999999999997e-05 2024-08-19T14:38:23.560 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.560 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.560 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:38:23.560 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:23.918712+0000", 2024-08-19T14:38:23.560 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1620000000000001e-05 2024-08-19T14:38:23.560 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.560 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.560 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:38:23.560 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:23.918719+0000", 2024-08-19T14:38:23.560 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.8920000000000003e-06 2024-08-19T14:38:23.561 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.561 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.561 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:38:23.561 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:23.930848+0000", 2024-08-19T14:38:23.561 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.012129423 2024-08-19T14:38:23.561 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.561 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.561 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:38:23.561 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:23.930871+0000", 2024-08-19T14:38:23.561 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2719000000000001e-05 2024-08-19T14:38:23.561 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.561 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.561 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:38:23.562 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:23.965882+0000", 2024-08-19T14:38:23.562 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.035011476999999999 2024-08-19T14:38:23.562 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.562 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.562 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:38:23.562 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:23.965910+0000", 2024-08-19T14:38:23.562 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.7447e-05 2024-08-19T14:38:23.562 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:38:23.562 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:38:23.562 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:38:23.562 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.563 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.563 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.16962.0:105364 7.f e627/617 7:f24bfbcd:::benchmark_data_smithi045_115842_object105363:head v 627'6473, mlcod=607'833)", 2024-08-19T14:38:23.563 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:34:47.948278+0000", 2024-08-19T14:38:23.563 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 215.575399514, 2024-08-19T14:38:23.563 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.067695303999999998, 2024-08-19T14:38:23.563 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:38:23.564 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:38:23.564 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:38:23.564 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:38:23.564 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.564 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:38:23.564 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.948278+0000", 2024-08-19T14:38:23.565 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:38:23.565 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.565 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.565 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:38:23.565 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.948278+0000", 2024-08-19T14:38:23.565 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:38:23.566 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.566 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.566 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:38:23.566 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.948279+0000", 2024-08-19T14:38:23.566 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.1200000000000002e-07 2024-08-19T14:38:23.566 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.566 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.566 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:38:23.566 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.948321+0000", 2024-08-19T14:38:23.566 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.2849999999999998e-05 2024-08-19T14:38:23.567 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.567 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.567 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:38:23.567 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.948336+0000", 2024-08-19T14:38:23.567 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4107999999999999e-05 2024-08-19T14:38:23.567 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.567 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.567 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:38:23.567 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.948343+0000", 2024-08-19T14:38:23.567 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.0840000000000003e-06 2024-08-19T14:38:23.567 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.567 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.568 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:38:23.568 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.949790+0000", 2024-08-19T14:38:23.568 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001447776 2024-08-19T14:38:23.568 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.568 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.568 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:38:23.568 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.949818+0000", 2024-08-19T14:38:23.568 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.7294999999999999e-05 2024-08-19T14:38:23.568 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.568 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.568 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:38:23.568 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.015860+0000", 2024-08-19T14:38:23.569 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.066042187000000002 2024-08-19T14:38:23.569 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.569 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.569 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:38:23.569 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.015973+0000", 2024-08-19T14:38:23.569 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000113192 2024-08-19T14:38:23.569 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:38:23.569 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:38:23.569 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:38:23.569 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.569 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.569 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.16962.0:105377 7.f e627/617 7:f91fc140:::benchmark_data_smithi045_115842_object105376:head v 627'6474, mlcod=607'833)", 2024-08-19T14:38:23.569 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:34:47.951714+0000", 2024-08-19T14:38:23.569 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 215.57196344100001, 2024-08-19T14:38:23.570 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.064366294000000004, 2024-08-19T14:38:23.570 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:38:23.570 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:38:23.570 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:38:23.570 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:38:23.570 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.570 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:38:23.570 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.951714+0000", 2024-08-19T14:38:23.570 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:38:23.570 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.570 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.571 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:38:23.571 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.951714+0000", 2024-08-19T14:38:23.571 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:38:23.571 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.571 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.571 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:38:23.572 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.951721+0000", 2024-08-19T14:38:23.572 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.2339999999999997e-06 2024-08-19T14:38:23.572 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.572 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.572 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:38:23.573 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.951750+0000", 2024-08-19T14:38:23.573 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.8495000000000001e-05 2024-08-19T14:38:23.573 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.573 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.573 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:38:23.573 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.951765+0000", 2024-08-19T14:38:23.573 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5506e-05 2024-08-19T14:38:23.573 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.573 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.573 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:38:23.573 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.951772+0000", 2024-08-19T14:38:23.574 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.3139999999999998e-06 2024-08-19T14:38:23.574 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.574 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.574 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:38:23.574 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.956679+0000", 2024-08-19T14:38:23.574 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0049063099999999997 2024-08-19T14:38:23.574 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.574 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.574 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:38:23.574 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.956786+0000", 2024-08-19T14:38:23.574 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000107295 2024-08-19T14:38:23.574 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.574 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.574 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:38:23.575 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.015978+0000", 2024-08-19T14:38:23.575 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.059191524000000002 2024-08-19T14:38:23.575 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.575 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.575 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:38:23.575 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.016080+0000", 2024-08-19T14:38:23.575 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00010261599999999999 2024-08-19T14:38:23.575 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:38:23.575 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:38:23.575 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:38:23.575 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.575 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.575 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.16962.0:105379 7.f e627/617 7:f49d0828:::benchmark_data_smithi045_115842_object105378:head v 627'6475, mlcod=607'833)", 2024-08-19T14:38:23.575 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:34:47.956593+0000", 2024-08-19T14:38:23.576 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 215.567083844, 2024-08-19T14:38:23.576 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.069375785999999995, 2024-08-19T14:38:23.576 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:38:23.576 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:38:23.576 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:38:23.576 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:38:23.576 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.576 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:38:23.576 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.956593+0000", 2024-08-19T14:38:23.576 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:38:23.576 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.576 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.576 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:38:23.576 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.956593+0000", 2024-08-19T14:38:23.577 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:38:23.577 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.577 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.577 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:38:23.577 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.956595+0000", 2024-08-19T14:38:23.577 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1289999999999999e-06 2024-08-19T14:38:23.577 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.577 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.577 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:38:23.577 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.956637+0000", 2024-08-19T14:38:23.577 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.2800999999999999e-05 2024-08-19T14:38:23.577 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.577 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.578 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:38:23.578 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.956652+0000", 2024-08-19T14:38:23.578 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4129e-05 2024-08-19T14:38:23.578 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.578 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.578 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:38:23.578 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.956661+0000", 2024-08-19T14:38:23.578 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.7559999999999994e-06 2024-08-19T14:38:23.578 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.578 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.578 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:38:23.578 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.964123+0000", 2024-08-19T14:38:23.578 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0074622000000000004 2024-08-19T14:38:23.579 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.579 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.579 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:38:23.579 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.964187+0000", 2024-08-19T14:38:23.579 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.3362999999999996e-05 2024-08-19T14:38:23.579 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.579 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.579 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:38:23.579 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.025913+0000", 2024-08-19T14:38:23.579 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.061725976000000002 2024-08-19T14:38:23.579 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.579 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.579 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:38:23.579 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.025969+0000", 2024-08-19T14:38:23.580 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.6431999999999998e-05 2024-08-19T14:38:23.580 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:38:23.580 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:38:23.580 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:38:23.580 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.580 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.580 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.16962.0:105380 7.f e627/617 7:fd82eef6:::benchmark_data_smithi045_115842_object105379:head v 627'6476, mlcod=607'833)", 2024-08-19T14:38:23.580 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:34:47.958978+0000", 2024-08-19T14:38:23.580 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 215.564699023, 2024-08-19T14:38:23.580 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.067030355, 2024-08-19T14:38:23.580 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:38:23.580 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:38:23.580 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:38:23.580 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:38:23.581 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.581 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:38:23.581 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.958978+0000", 2024-08-19T14:38:23.581 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:38:23.581 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.581 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.581 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:38:23.581 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.958978+0000", 2024-08-19T14:38:23.581 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:38:23.581 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.581 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.581 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:38:23.581 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.958979+0000", 2024-08-19T14:38:23.581 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.1299999999999999e-07 2024-08-19T14:38:23.582 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.582 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.582 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:38:23.582 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.959032+0000", 2024-08-19T14:38:23.582 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.3072999999999998e-05 2024-08-19T14:38:23.582 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.582 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.582 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:38:23.582 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.959046+0000", 2024-08-19T14:38:23.582 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3702e-05 2024-08-19T14:38:23.582 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.582 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.582 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:38:23.583 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.959053+0000", 2024-08-19T14:38:23.583 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.6579999999999996e-06 2024-08-19T14:38:23.583 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.583 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.583 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:38:23.583 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.974551+0000", 2024-08-19T14:38:23.583 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.015498168 2024-08-19T14:38:23.583 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.583 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.583 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:38:23.583 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.974576+0000", 2024-08-19T14:38:23.583 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.5502000000000001e-05 2024-08-19T14:38:23.583 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.583 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.584 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:38:23.584 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.025972+0000", 2024-08-19T14:38:23.584 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.051395733999999998 2024-08-19T14:38:23.584 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.584 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.584 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:38:23.584 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.026009+0000", 2024-08-19T14:38:23.584 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.6705000000000002e-05 2024-08-19T14:38:23.584 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:38:23.584 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:38:23.584 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:38:23.584 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.585 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.585 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.16962.0:105387 7.f e627/617 7:f7f9de62:::benchmark_data_smithi045_115842_object105386:head v 627'6477, mlcod=607'833)", 2024-08-19T14:38:23.585 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:34:47.962787+0000", 2024-08-19T14:38:23.585 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 215.56089054899999, 2024-08-19T14:38:23.585 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.063250878999999996, 2024-08-19T14:38:23.585 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:38:23.585 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:38:23.585 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:38:23.585 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:38:23.585 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.585 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:38:23.585 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.962787+0000", 2024-08-19T14:38:23.586 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:38:23.586 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.586 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.586 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:38:23.586 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.962787+0000", 2024-08-19T14:38:23.586 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:38:23.586 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.586 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.586 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:38:23.586 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.962788+0000", 2024-08-19T14:38:23.586 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0210000000000001e-06 2024-08-19T14:38:23.586 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.586 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.587 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:38:23.587 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.962824+0000", 2024-08-19T14:38:23.587 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.6328999999999997e-05 2024-08-19T14:38:23.587 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.587 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.587 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:38:23.587 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.962841+0000", 2024-08-19T14:38:23.587 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6858e-05 2024-08-19T14:38:23.587 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.587 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.587 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:38:23.587 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.962850+0000", 2024-08-19T14:38:23.587 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.8380000000000008e-06 2024-08-19T14:38:23.587 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.588 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.588 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:38:23.588 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.980518+0000", 2024-08-19T14:38:23.588 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.017668120999999998 2024-08-19T14:38:23.588 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.588 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.588 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:38:23.588 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.980548+0000", 2024-08-19T14:38:23.588 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.9825e-05 2024-08-19T14:38:23.588 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.588 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.588 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:38:23.588 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.026011+0000", 2024-08-19T14:38:23.589 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.045463389999999999 2024-08-19T14:38:23.589 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.589 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.589 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:38:23.589 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.026038+0000", 2024-08-19T14:38:23.589 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.6497000000000001e-05 2024-08-19T14:38:23.589 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:38:23.589 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:38:23.589 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:38:23.589 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.589 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.589 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.16962.0:105389 7.f e627/617 7:fe4b891a:::benchmark_data_smithi045_115842_object105388:head v 627'6478, mlcod=607'833)", 2024-08-19T14:38:23.589 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:34:47.971841+0000", 2024-08-19T14:38:23.590 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 215.551835931, 2024-08-19T14:38:23.590 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.054228620999999998, 2024-08-19T14:38:23.590 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:38:23.590 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:38:23.590 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:38:23.590 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:38:23.590 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.590 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:38:23.590 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.971841+0000", 2024-08-19T14:38:23.590 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:38:23.590 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.590 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.590 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:38:23.591 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.971841+0000", 2024-08-19T14:38:23.591 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:38:23.591 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.591 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.591 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:38:23.591 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.971842+0000", 2024-08-19T14:38:23.591 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.2500000000000004e-07 2024-08-19T14:38:23.591 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.591 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.591 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:38:23.591 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.971882+0000", 2024-08-19T14:38:23.591 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.0009999999999998e-05 2024-08-19T14:38:23.591 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.592 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.592 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:38:23.592 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.971900+0000", 2024-08-19T14:38:23.592 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7827000000000001e-05 2024-08-19T14:38:23.592 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.592 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.592 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:38:23.592 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.971908+0000", 2024-08-19T14:38:23.592 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.2060000000000007e-06 2024-08-19T14:38:23.592 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.592 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.592 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:38:23.592 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.986839+0000", 2024-08-19T14:38:23.592 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.014930986 2024-08-19T14:38:23.593 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.593 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.593 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:38:23.593 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.986866+0000", 2024-08-19T14:38:23.593 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.6441999999999999e-05 2024-08-19T14:38:23.593 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.593 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.593 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:38:23.593 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.026040+0000", 2024-08-19T14:38:23.593 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.039174266999999999 2024-08-19T14:38:23.593 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.593 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.593 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:38:23.593 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.026070+0000", 2024-08-19T14:38:23.594 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.0057999999999999e-05 2024-08-19T14:38:23.594 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:38:23.594 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:38:23.594 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:38:23.594 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.594 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.594 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.16962.0:105392 7.f e627/617 7:ff34f05a:::benchmark_data_smithi045_115842_object105391:head v 627'6479, mlcod=607'833)", 2024-08-19T14:38:23.594 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:34:47.975534+0000", 2024-08-19T14:38:23.594 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 215.548142872, 2024-08-19T14:38:23.594 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.050567490999999999, 2024-08-19T14:38:23.594 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:38:23.594 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:38:23.594 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:38:23.595 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:38:23.595 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.595 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:38:23.595 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.975534+0000", 2024-08-19T14:38:23.595 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:38:23.595 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.595 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.595 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:38:23.595 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.975534+0000", 2024-08-19T14:38:23.595 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:38:23.595 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.595 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.595 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:38:23.595 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.975536+0000", 2024-08-19T14:38:23.596 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3799999999999999e-06 2024-08-19T14:38:23.596 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.596 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.596 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:38:23.596 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.975580+0000", 2024-08-19T14:38:23.596 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.4029000000000001e-05 2024-08-19T14:38:23.596 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.596 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.596 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:38:23.596 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.975600+0000", 2024-08-19T14:38:23.596 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0081999999999998e-05 2024-08-19T14:38:23.596 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.596 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.596 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:38:23.597 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.975609+0000", 2024-08-19T14:38:23.597 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.1290000000000002e-06 2024-08-19T14:38:23.597 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.597 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.597 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:38:23.597 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.993986+0000", 2024-08-19T14:38:23.597 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.018377370000000001 2024-08-19T14:38:23.597 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.598 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.598 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:38:23.598 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.994018+0000", 2024-08-19T14:38:23.598 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.1971000000000003e-05 2024-08-19T14:38:23.598 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.598 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.599 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:38:23.599 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.026072+0000", 2024-08-19T14:38:23.599 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.032053566999999998 2024-08-19T14:38:23.599 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.599 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.599 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:38:23.600 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.026102+0000", 2024-08-19T14:38:23.600 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.9963000000000001e-05 2024-08-19T14:38:23.600 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:38:23.600 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:38:23.600 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:38:23.600 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.600 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.601 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.16962.0:105397 7.f e627/617 7:f80187d8:::benchmark_data_smithi045_115842_object105396:head v 627'6480, mlcod=607'833)", 2024-08-19T14:38:23.601 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:34:47.978095+0000", 2024-08-19T14:38:23.601 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 215.545582325, 2024-08-19T14:38:23.601 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.048037879999999998, 2024-08-19T14:38:23.601 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:38:23.601 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:38:23.602 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:38:23.602 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:38:23.602 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.602 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:38:23.602 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.978095+0000", 2024-08-19T14:38:23.602 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:38:23.603 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.603 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.603 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:38:23.603 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.978095+0000", 2024-08-19T14:38:23.603 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:38:23.604 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.604 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.604 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:38:23.604 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.978096+0000", 2024-08-19T14:38:23.604 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0589999999999999e-06 2024-08-19T14:38:23.604 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.605 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.605 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:38:23.605 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.978132+0000", 2024-08-19T14:38:23.605 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.6171999999999999e-05 2024-08-19T14:38:23.605 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.605 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.606 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:38:23.606 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.978152+0000", 2024-08-19T14:38:23.606 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9639e-05 2024-08-19T14:38:23.606 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.606 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.606 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:38:23.607 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:47.978161+0000", 2024-08-19T14:38:23.607 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.1290000000000002e-06 2024-08-19T14:38:23.607 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.607 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.607 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:38:23.607 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.000277+0000", 2024-08-19T14:38:23.608 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.022116327000000002 2024-08-19T14:38:23.608 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.608 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.608 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:38:23.608 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.000308+0000", 2024-08-19T14:38:23.608 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.1078000000000003e-05 2024-08-19T14:38:23.609 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.609 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.609 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:38:23.609 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.026104+0000", 2024-08-19T14:38:23.609 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.025795241 2024-08-19T14:38:23.609 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.610 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.610 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:38:23.610 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.026133+0000", 2024-08-19T14:38:23.610 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.9235e-05 2024-08-19T14:38:23.610 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:38:23.611 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:38:23.611 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:38:23.611 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.611 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.611 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.16962.0:106074 7.a e627/617 7:59660197:::benchmark_data_smithi045_115842_object106073:head v 627'6600, mlcod=608'1547)", 2024-08-19T14:38:23.611 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:34:48.721689+0000", 2024-08-19T14:38:23.612 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 214.80198875599999, 2024-08-19T14:38:23.612 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.052407031999999999, 2024-08-19T14:38:23.612 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:38:23.612 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:38:23.612 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:38:23.612 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:38:23.612 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.613 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:38:23.613 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.721689+0000", 2024-08-19T14:38:23.613 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:38:23.613 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.613 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.613 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:38:23.613 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.721689+0000", 2024-08-19T14:38:23.614 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:38:23.614 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.614 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.614 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:38:23.614 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.721689+0000", 2024-08-19T14:38:23.614 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.2099999999999996e-07 2024-08-19T14:38:23.614 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.614 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.614 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:38:23.614 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.721712+0000", 2024-08-19T14:38:23.615 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2974999999999999e-05 2024-08-19T14:38:23.615 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.615 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.615 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:38:23.615 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.721721+0000", 2024-08-19T14:38:23.615 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.6379999999999993e-06 2024-08-19T14:38:23.615 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.615 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.615 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:38:23.616 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:38:23.617 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.721726+0000", 2024-08-19T14:38:23.617 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.203e-06 2024-08-19T14:38:23.617 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.617 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.617 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:38:23.617 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.721937+0000", 2024-08-19T14:38:23.617 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00021136399999999999 2024-08-19T14:38:23.617 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.618 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.618 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:38:23.618 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.721957+0000", 2024-08-19T14:38:23.618 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9609000000000001e-05 2024-08-19T14:38:23.618 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.619 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.619 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:38:23.619 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.773975+0000", 2024-08-19T14:38:23.619 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.052018350999999997 2024-08-19T14:38:23.619 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.619 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.620 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:38:23.620 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.774096+0000", 2024-08-19T14:38:23.620 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000120271 2024-08-19T14:38:23.620 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:38:23.620 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:38:23.620 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:38:23.620 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.621 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.621 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.16962.0:106077 7.a e627/617 7:5eb1cc5e:::benchmark_data_smithi045_115842_object106076:head v 627'6601, mlcod=608'1547)", 2024-08-19T14:38:23.621 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:34:48.725534+0000", 2024-08-19T14:38:23.621 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 214.798143783, 2024-08-19T14:38:23.621 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.052283246999999998, 2024-08-19T14:38:23.621 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:38:23.621 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:38:23.622 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:38:23.622 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:38:23.622 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.622 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:38:23.622 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.725534+0000", 2024-08-19T14:38:23.622 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:38:23.622 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.623 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.623 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:38:23.623 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.725534+0000", 2024-08-19T14:38:23.623 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:38:23.623 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.623 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.624 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:38:23.624 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.725535+0000", 2024-08-19T14:38:23.624 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.356e-06 2024-08-19T14:38:23.624 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.624 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.624 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:38:23.625 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.725563+0000", 2024-08-19T14:38:23.625 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.8135000000000001e-05 2024-08-19T14:38:23.625 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.625 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.625 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:38:23.625 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.725577+0000", 2024-08-19T14:38:23.626 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4166e-05 2024-08-19T14:38:23.626 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.626 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.626 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:38:23.626 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.725586+0000", 2024-08-19T14:38:23.626 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.1409999999999994e-06 2024-08-19T14:38:23.626 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.626 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.626 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:38:23.626 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.728894+0000", 2024-08-19T14:38:23.627 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0033076749999999999 2024-08-19T14:38:23.627 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.627 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.627 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:38:23.627 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.728919+0000", 2024-08-19T14:38:23.627 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.5488000000000001e-05 2024-08-19T14:38:23.627 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.627 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.627 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:38:23.627 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.777773+0000", 2024-08-19T14:38:23.627 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.048853079000000001 2024-08-19T14:38:23.627 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.627 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.627 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:38:23.628 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.777817+0000", 2024-08-19T14:38:23.628 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.4206999999999999e-05 2024-08-19T14:38:23.628 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:38:23.628 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:38:23.628 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:38:23.628 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.628 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.628 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.16962.0:106082 7.5 e627/617 7:a84af00f:::benchmark_data_smithi045_115842_object106081:head v 627'6560, mlcod=608'1460)", 2024-08-19T14:38:23.628 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:34:48.729996+0000", 2024-08-19T14:38:23.628 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 214.793680932, 2024-08-19T14:38:23.628 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.051991066000000002, 2024-08-19T14:38:23.628 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:38:23.628 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:38:23.629 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:38:23.629 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:38:23.629 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.629 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:38:23.629 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.729996+0000", 2024-08-19T14:38:23.629 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:38:23.629 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.629 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.629 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:38:23.629 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.729996+0000", 2024-08-19T14:38:23.629 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:38:23.629 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.629 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.629 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:38:23.630 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.729997+0000", 2024-08-19T14:38:23.630 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.1500000000000004e-07 2024-08-19T14:38:23.630 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.630 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.630 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:38:23.630 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.730022+0000", 2024-08-19T14:38:23.630 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.5282e-05 2024-08-19T14:38:23.630 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.630 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.630 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:38:23.630 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.730033+0000", 2024-08-19T14:38:23.630 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0370000000000001e-05 2024-08-19T14:38:23.631 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.631 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.631 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:38:23.631 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.730039+0000", 2024-08-19T14:38:23.631 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.6699999999999997e-06 2024-08-19T14:38:23.631 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.631 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.631 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:38:23.631 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.732462+0000", 2024-08-19T14:38:23.631 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0024221939999999999 2024-08-19T14:38:23.631 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.631 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.631 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:38:23.631 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.732487+0000", 2024-08-19T14:38:23.632 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.5083e-05 2024-08-19T14:38:23.632 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.632 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.632 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:38:23.632 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.781953+0000", 2024-08-19T14:38:23.632 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.049466036999999997 2024-08-19T14:38:23.632 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.632 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.632 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:38:23.632 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.781987+0000", 2024-08-19T14:38:23.632 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.4715000000000002e-05 2024-08-19T14:38:23.632 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:38:23.632 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:38:23.633 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:38:23.633 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.633 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.633 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.16962.0:106085 7.0 e627/617 7:0349f56d:::benchmark_data_smithi045_115842_object106084:head v 627'6486, mlcod=607'921)", 2024-08-19T14:38:23.633 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:34:48.734348+0000", 2024-08-19T14:38:23.633 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 214.78932943800001, 2024-08-19T14:38:23.633 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.061543685000000001, 2024-08-19T14:38:23.633 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:38:23.633 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:38:23.633 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:38:23.633 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:38:23.633 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.633 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:38:23.633 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.734348+0000", 2024-08-19T14:38:23.634 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:38:23.634 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.634 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.634 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:38:23.634 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.734348+0000", 2024-08-19T14:38:23.634 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:38:23.634 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.634 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.634 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:38:23.634 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.734349+0000", 2024-08-19T14:38:23.634 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.6199999999999996e-07 2024-08-19T14:38:23.634 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.634 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.635 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:38:23.635 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.734410+0000", 2024-08-19T14:38:23.635 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.1273999999999996e-05 2024-08-19T14:38:23.635 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.635 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.635 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:38:23.635 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.734420+0000", 2024-08-19T14:38:23.635 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.027e-05 2024-08-19T14:38:23.635 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.635 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.635 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:38:23.635 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.734427+0000", 2024-08-19T14:38:23.635 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.6279999999999999e-06 2024-08-19T14:38:23.635 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.636 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.636 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:38:23.636 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.734476+0000", 2024-08-19T14:38:23.636 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.9348999999999998e-05 2024-08-19T14:38:23.636 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.636 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.636 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:38:23.636 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.734499+0000", 2024-08-19T14:38:23.636 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2558e-05 2024-08-19T14:38:23.636 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.637 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.637 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:38:23.637 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.795828+0000", 2024-08-19T14:38:23.637 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.061329231999999997 2024-08-19T14:38:23.637 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.637 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.637 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:38:23.637 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.795892+0000", 2024-08-19T14:38:23.637 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.3511999999999998e-05 2024-08-19T14:38:23.637 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:38:23.637 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:38:23.637 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:38:23.637 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.638 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.638 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.16962.0:106086 7.5 e627/617 7:aecea1dc:::benchmark_data_smithi045_115842_object106085:head v 627'6561, mlcod=608'1460)", 2024-08-19T14:38:23.638 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:34:48.734484+0000", 2024-08-19T14:38:23.638 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 214.789193068, 2024-08-19T14:38:23.638 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.058300642999999999, 2024-08-19T14:38:23.638 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:38:23.638 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:38:23.638 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:38:23.638 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:38:23.638 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.638 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:38:23.638 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.734484+0000", 2024-08-19T14:38:23.638 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:38:23.639 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.639 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.639 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:38:23.639 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.734484+0000", 2024-08-19T14:38:23.639 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:38:23.639 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.639 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.639 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:38:23.639 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.734485+0000", 2024-08-19T14:38:23.639 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.5999999999999999e-07 2024-08-19T14:38:23.639 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.639 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.639 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:38:23.640 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.734507+0000", 2024-08-19T14:38:23.640 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1950000000000002e-05 2024-08-19T14:38:23.640 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.640 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.640 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:38:23.640 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.734515+0000", 2024-08-19T14:38:23.640 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.405e-06 2024-08-19T14:38:23.640 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.640 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.640 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:38:23.641 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.734520+0000", 2024-08-19T14:38:23.641 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.3449999999999996e-06 2024-08-19T14:38:23.641 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.641 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.641 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:38:23.641 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.777620+0000", 2024-08-19T14:38:23.641 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.043100029999999998 2024-08-19T14:38:23.641 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.641 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.641 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:38:23.641 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.777648+0000", 2024-08-19T14:38:23.641 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.7144e-05 2024-08-19T14:38:23.641 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.642 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.642 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:38:23.642 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.792746+0000", 2024-08-19T14:38:23.642 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.015098741000000001 2024-08-19T14:38:23.642 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:23.642 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:23.642 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:38:23.642 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:48.792785+0000", 2024-08-19T14:38:23.642 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.8568000000000001e-05 2024-08-19T14:38:23.642 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:38:23.642 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:38:23.642 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:38:23.642 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:38:23.643 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:38:23.643 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:38:23.666 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:23.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:23.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:38:23.767 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:38:23.788 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:23.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:23.789 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:38:23.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:23 smithi045 ceph-mon[20143]: pgmap v1644: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.3 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 461 op/s; 16297/115308 objects degraded (14.133%); 4524/115308 objects misplaced (3.923%); 5.8 MiB/s, 137 objects/s recovering 2024-08-19T14:38:23.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:23 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 16297/115308 objects degraded (14.133%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-19T14:38:23.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:23 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:38:23.809 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:23 smithi032 ceph-mon[20004]: pgmap v1644: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.3 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 461 op/s; 16297/115308 objects degraded (14.133%); 4524/115308 objects misplaced (3.923%); 5.8 MiB/s, 137 objects/s recovering 2024-08-19T14:38:23.809 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:23 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 16297/115308 objects degraded (14.133%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-19T14:38:23.809 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:23 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:38:23.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:23 smithi031 ceph-mon[16336]: pgmap v1644: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.3 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 461 op/s; 16297/115308 objects degraded (14.133%); 4524/115308 objects misplaced (3.923%); 5.8 MiB/s, 137 objects/s recovering 2024-08-19T14:38:23.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:23 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 16297/115308 objects degraded (14.133%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-19T14:38:23.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:23 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:38:23.890 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:38:23.916 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:23.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:23.917 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:38:24.018 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:38:24.079 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:24.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:24.080 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:38:24.181 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:38:24.207 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:24.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:24.208 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:38:24.309 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:38:24.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:24.332 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:24.333 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:38:24.434 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:38:24.471 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:24.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:24.472 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:38:24.573 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:38:24.612 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:24.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:24.616 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:38:24.710 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-08-19T14:38:24.717 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:38:24.776 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:24.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:24.777 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:38:24.878 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:38:24.936 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:24.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:24.937 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:38:25.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:24 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:38:25.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:24 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:38:25.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:24 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:38:25.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:24 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:38:25.038 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:38:25.041 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:38:25.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:24 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:38:25.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:24 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:38:25.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:24 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:38:25.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:24 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:38:25.065 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:25.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:25.067 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:38:25.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:24 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:38:25.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:24 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:38:25.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:24 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:38:25.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:24 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:38:25.167 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:38:25.189 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:25.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:25.190 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:38:25.291 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:38:25.336 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:25.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:25.337 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:38:25.437 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:38:25.463 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:25.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:25.464 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:38:25.543 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-19T14:38:25.564 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:38:25.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:25.587 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:25.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:38:25.688 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:38:25.745 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:25.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:25.746 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:38:25.847 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:38:25.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:25 smithi031 ceph-mon[16336]: pgmap v1645: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.3 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 459 op/s; 16347/113421 objects degraded (14.413%); 4426/113421 objects misplaced (3.902%); 5.2 MiB/s, 131 objects/s recovering 2024-08-19T14:38:25.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:25 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:38:25.868 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:25 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:38:25.868 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:25 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:38:25.869 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:25.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:25.870 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:38:25.971 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:38:25.993 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:25.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:25.995 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:38:26.001 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:38:26.005 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:25 smithi045 ceph-mon[20143]: pgmap v1645: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.3 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 459 op/s; 16347/113421 objects degraded (14.413%); 4426/113421 objects misplaced (3.902%); 5.2 MiB/s, 131 objects/s recovering 2024-08-19T14:38:26.005 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:25 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:38:26.005 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:25 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:38:26.005 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:25 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:38:26.005 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:38:26.005 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [ 2024-08-19T14:38:26.005 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:26.006 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.16962.0:200788 7.a 7:5ed20f97:::benchmark_data_smithi045_115842_object82628:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e690)", 2024-08-19T14:38:26.006 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:38:25.615396+0000", 2024-08-19T14:38:26.006 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.015690382999999999, 2024-08-19T14:38:26.006 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.015714101000000001, 2024-08-19T14:38:26.006 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:38:26.006 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:38:26.006 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "waiting for sub ops", 2024-08-19T14:38:26.006 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:38:26.006 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.16962", 2024-08-19T14:38:26.006 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:38:26.006 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 200788 2024-08-19T14:38:26.006 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:26.007 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:38:26.007 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:26.007 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:38:26.007 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:25.615396+0000", 2024-08-19T14:38:26.007 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:38:26.007 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:26.007 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:26.007 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:38:26.007 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:25.615396+0000", 2024-08-19T14:38:26.007 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:38:26.007 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:26.007 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:26.008 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:38:26.008 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:25.615397+0000", 2024-08-19T14:38:26.008 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.0999999999999997e-07 2024-08-19T14:38:26.008 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:26.008 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:26.008 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:38:26.008 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:25.615401+0000", 2024-08-19T14:38:26.009 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.3970000000000004e-06 2024-08-19T14:38:26.009 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:26.009 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:26.009 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:38:26.009 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:25.615413+0000", 2024-08-19T14:38:26.009 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2194999999999999e-05 2024-08-19T14:38:26.009 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:26.009 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:26.009 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:38:26.009 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:25.615420+0000", 2024-08-19T14:38:26.009 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.0230000000000004e-06 2024-08-19T14:38:26.009 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:26.010 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:26.010 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:38:26.010 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:25.621861+0000", 2024-08-19T14:38:26.010 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0064407140000000002 2024-08-19T14:38:26.010 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:26.010 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:26.010 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:38:26.010 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:25.622041+0000", 2024-08-19T14:38:26.010 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00017962800000000001 2024-08-19T14:38:26.010 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:26.011 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:26.011 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2,9", 2024-08-19T14:38:26.011 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:25.622219+0000", 2024-08-19T14:38:26.011 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000177935 2024-08-19T14:38:26.011 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:26.011 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:26.011 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:38:26.011 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:25.629386+0000", 2024-08-19T14:38:26.011 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0071674809999999999 2024-08-19T14:38:26.011 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:26.011 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:26.012 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:38:26.012 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:25.629435+0000", 2024-08-19T14:38:26.012 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.9069000000000001e-05 2024-08-19T14:38:26.012 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:38:26.012 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:38:26.012 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:38:26.012 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:26.012 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:26.012 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.16962.0:200786 7.a 7:52a1c6e1:::benchmark_data_smithi045_115842_object82626:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e690)", 2024-08-19T14:38:26.012 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:38:25.615108+0000", 2024-08-19T14:38:26.012 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.015978315999999999, 2024-08-19T14:38:26.013 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.016056712000000001, 2024-08-19T14:38:26.013 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:38:26.013 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:38:26.013 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "waiting for sub ops", 2024-08-19T14:38:26.013 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:38:26.013 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.16962", 2024-08-19T14:38:26.013 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:38:26.013 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 200786 2024-08-19T14:38:26.013 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:26.013 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:38:26.013 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:26.013 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:38:26.013 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:25.615108+0000", 2024-08-19T14:38:26.014 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:38:26.014 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:26.014 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:26.014 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:38:26.014 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:25.615108+0000", 2024-08-19T14:38:26.014 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:38:26.014 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:26.014 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:26.014 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:38:26.014 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:25.615109+0000", 2024-08-19T14:38:26.014 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.065e-06 2024-08-19T14:38:26.014 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:26.015 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:26.015 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:38:26.015 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:25.615113+0000", 2024-08-19T14:38:26.015 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.9179999999999996e-06 2024-08-19T14:38:26.015 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:26.015 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:26.015 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:38:26.015 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:25.615125+0000", 2024-08-19T14:38:26.015 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1766000000000001e-05 2024-08-19T14:38:26.015 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:26.015 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:26.015 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:38:26.015 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:25.615132+0000", 2024-08-19T14:38:26.016 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.8220000000000007e-06 2024-08-19T14:38:26.016 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:26.016 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:26.016 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:38:26.016 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:25.617028+0000", 2024-08-19T14:38:26.016 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0018958340000000001 2024-08-19T14:38:26.016 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:26.016 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:26.016 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:38:26.016 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:25.617172+0000", 2024-08-19T14:38:26.016 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014326200000000001 2024-08-19T14:38:26.016 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:26.017 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:26.017 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2,9", 2024-08-19T14:38:26.017 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:25.617319+0000", 2024-08-19T14:38:26.017 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000147505 2024-08-19T14:38:26.017 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:26.017 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:26.017 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:38:26.017 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:25.629373+0000", 2024-08-19T14:38:26.017 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.012054436999999999 2024-08-19T14:38:26.017 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:26.017 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:26.017 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:38:26.018 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:25.629383+0000", 2024-08-19T14:38:26.018 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.1090000000000004e-06 2024-08-19T14:38:26.018 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:38:26.018 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:38:26.018 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:38:26.018 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:38:26.018 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-08-19T14:38:26.018 INFO:teuthology.orchestra.run.smithi045.stdout: "num_ops": 2 2024-08-19T14:38:26.018 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:38:26.019 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:25 smithi032 ceph-mon[20004]: pgmap v1645: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.3 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 459 op/s; 16347/113421 objects degraded (14.413%); 4426/113421 objects misplaced (3.902%); 5.2 MiB/s, 131 objects/s recovering 2024-08-19T14:38:26.019 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:25 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:38:26.019 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:25 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:38:26.019 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:25 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:38:26.096 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:38:26.140 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:26.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:26.141 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:38:26.242 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:38:26.265 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:26.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:26.266 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:38:26.367 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:38:26.413 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:26.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:26.414 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:38:26.515 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:38:26.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:26.551 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:26.552 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:38:26.652 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:38:26.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:26.695 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:26.696 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:38:26.796 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:38:26.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:26.820 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:26.820 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:38:26.921 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:38:26.965 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:26.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:26.967 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:38:27.046 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:38:27.047 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-08-19T14:38:27.068 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:38:27.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:27.114 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:27.114 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:38:27.215 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:38:27.250 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:27.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:27.251 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:38:27.342 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:38:27.352 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:38:27.373 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:27.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:27.375 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:38:27.475 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:38:27.498 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:27.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:27.499 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:38:27.600 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:38:27.621 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:27.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:27.622 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:38:27.723 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:38:27.735 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is unset 2024-08-19T14:38:27.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:27.747 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:27.747 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:38:27.848 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:38:27.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:27 smithi031 ceph-mon[16336]: pgmap v1646: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.2 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 509 op/s; 16078/108750 objects degraded (14.784%); 4426/108750 objects misplaced (4.070%); 4.3 MiB/s, 112 objects/s recovering 2024-08-19T14:38:27.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:27 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3488283543' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:38:27.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:27 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:38:27.874 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:27.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:27.875 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:38:27.913 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:38:27.913 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [], 2024-08-19T14:38:27.913 INFO:teuthology.orchestra.run.smithi045.stdout: "complaint_time": 30, 2024-08-19T14:38:27.913 INFO:teuthology.orchestra.run.smithi045.stdout: "num_blocked_ops": 0 2024-08-19T14:38:27.913 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:38:27.972 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:38:27.972 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-08-19T14:38:27.976 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:38:27.998 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:27.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:27.999 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:38:28.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:27 smithi032 ceph-mon[20004]: pgmap v1646: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.2 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 509 op/s; 16078/108750 objects degraded (14.784%); 4426/108750 objects misplaced (4.070%); 4.3 MiB/s, 112 objects/s recovering 2024-08-19T14:38:28.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:27 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3488283543' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:38:28.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:27 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:38:28.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:27 smithi045 ceph-mon[20143]: pgmap v1646: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.2 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 509 op/s; 16078/108750 objects degraded (14.784%); 4426/108750 objects misplaced (4.070%); 4.3 MiB/s, 112 objects/s recovering 2024-08-19T14:38:28.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:27 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3488283543' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:38:28.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:27 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:38:28.055 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:38:28.100 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:38:28.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:28.156 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:28.156 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:38:28.257 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:38:28.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:28.311 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:28.311 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:38:28.412 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:38:28.438 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:28.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:28.439 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:38:28.540 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:38:28.562 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:28.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:28.563 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:38:28.570 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-08-19T14:38:28.664 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:38:28.686 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:28.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:28.687 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:38:28.788 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:38:28.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:28 smithi045 ceph-mon[20143]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:38:28.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:28 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:38:28.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:28 smithi045 ceph-mon[20143]: osdmap e691: 12 total, 12 up, 10 in 2024-08-19T14:38:28.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:28 smithi045 ceph-mon[20143]: from='client.17250 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:38:28.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:28 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 16078/108750 objects degraded (14.784%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-19T14:38:28.814 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:28.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:28.815 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:38:28.843 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":1647,"stamp":"2024-08-19T14:38:27.734519+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2327970360,"num_objects":35516,"num_object_clones":0,"num_object_copies":106548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16030,"num_objects_misplaced":4333,"num_objects_unfound":0,"num_objects_dirty":35516,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":200802,"num_write_kb":7562049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43891,"num_bytes_recovered":2802782752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":165039,"ondisk_log_size":165039,"up":141,"acting":136,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":143,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":15498548,"kb_used_data":6994916,"kb_used_omap":18,"kb_used_meta":8503533,"kb_avail":921952972,"statfs":{"total":959950356480,"available":944079843328,"internally_reserved":0,"allocated":7162793984,"data_stored":7132260962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18760,"internal_metadata":8707618488},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,4,11,5,6,10,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-367525888,"num_objects":-5608,"num_object_clones":0,"num_object_copies":-16824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-495,"num_objects_misplaced":-494,"num_objects_unfound":0,"num_objects_dirty":-5608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5608,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1425,"num_bytes_recovered":48496640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.497233"},"pg_stats":[{"pgid":"7.6","version":"690'12781","reported_seq":13074,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-08-19T14:38:25.498482+0000","last_change":"2024-08-19T14:37:56.236536+0000","last_active":"2024-08-19T14:38:25.498482+0000","last_peered":"2024-08-19T14:38:25.498482+0000","last_clean":"2024-08-19T14:38:25.498482+0000","last_became_active":"2024-08-19T14:33:28.334317+0000","last_became_peered":"2024-08-19T14:33:28.334317+0000","last_unstale":"2024-08-19T14:38:25.498482+0000","last_undegraded":"2024-08-19T14:38:25.498482+0000","last_fullsized":"2024-08-19T14:38:25.498482+0000","mapping_epoch":600,"log_start":"611'2700","ondisk_log_start":"611'2700","created":600,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"684'11923","last_scrub_stamp":"2024-08-19T14:37:56.236394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:37:56.236394+0000","objects_scrubbed":3155,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:39:12.585833+0000","scrub_duration":1586,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":148963328,"num_objects":2273,"num_object_clones":0,"num_object_copies":6819,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":481728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"70'139","reported_seq":1699,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-08-19T14:38:15.403574+0000","last_change":"2024-08-19T14:37:57.701402+0000","last_active":"2024-08-19T14:38:15.403574+0000","last_peered":"2024-08-19T14:38:15.403574+0000","last_clean":"2024-08-19T14:38:15.403574+0000","last_became_active":"2024-08-19T14:35:13.950224+0000","last_became_peered":"2024-08-19T14:35:13.950224+0000","last_unstale":"2024-08-19T14:38:15.403574+0000","last_undegraded":"2024-08-19T14:38:15.403574+0000","last_fullsized":"2024-08-19T14:38:15.403574+0000","mapping_epoch":637,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":638,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:37:57.701290+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:19:58.072083+0000","last_clean_scrub_stamp":"2024-08-19T14:37:57.701290+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:39:18.431910+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,9],"acting":[0,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.5","version":"690'12420","reported_seq":21891,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-08-19T14:38:26.658335+0000","last_change":"2024-08-19T14:35:23.022361+0000","last_active":"2024-08-19T14:38:26.658335+0000","last_peered":"2024-08-19T14:38:26.658335+0000","last_clean":"2024-08-19T14:38:26.658335+0000","last_became_active":"2024-08-19T14:35:23.009238+0000","last_became_peered":"2024-08-19T14:35:23.009238+0000","last_unstale":"2024-08-19T14:38:26.658335+0000","last_undegraded":"2024-08-19T14:38:26.658335+0000","last_fullsized":"2024-08-19T14:38:26.658335+0000","mapping_epoch":641,"log_start":"610'2400","ondisk_log_start":"610'2400","created":600,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":148242432,"num_objects":2262,"num_object_clones":0,"num_object_copies":6786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12420,"num_write_kb":469824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4634,"num_bytes_recovered":303693824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,0],"acting":[11,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.3","version":"70'139","reported_seq":1543,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-08-19T14:38:15.421295+0000","last_change":"2024-08-19T14:33:18.804321+0000","last_active":"2024-08-19T14:38:15.421295+0000","last_peered":"2024-08-19T14:38:15.421295+0000","last_clean":"2024-08-19T14:38:15.421295+0000","last_became_active":"2024-08-19T14:18:13.804721+0000","last_became_peered":"2024-08-19T14:18:13.804721+0000","last_unstale":"2024-08-19T14:38:15.421295+0000","last_undegraded":"2024-08-19T14:38:15.421295+0000","last_fullsized":"2024-08-19T14:38:15.421295+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:33:18.804228+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:33:18.804228+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.4","version":"690'12712","reported_seq":13001,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-08-19T14:38:25.494070+0000","last_change":"2024-08-19T14:38:06.717577+0000","last_active":"2024-08-19T14:38:25.494070+0000","last_peered":"2024-08-19T14:38:25.494070+0000","last_clean":"2024-08-19T14:38:25.494070+0000","last_became_active":"2024-08-19T14:33:28.334446+0000","last_became_peered":"2024-08-19T14:33:28.334446+0000","last_unstale":"2024-08-19T14:38:25.494070+0000","last_undegraded":"2024-08-19T14:38:25.494070+0000","last_fullsized":"2024-08-19T14:38:25.494070+0000","mapping_epoch":600,"log_start":"611'2700","ondisk_log_start":"611'2700","created":600,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"686'12159","last_scrub_stamp":"2024-08-19T14:38:06.717004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:38:06.717004+0000","objects_scrubbed":2792,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:39:16.656979+0000","scrub_duration":1017,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146407424,"num_objects":2234,"num_object_clones":0,"num_object_copies":6702,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12712,"num_write_kb":478272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"70'139","reported_seq":1499,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-08-19T14:38:15.433893+0000","last_change":"2024-08-19T14:34:55.580212+0000","last_active":"2024-08-19T14:38:15.433893+0000","last_peered":"2024-08-19T14:38:15.433893+0000","last_clean":"2024-08-19T14:38:15.433893+0000","last_became_active":"2024-08-19T14:20:50.742354+0000","last_became_peered":"2024-08-19T14:20:50.742354+0000","last_unstale":"2024-08-19T14:38:15.433893+0000","last_undegraded":"2024-08-19T14:38:15.433893+0000","last_fullsized":"2024-08-19T14:38:15.433893+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:34:55.580141+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:19:59.302931+0000","last_clean_scrub_stamp":"2024-08-19T14:34:55.580141+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"690'12258","reported_seq":12549,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-08-19T14:38:25.261161+0000","last_change":"2024-08-19T14:37:43.229201+0000","last_active":"2024-08-19T14:38:25.261161+0000","last_peered":"2024-08-19T14:38:25.261161+0000","last_clean":"2024-08-19T14:38:25.261161+0000","last_became_active":"2024-08-19T14:33:28.331833+0000","last_became_peered":"2024-08-19T14:33:28.331833+0000","last_unstale":"2024-08-19T14:38:25.261161+0000","last_undegraded":"2024-08-19T14:38:25.261161+0000","last_fullsized":"2024-08-19T14:38:25.261161+0000","mapping_epoch":600,"log_start":"609'2200","ondisk_log_start":"609'2200","created":600,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"678'11114","last_scrub_stamp":"2024-08-19T14:37:43.229072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:37:43.229072+0000","objects_scrubbed":3413,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:39:12.361681+0000","scrub_duration":910,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":148111360,"num_objects":2260,"num_object_clones":0,"num_object_copies":6780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2260,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12258,"num_write_kb":464576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,1],"acting":[8,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1","version":"70'139","reported_seq":797,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-08-19T14:38:15.403902+0000","last_change":"2024-08-19T14:37:42.630047+0000","last_active":"2024-08-19T14:38:15.403902+0000","last_peered":"2024-08-19T14:38:15.403902+0000","last_clean":"2024-08-19T14:38:15.403902+0000","last_became_active":"2024-08-19T14:35:12.530566+0000","last_became_peered":"2024-08-19T14:35:12.530566+0000","last_unstale":"2024-08-19T14:38:15.403902+0000","last_undegraded":"2024-08-19T14:38:15.403902+0000","last_fullsized":"2024-08-19T14:38:15.403902+0000","mapping_epoch":635,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":636,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:37:42.629919+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:37:42.629919+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:39:05.497590+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,2],"acting":[0,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"690'12387","reported_seq":12656,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-08-19T14:38:23.658419+0000","last_change":"2024-08-19T14:34:52.869444+0000","last_active":"2024-08-19T14:38:23.658419+0000","last_peered":"2024-08-19T14:38:23.658419+0000","last_clean":"2024-08-19T14:38:23.658419+0000","last_became_active":"2024-08-19T14:33:28.332994+0000","last_became_peered":"2024-08-19T14:33:28.332994+0000","last_unstale":"2024-08-19T14:38:23.658419+0000","last_undegraded":"2024-08-19T14:38:23.658419+0000","last_fullsized":"2024-08-19T14:38:23.658419+0000","mapping_epoch":600,"log_start":"609'2300","ondisk_log_start":"609'2300","created":600,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"627'6737","last_scrub_stamp":"2024-08-19T14:34:52.869233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:34:52.869233+0000","objects_scrubbed":6705,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1588,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":143589376,"num_objects":2191,"num_object_clones":0,"num_object_copies":6573,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12387,"num_write_kb":466496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"70'139","reported_seq":1201,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-08-19T14:38:15.427092+0000","last_change":"2024-08-19T14:35:11.537368+0000","last_active":"2024-08-19T14:38:15.427092+0000","last_peered":"2024-08-19T14:38:15.427092+0000","last_clean":"2024-08-19T14:38:15.427092+0000","last_became_active":"2024-08-19T14:35:11.536828+0000","last_became_peered":"2024-08-19T14:35:11.536828+0000","last_unstale":"2024-08-19T14:38:15.427092+0000","last_undegraded":"2024-08-19T14:38:15.427092+0000","last_fullsized":"2024-08-19T14:38:15.427092+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:27.536928+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:21:46.057624+0000","last_clean_scrub_stamp":"2024-08-19T14:28:27.536928+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"690'12289","reported_seq":21714,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-08-19T14:38:24.706985+0000","last_change":"2024-08-19T14:36:15.120872+0000","last_active":"2024-08-19T14:38:24.706985+0000","last_peered":"2024-08-19T14:38:24.706985+0000","last_clean":"2024-08-19T14:38:24.706985+0000","last_became_active":"2024-08-19T14:36:15.096070+0000","last_became_peered":"2024-08-19T14:36:15.096070+0000","last_unstale":"2024-08-19T14:38:24.706985+0000","last_undegraded":"2024-08-19T14:38:24.706985+0000","last_fullsized":"2024-08-19T14:38:24.706985+0000","mapping_epoch":653,"log_start":"609'2200","ondisk_log_start":"609'2200","created":600,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":148832256,"num_objects":2271,"num_object_clones":0,"num_object_copies":6813,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2271,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12289,"num_write_kb":465920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4611,"num_bytes_recovered":302186496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,9],"acting":[3,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"70'139","reported_seq":769,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-08-19T14:38:15.394931+0000","last_change":"2024-08-19T14:37:45.357396+0000","last_active":"2024-08-19T14:38:15.394931+0000","last_peered":"2024-08-19T14:38:15.394931+0000","last_clean":"2024-08-19T14:38:15.394931+0000","last_became_active":"2024-08-19T14:34:15.717054+0000","last_became_peered":"2024-08-19T14:34:15.717054+0000","last_unstale":"2024-08-19T14:38:15.394931+0000","last_undegraded":"2024-08-19T14:38:15.394931+0000","last_fullsized":"2024-08-19T14:38:15.394931+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:37:45.357293+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:27.848870+0000","last_clean_scrub_stamp":"2024-08-19T14:37:45.357293+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:38:48.964771+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"7.3","version":"690'12758","reported_seq":19359,"reported_epoch":690,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-19T14:38:24.810331+0000","last_change":"2024-08-19T14:35:13.136988+0000","last_active":"2024-08-19T14:38:24.810331+0000","last_peered":"2024-08-19T14:38:24.810331+0000","last_clean":"2024-08-19T14:33:27.312789+0000","last_became_active":"2024-08-19T14:35:13.021078+0000","last_became_peered":"2024-08-19T14:35:13.021078+0000","last_unstale":"2024-08-19T14:38:24.810331+0000","last_undegraded":"2024-08-19T14:35:12.592288+0000","last_fullsized":"2024-08-19T14:35:12.526971+0000","mapping_epoch":635,"log_start":"611'2700","ondisk_log_start":"611'2700","created":600,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150470656,"num_objects":2296,"num_object_clones":0,"num_object_copies":6888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2296,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12758,"num_write_kb":481728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3204,"num_bytes_recovered":207421440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":-1758}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"70'139","reported_seq":723,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-08-19T14:38:15.418116+0000","last_change":"2024-08-19T14:35:11.533207+0000","last_active":"2024-08-19T14:38:15.418116+0000","last_peered":"2024-08-19T14:38:15.418116+0000","last_clean":"2024-08-19T14:38:15.418116+0000","last_became_active":"2024-08-19T14:35:11.532858+0000","last_became_peered":"2024-08-19T14:35:11.532858+0000","last_unstale":"2024-08-19T14:38:15.418116+0000","last_undegraded":"2024-08-19T14:38:15.418116+0000","last_fullsized":"2024-08-19T14:38:15.418116+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:27.331041+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:28:27.331041+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,0],"acting":[11,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.2","version":"690'12678","reported_seq":19229,"reported_epoch":690,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-19T14:38:24.814633+0000","last_change":"2024-08-19T14:35:13.121204+0000","last_active":"2024-08-19T14:38:24.814633+0000","last_peered":"2024-08-19T14:38:24.814633+0000","last_clean":"2024-08-19T14:33:27.312789+0000","last_became_active":"2024-08-19T14:35:13.013347+0000","last_became_peered":"2024-08-19T14:35:13.013347+0000","last_unstale":"2024-08-19T14:38:24.814633+0000","last_undegraded":"2024-08-19T14:35:12.598259+0000","last_fullsized":"2024-08-19T14:35:12.526602+0000","mapping_epoch":635,"log_start":"611'2600","ondisk_log_start":"611'2600","created":600,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150863872,"num_objects":2302,"num_object_clones":0,"num_object_copies":6906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12678,"num_write_kb":479360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3179,"num_bytes_recovered":205979648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":-1754}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"70'139","reported_seq":600,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-08-19T14:38:15.418026+0000","last_change":"2024-08-19T14:33:16.760928+0000","last_active":"2024-08-19T14:38:15.418026+0000","last_peered":"2024-08-19T14:38:15.418026+0000","last_clean":"2024-08-19T14:38:15.418026+0000","last_became_active":"2024-08-19T14:30:47.909067+0000","last_became_peered":"2024-08-19T14:30:47.909067+0000","last_unstale":"2024-08-19T14:38:15.418026+0000","last_undegraded":"2024-08-19T14:38:15.418026+0000","last_fullsized":"2024-08-19T14:38:15.418026+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:33:16.760810+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:33:16.760810+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.c","version":"690'12343","reported_seq":18966,"reported_epoch":690,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-19T14:38:24.816710+0000","last_change":"2024-08-19T14:35:13.133729+0000","last_active":"2024-08-19T14:38:24.816710+0000","last_peered":"2024-08-19T14:38:24.816710+0000","last_clean":"2024-08-19T14:33:27.312789+0000","last_became_active":"2024-08-19T14:35:13.003266+0000","last_became_peered":"2024-08-19T14:35:13.003266+0000","last_unstale":"2024-08-19T14:38:24.816710+0000","last_undegraded":"2024-08-19T14:35:12.593502+0000","last_fullsized":"2024-08-19T14:35:12.526375+0000","mapping_epoch":635,"log_start":"609'2300","ondisk_log_start":"609'2300","created":600,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":145031168,"num_objects":2213,"num_object_clones":0,"num_object_copies":6639,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12343,"num_write_kb":465792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3215,"num_bytes_recovered":209256448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,2],"acting":[3,11],"avail_no_missing":["3","11"],"object_location_counts":[{"shards":"3,11","objects":-1581}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"70'139","reported_seq":766,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-08-19T14:38:15.405667+0000","last_change":"2024-08-19T14:34:50.256878+0000","last_active":"2024-08-19T14:38:15.405667+0000","last_peered":"2024-08-19T14:38:15.405667+0000","last_clean":"2024-08-19T14:38:15.405667+0000","last_became_active":"2024-08-19T14:34:15.717472+0000","last_became_peered":"2024-08-19T14:34:15.717472+0000","last_unstale":"2024-08-19T14:38:15.405667+0000","last_undegraded":"2024-08-19T14:38:15.405667+0000","last_fullsized":"2024-08-19T14:38:15.405667+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:34:50.256808+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:28.864870+0000","last_clean_scrub_stamp":"2024-08-19T14:34:50.256808+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,0],"acting":[4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"690'16046","reported_seq":25843,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-08-19T14:38:25.411853+0000","last_change":"2024-08-19T14:37:42.029576+0000","last_active":"2024-08-19T14:38:25.411853+0000","last_peered":"2024-08-19T14:38:25.411853+0000","last_clean":"2024-08-19T14:38:25.411853+0000","last_became_active":"2024-08-19T14:36:14.098166+0000","last_became_peered":"2024-08-19T14:36:14.098166+0000","last_unstale":"2024-08-19T14:38:25.411853+0000","last_undegraded":"2024-08-19T14:38:25.411853+0000","last_fullsized":"2024-08-19T14:38:25.411853+0000","mapping_epoch":652,"log_start":"624'6002","ondisk_log_start":"624'6002","created":600,"last_epoch_clean":653,"parent":"0.0","parent_split_bits":0,"last_scrub":"678'14831","last_scrub_stamp":"2024-08-19T14:37:42.029449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:37:42.029449+0000","objects_scrubbed":3383,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:38:44.946893+0000","scrub_duration":1404,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":140902400,"num_objects":2150,"num_object_clones":0,"num_object_copies":6450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2150,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12416,"num_write_kb":466112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4700,"num_bytes_recovered":308019200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.9","version":"70'139","reported_seq":1701,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-08-19T14:38:15.433933+0000","last_change":"2024-08-19T14:35:12.529993+0000","last_active":"2024-08-19T14:38:15.433933+0000","last_peered":"2024-08-19T14:38:15.433933+0000","last_clean":"2024-08-19T14:38:15.433933+0000","last_became_active":"2024-08-19T14:35:12.529628+0000","last_became_peered":"2024-08-19T14:35:12.529628+0000","last_unstale":"2024-08-19T14:38:15.433933+0000","last_undegraded":"2024-08-19T14:38:15.433933+0000","last_fullsized":"2024-08-19T14:38:15.433933+0000","mapping_epoch":635,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":636,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:34:51.561893+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:34:51.561893+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.e","version":"690'12736","reported_seq":24534,"reported_epoch":690,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-19T14:38:27.713472+0000","last_change":"2024-08-19T14:35:11.863818+0000","last_active":"2024-08-19T14:38:27.713472+0000","last_peered":"2024-08-19T14:38:27.713472+0000","last_clean":"2024-08-19T14:33:27.312789+0000","last_became_active":"2024-08-19T14:35:11.806304+0000","last_became_peered":"2024-08-19T14:35:11.806304+0000","last_unstale":"2024-08-19T14:38:27.713472+0000","last_undegraded":"2024-08-19T14:35:11.779866+0000","last_fullsized":"2024-08-19T14:35:11.528207+0000","mapping_epoch":634,"log_start":"611'2700","ondisk_log_start":"611'2700","created":600,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":135004160,"num_objects":2060,"num_object_clones":0,"num_object_copies":6180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2273,"num_objects_misplaced":4333,"num_objects_unfound":0,"num_objects_dirty":2060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12736,"num_write_kb":473472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4419,"num_bytes_recovered":266665984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":-2273},{"shards":"1,5,9","objects":2060}],"blocked_by":[],"up_primary":2,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"70'139","reported_seq":731,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-08-19T14:38:15.426776+0000","last_change":"2024-08-19T14:35:11.537896+0000","last_active":"2024-08-19T14:38:15.426776+0000","last_peered":"2024-08-19T14:38:15.426776+0000","last_clean":"2024-08-19T14:38:15.426776+0000","last_became_active":"2024-08-19T14:35:11.537474+0000","last_became_peered":"2024-08-19T14:35:11.537474+0000","last_unstale":"2024-08-19T14:38:15.426776+0000","last_undegraded":"2024-08-19T14:38:15.426776+0000","last_fullsized":"2024-08-19T14:38:15.426776+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:59.829567+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:28:59.829567+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"690'12622","reported_seq":12815,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-08-19T14:38:27.608555+0000","last_change":"2024-08-19T14:33:28.335586+0000","last_active":"2024-08-19T14:38:27.608555+0000","last_peered":"2024-08-19T14:38:27.608555+0000","last_clean":"2024-08-19T14:38:27.608555+0000","last_became_active":"2024-08-19T14:33:28.335298+0000","last_became_peered":"2024-08-19T14:33:28.335298+0000","last_unstale":"2024-08-19T14:38:27.608555+0000","last_undegraded":"2024-08-19T14:38:27.608555+0000","last_fullsized":"2024-08-19T14:38:27.608555+0000","mapping_epoch":600,"log_start":"611'2600","ondisk_log_start":"611'2600","created":600,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":137822232,"num_objects":2104,"num_object_clones":0,"num_object_copies":6312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2104,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12622,"num_write_kb":471169,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"70'139","reported_seq":1052,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-08-19T14:38:15.417979+0000","last_change":"2024-08-19T14:33:15.753668+0000","last_active":"2024-08-19T14:38:15.417979+0000","last_peered":"2024-08-19T14:38:15.417979+0000","last_clean":"2024-08-19T14:38:15.417979+0000","last_became_active":"2024-08-19T14:30:47.941200+0000","last_became_peered":"2024-08-19T14:30:47.941200+0000","last_unstale":"2024-08-19T14:38:15.417979+0000","last_undegraded":"2024-08-19T14:38:15.417979+0000","last_fullsized":"2024-08-19T14:38:15.417979+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:33:15.753595+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:33:15.753595+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.8","version":"690'12819","reported_seq":28129,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-08-19T14:38:25.391718+0000","last_change":"2024-08-19T14:38:13.418089+0000","last_active":"2024-08-19T14:38:25.391718+0000","last_peered":"2024-08-19T14:38:25.391718+0000","last_clean":"2024-08-19T14:38:25.391718+0000","last_became_active":"2024-08-19T14:38:13.389656+0000","last_became_peered":"2024-08-19T14:38:13.389656+0000","last_unstale":"2024-08-19T14:38:25.391718+0000","last_undegraded":"2024-08-19T14:38:25.391718+0000","last_fullsized":"2024-08-19T14:38:25.391718+0000","mapping_epoch":688,"log_start":"611'2800","ondisk_log_start":"611'2800","created":600,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150142976,"num_objects":2291,"num_object_clones":0,"num_object_copies":6873,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2291,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12819,"num_write_kb":483520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7555,"num_bytes_recovered":476577792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.e","version":"70'139","reported_seq":1242,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-08-19T14:38:15.405719+0000","last_change":"2024-08-19T14:34:55.333177+0000","last_active":"2024-08-19T14:38:15.405719+0000","last_peered":"2024-08-19T14:38:15.405719+0000","last_clean":"2024-08-19T14:38:15.405719+0000","last_became_active":"2024-08-19T14:18:26.884105+0000","last_became_peered":"2024-08-19T14:18:26.884105+0000","last_unstale":"2024-08-19T14:38:15.405719+0000","last_undegraded":"2024-08-19T14:38:15.405719+0000","last_fullsized":"2024-08-19T14:38:15.405719+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:34:55.333065+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:32:31.196288+0000","last_clean_scrub_stamp":"2024-08-19T14:34:55.333065+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"690'12639","reported_seq":12967,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-08-19T14:38:25.493703+0000","last_change":"2024-08-19T14:37:54.857799+0000","last_active":"2024-08-19T14:38:25.493703+0000","last_peered":"2024-08-19T14:38:25.493703+0000","last_clean":"2024-08-19T14:38:25.493703+0000","last_became_active":"2024-08-19T14:33:28.334547+0000","last_became_peered":"2024-08-19T14:33:28.334547+0000","last_unstale":"2024-08-19T14:38:25.493703+0000","last_undegraded":"2024-08-19T14:38:25.493703+0000","last_fullsized":"2024-08-19T14:38:25.493703+0000","mapping_epoch":600,"log_start":"611'2600","ondisk_log_start":"611'2600","created":600,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"683'11737","last_scrub_stamp":"2024-08-19T14:37:54.851356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:37:54.851356+0000","objects_scrubbed":3161,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:39:09.304335+0000","scrub_duration":1254,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146866176,"num_objects":2241,"num_object_clones":0,"num_object_copies":6723,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12639,"num_write_kb":476160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,8],"acting":[0,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"70'139","reported_seq":994,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-08-19T14:38:15.405751+0000","last_change":"2024-08-19T14:34:12.199530+0000","last_active":"2024-08-19T14:38:15.405751+0000","last_peered":"2024-08-19T14:38:15.405751+0000","last_clean":"2024-08-19T14:38:15.405751+0000","last_became_active":"2024-08-19T14:27:31.837754+0000","last_became_peered":"2024-08-19T14:27:31.837754+0000","last_unstale":"2024-08-19T14:38:15.405751+0000","last_undegraded":"2024-08-19T14:38:15.405751+0000","last_fullsized":"2024-08-19T14:38:15.405751+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:34:12.199449+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:34:12.199449+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.a","version":"690'12503","reported_seq":31296,"reported_epoch":690,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-19T14:38:26.701373+0000","last_change":"2024-08-19T14:35:13.046246+0000","last_active":"2024-08-19T14:38:26.701373+0000","last_peered":"2024-08-19T14:38:26.701373+0000","last_clean":"2024-08-19T14:33:27.312789+0000","last_became_active":"2024-08-19T14:35:13.005199+0000","last_became_peered":"2024-08-19T14:35:13.005199+0000","last_unstale":"2024-08-19T14:38:26.701373+0000","last_undegraded":"2024-08-19T14:35:12.692218+0000","last_fullsized":"2024-08-19T14:35:12.524923+0000","mapping_epoch":635,"log_start":"611'2500","ondisk_log_start":"611'2500","created":600,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":145686528,"num_objects":2223,"num_object_clones":0,"num_object_copies":6669,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1853,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2223,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12503,"num_write_kb":471232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8371,"num_bytes_recovered":522125312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,2],"acting":[11,9],"avail_no_missing":["11","3","9"],"object_location_counts":[{"shards":"9,11","objects":370}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.c","version":"70'139","reported_seq":1349,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-08-19T14:38:15.429495+0000","last_change":"2024-08-19T14:34:15.738488+0000","last_active":"2024-08-19T14:38:15.429495+0000","last_peered":"2024-08-19T14:38:15.429495+0000","last_clean":"2024-08-19T14:38:15.429495+0000","last_became_active":"2024-08-19T14:34:15.733282+0000","last_became_peered":"2024-08-19T14:34:15.733282+0000","last_unstale":"2024-08-19T14:38:15.429495+0000","last_undegraded":"2024-08-19T14:38:15.429495+0000","last_fullsized":"2024-08-19T14:38:15.429495+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:33:17.453290+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:22:52.164586+0000","last_clean_scrub_stamp":"2024-08-19T14:33:17.453290+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,9],"acting":[3,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"690'12441","reported_seq":12633,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-08-19T14:38:27.536928+0000","last_change":"2024-08-19T14:33:28.623717+0000","last_active":"2024-08-19T14:38:27.536928+0000","last_peered":"2024-08-19T14:38:27.536928+0000","last_clean":"2024-08-19T14:38:27.536928+0000","last_became_active":"2024-08-19T14:33:28.623318+0000","last_became_peered":"2024-08-19T14:33:28.623318+0000","last_unstale":"2024-08-19T14:38:27.536928+0000","last_undegraded":"2024-08-19T14:38:27.536928+0000","last_fullsized":"2024-08-19T14:38:27.536928+0000","mapping_epoch":600,"log_start":"610'2400","ondisk_log_start":"610'2400","created":600,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":140443648,"num_objects":2143,"num_object_clones":0,"num_object_copies":6429,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2143,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12441,"num_write_kb":466688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"70'139","reported_seq":1246,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-08-19T14:38:15.434077+0000","last_change":"2024-08-19T14:34:49.476489+0000","last_active":"2024-08-19T14:38:15.434077+0000","last_peered":"2024-08-19T14:38:15.434077+0000","last_clean":"2024-08-19T14:38:15.434077+0000","last_became_active":"2024-08-19T14:18:26.882002+0000","last_became_peered":"2024-08-19T14:18:26.882002+0000","last_unstale":"2024-08-19T14:38:15.434077+0000","last_undegraded":"2024-08-19T14:38:15.434077+0000","last_fullsized":"2024-08-19T14:38:15.434077+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:34:49.476415+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:24:50.276408+0000","last_clean_scrub_stamp":"2024-08-19T14:34:49.476415+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"70'139","reported_seq":1025,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-08-19T14:38:15.417624+0000","last_change":"2024-08-19T14:33:19.781527+0000","last_active":"2024-08-19T14:38:15.417624+0000","last_peered":"2024-08-19T14:38:15.417624+0000","last_clean":"2024-08-19T14:38:15.417624+0000","last_became_active":"2024-08-19T14:27:31.838968+0000","last_became_peered":"2024-08-19T14:27:31.838968+0000","last_unstale":"2024-08-19T14:38:15.417624+0000","last_undegraded":"2024-08-19T14:38:15.417624+0000","last_fullsized":"2024-08-19T14:38:15.417624+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:33:19.781431+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:28:35.206115+0000","last_clean_scrub_stamp":"2024-08-19T14:33:19.781431+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.13","version":"70'139","reported_seq":1567,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-08-19T14:38:15.421200+0000","last_change":"2024-08-19T14:33:35.786236+0000","last_active":"2024-08-19T14:38:15.421200+0000","last_peered":"2024-08-19T14:38:15.421200+0000","last_clean":"2024-08-19T14:38:15.421200+0000","last_became_active":"2024-08-19T14:30:47.939849+0000","last_became_peered":"2024-08-19T14:30:47.939849+0000","last_unstale":"2024-08-19T14:38:15.421200+0000","last_undegraded":"2024-08-19T14:38:15.421200+0000","last_fullsized":"2024-08-19T14:38:15.421200+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:33:35.786105+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:33:35.786105+0000","last_clean_scrub_stamp":"2024-08-19T14:33:35.786105+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.10","version":"70'139","reported_seq":1195,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-08-19T14:38:15.434060+0000","last_change":"2024-08-19T14:34:58.616722+0000","last_active":"2024-08-19T14:38:15.434060+0000","last_peered":"2024-08-19T14:38:15.434060+0000","last_clean":"2024-08-19T14:38:15.434060+0000","last_became_active":"2024-08-19T14:27:31.839611+0000","last_became_peered":"2024-08-19T14:27:31.839611+0000","last_unstale":"2024-08-19T14:38:15.434060+0000","last_undegraded":"2024-08-19T14:38:15.434060+0000","last_fullsized":"2024-08-19T14:38:15.434060+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:34:58.616646+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:34:58.616646+0000","last_clean_scrub_stamp":"2024-08-19T14:34:58.616646+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.11","version":"70'139","reported_seq":1243,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-08-19T14:38:15.403656+0000","last_change":"2024-08-19T14:36:43.798545+0000","last_active":"2024-08-19T14:38:15.403656+0000","last_peered":"2024-08-19T14:38:15.403656+0000","last_clean":"2024-08-19T14:38:15.403656+0000","last_became_active":"2024-08-19T14:34:15.731049+0000","last_became_peered":"2024-08-19T14:34:15.731049+0000","last_unstale":"2024-08-19T14:38:15.403656+0000","last_undegraded":"2024-08-19T14:38:15.403656+0000","last_fullsized":"2024-08-19T14:38:15.403656+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:36:43.798385+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:24:50.547428+0000","last_clean_scrub_stamp":"2024-08-19T14:36:43.798385+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:38:16.708216+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"70'139","reported_seq":1178,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-08-19T14:38:15.456461+0000","last_change":"2024-08-19T14:33:27.737496+0000","last_active":"2024-08-19T14:38:15.456461+0000","last_peered":"2024-08-19T14:38:15.456461+0000","last_clean":"2024-08-19T14:38:15.456461+0000","last_became_active":"2024-08-19T14:20:28.204724+0000","last_became_peered":"2024-08-19T14:20:28.204724+0000","last_unstale":"2024-08-19T14:38:15.456461+0000","last_undegraded":"2024-08-19T14:38:15.456461+0000","last_fullsized":"2024-08-19T14:38:15.456461+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:33:27.737367+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:28:34.216370+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.737367+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"70'139","reported_seq":1321,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-08-19T14:38:15.395103+0000","last_change":"2024-08-19T14:37:38.355694+0000","last_active":"2024-08-19T14:38:15.395103+0000","last_peered":"2024-08-19T14:38:15.395103+0000","last_clean":"2024-08-19T14:38:15.395103+0000","last_became_active":"2024-08-19T14:18:31.901556+0000","last_became_peered":"2024-08-19T14:18:31.901556+0000","last_unstale":"2024-08-19T14:38:15.395103+0000","last_undegraded":"2024-08-19T14:38:15.395103+0000","last_fullsized":"2024-08-19T14:38:15.395103+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:37:38.355621+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:37:38.355621+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:38:39.528989+0000","scrub_duration":56,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,0],"acting":[8,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.14","version":"70'139","reported_seq":1350,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-08-19T14:38:15.433933+0000","last_change":"2024-08-19T14:35:12.530145+0000","last_active":"2024-08-19T14:38:15.433933+0000","last_peered":"2024-08-19T14:38:15.433933+0000","last_clean":"2024-08-19T14:38:15.433933+0000","last_became_active":"2024-08-19T14:35:12.529792+0000","last_became_peered":"2024-08-19T14:35:12.529792+0000","last_unstale":"2024-08-19T14:38:15.433933+0000","last_undegraded":"2024-08-19T14:38:15.433933+0000","last_fullsized":"2024-08-19T14:38:15.433933+0000","mapping_epoch":635,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":636,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:34:53.617013+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:34:53.617013+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,9],"acting":[7,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.15","version":"70'139","reported_seq":1202,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-08-19T14:38:15.434066+0000","last_change":"2024-08-19T14:34:57.640316+0000","last_active":"2024-08-19T14:38:15.434066+0000","last_peered":"2024-08-19T14:38:15.434066+0000","last_clean":"2024-08-19T14:38:15.434066+0000","last_became_active":"2024-08-19T14:18:31.912496+0000","last_became_peered":"2024-08-19T14:18:31.912496+0000","last_unstale":"2024-08-19T14:38:15.434066+0000","last_undegraded":"2024-08-19T14:38:15.434066+0000","last_fullsized":"2024-08-19T14:38:15.434066+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:34:57.640213+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:20:09.180923+0000","last_clean_scrub_stamp":"2024-08-19T14:34:57.640213+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"70'139","reported_seq":1187,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-08-19T14:38:15.405771+0000","last_change":"2024-08-19T14:34:06.176510+0000","last_active":"2024-08-19T14:38:15.405771+0000","last_peered":"2024-08-19T14:38:15.405771+0000","last_clean":"2024-08-19T14:38:15.405771+0000","last_became_active":"2024-08-19T14:18:31.901229+0000","last_became_peered":"2024-08-19T14:18:31.901229+0000","last_unstale":"2024-08-19T14:38:15.405771+0000","last_undegraded":"2024-08-19T14:38:15.405771+0000","last_fullsized":"2024-08-19T14:38:15.405771+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:34:06.176403+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:34:06.176403+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1b","version":"70'139","reported_seq":1568,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-08-19T14:38:15.394865+0000","last_change":"2024-08-19T14:37:59.510774+0000","last_active":"2024-08-19T14:38:15.394865+0000","last_peered":"2024-08-19T14:38:15.394865+0000","last_clean":"2024-08-19T14:38:15.394865+0000","last_became_active":"2024-08-19T14:34:15.718427+0000","last_became_peered":"2024-08-19T14:34:15.718427+0000","last_unstale":"2024-08-19T14:38:15.394865+0000","last_undegraded":"2024-08-19T14:38:15.394865+0000","last_fullsized":"2024-08-19T14:38:15.394865+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:37:59.510697+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:37:59.510697+0000","last_clean_scrub_stamp":"2024-08-19T14:37:59.510697+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:39:04.057695+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.18","version":"70'139","reported_seq":1130,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-08-19T14:38:15.433851+0000","last_change":"2024-08-19T14:35:11.528963+0000","last_active":"2024-08-19T14:38:15.433851+0000","last_peered":"2024-08-19T14:38:15.433851+0000","last_clean":"2024-08-19T14:38:15.433851+0000","last_became_active":"2024-08-19T14:35:11.528530+0000","last_became_peered":"2024-08-19T14:35:11.528530+0000","last_unstale":"2024-08-19T14:38:15.433851+0000","last_undegraded":"2024-08-19T14:38:15.433851+0000","last_fullsized":"2024-08-19T14:38:15.433851+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:29:05.336156+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:27:33.084161+0000","last_clean_scrub_stamp":"2024-08-19T14:29:05.336156+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.19","version":"70'139","reported_seq":1548,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-08-19T14:38:15.403560+0000","last_change":"2024-08-19T14:37:41.671836+0000","last_active":"2024-08-19T14:38:15.403560+0000","last_peered":"2024-08-19T14:38:15.403560+0000","last_clean":"2024-08-19T14:38:15.403560+0000","last_became_active":"2024-08-19T14:18:31.902547+0000","last_became_peered":"2024-08-19T14:18:31.902547+0000","last_unstale":"2024-08-19T14:38:15.403560+0000","last_undegraded":"2024-08-19T14:38:15.403560+0000","last_fullsized":"2024-08-19T14:38:15.403560+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:37:41.671731+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:37:41.671731+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:38:59.596992+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1e","version":"70'139","reported_seq":572,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-08-19T14:38:15.418085+0000","last_change":"2024-08-19T14:33:17.778415+0000","last_active":"2024-08-19T14:38:15.418085+0000","last_peered":"2024-08-19T14:38:15.418085+0000","last_clean":"2024-08-19T14:38:15.418085+0000","last_became_active":"2024-08-19T14:30:47.909974+0000","last_became_peered":"2024-08-19T14:30:47.909974+0000","last_unstale":"2024-08-19T14:38:15.418085+0000","last_undegraded":"2024-08-19T14:38:15.418085+0000","last_fullsized":"2024-08-19T14:38:15.418085+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:33:17.778295+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:27:04.780939+0000","last_clean_scrub_stamp":"2024-08-19T14:33:17.778295+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1c","version":"70'139","reported_seq":1322,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-08-19T14:38:15.394938+0000","last_change":"2024-08-19T14:38:01.463630+0000","last_active":"2024-08-19T14:38:15.394938+0000","last_peered":"2024-08-19T14:38:15.394938+0000","last_clean":"2024-08-19T14:38:15.394938+0000","last_became_active":"2024-08-19T14:30:47.908866+0000","last_became_peered":"2024-08-19T14:30:47.908866+0000","last_unstale":"2024-08-19T14:38:15.394938+0000","last_undegraded":"2024-08-19T14:38:15.394938+0000","last_fullsized":"2024-08-19T14:38:15.394938+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:38:01.463524+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:38:01.463524+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:39:18.496929+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,11],"acting":[8,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1d","version":"70'139","reported_seq":1231,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-08-19T14:38:15.417845+0000","last_change":"2024-08-19T14:35:11.542467+0000","last_active":"2024-08-19T14:38:15.417845+0000","last_peered":"2024-08-19T14:38:15.417845+0000","last_clean":"2024-08-19T14:38:15.417845+0000","last_became_active":"2024-08-19T14:35:11.540488+0000","last_became_peered":"2024-08-19T14:35:11.540488+0000","last_unstale":"2024-08-19T14:38:15.417845+0000","last_undegraded":"2024-08-19T14:38:15.417845+0000","last_fullsized":"2024-08-19T14:38:15.417845+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:29.194639+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:28:29.194639+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":2327379992,"num_objects":35514,"num_object_clones":0,"num_object_copies":106542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16030,"num_objects_misplaced":4333,"num_objects_unfound":0,"num_objects_dirty":35514,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":200802,"num_write_kb":7562049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43888,"num_bytes_recovered":2801926144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":7024881664,"data_stored":7024869448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160730,"ondisk_log_size":160730,"up":48,"acting":43,"num_store_stats":10},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":856608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187781,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1582312,"kb_used_data":720928,"kb_used_omap":1,"kb_used_meta":861374,"kb_avail":92162840,"statfs":{"total":95995035648,"available":94374748160,"internally_reserved":0,"allocated":738230272,"data_stored":735179257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":882047428},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":2314987372648,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":74,"seq":317827580238,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1566632,"kb_used_data":741344,"kb_used_omap":1,"kb_used_meta":825278,"kb_avail":92178520,"statfs":{"total":95995035648,"available":94390804480,"internally_reserved":0,"allocated":759136256,"data_stored":756085241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":845085121},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874581,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1215784,"kb_used_data":440480,"kb_used_omap":1,"kb_used_meta":775294,"kb_avail":92529368,"statfs":{"total":95995035648,"available":94750072832,"internally_reserved":0,"allocated":451051520,"data_stored":448000505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":793901509},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":57,"seq":244813136216,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1390320,"kb_used_data":452584,"kb_used_omap":1,"kb_used_meta":937726,"kb_avail":92354832,"statfs":{"total":95995035648,"available":94571347968,"internally_reserved":0,"allocated":463446016,"data_stored":460391449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":960231882},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":206158430555,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692191,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1366320,"kb_used_data":420264,"kb_used_omap":1,"kb_used_meta":946046,"kb_avail":92378832,"statfs":{"total":95995035648,"available":94595923968,"internally_reserved":0,"allocated":430350336,"data_stored":427295769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":968751557},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986531,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1566952,"kb_used_data":593120,"kb_used_omap":1,"kb_used_meta":973822,"kb_avail":92178200,"statfs":{"total":95995035648,"available":94390476800,"internally_reserved":0,"allocated":607354880,"data_stored":604303865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":997194181},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150058,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2601004,"kb_used_data":1617444,"kb_used_omap":1,"kb_used_meta":983550,"kb_avail":91144148,"statfs":{"total":95995035648,"available":93331607552,"internally_reserved":0,"allocated":1656262656,"data_stored":1653207569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1007155655},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,0,2,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":634,"seq":2723009265704,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":565936,"kb_used_data":304808,"kb_used_omap":4,"kb_used_meta":261115,"kb_avail":93179216,"statfs":{"total":95995035648,"available":95415517184,"internally_reserved":0,"allocated":312123392,"data_stored":309068825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4407,"internal_metadata":267382473},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575219,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1923820,"kb_used_data":831972,"kb_used_omap":1,"kb_used_meta":1091838,"kb_avail":91821332,"statfs":{"total":95995035648,"available":94025043968,"internally_reserved":0,"allocated":851939328,"data_stored":848884241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1118042565},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,3,2,3,1,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738743,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1719468,"kb_used_data":871972,"kb_used_omap":1,"kb_used_meta":847486,"kb_avail":92025684,"statfs":{"total":95995035648,"available":94234300416,"internally_reserved":0,"allocated":892899328,"data_stored":889844241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":867826117},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,0,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":875761664,"data_stored":875757592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":834736128,"data_stored":834732056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":300154880,"data_stored":300154880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1642467328,"data_stored":1642463256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":598802432,"data_stored":598802432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":418840576,"data_stored":418840576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":448724992,"data_stored":448724992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":436404224,"data_stored":436404224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":747765760,"data_stored":747765760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":721223680,"data_stored":721223680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T14:38:28.845 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph status --format=json 2024-08-19T14:38:28.865 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:38:28.916 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:38:28.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:28.987 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:28.988 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:38:29.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:28 smithi032 ceph-mon[20004]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:38:29.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:28 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:38:29.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:28 smithi032 ceph-mon[20004]: osdmap e691: 12 total, 12 up, 10 in 2024-08-19T14:38:29.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:28 smithi032 ceph-mon[20004]: from='client.17250 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:38:29.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:28 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 16078/108750 objects degraded (14.784%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-19T14:38:29.010 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:28 smithi031 ceph-mon[16336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:38:29.010 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:28 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:38:29.010 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:28 smithi031 ceph-mon[16336]: osdmap e691: 12 total, 12 up, 10 in 2024-08-19T14:38:29.010 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:28 smithi031 ceph-mon[16336]: from='client.17250 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:38:29.011 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:28 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 16078/108750 objects degraded (14.784%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-19T14:38:29.088 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:38:29.111 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:29.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:29.112 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:38:29.213 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:38:29.234 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:29.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:29.235 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:38:29.336 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:38:29.358 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:29.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:29.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:38:29.407 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:38:29.407 INFO:teuthology.orchestra.run.smithi045.stdout: "size": 20, 2024-08-19T14:38:29.407 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 600, 2024-08-19T14:38:29.407 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [ 2024-08-19T14:38:29.407 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.407 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.16962.0:101868 7.5 7:a9b55934:::benchmark_data_smithi045_115842_object101867:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e626)", 2024-08-19T14:38:29.407 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:34:44.024260+0000", 2024-08-19T14:38:29.407 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 225.380463158, 2024-08-19T14:38:29.408 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.22134078099999999, 2024-08-19T14:38:29.408 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:38:29.408 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:38:29.409 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:38:29.409 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:38:29.409 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.16962", 2024-08-19T14:38:29.409 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:38:29.409 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 101868 2024-08-19T14:38:29.409 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.409 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:38:29.409 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.410 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:38:29.410 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.024260+0000", 2024-08-19T14:38:29.410 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:38:29.410 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.410 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.410 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:38:29.410 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.024260+0000", 2024-08-19T14:38:29.410 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:38:29.410 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.410 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.410 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:38:29.410 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.024261+0000", 2024-08-19T14:38:29.410 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.75e-06 2024-08-19T14:38:29.411 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.411 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.411 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:38:29.411 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.024286+0000", 2024-08-19T14:38:29.411 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.5000000000000001e-05 2024-08-19T14:38:29.411 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.411 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.411 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:38:29.411 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.024300+0000", 2024-08-19T14:38:29.411 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3848e-05 2024-08-19T14:38:29.411 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.411 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.411 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:38:29.411 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.024308+0000", 2024-08-19T14:38:29.412 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.0800000000000006e-06 2024-08-19T14:38:29.412 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.412 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.412 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:38:29.412 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.028092+0000", 2024-08-19T14:38:29.412 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0037835730000000001 2024-08-19T14:38:29.412 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.412 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.412 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:38:29.412 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.028226+0000", 2024-08-19T14:38:29.412 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013406799999999999 2024-08-19T14:38:29.412 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.412 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.412 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,9", 2024-08-19T14:38:29.413 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.028410+0000", 2024-08-19T14:38:29.413 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00018418199999999999 2024-08-19T14:38:29.413 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.413 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.413 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:38:29.413 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.042589+0000", 2024-08-19T14:38:29.413 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.014178619 2024-08-19T14:38:29.413 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.413 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.413 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:38:29.413 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.042636+0000", 2024-08-19T14:38:29.413 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.7123999999999998e-05 2024-08-19T14:38:29.413 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.413 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.414 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:38:29.414 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.243010+0000", 2024-08-19T14:38:29.414 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.200373724 2024-08-19T14:38:29.414 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.414 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.414 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:38:29.414 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.243072+0000", 2024-08-19T14:38:29.414 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.2238999999999998e-05 2024-08-19T14:38:29.414 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.414 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.414 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:38:29.414 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.245600+0000", 2024-08-19T14:38:29.414 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.002528574 2024-08-19T14:38:29.415 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:38:29.415 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:38:29.415 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:38:29.415 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.415 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.415 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.16962.0:101874 7.5 7:aba2a64c:::benchmark_data_smithi045_115842_object101873:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e626)", 2024-08-19T14:38:29.415 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:34:44.030050+0000", 2024-08-19T14:38:29.415 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 225.37467326500001, 2024-08-19T14:38:29.415 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.23052292099999999, 2024-08-19T14:38:29.415 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:38:29.415 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:38:29.415 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:38:29.415 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:38:29.416 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.16962", 2024-08-19T14:38:29.416 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:38:29.416 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 101874 2024-08-19T14:38:29.416 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.416 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:38:29.416 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.416 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:38:29.416 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.030050+0000", 2024-08-19T14:38:29.416 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:38:29.416 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.416 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.416 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:38:29.416 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.030050+0000", 2024-08-19T14:38:29.416 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:38:29.416 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.417 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.417 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:38:29.417 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.030051+0000", 2024-08-19T14:38:29.417 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9099999999999999e-06 2024-08-19T14:38:29.417 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.417 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.417 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:38:29.417 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.030092+0000", 2024-08-19T14:38:29.417 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.1050999999999997e-05 2024-08-19T14:38:29.417 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.417 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.417 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:38:29.417 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.030113+0000", 2024-08-19T14:38:29.417 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0358999999999999e-05 2024-08-19T14:38:29.418 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.418 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.418 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:38:29.418 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.030123+0000", 2024-08-19T14:38:29.418 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0172e-05 2024-08-19T14:38:29.418 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.418 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.418 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:38:29.418 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.038789+0000", 2024-08-19T14:38:29.418 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0086658689999999997 2024-08-19T14:38:29.419 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.419 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.419 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:38:29.419 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.038947+0000", 2024-08-19T14:38:29.419 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015787700000000001 2024-08-19T14:38:29.419 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.419 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.419 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,9", 2024-08-19T14:38:29.419 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.039142+0000", 2024-08-19T14:38:29.419 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00019552400000000001 2024-08-19T14:38:29.419 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.419 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.419 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:38:29.420 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.044827+0000", 2024-08-19T14:38:29.420 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0056849190000000001 2024-08-19T14:38:29.420 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.420 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.420 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:38:29.420 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.054485+0000", 2024-08-19T14:38:29.420 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0096582260000000007 2024-08-19T14:38:29.420 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.420 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.420 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:38:29.420 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.257671+0000", 2024-08-19T14:38:29.420 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20318518099999999 2024-08-19T14:38:29.420 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.421 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.421 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:38:29.421 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.257731+0000", 2024-08-19T14:38:29.421 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.0825000000000001e-05 2024-08-19T14:38:29.421 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.421 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.421 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:38:29.421 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.260572+0000", 2024-08-19T14:38:29.421 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0028410079999999999 2024-08-19T14:38:29.421 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:38:29.421 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:38:29.421 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:38:29.421 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.421 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.421 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.16962.0:101888 7.5 7:a3e28936:::benchmark_data_smithi045_115842_object101887:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e626)", 2024-08-19T14:38:29.422 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:34:44.046545+0000", 2024-08-19T14:38:29.422 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 225.358177568, 2024-08-19T14:38:29.422 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.221824155, 2024-08-19T14:38:29.422 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:38:29.422 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:38:29.422 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:38:29.423 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:38:29.423 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.16962", 2024-08-19T14:38:29.423 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:38:29.423 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 101888 2024-08-19T14:38:29.423 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.423 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:38:29.423 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.423 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:38:29.424 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.046545+0000", 2024-08-19T14:38:29.424 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:38:29.424 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.424 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.424 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:38:29.424 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.046545+0000", 2024-08-19T14:38:29.424 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:38:29.424 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.424 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.424 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:38:29.424 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.046547+0000", 2024-08-19T14:38:29.424 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8780000000000001e-06 2024-08-19T14:38:29.424 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.424 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.425 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:38:29.425 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.046570+0000", 2024-08-19T14:38:29.425 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2459999999999998e-05 2024-08-19T14:38:29.425 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.425 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.425 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:38:29.425 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.046583+0000", 2024-08-19T14:38:29.425 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3886000000000001e-05 2024-08-19T14:38:29.425 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.425 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.425 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:38:29.425 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.046591+0000", 2024-08-19T14:38:29.425 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.6450000000000005e-06 2024-08-19T14:38:29.426 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.426 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.426 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:38:29.426 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.046648+0000", 2024-08-19T14:38:29.426 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.6694999999999998e-05 2024-08-19T14:38:29.426 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.426 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.426 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:38:29.426 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.046816+0000", 2024-08-19T14:38:29.426 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00016854899999999999 2024-08-19T14:38:29.426 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.426 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.426 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,9", 2024-08-19T14:38:29.427 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.046999+0000", 2024-08-19T14:38:29.427 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00018276100000000001 2024-08-19T14:38:29.427 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.427 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.427 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:38:29.427 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.054535+0000", 2024-08-19T14:38:29.427 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0075362509999999999 2024-08-19T14:38:29.427 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.427 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.427 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:38:29.427 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.054545+0000", 2024-08-19T14:38:29.427 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0084999999999999e-05 2024-08-19T14:38:29.427 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.427 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.428 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:38:29.428 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.266112+0000", 2024-08-19T14:38:29.428 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.211566105 2024-08-19T14:38:29.428 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.428 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.428 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:38:29.428 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.266179+0000", 2024-08-19T14:38:29.428 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.7485999999999997e-05 2024-08-19T14:38:29.428 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.428 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.428 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:38:29.428 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.268369+0000", 2024-08-19T14:38:29.428 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0021903539999999998 2024-08-19T14:38:29.428 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:38:29.429 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:38:29.429 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:38:29.429 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.429 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.429 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.16962.0:101894 7.5 7:a67a0241:::benchmark_data_smithi045_115842_object101893:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e626)", 2024-08-19T14:38:29.429 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:34:44.050292+0000", 2024-08-19T14:38:29.429 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 225.35443084100001, 2024-08-19T14:38:29.429 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.22549364399999999, 2024-08-19T14:38:29.429 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:38:29.429 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:38:29.429 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:38:29.429 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:38:29.429 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.16962", 2024-08-19T14:38:29.429 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:38:29.430 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 101894 2024-08-19T14:38:29.430 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.430 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:38:29.430 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.430 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:38:29.430 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.050292+0000", 2024-08-19T14:38:29.430 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:38:29.430 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.430 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.430 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:38:29.430 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.050292+0000", 2024-08-19T14:38:29.430 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:38:29.430 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.430 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.431 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:38:29.431 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.050293+0000", 2024-08-19T14:38:29.431 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.226e-06 2024-08-19T14:38:29.431 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.431 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.431 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:38:29.431 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.050314+0000", 2024-08-19T14:38:29.431 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1043000000000002e-05 2024-08-19T14:38:29.431 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.431 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.431 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:38:29.431 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.050326+0000", 2024-08-19T14:38:29.431 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1999e-05 2024-08-19T14:38:29.431 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.432 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.432 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:38:29.432 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.050332+0000", 2024-08-19T14:38:29.432 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.818e-06 2024-08-19T14:38:29.432 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.432 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.432 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:38:29.432 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.054572+0000", 2024-08-19T14:38:29.432 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0042403340000000001 2024-08-19T14:38:29.432 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.432 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.432 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:38:29.432 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.054692+0000", 2024-08-19T14:38:29.432 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000119597 2024-08-19T14:38:29.433 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.433 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.433 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,9", 2024-08-19T14:38:29.433 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.054807+0000", 2024-08-19T14:38:29.433 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00011525499999999999 2024-08-19T14:38:29.433 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.433 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.433 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:38:29.433 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.060833+0000", 2024-08-19T14:38:29.433 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0060257929999999998 2024-08-19T14:38:29.433 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.433 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.433 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:38:29.433 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.067717+0000", 2024-08-19T14:38:29.434 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0068838500000000004 2024-08-19T14:38:29.434 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.434 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.434 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:38:29.434 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.273501+0000", 2024-08-19T14:38:29.434 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.205783779 2024-08-19T14:38:29.434 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.434 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.434 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:38:29.434 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.273561+0000", 2024-08-19T14:38:29.434 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.0613999999999999e-05 2024-08-19T14:38:29.434 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.434 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.434 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:38:29.435 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.275786+0000", 2024-08-19T14:38:29.435 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0022243359999999999 2024-08-19T14:38:29.435 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:38:29.435 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:38:29.435 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:38:29.435 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.435 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.435 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.16962.0:101896 7.5 7:a83fdd11:::benchmark_data_smithi045_115842_object101895:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e626)", 2024-08-19T14:38:29.435 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:34:44.050859+0000", 2024-08-19T14:38:29.435 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 225.35386340700001, 2024-08-19T14:38:29.435 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.22712142900000001, 2024-08-19T14:38:29.435 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:38:29.435 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:38:29.435 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:38:29.435 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:38:29.436 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.16962", 2024-08-19T14:38:29.436 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:38:29.436 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 101896 2024-08-19T14:38:29.436 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.436 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:38:29.436 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.436 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:38:29.436 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.050859+0000", 2024-08-19T14:38:29.436 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:38:29.436 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.436 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.436 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:38:29.436 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.050859+0000", 2024-08-19T14:38:29.436 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:38:29.437 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.437 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.437 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:38:29.437 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.050860+0000", 2024-08-19T14:38:29.437 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.5900000000000005e-07 2024-08-19T14:38:29.437 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.437 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.437 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:38:29.437 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.050895+0000", 2024-08-19T14:38:29.437 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.4251000000000001e-05 2024-08-19T14:38:29.437 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.437 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.437 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:38:29.437 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.050904+0000", 2024-08-19T14:38:29.438 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.8779999999999993e-06 2024-08-19T14:38:29.438 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.438 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.438 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:38:29.438 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.050910+0000", 2024-08-19T14:38:29.438 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7370000000000003e-06 2024-08-19T14:38:29.438 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.438 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.438 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:38:29.438 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.057488+0000", 2024-08-19T14:38:29.438 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0065775440000000003 2024-08-19T14:38:29.438 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.438 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.438 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:38:29.439 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.057650+0000", 2024-08-19T14:38:29.439 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00016202 2024-08-19T14:38:29.439 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.439 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.439 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,9", 2024-08-19T14:38:29.439 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.057823+0000", 2024-08-19T14:38:29.439 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000173501 2024-08-19T14:38:29.439 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.439 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.439 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:38:29.439 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.067740+0000", 2024-08-19T14:38:29.439 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0099169500000000008 2024-08-19T14:38:29.439 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.439 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.440 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:38:29.440 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.067799+0000", 2024-08-19T14:38:29.440 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.9108999999999999e-05 2024-08-19T14:38:29.440 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.440 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.440 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:38:29.440 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.275858+0000", 2024-08-19T14:38:29.440 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20805822500000001 2024-08-19T14:38:29.440 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.440 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.440 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:38:29.440 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.275910+0000", 2024-08-19T14:38:29.440 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2253999999999998e-05 2024-08-19T14:38:29.440 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.441 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.441 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:38:29.441 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.277981+0000", 2024-08-19T14:38:29.441 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0020710009999999998 2024-08-19T14:38:29.441 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:38:29.441 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:38:29.441 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:38:29.441 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.441 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.441 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.16962.0:101904 7.5 7:ad2dd20f:::benchmark_data_smithi045_115842_object101903:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e626)", 2024-08-19T14:38:29.441 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:34:44.062127+0000", 2024-08-19T14:38:29.441 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 225.342596181, 2024-08-19T14:38:29.441 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.21785035, 2024-08-19T14:38:29.441 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:38:29.442 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:38:29.442 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:38:29.442 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:38:29.442 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.16962", 2024-08-19T14:38:29.442 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:38:29.442 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 101904 2024-08-19T14:38:29.442 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.442 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:38:29.442 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.442 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:38:29.442 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.062127+0000", 2024-08-19T14:38:29.442 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:38:29.442 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.442 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.443 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:38:29.443 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.062127+0000", 2024-08-19T14:38:29.443 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:38:29.443 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.443 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.443 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:38:29.443 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.062128+0000", 2024-08-19T14:38:29.443 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8780000000000001e-06 2024-08-19T14:38:29.443 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.444 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.444 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:38:29.444 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.062156+0000", 2024-08-19T14:38:29.444 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.7317e-05 2024-08-19T14:38:29.444 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.444 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.444 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:38:29.444 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.062170+0000", 2024-08-19T14:38:29.444 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4059e-05 2024-08-19T14:38:29.444 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.444 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.444 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:38:29.444 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.062177+0000", 2024-08-19T14:38:29.445 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.0360000000000001e-06 2024-08-19T14:38:29.445 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.445 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.445 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:38:29.445 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.067875+0000", 2024-08-19T14:38:29.445 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0056985459999999996 2024-08-19T14:38:29.445 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.445 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.445 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:38:29.445 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.068063+0000", 2024-08-19T14:38:29.445 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00018802399999999999 2024-08-19T14:38:29.445 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.445 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.445 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,9", 2024-08-19T14:38:29.446 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.068227+0000", 2024-08-19T14:38:29.446 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00016347900000000001 2024-08-19T14:38:29.446 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.446 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.446 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:38:29.446 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.075326+0000", 2024-08-19T14:38:29.446 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0070986160000000003 2024-08-19T14:38:29.446 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.446 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.446 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:38:29.446 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.092809+0000", 2024-08-19T14:38:29.446 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.017483207000000001 2024-08-19T14:38:29.446 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.446 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.447 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:38:29.447 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.278038+0000", 2024-08-19T14:38:29.447 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.185228804 2024-08-19T14:38:29.447 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.447 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.447 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:38:29.447 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.278077+0000", 2024-08-19T14:38:29.447 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.8949000000000003e-05 2024-08-19T14:38:29.447 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.447 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.447 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:38:29.447 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.279977+0000", 2024-08-19T14:38:29.447 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001900435 2024-08-19T14:38:29.447 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:38:29.447 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:38:29.448 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:38:29.448 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.448 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.448 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.16962.0:101907 7.5 7:a0598159:::benchmark_data_smithi045_115842_object101906:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e626)", 2024-08-19T14:38:29.448 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:34:44.064648+0000", 2024-08-19T14:38:29.448 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 225.340075184, 2024-08-19T14:38:29.448 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.217895121, 2024-08-19T14:38:29.448 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:38:29.448 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:38:29.448 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:38:29.448 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:38:29.448 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.16962", 2024-08-19T14:38:29.448 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:38:29.448 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 101907 2024-08-19T14:38:29.449 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.449 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:38:29.449 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.449 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:38:29.449 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.064648+0000", 2024-08-19T14:38:29.449 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:38:29.449 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.449 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.449 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:38:29.449 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.064648+0000", 2024-08-19T14:38:29.449 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:38:29.449 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.449 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.449 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:38:29.449 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.064649+0000", 2024-08-19T14:38:29.450 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5400000000000001e-06 2024-08-19T14:38:29.450 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.450 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.450 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:38:29.450 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.064676+0000", 2024-08-19T14:38:29.450 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.6812999999999999e-05 2024-08-19T14:38:29.450 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.450 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.450 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:38:29.450 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.064691+0000", 2024-08-19T14:38:29.450 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4763000000000001e-05 2024-08-19T14:38:29.450 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.450 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.451 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:38:29.451 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.064701+0000", 2024-08-19T14:38:29.451 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.9930000000000007e-06 2024-08-19T14:38:29.451 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.451 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.451 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:38:29.451 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.071510+0000", 2024-08-19T14:38:29.451 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0068096909999999997 2024-08-19T14:38:29.451 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.451 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.451 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:38:29.451 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.071680+0000", 2024-08-19T14:38:29.451 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00016987400000000001 2024-08-19T14:38:29.451 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.452 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.452 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,9", 2024-08-19T14:38:29.452 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.071848+0000", 2024-08-19T14:38:29.452 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00016773299999999999 2024-08-19T14:38:29.452 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.452 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.452 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:38:29.452 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.075335+0000", 2024-08-19T14:38:29.452 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.003486617 2024-08-19T14:38:29.452 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.452 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.452 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:38:29.452 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.092867+0000", 2024-08-19T14:38:29.452 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.017532749 2024-08-19T14:38:29.453 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.453 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.453 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:38:29.453 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.280052+0000", 2024-08-19T14:38:29.453 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.18718484899999999 2024-08-19T14:38:29.453 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.453 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.453 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:38:29.453 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.280118+0000", 2024-08-19T14:38:29.453 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.6191000000000004e-05 2024-08-19T14:38:29.453 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.453 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.453 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:38:29.453 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.282543+0000", 2024-08-19T14:38:29.453 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.002424308 2024-08-19T14:38:29.454 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:38:29.454 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:38:29.454 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:38:29.454 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.454 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.454 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.16962.0:101915 7.5 7:a194bcf3:::benchmark_data_smithi045_115842_object101914:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e626)", 2024-08-19T14:38:29.454 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:34:44.075596+0000", 2024-08-19T14:38:29.454 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 225.329126328, 2024-08-19T14:38:29.454 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.210082677, 2024-08-19T14:38:29.454 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:38:29.454 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:38:29.454 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:38:29.454 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:38:29.454 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.16962", 2024-08-19T14:38:29.455 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:38:29.455 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 101915 2024-08-19T14:38:29.455 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.455 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:38:29.455 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.455 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:38:29.455 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.075596+0000", 2024-08-19T14:38:29.455 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:38:29.455 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.455 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.455 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:38:29.455 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.075596+0000", 2024-08-19T14:38:29.455 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:38:29.455 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.456 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.456 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:38:29.456 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.075598+0000", 2024-08-19T14:38:29.456 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3400000000000001e-06 2024-08-19T14:38:29.456 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.456 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.456 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:38:29.456 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.075620+0000", 2024-08-19T14:38:29.456 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2259000000000002e-05 2024-08-19T14:38:29.456 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.456 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.456 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:38:29.456 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.075634+0000", 2024-08-19T14:38:29.456 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3961000000000001e-05 2024-08-19T14:38:29.457 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.457 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.457 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:38:29.457 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.075642+0000", 2024-08-19T14:38:29.457 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.2830000000000001e-06 2024-08-19T14:38:29.457 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.457 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.457 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:38:29.457 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.075703+0000", 2024-08-19T14:38:29.457 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.0565e-05 2024-08-19T14:38:29.457 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.457 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.457 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:38:29.457 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.075888+0000", 2024-08-19T14:38:29.457 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000185138 2024-08-19T14:38:29.458 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.458 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.458 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,9", 2024-08-19T14:38:29.458 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.076067+0000", 2024-08-19T14:38:29.458 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00017943000000000001 2024-08-19T14:38:29.458 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.458 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.458 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:38:29.458 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.079463+0000", 2024-08-19T14:38:29.458 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0033953920000000001 2024-08-19T14:38:29.458 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.458 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.458 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:38:29.459 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.092925+0000", 2024-08-19T14:38:29.459 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.013462399999999999 2024-08-19T14:38:29.459 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.459 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.459 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:38:29.459 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.282636+0000", 2024-08-19T14:38:29.459 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.18971106800000001 2024-08-19T14:38:29.459 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.459 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.459 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:38:29.459 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.282704+0000", 2024-08-19T14:38:29.459 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.7532000000000003e-05 2024-08-19T14:38:29.459 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.459 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.459 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:38:29.460 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:34:44.285679+0000", 2024-08-19T14:38:29.460 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0029753090000000002 2024-08-19T14:38:29.460 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:38:29.460 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:38:29.460 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:38:29.460 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.460 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.460 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.16962.0:122539 7.a 7:5342145d:::benchmark_data_smithi045_115842_object4379:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e636)", 2024-08-19T14:38:29.460 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:35:12.692440+0000", 2024-08-19T14:38:29.460 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 196.71228319400001, 2024-08-19T14:38:29.460 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.35142216700000001, 2024-08-19T14:38:29.460 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:38:29.460 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:38:29.460 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-08-19T14:38:29.461 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:38:29.461 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.16962", 2024-08-19T14:38:29.461 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:38:29.461 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 122539 2024-08-19T14:38:29.461 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.461 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:38:29.461 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.461 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:38:29.461 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:35:12.692440+0000", 2024-08-19T14:38:29.461 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:38:29.461 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.461 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.461 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:38:29.461 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:35:12.692440+0000", 2024-08-19T14:38:29.462 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:38:29.462 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.462 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.462 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:38:29.462 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:35:12.692441+0000", 2024-08-19T14:38:29.462 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.782e-06 2024-08-19T14:38:29.462 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.462 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.462 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:38:29.462 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:35:12.692447+0000", 2024-08-19T14:38:29.462 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.1229999999999997e-06 2024-08-19T14:38:29.462 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.462 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.462 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:38:29.463 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:35:12.692460+0000", 2024-08-19T14:38:29.463 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2275e-05 2024-08-19T14:38:29.463 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.463 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.463 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:38:29.463 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:35:12.692467+0000", 2024-08-19T14:38:29.463 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.3409999999999997e-06 2024-08-19T14:38:29.463 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.463 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.463 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:38:29.463 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:35:12.696645+0000", 2024-08-19T14:38:29.463 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0041775270000000003 2024-08-19T14:38:29.463 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.463 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.464 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for peered", 2024-08-19T14:38:29.464 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:35:12.696656+0000", 2024-08-19T14:38:29.464 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0908e-05 2024-08-19T14:38:29.464 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.464 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.464 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:38:29.464 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:35:13.020687+0000", 2024-08-19T14:38:29.464 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.32403147700000001 2024-08-19T14:38:29.464 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.464 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.464 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for missing object", 2024-08-19T14:38:29.464 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:35:13.020829+0000", 2024-08-19T14:38:29.464 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014225500000000001 2024-08-19T14:38:29.465 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.465 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.465 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:38:29.465 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:35:13.043783+0000", 2024-08-19T14:38:29.465 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.022953978999999999 2024-08-19T14:38:29.465 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.465 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.465 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:38:29.465 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:35:13.043862+0000", 2024-08-19T14:38:29.465 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.8499999999999997e-05 2024-08-19T14:38:29.465 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:38:29.465 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:38:29.465 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:38:29.465 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.466 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.466 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.16962.0:135912 7.a 7:5fb95132:::benchmark_data_smithi045_115842_object17752:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e647)", 2024-08-19T14:38:29.466 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:35:48.309187+0000", 2024-08-19T14:38:29.466 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 161.09553532499999, 2024-08-19T14:38:29.466 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6817382410000001, 2024-08-19T14:38:29.466 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:38:29.466 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:38:29.466 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-08-19T14:38:29.466 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:38:29.466 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.16962", 2024-08-19T14:38:29.466 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:38:29.466 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 135912 2024-08-19T14:38:29.466 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.467 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:38:29.467 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.467 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:38:29.467 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:35:48.309187+0000", 2024-08-19T14:38:29.467 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:38:29.467 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.467 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.467 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:38:29.467 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:35:48.309187+0000", 2024-08-19T14:38:29.467 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:38:29.467 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.467 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.467 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:38:29.467 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:35:48.309188+0000", 2024-08-19T14:38:29.468 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.8999999999999997e-07 2024-08-19T14:38:29.468 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.468 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.468 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:38:29.468 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:35:48.309191+0000", 2024-08-19T14:38:29.468 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.123e-06 2024-08-19T14:38:29.468 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.468 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.468 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:38:29.468 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:35:48.309197+0000", 2024-08-19T14:38:29.468 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.2779999999999999e-06 2024-08-19T14:38:29.468 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.468 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.468 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:38:29.469 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:35:48.309203+0000", 2024-08-19T14:38:29.469 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.6119999999999998e-06 2024-08-19T14:38:29.469 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.469 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.469 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:38:29.469 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:35:48.319256+0000", 2024-08-19T14:38:29.469 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.010052922000000001 2024-08-19T14:38:29.469 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.469 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.469 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for missing object", 2024-08-19T14:38:29.469 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:35:48.319319+0000", 2024-08-19T14:38:29.469 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.3331999999999996e-05 2024-08-19T14:38:29.469 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.469 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.470 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:38:29.470 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:35:49.990910+0000", 2024-08-19T14:38:29.470 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6715909369999999 2024-08-19T14:38:29.470 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.470 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.470 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:38:29.470 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:35:49.990926+0000", 2024-08-19T14:38:29.470 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5546999999999999e-05 2024-08-19T14:38:29.470 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:38:29.470 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:38:29.470 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:38:29.470 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.470 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.471 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.16962.0:153323 7.a 7:5d791733:::benchmark_data_smithi045_115842_object35163:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e657)", 2024-08-19T14:38:29.471 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:36:31.864504+0000", 2024-08-19T14:38:29.471 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 117.540218727, 2024-08-19T14:38:29.471 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.21933270299999999, 2024-08-19T14:38:29.471 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:38:29.471 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:38:29.471 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:38:29.471 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:38:29.471 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.16962", 2024-08-19T14:38:29.471 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:38:29.471 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 153323 2024-08-19T14:38:29.471 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.471 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:38:29.472 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.472 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:38:29.472 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:36:31.864504+0000", 2024-08-19T14:38:29.472 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:38:29.472 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.472 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.472 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:38:29.472 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:36:31.864504+0000", 2024-08-19T14:38:29.472 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:38:29.472 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.472 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.472 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:38:29.472 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:36:31.864505+0000", 2024-08-19T14:38:29.472 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2559999999999999e-06 2024-08-19T14:38:29.473 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.473 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.473 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:38:29.473 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:36:31.864510+0000", 2024-08-19T14:38:29.473 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.7160000000000002e-06 2024-08-19T14:38:29.473 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.473 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.473 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:38:29.473 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:36:31.864522+0000", 2024-08-19T14:38:29.473 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1926000000000001e-05 2024-08-19T14:38:29.473 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.473 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.473 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:38:29.474 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:36:31.864528+0000", 2024-08-19T14:38:29.474 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7019999999999998e-06 2024-08-19T14:38:29.474 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.474 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.474 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:38:29.474 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:36:31.864584+0000", 2024-08-19T14:38:29.474 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.6187999999999998e-05 2024-08-19T14:38:29.474 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.474 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.474 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:38:29.474 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:36:31.864708+0000", 2024-08-19T14:38:29.474 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012413699999999999 2024-08-19T14:38:29.474 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.474 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.475 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2,9", 2024-08-19T14:38:29.475 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:36:31.864811+0000", 2024-08-19T14:38:29.475 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00010294299999999999 2024-08-19T14:38:29.475 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.475 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.475 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:38:29.475 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:36:31.868416+0000", 2024-08-19T14:38:29.475 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0036052359999999999 2024-08-19T14:38:29.475 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.475 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.475 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:38:29.475 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:36:31.894736+0000", 2024-08-19T14:38:29.475 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.026319683999999999 2024-08-19T14:38:29.475 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.476 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.476 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:38:29.476 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:36:32.080712+0000", 2024-08-19T14:38:29.476 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.18597578100000001 2024-08-19T14:38:29.476 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.476 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.476 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:38:29.476 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:36:32.080779+0000", 2024-08-19T14:38:29.476 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.7279000000000004e-05 2024-08-19T14:38:29.476 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.476 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.476 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:38:29.476 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:36:32.083837+0000", 2024-08-19T14:38:29.476 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0030578549999999999 2024-08-19T14:38:29.477 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:38:29.477 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:38:29.477 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:38:29.477 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.477 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.477 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.16962.0:179498 7.5 7:afb271fb:::benchmark_data_smithi045_115842_object61338:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e678)", 2024-08-19T14:38:29.477 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:37:38.086645+0000", 2024-08-19T14:38:29.477 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 51.318078208999999, 2024-08-19T14:38:29.477 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.26942593799999998, 2024-08-19T14:38:29.477 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:38:29.477 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:38:29.477 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:38:29.477 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:38:29.477 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.16962", 2024-08-19T14:38:29.477 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:38:29.478 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:38:29.479 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:38:29.479 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:38:29.480 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 179498 2024-08-19T14:38:29.480 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.480 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:38:29.480 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.480 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:38:29.480 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:37:38.086645+0000", 2024-08-19T14:38:29.480 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:38:29.480 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.480 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.480 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:38:29.481 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:37:38.086645+0000", 2024-08-19T14:38:29.481 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:38:29.481 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.481 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.481 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:38:29.481 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:37:38.086646+0000", 2024-08-19T14:38:29.481 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.47e-07 2024-08-19T14:38:29.481 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.481 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.481 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:38:29.481 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:37:38.086650+0000", 2024-08-19T14:38:29.481 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.1570000000000002e-06 2024-08-19T14:38:29.481 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.481 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.482 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:38:29.482 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:37:38.086659+0000", 2024-08-19T14:38:29.482 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.3049999999999998e-06 2024-08-19T14:38:29.482 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.482 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.482 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:38:29.482 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:37:38.086665+0000", 2024-08-19T14:38:29.482 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.5890000000000002e-06 2024-08-19T14:38:29.482 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.482 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.482 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:38:29.482 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:37:38.086721+0000", 2024-08-19T14:38:29.482 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.6484999999999999e-05 2024-08-19T14:38:29.482 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.483 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.483 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:38:29.483 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:37:38.086809+0000", 2024-08-19T14:38:29.483 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.7674000000000001e-05 2024-08-19T14:38:29.483 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.483 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.483 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,9", 2024-08-19T14:38:29.483 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:37:38.086893+0000", 2024-08-19T14:38:29.483 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.3869000000000006e-05 2024-08-19T14:38:29.483 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.483 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.483 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:38:29.483 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:37:38.090749+0000", 2024-08-19T14:38:29.483 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0038561440000000002 2024-08-19T14:38:29.484 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.484 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.484 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:38:29.484 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:37:38.090849+0000", 2024-08-19T14:38:29.484 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000100467 2024-08-19T14:38:29.484 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.484 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.484 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:38:29.484 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:37:38.355958+0000", 2024-08-19T14:38:29.484 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.26510836700000001 2024-08-19T14:38:29.484 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.484 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.484 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:38:29.484 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:37:38.356014+0000", 2024-08-19T14:38:29.484 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.6635999999999998e-05 2024-08-19T14:38:29.485 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.485 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.485 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:38:29.485 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:37:38.356071+0000", 2024-08-19T14:38:29.485 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.6298000000000002e-05 2024-08-19T14:38:29.485 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:38:29.485 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:38:29.485 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:38:29.485 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.485 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.485 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.16962.0:179502 7.5 7:ad25005f:::benchmark_data_smithi045_115842_object61342:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e678)", 2024-08-19T14:38:29.485 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:37:38.109513+0000", 2024-08-19T14:38:29.485 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 51.295209753999998, 2024-08-19T14:38:29.486 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.246700533, 2024-08-19T14:38:29.486 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:38:29.486 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:38:29.486 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:38:29.486 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:38:29.486 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.16962", 2024-08-19T14:38:29.486 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:38:29.486 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 179502 2024-08-19T14:38:29.486 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.486 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:38:29.486 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.486 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:38:29.486 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:37:38.109513+0000", 2024-08-19T14:38:29.487 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:38:29.487 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.487 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.487 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:38:29.487 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:37:38.109513+0000", 2024-08-19T14:38:29.487 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:38:29.487 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.487 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.487 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:38:29.487 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:37:38.109515+0000", 2024-08-19T14:38:29.487 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.477e-06 2024-08-19T14:38:29.487 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.487 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.487 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:38:29.488 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:37:38.109520+0000", 2024-08-19T14:38:29.488 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.254e-06 2024-08-19T14:38:29.488 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.488 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.488 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:38:29.488 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:37:38.109532+0000", 2024-08-19T14:38:29.488 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2439999999999999e-05 2024-08-19T14:38:29.488 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.488 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.488 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:38:29.488 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:37:38.109538+0000", 2024-08-19T14:38:29.488 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4979999999999997e-06 2024-08-19T14:38:29.488 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.488 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.489 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:38:29.489 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:37:38.109594+0000", 2024-08-19T14:38:29.489 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.5862000000000001e-05 2024-08-19T14:38:29.489 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.489 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.489 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:38:29.489 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:37:38.109683+0000", 2024-08-19T14:38:29.489 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.9009000000000001e-05 2024-08-19T14:38:29.489 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.489 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.489 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,9", 2024-08-19T14:38:29.489 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:37:38.109759+0000", 2024-08-19T14:38:29.489 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.6866000000000006e-05 2024-08-19T14:38:29.489 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.490 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.490 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:38:29.490 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:37:38.113563+0000", 2024-08-19T14:38:29.490 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0038036659999999998 2024-08-19T14:38:29.490 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.490 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.490 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:38:29.490 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:37:38.113640+0000", 2024-08-19T14:38:29.490 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.7385999999999994e-05 2024-08-19T14:38:29.490 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.490 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.490 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:38:29.490 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:37:38.356114+0000", 2024-08-19T14:38:29.490 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.24247331799999999 2024-08-19T14:38:29.491 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.491 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.491 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:38:29.491 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:37:38.356172+0000", 2024-08-19T14:38:29.491 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7705999999999999e-05 2024-08-19T14:38:29.491 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.491 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.491 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:38:29.491 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:37:38.356214+0000", 2024-08-19T14:38:29.491 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.2051000000000001e-05 2024-08-19T14:38:29.491 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:38:29.491 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:38:29.491 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:38:29.491 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.492 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.492 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.16962.0:179504 7.5 7:aeb75f64:::benchmark_data_smithi045_115842_object61344:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e678)", 2024-08-19T14:38:29.492 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:37:38.120691+0000", 2024-08-19T14:38:29.492 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 51.284032107999998, 2024-08-19T14:38:29.492 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.23564085600000001, 2024-08-19T14:38:29.492 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:38:29.492 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:38:29.492 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:38:29.492 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:38:29.492 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.16962", 2024-08-19T14:38:29.492 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:38:29.492 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 179504 2024-08-19T14:38:29.492 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.492 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:38:29.493 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.493 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:38:29.493 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:37:38.120691+0000", 2024-08-19T14:38:29.493 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:38:29.493 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.493 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.493 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:38:29.493 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:37:38.120691+0000", 2024-08-19T14:38:29.493 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:38:29.493 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.493 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.493 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:38:29.493 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:37:38.120692+0000", 2024-08-19T14:38:29.493 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.584e-06 2024-08-19T14:38:29.493 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.494 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.494 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:38:29.494 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:37:38.120698+0000", 2024-08-19T14:38:29.494 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.0390000000000001e-06 2024-08-19T14:38:29.494 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.494 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.494 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:38:29.494 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:37:38.120711+0000", 2024-08-19T14:38:29.494 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3069e-05 2024-08-19T14:38:29.494 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.494 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.494 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:38:29.494 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:37:38.120718+0000", 2024-08-19T14:38:29.495 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.6900000000000003e-06 2024-08-19T14:38:29.495 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.495 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.495 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:38:29.495 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:37:38.120773+0000", 2024-08-19T14:38:29.495 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4466999999999999e-05 2024-08-19T14:38:29.495 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.495 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.495 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:38:29.495 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:37:38.120908+0000", 2024-08-19T14:38:29.495 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000135368 2024-08-19T14:38:29.495 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.495 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.495 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,9", 2024-08-19T14:38:29.496 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:37:38.121046+0000", 2024-08-19T14:38:29.496 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013834399999999999 2024-08-19T14:38:29.496 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.496 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.496 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:38:29.496 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:37:38.124953+0000", 2024-08-19T14:38:29.496 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0039070240000000003 2024-08-19T14:38:29.496 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.496 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.496 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:38:29.496 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:37:38.125677+0000", 2024-08-19T14:38:29.496 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00072422299999999997 2024-08-19T14:38:29.496 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.496 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.497 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:38:29.497 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:37:38.356251+0000", 2024-08-19T14:38:29.497 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.23057392600000001 2024-08-19T14:38:29.497 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.497 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.497 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:38:29.497 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:37:38.356296+0000", 2024-08-19T14:38:29.497 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.4264000000000002e-05 2024-08-19T14:38:29.497 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.497 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.497 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:38:29.497 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:37:38.356332+0000", 2024-08-19T14:38:29.497 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.5858000000000001e-05 2024-08-19T14:38:29.497 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:38:29.498 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:38:29.498 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:38:29.498 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.498 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.498 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.16962.0:192062 7.5 7:a47b837d:::benchmark_data_smithi045_115842_object73902:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e686)", 2024-08-19T14:38:29.498 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:38:06.950230+0000", 2024-08-19T14:38:29.498 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 22.454492650999999, 2024-08-19T14:38:29.498 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.21419580199999999, 2024-08-19T14:38:29.498 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:38:29.498 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:38:29.498 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:38:29.498 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:38:29.498 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.16962", 2024-08-19T14:38:29.499 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:38:29.499 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 192062 2024-08-19T14:38:29.499 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.499 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:38:29.499 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.499 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:38:29.499 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:06.950230+0000", 2024-08-19T14:38:29.499 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:38:29.499 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.499 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.499 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:38:29.499 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:06.950230+0000", 2024-08-19T14:38:29.499 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:38:29.499 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.499 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.500 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:38:29.500 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:06.950231+0000", 2024-08-19T14:38:29.500 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1769999999999999e-06 2024-08-19T14:38:29.500 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.500 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.500 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:38:29.500 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:06.950236+0000", 2024-08-19T14:38:29.500 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.8180000000000003e-06 2024-08-19T14:38:29.500 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.500 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.500 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:38:29.500 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:06.950247+0000", 2024-08-19T14:38:29.500 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.116e-05 2024-08-19T14:38:29.501 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.501 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.501 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:38:29.501 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:06.950254+0000", 2024-08-19T14:38:29.501 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.0659999999999998e-06 2024-08-19T14:38:29.501 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.501 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.501 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:38:29.501 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:06.950303+0000", 2024-08-19T14:38:29.501 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.9024999999999999e-05 2024-08-19T14:38:29.501 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.501 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.501 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:38:29.501 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:06.950389+0000", 2024-08-19T14:38:29.502 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.5402999999999995e-05 2024-08-19T14:38:29.502 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.502 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.502 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,9", 2024-08-19T14:38:29.502 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:06.950470+0000", 2024-08-19T14:38:29.502 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.0797000000000006e-05 2024-08-19T14:38:29.502 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.502 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.502 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:38:29.502 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:06.954233+0000", 2024-08-19T14:38:29.502 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.003762998 2024-08-19T14:38:29.502 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.502 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.503 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:38:29.503 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:06.968506+0000", 2024-08-19T14:38:29.503 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.014273255 2024-08-19T14:38:29.503 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.503 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.505 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:38:29.505 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:07.164358+0000", 2024-08-19T14:38:29.505 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.19585237599999999 2024-08-19T14:38:29.505 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.505 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.505 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:38:29.505 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:07.164396+0000", 2024-08-19T14:38:29.505 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.7744999999999999e-05 2024-08-19T14:38:29.506 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.506 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.506 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:38:29.506 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:07.164426+0000", 2024-08-19T14:38:29.506 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.9981999999999999e-05 2024-08-19T14:38:29.506 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:38:29.506 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:38:29.506 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:38:29.506 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.506 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.506 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.16962.0:192064 7.a 7:56f07884:::benchmark_data_smithi045_115842_object73904:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e686)", 2024-08-19T14:38:29.506 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:38:06.954647+0000", 2024-08-19T14:38:29.506 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 22.450075974000001, 2024-08-19T14:38:29.506 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20293466199999999, 2024-08-19T14:38:29.507 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:38:29.507 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:38:29.507 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:38:29.507 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:38:29.507 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.16962", 2024-08-19T14:38:29.507 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:38:29.507 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 192064 2024-08-19T14:38:29.507 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.507 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:38:29.507 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.507 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:38:29.507 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:06.954647+0000", 2024-08-19T14:38:29.507 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:38:29.507 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.508 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.508 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:38:29.508 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:06.954647+0000", 2024-08-19T14:38:29.508 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:38:29.508 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.508 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.508 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:38:29.508 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:06.954648+0000", 2024-08-19T14:38:29.508 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1710000000000001e-06 2024-08-19T14:38:29.508 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.508 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.508 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:38:29.508 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:06.954652+0000", 2024-08-19T14:38:29.508 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.2869999999999998e-06 2024-08-19T14:38:29.508 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.509 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.509 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:38:29.509 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:06.954662+0000", 2024-08-19T14:38:29.509 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.013e-05 2024-08-19T14:38:29.509 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.509 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.509 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:38:29.509 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:06.954670+0000", 2024-08-19T14:38:29.509 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.1559999999999998e-06 2024-08-19T14:38:29.509 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.509 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.509 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:38:29.509 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:06.954715+0000", 2024-08-19T14:38:29.509 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.5309000000000003e-05 2024-08-19T14:38:29.510 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.510 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.510 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:38:29.510 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:06.954810+0000", 2024-08-19T14:38:29.510 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.5476999999999994e-05 2024-08-19T14:38:29.510 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.510 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.510 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2,9", 2024-08-19T14:38:29.510 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:06.954911+0000", 2024-08-19T14:38:29.510 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000101111 2024-08-19T14:38:29.510 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.510 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.510 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:38:29.511 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:06.958488+0000", 2024-08-19T14:38:29.511 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.003576531 2024-08-19T14:38:29.511 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.511 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.511 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:38:29.511 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:06.971269+0000", 2024-08-19T14:38:29.511 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.012780659999999999 2024-08-19T14:38:29.511 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.511 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.511 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:38:29.511 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:07.156273+0000", 2024-08-19T14:38:29.511 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.18500446700000001 2024-08-19T14:38:29.511 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.511 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.511 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:38:29.512 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:07.156346+0000", 2024-08-19T14:38:29.512 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.3046999999999994e-05 2024-08-19T14:38:29.512 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.512 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.512 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:38:29.512 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:07.157581+0000", 2024-08-19T14:38:29.512 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001235316 2024-08-19T14:38:29.512 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:38:29.512 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:38:29.512 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:38:29.512 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.512 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.512 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.16962.0:199422 7.a 7:5ea5870d:::benchmark_data_smithi045_115842_object81262:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e690)", 2024-08-19T14:38:29.512 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:38:22.756822+0000", 2024-08-19T14:38:29.513 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 6.6479005610000002, 2024-08-19T14:38:29.513 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.247167585, 2024-08-19T14:38:29.513 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:38:29.513 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:38:29.513 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:38:29.513 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:38:29.513 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.16962", 2024-08-19T14:38:29.513 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:38:29.513 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 199422 2024-08-19T14:38:29.513 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.513 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:38:29.513 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.513 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:38:29.513 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:22.756822+0000", 2024-08-19T14:38:29.514 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:38:29.514 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.514 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.514 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:38:29.514 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:22.756822+0000", 2024-08-19T14:38:29.514 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:38:29.514 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.514 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.514 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:38:29.514 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:22.756824+0000", 2024-08-19T14:38:29.514 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.488e-06 2024-08-19T14:38:29.514 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.514 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.514 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:38:29.515 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:22.756829+0000", 2024-08-19T14:38:29.515 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.6810000000000003e-06 2024-08-19T14:38:29.515 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.515 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.515 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:38:29.515 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:22.756842+0000", 2024-08-19T14:38:29.515 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2442e-05 2024-08-19T14:38:29.515 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.515 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.515 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:38:29.515 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:22.756849+0000", 2024-08-19T14:38:29.515 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.2069999999999998e-06 2024-08-19T14:38:29.515 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.515 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.516 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:38:29.516 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:22.756922+0000", 2024-08-19T14:38:29.516 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.2577000000000007e-05 2024-08-19T14:38:29.516 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.516 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.516 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:38:29.516 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:22.757021+0000", 2024-08-19T14:38:29.516 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.9148999999999995e-05 2024-08-19T14:38:29.516 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.516 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.516 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2,9", 2024-08-19T14:38:29.516 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:22.757104+0000", 2024-08-19T14:38:29.516 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.3244999999999993e-05 2024-08-19T14:38:29.516 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.516 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.517 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:38:29.517 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:22.760873+0000", 2024-08-19T14:38:29.517 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0037694849999999999 2024-08-19T14:38:29.517 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.517 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.517 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:38:29.517 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:22.762427+0000", 2024-08-19T14:38:29.517 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0015536689999999999 2024-08-19T14:38:29.517 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.517 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.517 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:38:29.517 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:23.003068+0000", 2024-08-19T14:38:29.517 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.24064074999999999 2024-08-19T14:38:29.518 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.518 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.518 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:38:29.518 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:23.003118+0000", 2024-08-19T14:38:29.518 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0061000000000001e-05 2024-08-19T14:38:29.518 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.518 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.518 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:38:29.518 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:23.003990+0000", 2024-08-19T14:38:29.518 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00087183099999999999 2024-08-19T14:38:29.518 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:38:29.518 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:38:29.518 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:38:29.519 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.519 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.519 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.16962.0:199432 7.a 7:55097736:::benchmark_data_smithi045_115842_object81272:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e690)", 2024-08-19T14:38:29.519 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:38:22.776920+0000", 2024-08-19T14:38:29.519 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 6.6278023939999997, 2024-08-19T14:38:29.519 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.23688290400000001, 2024-08-19T14:38:29.519 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:38:29.519 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:38:29.519 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:38:29.519 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:38:29.519 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.16962", 2024-08-19T14:38:29.519 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:38:29.519 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 199432 2024-08-19T14:38:29.519 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.520 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:38:29.520 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.520 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:38:29.520 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:22.776920+0000", 2024-08-19T14:38:29.520 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:38:29.520 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.520 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.520 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:38:29.520 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:22.776920+0000", 2024-08-19T14:38:29.520 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:38:29.520 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.520 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.520 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:38:29.520 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:22.776922+0000", 2024-08-19T14:38:29.520 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7710000000000001e-06 2024-08-19T14:38:29.521 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.521 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.521 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:38:29.521 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:22.776930+0000", 2024-08-19T14:38:29.521 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.8369999999999997e-06 2024-08-19T14:38:29.521 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.521 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.521 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:38:29.521 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:22.776949+0000", 2024-08-19T14:38:29.521 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8981e-05 2024-08-19T14:38:29.521 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.521 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.521 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:38:29.521 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:22.776958+0000", 2024-08-19T14:38:29.522 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.037e-06 2024-08-19T14:38:29.522 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.522 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.522 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:38:29.522 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:22.777027+0000", 2024-08-19T14:38:29.522 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.8512999999999999e-05 2024-08-19T14:38:29.522 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.522 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.522 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:38:29.522 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:22.777182+0000", 2024-08-19T14:38:29.522 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015498700000000001 2024-08-19T14:38:29.522 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.522 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.522 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2,9", 2024-08-19T14:38:29.523 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:22.777337+0000", 2024-08-19T14:38:29.523 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000155494 2024-08-19T14:38:29.523 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.523 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.523 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:38:29.523 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:22.781561+0000", 2024-08-19T14:38:29.523 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0042234919999999997 2024-08-19T14:38:29.523 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.523 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.523 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:38:29.523 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:22.783069+0000", 2024-08-19T14:38:29.523 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001508941 2024-08-19T14:38:29.523 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.523 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.524 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:38:29.524 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:23.012725+0000", 2024-08-19T14:38:29.524 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.22965543399999999 2024-08-19T14:38:29.524 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.524 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.524 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:38:29.524 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:23.012801+0000", 2024-08-19T14:38:29.524 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.5622000000000002e-05 2024-08-19T14:38:29.524 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.524 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.524 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:38:29.524 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:23.013803+0000", 2024-08-19T14:38:29.524 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0010027949999999999 2024-08-19T14:38:29.524 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:38:29.525 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:38:29.525 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:38:29.525 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.525 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.525 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.16962.0:199450 7.a 7:5da024d9:::benchmark_data_smithi045_115842_object81290:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e690)", 2024-08-19T14:38:29.525 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:38:22.802980+0000", 2024-08-19T14:38:29.525 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 6.6017425730000001, 2024-08-19T14:38:29.525 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.21136637699999999, 2024-08-19T14:38:29.525 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:38:29.525 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:38:29.525 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:38:29.525 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:38:29.525 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.16962", 2024-08-19T14:38:29.525 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:38:29.526 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 199450 2024-08-19T14:38:29.526 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.526 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:38:29.526 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.526 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:38:29.526 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:22.802980+0000", 2024-08-19T14:38:29.526 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:38:29.526 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.526 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.526 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:38:29.526 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:22.802980+0000", 2024-08-19T14:38:29.526 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:38:29.526 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.526 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.527 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:38:29.527 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:22.802981+0000", 2024-08-19T14:38:29.527 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.246e-06 2024-08-19T14:38:29.527 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.527 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.527 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:38:29.527 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:22.802987+0000", 2024-08-19T14:38:29.527 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.8649999999999998e-06 2024-08-19T14:38:29.527 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.527 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.527 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:38:29.527 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:22.802999+0000", 2024-08-19T14:38:29.527 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2033e-05 2024-08-19T14:38:29.527 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.528 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.528 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:38:29.528 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:22.803007+0000", 2024-08-19T14:38:29.528 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.5079999999999999e-06 2024-08-19T14:38:29.528 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.528 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.528 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:38:29.528 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:22.803061+0000", 2024-08-19T14:38:29.528 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4282e-05 2024-08-19T14:38:29.528 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.528 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.528 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:38:29.528 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:22.803179+0000", 2024-08-19T14:38:29.528 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000117853 2024-08-19T14:38:29.529 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.529 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.529 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2,9", 2024-08-19T14:38:29.529 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:22.803271+0000", 2024-08-19T14:38:29.529 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.1769999999999997e-05 2024-08-19T14:38:29.529 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.529 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.529 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:38:29.529 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:22.807206+0000", 2024-08-19T14:38:29.529 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0039349579999999997 2024-08-19T14:38:29.529 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.529 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.529 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:38:29.529 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:22.807451+0000", 2024-08-19T14:38:29.530 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00024569899999999999 2024-08-19T14:38:29.530 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.530 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.530 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:38:29.530 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:23.013860+0000", 2024-08-19T14:38:29.530 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.206408435 2024-08-19T14:38:29.530 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.530 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.530 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:38:29.530 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:23.013912+0000", 2024-08-19T14:38:29.530 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2590000000000003e-05 2024-08-19T14:38:29.530 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.530 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.530 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:38:29.531 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:23.014347+0000", 2024-08-19T14:38:29.531 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00043413799999999998 2024-08-19T14:38:29.531 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:38:29.531 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:38:29.531 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:38:29.531 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.531 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.531 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.16962.0:199454 7.a 7:58b3f1d9:::benchmark_data_smithi045_115842_object81294:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e690)", 2024-08-19T14:38:29.531 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:38:22.813391+0000", 2024-08-19T14:38:29.531 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 6.5913322340000002, 2024-08-19T14:38:29.531 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.201428947, 2024-08-19T14:38:29.531 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:38:29.531 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:38:29.531 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:38:29.532 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:38:29.532 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.16962", 2024-08-19T14:38:29.532 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:38:29.532 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 199454 2024-08-19T14:38:29.532 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.532 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:38:29.532 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.532 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:38:29.532 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:22.813391+0000", 2024-08-19T14:38:29.532 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:38:29.532 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.532 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.532 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:38:29.532 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:22.813391+0000", 2024-08-19T14:38:29.532 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:38:29.533 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.533 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.533 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:38:29.533 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:22.813392+0000", 2024-08-19T14:38:29.533 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7400000000000001e-06 2024-08-19T14:38:29.533 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.533 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.533 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:38:29.533 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:22.813398+0000", 2024-08-19T14:38:29.533 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7930000000000003e-06 2024-08-19T14:38:29.533 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.533 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.533 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:38:29.533 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:22.813411+0000", 2024-08-19T14:38:29.534 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3067e-05 2024-08-19T14:38:29.534 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.534 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.534 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:38:29.534 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:22.813418+0000", 2024-08-19T14:38:29.534 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.6100000000000002e-06 2024-08-19T14:38:29.534 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.534 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.534 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:38:29.534 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:22.813483+0000", 2024-08-19T14:38:29.534 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.5067000000000006e-05 2024-08-19T14:38:29.534 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.534 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.534 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:38:29.535 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:22.813648+0000", 2024-08-19T14:38:29.535 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00016545899999999999 2024-08-19T14:38:29.535 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.535 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.535 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2,9", 2024-08-19T14:38:29.535 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:22.813816+0000", 2024-08-19T14:38:29.535 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00016726800000000001 2024-08-19T14:38:29.535 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.535 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.535 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:38:29.535 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:22.817890+0000", 2024-08-19T14:38:29.535 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0040742449999999998 2024-08-19T14:38:29.535 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.535 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.535 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:38:29.536 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:22.817988+0000", 2024-08-19T14:38:29.536 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.8503000000000002e-05 2024-08-19T14:38:29.536 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.536 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.536 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:38:29.536 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:23.014403+0000", 2024-08-19T14:38:29.536 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.19641430400000001 2024-08-19T14:38:29.536 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.536 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.536 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:38:29.536 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:23.014458+0000", 2024-08-19T14:38:29.536 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.5183999999999999e-05 2024-08-19T14:38:29.536 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:38:29.536 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:38:29.537 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:38:29.537 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:38:23.014819+0000", 2024-08-19T14:38:29.537 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00036170700000000001 2024-08-19T14:38:29.537 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:38:29.537 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:38:29.537 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:38:29.537 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:38:29.537 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:38:29.537 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:38:29.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:29.586 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:29.586 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:38:29.687 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:38:29.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:29.711 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:29.711 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:38:29.812 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:38:29.834 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:29.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:29.836 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:38:29.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:29 smithi031 ceph-mon[16336]: pgmap v1648: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.2 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 498 op/s; 16030/106548 objects degraded (15.045%); 4333/106548 objects misplaced (4.067%); 3.4 MiB/s, 108 objects/s recovering 2024-08-19T14:38:29.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:29 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:38:29.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:29 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:38:29.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:29 smithi031 ceph-mon[16336]: osdmap e692: 12 total, 12 up, 10 in 2024-08-19T14:38:29.937 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:38:29.958 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:29.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:29.960 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:38:30.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:29 smithi032 ceph-mon[20004]: pgmap v1648: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.2 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 498 op/s; 16030/106548 objects degraded (15.045%); 4333/106548 objects misplaced (4.067%); 3.4 MiB/s, 108 objects/s recovering 2024-08-19T14:38:30.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:29 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:38:30.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:29 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:38:30.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:29 smithi032 ceph-mon[20004]: osdmap e692: 12 total, 12 up, 10 in 2024-08-19T14:38:30.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:29 smithi045 ceph-mon[20143]: pgmap v1648: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.2 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 498 op/s; 16030/106548 objects degraded (15.045%); 4333/106548 objects misplaced (4.067%); 3.4 MiB/s, 108 objects/s recovering 2024-08-19T14:38:30.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:29 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:38:30.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:29 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:38:30.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:29 smithi045 ceph-mon[20143]: osdmap e692: 12 total, 12 up, 10 in 2024-08-19T14:38:30.061 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:38:30.082 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:30.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:30.083 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:38:30.183 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:38:30.218 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:30.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:30.219 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:38:30.320 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:38:30.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:30.354 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:30.354 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:38:30.455 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:38:30.477 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:30.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:30.478 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:38:30.579 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:38:30.583 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-08-19T14:38:30.601 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:30.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:30.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:38:30.704 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:38:30.725 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:30.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:30.726 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:38:30.739 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:38:30.739 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"ed7d1c6e-5e33-11ef-bccf-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"nodeep-scrub flag(s) set","count":12},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 16030/106548 objects degraded (15.045%), 5 pgs degraded, 5 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":1888,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":692,"num_osds":12,"num_up_osds":12,"osd_up_since":1724078110,"num_in_osds":10,"osd_in_since":1724078051,"num_remapped_pgs":5},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":42},{"state_name":"active+recovering+undersized+degraded+remapped","count":5}],"num_pgs":47,"num_pools":2,"num_objects":35516,"data_bytes":2327970360,"bytes_used":15870513152,"bytes_avail":944079843328,"bytes_total":959950356480,"degraded_objects":16030,"degraded_total":106548,"degraded_ratio":0.15044862409430493,"misplaced_objects":4333,"misplaced_total":106548,"misplaced_ratio":0.04066711716784923,"recovering_objects_per_sec":108,"recovering_bytes_per_sec":3517322,"recovering_keys_per_sec":0,"num_objects_recovered":1081,"num_bytes_recovered":35192832,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":498},"fsmap":{"epoch":1,"btime":"2024-08-19T14:05:22:841940+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.31:8443/"}},"servicemap":{"epoch":43,"modified":"2024-08-19T14:38:18.239512+0000","services":{"mon":{"daemons":{"summary":"","c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-08-19T14:38:30.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:30 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3616316933' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:38:30.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:30 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3001099316' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-08-19T14:38:30.827 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:38:30.851 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:30.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:30.853 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:38:30.863 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:38:30.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:30 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3616316933' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:38:30.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:30 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3001099316' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-08-19T14:38:30.954 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:38:30.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:30.979 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:30.979 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:38:31.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:30 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3616316933' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:38:31.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:30 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3001099316' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-08-19T14:38:31.080 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:38:31.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:31.129 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:31.129 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:38:31.230 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:38:31.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:31.254 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:31.254 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:38:31.355 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:38:31.371 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-19T14:38:31.377 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:31.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:31.379 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:38:31.382 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:38:31.383 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:38:31.480 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:38:31.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:31.524 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:31.524 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:38:31.626 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:38:31.648 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:31.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:31.649 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:38:31.733 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:38:31.733 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:38:31.733 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:38:31.734 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:38:31.734 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:38:31.734 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:38:31.734 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:38:31.734 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:38:31.734 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:38:31.734 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:38:31.735 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:38:31.735 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:38:31.735 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:38:31.735 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:38:31.735 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:38:31.735 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:38:31.735 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:38:31.735 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:38:31.735 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:38:31.735 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:38:31.735 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:38:31.735 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:38:31.736 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:38:31.736 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:38:31.749 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:38:31.770 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:31.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:31.771 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:38:31.872 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:38:31.895 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:31.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:31.896 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:38:31.997 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:38:32.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:31 smithi032 ceph-mon[20004]: pgmap v1650: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.1 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 458 op/s; 15745/104295 objects degraded (15.097%); 4333/104295 objects misplaced (4.155%); 4.3 MiB/s, 116 objects/s recovering 2024-08-19T14:38:32.018 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:32.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:32.019 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:38:32.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:31 smithi045 ceph-mon[20143]: pgmap v1650: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.1 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 458 op/s; 15745/104295 objects degraded (15.097%); 4333/104295 objects misplaced (4.155%); 4.3 MiB/s, 116 objects/s recovering 2024-08-19T14:38:32.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:31 smithi031 ceph-mon[16336]: pgmap v1650: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.1 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 458 op/s; 15745/104295 objects degraded (15.097%); 4333/104295 objects misplaced (4.155%); 4.3 MiB/s, 116 objects/s recovering 2024-08-19T14:38:32.120 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:38:32.173 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:32.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:32.175 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:38:32.275 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:38:32.315 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:32.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:32.316 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:38:32.417 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:38:32.442 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:32.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:32.443 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:38:32.544 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:38:32.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:32.570 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:32.570 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:38:32.671 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:38:32.707 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:32.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:32.708 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:38:32.748 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-08-19T14:38:32.809 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:38:32.835 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:32.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:32.836 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:38:32.937 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:38:32.964 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:32.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:32.965 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:38:33.049 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:38:33.066 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:38:33.087 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:33.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:33.089 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:38:33.189 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:38:33.213 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:33.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:33.214 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:38:33.315 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:38:33.353 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:33.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:33.354 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:38:33.455 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:38:33.478 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:33.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:33.479 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:38:33.580 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:38:33.603 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:33.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:33.605 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:38:33.611 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:38:33.611 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:38:33.706 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:38:33.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:33.747 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:33.747 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:38:33.848 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:38:33.884 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:33.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:33.885 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:38:33.986 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:38:34.009 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:34.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:34.011 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:38:34.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:33 smithi045 ceph-mon[20143]: pgmap v1651: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.1 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 526 op/s; 15520/100785 objects degraded (15.399%); 4333/100785 objects misplaced (4.299%); 4.3 MiB/s, 131 objects/s recovering 2024-08-19T14:38:34.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:33 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 15520/100785 objects degraded (15.399%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-19T14:38:34.111 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:38:34.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:33 smithi031 ceph-mon[16336]: pgmap v1651: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.1 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 526 op/s; 15520/100785 objects degraded (15.399%); 4333/100785 objects misplaced (4.299%); 4.3 MiB/s, 131 objects/s recovering 2024-08-19T14:38:34.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:33 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 15520/100785 objects degraded (15.399%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-19T14:38:34.134 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:34.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:34.135 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:38:34.236 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:38:34.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:33 smithi032 ceph-mon[20004]: pgmap v1651: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.1 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 526 op/s; 15520/100785 objects degraded (15.399%); 4333/100785 objects misplaced (4.299%); 4.3 MiB/s, 131 objects/s recovering 2024-08-19T14:38:34.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:33 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 15520/100785 objects degraded (15.399%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-19T14:38:34.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:34.259 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:34.259 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:38:34.274 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-08-19T14:38:34.360 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:38:34.382 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:34.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:34.383 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:38:34.485 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:38:34.506 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:34.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:34.507 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:38:34.609 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:38:34.615 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:38:34.658 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:34.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:34.659 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:38:34.760 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:38:34.810 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:34.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:34.812 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:38:34.912 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:38:34.944 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:34.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:34.945 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:38:35.046 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:38:35.069 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:35.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:35.070 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:38:35.172 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:38:35.203 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:35.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:35.204 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:38:35.305 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:38:35.339 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:35.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:35.340 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:38:35.387 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:38:35.387 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:38:35.441 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:38:35.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:35.483 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:35.483 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:38:35.584 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:38:35.630 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:35.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:35.631 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:38:35.731 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:38:35.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:35.786 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:35.787 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:38:35.888 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:38:35.927 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:35.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:35.928 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:38:36.029 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:38:36.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:35 smithi045 ceph-mon[20143]: pgmap v1652: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.0 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 399 op/s; 15548/99162 objects degraded (15.679%); 4225/99162 objects misplaced (4.261%); 2.2 MiB/s, 111 objects/s recovering 2024-08-19T14:38:36.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:35 smithi045 ceph-mon[20143]: 1.11 scrub starts 2024-08-19T14:38:36.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:35 smithi045 ceph-mon[20143]: 1.11 scrub ok 2024-08-19T14:38:36.054 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:36.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:36.055 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:38:36.063 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-08-19T14:38:36.086 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:35 smithi031 ceph-mon[16336]: pgmap v1652: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.0 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 399 op/s; 15548/99162 objects degraded (15.679%); 4225/99162 objects misplaced (4.261%); 2.2 MiB/s, 111 objects/s recovering 2024-08-19T14:38:36.086 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:35 smithi031 ceph-mon[16336]: 1.11 scrub starts 2024-08-19T14:38:36.086 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:35 smithi031 ceph-mon[16336]: 1.11 scrub ok 2024-08-19T14:38:36.156 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:38:36.178 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:36.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:36.179 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:38:36.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:35 smithi032 ceph-mon[20004]: pgmap v1652: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.0 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 399 op/s; 15548/99162 objects degraded (15.679%); 4225/99162 objects misplaced (4.261%); 2.2 MiB/s, 111 objects/s recovering 2024-08-19T14:38:36.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:35 smithi032 ceph-mon[20004]: 1.11 scrub starts 2024-08-19T14:38:36.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:35 smithi032 ceph-mon[20004]: 1.11 scrub ok 2024-08-19T14:38:36.280 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:38:36.303 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:36.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:36.304 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:38:36.405 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:38:36.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:36.429 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:36.429 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:38:36.530 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:38:36.554 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:36.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:36.555 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:38:36.655 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:38:36.700 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:36.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:36.701 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:38:36.802 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:38:36.824 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:36.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:36.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:38:36.926 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:38:36.966 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:36.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:36.967 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:38:37.068 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:38:37.092 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:37.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:37.093 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:38:37.194 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:38:37.216 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:37.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:37.217 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:38:37.318 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:38:37.340 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:37.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:37.341 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:38:37.365 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-08-19T14:38:37.365 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-19T14:38:37.442 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:38:37.502 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:38:37.540 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:38:37.542 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:37.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:37.544 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:38:37.645 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:38:37.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:37.700 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:37.700 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:38:37.801 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:38:37.822 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:37.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:37.823 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:38:37.925 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:38:37.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:37.948 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:37.948 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:38:37.966 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:37 smithi031 ceph-mon[16336]: pgmap v1653: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.9 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 467 op/s; 15221/94617 objects degraded (16.087%); 4225/94617 objects misplaced (4.465%); 4.4 MiB/s, 111 objects/s recovering 2024-08-19T14:38:38.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:37 smithi045 ceph-mon[20143]: pgmap v1653: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.9 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 467 op/s; 15221/94617 objects degraded (16.087%); 4225/94617 objects misplaced (4.465%); 4.4 MiB/s, 111 objects/s recovering 2024-08-19T14:38:38.049 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:38:38.070 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:38.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:38.071 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:38:38.172 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:38:38.193 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:38.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:38.195 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:38:38.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:37 smithi032 ceph-mon[20004]: pgmap v1653: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.9 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 467 op/s; 15221/94617 objects degraded (16.087%); 4225/94617 objects misplaced (4.465%); 4.4 MiB/s, 111 objects/s recovering 2024-08-19T14:38:38.295 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:38:38.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:38.344 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:38.344 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:38:38.445 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:38:38.467 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:38.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:38.468 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:38:38.570 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:38:38.607 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:38.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:38.609 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:38:38.710 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:38:38.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:38.731 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:38.732 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:38:38.765 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:38:38.832 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:38:38.836 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is unset 2024-08-19T14:38:38.855 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:38.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:38.856 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:38:38.957 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:38:38.978 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:38.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:38.980 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:38:39.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:38 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 15221/94617 objects degraded (16.087%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-19T14:38:39.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:38 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:38:39.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:38 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:38:39.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:38 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:38:39.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:38 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:38:39.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:38 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1061989589' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:38:39.080 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:38:39.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:38 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 15221/94617 objects degraded (16.087%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-19T14:38:39.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:38 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:38:39.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:38 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:38:39.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:38 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:38:39.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:38 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:38:39.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:38 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1061989589' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:38:39.134 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:39.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:39.136 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:38:39.237 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:38:39.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:38 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 15221/94617 objects degraded (16.087%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-19T14:38:39.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:38 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:38:39.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:38 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:38:39.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:38 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:38:39.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:38 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:38:39.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:38 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1061989589' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:38:39.259 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:39.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:39.260 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:38:39.361 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:38:39.382 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:39.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:39.384 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:38:39.485 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:38:39.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:39.510 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:39.511 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:38:39.612 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:38:39.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:39.635 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:39.635 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:38:39.736 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:38:39.759 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:39.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:39.760 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:38:39.861 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:38:39.882 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:39.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:39.884 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:38:39.985 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:38:40.006 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:40.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:40.007 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:38:40.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:39 smithi045 ceph-mon[20143]: pgmap v1654: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.9 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 477 op/s; 15059/92232 objects degraded (16.327%); 4130/92232 objects misplaced (4.478%); 3.8 MiB/s, 125 objects/s recovering 2024-08-19T14:38:40.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:39 smithi045 ceph-mon[20143]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:38:40.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:39 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1061989589' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:38:40.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:39 smithi045 ceph-mon[20143]: osdmap e693: 12 total, 12 up, 10 in 2024-08-19T14:38:40.108 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:38:40.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:39 smithi031 ceph-mon[16336]: pgmap v1654: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.9 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 477 op/s; 15059/92232 objects degraded (16.327%); 4130/92232 objects misplaced (4.478%); 3.8 MiB/s, 125 objects/s recovering 2024-08-19T14:38:40.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:39 smithi031 ceph-mon[16336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:38:40.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:39 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1061989589' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:38:40.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:39 smithi031 ceph-mon[16336]: osdmap e693: 12 total, 12 up, 10 in 2024-08-19T14:38:40.130 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:40.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:40.131 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:38:40.232 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:38:40.253 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:40.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:40.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:38:40.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:39 smithi032 ceph-mon[20004]: pgmap v1654: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.9 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 477 op/s; 15059/92232 objects degraded (16.327%); 4130/92232 objects misplaced (4.478%); 3.8 MiB/s, 125 objects/s recovering 2024-08-19T14:38:40.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:39 smithi032 ceph-mon[20004]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:38:40.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:39 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1061989589' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:38:40.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:39 smithi032 ceph-mon[20004]: osdmap e693: 12 total, 12 up, 10 in 2024-08-19T14:38:40.355 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:38:40.399 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:40.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:40.400 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:38:40.501 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:38:40.522 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:40.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:40.523 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:38:40.624 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:38:40.646 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:40.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:40.647 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:38:40.748 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:38:40.769 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:40.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:40.770 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:38:40.871 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:38:40.911 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:40.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:40.912 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:38:41.014 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:38:41.039 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:41.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:41.040 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:38:41.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:40 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:38:41.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:40 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:38:41.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:40 smithi045 ceph-mon[20143]: osdmap e694: 12 total, 12 up, 10 in 2024-08-19T14:38:41.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:40 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:38:41.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:40 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:38:41.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:40 smithi031 ceph-mon[16336]: osdmap e694: 12 total, 12 up, 10 in 2024-08-19T14:38:41.142 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:38:41.164 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:41.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:41.165 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:38:41.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:40 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:38:41.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:40 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:38:41.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:40 smithi032 ceph-mon[20004]: osdmap e694: 12 total, 12 up, 10 in 2024-08-19T14:38:41.266 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:38:41.288 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:41.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:41.289 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:38:41.390 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:38:41.416 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:41.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:41.417 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:38:41.517 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:38:41.540 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:41.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:41.541 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:38:41.642 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:38:41.663 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:41.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:41.665 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:38:41.766 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:38:41.767 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:38:41.767 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:38:41.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:41.811 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:41.811 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:38:41.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:41 smithi031 ceph-mon[16336]: pgmap v1657: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.8 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 441 op/s; 14764/90192 objects degraded (16.370%); 4130/90192 objects misplaced (4.579%); 4.8 MiB/s, 130 objects/s recovering 2024-08-19T14:38:41.912 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:38:41.934 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:41.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:41.935 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:38:42.036 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:38:42.079 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:42.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:42.080 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:38:42.181 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:38:42.203 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:42.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:42.204 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:38:42.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:41 smithi032 ceph-mon[20004]: pgmap v1657: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.8 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 441 op/s; 14764/90192 objects degraded (16.370%); 4130/90192 objects misplaced (4.579%); 4.8 MiB/s, 130 objects/s recovering 2024-08-19T14:38:42.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:41 smithi045 ceph-mon[20143]: pgmap v1657: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.8 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 441 op/s; 14764/90192 objects degraded (16.370%); 4130/90192 objects misplaced (4.579%); 4.8 MiB/s, 130 objects/s recovering 2024-08-19T14:38:42.305 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:38:42.327 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:42.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:42.328 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:38:42.429 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:38:42.468 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:42.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:42.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:38:42.570 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:38:42.591 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:42.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:42.592 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:38:42.693 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:38:42.716 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:42.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:42.717 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:38:42.818 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:38:42.860 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:42.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:42.861 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:38:42.963 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:38:42.985 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:42.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:42.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:38:43.088 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:38:43.097 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-08-19T14:38:43.107 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:38:43.109 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:43.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:43.110 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:38:43.138 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:38:43.211 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:38:43.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:43.264 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:43.264 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:38:43.365 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:38:43.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:43.388 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:43.389 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:38:43.490 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:38:43.513 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:43.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:43.515 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:38:43.616 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:38:43.639 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:43.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:43.640 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:38:43.741 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:38:43.786 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:43.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:43.790 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:38:43.864 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:38:43.864 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:38:43.891 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:38:43.899 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:38:43.899 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:38:43.913 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:43.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:43.914 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:38:43.952 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:38:43.953 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:38:44.008 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:38:44.008 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:38:44.015 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:38:44.035 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:43 smithi031 ceph-mon[16336]: pgmap v1658: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.8 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 516 op/s; 14512/86769 objects degraded (16.725%); 4130/86769 objects misplaced (4.760%); 4.8 MiB/s, 148 objects/s recovering 2024-08-19T14:38:44.035 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:43 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 14512/86769 objects degraded (16.725%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-19T14:38:44.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:44.036 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:44.037 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:38:44.061 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:38:44.062 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:38:44.093 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:38:44.094 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:38:44.130 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:38:44.130 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:38:44.137 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:38:44.159 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:44.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:44.160 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:38:44.165 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:38:44.166 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:38:44.205 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:38:44.205 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:38:44.249 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:38:44.249 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:38:44.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:43 smithi032 ceph-mon[20004]: pgmap v1658: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.8 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 516 op/s; 14512/86769 objects degraded (16.725%); 4130/86769 objects misplaced (4.760%); 4.8 MiB/s, 148 objects/s recovering 2024-08-19T14:38:44.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:43 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 14512/86769 objects degraded (16.725%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-19T14:38:44.261 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:38:44.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:43 smithi045 ceph-mon[20143]: pgmap v1658: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.8 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 516 op/s; 14512/86769 objects degraded (16.725%); 4130/86769 objects misplaced (4.760%); 4.8 MiB/s, 148 objects/s recovering 2024-08-19T14:38:44.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:43 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 14512/86769 objects degraded (16.725%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-19T14:38:44.306 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:38:44.307 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:38:44.329 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:44.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:44.330 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:38:44.336 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:38:44.336 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:38:44.367 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:38:44.393 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-19T14:38:44.431 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:38:44.453 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:44.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:44.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:38:44.555 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:38:44.579 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:44.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:44.580 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:38:44.673 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:38:44.675 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-08-19T14:38:44.680 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:38:44.702 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:44.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:44.703 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:38:44.804 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:38:44.853 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:44.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:44.854 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:38:44.955 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:38:45.009 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:45.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:45.010 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:38:45.112 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:38:45.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:44 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2460807708' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:38:45.134 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:45.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:45.135 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:38:45.238 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:38:45.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:44 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2460807708' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:38:45.259 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:45.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:45.261 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:38:45.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:44 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2460807708' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:38:45.362 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:38:45.407 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:45.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:45.408 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:38:45.482 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":1659,"stamp":"2024-08-19T14:38:44.249266+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1856242232,"num_objects":28318,"num_object_clones":0,"num_object_copies":84954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14565,"num_objects_misplaced":4032,"num_objects_unfound":0,"num_objects_dirty":28318,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":208000,"num_write_kb":7562049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45796,"num_bytes_recovered":2861503008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":165237,"ondisk_log_size":165237,"up":141,"acting":136,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":143,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":14250112,"kb_used_data":5653396,"kb_used_omap":18,"kb_used_meta":8596525,"kb_avail":923201408,"statfs":{"total":959950356480,"available":945358241792,"internally_reserved":0,"allocated":5789077504,"data_stored":5758399510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18760,"internal_metadata":8802842296},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-211091456,"num_objects":-3221,"num_object_clones":0,"num_object_copies":-9663,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-656,"num_objects_misplaced":-193,"num_objects_unfound":0,"num_objects_dirty":-3221,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3221,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":960,"num_bytes_recovered":19267584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002337"},"pg_stats":[{"pgid":"7.6","version":"694'13234","reported_seq":13535,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-08-19T14:38:40.559622+0000","last_change":"2024-08-19T14:37:56.236536+0000","last_active":"2024-08-19T14:38:40.559622+0000","last_peered":"2024-08-19T14:38:40.559622+0000","last_clean":"2024-08-19T14:38:40.559622+0000","last_became_active":"2024-08-19T14:33:28.334317+0000","last_became_peered":"2024-08-19T14:33:28.334317+0000","last_unstale":"2024-08-19T14:38:40.559622+0000","last_undegraded":"2024-08-19T14:38:40.559622+0000","last_fullsized":"2024-08-19T14:38:40.559622+0000","mapping_epoch":600,"log_start":"611'3200","ondisk_log_start":"611'3200","created":600,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"684'11923","last_scrub_stamp":"2024-08-19T14:37:56.236394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:37:56.236394+0000","objects_scrubbed":3155,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:39:12.585833+0000","scrub_duration":1586,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":119275520,"num_objects":1820,"num_object_clones":0,"num_object_copies":5460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13234,"num_write_kb":481728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"70'139","reported_seq":1707,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-08-19T14:38:39.805831+0000","last_change":"2024-08-19T14:37:57.701402+0000","last_active":"2024-08-19T14:38:39.805831+0000","last_peered":"2024-08-19T14:38:39.805831+0000","last_clean":"2024-08-19T14:38:39.805831+0000","last_became_active":"2024-08-19T14:35:13.950224+0000","last_became_peered":"2024-08-19T14:35:13.950224+0000","last_unstale":"2024-08-19T14:38:39.805831+0000","last_undegraded":"2024-08-19T14:38:39.805831+0000","last_fullsized":"2024-08-19T14:38:39.805831+0000","mapping_epoch":637,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":638,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:37:57.701290+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:19:58.072083+0000","last_clean_scrub_stamp":"2024-08-19T14:37:57.701290+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:39:18.431910+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,9],"acting":[0,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.5","version":"694'12881","reported_seq":22360,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-08-19T14:38:41.703965+0000","last_change":"2024-08-19T14:35:23.022361+0000","last_active":"2024-08-19T14:38:41.703965+0000","last_peered":"2024-08-19T14:38:41.703965+0000","last_clean":"2024-08-19T14:38:41.703965+0000","last_became_active":"2024-08-19T14:35:23.009238+0000","last_became_peered":"2024-08-19T14:35:23.009238+0000","last_unstale":"2024-08-19T14:38:41.703965+0000","last_undegraded":"2024-08-19T14:38:41.703965+0000","last_fullsized":"2024-08-19T14:38:41.703965+0000","mapping_epoch":641,"log_start":"611'2800","ondisk_log_start":"611'2800","created":600,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":118030336,"num_objects":1801,"num_object_clones":0,"num_object_copies":5403,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1801,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12881,"num_write_kb":469824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4634,"num_bytes_recovered":303693824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,0],"acting":[11,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.3","version":"70'139","reported_seq":1551,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-08-19T14:38:39.797900+0000","last_change":"2024-08-19T14:33:18.804321+0000","last_active":"2024-08-19T14:38:39.797900+0000","last_peered":"2024-08-19T14:38:39.797900+0000","last_clean":"2024-08-19T14:38:39.797900+0000","last_became_active":"2024-08-19T14:18:13.804721+0000","last_became_peered":"2024-08-19T14:18:13.804721+0000","last_unstale":"2024-08-19T14:38:39.797900+0000","last_undegraded":"2024-08-19T14:38:39.797900+0000","last_fullsized":"2024-08-19T14:38:39.797900+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:33:18.804228+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:33:18.804228+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.4","version":"694'13139","reported_seq":13435,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-08-19T14:38:40.552491+0000","last_change":"2024-08-19T14:38:06.717577+0000","last_active":"2024-08-19T14:38:40.552491+0000","last_peered":"2024-08-19T14:38:40.552491+0000","last_clean":"2024-08-19T14:38:40.552491+0000","last_became_active":"2024-08-19T14:33:28.334446+0000","last_became_peered":"2024-08-19T14:33:28.334446+0000","last_unstale":"2024-08-19T14:38:40.552491+0000","last_undegraded":"2024-08-19T14:38:40.552491+0000","last_fullsized":"2024-08-19T14:38:40.552491+0000","mapping_epoch":600,"log_start":"611'3100","ondisk_log_start":"611'3100","created":600,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"686'12159","last_scrub_stamp":"2024-08-19T14:38:06.717004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:38:06.717004+0000","objects_scrubbed":2792,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:39:16.656979+0000","scrub_duration":1017,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":118423552,"num_objects":1807,"num_object_clones":0,"num_object_copies":5421,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1807,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13139,"num_write_kb":478272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"70'139","reported_seq":1507,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-08-19T14:38:39.795344+0000","last_change":"2024-08-19T14:34:55.580212+0000","last_active":"2024-08-19T14:38:39.795344+0000","last_peered":"2024-08-19T14:38:39.795344+0000","last_clean":"2024-08-19T14:38:39.795344+0000","last_became_active":"2024-08-19T14:20:50.742354+0000","last_became_peered":"2024-08-19T14:20:50.742354+0000","last_unstale":"2024-08-19T14:38:39.795344+0000","last_undegraded":"2024-08-19T14:38:39.795344+0000","last_fullsized":"2024-08-19T14:38:39.795344+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:34:55.580141+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:19:59.302931+0000","last_clean_scrub_stamp":"2024-08-19T14:34:55.580141+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"694'12697","reported_seq":12996,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-08-19T14:38:40.249808+0000","last_change":"2024-08-19T14:37:43.229201+0000","last_active":"2024-08-19T14:38:40.249808+0000","last_peered":"2024-08-19T14:38:40.249808+0000","last_clean":"2024-08-19T14:38:40.249808+0000","last_became_active":"2024-08-19T14:33:28.331833+0000","last_became_peered":"2024-08-19T14:33:28.331833+0000","last_unstale":"2024-08-19T14:38:40.249808+0000","last_undegraded":"2024-08-19T14:38:40.249808+0000","last_fullsized":"2024-08-19T14:38:40.249808+0000","mapping_epoch":600,"log_start":"611'2600","ondisk_log_start":"611'2600","created":600,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"678'11114","last_scrub_stamp":"2024-08-19T14:37:43.229072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:37:43.229072+0000","objects_scrubbed":3413,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:39:12.361681+0000","scrub_duration":910,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":119341056,"num_objects":1821,"num_object_clones":0,"num_object_copies":5463,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1821,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12697,"num_write_kb":464576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,1],"acting":[8,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1","version":"70'139","reported_seq":805,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-08-19T14:38:39.806104+0000","last_change":"2024-08-19T14:37:42.630047+0000","last_active":"2024-08-19T14:38:39.806104+0000","last_peered":"2024-08-19T14:38:39.806104+0000","last_clean":"2024-08-19T14:38:39.806104+0000","last_became_active":"2024-08-19T14:35:12.530566+0000","last_became_peered":"2024-08-19T14:35:12.530566+0000","last_unstale":"2024-08-19T14:38:39.806104+0000","last_undegraded":"2024-08-19T14:38:39.806104+0000","last_fullsized":"2024-08-19T14:38:39.806104+0000","mapping_epoch":635,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":636,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:37:42.629919+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:37:42.629919+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:39:05.497590+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,2],"acting":[0,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"694'12923","reported_seq":13200,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-08-19T14:38:43.589408+0000","last_change":"2024-08-19T14:34:52.869444+0000","last_active":"2024-08-19T14:38:43.589408+0000","last_peered":"2024-08-19T14:38:43.589408+0000","last_clean":"2024-08-19T14:38:43.589408+0000","last_became_active":"2024-08-19T14:33:28.332994+0000","last_became_peered":"2024-08-19T14:33:28.332994+0000","last_unstale":"2024-08-19T14:38:43.589408+0000","last_undegraded":"2024-08-19T14:38:43.589408+0000","last_fullsized":"2024-08-19T14:38:43.589408+0000","mapping_epoch":600,"log_start":"611'2900","ondisk_log_start":"611'2900","created":600,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"627'6737","last_scrub_stamp":"2024-08-19T14:34:52.869233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:34:52.869233+0000","objects_scrubbed":6705,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1588,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108462080,"num_objects":1655,"num_object_clones":0,"num_object_copies":4965,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12923,"num_write_kb":466496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"70'139","reported_seq":1209,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-08-19T14:38:39.791844+0000","last_change":"2024-08-19T14:35:11.537368+0000","last_active":"2024-08-19T14:38:39.791844+0000","last_peered":"2024-08-19T14:38:39.791844+0000","last_clean":"2024-08-19T14:38:39.791844+0000","last_became_active":"2024-08-19T14:35:11.536828+0000","last_became_peered":"2024-08-19T14:35:11.536828+0000","last_unstale":"2024-08-19T14:38:39.791844+0000","last_undegraded":"2024-08-19T14:38:39.791844+0000","last_fullsized":"2024-08-19T14:38:39.791844+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:27.536928+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:21:46.057624+0000","last_clean_scrub_stamp":"2024-08-19T14:28:27.536928+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"694'12708","reported_seq":22141,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-08-19T14:38:40.024526+0000","last_change":"2024-08-19T14:36:15.120872+0000","last_active":"2024-08-19T14:38:40.024526+0000","last_peered":"2024-08-19T14:38:40.024526+0000","last_clean":"2024-08-19T14:38:40.024526+0000","last_became_active":"2024-08-19T14:36:15.096070+0000","last_became_peered":"2024-08-19T14:36:15.096070+0000","last_unstale":"2024-08-19T14:38:40.024526+0000","last_undegraded":"2024-08-19T14:38:40.024526+0000","last_fullsized":"2024-08-19T14:38:40.024526+0000","mapping_epoch":653,"log_start":"611'2700","ondisk_log_start":"611'2700","created":600,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121372672,"num_objects":1852,"num_object_clones":0,"num_object_copies":5556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12708,"num_write_kb":465920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4611,"num_bytes_recovered":302186496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,9],"acting":[3,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"70'139","reported_seq":777,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-08-19T14:38:39.793941+0000","last_change":"2024-08-19T14:37:45.357396+0000","last_active":"2024-08-19T14:38:39.793941+0000","last_peered":"2024-08-19T14:38:39.793941+0000","last_clean":"2024-08-19T14:38:39.793941+0000","last_became_active":"2024-08-19T14:34:15.717054+0000","last_became_peered":"2024-08-19T14:34:15.717054+0000","last_unstale":"2024-08-19T14:38:39.793941+0000","last_undegraded":"2024-08-19T14:38:39.793941+0000","last_fullsized":"2024-08-19T14:38:39.793941+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:37:45.357293+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:27.848870+0000","last_clean_scrub_stamp":"2024-08-19T14:37:45.357293+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:38:48.964771+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"7.3","version":"694'13218","reported_seq":20448,"reported_epoch":694,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-19T14:38:40.071063+0000","last_change":"2024-08-19T14:35:13.136988+0000","last_active":"2024-08-19T14:38:40.071063+0000","last_peered":"2024-08-19T14:38:40.071063+0000","last_clean":"2024-08-19T14:33:27.312789+0000","last_became_active":"2024-08-19T14:35:13.021078+0000","last_became_peered":"2024-08-19T14:35:13.021078+0000","last_unstale":"2024-08-19T14:38:40.071063+0000","last_undegraded":"2024-08-19T14:35:12.592288+0000","last_fullsized":"2024-08-19T14:35:12.526971+0000","mapping_epoch":635,"log_start":"611'3200","ondisk_log_start":"611'3200","created":600,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":120324096,"num_objects":1836,"num_object_clones":0,"num_object_copies":5508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3743,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13218,"num_write_kb":481728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3515,"num_bytes_recovered":227409920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":-1907}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"70'139","reported_seq":731,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-08-19T14:38:39.803188+0000","last_change":"2024-08-19T14:35:11.533207+0000","last_active":"2024-08-19T14:38:39.803188+0000","last_peered":"2024-08-19T14:38:39.803188+0000","last_clean":"2024-08-19T14:38:39.803188+0000","last_became_active":"2024-08-19T14:35:11.532858+0000","last_became_peered":"2024-08-19T14:35:11.532858+0000","last_unstale":"2024-08-19T14:38:39.803188+0000","last_undegraded":"2024-08-19T14:38:39.803188+0000","last_fullsized":"2024-08-19T14:38:39.803188+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:27.331041+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:28:27.331041+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,0],"acting":[11,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.2","version":"694'13109","reported_seq":20249,"reported_epoch":694,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-19T14:38:40.052606+0000","last_change":"2024-08-19T14:35:13.121204+0000","last_active":"2024-08-19T14:38:40.052606+0000","last_peered":"2024-08-19T14:38:40.052606+0000","last_clean":"2024-08-19T14:33:27.312789+0000","last_became_active":"2024-08-19T14:35:13.013347+0000","last_became_peered":"2024-08-19T14:35:13.013347+0000","last_unstale":"2024-08-19T14:38:40.052606+0000","last_undegraded":"2024-08-19T14:35:12.598259+0000","last_fullsized":"2024-08-19T14:35:12.526602+0000","mapping_epoch":635,"log_start":"611'3100","ondisk_log_start":"611'3100","created":600,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":122617856,"num_objects":1871,"num_object_clones":0,"num_object_copies":5613,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3764,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1871,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13109,"num_write_kb":479360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3471,"num_bytes_recovered":224919552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":-1893}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"70'139","reported_seq":608,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-08-19T14:38:39.803127+0000","last_change":"2024-08-19T14:33:16.760928+0000","last_active":"2024-08-19T14:38:39.803127+0000","last_peered":"2024-08-19T14:38:39.803127+0000","last_clean":"2024-08-19T14:38:39.803127+0000","last_became_active":"2024-08-19T14:30:47.909067+0000","last_became_peered":"2024-08-19T14:30:47.909067+0000","last_unstale":"2024-08-19T14:38:39.803127+0000","last_undegraded":"2024-08-19T14:38:39.803127+0000","last_fullsized":"2024-08-19T14:38:39.803127+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:33:16.760810+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:33:16.760810+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.c","version":"694'12800","reported_seq":20039,"reported_epoch":694,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-19T14:38:40.113693+0000","last_change":"2024-08-19T14:35:13.133729+0000","last_active":"2024-08-19T14:38:40.113693+0000","last_peered":"2024-08-19T14:38:40.113693+0000","last_clean":"2024-08-19T14:33:27.312789+0000","last_became_active":"2024-08-19T14:35:13.003266+0000","last_became_peered":"2024-08-19T14:35:13.003266+0000","last_unstale":"2024-08-19T14:38:40.113693+0000","last_undegraded":"2024-08-19T14:35:12.593502+0000","last_fullsized":"2024-08-19T14:35:12.526375+0000","mapping_epoch":635,"log_start":"611'2700","ondisk_log_start":"611'2700","created":600,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":115081216,"num_objects":1756,"num_object_clones":0,"num_object_copies":5268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3490,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12800,"num_write_kb":465792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3519,"num_bytes_recovered":229048320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,2],"acting":[3,11],"avail_no_missing":["3","11"],"object_location_counts":[{"shards":"3,11","objects":-1734}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"70'139","reported_seq":774,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-08-19T14:38:39.794029+0000","last_change":"2024-08-19T14:34:50.256878+0000","last_active":"2024-08-19T14:38:39.794029+0000","last_peered":"2024-08-19T14:38:39.794029+0000","last_clean":"2024-08-19T14:38:39.794029+0000","last_became_active":"2024-08-19T14:34:15.717472+0000","last_became_peered":"2024-08-19T14:34:15.717472+0000","last_unstale":"2024-08-19T14:38:39.794029+0000","last_undegraded":"2024-08-19T14:38:39.794029+0000","last_fullsized":"2024-08-19T14:38:39.794029+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:34:50.256808+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:28.864870+0000","last_clean_scrub_stamp":"2024-08-19T14:34:50.256808+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,0],"acting":[4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"694'16474","reported_seq":26279,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-08-19T14:38:40.557135+0000","last_change":"2024-08-19T14:37:42.029576+0000","last_active":"2024-08-19T14:38:40.557135+0000","last_peered":"2024-08-19T14:38:40.557135+0000","last_clean":"2024-08-19T14:38:40.557135+0000","last_became_active":"2024-08-19T14:36:14.098166+0000","last_became_peered":"2024-08-19T14:36:14.098166+0000","last_unstale":"2024-08-19T14:38:40.557135+0000","last_undegraded":"2024-08-19T14:38:40.557135+0000","last_fullsized":"2024-08-19T14:38:40.557135+0000","mapping_epoch":652,"log_start":"626'6402","ondisk_log_start":"626'6402","created":600,"last_epoch_clean":653,"parent":"0.0","parent_split_bits":0,"last_scrub":"678'14831","last_scrub_stamp":"2024-08-19T14:37:42.029449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:37:42.029449+0000","objects_scrubbed":3383,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:38:44.946893+0000","scrub_duration":1404,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112852992,"num_objects":1722,"num_object_clones":0,"num_object_copies":5166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1722,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12844,"num_write_kb":466112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4700,"num_bytes_recovered":308019200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.9","version":"70'139","reported_seq":1709,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-08-19T14:38:39.795300+0000","last_change":"2024-08-19T14:35:12.529993+0000","last_active":"2024-08-19T14:38:39.795300+0000","last_peered":"2024-08-19T14:38:39.795300+0000","last_clean":"2024-08-19T14:38:39.795300+0000","last_became_active":"2024-08-19T14:35:12.529628+0000","last_became_peered":"2024-08-19T14:35:12.529628+0000","last_unstale":"2024-08-19T14:38:39.795300+0000","last_undegraded":"2024-08-19T14:38:39.795300+0000","last_fullsized":"2024-08-19T14:38:39.795300+0000","mapping_epoch":635,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":636,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:34:51.561893+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:34:51.561893+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.e","version":"694'13176","reported_seq":25874,"reported_epoch":694,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-19T14:38:42.742402+0000","last_change":"2024-08-19T14:35:11.863818+0000","last_active":"2024-08-19T14:38:42.742402+0000","last_peered":"2024-08-19T14:38:42.742402+0000","last_clean":"2024-08-19T14:33:27.312789+0000","last_became_active":"2024-08-19T14:35:11.806304+0000","last_became_peered":"2024-08-19T14:35:11.806304+0000","last_unstale":"2024-08-19T14:38:42.742402+0000","last_undegraded":"2024-08-19T14:35:11.779866+0000","last_fullsized":"2024-08-19T14:35:11.528207+0000","mapping_epoch":634,"log_start":"611'3100","ondisk_log_start":"611'3100","created":600,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106168320,"num_objects":1620,"num_object_clones":0,"num_object_copies":4860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2412,"num_objects_misplaced":4032,"num_objects_unfound":0,"num_objects_dirty":1620,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13176,"num_write_kb":473472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4720,"num_bytes_recovered":266665984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":-2412},{"shards":"1,5,9","objects":1620}],"blocked_by":[],"up_primary":2,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"70'139","reported_seq":740,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-08-19T14:38:39.792120+0000","last_change":"2024-08-19T14:35:11.537896+0000","last_active":"2024-08-19T14:38:39.792120+0000","last_peered":"2024-08-19T14:38:39.792120+0000","last_clean":"2024-08-19T14:38:39.792120+0000","last_became_active":"2024-08-19T14:35:11.537474+0000","last_became_peered":"2024-08-19T14:35:11.537474+0000","last_unstale":"2024-08-19T14:38:39.792120+0000","last_undegraded":"2024-08-19T14:38:39.792120+0000","last_fullsized":"2024-08-19T14:38:39.792120+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:59.829567+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:28:59.829567+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"694'13097","reported_seq":13298,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-08-19T14:38:42.716868+0000","last_change":"2024-08-19T14:33:28.335586+0000","last_active":"2024-08-19T14:38:42.716868+0000","last_peered":"2024-08-19T14:38:42.716868+0000","last_clean":"2024-08-19T14:38:42.716868+0000","last_became_active":"2024-08-19T14:33:28.335298+0000","last_became_peered":"2024-08-19T14:33:28.335298+0000","last_unstale":"2024-08-19T14:38:42.716868+0000","last_undegraded":"2024-08-19T14:38:42.716868+0000","last_fullsized":"2024-08-19T14:38:42.716868+0000","mapping_epoch":600,"log_start":"611'3000","ondisk_log_start":"611'3000","created":600,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106692632,"num_objects":1629,"num_object_clones":0,"num_object_copies":4887,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1629,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13097,"num_write_kb":471169,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"70'139","reported_seq":1060,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-08-19T14:38:39.803049+0000","last_change":"2024-08-19T14:33:15.753668+0000","last_active":"2024-08-19T14:38:39.803049+0000","last_peered":"2024-08-19T14:38:39.803049+0000","last_clean":"2024-08-19T14:38:39.803049+0000","last_became_active":"2024-08-19T14:30:47.941200+0000","last_became_peered":"2024-08-19T14:30:47.941200+0000","last_unstale":"2024-08-19T14:38:39.803049+0000","last_undegraded":"2024-08-19T14:38:39.803049+0000","last_fullsized":"2024-08-19T14:38:39.803049+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:33:15.753595+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:33:15.753595+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.8","version":"694'13285","reported_seq":28603,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-08-19T14:38:40.393379+0000","last_change":"2024-08-19T14:38:13.418089+0000","last_active":"2024-08-19T14:38:40.393379+0000","last_peered":"2024-08-19T14:38:40.393379+0000","last_clean":"2024-08-19T14:38:40.393379+0000","last_became_active":"2024-08-19T14:38:13.389656+0000","last_became_peered":"2024-08-19T14:38:13.389656+0000","last_unstale":"2024-08-19T14:38:40.393379+0000","last_undegraded":"2024-08-19T14:38:40.393379+0000","last_fullsized":"2024-08-19T14:38:40.393379+0000","mapping_epoch":688,"log_start":"611'3200","ondisk_log_start":"611'3200","created":600,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":119603200,"num_objects":1825,"num_object_clones":0,"num_object_copies":5475,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1825,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13285,"num_write_kb":483520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7555,"num_bytes_recovered":476577792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.e","version":"70'139","reported_seq":1251,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-08-19T14:38:39.794052+0000","last_change":"2024-08-19T14:34:55.333177+0000","last_active":"2024-08-19T14:38:39.794052+0000","last_peered":"2024-08-19T14:38:39.794052+0000","last_clean":"2024-08-19T14:38:39.794052+0000","last_became_active":"2024-08-19T14:18:26.884105+0000","last_became_peered":"2024-08-19T14:18:26.884105+0000","last_unstale":"2024-08-19T14:38:39.794052+0000","last_undegraded":"2024-08-19T14:38:39.794052+0000","last_fullsized":"2024-08-19T14:38:39.794052+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:34:55.333065+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:32:31.196288+0000","last_clean_scrub_stamp":"2024-08-19T14:34:55.333065+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"694'13071","reported_seq":13407,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-08-19T14:38:40.380736+0000","last_change":"2024-08-19T14:37:54.857799+0000","last_active":"2024-08-19T14:38:40.380736+0000","last_peered":"2024-08-19T14:38:40.380736+0000","last_clean":"2024-08-19T14:38:40.380736+0000","last_became_active":"2024-08-19T14:33:28.334547+0000","last_became_peered":"2024-08-19T14:33:28.334547+0000","last_unstale":"2024-08-19T14:38:40.380736+0000","last_undegraded":"2024-08-19T14:38:40.380736+0000","last_fullsized":"2024-08-19T14:38:40.380736+0000","mapping_epoch":600,"log_start":"611'3000","ondisk_log_start":"611'3000","created":600,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"683'11737","last_scrub_stamp":"2024-08-19T14:37:54.851356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:37:54.851356+0000","objects_scrubbed":3161,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:39:09.304335+0000","scrub_duration":1254,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":118554624,"num_objects":1809,"num_object_clones":0,"num_object_copies":5427,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1809,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13071,"num_write_kb":476160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,8],"acting":[0,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"70'139","reported_seq":1002,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-08-19T14:38:39.794074+0000","last_change":"2024-08-19T14:34:12.199530+0000","last_active":"2024-08-19T14:38:39.794074+0000","last_peered":"2024-08-19T14:38:39.794074+0000","last_clean":"2024-08-19T14:38:39.794074+0000","last_became_active":"2024-08-19T14:27:31.837754+0000","last_became_peered":"2024-08-19T14:27:31.837754+0000","last_unstale":"2024-08-19T14:38:39.794074+0000","last_undegraded":"2024-08-19T14:38:39.794074+0000","last_fullsized":"2024-08-19T14:38:39.794074+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:34:12.199449+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:34:12.199449+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.a","version":"694'12938","reported_seq":33134,"reported_epoch":694,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-19T14:38:41.724716+0000","last_change":"2024-08-19T14:35:13.046246+0000","last_active":"2024-08-19T14:38:41.724716+0000","last_peered":"2024-08-19T14:38:41.724716+0000","last_clean":"2024-08-19T14:33:27.312789+0000","last_became_active":"2024-08-19T14:35:13.005199+0000","last_became_peered":"2024-08-19T14:35:13.005199+0000","last_unstale":"2024-08-19T14:38:41.724716+0000","last_undegraded":"2024-08-19T14:35:12.692218+0000","last_fullsized":"2024-08-19T14:35:12.524923+0000","mapping_epoch":635,"log_start":"611'2900","ondisk_log_start":"611'2900","created":600,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117178368,"num_objects":1788,"num_object_clones":0,"num_object_copies":5364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1156,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12938,"num_write_kb":471232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9068,"num_bytes_recovered":522125312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,2],"acting":[11,9],"avail_no_missing":["11","3","9"],"object_location_counts":[{"shards":"9,11","objects":632}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.c","version":"70'139","reported_seq":1358,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-08-19T14:38:39.798888+0000","last_change":"2024-08-19T14:34:15.738488+0000","last_active":"2024-08-19T14:38:39.798888+0000","last_peered":"2024-08-19T14:38:39.798888+0000","last_clean":"2024-08-19T14:38:39.798888+0000","last_became_active":"2024-08-19T14:34:15.733282+0000","last_became_peered":"2024-08-19T14:34:15.733282+0000","last_unstale":"2024-08-19T14:38:39.798888+0000","last_undegraded":"2024-08-19T14:38:39.798888+0000","last_fullsized":"2024-08-19T14:38:39.798888+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:33:17.453290+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:22:52.164586+0000","last_clean_scrub_stamp":"2024-08-19T14:33:17.453290+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,9],"acting":[3,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"694'12880","reported_seq":13080,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-08-19T14:38:42.756170+0000","last_change":"2024-08-19T14:33:28.623717+0000","last_active":"2024-08-19T14:38:42.756170+0000","last_peered":"2024-08-19T14:38:42.756170+0000","last_clean":"2024-08-19T14:38:42.756170+0000","last_became_active":"2024-08-19T14:33:28.623318+0000","last_became_peered":"2024-08-19T14:33:28.623318+0000","last_unstale":"2024-08-19T14:38:42.756170+0000","last_undegraded":"2024-08-19T14:38:42.756170+0000","last_fullsized":"2024-08-19T14:38:42.756170+0000","mapping_epoch":600,"log_start":"611'2800","ondisk_log_start":"611'2800","created":600,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":111673344,"num_objects":1704,"num_object_clones":0,"num_object_copies":5112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1704,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12880,"num_write_kb":466688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"70'139","reported_seq":1255,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-08-19T14:38:39.795537+0000","last_change":"2024-08-19T14:34:49.476489+0000","last_active":"2024-08-19T14:38:39.795537+0000","last_peered":"2024-08-19T14:38:39.795537+0000","last_clean":"2024-08-19T14:38:39.795537+0000","last_became_active":"2024-08-19T14:18:26.882002+0000","last_became_peered":"2024-08-19T14:18:26.882002+0000","last_unstale":"2024-08-19T14:38:39.795537+0000","last_undegraded":"2024-08-19T14:38:39.795537+0000","last_fullsized":"2024-08-19T14:38:39.795537+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:34:49.476415+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:24:50.276408+0000","last_clean_scrub_stamp":"2024-08-19T14:34:49.476415+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"70'139","reported_seq":1033,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-08-19T14:38:39.797450+0000","last_change":"2024-08-19T14:33:19.781527+0000","last_active":"2024-08-19T14:38:39.797450+0000","last_peered":"2024-08-19T14:38:39.797450+0000","last_clean":"2024-08-19T14:38:39.797450+0000","last_became_active":"2024-08-19T14:27:31.838968+0000","last_became_peered":"2024-08-19T14:27:31.838968+0000","last_unstale":"2024-08-19T14:38:39.797450+0000","last_undegraded":"2024-08-19T14:38:39.797450+0000","last_fullsized":"2024-08-19T14:38:39.797450+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:33:19.781431+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:28:35.206115+0000","last_clean_scrub_stamp":"2024-08-19T14:33:19.781431+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.13","version":"70'139","reported_seq":1575,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-08-19T14:38:39.797803+0000","last_change":"2024-08-19T14:33:35.786236+0000","last_active":"2024-08-19T14:38:39.797803+0000","last_peered":"2024-08-19T14:38:39.797803+0000","last_clean":"2024-08-19T14:38:39.797803+0000","last_became_active":"2024-08-19T14:30:47.939849+0000","last_became_peered":"2024-08-19T14:30:47.939849+0000","last_unstale":"2024-08-19T14:38:39.797803+0000","last_undegraded":"2024-08-19T14:38:39.797803+0000","last_fullsized":"2024-08-19T14:38:39.797803+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:33:35.786105+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:33:35.786105+0000","last_clean_scrub_stamp":"2024-08-19T14:33:35.786105+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.10","version":"70'139","reported_seq":1203,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-08-19T14:38:39.795446+0000","last_change":"2024-08-19T14:34:58.616722+0000","last_active":"2024-08-19T14:38:39.795446+0000","last_peered":"2024-08-19T14:38:39.795446+0000","last_clean":"2024-08-19T14:38:39.795446+0000","last_became_active":"2024-08-19T14:27:31.839611+0000","last_became_peered":"2024-08-19T14:27:31.839611+0000","last_unstale":"2024-08-19T14:38:39.795446+0000","last_undegraded":"2024-08-19T14:38:39.795446+0000","last_fullsized":"2024-08-19T14:38:39.795446+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:34:58.616646+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:34:58.616646+0000","last_clean_scrub_stamp":"2024-08-19T14:34:58.616646+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.11","version":"70'139","reported_seq":1259,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-08-19T14:38:39.805999+0000","last_change":"2024-08-19T14:38:34.904558+0000","last_active":"2024-08-19T14:38:39.805999+0000","last_peered":"2024-08-19T14:38:39.805999+0000","last_clean":"2024-08-19T14:38:39.805999+0000","last_became_active":"2024-08-19T14:34:15.731049+0000","last_became_peered":"2024-08-19T14:34:15.731049+0000","last_unstale":"2024-08-19T14:38:39.805999+0000","last_undegraded":"2024-08-19T14:38:39.805999+0000","last_fullsized":"2024-08-19T14:38:39.805999+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:38:34.904308+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:24:50.547428+0000","last_clean_scrub_stamp":"2024-08-19T14:38:34.904308+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:40:02.722075+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"70'139","reported_seq":1186,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-08-19T14:38:39.799921+0000","last_change":"2024-08-19T14:33:27.737496+0000","last_active":"2024-08-19T14:38:39.799921+0000","last_peered":"2024-08-19T14:38:39.799921+0000","last_clean":"2024-08-19T14:38:39.799921+0000","last_became_active":"2024-08-19T14:20:28.204724+0000","last_became_peered":"2024-08-19T14:20:28.204724+0000","last_unstale":"2024-08-19T14:38:39.799921+0000","last_undegraded":"2024-08-19T14:38:39.799921+0000","last_fullsized":"2024-08-19T14:38:39.799921+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:33:27.737367+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:28:34.216370+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.737367+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"70'139","reported_seq":1329,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-08-19T14:38:39.794052+0000","last_change":"2024-08-19T14:37:38.355694+0000","last_active":"2024-08-19T14:38:39.794052+0000","last_peered":"2024-08-19T14:38:39.794052+0000","last_clean":"2024-08-19T14:38:39.794052+0000","last_became_active":"2024-08-19T14:18:31.901556+0000","last_became_peered":"2024-08-19T14:18:31.901556+0000","last_unstale":"2024-08-19T14:38:39.794052+0000","last_undegraded":"2024-08-19T14:38:39.794052+0000","last_fullsized":"2024-08-19T14:38:39.794052+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:37:38.355621+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:37:38.355621+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":56,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,0],"acting":[8,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.14","version":"70'139","reported_seq":1358,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-08-19T14:38:39.795380+0000","last_change":"2024-08-19T14:35:12.530145+0000","last_active":"2024-08-19T14:38:39.795380+0000","last_peered":"2024-08-19T14:38:39.795380+0000","last_clean":"2024-08-19T14:38:39.795380+0000","last_became_active":"2024-08-19T14:35:12.529792+0000","last_became_peered":"2024-08-19T14:35:12.529792+0000","last_unstale":"2024-08-19T14:38:39.795380+0000","last_undegraded":"2024-08-19T14:38:39.795380+0000","last_fullsized":"2024-08-19T14:38:39.795380+0000","mapping_epoch":635,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":636,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:34:53.617013+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:34:53.617013+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,9],"acting":[7,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.15","version":"70'139","reported_seq":1210,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-08-19T14:38:39.795503+0000","last_change":"2024-08-19T14:34:57.640316+0000","last_active":"2024-08-19T14:38:39.795503+0000","last_peered":"2024-08-19T14:38:39.795503+0000","last_clean":"2024-08-19T14:38:39.795503+0000","last_became_active":"2024-08-19T14:18:31.912496+0000","last_became_peered":"2024-08-19T14:18:31.912496+0000","last_unstale":"2024-08-19T14:38:39.795503+0000","last_undegraded":"2024-08-19T14:38:39.795503+0000","last_fullsized":"2024-08-19T14:38:39.795503+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:34:57.640213+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:20:09.180923+0000","last_clean_scrub_stamp":"2024-08-19T14:34:57.640213+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"70'139","reported_seq":1196,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-08-19T14:38:39.793943+0000","last_change":"2024-08-19T14:34:06.176510+0000","last_active":"2024-08-19T14:38:39.793943+0000","last_peered":"2024-08-19T14:38:39.793943+0000","last_clean":"2024-08-19T14:38:39.793943+0000","last_became_active":"2024-08-19T14:18:31.901229+0000","last_became_peered":"2024-08-19T14:18:31.901229+0000","last_unstale":"2024-08-19T14:38:39.793943+0000","last_undegraded":"2024-08-19T14:38:39.793943+0000","last_fullsized":"2024-08-19T14:38:39.793943+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:34:06.176403+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:34:06.176403+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1b","version":"70'139","reported_seq":1577,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-08-19T14:38:39.793904+0000","last_change":"2024-08-19T14:37:59.510774+0000","last_active":"2024-08-19T14:38:39.793904+0000","last_peered":"2024-08-19T14:38:39.793904+0000","last_clean":"2024-08-19T14:38:39.793904+0000","last_became_active":"2024-08-19T14:34:15.718427+0000","last_became_peered":"2024-08-19T14:34:15.718427+0000","last_unstale":"2024-08-19T14:38:39.793904+0000","last_undegraded":"2024-08-19T14:38:39.793904+0000","last_fullsized":"2024-08-19T14:38:39.793904+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:37:59.510697+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:37:59.510697+0000","last_clean_scrub_stamp":"2024-08-19T14:37:59.510697+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:39:04.057695+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.18","version":"70'139","reported_seq":1138,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-08-19T14:38:39.795280+0000","last_change":"2024-08-19T14:35:11.528963+0000","last_active":"2024-08-19T14:38:39.795280+0000","last_peered":"2024-08-19T14:38:39.795280+0000","last_clean":"2024-08-19T14:38:39.795280+0000","last_became_active":"2024-08-19T14:35:11.528530+0000","last_became_peered":"2024-08-19T14:35:11.528530+0000","last_unstale":"2024-08-19T14:38:39.795280+0000","last_undegraded":"2024-08-19T14:38:39.795280+0000","last_fullsized":"2024-08-19T14:38:39.795280+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:29:05.336156+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:27:33.084161+0000","last_clean_scrub_stamp":"2024-08-19T14:29:05.336156+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.19","version":"70'139","reported_seq":1556,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-08-19T14:38:39.805892+0000","last_change":"2024-08-19T14:37:41.671836+0000","last_active":"2024-08-19T14:38:39.805892+0000","last_peered":"2024-08-19T14:38:39.805892+0000","last_clean":"2024-08-19T14:38:39.805892+0000","last_became_active":"2024-08-19T14:18:31.902547+0000","last_became_peered":"2024-08-19T14:18:31.902547+0000","last_unstale":"2024-08-19T14:38:39.805892+0000","last_undegraded":"2024-08-19T14:38:39.805892+0000","last_fullsized":"2024-08-19T14:38:39.805892+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:37:41.671731+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:37:41.671731+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:38:59.596992+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1e","version":"70'139","reported_seq":580,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-08-19T14:38:39.803046+0000","last_change":"2024-08-19T14:33:17.778415+0000","last_active":"2024-08-19T14:38:39.803046+0000","last_peered":"2024-08-19T14:38:39.803046+0000","last_clean":"2024-08-19T14:38:39.803046+0000","last_became_active":"2024-08-19T14:30:47.909974+0000","last_became_peered":"2024-08-19T14:30:47.909974+0000","last_unstale":"2024-08-19T14:38:39.803046+0000","last_undegraded":"2024-08-19T14:38:39.803046+0000","last_fullsized":"2024-08-19T14:38:39.803046+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:33:17.778295+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:27:04.780939+0000","last_clean_scrub_stamp":"2024-08-19T14:33:17.778295+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1c","version":"70'139","reported_seq":1331,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-08-19T14:38:39.793860+0000","last_change":"2024-08-19T14:38:01.463630+0000","last_active":"2024-08-19T14:38:39.793860+0000","last_peered":"2024-08-19T14:38:39.793860+0000","last_clean":"2024-08-19T14:38:39.793860+0000","last_became_active":"2024-08-19T14:30:47.908866+0000","last_became_peered":"2024-08-19T14:30:47.908866+0000","last_unstale":"2024-08-19T14:38:39.793860+0000","last_undegraded":"2024-08-19T14:38:39.793860+0000","last_fullsized":"2024-08-19T14:38:39.793860+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:38:01.463524+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:38:01.463524+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:39:18.496929+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,11],"acting":[8,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1d","version":"70'139","reported_seq":1240,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-08-19T14:38:39.797350+0000","last_change":"2024-08-19T14:35:11.542467+0000","last_active":"2024-08-19T14:38:39.797350+0000","last_peered":"2024-08-19T14:38:39.797350+0000","last_clean":"2024-08-19T14:38:39.797350+0000","last_became_active":"2024-08-19T14:35:11.540488+0000","last_became_peered":"2024-08-19T14:35:11.540488+0000","last_unstale":"2024-08-19T14:38:39.797350+0000","last_undegraded":"2024-08-19T14:38:39.797350+0000","last_fullsized":"2024-08-19T14:38:39.797350+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:29.194639+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:28:29.194639+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":1855651864,"num_objects":28316,"num_object_clones":0,"num_object_copies":84948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14565,"num_objects_misplaced":4032,"num_objects_unfound":0,"num_objects_dirty":28316,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":208000,"num_write_kb":7562049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45793,"num_bytes_recovered":2860646400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5644300288,"data_stored":5644288072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160928,"ondisk_log_size":160928,"up":48,"acting":43,"num_store_stats":10},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":856608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187784,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1454112,"kb_used_data":580240,"kb_used_omap":1,"kb_used_meta":873854,"kb_avail":92291040,"statfs":{"total":95995035648,"available":94506024960,"internally_reserved":0,"allocated":594165760,"data_stored":591098637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":894826948},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":2314987372651,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":74,"seq":317827580242,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1378456,"kb_used_data":553168,"kb_used_omap":1,"kb_used_meta":825278,"kb_avail":92366696,"statfs":{"total":95995035648,"available":94583496704,"internally_reserved":0,"allocated":566444032,"data_stored":563376909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":845085121},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874584,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1137168,"kb_used_data":357700,"kb_used_omap":1,"kb_used_meta":779454,"kb_avail":92607984,"statfs":{"total":95995035648,"available":94830575616,"internally_reserved":0,"allocated":366284800,"data_stored":363221704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":798161349},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":57,"seq":244813136219,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1307348,"kb_used_data":365452,"kb_used_omap":1,"kb_used_meta":941886,"kb_avail":92437804,"statfs":{"total":95995035648,"available":94656311296,"internally_reserved":0,"allocated":374222848,"data_stored":371156200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":964491722},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":206158430558,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692194,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1272928,"kb_used_data":339288,"kb_used_omap":1,"kb_used_meta":933566,"kb_avail":92472224,"statfs":{"total":95995035648,"available":94691557376,"internally_reserved":0,"allocated":347430912,"data_stored":344360237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":955972037},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986535,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1414232,"kb_used_data":444560,"kb_used_omap":1,"kb_used_meta":969662,"kb_avail":92330920,"statfs":{"total":95995035648,"available":94546862080,"internally_reserved":0,"allocated":455229440,"data_stored":452162317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":992934341},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150061,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2409748,"kb_used_data":1395656,"kb_used_omap":1,"kb_used_meta":1014078,"kb_avail":91335404,"statfs":{"total":95995035648,"available":93527453696,"internally_reserved":0,"allocated":1429151744,"data_stored":1426084576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1038416327},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":634,"seq":2723009265707,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":508888,"kb_used_data":247756,"kb_used_omap":4,"kb_used_meta":261115,"kb_avail":93236264,"statfs":{"total":95995035648,"available":95473934336,"internally_reserved":0,"allocated":253702144,"data_stored":250635496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4407,"internal_metadata":267382473},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575222,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1784668,"kb_used_data":666452,"kb_used_omap":1,"kb_used_meta":1118206,"kb_avail":91960484,"statfs":{"total":95995035648,"available":94167535616,"internally_reserved":0,"allocated":682446848,"data_stored":679375653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1145043397},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738746,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1582564,"kb_used_data":703124,"kb_used_omap":1,"kb_used_meta":879422,"kb_avail":92162588,"statfs":{"total":95995035648,"available":94374490112,"internally_reserved":0,"allocated":719998976,"data_stored":716927781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":900528581},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":701566976,"data_stored":701562904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":662114304,"data_stored":662110232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":242024448,"data_stored":242024448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1410535424,"data_stored":1410531352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":446562304,"data_stored":446562304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":335282176,"data_stored":335282176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":360841216,"data_stored":360841216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":351207424,"data_stored":351207424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":556072960,"data_stored":556072960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":578093056,"data_stored":578093056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T14:38:45.509 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:38:45.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:45.531 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:45.531 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:38:45.633 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:38:45.657 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:45.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:45.659 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:38:45.759 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:38:45.784 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:45.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:45.785 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:38:45.886 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:38:45.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:45.928 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:45.928 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:38:46.029 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:38:46.051 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:46.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:46.053 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:38:46.109 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:45 smithi031 ceph-mon[16336]: pgmap v1659: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.7 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 402 op/s; 14565/84954 objects degraded (17.145%); 4032/84954 objects misplaced (4.746%); 2.3 MiB/s, 119 objects/s recovering 2024-08-19T14:38:46.109 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:45 smithi031 ceph-mon[16336]: from='client.17277 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:38:46.153 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:38:46.205 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:46.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:46.206 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:38:46.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:45 smithi032 ceph-mon[20004]: pgmap v1659: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.7 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 402 op/s; 14565/84954 objects degraded (17.145%); 4032/84954 objects misplaced (4.746%); 2.3 MiB/s, 119 objects/s recovering 2024-08-19T14:38:46.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:45 smithi032 ceph-mon[20004]: from='client.17277 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:38:46.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:45 smithi045 ceph-mon[20143]: pgmap v1659: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.7 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 402 op/s; 14565/84954 objects degraded (17.145%); 4032/84954 objects misplaced (4.746%); 2.3 MiB/s, 119 objects/s recovering 2024-08-19T14:38:46.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:45 smithi045 ceph-mon[20143]: from='client.17277 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:38:46.306 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:38:46.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:46.352 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:46.353 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:38:46.397 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:38:46.397 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-08-19T14:38:46.397 INFO:teuthology.orchestra.run.smithi031.stdout: "complaint_time": 30, 2024-08-19T14:38:46.397 INFO:teuthology.orchestra.run.smithi031.stdout: "num_blocked_ops": 0 2024-08-19T14:38:46.397 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:38:46.397 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:38:46.454 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:38:46.512 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:46.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:46.513 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:38:46.614 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:38:46.636 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:46.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:46.637 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:38:46.738 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:38:46.760 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:46.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:46.761 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:38:46.862 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:38:46.884 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:46.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:46.885 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:38:46.986 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:38:47.026 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:47.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:47.027 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:38:47.038 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:46 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:38:47.039 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:46 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:38:47.039 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:46 smithi031 ceph-mon[16336]: osdmap e695: 12 total, 12 up, 10 in 2024-08-19T14:38:47.039 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:46 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:38:47.039 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:46 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:38:47.092 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-08-19T14:38:47.129 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:38:47.197 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:47.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:47.198 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:38:47.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:46 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:38:47.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:46 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:38:47.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:46 smithi032 ceph-mon[20004]: osdmap e695: 12 total, 12 up, 10 in 2024-08-19T14:38:47.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:46 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:38:47.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:46 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:38:47.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:46 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:38:47.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:46 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:38:47.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:46 smithi045 ceph-mon[20143]: osdmap e695: 12 total, 12 up, 10 in 2024-08-19T14:38:47.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:46 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:38:47.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:46 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:38:47.299 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:38:47.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:47.361 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:47.362 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:38:47.463 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:38:47.486 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:47.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:47.488 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:38:47.588 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:38:47.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:47.653 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:47.653 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:38:47.753 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:38:47.754 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:38:47.821 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:47.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:47.822 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:38:47.920 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:47 smithi031 ceph-mon[16336]: pgmap v1661: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.6 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 508 op/s; 14257/80487 objects degraded (17.713%); 4032/80487 objects misplaced (5.010%); 5.0 MiB/s, 127 objects/s recovering 2024-08-19T14:38:47.920 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:47 smithi031 ceph-mon[16336]: 1.17 scrub starts 2024-08-19T14:38:47.920 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:47 smithi031 ceph-mon[16336]: 1.17 scrub ok 2024-08-19T14:38:47.920 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:47 smithi031 ceph-mon[16336]: 7.f scrub starts 2024-08-19T14:38:47.920 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:47 smithi031 ceph-mon[16336]: osdmap e696: 12 total, 12 up, 10 in 2024-08-19T14:38:47.922 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:38:47.946 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:47.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:47.947 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:38:48.048 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:38:48.103 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:48.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:48.105 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:38:48.163 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:38:48.206 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:38:48.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:47 smithi032 ceph-mon[20004]: pgmap v1661: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.6 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 508 op/s; 14257/80487 objects degraded (17.713%); 4032/80487 objects misplaced (5.010%); 5.0 MiB/s, 127 objects/s recovering 2024-08-19T14:38:48.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:47 smithi032 ceph-mon[20004]: 1.17 scrub starts 2024-08-19T14:38:48.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:47 smithi032 ceph-mon[20004]: 1.17 scrub ok 2024-08-19T14:38:48.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:47 smithi032 ceph-mon[20004]: 7.f scrub starts 2024-08-19T14:38:48.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:47 smithi032 ceph-mon[20004]: osdmap e696: 12 total, 12 up, 10 in 2024-08-19T14:38:48.283 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:48.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:48.284 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:38:48.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:47 smithi045 ceph-mon[20143]: pgmap v1661: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.6 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 508 op/s; 14257/80487 objects degraded (17.713%); 4032/80487 objects misplaced (5.010%); 5.0 MiB/s, 127 objects/s recovering 2024-08-19T14:38:48.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:47 smithi045 ceph-mon[20143]: 1.17 scrub starts 2024-08-19T14:38:48.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:47 smithi045 ceph-mon[20143]: 1.17 scrub ok 2024-08-19T14:38:48.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:47 smithi045 ceph-mon[20143]: 7.f scrub starts 2024-08-19T14:38:48.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:47 smithi045 ceph-mon[20143]: osdmap e696: 12 total, 12 up, 10 in 2024-08-19T14:38:48.385 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:38:48.411 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:38:48.412 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 20, 2024-08-19T14:38:48.412 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 600, 2024-08-19T14:38:48.412 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [ 2024-08-19T14:38:48.412 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.412 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "MOSDPGPush(7.2 637/635 [PushOp(7:446a5b2b:::benchmark_data_smithi045_115842_object4789:head, version: 605'282, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:446a5b2b:::benchmark_data_smithi045_115842_object4789:head@605'282, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-19T14:38:48.412 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:35:13.931437+0000", 2024-08-19T14:38:48.412 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 214.47777916499999, 2024-08-19T14:38:48.412 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.123186772, 2024-08-19T14:38:48.412 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:38:48.412 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:38:48.412 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-08-19T14:38:48.413 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:38:48.413 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.413 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:38:48.413 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:13.931437+0000", 2024-08-19T14:38:48.413 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:38:48.413 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.413 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.413 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:38:48.413 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:13.931437+0000", 2024-08-19T14:38:48.413 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:38:48.413 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.413 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.413 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:38:48.414 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:13.931438+0000", 2024-08-19T14:38:48.414 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.2699999999999999e-07 2024-08-19T14:38:48.414 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.414 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.414 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:38:48.414 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:13.931468+0000", 2024-08-19T14:38:48.414 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.0803000000000003e-05 2024-08-19T14:38:48.414 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.414 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.414 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:38:48.414 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:13.931488+0000", 2024-08-19T14:38:48.414 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.9324999999999999e-05 2024-08-19T14:38:48.415 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.415 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.415 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:38:48.415 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:13.931503+0000", 2024-08-19T14:38:48.415 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4885000000000001e-05 2024-08-19T14:38:48.415 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.415 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.415 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:38:48.415 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:14.044073+0000", 2024-08-19T14:38:48.415 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.11257072999999999 2024-08-19T14:38:48.415 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.415 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.415 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:38:48.416 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:14.044086+0000", 2024-08-19T14:38:48.416 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2699e-05 2024-08-19T14:38:48.416 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.416 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.416 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:38:48.416 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:14.054624+0000", 2024-08-19T14:38:48.416 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.010537703000000001 2024-08-19T14:38:48.416 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:38:48.416 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:38:48.416 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:38:48.416 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.416 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.416 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "MOSDPGPush(7.a 648/635 [PushOp(7:506419e4:::benchmark_data_smithi045_115842_object61743:head, version: 613'3857, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:506419e4:::benchmark_data_smithi045_115842_object61743:head@613'3857, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-19T14:38:48.417 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:35:59.394012+0000", 2024-08-19T14:38:48.417 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 169.015203912, 2024-08-19T14:38:48.417 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.12366187200000001, 2024-08-19T14:38:48.417 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:38:48.417 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:38:48.417 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-08-19T14:38:48.417 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:38:48.417 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.417 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:38:48.417 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:59.394012+0000", 2024-08-19T14:38:48.417 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:38:48.417 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.417 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.417 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:38:48.418 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:59.394012+0000", 2024-08-19T14:38:48.418 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:38:48.418 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.418 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.418 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:38:48.418 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:59.394013+0000", 2024-08-19T14:38:48.418 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2580000000000001e-06 2024-08-19T14:38:48.418 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.418 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.418 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:38:48.418 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:59.394064+0000", 2024-08-19T14:38:48.418 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0339e-05 2024-08-19T14:38:48.419 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.419 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.419 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:38:48.419 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:59.394093+0000", 2024-08-19T14:38:48.420 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.9391999999999999e-05 2024-08-19T14:38:48.420 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.420 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.420 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:38:48.420 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:59.394113+0000", 2024-08-19T14:38:48.420 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.0186999999999998e-05 2024-08-19T14:38:48.420 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.420 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.420 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:38:48.421 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:59.514967+0000", 2024-08-19T14:38:48.421 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.12085404399999999 2024-08-19T14:38:48.421 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.421 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.421 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:38:48.421 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:59.514980+0000", 2024-08-19T14:38:48.421 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2432000000000001e-05 2024-08-19T14:38:48.421 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.421 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.421 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:38:48.421 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:59.517674+0000", 2024-08-19T14:38:48.421 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0026942200000000002 2024-08-19T14:38:48.421 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:38:48.421 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:38:48.422 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:38:48.422 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.422 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.422 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "MOSDPGPush(7.a 650/635 [PushOp(7:5c8dad9c:::benchmark_data_smithi045_115842_object63220:head, version: 613'3943, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:5c8dad9c:::benchmark_data_smithi045_115842_object63220:head@613'3943, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-19T14:38:48.422 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:36:03.715886+0000", 2024-08-19T14:38:48.422 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 164.69332992599999, 2024-08-19T14:38:48.422 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.15676975000000001, 2024-08-19T14:38:48.422 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:38:48.422 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:38:48.422 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-08-19T14:38:48.422 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:38:48.422 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.423 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:38:48.423 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:03.715886+0000", 2024-08-19T14:38:48.423 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:38:48.423 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.423 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.423 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:38:48.423 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:03.715886+0000", 2024-08-19T14:38:48.423 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:38:48.423 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.423 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.423 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:38:48.423 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:03.715887+0000", 2024-08-19T14:38:48.424 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.083e-06 2024-08-19T14:38:48.424 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.424 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.424 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:38:48.424 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:03.715946+0000", 2024-08-19T14:38:48.424 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.8717e-05 2024-08-19T14:38:48.424 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.424 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.424 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:38:48.424 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:03.715972+0000", 2024-08-19T14:38:48.424 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.6299999999999999e-05 2024-08-19T14:38:48.424 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.425 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.425 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:38:48.425 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:03.715992+0000", 2024-08-19T14:38:48.425 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.9729999999999999e-05 2024-08-19T14:38:48.425 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.426 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.426 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:38:48.426 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:03.869724+0000", 2024-08-19T14:38:48.426 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.153731864 2024-08-19T14:38:48.426 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.426 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.426 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:38:48.426 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:03.869766+0000", 2024-08-19T14:38:48.426 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.1819000000000001e-05 2024-08-19T14:38:48.426 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.426 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.426 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:38:48.426 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:03.872656+0000", 2024-08-19T14:38:48.427 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0028902369999999999 2024-08-19T14:38:48.427 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:38:48.427 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:38:48.427 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:38:48.427 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.427 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.427 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "MOSDPGPush(7.2 650/635 [PushOp(7:4789c4c6:::benchmark_data_smithi045_115842_object23753:head, version: 608'1512, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:4789c4c6:::benchmark_data_smithi045_115842_object23753:head@608'1512, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-19T14:38:48.427 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:36:03.725918+0000", 2024-08-19T14:38:48.427 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 164.68329799099999, 2024-08-19T14:38:48.427 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.16257834299999999, 2024-08-19T14:38:48.427 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:38:48.427 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:38:48.427 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-08-19T14:38:48.428 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:38:48.428 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.428 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:38:48.428 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:03.725918+0000", 2024-08-19T14:38:48.428 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:38:48.428 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.428 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.428 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:38:48.428 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:03.725918+0000", 2024-08-19T14:38:48.428 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:38:48.428 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.428 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.428 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:38:48.429 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:03.725919+0000", 2024-08-19T14:38:48.429 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.7499999999999999e-07 2024-08-19T14:38:48.429 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.429 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.429 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:38:48.429 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:03.726078+0000", 2024-08-19T14:38:48.429 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00015877400000000001 2024-08-19T14:38:48.429 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.429 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.429 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:38:48.429 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:03.726102+0000", 2024-08-19T14:38:48.429 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.3808e-05 2024-08-19T14:38:48.429 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.429 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.430 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:38:48.430 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:03.726119+0000", 2024-08-19T14:38:48.430 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.7694e-05 2024-08-19T14:38:48.430 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.430 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.430 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:38:48.430 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:03.879620+0000", 2024-08-19T14:38:48.430 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.153501208 2024-08-19T14:38:48.430 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.430 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.430 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:38:48.430 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:03.879634+0000", 2024-08-19T14:38:48.431 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.31e-05 2024-08-19T14:38:48.431 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.431 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.431 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:38:48.431 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:03.888496+0000", 2024-08-19T14:38:48.431 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0088629840000000008 2024-08-19T14:38:48.431 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:38:48.431 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:38:48.431 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:38:48.431 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.431 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.431 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "MOSDPGPush(7.a 660/635 [PushOp(7:5360cb8c:::benchmark_data_smithi045_115842_object37458:head, version: 610'2305, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:5360cb8c:::benchmark_data_smithi045_115842_object37458:head@610'2305, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-19T14:38:48.432 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:36:37.372724+0000", 2024-08-19T14:38:48.432 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 131.03649259599999, 2024-08-19T14:38:48.432 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.13786571, 2024-08-19T14:38:48.432 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:38:48.432 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:38:48.432 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-08-19T14:38:48.432 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:38:48.432 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.432 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:38:48.432 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:37.372724+0000", 2024-08-19T14:38:48.432 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:38:48.432 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.432 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.433 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:38:48.433 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:37.372724+0000", 2024-08-19T14:38:48.433 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:38:48.433 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.433 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.433 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:38:48.433 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:37.372725+0000", 2024-08-19T14:38:48.433 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0109999999999999e-06 2024-08-19T14:38:48.433 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.433 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.433 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:38:48.433 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:37.372776+0000", 2024-08-19T14:38:48.433 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1801000000000001e-05 2024-08-19T14:38:48.434 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.434 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.434 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:38:48.434 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:37.464620+0000", 2024-08-19T14:38:48.434 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.091843465999999999 2024-08-19T14:38:48.434 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.434 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.434 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:38:48.434 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:37.464647+0000", 2024-08-19T14:38:48.434 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.6849e-05 2024-08-19T14:38:48.434 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.434 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.434 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:38:48.435 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:37.506867+0000", 2024-08-19T14:38:48.435 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.042220013000000001 2024-08-19T14:38:48.435 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.435 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.435 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:38:48.435 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:37.506879+0000", 2024-08-19T14:38:48.435 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2320000000000001e-05 2024-08-19T14:38:48.435 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.435 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.435 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:38:48.435 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:37.510589+0000", 2024-08-19T14:38:48.435 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00371025 2024-08-19T14:38:48.436 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:38:48.436 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:38:48.436 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:38:48.436 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.436 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.436 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.16962.0:181780 7.e e678/634 7:7b43e48b:::benchmark_data_smithi045_115842_object63620:head v 678'11470, mlcod=613'4070)", 2024-08-19T14:38:48.436 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:37:43.454572+0000", 2024-08-19T14:38:48.436 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 64.954644259000005, 2024-08-19T14:38:48.436 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.161739627, 2024-08-19T14:38:48.436 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:38:48.436 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:38:48.436 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:38:48.436 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:38:48.436 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.437 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:38:48.437 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.454572+0000", 2024-08-19T14:38:48.437 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:38:48.437 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.437 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.437 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:38:48.437 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.454572+0000", 2024-08-19T14:38:48.437 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:38:48.437 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.437 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.437 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:38:48.438 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.454572+0000", 2024-08-19T14:38:48.438 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.48e-07 2024-08-19T14:38:48.438 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.438 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.438 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:38:48.438 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.454576+0000", 2024-08-19T14:38:48.438 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2910000000000001e-06 2024-08-19T14:38:48.438 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.438 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.438 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:38:48.439 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.454583+0000", 2024-08-19T14:38:48.439 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.3890000000000002e-06 2024-08-19T14:38:48.439 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.439 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.439 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:38:48.439 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.454588+0000", 2024-08-19T14:38:48.439 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0860000000000001e-06 2024-08-19T14:38:48.439 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.439 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.439 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:38:48.439 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.473215+0000", 2024-08-19T14:38:48.440 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.018626888000000001 2024-08-19T14:38:48.440 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.440 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.440 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:38:48.440 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.473241+0000", 2024-08-19T14:38:48.440 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.6454000000000001e-05 2024-08-19T14:38:48.440 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.440 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.440 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:38:48.440 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.616231+0000", 2024-08-19T14:38:48.440 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.14299003299999999 2024-08-19T14:38:48.440 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.441 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.441 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:38:48.441 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.616312+0000", 2024-08-19T14:38:48.441 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.0238e-05 2024-08-19T14:38:48.441 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:38:48.441 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:38:48.441 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:38:48.441 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.441 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.441 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.16962.0:181792 7.e e678/634 7:76fea455:::benchmark_data_smithi045_115842_object63632:head v 678'11471, mlcod=613'4070)", 2024-08-19T14:38:48.441 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:37:43.470051+0000", 2024-08-19T14:38:48.441 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 64.939164986999998, 2024-08-19T14:38:48.442 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.14628962500000001, 2024-08-19T14:38:48.442 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:38:48.442 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:38:48.442 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:38:48.442 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:38:48.442 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.442 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:38:48.442 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.470051+0000", 2024-08-19T14:38:48.442 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:38:48.442 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.442 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.442 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:38:48.442 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.470051+0000", 2024-08-19T14:38:48.443 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:38:48.443 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.443 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.443 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:38:48.443 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.470052+0000", 2024-08-19T14:38:48.443 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.032e-06 2024-08-19T14:38:48.443 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.443 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.443 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:38:48.443 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.470061+0000", 2024-08-19T14:38:48.443 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.0270000000000001e-06 2024-08-19T14:38:48.443 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.443 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.444 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:38:48.444 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.470074+0000", 2024-08-19T14:38:48.444 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2651e-05 2024-08-19T14:38:48.444 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.444 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.444 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:38:48.444 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.470082+0000", 2024-08-19T14:38:48.444 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.0469999999999994e-06 2024-08-19T14:38:48.444 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.444 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.444 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:38:48.444 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.487157+0000", 2024-08-19T14:38:48.444 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.017074807000000001 2024-08-19T14:38:48.445 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.445 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.445 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:38:48.445 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.487182+0000", 2024-08-19T14:38:48.445 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.5338e-05 2024-08-19T14:38:48.445 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.445 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.445 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:38:48.445 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.616318+0000", 2024-08-19T14:38:48.445 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.129135683 2024-08-19T14:38:48.445 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.445 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.445 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:38:48.446 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.616341+0000", 2024-08-19T14:38:48.446 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.304e-05 2024-08-19T14:38:48.446 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:38:48.446 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:38:48.446 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:38:48.446 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.446 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.446 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.16962.0:181793 7.e e678/634 7:73e8a3a5:::benchmark_data_smithi045_115842_object63633:head v 678'11472, mlcod=613'4070)", 2024-08-19T14:38:48.446 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:37:43.470096+0000", 2024-08-19T14:38:48.446 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 64.939119883999993, 2024-08-19T14:38:48.446 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.14626562000000001, 2024-08-19T14:38:48.446 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:38:48.447 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:38:48.447 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:38:48.447 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:38:48.447 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.447 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:38:48.447 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.470096+0000", 2024-08-19T14:38:48.447 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:38:48.447 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.447 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.447 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:38:48.447 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.470096+0000", 2024-08-19T14:38:48.447 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:38:48.447 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.447 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.447 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:38:48.448 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.470096+0000", 2024-08-19T14:38:48.448 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.3699999999999999e-07 2024-08-19T14:38:48.448 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.448 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.448 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:38:48.448 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.470100+0000", 2024-08-19T14:38:48.448 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.9829999999999998e-06 2024-08-19T14:38:48.448 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.448 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.448 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:38:48.448 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.470106+0000", 2024-08-19T14:38:48.448 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.8490000000000002e-06 2024-08-19T14:38:48.448 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.448 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.449 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:38:48.449 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.470111+0000", 2024-08-19T14:38:48.449 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.5480000000000003e-06 2024-08-19T14:38:48.449 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.449 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.449 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:38:48.449 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.501919+0000", 2024-08-19T14:38:48.449 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.031808014000000003 2024-08-19T14:38:48.449 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.449 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.449 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:38:48.450 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.501945+0000", 2024-08-19T14:38:48.450 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.6227000000000001e-05 2024-08-19T14:38:48.450 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.450 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.450 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:38:48.450 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.616343+0000", 2024-08-19T14:38:48.450 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.114397659 2024-08-19T14:38:48.450 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.450 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.450 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:38:48.450 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.616362+0000", 2024-08-19T14:38:48.451 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.9103e-05 2024-08-19T14:38:48.451 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:38:48.451 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:38:48.451 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:38:48.451 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.451 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.451 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.16962.0:181796 7.e e678/634 7:7a3875c2:::benchmark_data_smithi045_115842_object63636:head v 678'11473, mlcod=613'4070)", 2024-08-19T14:38:48.451 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:37:43.479955+0000", 2024-08-19T14:38:48.451 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 64.929260868, 2024-08-19T14:38:48.451 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.136429833, 2024-08-19T14:38:48.451 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:38:48.451 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:38:48.451 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:38:48.451 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:38:48.452 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.452 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:38:48.452 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.479955+0000", 2024-08-19T14:38:48.452 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:38:48.452 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.452 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.452 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:38:48.452 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.479955+0000", 2024-08-19T14:38:48.452 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:38:48.452 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.452 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.452 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:38:48.452 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.479956+0000", 2024-08-19T14:38:48.452 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0410000000000001e-06 2024-08-19T14:38:48.452 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.453 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.453 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:38:48.453 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.479965+0000", 2024-08-19T14:38:48.453 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.8610000000000004e-06 2024-08-19T14:38:48.453 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.453 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.453 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:38:48.453 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.479981+0000", 2024-08-19T14:38:48.453 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5909999999999998e-05 2024-08-19T14:38:48.453 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.453 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.453 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:38:48.453 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.479991+0000", 2024-08-19T14:38:48.453 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.4930000000000004e-06 2024-08-19T14:38:48.454 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.454 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.454 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:38:48.454 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.517448+0000", 2024-08-19T14:38:48.454 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.037457783000000001 2024-08-19T14:38:48.454 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.454 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.454 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:38:48.454 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.517472+0000", 2024-08-19T14:38:48.454 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.3768e-05 2024-08-19T14:38:48.454 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.454 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.454 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:38:48.454 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.616364+0000", 2024-08-19T14:38:48.455 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.098891862999999997 2024-08-19T14:38:48.455 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.455 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.455 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:38:48.455 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.616385+0000", 2024-08-19T14:38:48.455 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.1114000000000001e-05 2024-08-19T14:38:48.455 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:38:48.455 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:38:48.455 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:38:48.455 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.455 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.455 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.16962.0:181834 7.a e678/635 7:55a8c86a:::benchmark_data_smithi045_115842_object63674:head v 678'11332, mlcod=625'6138)", 2024-08-19T14:38:48.455 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:37:43.529824+0000", 2024-08-19T14:38:48.455 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 64.879392385000003, 2024-08-19T14:38:48.455 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.13102048499999999, 2024-08-19T14:38:48.455 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:38:48.456 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:38:48.456 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:38:48.456 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:38:48.456 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.456 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:38:48.456 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.529824+0000", 2024-08-19T14:38:48.456 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:38:48.456 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.456 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.456 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:38:48.456 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.529824+0000", 2024-08-19T14:38:48.456 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:38:48.457 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.457 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.457 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:38:48.457 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.529825+0000", 2024-08-19T14:38:48.457 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.7899999999999997e-07 2024-08-19T14:38:48.457 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.457 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.457 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:38:48.457 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.529831+0000", 2024-08-19T14:38:48.457 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.6109999999999999e-06 2024-08-19T14:38:48.457 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.457 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.457 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:38:48.458 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.615979+0000", 2024-08-19T14:38:48.458 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.086148107000000002 2024-08-19T14:38:48.458 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.458 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.458 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:38:48.458 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.615990+0000", 2024-08-19T14:38:48.458 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0397999999999999e-05 2024-08-19T14:38:48.458 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.458 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.458 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:38:48.458 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.622181+0000", 2024-08-19T14:38:48.458 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0061911120000000004 2024-08-19T14:38:48.458 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.459 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.459 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:38:48.459 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.622206+0000", 2024-08-19T14:38:48.459 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.5392e-05 2024-08-19T14:38:48.459 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.459 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.459 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:38:48.459 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.660773+0000", 2024-08-19T14:38:48.459 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.038566415 2024-08-19T14:38:48.459 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.459 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.459 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:38:48.459 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.660844+0000", 2024-08-19T14:38:48.460 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.1670999999999999e-05 2024-08-19T14:38:48.460 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:38:48.460 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:38:48.460 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:38:48.460 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.460 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.460 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.16962.0:193467 7.e e687/634 7:7507bbf2:::benchmark_data_smithi045_115842_object75307:head v 687'12210, mlcod=615'4810)", 2024-08-19T14:38:48.460 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:38:09.955684+0000", 2024-08-19T14:38:48.460 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 38.453531892999997, 2024-08-19T14:38:48.460 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.134407253, 2024-08-19T14:38:48.460 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:38:48.460 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:38:48.461 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:38:48.461 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:38:48.461 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.461 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:38:48.461 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:09.955684+0000", 2024-08-19T14:38:48.461 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:38:48.461 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.461 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.461 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:38:48.461 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:09.955684+0000", 2024-08-19T14:38:48.461 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:38:48.461 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.461 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.462 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:38:48.462 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:09.955685+0000", 2024-08-19T14:38:48.462 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.2900000000000003e-07 2024-08-19T14:38:48.462 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.462 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.462 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:38:48.462 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:09.955690+0000", 2024-08-19T14:38:48.462 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1710000000000001e-06 2024-08-19T14:38:48.462 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.462 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.462 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:38:48.463 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:09.955698+0000", 2024-08-19T14:38:48.463 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.3690000000000007e-06 2024-08-19T14:38:48.463 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.463 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.463 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:38:48.463 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:09.955704+0000", 2024-08-19T14:38:48.463 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.8370000000000002e-06 2024-08-19T14:38:48.463 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.463 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.463 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:38:48.463 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:09.960208+0000", 2024-08-19T14:38:48.463 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0045041059999999999 2024-08-19T14:38:48.463 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.463 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.464 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:38:48.464 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:09.960233+0000", 2024-08-19T14:38:48.464 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.5116000000000001e-05 2024-08-19T14:38:48.464 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.464 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.464 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:38:48.464 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:10.089745+0000", 2024-08-19T14:38:48.464 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.12951114399999999 2024-08-19T14:38:48.464 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.464 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.464 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:38:48.464 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:10.090091+0000", 2024-08-19T14:38:48.464 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000346881 2024-08-19T14:38:48.464 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:38:48.465 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:38:48.465 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:38:48.465 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.465 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.465 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "MOSDPGPush(7.a 687/635 [PushOp(7:5dfc064c:::benchmark_data_smithi045_115842_object113897:head, version: 629'7087, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:5dfc064c:::benchmark_data_smithi045_115842_object113897:head@629'7087, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-19T14:38:48.465 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:38:09.967652+0000", 2024-08-19T14:38:48.465 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 38.441563893999998, 2024-08-19T14:38:48.465 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.135725331, 2024-08-19T14:38:48.465 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:38:48.465 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:38:48.465 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-08-19T14:38:48.465 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:38:48.465 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.465 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:38:48.466 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:09.967652+0000", 2024-08-19T14:38:48.466 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:38:48.466 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.466 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.466 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:38:48.466 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:09.967652+0000", 2024-08-19T14:38:48.466 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:38:48.466 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.466 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.466 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:38:48.466 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:09.967654+0000", 2024-08-19T14:38:48.466 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2950000000000001e-06 2024-08-19T14:38:48.466 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.466 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.467 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:38:48.467 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:09.967705+0000", 2024-08-19T14:38:48.467 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1054000000000003e-05 2024-08-19T14:38:48.467 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.467 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.467 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:38:48.467 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:10.062409+0000", 2024-08-19T14:38:48.467 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.094704211999999996 2024-08-19T14:38:48.467 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.467 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.467 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:38:48.467 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:10.062460+0000", 2024-08-19T14:38:48.468 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1523000000000001e-05 2024-08-19T14:38:48.468 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.468 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.468 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:38:48.469 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:10.096545+0000", 2024-08-19T14:38:48.469 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.034085161000000003 2024-08-19T14:38:48.469 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.469 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.469 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:38:48.469 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:10.096559+0000", 2024-08-19T14:38:48.469 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3144e-05 2024-08-19T14:38:48.469 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.469 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.469 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:38:48.469 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:10.103378+0000", 2024-08-19T14:38:48.469 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0068189419999999997 2024-08-19T14:38:48.470 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:38:48.470 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:38:48.470 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:38:48.470 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.471 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.471 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.16962.0:199394 7.2 e690/635 7:406e87f5:::benchmark_data_smithi045_115842_object81234:head v 690'12614, mlcod=621'5122)", 2024-08-19T14:38:48.471 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:38:22.737244+0000", 2024-08-19T14:38:48.471 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 25.671972438000001, 2024-08-19T14:38:48.471 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.24429008799999999, 2024-08-19T14:38:48.471 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:38:48.471 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:38:48.471 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:38:48.471 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:38:48.471 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.471 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:38:48.471 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.737244+0000", 2024-08-19T14:38:48.472 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:38:48.472 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.472 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.472 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:38:48.472 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.737244+0000", 2024-08-19T14:38:48.472 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:38:48.472 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.472 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.472 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:38:48.473 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.737245+0000", 2024-08-19T14:38:48.473 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.6299999999999993e-07 2024-08-19T14:38:48.473 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.473 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.473 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:38:48.473 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.737251+0000", 2024-08-19T14:38:48.473 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.6930000000000001e-06 2024-08-19T14:38:48.473 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.473 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.473 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:38:48.473 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.737262+0000", 2024-08-19T14:38:48.474 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0922e-05 2024-08-19T14:38:48.474 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.474 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.474 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:38:48.474 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.737269+0000", 2024-08-19T14:38:48.474 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.6680000000000004e-06 2024-08-19T14:38:48.474 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.474 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.474 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:38:48.474 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.737648+0000", 2024-08-19T14:38:48.474 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00037954999999999998 2024-08-19T14:38:48.474 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.474 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.474 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:38:48.475 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.737671+0000", 2024-08-19T14:38:48.475 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.2784000000000001e-05 2024-08-19T14:38:48.475 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.475 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.475 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:38:48.475 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.981464+0000", 2024-08-19T14:38:48.475 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.243792642 2024-08-19T14:38:48.475 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.475 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.475 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:38:48.475 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.981534+0000", 2024-08-19T14:38:48.475 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.9865999999999998e-05 2024-08-19T14:38:48.475 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:38:48.476 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:38:48.476 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:38:48.476 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.476 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.476 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.16962.0:199392 7.2 e690/635 7:4580edf3:::benchmark_data_smithi045_115842_object81232:head v 690'12615, mlcod=621'5122)", 2024-08-19T14:38:48.476 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:38:22.746524+0000", 2024-08-19T14:38:48.476 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 25.662692478, 2024-08-19T14:38:48.476 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.23504549899999999, 2024-08-19T14:38:48.476 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:38:48.476 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:38:48.476 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:38:48.476 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:38:48.476 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.477 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:38:48.477 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.746524+0000", 2024-08-19T14:38:48.477 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:38:48.477 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.477 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.477 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:38:48.477 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.746524+0000", 2024-08-19T14:38:48.477 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:38:48.477 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.477 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.477 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:38:48.477 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.746525+0000", 2024-08-19T14:38:48.477 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0279999999999999e-06 2024-08-19T14:38:48.477 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.477 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.478 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:38:48.478 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.746533+0000", 2024-08-19T14:38:48.478 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.0069999999999997e-06 2024-08-19T14:38:48.478 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.478 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.478 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:38:48.478 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.746545+0000", 2024-08-19T14:38:48.478 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2584e-05 2024-08-19T14:38:48.478 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.478 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.478 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:38:48.478 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.746554+0000", 2024-08-19T14:38:48.478 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.0180000000000007e-06 2024-08-19T14:38:48.479 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.479 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.479 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:38:48.479 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.752313+0000", 2024-08-19T14:38:48.479 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0057586139999999996 2024-08-19T14:38:48.479 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.479 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.479 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:38:48.479 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.752334+0000", 2024-08-19T14:38:48.479 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.0735000000000001e-05 2024-08-19T14:38:48.479 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.479 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.479 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:38:48.480 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.981537+0000", 2024-08-19T14:38:48.480 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.22920327500000001 2024-08-19T14:38:48.480 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.480 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.480 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:38:48.480 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.981569+0000", 2024-08-19T14:38:48.480 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.2237999999999999e-05 2024-08-19T14:38:48.480 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:38:48.480 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:38:48.480 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:38:48.480 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.480 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.480 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.16962.0:199409 7.2 e690/635 7:4c8d089b:::benchmark_data_smithi045_115842_object81249:head v 690'12616, mlcod=621'5122)", 2024-08-19T14:38:48.481 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:38:22.750502+0000", 2024-08-19T14:38:48.481 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 25.658714282999998, 2024-08-19T14:38:48.481 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.255713405, 2024-08-19T14:38:48.481 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:38:48.481 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:38:48.481 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:38:48.481 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:38:48.481 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.481 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:38:48.481 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.750502+0000", 2024-08-19T14:38:48.481 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:38:48.481 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.481 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.482 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:38:48.482 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.750502+0000", 2024-08-19T14:38:48.482 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:38:48.482 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.482 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.482 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:38:48.482 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.750503+0000", 2024-08-19T14:38:48.482 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.09e-07 2024-08-19T14:38:48.482 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.482 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.482 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:38:48.482 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.750511+0000", 2024-08-19T14:38:48.482 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.9759999999999995e-06 2024-08-19T14:38:48.483 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.483 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.483 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:38:48.483 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.750522+0000", 2024-08-19T14:38:48.483 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1241e-05 2024-08-19T14:38:48.483 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.483 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.483 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:38:48.483 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.750529+0000", 2024-08-19T14:38:48.483 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.3629999999999996e-06 2024-08-19T14:38:48.483 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.483 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.484 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:38:48.484 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.765007+0000", 2024-08-19T14:38:48.484 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.014478058 2024-08-19T14:38:48.484 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.484 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.484 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:38:48.484 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.765023+0000", 2024-08-19T14:38:48.484 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5735000000000001e-05 2024-08-19T14:38:48.484 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.484 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.484 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:38:48.484 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:23.006112+0000", 2024-08-19T14:38:48.484 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.24108848799999999 2024-08-19T14:38:48.484 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.485 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.485 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:38:48.485 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:23.006215+0000", 2024-08-19T14:38:48.485 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00010363500000000001 2024-08-19T14:38:48.485 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:38:48.485 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:38:48.485 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:38:48.485 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.485 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.485 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.16962.0:199414 7.c e690/635 7:366d05c1:::benchmark_data_smithi045_115842_object81254:head v 690'12294, mlcod=621'5016)", 2024-08-19T14:38:48.485 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:38:22.751905+0000", 2024-08-19T14:38:48.485 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 25.657310894999998, 2024-08-19T14:38:48.485 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.20950311999999999, 2024-08-19T14:38:48.485 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:38:48.485 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:38:48.486 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:38:48.486 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:38:48.486 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.486 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:38:48.486 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.751905+0000", 2024-08-19T14:38:48.486 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:38:48.486 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.486 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.486 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:38:48.486 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.751905+0000", 2024-08-19T14:38:48.486 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:38:48.486 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.486 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.486 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:38:48.487 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.751906+0000", 2024-08-19T14:38:48.487 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.4000000000000001e-07 2024-08-19T14:38:48.487 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.487 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.487 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:38:48.487 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.751912+0000", 2024-08-19T14:38:48.487 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.0100000000000001e-06 2024-08-19T14:38:48.487 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.487 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.487 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:38:48.487 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.751923+0000", 2024-08-19T14:38:48.487 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.098e-05 2024-08-19T14:38:48.487 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.487 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.488 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:38:48.488 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.751930+0000", 2024-08-19T14:38:48.488 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.0600000000000002e-06 2024-08-19T14:38:48.488 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.488 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.488 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:38:48.488 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.752067+0000", 2024-08-19T14:38:48.488 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013742100000000001 2024-08-19T14:38:48.488 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.488 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.488 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:38:48.488 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.752089+0000", 2024-08-19T14:38:48.488 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.1389000000000001e-05 2024-08-19T14:38:48.489 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.489 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.489 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:38:48.489 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.766981+0000", 2024-08-19T14:38:48.489 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.014892396 2024-08-19T14:38:48.489 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.489 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.489 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:38:48.489 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.961408+0000", 2024-08-19T14:38:48.489 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.19442712400000001 2024-08-19T14:38:48.489 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:38:48.489 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:38:48.489 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:38:48.489 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.490 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.490 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.16962.0:199422 7.a e690/635 7:5ea5870d:::benchmark_data_smithi045_115842_object81262:head v 690'12384, mlcod=633'7611)", 2024-08-19T14:38:48.490 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:38:22.759906+0000", 2024-08-19T14:38:48.490 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 25.649310067000002, 2024-08-19T14:38:48.490 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.242679165, 2024-08-19T14:38:48.490 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:38:48.490 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:38:48.490 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:38:48.490 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:38:48.490 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.490 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:38:48.490 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.759906+0000", 2024-08-19T14:38:48.490 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:38:48.491 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.491 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.491 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:38:48.491 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.759906+0000", 2024-08-19T14:38:48.491 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:38:48.491 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.491 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.491 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:38:48.491 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.759908+0000", 2024-08-19T14:38:48.491 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5880000000000001e-06 2024-08-19T14:38:48.491 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.491 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.491 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:38:48.492 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.759916+0000", 2024-08-19T14:38:48.492 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.4950000000000008e-06 2024-08-19T14:38:48.492 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.492 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.492 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:38:48.492 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.759930+0000", 2024-08-19T14:38:48.492 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4236000000000001e-05 2024-08-19T14:38:48.492 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.492 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.492 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:38:48.493 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.759941+0000", 2024-08-19T14:38:48.493 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1052e-05 2024-08-19T14:38:48.493 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.493 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.493 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:38:48.493 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.765122+0000", 2024-08-19T14:38:48.493 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0051806439999999999 2024-08-19T14:38:48.493 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.493 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.493 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:38:48.493 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.765141+0000", 2024-08-19T14:38:48.493 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.8470000000000001e-05 2024-08-19T14:38:48.493 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.494 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.494 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:38:48.494 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:23.002541+0000", 2024-08-19T14:38:48.494 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.23740081599999999 2024-08-19T14:38:48.494 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.494 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.494 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:38:48.494 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:23.002585+0000", 2024-08-19T14:38:48.494 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.3863999999999999e-05 2024-08-19T14:38:48.494 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:38:48.494 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:38:48.494 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:38:48.495 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.495 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.495 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "MOSDPGPush(7.2 690/635 [PushOp(7:48749d7b:::benchmark_data_smithi045_115842_object81270:head, version: 621'5127, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:48749d7b:::benchmark_data_smithi045_115842_object81270:head@621'5127, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-19T14:38:48.495 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:38:22.764504+0000", 2024-08-19T14:38:48.495 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 25.644711906000001, 2024-08-19T14:38:48.495 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.20554665, 2024-08-19T14:38:48.495 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:38:48.495 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:38:48.495 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-08-19T14:38:48.495 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:38:48.495 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.495 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:38:48.495 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.764504+0000", 2024-08-19T14:38:48.496 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:38:48.496 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.496 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.496 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:38:48.496 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.764504+0000", 2024-08-19T14:38:48.496 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:38:48.496 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.496 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.496 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:38:48.496 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.764505+0000", 2024-08-19T14:38:48.496 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.6400000000000001e-07 2024-08-19T14:38:48.496 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.496 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.497 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:38:48.497 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.764536+0000", 2024-08-19T14:38:48.497 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.0719000000000002e-05 2024-08-19T14:38:48.497 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.497 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.497 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:38:48.497 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.764556+0000", 2024-08-19T14:38:48.497 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.0142000000000001e-05 2024-08-19T14:38:48.497 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.497 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.497 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:38:48.497 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.764573+0000", 2024-08-19T14:38:48.497 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6685999999999999e-05 2024-08-19T14:38:48.498 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.498 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.498 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:38:48.498 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.966878+0000", 2024-08-19T14:38:48.498 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.20230553300000001 2024-08-19T14:38:48.498 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.498 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.498 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:38:48.498 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.966888+0000", 2024-08-19T14:38:48.498 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.026e-05 2024-08-19T14:38:48.498 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.498 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.499 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:38:48.499 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.970051+0000", 2024-08-19T14:38:48.499 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.003162546 2024-08-19T14:38:48.499 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:38:48.499 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:38:48.499 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:38:48.499 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.499 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.499 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "MOSDPGRecoveryDelete(7.e e690,634 7:7b982ab6:::benchmark_data_smithi045_115842_object57353:head,673'11068)", 2024-08-19T14:38:48.499 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:38:22.767854+0000", 2024-08-19T14:38:48.499 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 25.641362100999999, 2024-08-19T14:38:48.499 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.19375594500000001, 2024-08-19T14:38:48.499 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:38:48.500 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:38:48.500 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-08-19T14:38:48.500 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:38:48.500 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.500 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:38:48.500 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.767854+0000", 2024-08-19T14:38:48.500 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:38:48.500 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.500 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.500 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:38:48.500 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.767854+0000", 2024-08-19T14:38:48.501 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:38:48.501 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.501 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.501 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:38:48.501 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.767855+0000", 2024-08-19T14:38:48.501 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2410000000000001e-06 2024-08-19T14:38:48.501 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.501 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.501 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:38:48.501 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.767872+0000", 2024-08-19T14:38:48.501 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6707000000000001e-05 2024-08-19T14:38:48.501 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.501 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.502 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:38:48.502 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.961428+0000", 2024-08-19T14:38:48.502 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.19355593600000001 2024-08-19T14:38:48.502 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.502 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.502 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:38:48.502 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.961441+0000", 2024-08-19T14:38:48.502 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3269e-05 2024-08-19T14:38:48.502 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.502 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.502 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:38:48.502 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.961526+0000", 2024-08-19T14:38:48.502 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.4942000000000001e-05 2024-08-19T14:38:48.503 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.503 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.503 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:38:48.503 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.961534+0000", 2024-08-19T14:38:48.503 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.6350000000000006e-06 2024-08-19T14:38:48.503 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.503 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.503 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:38:48.503 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.961610+0000", 2024-08-19T14:38:48.503 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.6215000000000001e-05 2024-08-19T14:38:48.503 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:38:48.504 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:38:48.504 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:38:48.504 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.504 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.504 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.16962.0:199399 7.3 e690/635 7:cecd84d1:::benchmark_data_smithi045_115842_object81239:head v 690'12702, mlcod=621'5174)", 2024-08-19T14:38:48.504 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:38:22.776198+0000", 2024-08-19T14:38:48.504 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 25.633018235000002, 2024-08-19T14:38:48.506 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.20653019, 2024-08-19T14:38:48.506 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:38:48.506 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:38:48.506 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:38:48.506 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:38:48.506 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.506 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:38:48.506 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.776198+0000", 2024-08-19T14:38:48.507 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:38:48.507 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.507 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.507 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:38:48.507 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.776198+0000", 2024-08-19T14:38:48.507 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:38:48.507 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.507 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.507 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:38:48.507 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.776199+0000", 2024-08-19T14:38:48.507 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2449999999999999e-06 2024-08-19T14:38:48.507 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.507 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.507 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:38:48.508 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.776210+0000", 2024-08-19T14:38:48.508 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1289e-05 2024-08-19T14:38:48.508 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.508 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.508 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:38:48.508 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.961366+0000", 2024-08-19T14:38:48.508 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.18515516700000001 2024-08-19T14:38:48.508 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.508 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.508 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:38:48.508 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.961387+0000", 2024-08-19T14:38:48.508 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.1248999999999999e-05 2024-08-19T14:38:48.508 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.508 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.509 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:38:48.509 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.961448+0000", 2024-08-19T14:38:48.509 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.1062000000000006e-05 2024-08-19T14:38:48.509 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.509 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.509 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:38:48.509 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.961480+0000", 2024-08-19T14:38:48.509 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.1775e-05 2024-08-19T14:38:48.509 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.509 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.509 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:38:48.509 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.982684+0000", 2024-08-19T14:38:48.509 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.021203914000000001 2024-08-19T14:38:48.509 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:38:48.510 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:38:48.510 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:38:48.510 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.982728+0000", 2024-08-19T14:38:48.510 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4489e-05 2024-08-19T14:38:48.510 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:38:48.510 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:38:48.510 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:38:48.510 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:38:48.510 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:38:48.510 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:38:48.510 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-19T14:38:48.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:48.512 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:48.512 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:38:48.613 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:38:48.635 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:48.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:48.636 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:38:48.737 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:38:48.760 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:48.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:48.761 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:38:48.862 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:38:48.876 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is set 2024-08-19T14:38:48.884 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:48.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:48.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:38:48.986 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:38:49.035 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:49.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:49.036 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:38:49.137 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:38:49.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:49.161 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:49.162 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:38:49.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:48 smithi032 ceph-mon[20004]: 7.f scrub ok 2024-08-19T14:38:49.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:48 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 14257/80487 objects degraded (17.713%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-19T14:38:49.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:48 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2649534186' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:38:49.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:48 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:38:49.262 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:38:49.286 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:49.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:49.287 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:38:49.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:48 smithi045 ceph-mon[20143]: 7.f scrub ok 2024-08-19T14:38:49.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:48 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 14257/80487 objects degraded (17.713%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-19T14:38:49.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:48 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2649534186' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:38:49.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:48 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:38:49.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:48 smithi031 ceph-mon[16336]: 7.f scrub ok 2024-08-19T14:38:49.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:48 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 14257/80487 objects degraded (17.713%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-19T14:38:49.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:48 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2649534186' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:38:49.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:48 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:38:49.388 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:38:49.412 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:49.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:49.413 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:38:49.483 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-08-19T14:38:49.513 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:38:49.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:49.587 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:49.587 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:38:49.688 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:38:49.732 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:49.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:49.734 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:38:49.835 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:38:49.856 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:49.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:49.857 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:38:49.958 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:38:49.980 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:49.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:49.981 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:38:50.082 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:38:50.099 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:49 smithi031 ceph-mon[16336]: pgmap v1663: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.6 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 498 op/s; 14095/78225 objects degraded (18.019%); 3931/78225 objects misplaced (5.025%); 2.4 MiB/s, 119 objects/s recovering 2024-08-19T14:38:50.099 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:49 smithi031 ceph-mon[16336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:38:50.099 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:49 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:38:50.099 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:49 smithi031 ceph-mon[16336]: osdmap e697: 12 total, 12 up, 10 in 2024-08-19T14:38:50.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:50.107 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:50.108 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:38:50.208 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:38:50.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:49 smithi032 ceph-mon[20004]: pgmap v1663: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.6 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 498 op/s; 14095/78225 objects degraded (18.019%); 3931/78225 objects misplaced (5.025%); 2.4 MiB/s, 119 objects/s recovering 2024-08-19T14:38:50.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:49 smithi032 ceph-mon[20004]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:38:50.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:49 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:38:50.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:49 smithi032 ceph-mon[20004]: osdmap e697: 12 total, 12 up, 10 in 2024-08-19T14:38:50.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:50.266 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:50.266 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:38:50.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:49 smithi045 ceph-mon[20143]: pgmap v1663: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.6 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 498 op/s; 14095/78225 objects degraded (18.019%); 3931/78225 objects misplaced (5.025%); 2.4 MiB/s, 119 objects/s recovering 2024-08-19T14:38:50.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:49 smithi045 ceph-mon[20143]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:38:50.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:49 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:38:50.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:49 smithi045 ceph-mon[20143]: osdmap e697: 12 total, 12 up, 10 in 2024-08-19T14:38:50.312 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:38:50.367 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:38:50.417 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:38:50.441 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:50.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:50.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:38:50.453 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:38:50.543 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:38:50.599 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:50.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:50.600 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:38:50.701 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:38:50.722 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:50.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:50.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:38:50.824 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:38:50.845 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:50.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:50.846 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:38:50.948 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:38:50.970 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:50.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:50.971 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:38:51.072 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:38:51.122 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:38:51.123 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:38:51.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:51.124 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:51.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:38:51.174 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:38:51.174 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:38:51.209 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:38:51.209 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:38:51.225 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:38:51.247 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:51.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:51.248 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:38:51.257 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:38:51.257 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:38:51.289 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:38:51.290 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:38:51.331 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:38:51.331 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:38:51.348 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:38:51.368 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:38:51.368 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:38:51.370 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:51.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:51.371 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:38:51.407 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:38:51.407 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:38:51.441 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:38:51.441 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:38:51.473 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:38:51.478 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:38:51.478 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:38:51.516 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:51.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:51.517 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:38:51.521 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:38:51.521 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:38:51.554 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:38:51.554 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:38:51.618 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:38:51.640 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:51.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:51.641 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:38:51.742 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:38:51.764 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:51.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:51.765 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:38:51.865 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:38:51.903 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:51.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:51.904 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:38:52.005 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:38:52.026 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:52.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:52.027 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:38:52.113 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:38:52.116 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-08-19T14:38:52.128 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:38:52.144 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:51 smithi031 ceph-mon[16336]: pgmap v1665: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.5 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 502 op/s; 13771/75900 objects degraded (18.144%); 3931/75900 objects misplaced (5.179%); 6.5 MiB/s, 154 objects/s recovering 2024-08-19T14:38:52.144 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:51 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/975650579' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:38:52.148 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:52.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:52.149 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:38:52.151 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-19T14:38:52.251 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:38:52.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:51 smithi032 ceph-mon[20004]: pgmap v1665: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.5 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 502 op/s; 13771/75900 objects degraded (18.144%); 3931/75900 objects misplaced (5.179%); 6.5 MiB/s, 154 objects/s recovering 2024-08-19T14:38:52.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:51 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/975650579' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:38:52.261 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:38:52.261 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:38:52.274 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:52.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:52.275 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:38:52.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:51 smithi045 ceph-mon[20143]: pgmap v1665: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.5 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 502 op/s; 13771/75900 objects degraded (18.144%); 3931/75900 objects misplaced (5.179%); 6.5 MiB/s, 154 objects/s recovering 2024-08-19T14:38:52.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:51 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/975650579' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:38:52.376 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:38:52.400 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:52.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:52.401 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:38:52.502 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:38:52.549 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:52.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:52.550 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:38:52.651 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:38:52.673 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:52.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:52.674 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:38:52.775 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:38:52.798 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:52.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:52.799 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:38:52.899 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:38:52.921 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:52.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:52.923 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:38:53.024 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:38:53.047 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:53.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:53.048 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:38:53.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:52 smithi031 ceph-mon[16336]: from='client.26927 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:38:53.149 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:38:53.157 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:52 smithi032 ceph-mon[20004]: from='client.26927 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:38:53.171 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:53.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:53.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:38:53.196 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:52 smithi045 ceph-mon[20143]: from='client.26927 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:38:53.271 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-08-19T14:38:53.273 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:38:53.297 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:53.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:53.298 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:38:53.399 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:38:53.421 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:53.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:53.422 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:38:53.523 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:38:53.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:53.573 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:53.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:38:53.674 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:38:53.709 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:53.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:53.710 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:38:53.810 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:38:53.835 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:53.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:53.837 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:38:53.937 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:38:53.962 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:53.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:53.963 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:38:54.064 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:38:54.087 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:54.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:54.089 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:38:54.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:53 smithi031 ceph-mon[16336]: pgmap v1666: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.5 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 422 op/s; 13601/72324 objects degraded (18.806%); 3931/72324 objects misplaced (5.435%); 3.1 MiB/s, 122 objects/s recovering 2024-08-19T14:38:54.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:53 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 13601/72324 objects degraded (18.806%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-19T14:38:54.190 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:38:54.215 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:54.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:54.216 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:38:54.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:53 smithi032 ceph-mon[20004]: pgmap v1666: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.5 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 422 op/s; 13601/72324 objects degraded (18.806%); 3931/72324 objects misplaced (5.435%); 3.1 MiB/s, 122 objects/s recovering 2024-08-19T14:38:54.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:53 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 13601/72324 objects degraded (18.806%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-19T14:38:54.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:53 smithi045 ceph-mon[20143]: pgmap v1666: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.5 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 422 op/s; 13601/72324 objects degraded (18.806%); 3931/72324 objects misplaced (5.435%); 3.1 MiB/s, 122 objects/s recovering 2024-08-19T14:38:54.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:53 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 13601/72324 objects degraded (18.806%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-19T14:38:54.299 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":1665,"stamp":"2024-08-19T14:38:50.253661+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1658454584,"num_objects":25300,"num_object_clones":0,"num_object_copies":75900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13771,"num_objects_misplaced":3931,"num_objects_unfound":0,"num_objects_dirty":25300,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":211018,"num_write_kb":7562049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46724,"num_bytes_recovered":2902594080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":164855,"ondisk_log_size":164855,"up":141,"acting":136,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":143,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":13685668,"kb_used_data":5118136,"kb_used_omap":18,"kb_used_meta":8567405,"kb_avail":923765852,"statfs":{"total":959950356480,"available":945936232448,"internally_reserved":0,"allocated":5240971264,"data_stored":5210216657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18760,"internal_metadata":8773023416},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,4,1,0,4,6,0,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-197787648,"num_objects":-3018,"num_object_clones":0,"num_object_copies":-9054,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-794,"num_objects_misplaced":-101,"num_objects_unfound":0,"num_objects_dirty":-3018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3018,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":928,"num_bytes_recovered":41091072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.004394"},"pg_stats":[{"pgid":"7.6","version":"694'13411","reported_seq":13712,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-08-19T14:38:45.441059+0000","last_change":"2024-08-19T14:37:56.236536+0000","last_active":"2024-08-19T14:38:45.441059+0000","last_peered":"2024-08-19T14:38:45.441059+0000","last_clean":"2024-08-19T14:38:45.441059+0000","last_became_active":"2024-08-19T14:33:28.334317+0000","last_became_peered":"2024-08-19T14:33:28.334317+0000","last_unstale":"2024-08-19T14:38:45.441059+0000","last_undegraded":"2024-08-19T14:38:45.441059+0000","last_fullsized":"2024-08-19T14:38:45.441059+0000","mapping_epoch":600,"log_start":"611'3400","ondisk_log_start":"611'3400","created":600,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"684'11923","last_scrub_stamp":"2024-08-19T14:37:56.236394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:37:56.236394+0000","objects_scrubbed":3155,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:39:12.585833+0000","scrub_duration":1586,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107675648,"num_objects":1643,"num_object_clones":0,"num_object_copies":4929,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1643,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13411,"num_write_kb":481728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"70'139","reported_seq":1707,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-08-19T14:38:39.805831+0000","last_change":"2024-08-19T14:37:57.701402+0000","last_active":"2024-08-19T14:38:39.805831+0000","last_peered":"2024-08-19T14:38:39.805831+0000","last_clean":"2024-08-19T14:38:39.805831+0000","last_became_active":"2024-08-19T14:35:13.950224+0000","last_became_peered":"2024-08-19T14:35:13.950224+0000","last_unstale":"2024-08-19T14:38:39.805831+0000","last_undegraded":"2024-08-19T14:38:39.805831+0000","last_fullsized":"2024-08-19T14:38:39.805831+0000","mapping_epoch":637,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":638,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:37:57.701290+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:19:58.072083+0000","last_clean_scrub_stamp":"2024-08-19T14:37:57.701290+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:39:18.431910+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,9],"acting":[0,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.5","version":"695'13024","reported_seq":22505,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-08-19T14:38:46.582725+0000","last_change":"2024-08-19T14:35:23.022361+0000","last_active":"2024-08-19T14:38:46.582725+0000","last_peered":"2024-08-19T14:38:46.582725+0000","last_clean":"2024-08-19T14:38:46.582725+0000","last_became_active":"2024-08-19T14:35:23.009238+0000","last_became_peered":"2024-08-19T14:35:23.009238+0000","last_unstale":"2024-08-19T14:38:46.582725+0000","last_undegraded":"2024-08-19T14:38:46.582725+0000","last_fullsized":"2024-08-19T14:38:46.582725+0000","mapping_epoch":641,"log_start":"611'3000","ondisk_log_start":"611'3000","created":600,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108658688,"num_objects":1658,"num_object_clones":0,"num_object_copies":4974,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1658,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13024,"num_write_kb":469824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4634,"num_bytes_recovered":303693824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,0],"acting":[11,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.3","version":"70'139","reported_seq":1555,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-08-19T14:38:46.840258+0000","last_change":"2024-08-19T14:33:18.804321+0000","last_active":"2024-08-19T14:38:46.840258+0000","last_peered":"2024-08-19T14:38:46.840258+0000","last_clean":"2024-08-19T14:38:46.840258+0000","last_became_active":"2024-08-19T14:18:13.804721+0000","last_became_peered":"2024-08-19T14:18:13.804721+0000","last_unstale":"2024-08-19T14:38:46.840258+0000","last_undegraded":"2024-08-19T14:38:46.840258+0000","last_fullsized":"2024-08-19T14:38:46.840258+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:33:18.804228+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:33:18.804228+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.4","version":"694'13282","reported_seq":13580,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-08-19T14:38:45.482084+0000","last_change":"2024-08-19T14:38:06.717577+0000","last_active":"2024-08-19T14:38:45.482084+0000","last_peered":"2024-08-19T14:38:45.482084+0000","last_clean":"2024-08-19T14:38:45.482084+0000","last_became_active":"2024-08-19T14:33:28.334446+0000","last_became_peered":"2024-08-19T14:33:28.334446+0000","last_unstale":"2024-08-19T14:38:45.482084+0000","last_undegraded":"2024-08-19T14:38:45.482084+0000","last_fullsized":"2024-08-19T14:38:45.482084+0000","mapping_epoch":600,"log_start":"611'3200","ondisk_log_start":"611'3200","created":600,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"686'12159","last_scrub_stamp":"2024-08-19T14:38:06.717004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:38:06.717004+0000","objects_scrubbed":2792,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:39:16.656979+0000","scrub_duration":1017,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":1664,"num_object_clones":0,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1664,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13282,"num_write_kb":478272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"70'139","reported_seq":1507,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-08-19T14:38:39.795344+0000","last_change":"2024-08-19T14:34:55.580212+0000","last_active":"2024-08-19T14:38:39.795344+0000","last_peered":"2024-08-19T14:38:39.795344+0000","last_clean":"2024-08-19T14:38:39.795344+0000","last_became_active":"2024-08-19T14:20:50.742354+0000","last_became_peered":"2024-08-19T14:20:50.742354+0000","last_unstale":"2024-08-19T14:38:39.795344+0000","last_undegraded":"2024-08-19T14:38:39.795344+0000","last_fullsized":"2024-08-19T14:38:39.795344+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:34:55.580141+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:19:59.302931+0000","last_clean_scrub_stamp":"2024-08-19T14:34:55.580141+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"694'12853","reported_seq":13153,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-08-19T14:38:45.314677+0000","last_change":"2024-08-19T14:37:43.229201+0000","last_active":"2024-08-19T14:38:45.314677+0000","last_peered":"2024-08-19T14:38:45.314677+0000","last_clean":"2024-08-19T14:38:45.314677+0000","last_became_active":"2024-08-19T14:33:28.331833+0000","last_became_peered":"2024-08-19T14:33:28.331833+0000","last_unstale":"2024-08-19T14:38:45.314677+0000","last_undegraded":"2024-08-19T14:38:45.314677+0000","last_fullsized":"2024-08-19T14:38:45.314677+0000","mapping_epoch":600,"log_start":"611'2800","ondisk_log_start":"611'2800","created":600,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"678'11114","last_scrub_stamp":"2024-08-19T14:37:43.229072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:37:43.229072+0000","objects_scrubbed":3413,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:39:12.361681+0000","scrub_duration":910,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109117440,"num_objects":1665,"num_object_clones":0,"num_object_copies":4995,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12853,"num_write_kb":464576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,1],"acting":[8,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1","version":"70'139","reported_seq":805,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-08-19T14:38:39.806104+0000","last_change":"2024-08-19T14:37:42.630047+0000","last_active":"2024-08-19T14:38:39.806104+0000","last_peered":"2024-08-19T14:38:39.806104+0000","last_clean":"2024-08-19T14:38:39.806104+0000","last_became_active":"2024-08-19T14:35:12.530566+0000","last_became_peered":"2024-08-19T14:35:12.530566+0000","last_unstale":"2024-08-19T14:38:39.806104+0000","last_undegraded":"2024-08-19T14:38:39.806104+0000","last_fullsized":"2024-08-19T14:38:39.806104+0000","mapping_epoch":635,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":636,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:37:42.629919+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:37:42.629919+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:39:05.497590+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,2],"acting":[0,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"696'13070","reported_seq":13351,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-08-19T14:38:48.641517+0000","last_change":"2024-08-19T14:34:52.869444+0000","last_active":"2024-08-19T14:38:48.641517+0000","last_peered":"2024-08-19T14:38:48.641517+0000","last_clean":"2024-08-19T14:38:48.641517+0000","last_became_active":"2024-08-19T14:33:28.332994+0000","last_became_peered":"2024-08-19T14:33:28.332994+0000","last_unstale":"2024-08-19T14:38:48.641517+0000","last_undegraded":"2024-08-19T14:38:48.641517+0000","last_fullsized":"2024-08-19T14:38:48.641517+0000","mapping_epoch":600,"log_start":"611'3000","ondisk_log_start":"611'3000","created":600,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"627'6737","last_scrub_stamp":"2024-08-19T14:34:52.869233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:34:52.869233+0000","objects_scrubbed":6705,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1588,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98828288,"num_objects":1508,"num_object_clones":0,"num_object_copies":4524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13070,"num_write_kb":466496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"70'139","reported_seq":1215,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-08-19T14:38:48.881263+0000","last_change":"2024-08-19T14:35:11.537368+0000","last_active":"2024-08-19T14:38:48.881263+0000","last_peered":"2024-08-19T14:38:48.881263+0000","last_clean":"2024-08-19T14:38:48.881263+0000","last_became_active":"2024-08-19T14:35:11.536828+0000","last_became_peered":"2024-08-19T14:35:11.536828+0000","last_unstale":"2024-08-19T14:38:48.881263+0000","last_undegraded":"2024-08-19T14:38:48.881263+0000","last_fullsized":"2024-08-19T14:38:48.881263+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:27.536928+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:21:46.057624+0000","last_clean_scrub_stamp":"2024-08-19T14:28:27.536928+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"697'13018","reported_seq":22457,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-08-19T14:38:50.209457+0000","last_change":"2024-08-19T14:36:15.120872+0000","last_active":"2024-08-19T14:38:50.209457+0000","last_peered":"2024-08-19T14:38:50.209457+0000","last_clean":"2024-08-19T14:38:50.209457+0000","last_became_active":"2024-08-19T14:36:15.096070+0000","last_became_peered":"2024-08-19T14:36:15.096070+0000","last_unstale":"2024-08-19T14:38:50.209457+0000","last_undegraded":"2024-08-19T14:38:50.209457+0000","last_fullsized":"2024-08-19T14:38:50.209457+0000","mapping_epoch":653,"log_start":"611'3000","ondisk_log_start":"611'3000","created":600,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":101056512,"num_objects":1542,"num_object_clones":0,"num_object_copies":4626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1542,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13018,"num_write_kb":465920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4611,"num_bytes_recovered":302186496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,9],"acting":[3,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"70'139","reported_seq":777,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-08-19T14:38:39.793941+0000","last_change":"2024-08-19T14:37:45.357396+0000","last_active":"2024-08-19T14:38:39.793941+0000","last_peered":"2024-08-19T14:38:39.793941+0000","last_clean":"2024-08-19T14:38:39.793941+0000","last_became_active":"2024-08-19T14:34:15.717054+0000","last_became_peered":"2024-08-19T14:34:15.717054+0000","last_unstale":"2024-08-19T14:38:39.793941+0000","last_undegraded":"2024-08-19T14:38:39.793941+0000","last_fullsized":"2024-08-19T14:38:39.793941+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:37:45.357293+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:27.848870+0000","last_clean_scrub_stamp":"2024-08-19T14:37:45.357293+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:38:48.964771+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"7.3","version":"697'13519","reported_seq":21182,"reported_epoch":697,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-19T14:38:50.200546+0000","last_change":"2024-08-19T14:35:13.136988+0000","last_active":"2024-08-19T14:38:50.200546+0000","last_peered":"2024-08-19T14:38:50.200546+0000","last_clean":"2024-08-19T14:33:27.312789+0000","last_became_active":"2024-08-19T14:35:13.021078+0000","last_became_peered":"2024-08-19T14:35:13.021078+0000","last_unstale":"2024-08-19T14:38:50.200546+0000","last_undegraded":"2024-08-19T14:35:12.592288+0000","last_fullsized":"2024-08-19T14:35:12.526971+0000","mapping_epoch":635,"log_start":"611'3500","ondisk_log_start":"611'3500","created":600,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100597760,"num_objects":1535,"num_object_clones":0,"num_object_copies":4605,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3530,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1535,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13519,"num_write_kb":481728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3728,"num_bytes_recovered":241303552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":-1995}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"70'139","reported_seq":733,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-08-19T14:38:45.821082+0000","last_change":"2024-08-19T14:35:11.533207+0000","last_active":"2024-08-19T14:38:45.821082+0000","last_peered":"2024-08-19T14:38:45.821082+0000","last_clean":"2024-08-19T14:38:45.821082+0000","last_became_active":"2024-08-19T14:35:11.532858+0000","last_became_peered":"2024-08-19T14:35:11.532858+0000","last_unstale":"2024-08-19T14:38:45.821082+0000","last_undegraded":"2024-08-19T14:38:45.821082+0000","last_fullsized":"2024-08-19T14:38:45.821082+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:27.331041+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:28:27.331041+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,0],"acting":[11,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.2","version":"697'13414","reported_seq":20967,"reported_epoch":697,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-19T14:38:50.235968+0000","last_change":"2024-08-19T14:35:13.121204+0000","last_active":"2024-08-19T14:38:50.235968+0000","last_peered":"2024-08-19T14:38:50.235968+0000","last_clean":"2024-08-19T14:33:27.312789+0000","last_became_active":"2024-08-19T14:35:13.013347+0000","last_became_peered":"2024-08-19T14:35:13.013347+0000","last_unstale":"2024-08-19T14:38:50.235968+0000","last_undegraded":"2024-08-19T14:35:12.598259+0000","last_fullsized":"2024-08-19T14:35:12.526602+0000","mapping_epoch":635,"log_start":"611'3400","ondisk_log_start":"611'3400","created":600,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102629376,"num_objects":1566,"num_object_clones":0,"num_object_copies":4698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3562,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1566,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13414,"num_write_kb":479360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3673,"num_bytes_recovered":237895680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":-1996}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"70'139","reported_seq":610,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-08-19T14:38:45.820812+0000","last_change":"2024-08-19T14:33:16.760928+0000","last_active":"2024-08-19T14:38:45.820812+0000","last_peered":"2024-08-19T14:38:45.820812+0000","last_clean":"2024-08-19T14:38:45.820812+0000","last_became_active":"2024-08-19T14:30:47.909067+0000","last_became_peered":"2024-08-19T14:30:47.909067+0000","last_unstale":"2024-08-19T14:38:45.820812+0000","last_undegraded":"2024-08-19T14:38:45.820812+0000","last_fullsized":"2024-08-19T14:38:45.820812+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:33:16.760810+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:33:16.760810+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.c","version":"697'13086","reported_seq":20765,"reported_epoch":697,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-19T14:38:50.221093+0000","last_change":"2024-08-19T14:35:13.133729+0000","last_active":"2024-08-19T14:38:50.221093+0000","last_peered":"2024-08-19T14:38:50.221093+0000","last_clean":"2024-08-19T14:33:27.312789+0000","last_became_active":"2024-08-19T14:35:13.003266+0000","last_became_peered":"2024-08-19T14:35:13.003266+0000","last_unstale":"2024-08-19T14:38:50.221093+0000","last_undegraded":"2024-08-19T14:35:12.593502+0000","last_fullsized":"2024-08-19T14:35:12.526375+0000","mapping_epoch":635,"log_start":"611'3000","ondisk_log_start":"611'3000","created":600,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96337920,"num_objects":1470,"num_object_clones":0,"num_object_copies":4410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3273,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1470,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":465792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3736,"num_bytes_recovered":243269632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,2],"acting":[3,11],"avail_no_missing":["3","11"],"object_location_counts":[{"shards":"3,11","objects":-1803}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"70'139","reported_seq":778,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-08-19T14:38:46.845319+0000","last_change":"2024-08-19T14:34:50.256878+0000","last_active":"2024-08-19T14:38:46.845319+0000","last_peered":"2024-08-19T14:38:46.845319+0000","last_clean":"2024-08-19T14:38:46.845319+0000","last_became_active":"2024-08-19T14:34:15.717472+0000","last_became_peered":"2024-08-19T14:34:15.717472+0000","last_unstale":"2024-08-19T14:38:46.845319+0000","last_undegraded":"2024-08-19T14:38:46.845319+0000","last_fullsized":"2024-08-19T14:38:46.845319+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:34:50.256808+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:28.864870+0000","last_clean_scrub_stamp":"2024-08-19T14:34:50.256808+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,0],"acting":[4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"694'16603","reported_seq":26408,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-08-19T14:38:45.523442+0000","last_change":"2024-08-19T14:37:42.029576+0000","last_active":"2024-08-19T14:38:45.523442+0000","last_peered":"2024-08-19T14:38:45.523442+0000","last_clean":"2024-08-19T14:38:45.523442+0000","last_became_active":"2024-08-19T14:36:14.098166+0000","last_became_peered":"2024-08-19T14:36:14.098166+0000","last_unstale":"2024-08-19T14:38:45.523442+0000","last_undegraded":"2024-08-19T14:38:45.523442+0000","last_fullsized":"2024-08-19T14:38:45.523442+0000","mapping_epoch":652,"log_start":"627'6602","ondisk_log_start":"627'6602","created":600,"last_epoch_clean":653,"parent":"0.0","parent_split_bits":0,"last_scrub":"678'14831","last_scrub_stamp":"2024-08-19T14:37:42.029449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:37:42.029449+0000","objects_scrubbed":3383,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1404,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104398848,"num_objects":1593,"num_object_clones":0,"num_object_copies":4779,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12973,"num_write_kb":466112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4700,"num_bytes_recovered":308019200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.9","version":"70'139","reported_seq":1709,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-08-19T14:38:39.795300+0000","last_change":"2024-08-19T14:35:12.529993+0000","last_active":"2024-08-19T14:38:39.795300+0000","last_peered":"2024-08-19T14:38:39.795300+0000","last_clean":"2024-08-19T14:38:39.795300+0000","last_became_active":"2024-08-19T14:35:12.529628+0000","last_became_peered":"2024-08-19T14:35:12.529628+0000","last_unstale":"2024-08-19T14:38:39.795300+0000","last_undegraded":"2024-08-19T14:38:39.795300+0000","last_fullsized":"2024-08-19T14:38:39.795300+0000","mapping_epoch":635,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":636,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:34:51.561893+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:34:51.561893+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.e","version":"696'13310","reported_seq":26309,"reported_epoch":696,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-19T14:38:47.757667+0000","last_change":"2024-08-19T14:35:11.863818+0000","last_active":"2024-08-19T14:38:47.757667+0000","last_peered":"2024-08-19T14:38:47.757667+0000","last_clean":"2024-08-19T14:33:27.312789+0000","last_became_active":"2024-08-19T14:35:11.806304+0000","last_became_peered":"2024-08-19T14:35:11.806304+0000","last_unstale":"2024-08-19T14:38:47.757667+0000","last_undegraded":"2024-08-19T14:35:11.779866+0000","last_fullsized":"2024-08-19T14:35:11.528207+0000","mapping_epoch":634,"log_start":"611'3300","ondisk_log_start":"611'3300","created":600,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":97386496,"num_objects":1486,"num_object_clones":0,"num_object_copies":4458,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2445,"num_objects_misplaced":3931,"num_objects_unfound":0,"num_objects_dirty":1486,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13310,"num_write_kb":473472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4821,"num_bytes_recovered":266665984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":-2445},{"shards":"1,5,9","objects":1486}],"blocked_by":[],"up_primary":2,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"70'139","reported_seq":746,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-08-19T14:38:48.883103+0000","last_change":"2024-08-19T14:35:11.537896+0000","last_active":"2024-08-19T14:38:48.883103+0000","last_peered":"2024-08-19T14:38:48.883103+0000","last_clean":"2024-08-19T14:38:48.883103+0000","last_became_active":"2024-08-19T14:35:11.537474+0000","last_became_peered":"2024-08-19T14:35:11.537474+0000","last_unstale":"2024-08-19T14:38:48.883103+0000","last_undegraded":"2024-08-19T14:38:48.883103+0000","last_fullsized":"2024-08-19T14:38:48.883103+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:59.829567+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:28:59.829567+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"696'13245","reported_seq":13450,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-08-19T14:38:47.691120+0000","last_change":"2024-08-19T14:33:28.335586+0000","last_active":"2024-08-19T14:38:47.691120+0000","last_peered":"2024-08-19T14:38:47.691120+0000","last_clean":"2024-08-19T14:38:47.691120+0000","last_became_active":"2024-08-19T14:33:28.335298+0000","last_became_peered":"2024-08-19T14:33:28.335298+0000","last_unstale":"2024-08-19T14:38:47.691120+0000","last_undegraded":"2024-08-19T14:38:47.691120+0000","last_fullsized":"2024-08-19T14:38:47.691120+0000","mapping_epoch":600,"log_start":"611'3200","ondisk_log_start":"611'3200","created":600,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96993304,"num_objects":1481,"num_object_clones":0,"num_object_copies":4443,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1481,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13245,"num_write_kb":471169,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"70'139","reported_seq":1062,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-08-19T14:38:45.824225+0000","last_change":"2024-08-19T14:33:15.753668+0000","last_active":"2024-08-19T14:38:45.824225+0000","last_peered":"2024-08-19T14:38:45.824225+0000","last_clean":"2024-08-19T14:38:45.824225+0000","last_became_active":"2024-08-19T14:30:47.941200+0000","last_became_peered":"2024-08-19T14:30:47.941200+0000","last_unstale":"2024-08-19T14:38:45.824225+0000","last_undegraded":"2024-08-19T14:38:45.824225+0000","last_fullsized":"2024-08-19T14:38:45.824225+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:33:15.753595+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:33:15.753595+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.8","version":"694'13438","reported_seq":28756,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-08-19T14:38:45.360997+0000","last_change":"2024-08-19T14:38:13.418089+0000","last_active":"2024-08-19T14:38:45.360997+0000","last_peered":"2024-08-19T14:38:45.360997+0000","last_clean":"2024-08-19T14:38:45.360997+0000","last_became_active":"2024-08-19T14:38:13.389656+0000","last_became_peered":"2024-08-19T14:38:13.389656+0000","last_unstale":"2024-08-19T14:38:45.360997+0000","last_undegraded":"2024-08-19T14:38:45.360997+0000","last_fullsized":"2024-08-19T14:38:45.360997+0000","mapping_epoch":688,"log_start":"611'3400","ondisk_log_start":"611'3400","created":600,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109576192,"num_objects":1672,"num_object_clones":0,"num_object_copies":5016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13438,"num_write_kb":483520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7555,"num_bytes_recovered":476577792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.e","version":"70'139","reported_seq":1255,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-08-19T14:38:46.845434+0000","last_change":"2024-08-19T14:34:55.333177+0000","last_active":"2024-08-19T14:38:46.845434+0000","last_peered":"2024-08-19T14:38:46.845434+0000","last_clean":"2024-08-19T14:38:46.845434+0000","last_became_active":"2024-08-19T14:18:26.884105+0000","last_became_peered":"2024-08-19T14:18:26.884105+0000","last_unstale":"2024-08-19T14:38:46.845434+0000","last_undegraded":"2024-08-19T14:38:46.845434+0000","last_fullsized":"2024-08-19T14:38:46.845434+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:34:55.333065+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:32:31.196288+0000","last_clean_scrub_stamp":"2024-08-19T14:34:55.333065+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"694'13228","reported_seq":13564,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-08-19T14:38:45.551397+0000","last_change":"2024-08-19T14:37:54.857799+0000","last_active":"2024-08-19T14:38:45.551397+0000","last_peered":"2024-08-19T14:38:45.551397+0000","last_clean":"2024-08-19T14:38:45.551397+0000","last_became_active":"2024-08-19T14:33:28.334547+0000","last_became_peered":"2024-08-19T14:33:28.334547+0000","last_unstale":"2024-08-19T14:38:45.551397+0000","last_undegraded":"2024-08-19T14:38:45.551397+0000","last_fullsized":"2024-08-19T14:38:45.551397+0000","mapping_epoch":600,"log_start":"611'3200","ondisk_log_start":"611'3200","created":600,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"683'11737","last_scrub_stamp":"2024-08-19T14:37:54.851356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:37:54.851356+0000","objects_scrubbed":3161,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:39:09.304335+0000","scrub_duration":1254,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108265472,"num_objects":1652,"num_object_clones":0,"num_object_copies":4956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1652,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13228,"num_write_kb":476160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,8],"acting":[0,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"70'139","reported_seq":1007,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-08-19T14:38:46.845270+0000","last_change":"2024-08-19T14:34:12.199530+0000","last_active":"2024-08-19T14:38:46.845270+0000","last_peered":"2024-08-19T14:38:46.845270+0000","last_clean":"2024-08-19T14:38:46.845270+0000","last_became_active":"2024-08-19T14:27:31.837754+0000","last_became_peered":"2024-08-19T14:27:31.837754+0000","last_unstale":"2024-08-19T14:38:46.845270+0000","last_undegraded":"2024-08-19T14:38:46.845270+0000","last_fullsized":"2024-08-19T14:38:46.845270+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:34:12.199449+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:34:12.199449+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.a","version":"695'13112","reported_seq":33699,"reported_epoch":695,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-19T14:38:46.730393+0000","last_change":"2024-08-19T14:35:13.046246+0000","last_active":"2024-08-19T14:38:46.730393+0000","last_peered":"2024-08-19T14:38:46.730393+0000","last_clean":"2024-08-19T14:33:27.312789+0000","last_became_active":"2024-08-19T14:35:13.005199+0000","last_became_peered":"2024-08-19T14:35:13.005199+0000","last_unstale":"2024-08-19T14:38:46.730393+0000","last_undegraded":"2024-08-19T14:35:12.692218+0000","last_fullsized":"2024-08-19T14:35:12.524923+0000","mapping_epoch":635,"log_start":"611'3100","ondisk_log_start":"611'3100","created":600,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":105775104,"num_objects":1614,"num_object_clones":0,"num_object_copies":4842,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":961,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13112,"num_write_kb":471232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9263,"num_bytes_recovered":522125312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,2],"acting":[11,9],"avail_no_missing":["11","3","9"],"object_location_counts":[{"shards":"9,11","objects":653}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.c","version":"70'139","reported_seq":1364,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-08-19T14:38:48.885646+0000","last_change":"2024-08-19T14:34:15.738488+0000","last_active":"2024-08-19T14:38:48.885646+0000","last_peered":"2024-08-19T14:38:48.885646+0000","last_clean":"2024-08-19T14:38:48.885646+0000","last_became_active":"2024-08-19T14:34:15.733282+0000","last_became_peered":"2024-08-19T14:34:15.733282+0000","last_unstale":"2024-08-19T14:38:48.885646+0000","last_undegraded":"2024-08-19T14:38:48.885646+0000","last_fullsized":"2024-08-19T14:38:48.885646+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:33:17.453290+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:22:52.164586+0000","last_clean_scrub_stamp":"2024-08-19T14:33:17.453290+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,9],"acting":[3,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"696'13035","reported_seq":13239,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-08-19T14:38:47.764156+0000","last_change":"2024-08-19T14:33:28.623717+0000","last_active":"2024-08-19T14:38:47.764156+0000","last_peered":"2024-08-19T14:38:47.764156+0000","last_clean":"2024-08-19T14:38:47.764156+0000","last_became_active":"2024-08-19T14:33:28.623318+0000","last_became_peered":"2024-08-19T14:33:28.623318+0000","last_unstale":"2024-08-19T14:38:47.764156+0000","last_undegraded":"2024-08-19T14:38:47.764156+0000","last_fullsized":"2024-08-19T14:38:47.764156+0000","mapping_epoch":600,"log_start":"611'3000","ondisk_log_start":"611'3000","created":600,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":101515264,"num_objects":1549,"num_object_clones":0,"num_object_copies":4647,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1549,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13035,"num_write_kb":466688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"70'139","reported_seq":1255,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-08-19T14:38:39.795537+0000","last_change":"2024-08-19T14:34:49.476489+0000","last_active":"2024-08-19T14:38:39.795537+0000","last_peered":"2024-08-19T14:38:39.795537+0000","last_clean":"2024-08-19T14:38:39.795537+0000","last_became_active":"2024-08-19T14:18:26.882002+0000","last_became_peered":"2024-08-19T14:18:26.882002+0000","last_unstale":"2024-08-19T14:38:39.795537+0000","last_undegraded":"2024-08-19T14:38:39.795537+0000","last_fullsized":"2024-08-19T14:38:39.795537+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:34:49.476415+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:24:50.276408+0000","last_clean_scrub_stamp":"2024-08-19T14:34:49.476415+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"70'139","reported_seq":1037,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-08-19T14:38:46.840105+0000","last_change":"2024-08-19T14:33:19.781527+0000","last_active":"2024-08-19T14:38:46.840105+0000","last_peered":"2024-08-19T14:38:46.840105+0000","last_clean":"2024-08-19T14:38:46.840105+0000","last_became_active":"2024-08-19T14:27:31.838968+0000","last_became_peered":"2024-08-19T14:27:31.838968+0000","last_unstale":"2024-08-19T14:38:46.840105+0000","last_undegraded":"2024-08-19T14:38:46.840105+0000","last_fullsized":"2024-08-19T14:38:46.840105+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:33:19.781431+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:28:35.206115+0000","last_clean_scrub_stamp":"2024-08-19T14:33:19.781431+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.13","version":"70'139","reported_seq":1579,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-08-19T14:38:46.840155+0000","last_change":"2024-08-19T14:33:35.786236+0000","last_active":"2024-08-19T14:38:46.840155+0000","last_peered":"2024-08-19T14:38:46.840155+0000","last_clean":"2024-08-19T14:38:46.840155+0000","last_became_active":"2024-08-19T14:30:47.939849+0000","last_became_peered":"2024-08-19T14:30:47.939849+0000","last_unstale":"2024-08-19T14:38:46.840155+0000","last_undegraded":"2024-08-19T14:38:46.840155+0000","last_fullsized":"2024-08-19T14:38:46.840155+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:33:35.786105+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:33:35.786105+0000","last_clean_scrub_stamp":"2024-08-19T14:33:35.786105+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.10","version":"70'139","reported_seq":1203,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-08-19T14:38:39.795446+0000","last_change":"2024-08-19T14:34:58.616722+0000","last_active":"2024-08-19T14:38:39.795446+0000","last_peered":"2024-08-19T14:38:39.795446+0000","last_clean":"2024-08-19T14:38:39.795446+0000","last_became_active":"2024-08-19T14:27:31.839611+0000","last_became_peered":"2024-08-19T14:27:31.839611+0000","last_unstale":"2024-08-19T14:38:39.795446+0000","last_undegraded":"2024-08-19T14:38:39.795446+0000","last_fullsized":"2024-08-19T14:38:39.795446+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:34:58.616646+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:34:58.616646+0000","last_clean_scrub_stamp":"2024-08-19T14:34:58.616646+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.11","version":"70'139","reported_seq":1259,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-08-19T14:38:39.805999+0000","last_change":"2024-08-19T14:38:34.904558+0000","last_active":"2024-08-19T14:38:39.805999+0000","last_peered":"2024-08-19T14:38:39.805999+0000","last_clean":"2024-08-19T14:38:39.805999+0000","last_became_active":"2024-08-19T14:34:15.731049+0000","last_became_peered":"2024-08-19T14:34:15.731049+0000","last_unstale":"2024-08-19T14:38:39.805999+0000","last_undegraded":"2024-08-19T14:38:39.805999+0000","last_fullsized":"2024-08-19T14:38:39.805999+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:38:34.904308+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:24:50.547428+0000","last_clean_scrub_stamp":"2024-08-19T14:38:34.904308+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:40:02.722075+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"70'139","reported_seq":1190,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-08-19T14:38:46.846473+0000","last_change":"2024-08-19T14:33:27.737496+0000","last_active":"2024-08-19T14:38:46.846473+0000","last_peered":"2024-08-19T14:38:46.846473+0000","last_clean":"2024-08-19T14:38:46.846473+0000","last_became_active":"2024-08-19T14:20:28.204724+0000","last_became_peered":"2024-08-19T14:20:28.204724+0000","last_unstale":"2024-08-19T14:38:46.846473+0000","last_undegraded":"2024-08-19T14:38:46.846473+0000","last_fullsized":"2024-08-19T14:38:46.846473+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:33:27.737367+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:28:34.216370+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.737367+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"70'139","reported_seq":1329,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-08-19T14:38:39.794052+0000","last_change":"2024-08-19T14:37:38.355694+0000","last_active":"2024-08-19T14:38:39.794052+0000","last_peered":"2024-08-19T14:38:39.794052+0000","last_clean":"2024-08-19T14:38:39.794052+0000","last_became_active":"2024-08-19T14:18:31.901556+0000","last_became_peered":"2024-08-19T14:18:31.901556+0000","last_unstale":"2024-08-19T14:38:39.794052+0000","last_undegraded":"2024-08-19T14:38:39.794052+0000","last_fullsized":"2024-08-19T14:38:39.794052+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:37:38.355621+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:37:38.355621+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":56,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,0],"acting":[8,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.14","version":"70'139","reported_seq":1358,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-08-19T14:38:39.795380+0000","last_change":"2024-08-19T14:35:12.530145+0000","last_active":"2024-08-19T14:38:39.795380+0000","last_peered":"2024-08-19T14:38:39.795380+0000","last_clean":"2024-08-19T14:38:39.795380+0000","last_became_active":"2024-08-19T14:35:12.529792+0000","last_became_peered":"2024-08-19T14:35:12.529792+0000","last_unstale":"2024-08-19T14:38:39.795380+0000","last_undegraded":"2024-08-19T14:38:39.795380+0000","last_fullsized":"2024-08-19T14:38:39.795380+0000","mapping_epoch":635,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":636,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:34:53.617013+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:34:53.617013+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,9],"acting":[7,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.15","version":"70'139","reported_seq":1210,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-08-19T14:38:39.795503+0000","last_change":"2024-08-19T14:34:57.640316+0000","last_active":"2024-08-19T14:38:39.795503+0000","last_peered":"2024-08-19T14:38:39.795503+0000","last_clean":"2024-08-19T14:38:39.795503+0000","last_became_active":"2024-08-19T14:18:31.912496+0000","last_became_peered":"2024-08-19T14:18:31.912496+0000","last_unstale":"2024-08-19T14:38:39.795503+0000","last_undegraded":"2024-08-19T14:38:39.795503+0000","last_fullsized":"2024-08-19T14:38:39.795503+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:34:57.640213+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:20:09.180923+0000","last_clean_scrub_stamp":"2024-08-19T14:34:57.640213+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"70'139","reported_seq":1200,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-08-19T14:38:46.845226+0000","last_change":"2024-08-19T14:34:06.176510+0000","last_active":"2024-08-19T14:38:46.845226+0000","last_peered":"2024-08-19T14:38:46.845226+0000","last_clean":"2024-08-19T14:38:46.845226+0000","last_became_active":"2024-08-19T14:18:31.901229+0000","last_became_peered":"2024-08-19T14:18:31.901229+0000","last_unstale":"2024-08-19T14:38:46.845226+0000","last_undegraded":"2024-08-19T14:38:46.845226+0000","last_fullsized":"2024-08-19T14:38:46.845226+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:34:06.176403+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:34:06.176403+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1b","version":"70'139","reported_seq":1577,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-08-19T14:38:39.793904+0000","last_change":"2024-08-19T14:37:59.510774+0000","last_active":"2024-08-19T14:38:39.793904+0000","last_peered":"2024-08-19T14:38:39.793904+0000","last_clean":"2024-08-19T14:38:39.793904+0000","last_became_active":"2024-08-19T14:34:15.718427+0000","last_became_peered":"2024-08-19T14:34:15.718427+0000","last_unstale":"2024-08-19T14:38:39.793904+0000","last_undegraded":"2024-08-19T14:38:39.793904+0000","last_fullsized":"2024-08-19T14:38:39.793904+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:37:59.510697+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:37:59.510697+0000","last_clean_scrub_stamp":"2024-08-19T14:37:59.510697+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:39:04.057695+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.18","version":"70'139","reported_seq":1138,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-08-19T14:38:39.795280+0000","last_change":"2024-08-19T14:35:11.528963+0000","last_active":"2024-08-19T14:38:39.795280+0000","last_peered":"2024-08-19T14:38:39.795280+0000","last_clean":"2024-08-19T14:38:39.795280+0000","last_became_active":"2024-08-19T14:35:11.528530+0000","last_became_peered":"2024-08-19T14:35:11.528530+0000","last_unstale":"2024-08-19T14:38:39.795280+0000","last_undegraded":"2024-08-19T14:38:39.795280+0000","last_fullsized":"2024-08-19T14:38:39.795280+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:29:05.336156+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:27:33.084161+0000","last_clean_scrub_stamp":"2024-08-19T14:29:05.336156+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.19","version":"70'139","reported_seq":1556,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-08-19T14:38:39.805892+0000","last_change":"2024-08-19T14:37:41.671836+0000","last_active":"2024-08-19T14:38:39.805892+0000","last_peered":"2024-08-19T14:38:39.805892+0000","last_clean":"2024-08-19T14:38:39.805892+0000","last_became_active":"2024-08-19T14:18:31.902547+0000","last_became_peered":"2024-08-19T14:18:31.902547+0000","last_unstale":"2024-08-19T14:38:39.805892+0000","last_undegraded":"2024-08-19T14:38:39.805892+0000","last_fullsized":"2024-08-19T14:38:39.805892+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:37:41.671731+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:37:41.671731+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:38:59.596992+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1e","version":"70'139","reported_seq":582,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-08-19T14:38:45.820698+0000","last_change":"2024-08-19T14:33:17.778415+0000","last_active":"2024-08-19T14:38:45.820698+0000","last_peered":"2024-08-19T14:38:45.820698+0000","last_clean":"2024-08-19T14:38:45.820698+0000","last_became_active":"2024-08-19T14:30:47.909974+0000","last_became_peered":"2024-08-19T14:30:47.909974+0000","last_unstale":"2024-08-19T14:38:45.820698+0000","last_undegraded":"2024-08-19T14:38:45.820698+0000","last_fullsized":"2024-08-19T14:38:45.820698+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:33:17.778295+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:27:04.780939+0000","last_clean_scrub_stamp":"2024-08-19T14:33:17.778295+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1c","version":"70'139","reported_seq":1331,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-08-19T14:38:39.793860+0000","last_change":"2024-08-19T14:38:01.463630+0000","last_active":"2024-08-19T14:38:39.793860+0000","last_peered":"2024-08-19T14:38:39.793860+0000","last_clean":"2024-08-19T14:38:39.793860+0000","last_became_active":"2024-08-19T14:30:47.908866+0000","last_became_peered":"2024-08-19T14:30:47.908866+0000","last_unstale":"2024-08-19T14:38:39.793860+0000","last_undegraded":"2024-08-19T14:38:39.793860+0000","last_fullsized":"2024-08-19T14:38:39.793860+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:38:01.463524+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:38:01.463524+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:39:18.496929+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,11],"acting":[8,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1d","version":"70'139","reported_seq":1244,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-08-19T14:38:46.840440+0000","last_change":"2024-08-19T14:35:11.542467+0000","last_active":"2024-08-19T14:38:46.840440+0000","last_peered":"2024-08-19T14:38:46.840440+0000","last_clean":"2024-08-19T14:38:46.840440+0000","last_became_active":"2024-08-19T14:35:11.540488+0000","last_became_peered":"2024-08-19T14:35:11.540488+0000","last_unstale":"2024-08-19T14:38:46.840440+0000","last_undegraded":"2024-08-19T14:38:46.840440+0000","last_fullsized":"2024-08-19T14:38:46.840440+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:29.194639+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:28:29.194639+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":1657864216,"num_objects":25298,"num_object_clones":0,"num_object_copies":75894,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13771,"num_objects_misplaced":3931,"num_objects_unfound":0,"num_objects_dirty":25298,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":211018,"num_write_kb":7562049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46721,"num_bytes_recovered":2901737472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5099827200,"data_stored":5099814984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160546,"ondisk_log_size":160546,"up":48,"acting":43,"num_store_stats":10},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":856608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187785,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1388200,"kb_used_data":530972,"kb_used_omap":1,"kb_used_meta":857214,"kb_avail":92356952,"statfs":{"total":95995035648,"available":94573518848,"internally_reserved":0,"allocated":543715328,"data_stored":540644178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":877787588},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":2314987372653,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":74,"seq":317827580243,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1336308,"kb_used_data":506856,"kb_used_omap":1,"kb_used_meta":829438,"kb_avail":92408844,"statfs":{"total":95995035648,"available":94626656256,"internally_reserved":0,"allocated":519020544,"data_stored":515945367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":849344961},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874585,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1107288,"kb_used_data":327824,"kb_used_omap":1,"kb_used_meta":779454,"kb_avail":92637864,"statfs":{"total":95995035648,"available":94861172736,"internally_reserved":0,"allocated":335691776,"data_stored":332624653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":798161349},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":57,"seq":244813136220,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1263968,"kb_used_data":334552,"kb_used_omap":1,"kb_used_meta":929406,"kb_avail":92481184,"statfs":{"total":95995035648,"available":94700732416,"internally_reserved":0,"allocated":342581248,"data_stored":339510573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":951712202},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":206158430560,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692195,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1242616,"kb_used_data":309040,"kb_used_omap":1,"kb_used_meta":933566,"kb_avail":92502536,"statfs":{"total":95995035648,"available":94722596864,"internally_reserved":0,"allocated":316456960,"data_stored":313378231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":955972037},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986536,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1379128,"kb_used_data":409448,"kb_used_omap":1,"kb_used_meta":969662,"kb_avail":92366024,"statfs":{"total":95995035648,"available":94582808576,"internally_reserved":0,"allocated":419274752,"data_stored":416199575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":992934341},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150063,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2246596,"kb_used_data":1240824,"kb_used_omap":1,"kb_used_meta":1005758,"kb_avail":91498556,"statfs":{"total":95995035648,"available":93694521344,"internally_reserved":0,"allocated":1270603776,"data_stored":1267520450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1029896647},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":634,"seq":2723009265709,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":464328,"kb_used_data":207356,"kb_used_omap":4,"kb_used_meta":256955,"kb_avail":93280824,"statfs":{"total":95995035648,"available":95519563776,"internally_reserved":0,"allocated":212332544,"data_stored":209249738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4407,"internal_metadata":263122633},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575223,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1730168,"kb_used_data":607788,"kb_used_omap":1,"kb_used_meta":1122366,"kb_avail":92014984,"statfs":{"total":95995035648,"available":94223343616,"internally_reserved":0,"allocated":622374912,"data_stored":619295663,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1149303237},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,0,1,3,0,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738747,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1527068,"kb_used_data":643476,"kb_used_omap":1,"kb_used_meta":883582,"kb_avail":92218084,"statfs":{"total":95995035648,"available":94431318016,"internally_reserved":0,"allocated":658919424,"data_stored":655848229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":904788421},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":641208320,"data_stored":641204248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":605097984,"data_stored":605093912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":200998912,"data_stored":200998912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1249710080,"data_stored":1249706008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":409665536,"data_stored":409665536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":305725440,"data_stored":305725440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":329056256,"data_stored":329056256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":323092480,"data_stored":323092480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":508821504,"data_stored":508821504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":526450688,"data_stored":526450688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T14:38:54.301 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph status --format=json 2024-08-19T14:38:54.317 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:38:54.337 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:38:54.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:54.339 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:54.339 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:38:54.440 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:38:54.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:54.463 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:54.463 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:38:54.564 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:38:54.585 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:54.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:54.586 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:38:54.686 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:38:54.738 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:54.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:54.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:38:54.840 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:38:54.862 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:54.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:54.863 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:38:54.964 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:38:54.985 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:54.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:54.986 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:38:55.087 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:38:55.105 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:38:55.144 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:55.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:55.145 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:38:55.157 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:38:55.157 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:38:55.246 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:38:55.271 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:55.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:55.272 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:38:55.372 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:38:55.406 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:55.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:55.408 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:38:55.509 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:38:55.531 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:55.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:55.533 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:38:55.634 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:38:55.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:55.659 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:55.659 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:38:55.760 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:38:55.782 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:55.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:55.783 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:38:55.809 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:38:55.883 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:38:55.907 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:55.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:55.908 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:38:55.969 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-08-19T14:38:56.008 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:38:56.086 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:56.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:56.087 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:38:56.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:55 smithi031 ceph-mon[16336]: pgmap v1667: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.4 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 411 op/s; 13629/70599 objects degraded (19.305%); 3828/70599 objects misplaced (5.422%); 2.7 MiB/s, 111 objects/s recovering 2024-08-19T14:38:56.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:55 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:38:56.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:55 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:38:56.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:55 smithi031 ceph-mon[16336]: osdmap e698: 12 total, 12 up, 10 in 2024-08-19T14:38:56.187 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:38:56.211 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:56.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:56.212 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:38:56.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:55 smithi032 ceph-mon[20004]: pgmap v1667: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.4 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 411 op/s; 13629/70599 objects degraded (19.305%); 3828/70599 objects misplaced (5.422%); 2.7 MiB/s, 111 objects/s recovering 2024-08-19T14:38:56.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:55 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:38:56.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:55 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:38:56.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:55 smithi032 ceph-mon[20004]: osdmap e698: 12 total, 12 up, 10 in 2024-08-19T14:38:56.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:55 smithi045 ceph-mon[20143]: pgmap v1667: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.4 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 411 op/s; 13629/70599 objects degraded (19.305%); 3828/70599 objects misplaced (5.422%); 2.7 MiB/s, 111 objects/s recovering 2024-08-19T14:38:56.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:55 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:38:56.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:55 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:38:56.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:55 smithi045 ceph-mon[20143]: osdmap e698: 12 total, 12 up, 10 in 2024-08-19T14:38:56.313 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:38:56.364 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:56.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:56.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:38:56.408 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:38:56.466 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:38:56.491 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:56.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:56.492 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:38:56.593 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:38:56.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:56.615 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:56.615 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:38:56.716 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:38:56.738 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:56.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:56.739 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:38:56.840 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:38:56.862 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:56.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:56.863 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:38:56.964 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:38:57.019 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:57.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:57.020 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:38:57.121 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:38:57.143 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:57.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:57.144 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:38:57.245 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:38:57.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:57.292 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:57.292 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:38:57.393 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:38:57.430 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:57.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:57.431 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:38:57.533 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:38:57.561 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:57.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:57.562 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:38:57.570 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:38:57.577 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:38:57.663 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:38:57.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:57.702 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:57.702 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:38:57.803 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:38:57.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:57.827 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:57.827 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:38:57.916 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is set 2024-08-19T14:38:57.928 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:38:57.950 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:57.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:57.951 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:38:57.975 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:57 smithi031 ceph-mon[16336]: pgmap v1669: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.3 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 507 op/s; 13312/66045 objects degraded (20.156%); 3828/66045 objects misplaced (5.796%); 5.1 MiB/s, 114 objects/s recovering 2024-08-19T14:38:57.975 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:57 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1728021109' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:38:58.052 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:38:58.090 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:38:58.090 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"ed7d1c6e-5e33-11ef-bccf-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub,nodeep-scrub flag(s) set","count":20},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 13312/66045 objects degraded (20.156%), 5 pgs degraded, 5 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":1915,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":699,"num_osds":12,"num_up_osds":12,"osd_up_since":1724078110,"num_in_osds":10,"osd_in_since":1724078051,"num_remapped_pgs":5},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":42},{"state_name":"active+recovering+undersized+degraded+remapped","count":5}],"num_pgs":47,"num_pools":2,"num_objects":22015,"data_bytes":1443168824,"bytes_used":13450489856,"bytes_avail":946499866624,"bytes_total":959950356480,"degraded_objects":13312,"degraded_total":66045,"degraded_ratio":0.20155954273601331,"misplaced_objects":3828,"misplaced_total":66045,"misplaced_ratio":0.057960481489893259,"recovering_objects_per_sec":114,"recovering_bytes_per_sec":5371629,"recovering_keys_per_sec":0,"num_objects_recovered":914,"num_bytes_recovered":42991616,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":507},"fsmap":{"epoch":1,"btime":"2024-08-19T14:05:22:841940+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.31:8443/"}},"servicemap":{"epoch":45,"modified":"2024-08-19T14:38:54.255402+0000","services":{"mon":{"daemons":{"summary":"","c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-08-19T14:38:58.093 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:58.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:58.094 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:38:58.195 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:38:58.217 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:58.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:58.218 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:38:58.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:57 smithi032 ceph-mon[20004]: pgmap v1669: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.3 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 507 op/s; 13312/66045 objects degraded (20.156%); 3828/66045 objects misplaced (5.796%); 5.1 MiB/s, 114 objects/s recovering 2024-08-19T14:38:58.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:57 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1728021109' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:38:58.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:57 smithi045 ceph-mon[20143]: pgmap v1669: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.3 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 507 op/s; 13312/66045 objects degraded (20.156%); 3828/66045 objects misplaced (5.796%); 5.1 MiB/s, 114 objects/s recovering 2024-08-19T14:38:58.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:57 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1728021109' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:38:58.320 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:38:58.341 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:58.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:58.342 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:38:58.443 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:38:58.465 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:58.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:58.466 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:38:58.531 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:38:58.531 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:38:58.566 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:38:58.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:58.603 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:58.603 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:38:58.704 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:38:58.725 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:58.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:58.727 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:38:58.828 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:38:58.849 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:58.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:58.850 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:38:58.951 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:38:58.975 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:58.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:58.976 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:38:59.077 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:38:59.095 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:58 smithi031 ceph-mon[16336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:38:59.096 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:58 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1728021109' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:38:59.096 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:58 smithi031 ceph-mon[16336]: osdmap e699: 12 total, 12 up, 10 in 2024-08-19T14:38:59.096 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:58 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 13312/66045 objects degraded (20.156%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-19T14:38:59.096 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:58 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/352915735' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-08-19T14:38:59.110 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:38:59.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:59.111 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:38:59.211 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:38:59.233 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:59.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:59.234 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:38:59.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:58 smithi032 ceph-mon[20004]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:38:59.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:58 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1728021109' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:38:59.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:58 smithi032 ceph-mon[20004]: osdmap e699: 12 total, 12 up, 10 in 2024-08-19T14:38:59.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:58 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 13312/66045 objects degraded (20.156%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-19T14:38:59.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:58 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/352915735' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-08-19T14:38:59.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:58 smithi045 ceph-mon[20143]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:38:59.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:58 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1728021109' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:38:59.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:58 smithi045 ceph-mon[20143]: osdmap e699: 12 total, 12 up, 10 in 2024-08-19T14:38:59.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:58 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 13312/66045 objects degraded (20.156%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-19T14:38:59.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:58 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/352915735' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-08-19T14:38:59.335 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:38:59.361 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:59.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:59.362 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:38:59.462 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:38:59.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:59.487 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:59.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:38:59.564 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-08-19T14:38:59.564 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-19T14:38:59.588 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:38:59.613 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:38:59.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:59.615 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:38:59.716 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:38:59.738 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:59.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:59.743 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:38:59.844 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:38:59.866 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:38:59.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:38:59.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:38:59.969 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:39:00.017 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:00.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:00.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:39:00.119 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:39:00.141 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:00.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:00.142 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:39:00.209 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:38:59 smithi031 ceph-mon[16336]: pgmap v1671: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.3 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 506 op/s; 13160/63744 objects degraded (20.645%); 3730/63744 objects misplaced (5.852%); 3.4 MiB/s, 111 objects/s recovering 2024-08-19T14:39:00.243 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:39:00.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:38:59 smithi032 ceph-mon[20004]: pgmap v1671: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.3 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 506 op/s; 13160/63744 objects degraded (20.645%); 3730/63744 objects misplaced (5.852%); 3.4 MiB/s, 111 objects/s recovering 2024-08-19T14:39:00.266 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:00.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:00.267 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:39:00.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:38:59 smithi045 ceph-mon[20143]: pgmap v1671: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.3 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 506 op/s; 13160/63744 objects degraded (20.645%); 3730/63744 objects misplaced (5.852%); 3.4 MiB/s, 111 objects/s recovering 2024-08-19T14:39:00.368 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:39:00.403 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:00.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:00.404 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:39:00.505 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:39:00.541 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:00.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:00.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:39:00.643 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:39:00.685 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:00.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:00.686 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:39:00.787 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:39:00.809 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:00.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:00.810 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:39:00.911 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:39:00.933 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:00.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:00.934 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:39:01.035 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:39:01.082 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:01.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:01.083 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:39:01.184 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:39:01.207 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:01.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:01.208 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:39:01.309 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:39:01.332 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:01.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:01.333 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:39:01.434 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:39:01.457 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:01.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:01.459 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:39:01.560 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:39:01.582 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:01.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:01.587 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:39:01.688 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:39:01.711 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:01.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:01.712 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:39:01.813 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:39:01.840 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:01.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:01.841 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:39:01.942 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:39:01.979 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:01.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:01.980 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:39:02.081 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:39:02.102 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:02.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:02.103 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:39:02.204 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:39:02.225 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:02.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:02.226 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:39:02.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:01 smithi032 ceph-mon[20004]: pgmap v1672: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.3 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 374 op/s; 13160/63330 objects degraded (20.780%); 3730/63330 objects misplaced (5.890%); 3.4 MiB/s, 90 objects/s recovering 2024-08-19T14:39:02.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:01 smithi045 ceph-mon[20143]: pgmap v1672: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.3 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 374 op/s; 13160/63330 objects degraded (20.780%); 3730/63330 objects misplaced (5.890%); 3.4 MiB/s, 90 objects/s recovering 2024-08-19T14:39:02.328 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:39:02.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:02.350 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:02.350 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:39:02.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:01 smithi031 ceph-mon[16336]: pgmap v1672: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.3 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 374 op/s; 13160/63330 objects degraded (20.780%); 3730/63330 objects misplaced (5.890%); 3.4 MiB/s, 90 objects/s recovering 2024-08-19T14:39:02.451 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:39:02.472 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:02.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:02.473 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:39:02.574 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:39:02.597 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:02.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:02.598 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:39:02.700 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:39:02.723 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:02.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:02.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:39:02.825 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:39:02.846 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:02.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:02.847 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:39:02.948 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:39:02.970 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:02.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:02.971 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:39:03.074 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:39:03.095 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:03.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:03.096 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:39:03.197 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:39:03.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:03.248 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:03.248 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:39:03.349 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:39:03.373 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:03.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:03.374 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:39:03.475 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:39:03.496 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:03.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:03.498 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:39:03.599 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:39:03.620 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:03.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:03.621 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:39:03.722 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:39:03.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:03.746 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:03.746 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:39:03.847 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:39:03.869 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:03.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:03.870 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:39:03.971 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:39:03.993 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:03.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:03.994 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:39:04.092 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:03 smithi031 ceph-mon[16336]: pgmap v1673: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.2 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 541 op/s; 12637/57591 objects degraded (21.943%); 3730/57591 objects misplaced (6.477%); 5.5 MiB/s, 143 objects/s recovering 2024-08-19T14:39:04.092 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:03 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 12637/57591 objects degraded (21.943%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-19T14:39:04.092 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:03 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1849424330' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:39:04.095 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:39:04.141 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:04.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:04.142 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:39:04.244 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:39:04.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:03 smithi032 ceph-mon[20004]: pgmap v1673: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.2 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 541 op/s; 12637/57591 objects degraded (21.943%); 3730/57591 objects misplaced (6.477%); 5.5 MiB/s, 143 objects/s recovering 2024-08-19T14:39:04.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:03 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 12637/57591 objects degraded (21.943%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-19T14:39:04.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:03 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1849424330' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:39:04.266 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:04.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:04.267 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:39:04.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:03 smithi045 ceph-mon[20143]: pgmap v1673: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.2 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 541 op/s; 12637/57591 objects degraded (21.943%); 3730/57591 objects misplaced (6.477%); 5.5 MiB/s, 143 objects/s recovering 2024-08-19T14:39:04.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:03 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 12637/57591 objects degraded (21.943%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-19T14:39:04.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:03 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1849424330' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:39:04.368 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:39:04.392 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:04.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:04.393 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:39:04.494 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:39:04.541 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:04.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:04.543 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:39:04.645 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:39:04.648 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:39:04.648 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:39:04.648 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:39:04.648 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:39:04.648 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:39:04.648 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:39:04.648 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:39:04.648 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:39:04.648 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:39:04.648 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:39:04.648 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:39:04.648 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:39:04.649 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:39:04.649 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:39:04.649 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:39:04.649 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:39:04.649 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:39:04.649 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:39:04.649 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:39:04.649 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:39:04.649 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:39:04.649 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:39:04.649 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:39:04.669 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:04.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:04.670 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:39:04.771 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:39:04.793 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:04.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:04.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:39:04.895 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:39:04.917 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:04.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:04.918 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:39:05.018 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:39:05.060 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:05.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:05.061 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:39:05.162 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:39:05.184 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:05.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:05.185 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:39:05.286 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:39:05.307 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:05.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:05.309 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:39:05.410 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:39:05.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:05.456 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:05.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:39:05.557 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:39:05.579 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:05.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:05.580 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:39:05.681 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:39:05.712 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:05.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:05.713 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:39:05.814 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:39:05.835 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:05.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:05.836 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:39:05.937 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:39:05.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:05.984 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:05.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:39:06.086 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:39:06.126 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:06.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:06.127 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:39:06.228 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:39:06.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:05 smithi032 ceph-mon[20004]: pgmap v1674: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.1 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 531 op/s; 12694/55662 objects degraded (22.806%); 3615/55662 objects misplaced (6.495%); 5.5 MiB/s, 134 objects/s recovering 2024-08-19T14:39:06.263 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:06.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:06.264 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:39:06.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:05 smithi045 ceph-mon[20143]: pgmap v1674: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.1 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 531 op/s; 12694/55662 objects degraded (22.806%); 3615/55662 objects misplaced (6.495%); 5.5 MiB/s, 134 objects/s recovering 2024-08-19T14:39:06.365 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:39:06.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:05 smithi031 ceph-mon[16336]: pgmap v1674: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.1 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 531 op/s; 12694/55662 objects degraded (22.806%); 3615/55662 objects misplaced (6.495%); 5.5 MiB/s, 134 objects/s recovering 2024-08-19T14:39:06.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:06.391 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:06.392 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:39:06.494 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:39:06.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:06.533 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:06.533 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:39:06.634 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:39:06.677 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:06.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:06.678 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:39:06.778 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:39:06.799 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:06.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:06.800 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:39:06.902 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:39:06.923 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:06.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:06.924 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:39:07.025 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:39:07.045 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:07.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:07.046 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:39:07.147 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:39:07.170 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:07.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:07.171 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:39:07.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:06 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:39:07.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:06 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:39:07.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:06 smithi032 ceph-mon[20004]: osdmap e700: 12 total, 12 up, 10 in 2024-08-19T14:39:07.272 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:39:07.293 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:07.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:07.294 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:39:07.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:06 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:39:07.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:06 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:39:07.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:06 smithi045 ceph-mon[20143]: osdmap e700: 12 total, 12 up, 10 in 2024-08-19T14:39:07.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:06 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:39:07.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:06 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:39:07.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:06 smithi031 ceph-mon[16336]: osdmap e700: 12 total, 12 up, 10 in 2024-08-19T14:39:07.394 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:39:07.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:07.443 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:07.443 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:39:07.544 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:39:07.566 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:07.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:07.567 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:39:07.668 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:39:07.713 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:07.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:07.714 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:39:07.814 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:39:07.838 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:07.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:07.839 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:39:07.940 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:39:07.963 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:07.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:07.964 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:39:08.065 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:39:08.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:08.087 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:08.087 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:39:08.189 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:39:08.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:08.211 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:08.211 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:39:08.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:07 smithi032 ceph-mon[20004]: pgmap v1676: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.0 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 519 op/s; 12382/50748 objects degraded (24.399%); 3615/50748 objects misplaced (7.123%); 5.3 MiB/s, 113 objects/s recovering 2024-08-19T14:39:08.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:07 smithi045 ceph-mon[20143]: pgmap v1676: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.0 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 519 op/s; 12382/50748 objects degraded (24.399%); 3615/50748 objects misplaced (7.123%); 5.3 MiB/s, 113 objects/s recovering 2024-08-19T14:39:08.313 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:39:08.354 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:08.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:08.355 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:39:08.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:07 smithi031 ceph-mon[16336]: pgmap v1676: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.0 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 519 op/s; 12382/50748 objects degraded (24.399%); 3615/50748 objects misplaced (7.123%); 5.3 MiB/s, 113 objects/s recovering 2024-08-19T14:39:08.456 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:39:08.478 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:08.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:08.479 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:39:08.580 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:39:08.609 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:08.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:08.611 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:39:08.712 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:39:08.733 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:08.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:08.735 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:39:08.836 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:39:08.863 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:08.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:08.864 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:39:08.965 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:39:08.999 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:09.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:09.000 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:39:09.102 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:39:09.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:09.124 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:09.124 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:39:09.225 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:39:09.247 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:09.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:09.249 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:39:09.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:08 smithi032 ceph-mon[20004]: osdmap e701: 12 total, 12 up, 10 in 2024-08-19T14:39:09.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:08 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 12382/50748 objects degraded (24.399%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-19T14:39:09.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:08 smithi045 ceph-mon[20143]: osdmap e701: 12 total, 12 up, 10 in 2024-08-19T14:39:09.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:08 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 12382/50748 objects degraded (24.399%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-19T14:39:09.350 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:39:09.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:08 smithi031 ceph-mon[16336]: osdmap e701: 12 total, 12 up, 10 in 2024-08-19T14:39:09.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:08 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 12382/50748 objects degraded (24.399%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-19T14:39:09.419 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:09.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:09.424 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:39:09.525 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:39:09.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:09.547 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:09.547 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:39:09.649 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:39:09.672 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:09.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:09.673 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:39:09.774 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:39:09.795 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:09.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:09.796 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:39:09.897 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:39:09.920 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:09.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:09.921 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:39:10.022 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:39:10.058 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:10.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:10.059 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:39:10.160 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:39:10.182 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:10.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:10.183 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:39:10.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:09 smithi032 ceph-mon[20004]: osdmap e702: 12 total, 12 up, 10 in 2024-08-19T14:39:10.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:09 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:39:10.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:09 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:39:10.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:09 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:39:10.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:09 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:39:10.284 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:39:10.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:09 smithi045 ceph-mon[20143]: osdmap e702: 12 total, 12 up, 10 in 2024-08-19T14:39:10.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:09 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:39:10.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:09 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:39:10.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:09 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:39:10.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:09 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:39:10.306 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:10.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:10.307 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:39:10.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:09 smithi031 ceph-mon[16336]: osdmap e702: 12 total, 12 up, 10 in 2024-08-19T14:39:10.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:09 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:39:10.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:09 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:39:10.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:09 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:39:10.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:09 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:39:10.408 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:39:10.430 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:10.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:10.431 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:39:10.532 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:39:10.570 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:10.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:10.572 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:39:10.673 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:39:10.695 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:10.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:10.696 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:39:10.797 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:39:10.821 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:10.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:10.823 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:39:10.924 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:39:10.974 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:10.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:10.975 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:39:11.076 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:39:11.121 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:11.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:11.122 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:39:11.223 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:39:11.245 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:11.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:11.246 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:39:11.347 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:39:11.412 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:11.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:11.414 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:39:11.515 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:39:11.563 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:11.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:11.564 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:39:11.665 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:39:11.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:11.688 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:11.689 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:39:11.790 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:39:11.833 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:11.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:11.834 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:39:11.935 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:39:11.957 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:11.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:11.958 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:39:12.060 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:39:12.082 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:12.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:12.083 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:39:12.184 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:39:12.206 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:12.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:12.207 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:39:12.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:11 smithi032 ceph-mon[20004]: pgmap v1680: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 1002 MiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 420 op/s; 12097/48084 objects degraded (25.158%); 3480/48084 objects misplaced (7.237%); 4.6 MiB/s, 122 objects/s recovering 2024-08-19T14:39:12.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:11 smithi045 ceph-mon[20143]: pgmap v1680: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 1002 MiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 420 op/s; 12097/48084 objects degraded (25.158%); 3480/48084 objects misplaced (7.237%); 4.6 MiB/s, 122 objects/s recovering 2024-08-19T14:39:12.309 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:39:12.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:12.345 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:12.345 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:39:12.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:11 smithi031 ceph-mon[16336]: pgmap v1680: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 1002 MiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 420 op/s; 12097/48084 objects degraded (25.158%); 3480/48084 objects misplaced (7.237%); 4.6 MiB/s, 122 objects/s recovering 2024-08-19T14:39:12.446 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:39:12.470 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:12.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:12.471 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:39:12.573 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:39:12.614 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:12.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:12.615 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:39:12.716 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:39:12.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:12.738 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:12.738 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:39:12.839 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:39:12.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:12.864 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:12.864 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:39:12.965 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:39:12.980 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-08-19T14:39:13.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:13.020 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:13.020 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:39:13.121 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:39:13.167 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:13.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:13.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:39:13.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:12 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:39:13.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:12 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:39:13.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:12 smithi032 ceph-mon[20004]: osdmap e703: 12 total, 12 up, 10 in 2024-08-19T14:39:13.269 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:39:13.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:13.291 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:13.292 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:39:13.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:12 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:39:13.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:12 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:39:13.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:12 smithi045 ceph-mon[20143]: osdmap e703: 12 total, 12 up, 10 in 2024-08-19T14:39:13.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:12 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:39:13.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:12 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:39:13.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:12 smithi031 ceph-mon[16336]: osdmap e703: 12 total, 12 up, 10 in 2024-08-19T14:39:13.392 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:39:13.414 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:13.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:13.415 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:39:13.516 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:39:13.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:13.557 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:13.557 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:39:13.658 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:39:13.680 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:13.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:13.681 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:39:13.782 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:39:13.805 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:13.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:13.807 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:39:13.908 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:39:13.929 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:13.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:13.930 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:39:14.031 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:39:14.052 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:14.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:14.053 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:39:14.092 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-19T14:39:14.154 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:39:14.246 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:14.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:14.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:39:14.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:13 smithi032 ceph-mon[20004]: pgmap v1682: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 890 MiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 446 op/s; 11531/42702 objects degraded (27.003%); 3480/42702 objects misplaced (8.150%); 6.6 MiB/s, 165 objects/s recovering 2024-08-19T14:39:14.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:13 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 11531/42702 objects degraded (27.003%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-19T14:39:14.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:13 smithi045 ceph-mon[20143]: pgmap v1682: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 890 MiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 446 op/s; 11531/42702 objects degraded (27.003%); 3480/42702 objects misplaced (8.150%); 6.6 MiB/s, 165 objects/s recovering 2024-08-19T14:39:14.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:13 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 11531/42702 objects degraded (27.003%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-19T14:39:14.313 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:13 smithi031 ceph-mon[16336]: pgmap v1682: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 890 MiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 446 op/s; 11531/42702 objects degraded (27.003%); 3480/42702 objects misplaced (8.150%); 6.6 MiB/s, 165 objects/s recovering 2024-08-19T14:39:14.313 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:14 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 11531/42702 objects degraded (27.003%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-19T14:39:14.349 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:39:14.370 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:14.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:14.371 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:39:14.472 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:39:14.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:14.530 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:14.530 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:39:14.631 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:39:14.654 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:14.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:14.655 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:39:14.756 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:39:14.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:14.781 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:14.781 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:39:14.882 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:39:14.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:14.915 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:14.915 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:39:15.016 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:39:15.038 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:15.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:15.039 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:39:15.140 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:39:15.162 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:15.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:15.163 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:39:15.264 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:39:15.286 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:15.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:15.287 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:39:15.388 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:39:15.434 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:15.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:15.435 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:39:15.536 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:39:15.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:15.567 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:15.567 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:39:15.668 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:39:15.710 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:15.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:15.711 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:39:15.812 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:39:15.834 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:15.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:15.835 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:39:15.936 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:39:15.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:15.994 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:15.994 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:39:16.095 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:39:16.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:16.118 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:16.118 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:39:16.219 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:39:16.243 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:16.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:16.245 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:39:16.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:15 smithi032 ceph-mon[20004]: pgmap v1683: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 850 MiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 410 op/s; 11553/40791 objects degraded (28.322%); 3348/40791 objects misplaced (8.208%); 6.3 MiB/s, 110 objects/s recovering 2024-08-19T14:39:16.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:15 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:39:16.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:15 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:39:16.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:15 smithi032 ceph-mon[20004]: osdmap e704: 12 total, 12 up, 10 in 2024-08-19T14:39:16.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:15 smithi045 ceph-mon[20143]: pgmap v1683: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 850 MiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 410 op/s; 11553/40791 objects degraded (28.322%); 3348/40791 objects misplaced (8.208%); 6.3 MiB/s, 110 objects/s recovering 2024-08-19T14:39:16.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:15 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:39:16.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:15 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:39:16.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:15 smithi045 ceph-mon[20143]: osdmap e704: 12 total, 12 up, 10 in 2024-08-19T14:39:16.346 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:39:16.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:15 smithi031 ceph-mon[16336]: pgmap v1683: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 850 MiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 410 op/s; 11553/40791 objects degraded (28.322%); 3348/40791 objects misplaced (8.208%); 6.3 MiB/s, 110 objects/s recovering 2024-08-19T14:39:16.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:15 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:39:16.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:15 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:39:16.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:15 smithi031 ceph-mon[16336]: osdmap e704: 12 total, 12 up, 10 in 2024-08-19T14:39:16.369 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:16.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:16.370 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:39:16.471 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:39:16.493 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:16.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:16.494 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:39:16.595 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:39:16.616 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:16.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:16.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:39:16.719 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:39:16.742 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:16.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:16.743 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:39:16.845 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:39:16.866 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:16.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:16.867 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:39:16.968 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:39:16.992 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:16.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:16.993 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:39:17.094 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:39:17.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:17.117 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:17.117 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:39:17.218 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:39:17.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:17 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:39:17.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:17 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:39:17.260 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:17.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:17.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:39:17.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:17 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:39:17.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:17 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:39:17.363 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:39:17.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:17 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:39:17.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:17 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:39:17.384 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:17.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:17.385 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:39:17.486 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:39:17.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:17.562 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:17.562 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:39:17.663 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:39:17.685 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:17.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:17.686 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:39:17.787 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:39:17.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:17.809 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:17.810 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:39:17.911 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:39:17.933 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:17.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:17.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:39:18.034 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:39:18.056 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:18.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:18.057 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:39:18.160 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:39:18.183 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:18.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:18.185 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:39:18.286 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:39:18.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:18 smithi045 ceph-mon[20143]: pgmap v1685: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 740 MiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 575 op/s; 11165/35493 objects degraded (31.457%); 3348/35493 objects misplaced (9.433%); 8.7 MiB/s, 148 objects/s recovering 2024-08-19T14:39:18.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:18 smithi045 ceph-mon[20143]: osdmap e705: 12 total, 12 up, 10 in 2024-08-19T14:39:18.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:18 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 11165/35493 objects degraded (31.457%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-19T14:39:18.310 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:18.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:18.311 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:39:18.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:18 smithi031 ceph-mon[16336]: pgmap v1685: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 740 MiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 575 op/s; 11165/35493 objects degraded (31.457%); 3348/35493 objects misplaced (9.433%); 8.7 MiB/s, 148 objects/s recovering 2024-08-19T14:39:18.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:18 smithi031 ceph-mon[16336]: osdmap e705: 12 total, 12 up, 10 in 2024-08-19T14:39:18.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:18 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 11165/35493 objects degraded (31.457%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-19T14:39:18.412 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:39:18.436 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:18.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:18.437 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:39:18.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:18 smithi032 ceph-mon[20004]: pgmap v1685: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 740 MiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 575 op/s; 11165/35493 objects degraded (31.457%); 3348/35493 objects misplaced (9.433%); 8.7 MiB/s, 148 objects/s recovering 2024-08-19T14:39:18.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:18 smithi032 ceph-mon[20004]: osdmap e705: 12 total, 12 up, 10 in 2024-08-19T14:39:18.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:18 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 11165/35493 objects degraded (31.457%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-19T14:39:18.524 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:39:18.538 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:39:18.561 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:18.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:18.562 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:39:18.663 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:39:18.716 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:18.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:18.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:39:18.818 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:39:18.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:18.868 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:18.868 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:39:18.969 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:39:18.991 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:18.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:18.992 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:39:19.093 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:39:19.137 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:19.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:19.138 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:39:19.239 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:39:19.260 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:19.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:19.261 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:39:19.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:19 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:39:19.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:19 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:39:19.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:19 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:39:19.362 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:39:19.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:19 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:39:19.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:19 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:39:19.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:19 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:39:19.413 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:19.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:19.414 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:39:19.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:19 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:39:19.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:19 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:39:19.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:19 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:39:19.514 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:39:19.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:19.537 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:19.538 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:39:19.638 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:39:19.689 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:19.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:19.690 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:39:19.791 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:39:19.837 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:19.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:19.838 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:39:19.939 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:39:19.979 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:19.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:19.980 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:39:20.081 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:39:20.124 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:20.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:20.125 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:39:20.225 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:39:20.226 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:39:20.227 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:39:20.248 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:20.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:20.249 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:39:20.252 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:20 smithi031 ceph-mon[16336]: pgmap v1687: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 690 MiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 509 op/s; 11184/33099 objects degraded (33.790%); 3211/33099 objects misplaced (9.701%); 6.5 MiB/s, 104 objects/s recovering 2024-08-19T14:39:20.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:20 smithi045 ceph-mon[20143]: pgmap v1687: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 690 MiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 509 op/s; 11184/33099 objects degraded (33.790%); 3211/33099 objects misplaced (9.701%); 6.5 MiB/s, 104 objects/s recovering 2024-08-19T14:39:20.349 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:39:20.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:20.372 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:20.373 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:39:20.474 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:39:20.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:20 smithi032 ceph-mon[20004]: pgmap v1687: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 690 MiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 509 op/s; 11184/33099 objects degraded (33.790%); 3211/33099 objects misplaced (9.701%); 6.5 MiB/s, 104 objects/s recovering 2024-08-19T14:39:20.523 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:39:20.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:20.530 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:20.531 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:39:20.632 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:39:20.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:20.716 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:20.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:39:20.818 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:39:20.841 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:20.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:20.843 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:39:20.944 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:39:20.965 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:20.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:20.966 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:39:21.067 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:39:21.088 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:21.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:21.089 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:39:21.190 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:39:21.212 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:21.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:21.214 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:39:21.315 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:39:21.336 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:21.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:21.337 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:39:21.438 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:39:21.486 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:21.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:21.487 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:39:21.588 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:39:21.609 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:21.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:21.610 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:39:21.711 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:39:21.733 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:21.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:21.734 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:39:21.835 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:39:21.857 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:21.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:21.858 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:39:21.960 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:39:21.983 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:21.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:21.984 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:39:22.085 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:39:22.108 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:22.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:22.109 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:39:22.210 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:39:22.233 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:22.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:22.234 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:39:22.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:22 smithi045 ceph-mon[20143]: pgmap v1688: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 680 MiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 419 op/s; 11184/32625 objects degraded (34.280%); 3211/32625 objects misplaced (9.842%); 5.1 MiB/s, 82 objects/s recovering 2024-08-19T14:39:22.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:22 smithi045 ceph-mon[20143]: from='client.26863 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:39:22.335 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:39:22.357 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:22.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:22.358 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:39:22.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:22 smithi031 ceph-mon[16336]: pgmap v1688: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 680 MiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 419 op/s; 11184/32625 objects degraded (34.280%); 3211/32625 objects misplaced (9.842%); 5.1 MiB/s, 82 objects/s recovering 2024-08-19T14:39:22.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:22 smithi031 ceph-mon[16336]: from='client.26863 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:39:22.459 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:39:22.480 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:22.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:22.482 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:39:22.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:22 smithi032 ceph-mon[20004]: pgmap v1688: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 680 MiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 419 op/s; 11184/32625 objects degraded (34.280%); 3211/32625 objects misplaced (9.842%); 5.1 MiB/s, 82 objects/s recovering 2024-08-19T14:39:22.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:22 smithi032 ceph-mon[20004]: from='client.26863 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:39:22.583 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:39:22.604 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:22.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:22.605 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:39:22.652 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:39:22.653 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-08-19T14:39:22.706 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:39:22.742 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:22.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:22.743 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:39:22.844 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:39:22.868 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:22.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:22.869 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:39:22.970 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:39:22.991 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:22.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:22.992 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:39:23.093 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:39:23.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:23.116 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:23.116 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:39:23.217 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:39:23.272 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:23.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:23.273 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:39:23.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:23 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:39:23.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:23 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:39:23.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:23 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 11184/32625 objects degraded (34.280%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-19T14:39:23.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:23 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:39:23.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:23 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:39:23.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:23 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 11184/32625 objects degraded (34.280%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-19T14:39:23.375 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:39:23.409 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:23 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:39:23.409 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:23 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:39:23.410 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:23 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 11184/32625 objects degraded (34.280%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-19T14:39:23.414 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:23.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:23.415 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:39:23.516 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:39:23.566 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:23.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:23.567 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:39:23.669 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:39:23.690 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:23.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:23.691 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:39:23.792 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:39:23.813 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:23.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:23.814 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:39:23.915 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:39:23.938 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:23.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:23.939 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:39:24.040 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:39:24.061 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:24.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:24.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:39:24.163 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:39:24.204 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:24.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:24.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:39:24.245 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":1688,"stamp":"2024-08-19T14:39:20.263942+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":713097784,"num_objects":10875,"num_object_clones":0,"num_object_copies":32625,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11184,"num_objects_misplaced":3211,"num_objects_unfound":0,"num_objects_dirty":10875,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":225443,"num_write_kb":7562049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":50218,"num_bytes_recovered":3062436384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":165080,"ondisk_log_size":165080,"up":141,"acting":137,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":142,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":10673328,"kb_used_data":2038988,"kb_used_omap":18,"kb_used_meta":8634029,"kb_avail":926778192,"statfs":{"total":959950356480,"available":949020868608,"internally_reserved":0,"allocated":2087923712,"data_stored":2056816008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18760,"internal_metadata":8841246392},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,3,7,8,10,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-220135424,"num_objects":-3359,"num_object_clones":0,"num_object_copies":-10077,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-347,"num_objects_misplaced":-269,"num_objects_unfound":0,"num_objects_dirty":-3359,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3359,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":657,"num_bytes_recovered":42926080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002436"},"pg_stats":[{"pgid":"7.6","version":"704'14376","reported_seq":14696,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-08-19T14:39:15.690494+0000","last_change":"2024-08-19T14:37:56.236536+0000","last_active":"2024-08-19T14:39:15.690494+0000","last_peered":"2024-08-19T14:39:15.690494+0000","last_clean":"2024-08-19T14:39:15.690494+0000","last_became_active":"2024-08-19T14:33:28.334317+0000","last_became_peered":"2024-08-19T14:33:28.334317+0000","last_unstale":"2024-08-19T14:39:15.690494+0000","last_undegraded":"2024-08-19T14:39:15.690494+0000","last_fullsized":"2024-08-19T14:39:15.690494+0000","mapping_epoch":600,"log_start":"614'4300","ondisk_log_start":"614'4300","created":600,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"684'11923","last_scrub_stamp":"2024-08-19T14:37:56.236394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:37:56.236394+0000","objects_scrubbed":3155,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:39:18.162476+0000","scrub_duration":1586,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44433408,"num_objects":678,"num_object_clones":0,"num_object_copies":2034,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":678,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14376,"num_write_kb":481728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"70'139","reported_seq":1727,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-08-19T14:39:15.005595+0000","last_change":"2024-08-19T14:37:57.701402+0000","last_active":"2024-08-19T14:39:15.005595+0000","last_peered":"2024-08-19T14:39:15.005595+0000","last_clean":"2024-08-19T14:39:15.005595+0000","last_became_active":"2024-08-19T14:35:13.950224+0000","last_became_peered":"2024-08-19T14:35:13.950224+0000","last_unstale":"2024-08-19T14:39:15.005595+0000","last_undegraded":"2024-08-19T14:39:15.005595+0000","last_fullsized":"2024-08-19T14:39:15.005595+0000","mapping_epoch":637,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":638,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:37:57.701290+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:19:58.072083+0000","last_clean_scrub_stamp":"2024-08-19T14:37:57.701290+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:39:18.431910+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,9],"acting":[0,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.5","version":"704'13995","reported_seq":23494,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-08-19T14:39:16.706112+0000","last_change":"2024-08-19T14:35:23.022361+0000","last_active":"2024-08-19T14:39:16.706112+0000","last_peered":"2024-08-19T14:39:16.706112+0000","last_clean":"2024-08-19T14:39:16.706112+0000","last_became_active":"2024-08-19T14:35:23.009238+0000","last_became_peered":"2024-08-19T14:35:23.009238+0000","last_unstale":"2024-08-19T14:39:16.706112+0000","last_undegraded":"2024-08-19T14:39:16.706112+0000","last_fullsized":"2024-08-19T14:39:16.706112+0000","mapping_epoch":641,"log_start":"614'3900","ondisk_log_start":"614'3900","created":600,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45023232,"num_objects":687,"num_object_clones":0,"num_object_copies":2061,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13995,"num_write_kb":469824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4634,"num_bytes_recovered":303693824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,0],"acting":[11,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.3","version":"70'139","reported_seq":1573,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-08-19T14:39:17.029333+0000","last_change":"2024-08-19T14:33:18.804321+0000","last_active":"2024-08-19T14:39:17.029333+0000","last_peered":"2024-08-19T14:39:17.029333+0000","last_clean":"2024-08-19T14:39:17.029333+0000","last_became_active":"2024-08-19T14:18:13.804721+0000","last_became_peered":"2024-08-19T14:18:13.804721+0000","last_unstale":"2024-08-19T14:39:17.029333+0000","last_undegraded":"2024-08-19T14:39:17.029333+0000","last_fullsized":"2024-08-19T14:39:17.029333+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:33:18.804228+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:33:18.804228+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.4","version":"704'14244","reported_seq":14562,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-08-19T14:39:15.430931+0000","last_change":"2024-08-19T14:38:06.717577+0000","last_active":"2024-08-19T14:39:15.430931+0000","last_peered":"2024-08-19T14:39:15.430931+0000","last_clean":"2024-08-19T14:39:15.430931+0000","last_became_active":"2024-08-19T14:33:28.334446+0000","last_became_peered":"2024-08-19T14:33:28.334446+0000","last_unstale":"2024-08-19T14:39:15.430931+0000","last_undegraded":"2024-08-19T14:39:15.430931+0000","last_fullsized":"2024-08-19T14:39:15.430931+0000","mapping_epoch":600,"log_start":"614'4200","ondisk_log_start":"614'4200","created":600,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"686'12159","last_scrub_stamp":"2024-08-19T14:38:06.717004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:38:06.717004+0000","objects_scrubbed":2792,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:39:16.656979+0000","scrub_duration":1017,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46006272,"num_objects":702,"num_object_clones":0,"num_object_copies":2106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":702,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14244,"num_write_kb":478272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"70'139","reported_seq":1527,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-08-19T14:39:15.012007+0000","last_change":"2024-08-19T14:34:55.580212+0000","last_active":"2024-08-19T14:39:15.012007+0000","last_peered":"2024-08-19T14:39:15.012007+0000","last_clean":"2024-08-19T14:39:15.012007+0000","last_became_active":"2024-08-19T14:20:50.742354+0000","last_became_peered":"2024-08-19T14:20:50.742354+0000","last_unstale":"2024-08-19T14:39:15.012007+0000","last_undegraded":"2024-08-19T14:39:15.012007+0000","last_fullsized":"2024-08-19T14:39:15.012007+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:34:55.580141+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:19:59.302931+0000","last_clean_scrub_stamp":"2024-08-19T14:34:55.580141+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"704'13782","reported_seq":14102,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-08-19T14:39:15.255433+0000","last_change":"2024-08-19T14:37:43.229201+0000","last_active":"2024-08-19T14:39:15.255433+0000","last_peered":"2024-08-19T14:39:15.255433+0000","last_clean":"2024-08-19T14:39:15.255433+0000","last_became_active":"2024-08-19T14:33:28.331833+0000","last_became_peered":"2024-08-19T14:33:28.331833+0000","last_unstale":"2024-08-19T14:39:15.255433+0000","last_undegraded":"2024-08-19T14:39:15.255433+0000","last_fullsized":"2024-08-19T14:39:15.255433+0000","mapping_epoch":600,"log_start":"612'3700","ondisk_log_start":"612'3700","created":600,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"678'11114","last_scrub_stamp":"2024-08-19T14:37:43.229072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:37:43.229072+0000","objects_scrubbed":3413,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":910,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48234496,"num_objects":736,"num_object_clones":0,"num_object_copies":2208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":736,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13782,"num_write_kb":464576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,1],"acting":[8,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1","version":"70'139","reported_seq":825,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-08-19T14:39:15.006005+0000","last_change":"2024-08-19T14:37:42.630047+0000","last_active":"2024-08-19T14:39:15.006005+0000","last_peered":"2024-08-19T14:39:15.006005+0000","last_clean":"2024-08-19T14:39:15.006005+0000","last_became_active":"2024-08-19T14:35:12.530566+0000","last_became_peered":"2024-08-19T14:35:12.530566+0000","last_unstale":"2024-08-19T14:39:15.006005+0000","last_undegraded":"2024-08-19T14:39:15.006005+0000","last_fullsized":"2024-08-19T14:39:15.006005+0000","mapping_epoch":635,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":636,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:37:42.629919+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:37:42.629919+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:39:16.127348+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,2],"acting":[0,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"705'13985","reported_seq":14284,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-08-19T14:39:18.696640+0000","last_change":"2024-08-19T14:34:52.869444+0000","last_active":"2024-08-19T14:39:18.696640+0000","last_peered":"2024-08-19T14:39:18.696640+0000","last_clean":"2024-08-19T14:39:18.696640+0000","last_became_active":"2024-08-19T14:33:28.332994+0000","last_became_peered":"2024-08-19T14:33:28.332994+0000","last_unstale":"2024-08-19T14:39:18.696640+0000","last_undegraded":"2024-08-19T14:39:18.696640+0000","last_fullsized":"2024-08-19T14:39:18.696640+0000","mapping_epoch":600,"log_start":"613'3900","ondisk_log_start":"613'3900","created":600,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"627'6737","last_scrub_stamp":"2024-08-19T14:34:52.869233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:34:52.869233+0000","objects_scrubbed":6705,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1588,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38862848,"num_objects":593,"num_object_clones":0,"num_object_copies":1779,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13985,"num_write_kb":466496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"70'139","reported_seq":1229,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-08-19T14:39:15.084329+0000","last_change":"2024-08-19T14:35:11.537368+0000","last_active":"2024-08-19T14:39:15.084329+0000","last_peered":"2024-08-19T14:39:15.084329+0000","last_clean":"2024-08-19T14:39:15.084329+0000","last_became_active":"2024-08-19T14:35:11.536828+0000","last_became_peered":"2024-08-19T14:35:11.536828+0000","last_unstale":"2024-08-19T14:39:15.084329+0000","last_undegraded":"2024-08-19T14:39:15.084329+0000","last_fullsized":"2024-08-19T14:39:15.084329+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:27.536928+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:21:46.057624+0000","last_clean_scrub_stamp":"2024-08-19T14:28:27.536928+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"704'13826","reported_seq":23278,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-08-19T14:39:15.475129+0000","last_change":"2024-08-19T14:36:15.120872+0000","last_active":"2024-08-19T14:39:15.475129+0000","last_peered":"2024-08-19T14:39:15.475129+0000","last_clean":"2024-08-19T14:39:15.475129+0000","last_became_active":"2024-08-19T14:36:15.096070+0000","last_became_peered":"2024-08-19T14:36:15.096070+0000","last_unstale":"2024-08-19T14:39:15.475129+0000","last_undegraded":"2024-08-19T14:39:15.475129+0000","last_fullsized":"2024-08-19T14:39:15.475129+0000","mapping_epoch":653,"log_start":"613'3800","ondisk_log_start":"613'3800","created":600,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48103424,"num_objects":734,"num_object_clones":0,"num_object_copies":2202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":734,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13826,"num_write_kb":465920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4611,"num_bytes_recovered":302186496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,9],"acting":[3,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"70'139","reported_seq":797,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-08-19T14:39:15.000697+0000","last_change":"2024-08-19T14:37:45.357396+0000","last_active":"2024-08-19T14:39:15.000697+0000","last_peered":"2024-08-19T14:39:15.000697+0000","last_clean":"2024-08-19T14:39:15.000697+0000","last_became_active":"2024-08-19T14:34:15.717054+0000","last_became_peered":"2024-08-19T14:34:15.717054+0000","last_unstale":"2024-08-19T14:39:15.000697+0000","last_undegraded":"2024-08-19T14:39:15.000697+0000","last_fullsized":"2024-08-19T14:39:15.000697+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:37:45.357293+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:27.848870+0000","last_clean_scrub_stamp":"2024-08-19T14:37:45.357293+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:39:15.366166+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"7.3","version":"704'14344","reported_seq":23241,"reported_epoch":704,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-19T14:39:15.468333+0000","last_change":"2024-08-19T14:35:13.136988+0000","last_active":"2024-08-19T14:39:15.468333+0000","last_peered":"2024-08-19T14:39:15.468333+0000","last_clean":"2024-08-19T14:33:27.312789+0000","last_became_active":"2024-08-19T14:35:13.021078+0000","last_became_peered":"2024-08-19T14:35:13.021078+0000","last_unstale":"2024-08-19T14:39:15.468333+0000","last_undegraded":"2024-08-19T14:35:12.592288+0000","last_fullsized":"2024-08-19T14:35:12.526971+0000","mapping_epoch":635,"log_start":"614'4300","ondisk_log_start":"614'4300","created":600,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46530560,"num_objects":710,"num_object_clones":0,"num_object_copies":2130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2920,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14344,"num_write_kb":481728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4338,"num_bytes_recovered":280952832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":-2210}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"70'139","reported_seq":752,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-08-19T14:39:15.007756+0000","last_change":"2024-08-19T14:35:11.533207+0000","last_active":"2024-08-19T14:39:15.007756+0000","last_peered":"2024-08-19T14:39:15.007756+0000","last_clean":"2024-08-19T14:39:15.007756+0000","last_became_active":"2024-08-19T14:35:11.532858+0000","last_became_peered":"2024-08-19T14:35:11.532858+0000","last_unstale":"2024-08-19T14:39:15.007756+0000","last_undegraded":"2024-08-19T14:39:15.007756+0000","last_fullsized":"2024-08-19T14:39:15.007756+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:27.331041+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:28:27.331041+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,0],"acting":[11,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.2","version":"704'14217","reported_seq":22965,"reported_epoch":704,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-19T14:39:15.462378+0000","last_change":"2024-08-19T14:35:13.121204+0000","last_active":"2024-08-19T14:39:15.462378+0000","last_peered":"2024-08-19T14:39:15.462378+0000","last_clean":"2024-08-19T14:33:27.312789+0000","last_became_active":"2024-08-19T14:35:13.013347+0000","last_became_peered":"2024-08-19T14:35:13.013347+0000","last_unstale":"2024-08-19T14:39:15.462378+0000","last_undegraded":"2024-08-19T14:35:12.598259+0000","last_fullsized":"2024-08-19T14:35:12.526602+0000","mapping_epoch":635,"log_start":"614'4200","ondisk_log_start":"614'4200","created":600,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50003968,"num_objects":763,"num_object_clones":0,"num_object_copies":2289,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2971,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":763,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14217,"num_write_kb":479360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4264,"num_bytes_recovered":276430848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":-2208}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"70'139","reported_seq":629,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-08-19T14:39:15.007729+0000","last_change":"2024-08-19T14:33:16.760928+0000","last_active":"2024-08-19T14:39:15.007729+0000","last_peered":"2024-08-19T14:39:15.007729+0000","last_clean":"2024-08-19T14:39:15.007729+0000","last_became_active":"2024-08-19T14:30:47.909067+0000","last_became_peered":"2024-08-19T14:30:47.909067+0000","last_unstale":"2024-08-19T14:39:15.007729+0000","last_undegraded":"2024-08-19T14:39:15.007729+0000","last_fullsized":"2024-08-19T14:39:15.007729+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:33:16.760810+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:33:16.760810+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.c","version":"704'13887","reported_seq":22803,"reported_epoch":704,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-19T14:39:15.460059+0000","last_change":"2024-08-19T14:35:13.133729+0000","last_active":"2024-08-19T14:39:15.460059+0000","last_peered":"2024-08-19T14:39:15.460059+0000","last_clean":"2024-08-19T14:33:27.312789+0000","last_became_active":"2024-08-19T14:35:13.003266+0000","last_became_peered":"2024-08-19T14:35:13.003266+0000","last_unstale":"2024-08-19T14:39:15.460059+0000","last_undegraded":"2024-08-19T14:35:12.593502+0000","last_fullsized":"2024-08-19T14:35:12.526375+0000","mapping_epoch":635,"log_start":"613'3800","ondisk_log_start":"613'3800","created":600,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43843584,"num_objects":669,"num_object_clones":0,"num_object_copies":2007,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2661,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":669,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13887,"num_write_kb":465792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4348,"num_bytes_recovered":283181056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,2],"acting":[3,11],"avail_no_missing":["3","11"],"object_location_counts":[{"shards":"3,11","objects":-1992}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"70'139","reported_seq":796,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-08-19T14:39:17.007786+0000","last_change":"2024-08-19T14:34:50.256878+0000","last_active":"2024-08-19T14:39:17.007786+0000","last_peered":"2024-08-19T14:39:17.007786+0000","last_clean":"2024-08-19T14:39:17.007786+0000","last_became_active":"2024-08-19T14:34:15.717472+0000","last_became_peered":"2024-08-19T14:34:15.717472+0000","last_unstale":"2024-08-19T14:39:17.007786+0000","last_undegraded":"2024-08-19T14:39:17.007786+0000","last_fullsized":"2024-08-19T14:39:17.007786+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:34:50.256808+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:28.864870+0000","last_clean_scrub_stamp":"2024-08-19T14:34:50.256808+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,0],"acting":[4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"704'17506","reported_seq":27353,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-08-19T14:39:15.688795+0000","last_change":"2024-08-19T14:38:47.357628+0000","last_active":"2024-08-19T14:39:15.688795+0000","last_peered":"2024-08-19T14:39:15.688795+0000","last_clean":"2024-08-19T14:39:15.688795+0000","last_became_active":"2024-08-19T14:36:14.098166+0000","last_became_peered":"2024-08-19T14:36:14.098166+0000","last_unstale":"2024-08-19T14:39:15.688795+0000","last_undegraded":"2024-08-19T14:39:15.688795+0000","last_fullsized":"2024-08-19T14:39:15.688795+0000","mapping_epoch":652,"log_start":"633'7502","ondisk_log_start":"633'7502","created":600,"last_epoch_clean":653,"parent":"0.0","parent_split_bits":0,"last_scrub":"696'16654","last_scrub_stamp":"2024-08-19T14:38:47.357047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:38:47.357047+0000","objects_scrubbed":1549,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:40:09.053409+0000","scrub_duration":564,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45219840,"num_objects":690,"num_object_clones":0,"num_object_copies":2070,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":690,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13876,"num_write_kb":466112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4700,"num_bytes_recovered":308019200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.9","version":"70'139","reported_seq":1730,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-08-19T14:39:15.011856+0000","last_change":"2024-08-19T14:35:12.529993+0000","last_active":"2024-08-19T14:39:15.011856+0000","last_peered":"2024-08-19T14:39:15.011856+0000","last_clean":"2024-08-19T14:39:15.011856+0000","last_became_active":"2024-08-19T14:35:12.529628+0000","last_became_peered":"2024-08-19T14:35:12.529628+0000","last_unstale":"2024-08-19T14:39:15.011856+0000","last_undegraded":"2024-08-19T14:39:15.011856+0000","last_fullsized":"2024-08-19T14:39:15.011856+0000","mapping_epoch":635,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":636,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:34:51.561893+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:34:51.561893+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.e","version":"705'14203","reported_seq":28740,"reported_epoch":705,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-19T14:39:17.897484+0000","last_change":"2024-08-19T14:35:11.863818+0000","last_active":"2024-08-19T14:39:17.897484+0000","last_peered":"2024-08-19T14:39:17.897484+0000","last_clean":"2024-08-19T14:33:27.312789+0000","last_became_active":"2024-08-19T14:35:11.806304+0000","last_became_peered":"2024-08-19T14:35:11.806304+0000","last_unstale":"2024-08-19T14:39:17.897484+0000","last_undegraded":"2024-08-19T14:35:11.779866+0000","last_fullsized":"2024-08-19T14:35:11.528207+0000","mapping_epoch":634,"log_start":"614'4200","ondisk_log_start":"614'4200","created":600,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38862848,"num_objects":593,"num_object_clones":0,"num_object_copies":1779,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2632,"num_objects_misplaced":3211,"num_objects_unfound":0,"num_objects_dirty":593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14203,"num_write_kb":473472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5541,"num_bytes_recovered":308412416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,5],"acting":[1,5],"avail_no_missing":["1","5","9"],"object_location_counts":[{"shards":"1,5","objects":-2618},{"shards":"1,5,9","objects":579}],"blocked_by":[],"up_primary":2,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"70'139","reported_seq":760,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-08-19T14:39:14.998205+0000","last_change":"2024-08-19T14:35:11.537896+0000","last_active":"2024-08-19T14:39:14.998205+0000","last_peered":"2024-08-19T14:39:14.998205+0000","last_clean":"2024-08-19T14:39:14.998205+0000","last_became_active":"2024-08-19T14:35:11.537474+0000","last_became_peered":"2024-08-19T14:35:11.537474+0000","last_unstale":"2024-08-19T14:39:14.998205+0000","last_undegraded":"2024-08-19T14:39:14.998205+0000","last_fullsized":"2024-08-19T14:39:14.998205+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:59.829567+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:28:59.829567+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"705'14128","reported_seq":14351,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-08-19T14:39:17.877549+0000","last_change":"2024-08-19T14:33:28.335586+0000","last_active":"2024-08-19T14:39:17.877549+0000","last_peered":"2024-08-19T14:39:17.877549+0000","last_clean":"2024-08-19T14:39:17.877549+0000","last_became_active":"2024-08-19T14:33:28.335298+0000","last_became_peered":"2024-08-19T14:33:28.335298+0000","last_unstale":"2024-08-19T14:39:17.877549+0000","last_undegraded":"2024-08-19T14:39:17.877549+0000","last_fullsized":"2024-08-19T14:39:17.877549+0000","mapping_epoch":600,"log_start":"614'4100","ondisk_log_start":"614'4100","created":600,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39125016,"num_objects":598,"num_object_clones":0,"num_object_copies":1794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":598,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14128,"num_write_kb":471169,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"70'139","reported_seq":1080,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-08-19T14:39:15.007608+0000","last_change":"2024-08-19T14:33:15.753668+0000","last_active":"2024-08-19T14:39:15.007608+0000","last_peered":"2024-08-19T14:39:15.007608+0000","last_clean":"2024-08-19T14:39:15.007608+0000","last_became_active":"2024-08-19T14:30:47.941200+0000","last_became_peered":"2024-08-19T14:30:47.941200+0000","last_unstale":"2024-08-19T14:39:15.007608+0000","last_undegraded":"2024-08-19T14:39:15.007608+0000","last_fullsized":"2024-08-19T14:39:15.007608+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:33:15.753595+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:33:15.753595+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.8","version":"704'14372","reported_seq":29706,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-08-19T14:39:15.415599+0000","last_change":"2024-08-19T14:38:13.418089+0000","last_active":"2024-08-19T14:39:15.415599+0000","last_peered":"2024-08-19T14:39:15.415599+0000","last_clean":"2024-08-19T14:39:15.415599+0000","last_became_active":"2024-08-19T14:38:13.389656+0000","last_became_peered":"2024-08-19T14:38:13.389656+0000","last_unstale":"2024-08-19T14:39:15.415599+0000","last_undegraded":"2024-08-19T14:39:15.415599+0000","last_fullsized":"2024-08-19T14:39:15.415599+0000","mapping_epoch":688,"log_start":"614'4300","ondisk_log_start":"614'4300","created":600,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48365568,"num_objects":738,"num_object_clones":0,"num_object_copies":2214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":738,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14372,"num_write_kb":483520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7555,"num_bytes_recovered":476577792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.e","version":"70'139","reported_seq":1273,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-08-19T14:39:17.007890+0000","last_change":"2024-08-19T14:34:55.333177+0000","last_active":"2024-08-19T14:39:17.007890+0000","last_peered":"2024-08-19T14:39:17.007890+0000","last_clean":"2024-08-19T14:39:17.007890+0000","last_became_active":"2024-08-19T14:18:26.884105+0000","last_became_peered":"2024-08-19T14:18:26.884105+0000","last_unstale":"2024-08-19T14:39:17.007890+0000","last_undegraded":"2024-08-19T14:39:17.007890+0000","last_fullsized":"2024-08-19T14:39:17.007890+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:34:55.333065+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:32:31.196288+0000","last_clean_scrub_stamp":"2024-08-19T14:34:55.333065+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"704'14154","reported_seq":14510,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-08-19T14:39:15.452912+0000","last_change":"2024-08-19T14:37:54.857799+0000","last_active":"2024-08-19T14:39:15.452912+0000","last_peered":"2024-08-19T14:39:15.452912+0000","last_clean":"2024-08-19T14:39:15.452912+0000","last_became_active":"2024-08-19T14:33:28.334547+0000","last_became_peered":"2024-08-19T14:33:28.334547+0000","last_unstale":"2024-08-19T14:39:15.452912+0000","last_undegraded":"2024-08-19T14:39:15.452912+0000","last_fullsized":"2024-08-19T14:39:15.452912+0000","mapping_epoch":600,"log_start":"614'4100","ondisk_log_start":"614'4100","created":600,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"683'11737","last_scrub_stamp":"2024-08-19T14:37:54.851356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:37:54.851356+0000","objects_scrubbed":3161,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:39:20.130518+0000","scrub_duration":1254,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47579136,"num_objects":726,"num_object_clones":0,"num_object_copies":2178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":726,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14154,"num_write_kb":476160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,8],"acting":[0,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"70'139","reported_seq":1025,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-08-19T14:39:17.007890+0000","last_change":"2024-08-19T14:34:12.199530+0000","last_active":"2024-08-19T14:39:17.007890+0000","last_peered":"2024-08-19T14:39:17.007890+0000","last_clean":"2024-08-19T14:39:17.007890+0000","last_became_active":"2024-08-19T14:27:31.837754+0000","last_became_peered":"2024-08-19T14:27:31.837754+0000","last_unstale":"2024-08-19T14:39:17.007890+0000","last_undegraded":"2024-08-19T14:39:17.007890+0000","last_fullsized":"2024-08-19T14:39:17.007890+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:34:12.199449+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:34:12.199449+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.a","version":"704'14052","reported_seq":36587,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-08-19T14:39:16.699146+0000","last_change":"2024-08-19T14:39:09.001385+0000","last_active":"2024-08-19T14:39:16.699146+0000","last_peered":"2024-08-19T14:39:16.699146+0000","last_clean":"2024-08-19T14:39:16.699146+0000","last_became_active":"2024-08-19T14:39:08.973126+0000","last_became_peered":"2024-08-19T14:39:08.973126+0000","last_unstale":"2024-08-19T14:39:16.699146+0000","last_undegraded":"2024-08-19T14:39:16.699146+0000","last_fullsized":"2024-08-19T14:39:16.699146+0000","mapping_epoch":701,"log_start":"614'4000","ondisk_log_start":"614'4000","created":600,"last_epoch_clean":702,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44171264,"num_objects":674,"num_object_clones":0,"num_object_copies":2022,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":674,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14052,"num_write_kb":471232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10224,"num_bytes_recovered":522125312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,2],"acting":[11,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.c","version":"70'139","reported_seq":1378,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-08-19T14:39:14.998924+0000","last_change":"2024-08-19T14:34:15.738488+0000","last_active":"2024-08-19T14:39:14.998924+0000","last_peered":"2024-08-19T14:39:14.998924+0000","last_clean":"2024-08-19T14:39:14.998924+0000","last_became_active":"2024-08-19T14:34:15.733282+0000","last_became_peered":"2024-08-19T14:34:15.733282+0000","last_unstale":"2024-08-19T14:39:14.998924+0000","last_undegraded":"2024-08-19T14:39:14.998924+0000","last_fullsized":"2024-08-19T14:39:14.998924+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:33:17.453290+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:22:52.164586+0000","last_clean_scrub_stamp":"2024-08-19T14:33:17.453290+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,9],"acting":[3,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"705'14002","reported_seq":14224,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-08-19T14:39:17.771734+0000","last_change":"2024-08-19T14:33:28.623717+0000","last_active":"2024-08-19T14:39:17.771734+0000","last_peered":"2024-08-19T14:39:17.771734+0000","last_clean":"2024-08-19T14:39:17.771734+0000","last_became_active":"2024-08-19T14:33:28.623318+0000","last_became_peered":"2024-08-19T14:33:28.623318+0000","last_unstale":"2024-08-19T14:39:17.771734+0000","last_undegraded":"2024-08-19T14:39:17.771734+0000","last_fullsized":"2024-08-19T14:39:17.771734+0000","mapping_epoch":600,"log_start":"614'4000","ondisk_log_start":"614'4000","created":600,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38141952,"num_objects":582,"num_object_clones":0,"num_object_copies":1746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14002,"num_write_kb":466688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"70'139","reported_seq":1275,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-08-19T14:39:15.012028+0000","last_change":"2024-08-19T14:34:49.476489+0000","last_active":"2024-08-19T14:39:15.012028+0000","last_peered":"2024-08-19T14:39:15.012028+0000","last_clean":"2024-08-19T14:39:15.012028+0000","last_became_active":"2024-08-19T14:18:26.882002+0000","last_became_peered":"2024-08-19T14:18:26.882002+0000","last_unstale":"2024-08-19T14:39:15.012028+0000","last_undegraded":"2024-08-19T14:39:15.012028+0000","last_fullsized":"2024-08-19T14:39:15.012028+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:34:49.476415+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:24:50.276408+0000","last_clean_scrub_stamp":"2024-08-19T14:34:49.476415+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"70'139","reported_seq":1056,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-08-19T14:39:17.017187+0000","last_change":"2024-08-19T14:33:19.781527+0000","last_active":"2024-08-19T14:39:17.017187+0000","last_peered":"2024-08-19T14:39:17.017187+0000","last_clean":"2024-08-19T14:39:17.017187+0000","last_became_active":"2024-08-19T14:27:31.838968+0000","last_became_peered":"2024-08-19T14:27:31.838968+0000","last_unstale":"2024-08-19T14:39:17.017187+0000","last_undegraded":"2024-08-19T14:39:17.017187+0000","last_fullsized":"2024-08-19T14:39:17.017187+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:33:19.781431+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:28:35.206115+0000","last_clean_scrub_stamp":"2024-08-19T14:33:19.781431+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.13","version":"70'139","reported_seq":1597,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-08-19T14:39:17.029201+0000","last_change":"2024-08-19T14:33:35.786236+0000","last_active":"2024-08-19T14:39:17.029201+0000","last_peered":"2024-08-19T14:39:17.029201+0000","last_clean":"2024-08-19T14:39:17.029201+0000","last_became_active":"2024-08-19T14:30:47.939849+0000","last_became_peered":"2024-08-19T14:30:47.939849+0000","last_unstale":"2024-08-19T14:39:17.029201+0000","last_undegraded":"2024-08-19T14:39:17.029201+0000","last_fullsized":"2024-08-19T14:39:17.029201+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:33:35.786105+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:33:35.786105+0000","last_clean_scrub_stamp":"2024-08-19T14:33:35.786105+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.10","version":"70'139","reported_seq":1224,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-08-19T14:39:15.011922+0000","last_change":"2024-08-19T14:34:58.616722+0000","last_active":"2024-08-19T14:39:15.011922+0000","last_peered":"2024-08-19T14:39:15.011922+0000","last_clean":"2024-08-19T14:39:15.011922+0000","last_became_active":"2024-08-19T14:27:31.839611+0000","last_became_peered":"2024-08-19T14:27:31.839611+0000","last_unstale":"2024-08-19T14:39:15.011922+0000","last_undegraded":"2024-08-19T14:39:15.011922+0000","last_fullsized":"2024-08-19T14:39:15.011922+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:34:58.616646+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:34:58.616646+0000","last_clean_scrub_stamp":"2024-08-19T14:34:58.616646+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.11","version":"70'139","reported_seq":1279,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-08-19T14:39:15.005882+0000","last_change":"2024-08-19T14:38:34.904558+0000","last_active":"2024-08-19T14:39:15.005882+0000","last_peered":"2024-08-19T14:39:15.005882+0000","last_clean":"2024-08-19T14:39:15.005882+0000","last_became_active":"2024-08-19T14:34:15.731049+0000","last_became_peered":"2024-08-19T14:34:15.731049+0000","last_unstale":"2024-08-19T14:39:15.005882+0000","last_undegraded":"2024-08-19T14:39:15.005882+0000","last_fullsized":"2024-08-19T14:39:15.005882+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:38:34.904308+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:24:50.547428+0000","last_clean_scrub_stamp":"2024-08-19T14:38:34.904308+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:40:02.722075+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"70'139","reported_seq":1208,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-08-19T14:39:17.020436+0000","last_change":"2024-08-19T14:33:27.737496+0000","last_active":"2024-08-19T14:39:17.020436+0000","last_peered":"2024-08-19T14:39:17.020436+0000","last_clean":"2024-08-19T14:39:17.020436+0000","last_became_active":"2024-08-19T14:20:28.204724+0000","last_became_peered":"2024-08-19T14:20:28.204724+0000","last_unstale":"2024-08-19T14:39:17.020436+0000","last_undegraded":"2024-08-19T14:39:17.020436+0000","last_fullsized":"2024-08-19T14:39:17.020436+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:33:27.737367+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:28:34.216370+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.737367+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"70'139","reported_seq":1357,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-08-19T14:39:15.000776+0000","last_change":"2024-08-19T14:38:46.389978+0000","last_active":"2024-08-19T14:39:15.000776+0000","last_peered":"2024-08-19T14:39:15.000776+0000","last_clean":"2024-08-19T14:39:15.000776+0000","last_became_active":"2024-08-19T14:18:31.901556+0000","last_became_peered":"2024-08-19T14:18:31.901556+0000","last_unstale":"2024-08-19T14:39:15.000776+0000","last_undegraded":"2024-08-19T14:39:15.000776+0000","last_fullsized":"2024-08-19T14:39:15.000776+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:38:46.389862+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:38:46.389862+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:40:08.907590+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,0],"acting":[8,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.14","version":"70'139","reported_seq":1378,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-08-19T14:39:15.011913+0000","last_change":"2024-08-19T14:35:12.530145+0000","last_active":"2024-08-19T14:39:15.011913+0000","last_peered":"2024-08-19T14:39:15.011913+0000","last_clean":"2024-08-19T14:39:15.011913+0000","last_became_active":"2024-08-19T14:35:12.529792+0000","last_became_peered":"2024-08-19T14:35:12.529792+0000","last_unstale":"2024-08-19T14:39:15.011913+0000","last_undegraded":"2024-08-19T14:39:15.011913+0000","last_fullsized":"2024-08-19T14:39:15.011913+0000","mapping_epoch":635,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":636,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:34:53.617013+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:34:53.617013+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,9],"acting":[7,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.15","version":"70'139","reported_seq":1230,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-08-19T14:39:15.011932+0000","last_change":"2024-08-19T14:34:57.640316+0000","last_active":"2024-08-19T14:39:15.011932+0000","last_peered":"2024-08-19T14:39:15.011932+0000","last_clean":"2024-08-19T14:39:15.011932+0000","last_became_active":"2024-08-19T14:18:31.912496+0000","last_became_peered":"2024-08-19T14:18:31.912496+0000","last_unstale":"2024-08-19T14:39:15.011932+0000","last_undegraded":"2024-08-19T14:39:15.011932+0000","last_fullsized":"2024-08-19T14:39:15.011932+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:34:57.640213+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:20:09.180923+0000","last_clean_scrub_stamp":"2024-08-19T14:34:57.640213+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"70'139","reported_seq":1218,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-08-19T14:39:17.007747+0000","last_change":"2024-08-19T14:34:06.176510+0000","last_active":"2024-08-19T14:39:17.007747+0000","last_peered":"2024-08-19T14:39:17.007747+0000","last_clean":"2024-08-19T14:39:17.007747+0000","last_became_active":"2024-08-19T14:18:31.901229+0000","last_became_peered":"2024-08-19T14:18:31.901229+0000","last_unstale":"2024-08-19T14:39:17.007747+0000","last_undegraded":"2024-08-19T14:39:17.007747+0000","last_fullsized":"2024-08-19T14:39:17.007747+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:34:06.176403+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:34:06.176403+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1b","version":"70'139","reported_seq":1597,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-08-19T14:39:15.000589+0000","last_change":"2024-08-19T14:37:59.510774+0000","last_active":"2024-08-19T14:39:15.000589+0000","last_peered":"2024-08-19T14:39:15.000589+0000","last_clean":"2024-08-19T14:39:15.000589+0000","last_became_active":"2024-08-19T14:34:15.718427+0000","last_became_peered":"2024-08-19T14:34:15.718427+0000","last_unstale":"2024-08-19T14:39:15.000589+0000","last_undegraded":"2024-08-19T14:39:15.000589+0000","last_fullsized":"2024-08-19T14:39:15.000589+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:37:59.510697+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:37:59.510697+0000","last_clean_scrub_stamp":"2024-08-19T14:37:59.510697+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:39:15.366187+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.18","version":"70'139","reported_seq":1158,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-08-19T14:39:15.011784+0000","last_change":"2024-08-19T14:35:11.528963+0000","last_active":"2024-08-19T14:39:15.011784+0000","last_peered":"2024-08-19T14:39:15.011784+0000","last_clean":"2024-08-19T14:39:15.011784+0000","last_became_active":"2024-08-19T14:35:11.528530+0000","last_became_peered":"2024-08-19T14:35:11.528530+0000","last_unstale":"2024-08-19T14:39:15.011784+0000","last_undegraded":"2024-08-19T14:39:15.011784+0000","last_fullsized":"2024-08-19T14:39:15.011784+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:29:05.336156+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:27:33.084161+0000","last_clean_scrub_stamp":"2024-08-19T14:29:05.336156+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.19","version":"70'139","reported_seq":1576,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-08-19T14:39:15.005630+0000","last_change":"2024-08-19T14:37:41.671836+0000","last_active":"2024-08-19T14:39:15.005630+0000","last_peered":"2024-08-19T14:39:15.005630+0000","last_clean":"2024-08-19T14:39:15.005630+0000","last_became_active":"2024-08-19T14:18:31.902547+0000","last_became_peered":"2024-08-19T14:18:31.902547+0000","last_unstale":"2024-08-19T14:39:15.005630+0000","last_undegraded":"2024-08-19T14:39:15.005630+0000","last_fullsized":"2024-08-19T14:39:15.005630+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:37:41.671731+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:37:41.671731+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:39:16.127320+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1e","version":"70'139","reported_seq":601,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-08-19T14:39:15.007630+0000","last_change":"2024-08-19T14:33:17.778415+0000","last_active":"2024-08-19T14:39:15.007630+0000","last_peered":"2024-08-19T14:39:15.007630+0000","last_clean":"2024-08-19T14:39:15.007630+0000","last_became_active":"2024-08-19T14:30:47.909974+0000","last_became_peered":"2024-08-19T14:30:47.909974+0000","last_unstale":"2024-08-19T14:39:15.007630+0000","last_undegraded":"2024-08-19T14:39:15.007630+0000","last_fullsized":"2024-08-19T14:39:15.007630+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:33:17.778295+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:27:04.780939+0000","last_clean_scrub_stamp":"2024-08-19T14:33:17.778295+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1c","version":"70'139","reported_seq":1351,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-08-19T14:39:15.000634+0000","last_change":"2024-08-19T14:38:01.463630+0000","last_active":"2024-08-19T14:39:15.000634+0000","last_peered":"2024-08-19T14:39:15.000634+0000","last_clean":"2024-08-19T14:39:15.000634+0000","last_became_active":"2024-08-19T14:30:47.908866+0000","last_became_peered":"2024-08-19T14:30:47.908866+0000","last_unstale":"2024-08-19T14:39:15.000634+0000","last_undegraded":"2024-08-19T14:39:15.000634+0000","last_fullsized":"2024-08-19T14:39:15.000634+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:38:01.463524+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:38:01.463524+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:39:18.496929+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,11],"acting":[8,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1d","version":"70'139","reported_seq":1262,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-08-19T14:39:17.019110+0000","last_change":"2024-08-19T14:35:11.542467+0000","last_active":"2024-08-19T14:39:17.019110+0000","last_peered":"2024-08-19T14:39:17.019110+0000","last_clean":"2024-08-19T14:39:17.019110+0000","last_became_active":"2024-08-19T14:35:11.540488+0000","last_became_peered":"2024-08-19T14:35:11.540488+0000","last_unstale":"2024-08-19T14:39:17.019110+0000","last_undegraded":"2024-08-19T14:39:17.019110+0000","last_fullsized":"2024-08-19T14:39:17.019110+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:29.194639+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:28:29.194639+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":712507416,"num_objects":10873,"num_object_clones":0,"num_object_copies":32619,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11184,"num_objects_misplaced":3211,"num_objects_unfound":0,"num_objects_dirty":10873,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":225443,"num_write_kb":7562049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":50215,"num_bytes_recovered":3061579776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1936797696,"data_stored":1936785480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160771,"ondisk_log_size":160771,"up":48,"acting":44,"num_store_stats":10},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":856608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187791,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1091604,"kb_used_data":230216,"kb_used_omap":1,"kb_used_meta":861374,"kb_avail":92653548,"statfs":{"total":95995035648,"available":94877233152,"internally_reserved":0,"allocated":235741184,"data_stored":232633512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":882047428},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":2314987372659,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":74,"seq":317827580249,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1056736,"kb_used_data":205076,"kb_used_omap":1,"kb_used_meta":851646,"kb_avail":92688416,"statfs":{"total":95995035648,"available":94912937984,"internally_reserved":0,"allocated":209997824,"data_stored":206886096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":872085953},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874591,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":928132,"kb_used_data":152828,"kb_used_omap":1,"kb_used_meta":775294,"kb_avail":92817020,"statfs":{"total":95995035648,"available":95044628480,"internally_reserved":0,"allocated":156495872,"data_stored":153392256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":793901509},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":57,"seq":244813136226,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1113548,"kb_used_data":153604,"kb_used_omap":1,"kb_used_meta":959934,"kb_avail":92631604,"statfs":{"total":95995035648,"available":94854762496,"internally_reserved":0,"allocated":157290496,"data_stored":154183328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":982972874},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":206158430566,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692201,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1059044,"kb_used_data":129628,"kb_used_omap":1,"kb_used_meta":929406,"kb_avail":92686108,"statfs":{"total":95995035648,"available":94910574592,"internally_reserved":0,"allocated":132739072,"data_stored":129623792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":951712197},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986542,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1139616,"kb_used_data":169940,"kb_used_omap":1,"kb_used_meta":969662,"kb_avail":92605536,"statfs":{"total":95995035648,"available":94828068864,"internally_reserved":0,"allocated":174018560,"data_stored":170906832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":992934341},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150068,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1396616,"kb_used_data":386688,"kb_used_omap":1,"kb_used_meta":1009918,"kb_avail":92348536,"statfs":{"total":95995035648,"available":94564900864,"internally_reserved":0,"allocated":395968512,"data_stored":392860824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1034156487},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":634,"seq":2723009265715,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":364516,"kb_used_data":85212,"kb_used_omap":4,"kb_used_meta":279163,"kb_avail":93380636,"statfs":{"total":95995035648,"available":95621771264,"internally_reserved":0,"allocated":87257088,"data_stored":84141808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4407,"internal_metadata":285863625},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,2,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575229,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1353892,"kb_used_data":243928,"kb_used_omap":1,"kb_used_meta":1109886,"kb_avail":92391260,"statfs":{"total":95995035648,"available":94608650240,"internally_reserved":0,"allocated":249782272,"data_stored":246666472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1136523717},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,3,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738753,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1169624,"kb_used_data":281868,"kb_used_omap":1,"kb_used_meta":887742,"kb_avail":92575528,"statfs":{"total":95995035648,"available":94797340672,"internally_reserved":0,"allocated":288632832,"data_stored":285521088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":909048261},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,3,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":273682432,"data_stored":273678360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":240128000,"data_stored":240123928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":74514432,"data_stored":74514432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":370741248,"data_stored":370737176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":158138368,"data_stored":158138368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":120258560,"data_stored":120258560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":143130624,"data_stored":143130624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":141754368,"data_stored":141754368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":198115328,"data_stored":198115328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":216334336,"data_stored":216334336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T14:39:24.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:24 smithi045 ceph-mon[20143]: pgmap v1689: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 545 MiB data, 9.9 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 609 op/s; 10819/26163 objects degraded (41.352%); 3211/26163 objects misplaced (12.273%); 6.9 MiB/s, 111 objects/s recovering 2024-08-19T14:39:24.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:24 smithi045 ceph-mon[20143]: osdmap e706: 12 total, 12 up, 10 in 2024-08-19T14:39:24.306 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:39:24.346 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:24.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:24.347 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:39:24.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:24 smithi031 ceph-mon[16336]: pgmap v1689: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 545 MiB data, 9.9 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 609 op/s; 10819/26163 objects degraded (41.352%); 3211/26163 objects misplaced (12.273%); 6.9 MiB/s, 111 objects/s recovering 2024-08-19T14:39:24.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:24 smithi031 ceph-mon[16336]: osdmap e706: 12 total, 12 up, 10 in 2024-08-19T14:39:24.448 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:39:24.492 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:24.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:24.493 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:39:24.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:24 smithi032 ceph-mon[20004]: pgmap v1689: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 545 MiB data, 9.9 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 609 op/s; 10819/26163 objects degraded (41.352%); 3211/26163 objects misplaced (12.273%); 6.9 MiB/s, 111 objects/s recovering 2024-08-19T14:39:24.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:24 smithi032 ceph-mon[20004]: osdmap e706: 12 total, 12 up, 10 in 2024-08-19T14:39:24.594 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:39:24.616 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:24.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:24.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:39:24.718 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:39:24.792 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:24.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:24.793 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:39:24.894 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:39:24.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:24.917 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:24.917 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:39:25.018 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:39:25.042 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:25.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:25.044 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:39:25.145 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:39:25.166 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:25.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:25.168 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:39:25.269 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:39:25.290 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:25.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:25.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:39:25.392 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:39:25.432 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:25.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:25.433 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:39:25.534 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:39:25.576 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:25.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:25.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:39:25.678 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:39:25.699 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:25.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:25.701 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:39:25.802 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:39:25.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:25.838 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:25.839 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:39:25.940 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:39:25.965 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:25.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:25.966 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:39:26.055 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is unset 2024-08-19T14:39:26.068 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:39:26.090 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:26.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:26.091 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:39:26.192 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:39:26.214 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:26.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:26.215 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:39:26.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:26 smithi045 ceph-mon[20143]: pgmap v1691: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 504 MiB data, 9.8 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 471 op/s; 10847/24177 objects degraded (44.865%); 3090/24177 objects misplaced (12.781%); 4.8 MiB/s, 77 objects/s recovering 2024-08-19T14:39:26.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:26 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3492717696' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:39:26.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:26 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:39:26.316 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:39:26.338 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:26.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:26.339 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:39:26.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:26 smithi031 ceph-mon[16336]: pgmap v1691: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 504 MiB data, 9.8 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 471 op/s; 10847/24177 objects degraded (44.865%); 3090/24177 objects misplaced (12.781%); 4.8 MiB/s, 77 objects/s recovering 2024-08-19T14:39:26.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:26 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3492717696' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:39:26.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:26 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:39:26.439 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:39:26.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:26.461 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:26.461 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:39:26.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:26 smithi032 ceph-mon[20004]: pgmap v1691: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 504 MiB data, 9.8 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 471 op/s; 10847/24177 objects degraded (44.865%); 3090/24177 objects misplaced (12.781%); 4.8 MiB/s, 77 objects/s recovering 2024-08-19T14:39:26.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:26 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3492717696' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:39:26.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:26 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:39:26.562 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:39:26.583 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:26.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:26.584 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:39:26.685 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:39:26.709 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:26.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:26.710 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:39:26.811 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:39:26.834 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:26.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:26.835 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:39:26.936 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:39:26.960 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:26.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:26.962 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:39:27.063 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:39:27.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:27.085 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:27.086 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:39:27.187 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:39:27.208 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:27.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:27.209 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:39:27.249 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-19T14:39:27.288 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:27 smithi031 ceph-mon[16336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:39:27.288 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:27 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:39:27.288 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:27 smithi031 ceph-mon[16336]: osdmap e707: 12 total, 12 up, 10 in 2024-08-19T14:39:27.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:27 smithi045 ceph-mon[20143]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:39:27.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:27 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:39:27.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:27 smithi045 ceph-mon[20143]: osdmap e707: 12 total, 12 up, 10 in 2024-08-19T14:39:27.310 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:39:27.365 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:27.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:27.366 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:39:27.467 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:39:27.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:27 smithi032 ceph-mon[20004]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:39:27.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:27 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:39:27.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:27 smithi032 ceph-mon[20004]: osdmap e707: 12 total, 12 up, 10 in 2024-08-19T14:39:27.529 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:27.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:27.530 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:39:27.631 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:39:27.683 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:27.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:27.684 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:39:27.784 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:39:27.806 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:27.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:27.807 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:39:27.908 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:39:27.959 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:27.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:27.960 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:39:28.061 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:39:28.083 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:28.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:28.084 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:39:28.185 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:39:28.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:28.209 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:28.209 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:39:28.310 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:39:28.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:28.362 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:28.362 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:39:28.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:28 smithi031 ceph-mon[16336]: pgmap v1693: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 393 MiB data, 9.5 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 592 op/s; 10440/18867 objects degraded (55.335%); 3090/18867 objects misplaced (16.378%); 6.9 MiB/s, 111 objects/s recovering 2024-08-19T14:39:28.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:28 smithi031 ceph-mon[16336]: 1.1c scrub starts 2024-08-19T14:39:28.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:28 smithi031 ceph-mon[16336]: 1.1c scrub ok 2024-08-19T14:39:28.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:28 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 10440/18867 objects degraded (55.335%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-19T14:39:28.463 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:39:28.486 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:28.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:28.487 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:39:28.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:28 smithi032 ceph-mon[20004]: pgmap v1693: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 393 MiB data, 9.5 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 592 op/s; 10440/18867 objects degraded (55.335%); 3090/18867 objects misplaced (16.378%); 6.9 MiB/s, 111 objects/s recovering 2024-08-19T14:39:28.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:28 smithi032 ceph-mon[20004]: 1.1c scrub starts 2024-08-19T14:39:28.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:28 smithi032 ceph-mon[20004]: 1.1c scrub ok 2024-08-19T14:39:28.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:28 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 10440/18867 objects degraded (55.335%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-19T14:39:28.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:28 smithi045 ceph-mon[20143]: pgmap v1693: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 393 MiB data, 9.5 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 592 op/s; 10440/18867 objects degraded (55.335%); 3090/18867 objects misplaced (16.378%); 6.9 MiB/s, 111 objects/s recovering 2024-08-19T14:39:28.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:28 smithi045 ceph-mon[20143]: 1.1c scrub starts 2024-08-19T14:39:28.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:28 smithi045 ceph-mon[20143]: 1.1c scrub ok 2024-08-19T14:39:28.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:28 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 10440/18867 objects degraded (55.335%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-19T14:39:28.588 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:39:28.656 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:28.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:28.657 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:39:28.758 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:39:28.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:28.793 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:28.793 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:39:28.894 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:39:28.916 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:28.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:28.917 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:39:29.018 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:39:29.043 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:29.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:29.044 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:39:29.145 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:39:29.194 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:29.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:29.195 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:39:29.244 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:39:29.244 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [ 2024-08-19T14:39:29.244 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:39:29.245 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:232503 7.d 7:b05fccfc:::benchmark_data_smithi045_115842_object114343:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e707)", 2024-08-19T14:39:29.245 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:39:29.227980+0000", 2024-08-19T14:39:29.245 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.014783205000000001, 2024-08-19T14:39:29.245 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.014818219000000001, 2024-08-19T14:39:29.245 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:39:29.245 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:39:29.245 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "waiting for sub ops", 2024-08-19T14:39:29.245 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:39:29.245 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:39:29.245 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:39:29.245 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 232503 2024-08-19T14:39:29.245 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:39:29.245 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:39:29.246 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:39:29.246 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:39:29.246 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.227980+0000", 2024-08-19T14:39:29.246 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:39:29.246 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:39:29.246 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:39:29.246 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:39:29.246 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.227980+0000", 2024-08-19T14:39:29.246 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:39:29.246 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:39:29.246 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:39:29.246 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:39:29.246 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.227981+0000", 2024-08-19T14:39:29.246 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.2e-07 2024-08-19T14:39:29.247 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:39:29.247 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:39:29.247 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:39:29.247 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.227986+0000", 2024-08-19T14:39:29.247 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.8459999999999999e-06 2024-08-19T14:39:29.247 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:39:29.247 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:39:29.247 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:39:29.247 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.227994+0000", 2024-08-19T14:39:29.247 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.6370000000000005e-06 2024-08-19T14:39:29.247 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:39:29.247 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:39:29.247 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:39:29.247 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.228001+0000", 2024-08-19T14:39:29.248 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.331e-06 2024-08-19T14:39:29.248 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:39:29.248 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:39:29.248 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:39:29.248 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.228182+0000", 2024-08-19T14:39:29.248 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00018181100000000001 2024-08-19T14:39:29.248 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:39:29.248 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:39:29.248 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:39:29.248 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.228331+0000", 2024-08-19T14:39:29.248 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00014867999999999999 2024-08-19T14:39:29.248 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:39:29.248 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:39:29.248 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,3", 2024-08-19T14:39:29.249 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.228469+0000", 2024-08-19T14:39:29.249 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013788200000000001 2024-08-19T14:39:29.249 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:39:29.249 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:39:29.249 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:39:29.249 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:39:29.249 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:39:29.249 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:232506 7.d 7.e92876dd (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e707)", 2024-08-19T14:39:29.249 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:39:29.228041+0000", 2024-08-19T14:39:29.249 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.014722722000000001, 2024-08-19T14:39:29.249 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.014846293999999999, 2024-08-19T14:39:29.249 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:39:29.249 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:39:29.249 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "queued for pg", 2024-08-19T14:39:29.250 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:39:29.250 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:39:29.250 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:39:29.250 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 232506 2024-08-19T14:39:29.250 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:39:29.250 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:39:29.250 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:39:29.250 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:39:29.250 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.228041+0000", 2024-08-19T14:39:29.250 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:39:29.250 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:39:29.250 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:39:29.250 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:39:29.251 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.228041+0000", 2024-08-19T14:39:29.251 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:39:29.251 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:39:29.251 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:39:29.251 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:39:29.251 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.228041+0000", 2024-08-19T14:39:29.251 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.0200000000000002e-07 2024-08-19T14:39:29.251 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:39:29.251 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:39:29.251 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:39:29.251 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.228045+0000", 2024-08-19T14:39:29.251 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.658e-06 2024-08-19T14:39:29.251 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:39:29.251 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:39:29.252 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:39:29.252 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.228056+0000", 2024-08-19T14:39:29.252 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0949e-05 2024-08-19T14:39:29.252 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:39:29.252 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:39:29.252 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:39:29.252 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.228063+0000", 2024-08-19T14:39:29.252 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.092e-06 2024-08-19T14:39:29.252 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:39:29.252 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:39:29.252 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:39:29.252 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:39:29.252 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:39:29.252 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "MOSDPGPushReply(7.e 707/634 [PushReplyOp(7:7aae7d56:::benchmark_data_smithi045_115842_object114435:head)])", 2024-08-19T14:39:29.253 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:39:29.228100+0000", 2024-08-19T14:39:29.253 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.014663321999999999, 2024-08-19T14:39:29.253 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.014842841000000001, 2024-08-19T14:39:29.253 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:39:29.253 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:39:29.253 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:39:29.253 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:39:29.253 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:39:29.253 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:39:29.253 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.228100+0000", 2024-08-19T14:39:29.253 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:39:29.253 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:39:29.253 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:39:29.253 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:39:29.253 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.228100+0000", 2024-08-19T14:39:29.254 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:39:29.254 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:39:29.254 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:39:29.254 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:39:29.254 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.228101+0000", 2024-08-19T14:39:29.254 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.1900000000000002e-07 2024-08-19T14:39:29.254 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:39:29.254 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:39:29.254 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:39:29.254 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.228106+0000", 2024-08-19T14:39:29.254 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.6500000000000001e-06 2024-08-19T14:39:29.254 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:39:29.254 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:39:29.254 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:39:29.255 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.228117+0000", 2024-08-19T14:39:29.255 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.079e-05 2024-08-19T14:39:29.255 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:39:29.255 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:39:29.255 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:39:29.255 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.228125+0000", 2024-08-19T14:39:29.255 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.3039999999999999e-06 2024-08-19T14:39:29.255 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:39:29.255 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:39:29.255 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:39:29.255 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.228529+0000", 2024-08-19T14:39:29.255 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00040467299999999998 2024-08-19T14:39:29.255 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:39:29.255 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:39:29.255 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:39:29.256 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:39:29.256 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:39:29.256 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.16962.0:232507 7.7 e707/600 7:ede83f09:::benchmark_data_smithi045_115842_object114347:head v 707'14249, mlcod=707'14247)", 2024-08-19T14:39:29.256 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:39:29.236517+0000", 2024-08-19T14:39:29.256 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.0062463520000000002, 2024-08-19T14:39:29.256 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0064828919999999996, 2024-08-19T14:39:29.256 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:39:29.256 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:39:29.256 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-08-19T14:39:29.256 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:39:29.256 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:39:29.256 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:39:29.256 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.236517+0000", 2024-08-19T14:39:29.256 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:39:29.256 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:39:29.257 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:39:29.257 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:39:29.257 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.236517+0000", 2024-08-19T14:39:29.257 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:39:29.257 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:39:29.257 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:39:29.257 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:39:29.257 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.236518+0000", 2024-08-19T14:39:29.257 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.3699999999999999e-07 2024-08-19T14:39:29.257 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:39:29.257 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:39:29.257 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:39:29.257 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.236526+0000", 2024-08-19T14:39:29.257 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.8590000000000005e-06 2024-08-19T14:39:29.258 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:39:29.258 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:39:29.258 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:39:29.258 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.236542+0000", 2024-08-19T14:39:29.258 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6126000000000001e-05 2024-08-19T14:39:29.258 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:39:29.258 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:39:29.258 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:39:29.258 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.236549+0000", 2024-08-19T14:39:29.258 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.4189999999999996e-06 2024-08-19T14:39:29.258 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:39:29.258 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:39:29.258 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:39:29.258 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.237805+0000", 2024-08-19T14:39:29.259 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.001255365 2024-08-19T14:39:29.259 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:39:29.259 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:39:29.259 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:39:29.259 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.237824+0000", 2024-08-19T14:39:29.259 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.9517999999999998e-05 2024-08-19T14:39:29.259 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:39:29.259 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:39:29.259 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:39:29.259 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:39:29.259 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:39:29.259 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:232509 7.d 7.4d5af9ad (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e707)", 2024-08-19T14:39:29.259 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:39:29.238693+0000", 2024-08-19T14:39:29.259 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.0040705519999999999, 2024-08-19T14:39:29.260 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00436631, 2024-08-19T14:39:29.260 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:39:29.260 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:39:29.260 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "queued for pg", 2024-08-19T14:39:29.260 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:39:29.260 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:39:29.260 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:39:29.260 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 232509 2024-08-19T14:39:29.260 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:39:29.260 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:39:29.260 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:39:29.260 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:39:29.260 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.238693+0000", 2024-08-19T14:39:29.260 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:39:29.260 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:39:29.261 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:39:29.261 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:39:29.261 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.238693+0000", 2024-08-19T14:39:29.261 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:39:29.261 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:39:29.261 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:39:29.261 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:39:29.261 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.238694+0000", 2024-08-19T14:39:29.261 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3430000000000001e-06 2024-08-19T14:39:29.261 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:39:29.261 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:39:29.261 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:39:29.261 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.238701+0000", 2024-08-19T14:39:29.261 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.4919999999999998e-06 2024-08-19T14:39:29.262 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:39:29.262 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:39:29.262 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:39:29.262 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.238712+0000", 2024-08-19T14:39:29.262 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1334e-05 2024-08-19T14:39:29.262 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:39:29.262 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:39:29.262 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:39:29.262 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.238720+0000", 2024-08-19T14:39:29.262 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.7419999999999997e-06 2024-08-19T14:39:29.262 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:39:29.262 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:39:29.262 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:39:29.262 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:39:29.263 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:39:29.263 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.16962.0:232510 7.7 e707/600)", 2024-08-19T14:39:29.263 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:39:29.240509+0000", 2024-08-19T14:39:29.263 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.0022542040000000001, 2024-08-19T14:39:29.263 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.002600415, 2024-08-19T14:39:29.263 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:39:29.263 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:39:29.264 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "queued for pg", 2024-08-19T14:39:29.264 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:39:29.264 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:39:29.264 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:39:29.264 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.240509+0000", 2024-08-19T14:39:29.264 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:39:29.264 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:39:29.264 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:39:29.264 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:39:29.264 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.240509+0000", 2024-08-19T14:39:29.264 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:39:29.264 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:39:29.264 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:39:29.264 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:39:29.265 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.240510+0000", 2024-08-19T14:39:29.265 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.4600000000000003e-07 2024-08-19T14:39:29.265 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:39:29.265 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:39:29.265 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:39:29.265 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.240520+0000", 2024-08-19T14:39:29.265 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.6330000000000008e-06 2024-08-19T14:39:29.265 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:39:29.265 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:39:29.265 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:39:29.265 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.240539+0000", 2024-08-19T14:39:29.265 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.8941e-05 2024-08-19T14:39:29.265 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:39:29.265 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:39:29.265 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:39:29.266 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.240546+0000", 2024-08-19T14:39:29.266 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.5800000000000003e-06 2024-08-19T14:39:29.266 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:39:29.266 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:39:29.266 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:39:29.266 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:39:29.266 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-08-19T14:39:29.266 INFO:teuthology.orchestra.run.smithi031.stdout: "num_ops": 6 2024-08-19T14:39:29.266 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:39:29.296 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:39:29.316 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:29.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:29.317 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:39:29.418 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:39:29.454 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:29.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:29.455 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:39:29.556 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:39:29.577 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:29.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:29.579 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:39:29.680 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:39:29.701 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:29.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:29.702 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:39:29.803 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:39:29.824 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:29.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:29.825 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:39:29.926 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:39:29.949 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:29.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:29.950 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:39:30.051 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:39:30.074 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:30.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:30.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:39:30.176 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:39:30.197 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:30.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:30.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:39:30.300 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:39:30.325 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:30.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:30.326 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:39:30.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:30 smithi031 ceph-mon[16336]: pgmap v1694: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 343 MiB data, 9.3 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 674 op/s; 10489/16428 objects degraded (63.848%); 2974/16428 objects misplaced (18.103%); 7.8 MiB/s, 126 objects/s recovering 2024-08-19T14:39:30.427 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:39:30.474 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:30.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:30.475 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:39:30.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:30 smithi032 ceph-mon[20004]: pgmap v1694: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 343 MiB data, 9.3 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 674 op/s; 10489/16428 objects degraded (63.848%); 2974/16428 objects misplaced (18.103%); 7.8 MiB/s, 126 objects/s recovering 2024-08-19T14:39:30.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:30 smithi045 ceph-mon[20143]: pgmap v1694: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 343 MiB data, 9.3 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 674 op/s; 10489/16428 objects degraded (63.848%); 2974/16428 objects misplaced (18.103%); 7.8 MiB/s, 126 objects/s recovering 2024-08-19T14:39:30.576 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:39:30.599 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:30.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:30.600 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:39:30.701 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:39:30.754 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:30.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:30.756 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:39:30.857 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:39:30.879 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:30.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:30.880 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:39:30.981 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:39:31.002 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:31.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:31.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:39:31.105 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:39:31.126 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:31.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:31.127 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:39:31.228 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:39:31.250 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:31.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:31.251 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:39:31.352 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:39:31.378 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:31.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:31.379 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:39:31.480 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:39:31.513 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:31.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:31.514 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:39:31.615 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:39:31.641 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:31.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:31.642 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:39:31.743 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:39:31.765 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:31.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:31.767 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:39:31.868 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:39:31.891 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:31.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:31.892 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:39:31.993 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:39:32.017 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:32.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:32.019 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:39:32.120 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:39:32.166 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:32.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:32.167 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:39:32.269 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:39:32.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:32.331 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:32.332 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:39:32.338 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:32 smithi031 ceph-mon[16336]: pgmap v1695: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 332 MiB data, 9.2 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 426 op/s; 10489/15921 objects degraded (65.882%); 2974/15921 objects misplaced (18.680%); 5.0 MiB/s, 80 objects/s recovering 2024-08-19T14:39:32.338 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:32 smithi031 ceph-mon[16336]: 1.6 scrub starts 2024-08-19T14:39:32.338 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:32 smithi031 ceph-mon[16336]: 1.6 scrub ok 2024-08-19T14:39:32.338 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:32 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:39:32.338 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:32 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:39:32.338 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:32 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:39:32.338 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:32 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:39:32.338 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:32 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:39:32.432 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:39:32.455 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:32.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:32.456 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:39:32.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:32 smithi032 ceph-mon[20004]: pgmap v1695: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 332 MiB data, 9.2 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 426 op/s; 10489/15921 objects degraded (65.882%); 2974/15921 objects misplaced (18.680%); 5.0 MiB/s, 80 objects/s recovering 2024-08-19T14:39:32.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:32 smithi032 ceph-mon[20004]: 1.6 scrub starts 2024-08-19T14:39:32.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:32 smithi032 ceph-mon[20004]: 1.6 scrub ok 2024-08-19T14:39:32.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:32 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:39:32.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:32 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:39:32.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:32 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:39:32.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:32 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:39:32.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:32 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:39:32.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:32 smithi045 ceph-mon[20143]: pgmap v1695: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 332 MiB data, 9.2 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 426 op/s; 10489/15921 objects degraded (65.882%); 2974/15921 objects misplaced (18.680%); 5.0 MiB/s, 80 objects/s recovering 2024-08-19T14:39:32.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:32 smithi045 ceph-mon[20143]: 1.6 scrub starts 2024-08-19T14:39:32.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:32 smithi045 ceph-mon[20143]: 1.6 scrub ok 2024-08-19T14:39:32.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:32 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:39:32.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:32 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:39:32.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:32 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:39:32.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:32 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:39:32.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:32 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:39:32.557 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:39:32.627 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:32.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:32.628 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:39:32.710 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-08-19T14:39:32.729 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:39:32.754 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:39:32.760 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:39:32.860 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:32.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:32.861 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:39:32.962 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:39:32.984 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:32.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:32.985 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:39:33.086 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:39:33.108 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:33.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:33.109 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:39:33.206 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:33 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:39:33.206 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:33 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:39:33.206 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:33 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:39:33.207 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:33 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:39:33.207 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:33 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 10489/15921 objects degraded (65.882%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-19T14:39:33.210 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:39:33.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:33.255 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:33.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:39:33.356 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:39:33.378 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:33.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:33.379 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:39:33.480 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:39:33.485 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-19T14:39:33.501 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:33.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:33.502 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:39:33.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:33 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:39:33.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:33 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:39:33.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:33 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:39:33.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:33 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:39:33.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:33 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 10489/15921 objects degraded (65.882%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-19T14:39:33.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:33 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:39:33.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:33 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:39:33.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:33 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:39:33.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:33 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:39:33.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:33 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 10489/15921 objects degraded (65.882%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-19T14:39:33.603 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:39:33.650 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:33.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:33.651 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:39:33.701 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:39:33.702 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:39:33.728 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:39:33.729 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:39:33.752 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:39:33.774 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:33.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:33.775 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:39:33.778 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:39:33.778 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:39:33.847 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:39:33.847 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:39:33.876 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:39:33.888 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:39:33.889 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:39:33.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:33.902 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:33.902 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:39:33.923 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:39:33.924 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:39:33.968 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:39:33.968 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:39:34.003 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:39:34.010 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:39:34.010 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:39:34.026 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:34.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:34.027 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:39:34.047 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:39:34.047 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:39:34.081 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:39:34.081 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:39:34.116 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:39:34.116 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:39:34.127 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:39:34.151 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:39:34.151 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:39:34.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:34.151 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:34.152 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:39:34.253 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:39:34.275 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:34.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:34.276 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:39:34.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:34 smithi031 ceph-mon[16336]: pgmap v1696: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 213 MiB data, 8.9 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 523 op/s; 10134/10185 objects degraded (99.499%); 2974/10185 objects misplaced (29.200%); 5.9 MiB/s, 95 objects/s recovering 2024-08-19T14:39:34.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:34 smithi031 ceph-mon[16336]: 1.1b scrub starts 2024-08-19T14:39:34.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:34 smithi031 ceph-mon[16336]: 1.1b scrub ok 2024-08-19T14:39:34.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:34 smithi031 ceph-mon[16336]: 1.19 scrub starts 2024-08-19T14:39:34.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:34 smithi031 ceph-mon[16336]: 1.19 scrub ok 2024-08-19T14:39:34.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:34 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1279801905' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:39:34.377 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:39:34.416 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:34.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:34.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:39:34.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:34 smithi032 ceph-mon[20004]: pgmap v1696: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 213 MiB data, 8.9 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 523 op/s; 10134/10185 objects degraded (99.499%); 2974/10185 objects misplaced (29.200%); 5.9 MiB/s, 95 objects/s recovering 2024-08-19T14:39:34.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:34 smithi032 ceph-mon[20004]: 1.1b scrub starts 2024-08-19T14:39:34.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:34 smithi032 ceph-mon[20004]: 1.1b scrub ok 2024-08-19T14:39:34.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:34 smithi032 ceph-mon[20004]: 1.19 scrub starts 2024-08-19T14:39:34.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:34 smithi032 ceph-mon[20004]: 1.19 scrub ok 2024-08-19T14:39:34.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:34 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1279801905' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:39:34.517 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:39:34.539 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:34.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:34.540 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:39:34.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:34 smithi045 ceph-mon[20143]: pgmap v1696: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 213 MiB data, 8.9 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 523 op/s; 10134/10185 objects degraded (99.499%); 2974/10185 objects misplaced (29.200%); 5.9 MiB/s, 95 objects/s recovering 2024-08-19T14:39:34.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:34 smithi045 ceph-mon[20143]: 1.1b scrub starts 2024-08-19T14:39:34.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:34 smithi045 ceph-mon[20143]: 1.1b scrub ok 2024-08-19T14:39:34.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:34 smithi045 ceph-mon[20143]: 1.19 scrub starts 2024-08-19T14:39:34.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:34 smithi045 ceph-mon[20143]: 1.19 scrub ok 2024-08-19T14:39:34.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:34 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1279801905' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:39:34.641 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:39:34.665 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:34.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:34.666 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:39:34.767 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:39:34.788 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:34.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:34.789 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:39:34.890 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:39:34.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:34.914 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:34.915 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:39:34.988 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:39:34.989 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-08-19T14:39:35.015 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:39:35.038 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:35.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:35.039 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:39:35.140 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:39:35.166 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:35.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:35.167 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:39:35.268 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:39:35.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:35.291 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:35.291 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:39:35.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:35 smithi031 ceph-mon[16336]: from='client.26875 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:39:35.392 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:39:35.416 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:35.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:35.417 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:39:35.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:35 smithi032 ceph-mon[20004]: from='client.26875 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:39:35.518 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:39:35.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:35.542 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:35.543 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:39:35.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:35 smithi045 ceph-mon[20143]: from='client.26875 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:39:35.643 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:39:35.667 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:35.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:35.668 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:39:35.769 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:39:35.790 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:35.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:35.792 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:39:35.893 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:39:35.913 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:35.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:35.914 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:39:36.015 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:39:36.036 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:36.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:36.037 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:39:36.138 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:39:36.161 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:36.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:36.162 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:39:36.263 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:39:36.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:36 smithi045 ceph-mon[20143]: pgmap v1697: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 173 MiB data, 8.8 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 530 op/s; 10153/8262 objects degraded (122.888%); 2829/8262 objects misplaced (34.241%); 6.4 MiB/s, 102 objects/s recovering 2024-08-19T14:39:36.316 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:36.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:36.317 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:39:36.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:36 smithi031 ceph-mon[16336]: pgmap v1697: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 173 MiB data, 8.8 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 530 op/s; 10153/8262 objects degraded (122.888%); 2829/8262 objects misplaced (34.241%); 6.4 MiB/s, 102 objects/s recovering 2024-08-19T14:39:36.418 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:39:36.439 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:36.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:36.440 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:39:36.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:36 smithi032 ceph-mon[20004]: pgmap v1697: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 173 MiB data, 8.8 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 530 op/s; 10153/8262 objects degraded (122.888%); 2829/8262 objects misplaced (34.241%); 6.4 MiB/s, 102 objects/s recovering 2024-08-19T14:39:36.541 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:39:36.562 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:36.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:36.563 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:39:36.664 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:39:36.687 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:36.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:36.688 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:39:36.724 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Removed 118157 objects 2024-08-19T14:39:36.724 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Clean up completed and total clean up time :272.267 2024-08-19T14:39:36.789 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:39:36.809 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:36.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:36.810 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:39:36.911 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:39:36.933 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:36.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:36.934 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:39:37.035 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:39:37.062 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:37.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:37.063 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:39:37.164 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:39:37.185 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:37.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:37.186 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:39:37.287 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:39:37.314 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:37.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:37.315 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:39:37.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:37 smithi031 ceph-mon[16336]: 7.7 scrub starts 2024-08-19T14:39:37.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:37 smithi031 ceph-mon[16336]: 7.7 scrub ok 2024-08-19T14:39:37.416 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:39:37.442 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:37.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:37.443 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:39:37.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:37 smithi032 ceph-mon[20004]: 7.7 scrub starts 2024-08-19T14:39:37.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:37 smithi032 ceph-mon[20004]: 7.7 scrub ok 2024-08-19T14:39:37.544 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:39:37.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:37 smithi045 ceph-mon[20143]: 7.7 scrub starts 2024-08-19T14:39:37.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:37 smithi045 ceph-mon[20143]: 7.7 scrub ok 2024-08-19T14:39:37.567 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:37.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:37.568 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:39:37.669 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:39:37.694 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:37.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:37.695 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:39:37.796 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:39:37.822 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:37.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:37.823 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:39:37.924 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:39:37.951 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:37.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:37.952 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:39:38.053 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:39:38.078 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:38.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:38.080 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:39:38.180 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:39:38.206 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:38.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:38.207 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:39:38.308 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:39:38.334 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:38.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:38.335 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:39:38.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:38 smithi031 ceph-mon[16336]: pgmap v1698: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 63 MiB data, 8.6 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 517 op/s; 9776/3003 objects degraded (325.541%); 2829/3003 objects misplaced (94.206%); 6.0 MiB/s, 97 objects/s recovering 2024-08-19T14:39:38.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:38 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 9776/3003 objects degraded (325.541%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-19T14:39:38.436 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:39:38.462 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:38.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:38.463 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:39:38.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:38 smithi032 ceph-mon[20004]: pgmap v1698: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 63 MiB data, 8.6 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 517 op/s; 9776/3003 objects degraded (325.541%); 2829/3003 objects misplaced (94.206%); 6.0 MiB/s, 97 objects/s recovering 2024-08-19T14:39:38.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:38 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 9776/3003 objects degraded (325.541%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-19T14:39:38.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:38 smithi045 ceph-mon[20143]: pgmap v1698: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 63 MiB data, 8.6 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 517 op/s; 9776/3003 objects degraded (325.541%); 2829/3003 objects misplaced (94.206%); 6.0 MiB/s, 97 objects/s recovering 2024-08-19T14:39:38.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:38 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 9776/3003 objects degraded (325.541%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-19T14:39:38.564 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:39:38.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:38.591 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:38.591 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:39:38.692 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:39:38.716 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:38.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:38.718 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:39:38.819 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:39:38.844 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:38.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:38.846 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:39:38.946 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:39:38.972 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:38.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:38.973 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:39:39.074 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:39:39.101 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:39.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:39.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:39:39.203 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:39:39.228 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:39.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:39.229 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:39:39.330 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:39:39.356 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:39.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:39.357 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:39:39.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:39 smithi031 ceph-mon[16336]: 1.1 scrub starts 2024-08-19T14:39:39.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:39 smithi031 ceph-mon[16336]: 1.1 scrub ok 2024-08-19T14:39:39.458 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:39:39.484 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:39.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:39.485 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:39:39.501 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_5 2024-08-19T14:39:39.501 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd pool rm unique_pool_5 unique_pool_5 --yes-i-really-really-mean-it 2024-08-19T14:39:39.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:39 smithi032 ceph-mon[20004]: 1.1 scrub starts 2024-08-19T14:39:39.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:39 smithi032 ceph-mon[20004]: 1.1 scrub ok 2024-08-19T14:39:39.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:39 smithi045 ceph-mon[20143]: 1.1 scrub starts 2024-08-19T14:39:39.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:39 smithi045 ceph-mon[20143]: 1.1 scrub ok 2024-08-19T14:39:39.586 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:39:39.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:39.611 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:39.611 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:39:39.712 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:39:39.737 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:39.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:39.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:39:39.840 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:39:39.861 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:39.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:39.862 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:39:39.963 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:39:39.988 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:39.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:39.989 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:39:40.090 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:39:40.116 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:40.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:40.118 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:39:40.219 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:39:40.244 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:40.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:40.245 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:39:40.346 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:39:40.361 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:40 smithi031 ceph-mon[16336]: pgmap v1699: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 24 MiB data, 8.5 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 492 op/s; 9790/1122 objects degraded (872.549%); 2742/1122 objects misplaced (244.385%); 5.5 MiB/s, 100 objects/s recovering 2024-08-19T14:39:40.361 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:40 smithi031 ceph-mon[16336]: 7.9 scrub starts 2024-08-19T14:39:40.362 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:40 smithi031 ceph-mon[16336]: 7.9 scrub ok 2024-08-19T14:39:40.372 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:40.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:40.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:39:40.474 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:39:40.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:40.501 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:40.501 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:39:40.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:40 smithi032 ceph-mon[20004]: pgmap v1699: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 24 MiB data, 8.5 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 492 op/s; 9790/1122 objects degraded (872.549%); 2742/1122 objects misplaced (244.385%); 5.5 MiB/s, 100 objects/s recovering 2024-08-19T14:39:40.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:40 smithi032 ceph-mon[20004]: 7.9 scrub starts 2024-08-19T14:39:40.509 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:40 smithi032 ceph-mon[20004]: 7.9 scrub ok 2024-08-19T14:39:40.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:40 smithi045 ceph-mon[20143]: pgmap v1699: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 24 MiB data, 8.5 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 492 op/s; 9790/1122 objects degraded (872.549%); 2742/1122 objects misplaced (244.385%); 5.5 MiB/s, 100 objects/s recovering 2024-08-19T14:39:40.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:40 smithi045 ceph-mon[20143]: 7.9 scrub starts 2024-08-19T14:39:40.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:40 smithi045 ceph-mon[20143]: 7.9 scrub ok 2024-08-19T14:39:40.602 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:39:40.628 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:40.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:40.629 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:39:40.730 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:39:40.756 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:40.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:40.757 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:39:40.858 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:39:40.885 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:40.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:40.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:39:40.987 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:39:41.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:41.015 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:41.015 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:39:41.116 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:39:41.141 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:41.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:41.142 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:39:41.243 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:39:41.269 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:41.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:41.270 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:39:41.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:41 smithi031 ceph-mon[16336]: 7.6 scrub starts 2024-08-19T14:39:41.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:41 smithi031 ceph-mon[16336]: 7.6 scrub ok 2024-08-19T14:39:41.371 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:39:41.398 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:41.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:41.399 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:39:41.500 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:39:41.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:41 smithi032 ceph-mon[20004]: 7.6 scrub starts 2024-08-19T14:39:41.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:41 smithi032 ceph-mon[20004]: 7.6 scrub ok 2024-08-19T14:39:41.526 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:41.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:41.527 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:39:41.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:41 smithi045 ceph-mon[20143]: 7.6 scrub starts 2024-08-19T14:39:41.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:41 smithi045 ceph-mon[20143]: 7.6 scrub ok 2024-08-19T14:39:41.627 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:39:41.653 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:41.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:41.654 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:39:41.755 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:39:41.782 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:41.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:41.783 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:39:41.884 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:39:41.910 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:41.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:41.911 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:39:42.012 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:39:42.038 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:42.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:42.040 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:39:42.141 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:39:42.166 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:42.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:42.167 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:39:42.268 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:39:42.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:42.295 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:42.296 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:39:42.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:42 smithi031 ceph-mon[16336]: pgmap v1700: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 18 MiB data, 8.4 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 432 op/s; 9790/849 objects degraded (1153.121%); 2742/849 objects misplaced (322.968%); 5.0 MiB/s, 90 objects/s recovering 2024-08-19T14:39:42.397 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:39:42.423 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:42.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:42.425 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:39:42.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:42 smithi032 ceph-mon[20004]: pgmap v1700: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 18 MiB data, 8.4 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 432 op/s; 9790/849 objects degraded (1153.121%); 2742/849 objects misplaced (322.968%); 5.0 MiB/s, 90 objects/s recovering 2024-08-19T14:39:42.526 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:39:42.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:42 smithi045 ceph-mon[20143]: pgmap v1700: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 18 MiB data, 8.4 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 432 op/s; 9790/849 objects degraded (1153.121%); 2742/849 objects misplaced (322.968%); 5.0 MiB/s, 90 objects/s recovering 2024-08-19T14:39:42.551 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:42.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:42.552 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:39:42.653 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:39:42.679 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:42.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:42.680 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:39:42.781 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:39:42.807 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:42.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:42.808 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:39:42.909 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:39:42.934 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:42.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:42.935 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:39:43.036 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:39:43.061 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:43.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:43.063 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:39:43.164 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:39:43.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:43.189 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:43.190 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:39:43.291 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:39:43.317 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:43.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:43.318 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:39:43.419 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:39:43.446 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:43.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:43.447 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:39:43.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:43 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 9790/849 objects degraded (1153.121%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-19T14:39:43.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:43 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 9790/849 objects degraded (1153.121%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-19T14:39:43.548 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:39:43.573 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:43.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:43.574 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:39:43.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:43 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 9790/849 objects degraded (1153.121%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-19T14:39:43.675 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:39:43.701 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:43.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:43.702 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:39:43.803 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:39:43.830 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:43.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:43.831 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:39:43.932 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:39:43.958 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:43.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:43.959 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:39:44.060 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:39:44.087 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:44.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:44.089 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:39:44.189 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:39:44.216 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:44.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:44.217 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:39:44.318 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:39:44.344 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:44.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:44.345 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:39:44.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:44 smithi031 ceph-mon[16336]: pgmap v1701: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 577 KiB data, 8.4 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 441 op/s; 9331/6 objects degraded (155516.667%); 2742/6 objects misplaced (45700.000%); 5.2 MiB/s, 194 objects/s recovering 2024-08-19T14:39:44.446 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:39:44.471 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:44.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:44.472 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:39:44.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:44 smithi032 ceph-mon[20004]: pgmap v1701: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 577 KiB data, 8.4 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 441 op/s; 9331/6 objects degraded (155516.667%); 2742/6 objects misplaced (45700.000%); 5.2 MiB/s, 194 objects/s recovering 2024-08-19T14:39:44.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:44 smithi045 ceph-mon[20143]: pgmap v1701: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 577 KiB data, 8.4 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 441 op/s; 9331/6 objects degraded (155516.667%); 2742/6 objects misplaced (45700.000%); 5.2 MiB/s, 194 objects/s recovering 2024-08-19T14:39:44.573 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:39:44.598 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:44.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:44.599 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:39:44.700 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:39:44.726 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:44.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:44.727 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:39:44.828 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:39:44.853 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:44.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:44.855 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:39:44.956 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:39:44.983 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:44.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:44.984 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:39:45.085 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:39:45.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:45.113 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:45.113 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:39:45.215 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:39:45.240 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:45.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:45.241 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:39:45.342 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:39:45.368 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:45.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:45.369 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:39:45.470 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:39:45.495 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:45.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:45.496 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:39:45.597 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:39:45.621 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:45.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:45.622 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:39:45.723 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:39:45.749 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:45.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:45.750 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:39:45.851 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:39:45.873 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:45.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:45.875 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:39:45.976 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:39:46.001 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:46.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:46.002 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:39:46.103 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:39:46.129 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:46.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:46.130 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:39:46.164 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:39:46.231 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:39:46.257 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:46.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:46.258 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:39:46.359 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:39:46.385 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:46.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:46.386 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:39:46.487 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:39:46.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:46 smithi032 ceph-mon[20004]: pgmap v1702: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 577 KiB data, 8.4 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 282 op/s; 9005/6 objects degraded (150083.333%); 2416/6 objects misplaced (40266.667%); 3.3 MiB/s, 205 objects/s recovering 2024-08-19T14:39:46.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:46 smithi032 ceph-mon[20004]: 7.4 scrub starts 2024-08-19T14:39:46.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:46 smithi032 ceph-mon[20004]: 7.4 scrub ok 2024-08-19T14:39:46.514 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:46.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:46.515 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:39:46.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:46 smithi045 ceph-mon[20143]: pgmap v1702: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 577 KiB data, 8.4 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 282 op/s; 9005/6 objects degraded (150083.333%); 2416/6 objects misplaced (40266.667%); 3.3 MiB/s, 205 objects/s recovering 2024-08-19T14:39:46.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:46 smithi045 ceph-mon[20143]: 7.4 scrub starts 2024-08-19T14:39:46.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:46 smithi045 ceph-mon[20143]: 7.4 scrub ok 2024-08-19T14:39:46.615 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:39:46.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:46 smithi031 ceph-mon[16336]: pgmap v1702: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 577 KiB data, 8.4 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 282 op/s; 9005/6 objects degraded (150083.333%); 2416/6 objects misplaced (40266.667%); 3.3 MiB/s, 205 objects/s recovering 2024-08-19T14:39:46.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:46 smithi031 ceph-mon[16336]: 7.4 scrub starts 2024-08-19T14:39:46.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:46 smithi031 ceph-mon[16336]: 7.4 scrub ok 2024-08-19T14:39:46.641 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:46.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:46.642 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:39:46.743 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:39:46.769 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:46.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:46.770 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:39:46.871 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:39:46.898 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:46.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:46.900 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:39:47.001 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:39:47.026 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:47.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:47.027 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:39:47.128 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:39:47.149 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:47.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:47.150 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:39:47.251 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:39:47.280 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:47.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:47.281 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:39:47.382 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:39:47.408 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:47.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:47.409 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:39:47.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:47 smithi032 ceph-mon[20004]: 1.0 scrub starts 2024-08-19T14:39:47.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:47 smithi032 ceph-mon[20004]: 1.0 scrub ok 2024-08-19T14:39:47.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:47 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:39:47.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:47 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:39:47.509 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:39:47.533 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:47.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:47.534 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:39:47.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:47 smithi045 ceph-mon[20143]: 1.0 scrub starts 2024-08-19T14:39:47.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:47 smithi045 ceph-mon[20143]: 1.0 scrub ok 2024-08-19T14:39:47.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:47 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:39:47.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:47 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:39:47.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:47 smithi031 ceph-mon[16336]: 1.0 scrub starts 2024-08-19T14:39:47.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:47 smithi031 ceph-mon[16336]: 1.0 scrub ok 2024-08-19T14:39:47.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:47 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:39:47.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:47 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:39:47.636 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:39:47.661 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:47.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:47.662 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:39:47.763 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:39:47.788 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:47.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:47.789 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:39:47.890 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:39:47.915 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:47.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:47.916 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:39:48.017 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:39:48.043 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:48.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:48.044 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:39:48.145 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:39:48.170 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:48.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:48.172 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:39:48.272 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:39:48.298 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:48.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:48.299 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:39:48.400 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:39:48.426 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:48.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:48.427 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:39:48.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:48 smithi032 ceph-mon[20004]: pgmap v1703: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 577 KiB data, 8.4 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 229 op/s; 7260/6 objects degraded (121000.000%); 2416/6 objects misplaced (40266.667%); 2.6 MiB/s, 338 objects/s recovering 2024-08-19T14:39:48.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:48 smithi032 ceph-mon[20004]: osdmap e708: 12 total, 12 up, 10 in 2024-08-19T14:39:48.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:48 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 7260/6 objects degraded (121000.000%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-19T14:39:48.528 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:39:48.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:48 smithi045 ceph-mon[20143]: pgmap v1703: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 577 KiB data, 8.4 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 229 op/s; 7260/6 objects degraded (121000.000%); 2416/6 objects misplaced (40266.667%); 2.6 MiB/s, 338 objects/s recovering 2024-08-19T14:39:48.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:48 smithi045 ceph-mon[20143]: osdmap e708: 12 total, 12 up, 10 in 2024-08-19T14:39:48.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:48 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 7260/6 objects degraded (121000.000%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-19T14:39:48.554 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:48.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:48.555 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:39:48.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:48 smithi031 ceph-mon[16336]: pgmap v1703: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 577 KiB data, 8.4 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 229 op/s; 7260/6 objects degraded (121000.000%); 2416/6 objects misplaced (40266.667%); 2.6 MiB/s, 338 objects/s recovering 2024-08-19T14:39:48.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:48 smithi031 ceph-mon[16336]: osdmap e708: 12 total, 12 up, 10 in 2024-08-19T14:39:48.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:48 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 7260/6 objects degraded (121000.000%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-19T14:39:48.656 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:39:48.682 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:48.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:48.683 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:39:48.784 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:39:48.810 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:48.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:48.811 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:39:48.912 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:39:48.939 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:48.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:48.940 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:39:49.041 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:39:49.067 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:49.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:49.068 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:39:49.169 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:39:49.195 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:49.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:49.196 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:39:49.297 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:39:49.323 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:49.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:49.324 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:39:49.425 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:39:49.450 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:49.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:49.451 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:39:49.552 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:39:49.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:49.578 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:49.578 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:39:49.679 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:39:49.705 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:49.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:49.706 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:39:49.805 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:39:49.807 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:39:49.832 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:49.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:49.833 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:39:49.851 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":1696,"stamp":"2024-08-19T14:39:32.270934+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":222888504,"num_objects":3395,"num_object_clones":0,"num_object_copies":10185,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10134,"num_objects_misplaced":2974,"num_objects_unfound":0,"num_objects_dirty":3395,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":232923,"num_write_kb":7562049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51582,"num_bytes_recovered":3151303200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":165160,"ondisk_log_size":165160,"up":141,"acting":137,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":142,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":9380076,"kb_used_data":754140,"kb_used_omap":18,"kb_used_meta":8625709,"kb_avail":928071444,"statfs":{"total":959950356480,"available":950345158656,"internally_reserved":0,"allocated":772239360,"data_stored":741017588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18760,"internal_metadata":8832726712},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,2,0,7,7,3,4,0,0,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-316866560,"num_objects":-4835,"num_object_clones":0,"num_object_copies":-14505,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-713,"num_objects_misplaced":-116,"num_objects_unfound":0,"num_objects_dirty":-4835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4835,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":878,"num_bytes_recovered":57212928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.235963"},"pg_stats":[{"pgid":"7.6","version":"707'14890","reported_seq":15217,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-08-19T14:39:30.677246+0000","last_change":"2024-08-19T14:37:56.236536+0000","last_active":"2024-08-19T14:39:30.677246+0000","last_peered":"2024-08-19T14:39:30.677246+0000","last_clean":"2024-08-19T14:39:30.677246+0000","last_became_active":"2024-08-19T14:33:28.334317+0000","last_became_peered":"2024-08-19T14:33:28.334317+0000","last_unstale":"2024-08-19T14:39:30.677246+0000","last_undegraded":"2024-08-19T14:39:30.677246+0000","last_fullsized":"2024-08-19T14:39:30.677246+0000","mapping_epoch":600,"log_start":"615'4800","ondisk_log_start":"615'4800","created":600,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"684'11923","last_scrub_stamp":"2024-08-19T14:37:56.236394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:37:56.236394+0000","objects_scrubbed":3155,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:39:30.851907+0000","scrub_duration":1586,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10747904,"num_objects":164,"num_object_clones":0,"num_object_copies":492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":164,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14890,"num_write_kb":481728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"70'139","reported_seq":1733,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-08-19T14:39:26.056932+0000","last_change":"2024-08-19T14:37:57.701402+0000","last_active":"2024-08-19T14:39:26.056932+0000","last_peered":"2024-08-19T14:39:26.056932+0000","last_clean":"2024-08-19T14:39:26.056932+0000","last_became_active":"2024-08-19T14:35:13.950224+0000","last_became_peered":"2024-08-19T14:35:13.950224+0000","last_unstale":"2024-08-19T14:39:26.056932+0000","last_undegraded":"2024-08-19T14:39:26.056932+0000","last_fullsized":"2024-08-19T14:39:26.056932+0000","mapping_epoch":637,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":638,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:37:57.701290+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:19:58.072083+0000","last_clean_scrub_stamp":"2024-08-19T14:37:57.701290+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:39:30.852008+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,9],"acting":[0,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.5","version":"707'14533","reported_seq":24038,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-08-19T14:39:31.738376+0000","last_change":"2024-08-19T14:35:23.022361+0000","last_active":"2024-08-19T14:39:31.738376+0000","last_peered":"2024-08-19T14:39:31.738376+0000","last_clean":"2024-08-19T14:39:31.738376+0000","last_became_active":"2024-08-19T14:35:23.009238+0000","last_became_peered":"2024-08-19T14:35:23.009238+0000","last_unstale":"2024-08-19T14:39:31.738376+0000","last_undegraded":"2024-08-19T14:39:31.738376+0000","last_fullsized":"2024-08-19T14:39:31.738376+0000","mapping_epoch":641,"log_start":"614'4500","ondisk_log_start":"614'4500","created":600,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9764864,"num_objects":149,"num_object_clones":0,"num_object_copies":447,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":149,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14533,"num_write_kb":469824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4634,"num_bytes_recovered":303693824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,0],"acting":[11,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.3","version":"70'139","reported_seq":1577,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-08-19T14:39:26.062149+0000","last_change":"2024-08-19T14:33:18.804321+0000","last_active":"2024-08-19T14:39:26.062149+0000","last_peered":"2024-08-19T14:39:26.062149+0000","last_clean":"2024-08-19T14:39:26.062149+0000","last_became_active":"2024-08-19T14:18:13.804721+0000","last_became_peered":"2024-08-19T14:18:13.804721+0000","last_unstale":"2024-08-19T14:39:26.062149+0000","last_undegraded":"2024-08-19T14:39:26.062149+0000","last_fullsized":"2024-08-19T14:39:26.062149+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:33:18.804228+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:33:18.804228+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.4","version":"707'14755","reported_seq":15079,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-08-19T14:39:30.705381+0000","last_change":"2024-08-19T14:38:06.717577+0000","last_active":"2024-08-19T14:39:30.705381+0000","last_peered":"2024-08-19T14:39:30.705381+0000","last_clean":"2024-08-19T14:39:30.705381+0000","last_became_active":"2024-08-19T14:33:28.334446+0000","last_became_peered":"2024-08-19T14:33:28.334446+0000","last_unstale":"2024-08-19T14:39:30.705381+0000","last_undegraded":"2024-08-19T14:39:30.705381+0000","last_fullsized":"2024-08-19T14:39:30.705381+0000","mapping_epoch":600,"log_start":"614'4700","ondisk_log_start":"614'4700","created":600,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"686'12159","last_scrub_stamp":"2024-08-19T14:38:06.717004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:38:06.717004+0000","objects_scrubbed":2792,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:39:30.851945+0000","scrub_duration":1017,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12517376,"num_objects":191,"num_object_clones":0,"num_object_copies":573,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14755,"num_write_kb":478272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"70'139","reported_seq":1533,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-08-19T14:39:26.057422+0000","last_change":"2024-08-19T14:34:55.580212+0000","last_active":"2024-08-19T14:39:26.057422+0000","last_peered":"2024-08-19T14:39:26.057422+0000","last_clean":"2024-08-19T14:39:26.057422+0000","last_became_active":"2024-08-19T14:20:50.742354+0000","last_became_peered":"2024-08-19T14:20:50.742354+0000","last_unstale":"2024-08-19T14:39:26.057422+0000","last_undegraded":"2024-08-19T14:39:26.057422+0000","last_fullsized":"2024-08-19T14:39:26.057422+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:34:55.580141+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:19:59.302931+0000","last_clean_scrub_stamp":"2024-08-19T14:34:55.580141+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"707'14289","reported_seq":14615,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-08-19T14:39:30.281458+0000","last_change":"2024-08-19T14:37:43.229201+0000","last_active":"2024-08-19T14:39:30.281458+0000","last_peered":"2024-08-19T14:39:30.281458+0000","last_clean":"2024-08-19T14:39:30.281458+0000","last_became_active":"2024-08-19T14:33:28.331833+0000","last_became_peered":"2024-08-19T14:33:28.331833+0000","last_unstale":"2024-08-19T14:39:30.281458+0000","last_undegraded":"2024-08-19T14:39:30.281458+0000","last_fullsized":"2024-08-19T14:39:30.281458+0000","mapping_epoch":600,"log_start":"614'4200","ondisk_log_start":"614'4200","created":600,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"678'11114","last_scrub_stamp":"2024-08-19T14:37:43.229072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:37:43.229072+0000","objects_scrubbed":3413,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":910,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15007744,"num_objects":229,"num_object_clones":0,"num_object_copies":687,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":229,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14289,"num_write_kb":464576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,1],"acting":[8,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1","version":"70'139","reported_seq":831,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-08-19T14:39:26.066108+0000","last_change":"2024-08-19T14:37:42.630047+0000","last_active":"2024-08-19T14:39:26.066108+0000","last_peered":"2024-08-19T14:39:26.066108+0000","last_clean":"2024-08-19T14:39:26.066108+0000","last_became_active":"2024-08-19T14:35:12.530566+0000","last_became_peered":"2024-08-19T14:35:12.530566+0000","last_unstale":"2024-08-19T14:39:26.066108+0000","last_undegraded":"2024-08-19T14:39:26.066108+0000","last_fullsized":"2024-08-19T14:39:26.066108+0000","mapping_epoch":635,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":636,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:37:42.629919+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:37:42.629919+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:39:30.851857+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,2],"acting":[0,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"707'14325","reported_seq":14628,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-08-19T14:39:28.695566+0000","last_change":"2024-08-19T14:34:52.869444+0000","last_active":"2024-08-19T14:39:28.695566+0000","last_peered":"2024-08-19T14:39:28.695566+0000","last_clean":"2024-08-19T14:39:28.695566+0000","last_became_active":"2024-08-19T14:33:28.332994+0000","last_became_peered":"2024-08-19T14:33:28.332994+0000","last_unstale":"2024-08-19T14:39:28.695566+0000","last_undegraded":"2024-08-19T14:39:28.695566+0000","last_fullsized":"2024-08-19T14:39:28.695566+0000","mapping_epoch":600,"log_start":"614'4300","ondisk_log_start":"614'4300","created":600,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"627'6737","last_scrub_stamp":"2024-08-19T14:34:52.869233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:34:52.869233+0000","objects_scrubbed":6705,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1588,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16580608,"num_objects":253,"num_object_clones":0,"num_object_copies":759,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14325,"num_write_kb":466496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"70'139","reported_seq":1235,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-08-19T14:39:26.066464+0000","last_change":"2024-08-19T14:35:11.537368+0000","last_active":"2024-08-19T14:39:26.066464+0000","last_peered":"2024-08-19T14:39:26.066464+0000","last_clean":"2024-08-19T14:39:26.066464+0000","last_became_active":"2024-08-19T14:35:11.536828+0000","last_became_peered":"2024-08-19T14:35:11.536828+0000","last_unstale":"2024-08-19T14:39:26.066464+0000","last_undegraded":"2024-08-19T14:39:26.066464+0000","last_fullsized":"2024-08-19T14:39:26.066464+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:27.536928+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:21:46.057624+0000","last_clean_scrub_stamp":"2024-08-19T14:28:27.536928+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"707'14351","reported_seq":23810,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-08-19T14:39:30.666004+0000","last_change":"2024-08-19T14:36:15.120872+0000","last_active":"2024-08-19T14:39:30.666004+0000","last_peered":"2024-08-19T14:39:30.666004+0000","last_clean":"2024-08-19T14:39:30.666004+0000","last_became_active":"2024-08-19T14:36:15.096070+0000","last_became_peered":"2024-08-19T14:36:15.096070+0000","last_unstale":"2024-08-19T14:39:30.666004+0000","last_undegraded":"2024-08-19T14:39:30.666004+0000","last_fullsized":"2024-08-19T14:39:30.666004+0000","mapping_epoch":653,"log_start":"614'4300","ondisk_log_start":"614'4300","created":600,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13697024,"num_objects":209,"num_object_clones":0,"num_object_copies":627,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":209,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14351,"num_write_kb":465920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4611,"num_bytes_recovered":302186496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,9],"acting":[3,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"70'139","reported_seq":803,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-08-19T14:39:26.053643+0000","last_change":"2024-08-19T14:37:45.357396+0000","last_active":"2024-08-19T14:39:26.053643+0000","last_peered":"2024-08-19T14:39:26.053643+0000","last_clean":"2024-08-19T14:39:26.053643+0000","last_became_active":"2024-08-19T14:34:15.717054+0000","last_became_peered":"2024-08-19T14:34:15.717054+0000","last_unstale":"2024-08-19T14:39:26.053643+0000","last_undegraded":"2024-08-19T14:39:26.053643+0000","last_fullsized":"2024-08-19T14:39:26.053643+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:37:45.357293+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:27.848870+0000","last_clean_scrub_stamp":"2024-08-19T14:37:45.357293+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:39:27.416697+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"7.3","version":"707'14844","reported_seq":24521,"reported_epoch":707,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-19T14:39:30.705983+0000","last_change":"2024-08-19T14:35:13.136988+0000","last_active":"2024-08-19T14:39:30.705983+0000","last_peered":"2024-08-19T14:39:30.705983+0000","last_clean":"2024-08-19T14:33:27.312789+0000","last_became_active":"2024-08-19T14:35:13.021078+0000","last_became_peered":"2024-08-19T14:35:13.021078+0000","last_unstale":"2024-08-19T14:39:30.705983+0000","last_undegraded":"2024-08-19T14:35:12.592288+0000","last_fullsized":"2024-08-19T14:35:12.526971+0000","mapping_epoch":635,"log_start":"615'4800","ondisk_log_start":"615'4800","created":600,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13762560,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2533,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14844,"num_write_kb":481728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4725,"num_bytes_recovered":306184192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":-2323}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"70'139","reported_seq":758,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-08-19T14:39:26.060951+0000","last_change":"2024-08-19T14:35:11.533207+0000","last_active":"2024-08-19T14:39:26.060951+0000","last_peered":"2024-08-19T14:39:26.060951+0000","last_clean":"2024-08-19T14:39:26.060951+0000","last_became_active":"2024-08-19T14:35:11.532858+0000","last_became_peered":"2024-08-19T14:35:11.532858+0000","last_unstale":"2024-08-19T14:39:26.060951+0000","last_undegraded":"2024-08-19T14:39:26.060951+0000","last_fullsized":"2024-08-19T14:39:26.060951+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:27.331041+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:28:27.331041+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,0],"acting":[11,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.2","version":"707'14782","reported_seq":24207,"reported_epoch":707,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-19T14:39:30.702686+0000","last_change":"2024-08-19T14:35:13.121204+0000","last_active":"2024-08-19T14:39:30.702686+0000","last_peered":"2024-08-19T14:39:30.702686+0000","last_clean":"2024-08-19T14:33:27.312789+0000","last_became_active":"2024-08-19T14:35:13.013347+0000","last_became_peered":"2024-08-19T14:35:13.013347+0000","last_unstale":"2024-08-19T14:39:30.702686+0000","last_undegraded":"2024-08-19T14:35:12.598259+0000","last_fullsized":"2024-08-19T14:35:12.526602+0000","mapping_epoch":635,"log_start":"614'4700","ondisk_log_start":"614'4700","created":600,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12976128,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2636,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14782,"num_write_kb":479360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4599,"num_bytes_recovered":298254336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":-2438}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"70'139","reported_seq":635,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-08-19T14:39:26.062348+0000","last_change":"2024-08-19T14:33:16.760928+0000","last_active":"2024-08-19T14:39:26.062348+0000","last_peered":"2024-08-19T14:39:26.062348+0000","last_clean":"2024-08-19T14:39:26.062348+0000","last_became_active":"2024-08-19T14:30:47.909067+0000","last_became_peered":"2024-08-19T14:30:47.909067+0000","last_unstale":"2024-08-19T14:39:26.062348+0000","last_undegraded":"2024-08-19T14:39:26.062348+0000","last_fullsized":"2024-08-19T14:39:26.062348+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:33:16.760810+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:33:16.760810+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.c","version":"707'14350","reported_seq":24083,"reported_epoch":707,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-19T14:39:30.714715+0000","last_change":"2024-08-19T14:35:13.133729+0000","last_active":"2024-08-19T14:39:30.714715+0000","last_peered":"2024-08-19T14:39:30.714715+0000","last_clean":"2024-08-19T14:33:27.312789+0000","last_became_active":"2024-08-19T14:35:13.003266+0000","last_became_peered":"2024-08-19T14:35:13.003266+0000","last_unstale":"2024-08-19T14:39:30.714715+0000","last_undegraded":"2024-08-19T14:35:12.593502+0000","last_fullsized":"2024-08-19T14:35:12.526375+0000","mapping_epoch":635,"log_start":"614'4300","ondisk_log_start":"614'4300","created":600,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13500416,"num_objects":206,"num_object_clones":0,"num_object_copies":618,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2256,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":206,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14350,"num_write_kb":465792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4753,"num_bytes_recovered":309657600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,2],"acting":[3,11],"avail_no_missing":["3","11"],"object_location_counts":[{"shards":"3,11","objects":-2050}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"70'139","reported_seq":800,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-08-19T14:39:26.052767+0000","last_change":"2024-08-19T14:34:50.256878+0000","last_active":"2024-08-19T14:39:26.052767+0000","last_peered":"2024-08-19T14:39:26.052767+0000","last_clean":"2024-08-19T14:39:26.052767+0000","last_became_active":"2024-08-19T14:34:15.717472+0000","last_became_peered":"2024-08-19T14:34:15.717472+0000","last_unstale":"2024-08-19T14:39:26.052767+0000","last_undegraded":"2024-08-19T14:39:26.052767+0000","last_fullsized":"2024-08-19T14:39:26.052767+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:34:50.256808+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:28.864870+0000","last_clean_scrub_stamp":"2024-08-19T14:34:50.256808+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,0],"acting":[4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"707'18010","reported_seq":27864,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-08-19T14:39:30.688884+0000","last_change":"2024-08-19T14:38:47.357628+0000","last_active":"2024-08-19T14:39:30.688884+0000","last_peered":"2024-08-19T14:39:30.688884+0000","last_clean":"2024-08-19T14:39:30.688884+0000","last_became_active":"2024-08-19T14:36:14.098166+0000","last_became_peered":"2024-08-19T14:36:14.098166+0000","last_unstale":"2024-08-19T14:39:30.688884+0000","last_undegraded":"2024-08-19T14:39:30.688884+0000","last_fullsized":"2024-08-19T14:39:30.688884+0000","mapping_epoch":652,"log_start":"644'8002","ondisk_log_start":"644'8002","created":600,"last_epoch_clean":653,"parent":"0.0","parent_split_bits":0,"last_scrub":"696'16654","last_scrub_stamp":"2024-08-19T14:38:47.357047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:38:47.357047+0000","objects_scrubbed":1549,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:40:09.053409+0000","scrub_duration":564,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12189696,"num_objects":186,"num_object_clones":0,"num_object_copies":558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14380,"num_write_kb":466112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4700,"num_bytes_recovered":308019200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.9","version":"70'139","reported_seq":1736,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-08-19T14:39:26.057345+0000","last_change":"2024-08-19T14:35:12.529993+0000","last_active":"2024-08-19T14:39:26.057345+0000","last_peered":"2024-08-19T14:39:26.057345+0000","last_clean":"2024-08-19T14:39:26.057345+0000","last_became_active":"2024-08-19T14:35:12.529628+0000","last_became_peered":"2024-08-19T14:35:12.529628+0000","last_unstale":"2024-08-19T14:39:26.057345+0000","last_undegraded":"2024-08-19T14:39:26.057345+0000","last_fullsized":"2024-08-19T14:39:26.057345+0000","mapping_epoch":635,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":636,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:34:51.561893+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:34:51.561893+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.e","version":"707'14531","reported_seq":29546,"reported_epoch":707,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-19T14:39:27.896677+0000","last_change":"2024-08-19T14:35:11.863818+0000","last_active":"2024-08-19T14:39:27.896677+0000","last_peered":"2024-08-19T14:39:27.896677+0000","last_clean":"2024-08-19T14:33:27.312789+0000","last_became_active":"2024-08-19T14:35:11.806304+0000","last_became_peered":"2024-08-19T14:35:11.806304+0000","last_unstale":"2024-08-19T14:39:27.896677+0000","last_undegraded":"2024-08-19T14:35:11.779866+0000","last_fullsized":"2024-08-19T14:35:11.528207+0000","mapping_epoch":634,"log_start":"614'4500","ondisk_log_start":"614'4500","created":600,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17367040,"num_objects":265,"num_object_clones":0,"num_object_copies":795,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2709,"num_objects_misplaced":2974,"num_objects_unfound":0,"num_objects_dirty":265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14531,"num_write_kb":473472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5778,"num_bytes_recovered":323747840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,5],"acting":[1,5],"avail_no_missing":["1","5","9"],"object_location_counts":[{"shards":"1,5","objects":-2709},{"shards":"1,5,9","objects":265}],"blocked_by":[],"up_primary":2,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"70'139","reported_seq":766,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-08-19T14:39:26.063682+0000","last_change":"2024-08-19T14:35:11.537896+0000","last_active":"2024-08-19T14:39:26.063682+0000","last_peered":"2024-08-19T14:39:26.063682+0000","last_clean":"2024-08-19T14:39:26.063682+0000","last_became_active":"2024-08-19T14:35:11.537474+0000","last_became_peered":"2024-08-19T14:35:11.537474+0000","last_unstale":"2024-08-19T14:39:26.063682+0000","last_undegraded":"2024-08-19T14:39:26.063682+0000","last_fullsized":"2024-08-19T14:39:26.063682+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:59.829567+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:28:59.829567+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"707'14443","reported_seq":14670,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-08-19T14:39:27.878428+0000","last_change":"2024-08-19T14:33:28.335586+0000","last_active":"2024-08-19T14:39:27.878428+0000","last_peered":"2024-08-19T14:39:27.878428+0000","last_clean":"2024-08-19T14:39:27.878428+0000","last_became_active":"2024-08-19T14:33:28.335298+0000","last_became_peered":"2024-08-19T14:33:28.335298+0000","last_unstale":"2024-08-19T14:39:27.878428+0000","last_undegraded":"2024-08-19T14:39:27.878428+0000","last_fullsized":"2024-08-19T14:39:27.878428+0000","mapping_epoch":600,"log_start":"614'4400","ondisk_log_start":"614'4400","created":600,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18481176,"num_objects":283,"num_object_clones":0,"num_object_copies":849,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":283,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14443,"num_write_kb":471169,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"70'139","reported_seq":1086,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-08-19T14:39:26.060791+0000","last_change":"2024-08-19T14:33:15.753668+0000","last_active":"2024-08-19T14:39:26.060791+0000","last_peered":"2024-08-19T14:39:26.060791+0000","last_clean":"2024-08-19T14:39:26.060791+0000","last_became_active":"2024-08-19T14:30:47.941200+0000","last_became_peered":"2024-08-19T14:30:47.941200+0000","last_unstale":"2024-08-19T14:39:26.060791+0000","last_undegraded":"2024-08-19T14:39:26.060791+0000","last_fullsized":"2024-08-19T14:39:26.060791+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:33:15.753595+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:33:15.753595+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.8","version":"707'14867","reported_seq":30211,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-08-19T14:39:30.393908+0000","last_change":"2024-08-19T14:38:13.418089+0000","last_active":"2024-08-19T14:39:30.393908+0000","last_peered":"2024-08-19T14:39:30.393908+0000","last_clean":"2024-08-19T14:39:30.393908+0000","last_became_active":"2024-08-19T14:38:13.389656+0000","last_became_peered":"2024-08-19T14:38:13.389656+0000","last_unstale":"2024-08-19T14:39:30.393908+0000","last_undegraded":"2024-08-19T14:39:30.393908+0000","last_fullsized":"2024-08-19T14:39:30.393908+0000","mapping_epoch":688,"log_start":"615'4800","ondisk_log_start":"615'4800","created":600,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15925248,"num_objects":243,"num_object_clones":0,"num_object_copies":729,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":243,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14867,"num_write_kb":483520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7555,"num_bytes_recovered":476577792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.e","version":"70'139","reported_seq":1277,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-08-19T14:39:26.053008+0000","last_change":"2024-08-19T14:34:55.333177+0000","last_active":"2024-08-19T14:39:26.053008+0000","last_peered":"2024-08-19T14:39:26.053008+0000","last_clean":"2024-08-19T14:39:26.053008+0000","last_became_active":"2024-08-19T14:18:26.884105+0000","last_became_peered":"2024-08-19T14:18:26.884105+0000","last_unstale":"2024-08-19T14:39:26.053008+0000","last_undegraded":"2024-08-19T14:39:26.053008+0000","last_fullsized":"2024-08-19T14:39:26.053008+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:34:55.333065+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:32:31.196288+0000","last_clean_scrub_stamp":"2024-08-19T14:34:55.333065+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"707'14682","reported_seq":15044,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-08-19T14:39:30.709771+0000","last_change":"2024-08-19T14:37:54.857799+0000","last_active":"2024-08-19T14:39:30.709771+0000","last_peered":"2024-08-19T14:39:30.709771+0000","last_clean":"2024-08-19T14:39:30.709771+0000","last_became_active":"2024-08-19T14:33:28.334547+0000","last_became_peered":"2024-08-19T14:33:28.334547+0000","last_unstale":"2024-08-19T14:39:30.709771+0000","last_undegraded":"2024-08-19T14:39:30.709771+0000","last_fullsized":"2024-08-19T14:39:30.709771+0000","mapping_epoch":600,"log_start":"614'4600","ondisk_log_start":"614'4600","created":600,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"683'11737","last_scrub_stamp":"2024-08-19T14:37:54.851356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:37:54.851356+0000","objects_scrubbed":3161,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1254,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12976128,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14682,"num_write_kb":476160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,8],"acting":[0,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"70'139","reported_seq":1029,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-08-19T14:39:26.053127+0000","last_change":"2024-08-19T14:34:12.199530+0000","last_active":"2024-08-19T14:39:26.053127+0000","last_peered":"2024-08-19T14:39:26.053127+0000","last_clean":"2024-08-19T14:39:26.053127+0000","last_became_active":"2024-08-19T14:27:31.837754+0000","last_became_peered":"2024-08-19T14:27:31.837754+0000","last_unstale":"2024-08-19T14:39:26.053127+0000","last_undegraded":"2024-08-19T14:39:26.053127+0000","last_fullsized":"2024-08-19T14:39:26.053127+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:34:12.199449+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:34:12.199449+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.a","version":"707'14579","reported_seq":37120,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-08-19T14:39:31.705444+0000","last_change":"2024-08-19T14:39:09.001385+0000","last_active":"2024-08-19T14:39:31.705444+0000","last_peered":"2024-08-19T14:39:31.705444+0000","last_clean":"2024-08-19T14:39:31.705444+0000","last_became_active":"2024-08-19T14:39:08.973126+0000","last_became_peered":"2024-08-19T14:39:08.973126+0000","last_unstale":"2024-08-19T14:39:31.705444+0000","last_undegraded":"2024-08-19T14:39:31.705444+0000","last_fullsized":"2024-08-19T14:39:31.705444+0000","mapping_epoch":701,"log_start":"614'4500","ondisk_log_start":"614'4500","created":600,"last_epoch_clean":702,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9633792,"num_objects":147,"num_object_clones":0,"num_object_copies":441,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":147,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14579,"num_write_kb":471232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10224,"num_bytes_recovered":522125312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,2],"acting":[11,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.c","version":"70'139","reported_seq":1384,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-08-19T14:39:26.065980+0000","last_change":"2024-08-19T14:34:15.738488+0000","last_active":"2024-08-19T14:39:26.065980+0000","last_peered":"2024-08-19T14:39:26.065980+0000","last_clean":"2024-08-19T14:39:26.065980+0000","last_became_active":"2024-08-19T14:34:15.733282+0000","last_became_peered":"2024-08-19T14:34:15.733282+0000","last_unstale":"2024-08-19T14:39:26.065980+0000","last_undegraded":"2024-08-19T14:39:26.065980+0000","last_fullsized":"2024-08-19T14:39:26.065980+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:33:17.453290+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:22:52.164586+0000","last_clean_scrub_stamp":"2024-08-19T14:33:17.453290+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,9],"acting":[3,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"707'14322","reported_seq":14548,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-08-19T14:39:27.892201+0000","last_change":"2024-08-19T14:33:28.623717+0000","last_active":"2024-08-19T14:39:27.892201+0000","last_peered":"2024-08-19T14:39:27.892201+0000","last_clean":"2024-08-19T14:39:27.892201+0000","last_became_active":"2024-08-19T14:33:28.623318+0000","last_became_peered":"2024-08-19T14:33:28.623318+0000","last_unstale":"2024-08-19T14:39:27.892201+0000","last_undegraded":"2024-08-19T14:39:27.892201+0000","last_fullsized":"2024-08-19T14:39:27.892201+0000","mapping_epoch":600,"log_start":"614'4300","ondisk_log_start":"614'4300","created":600,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:33:27.312789+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.312789+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17170432,"num_objects":262,"num_object_clones":0,"num_object_copies":786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14322,"num_write_kb":466688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"70'139","reported_seq":1281,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-08-19T14:39:26.058074+0000","last_change":"2024-08-19T14:34:49.476489+0000","last_active":"2024-08-19T14:39:26.058074+0000","last_peered":"2024-08-19T14:39:26.058074+0000","last_clean":"2024-08-19T14:39:26.058074+0000","last_became_active":"2024-08-19T14:18:26.882002+0000","last_became_peered":"2024-08-19T14:18:26.882002+0000","last_unstale":"2024-08-19T14:39:26.058074+0000","last_undegraded":"2024-08-19T14:39:26.058074+0000","last_fullsized":"2024-08-19T14:39:26.058074+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:34:49.476415+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:24:50.276408+0000","last_clean_scrub_stamp":"2024-08-19T14:34:49.476415+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"70'139","reported_seq":1060,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-08-19T14:39:26.061977+0000","last_change":"2024-08-19T14:33:19.781527+0000","last_active":"2024-08-19T14:39:26.061977+0000","last_peered":"2024-08-19T14:39:26.061977+0000","last_clean":"2024-08-19T14:39:26.061977+0000","last_became_active":"2024-08-19T14:27:31.838968+0000","last_became_peered":"2024-08-19T14:27:31.838968+0000","last_unstale":"2024-08-19T14:39:26.061977+0000","last_undegraded":"2024-08-19T14:39:26.061977+0000","last_fullsized":"2024-08-19T14:39:26.061977+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:33:19.781431+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:28:35.206115+0000","last_clean_scrub_stamp":"2024-08-19T14:33:19.781431+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.13","version":"70'139","reported_seq":1602,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-08-19T14:39:26.062048+0000","last_change":"2024-08-19T14:33:35.786236+0000","last_active":"2024-08-19T14:39:26.062048+0000","last_peered":"2024-08-19T14:39:26.062048+0000","last_clean":"2024-08-19T14:39:26.062048+0000","last_became_active":"2024-08-19T14:30:47.939849+0000","last_became_peered":"2024-08-19T14:30:47.939849+0000","last_unstale":"2024-08-19T14:39:26.062048+0000","last_undegraded":"2024-08-19T14:39:26.062048+0000","last_fullsized":"2024-08-19T14:39:26.062048+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:33:35.786105+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:33:35.786105+0000","last_clean_scrub_stamp":"2024-08-19T14:33:35.786105+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.10","version":"70'139","reported_seq":1230,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-08-19T14:39:26.057553+0000","last_change":"2024-08-19T14:34:58.616722+0000","last_active":"2024-08-19T14:39:26.057553+0000","last_peered":"2024-08-19T14:39:26.057553+0000","last_clean":"2024-08-19T14:39:26.057553+0000","last_became_active":"2024-08-19T14:27:31.839611+0000","last_became_peered":"2024-08-19T14:27:31.839611+0000","last_unstale":"2024-08-19T14:39:26.057553+0000","last_undegraded":"2024-08-19T14:39:26.057553+0000","last_fullsized":"2024-08-19T14:39:26.057553+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:34:58.616646+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:34:58.616646+0000","last_clean_scrub_stamp":"2024-08-19T14:34:58.616646+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.11","version":"70'139","reported_seq":1286,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-08-19T14:39:26.065981+0000","last_change":"2024-08-19T14:38:34.904558+0000","last_active":"2024-08-19T14:39:26.065981+0000","last_peered":"2024-08-19T14:39:26.065981+0000","last_clean":"2024-08-19T14:39:26.065981+0000","last_became_active":"2024-08-19T14:34:15.731049+0000","last_became_peered":"2024-08-19T14:34:15.731049+0000","last_unstale":"2024-08-19T14:39:26.065981+0000","last_undegraded":"2024-08-19T14:39:26.065981+0000","last_fullsized":"2024-08-19T14:39:26.065981+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:38:34.904308+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:24:50.547428+0000","last_clean_scrub_stamp":"2024-08-19T14:38:34.904308+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:40:02.722075+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"70'139","reported_seq":1212,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-08-19T14:39:26.071410+0000","last_change":"2024-08-19T14:33:27.737496+0000","last_active":"2024-08-19T14:39:26.071410+0000","last_peered":"2024-08-19T14:39:26.071410+0000","last_clean":"2024-08-19T14:39:26.071410+0000","last_became_active":"2024-08-19T14:20:28.204724+0000","last_became_peered":"2024-08-19T14:20:28.204724+0000","last_unstale":"2024-08-19T14:39:26.071410+0000","last_undegraded":"2024-08-19T14:39:26.071410+0000","last_fullsized":"2024-08-19T14:39:26.071410+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:33:27.737367+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:28:34.216370+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.737367+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"70'139","reported_seq":1363,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-08-19T14:39:26.053054+0000","last_change":"2024-08-19T14:38:46.389978+0000","last_active":"2024-08-19T14:39:26.053054+0000","last_peered":"2024-08-19T14:39:26.053054+0000","last_clean":"2024-08-19T14:39:26.053054+0000","last_became_active":"2024-08-19T14:18:31.901556+0000","last_became_peered":"2024-08-19T14:18:31.901556+0000","last_unstale":"2024-08-19T14:39:26.053054+0000","last_undegraded":"2024-08-19T14:39:26.053054+0000","last_fullsized":"2024-08-19T14:39:26.053054+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:38:46.389862+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:38:46.389862+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:40:08.907590+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,0],"acting":[8,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.14","version":"70'139","reported_seq":1385,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-08-19T14:39:26.057439+0000","last_change":"2024-08-19T14:35:12.530145+0000","last_active":"2024-08-19T14:39:26.057439+0000","last_peered":"2024-08-19T14:39:26.057439+0000","last_clean":"2024-08-19T14:39:26.057439+0000","last_became_active":"2024-08-19T14:35:12.529792+0000","last_became_peered":"2024-08-19T14:35:12.529792+0000","last_unstale":"2024-08-19T14:39:26.057439+0000","last_undegraded":"2024-08-19T14:39:26.057439+0000","last_fullsized":"2024-08-19T14:39:26.057439+0000","mapping_epoch":635,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":636,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:34:53.617013+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:34:53.617013+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,9],"acting":[7,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.15","version":"70'139","reported_seq":1236,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-08-19T14:39:26.058063+0000","last_change":"2024-08-19T14:34:57.640316+0000","last_active":"2024-08-19T14:39:26.058063+0000","last_peered":"2024-08-19T14:39:26.058063+0000","last_clean":"2024-08-19T14:39:26.058063+0000","last_became_active":"2024-08-19T14:18:31.912496+0000","last_became_peered":"2024-08-19T14:18:31.912496+0000","last_unstale":"2024-08-19T14:39:26.058063+0000","last_undegraded":"2024-08-19T14:39:26.058063+0000","last_fullsized":"2024-08-19T14:39:26.058063+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:34:57.640213+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:20:09.180923+0000","last_clean_scrub_stamp":"2024-08-19T14:34:57.640213+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"70'139","reported_seq":1222,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-08-19T14:39:26.052720+0000","last_change":"2024-08-19T14:34:06.176510+0000","last_active":"2024-08-19T14:39:26.052720+0000","last_peered":"2024-08-19T14:39:26.052720+0000","last_clean":"2024-08-19T14:39:26.052720+0000","last_became_active":"2024-08-19T14:18:31.901229+0000","last_became_peered":"2024-08-19T14:18:31.901229+0000","last_unstale":"2024-08-19T14:39:26.052720+0000","last_undegraded":"2024-08-19T14:39:26.052720+0000","last_fullsized":"2024-08-19T14:39:26.052720+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:34:06.176403+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:34:06.176403+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1b","version":"70'139","reported_seq":1603,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-08-19T14:39:26.053160+0000","last_change":"2024-08-19T14:37:59.510774+0000","last_active":"2024-08-19T14:39:26.053160+0000","last_peered":"2024-08-19T14:39:26.053160+0000","last_clean":"2024-08-19T14:39:26.053160+0000","last_became_active":"2024-08-19T14:34:15.718427+0000","last_became_peered":"2024-08-19T14:34:15.718427+0000","last_unstale":"2024-08-19T14:39:26.053160+0000","last_undegraded":"2024-08-19T14:39:26.053160+0000","last_fullsized":"2024-08-19T14:39:26.053160+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:37:59.510697+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:37:59.510697+0000","last_clean_scrub_stamp":"2024-08-19T14:37:59.510697+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:39:27.416748+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.18","version":"70'139","reported_seq":1164,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-08-19T14:39:26.057286+0000","last_change":"2024-08-19T14:35:11.528963+0000","last_active":"2024-08-19T14:39:26.057286+0000","last_peered":"2024-08-19T14:39:26.057286+0000","last_clean":"2024-08-19T14:39:26.057286+0000","last_became_active":"2024-08-19T14:35:11.528530+0000","last_became_peered":"2024-08-19T14:35:11.528530+0000","last_unstale":"2024-08-19T14:39:26.057286+0000","last_undegraded":"2024-08-19T14:39:26.057286+0000","last_fullsized":"2024-08-19T14:39:26.057286+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:29:05.336156+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:27:33.084161+0000","last_clean_scrub_stamp":"2024-08-19T14:29:05.336156+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.19","version":"70'139","reported_seq":1582,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-08-19T14:39:26.066453+0000","last_change":"2024-08-19T14:37:41.671836+0000","last_active":"2024-08-19T14:39:26.066453+0000","last_peered":"2024-08-19T14:39:26.066453+0000","last_clean":"2024-08-19T14:39:26.066453+0000","last_became_active":"2024-08-19T14:18:31.902547+0000","last_became_peered":"2024-08-19T14:18:31.902547+0000","last_unstale":"2024-08-19T14:39:26.066453+0000","last_undegraded":"2024-08-19T14:39:26.066453+0000","last_fullsized":"2024-08-19T14:39:26.066453+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:37:41.671731+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:37:41.671731+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:39:30.851811+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1e","version":"70'139","reported_seq":607,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-08-19T14:39:26.060819+0000","last_change":"2024-08-19T14:33:17.778415+0000","last_active":"2024-08-19T14:39:26.060819+0000","last_peered":"2024-08-19T14:39:26.060819+0000","last_clean":"2024-08-19T14:39:26.060819+0000","last_became_active":"2024-08-19T14:30:47.909974+0000","last_became_peered":"2024-08-19T14:30:47.909974+0000","last_unstale":"2024-08-19T14:39:26.060819+0000","last_undegraded":"2024-08-19T14:39:26.060819+0000","last_fullsized":"2024-08-19T14:39:26.060819+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:33:17.778295+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:27:04.780939+0000","last_clean_scrub_stamp":"2024-08-19T14:33:17.778295+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1c","version":"70'139","reported_seq":1365,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-08-19T14:39:26.405772+0000","last_change":"2024-08-19T14:39:26.405807+0000","last_active":"2024-08-19T14:39:26.405772+0000","last_peered":"2024-08-19T14:39:26.405772+0000","last_clean":"2024-08-19T14:39:26.405772+0000","last_became_active":"2024-08-19T14:30:47.908866+0000","last_became_peered":"2024-08-19T14:30:47.908866+0000","last_unstale":"2024-08-19T14:39:26.405772+0000","last_undegraded":"2024-08-19T14:39:26.405772+0000","last_fullsized":"2024-08-19T14:39:26.405772+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:39:26.405699+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:39:26.405699+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:40:29.159585+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,11],"acting":[8,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1d","version":"70'139","reported_seq":1266,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-08-19T14:39:26.063504+0000","last_change":"2024-08-19T14:35:11.542467+0000","last_active":"2024-08-19T14:39:26.063504+0000","last_peered":"2024-08-19T14:39:26.063504+0000","last_clean":"2024-08-19T14:39:26.063504+0000","last_became_active":"2024-08-19T14:35:11.540488+0000","last_became_peered":"2024-08-19T14:35:11.540488+0000","last_unstale":"2024-08-19T14:39:26.063504+0000","last_undegraded":"2024-08-19T14:39:26.063504+0000","last_fullsized":"2024-08-19T14:39:26.063504+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:28:29.194639+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:28:29.194639+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":222298136,"num_objects":3393,"num_object_clones":0,"num_object_copies":10179,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10134,"num_objects_misplaced":2974,"num_objects_unfound":0,"num_objects_dirty":3393,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":232923,"num_write_kb":7562049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51579,"num_bytes_recovered":3150446592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":639709184,"data_stored":639696968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160851,"ondisk_log_size":160851,"up":48,"acting":44,"num_store_stats":10},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":856608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187794,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":931700,"kb_used_data":57836,"kb_used_omap":1,"kb_used_meta":873854,"kb_avail":92813452,"statfs":{"total":95995035648,"available":95040974848,"internally_reserved":0,"allocated":59224064,"data_stored":56104174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":894826948},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":2314987372661,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":74,"seq":317827580251,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":933300,"kb_used_data":94124,"kb_used_omap":1,"kb_used_meta":839166,"kb_avail":92811852,"statfs":{"total":95995035648,"available":95039336448,"internally_reserved":0,"allocated":96382976,"data_stored":93263086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":859306433},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874594,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":834036,"kb_used_data":54572,"kb_used_omap":1,"kb_used_meta":779454,"kb_avail":92911116,"statfs":{"total":95995035648,"available":95140982784,"internally_reserved":0,"allocated":55881728,"data_stored":52761838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":798161349},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":57,"seq":244813136229,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1000700,"kb_used_data":53236,"kb_used_omap":1,"kb_used_meta":947454,"kb_avail":92744452,"statfs":{"total":95995035648,"available":94970318848,"internally_reserved":0,"allocated":54513664,"data_stored":51390222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":970193354},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":206158430568,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692204,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":976252,"kb_used_data":42676,"kb_used_omap":1,"kb_used_meta":933566,"kb_avail":92768900,"statfs":{"total":95995035648,"available":94995353600,"internally_reserved":0,"allocated":43700224,"data_stored":40576782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":955972037},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986544,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1057972,"kb_used_data":84140,"kb_used_omap":1,"kb_used_meta":973822,"kb_avail":92687180,"statfs":{"total":95995035648,"available":94911672320,"internally_reserved":0,"allocated":86159360,"data_stored":83039470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":997194181},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150071,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1133112,"kb_used_data":127344,"kb_used_omap":1,"kb_used_meta":1005758,"kb_avail":92612040,"statfs":{"total":95995035648,"available":94834728960,"internally_reserved":0,"allocated":130400256,"data_stored":127276294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1029896647},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,3,2,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":634,"seq":2723009265717,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":316348,"kb_used_data":41204,"kb_used_omap":4,"kb_used_meta":275003,"kb_avail":93428804,"statfs":{"total":95995035648,"available":95671095296,"internally_reserved":0,"allocated":42192896,"data_stored":39069454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4407,"internal_metadata":281603785},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,0,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575231,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1237304,"kb_used_data":119088,"kb_used_omap":1,"kb_used_meta":1118206,"kb_avail":92507848,"statfs":{"total":95995035648,"available":94728036352,"internally_reserved":0,"allocated":121946112,"data_stored":118822150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1145043397},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,0,0,0,0,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738756,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":959352,"kb_used_data":79920,"kb_used_omap":1,"kb_used_meta":879422,"kb_avail":92785800,"statfs":{"total":95995035648,"available":95012659200,"internally_reserved":0,"allocated":81838080,"data_stored":78714118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":900528581},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":72749056,"data_stored":72744984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":110759936,"data_stored":110755864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":33816576,"data_stored":33816576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":102305792,"data_stored":102301720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":69074944,"data_stored":69074944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":31981568,"data_stored":31981568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":40304640,"data_stored":40304640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":42532864,"data_stored":42532864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":85590016,"data_stored":85590016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":50593792,"data_stored":50593792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T14:39:49.854 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph status --format=json 2024-08-19T14:39:49.934 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:39:49.962 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:49.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:49.963 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:39:50.042 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:39:50.064 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:39:50.086 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:39:50.090 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:50.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:50.092 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:39:50.193 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:39:50.218 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:50.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:50.219 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:39:50.320 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:39:50.346 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:50.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:50.347 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:39:50.449 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:39:50.474 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:50.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:50.475 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:39:50.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:50 smithi032 ceph-mon[20004]: pgmap v1705: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 577 KiB data, 8.4 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 37 op/s; 6660/6 objects degraded (111000.000%); 1816/6 objects misplaced (30266.667%); 262 KiB/s, 407 objects/s recovering 2024-08-19T14:39:50.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:50 smithi045 ceph-mon[20143]: pgmap v1705: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 577 KiB data, 8.4 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 37 op/s; 6660/6 objects degraded (111000.000%); 1816/6 objects misplaced (30266.667%); 262 KiB/s, 407 objects/s recovering 2024-08-19T14:39:50.576 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:39:50.601 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:50.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:50.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:39:50.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:50 smithi031 ceph-mon[16336]: pgmap v1705: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 577 KiB data, 8.4 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 37 op/s; 6660/6 objects degraded (111000.000%); 1816/6 objects misplaced (30266.667%); 262 KiB/s, 407 objects/s recovering 2024-08-19T14:39:50.704 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:39:50.730 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:50.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:50.731 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:39:50.832 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:39:50.859 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:50.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:50.860 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:39:50.962 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:39:50.988 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:50.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:50.989 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:39:51.090 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:39:51.115 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:51.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:51.117 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:39:51.217 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:39:51.243 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:51.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:51.244 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:39:51.345 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:39:51.374 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:51.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:51.375 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:39:51.476 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:39:51.505 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:51.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:51.506 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:39:51.607 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:39:51.632 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:51.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:51.633 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:39:51.735 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:39:51.761 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:51.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:51.762 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:39:51.863 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:39:51.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:51.891 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:51.891 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:39:51.992 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:39:52.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:52.021 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:52.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:39:52.122 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:39:52.148 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:52.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:52.149 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:39:52.250 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:39:52.276 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:52.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:52.277 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:39:52.378 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:39:52.404 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:52.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:52.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:39:52.506 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:39:52.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:52 smithi032 ceph-mon[20004]: pgmap v1706: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 577 KiB data, 8.4 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 28 op/s; 6660/6 objects degraded (111000.000%); 1816/6 objects misplaced (30266.667%); 262 KiB/s, 407 objects/s recovering 2024-08-19T14:39:52.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:52 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:39:52.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:52 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:39:52.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:52 smithi032 ceph-mon[20004]: osdmap e709: 12 total, 12 up, 10 in 2024-08-19T14:39:52.532 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:52.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:52.533 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:39:52.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:52 smithi045 ceph-mon[20143]: pgmap v1706: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 577 KiB data, 8.4 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 28 op/s; 6660/6 objects degraded (111000.000%); 1816/6 objects misplaced (30266.667%); 262 KiB/s, 407 objects/s recovering 2024-08-19T14:39:52.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:52 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:39:52.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:52 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:39:52.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:52 smithi045 ceph-mon[20143]: osdmap e709: 12 total, 12 up, 10 in 2024-08-19T14:39:52.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:52 smithi031 ceph-mon[16336]: pgmap v1706: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 577 KiB data, 8.4 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 28 op/s; 6660/6 objects degraded (111000.000%); 1816/6 objects misplaced (30266.667%); 262 KiB/s, 407 objects/s recovering 2024-08-19T14:39:52.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:52 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:39:52.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:52 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:39:52.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:52 smithi031 ceph-mon[16336]: osdmap e709: 12 total, 12 up, 10 in 2024-08-19T14:39:52.635 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:39:52.661 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:52.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:52.662 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:39:52.763 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:39:52.788 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:52.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:52.789 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:39:52.890 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:39:52.916 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:52.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:52.917 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:39:53.018 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:39:53.044 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:53.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:53.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:39:53.145 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:39:53.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:53.172 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:53.172 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:39:53.273 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:39:53.301 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:53.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:53.302 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:39:53.403 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:39:53.408 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:53 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:39:53.409 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:53 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:39:53.409 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:53 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 6660/6 objects degraded (111000.000%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-19T14:39:53.411 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:53 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:39:53.412 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:53 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:39:53.412 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:53 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 6660/6 objects degraded (111000.000%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-19T14:39:53.435 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:53.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:53.436 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:39:53.448 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:53 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:39:53.448 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:53 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:39:53.448 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:53 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 6660/6 objects degraded (111000.000%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-19T14:39:53.537 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:39:53.563 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:53.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:53.564 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:39:53.665 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:39:53.690 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:53.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:53.691 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:39:53.793 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:39:53.818 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:53.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:53.819 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:39:53.920 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:39:53.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:53.946 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:53.947 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:39:54.047 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:39:54.073 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:54.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:54.074 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:39:54.175 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:39:54.201 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:54.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:54.202 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:39:54.277 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:39:54.278 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:39:54.303 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:39:54.328 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:54.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:54.329 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:39:54.430 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:39:54.456 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:54.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:54.457 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:39:54.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:54 smithi032 ceph-mon[20004]: pgmap v1708: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 577 KiB data, 8.4 GiB used, 886 GiB / 894 GiB avail; 4697/6 objects degraded (78283.333%); 1816/6 objects misplaced (30266.667%); 0 B/s, 538 objects/s recovering 2024-08-19T14:39:54.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:54 smithi032 ceph-mon[20004]: osdmap e710: 12 total, 12 up, 10 in 2024-08-19T14:39:54.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:54 smithi045 ceph-mon[20143]: pgmap v1708: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 577 KiB data, 8.4 GiB used, 886 GiB / 894 GiB avail; 4697/6 objects degraded (78283.333%); 1816/6 objects misplaced (30266.667%); 0 B/s, 538 objects/s recovering 2024-08-19T14:39:54.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:54 smithi045 ceph-mon[20143]: osdmap e710: 12 total, 12 up, 10 in 2024-08-19T14:39:54.558 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:39:54.583 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:54.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:54.584 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:39:54.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:54 smithi031 ceph-mon[16336]: pgmap v1708: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 577 KiB data, 8.4 GiB used, 886 GiB / 894 GiB avail; 4697/6 objects degraded (78283.333%); 1816/6 objects misplaced (30266.667%); 0 B/s, 538 objects/s recovering 2024-08-19T14:39:54.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:54 smithi031 ceph-mon[16336]: osdmap e710: 12 total, 12 up, 10 in 2024-08-19T14:39:54.685 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:39:54.710 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:54.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:54.711 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:39:54.812 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:39:54.837 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:54.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:54.838 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:39:54.940 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:39:54.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:54.966 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:54.966 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:39:55.067 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:39:55.091 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:55.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:55.092 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:39:55.193 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:39:55.216 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:55.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:55.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:39:55.318 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:39:55.341 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:55.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:55.342 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:39:55.443 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:39:55.469 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:55.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:55.470 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:39:55.571 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:39:55.596 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:55.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:55.597 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:39:55.698 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:39:55.722 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:55.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:55.723 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:39:55.824 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:39:55.849 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:55.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:55.850 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:39:55.887 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:39:55.951 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:39:55.977 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:55.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:55.978 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:39:56.079 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:39:56.105 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:56.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:56.106 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:39:56.207 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:39:56.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:56.235 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:56.236 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:39:56.337 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:39:56.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:56.365 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:56.365 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:39:56.466 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:39:56.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:56.493 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:56.494 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:39:56.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:56 smithi032 ceph-mon[20004]: pgmap v1710: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 577 KiB data, 8.4 GiB used, 886 GiB / 894 GiB avail; 3991/6 objects degraded (66516.667%); 1110/6 objects misplaced (18500.000%); 0 B/s, 457 objects/s recovering 2024-08-19T14:39:56.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:56 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:39:56.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:56 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:39:56.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:56 smithi032 ceph-mon[20004]: 1.5 scrub starts 2024-08-19T14:39:56.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:56 smithi032 ceph-mon[20004]: 1.5 scrub ok 2024-08-19T14:39:56.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:56 smithi032 ceph-mon[20004]: osdmap e711: 12 total, 12 up, 10 in 2024-08-19T14:39:56.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:56 smithi045 ceph-mon[20143]: pgmap v1710: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 577 KiB data, 8.4 GiB used, 886 GiB / 894 GiB avail; 3991/6 objects degraded (66516.667%); 1110/6 objects misplaced (18500.000%); 0 B/s, 457 objects/s recovering 2024-08-19T14:39:56.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:56 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:39:56.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:56 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:39:56.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:56 smithi045 ceph-mon[20143]: 1.5 scrub starts 2024-08-19T14:39:56.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:56 smithi045 ceph-mon[20143]: 1.5 scrub ok 2024-08-19T14:39:56.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:56 smithi045 ceph-mon[20143]: osdmap e711: 12 total, 12 up, 10 in 2024-08-19T14:39:56.595 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:39:56.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:56 smithi031 ceph-mon[16336]: pgmap v1710: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 577 KiB data, 8.4 GiB used, 886 GiB / 894 GiB avail; 3991/6 objects degraded (66516.667%); 1110/6 objects misplaced (18500.000%); 0 B/s, 457 objects/s recovering 2024-08-19T14:39:56.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:56 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:39:56.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:56 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:39:56.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:56 smithi031 ceph-mon[16336]: 1.5 scrub starts 2024-08-19T14:39:56.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:56 smithi031 ceph-mon[16336]: 1.5 scrub ok 2024-08-19T14:39:56.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:56 smithi031 ceph-mon[16336]: osdmap e711: 12 total, 12 up, 10 in 2024-08-19T14:39:56.620 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:56.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:56.621 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:39:56.722 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:39:56.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:56.749 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:56.749 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:39:56.850 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:39:56.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:56.877 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:56.877 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:39:56.979 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:39:57.005 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:57.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:57.006 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:39:57.107 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:39:57.133 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:57.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:57.134 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:39:57.235 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:39:57.261 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:57.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:57.262 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:39:57.363 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:39:57.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:57 smithi031 ceph-mon[16336]: osdmap e712: 12 total, 12 up, 10 in 2024-08-19T14:39:57.389 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:57.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:57.390 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:39:57.491 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:39:57.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:57 smithi032 ceph-mon[20004]: osdmap e712: 12 total, 12 up, 10 in 2024-08-19T14:39:57.516 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:57.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:57.517 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:39:57.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:57 smithi045 ceph-mon[20143]: osdmap e712: 12 total, 12 up, 10 in 2024-08-19T14:39:57.618 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:39:57.642 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:57.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:57.643 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:39:57.652 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-08-19T14:39:57.745 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:39:57.771 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:57.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:57.772 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:39:57.873 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:39:57.900 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:57.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:57.901 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:39:58.003 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:39:58.008 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is unset 2024-08-19T14:39:58.029 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:58.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:58.030 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:39:58.131 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:39:58.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:58.155 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:58.155 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:39:58.257 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:39:58.283 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:58.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:58.284 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:39:58.325 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:58 smithi031 ceph-mon[16336]: pgmap v1713: 47 pgs: 1 peering, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 577 KiB data, 8.4 GiB used, 886 GiB / 894 GiB avail; 1516/6 objects degraded (25266.667%); 1110/6 objects misplaced (18500.000%); 0 B/s, 619 objects/s recovering 2024-08-19T14:39:58.325 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:58 smithi031 ceph-mon[16336]: 7.a scrub starts 2024-08-19T14:39:58.325 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:58 smithi031 ceph-mon[16336]: 7.a scrub ok 2024-08-19T14:39:58.325 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:58 smithi031 ceph-mon[16336]: 1.7 scrub starts 2024-08-19T14:39:58.326 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:58 smithi031 ceph-mon[16336]: 1.7 scrub ok 2024-08-19T14:39:58.326 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:58 smithi031 ceph-mon[16336]: osdmap e713: 12 total, 12 up, 10 in 2024-08-19T14:39:58.326 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:58 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1896661355' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:39:58.326 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:58 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 1516/6 objects degraded (25266.667%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-19T14:39:58.326 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:58 smithi031 ceph-mon[16336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:39:58.326 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:58 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1896661355' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:39:58.327 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:58 smithi031 ceph-mon[16336]: osdmap e714: 12 total, 12 up, 10 in 2024-08-19T14:39:58.384 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:39:58.410 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:58.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:58.411 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:39:58.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:58 smithi032 ceph-mon[20004]: pgmap v1713: 47 pgs: 1 peering, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 577 KiB data, 8.4 GiB used, 886 GiB / 894 GiB avail; 1516/6 objects degraded (25266.667%); 1110/6 objects misplaced (18500.000%); 0 B/s, 619 objects/s recovering 2024-08-19T14:39:58.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:58 smithi032 ceph-mon[20004]: 7.a scrub starts 2024-08-19T14:39:58.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:58 smithi032 ceph-mon[20004]: 7.a scrub ok 2024-08-19T14:39:58.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:58 smithi032 ceph-mon[20004]: 1.7 scrub starts 2024-08-19T14:39:58.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:58 smithi032 ceph-mon[20004]: 1.7 scrub ok 2024-08-19T14:39:58.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:58 smithi032 ceph-mon[20004]: osdmap e713: 12 total, 12 up, 10 in 2024-08-19T14:39:58.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:58 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1896661355' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:39:58.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:58 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 1516/6 objects degraded (25266.667%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-19T14:39:58.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:58 smithi032 ceph-mon[20004]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:39:58.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:58 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1896661355' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:39:58.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:58 smithi032 ceph-mon[20004]: osdmap e714: 12 total, 12 up, 10 in 2024-08-19T14:39:58.512 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:39:58.537 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:58.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:58.538 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:39:58.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:58 smithi045 ceph-mon[20143]: pgmap v1713: 47 pgs: 1 peering, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 577 KiB data, 8.4 GiB used, 886 GiB / 894 GiB avail; 1516/6 objects degraded (25266.667%); 1110/6 objects misplaced (18500.000%); 0 B/s, 619 objects/s recovering 2024-08-19T14:39:58.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:58 smithi045 ceph-mon[20143]: 7.a scrub starts 2024-08-19T14:39:58.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:58 smithi045 ceph-mon[20143]: 7.a scrub ok 2024-08-19T14:39:58.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:58 smithi045 ceph-mon[20143]: 1.7 scrub starts 2024-08-19T14:39:58.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:58 smithi045 ceph-mon[20143]: 1.7 scrub ok 2024-08-19T14:39:58.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:58 smithi045 ceph-mon[20143]: osdmap e713: 12 total, 12 up, 10 in 2024-08-19T14:39:58.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:58 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1896661355' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:39:58.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:58 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 1516/6 objects degraded (25266.667%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-19T14:39:58.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:58 smithi045 ceph-mon[20143]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:39:58.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:58 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1896661355' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:39:58.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:58 smithi045 ceph-mon[20143]: osdmap e714: 12 total, 12 up, 10 in 2024-08-19T14:39:58.638 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:39:58.664 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:39:58.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:58.665 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:39:58.766 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:39:58.792 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:58.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:58.793 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:39:58.894 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:39:58.917 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:58.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:58.918 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:39:59.010 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'unique_pool_5' removed 2024-08-19T14:39:59.019 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:39:59.048 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:59.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:59.049 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:39:59.149 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:39:59.173 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:59.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:59.174 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:39:59.275 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:39:59.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:59.299 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:59.300 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:39:59.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:59 smithi031 ceph-mon[16336]: 1.1d scrub starts 2024-08-19T14:39:59.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:59 smithi031 ceph-mon[16336]: 1.1d scrub ok 2024-08-19T14:39:59.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:59 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/169790700' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]: dispatch 2024-08-19T14:39:59.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:59 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/169790700' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]': finished 2024-08-19T14:39:59.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:39:59 smithi031 ceph-mon[16336]: osdmap e715: 12 total, 12 up, 10 in 2024-08-19T14:39:59.400 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:39:59.421 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:59.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:59.422 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:39:59.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:59 smithi032 ceph-mon[20004]: 1.1d scrub starts 2024-08-19T14:39:59.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:59 smithi032 ceph-mon[20004]: 1.1d scrub ok 2024-08-19T14:39:59.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:59 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/169790700' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]: dispatch 2024-08-19T14:39:59.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:59 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/169790700' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]': finished 2024-08-19T14:39:59.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:39:59 smithi032 ceph-mon[20004]: osdmap e715: 12 total, 12 up, 10 in 2024-08-19T14:39:59.523 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:39:59.544 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:59.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:59.546 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:39:59.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:59 smithi045 ceph-mon[20143]: 1.1d scrub starts 2024-08-19T14:39:59.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:59 smithi045 ceph-mon[20143]: 1.1d scrub ok 2024-08-19T14:39:59.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:59 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/169790700' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]: dispatch 2024-08-19T14:39:59.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:59 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/169790700' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]': finished 2024-08-19T14:39:59.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:39:59 smithi045 ceph-mon[20143]: osdmap e715: 12 total, 12 up, 10 in 2024-08-19T14:39:59.646 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:39:59.669 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:59.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:59.670 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:39:59.694 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-08-19T14:39:59.695 INFO:tasks.radosbench:Beginning radosbench... 2024-08-19T14:39:59.696 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_6 2024-08-19T14:39:59.696 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd pool create unique_pool_6 16 2024-08-19T14:39:59.728 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:39:59.770 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:39:59.796 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:39:59.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:59.797 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:39:59.898 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:39:59.919 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:39:59.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:39:59.920 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:40:00.021 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:40:00.044 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:00.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:00.045 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:40:00.146 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:40:00.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:00.169 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:00.169 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:40:00.237 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:00 smithi031 ceph-mon[16336]: pgmap v1716: 47 pgs: 1 active+undersized+remapped, 1 peering, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 577 KiB data, 8.4 GiB used, 886 GiB / 894 GiB avail; 406/6 objects degraded (6766.667%); 0 B/s, 896 objects/s recovering 2024-08-19T14:40:00.238 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:00 smithi031 ceph-mon[16336]: 1.8 scrub starts 2024-08-19T14:40:00.238 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:00 smithi031 ceph-mon[16336]: 1.8 scrub ok 2024-08-19T14:40:00.238 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:00 smithi031 ceph-mon[16336]: 1.f scrub starts 2024-08-19T14:40:00.238 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:00 smithi031 ceph-mon[16336]: 1.f scrub ok 2024-08-19T14:40:00.238 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:00 smithi031 ceph-mon[16336]: Health detail: HEALTH_WARN Degraded data redundancy: 406/6 objects degraded (6766.667%), 2 pgs degraded, 3 pgs undersized 2024-08-19T14:40:00.238 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:00 smithi031 ceph-mon[16336]: [WRN] PG_DEGRADED: Degraded data redundancy: 406/6 objects degraded (6766.667%), 2 pgs degraded, 3 pgs undersized 2024-08-19T14:40:00.238 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:00 smithi031 ceph-mon[16336]: pg 7.2 is stuck undersized for 4m, current state active+recovering+undersized+degraded+remapped, last acting [3,7] 2024-08-19T14:40:00.238 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:00 smithi031 ceph-mon[16336]: pg 7.3 is stuck undersized for 4m, current state active+recovering+undersized+degraded+remapped, last acting [3,4] 2024-08-19T14:40:00.238 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:00 smithi031 ceph-mon[16336]: pg 7.e is stuck undersized for 4m, current state active+undersized+remapped, last acting [1,5] 2024-08-19T14:40:00.262 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:40:00.262 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:40:00.263 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:40:00.263 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:40:00.263 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:40:00.263 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:40:00.271 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:40:00.271 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:40:00.272 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:40:00.288 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:40:00.289 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:40:00.297 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:00.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:00.298 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:40:00.306 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:40:00.306 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:40:00.323 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:40:00.323 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:40:00.340 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:40:00.340 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:40:00.357 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:40:00.357 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:40:00.374 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:40:00.374 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:40:00.392 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:40:00.392 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:40:00.398 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:40:00.410 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:40:00.410 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:40:00.420 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:00.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:00.421 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:40:00.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:00 smithi032 ceph-mon[20004]: pgmap v1716: 47 pgs: 1 active+undersized+remapped, 1 peering, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 577 KiB data, 8.4 GiB used, 886 GiB / 894 GiB avail; 406/6 objects degraded (6766.667%); 0 B/s, 896 objects/s recovering 2024-08-19T14:40:00.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:00 smithi032 ceph-mon[20004]: 1.8 scrub starts 2024-08-19T14:40:00.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:00 smithi032 ceph-mon[20004]: 1.8 scrub ok 2024-08-19T14:40:00.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:00 smithi032 ceph-mon[20004]: 1.f scrub starts 2024-08-19T14:40:00.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:00 smithi032 ceph-mon[20004]: 1.f scrub ok 2024-08-19T14:40:00.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:00 smithi032 ceph-mon[20004]: Health detail: HEALTH_WARN Degraded data redundancy: 406/6 objects degraded (6766.667%), 2 pgs degraded, 3 pgs undersized 2024-08-19T14:40:00.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:00 smithi032 ceph-mon[20004]: [WRN] PG_DEGRADED: Degraded data redundancy: 406/6 objects degraded (6766.667%), 2 pgs degraded, 3 pgs undersized 2024-08-19T14:40:00.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:00 smithi032 ceph-mon[20004]: pg 7.2 is stuck undersized for 4m, current state active+recovering+undersized+degraded+remapped, last acting [3,7] 2024-08-19T14:40:00.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:00 smithi032 ceph-mon[20004]: pg 7.3 is stuck undersized for 4m, current state active+recovering+undersized+degraded+remapped, last acting [3,4] 2024-08-19T14:40:00.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:00 smithi032 ceph-mon[20004]: pg 7.e is stuck undersized for 4m, current state active+undersized+remapped, last acting [1,5] 2024-08-19T14:40:00.522 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:40:00.542 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:00.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:00.543 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:40:00.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:00 smithi045 ceph-mon[20143]: pgmap v1716: 47 pgs: 1 active+undersized+remapped, 1 peering, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 577 KiB data, 8.4 GiB used, 886 GiB / 894 GiB avail; 406/6 objects degraded (6766.667%); 0 B/s, 896 objects/s recovering 2024-08-19T14:40:00.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:00 smithi045 ceph-mon[20143]: 1.8 scrub starts 2024-08-19T14:40:00.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:00 smithi045 ceph-mon[20143]: 1.8 scrub ok 2024-08-19T14:40:00.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:00 smithi045 ceph-mon[20143]: 1.f scrub starts 2024-08-19T14:40:00.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:00 smithi045 ceph-mon[20143]: 1.f scrub ok 2024-08-19T14:40:00.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:00 smithi045 ceph-mon[20143]: Health detail: HEALTH_WARN Degraded data redundancy: 406/6 objects degraded (6766.667%), 2 pgs degraded, 3 pgs undersized 2024-08-19T14:40:00.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:00 smithi045 ceph-mon[20143]: [WRN] PG_DEGRADED: Degraded data redundancy: 406/6 objects degraded (6766.667%), 2 pgs degraded, 3 pgs undersized 2024-08-19T14:40:00.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:00 smithi045 ceph-mon[20143]: pg 7.2 is stuck undersized for 4m, current state active+recovering+undersized+degraded+remapped, last acting [3,7] 2024-08-19T14:40:00.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:00 smithi045 ceph-mon[20143]: pg 7.3 is stuck undersized for 4m, current state active+recovering+undersized+degraded+remapped, last acting [3,4] 2024-08-19T14:40:00.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:00 smithi045 ceph-mon[20143]: pg 7.e is stuck undersized for 4m, current state active+undersized+remapped, last acting [1,5] 2024-08-19T14:40:00.645 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:40:00.671 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:00.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:00.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:40:00.731 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:40:00.772 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:40:00.798 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:00.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:00.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:40:00.900 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:40:00.925 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:00.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:00.926 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:40:01.027 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:40:01.053 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:01.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:01.054 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:40:01.086 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-19T14:40:01.155 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:40:01.180 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:01.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:01.181 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:40:01.282 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:40:01.307 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:01.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:01.308 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:40:01.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:01 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:40:01.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:01 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:40:01.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:01 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:40:01.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:01 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:40:01.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:01 smithi031 ceph-mon[16336]: 1.18 scrub starts 2024-08-19T14:40:01.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:01 smithi031 ceph-mon[16336]: 1.18 scrub ok 2024-08-19T14:40:01.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:01 smithi031 ceph-mon[16336]: osdmap e716: 12 total, 12 up, 10 in 2024-08-19T14:40:01.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:01 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1365593892' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:40:01.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:01 smithi031 ceph-mon[16336]: 1.1a scrub starts 2024-08-19T14:40:01.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:01 smithi031 ceph-mon[16336]: 1.1a scrub ok 2024-08-19T14:40:01.404 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:40:01.409 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:40:01.435 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:01.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:01.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:40:01.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:01 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:40:01.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:01 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:40:01.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:01 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:40:01.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:01 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:40:01.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:01 smithi032 ceph-mon[20004]: 1.18 scrub starts 2024-08-19T14:40:01.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:01 smithi032 ceph-mon[20004]: 1.18 scrub ok 2024-08-19T14:40:01.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:01 smithi032 ceph-mon[20004]: osdmap e716: 12 total, 12 up, 10 in 2024-08-19T14:40:01.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:01 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1365593892' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:40:01.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:01 smithi032 ceph-mon[20004]: 1.1a scrub starts 2024-08-19T14:40:01.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:01 smithi032 ceph-mon[20004]: 1.1a scrub ok 2024-08-19T14:40:01.538 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:40:01.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:01 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:40:01.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:01 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:40:01.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:01 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:40:01.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:01 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:40:01.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:01 smithi045 ceph-mon[20143]: 1.18 scrub starts 2024-08-19T14:40:01.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:01 smithi045 ceph-mon[20143]: 1.18 scrub ok 2024-08-19T14:40:01.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:01 smithi045 ceph-mon[20143]: osdmap e716: 12 total, 12 up, 10 in 2024-08-19T14:40:01.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:01 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1365593892' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:40:01.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:01 smithi045 ceph-mon[20143]: 1.1a scrub starts 2024-08-19T14:40:01.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:01 smithi045 ceph-mon[20143]: 1.1a scrub ok 2024-08-19T14:40:01.562 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:01.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:01.563 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:40:01.624 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:40:01.624 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"ed7d1c6e-5e33-11ef-bccf-c7b262605968","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":1978,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":716,"num_osds":12,"num_up_osds":12,"osd_up_since":1724078110,"num_in_osds":10,"osd_in_since":1724078051,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":31}],"num_pgs":31,"num_pools":1,"num_objects":2,"data_bytes":590368,"bytes_used":8981061632,"bytes_avail":950969294848,"bytes_total":959950356480},"fsmap":{"epoch":1,"btime":"2024-08-19T14:05:22:841940+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.31:8443/"}},"servicemap":{"epoch":46,"modified":"2024-08-19T14:39:04.259049+0000","services":{}},"progress_events":{}} 2024-08-19T14:40:01.664 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:40:01.689 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:01.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:01.690 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:40:01.792 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:40:01.815 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:01.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:01.816 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:40:01.917 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:40:01.938 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:01.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:01.939 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:40:02.040 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:40:02.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:02.062 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:02.062 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:40:02.164 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:40:02.185 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:02.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:02.186 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:40:02.287 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:40:02.308 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:02.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:02.309 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:40:02.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:02 smithi031 ceph-mon[16336]: pgmap v1719: 31 pgs: 31 active+clean; 577 KiB data, 8.4 GiB used, 886 GiB / 894 GiB avail 2024-08-19T14:40:02.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:02 smithi031 ceph-mon[16336]: 1.c scrub starts 2024-08-19T14:40:02.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:02 smithi031 ceph-mon[16336]: 1.c scrub ok 2024-08-19T14:40:02.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:02 smithi031 ceph-mon[16336]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 406/6 objects degraded (6766.667%), 2 pgs degraded, 3 pgs undersized) 2024-08-19T14:40:02.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:02 smithi031 ceph-mon[16336]: Cluster is now healthy 2024-08-19T14:40:02.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:02 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1607317828' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-08-19T14:40:02.371 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:40:02.371 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 20, 2024-08-19T14:40:02.371 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 600, 2024-08-19T14:40:02.371 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [ 2024-08-19T14:40:02.371 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.372 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:174030 6.d 6:b7de3b0b:::benchmark_data_smithi045_97937_object66757:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e580)", 2024-08-19T14:40:02.372 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:32:12.456928+0000", 2024-08-19T14:40:02.372 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 469.91244915700003, 2024-08-19T14:40:02.372 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.31321180500000001, 2024-08-19T14:40:02.372 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:40:02.372 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:40:02.372 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:40:02.372 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:40:02.372 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:40:02.372 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:40:02.372 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 174030 2024-08-19T14:40:02.372 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.372 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:40:02.373 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.373 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:40:02.373 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.456928+0000", 2024-08-19T14:40:02.373 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:40:02.373 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.373 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.373 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:40:02.373 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.456928+0000", 2024-08-19T14:40:02.373 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:40:02.373 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.373 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.373 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:40:02.373 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.456929+0000", 2024-08-19T14:40:02.373 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5600000000000001e-06 2024-08-19T14:40:02.374 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.374 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.374 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:40:02.374 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.456935+0000", 2024-08-19T14:40:02.374 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.8599999999999998e-06 2024-08-19T14:40:02.374 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.374 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.374 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:40:02.374 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.456950+0000", 2024-08-19T14:40:02.374 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4413000000000001e-05 2024-08-19T14:40:02.374 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.374 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.374 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:40:02.375 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.456957+0000", 2024-08-19T14:40:02.375 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.3819999999999998e-06 2024-08-19T14:40:02.375 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.375 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.375 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:40:02.375 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.457017+0000", 2024-08-19T14:40:02.375 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.9490999999999997e-05 2024-08-19T14:40:02.375 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.375 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.375 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:40:02.375 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.457136+0000", 2024-08-19T14:40:02.375 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000119491 2024-08-19T14:40:02.375 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.375 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.376 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,5", 2024-08-19T14:40:02.376 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.457262+0000", 2024-08-19T14:40:02.376 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012542200000000001 2024-08-19T14:40:02.376 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.376 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.376 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:40:02.376 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.466456+0000", 2024-08-19T14:40:02.376 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0091943749999999994 2024-08-19T14:40:02.377 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.377 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.377 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:40:02.377 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.466505+0000", 2024-08-19T14:40:02.377 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.8785000000000001e-05 2024-08-19T14:40:02.377 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.377 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.377 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:40:02.377 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.770050+0000", 2024-08-19T14:40:02.377 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.30354529000000002 2024-08-19T14:40:02.377 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.377 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.377 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:40:02.377 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.770096+0000", 2024-08-19T14:40:02.378 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.6480000000000002e-05 2024-08-19T14:40:02.378 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.378 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.378 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:40:02.378 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.770140+0000", 2024-08-19T14:40:02.378 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.3256000000000001e-05 2024-08-19T14:40:02.378 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:40:02.378 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:40:02.378 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:40:02.378 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.378 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.378 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:174032 6.d 6:b8a5c293:::benchmark_data_smithi045_97937_object66759:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e580)", 2024-08-19T14:40:02.378 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:32:12.458962+0000", 2024-08-19T14:40:02.378 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 469.910414898, 2024-08-19T14:40:02.379 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.31142265299999999, 2024-08-19T14:40:02.379 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:40:02.379 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:40:02.379 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:40:02.379 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:40:02.379 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:40:02.379 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:40:02.379 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 174032 2024-08-19T14:40:02.379 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.379 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:40:02.379 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.379 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:40:02.379 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.458962+0000", 2024-08-19T14:40:02.379 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:40:02.380 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.380 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.380 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:40:02.380 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.458962+0000", 2024-08-19T14:40:02.380 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:40:02.380 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.380 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.380 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:40:02.380 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.458963+0000", 2024-08-19T14:40:02.380 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1289999999999999e-06 2024-08-19T14:40:02.380 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.380 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.380 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:40:02.380 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.458968+0000", 2024-08-19T14:40:02.381 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4830000000000001e-06 2024-08-19T14:40:02.381 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.381 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.381 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:40:02.381 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.458980+0000", 2024-08-19T14:40:02.381 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2476999999999999e-05 2024-08-19T14:40:02.381 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.381 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.381 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:40:02.381 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.458987+0000", 2024-08-19T14:40:02.381 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.3729999999999998e-06 2024-08-19T14:40:02.381 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.381 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.381 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:40:02.382 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.461794+0000", 2024-08-19T14:40:02.382 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0028076809999999998 2024-08-19T14:40:02.382 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.382 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.382 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:40:02.382 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.461948+0000", 2024-08-19T14:40:02.382 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00015364299999999999 2024-08-19T14:40:02.382 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.382 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.382 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,5", 2024-08-19T14:40:02.382 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.462046+0000", 2024-08-19T14:40:02.382 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.7576000000000002e-05 2024-08-19T14:40:02.382 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.382 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.383 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:40:02.383 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.466596+0000", 2024-08-19T14:40:02.383 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0045504179999999997 2024-08-19T14:40:02.383 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.383 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.383 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:40:02.383 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.470406+0000", 2024-08-19T14:40:02.383 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.003809977 2024-08-19T14:40:02.383 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.383 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.383 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:40:02.383 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.770183+0000", 2024-08-19T14:40:02.383 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.29977723699999997 2024-08-19T14:40:02.383 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.384 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.384 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:40:02.384 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.770239+0000", 2024-08-19T14:40:02.384 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.5961999999999997e-05 2024-08-19T14:40:02.384 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.384 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.384 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:40:02.384 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.770385+0000", 2024-08-19T14:40:02.384 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000145697 2024-08-19T14:40:02.384 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:40:02.384 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:40:02.384 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:40:02.384 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.384 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.385 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:174039 6.d 6:b7cfbf20:::benchmark_data_smithi045_97937_object66766:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e580)", 2024-08-19T14:40:02.385 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:32:12.470750+0000", 2024-08-19T14:40:02.385 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 469.89862692700001, 2024-08-19T14:40:02.385 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.299785941, 2024-08-19T14:40:02.385 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:40:02.385 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:40:02.385 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:40:02.385 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:40:02.385 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:40:02.385 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:40:02.386 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 174039 2024-08-19T14:40:02.386 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.386 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:40:02.386 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.387 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:40:02.387 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.470750+0000", 2024-08-19T14:40:02.387 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:40:02.387 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.387 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.387 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:40:02.387 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.470750+0000", 2024-08-19T14:40:02.387 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:40:02.387 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.387 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.387 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:40:02.387 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.470752+0000", 2024-08-19T14:40:02.387 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.739e-06 2024-08-19T14:40:02.387 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.388 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.388 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:40:02.388 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.470759+0000", 2024-08-19T14:40:02.388 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.754e-06 2024-08-19T14:40:02.388 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.388 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.388 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:40:02.388 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.470773+0000", 2024-08-19T14:40:02.388 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4365000000000001e-05 2024-08-19T14:40:02.388 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.388 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.388 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:40:02.388 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.470780+0000", 2024-08-19T14:40:02.388 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.4220000000000003e-06 2024-08-19T14:40:02.389 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.389 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.389 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:40:02.389 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.470837+0000", 2024-08-19T14:40:02.389 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.6246999999999999e-05 2024-08-19T14:40:02.389 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.389 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.389 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:40:02.389 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.471823+0000", 2024-08-19T14:40:02.389 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00098676800000000006 2024-08-19T14:40:02.389 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.389 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.389 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,5", 2024-08-19T14:40:02.389 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.471942+0000", 2024-08-19T14:40:02.390 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00011807399999999999 2024-08-19T14:40:02.390 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.390 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.390 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:40:02.390 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.476422+0000", 2024-08-19T14:40:02.390 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0044803660000000004 2024-08-19T14:40:02.390 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.390 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.390 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:40:02.390 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.477208+0000", 2024-08-19T14:40:02.390 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00078598299999999997 2024-08-19T14:40:02.390 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.390 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.391 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:40:02.391 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.770437+0000", 2024-08-19T14:40:02.391 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.293229036 2024-08-19T14:40:02.391 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.391 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.391 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:40:02.391 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.770497+0000", 2024-08-19T14:40:02.391 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.9998999999999999e-05 2024-08-19T14:40:02.391 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.391 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.391 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:40:02.391 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:12.770536+0000", 2024-08-19T14:40:02.391 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.9187999999999998e-05 2024-08-19T14:40:02.391 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:40:02.392 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:40:02.392 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:40:02.392 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.392 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.392 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:123536 7.e 7:74990b5e:::benchmark_data_smithi045_115842_object5376:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e638)", 2024-08-19T14:40:02.392 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:35:15.487240+0000", 2024-08-19T14:40:02.392 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 286.882137124, 2024-08-19T14:40:02.392 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.56778343899999995, 2024-08-19T14:40:02.392 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:40:02.392 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:40:02.392 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:40:02.392 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:40:02.392 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:40:02.392 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:40:02.393 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 123536 2024-08-19T14:40:02.393 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.393 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:40:02.393 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.393 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:40:02.393 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:15.487240+0000", 2024-08-19T14:40:02.393 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:40:02.393 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.393 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.394 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:40:02.394 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:15.487240+0000", 2024-08-19T14:40:02.394 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:40:02.394 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.394 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.394 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:40:02.394 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:15.487242+0000", 2024-08-19T14:40:02.394 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.61e-06 2024-08-19T14:40:02.394 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.394 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.394 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:40:02.394 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:15.487247+0000", 2024-08-19T14:40:02.395 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.3040000000000004e-06 2024-08-19T14:40:02.395 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.395 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.395 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:40:02.395 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:15.487264+0000", 2024-08-19T14:40:02.395 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.7110000000000001e-05 2024-08-19T14:40:02.395 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.395 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.395 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:40:02.395 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:15.487273+0000", 2024-08-19T14:40:02.395 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.6549999999999993e-06 2024-08-19T14:40:02.395 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.395 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.395 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:40:02.396 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:15.492302+0000", 2024-08-19T14:40:02.396 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00502908 2024-08-19T14:40:02.396 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.396 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.396 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for missing object", 2024-08-19T14:40:02.396 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:15.492390+0000", 2024-08-19T14:40:02.396 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.8712e-05 2024-08-19T14:40:02.396 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.396 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.396 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:40:02.396 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:16.055011+0000", 2024-08-19T14:40:02.396 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.56262038000000003 2024-08-19T14:40:02.396 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.396 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.397 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:40:02.397 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:16.055023+0000", 2024-08-19T14:40:02.397 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2588e-05 2024-08-19T14:40:02.397 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:40:02.397 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:40:02.397 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:40:02.397 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.397 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.397 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:177525 7.e 7:75808c59:::benchmark_data_smithi045_115842_object59365:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e675)", 2024-08-19T14:40:02.397 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:37:33.677940+0000", 2024-08-19T14:40:02.397 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 148.691437237, 2024-08-19T14:40:02.397 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.707863871, 2024-08-19T14:40:02.397 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:40:02.397 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:40:02.397 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:40:02.398 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:40:02.398 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:40:02.398 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:40:02.398 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 177525 2024-08-19T14:40:02.398 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.398 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:40:02.398 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.398 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:40:02.398 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:33.677940+0000", 2024-08-19T14:40:02.398 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:40:02.398 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.398 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.398 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:40:02.399 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:33.677940+0000", 2024-08-19T14:40:02.399 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:40:02.399 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.399 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.399 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:40:02.399 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:33.677940+0000", 2024-08-19T14:40:02.399 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.4700000000000001e-07 2024-08-19T14:40:02.399 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.399 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.399 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:40:02.399 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:33.677944+0000", 2024-08-19T14:40:02.399 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.1240000000000001e-06 2024-08-19T14:40:02.399 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.399 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.399 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:40:02.400 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:33.677952+0000", 2024-08-19T14:40:02.400 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.0469999999999994e-06 2024-08-19T14:40:02.400 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.400 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.400 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:40:02.400 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:33.677957+0000", 2024-08-19T14:40:02.400 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.7790000000000001e-06 2024-08-19T14:40:02.400 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.400 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.400 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:40:02.400 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:33.683942+0000", 2024-08-19T14:40:02.400 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0059850010000000002 2024-08-19T14:40:02.401 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.401 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.401 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for missing object", 2024-08-19T14:40:02.401 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:33.684027+0000", 2024-08-19T14:40:02.401 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.4586999999999995e-05 2024-08-19T14:40:02.401 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.401 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.401 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:40:02.401 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:35.385790+0000", 2024-08-19T14:40:02.401 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.701762964 2024-08-19T14:40:02.402 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.402 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.402 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:40:02.402 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:35.385804+0000", 2024-08-19T14:40:02.402 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3822000000000001e-05 2024-08-19T14:40:02.402 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:40:02.402 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:40:02.402 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:40:02.402 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.402 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.402 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:179487 7.d 7:be6e2ae7:::benchmark_data_smithi045_115842_object61327:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e678)", 2024-08-19T14:40:02.402 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:37:38.047749+0000", 2024-08-19T14:40:02.402 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 144.321628307, 2024-08-19T14:40:02.402 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.30256550599999998, 2024-08-19T14:40:02.403 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:40:02.403 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:40:02.403 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:40:02.403 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:40:02.403 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:40:02.403 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:40:02.403 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 179487 2024-08-19T14:40:02.403 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.403 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:40:02.403 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.403 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:40:02.403 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:38.047749+0000", 2024-08-19T14:40:02.403 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:40:02.404 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.404 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.404 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:40:02.404 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:38.047749+0000", 2024-08-19T14:40:02.404 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:40:02.404 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.404 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.404 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:40:02.404 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:38.047751+0000", 2024-08-19T14:40:02.404 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.2010000000000002e-06 2024-08-19T14:40:02.404 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.404 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.404 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:40:02.404 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:38.047759+0000", 2024-08-19T14:40:02.405 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.2060000000000007e-06 2024-08-19T14:40:02.405 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.405 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.405 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:40:02.405 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:38.047779+0000", 2024-08-19T14:40:02.405 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.0293999999999999e-05 2024-08-19T14:40:02.405 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.405 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.405 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:40:02.405 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:38.047789+0000", 2024-08-19T14:40:02.405 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.1540000000000008e-06 2024-08-19T14:40:02.405 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.406 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.406 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:40:02.407 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:38.047883+0000", 2024-08-19T14:40:02.407 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.4454000000000001e-05 2024-08-19T14:40:02.407 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.407 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.407 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op must wait for map", 2024-08-19T14:40:02.407 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:38.047889+0000", 2024-08-19T14:40:02.407 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.3659999999999997e-06 2024-08-19T14:40:02.407 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.407 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.407 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:40:02.408 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:38.049796+0000", 2024-08-19T14:40:02.408 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0019062529999999999 2024-08-19T14:40:02.408 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.408 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.408 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:40:02.408 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:38.049940+0000", 2024-08-19T14:40:02.408 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00014404199999999999 2024-08-19T14:40:02.408 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.408 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.408 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,3", 2024-08-19T14:40:02.408 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:38.050063+0000", 2024-08-19T14:40:02.408 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012279599999999999 2024-08-19T14:40:02.408 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.409 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.409 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:40:02.409 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:38.056330+0000", 2024-08-19T14:40:02.409 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0062678330000000004 2024-08-19T14:40:02.409 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.409 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.409 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:40:02.409 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:38.057291+0000", 2024-08-19T14:40:02.409 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00096042700000000003 2024-08-19T14:40:02.409 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.409 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.409 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:40:02.409 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:38.350212+0000", 2024-08-19T14:40:02.409 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.292920713 2024-08-19T14:40:02.409 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.410 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.410 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:40:02.410 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:38.350269+0000", 2024-08-19T14:40:02.410 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.7652000000000003e-05 2024-08-19T14:40:02.410 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.410 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.410 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:40:02.410 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:38.350314+0000", 2024-08-19T14:40:02.410 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.5114999999999997e-05 2024-08-19T14:40:02.411 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:40:02.411 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:40:02.411 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:40:02.411 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.411 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.411 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:180639 7.e 7:7b3191e7:::benchmark_data_smithi045_115842_object62479:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e678)", 2024-08-19T14:40:02.411 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:37:40.825823+0000", 2024-08-19T14:40:02.411 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 141.54355386700001, 2024-08-19T14:40:02.411 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.5895263179999999, 2024-08-19T14:40:02.411 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:40:02.411 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:40:02.411 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:40:02.411 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:40:02.411 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:40:02.412 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:40:02.412 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 180639 2024-08-19T14:40:02.412 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.412 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:40:02.412 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.412 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:40:02.412 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:40.825823+0000", 2024-08-19T14:40:02.412 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:40:02.412 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.412 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.412 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:40:02.412 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:40.825823+0000", 2024-08-19T14:40:02.412 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:40:02.412 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.413 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.413 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:40:02.413 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:40.825824+0000", 2024-08-19T14:40:02.413 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4700000000000002e-07 2024-08-19T14:40:02.413 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.413 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.413 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:40:02.413 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:40.825826+0000", 2024-08-19T14:40:02.413 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.6989999999999999e-06 2024-08-19T14:40:02.413 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.413 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.413 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:40:02.413 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:40.825833+0000", 2024-08-19T14:40:02.413 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.8109999999999997e-06 2024-08-19T14:40:02.414 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.414 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.414 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:40:02.415 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:40.825838+0000", 2024-08-19T14:40:02.415 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.322e-06 2024-08-19T14:40:02.415 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.415 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.415 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:40:02.415 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:40.826591+0000", 2024-08-19T14:40:02.415 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00075298699999999999 2024-08-19T14:40:02.416 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.416 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.416 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for missing object", 2024-08-19T14:40:02.416 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:40.826679+0000", 2024-08-19T14:40:02.416 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.7733999999999997e-05 2024-08-19T14:40:02.416 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.416 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.416 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:40:02.416 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.415332+0000", 2024-08-19T14:40:02.416 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.588653168 2024-08-19T14:40:02.416 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.416 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.416 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:40:02.416 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.415350+0000", 2024-08-19T14:40:02.417 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.715e-05 2024-08-19T14:40:02.417 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:40:02.417 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:40:02.417 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:40:02.417 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.417 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.417 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:180652 7.e 7:785f2c21:::benchmark_data_smithi045_115842_object62492:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e678)", 2024-08-19T14:40:02.417 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:37:40.825846+0000", 2024-08-19T14:40:02.417 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 141.543531409, 2024-08-19T14:40:02.417 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.5985306339999998, 2024-08-19T14:40:02.417 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:40:02.417 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:40:02.417 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:40:02.417 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:40:02.418 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:40:02.418 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:40:02.418 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 180652 2024-08-19T14:40:02.418 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.418 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:40:02.418 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.418 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:40:02.418 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:40.825846+0000", 2024-08-19T14:40:02.418 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:40:02.418 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.418 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.418 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:40:02.418 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:40.825846+0000", 2024-08-19T14:40:02.419 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:40:02.419 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.419 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.419 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:40:02.419 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:40.825846+0000", 2024-08-19T14:40:02.419 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.8299999999999998e-07 2024-08-19T14:40:02.419 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.419 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.419 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:40:02.419 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:40.825848+0000", 2024-08-19T14:40:02.419 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.0480000000000001e-06 2024-08-19T14:40:02.419 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.419 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.420 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:40:02.420 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:40.825854+0000", 2024-08-19T14:40:02.420 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.7729999999999996e-06 2024-08-19T14:40:02.420 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.420 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.420 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:40:02.420 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:40.825859+0000", 2024-08-19T14:40:02.420 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7079999999999996e-06 2024-08-19T14:40:02.420 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.420 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.420 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:40:02.420 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:40.826711+0000", 2024-08-19T14:40:02.420 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00085253599999999998 2024-08-19T14:40:02.420 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.421 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.421 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for missing object", 2024-08-19T14:40:02.421 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:40.826788+0000", 2024-08-19T14:40:02.421 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.7307999999999999e-05 2024-08-19T14:40:02.421 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.421 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.421 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:40:02.421 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.424327+0000", 2024-08-19T14:40:02.421 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.5975388829999999 2024-08-19T14:40:02.421 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.421 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.421 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:40:02.421 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.424376+0000", 2024-08-19T14:40:02.421 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.8995000000000001e-05 2024-08-19T14:40:02.422 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:40:02.422 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:40:02.422 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:40:02.422 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.422 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.422 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:180654 7.e 7:7efddc31:::benchmark_data_smithi045_115842_object62494:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e678)", 2024-08-19T14:40:02.422 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:37:40.825870+0000", 2024-08-19T14:40:02.422 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 141.54350724, 2024-08-19T14:40:02.422 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.6069602720000002, 2024-08-19T14:40:02.422 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:40:02.422 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:40:02.422 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:40:02.422 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:40:02.423 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:40:02.423 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:40:02.423 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 180654 2024-08-19T14:40:02.423 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.423 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:40:02.423 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.423 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:40:02.423 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:40.825870+0000", 2024-08-19T14:40:02.423 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:40:02.423 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.423 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.423 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:40:02.423 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:40.825870+0000", 2024-08-19T14:40:02.423 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:40:02.424 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.424 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.424 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:40:02.424 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:40.825870+0000", 2024-08-19T14:40:02.424 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.9999999999999998e-07 2024-08-19T14:40:02.424 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.424 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.424 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:40:02.424 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:40.825872+0000", 2024-08-19T14:40:02.424 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.1229999999999998e-06 2024-08-19T14:40:02.424 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.424 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.424 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:40:02.424 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:40.825878+0000", 2024-08-19T14:40:02.425 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.8810000000000001e-06 2024-08-19T14:40:02.425 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.425 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.425 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:40:02.425 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:40.825883+0000", 2024-08-19T14:40:02.425 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7600000000000002e-06 2024-08-19T14:40:02.425 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.425 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.425 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:40:02.425 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:40.826813+0000", 2024-08-19T14:40:02.425 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00093045599999999997 2024-08-19T14:40:02.425 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.425 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.426 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for missing object", 2024-08-19T14:40:02.426 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:40.826898+0000", 2024-08-19T14:40:02.426 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.4813000000000003e-05 2024-08-19T14:40:02.426 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.426 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.426 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:40:02.426 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.432817+0000", 2024-08-19T14:40:02.426 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.6059190700000001 2024-08-19T14:40:02.426 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.426 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.426 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:40:02.426 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.432830+0000", 2024-08-19T14:40:02.426 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2768999999999999e-05 2024-08-19T14:40:02.426 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:40:02.427 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:40:02.427 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:40:02.427 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.427 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.427 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:188913 7.e 7:7c2f3700:::benchmark_data_smithi045_115842_object70753:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e685)", 2024-08-19T14:40:02.427 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:37:59.679132+0000", 2024-08-19T14:40:02.427 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 122.690245507, 2024-08-19T14:40:02.427 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.8165226209999998, 2024-08-19T14:40:02.427 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:40:02.427 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:40:02.427 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:40:02.427 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:40:02.427 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:40:02.427 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:40:02.428 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 188913 2024-08-19T14:40:02.428 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.428 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:40:02.428 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.428 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:40:02.428 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:59.679132+0000", 2024-08-19T14:40:02.428 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:40:02.428 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.428 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.428 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:40:02.428 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:59.679132+0000", 2024-08-19T14:40:02.428 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:40:02.428 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.428 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.429 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:40:02.429 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:59.679132+0000", 2024-08-19T14:40:02.429 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.0400000000000002e-07 2024-08-19T14:40:02.429 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.429 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.429 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:40:02.429 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:59.679134+0000", 2024-08-19T14:40:02.429 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.2970000000000002e-06 2024-08-19T14:40:02.429 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.429 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.429 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:40:02.429 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:59.679140+0000", 2024-08-19T14:40:02.429 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.0090000000000004e-06 2024-08-19T14:40:02.429 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.430 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.430 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:40:02.430 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:59.679145+0000", 2024-08-19T14:40:02.430 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.8119999999999998e-06 2024-08-19T14:40:02.430 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.430 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.430 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:40:02.430 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:59.679901+0000", 2024-08-19T14:40:02.430 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00075560199999999997 2024-08-19T14:40:02.430 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.430 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.430 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for missing object", 2024-08-19T14:40:02.430 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:59.679989+0000", 2024-08-19T14:40:02.430 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.8046000000000004e-05 2024-08-19T14:40:02.431 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.431 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.431 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:40:02.431 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:03.495630+0000", 2024-08-19T14:40:02.431 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.8156409939999998 2024-08-19T14:40:02.431 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.431 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.431 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:40:02.431 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:03.495654+0000", 2024-08-19T14:40:02.431 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.4457e-05 2024-08-19T14:40:02.431 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:40:02.431 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:40:02.431 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:40:02.431 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.432 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.432 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:193857 7.e 7:7a7706f4:::benchmark_data_smithi045_115842_object75697:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e687)", 2024-08-19T14:40:02.432 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:38:10.869058+0000", 2024-08-19T14:40:02.432 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 111.500319381, 2024-08-19T14:40:02.432 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.435283927, 2024-08-19T14:40:02.432 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:40:02.432 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:40:02.432 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:40:02.432 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:40:02.432 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:40:02.432 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:40:02.432 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 193857 2024-08-19T14:40:02.432 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.433 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:40:02.433 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.433 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:40:02.433 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:10.869058+0000", 2024-08-19T14:40:02.433 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:40:02.433 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.433 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.433 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:40:02.433 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:10.869058+0000", 2024-08-19T14:40:02.433 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:40:02.433 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.433 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.433 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:40:02.433 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:10.869058+0000", 2024-08-19T14:40:02.434 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.8999999999999998e-07 2024-08-19T14:40:02.434 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.434 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.434 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:40:02.434 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:10.869060+0000", 2024-08-19T14:40:02.434 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.7439999999999999e-06 2024-08-19T14:40:02.434 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.434 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.434 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:40:02.434 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:10.869065+0000", 2024-08-19T14:40:02.434 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.485e-06 2024-08-19T14:40:02.434 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.434 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.434 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:40:02.435 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:10.869070+0000", 2024-08-19T14:40:02.435 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.6179999999999997e-06 2024-08-19T14:40:02.435 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.435 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.435 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:40:02.435 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:10.877850+0000", 2024-08-19T14:40:02.435 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0087805460000000002 2024-08-19T14:40:02.435 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.435 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.435 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for missing object", 2024-08-19T14:40:02.435 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:10.878074+0000", 2024-08-19T14:40:02.435 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00022323800000000001 2024-08-19T14:40:02.435 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.435 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.436 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:40:02.436 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:12.301236+0000", 2024-08-19T14:40:02.436 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.423162745 2024-08-19T14:40:02.436 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.436 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.436 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:40:02.436 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:12.304342+0000", 2024-08-19T14:40:02.436 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0031052609999999998 2024-08-19T14:40:02.436 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:40:02.436 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:40:02.436 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:40:02.436 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.436 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.436 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:201468 7.e 7:73a4f4d8:::benchmark_data_smithi045_115842_object83308:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e690)", 2024-08-19T14:40:02.437 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:38:27.119757+0000", 2024-08-19T14:40:02.437 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 95.249619640999995, 2024-08-19T14:40:02.437 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.76743869099999995, 2024-08-19T14:40:02.437 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:40:02.437 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:40:02.437 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:40:02.437 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:40:02.437 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:40:02.437 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:40:02.437 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 201468 2024-08-19T14:40:02.437 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.437 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:40:02.437 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.437 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:40:02.438 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:27.119757+0000", 2024-08-19T14:40:02.438 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:40:02.438 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.438 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.438 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:40:02.438 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:27.119757+0000", 2024-08-19T14:40:02.438 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:40:02.438 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.438 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.438 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:40:02.438 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:27.119758+0000", 2024-08-19T14:40:02.438 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.77e-07 2024-08-19T14:40:02.438 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.439 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.439 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:40:02.439 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:27.119760+0000", 2024-08-19T14:40:02.439 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.6690000000000002e-06 2024-08-19T14:40:02.439 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.439 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.439 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:40:02.439 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:27.119767+0000", 2024-08-19T14:40:02.439 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.472e-06 2024-08-19T14:40:02.439 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.439 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.439 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:40:02.439 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:27.119772+0000", 2024-08-19T14:40:02.440 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1599999999999997e-06 2024-08-19T14:40:02.440 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.440 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.440 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:40:02.440 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:27.123360+0000", 2024-08-19T14:40:02.440 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.003587898 2024-08-19T14:40:02.440 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.440 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.440 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for missing object", 2024-08-19T14:40:02.440 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:27.123444+0000", 2024-08-19T14:40:02.440 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.4469999999999996e-05 2024-08-19T14:40:02.440 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.440 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.440 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:40:02.441 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:27.887150+0000", 2024-08-19T14:40:02.441 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.76370524299999998 2024-08-19T14:40:02.441 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.441 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.441 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:40:02.441 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:27.887196+0000", 2024-08-19T14:40:02.441 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.6402000000000001e-05 2024-08-19T14:40:02.441 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:40:02.441 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:40:02.441 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:40:02.441 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.441 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.441 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:201473 7.e 7:7f9079d7:::benchmark_data_smithi045_115842_object83313:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e690)", 2024-08-19T14:40:02.441 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:38:27.119779+0000", 2024-08-19T14:40:02.442 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 95.249598414000005, 2024-08-19T14:40:02.442 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.77473406300000003, 2024-08-19T14:40:02.442 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:40:02.442 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:40:02.442 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:40:02.442 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:40:02.442 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:40:02.442 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:40:02.442 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 201473 2024-08-19T14:40:02.442 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.443 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:40:02.443 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.443 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:40:02.443 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:27.119779+0000", 2024-08-19T14:40:02.443 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:40:02.443 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.443 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.443 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:40:02.443 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:27.119779+0000", 2024-08-19T14:40:02.444 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:40:02.444 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.444 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.444 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:40:02.444 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:27.119779+0000", 2024-08-19T14:40:02.444 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.1399999999999998e-07 2024-08-19T14:40:02.444 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.444 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.444 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:40:02.444 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:27.119781+0000", 2024-08-19T14:40:02.444 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.176e-06 2024-08-19T14:40:02.444 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.444 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.444 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:40:02.445 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:27.119787+0000", 2024-08-19T14:40:02.445 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.6239999999999999e-06 2024-08-19T14:40:02.445 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.445 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.445 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:40:02.445 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:27.119791+0000", 2024-08-19T14:40:02.445 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.6550000000000003e-06 2024-08-19T14:40:02.445 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.445 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.445 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:40:02.445 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:27.123476+0000", 2024-08-19T14:40:02.445 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0036849719999999999 2024-08-19T14:40:02.445 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.445 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.446 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for missing object", 2024-08-19T14:40:02.446 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:27.123569+0000", 2024-08-19T14:40:02.446 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.2999999999999997e-05 2024-08-19T14:40:02.446 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.446 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.446 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:40:02.446 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:27.894493+0000", 2024-08-19T14:40:02.446 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:40:02.447 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.77092358100000002 2024-08-19T14:40:02.448 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.448 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.448 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:40:02.448 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:27.894513+0000", 2024-08-19T14:40:02.448 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.9741e-05 2024-08-19T14:40:02.448 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:40:02.448 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:40:02.448 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:40:02.448 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.448 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.448 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:201478 7.e 7:7de1c8fd:::benchmark_data_smithi045_115842_object83318:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e690)", 2024-08-19T14:40:02.448 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:38:27.119797+0000", 2024-08-19T14:40:02.448 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 95.249580162000001, 2024-08-19T14:40:02.449 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.78309812300000003, 2024-08-19T14:40:02.449 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:40:02.449 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:40:02.449 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:40:02.449 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:40:02.449 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:40:02.449 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:40:02.449 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 201478 2024-08-19T14:40:02.449 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.450 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:40:02.450 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.450 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:40:02.450 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:27.119797+0000", 2024-08-19T14:40:02.450 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:40:02.450 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.450 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.450 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:40:02.450 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:27.119797+0000", 2024-08-19T14:40:02.450 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:40:02.450 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.450 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.450 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:40:02.450 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:27.119797+0000", 2024-08-19T14:40:02.451 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.5899999999999998e-07 2024-08-19T14:40:02.451 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.451 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.451 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:40:02.451 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:27.119799+0000", 2024-08-19T14:40:02.451 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.9410000000000001e-06 2024-08-19T14:40:02.451 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.452 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.452 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:40:02.452 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:27.119805+0000", 2024-08-19T14:40:02.452 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.5350000000000004e-06 2024-08-19T14:40:02.452 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.452 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.452 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:40:02.452 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:27.119809+0000", 2024-08-19T14:40:02.452 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7829999999999998e-06 2024-08-19T14:40:02.452 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.452 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.453 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:40:02.453 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:27.123600+0000", 2024-08-19T14:40:02.453 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0037909290000000002 2024-08-19T14:40:02.453 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.453 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.453 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for missing object", 2024-08-19T14:40:02.453 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:27.123677+0000", 2024-08-19T14:40:02.453 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.6528999999999999e-05 2024-08-19T14:40:02.453 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.453 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.453 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:40:02.453 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:27.902882+0000", 2024-08-19T14:40:02.453 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.77920545500000005 2024-08-19T14:40:02.453 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.454 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.454 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:40:02.454 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:27.902895+0000", 2024-08-19T14:40:02.454 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2692e-05 2024-08-19T14:40:02.454 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:40:02.454 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:40:02.454 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:40:02.454 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.454 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.454 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:201492 7.e 7:785102fb:::benchmark_data_smithi045_115842_object83332:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e690)", 2024-08-19T14:40:02.454 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:38:27.119851+0000", 2024-08-19T14:40:02.454 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 95.249525585000001, 2024-08-19T14:40:02.454 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.78964315799999996, 2024-08-19T14:40:02.454 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:40:02.455 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:40:02.455 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:40:02.455 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:40:02.455 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:40:02.455 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:40:02.455 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 201492 2024-08-19T14:40:02.455 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.455 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:40:02.455 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.455 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:40:02.455 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:27.119851+0000", 2024-08-19T14:40:02.455 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:40:02.455 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.455 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.456 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:40:02.456 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:27.119851+0000", 2024-08-19T14:40:02.456 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:40:02.456 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.456 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.456 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:40:02.456 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:27.119852+0000", 2024-08-19T14:40:02.456 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.0100000000000001e-07 2024-08-19T14:40:02.456 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.456 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.456 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:40:02.456 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:27.119854+0000", 2024-08-19T14:40:02.456 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.8279999999999999e-06 2024-08-19T14:40:02.457 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.457 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.457 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:40:02.457 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:27.119865+0000", 2024-08-19T14:40:02.457 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0879e-05 2024-08-19T14:40:02.457 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.457 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.457 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:40:02.457 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:27.119870+0000", 2024-08-19T14:40:02.457 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1039999999999998e-06 2024-08-19T14:40:02.457 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.457 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.457 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:40:02.457 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:27.123881+0000", 2024-08-19T14:40:02.458 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0040112790000000004 2024-08-19T14:40:02.458 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.458 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.458 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for missing object", 2024-08-19T14:40:02.458 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:27.124100+0000", 2024-08-19T14:40:02.458 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000219182 2024-08-19T14:40:02.458 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.458 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.458 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:40:02.458 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:27.909481+0000", 2024-08-19T14:40:02.458 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.78538057900000002 2024-08-19T14:40:02.458 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.458 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.458 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:40:02.459 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:27.909495+0000", 2024-08-19T14:40:02.459 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4005999999999999e-05 2024-08-19T14:40:02.459 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:40:02.459 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:40:02.459 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:40:02.459 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.459 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.459 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:210125 7.e 7:70c40e16:::benchmark_data_smithi045_115842_object91965:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e694)", 2024-08-19T14:40:02.459 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:38:45.581565+0000", 2024-08-19T14:40:02.459 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 76.787812024999994, 2024-08-19T14:40:02.459 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.573032761, 2024-08-19T14:40:02.459 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:40:02.459 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:40:02.459 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:40:02.460 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:40:02.460 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:40:02.460 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:40:02.460 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 210125 2024-08-19T14:40:02.460 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.460 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:40:02.460 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.460 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:40:02.460 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:45.581565+0000", 2024-08-19T14:40:02.460 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:40:02.460 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.460 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.460 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:40:02.460 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:45.581565+0000", 2024-08-19T14:40:02.461 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:40:02.461 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.461 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.461 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:40:02.461 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:45.581565+0000", 2024-08-19T14:40:02.461 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.4200000000000002e-07 2024-08-19T14:40:02.461 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.461 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.461 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:40:02.461 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:45.581568+0000", 2024-08-19T14:40:02.461 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.1490000000000001e-06 2024-08-19T14:40:02.461 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.461 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.461 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:40:02.462 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:45.581573+0000", 2024-08-19T14:40:02.462 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.3739999999999997e-06 2024-08-19T14:40:02.462 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.462 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.462 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:40:02.462 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:45.581577+0000", 2024-08-19T14:40:02.462 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.544e-06 2024-08-19T14:40:02.462 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.462 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.462 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:40:02.462 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:45.592745+0000", 2024-08-19T14:40:02.462 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.011167662 2024-08-19T14:40:02.462 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.462 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.463 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for missing object", 2024-08-19T14:40:02.463 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:45.593010+0000", 2024-08-19T14:40:02.463 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00026508499999999997 2024-08-19T14:40:02.463 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.463 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.463 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:40:02.463 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:46.154555+0000", 2024-08-19T14:40:02.463 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.56154521700000004 2024-08-19T14:40:02.463 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.463 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.463 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:40:02.463 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:46.154598+0000", 2024-08-19T14:40:02.463 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2388000000000001e-05 2024-08-19T14:40:02.464 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:40:02.464 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:40:02.464 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:40:02.464 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.464 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.464 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:220620 7.d 7:b15dd3f1:::benchmark_data_smithi045_115842_object102460:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e700)", 2024-08-19T14:40:02.464 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:39:06.815895+0000", 2024-08-19T14:40:02.464 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 55.553482164999998, 2024-08-19T14:40:02.464 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.33205538600000001, 2024-08-19T14:40:02.464 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:40:02.464 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:40:02.464 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:40:02.464 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:40:02.464 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:40:02.465 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:40:02.465 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 220620 2024-08-19T14:40:02.465 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.465 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:40:02.465 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.465 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:40:02.465 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.815895+0000", 2024-08-19T14:40:02.465 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:40:02.465 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.465 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.465 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:40:02.465 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.815895+0000", 2024-08-19T14:40:02.465 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:40:02.465 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.466 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.466 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:40:02.466 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.815896+0000", 2024-08-19T14:40:02.466 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.505e-06 2024-08-19T14:40:02.466 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.466 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.466 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:40:02.466 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.815902+0000", 2024-08-19T14:40:02.466 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.305e-06 2024-08-19T14:40:02.466 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.466 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.466 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:40:02.466 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.815918+0000", 2024-08-19T14:40:02.467 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6093999999999999e-05 2024-08-19T14:40:02.467 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.467 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.467 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:40:02.467 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.815926+0000", 2024-08-19T14:40:02.467 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.0390000000000005e-06 2024-08-19T14:40:02.467 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.467 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.467 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:40:02.467 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.815987+0000", 2024-08-19T14:40:02.467 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.1014000000000002e-05 2024-08-19T14:40:02.467 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.467 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.467 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:40:02.468 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.816114+0000", 2024-08-19T14:40:02.468 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012700400000000001 2024-08-19T14:40:02.468 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.468 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.468 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,3", 2024-08-19T14:40:02.468 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.816244+0000", 2024-08-19T14:40:02.468 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013012700000000001 2024-08-19T14:40:02.468 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.468 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.468 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:40:02.468 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.827378+0000", 2024-08-19T14:40:02.468 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.011134333 2024-08-19T14:40:02.468 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.468 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.469 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:40:02.469 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.829710+0000", 2024-08-19T14:40:02.469 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0023313750000000001 2024-08-19T14:40:02.469 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.469 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.469 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:40:02.469 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:07.147836+0000", 2024-08-19T14:40:02.469 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.31812637199999999 2024-08-19T14:40:02.469 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.469 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.469 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:40:02.469 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:07.147903+0000", 2024-08-19T14:40:02.469 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.7144999999999995e-05 2024-08-19T14:40:02.470 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.470 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.470 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:40:02.470 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:07.147950+0000", 2024-08-19T14:40:02.470 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7073000000000001e-05 2024-08-19T14:40:02.470 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:40:02.470 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:40:02.470 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:40:02.470 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.470 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.470 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:220622 7.d 7:b91b5c37:::benchmark_data_smithi045_115842_object102462:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e700)", 2024-08-19T14:40:02.470 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:39:06.821281+0000", 2024-08-19T14:40:02.471 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 55.548096467999997, 2024-08-19T14:40:02.471 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.33110268700000001, 2024-08-19T14:40:02.471 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:40:02.471 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:40:02.471 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:40:02.471 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:40:02.471 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:40:02.471 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:40:02.471 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 220622 2024-08-19T14:40:02.471 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.472 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:40:02.472 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.472 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:40:02.472 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.821281+0000", 2024-08-19T14:40:02.472 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:40:02.472 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.472 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.472 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:40:02.472 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.821281+0000", 2024-08-19T14:40:02.472 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:40:02.472 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.472 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.472 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:40:02.472 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.821282+0000", 2024-08-19T14:40:02.473 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2860000000000001e-06 2024-08-19T14:40:02.473 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.473 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.473 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:40:02.473 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.821288+0000", 2024-08-19T14:40:02.473 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.1519999999999997e-06 2024-08-19T14:40:02.473 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.473 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.473 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:40:02.473 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.821299+0000", 2024-08-19T14:40:02.473 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1049e-05 2024-08-19T14:40:02.473 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.473 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.473 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:40:02.474 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.821307+0000", 2024-08-19T14:40:02.474 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.6699999999999994e-06 2024-08-19T14:40:02.474 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.474 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.474 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:40:02.474 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.822098+0000", 2024-08-19T14:40:02.474 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00079075100000000002 2024-08-19T14:40:02.474 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.474 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.474 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:40:02.474 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.822212+0000", 2024-08-19T14:40:02.474 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000114726 2024-08-19T14:40:02.474 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.474 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.475 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,3", 2024-08-19T14:40:02.475 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.822327+0000", 2024-08-19T14:40:02.475 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00011521799999999999 2024-08-19T14:40:02.475 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.475 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.475 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:40:02.475 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.827556+0000", 2024-08-19T14:40:02.475 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0052287649999999998 2024-08-19T14:40:02.475 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.475 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.476 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:40:02.476 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.829990+0000", 2024-08-19T14:40:02.476 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0024334809999999999 2024-08-19T14:40:02.476 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.476 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.476 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:40:02.476 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:07.152305+0000", 2024-08-19T14:40:02.476 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.32231495100000002 2024-08-19T14:40:02.476 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.476 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.476 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:40:02.476 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:07.152349+0000", 2024-08-19T14:40:02.476 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4141000000000003e-05 2024-08-19T14:40:02.476 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.477 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.477 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:40:02.477 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:07.152383+0000", 2024-08-19T14:40:02.477 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.4496999999999999e-05 2024-08-19T14:40:02.477 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:40:02.477 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:40:02.477 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:40:02.477 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.477 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.477 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:220629 7.d 7:b775bff3:::benchmark_data_smithi045_115842_object102469:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e700)", 2024-08-19T14:40:02.477 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:39:06.830315+0000", 2024-08-19T14:40:02.477 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 55.539061977999999, 2024-08-19T14:40:02.477 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.33290919200000002, 2024-08-19T14:40:02.478 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:40:02.478 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:40:02.478 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:40:02.478 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:40:02.478 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:40:02.478 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:40:02.478 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 220629 2024-08-19T14:40:02.478 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.478 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:40:02.478 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.478 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:40:02.479 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.830315+0000", 2024-08-19T14:40:02.479 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:40:02.479 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.479 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.479 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:40:02.479 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.830315+0000", 2024-08-19T14:40:02.479 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:40:02.479 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.479 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.479 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:40:02.479 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.830316+0000", 2024-08-19T14:40:02.479 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.063e-06 2024-08-19T14:40:02.479 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.479 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.480 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:40:02.480 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.830321+0000", 2024-08-19T14:40:02.480 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1059999999999999e-06 2024-08-19T14:40:02.480 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.480 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.480 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:40:02.480 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.830336+0000", 2024-08-19T14:40:02.480 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4856e-05 2024-08-19T14:40:02.480 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.480 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.480 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:40:02.480 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.830347+0000", 2024-08-19T14:40:02.480 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0604999999999999e-05 2024-08-19T14:40:02.480 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.481 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.481 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:40:02.481 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.830420+0000", 2024-08-19T14:40:02.481 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.3090999999999995e-05 2024-08-19T14:40:02.481 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.481 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.481 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:40:02.481 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.830540+0000", 2024-08-19T14:40:02.481 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00011983 2024-08-19T14:40:02.481 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.481 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.481 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 0,3", 2024-08-19T14:40:02.481 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.830660+0000", 2024-08-19T14:40:02.481 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012073 2024-08-19T14:40:02.482 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.482 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.482 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:40:02.482 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.835578+0000", 2024-08-19T14:40:02.482 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0049178490000000002 2024-08-19T14:40:02.482 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.482 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.482 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:40:02.482 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.836383+0000", 2024-08-19T14:40:02.482 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00080459300000000004 2024-08-19T14:40:02.482 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.482 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.482 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:40:02.483 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:07.163117+0000", 2024-08-19T14:40:02.483 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.32673422000000002 2024-08-19T14:40:02.483 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.483 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.483 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:40:02.483 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:07.163180+0000", 2024-08-19T14:40:02.483 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.2723000000000002e-05 2024-08-19T14:40:02.483 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.483 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.483 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:40:02.483 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:07.163224+0000", 2024-08-19T14:40:02.483 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4526e-05 2024-08-19T14:40:02.483 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:40:02.483 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:40:02.484 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:40:02.484 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.484 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.484 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:230020 7.e 7:7879f636:::benchmark_data_smithi045_115842_object111860:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e706)", 2024-08-19T14:40:02.484 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:39:24.532597+0000", 2024-08-19T14:40:02.484 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 37.836780066999999, 2024-08-19T14:40:02.484 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.74064174, 2024-08-19T14:40:02.484 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:40:02.484 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:40:02.484 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:40:02.484 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:40:02.484 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:40:02.484 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:40:02.484 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 230020 2024-08-19T14:40:02.485 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.485 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:40:02.485 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.485 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:40:02.485 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:24.532597+0000", 2024-08-19T14:40:02.485 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:40:02.485 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.485 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.485 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:40:02.485 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:24.532597+0000", 2024-08-19T14:40:02.485 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:40:02.485 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.485 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.485 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:40:02.486 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:24.532597+0000", 2024-08-19T14:40:02.486 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.2399999999999999e-07 2024-08-19T14:40:02.486 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.486 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.486 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:40:02.486 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:24.532600+0000", 2024-08-19T14:40:02.486 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.2249999999999999e-06 2024-08-19T14:40:02.486 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.486 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.486 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:40:02.486 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:24.532605+0000", 2024-08-19T14:40:02.486 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.4820000000000002e-06 2024-08-19T14:40:02.486 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.486 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.487 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:40:02.487 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:24.532610+0000", 2024-08-19T14:40:02.487 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.617e-06 2024-08-19T14:40:02.487 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.487 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.487 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:40:02.487 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:24.533407+0000", 2024-08-19T14:40:02.487 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00079693099999999996 2024-08-19T14:40:02.487 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.487 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.487 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for missing object", 2024-08-19T14:40:02.487 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:24.533491+0000", 2024-08-19T14:40:02.487 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.4686999999999997e-05 2024-08-19T14:40:02.487 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.488 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.488 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:40:02.488 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:27.273226+0000", 2024-08-19T14:40:02.488 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.7397351629999998 2024-08-19T14:40:02.488 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:02.488 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:02.488 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:40:02.488 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:27.273239+0000", 2024-08-19T14:40:02.488 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2311e-05 2024-08-19T14:40:02.488 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:40:02.488 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:40:02.488 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:40:02.488 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:40:02.488 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:40:02.488 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:40:02.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:02 smithi032 ceph-mon[20004]: pgmap v1719: 31 pgs: 31 active+clean; 577 KiB data, 8.4 GiB used, 886 GiB / 894 GiB avail 2024-08-19T14:40:02.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:02 smithi032 ceph-mon[20004]: 1.c scrub starts 2024-08-19T14:40:02.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:02 smithi032 ceph-mon[20004]: 1.c scrub ok 2024-08-19T14:40:02.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:02 smithi032 ceph-mon[20004]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 406/6 objects degraded (6766.667%), 2 pgs degraded, 3 pgs undersized) 2024-08-19T14:40:02.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:02 smithi032 ceph-mon[20004]: Cluster is now healthy 2024-08-19T14:40:02.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:02 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1607317828' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-08-19T14:40:02.509 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:02.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:02.510 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:40:02.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:02 smithi045 ceph-mon[20143]: pgmap v1719: 31 pgs: 31 active+clean; 577 KiB data, 8.4 GiB used, 886 GiB / 894 GiB avail 2024-08-19T14:40:02.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:02 smithi045 ceph-mon[20143]: 1.c scrub starts 2024-08-19T14:40:02.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:02 smithi045 ceph-mon[20143]: 1.c scrub ok 2024-08-19T14:40:02.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:02 smithi045 ceph-mon[20143]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 406/6 objects degraded (6766.667%), 2 pgs degraded, 3 pgs undersized) 2024-08-19T14:40:02.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:02 smithi045 ceph-mon[20143]: Cluster is now healthy 2024-08-19T14:40:02.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:02 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1607317828' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-08-19T14:40:02.611 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:40:02.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:02.633 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:02.634 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:40:02.736 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:40:02.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:02.758 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:02.759 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:40:02.860 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:40:02.881 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:02.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:02.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:40:02.984 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:40:03.010 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:03.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:03.011 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:40:03.112 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:40:03.134 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:03.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:03.138 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:40:03.181 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:40:03.187 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'unique_pool_6' created 2024-08-19T14:40:03.239 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:40:03.260 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:03.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:03.261 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:40:03.362 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:40:03.384 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:03.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:03.385 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:40:03.486 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:40:03.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:03 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/4164942721' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]: dispatch 2024-08-19T14:40:03.511 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:03.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:03.512 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:40:03.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:03 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/4164942721' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]: dispatch 2024-08-19T14:40:03.614 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:40:03.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:03 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/4164942721' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]: dispatch 2024-08-19T14:40:03.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:03.636 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:03.636 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:40:03.737 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:40:03.758 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:03.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:03.759 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:40:03.860 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:40:03.886 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:03.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:03.887 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:40:03.988 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:40:04.013 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:04.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:04.015 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:40:04.116 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:40:04.141 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:04.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:04.143 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:40:04.244 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:40:04.267 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:04.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:04.268 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:40:04.369 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:40:04.390 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:04.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:04.391 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:40:04.493 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:40:04.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:04 smithi032 ceph-mon[20004]: pgmap v1720: 31 pgs: 31 active+clean; 577 KiB data, 8.4 GiB used, 886 GiB / 894 GiB avail 2024-08-19T14:40:04.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:04 smithi032 ceph-mon[20004]: 1.3 scrub starts 2024-08-19T14:40:04.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:04 smithi032 ceph-mon[20004]: 1.3 scrub ok 2024-08-19T14:40:04.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:04 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/4164942721' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]': finished 2024-08-19T14:40:04.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:04 smithi032 ceph-mon[20004]: osdmap e717: 12 total, 12 up, 10 in 2024-08-19T14:40:04.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:04 smithi032 ceph-mon[20004]: 1.e deep-scrub starts 2024-08-19T14:40:04.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:04 smithi032 ceph-mon[20004]: 1.e deep-scrub ok 2024-08-19T14:40:04.518 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:04.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:04.519 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:40:04.534 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:04 smithi031 ceph-mon[16336]: pgmap v1720: 31 pgs: 31 active+clean; 577 KiB data, 8.4 GiB used, 886 GiB / 894 GiB avail 2024-08-19T14:40:04.534 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:04 smithi031 ceph-mon[16336]: 1.3 scrub starts 2024-08-19T14:40:04.535 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:04 smithi031 ceph-mon[16336]: 1.3 scrub ok 2024-08-19T14:40:04.535 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:04 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/4164942721' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]': finished 2024-08-19T14:40:04.535 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:04 smithi031 ceph-mon[16336]: osdmap e717: 12 total, 12 up, 10 in 2024-08-19T14:40:04.535 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:04 smithi031 ceph-mon[16336]: 1.e deep-scrub starts 2024-08-19T14:40:04.535 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:04 smithi031 ceph-mon[16336]: 1.e deep-scrub ok 2024-08-19T14:40:04.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:04 smithi045 ceph-mon[20143]: pgmap v1720: 31 pgs: 31 active+clean; 577 KiB data, 8.4 GiB used, 886 GiB / 894 GiB avail 2024-08-19T14:40:04.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:04 smithi045 ceph-mon[20143]: 1.3 scrub starts 2024-08-19T14:40:04.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:04 smithi045 ceph-mon[20143]: 1.3 scrub ok 2024-08-19T14:40:04.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:04 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/4164942721' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]': finished 2024-08-19T14:40:04.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:04 smithi045 ceph-mon[20143]: osdmap e717: 12 total, 12 up, 10 in 2024-08-19T14:40:04.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:04 smithi045 ceph-mon[20143]: 1.e deep-scrub starts 2024-08-19T14:40:04.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:04 smithi045 ceph-mon[20143]: 1.e deep-scrub ok 2024-08-19T14:40:04.621 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:40:04.645 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:04.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:04.646 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:40:04.747 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:40:04.772 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:04.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:04.773 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:40:04.874 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:40:04.899 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:04.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:04.900 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:40:05.001 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:40:05.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:05.028 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:05.028 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:40:05.129 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:40:05.146 INFO:tasks.cephadm.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-08-19T14:40:05.146 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-19T14:40:05.155 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:05.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:05.156 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:40:05.257 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:40:05.281 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:05.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:05.282 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:40:05.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:05 smithi031 ceph-mon[16336]: 1.16 scrub starts 2024-08-19T14:40:05.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:05 smithi031 ceph-mon[16336]: 1.16 scrub ok 2024-08-19T14:40:05.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:05 smithi031 ceph-mon[16336]: 1.b scrub starts 2024-08-19T14:40:05.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:05 smithi031 ceph-mon[16336]: 1.b scrub ok 2024-08-19T14:40:05.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:05 smithi031 ceph-mon[16336]: 1.14 scrub starts 2024-08-19T14:40:05.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:05 smithi031 ceph-mon[16336]: 1.14 scrub ok 2024-08-19T14:40:05.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:05 smithi031 ceph-mon[16336]: osdmap e718: 12 total, 12 up, 10 in 2024-08-19T14:40:05.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:05 smithi031 ceph-mon[16336]: 1.11 scrub starts 2024-08-19T14:40:05.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:05 smithi031 ceph-mon[16336]: 1.11 scrub ok 2024-08-19T14:40:05.384 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:40:05.405 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:05.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:05.406 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:40:05.507 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:40:05.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:05 smithi032 ceph-mon[20004]: 1.16 scrub starts 2024-08-19T14:40:05.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:05 smithi032 ceph-mon[20004]: 1.16 scrub ok 2024-08-19T14:40:05.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:05 smithi032 ceph-mon[20004]: 1.b scrub starts 2024-08-19T14:40:05.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:05 smithi032 ceph-mon[20004]: 1.b scrub ok 2024-08-19T14:40:05.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:05 smithi032 ceph-mon[20004]: 1.14 scrub starts 2024-08-19T14:40:05.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:05 smithi032 ceph-mon[20004]: 1.14 scrub ok 2024-08-19T14:40:05.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:05 smithi032 ceph-mon[20004]: osdmap e718: 12 total, 12 up, 10 in 2024-08-19T14:40:05.509 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:05 smithi032 ceph-mon[20004]: 1.11 scrub starts 2024-08-19T14:40:05.509 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:05 smithi032 ceph-mon[20004]: 1.11 scrub ok 2024-08-19T14:40:05.523 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-08-19T14:40:05.534 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:05.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:05.535 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:40:05.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:05 smithi045 ceph-mon[20143]: 1.16 scrub starts 2024-08-19T14:40:05.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:05 smithi045 ceph-mon[20143]: 1.16 scrub ok 2024-08-19T14:40:05.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:05 smithi045 ceph-mon[20143]: 1.b scrub starts 2024-08-19T14:40:05.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:05 smithi045 ceph-mon[20143]: 1.b scrub ok 2024-08-19T14:40:05.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:05 smithi045 ceph-mon[20143]: 1.14 scrub starts 2024-08-19T14:40:05.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:05 smithi045 ceph-mon[20143]: 1.14 scrub ok 2024-08-19T14:40:05.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:05 smithi045 ceph-mon[20143]: osdmap e718: 12 total, 12 up, 10 in 2024-08-19T14:40:05.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:05 smithi045 ceph-mon[20143]: 1.11 scrub starts 2024-08-19T14:40:05.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:05 smithi045 ceph-mon[20143]: 1.11 scrub ok 2024-08-19T14:40:05.601 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:40:05.636 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:40:05.662 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:05.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:05.663 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:40:05.765 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:40:05.790 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:05.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:05.792 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:40:05.893 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:40:05.914 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:05.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:05.915 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:40:06.011 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd pool application enable unique_pool_6 rados --yes-i-really-mean-it || true 2024-08-19T14:40:06.016 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:40:06.070 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:40:06.071 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:40:06.081 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:06.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:06.082 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:40:06.183 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:40:06.205 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:06.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:06.206 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:40:06.307 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:40:06.316 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:06 smithi031 ceph-mon[16336]: pgmap v1723: 47 pgs: 1 creating+peering, 15 unknown, 31 active+clean; 577 KiB data, 8.4 GiB used, 886 GiB / 894 GiB avail 2024-08-19T14:40:06.316 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:06 smithi031 ceph-mon[16336]: 1.12 scrub starts 2024-08-19T14:40:06.316 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:06 smithi031 ceph-mon[16336]: 1.12 scrub ok 2024-08-19T14:40:06.316 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:06 smithi031 ceph-mon[16336]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-19T14:40:06.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:06 smithi031 ceph-mon[16336]: osdmap e719: 12 total, 12 up, 10 in 2024-08-19T14:40:06.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:06 smithi031 ceph-mon[16336]: 1.a scrub starts 2024-08-19T14:40:06.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:06 smithi031 ceph-mon[16336]: 1.a scrub ok 2024-08-19T14:40:06.333 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:06.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:06.334 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:40:06.435 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:40:06.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:06.458 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:06.458 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:40:06.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:06 smithi032 ceph-mon[20004]: pgmap v1723: 47 pgs: 1 creating+peering, 15 unknown, 31 active+clean; 577 KiB data, 8.4 GiB used, 886 GiB / 894 GiB avail 2024-08-19T14:40:06.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:06 smithi032 ceph-mon[20004]: 1.12 scrub starts 2024-08-19T14:40:06.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:06 smithi032 ceph-mon[20004]: 1.12 scrub ok 2024-08-19T14:40:06.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:06 smithi032 ceph-mon[20004]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-19T14:40:06.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:06 smithi032 ceph-mon[20004]: osdmap e719: 12 total, 12 up, 10 in 2024-08-19T14:40:06.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:06 smithi032 ceph-mon[20004]: 1.a scrub starts 2024-08-19T14:40:06.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:06 smithi032 ceph-mon[20004]: 1.a scrub ok 2024-08-19T14:40:06.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:06 smithi045 ceph-mon[20143]: pgmap v1723: 47 pgs: 1 creating+peering, 15 unknown, 31 active+clean; 577 KiB data, 8.4 GiB used, 886 GiB / 894 GiB avail 2024-08-19T14:40:06.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:06 smithi045 ceph-mon[20143]: 1.12 scrub starts 2024-08-19T14:40:06.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:06 smithi045 ceph-mon[20143]: 1.12 scrub ok 2024-08-19T14:40:06.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:06 smithi045 ceph-mon[20143]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-19T14:40:06.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:06 smithi045 ceph-mon[20143]: osdmap e719: 12 total, 12 up, 10 in 2024-08-19T14:40:06.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:06 smithi045 ceph-mon[20143]: 1.a scrub starts 2024-08-19T14:40:06.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:06 smithi045 ceph-mon[20143]: 1.a scrub ok 2024-08-19T14:40:06.559 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:40:06.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:06.585 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:06.585 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:40:06.687 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:40:06.712 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:06.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:06.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:40:06.814 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:40:06.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:06.837 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:06.837 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:40:06.880 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:40:06.881 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-08-19T14:40:06.901 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:40:06.901 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:40:06.925 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:40:06.925 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:40:06.939 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:40:06.941 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:40:06.942 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:40:06.957 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:40:06.958 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:40:06.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:06.967 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:06.967 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:40:06.975 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:40:06.975 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:40:06.993 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:40:06.993 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:40:07.010 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:40:07.011 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:40:07.031 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:40:07.031 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:40:07.045 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:40:07.045 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:40:07.062 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:40:07.062 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:40:07.067 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:40:07.079 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:40:07.079 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:40:07.088 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:07.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:07.089 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:40:07.097 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:40:07.097 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:40:07.100 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:40:07.190 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:40:07.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:07.217 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:07.217 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:40:07.318 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:40:07.339 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:07.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:07.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:40:07.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:07 smithi031 ceph-mon[16336]: 1.13 scrub starts 2024-08-19T14:40:07.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:07 smithi031 ceph-mon[16336]: 1.13 scrub ok 2024-08-19T14:40:07.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:07 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:40:07.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:07 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:40:07.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:07 smithi031 ceph-mon[16336]: 1.9 scrub starts 2024-08-19T14:40:07.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:07 smithi031 ceph-mon[16336]: 1.9 scrub ok 2024-08-19T14:40:07.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:07 smithi031 ceph-mon[16336]: osdmap e720: 12 total, 12 up, 10 in 2024-08-19T14:40:07.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:07 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/4179695832' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:40:07.443 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:40:07.469 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:07.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:07.470 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:40:07.504 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":1726,"stamp":"2024-08-19T14:40:06.279409+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":856608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":120,"acting":120,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":8789304,"kb_used_data":87080,"kb_used_omap":18,"kb_used_meta":8702061,"kb_avail":928662216,"statfs":{"total":959950356480,"available":950950109184,"internally_reserved":0,"allocated":89169920,"data_stored":57504893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18760,"internal_metadata":8910911160},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000992"},"pg_stats":[{"pgid":"8.9","version":"0'0","reported_seq":13,"reported_epoch":719,"state":"active+clean","last_fresh":"2024-08-19T14:40:05.325295+0000","last_change":"2024-08-19T14:40:04.442466+0000","last_active":"2024-08-19T14:40:05.325295+0000","last_peered":"2024-08-19T14:40:05.325295+0000","last_clean":"2024-08-19T14:40:05.325295+0000","last_became_active":"2024-08-19T14:40:04.442173+0000","last_became_peered":"2024-08-19T14:40:04.442173+0000","last_unstale":"2024-08-19T14:40:05.325295+0000","last_undegraded":"2024-08-19T14:40:05.325295+0000","last_fullsized":"2024-08-19T14:40:05.325295+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":717,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.182354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:23.267013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"70'139","reported_seq":1765,"reported_epoch":719,"state":"active+clean","last_fresh":"2024-08-19T14:40:05.325082+0000","last_change":"2024-08-19T14:39:45.569137+0000","last_active":"2024-08-19T14:40:05.325082+0000","last_peered":"2024-08-19T14:40:05.325082+0000","last_clean":"2024-08-19T14:40:05.325082+0000","last_became_active":"2024-08-19T14:35:13.950224+0000","last_became_peered":"2024-08-19T14:35:13.950224+0000","last_unstale":"2024-08-19T14:40:05.325082+0000","last_undegraded":"2024-08-19T14:40:05.325082+0000","last_fullsized":"2024-08-19T14:40:05.325082+0000","mapping_epoch":637,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":638,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:39:45.569030+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:19:58.072083+0000","last_clean_scrub_stamp":"2024-08-19T14:39:45.569030+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:40:57.206713+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,9],"acting":[0,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":13,"reported_epoch":719,"state":"active+clean","last_fresh":"2024-08-19T14:40:05.234037+0000","last_change":"2024-08-19T14:40:04.442002+0000","last_active":"2024-08-19T14:40:05.234037+0000","last_peered":"2024-08-19T14:40:05.234037+0000","last_clean":"2024-08-19T14:40:05.234037+0000","last_became_active":"2024-08-19T14:40:04.441586+0000","last_became_peered":"2024-08-19T14:40:04.441586+0000","last_unstale":"2024-08-19T14:40:05.234037+0000","last_undegraded":"2024-08-19T14:40:05.234037+0000","last_fullsized":"2024-08-19T14:40:05.234037+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":717,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.182354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:21.392359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,9],"acting":[8,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"70'139","reported_seq":1603,"reported_epoch":716,"state":"active+clean","last_fresh":"2024-08-19T14:40:02.681274+0000","last_change":"2024-08-19T14:40:02.681316+0000","last_active":"2024-08-19T14:40:02.681274+0000","last_peered":"2024-08-19T14:40:02.681274+0000","last_clean":"2024-08-19T14:40:02.681274+0000","last_became_active":"2024-08-19T14:18:13.804721+0000","last_became_peered":"2024-08-19T14:18:13.804721+0000","last_unstale":"2024-08-19T14:40:02.681274+0000","last_undegraded":"2024-08-19T14:40:02.681274+0000","last_fullsized":"2024-08-19T14:40:02.681274+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:40:02.681185+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:40:02.681185+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:22.300124+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":13,"reported_epoch":719,"state":"active+clean","last_fresh":"2024-08-19T14:40:05.233595+0000","last_change":"2024-08-19T14:40:04.200633+0000","last_active":"2024-08-19T14:40:05.233595+0000","last_peered":"2024-08-19T14:40:05.233595+0000","last_clean":"2024-08-19T14:40:05.233595+0000","last_became_active":"2024-08-19T14:40:04.200299+0000","last_became_peered":"2024-08-19T14:40:04.200299+0000","last_unstale":"2024-08-19T14:40:05.233595+0000","last_undegraded":"2024-08-19T14:40:05.233595+0000","last_fullsized":"2024-08-19T14:40:05.233595+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":717,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.182354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:15.278167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,1],"acting":[8,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.2","version":"70'139","reported_seq":1558,"reported_epoch":719,"state":"active+clean","last_fresh":"2024-08-19T14:40:05.233525+0000","last_change":"2024-08-19T14:34:55.580212+0000","last_active":"2024-08-19T14:40:05.233525+0000","last_peered":"2024-08-19T14:40:05.233525+0000","last_clean":"2024-08-19T14:40:05.233525+0000","last_became_active":"2024-08-19T14:20:50.742354+0000","last_became_peered":"2024-08-19T14:20:50.742354+0000","last_unstale":"2024-08-19T14:40:05.233525+0000","last_undegraded":"2024-08-19T14:40:05.233525+0000","last_fullsized":"2024-08-19T14:40:05.233525+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:34:55.580141+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:19:59.302931+0000","last_clean_scrub_stamp":"2024-08-19T14:34:55.580141+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":13,"reported_epoch":719,"state":"active+clean","last_fresh":"2024-08-19T14:40:05.202310+0000","last_change":"2024-08-19T14:40:04.443203+0000","last_active":"2024-08-19T14:40:05.202310+0000","last_peered":"2024-08-19T14:40:05.202310+0000","last_clean":"2024-08-19T14:40:05.202310+0000","last_became_active":"2024-08-19T14:40:04.442802+0000","last_became_peered":"2024-08-19T14:40:04.442802+0000","last_unstale":"2024-08-19T14:40:05.202310+0000","last_undegraded":"2024-08-19T14:40:05.202310+0000","last_fullsized":"2024-08-19T14:40:05.202310+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":717,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.182354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:29.288558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,9],"acting":[3,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"70'139","reported_seq":864,"reported_epoch":719,"state":"active+clean","last_fresh":"2024-08-19T14:40:05.325478+0000","last_change":"2024-08-19T14:39:37.543259+0000","last_active":"2024-08-19T14:40:05.325478+0000","last_peered":"2024-08-19T14:40:05.325478+0000","last_clean":"2024-08-19T14:40:05.325478+0000","last_became_active":"2024-08-19T14:35:12.530566+0000","last_became_peered":"2024-08-19T14:35:12.530566+0000","last_unstale":"2024-08-19T14:40:05.325478+0000","last_undegraded":"2024-08-19T14:40:05.325478+0000","last_fullsized":"2024-08-19T14:40:05.325478+0000","mapping_epoch":635,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":636,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:39:37.543187+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:39:37.543187+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:40:56.083490+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,2],"acting":[0,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":13,"reported_epoch":719,"state":"active+clean","last_fresh":"2024-08-19T14:40:05.325628+0000","last_change":"2024-08-19T14:40:04.299744+0000","last_active":"2024-08-19T14:40:05.325628+0000","last_peered":"2024-08-19T14:40:05.325628+0000","last_clean":"2024-08-19T14:40:05.325628+0000","last_became_active":"2024-08-19T14:40:04.299403+0000","last_became_peered":"2024-08-19T14:40:04.299403+0000","last_unstale":"2024-08-19T14:40:05.325628+0000","last_undegraded":"2024-08-19T14:40:05.325628+0000","last_fullsized":"2024-08-19T14:40:05.325628+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":717,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.182354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:09.496839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"70'139","reported_seq":1268,"reported_epoch":719,"state":"active+clean","last_fresh":"2024-08-19T14:40:05.202312+0000","last_change":"2024-08-19T14:39:56.993699+0000","last_active":"2024-08-19T14:40:05.202312+0000","last_peered":"2024-08-19T14:40:05.202312+0000","last_clean":"2024-08-19T14:40:05.202312+0000","last_became_active":"2024-08-19T14:35:11.536828+0000","last_became_peered":"2024-08-19T14:35:11.536828+0000","last_unstale":"2024-08-19T14:40:05.202312+0000","last_undegraded":"2024-08-19T14:40:05.202312+0000","last_fullsized":"2024-08-19T14:40:05.202312+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:39:56.993616+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:21:46.057624+0000","last_clean_scrub_stamp":"2024-08-19T14:39:56.993616+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:05.362216+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-19T14:40:03.182354+0000","last_change":"2024-08-19T14:40:03.182354+0000","last_active":"2024-08-19T14:40:03.182354+0000","last_peered":"2024-08-19T14:40:03.182354+0000","last_clean":"2024-08-19T14:40:03.182354+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-19T14:40:03.182354+0000","last_undegraded":"2024-08-19T14:40:03.182354+0000","last_fullsized":"2024-08-19T14:40:03.182354+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.182354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.6","version":"70'139","reported_seq":835,"reported_epoch":719,"state":"active+clean","last_fresh":"2024-08-19T14:40:05.233840+0000","last_change":"2024-08-19T14:39:30.460768+0000","last_active":"2024-08-19T14:40:05.233840+0000","last_peered":"2024-08-19T14:40:05.233840+0000","last_clean":"2024-08-19T14:40:05.233840+0000","last_became_active":"2024-08-19T14:34:15.717054+0000","last_became_peered":"2024-08-19T14:34:15.717054+0000","last_unstale":"2024-08-19T14:40:05.233840+0000","last_undegraded":"2024-08-19T14:40:05.233840+0000","last_fullsized":"2024-08-19T14:40:05.233840+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:39:30.460649+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:27.848870+0000","last_clean_scrub_stamp":"2024-08-19T14:39:30.460649+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:40:59.351491+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":13,"reported_epoch":719,"state":"active+clean","last_fresh":"2024-08-19T14:40:05.233688+0000","last_change":"2024-08-19T14:40:04.442056+0000","last_active":"2024-08-19T14:40:05.233688+0000","last_peered":"2024-08-19T14:40:05.233688+0000","last_clean":"2024-08-19T14:40:05.233688+0000","last_became_active":"2024-08-19T14:40:04.441602+0000","last_became_peered":"2024-08-19T14:40:04.441602+0000","last_unstale":"2024-08-19T14:40:05.233688+0000","last_undegraded":"2024-08-19T14:40:05.233688+0000","last_fullsized":"2024-08-19T14:40:05.233688+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":717,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.182354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:14.564401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,3],"acting":[8,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.5","version":"70'139","reported_seq":784,"reported_epoch":716,"state":"active+clean","last_fresh":"2024-08-19T14:40:00.227454+0000","last_change":"2024-08-19T14:39:54.886149+0000","last_active":"2024-08-19T14:40:00.227454+0000","last_peered":"2024-08-19T14:40:00.227454+0000","last_clean":"2024-08-19T14:40:00.227454+0000","last_became_active":"2024-08-19T14:35:11.532858+0000","last_became_peered":"2024-08-19T14:35:11.532858+0000","last_unstale":"2024-08-19T14:40:00.227454+0000","last_undegraded":"2024-08-19T14:40:00.227454+0000","last_fullsized":"2024-08-19T14:40:00.227454+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:39:54.886056+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:39:54.886056+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:40:57.762183+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,0],"acting":[11,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-19T14:40:03.182354+0000","last_change":"2024-08-19T14:40:03.182354+0000","last_active":"2024-08-19T14:40:03.182354+0000","last_peered":"2024-08-19T14:40:03.182354+0000","last_clean":"2024-08-19T14:40:03.182354+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-19T14:40:03.182354+0000","last_undegraded":"2024-08-19T14:40:03.182354+0000","last_fullsized":"2024-08-19T14:40:03.182354+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.182354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.4","version":"70'139","reported_seq":653,"reported_epoch":716,"state":"active+clean","last_fresh":"2024-08-19T14:40:00.227254+0000","last_change":"2024-08-19T14:33:16.760928+0000","last_active":"2024-08-19T14:40:00.227254+0000","last_peered":"2024-08-19T14:40:00.227254+0000","last_clean":"2024-08-19T14:40:00.227254+0000","last_became_active":"2024-08-19T14:30:47.909067+0000","last_became_peered":"2024-08-19T14:30:47.909067+0000","last_unstale":"2024-08-19T14:40:00.227254+0000","last_undegraded":"2024-08-19T14:40:00.227254+0000","last_fullsized":"2024-08-19T14:40:00.227254+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:33:16.760810+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:33:16.760810+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-19T14:40:03.182354+0000","last_change":"2024-08-19T14:40:03.182354+0000","last_active":"2024-08-19T14:40:03.182354+0000","last_peered":"2024-08-19T14:40:03.182354+0000","last_clean":"2024-08-19T14:40:03.182354+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-19T14:40:03.182354+0000","last_undegraded":"2024-08-19T14:40:03.182354+0000","last_fullsized":"2024-08-19T14:40:03.182354+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.182354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.a","version":"70'139","reported_seq":820,"reported_epoch":717,"state":"active+clean","last_fresh":"2024-08-19T14:40:03.196319+0000","last_change":"2024-08-19T14:34:50.256878+0000","last_active":"2024-08-19T14:40:03.196319+0000","last_peered":"2024-08-19T14:40:03.196319+0000","last_clean":"2024-08-19T14:40:03.196319+0000","last_became_active":"2024-08-19T14:34:15.717472+0000","last_became_peered":"2024-08-19T14:34:15.717472+0000","last_unstale":"2024-08-19T14:40:03.196319+0000","last_undegraded":"2024-08-19T14:40:03.196319+0000","last_fullsized":"2024-08-19T14:40:03.196319+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:34:50.256808+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:28.864870+0000","last_clean_scrub_stamp":"2024-08-19T14:34:50.256808+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,0],"acting":[4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":13,"reported_epoch":719,"state":"active+clean","last_fresh":"2024-08-19T14:40:05.202321+0000","last_change":"2024-08-19T14:40:04.443641+0000","last_active":"2024-08-19T14:40:05.202321+0000","last_peered":"2024-08-19T14:40:05.202321+0000","last_clean":"2024-08-19T14:40:05.202321+0000","last_became_active":"2024-08-19T14:40:04.443254+0000","last_became_peered":"2024-08-19T14:40:04.443254+0000","last_unstale":"2024-08-19T14:40:05.202321+0000","last_undegraded":"2024-08-19T14:40:05.202321+0000","last_fullsized":"2024-08-19T14:40:05.202321+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":717,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.182354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:24.179363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,9],"acting":[3,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"70'139","reported_seq":1760,"reported_epoch":719,"state":"active+clean","last_fresh":"2024-08-19T14:40:05.233192+0000","last_change":"2024-08-19T14:35:12.529993+0000","last_active":"2024-08-19T14:40:05.233192+0000","last_peered":"2024-08-19T14:40:05.233192+0000","last_clean":"2024-08-19T14:40:05.233192+0000","last_became_active":"2024-08-19T14:35:12.529628+0000","last_became_peered":"2024-08-19T14:35:12.529628+0000","last_unstale":"2024-08-19T14:40:05.233192+0000","last_undegraded":"2024-08-19T14:40:05.233192+0000","last_fullsized":"2024-08-19T14:40:05.233192+0000","mapping_epoch":635,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":636,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:34:51.561893+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:34:51.561893+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-19T14:40:03.182354+0000","last_change":"2024-08-19T14:40:03.182354+0000","last_active":"2024-08-19T14:40:03.182354+0000","last_peered":"2024-08-19T14:40:03.182354+0000","last_clean":"2024-08-19T14:40:03.182354+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-19T14:40:03.182354+0000","last_undegraded":"2024-08-19T14:40:03.182354+0000","last_fullsized":"2024-08-19T14:40:03.182354+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.182354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.8","version":"70'139","reported_seq":798,"reported_epoch":719,"state":"active+clean","last_fresh":"2024-08-19T14:40:05.202368+0000","last_change":"2024-08-19T14:39:59.000415+0000","last_active":"2024-08-19T14:40:05.202368+0000","last_peered":"2024-08-19T14:40:05.202368+0000","last_clean":"2024-08-19T14:40:05.202368+0000","last_became_active":"2024-08-19T14:35:11.537474+0000","last_became_peered":"2024-08-19T14:35:11.537474+0000","last_unstale":"2024-08-19T14:40:05.202368+0000","last_undegraded":"2024-08-19T14:40:05.202368+0000","last_fullsized":"2024-08-19T14:40:05.202368+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:39:59.000300+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:39:59.000300+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:11.528949+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":13,"reported_epoch":719,"state":"active+clean","last_fresh":"2024-08-19T14:40:05.325292+0000","last_change":"2024-08-19T14:40:04.441786+0000","last_active":"2024-08-19T14:40:05.325292+0000","last_peered":"2024-08-19T14:40:05.325292+0000","last_clean":"2024-08-19T14:40:05.325292+0000","last_became_active":"2024-08-19T14:40:04.441445+0000","last_became_peered":"2024-08-19T14:40:04.441445+0000","last_unstale":"2024-08-19T14:40:05.325292+0000","last_undegraded":"2024-08-19T14:40:05.325292+0000","last_fullsized":"2024-08-19T14:40:05.325292+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":717,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.182354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:07.243931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,9],"acting":[0,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.b","version":"70'139","reported_seq":1105,"reported_epoch":716,"state":"active+clean","last_fresh":"2024-08-19T14:40:00.227338+0000","last_change":"2024-08-19T14:33:15.753668+0000","last_active":"2024-08-19T14:40:00.227338+0000","last_peered":"2024-08-19T14:40:00.227338+0000","last_clean":"2024-08-19T14:40:00.227338+0000","last_became_active":"2024-08-19T14:30:47.941200+0000","last_became_peered":"2024-08-19T14:30:47.941200+0000","last_unstale":"2024-08-19T14:40:00.227338+0000","last_undegraded":"2024-08-19T14:40:00.227338+0000","last_fullsized":"2024-08-19T14:40:00.227338+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:33:15.753595+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:33:15.753595+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-19T14:40:03.182354+0000","last_change":"2024-08-19T14:40:03.182354+0000","last_active":"2024-08-19T14:40:03.182354+0000","last_peered":"2024-08-19T14:40:03.182354+0000","last_clean":"2024-08-19T14:40:03.182354+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-19T14:40:03.182354+0000","last_undegraded":"2024-08-19T14:40:03.182354+0000","last_fullsized":"2024-08-19T14:40:03.182354+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.182354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.e","version":"70'139","reported_seq":1297,"reported_epoch":717,"state":"active+clean","last_fresh":"2024-08-19T14:40:03.196409+0000","last_change":"2024-08-19T14:34:55.333177+0000","last_active":"2024-08-19T14:40:03.196409+0000","last_peered":"2024-08-19T14:40:03.196409+0000","last_clean":"2024-08-19T14:40:03.196409+0000","last_became_active":"2024-08-19T14:18:26.884105+0000","last_became_peered":"2024-08-19T14:18:26.884105+0000","last_unstale":"2024-08-19T14:40:03.196409+0000","last_undegraded":"2024-08-19T14:40:03.196409+0000","last_fullsized":"2024-08-19T14:40:03.196409+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:34:55.333065+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:32:31.196288+0000","last_clean_scrub_stamp":"2024-08-19T14:34:55.333065+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-19T14:40:03.182354+0000","last_change":"2024-08-19T14:40:03.182354+0000","last_active":"2024-08-19T14:40:03.182354+0000","last_peered":"2024-08-19T14:40:03.182354+0000","last_clean":"2024-08-19T14:40:03.182354+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-19T14:40:03.182354+0000","last_undegraded":"2024-08-19T14:40:03.182354+0000","last_fullsized":"2024-08-19T14:40:03.182354+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.182354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.f","version":"70'139","reported_seq":1057,"reported_epoch":717,"state":"active+clean","last_fresh":"2024-08-19T14:40:03.196445+0000","last_change":"2024-08-19T14:39:59.812809+0000","last_active":"2024-08-19T14:40:03.196445+0000","last_peered":"2024-08-19T14:40:03.196445+0000","last_clean":"2024-08-19T14:40:03.196445+0000","last_became_active":"2024-08-19T14:27:31.837754+0000","last_became_peered":"2024-08-19T14:27:31.837754+0000","last_unstale":"2024-08-19T14:40:03.196445+0000","last_undegraded":"2024-08-19T14:40:03.196445+0000","last_fullsized":"2024-08-19T14:40:03.196445+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:39:59.812612+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:39:59.812612+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:07.074257+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":1,"reported_epoch":717,"state":"creating+peering","last_fresh":"2024-08-19T14:40:03.195850+0000","last_change":"2024-08-19T14:40:03.195850+0000","last_active":"2024-08-19T14:40:03.182354+0000","last_peered":"2024-08-19T14:40:03.182354+0000","last_clean":"2024-08-19T14:40:03.182354+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-19T14:40:03.195850+0000","last_undegraded":"2024-08-19T14:40:03.195850+0000","last_fullsized":"2024-08-19T14:40:03.195850+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":717,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.182354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,11],"acting":[4,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.c","version":"70'139","reported_seq":1416,"reported_epoch":719,"state":"active+clean","last_fresh":"2024-08-19T14:40:05.202314+0000","last_change":"2024-08-19T14:40:00.947434+0000","last_active":"2024-08-19T14:40:05.202314+0000","last_peered":"2024-08-19T14:40:05.202314+0000","last_clean":"2024-08-19T14:40:05.202314+0000","last_became_active":"2024-08-19T14:34:15.733282+0000","last_became_peered":"2024-08-19T14:34:15.733282+0000","last_unstale":"2024-08-19T14:40:05.202314+0000","last_undegraded":"2024-08-19T14:40:05.202314+0000","last_fullsized":"2024-08-19T14:40:05.202314+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:40:00.947341+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:22:52.164586+0000","last_clean_scrub_stamp":"2024-08-19T14:40:00.947341+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:21.334102+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,9],"acting":[3,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-19T14:40:03.182354+0000","last_change":"2024-08-19T14:40:03.182354+0000","last_active":"2024-08-19T14:40:03.182354+0000","last_peered":"2024-08-19T14:40:03.182354+0000","last_clean":"2024-08-19T14:40:03.182354+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-19T14:40:03.182354+0000","last_undegraded":"2024-08-19T14:40:03.182354+0000","last_fullsized":"2024-08-19T14:40:03.182354+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.182354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.d","version":"70'139","reported_seq":1305,"reported_epoch":719,"state":"active+clean","last_fresh":"2024-08-19T14:40:05.233626+0000","last_change":"2024-08-19T14:34:49.476489+0000","last_active":"2024-08-19T14:40:05.233626+0000","last_peered":"2024-08-19T14:40:05.233626+0000","last_clean":"2024-08-19T14:40:05.233626+0000","last_became_active":"2024-08-19T14:18:26.882002+0000","last_became_peered":"2024-08-19T14:18:26.882002+0000","last_unstale":"2024-08-19T14:40:05.233626+0000","last_undegraded":"2024-08-19T14:40:05.233626+0000","last_fullsized":"2024-08-19T14:40:05.233626+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:34:49.476415+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:24:50.276408+0000","last_clean_scrub_stamp":"2024-08-19T14:34:49.476415+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"70'139","reported_seq":1078,"reported_epoch":716,"state":"active+clean","last_fresh":"2024-08-19T14:40:00.177515+0000","last_change":"2024-08-19T14:33:19.781527+0000","last_active":"2024-08-19T14:40:00.177515+0000","last_peered":"2024-08-19T14:40:00.177515+0000","last_clean":"2024-08-19T14:40:00.177515+0000","last_became_active":"2024-08-19T14:27:31.838968+0000","last_became_peered":"2024-08-19T14:27:31.838968+0000","last_unstale":"2024-08-19T14:40:00.177515+0000","last_undegraded":"2024-08-19T14:40:00.177515+0000","last_fullsized":"2024-08-19T14:40:00.177515+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:33:19.781431+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:28:35.206115+0000","last_clean_scrub_stamp":"2024-08-19T14:33:19.781431+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.13","version":"70'139","reported_seq":1620,"reported_epoch":716,"state":"active+clean","last_fresh":"2024-08-19T14:40:00.177588+0000","last_change":"2024-08-19T14:33:35.786236+0000","last_active":"2024-08-19T14:40:00.177588+0000","last_peered":"2024-08-19T14:40:00.177588+0000","last_clean":"2024-08-19T14:40:00.177588+0000","last_became_active":"2024-08-19T14:30:47.939849+0000","last_became_peered":"2024-08-19T14:30:47.939849+0000","last_unstale":"2024-08-19T14:40:00.177588+0000","last_undegraded":"2024-08-19T14:40:00.177588+0000","last_fullsized":"2024-08-19T14:40:00.177588+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:33:35.786105+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:33:35.786105+0000","last_clean_scrub_stamp":"2024-08-19T14:33:35.786105+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.10","version":"70'139","reported_seq":1254,"reported_epoch":719,"state":"active+clean","last_fresh":"2024-08-19T14:40:05.233172+0000","last_change":"2024-08-19T14:34:58.616722+0000","last_active":"2024-08-19T14:40:05.233172+0000","last_peered":"2024-08-19T14:40:05.233172+0000","last_clean":"2024-08-19T14:40:05.233172+0000","last_became_active":"2024-08-19T14:27:31.839611+0000","last_became_peered":"2024-08-19T14:27:31.839611+0000","last_unstale":"2024-08-19T14:40:05.233172+0000","last_undegraded":"2024-08-19T14:40:05.233172+0000","last_fullsized":"2024-08-19T14:40:05.233172+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:34:58.616646+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:34:58.616646+0000","last_clean_scrub_stamp":"2024-08-19T14:34:58.616646+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.11","version":"70'139","reported_seq":1318,"reported_epoch":719,"state":"active+clean","last_fresh":"2024-08-19T14:40:05.325395+0000","last_change":"2024-08-19T14:40:04.812392+0000","last_active":"2024-08-19T14:40:05.325395+0000","last_peered":"2024-08-19T14:40:05.325395+0000","last_clean":"2024-08-19T14:40:05.325395+0000","last_became_active":"2024-08-19T14:34:15.731049+0000","last_became_peered":"2024-08-19T14:34:15.731049+0000","last_unstale":"2024-08-19T14:40:05.325395+0000","last_undegraded":"2024-08-19T14:40:05.325395+0000","last_fullsized":"2024-08-19T14:40:05.325395+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:40:04.812269+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:24:50.547428+0000","last_clean_scrub_stamp":"2024-08-19T14:40:04.812269+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:33.537909+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"70'139","reported_seq":1230,"reported_epoch":716,"state":"active+clean","last_fresh":"2024-08-19T14:40:00.177394+0000","last_change":"2024-08-19T14:33:27.737496+0000","last_active":"2024-08-19T14:40:00.177394+0000","last_peered":"2024-08-19T14:40:00.177394+0000","last_clean":"2024-08-19T14:40:00.177394+0000","last_became_active":"2024-08-19T14:20:28.204724+0000","last_became_peered":"2024-08-19T14:20:28.204724+0000","last_unstale":"2024-08-19T14:40:00.177394+0000","last_undegraded":"2024-08-19T14:40:00.177394+0000","last_fullsized":"2024-08-19T14:40:00.177394+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:33:27.737367+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:28:34.216370+0000","last_clean_scrub_stamp":"2024-08-19T14:33:27.737367+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"70'139","reported_seq":1387,"reported_epoch":719,"state":"active+clean","last_fresh":"2024-08-19T14:40:05.233690+0000","last_change":"2024-08-19T14:38:46.389978+0000","last_active":"2024-08-19T14:40:05.233690+0000","last_peered":"2024-08-19T14:40:05.233690+0000","last_clean":"2024-08-19T14:40:05.233690+0000","last_became_active":"2024-08-19T14:18:31.901556+0000","last_became_peered":"2024-08-19T14:18:31.901556+0000","last_unstale":"2024-08-19T14:40:05.233690+0000","last_undegraded":"2024-08-19T14:40:05.233690+0000","last_fullsized":"2024-08-19T14:40:05.233690+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:38:46.389862+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:38:46.389862+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:40:08.907590+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,0],"acting":[8,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.14","version":"70'139","reported_seq":1417,"reported_epoch":719,"state":"active+clean","last_fresh":"2024-08-19T14:40:05.233384+0000","last_change":"2024-08-19T14:40:03.932359+0000","last_active":"2024-08-19T14:40:05.233384+0000","last_peered":"2024-08-19T14:40:05.233384+0000","last_clean":"2024-08-19T14:40:05.233384+0000","last_became_active":"2024-08-19T14:35:12.529792+0000","last_became_peered":"2024-08-19T14:35:12.529792+0000","last_unstale":"2024-08-19T14:40:05.233384+0000","last_undegraded":"2024-08-19T14:40:05.233384+0000","last_fullsized":"2024-08-19T14:40:05.233384+0000","mapping_epoch":635,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":636,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:40:03.932271+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.932271+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:23.358249+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,9],"acting":[7,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.15","version":"70'139","reported_seq":1260,"reported_epoch":719,"state":"active+clean","last_fresh":"2024-08-19T14:40:05.233469+0000","last_change":"2024-08-19T14:34:57.640316+0000","last_active":"2024-08-19T14:40:05.233469+0000","last_peered":"2024-08-19T14:40:05.233469+0000","last_clean":"2024-08-19T14:40:05.233469+0000","last_became_active":"2024-08-19T14:18:31.912496+0000","last_became_peered":"2024-08-19T14:18:31.912496+0000","last_unstale":"2024-08-19T14:40:05.233469+0000","last_undegraded":"2024-08-19T14:40:05.233469+0000","last_fullsized":"2024-08-19T14:40:05.233469+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:34:57.640213+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:20:09.180923+0000","last_clean_scrub_stamp":"2024-08-19T14:34:57.640213+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"70'139","reported_seq":1250,"reported_epoch":717,"state":"active+clean","last_fresh":"2024-08-19T14:40:03.196205+0000","last_change":"2024-08-19T14:40:00.753735+0000","last_active":"2024-08-19T14:40:03.196205+0000","last_peered":"2024-08-19T14:40:03.196205+0000","last_clean":"2024-08-19T14:40:03.196205+0000","last_became_active":"2024-08-19T14:18:31.901229+0000","last_became_peered":"2024-08-19T14:18:31.901229+0000","last_unstale":"2024-08-19T14:40:03.196205+0000","last_undegraded":"2024-08-19T14:40:03.196205+0000","last_fullsized":"2024-08-19T14:40:03.196205+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:40:00.753636+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:40:00.753636+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:07.574708+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1b","version":"70'139","reported_seq":1635,"reported_epoch":719,"state":"active+clean","last_fresh":"2024-08-19T14:40:05.233519+0000","last_change":"2024-08-19T14:39:32.443757+0000","last_active":"2024-08-19T14:40:05.233519+0000","last_peered":"2024-08-19T14:40:05.233519+0000","last_clean":"2024-08-19T14:40:05.233519+0000","last_became_active":"2024-08-19T14:34:15.718427+0000","last_became_peered":"2024-08-19T14:34:15.718427+0000","last_unstale":"2024-08-19T14:40:05.233519+0000","last_undegraded":"2024-08-19T14:40:05.233519+0000","last_fullsized":"2024-08-19T14:40:05.233519+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:39:32.443656+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:37:59.510697+0000","last_clean_scrub_stamp":"2024-08-19T14:39:32.443656+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:40:40.877428+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.18","version":"70'139","reported_seq":1197,"reported_epoch":719,"state":"active+clean","last_fresh":"2024-08-19T14:40:05.233165+0000","last_change":"2024-08-19T14:39:59.937810+0000","last_active":"2024-08-19T14:40:05.233165+0000","last_peered":"2024-08-19T14:40:05.233165+0000","last_clean":"2024-08-19T14:40:05.233165+0000","last_became_active":"2024-08-19T14:35:11.528530+0000","last_became_peered":"2024-08-19T14:35:11.528530+0000","last_unstale":"2024-08-19T14:40:05.233165+0000","last_undegraded":"2024-08-19T14:40:05.233165+0000","last_fullsized":"2024-08-19T14:40:05.233165+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:39:59.937668+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:27:33.084161+0000","last_clean_scrub_stamp":"2024-08-19T14:39:59.937668+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:17.756316+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.19","version":"70'139","reported_seq":1614,"reported_epoch":719,"state":"active+clean","last_fresh":"2024-08-19T14:40:05.325352+0000","last_change":"2024-08-19T14:39:32.723246+0000","last_active":"2024-08-19T14:40:05.325352+0000","last_peered":"2024-08-19T14:40:05.325352+0000","last_clean":"2024-08-19T14:40:05.325352+0000","last_became_active":"2024-08-19T14:18:31.902547+0000","last_became_peered":"2024-08-19T14:18:31.902547+0000","last_unstale":"2024-08-19T14:40:05.325352+0000","last_undegraded":"2024-08-19T14:40:05.325352+0000","last_fullsized":"2024-08-19T14:40:05.325352+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:39:32.722703+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:39:32.722703+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:40:40.564491+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1e","version":"70'139","reported_seq":625,"reported_epoch":716,"state":"active+clean","last_fresh":"2024-08-19T14:40:00.227255+0000","last_change":"2024-08-19T14:33:17.778415+0000","last_active":"2024-08-19T14:40:00.227255+0000","last_peered":"2024-08-19T14:40:00.227255+0000","last_clean":"2024-08-19T14:40:00.227255+0000","last_became_active":"2024-08-19T14:30:47.909974+0000","last_became_peered":"2024-08-19T14:30:47.909974+0000","last_unstale":"2024-08-19T14:40:00.227255+0000","last_undegraded":"2024-08-19T14:40:00.227255+0000","last_fullsized":"2024-08-19T14:40:00.227255+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:33:17.778295+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:27:04.780939+0000","last_clean_scrub_stamp":"2024-08-19T14:33:17.778295+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1c","version":"70'139","reported_seq":1389,"reported_epoch":719,"state":"active+clean","last_fresh":"2024-08-19T14:40:05.233694+0000","last_change":"2024-08-19T14:39:26.405807+0000","last_active":"2024-08-19T14:40:05.233694+0000","last_peered":"2024-08-19T14:40:05.233694+0000","last_clean":"2024-08-19T14:40:05.233694+0000","last_became_active":"2024-08-19T14:30:47.908866+0000","last_became_peered":"2024-08-19T14:30:47.908866+0000","last_unstale":"2024-08-19T14:40:05.233694+0000","last_undegraded":"2024-08-19T14:40:05.233694+0000","last_fullsized":"2024-08-19T14:40:05.233694+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:39:26.405699+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:39:26.405699+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:40:29.159585+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,11],"acting":[8,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1d","version":"70'139","reported_seq":1292,"reported_epoch":716,"state":"active+clean","last_fresh":"2024-08-19T14:40:00.177404+0000","last_change":"2024-08-19T14:39:57.775710+0000","last_active":"2024-08-19T14:40:00.177404+0000","last_peered":"2024-08-19T14:40:00.177404+0000","last_clean":"2024-08-19T14:40:00.177404+0000","last_became_active":"2024-08-19T14:35:11.540488+0000","last_became_peered":"2024-08-19T14:35:11.540488+0000","last_unstale":"2024-08-19T14:40:00.177404+0000","last_undegraded":"2024-08-19T14:40:00.177404+0000","last_fullsized":"2024-08-19T14:40:00.177404+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:39:57.775631+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:39:57.775631+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:40:59.182147+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":27,"acting":27,"num_store_stats":6},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":856608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187800,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":883816,"kb_used_data":8540,"kb_used_omap":1,"kb_used_meta":875262,"kb_avail":92861336,"statfs":{"total":95995035648,"available":95090008064,"internally_reserved":0,"allocated":8744960,"data_stored":5583553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":896268740},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":10,"up_from":0,"seq":2314987372668,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":74,"seq":317827580258,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":856364,"kb_used_data":8860,"kb_used_omap":1,"kb_used_meta":847486,"kb_avail":92888788,"statfs":{"total":95995035648,"available":95118118912,"internally_reserved":0,"allocated":9072640,"data_stored":5911233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":867826113},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874601,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":775568,"kb_used_data":8576,"kb_used_omap":1,"kb_used_meta":766974,"kb_avail":92969584,"statfs":{"total":95995035648,"available":95200854016,"internally_reserved":0,"allocated":8781824,"data_stored":5608360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":785381829},"hb_peers":[0,1,2,3,4,5,6,7,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":57,"seq":244813136236,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":960392,"kb_used_data":8764,"kb_used_omap":1,"kb_used_meta":951614,"kb_avail":92784760,"statfs":{"total":95995035648,"available":95011594240,"internally_reserved":0,"allocated":8974336,"data_stored":5801261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":974453194},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":206158430575,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692210,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":946484,"kb_used_data":8740,"kb_used_omap":1,"kb_used_meta":937726,"kb_avail":92798668,"statfs":{"total":95995035648,"available":95025836032,"internally_reserved":0,"allocated":8949760,"data_stored":5784801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":960231877},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986551,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1004592,"kb_used_data":8540,"kb_used_omap":1,"kb_used_meta":996030,"kb_avail":92740560,"statfs":{"total":95995035648,"available":94966333440,"internally_reserved":0,"allocated":8744960,"data_stored":5583553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1019935173},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150078,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1041024,"kb_used_data":8884,"kb_used_omap":1,"kb_used_meta":1032126,"kb_avail":92704128,"statfs":{"total":95995035648,"available":94929027072,"internally_reserved":0,"allocated":9097216,"data_stored":5927693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1056897479},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":634,"seq":2723009265724,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":293500,"kb_used_data":8752,"kb_used_omap":4,"kb_used_meta":284731,"kb_avail":93451652,"statfs":{"total":95995035648,"available":95694491648,"internally_reserved":0,"allocated":8962048,"data_stored":5793193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4407,"internal_metadata":291565257},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575238,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1130924,"kb_used_data":8540,"kb_used_omap":1,"kb_used_meta":1122366,"kb_avail":92614228,"statfs":{"total":95995035648,"available":94836969472,"internally_reserved":0,"allocated":8744960,"data_stored":5583553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1149303237},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738763,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":896640,"kb_used_data":8884,"kb_used_omap":1,"kb_used_meta":887742,"kb_avail":92848512,"statfs":{"total":95995035648,"available":95076876288,"internally_reserved":0,"allocated":9097216,"data_stored":5927693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":909048261},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T14:40:07.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:07 smithi032 ceph-mon[20004]: 1.13 scrub starts 2024-08-19T14:40:07.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:07 smithi032 ceph-mon[20004]: 1.13 scrub ok 2024-08-19T14:40:07.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:07 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:40:07.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:07 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:40:07.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:07 smithi032 ceph-mon[20004]: 1.9 scrub starts 2024-08-19T14:40:07.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:07 smithi032 ceph-mon[20004]: 1.9 scrub ok 2024-08-19T14:40:07.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:07 smithi032 ceph-mon[20004]: osdmap e720: 12 total, 12 up, 10 in 2024-08-19T14:40:07.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:07 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/4179695832' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:40:07.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:07 smithi045 ceph-mon[20143]: 1.13 scrub starts 2024-08-19T14:40:07.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:07 smithi045 ceph-mon[20143]: 1.13 scrub ok 2024-08-19T14:40:07.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:07 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:40:07.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:07 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:40:07.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:07 smithi045 ceph-mon[20143]: 1.9 scrub starts 2024-08-19T14:40:07.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:07 smithi045 ceph-mon[20143]: 1.9 scrub ok 2024-08-19T14:40:07.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:07 smithi045 ceph-mon[20143]: osdmap e720: 12 total, 12 up, 10 in 2024-08-19T14:40:07.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:07 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/4179695832' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:40:07.571 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:40:07.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:07.598 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:07.599 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:40:07.700 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:40:07.726 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:07.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:07.727 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:40:07.761 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:40:07.828 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:40:07.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:07.850 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:07.851 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:40:07.952 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:40:07.977 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:07.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:07.978 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:40:08.079 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:40:08.104 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:08.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:08.105 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:40:08.205 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:40:08.230 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:08.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:08.231 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:40:08.333 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:40:08.359 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:08.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:08.360 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:40:08.373 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:08 smithi031 ceph-mon[16336]: pgmap v1726: 47 pgs: 1 creating+peering, 7 unknown, 39 active+clean; 577 KiB data, 8.4 GiB used, 886 GiB / 894 GiB avail 2024-08-19T14:40:08.373 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:08 smithi031 ceph-mon[16336]: from='client.17358 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:40:08.461 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:40:08.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:08.483 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:08.483 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:40:08.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:08 smithi032 ceph-mon[20004]: pgmap v1726: 47 pgs: 1 creating+peering, 7 unknown, 39 active+clean; 577 KiB data, 8.4 GiB used, 886 GiB / 894 GiB avail 2024-08-19T14:40:08.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:08 smithi032 ceph-mon[20004]: from='client.17358 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:40:08.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:08 smithi045 ceph-mon[20143]: pgmap v1726: 47 pgs: 1 creating+peering, 7 unknown, 39 active+clean; 577 KiB data, 8.4 GiB used, 886 GiB / 894 GiB avail 2024-08-19T14:40:08.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:08 smithi045 ceph-mon[20143]: from='client.17358 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:40:08.584 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:40:08.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:08.606 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:08.606 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:40:08.707 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:40:08.729 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:08.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:08.730 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:40:08.831 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:40:08.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:08.858 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:08.858 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:40:08.950 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:40:08.950 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:40:08.958 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:40:08.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:08.984 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:08.984 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:40:09.085 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:40:09.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:09.108 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:09.108 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:40:09.209 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:40:09.222 INFO:teuthology.orchestra.run.smithi031.stderr:enabled application 'rados' on pool 'unique_pool_6' 2024-08-19T14:40:09.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:09.237 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:09.237 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:40:09.338 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:40:09.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:09.362 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:09.362 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:40:09.464 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:40:09.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:09.490 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:09.490 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:40:09.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:09 smithi032 ceph-mon[20004]: 1.15 scrub starts 2024-08-19T14:40:09.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:09 smithi032 ceph-mon[20004]: 1.15 scrub ok 2024-08-19T14:40:09.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:09 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/937644212' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-08-19T14:40:09.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:09 smithi045 ceph-mon[20143]: 1.15 scrub starts 2024-08-19T14:40:09.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:09 smithi045 ceph-mon[20143]: 1.15 scrub ok 2024-08-19T14:40:09.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:09 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/937644212' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-08-19T14:40:09.562 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:09 smithi031 ceph-mon[16336]: 1.15 scrub starts 2024-08-19T14:40:09.562 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:09 smithi031 ceph-mon[16336]: 1.15 scrub ok 2024-08-19T14:40:09.562 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:09 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/937644212' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-08-19T14:40:09.591 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:40:09.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:09.619 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:09.619 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:40:09.720 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:40:09.745 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:09.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:09.746 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:40:09.847 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:40:09.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:09.870 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:09.871 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:40:09.972 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:40:09.998 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:09.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:09.999 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:40:10.100 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:40:10.122 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:10.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:10.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:40:10.225 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:40:10.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:10.248 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:10.248 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:40:10.350 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:40:10.371 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:10.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:10.372 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:40:10.475 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:40:10.501 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:10.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:10.502 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:40:10.508 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-19T14:40:10.510 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:10 smithi032 ceph-mon[20004]: pgmap v1727: 47 pgs: 1 creating+peering, 46 active+clean; 577 KiB data, 8.4 GiB used, 886 GiB / 894 GiB avail 2024-08-19T14:40:10.511 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:10 smithi032 ceph-mon[20004]: 1.d scrub starts 2024-08-19T14:40:10.511 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:10 smithi032 ceph-mon[20004]: 1.d scrub ok 2024-08-19T14:40:10.511 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:10 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/937644212' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-08-19T14:40:10.511 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:10 smithi032 ceph-mon[20004]: osdmap e721: 12 total, 12 up, 10 in 2024-08-19T14:40:10.530 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:10 smithi031 ceph-mon[16336]: pgmap v1727: 47 pgs: 1 creating+peering, 46 active+clean; 577 KiB data, 8.4 GiB used, 886 GiB / 894 GiB avail 2024-08-19T14:40:10.531 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:10 smithi031 ceph-mon[16336]: 1.d scrub starts 2024-08-19T14:40:10.531 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:10 smithi031 ceph-mon[16336]: 1.d scrub ok 2024-08-19T14:40:10.531 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:10 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/937644212' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-08-19T14:40:10.531 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:10 smithi031 ceph-mon[16336]: osdmap e721: 12 total, 12 up, 10 in 2024-08-19T14:40:10.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:10 smithi045 ceph-mon[20143]: pgmap v1727: 47 pgs: 1 creating+peering, 46 active+clean; 577 KiB data, 8.4 GiB used, 886 GiB / 894 GiB avail 2024-08-19T14:40:10.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:10 smithi045 ceph-mon[20143]: 1.d scrub starts 2024-08-19T14:40:10.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:10 smithi045 ceph-mon[20143]: 1.d scrub ok 2024-08-19T14:40:10.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:10 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/937644212' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-08-19T14:40:10.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:10 smithi045 ceph-mon[20143]: osdmap e721: 12 total, 12 up, 10 in 2024-08-19T14:40:10.604 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:40:10.629 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:10.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:10.630 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:40:10.731 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:40:10.752 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:10.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:10.753 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:40:10.854 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:40:10.880 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:10.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:10.881 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:40:10.974 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-08-19T14:40:10.982 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:40:11.038 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:11.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:11.039 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:40:11.109 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:40:11.140 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:40:11.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:11.166 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:11.166 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:40:11.214 DEBUG:teuthology.orchestra.run.smithi045:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_6 bench 90 write' 2024-08-19T14:40:11.217 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-08-19T14:40:11.217 INFO:teuthology.orchestra.run:waiting for 3000 2024-08-19T14:40:11.223 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is set 2024-08-19T14:40:11.268 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:40:11.286 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:hints = 1 2024-08-19T14:40:11.286 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-08-19T14:40:11.286 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Object prefix: benchmark_data_smithi045_137103 2024-08-19T14:40:11.286 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-19T14:40:11.286 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 0 0 0 0 0 0 - 0 2024-08-19T14:40:11.296 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:11.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:11.297 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:40:11.398 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:40:11.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:11.427 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:11.427 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:40:11.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:11 smithi032 ceph-mon[20004]: 1.4 scrub starts 2024-08-19T14:40:11.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:11 smithi032 ceph-mon[20004]: 1.4 scrub ok 2024-08-19T14:40:11.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:11 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3669624403' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:40:11.528 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:40:11.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:11 smithi045 ceph-mon[20143]: 1.4 scrub starts 2024-08-19T14:40:11.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:11 smithi045 ceph-mon[20143]: 1.4 scrub ok 2024-08-19T14:40:11.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:11 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3669624403' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:40:11.554 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:11.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:11.555 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:40:11.592 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:40:11.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:11 smithi031 ceph-mon[16336]: 1.4 scrub starts 2024-08-19T14:40:11.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:11 smithi031 ceph-mon[16336]: 1.4 scrub ok 2024-08-19T14:40:11.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:11 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3669624403' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:40:11.656 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:40:11.691 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:11.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:11.692 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:40:11.793 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:40:11.829 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:11.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:11.830 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:40:11.931 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:40:11.967 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:11.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:11.968 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:40:12.068 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:40:12.102 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:12.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:12.103 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:40:12.204 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:40:12.226 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:12.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:12.227 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:40:12.286 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 1 16 3392 3376 210.983 211 0.00142278 0.00394206 2024-08-19T14:40:12.294 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:40:12.328 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:40:12.354 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:40:12.361 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:12.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:12.362 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:40:12.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:12 smithi031 ceph-mon[16336]: pgmap v1729: 47 pgs: 47 active+clean; 577 KiB data, 8.4 GiB used, 886 GiB / 894 GiB avail 2024-08-19T14:40:12.368 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:12 smithi031 ceph-mon[16336]: 1.2 scrub starts 2024-08-19T14:40:12.368 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:12 smithi031 ceph-mon[16336]: 1.2 scrub ok 2024-08-19T14:40:12.368 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:12 smithi031 ceph-mon[16336]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-19T14:40:12.368 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:12 smithi031 ceph-mon[16336]: Cluster is now healthy 2024-08-19T14:40:12.368 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:12 smithi031 ceph-mon[16336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:40:12.368 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:12 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3669624403' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:40:12.368 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:12 smithi031 ceph-mon[16336]: osdmap e722: 12 total, 12 up, 10 in 2024-08-19T14:40:12.463 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:40:12.489 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:12.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:12.490 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:40:12.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:12 smithi032 ceph-mon[20004]: pgmap v1729: 47 pgs: 47 active+clean; 577 KiB data, 8.4 GiB used, 886 GiB / 894 GiB avail 2024-08-19T14:40:12.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:12 smithi032 ceph-mon[20004]: 1.2 scrub starts 2024-08-19T14:40:12.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:12 smithi032 ceph-mon[20004]: 1.2 scrub ok 2024-08-19T14:40:12.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:12 smithi032 ceph-mon[20004]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-19T14:40:12.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:12 smithi032 ceph-mon[20004]: Cluster is now healthy 2024-08-19T14:40:12.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:12 smithi032 ceph-mon[20004]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:40:12.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:12 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3669624403' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:40:12.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:12 smithi032 ceph-mon[20004]: osdmap e722: 12 total, 12 up, 10 in 2024-08-19T14:40:12.532 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:40:12.535 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-08-19T14:40:12.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:12 smithi045 ceph-mon[20143]: pgmap v1729: 47 pgs: 47 active+clean; 577 KiB data, 8.4 GiB used, 886 GiB / 894 GiB avail 2024-08-19T14:40:12.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:12 smithi045 ceph-mon[20143]: 1.2 scrub starts 2024-08-19T14:40:12.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:12 smithi045 ceph-mon[20143]: 1.2 scrub ok 2024-08-19T14:40:12.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:12 smithi045 ceph-mon[20143]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-19T14:40:12.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:12 smithi045 ceph-mon[20143]: Cluster is now healthy 2024-08-19T14:40:12.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:12 smithi045 ceph-mon[20143]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:40:12.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:12 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3669624403' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:40:12.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:12 smithi045 ceph-mon[20143]: osdmap e722: 12 total, 12 up, 10 in 2024-08-19T14:40:12.590 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:40:12.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:12.619 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:12.619 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:40:12.721 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:40:12.754 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:12.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:12.756 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:40:12.857 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:40:12.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:12.893 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:12.893 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:40:12.994 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:40:13.024 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:13.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:13.025 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:40:13.126 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:40:13.147 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:40:13.147 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:40:13.151 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:13.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:13.152 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:40:13.254 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:40:13.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:13.281 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:13.281 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:40:13.286 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 2 15 7271 7256 226.733 242.5 0.005462 0.00440277 2024-08-19T14:40:13.381 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:40:13.413 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:13.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:13.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:40:13.516 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:40:13.524 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:13 smithi031 ceph-mon[16336]: pgmap v1731: 47 pgs: 47 active+clean; 35 MiB data, 8.4 GiB used, 886 GiB / 894 GiB avail; 5.6 MiB/s wr, 89 op/s 2024-08-19T14:40:13.524 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:13 smithi031 ceph-mon[16336]: from='client.17391 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:40:13.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:13.538 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:13.538 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:40:13.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:13 smithi045 ceph-mon[20143]: pgmap v1731: 47 pgs: 47 active+clean; 35 MiB data, 8.4 GiB used, 886 GiB / 894 GiB avail; 5.6 MiB/s wr, 89 op/s 2024-08-19T14:40:13.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:13 smithi045 ceph-mon[20143]: from='client.17391 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:40:13.639 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:40:13.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:13.672 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:13.672 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:40:13.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:13 smithi032 ceph-mon[20004]: pgmap v1731: 47 pgs: 47 active+clean; 35 MiB data, 8.4 GiB used, 886 GiB / 894 GiB avail; 5.6 MiB/s wr, 89 op/s 2024-08-19T14:40:13.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:13 smithi032 ceph-mon[20004]: from='client.17391 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:40:13.773 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:40:13.807 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:13.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:13.809 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:40:13.909 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:40:13.945 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:13.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:13.947 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:40:14.048 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:40:14.059 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-19T14:40:14.078 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:14.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:14.079 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:40:14.091 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":1731,"stamp":"2024-08-19T14:40:12.280413+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":36373024,"num_objects":548,"num_object_clones":0,"num_object_copies":1644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":546,"num_write_kb":34944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":856608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4855,"ondisk_log_size":4855,"up":141,"acting":141,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":134,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":8789728,"kb_used_data":87500,"kb_used_omap":18,"kb_used_meta":8702061,"kb_avail":928661792,"statfs":{"total":959950356480,"available":950949675008,"internally_reserved":0,"allocated":89600000,"data_stored":57792949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18760,"internal_metadata":8910911160},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":35782656,"num_objects":546,"num_object_clones":0,"num_object_copies":1638,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":546,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":546,"num_write_kb":34944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.073068"},"pg_stats":[{"pgid":"8.9","version":"0'0","reported_seq":18,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-08-19T14:40:09.297197+0000","last_change":"2024-08-19T14:40:04.442466+0000","last_active":"2024-08-19T14:40:09.297197+0000","last_peered":"2024-08-19T14:40:09.297197+0000","last_clean":"2024-08-19T14:40:09.297197+0000","last_became_active":"2024-08-19T14:40:04.442173+0000","last_became_peered":"2024-08-19T14:40:04.442173+0000","last_unstale":"2024-08-19T14:40:09.297197+0000","last_undegraded":"2024-08-19T14:40:09.297197+0000","last_fullsized":"2024-08-19T14:40:09.297197+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":717,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.182354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:25.949039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"70'139","reported_seq":1769,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-08-19T14:40:09.297127+0000","last_change":"2024-08-19T14:39:45.569137+0000","last_active":"2024-08-19T14:40:09.297127+0000","last_peered":"2024-08-19T14:40:09.297127+0000","last_clean":"2024-08-19T14:40:09.297127+0000","last_became_active":"2024-08-19T14:35:13.950224+0000","last_became_peered":"2024-08-19T14:35:13.950224+0000","last_unstale":"2024-08-19T14:40:09.297127+0000","last_undegraded":"2024-08-19T14:40:09.297127+0000","last_fullsized":"2024-08-19T14:40:09.297127+0000","mapping_epoch":637,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":638,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:39:45.569030+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:19:58.072083+0000","last_clean_scrub_stamp":"2024-08-19T14:39:45.569030+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:40:57.206713+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,9],"acting":[0,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":18,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-08-19T14:40:09.798812+0000","last_change":"2024-08-19T14:40:04.442002+0000","last_active":"2024-08-19T14:40:09.798812+0000","last_peered":"2024-08-19T14:40:09.798812+0000","last_clean":"2024-08-19T14:40:09.798812+0000","last_became_active":"2024-08-19T14:40:04.441586+0000","last_became_peered":"2024-08-19T14:40:04.441586+0000","last_unstale":"2024-08-19T14:40:09.798812+0000","last_undegraded":"2024-08-19T14:40:09.798812+0000","last_fullsized":"2024-08-19T14:40:09.798812+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":717,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.182354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:26.482555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,9],"acting":[8,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"70'139","reported_seq":1611,"reported_epoch":720,"state":"active+clean","last_fresh":"2024-08-19T14:40:06.699817+0000","last_change":"2024-08-19T14:40:02.681316+0000","last_active":"2024-08-19T14:40:06.699817+0000","last_peered":"2024-08-19T14:40:06.699817+0000","last_clean":"2024-08-19T14:40:06.699817+0000","last_became_active":"2024-08-19T14:18:13.804721+0000","last_became_peered":"2024-08-19T14:18:13.804721+0000","last_unstale":"2024-08-19T14:40:06.699817+0000","last_undegraded":"2024-08-19T14:40:06.699817+0000","last_fullsized":"2024-08-19T14:40:06.699817+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:40:02.681185+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:40:02.681185+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:22.300124+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":18,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-08-19T14:40:09.798751+0000","last_change":"2024-08-19T14:40:04.200633+0000","last_active":"2024-08-19T14:40:09.798751+0000","last_peered":"2024-08-19T14:40:09.798751+0000","last_clean":"2024-08-19T14:40:09.798751+0000","last_became_active":"2024-08-19T14:40:04.200299+0000","last_became_peered":"2024-08-19T14:40:04.200299+0000","last_unstale":"2024-08-19T14:40:09.798751+0000","last_undegraded":"2024-08-19T14:40:09.798751+0000","last_fullsized":"2024-08-19T14:40:09.798751+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":717,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.182354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:27.707935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,1],"acting":[8,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.2","version":"70'139","reported_seq":1562,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-08-19T14:40:09.223825+0000","last_change":"2024-08-19T14:34:55.580212+0000","last_active":"2024-08-19T14:40:09.223825+0000","last_peered":"2024-08-19T14:40:09.223825+0000","last_clean":"2024-08-19T14:40:09.223825+0000","last_became_active":"2024-08-19T14:20:50.742354+0000","last_became_peered":"2024-08-19T14:20:50.742354+0000","last_unstale":"2024-08-19T14:40:09.223825+0000","last_undegraded":"2024-08-19T14:40:09.223825+0000","last_fullsized":"2024-08-19T14:40:09.223825+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:34:55.580141+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:19:59.302931+0000","last_clean_scrub_stamp":"2024-08-19T14:34:55.580141+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":18,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-08-19T14:40:09.223908+0000","last_change":"2024-08-19T14:40:04.443203+0000","last_active":"2024-08-19T14:40:09.223908+0000","last_peered":"2024-08-19T14:40:09.223908+0000","last_clean":"2024-08-19T14:40:09.223908+0000","last_became_active":"2024-08-19T14:40:04.442802+0000","last_became_peered":"2024-08-19T14:40:04.442802+0000","last_unstale":"2024-08-19T14:40:09.223908+0000","last_undegraded":"2024-08-19T14:40:09.223908+0000","last_fullsized":"2024-08-19T14:40:09.223908+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":717,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.182354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:14.120759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,9],"acting":[3,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"70'139","reported_seq":868,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-08-19T14:40:09.297301+0000","last_change":"2024-08-19T14:39:37.543259+0000","last_active":"2024-08-19T14:40:09.297301+0000","last_peered":"2024-08-19T14:40:09.297301+0000","last_clean":"2024-08-19T14:40:09.297301+0000","last_became_active":"2024-08-19T14:35:12.530566+0000","last_became_peered":"2024-08-19T14:35:12.530566+0000","last_unstale":"2024-08-19T14:40:09.297301+0000","last_undegraded":"2024-08-19T14:40:09.297301+0000","last_fullsized":"2024-08-19T14:40:09.297301+0000","mapping_epoch":635,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":636,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:39:37.543187+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:39:37.543187+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:40:56.083490+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,2],"acting":[0,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":18,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-08-19T14:40:09.297449+0000","last_change":"2024-08-19T14:40:04.299744+0000","last_active":"2024-08-19T14:40:09.297449+0000","last_peered":"2024-08-19T14:40:09.297449+0000","last_clean":"2024-08-19T14:40:09.297449+0000","last_became_active":"2024-08-19T14:40:04.299403+0000","last_became_peered":"2024-08-19T14:40:04.299403+0000","last_unstale":"2024-08-19T14:40:09.297449+0000","last_undegraded":"2024-08-19T14:40:09.297449+0000","last_fullsized":"2024-08-19T14:40:09.297449+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":717,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.182354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:22.448115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"70'139","reported_seq":1272,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-08-19T14:40:09.224022+0000","last_change":"2024-08-19T14:39:56.993699+0000","last_active":"2024-08-19T14:40:09.224022+0000","last_peered":"2024-08-19T14:40:09.224022+0000","last_clean":"2024-08-19T14:40:09.224022+0000","last_became_active":"2024-08-19T14:35:11.536828+0000","last_became_peered":"2024-08-19T14:35:11.536828+0000","last_unstale":"2024-08-19T14:40:09.224022+0000","last_undegraded":"2024-08-19T14:40:09.224022+0000","last_fullsized":"2024-08-19T14:40:09.224022+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:39:56.993616+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:21:46.057624+0000","last_clean_scrub_stamp":"2024-08-19T14:39:56.993616+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:05.362216+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.f","version":"722'94","reported_seq":114,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-08-19T14:40:11.746020+0000","last_change":"2024-08-19T14:40:04.299935+0000","last_active":"2024-08-19T14:40:11.746020+0000","last_peered":"2024-08-19T14:40:11.746020+0000","last_clean":"2024-08-19T14:40:11.746020+0000","last_became_active":"2024-08-19T14:40:04.299656+0000","last_became_peered":"2024-08-19T14:40:04.299656+0000","last_unstale":"2024-08-19T14:40:11.746020+0000","last_undegraded":"2024-08-19T14:40:11.746020+0000","last_fullsized":"2024-08-19T14:40:11.746020+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":717,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.182354+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:22.129102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6160384,"num_objects":94,"num_object_clones":0,"num_object_copies":282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":94,"num_write_kb":6016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,1],"acting":[11,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.6","version":"70'139","reported_seq":839,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-08-19T14:40:09.799002+0000","last_change":"2024-08-19T14:39:30.460768+0000","last_active":"2024-08-19T14:40:09.799002+0000","last_peered":"2024-08-19T14:40:09.799002+0000","last_clean":"2024-08-19T14:40:09.799002+0000","last_became_active":"2024-08-19T14:34:15.717054+0000","last_became_peered":"2024-08-19T14:34:15.717054+0000","last_unstale":"2024-08-19T14:40:09.799002+0000","last_undegraded":"2024-08-19T14:40:09.799002+0000","last_fullsized":"2024-08-19T14:40:09.799002+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:39:30.460649+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:27.848870+0000","last_clean_scrub_stamp":"2024-08-19T14:39:30.460649+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:40:59.351491+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":18,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-08-19T14:40:09.798786+0000","last_change":"2024-08-19T14:40:04.442056+0000","last_active":"2024-08-19T14:40:09.798786+0000","last_peered":"2024-08-19T14:40:09.798786+0000","last_clean":"2024-08-19T14:40:09.798786+0000","last_became_active":"2024-08-19T14:40:04.441602+0000","last_became_peered":"2024-08-19T14:40:04.441602+0000","last_unstale":"2024-08-19T14:40:09.798786+0000","last_undegraded":"2024-08-19T14:40:09.798786+0000","last_fullsized":"2024-08-19T14:40:09.798786+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":717,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.182354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:20.409556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,3],"acting":[8,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.5","version":"70'139","reported_seq":796,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-08-19T14:40:11.226832+0000","last_change":"2024-08-19T14:39:54.886149+0000","last_active":"2024-08-19T14:40:11.226832+0000","last_peered":"2024-08-19T14:40:11.226832+0000","last_clean":"2024-08-19T14:40:11.226832+0000","last_became_active":"2024-08-19T14:35:11.532858+0000","last_became_peered":"2024-08-19T14:35:11.532858+0000","last_unstale":"2024-08-19T14:40:11.226832+0000","last_undegraded":"2024-08-19T14:40:11.226832+0000","last_fullsized":"2024-08-19T14:40:11.226832+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:39:54.886056+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:39:54.886056+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:40:57.762183+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,0],"acting":[11,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.d","version":"722'113","reported_seq":133,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-08-19T14:40:11.745666+0000","last_change":"2024-08-19T14:40:04.443252+0000","last_active":"2024-08-19T14:40:11.745666+0000","last_peered":"2024-08-19T14:40:11.745666+0000","last_clean":"2024-08-19T14:40:11.745666+0000","last_became_active":"2024-08-19T14:40:04.442093+0000","last_became_peered":"2024-08-19T14:40:04.442093+0000","last_unstale":"2024-08-19T14:40:11.745666+0000","last_undegraded":"2024-08-19T14:40:11.745666+0000","last_fullsized":"2024-08-19T14:40:11.745666+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":717,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.182354+0000","objects_scrubbed":0,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:22.854336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7405568,"num_objects":113,"num_object_clones":0,"num_object_copies":339,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":113,"num_write_kb":7232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,2],"acting":[11,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.4","version":"70'139","reported_seq":673,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-08-19T14:40:11.226745+0000","last_change":"2024-08-19T14:40:09.799125+0000","last_active":"2024-08-19T14:40:11.226745+0000","last_peered":"2024-08-19T14:40:11.226745+0000","last_clean":"2024-08-19T14:40:11.226745+0000","last_became_active":"2024-08-19T14:30:47.909067+0000","last_became_peered":"2024-08-19T14:30:47.909067+0000","last_unstale":"2024-08-19T14:40:11.226745+0000","last_undegraded":"2024-08-19T14:40:11.226745+0000","last_fullsized":"2024-08-19T14:40:11.226745+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:40:09.798994+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:40:09.798994+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:32.468916+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.3","version":"722'124","reported_seq":143,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-08-19T14:40:11.745935+0000","last_change":"2024-08-19T14:40:04.443269+0000","last_active":"2024-08-19T14:40:11.745935+0000","last_peered":"2024-08-19T14:40:11.745935+0000","last_clean":"2024-08-19T14:40:11.745935+0000","last_became_active":"2024-08-19T14:40:04.442837+0000","last_became_peered":"2024-08-19T14:40:04.442837+0000","last_unstale":"2024-08-19T14:40:11.745935+0000","last_undegraded":"2024-08-19T14:40:11.745935+0000","last_fullsized":"2024-08-19T14:40:11.745935+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":717,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.182354+0000","objects_scrubbed":0,"log_size":124,"log_dups_size":0,"ondisk_log_size":124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:13.689400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8126464,"num_objects":124,"num_object_clones":0,"num_object_copies":372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":124,"num_write_kb":7936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.a","version":"70'139","reported_seq":834,"reported_epoch":720,"state":"active+clean","last_fresh":"2024-08-19T14:40:06.208321+0000","last_change":"2024-08-19T14:40:05.713501+0000","last_active":"2024-08-19T14:40:06.208321+0000","last_peered":"2024-08-19T14:40:06.208321+0000","last_clean":"2024-08-19T14:40:06.208321+0000","last_became_active":"2024-08-19T14:34:15.717472+0000","last_became_peered":"2024-08-19T14:34:15.717472+0000","last_unstale":"2024-08-19T14:40:06.208321+0000","last_undegraded":"2024-08-19T14:40:06.208321+0000","last_fullsized":"2024-08-19T14:40:06.208321+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:40:05.713411+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:28.864870+0000","last_clean_scrub_stamp":"2024-08-19T14:40:05.713411+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:12.701795+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,0],"acting":[4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":18,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-08-19T14:40:09.223908+0000","last_change":"2024-08-19T14:40:04.443641+0000","last_active":"2024-08-19T14:40:09.223908+0000","last_peered":"2024-08-19T14:40:09.223908+0000","last_clean":"2024-08-19T14:40:09.223908+0000","last_became_active":"2024-08-19T14:40:04.443254+0000","last_became_peered":"2024-08-19T14:40:04.443254+0000","last_unstale":"2024-08-19T14:40:09.223908+0000","last_undegraded":"2024-08-19T14:40:09.223908+0000","last_fullsized":"2024-08-19T14:40:09.223908+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":717,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.182354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:21.728211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,9],"acting":[3,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"70'139","reported_seq":1772,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-08-19T14:40:09.223825+0000","last_change":"2024-08-19T14:40:05.923941+0000","last_active":"2024-08-19T14:40:09.223825+0000","last_peered":"2024-08-19T14:40:09.223825+0000","last_clean":"2024-08-19T14:40:09.223825+0000","last_became_active":"2024-08-19T14:35:12.529628+0000","last_became_peered":"2024-08-19T14:35:12.529628+0000","last_unstale":"2024-08-19T14:40:09.223825+0000","last_undegraded":"2024-08-19T14:40:09.223825+0000","last_fullsized":"2024-08-19T14:40:09.223825+0000","mapping_epoch":635,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":636,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:40:05.923850+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:40:05.923850+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:22.262135+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":15,"reported_epoch":720,"state":"active+clean","last_fresh":"2024-08-19T14:40:06.699904+0000","last_change":"2024-08-19T14:40:04.441532+0000","last_active":"2024-08-19T14:40:06.699904+0000","last_peered":"2024-08-19T14:40:06.699904+0000","last_clean":"2024-08-19T14:40:06.699904+0000","last_became_active":"2024-08-19T14:40:04.441139+0000","last_became_peered":"2024-08-19T14:40:04.441139+0000","last_unstale":"2024-08-19T14:40:06.699904+0000","last_undegraded":"2024-08-19T14:40:06.699904+0000","last_fullsized":"2024-08-19T14:40:06.699904+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":717,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.182354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:32.439537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,2],"acting":[1,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"70'139","reported_seq":802,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-08-19T14:40:09.224028+0000","last_change":"2024-08-19T14:39:59.000415+0000","last_active":"2024-08-19T14:40:09.224028+0000","last_peered":"2024-08-19T14:40:09.224028+0000","last_clean":"2024-08-19T14:40:09.224028+0000","last_became_active":"2024-08-19T14:35:11.537474+0000","last_became_peered":"2024-08-19T14:35:11.537474+0000","last_unstale":"2024-08-19T14:40:09.224028+0000","last_undegraded":"2024-08-19T14:40:09.224028+0000","last_fullsized":"2024-08-19T14:40:09.224028+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:39:59.000300+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:39:59.000300+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:11.528949+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":18,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-08-19T14:40:09.297246+0000","last_change":"2024-08-19T14:40:04.441786+0000","last_active":"2024-08-19T14:40:09.297246+0000","last_peered":"2024-08-19T14:40:09.297246+0000","last_clean":"2024-08-19T14:40:09.297246+0000","last_became_active":"2024-08-19T14:40:04.441445+0000","last_became_peered":"2024-08-19T14:40:04.441445+0000","last_unstale":"2024-08-19T14:40:09.297246+0000","last_undegraded":"2024-08-19T14:40:09.297246+0000","last_fullsized":"2024-08-19T14:40:09.297246+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":717,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.182354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:23.583504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,9],"acting":[0,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.b","version":"70'139","reported_seq":1125,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-08-19T14:40:11.226853+0000","last_change":"2024-08-19T14:40:03.813911+0000","last_active":"2024-08-19T14:40:11.226853+0000","last_peered":"2024-08-19T14:40:11.226853+0000","last_clean":"2024-08-19T14:40:11.226853+0000","last_became_active":"2024-08-19T14:30:47.941200+0000","last_became_peered":"2024-08-19T14:30:47.941200+0000","last_unstale":"2024-08-19T14:40:11.226853+0000","last_undegraded":"2024-08-19T14:40:11.226853+0000","last_fullsized":"2024-08-19T14:40:11.226853+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:40:03.813794+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.813794+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:29.146651+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":15,"reported_epoch":720,"state":"active+clean","last_fresh":"2024-08-19T14:40:06.700121+0000","last_change":"2024-08-19T14:40:04.299820+0000","last_active":"2024-08-19T14:40:06.700121+0000","last_peered":"2024-08-19T14:40:06.700121+0000","last_clean":"2024-08-19T14:40:06.700121+0000","last_became_active":"2024-08-19T14:40:04.299448+0000","last_became_peered":"2024-08-19T14:40:04.299448+0000","last_unstale":"2024-08-19T14:40:06.700121+0000","last_undegraded":"2024-08-19T14:40:06.700121+0000","last_fullsized":"2024-08-19T14:40:06.700121+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":717,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.182354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:09.552673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.e","version":"70'139","reported_seq":1311,"reported_epoch":720,"state":"active+clean","last_fresh":"2024-08-19T14:40:06.208387+0000","last_change":"2024-08-19T14:40:03.764766+0000","last_active":"2024-08-19T14:40:06.208387+0000","last_peered":"2024-08-19T14:40:06.208387+0000","last_clean":"2024-08-19T14:40:06.208387+0000","last_became_active":"2024-08-19T14:18:26.884105+0000","last_became_peered":"2024-08-19T14:18:26.884105+0000","last_unstale":"2024-08-19T14:40:06.208387+0000","last_undegraded":"2024-08-19T14:40:06.208387+0000","last_fullsized":"2024-08-19T14:40:06.208387+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:40:03.764672+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:40:03.764672+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.764672+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:11.797638+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.6","version":"722'104","reported_seq":124,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-08-19T14:40:11.739432+0000","last_change":"2024-08-19T14:40:04.443511+0000","last_active":"2024-08-19T14:40:11.739432+0000","last_peered":"2024-08-19T14:40:11.739432+0000","last_clean":"2024-08-19T14:40:11.739432+0000","last_became_active":"2024-08-19T14:40:04.442943+0000","last_became_peered":"2024-08-19T14:40:04.442943+0000","last_unstale":"2024-08-19T14:40:11.739432+0000","last_undegraded":"2024-08-19T14:40:11.739432+0000","last_fullsized":"2024-08-19T14:40:11.739432+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":717,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.182354+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:29.082021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6815744,"num_objects":104,"num_object_clones":0,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":104,"num_write_kb":6656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,2],"acting":[11,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.f","version":"70'139","reported_seq":1063,"reported_epoch":720,"state":"active+clean","last_fresh":"2024-08-19T14:40:06.208424+0000","last_change":"2024-08-19T14:39:59.812809+0000","last_active":"2024-08-19T14:40:06.208424+0000","last_peered":"2024-08-19T14:40:06.208424+0000","last_clean":"2024-08-19T14:40:06.208424+0000","last_became_active":"2024-08-19T14:27:31.837754+0000","last_became_peered":"2024-08-19T14:27:31.837754+0000","last_unstale":"2024-08-19T14:40:06.208424+0000","last_undegraded":"2024-08-19T14:40:06.208424+0000","last_fullsized":"2024-08-19T14:40:06.208424+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:39:59.812612+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:39:59.812612+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:07.074257+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":15,"reported_epoch":720,"state":"active+clean","last_fresh":"2024-08-19T14:40:06.208280+0000","last_change":"2024-08-19T14:40:04.200422+0000","last_active":"2024-08-19T14:40:06.208280+0000","last_peered":"2024-08-19T14:40:06.208280+0000","last_clean":"2024-08-19T14:40:06.208280+0000","last_became_active":"2024-08-19T14:40:04.200106+0000","last_became_peered":"2024-08-19T14:40:04.200106+0000","last_unstale":"2024-08-19T14:40:06.208280+0000","last_undegraded":"2024-08-19T14:40:06.208280+0000","last_fullsized":"2024-08-19T14:40:06.208280+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":717,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.182354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:09.385596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,11],"acting":[4,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.c","version":"70'139","reported_seq":1420,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-08-19T14:40:09.223900+0000","last_change":"2024-08-19T14:40:00.947434+0000","last_active":"2024-08-19T14:40:09.223900+0000","last_peered":"2024-08-19T14:40:09.223900+0000","last_clean":"2024-08-19T14:40:09.223900+0000","last_became_active":"2024-08-19T14:34:15.733282+0000","last_became_peered":"2024-08-19T14:34:15.733282+0000","last_unstale":"2024-08-19T14:40:09.223900+0000","last_undegraded":"2024-08-19T14:40:09.223900+0000","last_fullsized":"2024-08-19T14:40:09.223900+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:40:00.947341+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:22:52.164586+0000","last_clean_scrub_stamp":"2024-08-19T14:40:00.947341+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:21.334102+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,9],"acting":[3,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"722'111","reported_seq":131,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-08-19T14:40:11.742630+0000","last_change":"2024-08-19T14:40:04.443126+0000","last_active":"2024-08-19T14:40:11.742630+0000","last_peered":"2024-08-19T14:40:11.742630+0000","last_clean":"2024-08-19T14:40:11.742630+0000","last_became_active":"2024-08-19T14:40:04.442757+0000","last_became_peered":"2024-08-19T14:40:04.442757+0000","last_unstale":"2024-08-19T14:40:11.742630+0000","last_undegraded":"2024-08-19T14:40:11.742630+0000","last_fullsized":"2024-08-19T14:40:11.742630+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":717,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.182354+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:06.385105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7274496,"num_objects":111,"num_object_clones":0,"num_object_copies":333,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":111,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":111,"num_write_kb":7104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,9],"acting":[11,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.d","version":"70'139","reported_seq":1317,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-08-19T14:40:09.224233+0000","last_change":"2024-08-19T14:40:08.911300+0000","last_active":"2024-08-19T14:40:09.224233+0000","last_peered":"2024-08-19T14:40:09.224233+0000","last_clean":"2024-08-19T14:40:09.224233+0000","last_became_active":"2024-08-19T14:18:26.882002+0000","last_became_peered":"2024-08-19T14:18:26.882002+0000","last_unstale":"2024-08-19T14:40:09.224233+0000","last_undegraded":"2024-08-19T14:40:09.224233+0000","last_fullsized":"2024-08-19T14:40:09.224233+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:40:08.911204+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:24:50.276408+0000","last_clean_scrub_stamp":"2024-08-19T14:40:08.911204+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:27.367708+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"70'139","reported_seq":1094,"reported_epoch":720,"state":"active+clean","last_fresh":"2024-08-19T14:40:06.699937+0000","last_change":"2024-08-19T14:40:04.665520+0000","last_active":"2024-08-19T14:40:06.699937+0000","last_peered":"2024-08-19T14:40:06.699937+0000","last_clean":"2024-08-19T14:40:06.699937+0000","last_became_active":"2024-08-19T14:27:31.838968+0000","last_became_peered":"2024-08-19T14:27:31.838968+0000","last_unstale":"2024-08-19T14:40:06.699937+0000","last_undegraded":"2024-08-19T14:40:06.699937+0000","last_fullsized":"2024-08-19T14:40:06.699937+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:40:04.665441+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:28:35.206115+0000","last_clean_scrub_stamp":"2024-08-19T14:40:04.665441+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:25.039402+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.13","version":"70'139","reported_seq":1636,"reported_epoch":720,"state":"active+clean","last_fresh":"2024-08-19T14:40:06.699968+0000","last_change":"2024-08-19T14:40:05.659348+0000","last_active":"2024-08-19T14:40:06.699968+0000","last_peered":"2024-08-19T14:40:06.699968+0000","last_clean":"2024-08-19T14:40:06.699968+0000","last_became_active":"2024-08-19T14:30:47.939849+0000","last_became_peered":"2024-08-19T14:30:47.939849+0000","last_unstale":"2024-08-19T14:40:06.699968+0000","last_undegraded":"2024-08-19T14:40:06.699968+0000","last_fullsized":"2024-08-19T14:40:06.699968+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:40:05.659275+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:33:35.786105+0000","last_clean_scrub_stamp":"2024-08-19T14:40:05.659275+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:30.843663+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.10","version":"70'139","reported_seq":1258,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-08-19T14:40:09.223996+0000","last_change":"2024-08-19T14:34:58.616722+0000","last_active":"2024-08-19T14:40:09.223996+0000","last_peered":"2024-08-19T14:40:09.223996+0000","last_clean":"2024-08-19T14:40:09.223996+0000","last_became_active":"2024-08-19T14:27:31.839611+0000","last_became_peered":"2024-08-19T14:27:31.839611+0000","last_unstale":"2024-08-19T14:40:09.223996+0000","last_undegraded":"2024-08-19T14:40:09.223996+0000","last_fullsized":"2024-08-19T14:40:09.223996+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:34:58.616646+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:34:58.616646+0000","last_clean_scrub_stamp":"2024-08-19T14:34:58.616646+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.11","version":"70'139","reported_seq":1322,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-08-19T14:40:09.297278+0000","last_change":"2024-08-19T14:40:04.812392+0000","last_active":"2024-08-19T14:40:09.297278+0000","last_peered":"2024-08-19T14:40:09.297278+0000","last_clean":"2024-08-19T14:40:09.297278+0000","last_became_active":"2024-08-19T14:34:15.731049+0000","last_became_peered":"2024-08-19T14:34:15.731049+0000","last_unstale":"2024-08-19T14:40:09.297278+0000","last_undegraded":"2024-08-19T14:40:09.297278+0000","last_fullsized":"2024-08-19T14:40:09.297278+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:40:04.812269+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:24:50.547428+0000","last_clean_scrub_stamp":"2024-08-19T14:40:04.812269+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:33.537909+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"70'139","reported_seq":1246,"reported_epoch":720,"state":"active+clean","last_fresh":"2024-08-19T14:40:06.700036+0000","last_change":"2024-08-19T14:40:03.700753+0000","last_active":"2024-08-19T14:40:06.700036+0000","last_peered":"2024-08-19T14:40:06.700036+0000","last_clean":"2024-08-19T14:40:06.700036+0000","last_became_active":"2024-08-19T14:20:28.204724+0000","last_became_peered":"2024-08-19T14:20:28.204724+0000","last_unstale":"2024-08-19T14:40:06.700036+0000","last_undegraded":"2024-08-19T14:40:06.700036+0000","last_fullsized":"2024-08-19T14:40:06.700036+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:40:03.700675+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:28:34.216370+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.700675+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:20.590299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"70'139","reported_seq":1391,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-08-19T14:40:09.799065+0000","last_change":"2024-08-19T14:38:46.389978+0000","last_active":"2024-08-19T14:40:09.799065+0000","last_peered":"2024-08-19T14:40:09.799065+0000","last_clean":"2024-08-19T14:40:09.799065+0000","last_became_active":"2024-08-19T14:18:31.901556+0000","last_became_peered":"2024-08-19T14:18:31.901556+0000","last_unstale":"2024-08-19T14:40:09.799065+0000","last_undegraded":"2024-08-19T14:40:09.799065+0000","last_fullsized":"2024-08-19T14:40:09.799065+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:38:46.389862+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:38:46.389862+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,0],"acting":[8,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.14","version":"70'139","reported_seq":1421,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-08-19T14:40:09.223850+0000","last_change":"2024-08-19T14:40:03.932359+0000","last_active":"2024-08-19T14:40:09.223850+0000","last_peered":"2024-08-19T14:40:09.223850+0000","last_clean":"2024-08-19T14:40:09.223850+0000","last_became_active":"2024-08-19T14:35:12.529792+0000","last_became_peered":"2024-08-19T14:35:12.529792+0000","last_unstale":"2024-08-19T14:40:09.223850+0000","last_undegraded":"2024-08-19T14:40:09.223850+0000","last_fullsized":"2024-08-19T14:40:09.223850+0000","mapping_epoch":635,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":636,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:40:03.932271+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.932271+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:23.358249+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,9],"acting":[7,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.15","version":"70'139","reported_seq":1273,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-08-19T14:40:09.224146+0000","last_change":"2024-08-19T14:40:07.881907+0000","last_active":"2024-08-19T14:40:09.224146+0000","last_peered":"2024-08-19T14:40:09.224146+0000","last_clean":"2024-08-19T14:40:09.224146+0000","last_became_active":"2024-08-19T14:18:31.912496+0000","last_became_peered":"2024-08-19T14:18:31.912496+0000","last_unstale":"2024-08-19T14:40:09.224146+0000","last_undegraded":"2024-08-19T14:40:09.224146+0000","last_fullsized":"2024-08-19T14:40:09.224146+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:40:07.881806+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:20:09.180923+0000","last_clean_scrub_stamp":"2024-08-19T14:40:07.881806+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:32.759701+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"70'139","reported_seq":1256,"reported_epoch":720,"state":"active+clean","last_fresh":"2024-08-19T14:40:06.208160+0000","last_change":"2024-08-19T14:40:00.753735+0000","last_active":"2024-08-19T14:40:06.208160+0000","last_peered":"2024-08-19T14:40:06.208160+0000","last_clean":"2024-08-19T14:40:06.208160+0000","last_became_active":"2024-08-19T14:18:31.901229+0000","last_became_peered":"2024-08-19T14:18:31.901229+0000","last_unstale":"2024-08-19T14:40:06.208160+0000","last_undegraded":"2024-08-19T14:40:06.208160+0000","last_fullsized":"2024-08-19T14:40:06.208160+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:40:00.753636+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:40:00.753636+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:07.574708+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1b","version":"70'139","reported_seq":1639,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-08-19T14:40:09.798691+0000","last_change":"2024-08-19T14:39:32.443757+0000","last_active":"2024-08-19T14:40:09.798691+0000","last_peered":"2024-08-19T14:40:09.798691+0000","last_clean":"2024-08-19T14:40:09.798691+0000","last_became_active":"2024-08-19T14:34:15.718427+0000","last_became_peered":"2024-08-19T14:34:15.718427+0000","last_unstale":"2024-08-19T14:40:09.798691+0000","last_undegraded":"2024-08-19T14:40:09.798691+0000","last_fullsized":"2024-08-19T14:40:09.798691+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:39:32.443656+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:37:59.510697+0000","last_clean_scrub_stamp":"2024-08-19T14:39:32.443656+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:40:40.877428+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.18","version":"70'139","reported_seq":1201,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-08-19T14:40:09.223825+0000","last_change":"2024-08-19T14:39:59.937810+0000","last_active":"2024-08-19T14:40:09.223825+0000","last_peered":"2024-08-19T14:40:09.223825+0000","last_clean":"2024-08-19T14:40:09.223825+0000","last_became_active":"2024-08-19T14:35:11.528530+0000","last_became_peered":"2024-08-19T14:35:11.528530+0000","last_unstale":"2024-08-19T14:40:09.223825+0000","last_undegraded":"2024-08-19T14:40:09.223825+0000","last_fullsized":"2024-08-19T14:40:09.223825+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:39:59.937668+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:27:33.084161+0000","last_clean_scrub_stamp":"2024-08-19T14:39:59.937668+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:17.756316+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.19","version":"70'139","reported_seq":1618,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-08-19T14:40:09.297134+0000","last_change":"2024-08-19T14:39:32.723246+0000","last_active":"2024-08-19T14:40:09.297134+0000","last_peered":"2024-08-19T14:40:09.297134+0000","last_clean":"2024-08-19T14:40:09.297134+0000","last_became_active":"2024-08-19T14:18:31.902547+0000","last_became_peered":"2024-08-19T14:18:31.902547+0000","last_unstale":"2024-08-19T14:40:09.297134+0000","last_undegraded":"2024-08-19T14:40:09.297134+0000","last_fullsized":"2024-08-19T14:40:09.297134+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:39:32.722703+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:39:32.722703+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:40:40.564491+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1e","version":"70'139","reported_seq":637,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-08-19T14:40:11.226814+0000","last_change":"2024-08-19T14:33:17.778415+0000","last_active":"2024-08-19T14:40:11.226814+0000","last_peered":"2024-08-19T14:40:11.226814+0000","last_clean":"2024-08-19T14:40:11.226814+0000","last_became_active":"2024-08-19T14:30:47.909974+0000","last_became_peered":"2024-08-19T14:30:47.909974+0000","last_unstale":"2024-08-19T14:40:11.226814+0000","last_undegraded":"2024-08-19T14:40:11.226814+0000","last_fullsized":"2024-08-19T14:40:11.226814+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:33:17.778295+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:27:04.780939+0000","last_clean_scrub_stamp":"2024-08-19T14:33:17.778295+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1c","version":"70'139","reported_seq":1393,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-08-19T14:40:09.798913+0000","last_change":"2024-08-19T14:39:26.405807+0000","last_active":"2024-08-19T14:40:09.798913+0000","last_peered":"2024-08-19T14:40:09.798913+0000","last_clean":"2024-08-19T14:40:09.798913+0000","last_became_active":"2024-08-19T14:30:47.908866+0000","last_became_peered":"2024-08-19T14:30:47.908866+0000","last_unstale":"2024-08-19T14:40:09.798913+0000","last_undegraded":"2024-08-19T14:40:09.798913+0000","last_fullsized":"2024-08-19T14:40:09.798913+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:39:26.405699+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:39:26.405699+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:40:29.159585+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,11],"acting":[8,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1d","version":"70'139","reported_seq":1300,"reported_epoch":720,"state":"active+clean","last_fresh":"2024-08-19T14:40:06.699993+0000","last_change":"2024-08-19T14:39:57.775710+0000","last_active":"2024-08-19T14:40:06.699993+0000","last_peered":"2024-08-19T14:40:06.699993+0000","last_clean":"2024-08-19T14:40:06.699993+0000","last_became_active":"2024-08-19T14:35:11.540488+0000","last_became_peered":"2024-08-19T14:35:11.540488+0000","last_unstale":"2024-08-19T14:40:06.699993+0000","last_undegraded":"2024-08-19T14:40:06.699993+0000","last_fullsized":"2024-08-19T14:40:06.699993+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:39:57.775631+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:39:57.775631+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:40:59.182147+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":35782656,"num_objects":546,"num_object_clones":0,"num_object_copies":1638,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":546,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":546,"num_write_kb":34944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":63963136,"data_stored":63963136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":546,"ondisk_log_size":546,"up":48,"acting":48,"num_store_stats":10},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":856608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187802,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":883876,"kb_used_data":8600,"kb_used_omap":1,"kb_used_meta":875262,"kb_avail":92861276,"statfs":{"total":95995035648,"available":95089946624,"internally_reserved":0,"allocated":8806400,"data_stored":5624803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":896268740},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":2314987372669,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":74,"seq":317827580259,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":856416,"kb_used_data":8908,"kb_used_omap":1,"kb_used_meta":847486,"kb_avail":92888736,"statfs":{"total":95995035648,"available":95118065664,"internally_reserved":0,"allocated":9121792,"data_stored":5944110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":867826113},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874602,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":775576,"kb_used_data":8588,"kb_used_omap":1,"kb_used_meta":766974,"kb_avail":92969576,"statfs":{"total":95995035648,"available":95200845824,"internally_reserved":0,"allocated":8794112,"data_stored":5616430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":785381829},"hb_peers":[0,1,2,3,4,5,6,7,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":57,"seq":244813136237,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":960432,"kb_used_data":8800,"kb_used_omap":1,"kb_used_meta":951614,"kb_avail":92784720,"statfs":{"total":95995035648,"available":95011553280,"internally_reserved":0,"allocated":9011200,"data_stored":5826051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":974453194},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":206158430576,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692212,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":946540,"kb_used_data":8800,"kb_used_omap":1,"kb_used_meta":937726,"kb_avail":92798612,"statfs":{"total":95995035648,"available":95025778688,"internally_reserved":0,"allocated":9011200,"data_stored":5826051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":960231877},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986552,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1004640,"kb_used_data":8588,"kb_used_omap":1,"kb_used_meta":996030,"kb_avail":92740512,"statfs":{"total":95995035648,"available":94966284288,"internally_reserved":0,"allocated":8794112,"data_stored":5616430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1019935173},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150079,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1041060,"kb_used_data":8920,"kb_used_omap":1,"kb_used_meta":1032126,"kb_avail":92704092,"statfs":{"total":95995035648,"available":94928990208,"internally_reserved":0,"allocated":9134080,"data_stored":5952483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1056897479},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":634,"seq":2723009265725,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":293544,"kb_used_data":8788,"kb_used_omap":4,"kb_used_meta":284731,"kb_avail":93451608,"statfs":{"total":95995035648,"available":95694446592,"internally_reserved":0,"allocated":8998912,"data_stored":5817678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4407,"internal_metadata":291565257},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575239,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1130968,"kb_used_data":8588,"kb_used_omap":1,"kb_used_meta":1122366,"kb_avail":92614184,"statfs":{"total":95995035648,"available":94836924416,"internally_reserved":0,"allocated":8794112,"data_stored":5616430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1149303237},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738764,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":896676,"kb_used_data":8920,"kb_used_omap":1,"kb_used_meta":887742,"kb_avail":92848476,"statfs":{"total":95995035648,"available":95076839424,"internally_reserved":0,"allocated":9134080,"data_stored":5952483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":909048261},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":11599872,"data_stored":11599872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":52363264,"data_stored":52363264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T14:40:14.093 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-08-19T14:40:14.093 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-19T14:40:14.094 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-08-19T14:40:14.094 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-19T14:40:14.180 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:40:14.203 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:14.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:14.204 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:40:14.286 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 3 15 9755 9740 202.9 155.25 0.0029234 0.00492116 2024-08-19T14:40:14.305 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:40:14.374 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:14.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:14.375 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:40:14.476 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:40:14.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:14.503 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:14.503 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:40:14.604 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:40:14.622 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-08-19T14:40:14.626 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:14.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:14.627 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:40:14.728 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:40:14.749 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:14.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:14.750 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:40:14.819 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:40:14.819 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:40:14.850 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:40:14.865 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:40:14.866 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:40:14.900 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:40:14.900 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:40:14.900 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:14.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:14.901 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:40:14.924 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:40:14.924 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:40:14.931 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:40:14.944 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:40:14.944 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:40:14.946 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:40:14.970 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:40:14.971 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:40:14.990 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:40:14.990 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:40:15.003 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:40:15.006 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:40:15.007 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:40:15.023 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:40:15.024 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:40:15.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:15.025 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:15.025 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:40:15.040 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:40:15.041 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:40:15.058 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:40:15.058 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:40:15.126 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:40:15.150 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:15.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:15.151 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:40:15.252 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:40:15.286 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 4 16 11899 11883 185.657 133.938 0.00238203 0.0053799 2024-08-19T14:40:15.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:15.288 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:15.288 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:40:15.297 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:40:15.297 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:40:15.389 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:40:15.415 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:15.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:15.416 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:40:15.517 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:40:15.542 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:15.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:15.544 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:40:15.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:15 smithi031 ceph-mon[16336]: pgmap v1732: 47 pgs: 47 active+clean; 115 MiB data, 8.7 GiB used, 885 GiB / 894 GiB avail; 14 MiB/s wr, 227 op/s 2024-08-19T14:40:15.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:15 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3289090797' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:40:15.645 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:40:15.671 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:15.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:15.672 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:40:15.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:15 smithi032 ceph-mon[20004]: pgmap v1732: 47 pgs: 47 active+clean; 115 MiB data, 8.7 GiB used, 885 GiB / 894 GiB avail; 14 MiB/s wr, 227 op/s 2024-08-19T14:40:15.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:15 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3289090797' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:40:15.772 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:40:15.793 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:15.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:15.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:40:15.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:15 smithi045 ceph-mon[20143]: pgmap v1732: 47 pgs: 47 active+clean; 115 MiB data, 8.7 GiB used, 885 GiB / 894 GiB avail; 14 MiB/s wr, 227 op/s 2024-08-19T14:40:15.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:15 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3289090797' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:40:15.896 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:40:15.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:15.934 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:15.934 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:40:16.035 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:40:16.061 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:16.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:16.062 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:40:16.163 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:40:16.187 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:16.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:16.188 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:40:16.287 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 5 15 14888 14873 185.897 186.875 0.00370604 0.00537306 2024-08-19T14:40:16.289 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:40:16.336 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:16.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:16.337 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:40:16.438 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:40:16.463 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:16.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:16.464 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:40:16.565 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:40:16.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:16.611 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:16.611 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:40:16.655 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:40:16.656 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-08-19T14:40:16.712 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:40:16.750 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:16.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:16.751 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:40:16.753 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:40:16.753 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 20, 2024-08-19T14:40:16.753 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 600, 2024-08-19T14:40:16.753 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [ 2024-08-19T14:40:16.753 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.753 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:120799 6.6 6:6d4cbd5c:::benchmark_data_smithi045_97937_object13526:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e552)", 2024-08-19T14:40:16.753 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:30:36.258857+0000", 2024-08-19T14:40:16.753 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 580.48965643999998, 2024-08-19T14:40:16.754 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2809968280000001, 2024-08-19T14:40:16.754 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:40:16.754 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:40:16.754 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:40:16.754 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:40:16.754 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:40:16.754 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:40:16.754 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 120799 2024-08-19T14:40:16.754 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.754 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:40:16.754 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.754 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:40:16.754 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:36.258857+0000", 2024-08-19T14:40:16.755 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:40:16.755 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.755 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.755 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:40:16.755 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:36.258857+0000", 2024-08-19T14:40:16.755 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:40:16.755 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.755 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.755 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:40:16.755 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:36.258857+0000", 2024-08-19T14:40:16.755 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.01e-07 2024-08-19T14:40:16.755 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.755 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.755 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:40:16.756 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:36.258863+0000", 2024-08-19T14:40:16.756 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.1419999999999998e-06 2024-08-19T14:40:16.756 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.756 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.756 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:40:16.756 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:36.258870+0000", 2024-08-19T14:40:16.756 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.4699999999999999e-06 2024-08-19T14:40:16.756 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.756 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.756 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:40:16.756 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:36.258875+0000", 2024-08-19T14:40:16.756 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.3580000000000002e-06 2024-08-19T14:40:16.756 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.757 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.757 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:40:16.757 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:36.261800+0000", 2024-08-19T14:40:16.757 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.002925422 2024-08-19T14:40:16.757 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.757 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.757 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for missing object", 2024-08-19T14:40:16.757 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:36.261888+0000", 2024-08-19T14:40:16.757 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.7714999999999996e-05 2024-08-19T14:40:16.757 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.757 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.757 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:40:16.757 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:40.539815+0000", 2024-08-19T14:40:16.758 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2779266820000004 2024-08-19T14:40:16.758 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.758 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.758 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:40:16.758 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:30:40.539853+0000", 2024-08-19T14:40:16.758 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.8637999999999998e-05 2024-08-19T14:40:16.758 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:40:16.758 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:40:16.758 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:40:16.758 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.758 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.758 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:137589 6.6 6:61484f3f:::benchmark_data_smithi045_97937_object30316:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e569)", 2024-08-19T14:40:16.758 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:31:13.695526+0000", 2024-08-19T14:40:16.758 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 543.05298700100002, 2024-08-19T14:40:16.759 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.8720014850000002, 2024-08-19T14:40:16.759 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:40:16.759 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:40:16.759 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:40:16.759 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:40:16.759 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:40:16.759 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:40:16.759 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 137589 2024-08-19T14:40:16.759 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.759 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:40:16.759 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.759 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:40:16.759 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:31:13.695526+0000", 2024-08-19T14:40:16.760 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:40:16.760 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.760 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.760 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:40:16.760 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:31:13.695526+0000", 2024-08-19T14:40:16.760 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:40:16.760 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.760 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.760 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:40:16.760 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:31:13.695526+0000", 2024-08-19T14:40:16.760 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.6100000000000001e-07 2024-08-19T14:40:16.760 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.760 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.760 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:40:16.760 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:31:13.695530+0000", 2024-08-19T14:40:16.761 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.0979999999999998e-06 2024-08-19T14:40:16.761 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.761 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.761 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:40:16.761 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:31:13.695537+0000", 2024-08-19T14:40:16.761 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.7109999999999995e-06 2024-08-19T14:40:16.761 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.761 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.761 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:40:16.762 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:31:13.695543+0000", 2024-08-19T14:40:16.762 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.5500000000000002e-06 2024-08-19T14:40:16.762 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.762 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.762 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:40:16.762 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:31:13.695732+0000", 2024-08-19T14:40:16.762 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000188801 2024-08-19T14:40:16.762 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.762 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.762 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for missing object", 2024-08-19T14:40:16.762 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:31:13.695811+0000", 2024-08-19T14:40:16.762 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.9010000000000004e-05 2024-08-19T14:40:16.762 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.763 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.763 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:40:16.763 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:31:16.567509+0000", 2024-08-19T14:40:16.763 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.8716988030000001 2024-08-19T14:40:16.763 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.763 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.763 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:40:16.763 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:31:16.567527+0000", 2024-08-19T14:40:16.763 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.8051e-05 2024-08-19T14:40:16.763 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:40:16.763 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:40:16.763 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:40:16.763 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.763 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.764 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:168001 6.6 6:607c6180:::benchmark_data_smithi045_97937_object60728:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e576)", 2024-08-19T14:40:16.764 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:32:03.729180+0000", 2024-08-19T14:40:16.764 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 493.01933256400002, 2024-08-19T14:40:16.764 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.1240902739999998, 2024-08-19T14:40:16.764 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:40:16.764 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:40:16.764 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:40:16.764 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:40:16.764 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:40:16.764 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:40:16.764 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 168001 2024-08-19T14:40:16.764 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.764 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:40:16.764 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.765 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:40:16.765 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:03.729180+0000", 2024-08-19T14:40:16.765 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:40:16.765 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.765 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.765 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:40:16.765 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:03.729180+0000", 2024-08-19T14:40:16.765 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:40:16.765 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.765 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.765 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:40:16.765 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:03.729181+0000", 2024-08-19T14:40:16.765 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.1199999999999999e-07 2024-08-19T14:40:16.765 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.766 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.766 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:40:16.766 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:03.729182+0000", 2024-08-19T14:40:16.766 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.319e-06 2024-08-19T14:40:16.766 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.766 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.766 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:40:16.766 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:03.729185+0000", 2024-08-19T14:40:16.766 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.0180000000000002e-06 2024-08-19T14:40:16.766 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.766 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.766 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:40:16.766 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:03.729188+0000", 2024-08-19T14:40:16.767 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.5859999999999999e-06 2024-08-19T14:40:16.767 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.767 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.767 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:40:16.767 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:03.734604+0000", 2024-08-19T14:40:16.767 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0054159550000000001 2024-08-19T14:40:16.767 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.767 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.767 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for missing object", 2024-08-19T14:40:16.767 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:03.743722+0000", 2024-08-19T14:40:16.767 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0091187130000000005 2024-08-19T14:40:16.767 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.767 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.767 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:40:16.768 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:05.853261+0000", 2024-08-19T14:40:16.768 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.1095391330000002 2024-08-19T14:40:16.768 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.768 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.768 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:40:16.768 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:05.853271+0000", 2024-08-19T14:40:16.768 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.3379999999999994e-06 2024-08-19T14:40:16.768 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:40:16.768 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:40:16.769 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:40:16.769 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.769 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.769 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:135520 7.c 7:35037f76:::benchmark_data_smithi045_115842_object17360:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e647)", 2024-08-19T14:40:16.769 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:35:47.323563+0000", 2024-08-19T14:40:16.769 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 269.42494998799998, 2024-08-19T14:40:16.769 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.8331218590000002, 2024-08-19T14:40:16.769 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:40:16.769 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:40:16.769 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:40:16.769 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:40:16.769 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:40:16.769 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:40:16.769 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 135520 2024-08-19T14:40:16.770 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.770 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:40:16.770 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.770 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:40:16.770 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:47.323563+0000", 2024-08-19T14:40:16.770 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:40:16.770 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.770 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.770 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:40:16.770 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:47.323563+0000", 2024-08-19T14:40:16.770 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:40:16.770 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.770 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.771 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:40:16.771 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:47.323563+0000", 2024-08-19T14:40:16.771 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.8500000000000002e-07 2024-08-19T14:40:16.771 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.771 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.771 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:40:16.771 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:47.323566+0000", 2024-08-19T14:40:16.771 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.7329999999999999e-06 2024-08-19T14:40:16.771 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.771 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.771 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:40:16.771 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:47.323573+0000", 2024-08-19T14:40:16.771 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.9829999999999999e-06 2024-08-19T14:40:16.771 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.772 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.772 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:40:16.772 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:47.323579+0000", 2024-08-19T14:40:16.772 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.4999999999999999e-06 2024-08-19T14:40:16.772 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.772 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.772 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:40:16.772 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:47.323888+0000", 2024-08-19T14:40:16.772 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00030948899999999998 2024-08-19T14:40:16.772 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.772 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.772 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for missing object", 2024-08-19T14:40:16.772 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:47.323969+0000", 2024-08-19T14:40:16.773 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.0506000000000005e-05 2024-08-19T14:40:16.773 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.773 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.773 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:40:16.773 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:51.156672+0000", 2024-08-19T14:40:16.773 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.8327035739999999 2024-08-19T14:40:16.773 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.773 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.773 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:40:16.773 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:51.156685+0000", 2024-08-19T14:40:16.773 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2689e-05 2024-08-19T14:40:16.773 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:40:16.773 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:40:16.773 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:40:16.774 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.774 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.774 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:135524 7.c 7:398fc3f7:::benchmark_data_smithi045_115842_object17364:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e647)", 2024-08-19T14:40:16.774 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:35:47.323615+0000", 2024-08-19T14:40:16.774 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 269.42489814800001, 2024-08-19T14:40:16.774 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.9784070759999999, 2024-08-19T14:40:16.774 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:40:16.774 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:40:16.774 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:40:16.774 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:40:16.774 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:40:16.775 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:40:16.775 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 135524 2024-08-19T14:40:16.775 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.775 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:40:16.775 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.775 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:40:16.775 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:47.323615+0000", 2024-08-19T14:40:16.775 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:40:16.775 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.775 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.775 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:40:16.775 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:47.323615+0000", 2024-08-19T14:40:16.775 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:40:16.776 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.776 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.776 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:40:16.776 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:47.323615+0000", 2024-08-19T14:40:16.776 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.1199999999999998e-07 2024-08-19T14:40:16.776 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.776 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.776 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:40:16.776 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:47.323618+0000", 2024-08-19T14:40:16.776 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.6730000000000001e-06 2024-08-19T14:40:16.776 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.776 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.776 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:40:16.776 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:47.323626+0000", 2024-08-19T14:40:16.777 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.0029999999999994e-06 2024-08-19T14:40:16.777 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.777 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.777 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:40:16.777 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:47.323632+0000", 2024-08-19T14:40:16.777 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.8799999999999996e-06 2024-08-19T14:40:16.777 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.777 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.777 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:40:16.777 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:47.323996+0000", 2024-08-19T14:40:16.777 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00036417099999999999 2024-08-19T14:40:16.777 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.777 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.777 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for missing object", 2024-08-19T14:40:16.778 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:47.324073+0000", 2024-08-19T14:40:16.778 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.7356999999999998e-05 2024-08-19T14:40:16.778 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.778 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.778 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:40:16.778 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:51.301999+0000", 2024-08-19T14:40:16.778 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.977925334 2024-08-19T14:40:16.778 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.778 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.778 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:40:16.778 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:51.302022+0000", 2024-08-19T14:40:16.778 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.3246000000000001e-05 2024-08-19T14:40:16.778 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:40:16.779 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:40:16.779 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:40:16.779 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.779 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.779 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:135528 7.3 7:cb1ca295:::benchmark_data_smithi045_115842_object17368:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e647)", 2024-08-19T14:40:16.779 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:35:47.323640+0000", 2024-08-19T14:40:16.779 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 269.42487262399999, 2024-08-19T14:40:16.779 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7852496709999999, 2024-08-19T14:40:16.779 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:40:16.779 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:40:16.779 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:40:16.779 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:40:16.780 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:40:16.780 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:40:16.780 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 135528 2024-08-19T14:40:16.780 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.780 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:40:16.780 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.780 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:40:16.780 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:47.323640+0000", 2024-08-19T14:40:16.780 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:40:16.780 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.780 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.780 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:40:16.781 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:47.323640+0000", 2024-08-19T14:40:16.781 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:40:16.781 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.781 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.781 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:40:16.781 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:47.323641+0000", 2024-08-19T14:40:16.781 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.15e-07 2024-08-19T14:40:16.781 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.781 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.781 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:40:16.781 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:47.323643+0000", 2024-08-19T14:40:16.781 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.5100000000000001e-06 2024-08-19T14:40:16.781 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.781 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.782 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:40:16.782 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:47.323651+0000", 2024-08-19T14:40:16.782 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.204e-06 2024-08-19T14:40:16.782 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.782 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.782 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:40:16.782 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:47.323656+0000", 2024-08-19T14:40:16.782 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0540000000000002e-06 2024-08-19T14:40:16.782 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.782 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.783 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:40:16.783 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:47.323930+0000", 2024-08-19T14:40:16.783 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00027444900000000001 2024-08-19T14:40:16.783 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.783 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.784 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for missing object", 2024-08-19T14:40:16.784 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:47.324005+0000", 2024-08-19T14:40:16.784 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.4727000000000005e-05 2024-08-19T14:40:16.784 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.784 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.784 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:40:16.784 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:51.108879+0000", 2024-08-19T14:40:16.784 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7848740510000001 2024-08-19T14:40:16.784 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.784 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.784 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:40:16.784 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:51.108890+0000", 2024-08-19T14:40:16.784 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1260999999999999e-05 2024-08-19T14:40:16.784 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:40:16.785 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:40:16.785 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:40:16.785 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.785 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.785 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:139379 7.2 7:47071399:::benchmark_data_smithi045_115842_object21219:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e648)", 2024-08-19T14:40:16.785 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:35:57.352771+0000", 2024-08-19T14:40:16.785 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 259.39574160299998, 2024-08-19T14:40:16.785 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.7060800060000001, 2024-08-19T14:40:16.785 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:40:16.785 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:40:16.785 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:40:16.785 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:40:16.785 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:40:16.785 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:40:16.786 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 139379 2024-08-19T14:40:16.786 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.786 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:40:16.786 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.786 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:40:16.786 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:57.352771+0000", 2024-08-19T14:40:16.786 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:40:16.786 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.786 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.786 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:40:16.786 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:57.352771+0000", 2024-08-19T14:40:16.786 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:40:16.786 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.787 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.787 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:40:16.787 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:57.352772+0000", 2024-08-19T14:40:16.787 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.8900000000000001e-07 2024-08-19T14:40:16.787 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.787 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.787 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:40:16.787 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:57.352774+0000", 2024-08-19T14:40:16.787 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.8929999999999999e-06 2024-08-19T14:40:16.787 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.787 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.787 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:40:16.787 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:57.352778+0000", 2024-08-19T14:40:16.787 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.6689999999999997e-06 2024-08-19T14:40:16.788 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.788 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.788 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:40:16.788 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:57.352782+0000", 2024-08-19T14:40:16.788 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.8970000000000001e-06 2024-08-19T14:40:16.788 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.788 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.788 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:40:16.788 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:57.373078+0000", 2024-08-19T14:40:16.788 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.020295853999999999 2024-08-19T14:40:16.788 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.788 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.788 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for missing object", 2024-08-19T14:40:16.788 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:57.373151+0000", 2024-08-19T14:40:16.789 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.2750999999999995e-05 2024-08-19T14:40:16.789 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.789 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.789 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:40:16.789 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:00.058840+0000", 2024-08-19T14:40:16.789 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.685688807 2024-08-19T14:40:16.789 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.789 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.789 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:40:16.789 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:00.058851+0000", 2024-08-19T14:40:16.789 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1846e-05 2024-08-19T14:40:16.789 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:40:16.789 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:40:16.789 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:40:16.790 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.790 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.790 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:139390 7.c 7:3982d552:::benchmark_data_smithi045_115842_object21230:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e648)", 2024-08-19T14:40:16.790 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:35:57.352808+0000", 2024-08-19T14:40:16.791 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 259.39570542299998, 2024-08-19T14:40:16.791 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.7041374769999997, 2024-08-19T14:40:16.791 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:40:16.791 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:40:16.791 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:40:16.791 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:40:16.791 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:40:16.791 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:40:16.791 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 139390 2024-08-19T14:40:16.791 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.791 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:40:16.792 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.792 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:40:16.792 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:57.352808+0000", 2024-08-19T14:40:16.792 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:40:16.792 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.792 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.792 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:40:16.792 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:57.352808+0000", 2024-08-19T14:40:16.792 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:40:16.792 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.793 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.793 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:40:16.793 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:57.352808+0000", 2024-08-19T14:40:16.793 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.8599999999999999e-07 2024-08-19T14:40:16.793 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.793 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.793 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:40:16.793 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:57.352810+0000", 2024-08-19T14:40:16.793 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.9410000000000001e-06 2024-08-19T14:40:16.793 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.794 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.794 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:40:16.794 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:57.352815+0000", 2024-08-19T14:40:16.794 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7720000000000002e-06 2024-08-19T14:40:16.794 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.794 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.794 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:40:16.794 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:57.352818+0000", 2024-08-19T14:40:16.794 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.9009999999999996e-06 2024-08-19T14:40:16.794 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.794 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.795 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:40:16.795 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:57.356159+0000", 2024-08-19T14:40:16.795 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0033409580000000002 2024-08-19T14:40:16.795 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.795 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.795 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for missing object", 2024-08-19T14:40:16.795 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:35:57.356239+0000", 2024-08-19T14:40:16.795 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.9319000000000004e-05 2024-08-19T14:40:16.795 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.795 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.795 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:40:16.795 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:00.052363+0000", 2024-08-19T14:40:16.796 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.696124121 2024-08-19T14:40:16.796 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.796 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.796 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:40:16.796 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:00.056945+0000", 2024-08-19T14:40:16.796 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0045821789999999996 2024-08-19T14:40:16.796 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:40:16.796 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:40:16.796 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:40:16.796 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.796 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.797 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:178361 7.3 7:cda41295:::benchmark_data_smithi045_115842_object60201:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e676)", 2024-08-19T14:40:16.797 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:37:35.698169+0000", 2024-08-19T14:40:16.797 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 161.050344295, 2024-08-19T14:40:16.797 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4301732630000004, 2024-08-19T14:40:16.797 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:40:16.797 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:40:16.797 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:40:16.797 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:40:16.797 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:40:16.797 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:40:16.797 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 178361 2024-08-19T14:40:16.797 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.798 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:40:16.798 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.798 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:40:16.798 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:35.698169+0000", 2024-08-19T14:40:16.798 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:40:16.798 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.798 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.798 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:40:16.798 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:35.698169+0000", 2024-08-19T14:40:16.798 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:40:16.798 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.799 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.799 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:40:16.799 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:35.698169+0000", 2024-08-19T14:40:16.799 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.1800000000000002e-07 2024-08-19T14:40:16.799 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.799 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.799 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:40:16.799 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:35.698172+0000", 2024-08-19T14:40:16.799 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.6910000000000002e-06 2024-08-19T14:40:16.799 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.800 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.800 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:40:16.800 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:35.698177+0000", 2024-08-19T14:40:16.800 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.5029999999999997e-06 2024-08-19T14:40:16.800 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.800 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.800 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:40:16.800 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:35.698182+0000", 2024-08-19T14:40:16.800 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4669999999999998e-06 2024-08-19T14:40:16.800 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.801 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.801 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:40:16.801 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:35.708410+0000", 2024-08-19T14:40:16.801 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.010228568 2024-08-19T14:40:16.801 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.801 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.801 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for missing object", 2024-08-19T14:40:16.801 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:35.708502+0000", 2024-08-19T14:40:16.801 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.1527000000000006e-05 2024-08-19T14:40:16.801 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.801 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.802 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:40:16.802 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:40.128321+0000", 2024-08-19T14:40:16.802 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4198188739999997 2024-08-19T14:40:16.802 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.802 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.802 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:40:16.802 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:40.128342+0000", 2024-08-19T14:40:16.802 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.1314999999999998e-05 2024-08-19T14:40:16.802 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:40:16.802 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:40:16.802 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:40:16.803 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.803 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.803 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:178375 7.3 7:cb0f4ae1:::benchmark_data_smithi045_115842_object60215:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e676)", 2024-08-19T14:40:16.803 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:37:35.698187+0000", 2024-08-19T14:40:16.803 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 161.05032573299999, 2024-08-19T14:40:16.803 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4423218430000002, 2024-08-19T14:40:16.803 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:40:16.803 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:40:16.803 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:40:16.804 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:40:16.804 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:40:16.804 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:40:16.804 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 178375 2024-08-19T14:40:16.804 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.804 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:40:16.804 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.804 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:40:16.804 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:35.698187+0000", 2024-08-19T14:40:16.804 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:40:16.804 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.805 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.805 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:40:16.805 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:35.698187+0000", 2024-08-19T14:40:16.805 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:40:16.805 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.805 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.805 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:40:16.805 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:35.698188+0000", 2024-08-19T14:40:16.805 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.7399999999999999e-07 2024-08-19T14:40:16.805 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.805 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.806 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:40:16.806 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:35.698189+0000", 2024-08-19T14:40:16.806 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.933e-06 2024-08-19T14:40:16.806 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.806 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.806 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:40:16.806 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:35.698195+0000", 2024-08-19T14:40:16.806 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1569999999999999e-06 2024-08-19T14:40:16.806 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.806 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.806 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:40:16.807 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:35.698199+0000", 2024-08-19T14:40:16.807 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.3900000000000003e-06 2024-08-19T14:40:16.807 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.807 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.807 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:40:16.807 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:35.708531+0000", 2024-08-19T14:40:16.807 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.010332221000000001 2024-08-19T14:40:16.807 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.807 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.807 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for missing object", 2024-08-19T14:40:16.807 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:35.708601+0000", 2024-08-19T14:40:16.808 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.9558e-05 2024-08-19T14:40:16.808 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.808 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.808 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:40:16.808 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:40.140497+0000", 2024-08-19T14:40:16.808 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4318961530000003 2024-08-19T14:40:16.808 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.808 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.808 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:40:16.808 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:40.140509+0000", 2024-08-19T14:40:16.809 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2157000000000001e-05 2024-08-19T14:40:16.809 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:40:16.809 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:40:16.809 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:40:16.809 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.809 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.809 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:191412 7.2 7:469e5aca:::benchmark_data_smithi045_115842_object73252:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e686)", 2024-08-19T14:40:16.809 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:38:05.333032+0000", 2024-08-19T14:40:16.809 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 131.415480527, 2024-08-19T14:40:16.810 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.173558978, 2024-08-19T14:40:16.810 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:40:16.810 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:40:16.810 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:40:16.810 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:40:16.810 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:40:16.810 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:40:16.810 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 191412 2024-08-19T14:40:16.810 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.810 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:40:16.811 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.811 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:40:16.811 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:05.333032+0000", 2024-08-19T14:40:16.811 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:40:16.811 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.811 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.811 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:40:16.811 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:05.333032+0000", 2024-08-19T14:40:16.811 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:40:16.811 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.811 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.812 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:40:16.812 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:05.333033+0000", 2024-08-19T14:40:16.812 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.9900000000000002e-07 2024-08-19T14:40:16.812 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.812 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.812 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:40:16.812 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:05.333035+0000", 2024-08-19T14:40:16.812 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.0990000000000001e-06 2024-08-19T14:40:16.812 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.812 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.812 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:40:16.813 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:05.333040+0000", 2024-08-19T14:40:16.813 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.2310000000000004e-06 2024-08-19T14:40:16.813 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.813 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.813 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:40:16.813 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:05.333045+0000", 2024-08-19T14:40:16.813 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4100000000000001e-06 2024-08-19T14:40:16.813 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.813 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.813 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:40:16.813 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:05.344687+0000", 2024-08-19T14:40:16.814 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.011642593 2024-08-19T14:40:16.814 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.814 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.814 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for missing object", 2024-08-19T14:40:16.814 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:05.344760+0000", 2024-08-19T14:40:16.814 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.2589999999999994e-05 2024-08-19T14:40:16.814 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.814 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.814 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:40:16.814 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:12.506579+0000", 2024-08-19T14:40:16.814 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.1618195030000003 2024-08-19T14:40:16.815 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.815 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.815 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:40:16.815 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:12.506591+0000", 2024-08-19T14:40:16.815 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2252999999999999e-05 2024-08-19T14:40:16.815 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:40:16.815 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:40:16.815 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:40:16.815 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.815 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.815 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:191415 7.c 7:3e0db92c:::benchmark_data_smithi045_115842_object73255:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e686)", 2024-08-19T14:40:16.816 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:38:05.333050+0000", 2024-08-19T14:40:16.816 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 131.41546324999999, 2024-08-19T14:40:16.816 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.1806932559999996, 2024-08-19T14:40:16.816 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:40:16.816 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:40:16.816 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:40:16.817 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:40:16.817 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:40:16.817 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:40:16.817 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 191415 2024-08-19T14:40:16.817 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.818 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:40:16.818 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.818 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:40:16.818 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:05.333050+0000", 2024-08-19T14:40:16.818 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:40:16.818 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.818 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.818 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:40:16.818 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:05.333050+0000", 2024-08-19T14:40:16.818 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:40:16.818 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.819 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.819 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:40:16.819 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:05.333050+0000", 2024-08-19T14:40:16.819 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.9999999999999999e-07 2024-08-19T14:40:16.819 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.819 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.819 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:40:16.819 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:05.333052+0000", 2024-08-19T14:40:16.819 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.902e-06 2024-08-19T14:40:16.819 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.819 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.819 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:40:16.820 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:05.333057+0000", 2024-08-19T14:40:16.820 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1370000000000001e-06 2024-08-19T14:40:16.820 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.820 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.820 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:40:16.820 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:05.333062+0000", 2024-08-19T14:40:16.820 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.6280000000000004e-06 2024-08-19T14:40:16.820 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.820 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.820 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:40:16.820 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:05.333145+0000", 2024-08-19T14:40:16.820 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.3194000000000003e-05 2024-08-19T14:40:16.820 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.820 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.821 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for missing object", 2024-08-19T14:40:16.821 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:05.333242+0000", 2024-08-19T14:40:16.821 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.6707999999999996e-05 2024-08-19T14:40:16.821 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.821 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.821 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:40:16.821 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:12.513727+0000", 2024-08-19T14:40:16.821 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.1804853890000002 2024-08-19T14:40:16.821 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.821 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.821 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:40:16.821 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:12.513743+0000", 2024-08-19T14:40:16.821 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5998000000000002e-05 2024-08-19T14:40:16.822 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:40:16.822 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:40:16.822 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:40:16.822 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.822 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.822 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:200330 7.c 7:3964c84b:::benchmark_data_smithi045_115842_object82170:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e690)", 2024-08-19T14:40:16.822 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:38:24.742215+0000", 2024-08-19T14:40:16.822 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 112.00629799399999, 2024-08-19T14:40:16.822 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.544114505, 2024-08-19T14:40:16.822 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:40:16.822 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:40:16.822 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:40:16.822 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:40:16.822 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:40:16.823 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:40:16.823 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 200330 2024-08-19T14:40:16.823 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.823 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:40:16.823 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.823 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:40:16.823 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:24.742215+0000", 2024-08-19T14:40:16.823 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:40:16.823 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.823 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.823 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:40:16.823 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:24.742215+0000", 2024-08-19T14:40:16.823 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:40:16.823 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.824 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.824 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:40:16.824 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:24.742215+0000", 2024-08-19T14:40:16.824 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.9999999999999999e-07 2024-08-19T14:40:16.824 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.824 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.824 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:40:16.824 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:24.742217+0000", 2024-08-19T14:40:16.824 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.0940000000000002e-06 2024-08-19T14:40:16.824 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.824 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.824 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:40:16.824 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:24.742223+0000", 2024-08-19T14:40:16.824 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.5149999999999997e-06 2024-08-19T14:40:16.824 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.825 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.825 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:40:16.825 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:24.742227+0000", 2024-08-19T14:40:16.825 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.5120000000000002e-06 2024-08-19T14:40:16.825 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.825 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.825 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:40:16.825 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:24.745674+0000", 2024-08-19T14:40:16.825 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0034464389999999999 2024-08-19T14:40:16.825 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.825 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.825 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for missing object", 2024-08-19T14:40:16.825 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:24.745757+0000", 2024-08-19T14:40:16.825 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.3162000000000001e-05 2024-08-19T14:40:16.826 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.826 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.826 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:40:16.826 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:28.286317+0000", 2024-08-19T14:40:16.826 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.5405595189999999 2024-08-19T14:40:16.826 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.826 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.826 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:40:16.826 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:28.286330+0000", 2024-08-19T14:40:16.826 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2964e-05 2024-08-19T14:40:16.826 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:40:16.826 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:40:16.826 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:40:16.827 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.827 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.827 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:201906 7.3 7:ca923fba:::benchmark_data_smithi045_115842_object83746:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e691)", 2024-08-19T14:40:16.827 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:38:28.215922+0000", 2024-08-19T14:40:16.827 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 108.53259135099999, 2024-08-19T14:40:16.827 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.587347399, 2024-08-19T14:40:16.827 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:40:16.827 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:40:16.827 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:40:16.827 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:40:16.827 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:40:16.827 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:40:16.827 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 201906 2024-08-19T14:40:16.828 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.828 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:40:16.828 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.828 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:40:16.828 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:28.215922+0000", 2024-08-19T14:40:16.828 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:40:16.828 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.828 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.828 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:40:16.828 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:28.215922+0000", 2024-08-19T14:40:16.828 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:40:16.828 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.828 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.828 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:40:16.829 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:28.215922+0000", 2024-08-19T14:40:16.829 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.4200000000000002e-07 2024-08-19T14:40:16.829 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.829 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.829 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:40:16.829 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:28.215925+0000", 2024-08-19T14:40:16.829 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.6249999999999999e-06 2024-08-19T14:40:16.829 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.829 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.829 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:40:16.829 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:28.215930+0000", 2024-08-19T14:40:16.829 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.2499999999999997e-06 2024-08-19T14:40:16.829 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.830 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.830 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:40:16.830 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:28.215935+0000", 2024-08-19T14:40:16.830 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0690000000000001e-06 2024-08-19T14:40:16.830 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.830 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.830 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:40:16.830 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:28.244218+0000", 2024-08-19T14:40:16.830 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.028282808999999999 2024-08-19T14:40:16.830 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.830 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.830 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for missing object", 2024-08-19T14:40:16.830 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:28.244310+0000", 2024-08-19T14:40:16.831 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.2227999999999999e-05 2024-08-19T14:40:16.831 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.831 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.831 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:40:16.831 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:29.803256+0000", 2024-08-19T14:40:16.831 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5589463349999999 2024-08-19T14:40:16.831 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.831 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.831 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:40:16.831 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:29.803269+0000", 2024-08-19T14:40:16.831 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2741e-05 2024-08-19T14:40:16.831 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:40:16.831 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:40:16.831 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:40:16.831 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.832 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.832 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:201910 7.3 7:cffdc6e7:::benchmark_data_smithi045_115842_object83750:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e691)", 2024-08-19T14:40:16.832 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:38:28.215941+0000", 2024-08-19T14:40:16.832 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 108.532571721, 2024-08-19T14:40:16.832 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.60242383, 2024-08-19T14:40:16.832 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:40:16.832 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:40:16.832 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:40:16.832 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:40:16.832 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:40:16.832 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:40:16.832 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 201910 2024-08-19T14:40:16.832 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.833 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:40:16.833 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.833 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:40:16.834 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:28.215941+0000", 2024-08-19T14:40:16.834 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:40:16.834 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.834 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.834 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:40:16.834 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:28.215941+0000", 2024-08-19T14:40:16.834 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:40:16.834 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.834 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.834 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:40:16.834 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:28.215942+0000", 2024-08-19T14:40:16.835 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.1100000000000002e-07 2024-08-19T14:40:16.835 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.835 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.835 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:40:16.835 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:28.215944+0000", 2024-08-19T14:40:16.835 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.9700000000000002e-06 2024-08-19T14:40:16.835 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.835 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.835 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:40:16.835 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:28.215949+0000", 2024-08-19T14:40:16.835 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0969999999999996e-06 2024-08-19T14:40:16.835 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.836 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.836 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:40:16.836 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:28.215952+0000", 2024-08-19T14:40:16.836 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.8190000000000002e-06 2024-08-19T14:40:16.836 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.836 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.836 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:40:16.836 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:28.244340+0000", 2024-08-19T14:40:16.836 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.028388002999999998 2024-08-19T14:40:16.836 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.836 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.836 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for missing object", 2024-08-19T14:40:16.836 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:28.244410+0000", 2024-08-19T14:40:16.836 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.9695000000000003e-05 2024-08-19T14:40:16.836 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.837 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.837 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:40:16.837 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:29.818349+0000", 2024-08-19T14:40:16.837 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5739392670000001 2024-08-19T14:40:16.837 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.837 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.837 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:40:16.837 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:29.818365+0000", 2024-08-19T14:40:16.837 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5668e-05 2024-08-19T14:40:16.837 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:40:16.837 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:40:16.837 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:40:16.837 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.837 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.838 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:201912 7.3 7:c625da36:::benchmark_data_smithi045_115842_object83752:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e691)", 2024-08-19T14:40:16.838 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:38:28.215958+0000", 2024-08-19T14:40:16.838 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 108.532555062, 2024-08-19T14:40:16.838 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.607718583, 2024-08-19T14:40:16.838 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:40:16.838 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:40:16.838 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:40:16.838 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:40:16.838 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:40:16.838 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:40:16.838 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 201912 2024-08-19T14:40:16.838 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.838 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:40:16.839 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.839 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:40:16.839 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:28.215958+0000", 2024-08-19T14:40:16.839 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:40:16.839 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.839 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.839 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:40:16.839 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:28.215958+0000", 2024-08-19T14:40:16.839 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:40:16.839 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.839 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.839 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:40:16.839 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:28.215958+0000", 2024-08-19T14:40:16.840 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.6199999999999999e-07 2024-08-19T14:40:16.840 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.840 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.840 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:40:16.840 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:28.215964+0000", 2024-08-19T14:40:16.840 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.0569999999999998e-06 2024-08-19T14:40:16.840 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.840 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.840 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:40:16.840 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:28.215970+0000", 2024-08-19T14:40:16.840 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.2440000000000001e-06 2024-08-19T14:40:16.840 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.840 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.841 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:40:16.841 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:28.215974+0000", 2024-08-19T14:40:16.841 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.0999999999999997e-06 2024-08-19T14:40:16.841 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.841 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.841 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:40:16.841 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:28.244439+0000", 2024-08-19T14:40:16.841 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.028465013000000001 2024-08-19T14:40:16.841 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.841 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.841 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for missing object", 2024-08-19T14:40:16.841 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:28.244509+0000", 2024-08-19T14:40:16.841 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.0322999999999997e-05 2024-08-19T14:40:16.841 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.842 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.842 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:40:16.842 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:29.823638+0000", 2024-08-19T14:40:16.842 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.579128614 2024-08-19T14:40:16.842 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.842 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.842 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:40:16.842 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:29.823677+0000", 2024-08-19T14:40:16.842 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.8970000000000001e-05 2024-08-19T14:40:16.842 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:40:16.842 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:40:16.842 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:40:16.842 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.842 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.843 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:219824 7.3 7:c228a6f2:::benchmark_data_smithi045_115842_object101664:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e699)", 2024-08-19T14:40:16.843 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:39:05.329302+0000", 2024-08-19T14:40:16.843 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 71.419211138999998, 2024-08-19T14:40:16.843 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.7693942160000002, 2024-08-19T14:40:16.843 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:40:16.843 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:40:16.843 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:40:16.843 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:40:16.843 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:40:16.843 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:40:16.843 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 219824 2024-08-19T14:40:16.843 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.843 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:40:16.843 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.844 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:40:16.844 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:05.329302+0000", 2024-08-19T14:40:16.844 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:40:16.844 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.844 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.844 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:40:16.844 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:05.329302+0000", 2024-08-19T14:40:16.844 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:40:16.844 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.844 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.844 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:40:16.844 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:05.329302+0000", 2024-08-19T14:40:16.844 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.22e-07 2024-08-19T14:40:16.844 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.844 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.845 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:40:16.845 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:05.329304+0000", 2024-08-19T14:40:16.845 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.7910000000000001e-06 2024-08-19T14:40:16.845 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.845 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.845 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:40:16.845 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:05.329309+0000", 2024-08-19T14:40:16.845 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.057e-06 2024-08-19T14:40:16.845 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.845 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.845 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:40:16.845 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:05.329314+0000", 2024-08-19T14:40:16.845 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4920000000000004e-06 2024-08-19T14:40:16.845 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.846 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.846 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:40:16.846 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:05.329839+0000", 2024-08-19T14:40:16.846 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00052567900000000003 2024-08-19T14:40:16.846 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.846 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.846 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for missing object", 2024-08-19T14:40:16.846 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:05.329918+0000", 2024-08-19T14:40:16.846 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.8475999999999999e-05 2024-08-19T14:40:16.846 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.846 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.846 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:40:16.846 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:08.098684+0000", 2024-08-19T14:40:16.847 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.7687666489999998 2024-08-19T14:40:16.847 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.847 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.847 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:40:16.847 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:08.098696+0000", 2024-08-19T14:40:16.847 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1749999999999999e-05 2024-08-19T14:40:16.847 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:40:16.847 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:40:16.847 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:40:16.847 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.847 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.847 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:219833 7.2 7:4faf1754:::benchmark_data_smithi045_115842_object101673:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e699)", 2024-08-19T14:40:16.847 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:39:05.329376+0000", 2024-08-19T14:40:16.847 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 71.419137508999995, 2024-08-19T14:40:16.848 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.770562398, 2024-08-19T14:40:16.848 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:40:16.848 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:40:16.848 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:40:16.848 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:40:16.848 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:40:16.848 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:40:16.848 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 219833 2024-08-19T14:40:16.848 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.848 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:40:16.848 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.848 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:40:16.848 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:05.329376+0000", 2024-08-19T14:40:16.849 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:40:16.849 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.849 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.849 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:40:16.849 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:05.329376+0000", 2024-08-19T14:40:16.849 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:40:16.849 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.849 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.849 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:40:16.849 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:05.329376+0000", 2024-08-19T14:40:16.849 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.8999999999999998e-07 2024-08-19T14:40:16.849 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.849 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.849 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:40:16.850 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:05.329378+0000", 2024-08-19T14:40:16.850 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.9769999999999999e-06 2024-08-19T14:40:16.850 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.850 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.850 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:40:16.850 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:05.329383+0000", 2024-08-19T14:40:16.850 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.3809999999999998e-06 2024-08-19T14:40:16.850 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.850 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.850 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:40:16.850 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:05.329388+0000", 2024-08-19T14:40:16.850 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.5510000000000001e-06 2024-08-19T14:40:16.850 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.851 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.851 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:40:16.851 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:05.340772+0000", 2024-08-19T14:40:16.851 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.011384215 2024-08-19T14:40:16.851 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.851 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.851 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for missing object", 2024-08-19T14:40:16.851 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:05.340845+0000", 2024-08-19T14:40:16.851 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.2634999999999998e-05 2024-08-19T14:40:16.851 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.851 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.851 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:40:16.851 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:08.099925+0000", 2024-08-19T14:40:16.851 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.7590808249999998 2024-08-19T14:40:16.852 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.852 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.852 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:40:16.852 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:08.099938+0000", 2024-08-19T14:40:16.852 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2524000000000001e-05 2024-08-19T14:40:16.852 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:40:16.852 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:40:16.852 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:40:16.852 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.852 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.852 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:233358 7.3 7:ce05daa3:::benchmark_data_smithi045_115842_object115198:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e707)", 2024-08-19T14:40:16.852 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:39:31.017243+0000", 2024-08-19T14:40:16.852 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 45.731269789000002, 2024-08-19T14:40:16.852 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4164580099999999, 2024-08-19T14:40:16.853 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:40:16.853 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:40:16.853 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:40:16.853 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:40:16.853 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:40:16.853 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:40:16.853 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 233358 2024-08-19T14:40:16.853 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.853 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:40:16.853 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.853 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:40:16.853 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:31.017243+0000", 2024-08-19T14:40:16.853 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:40:16.853 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.854 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.854 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:40:16.854 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:31.017243+0000", 2024-08-19T14:40:16.854 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:40:16.854 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.854 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.854 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:40:16.854 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:31.017244+0000", 2024-08-19T14:40:16.854 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.2599999999999998e-07 2024-08-19T14:40:16.854 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.854 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.854 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:40:16.855 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:31.017245+0000", 2024-08-19T14:40:16.855 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.9400000000000001e-06 2024-08-19T14:40:16.855 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.855 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.855 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:40:16.855 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:31.017251+0000", 2024-08-19T14:40:16.855 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.3789999999999997e-06 2024-08-19T14:40:16.855 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.855 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.855 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:40:16.855 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:31.017255+0000", 2024-08-19T14:40:16.855 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4390000000000002e-06 2024-08-19T14:40:16.855 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.856 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.856 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:40:16.856 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:31.021095+0000", 2024-08-19T14:40:16.856 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0038393619999999998 2024-08-19T14:40:16.856 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.856 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.856 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for missing object", 2024-08-19T14:40:16.857 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:31.021319+0000", 2024-08-19T14:40:16.857 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000224026 2024-08-19T14:40:16.857 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.857 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.858 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:40:16.858 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:32.433689+0000", 2024-08-19T14:40:16.858 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.41236988 2024-08-19T14:40:16.858 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.858 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.858 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:40:16.858 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:32.433701+0000", 2024-08-19T14:40:16.858 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2658e-05 2024-08-19T14:40:16.858 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:40:16.858 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:40:16.858 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:40:16.858 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.858 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.858 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:233367 7.3 7:c7fe5a37:::benchmark_data_smithi045_115842_object115207:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e707)", 2024-08-19T14:40:16.859 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:39:31.017282+0000", 2024-08-19T14:40:16.859 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 45.731230740999997, 2024-08-19T14:40:16.859 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4183822610000001, 2024-08-19T14:40:16.859 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:40:16.859 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:40:16.859 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:40:16.859 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:40:16.859 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:40:16.859 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:40:16.859 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 233367 2024-08-19T14:40:16.859 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.859 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:40:16.859 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.859 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:40:16.860 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:31.017282+0000", 2024-08-19T14:40:16.860 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:40:16.860 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.860 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.860 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:40:16.860 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:31.017282+0000", 2024-08-19T14:40:16.860 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:40:16.860 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.860 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.860 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:40:16.860 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:31.017283+0000", 2024-08-19T14:40:16.860 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.3999999999999997e-07 2024-08-19T14:40:16.860 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.860 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.861 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:40:16.861 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:31.017284+0000", 2024-08-19T14:40:16.861 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.849e-06 2024-08-19T14:40:16.861 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.861 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.861 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:40:16.861 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:31.017290+0000", 2024-08-19T14:40:16.861 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.3630000000000002e-06 2024-08-19T14:40:16.861 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.861 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.861 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:40:16.861 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:31.017295+0000", 2024-08-19T14:40:16.861 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7829999999999998e-06 2024-08-19T14:40:16.861 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.862 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.862 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:40:16.862 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:31.021414+0000", 2024-08-19T14:40:16.862 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0041193289999999997 2024-08-19T14:40:16.862 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.862 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.862 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for missing object", 2024-08-19T14:40:16.862 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:31.021514+0000", 2024-08-19T14:40:16.862 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000100438 2024-08-19T14:40:16.862 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.862 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.863 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:40:16.863 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:32.435654+0000", 2024-08-19T14:40:16.863 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.414139619 2024-08-19T14:40:16.863 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:40:16.863 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:40:16.863 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:40:16.863 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:32.435665+0000", 2024-08-19T14:40:16.863 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0540000000000001e-05 2024-08-19T14:40:16.863 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:40:16.863 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:40:16.863 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:40:16.863 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:40:16.863 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:40:16.863 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:40:16.864 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:40:16.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:16.891 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:16.892 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:40:16.992 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:40:17.019 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:17.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:17.020 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:40:17.121 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:40:17.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:17.147 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:17.147 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:40:17.248 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:40:17.283 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:17.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:17.284 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:40:17.286 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 6 16 18227 18211 189.682 208.625 0.00237832 0.00526581 2024-08-19T14:40:17.385 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:40:17.412 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:17.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:17.413 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:40:17.513 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:40:17.549 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:17.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:17.550 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:40:17.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:17 smithi031 ceph-mon[16336]: pgmap v1733: 47 pgs: 47 active+clean; 512 MiB data, 9.5 GiB used, 885 GiB / 894 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-08-19T14:40:17.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:17 smithi031 ceph-mon[16336]: from='client.17403 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:40:17.652 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:40:17.676 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:17.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:17.677 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:40:17.727 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":1733,"stamp":"2024-08-19T14:40:16.281918+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":537199136,"num_objects":8190,"num_object_clones":0,"num_object_copies":24570,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8190,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8188,"num_write_kb":524032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":856608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12497,"ondisk_log_size":12497,"up":141,"acting":141,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":141,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":9931980,"kb_used_data":1238120,"kb_used_omap":18,"kb_used_meta":8693741,"kb_avail":927519540,"statfs":{"total":959950356480,"available":949780008960,"internally_reserved":0,"allocated":1267834880,"data_stored":1235974214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18760,"internal_metadata":8902391480},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":536608768,"num_objects":8188,"num_object_clones":0,"num_object_copies":24564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8188,"num_write_kb":524032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002186"},"pg_stats":[{"pgid":"8.9","version":"722'811","reported_seq":831,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-08-19T14:40:15.641426+0000","last_change":"2024-08-19T14:40:04.442466+0000","last_active":"2024-08-19T14:40:15.641426+0000","last_peered":"2024-08-19T14:40:15.641426+0000","last_clean":"2024-08-19T14:40:15.641426+0000","last_became_active":"2024-08-19T14:40:04.442173+0000","last_became_peered":"2024-08-19T14:40:04.442173+0000","last_unstale":"2024-08-19T14:40:15.641426+0000","last_undegraded":"2024-08-19T14:40:15.641426+0000","last_fullsized":"2024-08-19T14:40:15.641426+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":717,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.182354+0000","objects_scrubbed":0,"log_size":811,"log_dups_size":0,"ondisk_log_size":811,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:25.949039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53149696,"num_objects":811,"num_object_clones":0,"num_object_copies":2433,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":811,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":811,"num_write_kb":51904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"70'139","reported_seq":1771,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-08-19T14:40:11.225931+0000","last_change":"2024-08-19T14:39:45.569137+0000","last_active":"2024-08-19T14:40:11.225931+0000","last_peered":"2024-08-19T14:40:11.225931+0000","last_clean":"2024-08-19T14:40:11.225931+0000","last_became_active":"2024-08-19T14:35:13.950224+0000","last_became_peered":"2024-08-19T14:35:13.950224+0000","last_unstale":"2024-08-19T14:40:11.225931+0000","last_undegraded":"2024-08-19T14:40:11.225931+0000","last_fullsized":"2024-08-19T14:40:11.225931+0000","mapping_epoch":637,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":638,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:39:45.569030+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:19:58.072083+0000","last_clean_scrub_stamp":"2024-08-19T14:39:45.569030+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:40:57.206713+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,9],"acting":[0,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"722'745","reported_seq":764,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-08-19T14:40:15.358077+0000","last_change":"2024-08-19T14:40:04.442002+0000","last_active":"2024-08-19T14:40:15.358077+0000","last_peered":"2024-08-19T14:40:15.358077+0000","last_clean":"2024-08-19T14:40:15.358077+0000","last_became_active":"2024-08-19T14:40:04.441586+0000","last_became_peered":"2024-08-19T14:40:04.441586+0000","last_unstale":"2024-08-19T14:40:15.358077+0000","last_undegraded":"2024-08-19T14:40:15.358077+0000","last_fullsized":"2024-08-19T14:40:15.358077+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":717,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.182354+0000","objects_scrubbed":0,"log_size":745,"log_dups_size":0,"ondisk_log_size":745,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:26.482555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48824320,"num_objects":745,"num_object_clones":0,"num_object_copies":2235,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":745,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":745,"num_write_kb":47680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,9],"acting":[8,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"70'139","reported_seq":1615,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-08-19T14:40:11.289703+0000","last_change":"2024-08-19T14:40:02.681316+0000","last_active":"2024-08-19T14:40:11.289703+0000","last_peered":"2024-08-19T14:40:11.289703+0000","last_clean":"2024-08-19T14:40:11.289703+0000","last_became_active":"2024-08-19T14:18:13.804721+0000","last_became_peered":"2024-08-19T14:18:13.804721+0000","last_unstale":"2024-08-19T14:40:11.289703+0000","last_undegraded":"2024-08-19T14:40:11.289703+0000","last_fullsized":"2024-08-19T14:40:11.289703+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:40:02.681185+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:40:02.681185+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:22.300124+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.b","version":"722'739","reported_seq":759,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-08-19T14:40:15.335744+0000","last_change":"2024-08-19T14:40:04.200633+0000","last_active":"2024-08-19T14:40:15.335744+0000","last_peered":"2024-08-19T14:40:15.335744+0000","last_clean":"2024-08-19T14:40:15.335744+0000","last_became_active":"2024-08-19T14:40:04.200299+0000","last_became_peered":"2024-08-19T14:40:04.200299+0000","last_unstale":"2024-08-19T14:40:15.335744+0000","last_undegraded":"2024-08-19T14:40:15.335744+0000","last_fullsized":"2024-08-19T14:40:15.335744+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":717,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.182354+0000","objects_scrubbed":0,"log_size":739,"log_dups_size":0,"ondisk_log_size":739,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:27.707935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48431104,"num_objects":739,"num_object_clones":0,"num_object_copies":2217,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":739,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":739,"num_write_kb":47296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,1],"acting":[8,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.2","version":"70'139","reported_seq":1572,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-08-19T14:40:11.226193+0000","last_change":"2024-08-19T14:40:10.977253+0000","last_active":"2024-08-19T14:40:11.226193+0000","last_peered":"2024-08-19T14:40:11.226193+0000","last_clean":"2024-08-19T14:40:11.226193+0000","last_became_active":"2024-08-19T14:20:50.742354+0000","last_became_peered":"2024-08-19T14:20:50.742354+0000","last_unstale":"2024-08-19T14:40:11.226193+0000","last_undegraded":"2024-08-19T14:40:11.226193+0000","last_fullsized":"2024-08-19T14:40:11.226193+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:40:10.977156+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:19:59.302931+0000","last_clean_scrub_stamp":"2024-08-19T14:40:10.977156+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:39.144591+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.8","version":"722'833","reported_seq":853,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-08-19T14:40:15.661612+0000","last_change":"2024-08-19T14:40:04.443203+0000","last_active":"2024-08-19T14:40:15.661612+0000","last_peered":"2024-08-19T14:40:15.661612+0000","last_clean":"2024-08-19T14:40:15.661612+0000","last_became_active":"2024-08-19T14:40:04.442802+0000","last_became_peered":"2024-08-19T14:40:04.442802+0000","last_unstale":"2024-08-19T14:40:15.661612+0000","last_undegraded":"2024-08-19T14:40:15.661612+0000","last_fullsized":"2024-08-19T14:40:15.661612+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":717,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.182354+0000","objects_scrubbed":0,"log_size":833,"log_dups_size":0,"ondisk_log_size":833,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:14.120759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54591488,"num_objects":833,"num_object_clones":0,"num_object_copies":2499,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":833,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":833,"num_write_kb":53312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,9],"acting":[3,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"70'139","reported_seq":870,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-08-19T14:40:11.226078+0000","last_change":"2024-08-19T14:39:37.543259+0000","last_active":"2024-08-19T14:40:11.226078+0000","last_peered":"2024-08-19T14:40:11.226078+0000","last_clean":"2024-08-19T14:40:11.226078+0000","last_became_active":"2024-08-19T14:35:12.530566+0000","last_became_peered":"2024-08-19T14:35:12.530566+0000","last_unstale":"2024-08-19T14:40:11.226078+0000","last_undegraded":"2024-08-19T14:40:11.226078+0000","last_fullsized":"2024-08-19T14:40:11.226078+0000","mapping_epoch":635,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":636,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:39:37.543187+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:39:37.543187+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:40:56.083490+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,2],"acting":[0,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.e","version":"722'809","reported_seq":829,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-08-19T14:40:15.679039+0000","last_change":"2024-08-19T14:40:04.299744+0000","last_active":"2024-08-19T14:40:15.679039+0000","last_peered":"2024-08-19T14:40:15.679039+0000","last_clean":"2024-08-19T14:40:15.679039+0000","last_became_active":"2024-08-19T14:40:04.299403+0000","last_became_peered":"2024-08-19T14:40:04.299403+0000","last_unstale":"2024-08-19T14:40:15.679039+0000","last_undegraded":"2024-08-19T14:40:15.679039+0000","last_fullsized":"2024-08-19T14:40:15.679039+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":717,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.182354+0000","objects_scrubbed":0,"log_size":809,"log_dups_size":0,"ondisk_log_size":809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:22.448115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53018624,"num_objects":809,"num_object_clones":0,"num_object_copies":2427,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":809,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":809,"num_write_kb":51776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"70'139","reported_seq":1274,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-08-19T14:40:11.225122+0000","last_change":"2024-08-19T14:39:56.993699+0000","last_active":"2024-08-19T14:40:11.225122+0000","last_peered":"2024-08-19T14:40:11.225122+0000","last_clean":"2024-08-19T14:40:11.225122+0000","last_became_active":"2024-08-19T14:35:11.536828+0000","last_became_peered":"2024-08-19T14:35:11.536828+0000","last_unstale":"2024-08-19T14:40:11.225122+0000","last_undegraded":"2024-08-19T14:40:11.225122+0000","last_fullsized":"2024-08-19T14:40:11.225122+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:39:56.993616+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:21:46.057624+0000","last_clean_scrub_stamp":"2024-08-19T14:39:56.993616+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:05.362216+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.f","version":"722'94","reported_seq":114,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-08-19T14:40:11.746020+0000","last_change":"2024-08-19T14:40:04.299935+0000","last_active":"2024-08-19T14:40:11.746020+0000","last_peered":"2024-08-19T14:40:11.746020+0000","last_clean":"2024-08-19T14:40:11.746020+0000","last_became_active":"2024-08-19T14:40:04.299656+0000","last_became_peered":"2024-08-19T14:40:04.299656+0000","last_unstale":"2024-08-19T14:40:11.746020+0000","last_undegraded":"2024-08-19T14:40:11.746020+0000","last_fullsized":"2024-08-19T14:40:11.746020+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":717,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.182354+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:22.129102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6160384,"num_objects":94,"num_object_clones":0,"num_object_copies":282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":94,"num_write_kb":6016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,1],"acting":[11,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.6","version":"70'139","reported_seq":842,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-08-19T14:40:11.227309+0000","last_change":"2024-08-19T14:39:30.460768+0000","last_active":"2024-08-19T14:40:11.227309+0000","last_peered":"2024-08-19T14:40:11.227309+0000","last_clean":"2024-08-19T14:40:11.227309+0000","last_became_active":"2024-08-19T14:34:15.717054+0000","last_became_peered":"2024-08-19T14:34:15.717054+0000","last_unstale":"2024-08-19T14:40:11.227309+0000","last_undegraded":"2024-08-19T14:40:11.227309+0000","last_fullsized":"2024-08-19T14:40:11.227309+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:39:30.460649+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:27.848870+0000","last_clean_scrub_stamp":"2024-08-19T14:39:30.460649+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:40:59.351491+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.c","version":"722'773","reported_seq":792,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-08-19T14:40:15.359607+0000","last_change":"2024-08-19T14:40:04.442056+0000","last_active":"2024-08-19T14:40:15.359607+0000","last_peered":"2024-08-19T14:40:15.359607+0000","last_clean":"2024-08-19T14:40:15.359607+0000","last_became_active":"2024-08-19T14:40:04.441602+0000","last_became_peered":"2024-08-19T14:40:04.441602+0000","last_unstale":"2024-08-19T14:40:15.359607+0000","last_undegraded":"2024-08-19T14:40:15.359607+0000","last_fullsized":"2024-08-19T14:40:15.359607+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":717,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.182354+0000","objects_scrubbed":0,"log_size":773,"log_dups_size":0,"ondisk_log_size":773,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:20.409556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50659328,"num_objects":773,"num_object_clones":0,"num_object_copies":2319,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":773,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":773,"num_write_kb":49472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,3],"acting":[8,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.5","version":"70'139","reported_seq":796,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-08-19T14:40:11.226832+0000","last_change":"2024-08-19T14:39:54.886149+0000","last_active":"2024-08-19T14:40:11.226832+0000","last_peered":"2024-08-19T14:40:11.226832+0000","last_clean":"2024-08-19T14:40:11.226832+0000","last_became_active":"2024-08-19T14:35:11.532858+0000","last_became_peered":"2024-08-19T14:35:11.532858+0000","last_unstale":"2024-08-19T14:40:11.226832+0000","last_undegraded":"2024-08-19T14:40:11.226832+0000","last_fullsized":"2024-08-19T14:40:11.226832+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:39:54.886056+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:39:54.886056+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:40:57.762183+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,0],"acting":[11,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.d","version":"722'113","reported_seq":133,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-08-19T14:40:11.745666+0000","last_change":"2024-08-19T14:40:04.443252+0000","last_active":"2024-08-19T14:40:11.745666+0000","last_peered":"2024-08-19T14:40:11.745666+0000","last_clean":"2024-08-19T14:40:11.745666+0000","last_became_active":"2024-08-19T14:40:04.442093+0000","last_became_peered":"2024-08-19T14:40:04.442093+0000","last_unstale":"2024-08-19T14:40:11.745666+0000","last_undegraded":"2024-08-19T14:40:11.745666+0000","last_fullsized":"2024-08-19T14:40:11.745666+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":717,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.182354+0000","objects_scrubbed":0,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:22.854336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7405568,"num_objects":113,"num_object_clones":0,"num_object_copies":339,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":113,"num_write_kb":7232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,2],"acting":[11,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.4","version":"70'139","reported_seq":673,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-08-19T14:40:11.226745+0000","last_change":"2024-08-19T14:40:09.799125+0000","last_active":"2024-08-19T14:40:11.226745+0000","last_peered":"2024-08-19T14:40:11.226745+0000","last_clean":"2024-08-19T14:40:11.226745+0000","last_became_active":"2024-08-19T14:30:47.909067+0000","last_became_peered":"2024-08-19T14:30:47.909067+0000","last_unstale":"2024-08-19T14:40:11.226745+0000","last_undegraded":"2024-08-19T14:40:11.226745+0000","last_fullsized":"2024-08-19T14:40:11.226745+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:40:09.798994+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:40:09.798994+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:32.468916+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.3","version":"722'124","reported_seq":143,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-08-19T14:40:11.745935+0000","last_change":"2024-08-19T14:40:04.443269+0000","last_active":"2024-08-19T14:40:11.745935+0000","last_peered":"2024-08-19T14:40:11.745935+0000","last_clean":"2024-08-19T14:40:11.745935+0000","last_became_active":"2024-08-19T14:40:04.442837+0000","last_became_peered":"2024-08-19T14:40:04.442837+0000","last_unstale":"2024-08-19T14:40:11.745935+0000","last_undegraded":"2024-08-19T14:40:11.745935+0000","last_fullsized":"2024-08-19T14:40:11.745935+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":717,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.182354+0000","objects_scrubbed":0,"log_size":124,"log_dups_size":0,"ondisk_log_size":124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:13.689400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8126464,"num_objects":124,"num_object_clones":0,"num_object_copies":372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":124,"num_write_kb":7936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.a","version":"70'139","reported_seq":838,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-08-19T14:40:11.225631+0000","last_change":"2024-08-19T14:40:05.713501+0000","last_active":"2024-08-19T14:40:11.225631+0000","last_peered":"2024-08-19T14:40:11.225631+0000","last_clean":"2024-08-19T14:40:11.225631+0000","last_became_active":"2024-08-19T14:34:15.717472+0000","last_became_peered":"2024-08-19T14:34:15.717472+0000","last_unstale":"2024-08-19T14:40:11.225631+0000","last_undegraded":"2024-08-19T14:40:11.225631+0000","last_fullsized":"2024-08-19T14:40:11.225631+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:40:05.713411+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:28.864870+0000","last_clean_scrub_stamp":"2024-08-19T14:40:05.713411+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:12.701795+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,0],"acting":[4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.0","version":"722'819","reported_seq":839,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-08-19T14:40:15.645489+0000","last_change":"2024-08-19T14:40:04.443641+0000","last_active":"2024-08-19T14:40:15.645489+0000","last_peered":"2024-08-19T14:40:15.645489+0000","last_clean":"2024-08-19T14:40:15.645489+0000","last_became_active":"2024-08-19T14:40:04.443254+0000","last_became_peered":"2024-08-19T14:40:04.443254+0000","last_unstale":"2024-08-19T14:40:15.645489+0000","last_undegraded":"2024-08-19T14:40:15.645489+0000","last_fullsized":"2024-08-19T14:40:15.645489+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":717,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.182354+0000","objects_scrubbed":0,"log_size":819,"log_dups_size":0,"ondisk_log_size":819,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:21.728211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53673984,"num_objects":819,"num_object_clones":0,"num_object_copies":2457,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":819,"num_write_kb":52416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,9],"acting":[3,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"70'139","reported_seq":1774,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-08-19T14:40:11.226172+0000","last_change":"2024-08-19T14:40:05.923941+0000","last_active":"2024-08-19T14:40:11.226172+0000","last_peered":"2024-08-19T14:40:11.226172+0000","last_clean":"2024-08-19T14:40:11.226172+0000","last_became_active":"2024-08-19T14:35:12.529628+0000","last_became_peered":"2024-08-19T14:35:12.529628+0000","last_unstale":"2024-08-19T14:40:11.226172+0000","last_undegraded":"2024-08-19T14:40:11.226172+0000","last_fullsized":"2024-08-19T14:40:11.226172+0000","mapping_epoch":635,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":636,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:40:05.923850+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:40:05.923850+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:22.262135+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1","version":"722'381","reported_seq":401,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-08-19T14:40:12.974303+0000","last_change":"2024-08-19T14:40:04.441532+0000","last_active":"2024-08-19T14:40:12.974303+0000","last_peered":"2024-08-19T14:40:12.974303+0000","last_clean":"2024-08-19T14:40:12.974303+0000","last_became_active":"2024-08-19T14:40:04.441139+0000","last_became_peered":"2024-08-19T14:40:04.441139+0000","last_unstale":"2024-08-19T14:40:12.974303+0000","last_undegraded":"2024-08-19T14:40:12.974303+0000","last_fullsized":"2024-08-19T14:40:12.974303+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":717,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.182354+0000","objects_scrubbed":0,"log_size":381,"log_dups_size":0,"ondisk_log_size":381,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:30.989902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24969216,"num_objects":381,"num_object_clones":0,"num_object_copies":1143,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":381,"num_write_kb":24384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,2],"acting":[1,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"70'139","reported_seq":804,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-08-19T14:40:11.225087+0000","last_change":"2024-08-19T14:39:59.000415+0000","last_active":"2024-08-19T14:40:11.225087+0000","last_peered":"2024-08-19T14:40:11.225087+0000","last_clean":"2024-08-19T14:40:11.225087+0000","last_became_active":"2024-08-19T14:35:11.537474+0000","last_became_peered":"2024-08-19T14:35:11.537474+0000","last_unstale":"2024-08-19T14:40:11.225087+0000","last_undegraded":"2024-08-19T14:40:11.225087+0000","last_fullsized":"2024-08-19T14:40:11.225087+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:39:59.000300+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:39:59.000300+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:11.528949+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.2","version":"722'835","reported_seq":855,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-08-19T14:40:15.668997+0000","last_change":"2024-08-19T14:40:04.441786+0000","last_active":"2024-08-19T14:40:15.668997+0000","last_peered":"2024-08-19T14:40:15.668997+0000","last_clean":"2024-08-19T14:40:15.668997+0000","last_became_active":"2024-08-19T14:40:04.441445+0000","last_became_peered":"2024-08-19T14:40:04.441445+0000","last_unstale":"2024-08-19T14:40:15.668997+0000","last_undegraded":"2024-08-19T14:40:15.668997+0000","last_fullsized":"2024-08-19T14:40:15.668997+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":717,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.182354+0000","objects_scrubbed":0,"log_size":835,"log_dups_size":0,"ondisk_log_size":835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:23.583504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54722560,"num_objects":835,"num_object_clones":0,"num_object_copies":2505,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":835,"num_write_kb":53440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,9],"acting":[0,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.b","version":"70'139","reported_seq":1125,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-08-19T14:40:11.226853+0000","last_change":"2024-08-19T14:40:03.813911+0000","last_active":"2024-08-19T14:40:11.226853+0000","last_peered":"2024-08-19T14:40:11.226853+0000","last_clean":"2024-08-19T14:40:11.226853+0000","last_became_active":"2024-08-19T14:30:47.941200+0000","last_became_peered":"2024-08-19T14:30:47.941200+0000","last_unstale":"2024-08-19T14:40:11.226853+0000","last_undegraded":"2024-08-19T14:40:11.226853+0000","last_fullsized":"2024-08-19T14:40:11.226853+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:40:03.813794+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.813794+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:29.146651+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.7","version":"722'385","reported_seq":405,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-08-19T14:40:12.979329+0000","last_change":"2024-08-19T14:40:04.299820+0000","last_active":"2024-08-19T14:40:12.979329+0000","last_peered":"2024-08-19T14:40:12.979329+0000","last_clean":"2024-08-19T14:40:12.979329+0000","last_became_active":"2024-08-19T14:40:04.299448+0000","last_became_peered":"2024-08-19T14:40:04.299448+0000","last_unstale":"2024-08-19T14:40:12.979329+0000","last_undegraded":"2024-08-19T14:40:12.979329+0000","last_fullsized":"2024-08-19T14:40:12.979329+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":717,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.182354+0000","objects_scrubbed":0,"log_size":385,"log_dups_size":0,"ondisk_log_size":385,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:05.592419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25231360,"num_objects":385,"num_object_clones":0,"num_object_copies":1155,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":385,"num_write_kb":24640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.e","version":"70'139","reported_seq":1315,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-08-19T14:40:11.225232+0000","last_change":"2024-08-19T14:40:03.764766+0000","last_active":"2024-08-19T14:40:11.225232+0000","last_peered":"2024-08-19T14:40:11.225232+0000","last_clean":"2024-08-19T14:40:11.225232+0000","last_became_active":"2024-08-19T14:18:26.884105+0000","last_became_peered":"2024-08-19T14:18:26.884105+0000","last_unstale":"2024-08-19T14:40:11.225232+0000","last_undegraded":"2024-08-19T14:40:11.225232+0000","last_fullsized":"2024-08-19T14:40:11.225232+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:40:03.764672+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:40:03.764672+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.764672+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:11.797638+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.6","version":"722'104","reported_seq":124,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-08-19T14:40:11.739432+0000","last_change":"2024-08-19T14:40:04.443511+0000","last_active":"2024-08-19T14:40:11.739432+0000","last_peered":"2024-08-19T14:40:11.739432+0000","last_clean":"2024-08-19T14:40:11.739432+0000","last_became_active":"2024-08-19T14:40:04.442943+0000","last_became_peered":"2024-08-19T14:40:04.442943+0000","last_unstale":"2024-08-19T14:40:11.739432+0000","last_undegraded":"2024-08-19T14:40:11.739432+0000","last_fullsized":"2024-08-19T14:40:11.739432+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":717,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.182354+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:29.082021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6815744,"num_objects":104,"num_object_clones":0,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":104,"num_write_kb":6656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,2],"acting":[11,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.f","version":"70'139","reported_seq":1067,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-08-19T14:40:11.225429+0000","last_change":"2024-08-19T14:39:59.812809+0000","last_active":"2024-08-19T14:40:11.225429+0000","last_peered":"2024-08-19T14:40:11.225429+0000","last_clean":"2024-08-19T14:40:11.225429+0000","last_became_active":"2024-08-19T14:27:31.837754+0000","last_became_peered":"2024-08-19T14:27:31.837754+0000","last_unstale":"2024-08-19T14:40:11.225429+0000","last_undegraded":"2024-08-19T14:40:11.225429+0000","last_fullsized":"2024-08-19T14:40:11.225429+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:39:59.812612+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:39:59.812612+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:07.074257+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.5","version":"722'512","reported_seq":532,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-08-19T14:40:13.734624+0000","last_change":"2024-08-19T14:40:04.200422+0000","last_active":"2024-08-19T14:40:13.734624+0000","last_peered":"2024-08-19T14:40:13.734624+0000","last_clean":"2024-08-19T14:40:13.734624+0000","last_became_active":"2024-08-19T14:40:04.200106+0000","last_became_peered":"2024-08-19T14:40:04.200106+0000","last_unstale":"2024-08-19T14:40:13.734624+0000","last_undegraded":"2024-08-19T14:40:13.734624+0000","last_fullsized":"2024-08-19T14:40:13.734624+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":717,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.182354+0000","objects_scrubbed":0,"log_size":512,"log_dups_size":0,"ondisk_log_size":512,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:09.342864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33554432,"num_objects":512,"num_object_clones":0,"num_object_copies":1536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":512,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":512,"num_write_kb":32768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,11],"acting":[4,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.c","version":"70'139","reported_seq":1422,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-08-19T14:40:11.225287+0000","last_change":"2024-08-19T14:40:00.947434+0000","last_active":"2024-08-19T14:40:11.225287+0000","last_peered":"2024-08-19T14:40:11.225287+0000","last_clean":"2024-08-19T14:40:11.225287+0000","last_became_active":"2024-08-19T14:34:15.733282+0000","last_became_peered":"2024-08-19T14:34:15.733282+0000","last_unstale":"2024-08-19T14:40:11.225287+0000","last_undegraded":"2024-08-19T14:40:11.225287+0000","last_fullsized":"2024-08-19T14:40:11.225287+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:40:00.947341+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:22:52.164586+0000","last_clean_scrub_stamp":"2024-08-19T14:40:00.947341+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:21.334102+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,9],"acting":[3,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"722'111","reported_seq":131,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-08-19T14:40:11.742630+0000","last_change":"2024-08-19T14:40:04.443126+0000","last_active":"2024-08-19T14:40:11.742630+0000","last_peered":"2024-08-19T14:40:11.742630+0000","last_clean":"2024-08-19T14:40:11.742630+0000","last_became_active":"2024-08-19T14:40:04.442757+0000","last_became_peered":"2024-08-19T14:40:04.442757+0000","last_unstale":"2024-08-19T14:40:11.742630+0000","last_undegraded":"2024-08-19T14:40:11.742630+0000","last_fullsized":"2024-08-19T14:40:11.742630+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":717,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.182354+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:06.385105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7274496,"num_objects":111,"num_object_clones":0,"num_object_copies":333,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":111,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":111,"num_write_kb":7104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,9],"acting":[11,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.d","version":"70'139","reported_seq":1319,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-08-19T14:40:11.226442+0000","last_change":"2024-08-19T14:40:08.911300+0000","last_active":"2024-08-19T14:40:11.226442+0000","last_peered":"2024-08-19T14:40:11.226442+0000","last_clean":"2024-08-19T14:40:11.226442+0000","last_became_active":"2024-08-19T14:18:26.882002+0000","last_became_peered":"2024-08-19T14:18:26.882002+0000","last_unstale":"2024-08-19T14:40:11.226442+0000","last_undegraded":"2024-08-19T14:40:11.226442+0000","last_fullsized":"2024-08-19T14:40:11.226442+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:40:08.911204+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:24:50.276408+0000","last_clean_scrub_stamp":"2024-08-19T14:40:08.911204+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:27.367708+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"70'139","reported_seq":1098,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-08-19T14:40:11.289579+0000","last_change":"2024-08-19T14:40:04.665520+0000","last_active":"2024-08-19T14:40:11.289579+0000","last_peered":"2024-08-19T14:40:11.289579+0000","last_clean":"2024-08-19T14:40:11.289579+0000","last_became_active":"2024-08-19T14:27:31.838968+0000","last_became_peered":"2024-08-19T14:27:31.838968+0000","last_unstale":"2024-08-19T14:40:11.289579+0000","last_undegraded":"2024-08-19T14:40:11.289579+0000","last_fullsized":"2024-08-19T14:40:11.289579+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:40:04.665441+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:28:35.206115+0000","last_clean_scrub_stamp":"2024-08-19T14:40:04.665441+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:25.039402+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.13","version":"70'139","reported_seq":1640,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-08-19T14:40:11.289680+0000","last_change":"2024-08-19T14:40:05.659348+0000","last_active":"2024-08-19T14:40:11.289680+0000","last_peered":"2024-08-19T14:40:11.289680+0000","last_clean":"2024-08-19T14:40:11.289680+0000","last_became_active":"2024-08-19T14:30:47.939849+0000","last_became_peered":"2024-08-19T14:30:47.939849+0000","last_unstale":"2024-08-19T14:40:11.289680+0000","last_undegraded":"2024-08-19T14:40:11.289680+0000","last_fullsized":"2024-08-19T14:40:11.289680+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:40:05.659275+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:33:35.786105+0000","last_clean_scrub_stamp":"2024-08-19T14:40:05.659275+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:30.843663+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.10","version":"70'139","reported_seq":1260,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-08-19T14:40:11.226225+0000","last_change":"2024-08-19T14:34:58.616722+0000","last_active":"2024-08-19T14:40:11.226225+0000","last_peered":"2024-08-19T14:40:11.226225+0000","last_clean":"2024-08-19T14:40:11.226225+0000","last_became_active":"2024-08-19T14:27:31.839611+0000","last_became_peered":"2024-08-19T14:27:31.839611+0000","last_unstale":"2024-08-19T14:40:11.226225+0000","last_undegraded":"2024-08-19T14:40:11.226225+0000","last_fullsized":"2024-08-19T14:40:11.226225+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:34:58.616646+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:34:58.616646+0000","last_clean_scrub_stamp":"2024-08-19T14:34:58.616646+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.11","version":"70'139","reported_seq":1324,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-08-19T14:40:11.226047+0000","last_change":"2024-08-19T14:40:04.812392+0000","last_active":"2024-08-19T14:40:11.226047+0000","last_peered":"2024-08-19T14:40:11.226047+0000","last_clean":"2024-08-19T14:40:11.226047+0000","last_became_active":"2024-08-19T14:34:15.731049+0000","last_became_peered":"2024-08-19T14:34:15.731049+0000","last_unstale":"2024-08-19T14:40:11.226047+0000","last_undegraded":"2024-08-19T14:40:11.226047+0000","last_fullsized":"2024-08-19T14:40:11.226047+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:40:04.812269+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:24:50.547428+0000","last_clean_scrub_stamp":"2024-08-19T14:40:04.812269+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:33.537909+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"70'139","reported_seq":1251,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-08-19T14:40:11.289739+0000","last_change":"2024-08-19T14:40:03.700753+0000","last_active":"2024-08-19T14:40:11.289739+0000","last_peered":"2024-08-19T14:40:11.289739+0000","last_clean":"2024-08-19T14:40:11.289739+0000","last_became_active":"2024-08-19T14:20:28.204724+0000","last_became_peered":"2024-08-19T14:20:28.204724+0000","last_unstale":"2024-08-19T14:40:11.289739+0000","last_undegraded":"2024-08-19T14:40:11.289739+0000","last_fullsized":"2024-08-19T14:40:11.289739+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:40:03.700675+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:28:34.216370+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.700675+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:20.590299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"70'139","reported_seq":1393,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-08-19T14:40:11.227508+0000","last_change":"2024-08-19T14:38:46.389978+0000","last_active":"2024-08-19T14:40:11.227508+0000","last_peered":"2024-08-19T14:40:11.227508+0000","last_clean":"2024-08-19T14:40:11.227508+0000","last_became_active":"2024-08-19T14:18:31.901556+0000","last_became_peered":"2024-08-19T14:18:31.901556+0000","last_unstale":"2024-08-19T14:40:11.227508+0000","last_undegraded":"2024-08-19T14:40:11.227508+0000","last_fullsized":"2024-08-19T14:40:11.227508+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:38:46.389862+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:38:46.389862+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,0],"acting":[8,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.14","version":"70'139","reported_seq":1423,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-08-19T14:40:11.226220+0000","last_change":"2024-08-19T14:40:03.932359+0000","last_active":"2024-08-19T14:40:11.226220+0000","last_peered":"2024-08-19T14:40:11.226220+0000","last_clean":"2024-08-19T14:40:11.226220+0000","last_became_active":"2024-08-19T14:35:12.529792+0000","last_became_peered":"2024-08-19T14:35:12.529792+0000","last_unstale":"2024-08-19T14:40:11.226220+0000","last_undegraded":"2024-08-19T14:40:11.226220+0000","last_fullsized":"2024-08-19T14:40:11.226220+0000","mapping_epoch":635,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":636,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:40:03.932271+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.932271+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:23.358249+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,9],"acting":[7,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.15","version":"70'139","reported_seq":1275,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-08-19T14:40:11.226389+0000","last_change":"2024-08-19T14:40:07.881907+0000","last_active":"2024-08-19T14:40:11.226389+0000","last_peered":"2024-08-19T14:40:11.226389+0000","last_clean":"2024-08-19T14:40:11.226389+0000","last_became_active":"2024-08-19T14:18:31.912496+0000","last_became_peered":"2024-08-19T14:18:31.912496+0000","last_unstale":"2024-08-19T14:40:11.226389+0000","last_undegraded":"2024-08-19T14:40:11.226389+0000","last_fullsized":"2024-08-19T14:40:11.226389+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:40:07.881806+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:20:09.180923+0000","last_clean_scrub_stamp":"2024-08-19T14:40:07.881806+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:32.759701+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"70'139","reported_seq":1260,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-08-19T14:40:11.225459+0000","last_change":"2024-08-19T14:40:00.753735+0000","last_active":"2024-08-19T14:40:11.225459+0000","last_peered":"2024-08-19T14:40:11.225459+0000","last_clean":"2024-08-19T14:40:11.225459+0000","last_became_active":"2024-08-19T14:18:31.901229+0000","last_became_peered":"2024-08-19T14:18:31.901229+0000","last_unstale":"2024-08-19T14:40:11.225459+0000","last_undegraded":"2024-08-19T14:40:11.225459+0000","last_fullsized":"2024-08-19T14:40:11.225459+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:40:00.753636+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:40:00.753636+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:07.574708+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1b","version":"70'139","reported_seq":1641,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-08-19T14:40:11.227489+0000","last_change":"2024-08-19T14:39:32.443757+0000","last_active":"2024-08-19T14:40:11.227489+0000","last_peered":"2024-08-19T14:40:11.227489+0000","last_clean":"2024-08-19T14:40:11.227489+0000","last_became_active":"2024-08-19T14:34:15.718427+0000","last_became_peered":"2024-08-19T14:34:15.718427+0000","last_unstale":"2024-08-19T14:40:11.227489+0000","last_undegraded":"2024-08-19T14:40:11.227489+0000","last_fullsized":"2024-08-19T14:40:11.227489+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:39:32.443656+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:37:59.510697+0000","last_clean_scrub_stamp":"2024-08-19T14:39:32.443656+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:40:40.877428+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.18","version":"70'139","reported_seq":1203,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-08-19T14:40:11.226155+0000","last_change":"2024-08-19T14:39:59.937810+0000","last_active":"2024-08-19T14:40:11.226155+0000","last_peered":"2024-08-19T14:40:11.226155+0000","last_clean":"2024-08-19T14:40:11.226155+0000","last_became_active":"2024-08-19T14:35:11.528530+0000","last_became_peered":"2024-08-19T14:35:11.528530+0000","last_unstale":"2024-08-19T14:40:11.226155+0000","last_undegraded":"2024-08-19T14:40:11.226155+0000","last_fullsized":"2024-08-19T14:40:11.226155+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:39:59.937668+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:27:33.084161+0000","last_clean_scrub_stamp":"2024-08-19T14:39:59.937668+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:17.756316+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.19","version":"70'139","reported_seq":1620,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-08-19T14:40:11.225970+0000","last_change":"2024-08-19T14:39:32.723246+0000","last_active":"2024-08-19T14:40:11.225970+0000","last_peered":"2024-08-19T14:40:11.225970+0000","last_clean":"2024-08-19T14:40:11.225970+0000","last_became_active":"2024-08-19T14:18:31.902547+0000","last_became_peered":"2024-08-19T14:18:31.902547+0000","last_unstale":"2024-08-19T14:40:11.225970+0000","last_undegraded":"2024-08-19T14:40:11.225970+0000","last_fullsized":"2024-08-19T14:40:11.225970+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:39:32.722703+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:39:32.722703+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:40:40.564491+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1e","version":"70'139","reported_seq":637,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-08-19T14:40:11.226814+0000","last_change":"2024-08-19T14:33:17.778415+0000","last_active":"2024-08-19T14:40:11.226814+0000","last_peered":"2024-08-19T14:40:11.226814+0000","last_clean":"2024-08-19T14:40:11.226814+0000","last_became_active":"2024-08-19T14:30:47.909974+0000","last_became_peered":"2024-08-19T14:30:47.909974+0000","last_unstale":"2024-08-19T14:40:11.226814+0000","last_undegraded":"2024-08-19T14:40:11.226814+0000","last_fullsized":"2024-08-19T14:40:11.226814+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:33:17.778295+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:27:04.780939+0000","last_clean_scrub_stamp":"2024-08-19T14:33:17.778295+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1c","version":"70'139","reported_seq":1395,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-08-19T14:40:11.227293+0000","last_change":"2024-08-19T14:39:26.405807+0000","last_active":"2024-08-19T14:40:11.227293+0000","last_peered":"2024-08-19T14:40:11.227293+0000","last_clean":"2024-08-19T14:40:11.227293+0000","last_became_active":"2024-08-19T14:30:47.908866+0000","last_became_peered":"2024-08-19T14:30:47.908866+0000","last_unstale":"2024-08-19T14:40:11.227293+0000","last_undegraded":"2024-08-19T14:40:11.227293+0000","last_fullsized":"2024-08-19T14:40:11.227293+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:39:26.405699+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:39:26.405699+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:40:29.159585+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,11],"acting":[8,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1d","version":"70'139","reported_seq":1304,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-08-19T14:40:11.289753+0000","last_change":"2024-08-19T14:39:57.775710+0000","last_active":"2024-08-19T14:40:11.289753+0000","last_peered":"2024-08-19T14:40:11.289753+0000","last_clean":"2024-08-19T14:40:11.289753+0000","last_became_active":"2024-08-19T14:35:11.540488+0000","last_became_peered":"2024-08-19T14:35:11.540488+0000","last_unstale":"2024-08-19T14:40:11.289753+0000","last_undegraded":"2024-08-19T14:40:11.289753+0000","last_fullsized":"2024-08-19T14:40:11.289753+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:39:57.775631+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:39:57.775631+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:40:59.182147+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":536608768,"num_objects":8188,"num_object_clones":0,"num_object_copies":24564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8188,"num_write_kb":524032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1534722048,"data_stored":1534722048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8188,"ondisk_log_size":8188,"up":48,"acting":48,"num_store_stats":10},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":856608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187802,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":883876,"kb_used_data":8600,"kb_used_omap":1,"kb_used_meta":875262,"kb_avail":92861276,"statfs":{"total":95995035648,"available":95089946624,"internally_reserved":0,"allocated":8806400,"data_stored":5624803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":896268740},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":2314987372670,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":74,"seq":317827580260,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1001328,"kb_used_data":145508,"kb_used_omap":1,"kb_used_meta":855806,"kb_avail":92743824,"statfs":{"total":95995035648,"available":94969675776,"internally_reserved":0,"allocated":149000192,"data_stored":145814340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":876345793},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874603,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":922092,"kb_used_data":146788,"kb_used_omap":1,"kb_used_meta":775294,"kb_avail":92823060,"statfs":{"total":95995035648,"available":95050813440,"internally_reserved":0,"allocated":150310912,"data_stored":147125060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":793901509},"hb_peers":[0,1,2,3,4,5,6,7,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":57,"seq":244813136238,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1057716,"kb_used_data":101932,"kb_used_omap":1,"kb_used_meta":955774,"kb_avail":92687436,"statfs":{"total":95995035648,"available":94911934464,"internally_reserved":0,"allocated":104378368,"data_stored":101188964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":978713034},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":206158430577,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692212,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":946540,"kb_used_data":8800,"kb_used_omap":1,"kb_used_meta":937726,"kb_avail":92798612,"statfs":{"total":95995035648,"available":95025778688,"internally_reserved":0,"allocated":9011200,"data_stored":5826051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":960231877},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986553,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1079028,"kb_used_data":91300,"kb_used_omap":1,"kb_used_meta":987710,"kb_avail":92666124,"statfs":{"total":95995035648,"available":94890110976,"internally_reserved":0,"allocated":93491200,"data_stored":90305348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1011415493},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150080,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1215980,"kb_used_data":188004,"kb_used_omap":1,"kb_used_meta":1027966,"kb_avail":92529172,"statfs":{"total":95995035648,"available":94749872128,"internally_reserved":0,"allocated":192516096,"data_stored":189330244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1052637639},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":634,"seq":2723009265726,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":571252,"kb_used_data":286508,"kb_used_omap":4,"kb_used_meta":284731,"kb_avail":93173900,"statfs":{"total":95995035648,"available":95410073600,"internally_reserved":0,"allocated":293384192,"data_stored":290194788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4407,"internal_metadata":291565257},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575240,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1232940,"kb_used_data":118884,"kb_used_omap":1,"kb_used_meta":1114046,"kb_avail":92512212,"statfs":{"total":95995035648,"available":94732505088,"internally_reserved":0,"allocated":121737216,"data_stored":118551364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1140783557},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738765,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1021228,"kb_used_data":141796,"kb_used_omap":1,"kb_used_meta":879422,"kb_avail":92723924,"statfs":{"total":95995035648,"available":94949298176,"internally_reserved":0,"allocated":145199104,"data_stored":142013252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":900528581},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":160890880,"data_stored":160890880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":151519232,"data_stored":151519232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":330104832,"data_stored":330104832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":216924160,"data_stored":216924160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":136249344,"data_stored":136249344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":11599872,"data_stored":11599872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":105447424,"data_stored":105447424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":147980288,"data_stored":147980288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":221642752,"data_stored":221642752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":52363264,"data_stored":52363264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T14:40:17.729 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-08-19T14:40:17.730 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd pool set .mgr pg_num 91 2024-08-19T14:40:17.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:17 smithi032 ceph-mon[20004]: pgmap v1733: 47 pgs: 47 active+clean; 512 MiB data, 9.5 GiB used, 885 GiB / 894 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-08-19T14:40:17.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:17 smithi032 ceph-mon[20004]: from='client.17403 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:40:17.779 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:40:17.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:17 smithi045 ceph-mon[20143]: pgmap v1733: 47 pgs: 47 active+clean; 512 MiB data, 9.5 GiB used, 885 GiB / 894 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-08-19T14:40:17.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:17 smithi045 ceph-mon[20143]: from='client.17403 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:40:17.803 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:17.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:17.804 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:40:17.905 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:40:17.942 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:17.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:17.943 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:40:18.044 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:40:18.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:18.066 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:18.066 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:40:18.167 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:40:18.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:18.203 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:18.203 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:40:18.287 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 7 16 20841 20825 185.922 163.375 0.00325282 0.00537243 2024-08-19T14:40:18.304 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:40:18.320 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:40:18.332 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:18.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:18.333 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:40:18.434 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:40:18.477 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:18.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:18.478 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:40:18.579 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:40:18.611 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:18.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:18.613 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:40:18.714 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:40:18.741 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:18.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:18.742 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:40:18.843 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:40:18.878 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:18.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:18.879 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:40:18.980 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:40:19.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:19.013 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:19.013 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:40:19.114 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:40:19.146 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:19.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:19.147 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:40:19.248 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:40:19.269 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:19.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:19.271 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:40:19.287 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 8 16 23346 23330 182.25 156.562 0.00165079 0.00538053 2024-08-19T14:40:19.372 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:40:19.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:19.407 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:19.407 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:40:19.509 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:40:19.546 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:19.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:19.547 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:40:19.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:19 smithi031 ceph-mon[16336]: pgmap v1734: 47 pgs: 47 active+clean; 914 MiB data, 10 GiB used, 884 GiB / 894 GiB avail; 101 MiB/s wr, 1.61k op/s 2024-08-19T14:40:19.648 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:40:19.683 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:19.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:19.684 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:40:19.731 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-08-19T14:40:19.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:19 smithi032 ceph-mon[20004]: pgmap v1734: 47 pgs: 47 active+clean; 914 MiB data, 10 GiB used, 884 GiB / 894 GiB avail; 101 MiB/s wr, 1.61k op/s 2024-08-19T14:40:19.780 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:40:19.785 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:40:19.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:19 smithi045 ceph-mon[20143]: pgmap v1734: 47 pgs: 47 active+clean; 914 MiB data, 10 GiB used, 884 GiB / 894 GiB avail; 101 MiB/s wr, 1.61k op/s 2024-08-19T14:40:19.814 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:19.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:19.815 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:40:19.918 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:40:19.953 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:19.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:19.954 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:40:20.055 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:40:20.057 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:40:20.084 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:40:20.095 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:20.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:20.096 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:40:20.197 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:40:20.223 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:20.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:20.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:40:20.287 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 9 15 26223 26208 181.984 179.875 0.00442934 0.0054892 2024-08-19T14:40:20.325 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:40:20.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:20.351 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:20.351 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:40:20.383 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:40:20.453 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:40:20.482 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:20.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:20.483 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:40:20.584 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:40:20.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:20.618 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:20.619 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:40:20.720 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:40:20.755 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:20.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:20.756 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:40:20.857 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:40:20.892 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:20.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:20.893 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:40:20.994 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:40:21.021 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:21.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:21.022 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:40:21.123 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:40:21.156 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:21.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:21.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:40:21.258 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:40:21.287 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 10 16 28929 28913 180.691 169.062 0.00749665 0.00552995 2024-08-19T14:40:21.291 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:21.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:21.292 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:40:21.357 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is set 2024-08-19T14:40:21.393 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:40:21.415 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:21.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:21.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:40:21.486 INFO:teuthology.orchestra.run.smithi031.stderr:Error EINVAL: specified pg_num 91 < pg_num_max 32 2024-08-19T14:40:21.486 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:21 smithi031 ceph-mon[16336]: pgmap v1735: 47 pgs: 47 active+clean; 970 MiB data, 11 GiB used, 883 GiB / 894 GiB avail; 97 MiB/s wr, 1.55k op/s 2024-08-19T14:40:21.486 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:21 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2313574280' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:40:21.486 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:21 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:40:21.487 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:40:21.488 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-08-19T14:40:21.488 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:40:21.488 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26911.0:27582 8.5 8.a29ed955 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e722)", 2024-08-19T14:40:21.488 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:40:20.796244+0000", 2024-08-19T14:40:21.488 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 6.3705000000000001e-05, 2024-08-19T14:40:21.488 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.5096e-05, 2024-08-19T14:40:21.488 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:40:21.488 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:40:21.488 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "queued for pg", 2024-08-19T14:40:21.488 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:40:21.488 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26911", 2024-08-19T14:40:21.488 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3494993995", 2024-08-19T14:40:21.488 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 27582 2024-08-19T14:40:21.489 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:40:21.489 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:40:21.489 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:40:21.489 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:40:21.489 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:40:20.796244+0000", 2024-08-19T14:40:21.489 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:40:21.489 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:40:21.489 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:40:21.489 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:40:21.489 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:40:20.796244+0000", 2024-08-19T14:40:21.489 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:40:21.489 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:40:21.489 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:40:21.489 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:40:21.490 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:40:20.796245+0000", 2024-08-19T14:40:21.490 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5200000000000001e-06 2024-08-19T14:40:21.490 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:40:21.490 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:40:21.490 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:40:21.490 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:40:20.796276+0000", 2024-08-19T14:40:21.490 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1106999999999999e-05 2024-08-19T14:40:21.490 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:40:21.490 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:40:21.490 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:40:21.490 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:40:20.796292+0000", 2024-08-19T14:40:21.490 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5817000000000001e-05 2024-08-19T14:40:21.490 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:40:21.491 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:40:21.491 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:40:21.491 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:40:20.796300+0000", 2024-08-19T14:40:21.491 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.4519999999999997e-06 2024-08-19T14:40:21.491 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:40:21.491 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:40:21.491 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:40:21.491 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:40:21.491 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-08-19T14:40:21.491 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 1 2024-08-19T14:40:21.491 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:40:21.517 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:40:21.544 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:21.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:21.545 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:40:21.646 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:40:21.679 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:21.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:21.680 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:40:21.750 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:21 smithi032 ceph-mon[20004]: pgmap v1735: 47 pgs: 47 active+clean; 970 MiB data, 11 GiB used, 883 GiB / 894 GiB avail; 97 MiB/s wr, 1.55k op/s 2024-08-19T14:40:21.750 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:21 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2313574280' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:40:21.750 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:21 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:40:21.781 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:40:21.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:21 smithi045 ceph-mon[20143]: pgmap v1735: 47 pgs: 47 active+clean; 970 MiB data, 11 GiB used, 883 GiB / 894 GiB avail; 97 MiB/s wr, 1.55k op/s 2024-08-19T14:40:21.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:21 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2313574280' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:40:21.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:21 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:40:21.802 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:21.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:21.803 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:40:21.904 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:40:21.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:21.935 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:21.935 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:40:22.036 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:40:22.073 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:22.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:22.074 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:40:22.175 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:40:22.203 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:22.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:22.204 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:40:22.287 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 11 16 30931 30915 175.638 125.125 0.00409928 0.00568802 2024-08-19T14:40:22.305 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:40:22.339 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:22.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:22.340 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:40:22.441 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:40:22.469 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-08-19T14:40:22.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:22.478 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:22.478 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:40:22.535 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:22 smithi031 ceph-mon[16336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:40:22.535 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:22 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:40:22.536 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:22 smithi031 ceph-mon[16336]: osdmap e723: 12 total, 12 up, 10 in 2024-08-19T14:40:22.536 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:22 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3453105899' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "91"}]: dispatch 2024-08-19T14:40:22.536 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:22 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "91"}]: dispatch 2024-08-19T14:40:22.579 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:40:22.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:22.616 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:22.617 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:40:22.718 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:40:22.747 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:22.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:22.748 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:40:22.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:22 smithi032 ceph-mon[20004]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:40:22.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:22 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:40:22.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:22 smithi032 ceph-mon[20004]: osdmap e723: 12 total, 12 up, 10 in 2024-08-19T14:40:22.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:22 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3453105899' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "91"}]: dispatch 2024-08-19T14:40:22.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:22 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "91"}]: dispatch 2024-08-19T14:40:22.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:22 smithi045 ceph-mon[20143]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:40:22.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:22 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:40:22.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:22 smithi045 ceph-mon[20143]: osdmap e723: 12 total, 12 up, 10 in 2024-08-19T14:40:22.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:22 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3453105899' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "91"}]: dispatch 2024-08-19T14:40:22.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:22 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "91"}]: dispatch 2024-08-19T14:40:22.809 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:40:22.849 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:40:22.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:22.871 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:22.871 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:40:22.972 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:40:22.999 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:22.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:23.000 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:40:23.101 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:40:23.126 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:23.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:23.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:40:23.228 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:40:23.264 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:23.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:23.265 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:40:23.287 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 12 16 33190 33174 172.766 141.188 0.00635135 0.00578339 2024-08-19T14:40:23.366 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:40:23.397 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:23.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:23.398 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:40:23.499 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:40:23.529 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:40:23.529 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:40:23.536 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:23.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:23.537 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:40:23.566 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:40:23.566 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:40:23.600 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:40:23.600 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:40:23.631 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:40:23.631 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:40:23.638 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:40:23.664 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:40:23.664 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:40:23.672 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:23.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:23.673 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:40:23.695 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:40:23.695 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:40:23.728 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:40:23.728 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:40:23.760 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:40:23.761 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:40:23.773 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:40:23.795 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:40:23.795 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:40:23.800 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:23.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:23.801 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:40:23.829 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:40:23.830 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:40:23.865 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:40:23.865 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:40:23.902 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:40:23.906 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:40:23.906 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:40:23.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:23.934 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:23.934 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:40:23.952 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-19T14:40:23.963 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:40:23.963 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:40:24.035 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:40:24.070 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:24.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:24.071 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:40:24.172 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:40:24.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:24.210 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:24.210 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:40:24.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:24 smithi032 ceph-mon[20004]: pgmap v1737: 47 pgs: 47 active+clean; 1.6 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 164 MiB/s wr, 2.62k op/s 2024-08-19T14:40:24.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:24 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2406451959' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:40:24.287 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 13 16 35733 35717 171.702 158.938 0.00732135 0.0058191 2024-08-19T14:40:24.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:24 smithi045 ceph-mon[20143]: pgmap v1737: 47 pgs: 47 active+clean; 1.6 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 164 MiB/s wr, 2.62k op/s 2024-08-19T14:40:24.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:24 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2406451959' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:40:24.312 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:40:24.340 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:24 smithi031 ceph-mon[16336]: pgmap v1737: 47 pgs: 47 active+clean; 1.6 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 164 MiB/s wr, 2.62k op/s 2024-08-19T14:40:24.341 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:24 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2406451959' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:40:24.344 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:24.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:24.346 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:40:24.446 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:40:24.483 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:24.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:24.484 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:40:24.585 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:40:24.612 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-08-19T14:40:24.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:24.626 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:24.626 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:40:24.728 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:40:24.756 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:24.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:24.757 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:40:24.858 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:40:24.892 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:24.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:24.894 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:40:24.946 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:40:24.995 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:40:25.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:25.030 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:25.030 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:40:25.131 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:40:25.157 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:25.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:25.158 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:40:25.221 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-19T14:40:25.259 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:40:25.287 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 14 16 38364 38348 171.182 164.438 0.00314482 0.00583424 2024-08-19T14:40:25.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:25.290 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:25.290 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:40:25.391 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:40:25.417 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:25.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:25.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:40:25.519 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:40:25.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:25.542 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:25.542 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:40:25.567 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:40:25.593 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:40:25.593 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:40:25.644 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:40:25.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:25.680 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:25.680 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:40:25.782 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:40:25.808 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:25.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:25.809 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:40:25.910 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:40:25.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:25.933 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:25.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:40:26.033 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:40:26.057 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:26.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:26.058 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:40:26.159 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:40:26.188 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:26.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:26.190 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:40:26.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:26 smithi032 ceph-mon[20004]: pgmap v1738: 47 pgs: 47 active+clean; 1.8 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 170 MiB/s wr, 2.72k op/s 2024-08-19T14:40:26.287 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 15 16 39739 39723 165.499 85.9375 0.00326402 0.00603717 2024-08-19T14:40:26.291 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:40:26.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:26 smithi045 ceph-mon[20143]: pgmap v1738: 47 pgs: 47 active+clean; 1.8 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 170 MiB/s wr, 2.72k op/s 2024-08-19T14:40:26.325 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:26.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:26.326 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:40:26.335 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-08-19T14:40:26.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:26 smithi031 ceph-mon[16336]: pgmap v1738: 47 pgs: 47 active+clean; 1.8 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 170 MiB/s wr, 2.72k op/s 2024-08-19T14:40:26.427 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:40:26.461 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:26.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:26.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:40:26.564 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:40:26.592 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:26.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:26.593 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:40:26.633 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:40:26.653 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:40:26.694 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:40:26.733 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:26.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:26.735 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:40:26.836 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:40:26.862 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:26.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:26.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:40:26.964 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:40:26.997 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:26.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:26.998 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:40:27.099 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:40:27.104 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is unset 2024-08-19T14:40:27.126 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:27.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:27.127 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:40:27.187 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:27 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/319382532' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:40:27.228 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:40:27.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:27.264 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:27.265 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:40:27.287 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 16 16 41897 41881 163.584 134.875 0.00516984 0.00610861 2024-08-19T14:40:27.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:27 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/319382532' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:40:27.325 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:40:27.326 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:40:27.365 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:40:27.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:27 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/319382532' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:40:27.400 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:27.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:27.401 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:40:27.502 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:40:27.537 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:27.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:27.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:40:27.639 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:40:27.673 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:27.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:27.678 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:40:27.779 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:40:27.801 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:27.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:27.802 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:40:27.903 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:40:27.942 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:27.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:27.943 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:40:27.991 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-08-19T14:40:28.043 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:40:28.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:28.065 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:28.065 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:40:28.166 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:40:28.195 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:28.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:28.196 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:40:28.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:28 smithi032 ceph-mon[20004]: pgmap v1739: 47 pgs: 47 active+clean; 2.1 GiB data, 15 GiB used, 880 GiB / 894 GiB avail; 167 MiB/s wr, 2.67k op/s 2024-08-19T14:40:28.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:28 smithi032 ceph-mon[20004]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:40:28.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:28 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/319382532' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:40:28.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:28 smithi032 ceph-mon[20004]: osdmap e724: 12 total, 12 up, 10 in 2024-08-19T14:40:28.269 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:40:28.288 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 17 16 43704 43688 160.604 112.938 0.00253866 0.006211 2024-08-19T14:40:28.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:28 smithi045 ceph-mon[20143]: pgmap v1739: 47 pgs: 47 active+clean; 2.1 GiB data, 15 GiB used, 880 GiB / 894 GiB avail; 167 MiB/s wr, 2.67k op/s 2024-08-19T14:40:28.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:28 smithi045 ceph-mon[20143]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:40:28.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:28 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/319382532' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:40:28.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:28 smithi045 ceph-mon[20143]: osdmap e724: 12 total, 12 up, 10 in 2024-08-19T14:40:28.297 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:40:28.318 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:28.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:28.319 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:40:28.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:28 smithi031 ceph-mon[16336]: pgmap v1739: 47 pgs: 47 active+clean; 2.1 GiB data, 15 GiB used, 880 GiB / 894 GiB avail; 167 MiB/s wr, 2.67k op/s 2024-08-19T14:40:28.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:28 smithi031 ceph-mon[16336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:40:28.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:28 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/319382532' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:40:28.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:28 smithi031 ceph-mon[16336]: osdmap e724: 12 total, 12 up, 10 in 2024-08-19T14:40:28.419 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:40:28.445 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:28.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:28.446 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:40:28.547 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:40:28.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:28.574 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:28.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:40:28.676 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:40:28.703 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:28.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:28.705 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:40:28.806 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:40:28.827 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:28.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:28.828 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:40:28.870 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:40:28.870 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:40:28.929 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:40:28.953 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 7, 8, 2, 11, 9] out_osds: [10, 6] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 8, 9, 11, 10, 2] 2024-08-19T14:40:28.954 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-19T14:40:28.954 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_6 2024-08-19T14:40:28.954 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-19T14:40:28.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:28.955 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:28.956 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:40:29.056 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:40:29.079 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:29.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:29.080 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:40:29.181 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:40:29.202 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:29.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:29.204 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:40:29.288 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 18 16 45232 45216 156.987 95.5 0.010408 0.00636563 2024-08-19T14:40:29.304 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:40:29.338 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:29.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:29.340 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:40:29.441 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:40:29.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:29.463 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:29.463 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:40:29.563 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:40:29.603 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:29.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:29.604 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:40:29.704 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:40:29.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:29.726 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:29.726 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:40:29.827 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:40:29.864 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:29.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:29.865 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:40:29.966 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:40:29.991 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:29.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:29.993 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:40:30.094 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:40:30.123 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:30.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:30.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:40:30.224 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:40:30.251 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:30.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:30.252 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:40:30.288 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 19 16 47202 47186 155.204 123.125 0.0027511 0.00643823 2024-08-19T14:40:30.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:30 smithi045 ceph-mon[20143]: pgmap v1741: 47 pgs: 47 active+clean; 2.4 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 189 MiB/s wr, 3.02k op/s 2024-08-19T14:40:30.352 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:40:30.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:30 smithi031 ceph-mon[16336]: pgmap v1741: 47 pgs: 47 active+clean; 2.4 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 189 MiB/s wr, 3.02k op/s 2024-08-19T14:40:30.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:30.380 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:30.380 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:40:30.455 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:30 smithi032 ceph-mon[20004]: pgmap v1741: 47 pgs: 47 active+clean; 2.4 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 189 MiB/s wr, 3.02k op/s 2024-08-19T14:40:30.481 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:40:30.496 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-08-19T14:40:30.507 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:30.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:30.508 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:40:30.608 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:40:30.642 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:30.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:30.643 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:40:30.744 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:40:30.777 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:30.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:30.778 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:40:30.821 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:40:30.879 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:40:30.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:30.913 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:30.913 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:40:31.015 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:40:31.051 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:31.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:31.052 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:40:31.153 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:40:31.178 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:31.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:31.179 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:40:31.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:31 smithi032 ceph-mon[20004]: 1.17 scrub starts 2024-08-19T14:40:31.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:31 smithi032 ceph-mon[20004]: 1.17 scrub ok 2024-08-19T14:40:31.280 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:40:31.288 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:2024-08-19T14:40:31.288437+0000 min lat: 0.00108187 max lat: 0.238328 avg lat: 0.00652846 2024-08-19T14:40:31.288 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-19T14:40:31.288 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 20 16 49000 48984 153.062 112.375 0.0105228 0.00652846 2024-08-19T14:40:31.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:31 smithi045 ceph-mon[20143]: 1.17 scrub starts 2024-08-19T14:40:31.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:31 smithi045 ceph-mon[20143]: 1.17 scrub ok 2024-08-19T14:40:31.305 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:31.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:31.306 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:40:31.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:31 smithi031 ceph-mon[16336]: 1.17 scrub starts 2024-08-19T14:40:31.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:31 smithi031 ceph-mon[16336]: 1.17 scrub ok 2024-08-19T14:40:31.406 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:40:31.438 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:31.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:31.439 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:40:31.540 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:40:31.573 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:31.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:31.574 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:40:31.675 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:40:31.697 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:31.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:31.698 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:40:31.799 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:40:31.820 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:31.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:31.821 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:40:31.849 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:40:31.849 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:40:31.921 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:40:31.948 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:31.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:31.948 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:40:32.050 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:40:32.071 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:32.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:32.072 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:40:32.174 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:40:32.194 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:32.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:32.195 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:40:32.288 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 21 16 50272 50256 149.559 79.5 0.00984633 0.0066814 2024-08-19T14:40:32.296 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:40:32.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:32 smithi045 ceph-mon[20143]: pgmap v1742: 47 pgs: 47 active+clean; 2.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 124 MiB/s wr, 1.99k op/s 2024-08-19T14:40:32.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:32 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:40:32.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:32 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:40:32.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:32 smithi045 ceph-mon[20143]: 1.1c scrub starts 2024-08-19T14:40:32.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:32 smithi045 ceph-mon[20143]: 1.1c scrub ok 2024-08-19T14:40:32.366 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:32.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:32 smithi031 ceph-mon[16336]: pgmap v1742: 47 pgs: 47 active+clean; 2.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 124 MiB/s wr, 1.99k op/s 2024-08-19T14:40:32.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:32 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:40:32.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:32 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:40:32.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:32 smithi031 ceph-mon[16336]: 1.1c scrub starts 2024-08-19T14:40:32.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:32 smithi031 ceph-mon[16336]: 1.1c scrub ok 2024-08-19T14:40:32.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:32.368 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:40:32.458 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:32 smithi032 ceph-mon[20004]: pgmap v1742: 47 pgs: 47 active+clean; 2.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 124 MiB/s wr, 1.99k op/s 2024-08-19T14:40:32.458 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:32 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:40:32.459 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:32 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:40:32.459 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:32 smithi032 ceph-mon[20004]: 1.1c scrub starts 2024-08-19T14:40:32.459 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:32 smithi032 ceph-mon[20004]: 1.1c scrub ok 2024-08-19T14:40:32.468 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:40:32.495 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-08-19T14:40:32.501 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:32.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:32.502 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:40:32.603 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:40:32.637 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:32.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:32.638 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:40:32.739 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:40:32.774 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:32.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:32.775 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:40:32.817 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:40:32.876 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:40:32.898 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:32.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:32.899 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:40:33.000 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:40:33.035 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:33.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:33.036 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:40:33.137 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:40:33.168 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:33.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:33.170 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:40:33.220 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:33 smithi032 ceph-mon[20004]: 1.1e scrub starts 2024-08-19T14:40:33.221 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:33 smithi032 ceph-mon[20004]: 1.1e scrub ok 2024-08-19T14:40:33.221 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:33 smithi032 ceph-mon[20004]: 1.10 scrub starts 2024-08-19T14:40:33.221 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:33 smithi032 ceph-mon[20004]: 1.10 scrub ok 2024-08-19T14:40:33.221 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:33 smithi032 ceph-mon[20004]: osdmap e725: 12 total, 12 up, 10 in 2024-08-19T14:40:33.221 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:33 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:40:33.221 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:33 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:40:33.271 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:40:33.288 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 22 16 52119 52103 148.007 115.438 0.0110793 0.0067516 2024-08-19T14:40:33.310 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:33.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:33.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:40:33.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:33 smithi031 ceph-mon[16336]: 1.1e scrub starts 2024-08-19T14:40:33.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:33 smithi031 ceph-mon[16336]: 1.1e scrub ok 2024-08-19T14:40:33.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:33 smithi031 ceph-mon[16336]: 1.10 scrub starts 2024-08-19T14:40:33.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:33 smithi031 ceph-mon[16336]: 1.10 scrub ok 2024-08-19T14:40:33.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:33 smithi031 ceph-mon[16336]: osdmap e725: 12 total, 12 up, 10 in 2024-08-19T14:40:33.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:33 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:40:33.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:33 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:40:33.367 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:40:33.367 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:40:33.412 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:40:33.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:33.440 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:33.440 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:40:33.542 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:40:33.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:33 smithi045 ceph-mon[20143]: 1.1e scrub starts 2024-08-19T14:40:33.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:33 smithi045 ceph-mon[20143]: 1.1e scrub ok 2024-08-19T14:40:33.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:33 smithi045 ceph-mon[20143]: 1.10 scrub starts 2024-08-19T14:40:33.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:33 smithi045 ceph-mon[20143]: 1.10 scrub ok 2024-08-19T14:40:33.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:33 smithi045 ceph-mon[20143]: osdmap e725: 12 total, 12 up, 10 in 2024-08-19T14:40:33.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:33 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:40:33.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:33 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:40:33.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:33.568 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:33.568 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:40:33.669 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:40:33.696 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:33.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:33.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:40:33.798 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:40:33.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:33.828 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:33.828 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:40:33.908 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:40:33.909 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:40:33.929 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:40:33.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:33.951 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:33.951 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:40:34.052 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:40:34.061 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-08-19T14:40:34.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:34.083 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:34.084 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:40:34.184 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:40:34.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:34.222 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:34.223 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:40:34.282 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:34 smithi032 ceph-mon[20004]: pgmap v1744: 47 pgs: 47 active+clean; 2.9 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 146 MiB/s wr, 2.34k op/s 2024-08-19T14:40:34.282 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:34 smithi032 ceph-mon[20004]: osdmap e726: 12 total, 12 up, 10 in 2024-08-19T14:40:34.283 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:34 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:40:34.288 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 23 16 53756 53740 146.02 102.312 0.0154479 0.00684394 2024-08-19T14:40:34.323 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:40:34.357 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:34.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:34.358 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:40:34.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:34 smithi031 ceph-mon[16336]: pgmap v1744: 47 pgs: 47 active+clean; 2.9 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 146 MiB/s wr, 2.34k op/s 2024-08-19T14:40:34.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:34 smithi031 ceph-mon[16336]: osdmap e726: 12 total, 12 up, 10 in 2024-08-19T14:40:34.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:34 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:40:34.459 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:40:34.486 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:34.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:34.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:40:34.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:34 smithi045 ceph-mon[20143]: pgmap v1744: 47 pgs: 47 active+clean; 2.9 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 146 MiB/s wr, 2.34k op/s 2024-08-19T14:40:34.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:34 smithi045 ceph-mon[20143]: osdmap e726: 12 total, 12 up, 10 in 2024-08-19T14:40:34.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:34 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:40:34.588 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:40:34.612 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:34.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:34.613 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:40:34.714 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:40:34.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:34.753 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:34.753 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:40:34.854 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:40:34.883 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:34.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:34.885 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:40:34.986 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:40:35.017 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:35.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:35.018 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:40:35.042 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:40:35.042 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:40:35.088 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:40:35.088 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:40:35.092 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:40:35.093 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-08-19T14:40:35.119 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:40:35.127 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:40:35.128 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:40:35.170 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:40:35.171 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:40:35.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:35.182 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:35.183 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:40:35.203 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:40:35.203 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:40:35.238 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:40:35.239 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:40:35.272 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:40:35.272 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:40:35.284 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:40:35.288 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 24 15 55603 55588 144.748 115.5 0.00676993 0.00690374 2024-08-19T14:40:35.304 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:40:35.304 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:40:35.320 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:35.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:35.321 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:40:35.338 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:35 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:40:35.338 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:35 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:40:35.338 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:35 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2133218029' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:40:35.339 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:40:35.339 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:40:35.375 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:40:35.375 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:40:35.404 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:35 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:40:35.404 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:35 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:40:35.404 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:35 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2133218029' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:40:35.411 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:40:35.411 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:40:35.422 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:40:35.449 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:40:35.450 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:40:35.454 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:35.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:35.455 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:40:35.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:35 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:40:35.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:35 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:40:35.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:35 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2133218029' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:40:35.556 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:40:35.590 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:35.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:35.591 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:40:35.691 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:40:35.729 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:35.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:35.730 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:40:35.831 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:40:35.870 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:35.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:35.872 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:40:35.882 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-19T14:40:35.972 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:40:35.999 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:36.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:36.000 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:40:36.001 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:40:36.101 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:40:36.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:36.131 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:36.132 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:40:36.233 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:40:36.280 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:36.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:36.281 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:40:36.288 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 25 16 57441 57425 143.55 114.812 0.00645988 0.0069614 2024-08-19T14:40:36.376 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:36 smithi032 ceph-mon[20004]: pgmap v1746: 47 pgs: 47 active+clean; 3.0 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 94 MiB/s wr, 1.50k op/s 2024-08-19T14:40:36.376 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:36 smithi032 ceph-mon[20004]: from='client.17427 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:40:36.376 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:36 smithi032 ceph-mon[20004]: osdmap e727: 12 total, 12 up, 10 in 2024-08-19T14:40:36.382 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:40:36.406 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:36.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:36.407 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:40:36.508 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:40:36.529 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:36.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:36.530 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:40:36.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:36 smithi045 ceph-mon[20143]: pgmap v1746: 47 pgs: 47 active+clean; 3.0 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 94 MiB/s wr, 1.50k op/s 2024-08-19T14:40:36.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:36 smithi045 ceph-mon[20143]: from='client.17427 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:40:36.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:36 smithi045 ceph-mon[20143]: osdmap e727: 12 total, 12 up, 10 in 2024-08-19T14:40:36.606 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:36 smithi031 ceph-mon[16336]: pgmap v1746: 47 pgs: 47 active+clean; 3.0 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 94 MiB/s wr, 1.50k op/s 2024-08-19T14:40:36.606 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:36 smithi031 ceph-mon[16336]: from='client.17427 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:40:36.607 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:36 smithi031 ceph-mon[16336]: osdmap e727: 12 total, 12 up, 10 in 2024-08-19T14:40:36.631 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:40:36.633 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":1746,"stamp":"2024-08-19T14:40:34.286046+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3228107296,"num_objects":49250,"num_object_clones":0,"num_object_copies":147750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49248,"num_write_kb":3151872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":856608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53557,"ondisk_log_size":53557,"up":141,"acting":141,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":141,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":18324400,"kb_used_data":9395924,"kb_used_omap":18,"kb_used_meta":8928365,"kb_avail":919127120,"statfs":{"total":959950356480,"available":941186170880,"internally_reserved":0,"allocated":9621426176,"data_stored":9589459285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18760,"internal_metadata":9142646456},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":714473472,"num_objects":10902,"num_object_clones":0,"num_object_copies":32706,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10902,"num_write_kb":697728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.253136"},"pg_stats":[{"pgid":"8.9","version":"724'3028","reported_seq":3052,"reported_epoch":724,"state":"active+clean","last_fresh":"2024-08-19T14:40:30.749786+0000","last_change":"2024-08-19T14:40:04.442466+0000","last_active":"2024-08-19T14:40:30.749786+0000","last_peered":"2024-08-19T14:40:30.749786+0000","last_clean":"2024-08-19T14:40:30.749786+0000","last_became_active":"2024-08-19T14:40:04.442173+0000","last_became_peered":"2024-08-19T14:40:04.442173+0000","last_unstale":"2024-08-19T14:40:30.749786+0000","last_undegraded":"2024-08-19T14:40:30.749786+0000","last_fullsized":"2024-08-19T14:40:30.749786+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":717,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.182354+0000","objects_scrubbed":0,"log_size":3028,"log_dups_size":0,"ondisk_log_size":3028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:25.949039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198443008,"num_objects":3028,"num_object_clones":0,"num_object_copies":9084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3028,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3028,"num_write_kb":193792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"70'139","reported_seq":1776,"reported_epoch":724,"state":"active+clean","last_fresh":"2024-08-19T14:40:27.039433+0000","last_change":"2024-08-19T14:39:45.569137+0000","last_active":"2024-08-19T14:40:27.039433+0000","last_peered":"2024-08-19T14:40:27.039433+0000","last_clean":"2024-08-19T14:40:27.039433+0000","last_became_active":"2024-08-19T14:35:13.950224+0000","last_became_peered":"2024-08-19T14:35:13.950224+0000","last_unstale":"2024-08-19T14:40:27.039433+0000","last_undegraded":"2024-08-19T14:40:27.039433+0000","last_fullsized":"2024-08-19T14:40:27.039433+0000","mapping_epoch":637,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":638,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:39:45.569030+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:19:58.072083+0000","last_clean_scrub_stamp":"2024-08-19T14:39:45.569030+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:40:57.206713+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,9],"acting":[0,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"724'2920","reported_seq":2943,"reported_epoch":724,"state":"active+clean","last_fresh":"2024-08-19T14:40:30.362436+0000","last_change":"2024-08-19T14:40:04.442002+0000","last_active":"2024-08-19T14:40:30.362436+0000","last_peered":"2024-08-19T14:40:30.362436+0000","last_clean":"2024-08-19T14:40:30.362436+0000","last_became_active":"2024-08-19T14:40:04.441586+0000","last_became_peered":"2024-08-19T14:40:04.441586+0000","last_unstale":"2024-08-19T14:40:30.362436+0000","last_undegraded":"2024-08-19T14:40:30.362436+0000","last_fullsized":"2024-08-19T14:40:30.362436+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":717,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.182354+0000","objects_scrubbed":0,"log_size":2920,"log_dups_size":0,"ondisk_log_size":2920,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:26.482555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":191365120,"num_objects":2920,"num_object_clones":0,"num_object_copies":8760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2920,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2920,"num_write_kb":186880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,9],"acting":[8,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"70'139","reported_seq":1622,"reported_epoch":725,"state":"active+clean","last_fresh":"2024-08-19T14:40:32.111640+0000","last_change":"2024-08-19T14:40:02.681316+0000","last_active":"2024-08-19T14:40:32.111640+0000","last_peered":"2024-08-19T14:40:32.111640+0000","last_clean":"2024-08-19T14:40:32.111640+0000","last_became_active":"2024-08-19T14:18:13.804721+0000","last_became_peered":"2024-08-19T14:18:13.804721+0000","last_unstale":"2024-08-19T14:40:32.111640+0000","last_undegraded":"2024-08-19T14:40:32.111640+0000","last_fullsized":"2024-08-19T14:40:32.111640+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:40:02.681185+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:40:02.681185+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:22.300124+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.b","version":"724'2921","reported_seq":2945,"reported_epoch":724,"state":"active+clean","last_fresh":"2024-08-19T14:40:30.350647+0000","last_change":"2024-08-19T14:40:04.200633+0000","last_active":"2024-08-19T14:40:30.350647+0000","last_peered":"2024-08-19T14:40:30.350647+0000","last_clean":"2024-08-19T14:40:30.350647+0000","last_became_active":"2024-08-19T14:40:04.200299+0000","last_became_peered":"2024-08-19T14:40:04.200299+0000","last_unstale":"2024-08-19T14:40:30.350647+0000","last_undegraded":"2024-08-19T14:40:30.350647+0000","last_fullsized":"2024-08-19T14:40:30.350647+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":717,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.182354+0000","objects_scrubbed":0,"log_size":2921,"log_dups_size":0,"ondisk_log_size":2921,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:27.707935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":191430656,"num_objects":2921,"num_object_clones":0,"num_object_copies":8763,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2921,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2921,"num_write_kb":186944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,1],"acting":[8,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.2","version":"70'139","reported_seq":1576,"reported_epoch":724,"state":"active+clean","last_fresh":"2024-08-19T14:40:27.041307+0000","last_change":"2024-08-19T14:40:10.977253+0000","last_active":"2024-08-19T14:40:27.041307+0000","last_peered":"2024-08-19T14:40:27.041307+0000","last_clean":"2024-08-19T14:40:27.041307+0000","last_became_active":"2024-08-19T14:20:50.742354+0000","last_became_peered":"2024-08-19T14:20:50.742354+0000","last_unstale":"2024-08-19T14:40:27.041307+0000","last_undegraded":"2024-08-19T14:40:27.041307+0000","last_fullsized":"2024-08-19T14:40:27.041307+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:40:10.977156+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:19:59.302931+0000","last_clean_scrub_stamp":"2024-08-19T14:40:10.977156+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:39.144591+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.8","version":"724'3061","reported_seq":3085,"reported_epoch":724,"state":"active+clean","last_fresh":"2024-08-19T14:40:30.754077+0000","last_change":"2024-08-19T14:40:04.443203+0000","last_active":"2024-08-19T14:40:30.754077+0000","last_peered":"2024-08-19T14:40:30.754077+0000","last_clean":"2024-08-19T14:40:30.754077+0000","last_became_active":"2024-08-19T14:40:04.442802+0000","last_became_peered":"2024-08-19T14:40:04.442802+0000","last_unstale":"2024-08-19T14:40:30.754077+0000","last_undegraded":"2024-08-19T14:40:30.754077+0000","last_fullsized":"2024-08-19T14:40:30.754077+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":717,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.182354+0000","objects_scrubbed":0,"log_size":3061,"log_dups_size":0,"ondisk_log_size":3061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:14.120759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":200605696,"num_objects":3061,"num_object_clones":0,"num_object_copies":9183,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3061,"num_write_kb":195904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,9],"acting":[3,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"70'139","reported_seq":874,"reported_epoch":724,"state":"active+clean","last_fresh":"2024-08-19T14:40:27.040182+0000","last_change":"2024-08-19T14:39:37.543259+0000","last_active":"2024-08-19T14:40:27.040182+0000","last_peered":"2024-08-19T14:40:27.040182+0000","last_clean":"2024-08-19T14:40:27.040182+0000","last_became_active":"2024-08-19T14:35:12.530566+0000","last_became_peered":"2024-08-19T14:35:12.530566+0000","last_unstale":"2024-08-19T14:40:27.040182+0000","last_undegraded":"2024-08-19T14:40:27.040182+0000","last_fullsized":"2024-08-19T14:40:27.040182+0000","mapping_epoch":635,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":636,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:39:37.543187+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:39:37.543187+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:40:56.083490+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,2],"acting":[0,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.e","version":"724'2966","reported_seq":2990,"reported_epoch":724,"state":"active+clean","last_fresh":"2024-08-19T14:40:30.756336+0000","last_change":"2024-08-19T14:40:04.299744+0000","last_active":"2024-08-19T14:40:30.756336+0000","last_peered":"2024-08-19T14:40:30.756336+0000","last_clean":"2024-08-19T14:40:30.756336+0000","last_became_active":"2024-08-19T14:40:04.299403+0000","last_became_peered":"2024-08-19T14:40:04.299403+0000","last_unstale":"2024-08-19T14:40:30.756336+0000","last_undegraded":"2024-08-19T14:40:30.756336+0000","last_fullsized":"2024-08-19T14:40:30.756336+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":717,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.182354+0000","objects_scrubbed":0,"log_size":2966,"log_dups_size":0,"ondisk_log_size":2966,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:22.448115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":194379776,"num_objects":2966,"num_object_clones":0,"num_object_copies":8898,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2966,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2966,"num_write_kb":189824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"70'139","reported_seq":1278,"reported_epoch":724,"state":"active+clean","last_fresh":"2024-08-19T14:40:27.037220+0000","last_change":"2024-08-19T14:39:56.993699+0000","last_active":"2024-08-19T14:40:27.037220+0000","last_peered":"2024-08-19T14:40:27.037220+0000","last_clean":"2024-08-19T14:40:27.037220+0000","last_became_active":"2024-08-19T14:35:11.536828+0000","last_became_peered":"2024-08-19T14:35:11.536828+0000","last_unstale":"2024-08-19T14:40:27.037220+0000","last_undegraded":"2024-08-19T14:40:27.037220+0000","last_fullsized":"2024-08-19T14:40:27.037220+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:39:56.993616+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:21:46.057624+0000","last_clean_scrub_stamp":"2024-08-19T14:39:56.993616+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:05.362216+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.f","version":"724'3167","reported_seq":3191,"reported_epoch":724,"state":"active+clean","last_fresh":"2024-08-19T14:40:31.676106+0000","last_change":"2024-08-19T14:40:04.299935+0000","last_active":"2024-08-19T14:40:31.676106+0000","last_peered":"2024-08-19T14:40:31.676106+0000","last_clean":"2024-08-19T14:40:31.676106+0000","last_became_active":"2024-08-19T14:40:04.299656+0000","last_became_peered":"2024-08-19T14:40:04.299656+0000","last_unstale":"2024-08-19T14:40:31.676106+0000","last_undegraded":"2024-08-19T14:40:31.676106+0000","last_fullsized":"2024-08-19T14:40:31.676106+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":717,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.182354+0000","objects_scrubbed":0,"log_size":3167,"log_dups_size":0,"ondisk_log_size":3167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:22.129102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":207552512,"num_objects":3167,"num_object_clones":0,"num_object_copies":9501,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3167,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3167,"num_write_kb":202688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,1],"acting":[11,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.6","version":"70'139","reported_seq":846,"reported_epoch":724,"state":"active+clean","last_fresh":"2024-08-19T14:40:27.036675+0000","last_change":"2024-08-19T14:39:30.460768+0000","last_active":"2024-08-19T14:40:27.036675+0000","last_peered":"2024-08-19T14:40:27.036675+0000","last_clean":"2024-08-19T14:40:27.036675+0000","last_became_active":"2024-08-19T14:34:15.717054+0000","last_became_peered":"2024-08-19T14:34:15.717054+0000","last_unstale":"2024-08-19T14:40:27.036675+0000","last_undegraded":"2024-08-19T14:40:27.036675+0000","last_fullsized":"2024-08-19T14:40:27.036675+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:39:30.460649+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:27.848870+0000","last_clean_scrub_stamp":"2024-08-19T14:39:30.460649+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:40:59.351491+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.c","version":"724'3016","reported_seq":3040,"reported_epoch":724,"state":"active+clean","last_fresh":"2024-08-19T14:40:30.332546+0000","last_change":"2024-08-19T14:40:04.442056+0000","last_active":"2024-08-19T14:40:30.332546+0000","last_peered":"2024-08-19T14:40:30.332546+0000","last_clean":"2024-08-19T14:40:30.332546+0000","last_became_active":"2024-08-19T14:40:04.441602+0000","last_became_peered":"2024-08-19T14:40:04.441602+0000","last_unstale":"2024-08-19T14:40:30.332546+0000","last_undegraded":"2024-08-19T14:40:30.332546+0000","last_fullsized":"2024-08-19T14:40:30.332546+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":717,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.182354+0000","objects_scrubbed":0,"log_size":3016,"log_dups_size":0,"ondisk_log_size":3016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:20.409556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197656576,"num_objects":3016,"num_object_clones":0,"num_object_copies":9048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3016,"num_write_kb":193024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,3],"acting":[8,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.5","version":"70'139","reported_seq":800,"reported_epoch":724,"state":"active+clean","last_fresh":"2024-08-19T14:40:27.039467+0000","last_change":"2024-08-19T14:39:54.886149+0000","last_active":"2024-08-19T14:40:27.039467+0000","last_peered":"2024-08-19T14:40:27.039467+0000","last_clean":"2024-08-19T14:40:27.039467+0000","last_became_active":"2024-08-19T14:35:11.532858+0000","last_became_peered":"2024-08-19T14:35:11.532858+0000","last_unstale":"2024-08-19T14:40:27.039467+0000","last_undegraded":"2024-08-19T14:40:27.039467+0000","last_fullsized":"2024-08-19T14:40:27.039467+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:39:54.886056+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:39:54.886056+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:40:57.762183+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,0],"acting":[11,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.d","version":"724'3143","reported_seq":3167,"reported_epoch":724,"state":"active+clean","last_fresh":"2024-08-19T14:40:31.691501+0000","last_change":"2024-08-19T14:40:04.443252+0000","last_active":"2024-08-19T14:40:31.691501+0000","last_peered":"2024-08-19T14:40:31.691501+0000","last_clean":"2024-08-19T14:40:31.691501+0000","last_became_active":"2024-08-19T14:40:04.442093+0000","last_became_peered":"2024-08-19T14:40:04.442093+0000","last_unstale":"2024-08-19T14:40:31.691501+0000","last_undegraded":"2024-08-19T14:40:31.691501+0000","last_fullsized":"2024-08-19T14:40:31.691501+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":717,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.182354+0000","objects_scrubbed":0,"log_size":3143,"log_dups_size":0,"ondisk_log_size":3143,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:22.854336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205979648,"num_objects":3143,"num_object_clones":0,"num_object_copies":9429,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3143,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3143,"num_write_kb":201152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,2],"acting":[11,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.4","version":"70'139","reported_seq":677,"reported_epoch":724,"state":"active+clean","last_fresh":"2024-08-19T14:40:27.037076+0000","last_change":"2024-08-19T14:40:09.799125+0000","last_active":"2024-08-19T14:40:27.037076+0000","last_peered":"2024-08-19T14:40:27.037076+0000","last_clean":"2024-08-19T14:40:27.037076+0000","last_became_active":"2024-08-19T14:30:47.909067+0000","last_became_peered":"2024-08-19T14:30:47.909067+0000","last_unstale":"2024-08-19T14:40:27.037076+0000","last_undegraded":"2024-08-19T14:40:27.037076+0000","last_fullsized":"2024-08-19T14:40:27.037076+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:40:09.798994+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:40:09.798994+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:32.468916+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.3","version":"724'3136","reported_seq":3160,"reported_epoch":724,"state":"active+clean","last_fresh":"2024-08-19T14:40:31.686153+0000","last_change":"2024-08-19T14:40:04.443269+0000","last_active":"2024-08-19T14:40:31.686153+0000","last_peered":"2024-08-19T14:40:31.686153+0000","last_clean":"2024-08-19T14:40:31.686153+0000","last_became_active":"2024-08-19T14:40:04.442837+0000","last_became_peered":"2024-08-19T14:40:04.442837+0000","last_unstale":"2024-08-19T14:40:31.686153+0000","last_undegraded":"2024-08-19T14:40:31.686153+0000","last_fullsized":"2024-08-19T14:40:31.686153+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":717,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.182354+0000","objects_scrubbed":0,"log_size":3136,"log_dups_size":0,"ondisk_log_size":3136,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:13.689400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205520896,"num_objects":3136,"num_object_clones":0,"num_object_copies":9408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3136,"num_write_kb":200704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.a","version":"70'139","reported_seq":847,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-08-19T14:40:33.117991+0000","last_change":"2024-08-19T14:40:05.713501+0000","last_active":"2024-08-19T14:40:33.117991+0000","last_peered":"2024-08-19T14:40:33.117991+0000","last_clean":"2024-08-19T14:40:33.117991+0000","last_became_active":"2024-08-19T14:34:15.717472+0000","last_became_peered":"2024-08-19T14:34:15.717472+0000","last_unstale":"2024-08-19T14:40:33.117991+0000","last_undegraded":"2024-08-19T14:40:33.117991+0000","last_fullsized":"2024-08-19T14:40:33.117991+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:40:05.713411+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:28.864870+0000","last_clean_scrub_stamp":"2024-08-19T14:40:05.713411+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:12.701795+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,0],"acting":[4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.0","version":"724'3007","reported_seq":3031,"reported_epoch":724,"state":"active+clean","last_fresh":"2024-08-19T14:40:30.782489+0000","last_change":"2024-08-19T14:40:04.443641+0000","last_active":"2024-08-19T14:40:30.782489+0000","last_peered":"2024-08-19T14:40:30.782489+0000","last_clean":"2024-08-19T14:40:30.782489+0000","last_became_active":"2024-08-19T14:40:04.443254+0000","last_became_peered":"2024-08-19T14:40:04.443254+0000","last_unstale":"2024-08-19T14:40:30.782489+0000","last_undegraded":"2024-08-19T14:40:30.782489+0000","last_fullsized":"2024-08-19T14:40:30.782489+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":717,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.182354+0000","objects_scrubbed":0,"log_size":3007,"log_dups_size":0,"ondisk_log_size":3007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:21.728211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197066752,"num_objects":3007,"num_object_clones":0,"num_object_copies":9021,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3007,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3007,"num_write_kb":192448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,9],"acting":[3,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"70'139","reported_seq":1778,"reported_epoch":724,"state":"active+clean","last_fresh":"2024-08-19T14:40:27.041402+0000","last_change":"2024-08-19T14:40:05.923941+0000","last_active":"2024-08-19T14:40:27.041402+0000","last_peered":"2024-08-19T14:40:27.041402+0000","last_clean":"2024-08-19T14:40:27.041402+0000","last_became_active":"2024-08-19T14:35:12.529628+0000","last_became_peered":"2024-08-19T14:35:12.529628+0000","last_unstale":"2024-08-19T14:40:27.041402+0000","last_undegraded":"2024-08-19T14:40:27.041402+0000","last_fullsized":"2024-08-19T14:40:27.041402+0000","mapping_epoch":635,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":636,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:40:05.923850+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:40:05.923850+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:22.262135+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1","version":"725'3282","reported_seq":3308,"reported_epoch":725,"state":"active+clean","last_fresh":"2024-08-19T14:40:32.950495+0000","last_change":"2024-08-19T14:40:04.441532+0000","last_active":"2024-08-19T14:40:32.950495+0000","last_peered":"2024-08-19T14:40:32.950495+0000","last_clean":"2024-08-19T14:40:32.950495+0000","last_became_active":"2024-08-19T14:40:04.441139+0000","last_became_peered":"2024-08-19T14:40:04.441139+0000","last_unstale":"2024-08-19T14:40:32.950495+0000","last_undegraded":"2024-08-19T14:40:32.950495+0000","last_fullsized":"2024-08-19T14:40:32.950495+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":717,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.182354+0000","objects_scrubbed":0,"log_size":3282,"log_dups_size":0,"ondisk_log_size":3282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:30.989902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":215089152,"num_objects":3282,"num_object_clones":0,"num_object_copies":9846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3282,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3282,"num_write_kb":210048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,2],"acting":[1,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"70'139","reported_seq":808,"reported_epoch":724,"state":"active+clean","last_fresh":"2024-08-19T14:40:27.037546+0000","last_change":"2024-08-19T14:39:59.000415+0000","last_active":"2024-08-19T14:40:27.037546+0000","last_peered":"2024-08-19T14:40:27.037546+0000","last_clean":"2024-08-19T14:40:27.037546+0000","last_became_active":"2024-08-19T14:35:11.537474+0000","last_became_peered":"2024-08-19T14:35:11.537474+0000","last_unstale":"2024-08-19T14:40:27.037546+0000","last_undegraded":"2024-08-19T14:40:27.037546+0000","last_fullsized":"2024-08-19T14:40:27.037546+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:39:59.000300+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:39:59.000300+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:11.528949+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.2","version":"724'3003","reported_seq":3027,"reported_epoch":724,"state":"active+clean","last_fresh":"2024-08-19T14:40:30.751080+0000","last_change":"2024-08-19T14:40:04.441786+0000","last_active":"2024-08-19T14:40:30.751080+0000","last_peered":"2024-08-19T14:40:30.751080+0000","last_clean":"2024-08-19T14:40:30.751080+0000","last_became_active":"2024-08-19T14:40:04.441445+0000","last_became_peered":"2024-08-19T14:40:04.441445+0000","last_unstale":"2024-08-19T14:40:30.751080+0000","last_undegraded":"2024-08-19T14:40:30.751080+0000","last_fullsized":"2024-08-19T14:40:30.751080+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":717,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.182354+0000","objects_scrubbed":0,"log_size":3003,"log_dups_size":0,"ondisk_log_size":3003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:23.583504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196804608,"num_objects":3003,"num_object_clones":0,"num_object_copies":9009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3003,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3003,"num_write_kb":192192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,9],"acting":[0,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.b","version":"70'139","reported_seq":1129,"reported_epoch":724,"state":"active+clean","last_fresh":"2024-08-19T14:40:27.037091+0000","last_change":"2024-08-19T14:40:03.813911+0000","last_active":"2024-08-19T14:40:27.037091+0000","last_peered":"2024-08-19T14:40:27.037091+0000","last_clean":"2024-08-19T14:40:27.037091+0000","last_became_active":"2024-08-19T14:30:47.941200+0000","last_became_peered":"2024-08-19T14:30:47.941200+0000","last_unstale":"2024-08-19T14:40:27.037091+0000","last_undegraded":"2024-08-19T14:40:27.037091+0000","last_fullsized":"2024-08-19T14:40:27.037091+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:40:03.813794+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.813794+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:29.146651+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.7","version":"725'3109","reported_seq":3135,"reported_epoch":725,"state":"active+clean","last_fresh":"2024-08-19T14:40:32.990663+0000","last_change":"2024-08-19T14:40:04.299820+0000","last_active":"2024-08-19T14:40:32.990663+0000","last_peered":"2024-08-19T14:40:32.990663+0000","last_clean":"2024-08-19T14:40:32.990663+0000","last_became_active":"2024-08-19T14:40:04.299448+0000","last_became_peered":"2024-08-19T14:40:04.299448+0000","last_unstale":"2024-08-19T14:40:32.990663+0000","last_undegraded":"2024-08-19T14:40:32.990663+0000","last_fullsized":"2024-08-19T14:40:32.990663+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":717,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.182354+0000","objects_scrubbed":0,"log_size":3109,"log_dups_size":0,"ondisk_log_size":3109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:05.592419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":203751424,"num_objects":3109,"num_object_clones":0,"num_object_copies":9327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3109,"num_write_kb":198976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.e","version":"70'139","reported_seq":1323,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-08-19T14:40:33.118734+0000","last_change":"2024-08-19T14:40:03.764766+0000","last_active":"2024-08-19T14:40:33.118734+0000","last_peered":"2024-08-19T14:40:33.118734+0000","last_clean":"2024-08-19T14:40:33.118734+0000","last_became_active":"2024-08-19T14:18:26.884105+0000","last_became_peered":"2024-08-19T14:18:26.884105+0000","last_unstale":"2024-08-19T14:40:33.118734+0000","last_undegraded":"2024-08-19T14:40:33.118734+0000","last_fullsized":"2024-08-19T14:40:33.118734+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:40:03.764672+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:40:03.764672+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.764672+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:11.797638+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.6","version":"724'3091","reported_seq":3115,"reported_epoch":724,"state":"active+clean","last_fresh":"2024-08-19T14:40:31.667406+0000","last_change":"2024-08-19T14:40:04.443511+0000","last_active":"2024-08-19T14:40:31.667406+0000","last_peered":"2024-08-19T14:40:31.667406+0000","last_clean":"2024-08-19T14:40:31.667406+0000","last_became_active":"2024-08-19T14:40:04.442943+0000","last_became_peered":"2024-08-19T14:40:04.442943+0000","last_unstale":"2024-08-19T14:40:31.667406+0000","last_undegraded":"2024-08-19T14:40:31.667406+0000","last_fullsized":"2024-08-19T14:40:31.667406+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":717,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.182354+0000","objects_scrubbed":0,"log_size":3091,"log_dups_size":0,"ondisk_log_size":3091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:29.082021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":202571776,"num_objects":3091,"num_object_clones":0,"num_object_copies":9273,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3091,"num_write_kb":197824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,2],"acting":[11,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.f","version":"70'139","reported_seq":1075,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-08-19T14:40:33.117779+0000","last_change":"2024-08-19T14:39:59.812809+0000","last_active":"2024-08-19T14:40:33.117779+0000","last_peered":"2024-08-19T14:40:33.117779+0000","last_clean":"2024-08-19T14:40:33.117779+0000","last_became_active":"2024-08-19T14:27:31.837754+0000","last_became_peered":"2024-08-19T14:27:31.837754+0000","last_unstale":"2024-08-19T14:40:33.117779+0000","last_undegraded":"2024-08-19T14:40:33.117779+0000","last_fullsized":"2024-08-19T14:40:33.117779+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:39:59.812612+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:39:59.812612+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:07.074257+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.5","version":"726'3345","reported_seq":3373,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-08-19T14:40:33.722970+0000","last_change":"2024-08-19T14:40:04.200422+0000","last_active":"2024-08-19T14:40:33.722970+0000","last_peered":"2024-08-19T14:40:33.722970+0000","last_clean":"2024-08-19T14:40:33.722970+0000","last_became_active":"2024-08-19T14:40:04.200106+0000","last_became_peered":"2024-08-19T14:40:04.200106+0000","last_unstale":"2024-08-19T14:40:33.722970+0000","last_undegraded":"2024-08-19T14:40:33.722970+0000","last_fullsized":"2024-08-19T14:40:33.722970+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":717,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.182354+0000","objects_scrubbed":0,"log_size":3345,"log_dups_size":0,"ondisk_log_size":3345,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:09.342864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":219217920,"num_objects":3345,"num_object_clones":0,"num_object_copies":10035,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3345,"num_write_kb":214080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,11],"acting":[4,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.c","version":"70'139","reported_seq":1426,"reported_epoch":724,"state":"active+clean","last_fresh":"2024-08-19T14:40:27.037342+0000","last_change":"2024-08-19T14:40:00.947434+0000","last_active":"2024-08-19T14:40:27.037342+0000","last_peered":"2024-08-19T14:40:27.037342+0000","last_clean":"2024-08-19T14:40:27.037342+0000","last_became_active":"2024-08-19T14:34:15.733282+0000","last_became_peered":"2024-08-19T14:34:15.733282+0000","last_unstale":"2024-08-19T14:40:27.037342+0000","last_undegraded":"2024-08-19T14:40:27.037342+0000","last_fullsized":"2024-08-19T14:40:27.037342+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:40:00.947341+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:22:52.164586+0000","last_clean_scrub_stamp":"2024-08-19T14:40:00.947341+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:21.334102+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,9],"acting":[3,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"724'3053","reported_seq":3077,"reported_epoch":724,"state":"active+clean","last_fresh":"2024-08-19T14:40:31.683516+0000","last_change":"2024-08-19T14:40:04.443126+0000","last_active":"2024-08-19T14:40:31.683516+0000","last_peered":"2024-08-19T14:40:31.683516+0000","last_clean":"2024-08-19T14:40:31.683516+0000","last_became_active":"2024-08-19T14:40:04.442757+0000","last_became_peered":"2024-08-19T14:40:04.442757+0000","last_unstale":"2024-08-19T14:40:31.683516+0000","last_undegraded":"2024-08-19T14:40:31.683516+0000","last_fullsized":"2024-08-19T14:40:31.683516+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":717,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.182354+0000","objects_scrubbed":0,"log_size":3053,"log_dups_size":0,"ondisk_log_size":3053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:06.385105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":200081408,"num_objects":3053,"num_object_clones":0,"num_object_copies":9159,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3053,"num_write_kb":195392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,9],"acting":[11,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.d","version":"70'139","reported_seq":1323,"reported_epoch":724,"state":"active+clean","last_fresh":"2024-08-19T14:40:27.041874+0000","last_change":"2024-08-19T14:40:08.911300+0000","last_active":"2024-08-19T14:40:27.041874+0000","last_peered":"2024-08-19T14:40:27.041874+0000","last_clean":"2024-08-19T14:40:27.041874+0000","last_became_active":"2024-08-19T14:18:26.882002+0000","last_became_peered":"2024-08-19T14:18:26.882002+0000","last_unstale":"2024-08-19T14:40:27.041874+0000","last_undegraded":"2024-08-19T14:40:27.041874+0000","last_fullsized":"2024-08-19T14:40:27.041874+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:40:08.911204+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:24:50.276408+0000","last_clean_scrub_stamp":"2024-08-19T14:40:08.911204+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:27.367708+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"70'139","reported_seq":1104,"reported_epoch":725,"state":"active+clean","last_fresh":"2024-08-19T14:40:32.109626+0000","last_change":"2024-08-19T14:40:04.665520+0000","last_active":"2024-08-19T14:40:32.109626+0000","last_peered":"2024-08-19T14:40:32.109626+0000","last_clean":"2024-08-19T14:40:32.109626+0000","last_became_active":"2024-08-19T14:27:31.838968+0000","last_became_peered":"2024-08-19T14:27:31.838968+0000","last_unstale":"2024-08-19T14:40:32.109626+0000","last_undegraded":"2024-08-19T14:40:32.109626+0000","last_fullsized":"2024-08-19T14:40:32.109626+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:40:04.665441+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:28:35.206115+0000","last_clean_scrub_stamp":"2024-08-19T14:40:04.665441+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:25.039402+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.13","version":"70'139","reported_seq":1646,"reported_epoch":725,"state":"active+clean","last_fresh":"2024-08-19T14:40:32.111542+0000","last_change":"2024-08-19T14:40:05.659348+0000","last_active":"2024-08-19T14:40:32.111542+0000","last_peered":"2024-08-19T14:40:32.111542+0000","last_clean":"2024-08-19T14:40:32.111542+0000","last_became_active":"2024-08-19T14:30:47.939849+0000","last_became_peered":"2024-08-19T14:30:47.939849+0000","last_unstale":"2024-08-19T14:40:32.111542+0000","last_undegraded":"2024-08-19T14:40:32.111542+0000","last_fullsized":"2024-08-19T14:40:32.111542+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:40:05.659275+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:33:35.786105+0000","last_clean_scrub_stamp":"2024-08-19T14:40:05.659275+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:30.843663+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.10","version":"70'139","reported_seq":1264,"reported_epoch":724,"state":"active+clean","last_fresh":"2024-08-19T14:40:27.041261+0000","last_change":"2024-08-19T14:34:58.616722+0000","last_active":"2024-08-19T14:40:27.041261+0000","last_peered":"2024-08-19T14:40:27.041261+0000","last_clean":"2024-08-19T14:40:27.041261+0000","last_became_active":"2024-08-19T14:27:31.839611+0000","last_became_peered":"2024-08-19T14:27:31.839611+0000","last_unstale":"2024-08-19T14:40:27.041261+0000","last_undegraded":"2024-08-19T14:40:27.041261+0000","last_fullsized":"2024-08-19T14:40:27.041261+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:34:58.616646+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:34:58.616646+0000","last_clean_scrub_stamp":"2024-08-19T14:34:58.616646+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:40:30.062147+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.11","version":"70'139","reported_seq":1328,"reported_epoch":724,"state":"active+clean","last_fresh":"2024-08-19T14:40:27.040083+0000","last_change":"2024-08-19T14:40:04.812392+0000","last_active":"2024-08-19T14:40:27.040083+0000","last_peered":"2024-08-19T14:40:27.040083+0000","last_clean":"2024-08-19T14:40:27.040083+0000","last_became_active":"2024-08-19T14:34:15.731049+0000","last_became_peered":"2024-08-19T14:34:15.731049+0000","last_unstale":"2024-08-19T14:40:27.040083+0000","last_undegraded":"2024-08-19T14:40:27.040083+0000","last_fullsized":"2024-08-19T14:40:27.040083+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:40:04.812269+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:24:50.547428+0000","last_clean_scrub_stamp":"2024-08-19T14:40:04.812269+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:33.537909+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"70'139","reported_seq":1257,"reported_epoch":725,"state":"active+clean","last_fresh":"2024-08-19T14:40:32.109529+0000","last_change":"2024-08-19T14:40:03.700753+0000","last_active":"2024-08-19T14:40:32.109529+0000","last_peered":"2024-08-19T14:40:32.109529+0000","last_clean":"2024-08-19T14:40:32.109529+0000","last_became_active":"2024-08-19T14:20:28.204724+0000","last_became_peered":"2024-08-19T14:20:28.204724+0000","last_unstale":"2024-08-19T14:40:32.109529+0000","last_undegraded":"2024-08-19T14:40:32.109529+0000","last_fullsized":"2024-08-19T14:40:32.109529+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:40:03.700675+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:28:34.216370+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.700675+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:20.590299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"70'139","reported_seq":1406,"reported_epoch":724,"state":"active+clean","last_fresh":"2024-08-19T14:40:30.299825+0000","last_change":"2024-08-19T14:40:30.299862+0000","last_active":"2024-08-19T14:40:30.299825+0000","last_peered":"2024-08-19T14:40:30.299825+0000","last_clean":"2024-08-19T14:40:30.299825+0000","last_became_active":"2024-08-19T14:18:31.901556+0000","last_became_peered":"2024-08-19T14:18:31.901556+0000","last_unstale":"2024-08-19T14:40:30.299825+0000","last_undegraded":"2024-08-19T14:40:30.299825+0000","last_fullsized":"2024-08-19T14:40:30.299825+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:40:30.299750+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:40:30.299750+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:36.225861+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,0],"acting":[8,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.14","version":"70'139","reported_seq":1427,"reported_epoch":724,"state":"active+clean","last_fresh":"2024-08-19T14:40:27.041368+0000","last_change":"2024-08-19T14:40:03.932359+0000","last_active":"2024-08-19T14:40:27.041368+0000","last_peered":"2024-08-19T14:40:27.041368+0000","last_clean":"2024-08-19T14:40:27.041368+0000","last_became_active":"2024-08-19T14:35:12.529792+0000","last_became_peered":"2024-08-19T14:35:12.529792+0000","last_unstale":"2024-08-19T14:40:27.041368+0000","last_undegraded":"2024-08-19T14:40:27.041368+0000","last_fullsized":"2024-08-19T14:40:27.041368+0000","mapping_epoch":635,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":636,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:40:03.932271+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.932271+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:23.358249+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,9],"acting":[7,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.15","version":"70'139","reported_seq":1279,"reported_epoch":724,"state":"active+clean","last_fresh":"2024-08-19T14:40:27.041814+0000","last_change":"2024-08-19T14:40:07.881907+0000","last_active":"2024-08-19T14:40:27.041814+0000","last_peered":"2024-08-19T14:40:27.041814+0000","last_clean":"2024-08-19T14:40:27.041814+0000","last_became_active":"2024-08-19T14:18:31.912496+0000","last_became_peered":"2024-08-19T14:18:31.912496+0000","last_unstale":"2024-08-19T14:40:27.041814+0000","last_undegraded":"2024-08-19T14:40:27.041814+0000","last_fullsized":"2024-08-19T14:40:27.041814+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:40:07.881806+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:20:09.180923+0000","last_clean_scrub_stamp":"2024-08-19T14:40:07.881806+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:32.759701+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"70'139","reported_seq":1268,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-08-19T14:40:33.117839+0000","last_change":"2024-08-19T14:40:00.753735+0000","last_active":"2024-08-19T14:40:33.117839+0000","last_peered":"2024-08-19T14:40:33.117839+0000","last_clean":"2024-08-19T14:40:33.117839+0000","last_became_active":"2024-08-19T14:18:31.901229+0000","last_became_peered":"2024-08-19T14:18:31.901229+0000","last_unstale":"2024-08-19T14:40:33.117839+0000","last_undegraded":"2024-08-19T14:40:33.117839+0000","last_fullsized":"2024-08-19T14:40:33.117839+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:40:00.753636+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:40:00.753636+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:07.574708+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1b","version":"70'139","reported_seq":1645,"reported_epoch":724,"state":"active+clean","last_fresh":"2024-08-19T14:40:27.035943+0000","last_change":"2024-08-19T14:39:32.443757+0000","last_active":"2024-08-19T14:40:27.035943+0000","last_peered":"2024-08-19T14:40:27.035943+0000","last_clean":"2024-08-19T14:40:27.035943+0000","last_became_active":"2024-08-19T14:34:15.718427+0000","last_became_peered":"2024-08-19T14:34:15.718427+0000","last_unstale":"2024-08-19T14:40:27.035943+0000","last_undegraded":"2024-08-19T14:40:27.035943+0000","last_fullsized":"2024-08-19T14:40:27.035943+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:39:32.443656+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:37:59.510697+0000","last_clean_scrub_stamp":"2024-08-19T14:39:32.443656+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:40:40.877428+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.18","version":"70'139","reported_seq":1207,"reported_epoch":724,"state":"active+clean","last_fresh":"2024-08-19T14:40:27.041253+0000","last_change":"2024-08-19T14:39:59.937810+0000","last_active":"2024-08-19T14:40:27.041253+0000","last_peered":"2024-08-19T14:40:27.041253+0000","last_clean":"2024-08-19T14:40:27.041253+0000","last_became_active":"2024-08-19T14:35:11.528530+0000","last_became_peered":"2024-08-19T14:35:11.528530+0000","last_unstale":"2024-08-19T14:40:27.041253+0000","last_undegraded":"2024-08-19T14:40:27.041253+0000","last_fullsized":"2024-08-19T14:40:27.041253+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:39:59.937668+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:27:33.084161+0000","last_clean_scrub_stamp":"2024-08-19T14:39:59.937668+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:17.756316+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.19","version":"70'139","reported_seq":1625,"reported_epoch":724,"state":"active+clean","last_fresh":"2024-08-19T14:40:27.039972+0000","last_change":"2024-08-19T14:39:32.723246+0000","last_active":"2024-08-19T14:40:27.039972+0000","last_peered":"2024-08-19T14:40:27.039972+0000","last_clean":"2024-08-19T14:40:27.039972+0000","last_became_active":"2024-08-19T14:18:31.902547+0000","last_became_peered":"2024-08-19T14:18:31.902547+0000","last_unstale":"2024-08-19T14:40:27.039972+0000","last_undegraded":"2024-08-19T14:40:27.039972+0000","last_fullsized":"2024-08-19T14:40:27.039972+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:39:32.722703+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:39:32.722703+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:40:40.564491+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1e","version":"70'139","reported_seq":641,"reported_epoch":724,"state":"active+clean","last_fresh":"2024-08-19T14:40:27.037088+0000","last_change":"2024-08-19T14:33:17.778415+0000","last_active":"2024-08-19T14:40:27.037088+0000","last_peered":"2024-08-19T14:40:27.037088+0000","last_clean":"2024-08-19T14:40:27.037088+0000","last_became_active":"2024-08-19T14:30:47.909974+0000","last_became_peered":"2024-08-19T14:30:47.909974+0000","last_unstale":"2024-08-19T14:40:27.037088+0000","last_undegraded":"2024-08-19T14:40:27.037088+0000","last_fullsized":"2024-08-19T14:40:27.037088+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:33:17.778295+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:27:04.780939+0000","last_clean_scrub_stamp":"2024-08-19T14:33:17.778295+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:40:29.827169+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1c","version":"70'139","reported_seq":1399,"reported_epoch":724,"state":"active+clean","last_fresh":"2024-08-19T14:40:27.035999+0000","last_change":"2024-08-19T14:39:26.405807+0000","last_active":"2024-08-19T14:40:27.035999+0000","last_peered":"2024-08-19T14:40:27.035999+0000","last_clean":"2024-08-19T14:40:27.035999+0000","last_became_active":"2024-08-19T14:30:47.908866+0000","last_became_peered":"2024-08-19T14:30:47.908866+0000","last_unstale":"2024-08-19T14:40:27.035999+0000","last_undegraded":"2024-08-19T14:40:27.035999+0000","last_fullsized":"2024-08-19T14:40:27.035999+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:39:26.405699+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:39:26.405699+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:40:29.159585+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,11],"acting":[8,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1d","version":"70'139","reported_seq":1310,"reported_epoch":725,"state":"active+clean","last_fresh":"2024-08-19T14:40:32.109736+0000","last_change":"2024-08-19T14:39:57.775710+0000","last_active":"2024-08-19T14:40:32.109736+0000","last_peered":"2024-08-19T14:40:32.109736+0000","last_clean":"2024-08-19T14:40:32.109736+0000","last_became_active":"2024-08-19T14:35:11.540488+0000","last_became_peered":"2024-08-19T14:35:11.540488+0000","last_unstale":"2024-08-19T14:40:32.109736+0000","last_undegraded":"2024-08-19T14:40:32.109736+0000","last_fullsized":"2024-08-19T14:40:32.109736+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:39:57.775631+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:39:57.775631+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:40:59.182147+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":3227516928,"num_objects":49248,"num_object_clones":0,"num_object_copies":147744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49248,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49248,"num_write_kb":3151872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9784983552,"data_stored":9784983552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":49248,"ondisk_log_size":49248,"up":48,"acting":48,"num_store_stats":10},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":856608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187806,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2262020,"kb_used_data":1356220,"kb_used_omap":1,"kb_used_meta":905790,"kb_avail":91483132,"statfs":{"total":95995035648,"available":93678727168,"internally_reserved":0,"allocated":1388769280,"data_stored":1385574918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":927529412},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":2314987372673,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":74,"seq":317827580264,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2333524,"kb_used_data":1449928,"kb_used_omap":1,"kb_used_meta":883582,"kb_avail":91411628,"statfs":{"total":95995035648,"available":93605507072,"internally_reserved":0,"allocated":1484726272,"data_stored":1481527705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":904788417},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874606,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1363268,"kb_used_data":574076,"kb_used_omap":1,"kb_used_meta":789182,"kb_avail":92381884,"statfs":{"total":95995035648,"available":94599049216,"internally_reserved":0,"allocated":587853824,"data_stored":584659462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":808122821},"hb_peers":[0,1,2,3,4,5,6,7,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":57,"seq":244813136241,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1339916,"kb_used_data":379972,"kb_used_omap":1,"kb_used_meta":959934,"kb_avail":92405236,"statfs":{"total":95995035648,"available":94622961664,"internally_reserved":0,"allocated":389091328,"data_stored":385893414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":982972874},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":206158430580,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692216,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1944524,"kb_used_data":974852,"kb_used_omap":1,"kb_used_meta":969662,"kb_avail":91800628,"statfs":{"total":95995035648,"available":94003843072,"internally_reserved":0,"allocated":998248448,"data_stored":995050534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":992934341},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986557,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1811796,"kb_used_data":828232,"kb_used_omap":1,"kb_used_meta":983550,"kb_avail":91933356,"statfs":{"total":95995035648,"available":94139756544,"internally_reserved":0,"allocated":848109568,"data_stored":844911001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1007155653},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150083,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1839300,"kb_used_data":757180,"kb_used_omap":1,"kb_used_meta":1082110,"kb_avail":91905852,"statfs":{"total":95995035648,"available":94111592448,"internally_reserved":0,"allocated":775352320,"data_stored":772157958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1108081095},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":634,"seq":2723009265729,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1582732,"kb_used_data":1275780,"kb_used_omap":4,"kb_used_meta":306939,"kb_avail":92162420,"statfs":{"total":95995035648,"available":94374318080,"internally_reserved":0,"allocated":1306398720,"data_stored":1303200806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4407,"internal_metadata":314306249},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575244,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2383892,"kb_used_data":1237896,"kb_used_omap":1,"kb_used_meta":1145982,"kb_avail":91361260,"statfs":{"total":95995035648,"available":93553930240,"internally_reserved":0,"allocated":1267605504,"data_stored":1264406937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1173486021},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738768,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1463428,"kb_used_data":561788,"kb_used_omap":1,"kb_used_meta":901630,"kb_avail":92281724,"statfs":{"total":95995035648,"available":94496485376,"internally_reserved":0,"allocated":575270912,"data_stored":572076550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":923269573},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":589824000,"data_stored":589824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1271398400,"data_stored":1271398400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1339686912,"data_stored":1339686912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":797573120,"data_stored":797573120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":871301120,"data_stored":871301120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1010302976,"data_stored":1010302976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":390135808,"data_stored":390135808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":580583424,"data_stored":580583424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1506541568,"data_stored":1506541568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1427636224,"data_stored":1427636224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T14:40:36.635 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-08-19T14:40:36.635 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd pool set unique_pool_6 pg_num 26 2024-08-19T14:40:36.655 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:36.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:36.656 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:40:36.757 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:40:36.790 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:36.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:36.791 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:40:36.892 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:40:36.920 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:36.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:36.921 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:40:37.022 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:40:37.066 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:37.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:37.067 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:40:37.168 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:40:37.201 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:37.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:37.202 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:40:37.270 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:37 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:40:37.270 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:37 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:40:37.270 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:37 smithi031 ceph-mon[16336]: osdmap e728: 12 total, 12 up, 10 in 2024-08-19T14:40:37.288 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 26 16 58760 58744 141.199 82.4375 0.00442591 0.00707742 2024-08-19T14:40:37.303 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:40:37.330 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:37.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:37.332 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:40:37.345 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:40:37.432 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:40:37.459 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:37.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:37.460 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:40:37.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:37 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:40:37.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:37 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:40:37.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:37 smithi032 ceph-mon[20004]: osdmap e728: 12 total, 12 up, 10 in 2024-08-19T14:40:37.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:37 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:40:37.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:37 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:40:37.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:37 smithi045 ceph-mon[20143]: osdmap e728: 12 total, 12 up, 10 in 2024-08-19T14:40:37.561 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:40:37.597 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:37.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:37.598 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:40:37.699 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:40:37.706 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:40:37.706 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-08-19T14:40:37.707 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-08-19T14:40:37.707 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-08-19T14:40:37.707 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:40:37.720 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:37.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:37.721 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:40:37.822 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:40:37.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:37.863 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:37.864 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:40:37.965 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:40:37.997 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:37.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:37.998 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:40:38.101 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:40:38.126 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:38.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:38.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:40:38.228 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:40:38.269 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:38.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:38.270 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:40:38.288 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 27 16 60470 60454 139.928 106.875 0.0153503 0.00714148 2024-08-19T14:40:38.338 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:40:38.372 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:40:38.394 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:38.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:38.395 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:40:38.471 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:38 smithi031 ceph-mon[16336]: pgmap v1749: 47 pgs: 47 active+clean; 3.3 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 86 MiB/s wr, 1.37k op/s 2024-08-19T14:40:38.496 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:40:38.509 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:38 smithi032 ceph-mon[20004]: pgmap v1749: 47 pgs: 47 active+clean; 3.3 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 86 MiB/s wr, 1.37k op/s 2024-08-19T14:40:38.517 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:38.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:38.518 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:40:38.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:38 smithi045 ceph-mon[20143]: pgmap v1749: 47 pgs: 47 active+clean; 3.3 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 86 MiB/s wr, 1.37k op/s 2024-08-19T14:40:38.619 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:40:38.648 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:38.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:38.649 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:40:38.750 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:40:38.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:38.783 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:38.783 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:40:38.884 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:40:38.918 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:38.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:38.919 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:40:39.021 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:40:39.050 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:39.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:39.052 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:40:39.152 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:40:39.154 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-08-19T14:40:39.183 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:39.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:39.184 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:40:39.285 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:40:39.288 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 28 16 61992 61976 138.327 95.125 0.0369112 0.00722323 2024-08-19T14:40:39.322 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is unset 2024-08-19T14:40:39.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:39.327 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:39.327 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:40:39.327 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:40:39.428 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:40:39.455 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:40:39.485 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:39.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:39.486 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:40:39.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:39 smithi032 ceph-mon[20004]: pgmap v1750: 47 pgs: 47 active+clean; 3.5 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 96 MiB/s wr, 1.53k op/s 2024-08-19T14:40:39.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:39 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:40:39.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:39 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:40:39.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:39 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/4200868755' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:40:39.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:39 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:40:39.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:39 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:40:39.587 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:40:39.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:39 smithi031 ceph-mon[16336]: pgmap v1750: 47 pgs: 47 active+clean; 3.5 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 96 MiB/s wr, 1.53k op/s 2024-08-19T14:40:39.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:39 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:40:39.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:39 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:40:39.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:39 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/4200868755' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:40:39.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:39 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:40:39.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:39 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:40:39.617 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:39.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:39.618 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:40:39.719 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:40:39.748 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:39.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:39.749 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:40:39.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:39 smithi045 ceph-mon[20143]: pgmap v1750: 47 pgs: 47 active+clean; 3.5 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 96 MiB/s wr, 1.53k op/s 2024-08-19T14:40:39.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:39 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:40:39.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:39 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:40:39.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:39 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/4200868755' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:40:39.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:39 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:40:39.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:39 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:40:39.850 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:40:39.881 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:39.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:39.882 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:40:39.983 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:40:40.007 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:40:40.007 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-08-19T14:40:40.007 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-08-19T14:40:40.007 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [] 2024-08-19T14:40:40.008 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:40:40.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:40.028 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:40.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:40:40.067 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:40:40.129 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:40:40.163 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:40.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:40.164 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:40:40.267 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:40:40.291 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 29 16 63668 63652 137.16 104.75 0.00970306 0.00728541 2024-08-19T14:40:40.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:40.309 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:40.310 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:40:40.349 INFO:teuthology.orchestra.run.smithi031.stderr:set pool 8 pg_num to 26 2024-08-19T14:40:40.410 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:40:40.434 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:40.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:40.435 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:40:40.537 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:40:40.571 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:40.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:40.572 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:40:40.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:40 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:40:40.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:40 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:40:40.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:40 smithi031 ceph-mon[16336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:40:40.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:40 smithi031 ceph-mon[16336]: Cluster is now healthy 2024-08-19T14:40:40.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:40 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/4200868755' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:40:40.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:40 smithi031 ceph-mon[16336]: osdmap e729: 12 total, 12 up, 10 in 2024-08-19T14:40:40.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:40 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:40:40.618 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:40 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:40:40.618 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:40 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:40:40.618 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:40 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2147968220' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num", "val": "26"}]: dispatch 2024-08-19T14:40:40.630 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:40 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:40:40.630 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:40 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:40:40.630 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:40 smithi032 ceph-mon[20004]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:40:40.630 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:40 smithi032 ceph-mon[20004]: Cluster is now healthy 2024-08-19T14:40:40.630 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:40 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/4200868755' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:40:40.630 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:40 smithi032 ceph-mon[20004]: osdmap e729: 12 total, 12 up, 10 in 2024-08-19T14:40:40.630 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:40 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:40:40.631 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:40 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:40:40.631 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:40 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:40:40.631 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:40 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2147968220' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num", "val": "26"}]: dispatch 2024-08-19T14:40:40.673 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:40:40.713 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:40.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:40.714 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:40:40.762 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-08-19T14:40:40.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:40 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:40:40.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:40 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:40:40.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:40 smithi045 ceph-mon[20143]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:40:40.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:40 smithi045 ceph-mon[20143]: Cluster is now healthy 2024-08-19T14:40:40.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:40 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/4200868755' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:40:40.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:40 smithi045 ceph-mon[20143]: osdmap e729: 12 total, 12 up, 10 in 2024-08-19T14:40:40.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:40 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:40:40.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:40 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:40:40.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:40 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:40:40.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:40 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2147968220' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num", "val": "26"}]: dispatch 2024-08-19T14:40:40.814 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:40:40.846 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:40.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:40.847 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:40:40.948 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:40:40.984 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:40.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:40.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:40:41.075 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:40:41.086 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:40:41.108 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:41.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:41.109 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:40:41.210 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:40:41.237 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:41.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:41.238 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:40:41.291 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 30 16 65298 65282 135.984 101.875 0.0117098 0.00734924 2024-08-19T14:40:41.338 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:40:41.366 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:41.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:41.367 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:40:41.465 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:41 smithi032 ceph-mon[20004]: pgmap v1752: 47 pgs: 47 active+clean; 3.5 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 86 MiB/s wr, 1.37k op/s 2024-08-19T14:40:41.465 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:41 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2147968220' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num", "val": "26"}]': finished 2024-08-19T14:40:41.465 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:41 smithi032 ceph-mon[20004]: osdmap e730: 12 total, 12 up, 10 in 2024-08-19T14:40:41.466 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:41 smithi031 ceph-mon[16336]: pgmap v1752: 47 pgs: 47 active+clean; 3.5 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 86 MiB/s wr, 1.37k op/s 2024-08-19T14:40:41.466 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:41 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2147968220' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num", "val": "26"}]': finished 2024-08-19T14:40:41.466 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:41 smithi031 ceph-mon[16336]: osdmap e730: 12 total, 12 up, 10 in 2024-08-19T14:40:41.468 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:40:41.503 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:41.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:41.504 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:40:41.547 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:40:41.547 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:40:41.588 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:40:41.589 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:40:41.591 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:40:41.591 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-08-19T14:40:41.591 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-08-19T14:40:41.591 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:40:41.605 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:40:41.626 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:40:41.627 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:40:41.633 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:41.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:41.634 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:40:41.661 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:40:41.661 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:40:41.691 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:40:41.691 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:40:41.727 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:40:41.727 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:40:41.735 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:40:41.760 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:40:41.760 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:40:41.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:41.775 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:41.775 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:40:41.793 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:40:41.793 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:40:41.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:41 smithi045 ceph-mon[20143]: pgmap v1752: 47 pgs: 47 active+clean; 3.5 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 86 MiB/s wr, 1.37k op/s 2024-08-19T14:40:41.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:41 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2147968220' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num", "val": "26"}]': finished 2024-08-19T14:40:41.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:41 smithi045 ceph-mon[20143]: osdmap e730: 12 total, 12 up, 10 in 2024-08-19T14:40:41.827 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:40:41.828 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:40:41.862 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:40:41.863 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:40:41.876 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:40:41.895 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:40:41.895 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:40:41.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:41.910 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:41.911 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:40:41.991 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-19T14:40:42.012 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:40:42.014 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:40:42.014 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:40:42.056 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:42.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:42.057 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:40:42.159 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:40:42.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:42.192 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:42.193 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:40:42.291 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 31 16 66894 66878 134.815 99.75 0.0075829 0.00741263 2024-08-19T14:40:42.293 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:40:42.322 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:42.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:42.323 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:40:42.407 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-08-19T14:40:42.424 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:40:42.479 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:42.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:42.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:40:42.581 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:40:42.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:42.603 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:42.604 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:40:42.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:42 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1964502977' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:40:42.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:42 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-08-19T14:40:42.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:42 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-08-19T14:40:42.704 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:40:42.705 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:40:42.727 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:42.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:42.728 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:40:42.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:42 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1964502977' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:40:42.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:42 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-08-19T14:40:42.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:42 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-08-19T14:40:42.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:42 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1964502977' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:40:42.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:42 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-08-19T14:40:42.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:42 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-08-19T14:40:42.829 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:40:42.842 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:40:42.851 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:42.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:42.852 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:40:42.954 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:40:42.975 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:42.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:42.976 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:40:43.077 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:40:43.099 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:43.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:43.100 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:40:43.202 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:40:43.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:43.225 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:43.225 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:40:43.291 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 32 16 67021 67005 130.85 7.9375 0.0250889 0.00742299 2024-08-19T14:40:43.326 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:40:43.347 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:43.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:43.348 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:40:43.449 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:40:43.484 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:43.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:43.485 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:40:43.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:43 smithi032 ceph-mon[20004]: pgmap v1754: 47 pgs: 47 active+clean; 3.9 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 108 MiB/s wr, 1.73k op/s 2024-08-19T14:40:43.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:43 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num_actual", "val": "26"}]': finished 2024-08-19T14:40:43.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:43 smithi032 ceph-mon[20004]: osdmap e731: 12 total, 12 up, 10 in 2024-08-19T14:40:43.528 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:40:43.528 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:40:43.586 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:40:43.625 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:43.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:43.626 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:40:43.727 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:40:43.763 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:43.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:43.764 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:40:43.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:43 smithi045 ceph-mon[20143]: pgmap v1754: 47 pgs: 47 active+clean; 3.9 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 108 MiB/s wr, 1.73k op/s 2024-08-19T14:40:43.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:43 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num_actual", "val": "26"}]': finished 2024-08-19T14:40:43.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:43 smithi045 ceph-mon[20143]: osdmap e731: 12 total, 12 up, 10 in 2024-08-19T14:40:43.864 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:40:43.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:43 smithi031 ceph-mon[16336]: pgmap v1754: 47 pgs: 47 active+clean; 3.9 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 108 MiB/s wr, 1.73k op/s 2024-08-19T14:40:43.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:43 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num_actual", "val": "26"}]': finished 2024-08-19T14:40:43.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:43 smithi031 ceph-mon[16336]: osdmap e731: 12 total, 12 up, 10 in 2024-08-19T14:40:43.894 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:43.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:43.895 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:40:43.996 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:40:44.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:44.032 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:44.032 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:40:44.133 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:40:44.165 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:44.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:44.167 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:40:44.261 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-08-19T14:40:44.268 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:40:44.291 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 33 16 68799 68783 130.252 111.125 0.00401786 0.00767192 2024-08-19T14:40:44.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:44.330 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:44.331 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:40:44.378 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is set 2024-08-19T14:40:44.431 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:40:44.453 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:44.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:44.454 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:40:44.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:44 smithi032 ceph-mon[20004]: 1.1b scrub starts 2024-08-19T14:40:44.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:44 smithi032 ceph-mon[20004]: 1.1b scrub ok 2024-08-19T14:40:44.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:44 smithi032 ceph-mon[20004]: osdmap e732: 12 total, 12 up, 10 in 2024-08-19T14:40:44.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:44 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1126445881' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:40:44.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:44 smithi032 ceph-mon[20004]: 8.5 scrub starts 2024-08-19T14:40:44.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:44 smithi032 ceph-mon[20004]: 8.9 scrub starts 2024-08-19T14:40:44.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:44 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-08-19T14:40:44.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:44 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-08-19T14:40:44.550 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:40:44.554 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:40:44.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:44.576 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:44.576 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:40:44.677 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:40:44.699 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:44.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:44.700 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:40:44.702 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:44 smithi031 ceph-mon[16336]: 1.1b scrub starts 2024-08-19T14:40:44.702 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:44 smithi031 ceph-mon[16336]: 1.1b scrub ok 2024-08-19T14:40:44.702 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:44 smithi031 ceph-mon[16336]: osdmap e732: 12 total, 12 up, 10 in 2024-08-19T14:40:44.702 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:44 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1126445881' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:40:44.703 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:44 smithi031 ceph-mon[16336]: 8.5 scrub starts 2024-08-19T14:40:44.703 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:44 smithi031 ceph-mon[16336]: 8.9 scrub starts 2024-08-19T14:40:44.703 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:44 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-08-19T14:40:44.703 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:44 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-08-19T14:40:44.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:44 smithi045 ceph-mon[20143]: 1.1b scrub starts 2024-08-19T14:40:44.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:44 smithi045 ceph-mon[20143]: 1.1b scrub ok 2024-08-19T14:40:44.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:44 smithi045 ceph-mon[20143]: osdmap e732: 12 total, 12 up, 10 in 2024-08-19T14:40:44.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:44 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1126445881' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:40:44.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:44 smithi045 ceph-mon[20143]: 8.5 scrub starts 2024-08-19T14:40:44.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:44 smithi045 ceph-mon[20143]: 8.9 scrub starts 2024-08-19T14:40:44.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:44 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-08-19T14:40:44.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:44 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-08-19T14:40:44.800 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:40:44.817 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:40:44.824 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:44.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:44.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:40:44.926 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:40:44.948 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:44.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:44.949 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:40:45.050 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:40:45.090 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:45.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:45.091 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:40:45.095 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:40:45.095 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:40:45.095 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:40:45.192 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:40:45.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:45.215 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:45.215 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:40:45.291 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 34 16 69005 68989 126.8 12.875 0.00302009 0.00767432 2024-08-19T14:40:45.316 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:40:45.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:45.338 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:45.339 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:40:45.440 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:40:45.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:45.462 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:45.463 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:40:45.564 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:40:45.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:45.585 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:45.586 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:40:45.648 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:45 smithi031 ceph-mon[16336]: 8.7 scrub starts 2024-08-19T14:40:45.649 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:45 smithi031 ceph-mon[16336]: 8.0 deep-scrub starts 2024-08-19T14:40:45.649 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:45 smithi031 ceph-mon[16336]: 8.4 scrub starts 2024-08-19T14:40:45.649 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:45 smithi031 ceph-mon[16336]: 8.5 scrub ok 2024-08-19T14:40:45.649 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:45 smithi031 ceph-mon[16336]: 8.7 scrub ok 2024-08-19T14:40:45.649 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:45 smithi031 ceph-mon[16336]: pgmap v1757: 57 pgs: 2 peering, 9 unknown, 46 active+clean; 4.0 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 100 MiB/s wr, 1.60k op/s 2024-08-19T14:40:45.649 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:45 smithi031 ceph-mon[16336]: 8.4 scrub ok 2024-08-19T14:40:45.649 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:45 smithi031 ceph-mon[16336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:40:45.649 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:45 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1126445881' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:40:45.649 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:45 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pgp_num_actual", "val": "26"}]': finished 2024-08-19T14:40:45.649 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:45 smithi031 ceph-mon[16336]: osdmap e733: 12 total, 12 up, 10 in 2024-08-19T14:40:45.664 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:40:45.665 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:40:45.680 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:45 smithi032 ceph-mon[20004]: 8.7 scrub starts 2024-08-19T14:40:45.680 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:45 smithi032 ceph-mon[20004]: 8.0 deep-scrub starts 2024-08-19T14:40:45.680 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:45 smithi032 ceph-mon[20004]: 8.4 scrub starts 2024-08-19T14:40:45.680 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:45 smithi032 ceph-mon[20004]: 8.5 scrub ok 2024-08-19T14:40:45.680 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:45 smithi032 ceph-mon[20004]: 8.7 scrub ok 2024-08-19T14:40:45.680 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:45 smithi032 ceph-mon[20004]: pgmap v1757: 57 pgs: 2 peering, 9 unknown, 46 active+clean; 4.0 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 100 MiB/s wr, 1.60k op/s 2024-08-19T14:40:45.680 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:45 smithi032 ceph-mon[20004]: 8.4 scrub ok 2024-08-19T14:40:45.680 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:45 smithi032 ceph-mon[20004]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:40:45.680 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:45 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1126445881' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:40:45.681 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:45 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pgp_num_actual", "val": "26"}]': finished 2024-08-19T14:40:45.681 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:45 smithi032 ceph-mon[20004]: osdmap e733: 12 total, 12 up, 10 in 2024-08-19T14:40:45.682 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:40:45.682 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:40:45.686 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:40:45.698 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:40:45.698 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:40:45.705 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-08-19T14:40:45.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:45.708 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:45.708 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:40:45.714 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:40:45.714 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:40:45.727 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:45 smithi045 ceph-mon[20143]: 8.7 scrub starts 2024-08-19T14:40:45.727 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:45 smithi045 ceph-mon[20143]: 8.0 deep-scrub starts 2024-08-19T14:40:45.728 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:45 smithi045 ceph-mon[20143]: 8.4 scrub starts 2024-08-19T14:40:45.728 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:45 smithi045 ceph-mon[20143]: 8.5 scrub ok 2024-08-19T14:40:45.728 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:45 smithi045 ceph-mon[20143]: 8.7 scrub ok 2024-08-19T14:40:45.728 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:45 smithi045 ceph-mon[20143]: pgmap v1757: 57 pgs: 2 peering, 9 unknown, 46 active+clean; 4.0 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 100 MiB/s wr, 1.60k op/s 2024-08-19T14:40:45.728 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:45 smithi045 ceph-mon[20143]: 8.4 scrub ok 2024-08-19T14:40:45.728 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:45 smithi045 ceph-mon[20143]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:40:45.728 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:45 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1126445881' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:40:45.728 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:45 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pgp_num_actual", "val": "26"}]': finished 2024-08-19T14:40:45.728 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:45 smithi045 ceph-mon[20143]: osdmap e733: 12 total, 12 up, 10 in 2024-08-19T14:40:45.731 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:40:45.731 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:40:45.747 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:40:45.747 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:40:45.764 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:40:45.764 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:40:45.781 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:40:45.782 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:40:45.798 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:40:45.798 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:40:45.809 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:40:45.815 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:40:45.815 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:40:45.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:45.832 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:45.832 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:40:45.833 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:40:45.833 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:40:45.851 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:40:45.851 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:40:45.933 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:40:45.954 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:45.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:45.955 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:40:45.961 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:40:46.056 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:40:46.079 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:46.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:46.080 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:40:46.181 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:40:46.200 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 7, 8, 2, 11, 9] out_osds: [10, 6] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 8, 9, 11, 10, 2] 2024-08-19T14:40:46.200 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-19T14:40:46.200 INFO:tasks.thrashosds.thrasher:Removing osd 3, in_osds are: [0, 1, 3, 4, 5, 7, 8, 2, 11, 9] 2024-08-19T14:40:46.201 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd out 3 2024-08-19T14:40:46.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:46.204 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:46.204 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:40:46.291 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 35 16 69005 68989 123.177 0 - 0.00767432 2024-08-19T14:40:46.305 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:40:46.325 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:46.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:46.326 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:40:46.427 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:40:46.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:46.463 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:46.463 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:40:46.519 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:46 smithi031 ceph-mon[16336]: osdmap e734: 12 total, 12 up, 10 in 2024-08-19T14:40:46.519 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:46 smithi031 ceph-mon[16336]: 8.0 deep-scrub ok 2024-08-19T14:40:46.519 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:46 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/550465885' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:40:46.557 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:40:46.557 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [], 2024-08-19T14:40:46.557 INFO:teuthology.orchestra.run.smithi045.stdout: "num_ops": 0 2024-08-19T14:40:46.557 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:40:46.563 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:40:46.593 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:46.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:46.594 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:40:46.669 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:40:46.695 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:40:46.722 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:46.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:46.723 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:40:46.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:46 smithi032 ceph-mon[20004]: osdmap e734: 12 total, 12 up, 10 in 2024-08-19T14:40:46.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:46 smithi032 ceph-mon[20004]: 8.0 deep-scrub ok 2024-08-19T14:40:46.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:46 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/550465885' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:40:46.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:46 smithi045 ceph-mon[20143]: osdmap e734: 12 total, 12 up, 10 in 2024-08-19T14:40:46.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:46 smithi045 ceph-mon[20143]: 8.0 deep-scrub ok 2024-08-19T14:40:46.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:46 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/550465885' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:40:46.824 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:40:46.851 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:46.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:46.852 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:40:46.953 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:40:46.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:46.980 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:46.980 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:40:47.043 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-19T14:40:47.081 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:40:47.125 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:47.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:47.126 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:40:47.227 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:40:47.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:47.291 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:47.291 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:40:47.292 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 36 16 69506 69490 120.625 15.6562 0.0204743 0.00828092 2024-08-19T14:40:47.351 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-08-19T14:40:47.393 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:40:47.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:47.417 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:47.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:40:47.518 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:40:47.545 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:47.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:47.546 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:40:47.590 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:40:47.647 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:40:47.671 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:47.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:47.672 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:40:47.679 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:40:47.705 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:47 smithi031 ceph-mon[16336]: pgmap v1760: 57 pgs: 2 peering, 4 remapped+peering, 5 unknown, 46 active+clean; 4.1 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 24 MiB/s wr, 388 op/s 2024-08-19T14:40:47.705 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:47 smithi031 ceph-mon[16336]: osdmap e735: 12 total, 12 up, 10 in 2024-08-19T14:40:47.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:47 smithi032 ceph-mon[20004]: pgmap v1760: 57 pgs: 2 peering, 4 remapped+peering, 5 unknown, 46 active+clean; 4.1 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 24 MiB/s wr, 388 op/s 2024-08-19T14:40:47.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:47 smithi032 ceph-mon[20004]: osdmap e735: 12 total, 12 up, 10 in 2024-08-19T14:40:47.773 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:40:47.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:47 smithi045 ceph-mon[20143]: pgmap v1760: 57 pgs: 2 peering, 4 remapped+peering, 5 unknown, 46 active+clean; 4.1 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 24 MiB/s wr, 388 op/s 2024-08-19T14:40:47.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:47 smithi045 ceph-mon[20143]: osdmap e735: 12 total, 12 up, 10 in 2024-08-19T14:40:47.803 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:47.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:47.804 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:40:47.906 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:40:47.948 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:47.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:47.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:40:48.050 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:40:48.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:48.079 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:48.080 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:40:48.082 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:47 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mon-a[16332]: 2024-08-19T14:40:47.702+0000 7f2ff503e640 -1 mon.a@0(leader).osd e735 definitely_dead 0 2024-08-19T14:40:48.180 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:40:48.205 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:48.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:48.206 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:40:48.291 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 37 16 70222 70206 118.575 44.75 0.0172058 0.00841682 2024-08-19T14:40:48.307 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:40:48.347 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:48.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:48.348 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:40:48.406 INFO:teuthology.orchestra.run.smithi031.stderr:marked out osd.3. 2024-08-19T14:40:48.414 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:48 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3639026993' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-08-19T14:40:48.414 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:48 smithi045 ceph-mon[20143]: Client client.admin marked osd.3 out, while it was still marked up 2024-08-19T14:40:48.448 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:40:48.488 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:48.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:48.489 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:40:48.529 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:40:48.529 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [], 2024-08-19T14:40:48.529 INFO:teuthology.orchestra.run.smithi045.stdout: "complaint_time": 30, 2024-08-19T14:40:48.529 INFO:teuthology.orchestra.run.smithi045.stdout: "num_blocked_ops": 0 2024-08-19T14:40:48.530 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:40:48.578 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:40:48.589 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:40:48.612 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:48.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:48.613 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:40:48.714 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:40:48.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:48.752 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:48.753 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:40:48.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:48 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3639026993' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-08-19T14:40:48.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:48 smithi032 ceph-mon[20004]: Client client.admin marked osd.3 out, while it was still marked up 2024-08-19T14:40:48.843 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:48 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3639026993' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-08-19T14:40:48.843 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:48 smithi031 ceph-mon[16336]: Client client.admin marked osd.3 out, while it was still marked up 2024-08-19T14:40:48.853 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:40:48.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:48.877 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:48.878 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:40:48.979 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:40:49.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:49.015 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:49.015 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:40:49.116 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:40:49.139 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:49.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:49.141 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:40:49.207 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-08-19T14:40:49.208 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:40:49.242 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:40:49.265 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:49.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:49.266 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:40:49.291 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 38 16 70336 70320 115.642 7.125 0.0220021 0.00845477 2024-08-19T14:40:49.367 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:40:49.401 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:49.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:49.402 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:40:49.421 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is set 2024-08-19T14:40:49.488 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:40:49.503 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:40:49.528 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:49.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:49.529 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:40:49.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:49 smithi045 ceph-mon[20143]: pgmap v1762: 57 pgs: 1 activating+undersized+degraded+remapped, 1 active+recovering+remapped, 4 active+recovering+undersized+remapped, 4 remapped+peering, 47 active+clean; 4.2 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 4380/207744 objects degraded (2.108%); 21076/207744 objects misplaced (10.145%); 2.5 MiB/s, 40 objects/s recovering 2024-08-19T14:40:49.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:49 smithi045 ceph-mon[20143]: Health check failed: Degraded data redundancy: 4380/207744 objects degraded (2.108%), 1 pg degraded (PG_DEGRADED) 2024-08-19T14:40:49.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:49 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3639026993' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2024-08-19T14:40:49.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:49 smithi045 ceph-mon[20143]: osdmap e736: 12 total, 12 up, 9 in 2024-08-19T14:40:49.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:49 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3583300162' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:40:49.629 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:40:49.654 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:49.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:49.655 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:40:49.734 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:49 smithi031 ceph-mon[16336]: pgmap v1762: 57 pgs: 1 activating+undersized+degraded+remapped, 1 active+recovering+remapped, 4 active+recovering+undersized+remapped, 4 remapped+peering, 47 active+clean; 4.2 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 4380/207744 objects degraded (2.108%); 21076/207744 objects misplaced (10.145%); 2.5 MiB/s, 40 objects/s recovering 2024-08-19T14:40:49.734 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:49 smithi031 ceph-mon[16336]: Health check failed: Degraded data redundancy: 4380/207744 objects degraded (2.108%), 1 pg degraded (PG_DEGRADED) 2024-08-19T14:40:49.734 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:49 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3639026993' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2024-08-19T14:40:49.734 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:49 smithi031 ceph-mon[16336]: osdmap e736: 12 total, 12 up, 9 in 2024-08-19T14:40:49.735 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:49 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3583300162' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:40:49.755 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:40:49.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:49 smithi032 ceph-mon[20004]: pgmap v1762: 57 pgs: 1 activating+undersized+degraded+remapped, 1 active+recovering+remapped, 4 active+recovering+undersized+remapped, 4 remapped+peering, 47 active+clean; 4.2 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 4380/207744 objects degraded (2.108%); 21076/207744 objects misplaced (10.145%); 2.5 MiB/s, 40 objects/s recovering 2024-08-19T14:40:49.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:49 smithi032 ceph-mon[20004]: Health check failed: Degraded data redundancy: 4380/207744 objects degraded (2.108%), 1 pg degraded (PG_DEGRADED) 2024-08-19T14:40:49.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:49 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3639026993' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2024-08-19T14:40:49.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:49 smithi032 ceph-mon[20004]: osdmap e736: 12 total, 12 up, 9 in 2024-08-19T14:40:49.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:49 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3583300162' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:40:49.781 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:49.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:49.782 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:40:49.882 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:40:49.918 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:49.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:49.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:40:50.023 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:40:50.048 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:50.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:50.049 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:40:50.108 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:40:50.108 INFO:teuthology.orchestra.run.smithi045.stdout: "size": 20, 2024-08-19T14:40:50.108 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 600, 2024-08-19T14:40:50.108 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [ 2024-08-19T14:40:50.108 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.108 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.16962.0:224666 7.7 7:e0cf291e:::benchmark_data_smithi045_115842_object106506:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e703)", 2024-08-19T14:40:50.108 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:39:14.759205+0000", 2024-08-19T14:40:50.109 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 95.346445662999997, 2024-08-19T14:40:50.109 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.212613626, 2024-08-19T14:40:50.109 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:40:50.109 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:40:50.109 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:40:50.109 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:40:50.109 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.16962", 2024-08-19T14:40:50.109 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:40:50.109 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 224666 2024-08-19T14:40:50.109 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.109 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:40:50.109 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.109 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:40:50.110 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.759205+0000", 2024-08-19T14:40:50.110 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:40:50.110 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.110 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.110 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:40:50.110 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.759205+0000", 2024-08-19T14:40:50.110 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:40:50.110 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.110 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.110 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:40:50.110 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.759206+0000", 2024-08-19T14:40:50.110 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.322e-06 2024-08-19T14:40:50.110 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.111 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.111 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:40:50.111 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.759211+0000", 2024-08-19T14:40:50.111 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.7859999999999996e-06 2024-08-19T14:40:50.111 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.111 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.111 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:40:50.111 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.759221+0000", 2024-08-19T14:40:50.111 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0529e-05 2024-08-19T14:40:50.111 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.111 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.111 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:40:50.112 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.759229+0000", 2024-08-19T14:40:50.112 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.5390000000000002e-06 2024-08-19T14:40:50.112 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.112 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.112 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:40:50.112 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.759280+0000", 2024-08-19T14:40:50.112 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0915999999999998e-05 2024-08-19T14:40:50.112 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.112 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.112 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:40:50.112 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.759363+0000", 2024-08-19T14:40:50.112 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.3714000000000005e-05 2024-08-19T14:40:50.113 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.113 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.113 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,3", 2024-08-19T14:40:50.113 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.759437+0000", 2024-08-19T14:40:50.113 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.3863999999999996e-05 2024-08-19T14:40:50.113 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.113 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.113 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:40:50.113 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.764416+0000", 2024-08-19T14:40:50.113 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0049787759999999999 2024-08-19T14:40:50.113 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.113 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.113 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:40:50.114 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.770232+0000", 2024-08-19T14:40:50.114 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0058158460000000004 2024-08-19T14:40:50.114 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.114 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.114 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:40:50.114 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.971750+0000", 2024-08-19T14:40:50.114 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20151855199999999 2024-08-19T14:40:50.114 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.114 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.114 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:40:50.114 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.971788+0000", 2024-08-19T14:40:50.115 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.7789e-05 2024-08-19T14:40:50.115 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.115 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.115 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:40:50.115 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.971818+0000", 2024-08-19T14:40:50.115 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.9992999999999999e-05 2024-08-19T14:40:50.115 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:40:50.115 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:40:50.115 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:40:50.115 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.115 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.115 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.16962.0:224669 7.7 7:ea42df57:::benchmark_data_smithi045_115842_object106509:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e703)", 2024-08-19T14:40:50.115 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:39:14.765265+0000", 2024-08-19T14:40:50.116 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 95.340385170000005, 2024-08-19T14:40:50.116 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.23464995899999999, 2024-08-19T14:40:50.116 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:40:50.116 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:40:50.116 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:40:50.116 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:40:50.116 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.16962", 2024-08-19T14:40:50.116 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:40:50.116 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 224669 2024-08-19T14:40:50.116 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.116 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:40:50.116 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.116 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:40:50.116 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.765265+0000", 2024-08-19T14:40:50.117 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:40:50.117 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.117 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.117 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:40:50.117 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.765265+0000", 2024-08-19T14:40:50.117 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:40:50.117 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.117 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.117 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:40:50.117 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.765267+0000", 2024-08-19T14:40:50.117 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5600000000000001e-06 2024-08-19T14:40:50.117 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.117 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.118 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:40:50.118 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.765272+0000", 2024-08-19T14:40:50.118 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.6219999999999997e-06 2024-08-19T14:40:50.118 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.118 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.118 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:40:50.118 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.765286+0000", 2024-08-19T14:40:50.118 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3998999999999999e-05 2024-08-19T14:40:50.118 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.118 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.118 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:40:50.118 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.765294+0000", 2024-08-19T14:40:50.118 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.9349999999999994e-06 2024-08-19T14:40:50.119 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.119 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.119 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:40:50.119 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.765350+0000", 2024-08-19T14:40:50.119 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.6243999999999999e-05 2024-08-19T14:40:50.119 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.119 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.119 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:40:50.119 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.765474+0000", 2024-08-19T14:40:50.119 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000123762 2024-08-19T14:40:50.119 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.119 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.119 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,3", 2024-08-19T14:40:50.119 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.765598+0000", 2024-08-19T14:40:50.120 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012404699999999999 2024-08-19T14:40:50.120 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.120 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.120 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:40:50.120 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.770614+0000", 2024-08-19T14:40:50.120 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0050154029999999999 2024-08-19T14:40:50.120 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.120 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.120 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:40:50.120 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.779808+0000", 2024-08-19T14:40:50.120 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0091942959999999994 2024-08-19T14:40:50.120 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.120 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.120 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:40:50.121 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.999824+0000", 2024-08-19T14:40:50.121 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.220016463 2024-08-19T14:40:50.121 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.121 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.121 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:40:50.121 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.999862+0000", 2024-08-19T14:40:50.121 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.8061e-05 2024-08-19T14:40:50.121 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.121 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.121 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:40:50.121 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.999915+0000", 2024-08-19T14:40:50.121 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2567e-05 2024-08-19T14:40:50.121 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:40:50.121 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:40:50.122 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:40:50.122 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.122 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.122 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.16962.0:224671 7.7 7:e66f70e9:::benchmark_data_smithi045_115842_object106511:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e703)", 2024-08-19T14:40:50.122 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:39:14.771305+0000", 2024-08-19T14:40:50.122 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 95.334345010999996, 2024-08-19T14:40:50.122 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.22877210000000001, 2024-08-19T14:40:50.122 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:40:50.122 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:40:50.122 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:40:50.122 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:40:50.122 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.16962", 2024-08-19T14:40:50.122 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:40:50.123 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 224671 2024-08-19T14:40:50.123 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.124 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:40:50.124 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.124 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:40:50.124 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.771305+0000", 2024-08-19T14:40:50.124 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:40:50.124 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.124 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.124 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:40:50.124 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.771305+0000", 2024-08-19T14:40:50.125 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:40:50.125 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.125 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.125 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:40:50.125 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.771307+0000", 2024-08-19T14:40:50.125 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.703e-06 2024-08-19T14:40:50.125 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.125 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.125 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:40:50.125 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.771313+0000", 2024-08-19T14:40:50.125 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.3659999999999997e-06 2024-08-19T14:40:50.125 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.125 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.126 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:40:50.126 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.771331+0000", 2024-08-19T14:40:50.126 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7513999999999999e-05 2024-08-19T14:40:50.126 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.126 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.126 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:40:50.126 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.771338+0000", 2024-08-19T14:40:50.126 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.6799999999999993e-06 2024-08-19T14:40:50.126 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.126 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.126 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:40:50.126 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.771392+0000", 2024-08-19T14:40:50.126 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.342e-05 2024-08-19T14:40:50.126 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.127 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.127 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:40:50.127 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.771508+0000", 2024-08-19T14:40:50.127 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00011576799999999999 2024-08-19T14:40:50.127 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.127 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.127 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,3", 2024-08-19T14:40:50.127 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.771620+0000", 2024-08-19T14:40:50.127 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000112139 2024-08-19T14:40:50.127 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.127 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.127 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:40:50.127 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.776775+0000", 2024-08-19T14:40:50.128 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0051554879999999997 2024-08-19T14:40:50.128 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.128 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.128 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:40:50.128 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.780515+0000", 2024-08-19T14:40:50.128 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0037401919999999998 2024-08-19T14:40:50.128 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.128 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.128 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:40:50.128 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.999975+0000", 2024-08-19T14:40:50.128 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.21945916400000001 2024-08-19T14:40:50.128 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.128 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.128 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:40:50.129 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:15.000031+0000", 2024-08-19T14:40:50.129 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.6450000000000003e-05 2024-08-19T14:40:50.129 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.129 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.129 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:40:50.129 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:15.000077+0000", 2024-08-19T14:40:50.129 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.6215999999999999e-05 2024-08-19T14:40:50.129 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:40:50.129 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:40:50.129 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:40:50.129 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.129 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.129 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.16962.0:224694 7.7 7:eb03b3dc:::benchmark_data_smithi045_115842_object106534:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e703)", 2024-08-19T14:40:50.130 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:39:14.810012+0000", 2024-08-19T14:40:50.130 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 95.295637889000005, 2024-08-19T14:40:50.130 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.19023380100000001, 2024-08-19T14:40:50.130 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:40:50.130 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:40:50.130 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:40:50.130 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:40:50.130 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.16962", 2024-08-19T14:40:50.130 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:40:50.130 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 224694 2024-08-19T14:40:50.130 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.130 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:40:50.130 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.130 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:40:50.131 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.810012+0000", 2024-08-19T14:40:50.131 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:40:50.131 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.131 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.131 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:40:50.131 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.810012+0000", 2024-08-19T14:40:50.131 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:40:50.131 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.131 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.131 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:40:50.131 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.810014+0000", 2024-08-19T14:40:50.131 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2839999999999999e-06 2024-08-19T14:40:50.131 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.132 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.132 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:40:50.132 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.810019+0000", 2024-08-19T14:40:50.132 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0359999999999997e-06 2024-08-19T14:40:50.132 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.132 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.132 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:40:50.132 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.810030+0000", 2024-08-19T14:40:50.132 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1812e-05 2024-08-19T14:40:50.132 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.132 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.132 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:40:50.132 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.810036+0000", 2024-08-19T14:40:50.132 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7969999999999997e-06 2024-08-19T14:40:50.133 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.133 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.133 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:40:50.133 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.810084+0000", 2024-08-19T14:40:50.133 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.7706e-05 2024-08-19T14:40:50.133 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.133 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.133 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:40:50.133 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.810175+0000", 2024-08-19T14:40:50.133 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.0964999999999994e-05 2024-08-19T14:40:50.133 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.133 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.133 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,3", 2024-08-19T14:40:50.134 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.810252+0000", 2024-08-19T14:40:50.134 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.7062000000000002e-05 2024-08-19T14:40:50.134 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.134 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.134 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:40:50.134 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.813920+0000", 2024-08-19T14:40:50.134 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0036680850000000002 2024-08-19T14:40:50.134 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.134 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.134 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:40:50.134 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.839214+0000", 2024-08-19T14:40:50.134 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.025294422 2024-08-19T14:40:50.134 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.134 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.135 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:40:50.135 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:15.000139+0000", 2024-08-19T14:40:50.135 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.160924393 2024-08-19T14:40:50.135 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.135 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.135 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:40:50.135 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:15.000199+0000", 2024-08-19T14:40:50.135 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.0343000000000001e-05 2024-08-19T14:40:50.135 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.135 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.135 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:40:50.135 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:15.000246+0000", 2024-08-19T14:40:50.135 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.6896e-05 2024-08-19T14:40:50.135 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:40:50.136 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:40:50.136 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:40:50.136 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.136 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.136 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.16962.0:224699 7.7 7:e3d167ee:::benchmark_data_smithi045_115842_object106539:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e703)", 2024-08-19T14:40:50.136 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:39:14.822504+0000", 2024-08-19T14:40:50.136 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 95.283146353999996, 2024-08-19T14:40:50.136 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.17785825699999999, 2024-08-19T14:40:50.136 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:40:50.136 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:40:50.137 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:40:50.137 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:40:50.137 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.16962", 2024-08-19T14:40:50.137 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:40:50.137 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 224699 2024-08-19T14:40:50.137 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.137 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:40:50.137 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.137 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:40:50.137 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.822504+0000", 2024-08-19T14:40:50.137 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:40:50.137 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.137 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.138 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:40:50.138 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.822504+0000", 2024-08-19T14:40:50.138 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:40:50.138 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.138 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.138 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:40:50.138 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.822505+0000", 2024-08-19T14:40:50.138 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4980000000000001e-06 2024-08-19T14:40:50.138 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.138 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.138 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:40:50.138 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.822510+0000", 2024-08-19T14:40:50.138 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.9110000000000001e-06 2024-08-19T14:40:50.139 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.139 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.139 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:40:50.139 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.822522+0000", 2024-08-19T14:40:50.139 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1481000000000001e-05 2024-08-19T14:40:50.139 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.139 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.139 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:40:50.139 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.822528+0000", 2024-08-19T14:40:50.139 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.2650000000000002e-06 2024-08-19T14:40:50.139 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.139 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.139 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:40:50.140 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.822574+0000", 2024-08-19T14:40:50.140 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.5652999999999998e-05 2024-08-19T14:40:50.140 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.140 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.140 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:40:50.140 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.822654+0000", 2024-08-19T14:40:50.140 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.0499000000000003e-05 2024-08-19T14:40:50.140 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.140 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.140 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,3", 2024-08-19T14:40:50.140 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.822729+0000", 2024-08-19T14:40:50.140 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.5037999999999995e-05 2024-08-19T14:40:50.140 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.140 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.141 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:40:50.141 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.835429+0000", 2024-08-19T14:40:50.141 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.012699391000000001 2024-08-19T14:40:50.141 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.141 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.141 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:40:50.141 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.847422+0000", 2024-08-19T14:40:50.141 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.011993311 2024-08-19T14:40:50.141 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.141 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.141 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:40:50.141 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:15.000293+0000", 2024-08-19T14:40:50.141 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.15287139699999999 2024-08-19T14:40:50.141 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.142 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.142 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:40:50.142 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:15.000322+0000", 2024-08-19T14:40:50.142 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.8561e-05 2024-08-19T14:40:50.142 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.142 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.142 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:40:50.142 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:15.000362+0000", 2024-08-19T14:40:50.142 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.0252e-05 2024-08-19T14:40:50.142 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:40:50.142 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:40:50.142 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:40:50.142 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.143 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.143 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.16962.0:224700 7.7 7:e378e4bd:::benchmark_data_smithi045_115842_object106540:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e703)", 2024-08-19T14:40:50.143 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:39:14.825059+0000", 2024-08-19T14:40:50.143 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 95.280590771999996, 2024-08-19T14:40:50.143 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.25962368200000002, 2024-08-19T14:40:50.143 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:40:50.143 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:40:50.143 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:40:50.143 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:40:50.143 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.16962", 2024-08-19T14:40:50.143 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:40:50.143 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 224700 2024-08-19T14:40:50.143 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.144 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:40:50.144 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.144 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:40:50.144 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.825059+0000", 2024-08-19T14:40:50.144 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:40:50.144 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.144 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.144 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:40:50.144 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.825059+0000", 2024-08-19T14:40:50.144 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:40:50.144 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.144 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.144 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:40:50.144 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.825061+0000", 2024-08-19T14:40:50.145 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1400000000000001e-06 2024-08-19T14:40:50.145 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.145 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.145 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:40:50.145 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.825066+0000", 2024-08-19T14:40:50.145 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0730000000000004e-06 2024-08-19T14:40:50.145 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.145 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.145 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:40:50.145 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.825078+0000", 2024-08-19T14:40:50.145 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2721000000000001e-05 2024-08-19T14:40:50.145 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.145 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.145 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:40:50.146 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.825085+0000", 2024-08-19T14:40:50.146 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.1350000000000003e-06 2024-08-19T14:40:50.146 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.146 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.146 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:40:50.146 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.826700+0000", 2024-08-19T14:40:50.146 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0016148079999999999 2024-08-19T14:40:50.146 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.146 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.146 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:40:50.146 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.826814+0000", 2024-08-19T14:40:50.146 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000113643 2024-08-19T14:40:50.146 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.147 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.147 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,3", 2024-08-19T14:40:50.149 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.826896+0000", 2024-08-19T14:40:50.150 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.2315e-05 2024-08-19T14:40:50.150 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.150 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.150 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:40:50.150 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.839158+0000", 2024-08-19T14:40:50.150 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.012261943000000001 2024-08-19T14:40:50.150 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.150 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.150 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:40:50.150 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.847478+0000", 2024-08-19T14:40:50.150 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0083198690000000006 2024-08-19T14:40:50.151 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.151 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.151 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:40:50.151 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:15.084574+0000", 2024-08-19T14:40:50.151 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.23709612699999999 2024-08-19T14:40:50.151 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.151 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.151 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:40:50.151 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:15.084631+0000", 2024-08-19T14:40:50.151 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7158999999999999e-05 2024-08-19T14:40:50.151 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.151 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.151 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:40:50.151 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:15.084683+0000", 2024-08-19T14:40:50.152 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1749000000000002e-05 2024-08-19T14:40:50.152 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:40:50.152 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:40:50.152 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:40:50.152 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.152 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.152 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.16962.0:224704 7.7 7:e5b596ea:::benchmark_data_smithi045_115842_object106544:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e703)", 2024-08-19T14:40:50.152 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:39:14.828060+0000", 2024-08-19T14:40:50.152 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 95.277589754000005, 2024-08-19T14:40:50.152 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.26613611799999998, 2024-08-19T14:40:50.152 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:40:50.152 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:40:50.152 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:40:50.152 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:40:50.153 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.16962", 2024-08-19T14:40:50.153 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:40:50.153 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 224704 2024-08-19T14:40:50.153 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.153 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:40:50.153 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.153 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:40:50.153 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.828060+0000", 2024-08-19T14:40:50.153 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:40:50.153 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.153 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.153 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:40:50.153 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.828060+0000", 2024-08-19T14:40:50.153 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:40:50.153 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.154 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.154 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:40:50.154 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.828062+0000", 2024-08-19T14:40:50.154 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2929999999999999e-06 2024-08-19T14:40:50.154 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.154 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.154 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:40:50.154 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.828067+0000", 2024-08-19T14:40:50.154 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.3179999999999997e-06 2024-08-19T14:40:50.154 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.154 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.154 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:40:50.154 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.828083+0000", 2024-08-19T14:40:50.155 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5843000000000001e-05 2024-08-19T14:40:50.155 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.155 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.155 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:40:50.155 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.828090+0000", 2024-08-19T14:40:50.155 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4290000000000004e-06 2024-08-19T14:40:50.155 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.155 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.155 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:40:50.155 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.830830+0000", 2024-08-19T14:40:50.155 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0027398330000000001 2024-08-19T14:40:50.155 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.155 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.155 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:40:50.156 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.830955+0000", 2024-08-19T14:40:50.156 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012513800000000001 2024-08-19T14:40:50.156 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.156 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.156 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,3", 2024-08-19T14:40:50.156 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.831089+0000", 2024-08-19T14:40:50.156 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013408800000000001 2024-08-19T14:40:50.156 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.156 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.156 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:40:50.156 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.842806+0000", 2024-08-19T14:40:50.156 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.011716183 2024-08-19T14:40:50.156 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.157 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.157 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:40:50.157 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.847659+0000", 2024-08-19T14:40:50.157 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0048539070000000002 2024-08-19T14:40:50.157 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.157 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.157 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:40:50.157 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:15.094133+0000", 2024-08-19T14:40:50.157 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.246473419 2024-08-19T14:40:50.157 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.157 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.157 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:40:50.157 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:15.094166+0000", 2024-08-19T14:40:50.157 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.2678000000000001e-05 2024-08-19T14:40:50.158 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.158 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.158 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:40:50.158 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:15.094197+0000", 2024-08-19T14:40:50.158 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.0988999999999998e-05 2024-08-19T14:40:50.158 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:40:50.158 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:40:50.158 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:40:50.158 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.158 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.158 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.16962.0:224705 7.7 7:e64435fe:::benchmark_data_smithi045_115842_object106545:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e703)", 2024-08-19T14:40:50.158 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:39:14.828443+0000", 2024-08-19T14:40:50.158 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 95.277206755999998, 2024-08-19T14:40:50.158 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.27032617399999997, 2024-08-19T14:40:50.159 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:40:50.159 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:40:50.159 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:40:50.159 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:40:50.159 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.16962", 2024-08-19T14:40:50.159 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:40:50.159 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 224705 2024-08-19T14:40:50.159 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.159 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:40:50.159 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.159 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:40:50.159 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.828443+0000", 2024-08-19T14:40:50.159 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:40:50.160 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.160 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.160 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:40:50.160 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.828443+0000", 2024-08-19T14:40:50.160 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:40:50.160 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.160 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.160 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:40:50.160 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.828444+0000", 2024-08-19T14:40:50.160 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.3099999999999996e-07 2024-08-19T14:40:50.160 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.160 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.160 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:40:50.160 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.828450+0000", 2024-08-19T14:40:50.161 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2059999999999998e-06 2024-08-19T14:40:50.161 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.161 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.161 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:40:50.161 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.828464+0000", 2024-08-19T14:40:50.161 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3969e-05 2024-08-19T14:40:50.161 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.161 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.161 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:40:50.161 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.828471+0000", 2024-08-19T14:40:50.161 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.7849999999999992e-06 2024-08-19T14:40:50.161 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.161 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.161 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:40:50.162 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.835455+0000", 2024-08-19T14:40:50.162 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0069832260000000004 2024-08-19T14:40:50.162 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.162 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.162 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:40:50.162 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.835539+0000", 2024-08-19T14:40:50.162 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.4761999999999999e-05 2024-08-19T14:40:50.162 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.162 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.162 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,3", 2024-08-19T14:40:50.162 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.835615+0000", 2024-08-19T14:40:50.162 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.5931000000000002e-05 2024-08-19T14:40:50.162 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.162 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.163 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:40:50.163 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.842812+0000", 2024-08-19T14:40:50.163 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0071963900000000004 2024-08-19T14:40:50.163 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.163 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.163 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:40:50.163 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.849992+0000", 2024-08-19T14:40:50.163 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0071800739999999998 2024-08-19T14:40:50.163 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.163 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.163 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:40:50.163 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:15.098646+0000", 2024-08-19T14:40:50.163 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.248654664 2024-08-19T14:40:50.164 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.164 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.164 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:40:50.164 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:15.098717+0000", 2024-08-19T14:40:50.164 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.1091000000000001e-05 2024-08-19T14:40:50.164 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.164 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.164 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:40:50.164 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:15.098770+0000", 2024-08-19T14:40:50.164 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2145000000000003e-05 2024-08-19T14:40:50.164 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:40:50.164 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:40:50.164 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:40:50.164 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.165 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.165 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.16962.0:224739 7.7 7:e1de92b6:::benchmark_data_smithi045_115842_object106579:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e703)", 2024-08-19T14:40:50.165 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:39:14.892352+0000", 2024-08-19T14:40:50.165 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 95.213298387999998, 2024-08-19T14:40:50.165 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20653886399999999, 2024-08-19T14:40:50.165 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:40:50.165 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:40:50.165 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:40:50.165 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:40:50.165 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.16962", 2024-08-19T14:40:50.165 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:40:50.165 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 224739 2024-08-19T14:40:50.165 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.165 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:40:50.166 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.166 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:40:50.166 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.892352+0000", 2024-08-19T14:40:50.166 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:40:50.166 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.166 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.166 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:40:50.166 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.892352+0000", 2024-08-19T14:40:50.166 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:40:50.166 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.166 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.166 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:40:50.166 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.892354+0000", 2024-08-19T14:40:50.166 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.4310000000000001e-06 2024-08-19T14:40:50.167 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.167 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.167 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:40:50.167 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.892362+0000", 2024-08-19T14:40:50.167 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.2279999999999998e-06 2024-08-19T14:40:50.167 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.167 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.167 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:40:50.167 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.892382+0000", 2024-08-19T14:40:50.167 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9238000000000001e-05 2024-08-19T14:40:50.167 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.167 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.167 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:40:50.167 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.892390+0000", 2024-08-19T14:40:50.168 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.7849999999999997e-06 2024-08-19T14:40:50.168 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.168 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.168 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:40:50.168 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.892459+0000", 2024-08-19T14:40:50.168 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.8820999999999997e-05 2024-08-19T14:40:50.168 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.168 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.168 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:40:50.168 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.892588+0000", 2024-08-19T14:40:50.168 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000129078 2024-08-19T14:40:50.168 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.168 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.168 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,3", 2024-08-19T14:40:50.169 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.892715+0000", 2024-08-19T14:40:50.169 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000126625 2024-08-19T14:40:50.169 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.169 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.169 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:40:50.169 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.896995+0000", 2024-08-19T14:40:50.169 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0042799379999999996 2024-08-19T14:40:50.169 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.169 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.169 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:40:50.169 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.900056+0000", 2024-08-19T14:40:50.169 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0030611689999999999 2024-08-19T14:40:50.169 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.170 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.170 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:40:50.170 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:15.098813+0000", 2024-08-19T14:40:50.170 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.198757133 2024-08-19T14:40:50.170 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.170 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.170 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:40:50.170 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:15.098859+0000", 2024-08-19T14:40:50.170 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.6051999999999999e-05 2024-08-19T14:40:50.170 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.170 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.170 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:40:50.170 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:15.098891+0000", 2024-08-19T14:40:50.170 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.1365999999999998e-05 2024-08-19T14:40:50.171 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:40:50.171 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:40:50.171 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:40:50.171 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.171 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.171 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.16962.0:224744 7.7 7:eaed24eb:::benchmark_data_smithi045_115842_object106584:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e703)", 2024-08-19T14:40:50.171 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:39:14.900081+0000", 2024-08-19T14:40:50.171 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 95.205568806000002, 2024-08-19T14:40:50.171 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.19897712300000001, 2024-08-19T14:40:50.171 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:40:50.171 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:40:50.171 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:40:50.171 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:40:50.172 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.16962", 2024-08-19T14:40:50.172 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:40:50.172 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 224744 2024-08-19T14:40:50.172 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.172 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:40:50.172 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.172 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:40:50.172 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.900081+0000", 2024-08-19T14:40:50.172 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:40:50.172 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.172 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.172 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:40:50.172 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.900081+0000", 2024-08-19T14:40:50.172 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:40:50.173 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.173 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.173 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:40:50.173 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.900083+0000", 2024-08-19T14:40:50.173 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1930000000000001e-06 2024-08-19T14:40:50.173 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.173 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.173 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:40:50.173 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.900087+0000", 2024-08-19T14:40:50.173 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.7230000000000004e-06 2024-08-19T14:40:50.173 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.173 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.173 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:40:50.174 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.900097+0000", 2024-08-19T14:40:50.174 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.5249999999999994e-06 2024-08-19T14:40:50.174 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.174 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.174 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:40:50.174 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.900103+0000", 2024-08-19T14:40:50.174 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.5910000000000001e-06 2024-08-19T14:40:50.174 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.174 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.174 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:40:50.174 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.900154+0000", 2024-08-19T14:40:50.174 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0482999999999998e-05 2024-08-19T14:40:50.174 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.175 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.175 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:40:50.175 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.900241+0000", 2024-08-19T14:40:50.175 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.7496000000000004e-05 2024-08-19T14:40:50.175 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.175 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.175 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,3", 2024-08-19T14:40:50.175 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.900325+0000", 2024-08-19T14:40:50.175 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.3237000000000003e-05 2024-08-19T14:40:50.175 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.175 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.175 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:40:50.175 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.903885+0000", 2024-08-19T14:40:50.176 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00356026 2024-08-19T14:40:50.176 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.176 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.176 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:40:50.176 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.916142+0000", 2024-08-19T14:40:50.176 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.012257308 2024-08-19T14:40:50.176 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.176 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.176 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:40:50.176 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:15.098962+0000", 2024-08-19T14:40:50.176 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.18281961099999999 2024-08-19T14:40:50.176 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.176 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.176 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:40:50.177 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:15.099013+0000", 2024-08-19T14:40:50.177 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1137999999999997e-05 2024-08-19T14:40:50.177 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.177 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.177 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:40:50.177 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:15.099059+0000", 2024-08-19T14:40:50.177 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.5558e-05 2024-08-19T14:40:50.177 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:40:50.177 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:40:50.177 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:40:50.177 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.177 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.177 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.16962.0:224750 7.7 7:e4f20221:::benchmark_data_smithi045_115842_object106590:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e703)", 2024-08-19T14:40:50.177 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:39:14.917954+0000", 2024-08-19T14:40:50.178 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 95.187696533999997, 2024-08-19T14:40:50.178 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.18120317699999999, 2024-08-19T14:40:50.178 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:40:50.178 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:40:50.178 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:40:50.178 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:40:50.178 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.16962", 2024-08-19T14:40:50.178 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:40:50.178 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 224750 2024-08-19T14:40:50.178 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.178 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:40:50.178 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.178 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:40:50.179 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.917954+0000", 2024-08-19T14:40:50.179 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:40:50.179 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.179 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.179 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:40:50.179 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.917954+0000", 2024-08-19T14:40:50.179 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:40:50.179 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.179 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.179 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:40:50.179 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.917955+0000", 2024-08-19T14:40:50.179 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0499999999999999e-06 2024-08-19T14:40:50.179 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.179 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.180 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:40:50.180 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.917959+0000", 2024-08-19T14:40:50.180 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.6410000000000001e-06 2024-08-19T14:40:50.180 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.180 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.180 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:40:50.180 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.917970+0000", 2024-08-19T14:40:50.180 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1078e-05 2024-08-19T14:40:50.180 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.180 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.180 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:40:50.180 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.917978+0000", 2024-08-19T14:40:50.180 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.7659999999999998e-06 2024-08-19T14:40:50.180 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.181 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.181 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:40:50.181 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.918032+0000", 2024-08-19T14:40:50.181 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.3882999999999999e-05 2024-08-19T14:40:50.181 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.181 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.181 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:40:50.181 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.918115+0000", 2024-08-19T14:40:50.181 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.2948999999999995e-05 2024-08-19T14:40:50.181 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.181 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.181 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,3", 2024-08-19T14:40:50.181 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.918189+0000", 2024-08-19T14:40:50.181 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.415e-05 2024-08-19T14:40:50.182 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.182 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.182 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:40:50.182 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.921766+0000", 2024-08-19T14:40:50.182 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0035766579999999999 2024-08-19T14:40:50.182 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.182 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.182 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:40:50.182 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.924116+0000", 2024-08-19T14:40:50.182 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0023503560000000001 2024-08-19T14:40:50.182 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.182 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.182 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:40:50.182 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:15.099106+0000", 2024-08-19T14:40:50.183 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.17498944699999999 2024-08-19T14:40:50.183 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.183 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.183 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:40:50.183 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:15.099136+0000", 2024-08-19T14:40:50.183 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.0490000000000001e-05 2024-08-19T14:40:50.183 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.183 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:40:50.185 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.185 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:40:50.185 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:15.099157+0000", 2024-08-19T14:40:50.185 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0709000000000001e-05 2024-08-19T14:40:50.185 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:40:50.185 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:40:50.185 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:40:50.185 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.185 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.186 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.16962.0:232608 7.7 7:e15b25b9:::benchmark_data_smithi045_115842_object114448:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e707)", 2024-08-19T14:40:50.186 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:39:29.466905+0000", 2024-08-19T14:40:50.186 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 80.638745232999995, 2024-08-19T14:40:50.186 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.29847353399999998, 2024-08-19T14:40:50.186 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:40:50.186 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:40:50.186 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:40:50.186 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:40:50.186 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.16962", 2024-08-19T14:40:50.186 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:40:50.186 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 232608 2024-08-19T14:40:50.186 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.186 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:40:50.186 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.187 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:40:50.187 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.466905+0000", 2024-08-19T14:40:50.187 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:40:50.187 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.187 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.187 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:40:50.187 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.466905+0000", 2024-08-19T14:40:50.187 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:40:50.187 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.187 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.187 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:40:50.187 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.466906+0000", 2024-08-19T14:40:50.187 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.099e-06 2024-08-19T14:40:50.188 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.188 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.188 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:40:50.188 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.466911+0000", 2024-08-19T14:40:50.188 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.5499999999999996e-06 2024-08-19T14:40:50.188 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.188 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.188 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:40:50.188 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.466921+0000", 2024-08-19T14:40:50.188 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0570999999999999e-05 2024-08-19T14:40:50.188 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.188 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.188 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:40:50.189 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.466927+0000", 2024-08-19T14:40:50.189 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.6269999999999997e-06 2024-08-19T14:40:50.189 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.189 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.189 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:40:50.189 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.466975+0000", 2024-08-19T14:40:50.189 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.7728000000000001e-05 2024-08-19T14:40:50.189 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.189 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.189 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:40:50.189 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.467057+0000", 2024-08-19T14:40:50.189 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.2019000000000002e-05 2024-08-19T14:40:50.189 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.189 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.190 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,3", 2024-08-19T14:40:50.190 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.467130+0000", 2024-08-19T14:40:50.190 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.3858999999999999e-05 2024-08-19T14:40:50.190 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.190 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.190 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:40:50.190 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.474856+0000", 2024-08-19T14:40:50.190 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.007725277 2024-08-19T14:40:50.190 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.190 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.190 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:40:50.190 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.491803+0000", 2024-08-19T14:40:50.190 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.016947051000000001 2024-08-19T14:40:50.191 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.191 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.191 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:40:50.191 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.765277+0000", 2024-08-19T14:40:50.191 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.27347464999999999 2024-08-19T14:40:50.191 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.191 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.191 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:40:50.191 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.765346+0000", 2024-08-19T14:40:50.191 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.8960000000000004e-05 2024-08-19T14:40:50.191 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.191 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.191 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:40:50.191 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.765378+0000", 2024-08-19T14:40:50.192 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.2143e-05 2024-08-19T14:40:50.192 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:40:50.192 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:40:50.192 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:40:50.192 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.192 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.192 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.16962.0:232610 7.7 7:e9c6b41b:::benchmark_data_smithi045_115842_object114450:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e707)", 2024-08-19T14:40:50.192 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:39:29.469702+0000", 2024-08-19T14:40:50.192 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 80.635947712000004, 2024-08-19T14:40:50.192 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.295804754, 2024-08-19T14:40:50.192 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:40:50.192 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:40:50.192 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:40:50.192 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:40:50.193 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.16962", 2024-08-19T14:40:50.193 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:40:50.193 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 232610 2024-08-19T14:40:50.193 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.193 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:40:50.193 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.193 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:40:50.193 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.469702+0000", 2024-08-19T14:40:50.193 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:40:50.193 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.193 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.193 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:40:50.193 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.469702+0000", 2024-08-19T14:40:50.194 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:40:50.194 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.194 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.194 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:40:50.194 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.469703+0000", 2024-08-19T14:40:50.194 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.2799999999999995e-07 2024-08-19T14:40:50.194 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.194 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.194 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:40:50.194 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.469707+0000", 2024-08-19T14:40:50.194 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.6729999999999998e-06 2024-08-19T14:40:50.194 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.194 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.195 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:40:50.195 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.469717+0000", 2024-08-19T14:40:50.195 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.9690000000000006e-06 2024-08-19T14:40:50.195 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.195 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.195 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:40:50.195 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.469725+0000", 2024-08-19T14:40:50.195 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.3469999999999999e-06 2024-08-19T14:40:50.195 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.195 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.195 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:40:50.195 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.470957+0000", 2024-08-19T14:40:50.195 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0012315749999999999 2024-08-19T14:40:50.195 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.196 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.196 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:40:50.196 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.471050+0000", 2024-08-19T14:40:50.196 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.3677000000000005e-05 2024-08-19T14:40:50.196 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.196 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.196 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,3", 2024-08-19T14:40:50.196 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.471177+0000", 2024-08-19T14:40:50.196 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012658999999999999 2024-08-19T14:40:50.196 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.196 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.196 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:40:50.196 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.475091+0000", 2024-08-19T14:40:50.197 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0039135089999999999 2024-08-19T14:40:50.197 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.197 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.197 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:40:50.197 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.494005+0000", 2024-08-19T14:40:50.197 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.018914362000000001 2024-08-19T14:40:50.197 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.197 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.197 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:40:50.197 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.765427+0000", 2024-08-19T14:40:50.197 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.27142210700000002 2024-08-19T14:40:50.197 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.197 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.197 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:40:50.198 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.765479+0000", 2024-08-19T14:40:50.198 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1789999999999997e-05 2024-08-19T14:40:50.198 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.198 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.198 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:40:50.198 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.765507+0000", 2024-08-19T14:40:50.198 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.8427000000000001e-05 2024-08-19T14:40:50.198 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:40:50.198 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:40:50.198 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:40:50.198 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.198 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.198 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.16962.0:232614 7.7 7:e6b4bf77:::benchmark_data_smithi045_115842_object114454:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e707)", 2024-08-19T14:40:50.198 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:39:29.475122+0000", 2024-08-19T14:40:50.199 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 80.630527865999994, 2024-08-19T14:40:50.199 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.29052661299999999, 2024-08-19T14:40:50.199 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:40:50.199 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:40:50.199 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:40:50.199 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:40:50.199 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.16962", 2024-08-19T14:40:50.199 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:40:50.199 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 232614 2024-08-19T14:40:50.199 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.199 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:40:50.199 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.199 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:40:50.200 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.475122+0000", 2024-08-19T14:40:50.200 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:40:50.200 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.200 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.200 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:40:50.200 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.475122+0000", 2024-08-19T14:40:50.200 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:40:50.200 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.200 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.200 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:40:50.200 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.475124+0000", 2024-08-19T14:40:50.200 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.176e-06 2024-08-19T14:40:50.200 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.201 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.201 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:40:50.201 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.475129+0000", 2024-08-19T14:40:50.201 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2020000000000003e-06 2024-08-19T14:40:50.201 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.201 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.201 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:40:50.201 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.475140+0000", 2024-08-19T14:40:50.201 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1039999999999999e-05 2024-08-19T14:40:50.201 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.201 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.201 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:40:50.201 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.475147+0000", 2024-08-19T14:40:50.202 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.3930000000000002e-06 2024-08-19T14:40:50.202 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.202 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.202 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:40:50.202 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.475196+0000", 2024-08-19T14:40:50.202 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.9073999999999998e-05 2024-08-19T14:40:50.202 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.202 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.202 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:40:50.202 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.475313+0000", 2024-08-19T14:40:50.202 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000116487 2024-08-19T14:40:50.202 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.202 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.202 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,3", 2024-08-19T14:40:50.203 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.475431+0000", 2024-08-19T14:40:50.203 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000118733 2024-08-19T14:40:50.203 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.203 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.203 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:40:50.203 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.479893+0000", 2024-08-19T14:40:50.203 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0044616609999999996 2024-08-19T14:40:50.203 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.203 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.203 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:40:50.203 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.494040+0000", 2024-08-19T14:40:50.203 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.014146832 2024-08-19T14:40:50.203 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.204 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.204 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:40:50.204 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.765567+0000", 2024-08-19T14:40:50.204 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.27152742099999999 2024-08-19T14:40:50.204 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.204 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.204 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:40:50.204 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.765613+0000", 2024-08-19T14:40:50.204 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.5884000000000003e-05 2024-08-19T14:40:50.204 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.204 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.204 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:40:50.204 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.765649+0000", 2024-08-19T14:40:50.204 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.5710000000000002e-05 2024-08-19T14:40:50.205 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:40:50.205 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:40:50.205 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:40:50.205 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.205 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.205 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.16962.0:232619 7.7 7:e24f6845:::benchmark_data_smithi045_115842_object114459:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e707)", 2024-08-19T14:40:50.205 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:39:29.487667+0000", 2024-08-19T14:40:50.205 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 80.617983097000007, 2024-08-19T14:40:50.205 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.27813914699999998, 2024-08-19T14:40:50.205 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:40:50.205 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:40:50.205 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:40:50.205 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:40:50.205 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.16962", 2024-08-19T14:40:50.206 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:40:50.206 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 232619 2024-08-19T14:40:50.206 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.206 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:40:50.206 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.206 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:40:50.206 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.487667+0000", 2024-08-19T14:40:50.206 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:40:50.206 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.206 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.206 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:40:50.206 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.487667+0000", 2024-08-19T14:40:50.207 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:40:50.207 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.207 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.207 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:40:50.207 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.487668+0000", 2024-08-19T14:40:50.207 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2899999999999999e-06 2024-08-19T14:40:50.207 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.207 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.207 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:40:50.207 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.487673+0000", 2024-08-19T14:40:50.207 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.3900000000000003e-06 2024-08-19T14:40:50.207 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.207 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.207 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:40:50.208 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.487684+0000", 2024-08-19T14:40:50.208 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0827e-05 2024-08-19T14:40:50.208 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.208 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.208 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:40:50.208 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.487689+0000", 2024-08-19T14:40:50.208 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.8170000000000004e-06 2024-08-19T14:40:50.208 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.208 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.208 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:40:50.208 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.487742+0000", 2024-08-19T14:40:50.208 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2960000000000001e-05 2024-08-19T14:40:50.208 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.209 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.209 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:40:50.209 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.487864+0000", 2024-08-19T14:40:50.209 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012173200000000001 2024-08-19T14:40:50.209 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.209 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.209 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,3", 2024-08-19T14:40:50.209 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.487998+0000", 2024-08-19T14:40:50.209 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013379900000000001 2024-08-19T14:40:50.209 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.209 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.209 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:40:50.209 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.491865+0000", 2024-08-19T14:40:50.209 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0038671370000000001 2024-08-19T14:40:50.210 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.210 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.210 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:40:50.210 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.494174+0000", 2024-08-19T14:40:50.210 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0023090509999999999 2024-08-19T14:40:50.210 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.210 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.210 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:40:50.210 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.765707+0000", 2024-08-19T14:40:50.210 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.27153272000000001 2024-08-19T14:40:50.210 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.210 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.210 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:40:50.211 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.765764+0000", 2024-08-19T14:40:50.211 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7079e-05 2024-08-19T14:40:50.211 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.211 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.211 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:40:50.211 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.765806+0000", 2024-08-19T14:40:50.211 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.2345000000000002e-05 2024-08-19T14:40:50.211 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:40:50.211 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:40:50.211 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:40:50.211 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.211 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.211 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.26911.0:56717 8.b 8:d86a8fa0:::benchmark_data_smithi045_137103_object56716:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e727)", 2024-08-19T14:40:50.211 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:40:35.843805+0000", 2024-08-19T14:40:50.212 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 14.261845279999999, 2024-08-19T14:40:50.212 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20949699999999999, 2024-08-19T14:40:50.212 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:40:50.212 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:40:50.212 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:40:50.212 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:40:50.212 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.26911", 2024-08-19T14:40:50.212 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/3494993995", 2024-08-19T14:40:50.212 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 56717 2024-08-19T14:40:50.212 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.212 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:40:50.212 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.212 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:40:50.212 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:40:35.843805+0000", 2024-08-19T14:40:50.213 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:40:50.213 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.213 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.213 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:40:50.213 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:40:35.843805+0000", 2024-08-19T14:40:50.213 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:40:50.213 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.213 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.213 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:40:50.213 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:40:35.843806+0000", 2024-08-19T14:40:50.213 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.229e-06 2024-08-19T14:40:50.213 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.213 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.214 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:40:50.214 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:40:35.843830+0000", 2024-08-19T14:40:50.214 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.3405e-05 2024-08-19T14:40:50.214 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.214 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.214 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:40:50.214 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:40:35.843840+0000", 2024-08-19T14:40:50.214 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0859999999999999e-05 2024-08-19T14:40:50.214 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.214 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.214 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:40:50.214 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:40:35.843848+0000", 2024-08-19T14:40:50.214 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.3479999999999998e-06 2024-08-19T14:40:50.215 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.215 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.215 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:40:50.215 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:40:35.843927+0000", 2024-08-19T14:40:50.215 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.9027000000000001e-05 2024-08-19T14:40:50.215 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.215 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.215 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:40:50.215 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:40:35.844060+0000", 2024-08-19T14:40:50.215 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000133683 2024-08-19T14:40:50.215 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.215 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.215 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,4", 2024-08-19T14:40:50.215 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:40:35.844183+0000", 2024-08-19T14:40:50.216 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012224600000000001 2024-08-19T14:40:50.216 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.216 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.216 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:40:50.216 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:40:35.846493+0000", 2024-08-19T14:40:50.216 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00231044 2024-08-19T14:40:50.216 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.216 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.216 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:40:50.216 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:40:35.854315+0000", 2024-08-19T14:40:50.216 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0078214180000000001 2024-08-19T14:40:50.216 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.216 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.216 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:40:50.217 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:40:36.053222+0000", 2024-08-19T14:40:50.217 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.198907745 2024-08-19T14:40:50.217 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.217 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.217 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:40:50.217 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:40:36.053258+0000", 2024-08-19T14:40:50.217 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.5679999999999997e-05 2024-08-19T14:40:50.217 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.217 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.217 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:40:50.217 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:40:36.053302+0000", 2024-08-19T14:40:50.217 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.3918999999999997e-05 2024-08-19T14:40:50.217 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:40:50.218 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:40:50.218 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:40:50.218 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.218 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.218 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.26911.0:56756 8.b 8:de712729:::benchmark_data_smithi045_137103_object56755:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e727)", 2024-08-19T14:40:50.218 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:40:35.867306+0000", 2024-08-19T14:40:50.218 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 14.238344197, 2024-08-19T14:40:50.218 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.18610792300000001, 2024-08-19T14:40:50.218 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:40:50.218 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:40:50.218 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:40:50.218 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:40:50.219 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.26911", 2024-08-19T14:40:50.219 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/3494993995", 2024-08-19T14:40:50.219 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 56756 2024-08-19T14:40:50.219 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.219 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:40:50.219 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.219 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:40:50.219 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:40:35.867306+0000", 2024-08-19T14:40:50.219 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:40:50.219 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.219 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.219 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:40:50.219 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:40:35.867306+0000", 2024-08-19T14:40:50.219 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:40:50.220 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.220 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.220 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:40:50.220 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:40:35.867307+0000", 2024-08-19T14:40:50.220 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0389999999999999e-06 2024-08-19T14:40:50.220 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.220 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.220 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:40:50.220 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:40:35.867337+0000", 2024-08-19T14:40:50.220 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.9641e-05 2024-08-19T14:40:50.220 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.220 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.220 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:40:50.220 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:40:35.867351+0000", 2024-08-19T14:40:50.221 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3923999999999999e-05 2024-08-19T14:40:50.221 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.221 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.221 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:40:50.221 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:40:35.867359+0000", 2024-08-19T14:40:50.221 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.2330000000000006e-06 2024-08-19T14:40:50.221 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.221 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.221 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:40:50.221 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:40:35.867427+0000", 2024-08-19T14:40:50.221 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.8589000000000003e-05 2024-08-19T14:40:50.221 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.221 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.221 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:40:50.222 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:40:35.867559+0000", 2024-08-19T14:40:50.222 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013178899999999999 2024-08-19T14:40:50.222 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.222 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.222 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,4", 2024-08-19T14:40:50.222 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:40:35.867766+0000", 2024-08-19T14:40:50.222 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00020723399999999999 2024-08-19T14:40:50.222 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.222 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.222 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:40:50.222 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:40:35.869804+0000", 2024-08-19T14:40:50.222 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0020372110000000001 2024-08-19T14:40:50.222 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.223 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.223 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:40:50.223 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:40:35.871816+0000", 2024-08-19T14:40:50.223 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0020122669999999999 2024-08-19T14:40:50.223 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.223 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.223 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:40:50.223 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:40:36.053345+0000", 2024-08-19T14:40:50.223 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.18152863599999999 2024-08-19T14:40:50.223 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.223 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.223 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:40:50.223 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:40:36.053384+0000", 2024-08-19T14:40:50.223 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.9269e-05 2024-08-19T14:40:50.224 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.224 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.224 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:40:50.224 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:40:36.053414+0000", 2024-08-19T14:40:50.224 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.0091e-05 2024-08-19T14:40:50.224 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:40:50.224 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:40:50.224 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:40:50.224 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.224 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.224 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.26911.0:56766 8.b 8:d111e4f4:::benchmark_data_smithi045_137103_object56765:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e727)", 2024-08-19T14:40:50.224 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:40:35.872756+0000", 2024-08-19T14:40:50.224 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 14.232893716, 2024-08-19T14:40:50.224 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.182063857, 2024-08-19T14:40:50.225 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:40:50.225 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:40:50.225 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:40:50.225 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:40:50.225 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.26911", 2024-08-19T14:40:50.225 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/3494993995", 2024-08-19T14:40:50.225 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 56766 2024-08-19T14:40:50.225 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.225 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:40:50.225 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.225 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:40:50.225 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:40:35.872756+0000", 2024-08-19T14:40:50.225 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:40:50.226 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.226 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.226 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:40:50.226 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:40:35.872756+0000", 2024-08-19T14:40:50.226 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:40:50.226 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.226 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.226 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:40:50.226 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:40:35.872758+0000", 2024-08-19T14:40:50.226 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0809999999999999e-06 2024-08-19T14:40:50.226 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.226 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.226 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:40:50.226 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:40:35.872785+0000", 2024-08-19T14:40:50.227 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.7764e-05 2024-08-19T14:40:50.227 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.227 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.227 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:40:50.227 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:40:35.872799+0000", 2024-08-19T14:40:50.227 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3244999999999999e-05 2024-08-19T14:40:50.227 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.227 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.227 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:40:50.227 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:40:35.872807+0000", 2024-08-19T14:40:50.227 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.1410000000000005e-06 2024-08-19T14:40:50.227 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.227 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.228 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:40:50.228 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:40:35.872905+0000", 2024-08-19T14:40:50.228 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.8237000000000002e-05 2024-08-19T14:40:50.228 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.228 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.228 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:40:50.228 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:40:35.873059+0000", 2024-08-19T14:40:50.228 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015365999999999999 2024-08-19T14:40:50.228 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.228 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.228 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,4", 2024-08-19T14:40:50.228 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:40:35.873186+0000", 2024-08-19T14:40:50.228 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000127844 2024-08-19T14:40:50.228 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.229 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.229 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:40:50.229 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:40:35.875795+0000", 2024-08-19T14:40:50.229 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0026089210000000002 2024-08-19T14:40:50.229 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.229 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.229 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:40:50.229 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:40:35.875807+0000", 2024-08-19T14:40:50.229 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1326e-05 2024-08-19T14:40:50.229 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.229 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.229 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:40:50.229 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:40:36.054754+0000", 2024-08-19T14:40:50.230 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.17894684599999999 2024-08-19T14:40:50.230 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.230 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.230 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:40:50.230 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:40:36.054793+0000", 2024-08-19T14:40:50.230 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.9821999999999999e-05 2024-08-19T14:40:50.230 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.230 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.230 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:40:50.230 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:40:36.054820+0000", 2024-08-19T14:40:50.230 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.6970000000000001e-05 2024-08-19T14:40:50.230 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:40:50.230 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:40:50.231 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:40:50.231 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.231 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.231 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.26911.0:56781 8.b 8:d8416c6a:::benchmark_data_smithi045_137103_object56780:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e727)", 2024-08-19T14:40:50.231 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:40:35.879884+0000", 2024-08-19T14:40:50.231 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 14.225766336, 2024-08-19T14:40:50.231 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.17742804600000001, 2024-08-19T14:40:50.231 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:40:50.231 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:40:50.231 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:40:50.231 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:40:50.231 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.26911", 2024-08-19T14:40:50.231 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/3494993995", 2024-08-19T14:40:50.231 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 56781 2024-08-19T14:40:50.232 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.232 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:40:50.232 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.232 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:40:50.232 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:40:35.879884+0000", 2024-08-19T14:40:50.232 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:40:50.232 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.232 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.232 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:40:50.232 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:40:35.879884+0000", 2024-08-19T14:40:50.232 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:40:50.232 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.232 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.232 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:40:50.233 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:40:35.879891+0000", 2024-08-19T14:40:50.233 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.1919999999999999e-06 2024-08-19T14:40:50.233 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.233 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.233 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:40:50.233 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:40:35.879913+0000", 2024-08-19T14:40:50.233 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2101000000000001e-05 2024-08-19T14:40:50.233 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.233 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.233 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:40:50.233 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:40:35.879926+0000", 2024-08-19T14:40:50.233 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3278000000000001e-05 2024-08-19T14:40:50.233 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.233 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.234 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:40:50.234 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:40:35.879933+0000", 2024-08-19T14:40:50.234 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.0319999999999998e-06 2024-08-19T14:40:50.234 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.234 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.234 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:40:50.234 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:40:35.880089+0000", 2024-08-19T14:40:50.234 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000155725 2024-08-19T14:40:50.234 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.234 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.234 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:40:50.234 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:40:35.880216+0000", 2024-08-19T14:40:50.234 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000126744 2024-08-19T14:40:50.234 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.235 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.235 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,4", 2024-08-19T14:40:50.235 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:40:35.880348+0000", 2024-08-19T14:40:50.235 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000131922 2024-08-19T14:40:50.235 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.235 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.235 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:40:50.235 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:40:35.882509+0000", 2024-08-19T14:40:50.235 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0021610219999999999 2024-08-19T14:40:50.235 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.235 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.235 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:40:50.235 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:40:35.886254+0000", 2024-08-19T14:40:50.235 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0037449089999999998 2024-08-19T14:40:50.236 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.236 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.236 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:40:50.236 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:40:36.057213+0000", 2024-08-19T14:40:50.236 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.170959165 2024-08-19T14:40:50.236 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.236 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.236 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:40:50.236 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:40:36.057271+0000", 2024-08-19T14:40:50.236 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7887999999999999e-05 2024-08-19T14:40:50.236 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.236 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.236 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:40:50.236 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:40:36.057312+0000", 2024-08-19T14:40:50.237 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.1068000000000001e-05 2024-08-19T14:40:50.237 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:40:50.237 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:40:50.237 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:40:50.237 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.237 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.237 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.26911.0:56796 8.b 8:d82ab3a0:::benchmark_data_smithi045_137103_object56795:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e727)", 2024-08-19T14:40:50.237 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:40:35.886940+0000", 2024-08-19T14:40:50.237 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 14.218709848, 2024-08-19T14:40:50.237 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.17052018799999999, 2024-08-19T14:40:50.237 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:40:50.237 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:40:50.237 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:40:50.238 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:40:50.238 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.26911", 2024-08-19T14:40:50.238 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/3494993995", 2024-08-19T14:40:50.238 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 56796 2024-08-19T14:40:50.238 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.238 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:40:50.238 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.238 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:40:50.238 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:40:35.886940+0000", 2024-08-19T14:40:50.238 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:40:50.238 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.238 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.238 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:40:50.238 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:40:35.886940+0000", 2024-08-19T14:40:50.239 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:40:50.239 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.239 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.239 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:40:50.239 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:40:35.886941+0000", 2024-08-19T14:40:50.239 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.4099999999999998e-07 2024-08-19T14:40:50.239 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.239 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.239 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:40:50.239 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:40:35.886962+0000", 2024-08-19T14:40:50.239 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1029000000000001e-05 2024-08-19T14:40:50.239 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.239 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.239 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:40:50.240 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:40:35.886969+0000", 2024-08-19T14:40:50.240 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4760000000000001e-06 2024-08-19T14:40:50.240 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.240 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.240 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:40:50.240 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:40:35.886975+0000", 2024-08-19T14:40:50.240 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4820000000000002e-06 2024-08-19T14:40:50.240 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.240 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.240 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:40:50.240 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:40:35.887026+0000", 2024-08-19T14:40:50.240 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0822999999999998e-05 2024-08-19T14:40:50.240 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.240 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.241 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:40:50.241 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:40:35.887133+0000", 2024-08-19T14:40:50.241 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000107036 2024-08-19T14:40:50.241 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.241 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.241 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,4", 2024-08-19T14:40:50.241 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:40:35.887271+0000", 2024-08-19T14:40:50.241 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0001381 2024-08-19T14:40:50.241 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.241 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.241 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:40:50.241 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:40:35.889994+0000", 2024-08-19T14:40:50.241 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0027227380000000002 2024-08-19T14:40:50.241 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.242 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.242 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:40:50.242 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:40:35.890497+0000", 2024-08-19T14:40:50.242 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000503174 2024-08-19T14:40:50.242 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.242 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.242 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:40:50.242 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:40:36.057366+0000", 2024-08-19T14:40:50.242 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.166869609 2024-08-19T14:40:50.242 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.242 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.242 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:40:50.242 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:40:36.057420+0000", 2024-08-19T14:40:50.242 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.3368999999999997e-05 2024-08-19T14:40:50.243 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:40:50.243 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:40:50.243 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:40:50.243 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:40:36.057461+0000", 2024-08-19T14:40:50.243 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.0710999999999997e-05 2024-08-19T14:40:50.243 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:40:50.243 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:40:50.243 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:40:50.243 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:40:50.243 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:40:50.243 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:40:50.244 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:40:50.244 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:40:50.256 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:40:50.257 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:40:50.265 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:50.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:50.266 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:40:50.289 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:40:50.290 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:40:50.291 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 39 16 70336 70320 112.677 0 - 0.00845477 2024-08-19T14:40:50.321 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:40:50.322 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:40:50.354 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:40:50.354 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:40:50.367 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:40:50.388 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:40:50.389 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:40:50.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:50.402 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:50.402 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:40:50.431 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:40:50.431 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:40:50.466 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:40:50.466 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:40:50.502 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:40:50.503 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:40:50.503 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:40:50.531 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:50.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:50.532 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:40:50.537 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:40:50.537 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:40:50.576 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:40:50.576 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:40:50.576 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:50 smithi031 ceph-mon[16336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:40:50.576 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:50 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3583300162' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:40:50.576 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:50 smithi031 ceph-mon[16336]: osdmap e737: 12 total, 12 up, 9 in 2024-08-19T14:40:50.577 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:50 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1703557782' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:40:50.608 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:40:50.609 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:40:50.632 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:40:50.689 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:50.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:50.690 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:40:50.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:50 smithi032 ceph-mon[20004]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:40:50.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:50 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3583300162' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:40:50.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:50 smithi032 ceph-mon[20004]: osdmap e737: 12 total, 12 up, 9 in 2024-08-19T14:40:50.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:50 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1703557782' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:40:50.791 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:40:50.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:50 smithi045 ceph-mon[20143]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:40:50.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:50 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3583300162' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:40:50.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:50 smithi045 ceph-mon[20143]: osdmap e737: 12 total, 12 up, 9 in 2024-08-19T14:40:50.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:50 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1703557782' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:40:50.817 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:50.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:50.818 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:40:50.919 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:40:50.946 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:50.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:50.947 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:40:51.048 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:40:51.065 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-08-19T14:40:51.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:51.085 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:51.085 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:40:51.186 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:40:51.229 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:51.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:51.230 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:40:51.291 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:2024-08-19T14:40:51.292133+0000 min lat: 0.00108187 max lat: 2.31637 avg lat: 0.00900898 2024-08-19T14:40:51.292 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-19T14:40:51.292 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 40 16 71022 71006 110.932 21.4375 0.00764607 0.00900898 2024-08-19T14:40:51.330 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:40:51.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:51.359 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:51.360 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:40:51.460 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:40:51.507 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:51.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:51.509 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:40:51.520 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:40:51.610 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:40:51.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:51 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:40:51.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:51 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:40:51.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:51 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:40:51.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:51 smithi031 ceph-mon[16336]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-08-19T14:40:51.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:51 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:40:51.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:51 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:40:51.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:51 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:40:51.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:51 smithi031 ceph-mon[16336]: osdmap e738: 12 total, 12 up, 9 in 2024-08-19T14:40:51.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:51 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:40:51.639 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:51.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:51.640 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:40:51.741 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:40:51.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:51 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:40:51.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:51 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:40:51.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:51 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:40:51.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:51 smithi032 ceph-mon[20004]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-08-19T14:40:51.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:51 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:40:51.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:51 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:40:51.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:51 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:40:51.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:51 smithi032 ceph-mon[20004]: osdmap e738: 12 total, 12 up, 9 in 2024-08-19T14:40:51.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:51 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:40:51.762 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:51 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:40:51.762 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:51 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:40:51.762 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:51 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:40:51.762 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:51 smithi045 ceph-mon[20143]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-08-19T14:40:51.762 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:51 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:40:51.762 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:51 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:40:51.762 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:51 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:40:51.762 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:51 smithi045 ceph-mon[20143]: osdmap e738: 12 total, 12 up, 9 in 2024-08-19T14:40:51.762 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:51 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:40:51.779 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:51.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:51.780 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:40:51.881 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:40:51.910 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:51.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:51.911 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:40:52.012 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:40:52.046 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:52.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:52.047 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:40:52.148 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:40:52.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:52.183 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:52.183 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:40:52.193 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:40:52.193 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:40:52.284 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:40:52.292 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 41 16 71918 71902 109.592 56 0.0275557 0.00911956 2024-08-19T14:40:52.313 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:52.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:52.314 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:40:52.416 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:40:52.446 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:52.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:52.447 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:40:52.548 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:40:52.575 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:52.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:52.576 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:40:52.677 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:40:52.703 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-19T14:40:52.724 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:52.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:52.725 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:40:52.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:52 smithi032 ceph-mon[20004]: osdmap e739: 12 total, 12 up, 9 in 2024-08-19T14:40:52.764 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:52 smithi031 ceph-mon[16336]: osdmap e739: 12 total, 12 up, 9 in 2024-08-19T14:40:52.765 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:52 smithi045 ceph-mon[20143]: osdmap e739: 12 total, 12 up, 9 in 2024-08-19T14:40:52.827 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:40:52.865 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:52.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:52.867 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:40:52.885 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-08-19T14:40:52.967 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:40:53.006 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:53.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:53.007 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:40:53.108 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:40:53.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:53.136 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:53.136 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:40:53.237 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:40:53.286 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:40:53.292 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 42 16 72708 72692 108.157 49.375 0.00801767 0.00923869 2024-08-19T14:40:53.298 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:53.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:53.300 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:40:53.307 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:40:53.366 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:40:53.401 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:40:53.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:53.467 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:53.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:40:53.568 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:40:53.595 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:53.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:53.596 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:40:53.697 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:40:53.733 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:53 smithi045 ceph-mon[20143]: pgmap v1768: 57 pgs: 1 active+undersized+remapped, 1 peering, 2 active+recovering+remapped, 11 active+recovering+undersized+remapped, 1 unknown, 3 remapped+peering, 38 active+clean; 4.3 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 511 B/s rd, 0 op/s; 46106/212211 objects misplaced (21.726%); 30 MiB/s, 484 objects/s recovering 2024-08-19T14:40:53.733 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:53 smithi045 ceph-mon[20143]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 4380/201423 objects degraded (2.175%), 1 pg degraded) 2024-08-19T14:40:53.733 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:53 smithi045 ceph-mon[20143]: osdmap e740: 12 total, 12 up, 9 in 2024-08-19T14:40:53.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:53 smithi032 ceph-mon[20004]: pgmap v1768: 57 pgs: 1 active+undersized+remapped, 1 peering, 2 active+recovering+remapped, 11 active+recovering+undersized+remapped, 1 unknown, 3 remapped+peering, 38 active+clean; 4.3 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 511 B/s rd, 0 op/s; 46106/212211 objects misplaced (21.726%); 30 MiB/s, 484 objects/s recovering 2024-08-19T14:40:53.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:53 smithi032 ceph-mon[20004]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 4380/201423 objects degraded (2.175%), 1 pg degraded) 2024-08-19T14:40:53.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:53 smithi032 ceph-mon[20004]: osdmap e740: 12 total, 12 up, 9 in 2024-08-19T14:40:53.760 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:53.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:53.761 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:40:53.862 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:40:53.868 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:53 smithi031 ceph-mon[16336]: pgmap v1768: 57 pgs: 1 active+undersized+remapped, 1 peering, 2 active+recovering+remapped, 11 active+recovering+undersized+remapped, 1 unknown, 3 remapped+peering, 38 active+clean; 4.3 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 511 B/s rd, 0 op/s; 46106/212211 objects misplaced (21.726%); 30 MiB/s, 484 objects/s recovering 2024-08-19T14:40:53.868 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:53 smithi031 ceph-mon[16336]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 4380/201423 objects degraded (2.175%), 1 pg degraded) 2024-08-19T14:40:53.868 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:53 smithi031 ceph-mon[16336]: osdmap e740: 12 total, 12 up, 9 in 2024-08-19T14:40:53.909 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:53.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:53.910 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:40:54.011 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:40:54.043 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:54.044 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:40:54.044 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:40:54.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:54.045 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:40:54.139 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 4, 5, 7, 8, 2, 11, 9] out_osds: [10, 6, 3] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 8, 9, 11, 10, 2] 2024-08-19T14:40:54.139 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-19T14:40:54.139 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-08-19T14:40:54.139 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-08-19T14:40:54.139 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-08-19T14:40:54.140 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.5 config set filestore_inject_stall 3 2024-08-19T14:40:54.146 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:40:54.178 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:54.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:54.179 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:40:54.280 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:40:54.292 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 43 16 73447 73431 106.716 46.1875 0.0150836 0.00936473 2024-08-19T14:40:54.312 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:54.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:54.313 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:40:54.414 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:40:54.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:54.439 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:54.440 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:40:54.541 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:40:54.579 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:54.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:54.580 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:40:54.681 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:40:54.716 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:54.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:54.717 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:40:54.770 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:40:54.817 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:40:54.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:54.855 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:54.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:40:54.911 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-08-19T14:40:54.956 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:40:54.990 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:54.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:54.991 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:40:55.093 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:40:55.128 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:55.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:55.129 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:40:55.230 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:40:55.258 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:55.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:55.259 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:40:55.292 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 44 16 74200 74184 105.36 47.0625 0.0237997 0.00948294 2024-08-19T14:40:55.321 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:40:55.360 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:40:55.400 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:55.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:55.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:40:55.502 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:40:55.534 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:55.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:55.535 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:40:55.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:55 smithi031 ceph-mon[16336]: pgmap v1770: 57 pgs: 1 active+undersized+remapped, 2 active+recovering+remapped, 11 active+recovering+undersized+remapped, 2 remapped+peering, 41 active+clean; 4.3 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 420 B/s rd, 25 MiB/s wr, 395 op/s; 44666/213711 objects misplaced (20.900%); 33 MiB/s, 528 objects/s recovering 2024-08-19T14:40:55.636 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:40:55.664 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:55.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:55.665 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:40:55.726 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:55 smithi032 ceph-mon[20004]: pgmap v1770: 57 pgs: 1 active+undersized+remapped, 2 active+recovering+remapped, 11 active+recovering+undersized+remapped, 2 remapped+peering, 41 active+clean; 4.3 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 420 B/s rd, 25 MiB/s wr, 395 op/s; 44666/213711 objects misplaced (20.900%); 33 MiB/s, 528 objects/s recovering 2024-08-19T14:40:55.766 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:40:55.767 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:55 smithi045 ceph-mon[20143]: pgmap v1770: 57 pgs: 1 active+undersized+remapped, 2 active+recovering+remapped, 11 active+recovering+undersized+remapped, 2 remapped+peering, 41 active+clean; 4.3 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 420 B/s rd, 25 MiB/s wr, 395 op/s; 44666/213711 objects misplaced (20.900%); 33 MiB/s, 528 objects/s recovering 2024-08-19T14:40:55.803 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:55.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:55.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:40:55.882 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:40:55.882 INFO:teuthology.orchestra.run.smithi032.stdout: "success": "" 2024-08-19T14:40:55.882 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:40:55.905 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:40:55.923 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:40:55.923 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:40:55.935 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:55.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:55.936 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:40:56.037 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:40:56.069 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:56.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:56.070 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:40:56.171 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:40:56.205 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:56.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:56.206 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:40:56.292 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 45 16 74932 74916 104.036 45.75 0.00507906 0.00960469 2024-08-19T14:40:56.307 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:40:56.331 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:56.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:56.332 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:40:56.433 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:40:56.464 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:56.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:56.465 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:40:56.566 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:40:56.621 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:56.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:56.622 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:40:56.637 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-08-19T14:40:56.723 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:40:56.748 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:56.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:56.750 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:40:56.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:56 smithi032 ceph-mon[20004]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-08-19T14:40:56.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:56 smithi045 ceph-mon[20143]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-08-19T14:40:56.851 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:40:56.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:56 smithi031 ceph-mon[16336]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-08-19T14:40:56.889 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:56.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:56.890 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:40:56.991 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:40:57.020 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:40:57.023 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:57.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:57.024 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:40:57.125 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:40:57.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:57.154 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:57.155 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:40:57.255 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:40:57.287 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:57.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:57.288 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:40:57.292 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 46 16 75556 75540 102.622 39 0.0112436 0.00973842 2024-08-19T14:40:57.389 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:40:57.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:57.420 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:57.420 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:40:57.521 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:40:57.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:57.551 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:57.551 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:40:57.653 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:40:57.684 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:57.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:57.685 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:40:57.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:57 smithi032 ceph-mon[20004]: pgmap v1771: 57 pgs: 2 active+recovering+remapped, 12 active+recovering+undersized+remapped, 43 active+clean; 4.5 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 341 B/s rd, 41 MiB/s wr, 653 op/s; 47549/219711 objects misplaced (21.642%); 38 MiB/s, 611 objects/s recovering 2024-08-19T14:40:57.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:57 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/325728069' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:40:57.786 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:40:57.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:57 smithi045 ceph-mon[20143]: pgmap v1771: 57 pgs: 2 active+recovering+remapped, 12 active+recovering+undersized+remapped, 43 active+clean; 4.5 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 341 B/s rd, 41 MiB/s wr, 653 op/s; 47549/219711 objects misplaced (21.642%); 38 MiB/s, 611 objects/s recovering 2024-08-19T14:40:57.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:57 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/325728069' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:40:57.819 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is unset 2024-08-19T14:40:57.820 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:57 smithi031 ceph-mon[16336]: pgmap v1771: 57 pgs: 2 active+recovering+remapped, 12 active+recovering+undersized+remapped, 43 active+clean; 4.5 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 341 B/s rd, 41 MiB/s wr, 653 op/s; 47549/219711 objects misplaced (21.642%); 38 MiB/s, 611 objects/s recovering 2024-08-19T14:40:57.821 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:57 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/325728069' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:40:57.830 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:57.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:57.831 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:40:57.932 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:40:57.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:57.967 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:57.967 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:40:58.068 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:40:58.111 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:58.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:58.112 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:40:58.213 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:40:58.237 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:58.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:58.238 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:40:58.292 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 47 16 76317 76301 101.45 47.5625 0.0029167 0.00985162 2024-08-19T14:40:58.338 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:40:58.368 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:58.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:58.369 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:40:58.470 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:40:58.499 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:58.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:58.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:40:58.584 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:40:58.584 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:40:58.600 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:40:58.630 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:58.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:58.631 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:40:58.732 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:40:58.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:58.774 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:58.774 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:40:58.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:58 smithi045 ceph-mon[20143]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:40:58.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:58 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/325728069' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:40:58.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:58 smithi045 ceph-mon[20143]: osdmap e741: 12 total, 12 up, 9 in 2024-08-19T14:40:58.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:58 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:40:58.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:58 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:40:58.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:58 smithi031 ceph-mon[16336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:40:58.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:58 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/325728069' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:40:58.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:58 smithi031 ceph-mon[16336]: osdmap e741: 12 total, 12 up, 9 in 2024-08-19T14:40:58.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:58 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:40:58.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:58 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:40:58.874 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:40:58.905 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:58.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:58.907 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:40:59.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:58 smithi032 ceph-mon[20004]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:40:59.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:58 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/325728069' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:40:59.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:58 smithi032 ceph-mon[20004]: osdmap e741: 12 total, 12 up, 9 in 2024-08-19T14:40:59.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:58 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:40:59.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:58 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:40:59.008 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:40:59.036 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:59.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:59.038 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:40:59.139 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:40:59.167 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:59.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:59.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:40:59.269 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:40:59.292 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 48 16 77173 77157 100.451 53.5 0.00998933 0.00994924 2024-08-19T14:40:59.298 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:59.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:59.299 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:40:59.400 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:40:59.416 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-08-19T14:40:59.439 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:40:59.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:59.440 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:40:59.541 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:40:59.567 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:59.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:59.568 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:40:59.669 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:40:59.691 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:40:59.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:59.692 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:40:59.786 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:40:59.794 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:40:59.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:59 smithi045 ceph-mon[20143]: pgmap v1773: 57 pgs: 2 active+recovering+remapped, 12 active+recovering+undersized+remapped, 43 active+clean; 4.5 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 40 MiB/s wr, 637 op/s; 45492/223245 objects misplaced (20.378%); 41 MiB/s, 660 objects/s recovering 2024-08-19T14:40:59.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:40:59 smithi045 ceph-mon[20143]: osdmap e742: 12 total, 12 up, 9 in 2024-08-19T14:40:59.824 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:59.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:59.825 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:40:59.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:59 smithi031 ceph-mon[16336]: pgmap v1773: 57 pgs: 2 active+recovering+remapped, 12 active+recovering+undersized+remapped, 43 active+clean; 4.5 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 40 MiB/s wr, 637 op/s; 45492/223245 objects misplaced (20.378%); 41 MiB/s, 660 objects/s recovering 2024-08-19T14:40:59.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:40:59 smithi031 ceph-mon[16336]: osdmap e742: 12 total, 12 up, 9 in 2024-08-19T14:40:59.926 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:40:59.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:40:59.953 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:40:59.953 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:41:00.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:59 smithi032 ceph-mon[20004]: pgmap v1773: 57 pgs: 2 active+recovering+remapped, 12 active+recovering+undersized+remapped, 43 active+clean; 4.5 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 40 MiB/s wr, 637 op/s; 45492/223245 objects misplaced (20.378%); 41 MiB/s, 660 objects/s recovering 2024-08-19T14:41:00.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:40:59 smithi032 ceph-mon[20004]: osdmap e742: 12 total, 12 up, 9 in 2024-08-19T14:41:00.054 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:41:00.083 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:00.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:00.084 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:41:00.185 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:41:00.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:00.224 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:00.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:41:00.293 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 49 16 77759 77743 99.1488 36.625 0.0166101 0.00999646 2024-08-19T14:41:00.326 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:41:00.329 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:41:00.329 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:41:00.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:00.351 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:00.351 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:41:00.452 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:41:00.476 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:00.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:00.477 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:41:00.578 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:41:00.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:00.623 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:00.623 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:41:00.725 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:41:00.765 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:00.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:00.766 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:41:00.867 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:41:00.868 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:00 smithi031 ceph-mon[16336]: osdmap e743: 12 total, 12 up, 9 in 2024-08-19T14:41:00.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:00.899 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:00.899 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:41:00.952 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:00 smithi045 ceph-mon[20143]: osdmap e743: 12 total, 12 up, 9 in 2024-08-19T14:41:01.000 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:41:01.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:00 smithi032 ceph-mon[20004]: osdmap e743: 12 total, 12 up, 9 in 2024-08-19T14:41:01.013 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-08-19T14:41:01.041 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:01.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:01.042 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:41:01.143 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:41:01.170 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:01.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:01.172 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:41:01.273 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:41:01.294 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 50 16 78427 78411 97.9981 41.75 0.0205794 0.0101978 2024-08-19T14:41:01.315 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:01.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:01.316 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:41:01.370 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:41:01.416 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:41:01.441 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:01.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:01.442 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:41:01.544 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:41:01.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:01.570 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:01.571 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:41:01.598 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 4, 5, 7, 8, 2, 11, 9] out_osds: [10, 6, 3] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 8, 9, 11, 10, 2] 2024-08-19T14:41:01.598 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-19T14:41:01.599 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-08-19T14:41:01.599 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-19T14:41:01.671 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:41:01.694 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:01.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:01.696 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:41:01.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:01 smithi045 ceph-mon[20143]: pgmap v1776: 57 pgs: 2 active+recovering+remapped, 12 active+recovering+undersized+remapped, 43 active+clean; 4.6 GiB data, 21 GiB used, 783 GiB / 805 GiB avail; 40 MiB/s wr, 642 op/s; 44508/225282 objects misplaced (19.757%); 32 MiB/s, 516 objects/s recovering 2024-08-19T14:41:01.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:01 smithi045 ceph-mon[20143]: osdmap e744: 12 total, 12 up, 9 in 2024-08-19T14:41:01.797 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:41:01.829 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:01.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:01.831 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:41:01.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:01 smithi031 ceph-mon[16336]: pgmap v1776: 57 pgs: 2 active+recovering+remapped, 12 active+recovering+undersized+remapped, 43 active+clean; 4.6 GiB data, 21 GiB used, 783 GiB / 805 GiB avail; 40 MiB/s wr, 642 op/s; 44508/225282 objects misplaced (19.757%); 32 MiB/s, 516 objects/s recovering 2024-08-19T14:41:01.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:01 smithi031 ceph-mon[16336]: osdmap e744: 12 total, 12 up, 9 in 2024-08-19T14:41:01.931 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:41:01.938 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:41:01.939 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:41:01.958 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:01.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:01.959 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:41:02.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:01 smithi032 ceph-mon[20004]: pgmap v1776: 57 pgs: 2 active+recovering+remapped, 12 active+recovering+undersized+remapped, 43 active+clean; 4.6 GiB data, 21 GiB used, 783 GiB / 805 GiB avail; 40 MiB/s wr, 642 op/s; 44508/225282 objects misplaced (19.757%); 32 MiB/s, 516 objects/s recovering 2024-08-19T14:41:02.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:01 smithi032 ceph-mon[20004]: osdmap e744: 12 total, 12 up, 9 in 2024-08-19T14:41:02.060 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:41:02.103 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:02.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:02.104 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:41:02.205 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:41:02.232 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:02.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:02.233 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:41:02.294 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 51 16 78980 78964 96.7543 34.5625 0.00425093 0.0103299 2024-08-19T14:41:02.334 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:41:02.363 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:02.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:02.364 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:41:02.465 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:41:02.493 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:02.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:02.494 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:41:02.595 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:41:02.626 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:02.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:02.628 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:41:02.729 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:41:02.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:02.756 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:02.757 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:41:02.858 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:41:02.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:02 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:41:02.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:02 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:41:02.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:02 smithi031 ceph-mon[16336]: osdmap e745: 12 total, 12 up, 9 in 2024-08-19T14:41:02.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:02.883 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:02.883 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:41:02.984 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:41:03.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:02 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:41:03.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:02 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:41:03.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:02 smithi032 ceph-mon[20004]: osdmap e745: 12 total, 12 up, 9 in 2024-08-19T14:41:03.021 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:03.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:03.023 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:41:03.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:02 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:41:03.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:02 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:41:03.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:02 smithi045 ceph-mon[20143]: osdmap e745: 12 total, 12 up, 9 in 2024-08-19T14:41:03.123 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:41:03.163 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:03.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:03.164 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:41:03.265 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:41:03.294 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 52 16 79559 79543 95.5896 36.1875 0.00727918 0.0104551 2024-08-19T14:41:03.296 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:03.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:03.297 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:41:03.398 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:41:03.438 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:03.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:03.439 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:41:03.540 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:41:03.585 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:03.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:03.586 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:41:03.687 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:41:03.713 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:03.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:03.714 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:41:03.815 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:41:03.837 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:03.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:03.838 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:41:03.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:03 smithi031 ceph-mon[16336]: pgmap v1779: 57 pgs: 2 active+recovering+remapped, 11 active+recovering+undersized+remapped, 44 active+clean; 4.8 GiB data, 22 GiB used, 783 GiB / 805 GiB avail; 54 MiB/s wr, 866 op/s; 40526/233646 objects misplaced (17.345%); 51 MiB/s, 820 objects/s recovering 2024-08-19T14:41:03.939 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:41:03.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:03.972 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:03.972 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:41:04.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:03 smithi032 ceph-mon[20004]: pgmap v1779: 57 pgs: 2 active+recovering+remapped, 11 active+recovering+undersized+remapped, 44 active+clean; 4.8 GiB data, 22 GiB used, 783 GiB / 805 GiB avail; 54 MiB/s wr, 866 op/s; 40526/233646 objects misplaced (17.345%); 51 MiB/s, 820 objects/s recovering 2024-08-19T14:41:04.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:03 smithi045 ceph-mon[20143]: pgmap v1779: 57 pgs: 2 active+recovering+remapped, 11 active+recovering+undersized+remapped, 44 active+clean; 4.8 GiB data, 22 GiB used, 783 GiB / 805 GiB avail; 54 MiB/s wr, 866 op/s; 40526/233646 objects misplaced (17.345%); 51 MiB/s, 820 objects/s recovering 2024-08-19T14:41:04.073 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:41:04.097 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:04.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:04.098 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:41:04.199 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:41:04.240 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:04.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:04.241 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:41:04.294 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 53 16 80409 80393 94.7883 53.125 0.00444608 0.0105451 2024-08-19T14:41:04.342 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:41:04.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:04.364 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:04.364 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:41:04.465 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:41:04.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:04.492 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:04.493 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:41:04.593 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:41:04.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:04.622 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:04.623 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:41:04.724 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:41:04.751 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:04.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:04.753 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:41:04.854 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:41:04.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:04.895 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:04.895 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:41:04.996 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:41:05.026 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:05.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:05.028 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:41:05.129 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:41:05.150 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:05.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:05.152 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:41:05.253 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:41:05.281 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:05.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:05.282 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:41:05.294 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 54 16 81242 81226 93.997 52.0625 0.0373201 0.0106327 2024-08-19T14:41:05.383 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:41:05.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:05.406 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:05.406 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:41:05.478 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-08-19T14:41:05.507 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:41:05.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:05.537 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:05.537 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:41:05.638 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:41:05.667 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:05.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:05.668 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:41:05.769 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:41:05.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:05 smithi045 ceph-mon[20143]: pgmap v1780: 57 pgs: 2 active+recovering+remapped, 11 active+recovering+undersized+remapped, 44 active+clean; 4.8 GiB data, 22 GiB used, 782 GiB / 805 GiB avail; 38 MiB/s wr, 609 op/s; 39106/235506 objects misplaced (16.605%); 38 MiB/s, 602 objects/s recovering 2024-08-19T14:41:05.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:05.798 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:05.798 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:41:05.808 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:41:05.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:05 smithi031 ceph-mon[16336]: pgmap v1780: 57 pgs: 2 active+recovering+remapped, 11 active+recovering+undersized+remapped, 44 active+clean; 4.8 GiB data, 22 GiB used, 782 GiB / 805 GiB avail; 38 MiB/s wr, 609 op/s; 39106/235506 objects misplaced (16.605%); 38 MiB/s, 602 objects/s recovering 2024-08-19T14:41:05.899 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:41:05.924 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:05.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:05.925 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:41:06.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:05 smithi032 ceph-mon[20004]: pgmap v1780: 57 pgs: 2 active+recovering+remapped, 11 active+recovering+undersized+remapped, 44 active+clean; 4.8 GiB data, 22 GiB used, 782 GiB / 805 GiB avail; 38 MiB/s wr, 609 op/s; 39106/235506 objects misplaced (16.605%); 38 MiB/s, 602 objects/s recovering 2024-08-19T14:41:06.026 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:41:06.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:06.050 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:06.050 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:41:06.152 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:41:06.177 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:06.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:06.178 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:41:06.279 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:41:06.294 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 55 16 81863 81847 92.9937 38.8125 0.00825577 0.0106944 2024-08-19T14:41:06.329 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:41:06.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:06.361 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:06.361 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:41:06.362 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:41:06.462 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:41:06.484 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:06.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:06.486 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:41:06.587 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:41:06.626 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:06.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:06.628 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:41:06.729 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:41:06.755 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:06.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:06.756 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:41:06.858 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:41:06.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:06 smithi031 ceph-mon[16336]: osdmap e746: 12 total, 12 up, 9 in 2024-08-19T14:41:06.884 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:06.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:06.885 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:41:06.970 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:06 smithi045 ceph-mon[20143]: osdmap e746: 12 total, 12 up, 9 in 2024-08-19T14:41:06.986 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:41:07.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:06 smithi032 ceph-mon[20004]: osdmap e746: 12 total, 12 up, 9 in 2024-08-19T14:41:07.014 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:07.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:07.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:41:07.117 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:41:07.143 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:07.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:07.144 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:41:07.245 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:41:07.282 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:07.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:07.283 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:41:07.295 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 56 16 82459 82443 91.9982 37.25 0.0326695 0.0108596 2024-08-19T14:41:07.383 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:41:07.411 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:07.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:07.412 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:41:07.513 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:41:07.544 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:07.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:07.545 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:41:07.646 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:41:07.687 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:07.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:07.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:41:07.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:07 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:41:07.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:07 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:41:07.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:07 smithi032 ceph-mon[20004]: pgmap v1782: 57 pgs: 2 active+recovering+remapped, 11 active+recovering+undersized+remapped, 44 active+clean; 4.9 GiB data, 22 GiB used, 782 GiB / 805 GiB avail; 54 MiB/s wr, 859 op/s; 37032/240756 objects misplaced (15.382%); 51 MiB/s, 819 objects/s recovering 2024-08-19T14:41:07.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:07 smithi032 ceph-mon[20004]: osdmap e747: 12 total, 12 up, 9 in 2024-08-19T14:41:07.788 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:41:07.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:07.819 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:07.819 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:41:07.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:07 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:41:07.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:07 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:41:07.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:07 smithi031 ceph-mon[16336]: pgmap v1782: 57 pgs: 2 active+recovering+remapped, 11 active+recovering+undersized+remapped, 44 active+clean; 4.9 GiB data, 22 GiB used, 782 GiB / 805 GiB avail; 54 MiB/s wr, 859 op/s; 37032/240756 objects misplaced (15.382%); 51 MiB/s, 819 objects/s recovering 2024-08-19T14:41:07.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:07 smithi031 ceph-mon[16336]: osdmap e747: 12 total, 12 up, 9 in 2024-08-19T14:41:07.920 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:41:07.948 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:07.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:07.949 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:41:08.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:07 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:41:08.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:07 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:41:08.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:07 smithi045 ceph-mon[20143]: pgmap v1782: 57 pgs: 2 active+recovering+remapped, 11 active+recovering+undersized+remapped, 44 active+clean; 4.9 GiB data, 22 GiB used, 782 GiB / 805 GiB avail; 54 MiB/s wr, 859 op/s; 37032/240756 objects misplaced (15.382%); 51 MiB/s, 819 objects/s recovering 2024-08-19T14:41:08.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:07 smithi045 ceph-mon[20143]: osdmap e747: 12 total, 12 up, 9 in 2024-08-19T14:41:08.050 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:41:08.078 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:08.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:08.079 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:41:08.180 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:41:08.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:08.253 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:08.253 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:41:08.295 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 57 16 83214 83198 91.212 47.1875 0.0170913 0.0109528 2024-08-19T14:41:08.320 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-19T14:41:08.354 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:41:08.382 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:08.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:08.383 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:41:08.484 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:41:08.508 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:08.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:08.509 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:41:08.610 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:41:08.634 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:08.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:08.635 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:41:08.736 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:41:08.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:08.762 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:08.763 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:41:08.863 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:41:08.892 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:08.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:08.893 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:41:08.993 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:41:09.023 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:09.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:09.024 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:41:09.125 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:41:09.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:09.152 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:09.152 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:41:09.253 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:41:09.283 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:09.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:09.284 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:41:09.295 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 58 16 83970 83954 90.4541 47.25 0.0123725 0.0110504 2024-08-19T14:41:09.386 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:41:09.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:09.413 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:09.413 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:41:09.513 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:41:09.547 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:09.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:09.548 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:41:09.649 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:41:09.680 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:09.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:09.681 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:41:09.782 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:41:09.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:09 smithi045 ceph-mon[20143]: pgmap v1784: 57 pgs: 2 active+recovering+remapped, 10 active+recovering+undersized+remapped, 45 active+clean; 5.0 GiB data, 23 GiB used, 782 GiB / 805 GiB avail; 41 MiB/s wr, 652 op/s; 35234/244110 objects misplaced (14.434%); 42 MiB/s, 670 objects/s recovering 2024-08-19T14:41:09.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:09 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:41:09.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:09 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:41:09.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:09 smithi045 ceph-mon[20143]: osdmap e748: 12 total, 12 up, 9 in 2024-08-19T14:41:09.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:09.804 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:09.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:41:09.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:09 smithi031 ceph-mon[16336]: pgmap v1784: 57 pgs: 2 active+recovering+remapped, 10 active+recovering+undersized+remapped, 45 active+clean; 5.0 GiB data, 23 GiB used, 782 GiB / 805 GiB avail; 41 MiB/s wr, 652 op/s; 35234/244110 objects misplaced (14.434%); 42 MiB/s, 670 objects/s recovering 2024-08-19T14:41:09.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:09 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:41:09.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:09 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:41:09.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:09 smithi031 ceph-mon[16336]: osdmap e748: 12 total, 12 up, 9 in 2024-08-19T14:41:09.905 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:41:09.941 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:09.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:09.942 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:41:10.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:09 smithi032 ceph-mon[20004]: pgmap v1784: 57 pgs: 2 active+recovering+remapped, 10 active+recovering+undersized+remapped, 45 active+clean; 5.0 GiB data, 23 GiB used, 782 GiB / 805 GiB avail; 41 MiB/s wr, 652 op/s; 35234/244110 objects misplaced (14.434%); 42 MiB/s, 670 objects/s recovering 2024-08-19T14:41:10.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:09 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:41:10.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:09 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:41:10.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:09 smithi032 ceph-mon[20004]: osdmap e748: 12 total, 12 up, 9 in 2024-08-19T14:41:10.043 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:41:10.069 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:10.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:10.070 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:41:10.171 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:41:10.209 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:10.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:10.210 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:41:10.295 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 59 16 84750 84734 89.7472 48.75 0.0110854 0.0111371 2024-08-19T14:41:10.310 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:41:10.338 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:10.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:10.340 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:41:10.357 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:41:10.358 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:41:10.441 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:41:10.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:10.475 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:10.475 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:41:10.576 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:41:10.608 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:10.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:10.609 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:41:10.710 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:41:10.734 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:10.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:10.735 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:41:10.836 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:41:10.862 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:10.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:10.863 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:41:10.964 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:41:11.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:11.011 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:11.011 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:41:11.112 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:41:11.158 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:11.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:11.159 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:41:11.260 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:41:11.295 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:2024-08-19T14:41:11.295619+0000 min lat: 0.00108187 max lat: 2.31637 avg lat: 0.0112476 2024-08-19T14:41:11.295 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-19T14:41:11.295 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 60 16 85330 85314 88.8556 36.25 0.0149598 0.0112476 2024-08-19T14:41:11.300 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:11.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:11.301 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:41:11.402 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:41:11.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:11.430 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:11.431 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:41:11.532 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:41:11.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:11.577 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:11.577 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:41:11.678 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:41:11.708 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:11.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:11.709 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:41:11.811 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:41:11.841 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:11.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:11.842 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:41:11.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:11 smithi031 ceph-mon[16336]: pgmap v1786: 57 pgs: 2 active+recovering+remapped, 10 active+recovering+undersized+remapped, 45 active+clean; 5.0 GiB data, 23 GiB used, 782 GiB / 805 GiB avail; 37 MiB/s wr, 597 op/s; 33953/246255 objects misplaced (13.788%); 33 MiB/s, 523 objects/s recovering 2024-08-19T14:41:11.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:11 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2472947774' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:41:11.943 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:41:11.964 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:11.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:11.965 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:41:12.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:11 smithi032 ceph-mon[20004]: pgmap v1786: 57 pgs: 2 active+recovering+remapped, 10 active+recovering+undersized+remapped, 45 active+clean; 5.0 GiB data, 23 GiB used, 782 GiB / 805 GiB avail; 37 MiB/s wr, 597 op/s; 33953/246255 objects misplaced (13.788%); 33 MiB/s, 523 objects/s recovering 2024-08-19T14:41:12.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:11 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2472947774' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:41:12.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:11 smithi045 ceph-mon[20143]: pgmap v1786: 57 pgs: 2 active+recovering+remapped, 10 active+recovering+undersized+remapped, 45 active+clean; 5.0 GiB data, 23 GiB used, 782 GiB / 805 GiB avail; 37 MiB/s wr, 597 op/s; 33953/246255 objects misplaced (13.788%); 33 MiB/s, 523 objects/s recovering 2024-08-19T14:41:12.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:11 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2472947774' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:41:12.065 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:41:12.094 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:12.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:12.095 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:41:12.176 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:41:12.177 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:41:12.177 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:41:12.177 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:41:12.177 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:41:12.177 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:41:12.177 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:41:12.177 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:41:12.178 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:41:12.178 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:41:12.178 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:41:12.178 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:41:12.178 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:41:12.178 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:41:12.178 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:41:12.178 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:41:12.179 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:41:12.179 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:41:12.179 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:41:12.179 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:41:12.179 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:41:12.179 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:41:12.179 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:41:12.179 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:41:12.195 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:41:12.227 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:12.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:12.228 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:41:12.295 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 61 16 86153 86137 88.2421 51.4375 0.0203582 0.0113204 2024-08-19T14:41:12.329 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:41:12.357 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:12.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:12.358 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:41:12.459 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:41:12.497 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:12.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:12.498 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:41:12.599 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:41:12.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:12.627 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:12.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:41:12.728 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:41:12.759 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:12.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:12.760 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:41:12.861 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:41:12.886 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:12.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:12.887 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:41:12.988 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:41:13.013 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:13.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:13.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:41:13.115 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:41:13.155 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:13.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:13.156 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:41:13.257 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:41:13.285 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:13.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:13.286 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:41:13.295 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 62 15 86968 86953 87.6414 51 0.0211666 0.0114049 2024-08-19T14:41:13.387 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:41:13.417 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:13.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:13.418 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:41:13.519 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:41:13.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:13.561 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:13.561 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:41:13.663 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:41:13.705 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:13.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:13.706 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:41:13.807 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:41:13.833 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:13.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:13.834 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:41:13.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:13 smithi031 ceph-mon[16336]: pgmap v1787: 57 pgs: 2 active+recovering+remapped, 10 active+recovering+undersized+remapped, 45 active+clean; 5.2 GiB data, 23 GiB used, 781 GiB / 805 GiB avail; 54 MiB/s wr, 863 op/s; 30884/254775 objects misplaced (12.122%); 44 MiB/s, 696 objects/s recovering 2024-08-19T14:41:13.935 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:41:13.962 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:13.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:13.964 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:41:14.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:13 smithi032 ceph-mon[20004]: pgmap v1787: 57 pgs: 2 active+recovering+remapped, 10 active+recovering+undersized+remapped, 45 active+clean; 5.2 GiB data, 23 GiB used, 781 GiB / 805 GiB avail; 54 MiB/s wr, 863 op/s; 30884/254775 objects misplaced (12.122%); 44 MiB/s, 696 objects/s recovering 2024-08-19T14:41:14.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:13 smithi045 ceph-mon[20143]: pgmap v1787: 57 pgs: 2 active+recovering+remapped, 10 active+recovering+undersized+remapped, 45 active+clean; 5.2 GiB data, 23 GiB used, 781 GiB / 805 GiB avail; 54 MiB/s wr, 863 op/s; 30884/254775 objects misplaced (12.122%); 44 MiB/s, 696 objects/s recovering 2024-08-19T14:41:14.065 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:41:14.091 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:14.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:14.093 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:41:14.194 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:41:14.237 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:14.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:14.238 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:41:14.295 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 63 16 87898 87882 87.1718 58.0625 0.00243247 0.0114637 2024-08-19T14:41:14.339 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:41:14.370 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:14.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:14.372 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:41:14.473 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:41:14.499 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:14.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:14.500 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:41:14.602 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:41:14.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:14.642 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:14.643 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:41:14.744 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:41:14.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:14.771 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:14.771 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:41:14.872 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:41:14.901 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:14.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:14.902 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:41:15.003 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:41:15.029 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:15.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:15.030 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:41:15.131 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:41:15.156 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:15.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:15.157 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:41:15.258 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:41:15.291 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:15.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:15.292 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:41:15.295 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 64 16 88748 88732 86.6398 53.125 0.00862567 0.0115361 2024-08-19T14:41:15.393 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:41:15.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:15.437 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:15.437 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:41:15.538 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:41:15.581 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:15.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:15.583 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:41:15.683 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:41:15.733 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:15.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:15.734 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:41:15.835 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:41:15.861 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:15.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:15.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:41:15.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:15 smithi031 ceph-mon[16336]: pgmap v1788: 57 pgs: 2 active+recovering+remapped, 10 active+recovering+undersized+remapped, 45 active+clean; 5.2 GiB data, 24 GiB used, 781 GiB / 805 GiB avail; 42 MiB/s wr, 667 op/s; 29596/256776 objects misplaced (11.526%); 32 MiB/s, 508 objects/s recovering 2024-08-19T14:41:15.963 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:41:15.990 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:15.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:15.992 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:41:16.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:15 smithi032 ceph-mon[20004]: pgmap v1788: 57 pgs: 2 active+recovering+remapped, 10 active+recovering+undersized+remapped, 45 active+clean; 5.2 GiB data, 24 GiB used, 781 GiB / 805 GiB avail; 42 MiB/s wr, 667 op/s; 29596/256776 objects misplaced (11.526%); 32 MiB/s, 508 objects/s recovering 2024-08-19T14:41:16.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:15 smithi045 ceph-mon[20143]: pgmap v1788: 57 pgs: 2 active+recovering+remapped, 10 active+recovering+undersized+remapped, 45 active+clean; 5.2 GiB data, 24 GiB used, 781 GiB / 805 GiB avail; 42 MiB/s wr, 667 op/s; 29596/256776 objects misplaced (11.526%); 32 MiB/s, 508 objects/s recovering 2024-08-19T14:41:16.093 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:41:16.125 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:16.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:16.126 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:41:16.227 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:41:16.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:16.276 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:16.277 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:41:16.295 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 65 16 89708 89692 86.2299 60 0.00377483 0.0115915 2024-08-19T14:41:16.377 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:41:16.407 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:16.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:16.408 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:41:16.510 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:41:16.534 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:16.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:16.535 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:41:16.636 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:41:16.679 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:16.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:16.680 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:41:16.782 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:41:16.805 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:16.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:16.806 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:41:16.907 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:41:16.932 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:16.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:16.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:41:17.034 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:41:17.064 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:17.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:17.065 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:41:17.166 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:41:17.196 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:17.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:17.198 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:41:17.295 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 66 16 90426 90410 85.6033 44.875 0.0160558 0.0116769 2024-08-19T14:41:17.298 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:41:17.325 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:17.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:17.327 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:41:17.413 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-08-19T14:41:17.427 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:41:17.464 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:17.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:17.465 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:41:17.566 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:41:17.599 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:17.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:17.600 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:41:17.701 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:41:17.733 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:17.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:17.735 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:41:17.751 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:41:17.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:17 smithi045 ceph-mon[20143]: pgmap v1789: 57 pgs: 2 active+recovering+remapped, 10 active+recovering+undersized+remapped, 45 active+clean; 5.4 GiB data, 24 GiB used, 780 GiB / 805 GiB avail; 47 MiB/s wr, 754 op/s; 27523/263043 objects misplaced (10.463%); 33 MiB/s, 532 objects/s recovering 2024-08-19T14:41:17.836 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:41:17.861 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:17.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:17.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:41:17.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:17 smithi031 ceph-mon[16336]: pgmap v1789: 57 pgs: 2 active+recovering+remapped, 10 active+recovering+undersized+remapped, 45 active+clean; 5.4 GiB data, 24 GiB used, 780 GiB / 805 GiB avail; 47 MiB/s wr, 754 op/s; 27523/263043 objects misplaced (10.463%); 33 MiB/s, 532 objects/s recovering 2024-08-19T14:41:17.963 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:41:18.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:17 smithi032 ceph-mon[20004]: pgmap v1789: 57 pgs: 2 active+recovering+remapped, 10 active+recovering+undersized+remapped, 45 active+clean; 5.4 GiB data, 24 GiB used, 780 GiB / 805 GiB avail; 47 MiB/s wr, 754 op/s; 27523/263043 objects misplaced (10.463%); 33 MiB/s, 532 objects/s recovering 2024-08-19T14:41:18.009 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:18.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:18.010 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:41:18.111 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:41:18.141 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:18.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:18.142 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:41:18.243 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:41:18.273 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:18.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:18.274 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:41:18.296 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 67 16 91219 91203 85.0654 49.5625 0.0046445 0.0117481 2024-08-19T14:41:18.375 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:41:18.405 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:18.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:18.407 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:41:18.508 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:41:18.537 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:18.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:18.538 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:41:18.639 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:41:18.663 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:18.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:18.664 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:41:18.765 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:41:18.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:18.811 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:18.811 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:41:18.913 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:41:18.938 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:18.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:18.939 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:41:19.040 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:41:19.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:19.069 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:19.069 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:41:19.170 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:41:19.202 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:19.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:19.203 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:41:19.297 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 68 16 92058 92042 84.5855 52.4375 0.0142419 0.0118167 2024-08-19T14:41:19.304 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:41:19.334 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:19.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:19.336 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:41:19.437 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:41:19.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:19.466 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:19.467 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:41:19.568 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:41:19.594 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:19.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:19.595 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:41:19.696 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:41:19.722 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:19.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:19.724 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:41:19.825 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:41:19.848 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:19.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:19.849 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:41:19.950 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:41:19.976 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:19.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:19.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:41:20.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:19 smithi032 ceph-mon[20004]: pgmap v1790: 57 pgs: 2 active+recovering+remapped, 10 active+recovering+undersized+remapped, 45 active+clean; 5.4 GiB data, 25 GiB used, 780 GiB / 805 GiB avail; 48 MiB/s wr, 769 op/s; 26353/267201 objects misplaced (9.863%); 30 MiB/s, 477 objects/s recovering 2024-08-19T14:41:20.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:19 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:41:20.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:19 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:41:20.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:19 smithi032 ceph-mon[20004]: osdmap e749: 12 total, 12 up, 9 in 2024-08-19T14:41:20.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:19 smithi045 ceph-mon[20143]: pgmap v1790: 57 pgs: 2 active+recovering+remapped, 10 active+recovering+undersized+remapped, 45 active+clean; 5.4 GiB data, 25 GiB used, 780 GiB / 805 GiB avail; 48 MiB/s wr, 769 op/s; 26353/267201 objects misplaced (9.863%); 30 MiB/s, 477 objects/s recovering 2024-08-19T14:41:20.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:19 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:41:20.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:19 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:41:20.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:19 smithi045 ceph-mon[20143]: osdmap e749: 12 total, 12 up, 9 in 2024-08-19T14:41:20.078 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:41:20.103 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:20.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:20.104 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:41:20.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:19 smithi031 ceph-mon[16336]: pgmap v1790: 57 pgs: 2 active+recovering+remapped, 10 active+recovering+undersized+remapped, 45 active+clean; 5.4 GiB data, 25 GiB used, 780 GiB / 805 GiB avail; 48 MiB/s wr, 769 op/s; 26353/267201 objects misplaced (9.863%); 30 MiB/s, 477 objects/s recovering 2024-08-19T14:41:20.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:19 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:41:20.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:19 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:41:20.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:19 smithi031 ceph-mon[16336]: osdmap e749: 12 total, 12 up, 9 in 2024-08-19T14:41:20.204 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:41:20.243 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:20.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:20.244 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:41:20.296 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 69 16 92751 92735 83.9873 43.3125 0.00801417 0.0118346 2024-08-19T14:41:20.345 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:41:20.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:20.370 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:20.370 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:41:20.471 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:41:20.495 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:20.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:20.496 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:41:20.597 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:41:20.623 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:20.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:20.624 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:41:20.725 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:41:20.752 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:20.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:20.753 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:41:20.854 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:41:20.876 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:20.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:20.877 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:41:20.935 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:20 smithi045 ceph-mon[20143]: osdmap e750: 12 total, 12 up, 9 in 2024-08-19T14:41:20.978 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:41:21.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:20 smithi032 ceph-mon[20004]: osdmap e750: 12 total, 12 up, 9 in 2024-08-19T14:41:21.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:21.011 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:21.011 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:41:21.112 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:41:21.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:20 smithi031 ceph-mon[16336]: osdmap e750: 12 total, 12 up, 9 in 2024-08-19T14:41:21.147 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:21.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:21.148 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:41:21.250 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:41:21.295 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:21.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:21.296 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:41:21.296 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 70 16 93256 93240 83.2384 31.5625 0.0164345 0.0120088 2024-08-19T14:41:21.397 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:41:21.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:21.442 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:21.442 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:41:21.543 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:41:21.584 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:21.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:21.585 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:41:21.686 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:41:21.712 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:21.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:21.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:41:21.814 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:41:21.831 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:41:21.831 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [], 2024-08-19T14:41:21.831 INFO:teuthology.orchestra.run.smithi045.stdout: "complaint_time": 30, 2024-08-19T14:41:21.831 INFO:teuthology.orchestra.run.smithi045.stdout: "num_blocked_ops": 0 2024-08-19T14:41:21.831 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:41:21.834 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:21 smithi045 ceph-mon[20143]: pgmap v1793: 57 pgs: 2 active+recovering+remapped, 10 active+recovering+undersized+remapped, 45 active+clean; 5.5 GiB data, 25 GiB used, 779 GiB / 805 GiB avail; 39 MiB/s wr, 620 op/s; 24875/269673 objects misplaced (9.224%); 24 MiB/s, 386 objects/s recovering 2024-08-19T14:41:21.834 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:21 smithi045 ceph-mon[20143]: osdmap e751: 12 total, 12 up, 9 in 2024-08-19T14:41:21.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:21.847 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:21.847 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:41:21.948 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:41:21.978 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:21.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:21.979 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:41:22.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:21 smithi032 ceph-mon[20004]: pgmap v1793: 57 pgs: 2 active+recovering+remapped, 10 active+recovering+undersized+remapped, 45 active+clean; 5.5 GiB data, 25 GiB used, 779 GiB / 805 GiB avail; 39 MiB/s wr, 620 op/s; 24875/269673 objects misplaced (9.224%); 24 MiB/s, 386 objects/s recovering 2024-08-19T14:41:22.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:21 smithi032 ceph-mon[20004]: osdmap e751: 12 total, 12 up, 9 in 2024-08-19T14:41:22.017 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:21 smithi031 ceph-mon[16336]: pgmap v1793: 57 pgs: 2 active+recovering+remapped, 10 active+recovering+undersized+remapped, 45 active+clean; 5.5 GiB data, 25 GiB used, 779 GiB / 805 GiB avail; 39 MiB/s wr, 620 op/s; 24875/269673 objects misplaced (9.224%); 24 MiB/s, 386 objects/s recovering 2024-08-19T14:41:22.017 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:21 smithi031 ceph-mon[16336]: osdmap e751: 12 total, 12 up, 9 in 2024-08-19T14:41:22.080 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:41:22.101 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:22.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:22.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:41:22.203 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:41:22.231 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:22.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:22.232 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:41:22.296 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 71 16 94115 94099 82.8221 53.6875 0.00684415 0.012069 2024-08-19T14:41:22.333 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:41:22.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:22.367 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:22.368 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:41:22.469 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:41:22.496 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:22.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:22.498 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:41:22.541 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-08-19T14:41:22.598 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:41:22.624 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:22.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:22.625 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:41:22.726 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:41:22.747 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:22.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:22.748 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:41:22.850 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:41:22.875 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:22.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:22.876 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:41:22.914 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:41:22.976 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:41:23.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:23.018 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:23.018 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:41:23.120 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:41:23.143 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:23.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:23.144 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:41:23.245 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:41:23.271 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:23.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:23.272 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:41:23.296 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 72 16 94840 94824 82.3012 45.3125 0.0149774 0.0121443 2024-08-19T14:41:23.373 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:41:23.401 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:23.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:23.402 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:41:23.485 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:41:23.485 INFO:teuthology.orchestra.run.smithi045.stdout: "size": 20, 2024-08-19T14:41:23.485 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 600, 2024-08-19T14:41:23.485 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [] 2024-08-19T14:41:23.485 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:41:23.503 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:41:23.534 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:23.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:23.535 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:41:23.636 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:41:23.667 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:23.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:23.668 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:41:23.769 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:41:23.797 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:23.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:23.798 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:41:23.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:23 smithi031 ceph-mon[16336]: pgmap v1795: 57 pgs: 2 active+recovering+remapped, 10 active+recovering+undersized+remapped, 45 active+clean; 5.7 GiB data, 26 GiB used, 779 GiB / 805 GiB avail; 53 MiB/s wr, 854 op/s; 22149/278427 objects misplaced (7.955%); 31 MiB/s, 491 objects/s recovering 2024-08-19T14:41:23.899 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:41:23.926 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:23.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:23.927 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:41:24.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:23 smithi032 ceph-mon[20004]: pgmap v1795: 57 pgs: 2 active+recovering+remapped, 10 active+recovering+undersized+remapped, 45 active+clean; 5.7 GiB data, 26 GiB used, 779 GiB / 805 GiB avail; 53 MiB/s wr, 854 op/s; 22149/278427 objects misplaced (7.955%); 31 MiB/s, 491 objects/s recovering 2024-08-19T14:41:24.031 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:41:24.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:23 smithi045 ceph-mon[20143]: pgmap v1795: 57 pgs: 2 active+recovering+remapped, 10 active+recovering+undersized+remapped, 45 active+clean; 5.7 GiB data, 26 GiB used, 779 GiB / 805 GiB avail; 53 MiB/s wr, 854 op/s; 22149/278427 objects misplaced (7.955%); 31 MiB/s, 491 objects/s recovering 2024-08-19T14:41:24.057 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:24.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:24.059 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:41:24.160 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:41:24.181 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:24.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:24.182 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:41:24.262 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-08-19T14:41:24.283 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:41:24.296 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 73 16 95720 95704 81.9271 55 0.00896129 0.0121998 2024-08-19T14:41:24.314 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:24.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:24.315 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:41:24.416 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:41:24.441 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:24.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:24.442 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:41:24.542 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:41:24.570 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:24.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:24.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:41:24.608 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:41:24.609 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-08-19T14:41:24.672 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:41:24.700 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:24.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:24.701 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:41:24.802 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:41:24.829 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:24.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:24.830 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:41:24.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:24 smithi031 ceph-mon[16336]: from='client.27001 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:41:24.931 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:41:24.959 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:24.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:24.960 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:41:25.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:24 smithi032 ceph-mon[20004]: from='client.27001 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:41:25.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:24 smithi045 ceph-mon[20143]: from='client.27001 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:41:25.061 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:41:25.094 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:25.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:25.095 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:41:25.196 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:41:25.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:25.241 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:25.241 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:41:25.296 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 74 16 96658 96642 81.6122 58.625 0.0211038 0.0122463 2024-08-19T14:41:25.342 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:41:25.394 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:25.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:25.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:41:25.496 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:41:25.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:25.527 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:25.527 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:41:25.628 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:41:25.650 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:25.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:25.651 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:41:25.752 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:41:25.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:25.782 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:25.783 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:41:25.884 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:41:25.915 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:25.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:25.916 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:41:26.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:25 smithi032 ceph-mon[20004]: pgmap v1796: 57 pgs: 2 active+recovering+remapped, 9 active+recovering+undersized+remapped, 46 active+clean; 5.7 GiB data, 26 GiB used, 779 GiB / 805 GiB avail; 47 MiB/s wr, 756 op/s; 19088/280815 objects misplaced (6.797%); 40 MiB/s, 640 objects/s recovering 2024-08-19T14:41:26.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:25 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:41:26.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:25 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:41:26.016 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:41:26.045 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:26.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:26.046 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:41:26.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:25 smithi045 ceph-mon[20143]: pgmap v1796: 57 pgs: 2 active+recovering+remapped, 9 active+recovering+undersized+remapped, 46 active+clean; 5.7 GiB data, 26 GiB used, 779 GiB / 805 GiB avail; 47 MiB/s wr, 756 op/s; 19088/280815 objects misplaced (6.797%); 40 MiB/s, 640 objects/s recovering 2024-08-19T14:41:26.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:25 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:41:26.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:25 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:41:26.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:25 smithi031 ceph-mon[16336]: pgmap v1796: 57 pgs: 2 active+recovering+remapped, 9 active+recovering+undersized+remapped, 46 active+clean; 5.7 GiB data, 26 GiB used, 779 GiB / 805 GiB avail; 47 MiB/s wr, 756 op/s; 19088/280815 objects misplaced (6.797%); 40 MiB/s, 640 objects/s recovering 2024-08-19T14:41:26.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:25 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:41:26.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:25 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:41:26.147 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:41:26.189 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:26.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:26.190 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:41:26.291 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:41:26.296 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 75 16 97535 97519 81.2548 54.8125 0.00351147 0.0122949 2024-08-19T14:41:26.326 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:26.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:26.327 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:41:26.428 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:41:26.460 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:26.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:26.461 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:41:26.562 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:41:26.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:26.584 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:26.584 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:41:26.684 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:41:26.711 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:26.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:26.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:41:26.814 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:41:26.847 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:26.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:26.848 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:41:26.949 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:41:26.972 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:26.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:26.973 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:41:27.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:26 smithi032 ceph-mon[20004]: osdmap e752: 12 total, 12 up, 9 in 2024-08-19T14:41:27.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:26 smithi045 ceph-mon[20143]: osdmap e752: 12 total, 12 up, 9 in 2024-08-19T14:41:27.074 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:41:27.108 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:27.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:27.109 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:41:27.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:26 smithi031 ceph-mon[16336]: osdmap e752: 12 total, 12 up, 9 in 2024-08-19T14:41:27.210 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:41:27.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:27.247 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:27.247 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:41:27.296 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 76 16 98494 98478 80.9743 59.9375 0.0589672 0.0123401 2024-08-19T14:41:27.348 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:41:27.374 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:27.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:27.375 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:41:27.476 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:41:27.504 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:27.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:27.507 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:41:27.609 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:41:27.650 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:27.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:27.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:41:27.752 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:41:27.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:27.781 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:27.782 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:41:27.883 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:41:27.913 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:27.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:27.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:41:28.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:27 smithi032 ceph-mon[20004]: pgmap v1798: 57 pgs: 2 active+recovering+remapped, 9 active+recovering+undersized+remapped, 46 active+clean; 5.8 GiB data, 26 GiB used, 778 GiB / 805 GiB avail; 55 MiB/s wr, 884 op/s; 16356/287343 objects misplaced (5.692%); 48 MiB/s, 761 objects/s recovering 2024-08-19T14:41:28.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:27 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:41:28.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:27 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:41:28.015 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:41:28.039 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:28.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:28.040 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:41:28.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:27 smithi045 ceph-mon[20143]: pgmap v1798: 57 pgs: 2 active+recovering+remapped, 9 active+recovering+undersized+remapped, 46 active+clean; 5.8 GiB data, 26 GiB used, 778 GiB / 805 GiB avail; 55 MiB/s wr, 884 op/s; 16356/287343 objects misplaced (5.692%); 48 MiB/s, 761 objects/s recovering 2024-08-19T14:41:28.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:27 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:41:28.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:27 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:41:28.071 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:27 smithi031 ceph-mon[16336]: pgmap v1798: 57 pgs: 2 active+recovering+remapped, 9 active+recovering+undersized+remapped, 46 active+clean; 5.8 GiB data, 26 GiB used, 778 GiB / 805 GiB avail; 55 MiB/s wr, 884 op/s; 16356/287343 objects misplaced (5.692%); 48 MiB/s, 761 objects/s recovering 2024-08-19T14:41:28.072 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:27 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:41:28.072 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:27 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:41:28.141 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:41:28.163 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:28.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:28.164 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:41:28.265 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:41:28.288 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:28.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:28.289 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:41:28.296 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 77 16 99122 99106 80.4324 39.25 0.00518208 0.0123464 2024-08-19T14:41:28.390 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:41:28.413 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:28.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:28.414 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:41:28.518 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:41:28.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:28.543 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:28.543 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:41:28.644 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:41:28.667 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:28.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:28.668 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:41:28.769 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:41:28.791 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:28.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:28.792 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:41:28.893 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:41:28.925 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:28.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:28.926 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:41:29.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:28 smithi032 ceph-mon[20004]: osdmap e753: 12 total, 12 up, 9 in 2024-08-19T14:41:29.027 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:41:29.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:28 smithi045 ceph-mon[20143]: osdmap e753: 12 total, 12 up, 9 in 2024-08-19T14:41:29.056 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:29.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:29.057 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:41:29.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:28 smithi031 ceph-mon[16336]: osdmap e753: 12 total, 12 up, 9 in 2024-08-19T14:41:29.157 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:41:29.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:29.185 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:29.186 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:41:29.287 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:41:29.297 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 78 16 99610 99594 79.7921 30.5 0.00439022 0.0125058 2024-08-19T14:41:29.312 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:29.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:29.313 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:41:29.415 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:41:29.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:29.443 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:29.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:41:29.544 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:41:29.570 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:29.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:29.572 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:41:29.672 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:41:29.700 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:29.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:29.702 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:41:29.802 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:41:29.830 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:29.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:29.831 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:41:29.932 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:41:29.964 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:29.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:29.966 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:41:30.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:29 smithi032 ceph-mon[20004]: pgmap v1800: 57 pgs: 2 active+recovering+remapped, 9 active+recovering+undersized+remapped, 46 active+clean; 5.9 GiB data, 27 GiB used, 778 GiB / 805 GiB avail; 52 MiB/s wr, 832 op/s; 16356/290889 objects misplaced (5.623%); 39 MiB/s, 623 objects/s recovering 2024-08-19T14:41:30.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:29 smithi032 ceph-mon[20004]: osdmap e754: 12 total, 12 up, 9 in 2024-08-19T14:41:30.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:29 smithi032 ceph-mon[20004]: osdmap e755: 12 total, 12 up, 9 in 2024-08-19T14:41:30.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:29 smithi045 ceph-mon[20143]: pgmap v1800: 57 pgs: 2 active+recovering+remapped, 9 active+recovering+undersized+remapped, 46 active+clean; 5.9 GiB data, 27 GiB used, 778 GiB / 805 GiB avail; 52 MiB/s wr, 832 op/s; 16356/290889 objects misplaced (5.623%); 39 MiB/s, 623 objects/s recovering 2024-08-19T14:41:30.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:29 smithi045 ceph-mon[20143]: osdmap e754: 12 total, 12 up, 9 in 2024-08-19T14:41:30.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:29 smithi045 ceph-mon[20143]: osdmap e755: 12 total, 12 up, 9 in 2024-08-19T14:41:30.067 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:41:30.088 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:30.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:30.089 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:41:30.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:29 smithi031 ceph-mon[16336]: pgmap v1800: 57 pgs: 2 active+recovering+remapped, 9 active+recovering+undersized+remapped, 46 active+clean; 5.9 GiB data, 27 GiB used, 778 GiB / 805 GiB avail; 52 MiB/s wr, 832 op/s; 16356/290889 objects misplaced (5.623%); 39 MiB/s, 623 objects/s recovering 2024-08-19T14:41:30.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:29 smithi031 ceph-mon[16336]: osdmap e754: 12 total, 12 up, 9 in 2024-08-19T14:41:30.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:29 smithi031 ceph-mon[16336]: osdmap e755: 12 total, 12 up, 9 in 2024-08-19T14:41:30.127 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:41:30.189 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:41:30.221 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:30.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:30.222 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:41:30.297 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 79 16 99983 99967 79.0772 23.3125 0.00903367 0.0125825 2024-08-19T14:41:30.323 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:41:30.347 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:30.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:30.348 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:41:30.449 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:41:30.473 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:30.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:30.474 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:41:30.574 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:41:30.597 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:30.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:30.598 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:41:30.698 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:41:30.731 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:30.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:30.733 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:41:30.834 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:41:30.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:30.872 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:30.872 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:41:30.972 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:41:30.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:30.994 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:30.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:41:31.095 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:41:31.120 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:31.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:31.121 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:41:31.222 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:41:31.246 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:31.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:31.247 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:41:31.297 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:2024-08-19T14:41:31.297418+0000 min lat: 0.00108187 max lat: 2.31637 avg lat: 0.0126802 2024-08-19T14:41:31.297 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-19T14:41:31.297 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 80 16 100500 100484 78.4926 32.3125 0.00283169 0.0126802 2024-08-19T14:41:31.348 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:41:31.369 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:31.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:31.370 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:41:31.471 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:41:31.494 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:31.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:31.495 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:41:31.596 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:41:31.619 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:31.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:31.620 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:41:31.720 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:41:31.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:31.753 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:31.754 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:41:31.854 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:41:31.885 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:31.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:31.890 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:41:31.991 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:41:32.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:31 smithi032 ceph-mon[20004]: pgmap v1803: 57 pgs: 1 active+undersized+remapped, 2 active+recovering+remapped, 8 active+recovering+undersized+remapped, 46 active+clean; 6.0 GiB data, 27 GiB used, 777 GiB / 805 GiB avail; 44 MiB/s wr, 705 op/s; 14802/292977 objects misplaced (5.052%); 30 MiB/s, 482 objects/s recovering 2024-08-19T14:41:32.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:31 smithi032 ceph-mon[20004]: osdmap e756: 12 total, 12 up, 9 in 2024-08-19T14:41:32.021 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:32.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:32.022 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:41:32.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:31 smithi045 ceph-mon[20143]: pgmap v1803: 57 pgs: 1 active+undersized+remapped, 2 active+recovering+remapped, 8 active+recovering+undersized+remapped, 46 active+clean; 6.0 GiB data, 27 GiB used, 777 GiB / 805 GiB avail; 44 MiB/s wr, 705 op/s; 14802/292977 objects misplaced (5.052%); 30 MiB/s, 482 objects/s recovering 2024-08-19T14:41:32.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:31 smithi045 ceph-mon[20143]: osdmap e756: 12 total, 12 up, 9 in 2024-08-19T14:41:32.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:31 smithi031 ceph-mon[16336]: pgmap v1803: 57 pgs: 1 active+undersized+remapped, 2 active+recovering+remapped, 8 active+recovering+undersized+remapped, 46 active+clean; 6.0 GiB data, 27 GiB used, 777 GiB / 805 GiB avail; 44 MiB/s wr, 705 op/s; 14802/292977 objects misplaced (5.052%); 30 MiB/s, 482 objects/s recovering 2024-08-19T14:41:32.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:31 smithi031 ceph-mon[16336]: osdmap e756: 12 total, 12 up, 9 in 2024-08-19T14:41:32.124 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:41:32.156 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:32.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:32.158 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:41:32.261 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:41:32.295 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:32.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:32.296 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:41:32.297 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 81 16 101264 101248 78.1131 47.75 0.0189987 0.0127971 2024-08-19T14:41:32.396 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:41:32.420 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:32.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:32.421 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:41:32.522 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:41:32.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:32.557 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:32.557 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:41:32.658 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:41:32.691 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:32.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:32.692 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:41:32.793 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:41:32.834 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:32.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:32.835 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:41:32.936 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:41:32.966 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:32.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:32.967 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:41:33.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:32 smithi032 ceph-mon[20004]: osdmap e757: 12 total, 12 up, 9 in 2024-08-19T14:41:33.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:32 smithi045 ceph-mon[20143]: osdmap e757: 12 total, 12 up, 9 in 2024-08-19T14:41:33.068 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:41:33.102 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:33.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:33.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:41:33.119 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:32 smithi031 ceph-mon[16336]: osdmap e757: 12 total, 12 up, 9 in 2024-08-19T14:41:33.205 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:41:33.248 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:33.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:33.249 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:41:33.297 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 82 16 102739 102723 78.2846 92.1875 0.0131559 0.0127691 2024-08-19T14:41:33.350 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:41:33.380 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:33.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:33.381 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:41:33.482 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:41:33.515 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:33.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:33.516 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:41:33.617 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:41:33.641 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:33.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:33.642 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:41:33.744 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:41:33.771 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:33.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:33.772 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:41:33.873 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:41:33.901 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:33.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:33.903 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:41:34.004 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:41:34.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:33 smithi032 ceph-mon[20004]: pgmap v1806: 57 pgs: 1 creating+peering, 2 active+recovering+remapped, 5 active+recovering+undersized+remapped, 49 active+clean; 6.1 GiB data, 28 GiB used, 777 GiB / 805 GiB avail; 49 MiB/s wr, 781 op/s; 11084/300264 objects misplaced (3.691%); 47 MiB/s, 749 objects/s recovering 2024-08-19T14:41:34.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:34.043 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:34.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:41:34.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:33 smithi045 ceph-mon[20143]: pgmap v1806: 57 pgs: 1 creating+peering, 2 active+recovering+remapped, 5 active+recovering+undersized+remapped, 49 active+clean; 6.1 GiB data, 28 GiB used, 777 GiB / 805 GiB avail; 49 MiB/s wr, 781 op/s; 11084/300264 objects misplaced (3.691%); 47 MiB/s, 749 objects/s recovering 2024-08-19T14:41:34.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:33 smithi031 ceph-mon[16336]: pgmap v1806: 57 pgs: 1 creating+peering, 2 active+recovering+remapped, 5 active+recovering+undersized+remapped, 49 active+clean; 6.1 GiB data, 28 GiB used, 777 GiB / 805 GiB avail; 49 MiB/s wr, 781 op/s; 11084/300264 objects misplaced (3.691%); 47 MiB/s, 749 objects/s recovering 2024-08-19T14:41:34.145 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:41:34.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:34.176 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:34.176 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:41:34.277 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:41:34.297 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 83 16 104226 104210 78.461 92.9375 0.00711833 0.0127405 2024-08-19T14:41:34.319 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:34.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:34.320 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:41:34.421 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:41:34.459 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:34.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:34.460 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:41:34.467 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:41:34.523 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":1795,"stamp":"2024-08-19T14:41:22.302609+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6082471185,"num_objects":92809,"num_object_clones":0,"num_object_copies":278427,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":22149,"num_objects_unfound":0,"num_objects_dirty":92809,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":84214,"num_write_kb":5389484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19255,"num_bytes_recovered":1262333606,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139192,"ondisk_log_size":139192,"up":171,"acting":161,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":185,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":26845908,"kb_used_data":18470704,"kb_used_omap":16,"kb_used_meta":8375087,"kb_avail":816860460,"statfs":{"total":863955320832,"available":836465111040,"internally_reserved":0,"allocated":18914000896,"data_stored":18884137092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17167,"internal_metadata":8576089329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[9,3,3,3,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":336068608,"num_objects":5128,"num_object_clones":0,"num_object_copies":15384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-5374,"num_objects_unfound":0,"num_objects_dirty":5128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5128,"num_write_kb":328192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2947,"num_bytes_recovered":193134592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001580"},"pg_stats":[{"pgid":"8.16","version":"751'4929","reported_seq":9310,"reported_epoch":751,"state":"active+clean","last_fresh":"2024-08-19T14:41:21.648859+0000","last_change":"2024-08-19T14:41:00.561784+0000","last_active":"2024-08-19T14:41:21.648859+0000","last_peered":"2024-08-19T14:41:21.648859+0000","last_clean":"2024-08-19T14:41:21.648859+0000","last_became_active":"2024-08-19T14:41:00.558578+0000","last_became_peered":"2024-08-19T14:41:00.558578+0000","last_unstale":"2024-08-19T14:41:21.648859+0000","last_undegraded":"2024-08-19T14:41:21.648859+0000","last_fullsized":"2024-08-19T14:41:21.648859+0000","mapping_epoch":743,"log_start":"0'0","ondisk_log_start":"0'0","created":731,"last_epoch_clean":744,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.182354+0000","objects_scrubbed":0,"log_size":4929,"log_dups_size":0,"ondisk_log_size":4929,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:22.025770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":187170816,"num_objects":2856,"num_object_clones":0,"num_object_copies":8568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2856,"num_write_kb":182784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2142,"num_bytes_recovered":140378112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.9","version":"751'5014","reported_seq":8436,"reported_epoch":751,"state":"active+recovering+undersized+remapped","last_fresh":"2024-08-19T14:41:20.969968+0000","last_change":"2024-08-19T14:40:50.580857+0000","last_active":"2024-08-19T14:41:20.969968+0000","last_peered":"2024-08-19T14:41:20.969968+0000","last_clean":"2024-08-19T14:40:48.248463+0000","last_became_active":"2024-08-19T14:40:50.543560+0000","last_became_peered":"2024-08-19T14:40:50.543560+0000","last_unstale":"2024-08-19T14:41:20.969968+0000","last_undegraded":"2024-08-19T14:41:20.969968+0000","last_fullsized":"2024-08-19T14:40:50.444995+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":717,"last_epoch_clean":732,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.182354+0000","objects_scrubbed":1299,"log_size":5014,"log_dups_size":0,"ondisk_log_size":5014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":191430656,"num_objects":2921,"num_object_clones":0,"num_object_copies":8763,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":522,"num_objects_unfound":0,"num_objects_dirty":2921,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2921,"num_write_kb":186944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1658,"num_bytes_recovered":108658688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,9],"acting":[0,5],"avail_no_missing":["0","3","5"],"object_location_counts":[{"shards":"0,3,5","objects":522},{"shards":"0,5","objects":2399}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"70'139","reported_seq":1830,"reported_epoch":751,"state":"active+clean","last_fresh":"2024-08-19T14:41:20.652566+0000","last_change":"2024-08-19T14:39:45.569137+0000","last_active":"2024-08-19T14:41:20.652566+0000","last_peered":"2024-08-19T14:41:20.652566+0000","last_clean":"2024-08-19T14:41:20.652566+0000","last_became_active":"2024-08-19T14:35:13.950224+0000","last_became_peered":"2024-08-19T14:35:13.950224+0000","last_unstale":"2024-08-19T14:41:20.652566+0000","last_undegraded":"2024-08-19T14:41:20.652566+0000","last_fullsized":"2024-08-19T14:41:20.652566+0000","mapping_epoch":637,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":638,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:39:45.569030+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:19:58.072083+0000","last_clean_scrub_stamp":"2024-08-19T14:39:45.569030+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,9],"acting":[0,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"750'5687","reported_seq":5766,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-08-19T14:41:20.279963+0000","last_change":"2024-08-19T14:40:04.442002+0000","last_active":"2024-08-19T14:41:20.279963+0000","last_peered":"2024-08-19T14:41:20.279963+0000","last_clean":"2024-08-19T14:41:20.279963+0000","last_became_active":"2024-08-19T14:40:04.441586+0000","last_became_peered":"2024-08-19T14:40:04.441586+0000","last_unstale":"2024-08-19T14:41:20.279963+0000","last_undegraded":"2024-08-19T14:41:20.279963+0000","last_fullsized":"2024-08-19T14:41:20.279963+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":717,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.182354+0000","objects_scrubbed":0,"log_size":5687,"log_dups_size":0,"ondisk_log_size":5687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":372703232,"num_objects":5687,"num_object_clones":0,"num_object_copies":17061,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5687,"num_write_kb":363968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,9],"acting":[8,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"70'139","reported_seq":1668,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-08-19T14:41:08.605379+0000","last_change":"2024-08-19T14:40:02.681316+0000","last_active":"2024-08-19T14:41:08.605379+0000","last_peered":"2024-08-19T14:41:08.605379+0000","last_clean":"2024-08-19T14:41:08.605379+0000","last_became_active":"2024-08-19T14:18:13.804721+0000","last_became_peered":"2024-08-19T14:18:13.804721+0000","last_unstale":"2024-08-19T14:41:08.605379+0000","last_undegraded":"2024-08-19T14:41:08.605379+0000","last_fullsized":"2024-08-19T14:41:08.605379+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:40:02.681185+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:40:02.681185+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:22.300124+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.b","version":"750'5739","reported_seq":5818,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-08-19T14:41:20.208635+0000","last_change":"2024-08-19T14:40:04.200633+0000","last_active":"2024-08-19T14:41:20.208635+0000","last_peered":"2024-08-19T14:41:20.208635+0000","last_clean":"2024-08-19T14:41:20.208635+0000","last_became_active":"2024-08-19T14:40:04.200299+0000","last_became_peered":"2024-08-19T14:40:04.200299+0000","last_unstale":"2024-08-19T14:41:20.208635+0000","last_undegraded":"2024-08-19T14:41:20.208635+0000","last_fullsized":"2024-08-19T14:41:20.208635+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":717,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.182354+0000","objects_scrubbed":0,"log_size":5739,"log_dups_size":0,"ondisk_log_size":5739,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:23.069839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":376111104,"num_objects":5739,"num_object_clones":0,"num_object_copies":17217,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5739,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5739,"num_write_kb":367296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,1],"acting":[8,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.2","version":"70'139","reported_seq":1628,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-08-19T14:41:19.651022+0000","last_change":"2024-08-19T14:40:10.977253+0000","last_active":"2024-08-19T14:41:19.651022+0000","last_peered":"2024-08-19T14:41:19.651022+0000","last_clean":"2024-08-19T14:41:19.651022+0000","last_became_active":"2024-08-19T14:20:50.742354+0000","last_became_peered":"2024-08-19T14:20:50.742354+0000","last_unstale":"2024-08-19T14:41:19.651022+0000","last_undegraded":"2024-08-19T14:41:19.651022+0000","last_fullsized":"2024-08-19T14:41:19.651022+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:40:10.977156+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:19:59.302931+0000","last_clean_scrub_stamp":"2024-08-19T14:40:10.977156+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:39.144591+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.8","version":"751'5094","reported_seq":6670,"reported_epoch":751,"state":"active+recovering+remapped","last_fresh":"2024-08-19T14:41:20.832831+0000","last_change":"2024-08-19T14:40:50.574034+0000","last_active":"2024-08-19T14:41:20.832831+0000","last_peered":"2024-08-19T14:41:20.832831+0000","last_clean":"2024-08-19T14:40:48.375631+0000","last_became_active":"2024-08-19T14:40:50.542682+0000","last_became_peered":"2024-08-19T14:40:50.542682+0000","last_unstale":"2024-08-19T14:41:20.832831+0000","last_undegraded":"2024-08-19T14:41:20.832831+0000","last_fullsized":"2024-08-19T14:41:20.832831+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":717,"last_epoch_clean":732,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.182354+0000","objects_scrubbed":0,"log_size":5094,"log_dups_size":0,"ondisk_log_size":5094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":195035136,"num_objects":2976,"num_object_clones":0,"num_object_copies":8928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1461,"num_objects_unfound":0,"num_objects_dirty":2976,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2976,"num_write_kb":190464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":743,"num_bytes_recovered":48693248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,9],"acting":[3,5,9],"avail_no_missing":["3","5","9"],"object_location_counts":[{"shards":"3,5,9","objects":1515}],"blocked_by":[],"up_primary":11,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"70'139","reported_seq":928,"reported_epoch":751,"state":"active+clean","last_fresh":"2024-08-19T14:41:20.656942+0000","last_change":"2024-08-19T14:39:37.543259+0000","last_active":"2024-08-19T14:41:20.656942+0000","last_peered":"2024-08-19T14:41:20.656942+0000","last_clean":"2024-08-19T14:41:20.656942+0000","last_became_active":"2024-08-19T14:35:12.530566+0000","last_became_peered":"2024-08-19T14:35:12.530566+0000","last_unstale":"2024-08-19T14:41:20.656942+0000","last_undegraded":"2024-08-19T14:41:20.656942+0000","last_fullsized":"2024-08-19T14:41:20.656942+0000","mapping_epoch":635,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":636,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:39:37.543187+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:39:37.543187+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,2],"acting":[0,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.e","version":"751'5821","reported_seq":5902,"reported_epoch":751,"state":"active+clean","last_fresh":"2024-08-19T14:41:20.839586+0000","last_change":"2024-08-19T14:40:04.299744+0000","last_active":"2024-08-19T14:41:20.839586+0000","last_peered":"2024-08-19T14:41:20.839586+0000","last_clean":"2024-08-19T14:41:20.839586+0000","last_became_active":"2024-08-19T14:40:04.299403+0000","last_became_peered":"2024-08-19T14:40:04.299403+0000","last_unstale":"2024-08-19T14:41:20.839586+0000","last_undegraded":"2024-08-19T14:41:20.839586+0000","last_fullsized":"2024-08-19T14:41:20.839586+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":717,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.182354+0000","objects_scrubbed":0,"log_size":5821,"log_dups_size":0,"ondisk_log_size":5821,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381485056,"num_objects":5821,"num_object_clones":0,"num_object_copies":17463,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5821,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5821,"num_write_kb":372544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"70'139","reported_seq":879,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-08-19T14:41:08.607075+0000","last_change":"2024-08-19T14:40:52.490529+0000","last_active":"2024-08-19T14:41:08.607075+0000","last_peered":"2024-08-19T14:41:08.607075+0000","last_clean":"2024-08-19T14:41:08.607075+0000","last_became_active":"2024-08-19T14:40:52.490194+0000","last_became_peered":"2024-08-19T14:40:52.490194+0000","last_unstale":"2024-08-19T14:41:08.607075+0000","last_undegraded":"2024-08-19T14:41:08.607075+0000","last_fullsized":"2024-08-19T14:41:08.607075+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:39:56.993616+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:21:46.057624+0000","last_clean_scrub_stamp":"2024-08-19T14:39:56.993616+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.f","version":"751'5872","reported_seq":5952,"reported_epoch":751,"state":"active+clean","last_fresh":"2024-08-19T14:41:21.700760+0000","last_change":"2024-08-19T14:40:04.299935+0000","last_active":"2024-08-19T14:41:21.700760+0000","last_peered":"2024-08-19T14:41:21.700760+0000","last_clean":"2024-08-19T14:41:21.700760+0000","last_became_active":"2024-08-19T14:40:04.299656+0000","last_became_peered":"2024-08-19T14:40:04.299656+0000","last_unstale":"2024-08-19T14:41:21.700760+0000","last_undegraded":"2024-08-19T14:41:21.700760+0000","last_fullsized":"2024-08-19T14:41:21.700760+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":717,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.182354+0000","objects_scrubbed":0,"log_size":5872,"log_dups_size":0,"ondisk_log_size":5872,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:32.460628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":384827392,"num_objects":5872,"num_object_clones":0,"num_object_copies":17616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5872,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5872,"num_write_kb":375808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,1],"acting":[11,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.6","version":"70'139","reported_seq":898,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-08-19T14:41:19.648807+0000","last_change":"2024-08-19T14:39:30.460768+0000","last_active":"2024-08-19T14:41:19.648807+0000","last_peered":"2024-08-19T14:41:19.648807+0000","last_clean":"2024-08-19T14:41:19.648807+0000","last_became_active":"2024-08-19T14:34:15.717054+0000","last_became_peered":"2024-08-19T14:34:15.717054+0000","last_unstale":"2024-08-19T14:41:19.648807+0000","last_undegraded":"2024-08-19T14:41:19.648807+0000","last_fullsized":"2024-08-19T14:41:19.648807+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:39:30.460649+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:27.848870+0000","last_clean_scrub_stamp":"2024-08-19T14:39:30.460649+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.c","version":"750'5896","reported_seq":9009,"reported_epoch":750,"state":"active+recovering+undersized+remapped","last_fresh":"2024-08-19T14:41:20.391755+0000","last_change":"2024-08-19T14:40:50.634786+0000","last_active":"2024-08-19T14:41:20.391755+0000","last_peered":"2024-08-19T14:41:20.391755+0000","last_clean":"2024-08-19T14:40:48.387410+0000","last_became_active":"2024-08-19T14:40:50.576096+0000","last_became_peered":"2024-08-19T14:40:50.576096+0000","last_unstale":"2024-08-19T14:41:20.391755+0000","last_undegraded":"2024-08-19T14:41:20.391755+0000","last_fullsized":"2024-08-19T14:40:50.434639+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":717,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.182354+0000","objects_scrubbed":0,"log_size":5896,"log_dups_size":0,"ondisk_log_size":5896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":386400256,"num_objects":5896,"num_object_clones":0,"num_object_copies":17688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2980,"num_objects_unfound":0,"num_objects_dirty":5896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5896,"num_write_kb":377344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1516,"num_bytes_recovered":99352576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,7],"acting":[8,9],"avail_no_missing":["8","3","9"],"object_location_counts":[{"shards":"3,8,9","objects":2980},{"shards":"8,9","objects":2916}],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.5","version":"70'139","reported_seq":855,"reported_epoch":751,"state":"active+clean","last_fresh":"2024-08-19T14:41:20.685750+0000","last_change":"2024-08-19T14:39:54.886149+0000","last_active":"2024-08-19T14:41:20.685750+0000","last_peered":"2024-08-19T14:41:20.685750+0000","last_clean":"2024-08-19T14:41:20.685750+0000","last_became_active":"2024-08-19T14:35:11.532858+0000","last_became_peered":"2024-08-19T14:35:11.532858+0000","last_unstale":"2024-08-19T14:41:20.685750+0000","last_undegraded":"2024-08-19T14:41:20.685750+0000","last_fullsized":"2024-08-19T14:41:20.685750+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:39:54.886056+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:39:54.886056+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,0],"acting":[11,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.d","version":"751'5885","reported_seq":5966,"reported_epoch":751,"state":"active+clean","last_fresh":"2024-08-19T14:41:21.707430+0000","last_change":"2024-08-19T14:40:04.443252+0000","last_active":"2024-08-19T14:41:21.707430+0000","last_peered":"2024-08-19T14:41:21.707430+0000","last_clean":"2024-08-19T14:41:21.707430+0000","last_became_active":"2024-08-19T14:40:04.442093+0000","last_became_peered":"2024-08-19T14:40:04.442093+0000","last_unstale":"2024-08-19T14:41:21.707430+0000","last_undegraded":"2024-08-19T14:41:21.707430+0000","last_fullsized":"2024-08-19T14:41:21.707430+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":717,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.182354+0000","objects_scrubbed":0,"log_size":5885,"log_dups_size":0,"ondisk_log_size":5885,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":385679360,"num_objects":5885,"num_object_clones":0,"num_object_copies":17655,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5885,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5885,"num_write_kb":376640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,2],"acting":[11,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.4","version":"70'139","reported_seq":732,"reported_epoch":751,"state":"active+clean","last_fresh":"2024-08-19T14:41:20.685764+0000","last_change":"2024-08-19T14:40:09.799125+0000","last_active":"2024-08-19T14:41:20.685764+0000","last_peered":"2024-08-19T14:41:20.685764+0000","last_clean":"2024-08-19T14:41:20.685764+0000","last_became_active":"2024-08-19T14:30:47.909067+0000","last_became_peered":"2024-08-19T14:30:47.909067+0000","last_unstale":"2024-08-19T14:41:20.685764+0000","last_undegraded":"2024-08-19T14:41:20.685764+0000","last_fullsized":"2024-08-19T14:41:20.685764+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:40:09.798994+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:40:09.798994+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:32.468916+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.3","version":"751'5055","reported_seq":5142,"reported_epoch":751,"state":"active+clean","last_fresh":"2024-08-19T14:41:21.785501+0000","last_change":"2024-08-19T14:40:43.372478+0000","last_active":"2024-08-19T14:41:21.785501+0000","last_peered":"2024-08-19T14:41:21.785501+0000","last_clean":"2024-08-19T14:41:21.785501+0000","last_became_active":"2024-08-19T14:40:43.369906+0000","last_became_peered":"2024-08-19T14:40:43.369906+0000","last_unstale":"2024-08-19T14:41:21.785501+0000","last_undegraded":"2024-08-19T14:41:21.785501+0000","last_fullsized":"2024-08-19T14:41:21.785501+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":717,"last_epoch_clean":732,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.182354+0000","objects_scrubbed":0,"log_size":5055,"log_dups_size":0,"ondisk_log_size":5055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:22.024365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":193331200,"num_objects":2950,"num_object_clones":0,"num_object_copies":8850,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2950,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2950,"num_write_kb":188800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.a","version":"70'139","reported_seq":893,"reported_epoch":749,"state":"active+clean","last_fresh":"2024-08-19T14:41:18.623657+0000","last_change":"2024-08-19T14:40:05.713501+0000","last_active":"2024-08-19T14:41:18.623657+0000","last_peered":"2024-08-19T14:41:18.623657+0000","last_clean":"2024-08-19T14:41:18.623657+0000","last_became_active":"2024-08-19T14:34:15.717472+0000","last_became_peered":"2024-08-19T14:34:15.717472+0000","last_unstale":"2024-08-19T14:41:18.623657+0000","last_undegraded":"2024-08-19T14:41:18.623657+0000","last_fullsized":"2024-08-19T14:41:18.623657+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:40:05.713411+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:28.864870+0000","last_clean_scrub_stamp":"2024-08-19T14:40:05.713411+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,0],"acting":[4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.0","version":"751'5194","reported_seq":7083,"reported_epoch":751,"state":"active+recovering+remapped","last_fresh":"2024-08-19T14:41:20.829471+0000","last_change":"2024-08-19T14:40:50.570904+0000","last_active":"2024-08-19T14:41:20.829471+0000","last_peered":"2024-08-19T14:41:20.829471+0000","last_clean":"2024-08-19T14:40:48.392218+0000","last_became_active":"2024-08-19T14:40:50.542375+0000","last_became_peered":"2024-08-19T14:40:50.542375+0000","last_unstale":"2024-08-19T14:41:20.829471+0000","last_undegraded":"2024-08-19T14:41:20.829471+0000","last_fullsized":"2024-08-19T14:41:20.829471+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":717,"last_epoch_clean":732,"parent":"0.0","parent_split_bits":0,"last_scrub":"734'4525","last_scrub_stamp":"2024-08-19T14:40:45.560285+0000","last_deep_scrub":"734'4525","last_deep_scrub_stamp":"2024-08-19T14:40:45.560285+0000","last_clean_scrub_stamp":"2024-08-19T14:40:45.560285+0000","objects_scrubbed":2176,"log_size":5194,"log_dups_size":0,"ondisk_log_size":5194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1668,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":186580992,"num_objects":2847,"num_object_clones":0,"num_object_copies":8541,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1364,"num_objects_unfound":0,"num_objects_dirty":2847,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":671,"num_write_kb":42944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":852,"num_bytes_recovered":55836672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,9],"acting":[3,5,9],"avail_no_missing":["3","5","9"],"object_location_counts":[{"shards":"3,5,9","objects":1483}],"blocked_by":[],"up_primary":4,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"70'139","reported_seq":1831,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-08-19T14:41:19.651079+0000","last_change":"2024-08-19T14:40:05.923941+0000","last_active":"2024-08-19T14:41:19.651079+0000","last_peered":"2024-08-19T14:41:19.651079+0000","last_clean":"2024-08-19T14:41:19.651079+0000","last_became_active":"2024-08-19T14:35:12.529628+0000","last_became_peered":"2024-08-19T14:35:12.529628+0000","last_unstale":"2024-08-19T14:41:19.651079+0000","last_undegraded":"2024-08-19T14:41:19.651079+0000","last_fullsized":"2024-08-19T14:41:19.651079+0000","mapping_epoch":635,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":636,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:40:05.923850+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:40:05.923850+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:22.262135+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1","version":"748'5007","reported_seq":5091,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-08-19T14:41:18.350190+0000","last_change":"2024-08-19T14:40:43.406403+0000","last_active":"2024-08-19T14:41:18.350190+0000","last_peered":"2024-08-19T14:41:18.350190+0000","last_clean":"2024-08-19T14:41:18.350190+0000","last_became_active":"2024-08-19T14:40:43.388885+0000","last_became_peered":"2024-08-19T14:40:43.388885+0000","last_unstale":"2024-08-19T14:41:18.350190+0000","last_undegraded":"2024-08-19T14:41:18.350190+0000","last_fullsized":"2024-08-19T14:41:18.350190+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":717,"last_epoch_clean":732,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.182354+0000","objects_scrubbed":0,"log_size":5007,"log_dups_size":0,"ondisk_log_size":5007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:22.963261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":190218240,"num_objects":2902,"num_object_clones":0,"num_object_copies":8706,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2902,"num_write_kb":185760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,2],"acting":[1,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"70'139","reported_seq":413,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-08-19T14:41:19.649197+0000","last_change":"2024-08-19T14:40:52.456223+0000","last_active":"2024-08-19T14:41:19.649197+0000","last_peered":"2024-08-19T14:41:19.649197+0000","last_clean":"2024-08-19T14:41:19.649197+0000","last_became_active":"2024-08-19T14:40:52.455488+0000","last_became_peered":"2024-08-19T14:40:52.455488+0000","last_unstale":"2024-08-19T14:41:19.649197+0000","last_undegraded":"2024-08-19T14:41:19.649197+0000","last_fullsized":"2024-08-19T14:41:19.649197+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:39:59.000300+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:39:59.000300+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,2],"acting":[8,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.2","version":"751'4957","reported_seq":5044,"reported_epoch":751,"state":"active+clean","last_fresh":"2024-08-19T14:41:20.838904+0000","last_change":"2024-08-19T14:40:43.374181+0000","last_active":"2024-08-19T14:41:20.838904+0000","last_peered":"2024-08-19T14:41:20.838904+0000","last_clean":"2024-08-19T14:41:20.838904+0000","last_became_active":"2024-08-19T14:40:43.371998+0000","last_became_peered":"2024-08-19T14:40:43.371998+0000","last_unstale":"2024-08-19T14:41:20.838904+0000","last_undegraded":"2024-08-19T14:41:20.838904+0000","last_fullsized":"2024-08-19T14:41:20.838904+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":717,"last_epoch_clean":732,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.182354+0000","objects_scrubbed":0,"log_size":4957,"log_dups_size":0,"ondisk_log_size":4957,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:23.907945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188973056,"num_objects":2883,"num_object_clones":0,"num_object_copies":8649,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2883,"num_write_kb":184544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,9],"acting":[0,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.b","version":"70'139","reported_seq":1183,"reported_epoch":751,"state":"active+clean","last_fresh":"2024-08-19T14:41:20.685677+0000","last_change":"2024-08-19T14:40:03.813911+0000","last_active":"2024-08-19T14:41:20.685677+0000","last_peered":"2024-08-19T14:41:20.685677+0000","last_clean":"2024-08-19T14:41:20.685677+0000","last_became_active":"2024-08-19T14:30:47.941200+0000","last_became_peered":"2024-08-19T14:30:47.941200+0000","last_unstale":"2024-08-19T14:41:20.685677+0000","last_undegraded":"2024-08-19T14:41:20.685677+0000","last_fullsized":"2024-08-19T14:41:20.685677+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:40:03.813794+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.813794+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:29.146651+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.7","version":"748'4859","reported_seq":4970,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-08-19T14:41:18.413892+0000","last_change":"2024-08-19T14:40:44.273658+0000","last_active":"2024-08-19T14:41:18.413892+0000","last_peered":"2024-08-19T14:41:18.413892+0000","last_clean":"2024-08-19T14:41:18.413892+0000","last_became_active":"2024-08-19T14:40:43.370885+0000","last_became_peered":"2024-08-19T14:40:43.370885+0000","last_unstale":"2024-08-19T14:41:18.413892+0000","last_undegraded":"2024-08-19T14:41:18.413892+0000","last_fullsized":"2024-08-19T14:41:18.413892+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":717,"last_epoch_clean":732,"parent":"0.0","parent_split_bits":0,"last_scrub":"732'4156","last_scrub_stamp":"2024-08-19T14:40:44.273366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:40:44.273366+0000","objects_scrubbed":2153,"log_size":4859,"log_dups_size":0,"ondisk_log_size":4859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:58.974295+0000","scrub_duration":570,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":187629568,"num_objects":2863,"num_object_clones":0,"num_object_copies":8589,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":710,"num_write_kb":45440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.e","version":"70'139","reported_seq":1379,"reported_epoch":749,"state":"active+clean","last_fresh":"2024-08-19T14:41:18.623475+0000","last_change":"2024-08-19T14:40:52.444822+0000","last_active":"2024-08-19T14:41:18.623475+0000","last_peered":"2024-08-19T14:41:18.623475+0000","last_clean":"2024-08-19T14:41:18.623475+0000","last_became_active":"2024-08-19T14:40:52.444460+0000","last_became_peered":"2024-08-19T14:40:52.444460+0000","last_unstale":"2024-08-19T14:41:18.623475+0000","last_undegraded":"2024-08-19T14:41:18.623475+0000","last_fullsized":"2024-08-19T14:41:18.623475+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:40:03.764672+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:40:03.764672+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.764672+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,8],"acting":[4,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.6","version":"751'4976","reported_seq":5063,"reported_epoch":751,"state":"active+clean","last_fresh":"2024-08-19T14:41:21.697357+0000","last_change":"2024-08-19T14:40:43.381073+0000","last_active":"2024-08-19T14:41:21.697357+0000","last_peered":"2024-08-19T14:41:21.697357+0000","last_clean":"2024-08-19T14:41:21.697357+0000","last_became_active":"2024-08-19T14:40:43.378698+0000","last_became_peered":"2024-08-19T14:40:43.378698+0000","last_unstale":"2024-08-19T14:41:21.697357+0000","last_undegraded":"2024-08-19T14:41:21.697357+0000","last_fullsized":"2024-08-19T14:41:21.697357+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":717,"last_epoch_clean":732,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.182354+0000","objects_scrubbed":0,"log_size":4976,"log_dups_size":0,"ondisk_log_size":4976,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:22.025740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":190251008,"num_objects":2903,"num_object_clones":0,"num_object_copies":8709,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2903,"num_write_kb":185792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,2],"acting":[11,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.f","version":"70'139","reported_seq":1131,"reported_epoch":749,"state":"active+clean","last_fresh":"2024-08-19T14:41:18.623634+0000","last_change":"2024-08-19T14:40:52.447312+0000","last_active":"2024-08-19T14:41:18.623634+0000","last_peered":"2024-08-19T14:41:18.623634+0000","last_clean":"2024-08-19T14:41:18.623634+0000","last_became_active":"2024-08-19T14:40:52.446953+0000","last_became_peered":"2024-08-19T14:40:52.446953+0000","last_unstale":"2024-08-19T14:41:18.623634+0000","last_undegraded":"2024-08-19T14:41:18.623634+0000","last_fullsized":"2024-08-19T14:41:18.623634+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:39:59.812612+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:39:59.812612+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:26.915462+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,8],"acting":[4,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.5","version":"749'5020","reported_seq":5133,"reported_epoch":749,"state":"active+clean","last_fresh":"2024-08-19T14:41:18.623549+0000","last_change":"2024-08-19T14:40:44.185558+0000","last_active":"2024-08-19T14:41:18.623549+0000","last_peered":"2024-08-19T14:41:18.623549+0000","last_clean":"2024-08-19T14:41:18.623549+0000","last_became_active":"2024-08-19T14:40:43.370152+0000","last_became_peered":"2024-08-19T14:40:43.370152+0000","last_unstale":"2024-08-19T14:41:18.623549+0000","last_undegraded":"2024-08-19T14:41:18.623549+0000","last_fullsized":"2024-08-19T14:41:18.623549+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":717,"last_epoch_clean":732,"parent":"0.0","parent_split_bits":0,"last_scrub":"732'4340","last_scrub_stamp":"2024-08-19T14:40:44.185450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:40:44.185450+0000","objects_scrubbed":2196,"log_size":5020,"log_dups_size":0,"ondisk_log_size":5020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:57.251230+0000","scrub_duration":371,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189005824,"num_objects":2884,"num_object_clones":0,"num_object_copies":8652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2884,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":688,"num_write_kb":44032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,11],"acting":[4,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.c","version":"70'139","reported_seq":434,"reported_epoch":751,"state":"active+clean","last_fresh":"2024-08-19T14:41:20.653099+0000","last_change":"2024-08-19T14:40:52.460885+0000","last_active":"2024-08-19T14:41:20.653099+0000","last_peered":"2024-08-19T14:41:20.653099+0000","last_clean":"2024-08-19T14:41:20.653099+0000","last_became_active":"2024-08-19T14:40:52.460464+0000","last_became_peered":"2024-08-19T14:40:52.460464+0000","last_unstale":"2024-08-19T14:41:20.653099+0000","last_undegraded":"2024-08-19T14:41:20.653099+0000","last_fullsized":"2024-08-19T14:41:20.653099+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:40:00.947341+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:22:52.164586+0000","last_clean_scrub_stamp":"2024-08-19T14:40:00.947341+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:21.023277+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8945,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":3,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":367750,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,5],"acting":[0,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"751'4995","reported_seq":5110,"reported_epoch":751,"state":"active+clean","last_fresh":"2024-08-19T14:41:21.784394+0000","last_change":"2024-08-19T14:40:44.356246+0000","last_active":"2024-08-19T14:41:21.784394+0000","last_peered":"2024-08-19T14:41:21.784394+0000","last_clean":"2024-08-19T14:41:21.784394+0000","last_became_active":"2024-08-19T14:40:43.364933+0000","last_became_peered":"2024-08-19T14:40:43.364933+0000","last_unstale":"2024-08-19T14:41:21.784394+0000","last_undegraded":"2024-08-19T14:41:21.784394+0000","last_fullsized":"2024-08-19T14:41:21.784394+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":717,"last_epoch_clean":732,"parent":"0.0","parent_split_bits":0,"last_scrub":"732'4215","last_scrub_stamp":"2024-08-19T14:40:44.355947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:40:44.355947+0000","objects_scrubbed":2072,"log_size":4995,"log_dups_size":0,"ondisk_log_size":4995,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:42:05.348091+0000","scrub_duration":403,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":187891712,"num_objects":2867,"num_object_clones":0,"num_object_copies":8601,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2867,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":795,"num_write_kb":50880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,9],"acting":[11,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.d","version":"70'139","reported_seq":1376,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-08-19T14:41:19.653180+0000","last_change":"2024-08-19T14:40:08.911300+0000","last_active":"2024-08-19T14:41:19.653180+0000","last_peered":"2024-08-19T14:41:19.653180+0000","last_clean":"2024-08-19T14:41:19.653180+0000","last_became_active":"2024-08-19T14:18:26.882002+0000","last_became_peered":"2024-08-19T14:18:26.882002+0000","last_unstale":"2024-08-19T14:41:19.653180+0000","last_undegraded":"2024-08-19T14:41:19.653180+0000","last_fullsized":"2024-08-19T14:41:19.653180+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:40:08.911204+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:24:50.276408+0000","last_clean_scrub_stamp":"2024-08-19T14:40:08.911204+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:27.367708+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"70'139","reported_seq":1150,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-08-19T14:41:08.604883+0000","last_change":"2024-08-19T14:40:04.665520+0000","last_active":"2024-08-19T14:41:08.604883+0000","last_peered":"2024-08-19T14:41:08.604883+0000","last_clean":"2024-08-19T14:41:08.604883+0000","last_became_active":"2024-08-19T14:27:31.838968+0000","last_became_peered":"2024-08-19T14:27:31.838968+0000","last_unstale":"2024-08-19T14:41:08.604883+0000","last_undegraded":"2024-08-19T14:41:08.604883+0000","last_fullsized":"2024-08-19T14:41:08.604883+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:40:04.665441+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:28:35.206115+0000","last_clean_scrub_stamp":"2024-08-19T14:40:04.665441+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:25.039402+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.13","version":"70'139","reported_seq":1692,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-08-19T14:41:08.605259+0000","last_change":"2024-08-19T14:40:05.659348+0000","last_active":"2024-08-19T14:41:08.605259+0000","last_peered":"2024-08-19T14:41:08.605259+0000","last_clean":"2024-08-19T14:41:08.605259+0000","last_became_active":"2024-08-19T14:30:47.939849+0000","last_became_peered":"2024-08-19T14:30:47.939849+0000","last_unstale":"2024-08-19T14:41:08.605259+0000","last_undegraded":"2024-08-19T14:41:08.605259+0000","last_fullsized":"2024-08-19T14:41:08.605259+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:40:05.659275+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:33:35.786105+0000","last_clean_scrub_stamp":"2024-08-19T14:40:05.659275+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:30.843663+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.19","version":"751'5027","reported_seq":8953,"reported_epoch":751,"state":"active+recovering+undersized+remapped","last_fresh":"2024-08-19T14:41:20.960906+0000","last_change":"2024-08-19T14:40:46.710101+0000","last_active":"2024-08-19T14:41:20.960906+0000","last_peered":"2024-08-19T14:41:20.960906+0000","last_clean":"2024-08-19T14:40:44.373041+0000","last_became_active":"2024-08-19T14:40:46.610566+0000","last_became_peered":"2024-08-19T14:40:46.610566+0000","last_unstale":"2024-08-19T14:41:20.960906+0000","last_undegraded":"2024-08-19T14:41:20.960906+0000","last_fullsized":"2024-08-19T14:40:46.386785+0000","mapping_epoch":734,"log_start":"0'0","ondisk_log_start":"0'0","created":731,"last_epoch_clean":732,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.182354+0000","objects_scrubbed":0,"log_size":5027,"log_dups_size":0,"ondisk_log_size":5027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":192282624,"num_objects":2934,"num_object_clones":0,"num_object_copies":8802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1796,"num_objects_unfound":0,"num_objects_dirty":2934,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2934,"num_write_kb":187776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1279,"num_bytes_recovered":83820544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,8],"acting":[0,5],"avail_no_missing":["0","3","5"],"object_location_counts":[{"shards":"0,3,5","objects":898},{"shards":"0,5","objects":2036}],"blocked_by":[],"up_primary":4,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"70'139","reported_seq":1324,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-08-19T14:41:19.651049+0000","last_change":"2024-08-19T14:40:32.035691+0000","last_active":"2024-08-19T14:41:19.651049+0000","last_peered":"2024-08-19T14:41:19.651049+0000","last_clean":"2024-08-19T14:41:19.651049+0000","last_became_active":"2024-08-19T14:27:31.839611+0000","last_became_peered":"2024-08-19T14:27:31.839611+0000","last_unstale":"2024-08-19T14:41:19.651049+0000","last_undegraded":"2024-08-19T14:41:19.651049+0000","last_fullsized":"2024-08-19T14:41:19.651049+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:40:32.035620+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:34:58.616646+0000","last_clean_scrub_stamp":"2024-08-19T14:40:32.035620+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:39.395084+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.18","version":"751'5080","reported_seq":7683,"reported_epoch":751,"state":"active+recovering+undersized+remapped","last_fresh":"2024-08-19T14:41:20.835001+0000","last_change":"2024-08-19T14:40:46.664649+0000","last_active":"2024-08-19T14:41:20.835001+0000","last_peered":"2024-08-19T14:41:20.835001+0000","last_clean":"2024-08-19T14:40:44.335364+0000","last_became_active":"2024-08-19T14:40:46.531622+0000","last_became_peered":"2024-08-19T14:40:46.531622+0000","last_unstale":"2024-08-19T14:41:20.835001+0000","last_undegraded":"2024-08-19T14:41:20.835001+0000","last_fullsized":"2024-08-19T14:40:46.385050+0000","mapping_epoch":734,"log_start":"0'0","ondisk_log_start":"0'0","created":731,"last_epoch_clean":732,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.182354+0000","objects_scrubbed":0,"log_size":5080,"log_dups_size":0,"ondisk_log_size":5080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":194117632,"num_objects":2962,"num_object_clones":0,"num_object_copies":8886,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2706,"num_objects_unfound":0,"num_objects_dirty":2962,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2962,"num_write_kb":189568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":838,"num_bytes_recovered":54919168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,5],"acting":[3,5],"avail_no_missing":["3","5","9"],"object_location_counts":[{"shards":"3,5","objects":1609},{"shards":"3,5,9","objects":1353}],"blocked_by":[],"up_primary":4,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.11","version":"70'139","reported_seq":1382,"reported_epoch":751,"state":"active+clean","last_fresh":"2024-08-19T14:41:20.656829+0000","last_change":"2024-08-19T14:40:04.812392+0000","last_active":"2024-08-19T14:41:20.656829+0000","last_peered":"2024-08-19T14:41:20.656829+0000","last_clean":"2024-08-19T14:41:20.656829+0000","last_became_active":"2024-08-19T14:34:15.731049+0000","last_became_peered":"2024-08-19T14:34:15.731049+0000","last_unstale":"2024-08-19T14:41:20.656829+0000","last_undegraded":"2024-08-19T14:41:20.656829+0000","last_fullsized":"2024-08-19T14:41:20.656829+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:40:04.812269+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:24:50.547428+0000","last_clean_scrub_stamp":"2024-08-19T14:40:04.812269+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:33.537909+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"70'139","reported_seq":1312,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-08-19T14:41:08.604754+0000","last_change":"2024-08-19T14:40:52.459431+0000","last_active":"2024-08-19T14:41:08.604754+0000","last_peered":"2024-08-19T14:41:08.604754+0000","last_clean":"2024-08-19T14:41:08.604754+0000","last_became_active":"2024-08-19T14:40:52.458344+0000","last_became_peered":"2024-08-19T14:40:52.458344+0000","last_unstale":"2024-08-19T14:41:08.604754+0000","last_undegraded":"2024-08-19T14:41:08.604754+0000","last_fullsized":"2024-08-19T14:41:08.604754+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:40:03.700675+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:28:34.216370+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.700675+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,11],"acting":[1,0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"70'139","reported_seq":1458,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-08-19T14:41:19.648633+0000","last_change":"2024-08-19T14:40:30.299862+0000","last_active":"2024-08-19T14:41:19.648633+0000","last_peered":"2024-08-19T14:41:19.648633+0000","last_clean":"2024-08-19T14:41:19.648633+0000","last_became_active":"2024-08-19T14:18:31.901556+0000","last_became_peered":"2024-08-19T14:18:31.901556+0000","last_unstale":"2024-08-19T14:41:19.648633+0000","last_undegraded":"2024-08-19T14:41:19.648633+0000","last_fullsized":"2024-08-19T14:41:19.648633+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:40:30.299750+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:40:30.299750+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:36.225861+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,0],"acting":[8,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.14","version":"70'139","reported_seq":1479,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-08-19T14:41:19.654371+0000","last_change":"2024-08-19T14:40:03.932359+0000","last_active":"2024-08-19T14:41:19.654371+0000","last_peered":"2024-08-19T14:41:19.654371+0000","last_clean":"2024-08-19T14:41:19.654371+0000","last_became_active":"2024-08-19T14:35:12.529792+0000","last_became_peered":"2024-08-19T14:35:12.529792+0000","last_unstale":"2024-08-19T14:41:19.654371+0000","last_undegraded":"2024-08-19T14:41:19.654371+0000","last_fullsized":"2024-08-19T14:41:19.654371+0000","mapping_epoch":635,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":636,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:40:03.932271+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.932271+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:23.358249+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,9],"acting":[7,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.15","version":"70'139","reported_seq":1331,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-08-19T14:41:19.653132+0000","last_change":"2024-08-19T14:40:07.881907+0000","last_active":"2024-08-19T14:41:19.653132+0000","last_peered":"2024-08-19T14:41:19.653132+0000","last_clean":"2024-08-19T14:41:19.653132+0000","last_became_active":"2024-08-19T14:18:31.912496+0000","last_became_peered":"2024-08-19T14:18:31.912496+0000","last_unstale":"2024-08-19T14:41:19.653132+0000","last_undegraded":"2024-08-19T14:41:19.653132+0000","last_fullsized":"2024-08-19T14:41:19.653132+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:40:07.881806+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:20:09.180923+0000","last_clean_scrub_stamp":"2024-08-19T14:40:07.881806+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:32.759701+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.13","version":"751'5024","reported_seq":9673,"reported_epoch":751,"state":"active+clean","last_fresh":"2024-08-19T14:41:21.718548+0000","last_change":"2024-08-19T14:41:06.594474+0000","last_active":"2024-08-19T14:41:21.718548+0000","last_peered":"2024-08-19T14:41:21.718548+0000","last_clean":"2024-08-19T14:41:21.718548+0000","last_became_active":"2024-08-19T14:41:06.590807+0000","last_became_peered":"2024-08-19T14:41:06.590807+0000","last_unstale":"2024-08-19T14:41:21.718548+0000","last_undegraded":"2024-08-19T14:41:21.718548+0000","last_fullsized":"2024-08-19T14:41:21.718548+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":731,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.182354+0000","objects_scrubbed":0,"log_size":5024,"log_dups_size":0,"ondisk_log_size":5024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:24.004746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":191299584,"num_objects":2919,"num_object_clones":0,"num_object_copies":8757,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2919,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2919,"num_write_kb":186816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2275,"num_bytes_recovered":149094400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,9],"acting":[11,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1a","version":"70'139","reported_seq":1315,"reported_epoch":749,"state":"active+clean","last_fresh":"2024-08-19T14:41:18.623591+0000","last_change":"2024-08-19T14:40:00.753735+0000","last_active":"2024-08-19T14:41:18.623591+0000","last_peered":"2024-08-19T14:41:18.623591+0000","last_clean":"2024-08-19T14:41:18.623591+0000","last_became_active":"2024-08-19T14:18:31.901229+0000","last_became_peered":"2024-08-19T14:18:31.901229+0000","last_unstale":"2024-08-19T14:41:18.623591+0000","last_undegraded":"2024-08-19T14:41:18.623591+0000","last_fullsized":"2024-08-19T14:41:18.623591+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:40:00.753636+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:40:00.753636+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.12","version":"751'4995","reported_seq":9453,"reported_epoch":751,"state":"active+recovering+undersized+remapped","last_fresh":"2024-08-19T14:41:20.970347+0000","last_change":"2024-08-19T14:40:46.653159+0000","last_active":"2024-08-19T14:41:20.970347+0000","last_peered":"2024-08-19T14:41:20.970347+0000","last_clean":"2024-08-19T14:40:44.376799+0000","last_became_active":"2024-08-19T14:40:46.573550+0000","last_became_peered":"2024-08-19T14:40:46.573550+0000","last_unstale":"2024-08-19T14:41:20.970347+0000","last_undegraded":"2024-08-19T14:41:20.970347+0000","last_fullsized":"2024-08-19T14:40:46.389991+0000","mapping_epoch":734,"log_start":"0'0","ondisk_log_start":"0'0","created":731,"last_epoch_clean":732,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.182354+0000","objects_scrubbed":0,"log_size":4995,"log_dups_size":0,"ondisk_log_size":4995,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":191463424,"num_objects":2922,"num_object_clones":0,"num_object_copies":8766,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1388,"num_objects_unfound":0,"num_objects_dirty":2922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2922,"num_write_kb":186976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1456,"num_bytes_recovered":95420416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[0,4],"avail_no_missing":["0","4","9"],"object_location_counts":[{"shards":"0,4","objects":2228},{"shards":"0,4,9","objects":694}],"blocked_by":[],"up_primary":7,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1b","version":"70'139","reported_seq":1706,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-08-19T14:41:19.649494+0000","last_change":"2024-08-19T14:40:43.238933+0000","last_active":"2024-08-19T14:41:19.649494+0000","last_peered":"2024-08-19T14:41:19.649494+0000","last_clean":"2024-08-19T14:41:19.649494+0000","last_became_active":"2024-08-19T14:34:15.718427+0000","last_became_peered":"2024-08-19T14:34:15.718427+0000","last_unstale":"2024-08-19T14:41:19.649494+0000","last_undegraded":"2024-08-19T14:41:19.649494+0000","last_fullsized":"2024-08-19T14:41:19.649494+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:40:43.238820+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:37:59.510697+0000","last_clean_scrub_stamp":"2024-08-19T14:40:43.238820+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:56.831246+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.11","version":"748'4989","reported_seq":9267,"reported_epoch":748,"state":"active+recovering+undersized+remapped","last_fresh":"2024-08-19T14:41:18.439967+0000","last_change":"2024-08-19T14:40:46.710903+0000","last_active":"2024-08-19T14:41:18.439967+0000","last_peered":"2024-08-19T14:41:18.439967+0000","last_clean":"2024-08-19T14:40:44.368364+0000","last_became_active":"2024-08-19T14:40:46.620378+0000","last_became_peered":"2024-08-19T14:40:46.620378+0000","last_unstale":"2024-08-19T14:41:18.439967+0000","last_undegraded":"2024-08-19T14:41:18.439967+0000","last_fullsized":"2024-08-19T14:40:46.387056+0000","mapping_epoch":734,"log_start":"0'0","ondisk_log_start":"0'0","created":731,"last_epoch_clean":732,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.182354+0000","objects_scrubbed":0,"log_size":4989,"log_dups_size":0,"ondisk_log_size":4989,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189038592,"num_objects":2885,"num_object_clones":0,"num_object_copies":8655,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1566,"num_objects_unfound":0,"num_objects_dirty":2885,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2885,"num_write_kb":184608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1398,"num_bytes_recovered":91619328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,9],"acting":[1,9],"avail_no_missing":["1","2","9"],"object_location_counts":[{"shards":"1,2,9","objects":783},{"shards":"1,9","objects":2102}],"blocked_by":[],"up_primary":7,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.18","version":"70'139","reported_seq":1259,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-08-19T14:41:19.650962+0000","last_change":"2024-08-19T14:39:59.937810+0000","last_active":"2024-08-19T14:41:19.650962+0000","last_peered":"2024-08-19T14:41:19.650962+0000","last_clean":"2024-08-19T14:41:19.650962+0000","last_became_active":"2024-08-19T14:35:11.528530+0000","last_became_peered":"2024-08-19T14:35:11.528530+0000","last_unstale":"2024-08-19T14:41:19.650962+0000","last_undegraded":"2024-08-19T14:41:19.650962+0000","last_fullsized":"2024-08-19T14:41:19.650962+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:39:59.937668+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:27:33.084161+0000","last_clean_scrub_stamp":"2024-08-19T14:39:59.937668+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"751'5012","reported_seq":7919,"reported_epoch":751,"state":"active+recovering+undersized+remapped","last_fresh":"2024-08-19T14:41:20.828440+0000","last_change":"2024-08-19T14:40:46.643254+0000","last_active":"2024-08-19T14:41:20.828440+0000","last_peered":"2024-08-19T14:41:20.828440+0000","last_clean":"2024-08-19T14:40:44.355963+0000","last_became_active":"2024-08-19T14:40:46.571954+0000","last_became_peered":"2024-08-19T14:40:46.571954+0000","last_unstale":"2024-08-19T14:41:20.828440+0000","last_undegraded":"2024-08-19T14:41:20.828440+0000","last_fullsized":"2024-08-19T14:40:46.385187+0000","mapping_epoch":734,"log_start":"0'0","ondisk_log_start":"0'0","created":731,"last_epoch_clean":732,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.182354+0000","objects_scrubbed":0,"log_size":5012,"log_dups_size":0,"ondisk_log_size":5012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":192937984,"num_objects":2944,"num_object_clones":0,"num_object_copies":8832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2306,"num_objects_unfound":0,"num_objects_dirty":2944,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2944,"num_write_kb":188416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":939,"num_bytes_recovered":61538304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,9],"acting":[3,9],"avail_no_missing":["3","5","9"],"object_location_counts":[{"shards":"3,5,9","objects":1153},{"shards":"3,9","objects":1791}],"blocked_by":[],"up_primary":7,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.19","version":"70'139","reported_seq":1679,"reported_epoch":751,"state":"active+clean","last_fresh":"2024-08-19T14:41:20.656728+0000","last_change":"2024-08-19T14:39:32.723246+0000","last_active":"2024-08-19T14:41:20.656728+0000","last_peered":"2024-08-19T14:41:20.656728+0000","last_clean":"2024-08-19T14:41:20.656728+0000","last_became_active":"2024-08-19T14:18:31.902547+0000","last_became_peered":"2024-08-19T14:18:31.902547+0000","last_unstale":"2024-08-19T14:41:20.656728+0000","last_undegraded":"2024-08-19T14:41:20.656728+0000","last_fullsized":"2024-08-19T14:41:20.656728+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:39:32.722703+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:39:32.722703+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.17","version":"748'4885","reported_seq":8417,"reported_epoch":748,"state":"active+recovering+undersized+remapped","last_fresh":"2024-08-19T14:41:18.411401+0000","last_change":"2024-08-19T14:40:46.728032+0000","last_active":"2024-08-19T14:41:18.411401+0000","last_peered":"2024-08-19T14:41:18.411401+0000","last_clean":"2024-08-19T14:40:44.365116+0000","last_became_active":"2024-08-19T14:40:46.673460+0000","last_became_peered":"2024-08-19T14:40:46.673460+0000","last_unstale":"2024-08-19T14:41:18.411401+0000","last_undegraded":"2024-08-19T14:41:18.411401+0000","last_fullsized":"2024-08-19T14:40:46.386686+0000","mapping_epoch":734,"log_start":"0'0","ondisk_log_start":"0'0","created":731,"last_epoch_clean":732,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.182354+0000","objects_scrubbed":0,"log_size":4885,"log_dups_size":0,"ondisk_log_size":4885,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":185729024,"num_objects":2834,"num_object_clones":0,"num_object_copies":8502,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3588,"num_objects_unfound":0,"num_objects_dirty":2834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2834,"num_write_kb":181376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":862,"num_bytes_recovered":56492032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,8],"acting":[1,0],"avail_no_missing":["1","2","11"],"object_location_counts":[{"shards":"0,1","objects":1638},{"shards":"1,2,11","objects":1196}],"blocked_by":[],"up_primary":7,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1e","version":"70'139","reported_seq":704,"reported_epoch":751,"state":"active+clean","last_fresh":"2024-08-19T14:41:20.685799+0000","last_change":"2024-08-19T14:40:31.847951+0000","last_active":"2024-08-19T14:41:20.685799+0000","last_peered":"2024-08-19T14:41:20.685799+0000","last_clean":"2024-08-19T14:41:20.685799+0000","last_became_active":"2024-08-19T14:30:47.909974+0000","last_became_peered":"2024-08-19T14:30:47.909974+0000","last_unstale":"2024-08-19T14:41:20.685799+0000","last_undegraded":"2024-08-19T14:41:20.685799+0000","last_fullsized":"2024-08-19T14:41:20.685799+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:40:31.847852+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:27:04.780939+0000","last_clean_scrub_stamp":"2024-08-19T14:40:31.847852+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:52.376557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.15","version":"748'5022","reported_seq":9275,"reported_epoch":748,"state":"active+recovering+undersized+remapped","last_fresh":"2024-08-19T14:41:18.445270+0000","last_change":"2024-08-19T14:40:46.662034+0000","last_active":"2024-08-19T14:41:18.445270+0000","last_peered":"2024-08-19T14:41:18.445270+0000","last_clean":"2024-08-19T14:40:44.330923+0000","last_became_active":"2024-08-19T14:40:46.614562+0000","last_became_peered":"2024-08-19T14:40:46.614562+0000","last_unstale":"2024-08-19T14:41:18.445270+0000","last_undegraded":"2024-08-19T14:41:18.445270+0000","last_fullsized":"2024-08-19T14:40:46.386996+0000","mapping_epoch":734,"log_start":"0'0","ondisk_log_start":"0'0","created":731,"last_epoch_clean":732,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.182354+0000","objects_scrubbed":0,"log_size":5022,"log_dups_size":0,"ondisk_log_size":5022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188907520,"num_objects":2883,"num_object_clones":0,"num_object_copies":8649,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1614,"num_objects_unfound":0,"num_objects_dirty":2883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2883,"num_write_kb":184480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1390,"num_bytes_recovered":91095040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,1],"acting":[1,7],"avail_no_missing":["1","4","11"],"object_location_counts":[{"shards":"1,4,11","objects":807},{"shards":"1,7","objects":2076}],"blocked_by":[],"up_primary":7,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"70'139","reported_seq":1460,"reported_epoch":750,"state":"active+clean","last_fresh":"2024-08-19T14:41:19.649405+0000","last_change":"2024-08-19T14:40:31.338726+0000","last_active":"2024-08-19T14:41:19.649405+0000","last_peered":"2024-08-19T14:41:19.649405+0000","last_clean":"2024-08-19T14:41:19.649405+0000","last_became_active":"2024-08-19T14:30:47.908866+0000","last_became_peered":"2024-08-19T14:30:47.908866+0000","last_unstale":"2024-08-19T14:41:19.649405+0000","last_undegraded":"2024-08-19T14:41:19.649405+0000","last_fullsized":"2024-08-19T14:41:19.649405+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:40:31.338409+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:40:31.338409+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:41:51.846061+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,11],"acting":[8,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.14","version":"748'4849","reported_seq":12548,"reported_epoch":748,"state":"active+recovering+undersized+remapped","last_fresh":"2024-08-19T14:41:16.538346+0000","last_change":"2024-08-19T14:40:46.602238+0000","last_active":"2024-08-19T14:41:16.538346+0000","last_peered":"2024-08-19T14:41:16.538346+0000","last_clean":"2024-08-19T14:40:44.367642+0000","last_became_active":"2024-08-19T14:40:46.566507+0000","last_became_peered":"2024-08-19T14:40:46.566507+0000","last_unstale":"2024-08-19T14:41:16.538346+0000","last_undegraded":"2024-08-19T14:41:16.538346+0000","last_fullsized":"2024-08-19T14:40:46.386657+0000","mapping_epoch":734,"log_start":"0'0","ondisk_log_start":"0'0","created":731,"last_epoch_clean":732,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:40:03.182354+0000","objects_scrubbed":0,"log_size":4849,"log_dups_size":0,"ondisk_log_size":4849,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":181698560,"num_objects":2773,"num_object_clones":0,"num_object_copies":8319,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":858,"num_objects_unfound":0,"num_objects_dirty":2773,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2773,"num_write_kb":177440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1904,"num_bytes_recovered":124780544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[5,0],"avail_no_missing":["5","9","11"],"object_location_counts":[{"shards":"0,5","objects":2487},{"shards":"5,9,11","objects":286}],"blocked_by":[],"up_primary":4,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1d","version":"70'139","reported_seq":1357,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-08-19T14:41:08.605997+0000","last_change":"2024-08-19T14:39:57.775710+0000","last_active":"2024-08-19T14:41:08.605997+0000","last_peered":"2024-08-19T14:41:08.605997+0000","last_clean":"2024-08-19T14:41:08.605997+0000","last_became_active":"2024-08-19T14:35:11.540488+0000","last_became_peered":"2024-08-19T14:35:11.540488+0000","last_unstale":"2024-08-19T14:41:08.605997+0000","last_undegraded":"2024-08-19T14:41:08.605997+0000","last_fullsized":"2024-08-19T14:41:08.605997+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:39:57.775631+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:39:57.775631+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":26,"stat_sum":{"num_bytes":6082199552,"num_objects":92808,"num_object_clones":0,"num_object_copies":278424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":22149,"num_objects_unfound":0,"num_objects_dirty":92808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":84211,"num_write_kb":5389440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19253,"num_bytes_recovered":1261764608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20351025152,"data_stored":20351025152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":134883,"ondisk_log_size":134883,"up":78,"acting":68,"num_store_stats":10},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":271633,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":3,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":568998,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1925120,"data_stored":1914464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187816,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3365484,"kb_used_data":2418008,"kb_used_omap":1,"kb_used_meta":947454,"kb_avail":90379668,"statfs":{"total":95995035648,"available":92548780032,"internally_reserved":0,"allocated":2476040192,"data_stored":2472716242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":970193348},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":2314987372683,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":74,"seq":317827580273,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3841084,"kb_used_data":2863092,"kb_used_omap":1,"kb_used_meta":977982,"kb_avail":89904068,"statfs":{"total":95995035648,"available":92061765632,"internally_reserved":0,"allocated":2931806208,"data_stored":2928493926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":1001454017},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874616,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2263112,"kb_used_data":1437824,"kb_used_omap":1,"kb_used_meta":825278,"kb_avail":91482040,"statfs":{"total":95995035648,"available":93677608960,"internally_reserved":0,"allocated":1472331776,"data_stored":1469015645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":845085125},"hb_peers":[0,1,2,3,4,5,6,7,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":57,"seq":244813136251,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2393760,"kb_used_data":1374100,"kb_used_omap":1,"kb_used_meta":1019646,"kb_avail":91351392,"statfs":{"total":95995035648,"available":93543825408,"internally_reserved":0,"allocated":1407078400,"data_stored":1403754818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":1044117962},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":206158430590,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692226,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3181596,"kb_used_data":2149460,"kb_used_omap":1,"kb_used_meta":1032126,"kb_avail":90563556,"statfs":{"total":95995035648,"available":92737081344,"internally_reserved":0,"allocated":2201047040,"data_stored":2197723458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1056897477},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986566,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3056380,"kb_used_data":1992308,"kb_used_omap":1,"kb_used_meta":1064062,"kb_avail":90688772,"statfs":{"total":95995035648,"available":92865302528,"internally_reserved":0,"allocated":2040123392,"data_stored":2036811110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1089599941},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":111669150093,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":634,"seq":2723009265739,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3167508,"kb_used_data":2806408,"kb_used_omap":4,"kb_used_meta":361083,"kb_avail":90577644,"statfs":{"total":95995035648,"available":92751507456,"internally_reserved":0,"allocated":2873761792,"data_stored":2870442109,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4407,"internal_metadata":369749705},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575253,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3204924,"kb_used_data":2013236,"kb_used_omap":1,"kb_used_meta":1191678,"kb_avail":90540228,"statfs":{"total":95995035648,"available":92713193472,"internally_reserved":0,"allocated":2061553664,"data_stored":2058241382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1220278725},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3],"upper_bound":2},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738778,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2372060,"kb_used_data":1416268,"kb_used_omap":1,"kb_used_meta":955774,"kb_avail":91373092,"statfs":{"total":95995035648,"available":93566046208,"internally_reserved":0,"allocated":1450258432,"data_stored":1446938402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":978713029},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1450967040,"data_stored":1450967040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2062155776,"data_stored":2062155776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2880897024,"data_stored":2880897024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1345978368,"data_stored":1345978368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2055602176,"data_stored":2055602176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2209415168,"data_stored":2209415168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1432748032,"data_stored":1432748032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1483145216,"data_stored":1483145216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":2945843200,"data_stored":2945843200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":2484273152,"data_stored":2484273152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T14:41:34.527 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd pool set .mgr pgp_num 91 2024-08-19T14:41:34.531 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:41:34.560 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:41:34.605 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:41:34.643 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:34.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:34.644 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:41:34.745 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:41:34.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:34.787 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:34.787 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:41:34.888 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:41:34.925 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:34.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:34.926 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:41:35.027 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:41:35.048 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:35.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:35.049 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:41:35.150 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:41:35.194 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:35.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:35.195 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:41:35.296 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:41:35.297 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 84 16 105448 105432 78.4361 76.375 0.0121238 0.0127448 2024-08-19T14:41:35.328 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:35.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:35.329 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:41:35.430 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:41:35.461 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:35.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:35.462 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:41:35.563 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:41:35.596 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:35.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:35.597 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:41:35.698 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:41:35.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:35.735 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:35.735 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:41:35.836 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:41:35.867 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:35.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:35.869 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:41:35.970 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:41:36.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:36.002 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:36.002 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:41:36.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:35 smithi032 ceph-mon[20004]: pgmap v1807: 57 pgs: 1 creating+peering, 2 active+recovering+remapped, 4 active+recovering+undersized+remapped, 50 active+clean; 6.2 GiB data, 28 GiB used, 777 GiB / 805 GiB avail; 39 MiB/s wr, 621 op/s; 8259/302817 objects misplaced (2.727%); 39 MiB/s, 619 objects/s recovering 2024-08-19T14:41:36.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:35 smithi045 ceph-mon[20143]: pgmap v1807: 57 pgs: 1 creating+peering, 2 active+recovering+remapped, 4 active+recovering+undersized+remapped, 50 active+clean; 6.2 GiB data, 28 GiB used, 777 GiB / 805 GiB avail; 39 MiB/s wr, 621 op/s; 8259/302817 objects misplaced (2.727%); 39 MiB/s, 619 objects/s recovering 2024-08-19T14:41:36.103 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:41:36.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:35 smithi031 ceph-mon[16336]: pgmap v1807: 57 pgs: 1 creating+peering, 2 active+recovering+remapped, 4 active+recovering+undersized+remapped, 50 active+clean; 6.2 GiB data, 28 GiB used, 777 GiB / 805 GiB avail; 39 MiB/s wr, 621 op/s; 8259/302817 objects misplaced (2.727%); 39 MiB/s, 619 objects/s recovering 2024-08-19T14:41:36.126 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:36.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:36.127 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:41:36.228 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:41:36.255 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:36.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:36.256 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:41:36.297 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 85 15 106851 106836 78.5456 87.75 0.00513252 0.012727 2024-08-19T14:41:36.356 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:41:36.391 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:36.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:36.392 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:41:36.493 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:41:36.522 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:36.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:36.523 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:41:36.624 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:41:36.657 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:36.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:36.658 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:41:36.759 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:41:36.788 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:36.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:36.790 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:41:36.890 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:41:36.919 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:36.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:36.920 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:41:37.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:36 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:41:37.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:36 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:41:37.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:36 smithi032 ceph-mon[20004]: osdmap e758: 12 total, 12 up, 9 in 2024-08-19T14:41:37.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:36 smithi032 ceph-mon[20004]: 1.19 scrub starts 2024-08-19T14:41:37.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:36 smithi032 ceph-mon[20004]: 1.19 scrub ok 2024-08-19T14:41:37.021 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:41:37.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:36 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:41:37.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:36 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:41:37.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:36 smithi045 ceph-mon[20143]: osdmap e758: 12 total, 12 up, 9 in 2024-08-19T14:41:37.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:36 smithi045 ceph-mon[20143]: 1.19 scrub starts 2024-08-19T14:41:37.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:36 smithi045 ceph-mon[20143]: 1.19 scrub ok 2024-08-19T14:41:37.048 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:37.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:37.050 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:41:37.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:36 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:41:37.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:36 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:41:37.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:36 smithi031 ceph-mon[16336]: osdmap e758: 12 total, 12 up, 9 in 2024-08-19T14:41:37.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:36 smithi031 ceph-mon[16336]: 1.19 scrub starts 2024-08-19T14:41:37.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:36 smithi031 ceph-mon[16336]: 1.19 scrub ok 2024-08-19T14:41:37.151 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:41:37.175 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:37.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:37.176 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:41:37.277 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:41:37.297 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 86 16 107956 107940 78.4345 69 0.0026262 0.0126901 2024-08-19T14:41:37.302 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:37.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:37.304 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:41:37.404 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:41:37.427 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:37.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:37.428 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:41:37.529 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:41:37.551 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:37.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:37.553 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:41:37.653 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:41:37.676 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:37.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:37.678 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:41:37.779 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:41:37.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:37.819 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:37.819 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:41:37.922 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:41:37.953 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:37.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:37.954 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:41:38.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:37 smithi032 ceph-mon[20004]: 1.7 scrub starts 2024-08-19T14:41:38.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:37 smithi032 ceph-mon[20004]: 1.7 scrub ok 2024-08-19T14:41:38.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:37 smithi032 ceph-mon[20004]: pgmap v1809: 57 pgs: 2 active+recovering+remapped, 4 active+recovering+undersized+remapped, 51 active+clean; 6.4 GiB data, 28 GiB used, 777 GiB / 805 GiB avail; 67 MiB/s wr, 1.08k op/s; 6203/312372 objects misplaced (1.986%); 52 MiB/s, 829 objects/s recovering 2024-08-19T14:41:38.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:37 smithi032 ceph-mon[20004]: osdmap e759: 12 total, 12 up, 9 in 2024-08-19T14:41:38.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:37 smithi032 ceph-mon[20004]: 1.c scrub starts 2024-08-19T14:41:38.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:37 smithi032 ceph-mon[20004]: 1.c scrub ok 2024-08-19T14:41:38.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:37 smithi045 ceph-mon[20143]: 1.7 scrub starts 2024-08-19T14:41:38.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:37 smithi045 ceph-mon[20143]: 1.7 scrub ok 2024-08-19T14:41:38.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:37 smithi045 ceph-mon[20143]: pgmap v1809: 57 pgs: 2 active+recovering+remapped, 4 active+recovering+undersized+remapped, 51 active+clean; 6.4 GiB data, 28 GiB used, 777 GiB / 805 GiB avail; 67 MiB/s wr, 1.08k op/s; 6203/312372 objects misplaced (1.986%); 52 MiB/s, 829 objects/s recovering 2024-08-19T14:41:38.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:37 smithi045 ceph-mon[20143]: osdmap e759: 12 total, 12 up, 9 in 2024-08-19T14:41:38.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:37 smithi045 ceph-mon[20143]: 1.c scrub starts 2024-08-19T14:41:38.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:37 smithi045 ceph-mon[20143]: 1.c scrub ok 2024-08-19T14:41:38.055 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:41:38.086 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:38.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:38.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:41:38.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:37 smithi031 ceph-mon[16336]: 1.7 scrub starts 2024-08-19T14:41:38.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:37 smithi031 ceph-mon[16336]: 1.7 scrub ok 2024-08-19T14:41:38.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:37 smithi031 ceph-mon[16336]: pgmap v1809: 57 pgs: 2 active+recovering+remapped, 4 active+recovering+undersized+remapped, 51 active+clean; 6.4 GiB data, 28 GiB used, 777 GiB / 805 GiB avail; 67 MiB/s wr, 1.08k op/s; 6203/312372 objects misplaced (1.986%); 52 MiB/s, 829 objects/s recovering 2024-08-19T14:41:38.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:37 smithi031 ceph-mon[16336]: osdmap e759: 12 total, 12 up, 9 in 2024-08-19T14:41:38.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:37 smithi031 ceph-mon[16336]: 1.c scrub starts 2024-08-19T14:41:38.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:37 smithi031 ceph-mon[16336]: 1.c scrub ok 2024-08-19T14:41:38.188 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:41:38.219 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:38.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:38.220 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:41:38.297 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 87 16 108752 108736 78.1048 49.75 0.00813405 0.0127988 2024-08-19T14:41:38.321 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:41:38.353 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:38.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:38.354 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:41:38.455 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:41:38.476 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:38.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:38.478 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:41:38.579 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:41:38.607 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:38.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:38.608 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:41:38.709 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:41:38.730 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:38.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:38.731 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:41:38.832 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:41:38.860 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:38.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:38.861 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:41:38.962 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:41:38.989 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:38.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:38.990 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:41:38.998 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:38 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:41:38.998 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:38 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:41:38.998 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:38 smithi031 ceph-mon[16336]: osdmap e760: 12 total, 12 up, 9 in 2024-08-19T14:41:38.998 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:38 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:41:38.998 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:38 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:41:39.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:38 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:41:39.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:38 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:41:39.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:38 smithi032 ceph-mon[20004]: osdmap e760: 12 total, 12 up, 9 in 2024-08-19T14:41:39.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:38 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:41:39.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:38 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:41:39.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:38 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:41:39.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:38 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:41:39.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:38 smithi045 ceph-mon[20143]: osdmap e760: 12 total, 12 up, 9 in 2024-08-19T14:41:39.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:38 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:41:39.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:38 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:41:39.091 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:41:39.127 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:39.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:39.128 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:41:39.229 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:41:39.261 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:39.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:39.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:41:39.298 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 88 16 110026 110010 78.122 79.625 0.0199683 0.0127959 2024-08-19T14:41:39.363 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:41:39.398 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:39.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:39.399 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:41:39.500 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:41:39.530 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:39.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:39.532 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:41:39.633 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:41:39.666 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:39.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:39.667 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:41:39.768 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:41:39.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:39.795 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:39.795 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:41:39.829 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is unset 2024-08-19T14:41:39.896 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:41:39.929 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:39.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:39.930 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:41:40.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:39 smithi032 ceph-mon[20004]: pgmap v1812: 57 pgs: 2 active+recovering+remapped, 4 active+recovering+undersized+remapped, 51 active+clean; 6.5 GiB data, 28 GiB used, 776 GiB / 805 GiB avail; 62 MiB/s wr, 989 op/s; 6203/318093 objects misplaced (1.950%); 28 MiB/s, 440 objects/s recovering 2024-08-19T14:41:40.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:39 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/871897250' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:41:40.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:39 smithi032 ceph-mon[20004]: osdmap e761: 12 total, 12 up, 9 in 2024-08-19T14:41:40.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:39 smithi032 ceph-mon[20004]: 8.2 scrub starts 2024-08-19T14:41:40.031 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:41:40.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:39 smithi045 ceph-mon[20143]: pgmap v1812: 57 pgs: 2 active+recovering+remapped, 4 active+recovering+undersized+remapped, 51 active+clean; 6.5 GiB data, 28 GiB used, 776 GiB / 805 GiB avail; 62 MiB/s wr, 989 op/s; 6203/318093 objects misplaced (1.950%); 28 MiB/s, 440 objects/s recovering 2024-08-19T14:41:40.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:39 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/871897250' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:41:40.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:39 smithi045 ceph-mon[20143]: osdmap e761: 12 total, 12 up, 9 in 2024-08-19T14:41:40.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:39 smithi045 ceph-mon[20143]: 8.2 scrub starts 2024-08-19T14:41:40.071 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:40.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:40.072 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:41:40.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:39 smithi031 ceph-mon[16336]: pgmap v1812: 57 pgs: 2 active+recovering+remapped, 4 active+recovering+undersized+remapped, 51 active+clean; 6.5 GiB data, 28 GiB used, 776 GiB / 805 GiB avail; 62 MiB/s wr, 989 op/s; 6203/318093 objects misplaced (1.950%); 28 MiB/s, 440 objects/s recovering 2024-08-19T14:41:40.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:39 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/871897250' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:41:40.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:39 smithi031 ceph-mon[16336]: osdmap e761: 12 total, 12 up, 9 in 2024-08-19T14:41:40.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:39 smithi031 ceph-mon[16336]: 8.2 scrub starts 2024-08-19T14:41:40.173 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:41:40.204 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:40.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:40.205 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:41:40.298 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 89 16 111198 111182 78.0672 73.25 0.0126198 0.0128043 2024-08-19T14:41:40.305 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:41:40.340 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:40.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:40.341 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:41:40.442 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:41:40.475 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:40.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:40.476 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:41:40.577 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:41:40.607 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:40.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:40.609 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:41:40.709 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:41:40.745 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:40.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:40.746 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:41:40.847 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:41:40.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:40.883 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:40.883 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:41:40.984 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:41:40.992 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:40 smithi045 ceph-mon[20143]: 1.1d scrub starts 2024-08-19T14:41:40.992 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:40 smithi045 ceph-mon[20143]: 1.1d scrub ok 2024-08-19T14:41:40.992 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:40 smithi045 ceph-mon[20143]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:41:40.992 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:40 smithi045 ceph-mon[20143]: Cluster is now healthy 2024-08-19T14:41:40.992 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:40 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/871897250' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:41:40.992 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:40 smithi045 ceph-mon[20143]: osdmap e762: 12 total, 12 up, 9 in 2024-08-19T14:41:41.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:40 smithi032 ceph-mon[20004]: 1.1d scrub starts 2024-08-19T14:41:41.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:40 smithi032 ceph-mon[20004]: 1.1d scrub ok 2024-08-19T14:41:41.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:40 smithi032 ceph-mon[20004]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:41:41.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:40 smithi032 ceph-mon[20004]: Cluster is now healthy 2024-08-19T14:41:41.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:40 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/871897250' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:41:41.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:40 smithi032 ceph-mon[20004]: osdmap e762: 12 total, 12 up, 9 in 2024-08-19T14:41:41.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:40 smithi032 ceph-mon[20004]: 8.2 scrub ok 2024-08-19T14:41:41.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:40 smithi032 ceph-mon[20004]: 8.9 deep-scrub starts 2024-08-19T14:41:41.017 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:41.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:41.018 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:41:41.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:40 smithi031 ceph-mon[16336]: 1.1d scrub starts 2024-08-19T14:41:41.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:40 smithi031 ceph-mon[16336]: 1.1d scrub ok 2024-08-19T14:41:41.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:40 smithi031 ceph-mon[16336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:41:41.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:40 smithi031 ceph-mon[16336]: Cluster is now healthy 2024-08-19T14:41:41.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:40 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/871897250' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:41:41.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:40 smithi031 ceph-mon[16336]: osdmap e762: 12 total, 12 up, 9 in 2024-08-19T14:41:41.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:40 smithi031 ceph-mon[16336]: 8.2 scrub ok 2024-08-19T14:41:41.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:40 smithi031 ceph-mon[16336]: 8.9 deep-scrub starts 2024-08-19T14:41:41.119 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:41:41.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:41.155 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:41.155 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:41:41.256 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:41:41.291 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:41.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:41.292 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:41:41.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:40 smithi045 ceph-mon[20143]: 8.2 scrub ok 2024-08-19T14:41:41.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:40 smithi045 ceph-mon[20143]: 8.9 deep-scrub starts 2024-08-19T14:41:41.297 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 90 2 112461 112459 78.0865 79.8125 0.00743478 0.0128014 2024-08-19T14:41:41.393 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:41:41.418 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:41.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:41.418 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:41:41.520 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:41:41.544 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:41.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:41.546 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:41:41.647 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:41:41.672 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:41.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:41.673 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:41:41.774 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:41:41.799 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:41.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:41.800 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:41:41.902 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:41:41.924 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:41.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:41.925 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:41:42.026 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:41:42.051 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:42.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:42.052 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:41:42.154 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:41:42.179 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:42.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:42.180 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:41:42.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:42 smithi032 ceph-mon[20004]: 8.1 scrub starts 2024-08-19T14:41:42.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:42 smithi032 ceph-mon[20004]: pgmap v1815: 57 pgs: 2 active+recovering+remapped, 4 active+recovering+undersized+remapped, 51 active+clean; 6.5 GiB data, 28 GiB used, 776 GiB / 805 GiB avail; 44 MiB/s wr, 707 op/s; 6203/320859 objects misplaced (1.933%) 2024-08-19T14:41:42.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:42 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:41:42.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:42 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:41:42.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:42 smithi032 ceph-mon[20004]: 8.1 scrub ok 2024-08-19T14:41:42.281 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:41:42.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:42 smithi045 ceph-mon[20143]: 8.1 scrub starts 2024-08-19T14:41:42.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:42 smithi045 ceph-mon[20143]: pgmap v1815: 57 pgs: 2 active+recovering+remapped, 4 active+recovering+undersized+remapped, 51 active+clean; 6.5 GiB data, 28 GiB used, 776 GiB / 805 GiB avail; 44 MiB/s wr, 707 op/s; 6203/320859 objects misplaced (1.933%) 2024-08-19T14:41:42.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:42 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:41:42.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:42 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:41:42.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:42 smithi045 ceph-mon[20143]: 8.1 scrub ok 2024-08-19T14:41:42.298 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Total time run: 90.0151 2024-08-19T14:41:42.298 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Total writes made: 112461 2024-08-19T14:41:42.298 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Write size: 65536 2024-08-19T14:41:42.298 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Object size: 65536 2024-08-19T14:41:42.298 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Bandwidth (MB/sec): 78.0849 2024-08-19T14:41:42.298 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Stddev Bandwidth: 51.1399 2024-08-19T14:41:42.298 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Max bandwidth (MB/sec): 242.5 2024-08-19T14:41:42.298 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Min bandwidth (MB/sec): 0 2024-08-19T14:41:42.298 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Average IOPS: 1249 2024-08-19T14:41:42.298 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Stddev IOPS: 818.245 2024-08-19T14:41:42.298 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Max IOPS: 3880 2024-08-19T14:41:42.299 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Min IOPS: 0 2024-08-19T14:41:42.299 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Average Latency(s): 0.0128016 2024-08-19T14:41:42.299 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Stddev Latency(s): 0.0495005 2024-08-19T14:41:42.299 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Max latency(s): 2.31637 2024-08-19T14:41:42.299 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Min latency(s): 0.00108187 2024-08-19T14:41:42.302 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Cleaning up (deleting benchmark objects) 2024-08-19T14:41:42.305 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:42.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:42.306 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:41:42.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:42 smithi031 ceph-mon[16336]: 8.1 scrub starts 2024-08-19T14:41:42.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:42 smithi031 ceph-mon[16336]: pgmap v1815: 57 pgs: 2 active+recovering+remapped, 4 active+recovering+undersized+remapped, 51 active+clean; 6.5 GiB data, 28 GiB used, 776 GiB / 805 GiB avail; 44 MiB/s wr, 707 op/s; 6203/320859 objects misplaced (1.933%) 2024-08-19T14:41:42.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:42 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:41:42.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:42 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:41:42.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:42 smithi031 ceph-mon[16336]: 8.1 scrub ok 2024-08-19T14:41:42.406 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:41:42.433 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:42.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:42.434 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:41:42.535 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:41:42.558 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:42.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:42.559 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:41:42.660 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:41:42.687 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:42.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:42.688 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:41:42.788 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:41:42.812 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:42.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:42.813 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:41:42.914 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:41:42.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:42.941 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:42.941 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:41:43.042 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:41:43.074 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:43.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:43.075 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:41:43.176 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:41:43.202 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:43.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:43.203 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:41:43.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:43 smithi045 ceph-mon[20143]: osdmap e763: 12 total, 12 up, 9 in 2024-08-19T14:41:43.304 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:41:43.326 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:43.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:43.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:41:43.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:43 smithi031 ceph-mon[16336]: osdmap e763: 12 total, 12 up, 9 in 2024-08-19T14:41:43.428 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:41:43.449 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:43.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:43.450 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:41:43.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:43 smithi032 ceph-mon[20004]: osdmap e763: 12 total, 12 up, 9 in 2024-08-19T14:41:43.550 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:41:43.572 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:43.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:43.573 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:41:43.674 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:41:43.696 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:43.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:43.697 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:41:43.798 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:41:43.820 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:43.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:43.822 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:41:43.923 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:41:43.946 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:43.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:43.947 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:41:44.048 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:41:44.076 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:44.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:44.077 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:41:44.178 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:41:44.210 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:44.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:44.211 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:41:44.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:44 smithi045 ceph-mon[20143]: pgmap v1817: 57 pgs: 1 active+clean+scrubbing+deep, 2 active+recovering+remapped, 3 active+recovering+undersized+remapped, 51 active+clean; 6.8 GiB data, 29 GiB used, 776 GiB / 805 GiB avail; 29 MiB/s wr, 469 op/s; 3047/334791 objects misplaced (0.910%); 28 MiB/s, 446 objects/s recovering 2024-08-19T14:41:44.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:44 smithi045 ceph-mon[20143]: 8.13 scrub starts 2024-08-19T14:41:44.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:44 smithi045 ceph-mon[20143]: 1.16 scrub starts 2024-08-19T14:41:44.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:44 smithi045 ceph-mon[20143]: 1.16 scrub ok 2024-08-19T14:41:44.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:44 smithi045 ceph-mon[20143]: osdmap e764: 12 total, 12 up, 9 in 2024-08-19T14:41:44.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:44 smithi045 ceph-mon[20143]: 1.1 scrub starts 2024-08-19T14:41:44.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:44 smithi045 ceph-mon[20143]: 1.1 scrub ok 2024-08-19T14:41:44.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:44 smithi045 ceph-mon[20143]: 8.13 scrub ok 2024-08-19T14:41:44.311 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:41:44.342 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:44.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:44.343 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:41:44.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:44 smithi031 ceph-mon[16336]: pgmap v1817: 57 pgs: 1 active+clean+scrubbing+deep, 2 active+recovering+remapped, 3 active+recovering+undersized+remapped, 51 active+clean; 6.8 GiB data, 29 GiB used, 776 GiB / 805 GiB avail; 29 MiB/s wr, 469 op/s; 3047/334791 objects misplaced (0.910%); 28 MiB/s, 446 objects/s recovering 2024-08-19T14:41:44.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:44 smithi031 ceph-mon[16336]: 8.13 scrub starts 2024-08-19T14:41:44.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:44 smithi031 ceph-mon[16336]: 1.16 scrub starts 2024-08-19T14:41:44.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:44 smithi031 ceph-mon[16336]: 1.16 scrub ok 2024-08-19T14:41:44.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:44 smithi031 ceph-mon[16336]: osdmap e764: 12 total, 12 up, 9 in 2024-08-19T14:41:44.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:44 smithi031 ceph-mon[16336]: 1.1 scrub starts 2024-08-19T14:41:44.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:44 smithi031 ceph-mon[16336]: 1.1 scrub ok 2024-08-19T14:41:44.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:44 smithi031 ceph-mon[16336]: 8.13 scrub ok 2024-08-19T14:41:44.443 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:41:44.467 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:44.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:44.468 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:41:44.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:44 smithi032 ceph-mon[20004]: pgmap v1817: 57 pgs: 1 active+clean+scrubbing+deep, 2 active+recovering+remapped, 3 active+recovering+undersized+remapped, 51 active+clean; 6.8 GiB data, 29 GiB used, 776 GiB / 805 GiB avail; 29 MiB/s wr, 469 op/s; 3047/334791 objects misplaced (0.910%); 28 MiB/s, 446 objects/s recovering 2024-08-19T14:41:44.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:44 smithi032 ceph-mon[20004]: 8.13 scrub starts 2024-08-19T14:41:44.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:44 smithi032 ceph-mon[20004]: 1.16 scrub starts 2024-08-19T14:41:44.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:44 smithi032 ceph-mon[20004]: 1.16 scrub ok 2024-08-19T14:41:44.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:44 smithi032 ceph-mon[20004]: osdmap e764: 12 total, 12 up, 9 in 2024-08-19T14:41:44.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:44 smithi032 ceph-mon[20004]: 1.1 scrub starts 2024-08-19T14:41:44.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:44 smithi032 ceph-mon[20004]: 1.1 scrub ok 2024-08-19T14:41:44.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:44 smithi032 ceph-mon[20004]: 8.13 scrub ok 2024-08-19T14:41:44.569 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:41:44.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:44.605 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:44.605 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:41:44.706 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:41:44.738 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:44.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:44.739 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:41:44.840 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:41:44.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:44.869 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:44.869 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:41:44.970 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:41:44.991 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:44.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:44.992 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:41:45.093 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:41:45.124 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:45.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:45.125 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:41:45.226 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:41:45.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:45.258 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:45.259 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:41:45.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:45 smithi045 ceph-mon[20143]: 8.14 scrub starts 2024-08-19T14:41:45.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:45 smithi045 ceph-mon[20143]: osdmap e765: 12 total, 12 up, 9 in 2024-08-19T14:41:45.359 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:41:45.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:45 smithi031 ceph-mon[16336]: 8.14 scrub starts 2024-08-19T14:41:45.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:45 smithi031 ceph-mon[16336]: osdmap e765: 12 total, 12 up, 9 in 2024-08-19T14:41:45.388 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:45.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:45.389 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:41:45.490 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:41:45.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:45 smithi032 ceph-mon[20004]: 8.14 scrub starts 2024-08-19T14:41:45.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:45 smithi032 ceph-mon[20004]: osdmap e765: 12 total, 12 up, 9 in 2024-08-19T14:41:45.523 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:45.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:45.524 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:41:45.625 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:41:45.651 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:45.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:45.653 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:41:45.753 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:41:45.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:45.783 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:45.783 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:41:45.884 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:41:45.918 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:45.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:45.919 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:41:46.020 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:41:46.056 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:46.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:46.057 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:41:46.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:46 smithi031 ceph-mon[16336]: pgmap v1820: 57 pgs: 2 creating+peering, 1 active+clean+scrubbing+deep, 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 51 active+clean; 6.8 GiB data, 29 GiB used, 775 GiB / 805 GiB avail; 2070/335916 objects misplaced (0.616%); 37 MiB/s, 589 objects/s recovering 2024-08-19T14:41:46.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:46 smithi031 ceph-mon[16336]: 8.14 scrub ok 2024-08-19T14:41:46.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:46 smithi031 ceph-mon[16336]: 8.19 scrub starts 2024-08-19T14:41:46.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:46 smithi031 ceph-mon[16336]: 8.8 scrub starts 2024-08-19T14:41:46.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:46 smithi031 ceph-mon[16336]: 1.0 deep-scrub starts 2024-08-19T14:41:46.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:46 smithi031 ceph-mon[16336]: 1.0 deep-scrub ok 2024-08-19T14:41:46.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:46 smithi031 ceph-mon[16336]: 8.8 scrub ok 2024-08-19T14:41:46.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:46 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:41:46.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:46 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:41:46.158 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:41:46.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:46.188 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:46.189 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:41:46.290 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:41:46.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:46 smithi045 ceph-mon[20143]: pgmap v1820: 57 pgs: 2 creating+peering, 1 active+clean+scrubbing+deep, 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 51 active+clean; 6.8 GiB data, 29 GiB used, 775 GiB / 805 GiB avail; 2070/335916 objects misplaced (0.616%); 37 MiB/s, 589 objects/s recovering 2024-08-19T14:41:46.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:46 smithi045 ceph-mon[20143]: 8.14 scrub ok 2024-08-19T14:41:46.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:46 smithi045 ceph-mon[20143]: 8.19 scrub starts 2024-08-19T14:41:46.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:46 smithi045 ceph-mon[20143]: 8.8 scrub starts 2024-08-19T14:41:46.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:46 smithi045 ceph-mon[20143]: 1.0 deep-scrub starts 2024-08-19T14:41:46.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:46 smithi045 ceph-mon[20143]: 1.0 deep-scrub ok 2024-08-19T14:41:46.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:46 smithi045 ceph-mon[20143]: 8.8 scrub ok 2024-08-19T14:41:46.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:46 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:41:46.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:46 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:41:46.320 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:46.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:46.321 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:41:46.422 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:41:46.453 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:46.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:46.455 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:41:46.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:46 smithi032 ceph-mon[20004]: pgmap v1820: 57 pgs: 2 creating+peering, 1 active+clean+scrubbing+deep, 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 51 active+clean; 6.8 GiB data, 29 GiB used, 775 GiB / 805 GiB avail; 2070/335916 objects misplaced (0.616%); 37 MiB/s, 589 objects/s recovering 2024-08-19T14:41:46.522 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:46 smithi032 ceph-mon[20004]: 8.14 scrub ok 2024-08-19T14:41:46.522 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:46 smithi032 ceph-mon[20004]: 8.19 scrub starts 2024-08-19T14:41:46.522 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:46 smithi032 ceph-mon[20004]: 8.8 scrub starts 2024-08-19T14:41:46.523 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:46 smithi032 ceph-mon[20004]: 1.0 deep-scrub starts 2024-08-19T14:41:46.523 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:46 smithi032 ceph-mon[20004]: 1.0 deep-scrub ok 2024-08-19T14:41:46.523 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:46 smithi032 ceph-mon[20004]: 8.8 scrub ok 2024-08-19T14:41:46.523 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:46 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:41:46.523 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:46 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:41:46.555 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:41:46.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:46.586 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:46.587 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:41:46.688 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:41:46.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:46.720 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:46.721 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:41:46.821 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:41:46.852 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:46.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:46.853 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:41:46.954 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:41:46.991 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:46.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:46.992 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:41:47.093 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:41:47.117 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:47.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:47.118 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:41:47.220 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:41:47.258 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:47.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:47.259 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:41:47.360 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:41:47.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:47 smithi031 ceph-mon[16336]: 8.19 scrub ok 2024-08-19T14:41:47.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:47 smithi031 ceph-mon[16336]: 8.18 scrub starts 2024-08-19T14:41:47.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:47 smithi031 ceph-mon[16336]: 8.3 scrub starts 2024-08-19T14:41:47.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:47 smithi031 ceph-mon[16336]: 1.3 scrub starts 2024-08-19T14:41:47.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:47 smithi031 ceph-mon[16336]: 1.3 scrub ok 2024-08-19T14:41:47.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:47 smithi031 ceph-mon[16336]: osdmap e766: 12 total, 12 up, 9 in 2024-08-19T14:41:47.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:47 smithi031 ceph-mon[16336]: 8.e scrub starts 2024-08-19T14:41:47.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:47 smithi031 ceph-mon[16336]: 8.3 scrub ok 2024-08-19T14:41:47.383 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:47.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:47.384 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:41:47.485 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:41:47.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:47 smithi032 ceph-mon[20004]: 8.19 scrub ok 2024-08-19T14:41:47.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:47 smithi032 ceph-mon[20004]: 8.18 scrub starts 2024-08-19T14:41:47.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:47 smithi032 ceph-mon[20004]: 8.3 scrub starts 2024-08-19T14:41:47.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:47 smithi032 ceph-mon[20004]: 1.3 scrub starts 2024-08-19T14:41:47.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:47 smithi032 ceph-mon[20004]: 1.3 scrub ok 2024-08-19T14:41:47.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:47 smithi032 ceph-mon[20004]: osdmap e766: 12 total, 12 up, 9 in 2024-08-19T14:41:47.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:47 smithi032 ceph-mon[20004]: 8.e scrub starts 2024-08-19T14:41:47.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:47 smithi032 ceph-mon[20004]: 8.3 scrub ok 2024-08-19T14:41:47.518 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:47.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:47.519 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:41:47.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:47 smithi045 ceph-mon[20143]: 8.19 scrub ok 2024-08-19T14:41:47.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:47 smithi045 ceph-mon[20143]: 8.18 scrub starts 2024-08-19T14:41:47.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:47 smithi045 ceph-mon[20143]: 8.3 scrub starts 2024-08-19T14:41:47.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:47 smithi045 ceph-mon[20143]: 1.3 scrub starts 2024-08-19T14:41:47.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:47 smithi045 ceph-mon[20143]: 1.3 scrub ok 2024-08-19T14:41:47.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:47 smithi045 ceph-mon[20143]: osdmap e766: 12 total, 12 up, 9 in 2024-08-19T14:41:47.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:47 smithi045 ceph-mon[20143]: 8.e scrub starts 2024-08-19T14:41:47.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:47 smithi045 ceph-mon[20143]: 8.3 scrub ok 2024-08-19T14:41:47.621 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:41:47.643 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:47.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:47.644 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:41:47.745 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:41:47.773 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:47.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:47.774 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:41:47.874 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:41:47.897 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:47.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:47.898 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:41:47.999 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:41:48.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:48.032 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:48.032 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:41:48.133 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:41:48.159 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:48.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:48.160 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:41:48.261 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:41:48.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:48 smithi045 ceph-mon[20143]: pgmap v1822: 57 pgs: 1 active+clean+scrubbing+deep, 2 creating+peering, 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 52 active+clean; 6.8 GiB data, 29 GiB used, 775 GiB / 805 GiB avail; 1107/333432 objects misplaced (0.332%); 21 MiB/s, 331 objects/s recovering 2024-08-19T14:41:48.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:48 smithi045 ceph-mon[20143]: 8.18 scrub ok 2024-08-19T14:41:48.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:48 smithi045 ceph-mon[20143]: 8.6 scrub starts 2024-08-19T14:41:48.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:48 smithi045 ceph-mon[20143]: 1.e scrub starts 2024-08-19T14:41:48.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:48 smithi045 ceph-mon[20143]: 1.e scrub ok 2024-08-19T14:41:48.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:48 smithi045 ceph-mon[20143]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-08-19T14:41:48.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:48 smithi045 ceph-mon[20143]: 1.11 deep-scrub starts 2024-08-19T14:41:48.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:48 smithi045 ceph-mon[20143]: 1.11 deep-scrub ok 2024-08-19T14:41:48.299 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:48.299 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:48 smithi031 ceph-mon[16336]: pgmap v1822: 57 pgs: 1 active+clean+scrubbing+deep, 2 creating+peering, 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 52 active+clean; 6.8 GiB data, 29 GiB used, 775 GiB / 805 GiB avail; 1107/333432 objects misplaced (0.332%); 21 MiB/s, 331 objects/s recovering 2024-08-19T14:41:48.299 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:48 smithi031 ceph-mon[16336]: 8.18 scrub ok 2024-08-19T14:41:48.299 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:48 smithi031 ceph-mon[16336]: 8.6 scrub starts 2024-08-19T14:41:48.300 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:48 smithi031 ceph-mon[16336]: 1.e scrub starts 2024-08-19T14:41:48.300 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:48 smithi031 ceph-mon[16336]: 1.e scrub ok 2024-08-19T14:41:48.300 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:48 smithi031 ceph-mon[16336]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-08-19T14:41:48.300 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:48 smithi031 ceph-mon[16336]: 1.11 deep-scrub starts 2024-08-19T14:41:48.300 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:48 smithi031 ceph-mon[16336]: 1.11 deep-scrub ok 2024-08-19T14:41:48.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:48.301 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:41:48.402 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:41:48.430 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:48.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:48.431 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:41:48.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:48 smithi032 ceph-mon[20004]: pgmap v1822: 57 pgs: 1 active+clean+scrubbing+deep, 2 creating+peering, 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 52 active+clean; 6.8 GiB data, 29 GiB used, 775 GiB / 805 GiB avail; 1107/333432 objects misplaced (0.332%); 21 MiB/s, 331 objects/s recovering 2024-08-19T14:41:48.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:48 smithi032 ceph-mon[20004]: 8.18 scrub ok 2024-08-19T14:41:48.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:48 smithi032 ceph-mon[20004]: 8.6 scrub starts 2024-08-19T14:41:48.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:48 smithi032 ceph-mon[20004]: 1.e scrub starts 2024-08-19T14:41:48.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:48 smithi032 ceph-mon[20004]: 1.e scrub ok 2024-08-19T14:41:48.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:48 smithi032 ceph-mon[20004]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-08-19T14:41:48.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:48 smithi032 ceph-mon[20004]: 1.11 deep-scrub starts 2024-08-19T14:41:48.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:48 smithi032 ceph-mon[20004]: 1.11 deep-scrub ok 2024-08-19T14:41:48.531 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:41:48.565 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:48.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:48.566 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:41:48.667 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:41:48.697 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:48.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:48.698 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:41:48.799 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:41:48.830 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:48.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:48.832 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:41:48.933 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:41:48.961 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:48.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:48.962 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:41:49.056 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:49 smithi031 ceph-mon[16336]: 8.6 scrub ok 2024-08-19T14:41:49.056 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:49 smithi031 ceph-mon[16336]: 8.16 scrub starts 2024-08-19T14:41:49.056 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:49 smithi031 ceph-mon[16336]: 8.e scrub ok 2024-08-19T14:41:49.063 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:41:49.098 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:49.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:49.099 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:41:49.200 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:41:49.234 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:49.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:49.235 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:41:49.336 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:41:49.358 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:49.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:49.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:41:49.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:49 smithi031 ceph-mon[16336]: 8.16 scrub ok 2024-08-19T14:41:49.460 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:41:49.491 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:49.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:49.492 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:41:49.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:49 smithi032 ceph-mon[20004]: 8.6 scrub ok 2024-08-19T14:41:49.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:49 smithi032 ceph-mon[20004]: 8.16 scrub starts 2024-08-19T14:41:49.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:49 smithi032 ceph-mon[20004]: 8.e scrub ok 2024-08-19T14:41:49.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:49 smithi032 ceph-mon[20004]: 8.16 scrub ok 2024-08-19T14:41:49.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:49 smithi045 ceph-mon[20143]: 8.6 scrub ok 2024-08-19T14:41:49.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:49 smithi045 ceph-mon[20143]: 8.16 scrub starts 2024-08-19T14:41:49.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:49 smithi045 ceph-mon[20143]: 8.e scrub ok 2024-08-19T14:41:49.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:49 smithi045 ceph-mon[20143]: 8.16 scrub ok 2024-08-19T14:41:49.593 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:41:49.621 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:49.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:49.622 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:41:49.723 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:41:49.752 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:49.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:49.753 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:41:49.854 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:41:49.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:49.880 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:49.880 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:41:49.981 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:41:50.009 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:50.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:50.010 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:41:50.111 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:41:50.138 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:50.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:50.139 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:41:50.240 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:41:50.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:50.274 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:50.274 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:41:50.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:50 smithi031 ceph-mon[16336]: pgmap v1823: 57 pgs: 1 active+clean+scrubbing+deep, 2 creating+peering, 1 active+recovering+undersized+remapped, 53 active+clean; 6.7 GiB data, 29 GiB used, 775 GiB / 805 GiB avail; 170 B/s rd, 0 op/s; 740/329595 objects misplaced (0.225%) 2024-08-19T14:41:50.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:50 smithi031 ceph-mon[16336]: 1.5 deep-scrub starts 2024-08-19T14:41:50.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:50 smithi031 ceph-mon[16336]: 1.5 deep-scrub ok 2024-08-19T14:41:50.375 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:41:50.405 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:50.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:50.406 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:41:50.507 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:41:50.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:50 smithi032 ceph-mon[20004]: pgmap v1823: 57 pgs: 1 active+clean+scrubbing+deep, 2 creating+peering, 1 active+recovering+undersized+remapped, 53 active+clean; 6.7 GiB data, 29 GiB used, 775 GiB / 805 GiB avail; 170 B/s rd, 0 op/s; 740/329595 objects misplaced (0.225%) 2024-08-19T14:41:50.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:50 smithi032 ceph-mon[20004]: 1.5 deep-scrub starts 2024-08-19T14:41:50.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:50 smithi032 ceph-mon[20004]: 1.5 deep-scrub ok 2024-08-19T14:41:50.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:50.539 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:50.539 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:41:50.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:50 smithi045 ceph-mon[20143]: pgmap v1823: 57 pgs: 1 active+clean+scrubbing+deep, 2 creating+peering, 1 active+recovering+undersized+remapped, 53 active+clean; 6.7 GiB data, 29 GiB used, 775 GiB / 805 GiB avail; 170 B/s rd, 0 op/s; 740/329595 objects misplaced (0.225%) 2024-08-19T14:41:50.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:50 smithi045 ceph-mon[20143]: 1.5 deep-scrub starts 2024-08-19T14:41:50.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:50 smithi045 ceph-mon[20143]: 1.5 deep-scrub ok 2024-08-19T14:41:50.640 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:41:50.679 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:50.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:50.680 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:41:50.781 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:41:50.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:50.818 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:50.818 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:41:50.919 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:41:50.951 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:50.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:50.952 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:41:51.053 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:41:51.080 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:51.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:51.082 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:41:51.182 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:41:51.221 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:51.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:51.222 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:41:51.322 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:41:51.350 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:51.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:51.352 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:41:51.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:51 smithi031 ceph-mon[16336]: 8.9 deep-scrub ok 2024-08-19T14:41:51.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:51 smithi031 ceph-mon[16336]: 1.12 deep-scrub starts 2024-08-19T14:41:51.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:51 smithi031 ceph-mon[16336]: 1.12 deep-scrub ok 2024-08-19T14:41:51.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:51 smithi031 ceph-mon[16336]: 8.d scrub starts 2024-08-19T14:41:51.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:51 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:41:51.452 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:41:51.493 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:51.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:51.494 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:41:51.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:51 smithi032 ceph-mon[20004]: 8.9 deep-scrub ok 2024-08-19T14:41:51.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:51 smithi032 ceph-mon[20004]: 1.12 deep-scrub starts 2024-08-19T14:41:51.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:51 smithi032 ceph-mon[20004]: 1.12 deep-scrub ok 2024-08-19T14:41:51.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:51 smithi032 ceph-mon[20004]: 8.d scrub starts 2024-08-19T14:41:51.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:51 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:41:51.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:51 smithi045 ceph-mon[20143]: 8.9 deep-scrub ok 2024-08-19T14:41:51.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:51 smithi045 ceph-mon[20143]: 1.12 deep-scrub starts 2024-08-19T14:41:51.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:51 smithi045 ceph-mon[20143]: 1.12 deep-scrub ok 2024-08-19T14:41:51.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:51 smithi045 ceph-mon[20143]: 8.d scrub starts 2024-08-19T14:41:51.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:51 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:41:51.594 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:41:51.643 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:51.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:51.644 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:41:51.745 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:41:51.782 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:51.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:51.783 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:41:51.884 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:41:51.913 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:51.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:51.915 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:41:52.015 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:41:52.046 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:52.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:52.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:41:52.148 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:41:52.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:52.169 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:52.169 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:41:52.270 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:41:52.293 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:52.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:52.294 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:41:52.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:52 smithi045 ceph-mon[20143]: pgmap v1824: 57 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+remapped, 55 active+clean; 6.6 GiB data, 29 GiB used, 776 GiB / 805 GiB avail; 140 B/s rd, 0 op/s; 740/325929 objects misplaced (0.227%) 2024-08-19T14:41:52.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:52 smithi045 ceph-mon[20143]: 1.b scrub starts 2024-08-19T14:41:52.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:52 smithi045 ceph-mon[20143]: 1.b scrub ok 2024-08-19T14:41:52.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:52 smithi045 ceph-mon[20143]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-08-19T14:41:52.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:52 smithi045 ceph-mon[20143]: Cluster is now healthy 2024-08-19T14:41:52.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:52 smithi045 ceph-mon[20143]: 1.1a deep-scrub starts 2024-08-19T14:41:52.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:52 smithi045 ceph-mon[20143]: 1.1a deep-scrub ok 2024-08-19T14:41:52.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:52 smithi031 ceph-mon[16336]: pgmap v1824: 57 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+remapped, 55 active+clean; 6.6 GiB data, 29 GiB used, 776 GiB / 805 GiB avail; 140 B/s rd, 0 op/s; 740/325929 objects misplaced (0.227%) 2024-08-19T14:41:52.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:52 smithi031 ceph-mon[16336]: 1.b scrub starts 2024-08-19T14:41:52.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:52 smithi031 ceph-mon[16336]: 1.b scrub ok 2024-08-19T14:41:52.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:52 smithi031 ceph-mon[16336]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-08-19T14:41:52.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:52 smithi031 ceph-mon[16336]: Cluster is now healthy 2024-08-19T14:41:52.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:52 smithi031 ceph-mon[16336]: 1.1a deep-scrub starts 2024-08-19T14:41:52.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:52 smithi031 ceph-mon[16336]: 1.1a deep-scrub ok 2024-08-19T14:41:52.396 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:41:52.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:52.435 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:52.435 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:41:52.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:52 smithi032 ceph-mon[20004]: pgmap v1824: 57 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+remapped, 55 active+clean; 6.6 GiB data, 29 GiB used, 776 GiB / 805 GiB avail; 140 B/s rd, 0 op/s; 740/325929 objects misplaced (0.227%) 2024-08-19T14:41:52.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:52 smithi032 ceph-mon[20004]: 1.b scrub starts 2024-08-19T14:41:52.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:52 smithi032 ceph-mon[20004]: 1.b scrub ok 2024-08-19T14:41:52.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:52 smithi032 ceph-mon[20004]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-08-19T14:41:52.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:52 smithi032 ceph-mon[20004]: Cluster is now healthy 2024-08-19T14:41:52.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:52 smithi032 ceph-mon[20004]: 1.1a deep-scrub starts 2024-08-19T14:41:52.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:52 smithi032 ceph-mon[20004]: 1.1a deep-scrub ok 2024-08-19T14:41:52.536 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:41:52.569 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:52.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:52.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:41:52.671 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:41:52.699 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:52.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:52.700 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:41:52.801 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:41:52.825 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:52.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:52.827 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:41:52.928 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:41:52.964 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:52.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:52.965 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:41:53.065 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:41:53.100 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:53.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:53.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:41:53.202 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:41:53.224 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:53.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:53.225 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:41:53.326 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:41:53.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:53.347 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:53.348 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:41:53.449 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:41:53.470 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:53.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:53.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:41:53.572 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:41:53.593 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:53.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:53.594 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:41:53.679 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:53 smithi032 ceph-mon[20004]: pgmap v1825: 57 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 55 active+clean; 6.3 GiB data, 28 GiB used, 776 GiB / 805 GiB avail; 123 B/s rd, 0 op/s; 200/309771 objects misplaced (0.065%) 2024-08-19T14:41:53.680 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:53 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:41:53.680 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:53 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:41:53.680 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:53 smithi032 ceph-mon[20004]: osdmap e767: 12 total, 12 up, 9 in 2024-08-19T14:41:53.680 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:53 smithi032 ceph-mon[20004]: Health check failed: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-08-19T14:41:53.695 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:41:53.719 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:53.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:53.720 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:41:53.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:53 smithi045 ceph-mon[20143]: pgmap v1825: 57 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 55 active+clean; 6.3 GiB data, 28 GiB used, 776 GiB / 805 GiB avail; 123 B/s rd, 0 op/s; 200/309771 objects misplaced (0.065%) 2024-08-19T14:41:53.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:53 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:41:53.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:53 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:41:53.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:53 smithi045 ceph-mon[20143]: osdmap e767: 12 total, 12 up, 9 in 2024-08-19T14:41:53.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:53 smithi045 ceph-mon[20143]: Health check failed: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-08-19T14:41:53.821 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:41:53.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:53.846 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:53.846 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:41:53.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:53 smithi031 ceph-mon[16336]: pgmap v1825: 57 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 55 active+clean; 6.3 GiB data, 28 GiB used, 776 GiB / 805 GiB avail; 123 B/s rd, 0 op/s; 200/309771 objects misplaced (0.065%) 2024-08-19T14:41:53.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:53 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:41:53.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:53 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:41:53.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:53 smithi031 ceph-mon[16336]: osdmap e767: 12 total, 12 up, 9 in 2024-08-19T14:41:53.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:53 smithi031 ceph-mon[16336]: Health check failed: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-08-19T14:41:53.947 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:41:53.971 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:53.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:53.972 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:41:54.073 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:41:54.099 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:54.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:54.100 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:41:54.201 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:41:54.205 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:41:54.205 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [ 2024-08-19T14:41:54.205 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:41:54.206 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.26911.0:115572 8.7 e765/731 8:e73ae20b:::benchmark_data_smithi045_137103_object3108:head v 765'5550, mlcod=765'5548)", 2024-08-19T14:41:54.206 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:41:46.001489+0000", 2024-08-19T14:41:54.206 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.0061600079999999998, 2024-08-19T14:41:54.206 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0061884749999999997, 2024-08-19T14:41:54.206 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:41:54.206 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:41:54.206 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-08-19T14:41:54.206 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:41:54.206 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:41:54.206 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:41:54.206 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:46.001489+0000", 2024-08-19T14:41:54.206 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:41:54.207 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:41:54.207 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:41:54.207 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:41:54.207 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:46.001489+0000", 2024-08-19T14:41:54.207 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:41:54.207 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:41:54.207 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:41:54.207 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:41:54.207 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:46.001490+0000", 2024-08-19T14:41:54.207 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.3399999999999998e-07 2024-08-19T14:41:54.207 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:41:54.207 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:41:54.207 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:41:54.208 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:46.001496+0000", 2024-08-19T14:41:54.208 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.908e-06 2024-08-19T14:41:54.208 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:41:54.208 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:41:54.208 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:41:54.208 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:46.001509+0000", 2024-08-19T14:41:54.208 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3499000000000001e-05 2024-08-19T14:41:54.208 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:41:54.208 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:41:54.208 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:41:54.209 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:46.001516+0000", 2024-08-19T14:41:54.209 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.7499999999999997e-06 2024-08-19T14:41:54.209 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:41:54.209 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:41:54.209 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:41:54.209 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:46.001656+0000", 2024-08-19T14:41:54.209 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000140168 2024-08-19T14:41:54.210 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:41:54.210 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:41:54.210 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:41:54.210 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:46.001674+0000", 2024-08-19T14:41:54.210 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.7703e-05 2024-08-19T14:41:54.210 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:41:54.210 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:41:54.210 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:41:54.210 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:41:54.210 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:41:54.211 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.26911.0:115575 8.a e765/717 8:58e653fd:::benchmark_data_smithi045_137103_object3111:head v 765'7157, mlcod=765'7155)", 2024-08-19T14:41:54.211 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:41:46.004532+0000", 2024-08-19T14:41:54.211 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.0031175590000000002, 2024-08-19T14:41:54.211 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0032110210000000001, 2024-08-19T14:41:54.211 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:41:54.211 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:41:54.211 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-08-19T14:41:54.211 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:41:54.211 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:41:54.211 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:41:54.211 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:46.004532+0000", 2024-08-19T14:41:54.211 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:41:54.211 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:41:54.212 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:41:54.212 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:41:54.212 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:46.004532+0000", 2024-08-19T14:41:54.212 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:41:54.212 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:41:54.212 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:41:54.212 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:41:54.212 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:46.004532+0000", 2024-08-19T14:41:54.212 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.5600000000000005e-07 2024-08-19T14:41:54.212 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:41:54.212 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:41:54.212 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:41:54.212 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:46.004538+0000", 2024-08-19T14:41:54.213 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.7440000000000004e-06 2024-08-19T14:41:54.213 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:41:54.213 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:41:54.213 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:41:54.213 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:46.004553+0000", 2024-08-19T14:41:54.213 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4538999999999999e-05 2024-08-19T14:41:54.213 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:41:54.213 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:41:54.213 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:41:54.213 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:46.004559+0000", 2024-08-19T14:41:54.213 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.4099999999999996e-06 2024-08-19T14:41:54.213 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:41:54.213 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:41:54.213 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:41:54.214 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:46.004621+0000", 2024-08-19T14:41:54.214 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.2343999999999998e-05 2024-08-19T14:41:54.214 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:41:54.214 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:41:54.214 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:41:54.214 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:46.004644+0000", 2024-08-19T14:41:54.214 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.2365e-05 2024-08-19T14:41:54.214 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:41:54.214 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:41:54.214 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:41:54.214 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:41:54.214 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:41:54.214 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.26911.0:115577 8.f e765/717 8:ffdbc9ad:::benchmark_data_smithi045_137103_object3113:head v 765'7232, mlcod=765'7230)", 2024-08-19T14:41:54.215 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:41:46.004684+0000", 2024-08-19T14:41:54.215 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.0029653190000000001, 2024-08-19T14:41:54.215 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0031152150000000002, 2024-08-19T14:41:54.215 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:41:54.215 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:41:54.215 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-08-19T14:41:54.215 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:41:54.215 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:41:54.215 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:41:54.215 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:46.004684+0000", 2024-08-19T14:41:54.215 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:41:54.215 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:41:54.215 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:41:54.215 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:41:54.216 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:46.004684+0000", 2024-08-19T14:41:54.216 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:41:54.216 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:41:54.216 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:41:54.216 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:41:54.216 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:46.004684+0000", 2024-08-19T14:41:54.216 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.82e-07 2024-08-19T14:41:54.216 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:41:54.216 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:41:54.216 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:41:54.216 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:46.004689+0000", 2024-08-19T14:41:54.216 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7249999999999997e-06 2024-08-19T14:41:54.216 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:41:54.216 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:41:54.217 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:41:54.217 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:46.004699+0000", 2024-08-19T14:41:54.217 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.6409999999999997e-06 2024-08-19T14:41:54.217 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:41:54.217 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:41:54.217 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:41:54.217 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:46.004705+0000", 2024-08-19T14:41:54.217 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.3600000000000001e-06 2024-08-19T14:41:54.217 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:41:54.217 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:41:54.217 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:41:54.217 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:46.005258+0000", 2024-08-19T14:41:54.217 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00055307100000000003 2024-08-19T14:41:54.217 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:41:54.217 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:41:54.218 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:41:54.218 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:46.005278+0000", 2024-08-19T14:41:54.218 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.9541000000000001e-05 2024-08-19T14:41:54.218 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:41:54.218 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:41:54.218 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:41:54.218 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:41:54.218 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-08-19T14:41:54.218 INFO:teuthology.orchestra.run.smithi031.stdout: "num_ops": 3 2024-08-19T14:41:54.218 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:41:54.236 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:54.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:54.238 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:41:54.339 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:41:54.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:54.372 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:54.372 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:41:54.473 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:41:54.500 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:54.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:54.501 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:41:54.602 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:41:54.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:54 smithi031 ceph-mon[16336]: 8.12 scrub starts 2024-08-19T14:41:54.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:54 smithi031 ceph-mon[16336]: 8.d scrub ok 2024-08-19T14:41:54.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:54 smithi031 ceph-mon[16336]: 8.12 scrub ok 2024-08-19T14:41:54.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:54 smithi031 ceph-mon[16336]: osdmap e768: 12 total, 12 up, 9 in 2024-08-19T14:41:54.631 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:54.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:54.633 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:41:54.733 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:41:54.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:54 smithi032 ceph-mon[20004]: 8.12 scrub starts 2024-08-19T14:41:54.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:54 smithi032 ceph-mon[20004]: 8.d scrub ok 2024-08-19T14:41:54.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:54 smithi032 ceph-mon[20004]: 8.12 scrub ok 2024-08-19T14:41:54.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:54 smithi032 ceph-mon[20004]: osdmap e768: 12 total, 12 up, 9 in 2024-08-19T14:41:54.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:54.771 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:54.771 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:41:54.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:54 smithi045 ceph-mon[20143]: 8.12 scrub starts 2024-08-19T14:41:54.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:54 smithi045 ceph-mon[20143]: 8.d scrub ok 2024-08-19T14:41:54.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:54 smithi045 ceph-mon[20143]: 8.12 scrub ok 2024-08-19T14:41:54.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:54 smithi045 ceph-mon[20143]: osdmap e768: 12 total, 12 up, 9 in 2024-08-19T14:41:54.872 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:41:54.897 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:54.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:54.899 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:41:55.000 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:41:55.029 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:55.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:55.030 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:41:55.132 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:41:55.161 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:55.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:55.162 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:41:55.263 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:41:55.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:55.295 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:55.296 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:41:55.396 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:41:55.423 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:55.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:55.424 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:41:55.526 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:41:55.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:55.559 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:55.559 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:41:55.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:55 smithi031 ceph-mon[16336]: 1.18 scrub starts 2024-08-19T14:41:55.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:55 smithi031 ceph-mon[16336]: 1.18 scrub ok 2024-08-19T14:41:55.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:55 smithi031 ceph-mon[16336]: 8.f deep-scrub starts 2024-08-19T14:41:55.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:55 smithi031 ceph-mon[16336]: pgmap v1828: 57 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 55 active+clean; 6.2 GiB data, 28 GiB used, 777 GiB / 805 GiB avail; 127 B/s rd, 0 op/s; 200/305370 objects misplaced (0.065%) 2024-08-19T14:41:55.660 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:41:55.690 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:55.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:55.691 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:41:55.755 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:55 smithi045 ceph-mon[20143]: 1.18 scrub starts 2024-08-19T14:41:55.755 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:55 smithi045 ceph-mon[20143]: 1.18 scrub ok 2024-08-19T14:41:55.755 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:55 smithi045 ceph-mon[20143]: 8.f deep-scrub starts 2024-08-19T14:41:55.755 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:55 smithi045 ceph-mon[20143]: pgmap v1828: 57 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 55 active+clean; 6.2 GiB data, 28 GiB used, 777 GiB / 805 GiB avail; 127 B/s rd, 0 op/s; 200/305370 objects misplaced (0.065%) 2024-08-19T14:41:55.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:55 smithi032 ceph-mon[20004]: 1.18 scrub starts 2024-08-19T14:41:55.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:55 smithi032 ceph-mon[20004]: 1.18 scrub ok 2024-08-19T14:41:55.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:55 smithi032 ceph-mon[20004]: 8.f deep-scrub starts 2024-08-19T14:41:55.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:55 smithi032 ceph-mon[20004]: pgmap v1828: 57 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 55 active+clean; 6.2 GiB data, 28 GiB used, 777 GiB / 805 GiB avail; 127 B/s rd, 0 op/s; 200/305370 objects misplaced (0.065%) 2024-08-19T14:41:55.793 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:41:55.820 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:55.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:55.821 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:41:55.922 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:41:55.956 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:55.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:55.958 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:41:56.059 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:41:56.091 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:56.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:56.092 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:41:56.193 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:41:56.228 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:56.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:56.229 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:41:56.330 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:41:56.362 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:56.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:56.363 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:41:56.464 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:41:56.500 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:56.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:56.501 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:41:56.602 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:41:56.634 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:56.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:56.635 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:41:56.736 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:41:56.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:56 smithi032 ceph-mon[20004]: 8.10 scrub starts 2024-08-19T14:41:56.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:56 smithi032 ceph-mon[20004]: 8.10 scrub ok 2024-08-19T14:41:56.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:56 smithi032 ceph-mon[20004]: 1.6 scrub starts 2024-08-19T14:41:56.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:56 smithi032 ceph-mon[20004]: 1.6 scrub ok 2024-08-19T14:41:56.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:56 smithi032 ceph-mon[20004]: 1.a scrub starts 2024-08-19T14:41:56.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:56 smithi032 ceph-mon[20004]: 1.a scrub ok 2024-08-19T14:41:56.762 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:56.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:56.763 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:41:56.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:56 smithi045 ceph-mon[20143]: 8.10 scrub starts 2024-08-19T14:41:56.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:56 smithi045 ceph-mon[20143]: 8.10 scrub ok 2024-08-19T14:41:56.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:56 smithi045 ceph-mon[20143]: 1.6 scrub starts 2024-08-19T14:41:56.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:56 smithi045 ceph-mon[20143]: 1.6 scrub ok 2024-08-19T14:41:56.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:56 smithi045 ceph-mon[20143]: 1.a scrub starts 2024-08-19T14:41:56.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:56 smithi045 ceph-mon[20143]: 1.a scrub ok 2024-08-19T14:41:56.864 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:41:56.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:56 smithi031 ceph-mon[16336]: 8.10 scrub starts 2024-08-19T14:41:56.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:56 smithi031 ceph-mon[16336]: 8.10 scrub ok 2024-08-19T14:41:56.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:56 smithi031 ceph-mon[16336]: 1.6 scrub starts 2024-08-19T14:41:56.868 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:56 smithi031 ceph-mon[16336]: 1.6 scrub ok 2024-08-19T14:41:56.868 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:56 smithi031 ceph-mon[16336]: 1.a scrub starts 2024-08-19T14:41:56.868 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:56 smithi031 ceph-mon[16336]: 1.a scrub ok 2024-08-19T14:41:56.887 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:56.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:56.888 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:41:56.989 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:41:57.014 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:57.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:57.015 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:41:57.116 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:41:57.146 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:57.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:57.148 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:41:57.249 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:41:57.286 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:57.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:57.287 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:41:57.388 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:41:57.418 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:57.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:57.419 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:41:57.520 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:41:57.550 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:57.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:57.551 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:41:57.652 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:41:57.683 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:57.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:57.684 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:41:57.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:57 smithi032 ceph-mon[20004]: 8.17 deep-scrub starts 2024-08-19T14:41:57.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:57 smithi032 ceph-mon[20004]: 1.13 scrub ok 2024-08-19T14:41:57.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:57 smithi032 ceph-mon[20004]: 1.4 deep-scrub starts 2024-08-19T14:41:57.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:57 smithi032 ceph-mon[20004]: 1.4 deep-scrub ok 2024-08-19T14:41:57.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:57 smithi032 ceph-mon[20004]: pgmap v1829: 57 pgs: 1 active+clean+scrubbing, 56 active+clean; 6.1 GiB data, 27 GiB used, 777 GiB / 805 GiB avail 2024-08-19T14:41:57.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:57 smithi032 ceph-mon[20004]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-08-19T14:41:57.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:57 smithi032 ceph-mon[20004]: Cluster is now healthy 2024-08-19T14:41:57.784 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:41:57.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:57 smithi045 ceph-mon[20143]: 8.17 deep-scrub starts 2024-08-19T14:41:57.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:57 smithi045 ceph-mon[20143]: 1.13 scrub ok 2024-08-19T14:41:57.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:57 smithi045 ceph-mon[20143]: 1.4 deep-scrub starts 2024-08-19T14:41:57.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:57 smithi045 ceph-mon[20143]: 1.4 deep-scrub ok 2024-08-19T14:41:57.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:57 smithi045 ceph-mon[20143]: pgmap v1829: 57 pgs: 1 active+clean+scrubbing, 56 active+clean; 6.1 GiB data, 27 GiB used, 777 GiB / 805 GiB avail 2024-08-19T14:41:57.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:57 smithi045 ceph-mon[20143]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-08-19T14:41:57.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:57 smithi045 ceph-mon[20143]: Cluster is now healthy 2024-08-19T14:41:57.810 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:57.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:57.811 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:41:57.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:57 smithi031 ceph-mon[16336]: 8.17 deep-scrub starts 2024-08-19T14:41:57.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:57 smithi031 ceph-mon[16336]: 1.13 scrub ok 2024-08-19T14:41:57.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:57 smithi031 ceph-mon[16336]: 1.4 deep-scrub starts 2024-08-19T14:41:57.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:57 smithi031 ceph-mon[16336]: 1.4 deep-scrub ok 2024-08-19T14:41:57.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:57 smithi031 ceph-mon[16336]: pgmap v1829: 57 pgs: 1 active+clean+scrubbing, 56 active+clean; 6.1 GiB data, 27 GiB used, 777 GiB / 805 GiB avail 2024-08-19T14:41:57.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:57 smithi031 ceph-mon[16336]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-08-19T14:41:57.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:57 smithi031 ceph-mon[16336]: Cluster is now healthy 2024-08-19T14:41:57.913 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:41:57.938 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:57.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:57.939 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:41:58.040 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:41:58.068 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:58.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:58.069 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:41:58.170 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:41:58.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:58.227 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:58.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:41:58.327 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:41:58.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:58.364 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:58.365 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:41:58.466 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:41:58.496 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:58.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:58.497 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:41:58.598 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:41:58.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:58.633 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:58.634 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:41:58.734 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:41:58.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:58 smithi032 ceph-mon[20004]: 8.11 scrub starts 2024-08-19T14:41:58.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:58 smithi032 ceph-mon[20004]: 8.11 scrub ok 2024-08-19T14:41:58.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:58 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2218076983' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:41:58.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:58.766 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:58.767 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:41:58.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:58 smithi045 ceph-mon[20143]: 8.11 scrub starts 2024-08-19T14:41:58.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:58 smithi045 ceph-mon[20143]: 8.11 scrub ok 2024-08-19T14:41:58.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:58 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2218076983' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:41:58.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:58 smithi031 ceph-mon[16336]: 8.11 scrub starts 2024-08-19T14:41:58.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:58 smithi031 ceph-mon[16336]: 8.11 scrub ok 2024-08-19T14:41:58.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:58 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2218076983' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:41:58.867 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:41:58.899 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:58.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:58.900 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:41:59.001 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:41:59.027 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:59.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:59.028 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:41:59.042 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:41:59.042 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:41:59.042 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:41:59.042 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:41:59.042 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:41:59.043 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:41:59.043 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:41:59.043 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:41:59.043 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:41:59.043 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:41:59.043 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:41:59.044 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:41:59.044 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:41:59.044 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:41:59.044 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:41:59.044 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:41:59.044 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:41:59.044 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:41:59.045 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:41:59.045 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:41:59.045 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:41:59.045 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:41:59.045 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:41:59.045 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:41:59.129 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:41:59.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:59.151 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:59.151 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:41:59.252 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:41:59.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:59.279 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:59.279 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:41:59.380 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:41:59.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:59.407 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:59.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:41:59.508 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:41:59.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:59.542 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:59.543 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:41:59.644 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:41:59.667 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:41:59.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:59.668 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:41:59.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:59 smithi032 ceph-mon[20004]: 8.15 scrub starts 2024-08-19T14:41:59.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:59 smithi032 ceph-mon[20004]: 8.15 scrub ok 2024-08-19T14:41:59.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:41:59 smithi032 ceph-mon[20004]: pgmap v1830: 57 pgs: 1 active+clean+scrubbing+deep, 56 active+clean; 6.0 GiB data, 27 GiB used, 778 GiB / 805 GiB avail 2024-08-19T14:41:59.768 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:41:59.796 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:41:59.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:59 smithi045 ceph-mon[20143]: 8.15 scrub starts 2024-08-19T14:41:59.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:59 smithi045 ceph-mon[20143]: 8.15 scrub ok 2024-08-19T14:41:59.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:41:59 smithi045 ceph-mon[20143]: pgmap v1830: 57 pgs: 1 active+clean+scrubbing+deep, 56 active+clean; 6.0 GiB data, 27 GiB used, 778 GiB / 805 GiB avail 2024-08-19T14:41:59.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:59.797 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:41:59.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:59 smithi031 ceph-mon[16336]: 8.15 scrub starts 2024-08-19T14:41:59.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:59 smithi031 ceph-mon[16336]: 8.15 scrub ok 2024-08-19T14:41:59.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:41:59 smithi031 ceph-mon[16336]: pgmap v1830: 57 pgs: 1 active+clean+scrubbing+deep, 56 active+clean; 6.0 GiB data, 27 GiB used, 778 GiB / 805 GiB avail 2024-08-19T14:41:59.898 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:41:59.925 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:41:59.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:41:59.926 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:42:00.027 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:42:00.050 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:00.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:00.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:42:00.152 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:42:00.178 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:00.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:00.180 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:42:00.281 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:42:00.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:00.309 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:00.309 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:42:00.411 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:42:00.449 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:00.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:00.450 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:42:00.551 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:42:00.583 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:00.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:00.584 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:42:00.685 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:42:00.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:00.715 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:00.715 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:42:00.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:00 smithi032 ceph-mon[20004]: 1.1e scrub starts 2024-08-19T14:42:00.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:00 smithi032 ceph-mon[20004]: 1.1e scrub ok 2024-08-19T14:42:00.768 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:00 smithi045 ceph-mon[20143]: 1.1e scrub starts 2024-08-19T14:42:00.769 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:00 smithi045 ceph-mon[20143]: 1.1e scrub ok 2024-08-19T14:42:00.816 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:42:00.840 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:00.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:00.842 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:42:00.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:00 smithi031 ceph-mon[16336]: 1.1e scrub starts 2024-08-19T14:42:00.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:00 smithi031 ceph-mon[16336]: 1.1e scrub ok 2024-08-19T14:42:00.943 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:42:00.971 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:00.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:00.972 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:42:01.073 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:42:01.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:01.122 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:01.122 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:42:01.223 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:42:01.255 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:01.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:01.256 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:42:01.357 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:42:01.392 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:01.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:01.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:42:01.494 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:42:01.517 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:01.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:01.518 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:42:01.619 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:42:01.646 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:01.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:01.648 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:42:01.749 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:42:01.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:01 smithi032 ceph-mon[20004]: pgmap v1831: 57 pgs: 1 active+clean+scrubbing+deep, 56 active+clean; 5.9 GiB data, 26 GiB used, 778 GiB / 805 GiB avail 2024-08-19T14:42:01.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:01 smithi032 ceph-mon[20004]: 1.9 scrub starts 2024-08-19T14:42:01.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:01 smithi032 ceph-mon[20004]: 1.9 scrub ok 2024-08-19T14:42:01.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:01 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:42:01.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:01 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:42:01.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:01 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:42:01.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:01 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:42:01.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:01 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:42:01.778 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:01.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:01.779 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:42:01.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:01 smithi045 ceph-mon[20143]: pgmap v1831: 57 pgs: 1 active+clean+scrubbing+deep, 56 active+clean; 5.9 GiB data, 26 GiB used, 778 GiB / 805 GiB avail 2024-08-19T14:42:01.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:01 smithi045 ceph-mon[20143]: 1.9 scrub starts 2024-08-19T14:42:01.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:01 smithi045 ceph-mon[20143]: 1.9 scrub ok 2024-08-19T14:42:01.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:01 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:42:01.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:01 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:42:01.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:01 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:42:01.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:01 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:42:01.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:01 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:42:01.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:01 smithi031 ceph-mon[16336]: pgmap v1831: 57 pgs: 1 active+clean+scrubbing+deep, 56 active+clean; 5.9 GiB data, 26 GiB used, 778 GiB / 805 GiB avail 2024-08-19T14:42:01.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:01 smithi031 ceph-mon[16336]: 1.9 scrub starts 2024-08-19T14:42:01.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:01 smithi031 ceph-mon[16336]: 1.9 scrub ok 2024-08-19T14:42:01.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:01 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:42:01.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:01 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:42:01.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:01 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:42:01.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:01 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:42:01.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:01 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:42:01.879 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:42:01.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:01.917 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:01.917 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:42:02.018 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:42:02.041 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:02.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:02.043 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:42:02.146 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:42:02.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:02.177 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:02.178 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:42:02.279 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:42:02.308 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:02.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:02.309 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:42:02.410 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:42:02.438 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:02.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:02.439 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:42:02.540 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:42:02.563 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:02.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:02.564 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:42:02.665 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:42:02.692 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:02.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:02.693 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:42:02.794 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:42:02.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:02.830 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:02.831 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:42:02.931 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:42:02.965 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:02.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:02.966 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:42:03.067 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:42:03.092 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:03.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:03.093 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:42:03.194 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:42:03.228 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:03.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:03.229 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:42:03.330 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:42:03.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:03.363 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:03.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:42:03.464 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:42:03.493 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:03.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:03.494 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:42:03.595 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:42:03.625 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:03.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:03.626 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:42:03.727 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:42:03.753 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:03.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:03.754 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:42:03.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:03 smithi032 ceph-mon[20004]: pgmap v1832: 57 pgs: 2 active+clean+scrubbing+deep, 55 active+clean; 5.6 GiB data, 26 GiB used, 779 GiB / 805 GiB avail 2024-08-19T14:42:03.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:03 smithi032 ceph-mon[20004]: 8.7 scrub starts 2024-08-19T14:42:03.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {auth=36,config=2,health=12,kv=50} crc {auth=3222154698,config=3457680788,health=3897446751,kv=2711045919}) 2024-08-19T14:42:03.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {kv=42,logm=58} crc {kv=958002028,logm=581608254}) 2024-08-19T14:42:03.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2457976474}) 2024-08-19T14:42:03.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3729210631}) 2024-08-19T14:42:03.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2230972430}) 2024-08-19T14:42:03.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4188039531}) 2024-08-19T14:42:03.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=479611951}) 2024-08-19T14:42:03.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3479498680}) 2024-08-19T14:42:03.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1247974059}) 2024-08-19T14:42:03.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3704473938}) 2024-08-19T14:42:03.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3546966034}) 2024-08-19T14:42:03.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=829432589}) 2024-08-19T14:42:03.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3030308662}) 2024-08-19T14:42:03.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=570465646}) 2024-08-19T14:42:03.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=178628098}) 2024-08-19T14:42:03.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3081414247}) 2024-08-19T14:42:03.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1170711257}) 2024-08-19T14:42:03.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2505545875}) 2024-08-19T14:42:03.760 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=72911326}) 2024-08-19T14:42:03.760 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3199878782}) 2024-08-19T14:42:03.760 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2646923020}) 2024-08-19T14:42:03.760 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3001182534}) 2024-08-19T14:42:03.760 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2190849151}) 2024-08-19T14:42:03.760 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2165405415}) 2024-08-19T14:42:03.760 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=873796896}) 2024-08-19T14:42:03.760 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2589210928}) 2024-08-19T14:42:03.760 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1763802486}) 2024-08-19T14:42:03.760 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=823901828}) 2024-08-19T14:42:03.760 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2030895044}) 2024-08-19T14:42:03.760 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1929075053}) 2024-08-19T14:42:03.761 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1781383395}) 2024-08-19T14:42:03.761 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=105868966}) 2024-08-19T14:42:03.761 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2327610751}) 2024-08-19T14:42:03.761 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2228951482}) 2024-08-19T14:42:03.761 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2865628179}) 2024-08-19T14:42:03.761 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1254641404}) 2024-08-19T14:42:03.761 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2495793379}) 2024-08-19T14:42:03.762 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=317524961}) 2024-08-19T14:42:03.762 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3168233751}) 2024-08-19T14:42:03.762 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=926848755}) 2024-08-19T14:42:03.762 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=397769427}) 2024-08-19T14:42:03.762 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1999511318}) 2024-08-19T14:42:03.762 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2741024824}) 2024-08-19T14:42:03.762 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2202145503}) 2024-08-19T14:42:03.762 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=914212547}) 2024-08-19T14:42:03.762 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2195185366}) 2024-08-19T14:42:03.762 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3411692791}) 2024-08-19T14:42:03.762 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2237803542}) 2024-08-19T14:42:03.762 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1697004363}) 2024-08-19T14:42:03.763 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2214339475}) 2024-08-19T14:42:03.763 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1548075756}) 2024-08-19T14:42:03.763 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1612270920}) 2024-08-19T14:42:03.763 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1156895575}) 2024-08-19T14:42:03.763 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3972633554}) 2024-08-19T14:42:03.763 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1553639209}) 2024-08-19T14:42:03.763 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=7250276}) 2024-08-19T14:42:03.763 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1437386178}) 2024-08-19T14:42:03.763 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2927545827}) 2024-08-19T14:42:03.763 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=453369477}) 2024-08-19T14:42:03.763 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=186316600}) 2024-08-19T14:42:03.763 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2311722870}) 2024-08-19T14:42:03.764 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2396447752}) 2024-08-19T14:42:03.764 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4238581967}) 2024-08-19T14:42:03.764 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3685228488}) 2024-08-19T14:42:03.764 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1067810075}) 2024-08-19T14:42:03.764 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3141699418}) 2024-08-19T14:42:03.764 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2979846439}) 2024-08-19T14:42:03.764 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2346258548}) 2024-08-19T14:42:03.764 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2929509397}) 2024-08-19T14:42:03.764 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=528266920}) 2024-08-19T14:42:03.765 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=499400314}) 2024-08-19T14:42:03.765 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2720366624}) 2024-08-19T14:42:03.765 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3465661353}) 2024-08-19T14:42:03.765 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=51,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=25} crc {logm=3095213747,mdsmap=2474562478,mgr=2924679688,mgr_command_descs=1300230762,mgr_metadata=482233419,mgrstat=1723312609}) 2024-08-19T14:42:03.765 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=33,mon_config_key=67} crc {mgrstat=2954130288,mon_config_key=1586131024}) 2024-08-19T14:42:03.765 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1757061686}) 2024-08-19T14:42:03.765 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3392079505}) 2024-08-19T14:42:03.765 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=12,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=70} crc {mon_config_key=2125561575,monmap=3193798142,osd_metadata=289051117,osd_pg_creating=2806808626,osdmap=3201246597}) 2024-08-19T14:42:03.765 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=913845312}) 2024-08-19T14:42:03.766 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3619438510}) 2024-08-19T14:42:03.766 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3140793736}) 2024-08-19T14:42:03.766 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3508553138}) 2024-08-19T14:42:03.766 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1698835763}) 2024-08-19T14:42:03.766 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=4194635033}) 2024-08-19T14:42:03.766 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1263353133}) 2024-08-19T14:42:03.766 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2830649815}) 2024-08-19T14:42:03.766 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2419199357}) 2024-08-19T14:42:03.766 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=76259912}) 2024-08-19T14:42:03.766 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=247711485}) 2024-08-19T14:42:03.766 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3695095570}) 2024-08-19T14:42:03.766 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3545774687}) 2024-08-19T14:42:03.767 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=162013159}) 2024-08-19T14:42:03.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:03 smithi045 ceph-mon[20143]: pgmap v1832: 57 pgs: 2 active+clean+scrubbing+deep, 55 active+clean; 5.6 GiB data, 26 GiB used, 779 GiB / 805 GiB avail 2024-08-19T14:42:03.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:03 smithi045 ceph-mon[20143]: 8.7 scrub starts 2024-08-19T14:42:03.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {auth=36,config=2,health=12,kv=50} crc {auth=3222154698,config=3457680788,health=3897446751,kv=2711045919}) 2024-08-19T14:42:03.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {kv=42,logm=58} crc {kv=958002028,logm=581608254}) 2024-08-19T14:42:03.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2457976474}) 2024-08-19T14:42:03.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3729210631}) 2024-08-19T14:42:03.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2230972430}) 2024-08-19T14:42:03.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4188039531}) 2024-08-19T14:42:03.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=479611951}) 2024-08-19T14:42:03.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3479498680}) 2024-08-19T14:42:03.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1247974059}) 2024-08-19T14:42:03.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3704473938}) 2024-08-19T14:42:03.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3546966034}) 2024-08-19T14:42:03.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=829432589}) 2024-08-19T14:42:03.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3030308662}) 2024-08-19T14:42:03.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=570465646}) 2024-08-19T14:42:03.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=178628098}) 2024-08-19T14:42:03.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3081414247}) 2024-08-19T14:42:03.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1170711257}) 2024-08-19T14:42:03.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2505545875}) 2024-08-19T14:42:03.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=72911326}) 2024-08-19T14:42:03.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3199878782}) 2024-08-19T14:42:03.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2646923020}) 2024-08-19T14:42:03.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3001182534}) 2024-08-19T14:42:03.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2190849151}) 2024-08-19T14:42:03.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2165405415}) 2024-08-19T14:42:03.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=873796896}) 2024-08-19T14:42:03.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2589210928}) 2024-08-19T14:42:03.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1763802486}) 2024-08-19T14:42:03.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=823901828}) 2024-08-19T14:42:03.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2030895044}) 2024-08-19T14:42:03.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1929075053}) 2024-08-19T14:42:03.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1781383395}) 2024-08-19T14:42:03.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=105868966}) 2024-08-19T14:42:03.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2327610751}) 2024-08-19T14:42:03.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2228951482}) 2024-08-19T14:42:03.801 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2865628179}) 2024-08-19T14:42:03.801 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1254641404}) 2024-08-19T14:42:03.801 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2495793379}) 2024-08-19T14:42:03.801 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=317524961}) 2024-08-19T14:42:03.801 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3168233751}) 2024-08-19T14:42:03.801 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=926848755}) 2024-08-19T14:42:03.801 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=397769427}) 2024-08-19T14:42:03.801 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1999511318}) 2024-08-19T14:42:03.801 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2741024824}) 2024-08-19T14:42:03.801 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2202145503}) 2024-08-19T14:42:03.801 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=914212547}) 2024-08-19T14:42:03.801 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2195185366}) 2024-08-19T14:42:03.802 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3411692791}) 2024-08-19T14:42:03.802 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2237803542}) 2024-08-19T14:42:03.802 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1697004363}) 2024-08-19T14:42:03.802 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2214339475}) 2024-08-19T14:42:03.802 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1548075756}) 2024-08-19T14:42:03.802 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1612270920}) 2024-08-19T14:42:03.802 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1156895575}) 2024-08-19T14:42:03.802 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3972633554}) 2024-08-19T14:42:03.802 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1553639209}) 2024-08-19T14:42:03.802 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=7250276}) 2024-08-19T14:42:03.802 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1437386178}) 2024-08-19T14:42:03.803 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2927545827}) 2024-08-19T14:42:03.803 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=453369477}) 2024-08-19T14:42:03.803 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=186316600}) 2024-08-19T14:42:03.803 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2311722870}) 2024-08-19T14:42:03.803 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2396447752}) 2024-08-19T14:42:03.803 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4238581967}) 2024-08-19T14:42:03.803 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3685228488}) 2024-08-19T14:42:03.803 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1067810075}) 2024-08-19T14:42:03.803 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3141699418}) 2024-08-19T14:42:03.803 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2979846439}) 2024-08-19T14:42:03.803 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2346258548}) 2024-08-19T14:42:03.803 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2929509397}) 2024-08-19T14:42:03.803 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=528266920}) 2024-08-19T14:42:03.803 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=499400314}) 2024-08-19T14:42:03.804 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2720366624}) 2024-08-19T14:42:03.804 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3465661353}) 2024-08-19T14:42:03.804 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=51,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=25} crc {logm=3095213747,mdsmap=2474562478,mgr=2924679688,mgr_command_descs=1300230762,mgr_metadata=482233419,mgrstat=1723312609}) 2024-08-19T14:42:03.804 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=33,mon_config_key=67} crc {mgrstat=2954130288,mon_config_key=1586131024}) 2024-08-19T14:42:03.804 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1757061686}) 2024-08-19T14:42:03.804 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3392079505}) 2024-08-19T14:42:03.804 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=12,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=70} crc {mon_config_key=2125561575,monmap=3193798142,osd_metadata=289051117,osd_pg_creating=2806808626,osdmap=3201246597}) 2024-08-19T14:42:03.804 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=913845312}) 2024-08-19T14:42:03.804 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3619438510}) 2024-08-19T14:42:03.804 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3140793736}) 2024-08-19T14:42:03.804 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3508553138}) 2024-08-19T14:42:03.804 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1698835763}) 2024-08-19T14:42:03.804 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=4194635033}) 2024-08-19T14:42:03.805 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1263353133}) 2024-08-19T14:42:03.805 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2830649815}) 2024-08-19T14:42:03.805 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2419199357}) 2024-08-19T14:42:03.805 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=76259912}) 2024-08-19T14:42:03.805 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=247711485}) 2024-08-19T14:42:03.805 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3695095570}) 2024-08-19T14:42:03.805 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3545774687}) 2024-08-19T14:42:03.805 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=162013159}) 2024-08-19T14:42:03.854 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:42:03.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:03 smithi031 ceph-mon[16336]: pgmap v1832: 57 pgs: 2 active+clean+scrubbing+deep, 55 active+clean; 5.6 GiB data, 26 GiB used, 779 GiB / 805 GiB avail 2024-08-19T14:42:03.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:03 smithi031 ceph-mon[16336]: 8.7 scrub starts 2024-08-19T14:42:03.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {auth=36,config=2,health=12,kv=50} crc {auth=3222154698,config=3457680788,health=3897446751,kv=2711045919}) 2024-08-19T14:42:03.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {kv=42,logm=58} crc {kv=958002028,logm=581608254}) 2024-08-19T14:42:03.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2457976474}) 2024-08-19T14:42:03.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3729210631}) 2024-08-19T14:42:03.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2230972430}) 2024-08-19T14:42:03.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4188039531}) 2024-08-19T14:42:03.868 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=479611951}) 2024-08-19T14:42:03.868 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3479498680}) 2024-08-19T14:42:03.868 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1247974059}) 2024-08-19T14:42:03.868 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3704473938}) 2024-08-19T14:42:03.868 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3546966034}) 2024-08-19T14:42:03.868 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=829432589}) 2024-08-19T14:42:03.868 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3030308662}) 2024-08-19T14:42:03.868 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=570465646}) 2024-08-19T14:42:03.868 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=178628098}) 2024-08-19T14:42:03.868 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3081414247}) 2024-08-19T14:42:03.868 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1170711257}) 2024-08-19T14:42:03.868 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2505545875}) 2024-08-19T14:42:03.868 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=72911326}) 2024-08-19T14:42:03.868 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3199878782}) 2024-08-19T14:42:03.869 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2646923020}) 2024-08-19T14:42:03.869 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3001182534}) 2024-08-19T14:42:03.869 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2190849151}) 2024-08-19T14:42:03.869 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2165405415}) 2024-08-19T14:42:03.869 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=873796896}) 2024-08-19T14:42:03.869 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2589210928}) 2024-08-19T14:42:03.869 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1763802486}) 2024-08-19T14:42:03.869 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=823901828}) 2024-08-19T14:42:03.869 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2030895044}) 2024-08-19T14:42:03.869 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1929075053}) 2024-08-19T14:42:03.869 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1781383395}) 2024-08-19T14:42:03.869 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=105868966}) 2024-08-19T14:42:03.869 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2327610751}) 2024-08-19T14:42:03.869 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2228951482}) 2024-08-19T14:42:03.870 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2865628179}) 2024-08-19T14:42:03.870 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1254641404}) 2024-08-19T14:42:03.870 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2495793379}) 2024-08-19T14:42:03.870 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=317524961}) 2024-08-19T14:42:03.870 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3168233751}) 2024-08-19T14:42:03.870 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=926848755}) 2024-08-19T14:42:03.870 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=397769427}) 2024-08-19T14:42:03.870 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1999511318}) 2024-08-19T14:42:03.870 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2741024824}) 2024-08-19T14:42:03.870 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2202145503}) 2024-08-19T14:42:03.870 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=914212547}) 2024-08-19T14:42:03.870 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2195185366}) 2024-08-19T14:42:03.870 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3411692791}) 2024-08-19T14:42:03.871 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2237803542}) 2024-08-19T14:42:03.871 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1697004363}) 2024-08-19T14:42:03.871 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2214339475}) 2024-08-19T14:42:03.871 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1548075756}) 2024-08-19T14:42:03.871 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1612270920}) 2024-08-19T14:42:03.871 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1156895575}) 2024-08-19T14:42:03.871 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3972633554}) 2024-08-19T14:42:03.871 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1553639209}) 2024-08-19T14:42:03.871 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=7250276}) 2024-08-19T14:42:03.871 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1437386178}) 2024-08-19T14:42:03.871 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2927545827}) 2024-08-19T14:42:03.871 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=453369477}) 2024-08-19T14:42:03.871 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=186316600}) 2024-08-19T14:42:03.871 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2311722870}) 2024-08-19T14:42:03.871 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2396447752}) 2024-08-19T14:42:03.872 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4238581967}) 2024-08-19T14:42:03.872 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3685228488}) 2024-08-19T14:42:03.872 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1067810075}) 2024-08-19T14:42:03.872 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3141699418}) 2024-08-19T14:42:03.872 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2979846439}) 2024-08-19T14:42:03.872 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2346258548}) 2024-08-19T14:42:03.872 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2929509397}) 2024-08-19T14:42:03.872 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=528266920}) 2024-08-19T14:42:03.872 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=499400314}) 2024-08-19T14:42:03.873 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2720366624}) 2024-08-19T14:42:03.873 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3465661353}) 2024-08-19T14:42:03.873 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=51,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=25} crc {logm=3095213747,mdsmap=2474562478,mgr=2924679688,mgr_command_descs=1300230762,mgr_metadata=482233419,mgrstat=1723312609}) 2024-08-19T14:42:03.873 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=33,mon_config_key=67} crc {mgrstat=2954130288,mon_config_key=1586131024}) 2024-08-19T14:42:03.873 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1757061686}) 2024-08-19T14:42:03.873 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3392079505}) 2024-08-19T14:42:03.873 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=12,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=70} crc {mon_config_key=2125561575,monmap=3193798142,osd_metadata=289051117,osd_pg_creating=2806808626,osdmap=3201246597}) 2024-08-19T14:42:03.873 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=913845312}) 2024-08-19T14:42:03.873 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3619438510}) 2024-08-19T14:42:03.873 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3140793736}) 2024-08-19T14:42:03.873 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3508553138}) 2024-08-19T14:42:03.873 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1698835763}) 2024-08-19T14:42:03.873 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=4194635033}) 2024-08-19T14:42:03.873 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1263353133}) 2024-08-19T14:42:03.874 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2830649815}) 2024-08-19T14:42:03.874 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2419199357}) 2024-08-19T14:42:03.874 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=76259912}) 2024-08-19T14:42:03.874 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=247711485}) 2024-08-19T14:42:03.874 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3695095570}) 2024-08-19T14:42:03.874 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3545774687}) 2024-08-19T14:42:03.874 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=162013159}) 2024-08-19T14:42:03.881 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:03.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:03.882 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:42:03.984 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:42:04.011 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:04.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:04.012 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:42:04.113 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:42:04.142 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:04.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:04.143 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:42:04.245 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:42:04.272 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:04.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:04.273 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:42:04.374 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:42:04.412 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:04.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:04.413 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:42:04.514 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:42:04.546 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:04.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:04.547 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:42:04.648 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:42:04.674 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:04.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:04.676 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:42:04.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:04 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=69} crc {osdmap=3025946257}) 2024-08-19T14:42:04.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:04 smithi032 ceph-mon[20004]: 1.f scrub starts 2024-08-19T14:42:04.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:04 smithi032 ceph-mon[20004]: 1.f scrub ok 2024-08-19T14:42:04.777 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:42:04.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:04 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=69} crc {osdmap=3025946257}) 2024-08-19T14:42:04.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:04 smithi045 ceph-mon[20143]: 1.f scrub starts 2024-08-19T14:42:04.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:04 smithi045 ceph-mon[20143]: 1.f scrub ok 2024-08-19T14:42:04.808 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:04.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:04.809 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:42:04.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:04 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=69} crc {osdmap=3025946257}) 2024-08-19T14:42:04.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:04 smithi031 ceph-mon[16336]: 1.f scrub starts 2024-08-19T14:42:04.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:04 smithi031 ceph-mon[16336]: 1.f scrub ok 2024-08-19T14:42:04.910 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:42:04.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:04.937 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:04.937 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:42:05.038 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:42:05.069 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:05.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:05.070 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:42:05.107 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-19T14:42:05.171 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:42:05.203 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:05.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:05.204 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:42:05.305 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:42:05.336 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:05.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:05.337 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:42:05.438 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:42:05.465 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:05.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:05.466 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:42:05.567 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:42:05.588 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:05.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:05.590 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:42:05.690 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:42:05.715 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:05.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:05.716 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:42:05.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:05 smithi032 ceph-mon[20004]: pgmap v1833: 57 pgs: 1 active+clean+scrubbing, 2 active+clean+scrubbing+deep, 54 active+clean; 5.6 GiB data, 26 GiB used, 779 GiB / 805 GiB avail 2024-08-19T14:42:05.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:05 smithi032 ceph-mon[20004]: 8.a scrub starts 2024-08-19T14:42:05.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:05 smithi045 ceph-mon[20143]: pgmap v1833: 57 pgs: 1 active+clean+scrubbing, 2 active+clean+scrubbing+deep, 54 active+clean; 5.6 GiB data, 26 GiB used, 779 GiB / 805 GiB avail 2024-08-19T14:42:05.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:05 smithi045 ceph-mon[20143]: 8.a scrub starts 2024-08-19T14:42:05.817 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:42:05.845 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:05.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:05.846 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:42:05.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:05 smithi031 ceph-mon[16336]: pgmap v1833: 57 pgs: 1 active+clean+scrubbing, 2 active+clean+scrubbing+deep, 54 active+clean; 5.6 GiB data, 26 GiB used, 779 GiB / 805 GiB avail 2024-08-19T14:42:05.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:05 smithi031 ceph-mon[16336]: 8.a scrub starts 2024-08-19T14:42:05.947 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:42:05.967 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:05.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:05.968 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:42:06.070 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:42:06.096 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:06.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:06.097 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:42:06.198 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:42:06.230 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:06.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:06.233 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:42:06.335 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:42:06.359 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:06.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:06.360 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:42:06.461 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:42:06.495 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:06.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:06.496 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:42:06.597 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:42:06.628 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:06.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:06.629 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:42:06.730 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:42:06.753 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:06.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:06.755 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:42:06.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:06 smithi032 ceph-mon[20004]: 8.17 deep-scrub ok 2024-08-19T14:42:06.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:06 smithi032 ceph-mon[20004]: 8.5 scrub starts 2024-08-19T14:42:06.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:06 smithi032 ceph-mon[20004]: 8.a scrub ok 2024-08-19T14:42:06.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:06 smithi045 ceph-mon[20143]: 8.17 deep-scrub ok 2024-08-19T14:42:06.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:06 smithi045 ceph-mon[20143]: 8.5 scrub starts 2024-08-19T14:42:06.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:06 smithi045 ceph-mon[20143]: 8.a scrub ok 2024-08-19T14:42:06.855 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:42:06.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:06 smithi031 ceph-mon[16336]: 8.17 deep-scrub ok 2024-08-19T14:42:06.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:06 smithi031 ceph-mon[16336]: 8.5 scrub starts 2024-08-19T14:42:06.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:06 smithi031 ceph-mon[16336]: 8.a scrub ok 2024-08-19T14:42:06.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:06.894 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:06.894 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:42:06.994 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:42:07.019 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:07.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:07.020 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:42:07.121 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:42:07.147 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:07.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:07.148 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:42:07.250 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:42:07.282 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:07.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:07.283 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:42:07.385 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:42:07.417 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:07.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:07.418 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:42:07.519 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:42:07.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:07.554 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:07.554 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:42:07.655 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:42:07.680 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:07.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:07.681 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:42:07.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:07 smithi032 ceph-mon[20004]: 8.4 scrub starts 2024-08-19T14:42:07.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:07 smithi032 ceph-mon[20004]: 8.5 scrub ok 2024-08-19T14:42:07.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:07 smithi032 ceph-mon[20004]: pgmap v1834: 57 pgs: 2 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 54 active+clean; 5.4 GiB data, 25 GiB used, 779 GiB / 805 GiB avail 2024-08-19T14:42:07.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:07 smithi032 ceph-mon[20004]: 8.4 scrub ok 2024-08-19T14:42:07.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:07 smithi032 ceph-mon[20004]: 1.14 scrub starts 2024-08-19T14:42:07.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:07 smithi032 ceph-mon[20004]: 1.14 scrub ok 2024-08-19T14:42:07.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:07 smithi032 ceph-mon[20004]: 8.c scrub starts 2024-08-19T14:42:07.782 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:42:07.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:07 smithi045 ceph-mon[20143]: 8.4 scrub starts 2024-08-19T14:42:07.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:07 smithi045 ceph-mon[20143]: 8.5 scrub ok 2024-08-19T14:42:07.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:07 smithi045 ceph-mon[20143]: pgmap v1834: 57 pgs: 2 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 54 active+clean; 5.4 GiB data, 25 GiB used, 779 GiB / 805 GiB avail 2024-08-19T14:42:07.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:07 smithi045 ceph-mon[20143]: 8.4 scrub ok 2024-08-19T14:42:07.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:07 smithi045 ceph-mon[20143]: 1.14 scrub starts 2024-08-19T14:42:07.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:07 smithi045 ceph-mon[20143]: 1.14 scrub ok 2024-08-19T14:42:07.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:07 smithi045 ceph-mon[20143]: 8.c scrub starts 2024-08-19T14:42:07.809 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:07.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:07.810 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:42:07.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:07 smithi031 ceph-mon[16336]: 8.4 scrub starts 2024-08-19T14:42:07.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:07 smithi031 ceph-mon[16336]: 8.5 scrub ok 2024-08-19T14:42:07.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:07 smithi031 ceph-mon[16336]: pgmap v1834: 57 pgs: 2 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 54 active+clean; 5.4 GiB data, 25 GiB used, 779 GiB / 805 GiB avail 2024-08-19T14:42:07.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:07 smithi031 ceph-mon[16336]: 8.4 scrub ok 2024-08-19T14:42:07.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:07 smithi031 ceph-mon[16336]: 1.14 scrub starts 2024-08-19T14:42:07.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:07 smithi031 ceph-mon[16336]: 1.14 scrub ok 2024-08-19T14:42:07.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:07 smithi031 ceph-mon[16336]: 8.c scrub starts 2024-08-19T14:42:07.910 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:42:07.940 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:07.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:07.941 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:42:08.042 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:42:08.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:08.070 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:08.071 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:42:08.172 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:42:08.201 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:08.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:08.202 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:42:08.303 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:42:08.329 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:08.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:08.330 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:42:08.431 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:42:08.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:08.460 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:08.460 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:42:08.561 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:42:08.586 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:08.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:08.588 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:42:08.689 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:42:08.719 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:08.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:08.720 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:42:08.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:08 smithi032 ceph-mon[20004]: 8.7 scrub ok 2024-08-19T14:42:08.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:08 smithi032 ceph-mon[20004]: 8.0 deep-scrub starts 2024-08-19T14:42:08.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:08 smithi032 ceph-mon[20004]: 8.c scrub ok 2024-08-19T14:42:08.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:08 smithi045 ceph-mon[20143]: 8.7 scrub ok 2024-08-19T14:42:08.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:08 smithi045 ceph-mon[20143]: 8.0 deep-scrub starts 2024-08-19T14:42:08.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:08 smithi045 ceph-mon[20143]: 8.c scrub ok 2024-08-19T14:42:08.821 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:42:08.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:08.856 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:08.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:42:08.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:08 smithi031 ceph-mon[16336]: 8.7 scrub ok 2024-08-19T14:42:08.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:08 smithi031 ceph-mon[16336]: 8.0 deep-scrub starts 2024-08-19T14:42:08.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:08 smithi031 ceph-mon[16336]: 8.c scrub ok 2024-08-19T14:42:08.957 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:42:08.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:08.987 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:08.988 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:42:09.088 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:42:09.124 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:09.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:09.125 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:42:09.227 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:42:09.252 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:09.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:09.254 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:42:09.355 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:42:09.379 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:09.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:09.380 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:42:09.482 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:42:09.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:09.515 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:09.516 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:42:09.617 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:42:09.643 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:09.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:09.644 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:42:09.745 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:42:09.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:09 smithi032 ceph-mon[20004]: pgmap v1835: 57 pgs: 1 active+clean+scrubbing+deep, 2 active+clean+scrubbing, 54 active+clean; 5.3 GiB data, 25 GiB used, 780 GiB / 805 GiB avail; 0 B/s wr, 14 op/s 2024-08-19T14:42:09.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:09 smithi032 ceph-mon[20004]: 1.8 scrub starts 2024-08-19T14:42:09.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:09 smithi032 ceph-mon[20004]: 1.8 scrub ok 2024-08-19T14:42:09.777 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:09.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:09.778 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:42:09.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:09 smithi045 ceph-mon[20143]: pgmap v1835: 57 pgs: 1 active+clean+scrubbing+deep, 2 active+clean+scrubbing, 54 active+clean; 5.3 GiB data, 25 GiB used, 780 GiB / 805 GiB avail; 0 B/s wr, 14 op/s 2024-08-19T14:42:09.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:09 smithi045 ceph-mon[20143]: 1.8 scrub starts 2024-08-19T14:42:09.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:09 smithi045 ceph-mon[20143]: 1.8 scrub ok 2024-08-19T14:42:09.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:09 smithi031 ceph-mon[16336]: pgmap v1835: 57 pgs: 1 active+clean+scrubbing+deep, 2 active+clean+scrubbing, 54 active+clean; 5.3 GiB data, 25 GiB used, 780 GiB / 805 GiB avail; 0 B/s wr, 14 op/s 2024-08-19T14:42:09.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:09 smithi031 ceph-mon[16336]: 1.8 scrub starts 2024-08-19T14:42:09.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:09 smithi031 ceph-mon[16336]: 1.8 scrub ok 2024-08-19T14:42:09.879 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:42:09.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:09.912 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:09.913 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:42:10.014 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:42:10.043 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:10.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:10.044 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:42:10.145 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:42:10.175 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:10.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:10.176 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:42:10.277 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:42:10.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:10.315 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:10.315 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:42:10.416 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:42:10.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:10.450 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:10.450 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:42:10.551 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:42:10.581 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:10.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:10.582 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:42:10.683 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:42:10.709 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:10.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:10.710 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:42:10.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:10 smithi032 ceph-mon[20004]: 1.d scrub starts 2024-08-19T14:42:10.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:10 smithi032 ceph-mon[20004]: 1.d scrub ok 2024-08-19T14:42:10.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:10 smithi045 ceph-mon[20143]: 1.d scrub starts 2024-08-19T14:42:10.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:10 smithi045 ceph-mon[20143]: 1.d scrub ok 2024-08-19T14:42:10.811 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:42:10.835 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:10.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:10.836 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:42:10.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:10 smithi031 ceph-mon[16336]: 1.d scrub starts 2024-08-19T14:42:10.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:10 smithi031 ceph-mon[16336]: 1.d scrub ok 2024-08-19T14:42:10.937 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:42:10.964 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:10.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:10.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:42:11.066 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:42:11.088 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:11.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:11.089 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:42:11.190 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:42:11.219 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:11.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:11.220 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:42:11.321 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:42:11.349 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:11.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:11.350 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:42:11.451 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:42:11.480 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:11.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:11.482 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:42:11.583 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:42:11.612 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:11.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:11.613 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:42:11.714 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:42:11.740 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:11.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:11.742 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:42:11.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:11 smithi032 ceph-mon[20004]: pgmap v1836: 57 pgs: 2 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 54 active+clean; 5.2 GiB data, 24 GiB used, 780 GiB / 805 GiB avail 2024-08-19T14:42:11.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:11 smithi045 ceph-mon[20143]: pgmap v1836: 57 pgs: 2 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 54 active+clean; 5.2 GiB data, 24 GiB used, 780 GiB / 805 GiB avail 2024-08-19T14:42:11.843 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:42:11.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:11 smithi031 ceph-mon[16336]: pgmap v1836: 57 pgs: 2 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 54 active+clean; 5.2 GiB data, 24 GiB used, 780 GiB / 805 GiB avail 2024-08-19T14:42:11.869 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:11.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:11.871 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:42:11.972 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:42:11.998 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:11.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:11.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:42:12.100 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:42:12.133 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:12.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:12.134 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:42:12.235 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:42:12.263 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:12.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:12.264 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:42:12.365 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:42:12.388 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:12.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:12.389 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:42:12.490 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:42:12.517 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:12.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:12.518 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:42:12.533 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:42:12.619 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:42:12.643 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:12.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:12.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:42:12.745 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:42:12.781 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:12.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:12.782 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:42:12.883 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:42:12.912 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:12.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:12.913 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:42:13.014 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:42:13.037 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:13.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:13.038 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:42:13.139 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:42:13.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:13.170 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:13.170 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:42:13.271 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:42:13.296 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:13.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:13.298 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:42:13.308 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-08-19T14:42:13.398 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:42:13.428 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:13.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:13.429 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:42:13.530 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:42:13.558 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:13.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:13.560 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:42:13.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:13 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:42:13.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:13 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:42:13.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:13 smithi031 ceph-mon[16336]: pgmap v1837: 57 pgs: 2 active+clean+scrubbing+deep, 55 active+clean; 5.0 GiB data, 24 GiB used, 781 GiB / 805 GiB avail; 0 B/s wr, 204 op/s 2024-08-19T14:42:13.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:13 smithi031 ceph-mon[16336]: osdmap e769: 12 total, 12 up, 9 in 2024-08-19T14:42:13.661 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:42:13.688 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:13.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:13.689 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:42:13.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:13 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:42:13.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:13 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:42:13.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:13 smithi032 ceph-mon[20004]: pgmap v1837: 57 pgs: 2 active+clean+scrubbing+deep, 55 active+clean; 5.0 GiB data, 24 GiB used, 781 GiB / 805 GiB avail; 0 B/s wr, 204 op/s 2024-08-19T14:42:13.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:13 smithi032 ceph-mon[20004]: osdmap e769: 12 total, 12 up, 9 in 2024-08-19T14:42:13.789 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:42:13.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:13 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:42:13.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:13 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:42:13.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:13 smithi045 ceph-mon[20143]: pgmap v1837: 57 pgs: 2 active+clean+scrubbing+deep, 55 active+clean; 5.0 GiB data, 24 GiB used, 781 GiB / 805 GiB avail; 0 B/s wr, 204 op/s 2024-08-19T14:42:13.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:13 smithi045 ceph-mon[20143]: osdmap e769: 12 total, 12 up, 9 in 2024-08-19T14:42:13.813 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:13.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:13.814 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:42:13.915 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:42:13.943 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:13.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:13.944 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:42:14.045 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:42:14.075 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:14.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:14.076 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:42:14.177 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:42:14.205 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:14.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:14.206 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:42:14.307 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:42:14.332 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:14.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:14.333 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:42:14.434 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:42:14.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:14.461 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:14.461 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:42:14.562 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:42:14.572 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:42:14.588 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:14.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:14.589 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:42:14.690 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:42:14.716 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:14.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:14.717 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:42:14.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:14 smithi045 ceph-mon[20143]: 8.b deep-scrub starts 2024-08-19T14:42:14.818 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:42:14.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:14.844 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:14.844 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:42:14.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:14 smithi031 ceph-mon[16336]: 8.b deep-scrub starts 2024-08-19T14:42:14.944 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:42:14.968 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:14.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:14.970 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:42:15.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:14 smithi032 ceph-mon[20004]: 8.b deep-scrub starts 2024-08-19T14:42:15.070 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:42:15.097 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:15.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:15.098 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:42:15.199 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:42:15.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:15.230 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:15.230 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:42:15.329 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:42:15.331 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:42:15.347 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:42:15.360 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:42:15.374 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:15.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:15.375 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:42:15.476 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:42:15.500 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:15.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:15.501 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:42:15.602 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:42:15.617 INFO:teuthology.orchestra.run.smithi031.stderr:Error EINVAL: specified pgp_num 91 > pg_num 31 2024-08-19T14:42:15.628 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:15.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:15.629 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:42:15.730 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:42:15.753 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:15.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:15.754 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:42:15.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:15 smithi045 ceph-mon[20143]: pgmap v1839: 57 pgs: 2 active+clean+scrubbing+deep, 55 active+clean; 4.9 GiB data, 24 GiB used, 781 GiB / 805 GiB avail; 0 B/s wr, 620 op/s 2024-08-19T14:42:15.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:15 smithi045 ceph-mon[20143]: 1.17 scrub starts 2024-08-19T14:42:15.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:15 smithi045 ceph-mon[20143]: 1.17 scrub ok 2024-08-19T14:42:15.854 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:42:15.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:15 smithi031 ceph-mon[16336]: pgmap v1839: 57 pgs: 2 active+clean+scrubbing+deep, 55 active+clean; 4.9 GiB data, 24 GiB used, 781 GiB / 805 GiB avail; 0 B/s wr, 620 op/s 2024-08-19T14:42:15.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:15 smithi031 ceph-mon[16336]: 1.17 scrub starts 2024-08-19T14:42:15.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:15 smithi031 ceph-mon[16336]: 1.17 scrub ok 2024-08-19T14:42:15.877 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:15.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:15.878 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:42:15.979 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:42:16.003 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:16.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:16.004 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:42:16.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:15 smithi032 ceph-mon[20004]: pgmap v1839: 57 pgs: 2 active+clean+scrubbing+deep, 55 active+clean; 4.9 GiB data, 24 GiB used, 781 GiB / 805 GiB avail; 0 B/s wr, 620 op/s 2024-08-19T14:42:16.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:15 smithi032 ceph-mon[20004]: 1.17 scrub starts 2024-08-19T14:42:16.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:15 smithi032 ceph-mon[20004]: 1.17 scrub ok 2024-08-19T14:42:16.105 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:42:16.133 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:16.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:16.134 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:42:16.235 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:42:16.261 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:16.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:16.262 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:42:16.363 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:42:16.388 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:16.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:16.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:42:16.491 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:42:16.522 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:16.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:16.523 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:42:16.624 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:42:16.659 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:16.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:16.660 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:42:16.761 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:42:16.781 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:16 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:42:16.781 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:16 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:42:16.781 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:16 smithi031 ceph-mon[16336]: 8.0 deep-scrub ok 2024-08-19T14:42:16.781 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:16 smithi031 ceph-mon[16336]: osdmap e770: 12 total, 12 up, 9 in 2024-08-19T14:42:16.781 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:16 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1833075417' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "91"}]: dispatch 2024-08-19T14:42:16.781 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:16 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "91"}]: dispatch 2024-08-19T14:42:16.787 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:16.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:16.788 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:42:16.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:16 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:42:16.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:16 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:42:16.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:16 smithi045 ceph-mon[20143]: 8.0 deep-scrub ok 2024-08-19T14:42:16.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:16 smithi045 ceph-mon[20143]: osdmap e770: 12 total, 12 up, 9 in 2024-08-19T14:42:16.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:16 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1833075417' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "91"}]: dispatch 2024-08-19T14:42:16.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:16 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "91"}]: dispatch 2024-08-19T14:42:16.889 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:42:16.922 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:16.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:16.923 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:42:17.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:16 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:42:17.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:16 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:42:17.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:16 smithi032 ceph-mon[20004]: 8.0 deep-scrub ok 2024-08-19T14:42:17.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:16 smithi032 ceph-mon[20004]: osdmap e770: 12 total, 12 up, 9 in 2024-08-19T14:42:17.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:16 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1833075417' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "91"}]: dispatch 2024-08-19T14:42:17.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:16 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "91"}]: dispatch 2024-08-19T14:42:17.023 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:42:17.052 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:17.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:17.053 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:42:17.154 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:42:17.186 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:17.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:17.208 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:42:17.308 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:42:17.331 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:17.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:17.332 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:42:17.433 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:42:17.455 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:17.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:17.456 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:42:17.531 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is set 2024-08-19T14:42:17.552 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:17 smithi031 ceph-mon[16336]: 1.15 scrub starts 2024-08-19T14:42:17.552 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:17 smithi031 ceph-mon[16336]: 1.15 scrub ok 2024-08-19T14:42:17.552 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:17 smithi031 ceph-mon[16336]: pgmap v1841: 57 pgs: 3 active+clean+scrubbing+deep, 54 active+clean; 4.8 GiB data, 23 GiB used, 781 GiB / 805 GiB avail; 0 B/s wr, 986 op/s 2024-08-19T14:42:17.552 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:17 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2064741081' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:42:17.552 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:17 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:42:17.556 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:42:17.578 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:17.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:17.579 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:42:17.658 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-19T14:42:17.680 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:42:17.703 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:17.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:17.704 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:42:17.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:17 smithi045 ceph-mon[20143]: 1.15 scrub starts 2024-08-19T14:42:17.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:17 smithi045 ceph-mon[20143]: 1.15 scrub ok 2024-08-19T14:42:17.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:17 smithi045 ceph-mon[20143]: pgmap v1841: 57 pgs: 3 active+clean+scrubbing+deep, 54 active+clean; 4.8 GiB data, 23 GiB used, 781 GiB / 805 GiB avail; 0 B/s wr, 986 op/s 2024-08-19T14:42:17.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:17 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2064741081' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:42:17.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:17 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:42:17.805 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:42:17.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:17.834 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:17.834 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:42:17.935 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:42:17.959 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:17.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:17.960 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:42:18.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:17 smithi032 ceph-mon[20004]: 1.15 scrub starts 2024-08-19T14:42:18.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:17 smithi032 ceph-mon[20004]: 1.15 scrub ok 2024-08-19T14:42:18.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:17 smithi032 ceph-mon[20004]: pgmap v1841: 57 pgs: 3 active+clean+scrubbing+deep, 54 active+clean; 4.8 GiB data, 23 GiB used, 781 GiB / 805 GiB avail; 0 B/s wr, 986 op/s 2024-08-19T14:42:18.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:17 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2064741081' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:42:18.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:17 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:42:18.061 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:42:18.098 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:18.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:18.100 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:42:18.172 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:42:18.173 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:42:18.200 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:42:18.201 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:42:18.201 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:42:18.222 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:18.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:18.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:42:18.225 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:42:18.225 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:42:18.254 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:42:18.254 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:42:18.281 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:42:18.281 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:42:18.306 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:42:18.307 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:42:18.324 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:42:18.328 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:42:18.328 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:42:18.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:18.352 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:18.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:42:18.357 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:42:18.357 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:42:18.365 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:42:18.365 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-08-19T14:42:18.365 INFO:teuthology.orchestra.run.smithi031.stdout: "complaint_time": 30, 2024-08-19T14:42:18.365 INFO:teuthology.orchestra.run.smithi031.stdout: "num_blocked_ops": 0 2024-08-19T14:42:18.365 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:42:18.387 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:42:18.387 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:42:18.418 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:42:18.419 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:42:18.448 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:42:18.448 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:42:18.452 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:42:18.471 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:42:18.471 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:42:18.476 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:18.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:18.477 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:42:18.578 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:42:18.607 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:18.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:18.608 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:42:18.709 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:42:18.734 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:18.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:18.736 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:42:18.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:18 smithi045 ceph-mon[20143]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:42:18.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:18 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:42:18.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:18 smithi045 ceph-mon[20143]: osdmap e771: 12 total, 12 up, 9 in 2024-08-19T14:42:18.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:18 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/634906476' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:42:18.836 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:42:18.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:18.862 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:18.862 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:42:18.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:18 smithi031 ceph-mon[16336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:42:18.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:18 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:42:18.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:18 smithi031 ceph-mon[16336]: osdmap e771: 12 total, 12 up, 9 in 2024-08-19T14:42:18.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:18 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/634906476' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:42:18.963 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:42:18.992 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:18.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:18.993 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:42:19.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:18 smithi032 ceph-mon[20004]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:42:19.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:18 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:42:19.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:18 smithi032 ceph-mon[20004]: osdmap e771: 12 total, 12 up, 9 in 2024-08-19T14:42:19.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:18 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/634906476' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:42:19.094 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:42:19.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:19.127 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:19.127 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:42:19.228 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:42:19.254 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:19.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:19.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:42:19.356 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:42:19.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:19.382 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:19.383 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:42:19.483 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:42:19.516 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:19.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:19.517 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:42:19.618 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:42:19.646 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:19.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:19.647 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:42:19.749 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:42:19.771 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:19.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:19.772 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:42:19.873 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:42:19.904 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:19.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:19.905 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:42:20.006 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:42:20.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:20.042 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:20.042 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:42:20.050 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-08-19T14:42:20.146 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:42:20.173 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:20.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:20.174 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:42:20.275 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:42:20.299 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:20.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:20.300 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:42:20.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:20 smithi031 ceph-mon[16336]: pgmap v1843: 57 pgs: 3 active+clean+scrubbing+deep, 54 active+clean; 4.8 GiB data, 23 GiB used, 781 GiB / 805 GiB avail; 0 B/s wr, 679 op/s 2024-08-19T14:42:20.401 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:42:20.431 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:20.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:20.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:42:20.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:20 smithi032 ceph-mon[20004]: pgmap v1843: 57 pgs: 3 active+clean+scrubbing+deep, 54 active+clean; 4.8 GiB data, 23 GiB used, 781 GiB / 805 GiB avail; 0 B/s wr, 679 op/s 2024-08-19T14:42:20.533 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:42:20.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:20 smithi045 ceph-mon[20143]: pgmap v1843: 57 pgs: 3 active+clean+scrubbing+deep, 54 active+clean; 4.8 GiB data, 23 GiB used, 781 GiB / 805 GiB avail; 0 B/s wr, 679 op/s 2024-08-19T14:42:20.559 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:20.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:20.560 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:42:20.661 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:42:20.689 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:20.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:20.690 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:42:20.776 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:42:20.791 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:42:20.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:20.825 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:20.825 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:42:20.927 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:42:20.957 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:20.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:20.959 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:42:21.028 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-19T14:42:21.059 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:42:21.115 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:21.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:21.116 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:42:21.218 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:42:21.242 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:21.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:21.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:42:21.344 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:42:21.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:21.383 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:21.383 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:42:21.403 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:42:21.403 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 20, 2024-08-19T14:42:21.403 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 600, 2024-08-19T14:42:21.403 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [ 2024-08-19T14:42:21.403 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.403 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "MOSDPGPush(7.a 650/635 [PushOp(7:5c8dad9c:::benchmark_data_smithi045_115842_object63220:head, version: 613'3943, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:5c8dad9c:::benchmark_data_smithi045_115842_object63220:head@613'3943, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-19T14:42:21.404 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:36:03.715886+0000", 2024-08-19T14:42:21.404 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 377.68436319800003, 2024-08-19T14:42:21.404 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.15676975000000001, 2024-08-19T14:42:21.404 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:42:21.404 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:42:21.404 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-08-19T14:42:21.404 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:42:21.404 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.404 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:42:21.404 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:03.715886+0000", 2024-08-19T14:42:21.404 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:42:21.405 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.405 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.405 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:42:21.405 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:03.715886+0000", 2024-08-19T14:42:21.405 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:42:21.405 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.405 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.405 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:42:21.405 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:03.715887+0000", 2024-08-19T14:42:21.405 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.083e-06 2024-08-19T14:42:21.406 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.406 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.406 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:42:21.406 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:03.715946+0000", 2024-08-19T14:42:21.406 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.8717e-05 2024-08-19T14:42:21.406 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.406 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.406 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:42:21.406 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:03.715972+0000", 2024-08-19T14:42:21.406 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.6299999999999999e-05 2024-08-19T14:42:21.407 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.407 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.407 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:42:21.407 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:03.715992+0000", 2024-08-19T14:42:21.407 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.9729999999999999e-05 2024-08-19T14:42:21.407 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.407 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.407 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:42:21.408 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:03.869724+0000", 2024-08-19T14:42:21.408 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.153731864 2024-08-19T14:42:21.408 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.408 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.408 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:42:21.408 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:03.869766+0000", 2024-08-19T14:42:21.408 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.1819000000000001e-05 2024-08-19T14:42:21.408 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.408 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.408 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:42:21.408 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:03.872656+0000", 2024-08-19T14:42:21.409 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0028902369999999999 2024-08-19T14:42:21.409 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:42:21.409 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:42:21.409 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:42:21.409 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.409 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.409 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "MOSDPGPush(7.2 650/635 [PushOp(7:4789c4c6:::benchmark_data_smithi045_115842_object23753:head, version: 608'1512, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:4789c4c6:::benchmark_data_smithi045_115842_object23753:head@608'1512, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-19T14:42:21.409 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:36:03.725918+0000", 2024-08-19T14:42:21.410 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 377.674331263, 2024-08-19T14:42:21.410 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.16257834299999999, 2024-08-19T14:42:21.410 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:42:21.410 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:42:21.410 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-08-19T14:42:21.410 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:42:21.410 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.410 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:42:21.411 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:03.725918+0000", 2024-08-19T14:42:21.411 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:42:21.411 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.411 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.411 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:42:21.411 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:03.725918+0000", 2024-08-19T14:42:21.411 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:42:21.411 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.411 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.412 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:42:21.412 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:03.725919+0000", 2024-08-19T14:42:21.412 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.7499999999999999e-07 2024-08-19T14:42:21.412 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.412 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.412 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:42:21.412 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:03.726078+0000", 2024-08-19T14:42:21.412 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00015877400000000001 2024-08-19T14:42:21.412 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.412 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.412 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:42:21.412 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:03.726102+0000", 2024-08-19T14:42:21.413 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.3808e-05 2024-08-19T14:42:21.413 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.413 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.413 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:42:21.413 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:03.726119+0000", 2024-08-19T14:42:21.413 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.7694e-05 2024-08-19T14:42:21.413 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.413 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.413 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:42:21.413 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:03.879620+0000", 2024-08-19T14:42:21.413 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.153501208 2024-08-19T14:42:21.413 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.413 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.413 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:42:21.414 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:03.879634+0000", 2024-08-19T14:42:21.414 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.31e-05 2024-08-19T14:42:21.414 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.414 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.414 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:42:21.414 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:03.888496+0000", 2024-08-19T14:42:21.414 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0088629840000000008 2024-08-19T14:42:21.414 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:42:21.414 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:42:21.414 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:42:21.414 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.414 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.415 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.16962.0:181780 7.e e678/634 7:7b43e48b:::benchmark_data_smithi045_115842_object63620:head v 678'11470, mlcod=613'4070)", 2024-08-19T14:42:21.415 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:37:43.454572+0000", 2024-08-19T14:42:21.415 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 277.945677531, 2024-08-19T14:42:21.415 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.161739627, 2024-08-19T14:42:21.415 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:42:21.415 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:42:21.415 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:42:21.415 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:42:21.415 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.415 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:42:21.415 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.454572+0000", 2024-08-19T14:42:21.415 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:42:21.415 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.415 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.416 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:42:21.416 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.454572+0000", 2024-08-19T14:42:21.416 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:42:21.416 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.416 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.416 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:42:21.416 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.454572+0000", 2024-08-19T14:42:21.416 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.48e-07 2024-08-19T14:42:21.416 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.416 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.416 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:42:21.417 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.454576+0000", 2024-08-19T14:42:21.417 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2910000000000001e-06 2024-08-19T14:42:21.417 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.417 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.417 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:42:21.417 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.454583+0000", 2024-08-19T14:42:21.417 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.3890000000000002e-06 2024-08-19T14:42:21.417 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.417 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.417 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:42:21.417 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.454588+0000", 2024-08-19T14:42:21.417 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0860000000000001e-06 2024-08-19T14:42:21.418 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.418 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.418 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:42:21.418 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.473215+0000", 2024-08-19T14:42:21.418 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.018626888000000001 2024-08-19T14:42:21.418 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.419 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.419 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:42:21.419 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.473241+0000", 2024-08-19T14:42:21.419 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.6454000000000001e-05 2024-08-19T14:42:21.419 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.419 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.419 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:42:21.420 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.616231+0000", 2024-08-19T14:42:21.420 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.14299003299999999 2024-08-19T14:42:21.421 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.421 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.421 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:42:21.421 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.616312+0000", 2024-08-19T14:42:21.421 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.0238e-05 2024-08-19T14:42:21.421 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:42:21.422 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:42:21.422 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:42:21.422 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.422 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.422 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.16962.0:181792 7.e e678/634 7:76fea455:::benchmark_data_smithi045_115842_object63632:head v 678'11471, mlcod=613'4070)", 2024-08-19T14:42:21.422 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:37:43.470051+0000", 2024-08-19T14:42:21.422 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 277.93019825900001, 2024-08-19T14:42:21.422 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.14628962500000001, 2024-08-19T14:42:21.423 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:42:21.423 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:42:21.423 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:42:21.423 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:42:21.423 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.423 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:42:21.423 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.470051+0000", 2024-08-19T14:42:21.423 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:42:21.423 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.423 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.423 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:42:21.423 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.470051+0000", 2024-08-19T14:42:21.423 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:42:21.424 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.424 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.424 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:42:21.424 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.470052+0000", 2024-08-19T14:42:21.424 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.032e-06 2024-08-19T14:42:21.424 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.424 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.424 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:42:21.424 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.470061+0000", 2024-08-19T14:42:21.424 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.0270000000000001e-06 2024-08-19T14:42:21.424 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.424 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.424 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:42:21.424 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.470074+0000", 2024-08-19T14:42:21.425 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2651e-05 2024-08-19T14:42:21.425 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.425 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.425 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:42:21.425 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.470082+0000", 2024-08-19T14:42:21.425 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.0469999999999994e-06 2024-08-19T14:42:21.425 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.425 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.425 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:42:21.425 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.487157+0000", 2024-08-19T14:42:21.425 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.017074807000000001 2024-08-19T14:42:21.425 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.425 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.425 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:42:21.425 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.487182+0000", 2024-08-19T14:42:21.426 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.5338e-05 2024-08-19T14:42:21.426 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.426 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.426 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:42:21.426 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.616318+0000", 2024-08-19T14:42:21.426 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.129135683 2024-08-19T14:42:21.426 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.426 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.426 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:42:21.426 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.616341+0000", 2024-08-19T14:42:21.426 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.304e-05 2024-08-19T14:42:21.426 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:42:21.426 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:42:21.427 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:42:21.427 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.427 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.427 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.16962.0:181793 7.e e678/634 7:73e8a3a5:::benchmark_data_smithi045_115842_object63633:head v 678'11472, mlcod=613'4070)", 2024-08-19T14:42:21.427 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:37:43.470096+0000", 2024-08-19T14:42:21.428 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 277.93015315600002, 2024-08-19T14:42:21.428 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.14626562000000001, 2024-08-19T14:42:21.428 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:42:21.428 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:42:21.428 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:42:21.428 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:42:21.428 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.428 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:42:21.428 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.470096+0000", 2024-08-19T14:42:21.428 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:42:21.428 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.428 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.428 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:42:21.428 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.470096+0000", 2024-08-19T14:42:21.428 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:42:21.429 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.429 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.429 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:42:21.429 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.470096+0000", 2024-08-19T14:42:21.429 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.3699999999999999e-07 2024-08-19T14:42:21.429 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.429 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.429 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:42:21.429 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.470100+0000", 2024-08-19T14:42:21.429 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.9829999999999998e-06 2024-08-19T14:42:21.429 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.429 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.429 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:42:21.429 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.470106+0000", 2024-08-19T14:42:21.430 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.8490000000000002e-06 2024-08-19T14:42:21.430 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.430 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.430 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:42:21.430 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.470111+0000", 2024-08-19T14:42:21.430 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.5480000000000003e-06 2024-08-19T14:42:21.430 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.431 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.431 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:42:21.431 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.501919+0000", 2024-08-19T14:42:21.431 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.031808014000000003 2024-08-19T14:42:21.431 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.431 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.432 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:42:21.432 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.501945+0000", 2024-08-19T14:42:21.432 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.6227000000000001e-05 2024-08-19T14:42:21.432 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.432 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.432 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:42:21.432 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.616343+0000", 2024-08-19T14:42:21.432 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.114397659 2024-08-19T14:42:21.432 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.432 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.432 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:42:21.432 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.616362+0000", 2024-08-19T14:42:21.432 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.9103e-05 2024-08-19T14:42:21.433 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:42:21.433 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:42:21.433 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:42:21.433 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.433 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.433 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.16962.0:199394 7.2 e690/635 7:406e87f5:::benchmark_data_smithi045_115842_object81234:head v 690'12614, mlcod=621'5122)", 2024-08-19T14:42:21.433 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:38:22.737244+0000", 2024-08-19T14:42:21.433 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 238.66300570999999, 2024-08-19T14:42:21.433 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.24429008799999999, 2024-08-19T14:42:21.433 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:42:21.433 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:42:21.433 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:42:21.433 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:42:21.433 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.434 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:42:21.434 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.737244+0000", 2024-08-19T14:42:21.434 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:42:21.434 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.434 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.434 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:42:21.434 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.737244+0000", 2024-08-19T14:42:21.434 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:42:21.434 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.434 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.434 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:42:21.434 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.737245+0000", 2024-08-19T14:42:21.434 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.6299999999999993e-07 2024-08-19T14:42:21.434 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.435 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.435 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:42:21.435 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.737251+0000", 2024-08-19T14:42:21.435 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.6930000000000001e-06 2024-08-19T14:42:21.435 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.435 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.435 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:42:21.435 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.737262+0000", 2024-08-19T14:42:21.435 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0922e-05 2024-08-19T14:42:21.435 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.435 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.435 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:42:21.435 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.737269+0000", 2024-08-19T14:42:21.435 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.6680000000000004e-06 2024-08-19T14:42:21.435 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.436 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.436 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:42:21.436 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.737648+0000", 2024-08-19T14:42:21.436 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00037954999999999998 2024-08-19T14:42:21.436 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.436 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.436 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:42:21.436 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.737671+0000", 2024-08-19T14:42:21.436 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.2784000000000001e-05 2024-08-19T14:42:21.436 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.436 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.436 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:42:21.436 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.981464+0000", 2024-08-19T14:42:21.436 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.243792642 2024-08-19T14:42:21.437 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.437 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.437 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:42:21.437 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.981534+0000", 2024-08-19T14:42:21.437 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.9865999999999998e-05 2024-08-19T14:42:21.437 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:42:21.437 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:42:21.437 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:42:21.437 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.437 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.437 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.16962.0:199392 7.2 e690/635 7:4580edf3:::benchmark_data_smithi045_115842_object81232:head v 690'12615, mlcod=621'5122)", 2024-08-19T14:42:21.437 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:38:22.746524+0000", 2024-08-19T14:42:21.437 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 238.65372575000001, 2024-08-19T14:42:21.438 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.23504549899999999, 2024-08-19T14:42:21.438 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:42:21.438 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:42:21.438 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:42:21.438 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:42:21.438 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.438 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:42:21.438 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.746524+0000", 2024-08-19T14:42:21.438 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:42:21.438 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.438 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.438 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:42:21.438 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.746524+0000", 2024-08-19T14:42:21.438 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:42:21.439 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.439 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.439 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:42:21.439 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.746525+0000", 2024-08-19T14:42:21.439 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0279999999999999e-06 2024-08-19T14:42:21.439 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.439 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.439 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:42:21.439 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.746533+0000", 2024-08-19T14:42:21.439 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.0069999999999997e-06 2024-08-19T14:42:21.439 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.439 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.439 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:42:21.440 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.746545+0000", 2024-08-19T14:42:21.440 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2584e-05 2024-08-19T14:42:21.440 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.440 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.440 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:42:21.440 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.746554+0000", 2024-08-19T14:42:21.440 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.0180000000000007e-06 2024-08-19T14:42:21.440 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.440 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.440 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:42:21.440 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.752313+0000", 2024-08-19T14:42:21.440 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0057586139999999996 2024-08-19T14:42:21.440 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.440 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.441 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:42:21.441 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.752334+0000", 2024-08-19T14:42:21.441 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.0735000000000001e-05 2024-08-19T14:42:21.441 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.441 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.441 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:42:21.441 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.981537+0000", 2024-08-19T14:42:21.441 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.22920327500000001 2024-08-19T14:42:21.441 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.441 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.441 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:42:21.441 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.981569+0000", 2024-08-19T14:42:21.441 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.2237999999999999e-05 2024-08-19T14:42:21.441 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:42:21.441 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:42:21.442 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:42:21.442 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.442 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.442 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.16962.0:199409 7.2 e690/635 7:4c8d089b:::benchmark_data_smithi045_115842_object81249:head v 690'12616, mlcod=621'5122)", 2024-08-19T14:42:21.442 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:38:22.750502+0000", 2024-08-19T14:42:21.442 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 238.649747555, 2024-08-19T14:42:21.442 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.255713405, 2024-08-19T14:42:21.442 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:42:21.442 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:42:21.442 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:42:21.442 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:42:21.442 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.442 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:42:21.443 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.750502+0000", 2024-08-19T14:42:21.443 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:42:21.443 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.443 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.443 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:42:21.443 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.750502+0000", 2024-08-19T14:42:21.443 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:42:21.443 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.443 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.443 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:42:21.443 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.750503+0000", 2024-08-19T14:42:21.443 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.09e-07 2024-08-19T14:42:21.443 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.443 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.444 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:42:21.444 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.750511+0000", 2024-08-19T14:42:21.444 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.9759999999999995e-06 2024-08-19T14:42:21.444 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.444 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.444 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:42:21.444 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.750522+0000", 2024-08-19T14:42:21.444 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1241e-05 2024-08-19T14:42:21.444 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.444 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.444 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:42:21.444 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.750529+0000", 2024-08-19T14:42:21.444 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.3629999999999996e-06 2024-08-19T14:42:21.444 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.445 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.445 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:42:21.445 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.765007+0000", 2024-08-19T14:42:21.445 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.014478058 2024-08-19T14:42:21.445 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.445 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.445 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:42:21.445 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.765023+0000", 2024-08-19T14:42:21.445 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5735000000000001e-05 2024-08-19T14:42:21.445 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.445 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.445 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:42:21.445 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:23.006112+0000", 2024-08-19T14:42:21.445 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.24108848799999999 2024-08-19T14:42:21.446 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.446 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.446 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:42:21.446 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:23.006215+0000", 2024-08-19T14:42:21.446 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00010363500000000001 2024-08-19T14:42:21.446 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:42:21.446 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:42:21.446 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:42:21.446 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.446 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.446 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.16962.0:199414 7.c e690/635 7:366d05c1:::benchmark_data_smithi045_115842_object81254:head v 690'12294, mlcod=621'5016)", 2024-08-19T14:42:21.446 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:38:22.751905+0000", 2024-08-19T14:42:21.446 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 238.648344167, 2024-08-19T14:42:21.447 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.20950311999999999, 2024-08-19T14:42:21.447 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:42:21.447 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:42:21.447 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:42:21.447 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:42:21.447 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.447 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:42:21.447 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.751905+0000", 2024-08-19T14:42:21.447 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:42:21.447 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.447 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.447 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:42:21.447 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.751905+0000", 2024-08-19T14:42:21.447 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:42:21.447 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.448 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.448 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:42:21.448 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.751906+0000", 2024-08-19T14:42:21.448 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.4000000000000001e-07 2024-08-19T14:42:21.448 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.448 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.448 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:42:21.448 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.751912+0000", 2024-08-19T14:42:21.448 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.0100000000000001e-06 2024-08-19T14:42:21.448 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.448 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.448 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:42:21.448 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.751923+0000", 2024-08-19T14:42:21.448 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.098e-05 2024-08-19T14:42:21.449 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.449 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.449 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:42:21.449 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.751930+0000", 2024-08-19T14:42:21.449 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.0600000000000002e-06 2024-08-19T14:42:21.449 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.449 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.449 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:42:21.449 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.752067+0000", 2024-08-19T14:42:21.449 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013742100000000001 2024-08-19T14:42:21.449 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.449 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.449 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:42:21.449 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.752089+0000", 2024-08-19T14:42:21.450 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.1389000000000001e-05 2024-08-19T14:42:21.450 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.450 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.450 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:42:21.450 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.766981+0000", 2024-08-19T14:42:21.450 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.014892396 2024-08-19T14:42:21.450 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.450 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.450 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:42:21.450 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.961408+0000", 2024-08-19T14:42:21.450 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.19442712400000001 2024-08-19T14:42:21.450 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:42:21.450 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:42:21.450 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:42:21.451 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.451 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.451 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.16962.0:199422 7.a e690/635 7:5ea5870d:::benchmark_data_smithi045_115842_object81262:head v 690'12384, mlcod=633'7611)", 2024-08-19T14:42:21.451 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:38:22.759906+0000", 2024-08-19T14:42:21.451 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 238.640343339, 2024-08-19T14:42:21.451 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.242679165, 2024-08-19T14:42:21.451 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:42:21.451 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:42:21.451 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:42:21.451 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:42:21.451 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.451 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:42:21.451 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.759906+0000", 2024-08-19T14:42:21.452 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:42:21.452 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.452 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.452 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:42:21.452 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.759906+0000", 2024-08-19T14:42:21.452 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:42:21.452 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.452 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.452 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:42:21.452 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.759908+0000", 2024-08-19T14:42:21.452 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5880000000000001e-06 2024-08-19T14:42:21.452 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.452 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.452 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:42:21.452 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.759916+0000", 2024-08-19T14:42:21.453 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.4950000000000008e-06 2024-08-19T14:42:21.453 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.453 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.453 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:42:21.453 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.759930+0000", 2024-08-19T14:42:21.453 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4236000000000001e-05 2024-08-19T14:42:21.453 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.453 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.453 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:42:21.453 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.759941+0000", 2024-08-19T14:42:21.453 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1052e-05 2024-08-19T14:42:21.453 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.453 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.453 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:42:21.454 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.765122+0000", 2024-08-19T14:42:21.454 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0051806439999999999 2024-08-19T14:42:21.454 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.454 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.454 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:42:21.454 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.765141+0000", 2024-08-19T14:42:21.454 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.8470000000000001e-05 2024-08-19T14:42:21.454 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.454 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.454 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:42:21.454 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:23.002541+0000", 2024-08-19T14:42:21.454 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.23740081599999999 2024-08-19T14:42:21.454 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.454 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.455 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:42:21.455 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:23.002585+0000", 2024-08-19T14:42:21.455 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.3863999999999999e-05 2024-08-19T14:42:21.455 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:42:21.455 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:42:21.455 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:42:21.455 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.455 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.455 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "MOSDPGPush(7.2 690/635 [PushOp(7:48749d7b:::benchmark_data_smithi045_115842_object81270:head, version: 621'5127, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:48749d7b:::benchmark_data_smithi045_115842_object81270:head@621'5127, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-19T14:42:21.455 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:38:22.764504+0000", 2024-08-19T14:42:21.455 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 238.63574517800001, 2024-08-19T14:42:21.455 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.20554665, 2024-08-19T14:42:21.455 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:42:21.456 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:42:21.456 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-08-19T14:42:21.456 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:42:21.456 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.456 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:42:21.456 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.764504+0000", 2024-08-19T14:42:21.456 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:42:21.456 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.456 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.456 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:42:21.456 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.764504+0000", 2024-08-19T14:42:21.456 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:42:21.456 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.456 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.457 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:42:21.457 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.764505+0000", 2024-08-19T14:42:21.457 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.6400000000000001e-07 2024-08-19T14:42:21.457 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.457 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.457 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:42:21.457 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.764536+0000", 2024-08-19T14:42:21.457 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.0719000000000002e-05 2024-08-19T14:42:21.457 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.457 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.457 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:42:21.457 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.764556+0000", 2024-08-19T14:42:21.457 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.0142000000000001e-05 2024-08-19T14:42:21.457 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.457 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.458 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:42:21.458 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.764573+0000", 2024-08-19T14:42:21.458 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6685999999999999e-05 2024-08-19T14:42:21.458 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.458 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.458 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:42:21.458 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.966878+0000", 2024-08-19T14:42:21.458 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.20230553300000001 2024-08-19T14:42:21.458 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.458 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.458 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:42:21.458 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.966888+0000", 2024-08-19T14:42:21.458 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.026e-05 2024-08-19T14:42:21.459 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.459 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.459 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:42:21.459 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.970051+0000", 2024-08-19T14:42:21.459 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.003162546 2024-08-19T14:42:21.459 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:42:21.459 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:42:21.459 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:42:21.459 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.459 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.459 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "MOSDPGRecoveryDelete(7.e e690,634 7:7b982ab6:::benchmark_data_smithi045_115842_object57353:head,673'11068)", 2024-08-19T14:42:21.459 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:38:22.767854+0000", 2024-08-19T14:42:21.459 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 238.63239537300001, 2024-08-19T14:42:21.459 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.19375594500000001, 2024-08-19T14:42:21.460 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:42:21.460 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:42:21.460 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-08-19T14:42:21.460 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:42:21.460 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.460 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:42:21.460 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.767854+0000", 2024-08-19T14:42:21.460 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:42:21.460 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.460 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.460 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:42:21.460 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.767854+0000", 2024-08-19T14:42:21.460 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:42:21.461 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.461 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.461 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:42:21.461 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.767855+0000", 2024-08-19T14:42:21.461 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2410000000000001e-06 2024-08-19T14:42:21.461 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.461 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.461 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:42:21.461 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.767872+0000", 2024-08-19T14:42:21.461 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6707000000000001e-05 2024-08-19T14:42:21.461 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.461 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.461 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:42:21.461 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.961428+0000", 2024-08-19T14:42:21.461 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.19355593600000001 2024-08-19T14:42:21.462 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.462 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.462 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:42:21.462 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.961441+0000", 2024-08-19T14:42:21.462 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3269e-05 2024-08-19T14:42:21.462 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.462 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.462 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:42:21.462 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.961526+0000", 2024-08-19T14:42:21.462 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.4942000000000001e-05 2024-08-19T14:42:21.462 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.462 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.462 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:42:21.463 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.961534+0000", 2024-08-19T14:42:21.463 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.6350000000000006e-06 2024-08-19T14:42:21.463 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.463 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.463 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:42:21.463 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.961610+0000", 2024-08-19T14:42:21.463 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.6215000000000001e-05 2024-08-19T14:42:21.463 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:42:21.463 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:42:21.463 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:42:21.463 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.463 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.463 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.16962.0:199399 7.3 e690/635 7:cecd84d1:::benchmark_data_smithi045_115842_object81239:head v 690'12702, mlcod=621'5174)", 2024-08-19T14:42:21.463 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:38:22.776198+0000", 2024-08-19T14:42:21.464 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 238.62405150699999, 2024-08-19T14:42:21.464 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.20653019, 2024-08-19T14:42:21.464 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:42:21.464 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:42:21.464 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:42:21.464 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:42:21.464 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.464 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:42:21.464 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.776198+0000", 2024-08-19T14:42:21.465 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:42:21.465 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.465 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.465 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:42:21.465 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.776198+0000", 2024-08-19T14:42:21.465 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:42:21.466 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.466 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.466 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:42:21.466 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.776199+0000", 2024-08-19T14:42:21.466 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2449999999999999e-06 2024-08-19T14:42:21.466 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.466 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.466 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:42:21.466 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.776210+0000", 2024-08-19T14:42:21.466 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1289e-05 2024-08-19T14:42:21.466 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.466 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.466 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:42:21.467 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.961366+0000", 2024-08-19T14:42:21.467 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.18515516700000001 2024-08-19T14:42:21.467 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.467 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.467 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:42:21.467 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.961387+0000", 2024-08-19T14:42:21.467 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.1248999999999999e-05 2024-08-19T14:42:21.468 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.468 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.468 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:42:21.468 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.961448+0000", 2024-08-19T14:42:21.468 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.1062000000000006e-05 2024-08-19T14:42:21.468 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.468 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.468 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:42:21.468 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.961480+0000", 2024-08-19T14:42:21.468 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.1775e-05 2024-08-19T14:42:21.468 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.468 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.468 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:42:21.468 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.982684+0000", 2024-08-19T14:42:21.469 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.021203914000000001 2024-08-19T14:42:21.469 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.469 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.469 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:42:21.469 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.982728+0000", 2024-08-19T14:42:21.469 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4489e-05 2024-08-19T14:42:21.469 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:42:21.469 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:42:21.469 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:42:21.469 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.469 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.469 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.26911.0:99126 8.e e754/717 8:714ccbf2:::benchmark_data_smithi045_137103_object99125:head v 754'6233, mlcod=753'6231)", 2024-08-19T14:42:21.469 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:41:28.704684+0000", 2024-08-19T14:42:21.469 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 52.695565248000001, 2024-08-19T14:42:21.470 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.14110450799999999, 2024-08-19T14:42:21.470 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:42:21.470 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:42:21.470 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:42:21.470 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:42:21.470 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.470 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:42:21.470 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:28.704684+0000", 2024-08-19T14:42:21.470 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:42:21.470 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.470 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.470 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:42:21.470 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:28.704684+0000", 2024-08-19T14:42:21.471 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:42:21.471 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.471 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.471 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:42:21.471 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:28.704685+0000", 2024-08-19T14:42:21.471 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0049999999999999e-06 2024-08-19T14:42:21.471 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.471 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.471 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:42:21.471 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:28.704712+0000", 2024-08-19T14:42:21.471 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.6897999999999999e-05 2024-08-19T14:42:21.471 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.471 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.471 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:42:21.472 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:28.704726+0000", 2024-08-19T14:42:21.472 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4328000000000001e-05 2024-08-19T14:42:21.472 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.472 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.472 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:42:21.472 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:28.704733+0000", 2024-08-19T14:42:21.472 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.0849999999999999e-06 2024-08-19T14:42:21.472 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.472 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.472 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:42:21.472 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:28.704791+0000", 2024-08-19T14:42:21.472 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.7868999999999998e-05 2024-08-19T14:42:21.472 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.473 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.473 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:42:21.473 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:28.704814+0000", 2024-08-19T14:42:21.473 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.2844e-05 2024-08-19T14:42:21.473 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.473 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.473 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:42:21.473 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:28.709511+0000", 2024-08-19T14:42:21.473 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0046969749999999999 2024-08-19T14:42:21.473 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.473 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.473 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:42:21.473 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:28.845789+0000", 2024-08-19T14:42:21.473 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.13627750399999999 2024-08-19T14:42:21.473 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:42:21.474 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:42:21.474 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:42:21.474 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.474 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.474 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.26911.0:99128 8.14 e754/750 8:2b98078d:::benchmark_data_smithi045_137103_object99127:head v 754'5132, mlcod=753'5130)", 2024-08-19T14:42:21.474 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:41:28.707998+0000", 2024-08-19T14:42:21.474 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 52.69225136, 2024-08-19T14:42:21.474 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.14108353500000001, 2024-08-19T14:42:21.474 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:42:21.474 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:42:21.474 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:42:21.474 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:42:21.474 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.475 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:42:21.475 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:28.707998+0000", 2024-08-19T14:42:21.475 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:42:21.475 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.475 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.475 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:42:21.475 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:28.707998+0000", 2024-08-19T14:42:21.475 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:42:21.475 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.475 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.475 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:42:21.475 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:28.707998+0000", 2024-08-19T14:42:21.475 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.6199999999999999e-07 2024-08-19T14:42:21.475 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.476 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.476 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:42:21.476 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:28.708028+0000", 2024-08-19T14:42:21.476 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.993e-05 2024-08-19T14:42:21.476 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.476 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.476 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:42:21.476 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:28.708040+0000", 2024-08-19T14:42:21.476 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1416999999999999e-05 2024-08-19T14:42:21.476 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.476 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.476 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:42:21.476 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:28.708047+0000", 2024-08-19T14:42:21.476 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.5920000000000003e-06 2024-08-19T14:42:21.476 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.477 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.477 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:42:21.477 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:28.845732+0000", 2024-08-19T14:42:21.477 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.13768438099999999 2024-08-19T14:42:21.477 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.477 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.477 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:42:21.477 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:28.845763+0000", 2024-08-19T14:42:21.477 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.0954000000000002e-05 2024-08-19T14:42:21.477 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.477 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.477 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:42:21.477 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:28.849040+0000", 2024-08-19T14:42:21.477 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0032775170000000002 2024-08-19T14:42:21.477 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.478 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.478 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:42:21.478 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:28.849082+0000", 2024-08-19T14:42:21.478 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.1381999999999998e-05 2024-08-19T14:42:21.478 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:42:21.478 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:42:21.478 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:42:21.478 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.478 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.478 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.26911.0:99129 8.14 e754/750 8:2ca1a9a1:::benchmark_data_smithi045_137103_object99128:head v 754'5133, mlcod=753'5130)", 2024-08-19T14:42:21.478 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:41:28.709638+0000", 2024-08-19T14:42:21.478 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 52.690611130999997, 2024-08-19T14:42:21.478 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.139467022, 2024-08-19T14:42:21.478 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:42:21.479 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:42:21.479 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:42:21.479 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:42:21.479 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.479 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:42:21.479 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:28.709638+0000", 2024-08-19T14:42:21.479 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:42:21.479 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.479 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.479 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:42:21.479 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:28.709638+0000", 2024-08-19T14:42:21.479 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:42:21.479 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.479 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.479 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:42:21.480 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:28.709639+0000", 2024-08-19T14:42:21.480 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.1500000000000003e-07 2024-08-19T14:42:21.480 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.480 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.480 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:42:21.480 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:28.709774+0000", 2024-08-19T14:42:21.480 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013489099999999999 2024-08-19T14:42:21.480 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.480 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.480 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:42:21.480 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:28.845802+0000", 2024-08-19T14:42:21.480 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.1360285 2024-08-19T14:42:21.480 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.481 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.481 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:42:21.481 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:28.845810+0000", 2024-08-19T14:42:21.481 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.8790000000000003e-06 2024-08-19T14:42:21.481 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.481 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.481 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:42:21.481 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:28.847275+0000", 2024-08-19T14:42:21.481 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.001464141 2024-08-19T14:42:21.481 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.481 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.481 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:42:21.481 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:28.847293+0000", 2024-08-19T14:42:21.481 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.8604999999999999e-05 2024-08-19T14:42:21.481 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.482 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.482 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:42:21.482 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:28.849084+0000", 2024-08-19T14:42:21.482 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0017911839999999999 2024-08-19T14:42:21.482 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.482 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.482 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:42:21.482 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:28.849105+0000", 2024-08-19T14:42:21.482 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.1007000000000001e-05 2024-08-19T14:42:21.482 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:42:21.482 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:42:21.482 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:42:21.482 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.482 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.483 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "MOSDPGPush(8.12 754/734 [PushOp(8:4b2adb7e:::benchmark_data_smithi045_137103_object65588:head, version: 730'4056, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:4b2adb7e:::benchmark_data_smithi045_137103_object65588:head@730'4056, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-19T14:42:21.483 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:41:28.711325+0000", 2024-08-19T14:42:21.483 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 52.688924286000002, 2024-08-19T14:42:21.483 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.142688281, 2024-08-19T14:42:21.483 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:42:21.483 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:42:21.483 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-08-19T14:42:21.483 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:42:21.483 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.483 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:42:21.483 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:28.711325+0000", 2024-08-19T14:42:21.483 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:42:21.483 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.483 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.484 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:42:21.484 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:28.711325+0000", 2024-08-19T14:42:21.484 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:42:21.484 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.484 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.484 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:42:21.484 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:28.711326+0000", 2024-08-19T14:42:21.484 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.95e-07 2024-08-19T14:42:21.484 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.484 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.484 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:42:21.484 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:28.711359+0000", 2024-08-19T14:42:21.484 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.3250000000000002e-05 2024-08-19T14:42:21.484 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.485 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.485 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:42:21.485 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:28.848011+0000", 2024-08-19T14:42:21.485 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.13665176800000001 2024-08-19T14:42:21.485 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.485 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.485 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:42:21.485 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:28.848519+0000", 2024-08-19T14:42:21.485 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00050756200000000003 2024-08-19T14:42:21.485 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.486 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.486 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:42:21.486 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:28.851404+0000", 2024-08-19T14:42:21.486 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.002885454 2024-08-19T14:42:21.486 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.486 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.486 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:42:21.486 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:28.851445+0000", 2024-08-19T14:42:21.486 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.0571999999999997e-05 2024-08-19T14:42:21.486 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.486 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.486 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:42:21.486 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:28.854013+0000", 2024-08-19T14:42:21.486 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0025686799999999998 2024-08-19T14:42:21.487 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:42:21.487 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:42:21.487 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:42:21.487 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.487 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.487 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.26911.0:112730 8.e e763/717 8:7b021f56:::benchmark_data_smithi045_137103_object266:head v 763'7120, mlcod=763'7118)", 2024-08-19T14:42:21.487 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:41:42.489992+0000", 2024-08-19T14:42:21.487 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 38.910257082999998, 2024-08-19T14:42:21.487 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.14127273100000001, 2024-08-19T14:42:21.487 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:42:21.487 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:42:21.487 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:42:21.487 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:42:21.487 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.488 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:42:21.488 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:42.489992+0000", 2024-08-19T14:42:21.488 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:42:21.488 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.488 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.488 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:42:21.488 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:42.489992+0000", 2024-08-19T14:42:21.488 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:42:21.488 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.488 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.488 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:42:21.488 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:42.489993+0000", 2024-08-19T14:42:21.488 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.5499999999999996e-07 2024-08-19T14:42:21.488 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.489 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.489 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:42:21.489 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:42.490000+0000", 2024-08-19T14:42:21.489 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.8040000000000004e-06 2024-08-19T14:42:21.489 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.489 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.489 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:42:21.489 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:42.490011+0000", 2024-08-19T14:42:21.489 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.064e-05 2024-08-19T14:42:21.489 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.489 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.489 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:42:21.489 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:42.490018+0000", 2024-08-19T14:42:21.489 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.2089999999999999e-06 2024-08-19T14:42:21.489 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.490 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.490 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:42:21.490 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:42.490071+0000", 2024-08-19T14:42:21.490 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.346e-05 2024-08-19T14:42:21.490 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.490 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.490 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:42:21.490 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:42.490091+0000", 2024-08-19T14:42:21.490 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.9123e-05 2024-08-19T14:42:21.490 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.490 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.490 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:42:21.490 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:42.631226+0000", 2024-08-19T14:42:21.490 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.141135608 2024-08-19T14:42:21.491 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.491 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.491 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:42:21.491 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:42.631265+0000", 2024-08-19T14:42:21.491 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.8931999999999999e-05 2024-08-19T14:42:21.491 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:42:21.491 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:42:21.491 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:42:21.491 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.491 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.491 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.26911.0:112731 8.f e763/717 8:ff6e92ec:::benchmark_data_smithi045_137103_object267:head v 763'7064, mlcod=763'7062)", 2024-08-19T14:42:21.491 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:41:42.490155+0000", 2024-08-19T14:42:21.491 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 38.910094555000001, 2024-08-19T14:42:21.491 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.14124139899999999, 2024-08-19T14:42:21.492 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:42:21.492 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:42:21.492 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:42:21.492 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:42:21.492 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.492 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:42:21.492 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:42.490155+0000", 2024-08-19T14:42:21.492 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:42:21.492 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.492 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.492 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:42:21.492 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:42.490155+0000", 2024-08-19T14:42:21.492 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:42:21.492 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.493 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.493 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:42:21.493 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:42.490155+0000", 2024-08-19T14:42:21.493 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.1399999999999997e-07 2024-08-19T14:42:21.493 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.493 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.493 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:42:21.493 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:42.490161+0000", 2024-08-19T14:42:21.493 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0839999999999999e-06 2024-08-19T14:42:21.493 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.493 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.493 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:42:21.493 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:42.490170+0000", 2024-08-19T14:42:21.493 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.1349999999999998e-06 2024-08-19T14:42:21.494 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.494 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.494 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:42:21.494 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:42.490175+0000", 2024-08-19T14:42:21.494 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.587e-06 2024-08-19T14:42:21.494 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.494 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.494 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:42:21.494 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:42.490225+0000", 2024-08-19T14:42:21.494 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.9937e-05 2024-08-19T14:42:21.494 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.494 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.494 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:42:21.494 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:42.490244+0000", 2024-08-19T14:42:21.495 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.9094e-05 2024-08-19T14:42:21.495 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.495 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.495 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:42:21.495 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:42.631352+0000", 2024-08-19T14:42:21.495 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.14110751199999999 2024-08-19T14:42:21.495 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.495 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.495 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:42:21.495 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:42.631396+0000", 2024-08-19T14:42:21.495 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4435999999999999e-05 2024-08-19T14:42:21.495 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:42:21.495 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:42:21.495 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:42:21.496 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.496 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.496 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.26911.0:112733 8.d e763/717 8:b1f595a5:::benchmark_data_smithi045_137103_object269:head v 763'7096, mlcod=763'7094)", 2024-08-19T14:42:21.496 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:41:42.493271+0000", 2024-08-19T14:42:21.496 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 38.906978877999997, 2024-08-19T14:42:21.496 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.14344847499999999, 2024-08-19T14:42:21.496 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:42:21.496 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:42:21.496 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:42:21.496 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:42:21.496 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.496 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:42:21.496 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:42.493271+0000", 2024-08-19T14:42:21.496 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:42:21.497 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.497 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.497 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:42:21.497 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:42.493271+0000", 2024-08-19T14:42:21.497 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:42:21.497 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.497 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.497 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:42:21.497 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:42.493271+0000", 2024-08-19T14:42:21.497 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.5799999999999999e-07 2024-08-19T14:42:21.497 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.497 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.497 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:42:21.497 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:42.493277+0000", 2024-08-19T14:42:21.497 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.3890000000000004e-06 2024-08-19T14:42:21.498 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.498 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.498 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:42:21.498 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:42.493285+0000", 2024-08-19T14:42:21.498 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.3939999999999996e-06 2024-08-19T14:42:21.498 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.498 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.498 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:42:21.498 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:42.493292+0000", 2024-08-19T14:42:21.498 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.596e-06 2024-08-19T14:42:21.498 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.498 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.498 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:42:21.498 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:42.631429+0000", 2024-08-19T14:42:21.499 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.13813727100000001 2024-08-19T14:42:21.499 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.499 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.499 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:42:21.499 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:42.631449+0000", 2024-08-19T14:42:21.499 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.0353e-05 2024-08-19T14:42:21.499 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.499 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.499 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:42:21.499 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:42.636691+0000", 2024-08-19T14:42:21.499 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0052417139999999997 2024-08-19T14:42:21.499 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:21.499 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:21.499 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:42:21.499 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:42.636719+0000", 2024-08-19T14:42:21.500 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.8099999999999999e-05 2024-08-19T14:42:21.500 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:42:21.500 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:42:21.500 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:42:21.500 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:42:21.500 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:42:21.500 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:42:21.500 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:42:21.531 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:21.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:21.532 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:42:21.633 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:42:21.655 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:21.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:21.657 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:42:21.758 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:42:21.786 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:21.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:21.787 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:42:21.888 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:42:21.915 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:21.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:21.916 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:42:22.017 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:42:22.049 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:22.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:22.050 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:42:22.151 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:42:22.173 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:22.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:22.174 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:42:22.275 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:42:22.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:22.309 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:22.309 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:42:22.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:22 smithi031 ceph-mon[16336]: pgmap v1844: 57 pgs: 2 active+clean+scrubbing+deep, 55 active+clean; 4.7 GiB data, 23 GiB used, 782 GiB / 805 GiB avail; 0 B/s wr, 625 op/s 2024-08-19T14:42:22.410 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:42:22.437 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:22.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:22.438 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:42:22.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:22 smithi032 ceph-mon[20004]: pgmap v1844: 57 pgs: 2 active+clean+scrubbing+deep, 55 active+clean; 4.7 GiB data, 23 GiB used, 782 GiB / 805 GiB avail; 0 B/s wr, 625 op/s 2024-08-19T14:42:22.539 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:42:22.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:22 smithi045 ceph-mon[20143]: pgmap v1844: 57 pgs: 2 active+clean+scrubbing+deep, 55 active+clean; 4.7 GiB data, 23 GiB used, 782 GiB / 805 GiB avail; 0 B/s wr, 625 op/s 2024-08-19T14:42:22.566 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:22.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:22.567 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:42:22.639 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:42:22.659 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 4, 5, 7, 8, 2, 11, 9] out_osds: [10, 6, 3] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 8, 9, 11, 10, 2] 2024-08-19T14:42:22.660 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-19T14:42:22.660 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2024-08-19T14:42:22.660 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-08-19T14:42:22.660 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-08-19T14:42:22.660 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.6 config set filestore_inject_stall 3 2024-08-19T14:42:22.668 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:42:22.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:22.702 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:22.703 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:42:22.805 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:42:22.835 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:22.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:22.836 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:42:22.937 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:42:22.963 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:22.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:22.964 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:42:22.970 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:42:23.064 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:42:23.093 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:23.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:23.094 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:42:23.195 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:42:23.221 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:23.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:23.222 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:42:23.322 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:42:23.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:23.354 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:23.354 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:42:23.456 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:42:23.485 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-08-19T14:42:23.488 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:23.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:23.489 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:42:23.510 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:42:23.590 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:42:23.615 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:23.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:23.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:42:23.717 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:42:23.758 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:23.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:23.759 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:42:23.797 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:42:23.797 INFO:teuthology.orchestra.run.smithi032.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-08-19T14:42:23.797 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:42:23.860 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:42:23.882 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:23.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:23.883 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:42:23.984 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:42:24.016 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:24.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:24.017 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:42:24.115 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:24 smithi031 ceph-mon[16336]: pgmap v1845: 57 pgs: 2 active+clean+scrubbing+deep, 55 active+clean; 4.5 GiB data, 23 GiB used, 782 GiB / 805 GiB avail; 0 B/s wr, 882 op/s 2024-08-19T14:42:24.118 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:42:24.145 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:24.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:24.146 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:42:24.196 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:42:24.247 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:42:24.249 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:42:24.273 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:24.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:24.274 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:42:24.348 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:24 smithi045 ceph-mon[20143]: pgmap v1845: 57 pgs: 2 active+clean+scrubbing+deep, 55 active+clean; 4.5 GiB data, 23 GiB used, 782 GiB / 805 GiB avail; 0 B/s wr, 882 op/s 2024-08-19T14:42:24.374 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:42:24.389 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:24 smithi032 ceph-mon[20004]: pgmap v1845: 57 pgs: 2 active+clean+scrubbing+deep, 55 active+clean; 4.5 GiB data, 23 GiB used, 782 GiB / 805 GiB avail; 0 B/s wr, 882 op/s 2024-08-19T14:42:24.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:24.412 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:24.413 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:42:24.513 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:42:24.541 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:24.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:24.542 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:42:24.643 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:42:24.671 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:24.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:24.672 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:42:24.755 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:42:24.756 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:42:24.772 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:42:24.784 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:42:24.784 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:42:24.794 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:24.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:24.795 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:42:24.803 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:42:24.804 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:42:24.828 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:42:24.828 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:42:24.850 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:42:24.851 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:42:24.884 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:42:24.884 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:42:24.896 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:42:24.915 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:42:24.915 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:42:24.923 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:24.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:24.924 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:42:24.946 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:42:24.946 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:42:24.968 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:42:24.968 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:42:24.997 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:42:24.997 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:42:25.018 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:42:25.018 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:42:25.025 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:42:25.045 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:42:25.046 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:42:25.050 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:25.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:25.051 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:42:25.068 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is set 2024-08-19T14:42:25.152 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:42:25.176 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:25.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:25.177 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:42:25.278 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:42:25.309 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:25.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:25.310 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:42:25.324 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:25 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3739103761' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:42:25.325 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:25 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/4160303720' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:42:25.412 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:42:25.441 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:25.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:25.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:42:25.458 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:42:25.458 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:42:25.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:25 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3739103761' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:42:25.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:25 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/4160303720' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:42:25.544 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:42:25.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:25 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3739103761' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:42:25.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:25 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/4160303720' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:42:25.575 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:25.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:25.576 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:42:25.678 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:42:25.711 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:25.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:25.712 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:42:25.813 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:42:25.841 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:25.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:25.842 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:42:25.942 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:42:25.973 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:25.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:25.974 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:42:26.075 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:42:26.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:26.107 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:26.107 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:42:26.148 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-08-19T14:42:26.208 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:42:26.236 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:26.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:26.237 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:42:26.338 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:42:26.364 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:26 smithi031 ceph-mon[16336]: pgmap v1846: 57 pgs: 2 active+clean+scrubbing+deep, 55 active+clean; 4.5 GiB data, 22 GiB used, 782 GiB / 805 GiB avail; 0 B/s wr, 826 op/s 2024-08-19T14:42:26.364 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:26 smithi031 ceph-mon[16336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:42:26.364 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:26 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3739103761' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:42:26.365 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:26 smithi031 ceph-mon[16336]: osdmap e772: 12 total, 12 up, 9 in 2024-08-19T14:42:26.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:26.373 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:26.373 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:42:26.474 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:42:26.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:26 smithi032 ceph-mon[20004]: pgmap v1846: 57 pgs: 2 active+clean+scrubbing+deep, 55 active+clean; 4.5 GiB data, 22 GiB used, 782 GiB / 805 GiB avail; 0 B/s wr, 826 op/s 2024-08-19T14:42:26.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:26 smithi032 ceph-mon[20004]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:42:26.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:26 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3739103761' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:42:26.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:26 smithi032 ceph-mon[20004]: osdmap e772: 12 total, 12 up, 9 in 2024-08-19T14:42:26.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:26.509 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:26.509 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:42:26.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:26 smithi045 ceph-mon[20143]: pgmap v1846: 57 pgs: 2 active+clean+scrubbing+deep, 55 active+clean; 4.5 GiB data, 22 GiB used, 782 GiB / 805 GiB avail; 0 B/s wr, 826 op/s 2024-08-19T14:42:26.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:26 smithi045 ceph-mon[20143]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:42:26.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:26 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3739103761' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:42:26.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:26 smithi045 ceph-mon[20143]: osdmap e772: 12 total, 12 up, 9 in 2024-08-19T14:42:26.610 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:42:26.637 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:26.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:26.638 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:42:26.739 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:42:26.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:26.770 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:26.770 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:42:26.871 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:42:26.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:26.907 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:26.907 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:42:27.008 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:42:27.034 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:27.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:27.036 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:42:27.137 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:42:27.172 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:27.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:27.173 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:42:27.274 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:42:27.302 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:27.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:27.303 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:42:27.404 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:42:27.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:27.440 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:27.440 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:42:27.541 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:42:27.555 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:42:27.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:27.579 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:27.579 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:42:27.680 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:42:27.713 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:27.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:27.714 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:42:27.816 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:42:27.850 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:27.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:27.851 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:42:27.952 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:42:27.983 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:27.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:27.984 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:42:28.085 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:42:28.115 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:28.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:28.116 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:42:28.134 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:42:28.134 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:42:28.217 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:42:28.248 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:28.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:28.249 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:42:28.350 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:42:28.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:28 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:42:28.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:28 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:42:28.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:28 smithi031 ceph-mon[16336]: pgmap v1848: 57 pgs: 1 active+clean+scrubbing+deep, 56 active+clean; 4.3 GiB data, 22 GiB used, 783 GiB / 805 GiB avail; 0 B/s wr, 809 op/s 2024-08-19T14:42:28.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:28 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:42:28.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:28 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:42:28.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:28 smithi031 ceph-mon[16336]: osdmap e773: 12 total, 12 up, 9 in 2024-08-19T14:42:28.381 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:28.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:28.382 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:42:28.483 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:42:28.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:28 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:42:28.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:28 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:42:28.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:28 smithi032 ceph-mon[20004]: pgmap v1848: 57 pgs: 1 active+clean+scrubbing+deep, 56 active+clean; 4.3 GiB data, 22 GiB used, 783 GiB / 805 GiB avail; 0 B/s wr, 809 op/s 2024-08-19T14:42:28.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:28 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:42:28.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:28 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:42:28.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:28 smithi032 ceph-mon[20004]: osdmap e773: 12 total, 12 up, 9 in 2024-08-19T14:42:28.509 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:28.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:28.510 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:42:28.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:28 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:42:28.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:28 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:42:28.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:28 smithi045 ceph-mon[20143]: pgmap v1848: 57 pgs: 1 active+clean+scrubbing+deep, 56 active+clean; 4.3 GiB data, 22 GiB used, 783 GiB / 805 GiB avail; 0 B/s wr, 809 op/s 2024-08-19T14:42:28.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:28 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:42:28.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:28 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:42:28.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:28 smithi045 ceph-mon[20143]: osdmap e773: 12 total, 12 up, 9 in 2024-08-19T14:42:28.610 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:42:28.642 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:28.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:28.643 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:42:28.744 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:42:28.771 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:28.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:28.772 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:42:28.812 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:42:28.873 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:42:28.908 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:28.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:28.909 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:42:28.977 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-08-19T14:42:29.010 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:42:29.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:29.048 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:29.048 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:42:29.150 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:42:29.179 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:29.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:29.180 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:42:29.231 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:42:29.281 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:42:29.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:29.311 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:29.311 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:42:29.412 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:42:29.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:29.445 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:29.445 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:42:29.455 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-19T14:42:29.546 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:42:29.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:29.576 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:29.576 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:42:29.633 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 4, 5, 7, 8, 2, 11, 9] out_osds: [10, 6, 3] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 8, 9, 11, 10, 2] 2024-08-19T14:42:29.633 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-19T14:42:29.634 INFO:tasks.thrashosds.thrasher:Removing osd 0, in_osds are: [0, 1, 4, 5, 7, 8, 2, 11, 9] 2024-08-19T14:42:29.634 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd out 0 2024-08-19T14:42:29.676 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:42:29.699 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:29.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:29.700 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:42:29.801 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:42:29.828 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:29.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:29.829 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:42:29.868 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:42:29.869 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:42:29.877 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:42:29.902 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:42:29.915 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:42:29.915 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:42:29.930 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:42:29.955 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:42:29.955 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:42:29.964 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:29.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:29.965 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:42:29.984 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:42:29.984 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:42:30.010 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:42:30.011 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:42:30.040 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:42:30.040 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:42:30.066 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:42:30.069 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:42:30.070 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:42:30.103 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:42:30.104 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:42:30.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:30.114 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:30.114 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:42:30.136 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:42:30.136 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:42:30.162 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:42:30.163 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:42:30.196 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:30 smithi031 ceph-mon[16336]: pgmap v1850: 57 pgs: 57 active+clean; 4.2 GiB data, 22 GiB used, 783 GiB / 805 GiB avail; 0 B/s wr, 1.01k op/s 2024-08-19T14:42:30.196 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:30 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2028009605' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:42:30.196 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:42:30.196 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:42:30.215 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:42:30.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:30.242 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:30.242 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:42:30.247 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:42:30.248 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:42:30.343 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:42:30.373 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:30.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:30.374 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:42:30.475 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:42:30.500 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:30.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:30.501 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:42:30.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:30 smithi032 ceph-mon[20004]: pgmap v1850: 57 pgs: 57 active+clean; 4.2 GiB data, 22 GiB used, 783 GiB / 805 GiB avail; 0 B/s wr, 1.01k op/s 2024-08-19T14:42:30.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:30 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2028009605' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:42:30.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:30 smithi045 ceph-mon[20143]: pgmap v1850: 57 pgs: 57 active+clean; 4.2 GiB data, 22 GiB used, 783 GiB / 805 GiB avail; 0 B/s wr, 1.01k op/s 2024-08-19T14:42:30.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:30 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2028009605' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:42:30.602 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:42:30.633 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:30.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:30.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:42:30.735 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:42:30.769 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:30.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:30.770 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:42:30.871 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:42:30.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:30.903 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:30.903 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:42:31.004 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:42:31.036 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:31.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:31.037 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:42:31.138 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:42:31.171 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:31.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:31.172 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:42:31.273 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:42:31.297 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:31.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:31.298 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:42:31.400 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:42:31.442 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:31.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:31.443 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:42:31.459 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:42:31.459 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 20, 2024-08-19T14:42:31.459 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 600, 2024-08-19T14:42:31.459 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [ 2024-08-19T14:42:31.459 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.460 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26191.0:187356 6.a 6:575cb98d:::benchmark_data_smithi045_97937_object80083:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e583)", 2024-08-19T14:42:31.460 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:32:32.420297+0000", 2024-08-19T14:42:31.460 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 598.36219290600002, 2024-08-19T14:42:31.460 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.34771512199999999, 2024-08-19T14:42:31.460 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:42:31.460 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:42:31.460 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:42:31.460 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:42:31.460 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26191", 2024-08-19T14:42:31.460 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3586680839", 2024-08-19T14:42:31.460 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 187356 2024-08-19T14:42:31.461 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.461 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:42:31.461 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.461 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:42:31.461 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.420297+0000", 2024-08-19T14:42:31.461 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:42:31.461 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.461 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.461 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:42:31.461 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.420297+0000", 2024-08-19T14:42:31.461 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:42:31.461 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.461 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.461 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:42:31.462 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.420298+0000", 2024-08-19T14:42:31.462 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.8100000000000001e-07 2024-08-19T14:42:31.462 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.462 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.462 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:42:31.462 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.420302+0000", 2024-08-19T14:42:31.462 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.374e-06 2024-08-19T14:42:31.462 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.462 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.462 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:42:31.462 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.420317+0000", 2024-08-19T14:42:31.462 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4683e-05 2024-08-19T14:42:31.462 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.463 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.463 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:42:31.463 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.420324+0000", 2024-08-19T14:42:31.463 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.3869999999999997e-06 2024-08-19T14:42:31.463 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.463 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.463 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:42:31.463 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.551342+0000", 2024-08-19T14:42:31.463 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.13101795199999999 2024-08-19T14:42:31.463 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.463 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.463 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:42:31.463 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.551468+0000", 2024-08-19T14:42:31.463 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012597899999999999 2024-08-19T14:42:31.464 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.464 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.464 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 1,5", 2024-08-19T14:42:31.464 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.551599+0000", 2024-08-19T14:42:31.464 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000130369 2024-08-19T14:42:31.464 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.464 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.464 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:42:31.464 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.557051+0000", 2024-08-19T14:42:31.464 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0054523949999999996 2024-08-19T14:42:31.465 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.465 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.465 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:42:31.465 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.560585+0000", 2024-08-19T14:42:31.465 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0035343240000000001 2024-08-19T14:42:31.465 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.465 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.465 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:42:31.465 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.767809+0000", 2024-08-19T14:42:31.465 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.20722385400000001 2024-08-19T14:42:31.465 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.466 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.466 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:42:31.466 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.767907+0000", 2024-08-19T14:42:31.466 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.7996000000000002e-05 2024-08-19T14:42:31.466 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.466 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.466 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:42:31.466 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:32:32.768012+0000", 2024-08-19T14:42:31.466 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000104928 2024-08-19T14:42:31.466 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:42:31.466 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:42:31.467 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:42:31.467 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.467 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.467 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:157465 7.f 7:f1de090e:::benchmark_data_smithi045_115842_object39305:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e663)", 2024-08-19T14:42:31.467 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:36:41.844976+0000", 2024-08-19T14:42:31.467 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 348.93751413500001, 2024-08-19T14:42:31.467 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.32793317300000002, 2024-08-19T14:42:31.467 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:42:31.467 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:42:31.467 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:42:31.467 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:42:31.468 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:42:31.468 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:42:31.468 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 157465 2024-08-19T14:42:31.468 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.468 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:42:31.468 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.468 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:42:31.468 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:41.844976+0000", 2024-08-19T14:42:31.468 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:42:31.468 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.468 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.469 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:42:31.469 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:41.844976+0000", 2024-08-19T14:42:31.469 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:42:31.469 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.469 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.469 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:42:31.469 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:41.844976+0000", 2024-08-19T14:42:31.469 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2e-07 2024-08-19T14:42:31.469 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.469 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.469 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:42:31.469 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:41.844979+0000", 2024-08-19T14:42:31.469 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.1889999999999999e-06 2024-08-19T14:42:31.470 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.470 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.470 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:42:31.470 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:41.844987+0000", 2024-08-19T14:42:31.470 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.0069999999999997e-06 2024-08-19T14:42:31.470 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.470 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.470 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:42:31.470 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:41.844993+0000", 2024-08-19T14:42:31.470 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.835e-06 2024-08-19T14:42:31.470 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.470 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.471 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:42:31.471 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:41.963617+0000", 2024-08-19T14:42:31.471 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.118623785 2024-08-19T14:42:31.471 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.471 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.471 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:42:31.471 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:41.963705+0000", 2024-08-19T14:42:31.471 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.7458000000000002e-05 2024-08-19T14:42:31.471 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.471 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.471 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 8,9", 2024-08-19T14:42:31.471 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:41.963780+0000", 2024-08-19T14:42:31.471 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.5042000000000004e-05 2024-08-19T14:42:31.471 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.472 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.472 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:42:31.472 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:41.977760+0000", 2024-08-19T14:42:31.472 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.013980461 2024-08-19T14:42:31.472 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.472 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.472 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:42:31.472 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:41.977797+0000", 2024-08-19T14:42:31.472 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7088999999999997e-05 2024-08-19T14:42:31.472 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.472 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.472 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:42:31.472 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:42.172777+0000", 2024-08-19T14:42:31.472 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.19497942300000001 2024-08-19T14:42:31.472 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.473 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.473 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:42:31.473 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:42.172837+0000", 2024-08-19T14:42:31.473 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.0047999999999998e-05 2024-08-19T14:42:31.473 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.473 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.473 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:42:31.473 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:42.172909+0000", 2024-08-19T14:42:31.473 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.2416000000000006e-05 2024-08-19T14:42:31.473 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:42:31.473 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:42:31.474 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:42:31.474 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.474 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.474 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:163704 7.f 7:f2a17ec5:::benchmark_data_smithi045_115842_object45544:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e667)", 2024-08-19T14:42:31.474 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:36:59.137143+0000", 2024-08-19T14:42:31.474 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 331.64534676400001, 2024-08-19T14:42:31.474 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.409542774, 2024-08-19T14:42:31.474 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:42:31.474 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:42:31.475 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:42:31.475 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:42:31.476 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:42:31.476 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:42:31.476 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 163704 2024-08-19T14:42:31.476 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.476 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:42:31.476 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.476 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:42:31.476 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.137143+0000", 2024-08-19T14:42:31.476 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:42:31.476 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.476 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.476 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:42:31.476 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.137143+0000", 2024-08-19T14:42:31.477 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:42:31.477 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.477 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.477 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:42:31.477 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.137145+0000", 2024-08-19T14:42:31.477 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.615e-06 2024-08-19T14:42:31.477 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.477 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.477 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:42:31.478 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.137151+0000", 2024-08-19T14:42:31.478 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.2360000000000001e-06 2024-08-19T14:42:31.478 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.478 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.478 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:42:31.478 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.137165+0000", 2024-08-19T14:42:31.478 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3504e-05 2024-08-19T14:42:31.479 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.479 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.479 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:42:31.479 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.137173+0000", 2024-08-19T14:42:31.479 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.3999999999999992e-06 2024-08-19T14:42:31.479 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.479 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.479 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:42:31.479 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.332675+0000", 2024-08-19T14:42:31.479 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.19550209499999999 2024-08-19T14:42:31.479 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.480 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.480 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:42:31.480 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.332815+0000", 2024-08-19T14:42:31.480 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013971799999999999 2024-08-19T14:42:31.480 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.480 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.480 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 8,9", 2024-08-19T14:42:31.480 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.332962+0000", 2024-08-19T14:42:31.480 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00014688900000000001 2024-08-19T14:42:31.480 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.480 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.480 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:42:31.481 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.348845+0000", 2024-08-19T14:42:31.481 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.015883026000000001 2024-08-19T14:42:31.481 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.481 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.481 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:42:31.481 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.349034+0000", 2024-08-19T14:42:31.481 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000189624 2024-08-19T14:42:31.481 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.481 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.481 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:42:31.481 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.546583+0000", 2024-08-19T14:42:31.481 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.19754886199999999 2024-08-19T14:42:31.481 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.482 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.482 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:42:31.482 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.546631+0000", 2024-08-19T14:42:31.482 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7682000000000002e-05 2024-08-19T14:42:31.482 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.482 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.482 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:42:31.482 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.546686+0000", 2024-08-19T14:42:31.482 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.5123000000000001e-05 2024-08-19T14:42:31.482 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:42:31.482 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:42:31.482 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:42:31.482 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.482 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.483 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:163711 7.f 7:f8f1f6f9:::benchmark_data_smithi045_115842_object45551:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e667)", 2024-08-19T14:42:31.483 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:36:59.161886+0000", 2024-08-19T14:42:31.483 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 331.620603534, 2024-08-19T14:42:31.483 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.38488678900000001, 2024-08-19T14:42:31.483 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:42:31.483 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:42:31.483 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:42:31.483 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:42:31.483 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:42:31.483 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:42:31.483 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 163711 2024-08-19T14:42:31.483 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.483 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:42:31.484 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.484 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:42:31.484 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.161886+0000", 2024-08-19T14:42:31.484 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:42:31.484 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.484 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.484 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:42:31.484 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.161886+0000", 2024-08-19T14:42:31.484 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:42:31.484 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.484 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.484 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:42:31.484 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.161888+0000", 2024-08-19T14:42:31.484 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6199999999999999e-06 2024-08-19T14:42:31.485 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.485 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.485 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:42:31.485 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.161895+0000", 2024-08-19T14:42:31.485 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.8329999999999996e-06 2024-08-19T14:42:31.485 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.485 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.485 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:42:31.485 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.161909+0000", 2024-08-19T14:42:31.485 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3913e-05 2024-08-19T14:42:31.485 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.485 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.485 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:42:31.486 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.161917+0000", 2024-08-19T14:42:31.486 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.2239999999999995e-06 2024-08-19T14:42:31.486 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.486 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.486 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:42:31.486 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.338110+0000", 2024-08-19T14:42:31.486 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.176193025 2024-08-19T14:42:31.486 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.486 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.486 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:42:31.486 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.338242+0000", 2024-08-19T14:42:31.486 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013215699999999999 2024-08-19T14:42:31.486 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.487 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.487 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 8,9", 2024-08-19T14:42:31.487 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.338366+0000", 2024-08-19T14:42:31.487 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012349299999999999 2024-08-19T14:42:31.487 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.487 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.487 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:42:31.487 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.349206+0000", 2024-08-19T14:42:31.487 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.010840214000000001 2024-08-19T14:42:31.488 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.488 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.488 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:42:31.488 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.349395+0000", 2024-08-19T14:42:31.488 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00018901599999999999 2024-08-19T14:42:31.488 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.488 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.488 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:42:31.488 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.546698+0000", 2024-08-19T14:42:31.488 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.197302739 2024-08-19T14:42:31.488 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.488 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.489 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:42:31.489 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.546734+0000", 2024-08-19T14:42:31.489 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.5914000000000002e-05 2024-08-19T14:42:31.489 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.489 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.489 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:42:31.489 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.546773+0000", 2024-08-19T14:42:31.489 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.9641000000000002e-05 2024-08-19T14:42:31.489 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:42:31.489 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:42:31.489 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:42:31.489 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.489 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.490 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:163728 7.f 7:f3d138f8:::benchmark_data_smithi045_115842_object45568:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e667)", 2024-08-19T14:42:31.490 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:36:59.213851+0000", 2024-08-19T14:42:31.490 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 331.56863904800002, 2024-08-19T14:42:31.490 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.33300425, 2024-08-19T14:42:31.490 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:42:31.490 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:42:31.490 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:42:31.490 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:42:31.490 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:42:31.490 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:42:31.490 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 163728 2024-08-19T14:42:31.491 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.491 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:42:31.491 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.491 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:42:31.491 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.213851+0000", 2024-08-19T14:42:31.491 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:42:31.491 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.491 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.491 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:42:31.491 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.213851+0000", 2024-08-19T14:42:31.492 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:42:31.492 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.492 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.492 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:42:31.492 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.213853+0000", 2024-08-19T14:42:31.492 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.984e-06 2024-08-19T14:42:31.492 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.492 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.492 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:42:31.492 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.213888+0000", 2024-08-19T14:42:31.492 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.5466000000000002e-05 2024-08-19T14:42:31.492 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.493 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.493 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:42:31.493 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.213910+0000", 2024-08-19T14:42:31.493 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.1855e-05 2024-08-19T14:42:31.493 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.493 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.493 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:42:31.493 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.213920+0000", 2024-08-19T14:42:31.493 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.9150000000000008e-06 2024-08-19T14:42:31.493 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.493 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.493 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:42:31.494 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.343357+0000", 2024-08-19T14:42:31.494 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.129436843 2024-08-19T14:42:31.494 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.494 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.494 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:42:31.494 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.343470+0000", 2024-08-19T14:42:31.494 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000113356 2024-08-19T14:42:31.494 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.494 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.494 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 8,9", 2024-08-19T14:42:31.494 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.343598+0000", 2024-08-19T14:42:31.494 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012727800000000001 2024-08-19T14:42:31.495 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.495 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.495 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:42:31.495 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.349571+0000", 2024-08-19T14:42:31.495 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0059730740000000001 2024-08-19T14:42:31.495 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.496 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.496 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:42:31.496 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.349748+0000", 2024-08-19T14:42:31.496 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000177329 2024-08-19T14:42:31.496 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.496 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.496 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:42:31.496 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.546783+0000", 2024-08-19T14:42:31.496 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.19703465000000001 2024-08-19T14:42:31.496 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.496 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.496 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:42:31.497 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.546817+0000", 2024-08-19T14:42:31.497 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.4446999999999997e-05 2024-08-19T14:42:31.497 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.497 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.497 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:42:31.497 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.546855+0000", 2024-08-19T14:42:31.497 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.8053000000000003e-05 2024-08-19T14:42:31.497 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:42:31.497 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:42:31.497 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:42:31.498 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.498 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.498 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:163735 7.f 7:fba5fed5:::benchmark_data_smithi045_115842_object45575:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e667)", 2024-08-19T14:42:31.498 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:36:59.227314+0000", 2024-08-19T14:42:31.498 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 331.555175604, 2024-08-19T14:42:31.498 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.31962848100000002, 2024-08-19T14:42:31.498 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:42:31.498 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:42:31.498 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:42:31.499 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:42:31.499 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:42:31.499 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:42:31.499 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 163735 2024-08-19T14:42:31.499 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.499 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:42:31.499 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.499 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:42:31.499 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.227314+0000", 2024-08-19T14:42:31.499 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:42:31.499 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.499 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.500 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:42:31.500 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.227314+0000", 2024-08-19T14:42:31.500 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:42:31.500 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.500 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.500 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:42:31.500 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.227316+0000", 2024-08-19T14:42:31.500 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6130000000000001e-06 2024-08-19T14:42:31.500 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.500 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.500 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:42:31.500 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.227324+0000", 2024-08-19T14:42:31.500 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.2029999999999992e-06 2024-08-19T14:42:31.500 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.501 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.501 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:42:31.501 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.227340+0000", 2024-08-19T14:42:31.501 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5914e-05 2024-08-19T14:42:31.501 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.501 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.501 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:42:31.501 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.227347+0000", 2024-08-19T14:42:31.501 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.9310000000000002e-06 2024-08-19T14:42:31.501 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.501 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.501 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:42:31.501 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.349894+0000", 2024-08-19T14:42:31.502 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.122547172 2024-08-19T14:42:31.502 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.502 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.502 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:42:31.502 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.350230+0000", 2024-08-19T14:42:31.502 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00033566800000000001 2024-08-19T14:42:31.502 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.502 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.502 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 8,9", 2024-08-19T14:42:31.502 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.350604+0000", 2024-08-19T14:42:31.502 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00037375800000000002 2024-08-19T14:42:31.502 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.502 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.503 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:42:31.503 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.365945+0000", 2024-08-19T14:42:31.503 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.015341059000000001 2024-08-19T14:42:31.503 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.503 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.503 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:42:31.503 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.365989+0000", 2024-08-19T14:42:31.503 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.3773999999999998e-05 2024-08-19T14:42:31.503 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.503 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.503 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:42:31.503 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.546869+0000", 2024-08-19T14:42:31.504 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.180880397 2024-08-19T14:42:31.504 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.504 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.504 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:42:31.504 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.546905+0000", 2024-08-19T14:42:31.504 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.5914000000000002e-05 2024-08-19T14:42:31.504 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.504 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.504 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:42:31.504 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.546943+0000", 2024-08-19T14:42:31.504 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.8077999999999997e-05 2024-08-19T14:42:31.505 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:42:31.505 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:42:31.505 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:42:31.505 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.505 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.505 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:220612 7.4 7:2bb015c3:::benchmark_data_smithi045_115842_object102452:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e700)", 2024-08-19T14:42:31.505 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:39:06.800533+0000", 2024-08-19T14:42:31.505 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 203.981957311, 2024-08-19T14:42:31.505 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.33481899700000001, 2024-08-19T14:42:31.505 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:42:31.505 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:42:31.505 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:42:31.505 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:42:31.505 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:42:31.505 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:42:31.506 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 220612 2024-08-19T14:42:31.506 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.506 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:42:31.506 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.506 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:42:31.506 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.800533+0000", 2024-08-19T14:42:31.506 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:42:31.506 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.506 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.506 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:42:31.506 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.800533+0000", 2024-08-19T14:42:31.506 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:42:31.506 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.507 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.507 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:42:31.507 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.800534+0000", 2024-08-19T14:42:31.507 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.22e-06 2024-08-19T14:42:31.507 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.507 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.507 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:42:31.507 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.800541+0000", 2024-08-19T14:42:31.507 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.6760000000000001e-06 2024-08-19T14:42:31.507 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.507 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.507 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:42:31.507 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.800551+0000", 2024-08-19T14:42:31.507 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0584000000000001e-05 2024-08-19T14:42:31.508 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.508 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.508 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:42:31.508 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.800558+0000", 2024-08-19T14:42:31.508 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.233e-06 2024-08-19T14:42:31.508 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.508 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.508 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:42:31.508 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.800660+0000", 2024-08-19T14:42:31.508 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00010177499999999999 2024-08-19T14:42:31.508 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.508 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.509 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:42:31.509 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.800786+0000", 2024-08-19T14:42:31.509 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000125353 2024-08-19T14:42:31.509 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.509 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.509 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 1,4", 2024-08-19T14:42:31.509 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.800914+0000", 2024-08-19T14:42:31.509 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012890800000000001 2024-08-19T14:42:31.509 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.509 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.509 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:42:31.509 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.820713+0000", 2024-08-19T14:42:31.510 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.019798382999999999 2024-08-19T14:42:31.510 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.510 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.510 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:42:31.510 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:07.135163+0000", 2024-08-19T14:42:31.510 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.31445014900000001 2024-08-19T14:42:31.510 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.510 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.510 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:42:31.510 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:07.135264+0000", 2024-08-19T14:42:31.510 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000100667 2024-08-19T14:42:31.510 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.511 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.511 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:42:31.511 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:07.135310+0000", 2024-08-19T14:42:31.511 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.6817999999999998e-05 2024-08-19T14:42:31.511 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.511 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.511 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:42:31.511 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:07.135352+0000", 2024-08-19T14:42:31.511 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.1230999999999999e-05 2024-08-19T14:42:31.511 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:42:31.511 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:42:31.511 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:42:31.511 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.512 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.512 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:220613 7.4 7:248c3256:::benchmark_data_smithi045_115842_object102453:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e700)", 2024-08-19T14:42:31.512 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:39:06.800580+0000", 2024-08-19T14:42:31.512 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 203.98190991300001, 2024-08-19T14:42:31.512 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.334942236, 2024-08-19T14:42:31.512 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:42:31.512 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:42:31.512 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:42:31.512 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:42:31.512 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:42:31.512 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:42:31.512 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 220613 2024-08-19T14:42:31.512 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.512 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:42:31.513 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.513 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:42:31.513 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.800580+0000", 2024-08-19T14:42:31.513 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:42:31.513 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.513 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.513 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:42:31.513 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.800580+0000", 2024-08-19T14:42:31.513 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:42:31.513 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.513 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.514 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:42:31.514 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.800581+0000", 2024-08-19T14:42:31.514 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2500000000000001e-07 2024-08-19T14:42:31.514 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.514 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.514 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:42:31.514 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.800583+0000", 2024-08-19T14:42:31.514 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.6290000000000002e-06 2024-08-19T14:42:31.514 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.514 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.514 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:42:31.514 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.800590+0000", 2024-08-19T14:42:31.515 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.9280000000000004e-06 2024-08-19T14:42:31.515 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.515 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.515 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:42:31.515 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.800595+0000", 2024-08-19T14:42:31.515 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.079e-06 2024-08-19T14:42:31.515 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.515 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.515 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:42:31.515 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.806658+0000", 2024-08-19T14:42:31.515 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0060631569999999996 2024-08-19T14:42:31.515 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.516 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.516 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:42:31.516 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.806760+0000", 2024-08-19T14:42:31.516 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000101505 2024-08-19T14:42:31.516 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.516 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.516 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 1,4", 2024-08-19T14:42:31.516 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.806857+0000", 2024-08-19T14:42:31.516 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.6738999999999997e-05 2024-08-19T14:42:31.516 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.516 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.516 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:42:31.517 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.820757+0000", 2024-08-19T14:42:31.517 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.013900279999999999 2024-08-19T14:42:31.517 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.517 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.517 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:42:31.517 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:07.135383+0000", 2024-08-19T14:42:31.517 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.31462581899999997 2024-08-19T14:42:31.517 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.517 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.518 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:42:31.518 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:07.135425+0000", 2024-08-19T14:42:31.518 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2536000000000001e-05 2024-08-19T14:42:31.518 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.518 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.518 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:42:31.518 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:07.135490+0000", 2024-08-19T14:42:31.518 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.5147000000000005e-05 2024-08-19T14:42:31.518 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.518 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.518 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:42:31.519 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:07.135522+0000", 2024-08-19T14:42:31.519 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.1992000000000001e-05 2024-08-19T14:42:31.519 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:42:31.519 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:42:31.519 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:42:31.519 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.519 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.519 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:220618 7.4 7:289c7a94:::benchmark_data_smithi045_115842_object102458:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e700)", 2024-08-19T14:42:31.519 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:39:06.812624+0000", 2024-08-19T14:42:31.519 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 203.96986645600001, 2024-08-19T14:42:31.520 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.32299076100000002, 2024-08-19T14:42:31.520 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:42:31.520 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:42:31.520 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:42:31.520 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:42:31.520 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:42:31.520 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:42:31.520 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 220618 2024-08-19T14:42:31.520 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.520 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:42:31.520 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.521 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:42:31.521 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.812624+0000", 2024-08-19T14:42:31.521 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:42:31.521 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.521 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.521 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:42:31.521 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.812624+0000", 2024-08-19T14:42:31.521 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:42:31.521 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.521 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.521 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:42:31.522 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.812625+0000", 2024-08-19T14:42:31.522 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.7579999999999999e-06 2024-08-19T14:42:31.522 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.522 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.522 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:42:31.522 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.812632+0000", 2024-08-19T14:42:31.522 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.7580000000000003e-06 2024-08-19T14:42:31.522 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.522 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.522 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:42:31.522 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.812646+0000", 2024-08-19T14:42:31.522 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4157e-05 2024-08-19T14:42:31.522 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.523 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.523 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:42:31.523 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.812654+0000", 2024-08-19T14:42:31.523 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.542e-06 2024-08-19T14:42:31.523 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.523 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.523 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:42:31.523 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.820783+0000", 2024-08-19T14:42:31.523 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0081295340000000008 2024-08-19T14:42:31.523 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.523 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.523 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:42:31.523 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.820928+0000", 2024-08-19T14:42:31.523 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00014483899999999999 2024-08-19T14:42:31.524 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.524 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.524 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 1,4", 2024-08-19T14:42:31.524 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.821050+0000", 2024-08-19T14:42:31.524 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000121371 2024-08-19T14:42:31.524 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.524 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.524 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:42:31.524 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:07.135219+0000", 2024-08-19T14:42:31.524 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.31416936099999998 2024-08-19T14:42:31.524 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.524 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.524 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:42:31.525 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:07.135416+0000", 2024-08-19T14:42:31.525 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00019753300000000001 2024-08-19T14:42:31.525 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.525 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.525 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:42:31.525 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:07.135527+0000", 2024-08-19T14:42:31.525 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00011103900000000001 2024-08-19T14:42:31.525 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.525 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.525 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:42:31.525 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:07.135586+0000", 2024-08-19T14:42:31.526 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.8316000000000002e-05 2024-08-19T14:42:31.526 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.526 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.526 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:42:31.526 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:07.135614+0000", 2024-08-19T14:42:31.526 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.8552999999999999e-05 2024-08-19T14:42:31.526 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:42:31.526 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:42:31.526 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:42:31.526 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.526 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.526 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:220619 7.6 7:65071c03:::benchmark_data_smithi045_115842_object102459:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e700)", 2024-08-19T14:42:31.526 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:39:06.813741+0000", 2024-08-19T14:42:31.527 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 203.96874948600001, 2024-08-19T14:42:31.527 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.33041151699999999, 2024-08-19T14:42:31.527 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:42:31.527 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:42:31.527 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:42:31.527 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:42:31.527 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:42:31.527 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:42:31.527 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 220619 2024-08-19T14:42:31.527 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.527 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:42:31.527 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.527 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:42:31.527 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.813741+0000", 2024-08-19T14:42:31.528 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:42:31.528 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.528 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.528 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:42:31.528 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.813741+0000", 2024-08-19T14:42:31.528 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:42:31.528 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.528 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.528 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:42:31.528 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.813742+0000", 2024-08-19T14:42:31.528 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.04e-06 2024-08-19T14:42:31.528 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.528 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.528 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:42:31.529 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.813746+0000", 2024-08-19T14:42:31.529 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7199999999999997e-06 2024-08-19T14:42:31.529 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.529 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.529 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:42:31.529 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.813757+0000", 2024-08-19T14:42:31.529 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0397999999999999e-05 2024-08-19T14:42:31.529 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.529 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.529 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:42:31.529 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.813764+0000", 2024-08-19T14:42:31.530 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.4100000000000002e-06 2024-08-19T14:42:31.530 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.530 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.530 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:42:31.530 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.816627+0000", 2024-08-19T14:42:31.531 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0028632319999999998 2024-08-19T14:42:31.531 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.531 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.531 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:42:31.531 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.816735+0000", 2024-08-19T14:42:31.531 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000107346 2024-08-19T14:42:31.531 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.532 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.532 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,7", 2024-08-19T14:42:31.532 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.816845+0000", 2024-08-19T14:42:31.532 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000110503 2024-08-19T14:42:31.532 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.532 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.533 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:42:31.533 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:07.143968+0000", 2024-08-19T14:42:31.533 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.32712271799999998 2024-08-19T14:42:31.533 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.533 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.533 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:42:31.533 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:07.144006+0000", 2024-08-19T14:42:31.534 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.8560999999999999e-05 2024-08-19T14:42:31.534 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.534 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.534 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:42:31.534 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:07.144043+0000", 2024-08-19T14:42:31.534 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.6284e-05 2024-08-19T14:42:31.534 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.535 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.535 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:42:31.535 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:07.144102+0000", 2024-08-19T14:42:31.535 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.9067999999999998e-05 2024-08-19T14:42:31.535 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.536 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.536 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:42:31.536 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:07.144152+0000", 2024-08-19T14:42:31.536 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0237e-05 2024-08-19T14:42:31.536 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:42:31.536 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:42:31.537 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:42:31.537 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.537 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.537 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:220621 7.6 7:6e439cf0:::benchmark_data_smithi045_115842_object102461:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e700)", 2024-08-19T14:42:31.537 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:39:06.819474+0000", 2024-08-19T14:42:31.537 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 203.96301620700001, 2024-08-19T14:42:31.538 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.34391892600000001, 2024-08-19T14:42:31.538 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:42:31.538 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:42:31.538 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:42:31.538 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:42:31.539 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:42:31.539 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:42:31.539 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 220621 2024-08-19T14:42:31.539 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.539 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:42:31.539 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.539 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:42:31.539 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.819474+0000", 2024-08-19T14:42:31.540 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:42:31.540 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.540 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.540 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:42:31.540 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.819474+0000", 2024-08-19T14:42:31.540 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:42:31.540 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.541 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.541 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:42:31.541 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.819476+0000", 2024-08-19T14:42:31.541 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.9290000000000001e-06 2024-08-19T14:42:31.541 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.541 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.541 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:42:31.541 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.819482+0000", 2024-08-19T14:42:31.541 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.0920000000000003e-06 2024-08-19T14:42:31.541 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.541 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.541 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:42:31.541 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.819499+0000", 2024-08-19T14:42:31.541 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6688999999999999e-05 2024-08-19T14:42:31.542 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.542 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.542 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:42:31.542 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.819508+0000", 2024-08-19T14:42:31.542 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.5510000000000005e-06 2024-08-19T14:42:31.542 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.542 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.542 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:42:31.542 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.821834+0000", 2024-08-19T14:42:31.542 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0023259280000000001 2024-08-19T14:42:31.542 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.542 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.542 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:42:31.543 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:07.134834+0000", 2024-08-19T14:42:31.543 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.31300021900000002 2024-08-19T14:42:31.543 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.543 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.543 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,7", 2024-08-19T14:42:31.543 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:07.139030+0000", 2024-08-19T14:42:31.543 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.004196137 2024-08-19T14:42:31.543 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.543 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.543 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:42:31.543 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:07.144192+0000", 2024-08-19T14:42:31.543 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0051618660000000002 2024-08-19T14:42:31.543 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.543 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.544 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:42:31.544 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:07.144203+0000", 2024-08-19T14:42:31.544 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.117e-05 2024-08-19T14:42:31.544 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.544 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.544 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:42:31.544 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:07.163158+0000", 2024-08-19T14:42:31.544 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.018954649 2024-08-19T14:42:31.544 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.544 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.544 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:42:31.544 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:07.163258+0000", 2024-08-19T14:42:31.544 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.9543000000000006e-05 2024-08-19T14:42:31.544 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.545 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.545 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:42:31.545 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:07.163393+0000", 2024-08-19T14:42:31.545 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000135153 2024-08-19T14:42:31.545 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:42:31.545 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:42:31.545 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:42:31.545 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.545 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.545 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.16962.0:220620 7.d e700/600 7:b15dd3f1:::benchmark_data_smithi045_115842_object102460:head v 700'13832, mlcod=700'13829)", 2024-08-19T14:42:31.545 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:39:06.821415+0000", 2024-08-19T14:42:31.545 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 203.96107522400001, 2024-08-19T14:42:31.545 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.31966576000000002, 2024-08-19T14:42:31.546 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:42:31.546 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:42:31.546 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:42:31.546 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:42:31.546 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.546 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:42:31.546 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.821415+0000", 2024-08-19T14:42:31.546 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:42:31.546 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.546 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.546 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:42:31.547 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.821415+0000", 2024-08-19T14:42:31.547 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:42:31.547 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.547 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.547 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:42:31.547 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.821416+0000", 2024-08-19T14:42:31.547 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.9800000000000003e-07 2024-08-19T14:42:31.547 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.547 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.547 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:42:31.547 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.821421+0000", 2024-08-19T14:42:31.547 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.3859999999999998e-06 2024-08-19T14:42:31.547 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.547 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.547 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:42:31.548 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.821430+0000", 2024-08-19T14:42:31.548 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.0650000000000005e-06 2024-08-19T14:42:31.548 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.548 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.548 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:42:31.548 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.821437+0000", 2024-08-19T14:42:31.548 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.9040000000000003e-06 2024-08-19T14:42:31.548 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.548 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.548 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:42:31.548 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.821504+0000", 2024-08-19T14:42:31.548 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.7256999999999997e-05 2024-08-19T14:42:31.548 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.549 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.549 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:42:31.549 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.821522+0000", 2024-08-19T14:42:31.549 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.8218000000000001e-05 2024-08-19T14:42:31.549 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.549 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.549 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:42:31.549 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:07.141002+0000", 2024-08-19T14:42:31.549 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.31947991100000001 2024-08-19T14:42:31.549 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.549 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.549 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:42:31.549 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:07.141081+0000", 2024-08-19T14:42:31.549 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.8220999999999995e-05 2024-08-19T14:42:31.550 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:42:31.550 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:42:31.550 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:42:31.550 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.550 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.550 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:220624 7.f 7:f8cdd8b1:::benchmark_data_smithi045_115842_object102464:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e700)", 2024-08-19T14:42:31.550 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:39:06.827029+0000", 2024-08-19T14:42:31.550 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 203.95546138899999, 2024-08-19T14:42:31.550 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.32098705700000002, 2024-08-19T14:42:31.550 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:42:31.551 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:42:31.551 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:42:31.551 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:42:31.551 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:42:31.551 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:42:31.551 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 220624 2024-08-19T14:42:31.551 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.551 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:42:31.551 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.551 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:42:31.551 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.827029+0000", 2024-08-19T14:42:31.552 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:42:31.552 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.552 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.552 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:42:31.552 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.827029+0000", 2024-08-19T14:42:31.552 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:42:31.552 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.552 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.552 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:42:31.552 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.827030+0000", 2024-08-19T14:42:31.552 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.694e-06 2024-08-19T14:42:31.552 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.552 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.552 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:42:31.553 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.827040+0000", 2024-08-19T14:42:31.553 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.3009999999999995e-06 2024-08-19T14:42:31.553 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.553 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.553 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:42:31.553 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:07.134952+0000", 2024-08-19T14:42:31.553 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.30791282599999997 2024-08-19T14:42:31.553 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.553 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.553 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:42:31.553 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:07.134963+0000", 2024-08-19T14:42:31.553 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0482999999999999e-05 2024-08-19T14:42:31.553 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.554 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.554 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:42:31.554 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:07.135025+0000", 2024-08-19T14:42:31.554 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.2508999999999994e-05 2024-08-19T14:42:31.554 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.554 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.554 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:42:31.554 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:07.135155+0000", 2024-08-19T14:42:31.554 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012912700000000001 2024-08-19T14:42:31.554 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.554 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.554 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 8,9", 2024-08-19T14:42:31.554 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:07.135911+0000", 2024-08-19T14:42:31.554 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000756878 2024-08-19T14:42:31.555 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.555 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.555 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:42:31.555 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:07.147742+0000", 2024-08-19T14:42:31.555 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.011830235999999999 2024-08-19T14:42:31.555 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.555 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.555 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:42:31.555 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:07.147783+0000", 2024-08-19T14:42:31.555 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.1764999999999997e-05 2024-08-19T14:42:31.555 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.555 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.555 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:42:31.556 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:07.147908+0000", 2024-08-19T14:42:31.556 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012414899999999999 2024-08-19T14:42:31.556 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.556 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.556 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:42:31.556 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:07.147970+0000", 2024-08-19T14:42:31.556 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.2646999999999998e-05 2024-08-19T14:42:31.556 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.556 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.556 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:42:31.556 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:07.148016+0000", 2024-08-19T14:42:31.556 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.5442000000000003e-05 2024-08-19T14:42:31.556 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:42:31.556 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:42:31.556 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:42:31.557 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.557 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.557 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:232591 7.6 7:6076b785:::benchmark_data_smithi045_115842_object114431:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e707)", 2024-08-19T14:42:31.557 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:39:29.424695+0000", 2024-08-19T14:42:31.557 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 181.35779523299999, 2024-08-19T14:42:31.557 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.35295967, 2024-08-19T14:42:31.557 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:42:31.557 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:42:31.557 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:42:31.557 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:42:31.557 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:42:31.557 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:42:31.557 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 232591 2024-08-19T14:42:31.557 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.558 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:42:31.558 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.558 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:42:31.558 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.424695+0000", 2024-08-19T14:42:31.558 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:42:31.558 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.558 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.558 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:42:31.558 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.424695+0000", 2024-08-19T14:42:31.558 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:42:31.558 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.558 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.559 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:42:31.559 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.424697+0000", 2024-08-19T14:42:31.559 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.73e-06 2024-08-19T14:42:31.559 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.559 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.559 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:42:31.559 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.424705+0000", 2024-08-19T14:42:31.559 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.1389999999999995e-06 2024-08-19T14:42:31.559 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.559 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.559 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:42:31.559 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.424721+0000", 2024-08-19T14:42:31.559 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6158999999999999e-05 2024-08-19T14:42:31.560 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.560 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.560 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:42:31.560 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.424730+0000", 2024-08-19T14:42:31.560 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.6770000000000007e-06 2024-08-19T14:42:31.560 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.560 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.560 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:42:31.560 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.424797+0000", 2024-08-19T14:42:31.560 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.6987000000000001e-05 2024-08-19T14:42:31.560 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.560 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.560 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:42:31.560 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.424928+0000", 2024-08-19T14:42:31.561 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013101399999999999 2024-08-19T14:42:31.561 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.561 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.561 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,7", 2024-08-19T14:42:31.561 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.425048+0000", 2024-08-19T14:42:31.561 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00011969899999999999 2024-08-19T14:42:31.561 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.561 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.561 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:42:31.561 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.431710+0000", 2024-08-19T14:42:31.562 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0066618270000000004 2024-08-19T14:42:31.562 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.562 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.562 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:42:31.562 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.431963+0000", 2024-08-19T14:42:31.562 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00025306500000000002 2024-08-19T14:42:31.562 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.562 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.562 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:42:31.562 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.777326+0000", 2024-08-19T14:42:31.562 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.34536338500000002 2024-08-19T14:42:31.562 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.563 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.563 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:42:31.563 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.777524+0000", 2024-08-19T14:42:31.563 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00019711899999999999 2024-08-19T14:42:31.563 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.563 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.563 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:42:31.563 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.777654+0000", 2024-08-19T14:42:31.563 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013086900000000001 2024-08-19T14:42:31.563 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:42:31.563 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:42:31.564 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:42:31.564 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.564 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.564 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:232592 7.6 7:6148bd1d:::benchmark_data_smithi045_115842_object114432:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e707)", 2024-08-19T14:42:31.564 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:39:29.434124+0000", 2024-08-19T14:42:31.564 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 181.348365949, 2024-08-19T14:42:31.564 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.34459850199999997, 2024-08-19T14:42:31.564 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:42:31.564 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:42:31.564 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:42:31.564 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:42:31.564 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:42:31.565 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:42:31.565 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 232592 2024-08-19T14:42:31.565 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.565 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:42:31.565 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.565 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:42:31.565 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.434124+0000", 2024-08-19T14:42:31.565 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:42:31.565 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.565 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.565 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:42:31.565 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.434124+0000", 2024-08-19T14:42:31.565 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:42:31.566 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.566 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.566 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:42:31.566 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.434126+0000", 2024-08-19T14:42:31.566 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.984e-06 2024-08-19T14:42:31.566 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.566 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.566 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:42:31.566 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.434134+0000", 2024-08-19T14:42:31.566 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.6899999999999992e-06 2024-08-19T14:42:31.566 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.566 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.567 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:42:31.567 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.434150+0000", 2024-08-19T14:42:31.567 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6526999999999999e-05 2024-08-19T14:42:31.567 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.567 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.567 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:42:31.567 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.434160+0000", 2024-08-19T14:42:31.567 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.6199999999999994e-06 2024-08-19T14:42:31.567 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.567 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.567 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:42:31.567 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.434286+0000", 2024-08-19T14:42:31.567 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000126191 2024-08-19T14:42:31.568 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.568 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.568 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:42:31.568 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.434411+0000", 2024-08-19T14:42:31.569 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012469400000000001 2024-08-19T14:42:31.569 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.569 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.569 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,7", 2024-08-19T14:42:31.569 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.434536+0000", 2024-08-19T14:42:31.569 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012499599999999999 2024-08-19T14:42:31.569 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.569 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.569 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:42:31.570 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.445685+0000", 2024-08-19T14:42:31.570 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.011149133 2024-08-19T14:42:31.570 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.570 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.570 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:42:31.570 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.445745+0000", 2024-08-19T14:42:31.570 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.0584000000000001e-05 2024-08-19T14:42:31.570 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.570 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.571 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:42:31.571 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.778598+0000", 2024-08-19T14:42:31.571 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.332852698 2024-08-19T14:42:31.571 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.571 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.571 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:42:31.571 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.778682+0000", 2024-08-19T14:42:31.571 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.3382999999999998e-05 2024-08-19T14:42:31.571 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.571 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.571 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:42:31.571 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.778723+0000", 2024-08-19T14:42:31.571 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.1001999999999998e-05 2024-08-19T14:42:31.571 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:42:31.572 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:42:31.572 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:42:31.572 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.572 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.572 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:232594 7.6 7:69e63e5e:::benchmark_data_smithi045_115842_object114434:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e707)", 2024-08-19T14:42:31.572 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:39:29.434195+0000", 2024-08-19T14:42:31.572 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 181.348295479, 2024-08-19T14:42:31.572 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.34467441900000001, 2024-08-19T14:42:31.572 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:42:31.572 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:42:31.572 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:42:31.572 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:42:31.573 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:42:31.573 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:42:31.573 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 232594 2024-08-19T14:42:31.573 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.573 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:42:31.573 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.573 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:42:31.573 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.434195+0000", 2024-08-19T14:42:31.573 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:42:31.573 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.573 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.573 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:42:31.573 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.434195+0000", 2024-08-19T14:42:31.573 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:42:31.574 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.574 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.574 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:42:31.574 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.434195+0000", 2024-08-19T14:42:31.574 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.9200000000000001e-07 2024-08-19T14:42:31.574 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.574 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.574 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:42:31.574 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.434198+0000", 2024-08-19T14:42:31.574 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.3170000000000002e-06 2024-08-19T14:42:31.574 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.574 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.574 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:42:31.574 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.434207+0000", 2024-08-19T14:42:31.575 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.1969999999999996e-06 2024-08-19T14:42:31.575 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.575 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.575 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:42:31.575 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.434213+0000", 2024-08-19T14:42:31.575 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.9970000000000004e-06 2024-08-19T14:42:31.575 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.575 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.575 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:42:31.575 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.440124+0000", 2024-08-19T14:42:31.575 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0059109899999999996 2024-08-19T14:42:31.575 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.575 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.576 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:42:31.576 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.440241+0000", 2024-08-19T14:42:31.576 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000117793 2024-08-19T14:42:31.576 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.576 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.576 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,7", 2024-08-19T14:42:31.576 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.440354+0000", 2024-08-19T14:42:31.576 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000113087 2024-08-19T14:42:31.576 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.576 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.576 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:42:31.576 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.445733+0000", 2024-08-19T14:42:31.576 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0053782320000000002 2024-08-19T14:42:31.576 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.577 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.577 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:42:31.577 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.445751+0000", 2024-08-19T14:42:31.577 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.8042000000000001e-05 2024-08-19T14:42:31.577 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.577 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.577 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:42:31.577 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.778759+0000", 2024-08-19T14:42:31.577 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.33300808399999998 2024-08-19T14:42:31.577 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.577 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.577 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:42:31.577 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.778830+0000", 2024-08-19T14:42:31.578 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.0961e-05 2024-08-19T14:42:31.578 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.578 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.578 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:42:31.578 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.778869+0000", 2024-08-19T14:42:31.578 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.9227000000000002e-05 2024-08-19T14:42:31.578 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:42:31.578 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:42:31.578 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:42:31.578 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.578 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.578 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26911.0:70283 8.9 8.b4edcc9 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e736)", 2024-08-19T14:42:31.579 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:40:48.417481+0000", 2024-08-19T14:42:31.579 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 102.365009466, 2024-08-19T14:42:31.579 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.028042347, 2024-08-19T14:42:31.579 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:42:31.579 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:42:31.579 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:42:31.579 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:42:31.579 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26911", 2024-08-19T14:42:31.579 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3494993995", 2024-08-19T14:42:31.579 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 70283 2024-08-19T14:42:31.579 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.580 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:42:31.580 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.580 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:42:31.580 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:40:48.417481+0000", 2024-08-19T14:42:31.580 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:42:31.580 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.580 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.580 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:42:31.580 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:40:48.417481+0000", 2024-08-19T14:42:31.580 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:42:31.581 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.581 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.581 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:42:31.581 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:40:48.417481+0000", 2024-08-19T14:42:31.581 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.5199999999999997e-07 2024-08-19T14:42:31.581 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.581 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.581 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:42:31.581 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:40:48.417549+0000", 2024-08-19T14:42:31.581 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.7627999999999997e-05 2024-08-19T14:42:31.581 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.581 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.581 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:42:31.582 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:40:48.417557+0000", 2024-08-19T14:42:31.582 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.6130000000000004e-06 2024-08-19T14:42:31.582 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.582 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.582 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:42:31.582 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:40:48.417564+0000", 2024-08-19T14:42:31.582 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.3049999999999998e-06 2024-08-19T14:42:31.582 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.582 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.582 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:42:31.582 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:40:48.429627+0000", 2024-08-19T14:42:31.582 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.012062941000000001 2024-08-19T14:42:31.582 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.583 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.583 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:42:31.583 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:40:48.429668+0000", 2024-08-19T14:42:31.583 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.0948999999999997e-05 2024-08-19T14:42:31.583 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.583 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.583 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:42:31.583 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:40:49.445455+0000", 2024-08-19T14:42:31.583 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0157872729999999 2024-08-19T14:42:31.583 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.583 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.583 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:42:31.584 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:40:49.445523+0000", 2024-08-19T14:42:31.584 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.8085999999999998e-05 2024-08-19T14:42:31.584 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:42:31.584 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:42:31.584 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:42:31.584 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.584 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.584 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26911.0:70293 8.9 8.189a5e49 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e736)", 2024-08-19T14:42:31.584 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:40:48.428534+0000", 2024-08-19T14:42:31.584 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 102.35395621000001, 2024-08-19T14:42:31.584 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0170277210000001, 2024-08-19T14:42:31.584 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:42:31.584 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:42:31.584 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:42:31.585 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:42:31.585 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26911", 2024-08-19T14:42:31.585 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3494993995", 2024-08-19T14:42:31.585 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 70293 2024-08-19T14:42:31.585 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.585 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:42:31.585 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.585 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:42:31.585 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:40:48.428534+0000", 2024-08-19T14:42:31.585 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:42:31.585 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.585 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.585 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:42:31.586 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:40:48.428534+0000", 2024-08-19T14:42:31.586 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:42:31.586 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.586 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.586 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:42:31.586 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:40:48.428534+0000", 2024-08-19T14:42:31.586 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7999999999999996e-07 2024-08-19T14:42:31.586 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.586 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.586 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:42:31.586 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:40:48.428592+0000", 2024-08-19T14:42:31.586 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.7955999999999999e-05 2024-08-19T14:42:31.587 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.587 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.587 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:42:31.587 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:40:48.428600+0000", 2024-08-19T14:42:31.587 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.072e-06 2024-08-19T14:42:31.587 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.587 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.587 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:42:31.587 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:40:48.428606+0000", 2024-08-19T14:42:31.587 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.9529999999999996e-06 2024-08-19T14:42:31.587 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.587 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.587 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:42:31.588 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:40:48.429966+0000", 2024-08-19T14:42:31.588 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.001359751 2024-08-19T14:42:31.588 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.588 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.588 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:42:31.588 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:40:48.430156+0000", 2024-08-19T14:42:31.588 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000190034 2024-08-19T14:42:31.588 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.588 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.588 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:42:31.588 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:40:49.445554+0000", 2024-08-19T14:42:31.589 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0153980250000001 2024-08-19T14:42:31.589 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.589 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.589 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:42:31.589 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:40:49.445562+0000", 2024-08-19T14:42:31.589 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.4499999999999998e-06 2024-08-19T14:42:31.589 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:42:31.589 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:42:31.589 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:42:31.589 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.590 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.590 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26911.0:70295 8.9 8.6d05a669 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e736)", 2024-08-19T14:42:31.590 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:40:48.440426+0000", 2024-08-19T14:42:31.590 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 102.342064244, 2024-08-19T14:42:31.590 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0051761340000001, 2024-08-19T14:42:31.590 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:42:31.590 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:42:31.590 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:42:31.591 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:42:31.591 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26911", 2024-08-19T14:42:31.591 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3494993995", 2024-08-19T14:42:31.591 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 70295 2024-08-19T14:42:31.591 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.591 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:42:31.591 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.591 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:42:31.591 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:40:48.440426+0000", 2024-08-19T14:42:31.591 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:42:31.591 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.591 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.592 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:42:31.592 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:40:48.440426+0000", 2024-08-19T14:42:31.592 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:42:31.592 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.592 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.592 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:42:31.592 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:40:48.440427+0000", 2024-08-19T14:42:31.592 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5409999999999999e-06 2024-08-19T14:42:31.592 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.592 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.592 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:42:31.592 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:40:48.440467+0000", 2024-08-19T14:42:31.592 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.9473e-05 2024-08-19T14:42:31.593 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.593 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.593 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:42:31.593 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:40:48.440486+0000", 2024-08-19T14:42:31.593 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.9219999999999999e-05 2024-08-19T14:42:31.593 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.593 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.593 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:42:31.593 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:40:48.440494+0000", 2024-08-19T14:42:31.593 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.0239999999999997e-06 2024-08-19T14:42:31.593 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.593 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.593 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:42:31.593 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:40:48.441288+0000", 2024-08-19T14:42:31.594 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00079417799999999998 2024-08-19T14:42:31.594 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.594 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.594 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:42:31.594 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:40:48.441300+0000", 2024-08-19T14:42:31.594 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2167999999999999e-05 2024-08-19T14:42:31.594 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.594 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.594 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:42:31.594 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:40:49.445595+0000", 2024-08-19T14:42:31.595 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0042943070000001 2024-08-19T14:42:31.595 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.595 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.595 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:42:31.595 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:40:49.445602+0000", 2024-08-19T14:42:31.595 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.2230000000000001e-06 2024-08-19T14:42:31.595 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:42:31.595 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:42:31.595 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:42:31.595 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.595 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.595 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26911.0:70302 8.9 8.666f8769 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e736)", 2024-08-19T14:42:31.595 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:40:48.463513+0000", 2024-08-19T14:42:31.595 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 102.318977137, 2024-08-19T14:42:31.596 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.98211852200000005, 2024-08-19T14:42:31.596 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:42:31.596 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:42:31.596 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:42:31.596 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:42:31.596 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26911", 2024-08-19T14:42:31.596 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3494993995", 2024-08-19T14:42:31.596 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 70302 2024-08-19T14:42:31.596 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.596 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:42:31.596 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.597 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:42:31.597 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:40:48.463513+0000", 2024-08-19T14:42:31.597 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:42:31.597 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.597 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.597 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:42:31.597 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:40:48.463513+0000", 2024-08-19T14:42:31.597 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:42:31.597 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.597 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.597 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:42:31.597 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:40:48.463514+0000", 2024-08-19T14:42:31.597 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.004e-06 2024-08-19T14:42:31.597 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.598 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.598 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:42:31.598 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:40:48.463544+0000", 2024-08-19T14:42:31.598 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.04e-05 2024-08-19T14:42:31.598 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.598 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.598 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:42:31.598 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:40:48.463557+0000", 2024-08-19T14:42:31.598 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2952000000000001e-05 2024-08-19T14:42:31.598 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.598 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.599 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:42:31.599 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:40:48.463565+0000", 2024-08-19T14:42:31.599 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.3300000000000001e-06 2024-08-19T14:42:31.599 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.599 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.599 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:42:31.599 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:40:48.463637+0000", 2024-08-19T14:42:31.599 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.2792000000000004e-05 2024-08-19T14:42:31.599 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.599 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.599 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:42:31.599 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:40:48.463649+0000", 2024-08-19T14:42:31.599 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1593999999999999e-05 2024-08-19T14:42:31.599 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.600 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.600 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:42:31.600 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:40:49.445626+0000", 2024-08-19T14:42:31.600 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.98197673699999999 2024-08-19T14:42:31.600 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:42:31.600 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:42:31.600 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:42:31.600 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:40:49.445631+0000", 2024-08-19T14:42:31.601 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.7130000000000002e-06 2024-08-19T14:42:31.601 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:42:31.601 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:42:31.601 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:42:31.601 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:42:31.601 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:42:31.601 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:42:31.602 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:42:31.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:31.639 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:31.639 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:42:31.740 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:42:31.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:31.771 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:31.771 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:42:31.872 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:42:31.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:31.906 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:31.906 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:42:32.007 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:42:32.041 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:32.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:32.042 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:42:32.098 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is unset 2024-08-19T14:42:32.143 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:42:32.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:32.174 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:32.174 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:42:32.275 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:42:32.305 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:32.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:32.307 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:42:32.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:32 smithi031 ceph-mon[16336]: pgmap v1851: 57 pgs: 57 active+clean; 4.1 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 0 B/s wr, 791 op/s 2024-08-19T14:42:32.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:32 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/4029055519' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:42:32.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:32 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:42:32.407 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:42:32.439 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:32.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:32.440 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:42:32.488 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:42:32.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:32 smithi032 ceph-mon[20004]: pgmap v1851: 57 pgs: 57 active+clean; 4.1 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 0 B/s wr, 791 op/s 2024-08-19T14:42:32.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:32 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/4029055519' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:42:32.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:32 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:42:32.541 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:42:32.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:32 smithi045 ceph-mon[20143]: pgmap v1851: 57 pgs: 57 active+clean; 4.1 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 0 B/s wr, 791 op/s 2024-08-19T14:42:32.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:32 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/4029055519' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:42:32.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:32 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:42:32.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:32.579 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:32.579 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:42:32.680 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:42:32.713 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:32.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:32.714 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:42:32.815 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:42:32.847 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:32.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:32.848 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:42:32.949 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:42:32.983 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:32.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:32.984 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:42:33.085 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:42:33.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:33.114 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:33.114 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:42:33.215 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:42:33.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:33.241 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:33.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:42:33.342 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:42:33.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:33 smithi031 ceph-mon[16336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:42:33.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:33 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:42:33.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:33 smithi031 ceph-mon[16336]: osdmap e774: 12 total, 12 up, 9 in 2024-08-19T14:42:33.370 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:33.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:33.372 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:42:33.473 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:42:33.504 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:33.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:33.505 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:42:33.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:33 smithi032 ceph-mon[20004]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:42:33.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:33 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:42:33.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:33 smithi032 ceph-mon[20004]: osdmap e774: 12 total, 12 up, 9 in 2024-08-19T14:42:33.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:33 smithi045 ceph-mon[20143]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:42:33.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:33 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:42:33.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:33 smithi045 ceph-mon[20143]: osdmap e774: 12 total, 12 up, 9 in 2024-08-19T14:42:33.606 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:42:33.649 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:33.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:33.650 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:42:33.751 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:42:33.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:33.784 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:33.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:42:33.885 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:42:33.913 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:33.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:33.914 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:42:33.996 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-08-19T14:42:34.015 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:42:34.073 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:34.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:34.075 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:42:34.176 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:42:34.210 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:34.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:34.211 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:42:34.253 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:34 smithi031 ceph-mon[16336]: pgmap v1853: 57 pgs: 57 active+clean; 3.8 GiB data, 20 GiB used, 784 GiB / 805 GiB avail; 0 B/s wr, 1.48k op/s 2024-08-19T14:42:34.312 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:42:34.336 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:34.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:34.337 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:42:34.438 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:42:34.465 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:34.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:34.466 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:42:34.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:34 smithi032 ceph-mon[20004]: pgmap v1853: 57 pgs: 57 active+clean; 3.8 GiB data, 20 GiB used, 784 GiB / 805 GiB avail; 0 B/s wr, 1.48k op/s 2024-08-19T14:42:34.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:34 smithi045 ceph-mon[20143]: pgmap v1853: 57 pgs: 57 active+clean; 3.8 GiB data, 20 GiB used, 784 GiB / 805 GiB avail; 0 B/s wr, 1.48k op/s 2024-08-19T14:42:34.567 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:42:34.605 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:42:34.628 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:34.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:34.630 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:42:34.730 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:42:34.756 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:34.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:34.758 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:42:34.859 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:42:34.905 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:34.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:34.906 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:42:35.007 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:42:35.041 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-19T14:42:35.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:35.046 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:35.046 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:42:35.068 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:34 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mon-a[16332]: 2024-08-19T14:42:34.900+0000 7f2ff503e640 -1 mon.a@0(leader).osd e774 definitely_dead 0 2024-08-19T14:42:35.121 INFO:teuthology.orchestra.run.smithi031.stderr:marked out osd.0. 2024-08-19T14:42:35.147 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:42:35.170 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:35.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:35.172 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:42:35.237 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:42:35.237 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-08-19T14:42:35.237 INFO:teuthology.orchestra.run.smithi031.stdout: "num_ops": 0 2024-08-19T14:42:35.237 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:42:35.273 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:42:35.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:35.296 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:35.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:42:35.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:35 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:42:35.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:35 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:42:35.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:35 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2987799653' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-08-19T14:42:35.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:35 smithi031 ceph-mon[16336]: Client client.admin marked osd.0 out, while it was still marked up 2024-08-19T14:42:35.397 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:42:35.418 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:35.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:35.420 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:42:35.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:35 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:42:35.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:35 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:42:35.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:35 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2987799653' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-08-19T14:42:35.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:35 smithi032 ceph-mon[20004]: Client client.admin marked osd.0 out, while it was still marked up 2024-08-19T14:42:35.520 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:42:35.537 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:42:35.542 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:35.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:35.543 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:42:35.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:35 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:42:35.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:35 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:42:35.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:35 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2987799653' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-08-19T14:42:35.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:35 smithi045 ceph-mon[20143]: Client client.admin marked osd.0 out, while it was still marked up 2024-08-19T14:42:35.643 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:42:35.664 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:35.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:35.665 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:42:35.766 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:42:35.792 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:35.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:35.793 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:42:35.894 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:42:35.915 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:35.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:35.916 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:42:36.017 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-08-19T14:42:36.018 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:42:36.041 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:36.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:36.042 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:42:36.143 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:42:36.165 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:36.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:36.166 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:42:36.267 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:42:36.292 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:36.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:36.294 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:42:36.394 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:42:36.419 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:36.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:36.420 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:42:36.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:36 smithi032 ceph-mon[20004]: pgmap v1854: 57 pgs: 57 active+clean; 3.7 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 1.25k op/s 2024-08-19T14:42:36.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:36 smithi032 ceph-mon[20004]: 8.b scrub starts 2024-08-19T14:42:36.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:36 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2987799653' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["0"]}]': finished 2024-08-19T14:42:36.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:36 smithi032 ceph-mon[20004]: osdmap e775: 12 total, 12 up, 8 in 2024-08-19T14:42:36.521 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:42:36.544 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:36.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:36.545 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:42:36.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:36 smithi045 ceph-mon[20143]: pgmap v1854: 57 pgs: 57 active+clean; 3.7 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 1.25k op/s 2024-08-19T14:42:36.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:36 smithi045 ceph-mon[20143]: 8.b scrub starts 2024-08-19T14:42:36.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:36 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2987799653' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["0"]}]': finished 2024-08-19T14:42:36.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:36 smithi045 ceph-mon[20143]: osdmap e775: 12 total, 12 up, 8 in 2024-08-19T14:42:36.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:36 smithi031 ceph-mon[16336]: pgmap v1854: 57 pgs: 57 active+clean; 3.7 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 1.25k op/s 2024-08-19T14:42:36.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:36 smithi031 ceph-mon[16336]: 8.b scrub starts 2024-08-19T14:42:36.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:36 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2987799653' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["0"]}]': finished 2024-08-19T14:42:36.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:36 smithi031 ceph-mon[16336]: osdmap e775: 12 total, 12 up, 8 in 2024-08-19T14:42:36.646 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:42:36.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:36.673 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:36.673 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:42:36.774 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:42:36.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:36.800 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:36.801 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:42:36.901 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:42:36.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:36.928 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:36.928 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:42:37.029 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:42:37.054 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:37.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:37.055 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:42:37.156 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:42:37.188 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:37.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:37.189 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:42:37.290 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:42:37.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:37.323 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:37.323 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:42:37.424 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:42:37.448 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:37.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:37.449 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:42:37.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:37 smithi032 ceph-mon[20004]: 8.b scrub ok 2024-08-19T14:42:37.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:37 smithi032 ceph-mon[20004]: osdmap e776: 12 total, 12 up, 8 in 2024-08-19T14:42:37.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:37 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:42:37.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:37 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:42:37.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:37 smithi045 ceph-mon[20143]: 8.b scrub ok 2024-08-19T14:42:37.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:37 smithi045 ceph-mon[20143]: osdmap e776: 12 total, 12 up, 8 in 2024-08-19T14:42:37.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:37 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:42:37.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:37 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:42:37.550 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:42:37.576 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:37.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:37.577 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:42:37.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:37 smithi031 ceph-mon[16336]: 8.b scrub ok 2024-08-19T14:42:37.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:37 smithi031 ceph-mon[16336]: osdmap e776: 12 total, 12 up, 8 in 2024-08-19T14:42:37.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:37 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:42:37.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:37 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:42:37.678 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:42:37.706 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:37.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:37.708 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:42:37.809 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:42:37.845 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:37.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:37.846 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:42:37.947 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:42:37.976 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:37.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:37.977 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:42:38.078 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:42:38.104 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:38.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:38.105 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:42:38.206 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:42:38.226 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:42:38.232 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:38.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:38.234 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:42:38.335 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:42:38.342 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:42:38.362 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:38.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:38.363 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:42:38.464 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:42:38.496 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:38.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:38.497 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:42:38.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:38 smithi032 ceph-mon[20004]: pgmap v1857: 57 pgs: 4 peering, 3 unknown, 50 active+clean; 3.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 22 KiB/s wr, 1.51k op/s; 16 KiB/s, 0 objects/s recovering 2024-08-19T14:42:38.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:38 smithi032 ceph-mon[20004]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-08-19T14:42:38.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:38 smithi032 ceph-mon[20004]: 8.e starting backfill to osd.7 from (0'0,0'0] MAX to 774'10598 2024-08-19T14:42:38.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:38 smithi032 ceph-mon[20004]: osdmap e777: 12 total, 12 up, 8 in 2024-08-19T14:42:38.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:38 smithi032 ceph-mon[20004]: 8.f scrub starts 2024-08-19T14:42:38.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:38 smithi032 ceph-mon[20004]: osdmap e778: 12 total, 12 up, 8 in 2024-08-19T14:42:38.510 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:38 smithi031 ceph-mon[16336]: pgmap v1857: 57 pgs: 4 peering, 3 unknown, 50 active+clean; 3.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 22 KiB/s wr, 1.51k op/s; 16 KiB/s, 0 objects/s recovering 2024-08-19T14:42:38.510 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:38 smithi031 ceph-mon[16336]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-08-19T14:42:38.510 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:38 smithi031 ceph-mon[16336]: 8.e starting backfill to osd.7 from (0'0,0'0] MAX to 774'10598 2024-08-19T14:42:38.510 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:38 smithi031 ceph-mon[16336]: osdmap e777: 12 total, 12 up, 8 in 2024-08-19T14:42:38.511 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:38 smithi031 ceph-mon[16336]: 8.f scrub starts 2024-08-19T14:42:38.511 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:38 smithi031 ceph-mon[16336]: osdmap e778: 12 total, 12 up, 8 in 2024-08-19T14:42:38.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:38 smithi045 ceph-mon[20143]: pgmap v1857: 57 pgs: 4 peering, 3 unknown, 50 active+clean; 3.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 22 KiB/s wr, 1.51k op/s; 16 KiB/s, 0 objects/s recovering 2024-08-19T14:42:38.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:38 smithi045 ceph-mon[20143]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-08-19T14:42:38.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:38 smithi045 ceph-mon[20143]: 8.e starting backfill to osd.7 from (0'0,0'0] MAX to 774'10598 2024-08-19T14:42:38.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:38 smithi045 ceph-mon[20143]: osdmap e777: 12 total, 12 up, 8 in 2024-08-19T14:42:38.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:38 smithi045 ceph-mon[20143]: 8.f scrub starts 2024-08-19T14:42:38.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:38 smithi045 ceph-mon[20143]: osdmap e778: 12 total, 12 up, 8 in 2024-08-19T14:42:38.598 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:42:38.625 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:38.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:38.626 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:42:38.727 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:42:38.754 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:38.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:38.755 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:42:38.856 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:42:38.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:38.892 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:38.892 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:42:38.994 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:42:39.033 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:39.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:39.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:42:39.136 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:42:39.164 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:39.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:39.166 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:42:39.267 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:42:39.296 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:39.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:39.298 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:42:39.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:39 smithi031 ceph-mon[16336]: 8.f scrub ok 2024-08-19T14:42:39.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:39 smithi031 ceph-mon[16336]: osdmap e779: 12 total, 12 up, 8 in 2024-08-19T14:42:39.399 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:42:39.432 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:39.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:39.433 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:42:39.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:39 smithi032 ceph-mon[20004]: 8.f scrub ok 2024-08-19T14:42:39.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:39 smithi032 ceph-mon[20004]: osdmap e779: 12 total, 12 up, 8 in 2024-08-19T14:42:39.533 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:42:39.534 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:42:39.538 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:42:39.538 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-08-19T14:42:39.538 INFO:teuthology.orchestra.run.smithi031.stdout: "complaint_time": 30, 2024-08-19T14:42:39.539 INFO:teuthology.orchestra.run.smithi031.stdout: "num_blocked_ops": 0 2024-08-19T14:42:39.539 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:42:39.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:39 smithi045 ceph-mon[20143]: 8.f scrub ok 2024-08-19T14:42:39.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:39 smithi045 ceph-mon[20143]: osdmap e779: 12 total, 12 up, 8 in 2024-08-19T14:42:39.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:39.583 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:39.583 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:42:39.684 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:42:39.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:39.717 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:39.718 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:42:39.819 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:42:39.845 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:39.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:39.846 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:42:39.947 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:42:39.974 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:39.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:39.975 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:42:40.076 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:42:40.102 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:40.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:40.103 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:42:40.154 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is unset 2024-08-19T14:42:40.204 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:42:40.233 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:40.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:40.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:42:40.312 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:40 smithi031 ceph-mon[16336]: pgmap v1860: 57 pgs: 2 active+undersized+remapped, 4 peering, 3 unknown, 48 active+clean; 3.5 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 33 KiB/s wr, 858 op/s; 23 KiB/s, 0 objects/s recovering 2024-08-19T14:42:40.312 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:40 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2738230994' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:42:40.312 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:40 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:42:40.312 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:40 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:42:40.335 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:42:40.364 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:40.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:40.365 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:42:40.466 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:42:40.493 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:40.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:40.495 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:42:40.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:40 smithi032 ceph-mon[20004]: pgmap v1860: 57 pgs: 2 active+undersized+remapped, 4 peering, 3 unknown, 48 active+clean; 3.5 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 33 KiB/s wr, 858 op/s; 23 KiB/s, 0 objects/s recovering 2024-08-19T14:42:40.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:40 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2738230994' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:42:40.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:40 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:42:40.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:40 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:42:40.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:40 smithi045 ceph-mon[20143]: pgmap v1860: 57 pgs: 2 active+undersized+remapped, 4 peering, 3 unknown, 48 active+clean; 3.5 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 33 KiB/s wr, 858 op/s; 23 KiB/s, 0 objects/s recovering 2024-08-19T14:42:40.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:40 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2738230994' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:42:40.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:40 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:42:40.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:40 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:42:40.595 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:42:40.626 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:40.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:40.627 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:42:40.728 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:42:40.729 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:42:40.730 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:42:40.758 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:42:40.758 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:42:40.759 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:40.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:40.760 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:42:40.787 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:42:40.787 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:42:40.810 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:42:40.810 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:42:40.840 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:42:40.840 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:42:40.860 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:42:40.872 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:42:40.872 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:42:40.896 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:40.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:40.897 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:42:40.897 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:42:40.898 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:42:40.997 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:42:41.044 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:41.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:41.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:42:41.144 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:42:41.145 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:42:41.146 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:42:41.175 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:42:41.176 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:42:41.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:41.180 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:41.180 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:42:41.202 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:42:41.202 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:42:41.228 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:42:41.228 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:42:41.255 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:42:41.255 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:42:41.281 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:42:41.307 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:41.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:41.308 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:42:41.409 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:42:41.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:41.451 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:41.451 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:42:41.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:41 smithi032 ceph-mon[20004]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:42:41.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:41 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2738230994' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:42:41.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:41 smithi032 ceph-mon[20004]: osdmap e780: 12 total, 12 up, 8 in 2024-08-19T14:42:41.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:41 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/4230882311' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:42:41.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:41 smithi045 ceph-mon[20143]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:42:41.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:41 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2738230994' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:42:41.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:41 smithi045 ceph-mon[20143]: osdmap e780: 12 total, 12 up, 8 in 2024-08-19T14:42:41.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:41 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/4230882311' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:42:41.552 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:42:41.576 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:41.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:41.577 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:42:41.583 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:41 smithi031 ceph-mon[16336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:42:41.584 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:41 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2738230994' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:42:41.584 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:41 smithi031 ceph-mon[16336]: osdmap e780: 12 total, 12 up, 8 in 2024-08-19T14:42:41.584 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:41 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/4230882311' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:42:41.678 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:42:41.721 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:41.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:41.722 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:42:41.823 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:42:41.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:41.858 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:41.858 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:42:41.959 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:42:41.987 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:41.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:41.988 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:42:42.067 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-08-19T14:42:42.089 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:42:42.144 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:42.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:42.145 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:42:42.246 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:42:42.281 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:42.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:42.282 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:42:42.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:42 smithi031 ceph-mon[16336]: pgmap v1863: 57 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+remapped, 8 peering, 3 unknown, 41 active+clean; 3.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 665 op/s; 10/167736 objects degraded (0.006%); 6747/167736 objects misplaced (4.022%); 3.9 MiB/s, 60 objects/s recovering 2024-08-19T14:42:42.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:42 smithi031 ceph-mon[16336]: Health check failed: Degraded data redundancy: 10/167736 objects degraded (0.006%), 2 pgs degraded (PG_DEGRADED) 2024-08-19T14:42:42.383 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:42:42.411 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:42.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:42.412 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:42:42.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:42 smithi032 ceph-mon[20004]: pgmap v1863: 57 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+remapped, 8 peering, 3 unknown, 41 active+clean; 3.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 665 op/s; 10/167736 objects degraded (0.006%); 6747/167736 objects misplaced (4.022%); 3.9 MiB/s, 60 objects/s recovering 2024-08-19T14:42:42.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:42 smithi032 ceph-mon[20004]: Health check failed: Degraded data redundancy: 10/167736 objects degraded (0.006%), 2 pgs degraded (PG_DEGRADED) 2024-08-19T14:42:42.513 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:42:42.541 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:42.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:42.542 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:42:42.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:42 smithi045 ceph-mon[20143]: pgmap v1863: 57 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+remapped, 8 peering, 3 unknown, 41 active+clean; 3.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 665 op/s; 10/167736 objects degraded (0.006%); 6747/167736 objects misplaced (4.022%); 3.9 MiB/s, 60 objects/s recovering 2024-08-19T14:42:42.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:42 smithi045 ceph-mon[20143]: Health check failed: Degraded data redundancy: 10/167736 objects degraded (0.006%), 2 pgs degraded (PG_DEGRADED) 2024-08-19T14:42:42.643 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:42:42.670 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:42.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:42.671 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:42:42.767 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:42:42.772 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:42:42.801 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:42.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:42.802 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:42:42.903 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:42:42.929 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:42.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:42.930 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:42:43.031 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:42:43.064 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:43.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:43.067 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:42:43.168 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:42:43.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:43.199 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:43.200 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:42:43.235 INFO:tasks.thrashosds.thrasher:in_osds: [1, 4, 5, 7, 8, 2, 11, 9] out_osds: [10, 6, 3, 0] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 8, 9, 11, 10, 2] 2024-08-19T14:42:43.235 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-19T14:42:43.236 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-08-19T14:42:43.236 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-19T14:42:43.301 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:42:43.316 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:42:43.316 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:42:43.347 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:43.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:43.348 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:42:43.449 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:42:43.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:43.481 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:43.481 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:42:43.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:43 smithi032 ceph-mon[20004]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-08-19T14:42:43.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:43 smithi045 ceph-mon[20143]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-08-19T14:42:43.582 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:42:43.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:43.611 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:43.611 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:42:43.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:43 smithi031 ceph-mon[16336]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-08-19T14:42:43.662 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-19T14:42:43.711 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:42:43.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:43.746 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:43.746 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:42:43.847 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:42:43.888 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:43.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:43.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:42:43.990 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:42:44.013 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-08-19T14:42:44.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:44.044 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:44.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:42:44.073 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:42:44.083 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:42:44.145 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:42:44.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:44.178 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:44.178 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:42:44.204 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:44 smithi031 ceph-mon[16336]: pgmap v1864: 57 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 3 active+recovering+undersized+degraded+remapped, 4 peering, 48 active+clean; 3.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 1.09k op/s; 114/158679 objects degraded (0.072%); 9863/158679 objects misplaced (6.216%); 7.3 MiB/s, 117 objects/s recovering 2024-08-19T14:42:44.279 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:42:44.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:44.304 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:44.304 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:42:44.405 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:42:44.435 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:44.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:44.436 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:42:44.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:44 smithi032 ceph-mon[20004]: pgmap v1864: 57 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 3 active+recovering+undersized+degraded+remapped, 4 peering, 48 active+clean; 3.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 1.09k op/s; 114/158679 objects degraded (0.072%); 9863/158679 objects misplaced (6.216%); 7.3 MiB/s, 117 objects/s recovering 2024-08-19T14:42:44.537 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:42:44.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:44 smithi045 ceph-mon[20143]: pgmap v1864: 57 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 3 active+recovering+undersized+degraded+remapped, 4 peering, 48 active+clean; 3.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 1.09k op/s; 114/158679 objects degraded (0.072%); 9863/158679 objects misplaced (6.216%); 7.3 MiB/s, 117 objects/s recovering 2024-08-19T14:42:44.564 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:44.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:44.566 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:42:44.653 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:42:44.666 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:42:44.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:44.745 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:44.746 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:42:44.847 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:42:44.900 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:44.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:44.901 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:42:45.002 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:42:45.051 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:45.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:45.052 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:42:45.134 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:42:45.137 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:42:45.153 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:42:45.161 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:42:45.162 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-08-19T14:42:45.184 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:45.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:45.185 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:42:45.286 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:42:45.314 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:45.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:45.315 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:42:45.416 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:42:45.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:45.443 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:45.443 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:42:45.544 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:42:45.574 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:45.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:45.575 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:42:45.676 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:42:45.703 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:45.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:45.704 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:42:45.805 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:42:45.837 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:45.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:45.838 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:42:45.939 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:42:45.960 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:45.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:45.961 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:42:46.062 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:42:46.085 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:46.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:46.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:42:46.187 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:42:46.208 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is set 2024-08-19T14:42:46.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:46.215 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:46.215 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:42:46.317 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:42:46.360 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:46.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:46.361 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:42:46.462 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:42:46.487 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:46.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:46.488 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:42:46.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:46 smithi032 ceph-mon[20004]: pgmap v1865: 57 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 52 active+clean; 3.1 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 975 op/s; 114/153762 objects degraded (0.074%); 8601/153762 objects misplaced (5.594%); 16 MiB/s, 250 objects/s recovering 2024-08-19T14:42:46.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:46 smithi032 ceph-mon[20004]: from='client.17568 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:42:46.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:46 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/4120536872' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:42:46.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:46 smithi045 ceph-mon[20143]: pgmap v1865: 57 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 52 active+clean; 3.1 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 975 op/s; 114/153762 objects degraded (0.074%); 8601/153762 objects misplaced (5.594%); 16 MiB/s, 250 objects/s recovering 2024-08-19T14:42:46.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:46 smithi045 ceph-mon[20143]: from='client.17568 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:42:46.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:46 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/4120536872' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:42:46.589 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:42:46.614 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:46.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:46.616 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:42:46.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:46 smithi031 ceph-mon[16336]: pgmap v1865: 57 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 52 active+clean; 3.1 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 975 op/s; 114/153762 objects degraded (0.074%); 8601/153762 objects misplaced (5.594%); 16 MiB/s, 250 objects/s recovering 2024-08-19T14:42:46.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:46 smithi031 ceph-mon[16336]: from='client.17568 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:42:46.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:46 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/4120536872' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:42:46.716 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:42:46.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:46.753 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:46.753 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:42:46.854 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:42:46.881 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:46.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:46.882 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:42:46.983 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:42:47.009 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:47.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:47.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:42:47.111 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:42:47.138 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:47.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:47.139 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:42:47.240 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:42:47.278 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:47.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:47.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:42:47.380 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:42:47.418 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:47.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:47.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:42:47.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:47 smithi032 ceph-mon[20004]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:42:47.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:47 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/4120536872' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:42:47.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:47 smithi032 ceph-mon[20004]: osdmap e781: 12 total, 12 up, 8 in 2024-08-19T14:42:47.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:47 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:42:47.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:47 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:42:47.521 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:42:47.545 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:47.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:47.546 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:42:47.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:47 smithi045 ceph-mon[20143]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:42:47.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:47 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/4120536872' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:42:47.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:47 smithi045 ceph-mon[20143]: osdmap e781: 12 total, 12 up, 8 in 2024-08-19T14:42:47.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:47 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:42:47.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:47 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:42:47.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:47 smithi031 ceph-mon[16336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:42:47.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:47 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/4120536872' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:42:47.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:47 smithi031 ceph-mon[16336]: osdmap e781: 12 total, 12 up, 8 in 2024-08-19T14:42:47.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:47 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:42:47.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:47 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:42:47.647 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:42:47.674 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:47.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:47.675 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:42:47.776 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:42:47.809 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:47.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:47.810 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:42:47.911 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:42:48.066 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:48.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:48.067 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:42:48.168 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:42:48.205 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:48.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:48.206 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:42:48.307 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:42:48.334 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:48.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:48.335 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:42:48.436 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:42:48.480 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:48.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:48.481 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:42:48.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:48 smithi032 ceph-mon[20004]: pgmap v1867: 57 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 3 active+recovering+undersized+degraded+remapped, 52 active+clean; 3.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 913 op/s; 272/147822 objects degraded (0.184%); 7974/147822 objects misplaced (5.394%); 15 MiB/s, 249 objects/s recovering 2024-08-19T14:42:48.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:48 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 272/147822 objects degraded (0.184%), 3 pgs degraded (PG_DEGRADED) 2024-08-19T14:42:48.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:48 smithi032 ceph-mon[20004]: osdmap e782: 12 total, 12 up, 8 in 2024-08-19T14:42:48.536 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:48 smithi031 ceph-mon[16336]: pgmap v1867: 57 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 3 active+recovering+undersized+degraded+remapped, 52 active+clean; 3.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 913 op/s; 272/147822 objects degraded (0.184%); 7974/147822 objects misplaced (5.394%); 15 MiB/s, 249 objects/s recovering 2024-08-19T14:42:48.536 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:48 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 272/147822 objects degraded (0.184%), 3 pgs degraded (PG_DEGRADED) 2024-08-19T14:42:48.536 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:48 smithi031 ceph-mon[16336]: osdmap e782: 12 total, 12 up, 8 in 2024-08-19T14:42:48.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:48 smithi045 ceph-mon[20143]: pgmap v1867: 57 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 3 active+recovering+undersized+degraded+remapped, 52 active+clean; 3.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 913 op/s; 272/147822 objects degraded (0.184%); 7974/147822 objects misplaced (5.394%); 15 MiB/s, 249 objects/s recovering 2024-08-19T14:42:48.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:48 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 272/147822 objects degraded (0.184%), 3 pgs degraded (PG_DEGRADED) 2024-08-19T14:42:48.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:48 smithi045 ceph-mon[20143]: osdmap e782: 12 total, 12 up, 8 in 2024-08-19T14:42:48.582 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:42:48.615 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:48.616 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:42:48.616 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:42:48.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:48.617 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:42:48.649 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:42:48.649 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:42:48.677 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:42:48.677 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:42:48.707 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:42:48.707 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:42:48.718 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:42:48.753 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:48.753 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:42:48.754 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:42:48.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:48.755 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:42:48.795 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:42:48.795 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:42:48.829 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:42:48.829 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:42:48.856 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:42:48.881 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:42:48.881 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:42:48.895 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:48.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:48.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:42:48.906 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:42:48.906 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:42:48.931 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:42:48.932 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:42:48.960 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:42:48.961 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:42:48.988 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:42:48.989 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:42:48.997 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:42:49.039 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:49.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:49.040 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:42:49.141 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:42:49.188 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:49.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:49.189 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:42:49.290 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:42:49.322 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:49.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:49.323 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:42:49.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:49 smithi031 ceph-mon[16336]: from='client.17577 172.21.15.31:0/475164173' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:42:49.424 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:42:49.464 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:49.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:49.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:42:49.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:49 smithi032 ceph-mon[20004]: from='client.17577 172.21.15.31:0/475164173' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:42:49.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:49 smithi045 ceph-mon[20143]: from='client.17577 172.21.15.31:0/475164173' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:42:49.565 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:42:49.596 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:49.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:49.597 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:42:49.698 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:42:49.731 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:49.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:49.732 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:42:49.833 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:42:49.858 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:49.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:49.860 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:42:49.925 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:42:49.925 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-08-19T14:42:49.925 INFO:teuthology.orchestra.run.smithi031.stdout: "num_ops": 0 2024-08-19T14:42:49.925 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:42:49.961 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:42:49.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:49.994 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:49.994 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:42:50.095 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:42:50.138 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:50.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:50.139 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:42:50.240 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:42:50.268 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:50.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:50.270 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:42:50.370 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:42:50.400 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:50.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:50.401 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:42:50.502 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:42:50.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:50 smithi032 ceph-mon[20004]: pgmap v1869: 57 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 3 active+recovering+undersized+degraded+remapped, 52 active+clean; 2.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s; 272/143238 objects degraded (0.190%); 7974/143238 objects misplaced (5.567%); 14 MiB/s, 226 objects/s recovering 2024-08-19T14:42:50.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:50 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:42:50.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:50 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:42:50.540 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:50.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:50.541 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:42:50.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:50 smithi045 ceph-mon[20143]: pgmap v1869: 57 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 3 active+recovering+undersized+degraded+remapped, 52 active+clean; 2.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s; 272/143238 objects degraded (0.190%); 7974/143238 objects misplaced (5.567%); 14 MiB/s, 226 objects/s recovering 2024-08-19T14:42:50.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:50 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:42:50.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:50 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:42:50.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:50 smithi031 ceph-mon[16336]: pgmap v1869: 57 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 3 active+recovering+undersized+degraded+remapped, 52 active+clean; 2.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s; 272/143238 objects degraded (0.190%); 7974/143238 objects misplaced (5.567%); 14 MiB/s, 226 objects/s recovering 2024-08-19T14:42:50.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:50 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:42:50.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:50 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:42:50.642 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:42:50.673 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:50.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:50.674 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:42:50.775 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:42:50.803 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:50.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:50.804 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:42:50.905 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:42:50.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:50.944 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:50.944 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:42:51.038 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-08-19T14:42:51.045 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:42:51.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:51.123 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:51.123 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:42:51.139 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":1865,"stamp":"2024-08-19T14:42:44.326311+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3359337227,"num_objects":51254,"num_object_clones":0,"num_object_copies":153762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":114,"num_objects_misplaced":8601,"num_objects_unfound":0,"num_objects_dirty":51254,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":105805,"num_write_kb":3055175,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8282,"num_bytes_recovered":543066569,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":222812,"ondisk_log_size":222812,"up":171,"acting":168,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":176,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":17469468,"kb_used_data":9529244,"kb_used_omap":15,"kb_used_meta":7938544,"kb_avail":732491748,"statfs":{"total":767960285184,"available":750071549952,"internally_reserved":0,"allocated":9757945856,"data_stored":9735722801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15572,"internal_metadata":8129069868},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,2,4,4,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-400752640,"num_objects":-6115,"num_object_clones":0,"num_object_copies":-18345,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":114,"num_objects_misplaced":8601,"num_objects_unfound":0,"num_objects_dirty":-6115,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6115,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1568,"num_bytes_recovered":102891520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.271043"},"pg_stats":[{"pgid":"8.16","version":"780'7418","reported_seq":11898,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-08-19T14:42:42.177442+0000","last_change":"2024-08-19T14:41:48.539318+0000","last_active":"2024-08-19T14:42:42.177442+0000","last_peered":"2024-08-19T14:42:42.177442+0000","last_clean":"2024-08-19T14:42:42.177442+0000","last_became_active":"2024-08-19T14:41:00.558578+0000","last_became_peered":"2024-08-19T14:41:00.558578+0000","last_unstale":"2024-08-19T14:42:42.177442+0000","last_undegraded":"2024-08-19T14:42:42.177442+0000","last_fullsized":"2024-08-19T14:42:42.177442+0000","mapping_epoch":743,"log_start":"0'0","ondisk_log_start":"0'0","created":731,"last_epoch_clean":744,"parent":"0.0","parent_split_bits":5,"last_scrub":"766'5665","last_scrub_stamp":"2024-08-19T14:41:48.539024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:41:48.539024+0000","objects_scrubbed":3281,"log_size":7418,"log_dups_size":0,"ondisk_log_size":7418,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:42:53.278875+0000","scrub_duration":640,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":99287040,"num_objects":1515,"num_object_clones":0,"num_object_copies":4545,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1515,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1766,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.9","version":"780'10191","reported_seq":15281,"reported_epoch":780,"state":"active+recovering+remapped","last_fresh":"2024-08-19T14:42:41.042853+0000","last_change":"2024-08-19T14:42:37.406834+0000","last_active":"2024-08-19T14:42:41.042853+0000","last_peered":"2024-08-19T14:42:41.042853+0000","last_clean":"2024-08-19T14:42:35.097858+0000","last_became_active":"2024-08-19T14:42:37.376450+0000","last_became_peered":"2024-08-19T14:42:37.376450+0000","last_unstale":"2024-08-19T14:42:41.042853+0000","last_undegraded":"2024-08-19T14:42:41.042853+0000","last_fullsized":"2024-08-19T14:42:41.042853+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":717,"last_epoch_clean":754,"parent":"0.0","parent_split_bits":0,"last_scrub":"766'8581","last_scrub_stamp":"2024-08-19T14:41:49.579113+0000","last_deep_scrub":"766'8581","last_deep_scrub_stamp":"2024-08-19T14:41:49.579113+0000","last_clean_scrub_stamp":"2024-08-19T14:41:49.579113+0000","objects_scrubbed":3491,"log_size":10191,"log_dups_size":0,"ondisk_log_size":10191,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":10,"scrub_schedule":"no scrub is scheduled","scrub_duration":9452,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":111345664,"num_objects":1699,"num_object_clones":0,"num_object_copies":5097,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1610,"num_objects_unfound":0,"num_objects_dirty":1699,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1794,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":13303808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,9],"acting":[0,5,9],"avail_no_missing":["0","5","9"],"object_location_counts":[{"shards":"0,5,9","objects":89}],"blocked_by":[],"up_primary":4,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"70'139","reported_seq":1900,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-08-19T14:42:40.162304+0000","last_change":"2024-08-19T14:42:39.076379+0000","last_active":"2024-08-19T14:42:40.162304+0000","last_peered":"2024-08-19T14:42:40.162304+0000","last_clean":"2024-08-19T14:42:40.162304+0000","last_became_active":"2024-08-19T14:42:39.075903+0000","last_became_peered":"2024-08-19T14:42:39.075903+0000","last_unstale":"2024-08-19T14:42:40.162304+0000","last_undegraded":"2024-08-19T14:42:40.162304+0000","last_fullsized":"2024-08-19T14:42:40.162304+0000","mapping_epoch":778,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":779,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:41:45.177828+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:41:45.177828+0000","last_clean_scrub_stamp":"2024-08-19T14:41:45.177828+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:43:03.749002+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,9],"acting":[7,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"780'10617","reported_seq":10819,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-08-19T14:42:40.480236+0000","last_change":"2024-08-19T14:42:06.047732+0000","last_active":"2024-08-19T14:42:40.480236+0000","last_peered":"2024-08-19T14:42:40.480236+0000","last_clean":"2024-08-19T14:42:40.480236+0000","last_became_active":"2024-08-19T14:40:04.441586+0000","last_became_peered":"2024-08-19T14:40:04.441586+0000","last_unstale":"2024-08-19T14:42:40.480236+0000","last_undegraded":"2024-08-19T14:42:40.480236+0000","last_fullsized":"2024-08-19T14:42:40.480236+0000","mapping_epoch":717,"log_start":"722'600","ondisk_log_start":"722'600","created":717,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"768'8519","last_scrub_stamp":"2024-08-19T14:42:06.047506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:42:06.047506+0000","objects_scrubbed":5454,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:43:34.547025+0000","scrub_duration":1427,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217513984,"num_objects":3319,"num_object_clones":0,"num_object_copies":9957,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10617,"num_write_kb":445952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,9],"acting":[8,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"70'139","reported_seq":1741,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-08-19T14:42:40.158800+0000","last_change":"2024-08-19T14:41:45.936228+0000","last_active":"2024-08-19T14:42:40.158800+0000","last_peered":"2024-08-19T14:42:40.158800+0000","last_clean":"2024-08-19T14:42:40.158800+0000","last_became_active":"2024-08-19T14:18:13.804721+0000","last_became_peered":"2024-08-19T14:18:13.804721+0000","last_unstale":"2024-08-19T14:42:40.158800+0000","last_undegraded":"2024-08-19T14:42:40.158800+0000","last_fullsized":"2024-08-19T14:42:40.158800+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:41:45.936114+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:41:45.936114+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:43:06.573254+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.b","version":"780'12403","reported_seq":12650,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-08-19T14:42:40.445329+0000","last_change":"2024-08-19T14:42:35.439902+0000","last_active":"2024-08-19T14:42:40.445329+0000","last_peered":"2024-08-19T14:42:40.445329+0000","last_clean":"2024-08-19T14:42:40.445329+0000","last_became_active":"2024-08-19T14:40:04.200299+0000","last_became_peered":"2024-08-19T14:40:04.200299+0000","last_unstale":"2024-08-19T14:42:40.445329+0000","last_undegraded":"2024-08-19T14:42:40.445329+0000","last_fullsized":"2024-08-19T14:42:40.445329+0000","mapping_epoch":717,"log_start":"723'2400","ondisk_log_start":"723'2400","created":717,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"775'12195","last_scrub_stamp":"2024-08-19T14:42:35.439765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:42:35.439765+0000","objects_scrubbed":3468,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:43:54.357189+0000","scrub_duration":622,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213450752,"num_objects":3257,"num_object_clones":0,"num_object_copies":9771,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3257,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10729,"num_write_kb":447552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,1],"acting":[8,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.2","version":"70'139","reported_seq":1698,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-08-19T14:42:40.164313+0000","last_change":"2024-08-19T14:42:39.099060+0000","last_active":"2024-08-19T14:42:40.164313+0000","last_peered":"2024-08-19T14:42:40.164313+0000","last_clean":"2024-08-19T14:42:40.164313+0000","last_became_active":"2024-08-19T14:42:39.098732+0000","last_became_peered":"2024-08-19T14:42:39.098732+0000","last_unstale":"2024-08-19T14:42:40.164313+0000","last_undegraded":"2024-08-19T14:42:40.164313+0000","last_fullsized":"2024-08-19T14:42:40.164313+0000","mapping_epoch":778,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":779,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:40:10.977156+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:19:59.302931+0000","last_clean_scrub_stamp":"2024-08-19T14:40:10.977156+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,8],"acting":[7,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.8","version":"780'7682","reported_seq":12291,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-08-19T14:42:42.200741+0000","last_change":"2024-08-19T14:41:45.465065+0000","last_active":"2024-08-19T14:42:42.200741+0000","last_peered":"2024-08-19T14:42:42.200741+0000","last_clean":"2024-08-19T14:42:42.200741+0000","last_became_active":"2024-08-19T14:41:44.018646+0000","last_became_peered":"2024-08-19T14:41:44.018646+0000","last_unstale":"2024-08-19T14:42:42.200741+0000","last_undegraded":"2024-08-19T14:42:42.200741+0000","last_fullsized":"2024-08-19T14:42:42.200741+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":717,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":0,"last_scrub":"765'5810","last_scrub_stamp":"2024-08-19T14:41:45.464802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:41:45.464802+0000","objects_scrubbed":3565,"log_size":7682,"log_dups_size":0,"ondisk_log_size":7682,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:42:57.311689+0000","scrub_duration":595,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110297088,"num_objects":1683,"num_object_clones":0,"num_object_copies":5049,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1683,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1882,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,9],"acting":[11,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1","version":"70'139","reported_seq":999,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-08-19T14:42:40.161014+0000","last_change":"2024-08-19T14:42:39.065514+0000","last_active":"2024-08-19T14:42:40.161014+0000","last_peered":"2024-08-19T14:42:40.161014+0000","last_clean":"2024-08-19T14:42:40.161014+0000","last_became_active":"2024-08-19T14:42:39.065160+0000","last_became_peered":"2024-08-19T14:42:39.065160+0000","last_unstale":"2024-08-19T14:42:40.161014+0000","last_undegraded":"2024-08-19T14:42:40.161014+0000","last_fullsized":"2024-08-19T14:42:40.161014+0000","mapping_epoch":778,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":779,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:41:43.141372+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:41:43.141372+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:43:05.283028+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,2],"acting":[11,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.e","version":"780'11049","reported_seq":12545,"reported_epoch":780,"state":"active+remapped+backfilling","last_fresh":"2024-08-19T14:42:43.881815+0000","last_change":"2024-08-19T14:42:37.444131+0000","last_active":"2024-08-19T14:42:43.881815+0000","last_peered":"2024-08-19T14:42:43.881815+0000","last_clean":"2024-08-19T14:42:35.077699+0000","last_became_active":"2024-08-19T14:42:37.350538+0000","last_became_peered":"2024-08-19T14:42:37.350538+0000","last_unstale":"2024-08-19T14:42:43.881815+0000","last_undegraded":"2024-08-19T14:42:43.881815+0000","last_fullsized":"2024-08-19T14:42:43.881815+0000","mapping_epoch":776,"log_start":"722'1000","ondisk_log_start":"722'1000","created":717,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"766'7447","last_scrub_stamp":"2024-08-19T14:41:47.953594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:41:47.953594+0000","objects_scrubbed":6806,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1829,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":207159296,"num_objects":3161,"num_object_clones":0,"num_object_copies":9483,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2556,"num_objects_unfound":0,"num_objects_dirty":3161,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11049,"num_write_kb":454720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":637,"num_bytes_recovered":41746432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,0],"avail_no_missing":["4","0","2"],"object_location_counts":[{"shards":"0,2,4","objects":3161}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"70'139","reported_seq":951,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-08-19T14:42:40.156292+0000","last_change":"2024-08-19T14:41:35.981885+0000","last_active":"2024-08-19T14:42:40.156292+0000","last_peered":"2024-08-19T14:42:40.156292+0000","last_clean":"2024-08-19T14:42:40.156292+0000","last_became_active":"2024-08-19T14:40:52.490194+0000","last_became_peered":"2024-08-19T14:40:52.490194+0000","last_unstale":"2024-08-19T14:42:40.156292+0000","last_undegraded":"2024-08-19T14:42:40.156292+0000","last_fullsized":"2024-08-19T14:42:40.156292+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:41:35.981666+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:21:46.057624+0000","last_clean_scrub_stamp":"2024-08-19T14:41:35.981666+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:42:53.801491+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.f","version":"780'14722","reported_seq":15073,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-08-19T14:42:42.236350+0000","last_change":"2024-08-19T14:42:38.271767+0000","last_active":"2024-08-19T14:42:42.236350+0000","last_peered":"2024-08-19T14:42:42.236350+0000","last_clean":"2024-08-19T14:42:42.236350+0000","last_became_active":"2024-08-19T14:40:04.299656+0000","last_became_peered":"2024-08-19T14:40:04.299656+0000","last_unstale":"2024-08-19T14:42:42.236350+0000","last_undegraded":"2024-08-19T14:42:42.236350+0000","last_fullsized":"2024-08-19T14:42:42.236350+0000","mapping_epoch":717,"log_start":"740'4701","ondisk_log_start":"740'4701","created":717,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"778'14456","last_scrub_stamp":"2024-08-19T14:42:38.271577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:42:38.271577+0000","objects_scrubbed":3467,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:44:04.947656+0000","scrub_duration":861,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":208338944,"num_objects":3179,"num_object_clones":0,"num_object_copies":9537,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10925,"num_write_kb":451328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,1],"acting":[11,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.6","version":"70'139","reported_seq":967,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-08-19T14:42:40.158528+0000","last_change":"2024-08-19T14:41:55.664761+0000","last_active":"2024-08-19T14:42:40.158528+0000","last_peered":"2024-08-19T14:42:40.158528+0000","last_clean":"2024-08-19T14:42:40.158528+0000","last_became_active":"2024-08-19T14:34:15.717054+0000","last_became_peered":"2024-08-19T14:34:15.717054+0000","last_unstale":"2024-08-19T14:42:40.158528+0000","last_undegraded":"2024-08-19T14:42:40.158528+0000","last_fullsized":"2024-08-19T14:42:40.158528+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:41:55.664660+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:27.848870+0000","last_clean_scrub_stamp":"2024-08-19T14:41:55.664660+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:43:14.920500+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.c","version":"780'10942","reported_seq":20145,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-08-19T14:42:40.458279+0000","last_change":"2024-08-19T14:42:08.018783+0000","last_active":"2024-08-19T14:42:40.458279+0000","last_peered":"2024-08-19T14:42:40.458279+0000","last_clean":"2024-08-19T14:42:40.458279+0000","last_became_active":"2024-08-19T14:41:54.046067+0000","last_became_peered":"2024-08-19T14:41:54.046067+0000","last_unstale":"2024-08-19T14:42:40.458279+0000","last_undegraded":"2024-08-19T14:42:40.458279+0000","last_fullsized":"2024-08-19T14:42:40.458279+0000","mapping_epoch":767,"log_start":"722'900","ondisk_log_start":"722'900","created":717,"last_epoch_clean":768,"parent":"0.0","parent_split_bits":0,"last_scrub":"768'8949","last_scrub_stamp":"2024-08-19T14:42:08.018650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:42:08.018650+0000","objects_scrubbed":5375,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:43:29.396546+0000","scrub_duration":1333,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218890240,"num_objects":3340,"num_object_clones":0,"num_object_copies":10020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10942,"num_write_kb":457024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4496,"num_bytes_recovered":294649856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,7],"acting":[8,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.5","version":"70'139","reported_seq":930,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-08-19T14:42:40.160728+0000","last_change":"2024-08-19T14:42:39.073508+0000","last_active":"2024-08-19T14:42:40.160728+0000","last_peered":"2024-08-19T14:42:40.160728+0000","last_clean":"2024-08-19T14:42:40.160728+0000","last_became_active":"2024-08-19T14:42:39.073043+0000","last_became_peered":"2024-08-19T14:42:39.073043+0000","last_unstale":"2024-08-19T14:42:40.160728+0000","last_undegraded":"2024-08-19T14:42:40.160728+0000","last_fullsized":"2024-08-19T14:42:40.160728+0000","mapping_epoch":778,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":779,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:41:48.878043+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:41:48.878043+0000","last_clean_scrub_stamp":"2024-08-19T14:41:48.878043+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:43:11.622572+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,4],"acting":[11,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.d","version":"780'10911","reported_seq":11126,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-08-19T14:42:42.264658+0000","last_change":"2024-08-19T14:41:52.948980+0000","last_active":"2024-08-19T14:42:42.264658+0000","last_peered":"2024-08-19T14:42:42.264658+0000","last_clean":"2024-08-19T14:42:42.264658+0000","last_became_active":"2024-08-19T14:40:04.442093+0000","last_became_peered":"2024-08-19T14:40:04.442093+0000","last_unstale":"2024-08-19T14:42:42.264658+0000","last_undegraded":"2024-08-19T14:42:42.264658+0000","last_fullsized":"2024-08-19T14:42:42.264658+0000","mapping_epoch":717,"log_start":"722'900","ondisk_log_start":"722'900","created":717,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"766'7908","last_scrub_stamp":"2024-08-19T14:41:52.948595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:41:52.948595+0000","objects_scrubbed":6397,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:43:03.837104+0000","scrub_duration":3061,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213254168,"num_objects":3255,"num_object_clones":0,"num_object_copies":9765,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3255,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10911,"num_write_kb":453249,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,2],"acting":[11,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.4","version":"70'139","reported_seq":798,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-08-19T14:42:40.160500+0000","last_change":"2024-08-19T14:41:55.868740+0000","last_active":"2024-08-19T14:42:40.160500+0000","last_peered":"2024-08-19T14:42:40.160500+0000","last_clean":"2024-08-19T14:42:40.160500+0000","last_became_active":"2024-08-19T14:30:47.909067+0000","last_became_peered":"2024-08-19T14:30:47.909067+0000","last_unstale":"2024-08-19T14:42:40.160500+0000","last_undegraded":"2024-08-19T14:42:40.160500+0000","last_fullsized":"2024-08-19T14:42:40.160500+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:41:55.868614+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:41:55.868614+0000","last_clean_scrub_stamp":"2024-08-19T14:41:55.868614+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:43:09.847997+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.3","version":"780'7541","reported_seq":7728,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-08-19T14:42:42.247420+0000","last_change":"2024-08-19T14:41:46.600418+0000","last_active":"2024-08-19T14:42:42.247420+0000","last_peered":"2024-08-19T14:42:42.247420+0000","last_clean":"2024-08-19T14:42:42.247420+0000","last_became_active":"2024-08-19T14:40:43.369906+0000","last_became_peered":"2024-08-19T14:40:43.369906+0000","last_unstale":"2024-08-19T14:42:42.247420+0000","last_undegraded":"2024-08-19T14:42:42.247420+0000","last_fullsized":"2024-08-19T14:42:42.247420+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":717,"last_epoch_clean":732,"parent":"0.0","parent_split_bits":0,"last_scrub":"766'5736","last_scrub_stamp":"2024-08-19T14:41:46.599983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:41:46.599983+0000","objects_scrubbed":3381,"log_size":7541,"log_dups_size":0,"ondisk_log_size":7541,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:43:05.327404+0000","scrub_duration":719,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":101711872,"num_objects":1552,"num_object_clones":0,"num_object_copies":4656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1552,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1829,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.a","version":"70'139","reported_seq":973,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-08-19T14:42:40.169744+0000","last_change":"2024-08-19T14:42:39.072075+0000","last_active":"2024-08-19T14:42:40.169744+0000","last_peered":"2024-08-19T14:42:40.169744+0000","last_clean":"2024-08-19T14:42:40.169744+0000","last_became_active":"2024-08-19T14:42:39.071701+0000","last_became_peered":"2024-08-19T14:42:39.071701+0000","last_unstale":"2024-08-19T14:42:40.169744+0000","last_undegraded":"2024-08-19T14:42:40.169744+0000","last_fullsized":"2024-08-19T14:42:40.169744+0000","mapping_epoch":778,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":779,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:41:55.751274+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:28.864870+0000","last_clean_scrub_stamp":"2024-08-19T14:41:55.751274+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:42:59.343342+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,2],"acting":[4,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.0","version":"780'10376","reported_seq":15172,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-08-19T14:42:43.864467+0000","last_change":"2024-08-19T14:42:15.406925+0000","last_active":"2024-08-19T14:42:43.864467+0000","last_peered":"2024-08-19T14:42:43.864467+0000","last_clean":"2024-08-19T14:42:43.864467+0000","last_became_active":"2024-08-19T14:41:44.018073+0000","last_became_peered":"2024-08-19T14:41:44.018073+0000","last_unstale":"2024-08-19T14:42:43.864467+0000","last_undegraded":"2024-08-19T14:42:43.864467+0000","last_fullsized":"2024-08-19T14:42:43.864467+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":717,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":0,"last_scrub":"769'9447","last_scrub_stamp":"2024-08-19T14:42:15.406827+0000","last_deep_scrub":"769'9447","last_deep_scrub_stamp":"2024-08-19T14:42:15.406827+0000","last_clean_scrub_stamp":"2024-08-19T14:42:15.406827+0000","objects_scrubbed":2569,"log_size":10376,"log_dups_size":0,"ondisk_log_size":10376,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:43:37.129147+0000","scrub_duration":7640,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100335616,"num_objects":1531,"num_object_clones":0,"num_object_copies":4593,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3327,"num_write_kb":85824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2216,"num_bytes_recovered":145227776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,9],"acting":[4,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"70'139","reported_seq":1899,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-08-19T14:42:40.163032+0000","last_change":"2024-08-19T14:42:00.561162+0000","last_active":"2024-08-19T14:42:40.163032+0000","last_peered":"2024-08-19T14:42:40.163032+0000","last_clean":"2024-08-19T14:42:40.163032+0000","last_became_active":"2024-08-19T14:35:12.529628+0000","last_became_peered":"2024-08-19T14:35:12.529628+0000","last_unstale":"2024-08-19T14:42:40.163032+0000","last_undegraded":"2024-08-19T14:42:40.163032+0000","last_fullsized":"2024-08-19T14:42:40.163032+0000","mapping_epoch":635,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":636,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:42:00.561080+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:42:00.561080+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:43:16.722031+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1","version":"780'7640","reported_seq":7833,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-08-19T14:42:43.554445+0000","last_change":"2024-08-19T14:41:40.957892+0000","last_active":"2024-08-19T14:42:43.554445+0000","last_peered":"2024-08-19T14:42:43.554445+0000","last_clean":"2024-08-19T14:42:43.554445+0000","last_became_active":"2024-08-19T14:40:43.388885+0000","last_became_peered":"2024-08-19T14:40:43.388885+0000","last_unstale":"2024-08-19T14:42:43.554445+0000","last_undegraded":"2024-08-19T14:42:43.554445+0000","last_fullsized":"2024-08-19T14:42:43.554445+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":717,"last_epoch_clean":732,"parent":"0.0","parent_split_bits":0,"last_scrub":"762'5656","last_scrub_stamp":"2024-08-19T14:41:40.957738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:41:40.957738+0000","objects_scrubbed":3522,"log_size":7640,"log_dups_size":0,"ondisk_log_size":7640,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:43:07.251310+0000","scrub_duration":1059,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104005632,"num_objects":1587,"num_object_clones":0,"num_object_copies":4761,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2005,"num_write_kb":2240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,2],"acting":[1,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"70'139","reported_seq":481,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-08-19T14:42:40.158302+0000","last_change":"2024-08-19T14:42:08.692385+0000","last_active":"2024-08-19T14:42:40.158302+0000","last_peered":"2024-08-19T14:42:40.158302+0000","last_clean":"2024-08-19T14:42:40.158302+0000","last_became_active":"2024-08-19T14:40:52.455488+0000","last_became_peered":"2024-08-19T14:40:52.455488+0000","last_unstale":"2024-08-19T14:42:40.158302+0000","last_undegraded":"2024-08-19T14:42:40.158302+0000","last_fullsized":"2024-08-19T14:42:40.158302+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:42:08.692264+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:42:08.692264+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:43:10.218844+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,2],"acting":[8,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.2","version":"780'7499","reported_seq":8321,"reported_epoch":780,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-19T14:42:43.873864+0000","last_change":"2024-08-19T14:42:37.369966+0000","last_active":"2024-08-19T14:42:43.873864+0000","last_peered":"2024-08-19T14:42:43.873864+0000","last_clean":"2024-08-19T14:42:35.099810+0000","last_became_active":"2024-08-19T14:42:37.325427+0000","last_became_peered":"2024-08-19T14:42:37.325427+0000","last_unstale":"2024-08-19T14:42:43.873864+0000","last_undegraded":"2024-08-19T14:42:37.170402+0000","last_fullsized":"2024-08-19T14:42:37.133403+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":717,"last_epoch_clean":732,"parent":"0.0","parent_split_bits":0,"last_scrub":"762'5512","last_scrub_stamp":"2024-08-19T14:41:40.095431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:41:40.095431+0000","objects_scrubbed":3411,"log_size":7499,"log_dups_size":0,"ondisk_log_size":7499,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":950,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100925440,"num_objects":1540,"num_object_clones":0,"num_object_copies":4620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":1418,"num_objects_unfound":0,"num_objects_dirty":1540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2003,"num_write_kb":4224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":315,"num_bytes_recovered":20578304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,5],"acting":[4,9],"avail_no_missing":["4","0","9"],"object_location_counts":[{"shards":"0,4,9","objects":1418},{"shards":"4,9","objects":117}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"70'139","reported_seq":1259,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-08-19T14:42:40.160753+0000","last_change":"2024-08-19T14:42:39.077445+0000","last_active":"2024-08-19T14:42:40.160753+0000","last_peered":"2024-08-19T14:42:40.160753+0000","last_clean":"2024-08-19T14:42:40.160753+0000","last_became_active":"2024-08-19T14:42:39.077214+0000","last_became_peered":"2024-08-19T14:42:39.077214+0000","last_unstale":"2024-08-19T14:42:40.160753+0000","last_undegraded":"2024-08-19T14:42:40.160753+0000","last_fullsized":"2024-08-19T14:42:40.160753+0000","mapping_epoch":778,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":779,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:41:50.892343+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:41:50.892343+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:42:59.875814+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,2],"acting":[11,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.7","version":"780'7406","reported_seq":7618,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-08-19T14:42:43.585014+0000","last_change":"2024-08-19T14:42:07.286608+0000","last_active":"2024-08-19T14:42:43.585014+0000","last_peered":"2024-08-19T14:42:43.585014+0000","last_clean":"2024-08-19T14:42:43.585014+0000","last_became_active":"2024-08-19T14:40:43.370885+0000","last_became_peered":"2024-08-19T14:40:43.370885+0000","last_unstale":"2024-08-19T14:42:43.585014+0000","last_undegraded":"2024-08-19T14:42:43.585014+0000","last_fullsized":"2024-08-19T14:42:43.585014+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":717,"last_epoch_clean":732,"parent":"0.0","parent_split_bits":0,"last_scrub":"768'6303","last_scrub_stamp":"2024-08-19T14:42:07.286464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:42:07.286464+0000","objects_scrubbed":2670,"log_size":7406,"log_dups_size":0,"ondisk_log_size":7406,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:43:22.343308+0000","scrub_duration":4551,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":97910784,"num_objects":1494,"num_object_clones":0,"num_object_copies":4482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3257,"num_write_kb":83136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.e","version":"70'139","reported_seq":1449,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-08-19T14:42:40.160668+0000","last_change":"2024-08-19T14:41:46.892341+0000","last_active":"2024-08-19T14:42:40.160668+0000","last_peered":"2024-08-19T14:42:40.160668+0000","last_clean":"2024-08-19T14:42:40.160668+0000","last_became_active":"2024-08-19T14:40:52.444460+0000","last_became_peered":"2024-08-19T14:40:52.444460+0000","last_unstale":"2024-08-19T14:42:40.160668+0000","last_undegraded":"2024-08-19T14:42:40.160668+0000","last_fullsized":"2024-08-19T14:42:40.160668+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:41:46.892116+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:40:03.764672+0000","last_clean_scrub_stamp":"2024-08-19T14:41:46.892116+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:42:56.457285+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,8],"acting":[4,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.6","version":"780'7473","reported_seq":7660,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-08-19T14:42:42.209980+0000","last_change":"2024-08-19T14:41:47.818694+0000","last_active":"2024-08-19T14:42:42.209980+0000","last_peered":"2024-08-19T14:42:42.209980+0000","last_clean":"2024-08-19T14:42:42.209980+0000","last_became_active":"2024-08-19T14:40:43.378698+0000","last_became_peered":"2024-08-19T14:40:43.378698+0000","last_unstale":"2024-08-19T14:42:42.209980+0000","last_undegraded":"2024-08-19T14:42:42.209980+0000","last_fullsized":"2024-08-19T14:42:42.209980+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":717,"last_epoch_clean":732,"parent":"0.0","parent_split_bits":0,"last_scrub":"766'5757","last_scrub_stamp":"2024-08-19T14:41:47.818557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:41:47.818557+0000","objects_scrubbed":3342,"log_size":7473,"log_dups_size":0,"ondisk_log_size":7473,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:43:05.485710+0000","scrub_duration":947,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":105381888,"num_objects":1608,"num_object_clones":0,"num_object_copies":4824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1734,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,2],"acting":[11,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.f","version":"70'139","reported_seq":1201,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-08-19T14:42:40.160486+0000","last_change":"2024-08-19T14:42:03.775080+0000","last_active":"2024-08-19T14:42:40.160486+0000","last_peered":"2024-08-19T14:42:40.160486+0000","last_clean":"2024-08-19T14:42:40.160486+0000","last_became_active":"2024-08-19T14:40:52.446953+0000","last_became_peered":"2024-08-19T14:40:52.446953+0000","last_unstale":"2024-08-19T14:42:40.160486+0000","last_undegraded":"2024-08-19T14:42:40.160486+0000","last_fullsized":"2024-08-19T14:42:40.160486+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:42:03.774987+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:42:03.774987+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:43:17.121159+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,8],"acting":[4,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.5","version":"780'7671","reported_seq":7881,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-08-19T14:42:43.861439+0000","last_change":"2024-08-19T14:42:06.297246+0000","last_active":"2024-08-19T14:42:43.861439+0000","last_peered":"2024-08-19T14:42:43.861439+0000","last_clean":"2024-08-19T14:42:43.861439+0000","last_became_active":"2024-08-19T14:40:43.370152+0000","last_became_peered":"2024-08-19T14:40:43.370152+0000","last_unstale":"2024-08-19T14:42:43.861439+0000","last_undegraded":"2024-08-19T14:42:43.861439+0000","last_fullsized":"2024-08-19T14:42:43.861439+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":717,"last_epoch_clean":732,"parent":"0.0","parent_split_bits":0,"last_scrub":"768'6479","last_scrub_stamp":"2024-08-19T14:42:06.297116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:42:06.297116+0000","objects_scrubbed":2666,"log_size":7671,"log_dups_size":0,"ondisk_log_size":7671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:43:08.219715+0000","scrub_duration":500,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96010240,"num_objects":1465,"num_object_clones":0,"num_object_copies":4395,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1465,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3339,"num_write_kb":83456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,11],"acting":[4,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.c","version":"70'139","reported_seq":504,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-08-19T14:42:40.155321+0000","last_change":"2024-08-19T14:42:39.073450+0000","last_active":"2024-08-19T14:42:40.155321+0000","last_peered":"2024-08-19T14:42:40.155321+0000","last_clean":"2024-08-19T14:42:40.155321+0000","last_became_active":"2024-08-19T14:42:39.073086+0000","last_became_peered":"2024-08-19T14:42:39.073086+0000","last_unstale":"2024-08-19T14:42:40.155321+0000","last_undegraded":"2024-08-19T14:42:40.155321+0000","last_fullsized":"2024-08-19T14:42:40.155321+0000","mapping_epoch":778,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":779,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:41:37.152446+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:22:52.164586+0000","last_clean_scrub_stamp":"2024-08-19T14:41:37.152446+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:42:51.252472+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,5],"acting":[1,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.4","version":"780'7409","reported_seq":7617,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-08-19T14:42:42.223240+0000","last_change":"2024-08-19T14:42:06.348357+0000","last_active":"2024-08-19T14:42:42.223240+0000","last_peered":"2024-08-19T14:42:42.223240+0000","last_clean":"2024-08-19T14:42:42.223240+0000","last_became_active":"2024-08-19T14:40:43.364933+0000","last_became_peered":"2024-08-19T14:40:43.364933+0000","last_unstale":"2024-08-19T14:42:42.223240+0000","last_undegraded":"2024-08-19T14:42:42.223240+0000","last_fullsized":"2024-08-19T14:42:42.223240+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":717,"last_epoch_clean":732,"parent":"0.0","parent_split_bits":0,"last_scrub":"768'6335","last_scrub_stamp":"2024-08-19T14:42:06.348178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:42:06.348178+0000","objects_scrubbed":2643,"log_size":7409,"log_dups_size":0,"ondisk_log_size":7409,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:43:17.876356+0000","scrub_duration":438,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102170624,"num_objects":1559,"num_object_clones":0,"num_object_copies":4677,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1559,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3209,"num_write_kb":86272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,9],"acting":[11,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.d","version":"70'139","reported_seq":1444,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-08-19T14:42:40.164045+0000","last_change":"2024-08-19T14:42:09.549992+0000","last_active":"2024-08-19T14:42:40.164045+0000","last_peered":"2024-08-19T14:42:40.164045+0000","last_clean":"2024-08-19T14:42:40.164045+0000","last_became_active":"2024-08-19T14:18:26.882002+0000","last_became_peered":"2024-08-19T14:18:26.882002+0000","last_unstale":"2024-08-19T14:42:40.164045+0000","last_undegraded":"2024-08-19T14:42:40.164045+0000","last_fullsized":"2024-08-19T14:42:40.164045+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:42:09.549880+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:24:50.276408+0000","last_clean_scrub_stamp":"2024-08-19T14:42:09.549880+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:43:32.809470+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"70'139","reported_seq":1223,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-08-19T14:42:40.155253+0000","last_change":"2024-08-19T14:41:49.815945+0000","last_active":"2024-08-19T14:42:40.155253+0000","last_peered":"2024-08-19T14:42:40.155253+0000","last_clean":"2024-08-19T14:42:40.155253+0000","last_became_active":"2024-08-19T14:27:31.838968+0000","last_became_peered":"2024-08-19T14:27:31.838968+0000","last_unstale":"2024-08-19T14:42:40.155253+0000","last_undegraded":"2024-08-19T14:42:40.155253+0000","last_fullsized":"2024-08-19T14:42:40.155253+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:41:49.815826+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:41:49.815826+0000","last_clean_scrub_stamp":"2024-08-19T14:41:49.815826+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:42:52.631709+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.13","version":"70'139","reported_seq":1774,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-08-19T14:42:40.158647+0000","last_change":"2024-08-19T14:42:39.080808+0000","last_active":"2024-08-19T14:42:40.158647+0000","last_peered":"2024-08-19T14:42:40.158647+0000","last_clean":"2024-08-19T14:42:40.158647+0000","last_became_active":"2024-08-19T14:42:39.080282+0000","last_became_peered":"2024-08-19T14:42:39.080282+0000","last_unstale":"2024-08-19T14:42:40.158647+0000","last_undegraded":"2024-08-19T14:42:40.158647+0000","last_fullsized":"2024-08-19T14:42:40.158647+0000","mapping_epoch":778,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":779,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:41:55.847727+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:33:35.786105+0000","last_clean_scrub_stamp":"2024-08-19T14:41:55.847727+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:43:22.169662+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,4],"acting":[1,11,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.19","version":"780'7608","reported_seq":14337,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-08-19T14:42:43.798445+0000","last_change":"2024-08-19T14:41:45.605464+0000","last_active":"2024-08-19T14:42:43.798445+0000","last_peered":"2024-08-19T14:42:43.798445+0000","last_clean":"2024-08-19T14:42:43.798445+0000","last_became_active":"2024-08-19T14:41:31.697673+0000","last_became_peered":"2024-08-19T14:41:31.697673+0000","last_unstale":"2024-08-19T14:42:43.798445+0000","last_undegraded":"2024-08-19T14:42:43.798445+0000","last_fullsized":"2024-08-19T14:42:43.798445+0000","mapping_epoch":756,"log_start":"0'0","ondisk_log_start":"0'0","created":731,"last_epoch_clean":757,"parent":"0.0","parent_split_bits":5,"last_scrub":"765'5724","last_scrub_stamp":"2024-08-19T14:41:45.605377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:41:45.605377+0000","objects_scrubbed":3506,"log_size":7608,"log_dups_size":0,"ondisk_log_size":7608,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:42:58.500861+0000","scrub_duration":763,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":105185280,"num_objects":1605,"num_object_clones":0,"num_object_copies":4815,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1901,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,8],"acting":[4,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.10","version":"70'139","reported_seq":1385,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-08-19T14:42:40.162914+0000","last_change":"2024-08-19T14:40:32.035691+0000","last_active":"2024-08-19T14:42:40.162914+0000","last_peered":"2024-08-19T14:42:40.162914+0000","last_clean":"2024-08-19T14:42:40.162914+0000","last_became_active":"2024-08-19T14:27:31.839611+0000","last_became_peered":"2024-08-19T14:27:31.839611+0000","last_unstale":"2024-08-19T14:42:40.162914+0000","last_undegraded":"2024-08-19T14:42:40.162914+0000","last_fullsized":"2024-08-19T14:42:40.162914+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:40:32.035620+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:34:58.616646+0000","last_clean_scrub_stamp":"2024-08-19T14:40:32.035620+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.18","version":"780'7706","reported_seq":14478,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-08-19T14:42:43.815526+0000","last_change":"2024-08-19T14:41:46.506136+0000","last_active":"2024-08-19T14:42:43.815526+0000","last_peered":"2024-08-19T14:42:43.815526+0000","last_clean":"2024-08-19T14:42:43.815526+0000","last_became_active":"2024-08-19T14:41:44.022738+0000","last_became_peered":"2024-08-19T14:41:44.022738+0000","last_unstale":"2024-08-19T14:42:43.815526+0000","last_undegraded":"2024-08-19T14:42:43.815526+0000","last_fullsized":"2024-08-19T14:42:43.815526+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":731,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":5,"last_scrub":"766'5821","last_scrub_stamp":"2024-08-19T14:41:46.505899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:41:46.505899+0000","objects_scrubbed":3475,"log_size":7706,"log_dups_size":0,"ondisk_log_size":7706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:43:11.646115+0000","scrub_duration":642,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103219200,"num_objects":1575,"num_object_clones":0,"num_object_copies":4725,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1575,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1900,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,5],"acting":[4,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.11","version":"70'139","reported_seq":298,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-08-19T14:42:40.158390+0000","last_change":"2024-08-19T14:42:39.064912+0000","last_active":"2024-08-19T14:42:40.158390+0000","last_peered":"2024-08-19T14:42:40.158390+0000","last_clean":"2024-08-19T14:42:40.158390+0000","last_became_active":"2024-08-19T14:42:39.064403+0000","last_became_peered":"2024-08-19T14:42:39.064403+0000","last_unstale":"2024-08-19T14:42:40.158390+0000","last_undegraded":"2024-08-19T14:42:40.158390+0000","last_fullsized":"2024-08-19T14:42:40.158390+0000","mapping_epoch":778,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":779,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:41:47.147376+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:41:47.147376+0000","last_clean_scrub_stamp":"2024-08-19T14:41:47.147376+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:42:44.825898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":5,"num_write":9,"num_write_kb":134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":96169,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,1],"acting":[8,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"70'139","reported_seq":1394,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-08-19T14:42:40.155495+0000","last_change":"2024-08-19T14:42:39.067076+0000","last_active":"2024-08-19T14:42:40.155495+0000","last_peered":"2024-08-19T14:42:40.155495+0000","last_clean":"2024-08-19T14:42:40.155495+0000","last_became_active":"2024-08-19T14:42:39.066670+0000","last_became_peered":"2024-08-19T14:42:39.066670+0000","last_unstale":"2024-08-19T14:42:40.155495+0000","last_undegraded":"2024-08-19T14:42:40.155495+0000","last_fullsized":"2024-08-19T14:42:40.155495+0000","mapping_epoch":778,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":779,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:41:42.882576+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:28:34.216370+0000","last_clean_scrub_stamp":"2024-08-19T14:41:42.882576+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:43:03.755107+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,5],"acting":[1,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"70'139","reported_seq":1535,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-08-19T14:42:40.158572+0000","last_change":"2024-08-19T14:42:39.066158+0000","last_active":"2024-08-19T14:42:40.158572+0000","last_peered":"2024-08-19T14:42:40.158572+0000","last_clean":"2024-08-19T14:42:40.158572+0000","last_became_active":"2024-08-19T14:42:39.065831+0000","last_became_peered":"2024-08-19T14:42:39.065831+0000","last_unstale":"2024-08-19T14:42:40.158572+0000","last_undegraded":"2024-08-19T14:42:40.158572+0000","last_fullsized":"2024-08-19T14:42:40.158572+0000","mapping_epoch":778,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":779,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:42:14.678271+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:42:14.678271+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:43:24.218124+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,11],"acting":[8,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.14","version":"70'139","reported_seq":1548,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-08-19T14:42:40.166113+0000","last_change":"2024-08-19T14:42:06.543869+0000","last_active":"2024-08-19T14:42:40.166113+0000","last_peered":"2024-08-19T14:42:40.166113+0000","last_clean":"2024-08-19T14:42:40.166113+0000","last_became_active":"2024-08-19T14:35:12.529792+0000","last_became_peered":"2024-08-19T14:35:12.529792+0000","last_unstale":"2024-08-19T14:42:40.166113+0000","last_undegraded":"2024-08-19T14:42:40.166113+0000","last_fullsized":"2024-08-19T14:42:40.166113+0000","mapping_epoch":635,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":636,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:42:06.543798+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:42:06.543798+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:43:13.641424+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,9],"acting":[7,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.15","version":"70'139","reported_seq":1400,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-08-19T14:42:40.162541+0000","last_change":"2024-08-19T14:42:15.608090+0000","last_active":"2024-08-19T14:42:40.162541+0000","last_peered":"2024-08-19T14:42:40.162541+0000","last_clean":"2024-08-19T14:42:40.162541+0000","last_became_active":"2024-08-19T14:18:31.912496+0000","last_became_peered":"2024-08-19T14:18:31.912496+0000","last_unstale":"2024-08-19T14:42:40.162541+0000","last_undegraded":"2024-08-19T14:42:40.162541+0000","last_fullsized":"2024-08-19T14:42:40.162541+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:42:15.607984+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:20:09.180923+0000","last_clean_scrub_stamp":"2024-08-19T14:42:15.607984+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:43:27.095836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.13","version":"780'7497","reported_seq":12245,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-08-19T14:42:42.254059+0000","last_change":"2024-08-19T14:41:43.273122+0000","last_active":"2024-08-19T14:42:42.254059+0000","last_peered":"2024-08-19T14:42:42.254059+0000","last_clean":"2024-08-19T14:42:42.254059+0000","last_became_active":"2024-08-19T14:41:06.590807+0000","last_became_peered":"2024-08-19T14:41:06.590807+0000","last_unstale":"2024-08-19T14:42:42.254059+0000","last_undegraded":"2024-08-19T14:42:42.254059+0000","last_fullsized":"2024-08-19T14:42:42.254059+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":731,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":5,"last_scrub":"764'5616","last_scrub_stamp":"2024-08-19T14:41:43.273000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:41:43.273000+0000","objects_scrubbed":3456,"log_size":7497,"log_dups_size":0,"ondisk_log_size":7497,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:43:11.150161+0000","scrub_duration":418,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103153664,"num_objects":1574,"num_object_clones":0,"num_object_copies":4722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1882,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,9],"acting":[11,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1a","version":"70'139","reported_seq":1385,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-08-19T14:42:40.169025+0000","last_change":"2024-08-19T14:41:51.832495+0000","last_active":"2024-08-19T14:42:40.169025+0000","last_peered":"2024-08-19T14:42:40.169025+0000","last_clean":"2024-08-19T14:42:40.169025+0000","last_became_active":"2024-08-19T14:18:31.901229+0000","last_became_peered":"2024-08-19T14:18:31.901229+0000","last_unstale":"2024-08-19T14:42:40.169025+0000","last_undegraded":"2024-08-19T14:42:40.169025+0000","last_fullsized":"2024-08-19T14:42:40.169025+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:41:51.832284+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:41:51.832284+0000","last_clean_scrub_stamp":"2024-08-19T14:41:51.832284+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:43:11.031980+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.12","version":"780'7444","reported_seq":13991,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-08-19T14:42:40.598695+0000","last_change":"2024-08-19T14:41:53.183280+0000","last_active":"2024-08-19T14:42:40.598695+0000","last_peered":"2024-08-19T14:42:40.598695+0000","last_clean":"2024-08-19T14:42:40.598695+0000","last_became_active":"2024-08-19T14:41:30.697627+0000","last_became_peered":"2024-08-19T14:41:30.697627+0000","last_unstale":"2024-08-19T14:42:40.598695+0000","last_undegraded":"2024-08-19T14:42:40.598695+0000","last_fullsized":"2024-08-19T14:42:40.598695+0000","mapping_epoch":755,"log_start":"0'0","ondisk_log_start":"0'0","created":731,"last_epoch_clean":756,"parent":"0.0","parent_split_bits":5,"last_scrub":"767'5979","last_scrub_stamp":"2024-08-19T14:41:53.183189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:41:53.183189+0000","objects_scrubbed":3133,"log_size":7444,"log_dups_size":0,"ondisk_log_size":7444,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:43:07.463075+0000","scrub_duration":570,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108789760,"num_objects":1660,"num_object_clones":0,"num_object_copies":4980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1660,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1473,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1b","version":"70'139","reported_seq":1775,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-08-19T14:42:40.161855+0000","last_change":"2024-08-19T14:42:39.066054+0000","last_active":"2024-08-19T14:42:40.161855+0000","last_peered":"2024-08-19T14:42:40.161855+0000","last_clean":"2024-08-19T14:42:40.161855+0000","last_became_active":"2024-08-19T14:42:39.065604+0000","last_became_peered":"2024-08-19T14:42:39.065604+0000","last_unstale":"2024-08-19T14:42:40.161855+0000","last_undegraded":"2024-08-19T14:42:40.161855+0000","last_fullsized":"2024-08-19T14:42:40.161855+0000","mapping_epoch":778,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":779,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:40:43.238820+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:37:59.510697+0000","last_clean_scrub_stamp":"2024-08-19T14:40:43.238820+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,2],"acting":[8,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.11","version":"780'7516","reported_seq":14254,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-08-19T14:42:40.607532+0000","last_change":"2024-08-19T14:41:57.023194+0000","last_active":"2024-08-19T14:42:40.607532+0000","last_peered":"2024-08-19T14:42:40.607532+0000","last_clean":"2024-08-19T14:42:40.607532+0000","last_became_active":"2024-08-19T14:41:28.692880+0000","last_became_peered":"2024-08-19T14:41:28.692880+0000","last_unstale":"2024-08-19T14:42:40.607532+0000","last_undegraded":"2024-08-19T14:42:40.607532+0000","last_fullsized":"2024-08-19T14:42:40.607532+0000","mapping_epoch":753,"log_start":"0'0","ondisk_log_start":"0'0","created":731,"last_epoch_clean":754,"parent":"0.0","parent_split_bits":5,"last_scrub":"768'6133","last_scrub_stamp":"2024-08-19T14:41:57.023094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:41:57.023094+0000","objects_scrubbed":3019,"log_size":7516,"log_dups_size":0,"ondisk_log_size":7516,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:43:16.626452+0000","scrub_duration":450,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106954752,"num_objects":1632,"num_object_clones":0,"num_object_copies":4896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1632,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1387,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,9],"acting":[7,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.18","version":"70'139","reported_seq":1328,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-08-19T14:42:40.162506+0000","last_change":"2024-08-19T14:41:53.603647+0000","last_active":"2024-08-19T14:42:40.162506+0000","last_peered":"2024-08-19T14:42:40.162506+0000","last_clean":"2024-08-19T14:42:40.162506+0000","last_became_active":"2024-08-19T14:35:11.528530+0000","last_became_peered":"2024-08-19T14:35:11.528530+0000","last_unstale":"2024-08-19T14:42:40.162506+0000","last_undegraded":"2024-08-19T14:42:40.162506+0000","last_fullsized":"2024-08-19T14:42:40.162506+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:41:53.603570+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:27:33.084161+0000","last_clean_scrub_stamp":"2024-08-19T14:41:53.603570+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:43:15.176682+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"780'7416","reported_seq":13584,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-08-19T14:42:40.527117+0000","last_change":"2024-08-19T14:41:55.126749+0000","last_active":"2024-08-19T14:42:40.527117+0000","last_peered":"2024-08-19T14:42:40.527117+0000","last_clean":"2024-08-19T14:42:40.527117+0000","last_became_active":"2024-08-19T14:41:43.014927+0000","last_became_peered":"2024-08-19T14:41:43.014927+0000","last_unstale":"2024-08-19T14:42:40.527117+0000","last_undegraded":"2024-08-19T14:42:40.527117+0000","last_fullsized":"2024-08-19T14:42:40.527117+0000","mapping_epoch":763,"log_start":"0'0","ondisk_log_start":"0'0","created":731,"last_epoch_clean":764,"parent":"0.0","parent_split_bits":5,"last_scrub":"768'6063","last_scrub_stamp":"2024-08-19T14:41:55.126598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:41:55.126598+0000","objects_scrubbed":3069,"log_size":7416,"log_dups_size":0,"ondisk_log_size":7416,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:43:04.050339+0000","scrub_duration":513,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112001024,"num_objects":1709,"num_object_clones":0,"num_object_copies":5127,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1360,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,9],"acting":[7,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.19","version":"70'139","reported_seq":795,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-08-19T14:42:40.162903+0000","last_change":"2024-08-19T14:42:39.091223+0000","last_active":"2024-08-19T14:42:40.162903+0000","last_peered":"2024-08-19T14:42:40.162903+0000","last_clean":"2024-08-19T14:42:40.162903+0000","last_became_active":"2024-08-19T14:42:39.090868+0000","last_became_peered":"2024-08-19T14:42:39.090868+0000","last_unstale":"2024-08-19T14:42:40.162903+0000","last_undegraded":"2024-08-19T14:42:40.162903+0000","last_fullsized":"2024-08-19T14:42:40.162903+0000","mapping_epoch":778,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":779,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:41:36.108239+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:41:36.108239+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:42:44.625892+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,8],"acting":[7,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.17","version":"780'9807","reported_seq":18646,"reported_epoch":780,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-19T14:42:40.631799+0000","last_change":"2024-08-19T14:42:37.365106+0000","last_active":"2024-08-19T14:42:40.631799+0000","last_peered":"2024-08-19T14:42:40.631799+0000","last_clean":"2024-08-19T14:42:35.085267+0000","last_became_active":"2024-08-19T14:42:37.328132+0000","last_became_peered":"2024-08-19T14:42:37.328132+0000","last_unstale":"2024-08-19T14:42:40.631799+0000","last_undegraded":"2024-08-19T14:42:37.165475+0000","last_fullsized":"2024-08-19T14:42:37.136256+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":731,"last_epoch_clean":760,"parent":"0.0","parent_split_bits":5,"last_scrub":"768'8732","last_scrub_stamp":"2024-08-19T14:42:04.677577+0000","last_deep_scrub":"768'8732","last_deep_scrub_stamp":"2024-08-19T14:42:04.677577+0000","last_clean_scrub_stamp":"2024-08-19T14:42:04.677577+0000","objects_scrubbed":2883,"log_size":9807,"log_dups_size":0,"ondisk_log_size":9807,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":10,"scrub_schedule":"no scrub is scheduled","scrub_duration":9059,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109182976,"num_objects":1666,"num_object_clones":0,"num_object_copies":4998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":104,"num_objects_misplaced":1506,"num_objects_unfound":0,"num_objects_dirty":1666,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1217,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":161,"num_bytes_recovered":10485760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,2],"acting":[7,8],"avail_no_missing":["7","0","8"],"object_location_counts":[{"shards":"0,7,8","objects":1506},{"shards":"7,8","objects":56}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1e","version":"70'139","reported_seq":770,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-08-19T14:42:40.161376+0000","last_change":"2024-08-19T14:41:58.896792+0000","last_active":"2024-08-19T14:42:40.161376+0000","last_peered":"2024-08-19T14:42:40.161376+0000","last_clean":"2024-08-19T14:42:40.161376+0000","last_became_active":"2024-08-19T14:30:47.909974+0000","last_became_peered":"2024-08-19T14:30:47.909974+0000","last_unstale":"2024-08-19T14:42:40.161376+0000","last_undegraded":"2024-08-19T14:42:40.161376+0000","last_fullsized":"2024-08-19T14:42:40.161376+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:41:58.896684+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:27:04.780939+0000","last_clean_scrub_stamp":"2024-08-19T14:41:58.896684+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:43:11.035925+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.15","version":"780'7497","reported_seq":14041,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-08-19T14:42:40.602958+0000","last_change":"2024-08-19T14:41:58.178892+0000","last_active":"2024-08-19T14:42:40.602958+0000","last_peered":"2024-08-19T14:42:40.602958+0000","last_clean":"2024-08-19T14:42:40.602958+0000","last_became_active":"2024-08-19T14:41:29.692988+0000","last_became_peered":"2024-08-19T14:41:29.692988+0000","last_unstale":"2024-08-19T14:42:40.602958+0000","last_undegraded":"2024-08-19T14:42:40.602958+0000","last_fullsized":"2024-08-19T14:42:40.602958+0000","mapping_epoch":754,"log_start":"0'0","ondisk_log_start":"0'0","created":731,"last_epoch_clean":755,"parent":"0.0","parent_split_bits":5,"last_scrub":"768'6135","last_scrub_stamp":"2024-08-19T14:41:58.178684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:41:58.178684+0000","objects_scrubbed":2932,"log_size":7497,"log_dups_size":0,"ondisk_log_size":7497,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:43:16.490802+0000","scrub_duration":607,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":101974016,"num_objects":1556,"num_object_clones":0,"num_object_copies":4668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1556,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1376,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,1],"acting":[7,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1c","version":"70'139","reported_seq":1520,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-08-19T14:42:40.158460+0000","last_change":"2024-08-19T14:40:31.338726+0000","last_active":"2024-08-19T14:42:40.158460+0000","last_peered":"2024-08-19T14:42:40.158460+0000","last_clean":"2024-08-19T14:42:40.158460+0000","last_became_active":"2024-08-19T14:30:47.908866+0000","last_became_peered":"2024-08-19T14:30:47.908866+0000","last_unstale":"2024-08-19T14:42:40.158460+0000","last_undegraded":"2024-08-19T14:42:40.158460+0000","last_fullsized":"2024-08-19T14:42:40.158460+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:40:31.338409+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:40:31.338409+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,11],"acting":[8,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.14","version":"780'7563","reported_seq":17026,"reported_epoch":780,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-19T14:42:43.875494+0000","last_change":"2024-08-19T14:42:37.421277+0000","last_active":"2024-08-19T14:42:43.875494+0000","last_peered":"2024-08-19T14:42:43.875494+0000","last_clean":"2024-08-19T14:42:35.108655+0000","last_became_active":"2024-08-19T14:42:37.362977+0000","last_became_peered":"2024-08-19T14:42:37.362977+0000","last_unstale":"2024-08-19T14:42:43.875494+0000","last_undegraded":"2024-08-19T14:42:37.173060+0000","last_fullsized":"2024-08-19T14:42:37.134077+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":731,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":5,"last_scrub":"765'5594","last_scrub_stamp":"2024-08-19T14:41:44.557906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:40:03.182354+0000","last_clean_scrub_stamp":"2024-08-19T14:41:44.557906+0000","objects_scrubbed":3512,"log_size":7563,"log_dups_size":0,"ondisk_log_size":7563,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":712,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100270080,"num_objects":1530,"num_object_clones":0,"num_object_copies":4590,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":1511,"num_objects_unfound":0,"num_objects_dirty":1530,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1982,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":251,"num_bytes_recovered":16449536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2],"avail_no_missing":["4","0","2","5"],"object_location_counts":[{"shards":"0,2,4","objects":1511},{"shards":"2,4","objects":14}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1d","version":"70'139","reported_seq":1429,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-08-19T14:42:40.156028+0000","last_change":"2024-08-19T14:41:38.948464+0000","last_active":"2024-08-19T14:42:40.156028+0000","last_peered":"2024-08-19T14:42:40.156028+0000","last_clean":"2024-08-19T14:42:40.156028+0000","last_became_active":"2024-08-19T14:35:11.540488+0000","last_became_peered":"2024-08-19T14:35:11.540488+0000","last_unstale":"2024-08-19T14:42:40.156028+0000","last_undegraded":"2024-08-19T14:42:40.156028+0000","last_fullsized":"2024-08-19T14:42:40.156028+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:41:38.947168+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:41:38.947168+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:42:58.810121+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":26,"stat_sum":{"num_bytes":3358720024,"num_objects":51251,"num_object_clones":0,"num_object_copies":153753,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":114,"num_objects_misplaced":8601,"num_objects_unfound":0,"num_objects_dirty":51251,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":105796,"num_write_kb":3055041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8279,"num_bytes_recovered":542441472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10559434752,"data_stored":10559422536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":218503,"ondisk_log_size":218503,"up":78,"acting":75,"num_store_stats":10},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":617203,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":5,"num_write":9,"num_write_kb":134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":625097,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2252800,"data_stored":2242144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187832,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2230212,"kb_used_data":1242308,"kb_used_omap":1,"kb_used_meta":987710,"kb_avail":91514940,"statfs":{"total":95995035648,"available":93711298560,"internally_reserved":0,"allocated":1272123392,"data_stored":1269366067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":1011415492},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":2314987372699,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":74,"seq":317827580290,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2467400,"kb_used_data":1447744,"kb_used_omap":1,"kb_used_meta":1019646,"kb_avail":91277752,"statfs":{"total":95995035648,"available":93468418048,"internally_reserved":0,"allocated":1482489856,"data_stored":1479736603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":1044117953},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874632,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1900192,"kb_used_data":980136,"kb_used_omap":1,"kb_used_meta":919678,"kb_avail":91844960,"statfs":{"total":95995035648,"available":94049239040,"internally_reserved":0,"allocated":1003659264,"data_stored":1000847958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":941750725},"hb_peers":[0,1,2,3,4,5,6,7,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":57,"seq":244813136267,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2201068,"kb_used_data":1100404,"kb_used_omap":1,"kb_used_meta":1100158,"kb_avail":91544084,"statfs":{"total":95995035648,"available":93741142016,"internally_reserved":0,"allocated":1126813696,"data_stored":1123994766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":1126562250},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,3,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":0,"seq":206158430606,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692242,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2195008,"kb_used_data":1116680,"kb_used_omap":1,"kb_used_meta":1077950,"kb_avail":91550144,"statfs":{"total":95995035648,"available":93747347456,"internally_reserved":0,"allocated":1143480320,"data_stored":1140723515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1103821253},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986583,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2355848,"kb_used_data":1219584,"kb_used_omap":1,"kb_used_meta":1136254,"kb_avail":91389304,"statfs":{"total":95995035648,"available":93582647296,"internally_reserved":0,"allocated":1248854016,"data_stored":1246100763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1163524549},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":111669150109,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":634,"seq":2723009265755,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1999028,"kb_used_data":1529460,"kb_used_omap":4,"kb_used_meta":469371,"kb_avail":91746124,"statfs":{"total":95995035648,"available":93948030976,"internally_reserved":0,"allocated":1566167040,"data_stored":1563348110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4407,"internal_metadata":480636617},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575270,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2120712,"kb_used_data":892928,"kb_used_omap":1,"kb_used_meta":1227774,"kb_avail":91624440,"statfs":{"total":95995035648,"available":93823426560,"internally_reserved":0,"allocated":914358272,"data_stored":911605019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1257241029},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":34359738794,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":676659200,"data_stored":676659200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":893190144,"data_stored":893190144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1534726144,"data_stored":1534722072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":294649856,"data_stored":294649856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1239744512,"data_stored":1239744512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1132003328,"data_stored":1132003328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1107562496,"data_stored":1107558424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":982712320,"data_stored":982712320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1454374912,"data_stored":1454374912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1243811840,"data_stored":1243807768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T14:42:51.142 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-08-19T14:42:51.142 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd pool set .mgr pg_num 101 2024-08-19T14:42:51.224 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:42:51.268 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:51.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:51.269 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:42:51.370 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:42:51.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:51.405 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:51.405 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:42:51.506 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:42:51.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:51 smithi032 ceph-mon[20004]: osdmap e783: 12 total, 12 up, 8 in 2024-08-19T14:42:51.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:51 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:42:51.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:51 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:42:51.543 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:51.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:51.544 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:42:51.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:51 smithi045 ceph-mon[20143]: osdmap e783: 12 total, 12 up, 8 in 2024-08-19T14:42:51.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:51 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:42:51.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:51 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:42:51.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:51 smithi031 ceph-mon[16336]: osdmap e783: 12 total, 12 up, 8 in 2024-08-19T14:42:51.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:51 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:42:51.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:51 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:42:51.645 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:42:51.683 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:51.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:51.684 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:42:51.729 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-19T14:42:51.785 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:42:51.809 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:51.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:51.810 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:42:51.911 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:42:51.947 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:51.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:51.948 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:42:51.997 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:42:52.048 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:42:52.087 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:42:52.090 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:52.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:52.091 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:42:52.192 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:42:52.220 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:52.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:52.221 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:42:52.321 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:42:52.348 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:52.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:52.349 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:42:52.406 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:52 smithi031 ceph-mon[16336]: pgmap v1871: 57 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 52 active+clean; 2.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 854 op/s; 272/138375 objects degraded (0.197%); 6564/138375 objects misplaced (4.744%); 17 MiB/s, 281 objects/s recovering 2024-08-19T14:42:52.406 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:52 smithi031 ceph-mon[16336]: osdmap e784: 12 total, 12 up, 8 in 2024-08-19T14:42:52.450 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:42:52.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:52.486 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:52.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:42:52.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:52 smithi032 ceph-mon[20004]: pgmap v1871: 57 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 52 active+clean; 2.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 854 op/s; 272/138375 objects degraded (0.197%); 6564/138375 objects misplaced (4.744%); 17 MiB/s, 281 objects/s recovering 2024-08-19T14:42:52.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:52 smithi032 ceph-mon[20004]: osdmap e784: 12 total, 12 up, 8 in 2024-08-19T14:42:52.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:52 smithi045 ceph-mon[20143]: pgmap v1871: 57 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 52 active+clean; 2.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 854 op/s; 272/138375 objects degraded (0.197%); 6564/138375 objects misplaced (4.744%); 17 MiB/s, 281 objects/s recovering 2024-08-19T14:42:52.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:52 smithi045 ceph-mon[20143]: osdmap e784: 12 total, 12 up, 8 in 2024-08-19T14:42:52.587 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:42:52.613 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:52.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:52.614 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:42:52.617 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:42:52.715 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:42:52.754 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:52.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:52.756 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:42:52.857 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:42:52.886 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:52.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:52.887 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:42:52.954 INFO:teuthology.orchestra.run.smithi031.stderr:Error EINVAL: specified pg_num 101 < pg_num_max 32 2024-08-19T14:42:52.988 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:42:53.017 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:53.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:53.018 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:42:53.119 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:42:53.146 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:53.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:53.147 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:42:53.248 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:42:53.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:53.285 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:53.285 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:42:53.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:53 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:42:53.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:53 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:42:53.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:53 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3116430503' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "101"}]: dispatch 2024-08-19T14:42:53.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:53 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "101"}]: dispatch 2024-08-19T14:42:53.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:53 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 272/138375 objects degraded (0.197%), 3 pgs degraded (PG_DEGRADED) 2024-08-19T14:42:53.386 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:42:53.418 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:53.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:53.419 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:42:53.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:53 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:42:53.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:53 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:42:53.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:53 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3116430503' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "101"}]: dispatch 2024-08-19T14:42:53.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:53 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "101"}]: dispatch 2024-08-19T14:42:53.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:53 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 272/138375 objects degraded (0.197%), 3 pgs degraded (PG_DEGRADED) 2024-08-19T14:42:53.519 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:42:53.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:53 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:42:53.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:53 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:42:53.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:53 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3116430503' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "101"}]: dispatch 2024-08-19T14:42:53.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:53 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "101"}]: dispatch 2024-08-19T14:42:53.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:53 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 272/138375 objects degraded (0.197%), 3 pgs degraded (PG_DEGRADED) 2024-08-19T14:42:53.547 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:53.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:53.548 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:42:53.649 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:42:53.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:53.676 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:53.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:42:53.777 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:42:53.805 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:53.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:53.807 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:42:53.895 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:42:53.910 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:42:53.972 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:53.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:53.973 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:42:54.074 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:42:54.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:54.107 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:54.107 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:42:54.208 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:42:54.233 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:54.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:54.234 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:42:54.260 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is set 2024-08-19T14:42:54.334 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:42:54.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:54.361 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:54.361 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:42:54.462 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:42:54.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:54.487 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:54.487 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:42:54.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:54 smithi045 ceph-mon[20143]: pgmap v1873: 57 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 3 active+recovering+undersized+degraded+remapped, 52 active+clean; 2.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.13k op/s; 440/127500 objects degraded (0.345%); 6116/127500 objects misplaced (4.797%); 15 MiB/s, 287 objects/s recovering 2024-08-19T14:42:54.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:54 smithi045 ceph-mon[20143]: osdmap e785: 12 total, 12 up, 8 in 2024-08-19T14:42:54.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:54 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/4005179113' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:42:54.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:54 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:42:54.588 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:42:54.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:54 smithi031 ceph-mon[16336]: pgmap v1873: 57 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 3 active+recovering+undersized+degraded+remapped, 52 active+clean; 2.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.13k op/s; 440/127500 objects degraded (0.345%); 6116/127500 objects misplaced (4.797%); 15 MiB/s, 287 objects/s recovering 2024-08-19T14:42:54.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:54 smithi031 ceph-mon[16336]: osdmap e785: 12 total, 12 up, 8 in 2024-08-19T14:42:54.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:54 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/4005179113' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:42:54.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:54 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:42:54.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:54.617 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:54.618 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:42:54.719 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:42:54.745 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:54.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:54.746 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:42:54.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:54 smithi032 ceph-mon[20004]: pgmap v1873: 57 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 3 active+recovering+undersized+degraded+remapped, 52 active+clean; 2.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.13k op/s; 440/127500 objects degraded (0.345%); 6116/127500 objects misplaced (4.797%); 15 MiB/s, 287 objects/s recovering 2024-08-19T14:42:54.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:54 smithi032 ceph-mon[20004]: osdmap e785: 12 total, 12 up, 8 in 2024-08-19T14:42:54.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:54 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/4005179113' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:42:54.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:54 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:42:54.847 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:42:54.877 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:54.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:54.878 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:42:54.979 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:42:55.007 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:55.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:55.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:42:55.109 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:42:55.144 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:55.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:55.145 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:42:55.246 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:42:55.268 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:55.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:55.269 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:42:55.370 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:42:55.402 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:55.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:55.403 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:42:55.504 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:42:55.511 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:55 smithi031 ceph-mon[16336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:42:55.511 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:55 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:42:55.511 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:55 smithi031 ceph-mon[16336]: osdmap e786: 12 total, 12 up, 8 in 2024-08-19T14:42:55.528 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:55.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:55.529 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:42:55.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:55 smithi045 ceph-mon[20143]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:42:55.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:55 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:42:55.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:55 smithi045 ceph-mon[20143]: osdmap e786: 12 total, 12 up, 8 in 2024-08-19T14:42:55.630 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:42:55.644 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:42:55.645 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-08-19T14:42:55.645 INFO:teuthology.orchestra.run.smithi031.stdout: "complaint_time": 30, 2024-08-19T14:42:55.645 INFO:teuthology.orchestra.run.smithi031.stdout: "num_blocked_ops": 0 2024-08-19T14:42:55.645 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:42:55.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:55.665 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:55.666 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:42:55.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:55 smithi032 ceph-mon[20004]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:42:55.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:55 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:42:55.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:55 smithi032 ceph-mon[20004]: osdmap e786: 12 total, 12 up, 8 in 2024-08-19T14:42:55.766 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:42:55.792 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:55.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:55.793 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:42:55.894 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:42:55.917 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:55.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:55.918 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:42:56.019 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:42:56.044 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:56.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:56.045 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:42:56.146 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:42:56.173 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:56.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:56.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:42:56.215 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-19T14:42:56.275 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:42:56.309 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:56.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:56.310 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:42:56.410 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:56 smithi031 ceph-mon[16336]: pgmap v1876: 57 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 52 active+clean; 2.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.31k op/s; 440/122313 objects degraded (0.360%); 4957/122313 objects misplaced (4.053%); 20 MiB/s, 374 objects/s recovering 2024-08-19T14:42:56.411 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:42:56.441 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:56.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:56.442 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:42:56.543 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:42:56.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:56 smithi045 ceph-mon[20143]: pgmap v1876: 57 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 52 active+clean; 2.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.31k op/s; 440/122313 objects degraded (0.360%); 4957/122313 objects misplaced (4.053%); 20 MiB/s, 374 objects/s recovering 2024-08-19T14:42:56.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:56.578 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:56.579 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:42:56.653 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:56 smithi032 ceph-mon[20004]: pgmap v1876: 57 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 52 active+clean; 2.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.31k op/s; 440/122313 objects degraded (0.360%); 4957/122313 objects misplaced (4.053%); 20 MiB/s, 374 objects/s recovering 2024-08-19T14:42:56.680 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:42:56.706 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:56.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:56.708 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:42:56.809 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:42:56.838 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:56.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:56.839 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:42:56.940 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:42:56.966 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:56.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:56.967 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:42:57.068 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:42:57.098 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:57.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:57.100 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:42:57.201 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:42:57.225 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:57.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:57.227 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:42:57.327 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:42:57.353 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:57.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:57.354 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:42:57.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:57 smithi031 ceph-mon[16336]: pgmap v1877: 57 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 3 active+recovering+undersized+degraded+remapped, 52 active+clean; 2.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.22k op/s; 580/116412 objects degraded (0.498%); 4467/116412 objects misplaced (3.837%); 17 MiB/s, 346 objects/s recovering 2024-08-19T14:42:57.455 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:42:57.480 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:57.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:57.481 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:42:57.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:57 smithi045 ceph-mon[20143]: pgmap v1877: 57 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 3 active+recovering+undersized+degraded+remapped, 52 active+clean; 2.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.22k op/s; 580/116412 objects degraded (0.498%); 4467/116412 objects misplaced (3.837%); 17 MiB/s, 346 objects/s recovering 2024-08-19T14:42:57.581 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:42:57.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:57.615 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:57.615 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:42:57.621 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:42:57.629 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-08-19T14:42:57.716 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:42:57.747 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:57.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:57.748 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:42:57.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:57 smithi032 ceph-mon[20004]: pgmap v1877: 57 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 3 active+recovering+undersized+degraded+remapped, 52 active+clean; 2.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.22k op/s; 580/116412 objects degraded (0.498%); 4467/116412 objects misplaced (3.837%); 17 MiB/s, 346 objects/s recovering 2024-08-19T14:42:57.849 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:42:57.887 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:57.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:57.888 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:42:57.989 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:42:58.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:58.023 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:58.023 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:42:58.125 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:42:58.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:58.156 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:58.157 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:42:58.194 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:42:58.195 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:42:58.221 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:42:58.222 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:42:58.254 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:42:58.254 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:42:58.257 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:42:58.261 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:42:58.281 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:42:58.281 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:42:58.284 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:58.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:58.286 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:42:58.311 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:42:58.312 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:42:58.339 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:42:58.339 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:42:58.363 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:42:58.363 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:42:58.387 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:42:58.397 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:42:58.397 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:42:58.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:58.420 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:58.420 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:42:58.422 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:42:58.423 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:42:58.423 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:58 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 580/116412 objects degraded (0.498%), 3 pgs degraded (PG_DEGRADED) 2024-08-19T14:42:58.423 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:58 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3917505427' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:42:58.444 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:42:58.444 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:42:58.464 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:42:58.464 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:42:58.491 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:42:58.492 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:42:58.522 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:42:58.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:58 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 580/116412 objects degraded (0.498%), 3 pgs degraded (PG_DEGRADED) 2024-08-19T14:42:58.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:58 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3917505427' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:42:58.562 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:58.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:58.563 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:42:58.664 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:42:58.692 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:58.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:58.693 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:42:58.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:58 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 580/116412 objects degraded (0.498%), 3 pgs degraded (PG_DEGRADED) 2024-08-19T14:42:58.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:58 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3917505427' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:42:58.794 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:42:58.826 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:58.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:58.829 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:42:58.930 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:42:58.959 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:58.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:58.960 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:42:59.061 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:42:59.105 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:59.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:59.106 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:42:59.109 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-19T14:42:59.207 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:42:59.228 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:59.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:59.229 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:42:59.330 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:42:59.359 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:59.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:59.360 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:42:59.461 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:42:59.469 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:42:59.469 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:42:59.491 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:59.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:59.492 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:42:59.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:42:59 smithi045 ceph-mon[20143]: pgmap v1878: 57 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 3 active+recovering+undersized+degraded+remapped, 52 active+clean; 2.3 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 975 op/s; 580/111756 objects degraded (0.519%); 4467/111756 objects misplaced (3.997%); 12 MiB/s, 218 objects/s recovering 2024-08-19T14:42:59.592 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:42:59.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:42:59 smithi031 ceph-mon[16336]: pgmap v1878: 57 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 3 active+recovering+undersized+degraded+remapped, 52 active+clean; 2.3 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 975 op/s; 580/111756 objects degraded (0.519%); 4467/111756 objects misplaced (3.997%); 12 MiB/s, 218 objects/s recovering 2024-08-19T14:42:59.624 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:42:59.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:59.625 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:42:59.726 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:42:59.749 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:42:59.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:59.750 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:42:59.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:42:59 smithi032 ceph-mon[20004]: pgmap v1878: 57 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 3 active+recovering+undersized+degraded+remapped, 52 active+clean; 2.3 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 975 op/s; 580/111756 objects degraded (0.519%); 4467/111756 objects misplaced (3.997%); 12 MiB/s, 218 objects/s recovering 2024-08-19T14:42:59.851 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:42:59.889 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:42:59.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:42:59.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:42:59.991 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:43:00.034 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:00.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:00.035 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:43:00.137 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:43:00.168 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:00.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:00.169 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:43:00.270 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:43:00.295 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:00.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:00.296 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:43:00.398 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:43:00.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:00.429 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:00.429 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:43:00.530 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:43:00.557 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:00.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:00.558 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:43:00.659 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:43:00.681 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:00.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:00.682 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:43:00.783 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:43:00.813 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:00.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:00.814 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:43:00.915 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:43:00.941 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:00.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:00.943 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:43:01.044 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:43:01.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:01.083 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:01.083 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:43:01.184 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:43:01.216 INFO:tasks.thrashosds.thrasher:in_osds: [1, 4, 5, 7, 8, 2, 11, 9] out_osds: [10, 6, 3, 0] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 8, 9, 11, 10, 2] 2024-08-19T14:43:01.216 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-19T14:43:01.217 INFO:tasks.thrashosds.thrasher:Removing osd 7, in_osds are: [1, 4, 5, 7, 8, 2, 11, 9] 2024-08-19T14:43:01.217 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd out 7 2024-08-19T14:43:01.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:01.221 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:01.221 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:43:01.322 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:43:01.360 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:01.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:01.362 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:43:01.463 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:43:01.519 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:01.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:01.520 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:43:01.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:01 smithi031 ceph-mon[16336]: pgmap v1879: 57 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 52 active+clean; 2.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 862 op/s; 580/106806 objects degraded (0.543%); 3460/106806 objects misplaced (3.240%); 18 MiB/s, 307 objects/s recovering 2024-08-19T14:43:01.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:01 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:43:01.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:01 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:43:01.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:01 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:43:01.620 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:43:01.653 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:01.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:01.654 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:43:01.755 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:43:01.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:01 smithi032 ceph-mon[20004]: pgmap v1879: 57 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 52 active+clean; 2.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 862 op/s; 580/106806 objects degraded (0.543%); 3460/106806 objects misplaced (3.240%); 18 MiB/s, 307 objects/s recovering 2024-08-19T14:43:01.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:01 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:43:01.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:01 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:43:01.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:01 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:43:01.786 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:01.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:01.787 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:43:01.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:01 smithi045 ceph-mon[20143]: pgmap v1879: 57 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 52 active+clean; 2.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 862 op/s; 580/106806 objects degraded (0.543%); 3460/106806 objects misplaced (3.240%); 18 MiB/s, 307 objects/s recovering 2024-08-19T14:43:01.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:01 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:43:01.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:01 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:43:01.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:01 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:43:01.888 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:43:01.913 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:01.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:01.915 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:43:02.016 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:43:02.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:02.058 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:02.058 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:43:02.158 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:43:02.192 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:02.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:02.193 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:43:02.294 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:43:02.322 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:02.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:02.323 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:43:02.424 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:43:02.451 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:02.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:02.453 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:43:02.475 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:02 smithi031 ceph-mon[16336]: osdmap e787: 12 total, 12 up, 8 in 2024-08-19T14:43:02.475 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:02 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:43:02.475 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:02 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:43:02.475 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:02 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:43:02.553 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:43:02.578 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:02.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:02.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:43:02.680 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:43:02.705 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:02.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:02.706 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:43:02.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:02 smithi032 ceph-mon[20004]: osdmap e787: 12 total, 12 up, 8 in 2024-08-19T14:43:02.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:02 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:43:02.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:02 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:43:02.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:02 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:43:02.770 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-08-19T14:43:02.795 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:43:02.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:02 smithi045 ceph-mon[20143]: osdmap e787: 12 total, 12 up, 8 in 2024-08-19T14:43:02.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:02 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:43:02.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:02 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:43:02.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:02 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:43:02.807 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:43:02.818 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:43:02.835 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:02.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:02.836 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:43:02.937 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:43:02.959 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:02.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:02.960 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:43:03.061 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:43:03.083 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:03.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:03.084 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:43:03.127 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:43:03.185 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:43:03.207 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:03.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:03.208 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:43:03.309 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:43:03.333 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:03.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:03.334 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:43:03.435 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:43:03.459 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:03.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:03.461 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:43:03.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:03 smithi032 ceph-mon[20004]: pgmap v1881: 57 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 3 active+recovering+undersized+degraded+remapped, 52 active+clean; 2.0 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.07k op/s; 691/96315 objects degraded (0.717%); 2950/96315 objects misplaced (3.063%); 13 MiB/s, 262 objects/s recovering 2024-08-19T14:43:03.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:03 smithi032 ceph-mon[20004]: osdmap e788: 12 total, 12 up, 8 in 2024-08-19T14:43:03.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:03 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:43:03.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:03 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:43:03.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:03 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:43:03.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:03 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:43:03.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:03 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:43:03.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:03 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:43:03.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:03 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:43:03.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:03 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 691/96315 objects degraded (0.717%), 3 pgs degraded (PG_DEGRADED) 2024-08-19T14:43:03.561 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:43:03.593 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:03.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:03.594 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:43:03.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:03 smithi031 ceph-mon[16336]: pgmap v1881: 57 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 3 active+recovering+undersized+degraded+remapped, 52 active+clean; 2.0 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.07k op/s; 691/96315 objects degraded (0.717%); 2950/96315 objects misplaced (3.063%); 13 MiB/s, 262 objects/s recovering 2024-08-19T14:43:03.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:03 smithi031 ceph-mon[16336]: osdmap e788: 12 total, 12 up, 8 in 2024-08-19T14:43:03.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:03 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:43:03.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:03 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:43:03.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:03 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:43:03.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:03 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:43:03.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:03 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:43:03.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:03 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:43:03.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:03 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:43:03.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:03 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 691/96315 objects degraded (0.717%), 3 pgs degraded (PG_DEGRADED) 2024-08-19T14:43:03.695 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:43:03.719 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:03.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:03.720 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:43:03.745 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:43:03.745 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:43:03.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:03 smithi045 ceph-mon[20143]: pgmap v1881: 57 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 3 active+recovering+undersized+degraded+remapped, 52 active+clean; 2.0 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.07k op/s; 691/96315 objects degraded (0.717%); 2950/96315 objects misplaced (3.063%); 13 MiB/s, 262 objects/s recovering 2024-08-19T14:43:03.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:03 smithi045 ceph-mon[20143]: osdmap e788: 12 total, 12 up, 8 in 2024-08-19T14:43:03.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:03 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:43:03.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:03 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:43:03.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:03 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:43:03.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:03 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:43:03.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:03 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:43:03.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:03 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:43:03.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:03 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:43:03.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:03 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 691/96315 objects degraded (0.717%), 3 pgs degraded (PG_DEGRADED) 2024-08-19T14:43:03.821 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:43:03.850 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:03.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:03.851 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:43:03.952 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:43:03.981 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:03.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:03.982 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:43:04.083 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:43:04.124 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:04.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:04.125 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:43:04.226 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:43:04.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:04.257 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:04.258 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:43:04.359 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:43:04.405 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:04.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:04.406 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:43:04.426 INFO:teuthology.orchestra.run.smithi031.stderr:marked out osd.7. 2024-08-19T14:43:04.432 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is unset 2024-08-19T14:43:04.507 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:43:04.531 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:04.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:04.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:43:04.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:04 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mon-a[16332]: 2024-08-19T14:43:04.213+0000 7f2ff503e640 -1 mon.a@0(leader).osd e789 definitely_dead 0 2024-08-19T14:43:04.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:04 smithi031 ceph-mon[16336]: osdmap e789: 12 total, 12 up, 8 in 2024-08-19T14:43:04.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:04 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/4163352547' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:43:04.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:04 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:43:04.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:04 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2216917011' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2024-08-19T14:43:04.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:04 smithi031 ceph-mon[16336]: Client client.admin marked osd.7 out, while it was still marked up 2024-08-19T14:43:04.633 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:43:04.656 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:04.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:04.657 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:43:04.689 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:43:04.758 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:43:04.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:04 smithi032 ceph-mon[20004]: osdmap e789: 12 total, 12 up, 8 in 2024-08-19T14:43:04.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:04 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/4163352547' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:43:04.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:04 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:43:04.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:04 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2216917011' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2024-08-19T14:43:04.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:04 smithi032 ceph-mon[20004]: Client client.admin marked osd.7 out, while it was still marked up 2024-08-19T14:43:04.781 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:04.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:04.783 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:43:04.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:04 smithi045 ceph-mon[20143]: osdmap e789: 12 total, 12 up, 8 in 2024-08-19T14:43:04.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:04 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/4163352547' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:43:04.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:04 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:43:04.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:04 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2216917011' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2024-08-19T14:43:04.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:04 smithi045 ceph-mon[20143]: Client client.admin marked osd.7 out, while it was still marked up 2024-08-19T14:43:04.883 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:43:04.906 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:04.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:04.908 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:43:05.009 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:43:05.032 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:05.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:05.033 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:43:05.134 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:43:05.155 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:05.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:05.156 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:43:05.256 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:43:05.279 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:05.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:05.280 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:43:05.381 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:43:05.403 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:05.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:05.404 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:43:05.505 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:43:05.532 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:05.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:05.533 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:43:05.634 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:43:05.661 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:05.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:05.662 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:43:05.747 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:05 smithi032 ceph-mon[20004]: pgmap v1884: 57 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 53 active+clean; 1.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s; 691/92637 objects degraded (0.746%); 1620/92637 objects misplaced (1.749%); 27 MiB/s, 479 objects/s recovering 2024-08-19T14:43:05.747 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:05 smithi032 ceph-mon[20004]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:43:05.747 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:05 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:43:05.747 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:05 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2216917011' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2024-08-19T14:43:05.747 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:05 smithi032 ceph-mon[20004]: osdmap e790: 12 total, 12 up, 7 in 2024-08-19T14:43:05.747 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:05 smithi032 ceph-mon[20004]: 1.1c scrub starts 2024-08-19T14:43:05.747 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:05 smithi032 ceph-mon[20004]: 1.1c scrub ok 2024-08-19T14:43:05.748 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:05 smithi031 ceph-mon[16336]: pgmap v1884: 57 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 53 active+clean; 1.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s; 691/92637 objects degraded (0.746%); 1620/92637 objects misplaced (1.749%); 27 MiB/s, 479 objects/s recovering 2024-08-19T14:43:05.748 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:05 smithi031 ceph-mon[16336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:43:05.748 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:05 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:43:05.748 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:05 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2216917011' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2024-08-19T14:43:05.748 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:05 smithi031 ceph-mon[16336]: osdmap e790: 12 total, 12 up, 7 in 2024-08-19T14:43:05.748 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:05 smithi031 ceph-mon[16336]: 1.1c scrub starts 2024-08-19T14:43:05.748 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:05 smithi031 ceph-mon[16336]: 1.1c scrub ok 2024-08-19T14:43:05.763 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:43:05.787 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:05.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:05.788 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:43:05.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:05 smithi045 ceph-mon[20143]: pgmap v1884: 57 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 53 active+clean; 1.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s; 691/92637 objects degraded (0.746%); 1620/92637 objects misplaced (1.749%); 27 MiB/s, 479 objects/s recovering 2024-08-19T14:43:05.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:05 smithi045 ceph-mon[20143]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:43:05.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:05 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:43:05.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:05 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2216917011' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2024-08-19T14:43:05.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:05 smithi045 ceph-mon[20143]: osdmap e790: 12 total, 12 up, 7 in 2024-08-19T14:43:05.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:05 smithi045 ceph-mon[20143]: 1.1c scrub starts 2024-08-19T14:43:05.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:05 smithi045 ceph-mon[20143]: 1.1c scrub ok 2024-08-19T14:43:05.887 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:43:05.889 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:43:05.913 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:05.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:05.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:43:06.015 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:43:06.038 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:06.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:06.039 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:43:06.140 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:43:06.165 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:06.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:06.167 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:43:06.268 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:43:06.293 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:06.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:06.294 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:43:06.395 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:43:06.418 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:06.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:06.419 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:43:06.450 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:43:06.451 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:43:06.478 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:43:06.479 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:43:06.502 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:43:06.502 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:43:06.519 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:43:06.528 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:43:06.528 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:43:06.546 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:43:06.547 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:43:06.560 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:06.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:06.562 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:43:06.571 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:43:06.571 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:43:06.594 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:43:06.594 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:43:06.594 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:06 smithi031 ceph-mon[16336]: 1.16 scrub starts 2024-08-19T14:43:06.595 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:06 smithi031 ceph-mon[16336]: 1.16 scrub ok 2024-08-19T14:43:06.595 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:06 smithi031 ceph-mon[16336]: osdmap e791: 12 total, 12 up, 7 in 2024-08-19T14:43:06.595 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:06 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/170998000' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:43:06.613 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:43:06.614 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:43:06.634 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:43:06.635 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:43:06.658 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:43:06.658 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:43:06.663 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:43:06.678 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:43:06.678 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:43:06.696 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:06.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:06.697 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:43:06.700 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:43:06.700 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:43:06.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:06 smithi032 ceph-mon[20004]: 1.16 scrub starts 2024-08-19T14:43:06.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:06 smithi032 ceph-mon[20004]: 1.16 scrub ok 2024-08-19T14:43:06.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:06 smithi032 ceph-mon[20004]: osdmap e791: 12 total, 12 up, 7 in 2024-08-19T14:43:06.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:06 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/170998000' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:43:06.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:06 smithi045 ceph-mon[20143]: 1.16 scrub starts 2024-08-19T14:43:06.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:06 smithi045 ceph-mon[20143]: 1.16 scrub ok 2024-08-19T14:43:06.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:06 smithi045 ceph-mon[20143]: osdmap e791: 12 total, 12 up, 7 in 2024-08-19T14:43:06.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:06 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/170998000' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:43:06.798 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:43:06.832 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:06.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:06.833 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:43:06.934 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:43:06.956 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:06.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:06.957 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:43:07.055 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-08-19T14:43:07.058 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:43:07.084 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:07.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:07.085 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:43:07.186 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:43:07.218 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:07.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:07.219 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:43:07.320 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:43:07.349 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:07.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:07.350 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:43:07.451 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:43:07.482 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:07.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:07.483 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:43:07.487 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:43:07.583 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:43:07.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:07.615 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:07.615 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:43:07.716 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:43:07.739 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:07.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:07.740 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:43:07.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:07 smithi032 ceph-mon[20004]: pgmap v1887: 57 pgs: 1 active+recovering+remapped, 4 remapped+peering, 2 active+recovering+undersized+degraded+remapped, 50 active+clean; 1.8 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 767 B/s rd, 17 KiB/s wr, 499 op/s; 10/90345 objects degraded (0.011%); 1229/90345 objects misplaced (1.360%); 24 MiB/s, 583 objects/s recovering 2024-08-19T14:43:07.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:07 smithi032 ceph-mon[20004]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-08-19T14:43:07.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:07 smithi032 ceph-mon[20004]: 8.c starting backfill to osd.11 from (0'0,0'0] MAX to 789'12439 2024-08-19T14:43:07.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:07 smithi032 ceph-mon[20004]: 8.d starting backfill to osd.5 from (0'0,0'0] MAX to 789'12279 2024-08-19T14:43:07.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:07 smithi032 ceph-mon[20004]: osdmap e792: 12 total, 12 up, 7 in 2024-08-19T14:43:07.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:07 smithi032 ceph-mon[20004]: 8.e starting backfill to osd.11 from (0'0,0'0] MAX to 789'12312 2024-08-19T14:43:07.778 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-19T14:43:07.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:07 smithi045 ceph-mon[20143]: pgmap v1887: 57 pgs: 1 active+recovering+remapped, 4 remapped+peering, 2 active+recovering+undersized+degraded+remapped, 50 active+clean; 1.8 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 767 B/s rd, 17 KiB/s wr, 499 op/s; 10/90345 objects degraded (0.011%); 1229/90345 objects misplaced (1.360%); 24 MiB/s, 583 objects/s recovering 2024-08-19T14:43:07.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:07 smithi045 ceph-mon[20143]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-08-19T14:43:07.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:07 smithi045 ceph-mon[20143]: 8.c starting backfill to osd.11 from (0'0,0'0] MAX to 789'12439 2024-08-19T14:43:07.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:07 smithi045 ceph-mon[20143]: 8.d starting backfill to osd.5 from (0'0,0'0] MAX to 789'12279 2024-08-19T14:43:07.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:07 smithi045 ceph-mon[20143]: osdmap e792: 12 total, 12 up, 7 in 2024-08-19T14:43:07.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:07 smithi045 ceph-mon[20143]: 8.e starting backfill to osd.11 from (0'0,0'0] MAX to 789'12312 2024-08-19T14:43:07.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:07 smithi031 ceph-mon[16336]: pgmap v1887: 57 pgs: 1 active+recovering+remapped, 4 remapped+peering, 2 active+recovering+undersized+degraded+remapped, 50 active+clean; 1.8 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 767 B/s rd, 17 KiB/s wr, 499 op/s; 10/90345 objects degraded (0.011%); 1229/90345 objects misplaced (1.360%); 24 MiB/s, 583 objects/s recovering 2024-08-19T14:43:07.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:07 smithi031 ceph-mon[16336]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-08-19T14:43:07.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:07 smithi031 ceph-mon[16336]: 8.c starting backfill to osd.11 from (0'0,0'0] MAX to 789'12439 2024-08-19T14:43:07.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:07 smithi031 ceph-mon[16336]: 8.d starting backfill to osd.5 from (0'0,0'0] MAX to 789'12279 2024-08-19T14:43:07.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:07 smithi031 ceph-mon[16336]: osdmap e792: 12 total, 12 up, 7 in 2024-08-19T14:43:07.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:07 smithi031 ceph-mon[16336]: 8.e starting backfill to osd.11 from (0'0,0'0] MAX to 789'12312 2024-08-19T14:43:07.841 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:43:07.872 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:07.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:07.873 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:43:07.974 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:43:08.005 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:08.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:08.006 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:43:08.091 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:43:08.106 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:43:08.130 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:08.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:08.131 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:43:08.233 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:43:08.262 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:08.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:08.264 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:43:08.365 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:43:08.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:08.387 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:08.388 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:43:08.489 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:43:08.524 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:08.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:08.525 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:43:08.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:08 smithi031 ceph-mon[16336]: osdmap e793: 12 total, 12 up, 7 in 2024-08-19T14:43:08.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:08 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 10/90345 objects degraded (0.011%), 2 pgs degraded (PG_DEGRADED) 2024-08-19T14:43:08.626 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:43:08.655 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:08.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:08.656 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:43:08.757 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:43:08.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:08 smithi032 ceph-mon[20004]: osdmap e793: 12 total, 12 up, 7 in 2024-08-19T14:43:08.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:08 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 10/90345 objects degraded (0.011%), 2 pgs degraded (PG_DEGRADED) 2024-08-19T14:43:08.780 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:08.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:08.781 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:43:08.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:08 smithi045 ceph-mon[20143]: osdmap e793: 12 total, 12 up, 7 in 2024-08-19T14:43:08.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:08 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 10/90345 objects degraded (0.011%), 2 pgs degraded (PG_DEGRADED) 2024-08-19T14:43:08.882 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:43:08.915 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:08.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:08.917 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:43:09.018 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:43:09.048 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:09.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:09.049 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:43:09.150 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:43:09.179 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:09.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:09.180 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:43:09.281 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:43:09.308 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:09.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:09.309 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:43:09.410 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:43:09.440 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:09.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:09.441 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:43:09.454 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:43:09.542 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:43:09.572 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:09.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:09.573 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:43:09.674 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:43:09.697 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:09.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:09.698 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:43:09.753 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:43:09.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:09 smithi032 ceph-mon[20004]: pgmap v1890: 57 pgs: 1 active+remapped+backfilling, 3 active+undersized+remapped, 1 active+recovering+remapped, 4 remapped+peering, 4 active+recovering+undersized+degraded+remapped, 44 active+clean; 1.8 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 767 B/s rd, 17 KiB/s wr, 333 op/s; 12/88656 objects degraded (0.014%); 4695/88656 objects misplaced (5.296%); 7.2 MiB/s, 308 objects/s recovering 2024-08-19T14:43:09.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:09 smithi032 ceph-mon[20004]: osdmap e794: 12 total, 12 up, 7 in 2024-08-19T14:43:09.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:09 smithi045 ceph-mon[20143]: pgmap v1890: 57 pgs: 1 active+remapped+backfilling, 3 active+undersized+remapped, 1 active+recovering+remapped, 4 remapped+peering, 4 active+recovering+undersized+degraded+remapped, 44 active+clean; 1.8 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 767 B/s rd, 17 KiB/s wr, 333 op/s; 12/88656 objects degraded (0.014%); 4695/88656 objects misplaced (5.296%); 7.2 MiB/s, 308 objects/s recovering 2024-08-19T14:43:09.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:09 smithi045 ceph-mon[20143]: osdmap e794: 12 total, 12 up, 7 in 2024-08-19T14:43:09.799 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:43:09.829 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:09.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:09.831 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:43:09.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:09 smithi031 ceph-mon[16336]: pgmap v1890: 57 pgs: 1 active+remapped+backfilling, 3 active+undersized+remapped, 1 active+recovering+remapped, 4 remapped+peering, 4 active+recovering+undersized+degraded+remapped, 44 active+clean; 1.8 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 767 B/s rd, 17 KiB/s wr, 333 op/s; 12/88656 objects degraded (0.014%); 4695/88656 objects misplaced (5.296%); 7.2 MiB/s, 308 objects/s recovering 2024-08-19T14:43:09.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:09 smithi031 ceph-mon[16336]: osdmap e794: 12 total, 12 up, 7 in 2024-08-19T14:43:09.932 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:43:09.959 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:09.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:09.960 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:43:10.062 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:43:10.084 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:10.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:10.085 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:43:10.145 INFO:tasks.thrashosds.thrasher:in_osds: [1, 4, 5, 8, 2, 11, 9] out_osds: [10, 6, 3, 0, 7] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 8, 9, 11, 10, 2] 2024-08-19T14:43:10.145 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-19T14:43:10.145 INFO:tasks.thrashosds.thrasher:Adding osd 10 2024-08-19T14:43:10.145 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd in 10 2024-08-19T14:43:10.186 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:43:10.211 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:10.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:10.212 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:43:10.281 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:43:10.281 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:43:10.300 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:43:10.300 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:43:10.313 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:43:10.319 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:43:10.320 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:43:10.344 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:10.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:10.345 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:43:10.355 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:43:10.355 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:43:10.379 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:43:10.380 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:43:10.399 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:43:10.400 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:43:10.422 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:43:10.423 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:43:10.446 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:43:10.447 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:43:10.447 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:43:10.461 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is unset 2024-08-19T14:43:10.466 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:43:10.474 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:43:10.475 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:43:10.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:10.480 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:10.481 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:43:10.504 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:43:10.504 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:43:10.530 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:43:10.530 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:43:10.557 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:43:10.557 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:43:10.582 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:43:10.612 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:10.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:10.614 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:43:10.714 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:43:10.744 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:10.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:10.746 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:43:10.754 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:10 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2980103446' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:43:10.754 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:10 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2789132377' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:43:10.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:10 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2980103446' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:43:10.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:10 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2789132377' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:43:10.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:10 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2980103446' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:43:10.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:10 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2789132377' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:43:10.846 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:43:10.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:10.883 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:10.883 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:43:10.984 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:43:11.007 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:11.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:11.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:43:11.109 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:43:11.141 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:11.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:11.142 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:43:11.243 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:43:11.265 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:11.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:11.267 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:43:11.368 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:43:11.393 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:11.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:11.394 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:43:11.495 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:43:11.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:11.526 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:11.526 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:43:11.627 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:43:11.629 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:11 smithi031 ceph-mon[16336]: pgmap v1892: 57 pgs: 1 activating+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 activating+undersized+remapped, 2 active+remapped+backfilling, 4 active+undersized+remapped, 1 active+recovering+remapped, 4 remapped+peering, 4 active+recovering+undersized+degraded+remapped, 38 active+clean; 1.8 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 1.6 KiB/s rd, 41 KiB/s wr, 385 op/s; 3531/87030 objects degraded (4.057%); 7034/87030 objects misplaced (8.082%); 28 MiB/s, 709 objects/s recovering 2024-08-19T14:43:11.629 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:11 smithi031 ceph-mon[16336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:43:11.629 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:11 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2980103446' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:43:11.629 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:11 smithi031 ceph-mon[16336]: osdmap e795: 12 total, 12 up, 7 in 2024-08-19T14:43:11.665 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:11.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:11.666 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:43:11.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:11 smithi032 ceph-mon[20004]: pgmap v1892: 57 pgs: 1 activating+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 activating+undersized+remapped, 2 active+remapped+backfilling, 4 active+undersized+remapped, 1 active+recovering+remapped, 4 remapped+peering, 4 active+recovering+undersized+degraded+remapped, 38 active+clean; 1.8 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 1.6 KiB/s rd, 41 KiB/s wr, 385 op/s; 3531/87030 objects degraded (4.057%); 7034/87030 objects misplaced (8.082%); 28 MiB/s, 709 objects/s recovering 2024-08-19T14:43:11.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:11 smithi032 ceph-mon[20004]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:43:11.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:11 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2980103446' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:43:11.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:11 smithi032 ceph-mon[20004]: osdmap e795: 12 total, 12 up, 7 in 2024-08-19T14:43:11.767 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:43:11.796 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:11.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:11 smithi045 ceph-mon[20143]: pgmap v1892: 57 pgs: 1 activating+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 activating+undersized+remapped, 2 active+remapped+backfilling, 4 active+undersized+remapped, 1 active+recovering+remapped, 4 remapped+peering, 4 active+recovering+undersized+degraded+remapped, 38 active+clean; 1.8 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 1.6 KiB/s rd, 41 KiB/s wr, 385 op/s; 3531/87030 objects degraded (4.057%); 7034/87030 objects misplaced (8.082%); 28 MiB/s, 709 objects/s recovering 2024-08-19T14:43:11.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:11 smithi045 ceph-mon[20143]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:43:11.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:11 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2980103446' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:43:11.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:11 smithi045 ceph-mon[20143]: osdmap e795: 12 total, 12 up, 7 in 2024-08-19T14:43:11.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:11.799 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:43:11.899 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:43:11.933 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:11.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:11.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:43:12.035 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:43:12.059 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:12.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:12.061 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:43:12.161 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:43:12.192 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:12.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:12.193 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:43:12.294 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:43:12.317 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:12.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:12.318 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:43:12.361 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:43:12.361 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:43:12.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:12 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mon-a[16332]: 2024-08-19T14:43:12.129+0000 7f2ff503e640 -1 mon.a@0(leader).osd e796 definitely_dead 0 2024-08-19T14:43:12.420 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:43:12.447 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:12.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:12.448 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:43:12.466 INFO:teuthology.orchestra.run.smithi031.stderr:marked in osd.10. 2024-08-19T14:43:12.549 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:43:12.571 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:12.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:12.572 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:43:12.673 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:43:12.701 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:12.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:12.702 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:43:12.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:12 smithi032 ceph-mon[20004]: osdmap e796: 12 total, 12 up, 7 in 2024-08-19T14:43:12.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:12 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/225137108' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["10"]}]: dispatch 2024-08-19T14:43:12.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:12 smithi045 ceph-mon[20143]: osdmap e796: 12 total, 12 up, 7 in 2024-08-19T14:43:12.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:12 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/225137108' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["10"]}]: dispatch 2024-08-19T14:43:12.803 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:43:12.825 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:12.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:12.826 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:43:12.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:12 smithi031 ceph-mon[16336]: osdmap e796: 12 total, 12 up, 7 in 2024-08-19T14:43:12.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:12 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/225137108' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["10"]}]: dispatch 2024-08-19T14:43:12.927 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:43:12.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:12.953 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:12.953 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:43:13.055 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:43:13.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:13.089 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:13.089 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:43:13.109 INFO:tasks.thrashosds.thrasher:Added osd 10 2024-08-19T14:43:13.190 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:43:13.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:13.218 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:13.219 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:43:13.320 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:43:13.341 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:13.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:13.342 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:43:13.411 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:43:13.443 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:43:13.483 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:13.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:13.484 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:43:13.585 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:43:13.610 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:13.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:13.612 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:43:13.713 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:43:13.723 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:43:13.738 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:13.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:13.739 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:43:13.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:13 smithi032 ceph-mon[20004]: pgmap v1895: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 peering, 1 active+recovering+remapped, 1 activating+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 activating+undersized+remapped, 1 active+remapped+backfilling, 1 active+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.8 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.0 KiB/s rd, 27 KiB/s wr, 167 op/s; 8800/86247 objects degraded (10.203%); 4321/86247 objects misplaced (5.010%); 80 MiB/s, 1.56k objects/s recovering 2024-08-19T14:43:13.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:13 smithi032 ceph-mon[20004]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-08-19T14:43:13.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:13 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/225137108' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["10"]}]': finished 2024-08-19T14:43:13.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:13 smithi032 ceph-mon[20004]: osdmap e797: 12 total, 12 up, 8 in 2024-08-19T14:43:13.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:13 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 8800/86247 objects degraded (10.203%), 6 pgs degraded (PG_DEGRADED) 2024-08-19T14:43:13.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:13 smithi032 ceph-mon[20004]: osdmap e798: 12 total, 12 up, 8 in 2024-08-19T14:43:13.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:13 smithi045 ceph-mon[20143]: pgmap v1895: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 peering, 1 active+recovering+remapped, 1 activating+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 activating+undersized+remapped, 1 active+remapped+backfilling, 1 active+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.8 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.0 KiB/s rd, 27 KiB/s wr, 167 op/s; 8800/86247 objects degraded (10.203%); 4321/86247 objects misplaced (5.010%); 80 MiB/s, 1.56k objects/s recovering 2024-08-19T14:43:13.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:13 smithi045 ceph-mon[20143]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-08-19T14:43:13.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:13 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/225137108' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["10"]}]': finished 2024-08-19T14:43:13.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:13 smithi045 ceph-mon[20143]: osdmap e797: 12 total, 12 up, 8 in 2024-08-19T14:43:13.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:13 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 8800/86247 objects degraded (10.203%), 6 pgs degraded (PG_DEGRADED) 2024-08-19T14:43:13.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:13 smithi045 ceph-mon[20143]: osdmap e798: 12 total, 12 up, 8 in 2024-08-19T14:43:13.841 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:43:13.848 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:13 smithi031 ceph-mon[16336]: pgmap v1895: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 peering, 1 active+recovering+remapped, 1 activating+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 activating+undersized+remapped, 1 active+remapped+backfilling, 1 active+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.8 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.0 KiB/s rd, 27 KiB/s wr, 167 op/s; 8800/86247 objects degraded (10.203%); 4321/86247 objects misplaced (5.010%); 80 MiB/s, 1.56k objects/s recovering 2024-08-19T14:43:13.849 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:13 smithi031 ceph-mon[16336]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-08-19T14:43:13.849 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:13 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/225137108' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["10"]}]': finished 2024-08-19T14:43:13.849 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:13 smithi031 ceph-mon[16336]: osdmap e797: 12 total, 12 up, 8 in 2024-08-19T14:43:13.849 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:13 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 8800/86247 objects degraded (10.203%), 6 pgs degraded (PG_DEGRADED) 2024-08-19T14:43:13.849 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:13 smithi031 ceph-mon[16336]: osdmap e798: 12 total, 12 up, 8 in 2024-08-19T14:43:13.864 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:13.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:13.865 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:43:13.967 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:43:13.994 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:13.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:13.995 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:43:14.096 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:43:14.132 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:14.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:14.133 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:43:14.234 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:43:14.255 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:14.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:14.256 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:43:14.357 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:43:14.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:14.393 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:14.393 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:43:14.453 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-19T14:43:14.494 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:43:14.526 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:14.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:14.527 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:43:14.628 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:43:14.655 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:14.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:14.657 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:43:14.758 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:43:14.781 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:14.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:14.782 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:43:14.883 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:43:14.915 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:14.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:14.916 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:43:15.017 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:43:15.042 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:15.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:15.043 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:43:15.144 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:43:15.170 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:15.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:15.171 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:43:15.200 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:43:15.200 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:43:15.200 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:43:15.200 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:43:15.200 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:43:15.201 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:43:15.201 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:43:15.201 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:43:15.201 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:43:15.201 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:43:15.201 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:43:15.201 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:43:15.201 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:43:15.201 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:43:15.201 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:43:15.202 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:43:15.202 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:43:15.202 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:43:15.202 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:43:15.202 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:43:15.202 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:43:15.202 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:43:15.202 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:43:15.202 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:43:15.272 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:43:15.298 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:15.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:15.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:43:15.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:15 smithi031 ceph-mon[16336]: osdmap e799: 12 total, 12 up, 8 in 2024-08-19T14:43:15.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:15 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/4190459655' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:43:15.400 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:43:15.428 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:15.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:15.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:43:15.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:15 smithi032 ceph-mon[20004]: osdmap e799: 12 total, 12 up, 8 in 2024-08-19T14:43:15.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:15 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/4190459655' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:43:15.530 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:43:15.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:15 smithi045 ceph-mon[20143]: osdmap e799: 12 total, 12 up, 8 in 2024-08-19T14:43:15.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:15 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/4190459655' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:43:15.578 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:15.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:15.579 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:43:15.680 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:43:15.732 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:15.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:15.733 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:43:15.834 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:43:15.859 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:15.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:15.860 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:43:15.961 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:43:16.006 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:16.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:16.007 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:43:16.108 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:43:16.141 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:16.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:16.142 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:43:16.243 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:43:16.269 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:16.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:16.270 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:43:16.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:16 smithi031 ceph-mon[16336]: pgmap v1899: 57 pgs: 4 remapped+peering, 1 active+recovery_wait+undersized+degraded+remapped, 2 peering, 1 active+recovering+remapped, 1 activating+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 activating+undersized+remapped, 1 active+remapped+backfilling, 1 active+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 39 active+clean; 1.8 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 84 op/s; 8800/86049 objects degraded (10.227%); 3769/86049 objects misplaced (4.380%); 82 MiB/s, 1.57k objects/s recovering 2024-08-19T14:43:16.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:16 smithi031 ceph-mon[16336]: osdmap e800: 12 total, 12 up, 8 in 2024-08-19T14:43:16.371 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:43:16.398 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:16.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:16.399 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:43:16.500 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:43:16.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:16 smithi032 ceph-mon[20004]: pgmap v1899: 57 pgs: 4 remapped+peering, 1 active+recovery_wait+undersized+degraded+remapped, 2 peering, 1 active+recovering+remapped, 1 activating+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 activating+undersized+remapped, 1 active+remapped+backfilling, 1 active+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 39 active+clean; 1.8 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 84 op/s; 8800/86049 objects degraded (10.227%); 3769/86049 objects misplaced (4.380%); 82 MiB/s, 1.57k objects/s recovering 2024-08-19T14:43:16.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:16 smithi032 ceph-mon[20004]: osdmap e800: 12 total, 12 up, 8 in 2024-08-19T14:43:16.526 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:16.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:16.528 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:43:16.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:16 smithi045 ceph-mon[20143]: pgmap v1899: 57 pgs: 4 remapped+peering, 1 active+recovery_wait+undersized+degraded+remapped, 2 peering, 1 active+recovering+remapped, 1 activating+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 activating+undersized+remapped, 1 active+remapped+backfilling, 1 active+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 39 active+clean; 1.8 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 84 op/s; 8800/86049 objects degraded (10.227%); 3769/86049 objects misplaced (4.380%); 82 MiB/s, 1.57k objects/s recovering 2024-08-19T14:43:16.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:16 smithi045 ceph-mon[20143]: osdmap e800: 12 total, 12 up, 8 in 2024-08-19T14:43:16.628 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:43:16.654 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:16.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:16.655 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:43:16.756 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:43:16.778 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:16.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:16.779 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:43:16.880 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:43:16.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:16.904 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:16.904 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:43:17.006 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:43:17.034 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:17.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:17.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:43:17.136 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:43:17.172 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:17.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:17.173 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:43:17.274 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:43:17.298 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:17.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:17.299 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:43:17.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:17 smithi031 ceph-mon[16336]: osdmap e801: 12 total, 12 up, 8 in 2024-08-19T14:43:17.400 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:43:17.430 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:17.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:17.432 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:43:17.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:17 smithi032 ceph-mon[20004]: osdmap e801: 12 total, 12 up, 8 in 2024-08-19T14:43:17.532 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:43:17.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:17 smithi045 ceph-mon[20143]: osdmap e801: 12 total, 12 up, 8 in 2024-08-19T14:43:17.558 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:17.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:17.559 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:43:17.661 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:43:17.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:17.687 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:17.687 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:43:17.788 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:43:17.813 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:17.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:17.814 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:43:17.915 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:43:17.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:17.938 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:17.938 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:43:18.039 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:43:18.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:18.077 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:18.077 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:43:18.110 INFO:tasks.thrashosds.thrasher:in_osds: [1, 4, 5, 8, 2, 11, 9, 10] out_osds: [6, 3, 0, 7] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 8, 9, 11, 10, 2] 2024-08-19T14:43:18.110 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-19T14:43:18.110 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2024-08-19T14:43:18.110 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-08-19T14:43:18.110 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-08-19T14:43:18.111 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.4 config set filestore_inject_stall 3 2024-08-19T14:43:18.178 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:43:18.202 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:18.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:18.203 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:43:18.304 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:43:18.341 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:18.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:18.342 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:43:18.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:18 smithi031 ceph-mon[16336]: pgmap v1902: 57 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 3 remapped+peering, 9 peering, 7 active+recovering+undersized+degraded+remapped, 36 active+clean; 1.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 529 B/s rd, 17 KiB/s wr, 174 op/s; 51/84237 objects degraded (0.061%); 4667/84237 objects misplaced (5.540%); 51 MiB/s, 1.06k objects/s recovering 2024-08-19T14:43:18.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:18 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 51/84237 objects degraded (0.061%), 7 pgs degraded (PG_DEGRADED) 2024-08-19T14:43:18.443 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:43:18.470 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:18.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:18.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:43:18.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:18 smithi032 ceph-mon[20004]: pgmap v1902: 57 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 3 remapped+peering, 9 peering, 7 active+recovering+undersized+degraded+remapped, 36 active+clean; 1.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 529 B/s rd, 17 KiB/s wr, 174 op/s; 51/84237 objects degraded (0.061%); 4667/84237 objects misplaced (5.540%); 51 MiB/s, 1.06k objects/s recovering 2024-08-19T14:43:18.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:18 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 51/84237 objects degraded (0.061%), 7 pgs degraded (PG_DEGRADED) 2024-08-19T14:43:18.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:18 smithi045 ceph-mon[20143]: pgmap v1902: 57 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 3 remapped+peering, 9 peering, 7 active+recovering+undersized+degraded+remapped, 36 active+clean; 1.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 529 B/s rd, 17 KiB/s wr, 174 op/s; 51/84237 objects degraded (0.061%); 4667/84237 objects misplaced (5.540%); 51 MiB/s, 1.06k objects/s recovering 2024-08-19T14:43:18.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:18 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 51/84237 objects degraded (0.061%), 7 pgs degraded (PG_DEGRADED) 2024-08-19T14:43:18.572 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:43:18.599 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:18.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:18.600 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:43:18.701 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:43:18.749 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:18.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:18.750 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:43:18.851 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:43:18.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:18.902 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:18.903 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:43:19.004 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:43:19.050 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:19.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:19.051 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:43:19.152 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:43:19.183 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:19.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:19.184 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:43:19.285 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:43:19.331 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:19.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:19.332 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:43:19.433 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:43:19.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:19.457 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:19.457 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:43:19.543 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-08-19T14:43:19.558 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:43:19.582 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:43:19.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:19.595 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:19.595 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:43:19.697 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:43:19.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:19.722 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:19.723 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:43:19.824 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:43:19.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:19.863 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:19.864 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:43:19.965 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:43:19.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:19.994 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:19.994 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:43:20.095 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:43:20.120 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:20.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:20.121 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:43:20.222 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:43:20.249 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:20.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:20.250 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:43:20.351 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:43:20.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:20 smithi031 ceph-mon[16336]: pgmap v1903: 57 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 3 remapped+peering, 8 peering, 9 active+recovering+undersized+degraded+remapped, 35 active+clean; 1.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 388 B/s rd, 13 KiB/s wr, 243 op/s; 59/82410 objects degraded (0.072%); 6696/82410 objects misplaced (8.125%); 47 MiB/s, 936 objects/s recovering 2024-08-19T14:43:20.382 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:20.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:20.383 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:43:20.484 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:43:20.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:20 smithi032 ceph-mon[20004]: pgmap v1903: 57 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 3 remapped+peering, 8 peering, 9 active+recovering+undersized+degraded+remapped, 35 active+clean; 1.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 388 B/s rd, 13 KiB/s wr, 243 op/s; 59/82410 objects degraded (0.072%); 6696/82410 objects misplaced (8.125%); 47 MiB/s, 936 objects/s recovering 2024-08-19T14:43:20.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:20.515 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:20.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:43:20.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:20 smithi045 ceph-mon[20143]: pgmap v1903: 57 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 3 remapped+peering, 8 peering, 9 active+recovering+undersized+degraded+remapped, 35 active+clean; 1.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 388 B/s rd, 13 KiB/s wr, 243 op/s; 59/82410 objects degraded (0.072%); 6696/82410 objects misplaced (8.125%); 47 MiB/s, 936 objects/s recovering 2024-08-19T14:43:20.616 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:43:20.646 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:20.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:20.647 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:43:20.748 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:43:20.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:20.788 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:20.788 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:43:20.890 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:43:20.916 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:20.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:20.917 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:43:21.018 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:43:21.059 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:21.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:21.060 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:43:21.161 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:43:21.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:21.198 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:21.198 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:43:21.299 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:43:21.345 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:21.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:21.346 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:43:21.447 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:43:21.475 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:21.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:21.477 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:43:21.578 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:43:21.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:21.622 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:21.622 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:43:21.723 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:43:21.748 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:21.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:21.749 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:43:21.849 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:43:21.874 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:21.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:21.875 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:43:21.976 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:43:22.009 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:22.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:22.010 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:43:22.111 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:43:22.137 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:22.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:22.138 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:43:22.239 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:43:22.267 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:22.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:22.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:43:22.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:22 smithi031 ceph-mon[16336]: pgmap v1904: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 1 peering, 10 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 326 B/s rd, 11 KiB/s wr, 382 op/s; 3518/78876 objects degraded (4.460%); 6647/78876 objects misplaced (8.427%); 43 MiB/s, 970 objects/s recovering 2024-08-19T14:43:22.369 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:43:22.396 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:22.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:22.397 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:43:22.498 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:43:22.509 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:22 smithi032 ceph-mon[20004]: pgmap v1904: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 1 peering, 10 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 326 B/s rd, 11 KiB/s wr, 382 op/s; 3518/78876 objects degraded (4.460%); 6647/78876 objects misplaced (8.427%); 43 MiB/s, 970 objects/s recovering 2024-08-19T14:43:22.533 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:22.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:22.534 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:43:22.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:22 smithi045 ceph-mon[20143]: pgmap v1904: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 1 peering, 10 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 326 B/s rd, 11 KiB/s wr, 382 op/s; 3518/78876 objects degraded (4.460%); 6647/78876 objects misplaced (8.427%); 43 MiB/s, 970 objects/s recovering 2024-08-19T14:43:22.634 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:43:22.677 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:22.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:22.678 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:43:22.779 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:43:22.835 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:22.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:22.836 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:43:22.842 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:43:22.937 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:43:22.967 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:22.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:22.968 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:43:23.069 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:43:23.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:23.096 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:23.097 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:43:23.198 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:43:23.226 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:23.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:23.227 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:43:23.328 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:43:23.357 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:23.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:23.358 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:43:23.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:23 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 3518/78876 objects degraded (4.460%), 11 pgs degraded (PG_DEGRADED) 2024-08-19T14:43:23.375 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:23 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 3518/78876 objects degraded (4.460%), 11 pgs degraded (PG_DEGRADED) 2024-08-19T14:43:23.459 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:43:23.483 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:23.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:23.484 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:43:23.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:23 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 3518/78876 objects degraded (4.460%), 11 pgs degraded (PG_DEGRADED) 2024-08-19T14:43:23.584 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:43:23.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:23.614 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:23.614 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:43:23.715 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:43:23.739 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:23.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:23.740 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:43:23.841 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:43:23.874 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:23.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:23.875 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:43:23.976 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:43:24.002 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:24.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:24.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:43:24.104 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:43:24.141 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:24.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:24.142 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:43:24.150 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:43:24.150 INFO:teuthology.orchestra.run.smithi032.stdout: "success": "" 2024-08-19T14:43:24.151 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:43:24.243 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:43:24.270 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:24.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:24.271 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:43:24.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:24 smithi031 ceph-mon[16336]: pgmap v1905: 57 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 10 active+recovering+undersized+degraded+remapped, 44 active+clean; 1.5 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 255 B/s rd, 8.2 KiB/s wr, 443 op/s; 3605/75411 objects degraded (4.780%); 5615/75411 objects misplaced (7.446%); 41 MiB/s, 900 objects/s recovering 2024-08-19T14:43:24.373 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:43:24.395 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:24.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:24.397 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:43:24.447 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:24 smithi045 ceph-mon[20143]: pgmap v1905: 57 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 10 active+recovering+undersized+degraded+remapped, 44 active+clean; 1.5 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 255 B/s rd, 8.2 KiB/s wr, 443 op/s; 3605/75411 objects degraded (4.780%); 5615/75411 objects misplaced (7.446%); 41 MiB/s, 900 objects/s recovering 2024-08-19T14:43:24.498 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:43:24.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:24 smithi032 ceph-mon[20004]: pgmap v1905: 57 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 10 active+recovering+undersized+degraded+remapped, 44 active+clean; 1.5 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 255 B/s rd, 8.2 KiB/s wr, 443 op/s; 3605/75411 objects degraded (4.780%); 5615/75411 objects misplaced (7.446%); 41 MiB/s, 900 objects/s recovering 2024-08-19T14:43:24.530 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:24.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:24.531 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:43:24.632 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:43:24.656 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:24.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:24.657 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:43:24.758 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:43:24.787 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:24.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:24.788 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:43:24.839 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:43:24.889 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:43:24.934 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:24.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:24.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:43:25.036 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:43:25.086 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:25.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:25.087 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:43:25.188 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:43:25.212 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:25.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:25.214 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:43:25.315 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:43:25.338 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:25.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:25.339 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:43:25.440 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:43:25.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:25.480 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:25.480 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:43:25.581 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:43:25.604 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:25.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:25.605 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:43:25.706 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:43:25.749 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:25.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:25.750 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:43:25.852 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:43:25.876 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:25.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:25.878 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:43:25.979 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:43:26.012 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:26.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:26.013 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:43:26.114 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:43:26.139 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:26.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:26.140 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:43:26.241 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:43:26.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:26 smithi032 ceph-mon[20004]: pgmap v1906: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 10 active+recovering+undersized+degraded+remapped, 44 active+clean; 1.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 493 op/s; 3604/71505 objects degraded (5.040%); 5085/71505 objects misplaced (7.111%); 23 MiB/s, 579 objects/s recovering 2024-08-19T14:43:26.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:26.291 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:26.291 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:43:26.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:26 smithi031 ceph-mon[16336]: pgmap v1906: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 10 active+recovering+undersized+degraded+remapped, 44 active+clean; 1.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 493 op/s; 3604/71505 objects degraded (5.040%); 5085/71505 objects misplaced (7.111%); 23 MiB/s, 579 objects/s recovering 2024-08-19T14:43:26.392 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:43:26.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:26.419 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:26.420 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:43:26.520 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:43:26.545 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:26.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:26.546 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:43:26.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:26 smithi045 ceph-mon[20143]: pgmap v1906: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 10 active+recovering+undersized+degraded+remapped, 44 active+clean; 1.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 493 op/s; 3604/71505 objects degraded (5.040%); 5085/71505 objects misplaced (7.111%); 23 MiB/s, 579 objects/s recovering 2024-08-19T14:43:26.646 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:43:26.680 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:26.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:26.682 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:43:26.732 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:43:26.783 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:43:26.820 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:26.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:26.821 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:43:26.922 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:43:26.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:26.965 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:26.965 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:43:27.066 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:43:27.109 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:27.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:27.110 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:43:27.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:27 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3526045688' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:43:27.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:27 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:43:27.123 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is set 2024-08-19T14:43:27.211 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:43:27.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:27.243 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:27.244 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:43:27.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:27 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3526045688' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:43:27.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:27 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:43:27.345 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:43:27.385 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:27.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:27.386 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:43:27.487 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:43:27.539 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:27.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:27.540 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:43:27.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:27 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3526045688' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:43:27.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:27 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:43:27.641 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:43:27.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:27.667 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:27.667 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:43:27.768 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:43:27.795 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:27.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:27.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:43:27.897 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:43:27.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:27.946 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:27.946 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:43:28.047 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:43:28.076 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:28.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:28.077 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:43:28.178 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:43:28.204 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:28.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:28.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:43:28.306 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:43:28.330 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:28.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:28.331 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:43:28.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:28 smithi031 ceph-mon[16336]: pgmap v1907: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 10 active+recovering+undersized+degraded+remapped, 44 active+clean; 1.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 582 op/s; 3701/66288 objects degraded (5.583%); 4165/66288 objects misplaced (6.283%); 24 MiB/s, 600 objects/s recovering 2024-08-19T14:43:28.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:28 smithi031 ceph-mon[16336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:43:28.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:28 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:43:28.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:28 smithi031 ceph-mon[16336]: osdmap e802: 12 total, 12 up, 8 in 2024-08-19T14:43:28.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:28 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 3701/66288 objects degraded (5.583%), 11 pgs degraded (PG_DEGRADED) 2024-08-19T14:43:28.433 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:43:28.459 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:28.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:28.460 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:43:28.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:28 smithi032 ceph-mon[20004]: pgmap v1907: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 10 active+recovering+undersized+degraded+remapped, 44 active+clean; 1.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 582 op/s; 3701/66288 objects degraded (5.583%); 4165/66288 objects misplaced (6.283%); 24 MiB/s, 600 objects/s recovering 2024-08-19T14:43:28.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:28 smithi032 ceph-mon[20004]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:43:28.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:28 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:43:28.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:28 smithi032 ceph-mon[20004]: osdmap e802: 12 total, 12 up, 8 in 2024-08-19T14:43:28.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:28 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 3701/66288 objects degraded (5.583%), 11 pgs degraded (PG_DEGRADED) 2024-08-19T14:43:28.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:28 smithi045 ceph-mon[20143]: pgmap v1907: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 10 active+recovering+undersized+degraded+remapped, 44 active+clean; 1.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 582 op/s; 3701/66288 objects degraded (5.583%); 4165/66288 objects misplaced (6.283%); 24 MiB/s, 600 objects/s recovering 2024-08-19T14:43:28.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:28 smithi045 ceph-mon[20143]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:43:28.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:28 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:43:28.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:28 smithi045 ceph-mon[20143]: osdmap e802: 12 total, 12 up, 8 in 2024-08-19T14:43:28.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:28 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 3701/66288 objects degraded (5.583%), 11 pgs degraded (PG_DEGRADED) 2024-08-19T14:43:28.560 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:43:28.588 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:28.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:28.589 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:43:28.690 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:43:28.716 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:28.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:28.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:43:28.818 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:43:28.847 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:28.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:28.848 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:43:28.949 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:43:28.976 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:28.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:28.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:43:29.078 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:43:29.106 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:29.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:29.107 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:43:29.208 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:43:29.232 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:29.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:29.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:43:29.334 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:43:29.361 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:29.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:29.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:43:29.464 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:43:29.479 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:43:29.508 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:29.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:29.509 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:43:29.610 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:43:29.643 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:29.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:29.644 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:43:29.745 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:43:29.773 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:29.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:29.774 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:43:29.875 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:43:29.902 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:29.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:29.903 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:43:30.004 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:43:30.028 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:30.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:30.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:43:30.130 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:43:30.159 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:30.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:30.160 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:43:30.261 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:43:30.303 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:30.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:30.304 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:43:30.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:30 smithi031 ceph-mon[16336]: pgmap v1909: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 10 active+recovering+undersized+degraded+remapped, 44 active+clean; 1.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 648 op/s; 3701/62943 objects degraded (5.880%); 3258/62943 objects misplaced (5.176%); 26 MiB/s, 624 objects/s recovering 2024-08-19T14:43:30.373 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:30 smithi032 ceph-mon[20004]: pgmap v1909: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 10 active+recovering+undersized+degraded+remapped, 44 active+clean; 1.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 648 op/s; 3701/62943 objects degraded (5.880%); 3258/62943 objects misplaced (5.176%); 26 MiB/s, 624 objects/s recovering 2024-08-19T14:43:30.404 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:43:30.413 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:30 smithi045 ceph-mon[20143]: pgmap v1909: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 10 active+recovering+undersized+degraded+remapped, 44 active+clean; 1.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 648 op/s; 3701/62943 objects degraded (5.880%); 3258/62943 objects misplaced (5.176%); 26 MiB/s, 624 objects/s recovering 2024-08-19T14:43:30.437 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:30.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:30.438 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:43:30.538 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:43:30.565 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:30.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:30.566 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:43:30.667 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:43:30.723 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:30.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:30.724 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:43:30.825 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:43:30.856 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:30.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:30.857 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:43:30.958 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:43:30.985 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:30.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:30.987 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:43:31.088 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:43:31.111 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:31.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:31.112 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:43:31.214 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:43:31.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:31.241 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:31.242 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:43:31.343 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:43:31.371 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:31.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:31.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:43:31.473 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:43:31.522 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:31.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:31.523 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:43:31.624 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:43:31.649 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:31.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:31.650 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:43:31.753 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:43:31.783 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:31.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:31.784 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:43:31.885 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:43:31.912 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:31.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:31.913 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:43:32.014 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:43:32.040 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:32.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:32.041 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:43:32.142 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:43:32.178 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:32.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:32.179 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:43:32.264 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:32 smithi031 ceph-mon[16336]: pgmap v1910: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 10 active+recovering+undersized+degraded+remapped, 44 active+clean; 1.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 670 op/s; 3701/58764 objects degraded (6.298%); 2405/58764 objects misplaced (4.093%); 25 MiB/s, 643 objects/s recovering 2024-08-19T14:43:32.264 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:32 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/516385396' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:43:32.280 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:43:32.305 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:32.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:32.306 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:43:32.407 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:43:32.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:32.433 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:32.433 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:43:32.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:32 smithi032 ceph-mon[20004]: pgmap v1910: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 10 active+recovering+undersized+degraded+remapped, 44 active+clean; 1.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 670 op/s; 3701/58764 objects degraded (6.298%); 2405/58764 objects misplaced (4.093%); 25 MiB/s, 643 objects/s recovering 2024-08-19T14:43:32.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:32 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/516385396' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:43:32.534 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:43:32.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:32 smithi045 ceph-mon[20143]: pgmap v1910: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 10 active+recovering+undersized+degraded+remapped, 44 active+clean; 1.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 670 op/s; 3701/58764 objects degraded (6.298%); 2405/58764 objects misplaced (4.093%); 25 MiB/s, 643 objects/s recovering 2024-08-19T14:43:32.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:32 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/516385396' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:43:32.557 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:32.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:32.558 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:43:32.659 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:43:32.686 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:32.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:32.687 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:43:32.787 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:43:32.809 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:32.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:32.810 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:43:32.911 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:43:32.945 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:32.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:32.946 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:43:33.047 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:43:33.057 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:43:33.057 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:43:33.057 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:43:33.058 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:43:33.058 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:43:33.058 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:43:33.058 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:43:33.058 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:43:33.058 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:43:33.058 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:43:33.059 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:43:33.059 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:43:33.059 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:43:33.059 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:43:33.059 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:43:33.059 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:43:33.059 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:43:33.059 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:43:33.059 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:43:33.059 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:43:33.060 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:43:33.060 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:43:33.060 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:43:33.060 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:43:33.080 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:33.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:33.082 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:43:33.183 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:43:33.204 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:33.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:33.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:43:33.306 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:43:33.332 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:33.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:33.333 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:43:33.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:33 smithi031 ceph-mon[16336]: osdmap e803: 12 total, 12 up, 8 in 2024-08-19T14:43:33.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:33 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 3701/58764 objects degraded (6.298%), 11 pgs degraded (PG_DEGRADED) 2024-08-19T14:43:33.434 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:43:33.466 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:33.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:33.467 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:43:33.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:33 smithi032 ceph-mon[20004]: osdmap e803: 12 total, 12 up, 8 in 2024-08-19T14:43:33.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:33 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 3701/58764 objects degraded (6.298%), 11 pgs degraded (PG_DEGRADED) 2024-08-19T14:43:33.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:33 smithi045 ceph-mon[20143]: osdmap e803: 12 total, 12 up, 8 in 2024-08-19T14:43:33.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:33 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 3701/58764 objects degraded (6.298%), 11 pgs degraded (PG_DEGRADED) 2024-08-19T14:43:33.568 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:43:33.592 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:33.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:33.593 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:43:33.694 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:43:33.717 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:33.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:33.718 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:43:33.819 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:43:33.842 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:33.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:33.843 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:43:33.859 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:43:33.859 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:43:33.944 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:43:33.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:33.994 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:33.994 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:43:34.095 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:43:34.120 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:34.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:34.121 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:43:34.222 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:43:34.253 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:34.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:34.254 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:43:34.355 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:43:34.379 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:34.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:34.380 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:43:34.431 INFO:tasks.thrashosds.thrasher:in_osds: [1, 4, 5, 8, 2, 11, 9, 10] out_osds: [6, 3, 0, 7] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 8, 9, 11, 10, 2] 2024-08-19T14:43:34.431 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd reweight 1 1 2024-08-19T14:43:34.457 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:34 smithi031 ceph-mon[16336]: pgmap v1912: 57 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 10 active+recovering+undersized+degraded+remapped, 44 active+clean; 1.1 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 664 op/s; 3775/55545 objects degraded (6.796%); 1616/55545 objects misplaced (2.909%); 26 MiB/s, 706 objects/s recovering 2024-08-19T14:43:34.457 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:34 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:43:34.457 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:34 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:43:34.457 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:34 smithi031 ceph-mon[16336]: osdmap e804: 12 total, 12 up, 8 in 2024-08-19T14:43:34.480 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:43:34.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:34 smithi032 ceph-mon[20004]: pgmap v1912: 57 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 10 active+recovering+undersized+degraded+remapped, 44 active+clean; 1.1 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 664 op/s; 3775/55545 objects degraded (6.796%); 1616/55545 objects misplaced (2.909%); 26 MiB/s, 706 objects/s recovering 2024-08-19T14:43:34.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:34 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:43:34.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:34 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:43:34.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:34 smithi032 ceph-mon[20004]: osdmap e804: 12 total, 12 up, 8 in 2024-08-19T14:43:34.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:34.522 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:34.522 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:43:34.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:34 smithi045 ceph-mon[20143]: pgmap v1912: 57 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 10 active+recovering+undersized+degraded+remapped, 44 active+clean; 1.1 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 664 op/s; 3775/55545 objects degraded (6.796%); 1616/55545 objects misplaced (2.909%); 26 MiB/s, 706 objects/s recovering 2024-08-19T14:43:34.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:34 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:43:34.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:34 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:43:34.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:34 smithi045 ceph-mon[20143]: osdmap e804: 12 total, 12 up, 8 in 2024-08-19T14:43:34.623 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:43:34.664 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:34.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:34.665 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:43:34.766 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:43:34.790 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:34.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:34.791 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:43:34.892 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:43:34.929 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:34.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:34.931 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:43:35.032 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:43:35.056 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:35.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:35.058 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:43:35.159 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:43:35.184 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:35.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:35.185 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:43:35.286 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:43:35.311 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:35.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:35.313 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:43:35.414 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:43:35.437 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:35.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:35.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:43:35.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:35 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:43:35.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:35 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:43:35.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:35 smithi032 ceph-mon[20004]: osdmap e805: 12 total, 12 up, 8 in 2024-08-19T14:43:35.539 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:43:35.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:35 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:43:35.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:35 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:43:35.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:35 smithi045 ceph-mon[20143]: osdmap e805: 12 total, 12 up, 8 in 2024-08-19T14:43:35.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:35.568 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:35.568 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:43:35.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:35 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:43:35.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:35 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:43:35.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:35 smithi031 ceph-mon[16336]: osdmap e805: 12 total, 12 up, 8 in 2024-08-19T14:43:35.669 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:43:35.695 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:35.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:35.697 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:43:35.799 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:43:35.824 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:35.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:35.826 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:43:35.927 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:43:35.955 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:35.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:35.956 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:43:36.058 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:43:36.086 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:36.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:36.087 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:43:36.189 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:43:36.214 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:36.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:36.215 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:43:36.316 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:43:36.342 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:36.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:36.343 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:43:36.444 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:43:36.469 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:36.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:36.470 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:43:36.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:36 smithi032 ceph-mon[20004]: pgmap v1915: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 10 active+recovering+undersized+degraded+remapped, 44 active+clean; 1.1 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 607 op/s; 3775/52008 objects degraded (7.258%); 1443/52008 objects misplaced (2.775%); 18 MiB/s, 691 objects/s recovering 2024-08-19T14:43:36.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:36 smithi045 ceph-mon[20143]: pgmap v1915: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 10 active+recovering+undersized+degraded+remapped, 44 active+clean; 1.1 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 607 op/s; 3775/52008 objects degraded (7.258%); 1443/52008 objects misplaced (2.775%); 18 MiB/s, 691 objects/s recovering 2024-08-19T14:43:36.566 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:36 smithi031 ceph-mon[16336]: pgmap v1915: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 10 active+recovering+undersized+degraded+remapped, 44 active+clean; 1.1 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 607 op/s; 3775/52008 objects degraded (7.258%); 1443/52008 objects misplaced (2.775%); 18 MiB/s, 691 objects/s recovering 2024-08-19T14:43:36.571 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:43:36.603 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:36.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:36.604 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:43:36.705 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:43:36.731 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:36.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:36.733 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:43:36.834 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:43:36.859 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:36.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:36.860 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:43:36.961 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:43:36.999 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:37.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:37.000 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:43:37.101 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:43:37.156 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:37.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:37.157 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:43:37.258 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:43:37.295 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:37.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:37.296 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:43:37.397 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:43:37.432 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:37.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:37.433 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:43:37.534 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:43:37.559 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:37.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:37.560 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:43:37.661 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:43:37.706 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:37.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:37.707 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:43:37.808 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:43:37.815 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:43:37.833 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:37.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:37.835 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:43:37.936 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:43:37.959 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:37.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:37.960 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:43:38.061 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:43:38.089 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:38.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:38.090 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:43:38.192 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:43:38.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:38.234 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:38.235 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:43:38.336 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:43:38.349 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:38 smithi031 ceph-mon[16336]: pgmap v1916: 57 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 10 active+recovering+undersized+degraded+remapped, 45 active+clean; 979 MiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 653 op/s; 3864/46995 objects degraded (8.222%); 751/46995 objects misplaced (1.598%); 15 MiB/s, 651 objects/s recovering 2024-08-19T14:43:38.350 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:38 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 3864/46995 objects degraded (8.222%), 11 pgs degraded (PG_DEGRADED) 2024-08-19T14:43:38.362 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:38.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:38.363 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:43:38.464 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:43:38.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:38 smithi032 ceph-mon[20004]: pgmap v1916: 57 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 10 active+recovering+undersized+degraded+remapped, 45 active+clean; 979 MiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 653 op/s; 3864/46995 objects degraded (8.222%); 751/46995 objects misplaced (1.598%); 15 MiB/s, 651 objects/s recovering 2024-08-19T14:43:38.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:38 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 3864/46995 objects degraded (8.222%), 11 pgs degraded (PG_DEGRADED) 2024-08-19T14:43:38.509 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:38.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:38.510 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:43:38.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:38 smithi045 ceph-mon[20143]: pgmap v1916: 57 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 10 active+recovering+undersized+degraded+remapped, 45 active+clean; 979 MiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 653 op/s; 3864/46995 objects degraded (8.222%); 751/46995 objects misplaced (1.598%); 15 MiB/s, 651 objects/s recovering 2024-08-19T14:43:38.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:38 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 3864/46995 objects degraded (8.222%), 11 pgs degraded (PG_DEGRADED) 2024-08-19T14:43:38.611 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:43:38.639 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:38.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:38.640 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:43:38.741 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:43:38.766 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:38.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:38.767 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:43:38.869 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:43:38.912 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:38.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:38.914 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:43:39.015 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:43:39.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:39.039 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:39.039 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:43:39.140 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:43:39.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:39.182 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:39.182 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:43:39.204 INFO:teuthology.orchestra.run.smithi031.stderr:reweighted osd.1 to 1 (10000) 2024-08-19T14:43:39.284 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:43:39.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:39.325 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:39.325 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:43:39.421 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:43:39.427 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:43:39.445 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:39 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3116558455' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-08-19T14:43:39.475 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:39.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:39.476 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:43:39.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:39 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3116558455' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-08-19T14:43:39.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:39 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3116558455' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-08-19T14:43:39.577 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:43:39.607 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:39.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:39.609 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:43:39.709 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:43:39.747 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:39.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:39.748 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:43:39.808 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-19T14:43:39.848 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:43:39.875 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:39.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:39.876 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:43:39.977 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:43:40.013 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:40.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:40.015 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:43:40.023 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd reweight 4 1 2024-08-19T14:43:40.100 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:43:40.115 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:43:40.147 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:40.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:40.148 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:43:40.148 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:43:40.248 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:43:40.285 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:40.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:40.286 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:43:40.387 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:43:40.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:40.415 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:40.415 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:43:40.460 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:40 smithi031 ceph-mon[16336]: pgmap v1917: 57 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 10 active+recovering+undersized+degraded+remapped, 45 active+clean; 914 MiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 627 op/s; 3864/43869 objects degraded (8.808%); 529/43869 objects misplaced (1.206%); 10 MiB/s, 641 objects/s recovering 2024-08-19T14:43:40.461 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:40 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:43:40.461 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:40 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:43:40.461 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:40 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3116558455' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-08-19T14:43:40.461 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:40 smithi031 ceph-mon[16336]: osdmap e806: 12 total, 12 up, 8 in 2024-08-19T14:43:40.469 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:40 smithi032 ceph-mon[20004]: pgmap v1917: 57 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 10 active+recovering+undersized+degraded+remapped, 45 active+clean; 914 MiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 627 op/s; 3864/43869 objects degraded (8.808%); 529/43869 objects misplaced (1.206%); 10 MiB/s, 641 objects/s recovering 2024-08-19T14:43:40.469 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:40 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:43:40.469 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:40 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:43:40.469 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:40 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3116558455' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-08-19T14:43:40.469 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:40 smithi032 ceph-mon[20004]: osdmap e806: 12 total, 12 up, 8 in 2024-08-19T14:43:40.471 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-08-19T14:43:40.516 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:43:40.522 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:40 smithi045 ceph-mon[20143]: pgmap v1917: 57 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 10 active+recovering+undersized+degraded+remapped, 45 active+clean; 914 MiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 627 op/s; 3864/43869 objects degraded (8.808%); 529/43869 objects misplaced (1.206%); 10 MiB/s, 641 objects/s recovering 2024-08-19T14:43:40.523 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:40 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:43:40.523 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:40 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:43:40.523 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:40 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3116558455' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-08-19T14:43:40.523 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:40 smithi045 ceph-mon[20143]: osdmap e806: 12 total, 12 up, 8 in 2024-08-19T14:43:40.543 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:40.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:40.544 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:43:40.644 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:43:40.684 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:40.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:40.685 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:43:40.786 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:43:40.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:40.816 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:40.816 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:43:40.816 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:43:40.825 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:43:40.826 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:43:40.850 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:43:40.851 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:43:40.877 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:43:40.878 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:43:40.903 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:43:40.903 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:43:40.917 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:43:40.923 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:43:40.923 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:43:40.948 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:43:40.948 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:43:40.965 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:40.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:40.967 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:43:40.969 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:43:40.969 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:43:40.997 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:43:40.997 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:43:41.025 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:43:41.025 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:43:41.067 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:43:41.068 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:43:41.068 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:43:41.100 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:43:41.100 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:43:41.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:41.102 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:41.103 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:43:41.126 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:43:41.127 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:43:41.130 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:43:41.204 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:43:41.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:41.240 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:41.240 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:43:41.342 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:43:41.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:41 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/425256965' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:43:41.369 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:41.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:41.370 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:43:41.471 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:43:41.502 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:41.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:41.503 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:43:41.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:41 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/425256965' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:43:41.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:41 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/425256965' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:43:41.604 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:43:41.628 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:41.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:41.629 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:43:41.730 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:43:41.754 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:41.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:41.755 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:43:41.856 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:43:41.909 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:41.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:41.910 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:43:42.012 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:43:42.036 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:42.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:42.037 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:43:42.138 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:43:42.165 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:42.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:42.166 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:43:42.267 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:43:42.273 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is set 2024-08-19T14:43:42.298 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:42.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:42.299 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:43:42.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:42 smithi031 ceph-mon[16336]: pgmap v1919: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 10 active+recovering+undersized+degraded+remapped, 45 active+clean; 829 MiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 615 op/s; 3864/39762 objects degraded (9.718%); 529/39762 objects misplaced (1.330%); 7.2 MiB/s, 718 objects/s recovering 2024-08-19T14:43:42.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:42 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/624126875' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:43:42.400 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:43:42.436 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:42.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:42.437 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:43:42.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:42 smithi032 ceph-mon[20004]: pgmap v1919: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 10 active+recovering+undersized+degraded+remapped, 45 active+clean; 829 MiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 615 op/s; 3864/39762 objects degraded (9.718%); 529/39762 objects misplaced (1.330%); 7.2 MiB/s, 718 objects/s recovering 2024-08-19T14:43:42.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:42 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/624126875' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:43:42.538 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:43:42.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:42 smithi045 ceph-mon[20143]: pgmap v1919: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 10 active+recovering+undersized+degraded+remapped, 45 active+clean; 829 MiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 615 op/s; 3864/39762 objects degraded (9.718%); 529/39762 objects misplaced (1.330%); 7.2 MiB/s, 718 objects/s recovering 2024-08-19T14:43:42.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:42 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/624126875' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:43:42.580 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:42.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:42.581 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:43:42.682 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:43:42.707 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:42.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:42.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:43:42.809 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:43:42.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:42.861 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:42.861 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:43:42.962 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:43:42.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:42.987 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:42.987 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:43:43.088 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:43:43.124 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:43.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:43.125 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:43:43.226 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:43:43.252 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:43.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:43.253 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:43:43.311 INFO:teuthology.orchestra.run.smithi031.stderr:reweighted osd.4 to 1 (10000) 2024-08-19T14:43:43.354 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:43:43.379 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:43.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:43.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:43:43.481 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:43:43.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:43 smithi032 ceph-mon[20004]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:43:43.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:43 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/624126875' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:43:43.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:43 smithi032 ceph-mon[20004]: osdmap e807: 12 total, 12 up, 8 in 2024-08-19T14:43:43.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:43 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2433167305' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-08-19T14:43:43.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:43 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-08-19T14:43:43.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:43 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 3864/39762 objects degraded (9.718%), 11 pgs degraded (PG_DEGRADED) 2024-08-19T14:43:43.526 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:43.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:43.527 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:43:43.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:43 smithi045 ceph-mon[20143]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:43:43.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:43 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/624126875' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:43:43.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:43 smithi045 ceph-mon[20143]: osdmap e807: 12 total, 12 up, 8 in 2024-08-19T14:43:43.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:43 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2433167305' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-08-19T14:43:43.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:43 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-08-19T14:43:43.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:43 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 3864/39762 objects degraded (9.718%), 11 pgs degraded (PG_DEGRADED) 2024-08-19T14:43:43.553 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:43 smithi031 ceph-mon[16336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:43:43.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:43 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/624126875' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:43:43.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:43 smithi031 ceph-mon[16336]: osdmap e807: 12 total, 12 up, 8 in 2024-08-19T14:43:43.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:43 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2433167305' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-08-19T14:43:43.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:43 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-08-19T14:43:43.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:43 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 3864/39762 objects degraded (9.718%), 11 pgs degraded (PG_DEGRADED) 2024-08-19T14:43:43.628 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:43:43.652 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:43.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:43.653 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:43:43.755 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:43:43.778 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:43.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:43.779 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:43:43.881 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:43:43.920 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:43.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:43.921 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:43:44.022 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:43:44.062 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:44.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:44.063 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:43:44.164 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:43:44.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:44.187 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:44.187 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:43:44.288 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:43:44.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:44.318 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:44.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:43:44.419 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:43:44.447 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:44.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:44.448 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:43:44.462 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd reweight 5 1 2024-08-19T14:43:44.549 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:43:44.580 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:44.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:44.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:43:44.604 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:44 smithi032 ceph-mon[20004]: pgmap v1921: 57 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 10 active+recovering+undersized+degraded+remapped, 45 active+clean; 758 MiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 651 op/s; 3945/36372 objects degraded (10.846%); 445/36372 objects misplaced (1.223%); 6.4 MiB/s, 708 objects/s recovering 2024-08-19T14:43:44.604 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:44 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-08-19T14:43:44.604 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:44 smithi032 ceph-mon[20004]: osdmap e808: 12 total, 12 up, 8 in 2024-08-19T14:43:44.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:44 smithi031 ceph-mon[16336]: pgmap v1921: 57 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 10 active+recovering+undersized+degraded+remapped, 45 active+clean; 758 MiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 651 op/s; 3945/36372 objects degraded (10.846%); 445/36372 objects misplaced (1.223%); 6.4 MiB/s, 708 objects/s recovering 2024-08-19T14:43:44.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:44 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-08-19T14:43:44.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:44 smithi031 ceph-mon[16336]: osdmap e808: 12 total, 12 up, 8 in 2024-08-19T14:43:44.682 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:43:44.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:44.710 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:44.710 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:43:44.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:44 smithi045 ceph-mon[20143]: pgmap v1921: 57 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 10 active+recovering+undersized+degraded+remapped, 45 active+clean; 758 MiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 651 op/s; 3945/36372 objects degraded (10.846%); 445/36372 objects misplaced (1.223%); 6.4 MiB/s, 708 objects/s recovering 2024-08-19T14:43:44.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:44 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-08-19T14:43:44.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:44 smithi045 ceph-mon[20143]: osdmap e808: 12 total, 12 up, 8 in 2024-08-19T14:43:44.811 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:43:44.822 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:43:44.840 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:44.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:44.841 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:43:44.865 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:43:44.942 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:43:44.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:44.969 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:44.970 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:43:45.071 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:43:45.095 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:45.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:45.096 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:43:45.197 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:43:45.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:45.224 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:45.224 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:43:45.325 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:43:45.349 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:45.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:45.350 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:43:45.418 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:43:45.418 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-08-19T14:43:45.418 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:43:45.419 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26911.0:216078 8.4 8:2075532f:::benchmark_data_smithi045_137103_object103614:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e808)", 2024-08-19T14:43:45.419 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:43:44.693092+0000", 2024-08-19T14:43:45.419 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.0075023110000000002, 2024-08-19T14:43:45.419 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0075323940000000004, 2024-08-19T14:43:45.419 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:43:45.419 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:43:45.419 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "waiting for sub ops", 2024-08-19T14:43:45.419 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:43:45.419 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26911", 2024-08-19T14:43:45.419 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3494993995", 2024-08-19T14:43:45.419 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 216078 2024-08-19T14:43:45.419 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:43:45.420 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:43:45.420 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:43:45.420 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:43:45.420 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:44.693092+0000", 2024-08-19T14:43:45.420 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:43:45.420 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:43:45.420 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:43:45.420 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:43:45.420 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:44.693092+0000", 2024-08-19T14:43:45.420 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:43:45.420 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:43:45.420 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:43:45.420 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:43:45.421 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:44.693093+0000", 2024-08-19T14:43:45.421 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4759999999999999e-06 2024-08-19T14:43:45.421 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:43:45.421 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:43:45.421 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:43:45.421 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:44.693099+0000", 2024-08-19T14:43:45.421 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0750000000000003e-06 2024-08-19T14:43:45.421 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:43:45.421 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:43:45.421 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:43:45.421 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:44.693113+0000", 2024-08-19T14:43:45.421 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3397000000000001e-05 2024-08-19T14:43:45.421 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:43:45.422 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:43:45.422 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:43:45.422 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:44.693121+0000", 2024-08-19T14:43:45.422 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.9810000000000003e-06 2024-08-19T14:43:45.422 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:43:45.422 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:43:45.422 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:43:45.422 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:44.693179+0000", 2024-08-19T14:43:45.422 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8436000000000001e-05 2024-08-19T14:43:45.422 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:43:45.422 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:43:45.422 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:43:45.422 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:44.693274+0000", 2024-08-19T14:43:45.423 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.5307000000000001e-05 2024-08-19T14:43:45.423 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:43:45.423 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:43:45.423 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 9,10", 2024-08-19T14:43:45.423 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:44.693346+0000", 2024-08-19T14:43:45.423 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1622e-05 2024-08-19T14:43:45.423 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:43:45.423 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:43:45.424 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:43:45.424 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:43:45.424 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:43:45.424 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26911.0:216080 8.4 8:230772bf:::benchmark_data_smithi045_137103_object103616:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e808)", 2024-08-19T14:43:45.424 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:43:44.694494+0000", 2024-08-19T14:43:45.424 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.0060996449999999999, 2024-08-19T14:43:45.424 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0061772229999999999, 2024-08-19T14:43:45.424 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:43:45.425 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:43:45.425 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "waiting for sub ops", 2024-08-19T14:43:45.425 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:43:45.425 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26911", 2024-08-19T14:43:45.425 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3494993995", 2024-08-19T14:43:45.425 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 216080 2024-08-19T14:43:45.425 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:43:45.425 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:43:45.426 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:43:45.426 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:43:45.426 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:44.694494+0000", 2024-08-19T14:43:45.426 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:43:45.426 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:43:45.426 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:43:45.426 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:43:45.426 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:44.694494+0000", 2024-08-19T14:43:45.426 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:43:45.426 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:43:45.427 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:43:45.427 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:43:45.427 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:44.694496+0000", 2024-08-19T14:43:45.427 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1880000000000001e-06 2024-08-19T14:43:45.427 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:43:45.427 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:43:45.427 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:43:45.427 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:44.694503+0000", 2024-08-19T14:43:45.427 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.8159999999999994e-06 2024-08-19T14:43:45.427 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:43:45.427 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:43:45.427 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:43:45.427 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:44.694517+0000", 2024-08-19T14:43:45.427 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.324e-05 2024-08-19T14:43:45.428 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:43:45.428 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:43:45.428 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:43:45.428 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:44.694524+0000", 2024-08-19T14:43:45.428 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7349999999999996e-06 2024-08-19T14:43:45.428 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:43:45.428 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:43:45.428 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:43:45.428 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:44.696618+0000", 2024-08-19T14:43:45.428 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0020933710000000001 2024-08-19T14:43:45.428 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:43:45.428 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:43:45.428 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:43:45.429 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:44.696756+0000", 2024-08-19T14:43:45.429 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000138615 2024-08-19T14:43:45.429 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:43:45.429 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:43:45.429 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 9,10", 2024-08-19T14:43:45.429 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:44.696913+0000", 2024-08-19T14:43:45.429 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00015673099999999999 2024-08-19T14:43:45.429 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:43:45.429 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:43:45.429 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:43:45.429 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:43:45.429 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:43:45.429 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26911.0:216082 8.4 8:27579643:::benchmark_data_smithi045_137103_object103618:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e808)", 2024-08-19T14:43:45.429 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:43:44.694693+0000", 2024-08-19T14:43:45.430 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.0059010440000000003, 2024-08-19T14:43:45.430 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0060210610000000003, 2024-08-19T14:43:45.430 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:43:45.430 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:43:45.430 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "waiting for sub ops", 2024-08-19T14:43:45.430 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:43:45.430 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26911", 2024-08-19T14:43:45.430 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3494993995", 2024-08-19T14:43:45.430 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 216082 2024-08-19T14:43:45.430 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:43:45.430 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:43:45.430 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:43:45.430 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:43:45.430 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:44.694693+0000", 2024-08-19T14:43:45.431 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:43:45.431 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:43:45.431 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:43:45.431 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:43:45.431 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:44.694693+0000", 2024-08-19T14:43:45.431 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:43:45.431 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:43:45.431 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:43:45.431 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:43:45.431 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:44.694694+0000", 2024-08-19T14:43:45.431 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4099999999999998e-07 2024-08-19T14:43:45.431 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:43:45.431 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:43:45.431 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:43:45.432 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:44.694697+0000", 2024-08-19T14:43:45.432 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.681e-06 2024-08-19T14:43:45.432 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:43:45.432 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:43:45.432 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:43:45.432 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:44.694707+0000", 2024-08-19T14:43:45.432 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.7629999999999995e-06 2024-08-19T14:43:45.432 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:43:45.432 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:43:45.432 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:43:45.433 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:44.694713+0000", 2024-08-19T14:43:45.433 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9039999999999997e-06 2024-08-19T14:43:45.433 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:43:45.433 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:43:45.433 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:43:45.433 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:44.699934+0000", 2024-08-19T14:43:45.433 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0052206370000000002 2024-08-19T14:43:45.433 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:43:45.433 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:43:45.433 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:43:45.433 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:44.700081+0000", 2024-08-19T14:43:45.433 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014751900000000001 2024-08-19T14:43:45.433 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:43:45.433 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:43:45.434 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 9,10", 2024-08-19T14:43:45.434 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:44.700217+0000", 2024-08-19T14:43:45.434 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013605999999999999 2024-08-19T14:43:45.434 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:43:45.434 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:43:45.434 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:43:45.434 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:43:45.434 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-08-19T14:43:45.434 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 3 2024-08-19T14:43:45.434 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:43:45.444 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:43:45.451 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:43:45.474 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:45.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:45.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:43:45.576 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:43:45.579 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:45 smithi031 ceph-mon[16336]: pgmap v1923: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 10 active+recovering+undersized+degraded+remapped, 45 active+clean; 684 MiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 613 op/s; 3942/32820 objects degraded (12.011%); 445/32820 objects misplaced (1.356%); 0 B/s, 633 objects/s recovering 2024-08-19T14:43:45.593 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-19T14:43:45.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:45.606 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:45.606 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:43:45.707 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:43:45.748 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:45.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:45.750 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:43:45.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:45 smithi032 ceph-mon[20004]: pgmap v1923: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 10 active+recovering+undersized+degraded+remapped, 45 active+clean; 684 MiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 613 op/s; 3942/32820 objects degraded (12.011%); 445/32820 objects misplaced (1.356%); 0 B/s, 633 objects/s recovering 2024-08-19T14:43:45.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:45 smithi045 ceph-mon[20143]: pgmap v1923: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 10 active+recovering+undersized+degraded+remapped, 45 active+clean; 684 MiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 613 op/s; 3942/32820 objects degraded (12.011%); 445/32820 objects misplaced (1.356%); 0 B/s, 633 objects/s recovering 2024-08-19T14:43:45.851 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:43:45.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:45.889 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:45.889 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:43:45.990 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:43:46.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:46.020 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:46.020 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:43:46.051 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:43:46.052 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:43:46.078 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:43:46.078 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:43:46.107 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:43:46.108 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:43:46.120 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:43:46.122 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:43:46.131 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:43:46.131 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:43:46.151 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:46.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:46.152 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:43:46.155 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:43:46.155 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:43:46.179 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:43:46.180 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:43:46.202 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:43:46.202 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:43:46.223 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:43:46.223 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:43:46.253 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:43:46.255 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:43:46.255 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:43:46.282 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:43:46.283 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:43:46.302 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:46.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:46.303 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:43:46.316 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:43:46.316 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:43:46.319 INFO:teuthology.orchestra.run.smithi031.stderr:reweighted osd.5 to 1 (10000) 2024-08-19T14:43:46.354 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:43:46.355 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:43:46.404 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:43:46.447 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:46.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:46.449 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:43:46.465 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:46 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:43:46.465 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:46 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:43:46.465 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:46 smithi031 ceph-mon[16336]: osdmap e809: 12 total, 12 up, 8 in 2024-08-19T14:43:46.465 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:46 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1211488401' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-08-19T14:43:46.465 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:46 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2593970826' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:43:46.550 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:43:46.589 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:46.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:46.590 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:43:46.691 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:43:46.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:46.735 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:46.735 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:43:46.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:46 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:43:46.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:46 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:43:46.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:46 smithi032 ceph-mon[20004]: osdmap e809: 12 total, 12 up, 8 in 2024-08-19T14:43:46.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:46 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1211488401' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-08-19T14:43:46.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:46 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2593970826' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:43:46.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:46 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:43:46.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:46 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:43:46.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:46 smithi045 ceph-mon[20143]: osdmap e809: 12 total, 12 up, 8 in 2024-08-19T14:43:46.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:46 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1211488401' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-08-19T14:43:46.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:46 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2593970826' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:43:46.837 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:43:46.863 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:46.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:46.864 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:43:46.965 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:43:46.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:46.990 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:46.990 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:43:47.091 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:43:47.120 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:47.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:47.121 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:43:47.222 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:43:47.266 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:47.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:47.268 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:43:47.368 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:43:47.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:47.412 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:47.412 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:43:47.512 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:43:47.538 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:47.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:47.539 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:43:47.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:47 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1211488401' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-08-19T14:43:47.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:47 smithi031 ceph-mon[16336]: osdmap e810: 12 total, 12 up, 8 in 2024-08-19T14:43:47.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:47 smithi031 ceph-mon[16336]: pgmap v1926: 57 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 10 active+recovering+undersized+degraded+remapped, 45 active+clean; 576 MiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 718 op/s; 4034/27606 objects degraded (14.613%); 352/27606 objects misplaced (1.275%); 0 B/s, 564 objects/s recovering 2024-08-19T14:43:47.640 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:43:47.668 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:47.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:47.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:43:47.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:47 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1211488401' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-08-19T14:43:47.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:47 smithi032 ceph-mon[20004]: osdmap e810: 12 total, 12 up, 8 in 2024-08-19T14:43:47.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:47 smithi032 ceph-mon[20004]: pgmap v1926: 57 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 10 active+recovering+undersized+degraded+remapped, 45 active+clean; 576 MiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 718 op/s; 4034/27606 objects degraded (14.613%); 352/27606 objects misplaced (1.275%); 0 B/s, 564 objects/s recovering 2024-08-19T14:43:47.770 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:43:47.795 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:47.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:47.796 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:43:47.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:47 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1211488401' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-08-19T14:43:47.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:47 smithi045 ceph-mon[20143]: osdmap e810: 12 total, 12 up, 8 in 2024-08-19T14:43:47.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:47 smithi045 ceph-mon[20143]: pgmap v1926: 57 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 10 active+recovering+undersized+degraded+remapped, 45 active+clean; 576 MiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 718 op/s; 4034/27606 objects degraded (14.613%); 352/27606 objects misplaced (1.275%); 0 B/s, 564 objects/s recovering 2024-08-19T14:43:47.897 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:43:47.928 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:47.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:47.929 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:43:48.030 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:43:48.057 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:48.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:48.058 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:43:48.159 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:43:48.184 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:48.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:48.185 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:43:48.286 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:43:48.312 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:48.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:48.313 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:43:48.414 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:43:48.442 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:48.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:48.443 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:43:48.544 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:43:48.577 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:48.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:48.578 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:43:48.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:48 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3819940253' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:43:48.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:48 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 4034/27606 objects degraded (14.613%), 11 pgs degraded (PG_DEGRADED) 2024-08-19T14:43:48.679 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:43:48.728 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:48.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:48.729 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:43:48.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:48 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3819940253' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:43:48.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:48 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 4034/27606 objects degraded (14.613%), 11 pgs degraded (PG_DEGRADED) 2024-08-19T14:43:48.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:48 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3819940253' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:43:48.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:48 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 4034/27606 objects degraded (14.613%), 11 pgs degraded (PG_DEGRADED) 2024-08-19T14:43:48.830 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:43:48.842 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is unset 2024-08-19T14:43:48.889 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:48.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:48.890 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:43:48.993 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:43:49.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:49.069 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:49.069 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:43:49.170 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:43:49.194 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:49.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:49.198 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:43:49.299 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:43:49.331 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:49.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:49.333 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:43:49.434 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:43:49.602 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:49.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:49.603 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:43:49.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:49 smithi031 ceph-mon[16336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:43:49.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:49 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:43:49.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:49 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3819940253' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:43:49.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:49 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:43:49.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:49 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:43:49.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:49 smithi031 ceph-mon[16336]: osdmap e811: 12 total, 12 up, 8 in 2024-08-19T14:43:49.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:49 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:43:49.705 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:43:49.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:49.742 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:49.742 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:43:49.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:49 smithi032 ceph-mon[20004]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:43:49.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:49 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:43:49.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:49 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3819940253' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:43:49.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:49 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:43:49.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:49 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:43:49.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:49 smithi032 ceph-mon[20004]: osdmap e811: 12 total, 12 up, 8 in 2024-08-19T14:43:49.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:49 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:43:49.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:49 smithi045 ceph-mon[20143]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:43:49.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:49 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:43:49.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:49 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3819940253' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:43:49.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:49 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:43:49.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:49 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:43:49.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:49 smithi045 ceph-mon[20143]: osdmap e811: 12 total, 12 up, 8 in 2024-08-19T14:43:49.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:49 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:43:49.843 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:43:49.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:49.865 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:49.865 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:43:49.967 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:43:49.990 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:49.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:49.991 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:43:50.092 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:43:50.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:50.116 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:50.116 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:43:50.217 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:43:50.261 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:50.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:50.262 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:43:50.363 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:43:50.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:50.399 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:50.399 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:43:50.501 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:43:50.530 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-08-19T14:43:50.531 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:50.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:50.532 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:43:50.634 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:43:50.656 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:50.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:50.657 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:43:50.758 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:43:50.779 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:50.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:50.780 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:43:50.870 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:43:50.881 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:43:50.908 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:50.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:50.909 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:43:51.010 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:43:51.064 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:51.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:51.065 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:43:51.165 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:43:51.187 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:51.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:51.189 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:43:51.290 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:43:51.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:51.325 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:51.325 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:43:51.427 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:43:51.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:51.455 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:51.455 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:43:51.556 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:43:51.579 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:51.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:51.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:43:51.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:51 smithi031 ceph-mon[16336]: osdmap e812: 12 total, 12 up, 8 in 2024-08-19T14:43:51.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:51 smithi031 ceph-mon[16336]: pgmap v1930: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 10 active+recovering+undersized+degraded+remapped, 45 active+clean; 417 MiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 674 op/s; 4034/19995 objects degraded (20.175%); 352/19995 objects misplaced (1.760%); 0 B/s, 609 objects/s recovering 2024-08-19T14:43:51.681 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:43:51.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:51.706 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:51.706 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:43:51.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:51 smithi032 ceph-mon[20004]: osdmap e812: 12 total, 12 up, 8 in 2024-08-19T14:43:51.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:51 smithi032 ceph-mon[20004]: pgmap v1930: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 10 active+recovering+undersized+degraded+remapped, 45 active+clean; 417 MiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 674 op/s; 4034/19995 objects degraded (20.175%); 352/19995 objects misplaced (1.760%); 0 B/s, 609 objects/s recovering 2024-08-19T14:43:51.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:51 smithi045 ceph-mon[20143]: osdmap e812: 12 total, 12 up, 8 in 2024-08-19T14:43:51.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:51 smithi045 ceph-mon[20143]: pgmap v1930: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 10 active+recovering+undersized+degraded+remapped, 45 active+clean; 417 MiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 674 op/s; 4034/19995 objects degraded (20.175%); 352/19995 objects misplaced (1.760%); 0 B/s, 609 objects/s recovering 2024-08-19T14:43:51.807 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:43:51.832 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:51.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:51.833 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:43:51.934 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:43:51.959 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:51.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:51.960 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:43:52.061 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:43:52.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:52.086 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:52.086 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:43:52.187 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:43:52.214 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:52.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:52.215 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:43:52.316 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:43:52.342 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:52.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:52.343 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:43:52.444 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:43:52.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:52.470 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:52.470 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:43:52.571 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:43:52.595 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:52.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:52.596 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:43:52.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:52 smithi031 ceph-mon[16336]: osdmap e813: 12 total, 12 up, 8 in 2024-08-19T14:43:52.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:52 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:43:52.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:52 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:43:52.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:52 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:43:52.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:52 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:43:52.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:52 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:43:52.696 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:43:52.742 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:52.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:52.744 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:43:52.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:52 smithi032 ceph-mon[20004]: osdmap e813: 12 total, 12 up, 8 in 2024-08-19T14:43:52.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:52 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:43:52.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:52 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:43:52.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:52 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:43:52.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:52 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:43:52.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:52 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:43:52.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:52 smithi045 ceph-mon[20143]: osdmap e813: 12 total, 12 up, 8 in 2024-08-19T14:43:52.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:52 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:43:52.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:52 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:43:52.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:52 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:43:52.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:52 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:43:52.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:52 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:43:52.844 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:43:52.869 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:52.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:52.871 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:43:52.972 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:43:52.996 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:52.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:52.998 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:43:53.099 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:43:53.122 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:53.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:53.123 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:43:53.224 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:43:53.249 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:53.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:53.250 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:43:53.351 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:43:53.389 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:53.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:53.390 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:43:53.490 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:43:53.538 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:53.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:53.539 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:43:53.640 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:43:53.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:53.687 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:53.688 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:43:53.788 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:43:53.813 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:53.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:53.814 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:43:53.915 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:43:53.959 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:53.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:53.960 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:43:54.061 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:43:54.087 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:54.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:54.088 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:43:54.189 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:43:54.193 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:53 smithi031 ceph-mon[16336]: pgmap v1932: 57 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 10 active+recovering+undersized+degraded+remapped, 45 active+clean; 343 MiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 618 op/s; 4118/16467 objects degraded (25.008%); 267/16467 objects misplaced (1.621%); 0 B/s, 624 objects/s recovering 2024-08-19T14:43:54.194 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:53 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 4118/16467 objects degraded (25.008%), 11 pgs degraded (PG_DEGRADED) 2024-08-19T14:43:54.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:54.217 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:54.217 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:43:54.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:53 smithi032 ceph-mon[20004]: pgmap v1932: 57 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 10 active+recovering+undersized+degraded+remapped, 45 active+clean; 343 MiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 618 op/s; 4118/16467 objects degraded (25.008%); 267/16467 objects misplaced (1.621%); 0 B/s, 624 objects/s recovering 2024-08-19T14:43:54.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:53 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 4118/16467 objects degraded (25.008%), 11 pgs degraded (PG_DEGRADED) 2024-08-19T14:43:54.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:53 smithi045 ceph-mon[20143]: pgmap v1932: 57 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 10 active+recovering+undersized+degraded+remapped, 45 active+clean; 343 MiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 618 op/s; 4118/16467 objects degraded (25.008%); 267/16467 objects misplaced (1.621%); 0 B/s, 624 objects/s recovering 2024-08-19T14:43:54.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:53 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 4118/16467 objects degraded (25.008%), 11 pgs degraded (PG_DEGRADED) 2024-08-19T14:43:54.318 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:43:54.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:54.343 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:54.344 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:43:54.445 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:43:54.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:54.471 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:54.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:43:54.573 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:43:54.594 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:54.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:54.597 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:43:54.698 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:43:54.723 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:54.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:54.724 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:43:54.825 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:43:54.850 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:54.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:54.851 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:43:54.952 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:43:54.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:54.981 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:54.981 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:43:55.082 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:43:55.111 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:55.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:55.112 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:43:55.213 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:43:55.236 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:55.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:55.237 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:43:55.296 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:43:55.296 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:43:55.338 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:43:55.360 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:55.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:55.361 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:43:55.463 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd reweight 8 1 2024-08-19T14:43:55.463 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:43:55.491 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:55.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:55.492 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:43:55.593 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:43:55.621 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:55.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:55.623 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:43:55.724 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:43:55.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:55.755 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:55.756 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:43:55.777 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:43:55.856 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:43:55.888 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:55.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:55.889 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:43:55.990 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:43:56.014 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:56.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:56.015 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:43:56.116 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:43:56.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:56.153 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:56.153 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:43:56.207 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:55 smithi031 ceph-mon[16336]: pgmap v1933: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 9 active+recovering+undersized+degraded+remapped, 46 active+clean; 267 MiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 639 op/s; 4117/12786 objects degraded (32.199%); 267/12786 objects misplaced (2.088%); 0 B/s, 569 objects/s recovering 2024-08-19T14:43:56.232 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-19T14:43:56.254 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:43:56.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:55 smithi032 ceph-mon[20004]: pgmap v1933: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 9 active+recovering+undersized+degraded+remapped, 46 active+clean; 267 MiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 639 op/s; 4117/12786 objects degraded (32.199%); 267/12786 objects misplaced (2.088%); 0 B/s, 569 objects/s recovering 2024-08-19T14:43:56.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:56.291 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:56.292 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:43:56.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:55 smithi045 ceph-mon[20143]: pgmap v1933: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 9 active+recovering+undersized+degraded+remapped, 46 active+clean; 267 MiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 639 op/s; 4117/12786 objects degraded (32.199%); 267/12786 objects misplaced (2.088%); 0 B/s, 569 objects/s recovering 2024-08-19T14:43:56.393 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:43:56.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:56.434 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:56.434 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:43:56.535 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:43:56.559 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:56.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:56.560 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:43:56.565 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:43:56.661 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:43:56.686 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:56.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:56.687 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:43:56.788 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:43:56.835 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:56.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:56.836 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:43:56.900 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:43:56.932 INFO:teuthology.orchestra.run.smithi031.stderr:reweighted osd.8 to 1 (10000) 2024-08-19T14:43:56.937 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:43:56.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:56 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2432979165' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-08-19T14:43:56.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:56.958 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:56.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:43:57.059 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:43:57.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:57.094 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:57.094 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:43:57.196 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:43:57.227 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:57.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:57.228 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:43:57.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:56 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2432979165' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-08-19T14:43:57.275 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Removed 112461 objects 2024-08-19T14:43:57.276 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Clean up completed and total clean up time :134.973 2024-08-19T14:43:57.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:56 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2432979165' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-08-19T14:43:57.328 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:43:57.350 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:57.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:57.351 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:43:57.452 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:43:57.474 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:57.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:57.475 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:43:57.576 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:43:57.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:57.603 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:57.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:43:57.704 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:43:57.729 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:57.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:57.730 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:43:57.821 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd reweight 2 1 2024-08-19T14:43:57.831 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:43:57.889 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:57.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:57.891 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:43:57.947 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is unset 2024-08-19T14:43:57.992 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:43:58.015 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:58.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:58.016 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:43:58.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:57 smithi031 ceph-mon[16336]: pgmap v1934: 57 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 9 active+recovering+undersized+degraded+remapped, 46 active+clean; 147 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 719 op/s; 4128/7032 objects degraded (58.703%); 108/7032 objects misplaced (1.536%); 1.1 MiB/s, 572 objects/s recovering 2024-08-19T14:43:58.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:57 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2432979165' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-08-19T14:43:58.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:57 smithi031 ceph-mon[16336]: osdmap e814: 12 total, 12 up, 8 in 2024-08-19T14:43:58.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:57 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2857750670' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:43:58.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:57 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:43:58.117 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:43:58.146 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:58.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:58.147 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:43:58.249 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:43:58.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:57 smithi032 ceph-mon[20004]: pgmap v1934: 57 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 9 active+recovering+undersized+degraded+remapped, 46 active+clean; 147 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 719 op/s; 4128/7032 objects degraded (58.703%); 108/7032 objects misplaced (1.536%); 1.1 MiB/s, 572 objects/s recovering 2024-08-19T14:43:58.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:57 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2432979165' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-08-19T14:43:58.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:57 smithi032 ceph-mon[20004]: osdmap e814: 12 total, 12 up, 8 in 2024-08-19T14:43:58.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:57 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2857750670' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:43:58.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:57 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:43:58.271 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:58.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:58.272 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:43:58.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:57 smithi045 ceph-mon[20143]: pgmap v1934: 57 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 9 active+recovering+undersized+degraded+remapped, 46 active+clean; 147 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 719 op/s; 4128/7032 objects degraded (58.703%); 108/7032 objects misplaced (1.536%); 1.1 MiB/s, 572 objects/s recovering 2024-08-19T14:43:58.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:57 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2432979165' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-08-19T14:43:58.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:57 smithi045 ceph-mon[20143]: osdmap e814: 12 total, 12 up, 8 in 2024-08-19T14:43:58.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:57 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2857750670' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:43:58.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:57 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:43:58.372 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:43:58.398 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:58.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:58.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:43:58.501 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:43:58.536 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:58.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:58.538 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:43:58.639 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:43:58.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:58.668 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:43:58.668 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:43:58.770 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:43:58.796 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:58.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:58.798 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:43:58.899 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:43:58.907 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:43:58.913 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:43:58.929 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:58.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:58.930 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:43:59.034 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:43:59.058 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:58 smithi031 ceph-mon[16336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:43:59.058 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:58 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:43:59.058 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:58 smithi031 ceph-mon[16336]: osdmap e815: 12 total, 12 up, 8 in 2024-08-19T14:43:59.058 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:58 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 4128/7032 objects degraded (58.703%), 10 pgs degraded (PG_DEGRADED) 2024-08-19T14:43:59.059 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:59.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:59.060 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:43:59.160 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:43:59.184 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:59.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:59.185 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:43:59.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:58 smithi032 ceph-mon[20004]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:43:59.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:58 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:43:59.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:58 smithi032 ceph-mon[20004]: osdmap e815: 12 total, 12 up, 8 in 2024-08-19T14:43:59.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:58 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 4128/7032 objects degraded (58.703%), 10 pgs degraded (PG_DEGRADED) 2024-08-19T14:43:59.285 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:43:59.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:58 smithi045 ceph-mon[20143]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:43:59.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:58 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:43:59.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:58 smithi045 ceph-mon[20143]: osdmap e815: 12 total, 12 up, 8 in 2024-08-19T14:43:59.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:58 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 4128/7032 objects degraded (58.703%), 10 pgs degraded (PG_DEGRADED) 2024-08-19T14:43:59.317 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:59.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:59.318 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:43:59.419 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:43:59.444 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:59.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:59.445 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:43:59.545 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:43:59.567 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:43:59.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:59.568 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:43:59.669 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:43:59.700 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:59.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:59.701 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:43:59.802 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:43:59.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:59.834 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:59.834 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:43:59.935 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:43:59.966 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:43:59.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:43:59.967 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:44:00.068 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:44:00.090 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:00.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:00.091 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:44:00.192 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:44:00.222 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:00.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:00.223 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:44:00.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:59 smithi032 ceph-mon[20004]: pgmap v1937: 57 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 9 active+recovering+undersized+degraded+remapped, 46 active+clean; 69 MiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 675 op/s; 4128/3282 objects degraded (125.777%); 108/3282 objects misplaced (3.291%); 1.3 MiB/s, 532 objects/s recovering 2024-08-19T14:44:00.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:43:59 smithi032 ceph-mon[20004]: osdmap e816: 12 total, 12 up, 8 in 2024-08-19T14:44:00.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:59 smithi045 ceph-mon[20143]: pgmap v1937: 57 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 9 active+recovering+undersized+degraded+remapped, 46 active+clean; 69 MiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 675 op/s; 4128/3282 objects degraded (125.777%); 108/3282 objects misplaced (3.291%); 1.3 MiB/s, 532 objects/s recovering 2024-08-19T14:44:00.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:43:59 smithi045 ceph-mon[20143]: osdmap e816: 12 total, 12 up, 8 in 2024-08-19T14:44:00.324 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:44:00.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:00.356 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:00.356 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:44:00.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:59 smithi031 ceph-mon[16336]: pgmap v1937: 57 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 9 active+recovering+undersized+degraded+remapped, 46 active+clean; 69 MiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 675 op/s; 4128/3282 objects degraded (125.777%); 108/3282 objects misplaced (3.291%); 1.3 MiB/s, 532 objects/s recovering 2024-08-19T14:44:00.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:43:59 smithi031 ceph-mon[16336]: osdmap e816: 12 total, 12 up, 8 in 2024-08-19T14:44:00.458 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:44:00.489 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:00.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:00.490 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:44:00.590 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:44:00.616 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:00.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:00.617 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:44:00.718 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:44:00.741 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:00.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:00.742 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:44:00.839 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-19T14:44:00.843 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:44:00.906 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:00.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:00.907 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:44:01.008 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:44:01.029 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:01.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:01.030 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:44:01.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:00 smithi031 ceph-mon[16336]: osdmap e817: 12 total, 12 up, 8 in 2024-08-19T14:44:01.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:00 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:44:01.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:00 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:44:01.131 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:44:01.153 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:01.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:01.154 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:44:01.255 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:44:01.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:00 smithi032 ceph-mon[20004]: osdmap e817: 12 total, 12 up, 8 in 2024-08-19T14:44:01.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:00 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:44:01.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:00 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:44:01.280 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:01.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:01.281 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:44:01.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:00 smithi045 ceph-mon[20143]: osdmap e817: 12 total, 12 up, 8 in 2024-08-19T14:44:01.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:00 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:44:01.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:00 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:44:01.381 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:44:01.404 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:01.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:01.405 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:44:01.506 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:44:01.528 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:01.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:01.529 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:44:01.580 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:44:01.580 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:44:01.599 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:44:01.599 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:44:01.620 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:44:01.620 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:44:01.630 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:44:01.648 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:44:01.649 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:44:01.657 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:01.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:01.658 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:44:01.669 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:44:01.673 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:44:01.673 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:44:01.698 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:44:01.699 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:44:01.714 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:44:01.714 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:44:01.733 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:44:01.733 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:44:01.753 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:44:01.753 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:44:01.758 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:44:01.772 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:44:01.772 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:44:01.781 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:01.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:01.782 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:44:01.792 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:44:01.792 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:44:01.812 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:44:01.812 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:44:01.883 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:44:01.906 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:01.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:01.907 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:44:01.972 INFO:teuthology.orchestra.run.smithi031.stderr:reweighted osd.2 to 1 (10000) 2024-08-19T14:44:01.984 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:01 smithi031 ceph-mon[16336]: pgmap v1940: 57 pgs: 1 active+recovering+remapped, 9 active+recovering+undersized+degraded+remapped, 47 active+clean; 20 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 508 op/s; 1469/927 objects degraded (158.468%); 108/927 objects misplaced (11.650%); 0 B/s, 795 objects/s recovering 2024-08-19T14:44:01.984 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:01 smithi031 ceph-mon[16336]: osdmap e818: 12 total, 12 up, 8 in 2024-08-19T14:44:01.984 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:01 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3339008221' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-08-19T14:44:01.984 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:01 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2485519596' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:44:02.008 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:44:02.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:02.033 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:02.033 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:44:02.135 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:44:02.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:02.161 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:02.162 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:44:02.220 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:01 smithi032 ceph-mon[20004]: pgmap v1940: 57 pgs: 1 active+recovering+remapped, 9 active+recovering+undersized+degraded+remapped, 47 active+clean; 20 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 508 op/s; 1469/927 objects degraded (158.468%); 108/927 objects misplaced (11.650%); 0 B/s, 795 objects/s recovering 2024-08-19T14:44:02.220 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:01 smithi032 ceph-mon[20004]: osdmap e818: 12 total, 12 up, 8 in 2024-08-19T14:44:02.221 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:01 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3339008221' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-08-19T14:44:02.221 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:01 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2485519596' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:44:02.222 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-08-19T14:44:02.262 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:44:02.290 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:02.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:02.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:44:02.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:01 smithi045 ceph-mon[20143]: pgmap v1940: 57 pgs: 1 active+recovering+remapped, 9 active+recovering+undersized+degraded+remapped, 47 active+clean; 20 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 508 op/s; 1469/927 objects degraded (158.468%); 108/927 objects misplaced (11.650%); 0 B/s, 795 objects/s recovering 2024-08-19T14:44:02.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:01 smithi045 ceph-mon[20143]: osdmap e818: 12 total, 12 up, 8 in 2024-08-19T14:44:02.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:01 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3339008221' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-08-19T14:44:02.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:01 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2485519596' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:44:02.392 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:44:02.419 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:02.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:02.420 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:44:02.521 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:44:02.530 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:44:02.544 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:02.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:02.545 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:44:02.646 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:44:02.668 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:02.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:02.669 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:44:02.770 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:44:02.793 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:02.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:02.794 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:44:02.895 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:44:02.921 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:02.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:02.922 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:44:03.022 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:44:03.049 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:03.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:03.050 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:44:03.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:02 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3339008221' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-08-19T14:44:03.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:02 smithi031 ceph-mon[16336]: osdmap e819: 12 total, 12 up, 8 in 2024-08-19T14:44:03.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:02 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:44:03.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:02 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:44:03.121 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_6 2024-08-19T14:44:03.121 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd pool rm unique_pool_6 unique_pool_6 --yes-i-really-really-mean-it 2024-08-19T14:44:03.151 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:44:03.187 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:03.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:03.189 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:44:03.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:02 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3339008221' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-08-19T14:44:03.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:02 smithi032 ceph-mon[20004]: osdmap e819: 12 total, 12 up, 8 in 2024-08-19T14:44:03.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:02 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:44:03.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:02 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:44:03.289 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:44:03.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:02 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3339008221' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-08-19T14:44:03.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:02 smithi045 ceph-mon[20143]: osdmap e819: 12 total, 12 up, 8 in 2024-08-19T14:44:03.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:02 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:44:03.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:02 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:44:03.315 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:03.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:03.316 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:44:03.417 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:44:03.442 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:03.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:03.443 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:44:03.523 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd reweight 11 1 2024-08-19T14:44:03.544 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:44:03.588 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:44:03.593 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:03.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:03.595 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:44:03.695 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:44:03.720 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:03.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:03.721 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:44:03.823 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:44:03.848 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:03.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:03.849 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:44:03.950 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:44:03.971 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:03.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:03.972 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:44:03.978 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is set 2024-08-19T14:44:03.994 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:03 smithi031 ceph-mon[16336]: pgmap v1943: 57 pgs: 1 active+recovering+remapped, 9 active+recovering+undersized+degraded+remapped, 47 active+clean; 1.3 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 270 op/s; 1289/39 objects degraded (3305.128%); 108/39 objects misplaced (276.923%); 0 B/s, 1.25k objects/s recovering 2024-08-19T14:44:03.994 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:03 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:44:03.994 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:03 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:44:03.994 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:03 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:44:03.995 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:03 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:44:03.995 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:03 smithi031 ceph-mon[16336]: osdmap e820: 12 total, 12 up, 8 in 2024-08-19T14:44:03.995 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:03 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 1289/39 objects degraded (3305.128%), 9 pgs degraded (PG_DEGRADED) 2024-08-19T14:44:03.995 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:03 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2412793624' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:44:04.072 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:44:04.098 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:04.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:04.099 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:44:04.200 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:44:04.224 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:04.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:04.226 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:44:04.227 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:03 smithi032 ceph-mon[20004]: pgmap v1943: 57 pgs: 1 active+recovering+remapped, 9 active+recovering+undersized+degraded+remapped, 47 active+clean; 1.3 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 270 op/s; 1289/39 objects degraded (3305.128%); 108/39 objects misplaced (276.923%); 0 B/s, 1.25k objects/s recovering 2024-08-19T14:44:04.227 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:03 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:44:04.227 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:03 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:44:04.227 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:03 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:44:04.227 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:03 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:44:04.227 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:03 smithi032 ceph-mon[20004]: osdmap e820: 12 total, 12 up, 8 in 2024-08-19T14:44:04.228 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:03 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 1289/39 objects degraded (3305.128%), 9 pgs degraded (PG_DEGRADED) 2024-08-19T14:44:04.228 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:03 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2412793624' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:44:04.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:03 smithi045 ceph-mon[20143]: pgmap v1943: 57 pgs: 1 active+recovering+remapped, 9 active+recovering+undersized+degraded+remapped, 47 active+clean; 1.3 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 270 op/s; 1289/39 objects degraded (3305.128%); 108/39 objects misplaced (276.923%); 0 B/s, 1.25k objects/s recovering 2024-08-19T14:44:04.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:03 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:44:04.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:03 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:44:04.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:03 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:44:04.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:03 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:44:04.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:03 smithi045 ceph-mon[20143]: osdmap e820: 12 total, 12 up, 8 in 2024-08-19T14:44:04.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:03 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 1289/39 objects degraded (3305.128%), 9 pgs degraded (PG_DEGRADED) 2024-08-19T14:44:04.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:03 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2412793624' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:44:04.326 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:44:04.353 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:04.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:04.354 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:44:04.455 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:44:04.476 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:04.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:04.477 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:44:04.578 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:44:04.603 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:04.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:04.604 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:44:04.705 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:44:04.732 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:04.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:04.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:44:04.833 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:44:04.842 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:44:04.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:04.858 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:04.858 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:44:04.929 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:44:04.959 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:44:04.980 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'unique_pool_6' removed 2024-08-19T14:44:04.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:04.986 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:04.986 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:44:04.995 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:04 smithi031 ceph-mon[16336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:44:04.995 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:04 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2412793624' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:44:04.995 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:04 smithi031 ceph-mon[16336]: osdmap e821: 12 total, 12 up, 8 in 2024-08-19T14:44:04.995 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:04 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1912524657' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]: dispatch 2024-08-19T14:44:04.996 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:04 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1912524657' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]': finished 2024-08-19T14:44:04.996 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:04 smithi031 ceph-mon[16336]: osdmap e822: 12 total, 12 up, 8 in 2024-08-19T14:44:05.088 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:44:05.110 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:05.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:05.111 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:44:05.212 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:44:05.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:05.237 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:05.237 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:44:05.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:04 smithi032 ceph-mon[20004]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:44:05.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:04 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2412793624' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:44:05.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:04 smithi032 ceph-mon[20004]: osdmap e821: 12 total, 12 up, 8 in 2024-08-19T14:44:05.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:04 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1912524657' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]: dispatch 2024-08-19T14:44:05.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:04 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1912524657' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]': finished 2024-08-19T14:44:05.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:04 smithi032 ceph-mon[20004]: osdmap e822: 12 total, 12 up, 8 in 2024-08-19T14:44:05.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:04 smithi045 ceph-mon[20143]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:44:05.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:04 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2412793624' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:44:05.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:04 smithi045 ceph-mon[20143]: osdmap e821: 12 total, 12 up, 8 in 2024-08-19T14:44:05.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:04 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1912524657' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]: dispatch 2024-08-19T14:44:05.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:04 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1912524657' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]': finished 2024-08-19T14:44:05.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:04 smithi045 ceph-mon[20143]: osdmap e822: 12 total, 12 up, 8 in 2024-08-19T14:44:05.337 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:44:05.359 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:05.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:05.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:44:05.460 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:44:05.482 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:05.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:05.483 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:44:05.584 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:44:05.605 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:05.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:05.606 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:44:05.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-08-19T14:44:05.684+0000 7fadd2d9d640 0 --2- 172.21.15.31:0/30780600 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fadd4102c20 0x7fadd4197760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-08-19T14:44:05.707 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:44:05.730 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:05.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:05.731 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:44:05.833 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:44:05.855 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:05.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:05.856 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:44:05.957 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:44:05.978 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:05.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:05.979 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:44:05.995 INFO:teuthology.orchestra.run.smithi031.stderr:reweighted osd.11 to 1 (10000) 2024-08-19T14:44:06.080 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:44:06.101 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:06.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:06.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:44:06.107 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:05 smithi031 ceph-mon[16336]: pgmap v1946: 57 pgs: 1 peering, 1 active+recovering+remapped, 8 active+recovering+undersized+degraded+remapped, 47 active+clean; 1.3 MiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 73 op/s; 1287/39 objects degraded (3300.000%); 108/39 objects misplaced (276.923%); 0 B/s, 1.27k objects/s recovering 2024-08-19T14:44:06.107 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:05 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/30780600' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-08-19T14:44:06.133 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-08-19T14:44:06.134 INFO:tasks.radosbench:Beginning radosbench... 2024-08-19T14:44:06.134 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_7 2024-08-19T14:44:06.134 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd pool create unique_pool_7 16 2024-08-19T14:44:06.204 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:44:06.236 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:06.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:06.237 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:44:06.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:05 smithi032 ceph-mon[20004]: pgmap v1946: 57 pgs: 1 peering, 1 active+recovering+remapped, 8 active+recovering+undersized+degraded+remapped, 47 active+clean; 1.3 MiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 73 op/s; 1287/39 objects degraded (3300.000%); 108/39 objects misplaced (276.923%); 0 B/s, 1.27k objects/s recovering 2024-08-19T14:44:06.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:05 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/30780600' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-08-19T14:44:06.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:05 smithi045 ceph-mon[20143]: pgmap v1946: 57 pgs: 1 peering, 1 active+recovering+remapped, 8 active+recovering+undersized+degraded+remapped, 47 active+clean; 1.3 MiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 73 op/s; 1287/39 objects degraded (3300.000%); 108/39 objects misplaced (276.923%); 0 B/s, 1.27k objects/s recovering 2024-08-19T14:44:06.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:05 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/30780600' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-08-19T14:44:06.338 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:44:06.361 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:06.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:06.363 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:44:06.464 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:44:06.485 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:06.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:06.486 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:44:06.588 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:44:06.614 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:06.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:06.615 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:44:06.716 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:44:06.740 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:06.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:06.741 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:44:06.842 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:44:06.866 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:06.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:06.867 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:44:06.895 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-19T14:44:06.968 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:44:06.993 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:06.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:06.994 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:44:07.095 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:44:07.119 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:07.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:07.121 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:44:07.221 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:44:07.247 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:07.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:07.248 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:44:07.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:07 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/30780600' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-08-19T14:44:07.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:07 smithi032 ceph-mon[20004]: osdmap e823: 12 total, 12 up, 8 in 2024-08-19T14:44:07.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:07 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/30780600' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-08-19T14:44:07.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:07 smithi045 ceph-mon[20143]: osdmap e823: 12 total, 12 up, 8 in 2024-08-19T14:44:07.350 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:44:07.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:07 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/30780600' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-08-19T14:44:07.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:07 smithi031 ceph-mon[16336]: osdmap e823: 12 total, 12 up, 8 in 2024-08-19T14:44:07.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:07.376 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:07.376 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:44:07.403 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:44:07.403 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-08-19T14:44:07.403 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-08-19T14:44:07.403 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-08-19T14:44:07.403 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.404 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26911.0:198785 8.8 8:13c17521:::benchmark_data_smithi045_137103_object86321:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e801)", 2024-08-19T14:44:07.404 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:43:17.891055+0000", 2024-08-19T14:44:07.404 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 48.509309770000002, 2024-08-19T14:44:07.404 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.112352038, 2024-08-19T14:44:07.404 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:44:07.404 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:44:07.404 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:44:07.404 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:44:07.404 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26911", 2024-08-19T14:44:07.404 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3494993995", 2024-08-19T14:44:07.405 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 198785 2024-08-19T14:44:07.405 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.405 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:44:07.405 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.405 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:44:07.405 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:17.891055+0000", 2024-08-19T14:44:07.405 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:44:07.405 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.405 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.405 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:44:07.405 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:17.891055+0000", 2024-08-19T14:44:07.405 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:44:07.405 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.405 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.406 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:44:07.406 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:17.891057+0000", 2024-08-19T14:44:07.406 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.136e-06 2024-08-19T14:44:07.406 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.406 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.406 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:44:07.406 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:17.891066+0000", 2024-08-19T14:44:07.406 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.38e-06 2024-08-19T14:44:07.406 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.406 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.406 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:44:07.406 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:17.891090+0000", 2024-08-19T14:44:07.406 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3473e-05 2024-08-19T14:44:07.407 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.407 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.407 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:44:07.407 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:17.891104+0000", 2024-08-19T14:44:07.407 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4239999999999999e-05 2024-08-19T14:44:07.407 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.407 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.408 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:44:07.408 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:17.891206+0000", 2024-08-19T14:44:07.408 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000101908 2024-08-19T14:44:07.408 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.408 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.408 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:44:07.408 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:17.891339+0000", 2024-08-19T14:44:07.408 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013358200000000001 2024-08-19T14:44:07.408 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.408 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.408 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 9,10", 2024-08-19T14:44:07.408 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:17.966095+0000", 2024-08-19T14:44:07.408 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.074756053000000003 2024-08-19T14:44:07.409 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.409 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.409 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:44:07.409 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:17.973930+0000", 2024-08-19T14:44:07.409 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0078345680000000001 2024-08-19T14:44:07.409 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.409 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.409 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:44:07.409 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:17.981951+0000", 2024-08-19T14:44:07.409 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0080210820000000006 2024-08-19T14:44:07.409 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.409 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.409 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:44:07.410 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:18.002607+0000", 2024-08-19T14:44:07.410 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.020656437999999999 2024-08-19T14:44:07.410 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.410 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.410 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:44:07.410 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:18.002681+0000", 2024-08-19T14:44:07.410 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3963999999999999e-05 2024-08-19T14:44:07.410 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.410 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.410 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:44:07.410 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:18.003407+0000", 2024-08-19T14:44:07.410 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00072521399999999996 2024-08-19T14:44:07.410 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:44:07.411 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:44:07.411 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:44:07.411 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.411 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.411 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26911.0:198786 8.8 8:136c67d1:::benchmark_data_smithi045_137103_object86322:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e801)", 2024-08-19T14:44:07.411 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:43:17.897185+0000", 2024-08-19T14:44:07.411 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 48.503179633999999, 2024-08-19T14:44:07.411 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.106900863, 2024-08-19T14:44:07.411 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:44:07.411 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:44:07.412 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:44:07.412 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:44:07.412 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26911", 2024-08-19T14:44:07.412 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3494993995", 2024-08-19T14:44:07.412 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 198786 2024-08-19T14:44:07.412 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.412 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:44:07.412 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.413 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:44:07.414 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:17.897185+0000", 2024-08-19T14:44:07.414 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:44:07.414 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.414 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.414 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:44:07.414 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:17.897185+0000", 2024-08-19T14:44:07.414 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:44:07.414 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.414 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.414 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:44:07.414 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:17.897187+0000", 2024-08-19T14:44:07.414 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8840000000000001e-06 2024-08-19T14:44:07.415 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.415 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.415 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:44:07.415 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:17.897197+0000", 2024-08-19T14:44:07.415 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.9490000000000008e-06 2024-08-19T14:44:07.415 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.415 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.415 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:44:07.415 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:17.966321+0000", 2024-08-19T14:44:07.415 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.069124367000000006 2024-08-19T14:44:07.416 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.416 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.416 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:44:07.416 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:17.966330+0000", 2024-08-19T14:44:07.416 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.0550000000000005e-06 2024-08-19T14:44:07.416 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.416 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.416 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:44:07.416 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:17.973960+0000", 2024-08-19T14:44:07.416 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.007629907 2024-08-19T14:44:07.416 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.416 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.416 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:44:07.417 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:17.974104+0000", 2024-08-19T14:44:07.417 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000144165 2024-08-19T14:44:07.417 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.417 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.417 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 9,10", 2024-08-19T14:44:07.417 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:17.974243+0000", 2024-08-19T14:44:07.417 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013851099999999999 2024-08-19T14:44:07.417 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.417 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.417 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:44:07.417 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:17.986248+0000", 2024-08-19T14:44:07.417 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.012005277999999999 2024-08-19T14:44:07.417 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.417 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.418 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:44:07.418 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:17.988393+0000", 2024-08-19T14:44:07.418 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0021452760000000002 2024-08-19T14:44:07.418 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.418 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.418 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:44:07.418 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:18.003477+0000", 2024-08-19T14:44:07.418 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0150844 2024-08-19T14:44:07.418 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.418 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.418 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:44:07.423 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:18.003539+0000", 2024-08-19T14:44:07.423 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1297000000000006e-05 2024-08-19T14:44:07.424 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.424 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.424 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:44:07.424 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:18.004086+0000", 2024-08-19T14:44:07.424 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00054677399999999998 2024-08-19T14:44:07.424 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:44:07.424 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:44:07.424 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:44:07.424 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.424 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.424 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26911.0:202285 8.11 8:8c27a3a6:::benchmark_data_smithi045_137103_object89821:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e801)", 2024-08-19T14:44:07.424 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:43:23.118440+0000", 2024-08-19T14:44:07.424 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 43.281924478000001, 2024-08-19T14:44:07.425 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.102394708, 2024-08-19T14:44:07.425 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:44:07.425 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:44:07.425 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:44:07.425 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:44:07.425 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26911", 2024-08-19T14:44:07.425 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3494993995", 2024-08-19T14:44:07.425 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 202285 2024-08-19T14:44:07.425 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.425 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:44:07.425 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.425 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:44:07.426 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:23.118440+0000", 2024-08-19T14:44:07.426 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:44:07.426 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.426 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.426 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:44:07.426 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:23.118440+0000", 2024-08-19T14:44:07.426 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:44:07.426 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.426 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.426 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:44:07.427 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:23.118442+0000", 2024-08-19T14:44:07.427 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.86e-06 2024-08-19T14:44:07.427 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.427 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.427 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:44:07.427 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:23.118448+0000", 2024-08-19T14:44:07.427 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1040000000000003e-06 2024-08-19T14:44:07.427 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.427 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.427 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:44:07.427 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:23.118462+0000", 2024-08-19T14:44:07.427 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4674999999999999e-05 2024-08-19T14:44:07.427 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.427 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.428 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:44:07.428 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:23.118470+0000", 2024-08-19T14:44:07.428 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1600000000000001e-06 2024-08-19T14:44:07.428 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.428 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.428 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:44:07.428 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:23.120227+0000", 2024-08-19T14:44:07.428 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001756954 2024-08-19T14:44:07.428 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.428 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.428 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:44:07.428 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:23.120383+0000", 2024-08-19T14:44:07.428 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00015682200000000001 2024-08-19T14:44:07.428 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.429 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.429 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 9,10", 2024-08-19T14:44:07.429 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:23.120515+0000", 2024-08-19T14:44:07.429 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000131383 2024-08-19T14:44:07.429 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.429 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.429 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:44:07.429 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:23.123992+0000", 2024-08-19T14:44:07.429 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0034774419999999999 2024-08-19T14:44:07.429 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.429 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.429 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:44:07.429 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:23.127960+0000", 2024-08-19T14:44:07.429 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00396789 2024-08-19T14:44:07.430 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.430 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.430 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:44:07.430 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:23.155086+0000", 2024-08-19T14:44:07.430 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.027126199 2024-08-19T14:44:07.430 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.430 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.430 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:44:07.430 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:23.219211+0000", 2024-08-19T14:44:07.430 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.064124347999999998 2024-08-19T14:44:07.430 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.430 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.430 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:44:07.430 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:23.220835+0000", 2024-08-19T14:44:07.431 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001623871 2024-08-19T14:44:07.431 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:44:07.431 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:44:07.431 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:44:07.431 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.431 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.431 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDPGPushReply(8.8 801/798 [PushReplyOp(8:13999242:::benchmark_data_smithi045_137103_object99957:head)])", 2024-08-19T14:44:07.431 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:43:26.508822+0000", 2024-08-19T14:44:07.431 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 39.891542358000002, 2024-08-19T14:44:07.431 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.11501145, 2024-08-19T14:44:07.431 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:44:07.431 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:44:07.431 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:44:07.431 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:44:07.431 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.432 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:44:07.432 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:26.508822+0000", 2024-08-19T14:44:07.432 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:44:07.432 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.432 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.432 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:44:07.432 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:26.508822+0000", 2024-08-19T14:44:07.432 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:44:07.432 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.432 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.432 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:44:07.432 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:26.508822+0000", 2024-08-19T14:44:07.432 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8100000000000003e-07 2024-08-19T14:44:07.432 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.433 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.433 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:44:07.433 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:26.508827+0000", 2024-08-19T14:44:07.433 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.065e-06 2024-08-19T14:44:07.433 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.433 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.433 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:44:07.433 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:26.508835+0000", 2024-08-19T14:44:07.433 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.0919999999999998e-06 2024-08-19T14:44:07.433 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.433 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.433 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:44:07.433 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:26.508839+0000", 2024-08-19T14:44:07.433 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1810000000000003e-06 2024-08-19T14:44:07.434 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.434 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.434 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:44:07.434 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:26.508892+0000", 2024-08-19T14:44:07.434 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2902000000000003e-05 2024-08-19T14:44:07.434 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.434 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.434 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:44:07.434 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:26.623833+0000", 2024-08-19T14:44:07.434 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.11494172900000001 2024-08-19T14:44:07.434 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:44:07.434 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:44:07.434 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:44:07.434 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.435 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.435 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.26911.0:204467 8.3 e801/731 8:c3fde17f:::benchmark_data_smithi045_137103_object92003:head v 801'8497, mlcod=801'8495)", 2024-08-19T14:44:07.435 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:43:26.509468+0000", 2024-08-19T14:44:07.435 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 39.890896224000002, 2024-08-19T14:44:07.435 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.114393409, 2024-08-19T14:44:07.435 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:44:07.435 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:44:07.435 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:44:07.435 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:44:07.435 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.435 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:44:07.435 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:26.509468+0000", 2024-08-19T14:44:07.435 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:44:07.435 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.436 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.436 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:44:07.436 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:26.509468+0000", 2024-08-19T14:44:07.436 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:44:07.436 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.436 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.436 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:44:07.436 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:26.509469+0000", 2024-08-19T14:44:07.436 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4099999999999999e-07 2024-08-19T14:44:07.436 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.436 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.436 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:44:07.436 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:26.509473+0000", 2024-08-19T14:44:07.436 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7890000000000002e-06 2024-08-19T14:44:07.437 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.437 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.437 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:44:07.437 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:26.509481+0000", 2024-08-19T14:44:07.437 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.379e-06 2024-08-19T14:44:07.437 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.437 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.437 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:44:07.437 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:26.509485+0000", 2024-08-19T14:44:07.437 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3309999999999997e-06 2024-08-19T14:44:07.437 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.437 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.437 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:44:07.437 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:26.509532+0000", 2024-08-19T14:44:07.438 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7372e-05 2024-08-19T14:44:07.438 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.438 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.438 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:44:07.438 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:26.509551+0000", 2024-08-19T14:44:07.438 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8887000000000001e-05 2024-08-19T14:44:07.438 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.438 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.438 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:44:07.438 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:26.623441+0000", 2024-08-19T14:44:07.438 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.113889563 2024-08-19T14:44:07.438 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.438 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.439 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:44:07.439 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:26.623862+0000", 2024-08-19T14:44:07.439 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00042064700000000001 2024-08-19T14:44:07.439 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:44:07.439 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:44:07.439 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:44:07.439 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.439 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.439 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.26911.0:204473 8.16 e801/743 8:6fd26986:::benchmark_data_smithi045_137103_object92009:head v 801'8325, mlcod=801'8323)", 2024-08-19T14:44:07.439 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:43:26.524315+0000", 2024-08-19T14:44:07.439 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 39.876049170999998, 2024-08-19T14:44:07.439 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.10284431300000001, 2024-08-19T14:44:07.439 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:44:07.439 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:44:07.440 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:44:07.440 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:44:07.440 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.440 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:44:07.440 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:26.524315+0000", 2024-08-19T14:44:07.440 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:44:07.440 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.440 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.440 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:44:07.440 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:26.524315+0000", 2024-08-19T14:44:07.440 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:44:07.440 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.440 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.441 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:44:07.441 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:26.524317+0000", 2024-08-19T14:44:07.441 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.66e-06 2024-08-19T14:44:07.441 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.441 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.441 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:44:07.441 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:26.524328+0000", 2024-08-19T14:44:07.441 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1600000000000001e-05 2024-08-19T14:44:07.441 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.441 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.441 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:44:07.441 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:26.623642+0000", 2024-08-19T14:44:07.441 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.099313296999999995 2024-08-19T14:44:07.441 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.442 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.442 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:44:07.442 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:26.623828+0000", 2024-08-19T14:44:07.442 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00018657400000000001 2024-08-19T14:44:07.442 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.442 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.442 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:44:07.442 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:26.623885+0000", 2024-08-19T14:44:07.442 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6549999999999999e-05 2024-08-19T14:44:07.442 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.442 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.442 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:44:07.442 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:26.623907+0000", 2024-08-19T14:44:07.443 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2368e-05 2024-08-19T14:44:07.443 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.443 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.443 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:44:07.443 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:26.627115+0000", 2024-08-19T14:44:07.443 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0032078509999999998 2024-08-19T14:44:07.443 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.443 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.443 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:44:07.443 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:26.627159+0000", 2024-08-19T14:44:07.443 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4413000000000003e-05 2024-08-19T14:44:07.443 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:44:07.443 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:44:07.444 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:44:07.444 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.444 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.444 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26911.0:204475 8.8 8:119f53dd:::benchmark_data_smithi045_137103_object92011:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e801)", 2024-08-19T14:44:07.444 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:43:26.527790+0000", 2024-08-19T14:44:07.444 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 39.872574583000002, 2024-08-19T14:44:07.444 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.12545067500000001, 2024-08-19T14:44:07.444 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:44:07.445 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:44:07.445 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:44:07.445 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:44:07.445 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26911", 2024-08-19T14:44:07.445 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3494993995", 2024-08-19T14:44:07.445 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 204475 2024-08-19T14:44:07.445 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.445 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:44:07.445 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.445 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:44:07.445 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:26.527790+0000", 2024-08-19T14:44:07.445 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:44:07.445 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.446 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.446 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:44:07.446 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:26.527790+0000", 2024-08-19T14:44:07.446 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:44:07.446 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.446 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.446 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:44:07.446 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:26.527791+0000", 2024-08-19T14:44:07.446 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0759999999999999e-06 2024-08-19T14:44:07.446 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.446 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.446 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:44:07.446 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:26.527795+0000", 2024-08-19T14:44:07.447 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3680000000000004e-06 2024-08-19T14:44:07.447 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.447 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.447 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:44:07.447 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:26.623415+0000", 2024-08-19T14:44:07.447 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.095620236999999997 2024-08-19T14:44:07.447 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.447 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.447 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:44:07.447 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:26.623648+0000", 2024-08-19T14:44:07.447 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00023223300000000001 2024-08-19T14:44:07.447 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.447 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.448 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:44:07.448 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:26.631750+0000", 2024-08-19T14:44:07.448 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0081020580000000005 2024-08-19T14:44:07.448 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.448 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.448 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:44:07.448 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:26.631886+0000", 2024-08-19T14:44:07.448 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013607499999999999 2024-08-19T14:44:07.448 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.448 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.448 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 9,10", 2024-08-19T14:44:07.448 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:26.632025+0000", 2024-08-19T14:44:07.448 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013947100000000001 2024-08-19T14:44:07.448 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.449 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.449 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:44:07.449 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:26.640744+0000", 2024-08-19T14:44:07.449 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0087185460000000006 2024-08-19T14:44:07.449 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.449 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.449 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:44:07.449 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:26.648204+0000", 2024-08-19T14:44:07.449 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0074605899999999996 2024-08-19T14:44:07.449 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.449 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.449 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:44:07.449 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:26.652477+0000", 2024-08-19T14:44:07.450 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0042728339999999997 2024-08-19T14:44:07.450 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.450 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.450 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:44:07.450 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:26.652623+0000", 2024-08-19T14:44:07.450 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014620900000000001 2024-08-19T14:44:07.450 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.450 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.450 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:44:07.450 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:26.653240+0000", 2024-08-19T14:44:07.450 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00061697800000000001 2024-08-19T14:44:07.450 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:44:07.450 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:44:07.450 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:44:07.451 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.451 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.451 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26911.0:213835 8.11 8:8deb7a71:::benchmark_data_smithi045_137103_object101371:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e806)", 2024-08-19T14:44:07.451 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:43:41.042187+0000", 2024-08-19T14:44:07.451 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 25.358176960000002, 2024-08-19T14:44:07.451 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.138051496, 2024-08-19T14:44:07.451 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:44:07.451 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:44:07.451 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:44:07.451 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:44:07.451 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26911", 2024-08-19T14:44:07.451 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3494993995", 2024-08-19T14:44:07.451 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 213835 2024-08-19T14:44:07.452 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.452 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:44:07.452 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.452 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:44:07.452 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.042187+0000", 2024-08-19T14:44:07.452 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:44:07.452 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.452 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.452 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:44:07.452 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.042187+0000", 2024-08-19T14:44:07.453 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:44:07.453 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.453 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.453 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:44:07.453 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.042189+0000", 2024-08-19T14:44:07.453 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4640000000000001e-06 2024-08-19T14:44:07.453 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.453 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.453 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:44:07.453 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.042194+0000", 2024-08-19T14:44:07.453 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0459999999999996e-06 2024-08-19T14:44:07.453 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.453 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.454 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:44:07.454 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.042205+0000", 2024-08-19T14:44:07.454 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0665e-05 2024-08-19T14:44:07.454 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.454 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.454 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:44:07.454 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.042211+0000", 2024-08-19T14:44:07.454 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4999999999999996e-06 2024-08-19T14:44:07.454 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.454 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.454 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:44:07.454 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.042261+0000", 2024-08-19T14:44:07.455 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9631e-05 2024-08-19T14:44:07.455 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.455 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.455 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:44:07.455 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.042357+0000", 2024-08-19T14:44:07.455 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.6417999999999997e-05 2024-08-19T14:44:07.455 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.455 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.455 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 9,10", 2024-08-19T14:44:07.455 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.042438+0000", 2024-08-19T14:44:07.455 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1037000000000004e-05 2024-08-19T14:44:07.455 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.455 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.455 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:44:07.456 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.049222+0000", 2024-08-19T14:44:07.456 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0067837069999999999 2024-08-19T14:44:07.456 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.456 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.456 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:44:07.456 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.065197+0000", 2024-08-19T14:44:07.456 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.015974743999999999 2024-08-19T14:44:07.456 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.456 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.456 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:44:07.456 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.179820+0000", 2024-08-19T14:44:07.456 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.114623725 2024-08-19T14:44:07.456 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.456 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.457 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:44:07.457 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.179879+0000", 2024-08-19T14:44:07.457 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8709000000000003e-05 2024-08-19T14:44:07.457 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.457 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.457 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:44:07.457 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.180239+0000", 2024-08-19T14:44:07.457 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00035984999999999999 2024-08-19T14:44:07.457 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:44:07.457 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:44:07.457 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:44:07.457 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.457 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.457 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26911.0:213837 8.11 8:8fa89269:::benchmark_data_smithi045_137103_object101373:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e806)", 2024-08-19T14:44:07.458 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:43:41.045341+0000", 2024-08-19T14:44:07.458 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 25.35502292, 2024-08-19T14:44:07.458 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.13516994500000001, 2024-08-19T14:44:07.458 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:44:07.458 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:44:07.458 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:44:07.458 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:44:07.458 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26911", 2024-08-19T14:44:07.458 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3494993995", 2024-08-19T14:44:07.458 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 213837 2024-08-19T14:44:07.458 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.458 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:44:07.458 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.458 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:44:07.459 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.045341+0000", 2024-08-19T14:44:07.459 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:44:07.459 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.459 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.459 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:44:07.459 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.045341+0000", 2024-08-19T14:44:07.459 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:44:07.459 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.459 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.460 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:44:07.460 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.045342+0000", 2024-08-19T14:44:07.460 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.5000000000000001e-07 2024-08-19T14:44:07.461 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.461 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.461 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:44:07.461 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.045348+0000", 2024-08-19T14:44:07.461 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8359999999999997e-06 2024-08-19T14:44:07.461 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.461 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.461 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:44:07.461 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.045362+0000", 2024-08-19T14:44:07.461 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4302e-05 2024-08-19T14:44:07.461 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.461 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.461 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:44:07.462 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.045370+0000", 2024-08-19T14:44:07.462 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.841e-06 2024-08-19T14:44:07.462 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.462 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.462 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:44:07.462 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.045725+0000", 2024-08-19T14:44:07.462 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00035418899999999998 2024-08-19T14:44:07.462 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.462 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.462 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:44:07.462 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.045857+0000", 2024-08-19T14:44:07.462 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013295700000000001 2024-08-19T14:44:07.463 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.463 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.463 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 9,10", 2024-08-19T14:44:07.463 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.046001+0000", 2024-08-19T14:44:07.463 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000143967 2024-08-19T14:44:07.463 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.463 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.463 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:44:07.463 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.049443+0000", 2024-08-19T14:44:07.463 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00344189 2024-08-19T14:44:07.463 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.463 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.463 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:44:07.463 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.179737+0000", 2024-08-19T14:44:07.464 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.13029397300000001 2024-08-19T14:44:07.464 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.464 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.464 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:44:07.464 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.180284+0000", 2024-08-19T14:44:07.464 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00054647099999999998 2024-08-19T14:44:07.464 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.464 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.464 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:44:07.464 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.180334+0000", 2024-08-19T14:44:07.464 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9863e-05 2024-08-19T14:44:07.464 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.464 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.464 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:44:07.465 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.180511+0000", 2024-08-19T14:44:07.465 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000177706 2024-08-19T14:44:07.465 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:44:07.465 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:44:07.465 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:44:07.465 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.465 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.465 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26911.0:213847 8.8 8:10ba7478:::benchmark_data_smithi045_137103_object101383:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e806)", 2024-08-19T14:44:07.465 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:43:41.052505+0000", 2024-08-19T14:44:07.465 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 25.347859824, 2024-08-19T14:44:07.465 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.127902971, 2024-08-19T14:44:07.465 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:44:07.465 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:44:07.465 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:44:07.466 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:44:07.466 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26911", 2024-08-19T14:44:07.466 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3494993995", 2024-08-19T14:44:07.466 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 213847 2024-08-19T14:44:07.466 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.466 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:44:07.466 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.466 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:44:07.466 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.052505+0000", 2024-08-19T14:44:07.466 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:44:07.466 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.466 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.466 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:44:07.466 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.052505+0000", 2024-08-19T14:44:07.467 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:44:07.467 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.467 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.467 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:44:07.467 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.052506+0000", 2024-08-19T14:44:07.467 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.215e-06 2024-08-19T14:44:07.467 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.467 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.467 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:44:07.467 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.052510+0000", 2024-08-19T14:44:07.467 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0369999999999996e-06 2024-08-19T14:44:07.467 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.467 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.468 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:44:07.468 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.052520+0000", 2024-08-19T14:44:07.468 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.8099999999999992e-06 2024-08-19T14:44:07.468 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.468 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.468 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:44:07.468 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.052525+0000", 2024-08-19T14:44:07.468 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3990000000000003e-06 2024-08-19T14:44:07.468 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.468 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.468 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:44:07.468 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.052577+0000", 2024-08-19T14:44:07.468 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2061999999999997e-05 2024-08-19T14:44:07.468 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.469 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.469 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:44:07.469 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.052706+0000", 2024-08-19T14:44:07.469 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012864300000000001 2024-08-19T14:44:07.469 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.469 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.469 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 9,10", 2024-08-19T14:44:07.469 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.052823+0000", 2024-08-19T14:44:07.469 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000117704 2024-08-19T14:44:07.469 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.469 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.469 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:44:07.469 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.058510+0000", 2024-08-19T14:44:07.470 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0056866570000000003 2024-08-19T14:44:07.470 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.470 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.470 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:44:07.470 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.180008+0000", 2024-08-19T14:44:07.470 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.12149813800000001 2024-08-19T14:44:07.470 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.470 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.470 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:44:07.470 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.180055+0000", 2024-08-19T14:44:07.470 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6425999999999999e-05 2024-08-19T14:44:07.470 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.470 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.470 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:44:07.471 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.180121+0000", 2024-08-19T14:44:07.471 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.648e-05 2024-08-19T14:44:07.471 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.471 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.471 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:44:07.471 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.180407+0000", 2024-08-19T14:44:07.471 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00028640000000000002 2024-08-19T14:44:07.471 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:44:07.471 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:44:07.471 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:44:07.471 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.471 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.471 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26911.0:213849 8.8 8:1287d6a6:::benchmark_data_smithi045_137103_object101385:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e806)", 2024-08-19T14:44:07.471 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:43:41.054782+0000", 2024-08-19T14:44:07.472 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 25.345582279999999, 2024-08-19T14:44:07.472 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.125901599, 2024-08-19T14:44:07.472 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:44:07.472 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:44:07.472 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:44:07.472 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:44:07.472 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26911", 2024-08-19T14:44:07.472 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3494993995", 2024-08-19T14:44:07.472 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 213849 2024-08-19T14:44:07.472 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.472 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:44:07.472 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.473 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:44:07.473 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.054782+0000", 2024-08-19T14:44:07.473 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:44:07.473 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.473 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.473 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:44:07.473 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.054782+0000", 2024-08-19T14:44:07.473 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:44:07.473 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.473 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.473 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:44:07.473 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.054783+0000", 2024-08-19T14:44:07.474 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.7599999999999996e-07 2024-08-19T14:44:07.474 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.474 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.474 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:44:07.474 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.054795+0000", 2024-08-19T14:44:07.474 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1892000000000001e-05 2024-08-19T14:44:07.474 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.474 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.474 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:44:07.474 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.054805+0000", 2024-08-19T14:44:07.474 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.7450000000000007e-06 2024-08-19T14:44:07.475 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.475 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.475 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:44:07.475 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.054809+0000", 2024-08-19T14:44:07.475 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9200000000000003e-06 2024-08-19T14:44:07.475 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.475 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.475 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:44:07.475 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.055795+0000", 2024-08-19T14:44:07.475 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00098545899999999994 2024-08-19T14:44:07.475 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.475 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.475 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:44:07.475 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.055921+0000", 2024-08-19T14:44:07.476 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012625399999999999 2024-08-19T14:44:07.476 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.476 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.476 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 9,10", 2024-08-19T14:44:07.476 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.056010+0000", 2024-08-19T14:44:07.476 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.8508e-05 2024-08-19T14:44:07.476 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.476 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.476 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:44:07.476 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.058600+0000", 2024-08-19T14:44:07.476 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0025900419999999999 2024-08-19T14:44:07.476 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.476 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.477 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:44:07.477 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.180449+0000", 2024-08-19T14:44:07.477 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.12184884 2024-08-19T14:44:07.477 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.477 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.477 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:44:07.477 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.180487+0000", 2024-08-19T14:44:07.477 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8046000000000001e-05 2024-08-19T14:44:07.477 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.477 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.477 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:44:07.477 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.180533+0000", 2024-08-19T14:44:07.477 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6626999999999999e-05 2024-08-19T14:44:07.477 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.478 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.478 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:44:07.478 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.180684+0000", 2024-08-19T14:44:07.478 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00015039 2024-08-19T14:44:07.478 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:44:07.478 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:44:07.478 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:44:07.478 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.478 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.478 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26911.0:213850 8.4 8:214dc11c:::benchmark_data_smithi045_137103_object101386:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e806)", 2024-08-19T14:44:07.478 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:43:41.057187+0000", 2024-08-19T14:44:07.478 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 25.343177159, 2024-08-19T14:44:07.478 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.123130881, 2024-08-19T14:44:07.479 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:44:07.479 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:44:07.479 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:44:07.479 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:44:07.479 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26911", 2024-08-19T14:44:07.479 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3494993995", 2024-08-19T14:44:07.479 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 213850 2024-08-19T14:44:07.479 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.479 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:44:07.479 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.479 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:44:07.479 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.057187+0000", 2024-08-19T14:44:07.479 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:44:07.480 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.480 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.480 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:44:07.480 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.057187+0000", 2024-08-19T14:44:07.480 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:44:07.480 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.480 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.480 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:44:07.480 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.057188+0000", 2024-08-19T14:44:07.480 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.7000000000000003e-07 2024-08-19T14:44:07.480 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.480 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.480 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:44:07.480 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.057193+0000", 2024-08-19T14:44:07.481 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0309999999999998e-06 2024-08-19T14:44:07.481 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.481 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.481 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:44:07.481 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.057202+0000", 2024-08-19T14:44:07.481 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.568e-06 2024-08-19T14:44:07.481 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.481 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.481 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:44:07.481 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.057207+0000", 2024-08-19T14:44:07.481 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7639999999999996e-06 2024-08-19T14:44:07.481 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.481 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.481 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:44:07.482 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.057247+0000", 2024-08-19T14:44:07.482 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0741000000000002e-05 2024-08-19T14:44:07.482 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.482 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.482 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:44:07.482 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.057329+0000", 2024-08-19T14:44:07.482 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1919e-05 2024-08-19T14:44:07.482 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.482 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.483 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 9,10", 2024-08-19T14:44:07.483 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.057401+0000", 2024-08-19T14:44:07.483 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1532999999999994e-05 2024-08-19T14:44:07.483 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.483 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.483 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:44:07.483 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.059863+0000", 2024-08-19T14:44:07.483 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0024627809999999998 2024-08-19T14:44:07.483 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.483 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.483 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:44:07.483 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.071672+0000", 2024-08-19T14:44:07.483 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011808213999999999 2024-08-19T14:44:07.483 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.484 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.484 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:44:07.484 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.180118+0000", 2024-08-19T14:44:07.484 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.108446231 2024-08-19T14:44:07.484 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.484 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.484 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:44:07.484 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.180172+0000", 2024-08-19T14:44:07.484 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3862e-05 2024-08-19T14:44:07.484 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.484 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.485 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:44:07.485 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.180318+0000", 2024-08-19T14:44:07.485 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000146267 2024-08-19T14:44:07.485 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:44:07.485 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:44:07.485 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:44:07.485 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.485 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.485 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26911.0:213854 8.11 8:8d4019d9:::benchmark_data_smithi045_137103_object101390:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e806)", 2024-08-19T14:44:07.485 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:43:41.068889+0000", 2024-08-19T14:44:07.485 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 25.331475786999999, 2024-08-19T14:44:07.485 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.112400208, 2024-08-19T14:44:07.486 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:44:07.486 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:44:07.486 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:44:07.486 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:44:07.486 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26911", 2024-08-19T14:44:07.486 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3494993995", 2024-08-19T14:44:07.486 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 213854 2024-08-19T14:44:07.486 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.486 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:44:07.486 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.486 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:44:07.487 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.068889+0000", 2024-08-19T14:44:07.487 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:44:07.487 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.487 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.487 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:44:07.487 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.068889+0000", 2024-08-19T14:44:07.487 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:44:07.487 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.487 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.487 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:44:07.487 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.068889+0000", 2024-08-19T14:44:07.487 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2799999999999996e-07 2024-08-19T14:44:07.487 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.488 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.488 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:44:07.488 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.068893+0000", 2024-08-19T14:44:07.488 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.41e-06 2024-08-19T14:44:07.488 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.488 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.488 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:44:07.488 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.068901+0000", 2024-08-19T14:44:07.488 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.2039999999999997e-06 2024-08-19T14:44:07.488 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.488 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.488 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:44:07.488 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.068905+0000", 2024-08-19T14:44:07.488 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6149999999999999e-06 2024-08-19T14:44:07.489 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.489 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.489 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:44:07.489 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.068956+0000", 2024-08-19T14:44:07.489 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0314e-05 2024-08-19T14:44:07.489 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.489 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.489 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:44:07.489 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.069062+0000", 2024-08-19T14:44:07.489 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00010625599999999999 2024-08-19T14:44:07.489 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.489 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.489 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 9,10", 2024-08-19T14:44:07.490 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.069133+0000", 2024-08-19T14:44:07.490 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1508999999999996e-05 2024-08-19T14:44:07.490 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.490 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.490 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:44:07.490 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.071605+0000", 2024-08-19T14:44:07.490 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0024713930000000001 2024-08-19T14:44:07.490 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.490 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.490 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:44:07.491 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.179778+0000", 2024-08-19T14:44:07.491 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.10817286199999999 2024-08-19T14:44:07.491 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.491 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.491 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:44:07.491 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.181030+0000", 2024-08-19T14:44:07.491 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0012519009999999999 2024-08-19T14:44:07.491 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.491 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.491 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:44:07.491 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.181081+0000", 2024-08-19T14:44:07.492 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1615e-05 2024-08-19T14:44:07.492 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.492 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.492 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:44:07.492 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.181289+0000", 2024-08-19T14:44:07.492 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00020750099999999999 2024-08-19T14:44:07.492 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:44:07.492 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:44:07.492 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:44:07.492 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.492 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.492 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop_reply(client.26911.0:213837 8.11 e806/797 ondisk, result = 0)", 2024-08-19T14:44:07.492 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:43:41.075721+0000", 2024-08-19T14:44:07.493 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 25.324643597000001, 2024-08-19T14:44:07.493 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.104024326, 2024-08-19T14:44:07.493 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:44:07.493 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:44:07.493 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-08-19T14:44:07.493 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:44:07.493 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.493 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:44:07.493 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.075721+0000", 2024-08-19T14:44:07.493 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:44:07.493 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.493 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.493 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:44:07.494 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.075721+0000", 2024-08-19T14:44:07.494 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:44:07.494 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.494 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.494 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:44:07.494 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.075721+0000", 2024-08-19T14:44:07.494 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0900000000000002e-07 2024-08-19T14:44:07.494 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.494 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.494 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:44:07.494 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.075725+0000", 2024-08-19T14:44:07.494 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.4400000000000001e-06 2024-08-19T14:44:07.494 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.495 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.495 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:44:07.495 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.075732+0000", 2024-08-19T14:44:07.495 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5789999999999998e-06 2024-08-19T14:44:07.495 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.495 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.495 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:44:07.495 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.075736+0000", 2024-08-19T14:44:07.495 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2089999999999999e-06 2024-08-19T14:44:07.495 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.495 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.495 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:44:07.496 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.179723+0000", 2024-08-19T14:44:07.496 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.10398642700000001 2024-08-19T14:44:07.496 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.496 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.496 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:44:07.496 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.179731+0000", 2024-08-19T14:44:07.496 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.9519999999999994e-06 2024-08-19T14:44:07.496 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.496 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.496 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:44:07.496 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.179745+0000", 2024-08-19T14:44:07.496 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4209999999999999e-05 2024-08-19T14:44:07.496 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:44:07.497 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:44:07.497 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:44:07.497 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.497 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.497 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.26911.0:215066 8.18 e807/764 8:1b403523:::benchmark_data_smithi045_137103_object102602:head v 807'8984, mlcod=807'8981)", 2024-08-19T14:44:07.497 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:43:42.907507+0000", 2024-08-19T14:44:07.497 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 23.492856924000002, 2024-08-19T14:44:07.497 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.14726671099999999, 2024-08-19T14:44:07.497 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:44:07.498 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:44:07.498 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:44:07.498 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:44:07.498 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.498 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:44:07.498 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:42.907507+0000", 2024-08-19T14:44:07.498 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:44:07.498 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.498 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.498 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:44:07.498 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:42.907507+0000", 2024-08-19T14:44:07.498 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:44:07.498 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.498 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.499 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:44:07.499 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:42.907508+0000", 2024-08-19T14:44:07.499 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4600000000000004e-07 2024-08-19T14:44:07.499 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.499 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.499 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:44:07.499 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:42.907514+0000", 2024-08-19T14:44:07.499 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7189999999999998e-06 2024-08-19T14:44:07.499 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.499 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.499 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:44:07.499 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:42.907526+0000", 2024-08-19T14:44:07.499 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1684e-05 2024-08-19T14:44:07.499 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.500 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.500 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:44:07.500 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:42.907531+0000", 2024-08-19T14:44:07.500 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0649999999999998e-06 2024-08-19T14:44:07.500 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.500 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.500 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:44:07.500 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:42.907571+0000", 2024-08-19T14:44:07.500 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0030000000000001e-05 2024-08-19T14:44:07.500 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.500 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.500 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:44:07.500 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:42.907586+0000", 2024-08-19T14:44:07.500 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5095e-05 2024-08-19T14:44:07.501 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.501 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.501 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:44:07.501 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:43.054726+0000", 2024-08-19T14:44:07.501 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.14714057 2024-08-19T14:44:07.501 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.501 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.501 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:44:07.501 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:43.054774+0000", 2024-08-19T14:44:07.501 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7802000000000001e-05 2024-08-19T14:44:07.501 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:44:07.501 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:44:07.501 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:44:07.502 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.502 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.502 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDPGRecoveryDeleteReply(8.11 e807,797 8:8cd5f3a3:::benchmark_data_smithi045_137103_object16443:head,768'6138)", 2024-08-19T14:44:07.502 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:43:42.910142+0000", 2024-08-19T14:44:07.502 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 23.490221998999999, 2024-08-19T14:44:07.502 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.14472585199999999, 2024-08-19T14:44:07.502 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:44:07.502 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:44:07.502 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:44:07.502 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:44:07.502 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.502 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:44:07.502 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:42.910142+0000", 2024-08-19T14:44:07.502 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:44:07.503 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.503 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.503 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:44:07.503 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:42.910142+0000", 2024-08-19T14:44:07.503 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:44:07.503 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.503 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.503 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:44:07.503 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:42.910144+0000", 2024-08-19T14:44:07.503 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.271e-06 2024-08-19T14:44:07.503 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.503 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.503 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:44:07.504 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:42.910149+0000", 2024-08-19T14:44:07.504 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0089999999999998e-06 2024-08-19T14:44:07.504 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.504 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.504 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:44:07.504 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:42.910158+0000", 2024-08-19T14:44:07.504 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.4890000000000001e-06 2024-08-19T14:44:07.504 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.504 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.504 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:44:07.504 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:42.910164+0000", 2024-08-19T14:44:07.504 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0039999999999996e-06 2024-08-19T14:44:07.504 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.505 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.505 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:44:07.505 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:42.910216+0000", 2024-08-19T14:44:07.505 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2201999999999999e-05 2024-08-19T14:44:07.505 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.505 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.505 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:44:07.505 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:43.054868+0000", 2024-08-19T14:44:07.505 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.14465187700000001 2024-08-19T14:44:07.505 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:44:07.505 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:44:07.505 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:44:07.505 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.505 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.505 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.26911.0:215065 8.d e807/796 8:ba5bdacb:::benchmark_data_smithi045_137103_object102601:head v 807'13514, mlcod=807'13511)", 2024-08-19T14:44:07.506 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:43:42.914765+0000", 2024-08-19T14:44:07.506 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 23.485599622999999, 2024-08-19T14:44:07.506 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.14674675500000001, 2024-08-19T14:44:07.506 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:44:07.506 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:44:07.506 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:44:07.506 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:44:07.506 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.506 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:44:07.506 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:42.914765+0000", 2024-08-19T14:44:07.506 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:44:07.506 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.506 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.507 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:44:07.507 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:42.914765+0000", 2024-08-19T14:44:07.507 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:44:07.507 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.507 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.507 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:44:07.507 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:42.914765+0000", 2024-08-19T14:44:07.507 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1200000000000003e-07 2024-08-19T14:44:07.507 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.507 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.507 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:44:07.507 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:42.914770+0000", 2024-08-19T14:44:07.507 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0799999999999996e-06 2024-08-19T14:44:07.507 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.507 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.508 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:44:07.508 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:43.054536+0000", 2024-08-19T14:44:07.508 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.13976527599999999 2024-08-19T14:44:07.508 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.508 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.508 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:44:07.508 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:43.054547+0000", 2024-08-19T14:44:07.508 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1235e-05 2024-08-19T14:44:07.508 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.508 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.508 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:44:07.508 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:43.054605+0000", 2024-08-19T14:44:07.508 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7979000000000002e-05 2024-08-19T14:44:07.509 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.509 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.509 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:44:07.509 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:43.054630+0000", 2024-08-19T14:44:07.509 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5074000000000001e-05 2024-08-19T14:44:07.509 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.509 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.509 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:44:07.509 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:43.061471+0000", 2024-08-19T14:44:07.509 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0068414390000000004 2024-08-19T14:44:07.509 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.509 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.510 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:44:07.510 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:43.061511+0000", 2024-08-19T14:44:07.510 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0059999999999999e-05 2024-08-19T14:44:07.510 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:44:07.510 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:44:07.510 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:44:07.510 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.510 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.510 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDPGRecoveryDeleteReply(8.4 e807,798 8:212f1bd2:::benchmark_data_smithi045_137103_object29107:head,768'6423)", 2024-08-19T14:44:07.510 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:43:42.914964+0000", 2024-08-19T14:44:07.510 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 23.485400027000001, 2024-08-19T14:44:07.510 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.1402487, 2024-08-19T14:44:07.510 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:44:07.510 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:44:07.511 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:44:07.511 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:44:07.511 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.511 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:44:07.511 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:42.914964+0000", 2024-08-19T14:44:07.511 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:44:07.511 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.511 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.511 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:44:07.511 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:42.914964+0000", 2024-08-19T14:44:07.511 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:44:07.511 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.511 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.511 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:44:07.511 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:42.914965+0000", 2024-08-19T14:44:07.512 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4900000000000001e-07 2024-08-19T14:44:07.512 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.512 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.512 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:44:07.512 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:42.914968+0000", 2024-08-19T14:44:07.512 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5310000000000002e-06 2024-08-19T14:44:07.512 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.512 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.512 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:44:07.512 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:43.054533+0000", 2024-08-19T14:44:07.512 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.139564823 2024-08-19T14:44:07.512 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.512 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.512 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:44:07.513 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:43.054549+0000", 2024-08-19T14:44:07.513 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5739e-05 2024-08-19T14:44:07.513 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.513 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.513 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:44:07.513 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:43.054649+0000", 2024-08-19T14:44:07.513 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000100251 2024-08-19T14:44:07.513 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.513 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.513 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:44:07.513 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:43.055213+0000", 2024-08-19T14:44:07.513 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00056390699999999999 2024-08-19T14:44:07.513 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:44:07.513 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:44:07.514 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:44:07.514 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.514 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.514 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDPGRecoveryDeleteReply(8.11 e818,797 8:89c97240:::benchmark_data_smithi045_137103_object54448:head,774'7379)", 2024-08-19T14:44:07.514 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:44:01.776936+0000", 2024-08-19T14:44:07.514 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 4.6234278580000003, 2024-08-19T14:44:07.514 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.10370937, 2024-08-19T14:44:07.514 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:44:07.514 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:44:07.514 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:44:07.514 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:44:07.514 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.514 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:44:07.514 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:44:01.776936+0000", 2024-08-19T14:44:07.515 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:44:07.515 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.515 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.515 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:44:07.515 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:44:01.776936+0000", 2024-08-19T14:44:07.515 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:44:07.515 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.515 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.515 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:44:07.515 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:44:01.776938+0000", 2024-08-19T14:44:07.515 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.42e-06 2024-08-19T14:44:07.515 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.515 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.515 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:44:07.516 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:44:01.776948+0000", 2024-08-19T14:44:07.516 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0073e-05 2024-08-19T14:44:07.516 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.516 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.516 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:44:07.516 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:44:01.776972+0000", 2024-08-19T14:44:07.516 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3816e-05 2024-08-19T14:44:07.516 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.516 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.516 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:44:07.516 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:44:01.777000+0000", 2024-08-19T14:44:07.516 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7926999999999999e-05 2024-08-19T14:44:07.516 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.516 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.517 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:44:07.517 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:44:01.777160+0000", 2024-08-19T14:44:07.517 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000159977 2024-08-19T14:44:07.517 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.517 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.517 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:44:07.517 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:44:01.880646+0000", 2024-08-19T14:44:07.517 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.103486157 2024-08-19T14:44:07.517 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:44:07.517 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:44:07.517 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:44:07.517 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.517 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.518 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDPGRecoveryDeleteReply(8.8 e818,798 8:14964ac9:::benchmark_data_smithi045_137103_object49588:head,773'7316)", 2024-08-19T14:44:07.518 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:44:01.777028+0000", 2024-08-19T14:44:07.518 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 4.6233358999999998, 2024-08-19T14:44:07.518 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.10353796799999999, 2024-08-19T14:44:07.518 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:44:07.518 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:44:07.518 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:44:07.518 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:44:07.518 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.518 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:44:07.519 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:44:01.777028+0000", 2024-08-19T14:44:07.519 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:44:07.519 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.519 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.519 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:44:07.519 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:44:01.777028+0000", 2024-08-19T14:44:07.519 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:44:07.519 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.519 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.519 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:44:07.519 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:44:01.777029+0000", 2024-08-19T14:44:07.519 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7400000000000003e-07 2024-08-19T14:44:07.519 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.519 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.520 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:44:07.520 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:44:01.777035+0000", 2024-08-19T14:44:07.520 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5559999999999998e-06 2024-08-19T14:44:07.520 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.520 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.520 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:44:07.520 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:44:01.777047+0000", 2024-08-19T14:44:07.520 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2483999999999999e-05 2024-08-19T14:44:07.520 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.520 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.520 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:44:07.520 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:44:01.777056+0000", 2024-08-19T14:44:07.520 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6629999999999999e-06 2024-08-19T14:44:07.521 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.521 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.521 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:44:07.521 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:44:01.777167+0000", 2024-08-19T14:44:07.521 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00011089799999999999 2024-08-19T14:44:07.521 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:44:07.521 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:44:07.521 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:44:07.521 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:44:01.880566+0000", 2024-08-19T14:44:07.521 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.103399793 2024-08-19T14:44:07.521 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:44:07.521 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:44:07.521 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:44:07.521 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:44:07.522 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:44:07.522 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:44:07.522 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:44:07.547 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:07.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:07.549 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:44:07.649 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:44:07.675 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:07.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:07.676 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:44:07.777 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:44:07.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:07.803 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:07.803 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:44:07.904 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:44:07.930 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:07.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:07.931 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:44:08.032 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:44:08.057 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:08.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:08.058 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:44:08.159 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:44:08.184 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:08.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:08.185 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:44:08.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:08 smithi032 ceph-mon[20004]: pgmap v1949: 31 pgs: 31 active+clean; 655 KiB data, 7.3 GiB used, 708 GiB / 715 GiB avail 2024-08-19T14:44:08.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:08 smithi032 ceph-mon[20004]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1287/39 objects degraded (3300.000%), 8 pgs degraded) 2024-08-19T14:44:08.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:08 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:44:08.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:08 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:44:08.286 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:44:08.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:08 smithi045 ceph-mon[20143]: pgmap v1949: 31 pgs: 31 active+clean; 655 KiB data, 7.3 GiB used, 708 GiB / 715 GiB avail 2024-08-19T14:44:08.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:08 smithi045 ceph-mon[20143]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1287/39 objects degraded (3300.000%), 8 pgs degraded) 2024-08-19T14:44:08.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:08 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:44:08.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:08 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:44:08.312 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:08.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:08.313 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:44:08.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:08 smithi031 ceph-mon[16336]: pgmap v1949: 31 pgs: 31 active+clean; 655 KiB data, 7.3 GiB used, 708 GiB / 715 GiB avail 2024-08-19T14:44:08.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:08 smithi031 ceph-mon[16336]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1287/39 objects degraded (3300.000%), 8 pgs degraded) 2024-08-19T14:44:08.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:08 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:44:08.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:08 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:44:08.414 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:44:08.444 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:08.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:08.445 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:44:08.546 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:44:08.572 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:08.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:08.573 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:44:08.673 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:44:08.699 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:08.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:08.700 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:44:08.801 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:44:08.826 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:08.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:08.827 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:44:08.928 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:44:08.953 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:08.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:08.954 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:44:09.056 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:44:09.081 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:09.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:09.083 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:44:09.184 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:44:09.209 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:09.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:09.211 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:44:09.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:09 smithi045 ceph-mon[20143]: osdmap e824: 12 total, 12 up, 8 in 2024-08-19T14:44:09.312 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:44:09.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:09.338 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:09.338 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:44:09.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:09 smithi031 ceph-mon[16336]: osdmap e824: 12 total, 12 up, 8 in 2024-08-19T14:44:09.439 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:44:09.464 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:09.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:09.465 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:44:09.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:09 smithi032 ceph-mon[20004]: osdmap e824: 12 total, 12 up, 8 in 2024-08-19T14:44:09.566 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:44:09.592 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:09.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:09.593 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:44:09.653 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd reweight 9 1 2024-08-19T14:44:09.667 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:44:09.668 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:44:09.683 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:44:09.694 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:44:09.720 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:09.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:09.722 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:44:09.823 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:44:09.849 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:09.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:09.850 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:44:09.952 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:44:09.978 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:09.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:09.979 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:44:10.080 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:44:10.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:10.107 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:10.107 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:44:10.208 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:44:10.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:10.233 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:10.233 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:44:10.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:10 smithi045 ceph-mon[20143]: pgmap v1951: 31 pgs: 31 active+clean; 655 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail 2024-08-19T14:44:10.334 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:44:10.360 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:10.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:10.361 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:44:10.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:10 smithi031 ceph-mon[16336]: pgmap v1951: 31 pgs: 31 active+clean; 655 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail 2024-08-19T14:44:10.462 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:44:10.487 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:10.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:10.488 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:44:10.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:10 smithi032 ceph-mon[20004]: pgmap v1951: 31 pgs: 31 active+clean; 655 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail 2024-08-19T14:44:10.589 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:44:10.614 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:10.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:10.615 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:44:10.716 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:44:10.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:10.743 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:10.743 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:44:10.844 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:44:10.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:10.871 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:10.871 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:44:10.972 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:44:10.999 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:11.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:11.000 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:44:11.101 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:44:11.127 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:11.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:11.128 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:44:11.229 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:44:11.255 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:11.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:11.256 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:44:11.357 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:44:11.383 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:11.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:11.384 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:44:11.485 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:44:11.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:11.514 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:11.514 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:44:11.615 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:44:11.640 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:11.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:11.641 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:44:11.742 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:44:11.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:11.770 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:11.770 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:44:11.871 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:44:11.897 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:11.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:11.898 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:44:11.999 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:44:12.024 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:12.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:12.025 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:44:12.126 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:44:12.151 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:12.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:12.152 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:44:12.253 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:44:12.279 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:12.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:12.280 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:44:12.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:12 smithi045 ceph-mon[20143]: pgmap v1952: 31 pgs: 31 active+clean; 655 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail 2024-08-19T14:44:12.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:12 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:44:12.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:12 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:44:12.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:12 smithi031 ceph-mon[16336]: pgmap v1952: 31 pgs: 31 active+clean; 655 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail 2024-08-19T14:44:12.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:12 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:44:12.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:12 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:44:12.381 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:44:12.407 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:12.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:12.408 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:44:12.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:12 smithi032 ceph-mon[20004]: pgmap v1952: 31 pgs: 31 active+clean; 655 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail 2024-08-19T14:44:12.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:12 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:44:12.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:12 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:44:12.509 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:44:12.534 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:12.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:12.535 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:44:12.636 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:44:12.662 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:12.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:12.663 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:44:12.764 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:44:12.817 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:12.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:12.818 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:44:12.919 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:44:12.941 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:12.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:12.942 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:44:13.043 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:44:13.064 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:13.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:13.065 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:44:13.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:13 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:44:13.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:13 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:44:13.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:13 smithi031 ceph-mon[16336]: osdmap e825: 12 total, 12 up, 8 in 2024-08-19T14:44:13.167 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:44:13.189 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:13.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:13.190 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:44:13.291 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:44:13.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:13 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:44:13.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:13 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:44:13.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:13 smithi045 ceph-mon[20143]: osdmap e825: 12 total, 12 up, 8 in 2024-08-19T14:44:13.317 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:13.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:13.318 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:44:13.419 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:44:13.445 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:13.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:13.446 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:44:13.510 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:13 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:44:13.510 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:13 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:44:13.511 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:13 smithi032 ceph-mon[20004]: osdmap e825: 12 total, 12 up, 8 in 2024-08-19T14:44:13.547 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:44:13.569 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:13.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:13.571 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:44:13.666 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:44:13.666 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:44:13.667 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:44:13.667 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:44:13.671 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:44:13.672 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:44:13.686 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:44:13.686 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:44:13.697 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:13.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:13.698 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:44:13.703 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:44:13.704 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:44:13.721 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:44:13.721 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:44:13.739 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:44:13.740 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:44:13.757 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:44:13.757 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:44:13.773 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:44:13.774 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:44:13.791 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:44:13.791 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:44:13.799 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:44:13.808 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:44:13.809 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:44:13.824 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:13.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:13.826 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:44:13.826 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:44:13.826 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:44:13.844 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:44:13.844 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:44:13.926 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:44:13.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:13.948 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:13.949 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:44:14.048 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'unique_pool_7' created 2024-08-19T14:44:14.049 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is set 2024-08-19T14:44:14.049 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:44:14.064 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:14 smithi031 ceph-mon[16336]: pgmap v1954: 31 pgs: 31 active+clean; 655 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail 2024-08-19T14:44:14.064 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:14 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1369545097' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:44:14.064 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:14 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/4271462601' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]: dispatch 2024-08-19T14:44:14.064 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:14 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2569815313' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:44:14.074 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:14.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:14.075 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:44:14.176 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:44:14.199 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:14.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:14.200 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:44:14.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:14 smithi045 ceph-mon[20143]: pgmap v1954: 31 pgs: 31 active+clean; 655 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail 2024-08-19T14:44:14.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:14 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1369545097' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:44:14.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:14 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/4271462601' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]: dispatch 2024-08-19T14:44:14.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:14 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2569815313' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:44:14.298 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:14 smithi032 ceph-mon[20004]: pgmap v1954: 31 pgs: 31 active+clean; 655 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail 2024-08-19T14:44:14.299 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:14 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1369545097' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:44:14.299 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:14 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/4271462601' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]: dispatch 2024-08-19T14:44:14.299 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:14 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2569815313' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:44:14.300 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:44:14.326 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:14.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:14.327 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:44:14.332 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-08-19T14:44:14.428 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:44:14.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:14.451 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:14.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:44:14.552 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:44:14.574 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:14.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:14.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:44:14.590 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:44:14.676 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:44:14.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:14.698 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:14.698 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:44:14.799 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:44:14.824 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:14.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:14.825 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:44:14.926 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:44:14.952 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:14.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:14.953 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:44:15.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-08-19T14:44:15.044+0000 7f1f51923640 0 --2- 172.21.15.31:0/2080012105 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f4c100b50 0x7f1f4c198670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-08-19T14:44:15.054 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:44:15.083 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:15.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:15.084 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:44:15.185 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:44:15.206 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:15.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:15.207 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:44:15.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:15 smithi045 ceph-mon[20143]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:44:15.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:15 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1369545097' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:44:15.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:15 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/4271462601' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]': finished 2024-08-19T14:44:15.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:15 smithi045 ceph-mon[20143]: osdmap e826: 12 total, 12 up, 8 in 2024-08-19T14:44:15.308 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:44:15.334 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:15.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:15.335 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:44:15.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:15 smithi031 ceph-mon[16336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:44:15.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:15 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1369545097' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:44:15.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:15 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/4271462601' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]': finished 2024-08-19T14:44:15.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:15 smithi031 ceph-mon[16336]: osdmap e826: 12 total, 12 up, 8 in 2024-08-19T14:44:15.435 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:44:15.461 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:15.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:15.463 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:44:15.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:15 smithi032 ceph-mon[20004]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:44:15.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:15 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1369545097' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:44:15.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:15 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/4271462601' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]': finished 2024-08-19T14:44:15.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:15 smithi032 ceph-mon[20004]: osdmap e826: 12 total, 12 up, 8 in 2024-08-19T14:44:15.565 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:44:15.591 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:15.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:15.592 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:44:15.693 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd pool application enable unique_pool_7 rados --yes-i-really-mean-it || true 2024-08-19T14:44:15.693 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:44:15.719 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:15.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:15.721 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:44:15.822 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:44:15.846 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:15.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:15.847 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:44:15.948 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:44:15.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:15.976 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:15.976 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:44:16.058 INFO:teuthology.orchestra.run.smithi031.stderr:reweighted osd.9 to 1 (10000) 2024-08-19T14:44:16.077 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:44:16.098 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:16.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:16.100 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:44:16.201 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:44:16.227 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:16.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:16.228 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:44:16.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:16 smithi032 ceph-mon[20004]: pgmap v1956: 47 pgs: 16 unknown, 31 active+clean; 655 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail 2024-08-19T14:44:16.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:16 smithi032 ceph-mon[20004]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-19T14:44:16.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:16 smithi032 ceph-mon[20004]: osdmap e827: 12 total, 12 up, 8 in 2024-08-19T14:44:16.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:16 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2080012105' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-08-19T14:44:16.328 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:44:16.354 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:16.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:16.355 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:44:16.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:16 smithi031 ceph-mon[16336]: pgmap v1956: 47 pgs: 16 unknown, 31 active+clean; 655 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail 2024-08-19T14:44:16.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:16 smithi031 ceph-mon[16336]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-19T14:44:16.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:16 smithi031 ceph-mon[16336]: osdmap e827: 12 total, 12 up, 8 in 2024-08-19T14:44:16.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:16 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2080012105' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-08-19T14:44:16.456 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:44:16.482 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:16.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:16.484 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:44:16.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:16 smithi045 ceph-mon[20143]: pgmap v1956: 47 pgs: 16 unknown, 31 active+clean; 655 KiB data, 7.2 GiB used, 708 GiB / 715 GiB avail 2024-08-19T14:44:16.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:16 smithi045 ceph-mon[20143]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-19T14:44:16.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:16 smithi045 ceph-mon[20143]: osdmap e827: 12 total, 12 up, 8 in 2024-08-19T14:44:16.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:16 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2080012105' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-08-19T14:44:16.585 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:44:16.610 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:16.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:16.611 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:44:16.669 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:44:16.712 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:44:16.738 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:16.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:16.739 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:44:16.840 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:44:16.856 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd reweight 10 1 2024-08-19T14:44:16.866 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:16.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:16.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:44:16.928 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:44:16.968 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:44:16.989 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:16.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:16.990 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:44:17.079 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:17 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2080012105' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-08-19T14:44:17.079 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:17 smithi031 ceph-mon[16336]: osdmap e828: 12 total, 12 up, 8 in 2024-08-19T14:44:17.091 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:44:17.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:17.115 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:17.115 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:44:17.216 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:44:17.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:17.240 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:17.240 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:44:17.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:17 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2080012105' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-08-19T14:44:17.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:17 smithi032 ceph-mon[20004]: osdmap e828: 12 total, 12 up, 8 in 2024-08-19T14:44:17.341 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:44:17.367 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:17.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:17.368 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:44:17.445 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:44:17.460 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:44:17.460 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:44:17.469 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:44:17.486 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:44:17.486 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:44:17.495 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:17.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:17.496 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:44:17.506 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:44:17.507 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:44:17.523 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:44:17.523 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:44:17.540 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:44:17.540 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:44:17.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:17 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2080012105' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-08-19T14:44:17.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:17 smithi045 ceph-mon[20143]: osdmap e828: 12 total, 12 up, 8 in 2024-08-19T14:44:17.556 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:44:17.557 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:44:17.574 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:44:17.574 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:44:17.591 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:44:17.591 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:44:17.596 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:44:17.608 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:44:17.609 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:44:17.619 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:17.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:17.621 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:44:17.723 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:44:17.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:17.750 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:17.750 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:44:17.827 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:44:17.827 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:44:17.844 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:44:17.844 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:44:17.851 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:44:17.861 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:44:17.861 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:44:17.875 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:17.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:17.876 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:44:17.977 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:44:18.003 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:18.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:18.004 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:44:18.105 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:44:18.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:18 smithi031 ceph-mon[16336]: pgmap v1959: 47 pgs: 2 creating+activating, 1 creating+peering, 8 unknown, 36 active+clean; 655 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail 2024-08-19T14:44:18.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:18 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1383240945' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:44:18.131 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:18.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:18.133 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:44:18.194 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-19T14:44:18.234 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:44:18.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:18.259 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:18.259 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:44:18.360 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:44:18.386 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:18.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:18.387 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:44:18.488 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:44:18.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:18 smithi032 ceph-mon[20004]: pgmap v1959: 47 pgs: 2 creating+activating, 1 creating+peering, 8 unknown, 36 active+clean; 655 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail 2024-08-19T14:44:18.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:18 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1383240945' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:44:18.510 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:18.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:18.511 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:44:18.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:18 smithi045 ceph-mon[20143]: pgmap v1959: 47 pgs: 2 creating+activating, 1 creating+peering, 8 unknown, 36 active+clean; 655 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail 2024-08-19T14:44:18.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:18 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1383240945' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:44:18.612 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:44:18.634 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:18.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:18.635 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:44:18.662 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:44:18.662 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:44:18.736 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:44:18.757 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:18.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:18.758 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:44:18.859 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:44:18.881 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:18.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:18.882 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:44:18.890 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:44:18.890 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-08-19T14:44:18.891 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-08-19T14:44:18.891 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:44:18.983 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:44:19.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:19.006 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:19.006 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:44:19.070 INFO:teuthology.orchestra.run.smithi031.stderr:enabled application 'rados' on pool 'unique_pool_7' 2024-08-19T14:44:19.107 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:44:19.128 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:19.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:19.129 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:44:19.231 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:44:19.257 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:19.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:19.258 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:44:19.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:19 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3310799048' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-08-19T14:44:19.359 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:44:19.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:19 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3310799048' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-08-19T14:44:19.385 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:19.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:19.386 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:44:19.487 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:44:19.512 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:19.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:19.514 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:44:19.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:19 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3310799048' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-08-19T14:44:19.615 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:44:19.640 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:19.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:19.641 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:44:19.742 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:44:19.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:19.767 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:19.768 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:44:19.868 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:44:19.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:19.895 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:19.895 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:44:19.996 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:44:20.018 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:20.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:20.019 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:44:20.071 INFO:teuthology.orchestra.run.smithi031.stderr:reweighted osd.10 to 1 (10000) 2024-08-19T14:44:20.120 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:44:20.141 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:20.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:20.142 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:44:20.243 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:44:20.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:20 smithi032 ceph-mon[20004]: pgmap v1960: 47 pgs: 2 creating+activating, 1 creating+peering, 6 unknown, 38 active+clean; 655 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail 2024-08-19T14:44:20.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:20 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3310799048' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-08-19T14:44:20.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:20 smithi032 ceph-mon[20004]: osdmap e829: 12 total, 12 up, 8 in 2024-08-19T14:44:20.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:20 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3619491550' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-08-19T14:44:20.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:20 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3727517106' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:44:20.264 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:20.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:20.265 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:44:20.366 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:44:20.417 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:20 smithi031 ceph-mon[16336]: pgmap v1960: 47 pgs: 2 creating+activating, 1 creating+peering, 6 unknown, 38 active+clean; 655 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail 2024-08-19T14:44:20.417 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:20 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3310799048' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-08-19T14:44:20.417 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:20 smithi031 ceph-mon[16336]: osdmap e829: 12 total, 12 up, 8 in 2024-08-19T14:44:20.417 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:20 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3619491550' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-08-19T14:44:20.417 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:20 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3727517106' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:44:20.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:20.440 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:20.440 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:44:20.502 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:44:20.541 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:44:20.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:20 smithi045 ceph-mon[20143]: pgmap v1960: 47 pgs: 2 creating+activating, 1 creating+peering, 6 unknown, 38 active+clean; 655 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail 2024-08-19T14:44:20.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:20 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3310799048' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-08-19T14:44:20.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:20 smithi045 ceph-mon[20143]: osdmap e829: 12 total, 12 up, 8 in 2024-08-19T14:44:20.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:20 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3619491550' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-08-19T14:44:20.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:20 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3727517106' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:44:20.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:20.568 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:20.568 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:44:20.670 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:44:20.692 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:20.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:20.693 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:44:20.794 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:44:20.815 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:20.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:20.817 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:44:20.918 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:44:20.930 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-08-19T14:44:20.930 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-19T14:44:20.943 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:20.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:20.944 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:44:21.045 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:44:21.053 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is unset 2024-08-19T14:44:21.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:21.067 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:21.067 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:44:21.168 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:44:21.188 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:21.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:21.190 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:44:21.253 DEBUG:teuthology.orchestra.run.smithi045:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_7 bench 90 write' 2024-08-19T14:44:21.256 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-08-19T14:44:21.256 INFO:teuthology.orchestra.run:waiting for 3000 2024-08-19T14:44:21.290 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:44:21.307 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:hints = 1 2024-08-19T14:44:21.307 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-08-19T14:44:21.307 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Object prefix: benchmark_data_smithi045_152862 2024-08-19T14:44:21.307 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-19T14:44:21.308 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 0 0 0 0 0 0 - 0 2024-08-19T14:44:21.313 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:21.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:21.314 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:44:21.357 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:21 smithi031 ceph-mon[16336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:44:21.358 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:21 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3619491550' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-08-19T14:44:21.358 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:21 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3727517106' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:44:21.358 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:21 smithi031 ceph-mon[16336]: osdmap e830: 12 total, 12 up, 8 in 2024-08-19T14:44:21.415 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:44:21.442 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:21.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:21.443 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:44:21.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:21 smithi032 ceph-mon[20004]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:44:21.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:21 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3619491550' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-08-19T14:44:21.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:21 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3727517106' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:44:21.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:21 smithi032 ceph-mon[20004]: osdmap e830: 12 total, 12 up, 8 in 2024-08-19T14:44:21.543 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:44:21.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:21 smithi045 ceph-mon[20143]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:44:21.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:21 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3619491550' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-08-19T14:44:21.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:21 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3727517106' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:44:21.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:21 smithi045 ceph-mon[20143]: osdmap e830: 12 total, 12 up, 8 in 2024-08-19T14:44:21.570 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:21.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:21.571 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:44:21.672 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:44:21.697 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:21.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:21.698 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:44:21.728 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:44:21.799 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:44:21.820 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:21.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:21.821 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:44:21.922 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:44:21.947 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:21.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:21.948 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:44:22.049 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:44:22.070 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:22.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:22.071 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:44:22.173 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:44:22.210 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:22.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:22.211 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:44:22.219 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:44:22.219 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:44:22.224 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:44:22.249 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:44:22.249 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:44:22.249 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:22 smithi031 ceph-mon[16336]: pgmap v1963: 47 pgs: 47 active+clean; 655 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail 2024-08-19T14:44:22.250 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:22 smithi031 ceph-mon[16336]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-19T14:44:22.276 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:44:22.276 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:44:22.301 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:44:22.301 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:44:22.307 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 1 16 2403 2387 149.179 149.188 0.00215462 0.00668404 2024-08-19T14:44:22.312 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:44:22.324 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:44:22.324 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:44:22.337 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:22.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:22.338 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:44:22.350 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:44:22.351 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:44:22.377 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:44:22.377 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:44:22.403 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:44:22.404 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:44:22.425 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:44:22.425 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:44:22.439 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:44:22.450 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:44:22.451 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:44:22.470 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:22.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:22.472 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:44:22.493 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:44:22.494 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:44:22.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:22 smithi032 ceph-mon[20004]: pgmap v1963: 47 pgs: 47 active+clean; 655 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail 2024-08-19T14:44:22.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:22 smithi032 ceph-mon[20004]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-19T14:44:22.520 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:44:22.520 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:44:22.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:22 smithi045 ceph-mon[20143]: pgmap v1963: 47 pgs: 47 active+clean; 655 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail 2024-08-19T14:44:22.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:22 smithi045 ceph-mon[20143]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-19T14:44:22.572 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:44:22.600 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:22.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:22.601 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:44:22.702 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:44:22.728 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:22.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:22.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:44:22.830 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:44:22.868 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:22.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:22.869 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:44:22.875 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-08-19T14:44:22.969 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:44:22.990 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:22.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:22.992 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:44:23.093 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:44:23.116 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:23.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:23.117 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:44:23.153 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:44:23.218 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:44:23.245 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:23.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:23.246 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:44:23.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:23 smithi032 ceph-mon[20004]: 1.6 scrub starts 2024-08-19T14:44:23.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:23 smithi032 ceph-mon[20004]: 1.6 scrub ok 2024-08-19T14:44:23.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:23 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2256110047' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:44:23.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:23 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:44:23.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:23 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:44:23.300 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:44:23.301 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-08-19T14:44:23.308 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 2 16 5134 5118 159.925 170.688 0.00410921 0.00624213 2024-08-19T14:44:23.347 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:44:23.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:23 smithi031 ceph-mon[16336]: 1.6 scrub starts 2024-08-19T14:44:23.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:23 smithi031 ceph-mon[16336]: 1.6 scrub ok 2024-08-19T14:44:23.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:23 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2256110047' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:44:23.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:23 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:44:23.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:23 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:44:23.376 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:23.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:23.377 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:44:23.478 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:44:23.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:23.500 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:23.501 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:44:23.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:23 smithi045 ceph-mon[20143]: 1.6 scrub starts 2024-08-19T14:44:23.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:23 smithi045 ceph-mon[20143]: 1.6 scrub ok 2024-08-19T14:44:23.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:23 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2256110047' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:44:23.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:23 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:44:23.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:23 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:44:23.601 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:44:23.623 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:23.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:23.624 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:44:23.720 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-19T14:44:23.724 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:44:23.749 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:23.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:23.751 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:44:23.852 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:44:23.880 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:23.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:23.881 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:44:23.982 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:44:24.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:24.018 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:24.018 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:44:24.119 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:44:24.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:24.157 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:24.157 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:44:24.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:24 smithi032 ceph-mon[20004]: pgmap v1964: 47 pgs: 47 active+clean; 655 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail 2024-08-19T14:44:24.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:24 smithi032 ceph-mon[20004]: 1.19 scrub starts 2024-08-19T14:44:24.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:24 smithi032 ceph-mon[20004]: 1.19 scrub ok 2024-08-19T14:44:24.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:24 smithi032 ceph-mon[20004]: osdmap e831: 12 total, 12 up, 8 in 2024-08-19T14:44:24.258 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:44:24.287 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:24.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:24.288 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:44:24.308 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 3 16 8122 8106 168.862 186.75 0.00185173 0.00591433 2024-08-19T14:44:24.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:24 smithi031 ceph-mon[16336]: pgmap v1964: 47 pgs: 47 active+clean; 655 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail 2024-08-19T14:44:24.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:24 smithi031 ceph-mon[16336]: 1.19 scrub starts 2024-08-19T14:44:24.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:24 smithi031 ceph-mon[16336]: 1.19 scrub ok 2024-08-19T14:44:24.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:24 smithi031 ceph-mon[16336]: osdmap e831: 12 total, 12 up, 8 in 2024-08-19T14:44:24.389 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:44:24.426 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:24.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:24.427 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:44:24.447 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:24 smithi045 ceph-mon[20143]: pgmap v1964: 47 pgs: 47 active+clean; 655 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail 2024-08-19T14:44:24.447 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:24 smithi045 ceph-mon[20143]: 1.19 scrub starts 2024-08-19T14:44:24.447 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:24 smithi045 ceph-mon[20143]: 1.19 scrub ok 2024-08-19T14:44:24.447 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:24 smithi045 ceph-mon[20143]: osdmap e831: 12 total, 12 up, 8 in 2024-08-19T14:44:24.528 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:44:24.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:24.551 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:24.551 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:44:24.589 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":1965,"stamp":"2024-08-19T14:44:23.083446+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":26688665,"num_objects":400,"num_object_clones":0,"num_object_copies":1200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":400,"num_whiteouts":0,"num_read":17,"num_read_kb":15,"num_write":422,"num_write_kb":25808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":841477,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4706,"ondisk_log_size":4706,"up":141,"acting":141,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":144,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":7486124,"kb_used_data":152012,"kb_used_omap":17,"kb_used_meta":7333998,"kb_avail":742475092,"statfs":{"total":767960285184,"available":760294494208,"internally_reserved":0,"allocated":155660288,"data_stored":136543207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18389,"internal_metadata":7510013995},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26017792,"num_objects":397,"num_object_clones":0,"num_object_copies":1191,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":397,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":397,"num_write_kb":25408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.734533"},"pg_stats":[{"pgid":"9.8","version":"830'210","reported_seq":228,"reported_epoch":830,"state":"active+clean","last_fresh":"2024-08-19T14:44:22.501473+0000","last_change":"2024-08-19T14:44:15.415491+0000","last_active":"2024-08-19T14:44:22.501473+0000","last_peered":"2024-08-19T14:44:22.501473+0000","last_clean":"2024-08-19T14:44:22.501473+0000","last_became_active":"2024-08-19T14:44:15.414629+0000","last_became_peered":"2024-08-19T14:44:15.414629+0000","last_unstale":"2024-08-19T14:44:22.501473+0000","last_undegraded":"2024-08-19T14:44:22.501473+0000","last_fullsized":"2024-08-19T14:44:22.501473+0000","mapping_epoch":826,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":827,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":210,"log_dups_size":0,"ondisk_log_size":210,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:23.965899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13762560,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":210,"num_write_kb":13440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,5],"acting":[11,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"70'139","reported_seq":1988,"reported_epoch":830,"state":"active+clean","last_fresh":"2024-08-19T14:44:20.138277+0000","last_change":"2024-08-19T14:43:16.089998+0000","last_active":"2024-08-19T14:44:20.138277+0000","last_peered":"2024-08-19T14:44:20.138277+0000","last_clean":"2024-08-19T14:44:20.138277+0000","last_became_active":"2024-08-19T14:43:16.089672+0000","last_became_peered":"2024-08-19T14:43:16.089672+0000","last_unstale":"2024-08-19T14:44:20.138277+0000","last_undegraded":"2024-08-19T14:44:20.138277+0000","last_fullsized":"2024-08-19T14:44:20.138277+0000","mapping_epoch":800,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":801,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:41:45.177828+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:41:45.177828+0000","last_clean_scrub_stamp":"2024-08-19T14:41:45.177828+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:44:25.137293+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,10],"acting":[11,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":18,"reported_epoch":830,"state":"active+clean","last_fresh":"2024-08-19T14:44:20.298599+0000","last_change":"2024-08-19T14:44:15.269581+0000","last_active":"2024-08-19T14:44:20.298599+0000","last_peered":"2024-08-19T14:44:20.298599+0000","last_clean":"2024-08-19T14:44:20.298599+0000","last_became_active":"2024-08-19T14:44:15.269188+0000","last_became_peered":"2024-08-19T14:44:15.269188+0000","last_unstale":"2024-08-19T14:44:20.298599+0000","last_undegraded":"2024-08-19T14:44:20.298599+0000","last_fullsized":"2024-08-19T14:44:20.298599+0000","mapping_epoch":826,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":827,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:17.058067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"70'139","reported_seq":1837,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-08-19T14:44:17.397058+0000","last_change":"2024-08-19T14:41:45.936228+0000","last_active":"2024-08-19T14:44:17.397058+0000","last_peered":"2024-08-19T14:44:17.397058+0000","last_clean":"2024-08-19T14:44:17.397058+0000","last_became_active":"2024-08-19T14:18:13.804721+0000","last_became_peered":"2024-08-19T14:18:13.804721+0000","last_unstale":"2024-08-19T14:44:17.397058+0000","last_undegraded":"2024-08-19T14:44:17.397058+0000","last_fullsized":"2024-08-19T14:44:17.397058+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:41:45.936114+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:41:45.936114+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":13,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-08-19T14:44:17.397052+0000","last_change":"2024-08-19T14:44:15.267888+0000","last_active":"2024-08-19T14:44:17.397052+0000","last_peered":"2024-08-19T14:44:17.397052+0000","last_clean":"2024-08-19T14:44:17.397052+0000","last_became_active":"2024-08-19T14:44:15.267604+0000","last_became_peered":"2024-08-19T14:44:15.267604+0000","last_unstale":"2024-08-19T14:44:17.397052+0000","last_undegraded":"2024-08-19T14:44:17.397052+0000","last_fullsized":"2024-08-19T14:44:17.397052+0000","mapping_epoch":826,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":827,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:27.344785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,8],"acting":[1,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"70'139","reported_seq":479,"reported_epoch":829,"state":"active+clean","last_fresh":"2024-08-19T14:44:19.070832+0000","last_change":"2024-08-19T14:43:11.471722+0000","last_active":"2024-08-19T14:44:19.070832+0000","last_peered":"2024-08-19T14:44:19.070832+0000","last_clean":"2024-08-19T14:44:19.070832+0000","last_became_active":"2024-08-19T14:43:11.470639+0000","last_became_peered":"2024-08-19T14:43:11.470639+0000","last_unstale":"2024-08-19T14:44:19.070832+0000","last_undegraded":"2024-08-19T14:44:19.070832+0000","last_fullsized":"2024-08-19T14:44:19.070832+0000","mapping_epoch":795,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":796,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:40:10.977156+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:19:59.302931+0000","last_clean_scrub_stamp":"2024-08-19T14:40:10.977156+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:44:19.766004+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":5,"num_write":8,"num_write_kb":133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":120211,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":18,"reported_epoch":830,"state":"active+clean","last_fresh":"2024-08-19T14:44:20.298225+0000","last_change":"2024-08-19T14:44:15.269253+0000","last_active":"2024-08-19T14:44:20.298225+0000","last_peered":"2024-08-19T14:44:20.298225+0000","last_clean":"2024-08-19T14:44:20.298225+0000","last_became_active":"2024-08-19T14:44:15.268485+0000","last_became_peered":"2024-08-19T14:44:15.268485+0000","last_unstale":"2024-08-19T14:44:20.298225+0000","last_undegraded":"2024-08-19T14:44:20.298225+0000","last_fullsized":"2024-08-19T14:44:20.298225+0000","mapping_epoch":826,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":827,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:36.120880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,5],"acting":[10,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1","version":"70'139","reported_seq":482,"reported_epoch":830,"state":"active+clean","last_fresh":"2024-08-19T14:44:20.298056+0000","last_change":"2024-08-19T14:43:16.098691+0000","last_active":"2024-08-19T14:44:20.298056+0000","last_peered":"2024-08-19T14:44:20.298056+0000","last_clean":"2024-08-19T14:44:20.298056+0000","last_became_active":"2024-08-19T14:43:16.098337+0000","last_became_peered":"2024-08-19T14:43:16.098337+0000","last_unstale":"2024-08-19T14:44:20.298056+0000","last_undegraded":"2024-08-19T14:44:20.298056+0000","last_fullsized":"2024-08-19T14:44:20.298056+0000","mapping_epoch":800,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":801,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:41:43.141372+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:41:43.141372+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,2],"acting":[10,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":13,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-08-19T14:44:17.397442+0000","last_change":"2024-08-19T14:44:15.415460+0000","last_active":"2024-08-19T14:44:17.397442+0000","last_peered":"2024-08-19T14:44:17.397442+0000","last_clean":"2024-08-19T14:44:17.397442+0000","last_became_active":"2024-08-19T14:44:15.415202+0000","last_became_peered":"2024-08-19T14:44:15.415202+0000","last_unstale":"2024-08-19T14:44:17.397442+0000","last_undegraded":"2024-08-19T14:44:17.397442+0000","last_fullsized":"2024-08-19T14:44:17.397442+0000","mapping_epoch":826,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":827,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:23.555885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,9],"acting":[1,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"70'139","reported_seq":1056,"reported_epoch":830,"state":"active+clean","last_fresh":"2024-08-19T14:44:20.298823+0000","last_change":"2024-08-19T14:43:16.114765+0000","last_active":"2024-08-19T14:44:20.298823+0000","last_peered":"2024-08-19T14:44:20.298823+0000","last_clean":"2024-08-19T14:44:20.298823+0000","last_became_active":"2024-08-19T14:43:16.100793+0000","last_became_peered":"2024-08-19T14:43:16.100793+0000","last_unstale":"2024-08-19T14:44:20.298823+0000","last_undegraded":"2024-08-19T14:44:20.298823+0000","last_fullsized":"2024-08-19T14:44:20.298823+0000","mapping_epoch":800,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":801,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:41:35.981666+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:21:46.057624+0000","last_clean_scrub_stamp":"2024-08-19T14:41:35.981666+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,2],"acting":[10,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":18,"reported_epoch":830,"state":"active+clean","last_fresh":"2024-08-19T14:44:20.298634+0000","last_change":"2024-08-19T14:44:15.415042+0000","last_active":"2024-08-19T14:44:20.298634+0000","last_peered":"2024-08-19T14:44:20.298634+0000","last_clean":"2024-08-19T14:44:20.298634+0000","last_became_active":"2024-08-19T14:44:15.414595+0000","last_became_peered":"2024-08-19T14:44:15.414595+0000","last_unstale":"2024-08-19T14:44:20.298634+0000","last_undegraded":"2024-08-19T14:44:20.298634+0000","last_fullsized":"2024-08-19T14:44:20.298634+0000","mapping_epoch":826,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":827,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:29.023228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,9],"acting":[10,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.6","version":"70'139","reported_seq":348,"reported_epoch":830,"state":"active+clean","last_fresh":"2024-08-19T14:44:20.298094+0000","last_change":"2024-08-19T14:43:16.091622+0000","last_active":"2024-08-19T14:44:20.298094+0000","last_peered":"2024-08-19T14:44:20.298094+0000","last_clean":"2024-08-19T14:44:20.298094+0000","last_became_active":"2024-08-19T14:43:16.091256+0000","last_became_peered":"2024-08-19T14:43:16.091256+0000","last_unstale":"2024-08-19T14:44:20.298094+0000","last_undegraded":"2024-08-19T14:44:20.298094+0000","last_fullsized":"2024-08-19T14:44:20.298094+0000","mapping_epoch":800,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":801,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:41:55.664660+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:27.848870+0000","last_clean_scrub_stamp":"2024-08-19T14:41:55.664660+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:44:21.132031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13417,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":66,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":48084,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":13,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-08-19T14:44:17.397262+0000","last_change":"2024-08-19T14:44:15.267600+0000","last_active":"2024-08-19T14:44:17.397262+0000","last_peered":"2024-08-19T14:44:17.397262+0000","last_clean":"2024-08-19T14:44:17.397262+0000","last_became_active":"2024-08-19T14:44:15.267306+0000","last_became_peered":"2024-08-19T14:44:15.267306+0000","last_unstale":"2024-08-19T14:44:17.397262+0000","last_undegraded":"2024-08-19T14:44:17.397262+0000","last_fullsized":"2024-08-19T14:44:17.397262+0000","mapping_epoch":826,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":827,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:25.918180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"70'139","reported_seq":438,"reported_epoch":830,"state":"active+clean","last_fresh":"2024-08-19T14:44:20.298255+0000","last_change":"2024-08-19T14:43:16.116905+0000","last_active":"2024-08-19T14:44:20.298255+0000","last_peered":"2024-08-19T14:44:20.298255+0000","last_clean":"2024-08-19T14:44:20.298255+0000","last_became_active":"2024-08-19T14:43:16.099747+0000","last_became_peered":"2024-08-19T14:43:16.099747+0000","last_unstale":"2024-08-19T14:44:20.298255+0000","last_undegraded":"2024-08-19T14:44:20.298255+0000","last_fullsized":"2024-08-19T14:44:20.298255+0000","mapping_epoch":800,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":801,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:41:48.878043+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:41:48.878043+0000","last_clean_scrub_stamp":"2024-08-19T14:41:48.878043+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,11],"acting":[10,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":16,"reported_epoch":829,"state":"active+clean","last_fresh":"2024-08-19T14:44:19.071017+0000","last_change":"2024-08-19T14:44:15.416651+0000","last_active":"2024-08-19T14:44:19.071017+0000","last_peered":"2024-08-19T14:44:19.071017+0000","last_clean":"2024-08-19T14:44:19.071017+0000","last_became_active":"2024-08-19T14:44:15.416278+0000","last_became_peered":"2024-08-19T14:44:15.416278+0000","last_unstale":"2024-08-19T14:44:19.071017+0000","last_undegraded":"2024-08-19T14:44:19.071017+0000","last_fullsized":"2024-08-19T14:44:19.071017+0000","mapping_epoch":826,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":827,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:21.705266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,10],"acting":[4,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"70'139","reported_seq":908,"reported_epoch":830,"state":"active+clean","last_fresh":"2024-08-19T14:44:20.138473+0000","last_change":"2024-08-19T14:43:08.462532+0000","last_active":"2024-08-19T14:44:20.138473+0000","last_peered":"2024-08-19T14:44:20.138473+0000","last_clean":"2024-08-19T14:44:20.138473+0000","last_became_active":"2024-08-19T14:43:08.462160+0000","last_became_peered":"2024-08-19T14:43:08.462160+0000","last_unstale":"2024-08-19T14:44:20.138473+0000","last_undegraded":"2024-08-19T14:44:20.138473+0000","last_fullsized":"2024-08-19T14:44:20.138473+0000","mapping_epoch":793,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":794,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:41:55.868614+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:41:55.868614+0000","last_clean_scrub_stamp":"2024-08-19T14:41:55.868614+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:44:25.137334+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,4],"acting":[11,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":13,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-08-19T14:44:17.397027+0000","last_change":"2024-08-19T14:44:15.415441+0000","last_active":"2024-08-19T14:44:17.397027+0000","last_peered":"2024-08-19T14:44:17.397027+0000","last_clean":"2024-08-19T14:44:17.397027+0000","last_became_active":"2024-08-19T14:44:15.415137+0000","last_became_peered":"2024-08-19T14:44:15.415137+0000","last_unstale":"2024-08-19T14:44:17.397027+0000","last_undegraded":"2024-08-19T14:44:17.397027+0000","last_fullsized":"2024-08-19T14:44:17.397027+0000","mapping_epoch":826,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":827,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:19.315374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,11],"acting":[1,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"70'139","reported_seq":1070,"reported_epoch":829,"state":"active+clean","last_fresh":"2024-08-19T14:44:19.070940+0000","last_change":"2024-08-19T14:42:39.072075+0000","last_active":"2024-08-19T14:44:19.070940+0000","last_peered":"2024-08-19T14:44:19.070940+0000","last_clean":"2024-08-19T14:44:19.070940+0000","last_became_active":"2024-08-19T14:42:39.071701+0000","last_became_peered":"2024-08-19T14:42:39.071701+0000","last_unstale":"2024-08-19T14:44:19.070940+0000","last_undegraded":"2024-08-19T14:44:19.070940+0000","last_fullsized":"2024-08-19T14:44:19.070940+0000","mapping_epoch":778,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":779,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:41:55.751274+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:28.864870+0000","last_clean_scrub_stamp":"2024-08-19T14:41:55.751274+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:44:23.749452+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,2],"acting":[4,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":18,"reported_epoch":830,"state":"active+clean","last_fresh":"2024-08-19T14:44:20.298133+0000","last_change":"2024-08-19T14:44:15.416094+0000","last_active":"2024-08-19T14:44:20.298133+0000","last_peered":"2024-08-19T14:44:20.298133+0000","last_clean":"2024-08-19T14:44:20.298133+0000","last_became_active":"2024-08-19T14:44:15.415731+0000","last_became_peered":"2024-08-19T14:44:15.415731+0000","last_unstale":"2024-08-19T14:44:20.298133+0000","last_undegraded":"2024-08-19T14:44:20.298133+0000","last_fullsized":"2024-08-19T14:44:20.298133+0000","mapping_epoch":826,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":827,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:30.021574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,2],"acting":[10,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.9","version":"70'139","reported_seq":1618,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-08-19T14:44:17.564569+0000","last_change":"2024-08-19T14:43:08.469441+0000","last_active":"2024-08-19T14:44:17.564569+0000","last_peered":"2024-08-19T14:44:17.564569+0000","last_clean":"2024-08-19T14:44:17.564569+0000","last_became_active":"2024-08-19T14:43:08.469067+0000","last_became_peered":"2024-08-19T14:43:08.469067+0000","last_unstale":"2024-08-19T14:44:17.564569+0000","last_undegraded":"2024-08-19T14:44:17.564569+0000","last_fullsized":"2024-08-19T14:44:17.564569+0000","mapping_epoch":793,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":794,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:42:00.561080+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:42:00.561080+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,5],"acting":[9,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":13,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-08-19T14:44:17.397012+0000","last_change":"2024-08-19T14:44:15.066444+0000","last_active":"2024-08-19T14:44:17.397012+0000","last_peered":"2024-08-19T14:44:17.397012+0000","last_clean":"2024-08-19T14:44:17.397012+0000","last_became_active":"2024-08-19T14:44:15.065971+0000","last_became_peered":"2024-08-19T14:44:15.065971+0000","last_unstale":"2024-08-19T14:44:17.397012+0000","last_undegraded":"2024-08-19T14:44:17.397012+0000","last_fullsized":"2024-08-19T14:44:17.397012+0000","mapping_epoch":826,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":827,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:20.903990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,8],"acting":[1,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"70'139","reported_seq":582,"reported_epoch":830,"state":"active+clean","last_fresh":"2024-08-19T14:44:20.298179+0000","last_change":"2024-08-19T14:42:08.692385+0000","last_active":"2024-08-19T14:44:20.298179+0000","last_peered":"2024-08-19T14:44:20.298179+0000","last_clean":"2024-08-19T14:44:20.298179+0000","last_became_active":"2024-08-19T14:40:52.455488+0000","last_became_peered":"2024-08-19T14:40:52.455488+0000","last_unstale":"2024-08-19T14:44:20.298179+0000","last_undegraded":"2024-08-19T14:44:20.298179+0000","last_fullsized":"2024-08-19T14:44:20.298179+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:42:08.692264+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:42:08.692264+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:44:20.824187+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,2],"acting":[8,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":18,"reported_epoch":830,"state":"active+clean","last_fresh":"2024-08-19T14:44:20.298377+0000","last_change":"2024-08-19T14:44:15.414974+0000","last_active":"2024-08-19T14:44:20.298377+0000","last_peered":"2024-08-19T14:44:20.298377+0000","last_clean":"2024-08-19T14:44:20.298377+0000","last_became_active":"2024-08-19T14:44:15.414623+0000","last_became_peered":"2024-08-19T14:44:15.414623+0000","last_unstale":"2024-08-19T14:44:20.298377+0000","last_undegraded":"2024-08-19T14:44:20.298377+0000","last_fullsized":"2024-08-19T14:44:20.298377+0000","mapping_epoch":826,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":827,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:18.725495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.b","version":"70'139","reported_seq":1360,"reported_epoch":830,"state":"active+clean","last_fresh":"2024-08-19T14:44:20.138470+0000","last_change":"2024-08-19T14:42:39.077445+0000","last_active":"2024-08-19T14:44:20.138470+0000","last_peered":"2024-08-19T14:44:20.138470+0000","last_clean":"2024-08-19T14:44:20.138470+0000","last_became_active":"2024-08-19T14:42:39.077214+0000","last_became_peered":"2024-08-19T14:42:39.077214+0000","last_unstale":"2024-08-19T14:44:20.138470+0000","last_undegraded":"2024-08-19T14:44:20.138470+0000","last_fullsized":"2024-08-19T14:44:20.138470+0000","mapping_epoch":778,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":779,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:41:50.892343+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:41:50.892343+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:44:25.137316+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,2],"acting":[11,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":18,"reported_epoch":830,"state":"active+clean","last_fresh":"2024-08-19T14:44:20.298451+0000","last_change":"2024-08-19T14:44:15.268309+0000","last_active":"2024-08-19T14:44:20.298451+0000","last_peered":"2024-08-19T14:44:20.298451+0000","last_clean":"2024-08-19T14:44:20.298451+0000","last_became_active":"2024-08-19T14:44:15.267907+0000","last_became_peered":"2024-08-19T14:44:15.267907+0000","last_unstale":"2024-08-19T14:44:20.298451+0000","last_undegraded":"2024-08-19T14:44:20.298451+0000","last_fullsized":"2024-08-19T14:44:20.298451+0000","mapping_epoch":826,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":827,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:42.468389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,1],"acting":[10,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.e","version":"70'139","reported_seq":1557,"reported_epoch":829,"state":"active+clean","last_fresh":"2024-08-19T14:44:19.071187+0000","last_change":"2024-08-19T14:43:08.463874+0000","last_active":"2024-08-19T14:44:19.071187+0000","last_peered":"2024-08-19T14:44:19.071187+0000","last_clean":"2024-08-19T14:44:19.071187+0000","last_became_active":"2024-08-19T14:43:08.463102+0000","last_became_peered":"2024-08-19T14:43:08.463102+0000","last_unstale":"2024-08-19T14:44:19.071187+0000","last_undegraded":"2024-08-19T14:44:19.071187+0000","last_fullsized":"2024-08-19T14:44:19.071187+0000","mapping_epoch":793,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":794,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:41:46.892116+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:40:03.764672+0000","last_clean_scrub_stamp":"2024-08-19T14:41:46.892116+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:44:23.749470+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,1],"acting":[4,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":18,"reported_epoch":830,"state":"active+clean","last_fresh":"2024-08-19T14:44:20.298629+0000","last_change":"2024-08-19T14:44:15.068003+0000","last_active":"2024-08-19T14:44:20.298629+0000","last_peered":"2024-08-19T14:44:20.298629+0000","last_clean":"2024-08-19T14:44:20.298629+0000","last_became_active":"2024-08-19T14:44:15.067635+0000","last_became_peered":"2024-08-19T14:44:15.067635+0000","last_unstale":"2024-08-19T14:44:20.298629+0000","last_undegraded":"2024-08-19T14:44:20.298629+0000","last_fullsized":"2024-08-19T14:44:20.298629+0000","mapping_epoch":826,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":827,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:40.928253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,1],"acting":[10,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.f","version":"70'139","reported_seq":1300,"reported_epoch":829,"state":"active+clean","last_fresh":"2024-08-19T14:44:19.071043+0000","last_change":"2024-08-19T14:42:03.775080+0000","last_active":"2024-08-19T14:44:19.071043+0000","last_peered":"2024-08-19T14:44:19.071043+0000","last_clean":"2024-08-19T14:44:19.071043+0000","last_became_active":"2024-08-19T14:40:52.446953+0000","last_became_peered":"2024-08-19T14:40:52.446953+0000","last_unstale":"2024-08-19T14:44:19.071043+0000","last_undegraded":"2024-08-19T14:44:19.071043+0000","last_fullsized":"2024-08-19T14:44:19.071043+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:42:03.774987+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:42:03.774987+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:44:23.749503+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,8],"acting":[4,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":13,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-08-19T14:44:17.397212+0000","last_change":"2024-08-19T14:44:15.066789+0000","last_active":"2024-08-19T14:44:17.397212+0000","last_peered":"2024-08-19T14:44:17.397212+0000","last_clean":"2024-08-19T14:44:17.397212+0000","last_became_active":"2024-08-19T14:44:15.065981+0000","last_became_peered":"2024-08-19T14:44:15.065981+0000","last_unstale":"2024-08-19T14:44:17.397212+0000","last_undegraded":"2024-08-19T14:44:17.397212+0000","last_fullsized":"2024-08-19T14:44:17.397212+0000","mapping_epoch":826,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":827,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:38.445367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.c","version":"70'139","reported_seq":600,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-08-19T14:44:17.397176+0000","last_change":"2024-08-19T14:42:39.073450+0000","last_active":"2024-08-19T14:44:17.397176+0000","last_peered":"2024-08-19T14:44:17.397176+0000","last_clean":"2024-08-19T14:44:17.397176+0000","last_became_active":"2024-08-19T14:42:39.073086+0000","last_became_peered":"2024-08-19T14:42:39.073086+0000","last_unstale":"2024-08-19T14:44:17.397176+0000","last_undegraded":"2024-08-19T14:44:17.397176+0000","last_fullsized":"2024-08-19T14:44:17.397176+0000","mapping_epoch":778,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":779,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:41:37.152446+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:22:52.164586+0000","last_clean_scrub_stamp":"2024-08-19T14:41:37.152446+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,5],"acting":[1,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.5","version":"830'187","reported_seq":205,"reported_epoch":830,"state":"active+clean","last_fresh":"2024-08-19T14:44:22.505962+0000","last_change":"2024-08-19T14:44:15.268306+0000","last_active":"2024-08-19T14:44:22.505962+0000","last_peered":"2024-08-19T14:44:22.505962+0000","last_clean":"2024-08-19T14:44:22.505962+0000","last_became_active":"2024-08-19T14:44:15.267867+0000","last_became_peered":"2024-08-19T14:44:15.267867+0000","last_unstale":"2024-08-19T14:44:22.505962+0000","last_undegraded":"2024-08-19T14:44:22.505962+0000","last_fullsized":"2024-08-19T14:44:22.505962+0000","mapping_epoch":826,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":827,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:31.746018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12255232,"num_objects":187,"num_object_clones":0,"num_object_copies":561,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":187,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":187,"num_write_kb":11968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,1],"acting":[11,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.d","version":"70'139","reported_seq":391,"reported_epoch":829,"state":"active+clean","last_fresh":"2024-08-19T14:44:19.071040+0000","last_change":"2024-08-19T14:43:08.459939+0000","last_active":"2024-08-19T14:44:19.071040+0000","last_peered":"2024-08-19T14:44:19.071040+0000","last_clean":"2024-08-19T14:44:19.071040+0000","last_became_active":"2024-08-19T14:43:08.459577+0000","last_became_peered":"2024-08-19T14:43:08.459577+0000","last_unstale":"2024-08-19T14:44:19.071040+0000","last_undegraded":"2024-08-19T14:44:19.071040+0000","last_fullsized":"2024-08-19T14:44:19.071040+0000","mapping_epoch":793,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":794,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:42:09.549880+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:24:50.276408+0000","last_clean_scrub_stamp":"2024-08-19T14:42:09.549880+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:44:19.765969+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,8],"acting":[4,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.12","version":"70'139","reported_seq":1329,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-08-19T14:44:17.397129+0000","last_change":"2024-08-19T14:43:08.458336+0000","last_active":"2024-08-19T14:44:17.397129+0000","last_peered":"2024-08-19T14:44:17.397129+0000","last_clean":"2024-08-19T14:44:17.397129+0000","last_became_active":"2024-08-19T14:43:08.457897+0000","last_became_peered":"2024-08-19T14:43:08.457897+0000","last_unstale":"2024-08-19T14:44:17.397129+0000","last_undegraded":"2024-08-19T14:44:17.397129+0000","last_fullsized":"2024-08-19T14:44:17.397129+0000","mapping_epoch":793,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":794,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:41:49.815826+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:41:49.815826+0000","last_clean_scrub_stamp":"2024-08-19T14:41:49.815826+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.13","version":"70'139","reported_seq":1880,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-08-19T14:44:17.396996+0000","last_change":"2024-08-19T14:43:16.087256+0000","last_active":"2024-08-19T14:44:17.396996+0000","last_peered":"2024-08-19T14:44:17.396996+0000","last_clean":"2024-08-19T14:44:17.396996+0000","last_became_active":"2024-08-19T14:43:16.086968+0000","last_became_peered":"2024-08-19T14:43:16.086968+0000","last_unstale":"2024-08-19T14:44:17.396996+0000","last_undegraded":"2024-08-19T14:44:17.396996+0000","last_fullsized":"2024-08-19T14:44:17.396996+0000","mapping_epoch":800,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":801,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:41:55.847727+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:33:35.786105+0000","last_clean_scrub_stamp":"2024-08-19T14:41:55.847727+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.10","version":"70'139","reported_seq":808,"reported_epoch":830,"state":"active+clean","last_fresh":"2024-08-19T14:44:20.298365+0000","last_change":"2024-08-19T14:43:16.114296+0000","last_active":"2024-08-19T14:44:20.298365+0000","last_peered":"2024-08-19T14:44:20.298365+0000","last_clean":"2024-08-19T14:44:20.298365+0000","last_became_active":"2024-08-19T14:43:16.113982+0000","last_became_peered":"2024-08-19T14:43:16.113982+0000","last_unstale":"2024-08-19T14:44:20.298365+0000","last_undegraded":"2024-08-19T14:44:20.298365+0000","last_fullsized":"2024-08-19T14:44:20.298365+0000","mapping_epoch":800,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":801,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:40:32.035620+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:34:58.616646+0000","last_clean_scrub_stamp":"2024-08-19T14:40:32.035620+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:44:21.132071+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,8],"acting":[10,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.11","version":"70'139","reported_seq":398,"reported_epoch":830,"state":"active+clean","last_fresh":"2024-08-19T14:44:20.298379+0000","last_change":"2024-08-19T14:42:39.064912+0000","last_active":"2024-08-19T14:44:20.298379+0000","last_peered":"2024-08-19T14:44:20.298379+0000","last_clean":"2024-08-19T14:44:20.298379+0000","last_became_active":"2024-08-19T14:42:39.064403+0000","last_became_peered":"2024-08-19T14:42:39.064403+0000","last_unstale":"2024-08-19T14:44:20.298379+0000","last_undegraded":"2024-08-19T14:44:20.298379+0000","last_fullsized":"2024-08-19T14:44:20.298379+0000","mapping_epoch":778,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":779,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:41:47.147376+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:41:47.147376+0000","last_clean_scrub_stamp":"2024-08-19T14:41:47.147376+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:44:22.883505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":5,"num_write":9,"num_write_kb":134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":96169,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,1],"acting":[8,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"70'139","reported_seq":1498,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-08-19T14:44:17.397333+0000","last_change":"2024-08-19T14:43:04.925737+0000","last_active":"2024-08-19T14:44:17.397333+0000","last_peered":"2024-08-19T14:44:17.397333+0000","last_clean":"2024-08-19T14:44:17.397333+0000","last_became_active":"2024-08-19T14:42:39.066670+0000","last_became_peered":"2024-08-19T14:42:39.066670+0000","last_unstale":"2024-08-19T14:44:17.397333+0000","last_undegraded":"2024-08-19T14:44:17.397333+0000","last_fullsized":"2024-08-19T14:44:17.397333+0000","mapping_epoch":778,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":779,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:43:04.925661+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:28:34.216370+0000","last_clean_scrub_stamp":"2024-08-19T14:43:04.925661+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,5],"acting":[1,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"70'139","reported_seq":1644,"reported_epoch":830,"state":"active+clean","last_fresh":"2024-08-19T14:44:20.298729+0000","last_change":"2024-08-19T14:43:16.127472+0000","last_active":"2024-08-19T14:44:20.298729+0000","last_peered":"2024-08-19T14:44:20.298729+0000","last_clean":"2024-08-19T14:44:20.298729+0000","last_became_active":"2024-08-19T14:43:16.123238+0000","last_became_peered":"2024-08-19T14:43:16.123238+0000","last_unstale":"2024-08-19T14:44:20.298729+0000","last_undegraded":"2024-08-19T14:44:20.298729+0000","last_fullsized":"2024-08-19T14:44:20.298729+0000","mapping_epoch":800,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":801,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:42:14.678271+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:42:14.678271+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:44:20.824207+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.14","version":"70'139","reported_seq":1063,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-08-19T14:44:17.397056+0000","last_change":"2024-08-19T14:43:08.454671+0000","last_active":"2024-08-19T14:44:17.397056+0000","last_peered":"2024-08-19T14:44:17.397056+0000","last_clean":"2024-08-19T14:44:17.397056+0000","last_became_active":"2024-08-19T14:43:08.454350+0000","last_became_peered":"2024-08-19T14:43:08.454350+0000","last_unstale":"2024-08-19T14:44:17.397056+0000","last_undegraded":"2024-08-19T14:44:17.397056+0000","last_fullsized":"2024-08-19T14:44:17.397056+0000","mapping_epoch":793,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":794,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:42:06.543798+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:42:06.543798+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,9],"acting":[1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"70'139","reported_seq":395,"reported_epoch":830,"state":"active+clean","last_fresh":"2024-08-19T14:44:20.298843+0000","last_change":"2024-08-19T14:43:08.457011+0000","last_active":"2024-08-19T14:44:20.298843+0000","last_peered":"2024-08-19T14:44:20.298843+0000","last_clean":"2024-08-19T14:44:20.298843+0000","last_became_active":"2024-08-19T14:43:08.456550+0000","last_became_peered":"2024-08-19T14:43:08.456550+0000","last_unstale":"2024-08-19T14:44:20.298843+0000","last_undegraded":"2024-08-19T14:44:20.298843+0000","last_fullsized":"2024-08-19T14:44:20.298843+0000","mapping_epoch":793,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":794,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:42:15.607984+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:20:09.180923+0000","last_clean_scrub_stamp":"2024-08-19T14:42:15.607984+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:44:22.883525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13418,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":48085,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,1],"acting":[8,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"70'139","reported_seq":1492,"reported_epoch":829,"state":"active+clean","last_fresh":"2024-08-19T14:44:19.070809+0000","last_change":"2024-08-19T14:43:11.472105+0000","last_active":"2024-08-19T14:44:19.070809+0000","last_peered":"2024-08-19T14:44:19.070809+0000","last_clean":"2024-08-19T14:44:19.070809+0000","last_became_active":"2024-08-19T14:43:11.471324+0000","last_became_peered":"2024-08-19T14:43:11.471324+0000","last_unstale":"2024-08-19T14:44:19.070809+0000","last_undegraded":"2024-08-19T14:44:19.070809+0000","last_fullsized":"2024-08-19T14:44:19.070809+0000","mapping_epoch":795,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":796,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:41:51.832284+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:41:51.832284+0000","last_clean_scrub_stamp":"2024-08-19T14:41:51.832284+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:44:23.749487+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,11],"acting":[4,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1b","version":"70'139","reported_seq":1876,"reported_epoch":830,"state":"active+clean","last_fresh":"2024-08-19T14:44:20.298382+0000","last_change":"2024-08-19T14:42:39.066054+0000","last_active":"2024-08-19T14:44:20.298382+0000","last_peered":"2024-08-19T14:44:20.298382+0000","last_clean":"2024-08-19T14:44:20.298382+0000","last_became_active":"2024-08-19T14:42:39.065604+0000","last_became_peered":"2024-08-19T14:42:39.065604+0000","last_unstale":"2024-08-19T14:44:20.298382+0000","last_undegraded":"2024-08-19T14:44:20.298382+0000","last_fullsized":"2024-08-19T14:44:20.298382+0000","mapping_epoch":778,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":779,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:40:43.238820+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:37:59.510697+0000","last_clean_scrub_stamp":"2024-08-19T14:40:43.238820+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:44:20.824165+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,2],"acting":[8,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.18","version":"70'139","reported_seq":855,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-08-19T14:44:17.396904+0000","last_change":"2024-08-19T14:43:08.458371+0000","last_active":"2024-08-19T14:44:17.396904+0000","last_peered":"2024-08-19T14:44:17.396904+0000","last_clean":"2024-08-19T14:44:17.396904+0000","last_became_active":"2024-08-19T14:43:08.457799+0000","last_became_peered":"2024-08-19T14:43:08.457799+0000","last_unstale":"2024-08-19T14:44:17.396904+0000","last_undegraded":"2024-08-19T14:44:17.396904+0000","last_fullsized":"2024-08-19T14:44:17.396904+0000","mapping_epoch":793,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":794,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:41:53.603570+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:27:33.084161+0000","last_clean_scrub_stamp":"2024-08-19T14:41:53.603570+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.19","version":"70'139","reported_seq":720,"reported_epoch":830,"state":"active+clean","last_fresh":"2024-08-19T14:44:20.298271+0000","last_change":"2024-08-19T14:43:16.097396+0000","last_active":"2024-08-19T14:44:20.298271+0000","last_peered":"2024-08-19T14:44:20.298271+0000","last_clean":"2024-08-19T14:44:20.298271+0000","last_became_active":"2024-08-19T14:43:16.096717+0000","last_became_peered":"2024-08-19T14:43:16.096717+0000","last_unstale":"2024-08-19T14:44:20.298271+0000","last_undegraded":"2024-08-19T14:44:20.298271+0000","last_fullsized":"2024-08-19T14:44:20.298271+0000","mapping_epoch":800,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":801,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:41:36.108239+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:41:36.108239+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:44:21.132052+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,8],"acting":[10,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1e","version":"70'139","reported_seq":889,"reported_epoch":830,"state":"active+clean","last_fresh":"2024-08-19T14:44:20.138563+0000","last_change":"2024-08-19T14:43:16.126022+0000","last_active":"2024-08-19T14:44:20.138563+0000","last_peered":"2024-08-19T14:44:20.138563+0000","last_clean":"2024-08-19T14:44:20.138563+0000","last_became_active":"2024-08-19T14:43:16.124258+0000","last_became_peered":"2024-08-19T14:43:16.124258+0000","last_unstale":"2024-08-19T14:44:20.138563+0000","last_undegraded":"2024-08-19T14:44:20.138563+0000","last_fullsized":"2024-08-19T14:44:20.138563+0000","mapping_epoch":800,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":801,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:41:58.896684+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:27:04.780939+0000","last_clean_scrub_stamp":"2024-08-19T14:41:58.896684+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:44:25.137350+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,10],"acting":[11,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1c","version":"70'139","reported_seq":1629,"reported_epoch":830,"state":"active+clean","last_fresh":"2024-08-19T14:44:20.298558+0000","last_change":"2024-08-19T14:43:04.811401+0000","last_active":"2024-08-19T14:44:20.298558+0000","last_peered":"2024-08-19T14:44:20.298558+0000","last_clean":"2024-08-19T14:44:20.298558+0000","last_became_active":"2024-08-19T14:30:47.908866+0000","last_became_peered":"2024-08-19T14:30:47.908866+0000","last_unstale":"2024-08-19T14:44:20.298558+0000","last_undegraded":"2024-08-19T14:44:20.298558+0000","last_fullsized":"2024-08-19T14:44:20.298558+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:43:04.811320+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:43:04.811320+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,11],"acting":[8,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1d","version":"70'139","reported_seq":1526,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-08-19T14:44:17.397228+0000","last_change":"2024-08-19T14:41:38.948464+0000","last_active":"2024-08-19T14:44:17.397228+0000","last_peered":"2024-08-19T14:44:17.397228+0000","last_clean":"2024-08-19T14:44:17.397228+0000","last_became_active":"2024-08-19T14:35:11.540488+0000","last_became_peered":"2024-08-19T14:35:11.540488+0000","last_unstale":"2024-08-19T14:44:17.397228+0000","last_undegraded":"2024-08-19T14:44:17.397228+0000","last_fullsized":"2024-08-19T14:44:17.397228+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:41:38.947168+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:41:38.947168+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":26017792,"num_objects":397,"num_object_clones":0,"num_object_copies":1191,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":397,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":397,"num_write_kb":25408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":119406592,"data_stored":119406592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":397,"ondisk_log_size":397,"up":48,"acting":48,"num_store_stats":8},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":670873,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":15,"num_write":25,"num_write_kb":400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":841477,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2252800,"data_stored":2242144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187852,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1115596,"kb_used_data":47364,"kb_used_omap":1,"kb_used_meta":1068222,"kb_avail":92629556,"statfs":{"total":95995035648,"available":94852665344,"internally_reserved":0,"allocated":48500736,"data_stored":46107738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":1093859780},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":539,"seq":2314987372719,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":147076,"kb_used_data":6840,"kb_used_omap":4,"kb_used_meta":140219,"kb_avail":93598076,"statfs":{"total":95995035648,"available":95844429824,"internally_reserved":0,"allocated":7004160,"data_stored":4615409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4407,"internal_metadata":143584969},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":74,"seq":317827580309,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1080956,"kb_used_data":7148,"kb_used_omap":1,"kb_used_meta":1073790,"kb_avail":92664196,"statfs":{"total":95995035648,"available":94888136704,"internally_reserved":0,"allocated":7319552,"data_stored":4934716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":1099561409},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874652,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":958468,"kb_used_data":6840,"kb_used_omap":1,"kb_used_meta":951614,"kb_avail":92786684,"statfs":{"total":95995035648,"available":95013564416,"internally_reserved":0,"allocated":7004160,"data_stored":4615409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":974453189},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":0,"seq":244813136287,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":206158430627,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692262,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1151628,"kb_used_data":33408,"kb_used_omap":1,"kb_used_meta":1118206,"kb_avail":92593524,"statfs":{"total":95995035648,"available":94815768576,"internally_reserved":0,"allocated":34209792,"data_stored":31817489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1145043397},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986602,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1240324,"kb_used_data":36024,"kb_used_omap":1,"kb_used_meta":1204286,"kb_avail":92504828,"statfs":{"total":95995035648,"available":94724943872,"internally_reserved":0,"allocated":36888576,"data_stored":34499825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1233189317},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":111669150129,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":634,"seq":2723009265775,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":516684,"kb_used_data":7040,"kb_used_omap":4,"kb_used_meta":509627,"kb_avail":93228468,"statfs":{"total":95995035648,"available":95465951232,"internally_reserved":0,"allocated":7208960,"data_stored":4816657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4407,"internal_metadata":521858761},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575289,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1275392,"kb_used_data":7348,"kb_used_omap":1,"kb_used_meta":1268030,"kb_avail":92469760,"statfs":{"total":95995035648,"available":94689034240,"internally_reserved":0,"allocated":7524352,"data_stored":5135964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1298463173},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":34359738814,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":56229888,"data_stored":56229888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":63176704,"data_stored":63176704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T14:44:24.591 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-19T14:44:24.638 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:44:24.652 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:44:24.680 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:24.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:24.681 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:44:24.782 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:44:24.810 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:24.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:24.811 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:44:24.912 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:44:24.933 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:24.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:24.934 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:44:25.035 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:44:25.060 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:25.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:25.061 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:44:25.162 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:44:25.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:25.196 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:25.197 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:44:25.247 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:44:25.298 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:44:25.308 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 4 16 11087 11071 172.97 185.312 0.00363285 0.00577267 2024-08-19T14:44:25.329 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:25.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:25.331 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:44:25.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:25 smithi031 ceph-mon[16336]: from='client.17799 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:44:25.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:25 smithi031 ceph-mon[16336]: 1.11 scrub starts 2024-08-19T14:44:25.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:25 smithi031 ceph-mon[16336]: 1.11 scrub ok 2024-08-19T14:44:25.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:25 smithi031 ceph-mon[16336]: 1.10 scrub starts 2024-08-19T14:44:25.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:25 smithi031 ceph-mon[16336]: 1.10 scrub ok 2024-08-19T14:44:25.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:25 smithi031 ceph-mon[16336]: 1.c scrub starts 2024-08-19T14:44:25.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:25 smithi031 ceph-mon[16336]: 1.c scrub ok 2024-08-19T14:44:25.432 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:44:25.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:25.464 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:25.464 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:44:25.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:25 smithi032 ceph-mon[20004]: from='client.17799 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:44:25.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:25 smithi032 ceph-mon[20004]: 1.11 scrub starts 2024-08-19T14:44:25.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:25 smithi032 ceph-mon[20004]: 1.11 scrub ok 2024-08-19T14:44:25.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:25 smithi032 ceph-mon[20004]: 1.10 scrub starts 2024-08-19T14:44:25.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:25 smithi032 ceph-mon[20004]: 1.10 scrub ok 2024-08-19T14:44:25.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:25 smithi032 ceph-mon[20004]: 1.c scrub starts 2024-08-19T14:44:25.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:25 smithi032 ceph-mon[20004]: 1.c scrub ok 2024-08-19T14:44:25.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:25 smithi045 ceph-mon[20143]: from='client.17799 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:44:25.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:25 smithi045 ceph-mon[20143]: 1.11 scrub starts 2024-08-19T14:44:25.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:25 smithi045 ceph-mon[20143]: 1.11 scrub ok 2024-08-19T14:44:25.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:25 smithi045 ceph-mon[20143]: 1.10 scrub starts 2024-08-19T14:44:25.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:25 smithi045 ceph-mon[20143]: 1.10 scrub ok 2024-08-19T14:44:25.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:25 smithi045 ceph-mon[20143]: 1.c scrub starts 2024-08-19T14:44:25.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:25 smithi045 ceph-mon[20143]: 1.c scrub ok 2024-08-19T14:44:25.565 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:44:25.598 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:25.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:25.600 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:44:25.700 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:44:25.722 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:25.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:25.723 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:44:25.819 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:44:25.819 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:44:25.824 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:44:25.855 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:25.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:25.856 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:44:25.957 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:44:25.986 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:25.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:25.987 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:44:26.057 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:44:26.075 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:44:26.075 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-08-19T14:44:26.088 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:44:26.105 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is unset 2024-08-19T14:44:26.127 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:26.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:26.128 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:44:26.229 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:44:26.260 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:26.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:26.261 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:44:26.308 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 5 16 13850 13834 172.911 172.688 0.00526132 0.0057763 2024-08-19T14:44:26.362 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:44:26.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:26 smithi031 ceph-mon[16336]: pgmap v1966: 47 pgs: 47 active+clean; 160 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 27 MiB/s wr, 425 op/s 2024-08-19T14:44:26.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:26 smithi031 ceph-mon[16336]: 1.d scrub starts 2024-08-19T14:44:26.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:26 smithi031 ceph-mon[16336]: 1.15 scrub starts 2024-08-19T14:44:26.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:26 smithi031 ceph-mon[16336]: 1.d scrub ok 2024-08-19T14:44:26.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:26 smithi031 ceph-mon[16336]: 1.15 scrub ok 2024-08-19T14:44:26.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:26 smithi031 ceph-mon[16336]: 1.9 scrub starts 2024-08-19T14:44:26.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:26 smithi031 ceph-mon[16336]: 1.9 scrub ok 2024-08-19T14:44:26.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:26 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3426516538' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:44:26.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:26 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:44:26.401 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:26.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:26.403 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:44:26.503 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:44:26.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:26 smithi032 ceph-mon[20004]: pgmap v1966: 47 pgs: 47 active+clean; 160 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 27 MiB/s wr, 425 op/s 2024-08-19T14:44:26.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:26 smithi032 ceph-mon[20004]: 1.d scrub starts 2024-08-19T14:44:26.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:26 smithi032 ceph-mon[20004]: 1.15 scrub starts 2024-08-19T14:44:26.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:26 smithi032 ceph-mon[20004]: 1.d scrub ok 2024-08-19T14:44:26.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:26 smithi032 ceph-mon[20004]: 1.15 scrub ok 2024-08-19T14:44:26.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:26 smithi032 ceph-mon[20004]: 1.9 scrub starts 2024-08-19T14:44:26.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:26 smithi032 ceph-mon[20004]: 1.9 scrub ok 2024-08-19T14:44:26.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:26 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3426516538' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:44:26.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:26 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:44:26.530 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:26.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:26.531 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:44:26.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:26 smithi045 ceph-mon[20143]: pgmap v1966: 47 pgs: 47 active+clean; 160 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 27 MiB/s wr, 425 op/s 2024-08-19T14:44:26.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:26 smithi045 ceph-mon[20143]: 1.d scrub starts 2024-08-19T14:44:26.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:26 smithi045 ceph-mon[20143]: 1.15 scrub starts 2024-08-19T14:44:26.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:26 smithi045 ceph-mon[20143]: 1.d scrub ok 2024-08-19T14:44:26.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:26 smithi045 ceph-mon[20143]: 1.15 scrub ok 2024-08-19T14:44:26.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:26 smithi045 ceph-mon[20143]: 1.9 scrub starts 2024-08-19T14:44:26.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:26 smithi045 ceph-mon[20143]: 1.9 scrub ok 2024-08-19T14:44:26.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:26 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3426516538' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:44:26.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:26 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:44:26.632 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:44:26.661 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:26.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:26.662 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:44:26.763 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:44:26.793 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:26.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:26.794 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:44:26.895 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:44:26.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:26.917 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:26.917 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:44:27.018 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:44:27.045 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:27.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:27.046 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:44:27.147 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:44:27.173 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:27.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:27.174 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:44:27.275 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:44:27.296 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:27.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:27.297 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:44:27.308 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 6 16 16524 16508 171.942 167.125 0.00132857 0.00574133 2024-08-19T14:44:27.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:27 smithi031 ceph-mon[16336]: 1.2 scrub starts 2024-08-19T14:44:27.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:27 smithi031 ceph-mon[16336]: 1.2 scrub ok 2024-08-19T14:44:27.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:27 smithi031 ceph-mon[16336]: 1.1b scrub starts 2024-08-19T14:44:27.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:27 smithi031 ceph-mon[16336]: 1.1b scrub ok 2024-08-19T14:44:27.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:27 smithi031 ceph-mon[16336]: from='client.17811 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:44:27.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:27 smithi031 ceph-mon[16336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:44:27.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:27 smithi031 ceph-mon[16336]: Cluster is now healthy 2024-08-19T14:44:27.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:27 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:44:27.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:27 smithi031 ceph-mon[16336]: 1.0 scrub starts 2024-08-19T14:44:27.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:27 smithi031 ceph-mon[16336]: osdmap e832: 12 total, 12 up, 8 in 2024-08-19T14:44:27.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:27 smithi031 ceph-mon[16336]: 1.0 scrub ok 2024-08-19T14:44:27.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:27 smithi031 ceph-mon[16336]: 1.7 scrub starts 2024-08-19T14:44:27.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:27 smithi031 ceph-mon[16336]: 1.7 scrub ok 2024-08-19T14:44:27.398 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:44:27.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:27.421 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:27.421 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:44:27.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:27 smithi032 ceph-mon[20004]: 1.2 scrub starts 2024-08-19T14:44:27.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:27 smithi032 ceph-mon[20004]: 1.2 scrub ok 2024-08-19T14:44:27.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:27 smithi032 ceph-mon[20004]: 1.1b scrub starts 2024-08-19T14:44:27.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:27 smithi032 ceph-mon[20004]: 1.1b scrub ok 2024-08-19T14:44:27.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:27 smithi032 ceph-mon[20004]: from='client.17811 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:44:27.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:27 smithi032 ceph-mon[20004]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:44:27.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:27 smithi032 ceph-mon[20004]: Cluster is now healthy 2024-08-19T14:44:27.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:27 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:44:27.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:27 smithi032 ceph-mon[20004]: 1.0 scrub starts 2024-08-19T14:44:27.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:27 smithi032 ceph-mon[20004]: osdmap e832: 12 total, 12 up, 8 in 2024-08-19T14:44:27.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:27 smithi032 ceph-mon[20004]: 1.0 scrub ok 2024-08-19T14:44:27.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:27 smithi032 ceph-mon[20004]: 1.7 scrub starts 2024-08-19T14:44:27.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:27 smithi032 ceph-mon[20004]: 1.7 scrub ok 2024-08-19T14:44:27.522 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:44:27.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:27 smithi045 ceph-mon[20143]: 1.2 scrub starts 2024-08-19T14:44:27.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:27 smithi045 ceph-mon[20143]: 1.2 scrub ok 2024-08-19T14:44:27.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:27 smithi045 ceph-mon[20143]: 1.1b scrub starts 2024-08-19T14:44:27.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:27 smithi045 ceph-mon[20143]: 1.1b scrub ok 2024-08-19T14:44:27.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:27 smithi045 ceph-mon[20143]: from='client.17811 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:44:27.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:27 smithi045 ceph-mon[20143]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:44:27.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:27 smithi045 ceph-mon[20143]: Cluster is now healthy 2024-08-19T14:44:27.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:27 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:44:27.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:27 smithi045 ceph-mon[20143]: 1.0 scrub starts 2024-08-19T14:44:27.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:27 smithi045 ceph-mon[20143]: osdmap e832: 12 total, 12 up, 8 in 2024-08-19T14:44:27.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:27 smithi045 ceph-mon[20143]: 1.0 scrub ok 2024-08-19T14:44:27.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:27 smithi045 ceph-mon[20143]: 1.7 scrub starts 2024-08-19T14:44:27.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:27 smithi045 ceph-mon[20143]: 1.7 scrub ok 2024-08-19T14:44:27.548 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:27.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:27.549 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:44:27.650 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:44:27.679 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:27.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:27.680 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:44:27.781 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:44:27.806 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:27.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:27.807 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:44:27.908 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:44:27.933 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:27.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:27.934 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:44:28.015 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":1966,"stamp":"2024-08-19T14:44:24.350689+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":167918745,"num_objects":2555,"num_object_clones":0,"num_object_copies":7665,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2555,"num_whiteouts":0,"num_read":17,"num_read_kb":15,"num_write":2577,"num_write_kb":163728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":841477,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6861,"ondisk_log_size":6861,"up":141,"acting":141,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":144,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":7798868,"kb_used_data":430076,"kb_used_omap":17,"kb_used_meta":7368686,"kb_avail":742162348,"statfs":{"total":767960285184,"available":759974244352,"internally_reserved":0,"allocated":440397824,"data_stored":421264419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18389,"internal_metadata":7545534507},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":167247872,"num_objects":2552,"num_object_clones":0,"num_object_copies":7656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2552,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2552,"num_write_kb":163328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001302"},"pg_stats":[{"pgid":"9.8","version":"830'210","reported_seq":228,"reported_epoch":830,"state":"active+clean","last_fresh":"2024-08-19T14:44:22.501473+0000","last_change":"2024-08-19T14:44:15.415491+0000","last_active":"2024-08-19T14:44:22.501473+0000","last_peered":"2024-08-19T14:44:22.501473+0000","last_clean":"2024-08-19T14:44:22.501473+0000","last_became_active":"2024-08-19T14:44:15.414629+0000","last_became_peered":"2024-08-19T14:44:15.414629+0000","last_unstale":"2024-08-19T14:44:22.501473+0000","last_undegraded":"2024-08-19T14:44:22.501473+0000","last_fullsized":"2024-08-19T14:44:22.501473+0000","mapping_epoch":826,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":827,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":210,"log_dups_size":0,"ondisk_log_size":210,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:23.965899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13762560,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":210,"num_write_kb":13440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,5],"acting":[11,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"70'139","reported_seq":1988,"reported_epoch":830,"state":"active+clean","last_fresh":"2024-08-19T14:44:20.138277+0000","last_change":"2024-08-19T14:43:16.089998+0000","last_active":"2024-08-19T14:44:20.138277+0000","last_peered":"2024-08-19T14:44:20.138277+0000","last_clean":"2024-08-19T14:44:20.138277+0000","last_became_active":"2024-08-19T14:43:16.089672+0000","last_became_peered":"2024-08-19T14:43:16.089672+0000","last_unstale":"2024-08-19T14:44:20.138277+0000","last_undegraded":"2024-08-19T14:44:20.138277+0000","last_fullsized":"2024-08-19T14:44:20.138277+0000","mapping_epoch":800,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":801,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:41:45.177828+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:41:45.177828+0000","last_clean_scrub_stamp":"2024-08-19T14:41:45.177828+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:44:25.137293+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,10],"acting":[11,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":18,"reported_epoch":830,"state":"active+clean","last_fresh":"2024-08-19T14:44:20.298599+0000","last_change":"2024-08-19T14:44:15.269581+0000","last_active":"2024-08-19T14:44:20.298599+0000","last_peered":"2024-08-19T14:44:20.298599+0000","last_clean":"2024-08-19T14:44:20.298599+0000","last_became_active":"2024-08-19T14:44:15.269188+0000","last_became_peered":"2024-08-19T14:44:15.269188+0000","last_unstale":"2024-08-19T14:44:20.298599+0000","last_undegraded":"2024-08-19T14:44:20.298599+0000","last_fullsized":"2024-08-19T14:44:20.298599+0000","mapping_epoch":826,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":827,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:17.058067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"70'139","reported_seq":1843,"reported_epoch":831,"state":"active+clean","last_fresh":"2024-08-19T14:44:23.149267+0000","last_change":"2024-08-19T14:41:45.936228+0000","last_active":"2024-08-19T14:44:23.149267+0000","last_peered":"2024-08-19T14:44:23.149267+0000","last_clean":"2024-08-19T14:44:23.149267+0000","last_became_active":"2024-08-19T14:18:13.804721+0000","last_became_peered":"2024-08-19T14:18:13.804721+0000","last_unstale":"2024-08-19T14:44:23.149267+0000","last_undegraded":"2024-08-19T14:44:23.149267+0000","last_fullsized":"2024-08-19T14:44:23.149267+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:41:45.936114+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:41:45.936114+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:44:23.807497+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.a","version":"831'365","reported_seq":385,"reported_epoch":831,"state":"active+clean","last_fresh":"2024-08-19T14:44:23.672557+0000","last_change":"2024-08-19T14:44:15.267888+0000","last_active":"2024-08-19T14:44:23.672557+0000","last_peered":"2024-08-19T14:44:23.672557+0000","last_clean":"2024-08-19T14:44:23.672557+0000","last_became_active":"2024-08-19T14:44:15.267604+0000","last_became_peered":"2024-08-19T14:44:15.267604+0000","last_unstale":"2024-08-19T14:44:23.672557+0000","last_undegraded":"2024-08-19T14:44:23.672557+0000","last_fullsized":"2024-08-19T14:44:23.672557+0000","mapping_epoch":826,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":827,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":365,"log_dups_size":0,"ondisk_log_size":365,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:42.661379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23920640,"num_objects":365,"num_object_clones":0,"num_object_copies":1095,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":365,"num_write_kb":23360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,8],"acting":[1,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"70'139","reported_seq":479,"reported_epoch":829,"state":"active+clean","last_fresh":"2024-08-19T14:44:19.070832+0000","last_change":"2024-08-19T14:43:11.471722+0000","last_active":"2024-08-19T14:44:19.070832+0000","last_peered":"2024-08-19T14:44:19.070832+0000","last_clean":"2024-08-19T14:44:19.070832+0000","last_became_active":"2024-08-19T14:43:11.470639+0000","last_became_peered":"2024-08-19T14:43:11.470639+0000","last_unstale":"2024-08-19T14:44:19.070832+0000","last_undegraded":"2024-08-19T14:44:19.070832+0000","last_fullsized":"2024-08-19T14:44:19.070832+0000","mapping_epoch":795,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":796,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:40:10.977156+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:19:59.302931+0000","last_clean_scrub_stamp":"2024-08-19T14:40:10.977156+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:44:19.766004+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":5,"num_write":8,"num_write_kb":133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":120211,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":18,"reported_epoch":830,"state":"active+clean","last_fresh":"2024-08-19T14:44:20.298225+0000","last_change":"2024-08-19T14:44:15.269253+0000","last_active":"2024-08-19T14:44:20.298225+0000","last_peered":"2024-08-19T14:44:20.298225+0000","last_clean":"2024-08-19T14:44:20.298225+0000","last_became_active":"2024-08-19T14:44:15.268485+0000","last_became_peered":"2024-08-19T14:44:15.268485+0000","last_unstale":"2024-08-19T14:44:20.298225+0000","last_undegraded":"2024-08-19T14:44:20.298225+0000","last_fullsized":"2024-08-19T14:44:20.298225+0000","mapping_epoch":826,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":827,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:36.120880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,5],"acting":[10,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1","version":"70'139","reported_seq":482,"reported_epoch":830,"state":"active+clean","last_fresh":"2024-08-19T14:44:20.298056+0000","last_change":"2024-08-19T14:43:16.098691+0000","last_active":"2024-08-19T14:44:20.298056+0000","last_peered":"2024-08-19T14:44:20.298056+0000","last_clean":"2024-08-19T14:44:20.298056+0000","last_became_active":"2024-08-19T14:43:16.098337+0000","last_became_peered":"2024-08-19T14:43:16.098337+0000","last_unstale":"2024-08-19T14:44:20.298056+0000","last_undegraded":"2024-08-19T14:44:20.298056+0000","last_fullsized":"2024-08-19T14:44:20.298056+0000","mapping_epoch":800,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":801,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:41:43.141372+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:41:43.141372+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,2],"acting":[10,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.f","version":"831'349","reported_seq":369,"reported_epoch":831,"state":"active+clean","last_fresh":"2024-08-19T14:44:23.662528+0000","last_change":"2024-08-19T14:44:15.415460+0000","last_active":"2024-08-19T14:44:23.662528+0000","last_peered":"2024-08-19T14:44:23.662528+0000","last_clean":"2024-08-19T14:44:23.662528+0000","last_became_active":"2024-08-19T14:44:15.415202+0000","last_became_peered":"2024-08-19T14:44:15.415202+0000","last_unstale":"2024-08-19T14:44:23.662528+0000","last_undegraded":"2024-08-19T14:44:23.662528+0000","last_fullsized":"2024-08-19T14:44:23.662528+0000","mapping_epoch":826,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":827,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":349,"log_dups_size":0,"ondisk_log_size":349,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:20.975128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22872064,"num_objects":349,"num_object_clones":0,"num_object_copies":1047,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":349,"num_write_kb":22336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,9],"acting":[1,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"70'139","reported_seq":1056,"reported_epoch":830,"state":"active+clean","last_fresh":"2024-08-19T14:44:20.298823+0000","last_change":"2024-08-19T14:43:16.114765+0000","last_active":"2024-08-19T14:44:20.298823+0000","last_peered":"2024-08-19T14:44:20.298823+0000","last_clean":"2024-08-19T14:44:20.298823+0000","last_became_active":"2024-08-19T14:43:16.100793+0000","last_became_peered":"2024-08-19T14:43:16.100793+0000","last_unstale":"2024-08-19T14:44:20.298823+0000","last_undegraded":"2024-08-19T14:44:20.298823+0000","last_fullsized":"2024-08-19T14:44:20.298823+0000","mapping_epoch":800,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":801,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:41:35.981666+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:21:46.057624+0000","last_clean_scrub_stamp":"2024-08-19T14:41:35.981666+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,2],"acting":[10,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":18,"reported_epoch":830,"state":"active+clean","last_fresh":"2024-08-19T14:44:20.298634+0000","last_change":"2024-08-19T14:44:15.415042+0000","last_active":"2024-08-19T14:44:20.298634+0000","last_peered":"2024-08-19T14:44:20.298634+0000","last_clean":"2024-08-19T14:44:20.298634+0000","last_became_active":"2024-08-19T14:44:15.414595+0000","last_became_peered":"2024-08-19T14:44:15.414595+0000","last_unstale":"2024-08-19T14:44:20.298634+0000","last_undegraded":"2024-08-19T14:44:20.298634+0000","last_fullsized":"2024-08-19T14:44:20.298634+0000","mapping_epoch":826,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":827,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:29.023228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,9],"acting":[10,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.6","version":"70'139","reported_seq":348,"reported_epoch":830,"state":"active+clean","last_fresh":"2024-08-19T14:44:20.298094+0000","last_change":"2024-08-19T14:43:16.091622+0000","last_active":"2024-08-19T14:44:20.298094+0000","last_peered":"2024-08-19T14:44:20.298094+0000","last_clean":"2024-08-19T14:44:20.298094+0000","last_became_active":"2024-08-19T14:43:16.091256+0000","last_became_peered":"2024-08-19T14:43:16.091256+0000","last_unstale":"2024-08-19T14:44:20.298094+0000","last_undegraded":"2024-08-19T14:44:20.298094+0000","last_fullsized":"2024-08-19T14:44:20.298094+0000","mapping_epoch":800,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":801,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:41:55.664660+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:27.848870+0000","last_clean_scrub_stamp":"2024-08-19T14:41:55.664660+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:44:21.132031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13417,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":66,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":48084,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.d","version":"831'378","reported_seq":398,"reported_epoch":831,"state":"active+clean","last_fresh":"2024-08-19T14:44:23.669579+0000","last_change":"2024-08-19T14:44:15.267600+0000","last_active":"2024-08-19T14:44:23.669579+0000","last_peered":"2024-08-19T14:44:23.669579+0000","last_clean":"2024-08-19T14:44:23.669579+0000","last_became_active":"2024-08-19T14:44:15.267306+0000","last_became_peered":"2024-08-19T14:44:15.267306+0000","last_unstale":"2024-08-19T14:44:23.669579+0000","last_undegraded":"2024-08-19T14:44:23.669579+0000","last_fullsized":"2024-08-19T14:44:23.669579+0000","mapping_epoch":826,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":827,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":378,"log_dups_size":0,"ondisk_log_size":378,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:29.202494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24772608,"num_objects":378,"num_object_clones":0,"num_object_copies":1134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":378,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":378,"num_write_kb":24192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"70'139","reported_seq":438,"reported_epoch":830,"state":"active+clean","last_fresh":"2024-08-19T14:44:20.298255+0000","last_change":"2024-08-19T14:43:16.116905+0000","last_active":"2024-08-19T14:44:20.298255+0000","last_peered":"2024-08-19T14:44:20.298255+0000","last_clean":"2024-08-19T14:44:20.298255+0000","last_became_active":"2024-08-19T14:43:16.099747+0000","last_became_peered":"2024-08-19T14:43:16.099747+0000","last_unstale":"2024-08-19T14:44:20.298255+0000","last_undegraded":"2024-08-19T14:44:20.298255+0000","last_fullsized":"2024-08-19T14:44:20.298255+0000","mapping_epoch":800,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":801,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:41:48.878043+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:41:48.878043+0000","last_clean_scrub_stamp":"2024-08-19T14:41:48.878043+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,11],"acting":[10,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":16,"reported_epoch":829,"state":"active+clean","last_fresh":"2024-08-19T14:44:19.071017+0000","last_change":"2024-08-19T14:44:15.416651+0000","last_active":"2024-08-19T14:44:19.071017+0000","last_peered":"2024-08-19T14:44:19.071017+0000","last_clean":"2024-08-19T14:44:19.071017+0000","last_became_active":"2024-08-19T14:44:15.416278+0000","last_became_peered":"2024-08-19T14:44:15.416278+0000","last_unstale":"2024-08-19T14:44:19.071017+0000","last_undegraded":"2024-08-19T14:44:19.071017+0000","last_fullsized":"2024-08-19T14:44:19.071017+0000","mapping_epoch":826,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":827,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:21.705266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,10],"acting":[4,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"70'139","reported_seq":908,"reported_epoch":830,"state":"active+clean","last_fresh":"2024-08-19T14:44:20.138473+0000","last_change":"2024-08-19T14:43:08.462532+0000","last_active":"2024-08-19T14:44:20.138473+0000","last_peered":"2024-08-19T14:44:20.138473+0000","last_clean":"2024-08-19T14:44:20.138473+0000","last_became_active":"2024-08-19T14:43:08.462160+0000","last_became_peered":"2024-08-19T14:43:08.462160+0000","last_unstale":"2024-08-19T14:44:20.138473+0000","last_undegraded":"2024-08-19T14:44:20.138473+0000","last_fullsized":"2024-08-19T14:44:20.138473+0000","mapping_epoch":793,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":794,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:41:55.868614+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:41:55.868614+0000","last_clean_scrub_stamp":"2024-08-19T14:41:55.868614+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:44:25.137334+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,4],"acting":[11,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.2","version":"831'362","reported_seq":382,"reported_epoch":831,"state":"active+clean","last_fresh":"2024-08-19T14:44:23.667285+0000","last_change":"2024-08-19T14:44:15.415441+0000","last_active":"2024-08-19T14:44:23.667285+0000","last_peered":"2024-08-19T14:44:23.667285+0000","last_clean":"2024-08-19T14:44:23.667285+0000","last_became_active":"2024-08-19T14:44:15.415137+0000","last_became_peered":"2024-08-19T14:44:15.415137+0000","last_unstale":"2024-08-19T14:44:23.667285+0000","last_undegraded":"2024-08-19T14:44:23.667285+0000","last_fullsized":"2024-08-19T14:44:23.667285+0000","mapping_epoch":826,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":827,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":362,"log_dups_size":0,"ondisk_log_size":362,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:28.806060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23724032,"num_objects":362,"num_object_clones":0,"num_object_copies":1086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":362,"num_write_kb":23168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,11],"acting":[1,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"70'139","reported_seq":1070,"reported_epoch":829,"state":"active+clean","last_fresh":"2024-08-19T14:44:19.070940+0000","last_change":"2024-08-19T14:42:39.072075+0000","last_active":"2024-08-19T14:44:19.070940+0000","last_peered":"2024-08-19T14:44:19.070940+0000","last_clean":"2024-08-19T14:44:19.070940+0000","last_became_active":"2024-08-19T14:42:39.071701+0000","last_became_peered":"2024-08-19T14:42:39.071701+0000","last_unstale":"2024-08-19T14:44:19.070940+0000","last_undegraded":"2024-08-19T14:44:19.070940+0000","last_fullsized":"2024-08-19T14:44:19.070940+0000","mapping_epoch":778,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":779,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:41:55.751274+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:28.864870+0000","last_clean_scrub_stamp":"2024-08-19T14:41:55.751274+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:44:23.749452+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,2],"acting":[4,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":18,"reported_epoch":830,"state":"active+clean","last_fresh":"2024-08-19T14:44:20.298133+0000","last_change":"2024-08-19T14:44:15.416094+0000","last_active":"2024-08-19T14:44:20.298133+0000","last_peered":"2024-08-19T14:44:20.298133+0000","last_clean":"2024-08-19T14:44:20.298133+0000","last_became_active":"2024-08-19T14:44:15.415731+0000","last_became_peered":"2024-08-19T14:44:15.415731+0000","last_unstale":"2024-08-19T14:44:20.298133+0000","last_undegraded":"2024-08-19T14:44:20.298133+0000","last_fullsized":"2024-08-19T14:44:20.298133+0000","mapping_epoch":826,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":827,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:30.021574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,2],"acting":[10,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.9","version":"70'139","reported_seq":1624,"reported_epoch":831,"state":"active+clean","last_fresh":"2024-08-19T14:44:23.151778+0000","last_change":"2024-08-19T14:43:08.469441+0000","last_active":"2024-08-19T14:44:23.151778+0000","last_peered":"2024-08-19T14:44:23.151778+0000","last_clean":"2024-08-19T14:44:23.151778+0000","last_became_active":"2024-08-19T14:43:08.469067+0000","last_became_peered":"2024-08-19T14:43:08.469067+0000","last_unstale":"2024-08-19T14:44:23.151778+0000","last_undegraded":"2024-08-19T14:44:23.151778+0000","last_fullsized":"2024-08-19T14:44:23.151778+0000","mapping_epoch":793,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":794,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:42:00.561080+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:42:00.561080+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,5],"acting":[9,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.0","version":"831'368","reported_seq":387,"reported_epoch":831,"state":"active+clean","last_fresh":"2024-08-19T14:44:23.676686+0000","last_change":"2024-08-19T14:44:15.066444+0000","last_active":"2024-08-19T14:44:23.676686+0000","last_peered":"2024-08-19T14:44:23.676686+0000","last_clean":"2024-08-19T14:44:23.676686+0000","last_became_active":"2024-08-19T14:44:15.065971+0000","last_became_peered":"2024-08-19T14:44:15.065971+0000","last_unstale":"2024-08-19T14:44:23.676686+0000","last_undegraded":"2024-08-19T14:44:23.676686+0000","last_fullsized":"2024-08-19T14:44:23.676686+0000","mapping_epoch":826,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":827,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":368,"log_dups_size":0,"ondisk_log_size":368,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:29.855885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24117248,"num_objects":368,"num_object_clones":0,"num_object_copies":1104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":368,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":368,"num_write_kb":23552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,8],"acting":[1,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"70'139","reported_seq":582,"reported_epoch":830,"state":"active+clean","last_fresh":"2024-08-19T14:44:20.298179+0000","last_change":"2024-08-19T14:42:08.692385+0000","last_active":"2024-08-19T14:44:20.298179+0000","last_peered":"2024-08-19T14:44:20.298179+0000","last_clean":"2024-08-19T14:44:20.298179+0000","last_became_active":"2024-08-19T14:40:52.455488+0000","last_became_peered":"2024-08-19T14:40:52.455488+0000","last_unstale":"2024-08-19T14:44:20.298179+0000","last_undegraded":"2024-08-19T14:44:20.298179+0000","last_fullsized":"2024-08-19T14:44:20.298179+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:42:08.692264+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:42:08.692264+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:44:20.824187+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,2],"acting":[8,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":18,"reported_epoch":830,"state":"active+clean","last_fresh":"2024-08-19T14:44:20.298377+0000","last_change":"2024-08-19T14:44:15.414974+0000","last_active":"2024-08-19T14:44:20.298377+0000","last_peered":"2024-08-19T14:44:20.298377+0000","last_clean":"2024-08-19T14:44:20.298377+0000","last_became_active":"2024-08-19T14:44:15.414623+0000","last_became_peered":"2024-08-19T14:44:15.414623+0000","last_unstale":"2024-08-19T14:44:20.298377+0000","last_undegraded":"2024-08-19T14:44:20.298377+0000","last_fullsized":"2024-08-19T14:44:20.298377+0000","mapping_epoch":826,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":827,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:18.725495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.b","version":"70'139","reported_seq":1360,"reported_epoch":830,"state":"active+clean","last_fresh":"2024-08-19T14:44:20.138470+0000","last_change":"2024-08-19T14:42:39.077445+0000","last_active":"2024-08-19T14:44:20.138470+0000","last_peered":"2024-08-19T14:44:20.138470+0000","last_clean":"2024-08-19T14:44:20.138470+0000","last_became_active":"2024-08-19T14:42:39.077214+0000","last_became_peered":"2024-08-19T14:42:39.077214+0000","last_unstale":"2024-08-19T14:44:20.138470+0000","last_undegraded":"2024-08-19T14:44:20.138470+0000","last_fullsized":"2024-08-19T14:44:20.138470+0000","mapping_epoch":778,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":779,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:41:50.892343+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:41:50.892343+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:44:25.137316+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,2],"acting":[11,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":18,"reported_epoch":830,"state":"active+clean","last_fresh":"2024-08-19T14:44:20.298451+0000","last_change":"2024-08-19T14:44:15.268309+0000","last_active":"2024-08-19T14:44:20.298451+0000","last_peered":"2024-08-19T14:44:20.298451+0000","last_clean":"2024-08-19T14:44:20.298451+0000","last_became_active":"2024-08-19T14:44:15.267907+0000","last_became_peered":"2024-08-19T14:44:15.267907+0000","last_unstale":"2024-08-19T14:44:20.298451+0000","last_undegraded":"2024-08-19T14:44:20.298451+0000","last_fullsized":"2024-08-19T14:44:20.298451+0000","mapping_epoch":826,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":827,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:42.468389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,1],"acting":[10,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.e","version":"70'139","reported_seq":1557,"reported_epoch":829,"state":"active+clean","last_fresh":"2024-08-19T14:44:19.071187+0000","last_change":"2024-08-19T14:43:08.463874+0000","last_active":"2024-08-19T14:44:19.071187+0000","last_peered":"2024-08-19T14:44:19.071187+0000","last_clean":"2024-08-19T14:44:19.071187+0000","last_became_active":"2024-08-19T14:43:08.463102+0000","last_became_peered":"2024-08-19T14:43:08.463102+0000","last_unstale":"2024-08-19T14:44:19.071187+0000","last_undegraded":"2024-08-19T14:44:19.071187+0000","last_fullsized":"2024-08-19T14:44:19.071187+0000","mapping_epoch":793,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":794,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:41:46.892116+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:40:03.764672+0000","last_clean_scrub_stamp":"2024-08-19T14:41:46.892116+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:44:23.749470+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,1],"acting":[4,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":18,"reported_epoch":830,"state":"active+clean","last_fresh":"2024-08-19T14:44:20.298629+0000","last_change":"2024-08-19T14:44:15.068003+0000","last_active":"2024-08-19T14:44:20.298629+0000","last_peered":"2024-08-19T14:44:20.298629+0000","last_clean":"2024-08-19T14:44:20.298629+0000","last_became_active":"2024-08-19T14:44:15.067635+0000","last_became_peered":"2024-08-19T14:44:15.067635+0000","last_unstale":"2024-08-19T14:44:20.298629+0000","last_undegraded":"2024-08-19T14:44:20.298629+0000","last_fullsized":"2024-08-19T14:44:20.298629+0000","mapping_epoch":826,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":827,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:40.928253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,1],"acting":[10,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.f","version":"70'139","reported_seq":1300,"reported_epoch":829,"state":"active+clean","last_fresh":"2024-08-19T14:44:19.071043+0000","last_change":"2024-08-19T14:42:03.775080+0000","last_active":"2024-08-19T14:44:19.071043+0000","last_peered":"2024-08-19T14:44:19.071043+0000","last_clean":"2024-08-19T14:44:19.071043+0000","last_became_active":"2024-08-19T14:40:52.446953+0000","last_became_peered":"2024-08-19T14:40:52.446953+0000","last_unstale":"2024-08-19T14:44:19.071043+0000","last_undegraded":"2024-08-19T14:44:19.071043+0000","last_fullsized":"2024-08-19T14:44:19.071043+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:42:03.774987+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:42:03.774987+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:44:23.749503+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,8],"acting":[4,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.4","version":"831'333","reported_seq":353,"reported_epoch":831,"state":"active+clean","last_fresh":"2024-08-19T14:44:23.482779+0000","last_change":"2024-08-19T14:44:15.066789+0000","last_active":"2024-08-19T14:44:23.482779+0000","last_peered":"2024-08-19T14:44:23.482779+0000","last_clean":"2024-08-19T14:44:23.482779+0000","last_became_active":"2024-08-19T14:44:15.065981+0000","last_became_peered":"2024-08-19T14:44:15.065981+0000","last_unstale":"2024-08-19T14:44:23.482779+0000","last_undegraded":"2024-08-19T14:44:23.482779+0000","last_fullsized":"2024-08-19T14:44:23.482779+0000","mapping_epoch":826,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":827,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":333,"log_dups_size":0,"ondisk_log_size":333,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:39.692151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21823488,"num_objects":333,"num_object_clones":0,"num_object_copies":999,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":333,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":333,"num_write_kb":21312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.c","version":"70'139","reported_seq":606,"reported_epoch":831,"state":"active+clean","last_fresh":"2024-08-19T14:44:23.149578+0000","last_change":"2024-08-19T14:42:39.073450+0000","last_active":"2024-08-19T14:44:23.149578+0000","last_peered":"2024-08-19T14:44:23.149578+0000","last_clean":"2024-08-19T14:44:23.149578+0000","last_became_active":"2024-08-19T14:42:39.073086+0000","last_became_peered":"2024-08-19T14:42:39.073086+0000","last_unstale":"2024-08-19T14:44:23.149578+0000","last_undegraded":"2024-08-19T14:44:23.149578+0000","last_fullsized":"2024-08-19T14:44:23.149578+0000","mapping_epoch":778,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":779,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:41:37.152446+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:22:52.164586+0000","last_clean_scrub_stamp":"2024-08-19T14:41:37.152446+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:44:23.807441+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,5],"acting":[1,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.5","version":"830'187","reported_seq":205,"reported_epoch":830,"state":"active+clean","last_fresh":"2024-08-19T14:44:22.505962+0000","last_change":"2024-08-19T14:44:15.268306+0000","last_active":"2024-08-19T14:44:22.505962+0000","last_peered":"2024-08-19T14:44:22.505962+0000","last_clean":"2024-08-19T14:44:22.505962+0000","last_became_active":"2024-08-19T14:44:15.267867+0000","last_became_peered":"2024-08-19T14:44:15.267867+0000","last_unstale":"2024-08-19T14:44:22.505962+0000","last_undegraded":"2024-08-19T14:44:22.505962+0000","last_fullsized":"2024-08-19T14:44:22.505962+0000","mapping_epoch":826,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":827,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:31.746018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12255232,"num_objects":187,"num_object_clones":0,"num_object_copies":561,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":187,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":187,"num_write_kb":11968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,1],"acting":[11,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.d","version":"70'139","reported_seq":391,"reported_epoch":829,"state":"active+clean","last_fresh":"2024-08-19T14:44:19.071040+0000","last_change":"2024-08-19T14:43:08.459939+0000","last_active":"2024-08-19T14:44:19.071040+0000","last_peered":"2024-08-19T14:44:19.071040+0000","last_clean":"2024-08-19T14:44:19.071040+0000","last_became_active":"2024-08-19T14:43:08.459577+0000","last_became_peered":"2024-08-19T14:43:08.459577+0000","last_unstale":"2024-08-19T14:44:19.071040+0000","last_undegraded":"2024-08-19T14:44:19.071040+0000","last_fullsized":"2024-08-19T14:44:19.071040+0000","mapping_epoch":793,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":794,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:42:09.549880+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:24:50.276408+0000","last_clean_scrub_stamp":"2024-08-19T14:42:09.549880+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:44:19.765969+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,8],"acting":[4,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.12","version":"70'139","reported_seq":1335,"reported_epoch":831,"state":"active+clean","last_fresh":"2024-08-19T14:44:23.150501+0000","last_change":"2024-08-19T14:43:08.458336+0000","last_active":"2024-08-19T14:44:23.150501+0000","last_peered":"2024-08-19T14:44:23.150501+0000","last_clean":"2024-08-19T14:44:23.150501+0000","last_became_active":"2024-08-19T14:43:08.457897+0000","last_became_peered":"2024-08-19T14:43:08.457897+0000","last_unstale":"2024-08-19T14:44:23.150501+0000","last_undegraded":"2024-08-19T14:44:23.150501+0000","last_fullsized":"2024-08-19T14:44:23.150501+0000","mapping_epoch":793,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":794,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:41:49.815826+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:41:49.815826+0000","last_clean_scrub_stamp":"2024-08-19T14:41:49.815826+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:44:23.807480+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.13","version":"70'139","reported_seq":1886,"reported_epoch":831,"state":"active+clean","last_fresh":"2024-08-19T14:44:23.149159+0000","last_change":"2024-08-19T14:43:16.087256+0000","last_active":"2024-08-19T14:44:23.149159+0000","last_peered":"2024-08-19T14:44:23.149159+0000","last_clean":"2024-08-19T14:44:23.149159+0000","last_became_active":"2024-08-19T14:43:16.086968+0000","last_became_peered":"2024-08-19T14:43:16.086968+0000","last_unstale":"2024-08-19T14:44:23.149159+0000","last_undegraded":"2024-08-19T14:44:23.149159+0000","last_fullsized":"2024-08-19T14:44:23.149159+0000","mapping_epoch":800,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":801,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:41:55.847727+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:33:35.786105+0000","last_clean_scrub_stamp":"2024-08-19T14:41:55.847727+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:44:23.807549+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.10","version":"70'139","reported_seq":808,"reported_epoch":830,"state":"active+clean","last_fresh":"2024-08-19T14:44:20.298365+0000","last_change":"2024-08-19T14:43:16.114296+0000","last_active":"2024-08-19T14:44:20.298365+0000","last_peered":"2024-08-19T14:44:20.298365+0000","last_clean":"2024-08-19T14:44:20.298365+0000","last_became_active":"2024-08-19T14:43:16.113982+0000","last_became_peered":"2024-08-19T14:43:16.113982+0000","last_unstale":"2024-08-19T14:44:20.298365+0000","last_undegraded":"2024-08-19T14:44:20.298365+0000","last_fullsized":"2024-08-19T14:44:20.298365+0000","mapping_epoch":800,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":801,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:40:32.035620+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:34:58.616646+0000","last_clean_scrub_stamp":"2024-08-19T14:40:32.035620+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:44:21.132071+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,8],"acting":[10,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.11","version":"70'139","reported_seq":398,"reported_epoch":830,"state":"active+clean","last_fresh":"2024-08-19T14:44:20.298379+0000","last_change":"2024-08-19T14:42:39.064912+0000","last_active":"2024-08-19T14:44:20.298379+0000","last_peered":"2024-08-19T14:44:20.298379+0000","last_clean":"2024-08-19T14:44:20.298379+0000","last_became_active":"2024-08-19T14:42:39.064403+0000","last_became_peered":"2024-08-19T14:42:39.064403+0000","last_unstale":"2024-08-19T14:44:20.298379+0000","last_undegraded":"2024-08-19T14:44:20.298379+0000","last_fullsized":"2024-08-19T14:44:20.298379+0000","mapping_epoch":778,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":779,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:41:47.147376+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:41:47.147376+0000","last_clean_scrub_stamp":"2024-08-19T14:41:47.147376+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:44:22.883505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":5,"num_write":9,"num_write_kb":134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":96169,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,1],"acting":[8,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"70'139","reported_seq":1504,"reported_epoch":831,"state":"active+clean","last_fresh":"2024-08-19T14:44:23.150228+0000","last_change":"2024-08-19T14:43:04.925737+0000","last_active":"2024-08-19T14:44:23.150228+0000","last_peered":"2024-08-19T14:44:23.150228+0000","last_clean":"2024-08-19T14:44:23.150228+0000","last_became_active":"2024-08-19T14:42:39.066670+0000","last_became_peered":"2024-08-19T14:42:39.066670+0000","last_unstale":"2024-08-19T14:44:23.150228+0000","last_undegraded":"2024-08-19T14:44:23.150228+0000","last_fullsized":"2024-08-19T14:44:23.150228+0000","mapping_epoch":778,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":779,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:43:04.925661+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:28:34.216370+0000","last_clean_scrub_stamp":"2024-08-19T14:43:04.925661+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:44:23.807565+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,5],"acting":[1,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"70'139","reported_seq":1644,"reported_epoch":830,"state":"active+clean","last_fresh":"2024-08-19T14:44:20.298729+0000","last_change":"2024-08-19T14:43:16.127472+0000","last_active":"2024-08-19T14:44:20.298729+0000","last_peered":"2024-08-19T14:44:20.298729+0000","last_clean":"2024-08-19T14:44:20.298729+0000","last_became_active":"2024-08-19T14:43:16.123238+0000","last_became_peered":"2024-08-19T14:43:16.123238+0000","last_unstale":"2024-08-19T14:44:20.298729+0000","last_undegraded":"2024-08-19T14:44:20.298729+0000","last_fullsized":"2024-08-19T14:44:20.298729+0000","mapping_epoch":800,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":801,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:42:14.678271+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:42:14.678271+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:44:20.824207+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.14","version":"70'139","reported_seq":1069,"reported_epoch":831,"state":"active+clean","last_fresh":"2024-08-19T14:44:23.149269+0000","last_change":"2024-08-19T14:43:08.454671+0000","last_active":"2024-08-19T14:44:23.149269+0000","last_peered":"2024-08-19T14:44:23.149269+0000","last_clean":"2024-08-19T14:44:23.149269+0000","last_became_active":"2024-08-19T14:43:08.454350+0000","last_became_peered":"2024-08-19T14:43:08.454350+0000","last_unstale":"2024-08-19T14:44:23.149269+0000","last_undegraded":"2024-08-19T14:44:23.149269+0000","last_fullsized":"2024-08-19T14:44:23.149269+0000","mapping_epoch":793,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":794,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:42:06.543798+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:42:06.543798+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:44:23.807531+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,9],"acting":[1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"70'139","reported_seq":395,"reported_epoch":830,"state":"active+clean","last_fresh":"2024-08-19T14:44:20.298843+0000","last_change":"2024-08-19T14:43:08.457011+0000","last_active":"2024-08-19T14:44:20.298843+0000","last_peered":"2024-08-19T14:44:20.298843+0000","last_clean":"2024-08-19T14:44:20.298843+0000","last_became_active":"2024-08-19T14:43:08.456550+0000","last_became_peered":"2024-08-19T14:43:08.456550+0000","last_unstale":"2024-08-19T14:44:20.298843+0000","last_undegraded":"2024-08-19T14:44:20.298843+0000","last_fullsized":"2024-08-19T14:44:20.298843+0000","mapping_epoch":793,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":794,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:42:15.607984+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:20:09.180923+0000","last_clean_scrub_stamp":"2024-08-19T14:42:15.607984+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:44:22.883525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13418,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":48085,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,1],"acting":[8,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"70'139","reported_seq":1492,"reported_epoch":829,"state":"active+clean","last_fresh":"2024-08-19T14:44:19.070809+0000","last_change":"2024-08-19T14:43:11.472105+0000","last_active":"2024-08-19T14:44:19.070809+0000","last_peered":"2024-08-19T14:44:19.070809+0000","last_clean":"2024-08-19T14:44:19.070809+0000","last_became_active":"2024-08-19T14:43:11.471324+0000","last_became_peered":"2024-08-19T14:43:11.471324+0000","last_unstale":"2024-08-19T14:44:19.070809+0000","last_undegraded":"2024-08-19T14:44:19.070809+0000","last_fullsized":"2024-08-19T14:44:19.070809+0000","mapping_epoch":795,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":796,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:41:51.832284+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:41:51.832284+0000","last_clean_scrub_stamp":"2024-08-19T14:41:51.832284+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:44:23.749487+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,11],"acting":[4,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1b","version":"70'139","reported_seq":1876,"reported_epoch":830,"state":"active+clean","last_fresh":"2024-08-19T14:44:20.298382+0000","last_change":"2024-08-19T14:42:39.066054+0000","last_active":"2024-08-19T14:44:20.298382+0000","last_peered":"2024-08-19T14:44:20.298382+0000","last_clean":"2024-08-19T14:44:20.298382+0000","last_became_active":"2024-08-19T14:42:39.065604+0000","last_became_peered":"2024-08-19T14:42:39.065604+0000","last_unstale":"2024-08-19T14:44:20.298382+0000","last_undegraded":"2024-08-19T14:44:20.298382+0000","last_fullsized":"2024-08-19T14:44:20.298382+0000","mapping_epoch":778,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":779,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:40:43.238820+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:37:59.510697+0000","last_clean_scrub_stamp":"2024-08-19T14:40:43.238820+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:44:20.824165+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,2],"acting":[8,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.18","version":"70'139","reported_seq":861,"reported_epoch":831,"state":"active+clean","last_fresh":"2024-08-19T14:44:23.149117+0000","last_change":"2024-08-19T14:43:08.458371+0000","last_active":"2024-08-19T14:44:23.149117+0000","last_peered":"2024-08-19T14:44:23.149117+0000","last_clean":"2024-08-19T14:44:23.149117+0000","last_became_active":"2024-08-19T14:43:08.457799+0000","last_became_peered":"2024-08-19T14:43:08.457799+0000","last_unstale":"2024-08-19T14:44:23.149117+0000","last_undegraded":"2024-08-19T14:44:23.149117+0000","last_fullsized":"2024-08-19T14:44:23.149117+0000","mapping_epoch":793,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":794,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:41:53.603570+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:27:33.084161+0000","last_clean_scrub_stamp":"2024-08-19T14:41:53.603570+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:44:23.807514+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.19","version":"70'139","reported_seq":720,"reported_epoch":830,"state":"active+clean","last_fresh":"2024-08-19T14:44:20.298271+0000","last_change":"2024-08-19T14:43:16.097396+0000","last_active":"2024-08-19T14:44:20.298271+0000","last_peered":"2024-08-19T14:44:20.298271+0000","last_clean":"2024-08-19T14:44:20.298271+0000","last_became_active":"2024-08-19T14:43:16.096717+0000","last_became_peered":"2024-08-19T14:43:16.096717+0000","last_unstale":"2024-08-19T14:44:20.298271+0000","last_undegraded":"2024-08-19T14:44:20.298271+0000","last_fullsized":"2024-08-19T14:44:20.298271+0000","mapping_epoch":800,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":801,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:41:36.108239+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:41:36.108239+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:44:21.132052+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,8],"acting":[10,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1e","version":"70'139","reported_seq":889,"reported_epoch":830,"state":"active+clean","last_fresh":"2024-08-19T14:44:20.138563+0000","last_change":"2024-08-19T14:43:16.126022+0000","last_active":"2024-08-19T14:44:20.138563+0000","last_peered":"2024-08-19T14:44:20.138563+0000","last_clean":"2024-08-19T14:44:20.138563+0000","last_became_active":"2024-08-19T14:43:16.124258+0000","last_became_peered":"2024-08-19T14:43:16.124258+0000","last_unstale":"2024-08-19T14:44:20.138563+0000","last_undegraded":"2024-08-19T14:44:20.138563+0000","last_fullsized":"2024-08-19T14:44:20.138563+0000","mapping_epoch":800,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":801,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:41:58.896684+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:27:04.780939+0000","last_clean_scrub_stamp":"2024-08-19T14:41:58.896684+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:44:25.137350+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,10],"acting":[11,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1c","version":"70'139","reported_seq":1629,"reported_epoch":830,"state":"active+clean","last_fresh":"2024-08-19T14:44:20.298558+0000","last_change":"2024-08-19T14:43:04.811401+0000","last_active":"2024-08-19T14:44:20.298558+0000","last_peered":"2024-08-19T14:44:20.298558+0000","last_clean":"2024-08-19T14:44:20.298558+0000","last_became_active":"2024-08-19T14:30:47.908866+0000","last_became_peered":"2024-08-19T14:30:47.908866+0000","last_unstale":"2024-08-19T14:44:20.298558+0000","last_undegraded":"2024-08-19T14:44:20.298558+0000","last_fullsized":"2024-08-19T14:44:20.298558+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:43:04.811320+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:43:04.811320+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,11],"acting":[8,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1d","version":"70'139","reported_seq":1532,"reported_epoch":831,"state":"active+clean","last_fresh":"2024-08-19T14:44:23.150111+0000","last_change":"2024-08-19T14:41:38.948464+0000","last_active":"2024-08-19T14:44:23.150111+0000","last_peered":"2024-08-19T14:44:23.150111+0000","last_clean":"2024-08-19T14:44:23.150111+0000","last_became_active":"2024-08-19T14:35:11.540488+0000","last_became_peered":"2024-08-19T14:35:11.540488+0000","last_unstale":"2024-08-19T14:44:23.150111+0000","last_undegraded":"2024-08-19T14:44:23.150111+0000","last_fullsized":"2024-08-19T14:44:23.150111+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:41:38.947168+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:41:38.947168+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:44:23.807462+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":167247872,"num_objects":2552,"num_object_clones":0,"num_object_copies":7656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2552,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2552,"num_write_kb":163328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":528678912,"data_stored":528678912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2552,"ondisk_log_size":2552,"up":48,"acting":48,"num_store_stats":8},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":670873,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":15,"num_write":25,"num_write_kb":400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":841477,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2252800,"data_stored":2242144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187852,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1115596,"kb_used_data":47364,"kb_used_omap":1,"kb_used_meta":1068222,"kb_avail":92629556,"statfs":{"total":95995035648,"available":94852665344,"internally_reserved":0,"allocated":48500736,"data_stored":46107738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":1093859780},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":539,"seq":2314987372719,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":147076,"kb_used_data":6840,"kb_used_omap":4,"kb_used_meta":140219,"kb_avail":93598076,"statfs":{"total":95995035648,"available":95844429824,"internally_reserved":0,"allocated":7004160,"data_stored":4615409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4407,"internal_metadata":143584969},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":74,"seq":317827580310,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1218188,"kb_used_data":136068,"kb_used_omap":1,"kb_used_meta":1082110,"kb_avail":92526964,"statfs":{"total":95995035648,"available":94747611136,"internally_reserved":0,"allocated":139333632,"data_stored":136940634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":1108081089},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874652,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":958468,"kb_used_data":6840,"kb_used_omap":1,"kb_used_meta":951614,"kb_avail":92786684,"statfs":{"total":95995035648,"available":95013564416,"internally_reserved":0,"allocated":7004160,"data_stored":4615409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":974453189},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":0,"seq":244813136287,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":206158430627,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692262,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1151628,"kb_used_data":33408,"kb_used_omap":1,"kb_used_meta":1118206,"kb_avail":92593524,"statfs":{"total":95995035648,"available":94815768576,"internally_reserved":0,"allocated":34209792,"data_stored":31817489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1145043397},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986602,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1240324,"kb_used_data":36024,"kb_used_omap":1,"kb_used_meta":1204286,"kb_avail":92504828,"statfs":{"total":95995035648,"available":94724943872,"internally_reserved":0,"allocated":36888576,"data_stored":34499825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1233189317},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":111669150129,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":634,"seq":2723009265775,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":516684,"kb_used_data":7040,"kb_used_omap":4,"kb_used_meta":509627,"kb_avail":93228468,"statfs":{"total":95995035648,"available":95465951232,"internally_reserved":0,"allocated":7208960,"data_stored":4816657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4407,"internal_metadata":521858761},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575290,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1450904,"kb_used_data":156492,"kb_used_omap":1,"kb_used_meta":1294398,"kb_avail":92294248,"statfs":{"total":95995035648,"available":94509309952,"internally_reserved":0,"allocated":160247808,"data_stored":157851258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1325464005},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":34359738814,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":234487808,"data_stored":234487808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":56229888,"data_stored":56229888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":174784512,"data_stored":174784512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":63176704,"data_stored":63176704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T14:44:28.017 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-08-19T14:44:28.017 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-19T14:44:28.017 INFO:tasks.thrashosds.thrasher:Removing osd 5, in_osds are: [1, 4, 5, 8, 2, 11, 9, 10] 2024-08-19T14:44:28.018 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd out 5 2024-08-19T14:44:28.035 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:44:28.060 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:44:28.072 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:28.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:28.073 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:44:28.174 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:44:28.198 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:28.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:28.199 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:44:28.300 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:44:28.308 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 7 16 18955 18939 169.077 151.938 0.0072086 0.00590808 2024-08-19T14:44:28.337 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:28.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:28.339 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:44:28.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:28 smithi031 ceph-mon[16336]: pgmap v1968: 47 pgs: 47 active+clean; 508 MiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 81 MiB/s wr, 1.29k op/s 2024-08-19T14:44:28.367 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:28 smithi032 ceph-mon[20004]: pgmap v1968: 47 pgs: 47 active+clean; 508 MiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 81 MiB/s wr, 1.29k op/s 2024-08-19T14:44:28.439 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:44:28.463 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:28.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:28.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:44:28.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:28 smithi045 ceph-mon[20143]: pgmap v1968: 47 pgs: 47 active+clean; 508 MiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 81 MiB/s wr, 1.29k op/s 2024-08-19T14:44:28.565 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:44:28.586 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:28.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:28.587 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:44:28.688 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:44:28.713 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:28.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:28.714 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:44:28.815 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:44:28.836 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:28.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:28.838 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:44:28.939 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:44:28.962 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:28.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:28.963 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:44:29.064 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:44:29.090 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:29.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:29.091 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:44:29.192 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:44:29.228 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:29.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:29.229 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:44:29.308 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 8 16 21422 21406 167.215 154.188 0.00312677 0.00594078 2024-08-19T14:44:29.330 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:44:29.358 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:29.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:29.359 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:44:29.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:29 smithi031 ceph-mon[16336]: 1.8 scrub starts 2024-08-19T14:44:29.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:29 smithi031 ceph-mon[16336]: 1.8 scrub ok 2024-08-19T14:44:29.392 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-19T14:44:29.461 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:44:29.481 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:29.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:29.482 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:44:29.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:29 smithi032 ceph-mon[20004]: 1.8 scrub starts 2024-08-19T14:44:29.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:29 smithi032 ceph-mon[20004]: 1.8 scrub ok 2024-08-19T14:44:29.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:29 smithi045 ceph-mon[20143]: 1.8 scrub starts 2024-08-19T14:44:29.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:29 smithi045 ceph-mon[20143]: 1.8 scrub ok 2024-08-19T14:44:29.584 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:44:29.607 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:29.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:29.608 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:44:29.709 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:44:29.747 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:29.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:29.748 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:44:29.849 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:44:29.886 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:29.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:29.887 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:44:29.988 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:44:30.016 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:30.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:30.017 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:44:30.118 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:44:30.141 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:30.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:30.142 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:44:30.243 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:44:30.272 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:30.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:30.273 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:44:30.308 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 9 15 23409 23394 162.44 124.25 0.00555341 0.00615015 2024-08-19T14:44:30.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:30 smithi031 ceph-mon[16336]: pgmap v1969: 47 pgs: 47 active+clean; 615 MiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-08-19T14:44:30.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:30 smithi031 ceph-mon[16336]: 1.a deep-scrub starts 2024-08-19T14:44:30.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:30 smithi031 ceph-mon[16336]: 1.17 scrub starts 2024-08-19T14:44:30.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:30 smithi031 ceph-mon[16336]: 1.a deep-scrub ok 2024-08-19T14:44:30.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:30 smithi031 ceph-mon[16336]: 1.17 scrub ok 2024-08-19T14:44:30.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:30 smithi031 ceph-mon[16336]: 1.5 scrub starts 2024-08-19T14:44:30.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:30 smithi031 ceph-mon[16336]: 1.5 scrub ok 2024-08-19T14:44:30.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:30 smithi031 ceph-mon[16336]: 1.b scrub starts 2024-08-19T14:44:30.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:30 smithi031 ceph-mon[16336]: 1.b scrub ok 2024-08-19T14:44:30.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:30 smithi031 ceph-mon[16336]: 1.1d scrub starts 2024-08-19T14:44:30.368 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:30 smithi031 ceph-mon[16336]: 1.1d scrub ok 2024-08-19T14:44:30.374 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:44:30.396 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:30.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:30.397 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:44:30.498 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:44:30.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:30 smithi032 ceph-mon[20004]: pgmap v1969: 47 pgs: 47 active+clean; 615 MiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-08-19T14:44:30.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:30 smithi032 ceph-mon[20004]: 1.a deep-scrub starts 2024-08-19T14:44:30.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:30 smithi032 ceph-mon[20004]: 1.17 scrub starts 2024-08-19T14:44:30.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:30 smithi032 ceph-mon[20004]: 1.a deep-scrub ok 2024-08-19T14:44:30.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:30 smithi032 ceph-mon[20004]: 1.17 scrub ok 2024-08-19T14:44:30.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:30 smithi032 ceph-mon[20004]: 1.5 scrub starts 2024-08-19T14:44:30.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:30 smithi032 ceph-mon[20004]: 1.5 scrub ok 2024-08-19T14:44:30.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:30 smithi032 ceph-mon[20004]: 1.b scrub starts 2024-08-19T14:44:30.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:30 smithi032 ceph-mon[20004]: 1.b scrub ok 2024-08-19T14:44:30.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:30 smithi032 ceph-mon[20004]: 1.1d scrub starts 2024-08-19T14:44:30.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:30 smithi032 ceph-mon[20004]: 1.1d scrub ok 2024-08-19T14:44:30.525 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:30.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:30.526 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:44:30.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:30 smithi045 ceph-mon[20143]: pgmap v1969: 47 pgs: 47 active+clean; 615 MiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-08-19T14:44:30.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:30 smithi045 ceph-mon[20143]: 1.a deep-scrub starts 2024-08-19T14:44:30.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:30 smithi045 ceph-mon[20143]: 1.17 scrub starts 2024-08-19T14:44:30.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:30 smithi045 ceph-mon[20143]: 1.a deep-scrub ok 2024-08-19T14:44:30.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:30 smithi045 ceph-mon[20143]: 1.17 scrub ok 2024-08-19T14:44:30.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:30 smithi045 ceph-mon[20143]: 1.5 scrub starts 2024-08-19T14:44:30.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:30 smithi045 ceph-mon[20143]: 1.5 scrub ok 2024-08-19T14:44:30.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:30 smithi045 ceph-mon[20143]: 1.b scrub starts 2024-08-19T14:44:30.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:30 smithi045 ceph-mon[20143]: 1.b scrub ok 2024-08-19T14:44:30.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:30 smithi045 ceph-mon[20143]: 1.1d scrub starts 2024-08-19T14:44:30.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:30 smithi045 ceph-mon[20143]: 1.1d scrub ok 2024-08-19T14:44:30.626 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:44:30.647 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:30.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:30.648 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:44:30.749 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:44:30.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:30.771 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:30.771 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:44:30.872 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:44:30.873 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:44:30.873 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:44:30.898 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:44:30.899 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:44:30.908 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:30.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:30.909 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:44:30.925 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:44:30.930 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:44:30.930 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:44:30.964 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:44:30.965 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:44:30.986 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:44:30.995 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:44:30.995 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:44:31.010 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:44:31.025 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:44:31.026 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:44:31.035 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:31.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:31.036 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:44:31.052 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:44:31.052 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:44:31.077 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:44:31.077 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:44:31.102 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:44:31.103 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:44:31.130 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:44:31.130 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:44:31.137 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:44:31.147 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:44:31.147 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:44:31.159 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:31.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:31.160 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:44:31.163 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:44:31.164 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:44:31.165 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-08-19T14:44:31.261 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:44:31.278 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:31 smithi031 ceph-mon[16336]: 1.e scrub starts 2024-08-19T14:44:31.279 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:31 smithi031 ceph-mon[16336]: 1.e scrub ok 2024-08-19T14:44:31.279 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:31 smithi031 ceph-mon[16336]: 1.12 scrub starts 2024-08-19T14:44:31.279 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:31 smithi031 ceph-mon[16336]: 1.12 scrub ok 2024-08-19T14:44:31.279 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:31 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2495790315' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:44:31.283 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:31.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:31.284 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:44:31.309 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 10 16 24987 24971 156.051 98.5625 0.00172542 0.00627763 2024-08-19T14:44:31.385 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:44:31.422 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:31.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:31.424 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:44:31.442 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:44:31.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:31 smithi032 ceph-mon[20004]: 1.e scrub starts 2024-08-19T14:44:31.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:31 smithi032 ceph-mon[20004]: 1.e scrub ok 2024-08-19T14:44:31.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:31 smithi032 ceph-mon[20004]: 1.12 scrub starts 2024-08-19T14:44:31.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:31 smithi032 ceph-mon[20004]: 1.12 scrub ok 2024-08-19T14:44:31.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:31 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2495790315' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:44:31.525 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:44:31.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:31 smithi045 ceph-mon[20143]: 1.e scrub starts 2024-08-19T14:44:31.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:31 smithi045 ceph-mon[20143]: 1.e scrub ok 2024-08-19T14:44:31.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:31 smithi045 ceph-mon[20143]: 1.12 scrub starts 2024-08-19T14:44:31.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:31 smithi045 ceph-mon[20143]: 1.12 scrub ok 2024-08-19T14:44:31.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:31 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2495790315' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:44:31.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:31.557 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:31.557 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:44:31.658 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:44:31.679 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:31.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:31.680 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:44:31.781 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:44:31.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:31.811 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:31.811 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:44:31.913 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:44:31.949 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:31.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:31.951 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:44:32.051 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:44:32.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:32.091 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:32.091 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:44:32.192 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:44:32.230 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:32.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:32.231 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:44:32.309 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 11 16 26966 26950 153.108 123.688 0.00149274 0.00651496 2024-08-19T14:44:32.332 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:44:32.357 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:32.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:32.359 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:44:32.408 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:32 smithi031 ceph-mon[16336]: pgmap v1970: 47 pgs: 47 active+clean; 1.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 154 MiB/s wr, 2.47k op/s 2024-08-19T14:44:32.408 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:32 smithi031 ceph-mon[16336]: 1.1a scrub starts 2024-08-19T14:44:32.408 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:32 smithi031 ceph-mon[16336]: 1.1a scrub ok 2024-08-19T14:44:32.424 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:32 smithi032 ceph-mon[20004]: pgmap v1970: 47 pgs: 47 active+clean; 1.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 154 MiB/s wr, 2.47k op/s 2024-08-19T14:44:32.424 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:32 smithi032 ceph-mon[20004]: 1.1a scrub starts 2024-08-19T14:44:32.424 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:32 smithi032 ceph-mon[20004]: 1.1a scrub ok 2024-08-19T14:44:32.460 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:44:32.486 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:32.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:32.487 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:44:32.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:32 smithi045 ceph-mon[20143]: pgmap v1970: 47 pgs: 47 active+clean; 1.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 154 MiB/s wr, 2.47k op/s 2024-08-19T14:44:32.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:32 smithi045 ceph-mon[20143]: 1.1a scrub starts 2024-08-19T14:44:32.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:32 smithi045 ceph-mon[20143]: 1.1a scrub ok 2024-08-19T14:44:32.566 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:44:32.566 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-08-19T14:44:32.566 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-08-19T14:44:32.566 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [] 2024-08-19T14:44:32.566 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:44:32.587 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:44:32.623 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:32.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:32.624 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:44:32.725 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:44:32.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:32.753 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:32.754 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:44:32.855 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:44:32.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:32.879 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:32.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:44:32.980 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:44:33.005 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:33.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:33.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:44:33.108 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:44:33.128 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is set 2024-08-19T14:44:33.132 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:33.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:33.134 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:44:33.235 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-08-19T14:44:33.235 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:44:33.260 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:33.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:33.261 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:44:33.309 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 12 16 29249 29233 152.239 142.688 0.00550434 0.00656219 2024-08-19T14:44:33.342 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:33 smithi031 ceph-mon[16336]: 1.1c scrub starts 2024-08-19T14:44:33.342 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:33 smithi031 ceph-mon[16336]: 1.1c scrub ok 2024-08-19T14:44:33.343 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:33 smithi031 ceph-mon[16336]: 1.f scrub starts 2024-08-19T14:44:33.343 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:33 smithi031 ceph-mon[16336]: 1.f scrub ok 2024-08-19T14:44:33.343 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:33 smithi031 ceph-mon[16336]: 1.1 scrub starts 2024-08-19T14:44:33.343 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:33 smithi031 ceph-mon[16336]: 1.1 scrub ok 2024-08-19T14:44:33.343 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:33 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3155923253' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:44:33.362 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:44:33.402 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:33.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:33.403 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:44:33.503 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:44:33.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:33 smithi032 ceph-mon[20004]: 1.1c scrub starts 2024-08-19T14:44:33.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:33 smithi032 ceph-mon[20004]: 1.1c scrub ok 2024-08-19T14:44:33.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:33 smithi032 ceph-mon[20004]: 1.f scrub starts 2024-08-19T14:44:33.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:33 smithi032 ceph-mon[20004]: 1.f scrub ok 2024-08-19T14:44:33.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:33 smithi032 ceph-mon[20004]: 1.1 scrub starts 2024-08-19T14:44:33.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:33 smithi032 ceph-mon[20004]: 1.1 scrub ok 2024-08-19T14:44:33.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:33 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3155923253' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:44:33.543 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:33.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:33.545 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:44:33.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:33 smithi045 ceph-mon[20143]: 1.1c scrub starts 2024-08-19T14:44:33.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:33 smithi045 ceph-mon[20143]: 1.1c scrub ok 2024-08-19T14:44:33.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:33 smithi045 ceph-mon[20143]: 1.f scrub starts 2024-08-19T14:44:33.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:33 smithi045 ceph-mon[20143]: 1.f scrub ok 2024-08-19T14:44:33.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:33 smithi045 ceph-mon[20143]: 1.1 scrub starts 2024-08-19T14:44:33.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:33 smithi045 ceph-mon[20143]: 1.1 scrub ok 2024-08-19T14:44:33.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:33 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3155923253' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:44:33.564 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:44:33.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:33 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mon-a[16332]: 2024-08-19T14:44:33.340+0000 7f2ff503e640 -1 mon.a@0(leader).osd e833 definitely_dead 0 2024-08-19T14:44:33.646 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:44:33.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:33.687 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:33.687 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:44:33.788 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:44:33.818 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:33.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:33.820 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:44:33.920 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:44:33.946 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:33.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:33.947 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:44:33.971 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:44:34.048 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:44:34.084 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:34.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:34.085 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:44:34.137 INFO:teuthology.orchestra.run.smithi031.stderr:marked out osd.5. 2024-08-19T14:44:34.186 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:44:34.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:34.208 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:34.208 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:44:34.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:34 smithi032 ceph-mon[20004]: pgmap v1971: 47 pgs: 47 active+clean; 1.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 141 MiB/s wr, 2.25k op/s 2024-08-19T14:44:34.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:34 smithi032 ceph-mon[20004]: 1.4 scrub starts 2024-08-19T14:44:34.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:34 smithi032 ceph-mon[20004]: 1.4 scrub ok 2024-08-19T14:44:34.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:34 smithi032 ceph-mon[20004]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:44:34.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:34 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3155923253' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:44:34.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:34 smithi032 ceph-mon[20004]: osdmap e833: 12 total, 12 up, 8 in 2024-08-19T14:44:34.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:34 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/4208374902' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-08-19T14:44:34.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:34 smithi032 ceph-mon[20004]: Client client.admin marked osd.5 out, while it was still marked up 2024-08-19T14:44:34.309 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 13 16 31088 31072 149.368 114.938 0.00212172 0.00660851 2024-08-19T14:44:34.309 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:44:34.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:34.331 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:34.332 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:44:34.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:34 smithi031 ceph-mon[16336]: pgmap v1971: 47 pgs: 47 active+clean; 1.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 141 MiB/s wr, 2.25k op/s 2024-08-19T14:44:34.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:34 smithi031 ceph-mon[16336]: 1.4 scrub starts 2024-08-19T14:44:34.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:34 smithi031 ceph-mon[16336]: 1.4 scrub ok 2024-08-19T14:44:34.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:34 smithi031 ceph-mon[16336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:44:34.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:34 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3155923253' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:44:34.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:34 smithi031 ceph-mon[16336]: osdmap e833: 12 total, 12 up, 8 in 2024-08-19T14:44:34.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:34 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/4208374902' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-08-19T14:44:34.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:34 smithi031 ceph-mon[16336]: Client client.admin marked osd.5 out, while it was still marked up 2024-08-19T14:44:34.433 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:44:34.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:34.455 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:34.455 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:44:34.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:34 smithi045 ceph-mon[20143]: pgmap v1971: 47 pgs: 47 active+clean; 1.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 141 MiB/s wr, 2.25k op/s 2024-08-19T14:44:34.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:34 smithi045 ceph-mon[20143]: 1.4 scrub starts 2024-08-19T14:44:34.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:34 smithi045 ceph-mon[20143]: 1.4 scrub ok 2024-08-19T14:44:34.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:34 smithi045 ceph-mon[20143]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:44:34.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:34 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3155923253' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:44:34.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:34 smithi045 ceph-mon[20143]: osdmap e833: 12 total, 12 up, 8 in 2024-08-19T14:44:34.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:34 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/4208374902' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-08-19T14:44:34.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:34 smithi045 ceph-mon[20143]: Client client.admin marked osd.5 out, while it was still marked up 2024-08-19T14:44:34.556 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:44:34.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:34.579 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:34.579 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:44:34.676 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:44:34.676 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-08-19T14:44:34.676 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-08-19T14:44:34.677 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:44:34.680 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:44:34.703 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:34.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:34.704 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:44:34.805 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:44:34.828 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:34.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:34.829 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:44:34.930 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:44:34.953 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:34.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:34.954 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:44:35.055 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:44:35.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:35.079 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:35.079 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:44:35.180 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:44:35.203 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:35.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:35.204 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:44:35.305 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:44:35.309 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 14 16 31088 31072 138.699 0 - 0.00660851 2024-08-19T14:44:35.328 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:35.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:35.329 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:44:35.395 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:44:35.430 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:44:35.456 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:35.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:35.457 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:44:35.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:35 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/4208374902' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-08-19T14:44:35.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:35 smithi032 ceph-mon[20004]: osdmap e834: 12 total, 12 up, 7 in 2024-08-19T14:44:35.518 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:35 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/4208374902' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-08-19T14:44:35.518 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:35 smithi031 ceph-mon[16336]: osdmap e834: 12 total, 12 up, 7 in 2024-08-19T14:44:35.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:35 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/4208374902' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-08-19T14:44:35.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:35 smithi045 ceph-mon[20143]: osdmap e834: 12 total, 12 up, 7 in 2024-08-19T14:44:35.558 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:44:35.583 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:35.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:35.584 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:44:35.683 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-08-19T14:44:35.685 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:44:35.708 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:35.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:35.709 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:44:35.810 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:44:35.834 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:35.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:35.835 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:44:35.936 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:44:35.950 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:44:35.957 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:35.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:35.958 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:44:36.059 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:44:36.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-08-19T14:44:36.077+0000 7fbdf0a97640 0 --2- 172.21.15.31:0/2477038842 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbdec1067c0 0x7fbdec106ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-08-19T14:44:36.085 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:36.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:36.086 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:44:36.100 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:44:36.101 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:44:36.118 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:44:36.118 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:44:36.134 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:44:36.134 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:44:36.156 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:44:36.185 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:44:36.185 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:44:36.187 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:44:36.212 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:36.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:36.213 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:44:36.214 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:44:36.214 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:44:36.234 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:44:36.234 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:44:36.255 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:44:36.256 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:44:36.289 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:44:36.289 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:44:36.310 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 15 16 31115 31099 129.564 0.84375 0.0109637 0.00757196 2024-08-19T14:44:36.314 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:44:36.315 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:44:36.315 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:44:36.327 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:36 smithi032 ceph-mon[20004]: pgmap v1974: 47 pgs: 47 active+clean; 1.6 GiB data, 9.9 GiB used, 616 GiB / 626 GiB avail; 143 MiB/s wr, 2.29k op/s 2024-08-19T14:44:36.327 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:36 smithi032 ceph-mon[20004]: osdmap e835: 12 total, 12 up, 7 in 2024-08-19T14:44:36.327 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:36 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3694847608' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:44:36.337 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:36.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:36.338 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:44:36.344 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:44:36.344 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:44:36.344 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:36 smithi031 ceph-mon[16336]: pgmap v1974: 47 pgs: 47 active+clean; 1.6 GiB data, 9.9 GiB used, 616 GiB / 626 GiB avail; 143 MiB/s wr, 2.29k op/s 2024-08-19T14:44:36.344 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:36 smithi031 ceph-mon[16336]: osdmap e835: 12 total, 12 up, 7 in 2024-08-19T14:44:36.344 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:36 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3694847608' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:44:36.386 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:44:36.386 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:44:36.439 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:44:36.443 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:44:36.443 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:44:36.467 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:36.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:36.468 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:44:36.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:36 smithi045 ceph-mon[20143]: pgmap v1974: 47 pgs: 47 active+clean; 1.6 GiB data, 9.9 GiB used, 616 GiB / 626 GiB avail; 143 MiB/s wr, 2.29k op/s 2024-08-19T14:44:36.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:36 smithi045 ceph-mon[20143]: osdmap e835: 12 total, 12 up, 7 in 2024-08-19T14:44:36.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:36 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3694847608' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:44:36.569 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:44:36.613 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:36.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:36.615 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:44:36.715 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:44:36.761 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:36.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:36.763 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:44:36.864 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:44:36.897 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-19T14:44:36.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:36.909 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:36.909 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:44:37.010 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:44:37.032 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:37.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:37.033 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:44:37.111 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-08-19T14:44:37.135 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:44:37.181 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:37.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:37.182 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:44:37.253 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:44:37.283 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:44:37.309 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 16 16 32039 32023 125.075 57.75 0.0101468 0.00798845 2024-08-19T14:44:37.318 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:37.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:37.319 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:44:37.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:37 smithi031 ceph-mon[16336]: osdmap e836: 12 total, 12 up, 7 in 2024-08-19T14:44:37.376 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:44:37.420 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:44:37.447 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:37.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:37.448 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:44:37.499 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:37 smithi032 ceph-mon[20004]: osdmap e836: 12 total, 12 up, 7 in 2024-08-19T14:44:37.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:37 smithi045 ceph-mon[20143]: osdmap e836: 12 total, 12 up, 7 in 2024-08-19T14:44:37.549 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:44:37.578 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:37.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:37.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:44:37.681 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:44:37.703 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:37.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:37.704 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:44:37.805 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:44:37.833 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:37.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:37.834 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:44:37.919 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:44:37.920 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:44:37.935 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:44:37.975 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:37.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:37.976 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:44:38.077 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:44:38.118 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:38.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:38.119 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:44:38.220 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:44:38.242 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:38.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:38.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:44:38.309 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 17 16 32950 32934 121.067 56.9375 0.00621645 0.00824904 2024-08-19T14:44:38.344 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:44:38.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:38.389 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:38.389 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:44:38.450 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is set 2024-08-19T14:44:38.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:38 smithi031 ceph-mon[16336]: pgmap v1977: 47 pgs: 10 remapped+peering, 1 peering, 36 active+clean; 1.8 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 139 MiB/s wr, 2.23k op/s 2024-08-19T14:44:38.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:38 smithi031 ceph-mon[16336]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-08-19T14:44:38.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:38 smithi031 ceph-mon[16336]: osdmap e837: 12 total, 12 up, 7 in 2024-08-19T14:44:38.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:38 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1165857990' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:44:38.490 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:44:38.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:38 smithi032 ceph-mon[20004]: pgmap v1977: 47 pgs: 10 remapped+peering, 1 peering, 36 active+clean; 1.8 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 139 MiB/s wr, 2.23k op/s 2024-08-19T14:44:38.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:38 smithi032 ceph-mon[20004]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-08-19T14:44:38.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:38 smithi032 ceph-mon[20004]: osdmap e837: 12 total, 12 up, 7 in 2024-08-19T14:44:38.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:38 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1165857990' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:44:38.534 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:38.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:38.535 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:44:38.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:38 smithi045 ceph-mon[20143]: pgmap v1977: 47 pgs: 10 remapped+peering, 1 peering, 36 active+clean; 1.8 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 139 MiB/s wr, 2.23k op/s 2024-08-19T14:44:38.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:38 smithi045 ceph-mon[20143]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-08-19T14:44:38.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:38 smithi045 ceph-mon[20143]: osdmap e837: 12 total, 12 up, 7 in 2024-08-19T14:44:38.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:38 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1165857990' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:44:38.602 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-08-19T14:44:38.636 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:44:38.695 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:38.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:38.696 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:44:38.797 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:44:38.823 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:38.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:38.824 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:44:38.925 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:44:38.951 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:38.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:38.952 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:44:39.044 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:44:39.053 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:44:39.090 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:44:39.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:39.112 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:39.112 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:44:39.213 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:44:39.260 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:39.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:39.261 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:44:39.310 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 18 16 33835 33819 117.413 55.3125 0.00509474 0.00850598 2024-08-19T14:44:39.362 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:44:39.393 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:39.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:39.394 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:44:39.398 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:44:39.496 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:44:39.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:39 smithi032 ceph-mon[20004]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:44:39.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:39 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1165857990' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:44:39.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:39 smithi032 ceph-mon[20004]: osdmap e838: 12 total, 12 up, 7 in 2024-08-19T14:44:39.528 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:39 smithi031 ceph-mon[16336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:44:39.528 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:39 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1165857990' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:44:39.528 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:39 smithi031 ceph-mon[16336]: osdmap e838: 12 total, 12 up, 7 in 2024-08-19T14:44:39.539 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:39.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:39.540 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:44:39.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:39 smithi045 ceph-mon[20143]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:44:39.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:39 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1165857990' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:44:39.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:39 smithi045 ceph-mon[20143]: osdmap e838: 12 total, 12 up, 7 in 2024-08-19T14:44:39.642 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:44:39.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:39.665 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:39.665 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:44:39.766 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:44:39.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:39.789 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:39.789 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:44:39.890 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:44:39.915 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:44:39.916 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:44:39.933 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:39.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:39.935 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:44:39.974 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:44:39.975 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:44:40.000 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:44:40.001 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:44:40.023 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:44:40.024 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:44:40.035 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:44:40.049 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:44:40.049 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:44:40.065 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:40.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:40.066 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:44:40.076 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:44:40.076 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:44:40.097 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:44:40.097 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:44:40.121 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:44:40.121 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:44:40.144 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:44:40.144 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:44:40.167 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:44:40.182 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:44:40.182 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:44:40.188 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:40.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:40.189 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:44:40.214 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:44:40.214 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:44:40.252 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:44:40.252 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:44:40.290 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:44:40.293 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:44:40.293 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:44:40.310 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 19 16 34691 34675 114.049 53.5 0.0232026 0.00876033 2024-08-19T14:44:40.322 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:40.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:40.323 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:44:40.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:40 smithi031 ceph-mon[16336]: pgmap v1980: 47 pgs: 2 active+recovering+undersized+remapped, 10 remapped+peering, 2 peering, 33 active+clean; 1.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 69 MiB/s wr, 1.10k op/s; 3820/92535 objects misplaced (4.128%); 2.0 MiB/s, 31 objects/s recovering 2024-08-19T14:44:40.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:40 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3114206116' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:44:40.370 INFO:tasks.thrashosds.thrasher:in_osds: [1, 4, 8, 2, 11, 9, 10] out_osds: [6, 3, 0, 7, 5] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 8, 9, 11, 10, 2] 2024-08-19T14:44:40.370 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd reweight 1 1 2024-08-19T14:44:40.424 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:44:40.468 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:40.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:40.469 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:44:40.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:40 smithi032 ceph-mon[20004]: pgmap v1980: 47 pgs: 2 active+recovering+undersized+remapped, 10 remapped+peering, 2 peering, 33 active+clean; 1.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 69 MiB/s wr, 1.10k op/s; 3820/92535 objects misplaced (4.128%); 2.0 MiB/s, 31 objects/s recovering 2024-08-19T14:44:40.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:40 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3114206116' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:44:40.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:40 smithi045 ceph-mon[20143]: pgmap v1980: 47 pgs: 2 active+recovering+undersized+remapped, 10 remapped+peering, 2 peering, 33 active+clean; 1.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 69 MiB/s wr, 1.10k op/s; 3820/92535 objects misplaced (4.128%); 2.0 MiB/s, 31 objects/s recovering 2024-08-19T14:44:40.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:40 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3114206116' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:44:40.570 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:44:40.611 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:40.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:40.612 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:44:40.713 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:44:40.759 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:40.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:40.760 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:44:40.861 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:44:40.898 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-08-19T14:44:40.917 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:40.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:40.918 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:44:41.019 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:44:41.061 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:44:41.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:41.069 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:41.069 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:44:41.170 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:44:41.218 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:41.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:41.222 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:44:41.304 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-19T14:44:41.310 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:2024-08-19T14:44:41.310509+0000 min lat: 0.00105666 max lat: 2.17994 avg lat: 0.00901237 2024-08-19T14:44:41.310 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-19T14:44:41.310 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 20 16 35481 35465 110.814 49.375 0.0389439 0.00901237 2024-08-19T14:44:41.323 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:44:41.345 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:41.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:41.346 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:44:41.401 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:44:41.446 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:44:41.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:41.477 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:41.477 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:44:41.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:41 smithi031 ceph-mon[16336]: Health check failed: Degraded data redundancy: 1/100143 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-08-19T14:44:41.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:41 smithi032 ceph-mon[20004]: Health check failed: Degraded data redundancy: 1/100143 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-08-19T14:44:41.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:41 smithi045 ceph-mon[20143]: Health check failed: Degraded data redundancy: 1/100143 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-08-19T14:44:41.578 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:44:41.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:41.600 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:41.601 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:44:41.698 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:44:41.701 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:44:41.756 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:41.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:41.757 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:44:41.858 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:44:41.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:41.900 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:41.900 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:44:42.002 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:44:42.028 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:42.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:42.030 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:44:42.050 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:44:42.051 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [], 2024-08-19T14:44:42.051 INFO:teuthology.orchestra.run.smithi045.stdout: "complaint_time": 30, 2024-08-19T14:44:42.051 INFO:teuthology.orchestra.run.smithi045.stdout: "num_blocked_ops": 0 2024-08-19T14:44:42.051 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:44:42.130 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:44:42.158 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:42.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:42.160 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:44:42.227 INFO:teuthology.orchestra.run.smithi031.stderr:reweighted osd.1 to 1 (10000) 2024-08-19T14:44:42.261 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:44:42.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:42.284 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:42.284 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:44:42.310 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 21 16 36269 36253 107.882 49.25 0.00964599 0.00925878 2024-08-19T14:44:42.385 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:44:42.426 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:42.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:42.427 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:44:42.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:42 smithi032 ceph-mon[20004]: pgmap v1981: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 6 remapped+peering, 1 peering, 33 active+clean; 2.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 77 MiB/s wr, 1.23k op/s; 1/100143 objects degraded (0.001%); 12554/100143 objects misplaced (12.536%); 13 MiB/s, 206 objects/s recovering 2024-08-19T14:44:42.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:42 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2179552510' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-08-19T14:44:42.528 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:44:42.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:42 smithi045 ceph-mon[20143]: pgmap v1981: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 6 remapped+peering, 1 peering, 33 active+clean; 2.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 77 MiB/s wr, 1.23k op/s; 1/100143 objects degraded (0.001%); 12554/100143 objects misplaced (12.536%); 13 MiB/s, 206 objects/s recovering 2024-08-19T14:44:42.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:42 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2179552510' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-08-19T14:44:42.565 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:42.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:42.566 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:44:42.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:42 smithi031 ceph-mon[16336]: pgmap v1981: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 6 remapped+peering, 1 peering, 33 active+clean; 2.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 77 MiB/s wr, 1.23k op/s; 1/100143 objects degraded (0.001%); 12554/100143 objects misplaced (12.536%); 13 MiB/s, 206 objects/s recovering 2024-08-19T14:44:42.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:42 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2179552510' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-08-19T14:44:42.667 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:44:42.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:42.690 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:42.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:44:42.791 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:44:42.797 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-08-19T14:44:42.816 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:42.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:42.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:44:42.918 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:44:42.941 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:42.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:42.943 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:44:43.006 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:44:43.044 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:44:43.066 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:43.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:43.067 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:44:43.169 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:44:43.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:43.192 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:43.192 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:44:43.262 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:44:43.293 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:44:43.313 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 22 16 36962 36946 104.945 43.3125 0.013603 0.0095201 2024-08-19T14:44:43.323 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:43.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:43.324 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:44:43.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:43 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2179552510' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-08-19T14:44:43.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:43 smithi031 ceph-mon[16336]: osdmap e839: 12 total, 12 up, 7 in 2024-08-19T14:44:43.425 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:44:43.454 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:43.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:43.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:44:43.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:43 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2179552510' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-08-19T14:44:43.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:43 smithi032 ceph-mon[20004]: osdmap e839: 12 total, 12 up, 7 in 2024-08-19T14:44:43.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:43 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2179552510' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-08-19T14:44:43.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:43 smithi045 ceph-mon[20143]: osdmap e839: 12 total, 12 up, 7 in 2024-08-19T14:44:43.556 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:44:43.587 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:43.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:43.589 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:44:43.690 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:44:43.711 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:43.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:43.712 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:44:43.813 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:44:43.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:43.835 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:43.835 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:44:43.899 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:44:43.899 INFO:teuthology.orchestra.run.smithi045.stdout: "size": 20, 2024-08-19T14:44:43.899 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 600, 2024-08-19T14:44:43.899 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [ 2024-08-19T14:44:43.899 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.899 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.16962.0:224666 7.7 7:e0cf291e:::benchmark_data_smithi045_115842_object106506:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e703)", 2024-08-19T14:44:43.899 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:39:14.759205+0000", 2024-08-19T14:44:43.899 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 329.13614467899998, 2024-08-19T14:44:43.899 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.212613626, 2024-08-19T14:44:43.899 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:44:43.899 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:44:43.900 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:44:43.900 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:44:43.900 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.16962", 2024-08-19T14:44:43.900 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:44:43.900 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 224666 2024-08-19T14:44:43.900 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.900 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:44:43.900 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.900 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:44:43.900 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.759205+0000", 2024-08-19T14:44:43.900 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:44:43.900 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.900 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.900 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:44:43.901 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.759205+0000", 2024-08-19T14:44:43.901 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:44:43.901 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.901 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.901 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:44:43.901 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.759206+0000", 2024-08-19T14:44:43.901 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.322e-06 2024-08-19T14:44:43.901 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.901 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.901 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:44:43.901 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.759211+0000", 2024-08-19T14:44:43.901 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.7859999999999996e-06 2024-08-19T14:44:43.901 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.902 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.902 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:44:43.902 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.759221+0000", 2024-08-19T14:44:43.902 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0529e-05 2024-08-19T14:44:43.902 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.902 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.902 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:44:43.902 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.759229+0000", 2024-08-19T14:44:43.902 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.5390000000000002e-06 2024-08-19T14:44:43.902 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.902 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.902 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:44:43.903 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.759280+0000", 2024-08-19T14:44:43.903 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0915999999999998e-05 2024-08-19T14:44:43.903 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.903 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.903 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:44:43.903 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.759363+0000", 2024-08-19T14:44:43.903 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.3714000000000005e-05 2024-08-19T14:44:43.903 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.903 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.903 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,3", 2024-08-19T14:44:43.903 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.759437+0000", 2024-08-19T14:44:43.903 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.3863999999999996e-05 2024-08-19T14:44:43.903 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.904 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.904 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:44:43.904 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.764416+0000", 2024-08-19T14:44:43.904 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0049787759999999999 2024-08-19T14:44:43.904 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.904 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.904 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:44:43.904 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.770232+0000", 2024-08-19T14:44:43.904 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0058158460000000004 2024-08-19T14:44:43.904 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.904 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.904 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:44:43.904 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.971750+0000", 2024-08-19T14:44:43.905 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20151855199999999 2024-08-19T14:44:43.905 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.905 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.905 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:44:43.905 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.971788+0000", 2024-08-19T14:44:43.905 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.7789e-05 2024-08-19T14:44:43.905 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.905 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.905 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:44:43.905 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.971818+0000", 2024-08-19T14:44:43.905 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.9992999999999999e-05 2024-08-19T14:44:43.905 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:44:43.905 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:44:43.905 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:44:43.906 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.906 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.906 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.16962.0:224669 7.7 7:ea42df57:::benchmark_data_smithi045_115842_object106509:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e703)", 2024-08-19T14:44:43.906 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:39:14.765265+0000", 2024-08-19T14:44:43.906 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 329.13008418599998, 2024-08-19T14:44:43.906 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.23464995899999999, 2024-08-19T14:44:43.906 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:44:43.906 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:44:43.906 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:44:43.906 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:44:43.906 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.16962", 2024-08-19T14:44:43.906 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:44:43.906 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 224669 2024-08-19T14:44:43.907 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.907 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:44:43.907 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.907 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:44:43.907 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.765265+0000", 2024-08-19T14:44:43.907 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:44:43.907 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.907 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.907 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:44:43.907 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.765265+0000", 2024-08-19T14:44:43.907 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:44:43.907 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.907 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.908 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:44:43.908 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.765267+0000", 2024-08-19T14:44:43.908 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5600000000000001e-06 2024-08-19T14:44:43.908 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.908 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.908 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:44:43.908 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.765272+0000", 2024-08-19T14:44:43.908 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.6219999999999997e-06 2024-08-19T14:44:43.908 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.908 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.908 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:44:43.908 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.765286+0000", 2024-08-19T14:44:43.908 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3998999999999999e-05 2024-08-19T14:44:43.908 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.909 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.909 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:44:43.909 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.765294+0000", 2024-08-19T14:44:43.909 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.9349999999999994e-06 2024-08-19T14:44:43.909 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.909 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.909 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:44:43.909 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.765350+0000", 2024-08-19T14:44:43.909 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.6243999999999999e-05 2024-08-19T14:44:43.909 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.909 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.909 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:44:43.909 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.765474+0000", 2024-08-19T14:44:43.909 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000123762 2024-08-19T14:44:43.909 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.910 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.910 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,3", 2024-08-19T14:44:43.910 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.765598+0000", 2024-08-19T14:44:43.910 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012404699999999999 2024-08-19T14:44:43.910 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.910 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.910 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:44:43.910 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.770614+0000", 2024-08-19T14:44:43.910 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0050154029999999999 2024-08-19T14:44:43.910 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.910 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.910 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:44:43.910 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.779808+0000", 2024-08-19T14:44:43.910 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0091942959999999994 2024-08-19T14:44:43.911 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.911 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.911 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:44:43.911 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.999824+0000", 2024-08-19T14:44:43.911 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.220016463 2024-08-19T14:44:43.911 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.911 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.911 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:44:43.911 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.999862+0000", 2024-08-19T14:44:43.911 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.8061e-05 2024-08-19T14:44:43.911 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.911 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.911 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:44:43.911 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.999915+0000", 2024-08-19T14:44:43.912 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2567e-05 2024-08-19T14:44:43.912 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:44:43.912 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:44:43.912 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:44:43.912 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.912 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.912 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.16962.0:224671 7.7 7:e66f70e9:::benchmark_data_smithi045_115842_object106511:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e703)", 2024-08-19T14:44:43.912 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:39:14.771305+0000", 2024-08-19T14:44:43.912 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 329.12404402700002, 2024-08-19T14:44:43.912 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.22877210000000001, 2024-08-19T14:44:43.912 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:44:43.912 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:44:43.912 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:44:43.912 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:44:43.913 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.16962", 2024-08-19T14:44:43.913 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:44:43.913 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 224671 2024-08-19T14:44:43.913 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.913 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:44:43.913 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.914 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:44:43.914 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.771305+0000", 2024-08-19T14:44:43.914 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:44:43.914 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.914 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.914 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:44:43.914 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.771305+0000", 2024-08-19T14:44:43.914 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:44:43.914 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.914 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.914 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:44:43.914 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.771307+0000", 2024-08-19T14:44:43.914 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.703e-06 2024-08-19T14:44:43.915 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.915 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.915 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:44:43.915 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.771313+0000", 2024-08-19T14:44:43.915 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.3659999999999997e-06 2024-08-19T14:44:43.915 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.915 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.915 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:44:43.915 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.771331+0000", 2024-08-19T14:44:43.915 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7513999999999999e-05 2024-08-19T14:44:43.915 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.915 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.915 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:44:43.915 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.771338+0000", 2024-08-19T14:44:43.916 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.6799999999999993e-06 2024-08-19T14:44:43.916 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.916 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.916 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:44:43.916 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.771392+0000", 2024-08-19T14:44:43.916 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.342e-05 2024-08-19T14:44:43.916 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.916 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.916 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:44:43.916 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.771508+0000", 2024-08-19T14:44:43.916 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00011576799999999999 2024-08-19T14:44:43.916 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.916 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.916 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,3", 2024-08-19T14:44:43.917 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.771620+0000", 2024-08-19T14:44:43.917 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000112139 2024-08-19T14:44:43.917 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.917 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.917 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:44:43.917 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.776775+0000", 2024-08-19T14:44:43.917 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0051554879999999997 2024-08-19T14:44:43.917 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.917 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.917 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:44:43.917 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.780515+0000", 2024-08-19T14:44:43.917 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0037401919999999998 2024-08-19T14:44:43.917 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.917 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.918 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:44:43.918 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.999975+0000", 2024-08-19T14:44:43.918 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.21945916400000001 2024-08-19T14:44:43.918 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.918 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.918 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:44:43.918 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:15.000031+0000", 2024-08-19T14:44:43.918 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.6450000000000003e-05 2024-08-19T14:44:43.918 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.918 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.918 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:44:43.918 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:15.000077+0000", 2024-08-19T14:44:43.918 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.6215999999999999e-05 2024-08-19T14:44:43.919 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:44:43.919 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:44:43.919 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:44:43.919 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.919 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.919 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.16962.0:224700 7.7 7:e378e4bd:::benchmark_data_smithi045_115842_object106540:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e703)", 2024-08-19T14:44:43.919 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:39:14.825059+0000", 2024-08-19T14:44:43.919 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 329.07028978800003, 2024-08-19T14:44:43.919 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.25962368200000002, 2024-08-19T14:44:43.919 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:44:43.919 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:44:43.919 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:44:43.919 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:44:43.919 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.16962", 2024-08-19T14:44:43.920 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:44:43.920 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 224700 2024-08-19T14:44:43.920 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.920 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:44:43.920 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.920 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:44:43.920 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.825059+0000", 2024-08-19T14:44:43.920 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:44:43.920 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.920 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.920 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:44:43.920 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.825059+0000", 2024-08-19T14:44:43.920 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:44:43.920 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.921 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.921 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:44:43.921 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.825061+0000", 2024-08-19T14:44:43.921 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1400000000000001e-06 2024-08-19T14:44:43.921 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.921 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.921 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:44:43.921 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.825066+0000", 2024-08-19T14:44:43.921 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0730000000000004e-06 2024-08-19T14:44:43.921 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.921 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.922 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:44:43.922 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.825078+0000", 2024-08-19T14:44:43.922 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2721000000000001e-05 2024-08-19T14:44:43.922 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.922 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.922 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:44:43.922 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.825085+0000", 2024-08-19T14:44:43.922 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.1350000000000003e-06 2024-08-19T14:44:43.922 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.922 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.922 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:44:43.922 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.826700+0000", 2024-08-19T14:44:43.922 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0016148079999999999 2024-08-19T14:44:43.922 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.923 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.923 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:44:43.923 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.826814+0000", 2024-08-19T14:44:43.923 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000113643 2024-08-19T14:44:43.923 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.923 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.923 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,3", 2024-08-19T14:44:43.923 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.826896+0000", 2024-08-19T14:44:43.923 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.2315e-05 2024-08-19T14:44:43.923 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.923 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.923 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:44:43.923 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.839158+0000", 2024-08-19T14:44:43.923 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.012261943000000001 2024-08-19T14:44:43.924 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.924 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.924 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:44:43.924 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.847478+0000", 2024-08-19T14:44:43.924 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0083198690000000006 2024-08-19T14:44:43.924 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.924 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.924 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:44:43.924 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:15.084574+0000", 2024-08-19T14:44:43.924 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.23709612699999999 2024-08-19T14:44:43.924 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.924 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.924 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:44:43.924 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:15.084631+0000", 2024-08-19T14:44:43.925 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7158999999999999e-05 2024-08-19T14:44:43.925 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.925 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.925 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:44:43.925 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:15.084683+0000", 2024-08-19T14:44:43.925 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1749000000000002e-05 2024-08-19T14:44:43.925 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:44:43.925 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:44:43.925 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:44:43.925 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.925 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.925 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.16962.0:224704 7.7 7:e5b596ea:::benchmark_data_smithi045_115842_object106544:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e703)", 2024-08-19T14:44:43.925 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:39:14.828060+0000", 2024-08-19T14:44:43.925 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 329.06728877, 2024-08-19T14:44:43.926 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.26613611799999998, 2024-08-19T14:44:43.926 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:44:43.926 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:44:43.926 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:44:43.926 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:44:43.926 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.16962", 2024-08-19T14:44:43.926 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:44:43.926 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 224704 2024-08-19T14:44:43.926 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.926 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:44:43.926 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.926 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:44:43.926 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.828060+0000", 2024-08-19T14:44:43.926 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:44:43.927 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.927 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.927 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:44:43.927 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.828060+0000", 2024-08-19T14:44:43.927 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:44:43.927 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.927 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.927 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:44:43.927 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.828062+0000", 2024-08-19T14:44:43.927 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2929999999999999e-06 2024-08-19T14:44:43.927 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.927 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.927 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:44:43.927 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.828067+0000", 2024-08-19T14:44:43.928 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.3179999999999997e-06 2024-08-19T14:44:43.928 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.928 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.928 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:44:43.928 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.828083+0000", 2024-08-19T14:44:43.928 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5843000000000001e-05 2024-08-19T14:44:43.928 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.928 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.928 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:44:43.928 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.828090+0000", 2024-08-19T14:44:43.928 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4290000000000004e-06 2024-08-19T14:44:43.928 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.928 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.928 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:44:43.929 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.830830+0000", 2024-08-19T14:44:43.929 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0027398330000000001 2024-08-19T14:44:43.929 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.929 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.929 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:44:43.929 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.830955+0000", 2024-08-19T14:44:43.929 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012513800000000001 2024-08-19T14:44:43.929 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.929 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.929 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,3", 2024-08-19T14:44:43.929 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.831089+0000", 2024-08-19T14:44:43.929 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013408800000000001 2024-08-19T14:44:43.929 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.929 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.929 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:44:43.930 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.842806+0000", 2024-08-19T14:44:43.930 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.011716183 2024-08-19T14:44:43.930 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.930 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.930 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:44:43.930 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.847659+0000", 2024-08-19T14:44:43.930 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0048539070000000002 2024-08-19T14:44:43.930 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.930 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.930 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:44:43.930 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:15.094133+0000", 2024-08-19T14:44:43.930 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.246473419 2024-08-19T14:44:43.930 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.931 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.931 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:44:43.931 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:15.094166+0000", 2024-08-19T14:44:43.931 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.2678000000000001e-05 2024-08-19T14:44:43.931 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.931 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.931 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:44:43.931 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:15.094197+0000", 2024-08-19T14:44:43.931 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.0988999999999998e-05 2024-08-19T14:44:43.931 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:44:43.931 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:44:43.931 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:44:43.931 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.931 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.931 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.16962.0:224705 7.7 7:e64435fe:::benchmark_data_smithi045_115842_object106545:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e703)", 2024-08-19T14:44:43.932 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:39:14.828443+0000", 2024-08-19T14:44:43.932 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 329.06690577199998, 2024-08-19T14:44:43.932 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.27032617399999997, 2024-08-19T14:44:43.932 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:44:43.932 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:44:43.932 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:44:43.932 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:44:43.932 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.16962", 2024-08-19T14:44:43.932 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:44:43.932 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 224705 2024-08-19T14:44:43.932 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.932 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:44:43.932 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.932 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:44:43.933 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.828443+0000", 2024-08-19T14:44:43.933 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:44:43.933 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.933 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.933 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:44:43.933 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.828443+0000", 2024-08-19T14:44:43.933 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:44:43.933 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.933 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.933 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:44:43.933 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.828444+0000", 2024-08-19T14:44:43.933 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.3099999999999996e-07 2024-08-19T14:44:43.933 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.933 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.934 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:44:43.934 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.828450+0000", 2024-08-19T14:44:43.934 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2059999999999998e-06 2024-08-19T14:44:43.934 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.934 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.934 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:44:43.934 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.828464+0000", 2024-08-19T14:44:43.934 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3969e-05 2024-08-19T14:44:43.934 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.934 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.934 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:44:43.934 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.828471+0000", 2024-08-19T14:44:43.934 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.7849999999999992e-06 2024-08-19T14:44:43.934 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.935 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.935 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:44:43.935 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.835455+0000", 2024-08-19T14:44:43.935 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0069832260000000004 2024-08-19T14:44:43.935 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.935 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.935 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:44:43.935 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.835539+0000", 2024-08-19T14:44:43.935 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.4761999999999999e-05 2024-08-19T14:44:43.935 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.935 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.935 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,3", 2024-08-19T14:44:43.935 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.835615+0000", 2024-08-19T14:44:43.935 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.5931000000000002e-05 2024-08-19T14:44:43.936 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.936 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.936 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:44:43.936 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.842812+0000", 2024-08-19T14:44:43.936 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0071963900000000004 2024-08-19T14:44:43.936 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.936 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.936 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:44:43.936 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:14.849992+0000", 2024-08-19T14:44:43.936 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0071800739999999998 2024-08-19T14:44:43.936 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.936 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.936 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:44:43.936 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:15.098646+0000", 2024-08-19T14:44:43.937 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.248654664 2024-08-19T14:44:43.937 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.937 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.937 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:44:43.937 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:15.098717+0000", 2024-08-19T14:44:43.937 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.1091000000000001e-05 2024-08-19T14:44:43.937 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.937 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.937 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:44:43.937 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:15.098770+0000", 2024-08-19T14:44:43.937 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2145000000000003e-05 2024-08-19T14:44:43.937 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:44:43.937 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:44:43.937 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:44:43.938 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.938 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.938 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.16962.0:232608 7.7 7:e15b25b9:::benchmark_data_smithi045_115842_object114448:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e707)", 2024-08-19T14:44:43.938 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:39:29.466905+0000", 2024-08-19T14:44:43.938 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 314.42844424899999, 2024-08-19T14:44:43.938 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.29847353399999998, 2024-08-19T14:44:43.938 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:44:43.938 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:44:43.938 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:44:43.938 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:44:43.938 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.16962", 2024-08-19T14:44:43.938 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:44:43.938 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 232608 2024-08-19T14:44:43.938 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.939 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:44:43.939 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.939 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:44:43.939 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.466905+0000", 2024-08-19T14:44:43.939 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:44:43.939 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.939 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.939 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:44:43.939 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.466905+0000", 2024-08-19T14:44:43.939 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:44:43.939 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.939 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.939 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:44:43.939 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.466906+0000", 2024-08-19T14:44:43.940 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.099e-06 2024-08-19T14:44:43.940 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.940 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.940 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:44:43.940 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.466911+0000", 2024-08-19T14:44:43.940 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.5499999999999996e-06 2024-08-19T14:44:43.940 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.940 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.940 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:44:43.940 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.466921+0000", 2024-08-19T14:44:43.940 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0570999999999999e-05 2024-08-19T14:44:43.940 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.940 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.940 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:44:43.941 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.466927+0000", 2024-08-19T14:44:43.941 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.6269999999999997e-06 2024-08-19T14:44:43.941 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.941 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.941 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:44:43.941 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:44:43.942 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.466975+0000", 2024-08-19T14:44:43.942 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.7728000000000001e-05 2024-08-19T14:44:43.943 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.943 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.943 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:44:43.943 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.467057+0000", 2024-08-19T14:44:43.943 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.2019000000000002e-05 2024-08-19T14:44:43.943 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.943 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.943 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,3", 2024-08-19T14:44:43.943 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.467130+0000", 2024-08-19T14:44:43.943 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.3858999999999999e-05 2024-08-19T14:44:43.943 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.943 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.944 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:44:43.944 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.474856+0000", 2024-08-19T14:44:43.944 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.007725277 2024-08-19T14:44:43.944 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.944 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.944 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:44:43.944 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.491803+0000", 2024-08-19T14:44:43.944 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.016947051000000001 2024-08-19T14:44:43.944 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.944 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.944 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:44:43.944 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.765277+0000", 2024-08-19T14:44:43.944 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.27347464999999999 2024-08-19T14:44:43.945 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.945 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.945 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:44:43.945 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.765346+0000", 2024-08-19T14:44:43.945 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.8960000000000004e-05 2024-08-19T14:44:43.945 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.945 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.945 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:44:43.945 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.765378+0000", 2024-08-19T14:44:43.945 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.2143e-05 2024-08-19T14:44:43.945 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:44:43.945 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:44:43.945 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:44:43.945 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.946 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.946 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.16962.0:232610 7.7 7:e9c6b41b:::benchmark_data_smithi045_115842_object114450:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e707)", 2024-08-19T14:44:43.946 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:39:29.469702+0000", 2024-08-19T14:44:43.946 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 314.425646728, 2024-08-19T14:44:43.946 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.295804754, 2024-08-19T14:44:43.946 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:44:43.946 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:44:43.946 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:44:43.946 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:44:43.946 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.16962", 2024-08-19T14:44:43.946 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:44:43.946 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 232610 2024-08-19T14:44:43.946 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.947 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:44:43.947 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.947 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:44:43.947 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.469702+0000", 2024-08-19T14:44:43.947 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:44:43.947 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.947 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.947 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:44:43.947 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.469702+0000", 2024-08-19T14:44:43.947 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:44:43.947 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.947 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.947 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:44:43.947 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.469703+0000", 2024-08-19T14:44:43.947 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.2799999999999995e-07 2024-08-19T14:44:43.948 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.948 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.948 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:44:43.948 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.469707+0000", 2024-08-19T14:44:43.948 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.6729999999999998e-06 2024-08-19T14:44:43.948 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.948 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.948 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:44:43.948 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.469717+0000", 2024-08-19T14:44:43.948 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.9690000000000006e-06 2024-08-19T14:44:43.948 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.948 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.948 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:44:43.948 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.469725+0000", 2024-08-19T14:44:43.949 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.3469999999999999e-06 2024-08-19T14:44:43.949 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.949 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.949 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:44:43.949 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.470957+0000", 2024-08-19T14:44:43.949 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0012315749999999999 2024-08-19T14:44:43.949 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.949 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.949 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:44:43.949 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.471050+0000", 2024-08-19T14:44:43.949 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.3677000000000005e-05 2024-08-19T14:44:43.949 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.949 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.949 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,3", 2024-08-19T14:44:43.950 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.471177+0000", 2024-08-19T14:44:43.950 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012658999999999999 2024-08-19T14:44:43.950 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.950 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.950 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:44:43.950 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.475091+0000", 2024-08-19T14:44:43.950 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0039135089999999999 2024-08-19T14:44:43.950 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.950 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.950 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:44:43.950 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.494005+0000", 2024-08-19T14:44:43.950 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.018914362000000001 2024-08-19T14:44:43.950 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.951 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.951 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:44:43.951 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.765427+0000", 2024-08-19T14:44:43.951 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.27142210700000002 2024-08-19T14:44:43.951 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.951 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.951 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:44:43.951 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.765479+0000", 2024-08-19T14:44:43.951 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1789999999999997e-05 2024-08-19T14:44:43.951 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.951 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.951 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:44:43.951 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.765507+0000", 2024-08-19T14:44:43.952 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.8427000000000001e-05 2024-08-19T14:44:43.952 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:44:43.952 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:44:43.952 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:44:43.952 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.952 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.952 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.16962.0:232614 7.7 7:e6b4bf77:::benchmark_data_smithi045_115842_object114454:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e707)", 2024-08-19T14:44:43.952 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:39:29.475122+0000", 2024-08-19T14:44:43.952 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 314.42022688200001, 2024-08-19T14:44:43.952 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.29052661299999999, 2024-08-19T14:44:43.952 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:44:43.952 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:44:43.952 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:44:43.952 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:44:43.953 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.16962", 2024-08-19T14:44:43.953 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:44:43.953 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 232614 2024-08-19T14:44:43.953 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.953 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:44:43.953 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.953 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:44:43.953 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.475122+0000", 2024-08-19T14:44:43.953 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:44:43.953 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.953 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.953 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:44:43.953 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.475122+0000", 2024-08-19T14:44:43.953 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:44:43.954 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.954 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.954 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:44:43.954 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.475124+0000", 2024-08-19T14:44:43.954 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.176e-06 2024-08-19T14:44:43.954 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.954 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.954 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:44:43.954 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.475129+0000", 2024-08-19T14:44:43.954 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2020000000000003e-06 2024-08-19T14:44:43.954 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.954 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.954 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:44:43.955 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.475140+0000", 2024-08-19T14:44:43.955 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1039999999999999e-05 2024-08-19T14:44:43.955 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.955 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.955 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:44:43.955 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.475147+0000", 2024-08-19T14:44:43.955 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.3930000000000002e-06 2024-08-19T14:44:43.955 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.955 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.955 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:44:43.955 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.475196+0000", 2024-08-19T14:44:43.955 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.9073999999999998e-05 2024-08-19T14:44:43.955 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.955 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.956 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:44:43.956 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.475313+0000", 2024-08-19T14:44:43.956 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000116487 2024-08-19T14:44:43.956 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.956 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.956 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,3", 2024-08-19T14:44:43.956 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.475431+0000", 2024-08-19T14:44:43.956 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000118733 2024-08-19T14:44:43.956 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.956 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.956 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:44:43.956 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.479893+0000", 2024-08-19T14:44:43.956 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0044616609999999996 2024-08-19T14:44:43.956 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.957 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.957 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:44:43.957 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.494040+0000", 2024-08-19T14:44:43.957 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.014146832 2024-08-19T14:44:43.957 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.957 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.957 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:44:43.957 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.765567+0000", 2024-08-19T14:44:43.957 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.27152742099999999 2024-08-19T14:44:43.957 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.957 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.957 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:44:43.957 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.765613+0000", 2024-08-19T14:44:43.958 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.5884000000000003e-05 2024-08-19T14:44:43.958 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.958 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.958 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:44:43.958 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.765649+0000", 2024-08-19T14:44:43.958 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.5710000000000002e-05 2024-08-19T14:44:43.958 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:44:43.958 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:44:43.958 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:44:43.958 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.958 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.958 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.16962.0:232619 7.7 7:e24f6845:::benchmark_data_smithi045_115842_object114459:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e707)", 2024-08-19T14:44:43.958 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:39:29.487667+0000", 2024-08-19T14:44:43.959 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 314.40768211300002, 2024-08-19T14:44:43.959 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.27813914699999998, 2024-08-19T14:44:43.959 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:44:43.959 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:44:43.959 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:44:43.959 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:44:43.959 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.16962", 2024-08-19T14:44:43.959 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:44:43.959 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 232619 2024-08-19T14:44:43.959 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.959 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:44:43.959 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.959 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:44:43.959 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.487667+0000", 2024-08-19T14:44:43.960 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:44:43.960 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.960 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.960 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:44:43.960 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.487667+0000", 2024-08-19T14:44:43.960 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:44:43.960 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.960 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.960 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:44:43.960 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.487668+0000", 2024-08-19T14:44:43.960 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2899999999999999e-06 2024-08-19T14:44:43.960 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.960 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.961 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:44:43.961 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.487673+0000", 2024-08-19T14:44:43.961 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.3900000000000003e-06 2024-08-19T14:44:43.961 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.961 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.961 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:44:43.961 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.487684+0000", 2024-08-19T14:44:43.961 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0827e-05 2024-08-19T14:44:43.961 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.961 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.961 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:44:43.961 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.487689+0000", 2024-08-19T14:44:43.961 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.8170000000000004e-06 2024-08-19T14:44:43.961 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.962 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.962 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:44:43.962 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.487742+0000", 2024-08-19T14:44:43.962 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2960000000000001e-05 2024-08-19T14:44:43.962 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.962 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.962 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:44:43.962 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.487864+0000", 2024-08-19T14:44:43.962 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012173200000000001 2024-08-19T14:44:43.962 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.962 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.962 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,3", 2024-08-19T14:44:43.962 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.487998+0000", 2024-08-19T14:44:43.962 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013379900000000001 2024-08-19T14:44:43.963 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.963 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.963 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:44:43.963 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.491865+0000", 2024-08-19T14:44:43.963 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0038671370000000001 2024-08-19T14:44:43.963 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.963 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.963 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:44:43.963 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.494174+0000", 2024-08-19T14:44:43.963 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0023090509999999999 2024-08-19T14:44:43.963 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.963 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.963 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:44:43.963 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.765707+0000", 2024-08-19T14:44:43.964 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.27153272000000001 2024-08-19T14:44:43.964 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.964 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.964 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:44:43.964 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.765764+0000", 2024-08-19T14:44:43.964 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7079e-05 2024-08-19T14:44:43.964 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.964 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.964 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:44:43.964 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:39:29.765806+0000", 2024-08-19T14:44:43.964 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.2345000000000002e-05 2024-08-19T14:44:43.964 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:44:43.964 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:44:43.964 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:44:43.965 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.965 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.965 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.26911.0:197684 8.17 8:ecc94a48:::benchmark_data_smithi045_137103_object85220:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e801)", 2024-08-19T14:44:43.965 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:43:16.205285+0000", 2024-08-19T14:44:43.965 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 87.690064657999997, 2024-08-19T14:44:43.965 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.51375581599999998, 2024-08-19T14:44:43.965 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:44:43.965 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:44:43.965 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-08-19T14:44:43.965 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:44:43.965 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.26911", 2024-08-19T14:44:43.965 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/3494993995", 2024-08-19T14:44:43.965 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 197684 2024-08-19T14:44:43.965 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.965 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:44:43.966 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.966 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:44:43.966 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:16.205285+0000", 2024-08-19T14:44:43.966 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:44:43.966 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.966 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.966 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:44:43.966 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:16.205285+0000", 2024-08-19T14:44:43.966 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:44:43.966 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.966 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.966 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:44:43.966 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:16.205290+0000", 2024-08-19T14:44:43.967 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.361e-06 2024-08-19T14:44:43.967 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.967 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.967 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:44:43.967 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:16.205293+0000", 2024-08-19T14:44:43.967 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.3459999999999998e-06 2024-08-19T14:44:43.967 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.967 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.967 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:44:43.967 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:16.205301+0000", 2024-08-19T14:44:43.967 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.374e-06 2024-08-19T14:44:43.967 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.967 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.967 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:44:43.968 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:16.205307+0000", 2024-08-19T14:44:43.968 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.1539999999999999e-06 2024-08-19T14:44:43.968 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.968 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.968 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:44:43.968 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:16.210314+0000", 2024-08-19T14:44:43.968 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0050069829999999996 2024-08-19T14:44:43.968 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.968 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.968 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for missing object", 2024-08-19T14:44:43.968 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:16.210424+0000", 2024-08-19T14:44:43.968 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000110019 2024-08-19T14:44:43.968 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.968 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.968 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:44:43.969 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:16.719019+0000", 2024-08-19T14:44:43.969 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.508595256 2024-08-19T14:44:43.969 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.969 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.969 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:44:43.969 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:16.719040+0000", 2024-08-19T14:44:43.969 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1322999999999999e-05 2024-08-19T14:44:43.969 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:44:43.969 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:44:43.969 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:44:43.969 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.969 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.969 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.26911.0:201718 8.17 8:ec3b6b96:::benchmark_data_smithi045_137103_object89254:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e801)", 2024-08-19T14:44:43.969 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:43:22.223517+0000", 2024-08-19T14:44:43.970 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 81.671832484999996, 2024-08-19T14:44:43.970 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.36028294, 2024-08-19T14:44:43.970 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:44:43.970 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:44:43.970 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-08-19T14:44:43.970 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:44:43.970 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.26911", 2024-08-19T14:44:43.970 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/3494993995", 2024-08-19T14:44:43.971 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 201718 2024-08-19T14:44:43.971 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.972 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:44:43.972 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.972 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:44:43.972 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:22.223517+0000", 2024-08-19T14:44:43.972 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:44:43.972 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.972 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.972 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:44:43.972 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:22.223517+0000", 2024-08-19T14:44:43.972 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:44:43.972 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.972 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.972 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:44:43.972 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:22.223518+0000", 2024-08-19T14:44:43.973 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6390000000000001e-06 2024-08-19T14:44:43.973 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.973 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.973 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:44:43.973 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:22.223555+0000", 2024-08-19T14:44:43.973 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.6325000000000002e-05 2024-08-19T14:44:43.973 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.973 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.973 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:44:43.973 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:22.223566+0000", 2024-08-19T14:44:43.973 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1294000000000001e-05 2024-08-19T14:44:43.973 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.973 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.973 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:44:43.974 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:22.223575+0000", 2024-08-19T14:44:43.974 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.8740000000000001e-06 2024-08-19T14:44:43.974 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.974 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.974 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:44:43.974 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:22.223647+0000", 2024-08-19T14:44:43.974 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.2279000000000004e-05 2024-08-19T14:44:43.974 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.974 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.974 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for missing object", 2024-08-19T14:44:43.974 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:22.223744+0000", 2024-08-19T14:44:43.974 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.6784e-05 2024-08-19T14:44:43.974 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.975 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.975 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:44:43.975 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:22.583783+0000", 2024-08-19T14:44:43.975 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.36003861300000001 2024-08-19T14:44:43.975 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.975 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.975 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:44:43.975 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:22.583800+0000", 2024-08-19T14:44:43.975 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7132000000000001e-05 2024-08-19T14:44:43.975 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:44:43.975 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:44:43.975 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:44:43.975 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.975 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.976 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.26911.0:201731 8.17 8:ecb992f9:::benchmark_data_smithi045_137103_object89267:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e801)", 2024-08-19T14:44:43.976 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:43:22.223598+0000", 2024-08-19T14:44:43.976 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 81.671751463000007, 2024-08-19T14:44:43.976 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.36147773700000002, 2024-08-19T14:44:43.976 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:44:43.976 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:44:43.976 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-08-19T14:44:43.976 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:44:43.976 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.26911", 2024-08-19T14:44:43.976 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/3494993995", 2024-08-19T14:44:43.976 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 201731 2024-08-19T14:44:43.976 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.976 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:44:43.976 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.977 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:44:43.977 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:22.223598+0000", 2024-08-19T14:44:43.977 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:44:43.977 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.977 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.977 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:44:43.977 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:22.223598+0000", 2024-08-19T14:44:43.977 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:44:43.977 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.977 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.977 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:44:43.977 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:22.223598+0000", 2024-08-19T14:44:43.977 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.9100000000000004e-07 2024-08-19T14:44:43.977 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.978 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.978 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:44:43.978 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:22.223601+0000", 2024-08-19T14:44:43.978 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.9830000000000001e-06 2024-08-19T14:44:43.978 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.978 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.978 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:44:43.978 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:22.223608+0000", 2024-08-19T14:44:43.978 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.6320000000000002e-06 2024-08-19T14:44:43.978 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.978 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.978 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:44:43.978 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:22.223613+0000", 2024-08-19T14:44:43.979 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.22e-06 2024-08-19T14:44:43.979 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.979 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.979 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:44:43.979 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:22.225163+0000", 2024-08-19T14:44:43.979 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0015494040000000001 2024-08-19T14:44:43.979 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.979 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.979 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for missing object", 2024-08-19T14:44:43.979 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:22.225267+0000", 2024-08-19T14:44:43.979 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000104558 2024-08-19T14:44:43.979 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.979 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.979 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:44:43.980 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:22.585065+0000", 2024-08-19T14:44:43.980 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.359798322 2024-08-19T14:44:43.980 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.980 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.980 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:44:43.980 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:22.585075+0000", 2024-08-19T14:44:43.980 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0027000000000001e-05 2024-08-19T14:44:43.980 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:44:43.980 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:44:43.980 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:44:43.980 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.980 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.980 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.26911.0:201920 8.17 8:ee9063b3:::benchmark_data_smithi045_137103_object89456:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e801)", 2024-08-19T14:44:43.980 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:43:22.536940+0000", 2024-08-19T14:44:43.981 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 81.358409006000002, 2024-08-19T14:44:43.981 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.48683706999999998, 2024-08-19T14:44:43.981 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:44:43.981 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:44:43.981 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-08-19T14:44:43.981 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:44:43.981 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.26911", 2024-08-19T14:44:43.981 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/3494993995", 2024-08-19T14:44:43.981 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 201920 2024-08-19T14:44:43.981 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.981 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:44:43.981 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.981 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:44:43.981 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:22.536940+0000", 2024-08-19T14:44:43.982 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:44:43.982 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.982 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.982 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:44:43.982 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:22.536940+0000", 2024-08-19T14:44:43.982 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:44:43.982 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.982 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.982 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:44:43.982 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:22.536941+0000", 2024-08-19T14:44:43.982 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.1199999999999998e-07 2024-08-19T14:44:43.982 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.982 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.982 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:44:43.983 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:22.536943+0000", 2024-08-19T14:44:43.983 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.621e-06 2024-08-19T14:44:43.983 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.983 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.983 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:44:43.983 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:22.536950+0000", 2024-08-19T14:44:43.983 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.6549999999999998e-06 2024-08-19T14:44:43.983 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.983 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.983 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:44:43.983 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:22.536956+0000", 2024-08-19T14:44:43.983 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.6489999999999996e-06 2024-08-19T14:44:43.983 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.983 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.984 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:44:43.984 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:22.537863+0000", 2024-08-19T14:44:43.984 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00090699400000000005 2024-08-19T14:44:43.984 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.984 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.984 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for missing object", 2024-08-19T14:44:43.984 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:22.537953+0000", 2024-08-19T14:44:43.984 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.0037000000000005e-05 2024-08-19T14:44:43.984 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.984 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.984 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:44:43.984 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:23.023762+0000", 2024-08-19T14:44:43.984 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.48580992200000001 2024-08-19T14:44:43.984 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.985 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.985 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:44:43.985 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:23.023777+0000", 2024-08-19T14:44:43.985 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4780000000000001e-05 2024-08-19T14:44:43.985 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:44:43.985 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:44:43.985 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:44:43.985 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.985 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.985 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.26911.0:202182 8.17 8:e9015a4c:::benchmark_data_smithi045_137103_object89718:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e801)", 2024-08-19T14:44:43.985 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:43:22.970702+0000", 2024-08-19T14:44:43.985 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 80.924646936000002, 2024-08-19T14:44:43.985 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.1532440560000001, 2024-08-19T14:44:43.985 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:44:43.986 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:44:43.986 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-08-19T14:44:43.986 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:44:43.986 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.26911", 2024-08-19T14:44:43.986 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/3494993995", 2024-08-19T14:44:43.986 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 202182 2024-08-19T14:44:43.986 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.986 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:44:43.986 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.986 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:44:43.986 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:22.970702+0000", 2024-08-19T14:44:43.986 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:44:43.986 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.987 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.987 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:44:43.987 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:22.970702+0000", 2024-08-19T14:44:43.987 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:44:43.987 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.987 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.987 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:44:43.987 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:22.970703+0000", 2024-08-19T14:44:43.987 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.0600000000000001e-07 2024-08-19T14:44:43.987 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.987 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.987 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:44:43.987 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:22.970705+0000", 2024-08-19T14:44:43.988 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.4430000000000002e-06 2024-08-19T14:44:43.988 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.988 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.988 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:44:43.988 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:22.970711+0000", 2024-08-19T14:44:43.988 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.6500000000000001e-06 2024-08-19T14:44:43.988 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.988 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.988 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:44:43.988 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:22.970716+0000", 2024-08-19T14:44:43.988 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0000000000000004e-06 2024-08-19T14:44:43.988 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.988 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.988 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:44:43.989 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:22.970957+0000", 2024-08-19T14:44:43.989 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00024096499999999999 2024-08-19T14:44:43.989 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.989 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.989 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for missing object", 2024-08-19T14:44:43.989 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:22.971036+0000", 2024-08-19T14:44:43.989 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.9688e-05 2024-08-19T14:44:43.989 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.989 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.989 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:44:43.989 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:26.123935+0000", 2024-08-19T14:44:43.989 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.1528983319999999 2024-08-19T14:44:43.989 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.989 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.990 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:44:43.990 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:26.123946+0000", 2024-08-19T14:44:43.990 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1572e-05 2024-08-19T14:44:43.990 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:44:43.990 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:44:43.990 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:44:43.990 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.990 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.990 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.26911.0:204420 8.17 8:edfef795:::benchmark_data_smithi045_137103_object91956:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e801)", 2024-08-19T14:44:43.990 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:43:26.442602+0000", 2024-08-19T14:44:43.990 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 77.452747017999997, 2024-08-19T14:44:43.990 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.55764363299999997, 2024-08-19T14:44:43.990 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:44:43.991 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:44:43.991 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-08-19T14:44:43.991 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:44:43.991 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.26911", 2024-08-19T14:44:43.991 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/3494993995", 2024-08-19T14:44:43.991 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 204420 2024-08-19T14:44:43.991 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.991 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:44:43.991 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.991 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:44:43.991 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:26.442602+0000", 2024-08-19T14:44:43.991 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:44:43.991 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.991 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.992 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:44:43.992 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:26.442602+0000", 2024-08-19T14:44:43.992 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:44:43.992 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.992 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.992 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:44:43.992 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:26.442603+0000", 2024-08-19T14:44:43.992 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.299e-06 2024-08-19T14:44:43.992 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.992 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.992 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:44:43.992 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:26.442608+0000", 2024-08-19T14:44:43.992 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.7700000000000001e-06 2024-08-19T14:44:43.993 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.993 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.993 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:44:43.993 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:26.442617+0000", 2024-08-19T14:44:43.993 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.6670000000000002e-06 2024-08-19T14:44:43.993 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.993 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.993 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:44:43.993 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:26.442624+0000", 2024-08-19T14:44:43.993 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.5599999999999996e-06 2024-08-19T14:44:43.993 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.993 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.993 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:44:43.993 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:26.445260+0000", 2024-08-19T14:44:43.994 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.002635465 2024-08-19T14:44:43.994 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.994 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.994 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for missing object", 2024-08-19T14:44:43.994 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:26.445354+0000", 2024-08-19T14:44:43.994 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.4092000000000006e-05 2024-08-19T14:44:43.994 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.994 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.994 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:44:43.994 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:27.000186+0000", 2024-08-19T14:44:43.994 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.55483226100000005 2024-08-19T14:44:43.994 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.994 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.995 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:44:43.995 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:27.000246+0000", 2024-08-19T14:44:43.995 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.9518999999999997e-05 2024-08-19T14:44:43.995 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:44:43.995 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:44:43.995 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:44:43.995 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.995 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.995 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.26911.0:214923 8.17 8:e8bf007e:::benchmark_data_smithi045_137103_object102459:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e807)", 2024-08-19T14:44:43.995 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:43:42.690868+0000", 2024-08-19T14:44:43.995 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 61.204480904999997, 2024-08-19T14:44:43.995 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.3503693710000002, 2024-08-19T14:44:43.995 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:44:43.995 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:44:43.996 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-08-19T14:44:43.996 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:44:43.996 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.26911", 2024-08-19T14:44:43.996 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/3494993995", 2024-08-19T14:44:43.996 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 214923 2024-08-19T14:44:43.996 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.996 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:44:43.996 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.996 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:44:43.996 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:42.690868+0000", 2024-08-19T14:44:43.996 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:44:43.996 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.996 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.996 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:44:43.997 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:42.690868+0000", 2024-08-19T14:44:43.997 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:44:43.997 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.997 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.997 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:44:43.997 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:42.690869+0000", 2024-08-19T14:44:43.997 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.41e-07 2024-08-19T14:44:43.997 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.997 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.997 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:44:43.997 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:42.690871+0000", 2024-08-19T14:44:43.997 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9910000000000001e-06 2024-08-19T14:44:43.997 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.997 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.998 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:44:43.998 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:42.690882+0000", 2024-08-19T14:44:43.998 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1443e-05 2024-08-19T14:44:43.998 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.998 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.998 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:44:43.998 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:42.690887+0000", 2024-08-19T14:44:43.998 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2569999999999998e-06 2024-08-19T14:44:43.998 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.998 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.998 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:44:43.998 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:42.707328+0000", 2024-08-19T14:44:43.998 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.016440293000000002 2024-08-19T14:44:43.998 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.999 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.999 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for missing object", 2024-08-19T14:44:43.999 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:42.707438+0000", 2024-08-19T14:44:43.999 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000110401 2024-08-19T14:44:43.999 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.999 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.999 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:44:43.999 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:46.041227+0000", 2024-08-19T14:44:43.999 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.3337893329999999 2024-08-19T14:44:43.999 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:43.999 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:43.999 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:44:44.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:44.000 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:44.000 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:44:44.001 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:46.041238+0000", 2024-08-19T14:44:44.001 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0312000000000001e-05 2024-08-19T14:44:44.001 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:44:44.001 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:44:44.001 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:44:44.001 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:44.001 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:44.002 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.26911.0:214923 8.17 8:e8bf007e:::benchmark_data_smithi045_137103_object102459:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e807)", 2024-08-19T14:44:44.002 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:43:42.743747+0000", 2024-08-19T14:44:44.002 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 61.151602466, 2024-08-19T14:44:44.002 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.2975287130000002, 2024-08-19T14:44:44.002 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:44:44.002 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:44:44.002 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-08-19T14:44:44.002 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:44:44.002 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.26911", 2024-08-19T14:44:44.002 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/3494993995", 2024-08-19T14:44:44.002 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 214923 2024-08-19T14:44:44.002 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:44.002 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:44:44.002 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:44.003 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:44:44.003 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:42.743747+0000", 2024-08-19T14:44:44.003 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:44:44.003 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:44.003 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:44.003 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:44:44.003 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:42.743747+0000", 2024-08-19T14:44:44.003 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:44:44.003 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:44.003 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:44.003 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:44:44.003 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:42.743748+0000", 2024-08-19T14:44:44.003 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3820000000000001e-06 2024-08-19T14:44:44.004 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:44.004 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:44.004 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:44:44.004 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:42.743752+0000", 2024-08-19T14:44:44.004 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.2950000000000004e-06 2024-08-19T14:44:44.004 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:44.004 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:44.004 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:44:44.004 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:42.743761+0000", 2024-08-19T14:44:44.004 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.1759999999999993e-06 2024-08-19T14:44:44.004 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:44.004 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:44.004 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:44:44.004 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:42.743768+0000", 2024-08-19T14:44:44.004 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.3980000000000002e-06 2024-08-19T14:44:44.005 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:44.005 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:44.005 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:44:44.005 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:42.743939+0000", 2024-08-19T14:44:44.005 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000170855 2024-08-19T14:44:44.005 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:44.005 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:44.005 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for missing object", 2024-08-19T14:44:44.005 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:42.744126+0000", 2024-08-19T14:44:44.005 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000187493 2024-08-19T14:44:44.005 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:44.005 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:44.005 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:44:44.005 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:46.041266+0000", 2024-08-19T14:44:44.006 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.2971391759999999 2024-08-19T14:44:44.006 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:44.006 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:44.006 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:44:44.006 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:46.041275+0000", 2024-08-19T14:44:44.006 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.9380000000000004e-06 2024-08-19T14:44:44.006 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:44:44.006 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:44:44.006 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:44:44.006 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:44.006 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:44.006 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.26911.0:214926 8.17 8:ec29e4df:::benchmark_data_smithi045_137103_object102462:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e807)", 2024-08-19T14:44:44.006 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:43:42.743803+0000", 2024-08-19T14:44:44.007 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 61.151545937999998, 2024-08-19T14:44:44.007 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.2972891880000001, 2024-08-19T14:44:44.007 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:44:44.007 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:44:44.007 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-08-19T14:44:44.007 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:44:44.007 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.26911", 2024-08-19T14:44:44.007 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/3494993995", 2024-08-19T14:44:44.007 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 214926 2024-08-19T14:44:44.007 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:44.007 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:44:44.007 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:44.007 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:44:44.007 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:42.743803+0000", 2024-08-19T14:44:44.007 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:44:44.008 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:44.008 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:44.008 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:44:44.008 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:42.743803+0000", 2024-08-19T14:44:44.008 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:44:44.008 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:44.008 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:44.008 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:44:44.008 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:42.743804+0000", 2024-08-19T14:44:44.008 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.5499999999999998e-07 2024-08-19T14:44:44.008 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:44.008 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:44.008 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:44:44.008 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:42.743807+0000", 2024-08-19T14:44:44.009 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.7379999999999999e-06 2024-08-19T14:44:44.009 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:44.009 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:44.009 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:44:44.009 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:42.743813+0000", 2024-08-19T14:44:44.009 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.5599999999999999e-06 2024-08-19T14:44:44.009 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:44.009 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:44.009 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:44:44.009 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:42.743818+0000", 2024-08-19T14:44:44.009 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1950000000000002e-06 2024-08-19T14:44:44.009 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:44.009 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:44.009 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:44:44.010 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:42.744190+0000", 2024-08-19T14:44:44.010 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00037210900000000002 2024-08-19T14:44:44.010 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:44.010 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:44.010 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for missing object", 2024-08-19T14:44:44.010 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:42.744356+0000", 2024-08-19T14:44:44.010 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00016553999999999999 2024-08-19T14:44:44.010 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:44.010 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:44.010 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:44:44.010 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:46.041081+0000", 2024-08-19T14:44:44.010 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.296724679 2024-08-19T14:44:44.010 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:44.011 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:44.011 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:44:44.011 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:46.041092+0000", 2024-08-19T14:44:44.011 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1812e-05 2024-08-19T14:44:44.011 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:44:44.011 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:44:44.011 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:44:44.011 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:44.011 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:44.011 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.26911.0:214935 8.17 8:ecfe546f:::benchmark_data_smithi045_137103_object102471:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e807)", 2024-08-19T14:44:44.011 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:43:42.743826+0000", 2024-08-19T14:44:44.011 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 61.151523087000001, 2024-08-19T14:44:44.011 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.3205305489999999, 2024-08-19T14:44:44.012 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:44:44.012 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:44:44.012 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-08-19T14:44:44.012 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:44:44.012 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.26911", 2024-08-19T14:44:44.012 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/3494993995", 2024-08-19T14:44:44.012 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 214935 2024-08-19T14:44:44.012 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:44.012 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:44:44.012 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:44.012 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:44:44.012 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:42.743826+0000", 2024-08-19T14:44:44.012 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:44:44.012 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:44.012 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:44.013 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:44:44.013 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:42.743826+0000", 2024-08-19T14:44:44.013 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:44:44.013 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:44.013 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:44.013 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:44:44.013 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:42.743826+0000", 2024-08-19T14:44:44.013 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.53e-07 2024-08-19T14:44:44.013 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:44.013 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:44.013 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:44:44.013 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:42.743828+0000", 2024-08-19T14:44:44.013 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9460000000000001e-06 2024-08-19T14:44:44.013 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:44.014 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:44.014 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:44:44.014 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:42.743834+0000", 2024-08-19T14:44:44.014 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4210000000000003e-06 2024-08-19T14:44:44.014 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:44.014 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:44.014 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:44:44.014 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:42.743838+0000", 2024-08-19T14:44:44.014 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.583e-06 2024-08-19T14:44:44.014 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:44.014 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:44.014 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:44:44.014 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:42.744385+0000", 2024-08-19T14:44:44.014 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00054647699999999997 2024-08-19T14:44:44.015 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:44.015 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:44.015 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for missing object", 2024-08-19T14:44:44.015 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:42.744462+0000", 2024-08-19T14:44:44.015 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.6909999999999994e-05 2024-08-19T14:44:44.015 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:44.015 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:44.015 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:44:44.015 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:46.064345+0000", 2024-08-19T14:44:44.015 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.3198834509999999 2024-08-19T14:44:44.015 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:44:44.015 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:44:44.015 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:44:44.015 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:43:46.064357+0000", 2024-08-19T14:44:44.015 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1408e-05 2024-08-19T14:44:44.016 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:44:44.016 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:44:44.016 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:44:44.016 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:44:44.016 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:44:44.016 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:44:44.101 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:44:44.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:44.137 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:44.138 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:44:44.239 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:44:44.253 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is unset 2024-08-19T14:44:44.278 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:44.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:44.280 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:44:44.311 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 23 16 37798 37782 102.654 52.25 0.0122605 0.00973268 2024-08-19T14:44:44.380 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:44:44.401 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:44.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:44.402 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:44:44.503 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:44:44.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:44 smithi032 ceph-mon[20004]: pgmap v1983: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 1 peering, 37 active+clean; 2.1 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 38 MiB/s wr, 611 op/s; 1/101694 objects degraded (0.001%); 16160/101694 objects misplaced (15.891%); 15 MiB/s, 231 objects/s recovering 2024-08-19T14:44:44.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:44 smithi032 ceph-mon[20004]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-08-19T14:44:44.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:44 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:44:44.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:44 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:44:44.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:44 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1913241755' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:44:44.533 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:44.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:44.535 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:44:44.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:44 smithi045 ceph-mon[20143]: pgmap v1983: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 1 peering, 37 active+clean; 2.1 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 38 MiB/s wr, 611 op/s; 1/101694 objects degraded (0.001%); 16160/101694 objects misplaced (15.891%); 15 MiB/s, 231 objects/s recovering 2024-08-19T14:44:44.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:44 smithi045 ceph-mon[20143]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-08-19T14:44:44.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:44 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:44:44.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:44 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:44:44.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:44 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1913241755' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:44:44.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:44 smithi031 ceph-mon[16336]: pgmap v1983: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 1 peering, 37 active+clean; 2.1 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 38 MiB/s wr, 611 op/s; 1/101694 objects degraded (0.001%); 16160/101694 objects misplaced (15.891%); 15 MiB/s, 231 objects/s recovering 2024-08-19T14:44:44.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:44 smithi031 ceph-mon[16336]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-08-19T14:44:44.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:44 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:44:44.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:44 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:44:44.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:44 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1913241755' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:44:44.636 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:44:44.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:44.685 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:44.685 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:44:44.786 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:44:44.826 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:44.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:44.827 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:44:44.893 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-08-19T14:44:44.928 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:44:44.987 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:44.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:44.988 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:44:45.090 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:44:45.119 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:45.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:45.120 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:44:45.221 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:44:45.248 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:45.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:45.249 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:44:45.312 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 24 16 38699 38683 100.716 56.3125 0.0105829 0.00992219 2024-08-19T14:44:45.350 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:44:45.377 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:44:45.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:45.409 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:45.409 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:44:45.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:45 smithi032 ceph-mon[20004]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:44:45.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:45 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1913241755' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:44:45.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:45 smithi032 ceph-mon[20004]: osdmap e840: 12 total, 12 up, 7 in 2024-08-19T14:44:45.510 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:44:45.524 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:45 smithi045 ceph-mon[20143]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:44:45.524 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:45 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1913241755' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:44:45.524 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:45 smithi045 ceph-mon[20143]: osdmap e840: 12 total, 12 up, 7 in 2024-08-19T14:44:45.571 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:45.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:45.572 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:44:45.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:45 smithi031 ceph-mon[16336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:44:45.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:45 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1913241755' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:44:45.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:45 smithi031 ceph-mon[16336]: osdmap e840: 12 total, 12 up, 7 in 2024-08-19T14:44:45.673 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:44:45.699 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:45.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:45.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:44:45.801 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:44:45.829 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:45.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:45.830 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:44:45.931 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:44:45.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:45.970 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:45.970 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:44:46.022 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:44:46.022 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [], 2024-08-19T14:44:46.022 INFO:teuthology.orchestra.run.smithi045.stdout: "num_ops": 0 2024-08-19T14:44:46.022 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:44:46.071 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:44:46.092 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:46.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:46.093 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:44:46.194 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:44:46.219 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:46.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:46.220 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:44:46.312 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 25 16 39531 39515 98.7676 52 0.0269669 0.0101165 2024-08-19T14:44:46.320 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:44:46.343 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:46.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:46.344 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:44:46.445 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:44:46.466 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:46.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:46.467 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:44:46.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:46 smithi045 ceph-mon[20143]: pgmap v1985: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 38 active+clean; 2.2 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 51 MiB/s wr, 816 op/s; 1/107697 objects degraded (0.001%); 14342/107697 objects misplaced (13.317%); 31 MiB/s, 497 objects/s recovering 2024-08-19T14:44:46.568 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:44:46.589 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:46.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:46.590 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:44:46.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:46 smithi031 ceph-mon[16336]: pgmap v1985: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 38 active+clean; 2.2 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 51 MiB/s wr, 816 op/s; 1/107697 objects degraded (0.001%); 14342/107697 objects misplaced (13.317%); 31 MiB/s, 497 objects/s recovering 2024-08-19T14:44:46.691 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:44:46.717 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:46.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:46.718 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:44:46.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:46 smithi032 ceph-mon[20004]: pgmap v1985: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 38 active+clean; 2.2 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 51 MiB/s wr, 816 op/s; 1/107697 objects degraded (0.001%); 14342/107697 objects misplaced (13.317%); 31 MiB/s, 497 objects/s recovering 2024-08-19T14:44:46.819 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:44:46.847 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:46.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:46.848 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:44:46.949 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:44:46.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:46.971 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:46.971 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:44:47.072 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:44:47.097 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:47.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:47.098 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:44:47.199 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:44:47.236 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:47.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:47.238 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:44:47.312 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 26 16 40320 40304 96.8655 49.3125 0.0103227 0.010283 2024-08-19T14:44:47.339 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:44:47.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:47.361 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:47.361 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:44:47.462 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:44:47.490 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:47.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:47.491 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:44:47.591 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:44:47.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:47 smithi031 ceph-mon[16336]: pgmap v1986: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 38 active+clean; 2.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 55 MiB/s wr, 885 op/s; 1/113784 objects degraded (0.001%); 13526/113784 objects misplaced (11.887%); 30 MiB/s, 486 objects/s recovering 2024-08-19T14:44:47.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:47 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 1/113784 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-08-19T14:44:47.619 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:47.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:47.621 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:44:47.722 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:44:47.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:47 smithi032 ceph-mon[20004]: pgmap v1986: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 38 active+clean; 2.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 55 MiB/s wr, 885 op/s; 1/113784 objects degraded (0.001%); 13526/113784 objects misplaced (11.887%); 30 MiB/s, 486 objects/s recovering 2024-08-19T14:44:47.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:47 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 1/113784 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-08-19T14:44:47.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:47.763 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:47.763 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:44:47.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:47 smithi045 ceph-mon[20143]: pgmap v1986: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 38 active+clean; 2.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 55 MiB/s wr, 885 op/s; 1/113784 objects degraded (0.001%); 13526/113784 objects misplaced (11.887%); 30 MiB/s, 486 objects/s recovering 2024-08-19T14:44:47.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:47 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 1/113784 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-08-19T14:44:47.864 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:44:47.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:47.887 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:47.887 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:44:47.988 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:44:48.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:48.010 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:48.011 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:44:48.112 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:44:48.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:48.133 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:48.133 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:44:48.235 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:44:48.258 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:48.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:48.259 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:44:48.313 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 27 16 41148 41132 95.1946 51.75 0.00524845 0.0104764 2024-08-19T14:44:48.360 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:44:48.382 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:48.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:48.383 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:44:48.484 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:44:48.505 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:48.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:48.506 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:44:48.607 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:44:48.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:48.630 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:48.630 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:44:48.731 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:44:48.765 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:48.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:48.766 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:44:48.867 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:44:48.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:48.902 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:48.902 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:44:49.003 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:44:49.025 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:49.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:49.026 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:44:49.102 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-08-19T14:44:49.127 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:44:49.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:49.150 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:49.150 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:44:49.252 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:44:49.301 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:49.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:49.303 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:44:49.313 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 28 16 41477 41461 92.5293 20.5625 0.0240799 0.0107994 2024-08-19T14:44:49.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:49 smithi031 ceph-mon[16336]: osdmap e841: 12 total, 12 up, 7 in 2024-08-19T14:44:49.405 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:44:49.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:49.474 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:49.474 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:44:49.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:49 smithi032 ceph-mon[20004]: osdmap e841: 12 total, 12 up, 7 in 2024-08-19T14:44:49.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:49 smithi045 ceph-mon[20143]: osdmap e841: 12 total, 12 up, 7 in 2024-08-19T14:44:49.575 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:44:49.594 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:44:49.598 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:49.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:49.602 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:44:49.703 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:44:49.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:49.725 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:49.725 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:44:49.826 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:44:49.847 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:49.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:49.848 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:44:49.949 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:44:49.986 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:49.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:49.987 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:44:50.088 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:44:50.111 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:50.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:50.112 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:44:50.214 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:44:50.239 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:50.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:50.240 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:44:50.313 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 29 16 42347 42331 91.2136 54.375 0.00327157 0.0109558 2024-08-19T14:44:50.316 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:50 smithi031 ceph-mon[16336]: pgmap v1988: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 38 active+clean; 2.3 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 46 MiB/s wr, 730 op/s; 1/115119 objects degraded (0.001%); 13003/115119 objects misplaced (11.295%); 32 MiB/s, 515 objects/s recovering 2024-08-19T14:44:50.316 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:50 smithi031 ceph-mon[16336]: osdmap e842: 12 total, 12 up, 7 in 2024-08-19T14:44:50.342 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:44:50.368 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:50.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:50.369 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:44:50.470 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:44:50.474 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd reweight 4 1 2024-08-19T14:44:50.493 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:44:50.496 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:50.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:50.498 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:44:50.509 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:50 smithi032 ceph-mon[20004]: pgmap v1988: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 38 active+clean; 2.3 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 46 MiB/s wr, 730 op/s; 1/115119 objects degraded (0.001%); 13003/115119 objects misplaced (11.295%); 32 MiB/s, 515 objects/s recovering 2024-08-19T14:44:50.509 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:50 smithi032 ceph-mon[20004]: osdmap e842: 12 total, 12 up, 7 in 2024-08-19T14:44:50.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:50 smithi045 ceph-mon[20143]: pgmap v1988: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 38 active+clean; 2.3 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 46 MiB/s wr, 730 op/s; 1/115119 objects degraded (0.001%); 13003/115119 objects misplaced (11.295%); 32 MiB/s, 515 objects/s recovering 2024-08-19T14:44:50.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:50 smithi045 ceph-mon[20143]: osdmap e842: 12 total, 12 up, 7 in 2024-08-19T14:44:50.599 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:44:50.640 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:50.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:50.641 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:44:50.742 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:44:50.769 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:50.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:50.770 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:44:50.872 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:44:50.896 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:50.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:50.897 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:44:50.998 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:44:51.023 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:51.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:51.024 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:44:51.064 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:44:51.065 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:44:51.093 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:44:51.125 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:44:51.150 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:51.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:51.151 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:44:51.214 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:51 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3109113679' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:44:51.252 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:44:51.288 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:44:51.288 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:44:51.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:51.371 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:51.371 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:44:51.371 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 30 16 43185 43169 89.919 52.375 0.0122512 0.0111143 2024-08-19T14:44:51.372 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:44:51.372 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:44:51.373 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:44:51.373 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:44:51.373 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:44:51.373 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:44:51.388 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:44:51.388 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:44:51.452 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:44:51.452 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:44:51.472 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:44:51.473 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:44:51.474 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:44:51.481 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:51 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3109113679' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:44:51.494 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:51.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:51.495 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:44:51.499 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:44:51.499 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:44:51.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:51 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3109113679' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:44:51.528 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:44:51.528 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:44:51.549 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:44:51.550 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:44:51.577 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:44:51.577 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:44:51.596 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:44:51.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:51.630 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:51.630 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:44:51.730 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:44:51.779 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:51.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:51.780 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:44:51.810 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:44:51.811 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:44:51.881 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:44:51.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:51.904 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:51.904 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:44:52.005 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:44:52.029 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:52.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:52.030 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:44:52.097 INFO:teuthology.orchestra.run.smithi031.stderr:reweighted osd.4 to 1 (10000) 2024-08-19T14:44:52.131 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:44:52.152 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:52.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:52.153 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:44:52.254 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:44:52.303 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:52.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:52.304 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:44:52.312 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:52 smithi031 ceph-mon[16336]: pgmap v1990: 47 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 38 active+clean; 2.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 44 MiB/s wr, 699 op/s; 1/120495 objects degraded (0.001%); 11250/120495 objects misplaced (9.336%); 32 MiB/s, 506 objects/s recovering 2024-08-19T14:44:52.312 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:52 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2815964478' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-08-19T14:44:52.312 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:52 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-08-19T14:44:52.312 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:52 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:44:52.313 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 31 16 44060 44044 88.781 54.6875 0.047766 0.0112559 2024-08-19T14:44:52.358 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-19T14:44:52.401 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:52 smithi045 ceph-mon[20143]: pgmap v1990: 47 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 38 active+clean; 2.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 44 MiB/s wr, 699 op/s; 1/120495 objects degraded (0.001%); 11250/120495 objects misplaced (9.336%); 32 MiB/s, 506 objects/s recovering 2024-08-19T14:44:52.401 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:52 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2815964478' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-08-19T14:44:52.401 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:52 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-08-19T14:44:52.401 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:52 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:44:52.405 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:44:52.430 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:52.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:52.431 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:44:52.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:52 smithi032 ceph-mon[20004]: pgmap v1990: 47 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 38 active+clean; 2.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 44 MiB/s wr, 699 op/s; 1/120495 objects degraded (0.001%); 11250/120495 objects misplaced (9.336%); 32 MiB/s, 506 objects/s recovering 2024-08-19T14:44:52.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:52 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2815964478' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-08-19T14:44:52.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:52 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-08-19T14:44:52.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:52 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:44:52.532 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:44:52.553 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:52.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:52.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:44:52.655 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:44:52.703 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:52.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:52.704 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:44:52.760 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-08-19T14:44:52.805 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:44:52.867 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:52.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:52.868 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:44:52.969 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:44:53.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:53.028 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:53.028 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:44:53.071 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd reweight 8 1 2024-08-19T14:44:53.126 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:44:53.129 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:44:53.150 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:53.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:53.151 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:44:53.252 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:44:53.270 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:44:53.279 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:53.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:53.280 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:44:53.285 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:53 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-08-19T14:44:53.285 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:53 smithi031 ceph-mon[16336]: osdmap e843: 12 total, 12 up, 7 in 2024-08-19T14:44:53.285 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:53 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:44:53.285 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:53 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:44:53.285 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:53 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:44:53.285 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:53 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 1/120495 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-08-19T14:44:53.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:53 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-08-19T14:44:53.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:53 smithi045 ceph-mon[20143]: osdmap e843: 12 total, 12 up, 7 in 2024-08-19T14:44:53.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:53 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:44:53.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:53 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:44:53.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:53 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:44:53.298 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:53 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 1/120495 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-08-19T14:44:53.314 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 32 16 44873 44857 87.5945 50.8125 0.0224834 0.0114063 2024-08-19T14:44:53.381 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:44:53.402 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:53.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:53.403 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:44:53.408 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:53 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-08-19T14:44:53.408 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:53 smithi032 ceph-mon[20004]: osdmap e843: 12 total, 12 up, 7 in 2024-08-19T14:44:53.408 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:53 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:44:53.408 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:53 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:44:53.408 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:53 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:44:53.408 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:53 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 1/120495 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-08-19T14:44:53.504 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:44:53.527 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:53.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:53.528 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:44:53.629 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:44:53.666 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:53.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:53.667 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:44:53.768 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:44:53.811 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:53.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:53.812 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:44:53.899 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:44:53.913 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:44:53.935 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:53.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:53.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:44:54.037 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:44:54.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:54.067 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:54.068 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:44:54.169 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:44:54.191 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:54.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:54.192 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:44:54.293 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:44:54.314 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 33 16 45791 45775 86.6787 57.375 0.011183 0.01153 2024-08-19T14:44:54.320 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:54.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:54.321 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:44:54.350 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:54 smithi031 ceph-mon[16336]: pgmap v1992: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 7 active+recovering+undersized+remapped, 38 active+clean; 2.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 40 MiB/s wr, 639 op/s; 1/125298 objects degraded (0.001%); 10162/125298 objects misplaced (8.110%); 35 MiB/s, 560 objects/s recovering 2024-08-19T14:44:54.350 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:54 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:44:54.350 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:54 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:44:54.350 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:54 smithi031 ceph-mon[16336]: osdmap e844: 12 total, 12 up, 7 in 2024-08-19T14:44:54.357 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:44:54.421 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:44:54.462 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:54.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:54.463 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:44:54.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:54 smithi032 ceph-mon[20004]: pgmap v1992: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 7 active+recovering+undersized+remapped, 38 active+clean; 2.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 40 MiB/s wr, 639 op/s; 1/125298 objects degraded (0.001%); 10162/125298 objects misplaced (8.110%); 35 MiB/s, 560 objects/s recovering 2024-08-19T14:44:54.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:54 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:44:54.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:54 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:44:54.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:54 smithi032 ceph-mon[20004]: osdmap e844: 12 total, 12 up, 7 in 2024-08-19T14:44:54.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:54 smithi045 ceph-mon[20143]: pgmap v1992: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 7 active+recovering+undersized+remapped, 38 active+clean; 2.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 40 MiB/s wr, 639 op/s; 1/125298 objects degraded (0.001%); 10162/125298 objects misplaced (8.110%); 35 MiB/s, 560 objects/s recovering 2024-08-19T14:44:54.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:54 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:44:54.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:54 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:44:54.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:54 smithi045 ceph-mon[20143]: osdmap e844: 12 total, 12 up, 7 in 2024-08-19T14:44:54.565 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:44:54.587 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:54.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:54.588 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:44:54.689 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:44:54.714 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:44:54.737 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:54.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:54.738 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:44:54.839 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:44:54.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:54.862 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:54.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:44:54.964 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:44:55.001 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:55.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:55.002 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:44:55.103 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:44:55.125 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:55.125 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is unset 2024-08-19T14:44:55.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:55.126 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:44:55.127 INFO:teuthology.orchestra.run.smithi031.stderr:reweighted osd.8 to 1 (10000) 2024-08-19T14:44:55.221 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:55 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/515510077' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:44:55.221 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:55 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3298839098' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-08-19T14:44:55.221 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:55 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-08-19T14:44:55.227 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:44:55.252 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:55.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:55.253 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:44:55.314 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 34 16 46511 46495 85.4529 45 0.00657149 0.0116873 2024-08-19T14:44:55.354 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:44:55.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:55 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/515510077' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:44:55.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:55 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3298839098' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-08-19T14:44:55.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:55 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-08-19T14:44:55.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:55.396 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:55.397 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:44:55.497 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:44:55.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:55 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/515510077' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:44:55.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:55 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3298839098' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-08-19T14:44:55.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:55 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-08-19T14:44:55.534 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:55.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:55.535 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:44:55.636 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:44:55.658 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:55.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:55.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:44:55.760 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:44:55.802 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:55.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:55.803 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:44:55.904 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:44:55.941 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:55.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:55.942 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:44:56.043 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:44:56.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:56.080 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:56.080 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:44:56.181 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:44:56.231 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:56.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:56.232 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:44:56.314 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 35 16 47258 47242 84.3453 46.6875 0.00589461 0.0118481 2024-08-19T14:44:56.332 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:44:56.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:56.358 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:56.359 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:44:56.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:56 smithi031 ceph-mon[16336]: pgmap v1994: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 2.7 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 56 MiB/s wr, 889 op/s; 1/131139 objects degraded (0.001%); 8627/131139 objects misplaced (6.579%); 46 MiB/s, 729 objects/s recovering 2024-08-19T14:44:56.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:56 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:44:56.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:56 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:44:56.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:56 smithi031 ceph-mon[16336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:44:56.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:56 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/515510077' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:44:56.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:56 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-08-19T14:44:56.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:56 smithi031 ceph-mon[16336]: osdmap e845: 12 total, 12 up, 7 in 2024-08-19T14:44:56.460 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:44:56.484 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:56.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:56.485 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:44:56.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:56 smithi032 ceph-mon[20004]: pgmap v1994: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 2.7 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 56 MiB/s wr, 889 op/s; 1/131139 objects degraded (0.001%); 8627/131139 objects misplaced (6.579%); 46 MiB/s, 729 objects/s recovering 2024-08-19T14:44:56.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:56 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:44:56.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:56 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:44:56.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:56 smithi032 ceph-mon[20004]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:44:56.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:56 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/515510077' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:44:56.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:56 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-08-19T14:44:56.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:56 smithi032 ceph-mon[20004]: osdmap e845: 12 total, 12 up, 7 in 2024-08-19T14:44:56.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:56 smithi045 ceph-mon[20143]: pgmap v1994: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 2.7 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 56 MiB/s wr, 889 op/s; 1/131139 objects degraded (0.001%); 8627/131139 objects misplaced (6.579%); 46 MiB/s, 729 objects/s recovering 2024-08-19T14:44:56.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:56 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:44:56.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:56 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:44:56.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:56 smithi045 ceph-mon[20143]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:44:56.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:56 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/515510077' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:44:56.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:56 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-08-19T14:44:56.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:56 smithi045 ceph-mon[20143]: osdmap e845: 12 total, 12 up, 7 in 2024-08-19T14:44:56.586 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:44:56.628 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:56.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:56.630 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:44:56.731 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:44:56.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:56.756 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:56.756 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:44:56.857 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:44:56.907 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:56.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:56.908 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:44:57.009 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:44:57.030 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:57.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:57.031 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:44:57.132 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:44:57.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:57.160 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:57.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:44:57.262 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:44:57.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:57.285 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:57.285 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:44:57.314 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 36 16 48071 48055 83.4137 50.8125 0.0153292 0.0119764 2024-08-19T14:44:57.386 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:44:57.428 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:57.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:57.429 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:44:57.530 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:44:57.570 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:57.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:57.571 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:44:57.672 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:44:57.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:57.716 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:57.716 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:44:57.716 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:44:57.716 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:44:57.816 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:44:57.837 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:57.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:57.839 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:44:57.939 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:44:57.982 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:57.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:57.983 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:44:58.084 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:44:58.106 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:58.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:58.107 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:44:58.208 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:44:58.235 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:58.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:58.236 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:44:58.257 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:44:58.258 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:44:58.280 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:44:58.281 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:44:58.302 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:44:58.303 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:44:58.314 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 37 16 48908 48892 82.5731 52.3125 0.0135741 0.0121039 2024-08-19T14:44:58.323 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:44:58.324 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:44:58.337 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:44:58.341 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:44:58.342 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:44:58.358 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:58.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:58.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:44:58.360 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:44:58.360 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:44:58.383 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:44:58.383 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:44:58.384 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:58 smithi031 ceph-mon[16336]: pgmap v1996: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 2.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 59 MiB/s wr, 939 op/s; 1/137418 objects degraded (0.001%); 7771/137418 objects misplaced (5.655%); 36 MiB/s, 579 objects/s recovering 2024-08-19T14:44:58.384 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:58 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 1/137418 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-08-19T14:44:58.402 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:44:58.402 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:44:58.427 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:44:58.428 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:44:58.459 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:44:58.460 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:44:58.460 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:44:58.491 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:44:58.491 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:44:58.503 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:58.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:58.504 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:44:58.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:58 smithi032 ceph-mon[20004]: pgmap v1996: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 2.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 59 MiB/s wr, 939 op/s; 1/137418 objects degraded (0.001%); 7771/137418 objects misplaced (5.655%); 36 MiB/s, 579 objects/s recovering 2024-08-19T14:44:58.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:58 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 1/137418 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-08-19T14:44:58.522 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:44:58.523 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:44:58.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:58 smithi045 ceph-mon[20143]: pgmap v1996: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 2.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 59 MiB/s wr, 939 op/s; 1/137418 objects degraded (0.001%); 7771/137418 objects misplaced (5.655%); 36 MiB/s, 579 objects/s recovering 2024-08-19T14:44:58.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:58 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 1/137418 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-08-19T14:44:58.605 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:44:58.626 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:58.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:58.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:44:58.728 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:44:58.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:58.770 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:58.770 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:44:58.871 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:44:58.893 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:58.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:58.894 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:44:58.995 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:44:59.021 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:59.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:59.022 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:44:59.123 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:44:59.188 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:44:59.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:59.189 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:44:59.289 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:44:59.311 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:59.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:59.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:44:59.314 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 38 16 49657 49641 81.632 46.8125 0.00334138 0.0122214 2024-08-19T14:44:59.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:44:59 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1925355656' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:44:59.413 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:44:59.444 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:59.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:59.445 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:44:59.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:44:59 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1925355656' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:44:59.546 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:44:59.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:44:59 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1925355656' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:44:59.579 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:59.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:59.580 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:44:59.681 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:44:59.705 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:59.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:59.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:44:59.807 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:44:59.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:59.831 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:44:59.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:44:59.932 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:44:59.966 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:44:59.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:44:59.968 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:45:00.068 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:45:00.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:00.091 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:00.091 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:45:00.192 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:45:00.214 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:00.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:00.215 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:45:00.314 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 39 16 49811 49795 79.7858 9.625 0.00685074 0.0125246 2024-08-19T14:45:00.316 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:45:00.318 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd reweight 2 1 2024-08-19T14:45:00.353 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:00.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:00.354 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:45:00.455 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:45:00.480 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:00 smithi031 ceph-mon[16336]: pgmap v1997: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 2.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 45 MiB/s wr, 724 op/s; 1/138906 objects degraded (0.001%); 7174/138906 objects misplaced (5.165%); 30 MiB/s, 477 objects/s recovering 2024-08-19T14:45:00.480 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:00 smithi031 ceph-mon[16336]: osdmap e846: 12 total, 12 up, 7 in 2024-08-19T14:45:00.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:00.487 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:00.487 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:45:00.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:00 smithi032 ceph-mon[20004]: pgmap v1997: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 2.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 45 MiB/s wr, 724 op/s; 1/138906 objects degraded (0.001%); 7174/138906 objects misplaced (5.165%); 30 MiB/s, 477 objects/s recovering 2024-08-19T14:45:00.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:00 smithi032 ceph-mon[20004]: osdmap e846: 12 total, 12 up, 7 in 2024-08-19T14:45:00.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:00 smithi045 ceph-mon[20143]: pgmap v1997: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 2.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 45 MiB/s wr, 724 op/s; 1/138906 objects degraded (0.001%); 7174/138906 objects misplaced (5.165%); 30 MiB/s, 477 objects/s recovering 2024-08-19T14:45:00.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:00 smithi045 ceph-mon[20143]: osdmap e846: 12 total, 12 up, 7 in 2024-08-19T14:45:00.588 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:45:00.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:00.616 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:00.616 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:45:00.716 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:45:00.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:00.741 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:00.741 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:45:00.842 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:45:00.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:00.864 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:00.865 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:45:00.965 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:45:00.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:00.992 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:00.993 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:45:01.026 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:45:01.094 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:45:01.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:01.117 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:01.117 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:45:01.125 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-19T14:45:01.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:01 smithi031 ceph-mon[16336]: osdmap e847: 12 total, 12 up, 7 in 2024-08-19T14:45:01.218 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:45:01.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:01.246 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:01.248 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:45:01.314 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:2024-08-19T14:45:01.314891+0000 min lat: 0.00105666 max lat: 2.17994 avg lat: 0.0127685 2024-08-19T14:45:01.315 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-19T14:45:01.315 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 40 16 50063 50047 78.1849 15.75 0.00273323 0.0127685 2024-08-19T14:45:01.349 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:45:01.392 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:01.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:01.393 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:45:01.494 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:45:01.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:01 smithi032 ceph-mon[20004]: osdmap e847: 12 total, 12 up, 7 in 2024-08-19T14:45:01.523 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:01 smithi045 ceph-mon[20143]: osdmap e847: 12 total, 12 up, 7 in 2024-08-19T14:45:01.539 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:01.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:01.540 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:45:01.641 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:45:01.664 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:01.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:01.665 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:45:01.708 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:45:01.765 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:45:01.793 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:01.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:01.795 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:45:01.896 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:45:01.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:01.923 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:01.923 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:45:02.024 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:45:02.055 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:02.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:02.056 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:45:02.138 INFO:teuthology.orchestra.run.smithi031.stderr:reweighted osd.2 to 1 (10000) 2024-08-19T14:45:02.157 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:45:02.189 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:02.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:02.190 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:45:02.291 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:45:02.312 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:02.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:02.313 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:45:02.314 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 41 16 50890 50874 77.5385 51.6875 0.0101284 0.0128715 2024-08-19T14:45:02.414 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:45:02.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:02.442 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:02.443 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:45:02.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:02 smithi032 ceph-mon[20004]: pgmap v2000: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 2.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 45 MiB/s wr, 712 op/s; 1/143973 objects degraded (0.001%); 6330/143973 objects misplaced (4.397%); 24 MiB/s, 382 objects/s recovering 2024-08-19T14:45:02.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:02 smithi032 ceph-mon[20004]: osdmap e848: 12 total, 12 up, 7 in 2024-08-19T14:45:02.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:02 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3347981692' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-08-19T14:45:02.513 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:02 smithi031 ceph-mon[16336]: pgmap v2000: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 2.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 45 MiB/s wr, 712 op/s; 1/143973 objects degraded (0.001%); 6330/143973 objects misplaced (4.397%); 24 MiB/s, 382 objects/s recovering 2024-08-19T14:45:02.514 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:02 smithi031 ceph-mon[16336]: osdmap e848: 12 total, 12 up, 7 in 2024-08-19T14:45:02.514 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:02 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3347981692' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-08-19T14:45:02.543 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:45:02.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:02 smithi045 ceph-mon[20143]: pgmap v2000: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 2.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 45 MiB/s wr, 712 op/s; 1/143973 objects degraded (0.001%); 6330/143973 objects misplaced (4.397%); 24 MiB/s, 382 objects/s recovering 2024-08-19T14:45:02.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:02 smithi045 ceph-mon[20143]: osdmap e848: 12 total, 12 up, 7 in 2024-08-19T14:45:02.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:02 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3347981692' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-08-19T14:45:02.565 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:02.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:02.566 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:45:02.667 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:45:02.693 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:02.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:02.694 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:45:02.795 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:45:02.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:02.818 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:02.818 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:45:02.920 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:45:02.943 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:02.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:02.944 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:45:02.958 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:45:03.045 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:45:03.063 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd reweight 11 1 2024-08-19T14:45:03.082 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:03.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:03.083 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:45:03.154 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is set 2024-08-19T14:45:03.184 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:45:03.204 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:03.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:03.206 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:45:03.307 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:45:03.314 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 42 15 51887 51872 77.1774 62.375 0.0155212 0.0129508 2024-08-19T14:45:03.328 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:03.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:03.329 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:45:03.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:03 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3347981692' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-08-19T14:45:03.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:03 smithi031 ceph-mon[16336]: osdmap e849: 12 total, 12 up, 7 in 2024-08-19T14:45:03.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:03 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3230393839' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:45:03.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:03 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 1/143973 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-08-19T14:45:03.430 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:45:03.452 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:03.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:03.453 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:45:03.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:03 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3347981692' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-08-19T14:45:03.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:03 smithi032 ceph-mon[20004]: osdmap e849: 12 total, 12 up, 7 in 2024-08-19T14:45:03.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:03 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3230393839' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:45:03.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:03 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 1/143973 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-08-19T14:45:03.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:03 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3347981692' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-08-19T14:45:03.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:03 smithi045 ceph-mon[20143]: osdmap e849: 12 total, 12 up, 7 in 2024-08-19T14:45:03.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:03 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3230393839' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:45:03.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:03 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 1/143973 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-08-19T14:45:03.554 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:45:03.580 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:03.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:03.581 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:45:03.682 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:45:03.707 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:03.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:03.708 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:45:03.809 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:45:03.850 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:03.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:03.851 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:45:03.952 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:45:03.977 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:03.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:03.978 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:45:03.996 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:45:04.034 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:45:04.079 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:45:04.100 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:04.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:04.102 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:45:04.203 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:45:04.230 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:04.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:04.231 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:45:04.260 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:04 smithi031 ceph-mon[16336]: pgmap v2003: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 3.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 48 MiB/s wr, 770 op/s; 1/148155 objects degraded (0.001%); 5031/148155 objects misplaced (3.396%); 33 MiB/s, 535 objects/s recovering 2024-08-19T14:45:04.261 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:04 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:45:04.261 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:04 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:45:04.261 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:04 smithi031 ceph-mon[16336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:45:04.261 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:04 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3230393839' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:45:04.261 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:04 smithi031 ceph-mon[16336]: osdmap e850: 12 total, 12 up, 7 in 2024-08-19T14:45:04.315 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 43 16 52933 52917 76.9013 65.3125 0.012708 0.0129976 2024-08-19T14:45:04.332 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:45:04.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:04.354 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:04.354 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:45:04.455 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:45:04.482 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:04.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:04.484 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:45:04.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:04 smithi032 ceph-mon[20004]: pgmap v2003: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 3.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 48 MiB/s wr, 770 op/s; 1/148155 objects degraded (0.001%); 5031/148155 objects misplaced (3.396%); 33 MiB/s, 535 objects/s recovering 2024-08-19T14:45:04.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:04 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:45:04.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:04 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:45:04.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:04 smithi032 ceph-mon[20004]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:45:04.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:04 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3230393839' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:45:04.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:04 smithi032 ceph-mon[20004]: osdmap e850: 12 total, 12 up, 7 in 2024-08-19T14:45:04.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:04 smithi045 ceph-mon[20143]: pgmap v2003: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 3.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 48 MiB/s wr, 770 op/s; 1/148155 objects degraded (0.001%); 5031/148155 objects misplaced (3.396%); 33 MiB/s, 535 objects/s recovering 2024-08-19T14:45:04.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:04 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:45:04.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:04 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:45:04.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:04 smithi045 ceph-mon[20143]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:45:04.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:04 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3230393839' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:45:04.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:04 smithi045 ceph-mon[20143]: osdmap e850: 12 total, 12 up, 7 in 2024-08-19T14:45:04.584 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:45:04.606 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:04.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:04.607 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:45:04.709 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:45:04.737 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:04.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:04.738 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:45:04.839 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:45:04.859 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-08-19T14:45:04.860 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:04.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:04.861 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:45:04.921 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:45:04.921 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:45:04.956 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:45:04.956 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:45:04.962 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:45:04.984 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:45:04.984 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:45:04.992 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:04.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:04.993 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:45:05.018 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:45:05.019 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:45:05.041 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:45:05.042 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:45:05.065 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:45:05.066 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:45:05.091 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:45:05.091 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:45:05.093 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:45:05.114 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:45:05.114 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:45:05.118 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:05.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:05.119 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:45:05.143 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:45:05.143 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:45:05.154 INFO:teuthology.orchestra.run.smithi031.stderr:reweighted osd.11 to 1 (10000) 2024-08-19T14:45:05.179 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:45:05.179 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:45:05.204 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:45:05.204 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:45:05.220 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:45:05.228 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:45:05.228 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:45:05.245 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:05.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:05.248 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:45:05.312 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:45:05.316 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 44 16 53850 53834 76.4543 57.3125 0.00985378 0.013067 2024-08-19T14:45:05.349 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:45:05.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:05 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2684216511' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:45:05.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:05 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1371544005' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-08-19T14:45:05.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:05.394 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:05.394 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:45:05.451 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:05 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2684216511' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:45:05.451 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:05 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1371544005' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-08-19T14:45:05.495 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:45:05.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:05 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2684216511' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:45:05.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:05 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1371544005' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-08-19T14:45:05.519 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:05.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:05.520 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:45:05.621 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:45:05.643 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:05.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:05.644 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:45:05.745 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:45:05.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:05.775 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:05.775 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:45:05.876 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:45:05.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:05.899 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:05.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:45:05.963 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-19T14:45:06.000 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:45:06.027 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:06.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:06.028 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:45:06.128 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:45:06.151 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:06.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:06.152 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:45:06.254 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:45:06.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:06.279 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:06.279 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:45:06.316 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 45 16 54189 54173 75.2262 21.1875 0.00194494 0.013269 2024-08-19T14:45:06.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:06 smithi031 ceph-mon[16336]: pgmap v2005: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 41 active+clean; 3.1 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 43 MiB/s wr, 695 op/s; 1/152820 objects degraded (0.001%); 3703/152820 objects misplaced (2.423%); 39 MiB/s, 619 objects/s recovering 2024-08-19T14:45:06.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:06 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1371544005' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-08-19T14:45:06.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:06 smithi031 ceph-mon[16336]: osdmap e851: 12 total, 12 up, 7 in 2024-08-19T14:45:06.380 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:45:06.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:06.405 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:06.405 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:45:06.507 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:45:06.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:06 smithi032 ceph-mon[20004]: pgmap v2005: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 41 active+clean; 3.1 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 43 MiB/s wr, 695 op/s; 1/152820 objects degraded (0.001%); 3703/152820 objects misplaced (2.423%); 39 MiB/s, 619 objects/s recovering 2024-08-19T14:45:06.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:06 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1371544005' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-08-19T14:45:06.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:06 smithi032 ceph-mon[20004]: osdmap e851: 12 total, 12 up, 7 in 2024-08-19T14:45:06.533 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:06.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:06.534 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:45:06.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:06 smithi045 ceph-mon[20143]: pgmap v2005: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 41 active+clean; 3.1 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 43 MiB/s wr, 695 op/s; 1/152820 objects degraded (0.001%); 3703/152820 objects misplaced (2.423%); 39 MiB/s, 619 objects/s recovering 2024-08-19T14:45:06.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:06 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1371544005' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-08-19T14:45:06.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:06 smithi045 ceph-mon[20143]: osdmap e851: 12 total, 12 up, 7 in 2024-08-19T14:45:06.635 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:45:06.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:06.659 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:06.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:45:06.674 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd reweight 9 1 2024-08-19T14:45:06.757 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:45:06.760 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:45:06.786 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:06.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:06.787 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:45:06.888 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:45:06.911 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:06.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:06.912 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:45:07.014 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:45:07.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:07.040 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:07.040 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:45:07.142 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:45:07.165 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:07.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:07.166 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:45:07.267 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:45:07.291 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:07.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:07.292 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:45:07.302 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:45:07.316 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 46 16 54560 54544 74.0949 23.1875 0.00301924 0.0134737 2024-08-19T14:45:07.393 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:45:07.415 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:07.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:07.416 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:45:07.425 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:07 smithi031 ceph-mon[16336]: osdmap e852: 12 total, 12 up, 7 in 2024-08-19T14:45:07.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:07 smithi032 ceph-mon[20004]: osdmap e852: 12 total, 12 up, 7 in 2024-08-19T14:45:07.517 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:07 smithi045 ceph-mon[20143]: osdmap e852: 12 total, 12 up, 7 in 2024-08-19T14:45:07.517 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:45:07.543 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:07.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:07.544 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:45:07.645 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:45:07.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:07.667 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:07.667 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:45:07.768 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:45:07.795 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:07.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:07.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:45:07.897 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:45:07.920 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:07.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:07.921 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:45:08.023 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:45:08.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:08.050 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:08.051 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:45:08.083 INFO:teuthology.orchestra.run.smithi031.stderr:reweighted osd.9 to 1 (10000) 2024-08-19T14:45:08.083 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is set 2024-08-19T14:45:08.133 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:45:08.151 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:45:08.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:08.173 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:08.173 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:45:08.273 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:45:08.294 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:08.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:08.295 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:45:08.316 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 47 16 54710 54694 72.718 9.375 0.00271917 0.0136911 2024-08-19T14:45:08.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:08 smithi031 ceph-mon[16336]: pgmap v2008: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 41 active+clean; 3.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 53 MiB/s wr, 846 op/s; 1/158892 objects degraded (0.001%); 2279/158892 objects misplaced (1.434%); 41 MiB/s, 650 objects/s recovering 2024-08-19T14:45:08.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:08 smithi031 ceph-mon[16336]: osdmap e853: 12 total, 12 up, 7 in 2024-08-19T14:45:08.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:08 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3522383090' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:45:08.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:08 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/384968244' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-08-19T14:45:08.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:08 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 1/158892 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-08-19T14:45:08.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:08 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3522383090' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:45:08.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:08 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/384968244' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-08-19T14:45:08.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:08 smithi031 ceph-mon[16336]: osdmap e854: 12 total, 12 up, 7 in 2024-08-19T14:45:08.396 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:45:08.417 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:08.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:08.418 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:45:08.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:08 smithi032 ceph-mon[20004]: pgmap v2008: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 41 active+clean; 3.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 53 MiB/s wr, 846 op/s; 1/158892 objects degraded (0.001%); 2279/158892 objects misplaced (1.434%); 41 MiB/s, 650 objects/s recovering 2024-08-19T14:45:08.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:08 smithi032 ceph-mon[20004]: osdmap e853: 12 total, 12 up, 7 in 2024-08-19T14:45:08.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:08 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3522383090' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:45:08.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:08 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/384968244' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-08-19T14:45:08.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:08 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 1/158892 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-08-19T14:45:08.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:08 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3522383090' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:45:08.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:08 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/384968244' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-08-19T14:45:08.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:08 smithi032 ceph-mon[20004]: osdmap e854: 12 total, 12 up, 7 in 2024-08-19T14:45:08.519 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:45:08.544 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:08.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:08.545 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:45:08.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:08 smithi045 ceph-mon[20143]: pgmap v2008: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 41 active+clean; 3.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 53 MiB/s wr, 846 op/s; 1/158892 objects degraded (0.001%); 2279/158892 objects misplaced (1.434%); 41 MiB/s, 650 objects/s recovering 2024-08-19T14:45:08.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:08 smithi045 ceph-mon[20143]: osdmap e853: 12 total, 12 up, 7 in 2024-08-19T14:45:08.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:08 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3522383090' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:45:08.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:08 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/384968244' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-08-19T14:45:08.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:08 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 1/158892 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-08-19T14:45:08.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:08 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3522383090' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:45:08.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:08 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/384968244' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-08-19T14:45:08.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:08 smithi045 ceph-mon[20143]: osdmap e854: 12 total, 12 up, 7 in 2024-08-19T14:45:08.646 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:45:08.668 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:08.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:08.669 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:45:08.770 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:45:08.795 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:08.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:08.796 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:45:08.897 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:45:08.923 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:08.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:08.924 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:45:09.025 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:45:09.051 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:09.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:09.052 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:45:09.130 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd reweight 10 1 2024-08-19T14:45:09.153 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:45:09.178 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:09.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:09.179 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:45:09.281 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:45:09.316 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 48 16 54878 54862 71.4219 10.5 0.00402434 0.0139948 2024-08-19T14:45:09.318 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:09.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:09.320 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:45:09.421 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:45:09.447 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:09.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:09.448 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:45:09.549 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:45:09.571 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:09.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:09.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:45:09.584 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:45:09.673 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:45:09.699 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:09.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:09.700 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:45:09.801 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:45:09.824 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:09.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:09.825 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:45:09.926 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:45:09.953 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:09.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:09.954 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:45:10.055 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:45:10.095 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:10.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:10.096 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:45:10.104 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:45:10.104 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:45:10.184 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:45:10.185 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:45:10.197 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:45:10.210 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:45:10.211 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:45:10.212 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:45:10.224 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:10.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:10.225 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:45:10.231 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:45:10.232 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:45:10.248 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:45:10.248 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:45:10.269 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:45:10.270 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:45:10.291 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:45:10.292 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:45:10.313 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:45:10.313 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:45:10.313 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:10 smithi031 ceph-mon[16336]: pgmap v2011: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 43 active+clean; 3.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 39 MiB/s wr, 616 op/s; 1/160215 objects degraded (0.001%); 1723/160215 objects misplaced (1.075%); 31 MiB/s, 494 objects/s recovering 2024-08-19T14:45:10.313 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:10 smithi031 ceph-mon[16336]: osdmap e855: 12 total, 12 up, 7 in 2024-08-19T14:45:10.313 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:10 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:45:10.313 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:10 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:45:10.316 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 49 16 56254 56238 71.7192 86 0.00878731 0.0139372 2024-08-19T14:45:10.325 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:45:10.333 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:45:10.333 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:45:10.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:10.363 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:10.363 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:45:10.368 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:45:10.368 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:45:10.394 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:45:10.394 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:45:10.423 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:45:10.423 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:45:10.453 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:45:10.453 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:45:10.463 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:45:10.503 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:10.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:10.504 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:45:10.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:10 smithi032 ceph-mon[20004]: pgmap v2011: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 43 active+clean; 3.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 39 MiB/s wr, 616 op/s; 1/160215 objects degraded (0.001%); 1723/160215 objects misplaced (1.075%); 31 MiB/s, 494 objects/s recovering 2024-08-19T14:45:10.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:10 smithi032 ceph-mon[20004]: osdmap e855: 12 total, 12 up, 7 in 2024-08-19T14:45:10.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:10 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:45:10.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:10 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:45:10.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:10 smithi045 ceph-mon[20143]: pgmap v2011: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 43 active+clean; 3.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 39 MiB/s wr, 616 op/s; 1/160215 objects degraded (0.001%); 1723/160215 objects misplaced (1.075%); 31 MiB/s, 494 objects/s recovering 2024-08-19T14:45:10.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:10 smithi045 ceph-mon[20143]: osdmap e855: 12 total, 12 up, 7 in 2024-08-19T14:45:10.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:10 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:45:10.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:10 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:45:10.605 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:45:10.644 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:10.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:10.645 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:45:10.746 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:45:10.774 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:10.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:10.775 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:45:10.876 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:45:10.913 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:10.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:10.914 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:45:11.015 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:45:11.036 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:11.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:11.037 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:45:11.138 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:45:11.166 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:11.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:11.167 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:45:11.268 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:45:11.289 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:11.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:11.290 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:45:11.316 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 50 16 57499 57483 71.8409 77.8125 0.00294042 0.0138757 2024-08-19T14:45:11.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:11 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3802980512' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:45:11.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:11 smithi031 ceph-mon[16336]: osdmap e856: 12 total, 12 up, 7 in 2024-08-19T14:45:11.390 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:45:11.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:11.428 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:11.428 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:45:11.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:11 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3802980512' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:45:11.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:11 smithi032 ceph-mon[20004]: osdmap e856: 12 total, 12 up, 7 in 2024-08-19T14:45:11.529 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:45:11.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:11 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3802980512' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:45:11.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:11 smithi045 ceph-mon[20143]: osdmap e856: 12 total, 12 up, 7 in 2024-08-19T14:45:11.550 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:11.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:11.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:45:11.583 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-19T14:45:11.652 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:45:11.673 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:11.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:11.674 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:45:11.776 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:45:11.813 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:11.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:11.815 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:45:11.916 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:45:11.923 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:45:11.954 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:11.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:11.955 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:45:12.056 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:45:12.080 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:12.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:12.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:45:12.177 INFO:teuthology.orchestra.run.smithi031.stderr:reweighted osd.10 to 1 (10000) 2024-08-19T14:45:12.182 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:45:12.218 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:12.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:12.219 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:45:12.316 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 51 16 58802 58786 72.029 81.4375 0.00573389 0.0138782 2024-08-19T14:45:12.320 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:45:12.341 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:12.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:12.342 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:45:12.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:12 smithi031 ceph-mon[16336]: pgmap v2014: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 43 active+clean; 3.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 23 MiB/s wr, 365 op/s; 1/163281 objects degraded (0.001%); 1723/163281 objects misplaced (1.055%); 8.7 MiB/s, 138 objects/s recovering 2024-08-19T14:45:12.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:12 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:45:12.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:12 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:45:12.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:12 smithi031 ceph-mon[16336]: osdmap e857: 12 total, 12 up, 7 in 2024-08-19T14:45:12.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:12 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1523002306' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-08-19T14:45:12.443 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:45:12.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:12.487 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:12.487 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:45:12.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:12 smithi032 ceph-mon[20004]: pgmap v2014: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 43 active+clean; 3.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 23 MiB/s wr, 365 op/s; 1/163281 objects degraded (0.001%); 1723/163281 objects misplaced (1.055%); 8.7 MiB/s, 138 objects/s recovering 2024-08-19T14:45:12.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:12 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:45:12.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:12 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:45:12.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:12 smithi032 ceph-mon[20004]: osdmap e857: 12 total, 12 up, 7 in 2024-08-19T14:45:12.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:12 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1523002306' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-08-19T14:45:12.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:12 smithi045 ceph-mon[20143]: pgmap v2014: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 43 active+clean; 3.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 23 MiB/s wr, 365 op/s; 1/163281 objects degraded (0.001%); 1723/163281 objects misplaced (1.055%); 8.7 MiB/s, 138 objects/s recovering 2024-08-19T14:45:12.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:12 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:45:12.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:12 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:45:12.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:12 smithi045 ceph-mon[20143]: osdmap e857: 12 total, 12 up, 7 in 2024-08-19T14:45:12.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:12 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1523002306' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-08-19T14:45:12.589 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:45:12.615 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:12.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:12.616 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:45:12.717 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:45:12.739 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:12.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:12.740 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:45:12.841 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:45:12.862 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:12.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:12.863 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:45:12.964 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:45:13.003 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:13.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:13.004 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:45:13.105 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:45:13.148 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:13.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:13.149 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:45:13.191 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-08-19T14:45:13.193 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-08-19T14:45:13.193 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-19T14:45:13.195 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is unset 2024-08-19T14:45:13.250 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:45:13.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:13.277 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:13.277 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:45:13.317 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 52 16 60167 60151 72.2838 85.3125 0.00711337 0.0138288 2024-08-19T14:45:13.378 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:45:13.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:13.400 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:13.400 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:45:13.502 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:45:13.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:13 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:45:13.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:13 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:45:13.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:13 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1523002306' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-08-19T14:45:13.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:13 smithi032 ceph-mon[20004]: osdmap e858: 12 total, 12 up, 7 in 2024-08-19T14:45:13.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:13 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2335234886' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:45:13.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:13 smithi032 ceph-mon[20004]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:45:13.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:13 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 1/163281 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-08-19T14:45:13.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:13 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:45:13.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:13 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:45:13.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:13 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1523002306' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-08-19T14:45:13.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:13 smithi045 ceph-mon[20143]: osdmap e858: 12 total, 12 up, 7 in 2024-08-19T14:45:13.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:13 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2335234886' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:45:13.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:13 smithi045 ceph-mon[20143]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:45:13.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:13 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 1/163281 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-08-19T14:45:13.550 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:13.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:13.553 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:45:13.573 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:45:13.592 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:13 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:45:13.592 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:13 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:45:13.592 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:13 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1523002306' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-08-19T14:45:13.592 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:13 smithi031 ceph-mon[16336]: osdmap e858: 12 total, 12 up, 7 in 2024-08-19T14:45:13.592 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:13 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2335234886' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:45:13.592 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:13 smithi031 ceph-mon[16336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:45:13.592 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:13 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 1/163281 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-08-19T14:45:13.653 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:45:13.680 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:13.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:13.681 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:45:13.686 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:45:13.782 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:45:13.803 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:13.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:13.804 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:45:13.905 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:45:13.948 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:13.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:13.950 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:45:14.051 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:45:14.056 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:45:14.083 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:14.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:14.084 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:45:14.185 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:45:14.210 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:14.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:14.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:45:14.291 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:45:14.291 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [], 2024-08-19T14:45:14.291 INFO:teuthology.orchestra.run.smithi045.stdout: "complaint_time": 30, 2024-08-19T14:45:14.292 INFO:teuthology.orchestra.run.smithi045.stdout: "num_blocked_ops": 0 2024-08-19T14:45:14.292 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:45:14.312 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:45:14.317 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 53 16 61446 61430 72.4281 79.9375 0.0110544 0.0138015 2024-08-19T14:45:14.329 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:14 smithi031 ceph-mon[16336]: pgmap v2017: 47 pgs: 47 active+clean; 3.4 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 34 MiB/s wr, 546 op/s; 27 MiB/s, 430 objects/s recovering 2024-08-19T14:45:14.329 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:14 smithi031 ceph-mon[16336]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/163281 objects degraded (0.001%), 1 pg degraded) 2024-08-19T14:45:14.329 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:14 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2335234886' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:45:14.329 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:14 smithi031 ceph-mon[16336]: osdmap e859: 12 total, 12 up, 7 in 2024-08-19T14:45:14.334 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:14.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:14.335 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:45:14.436 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:45:14.480 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:14.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:14.481 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:45:14.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:14 smithi032 ceph-mon[20004]: pgmap v2017: 47 pgs: 47 active+clean; 3.4 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 34 MiB/s wr, 546 op/s; 27 MiB/s, 430 objects/s recovering 2024-08-19T14:45:14.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:14 smithi032 ceph-mon[20004]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/163281 objects degraded (0.001%), 1 pg degraded) 2024-08-19T14:45:14.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:14 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2335234886' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:45:14.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:14 smithi032 ceph-mon[20004]: osdmap e859: 12 total, 12 up, 7 in 2024-08-19T14:45:14.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:14 smithi045 ceph-mon[20143]: pgmap v2017: 47 pgs: 47 active+clean; 3.4 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 34 MiB/s wr, 546 op/s; 27 MiB/s, 430 objects/s recovering 2024-08-19T14:45:14.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:14 smithi045 ceph-mon[20143]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/163281 objects degraded (0.001%), 1 pg degraded) 2024-08-19T14:45:14.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:14 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2335234886' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:45:14.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:14 smithi045 ceph-mon[20143]: osdmap e859: 12 total, 12 up, 7 in 2024-08-19T14:45:14.569 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:45:14.582 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:45:14.612 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:14.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:14.613 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:45:14.714 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:45:14.743 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:14.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:14.744 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:45:14.827 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:45:14.828 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-08-19T14:45:14.845 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:45:14.866 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:14.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:14.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:45:14.968 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:45:14.994 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:14.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:14.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:45:15.050 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-08-19T14:45:15.096 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:45:15.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:15.159 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:15.159 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:45:15.260 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:45:15.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:15.282 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:15.283 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:45:15.317 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 54 16 62577 62561 72.3957 70.6875 0.0202575 0.0138065 2024-08-19T14:45:15.376 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:45:15.377 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:45:15.383 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:45:15.396 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:45:15.396 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:45:15.404 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:15.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:15.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:45:15.421 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:45:15.421 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:45:15.442 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:45:15.442 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:45:15.462 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:45:15.463 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:45:15.484 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:45:15.485 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:45:15.505 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:45:15.506 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:45:15.506 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:45:15.517 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:45:15.527 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:15.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:15.528 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:45:15.529 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:45:15.529 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:45:15.556 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:45:15.556 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:45:15.585 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:45:15.586 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:45:15.612 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:45:15.612 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:45:15.629 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:45:15.636 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:45:15.636 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:45:15.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:15.665 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:15.665 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:45:15.766 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:45:15.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:15.806 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:15.806 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:45:15.907 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:45:15.929 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:15.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:15.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:45:15.979 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":2019,"stamp":"2024-08-19T14:45:14.367848+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3825533472,"num_objects":58366,"num_object_clones":0,"num_object_copies":175098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":58364,"num_write_kb":3735296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17552,"num_bytes_recovered":1151083584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":62673,"ondisk_log_size":62673,"up":141,"acting":141,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":141,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":17822424,"kb_used_data":11236860,"kb_used_omap":16,"kb_used_meta":6585455,"kb_avail":638393640,"statfs":{"total":671965249536,"available":653715087360,"internally_reserved":0,"allocated":11506544640,"data_stored":11488973910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16794,"internal_metadata":6743506534},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,1,7,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":258146304,"num_objects":3939,"num_object_clones":0,"num_object_copies":11817,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1,"num_objects_misplaced":-1723,"num_objects_unfound":0,"num_objects_dirty":3939,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3939,"num_write_kb":252096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1724,"num_bytes_recovered":112984064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.195952"},"pg_stats":[{"pgid":"9.8","version":"858'3723","reported_seq":7874,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-08-19T14:45:12.635697+0000","last_change":"2024-08-19T14:45:07.176081+0000","last_active":"2024-08-19T14:45:12.635697+0000","last_peered":"2024-08-19T14:45:12.635697+0000","last_clean":"2024-08-19T14:45:12.635697+0000","last_became_active":"2024-08-19T14:45:07.171026+0000","last_became_peered":"2024-08-19T14:45:07.171026+0000","last_unstale":"2024-08-19T14:45:12.635697+0000","last_undegraded":"2024-08-19T14:45:12.635697+0000","last_fullsized":"2024-08-19T14:45:12.635697+0000","mapping_epoch":852,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":853,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":3723,"log_dups_size":0,"ondisk_log_size":3723,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:24.356519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":243990528,"num_objects":3723,"num_object_clones":0,"num_object_copies":11169,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3723,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3723,"num_write_kb":238272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2033,"num_bytes_recovered":133234688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,1],"acting":[11,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"70'139","reported_seq":2053,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-08-19T14:45:12.189574+0000","last_change":"2024-08-19T14:44:26.109478+0000","last_active":"2024-08-19T14:45:12.189574+0000","last_peered":"2024-08-19T14:45:12.189574+0000","last_clean":"2024-08-19T14:45:12.189574+0000","last_became_active":"2024-08-19T14:43:16.089672+0000","last_became_peered":"2024-08-19T14:43:16.089672+0000","last_unstale":"2024-08-19T14:45:12.189574+0000","last_undegraded":"2024-08-19T14:45:12.189574+0000","last_fullsized":"2024-08-19T14:45:12.189574+0000","mapping_epoch":800,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":801,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:26.109206+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:41:45.177828+0000","last_clean_scrub_stamp":"2024-08-19T14:44:26.109206+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:50.139250+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,10],"acting":[11,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.b","version":"856'3564","reported_seq":7607,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-08-19T14:45:10.758866+0000","last_change":"2024-08-19T14:45:09.096009+0000","last_active":"2024-08-19T14:45:10.758866+0000","last_peered":"2024-08-19T14:45:10.758866+0000","last_clean":"2024-08-19T14:45:10.758866+0000","last_became_active":"2024-08-19T14:45:09.091550+0000","last_became_peered":"2024-08-19T14:45:09.091550+0000","last_unstale":"2024-08-19T14:45:10.758866+0000","last_undegraded":"2024-08-19T14:45:10.758866+0000","last_fullsized":"2024-08-19T14:45:10.758866+0000","mapping_epoch":854,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":3564,"log_dups_size":0,"ondisk_log_size":3564,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:42.050707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233570304,"num_objects":3564,"num_object_clones":0,"num_object_copies":10692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3564,"num_write_kb":228096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1981,"num_bytes_recovered":129826816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,1],"acting":[8,10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"70'139","reported_seq":1900,"reported_epoch":859,"state":"active+clean","last_fresh":"2024-08-19T14:45:13.201212+0000","last_change":"2024-08-19T14:41:45.936228+0000","last_active":"2024-08-19T14:45:13.201212+0000","last_peered":"2024-08-19T14:45:13.201212+0000","last_clean":"2024-08-19T14:45:13.201212+0000","last_became_active":"2024-08-19T14:18:13.804721+0000","last_became_peered":"2024-08-19T14:18:13.804721+0000","last_unstale":"2024-08-19T14:45:13.201212+0000","last_undegraded":"2024-08-19T14:45:13.201212+0000","last_fullsized":"2024-08-19T14:45:13.201212+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:41:45.936114+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:41:45.936114+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:17.675420+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.a","version":"859'3860","reported_seq":7831,"reported_epoch":859,"state":"active+clean","last_fresh":"2024-08-19T14:45:13.684418+0000","last_change":"2024-08-19T14:44:49.108491+0000","last_active":"2024-08-19T14:45:13.684418+0000","last_peered":"2024-08-19T14:45:13.684418+0000","last_clean":"2024-08-19T14:45:13.684418+0000","last_became_active":"2024-08-19T14:44:49.104676+0000","last_became_peered":"2024-08-19T14:44:49.104676+0000","last_unstale":"2024-08-19T14:45:13.684418+0000","last_undegraded":"2024-08-19T14:45:13.684418+0000","last_fullsized":"2024-08-19T14:45:13.684418+0000","mapping_epoch":841,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":842,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":3860,"log_dups_size":0,"ondisk_log_size":3860,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:43.293459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252968960,"num_objects":3860,"num_object_clones":0,"num_object_copies":11580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3860,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3860,"num_write_kb":247040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1942,"num_bytes_recovered":127270912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,2],"acting":[1,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"70'139","reported_seq":539,"reported_epoch":855,"state":"active+clean","last_fresh":"2024-08-19T14:45:09.108959+0000","last_change":"2024-08-19T14:44:25.759421+0000","last_active":"2024-08-19T14:45:09.108959+0000","last_peered":"2024-08-19T14:45:09.108959+0000","last_clean":"2024-08-19T14:45:09.108959+0000","last_became_active":"2024-08-19T14:43:11.470639+0000","last_became_peered":"2024-08-19T14:43:11.470639+0000","last_unstale":"2024-08-19T14:45:09.108959+0000","last_undegraded":"2024-08-19T14:45:09.108959+0000","last_fullsized":"2024-08-19T14:45:09.108959+0000","mapping_epoch":795,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":796,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:25.759337+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:19:59.302931+0000","last_clean_scrub_stamp":"2024-08-19T14:44:25.759337+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:30.192194+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.9","version":"856'3549","reported_seq":7572,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-08-19T14:45:10.393360+0000","last_change":"2024-08-19T14:45:09.098084+0000","last_active":"2024-08-19T14:45:10.393360+0000","last_peered":"2024-08-19T14:45:10.393360+0000","last_clean":"2024-08-19T14:45:10.393360+0000","last_became_active":"2024-08-19T14:45:09.091434+0000","last_became_peered":"2024-08-19T14:45:09.091434+0000","last_unstale":"2024-08-19T14:45:10.393360+0000","last_undegraded":"2024-08-19T14:45:10.393360+0000","last_fullsized":"2024-08-19T14:45:10.393360+0000","mapping_epoch":854,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":3549,"log_dups_size":0,"ondisk_log_size":3549,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:14.899998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":232587264,"num_objects":3549,"num_object_clones":0,"num_object_copies":10647,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3549,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3549,"num_write_kb":227136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":129236992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,8],"acting":[10,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1","version":"70'139","reported_seq":542,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-08-19T14:45:10.180462+0000","last_change":"2024-08-19T14:44:32.095248+0000","last_active":"2024-08-19T14:45:10.180462+0000","last_peered":"2024-08-19T14:45:10.180462+0000","last_clean":"2024-08-19T14:45:10.180462+0000","last_became_active":"2024-08-19T14:43:16.098337+0000","last_became_peered":"2024-08-19T14:43:16.098337+0000","last_unstale":"2024-08-19T14:45:10.180462+0000","last_undegraded":"2024-08-19T14:45:10.180462+0000","last_fullsized":"2024-08-19T14:45:10.180462+0000","mapping_epoch":800,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":801,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:32.094375+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:44:32.094375+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:40.928652+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,2],"acting":[10,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.f","version":"859'3842","reported_seq":7807,"reported_epoch":859,"state":"active+clean","last_fresh":"2024-08-19T14:45:13.672790+0000","last_change":"2024-08-19T14:45:01.147769+0000","last_active":"2024-08-19T14:45:13.672790+0000","last_peered":"2024-08-19T14:45:13.672790+0000","last_clean":"2024-08-19T14:45:13.672790+0000","last_became_active":"2024-08-19T14:45:01.143857+0000","last_became_peered":"2024-08-19T14:45:01.143857+0000","last_unstale":"2024-08-19T14:45:13.672790+0000","last_undegraded":"2024-08-19T14:45:13.672790+0000","last_fullsized":"2024-08-19T14:45:13.672790+0000","mapping_epoch":847,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":848,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":3842,"log_dups_size":0,"ondisk_log_size":3842,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:40.066370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251789312,"num_objects":3842,"num_object_clones":0,"num_object_copies":11526,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3842,"num_write_kb":245888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1939,"num_bytes_recovered":127074304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,11],"acting":[1,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"70'139","reported_seq":1125,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-08-19T14:45:10.179926+0000","last_change":"2024-08-19T14:44:38.181655+0000","last_active":"2024-08-19T14:45:10.179926+0000","last_peered":"2024-08-19T14:45:10.179926+0000","last_clean":"2024-08-19T14:45:10.179926+0000","last_became_active":"2024-08-19T14:44:38.181307+0000","last_became_peered":"2024-08-19T14:44:38.181307+0000","last_unstale":"2024-08-19T14:45:10.179926+0000","last_undegraded":"2024-08-19T14:45:10.179926+0000","last_fullsized":"2024-08-19T14:45:10.179926+0000","mapping_epoch":837,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:26.113799+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:21:46.057624+0000","last_clean_scrub_stamp":"2024-08-19T14:44:26.113799+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:52.928219+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,1],"acting":[10,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.e","version":"856'3543","reported_seq":3613,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-08-19T14:45:10.402685+0000","last_change":"2024-08-19T14:44:15.415042+0000","last_active":"2024-08-19T14:45:10.402685+0000","last_peered":"2024-08-19T14:45:10.402685+0000","last_clean":"2024-08-19T14:45:10.402685+0000","last_became_active":"2024-08-19T14:44:15.414595+0000","last_became_peered":"2024-08-19T14:44:15.414595+0000","last_unstale":"2024-08-19T14:45:10.402685+0000","last_undegraded":"2024-08-19T14:45:10.402685+0000","last_fullsized":"2024-08-19T14:45:10.402685+0000","mapping_epoch":826,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":827,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":3543,"log_dups_size":0,"ondisk_log_size":3543,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:29.023228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":232194048,"num_objects":3543,"num_object_clones":0,"num_object_copies":10629,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3543,"num_write_kb":226752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,9],"acting":[10,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.6","version":"70'139","reported_seq":408,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-08-19T14:45:10.182204+0000","last_change":"2024-08-19T14:44:22.089843+0000","last_active":"2024-08-19T14:45:10.182204+0000","last_peered":"2024-08-19T14:45:10.182204+0000","last_clean":"2024-08-19T14:45:10.182204+0000","last_became_active":"2024-08-19T14:43:16.091256+0000","last_became_peered":"2024-08-19T14:43:16.091256+0000","last_unstale":"2024-08-19T14:45:10.182204+0000","last_undegraded":"2024-08-19T14:45:10.182204+0000","last_fullsized":"2024-08-19T14:45:10.182204+0000","mapping_epoch":800,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":801,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:22.089766+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:27.848870+0000","last_clean_scrub_stamp":"2024-08-19T14:44:22.089766+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:42.204189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.d","version":"859'3706","reported_seq":7530,"reported_epoch":859,"state":"active+clean","last_fresh":"2024-08-19T14:45:13.676189+0000","last_change":"2024-08-19T14:45:00.248322+0000","last_active":"2024-08-19T14:45:13.676189+0000","last_peered":"2024-08-19T14:45:13.676189+0000","last_clean":"2024-08-19T14:45:13.676189+0000","last_became_active":"2024-08-19T14:45:00.244759+0000","last_became_peered":"2024-08-19T14:45:00.244759+0000","last_unstale":"2024-08-19T14:45:13.676189+0000","last_undegraded":"2024-08-19T14:45:13.676189+0000","last_fullsized":"2024-08-19T14:45:13.676189+0000","mapping_epoch":846,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":847,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":3706,"log_dups_size":0,"ondisk_log_size":3706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:26.231935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":242876416,"num_objects":3706,"num_object_clones":0,"num_object_copies":11118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3706,"num_write_kb":237184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1868,"num_bytes_recovered":122421248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"70'139","reported_seq":498,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-08-19T14:45:10.178320+0000","last_change":"2024-08-19T14:44:29.076011+0000","last_active":"2024-08-19T14:45:10.178320+0000","last_peered":"2024-08-19T14:45:10.178320+0000","last_clean":"2024-08-19T14:45:10.178320+0000","last_became_active":"2024-08-19T14:43:16.099747+0000","last_became_peered":"2024-08-19T14:43:16.099747+0000","last_unstale":"2024-08-19T14:45:10.178320+0000","last_undegraded":"2024-08-19T14:45:10.178320+0000","last_fullsized":"2024-08-19T14:45:10.178320+0000","mapping_epoch":800,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":801,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:29.075645+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:41:48.878043+0000","last_clean_scrub_stamp":"2024-08-19T14:44:29.075645+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:32.760424+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,11],"acting":[10,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.c","version":"855'3496","reported_seq":3564,"reported_epoch":855,"state":"active+clean","last_fresh":"2024-08-19T14:45:09.865501+0000","last_change":"2024-08-19T14:44:15.416651+0000","last_active":"2024-08-19T14:45:09.865501+0000","last_peered":"2024-08-19T14:45:09.865501+0000","last_clean":"2024-08-19T14:45:09.865501+0000","last_became_active":"2024-08-19T14:44:15.416278+0000","last_became_peered":"2024-08-19T14:44:15.416278+0000","last_unstale":"2024-08-19T14:45:09.865501+0000","last_undegraded":"2024-08-19T14:45:09.865501+0000","last_fullsized":"2024-08-19T14:45:09.865501+0000","mapping_epoch":826,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":827,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":3496,"log_dups_size":0,"ondisk_log_size":3496,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:21.705266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":229113856,"num_objects":3496,"num_object_clones":0,"num_object_copies":10488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3496,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3496,"num_write_kb":223744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,10],"acting":[4,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"70'139","reported_seq":972,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-08-19T14:45:12.187085+0000","last_change":"2024-08-19T14:44:33.078639+0000","last_active":"2024-08-19T14:45:12.187085+0000","last_peered":"2024-08-19T14:45:12.187085+0000","last_clean":"2024-08-19T14:45:12.187085+0000","last_became_active":"2024-08-19T14:43:08.462160+0000","last_became_peered":"2024-08-19T14:43:08.462160+0000","last_unstale":"2024-08-19T14:45:12.187085+0000","last_undegraded":"2024-08-19T14:45:12.187085+0000","last_fullsized":"2024-08-19T14:45:12.187085+0000","mapping_epoch":793,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":794,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:33.078526+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:41:55.868614+0000","last_clean_scrub_stamp":"2024-08-19T14:44:33.078526+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:46:03.059281+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,4],"acting":[11,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.2","version":"859'3785","reported_seq":3861,"reported_epoch":859,"state":"active+clean","last_fresh":"2024-08-19T14:45:13.648011+0000","last_change":"2024-08-19T14:44:15.415441+0000","last_active":"2024-08-19T14:45:13.648011+0000","last_peered":"2024-08-19T14:45:13.648011+0000","last_clean":"2024-08-19T14:45:13.648011+0000","last_became_active":"2024-08-19T14:44:15.415137+0000","last_became_peered":"2024-08-19T14:44:15.415137+0000","last_unstale":"2024-08-19T14:45:13.648011+0000","last_undegraded":"2024-08-19T14:45:13.648011+0000","last_fullsized":"2024-08-19T14:45:13.648011+0000","mapping_epoch":826,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":827,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":3785,"log_dups_size":0,"ondisk_log_size":3785,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:28.806060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":248053760,"num_objects":3785,"num_object_clones":0,"num_object_copies":11355,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3785,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3785,"num_write_kb":242240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,11],"acting":[1,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"70'139","reported_seq":1131,"reported_epoch":855,"state":"active+clean","last_fresh":"2024-08-19T14:45:09.109121+0000","last_change":"2024-08-19T14:44:28.862117+0000","last_active":"2024-08-19T14:45:09.109121+0000","last_peered":"2024-08-19T14:45:09.109121+0000","last_clean":"2024-08-19T14:45:09.109121+0000","last_became_active":"2024-08-19T14:42:39.071701+0000","last_became_peered":"2024-08-19T14:42:39.071701+0000","last_unstale":"2024-08-19T14:45:09.109121+0000","last_undegraded":"2024-08-19T14:45:09.109121+0000","last_fullsized":"2024-08-19T14:45:09.109121+0000","mapping_epoch":778,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":779,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:28.862033+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:44:28.862033+0000","last_clean_scrub_stamp":"2024-08-19T14:44:28.862033+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:35.447184+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,2],"acting":[4,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1","version":"856'3557","reported_seq":3627,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-08-19T14:45:10.408475+0000","last_change":"2024-08-19T14:44:15.416094+0000","last_active":"2024-08-19T14:45:10.408475+0000","last_peered":"2024-08-19T14:45:10.408475+0000","last_clean":"2024-08-19T14:45:10.408475+0000","last_became_active":"2024-08-19T14:44:15.415731+0000","last_became_peered":"2024-08-19T14:44:15.415731+0000","last_unstale":"2024-08-19T14:45:10.408475+0000","last_undegraded":"2024-08-19T14:45:10.408475+0000","last_fullsized":"2024-08-19T14:45:10.408475+0000","mapping_epoch":826,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":827,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":3557,"log_dups_size":0,"ondisk_log_size":3557,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:30.021574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233111552,"num_objects":3557,"num_object_clones":0,"num_object_copies":10671,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3557,"num_write_kb":227648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,2],"acting":[10,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.9","version":"70'139","reported_seq":1383,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-08-19T14:45:10.174866+0000","last_change":"2024-08-19T14:44:38.189250+0000","last_active":"2024-08-19T14:45:10.174866+0000","last_peered":"2024-08-19T14:45:10.174866+0000","last_clean":"2024-08-19T14:45:10.174866+0000","last_became_active":"2024-08-19T14:44:38.187345+0000","last_became_peered":"2024-08-19T14:44:38.187345+0000","last_unstale":"2024-08-19T14:45:10.174866+0000","last_undegraded":"2024-08-19T14:45:10.174866+0000","last_fullsized":"2024-08-19T14:45:10.174866+0000","mapping_epoch":837,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:24.958576+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:44:24.958576+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:41.929997+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,2],"acting":[8,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.0","version":"859'3752","reported_seq":3828,"reported_epoch":859,"state":"active+clean","last_fresh":"2024-08-19T14:45:13.683741+0000","last_change":"2024-08-19T14:44:15.066444+0000","last_active":"2024-08-19T14:45:13.683741+0000","last_peered":"2024-08-19T14:45:13.683741+0000","last_clean":"2024-08-19T14:45:13.683741+0000","last_became_active":"2024-08-19T14:44:15.065971+0000","last_became_peered":"2024-08-19T14:44:15.065971+0000","last_unstale":"2024-08-19T14:45:13.683741+0000","last_undegraded":"2024-08-19T14:45:13.683741+0000","last_fullsized":"2024-08-19T14:45:13.683741+0000","mapping_epoch":826,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":827,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":3752,"log_dups_size":0,"ondisk_log_size":3752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:29.855885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":245891072,"num_objects":3752,"num_object_clones":0,"num_object_copies":11256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3752,"num_write_kb":240128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,8],"acting":[1,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"70'139","reported_seq":651,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-08-19T14:45:10.175608+0000","last_change":"2024-08-19T14:44:38.183741+0000","last_active":"2024-08-19T14:45:10.175608+0000","last_peered":"2024-08-19T14:45:10.175608+0000","last_clean":"2024-08-19T14:45:10.175608+0000","last_became_active":"2024-08-19T14:44:38.183219+0000","last_became_peered":"2024-08-19T14:44:38.183219+0000","last_unstale":"2024-08-19T14:45:10.175608+0000","last_undegraded":"2024-08-19T14:45:10.175608+0000","last_fullsized":"2024-08-19T14:45:10.175608+0000","mapping_epoch":837,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:27.791867+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:44:27.791867+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:40.791822+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,4],"acting":[8,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.3","version":"856'3625","reported_seq":7577,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-08-19T14:45:10.743829+0000","last_change":"2024-08-19T14:45:09.095647+0000","last_active":"2024-08-19T14:45:10.743829+0000","last_peered":"2024-08-19T14:45:10.743829+0000","last_clean":"2024-08-19T14:45:10.743829+0000","last_became_active":"2024-08-19T14:45:09.091391+0000","last_became_peered":"2024-08-19T14:45:09.091391+0000","last_unstale":"2024-08-19T14:45:10.743829+0000","last_undegraded":"2024-08-19T14:45:10.743829+0000","last_fullsized":"2024-08-19T14:45:10.743829+0000","mapping_epoch":854,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":3625,"log_dups_size":0,"ondisk_log_size":3625,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:16.238416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":237568000,"num_objects":3625,"num_object_clones":0,"num_object_copies":10875,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3625,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3625,"num_write_kb":232000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1936,"num_bytes_recovered":126877696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,11],"acting":[8,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.b","version":"70'139","reported_seq":1433,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-08-19T14:45:12.187396+0000","last_change":"2024-08-19T14:44:38.183925+0000","last_active":"2024-08-19T14:45:12.187396+0000","last_peered":"2024-08-19T14:45:12.187396+0000","last_clean":"2024-08-19T14:45:12.187396+0000","last_became_active":"2024-08-19T14:44:38.183552+0000","last_became_peered":"2024-08-19T14:44:38.183552+0000","last_unstale":"2024-08-19T14:45:12.187396+0000","last_undegraded":"2024-08-19T14:45:12.187396+0000","last_fullsized":"2024-08-19T14:45:12.187396+0000","mapping_epoch":837,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:29.142655+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:44:29.142655+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:51.245420+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,1],"acting":[11,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.6","version":"856'3538","reported_seq":7543,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-08-19T14:45:10.398852+0000","last_change":"2024-08-19T14:45:08.093571+0000","last_active":"2024-08-19T14:45:10.398852+0000","last_peered":"2024-08-19T14:45:10.398852+0000","last_clean":"2024-08-19T14:45:10.398852+0000","last_became_active":"2024-08-19T14:45:08.088967+0000","last_became_peered":"2024-08-19T14:45:08.088967+0000","last_unstale":"2024-08-19T14:45:10.398852+0000","last_undegraded":"2024-08-19T14:45:10.398852+0000","last_fullsized":"2024-08-19T14:45:10.398852+0000","mapping_epoch":853,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":854,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":3538,"log_dups_size":0,"ondisk_log_size":3538,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:39.600021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":231866368,"num_objects":3538,"num_object_clones":0,"num_object_copies":10614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3538,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3538,"num_write_kb":226432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1962,"num_bytes_recovered":128581632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,9],"acting":[10,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.e","version":"70'139","reported_seq":1617,"reported_epoch":855,"state":"active+clean","last_fresh":"2024-08-19T14:45:09.109248+0000","last_change":"2024-08-19T14:44:29.815896+0000","last_active":"2024-08-19T14:45:09.109248+0000","last_peered":"2024-08-19T14:45:09.109248+0000","last_clean":"2024-08-19T14:45:09.109248+0000","last_became_active":"2024-08-19T14:43:08.463102+0000","last_became_peered":"2024-08-19T14:43:08.463102+0000","last_unstale":"2024-08-19T14:45:09.109248+0000","last_undegraded":"2024-08-19T14:45:09.109248+0000","last_fullsized":"2024-08-19T14:45:09.109248+0000","mapping_epoch":793,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":794,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:29.815798+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:40:03.764672+0000","last_clean_scrub_stamp":"2024-08-19T14:44:29.815798+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:44.328366+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,1],"acting":[4,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.7","version":"856'3436","reported_seq":3506,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-08-19T14:45:10.405465+0000","last_change":"2024-08-19T14:44:15.068003+0000","last_active":"2024-08-19T14:45:10.405465+0000","last_peered":"2024-08-19T14:45:10.405465+0000","last_clean":"2024-08-19T14:45:10.405465+0000","last_became_active":"2024-08-19T14:44:15.067635+0000","last_became_peered":"2024-08-19T14:44:15.067635+0000","last_unstale":"2024-08-19T14:45:10.405465+0000","last_undegraded":"2024-08-19T14:45:10.405465+0000","last_fullsized":"2024-08-19T14:45:10.405465+0000","mapping_epoch":826,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":827,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":3436,"log_dups_size":0,"ondisk_log_size":3436,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:40.928253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":225181696,"num_objects":3436,"num_object_clones":0,"num_object_copies":10308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3436,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3436,"num_write_kb":219904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,1],"acting":[10,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.f","version":"70'139","reported_seq":1369,"reported_epoch":855,"state":"active+clean","last_fresh":"2024-08-19T14:45:09.109193+0000","last_change":"2024-08-19T14:44:38.189541+0000","last_active":"2024-08-19T14:45:09.109193+0000","last_peered":"2024-08-19T14:45:09.109193+0000","last_clean":"2024-08-19T14:45:09.109193+0000","last_became_active":"2024-08-19T14:44:38.189293+0000","last_became_peered":"2024-08-19T14:44:38.189293+0000","last_unstale":"2024-08-19T14:45:09.109193+0000","last_undegraded":"2024-08-19T14:45:09.109193+0000","last_fullsized":"2024-08-19T14:45:09.109193+0000","mapping_epoch":837,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:31.800766+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:31.800766+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:46:01.031799+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,1],"acting":[4,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.4","version":"859'3731","reported_seq":3804,"reported_epoch":859,"state":"active+clean","last_fresh":"2024-08-19T14:45:13.671857+0000","last_change":"2024-08-19T14:44:15.066789+0000","last_active":"2024-08-19T14:45:13.671857+0000","last_peered":"2024-08-19T14:45:13.671857+0000","last_clean":"2024-08-19T14:45:13.671857+0000","last_became_active":"2024-08-19T14:44:15.065981+0000","last_became_peered":"2024-08-19T14:44:15.065981+0000","last_unstale":"2024-08-19T14:45:13.671857+0000","last_undegraded":"2024-08-19T14:45:13.671857+0000","last_fullsized":"2024-08-19T14:45:13.671857+0000","mapping_epoch":826,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":827,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":3731,"log_dups_size":0,"ondisk_log_size":3731,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:39.692151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":244514816,"num_objects":3731,"num_object_clones":0,"num_object_copies":11193,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3731,"num_write_kb":238784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.c","version":"70'139","reported_seq":684,"reported_epoch":859,"state":"active+clean","last_fresh":"2024-08-19T14:45:13.201385+0000","last_change":"2024-08-19T14:44:38.180450+0000","last_active":"2024-08-19T14:45:13.201385+0000","last_peered":"2024-08-19T14:45:13.201385+0000","last_clean":"2024-08-19T14:45:13.201385+0000","last_became_active":"2024-08-19T14:44:38.180018+0000","last_became_peered":"2024-08-19T14:44:38.180018+0000","last_unstale":"2024-08-19T14:45:13.201385+0000","last_undegraded":"2024-08-19T14:45:13.201385+0000","last_fullsized":"2024-08-19T14:45:13.201385+0000","mapping_epoch":837,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:24.706574+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:22:52.164586+0000","last_clean_scrub_stamp":"2024-08-19T14:44:24.706574+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:49.576031+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,4],"acting":[1,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.5","version":"858'3657","reported_seq":7571,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-08-19T14:45:12.604922+0000","last_change":"2024-08-19T14:45:06.219701+0000","last_active":"2024-08-19T14:45:12.604922+0000","last_peered":"2024-08-19T14:45:12.604922+0000","last_clean":"2024-08-19T14:45:12.604922+0000","last_became_active":"2024-08-19T14:45:06.216329+0000","last_became_peered":"2024-08-19T14:45:06.216329+0000","last_unstale":"2024-08-19T14:45:12.604922+0000","last_undegraded":"2024-08-19T14:45:12.604922+0000","last_fullsized":"2024-08-19T14:45:12.604922+0000","mapping_epoch":851,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":852,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":3657,"log_dups_size":0,"ondisk_log_size":3657,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:40.979416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":239665152,"num_objects":3657,"num_object_clones":0,"num_object_copies":10971,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3657,"num_write_kb":234048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1915,"num_bytes_recovered":125501440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,9],"acting":[11,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.d","version":"70'139","reported_seq":451,"reported_epoch":855,"state":"active+clean","last_fresh":"2024-08-19T14:45:09.109935+0000","last_change":"2024-08-19T14:44:24.806919+0000","last_active":"2024-08-19T14:45:09.109935+0000","last_peered":"2024-08-19T14:45:09.109935+0000","last_clean":"2024-08-19T14:45:09.109935+0000","last_became_active":"2024-08-19T14:43:08.459577+0000","last_became_peered":"2024-08-19T14:43:08.459577+0000","last_unstale":"2024-08-19T14:45:09.109935+0000","last_undegraded":"2024-08-19T14:45:09.109935+0000","last_fullsized":"2024-08-19T14:45:09.109935+0000","mapping_epoch":793,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":794,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:24.806784+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:24:50.276408+0000","last_clean_scrub_stamp":"2024-08-19T14:44:24.806784+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:52.198633+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,8],"acting":[4,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.12","version":"70'139","reported_seq":1408,"reported_epoch":859,"state":"active+clean","last_fresh":"2024-08-19T14:45:13.201375+0000","last_change":"2024-08-19T14:44:38.192587+0000","last_active":"2024-08-19T14:45:13.201375+0000","last_peered":"2024-08-19T14:45:13.201375+0000","last_clean":"2024-08-19T14:45:13.201375+0000","last_became_active":"2024-08-19T14:44:38.192380+0000","last_became_peered":"2024-08-19T14:44:38.192380+0000","last_unstale":"2024-08-19T14:45:13.201375+0000","last_undegraded":"2024-08-19T14:45:13.201375+0000","last_fullsized":"2024-08-19T14:45:13.201375+0000","mapping_epoch":837,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:30.604277+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:41:49.815826+0000","last_clean_scrub_stamp":"2024-08-19T14:44:30.604277+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:59.622743+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,8],"acting":[1,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.13","version":"70'139","reported_seq":1942,"reported_epoch":859,"state":"active+clean","last_fresh":"2024-08-19T14:45:13.201094+0000","last_change":"2024-08-19T14:43:16.087256+0000","last_active":"2024-08-19T14:45:13.201094+0000","last_peered":"2024-08-19T14:45:13.201094+0000","last_clean":"2024-08-19T14:45:13.201094+0000","last_became_active":"2024-08-19T14:43:16.086968+0000","last_became_peered":"2024-08-19T14:43:16.086968+0000","last_unstale":"2024-08-19T14:45:13.201094+0000","last_undegraded":"2024-08-19T14:45:13.201094+0000","last_fullsized":"2024-08-19T14:45:13.201094+0000","mapping_epoch":800,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":801,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:41:55.847727+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:33:35.786105+0000","last_clean_scrub_stamp":"2024-08-19T14:41:55.847727+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:17.675490+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.10","version":"70'139","reported_seq":877,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-08-19T14:45:10.178208+0000","last_change":"2024-08-19T14:44:38.187460+0000","last_active":"2024-08-19T14:45:10.178208+0000","last_peered":"2024-08-19T14:45:10.178208+0000","last_clean":"2024-08-19T14:45:10.178208+0000","last_became_active":"2024-08-19T14:44:38.187068+0000","last_became_peered":"2024-08-19T14:44:38.187068+0000","last_unstale":"2024-08-19T14:45:10.178208+0000","last_undegraded":"2024-08-19T14:45:10.178208+0000","last_fullsized":"2024-08-19T14:45:10.178208+0000","mapping_epoch":837,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:24.090078+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:34:58.616646+0000","last_clean_scrub_stamp":"2024-08-19T14:44:24.090078+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:44.737172+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,4],"acting":[10,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.11","version":"70'139","reported_seq":459,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-08-19T14:45:10.175078+0000","last_change":"2024-08-19T14:44:23.821840+0000","last_active":"2024-08-19T14:45:10.175078+0000","last_peered":"2024-08-19T14:45:10.175078+0000","last_clean":"2024-08-19T14:45:10.175078+0000","last_became_active":"2024-08-19T14:42:39.064403+0000","last_became_peered":"2024-08-19T14:42:39.064403+0000","last_unstale":"2024-08-19T14:45:10.175078+0000","last_undegraded":"2024-08-19T14:45:10.175078+0000","last_fullsized":"2024-08-19T14:45:10.175078+0000","mapping_epoch":778,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":779,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:23.821699+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:41:47.147376+0000","last_clean_scrub_stamp":"2024-08-19T14:44:23.821699+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:50.918005+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,1],"acting":[8,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"70'139","reported_seq":1570,"reported_epoch":859,"state":"active+clean","last_fresh":"2024-08-19T14:45:13.201596+0000","last_change":"2024-08-19T14:44:38.184076+0000","last_active":"2024-08-19T14:45:13.201596+0000","last_peered":"2024-08-19T14:45:13.201596+0000","last_clean":"2024-08-19T14:45:13.201596+0000","last_became_active":"2024-08-19T14:44:38.183608+0000","last_became_peered":"2024-08-19T14:44:38.183608+0000","last_unstale":"2024-08-19T14:45:13.201596+0000","last_undegraded":"2024-08-19T14:45:13.201596+0000","last_fullsized":"2024-08-19T14:45:13.201596+0000","mapping_epoch":837,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:43:04.925661+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:28:34.216370+0000","last_clean_scrub_stamp":"2024-08-19T14:43:04.925661+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:17.676007+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,4],"acting":[1,11,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"70'139","reported_seq":1714,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-08-19T14:45:10.175072+0000","last_change":"2024-08-19T14:44:38.184127+0000","last_active":"2024-08-19T14:45:10.175072+0000","last_peered":"2024-08-19T14:45:10.175072+0000","last_clean":"2024-08-19T14:45:10.175072+0000","last_became_active":"2024-08-19T14:44:38.183371+0000","last_became_peered":"2024-08-19T14:44:38.183371+0000","last_unstale":"2024-08-19T14:45:10.175072+0000","last_undegraded":"2024-08-19T14:45:10.175072+0000","last_fullsized":"2024-08-19T14:45:10.175072+0000","mapping_epoch":837,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:28.864662+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:28.864662+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:31.005692+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11],"acting":[8,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.14","version":"70'139","reported_seq":1125,"reported_epoch":859,"state":"active+clean","last_fresh":"2024-08-19T14:45:13.201247+0000","last_change":"2024-08-19T14:43:08.454671+0000","last_active":"2024-08-19T14:45:13.201247+0000","last_peered":"2024-08-19T14:45:13.201247+0000","last_clean":"2024-08-19T14:45:13.201247+0000","last_became_active":"2024-08-19T14:43:08.454350+0000","last_became_peered":"2024-08-19T14:43:08.454350+0000","last_unstale":"2024-08-19T14:45:13.201247+0000","last_undegraded":"2024-08-19T14:45:13.201247+0000","last_fullsized":"2024-08-19T14:45:13.201247+0000","mapping_epoch":793,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":794,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:42:06.543798+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:42:06.543798+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:17.675455+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,9],"acting":[1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"70'139","reported_seq":464,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-08-19T14:45:10.175245+0000","last_change":"2024-08-19T14:44:38.200208+0000","last_active":"2024-08-19T14:45:10.175245+0000","last_peered":"2024-08-19T14:45:10.175245+0000","last_clean":"2024-08-19T14:45:10.175245+0000","last_became_active":"2024-08-19T14:44:38.199514+0000","last_became_peered":"2024-08-19T14:44:38.199514+0000","last_unstale":"2024-08-19T14:45:10.175245+0000","last_undegraded":"2024-08-19T14:45:10.175245+0000","last_fullsized":"2024-08-19T14:45:10.175245+0000","mapping_epoch":837,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:24.808158+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:20:09.180923+0000","last_clean_scrub_stamp":"2024-08-19T14:44:24.808158+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:25.707601+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,11],"acting":[8,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"70'139","reported_seq":1552,"reported_epoch":855,"state":"active+clean","last_fresh":"2024-08-19T14:45:09.108954+0000","last_change":"2024-08-19T14:44:30.801086+0000","last_active":"2024-08-19T14:45:09.108954+0000","last_peered":"2024-08-19T14:45:09.108954+0000","last_clean":"2024-08-19T14:45:09.108954+0000","last_became_active":"2024-08-19T14:43:11.471324+0000","last_became_peered":"2024-08-19T14:43:11.471324+0000","last_unstale":"2024-08-19T14:45:09.108954+0000","last_undegraded":"2024-08-19T14:45:09.108954+0000","last_fullsized":"2024-08-19T14:45:09.108954+0000","mapping_epoch":795,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":796,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:30.801006+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:41:51.832284+0000","last_clean_scrub_stamp":"2024-08-19T14:44:30.801006+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:35.383040+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,11],"acting":[4,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1b","version":"70'139","reported_seq":1936,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-08-19T14:45:10.175018+0000","last_change":"2024-08-19T14:44:25.807033+0000","last_active":"2024-08-19T14:45:10.175018+0000","last_peered":"2024-08-19T14:45:10.175018+0000","last_clean":"2024-08-19T14:45:10.175018+0000","last_became_active":"2024-08-19T14:42:39.065604+0000","last_became_peered":"2024-08-19T14:42:39.065604+0000","last_unstale":"2024-08-19T14:45:10.175018+0000","last_undegraded":"2024-08-19T14:45:10.175018+0000","last_fullsized":"2024-08-19T14:45:10.175018+0000","mapping_epoch":778,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":779,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:25.806922+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:37:59.510697+0000","last_clean_scrub_stamp":"2024-08-19T14:44:25.806922+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:36.032376+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,2],"acting":[8,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.18","version":"70'139","reported_seq":930,"reported_epoch":859,"state":"active+clean","last_fresh":"2024-08-19T14:45:13.201133+0000","last_change":"2024-08-19T14:44:38.183317+0000","last_active":"2024-08-19T14:45:13.201133+0000","last_peered":"2024-08-19T14:45:13.201133+0000","last_clean":"2024-08-19T14:45:13.201133+0000","last_became_active":"2024-08-19T14:44:38.182712+0000","last_became_peered":"2024-08-19T14:44:38.182712+0000","last_unstale":"2024-08-19T14:45:13.201133+0000","last_undegraded":"2024-08-19T14:45:13.201133+0000","last_fullsized":"2024-08-19T14:45:13.201133+0000","mapping_epoch":837,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:41:53.603570+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:27:33.084161+0000","last_clean_scrub_stamp":"2024-08-19T14:41:53.603570+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:17.675626+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":402496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,11],"acting":[1,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.19","version":"70'139","reported_seq":780,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-08-19T14:45:10.180559+0000","last_change":"2024-08-19T14:44:23.083970+0000","last_active":"2024-08-19T14:45:10.180559+0000","last_peered":"2024-08-19T14:45:10.180559+0000","last_clean":"2024-08-19T14:45:10.180559+0000","last_became_active":"2024-08-19T14:43:16.096717+0000","last_became_peered":"2024-08-19T14:43:16.096717+0000","last_unstale":"2024-08-19T14:45:10.180559+0000","last_undegraded":"2024-08-19T14:45:10.180559+0000","last_fullsized":"2024-08-19T14:45:10.180559+0000","mapping_epoch":800,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":801,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:23.083891+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:44:23.083891+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:31.583064+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,8],"acting":[10,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1e","version":"70'139","reported_seq":946,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-08-19T14:45:12.187124+0000","last_change":"2024-08-19T14:43:16.126022+0000","last_active":"2024-08-19T14:45:12.187124+0000","last_peered":"2024-08-19T14:45:12.187124+0000","last_clean":"2024-08-19T14:45:12.187124+0000","last_became_active":"2024-08-19T14:43:16.124258+0000","last_became_peered":"2024-08-19T14:43:16.124258+0000","last_unstale":"2024-08-19T14:45:12.187124+0000","last_undegraded":"2024-08-19T14:45:12.187124+0000","last_fullsized":"2024-08-19T14:45:12.187124+0000","mapping_epoch":800,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":801,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:41:58.896684+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:27:04.780939+0000","last_clean_scrub_stamp":"2024-08-19T14:41:58.896684+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:13.168033+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,10],"acting":[11,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1c","version":"70'139","reported_seq":1689,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-08-19T14:45:10.182377+0000","last_change":"2024-08-19T14:44:31.796385+0000","last_active":"2024-08-19T14:45:10.182377+0000","last_peered":"2024-08-19T14:45:10.182377+0000","last_clean":"2024-08-19T14:45:10.182377+0000","last_became_active":"2024-08-19T14:30:47.908866+0000","last_became_peered":"2024-08-19T14:30:47.908866+0000","last_unstale":"2024-08-19T14:45:10.182377+0000","last_undegraded":"2024-08-19T14:45:10.182377+0000","last_fullsized":"2024-08-19T14:45:10.182377+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:31.796263+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:44:31.796263+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:38.646598+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,11],"acting":[8,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1d","version":"70'139","reported_seq":1596,"reported_epoch":859,"state":"active+clean","last_fresh":"2024-08-19T14:45:13.201299+0000","last_change":"2024-08-19T14:44:29.650415+0000","last_active":"2024-08-19T14:45:13.201299+0000","last_peered":"2024-08-19T14:45:13.201299+0000","last_clean":"2024-08-19T14:45:13.201299+0000","last_became_active":"2024-08-19T14:35:11.540488+0000","last_became_peered":"2024-08-19T14:35:11.540488+0000","last_unstale":"2024-08-19T14:45:13.201299+0000","last_undegraded":"2024-08-19T14:45:13.201299+0000","last_fullsized":"2024-08-19T14:45:13.201299+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:29.650238+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:44:29.650238+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:32.665361+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":3824943104,"num_objects":58364,"num_object_clones":0,"num_object_copies":175092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58364,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":58364,"num_write_kb":3735296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17548,"num_bytes_recovered":1150025728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12556369920,"data_stored":12556369920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":58364,"ondisk_log_size":58364,"up":48,"acting":48,"num_store_stats":8},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2785280,"data_stored":2771072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187862,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2800568,"kb_used_data":1654564,"kb_used_omap":1,"kb_used_meta":1145982,"kb_avail":90944584,"statfs":{"total":95995035648,"available":93127254016,"internally_reserved":0,"allocated":1694273536,"data_stored":1691756608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":1173486020},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":539,"seq":2314987372729,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1757008,"kb_used_data":1580676,"kb_used_omap":4,"kb_used_meta":176315,"kb_avail":91988144,"statfs":{"total":95995035648,"available":94195859456,"internally_reserved":0,"allocated":1618612224,"data_stored":1616107250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4407,"internal_metadata":180547273},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":74,"seq":317827580320,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3305572,"kb_used_data":2165148,"kb_used_omap":1,"kb_used_meta":1140414,"kb_avail":90439580,"statfs":{"total":95995035648,"available":92610129920,"internally_reserved":0,"allocated":2217111552,"data_stored":2214598176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":1167784385},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874662,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2564496,"kb_used_data":1576772,"kb_used_omap":1,"kb_used_meta":987710,"kb_avail":91180656,"statfs":{"total":95995035648,"available":93368991744,"internally_reserved":0,"allocated":1614614528,"data_stored":1612109554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1011415493},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":0,"seq":244813136297,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":206158430637,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":171798692272,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986612,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1906628,"kb_used_data":667256,"kb_used_omap":1,"kb_used_meta":1239358,"kb_avail":91838524,"statfs":{"total":95995035648,"available":94042648576,"internally_reserved":0,"allocated":683270144,"data_stored":680769375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1269103045},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":111669150139,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":634,"seq":2723009265785,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1188504,"kb_used_data":683020,"kb_used_omap":4,"kb_used_meta":505467,"kb_avail":92556648,"statfs":{"total":95995035648,"available":94778007552,"internally_reserved":0,"allocated":699412480,"data_stored":696903954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4407,"internal_metadata":517598921},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575300,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4299648,"kb_used_data":2909424,"kb_used_omap":1,"kb_used_meta":1390206,"kb_avail":89445504,"statfs":{"total":95995035648,"available":91592196096,"internally_reserved":0,"allocated":2979250176,"data_stored":2976728993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1423571397},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":34359738824,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2972516352,"data_stored":2972516352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":695205888,"data_stored":695205888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":675741696,"data_stored":675741696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1012858880,"data_stored":1012858880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1630273536,"data_stored":1630273536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":2248867840,"data_stored":2248867840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1618935808,"data_stored":1618935808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1701969920,"data_stored":1701969920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T14:45:15.982 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-19T14:45:16.031 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:45:16.052 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:16.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:16.053 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:45:16.055 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-19T14:45:16.154 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:45:16.179 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:16.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:16.180 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:45:16.194 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:45:16.194 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:45:16.280 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:45:16.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:16 smithi045 ceph-mon[20143]: pgmap v2019: 47 pgs: 47 active+clean; 3.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 59 MiB/s wr, 938 op/s; 26 MiB/s, 410 objects/s recovering 2024-08-19T14:45:16.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:16 smithi045 ceph-mon[20143]: from='client.27482 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:45:16.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:16 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:45:16.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:16 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:45:16.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:16 smithi045 ceph-mon[20143]: osdmap e860: 12 total, 12 up, 7 in 2024-08-19T14:45:16.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:16 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3295450738' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:45:16.317 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 55 16 63634 63618 72.2805 66.0625 0.0126025 0.0138295 2024-08-19T14:45:16.325 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:16.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:16.326 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:45:16.426 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:45:16.467 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:16.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:16.468 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:45:16.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:16 smithi032 ceph-mon[20004]: pgmap v2019: 47 pgs: 47 active+clean; 3.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 59 MiB/s wr, 938 op/s; 26 MiB/s, 410 objects/s recovering 2024-08-19T14:45:16.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:16 smithi032 ceph-mon[20004]: from='client.27482 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:45:16.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:16 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:45:16.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:16 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:45:16.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:16 smithi032 ceph-mon[20004]: osdmap e860: 12 total, 12 up, 7 in 2024-08-19T14:45:16.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:16 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3295450738' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:45:16.569 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:45:16.590 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:16.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:16.591 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:45:16.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:16 smithi031 ceph-mon[16336]: pgmap v2019: 47 pgs: 47 active+clean; 3.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 59 MiB/s wr, 938 op/s; 26 MiB/s, 410 objects/s recovering 2024-08-19T14:45:16.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:16 smithi031 ceph-mon[16336]: from='client.27482 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:45:16.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:16 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:45:16.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:16 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:45:16.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:16 smithi031 ceph-mon[16336]: osdmap e860: 12 total, 12 up, 7 in 2024-08-19T14:45:16.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:16 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3295450738' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:45:16.692 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:45:16.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:16.715 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:16.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:45:16.816 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:45:16.856 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:16.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:16.857 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:45:16.864 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:45:16.958 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:45:16.987 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:16.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:16.988 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:45:17.041 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-08-19T14:45:17.088 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:45:17.110 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:17.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:17.111 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:45:17.211 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:45:17.218 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:17 smithi045 ceph-mon[20143]: 1.1e scrub starts 2024-08-19T14:45:17.218 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:17 smithi045 ceph-mon[20143]: 1.1e scrub ok 2024-08-19T14:45:17.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:17.259 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:17.260 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:45:17.317 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 56 16 64909 64893 72.4126 79.6875 0.01003 0.0138034 2024-08-19T14:45:17.361 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:45:17.382 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:45:17.389 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:17.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:17.390 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:45:17.491 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:45:17.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:17 smithi032 ceph-mon[20004]: 1.1e scrub starts 2024-08-19T14:45:17.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:17 smithi032 ceph-mon[20004]: 1.1e scrub ok 2024-08-19T14:45:17.513 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:17.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:17.514 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:45:17.514 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:17 smithi031 ceph-mon[16336]: 1.1e scrub starts 2024-08-19T14:45:17.514 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:17 smithi031 ceph-mon[16336]: 1.1e scrub ok 2024-08-19T14:45:17.534 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:45:17.614 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:45:17.621 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:45:17.622 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-08-19T14:45:17.641 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:17.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:17.642 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:45:17.743 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:45:17.768 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:17.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:17.769 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:45:17.870 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:45:17.892 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:17.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:17.893 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:45:17.994 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:45:18.018 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:18.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:18.019 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:45:18.120 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:45:18.141 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:18.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:18.142 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:45:18.171 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:45:18.171 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:45:18.246 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:45:18.275 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:18.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:18.276 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:45:18.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:18 smithi045 ceph-mon[20143]: pgmap v2021: 47 pgs: 47 active+clean; 3.8 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 71 MiB/s wr, 1.13k op/s 2024-08-19T14:45:18.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:18 smithi045 ceph-mon[20143]: 9.9 scrub starts 2024-08-19T14:45:18.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:18 smithi045 ceph-mon[20143]: 9.3 scrub starts 2024-08-19T14:45:18.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:18 smithi045 ceph-mon[20143]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:45:18.318 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 57 16 65930 65914 72.2616 63.8125 0.00685407 0.0138326 2024-08-19T14:45:18.377 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:45:18.422 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:18.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:18.423 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:45:18.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:18 smithi032 ceph-mon[20004]: pgmap v2021: 47 pgs: 47 active+clean; 3.8 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 71 MiB/s wr, 1.13k op/s 2024-08-19T14:45:18.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:18 smithi032 ceph-mon[20004]: 9.9 scrub starts 2024-08-19T14:45:18.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:18 smithi032 ceph-mon[20004]: 9.3 scrub starts 2024-08-19T14:45:18.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:18 smithi032 ceph-mon[20004]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:45:18.524 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:45:18.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:18.547 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:18.547 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:45:18.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:18 smithi031 ceph-mon[16336]: pgmap v2021: 47 pgs: 47 active+clean; 3.8 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 71 MiB/s wr, 1.13k op/s 2024-08-19T14:45:18.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:18 smithi031 ceph-mon[16336]: 9.9 scrub starts 2024-08-19T14:45:18.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:18 smithi031 ceph-mon[16336]: 9.3 scrub starts 2024-08-19T14:45:18.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:18 smithi031 ceph-mon[16336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:45:18.648 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:45:18.674 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:18.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:18.675 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:45:18.776 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:45:18.799 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:18.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:18.800 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:45:18.867 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-08-19T14:45:18.881 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:45:18.901 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:45:18.953 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:18.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:18.954 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:45:19.047 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":2021,"stamp":"2024-08-19T14:45:16.368292+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4029088288,"num_objects":61472,"num_object_clones":0,"num_object_copies":184416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61470,"num_write_kb":3934080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17552,"num_bytes_recovered":1151083584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65779,"ondisk_log_size":65779,"up":141,"acting":141,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":141,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":18333820,"kb_used_data":11739940,"kb_used_omap":16,"kb_used_meta":6593775,"kb_avail":637882244,"statfs":{"total":671965249536,"available":653191417856,"internally_reserved":0,"allocated":12021698560,"data_stored":12004068792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16794,"internal_metadata":6752026214},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,3,8,6],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":385351680,"num_objects":5880,"num_object_clones":0,"num_object_copies":17640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5880,"num_write_kb":376320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.191889"},"pg_stats":[{"pgid":"9.8","version":"858'3723","reported_seq":7874,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-08-19T14:45:12.635697+0000","last_change":"2024-08-19T14:45:07.176081+0000","last_active":"2024-08-19T14:45:12.635697+0000","last_peered":"2024-08-19T14:45:12.635697+0000","last_clean":"2024-08-19T14:45:12.635697+0000","last_became_active":"2024-08-19T14:45:07.171026+0000","last_became_peered":"2024-08-19T14:45:07.171026+0000","last_unstale":"2024-08-19T14:45:12.635697+0000","last_undegraded":"2024-08-19T14:45:12.635697+0000","last_fullsized":"2024-08-19T14:45:12.635697+0000","mapping_epoch":852,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":853,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":3723,"log_dups_size":0,"ondisk_log_size":3723,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:24.356519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":243990528,"num_objects":3723,"num_object_clones":0,"num_object_copies":11169,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3723,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3723,"num_write_kb":238272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2033,"num_bytes_recovered":133234688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,1],"acting":[11,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"70'139","reported_seq":2053,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-08-19T14:45:12.189574+0000","last_change":"2024-08-19T14:44:26.109478+0000","last_active":"2024-08-19T14:45:12.189574+0000","last_peered":"2024-08-19T14:45:12.189574+0000","last_clean":"2024-08-19T14:45:12.189574+0000","last_became_active":"2024-08-19T14:43:16.089672+0000","last_became_peered":"2024-08-19T14:43:16.089672+0000","last_unstale":"2024-08-19T14:45:12.189574+0000","last_undegraded":"2024-08-19T14:45:12.189574+0000","last_fullsized":"2024-08-19T14:45:12.189574+0000","mapping_epoch":800,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":801,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:26.109206+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:41:45.177828+0000","last_clean_scrub_stamp":"2024-08-19T14:44:26.109206+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:50.139250+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,10],"acting":[11,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.b","version":"860'3930","reported_seq":7981,"reported_epoch":860,"state":"active+clean","last_fresh":"2024-08-19T14:45:15.782599+0000","last_change":"2024-08-19T14:45:09.096009+0000","last_active":"2024-08-19T14:45:15.782599+0000","last_peered":"2024-08-19T14:45:15.782599+0000","last_clean":"2024-08-19T14:45:15.782599+0000","last_became_active":"2024-08-19T14:45:09.091550+0000","last_became_peered":"2024-08-19T14:45:09.091550+0000","last_unstale":"2024-08-19T14:45:15.782599+0000","last_undegraded":"2024-08-19T14:45:15.782599+0000","last_fullsized":"2024-08-19T14:45:15.782599+0000","mapping_epoch":854,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":3930,"log_dups_size":0,"ondisk_log_size":3930,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:42.050707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":257556480,"num_objects":3930,"num_object_clones":0,"num_object_copies":11790,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3930,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3930,"num_write_kb":251520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1981,"num_bytes_recovered":129826816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,1],"acting":[8,10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"70'139","reported_seq":1900,"reported_epoch":859,"state":"active+clean","last_fresh":"2024-08-19T14:45:13.201212+0000","last_change":"2024-08-19T14:41:45.936228+0000","last_active":"2024-08-19T14:45:13.201212+0000","last_peered":"2024-08-19T14:45:13.201212+0000","last_clean":"2024-08-19T14:45:13.201212+0000","last_became_active":"2024-08-19T14:18:13.804721+0000","last_became_peered":"2024-08-19T14:18:13.804721+0000","last_unstale":"2024-08-19T14:45:13.201212+0000","last_undegraded":"2024-08-19T14:45:13.201212+0000","last_fullsized":"2024-08-19T14:45:13.201212+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:41:45.936114+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:41:45.936114+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:17.675420+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.a","version":"859'3860","reported_seq":7831,"reported_epoch":859,"state":"active+clean","last_fresh":"2024-08-19T14:45:13.684418+0000","last_change":"2024-08-19T14:44:49.108491+0000","last_active":"2024-08-19T14:45:13.684418+0000","last_peered":"2024-08-19T14:45:13.684418+0000","last_clean":"2024-08-19T14:45:13.684418+0000","last_became_active":"2024-08-19T14:44:49.104676+0000","last_became_peered":"2024-08-19T14:44:49.104676+0000","last_unstale":"2024-08-19T14:45:13.684418+0000","last_undegraded":"2024-08-19T14:45:13.684418+0000","last_fullsized":"2024-08-19T14:45:13.684418+0000","mapping_epoch":841,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":842,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":3860,"log_dups_size":0,"ondisk_log_size":3860,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:43.293459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252968960,"num_objects":3860,"num_object_clones":0,"num_object_copies":11580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3860,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3860,"num_write_kb":247040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1942,"num_bytes_recovered":127270912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,2],"acting":[1,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"70'139","reported_seq":548,"reported_epoch":859,"state":"active+clean","last_fresh":"2024-08-19T14:45:13.202433+0000","last_change":"2024-08-19T14:44:25.759421+0000","last_active":"2024-08-19T14:45:13.202433+0000","last_peered":"2024-08-19T14:45:13.202433+0000","last_clean":"2024-08-19T14:45:13.202433+0000","last_became_active":"2024-08-19T14:43:11.470639+0000","last_became_peered":"2024-08-19T14:43:11.470639+0000","last_unstale":"2024-08-19T14:45:13.202433+0000","last_undegraded":"2024-08-19T14:45:13.202433+0000","last_fullsized":"2024-08-19T14:45:13.202433+0000","mapping_epoch":795,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":796,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:25.759337+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:19:59.302931+0000","last_clean_scrub_stamp":"2024-08-19T14:44:25.759337+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:30.192194+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.9","version":"860'3952","reported_seq":7984,"reported_epoch":860,"state":"active+clean","last_fresh":"2024-08-19T14:45:15.394992+0000","last_change":"2024-08-19T14:45:09.098084+0000","last_active":"2024-08-19T14:45:15.394992+0000","last_peered":"2024-08-19T14:45:15.394992+0000","last_clean":"2024-08-19T14:45:15.394992+0000","last_became_active":"2024-08-19T14:45:09.091434+0000","last_became_peered":"2024-08-19T14:45:09.091434+0000","last_unstale":"2024-08-19T14:45:15.394992+0000","last_undegraded":"2024-08-19T14:45:15.394992+0000","last_fullsized":"2024-08-19T14:45:15.394992+0000","mapping_epoch":854,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":3952,"log_dups_size":0,"ondisk_log_size":3952,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":258998272,"num_objects":3952,"num_object_clones":0,"num_object_copies":11856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3952,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3952,"num_write_kb":252928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1972,"num_bytes_recovered":129236992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,8],"acting":[10,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1","version":"70'139","reported_seq":550,"reported_epoch":860,"state":"active+clean","last_fresh":"2024-08-19T14:45:15.216158+0000","last_change":"2024-08-19T14:44:32.095248+0000","last_active":"2024-08-19T14:45:15.216158+0000","last_peered":"2024-08-19T14:45:15.216158+0000","last_clean":"2024-08-19T14:45:15.216158+0000","last_became_active":"2024-08-19T14:43:16.098337+0000","last_became_peered":"2024-08-19T14:43:16.098337+0000","last_unstale":"2024-08-19T14:45:15.216158+0000","last_undegraded":"2024-08-19T14:45:15.216158+0000","last_fullsized":"2024-08-19T14:45:15.216158+0000","mapping_epoch":800,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":801,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:32.094375+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:44:32.094375+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:40.928652+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,2],"acting":[10,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.f","version":"859'3842","reported_seq":7807,"reported_epoch":859,"state":"active+clean","last_fresh":"2024-08-19T14:45:13.672790+0000","last_change":"2024-08-19T14:45:01.147769+0000","last_active":"2024-08-19T14:45:13.672790+0000","last_peered":"2024-08-19T14:45:13.672790+0000","last_clean":"2024-08-19T14:45:13.672790+0000","last_became_active":"2024-08-19T14:45:01.143857+0000","last_became_peered":"2024-08-19T14:45:01.143857+0000","last_unstale":"2024-08-19T14:45:13.672790+0000","last_undegraded":"2024-08-19T14:45:13.672790+0000","last_fullsized":"2024-08-19T14:45:13.672790+0000","mapping_epoch":847,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":848,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":3842,"log_dups_size":0,"ondisk_log_size":3842,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:40.066370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251789312,"num_objects":3842,"num_object_clones":0,"num_object_copies":11526,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3842,"num_write_kb":245888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1939,"num_bytes_recovered":127074304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,11],"acting":[1,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"70'139","reported_seq":1133,"reported_epoch":860,"state":"active+clean","last_fresh":"2024-08-19T14:45:15.217679+0000","last_change":"2024-08-19T14:44:38.181655+0000","last_active":"2024-08-19T14:45:15.217679+0000","last_peered":"2024-08-19T14:45:15.217679+0000","last_clean":"2024-08-19T14:45:15.217679+0000","last_became_active":"2024-08-19T14:44:38.181307+0000","last_became_peered":"2024-08-19T14:44:38.181307+0000","last_unstale":"2024-08-19T14:45:15.217679+0000","last_undegraded":"2024-08-19T14:45:15.217679+0000","last_fullsized":"2024-08-19T14:45:15.217679+0000","mapping_epoch":837,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:26.113799+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:21:46.057624+0000","last_clean_scrub_stamp":"2024-08-19T14:44:26.113799+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:52.928219+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,1],"acting":[10,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.e","version":"860'3945","reported_seq":4019,"reported_epoch":860,"state":"active+clean","last_fresh":"2024-08-19T14:45:15.388705+0000","last_change":"2024-08-19T14:44:15.415042+0000","last_active":"2024-08-19T14:45:15.388705+0000","last_peered":"2024-08-19T14:45:15.388705+0000","last_clean":"2024-08-19T14:45:15.388705+0000","last_became_active":"2024-08-19T14:44:15.414595+0000","last_became_peered":"2024-08-19T14:44:15.414595+0000","last_unstale":"2024-08-19T14:45:15.388705+0000","last_undegraded":"2024-08-19T14:45:15.388705+0000","last_fullsized":"2024-08-19T14:45:15.388705+0000","mapping_epoch":826,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":827,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":3945,"log_dups_size":0,"ondisk_log_size":3945,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:29.023228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":258539520,"num_objects":3945,"num_object_clones":0,"num_object_copies":11835,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3945,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3945,"num_write_kb":252480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,9],"acting":[10,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.6","version":"70'139","reported_seq":416,"reported_epoch":860,"state":"active+clean","last_fresh":"2024-08-19T14:45:15.217899+0000","last_change":"2024-08-19T14:44:22.089843+0000","last_active":"2024-08-19T14:45:15.217899+0000","last_peered":"2024-08-19T14:45:15.217899+0000","last_clean":"2024-08-19T14:45:15.217899+0000","last_became_active":"2024-08-19T14:43:16.091256+0000","last_became_peered":"2024-08-19T14:43:16.091256+0000","last_unstale":"2024-08-19T14:45:15.217899+0000","last_undegraded":"2024-08-19T14:45:15.217899+0000","last_fullsized":"2024-08-19T14:45:15.217899+0000","mapping_epoch":800,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":801,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:22.089766+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:27.848870+0000","last_clean_scrub_stamp":"2024-08-19T14:44:22.089766+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:42.204189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.d","version":"859'3706","reported_seq":7530,"reported_epoch":859,"state":"active+clean","last_fresh":"2024-08-19T14:45:13.676189+0000","last_change":"2024-08-19T14:45:00.248322+0000","last_active":"2024-08-19T14:45:13.676189+0000","last_peered":"2024-08-19T14:45:13.676189+0000","last_clean":"2024-08-19T14:45:13.676189+0000","last_became_active":"2024-08-19T14:45:00.244759+0000","last_became_peered":"2024-08-19T14:45:00.244759+0000","last_unstale":"2024-08-19T14:45:13.676189+0000","last_undegraded":"2024-08-19T14:45:13.676189+0000","last_fullsized":"2024-08-19T14:45:13.676189+0000","mapping_epoch":846,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":847,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":3706,"log_dups_size":0,"ondisk_log_size":3706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:26.231935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":242876416,"num_objects":3706,"num_object_clones":0,"num_object_copies":11118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3706,"num_write_kb":237184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1868,"num_bytes_recovered":122421248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"70'139","reported_seq":506,"reported_epoch":860,"state":"active+clean","last_fresh":"2024-08-19T14:45:15.216706+0000","last_change":"2024-08-19T14:44:29.076011+0000","last_active":"2024-08-19T14:45:15.216706+0000","last_peered":"2024-08-19T14:45:15.216706+0000","last_clean":"2024-08-19T14:45:15.216706+0000","last_became_active":"2024-08-19T14:43:16.099747+0000","last_became_peered":"2024-08-19T14:43:16.099747+0000","last_unstale":"2024-08-19T14:45:15.216706+0000","last_undegraded":"2024-08-19T14:45:15.216706+0000","last_fullsized":"2024-08-19T14:45:15.216706+0000","mapping_epoch":800,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":801,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:29.075645+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:41:48.878043+0000","last_clean_scrub_stamp":"2024-08-19T14:44:29.075645+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:32.760424+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,11],"acting":[10,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.c","version":"859'3901","reported_seq":3977,"reported_epoch":859,"state":"active+clean","last_fresh":"2024-08-19T14:45:14.914004+0000","last_change":"2024-08-19T14:44:15.416651+0000","last_active":"2024-08-19T14:45:14.914004+0000","last_peered":"2024-08-19T14:45:14.914004+0000","last_clean":"2024-08-19T14:45:14.914004+0000","last_became_active":"2024-08-19T14:44:15.416278+0000","last_became_peered":"2024-08-19T14:44:15.416278+0000","last_unstale":"2024-08-19T14:45:14.914004+0000","last_undegraded":"2024-08-19T14:45:14.914004+0000","last_fullsized":"2024-08-19T14:45:14.914004+0000","mapping_epoch":826,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":827,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":3901,"log_dups_size":0,"ondisk_log_size":3901,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:21.705266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":255655936,"num_objects":3901,"num_object_clones":0,"num_object_copies":11703,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3901,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3901,"num_write_kb":249664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,10],"acting":[4,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"70'139","reported_seq":972,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-08-19T14:45:12.187085+0000","last_change":"2024-08-19T14:44:33.078639+0000","last_active":"2024-08-19T14:45:12.187085+0000","last_peered":"2024-08-19T14:45:12.187085+0000","last_clean":"2024-08-19T14:45:12.187085+0000","last_became_active":"2024-08-19T14:43:08.462160+0000","last_became_peered":"2024-08-19T14:43:08.462160+0000","last_unstale":"2024-08-19T14:45:12.187085+0000","last_undegraded":"2024-08-19T14:45:12.187085+0000","last_fullsized":"2024-08-19T14:45:12.187085+0000","mapping_epoch":793,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":794,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:33.078526+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:41:55.868614+0000","last_clean_scrub_stamp":"2024-08-19T14:44:33.078526+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:46:03.059281+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,4],"acting":[11,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.2","version":"859'3785","reported_seq":3861,"reported_epoch":859,"state":"active+clean","last_fresh":"2024-08-19T14:45:13.648011+0000","last_change":"2024-08-19T14:44:15.415441+0000","last_active":"2024-08-19T14:45:13.648011+0000","last_peered":"2024-08-19T14:45:13.648011+0000","last_clean":"2024-08-19T14:45:13.648011+0000","last_became_active":"2024-08-19T14:44:15.415137+0000","last_became_peered":"2024-08-19T14:44:15.415137+0000","last_unstale":"2024-08-19T14:45:13.648011+0000","last_undegraded":"2024-08-19T14:45:13.648011+0000","last_fullsized":"2024-08-19T14:45:13.648011+0000","mapping_epoch":826,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":827,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":3785,"log_dups_size":0,"ondisk_log_size":3785,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:28.806060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":248053760,"num_objects":3785,"num_object_clones":0,"num_object_copies":11355,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3785,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3785,"num_write_kb":242240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,11],"acting":[1,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"70'139","reported_seq":1139,"reported_epoch":859,"state":"active+clean","last_fresh":"2024-08-19T14:45:13.202452+0000","last_change":"2024-08-19T14:44:28.862117+0000","last_active":"2024-08-19T14:45:13.202452+0000","last_peered":"2024-08-19T14:45:13.202452+0000","last_clean":"2024-08-19T14:45:13.202452+0000","last_became_active":"2024-08-19T14:42:39.071701+0000","last_became_peered":"2024-08-19T14:42:39.071701+0000","last_unstale":"2024-08-19T14:45:13.202452+0000","last_undegraded":"2024-08-19T14:45:13.202452+0000","last_fullsized":"2024-08-19T14:45:13.202452+0000","mapping_epoch":778,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":779,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:28.862033+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:44:28.862033+0000","last_clean_scrub_stamp":"2024-08-19T14:44:28.862033+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:35.447184+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,2],"acting":[4,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1","version":"860'3945","reported_seq":4023,"reported_epoch":860,"state":"active+clean","last_fresh":"2024-08-19T14:45:15.408885+0000","last_change":"2024-08-19T14:44:15.416094+0000","last_active":"2024-08-19T14:45:15.408885+0000","last_peered":"2024-08-19T14:45:15.408885+0000","last_clean":"2024-08-19T14:45:15.408885+0000","last_became_active":"2024-08-19T14:44:15.415731+0000","last_became_peered":"2024-08-19T14:44:15.415731+0000","last_unstale":"2024-08-19T14:45:15.408885+0000","last_undegraded":"2024-08-19T14:45:15.408885+0000","last_fullsized":"2024-08-19T14:45:15.408885+0000","mapping_epoch":826,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":827,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":3945,"log_dups_size":0,"ondisk_log_size":3945,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:30.021574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":258539520,"num_objects":3945,"num_object_clones":0,"num_object_copies":11835,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3945,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3945,"num_write_kb":252480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,2],"acting":[10,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.9","version":"70'139","reported_seq":1391,"reported_epoch":860,"state":"active+clean","last_fresh":"2024-08-19T14:45:15.215164+0000","last_change":"2024-08-19T14:44:38.189250+0000","last_active":"2024-08-19T14:45:15.215164+0000","last_peered":"2024-08-19T14:45:15.215164+0000","last_clean":"2024-08-19T14:45:15.215164+0000","last_became_active":"2024-08-19T14:44:38.187345+0000","last_became_peered":"2024-08-19T14:44:38.187345+0000","last_unstale":"2024-08-19T14:45:15.215164+0000","last_undegraded":"2024-08-19T14:45:15.215164+0000","last_fullsized":"2024-08-19T14:45:15.215164+0000","mapping_epoch":837,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:24.958576+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:44:24.958576+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:41.929997+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,2],"acting":[8,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.0","version":"859'3752","reported_seq":3828,"reported_epoch":859,"state":"active+clean","last_fresh":"2024-08-19T14:45:13.683741+0000","last_change":"2024-08-19T14:44:15.066444+0000","last_active":"2024-08-19T14:45:13.683741+0000","last_peered":"2024-08-19T14:45:13.683741+0000","last_clean":"2024-08-19T14:45:13.683741+0000","last_became_active":"2024-08-19T14:44:15.065971+0000","last_became_peered":"2024-08-19T14:44:15.065971+0000","last_unstale":"2024-08-19T14:45:13.683741+0000","last_undegraded":"2024-08-19T14:45:13.683741+0000","last_fullsized":"2024-08-19T14:45:13.683741+0000","mapping_epoch":826,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":827,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":3752,"log_dups_size":0,"ondisk_log_size":3752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:29.855885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":245891072,"num_objects":3752,"num_object_clones":0,"num_object_copies":11256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3752,"num_write_kb":240128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,8],"acting":[1,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"70'139","reported_seq":660,"reported_epoch":860,"state":"active+clean","last_fresh":"2024-08-19T14:45:15.214654+0000","last_change":"2024-08-19T14:44:38.183741+0000","last_active":"2024-08-19T14:45:15.214654+0000","last_peered":"2024-08-19T14:45:15.214654+0000","last_clean":"2024-08-19T14:45:15.214654+0000","last_became_active":"2024-08-19T14:44:38.183219+0000","last_became_peered":"2024-08-19T14:44:38.183219+0000","last_unstale":"2024-08-19T14:45:15.214654+0000","last_undegraded":"2024-08-19T14:45:15.214654+0000","last_fullsized":"2024-08-19T14:45:15.214654+0000","mapping_epoch":837,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:27.791867+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:44:27.791867+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:40.791822+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,4],"acting":[8,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.3","version":"860'3999","reported_seq":7959,"reported_epoch":860,"state":"active+clean","last_fresh":"2024-08-19T14:45:15.745387+0000","last_change":"2024-08-19T14:45:09.095647+0000","last_active":"2024-08-19T14:45:15.745387+0000","last_peered":"2024-08-19T14:45:15.745387+0000","last_clean":"2024-08-19T14:45:15.745387+0000","last_became_active":"2024-08-19T14:45:09.091391+0000","last_became_peered":"2024-08-19T14:45:09.091391+0000","last_unstale":"2024-08-19T14:45:15.745387+0000","last_undegraded":"2024-08-19T14:45:15.745387+0000","last_fullsized":"2024-08-19T14:45:15.745387+0000","mapping_epoch":854,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":3999,"log_dups_size":0,"ondisk_log_size":3999,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:16.238416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262078464,"num_objects":3999,"num_object_clones":0,"num_object_copies":11997,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3999,"num_write_kb":255936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1936,"num_bytes_recovered":126877696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,11],"acting":[8,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.b","version":"70'139","reported_seq":1433,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-08-19T14:45:12.187396+0000","last_change":"2024-08-19T14:44:38.183925+0000","last_active":"2024-08-19T14:45:12.187396+0000","last_peered":"2024-08-19T14:45:12.187396+0000","last_clean":"2024-08-19T14:45:12.187396+0000","last_became_active":"2024-08-19T14:44:38.183552+0000","last_became_peered":"2024-08-19T14:44:38.183552+0000","last_unstale":"2024-08-19T14:45:12.187396+0000","last_undegraded":"2024-08-19T14:45:12.187396+0000","last_fullsized":"2024-08-19T14:45:12.187396+0000","mapping_epoch":837,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:29.142655+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:44:29.142655+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:51.245420+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,1],"acting":[11,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.6","version":"860'3911","reported_seq":7924,"reported_epoch":860,"state":"active+clean","last_fresh":"2024-08-19T14:45:15.396761+0000","last_change":"2024-08-19T14:45:08.093571+0000","last_active":"2024-08-19T14:45:15.396761+0000","last_peered":"2024-08-19T14:45:15.396761+0000","last_clean":"2024-08-19T14:45:15.396761+0000","last_became_active":"2024-08-19T14:45:08.088967+0000","last_became_peered":"2024-08-19T14:45:08.088967+0000","last_unstale":"2024-08-19T14:45:15.396761+0000","last_undegraded":"2024-08-19T14:45:15.396761+0000","last_fullsized":"2024-08-19T14:45:15.396761+0000","mapping_epoch":853,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":854,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":3911,"log_dups_size":0,"ondisk_log_size":3911,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:39.600021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256311296,"num_objects":3911,"num_object_clones":0,"num_object_copies":11733,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3911,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3911,"num_write_kb":250304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1962,"num_bytes_recovered":128581632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,9],"acting":[10,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.e","version":"70'139","reported_seq":1625,"reported_epoch":859,"state":"active+clean","last_fresh":"2024-08-19T14:45:13.202455+0000","last_change":"2024-08-19T14:44:29.815896+0000","last_active":"2024-08-19T14:45:13.202455+0000","last_peered":"2024-08-19T14:45:13.202455+0000","last_clean":"2024-08-19T14:45:13.202455+0000","last_became_active":"2024-08-19T14:43:08.463102+0000","last_became_peered":"2024-08-19T14:43:08.463102+0000","last_unstale":"2024-08-19T14:45:13.202455+0000","last_undegraded":"2024-08-19T14:45:13.202455+0000","last_fullsized":"2024-08-19T14:45:13.202455+0000","mapping_epoch":793,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":794,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:29.815798+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:40:03.764672+0000","last_clean_scrub_stamp":"2024-08-19T14:44:29.815798+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:44.328366+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,1],"acting":[4,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.7","version":"860'3831","reported_seq":3909,"reported_epoch":860,"state":"active+clean","last_fresh":"2024-08-19T14:45:15.354577+0000","last_change":"2024-08-19T14:44:15.068003+0000","last_active":"2024-08-19T14:45:15.354577+0000","last_peered":"2024-08-19T14:45:15.354577+0000","last_clean":"2024-08-19T14:45:15.354577+0000","last_became_active":"2024-08-19T14:44:15.067635+0000","last_became_peered":"2024-08-19T14:44:15.067635+0000","last_unstale":"2024-08-19T14:45:15.354577+0000","last_undegraded":"2024-08-19T14:45:15.354577+0000","last_fullsized":"2024-08-19T14:45:15.354577+0000","mapping_epoch":826,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":827,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":3831,"log_dups_size":0,"ondisk_log_size":3831,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:40.928253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251068416,"num_objects":3831,"num_object_clones":0,"num_object_copies":11493,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3831,"num_write_kb":245184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,1],"acting":[10,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.f","version":"70'139","reported_seq":1377,"reported_epoch":859,"state":"active+clean","last_fresh":"2024-08-19T14:45:13.202415+0000","last_change":"2024-08-19T14:44:38.189541+0000","last_active":"2024-08-19T14:45:13.202415+0000","last_peered":"2024-08-19T14:45:13.202415+0000","last_clean":"2024-08-19T14:45:13.202415+0000","last_became_active":"2024-08-19T14:44:38.189293+0000","last_became_peered":"2024-08-19T14:44:38.189293+0000","last_unstale":"2024-08-19T14:45:13.202415+0000","last_undegraded":"2024-08-19T14:45:13.202415+0000","last_fullsized":"2024-08-19T14:45:13.202415+0000","mapping_epoch":837,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:31.800766+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:31.800766+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:46:01.031799+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,1],"acting":[4,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.4","version":"859'3731","reported_seq":3804,"reported_epoch":859,"state":"active+clean","last_fresh":"2024-08-19T14:45:13.671857+0000","last_change":"2024-08-19T14:44:15.066789+0000","last_active":"2024-08-19T14:45:13.671857+0000","last_peered":"2024-08-19T14:45:13.671857+0000","last_clean":"2024-08-19T14:45:13.671857+0000","last_became_active":"2024-08-19T14:44:15.065981+0000","last_became_peered":"2024-08-19T14:44:15.065981+0000","last_unstale":"2024-08-19T14:45:13.671857+0000","last_undegraded":"2024-08-19T14:45:13.671857+0000","last_fullsized":"2024-08-19T14:45:13.671857+0000","mapping_epoch":826,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":827,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":3731,"log_dups_size":0,"ondisk_log_size":3731,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:39.692151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":244514816,"num_objects":3731,"num_object_clones":0,"num_object_copies":11193,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3731,"num_write_kb":238784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.c","version":"70'139","reported_seq":684,"reported_epoch":859,"state":"active+clean","last_fresh":"2024-08-19T14:45:13.201385+0000","last_change":"2024-08-19T14:44:38.180450+0000","last_active":"2024-08-19T14:45:13.201385+0000","last_peered":"2024-08-19T14:45:13.201385+0000","last_clean":"2024-08-19T14:45:13.201385+0000","last_became_active":"2024-08-19T14:44:38.180018+0000","last_became_peered":"2024-08-19T14:44:38.180018+0000","last_unstale":"2024-08-19T14:45:13.201385+0000","last_undegraded":"2024-08-19T14:45:13.201385+0000","last_fullsized":"2024-08-19T14:45:13.201385+0000","mapping_epoch":837,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:24.706574+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:22:52.164586+0000","last_clean_scrub_stamp":"2024-08-19T14:44:24.706574+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:49.576031+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,4],"acting":[1,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.5","version":"858'3657","reported_seq":7571,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-08-19T14:45:12.604922+0000","last_change":"2024-08-19T14:45:06.219701+0000","last_active":"2024-08-19T14:45:12.604922+0000","last_peered":"2024-08-19T14:45:12.604922+0000","last_clean":"2024-08-19T14:45:12.604922+0000","last_became_active":"2024-08-19T14:45:06.216329+0000","last_became_peered":"2024-08-19T14:45:06.216329+0000","last_unstale":"2024-08-19T14:45:12.604922+0000","last_undegraded":"2024-08-19T14:45:12.604922+0000","last_fullsized":"2024-08-19T14:45:12.604922+0000","mapping_epoch":851,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":852,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":3657,"log_dups_size":0,"ondisk_log_size":3657,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:40.979416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":239665152,"num_objects":3657,"num_object_clones":0,"num_object_copies":10971,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3657,"num_write_kb":234048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1915,"num_bytes_recovered":125501440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,9],"acting":[11,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.d","version":"70'139","reported_seq":459,"reported_epoch":859,"state":"active+clean","last_fresh":"2024-08-19T14:45:13.203538+0000","last_change":"2024-08-19T14:44:24.806919+0000","last_active":"2024-08-19T14:45:13.203538+0000","last_peered":"2024-08-19T14:45:13.203538+0000","last_clean":"2024-08-19T14:45:13.203538+0000","last_became_active":"2024-08-19T14:43:08.459577+0000","last_became_peered":"2024-08-19T14:43:08.459577+0000","last_unstale":"2024-08-19T14:45:13.203538+0000","last_undegraded":"2024-08-19T14:45:13.203538+0000","last_fullsized":"2024-08-19T14:45:13.203538+0000","mapping_epoch":793,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":794,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:24.806784+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:24:50.276408+0000","last_clean_scrub_stamp":"2024-08-19T14:44:24.806784+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:52.198633+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,8],"acting":[4,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.12","version":"70'139","reported_seq":1408,"reported_epoch":859,"state":"active+clean","last_fresh":"2024-08-19T14:45:13.201375+0000","last_change":"2024-08-19T14:44:38.192587+0000","last_active":"2024-08-19T14:45:13.201375+0000","last_peered":"2024-08-19T14:45:13.201375+0000","last_clean":"2024-08-19T14:45:13.201375+0000","last_became_active":"2024-08-19T14:44:38.192380+0000","last_became_peered":"2024-08-19T14:44:38.192380+0000","last_unstale":"2024-08-19T14:45:13.201375+0000","last_undegraded":"2024-08-19T14:45:13.201375+0000","last_fullsized":"2024-08-19T14:45:13.201375+0000","mapping_epoch":837,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:30.604277+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:41:49.815826+0000","last_clean_scrub_stamp":"2024-08-19T14:44:30.604277+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:59.622743+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,8],"acting":[1,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.13","version":"70'139","reported_seq":1942,"reported_epoch":859,"state":"active+clean","last_fresh":"2024-08-19T14:45:13.201094+0000","last_change":"2024-08-19T14:43:16.087256+0000","last_active":"2024-08-19T14:45:13.201094+0000","last_peered":"2024-08-19T14:45:13.201094+0000","last_clean":"2024-08-19T14:45:13.201094+0000","last_became_active":"2024-08-19T14:43:16.086968+0000","last_became_peered":"2024-08-19T14:43:16.086968+0000","last_unstale":"2024-08-19T14:45:13.201094+0000","last_undegraded":"2024-08-19T14:45:13.201094+0000","last_fullsized":"2024-08-19T14:45:13.201094+0000","mapping_epoch":800,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":801,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:41:55.847727+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:33:35.786105+0000","last_clean_scrub_stamp":"2024-08-19T14:41:55.847727+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:17.675490+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.10","version":"70'139","reported_seq":885,"reported_epoch":860,"state":"active+clean","last_fresh":"2024-08-19T14:45:15.215781+0000","last_change":"2024-08-19T14:44:38.187460+0000","last_active":"2024-08-19T14:45:15.215781+0000","last_peered":"2024-08-19T14:45:15.215781+0000","last_clean":"2024-08-19T14:45:15.215781+0000","last_became_active":"2024-08-19T14:44:38.187068+0000","last_became_peered":"2024-08-19T14:44:38.187068+0000","last_unstale":"2024-08-19T14:45:15.215781+0000","last_undegraded":"2024-08-19T14:45:15.215781+0000","last_fullsized":"2024-08-19T14:45:15.215781+0000","mapping_epoch":837,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:24.090078+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:34:58.616646+0000","last_clean_scrub_stamp":"2024-08-19T14:44:24.090078+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:44.737172+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,4],"acting":[10,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.11","version":"70'139","reported_seq":467,"reported_epoch":860,"state":"active+clean","last_fresh":"2024-08-19T14:45:15.215336+0000","last_change":"2024-08-19T14:44:23.821840+0000","last_active":"2024-08-19T14:45:15.215336+0000","last_peered":"2024-08-19T14:45:15.215336+0000","last_clean":"2024-08-19T14:45:15.215336+0000","last_became_active":"2024-08-19T14:42:39.064403+0000","last_became_peered":"2024-08-19T14:42:39.064403+0000","last_unstale":"2024-08-19T14:45:15.215336+0000","last_undegraded":"2024-08-19T14:45:15.215336+0000","last_fullsized":"2024-08-19T14:45:15.215336+0000","mapping_epoch":778,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":779,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:23.821699+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:41:47.147376+0000","last_clean_scrub_stamp":"2024-08-19T14:44:23.821699+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:50.918005+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,1],"acting":[8,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"70'139","reported_seq":1570,"reported_epoch":859,"state":"active+clean","last_fresh":"2024-08-19T14:45:13.201596+0000","last_change":"2024-08-19T14:44:38.184076+0000","last_active":"2024-08-19T14:45:13.201596+0000","last_peered":"2024-08-19T14:45:13.201596+0000","last_clean":"2024-08-19T14:45:13.201596+0000","last_became_active":"2024-08-19T14:44:38.183608+0000","last_became_peered":"2024-08-19T14:44:38.183608+0000","last_unstale":"2024-08-19T14:45:13.201596+0000","last_undegraded":"2024-08-19T14:45:13.201596+0000","last_fullsized":"2024-08-19T14:45:13.201596+0000","mapping_epoch":837,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:43:04.925661+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:28:34.216370+0000","last_clean_scrub_stamp":"2024-08-19T14:43:04.925661+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:17.676007+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,4],"acting":[1,11,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"70'139","reported_seq":1722,"reported_epoch":860,"state":"active+clean","last_fresh":"2024-08-19T14:45:15.206828+0000","last_change":"2024-08-19T14:44:38.184127+0000","last_active":"2024-08-19T14:45:15.206828+0000","last_peered":"2024-08-19T14:45:15.206828+0000","last_clean":"2024-08-19T14:45:15.206828+0000","last_became_active":"2024-08-19T14:44:38.183371+0000","last_became_peered":"2024-08-19T14:44:38.183371+0000","last_unstale":"2024-08-19T14:45:15.206828+0000","last_undegraded":"2024-08-19T14:45:15.206828+0000","last_fullsized":"2024-08-19T14:45:15.206828+0000","mapping_epoch":837,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:28.864662+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:28.864662+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:31.005692+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11],"acting":[8,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.14","version":"70'139","reported_seq":1125,"reported_epoch":859,"state":"active+clean","last_fresh":"2024-08-19T14:45:13.201247+0000","last_change":"2024-08-19T14:43:08.454671+0000","last_active":"2024-08-19T14:45:13.201247+0000","last_peered":"2024-08-19T14:45:13.201247+0000","last_clean":"2024-08-19T14:45:13.201247+0000","last_became_active":"2024-08-19T14:43:08.454350+0000","last_became_peered":"2024-08-19T14:43:08.454350+0000","last_unstale":"2024-08-19T14:45:13.201247+0000","last_undegraded":"2024-08-19T14:45:13.201247+0000","last_fullsized":"2024-08-19T14:45:13.201247+0000","mapping_epoch":793,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":794,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:42:06.543798+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:42:06.543798+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:17.675455+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,9],"acting":[1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"70'139","reported_seq":472,"reported_epoch":860,"state":"active+clean","last_fresh":"2024-08-19T14:45:15.206687+0000","last_change":"2024-08-19T14:44:38.200208+0000","last_active":"2024-08-19T14:45:15.206687+0000","last_peered":"2024-08-19T14:45:15.206687+0000","last_clean":"2024-08-19T14:45:15.206687+0000","last_became_active":"2024-08-19T14:44:38.199514+0000","last_became_peered":"2024-08-19T14:44:38.199514+0000","last_unstale":"2024-08-19T14:45:15.206687+0000","last_undegraded":"2024-08-19T14:45:15.206687+0000","last_fullsized":"2024-08-19T14:45:15.206687+0000","mapping_epoch":837,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:24.808158+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:20:09.180923+0000","last_clean_scrub_stamp":"2024-08-19T14:44:24.808158+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:25.707601+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,11],"acting":[8,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"70'139","reported_seq":1560,"reported_epoch":859,"state":"active+clean","last_fresh":"2024-08-19T14:45:13.202344+0000","last_change":"2024-08-19T14:44:30.801086+0000","last_active":"2024-08-19T14:45:13.202344+0000","last_peered":"2024-08-19T14:45:13.202344+0000","last_clean":"2024-08-19T14:45:13.202344+0000","last_became_active":"2024-08-19T14:43:11.471324+0000","last_became_peered":"2024-08-19T14:43:11.471324+0000","last_unstale":"2024-08-19T14:45:13.202344+0000","last_undegraded":"2024-08-19T14:45:13.202344+0000","last_fullsized":"2024-08-19T14:45:13.202344+0000","mapping_epoch":795,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":796,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:30.801006+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:41:51.832284+0000","last_clean_scrub_stamp":"2024-08-19T14:44:30.801006+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:35.383040+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,11],"acting":[4,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1b","version":"70'139","reported_seq":1944,"reported_epoch":860,"state":"active+clean","last_fresh":"2024-08-19T14:45:15.207458+0000","last_change":"2024-08-19T14:44:25.807033+0000","last_active":"2024-08-19T14:45:15.207458+0000","last_peered":"2024-08-19T14:45:15.207458+0000","last_clean":"2024-08-19T14:45:15.207458+0000","last_became_active":"2024-08-19T14:42:39.065604+0000","last_became_peered":"2024-08-19T14:42:39.065604+0000","last_unstale":"2024-08-19T14:45:15.207458+0000","last_undegraded":"2024-08-19T14:45:15.207458+0000","last_fullsized":"2024-08-19T14:45:15.207458+0000","mapping_epoch":778,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":779,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:25.806922+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:37:59.510697+0000","last_clean_scrub_stamp":"2024-08-19T14:44:25.806922+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:36.032376+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,2],"acting":[8,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.18","version":"70'139","reported_seq":930,"reported_epoch":859,"state":"active+clean","last_fresh":"2024-08-19T14:45:13.201133+0000","last_change":"2024-08-19T14:44:38.183317+0000","last_active":"2024-08-19T14:45:13.201133+0000","last_peered":"2024-08-19T14:45:13.201133+0000","last_clean":"2024-08-19T14:45:13.201133+0000","last_became_active":"2024-08-19T14:44:38.182712+0000","last_became_peered":"2024-08-19T14:44:38.182712+0000","last_unstale":"2024-08-19T14:45:13.201133+0000","last_undegraded":"2024-08-19T14:45:13.201133+0000","last_fullsized":"2024-08-19T14:45:13.201133+0000","mapping_epoch":837,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:41:53.603570+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:27:33.084161+0000","last_clean_scrub_stamp":"2024-08-19T14:41:53.603570+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:17.675626+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":402496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,11],"acting":[1,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.19","version":"70'139","reported_seq":788,"reported_epoch":860,"state":"active+clean","last_fresh":"2024-08-19T14:45:15.216261+0000","last_change":"2024-08-19T14:44:23.083970+0000","last_active":"2024-08-19T14:45:15.216261+0000","last_peered":"2024-08-19T14:45:15.216261+0000","last_clean":"2024-08-19T14:45:15.216261+0000","last_became_active":"2024-08-19T14:43:16.096717+0000","last_became_peered":"2024-08-19T14:43:16.096717+0000","last_unstale":"2024-08-19T14:45:15.216261+0000","last_undegraded":"2024-08-19T14:45:15.216261+0000","last_fullsized":"2024-08-19T14:45:15.216261+0000","mapping_epoch":800,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":801,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:23.083891+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:44:23.083891+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:31.583064+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,8],"acting":[10,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1e","version":"70'139","reported_seq":946,"reported_epoch":858,"state":"active+clean","last_fresh":"2024-08-19T14:45:12.187124+0000","last_change":"2024-08-19T14:43:16.126022+0000","last_active":"2024-08-19T14:45:12.187124+0000","last_peered":"2024-08-19T14:45:12.187124+0000","last_clean":"2024-08-19T14:45:12.187124+0000","last_became_active":"2024-08-19T14:43:16.124258+0000","last_became_peered":"2024-08-19T14:43:16.124258+0000","last_unstale":"2024-08-19T14:45:12.187124+0000","last_undegraded":"2024-08-19T14:45:12.187124+0000","last_fullsized":"2024-08-19T14:45:12.187124+0000","mapping_epoch":800,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":801,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:41:58.896684+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:27:04.780939+0000","last_clean_scrub_stamp":"2024-08-19T14:41:58.896684+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:13.168033+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,10],"acting":[11,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1c","version":"70'139","reported_seq":1697,"reported_epoch":860,"state":"active+clean","last_fresh":"2024-08-19T14:45:15.211374+0000","last_change":"2024-08-19T14:44:31.796385+0000","last_active":"2024-08-19T14:45:15.211374+0000","last_peered":"2024-08-19T14:45:15.211374+0000","last_clean":"2024-08-19T14:45:15.211374+0000","last_became_active":"2024-08-19T14:30:47.908866+0000","last_became_peered":"2024-08-19T14:30:47.908866+0000","last_unstale":"2024-08-19T14:45:15.211374+0000","last_undegraded":"2024-08-19T14:45:15.211374+0000","last_fullsized":"2024-08-19T14:45:15.211374+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:31.796263+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:44:31.796263+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:38.646598+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,11],"acting":[8,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1d","version":"70'139","reported_seq":1596,"reported_epoch":859,"state":"active+clean","last_fresh":"2024-08-19T14:45:13.201299+0000","last_change":"2024-08-19T14:44:29.650415+0000","last_active":"2024-08-19T14:45:13.201299+0000","last_peered":"2024-08-19T14:45:13.201299+0000","last_clean":"2024-08-19T14:45:13.201299+0000","last_became_active":"2024-08-19T14:35:11.540488+0000","last_became_peered":"2024-08-19T14:35:11.540488+0000","last_unstale":"2024-08-19T14:45:13.201299+0000","last_undegraded":"2024-08-19T14:45:13.201299+0000","last_fullsized":"2024-08-19T14:45:13.201299+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:29.650238+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:44:29.650238+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:45:32.665361+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":4028497920,"num_objects":61470,"num_object_clones":0,"num_object_copies":184410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61470,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61470,"num_write_kb":3934080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17548,"num_bytes_recovered":1150025728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13064404992,"data_stored":13064404992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":61470,"ondisk_log_size":61470,"up":48,"acting":48,"num_store_stats":8},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2785280,"data_stored":2771072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187862,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2800568,"kb_used_data":1654564,"kb_used_omap":1,"kb_used_meta":1145982,"kb_avail":90944584,"statfs":{"total":95995035648,"available":93127254016,"internally_reserved":0,"allocated":1694273536,"data_stored":1691756608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":1173486020},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":539,"seq":2314987372730,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1941556,"kb_used_data":1756904,"kb_used_omap":4,"kb_used_meta":184635,"kb_avail":91803596,"statfs":{"total":95995035648,"available":94006882304,"internally_reserved":0,"allocated":1799069696,"data_stored":1796552065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4407,"internal_metadata":189066953},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":74,"seq":317827580320,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3305572,"kb_used_data":2165148,"kb_used_omap":1,"kb_used_meta":1140414,"kb_avail":90439580,"statfs":{"total":95995035648,"available":92610129920,"internally_reserved":0,"allocated":2217111552,"data_stored":2214598176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":1167784385},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874663,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2748612,"kb_used_data":1752564,"kb_used_omap":1,"kb_used_meta":996030,"kb_avail":90996540,"statfs":{"total":95995035648,"available":93180456960,"internally_reserved":0,"allocated":1794625536,"data_stored":1792103700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1019935173},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":0,"seq":244813136298,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":206158430638,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":171798692272,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986613,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1970224,"kb_used_data":743336,"kb_used_omap":1,"kb_used_meta":1226878,"kb_avail":91774928,"statfs":{"total":95995035648,"available":93977526272,"internally_reserved":0,"allocated":761176064,"data_stored":758658433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1256323525},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":111669150140,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":634,"seq":2723009265786,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1267640,"kb_used_data":758000,"kb_used_omap":4,"kb_used_meta":509627,"kb_avail":92477512,"statfs":{"total":95995035648,"available":94696972288,"internally_reserved":0,"allocated":776192000,"data_stored":773670817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4407,"internal_metadata":521858761},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575300,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4299648,"kb_used_data":2909424,"kb_used_omap":1,"kb_used_meta":1390206,"kb_avail":89445504,"statfs":{"total":95995035648,"available":91592196096,"internally_reserved":0,"allocated":2979250176,"data_stored":2976728993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1423571397},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":34359738825,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2972516352,"data_stored":2972516352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":771227648,"data_stored":771227648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":754909184,"data_stored":754909184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1012858880,"data_stored":1012858880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1804271616,"data_stored":1804271616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":2248867840,"data_stored":2248867840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1797783552,"data_stored":1797783552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1701969920,"data_stored":1701969920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T14:45:19.050 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-08-19T14:45:19.050 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-19T14:45:19.050 INFO:tasks.thrashosds.thrasher:Adding osd 6 2024-08-19T14:45:19.051 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd in 6 2024-08-19T14:45:19.054 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:45:19.110 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:19.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:19.111 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:45:19.212 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:45:19.223 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:45:19.228 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:19 smithi045 ceph-mon[20143]: from='client.17955 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:45:19.228 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:19 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3428282245' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:45:19.228 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:19 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:45:19.229 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is unset 2024-08-19T14:45:19.262 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:19.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:19.264 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:45:19.318 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 58 16 67053 67037 72.2258 70.1875 0.00801465 0.0138396 2024-08-19T14:45:19.337 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:45:19.365 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:45:19.410 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:19.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:19.412 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:45:19.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:19 smithi032 ceph-mon[20004]: from='client.17955 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:45:19.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:19 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3428282245' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:45:19.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:19 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:45:19.512 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:45:19.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:19.558 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:19.558 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:45:19.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:19 smithi031 ceph-mon[16336]: from='client.17955 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:45:19.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:19 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3428282245' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:45:19.561 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:19 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:45:19.659 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:45:19.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:19.687 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:19.687 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:45:19.788 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:45:19.813 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:19.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:19.814 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:45:19.915 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:45:19.940 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:19.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:19.941 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:45:19.983 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:45:19.983 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:45:20.041 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:45:20.063 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:20.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:20.064 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:45:20.165 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:45:20.206 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:20.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:20.207 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:45:20.308 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:45:20.317 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 59 16 68327 68311 72.3511 79.625 0.00842845 0.013816 2024-08-19T14:45:20.334 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:20.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:20.335 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:45:20.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:20 smithi031 ceph-mon[16336]: pgmap v2022: 47 pgs: 47 active+clean; 3.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 68 MiB/s wr, 1.08k op/s 2024-08-19T14:45:20.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:20 smithi031 ceph-mon[16336]: 9.3 scrub ok 2024-08-19T14:45:20.433 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:20 smithi031 ceph-mon[16336]: 9.9 scrub ok 2024-08-19T14:45:20.433 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:20 smithi031 ceph-mon[16336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:45:20.433 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:20 smithi031 ceph-mon[16336]: Cluster is now healthy 2024-08-19T14:45:20.433 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:20 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:45:20.433 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:20 smithi031 ceph-mon[16336]: osdmap e861: 12 total, 12 up, 7 in 2024-08-19T14:45:20.436 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:45:20.466 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:20.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:20.467 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:45:20.510 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:20 smithi032 ceph-mon[20004]: pgmap v2022: 47 pgs: 47 active+clean; 3.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 68 MiB/s wr, 1.08k op/s 2024-08-19T14:45:20.510 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:20 smithi032 ceph-mon[20004]: 9.3 scrub ok 2024-08-19T14:45:20.510 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:20 smithi032 ceph-mon[20004]: 9.9 scrub ok 2024-08-19T14:45:20.510 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:20 smithi032 ceph-mon[20004]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:45:20.510 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:20 smithi032 ceph-mon[20004]: Cluster is now healthy 2024-08-19T14:45:20.510 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:20 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:45:20.510 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:20 smithi032 ceph-mon[20004]: osdmap e861: 12 total, 12 up, 7 in 2024-08-19T14:45:20.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:20 smithi045 ceph-mon[20143]: pgmap v2022: 47 pgs: 47 active+clean; 3.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 68 MiB/s wr, 1.08k op/s 2024-08-19T14:45:20.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:20 smithi045 ceph-mon[20143]: 9.3 scrub ok 2024-08-19T14:45:20.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:20 smithi045 ceph-mon[20143]: 9.9 scrub ok 2024-08-19T14:45:20.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:20 smithi045 ceph-mon[20143]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:45:20.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:20 smithi045 ceph-mon[20143]: Cluster is now healthy 2024-08-19T14:45:20.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:20 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:45:20.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:20 smithi045 ceph-mon[20143]: osdmap e861: 12 total, 12 up, 7 in 2024-08-19T14:45:20.554 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:45:20.567 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:45:20.567 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:45:20.568 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:45:20.600 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:45:20.600 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:45:20.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:20.612 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:20.612 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:45:20.629 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:45:20.630 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:45:20.653 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:45:20.654 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:45:20.674 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:45:20.674 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:45:20.697 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:45:20.697 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:45:20.709 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-08-19T14:45:20.713 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:45:20.722 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:45:20.723 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:45:20.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:20.738 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:20.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:45:20.743 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:45:20.744 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:45:20.771 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:45:20.771 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:45:20.792 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:45:20.793 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:45:20.820 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:45:20.821 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:45:20.839 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:45:20.847 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:45:20.847 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:45:20.866 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:20.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:20.867 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:45:20.968 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:45:20.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:20.993 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:20.994 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:45:21.095 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:45:21.122 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:21.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:21.126 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:45:21.163 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:45:21.226 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:45:21.253 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:21.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:21.254 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:45:21.261 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:21 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/470153231' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:45:21.261 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:21 smithi031 ceph-mon[16336]: 1.3 scrub starts 2024-08-19T14:45:21.261 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:21 smithi031 ceph-mon[16336]: 1.3 scrub ok 2024-08-19T14:45:21.319 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:2024-08-19T14:45:21.318897+0000 min lat: 0.00105666 max lat: 2.17994 avg lat: 0.0138033 2024-08-19T14:45:21.319 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-19T14:45:21.319 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 60 16 69551 69535 72.4191 76.5 0.00841515 0.0138033 2024-08-19T14:45:21.355 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:45:21.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:21.403 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:21.404 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:45:21.504 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:45:21.506 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:21 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/470153231' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:45:21.506 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:21 smithi045 ceph-mon[20143]: 1.3 scrub starts 2024-08-19T14:45:21.506 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:21 smithi045 ceph-mon[20143]: 1.3 scrub ok 2024-08-19T14:45:21.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:21 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/470153231' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:45:21.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:21 smithi032 ceph-mon[20004]: 1.3 scrub starts 2024-08-19T14:45:21.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:21 smithi032 ceph-mon[20004]: 1.3 scrub ok 2024-08-19T14:45:21.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:21.529 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:21.530 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:45:21.631 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:45:21.653 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:21.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:21.654 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:45:21.755 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:45:21.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:21.796 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:21.796 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:45:21.809 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:45:21.809 INFO:teuthology.orchestra.run.smithi045.stdout: "size": 20, 2024-08-19T14:45:21.809 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 600, 2024-08-19T14:45:21.809 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [ 2024-08-19T14:45:21.809 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.809 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.27235.0:25430 9.1 9:8d490f00:::benchmark_data_smithi045_152862_object25429:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e832)", 2024-08-19T14:45:21.810 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:44:31.503088+0000", 2024-08-19T14:45:21.810 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 50.301010050999999, 2024-08-19T14:45:21.810 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.21076894600000001, 2024-08-19T14:45:21.810 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:45:21.810 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:45:21.810 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:45:21.810 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:45:21.810 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.27235", 2024-08-19T14:45:21.810 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2746512116", 2024-08-19T14:45:21.810 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 25430 2024-08-19T14:45:21.810 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.811 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:45:21.811 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.811 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:45:21.811 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.503088+0000", 2024-08-19T14:45:21.811 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:45:21.811 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.811 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.811 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:45:21.811 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.503088+0000", 2024-08-19T14:45:21.811 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:45:21.811 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.811 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.812 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:45:21.812 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.503088+0000", 2024-08-19T14:45:21.812 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.6899999999999997e-07 2024-08-19T14:45:21.812 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.812 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.812 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:45:21.812 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.503119+0000", 2024-08-19T14:45:21.812 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.0338e-05 2024-08-19T14:45:21.812 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.812 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.812 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:45:21.813 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.503132+0000", 2024-08-19T14:45:21.813 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2952999999999999e-05 2024-08-19T14:45:21.813 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.813 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.813 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:45:21.813 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.503138+0000", 2024-08-19T14:45:21.813 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.2179999999999996e-06 2024-08-19T14:45:21.813 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.813 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.813 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:45:21.813 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.503469+0000", 2024-08-19T14:45:21.813 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00033091300000000001 2024-08-19T14:45:21.814 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.814 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.814 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:45:21.814 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.503573+0000", 2024-08-19T14:45:21.814 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000104423 2024-08-19T14:45:21.814 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.814 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.814 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2,9", 2024-08-19T14:45:21.814 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.506232+0000", 2024-08-19T14:45:21.814 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0026583739999999998 2024-08-19T14:45:21.814 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.814 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.815 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:45:21.815 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.508215+0000", 2024-08-19T14:45:21.815 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0019829230000000002 2024-08-19T14:45:21.815 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.815 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.815 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:45:21.815 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.508675+0000", 2024-08-19T14:45:21.815 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00046045500000000002 2024-08-19T14:45:21.815 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.815 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.815 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:45:21.815 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.713757+0000", 2024-08-19T14:45:21.815 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20508206000000001 2024-08-19T14:45:21.816 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.816 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.816 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:45:21.816 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.713815+0000", 2024-08-19T14:45:21.816 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7584999999999998e-05 2024-08-19T14:45:21.816 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.816 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.817 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:45:21.817 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.713857+0000", 2024-08-19T14:45:21.817 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.2135000000000002e-05 2024-08-19T14:45:21.817 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:45:21.817 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:45:21.818 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:45:21.818 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.818 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.818 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.27235.0:25464 9.7 9:e1ad4beb:::benchmark_data_smithi045_152862_object25463:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e832)", 2024-08-19T14:45:21.818 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:44:31.516434+0000", 2024-08-19T14:45:21.818 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 50.287664231000001, 2024-08-19T14:45:21.818 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.19739658700000001, 2024-08-19T14:45:21.818 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:45:21.818 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:45:21.818 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:45:21.819 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:45:21.819 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.27235", 2024-08-19T14:45:21.819 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2746512116", 2024-08-19T14:45:21.819 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 25464 2024-08-19T14:45:21.819 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.819 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:45:21.819 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.819 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:45:21.819 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.516434+0000", 2024-08-19T14:45:21.819 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:45:21.820 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.820 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.820 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:45:21.820 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.516434+0000", 2024-08-19T14:45:21.820 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:45:21.820 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.820 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.820 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:45:21.820 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.516434+0000", 2024-08-19T14:45:21.820 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2799999999999996e-07 2024-08-19T14:45:21.821 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.821 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.821 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:45:21.821 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.516455+0000", 2024-08-19T14:45:21.821 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1100000000000001e-05 2024-08-19T14:45:21.821 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.821 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.821 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:45:21.821 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.516463+0000", 2024-08-19T14:45:21.821 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.1650000000000006e-06 2024-08-19T14:45:21.822 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.822 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.822 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:45:21.822 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.516469+0000", 2024-08-19T14:45:21.822 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.3159999999999996e-06 2024-08-19T14:45:21.822 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.822 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.822 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:45:21.822 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.517821+0000", 2024-08-19T14:45:21.822 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0013523809999999999 2024-08-19T14:45:21.822 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.822 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.823 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:45:21.823 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.517951+0000", 2024-08-19T14:45:21.823 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000129457 2024-08-19T14:45:21.823 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.823 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.823 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,2", 2024-08-19T14:45:21.823 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.518080+0000", 2024-08-19T14:45:21.823 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012928899999999999 2024-08-19T14:45:21.823 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.823 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.824 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:45:21.824 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.519589+0000", 2024-08-19T14:45:21.824 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0015089560000000001 2024-08-19T14:45:21.824 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.824 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.824 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:45:21.824 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.520529+0000", 2024-08-19T14:45:21.824 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00094052999999999997 2024-08-19T14:45:21.824 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.824 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.825 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:45:21.825 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.713762+0000", 2024-08-19T14:45:21.825 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.19323251699999999 2024-08-19T14:45:21.825 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.825 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.825 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:45:21.825 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.713799+0000", 2024-08-19T14:45:21.825 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.6772e-05 2024-08-19T14:45:21.825 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.825 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.825 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:45:21.825 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.713830+0000", 2024-08-19T14:45:21.825 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.1575999999999997e-05 2024-08-19T14:45:21.825 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:45:21.826 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:45:21.826 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:45:21.826 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.826 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.826 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.27235.0:25466 9.1 9:89684d63:::benchmark_data_smithi045_152862_object25465:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e832)", 2024-08-19T14:45:21.826 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:44:31.516483+0000", 2024-08-19T14:45:21.826 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 50.287614425999998, 2024-08-19T14:45:21.826 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.19753478199999999, 2024-08-19T14:45:21.827 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:45:21.827 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:45:21.827 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:45:21.827 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:45:21.827 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.27235", 2024-08-19T14:45:21.827 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2746512116", 2024-08-19T14:45:21.827 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 25466 2024-08-19T14:45:21.827 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.828 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:45:21.828 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.828 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:45:21.828 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.516483+0000", 2024-08-19T14:45:21.828 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:45:21.828 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.828 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.828 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:45:21.828 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.516483+0000", 2024-08-19T14:45:21.828 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:45:21.828 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.828 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.828 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:45:21.829 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.516484+0000", 2024-08-19T14:45:21.829 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.9500000000000003e-07 2024-08-19T14:45:21.829 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.829 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.829 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:45:21.829 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.516504+0000", 2024-08-19T14:45:21.829 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9953e-05 2024-08-19T14:45:21.829 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.829 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.829 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:45:21.829 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.516512+0000", 2024-08-19T14:45:21.830 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.2719999999999997e-06 2024-08-19T14:45:21.830 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.830 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.830 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:45:21.831 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.516518+0000", 2024-08-19T14:45:21.831 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.6450000000000002e-06 2024-08-19T14:45:21.831 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.831 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.831 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:45:21.831 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.516577+0000", 2024-08-19T14:45:21.831 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.9039000000000002e-05 2024-08-19T14:45:21.831 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.831 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.831 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:45:21.831 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.516698+0000", 2024-08-19T14:45:21.831 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012094700000000001 2024-08-19T14:45:21.831 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.832 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.832 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2,9", 2024-08-19T14:45:21.832 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.516824+0000", 2024-08-19T14:45:21.832 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012580299999999999 2024-08-19T14:45:21.832 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.832 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.832 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:45:21.832 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.518242+0000", 2024-08-19T14:45:21.832 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0014180250000000001 2024-08-19T14:45:21.832 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.832 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.833 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:45:21.833 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.523705+0000", 2024-08-19T14:45:21.833 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0054631660000000002 2024-08-19T14:45:21.833 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.833 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.833 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:45:21.833 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.713913+0000", 2024-08-19T14:45:21.833 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.190207969 2024-08-19T14:45:21.833 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.833 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.833 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:45:21.833 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.713978+0000", 2024-08-19T14:45:21.833 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.5530000000000004e-05 2024-08-19T14:45:21.834 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.834 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.834 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:45:21.834 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.714018+0000", 2024-08-19T14:45:21.834 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.9938000000000003e-05 2024-08-19T14:45:21.834 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:45:21.834 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:45:21.834 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:45:21.834 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.834 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.834 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.27235.0:25476 9.1 9:8dc83432:::benchmark_data_smithi045_152862_object25475:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e832)", 2024-08-19T14:45:21.834 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:44:31.519666+0000", 2024-08-19T14:45:21.834 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 50.284432289999998, 2024-08-19T14:45:21.834 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.19448850500000001, 2024-08-19T14:45:21.835 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:45:21.835 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:45:21.835 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:45:21.835 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:45:21.835 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.27235", 2024-08-19T14:45:21.835 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2746512116", 2024-08-19T14:45:21.835 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 25476 2024-08-19T14:45:21.835 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.835 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:45:21.835 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.835 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:45:21.835 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.519666+0000", 2024-08-19T14:45:21.835 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:45:21.835 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.836 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.836 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:45:21.836 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.519666+0000", 2024-08-19T14:45:21.836 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:45:21.836 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.836 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.836 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:45:21.836 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.519666+0000", 2024-08-19T14:45:21.836 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.7400000000000003e-07 2024-08-19T14:45:21.836 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.836 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.836 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:45:21.836 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.519702+0000", 2024-08-19T14:45:21.836 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.6047000000000002e-05 2024-08-19T14:45:21.837 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.837 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.837 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:45:21.837 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.519713+0000", 2024-08-19T14:45:21.837 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0893999999999999e-05 2024-08-19T14:45:21.837 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.837 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.837 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:45:21.837 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.519721+0000", 2024-08-19T14:45:21.837 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4370000000000001e-06 2024-08-19T14:45:21.837 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.837 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.837 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:45:21.837 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.519758+0000", 2024-08-19T14:45:21.838 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.7740000000000001e-05 2024-08-19T14:45:21.838 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.838 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.838 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:45:21.838 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.519849+0000", 2024-08-19T14:45:21.838 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.0550999999999994e-05 2024-08-19T14:45:21.838 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.838 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.838 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2,9", 2024-08-19T14:45:21.838 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.519989+0000", 2024-08-19T14:45:21.838 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000139646 2024-08-19T14:45:21.838 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.838 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.838 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:45:21.839 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.522150+0000", 2024-08-19T14:45:21.839 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0021612229999999999 2024-08-19T14:45:21.839 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.839 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.839 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:45:21.839 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.526472+0000", 2024-08-19T14:45:21.839 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0043217280000000004 2024-08-19T14:45:21.839 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.839 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.839 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:45:21.839 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.714072+0000", 2024-08-19T14:45:21.839 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.18760026499999999 2024-08-19T14:45:21.839 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.839 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.840 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:45:21.840 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.714128+0000", 2024-08-19T14:45:21.840 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.6057000000000002e-05 2024-08-19T14:45:21.840 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.840 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.840 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:45:21.841 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.714154+0000", 2024-08-19T14:45:21.841 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.6242999999999999e-05 2024-08-19T14:45:21.841 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:45:21.841 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:45:21.841 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:45:21.841 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.841 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.841 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.27235.0:25477 9.1 9:8dc4e08e:::benchmark_data_smithi045_152862_object25476:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e832)", 2024-08-19T14:45:21.841 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:44:31.519738+0000", 2024-08-19T14:45:21.841 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 50.284359791, 2024-08-19T14:45:21.841 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.194554594, 2024-08-19T14:45:21.841 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:45:21.841 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:45:21.841 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:45:21.842 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:45:21.842 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.27235", 2024-08-19T14:45:21.842 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2746512116", 2024-08-19T14:45:21.842 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 25477 2024-08-19T14:45:21.842 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.842 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:45:21.842 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.842 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:45:21.842 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.519738+0000", 2024-08-19T14:45:21.842 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:45:21.842 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.842 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.842 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:45:21.843 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.519738+0000", 2024-08-19T14:45:21.844 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:45:21.844 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.844 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.844 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:45:21.844 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.519739+0000", 2024-08-19T14:45:21.844 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.9999999999999998e-07 2024-08-19T14:45:21.845 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.845 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.845 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:45:21.845 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.519762+0000", 2024-08-19T14:45:21.845 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.3594000000000001e-05 2024-08-19T14:45:21.845 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.845 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.845 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:45:21.845 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.519770+0000", 2024-08-19T14:45:21.845 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.174e-06 2024-08-19T14:45:21.845 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.846 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.846 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:45:21.846 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.519776+0000", 2024-08-19T14:45:21.846 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.9540000000000001e-06 2024-08-19T14:45:21.846 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.846 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.846 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:45:21.846 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.520971+0000", 2024-08-19T14:45:21.846 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0011950750000000001 2024-08-19T14:45:21.846 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.846 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.846 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:45:21.846 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.521083+0000", 2024-08-19T14:45:21.847 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000111847 2024-08-19T14:45:21.847 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.847 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.847 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2,9", 2024-08-19T14:45:21.847 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.521212+0000", 2024-08-19T14:45:21.847 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000128849 2024-08-19T14:45:21.847 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.847 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.847 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:45:21.847 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.523892+0000", 2024-08-19T14:45:21.847 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0026795930000000001 2024-08-19T14:45:21.847 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.847 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.848 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:45:21.848 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.528034+0000", 2024-08-19T14:45:21.848 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.004142766 2024-08-19T14:45:21.848 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.848 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.848 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:45:21.848 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.714212+0000", 2024-08-19T14:45:21.848 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.18617767900000001 2024-08-19T14:45:21.848 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.848 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.848 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:45:21.848 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.714256+0000", 2024-08-19T14:45:21.848 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.3491000000000001e-05 2024-08-19T14:45:21.849 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.849 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.849 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:45:21.849 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.714293+0000", 2024-08-19T14:45:21.849 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.7072e-05 2024-08-19T14:45:21.849 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:45:21.849 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:45:21.849 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:45:21.849 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.849 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.849 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.27235.0:25481 9.1 9:88465d9f:::benchmark_data_smithi045_152862_object25480:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e832)", 2024-08-19T14:45:21.849 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:44:31.522230+0000", 2024-08-19T14:45:21.849 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 50.281868265999996, 2024-08-19T14:45:21.850 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.19220008399999999, 2024-08-19T14:45:21.850 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:45:21.850 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:45:21.850 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:45:21.850 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:45:21.850 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.27235", 2024-08-19T14:45:21.851 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2746512116", 2024-08-19T14:45:21.851 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 25481 2024-08-19T14:45:21.851 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.851 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:45:21.851 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.851 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:45:21.851 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.522230+0000", 2024-08-19T14:45:21.851 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:45:21.851 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.851 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.851 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:45:21.851 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.522230+0000", 2024-08-19T14:45:21.852 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:45:21.852 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.852 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.852 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:45:21.852 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.522231+0000", 2024-08-19T14:45:21.852 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.06e-06 2024-08-19T14:45:21.852 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.852 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.852 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:45:21.852 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.522248+0000", 2024-08-19T14:45:21.852 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7686e-05 2024-08-19T14:45:21.852 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.852 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.852 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:45:21.853 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.522264+0000", 2024-08-19T14:45:21.853 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5930000000000002e-05 2024-08-19T14:45:21.853 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.853 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.853 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:45:21.853 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.522272+0000", 2024-08-19T14:45:21.853 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4279999999999999e-06 2024-08-19T14:45:21.853 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.853 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.854 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:45:21.854 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.522324+0000", 2024-08-19T14:45:21.854 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2717999999999999e-05 2024-08-19T14:45:21.854 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.854 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.854 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:45:21.854 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.522620+0000", 2024-08-19T14:45:21.854 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00029540600000000001 2024-08-19T14:45:21.854 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.854 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.854 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2,9", 2024-08-19T14:45:21.854 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.522797+0000", 2024-08-19T14:45:21.854 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00017683 2024-08-19T14:45:21.855 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.855 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.855 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:45:21.855 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.524038+0000", 2024-08-19T14:45:21.855 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00124176 2024-08-19T14:45:21.855 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.855 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.855 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:45:21.855 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.528076+0000", 2024-08-19T14:45:21.855 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0040373880000000003 2024-08-19T14:45:21.855 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.855 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.855 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:45:21.855 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.714331+0000", 2024-08-19T14:45:21.855 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.186255114 2024-08-19T14:45:21.856 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.856 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.856 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:45:21.856 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.714385+0000", 2024-08-19T14:45:21.856 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.3995000000000001e-05 2024-08-19T14:45:21.856 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.856 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.856 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:45:21.856 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.714430+0000", 2024-08-19T14:45:21.856 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.4768999999999998e-05 2024-08-19T14:45:21.856 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:45:21.856 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:45:21.856 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:45:21.856 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.857 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.857 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.27235.0:25499 9.7 9:e6785d94:::benchmark_data_smithi045_152862_object25498:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e832)", 2024-08-19T14:45:21.857 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:44:31.530558+0000", 2024-08-19T14:45:21.857 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 50.273539767999999, 2024-08-19T14:45:21.857 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.18339192300000001, 2024-08-19T14:45:21.857 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:45:21.857 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:45:21.857 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:45:21.857 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:45:21.857 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.27235", 2024-08-19T14:45:21.857 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2746512116", 2024-08-19T14:45:21.857 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 25499 2024-08-19T14:45:21.857 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.857 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:45:21.857 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.858 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:45:21.858 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.530558+0000", 2024-08-19T14:45:21.858 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:45:21.858 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.858 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.858 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:45:21.858 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.530558+0000", 2024-08-19T14:45:21.858 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:45:21.858 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.858 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.858 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:45:21.858 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.530559+0000", 2024-08-19T14:45:21.859 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.9100000000000004e-07 2024-08-19T14:45:21.859 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.859 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.859 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:45:21.859 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.530581+0000", 2024-08-19T14:45:21.860 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2226999999999999e-05 2024-08-19T14:45:21.860 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.860 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.860 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:45:21.860 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.530589+0000", 2024-08-19T14:45:21.860 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.1110000000000008e-06 2024-08-19T14:45:21.860 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.860 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.860 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:45:21.860 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.530595+0000", 2024-08-19T14:45:21.861 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.5600000000000001e-06 2024-08-19T14:45:21.861 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.861 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.861 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:45:21.861 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.530874+0000", 2024-08-19T14:45:21.861 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00027961499999999999 2024-08-19T14:45:21.861 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.861 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.861 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:45:21.861 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.531037+0000", 2024-08-19T14:45:21.861 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000162671 2024-08-19T14:45:21.861 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.862 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.862 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,2", 2024-08-19T14:45:21.862 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.531184+0000", 2024-08-19T14:45:21.862 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014745700000000001 2024-08-19T14:45:21.862 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.862 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.862 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:45:21.862 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.533530+0000", 2024-08-19T14:45:21.862 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0023458590000000001 2024-08-19T14:45:21.862 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.862 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.862 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:45:21.862 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.534784+0000", 2024-08-19T14:45:21.863 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001253582 2024-08-19T14:45:21.863 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.863 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.863 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:45:21.863 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.713870+0000", 2024-08-19T14:45:21.863 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.17908626499999999 2024-08-19T14:45:21.863 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.863 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.863 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:45:21.863 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.713909+0000", 2024-08-19T14:45:21.863 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.9313000000000001e-05 2024-08-19T14:45:21.864 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.864 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.864 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:45:21.864 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.713950+0000", 2024-08-19T14:45:21.864 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.0672e-05 2024-08-19T14:45:21.864 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:45:21.864 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:45:21.864 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:45:21.864 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.864 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.864 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.27235.0:25507 9.7 9:e7c365af:::benchmark_data_smithi045_152862_object25506:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e832)", 2024-08-19T14:45:21.864 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:44:31.535380+0000", 2024-08-19T14:45:21.865 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 50.268717379000002, 2024-08-19T14:45:21.865 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.18069521499999999, 2024-08-19T14:45:21.865 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:45:21.865 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:45:21.865 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:45:21.865 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:45:21.865 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.27235", 2024-08-19T14:45:21.865 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2746512116", 2024-08-19T14:45:21.865 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 25507 2024-08-19T14:45:21.865 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.865 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:45:21.865 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.866 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:45:21.866 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.535380+0000", 2024-08-19T14:45:21.866 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:45:21.866 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.866 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.866 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:45:21.866 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.535380+0000", 2024-08-19T14:45:21.866 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:45:21.866 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.866 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.866 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:45:21.866 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.535382+0000", 2024-08-19T14:45:21.867 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.043e-06 2024-08-19T14:45:21.867 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.867 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.867 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:45:21.867 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.535406+0000", 2024-08-19T14:45:21.867 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.4451999999999999e-05 2024-08-19T14:45:21.867 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.867 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.867 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:45:21.867 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.535417+0000", 2024-08-19T14:45:21.867 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1088e-05 2024-08-19T14:45:21.867 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.867 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.868 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:45:21.868 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.535422+0000", 2024-08-19T14:45:21.868 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2719999999999997e-06 2024-08-19T14:45:21.868 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.868 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.868 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:45:21.868 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.535474+0000", 2024-08-19T14:45:21.868 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1907999999999998e-05 2024-08-19T14:45:21.868 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.868 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.868 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:45:21.868 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.535633+0000", 2024-08-19T14:45:21.868 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015829900000000001 2024-08-19T14:45:21.868 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.869 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.869 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,2", 2024-08-19T14:45:21.869 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.535791+0000", 2024-08-19T14:45:21.869 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015795499999999999 2024-08-19T14:45:21.869 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.869 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.869 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:45:21.869 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.537235+0000", 2024-08-19T14:45:21.869 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0014443220000000001 2024-08-19T14:45:21.869 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.869 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.869 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:45:21.869 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.538557+0000", 2024-08-19T14:45:21.869 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0013216840000000001 2024-08-19T14:45:21.870 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.870 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.870 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:45:21.870 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.715984+0000", 2024-08-19T14:45:21.870 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.177427581 2024-08-19T14:45:21.870 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.870 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.870 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:45:21.870 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.716036+0000", 2024-08-19T14:45:21.870 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2351000000000001e-05 2024-08-19T14:45:21.871 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.871 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.871 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:45:21.871 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.716076+0000", 2024-08-19T14:45:21.871 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.926e-05 2024-08-19T14:45:21.871 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:45:21.871 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:45:21.871 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:45:21.871 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.872 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.872 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.27235.0:25512 9.7 9:e24b56b9:::benchmark_data_smithi045_152862_object25511:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e832)", 2024-08-19T14:45:21.872 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:44:31.537229+0000", 2024-08-19T14:45:21.872 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 50.266869137999997, 2024-08-19T14:45:21.872 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.17905564099999999, 2024-08-19T14:45:21.872 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:45:21.872 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:45:21.872 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:45:21.872 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:45:21.873 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.27235", 2024-08-19T14:45:21.873 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2746512116", 2024-08-19T14:45:21.873 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 25512 2024-08-19T14:45:21.873 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.873 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:45:21.873 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.873 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:45:21.873 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.537229+0000", 2024-08-19T14:45:21.873 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:45:21.873 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.873 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.873 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:45:21.873 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.537229+0000", 2024-08-19T14:45:21.873 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:45:21.874 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.874 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.874 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:45:21.874 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.537230+0000", 2024-08-19T14:45:21.874 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1769999999999999e-06 2024-08-19T14:45:21.874 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.874 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.874 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:45:21.874 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.537253+0000", 2024-08-19T14:45:21.874 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.3023e-05 2024-08-19T14:45:21.874 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.874 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.874 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:45:21.875 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.537264+0000", 2024-08-19T14:45:21.875 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1127999999999999e-05 2024-08-19T14:45:21.875 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.875 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.875 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:45:21.875 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.537271+0000", 2024-08-19T14:45:21.875 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.8549999999999996e-06 2024-08-19T14:45:21.875 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.875 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.875 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:45:21.876 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.537317+0000", 2024-08-19T14:45:21.876 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.5884999999999999e-05 2024-08-19T14:45:21.876 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.876 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.876 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:45:21.876 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.537454+0000", 2024-08-19T14:45:21.876 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000137439 2024-08-19T14:45:21.876 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.876 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.876 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,2", 2024-08-19T14:45:21.876 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.537587+0000", 2024-08-19T14:45:21.876 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013312099999999999 2024-08-19T14:45:21.876 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.877 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.877 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:45:21.877 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.538741+0000", 2024-08-19T14:45:21.877 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0011533649999999999 2024-08-19T14:45:21.877 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.877 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.877 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:45:21.877 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.539188+0000", 2024-08-19T14:45:21.877 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00044710099999999999 2024-08-19T14:45:21.877 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.877 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.877 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:45:21.878 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.716113+0000", 2024-08-19T14:45:21.878 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.17692523099999999 2024-08-19T14:45:21.878 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.878 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.878 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:45:21.878 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.716245+0000", 2024-08-19T14:45:21.878 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000131667 2024-08-19T14:45:21.878 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.878 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.878 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:45:21.878 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.716284+0000", 2024-08-19T14:45:21.878 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.9648999999999999e-05 2024-08-19T14:45:21.878 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:45:21.879 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:45:21.879 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:45:21.879 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.879 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.879 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.27235.0:25520 9.1 9:849f197e:::benchmark_data_smithi045_152862_object25519:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e832)", 2024-08-19T14:45:21.879 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:44:31.539990+0000", 2024-08-19T14:45:21.879 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 50.264107453000001, 2024-08-19T14:45:21.879 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.17610424199999999, 2024-08-19T14:45:21.879 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:45:21.879 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:45:21.879 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:45:21.879 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:45:21.879 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.27235", 2024-08-19T14:45:21.879 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2746512116", 2024-08-19T14:45:21.880 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 25520 2024-08-19T14:45:21.880 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.880 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:45:21.880 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.880 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:45:21.880 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.539990+0000", 2024-08-19T14:45:21.880 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:45:21.880 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.880 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.880 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:45:21.880 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.539990+0000", 2024-08-19T14:45:21.880 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:45:21.880 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.880 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.881 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:45:21.881 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.539991+0000", 2024-08-19T14:45:21.881 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.2600000000000002e-07 2024-08-19T14:45:21.881 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.881 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.881 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:45:21.882 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.540013+0000", 2024-08-19T14:45:21.882 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1582000000000001e-05 2024-08-19T14:45:21.882 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.882 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.882 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:45:21.882 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.540021+0000", 2024-08-19T14:45:21.882 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.4209999999999995e-06 2024-08-19T14:45:21.882 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.882 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.882 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:45:21.882 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.540027+0000", 2024-08-19T14:45:21.882 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4709999999999998e-06 2024-08-19T14:45:21.882 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.883 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.883 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:45:21.883 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.540084+0000", 2024-08-19T14:45:21.883 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7544000000000003e-05 2024-08-19T14:45:21.883 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.883 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.883 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:45:21.883 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.540481+0000", 2024-08-19T14:45:21.883 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000396955 2024-08-19T14:45:21.884 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.884 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.884 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2,9", 2024-08-19T14:45:21.884 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.540618+0000", 2024-08-19T14:45:21.884 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013654699999999999 2024-08-19T14:45:21.884 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.884 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.884 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:45:21.884 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.542326+0000", 2024-08-19T14:45:21.884 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001708542 2024-08-19T14:45:21.884 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.884 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.884 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:45:21.886 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.542593+0000", 2024-08-19T14:45:21.886 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00026667100000000001 2024-08-19T14:45:21.886 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.886 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.886 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:45:21.886 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.716018+0000", 2024-08-19T14:45:21.887 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.173425158 2024-08-19T14:45:21.887 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.887 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.887 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:45:21.888 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.716063+0000", 2024-08-19T14:45:21.888 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.5130000000000003e-05 2024-08-19T14:45:21.888 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.888 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.888 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:45:21.888 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.716095+0000", 2024-08-19T14:45:21.888 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.1594999999999998e-05 2024-08-19T14:45:21.888 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:45:21.888 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:45:21.888 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:45:21.889 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.889 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.889 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.27235.0:25521 9.7 9:e1ac95ba:::benchmark_data_smithi045_152862_object25520:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e832)", 2024-08-19T14:45:21.889 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:44:31.541561+0000", 2024-08-19T14:45:21.889 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 50.262537076000001, 2024-08-19T14:45:21.889 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.17488937600000001, 2024-08-19T14:45:21.889 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:45:21.889 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:45:21.889 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:45:21.889 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:45:21.889 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.27235", 2024-08-19T14:45:21.889 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2746512116", 2024-08-19T14:45:21.889 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 25521 2024-08-19T14:45:21.890 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.890 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:45:21.890 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.890 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:45:21.890 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.541561+0000", 2024-08-19T14:45:21.890 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:45:21.890 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.890 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.891 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:45:21.891 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.541561+0000", 2024-08-19T14:45:21.891 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:45:21.891 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.891 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.891 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:45:21.891 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.541562+0000", 2024-08-19T14:45:21.892 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.6599999999999994e-07 2024-08-19T14:45:21.892 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.892 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.892 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:45:21.892 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.541579+0000", 2024-08-19T14:45:21.893 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6982000000000001e-05 2024-08-19T14:45:21.893 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.893 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.893 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:45:21.893 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.541588+0000", 2024-08-19T14:45:21.893 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.594e-06 2024-08-19T14:45:21.893 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.893 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.894 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:45:21.894 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.541594+0000", 2024-08-19T14:45:21.894 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.293e-06 2024-08-19T14:45:21.894 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.894 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.894 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:45:21.894 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.541651+0000", 2024-08-19T14:45:21.894 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7074999999999998e-05 2024-08-19T14:45:21.894 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.894 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.894 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:45:21.894 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.541784+0000", 2024-08-19T14:45:21.894 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013289899999999999 2024-08-19T14:45:21.895 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.895 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.895 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,2", 2024-08-19T14:45:21.895 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.542001+0000", 2024-08-19T14:45:21.895 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00021722199999999999 2024-08-19T14:45:21.895 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.895 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.896 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:45:21.896 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.543057+0000", 2024-08-19T14:45:21.896 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0010557609999999999 2024-08-19T14:45:21.896 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.896 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.897 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:21 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mon-a[16332]: 2024-08-19T14:45:21.611+0000 7f2ff503e640 -1 mon.a@0(leader).osd e861 definitely_dead 0 2024-08-19T14:45:21.898 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:45:21.898 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.543897+0000", 2024-08-19T14:45:21.898 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00084019300000000004 2024-08-19T14:45:21.898 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.898 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.898 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:45:21.898 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.716358+0000", 2024-08-19T14:45:21.898 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.17246145099999999 2024-08-19T14:45:21.898 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.898 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.898 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:45:21.898 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.716401+0000", 2024-08-19T14:45:21.899 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.2865999999999999e-05 2024-08-19T14:45:21.899 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.899 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.899 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:45:21.899 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.716450+0000", 2024-08-19T14:45:21.899 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.9073999999999998e-05 2024-08-19T14:45:21.899 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:45:21.899 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:45:21.899 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:45:21.899 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.899 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.899 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.27235.0:25527 9.7 9:e73de830:::benchmark_data_smithi045_152862_object25526:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e832)", 2024-08-19T14:45:21.899 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:44:31.545161+0000", 2024-08-19T14:45:21.899 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 50.258936962999996, 2024-08-19T14:45:21.900 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.17146969200000001, 2024-08-19T14:45:21.900 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:45:21.900 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:45:21.900 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:45:21.900 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:45:21.900 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.27235", 2024-08-19T14:45:21.900 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2746512116", 2024-08-19T14:45:21.900 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 25527 2024-08-19T14:45:21.900 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.900 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:45:21.900 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.900 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:45:21.900 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.545161+0000", 2024-08-19T14:45:21.901 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:45:21.901 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.901 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.901 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:45:21.901 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.545161+0000", 2024-08-19T14:45:21.901 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:45:21.901 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.901 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.901 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:45:21.901 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.545162+0000", 2024-08-19T14:45:21.901 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.7400000000000002e-07 2024-08-19T14:45:21.901 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.901 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.902 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:45:21.902 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.545181+0000", 2024-08-19T14:45:21.902 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9151999999999999e-05 2024-08-19T14:45:21.902 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.902 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.902 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:45:21.902 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.545188+0000", 2024-08-19T14:45:21.902 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.6179999999999997e-06 2024-08-19T14:45:21.902 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.902 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.902 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:45:21.902 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.545193+0000", 2024-08-19T14:45:21.902 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.6589999999999998e-06 2024-08-19T14:45:21.903 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.903 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.903 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:45:21.903 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.545233+0000", 2024-08-19T14:45:21.903 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.9422000000000003e-05 2024-08-19T14:45:21.903 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.903 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.903 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:45:21.903 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.545331+0000", 2024-08-19T14:45:21.903 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.8102000000000004e-05 2024-08-19T14:45:21.903 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.903 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.903 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,2", 2024-08-19T14:45:21.904 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.545437+0000", 2024-08-19T14:45:21.904 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00010631700000000001 2024-08-19T14:45:21.904 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.904 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.904 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:45:21.904 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.547449+0000", 2024-08-19T14:45:21.904 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.002012205 2024-08-19T14:45:21.905 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.905 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.905 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:45:21.905 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.547634+0000", 2024-08-19T14:45:21.906 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00018477500000000001 2024-08-19T14:45:21.906 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.906 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.906 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:45:21.907 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.716510+0000", 2024-08-19T14:45:21.907 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.16887623199999999 2024-08-19T14:45:21.907 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.907 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.907 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:45:21.907 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.716563+0000", 2024-08-19T14:45:21.907 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2516999999999999e-05 2024-08-19T14:45:21.907 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.907 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.907 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:45:21.907 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.716631+0000", 2024-08-19T14:45:21.908 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.7918999999999998e-05 2024-08-19T14:45:21.908 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:45:21.908 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:45:21.908 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:45:21.908 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.908 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.908 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.27235.0:25531 9.7 9:efc7b067:::benchmark_data_smithi045_152862_object25530:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e832)", 2024-08-19T14:45:21.908 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:44:31.546165+0000", 2024-08-19T14:45:21.909 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 50.257932992000001, 2024-08-19T14:45:21.909 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.170603053, 2024-08-19T14:45:21.909 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:45:21.909 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:45:21.909 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:45:21.909 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:45:21.909 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.27235", 2024-08-19T14:45:21.909 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2746512116", 2024-08-19T14:45:21.909 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 25531 2024-08-19T14:45:21.910 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.910 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:45:21.910 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.910 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:45:21.910 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.546165+0000", 2024-08-19T14:45:21.910 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:45:21.910 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.910 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.910 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:45:21.910 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.546165+0000", 2024-08-19T14:45:21.910 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:45:21.910 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.911 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.911 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:45:21.911 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.546166+0000", 2024-08-19T14:45:21.911 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0240000000000001e-06 2024-08-19T14:45:21.912 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.912 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.912 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:45:21.912 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.546187+0000", 2024-08-19T14:45:21.912 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0786999999999999e-05 2024-08-19T14:45:21.912 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.912 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.912 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:45:21.912 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.546201+0000", 2024-08-19T14:45:21.912 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4121000000000001e-05 2024-08-19T14:45:21.912 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.912 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.912 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:45:21.912 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.546209+0000", 2024-08-19T14:45:21.913 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.1429999999999998e-06 2024-08-19T14:45:21.913 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.913 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.913 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:45:21.913 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.546305+0000", 2024-08-19T14:45:21.913 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.5778000000000004e-05 2024-08-19T14:45:21.913 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.913 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.913 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:45:21.913 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.546434+0000", 2024-08-19T14:45:21.913 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000129226 2024-08-19T14:45:21.913 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.913 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.913 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,2", 2024-08-19T14:45:21.914 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.546528+0000", 2024-08-19T14:45:21.914 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.3697999999999996e-05 2024-08-19T14:45:21.914 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.914 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.914 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:45:21.914 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.547471+0000", 2024-08-19T14:45:21.914 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00094326499999999999 2024-08-19T14:45:21.914 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.915 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.915 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:45:21.915 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.547640+0000", 2024-08-19T14:45:21.915 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00016882799999999999 2024-08-19T14:45:21.915 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.915 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.915 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:45:21.916 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.716680+0000", 2024-08-19T14:45:21.916 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.16904027299999999 2024-08-19T14:45:21.916 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.916 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.916 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:45:21.916 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.716731+0000", 2024-08-19T14:45:21.916 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0924999999999998e-05 2024-08-19T14:45:21.916 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.916 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.916 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:45:21.916 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.716768+0000", 2024-08-19T14:45:21.916 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.6984999999999999e-05 2024-08-19T14:45:21.917 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:45:21.917 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:45:21.917 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:45:21.917 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.917 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.917 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.27235.0:25540 9.7 9:ec2cf3a5:::benchmark_data_smithi045_152862_object25539:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e832)", 2024-08-19T14:45:21.917 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:44:31.553529+0000", 2024-08-19T14:45:21.917 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 50.250569239999997, 2024-08-19T14:45:21.917 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.163431888, 2024-08-19T14:45:21.917 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:45:21.917 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:45:21.917 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:45:21.917 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:45:21.917 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.27235", 2024-08-19T14:45:21.918 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2746512116", 2024-08-19T14:45:21.918 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 25540 2024-08-19T14:45:21.918 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.918 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:45:21.918 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.918 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:45:21.918 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.553529+0000", 2024-08-19T14:45:21.918 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:45:21.918 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.918 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.918 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:45:21.918 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.553529+0000", 2024-08-19T14:45:21.918 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:45:21.919 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.919 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.919 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:45:21.919 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.553530+0000", 2024-08-19T14:45:21.919 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.6700000000000002e-07 2024-08-19T14:45:21.919 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.919 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.919 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:45:21.919 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.553548+0000", 2024-08-19T14:45:21.919 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8573e-05 2024-08-19T14:45:21.919 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.919 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.919 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:45:21.919 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.553559+0000", 2024-08-19T14:45:21.920 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0997e-05 2024-08-19T14:45:21.920 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.920 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.920 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:45:21.920 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.553565+0000", 2024-08-19T14:45:21.920 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.5210000000000002e-06 2024-08-19T14:45:21.920 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.920 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.920 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:45:21.920 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.553615+0000", 2024-08-19T14:45:21.920 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0482999999999998e-05 2024-08-19T14:45:21.920 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.920 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.920 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:45:21.921 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.553704+0000", 2024-08-19T14:45:21.921 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.8782000000000005e-05 2024-08-19T14:45:21.921 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.921 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.921 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,2", 2024-08-19T14:45:21.921 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.553790+0000", 2024-08-19T14:45:21.921 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.6522999999999998e-05 2024-08-19T14:45:21.921 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.922 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.922 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:45:21.922 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.554874+0000", 2024-08-19T14:45:21.922 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0010836349999999999 2024-08-19T14:45:21.922 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.922 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.922 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:45:21.922 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.554973+0000", 2024-08-19T14:45:21.922 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.8756000000000001e-05 2024-08-19T14:45:21.922 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.922 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.922 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:45:21.922 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.716855+0000", 2024-08-19T14:45:21.923 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.161881836 2024-08-19T14:45:21.923 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.923 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.923 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:45:21.923 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.716917+0000", 2024-08-19T14:45:21.923 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.2514999999999994e-05 2024-08-19T14:45:21.923 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.923 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.923 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:45:21.924 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.716961+0000", 2024-08-19T14:45:21.924 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.3300000000000002e-05 2024-08-19T14:45:21.924 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:45:21.924 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:45:21.924 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:45:21.924 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.924 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.924 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.27235.0:25561 9.7 9:e1acbb18:::benchmark_data_smithi045_152862_object25560:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e832)", 2024-08-19T14:45:21.924 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:44:31.573926+0000", 2024-08-19T14:45:21.924 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 50.230171730999999, 2024-08-19T14:45:21.924 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.143582351, 2024-08-19T14:45:21.924 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:45:21.924 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:45:21.925 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:45:21.925 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:45:21.925 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.27235", 2024-08-19T14:45:21.925 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2746512116", 2024-08-19T14:45:21.925 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 25561 2024-08-19T14:45:21.925 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.925 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:45:21.925 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.925 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:45:21.925 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.573926+0000", 2024-08-19T14:45:21.925 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:45:21.925 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.926 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.926 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:45:21.926 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.573926+0000", 2024-08-19T14:45:21.926 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:45:21.926 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.926 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.926 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:45:21.926 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.573927+0000", 2024-08-19T14:45:21.926 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.0200000000000001e-07 2024-08-19T14:45:21.926 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.926 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.926 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:45:21.926 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.573946+0000", 2024-08-19T14:45:21.927 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9538000000000002e-05 2024-08-19T14:45:21.927 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.927 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.927 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:45:21.927 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.573957+0000", 2024-08-19T14:45:21.927 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0522e-05 2024-08-19T14:45:21.927 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.927 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.927 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:45:21.927 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.573962+0000", 2024-08-19T14:45:21.927 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.045e-06 2024-08-19T14:45:21.927 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.927 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.927 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:45:21.928 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.574004+0000", 2024-08-19T14:45:21.928 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.1451e-05 2024-08-19T14:45:21.928 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.928 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.928 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:45:21.928 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.574086+0000", 2024-08-19T14:45:21.928 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.2383e-05 2024-08-19T14:45:21.928 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.928 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.928 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,2", 2024-08-19T14:45:21.928 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.574169+0000", 2024-08-19T14:45:21.928 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.3214000000000007e-05 2024-08-19T14:45:21.928 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.928 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.928 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:45:21.929 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.575026+0000", 2024-08-19T14:45:21.929 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00085690500000000004 2024-08-19T14:45:21.929 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.929 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.929 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:45:21.929 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.575801+0000", 2024-08-19T14:45:21.929 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00077500699999999997 2024-08-19T14:45:21.929 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.929 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.929 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:45:21.929 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.717422+0000", 2024-08-19T14:45:21.929 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.14162117399999999 2024-08-19T14:45:21.929 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.930 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.930 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:45:21.930 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.717474+0000", 2024-08-19T14:45:21.930 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2207999999999999e-05 2024-08-19T14:45:21.930 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.930 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.930 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:45:21.930 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:31.717508+0000", 2024-08-19T14:45:21.930 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.4102e-05 2024-08-19T14:45:21.930 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:45:21.930 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:45:21.930 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:45:21.930 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.930 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.931 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.27235.0:31081 9.6 9.8e690cd6 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e834)", 2024-08-19T14:45:21.931 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:44:34.157100+0000", 2024-08-19T14:45:21.931 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 47.646998343999996, 2024-08-19T14:45:21.931 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.99146695600000001, 2024-08-19T14:45:21.931 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:45:21.931 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:45:21.931 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-08-19T14:45:21.931 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:45:21.931 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.27235", 2024-08-19T14:45:21.931 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2746512116", 2024-08-19T14:45:21.931 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 31081 2024-08-19T14:45:21.931 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.932 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:45:21.932 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.932 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:45:21.932 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:34.157100+0000", 2024-08-19T14:45:21.932 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:45:21.932 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.932 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.932 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:45:21.932 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:34.157100+0000", 2024-08-19T14:45:21.932 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:45:21.932 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.932 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.932 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:45:21.933 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:34.157100+0000", 2024-08-19T14:45:21.933 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.7399999999999998e-07 2024-08-19T14:45:21.933 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.933 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.933 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:45:21.933 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:34.157120+0000", 2024-08-19T14:45:21.933 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0188999999999999e-05 2024-08-19T14:45:21.933 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.933 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.933 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:45:21.933 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:34.157127+0000", 2024-08-19T14:45:21.933 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.1679999999999998e-06 2024-08-19T14:45:21.933 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.934 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.934 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:45:21.934 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:34.157133+0000", 2024-08-19T14:45:21.934 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.6620000000000002e-06 2024-08-19T14:45:21.934 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.934 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.934 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:45:21.934 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:34.157262+0000", 2024-08-19T14:45:21.934 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012860700000000001 2024-08-19T14:45:21.934 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.934 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.934 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for peered", 2024-08-19T14:45:21.934 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:34.157274+0000", 2024-08-19T14:45:21.935 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2536e-05 2024-08-19T14:45:21.935 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.935 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.935 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:45:21.935 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:35.148528+0000", 2024-08-19T14:45:21.935 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.99125351699999997 2024-08-19T14:45:21.935 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.935 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.935 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:45:21.935 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:35.148566+0000", 2024-08-19T14:45:21.935 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.8803000000000001e-05 2024-08-19T14:45:21.935 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:45:21.935 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:45:21.935 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:45:21.936 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.936 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.936 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.27235.0:31085 9.6 9.d2431d16 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e834)", 2024-08-19T14:45:21.936 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:44:34.160262+0000", 2024-08-19T14:45:21.936 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 47.643836215, 2024-08-19T14:45:21.936 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.98964577799999998, 2024-08-19T14:45:21.936 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:45:21.936 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:45:21.936 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-08-19T14:45:21.936 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:45:21.936 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.27235", 2024-08-19T14:45:21.936 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2746512116", 2024-08-19T14:45:21.936 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 31085 2024-08-19T14:45:21.937 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.937 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:45:21.937 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.937 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:45:21.937 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:34.160262+0000", 2024-08-19T14:45:21.937 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:45:21.937 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.937 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.937 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:45:21.937 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:34.160262+0000", 2024-08-19T14:45:21.937 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:45:21.937 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.937 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.937 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:45:21.938 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:34.160263+0000", 2024-08-19T14:45:21.938 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.482e-06 2024-08-19T14:45:21.938 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.938 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.938 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:45:21.938 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:34.160283+0000", 2024-08-19T14:45:21.938 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0228999999999999e-05 2024-08-19T14:45:21.938 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.938 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.938 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:45:21.938 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:34.160293+0000", 2024-08-19T14:45:21.938 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.4010000000000003e-06 2024-08-19T14:45:21.938 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.938 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.939 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:45:21.939 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:34.160298+0000", 2024-08-19T14:45:21.939 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.8409999999999999e-06 2024-08-19T14:45:21.939 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.939 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.939 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:45:21.939 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:34.160341+0000", 2024-08-19T14:45:21.939 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.3633000000000001e-05 2024-08-19T14:45:21.939 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.939 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.939 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for peered", 2024-08-19T14:45:21.939 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:34.160349+0000", 2024-08-19T14:45:21.939 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.3239999999999997e-06 2024-08-19T14:45:21.940 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.940 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.940 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:45:21.940 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:35.149891+0000", 2024-08-19T14:45:21.940 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.98954210200000003 2024-08-19T14:45:21.940 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.940 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.940 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:45:21.940 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:35.149907+0000", 2024-08-19T14:45:21.940 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6765999999999998e-05 2024-08-19T14:45:21.940 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:45:21.940 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:45:21.940 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:45:21.941 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.941 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.941 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.27235.0:39781 9.6 9:6221f1a7:::benchmark_data_smithi045_152862_object39780:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e840)", 2024-08-19T14:45:21.941 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:44:46.561716+0000", 2024-08-19T14:45:21.941 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 35.242381561999998, 2024-08-19T14:45:21.941 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.134856429, 2024-08-19T14:45:21.941 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:45:21.941 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:45:21.941 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:45:21.941 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:45:21.941 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.27235", 2024-08-19T14:45:21.941 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2746512116", 2024-08-19T14:45:21.941 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 39781 2024-08-19T14:45:21.941 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.942 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:45:21.942 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.942 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:45:21.942 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:46.561716+0000", 2024-08-19T14:45:21.942 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:45:21.942 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.942 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.942 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:45:21.942 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:46.561716+0000", 2024-08-19T14:45:21.942 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:45:21.942 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.942 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.942 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:45:21.943 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:46.561718+0000", 2024-08-19T14:45:21.943 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5039999999999999e-06 2024-08-19T14:45:21.943 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.943 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.943 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:45:21.943 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:46.561751+0000", 2024-08-19T14:45:21.943 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.3541999999999998e-05 2024-08-19T14:45:21.943 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.943 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.943 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:45:21.943 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:46.561770+0000", 2024-08-19T14:45:21.943 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8570999999999999e-05 2024-08-19T14:45:21.943 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.943 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.944 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:45:21.944 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:46.561779+0000", 2024-08-19T14:45:21.944 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.0310000000000004e-06 2024-08-19T14:45:21.944 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.944 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.944 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:45:21.944 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:46.562236+0000", 2024-08-19T14:45:21.944 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00045689199999999998 2024-08-19T14:45:21.944 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.944 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.944 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:45:21.944 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:46.562410+0000", 2024-08-19T14:45:21.944 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000174198 2024-08-19T14:45:21.945 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.945 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.945 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,9", 2024-08-19T14:45:21.945 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:46.562588+0000", 2024-08-19T14:45:21.945 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00017804 2024-08-19T14:45:21.945 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.945 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.945 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:45:21.945 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:46.567020+0000", 2024-08-19T14:45:21.945 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0044319149999999998 2024-08-19T14:45:21.945 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.945 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.945 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:45:21.946 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:46.569062+0000", 2024-08-19T14:45:21.946 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0020420299999999998 2024-08-19T14:45:21.946 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.946 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.946 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:45:21.946 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:46.695046+0000", 2024-08-19T14:45:21.946 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.12598373700000001 2024-08-19T14:45:21.947 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.947 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.947 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:45:21.947 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:46.695107+0000", 2024-08-19T14:45:21.947 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.1311999999999998e-05 2024-08-19T14:45:21.947 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.947 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.947 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:45:21.947 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:46.696573+0000", 2024-08-19T14:45:21.948 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001465657 2024-08-19T14:45:21.948 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:45:21.948 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:45:21.948 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:45:21.949 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.949 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.949 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.27235.0:39782 9.6 9:69b39bcd:::benchmark_data_smithi045_152862_object39781:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e840)", 2024-08-19T14:45:21.949 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:44:46.562221+0000", 2024-08-19T14:45:21.949 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 35.241876859999998, 2024-08-19T14:45:21.949 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.137703459, 2024-08-19T14:45:21.949 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:45:21.949 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:45:21.949 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:45:21.949 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:45:21.949 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.27235", 2024-08-19T14:45:21.949 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2746512116", 2024-08-19T14:45:21.949 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 39782 2024-08-19T14:45:21.949 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.950 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:45:21.950 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.950 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:45:21.950 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:46.562221+0000", 2024-08-19T14:45:21.950 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:45:21.950 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.950 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.950 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:45:21.950 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:46.562221+0000", 2024-08-19T14:45:21.950 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:45:21.950 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.950 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.950 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:45:21.951 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:46.562222+0000", 2024-08-19T14:45:21.951 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.9700000000000005e-07 2024-08-19T14:45:21.951 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.951 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.951 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:45:21.951 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:46.562243+0000", 2024-08-19T14:45:21.951 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1209e-05 2024-08-19T14:45:21.951 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.951 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.951 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:45:21.951 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:46.562252+0000", 2024-08-19T14:45:21.951 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.8950000000000004e-06 2024-08-19T14:45:21.951 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.951 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.952 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:45:21.952 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:46.562259+0000", 2024-08-19T14:45:21.952 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.9140000000000002e-06 2024-08-19T14:45:21.952 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.952 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.952 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:45:21.952 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:46.564066+0000", 2024-08-19T14:45:21.952 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001806658 2024-08-19T14:45:21.952 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.952 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.952 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:45:21.952 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:46.564433+0000", 2024-08-19T14:45:21.952 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00036744600000000001 2024-08-19T14:45:21.953 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.953 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.953 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,9", 2024-08-19T14:45:21.953 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:46.564831+0000", 2024-08-19T14:45:21.953 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00039782000000000001 2024-08-19T14:45:21.953 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.953 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.953 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:45:21.953 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:46.569137+0000", 2024-08-19T14:45:21.953 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0043059930000000001 2024-08-19T14:45:21.953 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.953 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.953 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:45:21.954 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:46.569244+0000", 2024-08-19T14:45:21.954 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000107561 2024-08-19T14:45:21.954 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.954 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.954 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:45:21.954 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:46.696660+0000", 2024-08-19T14:45:21.954 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.127415422 2024-08-19T14:45:21.954 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.954 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.955 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:45:21.955 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:46.696983+0000", 2024-08-19T14:45:21.955 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00032312999999999998 2024-08-19T14:45:21.955 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.955 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.955 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:45:21.955 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:46.699924+0000", 2024-08-19T14:45:21.955 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0029415140000000001 2024-08-19T14:45:21.955 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:45:21.955 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:45:21.956 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:45:21.956 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.956 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.956 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.27235.0:49914 9.6 9:6f2c8487:::benchmark_data_smithi045_152862_object49913:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e847)", 2024-08-19T14:45:21.956 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:45:00.499787+0000", 2024-08-19T14:45:21.956 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 21.304310937, 2024-08-19T14:45:21.956 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.15710339300000001, 2024-08-19T14:45:21.956 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:45:21.956 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:45:21.956 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:45:21.956 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:45:21.956 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.27235", 2024-08-19T14:45:21.956 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2746512116", 2024-08-19T14:45:21.956 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 49914 2024-08-19T14:45:21.957 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.957 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:45:21.957 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.957 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:45:21.957 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:00.499787+0000", 2024-08-19T14:45:21.957 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:45:21.957 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.957 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.957 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:45:21.957 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:00.499787+0000", 2024-08-19T14:45:21.957 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:45:21.957 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.957 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.957 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:45:21.958 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:00.499789+0000", 2024-08-19T14:45:21.958 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.637e-06 2024-08-19T14:45:21.958 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.958 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.958 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:45:21.958 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:00.499814+0000", 2024-08-19T14:45:21.958 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.5063e-05 2024-08-19T14:45:21.958 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.958 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.958 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:45:21.958 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:00.499827+0000", 2024-08-19T14:45:21.958 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3179999999999999e-05 2024-08-19T14:45:21.958 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.958 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.959 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:45:21.959 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:00.499834+0000", 2024-08-19T14:45:21.959 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.6520000000000006e-06 2024-08-19T14:45:21.959 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.959 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.959 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:45:21.959 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:00.499890+0000", 2024-08-19T14:45:21.959 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.5497000000000001e-05 2024-08-19T14:45:21.959 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.959 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.959 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:45:21.959 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:00.500006+0000", 2024-08-19T14:45:21.959 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000115575 2024-08-19T14:45:21.959 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.960 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.960 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,9", 2024-08-19T14:45:21.960 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:00.500110+0000", 2024-08-19T14:45:21.960 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000104326 2024-08-19T14:45:21.960 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.960 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.960 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:45:21.960 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:00.501985+0000", 2024-08-19T14:45:21.960 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0018747309999999999 2024-08-19T14:45:21.960 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.960 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.960 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:45:21.960 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:00.502121+0000", 2024-08-19T14:45:21.960 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000136427 2024-08-19T14:45:21.961 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.961 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.961 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:45:21.961 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:00.655587+0000", 2024-08-19T14:45:21.961 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.153465664 2024-08-19T14:45:21.961 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.961 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.961 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:45:21.961 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:00.656002+0000", 2024-08-19T14:45:21.961 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00041564100000000001 2024-08-19T14:45:21.961 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:45:21.961 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:45:21.961 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:45:21.961 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:00.656890+0000", 2024-08-19T14:45:21.962 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00088800000000000001 2024-08-19T14:45:21.962 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:45:21.962 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:45:21.962 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:45:21.962 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:45:21.962 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:45:21.962 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:45:21.962 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:45:21.963 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-19T14:45:21.992 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:21.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:21.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:45:22.094 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:45:22.338 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 61 16 70515 70499 72.2195 60.25 0.00316654 0.0138274 2024-08-19T14:45:22.338 INFO:teuthology.orchestra.run.smithi031.stderr:marked in osd.6. 2024-08-19T14:45:22.360 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:22.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:22.361 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:45:22.462 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:45:22.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:22.484 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:22.485 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:45:22.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:22 smithi032 ceph-mon[20004]: pgmap v2024: 47 pgs: 47 active+clean; 4.0 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-08-19T14:45:22.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:22 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3487856029' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2024-08-19T14:45:22.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:22 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2024-08-19T14:45:22.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:22 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:45:22.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:22 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:45:22.527 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:22 smithi031 ceph-mon[16336]: pgmap v2024: 47 pgs: 47 active+clean; 4.0 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-08-19T14:45:22.527 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:22 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3487856029' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2024-08-19T14:45:22.527 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:22 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2024-08-19T14:45:22.527 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:22 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:45:22.527 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:22 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:45:22.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:22 smithi045 ceph-mon[20143]: pgmap v2024: 47 pgs: 47 active+clean; 4.0 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-08-19T14:45:22.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:22 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3487856029' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2024-08-19T14:45:22.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:22 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2024-08-19T14:45:22.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:22 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:45:22.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:22 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:45:22.585 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:45:22.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:22.608 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:22.609 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:45:22.711 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:45:22.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:22.737 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:22.737 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:45:22.787 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-08-19T14:45:22.838 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:45:22.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:22.863 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:22.864 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:45:22.964 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:45:22.984 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:45:22.986 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:22.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:22.987 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:45:23.016 INFO:tasks.thrashosds.thrasher:Added osd 6 2024-08-19T14:45:23.088 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:45:23.113 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:23.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:23.114 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:45:23.215 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:45:23.240 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:23.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:23.241 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:45:23.319 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 62 16 70515 70499 71.0548 0 - 0.0138274 2024-08-19T14:45:23.341 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:45:23.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:23.364 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:23.364 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:45:23.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:23 smithi031 ceph-mon[16336]: 9.c scrub starts 2024-08-19T14:45:23.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:23 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["6"]}]': finished 2024-08-19T14:45:23.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:23 smithi031 ceph-mon[16336]: osdmap e862: 12 total, 12 up, 8 in 2024-08-19T14:45:23.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:23 smithi031 ceph-mon[16336]: osdmap e863: 12 total, 12 up, 8 in 2024-08-19T14:45:23.464 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:45:23.487 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:23.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:23.488 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:45:23.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:23 smithi032 ceph-mon[20004]: 9.c scrub starts 2024-08-19T14:45:23.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:23 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["6"]}]': finished 2024-08-19T14:45:23.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:23 smithi032 ceph-mon[20004]: osdmap e862: 12 total, 12 up, 8 in 2024-08-19T14:45:23.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:23 smithi032 ceph-mon[20004]: osdmap e863: 12 total, 12 up, 8 in 2024-08-19T14:45:23.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:23 smithi045 ceph-mon[20143]: 9.c scrub starts 2024-08-19T14:45:23.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:23 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["6"]}]': finished 2024-08-19T14:45:23.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:23 smithi045 ceph-mon[20143]: osdmap e862: 12 total, 12 up, 8 in 2024-08-19T14:45:23.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:23 smithi045 ceph-mon[20143]: osdmap e863: 12 total, 12 up, 8 in 2024-08-19T14:45:23.588 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:45:23.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:23.616 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:23.616 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:45:23.717 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:45:23.741 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:23.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:23.742 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:45:23.843 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:45:23.869 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:23.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:23.870 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:45:23.971 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:45:23.996 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:23.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:23.997 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:45:24.061 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:45:24.099 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:45:24.116 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:45:24.126 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:24.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:24.127 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:45:24.228 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:45:24.250 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:24.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:24.251 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:45:24.319 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 63 16 70515 70499 69.9269 0 - 0.0138274 2024-08-19T14:45:24.352 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:45:24.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:24.374 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:24.375 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:45:24.476 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:45:24.498 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:24.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:24.499 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:45:24.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:24 smithi032 ceph-mon[20004]: pgmap v2026: 47 pgs: 47 active+clean; 4.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 74 MiB/s wr, 1.19k op/s 2024-08-19T14:45:24.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:24 smithi032 ceph-mon[20004]: 9.c scrub ok 2024-08-19T14:45:24.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:24 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:45:24.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:24 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:45:24.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:24 smithi032 ceph-mon[20004]: osdmap e864: 12 total, 12 up, 8 in 2024-08-19T14:45:24.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:24 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3446090653' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:45:24.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:24 smithi045 ceph-mon[20143]: pgmap v2026: 47 pgs: 47 active+clean; 4.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 74 MiB/s wr, 1.19k op/s 2024-08-19T14:45:24.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:24 smithi045 ceph-mon[20143]: 9.c scrub ok 2024-08-19T14:45:24.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:24 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:45:24.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:24 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:45:24.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:24 smithi045 ceph-mon[20143]: osdmap e864: 12 total, 12 up, 8 in 2024-08-19T14:45:24.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:24 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3446090653' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:45:24.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:24 smithi031 ceph-mon[16336]: pgmap v2026: 47 pgs: 47 active+clean; 4.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 74 MiB/s wr, 1.19k op/s 2024-08-19T14:45:24.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:24 smithi031 ceph-mon[16336]: 9.c scrub ok 2024-08-19T14:45:24.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:24 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:45:24.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:24 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:45:24.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:24 smithi031 ceph-mon[16336]: osdmap e864: 12 total, 12 up, 8 in 2024-08-19T14:45:24.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:24 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3446090653' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:45:24.600 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:45:24.634 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:24.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:24.635 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:45:24.713 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:45:24.713 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [ 2024-08-19T14:45:24.713 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:45:24.713 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.27235.0:70661 9.1 9:8b3b90de:::benchmark_data_smithi045_152862_object70660:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e864)", 2024-08-19T14:45:24.713 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:45:24.695007+0000", 2024-08-19T14:45:24.713 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.015584201000000001, 2024-08-19T14:45:24.713 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.015618699999999999, 2024-08-19T14:45:24.713 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:45:24.714 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:45:24.714 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "waiting for sub ops", 2024-08-19T14:45:24.714 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:45:24.714 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.27235", 2024-08-19T14:45:24.714 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2746512116", 2024-08-19T14:45:24.714 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 70661 2024-08-19T14:45:24.714 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:45:24.714 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:45:24.714 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:45:24.714 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:45:24.714 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.695007+0000", 2024-08-19T14:45:24.714 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:45:24.714 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:45:24.715 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:45:24.715 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:45:24.715 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.695007+0000", 2024-08-19T14:45:24.715 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:45:24.715 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:45:24.715 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:45:24.715 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:45:24.715 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.695009+0000", 2024-08-19T14:45:24.715 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.64e-06 2024-08-19T14:45:24.715 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:45:24.715 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:45:24.715 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:45:24.715 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.695071+0000", 2024-08-19T14:45:24.716 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.2025000000000003e-05 2024-08-19T14:45:24.716 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:45:24.716 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:45:24.716 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:45:24.716 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.695082+0000", 2024-08-19T14:45:24.716 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1253999999999999e-05 2024-08-19T14:45:24.716 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:45:24.716 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:45:24.716 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:45:24.716 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.695088+0000", 2024-08-19T14:45:24.716 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.0440000000000001e-06 2024-08-19T14:45:24.716 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:45:24.716 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:45:24.717 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:45:24.717 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.700272+0000", 2024-08-19T14:45:24.717 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0051838489999999999 2024-08-19T14:45:24.717 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:45:24.717 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:45:24.717 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:45:24.717 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.700406+0000", 2024-08-19T14:45:24.717 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013363200000000001 2024-08-19T14:45:24.717 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:45:24.717 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:45:24.717 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 6,9,10", 2024-08-19T14:45:24.717 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.700566+0000", 2024-08-19T14:45:24.717 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000160464 2024-08-19T14:45:24.718 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:45:24.718 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:45:24.718 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:45:24.718 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:45:24.718 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:45:24.718 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.27235.0:70662 9.1 9:86a856b5:::benchmark_data_smithi045_152862_object70661:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e864)", 2024-08-19T14:45:24.718 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:45:24.697448+0000", 2024-08-19T14:45:24.718 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.013143513000000001, 2024-08-19T14:45:24.718 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.01327475, 2024-08-19T14:45:24.718 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:45:24.718 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:45:24.718 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "waiting for sub ops", 2024-08-19T14:45:24.719 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:45:24.719 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.27235", 2024-08-19T14:45:24.719 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2746512116", 2024-08-19T14:45:24.719 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 70662 2024-08-19T14:45:24.719 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:45:24.719 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:45:24.719 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:45:24.719 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:45:24.719 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.697448+0000", 2024-08-19T14:45:24.719 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:45:24.719 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:45:24.719 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:45:24.719 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:45:24.720 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.697448+0000", 2024-08-19T14:45:24.720 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:45:24.720 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:45:24.720 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:45:24.720 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:45:24.720 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.697450+0000", 2024-08-19T14:45:24.720 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.34e-06 2024-08-19T14:45:24.720 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:45:24.720 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:45:24.720 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:45:24.720 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.697482+0000", 2024-08-19T14:45:24.720 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.1488000000000001e-05 2024-08-19T14:45:24.721 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:45:24.721 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:45:24.721 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:45:24.721 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.697503+0000", 2024-08-19T14:45:24.721 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.0937999999999998e-05 2024-08-19T14:45:24.721 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:45:24.721 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:45:24.721 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:45:24.721 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.697514+0000", 2024-08-19T14:45:24.721 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0903e-05 2024-08-19T14:45:24.721 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:45:24.721 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:45:24.721 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:45:24.721 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.702880+0000", 2024-08-19T14:45:24.722 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0053668609999999997 2024-08-19T14:45:24.722 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:45:24.722 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:45:24.722 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:45:24.722 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.703011+0000", 2024-08-19T14:45:24.722 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013098100000000001 2024-08-19T14:45:24.722 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:45:24.722 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:45:24.722 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 6,9,10", 2024-08-19T14:45:24.722 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.703160+0000", 2024-08-19T14:45:24.722 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000148317 2024-08-19T14:45:24.722 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:45:24.722 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:45:24.723 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:45:24.723 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:45:24.723 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:45:24.723 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop_reply(client.27235.0:70661 9.1 e864/863)", 2024-08-19T14:45:24.723 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:45:24.705916+0000", 2024-08-19T14:45:24.723 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.0046752950000000003, 2024-08-19T14:45:24.723 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0048784040000000002, 2024-08-19T14:45:24.723 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:45:24.723 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:45:24.723 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "queued for pg", 2024-08-19T14:45:24.723 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:45:24.723 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:45:24.723 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:45:24.723 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.705916+0000", 2024-08-19T14:45:24.724 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:45:24.724 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:45:24.724 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:45:24.724 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:45:24.724 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.705916+0000", 2024-08-19T14:45:24.724 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:45:24.724 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:45:24.724 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:45:24.724 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:45:24.724 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.705917+0000", 2024-08-19T14:45:24.724 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.246e-06 2024-08-19T14:45:24.724 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:45:24.724 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:45:24.725 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:45:24.725 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.705923+0000", 2024-08-19T14:45:24.725 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.818e-06 2024-08-19T14:45:24.725 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:45:24.725 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:45:24.725 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:45:24.725 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.705946+0000", 2024-08-19T14:45:24.725 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.3342000000000001e-05 2024-08-19T14:45:24.725 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:45:24.725 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:45:24.725 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:45:24.725 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.705956+0000", 2024-08-19T14:45:24.725 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.4579999999999999e-06 2024-08-19T14:45:24.725 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:45:24.726 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:45:24.726 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:45:24.726 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:45:24.726 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:45:24.726 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop_reply(client.27235.0:70662 9.1 e864/863)", 2024-08-19T14:45:24.726 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:45:24.706058+0000", 2024-08-19T14:45:24.726 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.0045331759999999999, 2024-08-19T14:45:24.726 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0047788550000000003, 2024-08-19T14:45:24.726 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:45:24.726 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:45:24.726 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "queued for pg", 2024-08-19T14:45:24.726 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:45:24.726 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:45:24.726 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:45:24.727 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.706058+0000", 2024-08-19T14:45:24.727 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:45:24.727 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:45:24.727 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:45:24.727 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:45:24.727 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.706058+0000", 2024-08-19T14:45:24.727 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:45:24.727 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:45:24.727 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:45:24.727 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:45:24.727 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.706059+0000", 2024-08-19T14:45:24.727 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.5199999999999996e-07 2024-08-19T14:45:24.727 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:45:24.727 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:45:24.728 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:45:24.728 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.706063+0000", 2024-08-19T14:45:24.728 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.3159999999999998e-06 2024-08-19T14:45:24.728 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:45:24.728 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:45:24.728 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:45:24.728 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.706075+0000", 2024-08-19T14:45:24.728 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1474999999999999e-05 2024-08-19T14:45:24.728 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:45:24.728 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:45:24.728 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:45:24.729 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.706081+0000", 2024-08-19T14:45:24.729 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.1319999999999999e-06 2024-08-19T14:45:24.729 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:45:24.729 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:45:24.729 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:45:24.729 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:45:24.729 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:45:24.729 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop_reply(client.27235.0:70661 9.1 e864/863)", 2024-08-19T14:45:24.729 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:45:24.708571+0000", 2024-08-19T14:45:24.729 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.0020198769999999998, 2024-08-19T14:45:24.729 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0023188060000000001, 2024-08-19T14:45:24.729 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:45:24.729 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:45:24.729 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "queued for pg", 2024-08-19T14:45:24.730 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:45:24.730 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:45:24.730 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:45:24.730 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.708571+0000", 2024-08-19T14:45:24.730 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:45:24.730 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:45:24.730 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:45:24.730 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:45:24.730 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.708571+0000", 2024-08-19T14:45:24.730 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:45:24.730 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:45:24.730 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:45:24.730 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:45:24.731 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.708573+0000", 2024-08-19T14:45:24.731 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2550000000000001e-06 2024-08-19T14:45:24.731 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:45:24.731 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:45:24.731 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:45:24.731 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.708583+0000", 2024-08-19T14:45:24.731 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.7729999999999994e-06 2024-08-19T14:45:24.731 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:45:24.731 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:45:24.731 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:45:24.731 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.708601+0000", 2024-08-19T14:45:24.731 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.8490000000000001e-05 2024-08-19T14:45:24.731 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:45:24.731 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:45:24.732 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:45:24.732 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.708611+0000", 2024-08-19T14:45:24.732 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.8190000000000003e-06 2024-08-19T14:45:24.732 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:45:24.732 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:45:24.732 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:45:24.732 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:45:24.732 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:45:24.732 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop_reply(client.27235.0:70662 9.1 e864/863)", 2024-08-19T14:45:24.732 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:45:24.708628+0000", 2024-08-19T14:45:24.732 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.0019629790000000001, 2024-08-19T14:45:24.732 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0023063060000000002, 2024-08-19T14:45:24.732 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:45:24.732 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:45:24.733 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "queued for pg", 2024-08-19T14:45:24.733 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:45:24.733 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:45:24.733 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:45:24.733 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.708628+0000", 2024-08-19T14:45:24.733 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:45:24.733 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:45:24.733 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:45:24.733 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:45:24.733 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.708628+0000", 2024-08-19T14:45:24.733 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:45:24.733 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:45:24.733 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:45:24.733 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:45:24.734 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.708629+0000", 2024-08-19T14:45:24.734 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.9999999999999999e-07 2024-08-19T14:45:24.734 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:45:24.734 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:45:24.734 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:45:24.734 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.708632+0000", 2024-08-19T14:45:24.734 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.8660000000000002e-06 2024-08-19T14:45:24.734 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:45:24.734 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:45:24.734 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:45:24.734 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.708638+0000", 2024-08-19T14:45:24.734 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.4420000000000003e-06 2024-08-19T14:45:24.734 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:45:24.734 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:45:24.735 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:45:24.735 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.708643+0000", 2024-08-19T14:45:24.735 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.6070000000000001e-06 2024-08-19T14:45:24.735 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:45:24.735 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:45:24.735 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:45:24.735 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:45:24.735 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:45:24.735 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.27235.0:70673 9.1 9.589412b1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e864)", 2024-08-19T14:45:24.735 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:45:24.710199+0000", 2024-08-19T14:45:24.735 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.00039282100000000001, 2024-08-19T14:45:24.735 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00077914599999999998, 2024-08-19T14:45:24.735 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:45:24.736 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:45:24.736 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "queued for pg", 2024-08-19T14:45:24.736 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:45:24.736 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.27235", 2024-08-19T14:45:24.736 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2746512116", 2024-08-19T14:45:24.736 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 70673 2024-08-19T14:45:24.736 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:45:24.736 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:45:24.736 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:45:24.736 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:45:24.736 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.710199+0000", 2024-08-19T14:45:24.736 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:45:24.736 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:45:24.736 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:45:24.737 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:45:24.737 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.710199+0000", 2024-08-19T14:45:24.737 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:45:24.737 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:45:24.737 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:45:24.737 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:45:24.737 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.710200+0000", 2024-08-19T14:45:24.737 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.9080000000000002e-06 2024-08-19T14:45:24.737 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:45:24.737 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:45:24.737 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:45:24.737 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.710279+0000", 2024-08-19T14:45:24.737 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.8617999999999998e-05 2024-08-19T14:45:24.737 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:45:24.738 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:45:24.738 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:45:24.738 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.710298+0000", 2024-08-19T14:45:24.738 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.9123999999999999e-05 2024-08-19T14:45:24.738 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:45:24.738 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:45:24.738 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:45:24.738 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.710308+0000", 2024-08-19T14:45:24.738 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.7089999999999997e-06 2024-08-19T14:45:24.738 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:45:24.738 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:45:24.738 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:45:24.738 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:45:24.738 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-08-19T14:45:24.739 INFO:teuthology.orchestra.run.smithi031.stdout: "num_ops": 7 2024-08-19T14:45:24.739 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:45:24.739 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:45:24.740 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:45:24.770 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:24.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:24.771 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:45:24.872 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:45:24.895 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:24.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:24.896 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:45:24.997 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:45:25.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:25.032 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:25.032 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:45:25.091 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is set 2024-08-19T14:45:25.133 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:45:25.157 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:25.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:25.158 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:45:25.259 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:45:25.290 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:25.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:25.291 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:45:25.319 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 64 16 71298 71282 69.599 16.3125 0.00729373 0.0143621 2024-08-19T14:45:25.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:25 smithi031 ceph-mon[16336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:45:25.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:25 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3446090653' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:45:25.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:25 smithi031 ceph-mon[16336]: osdmap e865: 12 total, 12 up, 8 in 2024-08-19T14:45:25.392 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:45:25.415 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:25.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:25.416 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:45:25.517 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:45:25.542 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:25.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:25.544 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:45:25.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:25 smithi045 ceph-mon[20143]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:45:25.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:25 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3446090653' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:45:25.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:25 smithi045 ceph-mon[20143]: osdmap e865: 12 total, 12 up, 8 in 2024-08-19T14:45:25.645 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:45:25.675 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:25.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:25.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:45:25.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:25 smithi032 ceph-mon[20004]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:45:25.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:25 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3446090653' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:45:25.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:25 smithi032 ceph-mon[20004]: osdmap e865: 12 total, 12 up, 8 in 2024-08-19T14:45:25.777 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:45:25.802 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:25.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:25.803 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:45:25.904 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:45:25.929 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:25.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:25.930 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:45:26.031 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:45:26.063 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:26.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:26.064 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:45:26.165 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:45:26.189 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:26.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:26.190 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:45:26.290 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:45:26.318 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:26.319 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 65 16 72343 72327 69.5329 65.3125 0.00358561 0.0143734 2024-08-19T14:45:26.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:26.320 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:45:26.421 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:45:26.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:26.459 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:26.459 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:45:26.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:26 smithi045 ceph-mon[20143]: pgmap v2029: 47 pgs: 7 remapped+peering, 2 peering, 38 active+clean; 4.2 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 62 MiB/s wr, 994 op/s 2024-08-19T14:45:26.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:26 smithi045 ceph-mon[20143]: osdmap e866: 12 total, 12 up, 8 in 2024-08-19T14:45:26.560 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:45:26.592 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:26.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:26.593 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:45:26.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:26 smithi031 ceph-mon[16336]: pgmap v2029: 47 pgs: 7 remapped+peering, 2 peering, 38 active+clean; 4.2 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 62 MiB/s wr, 994 op/s 2024-08-19T14:45:26.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:26 smithi031 ceph-mon[16336]: osdmap e866: 12 total, 12 up, 8 in 2024-08-19T14:45:26.694 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:45:26.727 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:26.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:26.728 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:45:26.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:26 smithi032 ceph-mon[20004]: pgmap v2029: 47 pgs: 7 remapped+peering, 2 peering, 38 active+clean; 4.2 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 62 MiB/s wr, 994 op/s 2024-08-19T14:45:26.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:26 smithi032 ceph-mon[20004]: osdmap e866: 12 total, 12 up, 8 in 2024-08-19T14:45:26.829 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:45:26.852 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:26.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:26.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:45:26.954 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:45:26.981 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:26.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:26.983 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:45:27.084 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:45:27.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:27.106 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:27.106 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:45:27.207 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:45:27.230 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:27.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:27.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:45:27.319 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 66 16 73210 73194 69.3004 54.1875 0.0175389 0.0144244 2024-08-19T14:45:27.333 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:45:27.344 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-08-19T14:45:27.372 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:27.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:27.373 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:45:27.474 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:45:27.501 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:27.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:27.502 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:45:27.603 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:45:27.636 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:27.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:27.638 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:45:27.739 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:45:27.774 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:27.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:27.775 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:45:27.876 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:45:27.898 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:27.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:27.899 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:45:27.929 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-19T14:45:28.000 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:45:28.017 INFO:tasks.thrashosds.thrasher:in_osds: [1, 4, 8, 2, 11, 9, 10, 6] out_osds: [3, 0, 7, 5] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 8, 9, 11, 10, 2] 2024-08-19T14:45:28.017 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-19T14:45:28.017 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-08-19T14:45:28.017 INFO:tasks.thrashosds.thrasher:Setting osd 6 primary_affinity to 1.000000 2024-08-19T14:45:28.017 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd primary-affinity 6 1 2024-08-19T14:45:28.031 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:28.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:28.032 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:45:28.133 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:45:28.154 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:28.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:28.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:45:28.256 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:45:28.266 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:28 smithi031 ceph-mon[16336]: pgmap v2032: 47 pgs: 2 active+recovering+undersized+remapped, 1 active+recovering+remapped, 7 remapped+peering, 5 peering, 32 active+clean; 4.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 58 MiB/s wr, 921 op/s; 13246/212931 objects misplaced (6.221%); 1.5 MiB/s, 24 objects/s recovering 2024-08-19T14:45:28.282 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:28.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:28.283 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:45:28.319 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 67 16 74041 74025 69.0412 51.9375 0.0123521 0.0144778 2024-08-19T14:45:28.384 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:45:28.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:28.417 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:28.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:45:28.518 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:45:28.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:28 smithi045 ceph-mon[20143]: pgmap v2032: 47 pgs: 2 active+recovering+undersized+remapped, 1 active+recovering+remapped, 7 remapped+peering, 5 peering, 32 active+clean; 4.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 58 MiB/s wr, 921 op/s; 13246/212931 objects misplaced (6.221%); 1.5 MiB/s, 24 objects/s recovering 2024-08-19T14:45:28.552 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:28.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:28.553 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:45:28.654 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:45:28.677 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:28.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:28.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:45:28.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:28 smithi032 ceph-mon[20004]: pgmap v2032: 47 pgs: 2 active+recovering+undersized+remapped, 1 active+recovering+remapped, 7 remapped+peering, 5 peering, 32 active+clean; 4.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 58 MiB/s wr, 921 op/s; 13246/212931 objects misplaced (6.221%); 1.5 MiB/s, 24 objects/s recovering 2024-08-19T14:45:28.780 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:45:28.805 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:28.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:28.806 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:45:28.907 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:45:28.931 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:28.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:28.932 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:45:29.033 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:45:29.057 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:29.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:29.058 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:45:29.159 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:45:29.200 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:29.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:29.201 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:45:29.302 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:45:29.322 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 68 16 74851 74835 68.7703 50.625 0.0170198 0.0145355 2024-08-19T14:45:29.328 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:29.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:29.329 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:45:29.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:29 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/646373284' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:45:29.430 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:45:29.453 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:29.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:29.454 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:45:29.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:29 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/646373284' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:45:29.555 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:45:29.579 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:29.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:29.580 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:45:29.681 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:45:29.704 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:29.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:29.705 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:45:29.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:29 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/646373284' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:45:29.807 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:45:29.815 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:45:29.815 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:45:29.815 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:45:29.816 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:45:29.816 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:45:29.816 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:45:29.816 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:45:29.816 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:45:29.816 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:45:29.816 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:45:29.816 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:45:29.817 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:45:29.817 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:45:29.817 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:45:29.817 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:45:29.817 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:45:29.817 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:45:29.817 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:45:29.818 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:45:29.818 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:45:29.818 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:45:29.818 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:45:29.818 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:45:29.818 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:45:29.832 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:29.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:29.833 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:45:29.934 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:45:29.959 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:29.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:29.960 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:45:30.061 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:45:30.094 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:30.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:30.095 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:45:30.196 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:45:30.225 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:30.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:30.226 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:45:30.319 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 69 16 75822 75806 68.6531 60.6875 0.0169106 0.0145606 2024-08-19T14:45:30.327 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:45:30.349 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:30.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:30.350 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:45:30.451 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:45:30.477 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:30.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:30.478 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:45:30.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:30 smithi045 ceph-mon[20143]: pgmap v2033: 47 pgs: 2 active+recovering+undersized+remapped, 1 active+recovering+remapped, 7 remapped+peering, 3 peering, 34 active+clean; 4.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 40 MiB/s wr, 637 op/s; 13246/213921 objects misplaced (6.192%); 1.1 MiB/s, 18 objects/s recovering 2024-08-19T14:45:30.578 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:45:30.611 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:30.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:30.612 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:45:30.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:30 smithi031 ceph-mon[16336]: pgmap v2033: 47 pgs: 2 active+recovering+undersized+remapped, 1 active+recovering+remapped, 7 remapped+peering, 3 peering, 34 active+clean; 4.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 40 MiB/s wr, 637 op/s; 13246/213921 objects misplaced (6.192%); 1.1 MiB/s, 18 objects/s recovering 2024-08-19T14:45:30.712 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:45:30.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:30.751 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:30.751 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:45:30.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:30 smithi032 ceph-mon[20004]: pgmap v2033: 47 pgs: 2 active+recovering+undersized+remapped, 1 active+recovering+remapped, 7 remapped+peering, 3 peering, 34 active+clean; 4.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 40 MiB/s wr, 637 op/s; 13246/213921 objects misplaced (6.192%); 1.1 MiB/s, 18 objects/s recovering 2024-08-19T14:45:30.852 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:45:30.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:30.889 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:30.889 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:45:30.990 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:45:31.024 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:31.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:31.025 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:45:31.126 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:45:31.165 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:31.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:31.166 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:45:31.266 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:45:31.296 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:31.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:31.297 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:45:31.319 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 70 16 76790 76774 68.5366 60.5 0.010116 0.0145843 2024-08-19T14:45:31.398 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:45:31.436 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:31.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:31.437 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:45:31.538 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:45:31.575 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:31.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:31.576 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:45:31.677 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:45:31.699 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:31.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:31.701 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:45:31.729 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:45:31.801 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:45:31.826 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:31.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:31.827 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:45:31.928 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:45:31.952 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:31.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:31.953 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:45:32.054 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:45:32.079 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:32.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:32.081 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:45:32.182 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:45:32.210 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:32.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:32.211 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:45:32.312 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:45:32.319 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 71 15 77763 77748 68.4286 60.875 0.00427797 0.0146084 2024-08-19T14:45:32.335 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:32.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:32.336 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:45:32.437 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:45:32.471 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:32.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:32.472 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:45:32.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:32 smithi045 ceph-mon[20143]: pgmap v2034: 47 pgs: 4 active+recovering+undersized+remapped, 1 active+recovering+remapped, 3 peering, 39 active+clean; 4.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 36 MiB/s wr, 577 op/s; 21760/219399 objects misplaced (9.918%); 3.9 MiB/s, 61 objects/s recovering 2024-08-19T14:45:32.573 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:45:32.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:32.610 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:32.610 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:45:32.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:32 smithi031 ceph-mon[16336]: pgmap v2034: 47 pgs: 4 active+recovering+undersized+remapped, 1 active+recovering+remapped, 3 peering, 39 active+clean; 4.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 36 MiB/s wr, 577 op/s; 21760/219399 objects misplaced (9.918%); 3.9 MiB/s, 61 objects/s recovering 2024-08-19T14:45:32.711 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:45:32.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:32.735 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:32.735 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:45:32.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:32 smithi032 ceph-mon[20004]: pgmap v2034: 47 pgs: 4 active+recovering+undersized+remapped, 1 active+recovering+remapped, 3 peering, 39 active+clean; 4.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 36 MiB/s wr, 577 op/s; 21760/219399 objects misplaced (9.918%); 3.9 MiB/s, 61 objects/s recovering 2024-08-19T14:45:32.836 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:45:32.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:32.860 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:32.861 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:45:32.961 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:45:32.987 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:32.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:32.988 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:45:33.089 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:45:33.113 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:33.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:33.115 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:45:33.215 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:45:33.240 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:33.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:33.241 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:45:33.319 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 72 16 78520 78504 68.1344 47.25 0.00773207 0.014666 2024-08-19T14:45:33.342 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:45:33.365 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:33.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:33.366 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:45:33.467 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:45:33.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:33.500 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:33.500 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:45:33.601 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:45:33.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:33.647 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:33.647 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:45:33.748 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:45:33.784 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:33.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:33.785 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:45:33.886 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:45:33.911 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:33.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:33.912 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:45:34.013 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:45:34.048 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:34.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:34.049 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:45:34.150 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:45:34.188 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:34.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:34.189 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:45:34.290 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:45:34.319 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:34.320 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 73 16 79513 79497 68.0512 62.0625 0.0392121 0.0146887 2024-08-19T14:45:34.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:34.320 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:45:34.422 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:45:34.452 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:34.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:34.453 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:45:34.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:34 smithi045 ceph-mon[20143]: pgmap v2035: 47 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+remapped, 42 active+clean; 4.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 43 MiB/s wr, 692 op/s; 21111/225144 objects misplaced (9.377%); 8.1 MiB/s, 129 objects/s recovering 2024-08-19T14:45:34.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:34 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:45:34.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:34 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:45:34.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:34 smithi045 ceph-mon[20143]: osdmap e867: 12 total, 12 up, 8 in 2024-08-19T14:45:34.554 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:45:34.583 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:34.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:34.584 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:45:34.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:34 smithi031 ceph-mon[16336]: pgmap v2035: 47 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+remapped, 42 active+clean; 4.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 43 MiB/s wr, 692 op/s; 21111/225144 objects misplaced (9.377%); 8.1 MiB/s, 129 objects/s recovering 2024-08-19T14:45:34.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:34 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:45:34.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:34 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:45:34.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:34 smithi031 ceph-mon[16336]: osdmap e867: 12 total, 12 up, 8 in 2024-08-19T14:45:34.685 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:45:34.716 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:34.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:34.718 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:45:34.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:34 smithi032 ceph-mon[20004]: pgmap v2035: 47 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+remapped, 42 active+clean; 4.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 43 MiB/s wr, 692 op/s; 21111/225144 objects misplaced (9.377%); 8.1 MiB/s, 129 objects/s recovering 2024-08-19T14:45:34.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:34 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:45:34.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:34 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:45:34.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:34 smithi032 ceph-mon[20004]: osdmap e867: 12 total, 12 up, 8 in 2024-08-19T14:45:34.819 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:45:34.842 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:34.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:34.843 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:45:34.944 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:45:34.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:34.974 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:34.974 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:45:35.075 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:45:35.099 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:35.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:35.100 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:45:35.201 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:45:35.235 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:35.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:35.236 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:45:35.320 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 74 16 80489 80473 67.9558 61 0.00805529 0.01471 2024-08-19T14:45:35.337 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:45:35.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:35.371 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:35.371 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:45:35.472 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:45:35.497 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:35.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:35.498 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:45:35.599 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:45:35.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:35.624 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:35.625 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:45:35.726 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:45:35.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:35.761 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:35.761 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:45:35.863 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:45:35.886 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:35.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:35.887 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:45:35.988 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:45:36.027 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:36.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:36.028 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:45:36.129 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:45:36.161 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:36.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:36.162 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:45:36.212 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:45:36.218 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:45:36.263 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:45:36.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:36.298 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:36.298 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:45:36.320 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 75 16 81442 81426 67.8438 59.5625 0.00623943 0.0147332 2024-08-19T14:45:36.400 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:45:36.431 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:36.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:36.432 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:45:36.533 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:45:36.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:36 smithi045 ceph-mon[20143]: pgmap v2037: 47 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+remapped, 42 active+clean; 4.7 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 48 MiB/s wr, 772 op/s; 20733/232110 objects misplaced (8.932%); 10 MiB/s, 158 objects/s recovering 2024-08-19T14:45:36.565 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:36.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:36.566 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:45:36.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:36 smithi031 ceph-mon[16336]: pgmap v2037: 47 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+remapped, 42 active+clean; 4.7 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 48 MiB/s wr, 772 op/s; 20733/232110 objects misplaced (8.932%); 10 MiB/s, 158 objects/s recovering 2024-08-19T14:45:36.667 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:45:36.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:36.691 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:36.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:45:36.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:36 smithi032 ceph-mon[20004]: pgmap v2037: 47 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+remapped, 42 active+clean; 4.7 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 48 MiB/s wr, 772 op/s; 20733/232110 objects misplaced (8.932%); 10 MiB/s, 158 objects/s recovering 2024-08-19T14:45:36.792 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:45:36.814 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:36.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:36.815 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:45:36.916 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:45:36.944 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:36.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:36.945 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:45:37.046 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:45:37.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:37.083 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:37.083 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:45:37.184 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:45:37.224 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:37.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:37.225 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:45:37.320 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 76 16 82415 82399 67.7512 60.8125 0.00326256 0.0147477 2024-08-19T14:45:37.325 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:45:37.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:37.352 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:37.352 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:45:37.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:37 smithi031 ceph-mon[16336]: pgmap v2038: 47 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+remapped, 42 active+clean; 4.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 55 MiB/s wr, 875 op/s; 20153/239193 objects misplaced (8.425%); 12 MiB/s, 189 objects/s recovering 2024-08-19T14:45:37.453 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:45:37.477 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:37.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:37.478 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:45:37.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:37 smithi045 ceph-mon[20143]: pgmap v2038: 47 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+remapped, 42 active+clean; 4.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 55 MiB/s wr, 875 op/s; 20153/239193 objects misplaced (8.425%); 12 MiB/s, 189 objects/s recovering 2024-08-19T14:45:37.578 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:45:37.601 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:37.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:37.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:45:37.703 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:45:37.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:37.735 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:37.735 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:45:37.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:37 smithi032 ceph-mon[20004]: pgmap v2038: 47 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+remapped, 42 active+clean; 4.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 55 MiB/s wr, 875 op/s; 20153/239193 objects misplaced (8.425%); 12 MiB/s, 189 objects/s recovering 2024-08-19T14:45:37.836 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:45:37.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:37.876 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:37.876 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:45:37.977 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:45:38.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:38.008 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:38.008 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:45:38.109 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:45:38.141 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:38.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:38.142 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:45:38.243 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:45:38.259 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:45:38.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:38.271 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:38.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:45:38.320 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 77 16 83287 83271 67.5791 54.5 0.0186826 0.0147812 2024-08-19T14:45:38.372 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:45:38.408 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:38.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:38.409 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:45:38.510 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:45:38.533 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:38.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:38.534 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:45:38.635 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:45:38.668 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:38.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:38.669 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:45:38.770 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:45:38.794 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:38.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:38.795 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:45:38.896 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:45:38.936 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:38.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:38.937 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:45:39.038 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:45:39.061 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:39.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:39.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:45:39.162 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:45:39.198 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:39.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:39.199 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:45:39.300 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:45:39.320 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 78 16 84185 84169 67.4322 56.125 0.00775803 0.0148239 2024-08-19T14:45:39.326 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:39.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:39.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:45:39.428 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:45:39.466 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:39.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:39.467 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:45:39.568 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:45:39.599 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:39 smithi031 ceph-mon[16336]: pgmap v2039: 47 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+remapped, 42 active+clean; 4.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 56 MiB/s wr, 900 op/s; 20153/240942 objects misplaced (8.364%); 12 MiB/s, 189 objects/s recovering 2024-08-19T14:45:39.603 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:39.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:39.604 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:45:39.705 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:45:39.739 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:39.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:39.740 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:45:39.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:39 smithi032 ceph-mon[20004]: pgmap v2039: 47 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+remapped, 42 active+clean; 4.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 56 MiB/s wr, 900 op/s; 20153/240942 objects misplaced (8.364%); 12 MiB/s, 189 objects/s recovering 2024-08-19T14:45:39.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:39 smithi045 ceph-mon[20143]: pgmap v2039: 47 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+remapped, 42 active+clean; 4.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 56 MiB/s wr, 900 op/s; 20153/240942 objects misplaced (8.364%); 12 MiB/s, 189 objects/s recovering 2024-08-19T14:45:39.841 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:45:39.880 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:39.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:39.881 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:45:39.982 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:45:40.006 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:40.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:40.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:45:40.108 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:45:40.145 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:40.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:40.146 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:45:40.247 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:45:40.271 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:40.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:40.272 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:45:40.320 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 79 16 85144 85128 67.3373 59.9375 0.0257816 0.0148443 2024-08-19T14:45:40.373 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:45:40.409 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:40.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:40.410 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:45:40.432 INFO:teuthology.orchestra.run.smithi031.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-08-19T14:45:40.483 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:45:40.483 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:45:40.511 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:45:40.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:40.551 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:40.551 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:45:40.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:40 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1370820306' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-08-19T14:45:40.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:40 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-08-19T14:45:40.652 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:45:40.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:40.676 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:40.676 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:45:40.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:40 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1370820306' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-08-19T14:45:40.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:40 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-08-19T14:45:40.777 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:45:40.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:40 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1370820306' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-08-19T14:45:40.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:40 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-08-19T14:45:40.799 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:40.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:40.801 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:45:40.902 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:45:40.926 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:40.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:40.927 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:45:41.028 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:45:41.051 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:41.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:41.053 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:45:41.154 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:45:41.197 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:41.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:41.198 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:45:41.299 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:45:41.320 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:2024-08-19T14:45:41.320771+0000 min lat: 0.00105666 max lat: 2.32965 avg lat: 0.0148879 2024-08-19T14:45:41.320 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-19T14:45:41.320 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 80 16 85940 85924 67.1174 49.75 0.0476284 0.0148879 2024-08-19T14:45:41.327 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:41.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:41.328 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:45:41.429 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:45:41.453 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is set 2024-08-19T14:45:41.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:41.454 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:41.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:45:41.555 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:45:41.589 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:41.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:41.590 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:45:41.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:41 smithi031 ceph-mon[16336]: pgmap v2040: 47 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+remapped, 42 active+clean; 5.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 58 MiB/s wr, 920 op/s; 19766/247017 objects misplaced (8.002%); 12 MiB/s, 199 objects/s recovering 2024-08-19T14:45:41.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:41 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-08-19T14:45:41.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:41 smithi031 ceph-mon[16336]: osdmap e868: 12 total, 12 up, 8 in 2024-08-19T14:45:41.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:41 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3014140114' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:45:41.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:41 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:45:41.691 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:45:41.725 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:41.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:41.726 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:45:41.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:41 smithi032 ceph-mon[20004]: pgmap v2040: 47 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+remapped, 42 active+clean; 5.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 58 MiB/s wr, 920 op/s; 19766/247017 objects misplaced (8.002%); 12 MiB/s, 199 objects/s recovering 2024-08-19T14:45:41.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:41 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-08-19T14:45:41.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:41 smithi032 ceph-mon[20004]: osdmap e868: 12 total, 12 up, 8 in 2024-08-19T14:45:41.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:41 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3014140114' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:45:41.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:41 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:45:41.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:41 smithi045 ceph-mon[20143]: pgmap v2040: 47 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+remapped, 42 active+clean; 5.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 58 MiB/s wr, 920 op/s; 19766/247017 objects misplaced (8.002%); 12 MiB/s, 199 objects/s recovering 2024-08-19T14:45:41.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:41 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-08-19T14:45:41.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:41 smithi045 ceph-mon[20143]: osdmap e868: 12 total, 12 up, 8 in 2024-08-19T14:45:41.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:41 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3014140114' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:45:41.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:41 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:45:41.826 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:45:41.852 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:41.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:41.853 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:45:41.954 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:45:41.984 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:41.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:41.985 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:45:42.086 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:45:42.109 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:42.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:42.110 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:45:42.211 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:45:42.246 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:42.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:42.247 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:45:42.320 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 81 16 86850 86834 66.9909 56.875 0.00541488 0.0149212 2024-08-19T14:45:42.348 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:45:42.375 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:42.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:42.376 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:45:42.476 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:45:42.505 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:42.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:42.507 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:45:42.607 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:45:42.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:42.646 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:42.647 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:45:42.748 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:45:42.760 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:42 smithi032 ceph-mon[20004]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:45:42.760 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:42 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:45:42.760 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:42 smithi032 ceph-mon[20004]: osdmap e869: 12 total, 12 up, 8 in 2024-08-19T14:45:42.780 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:42.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:42.781 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:45:42.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:42 smithi045 ceph-mon[20143]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:45:42.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:42 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:45:42.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:42 smithi045 ceph-mon[20143]: osdmap e869: 12 total, 12 up, 8 in 2024-08-19T14:45:42.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:42 smithi031 ceph-mon[16336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:45:42.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:42 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:45:42.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:42 smithi031 ceph-mon[16336]: osdmap e869: 12 total, 12 up, 8 in 2024-08-19T14:45:42.883 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:45:42.911 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:42.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:42.912 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:45:43.013 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:45:43.042 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:43.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:43.043 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:45:43.144 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:45:43.179 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:43.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:43.180 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:45:43.281 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:45:43.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:43.316 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:43.316 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:45:43.320 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 82 16 87758 87742 66.8659 56.75 0.00725792 0.0149486 2024-08-19T14:45:43.417 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:45:43.457 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:43.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:43.459 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:45:43.560 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:45:43.585 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:43.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:43.586 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:45:43.687 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:45:43.713 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:43.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:43.714 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:45:43.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:43 smithi032 ceph-mon[20004]: pgmap v2043: 47 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+remapped, 42 active+clean; 5.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 54 MiB/s wr, 871 op/s; 19165/253026 objects misplaced (7.574%); 12 MiB/s, 195 objects/s recovering 2024-08-19T14:45:43.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:43 smithi045 ceph-mon[20143]: pgmap v2043: 47 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+remapped, 42 active+clean; 5.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 54 MiB/s wr, 871 op/s; 19165/253026 objects misplaced (7.574%); 12 MiB/s, 195 objects/s recovering 2024-08-19T14:45:43.816 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:45:43.854 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:43.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:43.855 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:45:43.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:43 smithi031 ceph-mon[16336]: pgmap v2043: 47 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+remapped, 42 active+clean; 5.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 54 MiB/s wr, 871 op/s; 19165/253026 objects misplaced (7.574%); 12 MiB/s, 195 objects/s recovering 2024-08-19T14:45:43.955 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:45:43.980 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:43.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:43.981 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:45:44.082 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:45:44.124 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:44.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:44.125 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:45:44.226 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:45:44.261 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:44.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:44.262 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:45:44.320 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 83 16 88649 88633 66.7312 55.6875 0.00819051 0.0149796 2024-08-19T14:45:44.363 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:45:44.395 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:44.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:44.396 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:45:44.497 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:45:44.521 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:44.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:44.522 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:45:44.623 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:45:44.653 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:44.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:44.654 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:45:44.755 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:45:44.778 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:44.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:44.780 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:45:44.881 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:45:44.915 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:44.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:44.916 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:45:45.017 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:45:45.040 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:45.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:45.041 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:45:45.142 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:45:45.165 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:45.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:45.166 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:45:45.267 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:45:45.292 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:45.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:45.293 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:45:45.320 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 84 16 89427 89411 66.5156 48.625 0.00779377 0.0150285 2024-08-19T14:45:45.393 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:45:45.427 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:45.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:45.428 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:45:45.529 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:45:45.560 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:45.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:45.561 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:45:45.662 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:45:45.684 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:45.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:45.685 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:45:45.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:45 smithi032 ceph-mon[20004]: pgmap v2044: 47 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+remapped, 42 active+clean; 5.3 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 54 MiB/s wr, 858 op/s; 18792/259812 objects misplaced (7.233%); 11 MiB/s, 170 objects/s recovering 2024-08-19T14:45:45.786 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:45:45.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:45 smithi045 ceph-mon[20143]: pgmap v2044: 47 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+remapped, 42 active+clean; 5.3 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 54 MiB/s wr, 858 op/s; 18792/259812 objects misplaced (7.233%); 11 MiB/s, 170 objects/s recovering 2024-08-19T14:45:45.815 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:45.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:45.816 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:45:45.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:45 smithi031 ceph-mon[16336]: pgmap v2044: 47 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+remapped, 42 active+clean; 5.3 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 54 MiB/s wr, 858 op/s; 18792/259812 objects misplaced (7.233%); 11 MiB/s, 170 objects/s recovering 2024-08-19T14:45:45.917 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:45:45.941 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:45.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:45.942 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:45:46.043 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:45:46.082 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:46.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:46.083 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:45:46.184 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:45:46.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:46.220 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:46.220 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:45:46.321 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 85 16 90207 90191 66.3066 48.75 0.0165455 0.0150755 2024-08-19T14:45:46.321 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:45:46.365 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:46.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:46.366 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:45:46.467 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:45:46.493 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:46.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:46.494 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:45:46.595 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:45:46.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:46.625 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:46.626 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:45:46.726 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:45:46.760 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:46.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:46.761 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:45:46.862 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:45:46.886 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:46.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:46.887 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:45:46.988 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:45:47.015 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:47.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:47.016 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:45:47.117 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:45:47.147 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:47.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:47.148 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:45:47.249 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:45:47.283 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:47.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:47.284 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:45:47.321 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 86 16 91136 91120 66.2107 58.0625 0.0137912 0.0150977 2024-08-19T14:45:47.386 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:45:47.410 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:47.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:47.411 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:45:47.512 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:45:47.542 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:47.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:47.543 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:45:47.643 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:45:47.666 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:47.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:47.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:45:47.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:47 smithi032 ceph-mon[20004]: pgmap v2045: 47 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+remapped, 42 active+clean; 5.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 66 MiB/s wr, 1.06k op/s; 18124/266364 objects misplaced (6.804%); 16 MiB/s, 253 objects/s recovering 2024-08-19T14:45:47.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:47 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:45:47.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:47 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:45:47.768 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:45:47.789 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:47.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:47.791 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:45:47.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:47 smithi045 ceph-mon[20143]: pgmap v2045: 47 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+remapped, 42 active+clean; 5.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 66 MiB/s wr, 1.06k op/s; 18124/266364 objects misplaced (6.804%); 16 MiB/s, 253 objects/s recovering 2024-08-19T14:45:47.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:47 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:45:47.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:47 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:45:47.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:47 smithi031 ceph-mon[16336]: pgmap v2045: 47 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+remapped, 42 active+clean; 5.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 66 MiB/s wr, 1.06k op/s; 18124/266364 objects misplaced (6.804%); 16 MiB/s, 253 objects/s recovering 2024-08-19T14:45:47.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:47 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:45:47.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:47 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:45:47.894 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:45:47.917 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:47.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:47.918 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:45:48.019 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:45:48.043 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:48.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:48.044 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:45:48.145 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:45:48.177 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:48.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:48.178 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:45:48.279 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:45:48.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:48.303 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:48.303 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:45:48.322 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 87 16 91939 91923 66.0264 50.1875 0.00538799 0.0151389 2024-08-19T14:45:48.404 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:45:48.438 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:48.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:48.439 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:45:48.539 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:45:48.563 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:48.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:48.564 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:45:48.665 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:45:48.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:48.704 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:48.704 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:45:48.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:48 smithi032 ceph-mon[20004]: osdmap e870: 12 total, 12 up, 8 in 2024-08-19T14:45:48.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:48 smithi045 ceph-mon[20143]: osdmap e870: 12 total, 12 up, 8 in 2024-08-19T14:45:48.805 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:45:48.837 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:48.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:48.838 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:45:48.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:48 smithi031 ceph-mon[16336]: osdmap e870: 12 total, 12 up, 8 in 2024-08-19T14:45:48.939 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:45:48.973 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:48.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:48.974 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:45:49.075 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:45:49.098 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:49.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:49.099 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:45:49.201 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:45:49.224 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:49.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:49.225 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:45:49.323 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 88 16 92675 92659 65.7988 46 0.0370017 0.0151905 2024-08-19T14:45:49.326 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:45:49.350 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:49.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:49.351 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:45:49.452 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:45:49.484 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:49.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:49.485 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:45:49.585 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:45:49.609 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:49.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:49.612 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:45:49.713 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:45:49.742 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:49.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:49.743 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:45:49.844 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:45:49.881 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:49.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:49.882 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:45:49.983 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:45:50.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:50.030 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:50.030 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:45:50.131 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:45:50.166 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:50.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:50.167 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:45:50.268 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:45:50.295 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:50.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:50.296 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:45:50.321 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 89 16 93560 93544 65.681 55.3125 0.00802952 0.0152176 2024-08-19T14:45:50.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:50 smithi031 ceph-mon[16336]: pgmap v2047: 47 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+remapped, 42 active+clean; 5.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 55 MiB/s wr, 877 op/s; 18124/267936 objects misplaced (6.764%); 13 MiB/s, 206 objects/s recovering 2024-08-19T14:45:50.397 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:45:50.423 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:50.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:50.424 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:45:50.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:50 smithi032 ceph-mon[20004]: pgmap v2047: 47 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+remapped, 42 active+clean; 5.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 55 MiB/s wr, 877 op/s; 18124/267936 objects misplaced (6.764%); 13 MiB/s, 206 objects/s recovering 2024-08-19T14:45:50.525 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:45:50.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:50 smithi045 ceph-mon[20143]: pgmap v2047: 47 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+remapped, 42 active+clean; 5.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 55 MiB/s wr, 877 op/s; 18124/267936 objects misplaced (6.764%); 13 MiB/s, 206 objects/s recovering 2024-08-19T14:45:50.563 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:50.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:50.564 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:45:50.665 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:45:50.688 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:50.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:50.689 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:45:50.790 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:45:50.819 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:50.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:50.821 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:45:50.921 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:45:50.944 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:50.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:50.945 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:45:51.046 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:45:51.083 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:51.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:51.084 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:45:51.185 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:45:51.209 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:51.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:51.210 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:45:51.311 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:45:51.321 INFO:tasks.radosbench.radosbench.2.smithi045.stdout: 90 3 94502 94499 65.6143 59.6875 0.013762 0.0152349 2024-08-19T14:45:51.332 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:51.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:51.333 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:45:51.434 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:45:51.456 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:51.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:51.457 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:45:51.558 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:45:51.580 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:51.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:51.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:45:51.682 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:45:51.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:51.704 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:51.705 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:45:51.806 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:45:51.827 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:51.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:51.828 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:45:51.929 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:45:51.951 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:51.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:51.952 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:45:52.053 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:45:52.075 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:52.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:52.076 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:45:52.177 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:45:52.199 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:52.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:52.200 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:45:52.301 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:45:52.321 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Total time run: 90.0252 2024-08-19T14:45:52.322 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Total writes made: 94502 2024-08-19T14:45:52.322 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Write size: 65536 2024-08-19T14:45:52.322 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Object size: 65536 2024-08-19T14:45:52.322 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Bandwidth (MB/sec): 65.608 2024-08-19T14:45:52.322 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Stddev Bandwidth: 40.7167 2024-08-19T14:45:52.322 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Max bandwidth (MB/sec): 186.75 2024-08-19T14:45:52.322 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Min bandwidth (MB/sec): 0 2024-08-19T14:45:52.322 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Average IOPS: 1049 2024-08-19T14:45:52.322 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Stddev IOPS: 651.477 2024-08-19T14:45:52.322 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Max IOPS: 2988 2024-08-19T14:45:52.322 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Min IOPS: 0 2024-08-19T14:45:52.322 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Average Latency(s): 0.0152352 2024-08-19T14:45:52.323 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Stddev Latency(s): 0.0527767 2024-08-19T14:45:52.323 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Max latency(s): 2.32965 2024-08-19T14:45:52.323 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Min latency(s): 0.00105666 2024-08-19T14:45:52.324 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:52.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:52.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:45:52.328 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Cleaning up (deleting benchmark objects) 2024-08-19T14:45:52.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:52 smithi031 ceph-mon[16336]: pgmap v2048: 47 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+remapped, 42 active+clean; 5.6 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 48 MiB/s wr, 762 op/s; 17719/273447 objects misplaced (6.480%); 10 MiB/s, 162 objects/s recovering 2024-08-19T14:45:52.426 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:45:52.448 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:52.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:52.449 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:45:52.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:52 smithi032 ceph-mon[20004]: pgmap v2048: 47 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+remapped, 42 active+clean; 5.6 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 48 MiB/s wr, 762 op/s; 17719/273447 objects misplaced (6.480%); 10 MiB/s, 162 objects/s recovering 2024-08-19T14:45:52.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:52 smithi045 ceph-mon[20143]: pgmap v2048: 47 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+remapped, 42 active+clean; 5.6 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 48 MiB/s wr, 762 op/s; 17719/273447 objects misplaced (6.480%); 10 MiB/s, 162 objects/s recovering 2024-08-19T14:45:52.550 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:45:52.581 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:52.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:52.582 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:45:52.683 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:45:52.705 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:52.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:52.706 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:45:52.807 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:45:52.836 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:52.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:52.837 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:45:52.937 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:45:52.961 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:52.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:52.962 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:45:53.063 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:45:53.122 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:53.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:53.123 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:45:53.224 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:45:53.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:53.259 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:53.259 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:45:53.361 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:45:53.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:53 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:45:53.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:53 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:45:53.392 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:53.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:53.393 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:45:53.408 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:53 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:45:53.408 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:53 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:45:53.494 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:45:53.519 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:53.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:53.520 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:45:53.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:53 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:45:53.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:53 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:45:53.621 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:45:53.647 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:53.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:53.649 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:45:53.750 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:45:53.774 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:53.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:53.775 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:45:53.877 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:45:53.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:53.914 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:53.914 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:45:54.015 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:45:54.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:54.043 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:54.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:45:54.144 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:45:54.168 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:54.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:54.169 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:45:54.270 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:45:54.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:54.295 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:54.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:45:54.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:54 smithi031 ceph-mon[16336]: pgmap v2049: 47 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+remapped, 42 active+clean; 5.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 54 MiB/s wr, 862 op/s; 16998/278907 objects misplaced (6.095%); 14 MiB/s, 216 objects/s recovering 2024-08-19T14:45:54.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:54 smithi031 ceph-mon[16336]: osdmap e871: 12 total, 12 up, 8 in 2024-08-19T14:45:54.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:54 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:45:54.395 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:45:54.438 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:54.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:54.439 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:45:54.448 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:54 smithi045 ceph-mon[20143]: pgmap v2049: 47 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+remapped, 42 active+clean; 5.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 54 MiB/s wr, 862 op/s; 16998/278907 objects misplaced (6.095%); 14 MiB/s, 216 objects/s recovering 2024-08-19T14:45:54.448 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:54 smithi045 ceph-mon[20143]: osdmap e871: 12 total, 12 up, 8 in 2024-08-19T14:45:54.448 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:54 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:45:54.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:54 smithi032 ceph-mon[20004]: pgmap v2049: 47 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+remapped, 42 active+clean; 5.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 54 MiB/s wr, 862 op/s; 16998/278907 objects misplaced (6.095%); 14 MiB/s, 216 objects/s recovering 2024-08-19T14:45:54.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:54 smithi032 ceph-mon[20004]: osdmap e871: 12 total, 12 up, 8 in 2024-08-19T14:45:54.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:54 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:45:54.540 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:45:54.564 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:54.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:54.566 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:45:54.666 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:45:54.689 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:54.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:54.690 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:45:54.791 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:45:54.828 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:54.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:54.833 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:45:54.934 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:45:54.964 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:54.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:54.965 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:45:55.066 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:45:55.090 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:55.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:55.091 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:45:55.192 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:45:55.230 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:55.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:55.231 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:45:55.332 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:45:55.356 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:55.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:55.358 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:45:55.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:55 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:45:55.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:55 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:45:55.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:55 smithi031 ceph-mon[16336]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:45:55.458 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:45:55.486 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:55.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:55.487 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:45:55.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:55 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:45:55.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:55 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:45:55.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:55 smithi032 ceph-mon[20004]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:45:55.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:55 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:45:55.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:55 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:45:55.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:55 smithi045 ceph-mon[20143]: from='mgr.14150 ' entity='mgr.y' 2024-08-19T14:45:55.588 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:45:55.612 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:55.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:55.613 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:45:55.714 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:45:55.740 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:55.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:55.741 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:45:55.842 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:45:55.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:55.883 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:55.883 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:45:55.984 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:45:56.009 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:56.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:56.010 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:45:56.111 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:45:56.140 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:56.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:56.142 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:45:56.243 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:45:56.271 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:56.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:56.273 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:45:56.374 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:45:56.396 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:56.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:56.397 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:45:56.498 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:45:56.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:56 smithi032 ceph-mon[20004]: pgmap v2051: 47 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+remapped, 42 active+clean; 5.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 127 B/s rd, 42 MiB/s wr, 734 op/s; 16451/280821 objects misplaced (5.858%); 13 MiB/s, 209 objects/s recovering 2024-08-19T14:45:56.521 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:56.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:56.522 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:45:56.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:56 smithi045 ceph-mon[20143]: pgmap v2051: 47 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+remapped, 42 active+clean; 5.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 127 B/s rd, 42 MiB/s wr, 734 op/s; 16451/280821 objects misplaced (5.858%); 13 MiB/s, 209 objects/s recovering 2024-08-19T14:45:56.600 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:56 smithi031 ceph-mon[16336]: pgmap v2051: 47 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+remapped, 42 active+clean; 5.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 127 B/s rd, 42 MiB/s wr, 734 op/s; 16451/280821 objects misplaced (5.858%); 13 MiB/s, 209 objects/s recovering 2024-08-19T14:45:56.622 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:45:56.630 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-19T14:45:56.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:56.653 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:56.653 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:45:56.755 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:45:56.780 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:56.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:56.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:45:56.882 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:45:56.920 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:56.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:56.921 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:45:57.023 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:45:57.051 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:57.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:57.052 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:45:57.153 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:45:57.182 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:57.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:57.183 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:45:57.284 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:45:57.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:57.315 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:57.315 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:45:57.416 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:45:57.438 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:57.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:57.439 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:45:57.540 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:45:57.569 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:57.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:57.570 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:45:57.671 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:45:57.695 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:57.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:57.696 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:45:57.797 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:45:57.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:57.827 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:57.827 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:45:57.928 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:45:57.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:57.952 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:57.952 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:45:58.053 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:45:58.083 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:58.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:58.084 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:45:58.185 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:45:58.208 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:58.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:58.209 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:45:58.310 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:45:58.336 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:58.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:58.337 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:45:58.439 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:45:58.464 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:58.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:58.465 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:45:58.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:45:58 smithi032 ceph-mon[20004]: pgmap v2052: 47 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+remapped, 42 active+clean; 5.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 114 B/s rd, 40 MiB/s wr, 872 op/s; 15572/277368 objects misplaced (5.614%); 18 MiB/s, 286 objects/s recovering 2024-08-19T14:45:58.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:45:58 smithi045 ceph-mon[20143]: pgmap v2052: 47 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+remapped, 42 active+clean; 5.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 114 B/s rd, 40 MiB/s wr, 872 op/s; 15572/277368 objects misplaced (5.614%); 18 MiB/s, 286 objects/s recovering 2024-08-19T14:45:58.567 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:45:58.589 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:58.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:58.590 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:45:58.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:45:58 smithi031 ceph-mon[16336]: pgmap v2052: 47 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+remapped, 42 active+clean; 5.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 114 B/s rd, 40 MiB/s wr, 872 op/s; 15572/277368 objects misplaced (5.614%); 18 MiB/s, 286 objects/s recovering 2024-08-19T14:45:58.691 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:45:58.715 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:45:58.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:58.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:45:58.766 INFO:tasks.thrashosds.thrasher:in_osds: [1, 4, 8, 2, 11, 9, 10, 6] out_osds: [3, 0, 7, 5] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 8, 9, 11, 10, 2] 2024-08-19T14:45:58.767 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd reweight 1 1 2024-08-19T14:45:58.816 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:45:58.863 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:58.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:58.864 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:45:58.965 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:45:58.999 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:58.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:59.000 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:45:59.041 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:45:59.086 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-08-19T14:45:59.101 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:45:59.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:59.135 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:59.135 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:45:59.166 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:45:59.236 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:45:59.258 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:59.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:59.259 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:45:59.360 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:45:59.405 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:59.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:59.406 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:45:59.507 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:45:59.550 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:59.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:59.551 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:45:59.652 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:45:59.681 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:59.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:59.682 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:45:59.783 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:45:59.822 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:45:59.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:59.824 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:45:59.925 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:45:59.953 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:45:59.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:45:59.954 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:46:00.055 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:46:00.078 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:00.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:00.079 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:46:00.180 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:46:00.219 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:00.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:00.221 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:46:00.321 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:46:00.346 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:00.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:00.347 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:46:00.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:00 smithi031 ceph-mon[16336]: pgmap v2053: 47 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+remapped, 42 active+clean; 5.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 102 B/s rd, 32 MiB/s wr, 780 op/s; 15572/275682 objects misplaced (5.649%); 16 MiB/s, 255 objects/s recovering 2024-08-19T14:46:00.448 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:46:00.471 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:00.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:00.472 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:46:00.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:00 smithi032 ceph-mon[20004]: pgmap v2053: 47 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+remapped, 42 active+clean; 5.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 102 B/s rd, 32 MiB/s wr, 780 op/s; 15572/275682 objects misplaced (5.649%); 16 MiB/s, 255 objects/s recovering 2024-08-19T14:46:00.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:00 smithi045 ceph-mon[20143]: pgmap v2053: 47 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+remapped, 42 active+clean; 5.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 102 B/s rd, 32 MiB/s wr, 780 op/s; 15572/275682 objects misplaced (5.649%); 16 MiB/s, 255 objects/s recovering 2024-08-19T14:46:00.573 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:46:00.598 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:00.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:00.599 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:46:00.700 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:46:00.725 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:00.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:00.726 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:46:00.827 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:46:00.877 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:00.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:00.878 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:46:00.979 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:46:01.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:01.011 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:01.011 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:46:01.112 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:46:01.136 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:01.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:01.137 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:46:01.239 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:46:01.268 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:01.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:01.269 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:46:01.370 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:46:01.400 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:01.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:01.402 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:46:01.503 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:46:01.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:01 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:46:01.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:01 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:46:01.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:01 smithi032 ceph-mon[20004]: osdmap e872: 12 total, 12 up, 8 in 2024-08-19T14:46:01.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:01 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:46:01.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:01 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:46:01.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:01 smithi045 ceph-mon[20143]: osdmap e872: 12 total, 12 up, 8 in 2024-08-19T14:46:01.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:01.548 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:01.548 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:46:01.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:01 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:46:01.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:01 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:46:01.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:01 smithi031 ceph-mon[16336]: osdmap e872: 12 total, 12 up, 8 in 2024-08-19T14:46:01.650 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:46:01.684 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:01.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:01.685 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:46:01.786 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:46:01.811 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:01.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:01.812 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:46:01.891 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:46:01.901 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:46:01.901 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:46:01.914 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:46:01.931 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:46:01.931 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:46:01.939 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:01.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:01.940 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:46:01.960 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:46:01.960 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:46:01.984 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:46:01.984 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:46:02.017 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:46:02.018 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:46:02.040 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:46:02.043 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:46:02.043 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:46:02.070 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:46:02.070 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:46:02.080 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:02.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:02.082 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:46:02.092 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:46:02.092 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:46:02.124 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:46:02.125 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:46:02.158 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:46:02.158 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:46:02.182 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:46:02.187 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:46:02.188 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:46:02.215 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:02.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:02.216 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:46:02.221 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:46:02.221 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:46:02.316 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:46:02.339 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:02.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:02.341 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:46:02.359 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:02 smithi031 ceph-mon[16336]: pgmap v2055: 47 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+remapped, 42 active+clean; 5.5 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 127 B/s rd, 12 MiB/s wr, 765 op/s; 15087/269742 objects misplaced (5.593%); 15 MiB/s, 238 objects/s recovering 2024-08-19T14:46:02.359 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:02 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3285618221' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:46:02.441 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:46:02.464 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:02.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:02.465 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:46:02.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:02 smithi032 ceph-mon[20004]: pgmap v2055: 47 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+remapped, 42 active+clean; 5.5 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 127 B/s rd, 12 MiB/s wr, 765 op/s; 15087/269742 objects misplaced (5.593%); 15 MiB/s, 238 objects/s recovering 2024-08-19T14:46:02.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:02 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3285618221' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:46:02.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:02 smithi045 ceph-mon[20143]: pgmap v2055: 47 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+remapped, 42 active+clean; 5.5 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 127 B/s rd, 12 MiB/s wr, 765 op/s; 15087/269742 objects misplaced (5.593%); 15 MiB/s, 238 objects/s recovering 2024-08-19T14:46:02.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:02 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3285618221' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:46:02.566 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:46:02.594 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:02.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:02.595 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:46:02.696 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:46:02.725 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:02.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:02.726 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:46:02.828 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:46:02.865 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:02.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:02.866 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:46:02.967 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:46:02.997 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:02.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:02.998 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:46:03.078 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:46:03.099 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:46:03.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:03.152 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:03.152 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:46:03.253 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:46:03.290 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:03.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:03.291 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:46:03.392 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:46:03.426 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:03.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:03.427 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:46:03.528 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:46:03.562 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:03.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:03.564 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:46:03.665 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:46:03.706 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:03.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:03.707 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:46:03.808 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:46:03.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:03.847 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:03.848 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:46:03.949 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:46:03.979 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:03.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:03.980 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:46:04.081 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:46:04.104 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:04.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:04.105 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:46:04.151 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is unset 2024-08-19T14:46:04.154 INFO:teuthology.orchestra.run.smithi031.stderr:reweighted osd.1 to 1 (10000) 2024-08-19T14:46:04.206 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:46:04.242 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:04.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:04.243 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:46:04.344 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:46:04.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:04 smithi031 ceph-mon[16336]: pgmap v2056: 47 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+remapped, 42 active+clean; 5.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 110 B/s rd, 8.0 MiB/s wr, 813 op/s; 14255/264318 objects misplaced (5.393%); 18 MiB/s, 295 objects/s recovering 2024-08-19T14:46:04.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:04 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/905279678' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:46:04.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:04 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/4232172136' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-08-19T14:46:04.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:04 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-08-19T14:46:04.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:04 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:46:04.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:04 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:46:04.368 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:04.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:04.369 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:46:04.470 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:46:04.497 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:04.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:04.498 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:46:04.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:04 smithi032 ceph-mon[20004]: pgmap v2056: 47 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+remapped, 42 active+clean; 5.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 110 B/s rd, 8.0 MiB/s wr, 813 op/s; 14255/264318 objects misplaced (5.393%); 18 MiB/s, 295 objects/s recovering 2024-08-19T14:46:04.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:04 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/905279678' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:46:04.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:04 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/4232172136' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-08-19T14:46:04.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:04 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-08-19T14:46:04.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:04 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:46:04.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:04 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:46:04.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:04 smithi045 ceph-mon[20143]: pgmap v2056: 47 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+remapped, 42 active+clean; 5.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 110 B/s rd, 8.0 MiB/s wr, 813 op/s; 14255/264318 objects misplaced (5.393%); 18 MiB/s, 295 objects/s recovering 2024-08-19T14:46:04.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:04 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/905279678' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:46:04.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:04 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/4232172136' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-08-19T14:46:04.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:04 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-08-19T14:46:04.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:04 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:46:04.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:04 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:46:04.598 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:46:04.621 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:04.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:04.622 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:46:04.723 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:46:04.751 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:04.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:04.752 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:46:04.852 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:46:04.878 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:04.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:04.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:46:04.980 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:46:05.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:05.007 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:05.007 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:46:05.109 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:46:05.139 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:05.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:05.140 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:46:05.241 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:46:05.284 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:05.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:05.285 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:46:05.386 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:46:05.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:05.411 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:05.411 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:46:05.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:05 smithi032 ceph-mon[20004]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:46:05.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:05 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/905279678' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:46:05.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:05 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-08-19T14:46:05.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:05 smithi032 ceph-mon[20004]: osdmap e873: 12 total, 12 up, 8 in 2024-08-19T14:46:05.512 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:46:05.540 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:05.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:05.541 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:46:05.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:05 smithi045 ceph-mon[20143]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:46:05.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:05 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/905279678' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:46:05.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:05 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-08-19T14:46:05.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:05 smithi045 ceph-mon[20143]: osdmap e873: 12 total, 12 up, 8 in 2024-08-19T14:46:05.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:05 smithi031 ceph-mon[16336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:46:05.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:05 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/905279678' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:46:05.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:05 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-08-19T14:46:05.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:05 smithi031 ceph-mon[16336]: osdmap e873: 12 total, 12 up, 8 in 2024-08-19T14:46:05.643 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:46:05.666 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:05.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:05.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:46:05.768 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:46:05.790 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:46:05.792 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:05.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:05.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:46:05.893 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:46:05.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:05.938 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:05.938 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:46:06.039 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:46:06.071 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:06.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:06.072 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:46:06.173 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:46:06.196 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:06.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:06.197 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:46:06.298 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:46:06.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:06.340 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:06.340 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:46:06.441 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:46:06.478 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:06.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:06.479 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:46:06.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:06 smithi032 ceph-mon[20004]: pgmap v2058: 47 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+remapped, 42 active+clean; 5.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 819 op/s; 13735/257691 objects misplaced (5.330%); 14 MiB/s, 229 objects/s recovering 2024-08-19T14:46:06.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:06 smithi045 ceph-mon[20143]: pgmap v2058: 47 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+remapped, 42 active+clean; 5.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 819 op/s; 13735/257691 objects misplaced (5.330%); 14 MiB/s, 229 objects/s recovering 2024-08-19T14:46:06.581 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:46:06.610 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:06.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:06.611 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:46:06.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:06 smithi031 ceph-mon[16336]: pgmap v2058: 47 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+remapped, 42 active+clean; 5.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 819 op/s; 13735/257691 objects misplaced (5.330%); 14 MiB/s, 229 objects/s recovering 2024-08-19T14:46:06.712 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:46:06.744 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:06.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:06.745 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:46:06.846 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:46:06.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:06.884 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:06.886 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:46:06.987 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:46:07.025 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:07.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:07.027 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:46:07.128 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:46:07.151 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:07.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:07.153 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:46:07.254 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:46:07.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:07.295 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:07.296 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:46:07.397 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:46:07.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:07.431 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:07.431 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:46:07.533 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:46:07.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:07.561 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:07.561 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:46:07.662 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:46:07.690 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:07.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:07.691 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:46:07.792 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:46:07.827 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:07.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:07.828 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:46:07.929 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:46:07.934 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:46:07.934 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 20, 2024-08-19T14:46:07.934 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 600, 2024-08-19T14:46:07.934 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [ 2024-08-19T14:46:07.934 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.934 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.16962.0:181780 7.e e678/634 7:7b43e48b:::benchmark_data_smithi045_115842_object63620:head v 678'11470, mlcod=613'4070)", 2024-08-19T14:46:07.935 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:37:43.454572+0000", 2024-08-19T14:46:07.935 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 503.59290268799998, 2024-08-19T14:46:07.935 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.161739627, 2024-08-19T14:46:07.935 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:46:07.935 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:46:07.935 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:46:07.935 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:46:07.935 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.935 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:46:07.935 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.454572+0000", 2024-08-19T14:46:07.935 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:07.935 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.935 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.936 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:46:07.936 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.454572+0000", 2024-08-19T14:46:07.936 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:07.936 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.936 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.936 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:46:07.936 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.454572+0000", 2024-08-19T14:46:07.936 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.48e-07 2024-08-19T14:46:07.936 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.936 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.936 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:46:07.936 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.454576+0000", 2024-08-19T14:46:07.936 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2910000000000001e-06 2024-08-19T14:46:07.936 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.937 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.937 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:46:07.937 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.454583+0000", 2024-08-19T14:46:07.937 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.3890000000000002e-06 2024-08-19T14:46:07.937 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.937 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.937 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:46:07.937 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.454588+0000", 2024-08-19T14:46:07.937 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0860000000000001e-06 2024-08-19T14:46:07.937 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.937 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.937 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:46:07.937 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.473215+0000", 2024-08-19T14:46:07.938 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.018626888000000001 2024-08-19T14:46:07.938 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.938 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.938 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:46:07.938 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.473241+0000", 2024-08-19T14:46:07.938 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.6454000000000001e-05 2024-08-19T14:46:07.938 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.938 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.938 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:46:07.938 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.616231+0000", 2024-08-19T14:46:07.938 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.14299003299999999 2024-08-19T14:46:07.939 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.939 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.939 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:46:07.939 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.616312+0000", 2024-08-19T14:46:07.939 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.0238e-05 2024-08-19T14:46:07.939 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:07.939 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:46:07.939 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:07.939 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.939 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.939 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.16962.0:181792 7.e e678/634 7:76fea455:::benchmark_data_smithi045_115842_object63632:head v 678'11471, mlcod=613'4070)", 2024-08-19T14:46:07.939 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:37:43.470051+0000", 2024-08-19T14:46:07.940 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 503.57742341599999, 2024-08-19T14:46:07.940 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.14628962500000001, 2024-08-19T14:46:07.940 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:46:07.940 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:46:07.940 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:46:07.940 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:46:07.940 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.940 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:46:07.940 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.470051+0000", 2024-08-19T14:46:07.940 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:07.940 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.940 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.940 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:46:07.941 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.470051+0000", 2024-08-19T14:46:07.941 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:07.941 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.941 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.941 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:46:07.941 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.470052+0000", 2024-08-19T14:46:07.941 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.032e-06 2024-08-19T14:46:07.941 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.941 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.941 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:46:07.942 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.470061+0000", 2024-08-19T14:46:07.942 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.0270000000000001e-06 2024-08-19T14:46:07.942 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.942 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.942 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:46:07.942 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.470074+0000", 2024-08-19T14:46:07.942 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2651e-05 2024-08-19T14:46:07.942 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.942 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.942 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:46:07.942 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.470082+0000", 2024-08-19T14:46:07.942 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.0469999999999994e-06 2024-08-19T14:46:07.942 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.943 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.943 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:46:07.943 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.487157+0000", 2024-08-19T14:46:07.943 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.017074807000000001 2024-08-19T14:46:07.943 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.943 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.943 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:46:07.943 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.487182+0000", 2024-08-19T14:46:07.943 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.5338e-05 2024-08-19T14:46:07.943 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.943 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.943 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:46:07.943 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.616318+0000", 2024-08-19T14:46:07.944 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.129135683 2024-08-19T14:46:07.944 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.944 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.944 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:46:07.944 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.616341+0000", 2024-08-19T14:46:07.944 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.304e-05 2024-08-19T14:46:07.944 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:07.944 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:46:07.944 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:07.944 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.944 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.944 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.16962.0:181793 7.e e678/634 7:73e8a3a5:::benchmark_data_smithi045_115842_object63633:head v 678'11472, mlcod=613'4070)", 2024-08-19T14:46:07.944 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:37:43.470096+0000", 2024-08-19T14:46:07.944 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 503.577378313, 2024-08-19T14:46:07.944 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.14626562000000001, 2024-08-19T14:46:07.945 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:46:07.945 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:46:07.945 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:46:07.945 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:46:07.945 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.945 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:46:07.945 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.470096+0000", 2024-08-19T14:46:07.945 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:07.945 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.945 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.945 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:46:07.945 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.470096+0000", 2024-08-19T14:46:07.945 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:07.945 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.946 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.946 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:46:07.946 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.470096+0000", 2024-08-19T14:46:07.946 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.3699999999999999e-07 2024-08-19T14:46:07.946 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.946 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.946 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:46:07.946 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.470100+0000", 2024-08-19T14:46:07.946 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.9829999999999998e-06 2024-08-19T14:46:07.946 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.946 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.946 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:46:07.946 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.470106+0000", 2024-08-19T14:46:07.946 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.8490000000000002e-06 2024-08-19T14:46:07.947 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.947 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.947 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:46:07.947 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.470111+0000", 2024-08-19T14:46:07.947 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.5480000000000003e-06 2024-08-19T14:46:07.947 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.947 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.947 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:46:07.947 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.501919+0000", 2024-08-19T14:46:07.947 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.031808014000000003 2024-08-19T14:46:07.947 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.947 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.947 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:46:07.947 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.501945+0000", 2024-08-19T14:46:07.948 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.6227000000000001e-05 2024-08-19T14:46:07.948 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.948 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.948 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:46:07.948 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.616343+0000", 2024-08-19T14:46:07.948 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.114397659 2024-08-19T14:46:07.948 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.948 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.948 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:46:07.948 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:43.616362+0000", 2024-08-19T14:46:07.948 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.9103e-05 2024-08-19T14:46:07.948 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:07.949 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:46:07.949 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:07.949 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.949 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.949 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.16962.0:199394 7.2 e690/635 7:406e87f5:::benchmark_data_smithi045_115842_object81234:head v 690'12614, mlcod=621'5122)", 2024-08-19T14:46:07.949 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:38:22.737244+0000", 2024-08-19T14:46:07.949 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 464.31023086699997, 2024-08-19T14:46:07.949 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.24429008799999999, 2024-08-19T14:46:07.949 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:46:07.949 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:46:07.949 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:46:07.950 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:46:07.950 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.950 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:46:07.950 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.737244+0000", 2024-08-19T14:46:07.950 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:07.950 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.950 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.950 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:46:07.950 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.737244+0000", 2024-08-19T14:46:07.950 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:07.950 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.950 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.950 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:46:07.950 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.737245+0000", 2024-08-19T14:46:07.951 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.6299999999999993e-07 2024-08-19T14:46:07.951 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.951 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.951 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:46:07.951 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.737251+0000", 2024-08-19T14:46:07.951 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.6930000000000001e-06 2024-08-19T14:46:07.951 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.951 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.953 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:46:07.954 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.737262+0000", 2024-08-19T14:46:07.954 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0922e-05 2024-08-19T14:46:07.954 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.954 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.954 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:46:07.954 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.737269+0000", 2024-08-19T14:46:07.954 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.6680000000000004e-06 2024-08-19T14:46:07.954 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.954 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.954 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:46:07.954 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.737648+0000", 2024-08-19T14:46:07.954 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00037954999999999998 2024-08-19T14:46:07.955 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.955 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.955 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:46:07.955 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.737671+0000", 2024-08-19T14:46:07.955 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.2784000000000001e-05 2024-08-19T14:46:07.955 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.955 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.955 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:46:07.955 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.981464+0000", 2024-08-19T14:46:07.955 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.243792642 2024-08-19T14:46:07.955 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.955 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.955 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:46:07.955 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.981534+0000", 2024-08-19T14:46:07.956 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.9865999999999998e-05 2024-08-19T14:46:07.956 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:07.956 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:46:07.956 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:07.956 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.956 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.956 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.16962.0:199392 7.2 e690/635 7:4580edf3:::benchmark_data_smithi045_115842_object81232:head v 690'12615, mlcod=621'5122)", 2024-08-19T14:46:07.956 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:38:22.746524+0000", 2024-08-19T14:46:07.956 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 464.30095090700001, 2024-08-19T14:46:07.956 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.23504549899999999, 2024-08-19T14:46:07.956 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:46:07.956 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:46:07.956 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:46:07.956 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:46:07.957 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.957 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:46:07.957 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.746524+0000", 2024-08-19T14:46:07.957 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:07.957 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.957 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.957 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:46:07.957 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.746524+0000", 2024-08-19T14:46:07.957 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:07.957 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.957 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.957 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:46:07.957 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.746525+0000", 2024-08-19T14:46:07.957 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0279999999999999e-06 2024-08-19T14:46:07.957 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.958 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.958 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:46:07.958 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.746533+0000", 2024-08-19T14:46:07.958 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.0069999999999997e-06 2024-08-19T14:46:07.958 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.958 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.958 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:46:07.958 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.746545+0000", 2024-08-19T14:46:07.958 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2584e-05 2024-08-19T14:46:07.958 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.958 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.958 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:46:07.958 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.746554+0000", 2024-08-19T14:46:07.959 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.0180000000000007e-06 2024-08-19T14:46:07.959 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.959 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.959 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:46:07.959 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.752313+0000", 2024-08-19T14:46:07.959 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0057586139999999996 2024-08-19T14:46:07.959 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.959 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.959 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:46:07.959 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.752334+0000", 2024-08-19T14:46:07.959 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.0735000000000001e-05 2024-08-19T14:46:07.959 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.959 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.960 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:46:07.960 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.981537+0000", 2024-08-19T14:46:07.960 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.22920327500000001 2024-08-19T14:46:07.960 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.960 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.960 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:46:07.960 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.981569+0000", 2024-08-19T14:46:07.960 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.2237999999999999e-05 2024-08-19T14:46:07.960 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:07.960 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:46:07.960 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:07.960 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.960 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.960 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.16962.0:199409 7.2 e690/635 7:4c8d089b:::benchmark_data_smithi045_115842_object81249:head v 690'12616, mlcod=621'5122)", 2024-08-19T14:46:07.961 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:38:22.750502+0000", 2024-08-19T14:46:07.961 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 464.29697271200001, 2024-08-19T14:46:07.961 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.255713405, 2024-08-19T14:46:07.961 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:46:07.961 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:46:07.961 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:46:07.961 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:46:07.961 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.961 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:46:07.961 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.750502+0000", 2024-08-19T14:46:07.961 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:07.961 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.961 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.961 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:46:07.962 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.750502+0000", 2024-08-19T14:46:07.962 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:07.962 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.962 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.962 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:46:07.962 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.750503+0000", 2024-08-19T14:46:07.962 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.09e-07 2024-08-19T14:46:07.962 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.962 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.962 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:46:07.962 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.750511+0000", 2024-08-19T14:46:07.962 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.9759999999999995e-06 2024-08-19T14:46:07.962 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.962 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.963 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:46:07.963 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.750522+0000", 2024-08-19T14:46:07.963 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1241e-05 2024-08-19T14:46:07.963 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.963 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.963 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:46:07.963 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.750529+0000", 2024-08-19T14:46:07.963 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.3629999999999996e-06 2024-08-19T14:46:07.963 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.963 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.963 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:46:07.963 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.765007+0000", 2024-08-19T14:46:07.963 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.014478058 2024-08-19T14:46:07.964 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.964 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.964 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:46:07.964 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.765023+0000", 2024-08-19T14:46:07.964 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5735000000000001e-05 2024-08-19T14:46:07.964 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.964 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.964 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:46:07.964 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:23.006112+0000", 2024-08-19T14:46:07.964 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.24108848799999999 2024-08-19T14:46:07.964 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.964 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.964 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:46:07.964 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:23.006215+0000", 2024-08-19T14:46:07.965 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00010363500000000001 2024-08-19T14:46:07.965 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:07.965 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:46:07.965 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:07.965 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.965 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.965 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.16962.0:199414 7.c e690/635 7:366d05c1:::benchmark_data_smithi045_115842_object81254:head v 690'12294, mlcod=621'5016)", 2024-08-19T14:46:07.965 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:38:22.751905+0000", 2024-08-19T14:46:07.966 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 464.29556932399998, 2024-08-19T14:46:07.966 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.20950311999999999, 2024-08-19T14:46:07.966 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:46:07.966 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:46:07.966 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:46:07.966 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:46:07.966 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.966 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:46:07.966 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.751905+0000", 2024-08-19T14:46:07.966 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:07.966 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.966 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.966 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:46:07.967 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.751905+0000", 2024-08-19T14:46:07.967 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:07.967 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.967 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.967 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:46:07.967 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.751906+0000", 2024-08-19T14:46:07.967 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.4000000000000001e-07 2024-08-19T14:46:07.967 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.967 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.967 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:46:07.967 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.751912+0000", 2024-08-19T14:46:07.967 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.0100000000000001e-06 2024-08-19T14:46:07.967 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.968 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.968 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:46:07.968 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.751923+0000", 2024-08-19T14:46:07.968 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.098e-05 2024-08-19T14:46:07.968 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.968 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.968 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:46:07.968 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.751930+0000", 2024-08-19T14:46:07.968 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.0600000000000002e-06 2024-08-19T14:46:07.968 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.968 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.968 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:46:07.968 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.752067+0000", 2024-08-19T14:46:07.969 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013742100000000001 2024-08-19T14:46:07.969 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.969 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.969 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:46:07.969 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.752089+0000", 2024-08-19T14:46:07.969 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.1389000000000001e-05 2024-08-19T14:46:07.969 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.969 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.969 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:46:07.969 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.766981+0000", 2024-08-19T14:46:07.969 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.014892396 2024-08-19T14:46:07.969 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.969 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.970 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:46:07.970 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.961408+0000", 2024-08-19T14:46:07.970 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.19442712400000001 2024-08-19T14:46:07.970 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:07.970 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:46:07.970 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:07.970 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.970 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.970 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.16962.0:199422 7.a e690/635 7:5ea5870d:::benchmark_data_smithi045_115842_object81262:head v 690'12384, mlcod=633'7611)", 2024-08-19T14:46:07.970 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:38:22.759906+0000", 2024-08-19T14:46:07.970 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 464.28756849600001, 2024-08-19T14:46:07.970 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.242679165, 2024-08-19T14:46:07.970 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:46:07.970 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:46:07.971 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:46:07.971 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:46:07.971 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.971 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:46:07.971 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.759906+0000", 2024-08-19T14:46:07.971 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:07.971 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.971 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.971 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:46:07.971 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.759906+0000", 2024-08-19T14:46:07.971 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:07.971 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.971 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.972 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:46:07.972 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.759908+0000", 2024-08-19T14:46:07.972 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5880000000000001e-06 2024-08-19T14:46:07.972 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.972 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.972 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:46:07.972 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.759916+0000", 2024-08-19T14:46:07.972 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.4950000000000008e-06 2024-08-19T14:46:07.972 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.972 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.972 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:46:07.972 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.759930+0000", 2024-08-19T14:46:07.972 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4236000000000001e-05 2024-08-19T14:46:07.972 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.973 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.973 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:46:07.973 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.759941+0000", 2024-08-19T14:46:07.973 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1052e-05 2024-08-19T14:46:07.973 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.973 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.973 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:46:07.973 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.765122+0000", 2024-08-19T14:46:07.973 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0051806439999999999 2024-08-19T14:46:07.973 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.973 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.973 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:46:07.973 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.765141+0000", 2024-08-19T14:46:07.974 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.8470000000000001e-05 2024-08-19T14:46:07.974 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.974 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.974 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:46:07.974 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:23.002541+0000", 2024-08-19T14:46:07.974 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.23740081599999999 2024-08-19T14:46:07.974 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.974 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.974 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:46:07.974 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:23.002585+0000", 2024-08-19T14:46:07.974 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.3863999999999999e-05 2024-08-19T14:46:07.974 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:07.974 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:46:07.974 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:07.975 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.975 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.975 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "MOSDPGPush(7.2 690/635 [PushOp(7:48749d7b:::benchmark_data_smithi045_115842_object81270:head, version: 621'5127, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:48749d7b:::benchmark_data_smithi045_115842_object81270:head@621'5127, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-19T14:46:07.975 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:38:22.764504+0000", 2024-08-19T14:46:07.975 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 464.28297033500002, 2024-08-19T14:46:07.975 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.20554665, 2024-08-19T14:46:07.975 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:46:07.975 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:46:07.975 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-08-19T14:46:07.975 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:46:07.975 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.975 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:46:07.975 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.764504+0000", 2024-08-19T14:46:07.976 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:07.976 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.976 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.976 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:46:07.976 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.764504+0000", 2024-08-19T14:46:07.976 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:07.976 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.976 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.976 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:46:07.976 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.764505+0000", 2024-08-19T14:46:07.976 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.6400000000000001e-07 2024-08-19T14:46:07.976 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.976 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.977 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:46:07.977 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.764536+0000", 2024-08-19T14:46:07.977 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.0719000000000002e-05 2024-08-19T14:46:07.977 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.977 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.977 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:46:07.977 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.764556+0000", 2024-08-19T14:46:07.977 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.0142000000000001e-05 2024-08-19T14:46:07.977 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.977 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.977 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:46:07.977 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.764573+0000", 2024-08-19T14:46:07.977 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6685999999999999e-05 2024-08-19T14:46:07.977 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.977 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.978 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:46:07.978 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.966878+0000", 2024-08-19T14:46:07.978 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.20230553300000001 2024-08-19T14:46:07.978 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.978 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.978 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:46:07.978 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.966888+0000", 2024-08-19T14:46:07.978 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.026e-05 2024-08-19T14:46:07.978 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.978 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.978 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:46:07.978 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.970051+0000", 2024-08-19T14:46:07.978 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.003162546 2024-08-19T14:46:07.979 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:07.979 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:46:07.979 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:07.979 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.979 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.979 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "MOSDPGRecoveryDelete(7.e e690,634 7:7b982ab6:::benchmark_data_smithi045_115842_object57353:head,673'11068)", 2024-08-19T14:46:07.979 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:38:22.767854+0000", 2024-08-19T14:46:07.979 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 464.27962052999999, 2024-08-19T14:46:07.979 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.19375594500000001, 2024-08-19T14:46:07.979 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:46:07.979 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:46:07.979 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-08-19T14:46:07.979 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:46:07.979 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.980 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:46:07.980 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.767854+0000", 2024-08-19T14:46:07.980 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:07.980 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.980 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.980 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:46:07.980 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.767854+0000", 2024-08-19T14:46:07.980 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:07.980 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.980 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.980 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:46:07.980 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.767855+0000", 2024-08-19T14:46:07.980 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2410000000000001e-06 2024-08-19T14:46:07.981 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.981 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.981 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:46:07.981 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.767872+0000", 2024-08-19T14:46:07.981 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6707000000000001e-05 2024-08-19T14:46:07.981 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.981 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.981 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:46:07.981 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.961428+0000", 2024-08-19T14:46:07.981 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.19355593600000001 2024-08-19T14:46:07.981 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.981 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.981 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:46:07.981 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.961441+0000", 2024-08-19T14:46:07.981 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3269e-05 2024-08-19T14:46:07.982 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.982 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.982 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:46:07.982 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.961526+0000", 2024-08-19T14:46:07.982 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.4942000000000001e-05 2024-08-19T14:46:07.982 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.982 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.982 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:46:07.982 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.961534+0000", 2024-08-19T14:46:07.982 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.6350000000000006e-06 2024-08-19T14:46:07.982 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.982 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.982 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:46:07.983 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.961610+0000", 2024-08-19T14:46:07.983 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.6215000000000001e-05 2024-08-19T14:46:07.983 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:07.983 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:46:07.983 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:07.983 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.983 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.983 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.16962.0:199399 7.3 e690/635 7:cecd84d1:::benchmark_data_smithi045_115842_object81239:head v 690'12702, mlcod=621'5174)", 2024-08-19T14:46:07.983 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:38:22.776198+0000", 2024-08-19T14:46:07.983 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 464.27127666400003, 2024-08-19T14:46:07.983 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.20653019, 2024-08-19T14:46:07.983 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:46:07.983 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:46:07.983 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:46:07.984 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:46:07.984 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.984 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:46:07.984 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.776198+0000", 2024-08-19T14:46:07.984 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:07.984 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.984 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.984 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:46:07.984 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.776198+0000", 2024-08-19T14:46:07.984 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:07.984 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.984 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.984 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:46:07.984 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.776199+0000", 2024-08-19T14:46:07.984 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2449999999999999e-06 2024-08-19T14:46:07.985 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.985 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.985 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:46:07.985 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.776210+0000", 2024-08-19T14:46:07.985 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1289e-05 2024-08-19T14:46:07.985 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.985 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.985 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:46:07.985 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.961366+0000", 2024-08-19T14:46:07.985 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.18515516700000001 2024-08-19T14:46:07.985 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.985 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.985 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:46:07.986 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.961387+0000", 2024-08-19T14:46:07.986 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.1248999999999999e-05 2024-08-19T14:46:07.987 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.987 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.987 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:46:07.987 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.961448+0000", 2024-08-19T14:46:07.987 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.1062000000000006e-05 2024-08-19T14:46:07.987 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.988 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.988 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:46:07.988 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.961480+0000", 2024-08-19T14:46:07.988 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.1775e-05 2024-08-19T14:46:07.988 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.988 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.989 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:46:07.989 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.982684+0000", 2024-08-19T14:46:07.989 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.021203914000000001 2024-08-19T14:46:07.989 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.989 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.989 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:46:07.990 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.982728+0000", 2024-08-19T14:46:07.990 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4489e-05 2024-08-19T14:46:07.990 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:07.990 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:46:07.990 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:07.990 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.990 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.991 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "MOSDPGPush(8.12 754/734 [PushOp(8:4b2adb7e:::benchmark_data_smithi045_137103_object65588:head, version: 730'4056, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:4b2adb7e:::benchmark_data_smithi045_137103_object65588:head@730'4056, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-19T14:46:07.991 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:41:28.711325+0000", 2024-08-19T14:46:07.991 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 278.33614944300001, 2024-08-19T14:46:07.991 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.142688281, 2024-08-19T14:46:07.991 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:46:07.991 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:46:07.991 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-08-19T14:46:07.992 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:46:07.992 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.992 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:46:07.992 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:28.711325+0000", 2024-08-19T14:46:07.992 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:07.992 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.992 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.992 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:46:07.993 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:28.711325+0000", 2024-08-19T14:46:07.993 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:07.993 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.993 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.993 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:46:07.993 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:28.711326+0000", 2024-08-19T14:46:07.994 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.95e-07 2024-08-19T14:46:07.994 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.994 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.995 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:46:07.995 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:28.711359+0000", 2024-08-19T14:46:07.995 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.3250000000000002e-05 2024-08-19T14:46:07.995 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.996 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.996 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:46:07.996 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:28.848011+0000", 2024-08-19T14:46:07.996 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.13665176800000001 2024-08-19T14:46:07.996 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.996 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.997 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:46:07.997 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:28.848519+0000", 2024-08-19T14:46:07.997 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00050756200000000003 2024-08-19T14:46:07.997 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.997 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.997 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:46:07.998 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:28.851404+0000", 2024-08-19T14:46:07.998 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.002885454 2024-08-19T14:46:07.998 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.998 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.998 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:46:07.999 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:28.851445+0000", 2024-08-19T14:46:07.999 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.0571999999999997e-05 2024-08-19T14:46:07.999 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:07.999 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:07.999 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:46:07.999 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:28.854013+0000", 2024-08-19T14:46:08.000 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0025686799999999998 2024-08-19T14:46:08.000 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:08.000 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:46:08.000 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:08.000 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.000 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.000 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.26911.0:112730 8.e e763/717 8:7b021f56:::benchmark_data_smithi045_137103_object266:head v 763'7120, mlcod=763'7118)", 2024-08-19T14:46:08.001 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:41:42.489992+0000", 2024-08-19T14:46:08.001 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 264.55748224000001, 2024-08-19T14:46:08.001 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.14127273100000001, 2024-08-19T14:46:08.001 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:46:08.001 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:46:08.001 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:46:08.001 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:46:08.002 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.002 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:46:08.002 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:42.489992+0000", 2024-08-19T14:46:08.002 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:08.002 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.002 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.003 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:46:08.003 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:42.489992+0000", 2024-08-19T14:46:08.003 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:08.003 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.003 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.003 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:46:08.004 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:42.489993+0000", 2024-08-19T14:46:08.004 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.5499999999999996e-07 2024-08-19T14:46:08.004 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.004 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.004 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:46:08.004 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:42.490000+0000", 2024-08-19T14:46:08.005 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.8040000000000004e-06 2024-08-19T14:46:08.005 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.005 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.005 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:46:08.006 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:42.490011+0000", 2024-08-19T14:46:08.006 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.064e-05 2024-08-19T14:46:08.006 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.006 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.006 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:46:08.006 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:42.490018+0000", 2024-08-19T14:46:08.007 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.2089999999999999e-06 2024-08-19T14:46:08.007 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.007 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.007 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:46:08.007 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:42.490071+0000", 2024-08-19T14:46:08.007 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.346e-05 2024-08-19T14:46:08.007 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.007 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.007 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:46:08.007 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:42.490091+0000", 2024-08-19T14:46:08.007 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.9123e-05 2024-08-19T14:46:08.007 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.007 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.008 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:46:08.008 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:42.631226+0000", 2024-08-19T14:46:08.008 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.141135608 2024-08-19T14:46:08.008 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.008 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.008 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:46:08.008 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:42.631265+0000", 2024-08-19T14:46:08.008 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.8931999999999999e-05 2024-08-19T14:46:08.008 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:08.008 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:46:08.009 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:08.009 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.009 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.009 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.26911.0:112731 8.f e763/717 8:ff6e92ec:::benchmark_data_smithi045_137103_object267:head v 763'7064, mlcod=763'7062)", 2024-08-19T14:46:08.009 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:41:42.490155+0000", 2024-08-19T14:46:08.009 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 264.55731971199998, 2024-08-19T14:46:08.009 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.14124139899999999, 2024-08-19T14:46:08.009 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:46:08.009 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:46:08.009 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:46:08.009 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:46:08.009 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.009 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:46:08.010 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:42.490155+0000", 2024-08-19T14:46:08.010 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:08.010 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.010 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.010 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:46:08.010 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:42.490155+0000", 2024-08-19T14:46:08.010 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:08.010 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.010 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.010 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:46:08.010 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:42.490155+0000", 2024-08-19T14:46:08.010 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.1399999999999997e-07 2024-08-19T14:46:08.010 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.011 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.011 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:46:08.011 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:42.490161+0000", 2024-08-19T14:46:08.011 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0839999999999999e-06 2024-08-19T14:46:08.011 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.011 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.011 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:46:08.011 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:42.490170+0000", 2024-08-19T14:46:08.011 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.1349999999999998e-06 2024-08-19T14:46:08.011 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.011 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.011 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:46:08.011 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:42.490175+0000", 2024-08-19T14:46:08.011 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.587e-06 2024-08-19T14:46:08.012 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.012 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.012 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:46:08.012 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:42.490225+0000", 2024-08-19T14:46:08.012 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.9937e-05 2024-08-19T14:46:08.012 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.012 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.012 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:46:08.012 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:42.490244+0000", 2024-08-19T14:46:08.012 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.9094e-05 2024-08-19T14:46:08.012 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.012 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.012 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:46:08.012 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:42.631352+0000", 2024-08-19T14:46:08.013 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.14110751199999999 2024-08-19T14:46:08.013 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.013 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.013 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:46:08.013 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:42.631396+0000", 2024-08-19T14:46:08.013 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4435999999999999e-05 2024-08-19T14:46:08.013 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:08.013 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:46:08.013 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:08.013 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.014 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.014 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.26911.0:112733 8.d e763/717 8:b1f595a5:::benchmark_data_smithi045_137103_object269:head v 763'7096, mlcod=763'7094)", 2024-08-19T14:46:08.014 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:41:42.493271+0000", 2024-08-19T14:46:08.014 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 264.554204035, 2024-08-19T14:46:08.014 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.14344847499999999, 2024-08-19T14:46:08.014 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:46:08.014 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:46:08.014 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:46:08.014 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:46:08.014 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.014 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:46:08.014 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:42.493271+0000", 2024-08-19T14:46:08.015 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:08.015 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.015 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.015 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:46:08.015 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:42.493271+0000", 2024-08-19T14:46:08.015 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:08.015 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.015 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.015 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:46:08.015 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:42.493271+0000", 2024-08-19T14:46:08.016 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.5799999999999999e-07 2024-08-19T14:46:08.016 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.016 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.016 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:46:08.016 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:42.493277+0000", 2024-08-19T14:46:08.016 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.3890000000000004e-06 2024-08-19T14:46:08.016 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.016 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.016 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:46:08.016 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:42.493285+0000", 2024-08-19T14:46:08.016 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.3939999999999996e-06 2024-08-19T14:46:08.017 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.017 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.017 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:46:08.017 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:42.493292+0000", 2024-08-19T14:46:08.017 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.596e-06 2024-08-19T14:46:08.017 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.017 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.017 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:46:08.017 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:42.631429+0000", 2024-08-19T14:46:08.017 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.13813727100000001 2024-08-19T14:46:08.017 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.017 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.017 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:46:08.017 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:42.631449+0000", 2024-08-19T14:46:08.018 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.0353e-05 2024-08-19T14:46:08.018 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.018 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.018 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:46:08.018 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:42.636691+0000", 2024-08-19T14:46:08.018 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0052417139999999997 2024-08-19T14:46:08.018 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.018 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.018 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:46:08.018 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:42.636719+0000", 2024-08-19T14:46:08.018 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.8099999999999999e-05 2024-08-19T14:46:08.018 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:08.018 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:46:08.019 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:08.019 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.019 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.019 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.27235.0:70472 9.1 9:8eb9e7f2:::benchmark_data_smithi045_152862_object70471:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e864)", 2024-08-19T14:46:08.019 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:45:24.130851+0000", 2024-08-19T14:46:08.019 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 42.916623520000002, 2024-08-19T14:46:08.019 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.42378770399999999, 2024-08-19T14:46:08.019 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:46:08.019 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:46:08.019 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:46:08.019 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:46:08.019 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.27235", 2024-08-19T14:46:08.019 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2746512116", 2024-08-19T14:46:08.019 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 70472 2024-08-19T14:46:08.020 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.020 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:46:08.020 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.020 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:46:08.020 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.130851+0000", 2024-08-19T14:46:08.020 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:08.020 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.020 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.020 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:46:08.020 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.130851+0000", 2024-08-19T14:46:08.020 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:08.020 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.020 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.021 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:46:08.021 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.130853+0000", 2024-08-19T14:46:08.021 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.759e-06 2024-08-19T14:46:08.021 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.021 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.021 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:46:08.021 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.131053+0000", 2024-08-19T14:46:08.021 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000200118 2024-08-19T14:46:08.021 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.021 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.021 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:46:08.021 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.131079+0000", 2024-08-19T14:46:08.021 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.5715e-05 2024-08-19T14:46:08.022 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.022 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.022 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:46:08.022 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.131092+0000", 2024-08-19T14:46:08.022 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2952999999999999e-05 2024-08-19T14:46:08.022 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.022 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.022 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:46:08.022 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.132073+0000", 2024-08-19T14:46:08.022 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0009816829999999999 2024-08-19T14:46:08.022 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.022 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.023 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:46:08.023 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.132084+0000", 2024-08-19T14:46:08.023 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0973e-05 2024-08-19T14:46:08.023 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.023 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.023 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:46:08.023 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.553167+0000", 2024-08-19T14:46:08.023 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.42108286900000003 2024-08-19T14:46:08.023 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.023 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.023 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:46:08.023 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.554639+0000", 2024-08-19T14:46:08.023 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0014716340000000001 2024-08-19T14:46:08.024 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:08.024 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:46:08.024 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:08.024 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.024 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.024 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.27235.0:70476 9.1 9:81ce3d4f:::benchmark_data_smithi045_152862_object70475:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e864)", 2024-08-19T14:46:08.024 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:45:24.131125+0000", 2024-08-19T14:46:08.024 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 42.916349912999998, 2024-08-19T14:46:08.024 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.42668378600000001, 2024-08-19T14:46:08.024 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:46:08.024 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:46:08.024 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:46:08.024 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:46:08.025 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.27235", 2024-08-19T14:46:08.025 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2746512116", 2024-08-19T14:46:08.025 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 70476 2024-08-19T14:46:08.025 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.025 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:46:08.025 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.025 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:46:08.025 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.131125+0000", 2024-08-19T14:46:08.025 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:08.025 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.025 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.025 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:46:08.025 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.131125+0000", 2024-08-19T14:46:08.026 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:08.026 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.026 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.026 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:46:08.026 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.131125+0000", 2024-08-19T14:46:08.026 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.99e-07 2024-08-19T14:46:08.026 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.026 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.026 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:46:08.026 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.131196+0000", 2024-08-19T14:46:08.026 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.038e-05 2024-08-19T14:46:08.026 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.026 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.026 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:46:08.027 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.131206+0000", 2024-08-19T14:46:08.027 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.049e-05 2024-08-19T14:46:08.027 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.027 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.027 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:46:08.027 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.131213+0000", 2024-08-19T14:46:08.027 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.1010000000000003e-06 2024-08-19T14:46:08.027 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.027 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.027 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:46:08.027 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.132119+0000", 2024-08-19T14:46:08.027 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00090598899999999995 2024-08-19T14:46:08.027 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.027 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.028 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:46:08.028 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.132127+0000", 2024-08-19T14:46:08.028 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.2980000000000003e-06 2024-08-19T14:46:08.028 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.028 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.028 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:46:08.028 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.557684+0000", 2024-08-19T14:46:08.028 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.42555691099999998 2024-08-19T14:46:08.028 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.028 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.028 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:46:08.028 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.557808+0000", 2024-08-19T14:46:08.029 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012491800000000001 2024-08-19T14:46:08.029 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:08.029 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:46:08.029 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:08.029 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.029 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.029 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.27235.0:70485 9.1 9:80f87f5d:::benchmark_data_smithi045_152862_object70484:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e864)", 2024-08-19T14:46:08.029 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:45:24.131225+0000", 2024-08-19T14:46:08.029 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 42.916249790999998, 2024-08-19T14:46:08.029 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.42675184500000002, 2024-08-19T14:46:08.029 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:46:08.029 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:46:08.029 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:46:08.030 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:46:08.030 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.27235", 2024-08-19T14:46:08.030 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2746512116", 2024-08-19T14:46:08.030 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 70485 2024-08-19T14:46:08.030 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.030 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:46:08.030 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.030 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:46:08.030 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.131225+0000", 2024-08-19T14:46:08.030 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:08.030 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.030 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.030 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:46:08.031 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.131225+0000", 2024-08-19T14:46:08.031 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:08.031 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.031 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.031 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:46:08.031 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.131225+0000", 2024-08-19T14:46:08.031 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.5499999999999998e-07 2024-08-19T14:46:08.031 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.031 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.031 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:46:08.031 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.131290+0000", 2024-08-19T14:46:08.031 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.5104999999999994e-05 2024-08-19T14:46:08.031 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.031 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.032 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:46:08.032 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.131300+0000", 2024-08-19T14:46:08.032 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.5079999999999994e-06 2024-08-19T14:46:08.032 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.032 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.032 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:46:08.032 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.131306+0000", 2024-08-19T14:46:08.032 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.5320000000000003e-06 2024-08-19T14:46:08.032 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.032 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.032 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:46:08.032 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.132152+0000", 2024-08-19T14:46:08.032 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00084525799999999999 2024-08-19T14:46:08.032 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.033 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.033 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:46:08.033 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.132159+0000", 2024-08-19T14:46:08.033 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.7489999999999998e-06 2024-08-19T14:46:08.033 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.033 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.033 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:46:08.033 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.557856+0000", 2024-08-19T14:46:08.033 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.42569648100000002 2024-08-19T14:46:08.033 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.033 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.033 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:46:08.033 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.557977+0000", 2024-08-19T14:46:08.033 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012065699999999999 2024-08-19T14:46:08.034 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:08.034 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:46:08.034 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:08.034 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.034 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.034 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.27235.0:104455 9.1 9:89d1f275:::benchmark_data_smithi045_152862_object9950:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e872)", 2024-08-19T14:46:08.034 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:46:03.697722+0000", 2024-08-19T14:46:08.034 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 3.3497528270000001, 2024-08-19T14:46:08.034 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.089424409999999996, 2024-08-19T14:46:08.034 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:46:08.034 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:46:08.034 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:46:08.035 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:46:08.035 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.27235", 2024-08-19T14:46:08.035 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2746512116", 2024-08-19T14:46:08.035 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 104455 2024-08-19T14:46:08.035 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.035 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:46:08.035 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.035 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:46:08.035 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:03.697722+0000", 2024-08-19T14:46:08.035 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:08.035 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.035 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.035 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:46:08.035 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:03.697722+0000", 2024-08-19T14:46:08.036 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:08.036 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.036 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.036 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:46:08.036 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:03.697723+0000", 2024-08-19T14:46:08.036 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.8999999999999995e-07 2024-08-19T14:46:08.036 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.036 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.036 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:46:08.036 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:03.697727+0000", 2024-08-19T14:46:08.036 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.6820000000000002e-06 2024-08-19T14:46:08.036 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.037 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.037 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:46:08.037 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:03.697739+0000", 2024-08-19T14:46:08.037 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2167999999999999e-05 2024-08-19T14:46:08.037 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.037 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.037 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:46:08.037 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:03.697746+0000", 2024-08-19T14:46:08.037 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.5490000000000003e-06 2024-08-19T14:46:08.037 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.037 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.037 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:46:08.037 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:03.702904+0000", 2024-08-19T14:46:08.038 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0051579989999999999 2024-08-19T14:46:08.038 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.038 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.038 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:46:08.038 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:03.703050+0000", 2024-08-19T14:46:08.038 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00014614700000000001 2024-08-19T14:46:08.038 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.038 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.038 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 6,9,10", 2024-08-19T14:46:08.038 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:03.703186+0000", 2024-08-19T14:46:08.038 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013591499999999999 2024-08-19T14:46:08.038 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.038 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.039 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:46:08.039 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:03.709259+0000", 2024-08-19T14:46:08.039 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0060729850000000004 2024-08-19T14:46:08.039 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.039 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.039 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:46:08.039 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:03.709292+0000", 2024-08-19T14:46:08.039 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.2758e-05 2024-08-19T14:46:08.039 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.039 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.039 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:46:08.039 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:03.716094+0000", 2024-08-19T14:46:08.040 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0068022830000000001 2024-08-19T14:46:08.040 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.040 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.040 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:46:08.040 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:03.785650+0000", 2024-08-19T14:46:08.040 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.069556061000000002 2024-08-19T14:46:08.040 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.040 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.040 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:46:08.040 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:03.785714+0000", 2024-08-19T14:46:08.040 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.3733999999999997e-05 2024-08-19T14:46:08.040 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.040 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.041 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:46:08.041 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:03.787146+0000", 2024-08-19T14:46:08.041 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0014323389999999999 2024-08-19T14:46:08.041 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:08.041 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:46:08.041 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:08.041 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.041 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.041 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.27235.0:104747 9.1 9:8d2ddaee:::benchmark_data_smithi045_152862_object10242:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e872)", 2024-08-19T14:46:08.041 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:46:04.029673+0000", 2024-08-19T14:46:08.041 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 3.0178012340000002, 2024-08-19T14:46:08.041 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.055338624000000003, 2024-08-19T14:46:08.042 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:46:08.042 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:46:08.042 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:46:08.042 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:46:08.042 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.27235", 2024-08-19T14:46:08.042 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2746512116", 2024-08-19T14:46:08.042 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 104747 2024-08-19T14:46:08.042 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.042 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:46:08.042 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.042 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:46:08.042 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:04.029673+0000", 2024-08-19T14:46:08.042 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:08.043 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.043 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.043 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:46:08.043 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:04.029673+0000", 2024-08-19T14:46:08.043 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:08.043 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.043 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.043 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:46:08.043 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:04.029675+0000", 2024-08-19T14:46:08.043 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2759999999999999e-06 2024-08-19T14:46:08.043 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.043 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.043 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:46:08.044 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:04.029683+0000", 2024-08-19T14:46:08.044 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.7520000000000002e-06 2024-08-19T14:46:08.044 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.044 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.044 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:46:08.044 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:04.029698+0000", 2024-08-19T14:46:08.044 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.473e-05 2024-08-19T14:46:08.044 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.044 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.044 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:46:08.044 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:04.029704+0000", 2024-08-19T14:46:08.044 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.9900000000000002e-06 2024-08-19T14:46:08.044 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.045 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.045 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:46:08.045 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:04.035452+0000", 2024-08-19T14:46:08.045 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0057483179999999997 2024-08-19T14:46:08.045 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.045 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.045 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:46:08.045 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:04.035564+0000", 2024-08-19T14:46:08.045 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000112067 2024-08-19T14:46:08.045 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.045 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.045 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 6,9,10", 2024-08-19T14:46:08.045 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:04.035655+0000", 2024-08-19T14:46:08.045 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.0198999999999995e-05 2024-08-19T14:46:08.046 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.046 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.046 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:46:08.046 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:04.041778+0000", 2024-08-19T14:46:08.046 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0061228910000000001 2024-08-19T14:46:08.046 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.046 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.046 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:46:08.046 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:04.041792+0000", 2024-08-19T14:46:08.046 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4406000000000001e-05 2024-08-19T14:46:08.046 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.046 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.046 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:46:08.046 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:04.046711+0000", 2024-08-19T14:46:08.047 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0049189150000000003 2024-08-19T14:46:08.047 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.047 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.047 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:46:08.047 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:04.083521+0000", 2024-08-19T14:46:08.047 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.036809916999999998 2024-08-19T14:46:08.047 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.047 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.047 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:46:08.047 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:04.083591+0000", 2024-08-19T14:46:08.047 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.9789000000000006e-05 2024-08-19T14:46:08.047 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:08.047 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:08.047 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:46:08.048 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:04.085012+0000", 2024-08-19T14:46:08.048 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.001421374 2024-08-19T14:46:08.048 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:08.048 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:46:08.048 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:08.048 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:08.048 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:46:08.048 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:46:08.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:08.050 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:08.050 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:46:08.151 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:46:08.178 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:08.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:08.179 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:46:08.280 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:46:08.301 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:08.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:08.303 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:46:08.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:08 smithi031 ceph-mon[16336]: pgmap v2059: 47 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+remapped, 42 active+clean; 5.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 12871/251406 objects misplaced (5.120%); 21 MiB/s, 337 objects/s recovering 2024-08-19T14:46:08.403 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:46:08.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:08.441 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:08.441 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:46:08.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:08 smithi032 ceph-mon[20004]: pgmap v2059: 47 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+remapped, 42 active+clean; 5.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 12871/251406 objects misplaced (5.120%); 21 MiB/s, 337 objects/s recovering 2024-08-19T14:46:08.542 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:46:08.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:08 smithi045 ceph-mon[20143]: pgmap v2059: 47 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+remapped, 42 active+clean; 5.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 12871/251406 objects misplaced (5.120%); 21 MiB/s, 337 objects/s recovering 2024-08-19T14:46:08.571 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:08.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:08.572 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:46:08.673 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:46:08.697 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:08.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:08.698 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:46:08.799 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:46:08.825 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:08.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:08.826 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:46:08.927 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:46:08.951 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:08.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:08.952 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:46:09.053 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:46:09.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:09.076 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:09.076 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:46:09.177 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:46:09.207 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:09.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:09.208 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:46:09.309 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:46:09.350 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:09.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:09.351 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:46:09.452 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:46:09.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:09.498 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:09.499 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:46:09.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:09 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:46:09.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:09 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:46:09.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:09 smithi032 ceph-mon[20004]: 1.2 scrub starts 2024-08-19T14:46:09.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:09 smithi032 ceph-mon[20004]: 1.2 scrub ok 2024-08-19T14:46:09.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:09 smithi032 ceph-mon[20004]: osdmap e874: 12 total, 12 up, 8 in 2024-08-19T14:46:09.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:09 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:46:09.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:09 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:46:09.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:09 smithi045 ceph-mon[20143]: 1.2 scrub starts 2024-08-19T14:46:09.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:09 smithi045 ceph-mon[20143]: 1.2 scrub ok 2024-08-19T14:46:09.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:09 smithi045 ceph-mon[20143]: osdmap e874: 12 total, 12 up, 8 in 2024-08-19T14:46:09.600 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:46:09.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:09 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:46:09.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:09 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:46:09.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:09 smithi031 ceph-mon[16336]: 1.2 scrub starts 2024-08-19T14:46:09.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:09 smithi031 ceph-mon[16336]: 1.2 scrub ok 2024-08-19T14:46:09.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:09 smithi031 ceph-mon[16336]: osdmap e874: 12 total, 12 up, 8 in 2024-08-19T14:46:09.623 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:09.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:09.624 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:46:09.725 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:46:09.754 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:09.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:09.756 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:46:09.856 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:46:09.886 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:09.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:09.887 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:46:09.989 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:46:10.015 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:10.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:10.016 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:46:10.117 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:46:10.144 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:10.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:10.145 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:46:10.246 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:46:10.277 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:10.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:10.278 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:46:10.379 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:46:10.416 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:10.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:10.417 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:46:10.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:10 smithi032 ceph-mon[20004]: pgmap v2061: 47 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+remapped, 42 active+clean; 5.1 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 832 op/s; 12871/249756 objects misplaced (5.153%); 17 MiB/s, 276 objects/s recovering 2024-08-19T14:46:10.518 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:46:10.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:10 smithi045 ceph-mon[20143]: pgmap v2061: 47 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+remapped, 42 active+clean; 5.1 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 832 op/s; 12871/249756 objects misplaced (5.153%); 17 MiB/s, 276 objects/s recovering 2024-08-19T14:46:10.558 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:10.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:10.559 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:46:10.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:10 smithi031 ceph-mon[16336]: pgmap v2061: 47 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+remapped, 42 active+clean; 5.1 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 832 op/s; 12871/249756 objects misplaced (5.153%); 17 MiB/s, 276 objects/s recovering 2024-08-19T14:46:10.659 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:46:10.684 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:10.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:10.685 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:46:10.786 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:46:10.824 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:10.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:10.825 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:46:10.926 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:46:10.953 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:10.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:10.954 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:46:11.055 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:46:11.081 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:11.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:11.082 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:46:11.182 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:46:11.207 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:11.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:11.208 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:46:11.308 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:46:11.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:11.336 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:11.336 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:46:11.437 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:46:11.467 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:11.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:11.468 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:46:11.569 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:46:11.596 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:11.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:11.597 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:46:11.698 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:46:11.735 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:11.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:11.736 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:46:11.837 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:46:11.866 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:11.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:11.867 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:46:11.968 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:46:11.997 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:11.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:11.998 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:46:12.099 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:46:12.127 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:12.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:12.128 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:46:12.229 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:46:12.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:12.270 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:12.270 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:46:12.372 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:46:12.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:12.407 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:12.407 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:46:12.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:12 smithi032 ceph-mon[20004]: pgmap v2062: 47 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+remapped, 42 active+clean; 5.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 843 op/s; 12304/244074 objects misplaced (5.041%); 15 MiB/s, 243 objects/s recovering 2024-08-19T14:46:12.508 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:46:12.530 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:12.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:12.531 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:46:12.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:12 smithi045 ceph-mon[20143]: pgmap v2062: 47 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+remapped, 42 active+clean; 5.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 843 op/s; 12304/244074 objects misplaced (5.041%); 15 MiB/s, 243 objects/s recovering 2024-08-19T14:46:12.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:12 smithi031 ceph-mon[16336]: pgmap v2062: 47 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+remapped, 42 active+clean; 5.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 843 op/s; 12304/244074 objects misplaced (5.041%); 15 MiB/s, 243 objects/s recovering 2024-08-19T14:46:12.632 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:46:12.663 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:12.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:12.664 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:46:12.765 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:46:12.804 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:12.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:12.805 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:46:12.906 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:46:12.929 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:12.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:12.930 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:46:13.031 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:46:13.060 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:13.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:13.061 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:46:13.162 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:46:13.187 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:13.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:13.188 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:46:13.289 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:46:13.314 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:13.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:13.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:46:13.416 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:46:13.457 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:13.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:13.458 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:46:13.559 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:46:13.581 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:13.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:13.582 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:46:13.683 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:46:13.725 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:13.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:13.726 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:46:13.827 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:46:13.859 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:13.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:13.860 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:46:13.961 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:46:13.988 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:13.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:13.989 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:46:14.090 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:46:14.121 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:14.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:14.123 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:46:14.224 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:46:14.247 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:14.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:14.248 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:46:14.349 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:46:14.376 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:14.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:14.377 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:46:14.479 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:46:14.503 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:14.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:14.505 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:46:14.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:14 smithi032 ceph-mon[20004]: pgmap v2063: 47 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+remapped, 42 active+clean; 4.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 781 op/s; 11366/238398 objects misplaced (4.768%); 18 MiB/s, 287 objects/s recovering 2024-08-19T14:46:14.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:14 smithi045 ceph-mon[20143]: pgmap v2063: 47 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+remapped, 42 active+clean; 4.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 781 op/s; 11366/238398 objects misplaced (4.768%); 18 MiB/s, 287 objects/s recovering 2024-08-19T14:46:14.605 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:46:14.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:14 smithi031 ceph-mon[16336]: pgmap v2063: 47 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+remapped, 42 active+clean; 4.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 781 op/s; 11366/238398 objects misplaced (4.768%); 18 MiB/s, 287 objects/s recovering 2024-08-19T14:46:14.652 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:14.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:14.654 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:46:14.755 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:46:14.782 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:14.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:14.783 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:46:14.884 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:46:14.912 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:14.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:14.913 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:46:15.014 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:46:15.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:15.054 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:15.054 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:46:15.155 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:46:15.172 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd reweight 4 1 2024-08-19T14:46:15.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:15.188 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:15.188 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:46:15.290 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:46:15.316 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:15.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:15.317 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:46:15.418 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:46:15.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:15.450 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:15.450 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:46:15.552 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:46:15.584 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:15.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:15.585 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:46:15.686 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:46:15.712 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:15.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:15.713 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:46:15.814 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:46:15.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:15.839 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:15.840 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:46:15.935 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-08-19T14:46:15.940 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:46:15.974 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:15.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:15.975 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:46:15.988 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:46:16.039 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:46:16.075 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:46:16.099 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:16.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:16.100 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:46:16.201 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:46:16.225 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:16.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:16.226 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:46:16.327 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:46:16.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:16.360 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:16.360 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:46:16.461 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:46:16.474 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:16 smithi031 ceph-mon[16336]: pgmap v2064: 47 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+remapped, 42 active+clean; 4.7 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 846 op/s; 10594/232296 objects misplaced (4.561%); 20 MiB/s, 314 objects/s recovering 2024-08-19T14:46:16.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:16.496 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:16.496 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:46:16.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:16 smithi032 ceph-mon[20004]: pgmap v2064: 47 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+remapped, 42 active+clean; 4.7 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 846 op/s; 10594/232296 objects misplaced (4.561%); 20 MiB/s, 314 objects/s recovering 2024-08-19T14:46:16.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:16 smithi045 ceph-mon[20143]: pgmap v2064: 47 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+remapped, 42 active+clean; 4.7 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 846 op/s; 10594/232296 objects misplaced (4.561%); 20 MiB/s, 314 objects/s recovering 2024-08-19T14:46:16.597 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:46:16.620 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:16.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:16.621 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:46:16.652 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:46:16.721 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:46:16.743 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:16.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:16.744 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:46:16.845 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:46:16.869 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:16.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:16.870 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:46:16.971 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:46:16.993 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:16.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:16.995 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:46:17.096 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:46:17.119 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:17.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:17.121 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:46:17.204 INFO:teuthology.orchestra.run.smithi031.stderr:reweighted osd.4 to 1 (10000) 2024-08-19T14:46:17.222 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:46:17.258 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:17.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:17.259 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:46:17.316 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:46:17.316 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:46:17.352 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:46:17.352 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:46:17.360 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:46:17.373 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:46:17.374 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:46:17.389 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:17.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:17.390 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:46:17.402 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:17 smithi031 ceph-mon[16336]: 1.1a scrub starts 2024-08-19T14:46:17.402 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:17 smithi031 ceph-mon[16336]: 1.1a scrub ok 2024-08-19T14:46:17.402 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:17 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/934245997' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-08-19T14:46:17.402 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:17 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-08-19T14:46:17.402 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:46:17.403 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:46:17.427 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:46:17.427 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:46:17.449 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:46:17.449 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:46:17.470 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:46:17.470 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:46:17.491 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:46:17.492 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:46:17.492 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:46:17.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:17 smithi032 ceph-mon[20004]: 1.1a scrub starts 2024-08-19T14:46:17.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:17 smithi032 ceph-mon[20004]: 1.1a scrub ok 2024-08-19T14:46:17.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:17 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/934245997' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-08-19T14:46:17.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:17 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-08-19T14:46:17.512 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:17.513 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:46:17.513 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:46:17.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:17.514 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:46:17.533 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:46:17.533 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:46:17.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:17 smithi045 ceph-mon[20143]: 1.1a scrub starts 2024-08-19T14:46:17.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:17 smithi045 ceph-mon[20143]: 1.1a scrub ok 2024-08-19T14:46:17.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:17 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/934245997' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-08-19T14:46:17.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:17 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-08-19T14:46:17.589 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-19T14:46:17.615 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:46:17.637 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:17.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:17.638 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:46:17.740 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:46:17.752 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:46:17.753 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:46:17.763 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:17.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:17.764 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:46:17.771 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:46:17.772 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:46:17.864 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:46:17.887 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:17.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:17.888 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:46:17.989 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:46:18.011 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:18.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:18.012 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:46:18.113 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:46:18.136 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:18.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:18.137 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:46:18.238 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:46:18.264 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:18.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:18.265 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:46:18.366 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:46:18.396 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:18.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:18.397 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:46:18.498 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:46:18.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:18 smithi032 ceph-mon[20004]: pgmap v2065: 47 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+remapped, 42 active+clean; 4.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 843 op/s; 9422/226104 objects misplaced (4.167%); 22 MiB/s, 344 objects/s recovering 2024-08-19T14:46:18.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:18 smithi032 ceph-mon[20004]: 1.e scrub starts 2024-08-19T14:46:18.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:18 smithi032 ceph-mon[20004]: 1.e scrub ok 2024-08-19T14:46:18.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:18 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-08-19T14:46:18.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:18 smithi032 ceph-mon[20004]: osdmap e875: 12 total, 12 up, 8 in 2024-08-19T14:46:18.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:18 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:46:18.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:18 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:46:18.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:18 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2280986910' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:46:18.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:18 smithi032 ceph-mon[20004]: osdmap e876: 12 total, 12 up, 8 in 2024-08-19T14:46:18.525 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:18.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:18.526 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:46:18.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:18 smithi045 ceph-mon[20143]: pgmap v2065: 47 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+remapped, 42 active+clean; 4.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 843 op/s; 9422/226104 objects misplaced (4.167%); 22 MiB/s, 344 objects/s recovering 2024-08-19T14:46:18.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:18 smithi045 ceph-mon[20143]: 1.e scrub starts 2024-08-19T14:46:18.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:18 smithi045 ceph-mon[20143]: 1.e scrub ok 2024-08-19T14:46:18.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:18 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-08-19T14:46:18.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:18 smithi045 ceph-mon[20143]: osdmap e875: 12 total, 12 up, 8 in 2024-08-19T14:46:18.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:18 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:46:18.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:18 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:46:18.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:18 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2280986910' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:46:18.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:18 smithi045 ceph-mon[20143]: osdmap e876: 12 total, 12 up, 8 in 2024-08-19T14:46:18.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:18 smithi031 ceph-mon[16336]: pgmap v2065: 47 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+remapped, 42 active+clean; 4.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 843 op/s; 9422/226104 objects misplaced (4.167%); 22 MiB/s, 344 objects/s recovering 2024-08-19T14:46:18.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:18 smithi031 ceph-mon[16336]: 1.e scrub starts 2024-08-19T14:46:18.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:18 smithi031 ceph-mon[16336]: 1.e scrub ok 2024-08-19T14:46:18.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:18 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-08-19T14:46:18.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:18 smithi031 ceph-mon[16336]: osdmap e875: 12 total, 12 up, 8 in 2024-08-19T14:46:18.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:18 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:46:18.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:18 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:46:18.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:18 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2280986910' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:46:18.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:18 smithi031 ceph-mon[16336]: osdmap e876: 12 total, 12 up, 8 in 2024-08-19T14:46:18.626 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:46:18.665 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:18.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:18.666 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:46:18.767 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:46:18.795 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:18.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:18.796 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:46:18.897 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:46:18.940 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:18.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:18.941 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:46:18.949 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd reweight 8 1 2024-08-19T14:46:19.042 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:46:19.089 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:19.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:19.090 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:46:19.191 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:46:19.214 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:19.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:19.215 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:46:19.316 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:46:19.351 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:19.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:19.352 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:46:19.453 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:46:19.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:19.493 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:19.493 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:46:19.594 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:46:19.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:19.623 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:19.624 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:46:19.724 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:46:19.748 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:19.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:19.749 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:46:19.849 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:46:19.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:19.888 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:19.888 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:46:19.989 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:46:20.023 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:20.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:20.024 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:46:20.107 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:46:20.108 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:46:20.125 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:46:20.157 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:20.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:20.158 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:46:20.259 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:46:20.281 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:20.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:20.283 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:46:20.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:20 smithi031 ceph-mon[16336]: pgmap v2068: 47 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+remapped, 42 active+clean; 4.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 814 op/s; 9422/224511 objects misplaced (4.197%); 23 MiB/s, 360 objects/s recovering 2024-08-19T14:46:20.383 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:46:20.410 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:20.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:20.411 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:46:20.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:20 smithi032 ceph-mon[20004]: pgmap v2068: 47 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+remapped, 42 active+clean; 4.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 814 op/s; 9422/224511 objects misplaced (4.197%); 23 MiB/s, 360 objects/s recovering 2024-08-19T14:46:20.512 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:46:20.546 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:20.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:20 smithi045 ceph-mon[20143]: pgmap v2068: 47 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+remapped, 42 active+clean; 4.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 814 op/s; 9422/224511 objects misplaced (4.197%); 23 MiB/s, 360 objects/s recovering 2024-08-19T14:46:20.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:20.548 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:46:20.648 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:46:20.672 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:20.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:20.673 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:46:20.773 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:46:20.819 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-08-19T14:46:20.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:20.820 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:20.820 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:46:20.870 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:46:20.909 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:46:20.921 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:46:20.956 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:20.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:20.957 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:46:21.058 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:46:21.081 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:21.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:21.082 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:46:21.183 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:46:21.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:21.216 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:21.216 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:46:21.317 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:46:21.340 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:21.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:21.341 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:46:21.350 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:21 smithi031 ceph-mon[16336]: 1.d scrub starts 2024-08-19T14:46:21.350 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:21 smithi031 ceph-mon[16336]: 1.d scrub ok 2024-08-19T14:46:21.442 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:46:21.467 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:21.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:21.468 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:46:21.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:21 smithi032 ceph-mon[20004]: 1.d scrub starts 2024-08-19T14:46:21.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:21 smithi032 ceph-mon[20004]: 1.d scrub ok 2024-08-19T14:46:21.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:21 smithi045 ceph-mon[20143]: 1.d scrub starts 2024-08-19T14:46:21.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:21 smithi045 ceph-mon[20143]: 1.d scrub ok 2024-08-19T14:46:21.569 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:46:21.596 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:21.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:21.598 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:46:21.699 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:46:21.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:21.742 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:21.742 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:46:21.844 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:46:21.856 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:46:21.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:21.875 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:21.875 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:46:21.949 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:46:21.976 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:46:22.000 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:22.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:22.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:46:22.102 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:46:22.124 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:22.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:22.125 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:46:22.218 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is unset 2024-08-19T14:46:22.226 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:46:22.264 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:22.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:22.265 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:46:22.366 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:46:22.390 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:22.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:22.391 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:46:22.492 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:46:22.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:22 smithi032 ceph-mon[20004]: pgmap v2069: 47 pgs: 4 active+recovering+undersized+remapped, 43 active+clean; 4.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 789 op/s; 8193/219453 objects misplaced (3.733%); 25 MiB/s, 396 objects/s recovering 2024-08-19T14:46:22.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:22 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2731560141' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:46:22.535 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:22.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:22.536 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:46:22.540 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:22 smithi031 ceph-mon[16336]: pgmap v2069: 47 pgs: 4 active+recovering+undersized+remapped, 43 active+clean; 4.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 789 op/s; 8193/219453 objects misplaced (3.733%); 25 MiB/s, 396 objects/s recovering 2024-08-19T14:46:22.540 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:22 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2731560141' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:46:22.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:22 smithi045 ceph-mon[20143]: pgmap v2069: 47 pgs: 4 active+recovering+undersized+remapped, 43 active+clean; 4.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 789 op/s; 8193/219453 objects misplaced (3.733%); 25 MiB/s, 396 objects/s recovering 2024-08-19T14:46:22.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:22 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2731560141' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:46:22.637 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:46:22.659 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:22.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:22.660 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:46:22.761 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:46:22.788 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:22.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:22.789 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:46:22.890 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:46:22.913 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:22.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:22.914 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:46:23.015 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:46:23.037 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:23.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:23.038 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:46:23.138 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:46:23.139 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:46:23.139 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:46:23.177 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:23.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:23.178 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:46:23.225 INFO:teuthology.orchestra.run.smithi031.stderr:reweighted osd.8 to 1 (10000) 2024-08-19T14:46:23.279 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:46:23.305 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:23.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:23.306 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:46:23.353 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:23 smithi031 ceph-mon[16336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:46:23.354 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:23 smithi031 ceph-mon[16336]: Cluster is now healthy 2024-08-19T14:46:23.354 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:23 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2731560141' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:46:23.354 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:23 smithi031 ceph-mon[16336]: osdmap e877: 12 total, 12 up, 8 in 2024-08-19T14:46:23.354 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:23 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2922681715' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-08-19T14:46:23.354 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:23 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:46:23.354 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:23 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:46:23.407 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:46:23.433 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:23.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:23.434 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:46:23.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:23 smithi032 ceph-mon[20004]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:46:23.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:23 smithi032 ceph-mon[20004]: Cluster is now healthy 2024-08-19T14:46:23.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:23 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2731560141' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:46:23.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:23 smithi032 ceph-mon[20004]: osdmap e877: 12 total, 12 up, 8 in 2024-08-19T14:46:23.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:23 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2922681715' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-08-19T14:46:23.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:23 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:46:23.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:23 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:46:23.534 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:46:23.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:23 smithi045 ceph-mon[20143]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:46:23.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:23 smithi045 ceph-mon[20143]: Cluster is now healthy 2024-08-19T14:46:23.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:23 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2731560141' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:46:23.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:23 smithi045 ceph-mon[20143]: osdmap e877: 12 total, 12 up, 8 in 2024-08-19T14:46:23.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:23 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2922681715' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-08-19T14:46:23.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:23 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:46:23.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:23 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:46:23.559 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:23.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:23.560 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:46:23.661 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:46:23.684 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:23.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:23.685 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:46:23.786 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:46:23.810 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:23.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:23.812 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:46:23.913 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:46:23.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:23.957 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:23.957 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:46:24.058 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:46:24.144 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:24.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:24.146 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:46:24.247 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:46:24.272 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:24.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:24.273 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:46:24.374 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:46:24.394 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd reweight 2 1 2024-08-19T14:46:24.412 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:24.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:24.413 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:46:24.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:24 smithi032 ceph-mon[20004]: pgmap v2071: 47 pgs: 4 active+recovering+undersized+remapped, 43 active+clean; 4.4 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 601 op/s; 7634/215271 objects misplaced (3.546%); 19 MiB/s, 297 objects/s recovering 2024-08-19T14:46:24.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:24 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2922681715' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-08-19T14:46:24.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:24 smithi032 ceph-mon[20004]: osdmap e878: 12 total, 12 up, 8 in 2024-08-19T14:46:24.514 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:46:24.537 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:24.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:24.538 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:46:24.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:24 smithi045 ceph-mon[20143]: pgmap v2071: 47 pgs: 4 active+recovering+undersized+remapped, 43 active+clean; 4.4 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 601 op/s; 7634/215271 objects misplaced (3.546%); 19 MiB/s, 297 objects/s recovering 2024-08-19T14:46:24.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:24 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2922681715' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-08-19T14:46:24.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:24 smithi045 ceph-mon[20143]: osdmap e878: 12 total, 12 up, 8 in 2024-08-19T14:46:24.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:24 smithi031 ceph-mon[16336]: pgmap v2071: 47 pgs: 4 active+recovering+undersized+remapped, 43 active+clean; 4.4 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 601 op/s; 7634/215271 objects misplaced (3.546%); 19 MiB/s, 297 objects/s recovering 2024-08-19T14:46:24.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:24 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2922681715' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-08-19T14:46:24.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:24 smithi031 ceph-mon[16336]: osdmap e878: 12 total, 12 up, 8 in 2024-08-19T14:46:24.638 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:46:24.661 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:24.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:24.662 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:46:24.763 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:46:24.789 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:24.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:24.790 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:46:24.891 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:46:24.918 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:24.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:24.919 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:46:25.020 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:46:25.043 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:25.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:25.044 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:46:25.063 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-08-19T14:46:25.145 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:46:25.152 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:46:25.187 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:25.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:25.188 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:46:25.188 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:46:25.288 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:46:25.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:25.331 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:25.331 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:46:25.431 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:46:25.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:25.462 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:25.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:46:25.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:25 smithi045 ceph-mon[20143]: Health check failed: Degraded data redundancy: 4 pgs undersized (PG_DEGRADED) 2024-08-19T14:46:25.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:25 smithi045 ceph-mon[20143]: osdmap e879: 12 total, 12 up, 8 in 2024-08-19T14:46:25.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:25 smithi031 ceph-mon[16336]: Health check failed: Degraded data redundancy: 4 pgs undersized (PG_DEGRADED) 2024-08-19T14:46:25.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:25 smithi031 ceph-mon[16336]: osdmap e879: 12 total, 12 up, 8 in 2024-08-19T14:46:25.564 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:46:25.585 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:25.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:25.586 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:46:25.687 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:46:25.722 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:25.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:25.723 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:46:25.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:25 smithi032 ceph-mon[20004]: Health check failed: Degraded data redundancy: 4 pgs undersized (PG_DEGRADED) 2024-08-19T14:46:25.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:25 smithi032 ceph-mon[20004]: osdmap e879: 12 total, 12 up, 8 in 2024-08-19T14:46:25.824 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:46:25.834 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-19T14:46:25.853 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:25.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:25.854 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:46:25.955 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:46:25.979 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:25.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:25.980 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:46:25.984 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:46:25.984 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:46:26.009 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:46:26.010 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:46:26.036 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:46:26.036 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:46:26.067 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:46:26.067 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:46:26.080 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:46:26.099 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:46:26.099 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:46:26.105 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:26.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:26.106 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:46:26.132 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:46:26.133 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:46:26.162 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:46:26.162 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:46:26.186 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:46:26.186 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:46:26.207 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:46:26.211 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:46:26.211 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:46:26.244 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:26.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:26.245 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:46:26.250 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:46:26.250 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:46:26.289 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:46:26.290 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:46:26.325 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:46:26.332 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:46:26.332 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:46:26.346 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:46:26.353 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:26 smithi031 ceph-mon[16336]: pgmap v2073: 47 pgs: 4 active+recovering+undersized+remapped, 43 active+clean; 4.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 836 op/s; 6542/208986 objects misplaced (3.130%); 29 MiB/s, 465 objects/s recovering 2024-08-19T14:46:26.353 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:26 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:46:26.353 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:26 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:46:26.353 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:26 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2262171399' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:46:26.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:26.393 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:26.393 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:46:26.494 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:46:26.519 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:26.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:26.520 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:46:26.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:26 smithi045 ceph-mon[20143]: pgmap v2073: 47 pgs: 4 active+recovering+undersized+remapped, 43 active+clean; 4.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 836 op/s; 6542/208986 objects misplaced (3.130%); 29 MiB/s, 465 objects/s recovering 2024-08-19T14:46:26.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:26 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:46:26.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:26 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:46:26.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:26 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2262171399' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:46:26.620 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:46:26.670 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:26.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:26.671 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:46:26.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:26 smithi032 ceph-mon[20004]: pgmap v2073: 47 pgs: 4 active+recovering+undersized+remapped, 43 active+clean; 4.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 836 op/s; 6542/208986 objects misplaced (3.130%); 29 MiB/s, 465 objects/s recovering 2024-08-19T14:46:26.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:26 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:46:26.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:26 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:46:26.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:26 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2262171399' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:46:26.772 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:46:26.797 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:26.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:26.799 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:46:26.900 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:46:26.922 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:26.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:26.924 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:46:27.025 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:46:27.048 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:27.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:27.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:46:27.150 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:46:27.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:27.194 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:27.194 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:46:27.295 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:46:27.302 INFO:teuthology.orchestra.run.smithi031.stderr:reweighted osd.2 to 1 (10000) 2024-08-19T14:46:27.331 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:27.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:27.332 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:46:27.433 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:46:27.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:27.462 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:27.462 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:46:27.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:27 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3461309844' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-08-19T14:46:27.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:27 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-08-19T14:46:27.563 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:46:27.586 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:27.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:27.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:46:27.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:27 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3461309844' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-08-19T14:46:27.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:27 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-08-19T14:46:27.688 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:46:27.731 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:27.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:27.732 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:46:27.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:27 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3461309844' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-08-19T14:46:27.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:27 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-08-19T14:46:27.833 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:46:27.866 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:27.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:27.867 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:46:27.969 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:46:27.994 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:27.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:27.995 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:46:28.096 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:46:28.131 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:28.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:28.132 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:46:28.233 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:46:28.259 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:28.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:28.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:46:28.360 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:46:28.384 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:28.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:28.385 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:46:28.486 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:46:28.510 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:28.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:28.511 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:46:28.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:28 smithi045 ceph-mon[20143]: pgmap v2075: 47 pgs: 4 active+recovering+undersized+remapped, 43 active+clean; 4.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 927 op/s; 6092/202743 objects misplaced (3.005%); 22 MiB/s, 350 objects/s recovering 2024-08-19T14:46:28.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:28 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-08-19T14:46:28.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:28 smithi045 ceph-mon[20143]: osdmap e880: 12 total, 12 up, 8 in 2024-08-19T14:46:28.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:28 smithi045 ceph-mon[20143]: osdmap e881: 12 total, 12 up, 8 in 2024-08-19T14:46:28.612 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:46:28.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:28 smithi031 ceph-mon[16336]: pgmap v2075: 47 pgs: 4 active+recovering+undersized+remapped, 43 active+clean; 4.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 927 op/s; 6092/202743 objects misplaced (3.005%); 22 MiB/s, 350 objects/s recovering 2024-08-19T14:46:28.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:28 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-08-19T14:46:28.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:28 smithi031 ceph-mon[16336]: osdmap e880: 12 total, 12 up, 8 in 2024-08-19T14:46:28.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:28 smithi031 ceph-mon[16336]: osdmap e881: 12 total, 12 up, 8 in 2024-08-19T14:46:28.634 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:28.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:28.635 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:46:28.736 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:46:28.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:28 smithi032 ceph-mon[20004]: pgmap v2075: 47 pgs: 4 active+recovering+undersized+remapped, 43 active+clean; 4.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 927 op/s; 6092/202743 objects misplaced (3.005%); 22 MiB/s, 350 objects/s recovering 2024-08-19T14:46:28.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:28 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-08-19T14:46:28.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:28 smithi032 ceph-mon[20004]: osdmap e880: 12 total, 12 up, 8 in 2024-08-19T14:46:28.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:28 smithi032 ceph-mon[20004]: osdmap e881: 12 total, 12 up, 8 in 2024-08-19T14:46:28.758 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:28.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:28.759 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:46:28.860 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:46:28.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:28.882 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:28.882 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:46:28.983 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:46:29.005 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:29.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:29.006 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:46:29.107 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:46:29.130 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:29.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:29.131 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:46:29.232 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:46:29.257 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:29.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:29.258 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:46:29.359 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:46:29.383 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:29.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:29.384 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:46:29.485 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:46:29.507 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:29.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:29.508 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:46:29.609 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:46:29.634 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:29.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:29.635 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:46:29.670 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:46:29.736 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:46:29.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:29.766 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:29.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:46:29.867 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:46:29.891 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:29.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:29.892 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:46:29.993 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:46:30.037 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:30.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:30.038 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:46:30.139 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:46:30.177 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:30.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:30.178 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:46:30.279 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:46:30.304 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:30.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:30.305 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:46:30.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:30 smithi031 ceph-mon[16336]: pgmap v2078: 47 pgs: 4 active+recovering+undersized+remapped, 43 active+clean; 4.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 811 op/s; 6092/201285 objects misplaced (3.027%); 19 MiB/s, 298 objects/s recovering 2024-08-19T14:46:30.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:30 smithi031 ceph-mon[16336]: osdmap e882: 12 total, 12 up, 8 in 2024-08-19T14:46:30.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:30 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:46:30.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:30 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:46:30.406 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:46:30.431 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:30.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:30.432 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:46:30.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:30 smithi032 ceph-mon[20004]: pgmap v2078: 47 pgs: 4 active+recovering+undersized+remapped, 43 active+clean; 4.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 811 op/s; 6092/201285 objects misplaced (3.027%); 19 MiB/s, 298 objects/s recovering 2024-08-19T14:46:30.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:30 smithi032 ceph-mon[20004]: osdmap e882: 12 total, 12 up, 8 in 2024-08-19T14:46:30.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:30 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:46:30.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:30 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:46:30.532 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:46:30.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:30 smithi045 ceph-mon[20143]: pgmap v2078: 47 pgs: 4 active+recovering+undersized+remapped, 43 active+clean; 4.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 811 op/s; 6092/201285 objects misplaced (3.027%); 19 MiB/s, 298 objects/s recovering 2024-08-19T14:46:30.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:30 smithi045 ceph-mon[20143]: osdmap e882: 12 total, 12 up, 8 in 2024-08-19T14:46:30.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:30 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:46:30.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:30 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:46:30.555 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:30.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:30.556 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:46:30.658 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:46:30.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:30.684 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:30.685 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:46:30.786 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:46:30.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:30.831 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:30.831 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:46:30.932 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:46:30.957 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:30.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:30.959 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:46:31.060 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:46:31.084 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:31.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:31.085 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:46:31.186 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:46:31.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:31.211 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:31.212 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:46:31.312 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:46:31.337 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:31.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:31.338 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:46:31.439 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:46:31.465 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:31.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:31.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:46:31.567 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:46:31.592 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:31.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:31.593 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:46:31.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:31 smithi031 ceph-mon[16336]: osdmap e883: 12 total, 12 up, 8 in 2024-08-19T14:46:31.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:31 smithi031 ceph-mon[16336]: pgmap v2081: 47 pgs: 1 peering, 3 active+recovering+undersized+remapped, 43 active+clean; 4.0 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 555 op/s; 4810/196074 objects misplaced (2.453%); 20 MiB/s, 320 objects/s recovering 2024-08-19T14:46:31.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:31 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:46:31.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:31 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:46:31.694 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:46:31.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:31.745 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:31.745 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:46:31.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:31 smithi032 ceph-mon[20004]: osdmap e883: 12 total, 12 up, 8 in 2024-08-19T14:46:31.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:31 smithi032 ceph-mon[20004]: pgmap v2081: 47 pgs: 1 peering, 3 active+recovering+undersized+remapped, 43 active+clean; 4.0 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 555 op/s; 4810/196074 objects misplaced (2.453%); 20 MiB/s, 320 objects/s recovering 2024-08-19T14:46:31.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:31 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:46:31.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:31 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:46:31.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:31 smithi045 ceph-mon[20143]: osdmap e883: 12 total, 12 up, 8 in 2024-08-19T14:46:31.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:31 smithi045 ceph-mon[20143]: pgmap v2081: 47 pgs: 1 peering, 3 active+recovering+undersized+remapped, 43 active+clean; 4.0 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 555 op/s; 4810/196074 objects misplaced (2.453%); 20 MiB/s, 320 objects/s recovering 2024-08-19T14:46:31.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:31 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:46:31.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:31 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:46:31.846 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:46:31.870 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:31.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:31.871 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:46:31.972 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:46:31.995 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:31.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:31.996 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:46:32.097 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:46:32.135 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:32.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:32.136 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:46:32.237 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:46:32.279 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:32.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:32.280 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:46:32.381 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:46:32.416 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:32.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:32.418 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:46:32.519 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:46:32.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:32.555 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:32.555 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:46:32.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:32 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 3 pgs undersized (PG_DEGRADED) 2024-08-19T14:46:32.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:32 smithi031 ceph-mon[16336]: osdmap e884: 12 total, 12 up, 8 in 2024-08-19T14:46:32.656 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:46:32.677 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:32.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:32.679 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:46:32.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:32 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 3 pgs undersized (PG_DEGRADED) 2024-08-19T14:46:32.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:32 smithi032 ceph-mon[20004]: osdmap e884: 12 total, 12 up, 8 in 2024-08-19T14:46:32.780 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:46:32.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:32 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 3 pgs undersized (PG_DEGRADED) 2024-08-19T14:46:32.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:32 smithi045 ceph-mon[20143]: osdmap e884: 12 total, 12 up, 8 in 2024-08-19T14:46:32.806 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:32.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:32.807 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:46:32.908 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:46:32.931 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:32.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:32.933 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:46:33.034 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:46:33.058 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:33.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:33.059 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:46:33.160 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:46:33.182 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:33.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:33.183 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:46:33.284 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:46:33.307 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:33.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:33.308 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:46:33.409 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:46:33.450 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:33.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:33.451 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:46:33.552 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:46:33.576 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:33.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:33.577 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:46:33.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:33 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:46:33.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:33 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:46:33.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:33 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:46:33.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:33 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:46:33.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:33 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:46:33.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:33 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:46:33.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:33 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:46:33.678 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:46:33.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:33.721 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:33.721 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:46:33.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:33 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:46:33.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:33 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:46:33.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:33 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:46:33.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:33 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:46:33.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:33 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:46:33.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:33 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:46:33.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:33 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:46:33.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:33 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:46:33.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:33 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:46:33.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:33 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:46:33.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:33 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:46:33.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:33 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:46:33.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:33 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:46:33.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:33 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:46:33.822 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:46:33.860 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:33.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:33.861 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:46:33.962 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:46:33.988 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:33.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:33.989 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:46:34.089 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:46:34.127 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:34.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:34.128 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:46:34.197 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd reweight 11 1 2024-08-19T14:46:34.229 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:46:34.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:34.274 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:34.274 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:46:34.376 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:46:34.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:34.415 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:34.416 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:46:34.517 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:46:34.542 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:34.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:34.543 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:46:34.644 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:46:34.655 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:46:34.655 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 20, 2024-08-19T14:46:34.655 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 600, 2024-08-19T14:46:34.655 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [ 2024-08-19T14:46:34.655 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.655 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:157465 7.f 7:f1de090e:::benchmark_data_smithi045_115842_object39305:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e663)", 2024-08-19T14:46:34.655 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:36:41.844976+0000", 2024-08-19T14:46:34.655 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 592.80745012600005, 2024-08-19T14:46:34.655 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.32793317300000002, 2024-08-19T14:46:34.658 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:46:34.658 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:46:34.658 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:46:34.658 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:46:34.658 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:46:34.658 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:46:34.658 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 157465 2024-08-19T14:46:34.659 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.659 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:46:34.659 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.659 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:46:34.659 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:41.844976+0000", 2024-08-19T14:46:34.659 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:34.659 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.659 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.659 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:46:34.659 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:41.844976+0000", 2024-08-19T14:46:34.659 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:34.659 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.659 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.659 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:46:34.660 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:41.844976+0000", 2024-08-19T14:46:34.660 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2e-07 2024-08-19T14:46:34.660 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.660 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.660 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:46:34.660 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:41.844979+0000", 2024-08-19T14:46:34.660 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.1889999999999999e-06 2024-08-19T14:46:34.660 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.660 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.660 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:46:34.660 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:41.844987+0000", 2024-08-19T14:46:34.660 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.0069999999999997e-06 2024-08-19T14:46:34.661 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.661 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.661 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:46:34.661 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:41.844993+0000", 2024-08-19T14:46:34.661 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.835e-06 2024-08-19T14:46:34.661 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.661 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.661 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:46:34.661 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:41.963617+0000", 2024-08-19T14:46:34.661 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.118623785 2024-08-19T14:46:34.661 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.661 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.661 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:46:34.662 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:41.963705+0000", 2024-08-19T14:46:34.662 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.7458000000000002e-05 2024-08-19T14:46:34.662 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.662 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.662 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 8,9", 2024-08-19T14:46:34.662 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:41.963780+0000", 2024-08-19T14:46:34.662 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.5042000000000004e-05 2024-08-19T14:46:34.662 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.662 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.662 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:46:34.662 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:41.977760+0000", 2024-08-19T14:46:34.662 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.013980461 2024-08-19T14:46:34.663 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.663 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.663 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:46:34.663 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:41.977797+0000", 2024-08-19T14:46:34.663 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7088999999999997e-05 2024-08-19T14:46:34.663 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.663 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.663 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:46:34.663 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:42.172777+0000", 2024-08-19T14:46:34.663 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.19497942300000001 2024-08-19T14:46:34.663 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.663 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.663 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:46:34.664 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:42.172837+0000", 2024-08-19T14:46:34.664 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.0047999999999998e-05 2024-08-19T14:46:34.664 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.664 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.664 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:46:34.664 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:42.172909+0000", 2024-08-19T14:46:34.664 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.2416000000000006e-05 2024-08-19T14:46:34.664 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:34.664 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:46:34.664 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:34.664 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.664 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.664 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:163704 7.f 7:f2a17ec5:::benchmark_data_smithi045_115842_object45544:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e667)", 2024-08-19T14:46:34.665 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:36:59.137143+0000", 2024-08-19T14:46:34.665 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 575.51528275500004, 2024-08-19T14:46:34.665 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.409542774, 2024-08-19T14:46:34.665 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:46:34.665 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:46:34.665 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:46:34.665 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:46:34.665 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:46:34.665 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:46:34.665 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 163704 2024-08-19T14:46:34.665 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.665 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:46:34.665 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.665 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:46:34.666 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.137143+0000", 2024-08-19T14:46:34.666 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:34.666 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.666 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.666 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:46:34.666 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.137143+0000", 2024-08-19T14:46:34.666 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:34.666 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.666 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.666 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:46:34.666 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.137145+0000", 2024-08-19T14:46:34.666 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.615e-06 2024-08-19T14:46:34.667 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.667 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.667 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:46:34.667 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.137151+0000", 2024-08-19T14:46:34.667 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.2360000000000001e-06 2024-08-19T14:46:34.667 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.667 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.667 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:46:34.667 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.137165+0000", 2024-08-19T14:46:34.667 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3504e-05 2024-08-19T14:46:34.667 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.668 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.668 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:46:34.668 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.137173+0000", 2024-08-19T14:46:34.668 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.3999999999999992e-06 2024-08-19T14:46:34.668 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.668 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.668 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:46:34.668 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.332675+0000", 2024-08-19T14:46:34.669 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.19550209499999999 2024-08-19T14:46:34.669 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.669 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.669 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:46:34.669 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.332815+0000", 2024-08-19T14:46:34.669 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013971799999999999 2024-08-19T14:46:34.669 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.669 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.669 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 8,9", 2024-08-19T14:46:34.669 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.332962+0000", 2024-08-19T14:46:34.669 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00014688900000000001 2024-08-19T14:46:34.669 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.669 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.670 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:46:34.670 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.348845+0000", 2024-08-19T14:46:34.670 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.015883026000000001 2024-08-19T14:46:34.670 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.670 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.670 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:46:34.670 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.349034+0000", 2024-08-19T14:46:34.670 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000189624 2024-08-19T14:46:34.670 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.670 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.670 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:46:34.670 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.546583+0000", 2024-08-19T14:46:34.670 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.19754886199999999 2024-08-19T14:46:34.670 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.671 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.671 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:46:34.671 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.546631+0000", 2024-08-19T14:46:34.671 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7682000000000002e-05 2024-08-19T14:46:34.671 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.671 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.671 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:46:34.671 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.546686+0000", 2024-08-19T14:46:34.671 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.5123000000000001e-05 2024-08-19T14:46:34.671 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:34.671 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:46:34.671 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:34.671 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.672 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.672 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:163711 7.f 7:f8f1f6f9:::benchmark_data_smithi045_115842_object45551:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e667)", 2024-08-19T14:46:34.672 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:36:59.161886+0000", 2024-08-19T14:46:34.672 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 575.49053952500003, 2024-08-19T14:46:34.672 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.38488678900000001, 2024-08-19T14:46:34.672 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:46:34.672 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:46:34.672 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:46:34.672 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:46:34.672 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:46:34.672 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:46:34.672 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 163711 2024-08-19T14:46:34.673 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.673 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:46:34.673 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.673 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:46:34.673 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.161886+0000", 2024-08-19T14:46:34.673 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:34.673 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.673 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.673 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:46:34.673 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.161886+0000", 2024-08-19T14:46:34.673 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:34.673 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.674 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.674 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:46:34.674 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.161888+0000", 2024-08-19T14:46:34.674 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6199999999999999e-06 2024-08-19T14:46:34.674 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.674 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.674 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:46:34.674 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.161895+0000", 2024-08-19T14:46:34.674 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.8329999999999996e-06 2024-08-19T14:46:34.674 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.675 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.675 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:46:34.675 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.161909+0000", 2024-08-19T14:46:34.675 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3913e-05 2024-08-19T14:46:34.675 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.675 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.675 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:46:34.675 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.161917+0000", 2024-08-19T14:46:34.675 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.2239999999999995e-06 2024-08-19T14:46:34.675 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.675 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.675 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:46:34.676 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.338110+0000", 2024-08-19T14:46:34.676 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.176193025 2024-08-19T14:46:34.676 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.676 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.676 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:46:34.676 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.338242+0000", 2024-08-19T14:46:34.676 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013215699999999999 2024-08-19T14:46:34.676 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.676 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.676 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 8,9", 2024-08-19T14:46:34.676 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.338366+0000", 2024-08-19T14:46:34.677 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012349299999999999 2024-08-19T14:46:34.677 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.677 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.677 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:46:34.677 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.349206+0000", 2024-08-19T14:46:34.677 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.010840214000000001 2024-08-19T14:46:34.677 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.677 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.677 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:46:34.677 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.349395+0000", 2024-08-19T14:46:34.677 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00018901599999999999 2024-08-19T14:46:34.677 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.677 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.677 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:46:34.678 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.546698+0000", 2024-08-19T14:46:34.678 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.197302739 2024-08-19T14:46:34.678 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.678 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.678 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:46:34.678 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.546734+0000", 2024-08-19T14:46:34.678 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.5914000000000002e-05 2024-08-19T14:46:34.678 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.678 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.678 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:46:34.678 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.546773+0000", 2024-08-19T14:46:34.678 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.9641000000000002e-05 2024-08-19T14:46:34.679 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:34.679 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:46:34.679 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:34.679 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.679 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.679 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:163728 7.f 7:f3d138f8:::benchmark_data_smithi045_115842_object45568:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e667)", 2024-08-19T14:46:34.679 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:36:59.213851+0000", 2024-08-19T14:46:34.679 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 575.43857503900006, 2024-08-19T14:46:34.679 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.33300425, 2024-08-19T14:46:34.679 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:46:34.679 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:46:34.679 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:46:34.679 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:46:34.680 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:46:34.680 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:46:34.680 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 163728 2024-08-19T14:46:34.680 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.680 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:46:34.680 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.680 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:46:34.680 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.213851+0000", 2024-08-19T14:46:34.680 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:34.680 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.680 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.680 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:46:34.680 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.213851+0000", 2024-08-19T14:46:34.680 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:34.681 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.681 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.681 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:46:34.681 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.213853+0000", 2024-08-19T14:46:34.681 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.984e-06 2024-08-19T14:46:34.681 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.681 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.681 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:46:34.681 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.213888+0000", 2024-08-19T14:46:34.681 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.5466000000000002e-05 2024-08-19T14:46:34.681 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.681 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.681 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:46:34.682 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.213910+0000", 2024-08-19T14:46:34.682 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.1855e-05 2024-08-19T14:46:34.682 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.682 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.682 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:46:34.682 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.213920+0000", 2024-08-19T14:46:34.682 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.9150000000000008e-06 2024-08-19T14:46:34.682 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.682 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.682 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:46:34.682 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.343357+0000", 2024-08-19T14:46:34.682 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.129436843 2024-08-19T14:46:34.682 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.683 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.683 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:46:34.683 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.343470+0000", 2024-08-19T14:46:34.683 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000113356 2024-08-19T14:46:34.683 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.683 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.683 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 8,9", 2024-08-19T14:46:34.683 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.343598+0000", 2024-08-19T14:46:34.683 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012727800000000001 2024-08-19T14:46:34.683 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.683 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.683 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:46:34.683 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.349571+0000", 2024-08-19T14:46:34.683 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0059730740000000001 2024-08-19T14:46:34.684 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.684 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.684 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:46:34.684 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.349748+0000", 2024-08-19T14:46:34.684 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000177329 2024-08-19T14:46:34.684 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.684 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.684 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:46:34.684 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.546783+0000", 2024-08-19T14:46:34.684 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.19703465000000001 2024-08-19T14:46:34.684 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.684 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.684 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:46:34.684 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.546817+0000", 2024-08-19T14:46:34.685 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.4446999999999997e-05 2024-08-19T14:46:34.685 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.685 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.685 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:46:34.685 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.546855+0000", 2024-08-19T14:46:34.685 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.8053000000000003e-05 2024-08-19T14:46:34.685 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:34.685 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:46:34.685 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:34.685 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.685 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.685 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:163735 7.f 7:fba5fed5:::benchmark_data_smithi045_115842_object45575:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e667)", 2024-08-19T14:46:34.685 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:36:59.227314+0000", 2024-08-19T14:46:34.685 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 575.42511159499998, 2024-08-19T14:46:34.686 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.31962848100000002, 2024-08-19T14:46:34.686 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:46:34.686 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:46:34.686 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:46:34.686 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:46:34.686 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:46:34.686 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:46:34.686 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 163735 2024-08-19T14:46:34.686 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.686 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:46:34.686 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.686 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:46:34.687 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.227314+0000", 2024-08-19T14:46:34.687 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:34.687 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.687 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.687 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:46:34.687 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.227314+0000", 2024-08-19T14:46:34.687 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:34.687 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.687 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.687 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:46:34.687 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.227316+0000", 2024-08-19T14:46:34.687 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6130000000000001e-06 2024-08-19T14:46:34.687 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.687 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.688 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:46:34.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:34.690 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.227324+0000", 2024-08-19T14:46:34.690 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.2029999999999992e-06 2024-08-19T14:46:34.690 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.690 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.690 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:46:34.690 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.227340+0000", 2024-08-19T14:46:34.691 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5914e-05 2024-08-19T14:46:34.691 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.691 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.691 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:46:34.691 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.227347+0000", 2024-08-19T14:46:34.691 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.9310000000000002e-06 2024-08-19T14:46:34.691 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.691 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.691 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:46:34.691 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.349894+0000", 2024-08-19T14:46:34.691 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.122547172 2024-08-19T14:46:34.691 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.691 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.691 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:46:34.692 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.350230+0000", 2024-08-19T14:46:34.692 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00033566800000000001 2024-08-19T14:46:34.692 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.692 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.692 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 8,9", 2024-08-19T14:46:34.692 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.350604+0000", 2024-08-19T14:46:34.692 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00037375800000000002 2024-08-19T14:46:34.692 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.692 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.692 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:46:34.692 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.365945+0000", 2024-08-19T14:46:34.692 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.015341059000000001 2024-08-19T14:46:34.692 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.693 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.693 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:46:34.693 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.365989+0000", 2024-08-19T14:46:34.693 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.3773999999999998e-05 2024-08-19T14:46:34.693 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.693 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.693 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:46:34.693 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.546869+0000", 2024-08-19T14:46:34.693 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.180880397 2024-08-19T14:46:34.693 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.693 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.693 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:46:34.693 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.546905+0000", 2024-08-19T14:46:34.693 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.5914000000000002e-05 2024-08-19T14:46:34.694 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.694 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.694 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:46:34.694 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:36:59.546943+0000", 2024-08-19T14:46:34.694 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.8077999999999997e-05 2024-08-19T14:46:34.694 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:34.694 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:46:34.694 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:34.694 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.694 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.694 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:220612 7.4 7:2bb015c3:::benchmark_data_smithi045_115842_object102452:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e700)", 2024-08-19T14:46:34.694 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:39:06.800533+0000", 2024-08-19T14:46:34.694 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 447.85189330200001, 2024-08-19T14:46:34.695 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.33481899700000001, 2024-08-19T14:46:34.695 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:46:34.695 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:46:34.695 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:46:34.695 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:46:34.695 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:46:34.695 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:46:34.695 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 220612 2024-08-19T14:46:34.695 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.695 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:46:34.695 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.695 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:46:34.695 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.800533+0000", 2024-08-19T14:46:34.696 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:34.696 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.696 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.696 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:46:34.696 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.800533+0000", 2024-08-19T14:46:34.696 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:34.696 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.696 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.696 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:46:34.696 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.800534+0000", 2024-08-19T14:46:34.696 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.22e-06 2024-08-19T14:46:34.696 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.696 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.696 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:46:34.697 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.800541+0000", 2024-08-19T14:46:34.697 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.6760000000000001e-06 2024-08-19T14:46:34.697 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.697 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.697 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:46:34.697 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.800551+0000", 2024-08-19T14:46:34.697 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0584000000000001e-05 2024-08-19T14:46:34.697 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.697 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.697 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:46:34.697 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.800558+0000", 2024-08-19T14:46:34.697 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.233e-06 2024-08-19T14:46:34.697 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.697 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.698 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:46:34.698 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.800660+0000", 2024-08-19T14:46:34.698 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00010177499999999999 2024-08-19T14:46:34.698 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.698 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.698 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:46:34.698 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.800786+0000", 2024-08-19T14:46:34.698 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000125353 2024-08-19T14:46:34.698 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.698 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.698 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 1,4", 2024-08-19T14:46:34.698 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.800914+0000", 2024-08-19T14:46:34.698 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012890800000000001 2024-08-19T14:46:34.699 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.699 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.699 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:46:34.699 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.820713+0000", 2024-08-19T14:46:34.699 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.019798382999999999 2024-08-19T14:46:34.699 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.699 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.699 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:46:34.699 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:07.135163+0000", 2024-08-19T14:46:34.700 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.31445014900000001 2024-08-19T14:46:34.700 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.700 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.700 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:46:34.700 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:07.135264+0000", 2024-08-19T14:46:34.700 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000100667 2024-08-19T14:46:34.700 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.700 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.700 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:46:34.700 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:07.135310+0000", 2024-08-19T14:46:34.700 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.6817999999999998e-05 2024-08-19T14:46:34.700 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.700 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.700 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:46:34.701 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:07.135352+0000", 2024-08-19T14:46:34.701 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.1230999999999999e-05 2024-08-19T14:46:34.701 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:34.701 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:46:34.701 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:34.701 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.701 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.701 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:220613 7.4 7:248c3256:::benchmark_data_smithi045_115842_object102453:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e700)", 2024-08-19T14:46:34.701 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:39:06.800580+0000", 2024-08-19T14:46:34.701 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 447.85184590400002, 2024-08-19T14:46:34.701 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.334942236, 2024-08-19T14:46:34.701 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:46:34.701 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:46:34.702 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:46:34.702 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:46:34.702 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:46:34.702 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:46:34.702 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 220613 2024-08-19T14:46:34.702 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.702 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:46:34.702 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.702 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:46:34.702 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.800580+0000", 2024-08-19T14:46:34.702 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:34.702 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.703 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.703 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:46:34.703 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.800580+0000", 2024-08-19T14:46:34.703 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:34.703 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.703 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.703 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:46:34.703 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.800581+0000", 2024-08-19T14:46:34.703 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2500000000000001e-07 2024-08-19T14:46:34.703 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.703 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.704 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:46:34.704 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.800583+0000", 2024-08-19T14:46:34.704 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.6290000000000002e-06 2024-08-19T14:46:34.704 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.704 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.704 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:46:34.704 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.800590+0000", 2024-08-19T14:46:34.704 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.9280000000000004e-06 2024-08-19T14:46:34.704 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.704 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.704 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:46:34.704 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.800595+0000", 2024-08-19T14:46:34.705 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.079e-06 2024-08-19T14:46:34.705 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.705 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.705 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:46:34.705 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.806658+0000", 2024-08-19T14:46:34.705 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0060631569999999996 2024-08-19T14:46:34.705 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.705 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.705 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:46:34.705 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.806760+0000", 2024-08-19T14:46:34.705 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000101505 2024-08-19T14:46:34.705 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.706 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.706 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 1,4", 2024-08-19T14:46:34.706 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.806857+0000", 2024-08-19T14:46:34.706 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.6738999999999997e-05 2024-08-19T14:46:34.706 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.706 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.706 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:46:34.706 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.820757+0000", 2024-08-19T14:46:34.706 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.013900279999999999 2024-08-19T14:46:34.706 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.707 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.707 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:46:34.707 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:07.135383+0000", 2024-08-19T14:46:34.707 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.31462581899999997 2024-08-19T14:46:34.707 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.707 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.707 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:46:34.707 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:07.135425+0000", 2024-08-19T14:46:34.707 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2536000000000001e-05 2024-08-19T14:46:34.707 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.707 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.707 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:46:34.708 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:07.135490+0000", 2024-08-19T14:46:34.708 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.5147000000000005e-05 2024-08-19T14:46:34.708 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.708 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.708 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:46:34.708 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:07.135522+0000", 2024-08-19T14:46:34.708 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.1992000000000001e-05 2024-08-19T14:46:34.708 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:34.708 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:46:34.708 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:34.708 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.708 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.709 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:220618 7.4 7:289c7a94:::benchmark_data_smithi045_115842_object102458:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e700)", 2024-08-19T14:46:34.709 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:39:06.812624+0000", 2024-08-19T14:46:34.709 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 447.83980244700001, 2024-08-19T14:46:34.709 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.32299076100000002, 2024-08-19T14:46:34.709 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:46:34.709 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:46:34.709 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:46:34.709 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:46:34.709 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:46:34.709 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:46:34.709 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 220618 2024-08-19T14:46:34.709 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.710 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:46:34.710 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.710 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:46:34.710 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.812624+0000", 2024-08-19T14:46:34.710 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:34.710 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.710 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.710 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:46:34.710 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.812624+0000", 2024-08-19T14:46:34.710 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:34.710 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.710 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.710 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:46:34.710 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.812625+0000", 2024-08-19T14:46:34.711 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.7579999999999999e-06 2024-08-19T14:46:34.711 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.711 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.711 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:46:34.711 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.812632+0000", 2024-08-19T14:46:34.711 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.7580000000000003e-06 2024-08-19T14:46:34.711 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.711 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.711 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:46:34.711 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.812646+0000", 2024-08-19T14:46:34.711 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4157e-05 2024-08-19T14:46:34.711 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.711 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.712 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:46:34.712 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.812654+0000", 2024-08-19T14:46:34.712 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.542e-06 2024-08-19T14:46:34.712 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.712 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.712 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:46:34.712 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.820783+0000", 2024-08-19T14:46:34.712 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0081295340000000008 2024-08-19T14:46:34.712 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.712 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.712 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:46:34.712 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.820928+0000", 2024-08-19T14:46:34.712 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00014483899999999999 2024-08-19T14:46:34.712 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.713 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.713 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 1,4", 2024-08-19T14:46:34.713 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.821050+0000", 2024-08-19T14:46:34.713 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000121371 2024-08-19T14:46:34.713 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.713 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.713 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:46:34.713 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:07.135219+0000", 2024-08-19T14:46:34.713 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.31416936099999998 2024-08-19T14:46:34.713 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.713 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.713 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:46:34.713 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:07.135416+0000", 2024-08-19T14:46:34.714 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00019753300000000001 2024-08-19T14:46:34.714 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.714 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.714 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:46:34.714 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:07.135527+0000", 2024-08-19T14:46:34.714 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00011103900000000001 2024-08-19T14:46:34.714 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.714 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.714 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:46:34.714 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:07.135586+0000", 2024-08-19T14:46:34.714 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.8316000000000002e-05 2024-08-19T14:46:34.714 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.714 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.714 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:46:34.715 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:07.135614+0000", 2024-08-19T14:46:34.715 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.8552999999999999e-05 2024-08-19T14:46:34.715 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:34.715 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:46:34.715 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:34.715 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.715 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.715 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:220619 7.6 7:65071c03:::benchmark_data_smithi045_115842_object102459:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e700)", 2024-08-19T14:46:34.715 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:39:06.813741+0000", 2024-08-19T14:46:34.715 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 447.83868547700001, 2024-08-19T14:46:34.715 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.33041151699999999, 2024-08-19T14:46:34.715 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:46:34.715 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:46:34.716 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:46:34.716 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:46:34.716 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:46:34.716 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:46:34.716 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 220619 2024-08-19T14:46:34.716 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.716 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:46:34.716 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.716 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:46:34.716 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.813741+0000", 2024-08-19T14:46:34.716 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:34.716 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.716 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.716 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:46:34.717 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.813741+0000", 2024-08-19T14:46:34.717 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:34.717 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.717 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.717 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:46:34.717 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.813742+0000", 2024-08-19T14:46:34.717 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.04e-06 2024-08-19T14:46:34.717 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.717 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.717 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:46:34.717 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.813746+0000", 2024-08-19T14:46:34.717 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7199999999999997e-06 2024-08-19T14:46:34.717 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.717 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.718 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:46:34.718 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.813757+0000", 2024-08-19T14:46:34.718 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0397999999999999e-05 2024-08-19T14:46:34.718 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.718 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.718 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:46:34.718 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.813764+0000", 2024-08-19T14:46:34.718 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.4100000000000002e-06 2024-08-19T14:46:34.718 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.718 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.718 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:46:34.718 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.816627+0000", 2024-08-19T14:46:34.719 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0028632319999999998 2024-08-19T14:46:34.719 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.719 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.719 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:46:34.719 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.816735+0000", 2024-08-19T14:46:34.719 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000107346 2024-08-19T14:46:34.719 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.719 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.719 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,7", 2024-08-19T14:46:34.719 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.816845+0000", 2024-08-19T14:46:34.719 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000110503 2024-08-19T14:46:34.719 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.719 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.719 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:46:34.720 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:07.143968+0000", 2024-08-19T14:46:34.720 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.32712271799999998 2024-08-19T14:46:34.720 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.720 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.720 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:46:34.720 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:07.144006+0000", 2024-08-19T14:46:34.720 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.8560999999999999e-05 2024-08-19T14:46:34.720 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.720 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.720 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:46:34.720 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:07.144043+0000", 2024-08-19T14:46:34.720 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.6284e-05 2024-08-19T14:46:34.721 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.721 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.721 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:46:34.721 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:07.144102+0000", 2024-08-19T14:46:34.721 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.9067999999999998e-05 2024-08-19T14:46:34.721 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.721 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.721 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:46:34.721 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:07.144152+0000", 2024-08-19T14:46:34.721 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0237e-05 2024-08-19T14:46:34.721 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:34.721 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:46:34.721 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:34.721 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.722 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.722 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:220621 7.6 7:6e439cf0:::benchmark_data_smithi045_115842_object102461:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e700)", 2024-08-19T14:46:34.722 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:39:06.819474+0000", 2024-08-19T14:46:34.722 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 447.83295219799999, 2024-08-19T14:46:34.722 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.34391892600000001, 2024-08-19T14:46:34.722 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:46:34.722 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:46:34.722 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:46:34.722 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:46:34.722 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:46:34.722 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:46:34.722 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 220621 2024-08-19T14:46:34.722 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.723 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:46:34.723 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.723 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:46:34.723 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.819474+0000", 2024-08-19T14:46:34.723 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:34.723 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.723 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.723 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:46:34.723 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.819474+0000", 2024-08-19T14:46:34.723 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:34.723 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.723 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.723 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:46:34.723 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.819476+0000", 2024-08-19T14:46:34.724 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.9290000000000001e-06 2024-08-19T14:46:34.724 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.724 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.724 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:46:34.724 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.819482+0000", 2024-08-19T14:46:34.724 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.0920000000000003e-06 2024-08-19T14:46:34.724 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.724 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.724 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:46:34.724 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.819499+0000", 2024-08-19T14:46:34.724 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6688999999999999e-05 2024-08-19T14:46:34.724 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.724 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.724 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:46:34.725 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.819508+0000", 2024-08-19T14:46:34.725 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.5510000000000005e-06 2024-08-19T14:46:34.725 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.725 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.725 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:46:34.725 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.821834+0000", 2024-08-19T14:46:34.725 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0023259280000000001 2024-08-19T14:46:34.725 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.725 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.725 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:46:34.725 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:07.134834+0000", 2024-08-19T14:46:34.725 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.31300021900000002 2024-08-19T14:46:34.725 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.725 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.726 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,7", 2024-08-19T14:46:34.726 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:07.139030+0000", 2024-08-19T14:46:34.726 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.004196137 2024-08-19T14:46:34.726 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.726 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.726 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:46:34.726 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:07.144192+0000", 2024-08-19T14:46:34.726 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0051618660000000002 2024-08-19T14:46:34.726 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.726 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.726 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:46:34.726 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:07.144203+0000", 2024-08-19T14:46:34.726 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.117e-05 2024-08-19T14:46:34.726 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.727 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.727 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:46:34.727 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:07.163158+0000", 2024-08-19T14:46:34.727 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.018954649 2024-08-19T14:46:34.727 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.727 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.727 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:46:34.727 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:07.163258+0000", 2024-08-19T14:46:34.727 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.9543000000000006e-05 2024-08-19T14:46:34.727 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.727 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.727 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:46:34.727 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:07.163393+0000", 2024-08-19T14:46:34.728 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000135153 2024-08-19T14:46:34.728 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:34.728 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:46:34.728 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:34.728 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.728 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.728 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.16962.0:220620 7.d e700/600 7:b15dd3f1:::benchmark_data_smithi045_115842_object102460:head v 700'13832, mlcod=700'13829)", 2024-08-19T14:46:34.728 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:39:06.821415+0000", 2024-08-19T14:46:34.728 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 447.83101121499999, 2024-08-19T14:46:34.728 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.31966576000000002, 2024-08-19T14:46:34.728 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:46:34.728 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:46:34.728 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:46:34.728 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:46:34.729 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.729 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:46:34.729 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.821415+0000", 2024-08-19T14:46:34.729 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:34.729 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.729 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.729 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:46:34.729 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.821415+0000", 2024-08-19T14:46:34.729 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:34.729 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.729 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.729 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:46:34.729 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.821416+0000", 2024-08-19T14:46:34.729 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.9800000000000003e-07 2024-08-19T14:46:34.730 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.730 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.730 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:46:34.730 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.821421+0000", 2024-08-19T14:46:34.730 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.3859999999999998e-06 2024-08-19T14:46:34.730 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.730 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.730 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:46:34.730 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.821430+0000", 2024-08-19T14:46:34.730 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.0650000000000005e-06 2024-08-19T14:46:34.730 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.730 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.730 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:46:34.731 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.821437+0000", 2024-08-19T14:46:34.731 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.9040000000000003e-06 2024-08-19T14:46:34.731 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.731 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.731 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:46:34.731 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.821504+0000", 2024-08-19T14:46:34.731 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.7256999999999997e-05 2024-08-19T14:46:34.731 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.731 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.731 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:46:34.731 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.821522+0000", 2024-08-19T14:46:34.731 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.8218000000000001e-05 2024-08-19T14:46:34.731 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.732 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.732 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:46:34.732 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:07.141002+0000", 2024-08-19T14:46:34.732 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.31947991100000001 2024-08-19T14:46:34.732 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.732 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.732 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:46:34.732 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:07.141081+0000", 2024-08-19T14:46:34.732 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.8220999999999995e-05 2024-08-19T14:46:34.732 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:34.732 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:46:34.732 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:34.732 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.733 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.733 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:220624 7.f 7:f8cdd8b1:::benchmark_data_smithi045_115842_object102464:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e700)", 2024-08-19T14:46:34.733 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:39:06.827029+0000", 2024-08-19T14:46:34.733 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 447.82539738000003, 2024-08-19T14:46:34.733 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.32098705700000002, 2024-08-19T14:46:34.733 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:46:34.733 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:46:34.733 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:46:34.733 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:46:34.733 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:46:34.733 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:46:34.733 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 220624 2024-08-19T14:46:34.733 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.733 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:46:34.734 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.734 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:46:34.734 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.827029+0000", 2024-08-19T14:46:34.734 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:34.734 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.734 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.734 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:46:34.734 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.827029+0000", 2024-08-19T14:46:34.734 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:34.734 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.734 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.734 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:46:34.735 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.827030+0000", 2024-08-19T14:46:34.735 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.694e-06 2024-08-19T14:46:34.735 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.735 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.735 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:46:34.735 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:06.827040+0000", 2024-08-19T14:46:34.735 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.3009999999999995e-06 2024-08-19T14:46:34.735 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.735 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.735 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:46:34.735 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:07.134952+0000", 2024-08-19T14:46:34.735 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.30791282599999997 2024-08-19T14:46:34.735 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.735 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.736 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:46:34.736 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:07.134963+0000", 2024-08-19T14:46:34.736 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0482999999999999e-05 2024-08-19T14:46:34.736 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.736 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.736 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:46:34.736 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:07.135025+0000", 2024-08-19T14:46:34.736 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.2508999999999994e-05 2024-08-19T14:46:34.736 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.736 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.736 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:46:34.736 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:07.135155+0000", 2024-08-19T14:46:34.736 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012912700000000001 2024-08-19T14:46:34.737 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.737 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.737 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 8,9", 2024-08-19T14:46:34.737 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:07.135911+0000", 2024-08-19T14:46:34.737 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000756878 2024-08-19T14:46:34.737 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.737 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.737 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:46:34.737 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:07.147742+0000", 2024-08-19T14:46:34.737 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.011830235999999999 2024-08-19T14:46:34.737 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.737 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.737 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:46:34.738 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:07.147783+0000", 2024-08-19T14:46:34.738 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.1764999999999997e-05 2024-08-19T14:46:34.738 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.738 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.738 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:46:34.738 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:07.147908+0000", 2024-08-19T14:46:34.738 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012414899999999999 2024-08-19T14:46:34.738 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.738 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.738 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:46:34.738 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:07.147970+0000", 2024-08-19T14:46:34.738 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.2646999999999998e-05 2024-08-19T14:46:34.739 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.739 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.739 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:46:34.739 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:07.148016+0000", 2024-08-19T14:46:34.739 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.5442000000000003e-05 2024-08-19T14:46:34.739 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:34.739 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:46:34.739 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:34.739 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.739 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.739 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:232591 7.6 7:6076b785:::benchmark_data_smithi045_115842_object114431:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e707)", 2024-08-19T14:46:34.739 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:39:29.424695+0000", 2024-08-19T14:46:34.739 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 425.22773122400002, 2024-08-19T14:46:34.740 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.35295967, 2024-08-19T14:46:34.740 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:46:34.740 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:46:34.740 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:46:34.740 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:46:34.740 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:46:34.740 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:46:34.740 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 232591 2024-08-19T14:46:34.740 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.740 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:46:34.740 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.740 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:46:34.740 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.424695+0000", 2024-08-19T14:46:34.741 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:34.741 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.741 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.741 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:46:34.741 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.424695+0000", 2024-08-19T14:46:34.741 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:34.741 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.741 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.741 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:46:34.741 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.424697+0000", 2024-08-19T14:46:34.741 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.73e-06 2024-08-19T14:46:34.741 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.741 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.741 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:46:34.742 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.424705+0000", 2024-08-19T14:46:34.742 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.1389999999999995e-06 2024-08-19T14:46:34.742 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.742 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.742 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:46:34.742 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.424721+0000", 2024-08-19T14:46:34.742 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6158999999999999e-05 2024-08-19T14:46:34.742 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.742 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.742 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:46:34.742 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.424730+0000", 2024-08-19T14:46:34.742 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.6770000000000007e-06 2024-08-19T14:46:34.742 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.743 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.743 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:46:34.743 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.424797+0000", 2024-08-19T14:46:34.743 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.6987000000000001e-05 2024-08-19T14:46:34.743 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.743 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.743 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:46:34.743 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.424928+0000", 2024-08-19T14:46:34.743 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013101399999999999 2024-08-19T14:46:34.743 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.743 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.743 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,7", 2024-08-19T14:46:34.743 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.425048+0000", 2024-08-19T14:46:34.743 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00011969899999999999 2024-08-19T14:46:34.744 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.744 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.744 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:46:34.744 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.431710+0000", 2024-08-19T14:46:34.744 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0066618270000000004 2024-08-19T14:46:34.744 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.744 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.744 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:46:34.744 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.431963+0000", 2024-08-19T14:46:34.744 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00025306500000000002 2024-08-19T14:46:34.744 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.744 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.744 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:46:34.745 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.777326+0000", 2024-08-19T14:46:34.745 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.34536338500000002 2024-08-19T14:46:34.745 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.745 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.745 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:46:34.745 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.777524+0000", 2024-08-19T14:46:34.745 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00019711899999999999 2024-08-19T14:46:34.745 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.745 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.745 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:46:34.745 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.777654+0000", 2024-08-19T14:46:34.745 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013086900000000001 2024-08-19T14:46:34.745 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:34.745 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:46:34.745 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:34.746 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.746 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.746 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:232592 7.6 7:6148bd1d:::benchmark_data_smithi045_115842_object114432:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e707)", 2024-08-19T14:46:34.746 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:39:29.434124+0000", 2024-08-19T14:46:34.746 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 425.21830194, 2024-08-19T14:46:34.746 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.34459850199999997, 2024-08-19T14:46:34.746 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:46:34.746 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:46:34.746 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:46:34.746 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:46:34.746 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:46:34.746 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:46:34.747 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 232592 2024-08-19T14:46:34.747 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.747 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:46:34.747 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.747 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:46:34.747 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.434124+0000", 2024-08-19T14:46:34.747 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:34.747 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.747 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.747 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:46:34.747 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.434124+0000", 2024-08-19T14:46:34.747 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:34.747 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.747 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.748 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:46:34.748 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.434126+0000", 2024-08-19T14:46:34.748 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.984e-06 2024-08-19T14:46:34.748 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.748 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.748 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:46:34.748 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.434134+0000", 2024-08-19T14:46:34.748 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.6899999999999992e-06 2024-08-19T14:46:34.748 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.748 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.748 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:46:34.748 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.434150+0000", 2024-08-19T14:46:34.748 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6526999999999999e-05 2024-08-19T14:46:34.749 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.749 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.749 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:46:34.749 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.434160+0000", 2024-08-19T14:46:34.749 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.6199999999999994e-06 2024-08-19T14:46:34.749 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.749 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.749 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:46:34.749 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.434286+0000", 2024-08-19T14:46:34.749 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000126191 2024-08-19T14:46:34.749 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.749 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.749 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:46:34.750 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.434411+0000", 2024-08-19T14:46:34.750 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012469400000000001 2024-08-19T14:46:34.750 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.750 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.750 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,7", 2024-08-19T14:46:34.750 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.434536+0000", 2024-08-19T14:46:34.750 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012499599999999999 2024-08-19T14:46:34.750 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.750 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.750 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:46:34.750 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.445685+0000", 2024-08-19T14:46:34.750 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.011149133 2024-08-19T14:46:34.750 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.750 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.751 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:46:34.751 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.445745+0000", 2024-08-19T14:46:34.751 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.0584000000000001e-05 2024-08-19T14:46:34.751 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.751 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.751 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:46:34.751 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.778598+0000", 2024-08-19T14:46:34.751 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.332852698 2024-08-19T14:46:34.751 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.751 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.751 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:46:34.751 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.778682+0000", 2024-08-19T14:46:34.751 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.3382999999999998e-05 2024-08-19T14:46:34.752 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.752 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.752 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:46:34.752 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.778723+0000", 2024-08-19T14:46:34.752 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.1001999999999998e-05 2024-08-19T14:46:34.752 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:34.752 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:46:34.752 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:34.752 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.752 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.752 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:232594 7.6 7:69e63e5e:::benchmark_data_smithi045_115842_object114434:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e707)", 2024-08-19T14:46:34.752 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:39:29.434195+0000", 2024-08-19T14:46:34.752 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 425.21823146999998, 2024-08-19T14:46:34.753 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.34467441900000001, 2024-08-19T14:46:34.753 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:46:34.753 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:46:34.753 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:46:34.753 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:46:34.753 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:46:34.753 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:46:34.753 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 232594 2024-08-19T14:46:34.753 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.753 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:46:34.753 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.753 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:46:34.753 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.434195+0000", 2024-08-19T14:46:34.753 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:34.754 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.754 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.754 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:46:34.754 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.434195+0000", 2024-08-19T14:46:34.754 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:34.754 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.754 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.754 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:46:34.754 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.434195+0000", 2024-08-19T14:46:34.754 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.9200000000000001e-07 2024-08-19T14:46:34.754 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.754 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.754 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:46:34.755 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.434198+0000", 2024-08-19T14:46:34.755 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.3170000000000002e-06 2024-08-19T14:46:34.755 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.755 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.755 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:46:34.755 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.434207+0000", 2024-08-19T14:46:34.755 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.1969999999999996e-06 2024-08-19T14:46:34.755 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.755 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.755 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:46:34.755 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.434213+0000", 2024-08-19T14:46:34.755 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.9970000000000004e-06 2024-08-19T14:46:34.756 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.756 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.756 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:46:34.756 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.440124+0000", 2024-08-19T14:46:34.756 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0059109899999999996 2024-08-19T14:46:34.756 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.756 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.756 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:46:34.756 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.440241+0000", 2024-08-19T14:46:34.756 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000117793 2024-08-19T14:46:34.756 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.756 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.757 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3,7", 2024-08-19T14:46:34.757 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.440354+0000", 2024-08-19T14:46:34.757 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000113087 2024-08-19T14:46:34.757 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.757 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.757 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:46:34.757 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.445733+0000", 2024-08-19T14:46:34.757 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0053782320000000002 2024-08-19T14:46:34.757 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.757 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.757 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:46:34.757 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.445751+0000", 2024-08-19T14:46:34.757 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.8042000000000001e-05 2024-08-19T14:46:34.758 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.758 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.758 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:46:34.758 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.778759+0000", 2024-08-19T14:46:34.758 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.33300808399999998 2024-08-19T14:46:34.758 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.758 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.758 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:46:34.758 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.778830+0000", 2024-08-19T14:46:34.758 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.0961e-05 2024-08-19T14:46:34.758 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.758 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.758 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:46:34.758 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:29.778869+0000", 2024-08-19T14:46:34.759 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.9227000000000002e-05 2024-08-19T14:46:34.759 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:34.759 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:46:34.759 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:34.759 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.759 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.759 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26911.0:70283 8.9 8.b4edcc9 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e736)", 2024-08-19T14:46:34.759 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:40:48.417481+0000", 2024-08-19T14:46:34.759 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 346.23494545699998, 2024-08-19T14:46:34.759 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.028042347, 2024-08-19T14:46:34.759 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:46:34.759 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:46:34.759 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:46:34.760 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:46:34.760 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26911", 2024-08-19T14:46:34.760 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3494993995", 2024-08-19T14:46:34.760 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 70283 2024-08-19T14:46:34.760 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.760 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:46:34.760 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.760 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:46:34.760 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:40:48.417481+0000", 2024-08-19T14:46:34.760 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:34.760 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.760 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.760 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:46:34.761 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:40:48.417481+0000", 2024-08-19T14:46:34.761 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:34.761 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.761 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.761 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:46:34.761 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:40:48.417481+0000", 2024-08-19T14:46:34.761 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.5199999999999997e-07 2024-08-19T14:46:34.761 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.761 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.761 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:46:34.761 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:40:48.417549+0000", 2024-08-19T14:46:34.761 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.7627999999999997e-05 2024-08-19T14:46:34.761 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.761 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.762 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:46:34.762 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:40:48.417557+0000", 2024-08-19T14:46:34.762 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.6130000000000004e-06 2024-08-19T14:46:34.762 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.762 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.762 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:46:34.762 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:40:48.417564+0000", 2024-08-19T14:46:34.762 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.3049999999999998e-06 2024-08-19T14:46:34.762 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.762 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.765 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:46:34.766 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:40:48.429627+0000", 2024-08-19T14:46:34.766 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.012062941000000001 2024-08-19T14:46:34.766 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.766 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.766 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:46:34.766 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:40:48.429668+0000", 2024-08-19T14:46:34.766 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.0948999999999997e-05 2024-08-19T14:46:34.766 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.766 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.766 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:46:34.767 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:40:49.445455+0000", 2024-08-19T14:46:34.767 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0157872729999999 2024-08-19T14:46:34.767 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.767 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.767 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:46:34.767 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:40:49.445523+0000", 2024-08-19T14:46:34.767 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.8085999999999998e-05 2024-08-19T14:46:34.767 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:34.767 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:46:34.767 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:34.767 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.767 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.767 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26911.0:70293 8.9 8.189a5e49 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e736)", 2024-08-19T14:46:34.768 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:40:48.428534+0000", 2024-08-19T14:46:34.768 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 346.22389220100001, 2024-08-19T14:46:34.768 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0170277210000001, 2024-08-19T14:46:34.768 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:46:34.768 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:46:34.768 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:46:34.768 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:46:34.768 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26911", 2024-08-19T14:46:34.769 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3494993995", 2024-08-19T14:46:34.769 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 70293 2024-08-19T14:46:34.769 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.769 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:46:34.769 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.769 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:46:34.769 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:40:48.428534+0000", 2024-08-19T14:46:34.769 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:34.769 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.769 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.769 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:46:34.769 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:40:48.428534+0000", 2024-08-19T14:46:34.769 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:34.770 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.770 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.770 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:46:34.770 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:40:48.428534+0000", 2024-08-19T14:46:34.770 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7999999999999996e-07 2024-08-19T14:46:34.770 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.770 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.770 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:46:34.770 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:40:48.428592+0000", 2024-08-19T14:46:34.770 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.7955999999999999e-05 2024-08-19T14:46:34.770 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.770 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.770 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:46:34.771 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:40:48.428600+0000", 2024-08-19T14:46:34.771 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.072e-06 2024-08-19T14:46:34.771 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.771 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.771 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:46:34.771 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:40:48.428606+0000", 2024-08-19T14:46:34.771 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.9529999999999996e-06 2024-08-19T14:46:34.771 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.771 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.771 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:46:34.771 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:40:48.429966+0000", 2024-08-19T14:46:34.771 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.001359751 2024-08-19T14:46:34.772 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.772 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.772 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:46:34.772 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:40:48.430156+0000", 2024-08-19T14:46:34.772 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000190034 2024-08-19T14:46:34.772 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.772 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.772 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:46:34.772 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:40:49.445554+0000", 2024-08-19T14:46:34.772 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0153980250000001 2024-08-19T14:46:34.772 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.772 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.772 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:46:34.772 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:40:49.445562+0000", 2024-08-19T14:46:34.773 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.4499999999999998e-06 2024-08-19T14:46:34.773 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:34.773 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:46:34.773 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:34.773 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.773 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.773 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26911.0:70295 8.9 8.6d05a669 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e736)", 2024-08-19T14:46:34.773 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:40:48.440426+0000", 2024-08-19T14:46:34.773 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 346.212000235, 2024-08-19T14:46:34.773 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0051761340000001, 2024-08-19T14:46:34.773 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:46:34.773 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:46:34.773 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:46:34.774 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:46:34.774 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26911", 2024-08-19T14:46:34.774 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3494993995", 2024-08-19T14:46:34.774 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 70295 2024-08-19T14:46:34.774 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.774 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:46:34.774 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.774 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:46:34.774 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:40:48.440426+0000", 2024-08-19T14:46:34.774 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:34.774 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.774 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.774 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:46:34.774 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:40:48.440426+0000", 2024-08-19T14:46:34.775 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:34.775 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.775 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.775 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:46:34.775 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:40:48.440427+0000", 2024-08-19T14:46:34.775 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5409999999999999e-06 2024-08-19T14:46:34.775 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.775 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.775 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:46:34.776 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:40:48.440467+0000", 2024-08-19T14:46:34.776 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.9473e-05 2024-08-19T14:46:34.776 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.776 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.776 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:46:34.776 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:40:48.440486+0000", 2024-08-19T14:46:34.776 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.9219999999999999e-05 2024-08-19T14:46:34.776 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.776 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.776 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:46:34.776 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:40:48.440494+0000", 2024-08-19T14:46:34.776 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.0239999999999997e-06 2024-08-19T14:46:34.776 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.777 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.777 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:46:34.777 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:40:48.441288+0000", 2024-08-19T14:46:34.777 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00079417799999999998 2024-08-19T14:46:34.777 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.777 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.777 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:46:34.777 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:40:48.441300+0000", 2024-08-19T14:46:34.777 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2167999999999999e-05 2024-08-19T14:46:34.777 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.777 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.777 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:46:34.777 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:40:49.445595+0000", 2024-08-19T14:46:34.778 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0042943070000001 2024-08-19T14:46:34.778 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.778 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.778 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:46:34.778 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:40:49.445602+0000", 2024-08-19T14:46:34.778 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.2230000000000001e-06 2024-08-19T14:46:34.778 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:34.778 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:46:34.778 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:34.778 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.778 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.778 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26911.0:70302 8.9 8.666f8769 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e736)", 2024-08-19T14:46:34.778 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:40:48.463513+0000", 2024-08-19T14:46:34.779 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 346.18891312800002, 2024-08-19T14:46:34.779 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.98211852200000005, 2024-08-19T14:46:34.779 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:46:34.779 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:46:34.779 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:46:34.779 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:46:34.779 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26911", 2024-08-19T14:46:34.779 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3494993995", 2024-08-19T14:46:34.779 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 70302 2024-08-19T14:46:34.779 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.779 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:46:34.779 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.779 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:46:34.779 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:40:48.463513+0000", 2024-08-19T14:46:34.780 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:34.780 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.780 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.780 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:46:34.780 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:40:48.463513+0000", 2024-08-19T14:46:34.780 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:34.780 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.780 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.780 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:46:34.780 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:40:48.463514+0000", 2024-08-19T14:46:34.780 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.004e-06 2024-08-19T14:46:34.780 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.780 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.781 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:46:34.781 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:40:48.463544+0000", 2024-08-19T14:46:34.781 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.04e-05 2024-08-19T14:46:34.781 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.781 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.781 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:46:34.781 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:40:48.463557+0000", 2024-08-19T14:46:34.781 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2952000000000001e-05 2024-08-19T14:46:34.781 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.781 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.781 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:46:34.781 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:40:48.463565+0000", 2024-08-19T14:46:34.781 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.3300000000000001e-06 2024-08-19T14:46:34.782 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.782 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.782 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:46:34.782 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:40:48.463637+0000", 2024-08-19T14:46:34.782 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.2792000000000004e-05 2024-08-19T14:46:34.782 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.782 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.782 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:46:34.783 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:34.783 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:46:34.783 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:40:48.463649+0000", 2024-08-19T14:46:34.783 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1593999999999999e-05 2024-08-19T14:46:34.783 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.783 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.784 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:46:34.784 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:40:49.445626+0000", 2024-08-19T14:46:34.784 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.98197673699999999 2024-08-19T14:46:34.784 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.784 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.784 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:46:34.784 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:40:49.445631+0000", 2024-08-19T14:46:34.784 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.7130000000000002e-06 2024-08-19T14:46:34.784 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:34.784 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:46:34.784 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:34.784 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.784 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.784 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26911.0:168663 8.9 8:956ac69e:::benchmark_data_smithi045_137103_object56199:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e776)", 2024-08-19T14:46:34.785 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:42:37.166778+0000", 2024-08-19T14:46:34.785 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 237.48564790899999, 2024-08-19T14:46:34.785 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.24967547100000001, 2024-08-19T14:46:34.785 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:46:34.785 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:46:34.785 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:46:34.785 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:46:34.785 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26911", 2024-08-19T14:46:34.785 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3494993995", 2024-08-19T14:46:34.785 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 168663 2024-08-19T14:46:34.785 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.785 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:46:34.785 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.785 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:46:34.786 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:42:37.166778+0000", 2024-08-19T14:46:34.786 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:34.786 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.786 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.786 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:46:34.786 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:42:37.166778+0000", 2024-08-19T14:46:34.786 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:34.786 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.786 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.786 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:46:34.786 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:42:37.166779+0000", 2024-08-19T14:46:34.786 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.5000000000000003e-07 2024-08-19T14:46:34.786 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.786 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.787 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:46:34.787 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:42:37.166782+0000", 2024-08-19T14:46:34.787 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.6770000000000001e-06 2024-08-19T14:46:34.787 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.787 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.787 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:46:34.787 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:42:37.166791+0000", 2024-08-19T14:46:34.787 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.6030000000000005e-06 2024-08-19T14:46:34.787 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.787 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.787 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:46:34.787 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:42:37.166797+0000", 2024-08-19T14:46:34.787 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.6969999999999998e-06 2024-08-19T14:46:34.788 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.788 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.788 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:46:34.788 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:42:37.171254+0000", 2024-08-19T14:46:34.788 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0044574380000000002 2024-08-19T14:46:34.788 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.788 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.788 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:46:34.788 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:42:37.171263+0000", 2024-08-19T14:46:34.788 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.1730000000000001e-06 2024-08-19T14:46:34.788 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.788 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.788 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:46:34.788 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:42:37.395562+0000", 2024-08-19T14:46:34.788 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.22429860300000001 2024-08-19T14:46:34.789 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.789 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.789 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:46:34.789 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:42:37.395657+0000", 2024-08-19T14:46:34.789 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.5420999999999993e-05 2024-08-19T14:46:34.789 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.789 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.789 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 4,5,9", 2024-08-19T14:46:34.789 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:42:37.395754+0000", 2024-08-19T14:46:34.789 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.6797000000000002e-05 2024-08-19T14:46:34.789 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.789 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.789 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:46:34.789 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:42:37.401298+0000", 2024-08-19T14:46:34.790 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0055443860000000001 2024-08-19T14:46:34.790 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.790 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.790 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:46:34.790 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:42:37.402150+0000", 2024-08-19T14:46:34.790 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00085115699999999996 2024-08-19T14:46:34.790 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.790 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.790 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:46:34.790 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:42:37.406364+0000", 2024-08-19T14:46:34.790 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0042144599999999997 2024-08-19T14:46:34.790 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.790 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.790 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:46:34.791 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:42:37.415779+0000", 2024-08-19T14:46:34.791 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0094150589999999999 2024-08-19T14:46:34.791 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.791 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.791 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:46:34.791 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:42:37.415817+0000", 2024-08-19T14:46:34.791 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7483e-05 2024-08-19T14:46:34.791 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:34.791 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:34.791 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:46:34.791 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:42:37.416454+0000", 2024-08-19T14:46:34.791 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000636967 2024-08-19T14:46:34.791 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:34.791 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:46:34.792 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:34.792 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:34.792 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:46:34.792 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:46:34.883 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:46:34.906 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:34.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:34.907 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:46:35.008 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:46:35.029 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:35.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:35.030 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:46:35.131 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:46:35.152 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:35.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:35.153 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:46:35.254 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:46:35.275 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:35.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:35.276 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:46:35.316 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is set 2024-08-19T14:46:35.377 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:46:35.403 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:35.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:35.404 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:46:35.505 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:46:35.543 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:35.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:35.544 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:46:35.576 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:35 smithi031 ceph-mon[16336]: osdmap e885: 12 total, 12 up, 8 in 2024-08-19T14:46:35.576 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:35 smithi031 ceph-mon[16336]: pgmap v2085: 47 pgs: 1 active+undersized+remapped, 2 active+recovering+undersized+remapped, 44 active+clean; 3.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 707 op/s; 3449/186117 objects misplaced (1.853%); 16 MiB/s, 257 objects/s recovering 2024-08-19T14:46:35.576 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:35 smithi031 ceph-mon[16336]: 1.14 scrub starts 2024-08-19T14:46:35.576 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:35 smithi031 ceph-mon[16336]: 1.14 scrub ok 2024-08-19T14:46:35.576 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:35 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3524862316' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:46:35.645 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:46:35.667 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:35.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:35.668 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:46:35.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:35 smithi032 ceph-mon[20004]: osdmap e885: 12 total, 12 up, 8 in 2024-08-19T14:46:35.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:35 smithi032 ceph-mon[20004]: pgmap v2085: 47 pgs: 1 active+undersized+remapped, 2 active+recovering+undersized+remapped, 44 active+clean; 3.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 707 op/s; 3449/186117 objects misplaced (1.853%); 16 MiB/s, 257 objects/s recovering 2024-08-19T14:46:35.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:35 smithi032 ceph-mon[20004]: 1.14 scrub starts 2024-08-19T14:46:35.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:35 smithi032 ceph-mon[20004]: 1.14 scrub ok 2024-08-19T14:46:35.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:35 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3524862316' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:46:35.769 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:46:35.791 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:35.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:35.792 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:46:35.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:35 smithi045 ceph-mon[20143]: osdmap e885: 12 total, 12 up, 8 in 2024-08-19T14:46:35.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:35 smithi045 ceph-mon[20143]: pgmap v2085: 47 pgs: 1 active+undersized+remapped, 2 active+recovering+undersized+remapped, 44 active+clean; 3.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 707 op/s; 3449/186117 objects misplaced (1.853%); 16 MiB/s, 257 objects/s recovering 2024-08-19T14:46:35.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:35 smithi045 ceph-mon[20143]: 1.14 scrub starts 2024-08-19T14:46:35.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:35 smithi045 ceph-mon[20143]: 1.14 scrub ok 2024-08-19T14:46:35.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:35 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3524862316' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:46:35.893 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:46:35.937 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:35.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:35.938 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:46:36.039 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:46:36.063 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:36.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:36.064 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:46:36.165 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:46:36.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:36.203 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:36.204 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:46:36.299 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-08-19T14:46:36.304 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:46:36.326 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:36.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:36.327 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:46:36.428 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:46:36.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:36.467 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:36.468 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:46:36.568 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:46:36.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:36.592 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:36.592 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:46:36.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:36 smithi031 ceph-mon[16336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:46:36.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:36 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3524862316' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:46:36.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:36 smithi031 ceph-mon[16336]: osdmap e886: 12 total, 12 up, 8 in 2024-08-19T14:46:36.693 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:46:36.715 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:36.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:36.716 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:46:36.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:36 smithi032 ceph-mon[20004]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:46:36.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:36 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3524862316' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:46:36.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:36 smithi032 ceph-mon[20004]: osdmap e886: 12 total, 12 up, 8 in 2024-08-19T14:46:36.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:36 smithi045 ceph-mon[20143]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:46:36.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:36 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3524862316' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:46:36.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:36 smithi045 ceph-mon[20143]: osdmap e886: 12 total, 12 up, 8 in 2024-08-19T14:46:36.817 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:46:36.862 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:36.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:36.864 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:46:36.965 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:46:36.991 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:36.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:36.992 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:46:37.041 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:46:37.093 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:46:37.131 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:37.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:37.132 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:46:37.233 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:46:37.256 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:37.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:37.258 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:46:37.359 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:46:37.404 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:37.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:37.405 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:46:37.506 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:46:37.530 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:37.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:37.531 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:46:37.595 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:37 smithi031 ceph-mon[16336]: pgmap v2087: 47 pgs: 1 active+undersized+remapped, 2 active+recovering+undersized+remapped, 44 active+clean; 3.7 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 859 op/s; 2750/180597 objects misplaced (1.523%); 21 MiB/s, 343 objects/s recovering 2024-08-19T14:46:37.610 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:46:37.624 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:46:37.632 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:46:37.699 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:37.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:37.701 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:46:37.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:37 smithi032 ceph-mon[20004]: pgmap v2087: 47 pgs: 1 active+undersized+remapped, 2 active+recovering+undersized+remapped, 44 active+clean; 3.7 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 859 op/s; 2750/180597 objects misplaced (1.523%); 21 MiB/s, 343 objects/s recovering 2024-08-19T14:46:37.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:37 smithi045 ceph-mon[20143]: pgmap v2087: 47 pgs: 1 active+undersized+remapped, 2 active+recovering+undersized+remapped, 44 active+clean; 3.7 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 859 op/s; 2750/180597 objects misplaced (1.523%); 21 MiB/s, 343 objects/s recovering 2024-08-19T14:46:37.802 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:46:37.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:37.844 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:37.844 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:46:37.945 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:46:37.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:37.971 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:37.971 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:46:38.072 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:46:38.096 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:38.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:38.097 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:46:38.198 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:46:38.199 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:46:38.199 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [ 2024-08-19T14:46:38.199 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:38.199 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.27235.0:131437 9.2 9:4c4fd4a9:::benchmark_data_smithi045_152862_object36932:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e886)", 2024-08-19T14:46:38.200 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:46:38.194766+0000", 2024-08-19T14:46:38.200 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.0022256139999999999, 2024-08-19T14:46:38.200 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.002247013, 2024-08-19T14:46:38.200 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:46:38.200 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:46:38.200 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "waiting for sub ops", 2024-08-19T14:46:38.200 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:46:38.200 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.27235", 2024-08-19T14:46:38.200 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2746512116", 2024-08-19T14:46:38.200 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 131437 2024-08-19T14:46:38.200 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:38.200 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:46:38.201 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:38.201 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:46:38.201 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:38.194766+0000", 2024-08-19T14:46:38.201 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:38.201 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:38.201 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:38.201 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:46:38.201 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:38.194766+0000", 2024-08-19T14:46:38.201 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:38.202 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:38.202 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:38.202 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:46:38.202 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:38.194768+0000", 2024-08-19T14:46:38.202 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.221e-06 2024-08-19T14:46:38.202 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:38.202 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:38.202 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:46:38.202 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:38.194773+0000", 2024-08-19T14:46:38.202 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1789999999999999e-06 2024-08-19T14:46:38.202 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:38.202 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:38.202 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:46:38.202 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:38.194784+0000", 2024-08-19T14:46:38.203 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0969e-05 2024-08-19T14:46:38.203 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:38.203 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:38.203 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:46:38.203 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:38.194791+0000", 2024-08-19T14:46:38.203 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.17e-06 2024-08-19T14:46:38.203 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:38.203 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:38.203 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:46:38.203 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:38.194848+0000", 2024-08-19T14:46:38.203 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.6543999999999999e-05 2024-08-19T14:46:38.203 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:38.203 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:38.203 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:46:38.204 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:38.194968+0000", 2024-08-19T14:46:38.204 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000120803 2024-08-19T14:46:38.204 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:38.204 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:38.204 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 9,11", 2024-08-19T14:46:38.204 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:38.195085+0000", 2024-08-19T14:46:38.204 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000116678 2024-08-19T14:46:38.204 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:38.204 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:46:38.204 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:38.204 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:38.204 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:38.204 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.27235.0:131424 9.d 9:bdc0431d:::benchmark_data_smithi045_152862_object36919:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e886)", 2024-08-19T14:46:38.205 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:46:38.183727+0000", 2024-08-19T14:46:38.205 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.013264836, 2024-08-19T14:46:38.205 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.013346304, 2024-08-19T14:46:38.205 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:46:38.205 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:46:38.205 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "waiting for sub ops", 2024-08-19T14:46:38.205 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:46:38.205 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.27235", 2024-08-19T14:46:38.205 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2746512116", 2024-08-19T14:46:38.205 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 131424 2024-08-19T14:46:38.205 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:38.205 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:46:38.205 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:38.205 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:46:38.206 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:38.183727+0000", 2024-08-19T14:46:38.206 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:38.206 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:38.206 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:38.206 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:46:38.206 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:38.183727+0000", 2024-08-19T14:46:38.206 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:38.206 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:38.206 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:38.206 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:46:38.206 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:38.183728+0000", 2024-08-19T14:46:38.206 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.24e-06 2024-08-19T14:46:38.207 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:38.207 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:38.207 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:46:38.207 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:38.183733+0000", 2024-08-19T14:46:38.207 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4259999999999996e-06 2024-08-19T14:46:38.207 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:38.207 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:38.207 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:46:38.207 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:38.183744+0000", 2024-08-19T14:46:38.207 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.08e-05 2024-08-19T14:46:38.207 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:38.207 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:38.207 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:46:38.207 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:38.183751+0000", 2024-08-19T14:46:38.208 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.2969999999999998e-06 2024-08-19T14:46:38.208 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:38.208 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:38.208 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:46:38.208 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:38.190376+0000", 2024-08-19T14:46:38.208 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0066248970000000003 2024-08-19T14:46:38.208 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:38.208 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:38.208 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:46:38.208 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:38.190486+0000", 2024-08-19T14:46:38.208 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000110207 2024-08-19T14:46:38.208 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:38.208 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:38.209 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 4,8", 2024-08-19T14:46:38.209 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:38.190594+0000", 2024-08-19T14:46:38.209 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000107425 2024-08-19T14:46:38.209 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:38.209 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:38.209 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:46:38.209 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:38.194625+0000", 2024-08-19T14:46:38.209 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0040313930000000003 2024-08-19T14:46:38.209 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:38.209 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:46:38.209 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:38.209 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:38.209 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:38.209 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.27235.0:131425 9.d 9:b7b79ee4:::benchmark_data_smithi045_152862_object36920:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e886)", 2024-08-19T14:46:38.209 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:46:38.183775+0000", 2024-08-19T14:46:38.210 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.013216921, 2024-08-19T14:46:38.210 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.013347593, 2024-08-19T14:46:38.210 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:46:38.210 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:46:38.210 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "waiting for sub ops", 2024-08-19T14:46:38.210 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:46:38.210 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.27235", 2024-08-19T14:46:38.210 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2746512116", 2024-08-19T14:46:38.210 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 131425 2024-08-19T14:46:38.210 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:38.210 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:46:38.211 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:38.211 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:46:38.211 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:38.183775+0000", 2024-08-19T14:46:38.211 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:38.211 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:38.211 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:38.211 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:46:38.211 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:38.183775+0000", 2024-08-19T14:46:38.211 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:38.211 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:38.211 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:38.211 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:46:38.211 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:38.183776+0000", 2024-08-19T14:46:38.211 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0500000000000004e-07 2024-08-19T14:46:38.212 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:38.212 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:38.212 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:46:38.212 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:38.183778+0000", 2024-08-19T14:46:38.212 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.8119999999999999e-06 2024-08-19T14:46:38.212 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:38.212 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:38.212 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:46:38.212 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:38.183786+0000", 2024-08-19T14:46:38.212 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.6790000000000005e-06 2024-08-19T14:46:38.212 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:38.212 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:38.212 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:46:38.212 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:38.183791+0000", 2024-08-19T14:46:38.212 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.2970000000000003e-06 2024-08-19T14:46:38.213 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:38.213 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:38.213 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:46:38.213 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:38.194666+0000", 2024-08-19T14:46:38.213 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.010874701000000001 2024-08-19T14:46:38.213 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:38.213 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:38.213 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:46:38.213 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:38.194762+0000", 2024-08-19T14:46:38.213 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.5818999999999999e-05 2024-08-19T14:46:38.213 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:38.213 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:38.213 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 4,8", 2024-08-19T14:46:38.213 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:38.194892+0000", 2024-08-19T14:46:38.214 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012964199999999999 2024-08-19T14:46:38.214 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:38.214 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:46:38.214 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:38.214 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:38.214 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:38.214 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.27235.0:131426 9.2 9:41fe4204:::benchmark_data_smithi045_152862_object36921:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e886)", 2024-08-19T14:46:38.214 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:46:38.184165+0000", 2024-08-19T14:46:38.214 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.012826601, 2024-08-19T14:46:38.214 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.013000124, 2024-08-19T14:46:38.214 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:46:38.214 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:46:38.215 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "waiting for sub ops", 2024-08-19T14:46:38.215 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:46:38.215 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.27235", 2024-08-19T14:46:38.215 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2746512116", 2024-08-19T14:46:38.215 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 131426 2024-08-19T14:46:38.215 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:38.215 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:46:38.215 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:38.215 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:46:38.215 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:38.184165+0000", 2024-08-19T14:46:38.215 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:38.215 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:38.215 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:38.215 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:46:38.216 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:38.184165+0000", 2024-08-19T14:46:38.216 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:38.216 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:38.216 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:38.216 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:46:38.216 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:38.184166+0000", 2024-08-19T14:46:38.216 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.9400000000000004e-07 2024-08-19T14:46:38.216 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:38.216 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:38.216 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:46:38.216 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:38.184170+0000", 2024-08-19T14:46:38.216 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.0740000000000003e-06 2024-08-19T14:46:38.216 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:38.217 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:38.217 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:46:38.217 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:38.184182+0000", 2024-08-19T14:46:38.217 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1405e-05 2024-08-19T14:46:38.217 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:38.217 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:38.217 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:46:38.217 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:38.184189+0000", 2024-08-19T14:46:38.217 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.1289999999999998e-06 2024-08-19T14:46:38.217 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:38.217 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:38.217 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:46:38.217 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:38.184997+0000", 2024-08-19T14:46:38.217 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00080840600000000001 2024-08-19T14:46:38.218 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:38.218 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:38.218 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:46:38.218 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:38.185128+0000", 2024-08-19T14:46:38.218 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013111199999999999 2024-08-19T14:46:38.218 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:38.218 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:38.218 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 9,11", 2024-08-19T14:46:38.218 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:38.185250+0000", 2024-08-19T14:46:38.218 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000121902 2024-08-19T14:46:38.218 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:38.218 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:38.218 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:46:38.219 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:38.192761+0000", 2024-08-19T14:46:38.219 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0075102820000000001 2024-08-19T14:46:38.219 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:38.219 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:46:38.219 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:38.219 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:38.219 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:38.219 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.27235.0:131427 9.d 9.91f8c0cd (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e886)", 2024-08-19T14:46:38.219 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:46:38.184686+0000", 2024-08-19T14:46:38.219 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.012306345999999999, 2024-08-19T14:46:38.219 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0125217, 2024-08-19T14:46:38.219 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:46:38.219 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:46:38.219 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "queued for pg", 2024-08-19T14:46:38.220 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:46:38.220 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.27235", 2024-08-19T14:46:38.220 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2746512116", 2024-08-19T14:46:38.220 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 131427 2024-08-19T14:46:38.220 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:38.220 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:46:38.220 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:38.220 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:46:38.220 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:38.184686+0000", 2024-08-19T14:46:38.220 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:38.220 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:38.220 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:38.220 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:46:38.220 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:38.184686+0000", 2024-08-19T14:46:38.221 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:38.221 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:38.221 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:38.221 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:46:38.221 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:38.184687+0000", 2024-08-19T14:46:38.221 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.4099999999999997e-07 2024-08-19T14:46:38.221 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:38.221 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:38.221 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:46:38.221 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:38.184692+0000", 2024-08-19T14:46:38.221 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1290000000000004e-06 2024-08-19T14:46:38.221 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:38.221 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:38.221 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:46:38.222 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:38.184703+0000", 2024-08-19T14:46:38.222 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0859000000000001e-05 2024-08-19T14:46:38.222 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:38.222 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:38.222 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:46:38.222 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:38.184710+0000", 2024-08-19T14:46:38.222 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.272e-06 2024-08-19T14:46:38.222 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:38.222 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:46:38.222 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:38.222 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:38.222 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:38.222 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.27235.0:131429 9.2 9:48d5c19d:::benchmark_data_smithi045_152862_object36924:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e886)", 2024-08-19T14:46:38.223 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:46:38.185082+0000", 2024-08-19T14:46:38.223 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.011910106, 2024-08-19T14:46:38.223 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.012156639, 2024-08-19T14:46:38.223 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:46:38.223 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:46:38.223 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "waiting for sub ops", 2024-08-19T14:46:38.223 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:46:38.223 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.27235", 2024-08-19T14:46:38.223 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2746512116", 2024-08-19T14:46:38.223 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 131429 2024-08-19T14:46:38.223 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:38.223 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:46:38.223 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:38.223 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:46:38.224 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:38.185082+0000", 2024-08-19T14:46:38.224 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:38.224 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:38.224 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:38.224 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:46:38.224 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:38.185082+0000", 2024-08-19T14:46:38.224 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:38.224 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:38.224 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:38.224 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:46:38.224 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:38.185083+0000", 2024-08-19T14:46:38.224 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.85e-07 2024-08-19T14:46:38.224 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:38.224 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:38.225 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:46:38.225 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:38.185087+0000", 2024-08-19T14:46:38.225 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7310000000000001e-06 2024-08-19T14:46:38.225 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:38.225 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:38.225 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:46:38.225 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:38.185102+0000", 2024-08-19T14:46:38.225 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4718999999999999e-05 2024-08-19T14:46:38.225 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:38.225 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:38.225 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:46:38.225 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:38.185109+0000", 2024-08-19T14:46:38.225 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.2439999999999996e-06 2024-08-19T14:46:38.225 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:38.225 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:38.226 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:46:38.226 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:38.188667+0000", 2024-08-19T14:46:38.226 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.003557274 2024-08-19T14:46:38.226 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:38.226 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:38.226 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:46:38.226 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:38.188767+0000", 2024-08-19T14:46:38.226 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00010032 2024-08-19T14:46:38.226 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:38.226 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:38.226 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 9,11", 2024-08-19T14:46:38.226 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:38.188887+0000", 2024-08-19T14:46:38.226 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00011990899999999999 2024-08-19T14:46:38.226 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:38.227 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:38.227 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:46:38.227 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:38.192773+0000", 2024-08-19T14:46:38.227 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0038861379999999999 2024-08-19T14:46:38.227 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:38.227 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:46:38.227 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:38.227 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:38.227 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-08-19T14:46:38.227 INFO:teuthology.orchestra.run.smithi031.stdout: "num_ops": 6 2024-08-19T14:46:38.227 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:46:38.233 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:38.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:38.235 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:46:38.334 INFO:teuthology.orchestra.run.smithi031.stderr:reweighted osd.11 to 1 (10000) 2024-08-19T14:46:38.336 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:46:38.358 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:38.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:38.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:46:38.439 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:38 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1544022378' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-08-19T14:46:38.439 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:38 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-08-19T14:46:38.459 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:46:38.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:38.504 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:38.504 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:46:38.605 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:46:38.629 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:38.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:38.631 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:46:38.732 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:46:38.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:38 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1544022378' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-08-19T14:46:38.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:38 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-08-19T14:46:38.758 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:38.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:38.759 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:46:38.795 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:46:38.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:38 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1544022378' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-08-19T14:46:38.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:38 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-08-19T14:46:38.860 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:46:38.898 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:38.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:38.899 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:46:39.000 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:46:39.014 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-19T14:46:39.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:39.033 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:39.033 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:46:39.134 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:46:39.156 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:39.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:39.157 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:46:39.258 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:46:39.260 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd reweight 9 1 2024-08-19T14:46:39.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:39.292 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:39.292 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:46:39.393 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:46:39.420 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:39.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:39.421 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:46:39.522 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:46:39.530 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:39 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-08-19T14:46:39.530 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:39 smithi031 ceph-mon[16336]: osdmap e887: 12 total, 12 up, 8 in 2024-08-19T14:46:39.530 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:39 smithi031 ceph-mon[16336]: pgmap v2089: 47 pgs: 1 active+undersized+remapped, 2 active+recovering+undersized+remapped, 44 active+clean; 3.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 707 op/s; 2750/179268 objects misplaced (1.534%); 15 MiB/s, 242 objects/s recovering 2024-08-19T14:46:39.543 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:39.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:39.545 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:46:39.646 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:46:39.670 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:39.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:39.671 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:46:39.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:39 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-08-19T14:46:39.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:39 smithi032 ceph-mon[20004]: osdmap e887: 12 total, 12 up, 8 in 2024-08-19T14:46:39.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:39 smithi032 ceph-mon[20004]: pgmap v2089: 47 pgs: 1 active+undersized+remapped, 2 active+recovering+undersized+remapped, 44 active+clean; 3.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 707 op/s; 2750/179268 objects misplaced (1.534%); 15 MiB/s, 242 objects/s recovering 2024-08-19T14:46:39.772 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:46:39.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:39 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-08-19T14:46:39.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:39 smithi045 ceph-mon[20143]: osdmap e887: 12 total, 12 up, 8 in 2024-08-19T14:46:39.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:39 smithi045 ceph-mon[20143]: pgmap v2089: 47 pgs: 1 active+undersized+remapped, 2 active+recovering+undersized+remapped, 44 active+clean; 3.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 707 op/s; 2750/179268 objects misplaced (1.534%); 15 MiB/s, 242 objects/s recovering 2024-08-19T14:46:39.802 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:46:39.802 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:46:39.805 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:39.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:39.806 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:46:39.828 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:46:39.828 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:46:39.848 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:46:39.849 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:46:39.872 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:46:39.872 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:46:39.892 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:46:39.892 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:46:39.906 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:46:39.914 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:46:39.914 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:46:39.929 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:39.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:39.930 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:46:39.937 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:46:39.937 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:46:39.955 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:46:39.955 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:46:39.979 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:46:39.979 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:46:40.002 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:46:40.002 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:46:40.025 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:46:40.026 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:46:40.030 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:46:40.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:40.054 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:40.054 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:46:40.055 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:46:40.055 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:46:40.155 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:46:40.189 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:40.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:40.190 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:46:40.272 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-08-19T14:46:40.291 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:46:40.327 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:40.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:40.328 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:46:40.429 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:46:40.450 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:40.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:40.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:46:40.531 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:46:40.541 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:46:40.552 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:46:40.591 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:40.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:40.592 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:46:40.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:40 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2710410616' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:46:40.693 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:46:40.715 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:40.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:40.716 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:46:40.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:40 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2710410616' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:46:40.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:40 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2710410616' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:46:40.817 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:46:40.864 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:40.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:40.865 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:46:40.966 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:46:40.989 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:40.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:40.990 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:46:41.091 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:46:41.116 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:41.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:41.117 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:46:41.218 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:46:41.247 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:41.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:41.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:46:41.349 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:46:41.383 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:41.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:41.384 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:46:41.466 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:46:41.485 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:46:41.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:41.522 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:41.522 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:46:41.600 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:41 smithi031 ceph-mon[16336]: pgmap v2090: 47 pgs: 2 active+recovering+undersized+remapped, 45 active+clean; 3.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 757 op/s; 2499/172320 objects misplaced (1.450%); 9.8 MiB/s, 156 objects/s recovering 2024-08-19T14:46:41.623 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:46:41.661 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:41.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:41.662 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:46:41.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:41 smithi032 ceph-mon[20004]: pgmap v2090: 47 pgs: 2 active+recovering+undersized+remapped, 45 active+clean; 3.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 757 op/s; 2499/172320 objects misplaced (1.450%); 9.8 MiB/s, 156 objects/s recovering 2024-08-19T14:46:41.763 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:46:41.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:41 smithi045 ceph-mon[20143]: pgmap v2090: 47 pgs: 2 active+recovering+undersized+remapped, 45 active+clean; 3.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 757 op/s; 2499/172320 objects misplaced (1.450%); 9.8 MiB/s, 156 objects/s recovering 2024-08-19T14:46:41.807 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:41.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:41.808 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:46:41.909 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:46:41.947 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:41.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:41.948 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:46:42.007 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:46:42.007 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:46:42.049 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:46:42.072 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:42.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:42.073 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:46:42.174 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:46:42.197 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:42.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:42.198 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:46:42.298 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:46:42.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:42.339 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:42.340 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:46:42.368 INFO:teuthology.orchestra.run.smithi031.stderr:reweighted osd.9 to 1 (10000) 2024-08-19T14:46:42.371 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is set 2024-08-19T14:46:42.440 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:46:42.474 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:42.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:42.475 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:46:42.576 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:46:42.599 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:42.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:42.600 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:46:42.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:42 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 2 pgs undersized (PG_DEGRADED) 2024-08-19T14:46:42.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:42 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3220980897' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:46:42.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:42 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:46:42.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:42 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/4268444703' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-08-19T14:46:42.701 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:46:42.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:42.728 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:42.728 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:46:42.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:42 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 2 pgs undersized (PG_DEGRADED) 2024-08-19T14:46:42.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:42 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3220980897' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:46:42.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:42 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:46:42.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:42 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/4268444703' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-08-19T14:46:42.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:42 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 2 pgs undersized (PG_DEGRADED) 2024-08-19T14:46:42.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:42 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3220980897' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:46:42.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:42 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:46:42.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:42 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/4268444703' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-08-19T14:46:42.829 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:46:42.851 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:42.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:42.852 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:46:42.953 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:46:42.975 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:42.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:42.976 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:46:43.077 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:46:43.100 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:43.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:43.101 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:46:43.105 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-08-19T14:46:43.202 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:46:43.225 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:43.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:43.227 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:46:43.328 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:46:43.369 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:43.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:43.371 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:46:43.471 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:46:43.495 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:43.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:43.496 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:46:43.597 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:46:43.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:43 smithi031 ceph-mon[16336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:46:43.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:43 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:46:43.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:43 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/4268444703' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-08-19T14:46:43.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:43 smithi031 ceph-mon[16336]: osdmap e888: 12 total, 12 up, 8 in 2024-08-19T14:46:43.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:43 smithi031 ceph-mon[16336]: pgmap v2092: 47 pgs: 2 active+recovering+undersized+remapped, 45 active+clean; 3.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 729 op/s; 2298/169248 objects misplaced (1.358%); 10 MiB/s, 162 objects/s recovering 2024-08-19T14:46:43.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:43.639 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:43.639 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:46:43.740 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:46:43.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:43 smithi032 ceph-mon[20004]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:46:43.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:43 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:46:43.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:43 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/4268444703' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-08-19T14:46:43.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:43 smithi032 ceph-mon[20004]: osdmap e888: 12 total, 12 up, 8 in 2024-08-19T14:46:43.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:43 smithi032 ceph-mon[20004]: pgmap v2092: 47 pgs: 2 active+recovering+undersized+remapped, 45 active+clean; 3.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 729 op/s; 2298/169248 objects misplaced (1.358%); 10 MiB/s, 162 objects/s recovering 2024-08-19T14:46:43.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:43.767 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:43.767 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:46:43.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:43 smithi045 ceph-mon[20143]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:46:43.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:43 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:46:43.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:43 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/4268444703' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-08-19T14:46:43.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:43 smithi045 ceph-mon[20143]: osdmap e888: 12 total, 12 up, 8 in 2024-08-19T14:46:43.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:43 smithi045 ceph-mon[20143]: pgmap v2092: 47 pgs: 2 active+recovering+undersized+remapped, 45 active+clean; 3.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 729 op/s; 2298/169248 objects misplaced (1.358%); 10 MiB/s, 162 objects/s recovering 2024-08-19T14:46:43.868 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:46:43.890 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:43.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:43.891 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:46:43.992 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:46:44.015 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:44.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:44.016 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:46:44.117 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:46:44.142 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:44.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:44.143 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:46:44.242 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd reweight 10 1 2024-08-19T14:46:44.244 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:46:44.266 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:44.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:44.267 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:46:44.368 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:46:44.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:44.392 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:44.392 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:46:44.493 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:46:44.515 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:44.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:44.516 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:46:44.617 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:46:44.653 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:44.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:44.654 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:46:44.754 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:46:44.758 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:46:44.786 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:44.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:44.787 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:46:44.796 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:46:44.887 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:46:44.927 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:44.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:44.928 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:46:45.029 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:46:45.053 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:45.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:45.054 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:46:45.155 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:46:45.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:45.204 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:45.204 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:46:45.305 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:46:45.329 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:45.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:45.330 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:46:45.430 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:46:45.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:45.460 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:45.460 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:46:45.475 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:46:45.475 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:46:45.562 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:46:45.589 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:45.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:45.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:46:45.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:45 smithi031 ceph-mon[16336]: pgmap v2093: 47 pgs: 2 active+recovering+undersized+remapped, 45 active+clean; 3.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 735 op/s; 2298/162939 objects misplaced (1.410%); 3.5 MiB/s, 56 objects/s recovering 2024-08-19T14:46:45.692 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:46:45.715 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:45.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:45.716 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:46:45.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:45 smithi032 ceph-mon[20004]: pgmap v2093: 47 pgs: 2 active+recovering+undersized+remapped, 45 active+clean; 3.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 735 op/s; 2298/162939 objects misplaced (1.410%); 3.5 MiB/s, 56 objects/s recovering 2024-08-19T14:46:45.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:45 smithi045 ceph-mon[20143]: pgmap v2093: 47 pgs: 2 active+recovering+undersized+remapped, 45 active+clean; 3.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 735 op/s; 2298/162939 objects misplaced (1.410%); 3.5 MiB/s, 56 objects/s recovering 2024-08-19T14:46:45.803 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-19T14:46:45.816 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:46:45.854 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:45.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:45.855 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:46:45.956 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:46:45.978 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:45.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:45.979 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:46:46.080 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:46:46.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:46.121 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:46.121 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:46:46.222 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:46:46.249 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:46.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:46.250 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:46:46.351 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:46:46.373 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:46.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:46.374 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:46:46.424 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-08-19T14:46:46.453 INFO:teuthology.orchestra.run.smithi031.stderr:reweighted osd.10 to 1 (10000) 2024-08-19T14:46:46.475 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:46:46.512 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:46.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:46.513 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:46:46.614 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:46:46.642 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:46.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:46.643 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:46:46.704 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:46:46.729 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:46 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/248020309' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-08-19T14:46:46.744 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:46:46.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:46 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/248020309' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-08-19T14:46:46.765 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:46.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:46.766 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:46:46.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:46 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/248020309' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-08-19T14:46:46.867 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:46:46.902 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:46.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:46.903 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:46:47.005 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:46:47.028 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:47.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:47.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:46:47.129 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:46:47.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:47.173 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:47.173 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:46:47.274 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:46:47.298 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:47.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:47.299 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:46:47.384 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:46:47.400 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:46:47.423 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:46:47.439 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd reweight 6 1 2024-08-19T14:46:47.457 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:47.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:47.460 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:46:47.561 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:46:47.589 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:47.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:47.590 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:46:47.691 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:46:47.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:47.714 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:47.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:46:47.730 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:47 smithi031 ceph-mon[16336]: pgmap v2094: 47 pgs: 2 active+recovering+undersized+remapped, 45 active+clean; 3.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 939 op/s; 1776/156561 objects misplaced (1.134%); 7.6 MiB/s, 120 objects/s recovering 2024-08-19T14:46:47.730 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:47 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/248020309' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-08-19T14:46:47.730 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:47 smithi031 ceph-mon[16336]: osdmap e889: 12 total, 12 up, 8 in 2024-08-19T14:46:47.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:47 smithi032 ceph-mon[20004]: pgmap v2094: 47 pgs: 2 active+recovering+undersized+remapped, 45 active+clean; 3.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 939 op/s; 1776/156561 objects misplaced (1.134%); 7.6 MiB/s, 120 objects/s recovering 2024-08-19T14:46:47.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:47 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/248020309' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-08-19T14:46:47.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:47 smithi032 ceph-mon[20004]: osdmap e889: 12 total, 12 up, 8 in 2024-08-19T14:46:47.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:47 smithi045 ceph-mon[20143]: pgmap v2094: 47 pgs: 2 active+recovering+undersized+remapped, 45 active+clean; 3.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 939 op/s; 1776/156561 objects misplaced (1.134%); 7.6 MiB/s, 120 objects/s recovering 2024-08-19T14:46:47.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:47 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/248020309' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-08-19T14:46:47.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:47 smithi045 ceph-mon[20143]: osdmap e889: 12 total, 12 up, 8 in 2024-08-19T14:46:47.816 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:46:47.839 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:47.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:47.840 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:46:47.941 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:46:47.964 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:47.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:47.966 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:46:47.970 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:46:48.000 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:46:48.001 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:46:48.066 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:46:48.089 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:48.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:48.090 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:46:48.191 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:46:48.213 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:48.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:48.214 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:46:48.315 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:46:48.360 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:48.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:48.361 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:46:48.462 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:46:48.491 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:48.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:48.492 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:46:48.593 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:46:48.645 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:48.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:48.646 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:46:48.747 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:46:48.777 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:48.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:48.778 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:46:48.880 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:46:48.884 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:46:48.884 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:46:48.885 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:46:48.885 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:46:48.918 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:48.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:48.919 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:46:49.020 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:46:49.029 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:46:49.029 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:46:49.041 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:49.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:49.042 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:46:49.055 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:46:49.055 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:46:49.081 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:46:49.081 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:46:49.110 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:46:49.111 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:46:49.143 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:46:49.158 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:46:49.158 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:46:49.171 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:49.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:49.172 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:46:49.183 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:46:49.183 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:46:49.207 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:46:49.208 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:46:49.249 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:46:49.250 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:46:49.273 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:46:49.283 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:46:49.284 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:46:49.297 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:49.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:49.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:46:49.309 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:46:49.310 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:46:49.398 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:46:49.421 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:49.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:49.422 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:46:49.523 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:46:49.547 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:49.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:49.548 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:46:49.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:49 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:46:49.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:49 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:46:49.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:49 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:46:49.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:49 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:46:49.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:49 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:46:49.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:49 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:46:49.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:49 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:46:49.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:49 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1847610471' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:46:49.649 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:46:49.679 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:49.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:49.680 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:46:49.713 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:46:49.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:49 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:46:49.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:49 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:46:49.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:49 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:46:49.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:49 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:46:49.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:49 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:46:49.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:49 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:46:49.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:49 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:46:49.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:49 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1847610471' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:46:49.781 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:46:49.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:49 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:46:49.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:49 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:46:49.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:49 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:46:49.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:49 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:46:49.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:49 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:46:49.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:49 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:46:49.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:49 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:46:49.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:49 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1847610471' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:46:49.803 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:49.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:49.804 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:46:49.905 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:46:49.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:49.950 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:49.950 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:46:50.051 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:46:50.074 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:50.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:50.076 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:46:50.163 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-08-19T14:46:50.176 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:46:50.213 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:50.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:50.214 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:46:50.315 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:46:50.348 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:50.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:50.349 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:46:50.450 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:46:50.474 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:50.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:50.475 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:46:50.477 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is unset 2024-08-19T14:46:50.490 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:50 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:46:50.490 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:50 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:46:50.490 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:50 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2571835756' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:46:50.490 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:50 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:46:50.576 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:46:50.601 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:50.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:50.602 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:46:50.702 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:46:50.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:50.740 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:50.740 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:46:50.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:50 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:46:50.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:50 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:46:50.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:50 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2571835756' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:46:50.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:50 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:46:50.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:50 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:46:50.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:50 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:46:50.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:50 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2571835756' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:46:50.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:50 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:46:50.841 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:46:50.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:50.879 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:50.879 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:46:50.980 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:46:51.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:51.023 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:51.023 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:46:51.124 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:46:51.148 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:51.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:51.149 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:46:51.250 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:46:51.272 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:51.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:51.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:46:51.374 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:46:51.395 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:51.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:51.396 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:46:51.497 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:46:51.533 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:51.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:51.534 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:46:51.635 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:46:51.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:51.673 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:51.673 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:46:51.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:51 smithi032 ceph-mon[20004]: pgmap v2097: 47 pgs: 2 active+recovering+undersized+remapped, 45 active+clean; 3.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 910 op/s; 1440/147330 objects misplaced (0.977%); 6.7 MiB/s, 106 objects/s recovering 2024-08-19T14:46:51.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:51 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:46:51.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:51 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:46:51.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:51 smithi032 ceph-mon[20004]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:46:51.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:51 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:46:51.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:51 smithi032 ceph-mon[20004]: osdmap e890: 12 total, 12 up, 8 in 2024-08-19T14:46:51.763 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:51 smithi031 ceph-mon[16336]: pgmap v2097: 47 pgs: 2 active+recovering+undersized+remapped, 45 active+clean; 3.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 910 op/s; 1440/147330 objects misplaced (0.977%); 6.7 MiB/s, 106 objects/s recovering 2024-08-19T14:46:51.763 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:51 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:46:51.763 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:51 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:46:51.763 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:51 smithi031 ceph-mon[16336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:46:51.763 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:51 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:46:51.763 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:51 smithi031 ceph-mon[16336]: osdmap e890: 12 total, 12 up, 8 in 2024-08-19T14:46:51.774 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:46:51.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:51 smithi045 ceph-mon[20143]: pgmap v2097: 47 pgs: 2 active+recovering+undersized+remapped, 45 active+clean; 3.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 910 op/s; 1440/147330 objects misplaced (0.977%); 6.7 MiB/s, 106 objects/s recovering 2024-08-19T14:46:51.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:51 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:46:51.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:51 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:46:51.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:51 smithi045 ceph-mon[20143]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:46:51.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:51 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:46:51.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:51 smithi045 ceph-mon[20143]: osdmap e890: 12 total, 12 up, 8 in 2024-08-19T14:46:51.801 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:51.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:51.802 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:46:51.903 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:46:51.928 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:51.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:51.930 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:46:52.031 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:46:52.059 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:52.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:52.061 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:46:52.099 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:46:52.162 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:46:52.204 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:52.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:52.206 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:46:52.307 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:46:52.328 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:52.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:52.329 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:46:52.430 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:46:52.453 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:52.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:52.454 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:46:52.478 INFO:teuthology.orchestra.run.smithi031.stderr:reweighted osd.6 to 1 (10000) 2024-08-19T14:46:52.555 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:46:52.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:52.578 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:52.578 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:46:52.610 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:46:52.610 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-08-19T14:46:52.610 INFO:teuthology.orchestra.run.smithi031.stdout: "complaint_time": 30, 2024-08-19T14:46:52.610 INFO:teuthology.orchestra.run.smithi031.stdout: "num_blocked_ops": 0 2024-08-19T14:46:52.610 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:46:52.619 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:52 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2008121940' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-08-19T14:46:52.679 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:46:52.705 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:52.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:52.706 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:46:52.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:52 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2008121940' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-08-19T14:46:52.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:52 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2008121940' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-08-19T14:46:52.807 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:46:52.850 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:52.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:52.851 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:46:52.855 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:46:52.951 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:46:52.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:52.974 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:52.974 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:46:53.075 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:46:53.099 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:53.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:53.100 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:46:53.201 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:46:53.223 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:53.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:53.224 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:46:53.325 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:46:53.347 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:53.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:53.348 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:46:53.449 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:46:53.471 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:53.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:53.472 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:46:53.480 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-08-19T14:46:53.573 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:46:53.594 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:53.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:53.595 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:46:53.696 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:46:53.717 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:53.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:53.718 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:46:53.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:53 smithi032 ceph-mon[20004]: pgmap v2099: 47 pgs: 2 active+recovering+undersized+remapped, 45 active+clean; 2.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 776 op/s; 1176/144288 objects misplaced (0.815%); 8.8 MiB/s, 140 objects/s recovering 2024-08-19T14:46:53.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:53 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2008121940' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-08-19T14:46:53.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:53 smithi032 ceph-mon[20004]: osdmap e891: 12 total, 12 up, 8 in 2024-08-19T14:46:53.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:53 smithi032 ceph-mon[20004]: osdmap e892: 12 total, 12 up, 8 in 2024-08-19T14:46:53.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:53 smithi045 ceph-mon[20143]: pgmap v2099: 47 pgs: 2 active+recovering+undersized+remapped, 45 active+clean; 2.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 776 op/s; 1176/144288 objects misplaced (0.815%); 8.8 MiB/s, 140 objects/s recovering 2024-08-19T14:46:53.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:53 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2008121940' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-08-19T14:46:53.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:53 smithi045 ceph-mon[20143]: osdmap e891: 12 total, 12 up, 8 in 2024-08-19T14:46:53.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:53 smithi045 ceph-mon[20143]: osdmap e892: 12 total, 12 up, 8 in 2024-08-19T14:46:53.799 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-19T14:46:53.819 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:46:53.820 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:53 smithi031 ceph-mon[16336]: pgmap v2099: 47 pgs: 2 active+recovering+undersized+remapped, 45 active+clean; 2.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 776 op/s; 1176/144288 objects misplaced (0.815%); 8.8 MiB/s, 140 objects/s recovering 2024-08-19T14:46:53.820 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:53 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2008121940' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-08-19T14:46:53.820 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:53 smithi031 ceph-mon[16336]: osdmap e891: 12 total, 12 up, 8 in 2024-08-19T14:46:53.820 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:53 smithi031 ceph-mon[16336]: osdmap e892: 12 total, 12 up, 8 in 2024-08-19T14:46:53.840 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:53.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:53.841 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:46:53.880 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-08-19T14:46:53.880 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-19T14:46:53.943 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:46:53.964 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:53.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:53.965 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:46:53.978 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:46:54.002 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:46:54.066 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:46:54.087 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:54.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:54.088 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:46:54.189 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:46:54.221 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:54.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:54.222 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:46:54.323 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:46:54.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:54.345 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:54.345 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:46:54.446 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:46:54.473 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:54.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:54.474 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:46:54.575 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:46:54.598 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:54.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:54.599 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:46:54.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:54 smithi031 ceph-mon[16336]: 1.13 scrub starts 2024-08-19T14:46:54.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:54 smithi031 ceph-mon[16336]: 1.13 scrub ok 2024-08-19T14:46:54.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:54 smithi031 ceph-mon[16336]: osdmap e893: 12 total, 12 up, 8 in 2024-08-19T14:46:54.700 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:46:54.722 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:54.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:54.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:46:54.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:54 smithi032 ceph-mon[20004]: 1.13 scrub starts 2024-08-19T14:46:54.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:54 smithi032 ceph-mon[20004]: 1.13 scrub ok 2024-08-19T14:46:54.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:54 smithi032 ceph-mon[20004]: osdmap e893: 12 total, 12 up, 8 in 2024-08-19T14:46:54.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:54 smithi045 ceph-mon[20143]: 1.13 scrub starts 2024-08-19T14:46:54.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:54 smithi045 ceph-mon[20143]: 1.13 scrub ok 2024-08-19T14:46:54.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:54 smithi045 ceph-mon[20143]: osdmap e893: 12 total, 12 up, 8 in 2024-08-19T14:46:54.824 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:46:54.847 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:54.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:54.848 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:46:54.949 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:46:55.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:55.056 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:55.056 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:46:55.158 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:46:55.181 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:55.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:55.182 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:46:55.283 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:46:55.306 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:55.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:55.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:46:55.408 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:46:55.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:55.443 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:55.443 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:46:55.544 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:46:55.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:55.581 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:55.581 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:46:55.682 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:46:55.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:55.739 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:55.739 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:46:55.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:55 smithi032 ceph-mon[20004]: pgmap v2103: 47 pgs: 2 active+recovering+undersized+remapped, 45 active+clean; 2.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 727 op/s; 1176/138600 objects misplaced (0.848%); 4.1 MiB/s, 65 objects/s recovering 2024-08-19T14:46:55.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:55 smithi032 ceph-mon[20004]: 1.15 scrub starts 2024-08-19T14:46:55.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:55 smithi032 ceph-mon[20004]: 1.15 scrub ok 2024-08-19T14:46:55.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:55 smithi032 ceph-mon[20004]: 1.18 scrub starts 2024-08-19T14:46:55.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:55 smithi032 ceph-mon[20004]: 1.18 scrub ok 2024-08-19T14:46:55.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:55 smithi045 ceph-mon[20143]: pgmap v2103: 47 pgs: 2 active+recovering+undersized+remapped, 45 active+clean; 2.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 727 op/s; 1176/138600 objects misplaced (0.848%); 4.1 MiB/s, 65 objects/s recovering 2024-08-19T14:46:55.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:55 smithi045 ceph-mon[20143]: 1.15 scrub starts 2024-08-19T14:46:55.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:55 smithi045 ceph-mon[20143]: 1.15 scrub ok 2024-08-19T14:46:55.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:55 smithi045 ceph-mon[20143]: 1.18 scrub starts 2024-08-19T14:46:55.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:55 smithi045 ceph-mon[20143]: 1.18 scrub ok 2024-08-19T14:46:55.841 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:46:55.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:55 smithi031 ceph-mon[16336]: pgmap v2103: 47 pgs: 2 active+recovering+undersized+remapped, 45 active+clean; 2.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 727 op/s; 1176/138600 objects misplaced (0.848%); 4.1 MiB/s, 65 objects/s recovering 2024-08-19T14:46:55.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:55 smithi031 ceph-mon[16336]: 1.15 scrub starts 2024-08-19T14:46:55.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:55 smithi031 ceph-mon[16336]: 1.15 scrub ok 2024-08-19T14:46:55.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:55 smithi031 ceph-mon[16336]: 1.18 scrub starts 2024-08-19T14:46:55.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:55 smithi031 ceph-mon[16336]: 1.18 scrub ok 2024-08-19T14:46:55.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:55.868 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:55.868 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:46:55.969 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:46:55.990 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:55.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:55.991 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:46:56.092 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:46:56.113 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:56.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:56.115 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:46:56.215 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:46:56.253 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:56.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:56.254 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:46:56.355 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:46:56.377 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:56.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:56.379 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:46:56.480 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:46:56.503 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:56.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:56.504 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:46:56.541 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:56 smithi031 ceph-mon[16336]: 1.16 scrub starts 2024-08-19T14:46:56.541 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:56 smithi031 ceph-mon[16336]: 1.16 scrub ok 2024-08-19T14:46:56.604 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:46:56.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:56.632 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:56.632 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:46:56.695 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:46:56.695 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 20, 2024-08-19T14:46:56.695 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 600, 2024-08-19T14:46:56.695 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [ 2024-08-19T14:46:56.695 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.695 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:178361 7.3 7:cda41295:::benchmark_data_smithi045_115842_object60201:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e676)", 2024-08-19T14:46:56.695 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:37:35.698169+0000", 2024-08-19T14:46:56.695 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 560.81609340399996, 2024-08-19T14:46:56.695 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4301732630000004, 2024-08-19T14:46:56.696 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:46:56.696 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:46:56.696 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:46:56.696 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:46:56.696 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:46:56.696 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:46:56.696 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 178361 2024-08-19T14:46:56.696 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.696 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:46:56.696 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.696 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:46:56.696 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:35.698169+0000", 2024-08-19T14:46:56.696 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:56.697 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.697 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.697 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:46:56.697 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:35.698169+0000", 2024-08-19T14:46:56.697 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:56.697 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.697 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.697 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:46:56.697 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:35.698169+0000", 2024-08-19T14:46:56.697 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.1800000000000002e-07 2024-08-19T14:46:56.697 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.697 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.697 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:46:56.698 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:35.698172+0000", 2024-08-19T14:46:56.698 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.6910000000000002e-06 2024-08-19T14:46:56.698 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.698 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.698 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:46:56.698 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:35.698177+0000", 2024-08-19T14:46:56.698 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.5029999999999997e-06 2024-08-19T14:46:56.698 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.698 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.698 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:46:56.698 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:35.698182+0000", 2024-08-19T14:46:56.698 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4669999999999998e-06 2024-08-19T14:46:56.698 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.699 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.699 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:46:56.699 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:35.708410+0000", 2024-08-19T14:46:56.699 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.010228568 2024-08-19T14:46:56.699 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.699 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.699 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for missing object", 2024-08-19T14:46:56.699 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:35.708502+0000", 2024-08-19T14:46:56.699 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.1527000000000006e-05 2024-08-19T14:46:56.699 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.699 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.699 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:46:56.699 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:40.128321+0000", 2024-08-19T14:46:56.700 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4198188739999997 2024-08-19T14:46:56.700 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.700 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.700 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:46:56.700 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:40.128342+0000", 2024-08-19T14:46:56.700 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.1314999999999998e-05 2024-08-19T14:46:56.700 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:56.700 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:46:56.700 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:56.700 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.700 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.700 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:178375 7.3 7:cb0f4ae1:::benchmark_data_smithi045_115842_object60215:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e676)", 2024-08-19T14:46:56.701 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:37:35.698187+0000", 2024-08-19T14:46:56.701 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 560.81607484200003, 2024-08-19T14:46:56.701 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4423218430000002, 2024-08-19T14:46:56.701 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:46:56.701 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:46:56.701 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:46:56.701 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:46:56.701 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:46:56.701 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:46:56.701 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 178375 2024-08-19T14:46:56.701 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.701 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:46:56.701 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.701 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:46:56.702 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:35.698187+0000", 2024-08-19T14:46:56.702 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:56.702 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.702 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.702 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:46:56.702 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:35.698187+0000", 2024-08-19T14:46:56.702 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:56.702 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.702 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.702 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:46:56.702 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:35.698188+0000", 2024-08-19T14:46:56.702 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.7399999999999999e-07 2024-08-19T14:46:56.702 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.703 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.703 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:46:56.703 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:35.698189+0000", 2024-08-19T14:46:56.703 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.933e-06 2024-08-19T14:46:56.703 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.703 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.703 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:46:56.703 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:35.698195+0000", 2024-08-19T14:46:56.703 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1569999999999999e-06 2024-08-19T14:46:56.703 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.703 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.703 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:46:56.703 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:35.698199+0000", 2024-08-19T14:46:56.704 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.3900000000000003e-06 2024-08-19T14:46:56.704 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.704 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.704 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:46:56.704 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:35.708531+0000", 2024-08-19T14:46:56.704 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.010332221000000001 2024-08-19T14:46:56.704 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.704 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.704 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for missing object", 2024-08-19T14:46:56.704 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:35.708601+0000", 2024-08-19T14:46:56.704 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.9558e-05 2024-08-19T14:46:56.704 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.704 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.704 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:46:56.705 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:40.140497+0000", 2024-08-19T14:46:56.705 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4318961530000003 2024-08-19T14:46:56.705 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.705 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.705 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:46:56.705 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:37:40.140509+0000", 2024-08-19T14:46:56.705 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2157000000000001e-05 2024-08-19T14:46:56.705 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:56.705 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:46:56.705 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:56.706 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.706 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.706 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:191412 7.2 7:469e5aca:::benchmark_data_smithi045_115842_object73252:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e686)", 2024-08-19T14:46:56.706 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:38:05.333032+0000", 2024-08-19T14:46:56.706 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 531.18122963600001, 2024-08-19T14:46:56.706 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.173558978, 2024-08-19T14:46:56.706 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:46:56.706 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:46:56.706 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:46:56.706 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:46:56.706 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:46:56.706 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:46:56.706 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 191412 2024-08-19T14:46:56.707 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.707 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:46:56.707 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.707 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:46:56.707 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:05.333032+0000", 2024-08-19T14:46:56.707 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:56.707 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.707 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.707 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:46:56.707 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:05.333032+0000", 2024-08-19T14:46:56.707 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:56.707 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.707 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.707 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:46:56.708 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:05.333033+0000", 2024-08-19T14:46:56.708 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.9900000000000002e-07 2024-08-19T14:46:56.708 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.708 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.708 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:46:56.708 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:05.333035+0000", 2024-08-19T14:46:56.708 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.0990000000000001e-06 2024-08-19T14:46:56.708 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.708 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.708 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:46:56.708 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:05.333040+0000", 2024-08-19T14:46:56.708 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.2310000000000004e-06 2024-08-19T14:46:56.708 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.709 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.709 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:46:56.709 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:05.333045+0000", 2024-08-19T14:46:56.709 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4100000000000001e-06 2024-08-19T14:46:56.709 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.709 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.709 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:46:56.709 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:05.344687+0000", 2024-08-19T14:46:56.709 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.011642593 2024-08-19T14:46:56.709 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.709 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.709 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for missing object", 2024-08-19T14:46:56.709 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:05.344760+0000", 2024-08-19T14:46:56.709 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.2589999999999994e-05 2024-08-19T14:46:56.710 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.710 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.710 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:46:56.710 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:12.506579+0000", 2024-08-19T14:46:56.710 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.1618195030000003 2024-08-19T14:46:56.710 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.710 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.710 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:46:56.710 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:12.506591+0000", 2024-08-19T14:46:56.710 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2252999999999999e-05 2024-08-19T14:46:56.711 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:56.711 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:46:56.711 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:56.711 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.711 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.711 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:191415 7.c 7:3e0db92c:::benchmark_data_smithi045_115842_object73255:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e686)", 2024-08-19T14:46:56.711 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:38:05.333050+0000", 2024-08-19T14:46:56.711 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 531.18121235900003, 2024-08-19T14:46:56.711 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.1806932559999996, 2024-08-19T14:46:56.711 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:46:56.711 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:46:56.712 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:46:56.712 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:46:56.712 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:46:56.712 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:46:56.712 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 191415 2024-08-19T14:46:56.712 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.712 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:46:56.712 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.712 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:46:56.712 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:05.333050+0000", 2024-08-19T14:46:56.712 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:56.712 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.712 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.712 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:46:56.713 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:05.333050+0000", 2024-08-19T14:46:56.713 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:56.713 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.713 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.713 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:46:56.713 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:05.333050+0000", 2024-08-19T14:46:56.713 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.9999999999999999e-07 2024-08-19T14:46:56.713 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.713 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.713 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:46:56.713 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:05.333052+0000", 2024-08-19T14:46:56.713 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.902e-06 2024-08-19T14:46:56.713 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.714 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.714 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:46:56.714 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:05.333057+0000", 2024-08-19T14:46:56.714 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1370000000000001e-06 2024-08-19T14:46:56.714 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.714 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.714 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:46:56.714 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:05.333062+0000", 2024-08-19T14:46:56.714 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.6280000000000004e-06 2024-08-19T14:46:56.714 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.714 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.714 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:46:56.715 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:05.333145+0000", 2024-08-19T14:46:56.715 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.3194000000000003e-05 2024-08-19T14:46:56.715 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.715 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.715 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for missing object", 2024-08-19T14:46:56.715 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:05.333242+0000", 2024-08-19T14:46:56.715 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.6707999999999996e-05 2024-08-19T14:46:56.715 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.715 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.715 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:46:56.715 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:12.513727+0000", 2024-08-19T14:46:56.715 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.1804853890000002 2024-08-19T14:46:56.715 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.716 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.716 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:46:56.716 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:12.513743+0000", 2024-08-19T14:46:56.716 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5998000000000002e-05 2024-08-19T14:46:56.716 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:56.716 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:46:56.716 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:56.716 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.716 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.716 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:200330 7.c 7:3964c84b:::benchmark_data_smithi045_115842_object82170:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e690)", 2024-08-19T14:46:56.716 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:38:24.742215+0000", 2024-08-19T14:46:56.716 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 511.77204710299998, 2024-08-19T14:46:56.716 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.544114505, 2024-08-19T14:46:56.717 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:46:56.717 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:46:56.717 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:46:56.717 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:46:56.717 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:46:56.717 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:46:56.717 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 200330 2024-08-19T14:46:56.717 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.717 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:46:56.717 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.717 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:46:56.717 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:24.742215+0000", 2024-08-19T14:46:56.717 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:56.718 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.718 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.718 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:46:56.718 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:24.742215+0000", 2024-08-19T14:46:56.718 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:56.718 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.718 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.718 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:46:56.718 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:24.742215+0000", 2024-08-19T14:46:56.718 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.9999999999999999e-07 2024-08-19T14:46:56.718 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.718 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.718 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:46:56.719 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:24.742217+0000", 2024-08-19T14:46:56.719 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.0940000000000002e-06 2024-08-19T14:46:56.719 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.719 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.719 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:46:56.719 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:24.742223+0000", 2024-08-19T14:46:56.719 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.5149999999999997e-06 2024-08-19T14:46:56.719 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.719 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.719 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:46:56.719 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:24.742227+0000", 2024-08-19T14:46:56.719 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.5120000000000002e-06 2024-08-19T14:46:56.719 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.719 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.720 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:46:56.720 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:24.745674+0000", 2024-08-19T14:46:56.720 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0034464389999999999 2024-08-19T14:46:56.720 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.720 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.720 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for missing object", 2024-08-19T14:46:56.720 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:24.745757+0000", 2024-08-19T14:46:56.720 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.3162000000000001e-05 2024-08-19T14:46:56.720 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.720 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.720 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:46:56.720 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:28.286317+0000", 2024-08-19T14:46:56.720 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.5405595189999999 2024-08-19T14:46:56.720 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.721 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.721 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:46:56.721 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:28.286330+0000", 2024-08-19T14:46:56.721 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2964e-05 2024-08-19T14:46:56.721 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:56.721 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:46:56.721 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:56.721 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.721 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.721 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:201906 7.3 7:ca923fba:::benchmark_data_smithi045_115842_object83746:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e691)", 2024-08-19T14:46:56.721 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:38:28.215922+0000", 2024-08-19T14:46:56.721 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 508.29834046000002, 2024-08-19T14:46:56.721 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.587347399, 2024-08-19T14:46:56.721 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:46:56.722 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:46:56.722 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:46:56.722 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:46:56.722 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:46:56.722 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:46:56.722 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 201906 2024-08-19T14:46:56.722 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.722 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:46:56.722 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.722 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:46:56.723 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:28.215922+0000", 2024-08-19T14:46:56.723 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:56.723 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.723 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.723 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:46:56.723 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:28.215922+0000", 2024-08-19T14:46:56.723 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:56.723 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.723 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.723 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:46:56.723 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:28.215922+0000", 2024-08-19T14:46:56.723 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.4200000000000002e-07 2024-08-19T14:46:56.723 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.723 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.724 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:46:56.724 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:28.215925+0000", 2024-08-19T14:46:56.724 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.6249999999999999e-06 2024-08-19T14:46:56.724 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.724 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.724 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:46:56.724 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:28.215930+0000", 2024-08-19T14:46:56.724 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.2499999999999997e-06 2024-08-19T14:46:56.724 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.724 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.724 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:46:56.724 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:28.215935+0000", 2024-08-19T14:46:56.724 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0690000000000001e-06 2024-08-19T14:46:56.724 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.725 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.725 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:46:56.725 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:28.244218+0000", 2024-08-19T14:46:56.725 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.028282808999999999 2024-08-19T14:46:56.725 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.725 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.725 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for missing object", 2024-08-19T14:46:56.725 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:28.244310+0000", 2024-08-19T14:46:56.725 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.2227999999999999e-05 2024-08-19T14:46:56.725 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.725 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.725 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:46:56.725 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:29.803256+0000", 2024-08-19T14:46:56.725 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5589463349999999 2024-08-19T14:46:56.726 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.726 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.726 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:46:56.726 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:29.803269+0000", 2024-08-19T14:46:56.726 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2741e-05 2024-08-19T14:46:56.726 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:56.726 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:46:56.726 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:56.726 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.726 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.727 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:201910 7.3 7:cffdc6e7:::benchmark_data_smithi045_115842_object83750:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e691)", 2024-08-19T14:46:56.727 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:38:28.215941+0000", 2024-08-19T14:46:56.727 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 508.29832083000002, 2024-08-19T14:46:56.727 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.60242383, 2024-08-19T14:46:56.727 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:46:56.727 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:46:56.727 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:46:56.727 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:46:56.727 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:46:56.727 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:46:56.727 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 201910 2024-08-19T14:46:56.727 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.727 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:46:56.728 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.728 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:46:56.728 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:28.215941+0000", 2024-08-19T14:46:56.728 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:56.728 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.728 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.728 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:46:56.728 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:28.215941+0000", 2024-08-19T14:46:56.728 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:56.728 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.728 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.728 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:46:56.728 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:28.215942+0000", 2024-08-19T14:46:56.729 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.1100000000000002e-07 2024-08-19T14:46:56.729 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.729 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.729 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:46:56.729 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:28.215944+0000", 2024-08-19T14:46:56.729 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.9700000000000002e-06 2024-08-19T14:46:56.729 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.729 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.729 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:46:56.729 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:28.215949+0000", 2024-08-19T14:46:56.729 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0969999999999996e-06 2024-08-19T14:46:56.729 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.729 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.729 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:46:56.730 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:28.215952+0000", 2024-08-19T14:46:56.730 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.8190000000000002e-06 2024-08-19T14:46:56.730 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.730 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.730 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:46:56.730 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:28.244340+0000", 2024-08-19T14:46:56.730 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.028388002999999998 2024-08-19T14:46:56.730 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.730 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.730 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for missing object", 2024-08-19T14:46:56.730 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:28.244410+0000", 2024-08-19T14:46:56.730 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.9695000000000003e-05 2024-08-19T14:46:56.730 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.731 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.731 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:46:56.731 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:29.818349+0000", 2024-08-19T14:46:56.731 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5739392670000001 2024-08-19T14:46:56.731 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.731 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.731 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:46:56.731 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:29.818365+0000", 2024-08-19T14:46:56.731 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5668e-05 2024-08-19T14:46:56.731 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:56.731 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:46:56.731 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:56.732 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.732 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.732 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:201912 7.3 7:c625da36:::benchmark_data_smithi045_115842_object83752:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e691)", 2024-08-19T14:46:56.732 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:38:28.215958+0000", 2024-08-19T14:46:56.732 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 508.29830417099998, 2024-08-19T14:46:56.732 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.607718583, 2024-08-19T14:46:56.732 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:46:56.732 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:46:56.732 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:46:56.732 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:46:56.732 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:46:56.732 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:46:56.732 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 201912 2024-08-19T14:46:56.732 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.733 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:46:56.733 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.733 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:46:56.733 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:28.215958+0000", 2024-08-19T14:46:56.733 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:56.733 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.733 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.733 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:46:56.733 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:28.215958+0000", 2024-08-19T14:46:56.733 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:56.733 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.733 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.733 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:46:56.734 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:28.215958+0000", 2024-08-19T14:46:56.734 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.6199999999999999e-07 2024-08-19T14:46:56.734 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.734 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.734 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:46:56.734 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:28.215964+0000", 2024-08-19T14:46:56.734 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.0569999999999998e-06 2024-08-19T14:46:56.734 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.734 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.734 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:46:56.734 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:28.215970+0000", 2024-08-19T14:46:56.734 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.2440000000000001e-06 2024-08-19T14:46:56.734 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.735 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.735 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:46:56.735 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:28.215974+0000", 2024-08-19T14:46:56.735 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.0999999999999997e-06 2024-08-19T14:46:56.735 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.735 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.735 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:46:56.735 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:28.244439+0000", 2024-08-19T14:46:56.735 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.028465013000000001 2024-08-19T14:46:56.735 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.735 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.735 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for missing object", 2024-08-19T14:46:56.735 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:28.244509+0000", 2024-08-19T14:46:56.735 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.0322999999999997e-05 2024-08-19T14:46:56.735 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.736 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.736 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:46:56.736 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:29.823638+0000", 2024-08-19T14:46:56.736 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.579128614 2024-08-19T14:46:56.736 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.736 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.736 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:46:56.736 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:29.823677+0000", 2024-08-19T14:46:56.736 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.8970000000000001e-05 2024-08-19T14:46:56.736 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:56.736 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:46:56.736 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:56.736 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.736 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.737 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:219824 7.3 7:c228a6f2:::benchmark_data_smithi045_115842_object101664:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e699)", 2024-08-19T14:46:56.737 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:39:05.329302+0000", 2024-08-19T14:46:56.737 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 471.18496024799998, 2024-08-19T14:46:56.737 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.7693942160000002, 2024-08-19T14:46:56.737 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:46:56.737 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:46:56.737 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:46:56.737 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:46:56.737 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:46:56.737 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:46:56.737 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 219824 2024-08-19T14:46:56.737 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.737 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:46:56.737 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.738 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:46:56.738 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:05.329302+0000", 2024-08-19T14:46:56.738 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:56.738 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.738 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.738 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:46:56.738 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:05.329302+0000", 2024-08-19T14:46:56.738 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:56.738 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.738 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.738 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:46:56.738 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:05.329302+0000", 2024-08-19T14:46:56.738 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.22e-07 2024-08-19T14:46:56.739 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.739 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.739 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:46:56.739 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:05.329304+0000", 2024-08-19T14:46:56.739 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.7910000000000001e-06 2024-08-19T14:46:56.739 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.739 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.739 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:46:56.739 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:05.329309+0000", 2024-08-19T14:46:56.739 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.057e-06 2024-08-19T14:46:56.739 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.739 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.739 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:46:56.740 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:05.329314+0000", 2024-08-19T14:46:56.740 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4920000000000004e-06 2024-08-19T14:46:56.740 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.740 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.740 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:46:56.740 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:05.329839+0000", 2024-08-19T14:46:56.740 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00052567900000000003 2024-08-19T14:46:56.740 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.740 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.740 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for missing object", 2024-08-19T14:46:56.740 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:05.329918+0000", 2024-08-19T14:46:56.740 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.8475999999999999e-05 2024-08-19T14:46:56.740 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.741 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.741 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:46:56.741 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:08.098684+0000", 2024-08-19T14:46:56.741 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.7687666489999998 2024-08-19T14:46:56.741 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.741 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.741 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:46:56.741 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:08.098696+0000", 2024-08-19T14:46:56.741 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1749999999999999e-05 2024-08-19T14:46:56.741 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:56.741 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:46:56.741 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:56.741 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.741 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.742 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:219833 7.2 7:4faf1754:::benchmark_data_smithi045_115842_object101673:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e699)", 2024-08-19T14:46:56.742 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:39:05.329376+0000", 2024-08-19T14:46:56.742 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 471.18488661800001, 2024-08-19T14:46:56.742 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.770562398, 2024-08-19T14:46:56.742 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:46:56.742 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:46:56.742 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:46:56.742 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:46:56.742 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:46:56.742 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:46:56.742 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 219833 2024-08-19T14:46:56.742 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.743 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:46:56.743 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.743 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:46:56.743 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:05.329376+0000", 2024-08-19T14:46:56.743 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:56.743 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.743 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.743 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:46:56.743 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:05.329376+0000", 2024-08-19T14:46:56.743 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:56.743 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.743 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.743 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:46:56.743 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:05.329376+0000", 2024-08-19T14:46:56.744 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.8999999999999998e-07 2024-08-19T14:46:56.744 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.744 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.744 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:46:56.744 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:05.329378+0000", 2024-08-19T14:46:56.744 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.9769999999999999e-06 2024-08-19T14:46:56.744 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.744 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.744 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:46:56.744 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:05.329383+0000", 2024-08-19T14:46:56.744 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.3809999999999998e-06 2024-08-19T14:46:56.744 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.744 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.744 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:46:56.745 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:05.329388+0000", 2024-08-19T14:46:56.745 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.5510000000000001e-06 2024-08-19T14:46:56.745 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.745 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.745 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:46:56.745 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:05.340772+0000", 2024-08-19T14:46:56.745 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.011384215 2024-08-19T14:46:56.745 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.745 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.745 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for missing object", 2024-08-19T14:46:56.745 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:05.340845+0000", 2024-08-19T14:46:56.746 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.2634999999999998e-05 2024-08-19T14:46:56.746 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.746 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.746 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:46:56.746 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:08.099925+0000", 2024-08-19T14:46:56.746 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.7590808249999998 2024-08-19T14:46:56.746 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.746 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.746 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:46:56.746 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:08.099938+0000", 2024-08-19T14:46:56.746 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2524000000000001e-05 2024-08-19T14:46:56.746 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:56.746 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:46:56.747 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:56.747 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.747 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.747 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:233358 7.3 7:ce05daa3:::benchmark_data_smithi045_115842_object115198:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e707)", 2024-08-19T14:46:56.747 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:39:31.017243+0000", 2024-08-19T14:46:56.747 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 445.49701889800002, 2024-08-19T14:46:56.747 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4164580099999999, 2024-08-19T14:46:56.747 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:46:56.747 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:46:56.747 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:46:56.747 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:46:56.747 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:46:56.747 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:46:56.748 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 233358 2024-08-19T14:46:56.748 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.748 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:46:56.748 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.748 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:46:56.748 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:31.017243+0000", 2024-08-19T14:46:56.748 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:56.748 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.748 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.748 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:46:56.749 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:31.017243+0000", 2024-08-19T14:46:56.749 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:56.749 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.749 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.749 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:46:56.749 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:31.017244+0000", 2024-08-19T14:46:56.749 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.2599999999999998e-07 2024-08-19T14:46:56.749 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.749 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.749 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:46:56.749 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:31.017245+0000", 2024-08-19T14:46:56.749 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.9400000000000001e-06 2024-08-19T14:46:56.749 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.749 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.750 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:46:56.750 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:31.017251+0000", 2024-08-19T14:46:56.750 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.3789999999999997e-06 2024-08-19T14:46:56.750 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.750 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.750 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:46:56.750 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:31.017255+0000", 2024-08-19T14:46:56.750 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4390000000000002e-06 2024-08-19T14:46:56.750 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.750 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.750 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:46:56.751 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:31.021095+0000", 2024-08-19T14:46:56.751 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0038393619999999998 2024-08-19T14:46:56.751 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.751 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.751 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for missing object", 2024-08-19T14:46:56.751 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:31.021319+0000", 2024-08-19T14:46:56.751 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000224026 2024-08-19T14:46:56.751 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.751 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.751 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:46:56.751 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:32.433689+0000", 2024-08-19T14:46:56.751 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.41236988 2024-08-19T14:46:56.751 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.752 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.752 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:46:56.752 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:32.433701+0000", 2024-08-19T14:46:56.752 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2658e-05 2024-08-19T14:46:56.752 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:56.752 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:46:56.752 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:56.752 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.752 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.752 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.16962.0:233367 7.3 7:c7fe5a37:::benchmark_data_smithi045_115842_object115207:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e707)", 2024-08-19T14:46:56.752 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:39:31.017282+0000", 2024-08-19T14:46:56.752 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 445.49697985, 2024-08-19T14:46:56.752 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4183822610000001, 2024-08-19T14:46:56.752 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:46:56.753 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:46:56.753 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:46:56.753 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:46:56.753 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.16962", 2024-08-19T14:46:56.753 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:46:56.753 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 233367 2024-08-19T14:46:56.753 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.753 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:46:56.753 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.753 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:46:56.753 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:31.017282+0000", 2024-08-19T14:46:56.753 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:56.753 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.753 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.754 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:46:56.754 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:31.017282+0000", 2024-08-19T14:46:56.754 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:56.754 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.754 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.754 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:46:56.754 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:31.017283+0000", 2024-08-19T14:46:56.754 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.3999999999999997e-07 2024-08-19T14:46:56.754 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.754 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.754 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:46:56.754 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:31.017284+0000", 2024-08-19T14:46:56.754 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.849e-06 2024-08-19T14:46:56.755 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.755 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.755 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:46:56.755 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:31.017290+0000", 2024-08-19T14:46:56.755 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.3630000000000002e-06 2024-08-19T14:46:56.755 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.755 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.755 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:46:56.755 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:31.017295+0000", 2024-08-19T14:46:56.755 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7829999999999998e-06 2024-08-19T14:46:56.755 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.755 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.755 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:46:56.755 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:31.021414+0000", 2024-08-19T14:46:56.756 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0041193289999999997 2024-08-19T14:46:56.756 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.756 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.756 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for missing object", 2024-08-19T14:46:56.756 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:31.021514+0000", 2024-08-19T14:46:56.756 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000100438 2024-08-19T14:46:56.756 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.756 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.757 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:46:56.757 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:32.435654+0000", 2024-08-19T14:46:56.757 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.414139619 2024-08-19T14:46:56.757 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.757 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.757 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:46:56.757 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:39:32.435665+0000", 2024-08-19T14:46:56.757 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0540000000000001e-05 2024-08-19T14:46:56.757 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:56.757 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:46:56.757 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:56.757 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.757 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.757 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26911.0:95689 8.10 8:0bf81366:::benchmark_data_smithi045_137103_object95688:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e751)", 2024-08-19T14:46:56.758 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:41:24.256797+0000", 2024-08-19T14:46:56.758 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 332.25746502099997, 2024-08-19T14:46:56.758 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.225889702, 2024-08-19T14:46:56.758 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:46:56.758 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:46:56.758 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:46:56.758 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:46:56.758 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26911", 2024-08-19T14:46:56.758 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3494993995", 2024-08-19T14:46:56.758 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 95689 2024-08-19T14:46:56.758 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.758 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:46:56.758 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.759 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:46:56.759 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:24.256797+0000", 2024-08-19T14:46:56.759 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:56.759 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.759 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.759 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:46:56.759 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:24.256797+0000", 2024-08-19T14:46:56.759 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:56.759 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.759 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.759 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:46:56.759 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:24.256799+0000", 2024-08-19T14:46:56.759 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.46e-06 2024-08-19T14:46:56.760 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.760 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.760 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:46:56.760 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:24.256847+0000", 2024-08-19T14:46:56.760 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.8560999999999998e-05 2024-08-19T14:46:56.760 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.760 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.760 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:46:56.760 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:24.260339+0000", 2024-08-19T14:46:56.760 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.003491629 2024-08-19T14:46:56.760 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.760 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.760 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:46:56.760 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:24.260356+0000", 2024-08-19T14:46:56.760 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.7178999999999999e-05 2024-08-19T14:46:56.761 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.761 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.761 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:46:56.761 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:24.261219+0000", 2024-08-19T14:46:56.761 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00086325900000000001 2024-08-19T14:46:56.761 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.761 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.761 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:46:56.761 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:24.261398+0000", 2024-08-19T14:46:56.761 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000179199 2024-08-19T14:46:56.761 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.761 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.761 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 4,7,9", 2024-08-19T14:46:56.762 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:24.261575+0000", 2024-08-19T14:46:56.762 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00017631799999999999 2024-08-19T14:46:56.762 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.762 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.762 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:46:56.762 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:24.271194+0000", 2024-08-19T14:46:56.762 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0096189139999999992 2024-08-19T14:46:56.762 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.762 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.762 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:46:56.762 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:24.271259+0000", 2024-08-19T14:46:56.762 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.5747999999999995e-05 2024-08-19T14:46:56.762 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.763 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.763 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:46:56.763 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:24.281912+0000", 2024-08-19T14:46:56.763 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.010652745999999999 2024-08-19T14:46:56.763 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.763 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.763 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:46:56.763 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:24.481525+0000", 2024-08-19T14:46:56.763 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.19961247900000001 2024-08-19T14:46:56.763 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.763 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.763 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:46:56.763 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:24.481600+0000", 2024-08-19T14:46:56.764 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.5551999999999998e-05 2024-08-19T14:46:56.764 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.764 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.764 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:46:56.764 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:24.482687+0000", 2024-08-19T14:46:56.764 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0010866579999999999 2024-08-19T14:46:56.764 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:56.764 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:46:56.764 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:56.764 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.764 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.764 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.26911.0:97186 8.18 8:1ef12c34:::benchmark_data_smithi045_137103_object97185:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e752)", 2024-08-19T14:46:56.764 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:41:25.875198+0000", 2024-08-19T14:46:56.764 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 330.63906453499999, 2024-08-19T14:46:56.765 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.219503484, 2024-08-19T14:46:56.765 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:46:56.765 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:46:56.765 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:46:56.765 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:46:56.765 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.26911", 2024-08-19T14:46:56.765 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/3494993995", 2024-08-19T14:46:56.765 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 97186 2024-08-19T14:46:56.765 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.765 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:46:56.765 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.765 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:46:56.765 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:25.875198+0000", 2024-08-19T14:46:56.765 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:56.765 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.766 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.766 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:46:56.766 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:25.875198+0000", 2024-08-19T14:46:56.766 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:46:56.766 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.766 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.766 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:46:56.766 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:25.875199+0000", 2024-08-19T14:46:56.766 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2860000000000001e-06 2024-08-19T14:46:56.766 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.766 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.766 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:46:56.767 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:25.875242+0000", 2024-08-19T14:46:56.767 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2901000000000001e-05 2024-08-19T14:46:56.767 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.767 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.767 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:46:56.767 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:25.875260+0000", 2024-08-19T14:46:56.767 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.8179e-05 2024-08-19T14:46:56.767 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.767 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.767 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:46:56.767 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:25.875269+0000", 2024-08-19T14:46:56.767 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.2830000000000007e-06 2024-08-19T14:46:56.767 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.767 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.768 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:46:56.768 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:25.880274+0000", 2024-08-19T14:46:56.768 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.005004975 2024-08-19T14:46:56.768 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.768 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.768 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:46:56.768 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:25.881459+0000", 2024-08-19T14:46:56.768 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00118443 2024-08-19T14:46:56.768 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.768 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.768 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 4,5,8", 2024-08-19T14:46:56.768 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:25.881814+0000", 2024-08-19T14:46:56.768 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00035534599999999998 2024-08-19T14:46:56.768 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.769 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.769 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:46:56.769 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:25.888940+0000", 2024-08-19T14:46:56.769 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0071263439999999997 2024-08-19T14:46:56.769 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.769 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.769 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:46:56.769 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:25.889338+0000", 2024-08-19T14:46:56.769 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000397919 2024-08-19T14:46:56.769 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.769 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.769 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:46:56.769 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:25.892928+0000", 2024-08-19T14:46:56.769 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0035897580000000002 2024-08-19T14:46:56.770 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.770 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.770 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:46:56.770 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:26.093019+0000", 2024-08-19T14:46:56.770 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.20009076200000001 2024-08-19T14:46:56.770 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.770 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.770 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:46:56.770 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:26.093098+0000", 2024-08-19T14:46:56.770 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.8808999999999997e-05 2024-08-19T14:46:56.770 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:46:56.770 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:46:56.770 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:46:56.771 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:26.094701+0000", 2024-08-19T14:46:56.771 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.001603492 2024-08-19T14:46:56.771 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:56.771 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:46:56.771 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:56.771 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:46:56.771 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:46:56.771 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:46:56.771 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:46:56.772 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:56 smithi032 ceph-mon[20004]: 1.16 scrub starts 2024-08-19T14:46:56.772 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:56 smithi032 ceph-mon[20004]: 1.16 scrub ok 2024-08-19T14:46:56.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:56 smithi045 ceph-mon[20143]: 1.16 scrub starts 2024-08-19T14:46:56.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:56 smithi045 ceph-mon[20143]: 1.16 scrub ok 2024-08-19T14:46:56.811 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:56.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:56.813 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:46:56.913 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:46:56.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:56.952 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:56.952 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:46:57.053 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:46:57.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:57.084 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:57.084 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:46:57.186 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:46:57.207 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:57.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:57.208 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:46:57.272 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:46:57.272 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:46:57.308 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:46:57.308 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:46:57.308 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:46:57.333 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:46:57.333 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:46:57.335 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:57.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:57.336 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:46:57.354 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:46:57.355 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:46:57.373 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:46:57.373 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:46:57.395 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:46:57.395 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:46:57.416 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:46:57.416 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:46:57.435 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:46:57.436 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:46:57.437 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:46:57.460 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:46:57.460 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:46:57.463 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:57.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:57.464 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:46:57.493 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:46:57.493 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:46:57.525 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:46:57.525 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:46:57.558 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:46:57.558 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:46:57.565 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:46:57.583 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:46:57.584 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:46:57.589 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:57.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:57.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:46:57.691 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:46:57.714 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:57.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:57.715 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:46:57.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:57 smithi032 ceph-mon[20004]: 1.17 scrub starts 2024-08-19T14:46:57.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:57 smithi032 ceph-mon[20004]: 1.17 scrub ok 2024-08-19T14:46:57.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:57 smithi032 ceph-mon[20004]: pgmap v2104: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 2.7 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 777 op/s; 409/133527 objects misplaced (0.306%); 11 MiB/s, 174 objects/s recovering 2024-08-19T14:46:57.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:57 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-08-19T14:46:57.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:57 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3293109407' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:46:57.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:57 smithi045 ceph-mon[20143]: 1.17 scrub starts 2024-08-19T14:46:57.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:57 smithi045 ceph-mon[20143]: 1.17 scrub ok 2024-08-19T14:46:57.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:57 smithi045 ceph-mon[20143]: pgmap v2104: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 2.7 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 777 op/s; 409/133527 objects misplaced (0.306%); 11 MiB/s, 174 objects/s recovering 2024-08-19T14:46:57.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:57 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-08-19T14:46:57.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:57 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3293109407' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:46:57.816 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:46:57.850 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:57.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:57.851 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:46:57.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:57 smithi031 ceph-mon[16336]: 1.17 scrub starts 2024-08-19T14:46:57.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:57 smithi031 ceph-mon[16336]: 1.17 scrub ok 2024-08-19T14:46:57.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:57 smithi031 ceph-mon[16336]: pgmap v2104: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 2.7 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 777 op/s; 409/133527 objects misplaced (0.306%); 11 MiB/s, 174 objects/s recovering 2024-08-19T14:46:57.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:57 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-08-19T14:46:57.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:57 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3293109407' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:46:57.942 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-08-19T14:46:57.951 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:46:57.974 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:57.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:57.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:46:58.076 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:46:58.099 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:58.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:58.100 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:46:58.201 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:46:58.225 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:58.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:58.226 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:46:58.245 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:46:58.328 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:46:58.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:58.366 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:58.366 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:46:58.467 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:46:58.491 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:58.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:58.492 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:46:58.593 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:46:58.619 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:58.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:58.620 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:46:58.721 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:46:58.723 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:58 smithi032 ceph-mon[20004]: 9.a scrub starts 2024-08-19T14:46:58.744 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:58 smithi031 ceph-mon[16336]: 9.a scrub starts 2024-08-19T14:46:58.746 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:58.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:58.747 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:46:58.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:58 smithi045 ceph-mon[20143]: 9.a scrub starts 2024-08-19T14:46:58.848 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:46:58.891 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:58.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:58.892 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:46:58.992 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:46:59.003 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:46:59.003 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:46:59.020 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:59.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:59.021 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:46:59.121 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:46:59.146 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:59.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:59.147 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:46:59.248 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:46:59.273 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:46:59.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:59.275 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:46:59.376 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:46:59.400 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:59.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:59.401 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:46:59.502 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:46:59.505 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is unset 2024-08-19T14:46:59.546 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:59.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:59.547 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:46:59.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:59 smithi031 ceph-mon[16336]: 9.a scrub ok 2024-08-19T14:46:59.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:59 smithi031 ceph-mon[16336]: 1.1b scrub starts 2024-08-19T14:46:59.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:59 smithi031 ceph-mon[16336]: 1.1b scrub ok 2024-08-19T14:46:59.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:59 smithi031 ceph-mon[16336]: pgmap v2105: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 2.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 665 op/s; 409/132315 objects misplaced (0.309%); 8.0 MiB/s, 127 objects/s recovering 2024-08-19T14:46:59.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:59 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3729452903' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:46:59.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:46:59 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:46:59.647 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:46:59.675 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:59.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:59.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:46:59.693 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:59 smithi032 ceph-mon[20004]: 9.a scrub ok 2024-08-19T14:46:59.693 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:59 smithi032 ceph-mon[20004]: 1.1b scrub starts 2024-08-19T14:46:59.693 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:59 smithi032 ceph-mon[20004]: 1.1b scrub ok 2024-08-19T14:46:59.693 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:59 smithi032 ceph-mon[20004]: pgmap v2105: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 2.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 665 op/s; 409/132315 objects misplaced (0.309%); 8.0 MiB/s, 127 objects/s recovering 2024-08-19T14:46:59.693 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:59 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3729452903' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:46:59.693 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:46:59 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:46:59.776 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:46:59.779 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:46:59.781 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-08-19T14:46:59.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:59 smithi045 ceph-mon[20143]: 9.a scrub ok 2024-08-19T14:46:59.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:59 smithi045 ceph-mon[20143]: 1.1b scrub starts 2024-08-19T14:46:59.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:59 smithi045 ceph-mon[20143]: 1.1b scrub ok 2024-08-19T14:46:59.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:59 smithi045 ceph-mon[20143]: pgmap v2105: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 2.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 665 op/s; 409/132315 objects misplaced (0.309%); 8.0 MiB/s, 127 objects/s recovering 2024-08-19T14:46:59.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:59 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3729452903' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:46:59.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:46:59 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:46:59.803 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:46:59.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:59.804 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:46:59.905 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:46:59.950 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:46:59.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:46:59.951 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:47:00.052 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:47:00.076 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:00.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:00.078 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:47:00.156 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-08-19T14:47:00.179 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:47:00.215 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:00.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:00.216 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:47:00.317 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:47:00.364 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:00.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:00.366 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:47:00.467 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:47:00.491 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:00.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:00.492 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:47:00.510 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:47:00.593 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:47:00.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:00 smithi031 ceph-mon[16336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:47:00.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:00 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:47:00.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:00 smithi031 ceph-mon[16336]: osdmap e894: 12 total, 12 up, 8 in 2024-08-19T14:47:00.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:00 smithi031 ceph-mon[16336]: from='client.27656 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:47:00.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:00 smithi031 ceph-mon[16336]: 9.c scrub starts 2024-08-19T14:47:00.620 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:00.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:00.621 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:47:00.722 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:47:00.745 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:00.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:00.746 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:47:00.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:00 smithi032 ceph-mon[20004]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:47:00.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:00 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:47:00.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:00 smithi032 ceph-mon[20004]: osdmap e894: 12 total, 12 up, 8 in 2024-08-19T14:47:00.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:00 smithi032 ceph-mon[20004]: from='client.27656 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:47:00.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:00 smithi032 ceph-mon[20004]: 9.c scrub starts 2024-08-19T14:47:00.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:00 smithi045 ceph-mon[20143]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:47:00.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:00 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:47:00.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:00 smithi045 ceph-mon[20143]: osdmap e894: 12 total, 12 up, 8 in 2024-08-19T14:47:00.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:00 smithi045 ceph-mon[20143]: from='client.27656 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:47:00.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:00 smithi045 ceph-mon[20143]: 9.c scrub starts 2024-08-19T14:47:00.847 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:47:00.870 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:00.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:00.871 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:47:00.972 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:47:00.993 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:00.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:00.994 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:47:01.025 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:47:01.025 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:47:01.067 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:47:01.095 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:47:01.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:01.118 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:01.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:47:01.181 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":2106,"stamp":"2024-08-19T14:46:59.503756+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2801009208,"num_objects":42734,"num_object_clones":0,"num_object_copies":128202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":409,"num_objects_unfound":0,"num_objects_dirty":42734,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":146274,"num_write_kb":6048129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39287,"num_bytes_recovered":2575770688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150583,"ondisk_log_size":150583,"up":141,"acting":140,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":142,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":15714332,"kb_used_data":8575756,"kb_used_omap":17,"kb_used_meta":7138350,"kb_avail":734246884,"statfs":{"total":767960285184,"available":751868809216,"internally_reserved":0,"allocated":8781574144,"data_stored":8760246740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18388,"internal_metadata":7309670444},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,1,2,7,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-351404032,"num_objects":-5362,"num_object_clones":0,"num_object_copies":-16086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-767,"num_objects_unfound":0,"num_objects_dirty":-5362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5362,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":767,"num_bytes_recovered":50266112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.025743"},"pg_stats":[{"pgid":"9.8","version":"893'9212","reported_seq":13434,"reported_epoch":893,"state":"active+clean","last_fresh":"2024-08-19T14:46:57.737622+0000","last_change":"2024-08-19T14:45:07.176081+0000","last_active":"2024-08-19T14:46:57.737622+0000","last_peered":"2024-08-19T14:46:57.737622+0000","last_clean":"2024-08-19T14:46:57.737622+0000","last_became_active":"2024-08-19T14:45:07.171026+0000","last_became_peered":"2024-08-19T14:45:07.171026+0000","last_unstale":"2024-08-19T14:46:57.737622+0000","last_undegraded":"2024-08-19T14:46:57.737622+0000","last_fullsized":"2024-08-19T14:46:57.737622+0000","mapping_epoch":852,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":853,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":9212,"log_dups_size":0,"ondisk_log_size":9212,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":174063616,"num_objects":2656,"num_object_clones":0,"num_object_copies":7968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9212,"num_write_kb":379776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2033,"num_bytes_recovered":133234688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,1],"acting":[11,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"70'139","reported_seq":2132,"reported_epoch":893,"state":"active+clean","last_fresh":"2024-08-19T14:46:54.142921+0000","last_change":"2024-08-19T14:45:26.108478+0000","last_active":"2024-08-19T14:46:54.142921+0000","last_peered":"2024-08-19T14:46:54.142921+0000","last_clean":"2024-08-19T14:46:54.142921+0000","last_became_active":"2024-08-19T14:45:26.107697+0000","last_became_peered":"2024-08-19T14:45:26.107697+0000","last_unstale":"2024-08-19T14:46:54.142921+0000","last_undegraded":"2024-08-19T14:46:54.142921+0000","last_fullsized":"2024-08-19T14:46:54.142921+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:26.109206+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:41:45.177828+0000","last_clean_scrub_stamp":"2024-08-19T14:44:26.109206+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6,9],"acting":[11,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.b","version":"893'9023","reported_seq":13141,"reported_epoch":893,"state":"active+clean","last_fresh":"2024-08-19T14:46:55.223774+0000","last_change":"2024-08-19T14:45:09.096009+0000","last_active":"2024-08-19T14:46:55.223774+0000","last_peered":"2024-08-19T14:46:55.223774+0000","last_clean":"2024-08-19T14:46:55.223774+0000","last_became_active":"2024-08-19T14:45:09.091550+0000","last_became_peered":"2024-08-19T14:45:09.091550+0000","last_unstale":"2024-08-19T14:46:55.223774+0000","last_undegraded":"2024-08-19T14:46:55.223774+0000","last_fullsized":"2024-08-19T14:46:55.223774+0000","mapping_epoch":854,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":9023,"log_dups_size":0,"ondisk_log_size":9023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176619520,"num_objects":2695,"num_object_clones":0,"num_object_copies":8085,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2695,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9023,"num_write_kb":374976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1981,"num_bytes_recovered":129826816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,1],"acting":[8,10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"70'139","reported_seq":1986,"reported_epoch":893,"state":"active+clean","last_fresh":"2024-08-19T14:46:54.112565+0000","last_change":"2024-08-19T14:45:26.101740+0000","last_active":"2024-08-19T14:46:54.112565+0000","last_peered":"2024-08-19T14:46:54.112565+0000","last_clean":"2024-08-19T14:46:54.112565+0000","last_became_active":"2024-08-19T14:45:26.101280+0000","last_became_peered":"2024-08-19T14:45:26.101280+0000","last_unstale":"2024-08-19T14:46:54.112565+0000","last_undegraded":"2024-08-19T14:46:54.112565+0000","last_fullsized":"2024-08-19T14:46:54.112565+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:45:20.599964+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:45:20.599964+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.a","version":"893'9329","reported_seq":22345,"reported_epoch":893,"state":"active+clean","last_fresh":"2024-08-19T14:46:59.038245+0000","last_change":"2024-08-19T14:46:58.252113+0000","last_active":"2024-08-19T14:46:59.038245+0000","last_peered":"2024-08-19T14:46:59.038245+0000","last_clean":"2024-08-19T14:46:59.038245+0000","last_became_active":"2024-08-19T14:46:35.321997+0000","last_became_peered":"2024-08-19T14:46:35.321997+0000","last_unstale":"2024-08-19T14:46:59.038245+0000","last_undegraded":"2024-08-19T14:46:59.038245+0000","last_fullsized":"2024-08-19T14:46:59.038245+0000","mapping_epoch":885,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":886,"parent":"0.0","parent_split_bits":0,"last_scrub":"893'9283","last_scrub_stamp":"2024-08-19T14:46:58.251971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:46:58.251971+0000","objects_scrubbed":2574,"log_size":9329,"log_dups_size":0,"ondisk_log_size":9329,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:48:10.683585+0000","scrub_duration":524,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":165216256,"num_objects":2521,"num_object_clones":0,"num_object_copies":7563,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2521,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9329,"num_write_kb":379200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6408,"num_bytes_recovered":419954688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"70'139","reported_seq":624,"reported_epoch":893,"state":"active+clean","last_fresh":"2024-08-19T14:46:54.112205+0000","last_change":"2024-08-19T14:46:07.848793+0000","last_active":"2024-08-19T14:46:54.112205+0000","last_peered":"2024-08-19T14:46:54.112205+0000","last_clean":"2024-08-19T14:46:54.112205+0000","last_became_active":"2024-08-19T14:43:11.470639+0000","last_became_peered":"2024-08-19T14:43:11.470639+0000","last_unstale":"2024-08-19T14:46:54.112205+0000","last_undegraded":"2024-08-19T14:46:54.112205+0000","last_fullsized":"2024-08-19T14:46:54.112205+0000","mapping_epoch":795,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":796,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:46:07.848681+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:19:59.302931+0000","last_clean_scrub_stamp":"2024-08-19T14:46:07.848681+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:47:19.399442+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.9","version":"893'9144","reported_seq":21388,"reported_epoch":893,"state":"active+recovering+undersized+remapped","last_fresh":"2024-08-19T14:46:55.799731+0000","last_change":"2024-08-19T14:45:24.584370+0000","last_active":"2024-08-19T14:46:55.799731+0000","last_peered":"2024-08-19T14:46:55.799731+0000","last_clean":"2024-08-19T14:45:22.229561+0000","last_became_active":"2024-08-19T14:45:24.550442+0000","last_became_peered":"2024-08-19T14:45:24.550442+0000","last_unstale":"2024-08-19T14:46:55.799731+0000","last_undegraded":"2024-08-19T14:46:55.799731+0000","last_fullsized":"2024-08-19T14:45:24.094569+0000","mapping_epoch":863,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":0,"last_scrub":"860'4221","last_scrub_stamp":"2024-08-19T14:45:19.039376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:45:19.039376+0000","objects_scrubbed":4147,"log_size":9144,"log_dups_size":0,"ondisk_log_size":9144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1783,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":177733632,"num_objects":2712,"num_object_clones":0,"num_object_copies":8136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":409,"num_objects_unfound":0,"num_objects_dirty":2712,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9144,"num_write_kb":379392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6018,"num_bytes_recovered":394395648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,6],"acting":[10,11],"avail_no_missing":["10","5","8","11"],"object_location_counts":[{"shards":"8,10,11","objects":409},{"shards":"10,11","objects":2303}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1","version":"70'139","reported_seq":617,"reported_epoch":893,"state":"active+clean","last_fresh":"2024-08-19T14:46:54.115202+0000","last_change":"2024-08-19T14:44:32.095248+0000","last_active":"2024-08-19T14:46:54.115202+0000","last_peered":"2024-08-19T14:46:54.115202+0000","last_clean":"2024-08-19T14:46:54.115202+0000","last_became_active":"2024-08-19T14:43:16.098337+0000","last_became_peered":"2024-08-19T14:43:16.098337+0000","last_unstale":"2024-08-19T14:46:54.115202+0000","last_undegraded":"2024-08-19T14:46:54.115202+0000","last_fullsized":"2024-08-19T14:46:54.115202+0000","mapping_epoch":800,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":801,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:32.094375+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:44:32.094375+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,2],"acting":[10,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.f","version":"893'9335","reported_seq":13369,"reported_epoch":893,"state":"active+clean","last_fresh":"2024-08-19T14:46:58.999972+0000","last_change":"2024-08-19T14:45:01.147769+0000","last_active":"2024-08-19T14:46:58.999972+0000","last_peered":"2024-08-19T14:46:58.999972+0000","last_clean":"2024-08-19T14:46:58.999972+0000","last_became_active":"2024-08-19T14:45:01.143857+0000","last_became_peered":"2024-08-19T14:45:01.143857+0000","last_unstale":"2024-08-19T14:46:58.999972+0000","last_undegraded":"2024-08-19T14:46:58.999972+0000","last_fullsized":"2024-08-19T14:46:58.999972+0000","mapping_epoch":847,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":848,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":9335,"log_dups_size":0,"ondisk_log_size":9335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168624128,"num_objects":2573,"num_object_clones":0,"num_object_copies":7719,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2573,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9335,"num_write_kb":381056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1939,"num_bytes_recovered":127074304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,11],"acting":[1,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"70'139","reported_seq":1142,"reported_epoch":893,"state":"active+clean","last_fresh":"2024-08-19T14:46:54.115000+0000","last_change":"2024-08-19T14:45:26.107836+0000","last_active":"2024-08-19T14:46:54.115000+0000","last_peered":"2024-08-19T14:46:54.115000+0000","last_clean":"2024-08-19T14:46:54.115000+0000","last_became_active":"2024-08-19T14:45:26.107494+0000","last_became_peered":"2024-08-19T14:45:26.107494+0000","last_unstale":"2024-08-19T14:46:54.115000+0000","last_undegraded":"2024-08-19T14:46:54.115000+0000","last_fullsized":"2024-08-19T14:46:54.115000+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:26.113799+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:21:46.057624+0000","last_clean_scrub_stamp":"2024-08-19T14:44:26.113799+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,2],"acting":[6,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"893'9049","reported_seq":9192,"reported_epoch":893,"state":"active+clean","last_fresh":"2024-08-19T14:46:55.752997+0000","last_change":"2024-08-19T14:44:15.415042+0000","last_active":"2024-08-19T14:46:55.752997+0000","last_peered":"2024-08-19T14:46:55.752997+0000","last_clean":"2024-08-19T14:46:55.752997+0000","last_became_active":"2024-08-19T14:44:15.414595+0000","last_became_peered":"2024-08-19T14:44:15.414595+0000","last_unstale":"2024-08-19T14:46:55.752997+0000","last_undegraded":"2024-08-19T14:46:55.752997+0000","last_fullsized":"2024-08-19T14:46:55.752997+0000","mapping_epoch":826,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":827,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":9049,"log_dups_size":0,"ondisk_log_size":9049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176095232,"num_objects":2687,"num_object_clones":0,"num_object_copies":8061,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9049,"num_write_kb":375552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,9],"acting":[10,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.6","version":"70'139","reported_seq":483,"reported_epoch":893,"state":"active+clean","last_fresh":"2024-08-19T14:46:54.113440+0000","last_change":"2024-08-19T14:44:22.089843+0000","last_active":"2024-08-19T14:46:54.113440+0000","last_peered":"2024-08-19T14:46:54.113440+0000","last_clean":"2024-08-19T14:46:54.113440+0000","last_became_active":"2024-08-19T14:43:16.091256+0000","last_became_peered":"2024-08-19T14:43:16.091256+0000","last_unstale":"2024-08-19T14:46:54.113440+0000","last_undegraded":"2024-08-19T14:46:54.113440+0000","last_fullsized":"2024-08-19T14:46:54.113440+0000","mapping_epoch":800,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":801,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:22.089766+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:27.848870+0000","last_clean_scrub_stamp":"2024-08-19T14:44:22.089766+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.d","version":"893'9103","reported_seq":12997,"reported_epoch":893,"state":"active+clean","last_fresh":"2024-08-19T14:46:58.937078+0000","last_change":"2024-08-19T14:45:00.248322+0000","last_active":"2024-08-19T14:46:58.937078+0000","last_peered":"2024-08-19T14:46:58.937078+0000","last_clean":"2024-08-19T14:46:58.937078+0000","last_became_active":"2024-08-19T14:45:00.244759+0000","last_became_peered":"2024-08-19T14:45:00.244759+0000","last_unstale":"2024-08-19T14:46:58.937078+0000","last_undegraded":"2024-08-19T14:46:58.937078+0000","last_fullsized":"2024-08-19T14:46:58.937078+0000","mapping_epoch":846,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":847,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":9103,"log_dups_size":0,"ondisk_log_size":9103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171704344,"num_objects":2621,"num_object_clones":0,"num_object_copies":7863,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2621,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9103,"num_write_kb":375105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1868,"num_bytes_recovered":122421248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"70'139","reported_seq":582,"reported_epoch":893,"state":"active+clean","last_fresh":"2024-08-19T14:46:54.113015+0000","last_change":"2024-08-19T14:45:26.106248+0000","last_active":"2024-08-19T14:46:54.113015+0000","last_peered":"2024-08-19T14:46:54.113015+0000","last_clean":"2024-08-19T14:46:54.113015+0000","last_became_active":"2024-08-19T14:45:26.105833+0000","last_became_peered":"2024-08-19T14:45:26.105833+0000","last_unstale":"2024-08-19T14:46:54.113015+0000","last_undegraded":"2024-08-19T14:46:54.113015+0000","last_fullsized":"2024-08-19T14:46:54.113015+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:29.075645+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:41:48.878043+0000","last_clean_scrub_stamp":"2024-08-19T14:44:29.075645+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.c","version":"893'9051","reported_seq":9247,"reported_epoch":893,"state":"active+clean","last_fresh":"2024-08-19T14:46:54.875356+0000","last_change":"2024-08-19T14:45:22.398825+0000","last_active":"2024-08-19T14:46:54.875356+0000","last_peered":"2024-08-19T14:46:54.875356+0000","last_clean":"2024-08-19T14:46:54.875356+0000","last_became_active":"2024-08-19T14:44:15.416278+0000","last_became_peered":"2024-08-19T14:44:15.416278+0000","last_unstale":"2024-08-19T14:46:54.875356+0000","last_undegraded":"2024-08-19T14:46:54.875356+0000","last_fullsized":"2024-08-19T14:46:54.875356+0000","mapping_epoch":826,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":827,"parent":"0.0","parent_split_bits":0,"last_scrub":"862'4406","last_scrub_stamp":"2024-08-19T14:45:22.398707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:45:22.398707+0000","objects_scrubbed":4394,"log_size":9051,"log_dups_size":0,"ondisk_log_size":9051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":618,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":181600256,"num_objects":2771,"num_object_clones":0,"num_object_copies":8313,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2771,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9051,"num_write_kb":378304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,10],"acting":[4,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"70'139","reported_seq":1052,"reported_epoch":893,"state":"active+clean","last_fresh":"2024-08-19T14:46:54.142937+0000","last_change":"2024-08-19T14:45:26.105952+0000","last_active":"2024-08-19T14:46:54.142937+0000","last_peered":"2024-08-19T14:46:54.142937+0000","last_clean":"2024-08-19T14:46:54.142937+0000","last_became_active":"2024-08-19T14:45:26.105653+0000","last_became_peered":"2024-08-19T14:45:26.105653+0000","last_unstale":"2024-08-19T14:46:54.142937+0000","last_undegraded":"2024-08-19T14:46:54.142937+0000","last_fullsized":"2024-08-19T14:46:54.142937+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:33.078526+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:41:55.868614+0000","last_clean_scrub_stamp":"2024-08-19T14:44:33.078526+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,6],"acting":[11,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.2","version":"893'9319","reported_seq":9464,"reported_epoch":893,"state":"active+clean","last_fresh":"2024-08-19T14:46:59.051036+0000","last_change":"2024-08-19T14:44:15.415441+0000","last_active":"2024-08-19T14:46:59.051036+0000","last_peered":"2024-08-19T14:46:59.051036+0000","last_clean":"2024-08-19T14:46:59.051036+0000","last_became_active":"2024-08-19T14:44:15.415137+0000","last_became_peered":"2024-08-19T14:44:15.415137+0000","last_unstale":"2024-08-19T14:46:59.051036+0000","last_undegraded":"2024-08-19T14:46:59.051036+0000","last_fullsized":"2024-08-19T14:46:59.051036+0000","mapping_epoch":826,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":827,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":9319,"log_dups_size":0,"ondisk_log_size":9319,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":172818432,"num_objects":2637,"num_object_clones":0,"num_object_copies":7911,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2637,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9319,"num_write_kb":382592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,11],"acting":[1,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"70'139","reported_seq":1023,"reported_epoch":893,"state":"active+clean","last_fresh":"2024-08-19T14:46:54.114809+0000","last_change":"2024-08-19T14:45:26.103609+0000","last_active":"2024-08-19T14:46:54.114809+0000","last_peered":"2024-08-19T14:46:54.114809+0000","last_clean":"2024-08-19T14:46:54.114809+0000","last_became_active":"2024-08-19T14:45:26.103142+0000","last_became_peered":"2024-08-19T14:45:26.103142+0000","last_unstale":"2024-08-19T14:46:54.114809+0000","last_undegraded":"2024-08-19T14:46:54.114809+0000","last_fullsized":"2024-08-19T14:46:54.114809+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:28.862033+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:44:28.862033+0000","last_clean_scrub_stamp":"2024-08-19T14:44:28.862033+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,4],"acting":[6,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1","version":"893'9109","reported_seq":18129,"reported_epoch":893,"state":"active+clean","last_fresh":"2024-08-19T14:46:54.611890+0000","last_change":"2024-08-19T14:46:18.212911+0000","last_active":"2024-08-19T14:46:54.611890+0000","last_peered":"2024-08-19T14:46:54.611890+0000","last_clean":"2024-08-19T14:46:54.611890+0000","last_became_active":"2024-08-19T14:46:18.202172+0000","last_became_peered":"2024-08-19T14:46:18.202172+0000","last_unstale":"2024-08-19T14:46:54.611890+0000","last_undegraded":"2024-08-19T14:46:54.611890+0000","last_fullsized":"2024-08-19T14:46:54.611890+0000","mapping_epoch":875,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":876,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":9109,"log_dups_size":0,"ondisk_log_size":9109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184614912,"num_objects":2817,"num_object_clones":0,"num_object_copies":8451,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9109,"num_write_kb":381632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4433,"num_bytes_recovered":290521088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,10],"acting":[6,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"70'139","reported_seq":1409,"reported_epoch":893,"state":"active+clean","last_fresh":"2024-08-19T14:46:54.117033+0000","last_change":"2024-08-19T14:45:26.126939+0000","last_active":"2024-08-19T14:46:54.117033+0000","last_peered":"2024-08-19T14:46:54.117033+0000","last_clean":"2024-08-19T14:46:54.117033+0000","last_became_active":"2024-08-19T14:45:26.126712+0000","last_became_peered":"2024-08-19T14:45:26.126712+0000","last_unstale":"2024-08-19T14:46:54.117033+0000","last_undegraded":"2024-08-19T14:46:54.117033+0000","last_fullsized":"2024-08-19T14:46:54.117033+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:24.958576+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:44:24.958576+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,2],"acting":[6,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.0","version":"893'9131","reported_seq":17953,"reported_epoch":893,"state":"active+clean","last_fresh":"2024-08-19T14:46:59.021163+0000","last_change":"2024-08-19T14:46:29.123185+0000","last_active":"2024-08-19T14:46:59.021163+0000","last_peered":"2024-08-19T14:46:59.021163+0000","last_clean":"2024-08-19T14:46:59.021163+0000","last_became_active":"2024-08-19T14:46:29.110655+0000","last_became_peered":"2024-08-19T14:46:29.110655+0000","last_unstale":"2024-08-19T14:46:59.021163+0000","last_undegraded":"2024-08-19T14:46:59.021163+0000","last_fullsized":"2024-08-19T14:46:59.021163+0000","mapping_epoch":881,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":882,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":9131,"log_dups_size":0,"ondisk_log_size":9131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":164954112,"num_objects":2517,"num_object_clones":0,"num_object_copies":7551,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9131,"num_write_kb":372736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4334,"num_bytes_recovered":284033024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,11],"acting":[1,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"70'139","reported_seq":726,"reported_epoch":893,"state":"active+clean","last_fresh":"2024-08-19T14:46:54.113092+0000","last_change":"2024-08-19T14:44:38.183741+0000","last_active":"2024-08-19T14:46:54.113092+0000","last_peered":"2024-08-19T14:46:54.113092+0000","last_clean":"2024-08-19T14:46:54.113092+0000","last_became_active":"2024-08-19T14:44:38.183219+0000","last_became_peered":"2024-08-19T14:44:38.183219+0000","last_unstale":"2024-08-19T14:46:54.113092+0000","last_undegraded":"2024-08-19T14:46:54.113092+0000","last_fullsized":"2024-08-19T14:46:54.113092+0000","mapping_epoch":837,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:27.791867+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:44:27.791867+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,4],"acting":[8,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.3","version":"893'9173","reported_seq":22171,"reported_epoch":893,"state":"active+clean","last_fresh":"2024-08-19T14:46:55.199518+0000","last_change":"2024-08-19T14:46:54.137652+0000","last_active":"2024-08-19T14:46:55.199518+0000","last_peered":"2024-08-19T14:46:55.199518+0000","last_clean":"2024-08-19T14:46:55.199518+0000","last_became_active":"2024-08-19T14:46:54.116048+0000","last_became_peered":"2024-08-19T14:46:54.116048+0000","last_unstale":"2024-08-19T14:46:55.199518+0000","last_undegraded":"2024-08-19T14:46:55.199518+0000","last_fullsized":"2024-08-19T14:46:55.199518+0000","mapping_epoch":892,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":893,"parent":"0.0","parent_split_bits":0,"last_scrub":"860'4176","last_scrub_stamp":"2024-08-19T14:45:18.741455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:45:18.741455+0000","objects_scrubbed":4146,"log_size":9173,"log_dups_size":0,"ondisk_log_size":9173,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1454,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":183828480,"num_objects":2805,"num_object_clones":0,"num_object_copies":8415,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2805,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9173,"num_write_kb":383296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6391,"num_bytes_recovered":418840576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,6],"acting":[8,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.b","version":"70'139","reported_seq":1504,"reported_epoch":893,"state":"active+clean","last_fresh":"2024-08-19T14:46:54.143333+0000","last_change":"2024-08-19T14:44:38.183925+0000","last_active":"2024-08-19T14:46:54.143333+0000","last_peered":"2024-08-19T14:46:54.143333+0000","last_clean":"2024-08-19T14:46:54.143333+0000","last_became_active":"2024-08-19T14:44:38.183552+0000","last_became_peered":"2024-08-19T14:44:38.183552+0000","last_unstale":"2024-08-19T14:46:54.143333+0000","last_undegraded":"2024-08-19T14:46:54.143333+0000","last_fullsized":"2024-08-19T14:46:54.143333+0000","mapping_epoch":837,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:29.142655+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:44:29.142655+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,1],"acting":[11,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.6","version":"893'9109","reported_seq":13187,"reported_epoch":893,"state":"active+clean","last_fresh":"2024-08-19T14:46:55.770345+0000","last_change":"2024-08-19T14:45:08.093571+0000","last_active":"2024-08-19T14:46:55.770345+0000","last_peered":"2024-08-19T14:46:55.770345+0000","last_clean":"2024-08-19T14:46:55.770345+0000","last_became_active":"2024-08-19T14:45:08.088967+0000","last_became_peered":"2024-08-19T14:45:08.088967+0000","last_unstale":"2024-08-19T14:46:55.770345+0000","last_undegraded":"2024-08-19T14:46:55.770345+0000","last_fullsized":"2024-08-19T14:46:55.770345+0000","mapping_epoch":853,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":854,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":9109,"log_dups_size":0,"ondisk_log_size":9109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":178716672,"num_objects":2727,"num_object_clones":0,"num_object_copies":8181,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2727,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9109,"num_write_kb":378752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1962,"num_bytes_recovered":128581632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,9],"acting":[10,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.e","version":"70'139","reported_seq":1702,"reported_epoch":893,"state":"active+clean","last_fresh":"2024-08-19T14:46:54.112304+0000","last_change":"2024-08-19T14:46:17.013622+0000","last_active":"2024-08-19T14:46:54.112304+0000","last_peered":"2024-08-19T14:46:54.112304+0000","last_clean":"2024-08-19T14:46:54.112304+0000","last_became_active":"2024-08-19T14:43:08.463102+0000","last_became_peered":"2024-08-19T14:43:08.463102+0000","last_unstale":"2024-08-19T14:46:54.112304+0000","last_undegraded":"2024-08-19T14:46:54.112304+0000","last_fullsized":"2024-08-19T14:46:54.112304+0000","mapping_epoch":793,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":794,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:46:17.013541+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:40:03.764672+0000","last_clean_scrub_stamp":"2024-08-19T14:46:17.013541+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:47:21.570706+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,1],"acting":[4,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.7","version":"893'8924","reported_seq":9069,"reported_epoch":893,"state":"active+clean","last_fresh":"2024-08-19T14:46:55.789398+0000","last_change":"2024-08-19T14:44:15.068003+0000","last_active":"2024-08-19T14:46:55.789398+0000","last_peered":"2024-08-19T14:46:55.789398+0000","last_clean":"2024-08-19T14:46:55.789398+0000","last_became_active":"2024-08-19T14:44:15.067635+0000","last_became_peered":"2024-08-19T14:44:15.067635+0000","last_unstale":"2024-08-19T14:46:55.789398+0000","last_undegraded":"2024-08-19T14:46:55.789398+0000","last_fullsized":"2024-08-19T14:46:55.789398+0000","mapping_epoch":826,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":827,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":8924,"log_dups_size":0,"ondisk_log_size":8924,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180224000,"num_objects":2750,"num_object_clones":0,"num_object_copies":8250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8924,"num_write_kb":373568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,1],"acting":[10,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.f","version":"70'139","reported_seq":1388,"reported_epoch":893,"state":"active+clean","last_fresh":"2024-08-19T14:46:54.114946+0000","last_change":"2024-08-19T14:45:26.102407+0000","last_active":"2024-08-19T14:46:54.114946+0000","last_peered":"2024-08-19T14:46:54.114946+0000","last_clean":"2024-08-19T14:46:54.114946+0000","last_became_active":"2024-08-19T14:45:26.101929+0000","last_became_peered":"2024-08-19T14:45:26.101929+0000","last_unstale":"2024-08-19T14:46:54.114946+0000","last_undegraded":"2024-08-19T14:46:54.114946+0000","last_fullsized":"2024-08-19T14:46:54.114946+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:31.800766+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:31.800766+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,8],"acting":[6,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"893'9176","reported_seq":9321,"reported_epoch":893,"state":"active+clean","last_fresh":"2024-08-19T14:46:59.044802+0000","last_change":"2024-08-19T14:44:15.066789+0000","last_active":"2024-08-19T14:46:59.044802+0000","last_peered":"2024-08-19T14:46:59.044802+0000","last_clean":"2024-08-19T14:46:59.044802+0000","last_became_active":"2024-08-19T14:44:15.065981+0000","last_became_peered":"2024-08-19T14:44:15.065981+0000","last_unstale":"2024-08-19T14:46:59.044802+0000","last_undegraded":"2024-08-19T14:46:59.044802+0000","last_fullsized":"2024-08-19T14:46:59.044802+0000","mapping_epoch":826,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":827,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":9176,"log_dups_size":0,"ondisk_log_size":9176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173015040,"num_objects":2640,"num_object_clones":0,"num_object_copies":7920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2640,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9176,"num_write_kb":378112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.c","version":"70'139","reported_seq":756,"reported_epoch":893,"state":"active+clean","last_fresh":"2024-08-19T14:46:54.114887+0000","last_change":"2024-08-19T14:45:26.103908+0000","last_active":"2024-08-19T14:46:54.114887+0000","last_peered":"2024-08-19T14:46:54.114887+0000","last_clean":"2024-08-19T14:46:54.114887+0000","last_became_active":"2024-08-19T14:45:26.103255+0000","last_became_peered":"2024-08-19T14:45:26.103255+0000","last_unstale":"2024-08-19T14:46:54.114887+0000","last_undegraded":"2024-08-19T14:46:54.114887+0000","last_fullsized":"2024-08-19T14:46:54.114887+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:24.706574+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:22:52.164586+0000","last_clean_scrub_stamp":"2024-08-19T14:44:24.706574+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":983040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,1],"acting":[6,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"893'9087","reported_seq":13072,"reported_epoch":893,"state":"active+clean","last_fresh":"2024-08-19T14:46:57.745916+0000","last_change":"2024-08-19T14:45:06.219701+0000","last_active":"2024-08-19T14:46:57.745916+0000","last_peered":"2024-08-19T14:46:57.745916+0000","last_clean":"2024-08-19T14:46:57.745916+0000","last_became_active":"2024-08-19T14:45:06.216329+0000","last_became_peered":"2024-08-19T14:45:06.216329+0000","last_unstale":"2024-08-19T14:46:57.745916+0000","last_undegraded":"2024-08-19T14:46:57.745916+0000","last_fullsized":"2024-08-19T14:46:57.745916+0000","mapping_epoch":851,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":852,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":9087,"log_dups_size":0,"ondisk_log_size":9087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":170590208,"num_objects":2603,"num_object_clones":0,"num_object_copies":7809,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2603,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9087,"num_write_kb":374080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1915,"num_bytes_recovered":125501440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,9],"acting":[11,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.d","version":"70'139","reported_seq":536,"reported_epoch":893,"state":"active+clean","last_fresh":"2024-08-19T14:46:54.112253+0000","last_change":"2024-08-19T14:46:20.095131+0000","last_active":"2024-08-19T14:46:54.112253+0000","last_peered":"2024-08-19T14:46:54.112253+0000","last_clean":"2024-08-19T14:46:54.112253+0000","last_became_active":"2024-08-19T14:43:08.459577+0000","last_became_peered":"2024-08-19T14:43:08.459577+0000","last_unstale":"2024-08-19T14:46:54.112253+0000","last_undegraded":"2024-08-19T14:46:54.112253+0000","last_fullsized":"2024-08-19T14:46:54.112253+0000","mapping_epoch":793,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":794,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:46:20.095018+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:24:50.276408+0000","last_clean_scrub_stamp":"2024-08-19T14:46:20.095018+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:47:31.007933+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,8],"acting":[4,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.12","version":"70'139","reported_seq":1486,"reported_epoch":893,"state":"active+clean","last_fresh":"2024-08-19T14:46:54.112672+0000","last_change":"2024-08-19T14:45:26.109914+0000","last_active":"2024-08-19T14:46:54.112672+0000","last_peered":"2024-08-19T14:46:54.112672+0000","last_clean":"2024-08-19T14:46:54.112672+0000","last_became_active":"2024-08-19T14:45:26.109383+0000","last_became_peered":"2024-08-19T14:45:26.109383+0000","last_unstale":"2024-08-19T14:46:54.112672+0000","last_undegraded":"2024-08-19T14:46:54.112672+0000","last_fullsized":"2024-08-19T14:46:54.112672+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:30.604277+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:41:49.815826+0000","last_clean_scrub_stamp":"2024-08-19T14:44:30.604277+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,11],"acting":[1,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.13","version":"70'139","reported_seq":2019,"reported_epoch":893,"state":"active+clean","last_fresh":"2024-08-19T14:46:54.112454+0000","last_change":"2024-08-19T14:46:53.695559+0000","last_active":"2024-08-19T14:46:54.112454+0000","last_peered":"2024-08-19T14:46:54.112454+0000","last_clean":"2024-08-19T14:46:54.112454+0000","last_became_active":"2024-08-19T14:43:16.086968+0000","last_became_peered":"2024-08-19T14:43:16.086968+0000","last_unstale":"2024-08-19T14:46:54.112454+0000","last_undegraded":"2024-08-19T14:46:54.112454+0000","last_fullsized":"2024-08-19T14:46:54.112454+0000","mapping_epoch":800,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":801,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:46:53.695474+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:33:35.786105+0000","last_clean_scrub_stamp":"2024-08-19T14:46:53.695474+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:48:23.005145+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.10","version":"70'139","reported_seq":952,"reported_epoch":893,"state":"active+clean","last_fresh":"2024-08-19T14:46:54.112895+0000","last_change":"2024-08-19T14:44:38.187460+0000","last_active":"2024-08-19T14:46:54.112895+0000","last_peered":"2024-08-19T14:46:54.112895+0000","last_clean":"2024-08-19T14:46:54.112895+0000","last_became_active":"2024-08-19T14:44:38.187068+0000","last_became_peered":"2024-08-19T14:44:38.187068+0000","last_unstale":"2024-08-19T14:46:54.112895+0000","last_undegraded":"2024-08-19T14:46:54.112895+0000","last_fullsized":"2024-08-19T14:46:54.112895+0000","mapping_epoch":837,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:24.090078+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:34:58.616646+0000","last_clean_scrub_stamp":"2024-08-19T14:44:24.090078+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,4],"acting":[10,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.11","version":"70'139","reported_seq":533,"reported_epoch":893,"state":"active+clean","last_fresh":"2024-08-19T14:46:54.113354+0000","last_change":"2024-08-19T14:44:23.821840+0000","last_active":"2024-08-19T14:46:54.113354+0000","last_peered":"2024-08-19T14:46:54.113354+0000","last_clean":"2024-08-19T14:46:54.113354+0000","last_became_active":"2024-08-19T14:42:39.064403+0000","last_became_peered":"2024-08-19T14:42:39.064403+0000","last_unstale":"2024-08-19T14:46:54.113354+0000","last_undegraded":"2024-08-19T14:46:54.113354+0000","last_fullsized":"2024-08-19T14:46:54.113354+0000","mapping_epoch":778,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":779,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:23.821699+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:41:47.147376+0000","last_clean_scrub_stamp":"2024-08-19T14:44:23.821699+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,1],"acting":[8,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"70'139","reported_seq":1646,"reported_epoch":893,"state":"active+clean","last_fresh":"2024-08-19T14:46:55.739086+0000","last_change":"2024-08-19T14:46:55.739107+0000","last_active":"2024-08-19T14:46:55.739086+0000","last_peered":"2024-08-19T14:46:55.739086+0000","last_clean":"2024-08-19T14:46:55.739086+0000","last_became_active":"2024-08-19T14:44:38.183608+0000","last_became_peered":"2024-08-19T14:44:38.183608+0000","last_unstale":"2024-08-19T14:46:55.739086+0000","last_undegraded":"2024-08-19T14:46:55.739086+0000","last_fullsized":"2024-08-19T14:46:55.739086+0000","mapping_epoch":837,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:46:55.739034+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:28:34.216370+0000","last_clean_scrub_stamp":"2024-08-19T14:46:55.739034+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:48:22.771380+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,4],"acting":[1,11,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"70'139","reported_seq":1788,"reported_epoch":893,"state":"active+clean","last_fresh":"2024-08-19T14:46:54.113288+0000","last_change":"2024-08-19T14:44:38.184127+0000","last_active":"2024-08-19T14:46:54.113288+0000","last_peered":"2024-08-19T14:46:54.113288+0000","last_clean":"2024-08-19T14:46:54.113288+0000","last_became_active":"2024-08-19T14:44:38.183371+0000","last_became_peered":"2024-08-19T14:44:38.183371+0000","last_unstale":"2024-08-19T14:46:54.113288+0000","last_undegraded":"2024-08-19T14:46:54.113288+0000","last_fullsized":"2024-08-19T14:46:54.113288+0000","mapping_epoch":837,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:28.864662+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:28.864662+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11],"acting":[8,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.14","version":"70'139","reported_seq":1202,"reported_epoch":893,"state":"active+clean","last_fresh":"2024-08-19T14:46:54.112457+0000","last_change":"2024-08-19T14:46:34.796943+0000","last_active":"2024-08-19T14:46:54.112457+0000","last_peered":"2024-08-19T14:46:54.112457+0000","last_clean":"2024-08-19T14:46:54.112457+0000","last_became_active":"2024-08-19T14:43:08.454350+0000","last_became_peered":"2024-08-19T14:43:08.454350+0000","last_unstale":"2024-08-19T14:46:54.112457+0000","last_undegraded":"2024-08-19T14:46:54.112457+0000","last_fullsized":"2024-08-19T14:46:54.112457+0000","mapping_epoch":793,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":794,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:46:34.796863+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:46:34.796863+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:48:00.974974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,9],"acting":[1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"70'139","reported_seq":547,"reported_epoch":893,"state":"active+clean","last_fresh":"2024-08-19T14:46:54.433357+0000","last_change":"2024-08-19T14:46:54.433540+0000","last_active":"2024-08-19T14:46:54.433357+0000","last_peered":"2024-08-19T14:46:54.433357+0000","last_clean":"2024-08-19T14:46:54.433357+0000","last_became_active":"2024-08-19T14:44:38.199514+0000","last_became_peered":"2024-08-19T14:44:38.199514+0000","last_unstale":"2024-08-19T14:46:54.433357+0000","last_undegraded":"2024-08-19T14:46:54.433357+0000","last_fullsized":"2024-08-19T14:46:54.433357+0000","mapping_epoch":837,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:46:54.431820+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:20:09.180923+0000","last_clean_scrub_stamp":"2024-08-19T14:46:54.431820+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:48:09.163994+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,11],"acting":[8,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"70'139","reported_seq":1637,"reported_epoch":893,"state":"active+clean","last_fresh":"2024-08-19T14:46:54.112160+0000","last_change":"2024-08-19T14:46:15.973722+0000","last_active":"2024-08-19T14:46:54.112160+0000","last_peered":"2024-08-19T14:46:54.112160+0000","last_clean":"2024-08-19T14:46:54.112160+0000","last_became_active":"2024-08-19T14:43:11.471324+0000","last_became_peered":"2024-08-19T14:43:11.471324+0000","last_unstale":"2024-08-19T14:46:54.112160+0000","last_undegraded":"2024-08-19T14:46:54.112160+0000","last_fullsized":"2024-08-19T14:46:54.112160+0000","mapping_epoch":795,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":796,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:46:15.973642+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:41:51.832284+0000","last_clean_scrub_stamp":"2024-08-19T14:46:15.973642+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:47:25.118982+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,11],"acting":[4,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1b","version":"70'139","reported_seq":2011,"reported_epoch":893,"state":"active+clean","last_fresh":"2024-08-19T14:46:54.114376+0000","last_change":"2024-08-19T14:44:25.807033+0000","last_active":"2024-08-19T14:46:54.114376+0000","last_peered":"2024-08-19T14:46:54.114376+0000","last_clean":"2024-08-19T14:46:54.114376+0000","last_became_active":"2024-08-19T14:42:39.065604+0000","last_became_peered":"2024-08-19T14:42:39.065604+0000","last_unstale":"2024-08-19T14:46:54.114376+0000","last_undegraded":"2024-08-19T14:46:54.114376+0000","last_fullsized":"2024-08-19T14:46:54.114376+0000","mapping_epoch":778,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":779,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:25.806922+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:37:59.510697+0000","last_clean_scrub_stamp":"2024-08-19T14:44:25.806922+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,2],"acting":[8,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.18","version":"70'139","reported_seq":1007,"reported_epoch":893,"state":"active+clean","last_fresh":"2024-08-19T14:46:54.743777+0000","last_change":"2024-08-19T14:46:54.743821+0000","last_active":"2024-08-19T14:46:54.743777+0000","last_peered":"2024-08-19T14:46:54.743777+0000","last_clean":"2024-08-19T14:46:54.743777+0000","last_became_active":"2024-08-19T14:44:38.182712+0000","last_became_peered":"2024-08-19T14:44:38.182712+0000","last_unstale":"2024-08-19T14:46:54.743777+0000","last_undegraded":"2024-08-19T14:46:54.743777+0000","last_fullsized":"2024-08-19T14:46:54.743777+0000","mapping_epoch":837,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:46:54.743702+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:27:33.084161+0000","last_clean_scrub_stamp":"2024-08-19T14:46:54.743702+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:48:17.549331+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":402496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,11],"acting":[1,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.19","version":"70'139","reported_seq":864,"reported_epoch":893,"state":"active+clean","last_fresh":"2024-08-19T14:46:54.115243+0000","last_change":"2024-08-19T14:45:26.148752+0000","last_active":"2024-08-19T14:46:54.115243+0000","last_peered":"2024-08-19T14:46:54.115243+0000","last_clean":"2024-08-19T14:46:54.115243+0000","last_became_active":"2024-08-19T14:45:26.147921+0000","last_became_peered":"2024-08-19T14:45:26.147921+0000","last_unstale":"2024-08-19T14:46:54.115243+0000","last_undegraded":"2024-08-19T14:46:54.115243+0000","last_fullsized":"2024-08-19T14:46:54.115243+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:23.083891+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:44:23.083891+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,4],"acting":[10,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1e","version":"70'139","reported_seq":1025,"reported_epoch":893,"state":"active+clean","last_fresh":"2024-08-19T14:46:54.143174+0000","last_change":"2024-08-19T14:45:16.381698+0000","last_active":"2024-08-19T14:46:54.143174+0000","last_peered":"2024-08-19T14:46:54.143174+0000","last_clean":"2024-08-19T14:46:54.143174+0000","last_became_active":"2024-08-19T14:43:16.124258+0000","last_became_peered":"2024-08-19T14:43:16.124258+0000","last_unstale":"2024-08-19T14:46:54.143174+0000","last_undegraded":"2024-08-19T14:46:54.143174+0000","last_fullsized":"2024-08-19T14:46:54.143174+0000","mapping_epoch":800,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":801,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:45:16.381587+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:27:04.780939+0000","last_clean_scrub_stamp":"2024-08-19T14:45:16.381587+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,10],"acting":[11,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1c","version":"70'139","reported_seq":1773,"reported_epoch":893,"state":"active+clean","last_fresh":"2024-08-19T14:46:54.113136+0000","last_change":"2024-08-19T14:45:26.106081+0000","last_active":"2024-08-19T14:46:54.113136+0000","last_peered":"2024-08-19T14:46:54.113136+0000","last_clean":"2024-08-19T14:46:54.113136+0000","last_became_active":"2024-08-19T14:45:26.105382+0000","last_became_peered":"2024-08-19T14:45:26.105382+0000","last_unstale":"2024-08-19T14:46:54.113136+0000","last_undegraded":"2024-08-19T14:46:54.113136+0000","last_fullsized":"2024-08-19T14:46:54.113136+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:31.796263+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:44:31.796263+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1d","version":"70'139","reported_seq":1665,"reported_epoch":893,"state":"active+clean","last_fresh":"2024-08-19T14:46:54.112840+0000","last_change":"2024-08-19T14:44:29.650415+0000","last_active":"2024-08-19T14:46:54.112840+0000","last_peered":"2024-08-19T14:46:54.112840+0000","last_clean":"2024-08-19T14:46:54.112840+0000","last_became_active":"2024-08-19T14:35:11.540488+0000","last_became_peered":"2024-08-19T14:35:11.540488+0000","last_unstale":"2024-08-19T14:46:54.112840+0000","last_undegraded":"2024-08-19T14:46:54.112840+0000","last_fullsized":"2024-08-19T14:46:54.112840+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:29.650238+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:44:29.650238+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":2800418840,"num_objects":42732,"num_object_clones":0,"num_object_copies":128196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":409,"num_objects_unfound":0,"num_objects_dirty":42732,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":146274,"num_write_kb":6048129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39282,"num_bytes_recovered":2574385152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8858251264,"data_stored":8858239048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":146274,"ondisk_log_size":146274,"up":48,"acting":47,"num_store_stats":9},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1385536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2785280,"data_stored":2771072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187883,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2251096,"kb_used_data":1031952,"kb_used_omap":1,"kb_used_meta":1219070,"kb_avail":91494056,"statfs":{"total":95995035648,"available":93689913344,"internally_reserved":0,"allocated":1056718848,"data_stored":1054051175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":1248328132},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":539,"seq":2314987372750,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1457876,"kb_used_data":1245384,"kb_used_omap":4,"kb_used_meta":212411,"kb_avail":92287276,"statfs":{"total":95995035648,"available":94502170624,"internally_reserved":0,"allocated":1275273216,"data_stored":1272609095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4407,"internal_metadata":217509577},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":74,"seq":317827580341,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2739856,"kb_used_data":1506184,"kb_used_omap":1,"kb_used_meta":1233662,"kb_avail":91005296,"statfs":{"total":95995035648,"available":93189423104,"internally_reserved":0,"allocated":1542332416,"data_stored":1539668295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":1263270337},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874683,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2219224,"kb_used_data":1191244,"kb_used_omap":1,"kb_used_meta":1027966,"kb_avail":91525928,"statfs":{"total":95995035648,"available":93722550272,"internally_reserved":0,"allocated":1219833856,"data_stored":1217165663,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1052637637},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":0,"seq":244813136319,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430658,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1035524,"kb_used_data":864124,"kb_used_omap":1,"kb_used_meta":171390,"kb_avail":92709628,"statfs":{"total":95995035648,"available":94934659072,"internally_reserved":0,"allocated":884862976,"data_stored":882203073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":175503814},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":171798692294,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986635,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1810064,"kb_used_data":552640,"kb_used_omap":1,"kb_used_meta":1257406,"kb_avail":91935088,"statfs":{"total":95995035648,"available":94141530112,"internally_reserved":0,"allocated":565903360,"data_stored":563239385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1287584197},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":111669150160,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":634,"seq":2723009265806,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":736664,"kb_used_data":186768,"kb_used_omap":4,"kb_used_meta":549883,"kb_avail":93008488,"statfs":{"total":95995035648,"available":95240691712,"internally_reserved":0,"allocated":191250432,"data_stored":188582759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4407,"internal_metadata":563080905},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575322,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3464028,"kb_used_data":1997460,"kb_used_omap":1,"kb_used_meta":1466558,"kb_avail":90281124,"statfs":{"total":95995035648,"available":92447870976,"internally_reserved":0,"allocated":2045399040,"data_stored":2042727295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1501755845},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":34359738845,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2020806656,"data_stored":2020802584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":180748288,"data_stored":180748288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":550965248,"data_stored":550961176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":218497024,"data_stored":218497024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":876937216,"data_stored":876937216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1198133248,"data_stored":1198129176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1528889344,"data_stored":1528889344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1247608832,"data_stored":1247608832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1035665408,"data_stored":1035665408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T14:47:01.184 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-19T14:47:01.219 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:47:01.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:01.252 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:01.252 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:47:01.332 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:47:01.353 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:47:01.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:01.375 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:01.375 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:47:01.476 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:47:01.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:01.499 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:01.499 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:47:01.600 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:47:01.631 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:01.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:01.632 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:47:01.733 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:47:01.744 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-08-19T14:47:01.755 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:01.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:01.756 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:47:01.764 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:01 smithi031 ceph-mon[16336]: 9.1 scrub starts 2024-08-19T14:47:01.764 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:01 smithi031 ceph-mon[16336]: 9.8 scrub starts 2024-08-19T14:47:01.764 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:01 smithi031 ceph-mon[16336]: pgmap v2107: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 2.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 797 op/s; 409/125310 objects misplaced (0.326%); 6.6 MiB/s, 105 objects/s recovering 2024-08-19T14:47:01.764 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:01 smithi031 ceph-mon[16336]: 1.8 deep-scrub starts 2024-08-19T14:47:01.764 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:01 smithi031 ceph-mon[16336]: 1.8 deep-scrub ok 2024-08-19T14:47:01.764 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:01 smithi031 ceph-mon[16336]: osdmap e895: 12 total, 12 up, 8 in 2024-08-19T14:47:01.764 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:01 smithi031 ceph-mon[16336]: 9.1 scrub ok 2024-08-19T14:47:01.764 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:01 smithi031 ceph-mon[16336]: 9.c scrub ok 2024-08-19T14:47:01.764 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:01 smithi031 ceph-mon[16336]: 9.8 scrub ok 2024-08-19T14:47:01.765 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:01 smithi031 ceph-mon[16336]: 9.5 scrub starts 2024-08-19T14:47:01.777 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:01 smithi032 ceph-mon[20004]: 9.1 scrub starts 2024-08-19T14:47:01.777 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:01 smithi032 ceph-mon[20004]: 9.8 scrub starts 2024-08-19T14:47:01.777 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:01 smithi032 ceph-mon[20004]: pgmap v2107: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 2.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 797 op/s; 409/125310 objects misplaced (0.326%); 6.6 MiB/s, 105 objects/s recovering 2024-08-19T14:47:01.777 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:01 smithi032 ceph-mon[20004]: 1.8 deep-scrub starts 2024-08-19T14:47:01.777 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:01 smithi032 ceph-mon[20004]: 1.8 deep-scrub ok 2024-08-19T14:47:01.777 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:01 smithi032 ceph-mon[20004]: osdmap e895: 12 total, 12 up, 8 in 2024-08-19T14:47:01.777 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:01 smithi032 ceph-mon[20004]: 9.1 scrub ok 2024-08-19T14:47:01.777 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:01 smithi032 ceph-mon[20004]: 9.c scrub ok 2024-08-19T14:47:01.777 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:01 smithi032 ceph-mon[20004]: 9.8 scrub ok 2024-08-19T14:47:01.777 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:01 smithi032 ceph-mon[20004]: 9.5 scrub starts 2024-08-19T14:47:01.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:01 smithi045 ceph-mon[20143]: 9.1 scrub starts 2024-08-19T14:47:01.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:01 smithi045 ceph-mon[20143]: 9.8 scrub starts 2024-08-19T14:47:01.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:01 smithi045 ceph-mon[20143]: pgmap v2107: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 2.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 797 op/s; 409/125310 objects misplaced (0.326%); 6.6 MiB/s, 105 objects/s recovering 2024-08-19T14:47:01.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:01 smithi045 ceph-mon[20143]: 1.8 deep-scrub starts 2024-08-19T14:47:01.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:01 smithi045 ceph-mon[20143]: 1.8 deep-scrub ok 2024-08-19T14:47:01.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:01 smithi045 ceph-mon[20143]: osdmap e895: 12 total, 12 up, 8 in 2024-08-19T14:47:01.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:01 smithi045 ceph-mon[20143]: 9.1 scrub ok 2024-08-19T14:47:01.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:01 smithi045 ceph-mon[20143]: 9.c scrub ok 2024-08-19T14:47:01.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:01 smithi045 ceph-mon[20143]: 9.8 scrub ok 2024-08-19T14:47:01.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:01 smithi045 ceph-mon[20143]: 9.5 scrub starts 2024-08-19T14:47:01.857 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:47:01.880 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:01.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:01.881 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:47:01.892 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:47:01.892 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:47:01.912 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:47:01.912 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:47:01.932 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:47:01.932 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:47:01.951 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:47:01.951 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:47:01.978 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:47:01.979 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:47:01.983 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:47:02.008 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:47:02.009 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:47:02.020 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:02.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:02.021 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:47:02.033 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:47:02.033 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:47:02.037 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:47:02.039 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:47:02.057 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:47:02.057 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:47:02.092 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:47:02.092 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:47:02.122 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:47:02.123 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:47:02.123 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:47:02.146 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:02.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:02.147 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:47:02.151 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:47:02.152 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:47:02.190 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:47:02.190 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:47:02.247 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:47:02.270 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:02.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:02.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:47:02.372 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:47:02.407 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:02.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:02.408 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:47:02.509 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:47:02.534 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:02.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:02.535 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:47:02.590 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:47:02.590 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-08-19T14:47:02.590 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-08-19T14:47:02.590 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-08-19T14:47:02.590 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.590 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.16962.0:232603 7.1 7:8aa3ba8d:::benchmark_data_smithi045_115842_object114443:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e707)", 2024-08-19T14:47:02.590 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:39:29.443908+0000", 2024-08-19T14:47:02.591 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 453.11749383199998, 2024-08-19T14:47:02.591 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.32146603600000001, 2024-08-19T14:47:02.591 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:47:02.591 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:47:02.591 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:47:02.591 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:47:02.591 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.16962", 2024-08-19T14:47:02.591 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:47:02.591 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 232603 2024-08-19T14:47:02.591 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.591 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:47:02.591 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.591 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:47:02.591 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:39:29.443908+0000", 2024-08-19T14:47:02.592 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:02.592 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.592 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.592 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:47:02.592 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:39:29.443908+0000", 2024-08-19T14:47:02.592 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:02.592 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.592 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.592 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:47:02.592 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:39:29.443909+0000", 2024-08-19T14:47:02.592 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3689999999999999e-06 2024-08-19T14:47:02.592 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.592 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.592 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:47:02.593 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:39:29.443916+0000", 2024-08-19T14:47:02.593 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2219999999999999e-06 2024-08-19T14:47:02.593 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.593 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.593 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:47:02.593 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:39:29.443930+0000", 2024-08-19T14:47:02.593 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4688999999999999e-05 2024-08-19T14:47:02.593 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.593 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.593 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:47:02.593 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:39:29.443938+0000", 2024-08-19T14:47:02.593 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.926e-06 2024-08-19T14:47:02.593 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.593 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.594 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:47:02.594 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:39:29.444001+0000", 2024-08-19T14:47:02.594 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2948999999999996e-05 2024-08-19T14:47:02.594 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.594 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.594 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:47:02.594 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:39:29.444134+0000", 2024-08-19T14:47:02.594 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013257199999999999 2024-08-19T14:47:02.594 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.594 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.594 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 1,3", 2024-08-19T14:47:02.594 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:39:29.444262+0000", 2024-08-19T14:47:02.594 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000128627 2024-08-19T14:47:02.594 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.595 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.595 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:47:02.595 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:39:29.450410+0000", 2024-08-19T14:47:02.595 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0061473880000000002 2024-08-19T14:47:02.595 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.595 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.595 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:47:02.595 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:39:29.465848+0000", 2024-08-19T14:47:02.595 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.015437879 2024-08-19T14:47:02.595 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.595 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.595 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:47:02.595 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:39:29.765269+0000", 2024-08-19T14:47:02.595 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.29942160699999998 2024-08-19T14:47:02.596 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.596 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.596 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:47:02.596 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:39:29.765334+0000", 2024-08-19T14:47:02.596 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4339999999999997e-05 2024-08-19T14:47:02.596 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.596 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.596 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:47:02.596 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:39:29.765374+0000", 2024-08-19T14:47:02.596 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0467999999999999e-05 2024-08-19T14:47:02.596 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:02.596 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:47:02.596 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:02.596 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.597 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.597 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.16962.0:232604 7.1 7:828cc103:::benchmark_data_smithi045_115842_object114444:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e707)", 2024-08-19T14:47:02.597 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:39:29.447262+0000", 2024-08-19T14:47:02.597 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 453.11414026400001, 2024-08-19T14:47:02.597 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.31826677399999997, 2024-08-19T14:47:02.597 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:47:02.597 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:47:02.597 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:47:02.597 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:47:02.597 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.16962", 2024-08-19T14:47:02.597 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:47:02.597 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 232604 2024-08-19T14:47:02.597 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.597 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:47:02.598 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.598 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:47:02.598 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:39:29.447262+0000", 2024-08-19T14:47:02.598 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:02.598 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.598 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.598 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:47:02.598 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:39:29.447262+0000", 2024-08-19T14:47:02.598 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:02.598 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.598 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.598 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:47:02.598 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:39:29.447262+0000", 2024-08-19T14:47:02.598 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2500000000000005e-07 2024-08-19T14:47:02.599 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.599 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.599 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:47:02.599 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:39:29.447267+0000", 2024-08-19T14:47:02.599 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4270000000000001e-06 2024-08-19T14:47:02.599 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.599 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.599 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:47:02.599 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:39:29.447278+0000", 2024-08-19T14:47:02.599 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1442e-05 2024-08-19T14:47:02.599 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.599 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.599 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:47:02.600 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:39:29.447284+0000", 2024-08-19T14:47:02.600 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5809999999999996e-06 2024-08-19T14:47:02.600 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.601 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.601 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:47:02.601 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:39:29.448049+0000", 2024-08-19T14:47:02.601 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00076470100000000001 2024-08-19T14:47:02.601 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.601 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.602 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:47:02.602 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:39:29.448121+0000", 2024-08-19T14:47:02.602 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2477000000000004e-05 2024-08-19T14:47:02.602 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.602 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.602 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 1,3", 2024-08-19T14:47:02.602 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:39:29.448191+0000", 2024-08-19T14:47:02.602 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9882000000000006e-05 2024-08-19T14:47:02.602 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.602 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.602 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:47:02.602 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:39:29.450498+0000", 2024-08-19T14:47:02.602 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0023071279999999999 2024-08-19T14:47:02.603 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.603 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.603 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:47:02.603 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:39:29.471030+0000", 2024-08-19T14:47:02.603 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.020532321999999999 2024-08-19T14:47:02.603 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.603 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.603 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:47:02.603 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:39:29.765429+0000", 2024-08-19T14:47:02.603 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.29439832199999999 2024-08-19T14:47:02.603 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.603 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.603 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:47:02.603 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:39:29.765484+0000", 2024-08-19T14:47:02.604 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5464999999999998e-05 2024-08-19T14:47:02.604 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.604 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.604 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:47:02.604 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:39:29.765528+0000", 2024-08-19T14:47:02.604 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4301999999999997e-05 2024-08-19T14:47:02.604 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:02.604 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:47:02.604 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:02.604 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.604 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.604 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.16962.0:232606 7.1 7:843cca17:::benchmark_data_smithi045_115842_object114446:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e707)", 2024-08-19T14:47:02.604 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:39:29.462760+0000", 2024-08-19T14:47:02.604 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 453.09864231900002, 2024-08-19T14:47:02.605 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.30288721000000002, 2024-08-19T14:47:02.605 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:47:02.605 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:47:02.605 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:47:02.605 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:47:02.605 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.16962", 2024-08-19T14:47:02.605 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:47:02.605 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 232606 2024-08-19T14:47:02.605 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.605 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:47:02.605 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.605 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:47:02.605 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:39:29.462760+0000", 2024-08-19T14:47:02.605 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:02.606 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.606 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.606 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:47:02.606 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:39:29.462760+0000", 2024-08-19T14:47:02.606 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:02.606 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.606 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.606 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:47:02.606 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:39:29.462760+0000", 2024-08-19T14:47:02.606 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.4e-07 2024-08-19T14:47:02.606 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.606 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.606 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:47:02.606 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:39:29.462764+0000", 2024-08-19T14:47:02.607 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7790000000000002e-06 2024-08-19T14:47:02.607 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.607 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.607 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:47:02.607 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:39:29.462775+0000", 2024-08-19T14:47:02.607 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.059e-05 2024-08-19T14:47:02.607 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.607 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.607 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:47:02.607 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:39:29.462780+0000", 2024-08-19T14:47:02.607 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.468e-06 2024-08-19T14:47:02.607 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.607 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.607 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:47:02.608 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:39:29.462830+0000", 2024-08-19T14:47:02.608 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9886000000000003e-05 2024-08-19T14:47:02.608 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.608 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.608 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:47:02.608 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:39:29.462943+0000", 2024-08-19T14:47:02.608 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00011240599999999999 2024-08-19T14:47:02.608 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.608 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.608 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 1,3", 2024-08-19T14:47:02.608 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:39:29.463072+0000", 2024-08-19T14:47:02.608 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012900800000000001 2024-08-19T14:47:02.608 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.608 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.608 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:47:02.609 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:39:29.465642+0000", 2024-08-19T14:47:02.609 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0025699709999999999 2024-08-19T14:47:02.609 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.609 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.609 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:47:02.609 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:39:29.471119+0000", 2024-08-19T14:47:02.609 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0054773260000000002 2024-08-19T14:47:02.609 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.609 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.609 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:47:02.609 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:39:29.765570+0000", 2024-08-19T14:47:02.609 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.29445149599999998 2024-08-19T14:47:02.609 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.609 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.610 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:47:02.610 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:39:29.765607+0000", 2024-08-19T14:47:02.610 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6171000000000003e-05 2024-08-19T14:47:02.610 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.610 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.610 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:47:02.610 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:39:29.765647+0000", 2024-08-19T14:47:02.610 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0268999999999997e-05 2024-08-19T14:47:02.610 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:02.610 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:47:02.610 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:02.610 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.610 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.610 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.16962.0:232611 7.1 7:8e06892f:::benchmark_data_smithi045_115842_object114451:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e707)", 2024-08-19T14:47:02.611 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:39:29.471274+0000", 2024-08-19T14:47:02.611 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 453.09012791100002, 2024-08-19T14:47:02.611 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.29452909599999999, 2024-08-19T14:47:02.611 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:47:02.611 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:47:02.611 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:47:02.611 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:47:02.611 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.16962", 2024-08-19T14:47:02.611 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:47:02.611 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 232611 2024-08-19T14:47:02.611 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.611 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:47:02.611 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.611 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:47:02.612 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:39:29.471274+0000", 2024-08-19T14:47:02.612 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:02.612 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.612 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.612 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:47:02.612 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:39:29.471274+0000", 2024-08-19T14:47:02.612 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:02.612 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.612 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.612 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:47:02.612 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:39:29.471275+0000", 2024-08-19T14:47:02.612 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.4300000000000001e-07 2024-08-19T14:47:02.612 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.612 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.613 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:47:02.613 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:39:29.471279+0000", 2024-08-19T14:47:02.613 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0899999999999998e-06 2024-08-19T14:47:02.613 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.613 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.613 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:47:02.613 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:39:29.471289+0000", 2024-08-19T14:47:02.613 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0205e-05 2024-08-19T14:47:02.613 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.613 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.613 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:47:02.613 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:39:29.471295+0000", 2024-08-19T14:47:02.613 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2889999999999997e-06 2024-08-19T14:47:02.613 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.613 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.614 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:47:02.614 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:39:29.471323+0000", 2024-08-19T14:47:02.614 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8331999999999999e-05 2024-08-19T14:47:02.614 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.614 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.614 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:47:02.614 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:39:29.471399+0000", 2024-08-19T14:47:02.614 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6168999999999995e-05 2024-08-19T14:47:02.614 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.614 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.614 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 1,3", 2024-08-19T14:47:02.614 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:39:29.471466+0000", 2024-08-19T14:47:02.614 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6950000000000001e-05 2024-08-19T14:47:02.614 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.615 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.615 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:47:02.615 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:39:29.474823+0000", 2024-08-19T14:47:02.615 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0033573819999999999 2024-08-19T14:47:02.615 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.615 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.615 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:47:02.615 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:39:29.486682+0000", 2024-08-19T14:47:02.615 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011858535999999999 2024-08-19T14:47:02.615 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.615 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.615 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:47:02.615 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:39:29.765704+0000", 2024-08-19T14:47:02.615 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.279022402 2024-08-19T14:47:02.616 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.616 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.616 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:47:02.616 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:39:29.765756+0000", 2024-08-19T14:47:02.616 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2077000000000003e-05 2024-08-19T14:47:02.616 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.616 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.616 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:47:02.616 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:39:29.765803+0000", 2024-08-19T14:47:02.616 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6721000000000002e-05 2024-08-19T14:47:02.616 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:02.616 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:47:02.616 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:02.616 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.616 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.617 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.16962.0:232615 7.1 7:8184689b:::benchmark_data_smithi045_115842_object114455:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e707)", 2024-08-19T14:47:02.617 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:39:29.475906+0000", 2024-08-19T14:47:02.617 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 453.08549571499998, 2024-08-19T14:47:02.617 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.29003925200000003, 2024-08-19T14:47:02.617 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:47:02.617 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:47:02.617 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:47:02.617 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:47:02.617 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.16962", 2024-08-19T14:47:02.617 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/695557578", 2024-08-19T14:47:02.617 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 232615 2024-08-19T14:47:02.617 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.617 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:47:02.617 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.617 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:47:02.618 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:39:29.475906+0000", 2024-08-19T14:47:02.618 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:02.618 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.618 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.618 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:47:02.618 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:39:29.475906+0000", 2024-08-19T14:47:02.618 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:02.618 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.618 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.618 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:47:02.618 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:39:29.475907+0000", 2024-08-19T14:47:02.618 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.9899999999999999e-07 2024-08-19T14:47:02.618 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.618 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.619 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:47:02.619 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:39:29.475911+0000", 2024-08-19T14:47:02.619 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9550000000000002e-06 2024-08-19T14:47:02.619 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.619 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.619 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:47:02.619 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:39:29.475922+0000", 2024-08-19T14:47:02.619 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0858e-05 2024-08-19T14:47:02.619 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.619 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.619 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:47:02.619 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:39:29.475928+0000", 2024-08-19T14:47:02.619 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5790000000000003e-06 2024-08-19T14:47:02.619 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.620 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.620 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:47:02.620 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:39:29.475996+0000", 2024-08-19T14:47:02.620 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8455999999999996e-05 2024-08-19T14:47:02.620 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.620 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.620 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:47:02.620 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:39:29.476116+0000", 2024-08-19T14:47:02.620 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012011 2024-08-19T14:47:02.620 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.620 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.620 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 1,3", 2024-08-19T14:47:02.620 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:39:29.476243+0000", 2024-08-19T14:47:02.620 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012652 2024-08-19T14:47:02.620 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.621 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.621 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:47:02.621 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:39:29.479495+0000", 2024-08-19T14:47:02.621 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0032519210000000001 2024-08-19T14:47:02.621 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.621 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.621 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:47:02.621 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:39:29.486763+0000", 2024-08-19T14:47:02.621 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0072687020000000001 2024-08-19T14:47:02.621 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.621 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.621 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:47:02.621 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:39:29.765851+0000", 2024-08-19T14:47:02.621 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.279088004 2024-08-19T14:47:02.621 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.622 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.622 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:47:02.622 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:39:29.765905+0000", 2024-08-19T14:47:02.622 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3628999999999998e-05 2024-08-19T14:47:02.622 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.622 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.622 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:47:02.622 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:39:29.765945+0000", 2024-08-19T14:47:02.622 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0618999999999999e-05 2024-08-19T14:47:02.622 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:02.622 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:47:02.622 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:02.622 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.622 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.623 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26911.0:57625 8.5 8:a05814a0:::benchmark_data_smithi045_137103_object57624:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e728)", 2024-08-19T14:47:02.623 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:40:36.493525+0000", 2024-08-19T14:47:02.623 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 386.06787656400002, 2024-08-19T14:47:02.623 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.23983030699999999, 2024-08-19T14:47:02.623 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:47:02.623 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:47:02.623 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:47:02.623 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:47:02.623 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26911", 2024-08-19T14:47:02.623 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3494993995", 2024-08-19T14:47:02.623 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 57625 2024-08-19T14:47:02.623 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.623 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:47:02.623 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.624 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:47:02.624 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:40:36.493525+0000", 2024-08-19T14:47:02.624 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:02.624 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.624 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.624 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:47:02.624 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:40:36.493525+0000", 2024-08-19T14:47:02.624 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:02.624 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.624 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.624 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:47:02.624 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:40:36.493526+0000", 2024-08-19T14:47:02.624 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5100000000000001e-07 2024-08-19T14:47:02.624 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.624 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.625 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:47:02.625 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:40:36.493551+0000", 2024-08-19T14:47:02.625 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5664e-05 2024-08-19T14:47:02.625 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.625 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.625 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:47:02.625 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:40:36.493557+0000", 2024-08-19T14:47:02.625 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9950000000000002e-06 2024-08-19T14:47:02.625 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.625 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.625 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:47:02.625 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:40:36.493560+0000", 2024-08-19T14:47:02.625 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1580000000000001e-06 2024-08-19T14:47:02.625 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.626 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.626 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:47:02.626 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:40:36.506851+0000", 2024-08-19T14:47:02.626 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.013290786000000001 2024-08-19T14:47:02.626 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.626 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.626 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:47:02.626 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:40:36.506918+0000", 2024-08-19T14:47:02.626 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6676999999999999e-05 2024-08-19T14:47:02.626 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.626 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.626 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 1,11", 2024-08-19T14:47:02.626 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:40:36.507004+0000", 2024-08-19T14:47:02.626 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6033999999999997e-05 2024-08-19T14:47:02.627 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.627 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.627 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:47:02.627 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:40:36.528370+0000", 2024-08-19T14:47:02.627 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.021365884000000002 2024-08-19T14:47:02.627 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.627 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.627 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:47:02.627 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:40:36.528404+0000", 2024-08-19T14:47:02.627 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3844000000000003e-05 2024-08-19T14:47:02.627 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.627 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.627 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:47:02.627 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:40:36.733211+0000", 2024-08-19T14:47:02.627 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20480709699999999 2024-08-19T14:47:02.628 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.628 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.628 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:47:02.628 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:40:36.733301+0000", 2024-08-19T14:47:02.628 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.0598000000000002e-05 2024-08-19T14:47:02.628 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.628 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.628 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:47:02.628 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:40:36.733356+0000", 2024-08-19T14:47:02.628 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4318999999999999e-05 2024-08-19T14:47:02.628 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:02.628 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:47:02.628 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:02.628 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.629 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.629 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26911.0:168610 8.9 8.9e8d6cc9 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e775)", 2024-08-19T14:47:02.629 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:42:35.131749+0000", 2024-08-19T14:47:02.629 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 267.42965342899998, 2024-08-19T14:47:02.629 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.992208443, 2024-08-19T14:47:02.629 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:47:02.629 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:47:02.629 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:47:02.629 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:47:02.629 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26911", 2024-08-19T14:47:02.629 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3494993995", 2024-08-19T14:47:02.629 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 168610 2024-08-19T14:47:02.629 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.630 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:47:02.630 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.630 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:47:02.630 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.131749+0000", 2024-08-19T14:47:02.630 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:02.630 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.630 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.630 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:47:02.630 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.131749+0000", 2024-08-19T14:47:02.630 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:02.630 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.630 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.631 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:47:02.631 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.131750+0000", 2024-08-19T14:47:02.631 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3430000000000001e-06 2024-08-19T14:47:02.631 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.631 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.631 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:47:02.631 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.131755+0000", 2024-08-19T14:47:02.631 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5380000000000002e-06 2024-08-19T14:47:02.631 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.631 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.631 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:47:02.631 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.131768+0000", 2024-08-19T14:47:02.631 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2655e-05 2024-08-19T14:47:02.632 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.632 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.632 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:47:02.632 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.131775+0000", 2024-08-19T14:47:02.632 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3019999999999997e-06 2024-08-19T14:47:02.632 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.632 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.632 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:47:02.632 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.131817+0000", 2024-08-19T14:47:02.632 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1828e-05 2024-08-19T14:47:02.632 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.632 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.632 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:47:02.632 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.131830+0000", 2024-08-19T14:47:02.633 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2756e-05 2024-08-19T14:47:02.633 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.633 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.633 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:47:02.633 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:36.123945+0000", 2024-08-19T14:47:02.633 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.99211474600000005 2024-08-19T14:47:02.633 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.633 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.633 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:47:02.633 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:36.123957+0000", 2024-08-19T14:47:02.633 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2275e-05 2024-08-19T14:47:02.633 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:02.633 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:47:02.634 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:02.634 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.634 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.634 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26911.0:168627 8.14 8.ac90e994 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e775)", 2024-08-19T14:47:02.634 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:42:35.148878+0000", 2024-08-19T14:47:02.634 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 267.412523733, 2024-08-19T14:47:02.634 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.97531405299999996, 2024-08-19T14:47:02.634 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:47:02.634 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:47:02.634 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:47:02.634 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:47:02.634 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26911", 2024-08-19T14:47:02.634 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3494993995", 2024-08-19T14:47:02.634 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 168627 2024-08-19T14:47:02.635 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.635 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:47:02.635 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.635 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:47:02.635 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.148878+0000", 2024-08-19T14:47:02.635 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:02.635 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.635 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.635 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:47:02.635 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.148878+0000", 2024-08-19T14:47:02.635 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:02.635 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.635 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.635 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:47:02.636 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.148879+0000", 2024-08-19T14:47:02.636 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7800000000000001e-07 2024-08-19T14:47:02.636 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.636 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.636 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:47:02.636 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.148882+0000", 2024-08-19T14:47:02.636 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0970000000000002e-06 2024-08-19T14:47:02.636 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.636 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.636 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:47:02.636 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.148890+0000", 2024-08-19T14:47:02.636 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.3909999999999998e-06 2024-08-19T14:47:02.636 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.636 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.637 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:47:02.637 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.148895+0000", 2024-08-19T14:47:02.637 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5220000000000001e-06 2024-08-19T14:47:02.637 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.637 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.637 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:47:02.637 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.148943+0000", 2024-08-19T14:47:02.637 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7899000000000003e-05 2024-08-19T14:47:02.637 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.637 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.637 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:47:02.637 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.148950+0000", 2024-08-19T14:47:02.637 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.391e-06 2024-08-19T14:47:02.637 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.638 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.638 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:47:02.638 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:36.124139+0000", 2024-08-19T14:47:02.638 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.97518876300000001 2024-08-19T14:47:02.638 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.638 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.638 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:47:02.638 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:36.124192+0000", 2024-08-19T14:47:02.638 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3412000000000003e-05 2024-08-19T14:47:02.638 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:02.638 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:47:02.638 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:02.638 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.638 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.639 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26911.0:168640 8.14 8.9692eab4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e775)", 2024-08-19T14:47:02.639 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:42:35.172363+0000", 2024-08-19T14:47:02.639 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 267.389038513, 2024-08-19T14:47:02.639 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.95187207100000004, 2024-08-19T14:47:02.639 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:47:02.639 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:47:02.639 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:47:02.639 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:47:02.639 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26911", 2024-08-19T14:47:02.639 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3494993995", 2024-08-19T14:47:02.639 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 168640 2024-08-19T14:47:02.639 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.639 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:47:02.639 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.640 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:47:02.640 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.172363+0000", 2024-08-19T14:47:02.640 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:02.640 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.640 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.640 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:47:02.640 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.172363+0000", 2024-08-19T14:47:02.640 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:02.640 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.640 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.640 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:47:02.640 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.172365+0000", 2024-08-19T14:47:02.640 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0699999999999999e-06 2024-08-19T14:47:02.641 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.641 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.641 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:47:02.641 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.172369+0000", 2024-08-19T14:47:02.641 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8670000000000002e-06 2024-08-19T14:47:02.641 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.641 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.641 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:47:02.641 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.172379+0000", 2024-08-19T14:47:02.641 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.4520000000000003e-06 2024-08-19T14:47:02.641 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.641 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.641 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:47:02.641 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.172384+0000", 2024-08-19T14:47:02.641 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8550000000000001e-06 2024-08-19T14:47:02.642 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.642 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.642 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:47:02.642 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.172433+0000", 2024-08-19T14:47:02.642 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9790999999999998e-05 2024-08-19T14:47:02.642 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.642 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.642 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:47:02.642 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.172445+0000", 2024-08-19T14:47:02.642 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1873e-05 2024-08-19T14:47:02.642 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.642 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.642 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:47:02.642 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:36.124225+0000", 2024-08-19T14:47:02.642 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.95177953500000001 2024-08-19T14:47:02.643 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.643 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.643 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:47:02.643 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:36.124236+0000", 2024-08-19T14:47:02.643 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0628000000000001e-05 2024-08-19T14:47:02.643 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:02.643 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:47:02.643 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:02.643 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.643 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.643 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26911.0:168648 8.e 8.7b98db7e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e775)", 2024-08-19T14:47:02.643 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:42:35.178381+0000", 2024-08-19T14:47:02.643 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 267.38302052099999, 2024-08-19T14:47:02.643 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.94654472899999997, 2024-08-19T14:47:02.643 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:47:02.644 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:47:02.644 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:47:02.644 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:47:02.644 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26911", 2024-08-19T14:47:02.644 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3494993995", 2024-08-19T14:47:02.644 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 168648 2024-08-19T14:47:02.644 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.644 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:47:02.644 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.644 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:47:02.644 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.178381+0000", 2024-08-19T14:47:02.644 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:02.644 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.644 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.645 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:47:02.645 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.178381+0000", 2024-08-19T14:47:02.645 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:02.645 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.645 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.645 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:47:02.645 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.178382+0000", 2024-08-19T14:47:02.645 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7800000000000001e-07 2024-08-19T14:47:02.645 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.645 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.645 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:47:02.645 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.178385+0000", 2024-08-19T14:47:02.645 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.072e-06 2024-08-19T14:47:02.645 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.645 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.646 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:47:02.646 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.178393+0000", 2024-08-19T14:47:02.646 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.8849999999999999e-06 2024-08-19T14:47:02.646 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.646 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.646 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:47:02.646 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.178398+0000", 2024-08-19T14:47:02.646 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5049999999999998e-06 2024-08-19T14:47:02.646 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.646 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.646 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:47:02.646 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.178462+0000", 2024-08-19T14:47:02.646 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3324999999999994e-05 2024-08-19T14:47:02.646 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.646 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.647 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:47:02.647 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.178472+0000", 2024-08-19T14:47:02.647 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0339999999999999e-05 2024-08-19T14:47:02.647 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.647 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.647 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:47:02.647 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:36.124128+0000", 2024-08-19T14:47:02.647 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.94565564599999996 2024-08-19T14:47:02.647 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.647 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.647 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:47:02.647 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:36.124926+0000", 2024-08-19T14:47:02.647 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00079837799999999998 2024-08-19T14:47:02.647 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:02.648 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:47:02.648 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:02.648 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.648 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.648 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26911.0:168663 8.9 8.796356a9 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e775)", 2024-08-19T14:47:02.648 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:42:35.191873+0000", 2024-08-19T14:47:02.648 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 267.36952867799999, 2024-08-19T14:47:02.648 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.93212851900000004, 2024-08-19T14:47:02.648 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:47:02.648 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:47:02.648 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:47:02.648 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:47:02.648 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26911", 2024-08-19T14:47:02.648 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3494993995", 2024-08-19T14:47:02.648 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 168663 2024-08-19T14:47:02.649 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.649 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:47:02.649 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.649 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:47:02.649 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.191873+0000", 2024-08-19T14:47:02.649 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:02.649 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.649 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.649 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:47:02.649 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.191873+0000", 2024-08-19T14:47:02.649 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:02.649 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.649 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.649 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:47:02.650 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.191874+0000", 2024-08-19T14:47:02.650 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.093e-06 2024-08-19T14:47:02.650 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.650 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.650 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:47:02.650 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.191879+0000", 2024-08-19T14:47:02.650 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3869999999999997e-06 2024-08-19T14:47:02.650 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.650 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.650 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:47:02.650 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.191890+0000", 2024-08-19T14:47:02.650 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1591e-05 2024-08-19T14:47:02.650 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.650 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.650 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:47:02.651 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.191898+0000", 2024-08-19T14:47:02.651 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.9719999999999992e-06 2024-08-19T14:47:02.651 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.651 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.651 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:47:02.651 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.191944+0000", 2024-08-19T14:47:02.651 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5609999999999999e-05 2024-08-19T14:47:02.651 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.651 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.651 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:47:02.651 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.191949+0000", 2024-08-19T14:47:02.651 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7360000000000001e-06 2024-08-19T14:47:02.651 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.651 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.652 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:47:02.652 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:36.123993+0000", 2024-08-19T14:47:02.652 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.93204411200000004 2024-08-19T14:47:02.652 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.652 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.652 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:47:02.652 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:36.124002+0000", 2024-08-19T14:47:02.652 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.0180000000000007e-06 2024-08-19T14:47:02.652 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:02.652 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:47:02.652 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:02.652 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.652 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.652 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26911.0:168671 8.14 8.e5a7b0d4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e775)", 2024-08-19T14:47:02.652 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:42:35.200344+0000", 2024-08-19T14:47:02.653 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 267.36105781399999, 2024-08-19T14:47:02.653 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.92393055800000001, 2024-08-19T14:47:02.653 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:47:02.653 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:47:02.653 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:47:02.653 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:47:02.653 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26911", 2024-08-19T14:47:02.653 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3494993995", 2024-08-19T14:47:02.653 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 168671 2024-08-19T14:47:02.653 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.653 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:47:02.653 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.653 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:47:02.653 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.200344+0000", 2024-08-19T14:47:02.653 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:02.654 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.654 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.654 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:47:02.654 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.200344+0000", 2024-08-19T14:47:02.654 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:02.654 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.654 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.654 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:47:02.654 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.200345+0000", 2024-08-19T14:47:02.654 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5000000000000002e-07 2024-08-19T14:47:02.654 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.654 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.654 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:47:02.654 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.200349+0000", 2024-08-19T14:47:02.655 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0670000000000002e-06 2024-08-19T14:47:02.655 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.655 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.655 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:47:02.655 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.200359+0000", 2024-08-19T14:47:02.655 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.045e-05 2024-08-19T14:47:02.655 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.655 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.655 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:47:02.655 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.200365+0000", 2024-08-19T14:47:02.655 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5589999999999996e-06 2024-08-19T14:47:02.655 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.655 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.655 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:47:02.655 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.200412+0000", 2024-08-19T14:47:02.656 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7389999999999999e-05 2024-08-19T14:47:02.656 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.656 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.656 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:47:02.656 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.200419+0000", 2024-08-19T14:47:02.656 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6719999999999998e-06 2024-08-19T14:47:02.656 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.656 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.656 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:47:02.656 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:36.124267+0000", 2024-08-19T14:47:02.656 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.92384827300000005 2024-08-19T14:47:02.656 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.656 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.657 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:47:02.657 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:36.124275+0000", 2024-08-19T14:47:02.657 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3969999999999997e-06 2024-08-19T14:47:02.657 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:02.657 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:47:02.657 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:02.657 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.657 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.657 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26911.0:168683 8.e 8.af5b1b1e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e775)", 2024-08-19T14:47:02.657 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:42:35.216428+0000", 2024-08-19T14:47:02.657 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 267.34497398500002, 2024-08-19T14:47:02.657 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.90856147600000003, 2024-08-19T14:47:02.657 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:47:02.657 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:47:02.657 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:47:02.658 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:47:02.658 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26911", 2024-08-19T14:47:02.658 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3494993995", 2024-08-19T14:47:02.658 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 168683 2024-08-19T14:47:02.658 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.658 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:47:02.658 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.658 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:47:02.658 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.216428+0000", 2024-08-19T14:47:02.658 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:02.658 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.658 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.658 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:47:02.658 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.216428+0000", 2024-08-19T14:47:02.659 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:02.659 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.659 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.659 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:47:02.659 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.216429+0000", 2024-08-19T14:47:02.659 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.049e-06 2024-08-19T14:47:02.659 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.659 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.659 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:47:02.659 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.216433+0000", 2024-08-19T14:47:02.659 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3970000000000004e-06 2024-08-19T14:47:02.659 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.659 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.659 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:47:02.659 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.216444+0000", 2024-08-19T14:47:02.660 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0885e-05 2024-08-19T14:47:02.660 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.660 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.660 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:47:02.660 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.216449+0000", 2024-08-19T14:47:02.660 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1270000000000002e-06 2024-08-19T14:47:02.660 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.660 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.660 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:47:02.660 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.216498+0000", 2024-08-19T14:47:02.660 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8408999999999997e-05 2024-08-19T14:47:02.660 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.660 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.660 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:47:02.661 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.216505+0000", 2024-08-19T14:47:02.661 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4139999999999997e-06 2024-08-19T14:47:02.661 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.661 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.661 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:47:02.661 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:36.124969+0000", 2024-08-19T14:47:02.661 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.90846388 2024-08-19T14:47:02.661 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.661 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.661 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:47:02.661 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:36.124989+0000", 2024-08-19T14:47:02.661 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0315000000000001e-05 2024-08-19T14:47:02.661 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:02.661 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:47:02.661 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:02.662 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.662 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.662 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26911.0:168691 8.14 8.265f4994 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e775)", 2024-08-19T14:47:02.662 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:42:35.248197+0000", 2024-08-19T14:47:02.662 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 267.31320542399999, 2024-08-19T14:47:02.662 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.87613702699999996, 2024-08-19T14:47:02.662 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:47:02.662 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:47:02.662 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:47:02.662 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:47:02.662 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26911", 2024-08-19T14:47:02.662 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3494993995", 2024-08-19T14:47:02.662 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 168691 2024-08-19T14:47:02.662 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.663 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:47:02.663 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.663 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:47:02.663 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.248197+0000", 2024-08-19T14:47:02.663 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:02.663 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.663 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.663 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:47:02.663 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.248197+0000", 2024-08-19T14:47:02.663 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:02.663 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.663 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.663 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:47:02.663 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.248198+0000", 2024-08-19T14:47:02.664 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.217e-06 2024-08-19T14:47:02.664 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.664 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.664 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:47:02.664 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.248203+0000", 2024-08-19T14:47:02.664 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9339999999999997e-06 2024-08-19T14:47:02.664 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.664 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.664 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:47:02.664 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.248215+0000", 2024-08-19T14:47:02.664 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1878999999999999e-05 2024-08-19T14:47:02.664 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.664 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.664 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:47:02.664 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.248220+0000", 2024-08-19T14:47:02.665 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0420000000000002e-06 2024-08-19T14:47:02.665 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.665 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.665 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:47:02.665 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.248267+0000", 2024-08-19T14:47:02.665 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7797999999999999e-05 2024-08-19T14:47:02.665 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.665 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.665 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:47:02.665 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.248276+0000", 2024-08-19T14:47:02.665 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6659999999999997e-06 2024-08-19T14:47:02.665 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.665 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.665 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:47:02.666 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:36.124324+0000", 2024-08-19T14:47:02.666 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.87604772500000005 2024-08-19T14:47:02.666 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.666 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.666 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:47:02.666 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:36.124334+0000", 2024-08-19T14:47:02.666 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.7659999999999993e-06 2024-08-19T14:47:02.666 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:02.666 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:47:02.666 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:02.666 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.666 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.666 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26911.0:168683 8.e 8:78d8daf5:::benchmark_data_smithi045_137103_object56219:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e777)", 2024-08-19T14:47:02.666 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:42:37.168895+0000", 2024-08-19T14:47:02.666 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 265.39250657999997, 2024-08-19T14:47:02.667 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.255438844, 2024-08-19T14:47:02.667 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:47:02.667 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:47:02.667 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:47:02.667 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:47:02.667 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26911", 2024-08-19T14:47:02.667 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3494993995", 2024-08-19T14:47:02.667 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 168683 2024-08-19T14:47:02.667 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.667 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:47:02.667 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.667 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:47:02.667 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.168895+0000", 2024-08-19T14:47:02.667 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:02.668 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.668 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.668 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:47:02.668 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.168895+0000", 2024-08-19T14:47:02.668 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:02.668 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.668 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.668 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:47:02.668 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.168896+0000", 2024-08-19T14:47:02.668 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7800000000000002e-07 2024-08-19T14:47:02.668 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.668 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.668 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:47:02.668 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.168899+0000", 2024-08-19T14:47:02.668 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1939999999999998e-06 2024-08-19T14:47:02.669 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.669 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.669 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:47:02.669 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.168906+0000", 2024-08-19T14:47:02.669 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.489e-06 2024-08-19T14:47:02.669 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.669 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.669 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:47:02.669 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.168911+0000", 2024-08-19T14:47:02.669 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6210000000000001e-06 2024-08-19T14:47:02.669 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.669 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.669 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:47:02.669 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.169357+0000", 2024-08-19T14:47:02.669 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00044563799999999999 2024-08-19T14:47:02.670 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.670 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.670 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:47:02.670 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.169366+0000", 2024-08-19T14:47:02.670 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.1710000000000008e-06 2024-08-19T14:47:02.670 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.670 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.670 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:47:02.670 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.407153+0000", 2024-08-19T14:47:02.670 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.237787208 2024-08-19T14:47:02.670 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.670 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.670 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:47:02.671 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.407255+0000", 2024-08-19T14:47:02.671 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00010173199999999999 2024-08-19T14:47:02.671 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.671 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.671 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0,2,7", 2024-08-19T14:47:02.671 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.407369+0000", 2024-08-19T14:47:02.671 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00011373499999999999 2024-08-19T14:47:02.671 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.671 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.671 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:47:02.671 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.423128+0000", 2024-08-19T14:47:02.671 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.015759009000000001 2024-08-19T14:47:02.671 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.671 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.671 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:47:02.672 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.423387+0000", 2024-08-19T14:47:02.672 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00025941499999999998 2024-08-19T14:47:02.672 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.672 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.672 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:47:02.672 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.423401+0000", 2024-08-19T14:47:02.672 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3991999999999999e-05 2024-08-19T14:47:02.672 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.672 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.672 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:47:02.672 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.424168+0000", 2024-08-19T14:47:02.672 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00076712700000000004 2024-08-19T14:47:02.672 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.672 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.672 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:47:02.673 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.424228+0000", 2024-08-19T14:47:02.673 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9759000000000002e-05 2024-08-19T14:47:02.673 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.673 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.673 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:47:02.673 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:47:02.674 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:02 smithi031 ceph-mon[16336]: 1.c scrub starts 2024-08-19T14:47:02.675 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:02 smithi031 ceph-mon[16336]: 1.c scrub ok 2024-08-19T14:47:02.675 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:02 smithi031 ceph-mon[16336]: 9.5 scrub ok 2024-08-19T14:47:02.675 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:02 smithi031 ceph-mon[16336]: 9.b scrub starts 2024-08-19T14:47:02.675 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:02 smithi031 ceph-mon[16336]: osdmap e896: 12 total, 12 up, 8 in 2024-08-19T14:47:02.675 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:02 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/49993055' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:47:02.675 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.424334+0000", 2024-08-19T14:47:02.675 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000106276 2024-08-19T14:47:02.675 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:02.675 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:47:02.676 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:02.676 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.676 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.676 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26911.0:168627 8.14 8:29970935:::benchmark_data_smithi045_137103_object56163:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e777)", 2024-08-19T14:47:02.676 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:42:37.173239+0000", 2024-08-19T14:47:02.676 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 265.38816258200001, 2024-08-19T14:47:02.676 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.23880800299999999, 2024-08-19T14:47:02.676 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:47:02.676 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:47:02.676 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:47:02.676 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:47:02.676 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26911", 2024-08-19T14:47:02.676 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3494993995", 2024-08-19T14:47:02.676 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 168627 2024-08-19T14:47:02.677 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.677 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:47:02.677 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.677 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:47:02.677 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.173239+0000", 2024-08-19T14:47:02.677 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:02.677 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.677 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.677 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:47:02.677 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.173239+0000", 2024-08-19T14:47:02.677 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:02.677 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.677 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.678 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:47:02.678 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.173241+0000", 2024-08-19T14:47:02.678 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.322e-06 2024-08-19T14:47:02.678 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.678 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.678 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:47:02.678 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.173249+0000", 2024-08-19T14:47:02.678 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.8909999999999995e-06 2024-08-19T14:47:02.678 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.678 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.678 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:47:02.678 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.173262+0000", 2024-08-19T14:47:02.678 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3178e-05 2024-08-19T14:47:02.678 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.679 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.679 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:47:02.679 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.173269+0000", 2024-08-19T14:47:02.679 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1300000000000003e-06 2024-08-19T14:47:02.679 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.679 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.679 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:47:02.679 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.174228+0000", 2024-08-19T14:47:02.679 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00095882200000000002 2024-08-19T14:47:02.679 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.679 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.679 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:47:02.679 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.174238+0000", 2024-08-19T14:47:02.679 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0624e-05 2024-08-19T14:47:02.680 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.680 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.680 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:47:02.680 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.386217+0000", 2024-08-19T14:47:02.680 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.21197906899999999 2024-08-19T14:47:02.680 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.680 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.680 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:47:02.680 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.386363+0000", 2024-08-19T14:47:02.680 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014560699999999999 2024-08-19T14:47:02.680 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.680 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.680 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,7", 2024-08-19T14:47:02.680 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.386513+0000", 2024-08-19T14:47:02.680 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000149572 2024-08-19T14:47:02.681 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.681 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.681 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:47:02.681 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.395345+0000", 2024-08-19T14:47:02.681 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0088323470000000008 2024-08-19T14:47:02.681 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.681 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.681 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:47:02.681 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.395394+0000", 2024-08-19T14:47:02.681 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8973e-05 2024-08-19T14:47:02.681 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.681 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.681 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:47:02.681 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.410261+0000", 2024-08-19T14:47:02.681 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.014867293 2024-08-19T14:47:02.682 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.682 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.682 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:47:02.682 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.410322+0000", 2024-08-19T14:47:02.682 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1227000000000002e-05 2024-08-19T14:47:02.682 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.682 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.682 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:47:02.682 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.412047+0000", 2024-08-19T14:47:02.682 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001724948 2024-08-19T14:47:02.682 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:02.682 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:47:02.682 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:02.683 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.683 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.683 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26911.0:168640 8.14 8:2d574969:::benchmark_data_smithi045_137103_object56176:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e777)", 2024-08-19T14:47:02.683 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:42:37.173288+0000", 2024-08-19T14:47:02.683 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 265.38811441899998, 2024-08-19T14:47:02.683 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.24640339999999999, 2024-08-19T14:47:02.683 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:47:02.683 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:47:02.683 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:47:02.683 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:47:02.683 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26911", 2024-08-19T14:47:02.683 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3494993995", 2024-08-19T14:47:02.683 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 168640 2024-08-19T14:47:02.683 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.683 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:47:02.684 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.684 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:47:02.684 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.173288+0000", 2024-08-19T14:47:02.684 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:02.684 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.684 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.684 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:47:02.684 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.173288+0000", 2024-08-19T14:47:02.684 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:02.684 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.684 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.684 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:47:02.684 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.173288+0000", 2024-08-19T14:47:02.684 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.89e-07 2024-08-19T14:47:02.684 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.685 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.685 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:47:02.685 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.173291+0000", 2024-08-19T14:47:02.685 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.667e-06 2024-08-19T14:47:02.685 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.685 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.685 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:47:02.685 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.173297+0000", 2024-08-19T14:47:02.685 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6290000000000004e-06 2024-08-19T14:47:02.685 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.685 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.685 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:47:02.685 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.173301+0000", 2024-08-19T14:47:02.685 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8759999999999998e-06 2024-08-19T14:47:02.685 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.686 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.686 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:47:02.686 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.174287+0000", 2024-08-19T14:47:02.686 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00098535300000000001 2024-08-19T14:47:02.686 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.686 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.686 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:47:02.686 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.174297+0000", 2024-08-19T14:47:02.686 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0189e-05 2024-08-19T14:47:02.686 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.686 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.686 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:47:02.686 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.392184+0000", 2024-08-19T14:47:02.687 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.217886845 2024-08-19T14:47:02.687 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.687 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.687 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:47:02.687 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.392339+0000", 2024-08-19T14:47:02.687 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000155501 2024-08-19T14:47:02.687 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.687 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.687 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,7", 2024-08-19T14:47:02.687 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.392484+0000", 2024-08-19T14:47:02.687 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014522299999999999 2024-08-19T14:47:02.687 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.687 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.688 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:47:02.688 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.395438+0000", 2024-08-19T14:47:02.688 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0029532009999999999 2024-08-19T14:47:02.688 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.688 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.688 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:47:02.688 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.412059+0000", 2024-08-19T14:47:02.688 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.016621350999999999 2024-08-19T14:47:02.688 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.688 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.688 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:47:02.688 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.417958+0000", 2024-08-19T14:47:02.688 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0058995540000000004 2024-08-19T14:47:02.688 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.689 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.689 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:47:02.689 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.418029+0000", 2024-08-19T14:47:02.689 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0963999999999993e-05 2024-08-19T14:47:02.689 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.689 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.689 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:47:02.689 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.419691+0000", 2024-08-19T14:47:02.689 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001661558 2024-08-19T14:47:02.689 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:02.689 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:47:02.689 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:02.689 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.690 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.690 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26911.0:168671 8.14 8:2b0de5a7:::benchmark_data_smithi045_137103_object56207:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e777)", 2024-08-19T14:47:02.690 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:42:37.173306+0000", 2024-08-19T14:47:02.690 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 265.38809551000003, 2024-08-19T14:47:02.690 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.24790071299999999, 2024-08-19T14:47:02.690 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:47:02.690 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:47:02.690 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:47:02.690 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:47:02.690 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26911", 2024-08-19T14:47:02.690 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3494993995", 2024-08-19T14:47:02.690 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 168671 2024-08-19T14:47:02.690 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.691 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:47:02.691 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.691 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:47:02.691 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.173306+0000", 2024-08-19T14:47:02.691 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:02.691 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.691 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.691 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:47:02.691 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.173306+0000", 2024-08-19T14:47:02.691 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:02.691 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.691 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.691 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:47:02.691 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.173307+0000", 2024-08-19T14:47:02.691 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5699999999999999e-07 2024-08-19T14:47:02.692 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.692 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.692 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:47:02.692 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.173308+0000", 2024-08-19T14:47:02.692 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7129999999999999e-06 2024-08-19T14:47:02.692 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.692 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.692 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:47:02.692 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.173312+0000", 2024-08-19T14:47:02.692 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.557e-06 2024-08-19T14:47:02.692 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.692 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.692 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:47:02.692 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.173315+0000", 2024-08-19T14:47:02.693 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.4400000000000001e-06 2024-08-19T14:47:02.693 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.693 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.693 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:47:02.693 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.174345+0000", 2024-08-19T14:47:02.693 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001029805 2024-08-19T14:47:02.693 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.693 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.693 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:47:02.693 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.174354+0000", 2024-08-19T14:47:02.693 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.2910000000000007e-06 2024-08-19T14:47:02.693 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.693 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.693 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:47:02.694 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.395466+0000", 2024-08-19T14:47:02.694 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.221112105 2024-08-19T14:47:02.694 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.694 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.694 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:47:02.694 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.395592+0000", 2024-08-19T14:47:02.694 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012608000000000001 2024-08-19T14:47:02.694 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.694 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.694 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,7", 2024-08-19T14:47:02.694 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.395719+0000", 2024-08-19T14:47:02.694 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012759499999999999 2024-08-19T14:47:02.694 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.694 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.695 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:47:02.695 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.405349+0000", 2024-08-19T14:47:02.695 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0096295200000000008 2024-08-19T14:47:02.695 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.695 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.695 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:47:02.695 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.412065+0000", 2024-08-19T14:47:02.695 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0067166689999999998 2024-08-19T14:47:02.695 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.695 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.695 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:47:02.695 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.419744+0000", 2024-08-19T14:47:02.695 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0076781210000000004 2024-08-19T14:47:02.695 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.696 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.696 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:47:02.696 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.419801+0000", 2024-08-19T14:47:02.696 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7268999999999997e-05 2024-08-19T14:47:02.696 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.696 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.696 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:47:02.696 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.421207+0000", 2024-08-19T14:47:02.696 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0014062910000000001 2024-08-19T14:47:02.696 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:02.696 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:47:02.696 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:02.696 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.696 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.697 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26911.0:168691 8.14 8:2992fa64:::benchmark_data_smithi045_137103_object56227:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e777)", 2024-08-19T14:47:02.697 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:42:37.173322+0000", 2024-08-19T14:47:02.697 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 265.38808018399999, 2024-08-19T14:47:02.697 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.25123347499999998, 2024-08-19T14:47:02.697 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:47:02.697 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:47:02.697 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:47:02.697 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:47:02.697 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26911", 2024-08-19T14:47:02.697 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3494993995", 2024-08-19T14:47:02.697 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 168691 2024-08-19T14:47:02.697 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.697 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:47:02.697 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.698 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:47:02.698 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.173322+0000", 2024-08-19T14:47:02.698 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:02.698 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.698 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.698 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:47:02.698 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.173322+0000", 2024-08-19T14:47:02.698 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:02.698 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.698 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.698 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:47:02.698 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.173322+0000", 2024-08-19T14:47:02.698 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3300000000000001e-07 2024-08-19T14:47:02.699 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.699 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.699 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:47:02.699 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.173323+0000", 2024-08-19T14:47:02.699 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4899999999999999e-06 2024-08-19T14:47:02.699 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.699 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.699 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:47:02.699 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.173328+0000", 2024-08-19T14:47:02.699 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9019999999999998e-06 2024-08-19T14:47:02.699 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.699 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.699 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:47:02.699 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.173331+0000", 2024-08-19T14:47:02.700 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9059999999999998e-06 2024-08-19T14:47:02.700 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.700 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.700 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:47:02.700 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.174427+0000", 2024-08-19T14:47:02.700 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001095469 2024-08-19T14:47:02.700 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.700 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.700 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:47:02.700 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.174435+0000", 2024-08-19T14:47:02.700 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.2770000000000005e-06 2024-08-19T14:47:02.700 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.700 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.700 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:47:02.701 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.398537+0000", 2024-08-19T14:47:02.701 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.224102251 2024-08-19T14:47:02.701 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.701 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.701 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:47:02.701 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.398694+0000", 2024-08-19T14:47:02.701 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00015705100000000001 2024-08-19T14:47:02.701 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.701 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.701 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,7", 2024-08-19T14:47:02.701 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.398835+0000", 2024-08-19T14:47:02.701 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014098800000000001 2024-08-19T14:47:02.701 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.701 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.701 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:47:02.702 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.405424+0000", 2024-08-19T14:47:02.702 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0065890530000000001 2024-08-19T14:47:02.702 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.702 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.702 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:47:02.702 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.412074+0000", 2024-08-19T14:47:02.702 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0066499669999999997 2024-08-19T14:47:02.702 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.702 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.702 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:47:02.702 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.422567+0000", 2024-08-19T14:47:02.702 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010492873999999999 2024-08-19T14:47:02.702 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.703 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.703 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:47:02.703 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.422667+0000", 2024-08-19T14:47:02.703 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.9445999999999996e-05 2024-08-19T14:47:02.703 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.703 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.703 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:47:02.703 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.424555+0000", 2024-08-19T14:47:02.703 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001888568 2024-08-19T14:47:02.703 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:02.703 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:47:02.703 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:02.703 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.704 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.704 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.27235.0:110529 9.c 9:3dcbb5c0:::benchmark_data_smithi045_152862_object16024:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e874)", 2024-08-19T14:47:02.704 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:46:10.747280+0000", 2024-08-19T14:47:02.704 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 51.814122275999999, 2024-08-19T14:47:02.704 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.248260699, 2024-08-19T14:47:02.704 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:47:02.704 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:47:02.704 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:47:02.704 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:47:02.704 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.27235", 2024-08-19T14:47:02.704 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/2746512116", 2024-08-19T14:47:02.704 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 110529 2024-08-19T14:47:02.704 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.704 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:47:02.705 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.705 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:47:02.705 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:46:10.747280+0000", 2024-08-19T14:47:02.705 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:02.705 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.705 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.705 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:47:02.705 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:46:10.747280+0000", 2024-08-19T14:47:02.705 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:02.705 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.705 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.705 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:47:02.705 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:46:10.747281+0000", 2024-08-19T14:47:02.705 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.9700000000000005e-07 2024-08-19T14:47:02.706 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.706 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.706 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:47:02.706 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:46:10.747286+0000", 2024-08-19T14:47:02.706 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9219999999999996e-06 2024-08-19T14:47:02.706 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.706 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.706 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:47:02.706 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:46:10.747296+0000", 2024-08-19T14:47:02.706 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0366999999999999e-05 2024-08-19T14:47:02.706 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.706 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.706 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:47:02.707 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:46:10.747301+0000", 2024-08-19T14:47:02.707 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.091e-06 2024-08-19T14:47:02.707 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.707 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.707 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:47:02.707 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:46:10.747339+0000", 2024-08-19T14:47:02.707 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.824e-05 2024-08-19T14:47:02.707 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.707 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.707 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:47:02.707 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:46:10.747423+0000", 2024-08-19T14:47:02.707 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.4126000000000001e-05 2024-08-19T14:47:02.707 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.707 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.707 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 9,10", 2024-08-19T14:47:02.708 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:46:10.747494+0000", 2024-08-19T14:47:02.708 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0203999999999994e-05 2024-08-19T14:47:02.708 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.708 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.708 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:47:02.708 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:46:10.750790+0000", 2024-08-19T14:47:02.708 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0032967539999999998 2024-08-19T14:47:02.708 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.708 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.708 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:47:02.708 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:46:10.752310+0000", 2024-08-19T14:47:02.708 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0015201049999999999 2024-08-19T14:47:02.708 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.708 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.709 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:47:02.709 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:46:10.995468+0000", 2024-08-19T14:47:02.709 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.24315711300000001 2024-08-19T14:47:02.709 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.709 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.709 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:47:02.709 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:46:10.995509+0000", 2024-08-19T14:47:02.709 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1569000000000001e-05 2024-08-19T14:47:02.709 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:02.709 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:02.709 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:47:02.709 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:46:10.995540+0000", 2024-08-19T14:47:02.710 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1310999999999999e-05 2024-08-19T14:47:02.710 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:02.710 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:47:02.710 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:02.710 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:02.710 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:47:02.710 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:47:02.716 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:02.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:02.717 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:47:02.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:02 smithi032 ceph-mon[20004]: 1.c scrub starts 2024-08-19T14:47:02.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:02 smithi032 ceph-mon[20004]: 1.c scrub ok 2024-08-19T14:47:02.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:02 smithi032 ceph-mon[20004]: 9.5 scrub ok 2024-08-19T14:47:02.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:02 smithi032 ceph-mon[20004]: 9.b scrub starts 2024-08-19T14:47:02.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:02 smithi032 ceph-mon[20004]: osdmap e896: 12 total, 12 up, 8 in 2024-08-19T14:47:02.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:02 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/49993055' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:47:02.769 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-19T14:47:02.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:02 smithi045 ceph-mon[20143]: 1.c scrub starts 2024-08-19T14:47:02.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:02 smithi045 ceph-mon[20143]: 1.c scrub ok 2024-08-19T14:47:02.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:02 smithi045 ceph-mon[20143]: 9.5 scrub ok 2024-08-19T14:47:02.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:02 smithi045 ceph-mon[20143]: 9.b scrub starts 2024-08-19T14:47:02.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:02 smithi045 ceph-mon[20143]: osdmap e896: 12 total, 12 up, 8 in 2024-08-19T14:47:02.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:02 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/49993055' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:47:02.818 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:47:02.851 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:02.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:02.852 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:47:02.953 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:47:02.993 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:02.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:02.994 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:47:03.092 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:47:03.092 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-08-19T14:47:03.095 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:47:03.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:03.141 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:03.141 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:47:03.242 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:47:03.265 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:03.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:03.266 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:47:03.367 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:47:03.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:03.412 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:03.412 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:47:03.513 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:47:03.537 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:03.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:03.538 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:47:03.571 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-08-19T14:47:03.639 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:47:03.674 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:03.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:03.676 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:47:03.776 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:47:03.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:03 smithi045 ceph-mon[20143]: 9.e scrub starts 2024-08-19T14:47:03.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:03 smithi045 ceph-mon[20143]: 9.b scrub ok 2024-08-19T14:47:03.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:03 smithi045 ceph-mon[20143]: pgmap v2110: 47 pgs: 1 peering, 46 active+clean; 2.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 614 op/s; 4.3 MiB/s, 68 objects/s recovering 2024-08-19T14:47:03.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:03 smithi045 ceph-mon[20143]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-08-19T14:47:03.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:03 smithi045 ceph-mon[20143]: Cluster is now healthy 2024-08-19T14:47:03.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:03 smithi045 ceph-mon[20143]: 9.d scrub starts 2024-08-19T14:47:03.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:03 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:47:03.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:03 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:47:03.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:03 smithi045 ceph-mon[20143]: from='client.18123 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:47:03.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {auth=37,config=2,health=12,kv=49} crc {auth=1377161207,config=3457680788,health=3897446751,kv=712890808}) 2024-08-19T14:47:03.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {kv=52,logm=48} crc {kv=317465771,logm=1936600621}) 2024-08-19T14:47:03.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3045089614}) 2024-08-19T14:47:03.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2936916937}) 2024-08-19T14:47:03.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=397173111}) 2024-08-19T14:47:03.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4258523784}) 2024-08-19T14:47:03.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=843530821}) 2024-08-19T14:47:03.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=142205573}) 2024-08-19T14:47:03.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=445330544}) 2024-08-19T14:47:03.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3791367746}) 2024-08-19T14:47:03.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3778345534}) 2024-08-19T14:47:03.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=375354755}) 2024-08-19T14:47:03.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1371943234}) 2024-08-19T14:47:03.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3686152039}) 2024-08-19T14:47:03.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2380633542}) 2024-08-19T14:47:03.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2137648524}) 2024-08-19T14:47:03.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1709107726}) 2024-08-19T14:47:03.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2380748669}) 2024-08-19T14:47:03.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2708757337}) 2024-08-19T14:47:03.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2438119731}) 2024-08-19T14:47:03.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2129420428}) 2024-08-19T14:47:03.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1247602590}) 2024-08-19T14:47:03.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3789971044}) 2024-08-19T14:47:03.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=855894702}) 2024-08-19T14:47:03.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4119951111}) 2024-08-19T14:47:03.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2107021522}) 2024-08-19T14:47:03.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1102446228}) 2024-08-19T14:47:03.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4200598762}) 2024-08-19T14:47:03.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=174837066}) 2024-08-19T14:47:03.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1002356016}) 2024-08-19T14:47:03.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=691449663}) 2024-08-19T14:47:03.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3306126708}) 2024-08-19T14:47:03.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=196313875}) 2024-08-19T14:47:03.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4189194897}) 2024-08-19T14:47:03.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=204271178}) 2024-08-19T14:47:03.800 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3833701630}) 2024-08-19T14:47:03.801 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=803348292}) 2024-08-19T14:47:03.801 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1174296533}) 2024-08-19T14:47:03.801 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=5475840}) 2024-08-19T14:47:03.801 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2207725445}) 2024-08-19T14:47:03.801 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4256395641}) 2024-08-19T14:47:03.801 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1100472328}) 2024-08-19T14:47:03.801 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=491617764}) 2024-08-19T14:47:03.801 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=278938213}) 2024-08-19T14:47:03.801 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1568626822}) 2024-08-19T14:47:03.801 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3791461525}) 2024-08-19T14:47:03.801 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1833739206}) 2024-08-19T14:47:03.801 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=383959061}) 2024-08-19T14:47:03.801 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=947824193}) 2024-08-19T14:47:03.801 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2888894165}) 2024-08-19T14:47:03.802 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:03 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2249112383}) 2024-08-19T14:47:03.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:03.803 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:03.803 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:47:03.854 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:47:03.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:03 smithi031 ceph-mon[16336]: 9.e scrub starts 2024-08-19T14:47:03.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:03 smithi031 ceph-mon[16336]: 9.b scrub ok 2024-08-19T14:47:03.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:03 smithi031 ceph-mon[16336]: pgmap v2110: 47 pgs: 1 peering, 46 active+clean; 2.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 614 op/s; 4.3 MiB/s, 68 objects/s recovering 2024-08-19T14:47:03.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:03 smithi031 ceph-mon[16336]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-08-19T14:47:03.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:03 smithi031 ceph-mon[16336]: Cluster is now healthy 2024-08-19T14:47:03.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:03 smithi031 ceph-mon[16336]: 9.d scrub starts 2024-08-19T14:47:03.868 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:03 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:47:03.868 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:03 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:47:03.868 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:03 smithi031 ceph-mon[16336]: from='client.18123 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:47:03.868 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {auth=37,config=2,health=12,kv=49} crc {auth=1377161207,config=3457680788,health=3897446751,kv=712890808}) 2024-08-19T14:47:03.868 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {kv=52,logm=48} crc {kv=317465771,logm=1936600621}) 2024-08-19T14:47:03.868 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3045089614}) 2024-08-19T14:47:03.868 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2936916937}) 2024-08-19T14:47:03.868 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=397173111}) 2024-08-19T14:47:03.868 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4258523784}) 2024-08-19T14:47:03.868 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=843530821}) 2024-08-19T14:47:03.868 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=142205573}) 2024-08-19T14:47:03.868 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=445330544}) 2024-08-19T14:47:03.868 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3791367746}) 2024-08-19T14:47:03.868 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3778345534}) 2024-08-19T14:47:03.869 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=375354755}) 2024-08-19T14:47:03.869 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1371943234}) 2024-08-19T14:47:03.869 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3686152039}) 2024-08-19T14:47:03.869 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2380633542}) 2024-08-19T14:47:03.869 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2137648524}) 2024-08-19T14:47:03.869 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1709107726}) 2024-08-19T14:47:03.869 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2380748669}) 2024-08-19T14:47:03.869 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2708757337}) 2024-08-19T14:47:03.869 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2438119731}) 2024-08-19T14:47:03.869 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2129420428}) 2024-08-19T14:47:03.869 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1247602590}) 2024-08-19T14:47:03.869 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3789971044}) 2024-08-19T14:47:03.870 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=855894702}) 2024-08-19T14:47:03.870 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4119951111}) 2024-08-19T14:47:03.870 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2107021522}) 2024-08-19T14:47:03.870 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1102446228}) 2024-08-19T14:47:03.870 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4200598762}) 2024-08-19T14:47:03.870 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=174837066}) 2024-08-19T14:47:03.870 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1002356016}) 2024-08-19T14:47:03.870 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=691449663}) 2024-08-19T14:47:03.870 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3306126708}) 2024-08-19T14:47:03.870 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=196313875}) 2024-08-19T14:47:03.870 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4189194897}) 2024-08-19T14:47:03.870 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=204271178}) 2024-08-19T14:47:03.870 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3833701630}) 2024-08-19T14:47:03.871 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=803348292}) 2024-08-19T14:47:03.871 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1174296533}) 2024-08-19T14:47:03.871 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=5475840}) 2024-08-19T14:47:03.871 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2207725445}) 2024-08-19T14:47:03.871 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4256395641}) 2024-08-19T14:47:03.871 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1100472328}) 2024-08-19T14:47:03.871 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=491617764}) 2024-08-19T14:47:03.871 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=278938213}) 2024-08-19T14:47:03.871 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1568626822}) 2024-08-19T14:47:03.871 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3791461525}) 2024-08-19T14:47:03.871 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1833739206}) 2024-08-19T14:47:03.872 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=383959061}) 2024-08-19T14:47:03.872 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=947824193}) 2024-08-19T14:47:03.872 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2888894165}) 2024-08-19T14:47:03.872 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:03 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2249112383}) 2024-08-19T14:47:03.904 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":2110,"stamp":"2024-08-19T14:47:02.392518+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2675835448,"num_objects":40824,"num_object_clones":0,"num_object_copies":122472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40824,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":148184,"num_write_kb":6048129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39696,"num_bytes_recovered":2602574912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152493,"ondisk_log_size":152493,"up":141,"acting":141,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":142,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":15457984,"kb_used_data":8294332,"kb_used_omap":17,"kb_used_meta":7163310,"kb_avail":734503232,"statfs":{"total":767960285184,"available":752131309568,"internally_reserved":0,"allocated":8493395968,"data_stored":8472051676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18388,"internal_metadata":7335229484},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-241500160,"num_objects":-3685,"num_object_clones":0,"num_object_copies":-11055,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-409,"num_objects_unfound":0,"num_objects_dirty":-3685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3685,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":409,"num_bytes_recovered":26804224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001462"},"pg_stats":[{"pgid":"9.8","version":"893'9212","reported_seq":13434,"reported_epoch":893,"state":"active+clean","last_fresh":"2024-08-19T14:46:57.737622+0000","last_change":"2024-08-19T14:45:07.176081+0000","last_active":"2024-08-19T14:46:57.737622+0000","last_peered":"2024-08-19T14:46:57.737622+0000","last_clean":"2024-08-19T14:46:57.737622+0000","last_became_active":"2024-08-19T14:45:07.171026+0000","last_became_peered":"2024-08-19T14:45:07.171026+0000","last_unstale":"2024-08-19T14:46:57.737622+0000","last_undegraded":"2024-08-19T14:46:57.737622+0000","last_fullsized":"2024-08-19T14:46:57.737622+0000","mapping_epoch":852,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":853,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":9212,"log_dups_size":0,"ondisk_log_size":9212,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":174063616,"num_objects":2656,"num_object_clones":0,"num_object_copies":7968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9212,"num_write_kb":379776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2033,"num_bytes_recovered":133234688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,1],"acting":[11,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"70'139","reported_seq":2132,"reported_epoch":893,"state":"active+clean","last_fresh":"2024-08-19T14:46:54.142921+0000","last_change":"2024-08-19T14:45:26.108478+0000","last_active":"2024-08-19T14:46:54.142921+0000","last_peered":"2024-08-19T14:46:54.142921+0000","last_clean":"2024-08-19T14:46:54.142921+0000","last_became_active":"2024-08-19T14:45:26.107697+0000","last_became_peered":"2024-08-19T14:45:26.107697+0000","last_unstale":"2024-08-19T14:46:54.142921+0000","last_undegraded":"2024-08-19T14:46:54.142921+0000","last_fullsized":"2024-08-19T14:46:54.142921+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:26.109206+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:41:45.177828+0000","last_clean_scrub_stamp":"2024-08-19T14:44:26.109206+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6,9],"acting":[11,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.b","version":"894'9249","reported_seq":13369,"reported_epoch":894,"state":"active+clean","last_fresh":"2024-08-19T14:47:00.259860+0000","last_change":"2024-08-19T14:45:09.096009+0000","last_active":"2024-08-19T14:47:00.259860+0000","last_peered":"2024-08-19T14:47:00.259860+0000","last_clean":"2024-08-19T14:47:00.259860+0000","last_became_active":"2024-08-19T14:45:09.091550+0000","last_became_peered":"2024-08-19T14:45:09.091550+0000","last_unstale":"2024-08-19T14:47:00.259860+0000","last_undegraded":"2024-08-19T14:47:00.259860+0000","last_fullsized":"2024-08-19T14:47:00.259860+0000","mapping_epoch":854,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":9249,"log_dups_size":0,"ondisk_log_size":9249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":161808384,"num_objects":2469,"num_object_clones":0,"num_object_copies":7407,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9249,"num_write_kb":374976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1981,"num_bytes_recovered":129826816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,1],"acting":[8,10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"70'139","reported_seq":1986,"reported_epoch":893,"state":"active+clean","last_fresh":"2024-08-19T14:46:54.112565+0000","last_change":"2024-08-19T14:45:26.101740+0000","last_active":"2024-08-19T14:46:54.112565+0000","last_peered":"2024-08-19T14:46:54.112565+0000","last_clean":"2024-08-19T14:46:54.112565+0000","last_became_active":"2024-08-19T14:45:26.101280+0000","last_became_peered":"2024-08-19T14:45:26.101280+0000","last_unstale":"2024-08-19T14:46:54.112565+0000","last_undegraded":"2024-08-19T14:46:54.112565+0000","last_fullsized":"2024-08-19T14:46:54.112565+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:45:20.599964+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:45:20.599964+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.a","version":"893'9329","reported_seq":22345,"reported_epoch":893,"state":"active+clean","last_fresh":"2024-08-19T14:46:59.038245+0000","last_change":"2024-08-19T14:46:58.252113+0000","last_active":"2024-08-19T14:46:59.038245+0000","last_peered":"2024-08-19T14:46:59.038245+0000","last_clean":"2024-08-19T14:46:59.038245+0000","last_became_active":"2024-08-19T14:46:35.321997+0000","last_became_peered":"2024-08-19T14:46:35.321997+0000","last_unstale":"2024-08-19T14:46:59.038245+0000","last_undegraded":"2024-08-19T14:46:59.038245+0000","last_fullsized":"2024-08-19T14:46:59.038245+0000","mapping_epoch":885,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":886,"parent":"0.0","parent_split_bits":0,"last_scrub":"893'9283","last_scrub_stamp":"2024-08-19T14:46:58.251971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:46:58.251971+0000","objects_scrubbed":2574,"log_size":9329,"log_dups_size":0,"ondisk_log_size":9329,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:48:10.683585+0000","scrub_duration":524,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":165216256,"num_objects":2521,"num_object_clones":0,"num_object_copies":7563,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2521,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9329,"num_write_kb":379200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6408,"num_bytes_recovered":419954688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"70'139","reported_seq":626,"reported_epoch":894,"state":"active+clean","last_fresh":"2024-08-19T14:46:59.502699+0000","last_change":"2024-08-19T14:46:07.848793+0000","last_active":"2024-08-19T14:46:59.502699+0000","last_peered":"2024-08-19T14:46:59.502699+0000","last_clean":"2024-08-19T14:46:59.502699+0000","last_became_active":"2024-08-19T14:43:11.470639+0000","last_became_peered":"2024-08-19T14:43:11.470639+0000","last_unstale":"2024-08-19T14:46:59.502699+0000","last_undegraded":"2024-08-19T14:46:59.502699+0000","last_fullsized":"2024-08-19T14:46:59.502699+0000","mapping_epoch":795,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":796,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:46:07.848681+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:19:59.302931+0000","last_clean_scrub_stamp":"2024-08-19T14:46:07.848681+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:47:19.399442+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.9","version":"894'9361","reported_seq":22426,"reported_epoch":894,"state":"peering","last_fresh":"2024-08-19T14:47:00.512658+0000","last_change":"2024-08-19T14:47:00.527495+0000","last_active":"2024-08-19T14:47:00.527495+0000","last_peered":"2024-08-19T14:47:00.512658+0000","last_clean":"2024-08-19T14:45:22.229561+0000","last_became_active":"2024-08-19T14:45:24.550442+0000","last_became_peered":"2024-08-19T14:45:24.550442+0000","last_unstale":"2024-08-19T14:47:00.512658+0000","last_undegraded":"2024-08-19T14:47:00.512658+0000","last_fullsized":"2024-08-19T14:45:24.094569+0000","mapping_epoch":895,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":0,"last_scrub":"860'4221","last_scrub_stamp":"2024-08-19T14:45:19.039376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:45:19.039376+0000","objects_scrubbed":4147,"log_size":9361,"log_dups_size":0,"ondisk_log_size":9361,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1783,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163512320,"num_objects":2495,"num_object_clones":0,"num_object_copies":7485,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2495,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9361,"num_write_kb":379392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6427,"num_bytes_recovered":421199872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,6],"acting":[10,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6,8,11],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1","version":"70'139","reported_seq":621,"reported_epoch":895,"state":"active+clean","last_fresh":"2024-08-19T14:47:00.527177+0000","last_change":"2024-08-19T14:44:32.095248+0000","last_active":"2024-08-19T14:47:00.527177+0000","last_peered":"2024-08-19T14:47:00.527177+0000","last_clean":"2024-08-19T14:47:00.527177+0000","last_became_active":"2024-08-19T14:43:16.098337+0000","last_became_peered":"2024-08-19T14:43:16.098337+0000","last_unstale":"2024-08-19T14:47:00.527177+0000","last_undegraded":"2024-08-19T14:47:00.527177+0000","last_fullsized":"2024-08-19T14:47:00.527177+0000","mapping_epoch":800,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":801,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:32.094375+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:44:32.094375+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,2],"acting":[10,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.f","version":"893'9335","reported_seq":13369,"reported_epoch":893,"state":"active+clean","last_fresh":"2024-08-19T14:46:58.999972+0000","last_change":"2024-08-19T14:45:01.147769+0000","last_active":"2024-08-19T14:46:58.999972+0000","last_peered":"2024-08-19T14:46:58.999972+0000","last_clean":"2024-08-19T14:46:58.999972+0000","last_became_active":"2024-08-19T14:45:01.143857+0000","last_became_peered":"2024-08-19T14:45:01.143857+0000","last_unstale":"2024-08-19T14:46:58.999972+0000","last_undegraded":"2024-08-19T14:46:58.999972+0000","last_fullsized":"2024-08-19T14:46:58.999972+0000","mapping_epoch":847,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":848,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":9335,"log_dups_size":0,"ondisk_log_size":9335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168624128,"num_objects":2573,"num_object_clones":0,"num_object_copies":7719,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2573,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9335,"num_write_kb":381056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1939,"num_bytes_recovered":127074304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,11],"acting":[1,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"70'139","reported_seq":1144,"reported_epoch":894,"state":"active+clean","last_fresh":"2024-08-19T14:46:59.507956+0000","last_change":"2024-08-19T14:45:26.107836+0000","last_active":"2024-08-19T14:46:59.507956+0000","last_peered":"2024-08-19T14:46:59.507956+0000","last_clean":"2024-08-19T14:46:59.507956+0000","last_became_active":"2024-08-19T14:45:26.107494+0000","last_became_peered":"2024-08-19T14:45:26.107494+0000","last_unstale":"2024-08-19T14:46:59.507956+0000","last_undegraded":"2024-08-19T14:46:59.507956+0000","last_fullsized":"2024-08-19T14:46:59.507956+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:26.113799+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:21:46.057624+0000","last_clean_scrub_stamp":"2024-08-19T14:44:26.113799+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,2],"acting":[6,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"895'9306","reported_seq":9455,"reported_epoch":895,"state":"active+clean","last_fresh":"2024-08-19T14:47:00.754110+0000","last_change":"2024-08-19T14:44:15.415042+0000","last_active":"2024-08-19T14:47:00.754110+0000","last_peered":"2024-08-19T14:47:00.754110+0000","last_clean":"2024-08-19T14:47:00.754110+0000","last_became_active":"2024-08-19T14:44:15.414595+0000","last_became_peered":"2024-08-19T14:44:15.414595+0000","last_unstale":"2024-08-19T14:47:00.754110+0000","last_undegraded":"2024-08-19T14:47:00.754110+0000","last_fullsized":"2024-08-19T14:47:00.754110+0000","mapping_epoch":826,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":827,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":9306,"log_dups_size":0,"ondisk_log_size":9306,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159252480,"num_objects":2430,"num_object_clones":0,"num_object_copies":7290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2430,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9306,"num_write_kb":375552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,9],"acting":[10,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.6","version":"70'139","reported_seq":487,"reported_epoch":895,"state":"active+clean","last_fresh":"2024-08-19T14:47:00.524549+0000","last_change":"2024-08-19T14:44:22.089843+0000","last_active":"2024-08-19T14:47:00.524549+0000","last_peered":"2024-08-19T14:47:00.524549+0000","last_clean":"2024-08-19T14:47:00.524549+0000","last_became_active":"2024-08-19T14:43:16.091256+0000","last_became_peered":"2024-08-19T14:43:16.091256+0000","last_unstale":"2024-08-19T14:47:00.524549+0000","last_undegraded":"2024-08-19T14:47:00.524549+0000","last_fullsized":"2024-08-19T14:47:00.524549+0000","mapping_epoch":800,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":801,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:22.089766+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:27.848870+0000","last_clean_scrub_stamp":"2024-08-19T14:44:22.089766+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.d","version":"893'9103","reported_seq":12997,"reported_epoch":893,"state":"active+clean","last_fresh":"2024-08-19T14:46:58.937078+0000","last_change":"2024-08-19T14:45:00.248322+0000","last_active":"2024-08-19T14:46:58.937078+0000","last_peered":"2024-08-19T14:46:58.937078+0000","last_clean":"2024-08-19T14:46:58.937078+0000","last_became_active":"2024-08-19T14:45:00.244759+0000","last_became_peered":"2024-08-19T14:45:00.244759+0000","last_unstale":"2024-08-19T14:46:58.937078+0000","last_undegraded":"2024-08-19T14:46:58.937078+0000","last_fullsized":"2024-08-19T14:46:58.937078+0000","mapping_epoch":846,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":847,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":9103,"log_dups_size":0,"ondisk_log_size":9103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171704344,"num_objects":2621,"num_object_clones":0,"num_object_copies":7863,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2621,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9103,"num_write_kb":375105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1868,"num_bytes_recovered":122421248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"70'139","reported_seq":586,"reported_epoch":895,"state":"active+clean","last_fresh":"2024-08-19T14:47:00.523821+0000","last_change":"2024-08-19T14:45:26.106248+0000","last_active":"2024-08-19T14:47:00.523821+0000","last_peered":"2024-08-19T14:47:00.523821+0000","last_clean":"2024-08-19T14:47:00.523821+0000","last_became_active":"2024-08-19T14:45:26.105833+0000","last_became_peered":"2024-08-19T14:45:26.105833+0000","last_unstale":"2024-08-19T14:47:00.523821+0000","last_undegraded":"2024-08-19T14:47:00.523821+0000","last_fullsized":"2024-08-19T14:47:00.523821+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:29.075645+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:41:48.878043+0000","last_clean_scrub_stamp":"2024-08-19T14:44:29.075645+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.c","version":"894'9297","reported_seq":9495,"reported_epoch":894,"state":"active+clean","last_fresh":"2024-08-19T14:46:59.851122+0000","last_change":"2024-08-19T14:45:22.398825+0000","last_active":"2024-08-19T14:46:59.851122+0000","last_peered":"2024-08-19T14:46:59.851122+0000","last_clean":"2024-08-19T14:46:59.851122+0000","last_became_active":"2024-08-19T14:44:15.416278+0000","last_became_peered":"2024-08-19T14:44:15.416278+0000","last_unstale":"2024-08-19T14:46:59.851122+0000","last_undegraded":"2024-08-19T14:46:59.851122+0000","last_fullsized":"2024-08-19T14:46:59.851122+0000","mapping_epoch":826,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":827,"parent":"0.0","parent_split_bits":0,"last_scrub":"862'4406","last_scrub_stamp":"2024-08-19T14:45:22.398707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:45:22.398707+0000","objects_scrubbed":4394,"log_size":9297,"log_dups_size":0,"ondisk_log_size":9297,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":618,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":165478400,"num_objects":2525,"num_object_clones":0,"num_object_copies":7575,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9297,"num_write_kb":378304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,10],"acting":[4,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"70'139","reported_seq":1052,"reported_epoch":893,"state":"active+clean","last_fresh":"2024-08-19T14:46:54.142937+0000","last_change":"2024-08-19T14:45:26.105952+0000","last_active":"2024-08-19T14:46:54.142937+0000","last_peered":"2024-08-19T14:46:54.142937+0000","last_clean":"2024-08-19T14:46:54.142937+0000","last_became_active":"2024-08-19T14:45:26.105653+0000","last_became_peered":"2024-08-19T14:45:26.105653+0000","last_unstale":"2024-08-19T14:46:54.142937+0000","last_undegraded":"2024-08-19T14:46:54.142937+0000","last_fullsized":"2024-08-19T14:46:54.142937+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:33.078526+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:41:55.868614+0000","last_clean_scrub_stamp":"2024-08-19T14:44:33.078526+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,6],"acting":[11,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.2","version":"893'9319","reported_seq":9464,"reported_epoch":893,"state":"active+clean","last_fresh":"2024-08-19T14:46:59.051036+0000","last_change":"2024-08-19T14:44:15.415441+0000","last_active":"2024-08-19T14:46:59.051036+0000","last_peered":"2024-08-19T14:46:59.051036+0000","last_clean":"2024-08-19T14:46:59.051036+0000","last_became_active":"2024-08-19T14:44:15.415137+0000","last_became_peered":"2024-08-19T14:44:15.415137+0000","last_unstale":"2024-08-19T14:46:59.051036+0000","last_undegraded":"2024-08-19T14:46:59.051036+0000","last_fullsized":"2024-08-19T14:46:59.051036+0000","mapping_epoch":826,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":827,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":9319,"log_dups_size":0,"ondisk_log_size":9319,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":172818432,"num_objects":2637,"num_object_clones":0,"num_object_copies":7911,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2637,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9319,"num_write_kb":382592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,11],"acting":[1,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"70'139","reported_seq":1025,"reported_epoch":894,"state":"active+clean","last_fresh":"2024-08-19T14:46:59.507607+0000","last_change":"2024-08-19T14:45:26.103609+0000","last_active":"2024-08-19T14:46:59.507607+0000","last_peered":"2024-08-19T14:46:59.507607+0000","last_clean":"2024-08-19T14:46:59.507607+0000","last_became_active":"2024-08-19T14:45:26.103142+0000","last_became_peered":"2024-08-19T14:45:26.103142+0000","last_unstale":"2024-08-19T14:46:59.507607+0000","last_undegraded":"2024-08-19T14:46:59.507607+0000","last_fullsized":"2024-08-19T14:46:59.507607+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:28.862033+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:44:28.862033+0000","last_clean_scrub_stamp":"2024-08-19T14:44:28.862033+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,4],"acting":[6,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1","version":"894'9351","reported_seq":18373,"reported_epoch":894,"state":"active+clean","last_fresh":"2024-08-19T14:46:59.613302+0000","last_change":"2024-08-19T14:46:18.212911+0000","last_active":"2024-08-19T14:46:59.613302+0000","last_peered":"2024-08-19T14:46:59.613302+0000","last_clean":"2024-08-19T14:46:59.613302+0000","last_became_active":"2024-08-19T14:46:18.202172+0000","last_became_peered":"2024-08-19T14:46:18.202172+0000","last_unstale":"2024-08-19T14:46:59.613302+0000","last_undegraded":"2024-08-19T14:46:59.613302+0000","last_fullsized":"2024-08-19T14:46:59.613302+0000","mapping_epoch":875,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":876,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":9351,"log_dups_size":0,"ondisk_log_size":9351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168755200,"num_objects":2575,"num_object_clones":0,"num_object_copies":7725,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2575,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9351,"num_write_kb":381632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4433,"num_bytes_recovered":290521088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,10],"acting":[6,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"70'139","reported_seq":1411,"reported_epoch":894,"state":"active+clean","last_fresh":"2024-08-19T14:46:59.507561+0000","last_change":"2024-08-19T14:45:26.126939+0000","last_active":"2024-08-19T14:46:59.507561+0000","last_peered":"2024-08-19T14:46:59.507561+0000","last_clean":"2024-08-19T14:46:59.507561+0000","last_became_active":"2024-08-19T14:45:26.126712+0000","last_became_peered":"2024-08-19T14:45:26.126712+0000","last_unstale":"2024-08-19T14:46:59.507561+0000","last_undegraded":"2024-08-19T14:46:59.507561+0000","last_fullsized":"2024-08-19T14:46:59.507561+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:24.958576+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:44:24.958576+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,2],"acting":[6,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.0","version":"893'9131","reported_seq":17953,"reported_epoch":893,"state":"active+clean","last_fresh":"2024-08-19T14:46:59.021163+0000","last_change":"2024-08-19T14:46:29.123185+0000","last_active":"2024-08-19T14:46:59.021163+0000","last_peered":"2024-08-19T14:46:59.021163+0000","last_clean":"2024-08-19T14:46:59.021163+0000","last_became_active":"2024-08-19T14:46:29.110655+0000","last_became_peered":"2024-08-19T14:46:29.110655+0000","last_unstale":"2024-08-19T14:46:59.021163+0000","last_undegraded":"2024-08-19T14:46:59.021163+0000","last_fullsized":"2024-08-19T14:46:59.021163+0000","mapping_epoch":881,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":882,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":9131,"log_dups_size":0,"ondisk_log_size":9131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":164954112,"num_objects":2517,"num_object_clones":0,"num_object_copies":7551,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9131,"num_write_kb":372736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4334,"num_bytes_recovered":284033024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,11],"acting":[1,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"70'139","reported_seq":728,"reported_epoch":894,"state":"active+clean","last_fresh":"2024-08-19T14:46:59.512716+0000","last_change":"2024-08-19T14:44:38.183741+0000","last_active":"2024-08-19T14:46:59.512716+0000","last_peered":"2024-08-19T14:46:59.512716+0000","last_clean":"2024-08-19T14:46:59.512716+0000","last_became_active":"2024-08-19T14:44:38.183219+0000","last_became_peered":"2024-08-19T14:44:38.183219+0000","last_unstale":"2024-08-19T14:46:59.512716+0000","last_undegraded":"2024-08-19T14:46:59.512716+0000","last_fullsized":"2024-08-19T14:46:59.512716+0000","mapping_epoch":837,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:27.791867+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:44:27.791867+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,4],"acting":[8,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.3","version":"894'9423","reported_seq":22422,"reported_epoch":894,"state":"active+clean","last_fresh":"2024-08-19T14:47:00.242313+0000","last_change":"2024-08-19T14:46:54.137652+0000","last_active":"2024-08-19T14:47:00.242313+0000","last_peered":"2024-08-19T14:47:00.242313+0000","last_clean":"2024-08-19T14:47:00.242313+0000","last_became_active":"2024-08-19T14:46:54.116048+0000","last_became_peered":"2024-08-19T14:46:54.116048+0000","last_unstale":"2024-08-19T14:47:00.242313+0000","last_undegraded":"2024-08-19T14:47:00.242313+0000","last_fullsized":"2024-08-19T14:47:00.242313+0000","mapping_epoch":892,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":893,"parent":"0.0","parent_split_bits":0,"last_scrub":"860'4176","last_scrub_stamp":"2024-08-19T14:45:18.741455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:45:18.741455+0000","objects_scrubbed":4146,"log_size":9423,"log_dups_size":0,"ondisk_log_size":9423,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1454,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167444480,"num_objects":2555,"num_object_clones":0,"num_object_copies":7665,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9423,"num_write_kb":383296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6391,"num_bytes_recovered":418840576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,6],"acting":[8,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.b","version":"70'139","reported_seq":1504,"reported_epoch":893,"state":"active+clean","last_fresh":"2024-08-19T14:46:54.143333+0000","last_change":"2024-08-19T14:44:38.183925+0000","last_active":"2024-08-19T14:46:54.143333+0000","last_peered":"2024-08-19T14:46:54.143333+0000","last_clean":"2024-08-19T14:46:54.143333+0000","last_became_active":"2024-08-19T14:44:38.183552+0000","last_became_peered":"2024-08-19T14:44:38.183552+0000","last_unstale":"2024-08-19T14:46:54.143333+0000","last_undegraded":"2024-08-19T14:46:54.143333+0000","last_fullsized":"2024-08-19T14:46:54.143333+0000","mapping_epoch":837,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:29.142655+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:44:29.142655+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,1],"acting":[11,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.6","version":"895'9339","reported_seq":13423,"reported_epoch":895,"state":"active+clean","last_fresh":"2024-08-19T14:47:00.744442+0000","last_change":"2024-08-19T14:45:08.093571+0000","last_active":"2024-08-19T14:47:00.744442+0000","last_peered":"2024-08-19T14:47:00.744442+0000","last_clean":"2024-08-19T14:47:00.744442+0000","last_became_active":"2024-08-19T14:45:08.088967+0000","last_became_peered":"2024-08-19T14:45:08.088967+0000","last_unstale":"2024-08-19T14:47:00.744442+0000","last_undegraded":"2024-08-19T14:47:00.744442+0000","last_fullsized":"2024-08-19T14:47:00.744442+0000","mapping_epoch":853,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":854,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":9339,"log_dups_size":0,"ondisk_log_size":9339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163643392,"num_objects":2497,"num_object_clones":0,"num_object_copies":7491,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2497,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9339,"num_write_kb":378752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1962,"num_bytes_recovered":128581632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,9],"acting":[10,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.e","version":"70'139","reported_seq":1704,"reported_epoch":894,"state":"active+clean","last_fresh":"2024-08-19T14:46:59.502803+0000","last_change":"2024-08-19T14:46:17.013622+0000","last_active":"2024-08-19T14:46:59.502803+0000","last_peered":"2024-08-19T14:46:59.502803+0000","last_clean":"2024-08-19T14:46:59.502803+0000","last_became_active":"2024-08-19T14:43:08.463102+0000","last_became_peered":"2024-08-19T14:43:08.463102+0000","last_unstale":"2024-08-19T14:46:59.502803+0000","last_undegraded":"2024-08-19T14:46:59.502803+0000","last_fullsized":"2024-08-19T14:46:59.502803+0000","mapping_epoch":793,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":794,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:46:17.013541+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:40:03.764672+0000","last_clean_scrub_stamp":"2024-08-19T14:46:17.013541+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:47:21.570706+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,1],"acting":[4,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.7","version":"895'9166","reported_seq":9315,"reported_epoch":895,"state":"active+clean","last_fresh":"2024-08-19T14:47:00.785443+0000","last_change":"2024-08-19T14:44:15.068003+0000","last_active":"2024-08-19T14:47:00.785443+0000","last_peered":"2024-08-19T14:47:00.785443+0000","last_clean":"2024-08-19T14:47:00.785443+0000","last_became_active":"2024-08-19T14:44:15.067635+0000","last_became_peered":"2024-08-19T14:44:15.067635+0000","last_unstale":"2024-08-19T14:47:00.785443+0000","last_undegraded":"2024-08-19T14:47:00.785443+0000","last_fullsized":"2024-08-19T14:47:00.785443+0000","mapping_epoch":826,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":827,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":9166,"log_dups_size":0,"ondisk_log_size":9166,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":164364288,"num_objects":2508,"num_object_clones":0,"num_object_copies":7524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9166,"num_write_kb":373568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,1],"acting":[10,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.f","version":"70'139","reported_seq":1390,"reported_epoch":894,"state":"active+clean","last_fresh":"2024-08-19T14:46:59.507848+0000","last_change":"2024-08-19T14:45:26.102407+0000","last_active":"2024-08-19T14:46:59.507848+0000","last_peered":"2024-08-19T14:46:59.507848+0000","last_clean":"2024-08-19T14:46:59.507848+0000","last_became_active":"2024-08-19T14:45:26.101929+0000","last_became_peered":"2024-08-19T14:45:26.101929+0000","last_unstale":"2024-08-19T14:46:59.507848+0000","last_undegraded":"2024-08-19T14:46:59.507848+0000","last_fullsized":"2024-08-19T14:46:59.507848+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:31.800766+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:31.800766+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,8],"acting":[6,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"893'9176","reported_seq":9321,"reported_epoch":893,"state":"active+clean","last_fresh":"2024-08-19T14:46:59.044802+0000","last_change":"2024-08-19T14:44:15.066789+0000","last_active":"2024-08-19T14:46:59.044802+0000","last_peered":"2024-08-19T14:46:59.044802+0000","last_clean":"2024-08-19T14:46:59.044802+0000","last_became_active":"2024-08-19T14:44:15.065981+0000","last_became_peered":"2024-08-19T14:44:15.065981+0000","last_unstale":"2024-08-19T14:46:59.044802+0000","last_undegraded":"2024-08-19T14:46:59.044802+0000","last_fullsized":"2024-08-19T14:46:59.044802+0000","mapping_epoch":826,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":827,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":9176,"log_dups_size":0,"ondisk_log_size":9176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173015040,"num_objects":2640,"num_object_clones":0,"num_object_copies":7920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2640,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9176,"num_write_kb":378112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.c","version":"70'139","reported_seq":758,"reported_epoch":894,"state":"active+clean","last_fresh":"2024-08-19T14:46:59.507872+0000","last_change":"2024-08-19T14:45:26.103908+0000","last_active":"2024-08-19T14:46:59.507872+0000","last_peered":"2024-08-19T14:46:59.507872+0000","last_clean":"2024-08-19T14:46:59.507872+0000","last_became_active":"2024-08-19T14:45:26.103255+0000","last_became_peered":"2024-08-19T14:45:26.103255+0000","last_unstale":"2024-08-19T14:46:59.507872+0000","last_undegraded":"2024-08-19T14:46:59.507872+0000","last_fullsized":"2024-08-19T14:46:59.507872+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:24.706574+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:22:52.164586+0000","last_clean_scrub_stamp":"2024-08-19T14:44:24.706574+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":983040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,1],"acting":[6,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"893'9087","reported_seq":13072,"reported_epoch":893,"state":"active+clean","last_fresh":"2024-08-19T14:46:57.745916+0000","last_change":"2024-08-19T14:45:06.219701+0000","last_active":"2024-08-19T14:46:57.745916+0000","last_peered":"2024-08-19T14:46:57.745916+0000","last_clean":"2024-08-19T14:46:57.745916+0000","last_became_active":"2024-08-19T14:45:06.216329+0000","last_became_peered":"2024-08-19T14:45:06.216329+0000","last_unstale":"2024-08-19T14:46:57.745916+0000","last_undegraded":"2024-08-19T14:46:57.745916+0000","last_fullsized":"2024-08-19T14:46:57.745916+0000","mapping_epoch":851,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":852,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":9087,"log_dups_size":0,"ondisk_log_size":9087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":170590208,"num_objects":2603,"num_object_clones":0,"num_object_copies":7809,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2603,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9087,"num_write_kb":374080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1915,"num_bytes_recovered":125501440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,9],"acting":[11,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.d","version":"70'139","reported_seq":538,"reported_epoch":894,"state":"active+clean","last_fresh":"2024-08-19T14:46:59.502836+0000","last_change":"2024-08-19T14:46:20.095131+0000","last_active":"2024-08-19T14:46:59.502836+0000","last_peered":"2024-08-19T14:46:59.502836+0000","last_clean":"2024-08-19T14:46:59.502836+0000","last_became_active":"2024-08-19T14:43:08.459577+0000","last_became_peered":"2024-08-19T14:43:08.459577+0000","last_unstale":"2024-08-19T14:46:59.502836+0000","last_undegraded":"2024-08-19T14:46:59.502836+0000","last_fullsized":"2024-08-19T14:46:59.502836+0000","mapping_epoch":793,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":794,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:46:20.095018+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:24:50.276408+0000","last_clean_scrub_stamp":"2024-08-19T14:46:20.095018+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:47:31.007933+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,8],"acting":[4,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.12","version":"70'139","reported_seq":1486,"reported_epoch":893,"state":"active+clean","last_fresh":"2024-08-19T14:46:54.112672+0000","last_change":"2024-08-19T14:45:26.109914+0000","last_active":"2024-08-19T14:46:54.112672+0000","last_peered":"2024-08-19T14:46:54.112672+0000","last_clean":"2024-08-19T14:46:54.112672+0000","last_became_active":"2024-08-19T14:45:26.109383+0000","last_became_peered":"2024-08-19T14:45:26.109383+0000","last_unstale":"2024-08-19T14:46:54.112672+0000","last_undegraded":"2024-08-19T14:46:54.112672+0000","last_fullsized":"2024-08-19T14:46:54.112672+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:30.604277+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:41:49.815826+0000","last_clean_scrub_stamp":"2024-08-19T14:44:30.604277+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,11],"acting":[1,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.13","version":"70'139","reported_seq":2019,"reported_epoch":893,"state":"active+clean","last_fresh":"2024-08-19T14:46:54.112454+0000","last_change":"2024-08-19T14:46:53.695559+0000","last_active":"2024-08-19T14:46:54.112454+0000","last_peered":"2024-08-19T14:46:54.112454+0000","last_clean":"2024-08-19T14:46:54.112454+0000","last_became_active":"2024-08-19T14:43:16.086968+0000","last_became_peered":"2024-08-19T14:43:16.086968+0000","last_unstale":"2024-08-19T14:46:54.112454+0000","last_undegraded":"2024-08-19T14:46:54.112454+0000","last_fullsized":"2024-08-19T14:46:54.112454+0000","mapping_epoch":800,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":801,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:46:53.695474+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:33:35.786105+0000","last_clean_scrub_stamp":"2024-08-19T14:46:53.695474+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:48:23.005145+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.10","version":"70'139","reported_seq":956,"reported_epoch":895,"state":"active+clean","last_fresh":"2024-08-19T14:47:00.523719+0000","last_change":"2024-08-19T14:44:38.187460+0000","last_active":"2024-08-19T14:47:00.523719+0000","last_peered":"2024-08-19T14:47:00.523719+0000","last_clean":"2024-08-19T14:47:00.523719+0000","last_became_active":"2024-08-19T14:44:38.187068+0000","last_became_peered":"2024-08-19T14:44:38.187068+0000","last_unstale":"2024-08-19T14:47:00.523719+0000","last_undegraded":"2024-08-19T14:47:00.523719+0000","last_fullsized":"2024-08-19T14:47:00.523719+0000","mapping_epoch":837,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:24.090078+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:34:58.616646+0000","last_clean_scrub_stamp":"2024-08-19T14:44:24.090078+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,4],"acting":[10,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.11","version":"70'139","reported_seq":535,"reported_epoch":894,"state":"active+clean","last_fresh":"2024-08-19T14:46:59.512857+0000","last_change":"2024-08-19T14:44:23.821840+0000","last_active":"2024-08-19T14:46:59.512857+0000","last_peered":"2024-08-19T14:46:59.512857+0000","last_clean":"2024-08-19T14:46:59.512857+0000","last_became_active":"2024-08-19T14:42:39.064403+0000","last_became_peered":"2024-08-19T14:42:39.064403+0000","last_unstale":"2024-08-19T14:46:59.512857+0000","last_undegraded":"2024-08-19T14:46:59.512857+0000","last_fullsized":"2024-08-19T14:46:59.512857+0000","mapping_epoch":778,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":779,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:23.821699+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:41:47.147376+0000","last_clean_scrub_stamp":"2024-08-19T14:44:23.821699+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,1],"acting":[8,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"70'139","reported_seq":1646,"reported_epoch":893,"state":"active+clean","last_fresh":"2024-08-19T14:46:55.739086+0000","last_change":"2024-08-19T14:46:55.739107+0000","last_active":"2024-08-19T14:46:55.739086+0000","last_peered":"2024-08-19T14:46:55.739086+0000","last_clean":"2024-08-19T14:46:55.739086+0000","last_became_active":"2024-08-19T14:44:38.183608+0000","last_became_peered":"2024-08-19T14:44:38.183608+0000","last_unstale":"2024-08-19T14:46:55.739086+0000","last_undegraded":"2024-08-19T14:46:55.739086+0000","last_fullsized":"2024-08-19T14:46:55.739086+0000","mapping_epoch":837,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:46:55.739034+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:28:34.216370+0000","last_clean_scrub_stamp":"2024-08-19T14:46:55.739034+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:48:22.771380+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,4],"acting":[1,11,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"70'139","reported_seq":1798,"reported_epoch":894,"state":"active+clean","last_fresh":"2024-08-19T14:46:59.513393+0000","last_change":"2024-08-19T14:46:56.367773+0000","last_active":"2024-08-19T14:46:59.513393+0000","last_peered":"2024-08-19T14:46:59.513393+0000","last_clean":"2024-08-19T14:46:59.513393+0000","last_became_active":"2024-08-19T14:44:38.183371+0000","last_became_peered":"2024-08-19T14:44:38.183371+0000","last_unstale":"2024-08-19T14:46:59.513393+0000","last_undegraded":"2024-08-19T14:46:59.513393+0000","last_fullsized":"2024-08-19T14:46:59.513393+0000","mapping_epoch":837,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:46:56.367650+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:46:56.367650+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:48:22.698349+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11],"acting":[8,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.14","version":"70'139","reported_seq":1202,"reported_epoch":893,"state":"active+clean","last_fresh":"2024-08-19T14:46:54.112457+0000","last_change":"2024-08-19T14:46:34.796943+0000","last_active":"2024-08-19T14:46:54.112457+0000","last_peered":"2024-08-19T14:46:54.112457+0000","last_clean":"2024-08-19T14:46:54.112457+0000","last_became_active":"2024-08-19T14:43:08.454350+0000","last_became_peered":"2024-08-19T14:43:08.454350+0000","last_unstale":"2024-08-19T14:46:54.112457+0000","last_undegraded":"2024-08-19T14:46:54.112457+0000","last_fullsized":"2024-08-19T14:46:54.112457+0000","mapping_epoch":793,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":794,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:46:34.796863+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:46:34.796863+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:48:00.974974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,9],"acting":[1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"70'139","reported_seq":549,"reported_epoch":894,"state":"active+clean","last_fresh":"2024-08-19T14:46:59.514084+0000","last_change":"2024-08-19T14:46:54.433540+0000","last_active":"2024-08-19T14:46:59.514084+0000","last_peered":"2024-08-19T14:46:59.514084+0000","last_clean":"2024-08-19T14:46:59.514084+0000","last_became_active":"2024-08-19T14:44:38.199514+0000","last_became_peered":"2024-08-19T14:44:38.199514+0000","last_unstale":"2024-08-19T14:46:59.514084+0000","last_undegraded":"2024-08-19T14:46:59.514084+0000","last_fullsized":"2024-08-19T14:46:59.514084+0000","mapping_epoch":837,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:46:54.431820+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:20:09.180923+0000","last_clean_scrub_stamp":"2024-08-19T14:46:54.431820+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:48:09.163994+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,11],"acting":[8,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"70'139","reported_seq":1639,"reported_epoch":894,"state":"active+clean","last_fresh":"2024-08-19T14:46:59.502642+0000","last_change":"2024-08-19T14:46:15.973722+0000","last_active":"2024-08-19T14:46:59.502642+0000","last_peered":"2024-08-19T14:46:59.502642+0000","last_clean":"2024-08-19T14:46:59.502642+0000","last_became_active":"2024-08-19T14:43:11.471324+0000","last_became_peered":"2024-08-19T14:43:11.471324+0000","last_unstale":"2024-08-19T14:46:59.502642+0000","last_undegraded":"2024-08-19T14:46:59.502642+0000","last_fullsized":"2024-08-19T14:46:59.502642+0000","mapping_epoch":795,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":796,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:46:15.973642+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:41:51.832284+0000","last_clean_scrub_stamp":"2024-08-19T14:46:15.973642+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:47:25.118982+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,11],"acting":[4,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1b","version":"70'139","reported_seq":2021,"reported_epoch":894,"state":"active+clean","last_fresh":"2024-08-19T14:46:59.520053+0000","last_change":"2024-08-19T14:46:58.388681+0000","last_active":"2024-08-19T14:46:59.520053+0000","last_peered":"2024-08-19T14:46:59.520053+0000","last_clean":"2024-08-19T14:46:59.520053+0000","last_became_active":"2024-08-19T14:42:39.065604+0000","last_became_peered":"2024-08-19T14:42:39.065604+0000","last_unstale":"2024-08-19T14:46:59.520053+0000","last_undegraded":"2024-08-19T14:46:59.520053+0000","last_fullsized":"2024-08-19T14:46:59.520053+0000","mapping_epoch":778,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":779,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:46:58.388563+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:37:59.510697+0000","last_clean_scrub_stamp":"2024-08-19T14:46:58.388563+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:48:21.771287+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,2],"acting":[8,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.18","version":"70'139","reported_seq":1007,"reported_epoch":893,"state":"active+clean","last_fresh":"2024-08-19T14:46:54.743777+0000","last_change":"2024-08-19T14:46:54.743821+0000","last_active":"2024-08-19T14:46:54.743777+0000","last_peered":"2024-08-19T14:46:54.743777+0000","last_clean":"2024-08-19T14:46:54.743777+0000","last_became_active":"2024-08-19T14:44:38.182712+0000","last_became_peered":"2024-08-19T14:44:38.182712+0000","last_unstale":"2024-08-19T14:46:54.743777+0000","last_undegraded":"2024-08-19T14:46:54.743777+0000","last_fullsized":"2024-08-19T14:46:54.743777+0000","mapping_epoch":837,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:46:54.743702+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:27:33.084161+0000","last_clean_scrub_stamp":"2024-08-19T14:46:54.743702+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:48:17.549331+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":402496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,11],"acting":[1,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.19","version":"70'139","reported_seq":868,"reported_epoch":895,"state":"active+clean","last_fresh":"2024-08-19T14:47:00.527279+0000","last_change":"2024-08-19T14:45:26.148752+0000","last_active":"2024-08-19T14:47:00.527279+0000","last_peered":"2024-08-19T14:47:00.527279+0000","last_clean":"2024-08-19T14:47:00.527279+0000","last_became_active":"2024-08-19T14:45:26.147921+0000","last_became_peered":"2024-08-19T14:45:26.147921+0000","last_unstale":"2024-08-19T14:47:00.527279+0000","last_undegraded":"2024-08-19T14:47:00.527279+0000","last_fullsized":"2024-08-19T14:47:00.527279+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:23.083891+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:44:23.083891+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,4],"acting":[10,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1e","version":"70'139","reported_seq":1025,"reported_epoch":893,"state":"active+clean","last_fresh":"2024-08-19T14:46:54.143174+0000","last_change":"2024-08-19T14:45:16.381698+0000","last_active":"2024-08-19T14:46:54.143174+0000","last_peered":"2024-08-19T14:46:54.143174+0000","last_clean":"2024-08-19T14:46:54.143174+0000","last_became_active":"2024-08-19T14:43:16.124258+0000","last_became_peered":"2024-08-19T14:43:16.124258+0000","last_unstale":"2024-08-19T14:46:54.143174+0000","last_undegraded":"2024-08-19T14:46:54.143174+0000","last_fullsized":"2024-08-19T14:46:54.143174+0000","mapping_epoch":800,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":801,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:45:16.381587+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:27:04.780939+0000","last_clean_scrub_stamp":"2024-08-19T14:45:16.381587+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,10],"acting":[11,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1c","version":"70'139","reported_seq":1775,"reported_epoch":894,"state":"active+clean","last_fresh":"2024-08-19T14:46:59.512716+0000","last_change":"2024-08-19T14:45:26.106081+0000","last_active":"2024-08-19T14:46:59.512716+0000","last_peered":"2024-08-19T14:46:59.512716+0000","last_clean":"2024-08-19T14:46:59.512716+0000","last_became_active":"2024-08-19T14:45:26.105382+0000","last_became_peered":"2024-08-19T14:45:26.105382+0000","last_unstale":"2024-08-19T14:46:59.512716+0000","last_undegraded":"2024-08-19T14:46:59.512716+0000","last_fullsized":"2024-08-19T14:46:59.512716+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:31.796263+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:44:31.796263+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1d","version":"70'139","reported_seq":1665,"reported_epoch":893,"state":"active+clean","last_fresh":"2024-08-19T14:46:54.112840+0000","last_change":"2024-08-19T14:44:29.650415+0000","last_active":"2024-08-19T14:46:54.112840+0000","last_peered":"2024-08-19T14:46:54.112840+0000","last_clean":"2024-08-19T14:46:54.112840+0000","last_became_active":"2024-08-19T14:35:11.540488+0000","last_became_peered":"2024-08-19T14:35:11.540488+0000","last_unstale":"2024-08-19T14:46:54.112840+0000","last_undegraded":"2024-08-19T14:46:54.112840+0000","last_fullsized":"2024-08-19T14:46:54.112840+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:29.650238+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:44:29.650238+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":2675245080,"num_objects":40822,"num_object_clones":0,"num_object_copies":122466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40822,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":148184,"num_write_kb":6048129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39691,"num_bytes_recovered":2601189376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8542171136,"data_stored":8542158920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":148184,"ondisk_log_size":148184,"up":48,"acting":48,"num_store_stats":9},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1385536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2785280,"data_stored":2771072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187883,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2251096,"kb_used_data":1031952,"kb_used_omap":1,"kb_used_meta":1219070,"kb_avail":91494056,"statfs":{"total":95995035648,"available":93689913344,"internally_reserved":0,"allocated":1056718848,"data_stored":1054051175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":1248328132},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":539,"seq":2314987372751,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1358364,"kb_used_data":1137428,"kb_used_omap":4,"kb_used_meta":220731,"kb_avail":92386788,"statfs":{"total":95995035648,"available":94604070912,"internally_reserved":0,"allocated":1164726272,"data_stored":1162057925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4407,"internal_metadata":226029257},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":74,"seq":317827580341,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2739856,"kb_used_data":1506184,"kb_used_omap":1,"kb_used_meta":1233662,"kb_avail":91005296,"statfs":{"total":95995035648,"available":93189423104,"internally_reserved":0,"allocated":1542332416,"data_stored":1539668295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":1263270337},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874684,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2146516,"kb_used_data":1114380,"kb_used_omap":1,"kb_used_meta":1032126,"kb_avail":91598636,"statfs":{"total":95995035648,"available":93797003264,"internally_reserved":0,"allocated":1141125120,"data_stored":1138456927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1056897477},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":0,"seq":244813136319,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430659,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1005712,"kb_used_data":825992,"kb_used_omap":1,"kb_used_meta":179710,"kb_avail":92739440,"statfs":{"total":95995035648,"available":94965186560,"internally_reserved":0,"allocated":845815808,"data_stored":843151687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":184023494},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":171798692295,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986636,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1771092,"kb_used_data":509516,"kb_used_omap":1,"kb_used_meta":1261566,"kb_avail":91974060,"statfs":{"total":95995035648,"available":94181437440,"internally_reserved":0,"allocated":521744384,"data_stored":519076191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1291844037},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":111669150161,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":634,"seq":2723009265807,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":721320,"kb_used_data":171420,"kb_used_omap":4,"kb_used_meta":549883,"kb_avail":93023832,"statfs":{"total":95995035648,"available":95256403968,"internally_reserved":0,"allocated":175534080,"data_stored":172862181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4407,"internal_metadata":563080905},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575322,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3464028,"kb_used_data":1997460,"kb_used_omap":1,"kb_used_meta":1466558,"kb_avail":90281124,"statfs":{"total":95995035648,"available":92447870976,"internally_reserved":0,"allocated":2045399040,"data_stored":2042727295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1501755845},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":34359738846,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2020806656,"data_stored":2020802584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":165085184,"data_stored":165085184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":503517184,"data_stored":503513112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":198836224,"data_stored":198836224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":831193088,"data_stored":831193088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1119621120,"data_stored":1119617048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1528889344,"data_stored":1528889344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1138556928,"data_stored":1138556928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1035665408,"data_stored":1035665408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T14:47:03.908 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph status --format=json 2024-08-19T14:47:03.909 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:47:03.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:03.945 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:03.945 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:47:03.980 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:03 smithi032 ceph-mon[20004]: 9.e scrub starts 2024-08-19T14:47:03.981 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:03 smithi032 ceph-mon[20004]: 9.b scrub ok 2024-08-19T14:47:03.981 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:03 smithi032 ceph-mon[20004]: pgmap v2110: 47 pgs: 1 peering, 46 active+clean; 2.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 614 op/s; 4.3 MiB/s, 68 objects/s recovering 2024-08-19T14:47:03.981 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:03 smithi032 ceph-mon[20004]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-08-19T14:47:03.981 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:03 smithi032 ceph-mon[20004]: Cluster is now healthy 2024-08-19T14:47:03.981 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:03 smithi032 ceph-mon[20004]: 9.d scrub starts 2024-08-19T14:47:03.981 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:03 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:47:03.981 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:03 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:47:03.981 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:03 smithi032 ceph-mon[20004]: from='client.18123 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:47:03.981 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {auth=37,config=2,health=12,kv=49} crc {auth=1377161207,config=3457680788,health=3897446751,kv=712890808}) 2024-08-19T14:47:03.981 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {kv=52,logm=48} crc {kv=317465771,logm=1936600621}) 2024-08-19T14:47:03.982 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3045089614}) 2024-08-19T14:47:03.982 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2936916937}) 2024-08-19T14:47:03.982 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=397173111}) 2024-08-19T14:47:03.982 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4258523784}) 2024-08-19T14:47:03.982 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=843530821}) 2024-08-19T14:47:03.982 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=142205573}) 2024-08-19T14:47:03.982 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=445330544}) 2024-08-19T14:47:03.982 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3791367746}) 2024-08-19T14:47:03.982 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3778345534}) 2024-08-19T14:47:03.982 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=375354755}) 2024-08-19T14:47:03.983 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1371943234}) 2024-08-19T14:47:03.983 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3686152039}) 2024-08-19T14:47:03.983 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2380633542}) 2024-08-19T14:47:03.983 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2137648524}) 2024-08-19T14:47:03.983 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1709107726}) 2024-08-19T14:47:03.983 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2380748669}) 2024-08-19T14:47:03.983 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2708757337}) 2024-08-19T14:47:03.983 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2438119731}) 2024-08-19T14:47:03.983 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2129420428}) 2024-08-19T14:47:03.983 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1247602590}) 2024-08-19T14:47:03.984 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3789971044}) 2024-08-19T14:47:03.984 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=855894702}) 2024-08-19T14:47:03.984 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4119951111}) 2024-08-19T14:47:03.984 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2107021522}) 2024-08-19T14:47:03.984 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1102446228}) 2024-08-19T14:47:03.984 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4200598762}) 2024-08-19T14:47:03.984 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=174837066}) 2024-08-19T14:47:03.984 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1002356016}) 2024-08-19T14:47:03.984 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=691449663}) 2024-08-19T14:47:03.984 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3306126708}) 2024-08-19T14:47:03.984 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=196313875}) 2024-08-19T14:47:03.984 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4189194897}) 2024-08-19T14:47:03.984 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=204271178}) 2024-08-19T14:47:03.984 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3833701630}) 2024-08-19T14:47:03.984 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=803348292}) 2024-08-19T14:47:03.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1174296533}) 2024-08-19T14:47:03.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=5475840}) 2024-08-19T14:47:03.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2207725445}) 2024-08-19T14:47:03.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4256395641}) 2024-08-19T14:47:03.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1100472328}) 2024-08-19T14:47:03.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=491617764}) 2024-08-19T14:47:03.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=278938213}) 2024-08-19T14:47:03.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1568626822}) 2024-08-19T14:47:03.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3791461525}) 2024-08-19T14:47:03.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1833739206}) 2024-08-19T14:47:03.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=383959061}) 2024-08-19T14:47:03.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=947824193}) 2024-08-19T14:47:03.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2888894165}) 2024-08-19T14:47:03.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:03 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2249112383}) 2024-08-19T14:47:04.008 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:47:04.046 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:47:04.068 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:04.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:04.069 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:47:04.170 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:47:04.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:04.205 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:04.205 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:47:04.306 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:47:04.328 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:04.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:04.330 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:47:04.406 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:47:04.406 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-08-19T14:47:04.406 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-08-19T14:47:04.406 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:47:04.431 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:47:04.461 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:04.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:04.463 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:47:04.564 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:47:04.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:04.602 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:04.602 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:47:04.608 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:47:04.703 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:47:04.726 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:04.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:04.727 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:47:04.739 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:04 smithi031 ceph-mon[16336]: 1.9 scrub starts 2024-08-19T14:47:04.739 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:04 smithi031 ceph-mon[16336]: 1.9 scrub ok 2024-08-19T14:47:04.739 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:04 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:47:04.739 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:04 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:47:04.739 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:04 smithi031 ceph-mon[16336]: 9.d scrub ok 2024-08-19T14:47:04.739 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:04 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3213200565}) 2024-08-19T14:47:04.739 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:04 smithi031 ceph-mon[16336]: osdmap e897: 12 total, 12 up, 8 in 2024-08-19T14:47:04.740 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:04 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=316771497}) 2024-08-19T14:47:04.740 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:04 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2943740764}) 2024-08-19T14:47:04.740 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:04 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1363524529}) 2024-08-19T14:47:04.740 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:04 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1108606071}) 2024-08-19T14:47:04.740 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:04 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1035726813}) 2024-08-19T14:47:04.740 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:04 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3753145665}) 2024-08-19T14:47:04.740 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:04 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1605102415}) 2024-08-19T14:47:04.740 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:04 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=173592007}) 2024-08-19T14:47:04.740 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:04 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1608509793}) 2024-08-19T14:47:04.740 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:04 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4304000}) 2024-08-19T14:47:04.740 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:04 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4154024359}) 2024-08-19T14:47:04.740 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:04 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1605418230}) 2024-08-19T14:47:04.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:04 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1594033836}) 2024-08-19T14:47:04.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:04 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=658757203}) 2024-08-19T14:47:04.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:04 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3469264091}) 2024-08-19T14:47:04.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:04 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3440055162}) 2024-08-19T14:47:04.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:04 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2391642812}) 2024-08-19T14:47:04.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:04 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3510695015}) 2024-08-19T14:47:04.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:04 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3480919029}) 2024-08-19T14:47:04.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:04 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1971503379}) 2024-08-19T14:47:04.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:04 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=716574723}) 2024-08-19T14:47:04.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:04 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2461588400}) 2024-08-19T14:47:04.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:04 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3252574057}) 2024-08-19T14:47:04.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:04 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3684336475}) 2024-08-19T14:47:04.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:04 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2827382901}) 2024-08-19T14:47:04.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:04 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=930124493}) 2024-08-19T14:47:04.742 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:04 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3741544493}) 2024-08-19T14:47:04.742 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:04 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=267664478}) 2024-08-19T14:47:04.742 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:04 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2364304895}) 2024-08-19T14:47:04.742 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:04 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3335342760}) 2024-08-19T14:47:04.742 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:04 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3098890482}) 2024-08-19T14:47:04.742 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:04 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {logm=31,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=45} crc {logm=3506751807,mdsmap=2474562478,mgr=2924679688,mgr_command_descs=1300230762,mgr_metadata=482233419,mgrstat=324019580}) 2024-08-19T14:47:04.742 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:04 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=100} crc {mgrstat=116414799}) 2024-08-19T14:47:04.742 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:04 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=62,mon_config_key=38} crc {mgrstat=4070573312,mon_config_key=3242851253}) 2024-08-19T14:47:04.742 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:04 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=454784598}) 2024-08-19T14:47:04.742 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:04 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=828250284}) 2024-08-19T14:47:04.742 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:04 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=41,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=41} crc {mon_config_key=3976644904,monmap=3193798142,osd_metadata=289051117,osd_pg_creating=2422537692,osdmap=3042147239}) 2024-08-19T14:47:04.742 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:04 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2296170238}) 2024-08-19T14:47:04.742 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:04 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2657074974}) 2024-08-19T14:47:04.743 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:04 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1259212152}) 2024-08-19T14:47:04.743 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:04 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2416994630}) 2024-08-19T14:47:04.743 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:04 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1712751142}) 2024-08-19T14:47:04.743 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:04 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=336823981}) 2024-08-19T14:47:04.743 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:04 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=633633186}) 2024-08-19T14:47:04.743 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:04 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1802739298}) 2024-08-19T14:47:04.743 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:04 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=974594062}) 2024-08-19T14:47:04.743 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:04 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1377205422}) 2024-08-19T14:47:04.743 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:04 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1774636056}) 2024-08-19T14:47:04.743 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:04 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2528785403}) 2024-08-19T14:47:04.743 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:04 smithi031 ceph-mon[16336]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=16} crc {osdmap=3766021918}) 2024-08-19T14:47:04.743 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:04 smithi031 ceph-mon[16336]: 1.4 deep-scrub starts 2024-08-19T14:47:04.743 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:04 smithi031 ceph-mon[16336]: 1.4 deep-scrub ok 2024-08-19T14:47:04.827 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:47:04.852 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:04.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:04.853 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:47:04.954 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:47:04.978 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:04.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:04.979 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:47:05.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:04 smithi032 ceph-mon[20004]: 1.9 scrub starts 2024-08-19T14:47:05.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:04 smithi032 ceph-mon[20004]: 1.9 scrub ok 2024-08-19T14:47:05.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:04 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:47:05.009 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:04 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:47:05.009 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:04 smithi032 ceph-mon[20004]: 9.d scrub ok 2024-08-19T14:47:05.009 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:04 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3213200565}) 2024-08-19T14:47:05.009 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:04 smithi032 ceph-mon[20004]: osdmap e897: 12 total, 12 up, 8 in 2024-08-19T14:47:05.009 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:04 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=316771497}) 2024-08-19T14:47:05.009 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:04 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2943740764}) 2024-08-19T14:47:05.009 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:04 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1363524529}) 2024-08-19T14:47:05.009 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:04 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1108606071}) 2024-08-19T14:47:05.009 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:04 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1035726813}) 2024-08-19T14:47:05.009 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:04 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3753145665}) 2024-08-19T14:47:05.009 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:04 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1605102415}) 2024-08-19T14:47:05.009 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:04 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=173592007}) 2024-08-19T14:47:05.009 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:04 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1608509793}) 2024-08-19T14:47:05.009 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:04 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4304000}) 2024-08-19T14:47:05.010 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:04 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4154024359}) 2024-08-19T14:47:05.010 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:04 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1605418230}) 2024-08-19T14:47:05.010 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:04 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1594033836}) 2024-08-19T14:47:05.010 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:04 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=658757203}) 2024-08-19T14:47:05.010 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:04 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3469264091}) 2024-08-19T14:47:05.010 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:04 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3440055162}) 2024-08-19T14:47:05.010 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:04 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2391642812}) 2024-08-19T14:47:05.010 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:04 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3510695015}) 2024-08-19T14:47:05.010 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:04 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3480919029}) 2024-08-19T14:47:05.010 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:04 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1971503379}) 2024-08-19T14:47:05.010 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:04 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=716574723}) 2024-08-19T14:47:05.010 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:04 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2461588400}) 2024-08-19T14:47:05.010 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:04 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3252574057}) 2024-08-19T14:47:05.011 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:04 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3684336475}) 2024-08-19T14:47:05.011 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:04 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2827382901}) 2024-08-19T14:47:05.011 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:04 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=930124493}) 2024-08-19T14:47:05.011 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:04 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3741544493}) 2024-08-19T14:47:05.011 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:04 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=267664478}) 2024-08-19T14:47:05.011 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:04 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2364304895}) 2024-08-19T14:47:05.011 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:04 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3335342760}) 2024-08-19T14:47:05.011 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:04 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3098890482}) 2024-08-19T14:47:05.011 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:04 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {logm=31,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=45} crc {logm=3506751807,mdsmap=2474562478,mgr=2924679688,mgr_command_descs=1300230762,mgr_metadata=482233419,mgrstat=324019580}) 2024-08-19T14:47:05.011 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:04 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=100} crc {mgrstat=116414799}) 2024-08-19T14:47:05.011 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:04 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=62,mon_config_key=38} crc {mgrstat=4070573312,mon_config_key=3242851253}) 2024-08-19T14:47:05.011 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:04 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=454784598}) 2024-08-19T14:47:05.011 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:04 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=828250284}) 2024-08-19T14:47:05.011 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:04 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=41,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=41} crc {mon_config_key=3976644904,monmap=3193798142,osd_metadata=289051117,osd_pg_creating=2422537692,osdmap=3042147239}) 2024-08-19T14:47:05.011 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:04 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2296170238}) 2024-08-19T14:47:05.012 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:04 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2657074974}) 2024-08-19T14:47:05.012 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:04 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1259212152}) 2024-08-19T14:47:05.012 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:04 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2416994630}) 2024-08-19T14:47:05.012 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:04 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1712751142}) 2024-08-19T14:47:05.012 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:04 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=336823981}) 2024-08-19T14:47:05.012 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:04 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=633633186}) 2024-08-19T14:47:05.012 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:04 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1802739298}) 2024-08-19T14:47:05.012 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:04 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=974594062}) 2024-08-19T14:47:05.012 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:04 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1377205422}) 2024-08-19T14:47:05.012 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:04 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1774636056}) 2024-08-19T14:47:05.012 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:04 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2528785403}) 2024-08-19T14:47:05.012 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:04 smithi032 ceph-mon[20004]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=16} crc {osdmap=3766021918}) 2024-08-19T14:47:05.012 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:04 smithi032 ceph-mon[20004]: 1.4 deep-scrub starts 2024-08-19T14:47:05.012 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:04 smithi032 ceph-mon[20004]: 1.4 deep-scrub ok 2024-08-19T14:47:05.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:04 smithi045 ceph-mon[20143]: 1.9 scrub starts 2024-08-19T14:47:05.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:04 smithi045 ceph-mon[20143]: 1.9 scrub ok 2024-08-19T14:47:05.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:04 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:47:05.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:04 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:47:05.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:04 smithi045 ceph-mon[20143]: 9.d scrub ok 2024-08-19T14:47:05.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:04 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3213200565}) 2024-08-19T14:47:05.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:04 smithi045 ceph-mon[20143]: osdmap e897: 12 total, 12 up, 8 in 2024-08-19T14:47:05.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:04 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=316771497}) 2024-08-19T14:47:05.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:04 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2943740764}) 2024-08-19T14:47:05.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:04 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1363524529}) 2024-08-19T14:47:05.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:04 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1108606071}) 2024-08-19T14:47:05.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:04 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1035726813}) 2024-08-19T14:47:05.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:04 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3753145665}) 2024-08-19T14:47:05.048 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:04 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1605102415}) 2024-08-19T14:47:05.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:04 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=173592007}) 2024-08-19T14:47:05.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:04 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1608509793}) 2024-08-19T14:47:05.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:04 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4304000}) 2024-08-19T14:47:05.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:04 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4154024359}) 2024-08-19T14:47:05.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:04 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1605418230}) 2024-08-19T14:47:05.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:04 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1594033836}) 2024-08-19T14:47:05.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:04 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=658757203}) 2024-08-19T14:47:05.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:04 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3469264091}) 2024-08-19T14:47:05.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:04 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3440055162}) 2024-08-19T14:47:05.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:04 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2391642812}) 2024-08-19T14:47:05.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:04 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3510695015}) 2024-08-19T14:47:05.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:04 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3480919029}) 2024-08-19T14:47:05.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:04 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1971503379}) 2024-08-19T14:47:05.049 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:04 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=716574723}) 2024-08-19T14:47:05.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:04 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2461588400}) 2024-08-19T14:47:05.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:04 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3252574057}) 2024-08-19T14:47:05.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:04 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3684336475}) 2024-08-19T14:47:05.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:04 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2827382901}) 2024-08-19T14:47:05.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:04 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=930124493}) 2024-08-19T14:47:05.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:04 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3741544493}) 2024-08-19T14:47:05.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:04 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=267664478}) 2024-08-19T14:47:05.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:04 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2364304895}) 2024-08-19T14:47:05.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:04 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3335342760}) 2024-08-19T14:47:05.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:04 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3098890482}) 2024-08-19T14:47:05.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:04 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {logm=31,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=45} crc {logm=3506751807,mdsmap=2474562478,mgr=2924679688,mgr_command_descs=1300230762,mgr_metadata=482233419,mgrstat=324019580}) 2024-08-19T14:47:05.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:04 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=100} crc {mgrstat=116414799}) 2024-08-19T14:47:05.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:04 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=62,mon_config_key=38} crc {mgrstat=4070573312,mon_config_key=3242851253}) 2024-08-19T14:47:05.050 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:04 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=454784598}) 2024-08-19T14:47:05.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:04 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=828250284}) 2024-08-19T14:47:05.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:04 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=41,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=41} crc {mon_config_key=3976644904,monmap=3193798142,osd_metadata=289051117,osd_pg_creating=2422537692,osdmap=3042147239}) 2024-08-19T14:47:05.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:04 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2296170238}) 2024-08-19T14:47:05.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:04 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2657074974}) 2024-08-19T14:47:05.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:04 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1259212152}) 2024-08-19T14:47:05.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:04 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2416994630}) 2024-08-19T14:47:05.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:04 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1712751142}) 2024-08-19T14:47:05.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:04 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=336823981}) 2024-08-19T14:47:05.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:04 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=633633186}) 2024-08-19T14:47:05.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:04 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1802739298}) 2024-08-19T14:47:05.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:04 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=974594062}) 2024-08-19T14:47:05.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:04 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1377205422}) 2024-08-19T14:47:05.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:04 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1774636056}) 2024-08-19T14:47:05.051 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:04 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2528785403}) 2024-08-19T14:47:05.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:04 smithi045 ceph-mon[20143]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=16} crc {osdmap=3766021918}) 2024-08-19T14:47:05.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:04 smithi045 ceph-mon[20143]: 1.4 deep-scrub starts 2024-08-19T14:47:05.052 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:04 smithi045 ceph-mon[20143]: 1.4 deep-scrub ok 2024-08-19T14:47:05.079 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:47:05.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:05.116 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:05.117 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:47:05.218 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:47:05.240 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:05.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:05.241 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:47:05.244 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:47:05.341 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:47:05.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-08-19T14:47:05.355+0000 7f6df61b5640 0 --2- 172.21.15.31:0/3874162442 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6df010bec0 0x7f6df010e340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-08-19T14:47:05.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:05.379 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:05.379 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:47:05.382 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-08-19T14:47:05.480 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:47:05.504 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:05.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:05.505 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:47:05.561 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is set 2024-08-19T14:47:05.606 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:47:05.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:05 smithi031 ceph-mon[16336]: 9.e scrub ok 2024-08-19T14:47:05.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:05 smithi031 ceph-mon[16336]: pgmap v2112: 47 pgs: 1 peering, 46 active+clean; 2.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 725 op/s; 5.2 MiB/s, 83 objects/s recovering 2024-08-19T14:47:05.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:05 smithi031 ceph-mon[16336]: 9.2 scrub starts 2024-08-19T14:47:05.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:05 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/732153277' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:47:05.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:05 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:47:05.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:05 smithi031 ceph-mon[16336]: 1.b scrub starts 2024-08-19T14:47:05.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:05 smithi031 ceph-mon[16336]: 1.b scrub ok 2024-08-19T14:47:05.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:05.648 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:05.648 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:47:05.666 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:47:05.746 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:47:05.746 INFO:teuthology.orchestra.run.smithi031.stdout:{"fsid":"ed7d1c6e-5e33-11ef-bccf-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub flag(s) set","count":7},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":2403,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":898,"num_osds":12,"num_up_osds":12,"osd_up_since":1724078110,"num_in_osds":8,"osd_in_since":1724078722,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":46},{"state_name":"peering","count":1}],"num_pgs":47,"num_pools":2,"num_objects":39187,"data_bytes":2568553016,"bytes_used":15485657088,"bytes_avail":752474628096,"bytes_total":767960285184,"inactive_pgs_ratio":0.021276595070958138,"recovering_objects_per_sec":83,"recovering_bytes_per_sec":5482388,"recovering_keys_per_sec":0,"num_objects_recovered":409,"num_bytes_recovered":26804224,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":725},"fsmap":{"epoch":1,"btime":"2024-08-19T14:05:22:841940+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.31:8443/"}},"servicemap":{"epoch":53,"modified":"2024-08-19T14:46:58.391653+0000","services":{"osd":{"daemons":{"summary":"","8":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-08-19T14:47:05.748 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:47:05.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:05 smithi032 ceph-mon[20004]: 9.e scrub ok 2024-08-19T14:47:05.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:05 smithi032 ceph-mon[20004]: pgmap v2112: 47 pgs: 1 peering, 46 active+clean; 2.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 725 op/s; 5.2 MiB/s, 83 objects/s recovering 2024-08-19T14:47:05.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:05 smithi032 ceph-mon[20004]: 9.2 scrub starts 2024-08-19T14:47:05.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:05 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/732153277' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:47:05.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:05 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:47:05.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:05 smithi032 ceph-mon[20004]: 1.b scrub starts 2024-08-19T14:47:05.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:05 smithi032 ceph-mon[20004]: 1.b scrub ok 2024-08-19T14:47:05.774 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:05.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:05.775 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:47:05.876 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:47:05.914 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:05.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:05.915 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:47:06.016 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:47:06.037 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:47:06.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:06.038 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:06.038 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:47:06.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:05 smithi045 ceph-mon[20143]: 9.e scrub ok 2024-08-19T14:47:06.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:05 smithi045 ceph-mon[20143]: pgmap v2112: 47 pgs: 1 peering, 46 active+clean; 2.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 725 op/s; 5.2 MiB/s, 83 objects/s recovering 2024-08-19T14:47:06.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:05 smithi045 ceph-mon[20143]: 9.2 scrub starts 2024-08-19T14:47:06.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:05 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/732153277' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:47:06.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:05 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:47:06.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:05 smithi045 ceph-mon[20143]: 1.b scrub starts 2024-08-19T14:47:06.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:05 smithi045 ceph-mon[20143]: 1.b scrub ok 2024-08-19T14:47:06.139 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:47:06.172 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:06.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:06.174 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:47:06.275 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:47:06.300 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:06.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:06.301 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:47:06.402 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:47:06.426 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:06.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:06.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:47:06.499 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:47:06.499 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-08-19T14:47:06.499 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-08-19T14:47:06.500 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-08-19T14:47:06.500 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:47:06.501 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-08-19T14:47:06.501 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-19T14:47:06.529 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:47:06.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:06.567 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:06.567 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:47:06.668 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:47:06.691 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:06.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:06.692 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:47:06.743 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:06 smithi031 ceph-mon[16336]: 1.11 scrub starts 2024-08-19T14:47:06.743 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:06 smithi031 ceph-mon[16336]: 1.11 scrub ok 2024-08-19T14:47:06.743 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:06 smithi031 ceph-mon[16336]: 9.2 scrub ok 2024-08-19T14:47:06.743 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:06 smithi031 ceph-mon[16336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:47:06.744 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:06 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:47:06.744 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:06 smithi031 ceph-mon[16336]: osdmap e898: 12 total, 12 up, 8 in 2024-08-19T14:47:06.744 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:06 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/4105233224' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-08-19T14:47:06.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:06 smithi032 ceph-mon[20004]: 1.11 scrub starts 2024-08-19T14:47:06.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:06 smithi032 ceph-mon[20004]: 1.11 scrub ok 2024-08-19T14:47:06.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:06 smithi032 ceph-mon[20004]: 9.2 scrub ok 2024-08-19T14:47:06.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:06 smithi032 ceph-mon[20004]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:47:06.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:06 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:47:06.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:06 smithi032 ceph-mon[20004]: osdmap e898: 12 total, 12 up, 8 in 2024-08-19T14:47:06.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:06 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/4105233224' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-08-19T14:47:06.793 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:47:06.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:06.830 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:06.831 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:47:06.932 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:47:06.953 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:06.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:06.955 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:47:07.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:06 smithi045 ceph-mon[20143]: 1.11 scrub starts 2024-08-19T14:47:07.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:06 smithi045 ceph-mon[20143]: 1.11 scrub ok 2024-08-19T14:47:07.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:06 smithi045 ceph-mon[20143]: 9.2 scrub ok 2024-08-19T14:47:07.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:06 smithi045 ceph-mon[20143]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:47:07.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:06 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:47:07.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:06 smithi045 ceph-mon[20143]: osdmap e898: 12 total, 12 up, 8 in 2024-08-19T14:47:07.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:06 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/4105233224' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-08-19T14:47:07.055 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:47:07.081 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:07.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:07.082 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:47:07.183 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:47:07.207 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:07.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:07.208 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:47:07.218 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-08-19T14:47:07.309 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:47:07.348 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:07.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:07.350 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:47:07.451 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:47:07.473 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:07.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:07.474 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:47:07.496 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:47:07.575 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:47:07.616 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:07.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:07.618 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:47:07.719 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:47:07.745 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:07.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:07.746 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:47:07.847 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:47:07.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:07 smithi031 ceph-mon[16336]: pgmap v2114: 47 pgs: 47 active+clean; 2.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 733 op/s; 4.3 MiB/s, 69 objects/s recovering 2024-08-19T14:47:07.870 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:07.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:07.871 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:47:07.893 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:07 smithi032 ceph-mon[20004]: pgmap v2114: 47 pgs: 47 active+clean; 2.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 733 op/s; 4.3 MiB/s, 69 objects/s recovering 2024-08-19T14:47:07.971 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:47:08.005 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:47:08.006 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-08-19T14:47:08.006 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-08-19T14:47:08.006 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-08-19T14:47:08.006 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.006 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26911.0:198785 8.8 8:13c17521:::benchmark_data_smithi045_137103_object86321:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e801)", 2024-08-19T14:47:08.006 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:43:17.891055+0000", 2024-08-19T14:47:08.006 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 230.11174849, 2024-08-19T14:47:08.006 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.112352038, 2024-08-19T14:47:08.006 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:47:08.006 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:47:08.006 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:47:08.007 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:47:08.007 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26911", 2024-08-19T14:47:08.007 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3494993995", 2024-08-19T14:47:08.007 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 198785 2024-08-19T14:47:08.007 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.007 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:47:08.007 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.007 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:47:08.007 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:17.891055+0000", 2024-08-19T14:47:08.007 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:08.007 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.007 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.007 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:47:08.007 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:17.891055+0000", 2024-08-19T14:47:08.008 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:08.008 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.008 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.008 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:47:08.008 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:17.891057+0000", 2024-08-19T14:47:08.008 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.136e-06 2024-08-19T14:47:08.008 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.008 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.008 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:47:08.008 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:17.891066+0000", 2024-08-19T14:47:08.008 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.38e-06 2024-08-19T14:47:08.008 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.008 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.009 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:47:08.009 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:17.891090+0000", 2024-08-19T14:47:08.009 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3473e-05 2024-08-19T14:47:08.009 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.009 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.009 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:47:08.009 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:17.891104+0000", 2024-08-19T14:47:08.009 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4239999999999999e-05 2024-08-19T14:47:08.009 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.009 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.009 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:47:08.009 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:17.891206+0000", 2024-08-19T14:47:08.009 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000101908 2024-08-19T14:47:08.010 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.010 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.010 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:47:08.010 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:17.891339+0000", 2024-08-19T14:47:08.010 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013358200000000001 2024-08-19T14:47:08.010 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.010 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.010 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 9,10", 2024-08-19T14:47:08.010 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:17.966095+0000", 2024-08-19T14:47:08.010 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.074756053000000003 2024-08-19T14:47:08.010 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.010 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.011 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:47:08.011 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:17.973930+0000", 2024-08-19T14:47:08.011 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0078345680000000001 2024-08-19T14:47:08.011 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.011 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.011 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:47:08.011 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:17.981951+0000", 2024-08-19T14:47:08.011 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0080210820000000006 2024-08-19T14:47:08.011 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.011 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.011 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:47:08.011 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:18.002607+0000", 2024-08-19T14:47:08.011 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.020656437999999999 2024-08-19T14:47:08.011 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.012 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.012 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:47:08.012 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:18.002681+0000", 2024-08-19T14:47:08.012 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3963999999999999e-05 2024-08-19T14:47:08.012 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.012 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.012 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:47:08.012 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:18.003407+0000", 2024-08-19T14:47:08.012 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00072521399999999996 2024-08-19T14:47:08.012 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:08.012 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:47:08.012 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:08.012 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.012 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.013 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26911.0:198786 8.8 8:136c67d1:::benchmark_data_smithi045_137103_object86322:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e801)", 2024-08-19T14:47:08.013 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:43:17.897185+0000", 2024-08-19T14:47:08.013 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 230.105618354, 2024-08-19T14:47:08.013 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.106900863, 2024-08-19T14:47:08.013 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:47:08.013 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:47:08.013 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:47:08.013 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:47:08.013 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26911", 2024-08-19T14:47:08.013 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3494993995", 2024-08-19T14:47:08.013 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 198786 2024-08-19T14:47:08.013 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.013 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:47:08.013 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.014 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:47:08.014 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:17.897185+0000", 2024-08-19T14:47:08.014 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:08.014 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.014 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.014 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:47:08.014 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:17.897185+0000", 2024-08-19T14:47:08.014 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:08.014 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.014 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.014 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:47:08.014 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:17.897187+0000", 2024-08-19T14:47:08.014 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8840000000000001e-06 2024-08-19T14:47:08.015 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.015 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.015 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:47:08.015 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:17.897197+0000", 2024-08-19T14:47:08.015 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.9490000000000008e-06 2024-08-19T14:47:08.015 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.015 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.015 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:47:08.015 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:17.966321+0000", 2024-08-19T14:47:08.015 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.069124367000000006 2024-08-19T14:47:08.015 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.015 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.015 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:47:08.016 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:17.966330+0000", 2024-08-19T14:47:08.016 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.0550000000000005e-06 2024-08-19T14:47:08.016 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.016 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.016 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:47:08.016 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:17.973960+0000", 2024-08-19T14:47:08.016 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.007629907 2024-08-19T14:47:08.016 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.016 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.016 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:47:08.016 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:17.974104+0000", 2024-08-19T14:47:08.017 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000144165 2024-08-19T14:47:08.017 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.017 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.017 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 9,10", 2024-08-19T14:47:08.017 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:17.974243+0000", 2024-08-19T14:47:08.017 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013851099999999999 2024-08-19T14:47:08.017 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.017 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.017 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:47:08.018 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:17.986248+0000", 2024-08-19T14:47:08.018 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.012005277999999999 2024-08-19T14:47:08.018 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.018 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.018 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:47:08.018 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:17.988393+0000", 2024-08-19T14:47:08.018 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0021452760000000002 2024-08-19T14:47:08.018 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.018 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.018 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:47:08.018 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:18.003477+0000", 2024-08-19T14:47:08.018 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0150844 2024-08-19T14:47:08.019 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.019 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.019 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:47:08.019 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:18.003539+0000", 2024-08-19T14:47:08.019 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1297000000000006e-05 2024-08-19T14:47:08.019 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.019 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.019 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:47:08.019 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:18.004086+0000", 2024-08-19T14:47:08.019 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00054677399999999998 2024-08-19T14:47:08.019 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:08.019 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:47:08.019 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:08.020 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.020 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.020 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26911.0:202285 8.11 8:8c27a3a6:::benchmark_data_smithi045_137103_object89821:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e801)", 2024-08-19T14:47:08.020 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:43:23.118440+0000", 2024-08-19T14:47:08.020 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 224.88436319799999, 2024-08-19T14:47:08.020 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.102394708, 2024-08-19T14:47:08.020 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:47:08.020 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:47:08.020 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:47:08.020 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:47:08.020 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26911", 2024-08-19T14:47:08.020 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3494993995", 2024-08-19T14:47:08.020 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 202285 2024-08-19T14:47:08.020 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.021 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:47:08.021 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.021 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:47:08.021 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:23.118440+0000", 2024-08-19T14:47:08.021 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:08.021 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.021 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.021 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:47:08.021 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:23.118440+0000", 2024-08-19T14:47:08.021 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:08.021 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.021 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.021 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:47:08.022 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:23.118442+0000", 2024-08-19T14:47:08.022 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.86e-06 2024-08-19T14:47:08.022 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.022 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.022 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:47:08.022 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:23.118448+0000", 2024-08-19T14:47:08.022 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1040000000000003e-06 2024-08-19T14:47:08.022 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.022 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.022 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:47:08.022 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:23.118462+0000", 2024-08-19T14:47:08.022 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4674999999999999e-05 2024-08-19T14:47:08.022 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.023 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.023 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:47:08.023 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:23.118470+0000", 2024-08-19T14:47:08.023 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1600000000000001e-06 2024-08-19T14:47:08.023 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.023 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.023 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:47:08.023 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:23.120227+0000", 2024-08-19T14:47:08.023 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001756954 2024-08-19T14:47:08.023 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.023 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.023 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:47:08.023 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:23.120383+0000", 2024-08-19T14:47:08.024 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00015682200000000001 2024-08-19T14:47:08.024 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.024 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.024 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 9,10", 2024-08-19T14:47:08.024 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:23.120515+0000", 2024-08-19T14:47:08.024 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000131383 2024-08-19T14:47:08.024 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.024 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.024 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:47:08.024 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:23.123992+0000", 2024-08-19T14:47:08.024 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0034774419999999999 2024-08-19T14:47:08.024 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.024 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.025 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:47:08.025 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:23.127960+0000", 2024-08-19T14:47:08.025 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00396789 2024-08-19T14:47:08.025 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.025 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.025 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:47:08.025 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:23.155086+0000", 2024-08-19T14:47:08.025 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.027126199 2024-08-19T14:47:08.025 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.025 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.025 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:47:08.025 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:23.219211+0000", 2024-08-19T14:47:08.025 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.064124347999999998 2024-08-19T14:47:08.025 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.026 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.026 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:47:08.026 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:23.220835+0000", 2024-08-19T14:47:08.026 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001623871 2024-08-19T14:47:08.026 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:08.026 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:47:08.026 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:08.026 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.026 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.026 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDPGPushReply(8.8 801/798 [PushReplyOp(8:13999242:::benchmark_data_smithi045_137103_object99957:head)])", 2024-08-19T14:47:08.026 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:43:26.508822+0000", 2024-08-19T14:47:08.026 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 221.49398107799999, 2024-08-19T14:47:08.027 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.11501145, 2024-08-19T14:47:08.027 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:47:08.027 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:47:08.027 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:47:08.027 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:47:08.027 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.027 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:47:08.027 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:26.508822+0000", 2024-08-19T14:47:08.027 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:08.027 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.027 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.027 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:47:08.028 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:26.508822+0000", 2024-08-19T14:47:08.028 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:08.028 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.028 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.028 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:47:08.028 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:26.508822+0000", 2024-08-19T14:47:08.028 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8100000000000003e-07 2024-08-19T14:47:08.028 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.028 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.028 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:47:08.028 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:26.508827+0000", 2024-08-19T14:47:08.028 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.065e-06 2024-08-19T14:47:08.029 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.029 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.029 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:47:08.029 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:26.508835+0000", 2024-08-19T14:47:08.029 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.0919999999999998e-06 2024-08-19T14:47:08.029 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.029 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.029 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:47:08.029 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:26.508839+0000", 2024-08-19T14:47:08.029 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1810000000000003e-06 2024-08-19T14:47:08.029 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.029 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.029 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:47:08.030 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:26.508892+0000", 2024-08-19T14:47:08.030 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2902000000000003e-05 2024-08-19T14:47:08.030 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.030 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.030 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:47:08.030 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:26.623833+0000", 2024-08-19T14:47:08.030 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.11494172900000001 2024-08-19T14:47:08.030 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:08.030 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:47:08.030 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:08.030 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.030 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.030 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.26911.0:204467 8.3 e801/731 8:c3fde17f:::benchmark_data_smithi045_137103_object92003:head v 801'8497, mlcod=801'8495)", 2024-08-19T14:47:08.031 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:43:26.509468+0000", 2024-08-19T14:47:08.031 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 221.493334944, 2024-08-19T14:47:08.031 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.114393409, 2024-08-19T14:47:08.031 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:47:08.031 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:47:08.031 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:47:08.031 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:47:08.031 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.031 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:47:08.031 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:26.509468+0000", 2024-08-19T14:47:08.031 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:08.031 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.031 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.032 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:47:08.032 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:26.509468+0000", 2024-08-19T14:47:08.032 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:08.032 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.032 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.032 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:47:08.032 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:26.509469+0000", 2024-08-19T14:47:08.032 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4099999999999999e-07 2024-08-19T14:47:08.032 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.032 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.032 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:47:08.032 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:26.509473+0000", 2024-08-19T14:47:08.032 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7890000000000002e-06 2024-08-19T14:47:08.032 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.033 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.033 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:47:08.033 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:26.509481+0000", 2024-08-19T14:47:08.033 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.379e-06 2024-08-19T14:47:08.033 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.033 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.033 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:47:08.033 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:26.509485+0000", 2024-08-19T14:47:08.033 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3309999999999997e-06 2024-08-19T14:47:08.033 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.033 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.033 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:47:08.033 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:26.509532+0000", 2024-08-19T14:47:08.033 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7372e-05 2024-08-19T14:47:08.034 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.034 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.034 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:47:08.034 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:26.509551+0000", 2024-08-19T14:47:08.034 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8887000000000001e-05 2024-08-19T14:47:08.034 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.034 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.034 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:47:08.034 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:26.623441+0000", 2024-08-19T14:47:08.034 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.113889563 2024-08-19T14:47:08.034 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.034 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.034 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:47:08.034 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:26.623862+0000", 2024-08-19T14:47:08.035 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00042064700000000001 2024-08-19T14:47:08.035 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:08.035 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:47:08.035 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:08.035 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.035 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.035 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.26911.0:204473 8.16 e801/743 8:6fd26986:::benchmark_data_smithi045_137103_object92009:head v 801'8325, mlcod=801'8323)", 2024-08-19T14:47:08.035 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:43:26.524315+0000", 2024-08-19T14:47:08.035 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 221.47848789099999, 2024-08-19T14:47:08.035 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.10284431300000001, 2024-08-19T14:47:08.035 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:47:08.035 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:47:08.035 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:47:08.037 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:47:08.037 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.037 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:47:08.037 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:26.524315+0000", 2024-08-19T14:47:08.038 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:08.038 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.038 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.038 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:47:08.038 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:26.524315+0000", 2024-08-19T14:47:08.038 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:08.038 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.038 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.038 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:47:08.038 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:26.524317+0000", 2024-08-19T14:47:08.038 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.66e-06 2024-08-19T14:47:08.038 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.038 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.039 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:47:08.039 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:26.524328+0000", 2024-08-19T14:47:08.039 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1600000000000001e-05 2024-08-19T14:47:08.039 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.039 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.039 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:47:08.039 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:26.623642+0000", 2024-08-19T14:47:08.039 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.099313296999999995 2024-08-19T14:47:08.039 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.039 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.039 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:47:08.039 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:26.623828+0000", 2024-08-19T14:47:08.040 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00018657400000000001 2024-08-19T14:47:08.040 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.040 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.040 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:47:08.040 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:26.623885+0000", 2024-08-19T14:47:08.040 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6549999999999999e-05 2024-08-19T14:47:08.040 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.040 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.040 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:47:08.040 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:26.623907+0000", 2024-08-19T14:47:08.040 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2368e-05 2024-08-19T14:47:08.040 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.040 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.040 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:47:08.041 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:26.627115+0000", 2024-08-19T14:47:08.041 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0032078509999999998 2024-08-19T14:47:08.041 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.041 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.041 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:47:08.041 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:26.627159+0000", 2024-08-19T14:47:08.041 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4413000000000003e-05 2024-08-19T14:47:08.041 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:08.041 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:47:08.041 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:08.041 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.041 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.041 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26911.0:204475 8.8 8:119f53dd:::benchmark_data_smithi045_137103_object92011:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e801)", 2024-08-19T14:47:08.041 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:43:26.527790+0000", 2024-08-19T14:47:08.041 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 221.475013303, 2024-08-19T14:47:08.042 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.12545067500000001, 2024-08-19T14:47:08.042 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:47:08.042 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:47:08.042 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:47:08.042 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:47:08.042 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26911", 2024-08-19T14:47:08.042 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3494993995", 2024-08-19T14:47:08.042 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 204475 2024-08-19T14:47:08.042 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.042 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:47:08.042 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.042 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:47:08.042 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:26.527790+0000", 2024-08-19T14:47:08.043 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:08.043 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.043 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.043 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:47:08.043 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:26.527790+0000", 2024-08-19T14:47:08.043 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:08.043 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.043 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.043 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:47:08.043 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:26.527791+0000", 2024-08-19T14:47:08.043 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0759999999999999e-06 2024-08-19T14:47:08.043 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.043 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.043 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:47:08.044 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:26.527795+0000", 2024-08-19T14:47:08.044 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3680000000000004e-06 2024-08-19T14:47:08.044 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.044 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.044 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:47:08.044 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:26.623415+0000", 2024-08-19T14:47:08.044 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.095620236999999997 2024-08-19T14:47:08.044 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.044 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.044 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:47:08.044 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:26.623648+0000", 2024-08-19T14:47:08.044 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00023223300000000001 2024-08-19T14:47:08.044 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.044 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.044 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:47:08.045 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:26.631750+0000", 2024-08-19T14:47:08.045 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0081020580000000005 2024-08-19T14:47:08.045 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.045 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.045 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:47:08.045 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:26.631886+0000", 2024-08-19T14:47:08.045 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013607499999999999 2024-08-19T14:47:08.045 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.045 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.045 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 9,10", 2024-08-19T14:47:08.045 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:26.632025+0000", 2024-08-19T14:47:08.045 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013947100000000001 2024-08-19T14:47:08.045 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.046 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.046 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:47:08.046 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:26.640744+0000", 2024-08-19T14:47:08.046 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0087185460000000006 2024-08-19T14:47:08.046 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.046 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.046 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:47:08.046 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:26.648204+0000", 2024-08-19T14:47:08.046 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0074605899999999996 2024-08-19T14:47:08.046 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.046 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.046 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:47:08.046 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:26.652477+0000", 2024-08-19T14:47:08.047 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0042728339999999997 2024-08-19T14:47:08.047 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.047 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.047 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:47:08.047 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:26.652623+0000", 2024-08-19T14:47:08.047 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014620900000000001 2024-08-19T14:47:08.047 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.047 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.047 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:47:08.047 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:26.653240+0000", 2024-08-19T14:47:08.047 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00061697800000000001 2024-08-19T14:47:08.047 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:08.047 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:47:08.047 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:08.048 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.048 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.048 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26911.0:213835 8.11 8:8deb7a71:::benchmark_data_smithi045_137103_object101371:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e806)", 2024-08-19T14:47:08.048 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:43:41.042187+0000", 2024-08-19T14:47:08.048 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 206.96061567999999, 2024-08-19T14:47:08.048 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.138051496, 2024-08-19T14:47:08.048 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:47:08.048 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:47:08.048 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:47:08.048 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:47:08.048 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26911", 2024-08-19T14:47:08.048 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3494993995", 2024-08-19T14:47:08.048 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 213835 2024-08-19T14:47:08.048 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.049 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:47:08.049 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.049 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:47:08.049 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.042187+0000", 2024-08-19T14:47:08.049 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:08.049 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.049 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.049 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:47:08.049 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.042187+0000", 2024-08-19T14:47:08.049 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:08.049 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.049 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.049 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:47:08.049 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.042189+0000", 2024-08-19T14:47:08.050 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4640000000000001e-06 2024-08-19T14:47:08.050 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.050 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.050 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:47:08.050 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.042194+0000", 2024-08-19T14:47:08.050 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0459999999999996e-06 2024-08-19T14:47:08.050 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.050 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.050 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:47:08.050 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.042205+0000", 2024-08-19T14:47:08.050 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0665e-05 2024-08-19T14:47:08.050 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.050 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.050 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:47:08.051 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.042211+0000", 2024-08-19T14:47:08.051 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4999999999999996e-06 2024-08-19T14:47:08.051 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.051 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.051 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:47:08.051 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.042261+0000", 2024-08-19T14:47:08.051 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9631e-05 2024-08-19T14:47:08.051 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.051 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.051 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:47:08.051 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.042357+0000", 2024-08-19T14:47:08.051 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.6417999999999997e-05 2024-08-19T14:47:08.051 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.052 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.052 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 9,10", 2024-08-19T14:47:08.052 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.042438+0000", 2024-08-19T14:47:08.052 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1037000000000004e-05 2024-08-19T14:47:08.052 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.052 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.052 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:47:08.052 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.049222+0000", 2024-08-19T14:47:08.052 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0067837069999999999 2024-08-19T14:47:08.052 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.052 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.052 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:47:08.052 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.065197+0000", 2024-08-19T14:47:08.052 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.015974743999999999 2024-08-19T14:47:08.053 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.053 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.053 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:47:08.053 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.179820+0000", 2024-08-19T14:47:08.053 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.114623725 2024-08-19T14:47:08.053 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.053 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.053 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:47:08.053 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.179879+0000", 2024-08-19T14:47:08.053 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8709000000000003e-05 2024-08-19T14:47:08.053 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.053 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.053 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:47:08.053 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.180239+0000", 2024-08-19T14:47:08.054 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00035984999999999999 2024-08-19T14:47:08.054 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:08.054 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:47:08.054 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:08.054 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.054 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.054 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26911.0:213837 8.11 8:8fa89269:::benchmark_data_smithi045_137103_object101373:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e806)", 2024-08-19T14:47:08.054 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:43:41.045341+0000", 2024-08-19T14:47:08.054 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 206.95746163999999, 2024-08-19T14:47:08.054 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.13516994500000001, 2024-08-19T14:47:08.054 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:47:08.054 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:47:08.054 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:47:08.055 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:47:08.055 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26911", 2024-08-19T14:47:08.055 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3494993995", 2024-08-19T14:47:08.055 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 213837 2024-08-19T14:47:08.055 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.055 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:47:08.055 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.055 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:47:08.055 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.045341+0000", 2024-08-19T14:47:08.055 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:08.055 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.055 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.055 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:47:08.055 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.045341+0000", 2024-08-19T14:47:08.056 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:08.056 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.056 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.056 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:47:08.056 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.045342+0000", 2024-08-19T14:47:08.056 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.5000000000000001e-07 2024-08-19T14:47:08.056 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.056 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.056 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:47:08.056 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.045348+0000", 2024-08-19T14:47:08.056 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8359999999999997e-06 2024-08-19T14:47:08.056 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.056 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.057 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:47:08.057 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.045362+0000", 2024-08-19T14:47:08.057 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4302e-05 2024-08-19T14:47:08.057 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.057 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.057 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:47:08.057 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.045370+0000", 2024-08-19T14:47:08.057 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.841e-06 2024-08-19T14:47:08.057 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.057 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.057 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:47:08.057 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.045725+0000", 2024-08-19T14:47:08.057 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00035418899999999998 2024-08-19T14:47:08.057 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.058 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.058 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:47:08.058 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.045857+0000", 2024-08-19T14:47:08.058 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013295700000000001 2024-08-19T14:47:08.058 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.058 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.058 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 9,10", 2024-08-19T14:47:08.058 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.046001+0000", 2024-08-19T14:47:08.058 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000143967 2024-08-19T14:47:08.058 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.058 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.058 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:47:08.058 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.049443+0000", 2024-08-19T14:47:08.058 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00344189 2024-08-19T14:47:08.059 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.059 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.059 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:47:08.059 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.179737+0000", 2024-08-19T14:47:08.059 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.13029397300000001 2024-08-19T14:47:08.059 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.059 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.059 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:47:08.059 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.180284+0000", 2024-08-19T14:47:08.059 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00054647099999999998 2024-08-19T14:47:08.059 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.059 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.059 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:47:08.060 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.180334+0000", 2024-08-19T14:47:08.060 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9863e-05 2024-08-19T14:47:08.060 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.060 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.060 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:47:08.060 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.180511+0000", 2024-08-19T14:47:08.060 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000177706 2024-08-19T14:47:08.060 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:08.060 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:47:08.060 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:08.060 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.060 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.060 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26911.0:213847 8.8 8:10ba7478:::benchmark_data_smithi045_137103_object101383:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e806)", 2024-08-19T14:47:08.061 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:43:41.052505+0000", 2024-08-19T14:47:08.061 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 206.95029854399999, 2024-08-19T14:47:08.061 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.127902971, 2024-08-19T14:47:08.061 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:47:08.061 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:47:08.061 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:47:08.061 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:47:08.061 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26911", 2024-08-19T14:47:08.061 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3494993995", 2024-08-19T14:47:08.061 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 213847 2024-08-19T14:47:08.061 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.061 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:47:08.061 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.061 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:47:08.062 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.052505+0000", 2024-08-19T14:47:08.062 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:08.062 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.062 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.062 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:47:08.062 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.052505+0000", 2024-08-19T14:47:08.062 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:08.062 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.062 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.062 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:47:08.062 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.052506+0000", 2024-08-19T14:47:08.062 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.215e-06 2024-08-19T14:47:08.062 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.062 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.063 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:47:08.063 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.052510+0000", 2024-08-19T14:47:08.063 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0369999999999996e-06 2024-08-19T14:47:08.063 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.063 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.063 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:47:08.063 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.052520+0000", 2024-08-19T14:47:08.063 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.8099999999999992e-06 2024-08-19T14:47:08.063 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.063 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.063 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:47:08.063 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.052525+0000", 2024-08-19T14:47:08.063 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3990000000000003e-06 2024-08-19T14:47:08.063 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.064 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.064 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:47:08.064 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.052577+0000", 2024-08-19T14:47:08.064 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2061999999999997e-05 2024-08-19T14:47:08.064 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.064 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.064 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:47:08.064 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.052706+0000", 2024-08-19T14:47:08.064 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012864300000000001 2024-08-19T14:47:08.064 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.064 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.064 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 9,10", 2024-08-19T14:47:08.064 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.052823+0000", 2024-08-19T14:47:08.065 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000117704 2024-08-19T14:47:08.065 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.065 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.065 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:47:08.065 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.058510+0000", 2024-08-19T14:47:08.065 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0056866570000000003 2024-08-19T14:47:08.065 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.065 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.065 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:47:08.065 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.180008+0000", 2024-08-19T14:47:08.065 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.12149813800000001 2024-08-19T14:47:08.065 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.065 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.065 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:47:08.066 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.180055+0000", 2024-08-19T14:47:08.066 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6425999999999999e-05 2024-08-19T14:47:08.066 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.066 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.066 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:47:08.066 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.180121+0000", 2024-08-19T14:47:08.066 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.648e-05 2024-08-19T14:47:08.066 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.066 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.066 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:47:08.066 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.180407+0000", 2024-08-19T14:47:08.066 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00028640000000000002 2024-08-19T14:47:08.066 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:08.067 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:47:08.067 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:08.067 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.067 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.067 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26911.0:213849 8.8 8:1287d6a6:::benchmark_data_smithi045_137103_object101385:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e806)", 2024-08-19T14:47:08.067 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:43:41.054782+0000", 2024-08-19T14:47:08.067 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 206.94802100000001, 2024-08-19T14:47:08.067 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.125901599, 2024-08-19T14:47:08.067 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:47:08.067 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:47:08.067 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:47:08.067 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:47:08.067 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26911", 2024-08-19T14:47:08.067 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3494993995", 2024-08-19T14:47:08.068 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 213849 2024-08-19T14:47:08.068 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.068 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:47:08.068 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.068 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:47:08.068 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.054782+0000", 2024-08-19T14:47:08.068 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:08.068 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.068 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.068 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:47:08.068 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.054782+0000", 2024-08-19T14:47:08.068 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:08.068 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.068 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.069 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:47:08.069 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.054783+0000", 2024-08-19T14:47:08.069 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.7599999999999996e-07 2024-08-19T14:47:08.069 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.069 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.069 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:47:08.069 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.054795+0000", 2024-08-19T14:47:08.069 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1892000000000001e-05 2024-08-19T14:47:08.069 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.069 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.069 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:47:08.069 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.054805+0000", 2024-08-19T14:47:08.069 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.7450000000000007e-06 2024-08-19T14:47:08.070 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.070 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.070 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:47:08.070 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.054809+0000", 2024-08-19T14:47:08.070 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9200000000000003e-06 2024-08-19T14:47:08.070 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.070 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.070 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:47:08.070 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.055795+0000", 2024-08-19T14:47:08.070 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00098545899999999994 2024-08-19T14:47:08.071 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.071 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.071 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:47:08.071 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.055921+0000", 2024-08-19T14:47:08.071 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012625399999999999 2024-08-19T14:47:08.071 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.071 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.071 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 9,10", 2024-08-19T14:47:08.071 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.056010+0000", 2024-08-19T14:47:08.071 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.8508e-05 2024-08-19T14:47:08.071 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.071 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.072 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:47:08.072 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.058600+0000", 2024-08-19T14:47:08.072 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0025900419999999999 2024-08-19T14:47:08.072 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.072 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.072 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:47:08.072 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.180449+0000", 2024-08-19T14:47:08.072 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.12184884 2024-08-19T14:47:08.072 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.072 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.072 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:47:08.072 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.180487+0000", 2024-08-19T14:47:08.072 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8046000000000001e-05 2024-08-19T14:47:08.073 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.073 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.073 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:47:08.073 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.180533+0000", 2024-08-19T14:47:08.073 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6626999999999999e-05 2024-08-19T14:47:08.073 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.073 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.073 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:47:08.073 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.180684+0000", 2024-08-19T14:47:08.073 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00015039 2024-08-19T14:47:08.073 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:08.073 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:47:08.073 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:08.073 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.074 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.074 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26911.0:213850 8.4 8:214dc11c:::benchmark_data_smithi045_137103_object101386:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e806)", 2024-08-19T14:47:08.074 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:43:41.057187+0000", 2024-08-19T14:47:08.074 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 206.945615879, 2024-08-19T14:47:08.074 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.123130881, 2024-08-19T14:47:08.074 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:47:08.074 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:47:08.074 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:47:08.074 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:47:08.074 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26911", 2024-08-19T14:47:08.074 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3494993995", 2024-08-19T14:47:08.074 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 213850 2024-08-19T14:47:08.074 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.075 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:47:08.075 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.075 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:47:08.075 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.057187+0000", 2024-08-19T14:47:08.075 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:08.075 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.075 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.075 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:47:08.075 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.057187+0000", 2024-08-19T14:47:08.075 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:08.075 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.075 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.075 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:47:08.076 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.057188+0000", 2024-08-19T14:47:08.076 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.7000000000000003e-07 2024-08-19T14:47:08.076 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.076 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.076 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:47:08.076 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.057193+0000", 2024-08-19T14:47:08.076 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0309999999999998e-06 2024-08-19T14:47:08.076 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.076 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.076 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:47:08.076 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.057202+0000", 2024-08-19T14:47:08.076 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.568e-06 2024-08-19T14:47:08.076 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.077 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.077 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:47:08.077 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.057207+0000", 2024-08-19T14:47:08.077 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7639999999999996e-06 2024-08-19T14:47:08.077 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.077 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.077 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:47:08.077 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.057247+0000", 2024-08-19T14:47:08.077 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0741000000000002e-05 2024-08-19T14:47:08.077 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.077 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.077 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:47:08.077 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.057329+0000", 2024-08-19T14:47:08.077 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1919e-05 2024-08-19T14:47:08.078 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.078 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.078 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 9,10", 2024-08-19T14:47:08.078 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.057401+0000", 2024-08-19T14:47:08.078 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1532999999999994e-05 2024-08-19T14:47:08.078 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.078 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.078 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:47:08.078 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.059863+0000", 2024-08-19T14:47:08.078 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0024627809999999998 2024-08-19T14:47:08.078 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.078 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.078 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:47:08.078 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.071672+0000", 2024-08-19T14:47:08.079 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011808213999999999 2024-08-19T14:47:08.079 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.079 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.079 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:47:08.079 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.180118+0000", 2024-08-19T14:47:08.079 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.108446231 2024-08-19T14:47:08.079 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.079 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.079 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:47:08.079 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.180172+0000", 2024-08-19T14:47:08.079 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3862e-05 2024-08-19T14:47:08.079 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.079 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.079 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:47:08.080 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.180318+0000", 2024-08-19T14:47:08.080 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000146267 2024-08-19T14:47:08.080 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:08.080 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:47:08.080 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:08.080 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.080 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.080 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26911.0:213854 8.11 8:8d4019d9:::benchmark_data_smithi045_137103_object101390:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e806)", 2024-08-19T14:47:08.080 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:43:41.068889+0000", 2024-08-19T14:47:08.080 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 206.933914507, 2024-08-19T14:47:08.080 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.112400208, 2024-08-19T14:47:08.080 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:47:08.080 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:47:08.081 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:47:08.081 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:47:08.081 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26911", 2024-08-19T14:47:08.081 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3494993995", 2024-08-19T14:47:08.081 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 213854 2024-08-19T14:47:08.081 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.081 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:47:08.081 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.081 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:47:08.081 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.068889+0000", 2024-08-19T14:47:08.081 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:08.081 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.081 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.081 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:47:08.082 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.068889+0000", 2024-08-19T14:47:08.082 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:08.082 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.082 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.082 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:47:08.082 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.068889+0000", 2024-08-19T14:47:08.082 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2799999999999996e-07 2024-08-19T14:47:08.082 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.082 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.082 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:47:08.082 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.068893+0000", 2024-08-19T14:47:08.082 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.41e-06 2024-08-19T14:47:08.082 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.083 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.083 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:47:08.083 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.068901+0000", 2024-08-19T14:47:08.083 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.2039999999999997e-06 2024-08-19T14:47:08.083 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.083 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.083 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:47:08.083 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.068905+0000", 2024-08-19T14:47:08.083 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6149999999999999e-06 2024-08-19T14:47:08.083 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.083 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.083 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:47:08.083 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.068956+0000", 2024-08-19T14:47:08.083 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0314e-05 2024-08-19T14:47:08.084 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.084 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.084 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:47:08.084 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.069062+0000", 2024-08-19T14:47:08.084 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00010625599999999999 2024-08-19T14:47:08.084 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.084 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.084 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 9,10", 2024-08-19T14:47:08.084 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.069133+0000", 2024-08-19T14:47:08.084 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1508999999999996e-05 2024-08-19T14:47:08.084 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.084 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.084 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:47:08.085 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.071605+0000", 2024-08-19T14:47:08.085 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0024713930000000001 2024-08-19T14:47:08.085 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.085 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.085 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:47:08.085 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.179778+0000", 2024-08-19T14:47:08.085 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.10817286199999999 2024-08-19T14:47:08.085 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.085 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.085 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:47:08.085 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.181030+0000", 2024-08-19T14:47:08.085 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0012519009999999999 2024-08-19T14:47:08.085 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.086 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.086 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:47:08.086 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.181081+0000", 2024-08-19T14:47:08.086 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1615e-05 2024-08-19T14:47:08.086 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.086 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.086 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:47:08.086 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.181289+0000", 2024-08-19T14:47:08.086 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00020750099999999999 2024-08-19T14:47:08.086 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:08.086 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:47:08.086 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:08.087 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.087 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.087 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop_reply(client.26911.0:213837 8.11 e806/797 ondisk, result = 0)", 2024-08-19T14:47:08.087 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:43:41.075721+0000", 2024-08-19T14:47:08.087 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 206.92708231699999, 2024-08-19T14:47:08.087 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.104024326, 2024-08-19T14:47:08.087 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:47:08.087 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:47:08.087 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-08-19T14:47:08.087 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:47:08.087 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.087 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:47:08.087 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.075721+0000", 2024-08-19T14:47:08.087 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:08.088 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.088 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.088 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:47:08.088 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.075721+0000", 2024-08-19T14:47:08.088 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:08.088 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.088 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.088 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:47:08.088 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.075721+0000", 2024-08-19T14:47:08.088 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0900000000000002e-07 2024-08-19T14:47:08.088 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.088 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.088 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:47:08.088 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.075725+0000", 2024-08-19T14:47:08.089 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.4400000000000001e-06 2024-08-19T14:47:08.089 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.089 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.089 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:47:08.089 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.075732+0000", 2024-08-19T14:47:08.089 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5789999999999998e-06 2024-08-19T14:47:08.089 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.089 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.089 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:47:08.089 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.075736+0000", 2024-08-19T14:47:08.089 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2089999999999999e-06 2024-08-19T14:47:08.089 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.089 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.090 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:47:08.090 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.179723+0000", 2024-08-19T14:47:08.090 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.10398642700000001 2024-08-19T14:47:08.090 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.090 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.090 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:47:08.090 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.179731+0000", 2024-08-19T14:47:08.090 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.9519999999999994e-06 2024-08-19T14:47:08.090 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.090 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.090 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:47:08.090 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:41.179745+0000", 2024-08-19T14:47:08.090 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4209999999999999e-05 2024-08-19T14:47:08.090 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:08.091 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:47:08.091 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:08.091 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.091 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.091 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.26911.0:215066 8.18 e807/764 8:1b403523:::benchmark_data_smithi045_137103_object102602:head v 807'8984, mlcod=807'8981)", 2024-08-19T14:47:08.091 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:43:42.907507+0000", 2024-08-19T14:47:08.091 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 205.095295644, 2024-08-19T14:47:08.091 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.14726671099999999, 2024-08-19T14:47:08.091 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:47:08.091 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:47:08.091 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:47:08.091 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:47:08.091 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.092 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:47:08.092 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:42.907507+0000", 2024-08-19T14:47:08.092 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:08.092 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.092 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.092 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:47:08.092 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:42.907507+0000", 2024-08-19T14:47:08.092 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:08.092 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.092 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.092 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:47:08.092 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:42.907508+0000", 2024-08-19T14:47:08.092 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4600000000000004e-07 2024-08-19T14:47:08.092 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.092 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.093 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:47:08.093 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:42.907514+0000", 2024-08-19T14:47:08.093 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7189999999999998e-06 2024-08-19T14:47:08.093 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.093 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.093 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:47:08.093 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:42.907526+0000", 2024-08-19T14:47:08.093 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1684e-05 2024-08-19T14:47:08.093 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.093 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.094 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:47:08.094 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:42.907531+0000", 2024-08-19T14:47:08.094 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0649999999999998e-06 2024-08-19T14:47:08.094 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.094 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.094 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:47:08.094 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:42.907571+0000", 2024-08-19T14:47:08.094 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0030000000000001e-05 2024-08-19T14:47:08.094 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.094 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.094 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:47:08.094 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:42.907586+0000", 2024-08-19T14:47:08.095 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5095e-05 2024-08-19T14:47:08.095 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.095 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.095 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:47:08.095 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:43.054726+0000", 2024-08-19T14:47:08.095 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.14714057 2024-08-19T14:47:08.095 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.095 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.095 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:47:08.095 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:43.054774+0000", 2024-08-19T14:47:08.095 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7802000000000001e-05 2024-08-19T14:47:08.095 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:08.095 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:47:08.096 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:08.096 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.096 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.096 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDPGRecoveryDeleteReply(8.11 e807,797 8:8cd5f3a3:::benchmark_data_smithi045_137103_object16443:head,768'6138)", 2024-08-19T14:47:08.096 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:43:42.910142+0000", 2024-08-19T14:47:08.096 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 205.09266071900001, 2024-08-19T14:47:08.096 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.14472585199999999, 2024-08-19T14:47:08.096 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:47:08.096 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:47:08.096 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:47:08.096 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:47:08.096 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.096 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:47:08.097 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:42.910142+0000", 2024-08-19T14:47:08.097 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:08.097 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.097 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.097 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:47:08.097 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:42.910142+0000", 2024-08-19T14:47:08.097 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:08.097 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.097 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.097 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:47:08.097 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:42.910144+0000", 2024-08-19T14:47:08.097 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.271e-06 2024-08-19T14:47:08.097 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.098 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.098 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:47:08.098 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:42.910149+0000", 2024-08-19T14:47:08.098 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0089999999999998e-06 2024-08-19T14:47:08.098 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.098 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.098 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:47:08.098 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:42.910158+0000", 2024-08-19T14:47:08.098 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.4890000000000001e-06 2024-08-19T14:47:08.098 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.098 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.098 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:47:08.098 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:42.910164+0000", 2024-08-19T14:47:08.099 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0039999999999996e-06 2024-08-19T14:47:08.099 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.099 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.099 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:47:08.099 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:42.910216+0000", 2024-08-19T14:47:08.099 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2201999999999999e-05 2024-08-19T14:47:08.099 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.099 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.099 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:47:08.099 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:43.054868+0000", 2024-08-19T14:47:08.099 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.14465187700000001 2024-08-19T14:47:08.099 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:08.099 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:47:08.099 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:08.100 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.100 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.100 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.26911.0:215065 8.d e807/796 8:ba5bdacb:::benchmark_data_smithi045_137103_object102601:head v 807'13514, mlcod=807'13511)", 2024-08-19T14:47:08.100 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:43:42.914765+0000", 2024-08-19T14:47:08.100 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 205.08803834299999, 2024-08-19T14:47:08.100 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.14674675500000001, 2024-08-19T14:47:08.100 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:47:08.100 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:47:08.100 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:47:08.100 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:47:08.100 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.100 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:47:08.100 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:42.914765+0000", 2024-08-19T14:47:08.101 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:08.101 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.101 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.101 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:47:08.101 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:42.914765+0000", 2024-08-19T14:47:08.101 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:08.101 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.101 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.101 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:47:08.101 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:42.914765+0000", 2024-08-19T14:47:08.101 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1200000000000003e-07 2024-08-19T14:47:08.101 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.101 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.101 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:47:08.102 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:42.914770+0000", 2024-08-19T14:47:08.102 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0799999999999996e-06 2024-08-19T14:47:08.102 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.102 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.102 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:47:08.102 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:43.054536+0000", 2024-08-19T14:47:08.102 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.13976527599999999 2024-08-19T14:47:08.102 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.102 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.102 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:47:08.102 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:43.054547+0000", 2024-08-19T14:47:08.102 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1235e-05 2024-08-19T14:47:08.102 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.103 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.103 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:47:08.103 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:43.054605+0000", 2024-08-19T14:47:08.103 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7979000000000002e-05 2024-08-19T14:47:08.103 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.103 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.103 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:47:08.103 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:43.054630+0000", 2024-08-19T14:47:08.103 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5074000000000001e-05 2024-08-19T14:47:08.103 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.103 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.103 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:47:08.103 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:43.061471+0000", 2024-08-19T14:47:08.103 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0068414390000000004 2024-08-19T14:47:08.104 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.104 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.104 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:47:08.104 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:43.061511+0000", 2024-08-19T14:47:08.104 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0059999999999999e-05 2024-08-19T14:47:08.104 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:08.104 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:47:08.104 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:08.104 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.104 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.104 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDPGRecoveryDeleteReply(8.4 e807,798 8:212f1bd2:::benchmark_data_smithi045_137103_object29107:head,768'6423)", 2024-08-19T14:47:08.104 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:43:42.914964+0000", 2024-08-19T14:47:08.104 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 205.08783874700001, 2024-08-19T14:47:08.104 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.1402487, 2024-08-19T14:47:08.105 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:47:08.105 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:47:08.105 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:47:08.105 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:47:08.105 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.105 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:47:08.105 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:42.914964+0000", 2024-08-19T14:47:08.105 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:08.105 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.105 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.105 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:47:08.105 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:42.914964+0000", 2024-08-19T14:47:08.105 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:08.105 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.106 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.106 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:47:08.106 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:42.914965+0000", 2024-08-19T14:47:08.106 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4900000000000001e-07 2024-08-19T14:47:08.106 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.106 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.106 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:47:08.106 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:42.914968+0000", 2024-08-19T14:47:08.106 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5310000000000002e-06 2024-08-19T14:47:08.106 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.106 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.106 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:47:08.106 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:43.054533+0000", 2024-08-19T14:47:08.107 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.139564823 2024-08-19T14:47:08.107 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.107 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.107 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:47:08.107 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:43.054549+0000", 2024-08-19T14:47:08.107 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5739e-05 2024-08-19T14:47:08.107 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.107 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.107 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:47:08.107 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:43.054649+0000", 2024-08-19T14:47:08.107 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000100251 2024-08-19T14:47:08.107 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.107 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.107 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:47:08.107 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:43:43.055213+0000", 2024-08-19T14:47:08.108 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00056390699999999999 2024-08-19T14:47:08.108 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:08.108 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:47:08.108 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:08.108 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.108 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.108 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDPGRecoveryDeleteReply(8.11 e818,797 8:89c97240:::benchmark_data_smithi045_137103_object54448:head,774'7379)", 2024-08-19T14:47:08.108 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:44:01.776936+0000", 2024-08-19T14:47:08.108 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 186.22586657799999, 2024-08-19T14:47:08.108 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.10370937, 2024-08-19T14:47:08.108 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:47:08.108 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:47:08.108 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:47:08.108 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:47:08.108 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.109 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:47:08.109 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:44:01.776936+0000", 2024-08-19T14:47:08.109 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:08.109 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.109 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.109 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:47:08.109 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:44:01.776936+0000", 2024-08-19T14:47:08.109 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:08.109 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:08.110 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:08.110 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:47:08.110 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:07 smithi045 ceph-mon[20143]: pgmap v2114: 47 pgs: 47 active+clean; 2.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 733 op/s; 4.3 MiB/s, 69 objects/s recovering 2024-08-19T14:47:08.111 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.111 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:47:08.111 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:44:01.776938+0000", 2024-08-19T14:47:08.111 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.42e-06 2024-08-19T14:47:08.111 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.111 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.111 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:47:08.111 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:44:01.776948+0000", 2024-08-19T14:47:08.111 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0073e-05 2024-08-19T14:47:08.111 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.111 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.111 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:47:08.112 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:44:01.776972+0000", 2024-08-19T14:47:08.112 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3816e-05 2024-08-19T14:47:08.112 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.112 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.112 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:47:08.112 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:44:01.777000+0000", 2024-08-19T14:47:08.112 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7926999999999999e-05 2024-08-19T14:47:08.112 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.112 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.112 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:47:08.112 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:44:01.777160+0000", 2024-08-19T14:47:08.113 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000159977 2024-08-19T14:47:08.113 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.113 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.113 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:47:08.113 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:44:01.880646+0000", 2024-08-19T14:47:08.113 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.103486157 2024-08-19T14:47:08.113 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:08.113 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:47:08.113 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:08.113 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.113 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.113 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDPGRecoveryDeleteReply(8.8 e818,798 8:14964ac9:::benchmark_data_smithi045_137103_object49588:head,773'7316)", 2024-08-19T14:47:08.113 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:44:01.777028+0000", 2024-08-19T14:47:08.114 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 186.22577462000001, 2024-08-19T14:47:08.114 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.10353796799999999, 2024-08-19T14:47:08.114 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:47:08.114 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:47:08.114 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:47:08.114 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:47:08.114 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.114 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:47:08.114 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:44:01.777028+0000", 2024-08-19T14:47:08.114 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:08.114 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.114 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.115 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:47:08.115 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:44:01.777028+0000", 2024-08-19T14:47:08.115 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:08.115 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.115 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.115 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:47:08.115 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:44:01.777029+0000", 2024-08-19T14:47:08.115 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7400000000000003e-07 2024-08-19T14:47:08.115 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.115 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.116 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:47:08.116 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:44:01.777035+0000", 2024-08-19T14:47:08.116 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5559999999999998e-06 2024-08-19T14:47:08.116 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.116 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.116 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:47:08.116 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:44:01.777047+0000", 2024-08-19T14:47:08.116 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2483999999999999e-05 2024-08-19T14:47:08.116 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.116 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.116 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:47:08.116 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:44:01.777056+0000", 2024-08-19T14:47:08.116 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6629999999999999e-06 2024-08-19T14:47:08.117 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.117 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.117 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:47:08.117 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:44:01.777167+0000", 2024-08-19T14:47:08.117 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00011089799999999999 2024-08-19T14:47:08.117 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:08.117 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:08.117 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:47:08.117 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:44:01.880566+0000", 2024-08-19T14:47:08.117 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.103399793 2024-08-19T14:47:08.118 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:08.118 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:47:08.118 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:08.118 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:08.118 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:47:08.118 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:47:08.210 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:47:08.233 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:08.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:08.234 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:47:08.334 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:47:08.360 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:08.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:08.361 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:47:08.462 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:47:08.484 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:08.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:08.485 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:47:08.586 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:47:08.610 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:08.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:08.611 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:47:08.712 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:47:08.737 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:08.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:08.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:47:08.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:08 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2790938575' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:47:08.839 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:47:08.861 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:08.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:08.863 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:47:08.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:08 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2790938575' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:47:08.891 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:47:08.891 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:47:08.891 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:47:08.891 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:47:08.892 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:47:08.892 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:47:08.892 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:47:08.892 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:47:08.892 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:47:08.893 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:47:08.893 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:47:08.893 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:47:08.893 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:47:08.893 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:47:08.893 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:47:08.893 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:47:08.893 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:47:08.893 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:47:08.894 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:47:08.894 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:47:08.894 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:47:08.894 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:47:08.894 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:47:08.894 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:47:08.963 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:47:08.979 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-08-19T14:47:08.986 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:08.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:08.988 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:47:09.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:08 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2790938575' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:47:09.088 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:47:09.125 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:09.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:09.126 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:47:09.227 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:47:09.262 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:47:09.263 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:09.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:09.264 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:47:09.348 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-19T14:47:09.364 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:47:09.419 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:09.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:09.420 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:47:09.521 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:47:09.545 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:09.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:09.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:47:09.647 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:47:09.664 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:47:09.670 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:09.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:09.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:47:09.683 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:09 smithi032 ceph-mon[20004]: pgmap v2115: 47 pgs: 47 active+clean; 2.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 564 op/s 2024-08-19T14:47:09.694 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:47:09.772 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:47:09.795 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:09.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:09.796 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:47:09.807 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:47:09.808 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:47:09.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:09 smithi031 ceph-mon[16336]: pgmap v2115: 47 pgs: 47 active+clean; 2.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 564 op/s 2024-08-19T14:47:09.896 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:47:09.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:09.922 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:09.923 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:47:10.023 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:47:10.046 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:10.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:09 smithi045 ceph-mon[20143]: pgmap v2115: 47 pgs: 47 active+clean; 2.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 564 op/s 2024-08-19T14:47:10.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:10.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:47:10.150 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:47:10.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:10.190 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:10.191 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:47:10.292 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:47:10.329 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:10.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:10.331 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:47:10.432 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:47:10.616 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-08-19T14:47:10.703 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:10.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:10.704 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:47:10.729 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:47:10.738 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-08-19T14:47:10.804 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:47:10.836 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:10.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:10.838 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:47:10.924 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:47:10.938 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:47:10.964 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:10.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:10.965 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:47:11.066 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:47:11.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:11.089 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:11.090 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:47:11.191 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:47:11.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:11.217 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:11.218 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:47:11.319 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:47:11.344 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:11.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:11.345 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:47:11.378 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":2116,"stamp":"2024-08-19T14:47:10.394034+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2256470584,"num_objects":34425,"num_object_clones":0,"num_object_copies":103275,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34425,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154583,"num_write_kb":6048129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39696,"num_bytes_recovered":2602574912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158892,"ondisk_log_size":158892,"up":141,"acting":141,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":141,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":13877236,"kb_used_data":6723372,"kb_used_omap":17,"kb_used_meta":7153646,"kb_avail":736083980,"statfs":{"total":767960285184,"available":753749995520,"internally_reserved":0,"allocated":6884732928,"data_stored":6863236726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18388,"internal_metadata":7325333548},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,4,5,5,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-419364864,"num_objects":-6399,"num_object_clones":0,"num_object_copies":-19197,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6399,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001516"},"pg_stats":[{"pgid":"9.8","version":"898'9685","reported_seq":13950,"reported_epoch":898,"state":"active+clean","last_fresh":"2024-08-19T14:47:07.730395+0000","last_change":"2024-08-19T14:47:00.874950+0000","last_active":"2024-08-19T14:47:07.730395+0000","last_peered":"2024-08-19T14:47:07.730395+0000","last_clean":"2024-08-19T14:47:07.730395+0000","last_became_active":"2024-08-19T14:45:07.171026+0000","last_became_peered":"2024-08-19T14:45:07.171026+0000","last_unstale":"2024-08-19T14:47:07.730395+0000","last_undegraded":"2024-08-19T14:47:07.730395+0000","last_fullsized":"2024-08-19T14:47:07.730395+0000","mapping_epoch":852,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":853,"parent":"0.0","parent_split_bits":0,"last_scrub":"895'9389","last_scrub_stamp":"2024-08-19T14:47:00.874770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:47:00.874770+0000","objects_scrubbed":2494,"log_size":9685,"log_dups_size":0,"ondisk_log_size":9685,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:48:16.373272+0000","scrub_duration":845,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":143065088,"num_objects":2183,"num_object_clones":0,"num_object_copies":6549,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9685,"num_write_kb":379776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2033,"num_bytes_recovered":133234688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,1],"acting":[11,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"70'139","reported_seq":2143,"reported_epoch":898,"state":"active+clean","last_fresh":"2024-08-19T14:47:05.576645+0000","last_change":"2024-08-19T14:45:26.108478+0000","last_active":"2024-08-19T14:47:05.576645+0000","last_peered":"2024-08-19T14:47:05.576645+0000","last_clean":"2024-08-19T14:47:05.576645+0000","last_became_active":"2024-08-19T14:45:26.107697+0000","last_became_peered":"2024-08-19T14:45:26.107697+0000","last_unstale":"2024-08-19T14:47:05.576645+0000","last_undegraded":"2024-08-19T14:47:05.576645+0000","last_fullsized":"2024-08-19T14:47:05.576645+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:26.109206+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:41:45.177828+0000","last_clean_scrub_stamp":"2024-08-19T14:44:26.109206+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6,9],"acting":[11,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.b","version":"898'9702","reported_seq":13863,"reported_epoch":898,"state":"active+clean","last_fresh":"2024-08-19T14:47:10.245748+0000","last_change":"2024-08-19T14:47:02.193403+0000","last_active":"2024-08-19T14:47:10.245748+0000","last_peered":"2024-08-19T14:47:10.245748+0000","last_clean":"2024-08-19T14:47:10.245748+0000","last_became_active":"2024-08-19T14:45:09.091550+0000","last_became_peered":"2024-08-19T14:45:09.091550+0000","last_unstale":"2024-08-19T14:47:10.245748+0000","last_undegraded":"2024-08-19T14:47:10.245748+0000","last_fullsized":"2024-08-19T14:47:10.245748+0000","mapping_epoch":854,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":0,"last_scrub":"896'9312","last_scrub_stamp":"2024-08-19T14:47:02.192892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:47:02.192892+0000","objects_scrubbed":2437,"log_size":9702,"log_dups_size":0,"ondisk_log_size":9702,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:48:19.107707+0000","scrub_duration":769,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":132120576,"num_objects":2016,"num_object_clones":0,"num_object_copies":6048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9702,"num_write_kb":374976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1981,"num_bytes_recovered":129826816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,1],"acting":[8,10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"70'139","reported_seq":1996,"reported_epoch":898,"state":"active+clean","last_fresh":"2024-08-19T14:47:05.565416+0000","last_change":"2024-08-19T14:45:26.101740+0000","last_active":"2024-08-19T14:47:05.565416+0000","last_peered":"2024-08-19T14:47:05.565416+0000","last_clean":"2024-08-19T14:47:05.565416+0000","last_became_active":"2024-08-19T14:45:26.101280+0000","last_became_peered":"2024-08-19T14:45:26.101280+0000","last_unstale":"2024-08-19T14:47:05.565416+0000","last_undegraded":"2024-08-19T14:47:05.565416+0000","last_fullsized":"2024-08-19T14:47:05.565416+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:45:20.599964+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:45:20.599964+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.a","version":"898'9789","reported_seq":22815,"reported_epoch":898,"state":"active+clean","last_fresh":"2024-08-19T14:47:09.095337+0000","last_change":"2024-08-19T14:46:58.252113+0000","last_active":"2024-08-19T14:47:09.095337+0000","last_peered":"2024-08-19T14:47:09.095337+0000","last_clean":"2024-08-19T14:47:09.095337+0000","last_became_active":"2024-08-19T14:46:35.321997+0000","last_became_peered":"2024-08-19T14:46:35.321997+0000","last_unstale":"2024-08-19T14:47:09.095337+0000","last_undegraded":"2024-08-19T14:47:09.095337+0000","last_fullsized":"2024-08-19T14:47:09.095337+0000","mapping_epoch":885,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":886,"parent":"0.0","parent_split_bits":0,"last_scrub":"893'9283","last_scrub_stamp":"2024-08-19T14:46:58.251971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:46:58.251971+0000","objects_scrubbed":2574,"log_size":9789,"log_dups_size":0,"ondisk_log_size":9789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:48:10.683585+0000","scrub_duration":524,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":135069696,"num_objects":2061,"num_object_clones":0,"num_object_copies":6183,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9789,"num_write_kb":379200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6408,"num_bytes_recovered":419954688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"70'139","reported_seq":634,"reported_epoch":898,"state":"active+clean","last_fresh":"2024-08-19T14:47:05.563734+0000","last_change":"2024-08-19T14:46:07.848793+0000","last_active":"2024-08-19T14:47:05.563734+0000","last_peered":"2024-08-19T14:47:05.563734+0000","last_clean":"2024-08-19T14:47:05.563734+0000","last_became_active":"2024-08-19T14:43:11.470639+0000","last_became_peered":"2024-08-19T14:43:11.470639+0000","last_unstale":"2024-08-19T14:47:05.563734+0000","last_undegraded":"2024-08-19T14:47:05.563734+0000","last_fullsized":"2024-08-19T14:47:05.563734+0000","mapping_epoch":795,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":796,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:46:07.848681+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:19:59.302931+0000","last_clean_scrub_stamp":"2024-08-19T14:46:07.848681+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:47:19.399442+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.9","version":"898'9583","reported_seq":22661,"reported_epoch":898,"state":"active+clean","last_fresh":"2024-08-19T14:47:05.766825+0000","last_change":"2024-08-19T14:47:01.549114+0000","last_active":"2024-08-19T14:47:05.766825+0000","last_peered":"2024-08-19T14:47:05.766825+0000","last_clean":"2024-08-19T14:47:05.766825+0000","last_became_active":"2024-08-19T14:47:01.523307+0000","last_became_peered":"2024-08-19T14:47:01.523307+0000","last_unstale":"2024-08-19T14:47:05.766825+0000","last_undegraded":"2024-08-19T14:47:05.766825+0000","last_fullsized":"2024-08-19T14:47:05.766825+0000","mapping_epoch":895,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":896,"parent":"0.0","parent_split_bits":0,"last_scrub":"860'4221","last_scrub_stamp":"2024-08-19T14:45:19.039376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:45:19.039376+0000","objects_scrubbed":4147,"log_size":9583,"log_dups_size":0,"ondisk_log_size":9583,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1783,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":148963328,"num_objects":2273,"num_object_clones":0,"num_object_copies":6819,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9583,"num_write_kb":379392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6427,"num_bytes_recovered":421199872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,6],"acting":[10,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1","version":"70'139","reported_seq":627,"reported_epoch":898,"state":"active+clean","last_fresh":"2024-08-19T14:47:05.571571+0000","last_change":"2024-08-19T14:44:32.095248+0000","last_active":"2024-08-19T14:47:05.571571+0000","last_peered":"2024-08-19T14:47:05.571571+0000","last_clean":"2024-08-19T14:47:05.571571+0000","last_became_active":"2024-08-19T14:43:16.098337+0000","last_became_peered":"2024-08-19T14:43:16.098337+0000","last_unstale":"2024-08-19T14:47:05.571571+0000","last_undegraded":"2024-08-19T14:47:05.571571+0000","last_fullsized":"2024-08-19T14:47:05.571571+0000","mapping_epoch":800,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":801,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:32.094375+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:44:32.094375+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,2],"acting":[10,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.f","version":"898'9799","reported_seq":13843,"reported_epoch":898,"state":"active+clean","last_fresh":"2024-08-19T14:47:09.050735+0000","last_change":"2024-08-19T14:45:01.147769+0000","last_active":"2024-08-19T14:47:09.050735+0000","last_peered":"2024-08-19T14:47:09.050735+0000","last_clean":"2024-08-19T14:47:09.050735+0000","last_became_active":"2024-08-19T14:45:01.143857+0000","last_became_peered":"2024-08-19T14:45:01.143857+0000","last_unstale":"2024-08-19T14:47:09.050735+0000","last_undegraded":"2024-08-19T14:47:09.050735+0000","last_fullsized":"2024-08-19T14:47:09.050735+0000","mapping_epoch":847,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":848,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":9799,"log_dups_size":0,"ondisk_log_size":9799,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:47:10.668976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":138215424,"num_objects":2109,"num_object_clones":0,"num_object_copies":6327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9799,"num_write_kb":381056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1939,"num_bytes_recovered":127074304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,11],"acting":[1,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"70'139","reported_seq":1152,"reported_epoch":898,"state":"active+clean","last_fresh":"2024-08-19T14:47:05.573389+0000","last_change":"2024-08-19T14:45:26.107836+0000","last_active":"2024-08-19T14:47:05.573389+0000","last_peered":"2024-08-19T14:47:05.573389+0000","last_clean":"2024-08-19T14:47:05.573389+0000","last_became_active":"2024-08-19T14:45:26.107494+0000","last_became_peered":"2024-08-19T14:45:26.107494+0000","last_unstale":"2024-08-19T14:47:05.573389+0000","last_undegraded":"2024-08-19T14:47:05.573389+0000","last_fullsized":"2024-08-19T14:47:05.573389+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:26.113799+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:21:46.057624+0000","last_clean_scrub_stamp":"2024-08-19T14:44:26.113799+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,2],"acting":[6,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"898'9507","reported_seq":9697,"reported_epoch":898,"state":"active+clean","last_fresh":"2024-08-19T14:47:05.767279+0000","last_change":"2024-08-19T14:47:03.596330+0000","last_active":"2024-08-19T14:47:05.767279+0000","last_peered":"2024-08-19T14:47:05.767279+0000","last_clean":"2024-08-19T14:47:05.767279+0000","last_became_active":"2024-08-19T14:44:15.414595+0000","last_became_peered":"2024-08-19T14:44:15.414595+0000","last_unstale":"2024-08-19T14:47:05.767279+0000","last_undegraded":"2024-08-19T14:47:05.767279+0000","last_fullsized":"2024-08-19T14:47:05.767279+0000","mapping_epoch":826,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":827,"parent":"0.0","parent_split_bits":0,"last_scrub":"897'9404","last_scrub_stamp":"2024-08-19T14:47:03.596123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:47:03.596123+0000","objects_scrubbed":2367,"log_size":9507,"log_dups_size":0,"ondisk_log_size":9507,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:48:04.787924+0000","scrub_duration":1579,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146079744,"num_objects":2229,"num_object_clones":0,"num_object_copies":6687,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2229,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9507,"num_write_kb":375552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,9],"acting":[10,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.6","version":"70'139","reported_seq":493,"reported_epoch":898,"state":"active+clean","last_fresh":"2024-08-19T14:47:05.572951+0000","last_change":"2024-08-19T14:44:22.089843+0000","last_active":"2024-08-19T14:47:05.572951+0000","last_peered":"2024-08-19T14:47:05.572951+0000","last_clean":"2024-08-19T14:47:05.572951+0000","last_became_active":"2024-08-19T14:43:16.091256+0000","last_became_peered":"2024-08-19T14:43:16.091256+0000","last_unstale":"2024-08-19T14:47:05.572951+0000","last_undegraded":"2024-08-19T14:47:05.572951+0000","last_fullsized":"2024-08-19T14:47:05.572951+0000","mapping_epoch":800,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":801,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:22.089766+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:27.848870+0000","last_clean_scrub_stamp":"2024-08-19T14:44:22.089766+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.d","version":"898'9569","reported_seq":13505,"reported_epoch":898,"state":"active+clean","last_fresh":"2024-08-19T14:47:08.999041+0000","last_change":"2024-08-19T14:47:03.150890+0000","last_active":"2024-08-19T14:47:08.999041+0000","last_peered":"2024-08-19T14:47:08.999041+0000","last_clean":"2024-08-19T14:47:08.999041+0000","last_became_active":"2024-08-19T14:45:00.244759+0000","last_became_peered":"2024-08-19T14:45:00.244759+0000","last_unstale":"2024-08-19T14:47:08.999041+0000","last_undegraded":"2024-08-19T14:47:08.999041+0000","last_fullsized":"2024-08-19T14:47:08.999041+0000","mapping_epoch":846,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":847,"parent":"0.0","parent_split_bits":0,"last_scrub":"896'9263","last_scrub_stamp":"2024-08-19T14:47:03.150654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:47:03.150654+0000","objects_scrubbed":2471,"log_size":9569,"log_dups_size":0,"ondisk_log_size":9569,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:48:30.484912+0000","scrub_duration":549,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":141164568,"num_objects":2155,"num_object_clones":0,"num_object_copies":6465,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2155,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9569,"num_write_kb":375105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1868,"num_bytes_recovered":122421248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"70'139","reported_seq":592,"reported_epoch":898,"state":"active+clean","last_fresh":"2024-08-19T14:47:05.571562+0000","last_change":"2024-08-19T14:45:26.106248+0000","last_active":"2024-08-19T14:47:05.571562+0000","last_peered":"2024-08-19T14:47:05.571562+0000","last_clean":"2024-08-19T14:47:05.571562+0000","last_became_active":"2024-08-19T14:45:26.105833+0000","last_became_peered":"2024-08-19T14:45:26.105833+0000","last_unstale":"2024-08-19T14:47:05.571562+0000","last_undegraded":"2024-08-19T14:47:05.571562+0000","last_fullsized":"2024-08-19T14:47:05.571562+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:29.075645+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:41:48.878043+0000","last_clean_scrub_stamp":"2024-08-19T14:44:29.075645+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.c","version":"898'9770","reported_seq":10010,"reported_epoch":898,"state":"active+clean","last_fresh":"2024-08-19T14:47:09.850994+0000","last_change":"2024-08-19T14:47:00.707547+0000","last_active":"2024-08-19T14:47:09.850994+0000","last_peered":"2024-08-19T14:47:09.850994+0000","last_clean":"2024-08-19T14:47:09.850994+0000","last_became_active":"2024-08-19T14:44:15.416278+0000","last_became_peered":"2024-08-19T14:44:15.416278+0000","last_unstale":"2024-08-19T14:47:09.850994+0000","last_undegraded":"2024-08-19T14:47:09.850994+0000","last_fullsized":"2024-08-19T14:47:09.850994+0000","mapping_epoch":826,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":827,"parent":"0.0","parent_split_bits":0,"last_scrub":"895'9336","last_scrub_stamp":"2024-08-19T14:47:00.707453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:47:00.707453+0000","objects_scrubbed":2503,"log_size":9770,"log_dups_size":0,"ondisk_log_size":9770,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:48:15.877516+0000","scrub_duration":708,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":134479872,"num_objects":2052,"num_object_clones":0,"num_object_copies":6156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2052,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9770,"num_write_kb":378304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,10],"acting":[4,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"70'139","reported_seq":1070,"reported_epoch":898,"state":"active+clean","last_fresh":"2024-08-19T14:47:05.576268+0000","last_change":"2024-08-19T14:47:03.977859+0000","last_active":"2024-08-19T14:47:05.576268+0000","last_peered":"2024-08-19T14:47:05.576268+0000","last_clean":"2024-08-19T14:47:05.576268+0000","last_became_active":"2024-08-19T14:45:26.105653+0000","last_became_peered":"2024-08-19T14:45:26.105653+0000","last_unstale":"2024-08-19T14:47:05.576268+0000","last_undegraded":"2024-08-19T14:47:05.576268+0000","last_fullsized":"2024-08-19T14:47:05.576268+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:47:03.977741+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:47:03.977741+0000","last_clean_scrub_stamp":"2024-08-19T14:47:03.977741+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:48:15.967881+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,6],"acting":[11,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.2","version":"898'9765","reported_seq":9952,"reported_epoch":898,"state":"active+clean","last_fresh":"2024-08-19T14:47:09.055584+0000","last_change":"2024-08-19T14:47:05.477981+0000","last_active":"2024-08-19T14:47:09.055584+0000","last_peered":"2024-08-19T14:47:09.055584+0000","last_clean":"2024-08-19T14:47:09.055584+0000","last_became_active":"2024-08-19T14:44:15.415137+0000","last_became_peered":"2024-08-19T14:44:15.415137+0000","last_unstale":"2024-08-19T14:47:09.055584+0000","last_undegraded":"2024-08-19T14:47:09.055584+0000","last_fullsized":"2024-08-19T14:47:09.055584+0000","mapping_epoch":826,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":827,"parent":"0.0","parent_split_bits":0,"last_scrub":"897'9595","last_scrub_stamp":"2024-08-19T14:47:05.477792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:47:05.477792+0000","objects_scrubbed":2385,"log_size":9765,"log_dups_size":0,"ondisk_log_size":9765,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:48:07.682504+0000","scrub_duration":818,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":143589376,"num_objects":2191,"num_object_clones":0,"num_object_copies":6573,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9765,"num_write_kb":382592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,11],"acting":[1,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"70'139","reported_seq":1033,"reported_epoch":898,"state":"active+clean","last_fresh":"2024-08-19T14:47:05.573090+0000","last_change":"2024-08-19T14:45:26.103609+0000","last_active":"2024-08-19T14:47:05.573090+0000","last_peered":"2024-08-19T14:47:05.573090+0000","last_clean":"2024-08-19T14:47:05.573090+0000","last_became_active":"2024-08-19T14:45:26.103142+0000","last_became_peered":"2024-08-19T14:45:26.103142+0000","last_unstale":"2024-08-19T14:47:05.573090+0000","last_undegraded":"2024-08-19T14:47:05.573090+0000","last_fullsized":"2024-08-19T14:47:05.573090+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:28.862033+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:44:28.862033+0000","last_clean_scrub_stamp":"2024-08-19T14:44:28.862033+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,4],"acting":[6,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1","version":"898'9806","reported_seq":18869,"reported_epoch":898,"state":"active+clean","last_fresh":"2024-08-19T14:47:09.589767+0000","last_change":"2024-08-19T14:47:00.690148+0000","last_active":"2024-08-19T14:47:09.589767+0000","last_peered":"2024-08-19T14:47:09.589767+0000","last_clean":"2024-08-19T14:47:09.589767+0000","last_became_active":"2024-08-19T14:46:18.202172+0000","last_became_peered":"2024-08-19T14:46:18.202172+0000","last_unstale":"2024-08-19T14:47:09.589767+0000","last_undegraded":"2024-08-19T14:47:09.589767+0000","last_fullsized":"2024-08-19T14:47:09.589767+0000","mapping_epoch":875,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":876,"parent":"0.0","parent_split_bits":0,"last_scrub":"895'9410","last_scrub_stamp":"2024-08-19T14:47:00.689843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:47:00.689843+0000","objects_scrubbed":2534,"log_size":9806,"log_dups_size":0,"ondisk_log_size":9806,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:48:20.386135+0000","scrub_duration":775,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":138936320,"num_objects":2120,"num_object_clones":0,"num_object_copies":6360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9806,"num_write_kb":381632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4433,"num_bytes_recovered":290521088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,10],"acting":[6,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"70'139","reported_seq":1427,"reported_epoch":898,"state":"active+clean","last_fresh":"2024-08-19T14:47:05.575570+0000","last_change":"2024-08-19T14:47:02.958962+0000","last_active":"2024-08-19T14:47:05.575570+0000","last_peered":"2024-08-19T14:47:05.575570+0000","last_clean":"2024-08-19T14:47:05.575570+0000","last_became_active":"2024-08-19T14:45:26.126712+0000","last_became_peered":"2024-08-19T14:45:26.126712+0000","last_unstale":"2024-08-19T14:47:05.575570+0000","last_undegraded":"2024-08-19T14:47:05.575570+0000","last_fullsized":"2024-08-19T14:47:05.575570+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:47:02.958892+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:47:02.958892+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:48:21.986084+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,2],"acting":[6,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.0","version":"898'9580","reported_seq":18411,"reported_epoch":898,"state":"active+clean","last_fresh":"2024-08-19T14:47:09.080715+0000","last_change":"2024-08-19T14:46:29.123185+0000","last_active":"2024-08-19T14:47:09.080715+0000","last_peered":"2024-08-19T14:47:09.080715+0000","last_clean":"2024-08-19T14:47:09.080715+0000","last_became_active":"2024-08-19T14:46:29.110655+0000","last_became_peered":"2024-08-19T14:46:29.110655+0000","last_unstale":"2024-08-19T14:47:09.080715+0000","last_undegraded":"2024-08-19T14:47:09.080715+0000","last_fullsized":"2024-08-19T14:47:09.080715+0000","mapping_epoch":881,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":882,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":9580,"log_dups_size":0,"ondisk_log_size":9580,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:47:10.665984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":135528448,"num_objects":2068,"num_object_clones":0,"num_object_copies":6204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9580,"num_write_kb":372736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4334,"num_bytes_recovered":284033024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,11],"acting":[1,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"70'139","reported_seq":744,"reported_epoch":898,"state":"active+clean","last_fresh":"2024-08-19T14:47:05.562434+0000","last_change":"2024-08-19T14:47:00.426142+0000","last_active":"2024-08-19T14:47:05.562434+0000","last_peered":"2024-08-19T14:47:05.562434+0000","last_clean":"2024-08-19T14:47:05.562434+0000","last_became_active":"2024-08-19T14:44:38.183219+0000","last_became_peered":"2024-08-19T14:44:38.183219+0000","last_unstale":"2024-08-19T14:47:05.562434+0000","last_undegraded":"2024-08-19T14:47:05.562434+0000","last_fullsized":"2024-08-19T14:47:05.562434+0000","mapping_epoch":837,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:47:00.426007+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:47:00.426007+0000","last_clean_scrub_stamp":"2024-08-19T14:47:00.426007+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:48:18.773485+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,4],"acting":[8,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.3","version":"898'9902","reported_seq":22910,"reported_epoch":898,"state":"active+clean","last_fresh":"2024-08-19T14:47:10.249814+0000","last_change":"2024-08-19T14:46:54.137652+0000","last_active":"2024-08-19T14:47:10.249814+0000","last_peered":"2024-08-19T14:47:10.249814+0000","last_clean":"2024-08-19T14:47:10.249814+0000","last_became_active":"2024-08-19T14:46:54.116048+0000","last_became_peered":"2024-08-19T14:46:54.116048+0000","last_unstale":"2024-08-19T14:47:10.249814+0000","last_undegraded":"2024-08-19T14:47:10.249814+0000","last_fullsized":"2024-08-19T14:47:10.249814+0000","mapping_epoch":892,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":893,"parent":"0.0","parent_split_bits":0,"last_scrub":"860'4176","last_scrub_stamp":"2024-08-19T14:45:18.741455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:45:18.741455+0000","objects_scrubbed":4146,"log_size":9902,"log_dups_size":0,"ondisk_log_size":9902,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:47:11.416342+0000","scrub_duration":1454,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":136052736,"num_objects":2076,"num_object_clones":0,"num_object_copies":6228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2076,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9902,"num_write_kb":383296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6391,"num_bytes_recovered":418840576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,6],"acting":[8,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.b","version":"70'139","reported_seq":1522,"reported_epoch":898,"state":"active+clean","last_fresh":"2024-08-19T14:47:05.576326+0000","last_change":"2024-08-19T14:47:04.980934+0000","last_active":"2024-08-19T14:47:05.576326+0000","last_peered":"2024-08-19T14:47:05.576326+0000","last_clean":"2024-08-19T14:47:05.576326+0000","last_became_active":"2024-08-19T14:44:38.183552+0000","last_became_peered":"2024-08-19T14:44:38.183552+0000","last_unstale":"2024-08-19T14:47:05.576326+0000","last_undegraded":"2024-08-19T14:47:05.576326+0000","last_fullsized":"2024-08-19T14:47:05.576326+0000","mapping_epoch":837,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:47:04.980698+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:47:04.980698+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:48:16.893489+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,1],"acting":[11,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.6","version":"898'9554","reported_seq":13644,"reported_epoch":898,"state":"active+clean","last_fresh":"2024-08-19T14:47:05.789156+0000","last_change":"2024-08-19T14:45:08.093571+0000","last_active":"2024-08-19T14:47:05.789156+0000","last_peered":"2024-08-19T14:47:05.789156+0000","last_clean":"2024-08-19T14:47:05.789156+0000","last_became_active":"2024-08-19T14:45:08.088967+0000","last_became_peered":"2024-08-19T14:45:08.088967+0000","last_unstale":"2024-08-19T14:47:05.789156+0000","last_undegraded":"2024-08-19T14:47:05.789156+0000","last_fullsized":"2024-08-19T14:47:05.789156+0000","mapping_epoch":853,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":854,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":9554,"log_dups_size":0,"ondisk_log_size":9554,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":149553152,"num_objects":2282,"num_object_clones":0,"num_object_copies":6846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2282,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9554,"num_write_kb":378752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1962,"num_bytes_recovered":128581632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,9],"acting":[10,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.e","version":"70'139","reported_seq":1712,"reported_epoch":898,"state":"active+clean","last_fresh":"2024-08-19T14:47:05.563756+0000","last_change":"2024-08-19T14:46:17.013622+0000","last_active":"2024-08-19T14:47:05.563756+0000","last_peered":"2024-08-19T14:47:05.563756+0000","last_clean":"2024-08-19T14:47:05.563756+0000","last_became_active":"2024-08-19T14:43:08.463102+0000","last_became_peered":"2024-08-19T14:43:08.463102+0000","last_unstale":"2024-08-19T14:47:05.563756+0000","last_undegraded":"2024-08-19T14:47:05.563756+0000","last_fullsized":"2024-08-19T14:47:05.563756+0000","mapping_epoch":793,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":794,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:46:17.013541+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:40:03.764672+0000","last_clean_scrub_stamp":"2024-08-19T14:46:17.013541+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:47:21.570706+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,1],"acting":[4,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.7","version":"898'9391","reported_seq":9546,"reported_epoch":898,"state":"active+clean","last_fresh":"2024-08-19T14:47:05.712235+0000","last_change":"2024-08-19T14:44:15.068003+0000","last_active":"2024-08-19T14:47:05.712235+0000","last_peered":"2024-08-19T14:47:05.712235+0000","last_clean":"2024-08-19T14:47:05.712235+0000","last_became_active":"2024-08-19T14:44:15.067635+0000","last_became_peered":"2024-08-19T14:44:15.067635+0000","last_unstale":"2024-08-19T14:47:05.712235+0000","last_undegraded":"2024-08-19T14:47:05.712235+0000","last_fullsized":"2024-08-19T14:47:05.712235+0000","mapping_epoch":826,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":827,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":9391,"log_dups_size":0,"ondisk_log_size":9391,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":149618688,"num_objects":2283,"num_object_clones":0,"num_object_copies":6849,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2283,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9391,"num_write_kb":373568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,1],"acting":[10,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.f","version":"70'139","reported_seq":1398,"reported_epoch":898,"state":"active+clean","last_fresh":"2024-08-19T14:47:05.573344+0000","last_change":"2024-08-19T14:45:26.102407+0000","last_active":"2024-08-19T14:47:05.573344+0000","last_peered":"2024-08-19T14:47:05.573344+0000","last_clean":"2024-08-19T14:47:05.573344+0000","last_became_active":"2024-08-19T14:45:26.101929+0000","last_became_peered":"2024-08-19T14:45:26.101929+0000","last_unstale":"2024-08-19T14:47:05.573344+0000","last_undegraded":"2024-08-19T14:47:05.573344+0000","last_fullsized":"2024-08-19T14:47:05.573344+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:31.800766+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:31.800766+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,8],"acting":[6,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"898'9646","reported_seq":9801,"reported_epoch":898,"state":"active+clean","last_fresh":"2024-08-19T14:47:09.087081+0000","last_change":"2024-08-19T14:44:15.066789+0000","last_active":"2024-08-19T14:47:09.087081+0000","last_peered":"2024-08-19T14:47:09.087081+0000","last_clean":"2024-08-19T14:47:09.087081+0000","last_became_active":"2024-08-19T14:44:15.065981+0000","last_became_peered":"2024-08-19T14:44:15.065981+0000","last_unstale":"2024-08-19T14:47:09.087081+0000","last_undegraded":"2024-08-19T14:47:09.087081+0000","last_fullsized":"2024-08-19T14:47:09.087081+0000","mapping_epoch":826,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":827,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":9646,"log_dups_size":0,"ondisk_log_size":9646,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:47:10.668941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":142213120,"num_objects":2170,"num_object_clones":0,"num_object_copies":6510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2170,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9646,"num_write_kb":378112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.c","version":"70'139","reported_seq":774,"reported_epoch":898,"state":"active+clean","last_fresh":"2024-08-19T14:47:05.573274+0000","last_change":"2024-08-19T14:47:00.931286+0000","last_active":"2024-08-19T14:47:05.573274+0000","last_peered":"2024-08-19T14:47:05.573274+0000","last_clean":"2024-08-19T14:47:05.573274+0000","last_became_active":"2024-08-19T14:45:26.103255+0000","last_became_peered":"2024-08-19T14:45:26.103255+0000","last_unstale":"2024-08-19T14:47:05.573274+0000","last_undegraded":"2024-08-19T14:47:05.573274+0000","last_fullsized":"2024-08-19T14:47:05.573274+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:47:00.931196+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:22:52.164586+0000","last_clean_scrub_stamp":"2024-08-19T14:47:00.931196+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:48:03.326094+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":983040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,1],"acting":[6,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"898'9535","reported_seq":13562,"reported_epoch":898,"state":"active+clean","last_fresh":"2024-08-19T14:47:07.717832+0000","last_change":"2024-08-19T14:47:01.248180+0000","last_active":"2024-08-19T14:47:07.717832+0000","last_peered":"2024-08-19T14:47:07.717832+0000","last_clean":"2024-08-19T14:47:07.717832+0000","last_became_active":"2024-08-19T14:45:06.216329+0000","last_became_peered":"2024-08-19T14:45:06.216329+0000","last_unstale":"2024-08-19T14:47:07.717832+0000","last_undegraded":"2024-08-19T14:47:07.717832+0000","last_fullsized":"2024-08-19T14:47:07.717832+0000","mapping_epoch":851,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":852,"parent":"0.0","parent_split_bits":0,"last_scrub":"895'9226","last_scrub_stamp":"2024-08-19T14:47:01.247833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:47:01.247833+0000","objects_scrubbed":2464,"log_size":9535,"log_dups_size":0,"ondisk_log_size":9535,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:48:29.262680+0000","scrub_duration":256,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":141230080,"num_objects":2155,"num_object_clones":0,"num_object_copies":6465,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2155,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9535,"num_write_kb":374080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1915,"num_bytes_recovered":125501440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,9],"acting":[11,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.d","version":"70'139","reported_seq":546,"reported_epoch":898,"state":"active+clean","last_fresh":"2024-08-19T14:47:05.563734+0000","last_change":"2024-08-19T14:46:20.095131+0000","last_active":"2024-08-19T14:47:05.563734+0000","last_peered":"2024-08-19T14:47:05.563734+0000","last_clean":"2024-08-19T14:47:05.563734+0000","last_became_active":"2024-08-19T14:43:08.459577+0000","last_became_peered":"2024-08-19T14:43:08.459577+0000","last_unstale":"2024-08-19T14:47:05.563734+0000","last_undegraded":"2024-08-19T14:47:05.563734+0000","last_fullsized":"2024-08-19T14:47:05.563734+0000","mapping_epoch":793,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":794,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:46:20.095018+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:24:50.276408+0000","last_clean_scrub_stamp":"2024-08-19T14:46:20.095018+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:47:31.007933+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,8],"acting":[4,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.12","version":"70'139","reported_seq":1496,"reported_epoch":898,"state":"active+clean","last_fresh":"2024-08-19T14:47:05.564805+0000","last_change":"2024-08-19T14:45:26.109914+0000","last_active":"2024-08-19T14:47:05.564805+0000","last_peered":"2024-08-19T14:47:05.564805+0000","last_clean":"2024-08-19T14:47:05.564805+0000","last_became_active":"2024-08-19T14:45:26.109383+0000","last_became_peered":"2024-08-19T14:45:26.109383+0000","last_unstale":"2024-08-19T14:47:05.564805+0000","last_undegraded":"2024-08-19T14:47:05.564805+0000","last_fullsized":"2024-08-19T14:47:05.564805+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:30.604277+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:41:49.815826+0000","last_clean_scrub_stamp":"2024-08-19T14:44:30.604277+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,11],"acting":[1,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.13","version":"70'139","reported_seq":2029,"reported_epoch":898,"state":"active+clean","last_fresh":"2024-08-19T14:47:05.564753+0000","last_change":"2024-08-19T14:46:53.695559+0000","last_active":"2024-08-19T14:47:05.564753+0000","last_peered":"2024-08-19T14:47:05.564753+0000","last_clean":"2024-08-19T14:47:05.564753+0000","last_became_active":"2024-08-19T14:43:16.086968+0000","last_became_peered":"2024-08-19T14:43:16.086968+0000","last_unstale":"2024-08-19T14:47:05.564753+0000","last_undegraded":"2024-08-19T14:47:05.564753+0000","last_fullsized":"2024-08-19T14:47:05.564753+0000","mapping_epoch":800,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":801,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:46:53.695474+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:33:35.786105+0000","last_clean_scrub_stamp":"2024-08-19T14:46:53.695474+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:48:23.005145+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.10","version":"70'139","reported_seq":962,"reported_epoch":898,"state":"active+clean","last_fresh":"2024-08-19T14:47:05.571384+0000","last_change":"2024-08-19T14:44:38.187460+0000","last_active":"2024-08-19T14:47:05.571384+0000","last_peered":"2024-08-19T14:47:05.571384+0000","last_clean":"2024-08-19T14:47:05.571384+0000","last_became_active":"2024-08-19T14:44:38.187068+0000","last_became_peered":"2024-08-19T14:44:38.187068+0000","last_unstale":"2024-08-19T14:47:05.571384+0000","last_undegraded":"2024-08-19T14:47:05.571384+0000","last_fullsized":"2024-08-19T14:47:05.571384+0000","mapping_epoch":837,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:24.090078+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:34:58.616646+0000","last_clean_scrub_stamp":"2024-08-19T14:44:24.090078+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,4],"acting":[10,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.11","version":"70'139","reported_seq":551,"reported_epoch":898,"state":"active+clean","last_fresh":"2024-08-19T14:47:05.562806+0000","last_change":"2024-08-19T14:47:05.395106+0000","last_active":"2024-08-19T14:47:05.562806+0000","last_peered":"2024-08-19T14:47:05.562806+0000","last_clean":"2024-08-19T14:47:05.562806+0000","last_became_active":"2024-08-19T14:42:39.064403+0000","last_became_peered":"2024-08-19T14:42:39.064403+0000","last_unstale":"2024-08-19T14:47:05.562806+0000","last_undegraded":"2024-08-19T14:47:05.562806+0000","last_fullsized":"2024-08-19T14:47:05.562806+0000","mapping_epoch":778,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":779,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:47:05.394984+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:41:47.147376+0000","last_clean_scrub_stamp":"2024-08-19T14:47:05.394984+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:48:22.791670+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,1],"acting":[8,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"70'139","reported_seq":1656,"reported_epoch":898,"state":"active+clean","last_fresh":"2024-08-19T14:47:05.565098+0000","last_change":"2024-08-19T14:46:55.739107+0000","last_active":"2024-08-19T14:47:05.565098+0000","last_peered":"2024-08-19T14:47:05.565098+0000","last_clean":"2024-08-19T14:47:05.565098+0000","last_became_active":"2024-08-19T14:44:38.183608+0000","last_became_peered":"2024-08-19T14:44:38.183608+0000","last_unstale":"2024-08-19T14:47:05.565098+0000","last_undegraded":"2024-08-19T14:47:05.565098+0000","last_fullsized":"2024-08-19T14:47:05.565098+0000","mapping_epoch":837,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:46:55.739034+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:28:34.216370+0000","last_clean_scrub_stamp":"2024-08-19T14:46:55.739034+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:48:22.771380+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,4],"acting":[1,11,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"70'139","reported_seq":1806,"reported_epoch":898,"state":"active+clean","last_fresh":"2024-08-19T14:47:05.564159+0000","last_change":"2024-08-19T14:46:56.367773+0000","last_active":"2024-08-19T14:47:05.564159+0000","last_peered":"2024-08-19T14:47:05.564159+0000","last_clean":"2024-08-19T14:47:05.564159+0000","last_became_active":"2024-08-19T14:44:38.183371+0000","last_became_peered":"2024-08-19T14:44:38.183371+0000","last_unstale":"2024-08-19T14:47:05.564159+0000","last_undegraded":"2024-08-19T14:47:05.564159+0000","last_fullsized":"2024-08-19T14:47:05.564159+0000","mapping_epoch":837,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:46:56.367650+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:46:56.367650+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:48:22.698349+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11],"acting":[8,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.14","version":"70'139","reported_seq":1212,"reported_epoch":898,"state":"active+clean","last_fresh":"2024-08-19T14:47:05.565201+0000","last_change":"2024-08-19T14:46:34.796943+0000","last_active":"2024-08-19T14:47:05.565201+0000","last_peered":"2024-08-19T14:47:05.565201+0000","last_clean":"2024-08-19T14:47:05.565201+0000","last_became_active":"2024-08-19T14:43:08.454350+0000","last_became_peered":"2024-08-19T14:43:08.454350+0000","last_unstale":"2024-08-19T14:47:05.565201+0000","last_undegraded":"2024-08-19T14:47:05.565201+0000","last_fullsized":"2024-08-19T14:47:05.565201+0000","mapping_epoch":793,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":794,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:46:34.796863+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:46:34.796863+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:48:00.974974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,9],"acting":[1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"70'139","reported_seq":557,"reported_epoch":898,"state":"active+clean","last_fresh":"2024-08-19T14:47:05.564898+0000","last_change":"2024-08-19T14:46:54.433540+0000","last_active":"2024-08-19T14:47:05.564898+0000","last_peered":"2024-08-19T14:47:05.564898+0000","last_clean":"2024-08-19T14:47:05.564898+0000","last_became_active":"2024-08-19T14:44:38.199514+0000","last_became_peered":"2024-08-19T14:44:38.199514+0000","last_unstale":"2024-08-19T14:47:05.564898+0000","last_undegraded":"2024-08-19T14:47:05.564898+0000","last_fullsized":"2024-08-19T14:47:05.564898+0000","mapping_epoch":837,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:46:54.431820+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:20:09.180923+0000","last_clean_scrub_stamp":"2024-08-19T14:46:54.431820+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:48:09.163994+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,11],"acting":[8,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"70'139","reported_seq":1647,"reported_epoch":898,"state":"active+clean","last_fresh":"2024-08-19T14:47:05.563612+0000","last_change":"2024-08-19T14:46:15.973722+0000","last_active":"2024-08-19T14:47:05.563612+0000","last_peered":"2024-08-19T14:47:05.563612+0000","last_clean":"2024-08-19T14:47:05.563612+0000","last_became_active":"2024-08-19T14:43:11.471324+0000","last_became_peered":"2024-08-19T14:43:11.471324+0000","last_unstale":"2024-08-19T14:47:05.563612+0000","last_undegraded":"2024-08-19T14:47:05.563612+0000","last_fullsized":"2024-08-19T14:47:05.563612+0000","mapping_epoch":795,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":796,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:46:15.973642+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:41:51.832284+0000","last_clean_scrub_stamp":"2024-08-19T14:46:15.973642+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:47:25.118982+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,11],"acting":[4,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1b","version":"70'139","reported_seq":2029,"reported_epoch":898,"state":"active+clean","last_fresh":"2024-08-19T14:47:05.562655+0000","last_change":"2024-08-19T14:46:58.388681+0000","last_active":"2024-08-19T14:47:05.562655+0000","last_peered":"2024-08-19T14:47:05.562655+0000","last_clean":"2024-08-19T14:47:05.562655+0000","last_became_active":"2024-08-19T14:42:39.065604+0000","last_became_peered":"2024-08-19T14:42:39.065604+0000","last_unstale":"2024-08-19T14:47:05.562655+0000","last_undegraded":"2024-08-19T14:47:05.562655+0000","last_fullsized":"2024-08-19T14:47:05.562655+0000","mapping_epoch":778,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":779,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:46:58.388563+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:37:59.510697+0000","last_clean_scrub_stamp":"2024-08-19T14:46:58.388563+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:48:21.771287+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,2],"acting":[8,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.18","version":"70'139","reported_seq":1017,"reported_epoch":898,"state":"active+clean","last_fresh":"2024-08-19T14:47:05.564911+0000","last_change":"2024-08-19T14:46:54.743821+0000","last_active":"2024-08-19T14:47:05.564911+0000","last_peered":"2024-08-19T14:47:05.564911+0000","last_clean":"2024-08-19T14:47:05.564911+0000","last_became_active":"2024-08-19T14:44:38.182712+0000","last_became_peered":"2024-08-19T14:44:38.182712+0000","last_unstale":"2024-08-19T14:47:05.564911+0000","last_undegraded":"2024-08-19T14:47:05.564911+0000","last_fullsized":"2024-08-19T14:47:05.564911+0000","mapping_epoch":837,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:46:54.743702+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:27:33.084161+0000","last_clean_scrub_stamp":"2024-08-19T14:46:54.743702+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:48:17.549331+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":402496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,11],"acting":[1,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.19","version":"70'139","reported_seq":874,"reported_epoch":898,"state":"active+clean","last_fresh":"2024-08-19T14:47:05.571672+0000","last_change":"2024-08-19T14:45:26.148752+0000","last_active":"2024-08-19T14:47:05.571672+0000","last_peered":"2024-08-19T14:47:05.571672+0000","last_clean":"2024-08-19T14:47:05.571672+0000","last_became_active":"2024-08-19T14:45:26.147921+0000","last_became_peered":"2024-08-19T14:45:26.147921+0000","last_unstale":"2024-08-19T14:47:05.571672+0000","last_undegraded":"2024-08-19T14:47:05.571672+0000","last_fullsized":"2024-08-19T14:47:05.571672+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:23.083891+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:44:23.083891+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,4],"acting":[10,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1e","version":"70'139","reported_seq":1035,"reported_epoch":898,"state":"active+clean","last_fresh":"2024-08-19T14:47:05.576490+0000","last_change":"2024-08-19T14:45:16.381698+0000","last_active":"2024-08-19T14:47:05.576490+0000","last_peered":"2024-08-19T14:47:05.576490+0000","last_clean":"2024-08-19T14:47:05.576490+0000","last_became_active":"2024-08-19T14:43:16.124258+0000","last_became_peered":"2024-08-19T14:43:16.124258+0000","last_unstale":"2024-08-19T14:47:05.576490+0000","last_undegraded":"2024-08-19T14:47:05.576490+0000","last_fullsized":"2024-08-19T14:47:05.576490+0000","mapping_epoch":800,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":801,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:45:16.381587+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:27:04.780939+0000","last_clean_scrub_stamp":"2024-08-19T14:45:16.381587+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,10],"acting":[11,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1c","version":"70'139","reported_seq":1783,"reported_epoch":898,"state":"active+clean","last_fresh":"2024-08-19T14:47:05.562540+0000","last_change":"2024-08-19T14:45:26.106081+0000","last_active":"2024-08-19T14:47:05.562540+0000","last_peered":"2024-08-19T14:47:05.562540+0000","last_clean":"2024-08-19T14:47:05.562540+0000","last_became_active":"2024-08-19T14:45:26.105382+0000","last_became_peered":"2024-08-19T14:45:26.105382+0000","last_unstale":"2024-08-19T14:47:05.562540+0000","last_undegraded":"2024-08-19T14:47:05.562540+0000","last_fullsized":"2024-08-19T14:47:05.562540+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:31.796263+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:44:31.796263+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1d","version":"70'139","reported_seq":1675,"reported_epoch":898,"state":"active+clean","last_fresh":"2024-08-19T14:47:05.565977+0000","last_change":"2024-08-19T14:44:29.650415+0000","last_active":"2024-08-19T14:47:05.565977+0000","last_peered":"2024-08-19T14:47:05.565977+0000","last_clean":"2024-08-19T14:47:05.565977+0000","last_became_active":"2024-08-19T14:35:11.540488+0000","last_became_peered":"2024-08-19T14:35:11.540488+0000","last_unstale":"2024-08-19T14:47:05.565977+0000","last_undegraded":"2024-08-19T14:47:05.565977+0000","last_fullsized":"2024-08-19T14:47:05.565977+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:29.650238+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:44:29.650238+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":2255880216,"num_objects":34423,"num_object_clones":0,"num_object_copies":103269,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34423,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154583,"num_write_kb":6048129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39691,"num_bytes_recovered":2601189376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6899249152,"data_stored":6899236936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154583,"ondisk_log_size":154583,"up":48,"acting":48,"num_store_stats":9},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1385536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2785280,"data_stored":2771072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187885,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2073432,"kb_used_data":862668,"kb_used_omap":1,"kb_used_meta":1210750,"kb_avail":91671720,"statfs":{"total":95995035648,"available":93871841280,"internally_reserved":0,"allocated":883372032,"data_stored":880683250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":1239808452},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":539,"seq":2314987372752,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1248388,"kb_used_data":1040120,"kb_used_omap":4,"kb_used_meta":208251,"kb_avail":92496764,"statfs":{"total":95995035648,"available":94716686336,"internally_reserved":0,"allocated":1065082880,"data_stored":1062401905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4407,"internal_metadata":213249737},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":74,"seq":317827580343,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2471628,"kb_used_data":1242052,"kb_used_omap":1,"kb_used_meta":1229502,"kb_avail":91273524,"statfs":{"total":95995035648,"available":93464088576,"internally_reserved":0,"allocated":1271861248,"data_stored":1269176018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":1259010497},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874686,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1744848,"kb_used_data":680776,"kb_used_omap":1,"kb_used_meta":1064062,"kb_avail":92000304,"statfs":{"total":95995035648,"available":94208311296,"internally_reserved":0,"allocated":697114624,"data_stored":694425322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1089599941},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":0,"seq":244813136321,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430661,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":855500,"kb_used_data":684100,"kb_used_omap":1,"kb_used_meta":171390,"kb_avail":92889652,"statfs":{"total":95995035648,"available":95119003648,"internally_reserved":0,"allocated":700518400,"data_stored":697833170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":175503814},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":171798692296,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986638,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1660048,"kb_used_data":410952,"kb_used_omap":1,"kb_used_meta":1249086,"kb_avail":92085104,"statfs":{"total":95995035648,"available":94295146496,"internally_reserved":0,"allocated":420814848,"data_stored":418125546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1279064517},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":111669150162,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":634,"seq":2723009265808,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":707080,"kb_used_data":157184,"kb_used_omap":4,"kb_used_meta":549883,"kb_avail":93038072,"statfs":{"total":95995035648,"available":95270985728,"internally_reserved":0,"allocated":160956416,"data_stored":158271889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4407,"internal_metadata":563080905},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575324,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3116312,"kb_used_data":1645520,"kb_used_omap":1,"kb_used_meta":1470718,"kb_avail":90628840,"statfs":{"total":95995035648,"available":92803932160,"internally_reserved":0,"allocated":1685012480,"data_stored":1682319626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1506015685},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":34359738847,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1658982400,"data_stored":1658978328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":150274048,"data_stored":150274048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":412684288,"data_stored":412680216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":169803776,"data_stored":169803776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":681115648,"data_stored":681115648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":675418112,"data_stored":675414040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1254359040,"data_stored":1254359040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1038680064,"data_stored":1038680064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":857931776,"data_stored":857931776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T14:47:11.446 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:47:11.471 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:11.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:11.473 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:47:11.552 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:47:11.552 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:47:11.574 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:47:11.586 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is set 2024-08-19T14:47:11.602 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:11.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:11.603 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:47:11.639 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:47:11.704 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:47:11.744 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:11.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:11.746 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:47:11.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:11 smithi032 ceph-mon[20004]: pgmap v2116: 47 pgs: 47 active+clean; 2.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 799 op/s 2024-08-19T14:47:11.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:11 smithi032 ceph-mon[20004]: from='client.18147 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:47:11.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:11 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/272632636' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:47:11.846 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:47:11.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:11 smithi031 ceph-mon[16336]: pgmap v2116: 47 pgs: 47 active+clean; 2.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 799 op/s 2024-08-19T14:47:11.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:11 smithi031 ceph-mon[16336]: from='client.18147 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:47:11.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:11 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/272632636' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:47:11.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:11.879 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:11.880 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:47:11.981 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:47:12.019 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:12.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:12.020 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:47:12.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:11 smithi045 ceph-mon[20143]: pgmap v2116: 47 pgs: 47 active+clean; 2.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 799 op/s 2024-08-19T14:47:12.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:11 smithi045 ceph-mon[20143]: from='client.18147 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:47:12.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:11 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/272632636' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:47:12.121 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:47:12.144 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:12.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:12.145 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:47:12.246 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:47:12.255 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-08-19T14:47:12.294 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:12.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:12.295 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:47:12.387 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:47:12.395 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:47:12.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:12.431 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:12.431 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:47:12.532 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:47:12.538 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:47:12.556 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:12.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:12.557 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:47:12.659 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:47:12.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:12.698 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:12.699 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:47:12.799 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:47:12.824 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:12.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:12.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:47:12.847 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:12 smithi032 ceph-mon[20004]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:47:12.847 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:12 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/272632636' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:47:12.848 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:12 smithi032 ceph-mon[20004]: osdmap e899: 12 total, 12 up, 8 in 2024-08-19T14:47:12.850 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:12 smithi031 ceph-mon[16336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:47:12.850 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:12 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/272632636' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:47:12.850 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:12 smithi031 ceph-mon[16336]: osdmap e899: 12 total, 12 up, 8 in 2024-08-19T14:47:12.926 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:47:12.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:12.964 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:12.965 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:47:12.968 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:47:12.968 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:47:12.990 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:47:12.990 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:47:13.008 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:47:13.008 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:47:13.026 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:47:13.026 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:47:13.045 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:47:13.045 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:47:13.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:12 smithi045 ceph-mon[20143]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:47:13.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:12 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/272632636' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:47:13.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:12 smithi045 ceph-mon[20143]: osdmap e899: 12 total, 12 up, 8 in 2024-08-19T14:47:13.066 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:47:13.069 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:47:13.069 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:47:13.082 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:47:13.082 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:47:13.091 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:13.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:13.092 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:47:13.096 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:47:13.096 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:47:13.119 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:47:13.120 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:47:13.152 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:47:13.152 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:47:13.178 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:47:13.179 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:47:13.194 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:47:13.204 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:47:13.204 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:47:13.216 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:13.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:13.217 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:47:13.237 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:47:13.238 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:47:13.318 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:47:13.342 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:13.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:13.343 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:47:13.444 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:47:13.480 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:13.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:13.481 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:47:13.582 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:47:13.605 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:13.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:13.606 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:47:13.708 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:47:13.753 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:13.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:13.754 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:47:13.803 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-08-19T14:47:13.855 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:47:13.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:13 smithi031 ceph-mon[16336]: pgmap v2118: 47 pgs: 47 active+clean; 2.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 722 op/s 2024-08-19T14:47:13.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:13 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2959386832' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:47:13.889 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:13.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:13.890 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:47:13.991 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:47:14.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:13 smithi032 ceph-mon[20004]: pgmap v2118: 47 pgs: 47 active+clean; 2.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 722 op/s 2024-08-19T14:47:14.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:13 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2959386832' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:47:14.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:14.016 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:14.017 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:47:14.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:13 smithi045 ceph-mon[20143]: pgmap v2118: 47 pgs: 47 active+clean; 2.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 722 op/s 2024-08-19T14:47:14.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:13 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2959386832' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:47:14.082 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:47:14.118 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:47:14.143 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:14.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:14.144 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:47:14.245 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:47:14.268 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:14.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:14.269 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:47:14.286 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-19T14:47:14.370 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:47:14.381 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-19T14:47:14.407 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:14.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:14.408 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:47:14.509 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:47:14.542 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:14.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:14.544 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:47:14.595 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:47:14.596 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-08-19T14:47:14.596 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-08-19T14:47:14.596 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:47:14.606 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:14 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:47:14.607 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:14 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:47:14.607 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:14 smithi031 ceph-mon[16336]: osdmap e900: 12 total, 12 up, 8 in 2024-08-19T14:47:14.628 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:47:14.645 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:47:14.676 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:14.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:14.677 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:47:14.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:14 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:47:14.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:14 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:47:14.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:14 smithi032 ceph-mon[20004]: osdmap e900: 12 total, 12 up, 8 in 2024-08-19T14:47:14.778 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:47:14.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:14.822 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:14.822 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:47:14.923 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:47:14.966 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:14.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:14.968 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:47:15.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:14 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:47:15.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:14 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:47:15.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:14 smithi045 ceph-mon[20143]: osdmap e900: 12 total, 12 up, 8 in 2024-08-19T14:47:15.069 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:47:15.103 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:15.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:15.104 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:47:15.205 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:47:15.227 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:15.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:15.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:47:15.273 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-08-19T14:47:15.329 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:47:15.353 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:15.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:15.354 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:47:15.357 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:47:15.455 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:47:15.479 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:15.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:15.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:47:15.560 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:47:15.581 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:47:15.607 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is unset 2024-08-19T14:47:15.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:15.608 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:15.608 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:47:15.709 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:47:15.731 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:15.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:15.733 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:47:15.793 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:15 smithi031 ceph-mon[16336]: pgmap v2120: 47 pgs: 47 active+clean; 1.9 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 756 op/s 2024-08-19T14:47:15.793 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:15 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:47:15.793 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:15 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:47:15.793 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:15 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3245766640' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:47:15.833 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:47:15.875 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:15.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:15.877 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:47:15.917 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:47:15.948 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:15 smithi032 ceph-mon[20004]: pgmap v2120: 47 pgs: 47 active+clean; 1.9 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 756 op/s 2024-08-19T14:47:15.948 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:15 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:47:15.948 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:15 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:47:15.948 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:15 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3245766640' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:47:15.977 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:47:15.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:16.000 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:16.000 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:47:16.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:15 smithi045 ceph-mon[20143]: pgmap v2120: 47 pgs: 47 active+clean; 1.9 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 756 op/s 2024-08-19T14:47:16.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:15 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:47:16.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:15 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:47:16.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:15 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3245766640' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:47:16.101 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:47:16.143 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:16.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:16.144 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:47:16.194 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:47:16.195 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-08-19T14:47:16.195 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-08-19T14:47:16.195 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-08-19T14:47:16.195 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:47:16.244 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:47:16.279 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:16.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:16.280 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:47:16.381 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:47:16.422 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:47:16.423 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-08-19T14:47:16.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:16.426 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:16.427 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:47:16.528 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:47:16.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:16.550 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:16.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:47:16.652 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:47:16.675 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:16.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:16.676 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:47:16.777 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:47:16.800 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:16.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:16.801 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:47:16.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:16 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3245766640' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:47:16.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:16 smithi031 ceph-mon[16336]: osdmap e901: 12 total, 12 up, 8 in 2024-08-19T14:47:16.902 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:47:16.936 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:16.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:16.937 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:47:17.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:16 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3245766640' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:47:17.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:16 smithi032 ceph-mon[20004]: osdmap e901: 12 total, 12 up, 8 in 2024-08-19T14:47:17.038 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:47:17.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:16 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3245766640' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:47:17.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:16 smithi045 ceph-mon[20143]: osdmap e901: 12 total, 12 up, 8 in 2024-08-19T14:47:17.063 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:17.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:17.064 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:47:17.096 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-08-19T14:47:17.165 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:47:17.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:17.200 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:17.201 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:47:17.301 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:47:17.328 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:17.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:17.329 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:47:17.381 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:47:17.430 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:47:17.451 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:17.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:17.452 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:47:17.553 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:47:17.575 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:17.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:17.576 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:47:17.673 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":2122,"stamp":"2024-08-19T14:47:16.395210+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1928462904,"num_objects":29420,"num_object_clones":0,"num_object_copies":88260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29420,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":159588,"num_write_kb":6048129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39696,"num_bytes_recovered":2602574912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":163697,"ondisk_log_size":163697,"up":141,"acting":141,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":141,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":13030292,"kb_used_data":5829228,"kb_used_omap":17,"kb_used_meta":7200814,"kb_avail":736930924,"statfs":{"total":767960285184,"available":754617266176,"internally_reserved":0,"allocated":5969129472,"data_stored":5947565490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18388,"internal_metadata":7373633580},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,5,2,4,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-328007680,"num_objects":-5005,"num_object_clones":0,"num_object_copies":-15015,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5005,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5005,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001176"},"pg_stats":[{"pgid":"9.8","version":"899'9925","reported_seq":14192,"reported_epoch":899,"state":"active+clean","last_fresh":"2024-08-19T14:47:12.764365+0000","last_change":"2024-08-19T14:47:00.874950+0000","last_active":"2024-08-19T14:47:12.764365+0000","last_peered":"2024-08-19T14:47:12.764365+0000","last_clean":"2024-08-19T14:47:12.764365+0000","last_became_active":"2024-08-19T14:45:07.171026+0000","last_became_peered":"2024-08-19T14:45:07.171026+0000","last_unstale":"2024-08-19T14:47:12.764365+0000","last_undegraded":"2024-08-19T14:47:12.764365+0000","last_fullsized":"2024-08-19T14:47:12.764365+0000","mapping_epoch":852,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":853,"parent":"0.0","parent_split_bits":0,"last_scrub":"895'9389","last_scrub_stamp":"2024-08-19T14:47:00.874770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:47:00.874770+0000","objects_scrubbed":2494,"log_size":9925,"log_dups_size":0,"ondisk_log_size":9925,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:48:16.373272+0000","scrub_duration":845,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":127336448,"num_objects":1943,"num_object_clones":0,"num_object_copies":5829,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1943,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9925,"num_write_kb":379776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2033,"num_bytes_recovered":133234688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,1],"acting":[11,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"70'139","reported_seq":2145,"reported_epoch":899,"state":"active+clean","last_fresh":"2024-08-19T14:47:11.600241+0000","last_change":"2024-08-19T14:45:26.108478+0000","last_active":"2024-08-19T14:47:11.600241+0000","last_peered":"2024-08-19T14:47:11.600241+0000","last_clean":"2024-08-19T14:47:11.600241+0000","last_became_active":"2024-08-19T14:45:26.107697+0000","last_became_peered":"2024-08-19T14:45:26.107697+0000","last_unstale":"2024-08-19T14:47:11.600241+0000","last_undegraded":"2024-08-19T14:47:11.600241+0000","last_fullsized":"2024-08-19T14:47:11.600241+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:26.109206+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:41:45.177828+0000","last_clean_scrub_stamp":"2024-08-19T14:44:26.109206+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:47:14.977475+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6,9],"acting":[11,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.b","version":"900'9945","reported_seq":14110,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-08-19T14:47:15.253461+0000","last_change":"2024-08-19T14:47:02.193403+0000","last_active":"2024-08-19T14:47:15.253461+0000","last_peered":"2024-08-19T14:47:15.253461+0000","last_clean":"2024-08-19T14:47:15.253461+0000","last_became_active":"2024-08-19T14:45:09.091550+0000","last_became_peered":"2024-08-19T14:45:09.091550+0000","last_unstale":"2024-08-19T14:47:15.253461+0000","last_undegraded":"2024-08-19T14:47:15.253461+0000","last_fullsized":"2024-08-19T14:47:15.253461+0000","mapping_epoch":854,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":0,"last_scrub":"896'9312","last_scrub_stamp":"2024-08-19T14:47:02.192892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:47:02.192892+0000","objects_scrubbed":2437,"log_size":9945,"log_dups_size":0,"ondisk_log_size":9945,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:48:19.107707+0000","scrub_duration":769,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":116195328,"num_objects":1773,"num_object_clones":0,"num_object_copies":5319,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1773,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9945,"num_write_kb":374976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1981,"num_bytes_recovered":129826816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,1],"acting":[8,10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"70'139","reported_seq":2000,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-08-19T14:47:13.602928+0000","last_change":"2024-08-19T14:45:26.101740+0000","last_active":"2024-08-19T14:47:13.602928+0000","last_peered":"2024-08-19T14:47:13.602928+0000","last_clean":"2024-08-19T14:47:13.602928+0000","last_became_active":"2024-08-19T14:45:26.101280+0000","last_became_peered":"2024-08-19T14:45:26.101280+0000","last_unstale":"2024-08-19T14:47:13.602928+0000","last_undegraded":"2024-08-19T14:47:13.602928+0000","last_fullsized":"2024-08-19T14:47:13.602928+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:45:20.599964+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:45:20.599964+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:47:15.673337+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.a","version":"900'10024","reported_seq":23054,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-08-19T14:47:14.000925+0000","last_change":"2024-08-19T14:46:58.252113+0000","last_active":"2024-08-19T14:47:14.000925+0000","last_peered":"2024-08-19T14:47:14.000925+0000","last_clean":"2024-08-19T14:47:14.000925+0000","last_became_active":"2024-08-19T14:46:35.321997+0000","last_became_peered":"2024-08-19T14:46:35.321997+0000","last_unstale":"2024-08-19T14:47:14.000925+0000","last_undegraded":"2024-08-19T14:47:14.000925+0000","last_fullsized":"2024-08-19T14:47:14.000925+0000","mapping_epoch":885,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":886,"parent":"0.0","parent_split_bits":0,"last_scrub":"893'9283","last_scrub_stamp":"2024-08-19T14:46:58.251971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:46:58.251971+0000","objects_scrubbed":2574,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:48:10.683585+0000","scrub_duration":524,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":119668736,"num_objects":1826,"num_object_clones":0,"num_object_copies":5478,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1826,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10024,"num_write_kb":379200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6408,"num_bytes_recovered":419954688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"70'139","reported_seq":638,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-08-19T14:47:13.597505+0000","last_change":"2024-08-19T14:46:07.848793+0000","last_active":"2024-08-19T14:47:13.597505+0000","last_peered":"2024-08-19T14:47:13.597505+0000","last_clean":"2024-08-19T14:47:13.597505+0000","last_became_active":"2024-08-19T14:43:11.470639+0000","last_became_peered":"2024-08-19T14:43:11.470639+0000","last_unstale":"2024-08-19T14:47:13.597505+0000","last_undegraded":"2024-08-19T14:47:13.597505+0000","last_fullsized":"2024-08-19T14:47:13.597505+0000","mapping_epoch":795,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":796,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:46:07.848681+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:19:59.302931+0000","last_clean_scrub_stamp":"2024-08-19T14:46:07.848681+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:47:19.399442+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.9","version":"901'10104","reported_seq":23188,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-08-19T14:47:15.893038+0000","last_change":"2024-08-19T14:47:01.549114+0000","last_active":"2024-08-19T14:47:15.893038+0000","last_peered":"2024-08-19T14:47:15.893038+0000","last_clean":"2024-08-19T14:47:15.893038+0000","last_became_active":"2024-08-19T14:47:01.523307+0000","last_became_peered":"2024-08-19T14:47:01.523307+0000","last_unstale":"2024-08-19T14:47:15.893038+0000","last_undegraded":"2024-08-19T14:47:15.893038+0000","last_fullsized":"2024-08-19T14:47:15.893038+0000","mapping_epoch":895,"log_start":"830'100","ondisk_log_start":"830'100","created":826,"last_epoch_clean":896,"parent":"0.0","parent_split_bits":0,"last_scrub":"860'4221","last_scrub_stamp":"2024-08-19T14:45:19.039376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:45:19.039376+0000","objects_scrubbed":4147,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:47:19.896631+0000","scrub_duration":1783,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":114819072,"num_objects":1752,"num_object_clones":0,"num_object_copies":5256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10104,"num_write_kb":379392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6427,"num_bytes_recovered":421199872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,6],"acting":[10,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1","version":"70'139","reported_seq":633,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-08-19T14:47:15.619280+0000","last_change":"2024-08-19T14:44:32.095248+0000","last_active":"2024-08-19T14:47:15.619280+0000","last_peered":"2024-08-19T14:47:15.619280+0000","last_clean":"2024-08-19T14:47:15.619280+0000","last_became_active":"2024-08-19T14:43:16.098337+0000","last_became_peered":"2024-08-19T14:43:16.098337+0000","last_unstale":"2024-08-19T14:47:15.619280+0000","last_undegraded":"2024-08-19T14:47:15.619280+0000","last_fullsized":"2024-08-19T14:47:15.619280+0000","mapping_epoch":800,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":801,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:32.094375+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:44:32.094375+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:47:19.895747+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,2],"acting":[10,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.f","version":"900'10057","reported_seq":14105,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-08-19T14:47:14.080138+0000","last_change":"2024-08-19T14:45:01.147769+0000","last_active":"2024-08-19T14:47:14.080138+0000","last_peered":"2024-08-19T14:47:14.080138+0000","last_clean":"2024-08-19T14:47:14.080138+0000","last_became_active":"2024-08-19T14:45:01.143857+0000","last_became_peered":"2024-08-19T14:45:01.143857+0000","last_unstale":"2024-08-19T14:47:14.080138+0000","last_undegraded":"2024-08-19T14:47:14.080138+0000","last_fullsized":"2024-08-19T14:47:14.080138+0000","mapping_epoch":847,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":848,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:47:15.673267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121307136,"num_objects":1851,"num_object_clones":0,"num_object_copies":5553,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1851,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10057,"num_write_kb":381056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1939,"num_bytes_recovered":127074304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,11],"acting":[1,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"70'139","reported_seq":1156,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-08-19T14:47:13.601412+0000","last_change":"2024-08-19T14:45:26.107836+0000","last_active":"2024-08-19T14:47:13.601412+0000","last_peered":"2024-08-19T14:47:13.601412+0000","last_clean":"2024-08-19T14:47:13.601412+0000","last_became_active":"2024-08-19T14:45:26.107494+0000","last_became_peered":"2024-08-19T14:45:26.107494+0000","last_unstale":"2024-08-19T14:47:13.601412+0000","last_undegraded":"2024-08-19T14:47:13.601412+0000","last_fullsized":"2024-08-19T14:47:13.601412+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:26.113799+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:21:46.057624+0000","last_clean_scrub_stamp":"2024-08-19T14:44:26.113799+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,2],"acting":[6,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"901'10009","reported_seq":10205,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-08-19T14:47:15.839061+0000","last_change":"2024-08-19T14:47:03.596330+0000","last_active":"2024-08-19T14:47:15.839061+0000","last_peered":"2024-08-19T14:47:15.839061+0000","last_clean":"2024-08-19T14:47:15.839061+0000","last_became_active":"2024-08-19T14:44:15.414595+0000","last_became_peered":"2024-08-19T14:44:15.414595+0000","last_unstale":"2024-08-19T14:47:15.839061+0000","last_undegraded":"2024-08-19T14:47:15.839061+0000","last_fullsized":"2024-08-19T14:47:15.839061+0000","mapping_epoch":826,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":827,"parent":"0.0","parent_split_bits":0,"last_scrub":"897'9404","last_scrub_stamp":"2024-08-19T14:47:03.596123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:47:03.596123+0000","objects_scrubbed":2367,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:48:04.787924+0000","scrub_duration":1579,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113180672,"num_objects":1727,"num_object_clones":0,"num_object_copies":5181,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1727,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10009,"num_write_kb":375552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,9],"acting":[10,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.6","version":"70'139","reported_seq":499,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-08-19T14:47:15.614960+0000","last_change":"2024-08-19T14:44:22.089843+0000","last_active":"2024-08-19T14:47:15.614960+0000","last_peered":"2024-08-19T14:47:15.614960+0000","last_clean":"2024-08-19T14:47:15.614960+0000","last_became_active":"2024-08-19T14:43:16.091256+0000","last_became_peered":"2024-08-19T14:43:16.091256+0000","last_unstale":"2024-08-19T14:47:15.614960+0000","last_undegraded":"2024-08-19T14:47:15.614960+0000","last_fullsized":"2024-08-19T14:47:15.614960+0000","mapping_epoch":800,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":801,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:22.089766+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:27.848870+0000","last_clean_scrub_stamp":"2024-08-19T14:44:22.089766+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:47:19.896533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.d","version":"900'9817","reported_seq":13756,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-08-19T14:47:14.092596+0000","last_change":"2024-08-19T14:47:03.150890+0000","last_active":"2024-08-19T14:47:14.092596+0000","last_peered":"2024-08-19T14:47:14.092596+0000","last_clean":"2024-08-19T14:47:14.092596+0000","last_became_active":"2024-08-19T14:45:00.244759+0000","last_became_peered":"2024-08-19T14:45:00.244759+0000","last_unstale":"2024-08-19T14:47:14.092596+0000","last_undegraded":"2024-08-19T14:47:14.092596+0000","last_fullsized":"2024-08-19T14:47:14.092596+0000","mapping_epoch":846,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":847,"parent":"0.0","parent_split_bits":0,"last_scrub":"896'9263","last_scrub_stamp":"2024-08-19T14:47:03.150654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:47:03.150654+0000","objects_scrubbed":2471,"log_size":9817,"log_dups_size":0,"ondisk_log_size":9817,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:48:30.484912+0000","scrub_duration":549,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":124911640,"num_objects":1907,"num_object_clones":0,"num_object_copies":5721,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1907,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9817,"num_write_kb":375105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1868,"num_bytes_recovered":122421248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"70'139","reported_seq":598,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-08-19T14:47:15.614445+0000","last_change":"2024-08-19T14:45:26.106248+0000","last_active":"2024-08-19T14:47:15.614445+0000","last_peered":"2024-08-19T14:47:15.614445+0000","last_clean":"2024-08-19T14:47:15.614445+0000","last_became_active":"2024-08-19T14:45:26.105833+0000","last_became_peered":"2024-08-19T14:45:26.105833+0000","last_unstale":"2024-08-19T14:47:15.614445+0000","last_undegraded":"2024-08-19T14:47:15.614445+0000","last_fullsized":"2024-08-19T14:47:15.614445+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:29.075645+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:41:48.878043+0000","last_clean_scrub_stamp":"2024-08-19T14:44:29.075645+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:47:19.887052+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.c","version":"900'10017","reported_seq":10261,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-08-19T14:47:14.949668+0000","last_change":"2024-08-19T14:47:00.707547+0000","last_active":"2024-08-19T14:47:14.949668+0000","last_peered":"2024-08-19T14:47:14.949668+0000","last_clean":"2024-08-19T14:47:14.949668+0000","last_became_active":"2024-08-19T14:44:15.416278+0000","last_became_peered":"2024-08-19T14:44:15.416278+0000","last_unstale":"2024-08-19T14:47:14.949668+0000","last_undegraded":"2024-08-19T14:47:14.949668+0000","last_fullsized":"2024-08-19T14:47:14.949668+0000","mapping_epoch":826,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":827,"parent":"0.0","parent_split_bits":0,"last_scrub":"895'9336","last_scrub_stamp":"2024-08-19T14:47:00.707453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:47:00.707453+0000","objects_scrubbed":2503,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:48:15.877516+0000","scrub_duration":708,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":118292480,"num_objects":1805,"num_object_clones":0,"num_object_copies":5415,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1805,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10017,"num_write_kb":378304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,10],"acting":[4,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"70'139","reported_seq":1072,"reported_epoch":899,"state":"active+clean","last_fresh":"2024-08-19T14:47:11.599817+0000","last_change":"2024-08-19T14:47:03.977859+0000","last_active":"2024-08-19T14:47:11.599817+0000","last_peered":"2024-08-19T14:47:11.599817+0000","last_clean":"2024-08-19T14:47:11.599817+0000","last_became_active":"2024-08-19T14:45:26.105653+0000","last_became_peered":"2024-08-19T14:45:26.105653+0000","last_unstale":"2024-08-19T14:47:11.599817+0000","last_undegraded":"2024-08-19T14:47:11.599817+0000","last_fullsized":"2024-08-19T14:47:11.599817+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:47:03.977741+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:47:03.977741+0000","last_clean_scrub_stamp":"2024-08-19T14:47:03.977741+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:48:15.967881+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,6],"acting":[11,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.2","version":"900'10016","reported_seq":10207,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-08-19T14:47:14.068140+0000","last_change":"2024-08-19T14:47:05.477981+0000","last_active":"2024-08-19T14:47:14.068140+0000","last_peered":"2024-08-19T14:47:14.068140+0000","last_clean":"2024-08-19T14:47:14.068140+0000","last_became_active":"2024-08-19T14:44:15.415137+0000","last_became_peered":"2024-08-19T14:44:15.415137+0000","last_unstale":"2024-08-19T14:47:14.068140+0000","last_undegraded":"2024-08-19T14:47:14.068140+0000","last_fullsized":"2024-08-19T14:47:14.068140+0000","mapping_epoch":826,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":827,"parent":"0.0","parent_split_bits":0,"last_scrub":"897'9595","last_scrub_stamp":"2024-08-19T14:47:05.477792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:47:05.477792+0000","objects_scrubbed":2385,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:48:07.682504+0000","scrub_duration":818,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":127139840,"num_objects":1940,"num_object_clones":0,"num_object_copies":5820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1940,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10016,"num_write_kb":382592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,11],"acting":[1,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"70'139","reported_seq":1037,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-08-19T14:47:13.601271+0000","last_change":"2024-08-19T14:45:26.103609+0000","last_active":"2024-08-19T14:47:13.601271+0000","last_peered":"2024-08-19T14:47:13.601271+0000","last_clean":"2024-08-19T14:47:13.601271+0000","last_became_active":"2024-08-19T14:45:26.103142+0000","last_became_peered":"2024-08-19T14:45:26.103142+0000","last_unstale":"2024-08-19T14:47:13.601271+0000","last_undegraded":"2024-08-19T14:47:13.601271+0000","last_fullsized":"2024-08-19T14:47:13.601271+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:28.862033+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:44:28.862033+0000","last_clean_scrub_stamp":"2024-08-19T14:44:28.862033+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,4],"acting":[6,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1","version":"900'10074","reported_seq":19141,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-08-19T14:47:14.619594+0000","last_change":"2024-08-19T14:47:00.690148+0000","last_active":"2024-08-19T14:47:14.619594+0000","last_peered":"2024-08-19T14:47:14.619594+0000","last_clean":"2024-08-19T14:47:14.619594+0000","last_became_active":"2024-08-19T14:46:18.202172+0000","last_became_peered":"2024-08-19T14:46:18.202172+0000","last_unstale":"2024-08-19T14:47:14.619594+0000","last_undegraded":"2024-08-19T14:47:14.619594+0000","last_fullsized":"2024-08-19T14:47:14.619594+0000","mapping_epoch":875,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":876,"parent":"0.0","parent_split_bits":0,"last_scrub":"895'9410","last_scrub_stamp":"2024-08-19T14:47:00.689843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:47:00.689843+0000","objects_scrubbed":2534,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:48:20.386135+0000","scrub_duration":775,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121372672,"num_objects":1852,"num_object_clones":0,"num_object_copies":5556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10074,"num_write_kb":381632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4433,"num_bytes_recovered":290521088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,10],"acting":[6,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"70'139","reported_seq":1431,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-08-19T14:47:13.601253+0000","last_change":"2024-08-19T14:47:02.958962+0000","last_active":"2024-08-19T14:47:13.601253+0000","last_peered":"2024-08-19T14:47:13.601253+0000","last_clean":"2024-08-19T14:47:13.601253+0000","last_became_active":"2024-08-19T14:45:26.126712+0000","last_became_peered":"2024-08-19T14:45:26.126712+0000","last_unstale":"2024-08-19T14:47:13.601253+0000","last_undegraded":"2024-08-19T14:47:13.601253+0000","last_fullsized":"2024-08-19T14:47:13.601253+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:47:02.958892+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:47:02.958892+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:48:21.986084+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,2],"acting":[6,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.0","version":"900'9815","reported_seq":18651,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-08-19T14:47:14.060363+0000","last_change":"2024-08-19T14:46:29.123185+0000","last_active":"2024-08-19T14:47:14.060363+0000","last_peered":"2024-08-19T14:47:14.060363+0000","last_clean":"2024-08-19T14:47:14.060363+0000","last_became_active":"2024-08-19T14:46:29.110655+0000","last_became_peered":"2024-08-19T14:46:29.110655+0000","last_unstale":"2024-08-19T14:47:14.060363+0000","last_undegraded":"2024-08-19T14:47:14.060363+0000","last_fullsized":"2024-08-19T14:47:14.060363+0000","mapping_epoch":881,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":882,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":9815,"log_dups_size":0,"ondisk_log_size":9815,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:47:15.669405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":120127488,"num_objects":1833,"num_object_clones":0,"num_object_copies":5499,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1833,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9815,"num_write_kb":372736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4334,"num_bytes_recovered":284033024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,11],"acting":[1,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"70'139","reported_seq":748,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-08-19T14:47:13.597424+0000","last_change":"2024-08-19T14:47:00.426142+0000","last_active":"2024-08-19T14:47:13.597424+0000","last_peered":"2024-08-19T14:47:13.597424+0000","last_clean":"2024-08-19T14:47:13.597424+0000","last_became_active":"2024-08-19T14:44:38.183219+0000","last_became_peered":"2024-08-19T14:44:38.183219+0000","last_unstale":"2024-08-19T14:47:13.597424+0000","last_undegraded":"2024-08-19T14:47:13.597424+0000","last_fullsized":"2024-08-19T14:47:13.597424+0000","mapping_epoch":837,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:47:00.426007+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:47:00.426007+0000","last_clean_scrub_stamp":"2024-08-19T14:47:00.426007+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:48:18.773485+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,4],"acting":[8,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.3","version":"900'10131","reported_seq":23143,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-08-19T14:47:15.187024+0000","last_change":"2024-08-19T14:46:54.137652+0000","last_active":"2024-08-19T14:47:15.187024+0000","last_peered":"2024-08-19T14:47:15.187024+0000","last_clean":"2024-08-19T14:47:15.187024+0000","last_became_active":"2024-08-19T14:46:54.116048+0000","last_became_peered":"2024-08-19T14:46:54.116048+0000","last_unstale":"2024-08-19T14:47:15.187024+0000","last_undegraded":"2024-08-19T14:47:15.187024+0000","last_fullsized":"2024-08-19T14:47:15.187024+0000","mapping_epoch":892,"log_start":"830'100","ondisk_log_start":"830'100","created":826,"last_epoch_clean":893,"parent":"0.0","parent_split_bits":0,"last_scrub":"860'4176","last_scrub_stamp":"2024-08-19T14:45:18.741455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:45:18.741455+0000","objects_scrubbed":4146,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:47:17.388413+0000","scrub_duration":1454,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121044992,"num_objects":1847,"num_object_clones":0,"num_object_copies":5541,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1847,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10131,"num_write_kb":383296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6391,"num_bytes_recovered":418840576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,6],"acting":[8,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.b","version":"70'139","reported_seq":1524,"reported_epoch":899,"state":"active+clean","last_fresh":"2024-08-19T14:47:11.599477+0000","last_change":"2024-08-19T14:47:04.980934+0000","last_active":"2024-08-19T14:47:11.599477+0000","last_peered":"2024-08-19T14:47:11.599477+0000","last_clean":"2024-08-19T14:47:11.599477+0000","last_became_active":"2024-08-19T14:44:38.183552+0000","last_became_peered":"2024-08-19T14:44:38.183552+0000","last_unstale":"2024-08-19T14:47:11.599477+0000","last_undegraded":"2024-08-19T14:47:11.599477+0000","last_fullsized":"2024-08-19T14:47:11.599477+0000","mapping_epoch":837,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:47:04.980698+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:47:04.980698+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:48:16.893489+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,1],"acting":[11,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.6","version":"901'10063","reported_seq":14159,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-08-19T14:47:15.888422+0000","last_change":"2024-08-19T14:45:08.093571+0000","last_active":"2024-08-19T14:47:15.888422+0000","last_peered":"2024-08-19T14:47:15.888422+0000","last_clean":"2024-08-19T14:47:15.888422+0000","last_became_active":"2024-08-19T14:45:08.088967+0000","last_became_peered":"2024-08-19T14:45:08.088967+0000","last_unstale":"2024-08-19T14:47:15.888422+0000","last_undegraded":"2024-08-19T14:47:15.888422+0000","last_fullsized":"2024-08-19T14:47:15.888422+0000","mapping_epoch":853,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":854,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:47:19.895683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":116195328,"num_objects":1773,"num_object_clones":0,"num_object_copies":5319,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1773,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10063,"num_write_kb":378752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1962,"num_bytes_recovered":128581632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,9],"acting":[10,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.e","version":"70'139","reported_seq":1716,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-08-19T14:47:13.597823+0000","last_change":"2024-08-19T14:46:17.013622+0000","last_active":"2024-08-19T14:47:13.597823+0000","last_peered":"2024-08-19T14:47:13.597823+0000","last_clean":"2024-08-19T14:47:13.597823+0000","last_became_active":"2024-08-19T14:43:08.463102+0000","last_became_peered":"2024-08-19T14:43:08.463102+0000","last_unstale":"2024-08-19T14:47:13.597823+0000","last_undegraded":"2024-08-19T14:47:13.597823+0000","last_fullsized":"2024-08-19T14:47:13.597823+0000","mapping_epoch":793,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":794,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:46:17.013541+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:40:03.764672+0000","last_clean_scrub_stamp":"2024-08-19T14:46:17.013541+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:47:21.570706+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,1],"acting":[4,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.7","version":"901'9880","reported_seq":10040,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-08-19T14:47:15.892760+0000","last_change":"2024-08-19T14:44:15.068003+0000","last_active":"2024-08-19T14:47:15.892760+0000","last_peered":"2024-08-19T14:47:15.892760+0000","last_clean":"2024-08-19T14:47:15.892760+0000","last_became_active":"2024-08-19T14:44:15.067635+0000","last_became_peered":"2024-08-19T14:44:15.067635+0000","last_unstale":"2024-08-19T14:47:15.892760+0000","last_undegraded":"2024-08-19T14:47:15.892760+0000","last_fullsized":"2024-08-19T14:47:15.892760+0000","mapping_epoch":826,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":827,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":9880,"log_dups_size":0,"ondisk_log_size":9880,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:47:19.895718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117571584,"num_objects":1794,"num_object_clones":0,"num_object_copies":5382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1794,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9880,"num_write_kb":373568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,1],"acting":[10,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.f","version":"70'139","reported_seq":1402,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-08-19T14:47:13.601344+0000","last_change":"2024-08-19T14:45:26.102407+0000","last_active":"2024-08-19T14:47:13.601344+0000","last_peered":"2024-08-19T14:47:13.601344+0000","last_clean":"2024-08-19T14:47:13.601344+0000","last_became_active":"2024-08-19T14:45:26.101929+0000","last_became_peered":"2024-08-19T14:45:26.101929+0000","last_unstale":"2024-08-19T14:47:13.601344+0000","last_undegraded":"2024-08-19T14:47:13.601344+0000","last_fullsized":"2024-08-19T14:47:13.601344+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:31.800766+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:31.800766+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,8],"acting":[6,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"900'9871","reported_seq":10030,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-08-19T14:47:14.056601+0000","last_change":"2024-08-19T14:44:15.066789+0000","last_active":"2024-08-19T14:47:14.056601+0000","last_peered":"2024-08-19T14:47:14.056601+0000","last_clean":"2024-08-19T14:47:14.056601+0000","last_became_active":"2024-08-19T14:44:15.065981+0000","last_became_peered":"2024-08-19T14:44:15.065981+0000","last_unstale":"2024-08-19T14:47:14.056601+0000","last_undegraded":"2024-08-19T14:47:14.056601+0000","last_fullsized":"2024-08-19T14:47:14.056601+0000","mapping_epoch":826,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":827,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:44:14.042834+0000","objects_scrubbed":0,"log_size":9871,"log_dups_size":0,"ondisk_log_size":9871,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:47:15.673225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":127467520,"num_objects":1945,"num_object_clones":0,"num_object_copies":5835,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1945,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9871,"num_write_kb":378112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.c","version":"70'139","reported_seq":778,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-08-19T14:47:13.601376+0000","last_change":"2024-08-19T14:47:00.931286+0000","last_active":"2024-08-19T14:47:13.601376+0000","last_peered":"2024-08-19T14:47:13.601376+0000","last_clean":"2024-08-19T14:47:13.601376+0000","last_became_active":"2024-08-19T14:45:26.103255+0000","last_became_peered":"2024-08-19T14:45:26.103255+0000","last_unstale":"2024-08-19T14:47:13.601376+0000","last_undegraded":"2024-08-19T14:47:13.601376+0000","last_fullsized":"2024-08-19T14:47:13.601376+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:47:00.931196+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:22:52.164586+0000","last_clean_scrub_stamp":"2024-08-19T14:47:00.931196+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:48:03.326094+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":983040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,1],"acting":[6,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"899'9840","reported_seq":13869,"reported_epoch":899,"state":"active+clean","last_fresh":"2024-08-19T14:47:12.713735+0000","last_change":"2024-08-19T14:47:01.248180+0000","last_active":"2024-08-19T14:47:12.713735+0000","last_peered":"2024-08-19T14:47:12.713735+0000","last_clean":"2024-08-19T14:47:12.713735+0000","last_became_active":"2024-08-19T14:45:06.216329+0000","last_became_peered":"2024-08-19T14:45:06.216329+0000","last_unstale":"2024-08-19T14:47:12.713735+0000","last_undegraded":"2024-08-19T14:47:12.713735+0000","last_fullsized":"2024-08-19T14:47:12.713735+0000","mapping_epoch":851,"log_start":"0'0","ondisk_log_start":"0'0","created":826,"last_epoch_clean":852,"parent":"0.0","parent_split_bits":0,"last_scrub":"895'9226","last_scrub_stamp":"2024-08-19T14:47:01.247833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T14:44:14.042834+0000","last_clean_scrub_stamp":"2024-08-19T14:47:01.247833+0000","objects_scrubbed":2464,"log_size":9840,"log_dups_size":0,"ondisk_log_size":9840,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:48:29.262680+0000","scrub_duration":256,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121241600,"num_objects":1850,"num_object_clones":0,"num_object_copies":5550,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9840,"num_write_kb":374080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1915,"num_bytes_recovered":125501440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,9],"acting":[11,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.d","version":"70'139","reported_seq":550,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-08-19T14:47:13.597755+0000","last_change":"2024-08-19T14:46:20.095131+0000","last_active":"2024-08-19T14:47:13.597755+0000","last_peered":"2024-08-19T14:47:13.597755+0000","last_clean":"2024-08-19T14:47:13.597755+0000","last_became_active":"2024-08-19T14:43:08.459577+0000","last_became_peered":"2024-08-19T14:43:08.459577+0000","last_unstale":"2024-08-19T14:47:13.597755+0000","last_undegraded":"2024-08-19T14:47:13.597755+0000","last_fullsized":"2024-08-19T14:47:13.597755+0000","mapping_epoch":793,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":794,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:46:20.095018+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:24:50.276408+0000","last_clean_scrub_stamp":"2024-08-19T14:46:20.095018+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:47:31.007933+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,8],"acting":[4,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.12","version":"70'139","reported_seq":1500,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-08-19T14:47:13.603120+0000","last_change":"2024-08-19T14:45:26.109914+0000","last_active":"2024-08-19T14:47:13.603120+0000","last_peered":"2024-08-19T14:47:13.603120+0000","last_clean":"2024-08-19T14:47:13.603120+0000","last_became_active":"2024-08-19T14:45:26.109383+0000","last_became_peered":"2024-08-19T14:45:26.109383+0000","last_unstale":"2024-08-19T14:47:13.603120+0000","last_undegraded":"2024-08-19T14:47:13.603120+0000","last_fullsized":"2024-08-19T14:47:13.603120+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:30.604277+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:41:49.815826+0000","last_clean_scrub_stamp":"2024-08-19T14:44:30.604277+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:47:15.673303+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,11],"acting":[1,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.13","version":"70'139","reported_seq":2033,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-08-19T14:47:13.602809+0000","last_change":"2024-08-19T14:46:53.695559+0000","last_active":"2024-08-19T14:47:13.602809+0000","last_peered":"2024-08-19T14:47:13.602809+0000","last_clean":"2024-08-19T14:47:13.602809+0000","last_became_active":"2024-08-19T14:43:16.086968+0000","last_became_peered":"2024-08-19T14:43:16.086968+0000","last_unstale":"2024-08-19T14:47:13.602809+0000","last_undegraded":"2024-08-19T14:47:13.602809+0000","last_fullsized":"2024-08-19T14:47:13.602809+0000","mapping_epoch":800,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":801,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:46:53.695474+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:33:35.786105+0000","last_clean_scrub_stamp":"2024-08-19T14:46:53.695474+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:48:23.005145+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.10","version":"70'139","reported_seq":968,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-08-19T14:47:15.614048+0000","last_change":"2024-08-19T14:44:38.187460+0000","last_active":"2024-08-19T14:47:15.614048+0000","last_peered":"2024-08-19T14:47:15.614048+0000","last_clean":"2024-08-19T14:47:15.614048+0000","last_became_active":"2024-08-19T14:44:38.187068+0000","last_became_peered":"2024-08-19T14:44:38.187068+0000","last_unstale":"2024-08-19T14:47:15.614048+0000","last_undegraded":"2024-08-19T14:47:15.614048+0000","last_fullsized":"2024-08-19T14:47:15.614048+0000","mapping_epoch":837,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:24.090078+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:34:58.616646+0000","last_clean_scrub_stamp":"2024-08-19T14:44:24.090078+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:47:19.896565+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,4],"acting":[10,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.11","version":"70'139","reported_seq":556,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-08-19T14:47:13.597655+0000","last_change":"2024-08-19T14:47:05.395106+0000","last_active":"2024-08-19T14:47:13.597655+0000","last_peered":"2024-08-19T14:47:13.597655+0000","last_clean":"2024-08-19T14:47:13.597655+0000","last_became_active":"2024-08-19T14:42:39.064403+0000","last_became_peered":"2024-08-19T14:42:39.064403+0000","last_unstale":"2024-08-19T14:47:13.597655+0000","last_undegraded":"2024-08-19T14:47:13.597655+0000","last_fullsized":"2024-08-19T14:47:13.597655+0000","mapping_epoch":778,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":779,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:47:05.394984+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:41:47.147376+0000","last_clean_scrub_stamp":"2024-08-19T14:47:05.394984+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:48:22.791670+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,1],"acting":[8,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"70'139","reported_seq":1660,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-08-19T14:47:13.607503+0000","last_change":"2024-08-19T14:46:55.739107+0000","last_active":"2024-08-19T14:47:13.607503+0000","last_peered":"2024-08-19T14:47:13.607503+0000","last_clean":"2024-08-19T14:47:13.607503+0000","last_became_active":"2024-08-19T14:44:38.183608+0000","last_became_peered":"2024-08-19T14:44:38.183608+0000","last_unstale":"2024-08-19T14:47:13.607503+0000","last_undegraded":"2024-08-19T14:47:13.607503+0000","last_fullsized":"2024-08-19T14:47:13.607503+0000","mapping_epoch":837,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:46:55.739034+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:28:34.216370+0000","last_clean_scrub_stamp":"2024-08-19T14:46:55.739034+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:48:22.771380+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,4],"acting":[1,11,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"70'139","reported_seq":1810,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-08-19T14:47:13.597614+0000","last_change":"2024-08-19T14:46:56.367773+0000","last_active":"2024-08-19T14:47:13.597614+0000","last_peered":"2024-08-19T14:47:13.597614+0000","last_clean":"2024-08-19T14:47:13.597614+0000","last_became_active":"2024-08-19T14:44:38.183371+0000","last_became_peered":"2024-08-19T14:44:38.183371+0000","last_unstale":"2024-08-19T14:47:13.597614+0000","last_undegraded":"2024-08-19T14:47:13.597614+0000","last_fullsized":"2024-08-19T14:47:13.597614+0000","mapping_epoch":837,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:46:56.367650+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:46:56.367650+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:48:22.698349+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11],"acting":[8,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.14","version":"70'139","reported_seq":1216,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-08-19T14:47:13.602882+0000","last_change":"2024-08-19T14:46:34.796943+0000","last_active":"2024-08-19T14:47:13.602882+0000","last_peered":"2024-08-19T14:47:13.602882+0000","last_clean":"2024-08-19T14:47:13.602882+0000","last_became_active":"2024-08-19T14:43:08.454350+0000","last_became_peered":"2024-08-19T14:43:08.454350+0000","last_unstale":"2024-08-19T14:47:13.602882+0000","last_undegraded":"2024-08-19T14:47:13.602882+0000","last_fullsized":"2024-08-19T14:47:13.602882+0000","mapping_epoch":793,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":794,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:46:34.796863+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:46:34.796863+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:48:00.974974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,9],"acting":[1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"70'139","reported_seq":561,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-08-19T14:47:13.597885+0000","last_change":"2024-08-19T14:46:54.433540+0000","last_active":"2024-08-19T14:47:13.597885+0000","last_peered":"2024-08-19T14:47:13.597885+0000","last_clean":"2024-08-19T14:47:13.597885+0000","last_became_active":"2024-08-19T14:44:38.199514+0000","last_became_peered":"2024-08-19T14:44:38.199514+0000","last_unstale":"2024-08-19T14:47:13.597885+0000","last_undegraded":"2024-08-19T14:47:13.597885+0000","last_fullsized":"2024-08-19T14:47:13.597885+0000","mapping_epoch":837,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:46:54.431820+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:20:09.180923+0000","last_clean_scrub_stamp":"2024-08-19T14:46:54.431820+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:48:09.163994+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,11],"acting":[8,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"70'139","reported_seq":1651,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-08-19T14:47:13.597614+0000","last_change":"2024-08-19T14:46:15.973722+0000","last_active":"2024-08-19T14:47:13.597614+0000","last_peered":"2024-08-19T14:47:13.597614+0000","last_clean":"2024-08-19T14:47:13.597614+0000","last_became_active":"2024-08-19T14:43:11.471324+0000","last_became_peered":"2024-08-19T14:43:11.471324+0000","last_unstale":"2024-08-19T14:47:13.597614+0000","last_undegraded":"2024-08-19T14:47:13.597614+0000","last_fullsized":"2024-08-19T14:47:13.597614+0000","mapping_epoch":795,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":796,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:46:15.973642+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:41:51.832284+0000","last_clean_scrub_stamp":"2024-08-19T14:46:15.973642+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:47:25.118982+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,11],"acting":[4,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1b","version":"70'139","reported_seq":2033,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-08-19T14:47:13.597710+0000","last_change":"2024-08-19T14:46:58.388681+0000","last_active":"2024-08-19T14:47:13.597710+0000","last_peered":"2024-08-19T14:47:13.597710+0000","last_clean":"2024-08-19T14:47:13.597710+0000","last_became_active":"2024-08-19T14:42:39.065604+0000","last_became_peered":"2024-08-19T14:42:39.065604+0000","last_unstale":"2024-08-19T14:47:13.597710+0000","last_undegraded":"2024-08-19T14:47:13.597710+0000","last_fullsized":"2024-08-19T14:47:13.597710+0000","mapping_epoch":778,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":779,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:46:58.388563+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:37:59.510697+0000","last_clean_scrub_stamp":"2024-08-19T14:46:58.388563+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:48:21.771287+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,2],"acting":[8,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.18","version":"70'139","reported_seq":1021,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-08-19T14:47:13.603473+0000","last_change":"2024-08-19T14:46:54.743821+0000","last_active":"2024-08-19T14:47:13.603473+0000","last_peered":"2024-08-19T14:47:13.603473+0000","last_clean":"2024-08-19T14:47:13.603473+0000","last_became_active":"2024-08-19T14:44:38.182712+0000","last_became_peered":"2024-08-19T14:44:38.182712+0000","last_unstale":"2024-08-19T14:47:13.603473+0000","last_undegraded":"2024-08-19T14:47:13.603473+0000","last_fullsized":"2024-08-19T14:47:13.603473+0000","mapping_epoch":837,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:46:54.743702+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:27:33.084161+0000","last_clean_scrub_stamp":"2024-08-19T14:46:54.743702+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:48:17.549331+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":402496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,11],"acting":[1,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.19","version":"70'139","reported_seq":880,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-08-19T14:47:15.619405+0000","last_change":"2024-08-19T14:45:26.148752+0000","last_active":"2024-08-19T14:47:15.619405+0000","last_peered":"2024-08-19T14:47:15.619405+0000","last_clean":"2024-08-19T14:47:15.619405+0000","last_became_active":"2024-08-19T14:45:26.147921+0000","last_became_peered":"2024-08-19T14:45:26.147921+0000","last_unstale":"2024-08-19T14:47:15.619405+0000","last_undegraded":"2024-08-19T14:47:15.619405+0000","last_fullsized":"2024-08-19T14:47:15.619405+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:23.083891+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:44:23.083891+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:47:19.896597+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,4],"acting":[10,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1e","version":"70'139","reported_seq":1037,"reported_epoch":899,"state":"active+clean","last_fresh":"2024-08-19T14:47:11.600401+0000","last_change":"2024-08-19T14:45:16.381698+0000","last_active":"2024-08-19T14:47:11.600401+0000","last_peered":"2024-08-19T14:47:11.600401+0000","last_clean":"2024-08-19T14:47:11.600401+0000","last_became_active":"2024-08-19T14:43:16.124258+0000","last_became_peered":"2024-08-19T14:43:16.124258+0000","last_unstale":"2024-08-19T14:47:11.600401+0000","last_undegraded":"2024-08-19T14:47:11.600401+0000","last_fullsized":"2024-08-19T14:47:11.600401+0000","mapping_epoch":800,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":801,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:45:16.381587+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:27:04.780939+0000","last_clean_scrub_stamp":"2024-08-19T14:45:16.381587+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:47:14.977514+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,10],"acting":[11,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1c","version":"70'139","reported_seq":1787,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-08-19T14:47:13.598662+0000","last_change":"2024-08-19T14:45:26.106081+0000","last_active":"2024-08-19T14:47:13.598662+0000","last_peered":"2024-08-19T14:47:13.598662+0000","last_clean":"2024-08-19T14:47:13.598662+0000","last_became_active":"2024-08-19T14:45:26.105382+0000","last_became_peered":"2024-08-19T14:45:26.105382+0000","last_unstale":"2024-08-19T14:47:13.598662+0000","last_undegraded":"2024-08-19T14:47:13.598662+0000","last_fullsized":"2024-08-19T14:47:13.598662+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:31.796263+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:44:31.796263+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:47:17.388381+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1d","version":"70'139","reported_seq":1679,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-08-19T14:47:13.603126+0000","last_change":"2024-08-19T14:44:29.650415+0000","last_active":"2024-08-19T14:47:13.603126+0000","last_peered":"2024-08-19T14:47:13.603126+0000","last_clean":"2024-08-19T14:47:13.603126+0000","last_became_active":"2024-08-19T14:35:11.540488+0000","last_became_peered":"2024-08-19T14:35:11.540488+0000","last_unstale":"2024-08-19T14:47:13.603126+0000","last_undegraded":"2024-08-19T14:47:13.603126+0000","last_fullsized":"2024-08-19T14:47:13.603126+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:44:29.650238+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:44:29.650238+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:47:15.669454+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":1927872536,"num_objects":29418,"num_object_clones":0,"num_object_copies":88254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29418,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":159588,"num_write_kb":6048129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39691,"num_bytes_recovered":2601189376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5960511488,"data_stored":5960499272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159388,"ondisk_log_size":159388,"up":48,"acting":48,"num_store_stats":9},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1385536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2785280,"data_stored":2771072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187886,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1981220,"kb_used_data":762072,"kb_used_omap":1,"kb_used_meta":1219070,"kb_avail":91763932,"statfs":{"total":95995035648,"available":93966266368,"internally_reserved":0,"allocated":780361728,"data_stored":777668691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":1248328132},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":539,"seq":2314987372754,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1039592,"kb_used_data":818844,"kb_used_omap":4,"kb_used_meta":220731,"kb_avail":92705560,"statfs":{"total":95995035648,"available":94930493440,"internally_reserved":0,"allocated":838496256,"data_stored":835802566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4407,"internal_metadata":226029257},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":74,"seq":317827580344,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2333148,"kb_used_data":1095312,"kb_used_omap":1,"kb_used_meta":1237822,"kb_avail":91412004,"statfs":{"total":95995035648,"available":93605892096,"internally_reserved":0,"allocated":1121599488,"data_stored":1118910003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":1267530177},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874687,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1673712,"kb_used_data":605472,"kb_used_omap":1,"kb_used_meta":1068222,"kb_avail":92071440,"statfs":{"total":95995035648,"available":94281154560,"internally_reserved":0,"allocated":620003328,"data_stored":617305566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1093859781},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":0,"seq":244813136322,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430662,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":783404,"kb_used_data":603676,"kb_used_omap":1,"kb_used_meta":179710,"kb_avail":92961748,"statfs":{"total":95995035648,"available":95192829952,"internally_reserved":0,"allocated":618164224,"data_stored":615470534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":184023494},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":171798692297,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986639,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1636204,"kb_used_data":364896,"kb_used_omap":1,"kb_used_meta":1271294,"kb_avail":92108948,"statfs":{"total":95995035648,"available":94319562752,"internally_reserved":0,"allocated":373653504,"data_stored":370955742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1301805509},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":111669150164,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":634,"seq":2723009265810,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":663472,"kb_used_data":126052,"kb_used_omap":4,"kb_used_meta":537403,"kb_avail":93081680,"statfs":{"total":95995035648,"available":95315640320,"internally_reserved":0,"allocated":129077248,"data_stored":126380006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4407,"internal_metadata":550301385},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575325,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2919540,"kb_used_data":1452904,"kb_used_omap":1,"kb_used_meta":1466558,"kb_avail":90825612,"statfs":{"total":95995035648,"available":93005426688,"internally_reserved":0,"allocated":1487773696,"data_stored":1485072382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1501755845},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,0,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":34359738849,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1463619584,"data_stored":1463615512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":118751232,"data_stored":118751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":365170688,"data_stored":365166616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":150142976,"data_stored":150142976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":600506368,"data_stored":600506368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":597495808,"data_stored":597491736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1101463552,"data_stored":1101463552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":807469056,"data_stored":807469056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":755892224,"data_stored":755892224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T14:47:17.676 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-08-19T14:47:17.676 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-19T14:47:17.676 INFO:tasks.thrashosds.thrasher:Adding osd 7 2024-08-19T14:47:17.676 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd in 7 2024-08-19T14:47:17.677 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:47:17.677 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:47:17.702 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:17.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:17.704 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:47:17.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:17 smithi032 ceph-mon[20004]: 1.a scrub starts 2024-08-19T14:47:17.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:17 smithi032 ceph-mon[20004]: 1.a scrub ok 2024-08-19T14:47:17.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:17 smithi032 ceph-mon[20004]: pgmap v2122: 47 pgs: 47 active+clean; 1.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 834 op/s 2024-08-19T14:47:17.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:17 smithi032 ceph-mon[20004]: from='client.18171 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:47:17.804 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:47:17.808 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:17 smithi031 ceph-mon[16336]: 1.a scrub starts 2024-08-19T14:47:17.808 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:17 smithi031 ceph-mon[16336]: 1.a scrub ok 2024-08-19T14:47:17.808 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:17 smithi031 ceph-mon[16336]: pgmap v2122: 47 pgs: 47 active+clean; 1.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 834 op/s 2024-08-19T14:47:17.808 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:17 smithi031 ceph-mon[16336]: from='client.18171 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:47:17.842 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:17.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:17.843 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:47:17.944 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:47:17.955 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-19T14:47:17.956 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-08-19T14:47:17.956 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-08-19T14:47:17.956 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-08-19T14:47:17.956 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:17.956 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26911.0:153964 8.12 8:4d4545a9:::benchmark_data_smithi045_137103_object41500:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e771)", 2024-08-19T14:47:17.956 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:42:24.164475+0000", 2024-08-19T14:47:17.956 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 293.789611863, 2024-08-19T14:47:17.956 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.21338994999999999, 2024-08-19T14:47:17.956 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:47:17.956 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:47:17.956 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:47:17.957 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:47:17.957 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26911", 2024-08-19T14:47:17.957 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3494993995", 2024-08-19T14:47:17.957 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 153964 2024-08-19T14:47:17.957 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:17.957 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:47:17.957 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:17.957 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:47:17.957 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:24.164475+0000", 2024-08-19T14:47:17.957 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:17.957 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:17.958 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:17.958 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:47:17.958 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:24.164475+0000", 2024-08-19T14:47:17.958 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:17.958 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:17.958 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:17.958 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:47:17.958 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:24.164476+0000", 2024-08-19T14:47:17.958 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.125e-06 2024-08-19T14:47:17.958 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:17.959 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:17.959 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:47:17.959 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:24.164480+0000", 2024-08-19T14:47:17.959 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8e-06 2024-08-19T14:47:17.959 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:17.959 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:17.959 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:47:17.959 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:24.164491+0000", 2024-08-19T14:47:17.959 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1266e-05 2024-08-19T14:47:17.959 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:17.959 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:17.959 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:47:17.959 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:24.164499+0000", 2024-08-19T14:47:17.960 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5850000000000002e-06 2024-08-19T14:47:17.960 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:17.960 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:17.960 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:47:17.960 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:24.164556+0000", 2024-08-19T14:47:17.960 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7401999999999997e-05 2024-08-19T14:47:17.960 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:17.960 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:17.960 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:47:17.960 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:24.164693+0000", 2024-08-19T14:47:17.960 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000136623 2024-08-19T14:47:17.960 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:17.960 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:17.961 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,4", 2024-08-19T14:47:17.961 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:24.164802+0000", 2024-08-19T14:47:17.961 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000109221 2024-08-19T14:47:17.961 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:17.961 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:17.961 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:47:17.961 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:24.167783+0000", 2024-08-19T14:47:17.961 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.002980318 2024-08-19T14:47:17.961 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:17.961 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:17.961 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:47:17.961 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:24.171166+0000", 2024-08-19T14:47:17.961 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0033829820000000001 2024-08-19T14:47:17.962 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:17.962 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:17.962 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:47:17.962 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:24.377769+0000", 2024-08-19T14:47:17.962 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20660362900000001 2024-08-19T14:47:17.962 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:17.962 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:17.962 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:47:17.962 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:24.377824+0000", 2024-08-19T14:47:17.962 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5204000000000002e-05 2024-08-19T14:47:17.962 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:17.962 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:17.962 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:47:17.963 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:24.377865+0000", 2024-08-19T14:47:17.963 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0794999999999998e-05 2024-08-19T14:47:17.963 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:17.963 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:47:17.963 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:17.963 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:17.963 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:17.963 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26911.0:153968 8.12 8:48c7ff03:::benchmark_data_smithi045_137103_object41504:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e771)", 2024-08-19T14:47:17.963 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:42:24.168155+0000", 2024-08-19T14:47:17.963 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 293.785932264, 2024-08-19T14:47:17.963 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.21236127299999999, 2024-08-19T14:47:17.963 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:47:17.963 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:47:17.963 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:47:17.964 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:47:17.964 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26911", 2024-08-19T14:47:17.964 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3494993995", 2024-08-19T14:47:17.964 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 153968 2024-08-19T14:47:17.964 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:17.964 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:47:17.964 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:17.964 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:47:17.964 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:24.168155+0000", 2024-08-19T14:47:17.964 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:17.964 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:17.964 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:17.965 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:47:17.965 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:24.168155+0000", 2024-08-19T14:47:17.965 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:17.965 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:17.965 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:17.965 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:47:17.965 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:24.168156+0000", 2024-08-19T14:47:17.965 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.533e-06 2024-08-19T14:47:17.965 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:17.965 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:17.965 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:47:17.965 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:24.168162+0000", 2024-08-19T14:47:17.966 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8270000000000003e-06 2024-08-19T14:47:17.966 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:17.966 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:17.966 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:47:17.966 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:24.168617+0000", 2024-08-19T14:47:17.966 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00045464600000000001 2024-08-19T14:47:17.966 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:17.966 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:17.966 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:47:17.966 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:24.168628+0000", 2024-08-19T14:47:17.966 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1586999999999999e-05 2024-08-19T14:47:17.966 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:17.966 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:17.967 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:47:17.967 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:24.168685+0000", 2024-08-19T14:47:17.967 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.702e-05 2024-08-19T14:47:17.967 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:17.967 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:17.967 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:47:17.967 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:24.168809+0000", 2024-08-19T14:47:17.967 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012383199999999999 2024-08-19T14:47:17.967 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:17.967 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:17.967 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,4", 2024-08-19T14:47:17.967 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:24.168913+0000", 2024-08-19T14:47:17.967 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000103368 2024-08-19T14:47:17.967 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:17.968 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:17.968 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:47:17.968 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:24.171287+0000", 2024-08-19T14:47:17.968 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0023743710000000001 2024-08-19T14:47:17.968 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:17.968 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:17.968 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:47:17.968 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:24.177249+0000", 2024-08-19T14:47:17.968 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0059624370000000001 2024-08-19T14:47:17.968 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:17.969 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:17.969 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:47:17.969 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:24.380411+0000", 2024-08-19T14:47:17.969 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.20316157700000001 2024-08-19T14:47:17.969 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:17.969 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:17.969 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:47:17.969 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:24.380467+0000", 2024-08-19T14:47:17.969 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6344000000000001e-05 2024-08-19T14:47:17.969 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:17.969 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:17.969 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:47:17.969 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:24.380516+0000", 2024-08-19T14:47:17.969 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8730999999999998e-05 2024-08-19T14:47:17.970 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:17.970 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:47:17.970 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:17.970 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:17.970 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:17.970 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26911.0:168621 8.17 8.bc84b017 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e775)", 2024-08-19T14:47:17.970 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:42:35.141291+0000", 2024-08-19T14:47:17.970 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 282.812796499, 2024-08-19T14:47:17.970 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.98747045499999997, 2024-08-19T14:47:17.970 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:47:17.970 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:47:17.971 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:47:17.971 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:47:17.971 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26911", 2024-08-19T14:47:17.971 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3494993995", 2024-08-19T14:47:17.971 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 168621 2024-08-19T14:47:17.971 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:17.971 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:47:17.971 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:17.971 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:47:17.972 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.141291+0000", 2024-08-19T14:47:17.972 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:17.972 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:17.972 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:17.972 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:47:17.972 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.141291+0000", 2024-08-19T14:47:17.972 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:17.972 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:17.972 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:17.973 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:47:17.973 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.141292+0000", 2024-08-19T14:47:17.973 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3149999999999999e-06 2024-08-19T14:47:17.973 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:17.973 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:17.973 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:47:17.973 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.141297+0000", 2024-08-19T14:47:17.973 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5090000000000001e-06 2024-08-19T14:47:17.973 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:17.973 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:17.973 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:47:17.974 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.141311+0000", 2024-08-19T14:47:17.974 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3698e-05 2024-08-19T14:47:17.974 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:17.974 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:17.974 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:47:17.974 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.141318+0000", 2024-08-19T14:47:17.974 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1430000000000001e-06 2024-08-19T14:47:17.974 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:17.974 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:17.974 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:47:17.974 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.141377+0000", 2024-08-19T14:47:17.974 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9055999999999998e-05 2024-08-19T14:47:17.975 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:17.975 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:17.975 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:47:17.975 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.141389+0000", 2024-08-19T14:47:17.975 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1221e-05 2024-08-19T14:47:17.975 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:17.975 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:17.975 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:47:17.975 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:36.128724+0000", 2024-08-19T14:47:17.975 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.98733521400000002 2024-08-19T14:47:17.975 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:17.975 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:17.975 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:47:17.976 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:36.128761+0000", 2024-08-19T14:47:17.976 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7299000000000003e-05 2024-08-19T14:47:17.976 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:17.976 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:47:17.976 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:17.976 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:17.976 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:17.976 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26911.0:168656 8.17 8.3c3e7b97 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e775)", 2024-08-19T14:47:17.976 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:42:35.185953+0000", 2024-08-19T14:47:17.976 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 282.768133713, 2024-08-19T14:47:17.976 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.94288665400000005, 2024-08-19T14:47:17.976 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:47:17.976 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:47:17.977 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:47:17.977 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:47:17.977 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26911", 2024-08-19T14:47:17.977 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3494993995", 2024-08-19T14:47:17.977 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 168656 2024-08-19T14:47:17.977 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:17.977 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:47:17.977 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:17.977 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:47:17.977 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.185953+0000", 2024-08-19T14:47:17.977 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:17.977 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:17.978 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:17.978 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:47:17.978 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.185953+0000", 2024-08-19T14:47:17.978 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:17.978 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:17.978 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:17.978 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:47:17.978 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.185955+0000", 2024-08-19T14:47:17.978 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.877e-06 2024-08-19T14:47:17.978 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:17.978 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:17.978 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:47:17.978 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.185964+0000", 2024-08-19T14:47:17.978 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.0429999999999996e-06 2024-08-19T14:47:17.979 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:17.979 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:17.979 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:47:17.979 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.186012+0000", 2024-08-19T14:47:17.979 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7735999999999998e-05 2024-08-19T14:47:17.979 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:17.979 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:17.979 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:47:17.979 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.186028+0000", 2024-08-19T14:47:17.979 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5639999999999999e-05 2024-08-19T14:47:17.979 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:17.979 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:17.979 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:47:17.979 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.186189+0000", 2024-08-19T14:47:17.980 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00016155400000000001 2024-08-19T14:47:17.980 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:17.980 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:17.980 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:47:17.980 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.186201+0000", 2024-08-19T14:47:17.980 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1531e-05 2024-08-19T14:47:17.980 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:17.980 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:17.980 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:47:17.980 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:36.128827+0000", 2024-08-19T14:47:17.980 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.94262613100000003 2024-08-19T14:47:17.980 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:17.981 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:17.981 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:47:17.981 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:36.128840+0000", 2024-08-19T14:47:17.981 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3142000000000001e-05 2024-08-19T14:47:17.981 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:17.981 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:47:17.981 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:17.981 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:17.981 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:17.981 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26911.0:168680 8.17 8.f8788eb7 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e775)", 2024-08-19T14:47:17.981 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:42:35.211464+0000", 2024-08-19T14:47:17.981 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 282.74262360199998, 2024-08-19T14:47:17.981 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.91744495999999998, 2024-08-19T14:47:17.982 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:47:17.982 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:47:17.982 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:47:17.982 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:47:17.982 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26911", 2024-08-19T14:47:17.982 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3494993995", 2024-08-19T14:47:17.982 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 168680 2024-08-19T14:47:17.982 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:17.982 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:47:17.982 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:17.982 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:47:17.982 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.211464+0000", 2024-08-19T14:47:17.983 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:17.983 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:17.983 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:17.983 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:47:17.983 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.211464+0000", 2024-08-19T14:47:17.983 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:17.983 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:17.983 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:17.983 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:47:17.983 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.211465+0000", 2024-08-19T14:47:17.983 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.248e-06 2024-08-19T14:47:17.983 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:17.983 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:17.984 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:47:17.984 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.211470+0000", 2024-08-19T14:47:17.984 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3129999999999998e-06 2024-08-19T14:47:17.984 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:17.984 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:17.984 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:47:17.984 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.211483+0000", 2024-08-19T14:47:17.984 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2695e-05 2024-08-19T14:47:17.984 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:17.984 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:17.984 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:47:17.984 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.211490+0000", 2024-08-19T14:47:17.984 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0400000000000004e-06 2024-08-19T14:47:17.984 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:17.985 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:17.985 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:47:17.985 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.211556+0000", 2024-08-19T14:47:17.985 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6353999999999995e-05 2024-08-19T14:47:17.985 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:17.985 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:17.985 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:47:17.985 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.211568+0000", 2024-08-19T14:47:17.985 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2228999999999999e-05 2024-08-19T14:47:17.985 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:17.985 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:17.985 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:47:17.985 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:36.128897+0000", 2024-08-19T14:47:17.985 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.91732864700000005 2024-08-19T14:47:17.986 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:17.986 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:17.986 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:47:17.986 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:36.128908+0000", 2024-08-19T14:47:17.986 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1433999999999999e-05 2024-08-19T14:47:17.986 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:17.986 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:47:17.986 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:17.986 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:17.986 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:17.986 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26911.0:168688 8.17 8.2c134277 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e775)", 2024-08-19T14:47:17.986 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:42:35.234594+0000", 2024-08-19T14:47:17.986 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 282.719492844, 2024-08-19T14:47:17.987 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.89436076099999995, 2024-08-19T14:47:17.987 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:47:17.987 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:47:17.987 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:47:17.987 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:47:17.987 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26911", 2024-08-19T14:47:17.987 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3494993995", 2024-08-19T14:47:17.987 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 168688 2024-08-19T14:47:17.987 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:17.987 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:47:17.987 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:17.987 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:47:17.987 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.234594+0000", 2024-08-19T14:47:17.988 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:17.988 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:17.988 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:17.988 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:47:17.988 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.234594+0000", 2024-08-19T14:47:17.988 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:17.988 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:17.988 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:17.988 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:47:17.988 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.234596+0000", 2024-08-19T14:47:17.988 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2240000000000001e-06 2024-08-19T14:47:17.988 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:17.988 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:17.988 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:47:17.989 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.234601+0000", 2024-08-19T14:47:17.989 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4489999999999998e-06 2024-08-19T14:47:17.989 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:17.989 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:17.989 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:47:17.989 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.234613+0000", 2024-08-19T14:47:17.989 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2140999999999999e-05 2024-08-19T14:47:17.989 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:17.989 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:17.989 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:47:17.989 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.234619+0000", 2024-08-19T14:47:17.989 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.699e-06 2024-08-19T14:47:17.990 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:17.990 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:17.990 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:47:17.990 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.234681+0000", 2024-08-19T14:47:17.990 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2685e-05 2024-08-19T14:47:17.990 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:17.990 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:17.990 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:47:17.990 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:35.234694+0000", 2024-08-19T14:47:17.990 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2656e-05 2024-08-19T14:47:17.990 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:17.990 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:17.990 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:47:17.990 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:36.128946+0000", 2024-08-19T14:47:17.991 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.89425167400000005 2024-08-19T14:47:17.991 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:17.991 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:17.991 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:47:17.991 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:36.128955+0000", 2024-08-19T14:47:17.991 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.2329999999999995e-06 2024-08-19T14:47:17.991 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:17.991 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:47:17.991 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:17.991 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:17.991 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:17.991 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26911.0:168656 8.17 8:e9de7c3c:::benchmark_data_smithi045_137103_object56192:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e776)", 2024-08-19T14:47:17.991 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:42:37.165925+0000", 2024-08-19T14:47:17.992 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 280.788161704, 2024-08-19T14:47:17.992 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.211492706, 2024-08-19T14:47:17.992 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:47:17.992 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:47:17.992 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:47:17.992 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:47:17.992 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26911", 2024-08-19T14:47:17.992 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3494993995", 2024-08-19T14:47:17.992 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 168656 2024-08-19T14:47:17.992 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:17.992 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:47:17.992 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:17.993 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:47:17.993 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.165925+0000", 2024-08-19T14:47:17.993 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:17.993 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:17.993 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:17.993 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:47:17.993 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.165925+0000", 2024-08-19T14:47:17.993 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:17.993 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:17.993 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:17.993 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:47:17.993 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.165926+0000", 2024-08-19T14:47:17.993 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4199999999999996e-07 2024-08-19T14:47:17.993 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:17.994 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:17.994 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:47:17.994 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.165930+0000", 2024-08-19T14:47:17.994 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1200000000000004e-06 2024-08-19T14:47:17.994 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:17.994 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:17.994 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:47:17.994 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.165938+0000", 2024-08-19T14:47:17.994 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.0139999999999998e-06 2024-08-19T14:47:17.994 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:17.994 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:17.994 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:47:17.994 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.165944+0000", 2024-08-19T14:47:17.995 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9839999999999998e-06 2024-08-19T14:47:17.995 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:17.995 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:17.995 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:47:17.995 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.167146+0000", 2024-08-19T14:47:17.995 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001201988 2024-08-19T14:47:17.995 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:17.995 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:17.995 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:47:17.995 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.167156+0000", 2024-08-19T14:47:17.995 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0071999999999999e-05 2024-08-19T14:47:17.995 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:17.995 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:17.995 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:47:17.996 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.350255+0000", 2024-08-19T14:47:17.996 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.183098765 2024-08-19T14:47:17.996 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:17.996 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:17.996 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:47:17.996 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.350430+0000", 2024-08-19T14:47:17.996 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000175207 2024-08-19T14:47:17.996 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:17.996 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:17.996 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,8", 2024-08-19T14:47:17.996 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.350598+0000", 2024-08-19T14:47:17.996 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00016778100000000001 2024-08-19T14:47:17.996 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:17.996 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:17.997 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:47:17.997 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.354899+0000", 2024-08-19T14:47:17.997 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0043015800000000002 2024-08-19T14:47:17.997 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:17.997 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:17.997 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:47:17.997 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.354942+0000", 2024-08-19T14:47:17.997 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2781000000000003e-05 2024-08-19T14:47:17.997 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:17.997 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:17.997 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:47:17.997 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.376407+0000", 2024-08-19T14:47:17.997 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.021464496999999999 2024-08-19T14:47:17.998 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:17.998 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:17.998 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:47:17.998 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.376500+0000", 2024-08-19T14:47:17.998 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3450999999999997e-05 2024-08-19T14:47:17.998 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:17.998 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:17.998 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:47:17.998 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.377418+0000", 2024-08-19T14:47:17.998 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00091792399999999998 2024-08-19T14:47:17.998 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:17.998 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:47:17.998 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:17.999 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:17.999 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:17.999 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26911.0:168680 8.17 8:ed711e1f:::benchmark_data_smithi045_137103_object56216:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e776)", 2024-08-19T14:47:17.999 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:42:37.165952+0000", 2024-08-19T14:47:17.999 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 280.788135184, 2024-08-19T14:47:17.999 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.21468071799999999, 2024-08-19T14:47:17.999 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:47:17.999 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:47:17.999 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:47:17.999 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:47:17.999 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26911", 2024-08-19T14:47:17.999 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3494993995", 2024-08-19T14:47:17.999 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 168680 2024-08-19T14:47:18.000 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:18.000 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:47:18.000 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:18.000 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:47:18.000 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.165952+0000", 2024-08-19T14:47:18.000 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:18.000 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:18.000 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:18.000 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:47:18.000 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.165952+0000", 2024-08-19T14:47:18.000 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:18.000 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:18.000 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:18.001 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:47:18.001 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.165952+0000", 2024-08-19T14:47:18.001 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6300000000000001e-07 2024-08-19T14:47:18.001 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:18.001 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:18.001 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:47:18.001 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.165955+0000", 2024-08-19T14:47:18.001 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6649999999999999e-06 2024-08-19T14:47:18.001 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:18.001 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:18.001 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:47:18.001 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.165961+0000", 2024-08-19T14:47:18.002 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7790000000000001e-06 2024-08-19T14:47:18.002 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:18.002 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:18.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:18.003 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:18.003 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:47:18.003 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:47:18.003 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.165965+0000", 2024-08-19T14:47:18.004 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5190000000000003e-06 2024-08-19T14:47:18.004 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:18.004 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:18.004 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:47:18.004 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.167194+0000", 2024-08-19T14:47:18.004 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001229129 2024-08-19T14:47:18.004 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:18.004 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:18.004 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:47:18.004 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.167203+0000", 2024-08-19T14:47:18.004 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.7949999999999996e-06 2024-08-19T14:47:18.004 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:18.004 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:18.005 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:47:18.005 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.355085+0000", 2024-08-19T14:47:18.005 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.18788205999999999 2024-08-19T14:47:18.005 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:18.005 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:18.005 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:47:18.005 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.355374+0000", 2024-08-19T14:47:18.005 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00028833400000000001 2024-08-19T14:47:18.005 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:18.005 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:18.005 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,8", 2024-08-19T14:47:18.005 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.355520+0000", 2024-08-19T14:47:18.005 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014636400000000001 2024-08-19T14:47:18.006 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:18.006 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:18.006 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:47:18.006 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.364692+0000", 2024-08-19T14:47:18.006 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0091721400000000005 2024-08-19T14:47:18.006 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:18.006 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:18.006 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:47:18.006 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.364844+0000", 2024-08-19T14:47:18.006 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00015155999999999999 2024-08-19T14:47:18.006 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:18.006 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:18.006 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:47:18.006 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.377432+0000", 2024-08-19T14:47:18.007 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.012588217000000001 2024-08-19T14:47:18.007 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:18.007 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:18.007 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:47:18.007 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.377518+0000", 2024-08-19T14:47:18.007 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6471999999999995e-05 2024-08-19T14:47:18.007 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:18.007 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:18.007 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:47:18.007 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.380633+0000", 2024-08-19T14:47:18.007 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0031143210000000002 2024-08-19T14:47:18.007 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:18.007 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:47:18.007 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:18.008 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:18.008 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:18.008 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26911.0:168688 8.17 8:ee42c834:::benchmark_data_smithi045_137103_object56224:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e776)", 2024-08-19T14:47:18.008 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:42:37.165971+0000", 2024-08-19T14:47:18.008 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 280.78811621699998, 2024-08-19T14:47:18.008 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.22089492299999999, 2024-08-19T14:47:18.008 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:47:18.008 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:47:18.008 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:47:18.008 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:47:18.008 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26911", 2024-08-19T14:47:18.008 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3494993995", 2024-08-19T14:47:18.008 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 168688 2024-08-19T14:47:18.009 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:18.009 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:47:18.009 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:18.009 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:47:18.009 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.165971+0000", 2024-08-19T14:47:18.009 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:18.009 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:18.009 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:18.009 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:47:18.009 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.165971+0000", 2024-08-19T14:47:18.009 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:18.009 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:18.009 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:18.009 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:47:18.010 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.165971+0000", 2024-08-19T14:47:18.010 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5400000000000002e-07 2024-08-19T14:47:18.010 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:18.010 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:18.010 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:47:18.010 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.165973+0000", 2024-08-19T14:47:18.010 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1689999999999999e-06 2024-08-19T14:47:18.010 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:18.010 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:18.010 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:47:18.010 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.165994+0000", 2024-08-19T14:47:18.010 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0767e-05 2024-08-19T14:47:18.010 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:18.011 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:18.011 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:47:18.011 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.166000+0000", 2024-08-19T14:47:18.011 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5140000000000001e-06 2024-08-19T14:47:18.011 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:18.011 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:18.011 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:47:18.011 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.167240+0000", 2024-08-19T14:47:18.011 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0012404639999999999 2024-08-19T14:47:18.011 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:18.011 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:18.011 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-08-19T14:47:18.011 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.167250+0000", 2024-08-19T14:47:18.011 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.4390000000000006e-06 2024-08-19T14:47:18.012 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:18.012 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:18.012 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:47:18.012 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.359263+0000", 2024-08-19T14:47:18.012 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.192013814 2024-08-19T14:47:18.012 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:18.012 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:18.012 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-08-19T14:47:18.013 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.359426+0000", 2024-08-19T14:47:18.013 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000162335 2024-08-19T14:47:18.013 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:18.013 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:18.013 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2,8", 2024-08-19T14:47:18.013 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.359576+0000", 2024-08-19T14:47:18.013 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00015031800000000001 2024-08-19T14:47:18.013 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:18.013 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:18.013 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:47:18.013 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.364895+0000", 2024-08-19T14:47:18.013 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0053192669999999999 2024-08-19T14:47:18.013 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:18.013 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:18.014 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:47:18.014 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.365041+0000", 2024-08-19T14:47:18.014 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014546200000000001 2024-08-19T14:47:18.014 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:18.014 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:18.014 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-08-19T14:47:18.014 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.385060+0000", 2024-08-19T14:47:18.014 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.020019235 2024-08-19T14:47:18.014 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:18.014 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:18.014 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-08-19T14:47:18.014 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.385159+0000", 2024-08-19T14:47:18.014 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.9508999999999999e-05 2024-08-19T14:47:18.015 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:18.015 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:18.015 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:47:18.015 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:37.386866+0000", 2024-08-19T14:47:18.015 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0017063759999999999 2024-08-19T14:47:18.015 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:18.015 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:47:18.015 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:18.015 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:18.015 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:18.015 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26911.0:169443 8.17 8:ed201ba7:::benchmark_data_smithi045_137103_object56979:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e778)", 2024-08-19T14:47:18.015 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:42:38.114142+0000", 2024-08-19T14:47:18.015 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 279.83994514400001, 2024-08-19T14:47:18.015 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6826600630000002, 2024-08-19T14:47:18.016 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:47:18.016 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:47:18.016 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:47:18.016 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:47:18.016 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26911", 2024-08-19T14:47:18.016 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3494993995", 2024-08-19T14:47:18.016 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 169443 2024-08-19T14:47:18.016 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:18.016 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:47:18.016 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:18.016 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:47:18.016 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:38.114142+0000", 2024-08-19T14:47:18.017 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:18.017 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:18.017 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:18.017 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:47:18.017 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:38.114142+0000", 2024-08-19T14:47:18.017 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:18.017 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:18.017 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:18.017 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:47:18.017 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:38.114143+0000", 2024-08-19T14:47:18.017 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.569999999999999e-07 2024-08-19T14:47:18.017 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:18.017 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:18.017 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:47:18.017 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:38.114147+0000", 2024-08-19T14:47:18.018 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1999999999999996e-06 2024-08-19T14:47:18.018 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:18.018 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:18.018 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:47:18.018 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:38.114160+0000", 2024-08-19T14:47:18.018 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2843000000000001e-05 2024-08-19T14:47:18.018 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:18.018 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:18.018 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:47:18.018 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:38.114167+0000", 2024-08-19T14:47:18.018 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4599999999999997e-06 2024-08-19T14:47:18.018 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:18.018 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:18.019 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:47:18.019 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:38.114340+0000", 2024-08-19T14:47:18.019 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00017212400000000001 2024-08-19T14:47:18.019 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:18.019 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:18.019 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for missing object", 2024-08-19T14:47:18.019 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:38.116422+0000", 2024-08-19T14:47:18.019 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0020820000000000001 2024-08-19T14:47:18.019 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:18.019 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:18.019 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:47:18.019 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:39.796790+0000", 2024-08-19T14:47:18.019 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6803685939999999 2024-08-19T14:47:18.019 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:18.020 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:18.020 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:47:18.020 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:39.796802+0000", 2024-08-19T14:47:18.020 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1885000000000001e-05 2024-08-19T14:47:18.020 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:18.020 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:47:18.020 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:18.020 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:18.020 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:18.020 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.26911.0:186524 8.17 8:eabd739d:::benchmark_data_smithi045_137103_object74060:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e786)", 2024-08-19T14:47:18.020 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-08-19T14:42:54.339443+0000", 2024-08-19T14:47:18.020 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 263.61464405999999, 2024-08-19T14:47:18.020 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.562044556, 2024-08-19T14:47:18.021 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-08-19T14:47:18.021 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-08-19T14:47:18.021 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-08-19T14:47:18.021 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-08-19T14:47:18.021 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.26911", 2024-08-19T14:47:18.021 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.45:0/3494993995", 2024-08-19T14:47:18.021 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 186524 2024-08-19T14:47:18.021 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:18.021 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-08-19T14:47:18.021 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:18.021 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-08-19T14:47:18.021 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:54.339443+0000", 2024-08-19T14:47:18.021 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:18.021 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:18.022 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:18.022 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-08-19T14:47:18.022 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:54.339443+0000", 2024-08-19T14:47:18.022 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-08-19T14:47:18.022 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:18.022 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:18.022 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-08-19T14:47:18.022 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:54.339444+0000", 2024-08-19T14:47:18.022 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6700000000000002e-07 2024-08-19T14:47:18.022 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:18.022 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:18.022 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-08-19T14:47:18.022 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:54.339448+0000", 2024-08-19T14:47:18.022 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1640000000000003e-06 2024-08-19T14:47:18.023 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:18.023 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:18.023 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-08-19T14:47:18.023 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:54.339458+0000", 2024-08-19T14:47:18.023 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0314e-05 2024-08-19T14:47:18.023 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:18.023 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:18.023 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-08-19T14:47:18.023 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:54.339464+0000", 2024-08-19T14:47:18.023 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5049999999999998e-06 2024-08-19T14:47:18.023 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:18.023 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:18.023 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:47:18.024 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:54.339702+0000", 2024-08-19T14:47:18.024 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00023851300000000001 2024-08-19T14:47:18.024 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:18.024 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:18.024 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for missing object", 2024-08-19T14:47:18.024 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:54.339797+0000", 2024-08-19T14:47:18.024 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.4357000000000005e-05 2024-08-19T14:47:18.024 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:18.024 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:18.024 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-08-19T14:47:18.024 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:55.901476+0000", 2024-08-19T14:47:18.024 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5616788619999999 2024-08-19T14:47:18.024 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-19T14:47:18.024 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-19T14:47:18.025 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-08-19T14:47:18.025 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-08-19T14:42:55.901488+0000", 2024-08-19T14:47:18.025 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1973999999999999e-05 2024-08-19T14:47:18.025 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:18.025 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:47:18.025 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:18.025 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-19T14:47:18.025 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-19T14:47:18.025 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-19T14:47:18.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:17 smithi045 ceph-mon[20143]: 1.a scrub starts 2024-08-19T14:47:18.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:17 smithi045 ceph-mon[20143]: 1.a scrub ok 2024-08-19T14:47:18.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:17 smithi045 ceph-mon[20143]: pgmap v2122: 47 pgs: 47 active+clean; 1.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 834 op/s 2024-08-19T14:47:18.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:17 smithi045 ceph-mon[20143]: from='client.18171 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:47:18.104 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:47:18.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:18.127 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:18.128 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:47:18.229 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:47:18.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:18.259 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:18.259 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:47:18.290 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:47:18.290 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:47:18.303 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:47:18.318 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:47:18.318 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:47:18.349 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:47:18.349 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:47:18.360 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:47:18.376 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:47:18.376 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:47:18.400 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:47:18.400 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:47:18.402 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:18.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:18.403 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:47:18.425 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:47:18.425 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:47:18.450 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:47:18.450 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:47:18.472 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:47:18.472 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:47:18.503 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:47:18.510 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:47:18.510 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:47:18.526 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:18.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:18.527 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:47:18.542 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:47:18.542 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:47:18.577 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:47:18.578 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:47:18.611 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:47:18.612 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:47:18.628 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:47:18.669 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:18.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:18.670 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:47:18.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:18 smithi032 ceph-mon[20004]: 1.7 scrub starts 2024-08-19T14:47:18.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:18 smithi032 ceph-mon[20004]: 1.7 scrub ok 2024-08-19T14:47:18.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:18 smithi032 ceph-mon[20004]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:47:18.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:18 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2917342098' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:47:18.771 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:47:18.792 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:18.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:18.793 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:47:18.894 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:47:18.942 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-08-19T14:47:18.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:18.943 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:18.944 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:47:18.991 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:18 smithi045 ceph-mon[20143]: 1.7 scrub starts 2024-08-19T14:47:18.992 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:18 smithi045 ceph-mon[20143]: 1.7 scrub ok 2024-08-19T14:47:18.992 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:18 smithi045 ceph-mon[20143]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:47:18.992 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:18 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2917342098' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:47:19.044 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:47:19.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:19.076 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:19.076 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:47:19.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:18 smithi031 ceph-mon[16336]: 1.7 scrub starts 2024-08-19T14:47:19.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:18 smithi031 ceph-mon[16336]: 1.7 scrub ok 2024-08-19T14:47:19.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:18 smithi031 ceph-mon[16336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:47:19.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:18 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2917342098' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:47:19.177 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:47:19.199 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:19.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:19.200 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:47:19.301 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:47:19.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:19.358 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:19.358 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:47:19.418 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:47:19.459 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:47:19.479 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:19.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:19.480 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:47:19.546 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-19T14:47:19.572 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:19 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mon-a[16332]: 2024-08-19T14:47:19.247+0000 7f2ff503e640 -1 mon.a@0(leader).osd e901 definitely_dead 0 2024-08-19T14:47:19.581 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:47:19.607 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:19.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:19.609 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:47:19.637 INFO:teuthology.orchestra.run.smithi031.stderr:marked in osd.7. 2024-08-19T14:47:19.710 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:47:19.731 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:19.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:19.732 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:47:19.831 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:19 smithi045 ceph-mon[20143]: 1.0 scrub starts 2024-08-19T14:47:19.831 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:19 smithi045 ceph-mon[20143]: 1.0 scrub ok 2024-08-19T14:47:19.831 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:19 smithi045 ceph-mon[20143]: pgmap v2123: 47 pgs: 47 active+clean; 1.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 659 op/s 2024-08-19T14:47:19.831 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:19 smithi045 ceph-mon[20143]: 9.0 scrub starts 2024-08-19T14:47:19.831 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:19 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:47:19.831 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:19 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:47:19.832 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:19 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3882588761' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2024-08-19T14:47:19.832 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:19 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2024-08-19T14:47:19.833 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:47:19.856 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:19.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:19.857 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:47:19.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:19 smithi031 ceph-mon[16336]: 1.0 scrub starts 2024-08-19T14:47:19.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:19 smithi031 ceph-mon[16336]: 1.0 scrub ok 2024-08-19T14:47:19.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:19 smithi031 ceph-mon[16336]: pgmap v2123: 47 pgs: 47 active+clean; 1.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 659 op/s 2024-08-19T14:47:19.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:19 smithi031 ceph-mon[16336]: 9.0 scrub starts 2024-08-19T14:47:19.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:19 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:47:19.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:19 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:47:19.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:19 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3882588761' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2024-08-19T14:47:19.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:19 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2024-08-19T14:47:19.947 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:47:19.948 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:47:19.958 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:47:19.980 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:19.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:19.981 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:47:20.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:19 smithi032 ceph-mon[20004]: 1.0 scrub starts 2024-08-19T14:47:20.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:19 smithi032 ceph-mon[20004]: 1.0 scrub ok 2024-08-19T14:47:20.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:19 smithi032 ceph-mon[20004]: pgmap v2123: 47 pgs: 47 active+clean; 1.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 659 op/s 2024-08-19T14:47:20.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:19 smithi032 ceph-mon[20004]: 9.0 scrub starts 2024-08-19T14:47:20.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:19 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:47:20.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:19 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:47:20.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:19 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3882588761' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2024-08-19T14:47:20.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:19 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2024-08-19T14:47:20.082 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:47:20.103 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:20.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:20.104 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:47:20.205 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:47:20.227 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:20.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:20.228 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:47:20.329 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:47:20.351 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:20.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:20.352 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:47:20.453 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:47:20.474 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:20.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:20.475 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:47:20.576 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:47:20.598 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:20.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:20.599 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:47:20.700 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:47:20.725 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:20.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:20.727 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:47:20.830 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:47:20.851 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:20.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:20.852 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:47:20.953 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:47:20.974 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:20.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:20.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:47:21.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:20 smithi032 ceph-mon[20004]: 9.0 scrub ok 2024-08-19T14:47:21.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:20 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2024-08-19T14:47:21.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:20 smithi032 ceph-mon[20004]: osdmap e902: 12 total, 12 up, 9 in 2024-08-19T14:47:21.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:20 smithi045 ceph-mon[20143]: 9.0 scrub ok 2024-08-19T14:47:21.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:20 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2024-08-19T14:47:21.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:20 smithi045 ceph-mon[20143]: osdmap e902: 12 total, 12 up, 9 in 2024-08-19T14:47:21.076 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:47:21.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:21.103 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:21.103 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:47:21.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:20 smithi031 ceph-mon[16336]: 9.0 scrub ok 2024-08-19T14:47:21.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:20 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2024-08-19T14:47:21.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:20 smithi031 ceph-mon[16336]: osdmap e902: 12 total, 12 up, 9 in 2024-08-19T14:47:21.171 INFO:tasks.thrashosds.thrasher:Added osd 7 2024-08-19T14:47:21.204 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:47:21.229 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:21.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:21.230 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:47:21.331 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:47:21.354 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:21.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:21.355 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:47:21.456 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:47:21.474 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-08-19T14:47:21.478 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:21.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:21.478 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:47:21.580 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:47:21.600 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:21.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:21.601 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:47:21.630 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:47:21.702 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:47:21.725 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:21.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:21.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:47:21.750 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:47:21.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:21 smithi045 ceph-mon[20143]: 1.f scrub starts 2024-08-19T14:47:21.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:21 smithi045 ceph-mon[20143]: 1.f scrub ok 2024-08-19T14:47:21.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:21 smithi045 ceph-mon[20143]: pgmap v2125: 47 pgs: 2 peering, 1 active+clean+scrubbing, 44 active+clean; 1.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 932 op/s 2024-08-19T14:47:21.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:21 smithi045 ceph-mon[20143]: osdmap e903: 12 total, 12 up, 9 in 2024-08-19T14:47:21.827 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:47:21.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:21.852 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:21.852 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:47:21.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:21 smithi031 ceph-mon[16336]: 1.f scrub starts 2024-08-19T14:47:21.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:21 smithi031 ceph-mon[16336]: 1.f scrub ok 2024-08-19T14:47:21.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:21 smithi031 ceph-mon[16336]: pgmap v2125: 47 pgs: 2 peering, 1 active+clean+scrubbing, 44 active+clean; 1.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 932 op/s 2024-08-19T14:47:21.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:21 smithi031 ceph-mon[16336]: osdmap e903: 12 total, 12 up, 9 in 2024-08-19T14:47:21.954 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:47:22.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:22.005 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:22.005 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:47:22.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:21 smithi032 ceph-mon[20004]: 1.f scrub starts 2024-08-19T14:47:22.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:21 smithi032 ceph-mon[20004]: 1.f scrub ok 2024-08-19T14:47:22.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:21 smithi032 ceph-mon[20004]: pgmap v2125: 47 pgs: 2 peering, 1 active+clean+scrubbing, 44 active+clean; 1.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 932 op/s 2024-08-19T14:47:22.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:21 smithi032 ceph-mon[20004]: osdmap e903: 12 total, 12 up, 9 in 2024-08-19T14:47:22.106 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:47:22.137 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:22.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:22.138 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:47:22.239 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:47:22.269 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:22.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:22.270 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:47:22.329 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:47:22.329 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:47:22.371 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:47:22.398 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:22.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:22.399 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:47:22.500 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:47:22.524 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:22.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:22.525 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:47:22.626 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:47:22.653 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:22.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:22.654 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:47:22.659 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is unset 2024-08-19T14:47:22.756 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:47:22.783 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:22.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:22.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:47:22.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:22 smithi045 ceph-mon[20143]: osdmap e904: 12 total, 12 up, 9 in 2024-08-19T14:47:22.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:22 smithi045 ceph-mon[20143]: 9.e starting backfill to osd.7 from (0'0,0'0] MAX to 901'10177 2024-08-19T14:47:22.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:22 smithi045 ceph-mon[20143]: 9.5 starting backfill to osd.7 from (0'0,0'0] MAX to 901'10154 2024-08-19T14:47:22.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:22 smithi045 ceph-mon[20143]: 9.8 starting backfill to osd.7 from (0'0,0'0] MAX to 901'10288 2024-08-19T14:47:22.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:22 smithi045 ceph-mon[20143]: 9.a starting backfill to osd.7 from (0'0,0'0] MAX to 901'10307 2024-08-19T14:47:22.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:22 smithi045 ceph-mon[20143]: 1.1d scrub starts 2024-08-19T14:47:22.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:22 smithi045 ceph-mon[20143]: 1.1d scrub ok 2024-08-19T14:47:22.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:22 smithi045 ceph-mon[20143]: 1.5 scrub starts 2024-08-19T14:47:22.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:22 smithi045 ceph-mon[20143]: 1.5 scrub ok 2024-08-19T14:47:22.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:22 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2860082167' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:47:22.886 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:47:22.928 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:22.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:22.929 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:47:23.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:22 smithi032 ceph-mon[20004]: osdmap e904: 12 total, 12 up, 9 in 2024-08-19T14:47:23.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:22 smithi032 ceph-mon[20004]: 9.e starting backfill to osd.7 from (0'0,0'0] MAX to 901'10177 2024-08-19T14:47:23.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:22 smithi032 ceph-mon[20004]: 9.5 starting backfill to osd.7 from (0'0,0'0] MAX to 901'10154 2024-08-19T14:47:23.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:22 smithi032 ceph-mon[20004]: 9.8 starting backfill to osd.7 from (0'0,0'0] MAX to 901'10288 2024-08-19T14:47:23.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:22 smithi032 ceph-mon[20004]: 9.a starting backfill to osd.7 from (0'0,0'0] MAX to 901'10307 2024-08-19T14:47:23.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:22 smithi032 ceph-mon[20004]: 1.1d scrub starts 2024-08-19T14:47:23.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:22 smithi032 ceph-mon[20004]: 1.1d scrub ok 2024-08-19T14:47:23.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:22 smithi032 ceph-mon[20004]: 1.5 scrub starts 2024-08-19T14:47:23.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:22 smithi032 ceph-mon[20004]: 1.5 scrub ok 2024-08-19T14:47:23.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:22 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2860082167' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:47:23.030 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:47:23.035 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:22 smithi031 ceph-mon[16336]: osdmap e904: 12 total, 12 up, 9 in 2024-08-19T14:47:23.035 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:22 smithi031 ceph-mon[16336]: 9.e starting backfill to osd.7 from (0'0,0'0] MAX to 901'10177 2024-08-19T14:47:23.036 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:22 smithi031 ceph-mon[16336]: 9.5 starting backfill to osd.7 from (0'0,0'0] MAX to 901'10154 2024-08-19T14:47:23.036 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:22 smithi031 ceph-mon[16336]: 9.8 starting backfill to osd.7 from (0'0,0'0] MAX to 901'10288 2024-08-19T14:47:23.036 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:22 smithi031 ceph-mon[16336]: 9.a starting backfill to osd.7 from (0'0,0'0] MAX to 901'10307 2024-08-19T14:47:23.036 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:22 smithi031 ceph-mon[16336]: 1.1d scrub starts 2024-08-19T14:47:23.036 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:22 smithi031 ceph-mon[16336]: 1.1d scrub ok 2024-08-19T14:47:23.036 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:22 smithi031 ceph-mon[16336]: 1.5 scrub starts 2024-08-19T14:47:23.037 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:22 smithi031 ceph-mon[16336]: 1.5 scrub ok 2024-08-19T14:47:23.037 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:22 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2860082167' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:47:23.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:23.082 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:23.082 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:47:23.101 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-08-19T14:47:23.184 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:47:23.211 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:23.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:23.212 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:47:23.313 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:47:23.341 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:23.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:23.342 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:47:23.444 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:47:23.498 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:23.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:23.499 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:47:23.594 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:47:23.600 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:47:23.606 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:47:23.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:23.668 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:23.668 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:47:23.769 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:47:23.805 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:23.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:23.806 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:47:23.907 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:47:23.920 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:23 smithi032 ceph-mon[20004]: pgmap v2128: 47 pgs: 2 remapped+peering, 2 peering, 1 active+clean+scrubbing, 42 active+clean; 1.6 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 580 op/s 2024-08-19T14:47:23.920 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:23 smithi032 ceph-mon[20004]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:47:23.920 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:23 smithi032 ceph-mon[20004]: Cluster is now healthy 2024-08-19T14:47:23.920 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:23 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2860082167' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:47:23.920 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:23 smithi032 ceph-mon[20004]: osdmap e905: 12 total, 12 up, 9 in 2024-08-19T14:47:23.933 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:23 smithi045 ceph-mon[20143]: pgmap v2128: 47 pgs: 2 remapped+peering, 2 peering, 1 active+clean+scrubbing, 42 active+clean; 1.6 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 580 op/s 2024-08-19T14:47:23.933 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:23 smithi045 ceph-mon[20143]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:47:23.933 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:23 smithi045 ceph-mon[20143]: Cluster is now healthy 2024-08-19T14:47:23.934 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:23 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2860082167' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:47:23.934 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:23 smithi045 ceph-mon[20143]: osdmap e905: 12 total, 12 up, 9 in 2024-08-19T14:47:23.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:23.942 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:23.942 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:47:23.966 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:47:24.044 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:47:24.068 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:24.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:24.069 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:47:24.098 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:23 smithi031 ceph-mon[16336]: pgmap v2128: 47 pgs: 2 remapped+peering, 2 peering, 1 active+clean+scrubbing, 42 active+clean; 1.6 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 580 op/s 2024-08-19T14:47:24.098 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:23 smithi031 ceph-mon[16336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:47:24.098 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:23 smithi031 ceph-mon[16336]: Cluster is now healthy 2024-08-19T14:47:24.098 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:23 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2860082167' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:47:24.098 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:23 smithi031 ceph-mon[16336]: osdmap e905: 12 total, 12 up, 9 in 2024-08-19T14:47:24.170 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:47:24.205 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:24.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:24.206 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:47:24.219 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:47:24.219 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:47:24.307 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:47:24.336 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:24.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:24.337 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:47:24.438 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:47:24.466 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:24.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:24.467 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:47:24.568 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:47:24.602 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:24.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:24.603 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:47:24.704 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:47:24.745 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:24.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:24.746 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:47:24.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:24 smithi045 ceph-mon[20143]: 1.1c scrub starts 2024-08-19T14:47:24.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:24 smithi045 ceph-mon[20143]: 1.1c scrub ok 2024-08-19T14:47:24.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:24 smithi045 ceph-mon[20143]: osdmap e906: 12 total, 12 up, 9 in 2024-08-19T14:47:24.846 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:47:24.876 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:24.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:24.877 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:47:24.948 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:24 smithi031 ceph-mon[16336]: 1.1c scrub starts 2024-08-19T14:47:24.948 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:24 smithi031 ceph-mon[16336]: 1.1c scrub ok 2024-08-19T14:47:24.948 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:24 smithi031 ceph-mon[16336]: osdmap e906: 12 total, 12 up, 9 in 2024-08-19T14:47:24.977 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:47:24.997 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-08-19T14:47:25.006 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:25.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:25.007 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:47:25.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:24 smithi032 ceph-mon[20004]: 1.1c scrub starts 2024-08-19T14:47:25.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:24 smithi032 ceph-mon[20004]: 1.1c scrub ok 2024-08-19T14:47:25.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:24 smithi032 ceph-mon[20004]: osdmap e906: 12 total, 12 up, 9 in 2024-08-19T14:47:25.108 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:47:25.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:25.176 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:25.177 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:47:25.278 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:47:25.307 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:25.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:25.308 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:47:25.379 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-19T14:47:25.409 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:47:25.436 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:25.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:25.437 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:47:25.441 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:47:25.537 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:47:25.567 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:25.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:25.568 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:47:25.668 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:47:25.695 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:25.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:25.696 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:47:25.797 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:47:25.824 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:25.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:25.825 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:47:25.903 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:25 smithi045 ceph-mon[20143]: pgmap v2131: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfilling, 2 remapped+peering, 4 peering, 36 active+clean; 1.5 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 2.5 KiB/s rd, 67 KiB/s wr, 436 op/s; 20/74391 objects degraded (0.027%); 6108/74391 objects misplaced (8.211%); 24 MiB/s, 386 objects/s recovering 2024-08-19T14:47:25.904 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:25 smithi045 ceph-mon[20143]: Health check failed: Degraded data redundancy: 20/74391 objects degraded (0.027%), 1 pg degraded (PG_DEGRADED) 2024-08-19T14:47:25.925 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:47:26.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:25 smithi032 ceph-mon[20004]: pgmap v2131: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfilling, 2 remapped+peering, 4 peering, 36 active+clean; 1.5 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 2.5 KiB/s rd, 67 KiB/s wr, 436 op/s; 20/74391 objects degraded (0.027%); 6108/74391 objects misplaced (8.211%); 24 MiB/s, 386 objects/s recovering 2024-08-19T14:47:26.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:25 smithi032 ceph-mon[20004]: Health check failed: Degraded data redundancy: 20/74391 objects degraded (0.027%), 1 pg degraded (PG_DEGRADED) 2024-08-19T14:47:26.008 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:26.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:26.009 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:47:26.039 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:25 smithi031 ceph-mon[16336]: pgmap v2131: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfilling, 2 remapped+peering, 4 peering, 36 active+clean; 1.5 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 2.5 KiB/s rd, 67 KiB/s wr, 436 op/s; 20/74391 objects degraded (0.027%); 6108/74391 objects misplaced (8.211%); 24 MiB/s, 386 objects/s recovering 2024-08-19T14:47:26.040 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:25 smithi031 ceph-mon[16336]: Health check failed: Degraded data redundancy: 20/74391 objects degraded (0.027%), 1 pg degraded (PG_DEGRADED) 2024-08-19T14:47:26.063 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:47:26.063 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:47:26.110 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:47:26.137 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:26.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:26.138 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:47:26.172 INFO:tasks.thrashosds.thrasher:in_osds: [1, 4, 8, 2, 11, 9, 10, 6, 7] out_osds: [3, 0, 5] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 8, 9, 11, 10, 2] 2024-08-19T14:47:26.173 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-19T14:47:26.173 INFO:tasks.thrashosds.thrasher:inject_pause on osd.0 2024-08-19T14:47:26.173 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-08-19T14:47:26.173 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-08-19T14:47:26.173 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.0 config set filestore_inject_stall 3 2024-08-19T14:47:26.238 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:47:26.282 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:26.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:26.283 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:47:26.384 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:47:26.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:26.428 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:26.428 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:47:26.529 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:47:26.558 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:26.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:26.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:47:26.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-08-19T14:47:26.037+0000 7f90e7fff640 0 --2- 172.21.15.31:0/1601083445 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f90e81018c0 0x7f90e819d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-08-19T14:47:26.577 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:47:26.577 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:47:26.577 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:47:26.577 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:47:26.577 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:47:26.577 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:47:26.578 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:47:26.578 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:47:26.578 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:47:26.578 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:47:26.578 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:47:26.578 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:47:26.578 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:47:26.578 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:47:26.578 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:47:26.578 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:47:26.578 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:47:26.579 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:47:26.579 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:47:26.579 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:47:26.579 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:47:26.579 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:47:26.579 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:47:26.579 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:47:26.660 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:47:26.684 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:26.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:26.685 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:47:26.786 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:47:26.821 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-08-19T14:47:26.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:26.824 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:26.824 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:47:26.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:26 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1601083445' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:47:26.925 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:47:26.946 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:26.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:26.947 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:47:27.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:26 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1601083445' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:47:27.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:26 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1601083445' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:47:27.047 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:47:27.068 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:27.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:27.069 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:47:27.171 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:47:27.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:27.193 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:27.194 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:47:27.199 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:47:27.294 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:47:27.318 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:27.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:27.319 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:47:27.420 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:47:27.441 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:27.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:27.443 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:47:27.544 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:47:27.569 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:27.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:27.570 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:47:27.671 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:47:27.696 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:27.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:27.697 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:47:27.768 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:47:27.768 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [], 2024-08-19T14:47:27.768 INFO:teuthology.orchestra.run.smithi045.stdout: "complaint_time": 30, 2024-08-19T14:47:27.768 INFO:teuthology.orchestra.run.smithi045.stdout: "num_blocked_ops": 0 2024-08-19T14:47:27.768 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:47:27.798 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:47:27.821 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:27.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:27.822 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:47:27.923 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:47:27.948 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:27.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:27.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:47:28.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:27 smithi032 ceph-mon[20004]: pgmap v2132: 47 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfilling, 2 peering, 39 active+clean; 1.5 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 1.7 KiB/s rd, 46 KiB/s wr, 465 op/s; 67/71595 objects degraded (0.094%); 7544/71595 objects misplaced (10.537%); 18 MiB/s, 286 objects/s recovering 2024-08-19T14:47:28.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:27 smithi032 ceph-mon[20004]: osdmap e907: 12 total, 12 up, 9 in 2024-08-19T14:47:28.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:27 smithi045 ceph-mon[20143]: pgmap v2132: 47 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfilling, 2 peering, 39 active+clean; 1.5 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 1.7 KiB/s rd, 46 KiB/s wr, 465 op/s; 67/71595 objects degraded (0.094%); 7544/71595 objects misplaced (10.537%); 18 MiB/s, 286 objects/s recovering 2024-08-19T14:47:28.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:27 smithi045 ceph-mon[20143]: osdmap e907: 12 total, 12 up, 9 in 2024-08-19T14:47:28.050 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:47:28.090 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:28.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:28.091 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:47:28.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:27 smithi031 ceph-mon[16336]: pgmap v2132: 47 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfilling, 2 peering, 39 active+clean; 1.5 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 1.7 KiB/s rd, 46 KiB/s wr, 465 op/s; 67/71595 objects degraded (0.094%); 7544/71595 objects misplaced (10.537%); 18 MiB/s, 286 objects/s recovering 2024-08-19T14:47:28.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:27 smithi031 ceph-mon[16336]: osdmap e907: 12 total, 12 up, 9 in 2024-08-19T14:47:28.191 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:47:28.231 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:28.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:28.232 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:47:28.333 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:47:28.380 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:28.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:28.381 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:47:28.480 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-08-19T14:47:28.481 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:47:28.506 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:28.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:28.507 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:47:28.608 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:47:28.636 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:28.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:28.637 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:47:28.738 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:47:28.761 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:28.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:28.763 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:47:28.864 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:47:28.886 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:28.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:28.887 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:47:28.940 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:47:28.988 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:47:29.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:28 smithi032 ceph-mon[20004]: 9.3 scrub starts 2024-08-19T14:47:29.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:28 smithi032 ceph-mon[20004]: 9.3 scrub ok 2024-08-19T14:47:29.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:28 smithi032 ceph-mon[20004]: osdmap e908: 12 total, 12 up, 9 in 2024-08-19T14:47:29.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:29.033 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:29.033 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:47:29.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:28 smithi045 ceph-mon[20143]: 9.3 scrub starts 2024-08-19T14:47:29.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:28 smithi045 ceph-mon[20143]: 9.3 scrub ok 2024-08-19T14:47:29.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:28 smithi045 ceph-mon[20143]: osdmap e908: 12 total, 12 up, 9 in 2024-08-19T14:47:29.085 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:28 smithi031 ceph-mon[16336]: 9.3 scrub starts 2024-08-19T14:47:29.086 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:28 smithi031 ceph-mon[16336]: 9.3 scrub ok 2024-08-19T14:47:29.086 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:28 smithi031 ceph-mon[16336]: osdmap e908: 12 total, 12 up, 9 in 2024-08-19T14:47:29.134 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:47:29.160 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:29.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:29.161 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:47:29.262 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:47:29.287 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:29.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:29.288 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:47:29.388 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:47:29.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:29.415 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:29.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:47:29.516 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:47:29.555 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:47:29.556 INFO:teuthology.orchestra.run.smithi045.stdout: "size": 20, 2024-08-19T14:47:29.556 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 600, 2024-08-19T14:47:29.556 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [ 2024-08-19T14:47:29.556 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.556 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDPGPush(8.9 748/737 [PushOp(8:91183c5c:::benchmark_data_smithi045_137103_object46530:head, version: 724'2924, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:91183c5c:::benchmark_data_smithi045_137103_object46530:head@724'2924, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-19T14:47:29.556 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:41:18.572524+0000", 2024-08-19T14:47:29.556 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 370.97155551600002, 2024-08-19T14:47:29.556 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.080170134000000004, 2024-08-19T14:47:29.556 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:47:29.556 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:47:29.556 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-08-19T14:47:29.556 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:47:29.556 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.557 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:47:29.557 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:41:18.572524+0000", 2024-08-19T14:47:29.557 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:47:29.557 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.557 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.557 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:47:29.557 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:41:18.572524+0000", 2024-08-19T14:47:29.557 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:47:29.557 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.557 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.557 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:47:29.557 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:41:18.572524+0000", 2024-08-19T14:47:29.557 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.0800000000000004e-07 2024-08-19T14:47:29.558 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.558 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.558 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:47:29.558 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:41:18.572551+0000", 2024-08-19T14:47:29.558 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.6672000000000001e-05 2024-08-19T14:47:29.558 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.558 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.558 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:47:29.558 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:41:18.572565+0000", 2024-08-19T14:47:29.558 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4409999999999999e-05 2024-08-19T14:47:29.558 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.558 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.558 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:47:29.558 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:41:18.572575+0000", 2024-08-19T14:47:29.559 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.6190000000000006e-06 2024-08-19T14:47:29.559 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.559 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.559 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:47:29.559 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:41:18.572631+0000", 2024-08-19T14:47:29.559 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.5779999999999998e-05 2024-08-19T14:47:29.559 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.559 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.559 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:47:29.559 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:41:18.572638+0000", 2024-08-19T14:47:29.559 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.7270000000000001e-06 2024-08-19T14:47:29.559 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.559 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.560 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:47:29.560 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:41:18.652694+0000", 2024-08-19T14:47:29.560 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.080056318000000001 2024-08-19T14:47:29.560 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:47:29.560 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:47:29.560 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:47:29.560 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.560 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.560 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDPGPush(8.15 748/734 [PushOp(8:a8b5dffc:::benchmark_data_smithi045_137103_object45110:head, version: 724'2855, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:a8b5dffc:::benchmark_data_smithi045_137103_object45110:head@724'2855, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-19T14:47:29.560 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:41:18.574073+0000", 2024-08-19T14:47:29.560 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 370.97000680899998, 2024-08-19T14:47:29.560 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.079298409, 2024-08-19T14:47:29.560 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:47:29.561 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:47:29.561 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-08-19T14:47:29.561 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:47:29.561 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.561 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:47:29.561 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:41:18.574073+0000", 2024-08-19T14:47:29.561 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:47:29.561 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.561 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.561 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:47:29.561 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:41:18.574073+0000", 2024-08-19T14:47:29.561 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:47:29.561 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.561 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.562 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:47:29.562 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:41:18.574073+0000", 2024-08-19T14:47:29.562 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.6000000000000004e-07 2024-08-19T14:47:29.562 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.562 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.562 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:47:29.562 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:41:18.574101+0000", 2024-08-19T14:47:29.562 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.7673000000000001e-05 2024-08-19T14:47:29.562 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.562 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.562 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:47:29.562 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:41:18.651141+0000", 2024-08-19T14:47:29.562 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.077040569000000003 2024-08-19T14:47:29.563 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.563 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.563 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:47:29.563 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:41:18.651165+0000", 2024-08-19T14:47:29.563 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.3215000000000001e-05 2024-08-19T14:47:29.563 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.563 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.563 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:47:29.563 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:41:18.651253+0000", 2024-08-19T14:47:29.563 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.8805000000000001e-05 2024-08-19T14:47:29.563 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.563 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.563 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:47:29.563 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:41:18.651268+0000", 2024-08-19T14:47:29.563 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4494999999999999e-05 2024-08-19T14:47:29.564 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.564 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.564 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:47:29.564 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:41:18.653371+0000", 2024-08-19T14:47:29.564 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.002103092 2024-08-19T14:47:29.564 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:47:29.564 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:47:29.564 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:47:29.564 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.564 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.564 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.27235.0:49914 9.6 e847/835 9:6f2c8487:::benchmark_data_smithi045_152862_object49913:head v 847'3132, mlcod=832'1328)", 2024-08-19T14:47:29.564 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:45:00.500221+0000", 2024-08-19T14:47:29.564 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 149.04385872099999, 2024-08-19T14:47:29.564 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.15332016200000001, 2024-08-19T14:47:29.565 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:47:29.565 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:47:29.565 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:47:29.565 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:47:29.565 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.565 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:47:29.565 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:00.500221+0000", 2024-08-19T14:47:29.565 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:47:29.565 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.565 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.565 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:47:29.565 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:00.500221+0000", 2024-08-19T14:47:29.565 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:47:29.565 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.566 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.566 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:47:29.566 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:00.500221+0000", 2024-08-19T14:47:29.566 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.7299999999999995e-07 2024-08-19T14:47:29.566 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.566 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.566 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:47:29.566 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:00.500273+0000", 2024-08-19T14:47:29.566 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1486999999999997e-05 2024-08-19T14:47:29.566 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.566 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.566 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:47:29.566 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:00.500284+0000", 2024-08-19T14:47:29.566 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1649e-05 2024-08-19T14:47:29.567 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.567 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.567 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:47:29.567 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:00.500292+0000", 2024-08-19T14:47:29.567 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4410000000000004e-06 2024-08-19T14:47:29.567 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.567 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.567 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:47:29.567 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:00.500762+0000", 2024-08-19T14:47:29.567 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00047051800000000001 2024-08-19T14:47:29.567 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.567 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.567 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:47:29.567 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:00.500780+0000", 2024-08-19T14:47:29.568 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7672999999999998e-05 2024-08-19T14:47:29.568 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.568 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.568 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:47:29.568 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:00.653502+0000", 2024-08-19T14:47:29.568 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.15272184 2024-08-19T14:47:29.568 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.568 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.568 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:47:29.568 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:00.653541+0000", 2024-08-19T14:47:29.568 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.8881000000000003e-05 2024-08-19T14:47:29.568 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:47:29.568 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:47:29.568 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:47:29.569 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.569 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.569 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDPGPush(9.5 847/835 [PushOp(9:a3c0faf2:::benchmark_data_smithi045_152862_object24448:head, version: 832'1502, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(9:a3c0faf2:::benchmark_data_smithi045_152862_object24448:head@832'1502, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-19T14:47:29.569 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:45:00.501127+0000", 2024-08-19T14:47:29.569 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 149.04295270200001, 2024-08-19T14:47:29.569 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.15405698900000001, 2024-08-19T14:47:29.569 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:47:29.569 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:47:29.569 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-08-19T14:47:29.569 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:47:29.569 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.569 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:47:29.569 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:00.501127+0000", 2024-08-19T14:47:29.570 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:47:29.570 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.570 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.570 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:47:29.570 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:00.501127+0000", 2024-08-19T14:47:29.570 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:47:29.570 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.570 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.570 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:47:29.570 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:00.501127+0000", 2024-08-19T14:47:29.570 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.9400000000000005e-07 2024-08-19T14:47:29.570 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.570 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.570 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:47:29.571 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:00.501156+0000", 2024-08-19T14:47:29.571 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.9014999999999999e-05 2024-08-19T14:47:29.571 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.571 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.571 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:47:29.571 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:00.501176+0000", 2024-08-19T14:47:29.571 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9392999999999999e-05 2024-08-19T14:47:29.571 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.571 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.571 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:47:29.571 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:00.501193+0000", 2024-08-19T14:47:29.571 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6847e-05 2024-08-19T14:47:29.571 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.571 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.572 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:47:29.572 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:00.653442+0000", 2024-08-19T14:47:29.572 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.15224958699999999 2024-08-19T14:47:29.572 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.572 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.572 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:47:29.572 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:00.653476+0000", 2024-08-19T14:47:29.572 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.3470000000000003e-05 2024-08-19T14:47:29.572 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.572 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.572 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:47:29.572 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:00.655184+0000", 2024-08-19T14:47:29.572 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0017079829999999999 2024-08-19T14:47:29.572 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:47:29.573 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:47:29.573 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:47:29.573 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.573 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.573 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.27235.0:110977 9.2 e874/826 9:4d0b4ae9:::benchmark_data_smithi045_152862_object16472:head v 874'7018, mlcod=874'7016)", 2024-08-19T14:47:29.573 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:46:11.288985+0000", 2024-08-19T14:47:29.573 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 78.255094142999994, 2024-08-19T14:47:29.573 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.13314642700000001, 2024-08-19T14:47:29.573 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:47:29.573 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:47:29.573 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:47:29.573 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:47:29.573 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.573 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:47:29.574 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.288985+0000", 2024-08-19T14:47:29.574 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:47:29.574 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.574 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.574 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:47:29.574 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.288985+0000", 2024-08-19T14:47:29.574 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:47:29.574 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.574 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.574 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:47:29.574 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.288986+0000", 2024-08-19T14:47:29.574 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.0999999999999998e-07 2024-08-19T14:47:29.574 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.575 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.575 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:47:29.575 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.288993+0000", 2024-08-19T14:47:29.575 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.8889999999999996e-06 2024-08-19T14:47:29.575 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.575 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.575 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:47:29.575 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.289007+0000", 2024-08-19T14:47:29.575 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4718999999999999e-05 2024-08-19T14:47:29.575 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.575 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.575 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:47:29.575 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.289014+0000", 2024-08-19T14:47:29.575 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.8419999999999999e-06 2024-08-19T14:47:29.576 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.576 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.576 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:47:29.576 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.289251+0000", 2024-08-19T14:47:29.576 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00023693299999999999 2024-08-19T14:47:29.576 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.576 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.576 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:47:29.576 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.289298+0000", 2024-08-19T14:47:29.576 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.6567999999999999e-05 2024-08-19T14:47:29.576 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.576 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.576 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:47:29.576 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.422096+0000", 2024-08-19T14:47:29.577 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.13279822899999999 2024-08-19T14:47:29.577 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.577 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.577 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:47:29.577 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.422132+0000", 2024-08-19T14:47:29.577 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.5537000000000002e-05 2024-08-19T14:47:29.577 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:47:29.577 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:47:29.577 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:47:29.577 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.577 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.577 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.27235.0:110976 9.1 e874/863 9:8b7e4231:::benchmark_data_smithi045_152862_object16471:head v 874'7044, mlcod=855'3468)", 2024-08-19T14:47:29.577 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:46:11.290138+0000", 2024-08-19T14:47:29.577 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 78.253941252999994, 2024-08-19T14:47:29.578 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.131998166, 2024-08-19T14:47:29.578 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:47:29.578 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:47:29.578 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:47:29.578 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:47:29.578 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.578 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:47:29.578 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.290138+0000", 2024-08-19T14:47:29.578 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:47:29.578 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.578 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.578 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:47:29.578 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.290138+0000", 2024-08-19T14:47:29.578 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:47:29.579 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.579 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.579 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:47:29.579 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.290138+0000", 2024-08-19T14:47:29.579 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.2199999999999999e-07 2024-08-19T14:47:29.579 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.579 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.579 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:47:29.579 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.290143+0000", 2024-08-19T14:47:29.579 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.7509999999999999e-06 2024-08-19T14:47:29.579 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.579 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.579 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:47:29.579 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.290151+0000", 2024-08-19T14:47:29.580 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.1100000000000003e-06 2024-08-19T14:47:29.580 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.580 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.580 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:47:29.580 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.290157+0000", 2024-08-19T14:47:29.580 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.5149999999999997e-06 2024-08-19T14:47:29.580 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.580 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.580 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:47:29.580 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.290325+0000", 2024-08-19T14:47:29.580 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00016795800000000001 2024-08-19T14:47:29.580 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.580 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.580 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:47:29.581 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.290375+0000", 2024-08-19T14:47:29.581 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0392000000000001e-05 2024-08-19T14:47:29.581 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.581 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.581 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:47:29.581 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.422081+0000", 2024-08-19T14:47:29.581 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.131705509 2024-08-19T14:47:29.581 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.581 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.581 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:47:29.581 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.422136+0000", 2024-08-19T14:47:29.581 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.5509e-05 2024-08-19T14:47:29.581 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:47:29.581 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:47:29.582 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:47:29.582 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.582 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.582 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.27235.0:110978 9.2 e874/826 9:4e044f6b:::benchmark_data_smithi045_152862_object16473:head v 874'7019, mlcod=874'7016)", 2024-08-19T14:47:29.582 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:46:11.291231+0000", 2024-08-19T14:47:29.582 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 78.252847989000003, 2024-08-19T14:47:29.582 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.13111898, 2024-08-19T14:47:29.582 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:47:29.582 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:47:29.582 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:47:29.582 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:47:29.582 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.582 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:47:29.583 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.291231+0000", 2024-08-19T14:47:29.583 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:47:29.583 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.583 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.583 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:47:29.583 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.291231+0000", 2024-08-19T14:47:29.583 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:47:29.583 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.583 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.583 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:47:29.583 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.291232+0000", 2024-08-19T14:47:29.583 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.9500000000000002e-07 2024-08-19T14:47:29.583 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.583 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.584 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:47:29.584 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.291238+0000", 2024-08-19T14:47:29.584 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.9490000000000001e-06 2024-08-19T14:47:29.584 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.584 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.584 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:47:29.584 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.291249+0000", 2024-08-19T14:47:29.584 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0535e-05 2024-08-19T14:47:29.584 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.584 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.584 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:47:29.584 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.291255+0000", 2024-08-19T14:47:29.584 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.207e-06 2024-08-19T14:47:29.584 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.585 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.585 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:47:29.585 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.299464+0000", 2024-08-19T14:47:29.585 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0082091969999999997 2024-08-19T14:47:29.585 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.585 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.585 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:47:29.585 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.299486+0000", 2024-08-19T14:47:29.585 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1656000000000001e-05 2024-08-19T14:47:29.585 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.585 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.585 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:47:29.585 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.422243+0000", 2024-08-19T14:47:29.585 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.122757835 2024-08-19T14:47:29.586 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.586 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.586 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:47:29.586 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.422350+0000", 2024-08-19T14:47:29.586 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000106906 2024-08-19T14:47:29.586 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:47:29.586 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:47:29.586 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:47:29.586 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.586 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.586 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.27235.0:110981 9.3 e874/863 9:cce5e729:::benchmark_data_smithi045_152862_object16476:head v 874'6991, mlcod=833'1885)", 2024-08-19T14:47:29.586 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:46:11.297665+0000", 2024-08-19T14:47:29.586 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 78.246414690999998, 2024-08-19T14:47:29.586 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.124690812, 2024-08-19T14:47:29.587 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:47:29.587 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:47:29.587 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:47:29.587 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:47:29.587 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.587 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:47:29.587 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.297665+0000", 2024-08-19T14:47:29.587 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:47:29.587 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.587 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.587 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:47:29.587 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.297665+0000", 2024-08-19T14:47:29.587 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:47:29.587 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.588 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.588 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:47:29.588 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.297665+0000", 2024-08-19T14:47:29.588 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.0999999999999998e-07 2024-08-19T14:47:29.588 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.588 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.588 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:47:29.588 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.297671+0000", 2024-08-19T14:47:29.588 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.806e-06 2024-08-19T14:47:29.588 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.588 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.588 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:47:29.588 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.297685+0000", 2024-08-19T14:47:29.588 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3852999999999999e-05 2024-08-19T14:47:29.589 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.589 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.589 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:47:29.589 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.297692+0000", 2024-08-19T14:47:29.589 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.6209999999999998e-06 2024-08-19T14:47:29.589 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.589 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.589 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:47:29.589 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.297865+0000", 2024-08-19T14:47:29.589 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00017351700000000001 2024-08-19T14:47:29.589 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.589 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.589 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:47:29.589 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.297950+0000", 2024-08-19T14:47:29.590 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.4494999999999997e-05 2024-08-19T14:47:29.590 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.590 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.590 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:47:29.590 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.422308+0000", 2024-08-19T14:47:29.590 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.124358341 2024-08-19T14:47:29.590 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.590 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.590 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:47:29.590 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.422355+0000", 2024-08-19T14:47:29.590 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.7469000000000002e-05 2024-08-19T14:47:29.590 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:47:29.590 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:47:29.590 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:47:29.591 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.591 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.591 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.27235.0:110987 9.1 e874/863 9:862d1f71:::benchmark_data_smithi045_152862_object16482:head v 874'7045, mlcod=855'3471)", 2024-08-19T14:47:29.591 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:46:11.300675+0000", 2024-08-19T14:47:29.591 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 78.243404308999999, 2024-08-19T14:47:29.591 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.121615643, 2024-08-19T14:47:29.591 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:47:29.591 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:47:29.591 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:47:29.591 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:47:29.591 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.591 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:47:29.591 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.300675+0000", 2024-08-19T14:47:29.591 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:47:29.592 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.592 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.592 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:47:29.592 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.300675+0000", 2024-08-19T14:47:29.592 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:47:29.592 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.592 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.592 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:47:29.592 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.300676+0000", 2024-08-19T14:47:29.592 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.0200000000000001e-07 2024-08-19T14:47:29.592 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.592 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.592 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:47:29.592 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.300682+0000", 2024-08-19T14:47:29.593 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.9089999999999997e-06 2024-08-19T14:47:29.593 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.593 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.593 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:47:29.593 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.300697+0000", 2024-08-19T14:47:29.593 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5412000000000001e-05 2024-08-19T14:47:29.593 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.593 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.593 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:47:29.593 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.300704+0000", 2024-08-19T14:47:29.593 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.6630000000000004e-06 2024-08-19T14:47:29.593 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.593 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.593 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:47:29.593 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.300861+0000", 2024-08-19T14:47:29.594 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000156732 2024-08-19T14:47:29.594 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.594 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.594 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:47:29.594 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.300926+0000", 2024-08-19T14:47:29.594 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.4991000000000002e-05 2024-08-19T14:47:29.594 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.594 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.594 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:47:29.594 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.422260+0000", 2024-08-19T14:47:29.594 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.12133408399999999 2024-08-19T14:47:29.594 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.594 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.595 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:47:29.595 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.422291+0000", 2024-08-19T14:47:29.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:29.596 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:29.596 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:47:29.597 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.1050000000000003e-05 2024-08-19T14:47:29.597 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:47:29.597 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:47:29.597 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:47:29.597 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.597 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.597 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.27235.0:110983 9.3 e874/863 9:c59963d5:::benchmark_data_smithi045_152862_object16478:head v 874'6992, mlcod=833'1885)", 2024-08-19T14:47:29.597 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:46:11.302249+0000", 2024-08-19T14:47:29.597 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 78.241829844999998, 2024-08-19T14:47:29.598 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.120135223, 2024-08-19T14:47:29.598 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:47:29.598 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:47:29.598 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:47:29.598 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:47:29.598 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.598 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:47:29.598 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.302249+0000", 2024-08-19T14:47:29.598 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:47:29.598 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.598 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.598 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:47:29.598 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.302249+0000", 2024-08-19T14:47:29.598 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:47:29.599 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.599 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.599 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:47:29.599 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.302250+0000", 2024-08-19T14:47:29.599 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.0800000000000004e-07 2024-08-19T14:47:29.599 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.599 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.599 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:47:29.599 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.302257+0000", 2024-08-19T14:47:29.599 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.6420000000000001e-06 2024-08-19T14:47:29.599 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.599 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.599 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:47:29.599 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.302272+0000", 2024-08-19T14:47:29.600 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5231e-05 2024-08-19T14:47:29.600 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.600 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.600 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:47:29.600 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.302279+0000", 2024-08-19T14:47:29.600 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.7469999999999999e-06 2024-08-19T14:47:29.600 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.600 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.600 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:47:29.600 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.304054+0000", 2024-08-19T14:47:29.600 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001775034 2024-08-19T14:47:29.600 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.600 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.600 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:47:29.601 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.304078+0000", 2024-08-19T14:47:29.601 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.4451999999999999e-05 2024-08-19T14:47:29.601 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.601 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.601 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:47:29.601 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.422358+0000", 2024-08-19T14:47:29.601 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.118279911 2024-08-19T14:47:29.601 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.601 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.601 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:47:29.601 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.422385+0000", 2024-08-19T14:47:29.601 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.6497999999999999e-05 2024-08-19T14:47:29.601 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:47:29.601 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:47:29.602 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:47:29.602 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.602 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.602 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.27235.0:110984 9.e e874/826 9:7e2a3b1d:::benchmark_data_smithi045_152862_object16479:head v 874'6891, mlcod=874'6886)", 2024-08-19T14:47:29.602 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:46:11.302301+0000", 2024-08-19T14:47:29.602 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 78.241778042000007, 2024-08-19T14:47:29.602 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.120168275, 2024-08-19T14:47:29.602 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:47:29.602 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:47:29.602 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:47:29.602 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:47:29.602 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.602 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:47:29.603 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.302301+0000", 2024-08-19T14:47:29.603 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:47:29.603 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.603 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.603 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:47:29.603 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.302301+0000", 2024-08-19T14:47:29.603 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:47:29.603 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.603 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.603 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:47:29.603 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.302302+0000", 2024-08-19T14:47:29.603 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.5400000000000002e-07 2024-08-19T14:47:29.603 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.603 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.603 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:47:29.604 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.302306+0000", 2024-08-19T14:47:29.604 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.0969999999999999e-06 2024-08-19T14:47:29.604 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.604 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.604 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:47:29.604 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.302312+0000", 2024-08-19T14:47:29.604 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.6139999999999997e-06 2024-08-19T14:47:29.604 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.604 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.604 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:47:29.604 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.302317+0000", 2024-08-19T14:47:29.604 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.617e-06 2024-08-19T14:47:29.604 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.604 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.605 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:47:29.605 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.302428+0000", 2024-08-19T14:47:29.605 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000111073 2024-08-19T14:47:29.605 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.605 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.605 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:47:29.605 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.302447+0000", 2024-08-19T14:47:29.605 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8853000000000001e-05 2024-08-19T14:47:29.605 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.605 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.605 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:47:29.605 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.422438+0000", 2024-08-19T14:47:29.605 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.119991364 2024-08-19T14:47:29.605 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.606 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.606 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:47:29.606 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.422470+0000", 2024-08-19T14:47:29.606 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.1303000000000002e-05 2024-08-19T14:47:29.606 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:47:29.606 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:47:29.606 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:47:29.606 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.606 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.606 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.27235.0:110989 9.6 e874/853 9:6840f520:::benchmark_data_smithi045_152862_object16484:head v 874'6931, mlcod=874'6928)", 2024-08-19T14:47:29.606 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:46:11.306400+0000", 2024-08-19T14:47:29.606 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 78.237679647999997, 2024-08-19T14:47:29.606 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.11609773600000001, 2024-08-19T14:47:29.606 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:47:29.607 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:47:29.607 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:47:29.607 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:47:29.607 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.607 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:47:29.607 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.306400+0000", 2024-08-19T14:47:29.607 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:47:29.607 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.607 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.607 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:47:29.607 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.306400+0000", 2024-08-19T14:47:29.607 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:47:29.607 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.607 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.608 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:47:29.608 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.306401+0000", 2024-08-19T14:47:29.608 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.5600000000000004e-07 2024-08-19T14:47:29.608 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.608 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.608 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:47:29.608 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.306408+0000", 2024-08-19T14:47:29.608 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.498e-06 2024-08-19T14:47:29.608 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.608 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.608 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:47:29.608 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.306419+0000", 2024-08-19T14:47:29.608 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1e-05 2024-08-19T14:47:29.608 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.609 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.609 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:47:29.609 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.306425+0000", 2024-08-19T14:47:29.609 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.1929999999999998e-06 2024-08-19T14:47:29.609 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.609 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.609 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:47:29.609 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.306548+0000", 2024-08-19T14:47:29.609 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012242500000000001 2024-08-19T14:47:29.609 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.609 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.609 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:47:29.609 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.306565+0000", 2024-08-19T14:47:29.609 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7367000000000002e-05 2024-08-19T14:47:29.610 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.610 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.610 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:47:29.610 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.422472+0000", 2024-08-19T14:47:29.610 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.11590706100000001 2024-08-19T14:47:29.610 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.610 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.610 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:47:29.610 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.422497+0000", 2024-08-19T14:47:29.610 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.5335999999999999e-05 2024-08-19T14:47:29.610 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:47:29.610 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:47:29.610 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:47:29.610 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.611 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.611 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.27235.0:110992 9.8 e874/852 9:105ab922:::benchmark_data_smithi045_152862_object16487:head v 874'7025, mlcod=874'7023)", 2024-08-19T14:47:29.611 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:46:11.311968+0000", 2024-08-19T14:47:29.611 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 78.232111720000006, 2024-08-19T14:47:29.611 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.11043821700000001, 2024-08-19T14:47:29.611 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:47:29.611 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:47:29.611 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:47:29.611 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:47:29.611 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.611 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:47:29.611 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.311968+0000", 2024-08-19T14:47:29.611 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:47:29.611 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.612 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.612 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:47:29.612 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.311968+0000", 2024-08-19T14:47:29.612 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:47:29.612 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.612 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.612 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:47:29.612 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.311969+0000", 2024-08-19T14:47:29.612 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3e-06 2024-08-19T14:47:29.612 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.612 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.612 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:47:29.612 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.311978+0000", 2024-08-19T14:47:29.612 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.9819999999999997e-06 2024-08-19T14:47:29.613 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.613 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.613 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:47:29.613 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.311996+0000", 2024-08-19T14:47:29.613 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7975e-05 2024-08-19T14:47:29.613 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.613 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.613 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:47:29.613 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.312004+0000", 2024-08-19T14:47:29.613 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.7279999999999995e-06 2024-08-19T14:47:29.613 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.613 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.613 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:47:29.613 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.312347+0000", 2024-08-19T14:47:29.613 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00034309 2024-08-19T14:47:29.614 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.614 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.614 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:47:29.614 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.312365+0000", 2024-08-19T14:47:29.614 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8604999999999999e-05 2024-08-19T14:47:29.614 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.614 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.614 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:47:29.614 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.422377+0000", 2024-08-19T14:47:29.614 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.110011678 2024-08-19T14:47:29.614 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.614 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.614 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:47:29.615 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.422406+0000", 2024-08-19T14:47:29.615 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.8858999999999999e-05 2024-08-19T14:47:29.615 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:47:29.615 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:47:29.615 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:47:29.615 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.615 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.615 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.27235.0:110993 9.5 e874/851 9:aa4ad7f7:::benchmark_data_smithi045_152862_object16488:head v 874'6849, mlcod=874'6847)", 2024-08-19T14:47:29.615 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:46:11.312026+0000", 2024-08-19T14:47:29.615 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 78.232053140000005, 2024-08-19T14:47:29.615 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.110391424, 2024-08-19T14:47:29.615 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:47:29.615 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:47:29.615 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:47:29.616 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:47:29.616 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.616 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:47:29.616 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.312026+0000", 2024-08-19T14:47:29.616 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:47:29.616 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.616 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.616 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:47:29.616 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.312026+0000", 2024-08-19T14:47:29.616 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:47:29.616 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.616 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.616 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:47:29.616 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.312026+0000", 2024-08-19T14:47:29.617 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.8700000000000002e-07 2024-08-19T14:47:29.617 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.617 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.617 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:47:29.617 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.312031+0000", 2024-08-19T14:47:29.617 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.4490000000000001e-06 2024-08-19T14:47:29.617 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.617 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.617 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:47:29.617 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.312037+0000", 2024-08-19T14:47:29.617 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.178e-06 2024-08-19T14:47:29.617 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.617 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.617 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:47:29.618 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.312041+0000", 2024-08-19T14:47:29.618 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.3000000000000003e-06 2024-08-19T14:47:29.618 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.618 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.618 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:47:29.618 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.312090+0000", 2024-08-19T14:47:29.618 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.8971999999999998e-05 2024-08-19T14:47:29.618 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.618 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.618 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:47:29.618 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.312115+0000", 2024-08-19T14:47:29.618 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.4842999999999999e-05 2024-08-19T14:47:29.618 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.618 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.619 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:47:29.619 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.422327+0000", 2024-08-19T14:47:29.619 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.110212055 2024-08-19T14:47:29.619 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.619 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.619 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:47:29.619 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.422418+0000", 2024-08-19T14:47:29.619 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.0340000000000006e-05 2024-08-19T14:47:29.619 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:47:29.619 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:47:29.619 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:47:29.619 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.619 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.619 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.27235.0:110996 9.2 e874/826 9:4cd0604d:::benchmark_data_smithi045_152862_object16491:head v 874'7020, mlcod=874'7016)", 2024-08-19T14:47:29.620 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:46:11.314098+0000", 2024-08-19T14:47:29.620 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 78.229981542999994, 2024-08-19T14:47:29.620 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.10836283200000001, 2024-08-19T14:47:29.620 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:47:29.620 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:47:29.620 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:47:29.620 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:47:29.620 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.620 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:47:29.620 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.314098+0000", 2024-08-19T14:47:29.620 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:47:29.620 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.620 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.620 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:47:29.621 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.314098+0000", 2024-08-19T14:47:29.621 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:47:29.621 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.621 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.621 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:47:29.621 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.314099+0000", 2024-08-19T14:47:29.621 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.8100000000000001e-07 2024-08-19T14:47:29.621 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.621 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.621 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:47:29.621 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.314107+0000", 2024-08-19T14:47:29.621 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.4530000000000002e-06 2024-08-19T14:47:29.621 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.621 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.621 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:47:29.622 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.314134+0000", 2024-08-19T14:47:29.622 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.6985999999999999e-05 2024-08-19T14:47:29.622 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.622 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.622 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:47:29.622 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.314142+0000", 2024-08-19T14:47:29.622 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.8509999999999999e-06 2024-08-19T14:47:29.622 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.622 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.622 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:47:29.622 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.314216+0000", 2024-08-19T14:47:29.622 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.3802999999999998e-05 2024-08-19T14:47:29.622 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.622 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.623 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:47:29.623 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.314235+0000", 2024-08-19T14:47:29.623 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8729999999999999e-05 2024-08-19T14:47:29.623 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.623 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.623 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:47:29.623 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.422354+0000", 2024-08-19T14:47:29.623 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.108119837 2024-08-19T14:47:29.623 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.623 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.623 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:47:29.623 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.422461+0000", 2024-08-19T14:47:29.623 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000106191 2024-08-19T14:47:29.623 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:47:29.624 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:47:29.624 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:47:29.624 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.624 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.624 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.27235.0:110997 9.e e874/826 9:76c005d7:::benchmark_data_smithi045_152862_object16492:head v 874'6892, mlcod=874'6889)", 2024-08-19T14:47:29.624 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:46:11.316432+0000", 2024-08-19T14:47:29.624 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 78.227647610000005, 2024-08-19T14:47:29.624 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.106135057, 2024-08-19T14:47:29.624 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:47:29.624 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:47:29.624 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:47:29.624 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:47:29.624 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.624 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:47:29.624 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.316432+0000", 2024-08-19T14:47:29.625 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:47:29.625 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.625 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.625 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:47:29.625 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.316432+0000", 2024-08-19T14:47:29.625 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:47:29.625 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.625 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.625 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:47:29.625 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.316433+0000", 2024-08-19T14:47:29.625 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.3799999999999996e-07 2024-08-19T14:47:29.625 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.625 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.626 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:47:29.626 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.316441+0000", 2024-08-19T14:47:29.626 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.2290000000000003e-06 2024-08-19T14:47:29.626 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.626 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.626 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:47:29.626 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.316456+0000", 2024-08-19T14:47:29.626 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4708e-05 2024-08-19T14:47:29.626 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.626 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.626 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:47:29.626 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.316463+0000", 2024-08-19T14:47:29.626 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4789999999999999e-06 2024-08-19T14:47:29.626 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.627 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.627 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:47:29.627 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.316653+0000", 2024-08-19T14:47:29.627 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00018986400000000001 2024-08-19T14:47:29.627 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.627 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.627 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:47:29.627 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.316699+0000", 2024-08-19T14:47:29.627 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.5708999999999999e-05 2024-08-19T14:47:29.627 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.627 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.627 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:47:29.627 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.422499+0000", 2024-08-19T14:47:29.627 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.105800725 2024-08-19T14:47:29.628 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.628 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.628 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:47:29.628 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.422567+0000", 2024-08-19T14:47:29.628 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.7404999999999996e-05 2024-08-19T14:47:29.628 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:47:29.628 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:47:29.628 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:47:29.628 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.628 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.628 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.27235.0:110998 9.3 e874/863 9:c6334fed:::benchmark_data_smithi045_152862_object16493:head v 874'6993, mlcod=833'1886)", 2024-08-19T14:47:29.628 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:46:11.318166+0000", 2024-08-19T14:47:29.628 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 78.225913614999996, 2024-08-19T14:47:29.628 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.10443463, 2024-08-19T14:47:29.629 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:47:29.629 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:47:29.629 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:47:29.629 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:47:29.629 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.629 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:47:29.629 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.318166+0000", 2024-08-19T14:47:29.629 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:47:29.629 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.629 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.629 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:47:29.629 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.318166+0000", 2024-08-19T14:47:29.629 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:47:29.629 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.629 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.630 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:47:29.630 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.318167+0000", 2024-08-19T14:47:29.630 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.2900000000000002e-07 2024-08-19T14:47:29.630 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.630 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.630 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:47:29.630 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.318174+0000", 2024-08-19T14:47:29.630 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.9120000000000001e-06 2024-08-19T14:47:29.630 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.630 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.630 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:47:29.630 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.318186+0000", 2024-08-19T14:47:29.630 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2797e-05 2024-08-19T14:47:29.631 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.631 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.631 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:47:29.631 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.318193+0000", 2024-08-19T14:47:29.631 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.7669999999999998e-06 2024-08-19T14:47:29.631 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.631 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.631 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:47:29.631 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.318285+0000", 2024-08-19T14:47:29.631 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.1818999999999996e-05 2024-08-19T14:47:29.631 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.631 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.631 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:47:29.631 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.318312+0000", 2024-08-19T14:47:29.632 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.6874000000000001e-05 2024-08-19T14:47:29.632 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.632 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.632 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:47:29.632 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.422387+0000", 2024-08-19T14:47:29.632 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.104075665 2024-08-19T14:47:29.632 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.632 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.632 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:47:29.632 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.422600+0000", 2024-08-19T14:47:29.632 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00021286699999999999 2024-08-19T14:47:29.632 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:47:29.632 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:47:29.632 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:47:29.633 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.633 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.633 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.27235.0:110999 9.8 e874/852 9:1cb9f84e:::benchmark_data_smithi045_152862_object16494:head v 874'7026, mlcod=874'7023)", 2024-08-19T14:47:29.633 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:46:11.320845+0000", 2024-08-19T14:47:29.633 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 78.223234664000003, 2024-08-19T14:47:29.633 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.10167807600000001, 2024-08-19T14:47:29.633 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:47:29.633 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:47:29.633 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:47:29.633 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:47:29.633 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.633 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:47:29.633 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.320845+0000", 2024-08-19T14:47:29.633 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:47:29.634 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.634 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.634 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:47:29.634 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.320845+0000", 2024-08-19T14:47:29.634 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:47:29.634 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.634 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.634 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:47:29.634 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.320845+0000", 2024-08-19T14:47:29.634 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.5000000000000002e-07 2024-08-19T14:47:29.634 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.634 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.634 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:47:29.634 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.320851+0000", 2024-08-19T14:47:29.635 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.049e-06 2024-08-19T14:47:29.635 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.635 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.635 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:47:29.635 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.320863+0000", 2024-08-19T14:47:29.635 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1891e-05 2024-08-19T14:47:29.635 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.635 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.635 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:47:29.635 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.320869+0000", 2024-08-19T14:47:29.635 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.6330000000000001e-06 2024-08-19T14:47:29.635 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.635 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.635 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:47:29.635 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.320929+0000", 2024-08-19T14:47:29.636 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.0185000000000001e-05 2024-08-19T14:47:29.636 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.636 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.636 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:47:29.636 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.320949+0000", 2024-08-19T14:47:29.636 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0307999999999999e-05 2024-08-19T14:47:29.636 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.636 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.636 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:47:29.636 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.422409+0000", 2024-08-19T14:47:29.636 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.101459491 2024-08-19T14:47:29.636 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.636 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.636 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:47:29.637 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.422523+0000", 2024-08-19T14:47:29.637 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000113769 2024-08-19T14:47:29.637 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:47:29.637 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:47:29.637 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:47:29.637 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.637 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.637 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.27235.0:111001 9.2 e874/826 9:45311ef0:::benchmark_data_smithi045_152862_object16496:head v 874'7021, mlcod=874'7016)", 2024-08-19T14:47:29.637 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:46:11.323663+0000", 2024-08-19T14:47:29.637 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 78.220416220999994, 2024-08-19T14:47:29.637 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.098856208000000001, 2024-08-19T14:47:29.637 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:47:29.637 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:47:29.637 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:47:29.637 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:47:29.638 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.638 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:47:29.638 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.323663+0000", 2024-08-19T14:47:29.638 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:47:29.638 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.638 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.638 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:47:29.638 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.323663+0000", 2024-08-19T14:47:29.638 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:47:29.638 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.638 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.638 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:47:29.638 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.323664+0000", 2024-08-19T14:47:29.639 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.7800000000000001e-07 2024-08-19T14:47:29.639 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.639 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.639 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:47:29.639 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.323671+0000", 2024-08-19T14:47:29.639 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.5510000000000002e-06 2024-08-19T14:47:29.639 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.639 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.639 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:47:29.639 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.323687+0000", 2024-08-19T14:47:29.639 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5965999999999999e-05 2024-08-19T14:47:29.639 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.639 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.639 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:47:29.639 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.323694+0000", 2024-08-19T14:47:29.640 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.7610000000000002e-06 2024-08-19T14:47:29.640 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.640 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.640 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:47:29.640 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.323765+0000", 2024-08-19T14:47:29.640 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.0863000000000002e-05 2024-08-19T14:47:29.640 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.640 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.640 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:47:29.640 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.323783+0000", 2024-08-19T14:47:29.640 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8029e-05 2024-08-19T14:47:29.640 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.640 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.640 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:47:29.641 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.422464+0000", 2024-08-19T14:47:29.641 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.098680793000000003 2024-08-19T14:47:29.641 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.641 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.641 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:47:29.641 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.422519+0000", 2024-08-19T14:47:29.641 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.5467000000000003e-05 2024-08-19T14:47:29.641 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:47:29.641 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:47:29.641 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:47:29.641 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.641 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.641 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.27235.0:111004 9.6 e874/853 9:69fe6b0b:::benchmark_data_smithi045_152862_object16499:head v 874'6932, mlcod=874'6929)", 2024-08-19T14:47:29.641 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:46:11.334071+0000", 2024-08-19T14:47:29.642 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 78.210008125000002, 2024-08-19T14:47:29.642 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.088562975000000002, 2024-08-19T14:47:29.642 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:47:29.642 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:47:29.642 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:47:29.642 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:47:29.642 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.642 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:47:29.642 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.334071+0000", 2024-08-19T14:47:29.642 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:47:29.642 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.642 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.642 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:47:29.642 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.334071+0000", 2024-08-19T14:47:29.643 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:47:29.643 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.643 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.643 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:47:29.643 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.334073+0000", 2024-08-19T14:47:29.643 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.322e-06 2024-08-19T14:47:29.643 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.643 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.643 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:47:29.643 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.334083+0000", 2024-08-19T14:47:29.643 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0645000000000001e-05 2024-08-19T14:47:29.643 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.643 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.643 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:47:29.643 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.334099+0000", 2024-08-19T14:47:29.644 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5393999999999999e-05 2024-08-19T14:47:29.644 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.644 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.644 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:47:29.644 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.334105+0000", 2024-08-19T14:47:29.644 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.8290000000000002e-06 2024-08-19T14:47:29.644 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.644 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.644 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:47:29.644 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.334168+0000", 2024-08-19T14:47:29.644 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.2601999999999995e-05 2024-08-19T14:47:29.644 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.644 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.644 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:47:29.645 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.334186+0000", 2024-08-19T14:47:29.645 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.774e-05 2024-08-19T14:47:29.645 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.645 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.645 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:47:29.645 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.422569+0000", 2024-08-19T14:47:29.645 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.088383530000000002 2024-08-19T14:47:29.645 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:29.645 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:29.645 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:47:29.645 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:11.422634+0000", 2024-08-19T14:47:29.645 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.4912999999999993e-05 2024-08-19T14:47:29.645 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:47:29.645 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:47:29.645 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:47:29.646 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:47:29.646 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:47:29.646 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:47:29.697 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:47:29.722 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:29.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:29.724 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:47:29.825 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:47:29.848 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:29.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:29.849 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:47:29.950 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:47:29.973 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:29.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:29.974 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:47:30.007 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:29 smithi031 ceph-mon[16336]: pgmap v2135: 47 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfilling, 41 active+clean; 1.4 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 1.7 KiB/s rd, 47 KiB/s wr, 423 op/s; 67/70614 objects degraded (0.095%); 5121/70614 objects misplaced (7.252%); 42 MiB/s, 665 objects/s recovering 2024-08-19T14:47:30.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:29 smithi032 ceph-mon[20004]: pgmap v2135: 47 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfilling, 41 active+clean; 1.4 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 1.7 KiB/s rd, 47 KiB/s wr, 423 op/s; 67/70614 objects degraded (0.095%); 5121/70614 objects misplaced (7.252%); 42 MiB/s, 665 objects/s recovering 2024-08-19T14:47:30.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:29 smithi045 ceph-mon[20143]: pgmap v2135: 47 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfilling, 41 active+clean; 1.4 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 1.7 KiB/s rd, 47 KiB/s wr, 423 op/s; 67/70614 objects degraded (0.095%); 5121/70614 objects misplaced (7.252%); 42 MiB/s, 665 objects/s recovering 2024-08-19T14:47:30.075 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:47:30.097 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:30.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:30.098 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:47:30.099 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:47:30.142 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:47:30.199 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:47:30.221 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:30.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:30.222 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:47:30.323 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:47:30.345 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:30.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:30.345 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:47:30.400 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-08-19T14:47:30.447 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:47:30.468 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:30.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:30.469 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:47:30.570 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:47:30.592 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:30.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:30.593 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:47:30.666 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:47:30.694 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:47:30.716 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:30.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:30.717 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:47:30.818 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:47:30.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:30.843 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:30.843 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:47:30.944 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:47:30.967 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:30 smithi045 ceph-mon[20143]: osdmap e909: 12 total, 12 up, 9 in 2024-08-19T14:47:30.969 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:30.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:30.970 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:47:31.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:30 smithi032 ceph-mon[20004]: osdmap e909: 12 total, 12 up, 9 in 2024-08-19T14:47:31.071 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:47:31.094 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:31.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:31.095 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:47:31.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:30 smithi031 ceph-mon[16336]: osdmap e909: 12 total, 12 up, 9 in 2024-08-19T14:47:31.196 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:47:31.218 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:47:31.218 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [], 2024-08-19T14:47:31.218 INFO:teuthology.orchestra.run.smithi045.stdout: "num_ops": 0 2024-08-19T14:47:31.218 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:47:31.221 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:31.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:31.222 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:47:31.323 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:47:31.344 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:31.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:31.346 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:47:31.447 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:47:31.469 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:31.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:31.470 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:47:31.571 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:47:31.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:31.593 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:31.594 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:47:31.695 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:47:31.716 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:31.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:31.717 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:47:31.737 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:31 smithi031 ceph-mon[16336]: pgmap v2137: 47 pgs: 2 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfilling, 43 active+clean; 1.3 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 525 op/s; 72/64923 objects degraded (0.111%); 3207/64923 objects misplaced (4.940%); 43 MiB/s, 685 objects/s recovering 2024-08-19T14:47:31.737 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:31 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 72/64923 objects degraded (0.111%), 2 pgs degraded (PG_DEGRADED) 2024-08-19T14:47:31.818 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:47:31.854 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:31.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:31.855 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:47:31.880 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:31 smithi045 ceph-mon[20143]: pgmap v2137: 47 pgs: 2 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfilling, 43 active+clean; 1.3 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 525 op/s; 72/64923 objects degraded (0.111%); 3207/64923 objects misplaced (4.940%); 43 MiB/s, 685 objects/s recovering 2024-08-19T14:47:31.880 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:31 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 72/64923 objects degraded (0.111%), 2 pgs degraded (PG_DEGRADED) 2024-08-19T14:47:31.880 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:31 smithi045 ceph-mon[20143]: osdmap e910: 12 total, 12 up, 9 in 2024-08-19T14:47:31.930 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-08-19T14:47:31.956 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:47:31.980 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:31.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:31.982 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:47:32.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:31 smithi032 ceph-mon[20004]: pgmap v2137: 47 pgs: 2 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfilling, 43 active+clean; 1.3 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 525 op/s; 72/64923 objects degraded (0.111%); 3207/64923 objects misplaced (4.940%); 43 MiB/s, 685 objects/s recovering 2024-08-19T14:47:32.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:31 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 72/64923 objects degraded (0.111%), 2 pgs degraded (PG_DEGRADED) 2024-08-19T14:47:32.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:31 smithi032 ceph-mon[20004]: osdmap e910: 12 total, 12 up, 9 in 2024-08-19T14:47:32.038 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:47:32.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:31 smithi031 ceph-mon[16336]: osdmap e910: 12 total, 12 up, 9 in 2024-08-19T14:47:32.082 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:47:32.108 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:32.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:32.109 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:47:32.162 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:47:32.162 INFO:teuthology.orchestra.run.smithi031.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-08-19T14:47:32.162 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:47:32.210 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:47:32.266 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:32.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:32.267 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:47:32.368 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:47:32.392 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:32.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:32.393 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:47:32.400 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:47:32.494 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:47:32.519 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:32.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:32.520 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:47:32.621 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:47:32.658 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:32.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:32.659 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:47:32.760 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:47:32.783 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:32.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:32.784 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:47:32.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:32 smithi045 ceph-mon[20143]: 1.1e scrub starts 2024-08-19T14:47:32.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:32 smithi045 ceph-mon[20143]: 1.1e scrub ok 2024-08-19T14:47:32.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:32 smithi045 ceph-mon[20143]: osdmap e911: 12 total, 12 up, 9 in 2024-08-19T14:47:32.884 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:47:32.911 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:32.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:32.912 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:47:33.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:32 smithi032 ceph-mon[20004]: 1.1e scrub starts 2024-08-19T14:47:33.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:32 smithi032 ceph-mon[20004]: 1.1e scrub ok 2024-08-19T14:47:33.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:32 smithi032 ceph-mon[20004]: osdmap e911: 12 total, 12 up, 9 in 2024-08-19T14:47:33.013 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:47:33.061 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:33.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:33.062 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:47:33.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:32 smithi031 ceph-mon[16336]: 1.1e scrub starts 2024-08-19T14:47:33.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:32 smithi031 ceph-mon[16336]: 1.1e scrub ok 2024-08-19T14:47:33.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:32 smithi031 ceph-mon[16336]: osdmap e911: 12 total, 12 up, 9 in 2024-08-19T14:47:33.163 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:47:33.187 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:33.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:33.188 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:47:33.205 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:47:33.205 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [], 2024-08-19T14:47:33.205 INFO:teuthology.orchestra.run.smithi045.stdout: "complaint_time": 30, 2024-08-19T14:47:33.205 INFO:teuthology.orchestra.run.smithi045.stdout: "num_blocked_ops": 0 2024-08-19T14:47:33.205 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:47:33.289 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:47:33.334 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:33.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:33.336 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:47:33.403 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:47:33.437 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:47:33.462 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:33.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:33.464 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:47:33.565 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:47:33.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:33.591 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:33.591 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:47:33.692 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:47:33.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:33.724 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:33.725 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:47:33.736 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is set 2024-08-19T14:47:33.771 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:33 smithi031 ceph-mon[16336]: pgmap v2140: 47 pgs: 2 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfilling, 43 active+clean; 1.3 GiB data, 12 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 556 op/s; 80/63729 objects degraded (0.126%); 2937/63729 objects misplaced (4.609%); 57 MiB/s, 906 objects/s recovering 2024-08-19T14:47:33.771 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:33 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/649145101' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:47:33.826 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:47:33.858 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:33 smithi045 ceph-mon[20143]: pgmap v2140: 47 pgs: 2 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfilling, 43 active+clean; 1.3 GiB data, 12 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 556 op/s; 80/63729 objects degraded (0.126%); 2937/63729 objects misplaced (4.609%); 57 MiB/s, 906 objects/s recovering 2024-08-19T14:47:33.858 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:33 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/649145101' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:47:33.860 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:33.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:33.861 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:47:33.909 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-08-19T14:47:33.962 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:47:33.993 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:47:33.993 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:47:34.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:33 smithi032 ceph-mon[20004]: pgmap v2140: 47 pgs: 2 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfilling, 43 active+clean; 1.3 GiB data, 12 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 556 op/s; 80/63729 objects degraded (0.126%); 2937/63729 objects misplaced (4.609%); 57 MiB/s, 906 objects/s recovering 2024-08-19T14:47:34.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:33 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/649145101' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:47:34.008 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:34.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:34.009 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:47:34.018 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:47:34.018 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:47:34.038 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:47:34.038 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:47:34.058 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:47:34.058 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:47:34.076 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:47:34.076 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:47:34.099 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:47:34.100 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:47:34.110 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:47:34.126 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:47:34.127 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:47:34.146 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:47:34.147 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:47:34.148 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:34.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:34.149 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:47:34.177 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:47:34.177 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:47:34.203 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:47:34.203 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:47:34.238 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:47:34.239 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:47:34.250 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:47:34.264 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:47:34.264 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:47:34.275 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:34.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:34.276 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:47:34.364 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:47:34.376 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:47:34.425 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:34.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:34.427 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:47:34.527 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:47:34.553 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:34.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:34.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:47:34.654 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:47:34.690 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:34.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:34.692 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:47:34.793 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:47:34.817 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:34.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:34.818 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:47:34.919 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:47:34.946 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:34.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:34.948 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:47:34.978 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:34 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:47:34.979 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:34 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:47:34.979 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:34 smithi045 ceph-mon[20143]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:47:34.979 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:34 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/649145101' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:47:34.979 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:34 smithi045 ceph-mon[20143]: osdmap e912: 12 total, 12 up, 9 in 2024-08-19T14:47:34.979 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:34 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3352830120' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:47:35.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:34 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:47:35.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:34 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:47:35.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:34 smithi032 ceph-mon[20004]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:47:35.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:34 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/649145101' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:47:35.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:34 smithi032 ceph-mon[20004]: osdmap e912: 12 total, 12 up, 9 in 2024-08-19T14:47:35.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:34 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3352830120' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:47:35.042 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:34 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:47:35.042 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:34 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:47:35.043 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:34 smithi031 ceph-mon[16336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:47:35.043 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:34 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/649145101' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:47:35.043 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:34 smithi031 ceph-mon[16336]: osdmap e912: 12 total, 12 up, 9 in 2024-08-19T14:47:35.043 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:34 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3352830120' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:47:35.049 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:47:35.073 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:35.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:35.075 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:47:35.144 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:47:35.144 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:47:35.176 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:47:35.200 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:35.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:35.202 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:47:35.303 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:47:35.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:35.341 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:35.341 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:47:35.442 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:47:35.467 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:35.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:35.469 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:47:35.569 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:47:35.592 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:35.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:35.593 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:47:35.693 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:47:35.720 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:35.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:35.721 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:47:35.822 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:47:35.844 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:35.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:35.846 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:47:35.865 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-08-19T14:47:35.946 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:47:35.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:35.972 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:35.972 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:47:36.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:35 smithi032 ceph-mon[20004]: pgmap v2142: 47 pgs: 2 active+recovering+undersized+degraded+remapped, 45 active+clean; 1.2 GiB data, 12 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 486 op/s; 80/59535 objects degraded (0.134%); 2061/59535 objects misplaced (3.462%); 15 MiB/s, 239 objects/s recovering 2024-08-19T14:47:36.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:35 smithi045 ceph-mon[20143]: pgmap v2142: 47 pgs: 2 active+recovering+undersized+degraded+remapped, 45 active+clean; 1.2 GiB data, 12 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 486 op/s; 80/59535 objects degraded (0.134%); 2061/59535 objects misplaced (3.462%); 15 MiB/s, 239 objects/s recovering 2024-08-19T14:47:36.073 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:47:36.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:35 smithi031 ceph-mon[16336]: pgmap v2142: 47 pgs: 2 active+recovering+undersized+degraded+remapped, 45 active+clean; 1.2 GiB data, 12 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 486 op/s; 80/59535 objects degraded (0.134%); 2061/59535 objects misplaced (3.462%); 15 MiB/s, 239 objects/s recovering 2024-08-19T14:47:36.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:36.123 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:36.123 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:47:36.224 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:47:36.255 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:36.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:36.256 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:47:36.300 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:47:36.356 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:47:36.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:36.385 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:36.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:47:36.486 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:47:36.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:36.512 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:36.512 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:47:36.614 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:47:36.640 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:36.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:36.641 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:47:36.743 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:47:36.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:36.770 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:36.770 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:47:36.871 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:47:36.896 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:36.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:36.897 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:47:36.927 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-19T14:47:36.998 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:47:37.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:37.024 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:37.024 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:47:37.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:36 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 80/55728 objects degraded (0.144%), 2 pgs degraded (PG_DEGRADED) 2024-08-19T14:47:37.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:36 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 80/55728 objects degraded (0.144%), 2 pgs degraded (PG_DEGRADED) 2024-08-19T14:47:37.126 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:47:37.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:37.156 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:37.156 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:47:37.232 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:47:37.232 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:47:37.249 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:47:37.257 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:47:37.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:36 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 80/55728 objects degraded (0.144%), 2 pgs degraded (PG_DEGRADED) 2024-08-19T14:47:37.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:37.294 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:37.294 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:47:37.396 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:47:37.424 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:37.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:37.425 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:47:37.499 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:47:37.526 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:47:37.625 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:37.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:37.626 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:47:37.727 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:47:37.755 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:37.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:37.756 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:47:37.853 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:47:37.856 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:47:37.879 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:37.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:37.880 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:47:37.936 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:37 smithi045 ceph-mon[20143]: pgmap v2143: 47 pgs: 2 active+recovering+undersized+degraded+remapped, 45 active+clean; 1.1 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 510 op/s; 80/55728 objects degraded (0.144%); 1857/55728 objects misplaced (3.332%); 14 MiB/s, 221 objects/s recovering 2024-08-19T14:47:37.982 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:47:37.983 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:37 smithi031 ceph-mon[16336]: pgmap v2143: 47 pgs: 2 active+recovering+undersized+degraded+remapped, 45 active+clean; 1.1 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 510 op/s; 80/55728 objects degraded (0.144%); 1857/55728 objects misplaced (3.332%); 14 MiB/s, 221 objects/s recovering 2024-08-19T14:47:37.986 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-08-19T14:47:38.009 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:38.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:38.010 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:47:38.111 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:47:38.135 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:38.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:38.136 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:47:38.238 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:47:38.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:37 smithi032 ceph-mon[20004]: pgmap v2143: 47 pgs: 2 active+recovering+undersized+degraded+remapped, 45 active+clean; 1.1 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 510 op/s; 80/55728 objects degraded (0.144%); 1857/55728 objects misplaced (3.332%); 14 MiB/s, 221 objects/s recovering 2024-08-19T14:47:38.262 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:38.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:38.263 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:47:38.302 INFO:tasks.thrashosds.thrasher:in_osds: [1, 4, 8, 2, 11, 9, 10, 6, 7] out_osds: [3, 0, 5] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 8, 9, 11, 10, 2] 2024-08-19T14:47:38.302 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-19T14:47:38.303 INFO:tasks.thrashosds.thrasher:Adding osd 3 2024-08-19T14:47:38.303 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd in 3 2024-08-19T14:47:38.364 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:47:38.425 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:38.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:38.427 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:47:38.444 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:47:38.528 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:47:38.557 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:38.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:38.558 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:47:38.659 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:47:38.703 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:38.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:38.704 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:47:38.800 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is set 2024-08-19T14:47:38.804 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:47:38.833 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:38.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:38.834 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:47:38.898 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:38 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1614171102' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:47:38.935 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:47:38.962 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:38.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:38.963 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:47:39.044 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:38 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1614171102' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:47:39.063 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:47:39.064 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:47:39.065 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:47:39.088 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:39.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:39.089 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:47:39.190 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:47:39.216 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:39.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:39.217 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:47:39.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:38 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1614171102' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:47:39.319 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:47:39.347 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:39.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:39.348 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:47:39.449 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:47:39.479 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:47:39.480 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:47:39.480 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:47:39.480 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:47:39.480 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:47:39.480 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:47:39.480 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:47:39.480 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:47:39.481 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:47:39.481 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:47:39.481 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:47:39.481 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:47:39.481 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:47:39.481 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:47:39.481 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:47:39.481 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:47:39.481 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:47:39.482 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:47:39.482 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:47:39.482 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:47:39.482 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:47:39.482 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:47:39.482 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:47:39.482 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:47:39.489 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:39.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:39.490 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:47:39.590 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:47:39.615 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:39.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:39.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:47:39.718 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:47:39.727 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-08-19T14:47:39.740 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:39.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:39.741 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:47:39.842 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:47:39.866 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:39.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:39.871 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:47:39.972 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:47:40.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:40.031 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:40.031 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:47:40.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:39 smithi045 ceph-mon[20143]: pgmap v2144: 47 pgs: 2 active+recovering+undersized+degraded+remapped, 45 active+clean; 1.1 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 460 op/s; 80/54318 objects degraded (0.147%); 1857/54318 objects misplaced (3.419%); 11 MiB/s, 172 objects/s recovering 2024-08-19T14:47:40.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:39 smithi045 ceph-mon[20143]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:47:40.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:39 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1614171102' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:47:40.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:39 smithi045 ceph-mon[20143]: osdmap e913: 12 total, 12 up, 9 in 2024-08-19T14:47:40.047 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:39 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2387681935' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:47:40.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:39 smithi031 ceph-mon[16336]: pgmap v2144: 47 pgs: 2 active+recovering+undersized+degraded+remapped, 45 active+clean; 1.1 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 460 op/s; 80/54318 objects degraded (0.147%); 1857/54318 objects misplaced (3.419%); 11 MiB/s, 172 objects/s recovering 2024-08-19T14:47:40.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:39 smithi031 ceph-mon[16336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:47:40.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:39 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1614171102' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:47:40.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:39 smithi031 ceph-mon[16336]: osdmap e913: 12 total, 12 up, 9 in 2024-08-19T14:47:40.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:39 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2387681935' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:47:40.132 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:47:40.156 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:40.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:40.157 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:47:40.172 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:47:40.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:39 smithi032 ceph-mon[20004]: pgmap v2144: 47 pgs: 2 active+recovering+undersized+degraded+remapped, 45 active+clean; 1.1 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 460 op/s; 80/54318 objects degraded (0.147%); 1857/54318 objects misplaced (3.419%); 11 MiB/s, 172 objects/s recovering 2024-08-19T14:47:40.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:39 smithi032 ceph-mon[20004]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:47:40.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:39 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1614171102' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-19T14:47:40.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:39 smithi032 ceph-mon[20004]: osdmap e913: 12 total, 12 up, 9 in 2024-08-19T14:47:40.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:39 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2387681935' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:47:40.258 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:47:40.282 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:40.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:40.283 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:47:40.308 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:47:40.384 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:47:40.410 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:40.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:40.411 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:47:40.511 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:47:40.535 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:40.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:40.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:47:40.637 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:47:40.663 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:40.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:40.664 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:47:40.765 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:47:40.792 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:40.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:40.793 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:47:40.895 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:47:40.921 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:40.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:40.922 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:47:41.023 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:47:41.048 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:41.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:41.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:47:41.150 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:47:41.175 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:41.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:41.177 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:47:41.278 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:47:41.301 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:41.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:41.302 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:47:41.403 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:47:41.428 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:41.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:41.429 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:47:41.530 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:47:41.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:41.557 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:41.557 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:47:41.658 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:47:41.684 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:41.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:41.685 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:47:41.786 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:47:41.800 INFO:teuthology.orchestra.run.smithi031.stderr:marked in osd.3. 2024-08-19T14:47:41.813 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:41.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:41.814 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:47:41.914 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:47:41.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:41.936 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:41.937 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:47:42.038 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:47:42.059 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:42.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:42.060 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:47:42.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:41 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mon-a[16332]: 2024-08-19T14:47:41.666+0000 7f2ff503e640 -1 mon.a@0(leader).osd e913 definitely_dead 0 2024-08-19T14:47:42.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:41 smithi031 ceph-mon[16336]: pgmap v2146: 47 pgs: 2 active+recovering+undersized+degraded+remapped, 45 active+clean; 967 MiB data, 11 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 722 op/s; 80/46380 objects degraded (0.172%); 1571/46380 objects misplaced (3.387%); 11 MiB/s, 168 objects/s recovering 2024-08-19T14:47:42.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:41 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3915574850' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2024-08-19T14:47:42.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:41 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2024-08-19T14:47:42.161 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:47:42.183 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:42.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:42.184 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:47:42.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:41 smithi032 ceph-mon[20004]: pgmap v2146: 47 pgs: 2 active+recovering+undersized+degraded+remapped, 45 active+clean; 967 MiB data, 11 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 722 op/s; 80/46380 objects degraded (0.172%); 1571/46380 objects misplaced (3.387%); 11 MiB/s, 168 objects/s recovering 2024-08-19T14:47:42.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:41 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3915574850' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2024-08-19T14:47:42.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:41 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2024-08-19T14:47:42.278 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-19T14:47:42.285 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:47:42.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:41 smithi045 ceph-mon[20143]: pgmap v2146: 47 pgs: 2 active+recovering+undersized+degraded+remapped, 45 active+clean; 967 MiB data, 11 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 722 op/s; 80/46380 objects degraded (0.172%); 1571/46380 objects misplaced (3.387%); 11 MiB/s, 168 objects/s recovering 2024-08-19T14:47:42.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:41 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3915574850' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2024-08-19T14:47:42.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:41 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2024-08-19T14:47:42.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:42.309 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:42.309 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:47:42.410 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:47:42.431 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:42.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:42.432 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:47:42.533 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:47:42.555 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:42.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:42.557 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:47:42.624 INFO:tasks.thrashosds.thrasher:Added osd 3 2024-08-19T14:47:42.658 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:47:42.661 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:47:42.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:42.681 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:42.681 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:47:42.782 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:47:42.809 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:42.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:42.810 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:47:42.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:42 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["3"]}]': finished 2024-08-19T14:47:42.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:42 smithi031 ceph-mon[16336]: osdmap e914: 12 total, 12 up, 10 in 2024-08-19T14:47:42.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:42 smithi031 ceph-mon[16336]: osdmap e915: 12 total, 12 up, 10 in 2024-08-19T14:47:42.869 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:47:42.910 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:47:42.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:42.934 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:42.934 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:47:43.035 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:47:43.058 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:43.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:43.060 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:47:43.161 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:47:43.183 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:43.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:43.184 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:47:43.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:42 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["3"]}]': finished 2024-08-19T14:47:43.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:42 smithi032 ceph-mon[20004]: osdmap e914: 12 total, 12 up, 10 in 2024-08-19T14:47:43.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:42 smithi032 ceph-mon[20004]: osdmap e915: 12 total, 12 up, 10 in 2024-08-19T14:47:43.285 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:47:43.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:42 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["3"]}]': finished 2024-08-19T14:47:43.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:42 smithi045 ceph-mon[20143]: osdmap e914: 12 total, 12 up, 10 in 2024-08-19T14:47:43.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:42 smithi045 ceph-mon[20143]: osdmap e915: 12 total, 12 up, 10 in 2024-08-19T14:47:43.311 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:43.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:43.312 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:47:43.343 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:47:43.413 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:47:43.438 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:43.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:43.440 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:47:43.541 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:47:43.563 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:43.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:43.564 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:47:43.665 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:47:43.687 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:43.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:43.688 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:47:43.789 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:47:43.798 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is unset 2024-08-19T14:47:43.818 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:43.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:43.819 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:47:43.872 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:47:43.872 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:47:43.891 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:47:43.891 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:47:43.910 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:47:43.910 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:47:43.920 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:47:43.929 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:47:43.929 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:47:43.943 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:43.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:43.944 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:47:44.045 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:47:44.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:44.067 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:44.067 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:47:44.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:43 smithi031 ceph-mon[16336]: pgmap v2148: 47 pgs: 2 active+recovering+undersized+degraded+remapped, 45 active+clean; 921 MiB data, 11 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 639 op/s; 80/44187 objects degraded (0.181%); 1415/44187 objects misplaced (3.202%); 5.0 MiB/s, 80 objects/s recovering 2024-08-19T14:47:44.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:43 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 80/44187 objects degraded (0.181%), 2 pgs degraded (PG_DEGRADED) 2024-08-19T14:47:44.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:43 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2815505378' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:47:44.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:43 smithi031 ceph-mon[16336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:47:44.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:43 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2815505378' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:47:44.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:43 smithi031 ceph-mon[16336]: osdmap e916: 12 total, 12 up, 10 in 2024-08-19T14:47:44.148 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:47:44.148 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:47:44.166 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:47:44.167 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:47:44.168 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:47:44.185 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:47:44.185 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:47:44.190 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:44.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:44.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:47:44.205 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:47:44.205 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:47:44.222 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:47:44.222 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:47:44.241 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:47:44.241 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:47:44.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:43 smithi032 ceph-mon[20004]: pgmap v2148: 47 pgs: 2 active+recovering+undersized+degraded+remapped, 45 active+clean; 921 MiB data, 11 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 639 op/s; 80/44187 objects degraded (0.181%); 1415/44187 objects misplaced (3.202%); 5.0 MiB/s, 80 objects/s recovering 2024-08-19T14:47:44.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:43 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 80/44187 objects degraded (0.181%), 2 pgs degraded (PG_DEGRADED) 2024-08-19T14:47:44.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:43 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2815505378' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:47:44.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:43 smithi032 ceph-mon[20004]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:47:44.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:43 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2815505378' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:47:44.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:43 smithi032 ceph-mon[20004]: osdmap e916: 12 total, 12 up, 10 in 2024-08-19T14:47:44.260 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:47:44.260 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:47:44.277 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-08-19T14:47:44.277 INFO:teuthology.orchestra.run.smithi045.stdout: "size": 20, 2024-08-19T14:47:44.277 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 600, 2024-08-19T14:47:44.277 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [ 2024-08-19T14:47:44.278 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.278 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.27235.0:31081 9.6 9.8e690cd6 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e834)", 2024-08-19T14:47:44.278 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:44:34.157100+0000", 2024-08-19T14:47:44.278 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 189.394052598, 2024-08-19T14:47:44.278 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.99146695600000001, 2024-08-19T14:47:44.278 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:47:44.278 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:47:44.278 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-08-19T14:47:44.278 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:47:44.278 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.27235", 2024-08-19T14:47:44.279 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2746512116", 2024-08-19T14:47:44.279 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 31081 2024-08-19T14:47:44.279 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.279 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:47:44.279 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.279 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:47:44.279 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:34.157100+0000", 2024-08-19T14:47:44.279 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:47:44.279 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.279 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.279 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:47:44.279 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:34.157100+0000", 2024-08-19T14:47:44.280 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:47:44.280 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.280 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.280 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:47:44.280 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:34.157100+0000", 2024-08-19T14:47:44.280 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.7399999999999998e-07 2024-08-19T14:47:44.280 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.280 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.280 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:47:44.280 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:34.157120+0000", 2024-08-19T14:47:44.280 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0188999999999999e-05 2024-08-19T14:47:44.281 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.281 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.281 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:47:44.281 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:34.157127+0000", 2024-08-19T14:47:44.281 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.1679999999999998e-06 2024-08-19T14:47:44.281 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.281 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.281 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:47:44.281 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:34.157133+0000", 2024-08-19T14:47:44.281 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.6620000000000002e-06 2024-08-19T14:47:44.281 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.282 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.282 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:47:44.282 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:34.157262+0000", 2024-08-19T14:47:44.282 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012860700000000001 2024-08-19T14:47:44.282 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.282 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.282 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for peered", 2024-08-19T14:47:44.282 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:34.157274+0000", 2024-08-19T14:47:44.282 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2536e-05 2024-08-19T14:47:44.282 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.282 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.283 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:47:44.283 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:35.148528+0000", 2024-08-19T14:47:44.283 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.99125351699999997 2024-08-19T14:47:44.283 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.283 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.283 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:47:44.283 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:35.148566+0000", 2024-08-19T14:47:44.283 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.8803000000000001e-05 2024-08-19T14:47:44.283 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:47:44.283 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:47:44.283 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:47:44.283 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.284 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.284 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.27235.0:31085 9.6 9.d2431d16 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e834)", 2024-08-19T14:47:44.284 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:44:34.160262+0000", 2024-08-19T14:47:44.284 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 189.390890469, 2024-08-19T14:47:44.284 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.98964577799999998, 2024-08-19T14:47:44.284 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:47:44.284 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:47:44.284 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-08-19T14:47:44.284 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:47:44.284 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.27235", 2024-08-19T14:47:44.285 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2746512116", 2024-08-19T14:47:44.285 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 31085 2024-08-19T14:47:44.285 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.285 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:47:44.285 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.285 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:47:44.285 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:34.160262+0000", 2024-08-19T14:47:44.285 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:47:44.285 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.285 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.285 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:47:44.285 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:34.160262+0000", 2024-08-19T14:47:44.286 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:47:44.286 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.286 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.286 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:47:44.286 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:34.160263+0000", 2024-08-19T14:47:44.286 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.482e-06 2024-08-19T14:47:44.286 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.286 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.286 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:47:44.286 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:34.160283+0000", 2024-08-19T14:47:44.287 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0228999999999999e-05 2024-08-19T14:47:44.287 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.287 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.287 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:47:44.287 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:34.160293+0000", 2024-08-19T14:47:44.287 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.4010000000000003e-06 2024-08-19T14:47:44.287 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.287 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.287 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:47:44.287 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:34.160298+0000", 2024-08-19T14:47:44.287 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.8409999999999999e-06 2024-08-19T14:47:44.288 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.288 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.288 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:47:44.288 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:34.160341+0000", 2024-08-19T14:47:44.288 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.3633000000000001e-05 2024-08-19T14:47:44.288 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.288 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.288 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for peered", 2024-08-19T14:47:44.291 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:34.160349+0000", 2024-08-19T14:47:44.291 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.3239999999999997e-06 2024-08-19T14:47:44.291 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.291 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.291 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:47:44.291 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:35.149891+0000", 2024-08-19T14:47:44.291 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.98954210200000003 2024-08-19T14:47:44.291 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.291 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.292 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:47:44.292 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:44:35.149907+0000", 2024-08-19T14:47:44.292 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6765999999999998e-05 2024-08-19T14:47:44.292 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:47:44.292 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:47:44.292 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:47:44.292 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.292 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.292 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.27235.0:70483 9.9 9:9359de2b:::benchmark_data_smithi045_152862_object70482:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e864)", 2024-08-19T14:47:44.292 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:45:24.117771+0000", 2024-08-19T14:47:44.292 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 139.43338109699999, 2024-08-19T14:47:44.292 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.44251764799999999, 2024-08-19T14:47:44.293 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:47:44.293 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:47:44.293 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-08-19T14:47:44.293 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:47:44.293 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.27235", 2024-08-19T14:47:44.293 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2746512116", 2024-08-19T14:47:44.293 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 70483 2024-08-19T14:47:44.293 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.293 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:47:44.293 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.293 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:47:44.293 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:24.117771+0000", 2024-08-19T14:47:44.294 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:47:44.294 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.294 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.294 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:47:44.294 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:24.117771+0000", 2024-08-19T14:47:44.294 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:47:44.294 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.294 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.294 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:47:44.294 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:24.117773+0000", 2024-08-19T14:47:44.294 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0669999999999999e-06 2024-08-19T14:47:44.294 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.295 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.295 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:47:44.295 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:24.117811+0000", 2024-08-19T14:47:44.295 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.7528999999999999e-05 2024-08-19T14:47:44.295 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.295 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.295 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:47:44.295 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:24.117827+0000", 2024-08-19T14:47:44.295 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6035000000000001e-05 2024-08-19T14:47:44.295 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.295 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.295 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:47:44.296 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:24.117838+0000", 2024-08-19T14:47:44.296 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1102e-05 2024-08-19T14:47:44.296 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.296 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.296 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:47:44.296 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:24.121392+0000", 2024-08-19T14:47:44.296 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0035546010000000001 2024-08-19T14:47:44.296 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.296 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.296 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for peered", 2024-08-19T14:47:44.296 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:24.121405+0000", 2024-08-19T14:47:44.296 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3009e-05 2024-08-19T14:47:44.296 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.297 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.297 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:47:44.297 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:24.560154+0000", 2024-08-19T14:47:44.297 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.438748888 2024-08-19T14:47:44.297 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.297 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.297 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:47:44.297 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:24.560289+0000", 2024-08-19T14:47:44.297 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013441700000000001 2024-08-19T14:47:44.297 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:47:44.297 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:47:44.298 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:47:44.298 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.298 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.298 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.27235.0:70484 9.9 9:92ad6910:::benchmark_data_smithi045_152862_object70483:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e864)", 2024-08-19T14:47:44.298 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:45:24.117865+0000", 2024-08-19T14:47:44.298 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 139.43328732000001, 2024-08-19T14:47:44.298 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.44664219799999999, 2024-08-19T14:47:44.298 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:47:44.298 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:47:44.298 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-08-19T14:47:44.298 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:47:44.298 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.27235", 2024-08-19T14:47:44.299 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2746512116", 2024-08-19T14:47:44.299 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 70484 2024-08-19T14:47:44.299 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.299 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:47:44.299 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.299 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:47:44.299 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:24.117865+0000", 2024-08-19T14:47:44.299 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:47:44.299 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.299 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.299 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:47:44.300 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:24.117865+0000", 2024-08-19T14:47:44.300 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:47:44.300 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.300 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.300 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:47:44.300 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:24.117866+0000", 2024-08-19T14:47:44.300 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.2099999999999996e-07 2024-08-19T14:47:44.300 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.300 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.300 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:47:44.300 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:24.117914+0000", 2024-08-19T14:47:44.300 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.8887000000000001e-05 2024-08-19T14:47:44.301 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.301 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.301 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:47:44.301 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:24.117927+0000", 2024-08-19T14:47:44.301 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2819999999999999e-05 2024-08-19T14:47:44.301 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.301 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.301 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:47:44.301 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:24.117933+0000", 2024-08-19T14:47:44.301 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.1449999999999996e-06 2024-08-19T14:47:44.301 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.302 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.302 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:47:44.302 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:24.121437+0000", 2024-08-19T14:47:44.302 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.003503262 2024-08-19T14:47:44.302 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.302 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.302 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for peered", 2024-08-19T14:47:44.302 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:24.121443+0000", 2024-08-19T14:47:44.302 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.319e-06 2024-08-19T14:47:44.302 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.302 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.302 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:47:44.303 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:24.564415+0000", 2024-08-19T14:47:44.303 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.44297210100000001 2024-08-19T14:47:44.303 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.303 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.303 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:47:44.303 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:24.564507+0000", 2024-08-19T14:47:44.303 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.1942999999999997e-05 2024-08-19T14:47:44.303 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:47:44.303 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:47:44.303 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:47:44.303 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.303 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.304 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.27235.0:70498 9.9 9:9197760d:::benchmark_data_smithi045_152862_object70497:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e864)", 2024-08-19T14:47:44.304 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:45:24.117945+0000", 2024-08-19T14:47:44.304 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 139.43320736800001, 2024-08-19T14:47:44.304 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.45920555499999999, 2024-08-19T14:47:44.304 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:47:44.304 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:47:44.304 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:47:44.304 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:47:44.304 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.27235", 2024-08-19T14:47:44.304 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2746512116", 2024-08-19T14:47:44.304 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 70498 2024-08-19T14:47:44.304 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.305 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:47:44.305 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.305 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:47:44.305 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:24.117945+0000", 2024-08-19T14:47:44.305 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:47:44.305 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.305 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.305 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:47:44.305 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:24.117945+0000", 2024-08-19T14:47:44.305 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:47:44.305 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.305 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.306 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:47:44.306 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:24.117945+0000", 2024-08-19T14:47:44.306 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.8699999999999995e-07 2024-08-19T14:47:44.306 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.306 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.306 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:47:44.306 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:24.117972+0000", 2024-08-19T14:47:44.306 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.6673999999999999e-05 2024-08-19T14:47:44.306 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.306 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.306 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:47:44.307 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:24.117981+0000", 2024-08-19T14:47:44.307 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.465e-06 2024-08-19T14:47:44.307 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.307 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.307 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:47:44.307 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:24.117987+0000", 2024-08-19T14:47:44.307 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4990000000000002e-06 2024-08-19T14:47:44.307 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.307 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.307 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:47:44.307 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:24.121482+0000", 2024-08-19T14:47:44.308 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0034945879999999999 2024-08-19T14:47:44.308 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.308 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.308 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for peered", 2024-08-19T14:47:44.308 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:24.121487+0000", 2024-08-19T14:47:44.308 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.215e-06 2024-08-19T14:47:44.308 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.308 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.308 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:47:44.308 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:24.564547+0000", 2024-08-19T14:47:44.308 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.44306020699999998 2024-08-19T14:47:44.308 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.308 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.309 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:47:44.309 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:24.564700+0000", 2024-08-19T14:47:44.309 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015304500000000001 2024-08-19T14:47:44.309 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.309 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.309 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 6,11", 2024-08-19T14:47:44.309 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:24.564889+0000", 2024-08-19T14:47:44.309 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00018919999999999999 2024-08-19T14:47:44.309 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.309 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.309 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:47:44.309 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:24.573153+0000", 2024-08-19T14:47:44.310 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0082636749999999998 2024-08-19T14:47:44.310 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.310 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.310 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:47:44.310 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:24.573195+0000", 2024-08-19T14:47:44.310 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.1754e-05 2024-08-19T14:47:44.310 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.310 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.310 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:47:44.310 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:24.573381+0000", 2024-08-19T14:47:44.310 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00018598599999999999 2024-08-19T14:47:44.311 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.311 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.311 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:47:44.311 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:24.573445+0000", 2024-08-19T14:47:44.311 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.4599999999999998e-05 2024-08-19T14:47:44.311 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.311 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.311 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:47:44.311 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:24.577150+0000", 2024-08-19T14:47:44.311 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00370506 2024-08-19T14:47:44.311 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:47:44.311 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:47:44.312 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:47:44.312 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.312 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.312 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.27235.0:70499 9.9 9:9f15c880:::benchmark_data_smithi045_152862_object70498:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e864)", 2024-08-19T14:47:44.312 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:45:24.117999+0000", 2024-08-19T14:47:44.312 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 139.43315337199999, 2024-08-19T14:47:44.312 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.46267045899999998, 2024-08-19T14:47:44.312 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:47:44.312 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:47:44.312 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:47:44.312 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:47:44.312 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.27235", 2024-08-19T14:47:44.313 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2746512116", 2024-08-19T14:47:44.313 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:43 smithi045 ceph-mon[20143]: pgmap v2148: 47 pgs: 2 active+recovering+undersized+degraded+remapped, 45 active+clean; 921 MiB data, 11 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 639 op/s; 80/44187 objects degraded (0.181%); 1415/44187 objects misplaced (3.202%); 5.0 MiB/s, 80 objects/s recovering 2024-08-19T14:47:44.313 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:43 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 80/44187 objects degraded (0.181%), 2 pgs degraded (PG_DEGRADED) 2024-08-19T14:47:44.313 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:43 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2815505378' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:47:44.313 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:43 smithi045 ceph-mon[20143]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:47:44.313 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:43 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2815505378' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:47:44.314 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:43 smithi045 ceph-mon[20143]: osdmap e916: 12 total, 12 up, 10 in 2024-08-19T14:47:44.314 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 70499 2024-08-19T14:47:44.315 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.315 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:47:44.315 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.315 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:47:44.315 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:24.117999+0000", 2024-08-19T14:47:44.315 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:47:44.315 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.315 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.315 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:47:44.315 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:24.117999+0000", 2024-08-19T14:47:44.315 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:47:44.316 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.316 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.316 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:47:44.316 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:24.117999+0000", 2024-08-19T14:47:44.316 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4899999999999995e-07 2024-08-19T14:47:44.316 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.316 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.316 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:47:44.316 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:24.118025+0000", 2024-08-19T14:47:44.316 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.5564000000000001e-05 2024-08-19T14:47:44.316 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.316 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.317 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:47:44.317 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:24.118034+0000", 2024-08-19T14:47:44.317 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.0399999999999998e-06 2024-08-19T14:47:44.317 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.317 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.317 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:47:44.317 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:24.118040+0000", 2024-08-19T14:47:44.317 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.6790000000000002e-06 2024-08-19T14:47:44.317 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.317 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.317 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:47:44.318 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:24.121552+0000", 2024-08-19T14:47:44.318 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0035124990000000001 2024-08-19T14:47:44.318 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.318 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.318 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for peered", 2024-08-19T14:47:44.318 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:24.121558+0000", 2024-08-19T14:47:44.318 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7309999999999998e-06 2024-08-19T14:47:44.318 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.318 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.318 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:47:44.318 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:24.567981+0000", 2024-08-19T14:47:44.319 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.44642287000000003 2024-08-19T14:47:44.319 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.319 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.319 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:47:44.319 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:24.568120+0000", 2024-08-19T14:47:44.319 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000139381 2024-08-19T14:47:44.319 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.319 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.319 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 6,11", 2024-08-19T14:47:44.319 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:24.568283+0000", 2024-08-19T14:47:44.319 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00016279599999999999 2024-08-19T14:47:44.319 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.320 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.320 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:47:44.320 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:24.573305+0000", 2024-08-19T14:47:44.320 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0050219380000000001 2024-08-19T14:47:44.320 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.320 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.320 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:47:44.320 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:24.573358+0000", 2024-08-19T14:47:44.320 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.3124000000000002e-05 2024-08-19T14:47:44.320 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.320 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.320 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:47:44.321 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:24.577166+0000", 2024-08-19T14:47:44.321 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0038084350000000002 2024-08-19T14:47:44.321 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.321 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.321 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:47:44.321 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:24.577220+0000", 2024-08-19T14:47:44.321 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.3337000000000001e-05 2024-08-19T14:47:44.321 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.321 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.321 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:47:44.321 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:24.580669+0000", 2024-08-19T14:47:44.321 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0034495160000000001 2024-08-19T14:47:44.322 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:47:44.322 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:47:44.322 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:47:44.322 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.322 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.322 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.27235.0:70515 9.9 9:93b54f6d:::benchmark_data_smithi045_152862_object70514:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e864)", 2024-08-19T14:47:44.322 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:45:24.118049+0000", 2024-08-19T14:47:44.322 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 139.43310309200001, 2024-08-19T14:47:44.322 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.46964550300000002, 2024-08-19T14:47:44.322 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:47:44.322 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:47:44.323 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:47:44.323 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:47:44.323 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.27235", 2024-08-19T14:47:44.323 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2746512116", 2024-08-19T14:47:44.323 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 70515 2024-08-19T14:47:44.323 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.323 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:47:44.323 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.323 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:47:44.323 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:24.118049+0000", 2024-08-19T14:47:44.323 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:47:44.323 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.324 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.324 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:47:44.324 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:24.118049+0000", 2024-08-19T14:47:44.324 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:47:44.324 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.324 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.324 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:47:44.324 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:24.118050+0000", 2024-08-19T14:47:44.324 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.7199999999999999e-07 2024-08-19T14:47:44.324 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.324 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.325 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:47:44.325 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:24.118081+0000", 2024-08-19T14:47:44.325 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.1810999999999997e-05 2024-08-19T14:47:44.325 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.325 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.325 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:47:44.325 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:24.118092+0000", 2024-08-19T14:47:44.325 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0947999999999999e-05 2024-08-19T14:47:44.325 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.325 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.325 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:47:44.325 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:24.118098+0000", 2024-08-19T14:47:44.326 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.31e-06 2024-08-19T14:47:44.326 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.326 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.326 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:47:44.326 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:24.121580+0000", 2024-08-19T14:47:44.326 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.003482312 2024-08-19T14:47:44.326 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.326 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.326 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for peered", 2024-08-19T14:47:44.326 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:24.121586+0000", 2024-08-19T14:47:44.326 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.6760000000000004e-06 2024-08-19T14:47:44.327 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.327 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.327 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:47:44.327 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:24.570650+0000", 2024-08-19T14:47:44.327 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.44906478799999999 2024-08-19T14:47:44.327 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.327 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.327 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:47:44.327 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:24.570787+0000", 2024-08-19T14:47:44.328 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013684400000000001 2024-08-19T14:47:44.328 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.328 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.328 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 6,11", 2024-08-19T14:47:44.328 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:24.570963+0000", 2024-08-19T14:47:44.328 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000175899 2024-08-19T14:47:44.328 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.328 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.328 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:47:44.328 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:24.584203+0000", 2024-08-19T14:47:44.328 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.013239552 2024-08-19T14:47:44.329 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.329 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.329 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:47:44.329 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:24.584249+0000", 2024-08-19T14:47:44.329 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.6693000000000002e-05 2024-08-19T14:47:44.329 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.329 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.329 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:47:44.329 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:24.584304+0000", 2024-08-19T14:47:44.329 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4979999999999999e-05 2024-08-19T14:47:44.329 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.330 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.330 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:47:44.330 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:24.584368+0000", 2024-08-19T14:47:44.330 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.3405999999999996e-05 2024-08-19T14:47:44.330 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.330 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.330 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:47:44.330 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:45:24.587695+0000", 2024-08-19T14:47:44.330 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0033268120000000002 2024-08-19T14:47:44.330 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:47:44.330 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:47:44.331 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:47:44.331 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.331 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.331 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.27235.0:106748 9.9 9:946cc505:::benchmark_data_smithi045_152862_object12243:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e873)", 2024-08-19T14:47:44.331 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:46:06.310823+0000", 2024-08-19T14:47:44.331 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 97.240329021999997, 2024-08-19T14:47:44.331 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.22430883700000001, 2024-08-19T14:47:44.331 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:47:44.331 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:47:44.331 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:47:44.331 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:47:44.331 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.27235", 2024-08-19T14:47:44.332 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2746512116", 2024-08-19T14:47:44.332 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 106748 2024-08-19T14:47:44.332 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.332 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:47:44.332 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.332 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:47:44.332 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:06.310823+0000", 2024-08-19T14:47:44.332 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:47:44.332 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.332 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.332 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:47:44.332 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:06.310823+0000", 2024-08-19T14:47:44.333 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:47:44.333 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.333 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.333 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:47:44.333 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:06.310824+0000", 2024-08-19T14:47:44.333 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2759999999999999e-06 2024-08-19T14:47:44.333 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.333 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.333 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:47:44.333 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:06.310831+0000", 2024-08-19T14:47:44.333 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.6030000000000001e-06 2024-08-19T14:47:44.334 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.334 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.334 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:47:44.334 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:06.310848+0000", 2024-08-19T14:47:44.334 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.662e-05 2024-08-19T14:47:44.334 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.334 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.334 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:47:44.334 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:06.310856+0000", 2024-08-19T14:47:44.334 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.5399999999999996e-06 2024-08-19T14:47:44.334 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.335 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.335 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:47:44.335 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:06.311319+0000", 2024-08-19T14:47:44.335 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00046295499999999997 2024-08-19T14:47:44.335 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.335 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.335 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:47:44.335 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:06.311442+0000", 2024-08-19T14:47:44.335 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012281 2024-08-19T14:47:44.335 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.335 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.336 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 6,11", 2024-08-19T14:47:44.336 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:06.311557+0000", 2024-08-19T14:47:44.336 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000114957 2024-08-19T14:47:44.336 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.336 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.336 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:47:44.336 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:06.316682+0000", 2024-08-19T14:47:44.336 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0051252169999999996 2024-08-19T14:47:44.336 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.336 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.336 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:47:44.336 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:06.316849+0000", 2024-08-19T14:47:44.337 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00016656599999999999 2024-08-19T14:47:44.337 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.337 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.337 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:47:44.337 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:06.532778+0000", 2024-08-19T14:47:44.337 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.215929291 2024-08-19T14:47:44.337 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.337 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.337 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:47:44.337 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:06.532863+0000", 2024-08-19T14:47:44.337 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.4729999999999997e-05 2024-08-19T14:47:44.337 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.338 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.338 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:47:44.338 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:06.535132+0000", 2024-08-19T14:47:44.338 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0022692720000000001 2024-08-19T14:47:44.338 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:47:44.338 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:47:44.338 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:47:44.338 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.338 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.338 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.27235.0:106754 9.9 9:91c2bc64:::benchmark_data_smithi045_152862_object12249:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e873)", 2024-08-19T14:47:44.338 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:46:06.319256+0000", 2024-08-19T14:47:44.338 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 97.231896030000001, 2024-08-19T14:47:44.339 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.21748487499999999, 2024-08-19T14:47:44.339 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:47:44.339 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:47:44.339 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:47:44.339 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:47:44.339 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.27235", 2024-08-19T14:47:44.339 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2746512116", 2024-08-19T14:47:44.339 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 106754 2024-08-19T14:47:44.339 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.339 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:47:44.339 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.339 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:47:44.340 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:06.319256+0000", 2024-08-19T14:47:44.340 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:47:44.340 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.340 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.340 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:47:44.340 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:06.319256+0000", 2024-08-19T14:47:44.340 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:47:44.340 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.340 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.340 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:47:44.340 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:06.319258+0000", 2024-08-19T14:47:44.340 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2520000000000002e-06 2024-08-19T14:47:44.341 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.341 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.341 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:47:44.341 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:06.319264+0000", 2024-08-19T14:47:44.341 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.3410000000000002e-06 2024-08-19T14:47:44.341 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.341 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.341 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:47:44.341 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:06.319276+0000", 2024-08-19T14:47:44.341 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2311e-05 2024-08-19T14:47:44.341 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.341 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.341 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:47:44.342 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:06.319283+0000", 2024-08-19T14:47:44.342 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.1180000000000003e-06 2024-08-19T14:47:44.342 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.342 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.343 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:47:44.343 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:06.320792+0000", 2024-08-19T14:47:44.343 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001509217 2024-08-19T14:47:44.343 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.343 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.343 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:47:44.343 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:06.320935+0000", 2024-08-19T14:47:44.343 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014270099999999999 2024-08-19T14:47:44.343 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.344 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.344 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 6,11", 2024-08-19T14:47:44.344 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:06.321011+0000", 2024-08-19T14:47:44.344 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.6279000000000006e-05 2024-08-19T14:47:44.344 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.344 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.344 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:47:44.344 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:06.326124+0000", 2024-08-19T14:47:44.345 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0051122090000000004 2024-08-19T14:47:44.345 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.345 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.345 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:47:44.345 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:06.326139+0000", 2024-08-19T14:47:44.345 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5218000000000001e-05 2024-08-19T14:47:44.345 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.345 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.345 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:47:44.345 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:06.535185+0000", 2024-08-19T14:47:44.345 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20904619599999999 2024-08-19T14:47:44.345 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.346 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.346 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:47:44.346 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:06.535221+0000", 2024-08-19T14:47:44.346 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.6022999999999997e-05 2024-08-19T14:47:44.346 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.346 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.346 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:47:44.346 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:06.536741+0000", 2024-08-19T14:47:44.346 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00152001 2024-08-19T14:47:44.346 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:47:44.346 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:47:44.346 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:47:44.347 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.347 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.347 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.27235.0:115453 9.6 9:6de1d98d:::benchmark_data_smithi045_152862_object20948:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e874)", 2024-08-19T14:47:44.347 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:46:16.715524+0000", 2024-08-19T14:47:44.347 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 86.835628392999993, 2024-08-19T14:47:44.347 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.22357210899999999, 2024-08-19T14:47:44.347 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:47:44.347 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:47:44.347 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:47:44.347 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:47:44.347 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.27235", 2024-08-19T14:47:44.347 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2746512116", 2024-08-19T14:47:44.348 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 115453 2024-08-19T14:47:44.348 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.348 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:47:44.348 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.348 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:47:44.348 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.715524+0000", 2024-08-19T14:47:44.348 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:47:44.348 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.348 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.348 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:47:44.348 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.715524+0000", 2024-08-19T14:47:44.348 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:47:44.349 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.349 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.349 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:47:44.349 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.715526+0000", 2024-08-19T14:47:44.349 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9829999999999999e-06 2024-08-19T14:47:44.349 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.349 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.349 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:47:44.349 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.715531+0000", 2024-08-19T14:47:44.349 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.9089999999999999e-06 2024-08-19T14:47:44.349 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.350 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.350 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:47:44.350 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.715550+0000", 2024-08-19T14:47:44.350 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9020000000000001e-05 2024-08-19T14:47:44.350 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.350 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.350 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:47:44.350 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.715557+0000", 2024-08-19T14:47:44.350 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.6020000000000002e-06 2024-08-19T14:47:44.350 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.350 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.351 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:47:44.351 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.715707+0000", 2024-08-19T14:47:44.351 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014958199999999999 2024-08-19T14:47:44.351 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.351 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.351 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:47:44.351 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.719135+0000", 2024-08-19T14:47:44.351 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.003428314 2024-08-19T14:47:44.351 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.351 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.351 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,9", 2024-08-19T14:47:44.351 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.719455+0000", 2024-08-19T14:47:44.352 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00031948000000000002 2024-08-19T14:47:44.352 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.352 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.352 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:47:44.352 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.738265+0000", 2024-08-19T14:47:44.352 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.018810259999999999 2024-08-19T14:47:44.352 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.352 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.352 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:47:44.352 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.739884+0000", 2024-08-19T14:47:44.352 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0016195420000000001 2024-08-19T14:47:44.352 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.353 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.353 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:47:44.353 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.938991+0000", 2024-08-19T14:47:44.353 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.19910667800000001 2024-08-19T14:47:44.353 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.353 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.353 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:47:44.353 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.939051+0000", 2024-08-19T14:47:44.353 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.9466999999999998e-05 2024-08-19T14:47:44.353 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.353 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.353 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:47:44.354 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.939096+0000", 2024-08-19T14:47:44.354 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.5272000000000003e-05 2024-08-19T14:47:44.354 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:47:44.354 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:47:44.354 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:47:44.354 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.354 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.354 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.27235.0:115457 9.6 9:6a9ee07b:::benchmark_data_smithi045_152862_object20952:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e874)", 2024-08-19T14:47:44.354 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:46:16.717569+0000", 2024-08-19T14:47:44.354 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 86.833583298999997, 2024-08-19T14:47:44.354 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.221683884, 2024-08-19T14:47:44.354 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:47:44.355 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:47:44.355 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:47:44.355 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:47:44.355 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.27235", 2024-08-19T14:47:44.355 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2746512116", 2024-08-19T14:47:44.355 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 115457 2024-08-19T14:47:44.355 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.355 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:47:44.355 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.355 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:47:44.355 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.717569+0000", 2024-08-19T14:47:44.356 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:47:44.356 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.356 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.356 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:47:44.356 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.717569+0000", 2024-08-19T14:47:44.356 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:47:44.356 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.356 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.356 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:47:44.356 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.717570+0000", 2024-08-19T14:47:44.356 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0720000000000001e-06 2024-08-19T14:47:44.356 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.357 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.357 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:47:44.357 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.717575+0000", 2024-08-19T14:47:44.357 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.057e-06 2024-08-19T14:47:44.357 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.357 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.357 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:47:44.357 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.717586+0000", 2024-08-19T14:47:44.357 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0828e-05 2024-08-19T14:47:44.357 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.357 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.357 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:47:44.358 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.717593+0000", 2024-08-19T14:47:44.358 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.2080000000000003e-06 2024-08-19T14:47:44.358 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.358 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.358 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:47:44.358 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.723563+0000", 2024-08-19T14:47:44.358 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0059695629999999998 2024-08-19T14:47:44.358 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.358 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.358 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:47:44.358 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.723676+0000", 2024-08-19T14:47:44.359 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000113378 2024-08-19T14:47:44.359 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.359 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.359 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,9", 2024-08-19T14:47:44.359 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.723788+0000", 2024-08-19T14:47:44.359 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000111947 2024-08-19T14:47:44.359 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.359 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.359 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:47:44.359 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.738299+0000", 2024-08-19T14:47:44.359 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.014511391 2024-08-19T14:47:44.359 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.360 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.360 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:47:44.360 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.739892+0000", 2024-08-19T14:47:44.360 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001592256 2024-08-19T14:47:44.360 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.360 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.360 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:47:44.360 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.939154+0000", 2024-08-19T14:47:44.360 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.199262149 2024-08-19T14:47:44.360 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.360 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.360 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:47:44.361 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.939213+0000", 2024-08-19T14:47:44.361 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.8922999999999998e-05 2024-08-19T14:47:44.361 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.361 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.361 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:47:44.361 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.939253+0000", 2024-08-19T14:47:44.361 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.0111999999999998e-05 2024-08-19T14:47:44.361 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:47:44.361 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:47:44.361 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:47:44.361 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.361 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.362 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.27235.0:115459 9.6 9:64b47f6d:::benchmark_data_smithi045_152862_object20954:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e874)", 2024-08-19T14:47:44.362 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:46:16.721715+0000", 2024-08-19T14:47:44.362 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 86.829437495999997, 2024-08-19T14:47:44.362 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.23122553000000001, 2024-08-19T14:47:44.362 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:47:44.362 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:47:44.362 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:47:44.362 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:47:44.362 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.27235", 2024-08-19T14:47:44.362 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2746512116", 2024-08-19T14:47:44.362 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 115459 2024-08-19T14:47:44.363 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.363 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:47:44.363 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.363 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:47:44.363 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.721715+0000", 2024-08-19T14:47:44.363 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:47:44.363 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.363 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.363 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:47:44.363 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.721715+0000", 2024-08-19T14:47:44.363 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:47:44.363 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.364 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.364 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:47:44.364 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.721716+0000", 2024-08-19T14:47:44.364 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5260000000000001e-06 2024-08-19T14:47:44.365 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.365 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.365 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:47:44.365 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.721721+0000", 2024-08-19T14:47:44.366 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2329999999999997e-06 2024-08-19T14:47:44.366 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.366 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.366 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:47:44.366 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.721735+0000", 2024-08-19T14:47:44.366 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4012999999999999e-05 2024-08-19T14:47:44.366 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.366 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.366 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:47:44.366 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.721743+0000", 2024-08-19T14:47:44.366 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.1040000000000001e-06 2024-08-19T14:47:44.367 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.367 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.367 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:47:44.367 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.731362+0000", 2024-08-19T14:47:44.367 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0096199230000000007 2024-08-19T14:47:44.367 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.367 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.367 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:47:44.367 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.731644+0000", 2024-08-19T14:47:44.367 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00028108199999999999 2024-08-19T14:47:44.367 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.368 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.368 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,9", 2024-08-19T14:47:44.368 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.732053+0000", 2024-08-19T14:47:44.368 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00040926200000000003 2024-08-19T14:47:44.368 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.368 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.368 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:47:44.368 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.741843+0000", 2024-08-19T14:47:44.368 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0097904289999999998 2024-08-19T14:47:44.368 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.368 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.368 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:47:44.369 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.744351+0000", 2024-08-19T14:47:44.369 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0025073869999999998 2024-08-19T14:47:44.369 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.369 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.369 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:47:44.369 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.952841+0000", 2024-08-19T14:47:44.369 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.208489909 2024-08-19T14:47:44.369 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.369 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.369 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:47:44.369 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.952890+0000", 2024-08-19T14:47:44.369 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.9703000000000002e-05 2024-08-19T14:47:44.370 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.370 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.370 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:47:44.370 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.952940+0000", 2024-08-19T14:47:44.370 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.9959000000000001e-05 2024-08-19T14:47:44.370 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:47:44.370 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:47:44.370 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:47:44.370 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.370 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.371 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.27235.0:115462 9.e 9:7818abf4:::benchmark_data_smithi045_152862_object20957:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e874)", 2024-08-19T14:47:44.371 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:46:16.725795+0000", 2024-08-19T14:47:44.371 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 86.825357252000003, 2024-08-19T14:47:44.371 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.227145287, 2024-08-19T14:47:44.371 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:47:44.371 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:47:44.371 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:47:44.371 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:47:44.371 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.27235", 2024-08-19T14:47:44.371 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2746512116", 2024-08-19T14:47:44.371 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 115462 2024-08-19T14:47:44.372 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.373 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:47:44.373 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.373 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:47:44.373 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.725795+0000", 2024-08-19T14:47:44.373 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:47:44.373 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.373 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.373 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:47:44.374 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.725795+0000", 2024-08-19T14:47:44.374 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:47:44.374 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.374 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.374 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:47:44.374 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.725796+0000", 2024-08-19T14:47:44.374 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0699999999999999e-06 2024-08-19T14:47:44.374 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.374 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.374 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:47:44.374 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.725801+0000", 2024-08-19T14:47:44.374 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.8459999999999999e-06 2024-08-19T14:47:44.375 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.375 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.375 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:47:44.375 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.725815+0000", 2024-08-19T14:47:44.375 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3866000000000001e-05 2024-08-19T14:47:44.375 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.375 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.375 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:47:44.375 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.725822+0000", 2024-08-19T14:47:44.375 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.6529999999999994e-06 2024-08-19T14:47:44.376 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.376 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.376 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:47:44.376 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.729594+0000", 2024-08-19T14:47:44.376 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0037721590000000002 2024-08-19T14:47:44.376 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.376 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.376 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:47:44.376 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.729715+0000", 2024-08-19T14:47:44.376 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000120121 2024-08-19T14:47:44.376 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.376 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.377 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,9", 2024-08-19T14:47:44.377 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.729822+0000", 2024-08-19T14:47:44.377 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000107821 2024-08-19T14:47:44.377 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.377 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.377 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:47:44.377 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.741831+0000", 2024-08-19T14:47:44.377 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.012008649 2024-08-19T14:47:44.377 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.377 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.377 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:47:44.377 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.744309+0000", 2024-08-19T14:47:44.378 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0024783259999999999 2024-08-19T14:47:44.378 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.378 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.378 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:47:44.378 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.952822+0000", 2024-08-19T14:47:44.378 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20851298500000001 2024-08-19T14:47:44.378 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.378 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.378 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:47:44.378 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.952885+0000", 2024-08-19T14:47:44.378 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.2398999999999997e-05 2024-08-19T14:47:44.378 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.379 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.379 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:47:44.379 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.952940+0000", 2024-08-19T14:47:44.379 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.5392000000000001e-05 2024-08-19T14:47:44.379 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:47:44.379 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:47:44.379 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:47:44.379 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.379 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.379 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.27235.0:115466 9.e 9:75b251fa:::benchmark_data_smithi045_152862_object20961:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e874)", 2024-08-19T14:47:44.379 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:46:16.731009+0000", 2024-08-19T14:47:44.379 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 86.820143004000002, 2024-08-19T14:47:44.380 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.226066922, 2024-08-19T14:47:44.380 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:47:44.380 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:47:44.380 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:47:44.380 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:47:44.380 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.27235", 2024-08-19T14:47:44.380 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2746512116", 2024-08-19T14:47:44.380 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 115466 2024-08-19T14:47:44.380 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.380 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:47:44.381 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.381 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:47:44.381 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.731009+0000", 2024-08-19T14:47:44.381 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:47:44.381 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.381 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.381 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:47:44.381 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.731009+0000", 2024-08-19T14:47:44.381 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:47:44.381 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.382 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.382 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:47:44.382 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.731011+0000", 2024-08-19T14:47:44.382 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4419999999999999e-06 2024-08-19T14:47:44.382 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.382 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.382 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:47:44.382 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.731017+0000", 2024-08-19T14:47:44.382 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.7270000000000001e-06 2024-08-19T14:47:44.382 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.382 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.382 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:47:44.383 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.731032+0000", 2024-08-19T14:47:44.383 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4718e-05 2024-08-19T14:47:44.383 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.383 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.383 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:47:44.383 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.731039+0000", 2024-08-19T14:47:44.383 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4220000000000003e-06 2024-08-19T14:47:44.383 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.383 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.383 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:47:44.383 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.733764+0000", 2024-08-19T14:47:44.383 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.002724476 2024-08-19T14:47:44.384 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.384 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.384 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:47:44.384 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.733906+0000", 2024-08-19T14:47:44.384 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000141872 2024-08-19T14:47:44.384 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.384 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.384 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,9", 2024-08-19T14:47:44.384 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.734025+0000", 2024-08-19T14:47:44.384 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000118937 2024-08-19T14:47:44.384 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.384 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.385 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:47:44.385 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.741837+0000", 2024-08-19T14:47:44.385 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0078125729999999997 2024-08-19T14:47:44.385 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.385 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.385 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:47:44.385 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.751909+0000", 2024-08-19T14:47:44.385 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.010071503000000001 2024-08-19T14:47:44.385 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.385 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.385 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:47:44.385 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.957017+0000", 2024-08-19T14:47:44.386 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20510845999999999 2024-08-19T14:47:44.386 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.386 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.386 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:47:44.386 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.957048+0000", 2024-08-19T14:47:44.386 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.0301e-05 2024-08-19T14:47:44.386 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.386 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.386 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:47:44.386 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.957076+0000", 2024-08-19T14:47:44.386 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.8490999999999999e-05 2024-08-19T14:47:44.386 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:47:44.387 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:47:44.387 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:47:44.387 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.387 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.387 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.27235.0:115470 9.e 9:7bfed996:::benchmark_data_smithi045_152862_object20965:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e874)", 2024-08-19T14:47:44.387 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:46:16.733422+0000", 2024-08-19T14:47:44.387 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 86.817730620000006, 2024-08-19T14:47:44.387 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.23689764399999999, 2024-08-19T14:47:44.387 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:47:44.387 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:47:44.387 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:47:44.387 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:47:44.388 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.27235", 2024-08-19T14:47:44.388 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2746512116", 2024-08-19T14:47:44.388 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 115470 2024-08-19T14:47:44.388 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.388 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:47:44.388 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.388 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:47:44.388 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.733422+0000", 2024-08-19T14:47:44.388 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:47:44.388 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.388 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.388 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:47:44.389 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.733422+0000", 2024-08-19T14:47:44.389 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:47:44.389 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.389 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.389 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:47:44.389 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.733423+0000", 2024-08-19T14:47:44.389 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1790000000000001e-06 2024-08-19T14:47:44.389 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.389 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.389 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:47:44.389 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.733427+0000", 2024-08-19T14:47:44.390 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.6439999999999999e-06 2024-08-19T14:47:44.390 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.390 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.390 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:47:44.390 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.733442+0000", 2024-08-19T14:47:44.390 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4228e-05 2024-08-19T14:47:44.390 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.390 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.390 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:47:44.390 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.733450+0000", 2024-08-19T14:47:44.390 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.0499999999999992e-06 2024-08-19T14:47:44.391 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.391 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.391 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:47:44.391 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.738327+0000", 2024-08-19T14:47:44.391 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0048775609999999999 2024-08-19T14:47:44.391 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.391 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.391 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:47:44.391 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.738430+0000", 2024-08-19T14:47:44.391 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000102484 2024-08-19T14:47:44.391 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.392 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.392 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,9", 2024-08-19T14:47:44.392 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.738526+0000", 2024-08-19T14:47:44.392 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.6408000000000003e-05 2024-08-19T14:47:44.392 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.392 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.392 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:47:44.392 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.744492+0000", 2024-08-19T14:47:44.392 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0059654790000000001 2024-08-19T14:47:44.392 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.393 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.393 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:47:44.393 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.753970+0000", 2024-08-19T14:47:44.393 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0094780249999999993 2024-08-19T14:47:44.393 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.393 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.393 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:47:44.393 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.970169+0000", 2024-08-19T14:47:44.393 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.21619945600000001 2024-08-19T14:47:44.393 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.393 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.394 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:47:44.394 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.970239+0000", 2024-08-19T14:47:44.394 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.0234999999999994e-05 2024-08-19T14:47:44.394 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.394 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.394 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:47:44.394 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.970319+0000", 2024-08-19T14:47:44.394 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.9895000000000007e-05 2024-08-19T14:47:44.394 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:47:44.394 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:47:44.394 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:47:44.394 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.395 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.395 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.27235.0:115479 9.e 9:774bc315:::benchmark_data_smithi045_152862_object20974:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e874)", 2024-08-19T14:47:44.395 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:46:16.747488+0000", 2024-08-19T14:47:44.395 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 86.803663943000004, 2024-08-19T14:47:44.395 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.23003422000000001, 2024-08-19T14:47:44.395 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:47:44.395 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:47:44.395 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:47:44.395 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:47:44.395 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.27235", 2024-08-19T14:47:44.395 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2746512116", 2024-08-19T14:47:44.396 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 115479 2024-08-19T14:47:44.396 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.396 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:47:44.396 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.396 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:47:44.396 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.747488+0000", 2024-08-19T14:47:44.396 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:47:44.396 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.396 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.396 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:47:44.396 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.747488+0000", 2024-08-19T14:47:44.396 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:47:44.397 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.397 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.397 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:47:44.397 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.747490+0000", 2024-08-19T14:47:44.397 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.483e-06 2024-08-19T14:47:44.397 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.397 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.397 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:47:44.397 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.747495+0000", 2024-08-19T14:47:44.397 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1100000000000002e-06 2024-08-19T14:47:44.397 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.397 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.398 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:47:44.398 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.747510+0000", 2024-08-19T14:47:44.398 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5366000000000002e-05 2024-08-19T14:47:44.398 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.398 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.398 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:47:44.398 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.747518+0000", 2024-08-19T14:47:44.398 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.6149999999999999e-06 2024-08-19T14:47:44.398 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.398 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.398 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:47:44.399 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.747595+0000", 2024-08-19T14:47:44.399 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.7226999999999998e-05 2024-08-19T14:47:44.399 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.399 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.399 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-08-19T14:47:44.399 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.747718+0000", 2024-08-19T14:47:44.399 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012297999999999999 2024-08-19T14:47:44.399 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.399 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.399 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,9", 2024-08-19T14:47:44.399 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.747829+0000", 2024-08-19T14:47:44.399 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000111178 2024-08-19T14:47:44.400 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.400 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.400 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-08-19T14:47:44.400 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.751551+0000", 2024-08-19T14:47:44.400 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0037219129999999999 2024-08-19T14:47:44.400 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.400 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.400 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:47:44.400 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.754056+0000", 2024-08-19T14:47:44.400 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0025054370000000001 2024-08-19T14:47:44.400 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.400 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.400 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:47:44.401 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.977377+0000", 2024-08-19T14:47:44.401 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.22332007400000001 2024-08-19T14:47:44.401 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.401 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.401 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-08-19T14:47:44.401 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.977428+0000", 2024-08-19T14:47:44.401 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1455000000000001e-05 2024-08-19T14:47:44.401 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.401 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.401 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:47:44.401 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:46:16.977522+0000", 2024-08-19T14:47:44.402 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.4382000000000005e-05 2024-08-19T14:47:44.402 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:47:44.402 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:47:44.402 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:47:44.402 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.402 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.402 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.27235.0:176046 9.7 9.93e5a5e7 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e914)", 2024-08-19T14:47:44.402 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:47:41.814426+0000", 2024-08-19T14:47:44.402 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 1.7367257999999999, 2024-08-19T14:47:44.402 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.98927618399999995, 2024-08-19T14:47:44.402 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:47:44.403 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:47:44.403 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-08-19T14:47:44.403 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:47:44.403 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.27235", 2024-08-19T14:47:44.403 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2746512116", 2024-08-19T14:47:44.403 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 176046 2024-08-19T14:47:44.403 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.403 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:47:44.403 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.403 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:47:44.403 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:47:41.814426+0000", 2024-08-19T14:47:44.403 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:47:44.404 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.404 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.404 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:47:44.404 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:47:41.814426+0000", 2024-08-19T14:47:44.404 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:47:44.404 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.404 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.404 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:47:44.404 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:47:41.814428+0000", 2024-08-19T14:47:44.404 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5400000000000001e-06 2024-08-19T14:47:44.404 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.404 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.405 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:47:44.405 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:47:41.814434+0000", 2024-08-19T14:47:44.405 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.8640000000000001e-06 2024-08-19T14:47:44.405 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.405 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.405 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:47:44.405 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:47:41.814451+0000", 2024-08-19T14:47:44.405 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6850999999999998e-05 2024-08-19T14:47:44.405 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.405 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.405 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:47:44.405 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:47:41.814460+0000", 2024-08-19T14:47:44.406 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.4490000000000005e-06 2024-08-19T14:47:44.406 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.406 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.406 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:47:44.406 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:47:41.814597+0000", 2024-08-19T14:47:44.406 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000136494 2024-08-19T14:47:44.406 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.406 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.406 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for peered", 2024-08-19T14:47:44.406 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:47:41.814608+0000", 2024-08-19T14:47:44.406 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1323e-05 2024-08-19T14:47:44.406 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.407 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.407 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:47:44.407 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:47:42.803672+0000", 2024-08-19T14:47:44.407 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.98906423799999998 2024-08-19T14:47:44.407 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.407 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.407 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:47:44.407 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:47:42.803703+0000", 2024-08-19T14:47:44.407 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.0425000000000001e-05 2024-08-19T14:47:44.407 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:47:44.407 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:47:44.407 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:47:44.408 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.408 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.408 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.27235.0:176047 9.7 9.b20f0ec7 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e914)", 2024-08-19T14:47:44.408 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:47:41.814482+0000", 2024-08-19T14:47:44.408 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 1.736670511, 2024-08-19T14:47:44.408 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.98925941900000003, 2024-08-19T14:47:44.408 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:47:44.408 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:47:44.408 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-08-19T14:47:44.408 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:47:44.408 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.27235", 2024-08-19T14:47:44.408 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2746512116", 2024-08-19T14:47:44.409 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 176047 2024-08-19T14:47:44.409 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.409 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:47:44.409 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.409 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:47:44.409 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:47:41.814482+0000", 2024-08-19T14:47:44.409 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:47:44.409 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.409 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.409 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:47:44.409 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:47:41.814482+0000", 2024-08-19T14:47:44.409 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:47:44.410 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.410 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.410 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:47:44.410 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:47:41.814482+0000", 2024-08-19T14:47:44.410 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7299999999999996e-07 2024-08-19T14:47:44.410 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.410 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.410 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:47:44.410 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:47:41.814486+0000", 2024-08-19T14:47:44.410 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.3689999999999998e-06 2024-08-19T14:47:44.410 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.411 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.411 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:47:44.411 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:47:41.814494+0000", 2024-08-19T14:47:44.411 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.5620000000000004e-06 2024-08-19T14:47:44.411 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.411 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.411 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:47:44.411 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:47:41.814543+0000", 2024-08-19T14:47:44.411 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.8857999999999999e-05 2024-08-19T14:47:44.411 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.411 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.412 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:47:44.412 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:47:41.814635+0000", 2024-08-19T14:47:44.412 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.2312000000000007e-05 2024-08-19T14:47:44.412 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.412 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.412 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for peered", 2024-08-19T14:47:44.412 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:47:41.814644+0000", 2024-08-19T14:47:44.412 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.8060000000000001e-06 2024-08-19T14:47:44.412 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.412 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.412 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:47:44.412 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:47:42.803733+0000", 2024-08-19T14:47:44.413 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.98908848500000002 2024-08-19T14:47:44.413 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.413 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.413 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:47:44.413 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:47:42.803741+0000", 2024-08-19T14:47:44.413 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.4540000000000007e-06 2024-08-19T14:47:44.413 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:47:44.413 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:47:44.413 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:47:44.413 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.413 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.413 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.27235.0:176052 9.7 9.177d7337 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e914)", 2024-08-19T14:47:44.413 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:47:41.819746+0000", 2024-08-19T14:47:44.414 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 1.731406301, 2024-08-19T14:47:44.414 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.98402606500000001, 2024-08-19T14:47:44.414 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:47:44.414 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:47:44.414 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-08-19T14:47:44.414 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:47:44.414 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.27235", 2024-08-19T14:47:44.414 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2746512116", 2024-08-19T14:47:44.414 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 176052 2024-08-19T14:47:44.414 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.415 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:47:44.415 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.415 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:47:44.415 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:47:41.819746+0000", 2024-08-19T14:47:44.415 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:47:44.415 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.415 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.415 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:47:44.415 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:47:41.819746+0000", 2024-08-19T14:47:44.415 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:47:44.415 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.416 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.416 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:47:44.416 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:47:41.819747+0000", 2024-08-19T14:47:44.416 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.2200000000000002e-07 2024-08-19T14:47:44.416 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.416 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.416 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:47:44.416 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:47:41.819751+0000", 2024-08-19T14:47:44.416 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.6890000000000003e-06 2024-08-19T14:47:44.416 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.416 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.416 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:47:44.417 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:47:41.819764+0000", 2024-08-19T14:47:44.417 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2129e-05 2024-08-19T14:47:44.417 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.417 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.417 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:47:44.417 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:47:41.819769+0000", 2024-08-19T14:47:44.417 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.5929999999999996e-06 2024-08-19T14:47:44.417 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.417 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.417 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:47:44.417 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:47:41.819821+0000", 2024-08-19T14:47:44.417 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1947000000000003e-05 2024-08-19T14:47:44.417 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.418 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.418 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for peered", 2024-08-19T14:47:44.418 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:47:41.819828+0000", 2024-08-19T14:47:44.418 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.9670000000000004e-06 2024-08-19T14:47:44.418 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.418 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.418 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:47:44.418 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:47:42.803765+0000", 2024-08-19T14:47:44.418 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.98393696600000002 2024-08-19T14:47:44.418 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.419 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.419 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:47:44.419 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:47:42.803772+0000", 2024-08-19T14:47:44.419 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.8519999999999998e-06 2024-08-19T14:47:44.419 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:47:44.419 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:47:44.419 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:47:44.419 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.419 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.419 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.27235.0:176057 9.7 9.9c8eb0f7 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e914)", 2024-08-19T14:47:44.419 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-08-19T14:47:41.828116+0000", 2024-08-19T14:47:44.419 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 1.723036236, 2024-08-19T14:47:44.419 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.97569004699999995, 2024-08-19T14:47:44.420 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-08-19T14:47:44.420 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-08-19T14:47:44.420 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-08-19T14:47:44.420 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-08-19T14:47:44.420 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.27235", 2024-08-19T14:47:44.420 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2746512116", 2024-08-19T14:47:44.420 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 176057 2024-08-19T14:47:44.420 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.420 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-08-19T14:47:44.420 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.420 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-08-19T14:47:44.420 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:47:41.828116+0000", 2024-08-19T14:47:44.421 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:47:44.421 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.421 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.421 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-08-19T14:47:44.421 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:47:41.828116+0000", 2024-08-19T14:47:44.421 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-08-19T14:47:44.421 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.421 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.421 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-08-19T14:47:44.421 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:47:41.828117+0000", 2024-08-19T14:47:44.421 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3430000000000001e-06 2024-08-19T14:47:44.422 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.422 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.422 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-08-19T14:47:44.422 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:47:41.828124+0000", 2024-08-19T14:47:44.422 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.6630000000000004e-06 2024-08-19T14:47:44.422 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.422 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.422 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-08-19T14:47:44.422 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:47:41.828138+0000", 2024-08-19T14:47:44.422 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4456e-05 2024-08-19T14:47:44.422 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.422 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.423 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-08-19T14:47:44.423 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:47:41.828145+0000", 2024-08-19T14:47:44.423 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.4740000000000002e-06 2024-08-19T14:47:44.423 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.423 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.423 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:47:44.423 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:47:41.828201+0000", 2024-08-19T14:47:44.423 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.6363999999999998e-05 2024-08-19T14:47:44.423 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.423 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.423 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for peered", 2024-08-19T14:47:44.424 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:47:41.828209+0000", 2024-08-19T14:47:44.424 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.5050000000000001e-06 2024-08-19T14:47:44.424 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.424 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.424 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-08-19T14:47:44.424 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:47:42.803799+0000", 2024-08-19T14:47:44.424 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.97559061899999999 2024-08-19T14:47:44.424 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-08-19T14:47:44.424 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-08-19T14:47:44.424 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-08-19T14:47:44.424 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-08-19T14:47:42.803806+0000", 2024-08-19T14:47:44.424 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.6229999999999999e-06 2024-08-19T14:47:44.425 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:47:44.425 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:47:44.425 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:47:44.425 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-08-19T14:47:44.425 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-08-19T14:47:44.425 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-08-19T14:47:44.425 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:47:44.427 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:47:44.427 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:47:44.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:44.448 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:44.448 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:47:44.549 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:47:44.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:44.571 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:44.571 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:47:44.672 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:47:44.693 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:44.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:44.694 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:47:44.796 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:47:44.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:44.826 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:44.826 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:47:44.927 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:47:44.957 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:44.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:44.958 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:47:45.059 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:47:45.080 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:45.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:45.081 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:47:45.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:44 smithi031 ceph-mon[16336]: 9.c starting backfill to osd.3 from (0'0,0'0] MAX to 913'11015 2024-08-19T14:47:45.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:44 smithi031 ceph-mon[16336]: 9.9 starting backfill to osd.3 from (0'0,0'0] MAX to 913'11081 2024-08-19T14:47:45.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:44 smithi031 ceph-mon[16336]: 9.7 starting backfill to osd.3 from (0'0,0'0] MAX to 913'10863 2024-08-19T14:47:45.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:44 smithi031 ceph-mon[16336]: 9.1 starting backfill to osd.3 from (0'0,0'0] MAX to 913'11084 2024-08-19T14:47:45.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:44 smithi031 ceph-mon[16336]: 9.3 starting backfill to osd.3 from (0'0,0'0] MAX to 913'11139 2024-08-19T14:47:45.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:44 smithi031 ceph-mon[16336]: 9.a starting backfill to osd.3 from (0'0,0'0] MAX to 913'11082 2024-08-19T14:47:45.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:44 smithi031 ceph-mon[16336]: 9.2 starting backfill to osd.3 from (0'0,0'0] MAX to 913'11134 2024-08-19T14:47:45.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:44 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1712283219' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:47:45.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:44 smithi031 ceph-mon[16336]: 9.6 scrub starts 2024-08-19T14:47:45.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:44 smithi031 ceph-mon[16336]: 9.6 scrub ok 2024-08-19T14:47:45.182 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:47:45.218 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:45.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:45.219 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:47:45.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:44 smithi032 ceph-mon[20004]: 9.c starting backfill to osd.3 from (0'0,0'0] MAX to 913'11015 2024-08-19T14:47:45.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:44 smithi032 ceph-mon[20004]: 9.9 starting backfill to osd.3 from (0'0,0'0] MAX to 913'11081 2024-08-19T14:47:45.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:44 smithi032 ceph-mon[20004]: 9.7 starting backfill to osd.3 from (0'0,0'0] MAX to 913'10863 2024-08-19T14:47:45.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:44 smithi032 ceph-mon[20004]: 9.1 starting backfill to osd.3 from (0'0,0'0] MAX to 913'11084 2024-08-19T14:47:45.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:44 smithi032 ceph-mon[20004]: 9.3 starting backfill to osd.3 from (0'0,0'0] MAX to 913'11139 2024-08-19T14:47:45.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:44 smithi032 ceph-mon[20004]: 9.a starting backfill to osd.3 from (0'0,0'0] MAX to 913'11082 2024-08-19T14:47:45.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:44 smithi032 ceph-mon[20004]: 9.2 starting backfill to osd.3 from (0'0,0'0] MAX to 913'11134 2024-08-19T14:47:45.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:44 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1712283219' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:47:45.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:44 smithi032 ceph-mon[20004]: 9.6 scrub starts 2024-08-19T14:47:45.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:44 smithi032 ceph-mon[20004]: 9.6 scrub ok 2024-08-19T14:47:45.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:44 smithi045 ceph-mon[20143]: 9.c starting backfill to osd.3 from (0'0,0'0] MAX to 913'11015 2024-08-19T14:47:45.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:44 smithi045 ceph-mon[20143]: 9.9 starting backfill to osd.3 from (0'0,0'0] MAX to 913'11081 2024-08-19T14:47:45.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:44 smithi045 ceph-mon[20143]: 9.7 starting backfill to osd.3 from (0'0,0'0] MAX to 913'10863 2024-08-19T14:47:45.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:44 smithi045 ceph-mon[20143]: 9.1 starting backfill to osd.3 from (0'0,0'0] MAX to 913'11084 2024-08-19T14:47:45.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:44 smithi045 ceph-mon[20143]: 9.3 starting backfill to osd.3 from (0'0,0'0] MAX to 913'11139 2024-08-19T14:47:45.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:44 smithi045 ceph-mon[20143]: 9.a starting backfill to osd.3 from (0'0,0'0] MAX to 913'11082 2024-08-19T14:47:45.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:44 smithi045 ceph-mon[20143]: 9.2 starting backfill to osd.3 from (0'0,0'0] MAX to 913'11134 2024-08-19T14:47:45.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:44 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1712283219' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:47:45.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:44 smithi045 ceph-mon[20143]: 9.6 scrub starts 2024-08-19T14:47:45.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:44 smithi045 ceph-mon[20143]: 9.6 scrub ok 2024-08-19T14:47:45.321 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:47:45.364 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:45.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:45.365 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:47:45.466 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:47:45.493 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:45.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:45.494 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:47:45.595 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:47:45.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:45.625 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:45.625 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:47:45.727 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:47:45.762 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:45.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:45.763 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:47:45.864 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:47:45.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:45.903 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:45.903 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:47:46.003 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:47:46.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:46.037 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:46.037 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:47:46.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:45 smithi031 ceph-mon[16336]: pgmap v2151: 47 pgs: 1 activating+undersized+remapped, 3 activating+remapped, 2 active+recovering+undersized+degraded+remapped, 41 active+clean; 841 MiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 831 op/s; 80/40353 objects degraded (0.198%); 2438/40353 objects misplaced (6.042%); 11 MiB/s, 180 objects/s recovering 2024-08-19T14:47:46.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:45 smithi031 ceph-mon[16336]: osdmap e917: 12 total, 12 up, 10 in 2024-08-19T14:47:46.138 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:47:46.166 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:46.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:46.167 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:47:46.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:45 smithi032 ceph-mon[20004]: pgmap v2151: 47 pgs: 1 activating+undersized+remapped, 3 activating+remapped, 2 active+recovering+undersized+degraded+remapped, 41 active+clean; 841 MiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 831 op/s; 80/40353 objects degraded (0.198%); 2438/40353 objects misplaced (6.042%); 11 MiB/s, 180 objects/s recovering 2024-08-19T14:47:46.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:45 smithi032 ceph-mon[20004]: osdmap e917: 12 total, 12 up, 10 in 2024-08-19T14:47:46.268 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:47:46.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:45 smithi045 ceph-mon[20143]: pgmap v2151: 47 pgs: 1 activating+undersized+remapped, 3 activating+remapped, 2 active+recovering+undersized+degraded+remapped, 41 active+clean; 841 MiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 831 op/s; 80/40353 objects degraded (0.198%); 2438/40353 objects misplaced (6.042%); 11 MiB/s, 180 objects/s recovering 2024-08-19T14:47:46.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:45 smithi045 ceph-mon[20143]: osdmap e917: 12 total, 12 up, 10 in 2024-08-19T14:47:46.298 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:46.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:46.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:47:46.400 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:47:46.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:46.428 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:46.428 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:47:46.529 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:47:46.568 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:46.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:46.569 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:47:46.670 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:47:46.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:46.692 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:46.692 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:47:46.793 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:47:46.826 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:46.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:46.829 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:47:46.930 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:47:46.964 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:46.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:46.965 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:47:47.046 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:46 smithi045 ceph-mon[20143]: osdmap e918: 12 total, 12 up, 10 in 2024-08-19T14:47:47.066 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:47:47.100 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:47.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:47.101 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:47:47.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:46 smithi031 ceph-mon[16336]: osdmap e918: 12 total, 12 up, 10 in 2024-08-19T14:47:47.202 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:47:47.233 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:47.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:47.234 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:47:47.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:46 smithi032 ceph-mon[20004]: osdmap e918: 12 total, 12 up, 10 in 2024-08-19T14:47:47.335 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:47:47.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:47.370 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:47.370 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:47:47.471 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:47:47.506 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:47.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:47.508 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:47:47.609 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:47:47.625 INFO:tasks.thrashosds.thrasher:in_osds: [1, 4, 8, 2, 11, 9, 10, 6, 7, 3] out_osds: [0, 5] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 8, 9, 11, 10, 2] 2024-08-19T14:47:47.625 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-19T14:47:47.626 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-08-19T14:47:47.626 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd dump -f json-pretty 2024-08-19T14:47:47.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:47.641 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:47.641 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:47:47.742 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:47:47.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:47.785 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:47.786 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:47:47.886 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:47:47.920 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:47.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:47.921 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:47:48.022 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:47:48.046 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:48.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:48.047 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:47:48.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:47 smithi031 ceph-mon[16336]: pgmap v2154: 47 pgs: 1 active+recovering+remapped, 2 active+remapped+backfilling, 2 peering, 3 activating+undersized+remapped, 4 activating+remapped, 1 active+recovering+undersized+degraded+remapped, 34 active+clean; 799 MiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 489 op/s; 25/38316 objects degraded (0.065%); 6146/38316 objects misplaced (16.040%); 13 MiB/s, 200 objects/s recovering 2024-08-19T14:47:48.149 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:47:48.173 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:48.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:48.175 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:47:48.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:47 smithi032 ceph-mon[20004]: pgmap v2154: 47 pgs: 1 active+recovering+remapped, 2 active+remapped+backfilling, 2 peering, 3 activating+undersized+remapped, 4 activating+remapped, 1 active+recovering+undersized+degraded+remapped, 34 active+clean; 799 MiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 489 op/s; 25/38316 objects degraded (0.065%); 6146/38316 objects misplaced (16.040%); 13 MiB/s, 200 objects/s recovering 2024-08-19T14:47:48.275 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:47:48.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:47 smithi045 ceph-mon[20143]: pgmap v2154: 47 pgs: 1 active+recovering+remapped, 2 active+remapped+backfilling, 2 peering, 3 activating+undersized+remapped, 4 activating+remapped, 1 active+recovering+undersized+degraded+remapped, 34 active+clean; 799 MiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 489 op/s; 25/38316 objects degraded (0.065%); 6146/38316 objects misplaced (16.040%); 13 MiB/s, 200 objects/s recovering 2024-08-19T14:47:48.299 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:48.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:48.301 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:47:48.402 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:47:48.424 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:48.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:48.425 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:47:48.526 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:47:48.549 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:48.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:48.550 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:47:48.651 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:47:48.681 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:48.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:48.682 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:47:48.783 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:47:48.813 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:48.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:48.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:47:48.914 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:47:48.937 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:48.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:48.938 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:47:49.039 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:47:49.061 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:49.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:49.062 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:47:49.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:48 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:47:49.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:48 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:47:49.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:48 smithi031 ceph-mon[16336]: osdmap e919: 12 total, 12 up, 10 in 2024-08-19T14:47:49.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:48 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 25/38316 objects degraded (0.065%), 1 pg degraded (PG_DEGRADED) 2024-08-19T14:47:49.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:48 smithi031 ceph-mon[16336]: osdmap e920: 12 total, 12 up, 10 in 2024-08-19T14:47:49.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:48 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:47:49.163 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:47:49.184 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:49.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:49.185 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:47:49.257 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:48 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:47:49.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:48 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:47:49.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:48 smithi032 ceph-mon[20004]: osdmap e919: 12 total, 12 up, 10 in 2024-08-19T14:47:49.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:48 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 25/38316 objects degraded (0.065%), 1 pg degraded (PG_DEGRADED) 2024-08-19T14:47:49.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:48 smithi032 ceph-mon[20004]: osdmap e920: 12 total, 12 up, 10 in 2024-08-19T14:47:49.258 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:48 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:47:49.286 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:47:49.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:48 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:47:49.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:48 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:47:49.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:48 smithi045 ceph-mon[20143]: osdmap e919: 12 total, 12 up, 10 in 2024-08-19T14:47:49.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:48 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 25/38316 objects degraded (0.065%), 1 pg degraded (PG_DEGRADED) 2024-08-19T14:47:49.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:48 smithi045 ceph-mon[20143]: osdmap e920: 12 total, 12 up, 10 in 2024-08-19T14:47:49.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:48 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:47:49.310 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:49.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:49.311 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:47:49.412 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:47:49.434 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:49.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:49.435 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:47:49.474 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-08-19T14:47:49.536 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:47:49.558 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:49.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:49.559 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:47:49.660 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:47:49.681 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:49.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:49.682 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:47:49.783 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:47:49.807 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:49.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:49.808 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:47:49.909 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:47:49.931 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:49.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:49.932 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:47:50.033 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:47:50.056 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:50.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:50.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:47:50.158 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:47:50.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:50.186 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:50.186 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:47:50.287 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:47:50.312 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:50.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:50.313 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:47:50.414 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:47:50.438 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:50.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:50.439 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:47:50.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:50 smithi032 ceph-mon[20004]: pgmap v2157: 47 pgs: 1 active+recovering+remapped, 2 active+remapped+backfilling, 2 peering, 3 activating+undersized+remapped, 4 activating+remapped, 1 active+recovering+undersized+degraded+remapped, 34 active+clean; 781 MiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 240 op/s; 25/37461 objects degraded (0.067%); 6146/37461 objects misplaced (16.406%); 3.7 MiB/s, 58 objects/s recovering 2024-08-19T14:47:50.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:50 smithi032 ceph-mon[20004]: osdmap e921: 12 total, 12 up, 10 in 2024-08-19T14:47:50.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:50 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:47:50.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:50 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:47:50.539 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:47:50.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:50 smithi045 ceph-mon[20143]: pgmap v2157: 47 pgs: 1 active+recovering+remapped, 2 active+remapped+backfilling, 2 peering, 3 activating+undersized+remapped, 4 activating+remapped, 1 active+recovering+undersized+degraded+remapped, 34 active+clean; 781 MiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 240 op/s; 25/37461 objects degraded (0.067%); 6146/37461 objects misplaced (16.406%); 3.7 MiB/s, 58 objects/s recovering 2024-08-19T14:47:50.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:50 smithi045 ceph-mon[20143]: osdmap e921: 12 total, 12 up, 10 in 2024-08-19T14:47:50.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:50 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:47:50.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:50 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:47:50.562 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:50.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:50.563 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:47:50.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:50 smithi031 ceph-mon[16336]: pgmap v2157: 47 pgs: 1 active+recovering+remapped, 2 active+remapped+backfilling, 2 peering, 3 activating+undersized+remapped, 4 activating+remapped, 1 active+recovering+undersized+degraded+remapped, 34 active+clean; 781 MiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 240 op/s; 25/37461 objects degraded (0.067%); 6146/37461 objects misplaced (16.406%); 3.7 MiB/s, 58 objects/s recovering 2024-08-19T14:47:50.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:50 smithi031 ceph-mon[16336]: osdmap e921: 12 total, 12 up, 10 in 2024-08-19T14:47:50.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:50 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:47:50.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:50 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:47:50.664 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:47:50.686 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:50.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:50.687 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:47:50.788 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:47:50.810 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:50.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:50.811 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:47:50.911 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:47:50.933 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:50.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:50.934 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:47:51.035 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:47:51.055 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:51.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:51.056 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:47:51.157 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:47:51.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:51.182 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:51.182 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:47:51.283 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:47:51.312 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:51.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:51.313 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:47:51.414 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:47:51.463 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:51 smithi031 ceph-mon[16336]: osdmap e922: 12 total, 12 up, 10 in 2024-08-19T14:47:51.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:51.476 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:51.476 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:47:51.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:51 smithi032 ceph-mon[20004]: osdmap e922: 12 total, 12 up, 10 in 2024-08-19T14:47:51.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:51 smithi045 ceph-mon[20143]: osdmap e922: 12 total, 12 up, 10 in 2024-08-19T14:47:51.578 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:47:51.603 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:51.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:51.604 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:47:51.705 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:47:51.735 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:51.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:51.737 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:47:51.792 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-19T14:47:51.838 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:47:51.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:51.878 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:51.878 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:47:51.979 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:47:52.020 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:52.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:52.021 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:47:52.122 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:47:52.132 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:52 smithi032 ceph-mon[20004]: pgmap v2160: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 3 active+remapped+backfilling, 1 peering, 2 activating+remapped, 39 active+clean; 704 MiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 377 op/s; 25/33780 objects degraded (0.074%); 4522/33780 objects misplaced (13.387%); 22 MiB/s, 437 objects/s recovering 2024-08-19T14:47:52.156 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:52.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:52.159 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:47:52.261 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:47:52.297 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:52.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:52.298 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:47:52.399 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:47:52.426 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:52.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:52.427 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:47:52.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:52 smithi032 ceph-mon[20004]: osdmap e923: 12 total, 12 up, 10 in 2024-08-19T14:47:52.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:52 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:47:52.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:52 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:47:52.528 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:47:52.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:52 smithi045 ceph-mon[20143]: pgmap v2160: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 3 active+remapped+backfilling, 1 peering, 2 activating+remapped, 39 active+clean; 704 MiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 377 op/s; 25/33780 objects degraded (0.074%); 4522/33780 objects misplaced (13.387%); 22 MiB/s, 437 objects/s recovering 2024-08-19T14:47:52.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:52 smithi045 ceph-mon[20143]: osdmap e923: 12 total, 12 up, 10 in 2024-08-19T14:47:52.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:52 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:47:52.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:52 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:47:52.553 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:52.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:52.554 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:47:52.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:52 smithi031 ceph-mon[16336]: pgmap v2160: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 3 active+remapped+backfilling, 1 peering, 2 activating+remapped, 39 active+clean; 704 MiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 377 op/s; 25/33780 objects degraded (0.074%); 4522/33780 objects misplaced (13.387%); 22 MiB/s, 437 objects/s recovering 2024-08-19T14:47:52.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:52 smithi031 ceph-mon[16336]: osdmap e923: 12 total, 12 up, 10 in 2024-08-19T14:47:52.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:52 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:47:52.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:52 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:47:52.655 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:47:52.688 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:52.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:52.689 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:47:52.790 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:47:52.819 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:52.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:52.820 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:47:52.921 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:47:52.950 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:52.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:52.951 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:47:53.052 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:47:53.075 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:53.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:53.076 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:47:53.177 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:47:53.203 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:53.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:53.204 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:47:53.305 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:47:53.336 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:53.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:53.337 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:47:53.409 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:53 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 25/33780 objects degraded (0.074%), 1 pg degraded (PG_DEGRADED) 2024-08-19T14:47:53.438 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:47:53.479 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:53.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:53.481 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:47:53.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:53 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 25/33780 objects degraded (0.074%), 1 pg degraded (PG_DEGRADED) 2024-08-19T14:47:53.582 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:47:53.612 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:53.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:53.613 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:47:53.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:53 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 25/33780 objects degraded (0.074%), 1 pg degraded (PG_DEGRADED) 2024-08-19T14:47:53.714 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:47:53.740 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:53.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:53.741 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:47:53.842 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:47:53.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:53.871 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:53.872 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:47:53.973 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:47:54.011 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:54.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:54.012 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:47:54.113 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:47:54.148 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:54.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:54.149 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:47:54.230 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:47:54.251 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:47:54.261 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:54 smithi031 ceph-mon[16336]: pgmap v2162: 47 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 2 peering, 43 active+clean; 644 MiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 511 op/s; 25/30885 objects degraded (0.081%); 604/30885 objects misplaced (1.956%); 68 MiB/s, 1.16k objects/s recovering 2024-08-19T14:47:54.261 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:54 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:47:54.261 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:54 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:47:54.262 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:54 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:47:54.262 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:54 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:47:54.271 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:47:54.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:54.291 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:54.291 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:47:54.392 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:47:54.424 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:54.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:54.425 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:47:54.447 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:54 smithi045 ceph-mon[20143]: pgmap v2162: 47 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 2 peering, 43 active+clean; 644 MiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 511 op/s; 25/30885 objects degraded (0.081%); 604/30885 objects misplaced (1.956%); 68 MiB/s, 1.16k objects/s recovering 2024-08-19T14:47:54.447 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:54 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:47:54.447 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:54 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:47:54.447 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:54 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:47:54.447 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:54 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:47:54.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:54 smithi032 ceph-mon[20004]: pgmap v2162: 47 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 2 peering, 43 active+clean; 644 MiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 511 op/s; 25/30885 objects degraded (0.081%); 604/30885 objects misplaced (1.956%); 68 MiB/s, 1.16k objects/s recovering 2024-08-19T14:47:54.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:54 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:47:54.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:54 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:47:54.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:54 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:47:54.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:54 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:47:54.526 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:47:54.566 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:54.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:54.567 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:47:54.669 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:47:54.696 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:54.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:54.698 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:47:54.799 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:47:54.848 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:54.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:54.849 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:47:54.900 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:47:54.900 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:47:54.950 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:47:54.972 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:54.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:54.973 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:47:55.074 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:47:55.112 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:47:55.113 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:55.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:55.114 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:47:55.215 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:47:55.247 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:55.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:55.248 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:47:55.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:55 smithi045 ceph-mon[20143]: osdmap e924: 12 total, 12 up, 10 in 2024-08-19T14:47:55.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:55 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:47:55.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:55 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:47:55.297 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:55 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:47:55.349 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:47:55.375 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:55.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:55.376 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:47:55.477 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:47:55.501 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:55.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:55.502 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:47:55.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:55 smithi032 ceph-mon[20004]: osdmap e924: 12 total, 12 up, 10 in 2024-08-19T14:47:55.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:55 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:47:55.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:55 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:47:55.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:55 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:47:55.549 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:55 smithi031 ceph-mon[16336]: osdmap e924: 12 total, 12 up, 10 in 2024-08-19T14:47:55.549 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:55 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:47:55.549 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:55 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:47:55.549 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:55 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:47:55.603 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:47:55.625 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:55.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:55.626 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:47:55.727 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:47:55.754 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:55.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:55.756 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:47:55.856 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:47:55.884 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:55.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:55.885 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:47:55.986 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:47:56.024 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:56.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:56.025 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:47:56.126 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:47:56.152 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:56.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:56.153 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:47:56.254 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:47:56.263 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-19T14:47:56.282 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:56.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:56.283 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:47:56.384 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:47:56.405 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:56.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:56.406 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:47:56.453 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:56 smithi031 ceph-mon[16336]: pgmap v2164: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 1 peering, 44 active+clean; 551 MiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 697 op/s; 25/26412 objects degraded (0.095%); 604/26412 objects misplaced (2.287%); 55 MiB/s, 1.02k objects/s recovering 2024-08-19T14:47:56.507 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:47:56.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:56 smithi032 ceph-mon[20004]: pgmap v2164: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 1 peering, 44 active+clean; 551 MiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 697 op/s; 25/26412 objects degraded (0.095%); 604/26412 objects misplaced (2.287%); 55 MiB/s, 1.02k objects/s recovering 2024-08-19T14:47:56.530 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:56.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:56.531 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:47:56.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:56 smithi045 ceph-mon[20143]: pgmap v2164: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 1 peering, 44 active+clean; 551 MiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 697 op/s; 25/26412 objects degraded (0.095%); 604/26412 objects misplaced (2.287%); 55 MiB/s, 1.02k objects/s recovering 2024-08-19T14:47:56.632 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:47:56.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:56.665 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:56.665 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:47:56.766 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:47:56.790 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:56.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:56.792 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:47:56.893 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:47:56.906 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-08-19T14:47:57.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:57.076 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:57.076 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:47:57.177 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:47:57.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:57.225 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:57.225 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:47:57.326 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:47:57.360 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:57.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:57.362 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:47:57.452 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:57 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:47:57.452 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:57 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:47:57.463 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:47:57.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:57 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:47:57.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:57 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:47:57.509 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:57.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:57.510 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:47:57.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:57 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:47:57.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:57 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:47:57.611 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:47:57.634 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:57.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:57.635 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:47:57.736 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:47:57.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:57.773 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:57.773 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:47:57.815 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:47:57.851 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:47:57.874 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:47:57.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:57.933 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:57.933 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:47:58.034 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:47:58.068 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:58.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:58.069 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:47:58.112 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:47:58.112 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:47:58.112 INFO:teuthology.orchestra.run.smithi031.stdout: "epoch": 925, 2024-08-19T14:47:58.112 INFO:teuthology.orchestra.run.smithi031.stdout: "fsid": "ed7d1c6e-5e33-11ef-bccf-c7b262605968", 2024-08-19T14:47:58.112 INFO:teuthology.orchestra.run.smithi031.stdout: "created": "2024-08-19T14:05:22.842169+0000", 2024-08-19T14:47:58.112 INFO:teuthology.orchestra.run.smithi031.stdout: "modified": "2024-08-19T14:47:57.171764+0000", 2024-08-19T14:47:58.112 INFO:teuthology.orchestra.run.smithi031.stdout: "last_up_change": "2024-08-19T14:35:10.517930+0000", 2024-08-19T14:47:58.112 INFO:teuthology.orchestra.run.smithi031.stdout: "last_in_change": "2024-08-19T14:47:41.794306+0000", 2024-08-19T14:47:58.112 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-08-19T14:47:58.113 INFO:teuthology.orchestra.run.smithi031.stdout: "flags_num": 5804032, 2024-08-19T14:47:58.113 INFO:teuthology.orchestra.run.smithi031.stdout: "flags_set": [ 2024-08-19T14:47:58.113 INFO:teuthology.orchestra.run.smithi031.stdout: "nodeep-scrub", 2024-08-19T14:47:58.113 INFO:teuthology.orchestra.run.smithi031.stdout: "pglog_hardlimit", 2024-08-19T14:47:58.113 INFO:teuthology.orchestra.run.smithi031.stdout: "purged_snapdirs", 2024-08-19T14:47:58.113 INFO:teuthology.orchestra.run.smithi031.stdout: "recovery_deletes", 2024-08-19T14:47:58.113 INFO:teuthology.orchestra.run.smithi031.stdout: "sortbitwise" 2024-08-19T14:47:58.113 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-08-19T14:47:58.113 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_version": 29, 2024-08-19T14:47:58.113 INFO:teuthology.orchestra.run.smithi031.stdout: "full_ratio": 0.94999998807907104, 2024-08-19T14:47:58.113 INFO:teuthology.orchestra.run.smithi031.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-08-19T14:47:58.113 INFO:teuthology.orchestra.run.smithi031.stdout: "nearfull_ratio": 0.85000002384185791, 2024-08-19T14:47:58.113 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_snapshot": "", 2024-08-19T14:47:58.113 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_max": 9, 2024-08-19T14:47:58.114 INFO:teuthology.orchestra.run.smithi031.stdout: "max_osd": 12, 2024-08-19T14:47:58.114 INFO:teuthology.orchestra.run.smithi031.stdout: "require_min_compat_client": "luminous", 2024-08-19T14:47:58.114 INFO:teuthology.orchestra.run.smithi031.stdout: "min_compat_client": "luminous", 2024-08-19T14:47:58.114 INFO:teuthology.orchestra.run.smithi031.stdout: "require_osd_release": "squid", 2024-08-19T14:47:58.114 INFO:teuthology.orchestra.run.smithi031.stdout: "allow_crimson": false, 2024-08-19T14:47:58.114 INFO:teuthology.orchestra.run.smithi031.stdout: "pools": [ 2024-08-19T14:47:58.114 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.114 INFO:teuthology.orchestra.run.smithi031.stdout: "pool": 1, 2024-08-19T14:47:58.114 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_name": ".mgr", 2024-08-19T14:47:58.114 INFO:teuthology.orchestra.run.smithi031.stdout: "create_time": "2024-08-19T14:08:15.966221+0000", 2024-08-19T14:47:58.114 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-08-19T14:47:58.114 INFO:teuthology.orchestra.run.smithi031.stdout: "flags_names": "hashpspool", 2024-08-19T14:47:58.115 INFO:teuthology.orchestra.run.smithi031.stdout: "type": 1, 2024-08-19T14:47:58.115 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 3, 2024-08-19T14:47:58.115 INFO:teuthology.orchestra.run.smithi031.stdout: "min_size": 2, 2024-08-19T14:47:58.115 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_rule": 0, 2024-08-19T14:47:58.115 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_count": 0, 2024-08-19T14:47:58.115 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_target": 0, 2024-08-19T14:47:58.115 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_barrier": 0, 2024-08-19T14:47:58.115 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-08-19T14:47:58.115 INFO:teuthology.orchestra.run.smithi031.stdout: "object_hash": 2, 2024-08-19T14:47:58.115 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_autoscale_mode": "off", 2024-08-19T14:47:58.115 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num": 31, 2024-08-19T14:47:58.115 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_placement_num": 31, 2024-08-19T14:47:58.115 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_placement_num_target": 31, 2024-08-19T14:47:58.116 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num_target": 31, 2024-08-19T14:47:58.116 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num_pending": 31, 2024-08-19T14:47:58.116 INFO:teuthology.orchestra.run.smithi031.stdout: "last_pg_merge_meta": { 2024-08-19T14:47:58.116 INFO:teuthology.orchestra.run.smithi031.stdout: "source_pgid": "0.0", 2024-08-19T14:47:58.116 INFO:teuthology.orchestra.run.smithi031.stdout: "ready_epoch": 0, 2024-08-19T14:47:58.116 INFO:teuthology.orchestra.run.smithi031.stdout: "last_epoch_started": 0, 2024-08-19T14:47:58.116 INFO:teuthology.orchestra.run.smithi031.stdout: "last_epoch_clean": 0, 2024-08-19T14:47:58.116 INFO:teuthology.orchestra.run.smithi031.stdout: "source_version": "0'0", 2024-08-19T14:47:58.116 INFO:teuthology.orchestra.run.smithi031.stdout: "target_version": "0'0" 2024-08-19T14:47:58.116 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.116 INFO:teuthology.orchestra.run.smithi031.stdout: "last_change": "278", 2024-08-19T14:47:58.116 INFO:teuthology.orchestra.run.smithi031.stdout: "last_force_op_resend": "0", 2024-08-19T14:47:58.116 INFO:teuthology.orchestra.run.smithi031.stdout: "last_force_op_resend_prenautilus": "0", 2024-08-19T14:47:58.117 INFO:teuthology.orchestra.run.smithi031.stdout: "last_force_op_resend_preluminous": "276", 2024-08-19T14:47:58.117 INFO:teuthology.orchestra.run.smithi031.stdout: "auid": 0, 2024-08-19T14:47:58.117 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_mode": "selfmanaged", 2024-08-19T14:47:58.117 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_seq": 0, 2024-08-19T14:47:58.117 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_epoch": 0, 2024-08-19T14:47:58.117 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_snaps": [], 2024-08-19T14:47:58.117 INFO:teuthology.orchestra.run.smithi031.stdout: "removed_snaps": "[]", 2024-08-19T14:47:58.117 INFO:teuthology.orchestra.run.smithi031.stdout: "quota_max_bytes": 0, 2024-08-19T14:47:58.117 INFO:teuthology.orchestra.run.smithi031.stdout: "quota_max_objects": 0, 2024-08-19T14:47:58.117 INFO:teuthology.orchestra.run.smithi031.stdout: "tiers": [], 2024-08-19T14:47:58.117 INFO:teuthology.orchestra.run.smithi031.stdout: "tier_of": -1, 2024-08-19T14:47:58.117 INFO:teuthology.orchestra.run.smithi031.stdout: "read_tier": -1, 2024-08-19T14:47:58.117 INFO:teuthology.orchestra.run.smithi031.stdout: "write_tier": -1, 2024-08-19T14:47:58.117 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_mode": "none", 2024-08-19T14:47:58.118 INFO:teuthology.orchestra.run.smithi031.stdout: "target_max_bytes": 0, 2024-08-19T14:47:58.118 INFO:teuthology.orchestra.run.smithi031.stdout: "target_max_objects": 0, 2024-08-19T14:47:58.118 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-08-19T14:47:58.118 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-08-19T14:47:58.118 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_target_full_ratio_micro": 800000, 2024-08-19T14:47:58.118 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_min_flush_age": 0, 2024-08-19T14:47:58.118 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_min_evict_age": 0, 2024-08-19T14:47:58.118 INFO:teuthology.orchestra.run.smithi031.stdout: "erasure_code_profile": "", 2024-08-19T14:47:58.118 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_params": { 2024-08-19T14:47:58.118 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "none" 2024-08-19T14:47:58.118 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.118 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_period": 0, 2024-08-19T14:47:58.118 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_count": 0, 2024-08-19T14:47:58.119 INFO:teuthology.orchestra.run.smithi031.stdout: "use_gmt_hitset": true, 2024-08-19T14:47:58.119 INFO:teuthology.orchestra.run.smithi031.stdout: "min_read_recency_for_promote": 0, 2024-08-19T14:47:58.119 INFO:teuthology.orchestra.run.smithi031.stdout: "min_write_recency_for_promote": 0, 2024-08-19T14:47:58.119 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_grade_decay_rate": 0, 2024-08-19T14:47:58.119 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_search_last_n": 0, 2024-08-19T14:47:58.119 INFO:teuthology.orchestra.run.smithi031.stdout: "grade_table": [], 2024-08-19T14:47:58.119 INFO:teuthology.orchestra.run.smithi031.stdout: "stripe_width": 0, 2024-08-19T14:47:58.119 INFO:teuthology.orchestra.run.smithi031.stdout: "expected_num_objects": 0, 2024-08-19T14:47:58.119 INFO:teuthology.orchestra.run.smithi031.stdout: "fast_read": false, 2024-08-19T14:47:58.119 INFO:teuthology.orchestra.run.smithi031.stdout: "options": { 2024-08-19T14:47:58.119 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num_max": 32, 2024-08-19T14:47:58.119 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num_min": 1 2024-08-19T14:47:58.119 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.119 INFO:teuthology.orchestra.run.smithi031.stdout: "application_metadata": { 2024-08-19T14:47:58.120 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": {} 2024-08-19T14:47:58.120 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.120 INFO:teuthology.orchestra.run.smithi031.stdout: "read_balance": { 2024-08-19T14:47:58.120 INFO:teuthology.orchestra.run.smithi031.stdout: "score_type": "Fair distribution", 2024-08-19T14:47:58.120 INFO:teuthology.orchestra.run.smithi031.stdout: "score_acting": 2.5799999237060547, 2024-08-19T14:47:58.120 INFO:teuthology.orchestra.run.smithi031.stdout: "score_stable": 2.5799999237060547, 2024-08-19T14:47:58.120 INFO:teuthology.orchestra.run.smithi031.stdout: "optimal_score": 1, 2024-08-19T14:47:58.120 INFO:teuthology.orchestra.run.smithi031.stdout: "raw_score_acting": 2.5799999237060547, 2024-08-19T14:47:58.120 INFO:teuthology.orchestra.run.smithi031.stdout: "raw_score_stable": 2.5799999237060547, 2024-08-19T14:47:58.120 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity_weighted": 0.75614923238754272, 2024-08-19T14:47:58.120 INFO:teuthology.orchestra.run.smithi031.stdout: "average_primary_affinity": 0.9100000262260437, 2024-08-19T14:47:58.120 INFO:teuthology.orchestra.run.smithi031.stdout: "average_primary_affinity_weighted": 0.75999999046325684 2024-08-19T14:47:58.120 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:47:58.120 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.121 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.121 INFO:teuthology.orchestra.run.smithi031.stdout: "pool": 9, 2024-08-19T14:47:58.121 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_name": "unique_pool_7", 2024-08-19T14:47:58.121 INFO:teuthology.orchestra.run.smithi031.stdout: "create_time": "2024-08-19T14:44:13.804952+0000", 2024-08-19T14:47:58.121 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-08-19T14:47:58.121 INFO:teuthology.orchestra.run.smithi031.stdout: "flags_names": "hashpspool", 2024-08-19T14:47:58.121 INFO:teuthology.orchestra.run.smithi031.stdout: "type": 1, 2024-08-19T14:47:58.121 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 3, 2024-08-19T14:47:58.121 INFO:teuthology.orchestra.run.smithi031.stdout: "min_size": 2, 2024-08-19T14:47:58.121 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_rule": 0, 2024-08-19T14:47:58.121 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_count": 0, 2024-08-19T14:47:58.121 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_target": 0, 2024-08-19T14:47:58.121 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_barrier": 0, 2024-08-19T14:47:58.121 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-08-19T14:47:58.122 INFO:teuthology.orchestra.run.smithi031.stdout: "object_hash": 2, 2024-08-19T14:47:58.122 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_autoscale_mode": "off", 2024-08-19T14:47:58.122 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num": 16, 2024-08-19T14:47:58.122 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_placement_num": 16, 2024-08-19T14:47:58.122 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_placement_num_target": 16, 2024-08-19T14:47:58.122 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num_target": 16, 2024-08-19T14:47:58.122 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num_pending": 16, 2024-08-19T14:47:58.122 INFO:teuthology.orchestra.run.smithi031.stdout: "last_pg_merge_meta": { 2024-08-19T14:47:58.122 INFO:teuthology.orchestra.run.smithi031.stdout: "source_pgid": "0.0", 2024-08-19T14:47:58.122 INFO:teuthology.orchestra.run.smithi031.stdout: "ready_epoch": 0, 2024-08-19T14:47:58.122 INFO:teuthology.orchestra.run.smithi031.stdout: "last_epoch_started": 0, 2024-08-19T14:47:58.122 INFO:teuthology.orchestra.run.smithi031.stdout: "last_epoch_clean": 0, 2024-08-19T14:47:58.122 INFO:teuthology.orchestra.run.smithi031.stdout: "source_version": "0'0", 2024-08-19T14:47:58.123 INFO:teuthology.orchestra.run.smithi031.stdout: "target_version": "0'0" 2024-08-19T14:47:58.123 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.123 INFO:teuthology.orchestra.run.smithi031.stdout: "last_change": "829", 2024-08-19T14:47:58.123 INFO:teuthology.orchestra.run.smithi031.stdout: "last_force_op_resend": "0", 2024-08-19T14:47:58.123 INFO:teuthology.orchestra.run.smithi031.stdout: "last_force_op_resend_prenautilus": "0", 2024-08-19T14:47:58.123 INFO:teuthology.orchestra.run.smithi031.stdout: "last_force_op_resend_preluminous": "0", 2024-08-19T14:47:58.123 INFO:teuthology.orchestra.run.smithi031.stdout: "auid": 0, 2024-08-19T14:47:58.123 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_mode": "selfmanaged", 2024-08-19T14:47:58.123 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_seq": 0, 2024-08-19T14:47:58.123 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_epoch": 0, 2024-08-19T14:47:58.123 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_snaps": [], 2024-08-19T14:47:58.123 INFO:teuthology.orchestra.run.smithi031.stdout: "removed_snaps": "[]", 2024-08-19T14:47:58.123 INFO:teuthology.orchestra.run.smithi031.stdout: "quota_max_bytes": 0, 2024-08-19T14:47:58.123 INFO:teuthology.orchestra.run.smithi031.stdout: "quota_max_objects": 0, 2024-08-19T14:47:58.124 INFO:teuthology.orchestra.run.smithi031.stdout: "tiers": [], 2024-08-19T14:47:58.124 INFO:teuthology.orchestra.run.smithi031.stdout: "tier_of": -1, 2024-08-19T14:47:58.124 INFO:teuthology.orchestra.run.smithi031.stdout: "read_tier": -1, 2024-08-19T14:47:58.124 INFO:teuthology.orchestra.run.smithi031.stdout: "write_tier": -1, 2024-08-19T14:47:58.124 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_mode": "none", 2024-08-19T14:47:58.124 INFO:teuthology.orchestra.run.smithi031.stdout: "target_max_bytes": 0, 2024-08-19T14:47:58.124 INFO:teuthology.orchestra.run.smithi031.stdout: "target_max_objects": 0, 2024-08-19T14:47:58.124 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-08-19T14:47:58.124 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-08-19T14:47:58.124 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_target_full_ratio_micro": 800000, 2024-08-19T14:47:58.124 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_min_flush_age": 0, 2024-08-19T14:47:58.124 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_min_evict_age": 0, 2024-08-19T14:47:58.124 INFO:teuthology.orchestra.run.smithi031.stdout: "erasure_code_profile": "", 2024-08-19T14:47:58.125 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_params": { 2024-08-19T14:47:58.125 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "none" 2024-08-19T14:47:58.125 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.125 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_period": 0, 2024-08-19T14:47:58.125 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_count": 0, 2024-08-19T14:47:58.125 INFO:teuthology.orchestra.run.smithi031.stdout: "use_gmt_hitset": true, 2024-08-19T14:47:58.125 INFO:teuthology.orchestra.run.smithi031.stdout: "min_read_recency_for_promote": 0, 2024-08-19T14:47:58.125 INFO:teuthology.orchestra.run.smithi031.stdout: "min_write_recency_for_promote": 0, 2024-08-19T14:47:58.125 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_grade_decay_rate": 0, 2024-08-19T14:47:58.125 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_search_last_n": 0, 2024-08-19T14:47:58.125 INFO:teuthology.orchestra.run.smithi031.stdout: "grade_table": [], 2024-08-19T14:47:58.125 INFO:teuthology.orchestra.run.smithi031.stdout: "stripe_width": 0, 2024-08-19T14:47:58.125 INFO:teuthology.orchestra.run.smithi031.stdout: "expected_num_objects": 0, 2024-08-19T14:47:58.126 INFO:teuthology.orchestra.run.smithi031.stdout: "fast_read": false, 2024-08-19T14:47:58.126 INFO:teuthology.orchestra.run.smithi031.stdout: "options": {}, 2024-08-19T14:47:58.126 INFO:teuthology.orchestra.run.smithi031.stdout: "application_metadata": { 2024-08-19T14:47:58.126 INFO:teuthology.orchestra.run.smithi031.stdout: "rados": {} 2024-08-19T14:47:58.126 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.126 INFO:teuthology.orchestra.run.smithi031.stdout: "read_balance": { 2024-08-19T14:47:58.126 INFO:teuthology.orchestra.run.smithi031.stdout: "score_type": "Fair distribution", 2024-08-19T14:47:58.126 INFO:teuthology.orchestra.run.smithi031.stdout: "score_acting": 3.119999885559082, 2024-08-19T14:47:58.126 INFO:teuthology.orchestra.run.smithi031.stdout: "score_stable": 2.5, 2024-08-19T14:47:58.126 INFO:teuthology.orchestra.run.smithi031.stdout: "optimal_score": 0.89999997615814209, 2024-08-19T14:47:58.126 INFO:teuthology.orchestra.run.smithi031.stdout: "raw_score_acting": 2.809999942779541, 2024-08-19T14:47:58.126 INFO:teuthology.orchestra.run.smithi031.stdout: "raw_score_stable": 2.25, 2024-08-19T14:47:58.126 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity_weighted": 0.75614923238754272, 2024-08-19T14:47:58.126 INFO:teuthology.orchestra.run.smithi031.stdout: "average_primary_affinity": 0.9100000262260437, 2024-08-19T14:47:58.127 INFO:teuthology.orchestra.run.smithi031.stdout: "average_primary_affinity_weighted": 0.75999999046325684 2024-08-19T14:47:58.127 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:47:58.127 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:47:58.127 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-08-19T14:47:58.127 INFO:teuthology.orchestra.run.smithi031.stdout: "osds": [ 2024-08-19T14:47:58.127 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.127 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 0, 2024-08-19T14:47:58.127 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "d7a0fff3-23c6-4ee3-a275-a844862aabfb", 2024-08-19T14:47:58.127 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-08-19T14:47:58.127 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 0, 2024-08-19T14:47:58.127 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 0, 2024-08-19T14:47:58.127 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-08-19T14:47:58.127 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-08-19T14:47:58.127 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-08-19T14:47:58.128 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 8, 2024-08-19T14:47:58.128 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 820, 2024-08-19T14:47:58.128 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-08-19T14:47:58.128 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-08-19T14:47:58.128 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-08-19T14:47:58.128 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:47:58.128 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.128 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:47:58.128 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6802", 2024-08-19T14:47:58.128 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1108945097 2024-08-19T14:47:58.128 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.128 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.128 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:47:58.128 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6803", 2024-08-19T14:47:58.129 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1108945097 2024-08-19T14:47:58.129 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:47:58.129 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:47:58.129 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.129 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-08-19T14:47:58.129 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:47:58.129 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.130 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:47:58.130 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6804", 2024-08-19T14:47:58.130 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1108945097 2024-08-19T14:47:58.130 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.130 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.130 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:47:58.130 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6805", 2024-08-19T14:47:58.130 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1108945097 2024-08-19T14:47:58.130 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:47:58.130 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:47:58.130 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.131 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-08-19T14:47:58.131 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:47:58.131 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.131 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:47:58.131 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6808", 2024-08-19T14:47:58.131 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1108945097 2024-08-19T14:47:58.131 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.131 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.131 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:47:58.131 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6809", 2024-08-19T14:47:58.131 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1108945097 2024-08-19T14:47:58.131 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:47:58.131 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:47:58.132 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.132 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-08-19T14:47:58.132 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:47:58.132 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.132 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:47:58.132 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6806", 2024-08-19T14:47:58.132 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1108945097 2024-08-19T14:47:58.132 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.132 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.132 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:47:58.132 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6807", 2024-08-19T14:47:58.132 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1108945097 2024-08-19T14:47:58.132 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:47:58.132 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:47:58.133 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.133 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.31:6803/1108945097", 2024-08-19T14:47:58.133 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.31:6805/1108945097", 2024-08-19T14:47:58.133 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.31:6809/1108945097", 2024-08-19T14:47:58.133 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.31:6807/1108945097", 2024-08-19T14:47:58.133 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-08-19T14:47:58.133 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-08-19T14:47:58.133 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-08-19T14:47:58.133 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:47:58.133 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.133 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.133 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 1, 2024-08-19T14:47:58.133 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "145d5bb4-74dd-440f-bcc9-7e9ebda9c5d6", 2024-08-19T14:47:58.134 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-08-19T14:47:58.134 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-08-19T14:47:58.134 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-08-19T14:47:58.134 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-08-19T14:47:58.134 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-08-19T14:47:58.134 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-08-19T14:47:58.134 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 13, 2024-08-19T14:47:58.134 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 921, 2024-08-19T14:47:58.134 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-08-19T14:47:58.134 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-08-19T14:47:58.134 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-08-19T14:47:58.134 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:47:58.134 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.135 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:47:58.135 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6810", 2024-08-19T14:47:58.135 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3236186806 2024-08-19T14:47:58.135 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.135 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.135 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:47:58.135 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6811", 2024-08-19T14:47:58.135 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3236186806 2024-08-19T14:47:58.135 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:47:58.135 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:47:58.135 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.135 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-08-19T14:47:58.135 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:47:58.136 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.136 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:47:58.136 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6812", 2024-08-19T14:47:58.136 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3236186806 2024-08-19T14:47:58.136 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.136 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.136 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:47:58.136 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6813", 2024-08-19T14:47:58.136 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3236186806 2024-08-19T14:47:58.136 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:47:58.136 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:47:58.136 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.137 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-08-19T14:47:58.137 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:47:58.137 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.137 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:47:58.137 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6816", 2024-08-19T14:47:58.137 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3236186806 2024-08-19T14:47:58.137 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.137 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.137 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:47:58.137 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6817", 2024-08-19T14:47:58.137 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3236186806 2024-08-19T14:47:58.137 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:47:58.137 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:47:58.138 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.138 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-08-19T14:47:58.138 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:47:58.138 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.138 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:47:58.138 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6814", 2024-08-19T14:47:58.138 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3236186806 2024-08-19T14:47:58.138 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.138 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.138 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:47:58.138 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6815", 2024-08-19T14:47:58.138 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3236186806 2024-08-19T14:47:58.138 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:47:58.139 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:47:58.139 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.139 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.31:6811/3236186806", 2024-08-19T14:47:58.139 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.31:6813/3236186806", 2024-08-19T14:47:58.139 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.31:6817/3236186806", 2024-08-19T14:47:58.139 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.31:6815/3236186806", 2024-08-19T14:47:58.139 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-08-19T14:47:58.139 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-08-19T14:47:58.139 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-08-19T14:47:58.139 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:47:58.139 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.139 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.139 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 2, 2024-08-19T14:47:58.139 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "376c18d0-4768-46a0-bb60-e79c88ca909c", 2024-08-19T14:47:58.140 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-08-19T14:47:58.140 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-08-19T14:47:58.140 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-08-19T14:47:58.140 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 0.07379150390625, 2024-08-19T14:47:58.140 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 18, 2024-08-19T14:47:58.140 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 564, 2024-08-19T14:47:58.140 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 634, 2024-08-19T14:47:58.140 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 916, 2024-08-19T14:47:58.140 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 565, 2024-08-19T14:47:58.140 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-08-19T14:47:58.140 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-08-19T14:47:58.140 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:47:58.140 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.141 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:47:58.141 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6818", 2024-08-19T14:47:58.141 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 656997856 2024-08-19T14:47:58.141 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.141 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.141 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:47:58.141 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6819", 2024-08-19T14:47:58.141 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 656997856 2024-08-19T14:47:58.141 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:47:58.141 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:47:58.141 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.141 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-08-19T14:47:58.141 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:47:58.141 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.142 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:47:58.142 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6820", 2024-08-19T14:47:58.142 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 656997856 2024-08-19T14:47:58.142 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.142 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.142 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:47:58.142 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6821", 2024-08-19T14:47:58.142 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 656997856 2024-08-19T14:47:58.142 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:47:58.142 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:47:58.142 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.142 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-08-19T14:47:58.142 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:47:58.143 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.143 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:47:58.143 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6824", 2024-08-19T14:47:58.143 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 656997856 2024-08-19T14:47:58.143 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.143 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.143 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:47:58.143 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6825", 2024-08-19T14:47:58.143 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 656997856 2024-08-19T14:47:58.143 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:47:58.143 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:47:58.143 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.143 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-08-19T14:47:58.143 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:47:58.144 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.144 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:47:58.144 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6822", 2024-08-19T14:47:58.144 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 656997856 2024-08-19T14:47:58.144 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.144 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.144 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:47:58.144 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6823", 2024-08-19T14:47:58.144 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 656997856 2024-08-19T14:47:58.144 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:47:58.144 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:47:58.144 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.144 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.31:6819/656997856", 2024-08-19T14:47:58.144 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.31:6821/656997856", 2024-08-19T14:47:58.145 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.31:6825/656997856", 2024-08-19T14:47:58.145 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.31:6823/656997856", 2024-08-19T14:47:58.145 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-08-19T14:47:58.145 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-08-19T14:47:58.145 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-08-19T14:47:58.145 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:47:58.145 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.145 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.145 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 3, 2024-08-19T14:47:58.145 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "a787ea9f-b9ba-450f-b388-614cfba90634", 2024-08-19T14:47:58.145 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-08-19T14:47:58.145 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-08-19T14:47:58.145 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-08-19T14:47:58.146 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-08-19T14:47:58.146 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-08-19T14:47:58.146 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-08-19T14:47:58.146 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 26, 2024-08-19T14:47:58.146 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 922, 2024-08-19T14:47:58.146 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-08-19T14:47:58.146 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-08-19T14:47:58.146 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-08-19T14:47:58.146 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:47:58.146 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.146 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:47:58.146 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6826", 2024-08-19T14:47:58.146 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2288082716 2024-08-19T14:47:58.146 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.147 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.147 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:47:58.147 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6827", 2024-08-19T14:47:58.147 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2288082716 2024-08-19T14:47:58.147 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:47:58.147 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:47:58.147 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.147 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-08-19T14:47:58.147 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:47:58.147 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.147 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:47:58.147 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6828", 2024-08-19T14:47:58.147 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2288082716 2024-08-19T14:47:58.147 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.148 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.148 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:47:58.148 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6829", 2024-08-19T14:47:58.148 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2288082716 2024-08-19T14:47:58.148 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:47:58.148 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:47:58.148 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.148 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-08-19T14:47:58.148 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:47:58.148 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.148 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:47:58.148 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6832", 2024-08-19T14:47:58.148 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2288082716 2024-08-19T14:47:58.149 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.149 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.149 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:47:58.149 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6833", 2024-08-19T14:47:58.149 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2288082716 2024-08-19T14:47:58.149 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:47:58.149 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:47:58.149 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.149 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-08-19T14:47:58.149 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:47:58.149 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.149 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:47:58.149 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6830", 2024-08-19T14:47:58.150 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2288082716 2024-08-19T14:47:58.150 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.150 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.150 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:47:58.150 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6831", 2024-08-19T14:47:58.150 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2288082716 2024-08-19T14:47:58.150 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:47:58.150 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:47:58.150 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.150 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.31:6827/2288082716", 2024-08-19T14:47:58.150 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.31:6829/2288082716", 2024-08-19T14:47:58.150 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.31:6833/2288082716", 2024-08-19T14:47:58.150 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.31:6831/2288082716", 2024-08-19T14:47:58.151 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-08-19T14:47:58.151 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-08-19T14:47:58.151 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-08-19T14:47:58.151 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:47:58.151 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.151 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.151 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 4, 2024-08-19T14:47:58.151 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "7859751a-12f5-4a57-a332-435360479d53", 2024-08-19T14:47:58.151 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-08-19T14:47:58.151 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-08-19T14:47:58.151 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-08-19T14:47:58.151 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-08-19T14:47:58.151 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-08-19T14:47:58.151 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-08-19T14:47:58.152 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 31, 2024-08-19T14:47:58.152 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 918, 2024-08-19T14:47:58.152 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-08-19T14:47:58.152 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-08-19T14:47:58.152 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-08-19T14:47:58.152 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:47:58.152 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.152 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:47:58.152 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6800", 2024-08-19T14:47:58.152 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3934739984 2024-08-19T14:47:58.152 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.152 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.152 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:47:58.152 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6801", 2024-08-19T14:47:58.153 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3934739984 2024-08-19T14:47:58.153 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:47:58.153 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:47:58.153 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.153 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-08-19T14:47:58.153 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:47:58.153 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.153 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:47:58.153 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6802", 2024-08-19T14:47:58.153 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3934739984 2024-08-19T14:47:58.153 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.153 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.153 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:47:58.153 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6803", 2024-08-19T14:47:58.154 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3934739984 2024-08-19T14:47:58.154 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:47:58.154 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:47:58.154 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.154 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-08-19T14:47:58.154 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:47:58.154 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.154 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:47:58.154 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6806", 2024-08-19T14:47:58.154 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3934739984 2024-08-19T14:47:58.154 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.154 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.154 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:47:58.155 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6807", 2024-08-19T14:47:58.155 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3934739984 2024-08-19T14:47:58.155 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:47:58.155 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:47:58.155 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.155 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-08-19T14:47:58.155 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:47:58.155 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.155 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:47:58.155 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6804", 2024-08-19T14:47:58.155 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3934739984 2024-08-19T14:47:58.155 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.155 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.156 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:47:58.156 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6805", 2024-08-19T14:47:58.156 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3934739984 2024-08-19T14:47:58.156 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:47:58.156 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:47:58.156 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.156 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.32:6801/3934739984", 2024-08-19T14:47:58.156 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.32:6803/3934739984", 2024-08-19T14:47:58.156 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.32:6807/3934739984", 2024-08-19T14:47:58.156 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.32:6805/3934739984", 2024-08-19T14:47:58.156 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-08-19T14:47:58.156 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-08-19T14:47:58.156 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-08-19T14:47:58.156 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:47:58.157 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.157 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.157 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 5, 2024-08-19T14:47:58.157 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "017f3eef-ac60-4b82-8350-919017cb4ccd", 2024-08-19T14:47:58.157 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-08-19T14:47:58.157 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 0, 2024-08-19T14:47:58.157 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 0, 2024-08-19T14:47:58.157 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 0, 2024-08-19T14:47:58.157 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-08-19T14:47:58.157 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-08-19T14:47:58.157 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 40, 2024-08-19T14:47:58.157 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 817, 2024-08-19T14:47:58.157 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-08-19T14:47:58.157 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-08-19T14:47:58.158 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-08-19T14:47:58.158 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:47:58.158 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.158 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:47:58.158 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6808", 2024-08-19T14:47:58.158 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 526388985 2024-08-19T14:47:58.158 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.158 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.158 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:47:58.158 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6809", 2024-08-19T14:47:58.158 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 526388985 2024-08-19T14:47:58.158 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:47:58.158 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:47:58.158 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.159 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-08-19T14:47:58.159 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:47:58.159 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.159 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:47:58.159 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6810", 2024-08-19T14:47:58.159 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 526388985 2024-08-19T14:47:58.159 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.159 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.159 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:47:58.159 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6811", 2024-08-19T14:47:58.159 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 526388985 2024-08-19T14:47:58.159 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:47:58.159 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:47:58.159 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.160 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-08-19T14:47:58.160 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:47:58.160 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.160 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:47:58.160 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6814", 2024-08-19T14:47:58.160 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 526388985 2024-08-19T14:47:58.160 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.160 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.160 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:47:58.160 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6815", 2024-08-19T14:47:58.160 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 526388985 2024-08-19T14:47:58.160 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:47:58.160 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:47:58.160 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.161 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-08-19T14:47:58.161 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:47:58.161 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.161 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:47:58.161 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6812", 2024-08-19T14:47:58.161 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 526388985 2024-08-19T14:47:58.161 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.161 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.161 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:47:58.161 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6813", 2024-08-19T14:47:58.161 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 526388985 2024-08-19T14:47:58.161 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:47:58.161 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:47:58.161 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.162 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.32:6809/526388985", 2024-08-19T14:47:58.162 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.32:6811/526388985", 2024-08-19T14:47:58.162 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.32:6815/526388985", 2024-08-19T14:47:58.162 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.32:6813/526388985", 2024-08-19T14:47:58.162 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-08-19T14:47:58.162 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-08-19T14:47:58.162 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-08-19T14:47:58.162 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:47:58.162 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.162 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.162 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 6, 2024-08-19T14:47:58.162 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "44aa8229-0268-49d7-bd57-3bf1f384e5e6", 2024-08-19T14:47:58.162 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-08-19T14:47:58.163 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-08-19T14:47:58.163 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-08-19T14:47:58.163 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-08-19T14:47:58.163 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-08-19T14:47:58.163 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-08-19T14:47:58.163 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 48, 2024-08-19T14:47:58.163 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 919, 2024-08-19T14:47:58.163 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-08-19T14:47:58.163 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-08-19T14:47:58.163 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-08-19T14:47:58.163 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:47:58.163 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.163 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:47:58.164 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6816", 2024-08-19T14:47:58.164 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1131035844 2024-08-19T14:47:58.164 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.164 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.164 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:47:58.164 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6817", 2024-08-19T14:47:58.164 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1131035844 2024-08-19T14:47:58.164 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:47:58.164 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:47:58.164 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.164 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-08-19T14:47:58.164 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:47:58.164 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.164 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:47:58.165 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6818", 2024-08-19T14:47:58.165 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1131035844 2024-08-19T14:47:58.165 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.165 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.165 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:47:58.165 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6819", 2024-08-19T14:47:58.165 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1131035844 2024-08-19T14:47:58.165 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:47:58.165 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:47:58.165 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.165 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-08-19T14:47:58.165 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:47:58.165 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.166 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:47:58.166 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6822", 2024-08-19T14:47:58.166 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1131035844 2024-08-19T14:47:58.166 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.166 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.166 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:47:58.166 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6823", 2024-08-19T14:47:58.166 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1131035844 2024-08-19T14:47:58.166 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:47:58.166 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:47:58.166 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.166 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-08-19T14:47:58.167 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:47:58.167 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.167 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:47:58.167 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6820", 2024-08-19T14:47:58.167 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1131035844 2024-08-19T14:47:58.167 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.167 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.167 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:47:58.168 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6821", 2024-08-19T14:47:58.168 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1131035844 2024-08-19T14:47:58.168 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:47:58.168 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:47:58.168 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.168 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.32:6817/1131035844", 2024-08-19T14:47:58.168 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.32:6819/1131035844", 2024-08-19T14:47:58.169 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.32:6823/1131035844", 2024-08-19T14:47:58.169 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.32:6821/1131035844", 2024-08-19T14:47:58.169 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-08-19T14:47:58.169 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-08-19T14:47:58.169 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-08-19T14:47:58.169 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:47:58.169 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.169 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.169 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 7, 2024-08-19T14:47:58.169 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "263433e5-70b6-4192-86ff-dc2ad95927eb", 2024-08-19T14:47:58.169 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-08-19T14:47:58.169 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-08-19T14:47:58.170 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-08-19T14:47:58.170 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-08-19T14:47:58.170 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-08-19T14:47:58.170 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-08-19T14:47:58.170 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 57, 2024-08-19T14:47:58.170 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 907, 2024-08-19T14:47:58.170 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-08-19T14:47:58.170 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-08-19T14:47:58.170 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-08-19T14:47:58.170 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:47:58.170 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.170 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:47:58.170 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6824", 2024-08-19T14:47:58.171 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2014939947 2024-08-19T14:47:58.171 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.171 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.171 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:47:58.171 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6825", 2024-08-19T14:47:58.171 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2014939947 2024-08-19T14:47:58.171 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:47:58.171 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:47:58.171 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.171 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-08-19T14:47:58.171 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:47:58.171 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.171 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:47:58.171 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6826", 2024-08-19T14:47:58.172 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2014939947 2024-08-19T14:47:58.172 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.172 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.172 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:47:58.172 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6827", 2024-08-19T14:47:58.172 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2014939947 2024-08-19T14:47:58.172 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:47:58.172 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:47:58.172 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.172 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-08-19T14:47:58.172 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:47:58.172 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.172 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:47:58.172 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6830", 2024-08-19T14:47:58.173 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2014939947 2024-08-19T14:47:58.173 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.173 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.173 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:47:58.173 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6831", 2024-08-19T14:47:58.173 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2014939947 2024-08-19T14:47:58.173 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:47:58.173 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:47:58.173 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.173 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-08-19T14:47:58.173 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:47:58.173 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.173 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:47:58.173 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6828", 2024-08-19T14:47:58.174 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2014939947 2024-08-19T14:47:58.174 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.174 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.174 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:47:58.174 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.32:6829", 2024-08-19T14:47:58.174 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2014939947 2024-08-19T14:47:58.174 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:47:58.174 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:47:58.174 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.174 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.32:6825/2014939947", 2024-08-19T14:47:58.174 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.32:6827/2014939947", 2024-08-19T14:47:58.174 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.32:6831/2014939947", 2024-08-19T14:47:58.174 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.32:6829/2014939947", 2024-08-19T14:47:58.175 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-08-19T14:47:58.175 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-08-19T14:47:58.175 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-08-19T14:47:58.175 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:47:58.175 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.175 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.175 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 8, 2024-08-19T14:47:58.175 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "2bbcad36-0c74-4528-9fb9-a8812afa1662", 2024-08-19T14:47:58.175 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-08-19T14:47:58.175 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-08-19T14:47:58.175 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-08-19T14:47:58.175 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-08-19T14:47:58.175 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-08-19T14:47:58.175 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-08-19T14:47:58.176 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 65, 2024-08-19T14:47:58.176 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 917, 2024-08-19T14:47:58.176 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-08-19T14:47:58.176 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-08-19T14:47:58.176 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-08-19T14:47:58.176 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:47:58.176 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.176 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:47:58.176 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6800", 2024-08-19T14:47:58.176 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1445140892 2024-08-19T14:47:58.176 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.176 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.176 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:47:58.176 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6801", 2024-08-19T14:47:58.177 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1445140892 2024-08-19T14:47:58.177 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:47:58.177 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:47:58.177 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.177 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-08-19T14:47:58.177 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:47:58.177 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.177 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:47:58.177 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6802", 2024-08-19T14:47:58.177 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1445140892 2024-08-19T14:47:58.177 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.177 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.177 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:47:58.177 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6803", 2024-08-19T14:47:58.178 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1445140892 2024-08-19T14:47:58.178 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:47:58.178 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:47:58.178 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.178 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-08-19T14:47:58.178 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:47:58.178 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.178 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:47:58.178 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6806", 2024-08-19T14:47:58.178 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1445140892 2024-08-19T14:47:58.178 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.178 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.178 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:47:58.179 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6807", 2024-08-19T14:47:58.179 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1445140892 2024-08-19T14:47:58.179 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:47:58.179 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:47:58.179 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.179 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-08-19T14:47:58.179 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:47:58.179 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.179 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:47:58.179 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6804", 2024-08-19T14:47:58.179 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1445140892 2024-08-19T14:47:58.179 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.179 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.180 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:47:58.180 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6805", 2024-08-19T14:47:58.180 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1445140892 2024-08-19T14:47:58.180 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:47:58.180 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:47:58.180 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.180 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.45:6801/1445140892", 2024-08-19T14:47:58.180 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.45:6803/1445140892", 2024-08-19T14:47:58.180 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.45:6807/1445140892", 2024-08-19T14:47:58.180 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.45:6805/1445140892", 2024-08-19T14:47:58.180 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-08-19T14:47:58.180 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-08-19T14:47:58.180 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-08-19T14:47:58.180 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:47:58.181 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.181 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.181 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 9, 2024-08-19T14:47:58.181 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "c8ee4277-3bf9-46bf-9cc0-152c1c4b9508", 2024-08-19T14:47:58.181 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-08-19T14:47:58.181 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-08-19T14:47:58.181 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-08-19T14:47:58.181 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 0, 2024-08-19T14:47:58.181 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-08-19T14:47:58.181 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-08-19T14:47:58.181 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 74, 2024-08-19T14:47:58.181 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 793, 2024-08-19T14:47:58.181 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-08-19T14:47:58.181 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-08-19T14:47:58.182 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-08-19T14:47:58.182 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:47:58.182 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.182 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:47:58.182 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6808", 2024-08-19T14:47:58.182 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1213787169 2024-08-19T14:47:58.182 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.182 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.182 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:47:58.182 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6809", 2024-08-19T14:47:58.182 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1213787169 2024-08-19T14:47:58.182 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:47:58.182 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:47:58.182 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.183 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-08-19T14:47:58.183 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:47:58.183 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.183 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:47:58.183 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6810", 2024-08-19T14:47:58.183 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1213787169 2024-08-19T14:47:58.183 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.183 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.183 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:47:58.183 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6811", 2024-08-19T14:47:58.183 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1213787169 2024-08-19T14:47:58.183 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:47:58.183 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:47:58.184 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.184 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-08-19T14:47:58.184 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:47:58.184 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.184 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:47:58.184 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6814", 2024-08-19T14:47:58.184 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1213787169 2024-08-19T14:47:58.184 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.184 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.184 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:47:58.184 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6815", 2024-08-19T14:47:58.184 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1213787169 2024-08-19T14:47:58.184 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:47:58.184 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:47:58.185 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.185 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-08-19T14:47:58.185 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:47:58.185 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.185 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:47:58.185 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6812", 2024-08-19T14:47:58.185 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1213787169 2024-08-19T14:47:58.185 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.185 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.185 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:47:58.185 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6813", 2024-08-19T14:47:58.185 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1213787169 2024-08-19T14:47:58.185 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:47:58.186 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:47:58.186 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.186 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.45:6809/1213787169", 2024-08-19T14:47:58.187 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.45:6811/1213787169", 2024-08-19T14:47:58.187 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.45:6815/1213787169", 2024-08-19T14:47:58.187 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.45:6813/1213787169", 2024-08-19T14:47:58.187 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-08-19T14:47:58.187 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-08-19T14:47:58.187 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-08-19T14:47:58.187 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:47:58.187 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.187 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.187 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 10, 2024-08-19T14:47:58.187 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "29031395-5f5c-40f8-ac90-9f6ac48e5c9f", 2024-08-19T14:47:58.188 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-08-19T14:47:58.188 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-08-19T14:47:58.188 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-08-19T14:47:58.188 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-08-19T14:47:58.188 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 80, 2024-08-19T14:47:58.188 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 529, 2024-08-19T14:47:58.188 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 539, 2024-08-19T14:47:58.188 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 921, 2024-08-19T14:47:58.188 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 530, 2024-08-19T14:47:58.188 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-08-19T14:47:58.188 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-08-19T14:47:58.188 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:47:58.189 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.189 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:47:58.189 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6816", 2024-08-19T14:47:58.189 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1069743893 2024-08-19T14:47:58.189 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.189 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.189 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:47:58.189 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6817", 2024-08-19T14:47:58.189 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1069743893 2024-08-19T14:47:58.189 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:47:58.189 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:47:58.189 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.189 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-08-19T14:47:58.189 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:47:58.190 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.190 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:47:58.190 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6818", 2024-08-19T14:47:58.190 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1069743893 2024-08-19T14:47:58.190 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.190 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.190 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:47:58.190 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6819", 2024-08-19T14:47:58.190 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1069743893 2024-08-19T14:47:58.190 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:47:58.190 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:47:58.190 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.190 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-08-19T14:47:58.191 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:47:58.191 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.191 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:47:58.191 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6822", 2024-08-19T14:47:58.191 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1069743893 2024-08-19T14:47:58.191 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.191 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.191 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:47:58.191 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6823", 2024-08-19T14:47:58.191 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1069743893 2024-08-19T14:47:58.191 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:47:58.191 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:47:58.191 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.191 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-08-19T14:47:58.192 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:47:58.192 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.192 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:47:58.192 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6820", 2024-08-19T14:47:58.192 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1069743893 2024-08-19T14:47:58.192 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.192 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.192 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:47:58.192 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6821", 2024-08-19T14:47:58.192 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1069743893 2024-08-19T14:47:58.192 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:47:58.192 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:47:58.192 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.192 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.45:6817/1069743893", 2024-08-19T14:47:58.193 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.45:6819/1069743893", 2024-08-19T14:47:58.193 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.45:6823/1069743893", 2024-08-19T14:47:58.193 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.45:6821/1069743893", 2024-08-19T14:47:58.193 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-08-19T14:47:58.193 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-08-19T14:47:58.193 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-08-19T14:47:58.193 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:47:58.193 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.193 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.193 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 11, 2024-08-19T14:47:58.193 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "5ef0db4b-0fef-4ea0-a19b-d3677e082c29", 2024-08-19T14:47:58.193 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-08-19T14:47:58.193 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-08-19T14:47:58.194 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-08-19T14:47:58.194 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-08-19T14:47:58.194 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-08-19T14:47:58.194 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-08-19T14:47:58.194 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 86, 2024-08-19T14:47:58.194 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 910, 2024-08-19T14:47:58.194 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-08-19T14:47:58.195 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-08-19T14:47:58.195 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-08-19T14:47:58.195 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:47:58.195 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.195 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:47:58.195 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6824", 2024-08-19T14:47:58.195 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2438396298 2024-08-19T14:47:58.195 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.195 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.195 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:47:58.195 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6825", 2024-08-19T14:47:58.195 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2438396298 2024-08-19T14:47:58.195 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:47:58.196 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:47:58.196 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.196 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-08-19T14:47:58.196 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:47:58.196 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.196 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:47:58.196 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6826", 2024-08-19T14:47:58.196 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2438396298 2024-08-19T14:47:58.196 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.196 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.196 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:47:58.196 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6827", 2024-08-19T14:47:58.196 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2438396298 2024-08-19T14:47:58.196 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:47:58.197 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:47:58.197 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.197 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-08-19T14:47:58.197 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:47:58.197 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.197 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:47:58.197 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6830", 2024-08-19T14:47:58.197 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2438396298 2024-08-19T14:47:58.197 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.197 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.197 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:47:58.197 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6831", 2024-08-19T14:47:58.197 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2438396298 2024-08-19T14:47:58.198 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:47:58.198 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:47:58.198 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.198 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-08-19T14:47:58.198 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-08-19T14:47:58.198 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.198 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-08-19T14:47:58.198 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6828", 2024-08-19T14:47:58.198 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2438396298 2024-08-19T14:47:58.198 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.199 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.199 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-08-19T14:47:58.199 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.45:6829", 2024-08-19T14:47:58.199 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2438396298 2024-08-19T14:47:58.199 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:47:58.199 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:47:58.199 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.199 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.45:6825/2438396298", 2024-08-19T14:47:58.199 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.45:6827/2438396298", 2024-08-19T14:47:58.199 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.45:6831/2438396298", 2024-08-19T14:47:58.199 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.45:6829/2438396298", 2024-08-19T14:47:58.199 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-08-19T14:47:58.199 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-08-19T14:47:58.200 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-08-19T14:47:58.200 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:47:58.200 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:47:58.200 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-08-19T14:47:58.200 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_xinfo": [ 2024-08-19T14:47:58.200 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.200 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 0, 2024-08-19T14:47:58.200 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-08-19T14:47:58.200 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-08-19T14:47:58.200 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-08-19T14:47:58.200 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-08-19T14:47:58.200 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 65536, 2024-08-19T14:47:58.200 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-08-19T14:47:49.205675+0000", 2024-08-19T14:47:58.200 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-08-19T14:47:58.201 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.201 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.201 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 1, 2024-08-19T14:47:58.201 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-08-19T14:47:58.201 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-08-19T14:47:58.201 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-08-19T14:47:58.201 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-08-19T14:47:58.201 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-08-19T14:47:58.201 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-08-19T14:47:53.610800+0000", 2024-08-19T14:47:58.201 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-08-19T14:47:58.201 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.201 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.201 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 2, 2024-08-19T14:47:58.202 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-08-19T14:47:58.202 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-08-19T14:47:58.202 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-08-19T14:47:58.202 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-08-19T14:47:58.202 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-08-19T14:47:58.202 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-08-19T14:47:47.260684+0000", 2024-08-19T14:47:58.202 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 564 2024-08-19T14:47:58.202 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.202 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.202 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 3, 2024-08-19T14:47:58.202 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-08-19T14:47:58.202 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-08-19T14:47:58.202 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-08-19T14:47:58.202 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-08-19T14:47:58.203 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-08-19T14:47:58.203 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-08-19T14:46:49.576968+0000", 2024-08-19T14:47:58.203 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-08-19T14:47:58.203 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.203 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.203 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 4, 2024-08-19T14:47:58.203 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-08-19T14:47:58.203 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-08-19T14:47:58.203 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-08-19T14:47:58.203 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-08-19T14:47:58.203 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-08-19T14:47:58.203 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-08-19T14:47:33.446105+0000", 2024-08-19T14:47:58.203 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-08-19T14:47:58.204 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.204 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.204 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 5, 2024-08-19T14:47:58.204 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-08-19T14:47:58.204 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-08-19T14:47:58.204 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-08-19T14:47:58.204 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-08-19T14:47:58.204 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 65536, 2024-08-19T14:47:58.204 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-08-19T14:47:03.078224+0000", 2024-08-19T14:47:58.204 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-08-19T14:47:58.204 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.204 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.204 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 6, 2024-08-19T14:47:58.204 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-08-19T14:47:58.204 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-08-19T14:47:58.205 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-08-19T14:47:58.205 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-08-19T14:47:58.205 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-08-19T14:47:58.205 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-08-19T14:47:53.672486+0000", 2024-08-19T14:47:58.205 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-08-19T14:47:58.205 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.205 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.205 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 7, 2024-08-19T14:47:58.205 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-08-19T14:47:58.205 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-08-19T14:47:58.205 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-08-19T14:47:58.205 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-08-19T14:47:58.205 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-08-19T14:47:58.206 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-08-19T14:47:18.822276+0000", 2024-08-19T14:47:58.206 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-08-19T14:47:58.206 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.206 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.206 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 8, 2024-08-19T14:47:58.206 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-08-19T14:47:58.206 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-08-19T14:47:58.206 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-08-19T14:47:58.206 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-08-19T14:47:58.206 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-08-19T14:47:58.206 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-08-19T14:47:13.295962+0000", 2024-08-19T14:47:58.206 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-08-19T14:47:58.206 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.206 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.207 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 9, 2024-08-19T14:47:58.207 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-08-19T14:47:58.207 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-08-19T14:47:58.207 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-08-19T14:47:58.207 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-08-19T14:47:58.207 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-08-19T14:47:58.207 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-08-19T14:47:56.543823+0000", 2024-08-19T14:47:58.207 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-08-19T14:47:58.207 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.207 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.207 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 10, 2024-08-19T14:47:58.207 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-08-19T14:47:58.207 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-08-19T14:47:58.208 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-08-19T14:47:58.208 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-08-19T14:47:58.208 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-08-19T14:47:58.208 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-08-19T14:47:03.071701+0000", 2024-08-19T14:47:58.208 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 529 2024-08-19T14:47:58.208 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.208 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.208 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 11, 2024-08-19T14:47:58.208 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-08-19T14:47:58.208 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-08-19T14:47:58.208 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-08-19T14:47:58.208 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-08-19T14:47:58.208 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-08-19T14:47:58.208 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-08-19T14:47:14.659290+0000", 2024-08-19T14:47:58.209 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-08-19T14:47:58.209 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:47:58.209 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-08-19T14:47:58.209 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_upmap": [ 2024-08-19T14:47:58.209 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.209 INFO:teuthology.orchestra.run.smithi031.stdout: "pgid": "1.0", 2024-08-19T14:47:58.209 INFO:teuthology.orchestra.run.smithi031.stdout: "osds": [ 2024-08-19T14:47:58.209 INFO:teuthology.orchestra.run.smithi031.stdout: 2, 2024-08-19T14:47:58.209 INFO:teuthology.orchestra.run.smithi031.stdout: 9, 2024-08-19T14:47:58.209 INFO:teuthology.orchestra.run.smithi031.stdout: 0 2024-08-19T14:47:58.209 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:47:58.209 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:47:58.210 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-08-19T14:47:58.210 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_upmap_items": [], 2024-08-19T14:47:58.210 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_upmap_primaries": [], 2024-08-19T14:47:58.210 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_temp": [ 2024-08-19T14:47:58.210 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.210 INFO:teuthology.orchestra.run.smithi031.stdout: "pgid": "9.7", 2024-08-19T14:47:58.210 INFO:teuthology.orchestra.run.smithi031.stdout: "osds": [ 2024-08-19T14:47:58.210 INFO:teuthology.orchestra.run.smithi031.stdout: 10, 2024-08-19T14:47:58.210 INFO:teuthology.orchestra.run.smithi031.stdout: 7, 2024-08-19T14:47:58.210 INFO:teuthology.orchestra.run.smithi031.stdout: 2 2024-08-19T14:47:58.210 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:47:58.210 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.210 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:47:58.211 INFO:teuthology.orchestra.run.smithi031.stdout: "pgid": "9.d", 2024-08-19T14:47:58.211 INFO:teuthology.orchestra.run.smithi031.stdout: "osds": [ 2024-08-19T14:47:58.211 INFO:teuthology.orchestra.run.smithi031.stdout: 1, 2024-08-19T14:47:58.211 INFO:teuthology.orchestra.run.smithi031.stdout: 4 2024-08-19T14:47:58.211 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:47:58.211 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:47:58.211 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-08-19T14:47:58.211 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_temp": [], 2024-08-19T14:47:58.211 INFO:teuthology.orchestra.run.smithi031.stdout: "blocklist": { 2024-08-19T14:47:58.211 INFO:teuthology.orchestra.run.smithi031.stdout: "172.21.15.31:0/1451745969": "2024-08-20T14:06:13.660657+0000", 2024-08-19T14:47:58.211 INFO:teuthology.orchestra.run.smithi031.stdout: "172.21.15.31:0/1515698918": "2024-08-20T14:06:13.660657+0000", 2024-08-19T14:47:58.211 INFO:teuthology.orchestra.run.smithi031.stdout: "172.21.15.31:6801/3297539242": "2024-08-20T14:06:13.660657+0000", 2024-08-19T14:47:58.211 INFO:teuthology.orchestra.run.smithi031.stdout: "172.21.15.31:6800/3297539242": "2024-08-20T14:06:13.660657+0000", 2024-08-19T14:47:58.212 INFO:teuthology.orchestra.run.smithi031.stdout: "172.21.15.31:0/2217423818": "2024-08-20T14:05:48.827576+0000", 2024-08-19T14:47:58.212 INFO:teuthology.orchestra.run.smithi031.stdout: "172.21.15.31:0/3353498281": "2024-08-20T14:06:13.660657+0000", 2024-08-19T14:47:58.212 INFO:teuthology.orchestra.run.smithi031.stdout: "172.21.15.31:0/4013594220": "2024-08-20T14:05:48.827576+0000", 2024-08-19T14:47:58.212 INFO:teuthology.orchestra.run.smithi031.stdout: "172.21.15.31:0/1654518065": "2024-08-20T14:05:48.827576+0000", 2024-08-19T14:47:58.212 INFO:teuthology.orchestra.run.smithi031.stdout: "172.21.15.31:6801/2368821258": "2024-08-20T14:05:48.827576+0000", 2024-08-19T14:47:58.212 INFO:teuthology.orchestra.run.smithi031.stdout: "172.21.15.31:6800/2368821258": "2024-08-20T14:05:48.827576+0000" 2024-08-19T14:47:58.212 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.212 INFO:teuthology.orchestra.run.smithi031.stdout: "range_blocklist": {}, 2024-08-19T14:47:58.212 INFO:teuthology.orchestra.run.smithi031.stdout: "erasure_code_profiles": { 2024-08-19T14:47:58.212 INFO:teuthology.orchestra.run.smithi031.stdout: "default": { 2024-08-19T14:47:58.212 INFO:teuthology.orchestra.run.smithi031.stdout: "crush-failure-domain": "osd", 2024-08-19T14:47:58.212 INFO:teuthology.orchestra.run.smithi031.stdout: "k": "2", 2024-08-19T14:47:58.212 INFO:teuthology.orchestra.run.smithi031.stdout: "m": "1", 2024-08-19T14:47:58.213 INFO:teuthology.orchestra.run.smithi031.stdout: "plugin": "jerasure", 2024-08-19T14:47:58.213 INFO:teuthology.orchestra.run.smithi031.stdout: "technique": "reed_sol_van" 2024-08-19T14:47:58.213 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:47:58.213 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:47:58.213 INFO:teuthology.orchestra.run.smithi031.stdout: "removed_snaps_queue": [], 2024-08-19T14:47:58.213 INFO:teuthology.orchestra.run.smithi031.stdout: "new_removed_snaps": [], 2024-08-19T14:47:58.213 INFO:teuthology.orchestra.run.smithi031.stdout: "new_purged_snaps": [], 2024-08-19T14:47:58.213 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_node_flags": {}, 2024-08-19T14:47:58.213 INFO:teuthology.orchestra.run.smithi031.stdout: "device_class_flags": {}, 2024-08-19T14:47:58.213 INFO:teuthology.orchestra.run.smithi031.stdout: "stretch_mode": { 2024-08-19T14:47:58.213 INFO:teuthology.orchestra.run.smithi031.stdout: "stretch_mode_enabled": false, 2024-08-19T14:47:58.213 INFO:teuthology.orchestra.run.smithi031.stdout: "stretch_bucket_count": 0, 2024-08-19T14:47:58.213 INFO:teuthology.orchestra.run.smithi031.stdout: "degraded_stretch_mode": 0, 2024-08-19T14:47:58.214 INFO:teuthology.orchestra.run.smithi031.stdout: "recovering_stretch_mode": 0, 2024-08-19T14:47:58.214 INFO:teuthology.orchestra.run.smithi031.stdout: "stretch_mode_bucket": 0 2024-08-19T14:47:58.214 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:47:58.214 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:47:58.214 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:47:58.215 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is unset 2024-08-19T14:47:58.234 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:58 smithi031 ceph-mon[16336]: pgmap v2165: 47 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 45 active+clean; 431 MiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 718 op/s; 25/20661 objects degraded (0.121%); 6/20661 objects misplaced (0.029%); 35 MiB/s, 629 objects/s recovering 2024-08-19T14:47:58.234 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:58 smithi031 ceph-mon[16336]: osdmap e925: 12 total, 12 up, 10 in 2024-08-19T14:47:58.235 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:58 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1836235078' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:47:58.235 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:58 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3074486596' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-08-19T14:47:58.235 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:58 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 25/20661 objects degraded (0.121%), 1 pg degraded (PG_DEGRADED) 2024-08-19T14:47:58.241 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:58.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:58.241 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:47:58.342 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:47:58.367 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:58.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:58.368 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:47:58.469 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:47:58.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:58.496 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:58.496 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:47:58.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:58 smithi032 ceph-mon[20004]: pgmap v2165: 47 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 45 active+clean; 431 MiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 718 op/s; 25/20661 objects degraded (0.121%); 6/20661 objects misplaced (0.029%); 35 MiB/s, 629 objects/s recovering 2024-08-19T14:47:58.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:58 smithi032 ceph-mon[20004]: osdmap e925: 12 total, 12 up, 10 in 2024-08-19T14:47:58.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:58 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1836235078' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:47:58.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:58 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3074486596' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-08-19T14:47:58.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:58 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 25/20661 objects degraded (0.121%), 1 pg degraded (PG_DEGRADED) 2024-08-19T14:47:58.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:58 smithi045 ceph-mon[20143]: pgmap v2165: 47 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 45 active+clean; 431 MiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 718 op/s; 25/20661 objects degraded (0.121%); 6/20661 objects misplaced (0.029%); 35 MiB/s, 629 objects/s recovering 2024-08-19T14:47:58.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:58 smithi045 ceph-mon[20143]: osdmap e925: 12 total, 12 up, 10 in 2024-08-19T14:47:58.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:58 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1836235078' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:47:58.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:58 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3074486596' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-08-19T14:47:58.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:58 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 25/20661 objects degraded (0.121%), 1 pg degraded (PG_DEGRADED) 2024-08-19T14:47:58.597 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:47:58.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:58.621 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:58.622 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:47:58.723 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:47:58.761 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:58.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:58.762 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:47:58.863 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:47:58.884 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:47:58.885 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:47:58.892 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:58.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:58.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:47:58.910 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:47:58.911 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:47:58.942 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:47:58.942 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:47:58.973 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:47:58.973 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:47:58.994 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:47:59.002 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:47:59.002 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:47:59.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:59.017 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:59.017 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:47:59.031 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:47:59.031 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:47:59.056 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:47:59.057 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:47:59.082 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:47:59.083 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:47:59.110 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:47:59.111 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:47:59.118 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:47:59.145 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:47:59.146 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:47:59.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:59.154 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:59.154 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:47:59.175 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:47:59.175 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:47:59.208 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:47:59.209 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:47:59.254 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:47:59.291 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:59.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:59.292 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:47:59.393 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:47:59.418 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:59.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:59.420 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:47:59.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:59 smithi032 ceph-mon[20004]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:47:59.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:59 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1836235078' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:47:59.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:59 smithi032 ceph-mon[20004]: osdmap e926: 12 total, 12 up, 10 in 2024-08-19T14:47:59.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:47:59 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/341805804' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:47:59.520 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:47:59.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:59 smithi045 ceph-mon[20143]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:47:59.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:59 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1836235078' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:47:59.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:59 smithi045 ceph-mon[20143]: osdmap e926: 12 total, 12 up, 10 in 2024-08-19T14:47:59.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:47:59 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/341805804' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:47:59.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:59.555 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:47:59.555 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:47:59.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:59 smithi031 ceph-mon[16336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-19T14:47:59.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:59 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1836235078' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:47:59.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:59 smithi031 ceph-mon[16336]: osdmap e926: 12 total, 12 up, 10 in 2024-08-19T14:47:59.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:47:59 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/341805804' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:47:59.656 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:47:59.685 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:59.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:59.686 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:47:59.787 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:47:59.819 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:47:59.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:59.820 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:47:59.921 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:47:59.946 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:47:59.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:47:59.947 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:48:00.048 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:48:00.071 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:48:00.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:00.072 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:48:00.173 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:48:00.202 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:48:00.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:00.203 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:48:00.304 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:48:00.336 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:48:00.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:00.337 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:48:00.438 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:48:00.478 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:48:00.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:00.479 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:48:00.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:00 smithi032 ceph-mon[20004]: pgmap v2168: 47 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 45 active+clean; 400 MiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 649 op/s; 25/19197 objects degraded (0.130%); 6/19197 objects misplaced (0.031%); 3.1 MiB/s, 121 objects/s recovering 2024-08-19T14:48:00.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:00 smithi045 ceph-mon[20143]: pgmap v2168: 47 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 45 active+clean; 400 MiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 649 op/s; 25/19197 objects degraded (0.130%); 6/19197 objects misplaced (0.031%); 3.1 MiB/s, 121 objects/s recovering 2024-08-19T14:48:00.580 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:48:00.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:00 smithi031 ceph-mon[16336]: pgmap v2168: 47 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 45 active+clean; 400 MiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 649 op/s; 25/19197 objects degraded (0.130%); 6/19197 objects misplaced (0.031%); 3.1 MiB/s, 121 objects/s recovering 2024-08-19T14:48:00.618 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:48:00.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:00.619 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:48:00.720 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:48:00.749 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:48:00.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:00.750 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:48:00.851 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:48:00.882 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:48:00.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:00.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:48:00.984 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:48:01.012 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:48:01.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:01.013 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:48:01.114 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:48:01.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:01.147 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:48:01.147 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:48:01.248 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:48:01.274 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:48:01.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:01.275 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:48:01.376 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:48:01.404 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:48:01.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:01.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:48:01.507 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:48:01.530 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:48:01.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:01.531 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:48:01.632 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:48:01.658 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:48:01.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:01.660 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:48:01.761 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:48:01.783 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:48:01.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:01.785 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:48:01.886 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:48:01.920 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:48:01.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:01.921 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:48:02.022 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:48:02.047 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:48:02.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:02.048 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:48:02.148 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:48:02.180 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:48:02.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:02.181 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:48:02.283 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:48:02.307 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:48:02.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:02.309 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:48:02.410 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:48:02.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:02.445 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:48:02.445 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:48:02.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:02 smithi032 ceph-mon[20004]: pgmap v2169: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 45 active+clean; 294 MiB data, 11 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 851 op/s; 25/14082 objects degraded (0.178%); 6/14082 objects misplaced (0.043%); 3.0 MiB/s, 174 objects/s recovering 2024-08-19T14:48:02.546 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:48:02.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:02 smithi045 ceph-mon[20143]: pgmap v2169: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 45 active+clean; 294 MiB data, 11 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 851 op/s; 25/14082 objects degraded (0.178%); 6/14082 objects misplaced (0.043%); 3.0 MiB/s, 174 objects/s recovering 2024-08-19T14:48:02.577 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:48:02.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:02.578 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:48:02.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:02 smithi031 ceph-mon[16336]: pgmap v2169: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 45 active+clean; 294 MiB data, 11 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 851 op/s; 25/14082 objects degraded (0.178%); 6/14082 objects misplaced (0.043%); 3.0 MiB/s, 174 objects/s recovering 2024-08-19T14:48:02.679 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:48:02.716 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:48:02.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:02.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:48:02.818 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:48:02.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:02.850 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:48:02.850 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:48:02.858 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Removed 94502 objects 2024-08-19T14:48:02.858 INFO:tasks.radosbench.radosbench.2.smithi045.stdout:Clean up completed and total clean up time :130.53 2024-08-19T14:48:02.951 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:48:02.972 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:48:02.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:02.973 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:48:03.074 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:48:03.095 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:48:03.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:03.097 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:48:03.197 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:48:03.221 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:48:03.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:03.222 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:48:03.324 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:48:03.345 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:48:03.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:03.346 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:48:03.447 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:48:03.468 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:48:03.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:03.469 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:48:03.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:03 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 25/14082 objects degraded (0.178%), 1 pg degraded (PG_DEGRADED) 2024-08-19T14:48:03.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:03 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 25/14082 objects degraded (0.178%), 1 pg degraded (PG_DEGRADED) 2024-08-19T14:48:03.570 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:48:03.592 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:48:03.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:03.593 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:48:03.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:03 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 25/14082 objects degraded (0.178%), 1 pg degraded (PG_DEGRADED) 2024-08-19T14:48:03.693 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:48:03.714 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:48:03.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:03.715 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:48:03.816 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:48:03.838 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:48:03.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:03.839 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:48:03.940 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:48:03.961 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:48:03.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:03.962 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:48:04.063 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:48:04.085 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:48:04.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:04.086 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:48:04.187 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:48:04.211 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:48:04.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:04.212 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:48:04.312 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:48:04.334 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:48:04.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:04.335 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:48:04.438 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:48:04.460 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:48:04.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:04.461 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:48:04.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:04 smithi032 ceph-mon[20004]: pgmap v2170: 47 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 45 active+clean; 171 MiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 759 op/s; 25/8187 objects degraded (0.305%); 2.4 MiB/s, 119 objects/s recovering 2024-08-19T14:48:04.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:04 smithi045 ceph-mon[20143]: pgmap v2170: 47 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 45 active+clean; 171 MiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 759 op/s; 25/8187 objects degraded (0.305%); 2.4 MiB/s, 119 objects/s recovering 2024-08-19T14:48:04.562 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:48:04.583 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:48:04.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:04.584 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:48:04.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:04 smithi031 ceph-mon[16336]: pgmap v2170: 47 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 45 active+clean; 171 MiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 759 op/s; 25/8187 objects degraded (0.305%); 2.4 MiB/s, 119 objects/s recovering 2024-08-19T14:48:04.685 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:48:04.707 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:48:04.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:04.708 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:48:04.809 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:48:04.832 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:48:04.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:04.833 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:48:04.934 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:48:04.957 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:48:04.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:04.959 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:48:05.060 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:48:05.080 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:48:05.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:05.082 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:48:05.183 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:48:05.205 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:48:05.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:05.206 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:48:05.307 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:48:05.328 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:48:05.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:05.329 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:48:05.430 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:48:05.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:05.454 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:48:05.454 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:48:05.555 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:48:05.577 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:48:05.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:05.578 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:48:05.679 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:48:05.701 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:48:05.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:05.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:48:05.804 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:48:05.826 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:48:05.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:05.827 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:48:05.928 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:48:05.950 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:48:05.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:05.951 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:48:06.052 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:48:06.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:06.076 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:48:06.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:48:06.178 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:48:06.200 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:48:06.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:06.201 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:48:06.302 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:48:06.325 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:48:06.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:06.326 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:48:06.358 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-19T14:48:06.427 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:48:06.448 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:48:06.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:06.449 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:48:06.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:06 smithi032 ceph-mon[20004]: pgmap v2171: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 45 active+clean; 56 MiB data, 9.9 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 748 op/s; 25/2688 objects degraded (0.930%); 24 KiB/s, 147 objects/s recovering 2024-08-19T14:48:06.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:06 smithi045 ceph-mon[20143]: pgmap v2171: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 45 active+clean; 56 MiB data, 9.9 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 748 op/s; 25/2688 objects degraded (0.930%); 24 KiB/s, 147 objects/s recovering 2024-08-19T14:48:06.550 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:48:06.571 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:48:06.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:06.572 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:48:06.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:06 smithi031 ceph-mon[16336]: pgmap v2171: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 45 active+clean; 56 MiB data, 9.9 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 748 op/s; 25/2688 objects degraded (0.930%); 24 KiB/s, 147 objects/s recovering 2024-08-19T14:48:06.673 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:48:06.694 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:48:06.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:06.695 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:48:06.796 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:48:06.819 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:48:06.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:06.820 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:48:06.921 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:48:06.942 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:48:06.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:06.944 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:48:07.044 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:48:07.066 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:48:07.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:07.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:48:07.080 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_7 2024-08-19T14:48:07.080 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd pool rm unique_pool_7 unique_pool_7 --yes-i-really-really-mean-it 2024-08-19T14:48:07.167 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:48:07.193 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:48:07.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:07.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:48:07.203 INFO:tasks.thrashosds.thrasher:j is {'epoch': 925, 'fsid': 'ed7d1c6e-5e33-11ef-bccf-c7b262605968', 'created': '2024-08-19T14:05:22.842169+0000', 'modified': '2024-08-19T14:47:57.171764+0000', 'last_up_change': '2024-08-19T14:35:10.517930+0000', 'last_in_change': '2024-08-19T14:47:41.794306+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 29, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 9, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-08-19T14:08:15.966221+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 31, 'pg_placement_num': 31, 'pg_placement_num_target': 31, 'pg_num_target': 31, 'pg_num_pending': 31, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '278', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '276', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5799999237060547, 'score_stable': 2.5799999237060547, 'optimal_score': 1, 'raw_score_acting': 2.5799999237060547, 'raw_score_stable': 2.5799999237060547, 'primary_affinity_weighted': 0.7561492323875427, 'average_primary_affinity': 0.9100000262260437, 'average_primary_affinity_weighted': 0.7599999904632568}}, {'pool': 9, 'pool_name': 'unique_pool_7', 'create_time': '2024-08-19T14:44:13.804952+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '829', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3.119999885559082, 'score_stable': 2.5, 'optimal_score': 0.8999999761581421, 'raw_score_acting': 2.809999942779541, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.7561492323875427, 'average_primary_affinity': 0.9100000262260437, 'average_primary_affinity_weighted': 0.7599999904632568}}], 'osds': [{'osd': 0, 'uuid': 'd7a0fff3-23c6-4ee3-a275-a844862aabfb', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 820, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6802', 'nonce': 1108945097}, {'type': 'v1', 'addr': '172.21.15.31:6803', 'nonce': 1108945097}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6804', 'nonce': 1108945097}, {'type': 'v1', 'addr': '172.21.15.31:6805', 'nonce': 1108945097}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6808', 'nonce': 1108945097}, {'type': 'v1', 'addr': '172.21.15.31:6809', 'nonce': 1108945097}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6806', 'nonce': 1108945097}, {'type': 'v1', 'addr': '172.21.15.31:6807', 'nonce': 1108945097}]}, 'public_addr': '172.21.15.31:6803/1108945097', 'cluster_addr': '172.21.15.31:6805/1108945097', 'heartbeat_back_addr': '172.21.15.31:6809/1108945097', 'heartbeat_front_addr': '172.21.15.31:6807/1108945097', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '145d5bb4-74dd-440f-bcc9-7e9ebda9c5d6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 921, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6810', 'nonce': 3236186806}, {'type': 'v1', 'addr': '172.21.15.31:6811', 'nonce': 3236186806}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6812', 'nonce': 3236186806}, {'type': 'v1', 'addr': '172.21.15.31:6813', 'nonce': 3236186806}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6816', 'nonce': 3236186806}, {'type': 'v1', 'addr': '172.21.15.31:6817', 'nonce': 3236186806}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6814', 'nonce': 3236186806}, {'type': 'v1', 'addr': '172.21.15.31:6815', 'nonce': 3236186806}]}, 'public_addr': '172.21.15.31:6811/3236186806', 'cluster_addr': '172.21.15.31:6813/3236186806', 'heartbeat_back_addr': '172.21.15.31:6817/3236186806', 'heartbeat_front_addr': '172.21.15.31:6815/3236186806', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '376c18d0-4768-46a0-bb60-e79c88ca909c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.07379150390625, 'last_clean_begin': 18, 'last_clean_end': 564, 'up_from': 634, 'up_thru': 916, 'down_at': 565, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6818', 'nonce': 656997856}, {'type': 'v1', 'addr': '172.21.15.31:6819', 'nonce': 656997856}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6820', 'nonce': 656997856}, {'type': 'v1', 'addr': '172.21.15.31:6821', 'nonce': 656997856}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6824', 'nonce': 656997856}, {'type': 'v1', 'addr': '172.21.15.31:6825', 'nonce': 656997856}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6822', 'nonce': 656997856}, {'type': 'v1', 'addr': '172.21.15.31:6823', 'nonce': 656997856}]}, 'public_addr': '172.21.15.31:6819/656997856', 'cluster_addr': '172.21.15.31:6821/656997856', 'heartbeat_back_addr': '172.21.15.31:6825/656997856', 'heartbeat_front_addr': '172.21.15.31:6823/656997856', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'a787ea9f-b9ba-450f-b388-614cfba90634', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 922, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6826', 'nonce': 2288082716}, {'type': 'v1', 'addr': '172.21.15.31:6827', 'nonce': 2288082716}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6828', 'nonce': 2288082716}, {'type': 'v1', 'addr': '172.21.15.31:6829', 'nonce': 2288082716}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6832', 'nonce': 2288082716}, {'type': 'v1', 'addr': '172.21.15.31:6833', 'nonce': 2288082716}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6830', 'nonce': 2288082716}, {'type': 'v1', 'addr': '172.21.15.31:6831', 'nonce': 2288082716}]}, 'public_addr': '172.21.15.31:6827/2288082716', 'cluster_addr': '172.21.15.31:6829/2288082716', 'heartbeat_back_addr': '172.21.15.31:6833/2288082716', 'heartbeat_front_addr': '172.21.15.31:6831/2288082716', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '7859751a-12f5-4a57-a332-435360479d53', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 918, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6800', 'nonce': 3934739984}, {'type': 'v1', 'addr': '172.21.15.32:6801', 'nonce': 3934739984}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6802', 'nonce': 3934739984}, {'type': 'v1', 'addr': '172.21.15.32:6803', 'nonce': 3934739984}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6806', 'nonce': 3934739984}, {'type': 'v1', 'addr': '172.21.15.32:6807', 'nonce': 3934739984}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6804', 'nonce': 3934739984}, {'type': 'v1', 'addr': '172.21.15.32:6805', 'nonce': 3934739984}]}, 'public_addr': '172.21.15.32:6801/3934739984', 'cluster_addr': '172.21.15.32:6803/3934739984', 'heartbeat_back_addr': '172.21.15.32:6807/3934739984', 'heartbeat_front_addr': '172.21.15.32:6805/3934739984', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '017f3eef-ac60-4b82-8350-919017cb4ccd', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 40, 'up_thru': 817, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6808', 'nonce': 526388985}, {'type': 'v1', 'addr': '172.21.15.32:6809', 'nonce': 526388985}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6810', 'nonce': 526388985}, {'type': 'v1', 'addr': '172.21.15.32:6811', 'nonce': 526388985}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6814', 'nonce': 526388985}, {'type': 'v1', 'addr': '172.21.15.32:6815', 'nonce': 526388985}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6812', 'nonce': 526388985}, {'type': 'v1', 'addr': '172.21.15.32:6813', 'nonce': 526388985}]}, 'public_addr': '172.21.15.32:6809/526388985', 'cluster_addr': '172.21.15.32:6811/526388985', 'heartbeat_back_addr': '172.21.15.32:6815/526388985', 'heartbeat_front_addr': '172.21.15.32:6813/526388985', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '44aa8229-0268-49d7-bd57-3bf1f384e5e6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 48, 'up_thru': 919, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6816', 'nonce': 1131035844}, {'type': 'v1', 'addr': '172.21.15.32:6817', 'nonce': 1131035844}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6818', 'nonce': 1131035844}, {'type': 'v1', 'addr': '172.21.15.32:6819', 'nonce': 1131035844}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6822', 'nonce': 1131035844}, {'type': 'v1', 'addr': '172.21.15.32:6823', 'nonce': 1131035844}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6820', 'nonce': 1131035844}, {'type': 'v1', 'addr': '172.21.15.32:6821', 'nonce': 1131035844}]}, 'public_addr': '172.21.15.32:6817/1131035844', 'cluster_addr': '172.21.15.32:6819/1131035844', 'heartbeat_back_addr': '172.21.15.32:6823/1131035844', 'heartbeat_front_addr': '172.21.15.32:6821/1131035844', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '263433e5-70b6-4192-86ff-dc2ad95927eb', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 57, 'up_thru': 907, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6824', 'nonce': 2014939947}, {'type': 'v1', 'addr': '172.21.15.32:6825', 'nonce': 2014939947}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6826', 'nonce': 2014939947}, {'type': 'v1', 'addr': '172.21.15.32:6827', 'nonce': 2014939947}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6830', 'nonce': 2014939947}, {'type': 'v1', 'addr': '172.21.15.32:6831', 'nonce': 2014939947}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6828', 'nonce': 2014939947}, {'type': 'v1', 'addr': '172.21.15.32:6829', 'nonce': 2014939947}]}, 'public_addr': '172.21.15.32:6825/2014939947', 'cluster_addr': '172.21.15.32:6827/2014939947', 'heartbeat_back_addr': '172.21.15.32:6831/2014939947', 'heartbeat_front_addr': '172.21.15.32:6829/2014939947', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': '2bbcad36-0c74-4528-9fb9-a8812afa1662', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 65, 'up_thru': 917, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6800', 'nonce': 1445140892}, {'type': 'v1', 'addr': '172.21.15.45:6801', 'nonce': 1445140892}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6802', 'nonce': 1445140892}, {'type': 'v1', 'addr': '172.21.15.45:6803', 'nonce': 1445140892}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6806', 'nonce': 1445140892}, {'type': 'v1', 'addr': '172.21.15.45:6807', 'nonce': 1445140892}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6804', 'nonce': 1445140892}, {'type': 'v1', 'addr': '172.21.15.45:6805', 'nonce': 1445140892}]}, 'public_addr': '172.21.15.45:6801/1445140892', 'cluster_addr': '172.21.15.45:6803/1445140892', 'heartbeat_back_addr': '172.21.15.45:6807/1445140892', 'heartbeat_front_addr': '172.21.15.45:6805/1445140892', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': 'c8ee4277-3bf9-46bf-9cc0-152c1c4b9508', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 74, 'up_thru': 793, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6808', 'nonce': 1213787169}, {'type': 'v1', 'addr': '172.21.15.45:6809', 'nonce': 1213787169}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6810', 'nonce': 1213787169}, {'type': 'v1', 'addr': '172.21.15.45:6811', 'nonce': 1213787169}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6814', 'nonce': 1213787169}, {'type': 'v1', 'addr': '172.21.15.45:6815', 'nonce': 1213787169}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6812', 'nonce': 1213787169}, {'type': 'v1', 'addr': '172.21.15.45:6813', 'nonce': 1213787169}]}, 'public_addr': '172.21.15.45:6809/1213787169', 'cluster_addr': '172.21.15.45:6811/1213787169', 'heartbeat_back_addr': '172.21.15.45:6815/1213787169', 'heartbeat_front_addr': '172.21.15.45:6813/1213787169', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '29031395-5f5c-40f8-ac90-9f6ac48e5c9f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 80, 'last_clean_end': 529, 'up_from': 539, 'up_thru': 921, 'down_at': 530, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6816', 'nonce': 1069743893}, {'type': 'v1', 'addr': '172.21.15.45:6817', 'nonce': 1069743893}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6818', 'nonce': 1069743893}, {'type': 'v1', 'addr': '172.21.15.45:6819', 'nonce': 1069743893}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6822', 'nonce': 1069743893}, {'type': 'v1', 'addr': '172.21.15.45:6823', 'nonce': 1069743893}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6820', 'nonce': 1069743893}, {'type': 'v1', 'addr': '172.21.15.45:6821', 'nonce': 1069743893}]}, 'public_addr': '172.21.15.45:6817/1069743893', 'cluster_addr': '172.21.15.45:6819/1069743893', 'heartbeat_back_addr': '172.21.15.45:6823/1069743893', 'heartbeat_front_addr': '172.21.15.45:6821/1069743893', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': '5ef0db4b-0fef-4ea0-a19b-d3677e082c29', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 86, 'up_thru': 910, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6824', 'nonce': 2438396298}, {'type': 'v1', 'addr': '172.21.15.45:6825', 'nonce': 2438396298}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6826', 'nonce': 2438396298}, {'type': 'v1', 'addr': '172.21.15.45:6827', 'nonce': 2438396298}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6830', 'nonce': 2438396298}, {'type': 'v1', 'addr': '172.21.15.45:6831', 'nonce': 2438396298}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6828', 'nonce': 2438396298}, {'type': 'v1', 'addr': '172.21.15.45:6829', 'nonce': 2438396298}]}, 'public_addr': '172.21.15.45:6825/2438396298', 'cluster_addr': '172.21.15.45:6827/2438396298', 'heartbeat_back_addr': '172.21.15.45:6831/2438396298', 'heartbeat_front_addr': '172.21.15.45:6829/2438396298', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-08-19T14:47:49.205675+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-19T14:47:53.610800+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-19T14:47:47.260684+0000', 'dead_epoch': 564}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-19T14:46:49.576968+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-19T14:47:33.446105+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-08-19T14:47:03.078224+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-19T14:47:53.672486+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-19T14:47:18.822276+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-19T14:47:13.295962+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-19T14:47:56.543823+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-19T14:47:03.071701+0000', 'dead_epoch': 529}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-19T14:47:14.659290+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '1.0', 'osds': [2, 9, 0]}], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '9.7', 'osds': [10, 7, 2]}, {'pgid': '9.d', 'osds': [1, 4]}], 'primary_temp': [], 'blocklist': {'172.21.15.31:0/1451745969': '2024-08-20T14:06:13.660657+0000', '172.21.15.31:0/1515698918': '2024-08-20T14:06:13.660657+0000', '172.21.15.31:6801/3297539242': '2024-08-20T14:06:13.660657+0000', '172.21.15.31:6800/3297539242': '2024-08-20T14:06:13.660657+0000', '172.21.15.31:0/2217423818': '2024-08-20T14:05:48.827576+0000', '172.21.15.31:0/3353498281': '2024-08-20T14:06:13.660657+0000', '172.21.15.31:0/4013594220': '2024-08-20T14:05:48.827576+0000', '172.21.15.31:0/1654518065': '2024-08-20T14:05:48.827576+0000', '172.21.15.31:6801/2368821258': '2024-08-20T14:05:48.827576+0000', '172.21.15.31:6800/2368821258': '2024-08-20T14:05:48.827576+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-08-19T14:48:07.203 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2024-08-19T14:48:07.294 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:48:07.319 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:48:07.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:07.320 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:48:07.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:07 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:48:07.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:07 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:48:07.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:07 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:48:07.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:07 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:48:07.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:07 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:48:07.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:07 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:48:07.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:07 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:48:07.420 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:48:07.442 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:48:07.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:07.443 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:48:07.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:07 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:48:07.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:07 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:48:07.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:07 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:48:07.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:07 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:48:07.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:07 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:48:07.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:07 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:48:07.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:07 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:48:07.544 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:48:07.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:07 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:48:07.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:07 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:48:07.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:07 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:48:07.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:07 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:48:07.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:07 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:48:07.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:07 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:48:07.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:07 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:48:07.566 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:48:07.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:07.567 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:48:07.668 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:48:07.684 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:48:07.684 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-08-19T14:48:07.684 INFO:teuthology.orchestra.run.smithi031.stdout: "complaint_time": 30, 2024-08-19T14:48:07.684 INFO:teuthology.orchestra.run.smithi031.stdout: "num_blocked_ops": 0 2024-08-19T14:48:07.684 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:48:07.693 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:48:07.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:07.694 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:48:07.795 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:48:07.816 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:48:07.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:07.817 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:48:07.918 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:48:07.939 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:48:07.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:07.940 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:48:08.041 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:48:08.062 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:48:08.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:08.063 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:48:08.164 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:48:08.185 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:48:08.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:08.187 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:48:08.287 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:48:08.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:08.309 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:48:08.310 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:48:08.336 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:08 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:48:08.337 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:08 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:48:08.337 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:08 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 25/2688 objects degraded (0.930%), 1 pg degraded (PG_DEGRADED) 2024-08-19T14:48:08.366 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-08-19T14:48:08.410 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:48:08.432 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:48:08.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:08.433 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:48:08.437 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:48:08.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:08 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:48:08.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:08 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:48:08.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:08 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 25/2688 objects degraded (0.930%), 1 pg degraded (PG_DEGRADED) 2024-08-19T14:48:08.534 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:48:08.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:08 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:48:08.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:08 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:48:08.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:08 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 25/2688 objects degraded (0.930%), 1 pg degraded (PG_DEGRADED) 2024-08-19T14:48:08.556 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:48:08.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:08.557 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:48:08.658 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:48:08.680 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:48:08.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:08.681 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:48:08.782 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:48:08.804 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:48:08.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:08.805 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:48:08.906 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:48:08.928 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:48:08.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:08.929 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:48:09.030 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:48:09.052 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:48:09.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:09.053 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:48:09.084 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-19T14:48:09.154 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:48:09.179 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:48:09.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:09.180 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:48:09.280 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:48:09.303 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:48:09.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:09.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:48:09.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:09 smithi031 ceph-mon[16336]: osdmap e927: 12 total, 12 up, 10 in 2024-08-19T14:48:09.405 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:48:09.427 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:48:09.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:09.428 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:48:09.435 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:48:09.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:09 smithi032 ceph-mon[20004]: osdmap e927: 12 total, 12 up, 10 in 2024-08-19T14:48:09.530 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:48:09.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:09 smithi045 ceph-mon[20143]: osdmap e927: 12 total, 12 up, 10 in 2024-08-19T14:48:09.551 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:48:09.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:09.552 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:48:09.654 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:48:09.675 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:48:09.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:09.676 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:48:09.777 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:48:09.798 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:48:09.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:09.799 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:48:09.900 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:48:09.921 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:48:09.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:09.923 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:48:10.023 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:48:10.044 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:48:10.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:10.046 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:48:10.147 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:48:10.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:10.171 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:48:10.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:48:10.272 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:48:10.295 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:48:10.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:10.296 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:48:10.397 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:48:10.420 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:48:10.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:10.421 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:48:10.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:10 smithi032 ceph-mon[20004]: pgmap v2174: 47 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 45 active+clean; 629 KiB data, 9.6 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 639 op/s; 25/6 objects degraded (416.667%); 19 KiB/s, 180 objects/s recovering 2024-08-19T14:48:10.522 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:48:10.544 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:48:10.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:10.545 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:48:10.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:10 smithi045 ceph-mon[20143]: pgmap v2174: 47 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 45 active+clean; 629 KiB data, 9.6 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 639 op/s; 25/6 objects degraded (416.667%); 19 KiB/s, 180 objects/s recovering 2024-08-19T14:48:10.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:10 smithi031 ceph-mon[16336]: pgmap v2174: 47 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 45 active+clean; 629 KiB data, 9.6 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 639 op/s; 25/6 objects degraded (416.667%); 19 KiB/s, 180 objects/s recovering 2024-08-19T14:48:10.646 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:48:10.669 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:48:10.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:10.670 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:48:10.771 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:48:10.794 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:48:10.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:10.795 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:48:10.896 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:48:10.918 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:48:10.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:10.919 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:48:11.020 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:48:11.042 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:48:11.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:11.043 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:48:11.136 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:48:11.136 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:48:11.144 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:48:11.167 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:48:11.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:11.168 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:48:11.229 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'unique_pool_7' removed 2024-08-19T14:48:11.269 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0 2024-08-19T14:48:11.295 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:48:11.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:11.296 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.0' 2024-08-19T14:48:11.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:11 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1685818063' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]: dispatch 2024-08-19T14:48:11.397 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:48:11.417 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:48:11.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:11.419 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:48:11.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:11 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1685818063' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]: dispatch 2024-08-19T14:48:11.519 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:48:11.541 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:48:11.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:11.542 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:48:11.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:11 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1685818063' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]: dispatch 2024-08-19T14:48:11.642 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:48:11.664 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:48:11.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:11.665 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:48:11.766 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:48:11.788 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:48:11.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:11.789 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:48:11.890 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:48:11.913 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:48:11.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:11.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:48:12.015 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:48:12.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:12.038 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:48:12.039 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:48:12.139 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1 2024-08-19T14:48:12.169 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:48:12.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:12.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.1' 2024-08-19T14:48:12.204 INFO:tasks.thrashosds.thrasher:in_osds: [1, 4, 8, 2, 11, 9, 10, 6, 7, 3] out_osds: [0, 5] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 8, 9, 11, 10, 2] 2024-08-19T14:48:12.204 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-19T14:48:12.204 INFO:tasks.thrashosds.thrasher:inject_pause on osd.2 2024-08-19T14:48:12.205 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-08-19T14:48:12.205 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-08-19T14:48:12.205 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.2 config set filestore_inject_stall 3 2024-08-19T14:48:12.272 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:48:12.293 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:48:12.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:12.294 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:48:12.391 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:12 smithi031 ceph-mon[16336]: pgmap v2175: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 45 active+clean; 629 KiB data, 9.6 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 469 op/s; 25/6 objects degraded (416.667%); 19 KiB/s, 263 objects/s recovering 2024-08-19T14:48:12.391 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:12 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:48:12.391 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:12 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:48:12.391 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:12 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1685818063' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]': finished 2024-08-19T14:48:12.391 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:12 smithi031 ceph-mon[16336]: osdmap e928: 12 total, 12 up, 10 in 2024-08-19T14:48:12.395 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:48:12.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:12.419 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:48:12.419 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:48:12.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:12 smithi032 ceph-mon[20004]: pgmap v2175: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 45 active+clean; 629 KiB data, 9.6 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 469 op/s; 25/6 objects degraded (416.667%); 19 KiB/s, 263 objects/s recovering 2024-08-19T14:48:12.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:12 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:48:12.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:12 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:48:12.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:12 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1685818063' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]': finished 2024-08-19T14:48:12.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:12 smithi032 ceph-mon[20004]: osdmap e928: 12 total, 12 up, 10 in 2024-08-19T14:48:12.519 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6 2024-08-19T14:48:12.544 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:48:12.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:12.545 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.6' 2024-08-19T14:48:12.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:12 smithi045 ceph-mon[20143]: pgmap v2175: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 45 active+clean; 629 KiB data, 9.6 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 469 op/s; 25/6 objects degraded (416.667%); 19 KiB/s, 263 objects/s recovering 2024-08-19T14:48:12.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:12 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:48:12.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:12 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:48:12.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:12 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1685818063' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]': finished 2024-08-19T14:48:12.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:12 smithi045 ceph-mon[20143]: osdmap e928: 12 total, 12 up, 10 in 2024-08-19T14:48:12.605 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:48:12.605 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 20, 2024-08-19T14:48:12.606 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 600, 2024-08-19T14:48:12.606 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [ 2024-08-19T14:48:12.606 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.606 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.16962.0:199394 7.2 e690/635 7:406e87f5:::benchmark_data_smithi045_115842_object81234:head v 690'12614, mlcod=621'5122)", 2024-08-19T14:48:12.606 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:38:22.737244+0000", 2024-08-19T14:48:12.606 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 589.86640218800005, 2024-08-19T14:48:12.606 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.24429008799999999, 2024-08-19T14:48:12.606 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:48:12.606 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:48:12.606 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:48:12.606 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:48:12.606 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.606 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:48:12.607 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.737244+0000", 2024-08-19T14:48:12.607 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:48:12.607 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.607 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.607 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:48:12.607 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.737244+0000", 2024-08-19T14:48:12.607 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:48:12.607 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.607 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.607 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:48:12.607 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.737245+0000", 2024-08-19T14:48:12.607 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.6299999999999993e-07 2024-08-19T14:48:12.607 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.608 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.608 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:48:12.608 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.737251+0000", 2024-08-19T14:48:12.608 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.6930000000000001e-06 2024-08-19T14:48:12.608 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.608 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.608 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:48:12.608 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.737262+0000", 2024-08-19T14:48:12.608 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0922e-05 2024-08-19T14:48:12.608 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.608 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.608 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:48:12.608 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.737269+0000", 2024-08-19T14:48:12.608 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.6680000000000004e-06 2024-08-19T14:48:12.609 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.609 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.609 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:48:12.609 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.737648+0000", 2024-08-19T14:48:12.609 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00037954999999999998 2024-08-19T14:48:12.609 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.609 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.609 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:48:12.609 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.737671+0000", 2024-08-19T14:48:12.609 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.2784000000000001e-05 2024-08-19T14:48:12.609 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.609 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.609 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:48:12.609 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.981464+0000", 2024-08-19T14:48:12.610 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.243792642 2024-08-19T14:48:12.610 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.610 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.610 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:48:12.610 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.981534+0000", 2024-08-19T14:48:12.610 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.9865999999999998e-05 2024-08-19T14:48:12.610 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:48:12.610 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:48:12.610 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:48:12.610 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.610 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.610 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.16962.0:199392 7.2 e690/635 7:4580edf3:::benchmark_data_smithi045_115842_object81232:head v 690'12615, mlcod=621'5122)", 2024-08-19T14:48:12.610 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:38:22.746524+0000", 2024-08-19T14:48:12.611 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 589.85712222799998, 2024-08-19T14:48:12.611 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.23504549899999999, 2024-08-19T14:48:12.611 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:48:12.611 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:48:12.611 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:48:12.611 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:48:12.611 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.611 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:48:12.611 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.746524+0000", 2024-08-19T14:48:12.611 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:48:12.611 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.611 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.611 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:48:12.611 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.746524+0000", 2024-08-19T14:48:12.612 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:48:12.612 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.612 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.612 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:48:12.612 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.746525+0000", 2024-08-19T14:48:12.612 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0279999999999999e-06 2024-08-19T14:48:12.612 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.612 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.612 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:48:12.612 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.746533+0000", 2024-08-19T14:48:12.612 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.0069999999999997e-06 2024-08-19T14:48:12.612 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.612 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.613 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:48:12.613 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.746545+0000", 2024-08-19T14:48:12.613 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2584e-05 2024-08-19T14:48:12.613 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.613 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.613 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:48:12.613 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.746554+0000", 2024-08-19T14:48:12.613 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.0180000000000007e-06 2024-08-19T14:48:12.613 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.613 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.613 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:48:12.613 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.752313+0000", 2024-08-19T14:48:12.613 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0057586139999999996 2024-08-19T14:48:12.613 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.614 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.614 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:48:12.614 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.752334+0000", 2024-08-19T14:48:12.614 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.0735000000000001e-05 2024-08-19T14:48:12.614 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.614 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.614 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:48:12.614 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.981537+0000", 2024-08-19T14:48:12.614 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.22920327500000001 2024-08-19T14:48:12.614 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.614 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.614 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:48:12.614 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.981569+0000", 2024-08-19T14:48:12.614 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.2237999999999999e-05 2024-08-19T14:48:12.615 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:48:12.615 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:48:12.615 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:48:12.615 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.615 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.615 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.16962.0:199409 7.2 e690/635 7:4c8d089b:::benchmark_data_smithi045_115842_object81249:head v 690'12616, mlcod=621'5122)", 2024-08-19T14:48:12.615 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:38:22.750502+0000", 2024-08-19T14:48:12.615 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 589.85314403300004, 2024-08-19T14:48:12.615 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.255713405, 2024-08-19T14:48:12.615 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:48:12.615 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:48:12.615 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:48:12.615 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:48:12.615 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.616 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:48:12.616 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.750502+0000", 2024-08-19T14:48:12.616 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:48:12.616 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.616 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.616 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:48:12.616 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.750502+0000", 2024-08-19T14:48:12.616 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:48:12.616 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.616 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.616 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:48:12.616 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.750503+0000", 2024-08-19T14:48:12.616 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.09e-07 2024-08-19T14:48:12.616 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.617 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.617 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:48:12.617 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.750511+0000", 2024-08-19T14:48:12.617 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.9759999999999995e-06 2024-08-19T14:48:12.617 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.617 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.617 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:48:12.617 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.750522+0000", 2024-08-19T14:48:12.617 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1241e-05 2024-08-19T14:48:12.617 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.617 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.617 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:48:12.617 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.750529+0000", 2024-08-19T14:48:12.617 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.3629999999999996e-06 2024-08-19T14:48:12.617 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.618 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.618 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:48:12.618 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.765007+0000", 2024-08-19T14:48:12.618 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.014478058 2024-08-19T14:48:12.618 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.618 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.619 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:48:12.619 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.765023+0000", 2024-08-19T14:48:12.622 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5735000000000001e-05 2024-08-19T14:48:12.622 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.622 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.622 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:48:12.622 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:23.006112+0000", 2024-08-19T14:48:12.622 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.24108848799999999 2024-08-19T14:48:12.623 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.623 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.623 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:48:12.623 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:23.006215+0000", 2024-08-19T14:48:12.623 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00010363500000000001 2024-08-19T14:48:12.623 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:48:12.623 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:48:12.623 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:48:12.623 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.623 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.623 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.16962.0:199414 7.c e690/635 7:366d05c1:::benchmark_data_smithi045_115842_object81254:head v 690'12294, mlcod=621'5016)", 2024-08-19T14:48:12.623 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:38:22.751905+0000", 2024-08-19T14:48:12.623 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 589.85174064499995, 2024-08-19T14:48:12.624 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.20950311999999999, 2024-08-19T14:48:12.624 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:48:12.624 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:48:12.624 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:48:12.624 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:48:12.624 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.624 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:48:12.624 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.751905+0000", 2024-08-19T14:48:12.624 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:48:12.624 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.624 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.624 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:48:12.625 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.751905+0000", 2024-08-19T14:48:12.625 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:48:12.625 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.625 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.625 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:48:12.625 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.751906+0000", 2024-08-19T14:48:12.625 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.4000000000000001e-07 2024-08-19T14:48:12.625 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.625 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.625 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:48:12.625 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.751912+0000", 2024-08-19T14:48:12.625 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.0100000000000001e-06 2024-08-19T14:48:12.625 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.625 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.626 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:48:12.626 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.751923+0000", 2024-08-19T14:48:12.626 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.098e-05 2024-08-19T14:48:12.626 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.626 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.626 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:48:12.626 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.751930+0000", 2024-08-19T14:48:12.626 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.0600000000000002e-06 2024-08-19T14:48:12.626 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.626 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.626 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:48:12.626 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.752067+0000", 2024-08-19T14:48:12.626 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013742100000000001 2024-08-19T14:48:12.627 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.627 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.627 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:48:12.627 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.752089+0000", 2024-08-19T14:48:12.627 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.1389000000000001e-05 2024-08-19T14:48:12.627 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.627 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.627 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:48:12.627 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.766981+0000", 2024-08-19T14:48:12.627 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.014892396 2024-08-19T14:48:12.627 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.627 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.627 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:48:12.627 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.961408+0000", 2024-08-19T14:48:12.628 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.19442712400000001 2024-08-19T14:48:12.628 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:48:12.628 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:48:12.628 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:48:12.629 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.629 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.629 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.16962.0:199422 7.a e690/635 7:5ea5870d:::benchmark_data_smithi045_115842_object81262:head v 690'12384, mlcod=633'7611)", 2024-08-19T14:48:12.629 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:38:22.759906+0000", 2024-08-19T14:48:12.629 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 589.84373981700003, 2024-08-19T14:48:12.630 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.242679165, 2024-08-19T14:48:12.630 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:48:12.630 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:48:12.630 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:48:12.630 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:48:12.630 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.630 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:48:12.630 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.759906+0000", 2024-08-19T14:48:12.630 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:48:12.630 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.630 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.630 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:48:12.631 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.759906+0000", 2024-08-19T14:48:12.631 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:48:12.631 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.631 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.631 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:48:12.631 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.759908+0000", 2024-08-19T14:48:12.631 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5880000000000001e-06 2024-08-19T14:48:12.631 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.631 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.632 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:48:12.632 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.759916+0000", 2024-08-19T14:48:12.632 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.4950000000000008e-06 2024-08-19T14:48:12.632 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.632 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.633 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:48:12.633 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.759930+0000", 2024-08-19T14:48:12.633 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4236000000000001e-05 2024-08-19T14:48:12.633 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.633 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.633 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:48:12.634 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.759941+0000", 2024-08-19T14:48:12.634 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1052e-05 2024-08-19T14:48:12.634 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.634 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.634 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:48:12.635 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.765122+0000", 2024-08-19T14:48:12.635 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0051806439999999999 2024-08-19T14:48:12.635 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.635 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.635 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:48:12.635 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.765141+0000", 2024-08-19T14:48:12.635 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.8470000000000001e-05 2024-08-19T14:48:12.635 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.635 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.635 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:48:12.636 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:23.002541+0000", 2024-08-19T14:48:12.636 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.23740081599999999 2024-08-19T14:48:12.636 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.636 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.636 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:48:12.637 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:23.002585+0000", 2024-08-19T14:48:12.637 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.3863999999999999e-05 2024-08-19T14:48:12.637 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:48:12.637 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:48:12.637 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:48:12.637 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.637 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.637 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "MOSDPGPush(7.2 690/635 [PushOp(7:48749d7b:::benchmark_data_smithi045_115842_object81270:head, version: 621'5127, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:48749d7b:::benchmark_data_smithi045_115842_object81270:head@621'5127, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-19T14:48:12.637 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:38:22.764504+0000", 2024-08-19T14:48:12.637 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 589.83914165600004, 2024-08-19T14:48:12.637 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.20554665, 2024-08-19T14:48:12.637 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:48:12.638 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:48:12.638 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-08-19T14:48:12.638 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:48:12.638 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.638 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:48:12.638 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.764504+0000", 2024-08-19T14:48:12.638 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:48:12.638 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.639 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.639 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:48:12.639 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.764504+0000", 2024-08-19T14:48:12.639 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:48:12.639 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.639 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.639 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:48:12.639 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.764505+0000", 2024-08-19T14:48:12.639 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.6400000000000001e-07 2024-08-19T14:48:12.639 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.639 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.640 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:48:12.640 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.764536+0000", 2024-08-19T14:48:12.640 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.0719000000000002e-05 2024-08-19T14:48:12.640 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.640 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.640 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:48:12.640 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.764556+0000", 2024-08-19T14:48:12.640 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.0142000000000001e-05 2024-08-19T14:48:12.640 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.640 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.640 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:48:12.640 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.764573+0000", 2024-08-19T14:48:12.641 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6685999999999999e-05 2024-08-19T14:48:12.641 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.641 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.641 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:48:12.641 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.966878+0000", 2024-08-19T14:48:12.642 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.20230553300000001 2024-08-19T14:48:12.642 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.642 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.642 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:48:12.642 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.966888+0000", 2024-08-19T14:48:12.642 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.026e-05 2024-08-19T14:48:12.643 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.643 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.643 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:48:12.643 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.970051+0000", 2024-08-19T14:48:12.643 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.003162546 2024-08-19T14:48:12.643 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:48:12.644 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:48:12.644 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:48:12.644 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.644 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.644 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "MOSDPGRecoveryDelete(7.e e690,634 7:7b982ab6:::benchmark_data_smithi045_115842_object57353:head,673'11068)", 2024-08-19T14:48:12.644 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:38:22.767854+0000", 2024-08-19T14:48:12.644 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 589.83579185099995, 2024-08-19T14:48:12.646 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.19375594500000001, 2024-08-19T14:48:12.646 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:48:12.646 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:48:12.646 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-08-19T14:48:12.646 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:48:12.646 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.646 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:48:12.646 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.767854+0000", 2024-08-19T14:48:12.647 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:48:12.647 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.647 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.647 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:48:12.647 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.767854+0000", 2024-08-19T14:48:12.647 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:48:12.647 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.647 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.647 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:48:12.647 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.767855+0000", 2024-08-19T14:48:12.647 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2410000000000001e-06 2024-08-19T14:48:12.647 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.647 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.647 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:48:12.648 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.767872+0000", 2024-08-19T14:48:12.648 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6707000000000001e-05 2024-08-19T14:48:12.648 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.648 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.648 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:48:12.648 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.961428+0000", 2024-08-19T14:48:12.648 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.19355593600000001 2024-08-19T14:48:12.648 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.648 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.649 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:48:12.649 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.961441+0000", 2024-08-19T14:48:12.649 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3269e-05 2024-08-19T14:48:12.649 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.649 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.649 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:48:12.649 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.961526+0000", 2024-08-19T14:48:12.649 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.4942000000000001e-05 2024-08-19T14:48:12.649 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.649 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.649 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:48:12.649 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.961534+0000", 2024-08-19T14:48:12.649 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.6350000000000006e-06 2024-08-19T14:48:12.650 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.650 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.650 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:48:12.650 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.961610+0000", 2024-08-19T14:48:12.650 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.6215000000000001e-05 2024-08-19T14:48:12.650 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:48:12.650 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:48:12.651 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:48:12.651 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.651 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.651 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.16962.0:199399 7.3 e690/635 7:cecd84d1:::benchmark_data_smithi045_115842_object81239:head v 690'12702, mlcod=621'5174)", 2024-08-19T14:48:12.651 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:38:22.776198+0000", 2024-08-19T14:48:12.651 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 589.82744798500005, 2024-08-19T14:48:12.651 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.20653019, 2024-08-19T14:48:12.651 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:48:12.651 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:48:12.651 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:48:12.651 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:48:12.651 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.651 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:48:12.651 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.776198+0000", 2024-08-19T14:48:12.652 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:48:12.652 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.652 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.652 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:48:12.652 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.776198+0000", 2024-08-19T14:48:12.652 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:48:12.652 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.652 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.652 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:48:12.652 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.776199+0000", 2024-08-19T14:48:12.652 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2449999999999999e-06 2024-08-19T14:48:12.652 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.652 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.652 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:48:12.652 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.776210+0000", 2024-08-19T14:48:12.653 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1289e-05 2024-08-19T14:48:12.653 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.653 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.653 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:48:12.653 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.961366+0000", 2024-08-19T14:48:12.653 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.18515516700000001 2024-08-19T14:48:12.653 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.653 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.653 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:48:12.653 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.961387+0000", 2024-08-19T14:48:12.653 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.1248999999999999e-05 2024-08-19T14:48:12.653 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.653 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.653 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:48:12.654 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.961448+0000", 2024-08-19T14:48:12.654 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.1062000000000006e-05 2024-08-19T14:48:12.654 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.654 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.654 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:48:12.654 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.961480+0000", 2024-08-19T14:48:12.654 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.1775e-05 2024-08-19T14:48:12.654 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.654 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.655 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:48:12.655 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.982684+0000", 2024-08-19T14:48:12.655 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.021203914000000001 2024-08-19T14:48:12.655 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.655 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.655 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:48:12.655 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:38:22.982728+0000", 2024-08-19T14:48:12.656 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4489e-05 2024-08-19T14:48:12.656 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:48:12.656 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:48:12.656 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:48:12.656 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.656 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.656 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "MOSDPGPush(8.12 754/734 [PushOp(8:4b2adb7e:::benchmark_data_smithi045_137103_object65588:head, version: 730'4056, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:4b2adb7e:::benchmark_data_smithi045_137103_object65588:head@730'4056, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-19T14:48:12.656 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:41:28.711325+0000", 2024-08-19T14:48:12.657 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 403.89232076399998, 2024-08-19T14:48:12.657 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.142688281, 2024-08-19T14:48:12.657 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:48:12.657 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:48:12.657 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-08-19T14:48:12.657 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:48:12.657 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.658 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:48:12.658 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:28.711325+0000", 2024-08-19T14:48:12.658 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:48:12.658 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.658 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.658 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:48:12.659 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:28.711325+0000", 2024-08-19T14:48:12.659 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:48:12.659 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.659 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.659 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:48:12.659 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:28.711326+0000", 2024-08-19T14:48:12.659 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.95e-07 2024-08-19T14:48:12.660 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.660 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.660 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:48:12.660 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:28.711359+0000", 2024-08-19T14:48:12.660 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.3250000000000002e-05 2024-08-19T14:48:12.660 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.661 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.661 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:48:12.661 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:28.848011+0000", 2024-08-19T14:48:12.661 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.13665176800000001 2024-08-19T14:48:12.661 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.661 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.662 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:48:12.662 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:28.848519+0000", 2024-08-19T14:48:12.662 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00050756200000000003 2024-08-19T14:48:12.662 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.662 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.662 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:48:12.663 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:28.851404+0000", 2024-08-19T14:48:12.663 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.002885454 2024-08-19T14:48:12.663 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.663 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.663 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:48:12.663 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:28.851445+0000", 2024-08-19T14:48:12.663 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.0571999999999997e-05 2024-08-19T14:48:12.663 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.663 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.663 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:48:12.663 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:28.854013+0000", 2024-08-19T14:48:12.663 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0025686799999999998 2024-08-19T14:48:12.663 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:48:12.664 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:48:12.664 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:48:12.664 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.664 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.664 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.26911.0:112730 8.e e763/717 8:7b021f56:::benchmark_data_smithi045_137103_object266:head v 763'7120, mlcod=763'7118)", 2024-08-19T14:48:12.664 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:41:42.489992+0000", 2024-08-19T14:48:12.664 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 390.11365356099998, 2024-08-19T14:48:12.664 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.14127273100000001, 2024-08-19T14:48:12.664 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:48:12.664 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:48:12.664 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:48:12.664 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:48:12.664 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.664 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:48:12.665 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:42.489992+0000", 2024-08-19T14:48:12.665 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:48:12.665 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.665 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.665 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:48:12.665 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:42.489992+0000", 2024-08-19T14:48:12.665 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:48:12.665 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.665 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.665 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:48:12.665 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:42.489993+0000", 2024-08-19T14:48:12.665 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.5499999999999996e-07 2024-08-19T14:48:12.665 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.666 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.666 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:48:12.666 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:42.490000+0000", 2024-08-19T14:48:12.666 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.8040000000000004e-06 2024-08-19T14:48:12.666 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.666 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.666 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:48:12.666 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:42.490011+0000", 2024-08-19T14:48:12.666 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.064e-05 2024-08-19T14:48:12.666 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.666 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.666 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:48:12.666 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:42.490018+0000", 2024-08-19T14:48:12.667 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.2089999999999999e-06 2024-08-19T14:48:12.667 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.667 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.667 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:48:12.667 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:42.490071+0000", 2024-08-19T14:48:12.667 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.346e-05 2024-08-19T14:48:12.667 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.667 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.667 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:48:12.667 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:42.490091+0000", 2024-08-19T14:48:12.667 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.9123e-05 2024-08-19T14:48:12.667 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.667 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.667 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:48:12.668 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:42.631226+0000", 2024-08-19T14:48:12.668 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.141135608 2024-08-19T14:48:12.668 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.668 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.668 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:48:12.668 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:42.631265+0000", 2024-08-19T14:48:12.668 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.8931999999999999e-05 2024-08-19T14:48:12.668 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:48:12.668 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:48:12.668 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:48:12.668 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.668 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.668 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.26911.0:112731 8.f e763/717 8:ff6e92ec:::benchmark_data_smithi045_137103_object267:head v 763'7064, mlcod=763'7062)", 2024-08-19T14:48:12.669 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:41:42.490155+0000", 2024-08-19T14:48:12.669 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 390.113491033, 2024-08-19T14:48:12.669 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.14124139899999999, 2024-08-19T14:48:12.669 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:48:12.669 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:48:12.669 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:48:12.669 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:48:12.669 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.669 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:48:12.669 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:42.490155+0000", 2024-08-19T14:48:12.669 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:48:12.669 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.669 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.669 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:48:12.670 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:42.490155+0000", 2024-08-19T14:48:12.670 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:48:12.670 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.670 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.670 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:48:12.670 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:42.490155+0000", 2024-08-19T14:48:12.670 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.1399999999999997e-07 2024-08-19T14:48:12.670 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.670 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.670 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:48:12.670 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:42.490161+0000", 2024-08-19T14:48:12.670 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0839999999999999e-06 2024-08-19T14:48:12.671 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.671 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.671 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:48:12.671 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:42.490170+0000", 2024-08-19T14:48:12.671 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.1349999999999998e-06 2024-08-19T14:48:12.671 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.671 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.671 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:48:12.671 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:42.490175+0000", 2024-08-19T14:48:12.671 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.587e-06 2024-08-19T14:48:12.671 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.671 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.671 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:48:12.671 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:42.490225+0000", 2024-08-19T14:48:12.672 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.9937e-05 2024-08-19T14:48:12.672 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.672 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.672 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:48:12.672 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:42.490244+0000", 2024-08-19T14:48:12.672 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.9094e-05 2024-08-19T14:48:12.672 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.672 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.672 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:48:12.672 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:42.631352+0000", 2024-08-19T14:48:12.672 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.14110751199999999 2024-08-19T14:48:12.672 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.672 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.672 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:48:12.673 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:42.631396+0000", 2024-08-19T14:48:12.673 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4435999999999999e-05 2024-08-19T14:48:12.673 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:48:12.673 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:48:12.673 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:48:12.673 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.673 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.674 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.26911.0:112733 8.d e763/717 8:b1f595a5:::benchmark_data_smithi045_137103_object269:head v 763'7096, mlcod=763'7094)", 2024-08-19T14:48:12.674 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:41:42.493271+0000", 2024-08-19T14:48:12.674 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 390.11037535600002, 2024-08-19T14:48:12.674 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.14344847499999999, 2024-08-19T14:48:12.674 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:48:12.674 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:48:12.674 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:48:12.674 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:48:12.674 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.674 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:48:12.674 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:42.493271+0000", 2024-08-19T14:48:12.674 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:48:12.674 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.675 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.675 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:48:12.675 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:42.493271+0000", 2024-08-19T14:48:12.675 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:48:12.675 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.675 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.675 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:48:12.675 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:42.493271+0000", 2024-08-19T14:48:12.675 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.5799999999999999e-07 2024-08-19T14:48:12.675 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.675 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.675 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:48:12.676 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:42.493277+0000", 2024-08-19T14:48:12.676 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.3890000000000004e-06 2024-08-19T14:48:12.676 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.676 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.676 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:48:12.676 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:42.493285+0000", 2024-08-19T14:48:12.676 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.3939999999999996e-06 2024-08-19T14:48:12.676 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.676 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.676 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:48:12.676 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:42.493292+0000", 2024-08-19T14:48:12.676 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.596e-06 2024-08-19T14:48:12.677 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.677 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.677 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:48:12.677 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:42.631429+0000", 2024-08-19T14:48:12.677 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.13813727100000001 2024-08-19T14:48:12.677 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.677 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.677 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:48:12.677 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:42.631449+0000", 2024-08-19T14:48:12.677 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.0353e-05 2024-08-19T14:48:12.677 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.677 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.677 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:48:12.677 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:42.636691+0000", 2024-08-19T14:48:12.678 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0052417139999999997 2024-08-19T14:48:12.678 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.678 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.678 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:48:12.678 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:41:42.636719+0000", 2024-08-19T14:48:12.678 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.8099999999999999e-05 2024-08-19T14:48:12.678 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:48:12.678 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:48:12.678 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:48:12.678 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.678 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.678 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.27235.0:70472 9.1 9:8eb9e7f2:::benchmark_data_smithi045_152862_object70471:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e864)", 2024-08-19T14:48:12.678 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:45:24.130851+0000", 2024-08-19T14:48:12.678 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 168.472794841, 2024-08-19T14:48:12.679 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.42378770399999999, 2024-08-19T14:48:12.679 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:48:12.679 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:48:12.679 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:48:12.679 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:48:12.679 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.27235", 2024-08-19T14:48:12.679 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2746512116", 2024-08-19T14:48:12.679 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 70472 2024-08-19T14:48:12.679 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.679 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:48:12.679 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.679 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:48:12.679 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.130851+0000", 2024-08-19T14:48:12.680 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:48:12.680 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.680 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.680 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:48:12.680 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.130851+0000", 2024-08-19T14:48:12.680 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:48:12.680 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.680 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.680 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:48:12.680 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.130853+0000", 2024-08-19T14:48:12.680 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.759e-06 2024-08-19T14:48:12.680 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.680 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.681 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:48:12.681 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.131053+0000", 2024-08-19T14:48:12.681 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000200118 2024-08-19T14:48:12.681 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.681 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.681 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:48:12.681 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.131079+0000", 2024-08-19T14:48:12.681 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.5715e-05 2024-08-19T14:48:12.681 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.681 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.681 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:48:12.681 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.131092+0000", 2024-08-19T14:48:12.681 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2952999999999999e-05 2024-08-19T14:48:12.682 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.682 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.682 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:48:12.682 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.132073+0000", 2024-08-19T14:48:12.682 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0009816829999999999 2024-08-19T14:48:12.682 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.682 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.682 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:48:12.682 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.132084+0000", 2024-08-19T14:48:12.682 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0973e-05 2024-08-19T14:48:12.682 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.682 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.683 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:48:12.683 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.553167+0000", 2024-08-19T14:48:12.683 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.42108286900000003 2024-08-19T14:48:12.683 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.683 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.683 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:48:12.683 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.554639+0000", 2024-08-19T14:48:12.683 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0014716340000000001 2024-08-19T14:48:12.683 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:48:12.683 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:48:12.683 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:48:12.683 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.683 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.684 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.27235.0:70476 9.1 9:81ce3d4f:::benchmark_data_smithi045_152862_object70475:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e864)", 2024-08-19T14:48:12.684 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:45:24.131125+0000", 2024-08-19T14:48:12.684 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 168.472521234, 2024-08-19T14:48:12.684 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.42668378600000001, 2024-08-19T14:48:12.684 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:48:12.684 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:48:12.684 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:48:12.684 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:48:12.684 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.27235", 2024-08-19T14:48:12.684 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2746512116", 2024-08-19T14:48:12.685 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 70476 2024-08-19T14:48:12.685 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.685 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:48:12.685 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.685 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:48:12.685 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.131125+0000", 2024-08-19T14:48:12.685 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:48:12.685 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.685 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.685 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:48:12.685 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.131125+0000", 2024-08-19T14:48:12.685 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:48:12.685 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.685 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.686 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:48:12.686 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.131125+0000", 2024-08-19T14:48:12.686 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.99e-07 2024-08-19T14:48:12.686 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.686 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.686 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:48:12.686 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.131196+0000", 2024-08-19T14:48:12.686 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.038e-05 2024-08-19T14:48:12.686 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.686 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.686 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:48:12.686 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.131206+0000", 2024-08-19T14:48:12.686 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.049e-05 2024-08-19T14:48:12.687 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.687 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.687 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:48:12.687 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.131213+0000", 2024-08-19T14:48:12.687 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.1010000000000003e-06 2024-08-19T14:48:12.687 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.687 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.687 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:48:12.687 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.132119+0000", 2024-08-19T14:48:12.687 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00090598899999999995 2024-08-19T14:48:12.687 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.687 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.687 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:48:12.687 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.132127+0000", 2024-08-19T14:48:12.688 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.2980000000000003e-06 2024-08-19T14:48:12.688 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.688 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.688 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:48:12.688 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.557684+0000", 2024-08-19T14:48:12.688 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.42555691099999998 2024-08-19T14:48:12.688 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.688 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.688 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:48:12.688 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.557808+0000", 2024-08-19T14:48:12.689 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012491800000000001 2024-08-19T14:48:12.689 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:48:12.689 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:48:12.689 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:48:12.689 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.689 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.689 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.27235.0:70485 9.1 9:80f87f5d:::benchmark_data_smithi045_152862_object70484:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e864)", 2024-08-19T14:48:12.689 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:45:24.131225+0000", 2024-08-19T14:48:12.689 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 168.47242111200001, 2024-08-19T14:48:12.689 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.42675184500000002, 2024-08-19T14:48:12.689 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:48:12.689 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:48:12.690 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-08-19T14:48:12.690 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:48:12.690 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.27235", 2024-08-19T14:48:12.690 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2746512116", 2024-08-19T14:48:12.690 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 70485 2024-08-19T14:48:12.690 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.690 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:48:12.690 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.690 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:48:12.690 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.131225+0000", 2024-08-19T14:48:12.690 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:48:12.690 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.691 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.691 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:48:12.691 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.131225+0000", 2024-08-19T14:48:12.691 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:48:12.691 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.691 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.691 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:48:12.691 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.131225+0000", 2024-08-19T14:48:12.691 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.5499999999999998e-07 2024-08-19T14:48:12.691 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.691 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.691 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:48:12.691 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.131290+0000", 2024-08-19T14:48:12.691 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.5104999999999994e-05 2024-08-19T14:48:12.692 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.692 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.692 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:48:12.692 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.131300+0000", 2024-08-19T14:48:12.692 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.5079999999999994e-06 2024-08-19T14:48:12.692 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.692 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.693 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:48:12.693 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.131306+0000", 2024-08-19T14:48:12.693 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.5320000000000003e-06 2024-08-19T14:48:12.693 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.693 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.693 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:48:12.693 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.132152+0000", 2024-08-19T14:48:12.693 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00084525799999999999 2024-08-19T14:48:12.694 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.694 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.694 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for peered", 2024-08-19T14:48:12.694 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.132159+0000", 2024-08-19T14:48:12.694 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.7489999999999998e-06 2024-08-19T14:48:12.694 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.694 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.695 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:48:12.695 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.557856+0000", 2024-08-19T14:48:12.695 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.42569648100000002 2024-08-19T14:48:12.695 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.695 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.695 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:48:12.696 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:45:24.557977+0000", 2024-08-19T14:48:12.696 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012065699999999999 2024-08-19T14:48:12.696 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:48:12.696 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:48:12.696 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:48:12.696 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.696 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.696 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.27235.0:110976 9.1 9:8b7e4231:::benchmark_data_smithi045_152862_object16471:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e874)", 2024-08-19T14:48:12.696 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:46:11.284639+0000", 2024-08-19T14:48:12.696 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 121.319007236, 2024-08-19T14:48:12.696 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.138447722, 2024-08-19T14:48:12.696 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:48:12.696 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:48:12.697 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:48:12.697 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:48:12.697 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.27235", 2024-08-19T14:48:12.697 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2746512116", 2024-08-19T14:48:12.697 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 110976 2024-08-19T14:48:12.697 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.697 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:48:12.697 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.697 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:48:12.697 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:11.284639+0000", 2024-08-19T14:48:12.697 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:48:12.697 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.697 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.697 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:48:12.698 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:11.284639+0000", 2024-08-19T14:48:12.698 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:48:12.698 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.698 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.698 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:48:12.698 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:11.284639+0000", 2024-08-19T14:48:12.698 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.06e-07 2024-08-19T14:48:12.698 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.698 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.699 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:48:12.699 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:11.284643+0000", 2024-08-19T14:48:12.699 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.4230000000000001e-06 2024-08-19T14:48:12.699 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.699 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.699 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:48:12.699 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:11.284653+0000", 2024-08-19T14:48:12.700 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0635000000000001e-05 2024-08-19T14:48:12.700 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.700 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.700 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:48:12.700 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:11.284660+0000", 2024-08-19T14:48:12.700 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.4659999999999996e-06 2024-08-19T14:48:12.700 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.700 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.700 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:48:12.700 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:11.289766+0000", 2024-08-19T14:48:12.700 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0051067250000000003 2024-08-19T14:48:12.700 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.700 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.701 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:48:12.701 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:11.289870+0000", 2024-08-19T14:48:12.701 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000103631 2024-08-19T14:48:12.701 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.701 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.701 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 6,9,10", 2024-08-19T14:48:12.701 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:11.290005+0000", 2024-08-19T14:48:12.701 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013542200000000001 2024-08-19T14:48:12.701 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.701 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.702 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:48:12.702 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:11.293423+0000", 2024-08-19T14:48:12.702 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0034170649999999999 2024-08-19T14:48:12.702 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.702 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.702 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:48:12.702 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:11.297158+0000", 2024-08-19T14:48:12.702 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0037354659999999998 2024-08-19T14:48:12.702 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.702 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.703 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:48:12.703 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:11.313652+0000", 2024-08-19T14:48:12.703 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.016494194 2024-08-19T14:48:12.703 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.703 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.703 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:48:12.703 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:11.422312+0000", 2024-08-19T14:48:12.703 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.108659849 2024-08-19T14:48:12.703 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.703 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.703 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:48:12.703 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:11.422372+0000", 2024-08-19T14:48:12.703 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.9648000000000002e-05 2024-08-19T14:48:12.703 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.704 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.704 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:48:12.704 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:11.423086+0000", 2024-08-19T14:48:12.704 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00071469199999999995 2024-08-19T14:48:12.704 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:48:12.705 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:48:12.705 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:48:12.705 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.705 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.705 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.27235.0:110987 9.1 9:862d1f71:::benchmark_data_smithi045_152862_object16482:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e874)", 2024-08-19T14:48:12.705 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:46:11.299792+0000", 2024-08-19T14:48:12.706 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 121.30385402900001, 2024-08-19T14:48:12.706 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.123817388, 2024-08-19T14:48:12.706 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:48:12.706 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:48:12.706 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:48:12.707 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-08-19T14:48:12.707 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.27235", 2024-08-19T14:48:12.707 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.45:0/2746512116", 2024-08-19T14:48:12.707 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 110987 2024-08-19T14:48:12.707 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.708 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:48:12.708 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.708 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:48:12.708 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:11.299792+0000", 2024-08-19T14:48:12.708 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:48:12.708 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.709 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.709 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:48:12.709 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:11.299792+0000", 2024-08-19T14:48:12.709 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:48:12.709 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.709 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.709 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:48:12.709 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:11.299793+0000", 2024-08-19T14:48:12.709 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.485e-06 2024-08-19T14:48:12.710 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.710 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.710 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:48:12.710 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:11.299799+0000", 2024-08-19T14:48:12.710 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.8699999999999997e-06 2024-08-19T14:48:12.710 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.710 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.710 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:48:12.710 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:11.299813+0000", 2024-08-19T14:48:12.710 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4132e-05 2024-08-19T14:48:12.711 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.711 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.711 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:48:12.711 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:11.299821+0000", 2024-08-19T14:48:12.711 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.0800000000000006e-06 2024-08-19T14:48:12.711 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.711 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.711 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:48:12.711 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:11.300385+0000", 2024-08-19T14:48:12.712 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00056395800000000004 2024-08-19T14:48:12.712 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.712 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.712 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:48:12.712 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:11.300477+0000", 2024-08-19T14:48:12.712 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.1510999999999998e-05 2024-08-19T14:48:12.712 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.712 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.712 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 6,9,10", 2024-08-19T14:48:12.712 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:11.300559+0000", 2024-08-19T14:48:12.712 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.1873999999999996e-05 2024-08-19T14:48:12.713 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.713 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.713 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-08-19T14:48:12.713 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:11.306971+0000", 2024-08-19T14:48:12.713 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0064118789999999997 2024-08-19T14:48:12.713 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.713 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.713 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:48:12.713 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:11.306996+0000", 2024-08-19T14:48:12.713 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.5369999999999999e-05 2024-08-19T14:48:12.714 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.714 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.714 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:48:12.714 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:11.320033+0000", 2024-08-19T14:48:12.714 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.013036785 2024-08-19T14:48:12.714 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.714 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.714 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-08-19T14:48:12.714 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:11.423130+0000", 2024-08-19T14:48:12.714 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.103097018 2024-08-19T14:48:12.715 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.715 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.715 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:48:12.715 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:11.423167+0000", 2024-08-19T14:48:12.715 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.6844999999999997e-05 2024-08-19T14:48:12.715 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.715 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.715 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:48:12.715 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:46:11.423609+0000", 2024-08-19T14:48:12.715 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000442581 2024-08-19T14:48:12.715 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:48:12.716 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:48:12.716 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:48:12.716 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.716 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.716 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.27235.0:188101 9.7 e926/915 9:ea14a9ae:::benchmark_data_smithi045_152862_object93596:head v 926'11607, mlcod=871'6174)", 2024-08-19T14:48:12.716 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:48:01.888888+0000", 2024-08-19T14:48:12.716 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 10.714757906999999, 2024-08-19T14:48:12.716 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.014730765, 2024-08-19T14:48:12.716 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:48:12.716 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:48:12.717 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:48:12.717 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:48:12.717 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.717 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:48:12.717 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:48:01.888888+0000", 2024-08-19T14:48:12.717 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:48:12.717 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.717 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.717 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:48:12.717 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:48:01.888888+0000", 2024-08-19T14:48:12.717 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:48:12.717 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.718 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.718 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:48:12.718 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:48:01.888889+0000", 2024-08-19T14:48:12.718 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4950000000000001e-06 2024-08-19T14:48:12.718 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.719 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.719 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:48:12.719 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:48:01.888899+0000", 2024-08-19T14:48:12.719 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.5380000000000008e-06 2024-08-19T14:48:12.719 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.719 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.720 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:48:12.720 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:48:01.888921+0000", 2024-08-19T14:48:12.720 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.1695000000000002e-05 2024-08-19T14:48:12.720 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.720 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.721 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:48:12.721 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:48:01.888930+0000", 2024-08-19T14:48:12.721 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.1940000000000004e-06 2024-08-19T14:48:12.721 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.721 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.722 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:48:12.722 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:48:01.889016+0000", 2024-08-19T14:48:12.722 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.5811999999999998e-05 2024-08-19T14:48:12.722 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.722 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.722 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:48:12.723 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:48:01.889047+0000", 2024-08-19T14:48:12.723 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.095e-05 2024-08-19T14:48:12.723 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.723 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.723 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:48:12.724 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:48:01.903544+0000", 2024-08-19T14:48:12.724 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.014497200999999999 2024-08-19T14:48:12.724 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.724 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.724 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:48:12.724 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:48:01.903619+0000", 2024-08-19T14:48:12.725 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.4880000000000001e-05 2024-08-19T14:48:12.725 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:48:12.725 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:48:12.725 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:48:12.725 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.726 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.726 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.27235.0:188223 9.7 e926/915 9:e016dc80:::benchmark_data_smithi045_152862_object93718:head v 926'11623, mlcod=871'6178)", 2024-08-19T14:48:12.726 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:48:02.022400+0000", 2024-08-19T14:48:12.726 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 10.581246215, 2024-08-19T14:48:12.726 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.014619349, 2024-08-19T14:48:12.727 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:48:12.727 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:48:12.727 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:48:12.727 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:48:12.727 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.727 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:48:12.728 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:48:02.022400+0000", 2024-08-19T14:48:12.728 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:48:12.728 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.728 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.728 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:48:12.729 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:48:02.022400+0000", 2024-08-19T14:48:12.729 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:48:12.729 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.729 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.729 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:48:12.729 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:48:02.022401+0000", 2024-08-19T14:48:12.729 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1519999999999999e-06 2024-08-19T14:48:12.729 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.729 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.730 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:48:12.730 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:48:02.022409+0000", 2024-08-19T14:48:12.730 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.2910000000000007e-06 2024-08-19T14:48:12.730 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.730 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.730 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:48:12.731 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:48:02.022425+0000", 2024-08-19T14:48:12.731 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5639000000000001e-05 2024-08-19T14:48:12.731 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.731 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.731 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:48:12.732 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:48:02.022432+0000", 2024-08-19T14:48:12.732 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.6399999999999997e-06 2024-08-19T14:48:12.732 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.732 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.732 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:48:12.732 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:48:02.022491+0000", 2024-08-19T14:48:12.732 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.8619999999999998e-05 2024-08-19T14:48:12.732 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.732 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.732 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:48:12.732 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:48:02.022518+0000", 2024-08-19T14:48:12.732 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.6588999999999999e-05 2024-08-19T14:48:12.733 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.733 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.733 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:48:12.733 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:48:02.036981+0000", 2024-08-19T14:48:12.733 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.014463202 2024-08-19T14:48:12.733 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.733 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.733 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:48:12.733 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:48:02.037019+0000", 2024-08-19T14:48:12.734 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.8216000000000001e-05 2024-08-19T14:48:12.734 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:48:12.734 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:48:12.734 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:48:12.734 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.734 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.734 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.27235.0:188328 9.7 e926/915 9:ea99249e:::benchmark_data_smithi045_152862_object93823:head v 926'11633, mlcod=871'6182)", 2024-08-19T14:48:12.734 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-08-19T14:48:02.125104+0000", 2024-08-19T14:48:12.735 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 10.478541689, 2024-08-19T14:48:12.735 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.014675137, 2024-08-19T14:48:12.735 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-08-19T14:48:12.735 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-08-19T14:48:12.735 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-19T14:48:12.735 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-08-19T14:48:12.735 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.735 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-08-19T14:48:12.735 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:48:02.125104+0000", 2024-08-19T14:48:12.735 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:48:12.735 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.735 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.735 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-08-19T14:48:12.735 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:48:02.125104+0000", 2024-08-19T14:48:12.736 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-08-19T14:48:12.736 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.736 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.736 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-08-19T14:48:12.736 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:48:02.125105+0000", 2024-08-19T14:48:12.736 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.238e-06 2024-08-19T14:48:12.736 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.736 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.736 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-08-19T14:48:12.736 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:48:02.125116+0000", 2024-08-19T14:48:12.736 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0341e-05 2024-08-19T14:48:12.736 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.737 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.737 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-08-19T14:48:12.737 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:48:02.125138+0000", 2024-08-19T14:48:12.737 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.2359999999999999e-05 2024-08-19T14:48:12.737 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.737 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.737 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-08-19T14:48:12.737 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:48:02.125148+0000", 2024-08-19T14:48:12.737 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.6809999999999993e-06 2024-08-19T14:48:12.737 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.737 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.737 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-08-19T14:48:12.737 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:48:02.125229+0000", 2024-08-19T14:48:12.738 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.1450000000000001e-05 2024-08-19T14:48:12.738 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.738 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.738 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-08-19T14:48:12.738 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:48:02.125259+0000", 2024-08-19T14:48:12.738 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.9944999999999999e-05 2024-08-19T14:48:12.738 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.738 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.738 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-08-19T14:48:12.738 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:48:02.139723+0000", 2024-08-19T14:48:12.738 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.014464048 2024-08-19T14:48:12.739 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-08-19T14:48:12.739 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-08-19T14:48:12.739 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-08-19T14:48:12.739 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-08-19T14:48:02.139779+0000", 2024-08-19T14:48:12.739 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.6073999999999999e-05 2024-08-19T14:48:12.739 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:48:12.739 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:48:12.739 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:48:12.739 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-08-19T14:48:12.740 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-08-19T14:48:12.740 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:48:12.740 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:48:12.766 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:48:12.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:12.767 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:48:12.868 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:48:12.891 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:48:12.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:12.892 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:48:12.993 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5 2024-08-19T14:48:13.019 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:48:13.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:13.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.5' 2024-08-19T14:48:13.121 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:48:13.147 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:48:13.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:13.148 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:48:13.233 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is set 2024-08-19T14:48:13.249 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:48:13.274 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:48:13.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:13.275 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:48:13.376 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:48:13.399 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:48:13.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:13.400 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:48:13.502 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:48:13.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:13 smithi032 ceph-mon[20004]: 1.19 scrub starts 2024-08-19T14:48:13.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:13 smithi032 ceph-mon[20004]: 1.19 scrub ok 2024-08-19T14:48:13.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:13 smithi032 ceph-mon[20004]: 1.12 deep-scrub starts 2024-08-19T14:48:13.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:13 smithi032 ceph-mon[20004]: 1.12 deep-scrub ok 2024-08-19T14:48:13.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:13 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2380807967' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:48:13.509 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:13 smithi032 ceph-mon[20004]: Health check update: Degraded data redundancy: 25/6 objects degraded (416.667%), 1 pg degraded (PG_DEGRADED) 2024-08-19T14:48:13.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:13.527 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:48:13.527 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:48:13.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:13 smithi045 ceph-mon[20143]: 1.19 scrub starts 2024-08-19T14:48:13.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:13 smithi045 ceph-mon[20143]: 1.19 scrub ok 2024-08-19T14:48:13.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:13 smithi045 ceph-mon[20143]: 1.12 deep-scrub starts 2024-08-19T14:48:13.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:13 smithi045 ceph-mon[20143]: 1.12 deep-scrub ok 2024-08-19T14:48:13.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:13 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2380807967' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:48:13.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:13 smithi045 ceph-mon[20143]: Health check update: Degraded data redundancy: 25/6 objects degraded (416.667%), 1 pg degraded (PG_DEGRADED) 2024-08-19T14:48:13.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:13 smithi031 ceph-mon[16336]: 1.19 scrub starts 2024-08-19T14:48:13.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:13 smithi031 ceph-mon[16336]: 1.19 scrub ok 2024-08-19T14:48:13.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:13 smithi031 ceph-mon[16336]: 1.12 deep-scrub starts 2024-08-19T14:48:13.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:13 smithi031 ceph-mon[16336]: 1.12 deep-scrub ok 2024-08-19T14:48:13.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:13 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2380807967' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-08-19T14:48:13.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:13 smithi031 ceph-mon[16336]: Health check update: Degraded data redundancy: 25/6 objects degraded (416.667%), 1 pg degraded (PG_DEGRADED) 2024-08-19T14:48:13.628 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:48:13.651 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:48:13.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:13.652 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:48:13.753 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9 2024-08-19T14:48:13.779 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:48:13.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:13.780 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.9' 2024-08-19T14:48:13.881 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2 2024-08-19T14:48:13.903 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:48:13.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:13.904 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.2' 2024-08-19T14:48:14.005 DEBUG:teuthology.orchestra.run.smithi031:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3 2024-08-19T14:48:14.027 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: docker: command not found 2024-08-19T14:48:14.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:14.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi031 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.3' 2024-08-19T14:48:14.081 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-08-19T14:48:14.129 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:48:14.154 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:48:14.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:14.155 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:48:14.256 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4 2024-08-19T14:48:14.281 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:48:14.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:14.282 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.4' 2024-08-19T14:48:14.307 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:14 smithi031 ceph-mon[16336]: pgmap v2177: 31 pgs: 31 active+clean; 629 KiB data, 9.6 GiB used, 884 GiB / 894 GiB avail 2024-08-19T14:48:14.308 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:14 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:48:14.308 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:14 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:48:14.308 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:14 smithi031 ceph-mon[16336]: 1.1a scrub starts 2024-08-19T14:48:14.308 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:14 smithi031 ceph-mon[16336]: 1.1a scrub ok 2024-08-19T14:48:14.308 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:14 smithi031 ceph-mon[16336]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 25/6 objects degraded (416.667%), 1 pg degraded) 2024-08-19T14:48:14.308 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:14 smithi031 ceph-mon[16336]: Cluster is now healthy 2024-08-19T14:48:14.308 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:14 smithi031 ceph-mon[16336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:48:14.308 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:14 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2380807967' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:48:14.308 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:14 smithi031 ceph-mon[16336]: osdmap e929: 12 total, 12 up, 10 in 2024-08-19T14:48:14.308 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:14 smithi031 ceph-mon[16336]: 1.2 scrub starts 2024-08-19T14:48:14.383 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:48:14.409 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:48:14.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:14.410 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:48:14.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:14 smithi032 ceph-mon[20004]: pgmap v2177: 31 pgs: 31 active+clean; 629 KiB data, 9.6 GiB used, 884 GiB / 894 GiB avail 2024-08-19T14:48:14.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:14 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:48:14.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:14 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:48:14.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:14 smithi032 ceph-mon[20004]: 1.1a scrub starts 2024-08-19T14:48:14.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:14 smithi032 ceph-mon[20004]: 1.1a scrub ok 2024-08-19T14:48:14.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:14 smithi032 ceph-mon[20004]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 25/6 objects degraded (416.667%), 1 pg degraded) 2024-08-19T14:48:14.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:14 smithi032 ceph-mon[20004]: Cluster is now healthy 2024-08-19T14:48:14.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:14 smithi032 ceph-mon[20004]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:48:14.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:14 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2380807967' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:48:14.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:14 smithi032 ceph-mon[20004]: osdmap e929: 12 total, 12 up, 10 in 2024-08-19T14:48:14.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:14 smithi032 ceph-mon[20004]: 1.2 scrub starts 2024-08-19T14:48:14.511 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11 2024-08-19T14:48:14.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:14.536 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:48:14.536 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.11' 2024-08-19T14:48:14.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:14 smithi045 ceph-mon[20143]: pgmap v2177: 31 pgs: 31 active+clean; 629 KiB data, 9.6 GiB used, 884 GiB / 894 GiB avail 2024-08-19T14:48:14.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:14 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:48:14.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:14 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:48:14.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:14 smithi045 ceph-mon[20143]: 1.1a scrub starts 2024-08-19T14:48:14.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:14 smithi045 ceph-mon[20143]: 1.1a scrub ok 2024-08-19T14:48:14.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:14 smithi045 ceph-mon[20143]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 25/6 objects degraded (416.667%), 1 pg degraded) 2024-08-19T14:48:14.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:14 smithi045 ceph-mon[20143]: Cluster is now healthy 2024-08-19T14:48:14.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:14 smithi045 ceph-mon[20143]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-19T14:48:14.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:14 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2380807967' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-19T14:48:14.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:14 smithi045 ceph-mon[20143]: osdmap e929: 12 total, 12 up, 10 in 2024-08-19T14:48:14.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:14 smithi045 ceph-mon[20143]: 1.2 scrub starts 2024-08-19T14:48:14.637 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:48:14.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:14.663 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:48:14.663 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:48:14.765 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7 2024-08-19T14:48:14.790 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-08-19T14:48:14.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:14.791 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.7' 2024-08-19T14:48:14.893 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10 2024-08-19T14:48:14.919 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:48:14.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:14.920 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.10' 2024-08-19T14:48:15.021 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8 2024-08-19T14:48:15.047 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-08-19T14:48:15.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T14:48:15.048 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd.8' 2024-08-19T14:48:15.110 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-08-19T14:48:15.123 INFO:tasks.thrashosds:joining thrashosds 2024-08-19T14:48:15.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:15 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3874200120' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:48:15.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:15 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3874200120' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:48:15.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:15 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3874200120' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-08-19T14:48:15.796 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-08-19T14:48:15.796 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-08-19T14:48:15.796 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-08-19T14:48:15.796 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-08-19T14:48:15.796 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-19T14:48:15.797 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-19T14:48:15.797 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-19T14:48:15.797 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-19T14:48:15.797 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-08-19T14:48:15.797 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-08-19T14:48:15.797 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-08-19T14:48:15.797 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-08-19T14:48:15.797 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8: {} 2024-08-19T14:48:15.797 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9: {} 2024-08-19T14:48:15.798 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10: {} 2024-08-19T14:48:15.798 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-19T14:48:15.798 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-19T14:48:15.798 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-19T14:48:15.798 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-19T14:48:15.798 INFO:teuthology.orchestra.run.smithi031.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-19T14:48:15.798 INFO:teuthology.orchestra.run.smithi031.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-19T14:48:15.799 INFO:teuthology.orchestra.run.smithi031.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-19T14:48:15.799 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11: {} 2024-08-19T14:48:15.799 INFO:teuthology.orchestra.run.smithi031.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-19T14:48:16.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:16 smithi031 ceph-mon[16336]: pgmap v2179: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 629 KiB data, 9.5 GiB used, 884 GiB / 894 GiB avail 2024-08-19T14:48:16.441 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:48:16.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:16 smithi032 ceph-mon[20004]: pgmap v2179: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 629 KiB data, 9.5 GiB used, 884 GiB / 894 GiB avail 2024-08-19T14:48:16.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:16 smithi045 ceph-mon[20143]: pgmap v2179: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 629 KiB data, 9.5 GiB used, 884 GiB / 894 GiB avail 2024-08-19T14:48:16.926 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:48:16.947 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-08-19T14:48:16.947 INFO:teuthology.orchestra.run.smithi031.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-08-19T14:48:16.947 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-08-19T14:48:17.718 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-19T14:48:17.960 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:48:17.961 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:48:18.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:18 smithi032 ceph-mon[20004]: pgmap v2180: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 629 KiB data, 9.5 GiB used, 885 GiB / 894 GiB avail 2024-08-19T14:48:18.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:18 smithi045 ceph-mon[20143]: pgmap v2180: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 629 KiB data, 9.5 GiB used, 885 GiB / 894 GiB avail 2024-08-19T14:48:18.590 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:18 smithi031 ceph-mon[16336]: pgmap v2180: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 629 KiB data, 9.5 GiB used, 885 GiB / 894 GiB avail 2024-08-19T14:48:18.622 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-08-19T14:48:18.712 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:48:19.229 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:48:19.722 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:48:19.723 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:48:20.251 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is unset 2024-08-19T14:48:20.468 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:20 smithi031 ceph-mon[16336]: pgmap v2181: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 629 KiB data, 9.5 GiB used, 885 GiB / 894 GiB avail 2024-08-19T14:48:20.468 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:20 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2836896098' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:48:20.503 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-08-19T14:48:20.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:20 smithi032 ceph-mon[20004]: pgmap v2181: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 629 KiB data, 9.5 GiB used, 885 GiB / 894 GiB avail 2024-08-19T14:48:20.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:20 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2836896098' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:48:20.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:20 smithi045 ceph-mon[20143]: pgmap v2181: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 629 KiB data, 9.5 GiB used, 885 GiB / 894 GiB avail 2024-08-19T14:48:20.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:20 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2836896098' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-08-19T14:48:20.777 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:48:21.291 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:48:21.291 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:48:21.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:21 smithi032 ceph-mon[20004]: Health check cleared: OSDMAP_FLAGS (was: noscrub flag(s) set) 2024-08-19T14:48:21.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:21 smithi032 ceph-mon[20004]: Cluster is now healthy 2024-08-19T14:48:21.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:21 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2836896098' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:48:21.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:21 smithi032 ceph-mon[20004]: osdmap e930: 12 total, 12 up, 10 in 2024-08-19T14:48:21.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:21 smithi032 ceph-mon[20004]: 1.a scrub starts 2024-08-19T14:48:21.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:21 smithi032 ceph-mon[20004]: 1.a scrub ok 2024-08-19T14:48:21.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:21 smithi045 ceph-mon[20143]: Health check cleared: OSDMAP_FLAGS (was: noscrub flag(s) set) 2024-08-19T14:48:21.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:21 smithi045 ceph-mon[20143]: Cluster is now healthy 2024-08-19T14:48:21.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:21 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2836896098' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:48:21.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:21 smithi045 ceph-mon[20143]: osdmap e930: 12 total, 12 up, 10 in 2024-08-19T14:48:21.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:21 smithi045 ceph-mon[20143]: 1.a scrub starts 2024-08-19T14:48:21.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:21 smithi045 ceph-mon[20143]: 1.a scrub ok 2024-08-19T14:48:21.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:21 smithi031 ceph-mon[16336]: Health check cleared: OSDMAP_FLAGS (was: noscrub flag(s) set) 2024-08-19T14:48:21.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:21 smithi031 ceph-mon[16336]: Cluster is now healthy 2024-08-19T14:48:21.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:21 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2836896098' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-19T14:48:21.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:21 smithi031 ceph-mon[16336]: osdmap e930: 12 total, 12 up, 10 in 2024-08-19T14:48:21.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:21 smithi031 ceph-mon[16336]: 1.a scrub starts 2024-08-19T14:48:21.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:21 smithi031 ceph-mon[16336]: 1.a scrub ok 2024-08-19T14:48:21.950 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-19T14:48:22.398 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-08-19T14:48:22.437 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:48:22.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:22 smithi032 ceph-mon[20004]: pgmap v2183: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 629 KiB data, 9.5 GiB used, 885 GiB / 894 GiB avail 2024-08-19T14:48:22.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:22 smithi032 ceph-mon[20004]: 1.e scrub starts 2024-08-19T14:48:22.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:22 smithi032 ceph-mon[20004]: 1.e scrub ok 2024-08-19T14:48:22.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:22 smithi045 ceph-mon[20143]: pgmap v2183: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 629 KiB data, 9.5 GiB used, 885 GiB / 894 GiB avail 2024-08-19T14:48:22.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:22 smithi045 ceph-mon[20143]: 1.e scrub starts 2024-08-19T14:48:22.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:22 smithi045 ceph-mon[20143]: 1.e scrub ok 2024-08-19T14:48:22.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:22 smithi031 ceph-mon[16336]: pgmap v2183: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 629 KiB data, 9.5 GiB used, 885 GiB / 894 GiB avail 2024-08-19T14:48:22.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:22 smithi031 ceph-mon[16336]: 1.e scrub starts 2024-08-19T14:48:22.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:22 smithi031 ceph-mon[16336]: 1.e scrub ok 2024-08-19T14:48:22.756 INFO:tasks.thrashosds.thrasher:inning osd 2024-08-19T14:48:22.757 INFO:tasks.thrashosds.thrasher:Adding osd 0 2024-08-19T14:48:22.757 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd in 0 2024-08-19T14:48:23.001 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:48:23.026 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:48:23.256 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is unset 2024-08-19T14:48:23.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:23 smithi045 ceph-mon[20143]: 1.1 scrub starts 2024-08-19T14:48:23.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:23 smithi045 ceph-mon[20143]: 1.1 scrub ok 2024-08-19T14:48:23.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:23 smithi045 ceph-mon[20143]: 1.11 scrub starts 2024-08-19T14:48:23.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:23 smithi045 ceph-mon[20143]: 1.11 scrub ok 2024-08-19T14:48:23.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:23 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1163256269' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:48:23.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:23 smithi031 ceph-mon[16336]: 1.1 scrub starts 2024-08-19T14:48:23.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:23 smithi031 ceph-mon[16336]: 1.1 scrub ok 2024-08-19T14:48:23.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:23 smithi031 ceph-mon[16336]: 1.11 scrub starts 2024-08-19T14:48:23.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:23 smithi031 ceph-mon[16336]: 1.11 scrub ok 2024-08-19T14:48:23.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:23 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1163256269' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:48:23.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:23 smithi032 ceph-mon[20004]: 1.1 scrub starts 2024-08-19T14:48:23.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:23 smithi032 ceph-mon[20004]: 1.1 scrub ok 2024-08-19T14:48:23.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:23 smithi032 ceph-mon[20004]: 1.11 scrub starts 2024-08-19T14:48:23.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:23 smithi032 ceph-mon[20004]: 1.11 scrub ok 2024-08-19T14:48:23.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:23 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1163256269' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-08-19T14:48:23.768 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:48:23.768 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:48:24.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:24 smithi045 ceph-mon[20143]: pgmap v2184: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 629 KiB data, 9.5 GiB used, 885 GiB / 894 GiB avail 2024-08-19T14:48:24.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:24 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1163256269' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:48:24.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:24 smithi045 ceph-mon[20143]: osdmap e931: 12 total, 12 up, 10 in 2024-08-19T14:48:24.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:24 smithi045 ceph-mon[20143]: 1.9 deep-scrub starts 2024-08-19T14:48:24.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:24 smithi045 ceph-mon[20143]: 1.9 deep-scrub ok 2024-08-19T14:48:24.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:24 smithi045 ceph-mon[20143]: 1.1b scrub starts 2024-08-19T14:48:24.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:24 smithi045 ceph-mon[20143]: 1.1b scrub ok 2024-08-19T14:48:24.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:24 smithi045 ceph-mon[20143]: 1.13 scrub starts 2024-08-19T14:48:24.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:24 smithi045 ceph-mon[20143]: 1.13 scrub ok 2024-08-19T14:48:24.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:24 smithi045 ceph-mon[20143]: 1.f scrub starts 2024-08-19T14:48:24.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:24 smithi045 ceph-mon[20143]: 1.f scrub ok 2024-08-19T14:48:24.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:24 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:48:24.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:24 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:48:24.597 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:24 smithi031 ceph-mon[16336]: pgmap v2184: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 629 KiB data, 9.5 GiB used, 885 GiB / 894 GiB avail 2024-08-19T14:48:24.597 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:24 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1163256269' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:48:24.597 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:24 smithi031 ceph-mon[16336]: osdmap e931: 12 total, 12 up, 10 in 2024-08-19T14:48:24.598 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:24 smithi031 ceph-mon[16336]: 1.9 deep-scrub starts 2024-08-19T14:48:24.598 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:24 smithi031 ceph-mon[16336]: 1.9 deep-scrub ok 2024-08-19T14:48:24.598 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:24 smithi031 ceph-mon[16336]: 1.1b scrub starts 2024-08-19T14:48:24.598 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:24 smithi031 ceph-mon[16336]: 1.1b scrub ok 2024-08-19T14:48:24.598 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:24 smithi031 ceph-mon[16336]: 1.13 scrub starts 2024-08-19T14:48:24.598 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:24 smithi031 ceph-mon[16336]: 1.13 scrub ok 2024-08-19T14:48:24.598 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:24 smithi031 ceph-mon[16336]: 1.f scrub starts 2024-08-19T14:48:24.598 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:24 smithi031 ceph-mon[16336]: 1.f scrub ok 2024-08-19T14:48:24.598 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:24 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:48:24.598 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:24 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:48:24.624 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-08-19T14:48:24.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:24 smithi032 ceph-mon[20004]: pgmap v2184: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 629 KiB data, 9.5 GiB used, 885 GiB / 894 GiB avail 2024-08-19T14:48:24.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:24 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1163256269' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-19T14:48:24.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:24 smithi032 ceph-mon[20004]: osdmap e931: 12 total, 12 up, 10 in 2024-08-19T14:48:24.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:24 smithi032 ceph-mon[20004]: 1.9 deep-scrub starts 2024-08-19T14:48:24.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:24 smithi032 ceph-mon[20004]: 1.9 deep-scrub ok 2024-08-19T14:48:24.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:24 smithi032 ceph-mon[20004]: 1.1b scrub starts 2024-08-19T14:48:24.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:24 smithi032 ceph-mon[20004]: 1.1b scrub ok 2024-08-19T14:48:24.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:24 smithi032 ceph-mon[20004]: 1.13 scrub starts 2024-08-19T14:48:24.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:24 smithi032 ceph-mon[20004]: 1.13 scrub ok 2024-08-19T14:48:24.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:24 smithi032 ceph-mon[20004]: 1.f scrub starts 2024-08-19T14:48:24.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:24 smithi032 ceph-mon[20004]: 1.f scrub ok 2024-08-19T14:48:24.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:24 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:48:24.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:24 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:48:25.318 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:25 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mon-a[16332]: 2024-08-19T14:48:25.028+0000 7f2ff503e640 -1 mon.a@0(leader).osd e932 definitely_dead 0 2024-08-19T14:48:25.318 INFO:teuthology.orchestra.run.smithi031.stderr:marked in osd.0. 2024-08-19T14:48:25.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:25 smithi031 ceph-mon[16336]: 1.2 deep-scrub starts 2024-08-19T14:48:25.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:25 smithi031 ceph-mon[16336]: 1.2 deep-scrub ok 2024-08-19T14:48:25.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:25 smithi031 ceph-mon[16336]: osdmap e932: 12 total, 12 up, 10 in 2024-08-19T14:48:25.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:25 smithi031 ceph-mon[16336]: 1.17 scrub starts 2024-08-19T14:48:25.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:25 smithi031 ceph-mon[16336]: 1.17 scrub ok 2024-08-19T14:48:25.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:25 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/813756985' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["0"]}]: dispatch 2024-08-19T14:48:25.690 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:48:25.716 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:25 smithi045 ceph-mon[20143]: 1.2 deep-scrub starts 2024-08-19T14:48:25.716 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:25 smithi045 ceph-mon[20143]: 1.2 deep-scrub ok 2024-08-19T14:48:25.716 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:25 smithi045 ceph-mon[20143]: osdmap e932: 12 total, 12 up, 10 in 2024-08-19T14:48:25.716 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:25 smithi045 ceph-mon[20143]: 1.17 scrub starts 2024-08-19T14:48:25.716 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:25 smithi045 ceph-mon[20143]: 1.17 scrub ok 2024-08-19T14:48:25.716 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:25 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/813756985' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["0"]}]: dispatch 2024-08-19T14:48:25.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:25 smithi032 ceph-mon[20004]: 1.2 deep-scrub starts 2024-08-19T14:48:25.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:25 smithi032 ceph-mon[20004]: 1.2 deep-scrub ok 2024-08-19T14:48:25.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:25 smithi032 ceph-mon[20004]: osdmap e932: 12 total, 12 up, 10 in 2024-08-19T14:48:25.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:25 smithi032 ceph-mon[20004]: 1.17 scrub starts 2024-08-19T14:48:25.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:25 smithi032 ceph-mon[20004]: 1.17 scrub ok 2024-08-19T14:48:25.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:25 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/813756985' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["0"]}]: dispatch 2024-08-19T14:48:26.112 INFO:tasks.thrashosds.thrasher:Added osd 0 2024-08-19T14:48:26.113 INFO:tasks.thrashosds.thrasher:inning osd 2024-08-19T14:48:26.113 INFO:tasks.thrashosds.thrasher:Adding osd 5 2024-08-19T14:48:26.113 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd in 5 2024-08-19T14:48:26.355 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:26 smithi031 ceph-mon[16336]: pgmap v2187: 31 pgs: 31 active+clean; 629 KiB data, 9.5 GiB used, 885 GiB / 894 GiB avail 2024-08-19T14:48:26.355 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:26 smithi031 ceph-mon[16336]: 1.6 scrub starts 2024-08-19T14:48:26.355 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:26 smithi031 ceph-mon[16336]: 1.6 scrub ok 2024-08-19T14:48:26.355 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:26 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/813756985' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["0"]}]': finished 2024-08-19T14:48:26.355 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:26 smithi031 ceph-mon[16336]: osdmap e933: 12 total, 12 up, 11 in 2024-08-19T14:48:26.355 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:26 smithi031 ceph-mon[16336]: 1.d scrub starts 2024-08-19T14:48:26.355 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:26 smithi031 ceph-mon[16336]: 1.d scrub ok 2024-08-19T14:48:26.355 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:26 smithi031 ceph-mon[16336]: 1.3 scrub starts 2024-08-19T14:48:26.356 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:26 smithi031 ceph-mon[16336]: 1.3 scrub ok 2024-08-19T14:48:26.604 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:48:26.604 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:48:26.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:26 smithi032 ceph-mon[20004]: pgmap v2187: 31 pgs: 31 active+clean; 629 KiB data, 9.5 GiB used, 885 GiB / 894 GiB avail 2024-08-19T14:48:26.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:26 smithi032 ceph-mon[20004]: 1.6 scrub starts 2024-08-19T14:48:26.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:26 smithi032 ceph-mon[20004]: 1.6 scrub ok 2024-08-19T14:48:26.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:26 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/813756985' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["0"]}]': finished 2024-08-19T14:48:26.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:26 smithi032 ceph-mon[20004]: osdmap e933: 12 total, 12 up, 11 in 2024-08-19T14:48:26.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:26 smithi032 ceph-mon[20004]: 1.d scrub starts 2024-08-19T14:48:26.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:26 smithi032 ceph-mon[20004]: 1.d scrub ok 2024-08-19T14:48:26.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:26 smithi032 ceph-mon[20004]: 1.3 scrub starts 2024-08-19T14:48:26.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:26 smithi032 ceph-mon[20004]: 1.3 scrub ok 2024-08-19T14:48:26.771 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:26 smithi045 ceph-mon[20143]: pgmap v2187: 31 pgs: 31 active+clean; 629 KiB data, 9.5 GiB used, 885 GiB / 894 GiB avail 2024-08-19T14:48:26.771 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:26 smithi045 ceph-mon[20143]: 1.6 scrub starts 2024-08-19T14:48:26.771 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:26 smithi045 ceph-mon[20143]: 1.6 scrub ok 2024-08-19T14:48:26.771 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:26 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/813756985' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["0"]}]': finished 2024-08-19T14:48:26.771 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:26 smithi045 ceph-mon[20143]: osdmap e933: 12 total, 12 up, 11 in 2024-08-19T14:48:26.771 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:26 smithi045 ceph-mon[20143]: 1.d scrub starts 2024-08-19T14:48:26.772 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:26 smithi045 ceph-mon[20143]: 1.d scrub ok 2024-08-19T14:48:26.772 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:26 smithi045 ceph-mon[20143]: 1.3 scrub starts 2024-08-19T14:48:26.772 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:26 smithi045 ceph-mon[20143]: 1.3 scrub ok 2024-08-19T14:48:26.849 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:48:27.319 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-08-19T14:48:27.522 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:27 smithi031 ceph-mon[16336]: osdmap e934: 12 total, 12 up, 11 in 2024-08-19T14:48:27.522 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:27 smithi031 ceph-mon[16336]: 1.15 scrub starts 2024-08-19T14:48:27.522 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:27 smithi031 ceph-mon[16336]: 1.15 scrub ok 2024-08-19T14:48:27.522 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:27 smithi031 ceph-mon[16336]: pgmap v2190: 31 pgs: 3 peering, 2 unknown, 26 active+clean; 629 KiB data, 10 GiB used, 973 GiB / 983 GiB avail 2024-08-19T14:48:27.522 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:27 smithi031 ceph-mon[16336]: 1.8 scrub starts 2024-08-19T14:48:27.522 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:27 smithi031 ceph-mon[16336]: 1.8 scrub ok 2024-08-19T14:48:27.522 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:27 smithi031 ceph-mon[16336]: osdmap e935: 12 total, 12 up, 11 in 2024-08-19T14:48:27.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:27 smithi032 ceph-mon[20004]: osdmap e934: 12 total, 12 up, 11 in 2024-08-19T14:48:27.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:27 smithi032 ceph-mon[20004]: 1.15 scrub starts 2024-08-19T14:48:27.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:27 smithi032 ceph-mon[20004]: 1.15 scrub ok 2024-08-19T14:48:27.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:27 smithi032 ceph-mon[20004]: pgmap v2190: 31 pgs: 3 peering, 2 unknown, 26 active+clean; 629 KiB data, 10 GiB used, 973 GiB / 983 GiB avail 2024-08-19T14:48:27.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:27 smithi032 ceph-mon[20004]: 1.8 scrub starts 2024-08-19T14:48:27.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:27 smithi032 ceph-mon[20004]: 1.8 scrub ok 2024-08-19T14:48:27.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:27 smithi032 ceph-mon[20004]: osdmap e935: 12 total, 12 up, 11 in 2024-08-19T14:48:27.792 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:48:27.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:27 smithi045 ceph-mon[20143]: osdmap e934: 12 total, 12 up, 11 in 2024-08-19T14:48:27.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:27 smithi045 ceph-mon[20143]: 1.15 scrub starts 2024-08-19T14:48:27.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:27 smithi045 ceph-mon[20143]: 1.15 scrub ok 2024-08-19T14:48:27.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:27 smithi045 ceph-mon[20143]: pgmap v2190: 31 pgs: 3 peering, 2 unknown, 26 active+clean; 629 KiB data, 10 GiB used, 973 GiB / 983 GiB avail 2024-08-19T14:48:27.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:27 smithi045 ceph-mon[20143]: 1.8 scrub starts 2024-08-19T14:48:27.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:27 smithi045 ceph-mon[20143]: 1.8 scrub ok 2024-08-19T14:48:27.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:27 smithi045 ceph-mon[20143]: osdmap e935: 12 total, 12 up, 11 in 2024-08-19T14:48:28.128 INFO:teuthology.orchestra.run.smithi031.stderr:marked in osd.5. 2024-08-19T14:48:28.262 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:27 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mon-a[16332]: 2024-08-19T14:48:27.936+0000 7f2ff503e640 -1 mon.a@0(leader).osd e935 definitely_dead 0 2024-08-19T14:48:28.360 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:48:28.360 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:48:28.591 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:28 smithi031 ceph-mon[16336]: 1.10 scrub starts 2024-08-19T14:48:28.591 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:28 smithi031 ceph-mon[16336]: 1.10 scrub ok 2024-08-19T14:48:28.591 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:28 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/291636141' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["5"]}]: dispatch 2024-08-19T14:48:28.591 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:28 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/291636141' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-08-19T14:48:28.591 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:28 smithi031 ceph-mon[16336]: osdmap e936: 12 total, 12 up, 12 in 2024-08-19T14:48:28.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:28 smithi032 ceph-mon[20004]: 1.10 scrub starts 2024-08-19T14:48:28.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:28 smithi032 ceph-mon[20004]: 1.10 scrub ok 2024-08-19T14:48:28.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:28 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/291636141' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["5"]}]: dispatch 2024-08-19T14:48:28.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:28 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/291636141' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-08-19T14:48:28.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:28 smithi032 ceph-mon[20004]: osdmap e936: 12 total, 12 up, 12 in 2024-08-19T14:48:28.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:28 smithi045 ceph-mon[20143]: 1.10 scrub starts 2024-08-19T14:48:28.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:28 smithi045 ceph-mon[20143]: 1.10 scrub ok 2024-08-19T14:48:28.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:28 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/291636141' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["5"]}]: dispatch 2024-08-19T14:48:28.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:28 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/291636141' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-08-19T14:48:28.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:28 smithi045 ceph-mon[20143]: osdmap e936: 12 total, 12 up, 12 in 2024-08-19T14:48:29.391 INFO:tasks.thrashosds.thrasher:Added osd 5 2024-08-19T14:48:29.391 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-08-19T14:48:29.391 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd pool set .mgr pgp_num 101 2024-08-19T14:48:29.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:29 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:48:29.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:29 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:48:29.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:29 smithi031 ceph-mon[16336]: 1.4 scrub starts 2024-08-19T14:48:29.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:29 smithi031 ceph-mon[16336]: 1.4 scrub ok 2024-08-19T14:48:29.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:29 smithi031 ceph-mon[16336]: pgmap v2193: 31 pgs: 1 active+undersized+remapped, 3 peering, 2 unknown, 25 active+clean; 629 KiB data, 10 GiB used, 973 GiB / 983 GiB avail 2024-08-19T14:48:29.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:29 smithi031 ceph-mon[16336]: 1.14 scrub starts 2024-08-19T14:48:29.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:29 smithi031 ceph-mon[16336]: 1.14 scrub ok 2024-08-19T14:48:29.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:29 smithi031 ceph-mon[16336]: osdmap e937: 12 total, 12 up, 12 in 2024-08-19T14:48:29.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:29 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:48:29.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:29 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:48:29.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:29 smithi032 ceph-mon[20004]: 1.4 scrub starts 2024-08-19T14:48:29.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:29 smithi032 ceph-mon[20004]: 1.4 scrub ok 2024-08-19T14:48:29.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:29 smithi032 ceph-mon[20004]: pgmap v2193: 31 pgs: 1 active+undersized+remapped, 3 peering, 2 unknown, 25 active+clean; 629 KiB data, 10 GiB used, 973 GiB / 983 GiB avail 2024-08-19T14:48:29.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:29 smithi032 ceph-mon[20004]: 1.14 scrub starts 2024-08-19T14:48:29.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:29 smithi032 ceph-mon[20004]: 1.14 scrub ok 2024-08-19T14:48:29.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:29 smithi032 ceph-mon[20004]: osdmap e937: 12 total, 12 up, 12 in 2024-08-19T14:48:29.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:29 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:48:29.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:29 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:48:29.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:29 smithi045 ceph-mon[20143]: 1.4 scrub starts 2024-08-19T14:48:29.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:29 smithi045 ceph-mon[20143]: 1.4 scrub ok 2024-08-19T14:48:29.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:29 smithi045 ceph-mon[20143]: pgmap v2193: 31 pgs: 1 active+undersized+remapped, 3 peering, 2 unknown, 25 active+clean; 629 KiB data, 10 GiB used, 973 GiB / 983 GiB avail 2024-08-19T14:48:29.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:29 smithi045 ceph-mon[20143]: 1.14 scrub starts 2024-08-19T14:48:29.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:29 smithi045 ceph-mon[20143]: 1.14 scrub ok 2024-08-19T14:48:29.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:29 smithi045 ceph-mon[20143]: osdmap e937: 12 total, 12 up, 12 in 2024-08-19T14:48:29.871 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-08-19T14:48:29.982 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:48:30.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:30 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:48:30.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:30 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:48:30.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:30 smithi031 ceph-mon[16336]: osdmap e938: 12 total, 12 up, 12 in 2024-08-19T14:48:30.567 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:48:30.738 INFO:teuthology.orchestra.run.smithi031.stderr:Error EINVAL: specified pgp_num 101 > pg_num 31 2024-08-19T14:48:30.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:30 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:48:30.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:30 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:48:30.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:30 smithi032 ceph-mon[20004]: osdmap e938: 12 total, 12 up, 12 in 2024-08-19T14:48:30.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:30 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:48:30.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:30 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:48:30.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:30 smithi045 ceph-mon[20143]: osdmap e938: 12 total, 12 up, 12 in 2024-08-19T14:48:31.521 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:31 smithi031 ceph-mon[16336]: pgmap v2196: 31 pgs: 2 active+undersized+remapped, 3 peering, 5 remapped+peering, 2 unknown, 19 active+clean; 577 KiB data, 10 GiB used, 973 GiB / 983 GiB avail 2024-08-19T14:48:31.522 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:31 smithi031 ceph-mon[16336]: 1.16 scrub starts 2024-08-19T14:48:31.522 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:31 smithi031 ceph-mon[16336]: 1.16 scrub ok 2024-08-19T14:48:31.522 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:31 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/569191818' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "101"}]: dispatch 2024-08-19T14:48:31.522 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:31 smithi031 ceph-mon[16336]: osdmap e939: 12 total, 12 up, 12 in 2024-08-19T14:48:31.551 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-19T14:48:31.551 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-08-19T14:48:31.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:31 smithi032 ceph-mon[20004]: pgmap v2196: 31 pgs: 2 active+undersized+remapped, 3 peering, 5 remapped+peering, 2 unknown, 19 active+clean; 577 KiB data, 10 GiB used, 973 GiB / 983 GiB avail 2024-08-19T14:48:31.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:31 smithi032 ceph-mon[20004]: 1.16 scrub starts 2024-08-19T14:48:31.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:31 smithi032 ceph-mon[20004]: 1.16 scrub ok 2024-08-19T14:48:31.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:31 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/569191818' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "101"}]: dispatch 2024-08-19T14:48:31.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:31 smithi032 ceph-mon[20004]: osdmap e939: 12 total, 12 up, 12 in 2024-08-19T14:48:31.776 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:48:31.776 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:48:31.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:31 smithi045 ceph-mon[20143]: pgmap v2196: 31 pgs: 2 active+undersized+remapped, 3 peering, 5 remapped+peering, 2 unknown, 19 active+clean; 577 KiB data, 10 GiB used, 973 GiB / 983 GiB avail 2024-08-19T14:48:31.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:31 smithi045 ceph-mon[20143]: 1.16 scrub starts 2024-08-19T14:48:31.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:31 smithi045 ceph-mon[20143]: 1.16 scrub ok 2024-08-19T14:48:31.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:31 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/569191818' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "101"}]: dispatch 2024-08-19T14:48:31.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:31 smithi045 ceph-mon[20143]: osdmap e939: 12 total, 12 up, 12 in 2024-08-19T14:48:32.050 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:48:32.457 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-08-19T14:48:32.988 INFO:teuthology.orchestra.run.smithi031.stdout:mon.a: {} 2024-08-19T14:48:32.988 INFO:teuthology.orchestra.run.smithi031.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-08-19T14:48:32.998 INFO:teuthology.orchestra.run.smithi031.stdout:mon.c: {} 2024-08-19T14:48:32.999 INFO:teuthology.orchestra.run.smithi031.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-08-19T14:48:33.009 INFO:teuthology.orchestra.run.smithi031.stdout:mon.b: {} 2024-08-19T14:48:33.009 INFO:teuthology.orchestra.run.smithi031.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-08-19T14:48:33.254 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:48:33.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:33 smithi031 ceph-mon[16336]: osdmap e940: 12 total, 12 up, 12 in 2024-08-19T14:48:33.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:33 smithi031 ceph-mon[16336]: 1.b scrub starts 2024-08-19T14:48:33.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:33 smithi031 ceph-mon[16336]: 1.b scrub ok 2024-08-19T14:48:33.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:33 smithi031 ceph-mon[16336]: 1.c scrub starts 2024-08-19T14:48:33.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:33 smithi031 ceph-mon[16336]: 1.c scrub ok 2024-08-19T14:48:33.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:33 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3029847457' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-08-19T14:48:33.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:33 smithi032 ceph-mon[20004]: osdmap e940: 12 total, 12 up, 12 in 2024-08-19T14:48:33.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:33 smithi032 ceph-mon[20004]: 1.b scrub starts 2024-08-19T14:48:33.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:33 smithi032 ceph-mon[20004]: 1.b scrub ok 2024-08-19T14:48:33.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:33 smithi032 ceph-mon[20004]: 1.c scrub starts 2024-08-19T14:48:33.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:33 smithi032 ceph-mon[20004]: 1.c scrub ok 2024-08-19T14:48:33.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:33 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3029847457' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-08-19T14:48:33.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:33 smithi045 ceph-mon[20143]: osdmap e940: 12 total, 12 up, 12 in 2024-08-19T14:48:33.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:33 smithi045 ceph-mon[20143]: 1.b scrub starts 2024-08-19T14:48:33.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:33 smithi045 ceph-mon[20143]: 1.b scrub ok 2024-08-19T14:48:33.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:33 smithi045 ceph-mon[20143]: 1.c scrub starts 2024-08-19T14:48:33.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:33 smithi045 ceph-mon[20143]: 1.c scrub ok 2024-08-19T14:48:33.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:33 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3029847457' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-08-19T14:48:33.767 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd reweight 0 1 2024-08-19T14:48:34.298 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:48:34.298 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:48:34.298 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:34 smithi031 ceph-mon[16336]: pgmap v2199: 31 pgs: 3 active+undersized+remapped, 5 remapped+peering, 23 active+clean; 577 KiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s, 0 objects/s recovering 2024-08-19T14:48:34.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:34 smithi032 ceph-mon[20004]: pgmap v2199: 31 pgs: 3 active+undersized+remapped, 5 remapped+peering, 23 active+clean; 577 KiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s, 0 objects/s recovering 2024-08-19T14:48:34.526 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:48:34.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:34 smithi045 ceph-mon[20143]: pgmap v2199: 31 pgs: 3 active+undersized+remapped, 5 remapped+peering, 23 active+clean; 577 KiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s, 0 objects/s recovering 2024-08-19T14:48:35.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:35 smithi031 ceph-mon[16336]: 1.18 scrub starts 2024-08-19T14:48:35.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:35 smithi031 ceph-mon[16336]: 1.18 scrub ok 2024-08-19T14:48:35.422 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-08-19T14:48:35.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:35 smithi032 ceph-mon[20004]: 1.18 scrub starts 2024-08-19T14:48:35.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:35 smithi032 ceph-mon[20004]: 1.18 scrub ok 2024-08-19T14:48:35.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:35 smithi045 ceph-mon[20143]: 1.18 scrub starts 2024-08-19T14:48:35.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:35 smithi045 ceph-mon[20143]: 1.18 scrub ok 2024-08-19T14:48:35.673 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:48:36.151 INFO:teuthology.orchestra.run.smithi031.stderr:reweighted osd.0 to 1 (10000) 2024-08-19T14:48:36.196 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:48:36.196 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:48:36.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:36 smithi031 ceph-mon[16336]: pgmap v2200: 31 pgs: 2 active+undersized+remapped, 1 remapped+peering, 28 active+clean; 577 KiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s, 0 objects/s recovering 2024-08-19T14:48:36.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:36 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/14680989' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-08-19T14:48:36.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:36 smithi031 ceph-mon[16336]: 1.0 scrub starts 2024-08-19T14:48:36.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:36 smithi031 ceph-mon[16336]: 1.0 scrub ok 2024-08-19T14:48:36.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:36 smithi032 ceph-mon[20004]: pgmap v2200: 31 pgs: 2 active+undersized+remapped, 1 remapped+peering, 28 active+clean; 577 KiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s, 0 objects/s recovering 2024-08-19T14:48:36.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:36 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/14680989' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-08-19T14:48:36.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:36 smithi032 ceph-mon[20004]: 1.0 scrub starts 2024-08-19T14:48:36.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:36 smithi032 ceph-mon[20004]: 1.0 scrub ok 2024-08-19T14:48:36.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:36 smithi045 ceph-mon[20143]: pgmap v2200: 31 pgs: 2 active+undersized+remapped, 1 remapped+peering, 28 active+clean; 577 KiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s, 0 objects/s recovering 2024-08-19T14:48:36.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:36 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/14680989' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-08-19T14:48:36.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:36 smithi045 ceph-mon[20143]: 1.0 scrub starts 2024-08-19T14:48:36.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:36 smithi045 ceph-mon[20143]: 1.0 scrub ok 2024-08-19T14:48:36.814 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd primary-affinity 0 1 2024-08-19T14:48:37.447 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:37 smithi031 ceph-mon[16336]: 1.5 scrub starts 2024-08-19T14:48:37.447 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:37 smithi031 ceph-mon[16336]: 1.5 scrub ok 2024-08-19T14:48:37.447 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:37 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/14680989' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-08-19T14:48:37.448 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:37 smithi031 ceph-mon[16336]: osdmap e941: 12 total, 12 up, 12 in 2024-08-19T14:48:37.475 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-08-19T14:48:37.501 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:37 smithi032 ceph-mon[20004]: 1.5 scrub starts 2024-08-19T14:48:37.501 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:37 smithi032 ceph-mon[20004]: 1.5 scrub ok 2024-08-19T14:48:37.501 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:37 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/14680989' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-08-19T14:48:37.502 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:37 smithi032 ceph-mon[20004]: osdmap e941: 12 total, 12 up, 12 in 2024-08-19T14:48:37.534 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:48:37.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:37 smithi045 ceph-mon[20143]: 1.5 scrub starts 2024-08-19T14:48:37.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:37 smithi045 ceph-mon[20143]: 1.5 scrub ok 2024-08-19T14:48:37.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:37 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/14680989' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-08-19T14:48:37.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:37 smithi045 ceph-mon[20143]: osdmap e941: 12 total, 12 up, 12 in 2024-08-19T14:48:37.743 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:48:38.157 INFO:teuthology.orchestra.run.smithi031.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-08-19T14:48:38.395 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:38 smithi032 ceph-mon[20004]: pgmap v2202: 31 pgs: 31 active+clean; 577 KiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s, 0 objects/s recovering 2024-08-19T14:48:38.395 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:38 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1258069812' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-08-19T14:48:38.423 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:48:38.423 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:48:38.503 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:38 smithi031 ceph-mon[16336]: pgmap v2202: 31 pgs: 31 active+clean; 577 KiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s, 0 objects/s recovering 2024-08-19T14:48:38.503 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:38 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1258069812' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-08-19T14:48:38.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:38 smithi045 ceph-mon[20143]: pgmap v2202: 31 pgs: 31 active+clean; 577 KiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s, 0 objects/s recovering 2024-08-19T14:48:38.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:38 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1258069812' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-08-19T14:48:38.814 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd reweight 1 1 2024-08-19T14:48:39.071 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:48:39.183 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-08-19T14:48:39.473 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:39 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1258069812' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-08-19T14:48:39.474 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:39 smithi031 ceph-mon[16336]: osdmap e942: 12 total, 12 up, 12 in 2024-08-19T14:48:39.479 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:48:39.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:39 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1258069812' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-08-19T14:48:39.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:39 smithi032 ceph-mon[20004]: osdmap e942: 12 total, 12 up, 12 in 2024-08-19T14:48:39.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:39 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1258069812' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-08-19T14:48:39.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:39 smithi045 ceph-mon[20143]: osdmap e942: 12 total, 12 up, 12 in 2024-08-19T14:48:39.965 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:48:39.966 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:48:40.178 INFO:teuthology.orchestra.run.smithi031.stderr:reweighted osd.1 to 1 (10000) 2024-08-19T14:48:40.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:40 smithi032 ceph-mon[20004]: pgmap v2204: 31 pgs: 31 active+clean; 577 KiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s, 0 objects/s recovering 2024-08-19T14:48:40.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:40 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:48:40.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:40 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:48:40.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:40 smithi032 ceph-mon[20004]: osdmap e943: 12 total, 12 up, 12 in 2024-08-19T14:48:40.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:40 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3942718389' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-08-19T14:48:40.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:40 smithi045 ceph-mon[20143]: pgmap v2204: 31 pgs: 31 active+clean; 577 KiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s, 0 objects/s recovering 2024-08-19T14:48:40.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:40 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:48:40.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:40 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:48:40.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:40 smithi045 ceph-mon[20143]: osdmap e943: 12 total, 12 up, 12 in 2024-08-19T14:48:40.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:40 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3942718389' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-08-19T14:48:40.555 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-08-19T14:48:40.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:40 smithi031 ceph-mon[16336]: pgmap v2204: 31 pgs: 31 active+clean; 577 KiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s, 0 objects/s recovering 2024-08-19T14:48:40.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:40 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:48:40.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:40 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:48:40.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:40 smithi031 ceph-mon[16336]: osdmap e943: 12 total, 12 up, 12 in 2024-08-19T14:48:40.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:40 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3942718389' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-08-19T14:48:40.809 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:48:41.222 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd primary-affinity 1 1 2024-08-19T14:48:41.231 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:48:41.231 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:48:41.480 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:48:41.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:41 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3942718389' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-08-19T14:48:41.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:41 smithi032 ceph-mon[20004]: osdmap e944: 12 total, 12 up, 12 in 2024-08-19T14:48:41.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:41 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3942718389' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-08-19T14:48:41.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:41 smithi045 ceph-mon[20143]: osdmap e944: 12 total, 12 up, 12 in 2024-08-19T14:48:41.602 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:41 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3942718389' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-08-19T14:48:41.602 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:41 smithi031 ceph-mon[16336]: osdmap e944: 12 total, 12 up, 12 in 2024-08-19T14:48:41.889 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-08-19T14:48:42.144 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:48:42.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:42 smithi032 ceph-mon[20004]: pgmap v2207: 31 pgs: 31 active+clean; 577 KiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:48:42.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:42 smithi045 ceph-mon[20143]: pgmap v2207: 31 pgs: 31 active+clean; 577 KiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:48:42.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:42 smithi031 ceph-mon[16336]: pgmap v2207: 31 pgs: 31 active+clean; 577 KiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:48:42.630 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:48:42.630 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:48:43.205 INFO:teuthology.orchestra.run.smithi031.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-08-19T14:48:43.236 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:43 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3281370197' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-08-19T14:48:43.238 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-08-19T14:48:43.491 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:48:43.527 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:43 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3281370197' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-08-19T14:48:43.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:43 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3281370197' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-08-19T14:48:43.979 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd reweight 3 1 2024-08-19T14:48:44.013 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:48:44.013 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:48:44.237 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:48:44.459 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:44 smithi031 ceph-mon[16336]: pgmap v2208: 31 pgs: 31 active+clean; 577 KiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:48:44.459 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:44 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3281370197' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-08-19T14:48:44.459 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:44 smithi031 ceph-mon[16336]: osdmap e945: 12 total, 12 up, 12 in 2024-08-19T14:48:44.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:44 smithi032 ceph-mon[20004]: pgmap v2208: 31 pgs: 31 active+clean; 577 KiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:48:44.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:44 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3281370197' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-08-19T14:48:44.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:44 smithi032 ceph-mon[20004]: osdmap e945: 12 total, 12 up, 12 in 2024-08-19T14:48:44.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:44 smithi045 ceph-mon[20143]: pgmap v2208: 31 pgs: 31 active+clean; 577 KiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:48:44.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:44 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3281370197' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-08-19T14:48:44.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:44 smithi045 ceph-mon[20143]: osdmap e945: 12 total, 12 up, 12 in 2024-08-19T14:48:44.647 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-08-19T14:48:44.901 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:48:45.215 INFO:teuthology.orchestra.run.smithi031.stderr:reweighted osd.3 to 1 (10000) 2024-08-19T14:48:45.312 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:45 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1468385668' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-08-19T14:48:45.413 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:48:45.413 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:48:45.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:45 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1468385668' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-08-19T14:48:45.551 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:45 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1468385668' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-08-19T14:48:45.945 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd primary-affinity 3 1 2024-08-19T14:48:46.107 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-08-19T14:48:46.202 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:48:46.363 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:48:46.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:46 smithi032 ceph-mon[20004]: pgmap v2210: 31 pgs: 31 active+clean; 577 KiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:48:46.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:46 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1468385668' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-08-19T14:48:46.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:46 smithi032 ceph-mon[20004]: osdmap e946: 12 total, 12 up, 12 in 2024-08-19T14:48:46.538 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:46 smithi031 ceph-mon[16336]: pgmap v2210: 31 pgs: 31 active+clean; 577 KiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:48:46.538 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:46 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1468385668' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-08-19T14:48:46.538 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:46 smithi031 ceph-mon[16336]: osdmap e946: 12 total, 12 up, 12 in 2024-08-19T14:48:46.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:46 smithi045 ceph-mon[20143]: pgmap v2210: 31 pgs: 31 active+clean; 577 KiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:48:46.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:46 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1468385668' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-08-19T14:48:46.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:46 smithi045 ceph-mon[20143]: osdmap e946: 12 total, 12 up, 12 in 2024-08-19T14:48:46.902 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:48:46.902 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:48:47.217 INFO:teuthology.orchestra.run.smithi031.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-08-19T14:48:47.459 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:47 smithi032 ceph-mon[20004]: 1.7 scrub starts 2024-08-19T14:48:47.459 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:47 smithi032 ceph-mon[20004]: 1.7 scrub ok 2024-08-19T14:48:47.459 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:47 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1551678792' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-08-19T14:48:47.488 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-08-19T14:48:47.516 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:47 smithi031 ceph-mon[16336]: 1.7 scrub starts 2024-08-19T14:48:47.517 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:47 smithi031 ceph-mon[16336]: 1.7 scrub ok 2024-08-19T14:48:47.517 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:47 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1551678792' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-08-19T14:48:47.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:47 smithi045 ceph-mon[20143]: 1.7 scrub starts 2024-08-19T14:48:47.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:47 smithi045 ceph-mon[20143]: 1.7 scrub ok 2024-08-19T14:48:47.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:47 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1551678792' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-08-19T14:48:47.741 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:48:47.894 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd reweight 4 1 2024-08-19T14:48:48.159 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:48:48.206 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:48:48.206 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:48:48.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:48 smithi032 ceph-mon[20004]: pgmap v2212: 31 pgs: 31 active+clean; 577 KiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:48:48.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:48 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1551678792' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-08-19T14:48:48.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:48 smithi032 ceph-mon[20004]: osdmap e947: 12 total, 12 up, 12 in 2024-08-19T14:48:48.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:48 smithi045 ceph-mon[20143]: pgmap v2212: 31 pgs: 31 active+clean; 577 KiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:48:48.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:48 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1551678792' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-08-19T14:48:48.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:48 smithi045 ceph-mon[20143]: osdmap e947: 12 total, 12 up, 12 in 2024-08-19T14:48:48.595 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:48 smithi031 ceph-mon[16336]: pgmap v2212: 31 pgs: 31 active+clean; 577 KiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:48:48.595 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:48 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1551678792' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-08-19T14:48:48.595 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:48 smithi031 ceph-mon[16336]: osdmap e947: 12 total, 12 up, 12 in 2024-08-19T14:48:48.880 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-08-19T14:48:49.135 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:48:49.229 INFO:teuthology.orchestra.run.smithi031.stderr:reweighted osd.4 to 1 (10000) 2024-08-19T14:48:49.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:49 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/743547984' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-08-19T14:48:49.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:49 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/743547984' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-08-19T14:48:49.548 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:49 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/743547984' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-08-19T14:48:49.637 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:48:49.637 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:48:49.985 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd primary-affinity 4 1 2024-08-19T14:48:50.246 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:48:50.247 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-08-19T14:48:50.481 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:50 smithi031 ceph-mon[16336]: pgmap v2214: 31 pgs: 31 active+clean; 577 KiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:48:50.481 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:50 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/743547984' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-08-19T14:48:50.481 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:50 smithi031 ceph-mon[16336]: osdmap e948: 12 total, 12 up, 12 in 2024-08-19T14:48:50.481 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:50 smithi031 ceph-mon[16336]: 1.1c scrub starts 2024-08-19T14:48:50.481 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:50 smithi031 ceph-mon[16336]: 1.1c scrub ok 2024-08-19T14:48:50.507 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:48:50.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:50 smithi032 ceph-mon[20004]: pgmap v2214: 31 pgs: 31 active+clean; 577 KiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:48:50.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:50 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/743547984' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-08-19T14:48:50.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:50 smithi032 ceph-mon[20004]: osdmap e948: 12 total, 12 up, 12 in 2024-08-19T14:48:50.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:50 smithi032 ceph-mon[20004]: 1.1c scrub starts 2024-08-19T14:48:50.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:50 smithi032 ceph-mon[20004]: 1.1c scrub ok 2024-08-19T14:48:50.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:50 smithi045 ceph-mon[20143]: pgmap v2214: 31 pgs: 31 active+clean; 577 KiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:48:50.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:50 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/743547984' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-08-19T14:48:50.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:50 smithi045 ceph-mon[20143]: osdmap e948: 12 total, 12 up, 12 in 2024-08-19T14:48:50.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:50 smithi045 ceph-mon[20143]: 1.1c scrub starts 2024-08-19T14:48:50.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:50 smithi045 ceph-mon[20143]: 1.1c scrub ok 2024-08-19T14:48:51.020 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:48:51.021 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:48:51.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:51 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:48:51.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:51 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:48:51.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:51 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:48:51.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:51 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:48:51.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:51 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:48:51.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:51 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:48:51.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:51 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:48:51.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:51 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:48:51.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:51 smithi032 ceph-mon[20004]: 1.1d deep-scrub starts 2024-08-19T14:48:51.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:51 smithi032 ceph-mon[20004]: 1.1d deep-scrub ok 2024-08-19T14:48:51.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:51 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2274837445' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-08-19T14:48:51.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:51 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-08-19T14:48:51.509 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:51 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:48:51.509 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:51 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:48:51.509 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:51 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:48:51.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:51 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:48:51.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:51 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:48:51.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:51 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:48:51.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:51 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:48:51.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:51 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:48:51.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:51 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:48:51.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:51 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:48:51.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:51 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:48:51.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:51 smithi045 ceph-mon[20143]: 1.1d deep-scrub starts 2024-08-19T14:48:51.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:51 smithi045 ceph-mon[20143]: 1.1d deep-scrub ok 2024-08-19T14:48:51.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:51 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2274837445' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-08-19T14:48:51.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:51 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-08-19T14:48:51.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:51 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:48:51.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:51 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:48:51.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:51 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:48:51.560 INFO:teuthology.orchestra.run.smithi031.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-08-19T14:48:51.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:51 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-19T14:48:51.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:51 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-19T14:48:51.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:51 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-19T14:48:51.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:51 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:48:51.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:51 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:48:51.561 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:51 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:48:51.561 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:51 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:48:51.561 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:51 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:48:51.561 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:51 smithi031 ceph-mon[16336]: 1.1d deep-scrub starts 2024-08-19T14:48:51.561 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:51 smithi031 ceph-mon[16336]: 1.1d deep-scrub ok 2024-08-19T14:48:51.561 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:51 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2274837445' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-08-19T14:48:51.561 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:51 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-08-19T14:48:51.561 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:51 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:48:51.561 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:51 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:48:51.561 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:51 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:48:51.637 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-08-19T14:48:51.889 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:48:52.184 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd reweight 5 1 2024-08-19T14:48:52.320 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:52 smithi032 ceph-mon[20004]: pgmap v2216: 31 pgs: 31 active+clean; 577 KiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:48:52.320 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:52 smithi032 ceph-mon[20004]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-08-19T14:48:52.321 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:52 smithi032 ceph-mon[20004]: osdmap e949: 12 total, 12 up, 12 in 2024-08-19T14:48:52.337 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:48:52.337 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:48:52.443 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:48:52.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:52 smithi045 ceph-mon[20143]: pgmap v2216: 31 pgs: 31 active+clean; 577 KiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:48:52.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:52 smithi045 ceph-mon[20143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-08-19T14:48:52.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:52 smithi045 ceph-mon[20143]: osdmap e949: 12 total, 12 up, 12 in 2024-08-19T14:48:52.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:52 smithi031 ceph-mon[16336]: pgmap v2216: 31 pgs: 31 active+clean; 577 KiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:48:52.558 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:52 smithi031 ceph-mon[16336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-08-19T14:48:52.558 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:52 smithi031 ceph-mon[16336]: osdmap e949: 12 total, 12 up, 12 in 2024-08-19T14:48:52.965 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-08-19T14:48:53.221 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.b/config 2024-08-19T14:48:53.243 INFO:teuthology.orchestra.run.smithi031.stderr:reweighted osd.5 to 1 (10000) 2024-08-19T14:48:53.509 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:53 smithi032 ceph-mon[20004]: 1.1e scrub starts 2024-08-19T14:48:53.509 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:53 smithi032 ceph-mon[20004]: 1.1e scrub ok 2024-08-19T14:48:53.510 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:53 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3590567775' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-08-19T14:48:53.513 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:53 smithi031 ceph-mon[16336]: 1.1e scrub starts 2024-08-19T14:48:53.513 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:53 smithi031 ceph-mon[16336]: 1.1e scrub ok 2024-08-19T14:48:53.513 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:53 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3590567775' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-08-19T14:48:53.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:53 smithi045 ceph-mon[20143]: 1.1e scrub starts 2024-08-19T14:48:53.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:53 smithi045 ceph-mon[20143]: 1.1e scrub ok 2024-08-19T14:48:53.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:53 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3590567775' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-08-19T14:48:53.734 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-08-19T14:48:53.734 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:48:53.884 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd primary-affinity 5 1 2024-08-19T14:48:54.144 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:48:54.386 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:54 smithi032 ceph-mon[20004]: pgmap v2218: 31 pgs: 31 active+clean; 577 KiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:48:54.387 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:54 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3590567775' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-08-19T14:48:54.387 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:54 smithi032 ceph-mon[20004]: osdmap e950: 12 total, 12 up, 12 in 2024-08-19T14:48:54.416 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-08-19T14:48:54.518 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:54 smithi031 ceph-mon[16336]: pgmap v2218: 31 pgs: 31 active+clean; 577 KiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:48:54.518 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:54 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3590567775' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-08-19T14:48:54.518 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:54 smithi031 ceph-mon[16336]: osdmap e950: 12 total, 12 up, 12 in 2024-08-19T14:48:54.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:54 smithi045 ceph-mon[20143]: pgmap v2218: 31 pgs: 31 active+clean; 577 KiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:48:54.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:54 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3590567775' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-08-19T14:48:54.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:54 smithi045 ceph-mon[20143]: osdmap e950: 12 total, 12 up, 12 in 2024-08-19T14:48:54.682 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:48:55.256 INFO:teuthology.orchestra.run.smithi031.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-08-19T14:48:55.327 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:48:55.327 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:48:55.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:55 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3226479360' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-08-19T14:48:55.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:55 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3226479360' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-08-19T14:48:55.562 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:55 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3226479360' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-08-19T14:48:55.988 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd reweight 6 1 2024-08-19T14:48:56.011 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-08-19T14:48:56.247 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:48:56.269 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:48:56.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:56 smithi032 ceph-mon[20004]: pgmap v2220: 31 pgs: 31 active+clean; 577 KiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:48:56.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:56 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3226479360' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-08-19T14:48:56.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:56 smithi032 ceph-mon[20004]: osdmap e951: 12 total, 12 up, 12 in 2024-08-19T14:48:56.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:56 smithi045 ceph-mon[20143]: pgmap v2220: 31 pgs: 31 active+clean; 577 KiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:48:56.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:56 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3226479360' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-08-19T14:48:56.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:56 smithi045 ceph-mon[20143]: osdmap e951: 12 total, 12 up, 12 in 2024-08-19T14:48:56.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:56 smithi031 ceph-mon[16336]: pgmap v2220: 31 pgs: 31 active+clean; 577 KiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:48:56.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:56 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3226479360' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-08-19T14:48:56.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:56 smithi031 ceph-mon[16336]: osdmap e951: 12 total, 12 up, 12 in 2024-08-19T14:48:56.787 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:48:56.788 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:48:57.261 INFO:teuthology.orchestra.run.smithi031.stderr:reweighted osd.6 to 1 (10000) 2024-08-19T14:48:57.348 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:57 smithi045 ceph-mon[20143]: osdmap e952: 12 total, 12 up, 12 in 2024-08-19T14:48:57.348 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:57 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2350602721' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-08-19T14:48:57.377 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-08-19T14:48:57.562 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:57 smithi031 ceph-mon[16336]: osdmap e952: 12 total, 12 up, 12 in 2024-08-19T14:48:57.562 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:57 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2350602721' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-08-19T14:48:57.637 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:48:57.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:57 smithi032 ceph-mon[20004]: osdmap e952: 12 total, 12 up, 12 in 2024-08-19T14:48:57.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:57 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2350602721' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-08-19T14:48:58.024 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd primary-affinity 6 1 2024-08-19T14:48:58.122 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:48:58.122 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:48:58.286 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:48:58.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:58 smithi045 ceph-mon[20143]: pgmap v2223: 31 pgs: 31 active+clean; 577 KiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:48:58.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:58 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2350602721' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-08-19T14:48:58.296 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:58 smithi045 ceph-mon[20143]: osdmap e953: 12 total, 12 up, 12 in 2024-08-19T14:48:58.524 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:58 smithi031 ceph-mon[16336]: pgmap v2223: 31 pgs: 31 active+clean; 577 KiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:48:58.524 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:58 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2350602721' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-08-19T14:48:58.524 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:58 smithi031 ceph-mon[16336]: osdmap e953: 12 total, 12 up, 12 in 2024-08-19T14:48:58.724 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-08-19T14:48:58.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:58 smithi032 ceph-mon[20004]: pgmap v2223: 31 pgs: 31 active+clean; 577 KiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:48:58.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:58 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2350602721' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-08-19T14:48:58.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:58 smithi032 ceph-mon[20004]: osdmap e953: 12 total, 12 up, 12 in 2024-08-19T14:48:58.976 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:48:59.271 INFO:teuthology.orchestra.run.smithi031.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-08-19T14:48:59.367 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:48:59 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2568211619' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-08-19T14:48:59.470 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:48:59.470 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:48:59.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:48:59 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2568211619' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-08-19T14:48:59.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:48:59 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2568211619' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-08-19T14:48:59.952 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd reweight 7 1 2024-08-19T14:49:00.118 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-08-19T14:49:00.209 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:49:00.375 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:49:00.498 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:00 smithi045 ceph-mon[20143]: pgmap v2225: 31 pgs: 31 active+clean; 577 KiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:49:00.498 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:00 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2568211619' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-08-19T14:49:00.499 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:00 smithi045 ceph-mon[20143]: osdmap e954: 12 total, 12 up, 12 in 2024-08-19T14:49:00.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:00 smithi031 ceph-mon[16336]: pgmap v2225: 31 pgs: 31 active+clean; 577 KiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:49:00.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:00 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2568211619' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-08-19T14:49:00.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:00 smithi031 ceph-mon[16336]: osdmap e954: 12 total, 12 up, 12 in 2024-08-19T14:49:00.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:00 smithi032 ceph-mon[20004]: pgmap v2225: 31 pgs: 31 active+clean; 577 KiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:49:00.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:00 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2568211619' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-08-19T14:49:00.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:00 smithi032 ceph-mon[20004]: osdmap e954: 12 total, 12 up, 12 in 2024-08-19T14:49:00.891 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:49:00.892 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:49:01.278 INFO:teuthology.orchestra.run.smithi031.stderr:reweighted osd.7 to 1 (10000) 2024-08-19T14:49:01.430 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:01 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:49:01.430 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:01 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:49:01.430 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:01 smithi045 ceph-mon[20143]: osdmap e955: 12 total, 12 up, 12 in 2024-08-19T14:49:01.431 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:01 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1967876780' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-08-19T14:49:01.459 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-08-19T14:49:01.576 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:01 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:49:01.576 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:01 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:49:01.576 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:01 smithi031 ceph-mon[16336]: osdmap e955: 12 total, 12 up, 12 in 2024-08-19T14:49:01.576 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:01 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1967876780' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-08-19T14:49:01.709 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:49:01.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:01 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:49:01.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:01 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:49:01.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:01 smithi032 ceph-mon[20004]: osdmap e955: 12 total, 12 up, 12 in 2024-08-19T14:49:01.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:01 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1967876780' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-08-19T14:49:01.987 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd primary-affinity 7 1 2024-08-19T14:49:02.224 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:49:02.225 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:49:02.247 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:49:02.537 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:02 smithi031 ceph-mon[16336]: pgmap v2228: 31 pgs: 31 active+clean; 577 KiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:49:02.537 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:02 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1967876780' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-08-19T14:49:02.537 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:02 smithi031 ceph-mon[16336]: osdmap e956: 12 total, 12 up, 12 in 2024-08-19T14:49:02.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:02 smithi045 ceph-mon[20143]: pgmap v2228: 31 pgs: 31 active+clean; 577 KiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:49:02.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:02 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1967876780' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-08-19T14:49:02.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:02 smithi045 ceph-mon[20143]: osdmap e956: 12 total, 12 up, 12 in 2024-08-19T14:49:02.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:02 smithi032 ceph-mon[20004]: pgmap v2228: 31 pgs: 31 active+clean; 577 KiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:49:02.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:02 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1967876780' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-08-19T14:49:02.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:02 smithi032 ceph-mon[20004]: osdmap e956: 12 total, 12 up, 12 in 2024-08-19T14:49:02.798 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-08-19T14:49:03.049 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:49:03.290 INFO:teuthology.orchestra.run.smithi031.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-08-19T14:49:03.456 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:03 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:49:03.456 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:03 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:49:03.456 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:03 smithi045 ceph-mon[20143]: osdmap e957: 12 total, 12 up, 12 in 2024-08-19T14:49:03.457 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:03 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3451685626' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-08-19T14:49:03.482 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:49:03.482 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:49:03.609 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:03 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:49:03.609 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:03 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:49:03.609 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:03 smithi031 ceph-mon[16336]: osdmap e957: 12 total, 12 up, 12 in 2024-08-19T14:49:03.609 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:03 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3451685626' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-08-19T14:49:03.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:03 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:49:03.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:03 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:49:03.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:03 smithi032 ceph-mon[20004]: osdmap e957: 12 total, 12 up, 12 in 2024-08-19T14:49:03.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:03 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3451685626' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-08-19T14:49:03.983 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd reweight 8 1 2024-08-19T14:49:04.123 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-08-19T14:49:04.237 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:49:04.374 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:49:04.495 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:04 smithi045 ceph-mon[20143]: pgmap v2231: 31 pgs: 31 active+clean; 577 KiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:49:04.496 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:04 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3451685626' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-08-19T14:49:04.496 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:04 smithi045 ceph-mon[20143]: osdmap e958: 12 total, 12 up, 12 in 2024-08-19T14:49:04.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:04 smithi031 ceph-mon[16336]: pgmap v2231: 31 pgs: 31 active+clean; 577 KiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:49:04.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:04 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3451685626' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-08-19T14:49:04.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:04 smithi031 ceph-mon[16336]: osdmap e958: 12 total, 12 up, 12 in 2024-08-19T14:49:04.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:04 smithi032 ceph-mon[20004]: pgmap v2231: 31 pgs: 31 active+clean; 577 KiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:49:04.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:04 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3451685626' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-08-19T14:49:04.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:04 smithi032 ceph-mon[20004]: osdmap e958: 12 total, 12 up, 12 in 2024-08-19T14:49:04.830 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:49:04.831 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:49:05.298 INFO:teuthology.orchestra.run.smithi031.stderr:reweighted osd.8 to 1 (10000) 2024-08-19T14:49:05.418 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:05 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3719865853' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-08-19T14:49:05.447 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-08-19T14:49:05.507 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:05 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3719865853' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-08-19T14:49:05.608 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:05 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3719865853' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-08-19T14:49:05.738 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:49:06.019 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd primary-affinity 8 1 2024-08-19T14:49:06.213 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:49:06.213 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:49:06.278 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:49:06.516 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:06 smithi045 ceph-mon[20143]: pgmap v2233: 31 pgs: 31 active+clean; 577 KiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:49:06.516 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:06 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3719865853' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-08-19T14:49:06.517 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:06 smithi045 ceph-mon[20143]: osdmap e959: 12 total, 12 up, 12 in 2024-08-19T14:49:06.564 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:06 smithi031 ceph-mon[16336]: pgmap v2233: 31 pgs: 31 active+clean; 577 KiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:49:06.564 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:06 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3719865853' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-08-19T14:49:06.564 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:06 smithi031 ceph-mon[16336]: osdmap e959: 12 total, 12 up, 12 in 2024-08-19T14:49:06.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:06 smithi032 ceph-mon[20004]: pgmap v2233: 31 pgs: 31 active+clean; 577 KiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:49:06.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:06 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3719865853' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-08-19T14:49:06.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:06 smithi032 ceph-mon[20004]: osdmap e959: 12 total, 12 up, 12 in 2024-08-19T14:49:06.870 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-08-19T14:49:07.120 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:49:07.307 INFO:teuthology.orchestra.run.smithi031.stderr:set osd.8 primary-affinity to 1 (8655362) 2024-08-19T14:49:07.504 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:07 smithi045 ceph-mon[20143]: from='client.18402 172.21.15.31:0/4223236668' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]: dispatch 2024-08-19T14:49:07.613 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:49:07.613 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:49:07.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:07 smithi031 ceph-mon[16336]: from='client.18402 172.21.15.31:0/4223236668' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]: dispatch 2024-08-19T14:49:07.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:07 smithi032 ceph-mon[20004]: from='client.18402 172.21.15.31:0/4223236668' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]: dispatch 2024-08-19T14:49:08.049 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd reweight 9 1 2024-08-19T14:49:08.204 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-08-19T14:49:08.305 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:49:08.454 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:49:08.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:08 smithi045 ceph-mon[20143]: pgmap v2235: 31 pgs: 31 active+clean; 577 KiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:49:08.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:08 smithi045 ceph-mon[20143]: from='client.18402 172.21.15.31:0/4223236668' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]': finished 2024-08-19T14:49:08.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:08 smithi045 ceph-mon[20143]: osdmap e960: 12 total, 12 up, 12 in 2024-08-19T14:49:08.568 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:08 smithi031 ceph-mon[16336]: pgmap v2235: 31 pgs: 31 active+clean; 577 KiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:49:08.568 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:08 smithi031 ceph-mon[16336]: from='client.18402 172.21.15.31:0/4223236668' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]': finished 2024-08-19T14:49:08.568 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:08 smithi031 ceph-mon[16336]: osdmap e960: 12 total, 12 up, 12 in 2024-08-19T14:49:08.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:08 smithi032 ceph-mon[20004]: pgmap v2235: 31 pgs: 31 active+clean; 577 KiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:49:08.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:08 smithi032 ceph-mon[20004]: from='client.18402 172.21.15.31:0/4223236668' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]': finished 2024-08-19T14:49:08.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:08 smithi032 ceph-mon[20004]: osdmap e960: 12 total, 12 up, 12 in 2024-08-19T14:49:08.971 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:49:08.972 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:49:09.314 INFO:teuthology.orchestra.run.smithi031.stderr:reweighted osd.9 to 1 (10000) 2024-08-19T14:49:09.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:09 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2573056111' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-08-19T14:49:09.605 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-08-19T14:49:09.614 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:09 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2573056111' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-08-19T14:49:09.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:09 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2573056111' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-08-19T14:49:09.860 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.c/config 2024-08-19T14:49:09.955 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd primary-affinity 9 1 2024-08-19T14:49:10.209 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:49:10.312 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-08-19T14:49:10.313 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:49:10.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:10 smithi045 ceph-mon[20143]: pgmap v2237: 31 pgs: 31 active+clean; 577 KiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:49:10.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:10 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2573056111' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-08-19T14:49:10.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:10 smithi045 ceph-mon[20143]: osdmap e961: 12 total, 12 up, 12 in 2024-08-19T14:49:10.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:10 smithi031 ceph-mon[16336]: pgmap v2237: 31 pgs: 31 active+clean; 577 KiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:49:10.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:10 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2573056111' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-08-19T14:49:10.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:10 smithi031 ceph-mon[16336]: osdmap e961: 12 total, 12 up, 12 in 2024-08-19T14:49:10.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:10 smithi032 ceph-mon[20004]: pgmap v2237: 31 pgs: 31 active+clean; 577 KiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:49:10.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:10 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2573056111' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-08-19T14:49:10.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:10 smithi032 ceph-mon[20004]: osdmap e961: 12 total, 12 up, 12 in 2024-08-19T14:49:10.938 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-08-19T14:49:11.189 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:49:11.321 INFO:teuthology.orchestra.run.smithi031.stderr:set osd.9 primary-affinity to 1 (8655362) 2024-08-19T14:49:11.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:11 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3910628814' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]: dispatch 2024-08-19T14:49:11.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:11 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3910628814' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]: dispatch 2024-08-19T14:49:11.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:11 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3910628814' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]: dispatch 2024-08-19T14:49:12.093 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd reweight 11 1 2024-08-19T14:49:12.321 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:49:12.322 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:49:12.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:12 smithi031 ceph-mon[16336]: pgmap v2239: 31 pgs: 31 active+clean; 577 KiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:49:12.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:12 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3910628814' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]': finished 2024-08-19T14:49:12.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:12 smithi031 ceph-mon[16336]: osdmap e962: 12 total, 12 up, 12 in 2024-08-19T14:49:12.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:12 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:49:12.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:12 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:49:12.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:12 smithi032 ceph-mon[20004]: pgmap v2239: 31 pgs: 31 active+clean; 577 KiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:49:12.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:12 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3910628814' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]': finished 2024-08-19T14:49:12.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:12 smithi032 ceph-mon[20004]: osdmap e962: 12 total, 12 up, 12 in 2024-08-19T14:49:12.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:12 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:49:12.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:12 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:49:12.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:12 smithi045 ceph-mon[20143]: pgmap v2239: 31 pgs: 31 active+clean; 577 KiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:49:12.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:12 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3910628814' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]': finished 2024-08-19T14:49:12.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:12 smithi045 ceph-mon[20143]: osdmap e962: 12 total, 12 up, 12 in 2024-08-19T14:49:12.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:12 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:49:12.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:12 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:49:12.885 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:49:12.920 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-08-19T14:49:13.424 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:13 smithi031 ceph-mon[16336]: osdmap e963: 12 total, 12 up, 12 in 2024-08-19T14:49:13.424 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:13 smithi031 ceph-mon[16336]: pgmap v2242: 31 pgs: 31 active+clean; 577 KiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:49:13.458 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:49:13.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:13 smithi032 ceph-mon[20004]: osdmap e963: 12 total, 12 up, 12 in 2024-08-19T14:49:13.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:13 smithi032 ceph-mon[20004]: pgmap v2242: 31 pgs: 31 active+clean; 577 KiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:49:13.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:13 smithi045 ceph-mon[20143]: osdmap e963: 12 total, 12 up, 12 in 2024-08-19T14:49:13.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:13 smithi045 ceph-mon[20143]: pgmap v2242: 31 pgs: 31 active+clean; 577 KiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:49:13.968 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:49:13.968 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:49:14.336 INFO:teuthology.orchestra.run.smithi031.stderr:reweighted osd.11 to 1 (10000) 2024-08-19T14:49:14.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:14 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2305376172' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-08-19T14:49:14.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:14 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2305376172' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-08-19T14:49:14.777 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-08-19T14:49:14.799 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:14 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2305376172' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-08-19T14:49:15.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:15 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2305376172' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-08-19T14:49:15.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:15 smithi031 ceph-mon[16336]: osdmap e964: 12 total, 12 up, 12 in 2024-08-19T14:49:15.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:15 smithi031 ceph-mon[16336]: pgmap v2244: 31 pgs: 31 active+clean; 577 KiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:49:15.560 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd primary-affinity 11 1 2024-08-19T14:49:15.561 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:49:15.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:15 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2305376172' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-08-19T14:49:15.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:15 smithi032 ceph-mon[20004]: osdmap e964: 12 total, 12 up, 12 in 2024-08-19T14:49:15.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:15 smithi032 ceph-mon[20004]: pgmap v2244: 31 pgs: 31 active+clean; 577 KiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:49:15.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:15 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2305376172' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-08-19T14:49:15.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:15 smithi045 ceph-mon[20143]: osdmap e964: 12 total, 12 up, 12 in 2024-08-19T14:49:15.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:15 smithi045 ceph-mon[20143]: pgmap v2244: 31 pgs: 31 active+clean; 577 KiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:49:16.030 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-08-19T14:49:16.030 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-19T14:49:16.036 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:49:17.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:17 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:49:17.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:17 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:49:17.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:17 smithi031 ceph-mon[16336]: osdmap e965: 12 total, 12 up, 12 in 2024-08-19T14:49:17.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:17 smithi031 ceph-mon[16336]: pgmap v2246: 31 pgs: 31 active+clean; 577 KiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:49:17.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:17 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:49:17.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:17 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:49:17.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:17 smithi032 ceph-mon[20004]: osdmap e965: 12 total, 12 up, 12 in 2024-08-19T14:49:17.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:17 smithi032 ceph-mon[20004]: pgmap v2246: 31 pgs: 31 active+clean; 577 KiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:49:17.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:17 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:49:17.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:17 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:49:17.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:17 smithi045 ceph-mon[20143]: osdmap e965: 12 total, 12 up, 12 in 2024-08-19T14:49:17.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:17 smithi045 ceph-mon[20143]: pgmap v2246: 31 pgs: 31 active+clean; 577 KiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:49:18.349 INFO:teuthology.orchestra.run.smithi031.stderr:set osd.11 primary-affinity to 1 (8655362) 2024-08-19T14:49:18.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:18 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3623176406' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]: dispatch 2024-08-19T14:49:18.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:18 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:49:18.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:18 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:49:18.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:18 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3623176406' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]: dispatch 2024-08-19T14:49:18.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:18 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:49:18.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:18 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:49:18.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:18 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3623176406' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]: dispatch 2024-08-19T14:49:18.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:18 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:49:18.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:18 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:49:19.095 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd reweight 10 1 2024-08-19T14:49:19.352 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:49:19.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:19 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3623176406' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]': finished 2024-08-19T14:49:19.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:19 smithi031 ceph-mon[16336]: osdmap e966: 12 total, 12 up, 12 in 2024-08-19T14:49:19.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:19 smithi031 ceph-mon[16336]: pgmap v2248: 31 pgs: 31 active+clean; 577 KiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:49:19.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:19 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3623176406' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]': finished 2024-08-19T14:49:19.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:19 smithi032 ceph-mon[20004]: osdmap e966: 12 total, 12 up, 12 in 2024-08-19T14:49:19.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:19 smithi032 ceph-mon[20004]: pgmap v2248: 31 pgs: 31 active+clean; 577 KiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:49:19.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:19 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3623176406' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]': finished 2024-08-19T14:49:19.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:19 smithi045 ceph-mon[20143]: osdmap e966: 12 total, 12 up, 12 in 2024-08-19T14:49:19.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:19 smithi045 ceph-mon[20143]: pgmap v2248: 31 pgs: 31 active+clean; 577 KiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:49:20.368 INFO:teuthology.orchestra.run.smithi031.stderr:reweighted osd.10 to 1 (10000) 2024-08-19T14:49:20.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:20 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/121223627' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-08-19T14:49:20.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:20 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/121223627' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-08-19T14:49:20.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:20 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/121223627' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-08-19T14:49:21.136 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd primary-affinity 10 1 2024-08-19T14:49:21.391 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:49:21.634 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:21 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/121223627' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-08-19T14:49:21.634 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:21 smithi031 ceph-mon[16336]: osdmap e967: 12 total, 12 up, 12 in 2024-08-19T14:49:21.634 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:21 smithi031 ceph-mon[16336]: pgmap v2250: 31 pgs: 31 active+clean; 577 KiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:49:21.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:21 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/121223627' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-08-19T14:49:21.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:21 smithi032 ceph-mon[20004]: osdmap e967: 12 total, 12 up, 12 in 2024-08-19T14:49:21.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:21 smithi032 ceph-mon[20004]: pgmap v2250: 31 pgs: 31 active+clean; 577 KiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:49:21.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:21 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/121223627' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-08-19T14:49:21.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:21 smithi045 ceph-mon[20143]: osdmap e967: 12 total, 12 up, 12 in 2024-08-19T14:49:21.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:21 smithi045 ceph-mon[20143]: pgmap v2250: 31 pgs: 31 active+clean; 577 KiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:49:22.379 INFO:teuthology.orchestra.run.smithi031.stderr:set osd.10 primary-affinity to 1 (8655362) 2024-08-19T14:49:22.665 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:22 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2333832660' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]: dispatch 2024-08-19T14:49:22.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:22 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2333832660' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]: dispatch 2024-08-19T14:49:22.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:22 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2333832660' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]: dispatch 2024-08-19T14:49:23.146 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd reweight 2 1 2024-08-19T14:49:23.402 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:49:23.638 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:23 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2333832660' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]': finished 2024-08-19T14:49:23.639 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:23 smithi031 ceph-mon[16336]: osdmap e968: 12 total, 12 up, 12 in 2024-08-19T14:49:23.639 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:23 smithi031 ceph-mon[16336]: pgmap v2252: 31 pgs: 31 active+clean; 577 KiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:49:23.639 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:23 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:49:23.639 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:23 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:49:23.659 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:23 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2333832660' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]': finished 2024-08-19T14:49:23.659 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:23 smithi032 ceph-mon[20004]: osdmap e968: 12 total, 12 up, 12 in 2024-08-19T14:49:23.659 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:23 smithi032 ceph-mon[20004]: pgmap v2252: 31 pgs: 31 active+clean; 577 KiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:49:23.659 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:23 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:49:23.659 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:23 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:49:23.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:23 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2333832660' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]': finished 2024-08-19T14:49:23.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:23 smithi045 ceph-mon[20143]: osdmap e968: 12 total, 12 up, 12 in 2024-08-19T14:49:23.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:23 smithi045 ceph-mon[20143]: pgmap v2252: 31 pgs: 31 active+clean; 577 KiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:49:23.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:23 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:49:23.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:23 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:49:24.393 INFO:teuthology.orchestra.run.smithi031.stderr:reweighted osd.2 to 1 (10000) 2024-08-19T14:49:24.698 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:24 smithi045 ceph-mon[20143]: osdmap e969: 12 total, 12 up, 12 in 2024-08-19T14:49:24.698 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:24 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3842563598' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-08-19T14:49:24.742 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:24 smithi031 ceph-mon[16336]: osdmap e969: 12 total, 12 up, 12 in 2024-08-19T14:49:24.742 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:24 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3842563598' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-08-19T14:49:24.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:24 smithi032 ceph-mon[20004]: osdmap e969: 12 total, 12 up, 12 in 2024-08-19T14:49:24.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:24 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3842563598' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-08-19T14:49:25.152 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd primary-affinity 2 1 2024-08-19T14:49:25.410 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:49:25.666 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:25 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:49:25.667 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:25 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:49:25.667 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:25 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3842563598' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-08-19T14:49:25.667 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:25 smithi031 ceph-mon[16336]: osdmap e970: 12 total, 12 up, 12 in 2024-08-19T14:49:25.667 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:25 smithi031 ceph-mon[16336]: pgmap v2255: 31 pgs: 31 active+clean; 577 KiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:49:25.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:25 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:49:25.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:25 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:49:25.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:25 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3842563598' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-08-19T14:49:25.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:25 smithi032 ceph-mon[20004]: osdmap e970: 12 total, 12 up, 12 in 2024-08-19T14:49:25.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:25 smithi032 ceph-mon[20004]: pgmap v2255: 31 pgs: 31 active+clean; 577 KiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:49:25.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:25 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:49:25.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:25 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:49:25.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:25 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3842563598' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-08-19T14:49:25.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:25 smithi045 ceph-mon[20143]: osdmap e970: 12 total, 12 up, 12 in 2024-08-19T14:49:25.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:25 smithi045 ceph-mon[20143]: pgmap v2255: 31 pgs: 31 active+clean; 577 KiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:49:26.404 INFO:teuthology.orchestra.run.smithi031.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-08-19T14:49:26.699 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:26 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2845662179' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-08-19T14:49:26.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:26 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2845662179' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-08-19T14:49:26.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:26 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2845662179' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-08-19T14:49:27.161 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-08-19T14:49:27.161 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-08-19T14:49:27.161 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-08-19T14:49:27.161 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-08-19T14:49:27.161 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-08-19T14:49:27.162 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd dump --format=json 2024-08-19T14:49:27.410 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:27 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2845662179' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-08-19T14:49:27.410 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:27 smithi031 ceph-mon[16336]: osdmap e971: 12 total, 12 up, 12 in 2024-08-19T14:49:27.423 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:49:27.670 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:27 smithi031 ceph-mon[16336]: pgmap v2257: 31 pgs: 31 active+clean; 577 KiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:49:27.670 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:27 smithi031 ceph-mon[16336]: 1.12 scrub starts 2024-08-19T14:49:27.670 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:27 smithi031 ceph-mon[16336]: 1.12 scrub ok 2024-08-19T14:49:27.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:27 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2845662179' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-08-19T14:49:27.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:27 smithi032 ceph-mon[20004]: osdmap e971: 12 total, 12 up, 12 in 2024-08-19T14:49:27.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:27 smithi032 ceph-mon[20004]: pgmap v2257: 31 pgs: 31 active+clean; 577 KiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:49:27.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:27 smithi032 ceph-mon[20004]: 1.12 scrub starts 2024-08-19T14:49:27.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:27 smithi032 ceph-mon[20004]: 1.12 scrub ok 2024-08-19T14:49:27.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:27 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2845662179' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-08-19T14:49:27.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:27 smithi045 ceph-mon[20143]: osdmap e971: 12 total, 12 up, 12 in 2024-08-19T14:49:27.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:27 smithi045 ceph-mon[20143]: pgmap v2257: 31 pgs: 31 active+clean; 577 KiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:49:27.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:27 smithi045 ceph-mon[20143]: 1.12 scrub starts 2024-08-19T14:49:27.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:27 smithi045 ceph-mon[20143]: 1.12 scrub ok 2024-08-19T14:49:28.052 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:49:28.052 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":972,"fsid":"ed7d1c6e-5e33-11ef-bccf-c7b262605968","created":"2024-08-19T14:05:22.842169+0000","modified":"2024-08-19T14:49:27.403792+0000","last_up_change":"2024-08-19T14:35:10.517930+0000","last_in_change":"2024-08-19T14:48:28.125677+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":29,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-19T14:08:15.966221+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":31,"pg_placement_num_target":31,"pg_num_target":31,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"278","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"276","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5499999523162842,"score_stable":1.5499999523162842,"optimal_score":1,"raw_score_acting":1.5499999523162842,"raw_score_stable":1.5499999523162842,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d7a0fff3-23c6-4ee3-a275-a844862aabfb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":936,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":1108945097},{"type":"v1","addr":"172.21.15.31:6803","nonce":1108945097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":1108945097},{"type":"v1","addr":"172.21.15.31:6805","nonce":1108945097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1108945097},{"type":"v1","addr":"172.21.15.31:6809","nonce":1108945097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":1108945097},{"type":"v1","addr":"172.21.15.31:6807","nonce":1108945097}]},"public_addr":"172.21.15.31:6803/1108945097","cluster_addr":"172.21.15.31:6805/1108945097","heartbeat_back_addr":"172.21.15.31:6809/1108945097","heartbeat_front_addr":"172.21.15.31:6807/1108945097","state":["exists","up"]},{"osd":1,"uuid":"145d5bb4-74dd-440f-bcc9-7e9ebda9c5d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":939,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":3236186806},{"type":"v1","addr":"172.21.15.31:6811","nonce":3236186806}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":3236186806},{"type":"v1","addr":"172.21.15.31:6813","nonce":3236186806}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":3236186806},{"type":"v1","addr":"172.21.15.31:6817","nonce":3236186806}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":3236186806},{"type":"v1","addr":"172.21.15.31:6815","nonce":3236186806}]},"public_addr":"172.21.15.31:6811/3236186806","cluster_addr":"172.21.15.31:6813/3236186806","heartbeat_back_addr":"172.21.15.31:6817/3236186806","heartbeat_front_addr":"172.21.15.31:6815/3236186806","state":["exists","up"]},{"osd":2,"uuid":"376c18d0-4768-46a0-bb60-e79c88ca909c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":18,"last_clean_end":564,"up_from":634,"up_thru":971,"down_at":565,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":656997856},{"type":"v1","addr":"172.21.15.31:6819","nonce":656997856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":656997856},{"type":"v1","addr":"172.21.15.31:6821","nonce":656997856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":656997856},{"type":"v1","addr":"172.21.15.31:6825","nonce":656997856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":656997856},{"type":"v1","addr":"172.21.15.31:6823","nonce":656997856}]},"public_addr":"172.21.15.31:6819/656997856","cluster_addr":"172.21.15.31:6821/656997856","heartbeat_back_addr":"172.21.15.31:6825/656997856","heartbeat_front_addr":"172.21.15.31:6823/656997856","state":["exists","up"]},{"osd":3,"uuid":"a787ea9f-b9ba-450f-b388-614cfba90634","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":939,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":2288082716},{"type":"v1","addr":"172.21.15.31:6827","nonce":2288082716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":2288082716},{"type":"v1","addr":"172.21.15.31:6829","nonce":2288082716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2288082716},{"type":"v1","addr":"172.21.15.31:6833","nonce":2288082716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":2288082716},{"type":"v1","addr":"172.21.15.31:6831","nonce":2288082716}]},"public_addr":"172.21.15.31:6827/2288082716","cluster_addr":"172.21.15.31:6829/2288082716","heartbeat_back_addr":"172.21.15.31:6833/2288082716","heartbeat_front_addr":"172.21.15.31:6831/2288082716","state":["exists","up"]},{"osd":4,"uuid":"7859751a-12f5-4a57-a332-435360479d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":918,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":3934739984},{"type":"v1","addr":"172.21.15.32:6801","nonce":3934739984}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":3934739984},{"type":"v1","addr":"172.21.15.32:6803","nonce":3934739984}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":3934739984},{"type":"v1","addr":"172.21.15.32:6807","nonce":3934739984}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":3934739984},{"type":"v1","addr":"172.21.15.32:6805","nonce":3934739984}]},"public_addr":"172.21.15.32:6801/3934739984","cluster_addr":"172.21.15.32:6803/3934739984","heartbeat_back_addr":"172.21.15.32:6807/3934739984","heartbeat_front_addr":"172.21.15.32:6805/3934739984","state":["exists","up"]},{"osd":5,"uuid":"017f3eef-ac60-4b82-8350-919017cb4ccd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":951,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":526388985},{"type":"v1","addr":"172.21.15.32:6809","nonce":526388985}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":526388985},{"type":"v1","addr":"172.21.15.32:6811","nonce":526388985}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":526388985},{"type":"v1","addr":"172.21.15.32:6815","nonce":526388985}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":526388985},{"type":"v1","addr":"172.21.15.32:6813","nonce":526388985}]},"public_addr":"172.21.15.32:6809/526388985","cluster_addr":"172.21.15.32:6811/526388985","heartbeat_back_addr":"172.21.15.32:6815/526388985","heartbeat_front_addr":"172.21.15.32:6813/526388985","state":["exists","up"]},{"osd":6,"uuid":"44aa8229-0268-49d7-bd57-3bf1f384e5e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":939,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":1131035844},{"type":"v1","addr":"172.21.15.32:6817","nonce":1131035844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":1131035844},{"type":"v1","addr":"172.21.15.32:6819","nonce":1131035844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":1131035844},{"type":"v1","addr":"172.21.15.32:6823","nonce":1131035844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":1131035844},{"type":"v1","addr":"172.21.15.32:6821","nonce":1131035844}]},"public_addr":"172.21.15.32:6817/1131035844","cluster_addr":"172.21.15.32:6819/1131035844","heartbeat_back_addr":"172.21.15.32:6823/1131035844","heartbeat_front_addr":"172.21.15.32:6821/1131035844","state":["exists","up"]},{"osd":7,"uuid":"263433e5-70b6-4192-86ff-dc2ad95927eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":939,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":2014939947},{"type":"v1","addr":"172.21.15.32:6825","nonce":2014939947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":2014939947},{"type":"v1","addr":"172.21.15.32:6827","nonce":2014939947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":2014939947},{"type":"v1","addr":"172.21.15.32:6831","nonce":2014939947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":2014939947},{"type":"v1","addr":"172.21.15.32:6829","nonce":2014939947}]},"public_addr":"172.21.15.32:6825/2014939947","cluster_addr":"172.21.15.32:6827/2014939947","heartbeat_back_addr":"172.21.15.32:6831/2014939947","heartbeat_front_addr":"172.21.15.32:6829/2014939947","state":["exists","up"]},{"osd":8,"uuid":"2bbcad36-0c74-4528-9fb9-a8812afa1662","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":939,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1445140892},{"type":"v1","addr":"172.21.15.45:6801","nonce":1445140892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1445140892},{"type":"v1","addr":"172.21.15.45:6803","nonce":1445140892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1445140892},{"type":"v1","addr":"172.21.15.45:6807","nonce":1445140892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1445140892},{"type":"v1","addr":"172.21.15.45:6805","nonce":1445140892}]},"public_addr":"172.21.15.45:6801/1445140892","cluster_addr":"172.21.15.45:6803/1445140892","heartbeat_back_addr":"172.21.15.45:6807/1445140892","heartbeat_front_addr":"172.21.15.45:6805/1445140892","state":["exists","up"]},{"osd":9,"uuid":"c8ee4277-3bf9-46bf-9cc0-152c1c4b9508","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":962,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1213787169},{"type":"v1","addr":"172.21.15.45:6809","nonce":1213787169}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1213787169},{"type":"v1","addr":"172.21.15.45:6811","nonce":1213787169}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1213787169},{"type":"v1","addr":"172.21.15.45:6815","nonce":1213787169}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1213787169},{"type":"v1","addr":"172.21.15.45:6813","nonce":1213787169}]},"public_addr":"172.21.15.45:6809/1213787169","cluster_addr":"172.21.15.45:6811/1213787169","heartbeat_back_addr":"172.21.15.45:6815/1213787169","heartbeat_front_addr":"172.21.15.45:6813/1213787169","state":["exists","up"]},{"osd":10,"uuid":"29031395-5f5c-40f8-ac90-9f6ac48e5c9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":80,"last_clean_end":529,"up_from":539,"up_thru":936,"down_at":530,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":1069743893},{"type":"v1","addr":"172.21.15.45:6817","nonce":1069743893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":1069743893},{"type":"v1","addr":"172.21.15.45:6819","nonce":1069743893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":1069743893},{"type":"v1","addr":"172.21.15.45:6823","nonce":1069743893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":1069743893},{"type":"v1","addr":"172.21.15.45:6821","nonce":1069743893}]},"public_addr":"172.21.15.45:6817/1069743893","cluster_addr":"172.21.15.45:6819/1069743893","heartbeat_back_addr":"172.21.15.45:6823/1069743893","heartbeat_front_addr":"172.21.15.45:6821/1069743893","state":["exists","up"]},{"osd":11,"uuid":"5ef0db4b-0fef-4ea0-a19b-d3677e082c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":86,"up_thru":939,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2438396298},{"type":"v1","addr":"172.21.15.45:6825","nonce":2438396298}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":2438396298},{"type":"v1","addr":"172.21.15.45:6827","nonce":2438396298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":2438396298},{"type":"v1","addr":"172.21.15.45:6831","nonce":2438396298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":2438396298},{"type":"v1","addr":"172.21.15.45:6829","nonce":2438396298}]},"public_addr":"172.21.15.45:6825/2438396298","cluster_addr":"172.21.15.45:6827/2438396298","heartbeat_back_addr":"172.21.15.45:6831/2438396298","heartbeat_front_addr":"172.21.15.45:6829/2438396298","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T14:49:01.777625+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T14:49:17.933457+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T14:48:59.975037+0000","dead_epoch":564},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T14:49:24.369534+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T14:48:38.666696+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T14:49:16.245415+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T14:49:11.452371+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T14:48:27.847092+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T14:48:29.109925+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T14:49:22.840386+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T14:48:12.770531+0000","dead_epoch":529},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T14:48:23.685590+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"1.0","osds":[2,9,0]}],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/1451745969":"2024-08-20T14:06:13.660657+0000","172.21.15.31:0/1515698918":"2024-08-20T14:06:13.660657+0000","172.21.15.31:6801/3297539242":"2024-08-20T14:06:13.660657+0000","172.21.15.31:6800/3297539242":"2024-08-20T14:06:13.660657+0000","172.21.15.31:0/2217423818":"2024-08-20T14:05:48.827576+0000","172.21.15.31:0/3353498281":"2024-08-20T14:06:13.660657+0000","172.21.15.31:0/4013594220":"2024-08-20T14:05:48.827576+0000","172.21.15.31:0/1654518065":"2024-08-20T14:05:48.827576+0000","172.21.15.31:6801/2368821258":"2024-08-20T14:05:48.827576+0000","172.21.15.31:6800/2368821258":"2024-08-20T14:05:48.827576+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-19T14:49:28.690 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:28 smithi031 ceph-mon[16336]: osdmap e972: 12 total, 12 up, 12 in 2024-08-19T14:49:28.691 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:28 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3555098302' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-08-19T14:49:28.723 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-08-19T14:49:28.723 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd dump --format=json 2024-08-19T14:49:28.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:28 smithi032 ceph-mon[20004]: osdmap e972: 12 total, 12 up, 12 in 2024-08-19T14:49:28.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:28 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3555098302' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-08-19T14:49:28.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:28 smithi045 ceph-mon[20143]: osdmap e972: 12 total, 12 up, 12 in 2024-08-19T14:49:28.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:28 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3555098302' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-08-19T14:49:28.975 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:49:29.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:29 smithi031 ceph-mon[16336]: pgmap v2259: 31 pgs: 31 active+clean; 577 KiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:49:29.619 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:49:29.619 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":972,"fsid":"ed7d1c6e-5e33-11ef-bccf-c7b262605968","created":"2024-08-19T14:05:22.842169+0000","modified":"2024-08-19T14:49:27.403792+0000","last_up_change":"2024-08-19T14:35:10.517930+0000","last_in_change":"2024-08-19T14:48:28.125677+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":29,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-19T14:08:15.966221+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":31,"pg_placement_num_target":31,"pg_num_target":31,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"278","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"276","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5499999523162842,"score_stable":1.5499999523162842,"optimal_score":1,"raw_score_acting":1.5499999523162842,"raw_score_stable":1.5499999523162842,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d7a0fff3-23c6-4ee3-a275-a844862aabfb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":936,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":1108945097},{"type":"v1","addr":"172.21.15.31:6803","nonce":1108945097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":1108945097},{"type":"v1","addr":"172.21.15.31:6805","nonce":1108945097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1108945097},{"type":"v1","addr":"172.21.15.31:6809","nonce":1108945097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":1108945097},{"type":"v1","addr":"172.21.15.31:6807","nonce":1108945097}]},"public_addr":"172.21.15.31:6803/1108945097","cluster_addr":"172.21.15.31:6805/1108945097","heartbeat_back_addr":"172.21.15.31:6809/1108945097","heartbeat_front_addr":"172.21.15.31:6807/1108945097","state":["exists","up"]},{"osd":1,"uuid":"145d5bb4-74dd-440f-bcc9-7e9ebda9c5d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":939,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":3236186806},{"type":"v1","addr":"172.21.15.31:6811","nonce":3236186806}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":3236186806},{"type":"v1","addr":"172.21.15.31:6813","nonce":3236186806}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":3236186806},{"type":"v1","addr":"172.21.15.31:6817","nonce":3236186806}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":3236186806},{"type":"v1","addr":"172.21.15.31:6815","nonce":3236186806}]},"public_addr":"172.21.15.31:6811/3236186806","cluster_addr":"172.21.15.31:6813/3236186806","heartbeat_back_addr":"172.21.15.31:6817/3236186806","heartbeat_front_addr":"172.21.15.31:6815/3236186806","state":["exists","up"]},{"osd":2,"uuid":"376c18d0-4768-46a0-bb60-e79c88ca909c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":18,"last_clean_end":564,"up_from":634,"up_thru":971,"down_at":565,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":656997856},{"type":"v1","addr":"172.21.15.31:6819","nonce":656997856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":656997856},{"type":"v1","addr":"172.21.15.31:6821","nonce":656997856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":656997856},{"type":"v1","addr":"172.21.15.31:6825","nonce":656997856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":656997856},{"type":"v1","addr":"172.21.15.31:6823","nonce":656997856}]},"public_addr":"172.21.15.31:6819/656997856","cluster_addr":"172.21.15.31:6821/656997856","heartbeat_back_addr":"172.21.15.31:6825/656997856","heartbeat_front_addr":"172.21.15.31:6823/656997856","state":["exists","up"]},{"osd":3,"uuid":"a787ea9f-b9ba-450f-b388-614cfba90634","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":939,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":2288082716},{"type":"v1","addr":"172.21.15.31:6827","nonce":2288082716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":2288082716},{"type":"v1","addr":"172.21.15.31:6829","nonce":2288082716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2288082716},{"type":"v1","addr":"172.21.15.31:6833","nonce":2288082716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":2288082716},{"type":"v1","addr":"172.21.15.31:6831","nonce":2288082716}]},"public_addr":"172.21.15.31:6827/2288082716","cluster_addr":"172.21.15.31:6829/2288082716","heartbeat_back_addr":"172.21.15.31:6833/2288082716","heartbeat_front_addr":"172.21.15.31:6831/2288082716","state":["exists","up"]},{"osd":4,"uuid":"7859751a-12f5-4a57-a332-435360479d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":918,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":3934739984},{"type":"v1","addr":"172.21.15.32:6801","nonce":3934739984}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":3934739984},{"type":"v1","addr":"172.21.15.32:6803","nonce":3934739984}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":3934739984},{"type":"v1","addr":"172.21.15.32:6807","nonce":3934739984}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":3934739984},{"type":"v1","addr":"172.21.15.32:6805","nonce":3934739984}]},"public_addr":"172.21.15.32:6801/3934739984","cluster_addr":"172.21.15.32:6803/3934739984","heartbeat_back_addr":"172.21.15.32:6807/3934739984","heartbeat_front_addr":"172.21.15.32:6805/3934739984","state":["exists","up"]},{"osd":5,"uuid":"017f3eef-ac60-4b82-8350-919017cb4ccd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":951,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":526388985},{"type":"v1","addr":"172.21.15.32:6809","nonce":526388985}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":526388985},{"type":"v1","addr":"172.21.15.32:6811","nonce":526388985}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":526388985},{"type":"v1","addr":"172.21.15.32:6815","nonce":526388985}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":526388985},{"type":"v1","addr":"172.21.15.32:6813","nonce":526388985}]},"public_addr":"172.21.15.32:6809/526388985","cluster_addr":"172.21.15.32:6811/526388985","heartbeat_back_addr":"172.21.15.32:6815/526388985","heartbeat_front_addr":"172.21.15.32:6813/526388985","state":["exists","up"]},{"osd":6,"uuid":"44aa8229-0268-49d7-bd57-3bf1f384e5e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":939,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":1131035844},{"type":"v1","addr":"172.21.15.32:6817","nonce":1131035844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":1131035844},{"type":"v1","addr":"172.21.15.32:6819","nonce":1131035844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":1131035844},{"type":"v1","addr":"172.21.15.32:6823","nonce":1131035844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":1131035844},{"type":"v1","addr":"172.21.15.32:6821","nonce":1131035844}]},"public_addr":"172.21.15.32:6817/1131035844","cluster_addr":"172.21.15.32:6819/1131035844","heartbeat_back_addr":"172.21.15.32:6823/1131035844","heartbeat_front_addr":"172.21.15.32:6821/1131035844","state":["exists","up"]},{"osd":7,"uuid":"263433e5-70b6-4192-86ff-dc2ad95927eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":939,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":2014939947},{"type":"v1","addr":"172.21.15.32:6825","nonce":2014939947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":2014939947},{"type":"v1","addr":"172.21.15.32:6827","nonce":2014939947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":2014939947},{"type":"v1","addr":"172.21.15.32:6831","nonce":2014939947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":2014939947},{"type":"v1","addr":"172.21.15.32:6829","nonce":2014939947}]},"public_addr":"172.21.15.32:6825/2014939947","cluster_addr":"172.21.15.32:6827/2014939947","heartbeat_back_addr":"172.21.15.32:6831/2014939947","heartbeat_front_addr":"172.21.15.32:6829/2014939947","state":["exists","up"]},{"osd":8,"uuid":"2bbcad36-0c74-4528-9fb9-a8812afa1662","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":939,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1445140892},{"type":"v1","addr":"172.21.15.45:6801","nonce":1445140892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1445140892},{"type":"v1","addr":"172.21.15.45:6803","nonce":1445140892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1445140892},{"type":"v1","addr":"172.21.15.45:6807","nonce":1445140892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1445140892},{"type":"v1","addr":"172.21.15.45:6805","nonce":1445140892}]},"public_addr":"172.21.15.45:6801/1445140892","cluster_addr":"172.21.15.45:6803/1445140892","heartbeat_back_addr":"172.21.15.45:6807/1445140892","heartbeat_front_addr":"172.21.15.45:6805/1445140892","state":["exists","up"]},{"osd":9,"uuid":"c8ee4277-3bf9-46bf-9cc0-152c1c4b9508","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":962,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1213787169},{"type":"v1","addr":"172.21.15.45:6809","nonce":1213787169}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1213787169},{"type":"v1","addr":"172.21.15.45:6811","nonce":1213787169}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1213787169},{"type":"v1","addr":"172.21.15.45:6815","nonce":1213787169}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1213787169},{"type":"v1","addr":"172.21.15.45:6813","nonce":1213787169}]},"public_addr":"172.21.15.45:6809/1213787169","cluster_addr":"172.21.15.45:6811/1213787169","heartbeat_back_addr":"172.21.15.45:6815/1213787169","heartbeat_front_addr":"172.21.15.45:6813/1213787169","state":["exists","up"]},{"osd":10,"uuid":"29031395-5f5c-40f8-ac90-9f6ac48e5c9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":80,"last_clean_end":529,"up_from":539,"up_thru":936,"down_at":530,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":1069743893},{"type":"v1","addr":"172.21.15.45:6817","nonce":1069743893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":1069743893},{"type":"v1","addr":"172.21.15.45:6819","nonce":1069743893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":1069743893},{"type":"v1","addr":"172.21.15.45:6823","nonce":1069743893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":1069743893},{"type":"v1","addr":"172.21.15.45:6821","nonce":1069743893}]},"public_addr":"172.21.15.45:6817/1069743893","cluster_addr":"172.21.15.45:6819/1069743893","heartbeat_back_addr":"172.21.15.45:6823/1069743893","heartbeat_front_addr":"172.21.15.45:6821/1069743893","state":["exists","up"]},{"osd":11,"uuid":"5ef0db4b-0fef-4ea0-a19b-d3677e082c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":86,"up_thru":939,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2438396298},{"type":"v1","addr":"172.21.15.45:6825","nonce":2438396298}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":2438396298},{"type":"v1","addr":"172.21.15.45:6827","nonce":2438396298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":2438396298},{"type":"v1","addr":"172.21.15.45:6831","nonce":2438396298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":2438396298},{"type":"v1","addr":"172.21.15.45:6829","nonce":2438396298}]},"public_addr":"172.21.15.45:6825/2438396298","cluster_addr":"172.21.15.45:6827/2438396298","heartbeat_back_addr":"172.21.15.45:6831/2438396298","heartbeat_front_addr":"172.21.15.45:6829/2438396298","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T14:49:01.777625+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T14:49:17.933457+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T14:48:59.975037+0000","dead_epoch":564},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T14:49:24.369534+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T14:48:38.666696+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T14:49:16.245415+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T14:49:11.452371+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T14:48:27.847092+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T14:48:29.109925+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T14:49:22.840386+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T14:48:12.770531+0000","dead_epoch":529},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T14:48:23.685590+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"1.0","osds":[2,9,0]}],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/1451745969":"2024-08-20T14:06:13.660657+0000","172.21.15.31:0/1515698918":"2024-08-20T14:06:13.660657+0000","172.21.15.31:6801/3297539242":"2024-08-20T14:06:13.660657+0000","172.21.15.31:6800/3297539242":"2024-08-20T14:06:13.660657+0000","172.21.15.31:0/2217423818":"2024-08-20T14:05:48.827576+0000","172.21.15.31:0/3353498281":"2024-08-20T14:06:13.660657+0000","172.21.15.31:0/4013594220":"2024-08-20T14:05:48.827576+0000","172.21.15.31:0/1654518065":"2024-08-20T14:05:48.827576+0000","172.21.15.31:6801/2368821258":"2024-08-20T14:05:48.827576+0000","172.21.15.31:6800/2368821258":"2024-08-20T14:05:48.827576+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-19T14:49:29.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:29 smithi032 ceph-mon[20004]: pgmap v2259: 31 pgs: 31 active+clean; 577 KiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:49:29.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:29 smithi045 ceph-mon[20143]: pgmap v2259: 31 pgs: 31 active+clean; 577 KiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:49:30.228 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-08-19T14:49:30.229 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-08-19T14:49:30.229 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-08-19T14:49:30.230 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-08-19T14:49:30.231 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-08-19T14:49:30.232 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-08-19T14:49:30.233 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-08-19T14:49:30.233 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-08-19T14:49:30.234 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-08-19T14:49:30.234 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-08-19T14:49:30.235 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-08-19T14:49:30.235 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-08-19T14:49:30.421 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:30 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/249124536' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-08-19T14:49:30.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:30 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/249124536' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-08-19T14:49:30.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:30 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/249124536' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-08-19T14:49:30.889 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:49:30.918 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:49:31.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:31 smithi032 ceph-mon[20004]: pgmap v2260: 31 pgs: 31 active+clean; 577 KiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:49:31.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:31 smithi032 ceph-mon[20004]: 1.3 scrub starts 2024-08-19T14:49:31.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:31 smithi032 ceph-mon[20004]: 1.3 scrub ok 2024-08-19T14:49:31.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:31 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.08581289882473492]}]: dispatch 2024-08-19T14:49:31.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:31 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.0874377146109268]}]: dispatch 2024-08-19T14:49:31.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:31 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08781623285599573]}]: dispatch 2024-08-19T14:49:31.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:31 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09005373626739416]}]: dispatch 2024-08-19T14:49:31.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:31 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08781623285599573]}]: dispatch 2024-08-19T14:49:31.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:31 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08668556572075027]}]: dispatch 2024-08-19T14:49:31.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:31 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08581289882473492]}]: dispatch 2024-08-19T14:49:31.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:31 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08455405010917294]}]: dispatch 2024-08-19T14:49:31.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:31 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08668556572075027]}]: dispatch 2024-08-19T14:49:31.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:31 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08668556572075027]}]: dispatch 2024-08-19T14:49:31.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:31 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08668556572075027]}]: dispatch 2024-08-19T14:49:31.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:31 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.09150035948679357]}]: dispatch 2024-08-19T14:49:31.788 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:31 smithi031 ceph-mon[16336]: pgmap v2260: 31 pgs: 31 active+clean; 577 KiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:49:31.788 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:31 smithi031 ceph-mon[16336]: 1.3 scrub starts 2024-08-19T14:49:31.788 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:31 smithi031 ceph-mon[16336]: 1.3 scrub ok 2024-08-19T14:49:31.788 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:31 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.08581289882473492]}]: dispatch 2024-08-19T14:49:31.788 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:31 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.0874377146109268]}]: dispatch 2024-08-19T14:49:31.788 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:31 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08781623285599573]}]: dispatch 2024-08-19T14:49:31.788 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:31 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09005373626739416]}]: dispatch 2024-08-19T14:49:31.788 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:31 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08781623285599573]}]: dispatch 2024-08-19T14:49:31.788 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:31 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08668556572075027]}]: dispatch 2024-08-19T14:49:31.788 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:31 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08581289882473492]}]: dispatch 2024-08-19T14:49:31.789 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:31 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08455405010917294]}]: dispatch 2024-08-19T14:49:31.789 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:31 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08668556572075027]}]: dispatch 2024-08-19T14:49:31.789 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:31 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08668556572075027]}]: dispatch 2024-08-19T14:49:31.789 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:31 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08668556572075027]}]: dispatch 2024-08-19T14:49:31.789 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:31 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.09150035948679357]}]: dispatch 2024-08-19T14:49:31.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:31 smithi045 ceph-mon[20143]: pgmap v2260: 31 pgs: 31 active+clean; 577 KiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:49:31.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:31 smithi045 ceph-mon[20143]: 1.3 scrub starts 2024-08-19T14:49:31.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:31 smithi045 ceph-mon[20143]: 1.3 scrub ok 2024-08-19T14:49:31.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:31 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.08581289882473492]}]: dispatch 2024-08-19T14:49:31.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:31 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.0874377146109268]}]: dispatch 2024-08-19T14:49:31.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:31 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08781623285599573]}]: dispatch 2024-08-19T14:49:31.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:31 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09005373626739416]}]: dispatch 2024-08-19T14:49:31.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:31 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08781623285599573]}]: dispatch 2024-08-19T14:49:31.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:31 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08668556572075027]}]: dispatch 2024-08-19T14:49:31.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:31 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08581289882473492]}]: dispatch 2024-08-19T14:49:31.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:31 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08455405010917294]}]: dispatch 2024-08-19T14:49:31.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:31 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08668556572075027]}]: dispatch 2024-08-19T14:49:31.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:31 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08668556572075027]}]: dispatch 2024-08-19T14:49:31.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:31 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08668556572075027]}]: dispatch 2024-08-19T14:49:31.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:31 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.09150035948679357]}]: dispatch 2024-08-19T14:49:32.252 INFO:teuthology.orchestra.run.smithi031.stdout:55834575353 2024-08-19T14:49:32.252 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-08-19T14:49:32.299 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:49:32.309 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:49:32.311 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:49:32.311 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:49:32.313 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:49:32.313 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:49:32.316 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:49:32.323 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:49:32.336 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:49:32.543 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:32 smithi031 ceph-mon[16336]: 1.1 scrub starts 2024-08-19T14:49:32.544 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:32 smithi031 ceph-mon[16336]: 1.1 scrub ok 2024-08-19T14:49:32.544 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:32 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.08581289882473492]}]': finished 2024-08-19T14:49:32.544 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:32 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.0874377146109268]}]': finished 2024-08-19T14:49:32.544 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:32 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08781623285599573]}]': finished 2024-08-19T14:49:32.544 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:32 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09005373626739416]}]': finished 2024-08-19T14:49:32.544 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:32 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08781623285599573]}]': finished 2024-08-19T14:49:32.544 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:32 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08668556572075027]}]': finished 2024-08-19T14:49:32.544 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:32 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08581289882473492]}]': finished 2024-08-19T14:49:32.544 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:32 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08455405010917294]}]': finished 2024-08-19T14:49:32.544 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:32 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08668556572075027]}]': finished 2024-08-19T14:49:32.544 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:32 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08668556572075027]}]': finished 2024-08-19T14:49:32.544 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:32 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08668556572075027]}]': finished 2024-08-19T14:49:32.545 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:32 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.09150035948679357]}]': finished 2024-08-19T14:49:32.545 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:32 smithi031 ceph-mon[16336]: osdmap e973: 12 total, 12 up, 12 in 2024-08-19T14:49:32.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:32 smithi032 ceph-mon[20004]: 1.1 scrub starts 2024-08-19T14:49:32.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:32 smithi032 ceph-mon[20004]: 1.1 scrub ok 2024-08-19T14:49:32.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:32 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.08581289882473492]}]': finished 2024-08-19T14:49:32.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:32 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.0874377146109268]}]': finished 2024-08-19T14:49:32.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:32 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08781623285599573]}]': finished 2024-08-19T14:49:32.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:32 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09005373626739416]}]': finished 2024-08-19T14:49:32.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:32 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08781623285599573]}]': finished 2024-08-19T14:49:32.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:32 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08668556572075027]}]': finished 2024-08-19T14:49:32.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:32 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08581289882473492]}]': finished 2024-08-19T14:49:32.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:32 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08455405010917294]}]': finished 2024-08-19T14:49:32.759 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:32 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08668556572075027]}]': finished 2024-08-19T14:49:32.760 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:32 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08668556572075027]}]': finished 2024-08-19T14:49:32.760 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:32 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08668556572075027]}]': finished 2024-08-19T14:49:32.760 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:32 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.09150035948679357]}]': finished 2024-08-19T14:49:32.760 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:32 smithi032 ceph-mon[20004]: osdmap e973: 12 total, 12 up, 12 in 2024-08-19T14:49:32.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:32 smithi045 ceph-mon[20143]: 1.1 scrub starts 2024-08-19T14:49:32.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:32 smithi045 ceph-mon[20143]: 1.1 scrub ok 2024-08-19T14:49:32.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:32 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.08581289882473492]}]': finished 2024-08-19T14:49:32.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:32 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.0874377146109268]}]': finished 2024-08-19T14:49:32.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:32 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08781623285599573]}]': finished 2024-08-19T14:49:32.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:32 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09005373626739416]}]': finished 2024-08-19T14:49:32.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:32 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08781623285599573]}]': finished 2024-08-19T14:49:32.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:32 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08668556572075027]}]': finished 2024-08-19T14:49:32.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:32 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08581289882473492]}]': finished 2024-08-19T14:49:32.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:32 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08455405010917294]}]': finished 2024-08-19T14:49:32.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:32 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08668556572075027]}]': finished 2024-08-19T14:49:32.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:32 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08668556572075027]}]': finished 2024-08-19T14:49:32.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:32 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08668556572075027]}]': finished 2024-08-19T14:49:32.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:32 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.09150035948679357]}]': finished 2024-08-19T14:49:32.798 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:32 smithi045 ceph-mon[20143]: osdmap e973: 12 total, 12 up, 12 in 2024-08-19T14:49:33.735 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:33 smithi031 ceph-mon[16336]: osdmap e974: 12 total, 12 up, 12 in 2024-08-19T14:49:33.735 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:33 smithi031 ceph-mon[16336]: pgmap v2263: 31 pgs: 31 active+clean; 577 KiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:49:33.735 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:33 smithi031 ceph-mon[16336]: osdmap e975: 12 total, 12 up, 12 in 2024-08-19T14:49:33.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:33 smithi032 ceph-mon[20004]: osdmap e974: 12 total, 12 up, 12 in 2024-08-19T14:49:33.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:33 smithi032 ceph-mon[20004]: pgmap v2263: 31 pgs: 31 active+clean; 577 KiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:49:33.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:33 smithi032 ceph-mon[20004]: osdmap e975: 12 total, 12 up, 12 in 2024-08-19T14:49:33.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:33 smithi045 ceph-mon[20143]: osdmap e974: 12 total, 12 up, 12 in 2024-08-19T14:49:33.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:33 smithi045 ceph-mon[20143]: pgmap v2263: 31 pgs: 31 active+clean; 577 KiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:49:33.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:33 smithi045 ceph-mon[20143]: osdmap e975: 12 total, 12 up, 12 in 2024-08-19T14:49:34.609 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:34 smithi031 ceph-mon[16336]: 1.19 scrub starts 2024-08-19T14:49:34.609 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:34 smithi031 ceph-mon[16336]: 1.19 scrub ok 2024-08-19T14:49:34.609 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:34 smithi031 ceph-mon[16336]: 1.1b scrub starts 2024-08-19T14:49:34.609 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:34 smithi031 ceph-mon[16336]: 1.1b scrub ok 2024-08-19T14:49:34.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:34 smithi032 ceph-mon[20004]: 1.19 scrub starts 2024-08-19T14:49:34.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:34 smithi032 ceph-mon[20004]: 1.19 scrub ok 2024-08-19T14:49:34.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:34 smithi032 ceph-mon[20004]: 1.1b scrub starts 2024-08-19T14:49:34.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:34 smithi032 ceph-mon[20004]: 1.1b scrub ok 2024-08-19T14:49:34.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:34 smithi045 ceph-mon[20143]: 1.19 scrub starts 2024-08-19T14:49:34.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:34 smithi045 ceph-mon[20143]: 1.19 scrub ok 2024-08-19T14:49:34.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:34 smithi045 ceph-mon[20143]: 1.1b scrub starts 2024-08-19T14:49:34.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:34 smithi045 ceph-mon[20143]: 1.1b scrub ok 2024-08-19T14:49:35.009 INFO:teuthology.orchestra.run.smithi031.stdout:171798692326 2024-08-19T14:49:35.009 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-08-19T14:49:35.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:35 smithi031 ceph-mon[16336]: pgmap v2265: 31 pgs: 1 active+remapped, 30 active+clean; 577 KiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:49:35.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:35 smithi031 ceph-mon[16336]: osdmap e976: 12 total, 12 up, 12 in 2024-08-19T14:49:35.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:35 smithi032 ceph-mon[20004]: pgmap v2265: 31 pgs: 1 active+remapped, 30 active+clean; 577 KiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:49:35.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:35 smithi032 ceph-mon[20004]: osdmap e976: 12 total, 12 up, 12 in 2024-08-19T14:49:35.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:35 smithi045 ceph-mon[20143]: pgmap v2265: 31 pgs: 1 active+remapped, 30 active+clean; 577 KiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:49:35.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:35 smithi045 ceph-mon[20143]: osdmap e976: 12 total, 12 up, 12 in 2024-08-19T14:49:35.994 INFO:teuthology.orchestra.run.smithi031.stdout:2314987372782 2024-08-19T14:49:35.995 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-08-19T14:49:36.615 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:36 smithi031 ceph-mon[16336]: osdmap e977: 12 total, 12 up, 12 in 2024-08-19T14:49:36.615 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:36 smithi031 ceph-mon[16336]: 1.14 deep-scrub starts 2024-08-19T14:49:36.615 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:36 smithi031 ceph-mon[16336]: 1.14 deep-scrub ok 2024-08-19T14:49:36.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:36 smithi032 ceph-mon[20004]: osdmap e977: 12 total, 12 up, 12 in 2024-08-19T14:49:36.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:36 smithi032 ceph-mon[20004]: 1.14 deep-scrub starts 2024-08-19T14:49:36.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:36 smithi032 ceph-mon[20004]: 1.14 deep-scrub ok 2024-08-19T14:49:36.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:36 smithi045 ceph-mon[20143]: osdmap e977: 12 total, 12 up, 12 in 2024-08-19T14:49:36.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:36 smithi045 ceph-mon[20143]: 1.14 deep-scrub starts 2024-08-19T14:49:36.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:36 smithi045 ceph-mon[20143]: 1.14 deep-scrub ok 2024-08-19T14:49:37.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:37 smithi031 ceph-mon[16336]: pgmap v2268: 31 pgs: 31 active+clean; 577 KiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:49:37.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:37 smithi031 ceph-mon[16336]: 1.13 scrub starts 2024-08-19T14:49:37.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:37 smithi031 ceph-mon[16336]: 1.13 scrub ok 2024-08-19T14:49:37.648 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:49:37.685 INFO:teuthology.orchestra.run.smithi031.stdout:317827580374 2024-08-19T14:49:37.685 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-08-19T14:49:37.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:37 smithi032 ceph-mon[20004]: pgmap v2268: 31 pgs: 31 active+clean; 577 KiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:49:37.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:37 smithi032 ceph-mon[20004]: 1.13 scrub starts 2024-08-19T14:49:37.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:37 smithi032 ceph-mon[20004]: 1.13 scrub ok 2024-08-19T14:49:37.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:37 smithi045 ceph-mon[20143]: pgmap v2268: 31 pgs: 31 active+clean; 577 KiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:49:37.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:37 smithi045 ceph-mon[20143]: 1.13 scrub starts 2024-08-19T14:49:37.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:37 smithi045 ceph-mon[20143]: 1.13 scrub ok 2024-08-19T14:49:38.016 INFO:teuthology.orchestra.run.smithi031.stdout:111669150192 2024-08-19T14:49:38.016 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-08-19T14:49:38.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:38 smithi032 ceph-mon[20004]: 1.2 scrub starts 2024-08-19T14:49:38.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:38 smithi032 ceph-mon[20004]: 1.2 scrub ok 2024-08-19T14:49:38.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:38 smithi045 ceph-mon[20143]: 1.2 scrub starts 2024-08-19T14:49:38.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:38 smithi045 ceph-mon[20143]: 1.2 scrub ok 2024-08-19T14:49:38.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:38 smithi031 ceph-mon[16336]: 1.2 scrub starts 2024-08-19T14:49:38.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:38 smithi031 ceph-mon[16336]: 1.2 scrub ok 2024-08-19T14:49:39.296 INFO:teuthology.orchestra.run.smithi031.stdout:2723009265839 2024-08-19T14:49:39.297 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-08-19T14:49:39.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:39 smithi031 ceph-mon[16336]: pgmap v2269: 31 pgs: 31 active+clean; 577 KiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:49:39.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:39 smithi032 ceph-mon[20004]: pgmap v2269: 31 pgs: 31 active+clean; 577 KiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:49:39.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:39 smithi045 ceph-mon[20143]: pgmap v2269: 31 pgs: 31 active+clean; 577 KiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:49:40.074 INFO:teuthology.orchestra.run.smithi031.stdout:34359738878 2024-08-19T14:49:40.075 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-08-19T14:49:40.706 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:40 smithi031 ceph-mon[16336]: 1.15 scrub starts 2024-08-19T14:49:40.706 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:40 smithi031 ceph-mon[16336]: 1.15 scrub ok 2024-08-19T14:49:40.706 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:40 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:49:40.706 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:40 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:49:40.707 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:40 smithi031 ceph-mon[16336]: osdmap e978: 12 total, 12 up, 12 in 2024-08-19T14:49:40.733 INFO:teuthology.orchestra.run.smithi031.stdout:206158430691 2024-08-19T14:49:40.733 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-08-19T14:49:40.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:40 smithi032 ceph-mon[20004]: 1.15 scrub starts 2024-08-19T14:49:40.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:40 smithi032 ceph-mon[20004]: 1.15 scrub ok 2024-08-19T14:49:40.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:40 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:49:40.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:40 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:49:40.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:40 smithi032 ceph-mon[20004]: osdmap e978: 12 total, 12 up, 12 in 2024-08-19T14:49:40.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:40 smithi045 ceph-mon[20143]: 1.15 scrub starts 2024-08-19T14:49:40.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:40 smithi045 ceph-mon[20143]: 1.15 scrub ok 2024-08-19T14:49:40.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:40 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:49:40.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:40 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:49:40.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:40 smithi045 ceph-mon[20143]: osdmap e978: 12 total, 12 up, 12 in 2024-08-19T14:49:41.212 INFO:teuthology.orchestra.run.smithi031.stdout:279172874716 2024-08-19T14:49:41.212 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-08-19T14:49:41.225 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:49:41.226 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:49:41.227 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:49:41.228 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:49:41.745 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:41 smithi031 ceph-mon[16336]: pgmap v2271: 31 pgs: 31 active+clean; 577 KiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:49:41.745 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:41 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:49:41.745 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:41 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:49:41.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:41 smithi032 ceph-mon[20004]: pgmap v2271: 31 pgs: 31 active+clean; 577 KiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:49:41.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:41 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:49:41.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:41 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:49:41.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:41 smithi045 ceph-mon[20143]: pgmap v2271: 31 pgs: 31 active+clean; 577 KiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:49:41.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:41 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:49:41.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:41 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:49:42.401 INFO:teuthology.orchestra.run.smithi031.stdout:244813136352 2024-08-19T14:49:42.402 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-08-19T14:49:42.672 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:42 smithi031 ceph-mon[16336]: osdmap e979: 12 total, 12 up, 12 in 2024-08-19T14:49:42.672 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:42 smithi031 ceph-mon[16336]: 1.1a scrub starts 2024-08-19T14:49:42.672 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:42 smithi031 ceph-mon[16336]: 1.1a scrub ok 2024-08-19T14:49:42.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:42 smithi032 ceph-mon[20004]: osdmap e979: 12 total, 12 up, 12 in 2024-08-19T14:49:42.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:42 smithi032 ceph-mon[20004]: 1.1a scrub starts 2024-08-19T14:49:42.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:42 smithi032 ceph-mon[20004]: 1.1a scrub ok 2024-08-19T14:49:42.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:42 smithi045 ceph-mon[20143]: osdmap e979: 12 total, 12 up, 12 in 2024-08-19T14:49:42.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:42 smithi045 ceph-mon[20143]: 1.1a scrub starts 2024-08-19T14:49:42.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:42 smithi045 ceph-mon[20143]: 1.1a scrub ok 2024-08-19T14:49:43.372 INFO:teuthology.orchestra.run.smithi031.stdout:171798692328 2024-08-19T14:49:43.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:43 smithi032 ceph-mon[20004]: pgmap v2273: 31 pgs: 31 active+clean; 577 KiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:49:43.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:43 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3561876666' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-08-19T14:49:43.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:43 smithi045 ceph-mon[20143]: pgmap v2273: 31 pgs: 31 active+clean; 577 KiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:49:43.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:43 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3561876666' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-08-19T14:49:43.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:43 smithi031 ceph-mon[16336]: pgmap v2273: 31 pgs: 31 active+clean; 577 KiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:49:43.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:43 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3561876666' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-08-19T14:49:44.002 INFO:teuthology.orchestra.run.smithi031.stdout:369367187917 2024-08-19T14:49:44.003 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-08-19T14:49:44.428 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:49:44.428 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:49:44.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:44 smithi032 ceph-mon[20004]: 1.11 scrub starts 2024-08-19T14:49:44.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:44 smithi032 ceph-mon[20004]: 1.11 scrub ok 2024-08-19T14:49:44.783 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:44 smithi031 ceph-mon[16336]: 1.11 scrub starts 2024-08-19T14:49:44.783 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:44 smithi031 ceph-mon[16336]: 1.11 scrub ok 2024-08-19T14:49:44.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:44 smithi045 ceph-mon[20143]: 1.11 scrub starts 2024-08-19T14:49:44.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:44 smithi045 ceph-mon[20143]: 1.11 scrub ok 2024-08-19T14:49:45.553 INFO:tasks.cephadm.ceph_manager.ceph:need seq 171798692326 got 171798692328 for osd.5 2024-08-19T14:49:45.553 DEBUG:teuthology.parallel:result is None 2024-08-19T14:49:45.748 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:45 smithi031 ceph-mon[16336]: 1.5 deep-scrub starts 2024-08-19T14:49:45.749 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:45 smithi031 ceph-mon[16336]: 1.5 deep-scrub ok 2024-08-19T14:49:45.749 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:45 smithi031 ceph-mon[16336]: pgmap v2274: 31 pgs: 31 active+clean; 577 KiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:49:45.749 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:45 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:49:45.749 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:45 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:49:45.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:45 smithi032 ceph-mon[20004]: 1.5 deep-scrub starts 2024-08-19T14:49:45.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:45 smithi032 ceph-mon[20004]: 1.5 deep-scrub ok 2024-08-19T14:49:45.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:45 smithi032 ceph-mon[20004]: pgmap v2274: 31 pgs: 31 active+clean; 577 KiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:49:45.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:45 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:49:45.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:45 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:49:45.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:45 smithi045 ceph-mon[20143]: 1.5 deep-scrub starts 2024-08-19T14:49:45.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:45 smithi045 ceph-mon[20143]: 1.5 deep-scrub ok 2024-08-19T14:49:45.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:45 smithi045 ceph-mon[20143]: pgmap v2274: 31 pgs: 31 active+clean; 577 KiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:49:45.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:45 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:49:45.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:45 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:49:45.932 INFO:teuthology.orchestra.run.smithi031.stdout:111669150194 2024-08-19T14:49:46.504 INFO:teuthology.orchestra.run.smithi031.stdout:55834575356 2024-08-19T14:49:46.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:46 smithi031 ceph-mon[16336]: 1.d scrub starts 2024-08-19T14:49:46.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:46 smithi031 ceph-mon[16336]: 1.d scrub ok 2024-08-19T14:49:46.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:46 smithi031 ceph-mon[16336]: 1.17 scrub starts 2024-08-19T14:49:46.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:46 smithi031 ceph-mon[16336]: 1.17 scrub ok 2024-08-19T14:49:46.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:46 smithi031 ceph-mon[16336]: osdmap e980: 12 total, 12 up, 12 in 2024-08-19T14:49:46.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:46 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:49:46.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:46 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:49:46.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:46 smithi031 ceph-mon[16336]: 1.18 scrub starts 2024-08-19T14:49:46.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:46 smithi031 ceph-mon[16336]: 1.18 scrub ok 2024-08-19T14:49:46.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:46 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1065502513' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-08-19T14:49:46.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:46 smithi032 ceph-mon[20004]: 1.d scrub starts 2024-08-19T14:49:46.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:46 smithi032 ceph-mon[20004]: 1.d scrub ok 2024-08-19T14:49:46.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:46 smithi032 ceph-mon[20004]: 1.17 scrub starts 2024-08-19T14:49:46.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:46 smithi032 ceph-mon[20004]: 1.17 scrub ok 2024-08-19T14:49:46.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:46 smithi032 ceph-mon[20004]: osdmap e980: 12 total, 12 up, 12 in 2024-08-19T14:49:46.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:46 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:49:46.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:46 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:49:46.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:46 smithi032 ceph-mon[20004]: 1.18 scrub starts 2024-08-19T14:49:46.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:46 smithi032 ceph-mon[20004]: 1.18 scrub ok 2024-08-19T14:49:46.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:46 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1065502513' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-08-19T14:49:46.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:46 smithi045 ceph-mon[20143]: 1.d scrub starts 2024-08-19T14:49:46.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:46 smithi045 ceph-mon[20143]: 1.d scrub ok 2024-08-19T14:49:46.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:46 smithi045 ceph-mon[20143]: 1.17 scrub starts 2024-08-19T14:49:46.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:46 smithi045 ceph-mon[20143]: 1.17 scrub ok 2024-08-19T14:49:46.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:46 smithi045 ceph-mon[20143]: osdmap e980: 12 total, 12 up, 12 in 2024-08-19T14:49:46.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:46 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:49:46.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:46 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:49:46.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:46 smithi045 ceph-mon[20143]: 1.18 scrub starts 2024-08-19T14:49:46.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:46 smithi045 ceph-mon[20143]: 1.18 scrub ok 2024-08-19T14:49:46.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:46 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1065502513' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-08-19T14:49:47.535 INFO:teuthology.orchestra.run.smithi031.stdout:279172874719 2024-08-19T14:49:47.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:47 smithi031 ceph-mon[16336]: 1.f scrub starts 2024-08-19T14:49:47.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:47 smithi031 ceph-mon[16336]: 1.f scrub ok 2024-08-19T14:49:47.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:47 smithi031 ceph-mon[16336]: pgmap v2276: 31 pgs: 31 active+clean; 577 KiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:49:47.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:47 smithi031 ceph-mon[16336]: osdmap e981: 12 total, 12 up, 12 in 2024-08-19T14:49:47.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:47 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/857212115' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-08-19T14:49:47.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:47 smithi032 ceph-mon[20004]: 1.f scrub starts 2024-08-19T14:49:47.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:47 smithi032 ceph-mon[20004]: 1.f scrub ok 2024-08-19T14:49:47.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:47 smithi032 ceph-mon[20004]: pgmap v2276: 31 pgs: 31 active+clean; 577 KiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:49:47.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:47 smithi032 ceph-mon[20004]: osdmap e981: 12 total, 12 up, 12 in 2024-08-19T14:49:47.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:47 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/857212115' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-08-19T14:49:47.780 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834575353 got 55834575356 for osd.1 2024-08-19T14:49:47.780 DEBUG:teuthology.parallel:result is None 2024-08-19T14:49:47.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:47 smithi045 ceph-mon[20143]: 1.f scrub starts 2024-08-19T14:49:47.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:47 smithi045 ceph-mon[20143]: 1.f scrub ok 2024-08-19T14:49:47.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:47 smithi045 ceph-mon[20143]: pgmap v2276: 31 pgs: 31 active+clean; 577 KiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:49:47.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:47 smithi045 ceph-mon[20143]: osdmap e981: 12 total, 12 up, 12 in 2024-08-19T14:49:47.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:47 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/857212115' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-08-19T14:49:48.479 INFO:tasks.cephadm.ceph_manager.ceph:need seq 279172874716 got 279172874719 for osd.8 2024-08-19T14:49:48.480 DEBUG:teuthology.parallel:result is None 2024-08-19T14:49:48.675 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:49:48.675 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:49:48.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:48 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2185753636' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-08-19T14:49:48.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:48 smithi032 ceph-mon[20004]: 1.c scrub starts 2024-08-19T14:49:48.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:48 smithi032 ceph-mon[20004]: 1.c scrub ok 2024-08-19T14:49:48.778 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:48 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2185753636' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-08-19T14:49:48.778 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:48 smithi031 ceph-mon[16336]: 1.c scrub starts 2024-08-19T14:49:48.778 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:48 smithi031 ceph-mon[16336]: 1.c scrub ok 2024-08-19T14:49:48.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:48 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2185753636' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-08-19T14:49:48.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:48 smithi045 ceph-mon[20143]: 1.c scrub starts 2024-08-19T14:49:48.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:48 smithi045 ceph-mon[20143]: 1.c scrub ok 2024-08-19T14:49:49.125 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669150192 got 111669150194 for osd.3 2024-08-19T14:49:49.125 DEBUG:teuthology.parallel:result is None 2024-08-19T14:49:49.477 INFO:teuthology.orchestra.run.smithi031.stdout:317827580376 2024-08-19T14:49:49.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:49 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:49:49.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:49 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:49:49.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:49 smithi032 ceph-mon[20004]: pgmap v2278: 31 pgs: 31 active+clean; 577 KiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:49:49.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:49 smithi032 ceph-mon[20004]: osdmap e982: 12 total, 12 up, 12 in 2024-08-19T14:49:49.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:49 smithi032 ceph-mon[20004]: 1.8 scrub starts 2024-08-19T14:49:49.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:49 smithi032 ceph-mon[20004]: 1.8 scrub ok 2024-08-19T14:49:49.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:49 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/4057748843' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-08-19T14:49:49.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:49 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:49:49.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:49 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:49:49.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:49 smithi045 ceph-mon[20143]: pgmap v2278: 31 pgs: 31 active+clean; 577 KiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:49:49.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:49 smithi045 ceph-mon[20143]: osdmap e982: 12 total, 12 up, 12 in 2024-08-19T14:49:49.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:49 smithi045 ceph-mon[20143]: 1.8 scrub starts 2024-08-19T14:49:49.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:49 smithi045 ceph-mon[20143]: 1.8 scrub ok 2024-08-19T14:49:49.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:49 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/4057748843' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-08-19T14:49:49.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:49 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:49:49.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:49 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:49:49.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:49 smithi031 ceph-mon[16336]: pgmap v2278: 31 pgs: 31 active+clean; 577 KiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:49:49.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:49 smithi031 ceph-mon[16336]: osdmap e982: 12 total, 12 up, 12 in 2024-08-19T14:49:49.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:49 smithi031 ceph-mon[16336]: 1.8 scrub starts 2024-08-19T14:49:49.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:49 smithi031 ceph-mon[16336]: 1.8 scrub ok 2024-08-19T14:49:49.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:49 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/4057748843' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-08-19T14:49:50.364 INFO:teuthology.orchestra.run.smithi031.stdout:2723009265841 2024-08-19T14:49:50.618 INFO:tasks.cephadm.ceph_manager.ceph:need seq 317827580374 got 317827580376 for osd.9 2024-08-19T14:49:50.618 DEBUG:teuthology.parallel:result is None 2024-08-19T14:49:50.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:50 smithi032 ceph-mon[20004]: 1.a scrub starts 2024-08-19T14:49:50.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:50 smithi032 ceph-mon[20004]: 1.a scrub ok 2024-08-19T14:49:50.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:50 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3198929884' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-08-19T14:49:50.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:50 smithi045 ceph-mon[20143]: 1.a scrub starts 2024-08-19T14:49:50.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:50 smithi045 ceph-mon[20143]: 1.a scrub ok 2024-08-19T14:49:50.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:50 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3198929884' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-08-19T14:49:50.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:50 smithi031 ceph-mon[16336]: 1.a scrub starts 2024-08-19T14:49:50.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:50 smithi031 ceph-mon[16336]: 1.a scrub ok 2024-08-19T14:49:50.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:50 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3198929884' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-08-19T14:49:51.175 INFO:tasks.cephadm.ceph_manager.ceph:need seq 2723009265839 got 2723009265841 for osd.2 2024-08-19T14:49:51.175 DEBUG:teuthology.parallel:result is None 2024-08-19T14:49:51.663 INFO:teuthology.orchestra.run.smithi031.stdout:133143986670 2024-08-19T14:49:51.664 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-08-19T14:49:51.757 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:51 smithi032 ceph-mon[20004]: 1.b deep-scrub starts 2024-08-19T14:49:51.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:51 smithi032 ceph-mon[20004]: 1.b deep-scrub ok 2024-08-19T14:49:51.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:51 smithi032 ceph-mon[20004]: pgmap v2280: 31 pgs: 31 active+clean; 577 KiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:49:51.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:51 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:49:51.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:51 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:49:51.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:51 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:49:51.758 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:51 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:49:51.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:51 smithi045 ceph-mon[20143]: 1.b deep-scrub starts 2024-08-19T14:49:51.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:51 smithi045 ceph-mon[20143]: 1.b deep-scrub ok 2024-08-19T14:49:51.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:51 smithi045 ceph-mon[20143]: pgmap v2280: 31 pgs: 31 active+clean; 577 KiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:49:51.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:51 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:49:51.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:51 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:49:51.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:51 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:49:51.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:51 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:49:51.840 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:51 smithi031 ceph-mon[16336]: 1.b deep-scrub starts 2024-08-19T14:49:51.840 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:51 smithi031 ceph-mon[16336]: 1.b deep-scrub ok 2024-08-19T14:49:51.840 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:51 smithi031 ceph-mon[16336]: pgmap v2280: 31 pgs: 31 active+clean; 577 KiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:49:51.840 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:51 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:49:51.840 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:51 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:49:51.840 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:51 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:49:51.840 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:51 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:49:52.114 INFO:teuthology.orchestra.run.smithi031.stdout:2314987372786 2024-08-19T14:49:52.310 INFO:teuthology.orchestra.run.smithi031.stdout:34359738880 2024-08-19T14:49:52.764 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:52 smithi031 ceph-mon[16336]: 1.6 scrub starts 2024-08-19T14:49:52.764 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:52 smithi031 ceph-mon[16336]: 1.6 scrub ok 2024-08-19T14:49:52.764 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:52 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2384503844' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-08-19T14:49:52.765 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:52 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1090907935' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-08-19T14:49:52.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:52 smithi045 ceph-mon[20143]: 1.6 scrub starts 2024-08-19T14:49:52.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:52 smithi045 ceph-mon[20143]: 1.6 scrub ok 2024-08-19T14:49:52.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:52 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2384503844' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-08-19T14:49:52.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:52 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1090907935' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-08-19T14:49:53.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:52 smithi032 ceph-mon[20004]: 1.6 scrub starts 2024-08-19T14:49:53.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:52 smithi032 ceph-mon[20004]: 1.6 scrub ok 2024-08-19T14:49:53.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:52 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2384503844' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-08-19T14:49:53.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:52 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1090907935' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-08-19T14:49:53.071 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:49:53.072 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:49:53.106 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738878 got 34359738880 for osd.0 2024-08-19T14:49:53.106 DEBUG:teuthology.parallel:result is None 2024-08-19T14:49:53.326 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:49:53.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:53 smithi031 ceph-mon[16336]: pgmap v2281: 31 pgs: 31 active+clean; 577 KiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:49:53.728 INFO:tasks.cephadm.ceph_manager.ceph:need seq 2314987372782 got 2314987372786 for osd.10 2024-08-19T14:49:53.728 DEBUG:teuthology.parallel:result is None 2024-08-19T14:49:53.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:53 smithi045 ceph-mon[20143]: pgmap v2281: 31 pgs: 31 active+clean; 577 KiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:49:53.909 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:53 smithi032 ceph-mon[20004]: pgmap v2281: 31 pgs: 31 active+clean; 577 KiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:49:54.374 INFO:teuthology.orchestra.run.smithi031.stdout:244813136354 2024-08-19T14:49:54.732 INFO:teuthology.orchestra.run.smithi031.stdout:206158430694 2024-08-19T14:49:54.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:54 smithi031 ceph-mon[16336]: 1.16 scrub starts 2024-08-19T14:49:54.733 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:54 smithi031 ceph-mon[16336]: 1.16 scrub ok 2024-08-19T14:49:54.733 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:54 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/1989030936' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-08-19T14:49:54.750 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:49:54.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:54 smithi045 ceph-mon[20143]: 1.16 scrub starts 2024-08-19T14:49:54.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:54 smithi045 ceph-mon[20143]: 1.16 scrub ok 2024-08-19T14:49:54.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:54 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/1989030936' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-08-19T14:49:55.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:54 smithi032 ceph-mon[20004]: 1.16 scrub starts 2024-08-19T14:49:55.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:54 smithi032 ceph-mon[20004]: 1.16 scrub ok 2024-08-19T14:49:55.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:54 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/1989030936' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-08-19T14:49:55.651 INFO:tasks.cephadm.ceph_manager.ceph:need seq 244813136352 got 244813136354 for osd.7 2024-08-19T14:49:55.651 DEBUG:teuthology.parallel:result is None 2024-08-19T14:49:55.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:55 smithi045 ceph-mon[20143]: pgmap v2282: 31 pgs: 31 active+clean; 577 KiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:49:55.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:55 smithi045 ceph-mon[20143]: 1.e scrub starts 2024-08-19T14:49:55.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:55 smithi045 ceph-mon[20143]: 1.e scrub ok 2024-08-19T14:49:55.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:55 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/3785106285' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-08-19T14:49:55.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:55 smithi045 ceph-mon[20143]: 1.10 scrub starts 2024-08-19T14:49:55.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:55 smithi045 ceph-mon[20143]: 1.10 scrub ok 2024-08-19T14:49:55.814 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:55 smithi031 ceph-mon[16336]: pgmap v2282: 31 pgs: 31 active+clean; 577 KiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:49:55.815 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:55 smithi031 ceph-mon[16336]: 1.e scrub starts 2024-08-19T14:49:55.815 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:55 smithi031 ceph-mon[16336]: 1.e scrub ok 2024-08-19T14:49:55.815 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:55 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/3785106285' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-08-19T14:49:55.815 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:55 smithi031 ceph-mon[16336]: 1.10 scrub starts 2024-08-19T14:49:55.815 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:55 smithi031 ceph-mon[16336]: 1.10 scrub ok 2024-08-19T14:49:55.994 INFO:teuthology.orchestra.run.smithi031.stdout:133143986671 2024-08-19T14:49:56.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:55 smithi032 ceph-mon[20004]: pgmap v2282: 31 pgs: 31 active+clean; 577 KiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:49:56.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:55 smithi032 ceph-mon[20004]: 1.e scrub starts 2024-08-19T14:49:56.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:55 smithi032 ceph-mon[20004]: 1.e scrub ok 2024-08-19T14:49:56.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:55 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/3785106285' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-08-19T14:49:56.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:55 smithi032 ceph-mon[20004]: 1.10 scrub starts 2024-08-19T14:49:56.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:55 smithi032 ceph-mon[20004]: 1.10 scrub ok 2024-08-19T14:49:56.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:56 smithi031 ceph-mon[16336]: 1.4 scrub starts 2024-08-19T14:49:56.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:56 smithi031 ceph-mon[16336]: 1.4 scrub ok 2024-08-19T14:49:56.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:56 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2834315237' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-08-19T14:49:56.620 INFO:teuthology.orchestra.run.smithi031.stdout:369367187919 2024-08-19T14:49:56.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:56 smithi045 ceph-mon[20143]: 1.4 scrub starts 2024-08-19T14:49:56.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:56 smithi045 ceph-mon[20143]: 1.4 scrub ok 2024-08-19T14:49:56.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:56 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2834315237' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-08-19T14:49:56.941 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986670 got 133143986671 for osd.4 2024-08-19T14:49:56.941 DEBUG:teuthology.parallel:result is None 2024-08-19T14:49:57.009 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:56 smithi032 ceph-mon[20004]: 1.4 scrub starts 2024-08-19T14:49:57.009 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:56 smithi032 ceph-mon[20004]: 1.4 scrub ok 2024-08-19T14:49:57.009 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:56 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2834315237' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-08-19T14:49:57.570 INFO:tasks.cephadm.ceph_manager.ceph:need seq 369367187917 got 369367187919 for osd.11 2024-08-19T14:49:57.571 DEBUG:teuthology.parallel:result is None 2024-08-19T14:49:57.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:57 smithi045 ceph-mon[20143]: pgmap v2283: 31 pgs: 31 active+clean; 577 KiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:49:57.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:57 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/149654346' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-08-19T14:49:57.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:57 smithi031 ceph-mon[16336]: pgmap v2283: 31 pgs: 31 active+clean; 577 KiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:49:57.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:57 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/149654346' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-08-19T14:49:57.956 INFO:tasks.cephadm.ceph_manager.ceph:need seq 206158430691 got 206158430694 for osd.6 2024-08-19T14:49:57.956 DEBUG:teuthology.parallel:result is None 2024-08-19T14:49:57.956 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-08-19T14:49:57.956 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-19T14:49:58.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:57 smithi032 ceph-mon[20004]: pgmap v2283: 31 pgs: 31 active+clean; 577 KiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:49:58.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:57 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/149654346' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-08-19T14:49:58.217 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:49:58.842 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:49:58.843 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-08-19T14:49:59.569 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":2284,"stamp":"2024-08-19T14:49:58.439408+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2242144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":93,"acting":93,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":93,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":12056508,"kb_used_data":106104,"kb_used_omap":24,"kb_used_meta":11950247,"kb_avail":1112885316,"statfs":{"total":1151940427776,"available":1139594563584,"internally_reserved":0,"allocated":108650496,"data_stored":71808072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24761,"internal_metadata":12237053767},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002127"},"pg_stats":[{"pgid":"1.0","version":"70'139","reported_seq":2102,"reported_epoch":982,"state":"active+clean","last_fresh":"2024-08-19T14:49:49.514321+0000","last_change":"2024-08-19T14:49:27.410413+0000","last_active":"2024-08-19T14:49:49.514321+0000","last_peered":"2024-08-19T14:49:49.514321+0000","last_clean":"2024-08-19T14:49:49.514321+0000","last_became_active":"2024-08-19T14:49:27.410066+0000","last_became_peered":"2024-08-19T14:49:27.410066+0000","last_unstale":"2024-08-19T14:49:49.514321+0000","last_undegraded":"2024-08-19T14:49:49.514321+0000","last_fullsized":"2024-08-19T14:49:49.514321+0000","mapping_epoch":971,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":972,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:48:35.270407+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:41:45.177828+0000","last_clean_scrub_stamp":"2024-08-19T14:48:35.270407+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:49:51.310481+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,0],"acting":[2,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"70'139","reported_seq":2182,"reported_epoch":982,"state":"active+clean","last_fresh":"2024-08-19T14:49:49.514429+0000","last_change":"2024-08-19T14:49:35.443305+0000","last_active":"2024-08-19T14:49:49.514429+0000","last_peered":"2024-08-19T14:49:49.514429+0000","last_clean":"2024-08-19T14:49:49.514429+0000","last_became_active":"2024-08-19T14:49:35.442955+0000","last_became_peered":"2024-08-19T14:49:35.442955+0000","last_unstale":"2024-08-19T14:49:49.514429+0000","last_undegraded":"2024-08-19T14:49:49.514429+0000","last_fullsized":"2024-08-19T14:49:49.514429+0000","mapping_epoch":976,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":977,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:49:30.522127+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:49:30.522127+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:50:44.180831+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"70'139","reported_seq":584,"reported_epoch":982,"state":"active+clean","last_fresh":"2024-08-19T14:49:49.276672+0000","last_change":"2024-08-19T14:49:37.096769+0000","last_active":"2024-08-19T14:49:49.276672+0000","last_peered":"2024-08-19T14:49:49.276672+0000","last_clean":"2024-08-19T14:49:49.276672+0000","last_became_active":"2024-08-19T14:48:29.137710+0000","last_became_peered":"2024-08-19T14:48:29.137710+0000","last_unstale":"2024-08-19T14:49:49.276672+0000","last_undegraded":"2024-08-19T14:49:49.276672+0000","last_fullsized":"2024-08-19T14:49:49.276672+0000","mapping_epoch":936,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":937,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:49:37.096669+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:48:24.313364+0000","last_clean_scrub_stamp":"2024-08-19T14:49:37.096669+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:51:02.158270+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"70'139","reported_seq":820,"reported_epoch":982,"state":"active+clean","last_fresh":"2024-08-19T14:49:49.514591+0000","last_change":"2024-08-19T14:49:30.896533+0000","last_active":"2024-08-19T14:49:49.514591+0000","last_peered":"2024-08-19T14:49:49.514591+0000","last_clean":"2024-08-19T14:49:49.514591+0000","last_became_active":"2024-08-19T14:48:29.139386+0000","last_became_peered":"2024-08-19T14:48:29.139386+0000","last_unstale":"2024-08-19T14:49:49.514591+0000","last_undegraded":"2024-08-19T14:49:49.514591+0000","last_fullsized":"2024-08-19T14:49:49.514591+0000","mapping_epoch":936,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":937,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:49:30.896441+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:49:30.896441+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:50:57.035316+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.7","version":"70'139","reported_seq":1345,"reported_epoch":982,"state":"active+clean","last_fresh":"2024-08-19T14:49:49.459792+0000","last_change":"2024-08-19T14:48:46.622665+0000","last_active":"2024-08-19T14:49:49.459792+0000","last_peered":"2024-08-19T14:49:49.459792+0000","last_clean":"2024-08-19T14:49:49.459792+0000","last_became_active":"2024-08-19T14:47:45.871964+0000","last_became_peered":"2024-08-19T14:47:45.871964+0000","last_unstale":"2024-08-19T14:49:49.459792+0000","last_undegraded":"2024-08-19T14:49:49.459792+0000","last_fullsized":"2024-08-19T14:49:49.459792+0000","mapping_epoch":917,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":918,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:48:46.622591+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:21:46.057624+0000","last_clean_scrub_stamp":"2024-08-19T14:48:46.622591+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:50:15.453705+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"70'139","reported_seq":677,"reported_epoch":982,"state":"active+clean","last_fresh":"2024-08-19T14:49:50.775120+0000","last_change":"2024-08-19T14:49:50.775147+0000","last_active":"2024-08-19T14:49:50.775120+0000","last_peered":"2024-08-19T14:49:50.775120+0000","last_clean":"2024-08-19T14:49:50.775120+0000","last_became_active":"2024-08-19T14:43:16.091256+0000","last_became_peered":"2024-08-19T14:43:16.091256+0000","last_unstale":"2024-08-19T14:49:50.775120+0000","last_undegraded":"2024-08-19T14:49:50.775120+0000","last_fullsized":"2024-08-19T14:49:50.775120+0000","mapping_epoch":800,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":801,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:49:50.775050+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:27.848870+0000","last_clean_scrub_stamp":"2024-08-19T14:49:50.775050+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:50:58.991509+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.5","version":"70'139","reported_seq":784,"reported_epoch":982,"state":"active+clean","last_fresh":"2024-08-19T14:49:49.514670+0000","last_change":"2024-08-19T14:49:43.757857+0000","last_active":"2024-08-19T14:49:49.514670+0000","last_peered":"2024-08-19T14:49:49.514670+0000","last_clean":"2024-08-19T14:49:49.514670+0000","last_became_active":"2024-08-19T14:45:26.105833+0000","last_became_peered":"2024-08-19T14:45:26.105833+0000","last_unstale":"2024-08-19T14:49:49.514670+0000","last_undegraded":"2024-08-19T14:49:49.514670+0000","last_fullsized":"2024-08-19T14:49:49.514670+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:49:43.757756+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:49:43.757756+0000","last_clean_scrub_stamp":"2024-08-19T14:49:43.757756+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:50:53.218703+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.4","version":"70'139","reported_seq":1262,"reported_epoch":982,"state":"active+clean","last_fresh":"2024-08-19T14:49:55.156903+0000","last_change":"2024-08-19T14:49:55.156952+0000","last_active":"2024-08-19T14:49:55.156903+0000","last_peered":"2024-08-19T14:49:55.156903+0000","last_clean":"2024-08-19T14:49:55.156903+0000","last_became_active":"2024-08-19T14:47:23.691499+0000","last_became_peered":"2024-08-19T14:47:23.691499+0000","last_unstale":"2024-08-19T14:49:55.156903+0000","last_undegraded":"2024-08-19T14:49:55.156903+0000","last_fullsized":"2024-08-19T14:49:55.156903+0000","mapping_epoch":905,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":906,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:49:55.156813+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:47:03.977741+0000","last_clean_scrub_stamp":"2024-08-19T14:49:55.156813+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:51:12.319241+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.a","version":"70'139","reported_seq":656,"reported_epoch":982,"state":"active+clean","last_fresh":"2024-08-19T14:49:49.460070+0000","last_change":"2024-08-19T14:49:49.460095+0000","last_active":"2024-08-19T14:49:49.460070+0000","last_peered":"2024-08-19T14:49:49.460070+0000","last_clean":"2024-08-19T14:49:49.460070+0000","last_became_active":"2024-08-19T14:49:12.325624+0000","last_became_peered":"2024-08-19T14:49:12.325624+0000","last_unstale":"2024-08-19T14:49:49.460070+0000","last_undegraded":"2024-08-19T14:49:49.460070+0000","last_fullsized":"2024-08-19T14:49:49.460070+0000","mapping_epoch":962,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":963,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:49:49.459994+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:44:28.862033+0000","last_clean_scrub_stamp":"2024-08-19T14:49:49.459994+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:50:53.079997+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.9","version":"70'139","reported_seq":1520,"reported_epoch":982,"state":"active+clean","last_fresh":"2024-08-19T14:49:49.276777+0000","last_change":"2024-08-19T14:48:23.298293+0000","last_active":"2024-08-19T14:49:49.276777+0000","last_peered":"2024-08-19T14:49:49.276777+0000","last_clean":"2024-08-19T14:49:49.276777+0000","last_became_active":"2024-08-19T14:47:23.691771+0000","last_became_peered":"2024-08-19T14:47:23.691771+0000","last_unstale":"2024-08-19T14:49:49.276777+0000","last_undegraded":"2024-08-19T14:49:49.276777+0000","last_fullsized":"2024-08-19T14:49:49.276777+0000","mapping_epoch":905,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":906,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:48:23.298198+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:48:23.298198+0000","last_clean_scrub_stamp":"2024-08-19T14:48:23.298198+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:49:52.353178+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.8","version":"70'139","reported_seq":523,"reported_epoch":982,"state":"active+clean","last_fresh":"2024-08-19T14:49:49.514673+0000","last_change":"2024-08-19T14:49:48.881647+0000","last_active":"2024-08-19T14:49:49.514673+0000","last_peered":"2024-08-19T14:49:49.514673+0000","last_clean":"2024-08-19T14:49:49.514673+0000","last_became_active":"2024-08-19T14:48:56.264160+0000","last_became_peered":"2024-08-19T14:48:56.264160+0000","last_unstale":"2024-08-19T14:49:49.514673+0000","last_undegraded":"2024-08-19T14:49:49.514673+0000","last_fullsized":"2024-08-19T14:49:49.514673+0000","mapping_epoch":951,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":952,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:49:48.881554+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:47:00.426007+0000","last_clean_scrub_stamp":"2024-08-19T14:49:48.881554+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:51:14.523131+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"70'139","reported_seq":1716,"reported_epoch":982,"state":"active+clean","last_fresh":"2024-08-19T14:49:50.075157+0000","last_change":"2024-08-19T14:49:50.075183+0000","last_active":"2024-08-19T14:49:50.075157+0000","last_peered":"2024-08-19T14:49:50.075157+0000","last_clean":"2024-08-19T14:49:50.075157+0000","last_became_active":"2024-08-19T14:48:32.140551+0000","last_became_peered":"2024-08-19T14:48:32.140551+0000","last_unstale":"2024-08-19T14:49:50.075157+0000","last_undegraded":"2024-08-19T14:49:50.075157+0000","last_fullsized":"2024-08-19T14:49:50.075157+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:49:50.075085+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:49:50.075085+0000","last_clean_scrub_stamp":"2024-08-19T14:49:50.075085+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:51:05.959780+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.e","version":"70'139","reported_seq":1914,"reported_epoch":982,"state":"active+clean","last_fresh":"2024-08-19T14:49:54.682328+0000","last_change":"2024-08-19T14:49:54.682356+0000","last_active":"2024-08-19T14:49:54.682328+0000","last_peered":"2024-08-19T14:49:54.682328+0000","last_clean":"2024-08-19T14:49:54.682328+0000","last_became_active":"2024-08-19T14:47:45.836038+0000","last_became_peered":"2024-08-19T14:47:45.836038+0000","last_unstale":"2024-08-19T14:49:54.682328+0000","last_undegraded":"2024-08-19T14:49:54.682328+0000","last_fullsized":"2024-08-19T14:49:54.682328+0000","mapping_epoch":917,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":918,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:49:54.682259+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:40:03.764672+0000","last_clean_scrub_stamp":"2024-08-19T14:49:54.682259+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:51:20.128747+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.f","version":"70'139","reported_seq":1607,"reported_epoch":982,"state":"active+clean","last_fresh":"2024-08-19T14:49:49.459646+0000","last_change":"2024-08-19T14:49:45.861846+0000","last_active":"2024-08-19T14:49:49.459646+0000","last_peered":"2024-08-19T14:49:49.459646+0000","last_clean":"2024-08-19T14:49:49.459646+0000","last_became_active":"2024-08-19T14:48:32.141263+0000","last_became_peered":"2024-08-19T14:48:32.141263+0000","last_unstale":"2024-08-19T14:49:49.459646+0000","last_undegraded":"2024-08-19T14:49:49.459646+0000","last_fullsized":"2024-08-19T14:49:49.459646+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:49:45.861719+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:49:45.861719+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:51:04.377191+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.c","version":"70'139","reported_seq":866,"reported_epoch":982,"state":"active+clean","last_fresh":"2024-08-19T14:49:49.514263+0000","last_change":"2024-08-19T14:49:47.965142+0000","last_active":"2024-08-19T14:49:49.514263+0000","last_peered":"2024-08-19T14:49:49.514263+0000","last_clean":"2024-08-19T14:49:49.514263+0000","last_became_active":"2024-08-19T14:48:29.138129+0000","last_became_peered":"2024-08-19T14:48:29.138129+0000","last_unstale":"2024-08-19T14:49:49.514263+0000","last_undegraded":"2024-08-19T14:49:49.514263+0000","last_fullsized":"2024-08-19T14:49:49.514263+0000","mapping_epoch":936,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":937,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:49:47.965063+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:22:52.164586+0000","last_clean_scrub_stamp":"2024-08-19T14:49:47.965063+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:50:54.438352+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1638400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.d","version":"70'139","reported_seq":483,"reported_epoch":982,"state":"active+clean","last_fresh":"2024-08-19T14:49:49.276848+0000","last_change":"2024-08-19T14:49:45.083734+0000","last_active":"2024-08-19T14:49:49.276848+0000","last_peered":"2024-08-19T14:49:49.276848+0000","last_clean":"2024-08-19T14:49:49.276848+0000","last_became_active":"2024-08-19T14:47:23.719768+0000","last_became_peered":"2024-08-19T14:47:23.719768+0000","last_unstale":"2024-08-19T14:49:49.276848+0000","last_undegraded":"2024-08-19T14:49:49.276848+0000","last_fullsized":"2024-08-19T14:49:49.276848+0000","mapping_epoch":905,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":906,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:49:45.083641+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:24:50.276408+0000","last_clean_scrub_stamp":"2024-08-19T14:49:45.083641+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:51:00.129068+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"70'139","reported_seq":1667,"reported_epoch":982,"state":"active+clean","last_fresh":"2024-08-19T14:49:49.514594+0000","last_change":"2024-08-19T14:49:27.009743+0000","last_active":"2024-08-19T14:49:49.514594+0000","last_peered":"2024-08-19T14:49:49.514594+0000","last_clean":"2024-08-19T14:49:49.514594+0000","last_became_active":"2024-08-19T14:48:56.264570+0000","last_became_peered":"2024-08-19T14:48:56.264570+0000","last_unstale":"2024-08-19T14:49:49.514594+0000","last_undegraded":"2024-08-19T14:49:49.514594+0000","last_fullsized":"2024-08-19T14:49:49.514594+0000","mapping_epoch":951,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":952,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:49:27.009655+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:48:12.648665+0000","last_clean_scrub_stamp":"2024-08-19T14:49:27.009655+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:50:35.137096+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"70'139","reported_seq":2210,"reported_epoch":982,"state":"active+clean","last_fresh":"2024-08-19T14:49:48.735118+0000","last_change":"2024-08-19T14:49:36.742195+0000","last_active":"2024-08-19T14:49:48.735118+0000","last_peered":"2024-08-19T14:49:48.735118+0000","last_clean":"2024-08-19T14:49:48.735118+0000","last_became_active":"2024-08-19T14:43:16.086968+0000","last_became_peered":"2024-08-19T14:43:16.086968+0000","last_unstale":"2024-08-19T14:49:48.735118+0000","last_undegraded":"2024-08-19T14:49:48.735118+0000","last_fullsized":"2024-08-19T14:49:48.735118+0000","mapping_epoch":800,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":801,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:49:36.742116+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:33:35.786105+0000","last_clean_scrub_stamp":"2024-08-19T14:49:36.742116+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:51:05.396593+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.10","version":"70'139","reported_seq":985,"reported_epoch":982,"state":"active+clean","last_fresh":"2024-08-19T14:49:54.846670+0000","last_change":"2024-08-19T14:49:54.846695+0000","last_active":"2024-08-19T14:49:54.846670+0000","last_peered":"2024-08-19T14:49:54.846670+0000","last_clean":"2024-08-19T14:49:54.846670+0000","last_became_active":"2024-08-19T14:48:56.264569+0000","last_became_peered":"2024-08-19T14:48:56.264569+0000","last_unstale":"2024-08-19T14:49:54.846670+0000","last_undegraded":"2024-08-19T14:49:54.846670+0000","last_fullsized":"2024-08-19T14:49:54.846670+0000","mapping_epoch":951,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":952,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:49:54.846604+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:34:58.616646+0000","last_clean_scrub_stamp":"2024-08-19T14:49:54.846604+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:51:10.543471+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"70'139","reported_seq":690,"reported_epoch":982,"state":"active+clean","last_fresh":"2024-08-19T14:49:48.848306+0000","last_change":"2024-08-19T14:49:43.164136+0000","last_active":"2024-08-19T14:49:48.848306+0000","last_peered":"2024-08-19T14:49:48.848306+0000","last_clean":"2024-08-19T14:49:48.848306+0000","last_became_active":"2024-08-19T14:48:29.139282+0000","last_became_peered":"2024-08-19T14:48:29.139282+0000","last_unstale":"2024-08-19T14:49:48.848306+0000","last_undegraded":"2024-08-19T14:49:48.848306+0000","last_fullsized":"2024-08-19T14:49:48.848306+0000","mapping_epoch":936,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":937,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:49:43.164045+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:41:47.147376+0000","last_clean_scrub_stamp":"2024-08-19T14:49:43.164045+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:50:45.850314+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"70'139","reported_seq":1856,"reported_epoch":982,"state":"active+clean","last_fresh":"2024-08-19T14:49:53.682323+0000","last_change":"2024-08-19T14:49:53.682323+0000","last_active":"2024-08-19T14:49:53.682323+0000","last_peered":"2024-08-19T14:49:53.682323+0000","last_clean":"2024-08-19T14:49:53.682323+0000","last_became_active":"2024-08-19T14:48:29.137929+0000","last_became_peered":"2024-08-19T14:48:29.137929+0000","last_unstale":"2024-08-19T14:49:53.682323+0000","last_undegraded":"2024-08-19T14:49:53.682323+0000","last_fullsized":"2024-08-19T14:49:53.682323+0000","mapping_epoch":936,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":937,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:49:53.682211+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:28:34.216370+0000","last_clean_scrub_stamp":"2024-08-19T14:49:53.682211+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:51:07.373407+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"70'139","reported_seq":2002,"reported_epoch":982,"state":"active+clean","last_fresh":"2024-08-19T14:49:48.498550+0000","last_change":"2024-08-19T14:49:45.138668+0000","last_active":"2024-08-19T14:49:48.498550+0000","last_peered":"2024-08-19T14:49:48.498550+0000","last_clean":"2024-08-19T14:49:48.498550+0000","last_became_active":"2024-08-19T14:48:32.139389+0000","last_became_peered":"2024-08-19T14:48:32.139389+0000","last_unstale":"2024-08-19T14:49:48.498550+0000","last_undegraded":"2024-08-19T14:49:48.498550+0000","last_fullsized":"2024-08-19T14:49:48.498550+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:49:45.138574+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:49:45.138574+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:50:58.887087+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.14","version":"70'139","reported_seq":1027,"reported_epoch":982,"state":"active+clean","last_fresh":"2024-08-19T14:49:49.514373+0000","last_change":"2024-08-19T14:49:35.856925+0000","last_active":"2024-08-19T14:49:49.514373+0000","last_peered":"2024-08-19T14:49:49.514373+0000","last_clean":"2024-08-19T14:49:49.514373+0000","last_became_active":"2024-08-19T14:49:27.410044+0000","last_became_peered":"2024-08-19T14:49:27.410044+0000","last_unstale":"2024-08-19T14:49:49.514373+0000","last_undegraded":"2024-08-19T14:49:49.514373+0000","last_fullsized":"2024-08-19T14:49:49.514373+0000","mapping_epoch":971,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":972,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:49:35.856840+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:49:35.856840+0000","last_clean_scrub_stamp":"2024-08-19T14:49:35.856840+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:50:56.571169+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.15","version":"70'139","reported_seq":495,"reported_epoch":982,"state":"active+clean","last_fresh":"2024-08-19T14:49:49.276829+0000","last_change":"2024-08-19T14:49:39.043344+0000","last_active":"2024-08-19T14:49:49.276829+0000","last_peered":"2024-08-19T14:49:49.276829+0000","last_clean":"2024-08-19T14:49:49.276829+0000","last_became_active":"2024-08-19T14:48:32.138473+0000","last_became_peered":"2024-08-19T14:48:32.138473+0000","last_unstale":"2024-08-19T14:49:49.276829+0000","last_undegraded":"2024-08-19T14:49:49.276829+0000","last_fullsized":"2024-08-19T14:49:49.276829+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:49:39.043212+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:20:09.180923+0000","last_clean_scrub_stamp":"2024-08-19T14:49:39.043212+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:50:47.000457+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"70'139","reported_seq":1839,"reported_epoch":982,"state":"active+clean","last_fresh":"2024-08-19T14:49:48.498204+0000","last_change":"2024-08-19T14:49:41.778293+0000","last_active":"2024-08-19T14:49:48.498204+0000","last_peered":"2024-08-19T14:49:48.498204+0000","last_clean":"2024-08-19T14:49:48.498204+0000","last_became_active":"2024-08-19T14:47:23.684905+0000","last_became_peered":"2024-08-19T14:47:23.684905+0000","last_unstale":"2024-08-19T14:49:48.498204+0000","last_undegraded":"2024-08-19T14:49:48.498204+0000","last_fullsized":"2024-08-19T14:49:48.498204+0000","mapping_epoch":905,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":906,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:49:41.778202+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:41:51.832284+0000","last_clean_scrub_stamp":"2024-08-19T14:49:41.778202+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:50:56.075830+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1b","version":"70'139","reported_seq":2222,"reported_epoch":982,"state":"active+clean","last_fresh":"2024-08-19T14:49:48.498074+0000","last_change":"2024-08-19T14:49:33.165119+0000","last_active":"2024-08-19T14:49:48.498074+0000","last_peered":"2024-08-19T14:49:48.498074+0000","last_clean":"2024-08-19T14:49:48.498074+0000","last_became_active":"2024-08-19T14:48:29.138457+0000","last_became_peered":"2024-08-19T14:48:29.138457+0000","last_unstale":"2024-08-19T14:49:48.498074+0000","last_undegraded":"2024-08-19T14:49:48.498074+0000","last_fullsized":"2024-08-19T14:49:48.498074+0000","mapping_epoch":936,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":937,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:49:33.165022+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:37:59.510697+0000","last_clean_scrub_stamp":"2024-08-19T14:49:33.165022+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:50:37.755249+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.18","version":"70'139","reported_seq":1175,"reported_epoch":982,"state":"active+clean","last_fresh":"2024-08-19T14:49:49.514517+0000","last_change":"2024-08-19T14:49:45.931143+0000","last_active":"2024-08-19T14:49:49.514517+0000","last_peered":"2024-08-19T14:49:49.514517+0000","last_clean":"2024-08-19T14:49:49.514517+0000","last_became_active":"2024-08-19T14:48:56.264698+0000","last_became_peered":"2024-08-19T14:48:56.264698+0000","last_unstale":"2024-08-19T14:49:49.514517+0000","last_undegraded":"2024-08-19T14:49:49.514517+0000","last_fullsized":"2024-08-19T14:49:49.514517+0000","mapping_epoch":951,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":952,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:49:45.931053+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:27:33.084161+0000","last_clean_scrub_stamp":"2024-08-19T14:49:45.931053+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:50:52.901003+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":603744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.19","version":"70'139","reported_seq":1075,"reported_epoch":982,"state":"active+clean","last_fresh":"2024-08-19T14:49:49.514589+0000","last_change":"2024-08-19T14:49:32.945893+0000","last_active":"2024-08-19T14:49:49.514589+0000","last_peered":"2024-08-19T14:49:49.514589+0000","last_clean":"2024-08-19T14:49:49.514589+0000","last_became_active":"2024-08-19T14:48:29.139455+0000","last_became_peered":"2024-08-19T14:48:29.139455+0000","last_unstale":"2024-08-19T14:49:49.514589+0000","last_undegraded":"2024-08-19T14:49:49.514589+0000","last_fullsized":"2024-08-19T14:49:49.514589+0000","mapping_epoch":936,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":937,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:49:32.945787+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:49:32.945787+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:50:59.804983+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1e","version":"70'139","reported_seq":1226,"reported_epoch":982,"state":"active+clean","last_fresh":"2024-08-19T14:49:48.498365+0000","last_change":"2024-08-19T14:48:52.530911+0000","last_active":"2024-08-19T14:49:48.498365+0000","last_peered":"2024-08-19T14:49:48.498365+0000","last_clean":"2024-08-19T14:49:48.498365+0000","last_became_active":"2024-08-19T14:47:23.687739+0000","last_became_peered":"2024-08-19T14:47:23.687739+0000","last_unstale":"2024-08-19T14:49:48.498365+0000","last_undegraded":"2024-08-19T14:49:48.498365+0000","last_fullsized":"2024-08-19T14:49:48.498365+0000","mapping_epoch":905,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":906,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:48:52.530809+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:27:04.780939+0000","last_clean_scrub_stamp":"2024-08-19T14:48:52.530809+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:50:16.020198+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1c","version":"70'139","reported_seq":1967,"reported_epoch":982,"state":"active+clean","last_fresh":"2024-08-19T14:49:48.498136+0000","last_change":"2024-08-19T14:48:49.327071+0000","last_active":"2024-08-19T14:49:48.498136+0000","last_peered":"2024-08-19T14:49:48.498136+0000","last_clean":"2024-08-19T14:49:48.498136+0000","last_became_active":"2024-08-19T14:45:26.105382+0000","last_became_peered":"2024-08-19T14:45:26.105382+0000","last_unstale":"2024-08-19T14:49:48.498136+0000","last_undegraded":"2024-08-19T14:49:48.498136+0000","last_fullsized":"2024-08-19T14:49:48.498136+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:48:49.326969+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:48:49.326969+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:50:06.043344+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1d","version":"70'139","reported_seq":1856,"reported_epoch":982,"state":"active+clean","last_fresh":"2024-08-19T14:49:48.735125+0000","last_change":"2024-08-19T14:48:50.576212+0000","last_active":"2024-08-19T14:49:48.735125+0000","last_peered":"2024-08-19T14:49:48.735125+0000","last_clean":"2024-08-19T14:49:48.735125+0000","last_became_active":"2024-08-19T14:35:11.540488+0000","last_became_peered":"2024-08-19T14:35:11.540488+0000","last_unstale":"2024-08-19T14:49:48.735125+0000","last_undegraded":"2024-08-19T14:49:48.735125+0000","last_fullsized":"2024-08-19T14:49:48.735125+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:48:50.576131+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:48:50.576131+0000","last_clean_scrub_stamp":"2024-08-19T14:48:50.576131+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:50:18.801673+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2242144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187920,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1237520,"kb_used_data":8712,"kb_used_omap":1,"kb_used_meta":1228798,"kb_avail":92507632,"statfs":{"total":95995035648,"available":94727815168,"internally_reserved":0,"allocated":8921088,"data_stored":5851774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":1258289604},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":539,"seq":2314987372787,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":257236,"kb_used_data":8712,"kb_used_omap":4,"kb_used_meta":248507,"kb_avail":93487916,"statfs":{"total":95995035648,"available":95731625984,"internally_reserved":0,"allocated":8921088,"data_stored":5851774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4407,"internal_metadata":254471881},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":74,"seq":317827580378,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1286804,"kb_used_data":8712,"kb_used_omap":1,"kb_used_meta":1278078,"kb_avail":92458348,"statfs":{"total":95995035648,"available":94677348352,"internally_reserved":0,"allocated":8921088,"data_stored":5851774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":1308752321},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874721,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1072788,"kb_used_data":8712,"kb_used_omap":1,"kb_used_meta":1064062,"kb_avail":92672364,"statfs":{"total":95995035648,"available":94896500736,"internally_reserved":0,"allocated":8921088,"data_stored":5851774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1089599941},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":57,"seq":244813136355,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1276892,"kb_used_data":8912,"kb_used_omap":1,"kb_used_meta":1267966,"kb_avail":92468260,"statfs":{"total":95995035648,"available":94687498240,"internally_reserved":0,"allocated":9125888,"data_stored":6053022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":1298397642},"hb_peers":[0,1,2,3,4,5,6,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430695,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":220692,"kb_used_data":9032,"kb_used_omap":1,"kb_used_meta":211646,"kb_avail":93524460,"statfs":{"total":95995035648,"available":95769047040,"internally_reserved":0,"allocated":9248768,"data_stored":6179454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":216725958},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692331,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1149336,"kb_used_data":8912,"kb_used_omap":1,"kb_used_meta":1140414,"kb_avail":92595816,"statfs":{"total":95995035648,"available":94818115584,"internally_reserved":0,"allocated":9125888,"data_stored":6053022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1167784389},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986672,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1288340,"kb_used_data":8712,"kb_used_omap":1,"kb_used_meta":1279614,"kb_avail":92456812,"statfs":{"total":95995035648,"available":94675775488,"internally_reserved":0,"allocated":8921088,"data_stored":5851774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1310325189},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150197,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1186064,"kb_used_data":9032,"kb_used_omap":1,"kb_used_meta":1177022,"kb_avail":92559088,"statfs":{"total":95995035648,"available":94780506112,"internally_reserved":0,"allocated":9248768,"data_stored":6179454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1205270983},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":634,"seq":2723009265843,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":554652,"kb_used_data":8912,"kb_used_omap":4,"kb_used_meta":545723,"kb_avail":93190500,"statfs":{"total":95995035648,"available":95427072000,"internally_reserved":0,"allocated":9125888,"data_stored":6053022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4407,"internal_metadata":558821065},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575358,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1525272,"kb_used_data":8712,"kb_used_omap":1,"kb_used_meta":1516542,"kb_avail":92219880,"statfs":{"total":95995035648,"available":94433157120,"internally_reserved":0,"allocated":8921088,"data_stored":5851774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1552939461},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738882,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1000912,"kb_used_data":9032,"kb_used_omap":1,"kb_used_meta":991870,"kb_avail":92744240,"statfs":{"total":95995035648,"available":94970101760,"internally_reserved":0,"allocated":9248768,"data_stored":6179454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1015675333},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T14:49:59.573 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-19T14:49:59.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:59 smithi045 ceph-mon[20143]: pgmap v2284: 31 pgs: 31 active+clean; 577 KiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:49:59.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:49:59 smithi045 ceph-mon[20143]: from='client.18609 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:49:59.829 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:49:59.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:59 smithi031 ceph-mon[16336]: pgmap v2284: 31 pgs: 31 active+clean; 577 KiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:49:59.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:49:59 smithi031 ceph-mon[16336]: from='client.18609 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:50:00.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:59 smithi032 ceph-mon[20004]: pgmap v2284: 31 pgs: 31 active+clean; 577 KiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:50:00.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:49:59 smithi032 ceph-mon[20004]: from='client.18609 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:50:00.565 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-08-19T14:50:00.604 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:50:00 smithi031 ceph-mon[16336]: 1.0 scrub starts 2024-08-19T14:50:00.605 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:50:00 smithi031 ceph-mon[16336]: 1.0 scrub ok 2024-08-19T14:50:00.605 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:50:00 smithi031 ceph-mon[16336]: overall HEALTH_OK 2024-08-19T14:50:00.605 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-08-19T14:50:00.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:50:00 smithi045 ceph-mon[20143]: 1.0 scrub starts 2024-08-19T14:50:00.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:50:00 smithi045 ceph-mon[20143]: 1.0 scrub ok 2024-08-19T14:50:00.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:50:00 smithi045 ceph-mon[20143]: overall HEALTH_OK 2024-08-19T14:50:01.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:50:00 smithi032 ceph-mon[20004]: 1.0 scrub starts 2024-08-19T14:50:01.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:50:00 smithi032 ceph-mon[20004]: 1.0 scrub ok 2024-08-19T14:50:01.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:50:00 smithi032 ceph-mon[20004]: overall HEALTH_OK 2024-08-19T14:50:01.270 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":2285,"stamp":"2024-08-19T14:50:00.439802+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2242144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":93,"acting":93,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":93,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":12056492,"kb_used_data":106104,"kb_used_omap":24,"kb_used_meta":11950247,"kb_avail":1112885332,"statfs":{"total":1151940427776,"available":1139594579968,"internally_reserved":0,"allocated":108650496,"data_stored":71808072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24761,"internal_metadata":12237053767},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.942716"},"pg_stats":[{"pgid":"1.0","version":"70'139","reported_seq":2102,"reported_epoch":982,"state":"active+clean","last_fresh":"2024-08-19T14:49:49.514321+0000","last_change":"2024-08-19T14:49:27.410413+0000","last_active":"2024-08-19T14:49:49.514321+0000","last_peered":"2024-08-19T14:49:49.514321+0000","last_clean":"2024-08-19T14:49:49.514321+0000","last_became_active":"2024-08-19T14:49:27.410066+0000","last_became_peered":"2024-08-19T14:49:27.410066+0000","last_unstale":"2024-08-19T14:49:49.514321+0000","last_undegraded":"2024-08-19T14:49:49.514321+0000","last_fullsized":"2024-08-19T14:49:49.514321+0000","mapping_epoch":971,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":972,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:48:35.270407+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:41:45.177828+0000","last_clean_scrub_stamp":"2024-08-19T14:48:35.270407+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:49:51.310481+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,0],"acting":[2,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"70'139","reported_seq":2182,"reported_epoch":982,"state":"active+clean","last_fresh":"2024-08-19T14:49:49.514429+0000","last_change":"2024-08-19T14:49:35.443305+0000","last_active":"2024-08-19T14:49:49.514429+0000","last_peered":"2024-08-19T14:49:49.514429+0000","last_clean":"2024-08-19T14:49:49.514429+0000","last_became_active":"2024-08-19T14:49:35.442955+0000","last_became_peered":"2024-08-19T14:49:35.442955+0000","last_unstale":"2024-08-19T14:49:49.514429+0000","last_undegraded":"2024-08-19T14:49:49.514429+0000","last_fullsized":"2024-08-19T14:49:49.514429+0000","mapping_epoch":976,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":977,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:49:30.522127+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:49:30.522127+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:50:44.180831+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"70'139","reported_seq":584,"reported_epoch":982,"state":"active+clean","last_fresh":"2024-08-19T14:49:49.276672+0000","last_change":"2024-08-19T14:49:37.096769+0000","last_active":"2024-08-19T14:49:49.276672+0000","last_peered":"2024-08-19T14:49:49.276672+0000","last_clean":"2024-08-19T14:49:49.276672+0000","last_became_active":"2024-08-19T14:48:29.137710+0000","last_became_peered":"2024-08-19T14:48:29.137710+0000","last_unstale":"2024-08-19T14:49:49.276672+0000","last_undegraded":"2024-08-19T14:49:49.276672+0000","last_fullsized":"2024-08-19T14:49:49.276672+0000","mapping_epoch":936,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":937,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:49:37.096669+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:48:24.313364+0000","last_clean_scrub_stamp":"2024-08-19T14:49:37.096669+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:51:02.158270+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"70'139","reported_seq":820,"reported_epoch":982,"state":"active+clean","last_fresh":"2024-08-19T14:49:49.514591+0000","last_change":"2024-08-19T14:49:30.896533+0000","last_active":"2024-08-19T14:49:49.514591+0000","last_peered":"2024-08-19T14:49:49.514591+0000","last_clean":"2024-08-19T14:49:49.514591+0000","last_became_active":"2024-08-19T14:48:29.139386+0000","last_became_peered":"2024-08-19T14:48:29.139386+0000","last_unstale":"2024-08-19T14:49:49.514591+0000","last_undegraded":"2024-08-19T14:49:49.514591+0000","last_fullsized":"2024-08-19T14:49:49.514591+0000","mapping_epoch":936,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":937,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:49:30.896441+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:49:30.896441+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:50:57.035316+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.7","version":"70'139","reported_seq":1345,"reported_epoch":982,"state":"active+clean","last_fresh":"2024-08-19T14:49:49.459792+0000","last_change":"2024-08-19T14:48:46.622665+0000","last_active":"2024-08-19T14:49:49.459792+0000","last_peered":"2024-08-19T14:49:49.459792+0000","last_clean":"2024-08-19T14:49:49.459792+0000","last_became_active":"2024-08-19T14:47:45.871964+0000","last_became_peered":"2024-08-19T14:47:45.871964+0000","last_unstale":"2024-08-19T14:49:49.459792+0000","last_undegraded":"2024-08-19T14:49:49.459792+0000","last_fullsized":"2024-08-19T14:49:49.459792+0000","mapping_epoch":917,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":918,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:48:46.622591+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:21:46.057624+0000","last_clean_scrub_stamp":"2024-08-19T14:48:46.622591+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:50:15.453705+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"70'139","reported_seq":677,"reported_epoch":982,"state":"active+clean","last_fresh":"2024-08-19T14:49:50.775120+0000","last_change":"2024-08-19T14:49:50.775147+0000","last_active":"2024-08-19T14:49:50.775120+0000","last_peered":"2024-08-19T14:49:50.775120+0000","last_clean":"2024-08-19T14:49:50.775120+0000","last_became_active":"2024-08-19T14:43:16.091256+0000","last_became_peered":"2024-08-19T14:43:16.091256+0000","last_unstale":"2024-08-19T14:49:50.775120+0000","last_undegraded":"2024-08-19T14:49:50.775120+0000","last_fullsized":"2024-08-19T14:49:50.775120+0000","mapping_epoch":800,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":801,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:49:50.775050+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:18:27.848870+0000","last_clean_scrub_stamp":"2024-08-19T14:49:50.775050+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:50:58.991509+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.5","version":"70'139","reported_seq":784,"reported_epoch":982,"state":"active+clean","last_fresh":"2024-08-19T14:49:49.514670+0000","last_change":"2024-08-19T14:49:43.757857+0000","last_active":"2024-08-19T14:49:49.514670+0000","last_peered":"2024-08-19T14:49:49.514670+0000","last_clean":"2024-08-19T14:49:49.514670+0000","last_became_active":"2024-08-19T14:45:26.105833+0000","last_became_peered":"2024-08-19T14:45:26.105833+0000","last_unstale":"2024-08-19T14:49:49.514670+0000","last_undegraded":"2024-08-19T14:49:49.514670+0000","last_fullsized":"2024-08-19T14:49:49.514670+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:49:43.757756+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:49:43.757756+0000","last_clean_scrub_stamp":"2024-08-19T14:49:43.757756+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:50:53.218703+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.4","version":"70'139","reported_seq":1262,"reported_epoch":982,"state":"active+clean","last_fresh":"2024-08-19T14:49:55.156903+0000","last_change":"2024-08-19T14:49:55.156952+0000","last_active":"2024-08-19T14:49:55.156903+0000","last_peered":"2024-08-19T14:49:55.156903+0000","last_clean":"2024-08-19T14:49:55.156903+0000","last_became_active":"2024-08-19T14:47:23.691499+0000","last_became_peered":"2024-08-19T14:47:23.691499+0000","last_unstale":"2024-08-19T14:49:55.156903+0000","last_undegraded":"2024-08-19T14:49:55.156903+0000","last_fullsized":"2024-08-19T14:49:55.156903+0000","mapping_epoch":905,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":906,"parent":"0.0","parent_split_bits":3,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:49:55.156813+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:47:03.977741+0000","last_clean_scrub_stamp":"2024-08-19T14:49:55.156813+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:51:12.319241+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.a","version":"70'139","reported_seq":656,"reported_epoch":982,"state":"active+clean","last_fresh":"2024-08-19T14:49:49.460070+0000","last_change":"2024-08-19T14:49:49.460095+0000","last_active":"2024-08-19T14:49:49.460070+0000","last_peered":"2024-08-19T14:49:49.460070+0000","last_clean":"2024-08-19T14:49:49.460070+0000","last_became_active":"2024-08-19T14:49:12.325624+0000","last_became_peered":"2024-08-19T14:49:12.325624+0000","last_unstale":"2024-08-19T14:49:49.460070+0000","last_undegraded":"2024-08-19T14:49:49.460070+0000","last_fullsized":"2024-08-19T14:49:49.460070+0000","mapping_epoch":962,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":963,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:49:49.459994+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:44:28.862033+0000","last_clean_scrub_stamp":"2024-08-19T14:49:49.459994+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:50:53.079997+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.9","version":"70'139","reported_seq":1520,"reported_epoch":982,"state":"active+clean","last_fresh":"2024-08-19T14:49:49.276777+0000","last_change":"2024-08-19T14:48:23.298293+0000","last_active":"2024-08-19T14:49:49.276777+0000","last_peered":"2024-08-19T14:49:49.276777+0000","last_clean":"2024-08-19T14:49:49.276777+0000","last_became_active":"2024-08-19T14:47:23.691771+0000","last_became_peered":"2024-08-19T14:47:23.691771+0000","last_unstale":"2024-08-19T14:49:49.276777+0000","last_undegraded":"2024-08-19T14:49:49.276777+0000","last_fullsized":"2024-08-19T14:49:49.276777+0000","mapping_epoch":905,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":906,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:48:23.298198+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:48:23.298198+0000","last_clean_scrub_stamp":"2024-08-19T14:48:23.298198+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:49:52.353178+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.8","version":"70'139","reported_seq":523,"reported_epoch":982,"state":"active+clean","last_fresh":"2024-08-19T14:49:49.514673+0000","last_change":"2024-08-19T14:49:48.881647+0000","last_active":"2024-08-19T14:49:49.514673+0000","last_peered":"2024-08-19T14:49:49.514673+0000","last_clean":"2024-08-19T14:49:49.514673+0000","last_became_active":"2024-08-19T14:48:56.264160+0000","last_became_peered":"2024-08-19T14:48:56.264160+0000","last_unstale":"2024-08-19T14:49:49.514673+0000","last_undegraded":"2024-08-19T14:49:49.514673+0000","last_fullsized":"2024-08-19T14:49:49.514673+0000","mapping_epoch":951,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":952,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:49:48.881554+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:47:00.426007+0000","last_clean_scrub_stamp":"2024-08-19T14:49:48.881554+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:51:14.523131+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"70'139","reported_seq":1716,"reported_epoch":982,"state":"active+clean","last_fresh":"2024-08-19T14:49:50.075157+0000","last_change":"2024-08-19T14:49:50.075183+0000","last_active":"2024-08-19T14:49:50.075157+0000","last_peered":"2024-08-19T14:49:50.075157+0000","last_clean":"2024-08-19T14:49:50.075157+0000","last_became_active":"2024-08-19T14:48:32.140551+0000","last_became_peered":"2024-08-19T14:48:32.140551+0000","last_unstale":"2024-08-19T14:49:50.075157+0000","last_undegraded":"2024-08-19T14:49:50.075157+0000","last_fullsized":"2024-08-19T14:49:50.075157+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:49:50.075085+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:49:50.075085+0000","last_clean_scrub_stamp":"2024-08-19T14:49:50.075085+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:51:05.959780+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.e","version":"70'139","reported_seq":1914,"reported_epoch":982,"state":"active+clean","last_fresh":"2024-08-19T14:49:54.682328+0000","last_change":"2024-08-19T14:49:54.682356+0000","last_active":"2024-08-19T14:49:54.682328+0000","last_peered":"2024-08-19T14:49:54.682328+0000","last_clean":"2024-08-19T14:49:54.682328+0000","last_became_active":"2024-08-19T14:47:45.836038+0000","last_became_peered":"2024-08-19T14:47:45.836038+0000","last_unstale":"2024-08-19T14:49:54.682328+0000","last_undegraded":"2024-08-19T14:49:54.682328+0000","last_fullsized":"2024-08-19T14:49:54.682328+0000","mapping_epoch":917,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":918,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:49:54.682259+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:40:03.764672+0000","last_clean_scrub_stamp":"2024-08-19T14:49:54.682259+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:51:20.128747+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.f","version":"70'139","reported_seq":1608,"reported_epoch":982,"state":"active+clean","last_fresh":"2024-08-19T14:49:49.459724+0000","last_change":"2024-08-19T14:49:45.861846+0000","last_active":"2024-08-19T14:49:49.459724+0000","last_peered":"2024-08-19T14:49:49.459724+0000","last_clean":"2024-08-19T14:49:49.459724+0000","last_became_active":"2024-08-19T14:48:32.141263+0000","last_became_peered":"2024-08-19T14:48:32.141263+0000","last_unstale":"2024-08-19T14:49:49.459724+0000","last_undegraded":"2024-08-19T14:49:49.459724+0000","last_fullsized":"2024-08-19T14:49:49.459724+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:49:45.861719+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:49:45.861719+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:51:04.377191+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.c","version":"70'139","reported_seq":866,"reported_epoch":982,"state":"active+clean","last_fresh":"2024-08-19T14:49:49.514263+0000","last_change":"2024-08-19T14:49:47.965142+0000","last_active":"2024-08-19T14:49:49.514263+0000","last_peered":"2024-08-19T14:49:49.514263+0000","last_clean":"2024-08-19T14:49:49.514263+0000","last_became_active":"2024-08-19T14:48:29.138129+0000","last_became_peered":"2024-08-19T14:48:29.138129+0000","last_unstale":"2024-08-19T14:49:49.514263+0000","last_undegraded":"2024-08-19T14:49:49.514263+0000","last_fullsized":"2024-08-19T14:49:49.514263+0000","mapping_epoch":936,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":937,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:49:47.965063+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:22:52.164586+0000","last_clean_scrub_stamp":"2024-08-19T14:49:47.965063+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:50:54.438352+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1638400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.d","version":"70'139","reported_seq":483,"reported_epoch":982,"state":"active+clean","last_fresh":"2024-08-19T14:49:49.276848+0000","last_change":"2024-08-19T14:49:45.083734+0000","last_active":"2024-08-19T14:49:49.276848+0000","last_peered":"2024-08-19T14:49:49.276848+0000","last_clean":"2024-08-19T14:49:49.276848+0000","last_became_active":"2024-08-19T14:47:23.719768+0000","last_became_peered":"2024-08-19T14:47:23.719768+0000","last_unstale":"2024-08-19T14:49:49.276848+0000","last_undegraded":"2024-08-19T14:49:49.276848+0000","last_fullsized":"2024-08-19T14:49:49.276848+0000","mapping_epoch":905,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":906,"parent":"0.0","parent_split_bits":4,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:49:45.083641+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:24:50.276408+0000","last_clean_scrub_stamp":"2024-08-19T14:49:45.083641+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:51:00.129068+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"70'139","reported_seq":1667,"reported_epoch":982,"state":"active+clean","last_fresh":"2024-08-19T14:49:49.514594+0000","last_change":"2024-08-19T14:49:27.009743+0000","last_active":"2024-08-19T14:49:49.514594+0000","last_peered":"2024-08-19T14:49:49.514594+0000","last_clean":"2024-08-19T14:49:49.514594+0000","last_became_active":"2024-08-19T14:48:56.264570+0000","last_became_peered":"2024-08-19T14:48:56.264570+0000","last_unstale":"2024-08-19T14:49:49.514594+0000","last_undegraded":"2024-08-19T14:49:49.514594+0000","last_fullsized":"2024-08-19T14:49:49.514594+0000","mapping_epoch":951,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":952,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:49:27.009655+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:48:12.648665+0000","last_clean_scrub_stamp":"2024-08-19T14:49:27.009655+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:50:35.137096+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"70'139","reported_seq":2210,"reported_epoch":982,"state":"active+clean","last_fresh":"2024-08-19T14:49:48.735118+0000","last_change":"2024-08-19T14:49:36.742195+0000","last_active":"2024-08-19T14:49:48.735118+0000","last_peered":"2024-08-19T14:49:48.735118+0000","last_clean":"2024-08-19T14:49:48.735118+0000","last_became_active":"2024-08-19T14:43:16.086968+0000","last_became_peered":"2024-08-19T14:43:16.086968+0000","last_unstale":"2024-08-19T14:49:48.735118+0000","last_undegraded":"2024-08-19T14:49:48.735118+0000","last_fullsized":"2024-08-19T14:49:48.735118+0000","mapping_epoch":800,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":801,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:49:36.742116+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:33:35.786105+0000","last_clean_scrub_stamp":"2024-08-19T14:49:36.742116+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:51:05.396593+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.10","version":"70'139","reported_seq":985,"reported_epoch":982,"state":"active+clean","last_fresh":"2024-08-19T14:49:54.846670+0000","last_change":"2024-08-19T14:49:54.846695+0000","last_active":"2024-08-19T14:49:54.846670+0000","last_peered":"2024-08-19T14:49:54.846670+0000","last_clean":"2024-08-19T14:49:54.846670+0000","last_became_active":"2024-08-19T14:48:56.264569+0000","last_became_peered":"2024-08-19T14:48:56.264569+0000","last_unstale":"2024-08-19T14:49:54.846670+0000","last_undegraded":"2024-08-19T14:49:54.846670+0000","last_fullsized":"2024-08-19T14:49:54.846670+0000","mapping_epoch":951,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":952,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:49:54.846604+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:34:58.616646+0000","last_clean_scrub_stamp":"2024-08-19T14:49:54.846604+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:51:10.543471+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"70'139","reported_seq":690,"reported_epoch":982,"state":"active+clean","last_fresh":"2024-08-19T14:49:48.848306+0000","last_change":"2024-08-19T14:49:43.164136+0000","last_active":"2024-08-19T14:49:48.848306+0000","last_peered":"2024-08-19T14:49:48.848306+0000","last_clean":"2024-08-19T14:49:48.848306+0000","last_became_active":"2024-08-19T14:48:29.139282+0000","last_became_peered":"2024-08-19T14:48:29.139282+0000","last_unstale":"2024-08-19T14:49:48.848306+0000","last_undegraded":"2024-08-19T14:49:48.848306+0000","last_fullsized":"2024-08-19T14:49:48.848306+0000","mapping_epoch":936,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":937,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:49:43.164045+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:41:47.147376+0000","last_clean_scrub_stamp":"2024-08-19T14:49:43.164045+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:50:45.850314+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"70'139","reported_seq":1856,"reported_epoch":982,"state":"active+clean","last_fresh":"2024-08-19T14:49:53.682323+0000","last_change":"2024-08-19T14:49:53.682323+0000","last_active":"2024-08-19T14:49:53.682323+0000","last_peered":"2024-08-19T14:49:53.682323+0000","last_clean":"2024-08-19T14:49:53.682323+0000","last_became_active":"2024-08-19T14:48:29.137929+0000","last_became_peered":"2024-08-19T14:48:29.137929+0000","last_unstale":"2024-08-19T14:49:53.682323+0000","last_undegraded":"2024-08-19T14:49:53.682323+0000","last_fullsized":"2024-08-19T14:49:53.682323+0000","mapping_epoch":936,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":937,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:49:53.682211+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:28:34.216370+0000","last_clean_scrub_stamp":"2024-08-19T14:49:53.682211+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:51:07.373407+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"70'139","reported_seq":2003,"reported_epoch":982,"state":"active+clean","last_fresh":"2024-08-19T14:49:48.498610+0000","last_change":"2024-08-19T14:49:45.138668+0000","last_active":"2024-08-19T14:49:48.498610+0000","last_peered":"2024-08-19T14:49:48.498610+0000","last_clean":"2024-08-19T14:49:48.498610+0000","last_became_active":"2024-08-19T14:48:32.139389+0000","last_became_peered":"2024-08-19T14:48:32.139389+0000","last_unstale":"2024-08-19T14:49:48.498610+0000","last_undegraded":"2024-08-19T14:49:48.498610+0000","last_fullsized":"2024-08-19T14:49:48.498610+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:49:45.138574+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:10.672834+0000","last_clean_scrub_stamp":"2024-08-19T14:49:45.138574+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:50:58.887087+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.14","version":"70'139","reported_seq":1027,"reported_epoch":982,"state":"active+clean","last_fresh":"2024-08-19T14:49:49.514373+0000","last_change":"2024-08-19T14:49:35.856925+0000","last_active":"2024-08-19T14:49:49.514373+0000","last_peered":"2024-08-19T14:49:49.514373+0000","last_clean":"2024-08-19T14:49:49.514373+0000","last_became_active":"2024-08-19T14:49:27.410044+0000","last_became_peered":"2024-08-19T14:49:27.410044+0000","last_unstale":"2024-08-19T14:49:49.514373+0000","last_undegraded":"2024-08-19T14:49:49.514373+0000","last_fullsized":"2024-08-19T14:49:49.514373+0000","mapping_epoch":971,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":972,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:49:35.856840+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:49:35.856840+0000","last_clean_scrub_stamp":"2024-08-19T14:49:35.856840+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:50:56.571169+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.15","version":"70'139","reported_seq":495,"reported_epoch":982,"state":"active+clean","last_fresh":"2024-08-19T14:49:49.276829+0000","last_change":"2024-08-19T14:49:39.043344+0000","last_active":"2024-08-19T14:49:49.276829+0000","last_peered":"2024-08-19T14:49:49.276829+0000","last_clean":"2024-08-19T14:49:49.276829+0000","last_became_active":"2024-08-19T14:48:32.138473+0000","last_became_peered":"2024-08-19T14:48:32.138473+0000","last_unstale":"2024-08-19T14:49:49.276829+0000","last_undegraded":"2024-08-19T14:49:49.276829+0000","last_fullsized":"2024-08-19T14:49:49.276829+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:49:39.043212+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:20:09.180923+0000","last_clean_scrub_stamp":"2024-08-19T14:49:39.043212+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:50:47.000457+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"70'139","reported_seq":1839,"reported_epoch":982,"state":"active+clean","last_fresh":"2024-08-19T14:49:48.498204+0000","last_change":"2024-08-19T14:49:41.778293+0000","last_active":"2024-08-19T14:49:48.498204+0000","last_peered":"2024-08-19T14:49:48.498204+0000","last_clean":"2024-08-19T14:49:48.498204+0000","last_became_active":"2024-08-19T14:47:23.684905+0000","last_became_peered":"2024-08-19T14:47:23.684905+0000","last_unstale":"2024-08-19T14:49:48.498204+0000","last_undegraded":"2024-08-19T14:49:48.498204+0000","last_fullsized":"2024-08-19T14:49:48.498204+0000","mapping_epoch":905,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":906,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:49:41.778202+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:41:51.832284+0000","last_clean_scrub_stamp":"2024-08-19T14:49:41.778202+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:50:56.075830+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1b","version":"70'139","reported_seq":2222,"reported_epoch":982,"state":"active+clean","last_fresh":"2024-08-19T14:49:48.498074+0000","last_change":"2024-08-19T14:49:33.165119+0000","last_active":"2024-08-19T14:49:48.498074+0000","last_peered":"2024-08-19T14:49:48.498074+0000","last_clean":"2024-08-19T14:49:48.498074+0000","last_became_active":"2024-08-19T14:48:29.138457+0000","last_became_peered":"2024-08-19T14:48:29.138457+0000","last_unstale":"2024-08-19T14:49:48.498074+0000","last_undegraded":"2024-08-19T14:49:48.498074+0000","last_fullsized":"2024-08-19T14:49:48.498074+0000","mapping_epoch":936,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":937,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:49:33.165022+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:37:59.510697+0000","last_clean_scrub_stamp":"2024-08-19T14:49:33.165022+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:50:37.755249+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.18","version":"70'139","reported_seq":1175,"reported_epoch":982,"state":"active+clean","last_fresh":"2024-08-19T14:49:49.514517+0000","last_change":"2024-08-19T14:49:45.931143+0000","last_active":"2024-08-19T14:49:49.514517+0000","last_peered":"2024-08-19T14:49:49.514517+0000","last_clean":"2024-08-19T14:49:49.514517+0000","last_became_active":"2024-08-19T14:48:56.264698+0000","last_became_peered":"2024-08-19T14:48:56.264698+0000","last_unstale":"2024-08-19T14:49:49.514517+0000","last_undegraded":"2024-08-19T14:49:49.514517+0000","last_fullsized":"2024-08-19T14:49:49.514517+0000","mapping_epoch":951,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":952,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:49:45.931053+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:27:33.084161+0000","last_clean_scrub_stamp":"2024-08-19T14:49:45.931053+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:50:52.901003+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":603744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.19","version":"70'139","reported_seq":1075,"reported_epoch":982,"state":"active+clean","last_fresh":"2024-08-19T14:49:49.514589+0000","last_change":"2024-08-19T14:49:32.945893+0000","last_active":"2024-08-19T14:49:49.514589+0000","last_peered":"2024-08-19T14:49:49.514589+0000","last_clean":"2024-08-19T14:49:49.514589+0000","last_became_active":"2024-08-19T14:48:29.139455+0000","last_became_peered":"2024-08-19T14:48:29.139455+0000","last_unstale":"2024-08-19T14:49:49.514589+0000","last_undegraded":"2024-08-19T14:49:49.514589+0000","last_fullsized":"2024-08-19T14:49:49.514589+0000","mapping_epoch":936,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":937,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:49:32.945787+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:16:03.918046+0000","last_clean_scrub_stamp":"2024-08-19T14:49:32.945787+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:50:59.804983+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1e","version":"70'139","reported_seq":1226,"reported_epoch":982,"state":"active+clean","last_fresh":"2024-08-19T14:49:48.498365+0000","last_change":"2024-08-19T14:48:52.530911+0000","last_active":"2024-08-19T14:49:48.498365+0000","last_peered":"2024-08-19T14:49:48.498365+0000","last_clean":"2024-08-19T14:49:48.498365+0000","last_became_active":"2024-08-19T14:47:23.687739+0000","last_became_peered":"2024-08-19T14:47:23.687739+0000","last_unstale":"2024-08-19T14:49:48.498365+0000","last_undegraded":"2024-08-19T14:49:48.498365+0000","last_fullsized":"2024-08-19T14:49:48.498365+0000","mapping_epoch":905,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":906,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:48:52.530809+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:27:04.780939+0000","last_clean_scrub_stamp":"2024-08-19T14:48:52.530809+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:50:16.020198+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1c","version":"70'139","reported_seq":1967,"reported_epoch":982,"state":"active+clean","last_fresh":"2024-08-19T14:49:48.498136+0000","last_change":"2024-08-19T14:48:49.327071+0000","last_active":"2024-08-19T14:49:48.498136+0000","last_peered":"2024-08-19T14:49:48.498136+0000","last_clean":"2024-08-19T14:49:48.498136+0000","last_became_active":"2024-08-19T14:45:26.105382+0000","last_became_peered":"2024-08-19T14:45:26.105382+0000","last_unstale":"2024-08-19T14:49:48.498136+0000","last_undegraded":"2024-08-19T14:49:48.498136+0000","last_fullsized":"2024-08-19T14:49:48.498136+0000","mapping_epoch":865,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:48:49.326969+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:10:40.178629+0000","last_clean_scrub_stamp":"2024-08-19T14:48:49.326969+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:50:06.043344+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1d","version":"70'139","reported_seq":1856,"reported_epoch":982,"state":"active+clean","last_fresh":"2024-08-19T14:49:48.735125+0000","last_change":"2024-08-19T14:48:50.576212+0000","last_active":"2024-08-19T14:49:48.735125+0000","last_peered":"2024-08-19T14:49:48.735125+0000","last_clean":"2024-08-19T14:49:48.735125+0000","last_became_active":"2024-08-19T14:35:11.540488+0000","last_became_peered":"2024-08-19T14:35:11.540488+0000","last_unstale":"2024-08-19T14:49:48.735125+0000","last_undegraded":"2024-08-19T14:49:48.735125+0000","last_fullsized":"2024-08-19T14:49:48.735125+0000","mapping_epoch":634,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":635,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'139","last_scrub_stamp":"2024-08-19T14:48:50.576131+0000","last_deep_scrub":"70'139","last_deep_scrub_stamp":"2024-08-19T14:48:50.576131+0000","last_clean_scrub_stamp":"2024-08-19T14:48:50.576131+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T14:50:18.801673+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2242144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187920,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1237520,"kb_used_data":8712,"kb_used_omap":1,"kb_used_meta":1228798,"kb_avail":92507632,"statfs":{"total":95995035648,"available":94727815168,"internally_reserved":0,"allocated":8921088,"data_stored":5851774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":1258289604},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":539,"seq":2314987372788,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":257236,"kb_used_data":8712,"kb_used_omap":4,"kb_used_meta":248507,"kb_avail":93487916,"statfs":{"total":95995035648,"available":95731625984,"internally_reserved":0,"allocated":8921088,"data_stored":5851774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4407,"internal_metadata":254471881},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":74,"seq":317827580379,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1286804,"kb_used_data":8712,"kb_used_omap":1,"kb_used_meta":1278078,"kb_avail":92458348,"statfs":{"total":95995035648,"available":94677348352,"internally_reserved":0,"allocated":8921088,"data_stored":5851774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":1308752321},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874722,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1072788,"kb_used_data":8712,"kb_used_omap":1,"kb_used_meta":1064062,"kb_avail":92672364,"statfs":{"total":95995035648,"available":94896500736,"internally_reserved":0,"allocated":8921088,"data_stored":5851774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1089599941},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":57,"seq":244813136356,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1276888,"kb_used_data":8912,"kb_used_omap":1,"kb_used_meta":1267966,"kb_avail":92468264,"statfs":{"total":95995035648,"available":94687502336,"internally_reserved":0,"allocated":9125888,"data_stored":6053022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":1298397642},"hb_peers":[0,1,2,3,4,5,6,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430696,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":220692,"kb_used_data":9032,"kb_used_omap":1,"kb_used_meta":211646,"kb_avail":93524460,"statfs":{"total":95995035648,"available":95769047040,"internally_reserved":0,"allocated":9248768,"data_stored":6179454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":216725958},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692331,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1149336,"kb_used_data":8912,"kb_used_omap":1,"kb_used_meta":1140414,"kb_avail":92595816,"statfs":{"total":95995035648,"available":94818115584,"internally_reserved":0,"allocated":9125888,"data_stored":6053022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1167784389},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986673,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1288336,"kb_used_data":8712,"kb_used_omap":1,"kb_used_meta":1279614,"kb_avail":92456816,"statfs":{"total":95995035648,"available":94675779584,"internally_reserved":0,"allocated":8921088,"data_stored":5851774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1310325189},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150197,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1186064,"kb_used_data":9032,"kb_used_omap":1,"kb_used_meta":1177022,"kb_avail":92559088,"statfs":{"total":95995035648,"available":94780506112,"internally_reserved":0,"allocated":9248768,"data_stored":6179454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1205270983},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":634,"seq":2723009265843,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":554652,"kb_used_data":8912,"kb_used_omap":4,"kb_used_meta":545723,"kb_avail":93190500,"statfs":{"total":95995035648,"available":95427072000,"internally_reserved":0,"allocated":9125888,"data_stored":6053022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4407,"internal_metadata":558821065},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575359,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1525264,"kb_used_data":8712,"kb_used_omap":1,"kb_used_meta":1516542,"kb_avail":92219888,"statfs":{"total":95995035648,"available":94433165312,"internally_reserved":0,"allocated":8921088,"data_stored":5851774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1552939461},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738882,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1000912,"kb_used_data":9032,"kb_used_omap":1,"kb_used_meta":991870,"kb_avail":92744240,"statfs":{"total":95995035648,"available":94970101760,"internally_reserved":0,"allocated":9248768,"data_stored":6179454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1015675333},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T14:50:01.272 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-08-19T14:50:01.273 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833 shell --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 -- ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-08-19T14:50:01.529 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/mon.a/config 2024-08-19T14:50:01.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:50:01 smithi045 ceph-mon[20143]: 1.9 scrub starts 2024-08-19T14:50:01.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:50:01 smithi045 ceph-mon[20143]: 1.9 scrub ok 2024-08-19T14:50:01.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:50:01 smithi045 ceph-mon[20143]: pgmap v2285: 31 pgs: 31 active+clean; 577 KiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:50:01.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:50:01 smithi045 ceph-mon[20143]: from='client.18615 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:50:01.866 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:50:01 smithi031 ceph-mon[16336]: 1.9 scrub starts 2024-08-19T14:50:01.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:50:01 smithi031 ceph-mon[16336]: 1.9 scrub ok 2024-08-19T14:50:01.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:50:01 smithi031 ceph-mon[16336]: pgmap v2285: 31 pgs: 31 active+clean; 577 KiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:50:01.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:50:01 smithi031 ceph-mon[16336]: from='client.18615 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:50:02.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:50:01 smithi032 ceph-mon[20004]: 1.9 scrub starts 2024-08-19T14:50:02.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:50:01 smithi032 ceph-mon[20004]: 1.9 scrub ok 2024-08-19T14:50:02.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:50:01 smithi032 ceph-mon[20004]: pgmap v2285: 31 pgs: 31 active+clean; 577 KiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:50:02.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:50:01 smithi032 ceph-mon[20004]: from='client.18615 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-19T14:50:02.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:50:02 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2731901093' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-08-19T14:50:02.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:50:02 smithi031 ceph-mon[16336]: from='client.? 172.21.15.31:0/2731901093' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-08-19T14:50:02.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:50:02 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:50:02.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:50:02 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:50:02.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:50:02 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:50:02.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:50:02 smithi031 ceph-mon[16336]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:50:02.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:50:02 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2731901093' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-08-19T14:50:02.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:50:02 smithi045 ceph-mon[20143]: from='client.? 172.21.15.31:0/2731901093' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-08-19T14:50:02.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:50:02 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:50:02.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:50:02 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:50:02.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:50:02 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:50:02.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:50:02 smithi045 ceph-mon[20143]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:50:03.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:50:02 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2731901093' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-08-19T14:50:03.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:50:02 smithi032 ceph-mon[20004]: from='client.? 172.21.15.31:0/2731901093' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-08-19T14:50:03.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:50:02 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-19T14:50:03.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:50:02 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-19T14:50:03.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:50:02 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-19T14:50:03.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:50:02 smithi032 ceph-mon[20004]: from='mgr.14150 172.21.15.31:0/3603471188' entity='mgr.y' 2024-08-19T14:50:03.034 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-08-19T14:50:03.045 INFO:tasks.cephadm:Teardown begin 2024-08-19T14:50:03.045 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-08-19T14:50:03.084 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-08-19T14:50:03.117 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-08-19T14:50:03.169 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-08-19T14:50:03.170 DEBUG:teuthology.orchestra.run.smithi031:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-08-19T14:50:03.197 DEBUG:teuthology.orchestra.run.smithi032:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-08-19T14:50:03.225 DEBUG:teuthology.orchestra.run.smithi045:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-08-19T14:50:03.251 INFO:tasks.cephadm:Stopping all daemons... 2024-08-19T14:50:03.251 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-08-19T14:50:03.252 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968@mon.a 2024-08-19T14:50:03.539 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:50:03 smithi031 systemd[1]: Stopping Ceph mon.a for ed7d1c6e-5e33-11ef-bccf-c7b262605968... 2024-08-19T14:50:03.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:50:03 smithi045 ceph-mon[20143]: purged_snaps scrub starts 2024-08-19T14:50:03.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:50:03 smithi045 ceph-mon[20143]: purged_snaps scrub ok 2024-08-19T14:50:03.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:50:03 smithi045 ceph-mon[20143]: pgmap v2286: 31 pgs: 31 active+clean; 577 KiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:50:03.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:50:03 smithi045 ceph-mon[20143]: osdmap e983: 12 total, 12 up, 12 in 2024-08-19T14:50:03.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:50:03 smithi031 ceph-mon[16336]: purged_snaps scrub starts 2024-08-19T14:50:03.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:50:03 smithi031 ceph-mon[16336]: purged_snaps scrub ok 2024-08-19T14:50:03.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:50:03 smithi031 ceph-mon[16336]: pgmap v2286: 31 pgs: 31 active+clean; 577 KiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:50:03.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:50:03 smithi031 ceph-mon[16336]: osdmap e983: 12 total, 12 up, 12 in 2024-08-19T14:50:03.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:50:03 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mon-a[16332]: 2024-08-19T14:50:03.574+0000 7f2ffa849640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-08-19T14:50:03.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:50:03 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mon-a[16332]: 2024-08-19T14:50:03.574+0000 7f2ffa849640 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-08-19T14:50:03.867 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:50:03 smithi031 podman[257064]: 2024-08-19 14:50:03.738251458 +0000 UTC m=+0.269560297 container died 268c97faa6a34ab0a8ee48d8e11506efe1fadf98fee0a8802e9dcda667f18086 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mon-a, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-de09dd0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-08-19T14:50:04.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:50:03 smithi032 ceph-mon[20004]: purged_snaps scrub starts 2024-08-19T14:50:04.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:50:03 smithi032 ceph-mon[20004]: purged_snaps scrub ok 2024-08-19T14:50:04.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:50:03 smithi032 ceph-mon[20004]: pgmap v2286: 31 pgs: 31 active+clean; 577 KiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-19T14:50:04.008 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:50:03 smithi032 ceph-mon[20004]: osdmap e983: 12 total, 12 up, 12 in 2024-08-19T14:50:04.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:50:04 smithi031 podman[257064]: 2024-08-19 14:50:04.217234093 +0000 UTC m=+0.748542888 container cleanup 268c97faa6a34ab0a8ee48d8e11506efe1fadf98fee0a8802e9dcda667f18086 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mon-a, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, RELEASE=squid-de09dd0, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731) 2024-08-19T14:50:04.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Aug 19 14:50:04 smithi031 bash[257064]: ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mon-a 2024-08-19T14:50:04.667 DEBUG:teuthology.orchestra.run.smithi031:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968@mon.a.service' 2024-08-19T14:50:04.705 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-19T14:50:04.705 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-08-19T14:50:04.705 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-08-19T14:50:04.705 DEBUG:teuthology.orchestra.run.smithi032:> sudo systemctl stop ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968@mon.b 2024-08-19T14:50:05.007 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:50:04 smithi032 systemd[1]: Stopping Ceph mon.b for ed7d1c6e-5e33-11ef-bccf-c7b262605968... 2024-08-19T14:50:05.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:50:05 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mon-b[20000]: 2024-08-19T14:50:05.086+0000 7ff37160f640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-08-19T14:50:05.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:50:05 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mon-b[20000]: 2024-08-19T14:50:05.086+0000 7ff37160f640 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-08-19T14:50:05.508 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:50:05 smithi032 podman[161936]: 2024-08-19 14:50:05.23984715 +0000 UTC m=+0.263104603 container died 066967b48013e975d49dd757239af1a4e46f30fff847c4c9394bbd44a78939ff (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mon-b, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-de09dd0, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=) 2024-08-19T14:50:05.997 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:50:05 smithi032 podman[161936]: 2024-08-19 14:50:05.737100015 +0000 UTC m=+0.760357510 container cleanup 066967b48013e975d49dd757239af1a4e46f30fff847c4c9394bbd44a78939ff (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mon-b, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=squid-de09dd0, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-08-19T14:50:05.997 INFO:journalctl@ceph.mon.b.smithi032.stdout:Aug 19 14:50:05 smithi032 bash[161936]: ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mon-b 2024-08-19T14:50:06.132 DEBUG:teuthology.orchestra.run.smithi032:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968@mon.b.service' 2024-08-19T14:50:06.181 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-19T14:50:06.182 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-08-19T14:50:06.182 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-08-19T14:50:06.182 DEBUG:teuthology.orchestra.run.smithi045:> sudo systemctl stop ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968@mon.c 2024-08-19T14:50:06.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:50:06 smithi045 systemd[1]: Stopping Ceph mon.c for ed7d1c6e-5e33-11ef-bccf-c7b262605968... 2024-08-19T14:50:06.796 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:50:06 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mon-c[20139]: 2024-08-19T14:50:06.542+0000 7f11f23c8640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-08-19T14:50:06.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:50:06 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mon-c[20139]: 2024-08-19T14:50:06.542+0000 7f11f23c8640 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-08-19T14:50:06.797 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:50:06 smithi045 podman[167413]: 2024-08-19 14:50:06.613918782 +0000 UTC m=+0.200291607 container died 168d75d038216a2938bbf98b7348243fef43172e45f916a50eea621f2234813e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mon-c, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-de09dd0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True) 2024-08-19T14:50:07.546 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:50:07 smithi045 podman[167413]: 2024-08-19 14:50:07.276548061 +0000 UTC m=+0.862920886 container cleanup 168d75d038216a2938bbf98b7348243fef43172e45f916a50eea621f2234813e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mon-c, RELEASE=squid-de09dd0, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux ) 2024-08-19T14:50:07.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Aug 19 14:50:07 smithi045 bash[167413]: ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mon-c 2024-08-19T14:50:07.716 DEBUG:teuthology.orchestra.run.smithi045:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968@mon.c.service' 2024-08-19T14:50:07.753 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-19T14:50:07.754 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-08-19T14:50:07.754 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-08-19T14:50:07.754 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968@mgr.y 2024-08-19T14:50:08.048 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:50:07 smithi031 systemd[1]: Stopping Ceph mgr.y for ed7d1c6e-5e33-11ef-bccf-c7b262605968... 2024-08-19T14:50:08.306 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:50:08 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: 2024-08-19T14:50:08.046+0000 7ff4ea559640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-19T14:50:08.306 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:50:08 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y[16562]: 2024-08-19T14:50:08.046+0000 7ff4ea559640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-08-19T14:50:08.307 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:50:08 smithi031 podman[257185]: 2024-08-19 14:50:08.078309393 +0000 UTC m=+0.127657465 container died 5f1a20479194529c4b79d74a77cd8c89219f09873bb8b1c20a302a9886e97f60 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y, RELEASE=squid-de09dd0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux ) 2024-08-19T14:50:08.616 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:50:08 smithi031 podman[257185]: 2024-08-19 14:50:08.489234918 +0000 UTC m=+0.538582989 container cleanup 5f1a20479194529c4b79d74a77cd8c89219f09873bb8b1c20a302a9886e97f60 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:de09dd018de4baddfa6ca4d43bc8fb784cf75833, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=squid-de09dd0, ceph=True, GIT_CLEAN=True, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-08-19T14:50:08.616 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Aug 19 14:50:08 smithi031 bash[257185]: ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-y 2024-08-19T14:50:08.864 DEBUG:teuthology.orchestra.run.smithi031:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968@mgr.y.service' 2024-08-19T14:50:08.899 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-19T14:50:08.899 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-08-19T14:50:08.899 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-08-19T14:50:08.899 DEBUG:teuthology.orchestra.run.smithi032:> sudo systemctl stop ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968@mgr.x 2024-08-19T14:50:09.212 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Aug 19 14:50:08 smithi032 systemd[1]: Stopping Ceph mgr.x for ed7d1c6e-5e33-11ef-bccf-c7b262605968... 2024-08-19T14:50:09.507 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Aug 19 14:50:09 smithi032 podman[162060]: 2024-08-19 14:50:09.212697367 +0000 UTC m=+0.111669533 container died 70be1fbca0718275dc69c5dde99d1c2f2eaa172ee9483fd4808a7a8efc6a3569 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-x, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=squid-de09dd0) 2024-08-19T14:50:09.986 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Aug 19 14:50:09 smithi032 podman[162060]: 2024-08-19 14:50:09.600418465 +0000 UTC m=+0.499390584 container cleanup 70be1fbca0718275dc69c5dde99d1c2f2eaa172ee9483fd4808a7a8efc6a3569 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-x, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, ceph=True, org.label-schema.vendor=CentOS, RELEASE=squid-de09dd0, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-08-19T14:50:09.986 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Aug 19 14:50:09 smithi032 bash[162060]: ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-mgr-x 2024-08-19T14:50:10.201 DEBUG:teuthology.orchestra.run.smithi032:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968@mgr.x.service' 2024-08-19T14:50:10.235 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-19T14:50:10.235 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-08-19T14:50:10.236 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-08-19T14:50:10.236 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968@osd.0 2024-08-19T14:50:10.544 INFO:journalctl@ceph.osd.0.smithi031.stdout:Aug 19 14:50:10 smithi031 systemd[1]: Stopping Ceph osd.0 for ed7d1c6e-5e33-11ef-bccf-c7b262605968... 2024-08-19T14:50:10.866 INFO:journalctl@ceph.osd.0.smithi031.stdout:Aug 19 14:50:10 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-0[30082]: 2024-08-19T14:50:10.542+0000 7fd56a276640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-19T14:50:10.867 INFO:journalctl@ceph.osd.0.smithi031.stdout:Aug 19 14:50:10 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-0[30082]: 2024-08-19T14:50:10.542+0000 7fd56a276640 -1 osd.0 983 *** Got signal Terminated *** 2024-08-19T14:50:10.867 INFO:journalctl@ceph.osd.0.smithi031.stdout:Aug 19 14:50:10 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-0[30082]: 2024-08-19T14:50:10.542+0000 7fd56a276640 -1 osd.0 983 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-08-19T14:50:16.452 INFO:journalctl@ceph.osd.0.smithi031.stdout:Aug 19 14:50:16 smithi031 podman[257306]: 2024-08-19 14:50:16.182812268 +0000 UTC m=+5.750198256 container died ddf3252709965f5957e7e294b9cb0207aada8b2974d654d688362dd9a1a5cc55 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=squid-de09dd0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-08-19T14:50:16.803 INFO:journalctl@ceph.osd.0.smithi031.stdout:Aug 19 14:50:16 smithi031 podman[257306]: 2024-08-19 14:50:16.618957012 +0000 UTC m=+6.186343041 container cleanup ddf3252709965f5957e7e294b9cb0207aada8b2974d654d688362dd9a1a5cc55 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=squid-de09dd0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD) 2024-08-19T14:50:16.804 INFO:journalctl@ceph.osd.0.smithi031.stdout:Aug 19 14:50:16 smithi031 bash[257306]: ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-0 2024-08-19T14:50:17.116 INFO:journalctl@ceph.osd.0.smithi031.stdout:Aug 19 14:50:16 smithi031 podman[257323]: 2024-08-19 14:50:16.803617236 +0000 UTC m=+0.624090264 container remove ddf3252709965f5957e7e294b9cb0207aada8b2974d654d688362dd9a1a5cc55 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-de09dd0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731) 2024-08-19T14:50:17.484 INFO:journalctl@ceph.osd.0.smithi031.stdout:Aug 19 14:50:17 smithi031 podman[257409]: 2024-08-19 14:50:17.128710911 +0000 UTC m=+0.017085913 image pull ea7a5189f30bad3aca9524788e2965d2368d7ce786df09d4edcd8c3edf9cefba quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9 2024-08-19T14:50:17.484 INFO:journalctl@ceph.osd.0.smithi031.stdout:Aug 19 14:50:17 smithi031 podman[257409]: 2024-08-19 14:50:17.362876035 +0000 UTC m=+0.251251027 container create 6e453dfd1746ccf00a92db837efad35dd8d52648eb123037e616c3358134b44b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-0-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=squid-de09dd0, org.label-schema.build-date=20240731, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-08-19T14:50:17.896 INFO:journalctl@ceph.osd.0.smithi031.stdout:Aug 19 14:50:17 smithi031 podman[257409]: 2024-08-19 14:50:17.637986172 +0000 UTC m=+0.526361164 container init 6e453dfd1746ccf00a92db837efad35dd8d52648eb123037e616c3358134b44b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-0-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240731, ceph=True, org.label-schema.license=GPLv2, RELEASE=squid-de09dd0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux ) 2024-08-19T14:50:17.896 INFO:journalctl@ceph.osd.0.smithi031.stdout:Aug 19 14:50:17 smithi031 podman[257409]: 2024-08-19 14:50:17.641186839 +0000 UTC m=+0.529561834 container start 6e453dfd1746ccf00a92db837efad35dd8d52648eb123037e616c3358134b44b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-0-deactivate, GIT_CLEAN=True, RELEASE=squid-de09dd0, org.label-schema.build-date=20240731, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0) 2024-08-19T14:50:17.896 INFO:journalctl@ceph.osd.0.smithi031.stdout:Aug 19 14:50:17 smithi031 podman[257409]: 2024-08-19 14:50:17.740400645 +0000 UTC m=+0.628775651 container attach 6e453dfd1746ccf00a92db837efad35dd8d52648eb123037e616c3358134b44b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-0-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-de09dd0, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, io.buildah.version=1.36.0, ceph=True) 2024-08-19T14:50:18.149 INFO:journalctl@ceph.osd.0.smithi031.stdout:Aug 19 14:50:17 smithi031 podman[257409]: 2024-08-19 14:50:17.896208811 +0000 UTC m=+0.784583804 container died 6e453dfd1746ccf00a92db837efad35dd8d52648eb123037e616c3358134b44b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-0-deactivate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=squid-de09dd0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, io.buildah.version=1.36.0) 2024-08-19T14:50:18.652 DEBUG:teuthology.orchestra.run.smithi031:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968@osd.0.service' 2024-08-19T14:50:18.695 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-19T14:50:18.695 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-08-19T14:50:18.695 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-08-19T14:50:18.695 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968@osd.1 2024-08-19T14:50:19.032 INFO:journalctl@ceph.osd.1.smithi031.stdout:Aug 19 14:50:18 smithi031 systemd[1]: Stopping Ceph osd.1 for ed7d1c6e-5e33-11ef-bccf-c7b262605968... 2024-08-19T14:50:19.366 INFO:journalctl@ceph.osd.1.smithi031.stdout:Aug 19 14:50:19 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-1[35635]: 2024-08-19T14:50:19.031+0000 7fab7793d640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-19T14:50:19.366 INFO:journalctl@ceph.osd.1.smithi031.stdout:Aug 19 14:50:19 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-1[35635]: 2024-08-19T14:50:19.031+0000 7fab7793d640 -1 osd.1 983 *** Got signal Terminated *** 2024-08-19T14:50:19.366 INFO:journalctl@ceph.osd.1.smithi031.stdout:Aug 19 14:50:19 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-1[35635]: 2024-08-19T14:50:19.031+0000 7fab7793d640 -1 osd.1 983 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-08-19T14:50:24.866 INFO:journalctl@ceph.osd.1.smithi031.stdout:Aug 19 14:50:24 smithi031 podman[257513]: 2024-08-19 14:50:24.59068857 +0000 UTC m=+5.656067528 container died 909fd3cb3a2bd98ccb3bda01a58a12f0ada4a92a9cf26d9f9f0afcfda18433e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-1, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, RELEASE=squid-de09dd0, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-08-19T14:50:25.235 INFO:journalctl@ceph.osd.1.smithi031.stdout:Aug 19 14:50:25 smithi031 podman[257513]: 2024-08-19 14:50:25.131284342 +0000 UTC m=+6.196663297 container cleanup 909fd3cb3a2bd98ccb3bda01a58a12f0ada4a92a9cf26d9f9f0afcfda18433e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-1, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=squid-de09dd0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.36.0) 2024-08-19T14:50:25.235 INFO:journalctl@ceph.osd.1.smithi031.stdout:Aug 19 14:50:25 smithi031 bash[257513]: ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-1 2024-08-19T14:50:25.616 INFO:journalctl@ceph.osd.1.smithi031.stdout:Aug 19 14:50:25 smithi031 podman[257528]: 2024-08-19 14:50:25.32481349 +0000 UTC m=+0.729728771 container remove 909fd3cb3a2bd98ccb3bda01a58a12f0ada4a92a9cf26d9f9f0afcfda18433e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-1, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , RELEASE=squid-de09dd0, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-08-19T14:50:26.038 INFO:journalctl@ceph.osd.1.smithi031.stdout:Aug 19 14:50:25 smithi031 podman[257627]: 2024-08-19 14:50:25.682088473 +0000 UTC m=+0.019692898 image pull ea7a5189f30bad3aca9524788e2965d2368d7ce786df09d4edcd8c3edf9cefba quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9 2024-08-19T14:50:26.038 INFO:journalctl@ceph.osd.1.smithi031.stdout:Aug 19 14:50:25 smithi031 podman[257627]: 2024-08-19 14:50:25.792244356 +0000 UTC m=+0.129848751 container create 4cf363bdc942f771d3f5393216f4d429fe7ee31bdb23c93a5e819ff1f8bdb58e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-1-deactivate, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=squid-de09dd0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-08-19T14:50:26.038 INFO:journalctl@ceph.osd.1.smithi031.stdout:Aug 19 14:50:26 smithi031 podman[257627]: 2024-08-19 14:50:26.011319027 +0000 UTC m=+0.348923430 container init 4cf363bdc942f771d3f5393216f4d429fe7ee31bdb23c93a5e819ff1f8bdb58e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-1-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, RELEASE=squid-de09dd0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, org.label-schema.build-date=20240731) 2024-08-19T14:50:26.038 INFO:journalctl@ceph.osd.1.smithi031.stdout:Aug 19 14:50:26 smithi031 podman[257627]: 2024-08-19 14:50:26.014844308 +0000 UTC m=+0.352448708 container start 4cf363bdc942f771d3f5393216f4d429fe7ee31bdb23c93a5e819ff1f8bdb58e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, RELEASE=squid-de09dd0, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-08-19T14:50:26.366 INFO:journalctl@ceph.osd.1.smithi031.stdout:Aug 19 14:50:26 smithi031 podman[257627]: 2024-08-19 14:50:26.120063442 +0000 UTC m=+0.457667831 container attach 4cf363bdc942f771d3f5393216f4d429fe7ee31bdb23c93a5e819ff1f8bdb58e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-1-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=squid-de09dd0, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image) 2024-08-19T14:50:26.366 INFO:journalctl@ceph.osd.1.smithi031.stdout:Aug 19 14:50:26 smithi031 podman[257627]: 2024-08-19 14:50:26.223879464 +0000 UTC m=+0.561483853 container died 4cf363bdc942f771d3f5393216f4d429fe7ee31bdb23c93a5e819ff1f8bdb58e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-1-deactivate, GIT_BRANCH=HEAD, RELEASE=squid-de09dd0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True) 2024-08-19T14:50:26.867 DEBUG:teuthology.orchestra.run.smithi031:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968@osd.1.service' 2024-08-19T14:50:26.910 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-19T14:50:26.910 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-08-19T14:50:26.910 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-08-19T14:50:26.910 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968@osd.2 2024-08-19T14:50:27.117 INFO:journalctl@ceph.osd.2.smithi031.stdout:Aug 19 14:50:26 smithi031 systemd[1]: Stopping Ceph osd.2 for ed7d1c6e-5e33-11ef-bccf-c7b262605968... 2024-08-19T14:50:27.616 INFO:journalctl@ceph.osd.2.smithi031.stdout:Aug 19 14:50:27 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-2[181347]: 2024-08-19T14:50:27.252+0000 7fcb7f43a640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-19T14:50:27.616 INFO:journalctl@ceph.osd.2.smithi031.stdout:Aug 19 14:50:27 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-2[181347]: 2024-08-19T14:50:27.252+0000 7fcb7f43a640 -1 osd.2 983 *** Got signal Terminated *** 2024-08-19T14:50:27.616 INFO:journalctl@ceph.osd.2.smithi031.stdout:Aug 19 14:50:27 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-2[181347]: 2024-08-19T14:50:27.252+0000 7fcb7f43a640 -1 osd.2 983 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-08-19T14:50:33.060 INFO:journalctl@ceph.osd.2.smithi031.stdout:Aug 19 14:50:32 smithi031 podman[257731]: 2024-08-19 14:50:32.797040969 +0000 UTC m=+5.644919548 container died 6126bfc5c5138d398119df06d2108661bc6f5f7fcd67ca072724adc90759d7e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=squid-de09dd0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240731, ceph=True, io.buildah.version=1.36.0) 2024-08-19T14:50:33.366 INFO:journalctl@ceph.osd.2.smithi031.stdout:Aug 19 14:50:33 smithi031 podman[257731]: 2024-08-19 14:50:33.226879543 +0000 UTC m=+6.074758149 container cleanup 6126bfc5c5138d398119df06d2108661bc6f5f7fcd67ca072724adc90759d7e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-2, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, RELEASE=squid-de09dd0) 2024-08-19T14:50:33.367 INFO:journalctl@ceph.osd.2.smithi031.stdout:Aug 19 14:50:33 smithi031 bash[257731]: ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-2 2024-08-19T14:50:33.866 INFO:journalctl@ceph.osd.2.smithi031.stdout:Aug 19 14:50:33 smithi031 podman[257747]: 2024-08-19 14:50:33.461793749 +0000 UTC m=+0.660809290 container remove 6126bfc5c5138d398119df06d2108661bc6f5f7fcd67ca072724adc90759d7e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-2, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, RELEASE=squid-de09dd0, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731) 2024-08-19T14:50:34.215 INFO:journalctl@ceph.osd.2.smithi031.stdout:Aug 19 14:50:33 smithi031 podman[257834]: 2024-08-19 14:50:33.807118824 +0000 UTC m=+0.017100742 image pull ea7a5189f30bad3aca9524788e2965d2368d7ce786df09d4edcd8c3edf9cefba quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9 2024-08-19T14:50:34.215 INFO:journalctl@ceph.osd.2.smithi031.stdout:Aug 19 14:50:33 smithi031 podman[257834]: 2024-08-19 14:50:33.945971179 +0000 UTC m=+0.155953093 container create d63c9708fd789ba7ef3678aac6989f3e4d605cdf95ed434e8addb0a19fcd0043 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-2-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, RELEASE=squid-de09dd0, org.label-schema.license=GPLv2) 2024-08-19T14:50:34.215 INFO:journalctl@ceph.osd.2.smithi031.stdout:Aug 19 14:50:34 smithi031 podman[257834]: 2024-08-19 14:50:34.140610002 +0000 UTC m=+0.350591919 container init d63c9708fd789ba7ef3678aac6989f3e4d605cdf95ed434e8addb0a19fcd0043 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-2-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-de09dd0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, io.buildah.version=1.36.0, ceph=True) 2024-08-19T14:50:34.215 INFO:journalctl@ceph.osd.2.smithi031.stdout:Aug 19 14:50:34 smithi031 podman[257834]: 2024-08-19 14:50:34.144277839 +0000 UTC m=+0.354259755 container start d63c9708fd789ba7ef3678aac6989f3e4d605cdf95ed434e8addb0a19fcd0043 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-2-deactivate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=squid-de09dd0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, ceph=True, GIT_CLEAN=True) 2024-08-19T14:50:34.215 INFO:journalctl@ceph.osd.2.smithi031.stdout:Aug 19 14:50:34 smithi031 podman[257834]: 2024-08-19 14:50:34.215137997 +0000 UTC m=+0.425119935 container attach d63c9708fd789ba7ef3678aac6989f3e4d605cdf95ed434e8addb0a19fcd0043 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-2-deactivate, ceph=True, RELEASE=squid-de09dd0, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-08-19T14:50:34.616 INFO:journalctl@ceph.osd.2.smithi031.stdout:Aug 19 14:50:34 smithi031 podman[257834]: 2024-08-19 14:50:34.332414282 +0000 UTC m=+0.542396200 container died d63c9708fd789ba7ef3678aac6989f3e4d605cdf95ed434e8addb0a19fcd0043 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-2-deactivate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-de09dd0, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-08-19T14:50:34.879 INFO:journalctl@ceph.osd.2.smithi031.stdout:Aug 19 14:50:34 smithi031 podman[257857]: 2024-08-19 14:50:34.860187601 +0000 UTC m=+0.517929435 container remove d63c9708fd789ba7ef3678aac6989f3e4d605cdf95ed434e8addb0a19fcd0043 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-2-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-de09dd0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, io.buildah.version=1.36.0, GIT_BRANCH=HEAD) 2024-08-19T14:50:34.884 DEBUG:teuthology.orchestra.run.smithi031:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968@osd.2.service' 2024-08-19T14:50:34.922 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-19T14:50:34.922 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-08-19T14:50:34.922 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-08-19T14:50:34.922 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968@osd.3 2024-08-19T14:50:35.247 INFO:journalctl@ceph.osd.3.smithi031.stdout:Aug 19 14:50:35 smithi031 systemd[1]: Stopping Ceph osd.3 for ed7d1c6e-5e33-11ef-bccf-c7b262605968... 2024-08-19T14:50:35.500 INFO:journalctl@ceph.osd.3.smithi031.stdout:Aug 19 14:50:35 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-3[47445]: 2024-08-19T14:50:35.246+0000 7fd36ec47640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-19T14:50:35.500 INFO:journalctl@ceph.osd.3.smithi031.stdout:Aug 19 14:50:35 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-3[47445]: 2024-08-19T14:50:35.246+0000 7fd36ec47640 -1 osd.3 983 *** Got signal Terminated *** 2024-08-19T14:50:35.500 INFO:journalctl@ceph.osd.3.smithi031.stdout:Aug 19 14:50:35 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-3[47445]: 2024-08-19T14:50:35.246+0000 7fd36ec47640 -1 osd.3 983 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-08-19T14:50:36.547 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:50:36 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:50:36.275+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.724961+0000 front 2024-08-19T14:50:12.724727+0000 (oldest deadline 2024-08-19T14:50:36.224802+0000) 2024-08-19T14:50:37.257 INFO:journalctl@ceph.osd.6.smithi032.stdout:Aug 19 14:50:36 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-6[35351]: 2024-08-19T14:50:36.790+0000 7f8d3de10640 -1 osd.6 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:13.516154+0000 front 2024-08-19T14:50:13.516301+0000 (oldest deadline 2024-08-19T14:50:36.416116+0000) 2024-08-19T14:50:37.264 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:50:36 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:50:36.937+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.842003+0000 front 2024-08-19T14:50:12.842211+0000 (oldest deadline 2024-08-19T14:50:36.242616+0000) 2024-08-19T14:50:37.546 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:50:37 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:50:37.263+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.724961+0000 front 2024-08-19T14:50:12.724727+0000 (oldest deadline 2024-08-19T14:50:36.224802+0000) 2024-08-19T14:50:38.007 INFO:journalctl@ceph.osd.6.smithi032.stdout:Aug 19 14:50:37 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-6[35351]: 2024-08-19T14:50:37.771+0000 7f8d3de10640 -1 osd.6 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:13.516154+0000 front 2024-08-19T14:50:13.516301+0000 (oldest deadline 2024-08-19T14:50:36.416116+0000) 2024-08-19T14:50:38.249 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:50:37 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:50:37.921+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.842003+0000 front 2024-08-19T14:50:12.842211+0000 (oldest deadline 2024-08-19T14:50:36.242616+0000) 2024-08-19T14:50:38.366 INFO:journalctl@ceph.osd.3.smithi031.stdout:Aug 19 14:50:37 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-3[47445]: 2024-08-19T14:50:37.943+0000 7fd367255640 -1 osd.3 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.792136+0000 front 2024-08-19T14:50:12.792345+0000 (oldest deadline 2024-08-19T14:50:37.492267+0000) 2024-08-19T14:50:38.546 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:50:38 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:50:38.248+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.724961+0000 front 2024-08-19T14:50:12.724727+0000 (oldest deadline 2024-08-19T14:50:36.224802+0000) 2024-08-19T14:50:39.046 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:50:38 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:50:38.767+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:13.297511+0000 front 2024-08-19T14:50:13.297834+0000 (oldest deadline 2024-08-19T14:50:37.997679+0000) 2024-08-19T14:50:39.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:50:38 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:50:38.887+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.842003+0000 front 2024-08-19T14:50:12.842211+0000 (oldest deadline 2024-08-19T14:50:36.242616+0000) 2024-08-19T14:50:39.257 INFO:journalctl@ceph.osd.6.smithi032.stdout:Aug 19 14:50:38 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-6[35351]: 2024-08-19T14:50:38.776+0000 7f8d3de10640 -1 osd.6 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:13.516154+0000 front 2024-08-19T14:50:13.516301+0000 (oldest deadline 2024-08-19T14:50:36.416116+0000) 2024-08-19T14:50:39.366 INFO:journalctl@ceph.osd.3.smithi031.stdout:Aug 19 14:50:38 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-3[47445]: 2024-08-19T14:50:38.990+0000 7fd367255640 -1 osd.3 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.792136+0000 front 2024-08-19T14:50:12.792345+0000 (oldest deadline 2024-08-19T14:50:37.492267+0000) 2024-08-19T14:50:39.546 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:50:39 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:50:39.200+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.724961+0000 front 2024-08-19T14:50:12.724727+0000 (oldest deadline 2024-08-19T14:50:36.224802+0000) 2024-08-19T14:50:39.547 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:50:39 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:50:39.148+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:14.239016+0000 front 2024-08-19T14:50:14.238817+0000 (oldest deadline 2024-08-19T14:50:38.339035+0000) 2024-08-19T14:50:40.046 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:50:39 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:50:39.789+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:13.297511+0000 front 2024-08-19T14:50:13.297834+0000 (oldest deadline 2024-08-19T14:50:37.997679+0000) 2024-08-19T14:50:40.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:50:39 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:50:39.883+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.842003+0000 front 2024-08-19T14:50:12.842211+0000 (oldest deadline 2024-08-19T14:50:36.242616+0000) 2024-08-19T14:50:40.257 INFO:journalctl@ceph.osd.6.smithi032.stdout:Aug 19 14:50:39 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-6[35351]: 2024-08-19T14:50:39.796+0000 7f8d3de10640 -1 osd.6 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:13.516154+0000 front 2024-08-19T14:50:13.516301+0000 (oldest deadline 2024-08-19T14:50:36.416116+0000) 2024-08-19T14:50:40.366 INFO:journalctl@ceph.osd.3.smithi031.stdout:Aug 19 14:50:39 smithi031 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-3[47445]: 2024-08-19T14:50:39.977+0000 7fd367255640 -1 osd.3 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.792136+0000 front 2024-08-19T14:50:12.792345+0000 (oldest deadline 2024-08-19T14:50:37.492267+0000) 2024-08-19T14:50:40.546 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:50:40 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:50:40.128+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:14.239016+0000 front 2024-08-19T14:50:14.238817+0000 (oldest deadline 2024-08-19T14:50:38.339035+0000) 2024-08-19T14:50:40.547 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:50:40 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:50:40.226+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.724961+0000 front 2024-08-19T14:50:12.724727+0000 (oldest deadline 2024-08-19T14:50:36.224802+0000) 2024-08-19T14:50:40.758 INFO:journalctl@ceph.osd.4.smithi032.stdout:Aug 19 14:50:40 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-4[24402]: 2024-08-19T14:50:40.450+0000 7fad382a0640 -1 osd.4 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:15.418820+0000 front 2024-08-19T14:50:15.418895+0000 (oldest deadline 2024-08-19T14:50:39.518888+0000) 2024-08-19T14:50:41.047 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:50:40 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:50:40.765+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:13.297511+0000 front 2024-08-19T14:50:13.297834+0000 (oldest deadline 2024-08-19T14:50:37.997679+0000) 2024-08-19T14:50:41.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:50:40 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:50:40.842+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.842003+0000 front 2024-08-19T14:50:12.842211+0000 (oldest deadline 2024-08-19T14:50:36.242616+0000) 2024-08-19T14:50:41.080 INFO:journalctl@ceph.osd.3.smithi031.stdout:Aug 19 14:50:40 smithi031 podman[257939]: 2024-08-19 14:50:40.78624008 +0000 UTC m=+5.635029401 container died 9c2b59e05e3cf1790fe9ef3be05e49a8576021e84909b3370d1c63cf58ddb23a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-3, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, RELEASE=squid-de09dd0, org.label-schema.build-date=20240731, io.buildah.version=1.36.0) 2024-08-19T14:50:41.257 INFO:journalctl@ceph.osd.6.smithi032.stdout:Aug 19 14:50:40 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-6[35351]: 2024-08-19T14:50:40.806+0000 7f8d3de10640 -1 osd.6 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:13.516154+0000 front 2024-08-19T14:50:13.516301+0000 (oldest deadline 2024-08-19T14:50:36.416116+0000) 2024-08-19T14:50:41.366 INFO:journalctl@ceph.osd.3.smithi031.stdout:Aug 19 14:50:41 smithi031 podman[257939]: 2024-08-19 14:50:41.263066518 +0000 UTC m=+6.111855885 container cleanup 9c2b59e05e3cf1790fe9ef3be05e49a8576021e84909b3370d1c63cf58ddb23a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-3, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=squid-de09dd0, org.label-schema.vendor=CentOS) 2024-08-19T14:50:41.366 INFO:journalctl@ceph.osd.3.smithi031.stdout:Aug 19 14:50:41 smithi031 bash[257939]: ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-3 2024-08-19T14:50:41.546 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:50:41 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:50:41.115+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:14.239016+0000 front 2024-08-19T14:50:14.238817+0000 (oldest deadline 2024-08-19T14:50:38.339035+0000) 2024-08-19T14:50:41.547 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:50:41 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:50:41.260+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.724961+0000 front 2024-08-19T14:50:12.724727+0000 (oldest deadline 2024-08-19T14:50:36.224802+0000) 2024-08-19T14:50:41.757 INFO:journalctl@ceph.osd.4.smithi032.stdout:Aug 19 14:50:41 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-4[24402]: 2024-08-19T14:50:41.447+0000 7fad382a0640 -1 osd.4 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:15.418820+0000 front 2024-08-19T14:50:15.418895+0000 (oldest deadline 2024-08-19T14:50:39.518888+0000) 2024-08-19T14:50:41.866 INFO:journalctl@ceph.osd.3.smithi031.stdout:Aug 19 14:50:41 smithi031 podman[257956]: 2024-08-19 14:50:41.53973367 +0000 UTC m=+0.741972065 container remove 9c2b59e05e3cf1790fe9ef3be05e49a8576021e84909b3370d1c63cf58ddb23a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-3, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, RELEASE=squid-de09dd0, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=) 2024-08-19T14:50:42.070 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:50:41 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:50:41.802+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:13.297511+0000 front 2024-08-19T14:50:13.297834+0000 (oldest deadline 2024-08-19T14:50:37.997679+0000) 2024-08-19T14:50:42.070 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:50:41 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:50:41.884+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.842003+0000 front 2024-08-19T14:50:12.842211+0000 (oldest deadline 2024-08-19T14:50:36.242616+0000) 2024-08-19T14:50:42.143 INFO:journalctl@ceph.osd.3.smithi031.stdout:Aug 19 14:50:41 smithi031 podman[258045]: 2024-08-19 14:50:41.890539225 +0000 UTC m=+0.112731978 container create a24ed36fd5f76fa48c2b60427c115809c2f57ad87366a0a019b53d8b5a1fecab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-3-deactivate, GIT_CLEAN=True, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, RELEASE=squid-de09dd0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-08-19T14:50:42.143 INFO:journalctl@ceph.osd.3.smithi031.stdout:Aug 19 14:50:41 smithi031 podman[258045]: 2024-08-19 14:50:41.79471469 +0000 UTC m=+0.016907415 image pull ea7a5189f30bad3aca9524788e2965d2368d7ce786df09d4edcd8c3edf9cefba quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9 2024-08-19T14:50:42.143 INFO:journalctl@ceph.osd.3.smithi031.stdout:Aug 19 14:50:42 smithi031 podman[258045]: 2024-08-19 14:50:42.076069439 +0000 UTC m=+0.298262163 container init a24ed36fd5f76fa48c2b60427c115809c2f57ad87366a0a019b53d8b5a1fecab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-3-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, RELEASE=squid-de09dd0, io.buildah.version=1.36.0, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-08-19T14:50:42.143 INFO:journalctl@ceph.osd.3.smithi031.stdout:Aug 19 14:50:42 smithi031 podman[258045]: 2024-08-19 14:50:42.079428579 +0000 UTC m=+0.301621291 container start a24ed36fd5f76fa48c2b60427c115809c2f57ad87366a0a019b53d8b5a1fecab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-3-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=squid-de09dd0, io.buildah.version=1.36.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-08-19T14:50:42.143 INFO:journalctl@ceph.osd.3.smithi031.stdout:Aug 19 14:50:42 smithi031 podman[258045]: 2024-08-19 14:50:42.142747981 +0000 UTC m=+0.364940701 container attach a24ed36fd5f76fa48c2b60427c115809c2f57ad87366a0a019b53d8b5a1fecab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-3-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=squid-de09dd0, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=) 2024-08-19T14:50:42.257 INFO:journalctl@ceph.osd.6.smithi032.stdout:Aug 19 14:50:41 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-6[35351]: 2024-08-19T14:50:41.830+0000 7f8d3de10640 -1 osd.6 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:13.516154+0000 front 2024-08-19T14:50:13.516301+0000 (oldest deadline 2024-08-19T14:50:36.416116+0000) 2024-08-19T14:50:42.546 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:50:42 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:50:42.273+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.724961+0000 front 2024-08-19T14:50:12.724727+0000 (oldest deadline 2024-08-19T14:50:36.224802+0000) 2024-08-19T14:50:42.547 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:50:42 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:50:42.069+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:14.239016+0000 front 2024-08-19T14:50:14.238817+0000 (oldest deadline 2024-08-19T14:50:38.339035+0000) 2024-08-19T14:50:42.578 INFO:journalctl@ceph.osd.3.smithi031.stdout:Aug 19 14:50:42 smithi031 podman[258045]: 2024-08-19 14:50:42.277582323 +0000 UTC m=+0.499775049 container died a24ed36fd5f76fa48c2b60427c115809c2f57ad87366a0a019b53d8b5a1fecab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-3-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, RELEASE=squid-de09dd0, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2) 2024-08-19T14:50:42.757 INFO:journalctl@ceph.osd.4.smithi032.stdout:Aug 19 14:50:42 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-4[24402]: 2024-08-19T14:50:42.434+0000 7fad382a0640 -1 osd.4 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:15.418820+0000 front 2024-08-19T14:50:15.418895+0000 (oldest deadline 2024-08-19T14:50:39.518888+0000) 2024-08-19T14:50:42.758 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:50:42 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7[41448]: 2024-08-19T14:50:42.300+0000 7f1321f92640 -1 osd.7 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.778781+0000 front 2024-08-19T14:50:12.778916+0000 (oldest deadline 2024-08-19T14:50:41.579607+0000) 2024-08-19T14:50:42.889 DEBUG:teuthology.orchestra.run.smithi031:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968@osd.3.service' 2024-08-19T14:50:42.932 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-19T14:50:42.932 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-08-19T14:50:42.932 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-08-19T14:50:42.932 DEBUG:teuthology.orchestra.run.smithi032:> sudo systemctl stop ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968@osd.4 2024-08-19T14:50:43.046 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:50:42 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:50:42.765+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:13.297511+0000 front 2024-08-19T14:50:13.297834+0000 (oldest deadline 2024-08-19T14:50:37.997679+0000) 2024-08-19T14:50:43.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:50:42 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:50:42.902+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.842003+0000 front 2024-08-19T14:50:12.842211+0000 (oldest deadline 2024-08-19T14:50:36.242616+0000) 2024-08-19T14:50:43.246 INFO:journalctl@ceph.osd.6.smithi032.stdout:Aug 19 14:50:42 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-6[35351]: 2024-08-19T14:50:42.847+0000 7f8d3de10640 -1 osd.6 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:13.516154+0000 front 2024-08-19T14:50:13.516301+0000 (oldest deadline 2024-08-19T14:50:36.416116+0000) 2024-08-19T14:50:43.246 INFO:journalctl@ceph.osd.4.smithi032.stdout:Aug 19 14:50:42 smithi032 systemd[1]: Stopping Ceph osd.4 for ed7d1c6e-5e33-11ef-bccf-c7b262605968... 2024-08-19T14:50:43.246 INFO:journalctl@ceph.osd.4.smithi032.stdout:Aug 19 14:50:43 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-4[24402]: 2024-08-19T14:50:43.244+0000 7fad3bc8a640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-19T14:50:43.507 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:50:43 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7[41448]: 2024-08-19T14:50:43.272+0000 7f1321f92640 -1 osd.7 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.778781+0000 front 2024-08-19T14:50:12.778916+0000 (oldest deadline 2024-08-19T14:50:41.579607+0000) 2024-08-19T14:50:43.508 INFO:journalctl@ceph.osd.4.smithi032.stdout:Aug 19 14:50:43 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-4[24402]: 2024-08-19T14:50:43.244+0000 7fad3bc8a640 -1 osd.4 983 *** Got signal Terminated *** 2024-08-19T14:50:43.508 INFO:journalctl@ceph.osd.4.smithi032.stdout:Aug 19 14:50:43 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-4[24402]: 2024-08-19T14:50:43.244+0000 7fad3bc8a640 -1 osd.4 983 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-08-19T14:50:43.508 INFO:journalctl@ceph.osd.4.smithi032.stdout:Aug 19 14:50:43 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-4[24402]: 2024-08-19T14:50:43.481+0000 7fad382a0640 -1 osd.4 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:15.418820+0000 front 2024-08-19T14:50:15.418895+0000 (oldest deadline 2024-08-19T14:50:39.518888+0000) 2024-08-19T14:50:43.546 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:50:43 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:50:43.108+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:14.239016+0000 front 2024-08-19T14:50:14.238817+0000 (oldest deadline 2024-08-19T14:50:38.339035+0000) 2024-08-19T14:50:43.546 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:50:43 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:50:43.310+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.724961+0000 front 2024-08-19T14:50:12.724727+0000 (oldest deadline 2024-08-19T14:50:36.224802+0000) 2024-08-19T14:50:44.046 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:50:43 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:50:43.780+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:13.297511+0000 front 2024-08-19T14:50:13.297834+0000 (oldest deadline 2024-08-19T14:50:37.997679+0000) 2024-08-19T14:50:44.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:50:43 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:50:43.892+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.842003+0000 front 2024-08-19T14:50:12.842211+0000 (oldest deadline 2024-08-19T14:50:36.242616+0000) 2024-08-19T14:50:44.232 INFO:journalctl@ceph.osd.6.smithi032.stdout:Aug 19 14:50:43 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-6[35351]: 2024-08-19T14:50:43.847+0000 7f8d3de10640 -1 osd.6 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:13.516154+0000 front 2024-08-19T14:50:13.516301+0000 (oldest deadline 2024-08-19T14:50:36.416116+0000) 2024-08-19T14:50:44.507 INFO:journalctl@ceph.osd.4.smithi032.stdout:Aug 19 14:50:44 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-4[24402]: 2024-08-19T14:50:44.450+0000 7fad382a0640 -1 osd.4 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:15.418820+0000 front 2024-08-19T14:50:15.418895+0000 (oldest deadline 2024-08-19T14:50:39.518888+0000) 2024-08-19T14:50:44.508 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:50:44 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7[41448]: 2024-08-19T14:50:44.231+0000 7f1321f92640 -1 osd.7 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.778781+0000 front 2024-08-19T14:50:12.778916+0000 (oldest deadline 2024-08-19T14:50:41.579607+0000) 2024-08-19T14:50:44.546 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:50:44 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:50:44.143+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:14.239016+0000 front 2024-08-19T14:50:14.238817+0000 (oldest deadline 2024-08-19T14:50:38.339035+0000) 2024-08-19T14:50:44.546 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:50:44 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:50:44.341+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.724961+0000 front 2024-08-19T14:50:12.724727+0000 (oldest deadline 2024-08-19T14:50:36.224802+0000) 2024-08-19T14:50:45.047 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:50:44 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:50:44.771+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:13.297511+0000 front 2024-08-19T14:50:13.297834+0000 (oldest deadline 2024-08-19T14:50:37.997679+0000) 2024-08-19T14:50:45.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:50:44 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:50:44.850+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.842003+0000 front 2024-08-19T14:50:12.842211+0000 (oldest deadline 2024-08-19T14:50:36.242616+0000) 2024-08-19T14:50:45.257 INFO:journalctl@ceph.osd.6.smithi032.stdout:Aug 19 14:50:44 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-6[35351]: 2024-08-19T14:50:44.846+0000 7f8d3de10640 -1 osd.6 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:13.516154+0000 front 2024-08-19T14:50:13.516301+0000 (oldest deadline 2024-08-19T14:50:36.416116+0000) 2024-08-19T14:50:45.546 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:50:45 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:50:45.108+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:14.239016+0000 front 2024-08-19T14:50:14.238817+0000 (oldest deadline 2024-08-19T14:50:38.339035+0000) 2024-08-19T14:50:45.550 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:50:45 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:50:45.334+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.724961+0000 front 2024-08-19T14:50:12.724727+0000 (oldest deadline 2024-08-19T14:50:36.224802+0000) 2024-08-19T14:50:45.757 INFO:journalctl@ceph.osd.4.smithi032.stdout:Aug 19 14:50:45 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-4[24402]: 2024-08-19T14:50:45.455+0000 7fad382a0640 -1 osd.4 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:15.418820+0000 front 2024-08-19T14:50:15.418895+0000 (oldest deadline 2024-08-19T14:50:39.518888+0000) 2024-08-19T14:50:45.758 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:50:45 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7[41448]: 2024-08-19T14:50:45.277+0000 7f1321f92640 -1 osd.7 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.778781+0000 front 2024-08-19T14:50:12.778916+0000 (oldest deadline 2024-08-19T14:50:41.579607+0000) 2024-08-19T14:50:46.046 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:50:45 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:50:45.726+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:13.297511+0000 front 2024-08-19T14:50:13.297834+0000 (oldest deadline 2024-08-19T14:50:37.997679+0000) 2024-08-19T14:50:46.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:50:45 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:50:45.857+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.842003+0000 front 2024-08-19T14:50:12.842211+0000 (oldest deadline 2024-08-19T14:50:36.242616+0000) 2024-08-19T14:50:46.257 INFO:journalctl@ceph.osd.6.smithi032.stdout:Aug 19 14:50:45 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-6[35351]: 2024-08-19T14:50:45.836+0000 7f8d3de10640 -1 osd.6 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:13.516154+0000 front 2024-08-19T14:50:13.516301+0000 (oldest deadline 2024-08-19T14:50:36.416116+0000) 2024-08-19T14:50:46.379 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:50:46 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:50:46.087+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:14.239016+0000 front 2024-08-19T14:50:14.238817+0000 (oldest deadline 2024-08-19T14:50:38.339035+0000) 2024-08-19T14:50:46.716 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:50:46 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:50:46.378+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.724961+0000 front 2024-08-19T14:50:12.724727+0000 (oldest deadline 2024-08-19T14:50:36.224802+0000) 2024-08-19T14:50:46.757 INFO:journalctl@ceph.osd.4.smithi032.stdout:Aug 19 14:50:46 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-4[24402]: 2024-08-19T14:50:46.481+0000 7fad382a0640 -1 osd.4 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:15.418820+0000 front 2024-08-19T14:50:15.418895+0000 (oldest deadline 2024-08-19T14:50:39.518888+0000) 2024-08-19T14:50:46.758 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:50:46 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7[41448]: 2024-08-19T14:50:46.314+0000 7f1321f92640 -1 osd.7 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.778781+0000 front 2024-08-19T14:50:12.778916+0000 (oldest deadline 2024-08-19T14:50:41.579607+0000) 2024-08-19T14:50:46.758 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:50:46 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7[41448]: 2024-08-19T14:50:46.314+0000 7f1321f92640 -1 osd.7 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:21.580467+0000 front 2024-08-19T14:50:21.580200+0000 (oldest deadline 2024-08-19T14:50:46.280257+0000) 2024-08-19T14:50:47.044 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:50:46 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:50:46.889+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.842003+0000 front 2024-08-19T14:50:12.842211+0000 (oldest deadline 2024-08-19T14:50:36.242616+0000) 2024-08-19T14:50:47.044 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:50:46 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:50:46.715+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:13.297511+0000 front 2024-08-19T14:50:13.297834+0000 (oldest deadline 2024-08-19T14:50:37.997679+0000) 2024-08-19T14:50:47.257 INFO:journalctl@ceph.osd.6.smithi032.stdout:Aug 19 14:50:46 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-6[35351]: 2024-08-19T14:50:46.856+0000 7f8d3de10640 -1 osd.6 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:13.516154+0000 front 2024-08-19T14:50:13.516301+0000 (oldest deadline 2024-08-19T14:50:36.416116+0000) 2024-08-19T14:50:47.258 INFO:journalctl@ceph.osd.5.smithi032.stdout:Aug 19 14:50:46 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-5[29877]: 2024-08-19T14:50:46.874+0000 7f6c79151640 -1 osd.5 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:21.484752+0000 front 2024-08-19T14:50:21.484559+0000 (oldest deadline 2024-08-19T14:50:46.184536+0000) 2024-08-19T14:50:47.296 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:50:47 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:50:47.043+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:14.239016+0000 front 2024-08-19T14:50:14.238817+0000 (oldest deadline 2024-08-19T14:50:38.339035+0000) 2024-08-19T14:50:47.713 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:50:47 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:50:47.418+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.724961+0000 front 2024-08-19T14:50:12.724727+0000 (oldest deadline 2024-08-19T14:50:36.224802+0000) 2024-08-19T14:50:47.713 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:50:47 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:50:47.418+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:20.627490+0000 front 2024-08-19T14:50:20.627552+0000 (oldest deadline 2024-08-19T14:50:46.527500+0000) 2024-08-19T14:50:47.757 INFO:journalctl@ceph.osd.4.smithi032.stdout:Aug 19 14:50:47 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-4[24402]: 2024-08-19T14:50:47.517+0000 7fad382a0640 -1 osd.4 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:15.418820+0000 front 2024-08-19T14:50:15.418895+0000 (oldest deadline 2024-08-19T14:50:39.518888+0000) 2024-08-19T14:50:47.758 INFO:journalctl@ceph.osd.4.smithi032.stdout:Aug 19 14:50:47 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-4[24402]: 2024-08-19T14:50:47.517+0000 7fad382a0640 -1 osd.4 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:21.621746+0000 front 2024-08-19T14:50:21.621894+0000 (oldest deadline 2024-08-19T14:50:46.921869+0000) 2024-08-19T14:50:47.758 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:50:47 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7[41448]: 2024-08-19T14:50:47.345+0000 7f1321f92640 -1 osd.7 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.778781+0000 front 2024-08-19T14:50:12.778916+0000 (oldest deadline 2024-08-19T14:50:41.579607+0000) 2024-08-19T14:50:47.758 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:50:47 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7[41448]: 2024-08-19T14:50:47.345+0000 7f1321f92640 -1 osd.7 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:21.580467+0000 front 2024-08-19T14:50:21.580200+0000 (oldest deadline 2024-08-19T14:50:46.280257+0000) 2024-08-19T14:50:48.028 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:50:47 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:50:47.712+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:13.297511+0000 front 2024-08-19T14:50:13.297834+0000 (oldest deadline 2024-08-19T14:50:37.997679+0000) 2024-08-19T14:50:48.028 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:50:47 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:50:47.917+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.842003+0000 front 2024-08-19T14:50:12.842211+0000 (oldest deadline 2024-08-19T14:50:36.242616+0000) 2024-08-19T14:50:48.028 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:50:47 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:50:47.917+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.744487+0000 front 2024-08-19T14:50:23.744453+0000 (oldest deadline 2024-08-19T14:50:47.844529+0000) 2024-08-19T14:50:48.257 INFO:journalctl@ceph.osd.5.smithi032.stdout:Aug 19 14:50:47 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-5[29877]: 2024-08-19T14:50:47.867+0000 7f6c79151640 -1 osd.5 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:21.484752+0000 front 2024-08-19T14:50:21.484559+0000 (oldest deadline 2024-08-19T14:50:46.184536+0000) 2024-08-19T14:50:48.258 INFO:journalctl@ceph.osd.6.smithi032.stdout:Aug 19 14:50:47 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-6[35351]: 2024-08-19T14:50:47.901+0000 7f8d3de10640 -1 osd.6 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:13.516154+0000 front 2024-08-19T14:50:13.516301+0000 (oldest deadline 2024-08-19T14:50:36.416116+0000) 2024-08-19T14:50:48.258 INFO:journalctl@ceph.osd.6.smithi032.stdout:Aug 19 14:50:47 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-6[35351]: 2024-08-19T14:50:47.901+0000 7f8d3de10640 -1 osd.6 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:21.017791+0000 front 2024-08-19T14:50:21.017641+0000 (oldest deadline 2024-08-19T14:50:47.418155+0000) 2024-08-19T14:50:48.296 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:50:48 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:50:48.027+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:14.239016+0000 front 2024-08-19T14:50:14.238817+0000 (oldest deadline 2024-08-19T14:50:38.339035+0000) 2024-08-19T14:50:48.689 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:50:48 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:50:48.387+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.724961+0000 front 2024-08-19T14:50:12.724727+0000 (oldest deadline 2024-08-19T14:50:36.224802+0000) 2024-08-19T14:50:48.689 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:50:48 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:50:48.387+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:20.627490+0000 front 2024-08-19T14:50:20.627552+0000 (oldest deadline 2024-08-19T14:50:46.527500+0000) 2024-08-19T14:50:48.757 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:50:48 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7[41448]: 2024-08-19T14:50:48.326+0000 7f1321f92640 -1 osd.7 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.778781+0000 front 2024-08-19T14:50:12.778916+0000 (oldest deadline 2024-08-19T14:50:41.579607+0000) 2024-08-19T14:50:48.758 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:50:48 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7[41448]: 2024-08-19T14:50:48.326+0000 7f1321f92640 -1 osd.7 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:21.580467+0000 front 2024-08-19T14:50:21.580200+0000 (oldest deadline 2024-08-19T14:50:46.280257+0000) 2024-08-19T14:50:49.046 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:50:48 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:50:48.688+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:13.297511+0000 front 2024-08-19T14:50:13.297834+0000 (oldest deadline 2024-08-19T14:50:37.997679+0000) 2024-08-19T14:50:49.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:50:48 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:50:48.880+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.842003+0000 front 2024-08-19T14:50:12.842211+0000 (oldest deadline 2024-08-19T14:50:36.242616+0000) 2024-08-19T14:50:49.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:50:48 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:50:48.880+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.744487+0000 front 2024-08-19T14:50:23.744453+0000 (oldest deadline 2024-08-19T14:50:47.844529+0000) 2024-08-19T14:50:49.231 INFO:journalctl@ceph.osd.5.smithi032.stdout:Aug 19 14:50:48 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-5[29877]: 2024-08-19T14:50:48.818+0000 7f6c79151640 -1 osd.5 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:21.484752+0000 front 2024-08-19T14:50:21.484559+0000 (oldest deadline 2024-08-19T14:50:46.184536+0000) 2024-08-19T14:50:49.232 INFO:journalctl@ceph.osd.4.smithi032.stdout:Aug 19 14:50:48 smithi032 podman[162194]: 2024-08-19 14:50:48.820774728 +0000 UTC m=+5.673164246 container died 2e0f5380ea1eb111743d5bdd5768b706b875bbd7be0f5143e8e79de40af3b1df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-4, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, RELEASE=squid-de09dd0, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-08-19T14:50:49.232 INFO:journalctl@ceph.osd.6.smithi032.stdout:Aug 19 14:50:48 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-6[35351]: 2024-08-19T14:50:48.946+0000 7f8d3de10640 -1 osd.6 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:13.516154+0000 front 2024-08-19T14:50:13.516301+0000 (oldest deadline 2024-08-19T14:50:36.416116+0000) 2024-08-19T14:50:49.232 INFO:journalctl@ceph.osd.6.smithi032.stdout:Aug 19 14:50:48 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-6[35351]: 2024-08-19T14:50:48.946+0000 7f8d3de10640 -1 osd.6 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:21.017791+0000 front 2024-08-19T14:50:21.017641+0000 (oldest deadline 2024-08-19T14:50:47.418155+0000) 2024-08-19T14:50:49.391 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:50:49 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:50:49.051+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:14.239016+0000 front 2024-08-19T14:50:14.238817+0000 (oldest deadline 2024-08-19T14:50:38.339035+0000) 2024-08-19T14:50:49.391 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:50:49 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:50:49.051+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:22.841332+0000 front 2024-08-19T14:50:22.841705+0000 (oldest deadline 2024-08-19T14:50:48.741518+0000) 2024-08-19T14:50:49.508 INFO:journalctl@ceph.osd.4.smithi032.stdout:Aug 19 14:50:49 smithi032 podman[162194]: 2024-08-19 14:50:49.328722422 +0000 UTC m=+6.181111933 container cleanup 2e0f5380ea1eb111743d5bdd5768b706b875bbd7be0f5143e8e79de40af3b1df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-4, org.label-schema.build-date=20240731, RELEASE=squid-de09dd0, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux ) 2024-08-19T14:50:49.508 INFO:journalctl@ceph.osd.4.smithi032.stdout:Aug 19 14:50:49 smithi032 bash[162194]: ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-4 2024-08-19T14:50:49.508 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:50:49 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7[41448]: 2024-08-19T14:50:49.375+0000 7f1321f92640 -1 osd.7 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.778781+0000 front 2024-08-19T14:50:12.778916+0000 (oldest deadline 2024-08-19T14:50:41.579607+0000) 2024-08-19T14:50:49.508 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:50:49 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7[41448]: 2024-08-19T14:50:49.375+0000 7f1321f92640 -1 osd.7 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:21.580467+0000 front 2024-08-19T14:50:21.580200+0000 (oldest deadline 2024-08-19T14:50:46.280257+0000) 2024-08-19T14:50:49.649 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:50:49 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:50:49.390+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.724961+0000 front 2024-08-19T14:50:12.724727+0000 (oldest deadline 2024-08-19T14:50:36.224802+0000) 2024-08-19T14:50:49.650 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:50:49 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:50:49.390+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:20.627490+0000 front 2024-08-19T14:50:20.627552+0000 (oldest deadline 2024-08-19T14:50:46.527500+0000) 2024-08-19T14:50:49.931 INFO:journalctl@ceph.osd.4.smithi032.stdout:Aug 19 14:50:49 smithi032 podman[162211]: 2024-08-19 14:50:49.605366356 +0000 UTC m=+0.779024415 container remove 2e0f5380ea1eb111743d5bdd5768b706b875bbd7be0f5143e8e79de40af3b1df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-4, RELEASE=squid-de09dd0, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, org.label-schema.schema-version=1.0) 2024-08-19T14:50:49.932 INFO:journalctl@ceph.osd.5.smithi032.stdout:Aug 19 14:50:49 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-5[29877]: 2024-08-19T14:50:49.775+0000 7f6c79151640 -1 osd.5 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:21.484752+0000 front 2024-08-19T14:50:21.484559+0000 (oldest deadline 2024-08-19T14:50:46.184536+0000) 2024-08-19T14:50:50.015 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:50:49 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:50:49.843+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.842003+0000 front 2024-08-19T14:50:12.842211+0000 (oldest deadline 2024-08-19T14:50:36.242616+0000) 2024-08-19T14:50:50.015 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:50:49 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:50:49.843+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.744487+0000 front 2024-08-19T14:50:23.744453+0000 (oldest deadline 2024-08-19T14:50:47.844529+0000) 2024-08-19T14:50:50.015 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:50:49 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:50:49.648+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:13.297511+0000 front 2024-08-19T14:50:13.297834+0000 (oldest deadline 2024-08-19T14:50:37.997679+0000) 2024-08-19T14:50:50.015 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:50:49 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:50:49.648+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.298667+0000 front 2024-08-19T14:50:23.298816+0000 (oldest deadline 2024-08-19T14:50:49.198731+0000) 2024-08-19T14:50:50.242 INFO:journalctl@ceph.osd.6.smithi032.stdout:Aug 19 14:50:49 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-6[35351]: 2024-08-19T14:50:49.966+0000 7f8d3de10640 -1 osd.6 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:13.516154+0000 front 2024-08-19T14:50:13.516301+0000 (oldest deadline 2024-08-19T14:50:36.416116+0000) 2024-08-19T14:50:50.243 INFO:journalctl@ceph.osd.6.smithi032.stdout:Aug 19 14:50:49 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-6[35351]: 2024-08-19T14:50:49.966+0000 7f8d3de10640 -1 osd.6 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:21.017791+0000 front 2024-08-19T14:50:21.017641+0000 (oldest deadline 2024-08-19T14:50:47.418155+0000) 2024-08-19T14:50:50.243 INFO:journalctl@ceph.osd.4.smithi032.stdout:Aug 19 14:50:49 smithi032 podman[162299]: 2024-08-19 14:50:49.833739952 +0000 UTC m=+0.017173424 image pull ea7a5189f30bad3aca9524788e2965d2368d7ce786df09d4edcd8c3edf9cefba quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9 2024-08-19T14:50:50.243 INFO:journalctl@ceph.osd.4.smithi032.stdout:Aug 19 14:50:49 smithi032 podman[162299]: 2024-08-19 14:50:49.931597804 +0000 UTC m=+0.115031326 container create 24c729252802db67867f25144540e8898f881f7b82a1a25881f7944da213afa8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-4-deactivate, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, RELEASE=squid-de09dd0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2) 2024-08-19T14:50:50.243 INFO:journalctl@ceph.osd.4.smithi032.stdout:Aug 19 14:50:50 smithi032 podman[162299]: 2024-08-19 14:50:50.150857974 +0000 UTC m=+0.334291451 container init 24c729252802db67867f25144540e8898f881f7b82a1a25881f7944da213afa8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-4-deactivate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=squid-de09dd0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731) 2024-08-19T14:50:50.243 INFO:journalctl@ceph.osd.4.smithi032.stdout:Aug 19 14:50:50 smithi032 podman[162299]: 2024-08-19 14:50:50.15431792 +0000 UTC m=+0.337751386 container start 24c729252802db67867f25144540e8898f881f7b82a1a25881f7944da213afa8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-4-deactivate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=squid-de09dd0) 2024-08-19T14:50:50.243 INFO:journalctl@ceph.osd.4.smithi032.stdout:Aug 19 14:50:50 smithi032 podman[162299]: 2024-08-19 14:50:50.242441998 +0000 UTC m=+0.425875468 container attach 24c729252802db67867f25144540e8898f881f7b82a1a25881f7944da213afa8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-4-deactivate, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, GIT_CLEAN=True, RELEASE=squid-de09dd0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-08-19T14:50:50.296 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:50:50 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:50:50.014+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:14.239016+0000 front 2024-08-19T14:50:14.238817+0000 (oldest deadline 2024-08-19T14:50:38.339035+0000) 2024-08-19T14:50:50.296 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:50:50 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:50:50.014+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:22.841332+0000 front 2024-08-19T14:50:22.841705+0000 (oldest deadline 2024-08-19T14:50:48.741518+0000) 2024-08-19T14:50:50.507 INFO:journalctl@ceph.osd.4.smithi032.stdout:Aug 19 14:50:50 smithi032 podman[162299]: 2024-08-19 14:50:50.396595075 +0000 UTC m=+0.580028535 container died 24c729252802db67867f25144540e8898f881f7b82a1a25881f7944da213afa8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-4-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=squid-de09dd0, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0) 2024-08-19T14:50:50.508 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:50:50 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7[41448]: 2024-08-19T14:50:50.415+0000 7f1321f92640 -1 osd.7 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.778781+0000 front 2024-08-19T14:50:12.778916+0000 (oldest deadline 2024-08-19T14:50:41.579607+0000) 2024-08-19T14:50:50.508 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:50:50 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7[41448]: 2024-08-19T14:50:50.415+0000 7f1321f92640 -1 osd.7 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:21.580467+0000 front 2024-08-19T14:50:21.580200+0000 (oldest deadline 2024-08-19T14:50:46.280257+0000) 2024-08-19T14:50:50.646 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:50:50 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:50:50.389+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.724961+0000 front 2024-08-19T14:50:12.724727+0000 (oldest deadline 2024-08-19T14:50:36.224802+0000) 2024-08-19T14:50:50.646 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:50:50 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:50:50.389+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:20.627490+0000 front 2024-08-19T14:50:20.627552+0000 (oldest deadline 2024-08-19T14:50:46.527500+0000) 2024-08-19T14:50:50.950 INFO:journalctl@ceph.osd.5.smithi032.stdout:Aug 19 14:50:50 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-5[29877]: 2024-08-19T14:50:50.744+0000 7f6c79151640 -1 osd.5 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:21.484752+0000 front 2024-08-19T14:50:21.484559+0000 (oldest deadline 2024-08-19T14:50:46.184536+0000) 2024-08-19T14:50:50.981 DEBUG:teuthology.orchestra.run.smithi032:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968@osd.4.service' 2024-08-19T14:50:51.022 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-19T14:50:51.022 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-08-19T14:50:51.022 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-08-19T14:50:51.023 DEBUG:teuthology.orchestra.run.smithi032:> sudo systemctl stop ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968@osd.5 2024-08-19T14:50:51.046 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:50:50 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:50:50.645+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:13.297511+0000 front 2024-08-19T14:50:13.297834+0000 (oldest deadline 2024-08-19T14:50:37.997679+0000) 2024-08-19T14:50:51.047 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:50:50 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:50:50.645+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.298667+0000 front 2024-08-19T14:50:23.298816+0000 (oldest deadline 2024-08-19T14:50:49.198731+0000) 2024-08-19T14:50:51.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:50:50 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:50:50.859+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.842003+0000 front 2024-08-19T14:50:12.842211+0000 (oldest deadline 2024-08-19T14:50:36.242616+0000) 2024-08-19T14:50:51.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:50:50 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:50:50.859+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.744487+0000 front 2024-08-19T14:50:23.744453+0000 (oldest deadline 2024-08-19T14:50:47.844529+0000) 2024-08-19T14:50:51.257 INFO:journalctl@ceph.osd.5.smithi032.stdout:Aug 19 14:50:51 smithi032 systemd[1]: Stopping Ceph osd.5 for ed7d1c6e-5e33-11ef-bccf-c7b262605968... 2024-08-19T14:50:51.258 INFO:journalctl@ceph.osd.6.smithi032.stdout:Aug 19 14:50:50 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-6[35351]: 2024-08-19T14:50:50.949+0000 7f8d3de10640 -1 osd.6 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:13.516154+0000 front 2024-08-19T14:50:13.516301+0000 (oldest deadline 2024-08-19T14:50:36.416116+0000) 2024-08-19T14:50:51.258 INFO:journalctl@ceph.osd.6.smithi032.stdout:Aug 19 14:50:50 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-6[35351]: 2024-08-19T14:50:50.949+0000 7f8d3de10640 -1 osd.6 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:21.017791+0000 front 2024-08-19T14:50:21.017641+0000 (oldest deadline 2024-08-19T14:50:47.418155+0000) 2024-08-19T14:50:51.362 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:50:51 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:50:51.053+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:14.239016+0000 front 2024-08-19T14:50:14.238817+0000 (oldest deadline 2024-08-19T14:50:38.339035+0000) 2024-08-19T14:50:51.362 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:50:51 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:50:51.053+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:22.841332+0000 front 2024-08-19T14:50:22.841705+0000 (oldest deadline 2024-08-19T14:50:48.741518+0000) 2024-08-19T14:50:51.650 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:50:51 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:50:51.361+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.724961+0000 front 2024-08-19T14:50:12.724727+0000 (oldest deadline 2024-08-19T14:50:36.224802+0000) 2024-08-19T14:50:51.650 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:50:51 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:50:51.361+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:20.627490+0000 front 2024-08-19T14:50:20.627552+0000 (oldest deadline 2024-08-19T14:50:46.527500+0000) 2024-08-19T14:50:51.757 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:50:51 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7[41448]: 2024-08-19T14:50:51.433+0000 7f1321f92640 -1 osd.7 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.778781+0000 front 2024-08-19T14:50:12.778916+0000 (oldest deadline 2024-08-19T14:50:41.579607+0000) 2024-08-19T14:50:51.758 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:50:51 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7[41448]: 2024-08-19T14:50:51.433+0000 7f1321f92640 -1 osd.7 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:21.580467+0000 front 2024-08-19T14:50:21.580200+0000 (oldest deadline 2024-08-19T14:50:46.280257+0000) 2024-08-19T14:50:51.758 INFO:journalctl@ceph.osd.5.smithi032.stdout:Aug 19 14:50:51 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-5[29877]: 2024-08-19T14:50:51.365+0000 7f6c7cb3b640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-19T14:50:51.758 INFO:journalctl@ceph.osd.5.smithi032.stdout:Aug 19 14:50:51 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-5[29877]: 2024-08-19T14:50:51.365+0000 7f6c7cb3b640 -1 osd.5 983 *** Got signal Terminated *** 2024-08-19T14:50:51.758 INFO:journalctl@ceph.osd.5.smithi032.stdout:Aug 19 14:50:51 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-5[29877]: 2024-08-19T14:50:51.365+0000 7f6c7cb3b640 -1 osd.5 983 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-08-19T14:50:52.047 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:50:51 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:50:51.649+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:13.297511+0000 front 2024-08-19T14:50:13.297834+0000 (oldest deadline 2024-08-19T14:50:37.997679+0000) 2024-08-19T14:50:52.047 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:50:51 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:50:51.649+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.298667+0000 front 2024-08-19T14:50:23.298816+0000 (oldest deadline 2024-08-19T14:50:49.198731+0000) 2024-08-19T14:50:52.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:50:51 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:50:51.809+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.842003+0000 front 2024-08-19T14:50:12.842211+0000 (oldest deadline 2024-08-19T14:50:36.242616+0000) 2024-08-19T14:50:52.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:50:51 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:50:51.809+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.744487+0000 front 2024-08-19T14:50:23.744453+0000 (oldest deadline 2024-08-19T14:50:47.844529+0000) 2024-08-19T14:50:52.198 INFO:journalctl@ceph.osd.5.smithi032.stdout:Aug 19 14:50:51 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-5[29877]: 2024-08-19T14:50:51.777+0000 7f6c79151640 -1 osd.5 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:21.484752+0000 front 2024-08-19T14:50:21.484559+0000 (oldest deadline 2024-08-19T14:50:46.184536+0000) 2024-08-19T14:50:52.198 INFO:journalctl@ceph.osd.6.smithi032.stdout:Aug 19 14:50:51 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-6[35351]: 2024-08-19T14:50:51.943+0000 7f8d3de10640 -1 osd.6 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:13.516154+0000 front 2024-08-19T14:50:13.516301+0000 (oldest deadline 2024-08-19T14:50:36.416116+0000) 2024-08-19T14:50:52.198 INFO:journalctl@ceph.osd.6.smithi032.stdout:Aug 19 14:50:51 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-6[35351]: 2024-08-19T14:50:51.943+0000 7f8d3de10640 -1 osd.6 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:21.017791+0000 front 2024-08-19T14:50:21.017641+0000 (oldest deadline 2024-08-19T14:50:47.418155+0000) 2024-08-19T14:50:52.382 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:50:52 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:50:52.093+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:14.239016+0000 front 2024-08-19T14:50:14.238817+0000 (oldest deadline 2024-08-19T14:50:38.339035+0000) 2024-08-19T14:50:52.382 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:50:52 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:50:52.093+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:22.841332+0000 front 2024-08-19T14:50:22.841705+0000 (oldest deadline 2024-08-19T14:50:48.741518+0000) 2024-08-19T14:50:52.650 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:50:52 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:50:52.381+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.724961+0000 front 2024-08-19T14:50:12.724727+0000 (oldest deadline 2024-08-19T14:50:36.224802+0000) 2024-08-19T14:50:52.651 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:50:52 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:50:52.381+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:20.627490+0000 front 2024-08-19T14:50:20.627552+0000 (oldest deadline 2024-08-19T14:50:46.527500+0000) 2024-08-19T14:50:52.757 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:50:52 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7[41448]: 2024-08-19T14:50:52.461+0000 7f1321f92640 -1 osd.7 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.778781+0000 front 2024-08-19T14:50:12.778916+0000 (oldest deadline 2024-08-19T14:50:41.579607+0000) 2024-08-19T14:50:52.758 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:50:52 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7[41448]: 2024-08-19T14:50:52.461+0000 7f1321f92640 -1 osd.7 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:21.580467+0000 front 2024-08-19T14:50:21.580200+0000 (oldest deadline 2024-08-19T14:50:46.280257+0000) 2024-08-19T14:50:53.046 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:50:52 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:50:52.649+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:13.297511+0000 front 2024-08-19T14:50:13.297834+0000 (oldest deadline 2024-08-19T14:50:37.997679+0000) 2024-08-19T14:50:53.046 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:50:52 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:50:52.649+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.298667+0000 front 2024-08-19T14:50:23.298816+0000 (oldest deadline 2024-08-19T14:50:49.198731+0000) 2024-08-19T14:50:53.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:50:52 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:50:52.789+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.842003+0000 front 2024-08-19T14:50:12.842211+0000 (oldest deadline 2024-08-19T14:50:36.242616+0000) 2024-08-19T14:50:53.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:50:52 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:50:52.789+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.744487+0000 front 2024-08-19T14:50:23.744453+0000 (oldest deadline 2024-08-19T14:50:47.844529+0000) 2024-08-19T14:50:53.258 INFO:journalctl@ceph.osd.5.smithi032.stdout:Aug 19 14:50:52 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-5[29877]: 2024-08-19T14:50:52.820+0000 7f6c79151640 -1 osd.5 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:21.484752+0000 front 2024-08-19T14:50:21.484559+0000 (oldest deadline 2024-08-19T14:50:46.184536+0000) 2024-08-19T14:50:53.258 INFO:journalctl@ceph.osd.6.smithi032.stdout:Aug 19 14:50:52 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-6[35351]: 2024-08-19T14:50:52.947+0000 7f8d3de10640 -1 osd.6 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:13.516154+0000 front 2024-08-19T14:50:13.516301+0000 (oldest deadline 2024-08-19T14:50:36.416116+0000) 2024-08-19T14:50:53.258 INFO:journalctl@ceph.osd.6.smithi032.stdout:Aug 19 14:50:52 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-6[35351]: 2024-08-19T14:50:52.947+0000 7f8d3de10640 -1 osd.6 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:21.017791+0000 front 2024-08-19T14:50:21.017641+0000 (oldest deadline 2024-08-19T14:50:47.418155+0000) 2024-08-19T14:50:53.418 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:50:53 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:50:53.095+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:14.239016+0000 front 2024-08-19T14:50:14.238817+0000 (oldest deadline 2024-08-19T14:50:38.339035+0000) 2024-08-19T14:50:53.418 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:50:53 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:50:53.095+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:22.841332+0000 front 2024-08-19T14:50:22.841705+0000 (oldest deadline 2024-08-19T14:50:48.741518+0000) 2024-08-19T14:50:53.751 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:50:53 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:50:53.417+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.724961+0000 front 2024-08-19T14:50:12.724727+0000 (oldest deadline 2024-08-19T14:50:36.224802+0000) 2024-08-19T14:50:53.751 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:50:53 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:50:53.417+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:20.627490+0000 front 2024-08-19T14:50:20.627552+0000 (oldest deadline 2024-08-19T14:50:46.527500+0000) 2024-08-19T14:50:53.751 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:50:53 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:50:53.417+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:30.330088+0000 front 2024-08-19T14:50:30.330151+0000 (oldest deadline 2024-08-19T14:50:53.230021+0000) 2024-08-19T14:50:53.751 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:50:53 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:50:53.644+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:13.297511+0000 front 2024-08-19T14:50:13.297834+0000 (oldest deadline 2024-08-19T14:50:37.997679+0000) 2024-08-19T14:50:53.751 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:50:53 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:50:53.644+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.298667+0000 front 2024-08-19T14:50:23.298816+0000 (oldest deadline 2024-08-19T14:50:49.198731+0000) 2024-08-19T14:50:53.757 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:50:53 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7[41448]: 2024-08-19T14:50:53.492+0000 7f1321f92640 -1 osd.7 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.778781+0000 front 2024-08-19T14:50:12.778916+0000 (oldest deadline 2024-08-19T14:50:41.579607+0000) 2024-08-19T14:50:53.757 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:50:53 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7[41448]: 2024-08-19T14:50:53.492+0000 7f1321f92640 -1 osd.7 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:21.580467+0000 front 2024-08-19T14:50:21.580200+0000 (oldest deadline 2024-08-19T14:50:46.280257+0000) 2024-08-19T14:50:54.046 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:50:53 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:50:53.749+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.842003+0000 front 2024-08-19T14:50:12.842211+0000 (oldest deadline 2024-08-19T14:50:36.242616+0000) 2024-08-19T14:50:54.046 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:50:53 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:50:53.749+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.744487+0000 front 2024-08-19T14:50:23.744453+0000 (oldest deadline 2024-08-19T14:50:47.844529+0000) 2024-08-19T14:50:54.257 INFO:journalctl@ceph.osd.5.smithi032.stdout:Aug 19 14:50:53 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-5[29877]: 2024-08-19T14:50:53.781+0000 7f6c79151640 -1 osd.5 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:21.484752+0000 front 2024-08-19T14:50:21.484559+0000 (oldest deadline 2024-08-19T14:50:46.184536+0000) 2024-08-19T14:50:54.258 INFO:journalctl@ceph.osd.6.smithi032.stdout:Aug 19 14:50:53 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-6[35351]: 2024-08-19T14:50:53.947+0000 7f8d3de10640 -1 osd.6 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:13.516154+0000 front 2024-08-19T14:50:13.516301+0000 (oldest deadline 2024-08-19T14:50:36.416116+0000) 2024-08-19T14:50:54.258 INFO:journalctl@ceph.osd.6.smithi032.stdout:Aug 19 14:50:53 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-6[35351]: 2024-08-19T14:50:53.947+0000 7f8d3de10640 -1 osd.6 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:21.017791+0000 front 2024-08-19T14:50:21.017641+0000 (oldest deadline 2024-08-19T14:50:47.418155+0000) 2024-08-19T14:50:54.394 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:50:54 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:50:54.046+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:14.239016+0000 front 2024-08-19T14:50:14.238817+0000 (oldest deadline 2024-08-19T14:50:38.339035+0000) 2024-08-19T14:50:54.394 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:50:54 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:50:54.046+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:22.841332+0000 front 2024-08-19T14:50:22.841705+0000 (oldest deadline 2024-08-19T14:50:48.741518+0000) 2024-08-19T14:50:54.792 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:50:54 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:50:54.392+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.724961+0000 front 2024-08-19T14:50:12.724727+0000 (oldest deadline 2024-08-19T14:50:36.224802+0000) 2024-08-19T14:50:54.792 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:50:54 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:50:54.392+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:20.627490+0000 front 2024-08-19T14:50:20.627552+0000 (oldest deadline 2024-08-19T14:50:46.527500+0000) 2024-08-19T14:50:54.792 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:50:54 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:50:54.392+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:30.330088+0000 front 2024-08-19T14:50:30.330151+0000 (oldest deadline 2024-08-19T14:50:53.230021+0000) 2024-08-19T14:50:54.792 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:50:54 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:50:54.619+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:13.297511+0000 front 2024-08-19T14:50:13.297834+0000 (oldest deadline 2024-08-19T14:50:37.997679+0000) 2024-08-19T14:50:54.792 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:50:54 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:50:54.619+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.298667+0000 front 2024-08-19T14:50:23.298816+0000 (oldest deadline 2024-08-19T14:50:49.198731+0000) 2024-08-19T14:50:54.792 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:50:54 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:50:54.619+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:29.199201+0000 front 2024-08-19T14:50:29.199233+0000 (oldest deadline 2024-08-19T14:50:54.499327+0000) 2024-08-19T14:50:54.998 INFO:journalctl@ceph.osd.5.smithi032.stdout:Aug 19 14:50:54 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-5[29877]: 2024-08-19T14:50:54.755+0000 7f6c79151640 -1 osd.5 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:21.484752+0000 front 2024-08-19T14:50:21.484559+0000 (oldest deadline 2024-08-19T14:50:46.184536+0000) 2024-08-19T14:50:54.998 INFO:journalctl@ceph.osd.5.smithi032.stdout:Aug 19 14:50:54 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-5[29877]: 2024-08-19T14:50:54.755+0000 7f6c79151640 -1 osd.5 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:29.685471+0000 front 2024-08-19T14:50:29.685545+0000 (oldest deadline 2024-08-19T14:50:53.785445+0000) 2024-08-19T14:50:54.998 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:50:54 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7[41448]: 2024-08-19T14:50:54.527+0000 7f1321f92640 -1 osd.7 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.778781+0000 front 2024-08-19T14:50:12.778916+0000 (oldest deadline 2024-08-19T14:50:41.579607+0000) 2024-08-19T14:50:54.998 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:50:54 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7[41448]: 2024-08-19T14:50:54.527+0000 7f1321f92640 -1 osd.7 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:21.580467+0000 front 2024-08-19T14:50:21.580200+0000 (oldest deadline 2024-08-19T14:50:46.280257+0000) 2024-08-19T14:50:55.046 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:50:54 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:50:54.791+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.842003+0000 front 2024-08-19T14:50:12.842211+0000 (oldest deadline 2024-08-19T14:50:36.242616+0000) 2024-08-19T14:50:55.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:50:54 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:50:54.791+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.744487+0000 front 2024-08-19T14:50:23.744453+0000 (oldest deadline 2024-08-19T14:50:47.844529+0000) 2024-08-19T14:50:55.047 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:50:55 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:50:55.033+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:14.239016+0000 front 2024-08-19T14:50:14.238817+0000 (oldest deadline 2024-08-19T14:50:38.339035+0000) 2024-08-19T14:50:55.047 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:50:55 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:50:55.033+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:22.841332+0000 front 2024-08-19T14:50:22.841705+0000 (oldest deadline 2024-08-19T14:50:48.741518+0000) 2024-08-19T14:50:55.258 INFO:journalctl@ceph.osd.6.smithi032.stdout:Aug 19 14:50:54 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-6[35351]: 2024-08-19T14:50:54.996+0000 7f8d3de10640 -1 osd.6 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:13.516154+0000 front 2024-08-19T14:50:13.516301+0000 (oldest deadline 2024-08-19T14:50:36.416116+0000) 2024-08-19T14:50:55.258 INFO:journalctl@ceph.osd.6.smithi032.stdout:Aug 19 14:50:54 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-6[35351]: 2024-08-19T14:50:54.996+0000 7f8d3de10640 -1 osd.6 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:21.017791+0000 front 2024-08-19T14:50:21.017641+0000 (oldest deadline 2024-08-19T14:50:47.418155+0000) 2024-08-19T14:50:55.667 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:50:55 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:50:55.369+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.724961+0000 front 2024-08-19T14:50:12.724727+0000 (oldest deadline 2024-08-19T14:50:36.224802+0000) 2024-08-19T14:50:55.667 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:50:55 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:50:55.369+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:20.627490+0000 front 2024-08-19T14:50:20.627552+0000 (oldest deadline 2024-08-19T14:50:46.527500+0000) 2024-08-19T14:50:55.668 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:50:55 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:50:55.369+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:30.330088+0000 front 2024-08-19T14:50:30.330151+0000 (oldest deadline 2024-08-19T14:50:53.230021+0000) 2024-08-19T14:50:56.007 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:50:55 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7[41448]: 2024-08-19T14:50:55.549+0000 7f1321f92640 -1 osd.7 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.778781+0000 front 2024-08-19T14:50:12.778916+0000 (oldest deadline 2024-08-19T14:50:41.579607+0000) 2024-08-19T14:50:56.008 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:50:55 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7[41448]: 2024-08-19T14:50:55.549+0000 7f1321f92640 -1 osd.7 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:21.580467+0000 front 2024-08-19T14:50:21.580200+0000 (oldest deadline 2024-08-19T14:50:46.280257+0000) 2024-08-19T14:50:56.008 INFO:journalctl@ceph.osd.5.smithi032.stdout:Aug 19 14:50:55 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-5[29877]: 2024-08-19T14:50:55.708+0000 7f6c79151640 -1 osd.5 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:21.484752+0000 front 2024-08-19T14:50:21.484559+0000 (oldest deadline 2024-08-19T14:50:46.184536+0000) 2024-08-19T14:50:56.008 INFO:journalctl@ceph.osd.5.smithi032.stdout:Aug 19 14:50:55 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-5[29877]: 2024-08-19T14:50:55.708+0000 7f6c79151640 -1 osd.5 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:29.685471+0000 front 2024-08-19T14:50:29.685545+0000 (oldest deadline 2024-08-19T14:50:53.785445+0000) 2024-08-19T14:50:56.046 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:50:55 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:50:55.667+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:13.297511+0000 front 2024-08-19T14:50:13.297834+0000 (oldest deadline 2024-08-19T14:50:37.997679+0000) 2024-08-19T14:50:56.046 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:50:55 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:50:55.667+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.298667+0000 front 2024-08-19T14:50:23.298816+0000 (oldest deadline 2024-08-19T14:50:49.198731+0000) 2024-08-19T14:50:56.047 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:50:55 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:50:55.667+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:29.199201+0000 front 2024-08-19T14:50:29.199233+0000 (oldest deadline 2024-08-19T14:50:54.499327+0000) 2024-08-19T14:50:56.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:50:55 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:50:55.762+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.842003+0000 front 2024-08-19T14:50:12.842211+0000 (oldest deadline 2024-08-19T14:50:36.242616+0000) 2024-08-19T14:50:56.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:50:55 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:50:55.762+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.744487+0000 front 2024-08-19T14:50:23.744453+0000 (oldest deadline 2024-08-19T14:50:47.844529+0000) 2024-08-19T14:50:56.382 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:50:56 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:50:56.056+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:14.239016+0000 front 2024-08-19T14:50:14.238817+0000 (oldest deadline 2024-08-19T14:50:38.339035+0000) 2024-08-19T14:50:56.382 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:50:56 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:50:56.056+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:22.841332+0000 front 2024-08-19T14:50:22.841705+0000 (oldest deadline 2024-08-19T14:50:48.741518+0000) 2024-08-19T14:50:56.502 INFO:journalctl@ceph.osd.6.smithi032.stdout:Aug 19 14:50:56 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-6[35351]: 2024-08-19T14:50:56.012+0000 7f8d3de10640 -1 osd.6 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:13.516154+0000 front 2024-08-19T14:50:13.516301+0000 (oldest deadline 2024-08-19T14:50:36.416116+0000) 2024-08-19T14:50:56.502 INFO:journalctl@ceph.osd.6.smithi032.stdout:Aug 19 14:50:56 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-6[35351]: 2024-08-19T14:50:56.012+0000 7f8d3de10640 -1 osd.6 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:21.017791+0000 front 2024-08-19T14:50:21.017641+0000 (oldest deadline 2024-08-19T14:50:47.418155+0000) 2024-08-19T14:50:56.667 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:50:56 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:50:56.381+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.724961+0000 front 2024-08-19T14:50:12.724727+0000 (oldest deadline 2024-08-19T14:50:36.224802+0000) 2024-08-19T14:50:56.667 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:50:56 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:50:56.381+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:20.627490+0000 front 2024-08-19T14:50:20.627552+0000 (oldest deadline 2024-08-19T14:50:46.527500+0000) 2024-08-19T14:50:56.667 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:50:56 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:50:56.381+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:30.330088+0000 front 2024-08-19T14:50:30.330151+0000 (oldest deadline 2024-08-19T14:50:53.230021+0000) 2024-08-19T14:50:56.757 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:50:56 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7[41448]: 2024-08-19T14:50:56.501+0000 7f1321f92640 -1 osd.7 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.778781+0000 front 2024-08-19T14:50:12.778916+0000 (oldest deadline 2024-08-19T14:50:41.579607+0000) 2024-08-19T14:50:56.757 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:50:56 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7[41448]: 2024-08-19T14:50:56.501+0000 7f1321f92640 -1 osd.7 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:21.580467+0000 front 2024-08-19T14:50:21.580200+0000 (oldest deadline 2024-08-19T14:50:46.280257+0000) 2024-08-19T14:50:57.046 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:50:56 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:50:56.666+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:13.297511+0000 front 2024-08-19T14:50:13.297834+0000 (oldest deadline 2024-08-19T14:50:37.997679+0000) 2024-08-19T14:50:57.047 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:50:56 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:50:56.666+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.298667+0000 front 2024-08-19T14:50:23.298816+0000 (oldest deadline 2024-08-19T14:50:49.198731+0000) 2024-08-19T14:50:57.047 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:50:56 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:50:56.666+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:29.199201+0000 front 2024-08-19T14:50:29.199233+0000 (oldest deadline 2024-08-19T14:50:54.499327+0000) 2024-08-19T14:50:57.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:50:56 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:50:56.798+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.842003+0000 front 2024-08-19T14:50:12.842211+0000 (oldest deadline 2024-08-19T14:50:36.242616+0000) 2024-08-19T14:50:57.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:50:56 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:50:56.798+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.744487+0000 front 2024-08-19T14:50:23.744453+0000 (oldest deadline 2024-08-19T14:50:47.844529+0000) 2024-08-19T14:50:57.209 INFO:journalctl@ceph.osd.5.smithi032.stdout:Aug 19 14:50:56 smithi032 podman[162401]: 2024-08-19 14:50:56.940407036 +0000 UTC m=+5.683827773 container died c9887c3b59017cb428ba8fd282fbcc151a1c48cc410dee099bc1f0a6c15ce3f9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-5, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, RELEASE=squid-de09dd0, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0) 2024-08-19T14:50:57.209 INFO:journalctl@ceph.osd.6.smithi032.stdout:Aug 19 14:50:56 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-6[35351]: 2024-08-19T14:50:56.968+0000 7f8d3de10640 -1 osd.6 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:13.516154+0000 front 2024-08-19T14:50:13.516301+0000 (oldest deadline 2024-08-19T14:50:36.416116+0000) 2024-08-19T14:50:57.210 INFO:journalctl@ceph.osd.6.smithi032.stdout:Aug 19 14:50:56 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-6[35351]: 2024-08-19T14:50:56.968+0000 7f8d3de10640 -1 osd.6 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:21.017791+0000 front 2024-08-19T14:50:21.017641+0000 (oldest deadline 2024-08-19T14:50:47.418155+0000) 2024-08-19T14:50:57.210 INFO:journalctl@ceph.osd.6.smithi032.stdout:Aug 19 14:50:56 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-6[35351]: 2024-08-19T14:50:56.968+0000 7f8d3de10640 -1 osd.6 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:32.019846+0000 front 2024-08-19T14:50:32.019867+0000 (oldest deadline 2024-08-19T14:50:56.119827+0000) 2024-08-19T14:50:57.419 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:50:57 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:50:57.092+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:14.239016+0000 front 2024-08-19T14:50:14.238817+0000 (oldest deadline 2024-08-19T14:50:38.339035+0000) 2024-08-19T14:50:57.419 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:50:57 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:50:57.092+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:22.841332+0000 front 2024-08-19T14:50:22.841705+0000 (oldest deadline 2024-08-19T14:50:48.741518+0000) 2024-08-19T14:50:57.507 INFO:journalctl@ceph.osd.5.smithi032.stdout:Aug 19 14:50:57 smithi032 podman[162401]: 2024-08-19 14:50:57.308326516 +0000 UTC m=+6.051747298 container cleanup c9887c3b59017cb428ba8fd282fbcc151a1c48cc410dee099bc1f0a6c15ce3f9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-5, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=squid-de09dd0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-08-19T14:50:57.508 INFO:journalctl@ceph.osd.5.smithi032.stdout:Aug 19 14:50:57 smithi032 bash[162401]: ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-5 2024-08-19T14:50:57.793 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:50:57 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:50:57.419+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.724961+0000 front 2024-08-19T14:50:12.724727+0000 (oldest deadline 2024-08-19T14:50:36.224802+0000) 2024-08-19T14:50:57.794 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:50:57 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:50:57.419+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:20.627490+0000 front 2024-08-19T14:50:20.627552+0000 (oldest deadline 2024-08-19T14:50:46.527500+0000) 2024-08-19T14:50:57.794 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:50:57 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:50:57.419+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:30.330088+0000 front 2024-08-19T14:50:30.330151+0000 (oldest deadline 2024-08-19T14:50:53.230021+0000) 2024-08-19T14:50:57.794 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:50:57 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:50:57.658+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:13.297511+0000 front 2024-08-19T14:50:13.297834+0000 (oldest deadline 2024-08-19T14:50:37.997679+0000) 2024-08-19T14:50:57.794 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:50:57 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:50:57.658+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.298667+0000 front 2024-08-19T14:50:23.298816+0000 (oldest deadline 2024-08-19T14:50:49.198731+0000) 2024-08-19T14:50:57.794 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:50:57 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:50:57.658+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:29.199201+0000 front 2024-08-19T14:50:29.199233+0000 (oldest deadline 2024-08-19T14:50:54.499327+0000) 2024-08-19T14:50:57.950 INFO:journalctl@ceph.osd.5.smithi032.stdout:Aug 19 14:50:57 smithi032 podman[162427]: 2024-08-19 14:50:57.759679695 +0000 UTC m=+0.813352322 container remove c9887c3b59017cb428ba8fd282fbcc151a1c48cc410dee099bc1f0a6c15ce3f9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=squid-de09dd0, io.buildah.version=1.36.0, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731) 2024-08-19T14:50:57.950 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:50:57 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7[41448]: 2024-08-19T14:50:57.514+0000 7f1321f92640 -1 osd.7 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.778781+0000 front 2024-08-19T14:50:12.778916+0000 (oldest deadline 2024-08-19T14:50:41.579607+0000) 2024-08-19T14:50:57.950 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:50:57 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7[41448]: 2024-08-19T14:50:57.514+0000 7f1321f92640 -1 osd.7 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:21.580467+0000 front 2024-08-19T14:50:21.580200+0000 (oldest deadline 2024-08-19T14:50:46.280257+0000) 2024-08-19T14:50:58.046 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:50:57 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:50:57.793+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.842003+0000 front 2024-08-19T14:50:12.842211+0000 (oldest deadline 2024-08-19T14:50:36.242616+0000) 2024-08-19T14:50:58.046 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:50:57 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:50:57.793+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.744487+0000 front 2024-08-19T14:50:23.744453+0000 (oldest deadline 2024-08-19T14:50:47.844529+0000) 2024-08-19T14:50:58.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:50:57 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:50:57.793+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:31.345568+0000 front 2024-08-19T14:50:31.345651+0000 (oldest deadline 2024-08-19T14:50:57.745917+0000) 2024-08-19T14:50:58.210 INFO:journalctl@ceph.osd.5.smithi032.stdout:Aug 19 14:50:58 smithi032 podman[162515]: 2024-08-19 14:50:57.987421866 +0000 UTC m=+0.017124028 image pull ea7a5189f30bad3aca9524788e2965d2368d7ce786df09d4edcd8c3edf9cefba quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9 2024-08-19T14:50:58.210 INFO:journalctl@ceph.osd.5.smithi032.stdout:Aug 19 14:50:58 smithi032 podman[162515]: 2024-08-19 14:50:58.102372112 +0000 UTC m=+0.132074258 container create 30274bda4a7724e7d808b53cab3ad0af693a47800aa214b838f061df832fb672 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-5-deactivate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, RELEASE=squid-de09dd0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-08-19T14:50:58.210 INFO:journalctl@ceph.osd.6.smithi032.stdout:Aug 19 14:50:57 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-6[35351]: 2024-08-19T14:50:57.949+0000 7f8d3de10640 -1 osd.6 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:13.516154+0000 front 2024-08-19T14:50:13.516301+0000 (oldest deadline 2024-08-19T14:50:36.416116+0000) 2024-08-19T14:50:58.211 INFO:journalctl@ceph.osd.6.smithi032.stdout:Aug 19 14:50:57 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-6[35351]: 2024-08-19T14:50:57.949+0000 7f8d3de10640 -1 osd.6 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:21.017791+0000 front 2024-08-19T14:50:21.017641+0000 (oldest deadline 2024-08-19T14:50:47.418155+0000) 2024-08-19T14:50:58.211 INFO:journalctl@ceph.osd.6.smithi032.stdout:Aug 19 14:50:57 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-6[35351]: 2024-08-19T14:50:57.949+0000 7f8d3de10640 -1 osd.6 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:32.019846+0000 front 2024-08-19T14:50:32.019867+0000 (oldest deadline 2024-08-19T14:50:56.119827+0000) 2024-08-19T14:50:58.373 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:50:58 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:50:58.076+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:14.239016+0000 front 2024-08-19T14:50:14.238817+0000 (oldest deadline 2024-08-19T14:50:38.339035+0000) 2024-08-19T14:50:58.373 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:50:58 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:50:58.076+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:22.841332+0000 front 2024-08-19T14:50:22.841705+0000 (oldest deadline 2024-08-19T14:50:48.741518+0000) 2024-08-19T14:50:58.481 INFO:journalctl@ceph.osd.5.smithi032.stdout:Aug 19 14:50:58 smithi032 podman[162515]: 2024-08-19 14:50:58.321655356 +0000 UTC m=+0.351357503 container init 30274bda4a7724e7d808b53cab3ad0af693a47800aa214b838f061df832fb672 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-de09dd0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-08-19T14:50:58.481 INFO:journalctl@ceph.osd.5.smithi032.stdout:Aug 19 14:50:58 smithi032 podman[162515]: 2024-08-19 14:50:58.324826639 +0000 UTC m=+0.354528785 container start 30274bda4a7724e7d808b53cab3ad0af693a47800aa214b838f061df832fb672 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-5-deactivate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-de09dd0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-08-19T14:50:58.481 INFO:journalctl@ceph.osd.5.smithi032.stdout:Aug 19 14:50:58 smithi032 podman[162515]: 2024-08-19 14:50:58.429910089 +0000 UTC m=+0.459612243 container attach 30274bda4a7724e7d808b53cab3ad0af693a47800aa214b838f061df832fb672 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-5-deactivate, RELEASE=squid-de09dd0, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-08-19T14:50:58.695 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:50:58 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:50:58.373+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.724961+0000 front 2024-08-19T14:50:12.724727+0000 (oldest deadline 2024-08-19T14:50:36.224802+0000) 2024-08-19T14:50:58.695 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:50:58 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:50:58.373+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:20.627490+0000 front 2024-08-19T14:50:20.627552+0000 (oldest deadline 2024-08-19T14:50:46.527500+0000) 2024-08-19T14:50:58.695 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:50:58 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:50:58.373+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:30.330088+0000 front 2024-08-19T14:50:30.330151+0000 (oldest deadline 2024-08-19T14:50:53.230021+0000) 2024-08-19T14:50:58.757 INFO:journalctl@ceph.osd.5.smithi032.stdout:Aug 19 14:50:58 smithi032 podman[162515]: 2024-08-19 14:50:58.511090985 +0000 UTC m=+0.540793141 container died 30274bda4a7724e7d808b53cab3ad0af693a47800aa214b838f061df832fb672 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-5-deactivate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , RELEASE=squid-de09dd0, io.buildah.version=1.36.0) 2024-08-19T14:50:58.758 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:50:58 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7[41448]: 2024-08-19T14:50:58.479+0000 7f1321f92640 -1 osd.7 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.778781+0000 front 2024-08-19T14:50:12.778916+0000 (oldest deadline 2024-08-19T14:50:41.579607+0000) 2024-08-19T14:50:58.758 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:50:58 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7[41448]: 2024-08-19T14:50:58.479+0000 7f1321f92640 -1 osd.7 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:21.580467+0000 front 2024-08-19T14:50:21.580200+0000 (oldest deadline 2024-08-19T14:50:46.280257+0000) 2024-08-19T14:50:59.029 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:50:58 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:50:58.798+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.842003+0000 front 2024-08-19T14:50:12.842211+0000 (oldest deadline 2024-08-19T14:50:36.242616+0000) 2024-08-19T14:50:59.030 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:50:58 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:50:58.798+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.744487+0000 front 2024-08-19T14:50:23.744453+0000 (oldest deadline 2024-08-19T14:50:47.844529+0000) 2024-08-19T14:50:59.030 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:50:58 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:50:58.798+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:31.345568+0000 front 2024-08-19T14:50:31.345651+0000 (oldest deadline 2024-08-19T14:50:57.745917+0000) 2024-08-19T14:50:59.030 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:50:58 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:50:58.694+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:13.297511+0000 front 2024-08-19T14:50:13.297834+0000 (oldest deadline 2024-08-19T14:50:37.997679+0000) 2024-08-19T14:50:59.030 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:50:58 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:50:58.694+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.298667+0000 front 2024-08-19T14:50:23.298816+0000 (oldest deadline 2024-08-19T14:50:49.198731+0000) 2024-08-19T14:50:59.030 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:50:58 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:50:58.694+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:29.199201+0000 front 2024-08-19T14:50:29.199233+0000 (oldest deadline 2024-08-19T14:50:54.499327+0000) 2024-08-19T14:50:59.134 INFO:journalctl@ceph.osd.6.smithi032.stdout:Aug 19 14:50:58 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-6[35351]: 2024-08-19T14:50:58.985+0000 7f8d3de10640 -1 osd.6 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:13.516154+0000 front 2024-08-19T14:50:13.516301+0000 (oldest deadline 2024-08-19T14:50:36.416116+0000) 2024-08-19T14:50:59.134 INFO:journalctl@ceph.osd.6.smithi032.stdout:Aug 19 14:50:58 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-6[35351]: 2024-08-19T14:50:58.985+0000 7f8d3de10640 -1 osd.6 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:21.017791+0000 front 2024-08-19T14:50:21.017641+0000 (oldest deadline 2024-08-19T14:50:47.418155+0000) 2024-08-19T14:50:59.134 INFO:journalctl@ceph.osd.6.smithi032.stdout:Aug 19 14:50:58 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-6[35351]: 2024-08-19T14:50:58.985+0000 7f8d3de10640 -1 osd.6 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:32.019846+0000 front 2024-08-19T14:50:32.019867+0000 (oldest deadline 2024-08-19T14:50:56.119827+0000) 2024-08-19T14:50:59.165 DEBUG:teuthology.orchestra.run.smithi032:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968@osd.5.service' 2024-08-19T14:50:59.204 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-19T14:50:59.205 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-08-19T14:50:59.205 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-08-19T14:50:59.205 DEBUG:teuthology.orchestra.run.smithi032:> sudo systemctl stop ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968@osd.6 2024-08-19T14:50:59.296 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:50:59 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:50:59.027+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:14.239016+0000 front 2024-08-19T14:50:14.238817+0000 (oldest deadline 2024-08-19T14:50:38.339035+0000) 2024-08-19T14:50:59.296 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:50:59 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:50:59.027+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:22.841332+0000 front 2024-08-19T14:50:22.841705+0000 (oldest deadline 2024-08-19T14:50:48.741518+0000) 2024-08-19T14:50:59.440 INFO:journalctl@ceph.osd.6.smithi032.stdout:Aug 19 14:50:59 smithi032 systemd[1]: Stopping Ceph osd.6 for ed7d1c6e-5e33-11ef-bccf-c7b262605968... 2024-08-19T14:50:59.650 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:50:59 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:50:59.324+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.724961+0000 front 2024-08-19T14:50:12.724727+0000 (oldest deadline 2024-08-19T14:50:36.224802+0000) 2024-08-19T14:50:59.651 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:50:59 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:50:59.324+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:20.627490+0000 front 2024-08-19T14:50:20.627552+0000 (oldest deadline 2024-08-19T14:50:46.527500+0000) 2024-08-19T14:50:59.651 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:50:59 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:50:59.324+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:30.330088+0000 front 2024-08-19T14:50:30.330151+0000 (oldest deadline 2024-08-19T14:50:53.230021+0000) 2024-08-19T14:50:59.757 INFO:journalctl@ceph.osd.6.smithi032.stdout:Aug 19 14:50:59 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-6[35351]: 2024-08-19T14:50:59.561+0000 7f8d417fa640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-19T14:50:59.758 INFO:journalctl@ceph.osd.6.smithi032.stdout:Aug 19 14:50:59 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-6[35351]: 2024-08-19T14:50:59.561+0000 7f8d417fa640 -1 osd.6 983 *** Got signal Terminated *** 2024-08-19T14:50:59.758 INFO:journalctl@ceph.osd.6.smithi032.stdout:Aug 19 14:50:59 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-6[35351]: 2024-08-19T14:50:59.561+0000 7f8d417fa640 -1 osd.6 983 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-08-19T14:50:59.758 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:50:59 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7[41448]: 2024-08-19T14:50:59.438+0000 7f1321f92640 -1 osd.7 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.778781+0000 front 2024-08-19T14:50:12.778916+0000 (oldest deadline 2024-08-19T14:50:41.579607+0000) 2024-08-19T14:50:59.758 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:50:59 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7[41448]: 2024-08-19T14:50:59.438+0000 7f1321f92640 -1 osd.7 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:21.580467+0000 front 2024-08-19T14:50:21.580200+0000 (oldest deadline 2024-08-19T14:50:46.280257+0000) 2024-08-19T14:50:59.758 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:50:59 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7[41448]: 2024-08-19T14:50:59.438+0000 7f1321f92640 -1 osd.7 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:29.682438+0000 front 2024-08-19T14:50:29.682226+0000 (oldest deadline 2024-08-19T14:50:58.482746+0000) 2024-08-19T14:51:00.046 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:50:59 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:50:59.650+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:13.297511+0000 front 2024-08-19T14:50:13.297834+0000 (oldest deadline 2024-08-19T14:50:37.997679+0000) 2024-08-19T14:51:00.047 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:50:59 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:50:59.650+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.298667+0000 front 2024-08-19T14:50:23.298816+0000 (oldest deadline 2024-08-19T14:50:49.198731+0000) 2024-08-19T14:51:00.047 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:50:59 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:50:59.650+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:29.199201+0000 front 2024-08-19T14:50:29.199233+0000 (oldest deadline 2024-08-19T14:50:54.499327+0000) 2024-08-19T14:51:00.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:50:59 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:50:59.830+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.842003+0000 front 2024-08-19T14:50:12.842211+0000 (oldest deadline 2024-08-19T14:50:36.242616+0000) 2024-08-19T14:51:00.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:50:59 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:50:59.830+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.744487+0000 front 2024-08-19T14:50:23.744453+0000 (oldest deadline 2024-08-19T14:50:47.844529+0000) 2024-08-19T14:51:00.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:50:59 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:50:59.830+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:31.345568+0000 front 2024-08-19T14:50:31.345651+0000 (oldest deadline 2024-08-19T14:50:57.745917+0000) 2024-08-19T14:51:00.350 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:51:00 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:51:00.053+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:14.239016+0000 front 2024-08-19T14:50:14.238817+0000 (oldest deadline 2024-08-19T14:50:38.339035+0000) 2024-08-19T14:51:00.350 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:51:00 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:51:00.053+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:22.841332+0000 front 2024-08-19T14:50:22.841705+0000 (oldest deadline 2024-08-19T14:50:48.741518+0000) 2024-08-19T14:51:00.478 INFO:journalctl@ceph.osd.6.smithi032.stdout:Aug 19 14:51:00 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-6[35351]: 2024-08-19T14:51:00.023+0000 7f8d3de10640 -1 osd.6 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:13.516154+0000 front 2024-08-19T14:50:13.516301+0000 (oldest deadline 2024-08-19T14:50:36.416116+0000) 2024-08-19T14:51:00.478 INFO:journalctl@ceph.osd.6.smithi032.stdout:Aug 19 14:51:00 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-6[35351]: 2024-08-19T14:51:00.023+0000 7f8d3de10640 -1 osd.6 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:21.017791+0000 front 2024-08-19T14:50:21.017641+0000 (oldest deadline 2024-08-19T14:50:47.418155+0000) 2024-08-19T14:51:00.478 INFO:journalctl@ceph.osd.6.smithi032.stdout:Aug 19 14:51:00 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-6[35351]: 2024-08-19T14:51:00.023+0000 7f8d3de10640 -1 osd.6 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:32.019846+0000 front 2024-08-19T14:50:32.019867+0000 (oldest deadline 2024-08-19T14:50:56.119827+0000) 2024-08-19T14:51:00.610 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:00 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:00.349+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.724961+0000 front 2024-08-19T14:50:12.724727+0000 (oldest deadline 2024-08-19T14:50:36.224802+0000) 2024-08-19T14:51:00.611 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:00 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:00.349+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:20.627490+0000 front 2024-08-19T14:50:20.627552+0000 (oldest deadline 2024-08-19T14:50:46.527500+0000) 2024-08-19T14:51:00.611 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:00 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:00.349+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:30.330088+0000 front 2024-08-19T14:50:30.330151+0000 (oldest deadline 2024-08-19T14:50:53.230021+0000) 2024-08-19T14:51:00.757 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:51:00 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7[41448]: 2024-08-19T14:51:00.477+0000 7f1321f92640 -1 osd.7 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.778781+0000 front 2024-08-19T14:50:12.778916+0000 (oldest deadline 2024-08-19T14:50:41.579607+0000) 2024-08-19T14:51:00.757 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:51:00 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7[41448]: 2024-08-19T14:51:00.477+0000 7f1321f92640 -1 osd.7 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:21.580467+0000 front 2024-08-19T14:50:21.580200+0000 (oldest deadline 2024-08-19T14:50:46.280257+0000) 2024-08-19T14:51:00.757 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:51:00 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7[41448]: 2024-08-19T14:51:00.477+0000 7f1321f92640 -1 osd.7 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:29.682438+0000 front 2024-08-19T14:50:29.682226+0000 (oldest deadline 2024-08-19T14:50:58.482746+0000) 2024-08-19T14:51:00.873 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:00 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:00.609+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:13.297511+0000 front 2024-08-19T14:50:13.297834+0000 (oldest deadline 2024-08-19T14:50:37.997679+0000) 2024-08-19T14:51:00.873 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:00 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:00.609+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.298667+0000 front 2024-08-19T14:50:23.298816+0000 (oldest deadline 2024-08-19T14:50:49.198731+0000) 2024-08-19T14:51:00.873 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:00 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:00.609+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:29.199201+0000 front 2024-08-19T14:50:29.199233+0000 (oldest deadline 2024-08-19T14:50:54.499327+0000) 2024-08-19T14:51:01.296 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:51:01 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:51:01.068+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:14.239016+0000 front 2024-08-19T14:50:14.238817+0000 (oldest deadline 2024-08-19T14:50:38.339035+0000) 2024-08-19T14:51:01.296 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:51:01 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:51:01.068+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:22.841332+0000 front 2024-08-19T14:50:22.841705+0000 (oldest deadline 2024-08-19T14:50:48.741518+0000) 2024-08-19T14:51:01.297 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:00 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:00.872+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.842003+0000 front 2024-08-19T14:50:12.842211+0000 (oldest deadline 2024-08-19T14:50:36.242616+0000) 2024-08-19T14:51:01.297 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:00 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:00.872+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.744487+0000 front 2024-08-19T14:50:23.744453+0000 (oldest deadline 2024-08-19T14:50:47.844529+0000) 2024-08-19T14:51:01.297 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:00 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:00.872+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:31.345568+0000 front 2024-08-19T14:50:31.345651+0000 (oldest deadline 2024-08-19T14:50:57.745917+0000) 2024-08-19T14:51:01.456 INFO:journalctl@ceph.osd.6.smithi032.stdout:Aug 19 14:51:01 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-6[35351]: 2024-08-19T14:51:01.072+0000 7f8d3de10640 -1 osd.6 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:13.516154+0000 front 2024-08-19T14:50:13.516301+0000 (oldest deadline 2024-08-19T14:50:36.416116+0000) 2024-08-19T14:51:01.456 INFO:journalctl@ceph.osd.6.smithi032.stdout:Aug 19 14:51:01 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-6[35351]: 2024-08-19T14:51:01.072+0000 7f8d3de10640 -1 osd.6 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:21.017791+0000 front 2024-08-19T14:50:21.017641+0000 (oldest deadline 2024-08-19T14:50:47.418155+0000) 2024-08-19T14:51:01.456 INFO:journalctl@ceph.osd.6.smithi032.stdout:Aug 19 14:51:01 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-6[35351]: 2024-08-19T14:51:01.072+0000 7f8d3de10640 -1 osd.6 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:32.019846+0000 front 2024-08-19T14:50:32.019867+0000 (oldest deadline 2024-08-19T14:50:56.119827+0000) 2024-08-19T14:51:01.757 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:51:01 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7[41448]: 2024-08-19T14:51:01.455+0000 7f1321f92640 -1 osd.7 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.778781+0000 front 2024-08-19T14:50:12.778916+0000 (oldest deadline 2024-08-19T14:50:41.579607+0000) 2024-08-19T14:51:01.758 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:51:01 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7[41448]: 2024-08-19T14:51:01.455+0000 7f1321f92640 -1 osd.7 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:21.580467+0000 front 2024-08-19T14:50:21.580200+0000 (oldest deadline 2024-08-19T14:50:46.280257+0000) 2024-08-19T14:51:01.758 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:51:01 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7[41448]: 2024-08-19T14:51:01.455+0000 7f1321f92640 -1 osd.7 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:29.682438+0000 front 2024-08-19T14:50:29.682226+0000 (oldest deadline 2024-08-19T14:50:58.482746+0000) 2024-08-19T14:51:01.796 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:01 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:01.361+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.724961+0000 front 2024-08-19T14:50:12.724727+0000 (oldest deadline 2024-08-19T14:50:36.224802+0000) 2024-08-19T14:51:01.796 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:01 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:01.361+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:20.627490+0000 front 2024-08-19T14:50:20.627552+0000 (oldest deadline 2024-08-19T14:50:46.527500+0000) 2024-08-19T14:51:01.797 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:01 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:01.361+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:30.330088+0000 front 2024-08-19T14:50:30.330151+0000 (oldest deadline 2024-08-19T14:50:53.230021+0000) 2024-08-19T14:51:01.797 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:01 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:01.562+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:13.297511+0000 front 2024-08-19T14:50:13.297834+0000 (oldest deadline 2024-08-19T14:50:37.997679+0000) 2024-08-19T14:51:01.797 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:01 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:01.562+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.298667+0000 front 2024-08-19T14:50:23.298816+0000 (oldest deadline 2024-08-19T14:50:49.198731+0000) 2024-08-19T14:51:01.797 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:01 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:01.562+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:29.199201+0000 front 2024-08-19T14:50:29.199233+0000 (oldest deadline 2024-08-19T14:50:54.499327+0000) 2024-08-19T14:51:02.296 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:51:02 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:51:02.079+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:14.239016+0000 front 2024-08-19T14:50:14.238817+0000 (oldest deadline 2024-08-19T14:50:38.339035+0000) 2024-08-19T14:51:02.297 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:51:02 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:51:02.079+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:22.841332+0000 front 2024-08-19T14:50:22.841705+0000 (oldest deadline 2024-08-19T14:50:48.741518+0000) 2024-08-19T14:51:02.297 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:01 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:01.899+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.842003+0000 front 2024-08-19T14:50:12.842211+0000 (oldest deadline 2024-08-19T14:50:36.242616+0000) 2024-08-19T14:51:02.297 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:01 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:01.899+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.744487+0000 front 2024-08-19T14:50:23.744453+0000 (oldest deadline 2024-08-19T14:50:47.844529+0000) 2024-08-19T14:51:02.297 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:01 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:01.899+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:31.345568+0000 front 2024-08-19T14:50:31.345651+0000 (oldest deadline 2024-08-19T14:50:57.745917+0000) 2024-08-19T14:51:02.297 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:01 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:01.899+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6830 osd.3 since back 2024-08-19T14:50:38.846810+0000 front 2024-08-19T14:50:38.846826+0000 (oldest deadline 2024-08-19T14:51:01.746744+0000) 2024-08-19T14:51:02.439 INFO:journalctl@ceph.osd.6.smithi032.stdout:Aug 19 14:51:02 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-6[35351]: 2024-08-19T14:51:02.063+0000 7f8d3de10640 -1 osd.6 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:13.516154+0000 front 2024-08-19T14:50:13.516301+0000 (oldest deadline 2024-08-19T14:50:36.416116+0000) 2024-08-19T14:51:02.440 INFO:journalctl@ceph.osd.6.smithi032.stdout:Aug 19 14:51:02 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-6[35351]: 2024-08-19T14:51:02.063+0000 7f8d3de10640 -1 osd.6 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:21.017791+0000 front 2024-08-19T14:50:21.017641+0000 (oldest deadline 2024-08-19T14:50:47.418155+0000) 2024-08-19T14:51:02.440 INFO:journalctl@ceph.osd.6.smithi032.stdout:Aug 19 14:51:02 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-6[35351]: 2024-08-19T14:51:02.063+0000 7f8d3de10640 -1 osd.6 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:32.019846+0000 front 2024-08-19T14:50:32.019867+0000 (oldest deadline 2024-08-19T14:50:56.119827+0000) 2024-08-19T14:51:02.757 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:51:02 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7[41448]: 2024-08-19T14:51:02.438+0000 7f1321f92640 -1 osd.7 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.778781+0000 front 2024-08-19T14:50:12.778916+0000 (oldest deadline 2024-08-19T14:50:41.579607+0000) 2024-08-19T14:51:02.758 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:51:02 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7[41448]: 2024-08-19T14:51:02.438+0000 7f1321f92640 -1 osd.7 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:21.580467+0000 front 2024-08-19T14:50:21.580200+0000 (oldest deadline 2024-08-19T14:50:46.280257+0000) 2024-08-19T14:51:02.758 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:51:02 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7[41448]: 2024-08-19T14:51:02.438+0000 7f1321f92640 -1 osd.7 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:29.682438+0000 front 2024-08-19T14:50:29.682226+0000 (oldest deadline 2024-08-19T14:50:58.482746+0000) 2024-08-19T14:51:02.796 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:02 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:02.348+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.724961+0000 front 2024-08-19T14:50:12.724727+0000 (oldest deadline 2024-08-19T14:50:36.224802+0000) 2024-08-19T14:51:02.796 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:02 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:02.348+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:20.627490+0000 front 2024-08-19T14:50:20.627552+0000 (oldest deadline 2024-08-19T14:50:46.527500+0000) 2024-08-19T14:51:02.796 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:02 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:02.348+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:30.330088+0000 front 2024-08-19T14:50:30.330151+0000 (oldest deadline 2024-08-19T14:50:53.230021+0000) 2024-08-19T14:51:02.797 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:02 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:02.528+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:13.297511+0000 front 2024-08-19T14:50:13.297834+0000 (oldest deadline 2024-08-19T14:50:37.997679+0000) 2024-08-19T14:51:02.797 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:02 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:02.528+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.298667+0000 front 2024-08-19T14:50:23.298816+0000 (oldest deadline 2024-08-19T14:50:49.198731+0000) 2024-08-19T14:51:02.797 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:02 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:02.528+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:29.199201+0000 front 2024-08-19T14:50:29.199233+0000 (oldest deadline 2024-08-19T14:50:54.499327+0000) 2024-08-19T14:51:03.296 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:51:03 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:51:03.056+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:14.239016+0000 front 2024-08-19T14:50:14.238817+0000 (oldest deadline 2024-08-19T14:50:38.339035+0000) 2024-08-19T14:51:03.296 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:51:03 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:51:03.056+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:22.841332+0000 front 2024-08-19T14:50:22.841705+0000 (oldest deadline 2024-08-19T14:50:48.741518+0000) 2024-08-19T14:51:03.297 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:02 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:02.887+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.842003+0000 front 2024-08-19T14:50:12.842211+0000 (oldest deadline 2024-08-19T14:50:36.242616+0000) 2024-08-19T14:51:03.297 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:02 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:02.887+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.744487+0000 front 2024-08-19T14:50:23.744453+0000 (oldest deadline 2024-08-19T14:50:47.844529+0000) 2024-08-19T14:51:03.297 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:02 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:02.887+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:31.345568+0000 front 2024-08-19T14:50:31.345651+0000 (oldest deadline 2024-08-19T14:50:57.745917+0000) 2024-08-19T14:51:03.297 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:02 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:02.887+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6830 osd.3 since back 2024-08-19T14:50:38.846810+0000 front 2024-08-19T14:50:38.846826+0000 (oldest deadline 2024-08-19T14:51:01.746744+0000) 2024-08-19T14:51:03.390 INFO:journalctl@ceph.osd.6.smithi032.stdout:Aug 19 14:51:03 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-6[35351]: 2024-08-19T14:51:03.024+0000 7f8d3de10640 -1 osd.6 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:13.516154+0000 front 2024-08-19T14:50:13.516301+0000 (oldest deadline 2024-08-19T14:50:36.416116+0000) 2024-08-19T14:51:03.390 INFO:journalctl@ceph.osd.6.smithi032.stdout:Aug 19 14:51:03 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-6[35351]: 2024-08-19T14:51:03.024+0000 7f8d3de10640 -1 osd.6 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:21.017791+0000 front 2024-08-19T14:50:21.017641+0000 (oldest deadline 2024-08-19T14:50:47.418155+0000) 2024-08-19T14:51:03.391 INFO:journalctl@ceph.osd.6.smithi032.stdout:Aug 19 14:51:03 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-6[35351]: 2024-08-19T14:51:03.024+0000 7f8d3de10640 -1 osd.6 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:32.019846+0000 front 2024-08-19T14:50:32.019867+0000 (oldest deadline 2024-08-19T14:50:56.119827+0000) 2024-08-19T14:51:03.758 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:51:03 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7[41448]: 2024-08-19T14:51:03.389+0000 7f1321f92640 -1 osd.7 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.778781+0000 front 2024-08-19T14:50:12.778916+0000 (oldest deadline 2024-08-19T14:50:41.579607+0000) 2024-08-19T14:51:03.758 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:51:03 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7[41448]: 2024-08-19T14:51:03.389+0000 7f1321f92640 -1 osd.7 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:21.580467+0000 front 2024-08-19T14:50:21.580200+0000 (oldest deadline 2024-08-19T14:50:46.280257+0000) 2024-08-19T14:51:03.758 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:51:03 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7[41448]: 2024-08-19T14:51:03.389+0000 7f1321f92640 -1 osd.7 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:29.682438+0000 front 2024-08-19T14:50:29.682226+0000 (oldest deadline 2024-08-19T14:50:58.482746+0000) 2024-08-19T14:51:03.796 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:03 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:03.363+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.724961+0000 front 2024-08-19T14:50:12.724727+0000 (oldest deadline 2024-08-19T14:50:36.224802+0000) 2024-08-19T14:51:03.796 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:03 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:03.363+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:20.627490+0000 front 2024-08-19T14:50:20.627552+0000 (oldest deadline 2024-08-19T14:50:46.527500+0000) 2024-08-19T14:51:03.797 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:03 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:03.363+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:30.330088+0000 front 2024-08-19T14:50:30.330151+0000 (oldest deadline 2024-08-19T14:50:53.230021+0000) 2024-08-19T14:51:03.797 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:03 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:03.553+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:13.297511+0000 front 2024-08-19T14:50:13.297834+0000 (oldest deadline 2024-08-19T14:50:37.997679+0000) 2024-08-19T14:51:03.797 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:03 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:03.553+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.298667+0000 front 2024-08-19T14:50:23.298816+0000 (oldest deadline 2024-08-19T14:50:49.198731+0000) 2024-08-19T14:51:03.797 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:03 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:03.553+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:29.199201+0000 front 2024-08-19T14:50:29.199233+0000 (oldest deadline 2024-08-19T14:50:54.499327+0000) 2024-08-19T14:51:04.296 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:51:04 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:51:04.094+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:14.239016+0000 front 2024-08-19T14:50:14.238817+0000 (oldest deadline 2024-08-19T14:50:38.339035+0000) 2024-08-19T14:51:04.296 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:51:04 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:51:04.094+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:22.841332+0000 front 2024-08-19T14:50:22.841705+0000 (oldest deadline 2024-08-19T14:50:48.741518+0000) 2024-08-19T14:51:04.297 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:03 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:03.854+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.842003+0000 front 2024-08-19T14:50:12.842211+0000 (oldest deadline 2024-08-19T14:50:36.242616+0000) 2024-08-19T14:51:04.297 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:03 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:03.854+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.744487+0000 front 2024-08-19T14:50:23.744453+0000 (oldest deadline 2024-08-19T14:50:47.844529+0000) 2024-08-19T14:51:04.297 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:03 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:03.854+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:31.345568+0000 front 2024-08-19T14:50:31.345651+0000 (oldest deadline 2024-08-19T14:50:57.745917+0000) 2024-08-19T14:51:04.297 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:03 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:03.854+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6830 osd.3 since back 2024-08-19T14:50:38.846810+0000 front 2024-08-19T14:50:38.846826+0000 (oldest deadline 2024-08-19T14:51:01.746744+0000) 2024-08-19T14:51:04.385 INFO:journalctl@ceph.osd.6.smithi032.stdout:Aug 19 14:51:04 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-6[35351]: 2024-08-19T14:51:04.042+0000 7f8d3de10640 -1 osd.6 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:13.516154+0000 front 2024-08-19T14:50:13.516301+0000 (oldest deadline 2024-08-19T14:50:36.416116+0000) 2024-08-19T14:51:04.385 INFO:journalctl@ceph.osd.6.smithi032.stdout:Aug 19 14:51:04 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-6[35351]: 2024-08-19T14:51:04.042+0000 7f8d3de10640 -1 osd.6 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:21.017791+0000 front 2024-08-19T14:50:21.017641+0000 (oldest deadline 2024-08-19T14:50:47.418155+0000) 2024-08-19T14:51:04.385 INFO:journalctl@ceph.osd.6.smithi032.stdout:Aug 19 14:51:04 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-6[35351]: 2024-08-19T14:51:04.042+0000 7f8d3de10640 -1 osd.6 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:32.019846+0000 front 2024-08-19T14:50:32.019867+0000 (oldest deadline 2024-08-19T14:50:56.119827+0000) 2024-08-19T14:51:04.385 INFO:journalctl@ceph.osd.6.smithi032.stdout:Aug 19 14:51:04 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-6[35351]: 2024-08-19T14:51:04.042+0000 7f8d3de10640 -1 osd.6 983 heartbeat_check: no reply from 172.21.15.31:6830 osd.3 since back 2024-08-19T14:50:39.020952+0000 front 2024-08-19T14:50:39.021097+0000 (oldest deadline 2024-08-19T14:51:03.121032+0000) 2024-08-19T14:51:04.757 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:51:04 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7[41448]: 2024-08-19T14:51:04.383+0000 7f1321f92640 -1 osd.7 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.778781+0000 front 2024-08-19T14:50:12.778916+0000 (oldest deadline 2024-08-19T14:50:41.579607+0000) 2024-08-19T14:51:04.758 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:51:04 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7[41448]: 2024-08-19T14:51:04.383+0000 7f1321f92640 -1 osd.7 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:21.580467+0000 front 2024-08-19T14:50:21.580200+0000 (oldest deadline 2024-08-19T14:50:46.280257+0000) 2024-08-19T14:51:04.758 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:51:04 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7[41448]: 2024-08-19T14:51:04.383+0000 7f1321f92640 -1 osd.7 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:29.682438+0000 front 2024-08-19T14:50:29.682226+0000 (oldest deadline 2024-08-19T14:50:58.482746+0000) 2024-08-19T14:51:04.758 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:51:04 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7[41448]: 2024-08-19T14:51:04.383+0000 7f1321f92640 -1 osd.7 983 heartbeat_check: no reply from 172.21.15.31:6830 osd.3 since back 2024-08-19T14:50:38.483316+0000 front 2024-08-19T14:50:38.483244+0000 (oldest deadline 2024-08-19T14:51:03.783258+0000) 2024-08-19T14:51:04.796 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:04 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:04.362+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.724961+0000 front 2024-08-19T14:50:12.724727+0000 (oldest deadline 2024-08-19T14:50:36.224802+0000) 2024-08-19T14:51:04.796 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:04 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:04.362+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:20.627490+0000 front 2024-08-19T14:50:20.627552+0000 (oldest deadline 2024-08-19T14:50:46.527500+0000) 2024-08-19T14:51:04.796 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:04 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:04.362+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:30.330088+0000 front 2024-08-19T14:50:30.330151+0000 (oldest deadline 2024-08-19T14:50:53.230021+0000) 2024-08-19T14:51:04.797 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:04 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:04.506+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:13.297511+0000 front 2024-08-19T14:50:13.297834+0000 (oldest deadline 2024-08-19T14:50:37.997679+0000) 2024-08-19T14:51:04.797 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:04 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:04.506+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.298667+0000 front 2024-08-19T14:50:23.298816+0000 (oldest deadline 2024-08-19T14:50:49.198731+0000) 2024-08-19T14:51:04.797 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:04 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:04.506+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:29.199201+0000 front 2024-08-19T14:50:29.199233+0000 (oldest deadline 2024-08-19T14:50:54.499327+0000) 2024-08-19T14:51:05.297 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:51:05 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:51:05.051+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:14.239016+0000 front 2024-08-19T14:50:14.238817+0000 (oldest deadline 2024-08-19T14:50:38.339035+0000) 2024-08-19T14:51:05.297 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:51:05 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:51:05.051+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:22.841332+0000 front 2024-08-19T14:50:22.841705+0000 (oldest deadline 2024-08-19T14:50:48.741518+0000) 2024-08-19T14:51:05.297 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:04 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:04.861+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.842003+0000 front 2024-08-19T14:50:12.842211+0000 (oldest deadline 2024-08-19T14:50:36.242616+0000) 2024-08-19T14:51:05.297 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:04 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:04.861+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.744487+0000 front 2024-08-19T14:50:23.744453+0000 (oldest deadline 2024-08-19T14:50:47.844529+0000) 2024-08-19T14:51:05.297 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:04 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:04.861+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:31.345568+0000 front 2024-08-19T14:50:31.345651+0000 (oldest deadline 2024-08-19T14:50:57.745917+0000) 2024-08-19T14:51:05.297 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:04 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:04.861+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6830 osd.3 since back 2024-08-19T14:50:38.846810+0000 front 2024-08-19T14:50:38.846826+0000 (oldest deadline 2024-08-19T14:51:01.746744+0000) 2024-08-19T14:51:05.337 INFO:journalctl@ceph.osd.6.smithi032.stdout:Aug 19 14:51:05 smithi032 podman[162618]: 2024-08-19 14:51:05.066733889 +0000 UTC m=+5.616175134 container died a8125b3c21bcc858d3c24f30da329d8f12eb3e7debedc015b9adcef457999697 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-6, GIT_CLEAN=True, RELEASE=squid-de09dd0, ceph=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-08-19T14:51:05.757 INFO:journalctl@ceph.osd.6.smithi032.stdout:Aug 19 14:51:05 smithi032 podman[162618]: 2024-08-19 14:51:05.436881728 +0000 UTC m=+5.986322970 container cleanup a8125b3c21bcc858d3c24f30da329d8f12eb3e7debedc015b9adcef457999697 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-6, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-de09dd0, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0) 2024-08-19T14:51:05.757 INFO:journalctl@ceph.osd.6.smithi032.stdout:Aug 19 14:51:05 smithi032 bash[162618]: ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-6 2024-08-19T14:51:05.758 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:51:05 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7[41448]: 2024-08-19T14:51:05.336+0000 7f1321f92640 -1 osd.7 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.778781+0000 front 2024-08-19T14:50:12.778916+0000 (oldest deadline 2024-08-19T14:50:41.579607+0000) 2024-08-19T14:51:05.758 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:51:05 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7[41448]: 2024-08-19T14:51:05.336+0000 7f1321f92640 -1 osd.7 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:21.580467+0000 front 2024-08-19T14:50:21.580200+0000 (oldest deadline 2024-08-19T14:50:46.280257+0000) 2024-08-19T14:51:05.758 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:51:05 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7[41448]: 2024-08-19T14:51:05.336+0000 7f1321f92640 -1 osd.7 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:29.682438+0000 front 2024-08-19T14:50:29.682226+0000 (oldest deadline 2024-08-19T14:50:58.482746+0000) 2024-08-19T14:51:05.758 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:51:05 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7[41448]: 2024-08-19T14:51:05.336+0000 7f1321f92640 -1 osd.7 983 heartbeat_check: no reply from 172.21.15.31:6830 osd.3 since back 2024-08-19T14:50:38.483316+0000 front 2024-08-19T14:50:38.483244+0000 (oldest deadline 2024-08-19T14:51:03.783258+0000) 2024-08-19T14:51:05.797 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:05 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:05.552+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:13.297511+0000 front 2024-08-19T14:50:13.297834+0000 (oldest deadline 2024-08-19T14:50:37.997679+0000) 2024-08-19T14:51:05.797 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:05 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:05.552+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.298667+0000 front 2024-08-19T14:50:23.298816+0000 (oldest deadline 2024-08-19T14:50:49.198731+0000) 2024-08-19T14:51:05.797 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:05 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:05.552+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:29.199201+0000 front 2024-08-19T14:50:29.199233+0000 (oldest deadline 2024-08-19T14:50:54.499327+0000) 2024-08-19T14:51:05.797 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:05 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:05.336+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.724961+0000 front 2024-08-19T14:50:12.724727+0000 (oldest deadline 2024-08-19T14:50:36.224802+0000) 2024-08-19T14:51:05.798 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:05 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:05.336+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:20.627490+0000 front 2024-08-19T14:50:20.627552+0000 (oldest deadline 2024-08-19T14:50:46.527500+0000) 2024-08-19T14:51:05.798 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:05 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:05.336+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:30.330088+0000 front 2024-08-19T14:50:30.330151+0000 (oldest deadline 2024-08-19T14:50:53.230021+0000) 2024-08-19T14:51:06.234 INFO:journalctl@ceph.osd.6.smithi032.stdout:Aug 19 14:51:05 smithi032 podman[162633]: 2024-08-19 14:51:05.85526793 +0000 UTC m=+0.778530216 container remove a8125b3c21bcc858d3c24f30da329d8f12eb3e7debedc015b9adcef457999697 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-6, GIT_CLEAN=True, RELEASE=squid-de09dd0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.schema-version=1.0) 2024-08-19T14:51:06.294 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:51:06 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:51:06.077+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:14.239016+0000 front 2024-08-19T14:50:14.238817+0000 (oldest deadline 2024-08-19T14:50:38.339035+0000) 2024-08-19T14:51:06.295 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:51:06 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:51:06.077+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:22.841332+0000 front 2024-08-19T14:50:22.841705+0000 (oldest deadline 2024-08-19T14:50:48.741518+0000) 2024-08-19T14:51:06.295 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:51:06 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:51:06.077+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.31:6830 osd.3 since back 2024-08-19T14:50:39.645003+0000 front 2024-08-19T14:50:39.645158+0000 (oldest deadline 2024-08-19T14:51:05.545008+0000) 2024-08-19T14:51:06.295 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:05 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:05.882+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.842003+0000 front 2024-08-19T14:50:12.842211+0000 (oldest deadline 2024-08-19T14:50:36.242616+0000) 2024-08-19T14:51:06.295 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:05 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:05.882+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.744487+0000 front 2024-08-19T14:50:23.744453+0000 (oldest deadline 2024-08-19T14:50:47.844529+0000) 2024-08-19T14:51:06.295 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:05 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:05.882+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:31.345568+0000 front 2024-08-19T14:50:31.345651+0000 (oldest deadline 2024-08-19T14:50:57.745917+0000) 2024-08-19T14:51:06.295 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:05 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:05.882+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6830 osd.3 since back 2024-08-19T14:50:38.846810+0000 front 2024-08-19T14:50:38.846826+0000 (oldest deadline 2024-08-19T14:51:01.746744+0000) 2024-08-19T14:51:06.507 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:51:06 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7[41448]: 2024-08-19T14:51:06.344+0000 7f1321f92640 -1 osd.7 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.778781+0000 front 2024-08-19T14:50:12.778916+0000 (oldest deadline 2024-08-19T14:50:41.579607+0000) 2024-08-19T14:51:06.508 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:51:06 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7[41448]: 2024-08-19T14:51:06.344+0000 7f1321f92640 -1 osd.7 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:21.580467+0000 front 2024-08-19T14:50:21.580200+0000 (oldest deadline 2024-08-19T14:50:46.280257+0000) 2024-08-19T14:51:06.508 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:51:06 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7[41448]: 2024-08-19T14:51:06.344+0000 7f1321f92640 -1 osd.7 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:29.682438+0000 front 2024-08-19T14:50:29.682226+0000 (oldest deadline 2024-08-19T14:50:58.482746+0000) 2024-08-19T14:51:06.508 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:51:06 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7[41448]: 2024-08-19T14:51:06.344+0000 7f1321f92640 -1 osd.7 983 heartbeat_check: no reply from 172.21.15.31:6830 osd.3 since back 2024-08-19T14:50:38.483316+0000 front 2024-08-19T14:50:38.483244+0000 (oldest deadline 2024-08-19T14:51:03.783258+0000) 2024-08-19T14:51:06.508 INFO:journalctl@ceph.osd.6.smithi032.stdout:Aug 19 14:51:06 smithi032 podman[162727]: 2024-08-19 14:51:06.136414898 +0000 UTC m=+0.017809820 image pull ea7a5189f30bad3aca9524788e2965d2368d7ce786df09d4edcd8c3edf9cefba quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9 2024-08-19T14:51:06.508 INFO:journalctl@ceph.osd.6.smithi032.stdout:Aug 19 14:51:06 smithi032 podman[162727]: 2024-08-19 14:51:06.247630356 +0000 UTC m=+0.129025278 container create 844eaff94f5a9c1a9896be2dce3c86eb7ead269479fdcb7e8da524b3b2a1d188 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-6-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, GIT_CLEAN=True, RELEASE=squid-de09dd0, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-08-19T14:51:06.508 INFO:journalctl@ceph.osd.6.smithi032.stdout:Aug 19 14:51:06 smithi032 podman[162727]: 2024-08-19 14:51:06.433908759 +0000 UTC m=+0.315303692 container init 844eaff94f5a9c1a9896be2dce3c86eb7ead269479fdcb7e8da524b3b2a1d188 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-6-deactivate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, RELEASE=squid-de09dd0) 2024-08-19T14:51:06.508 INFO:journalctl@ceph.osd.6.smithi032.stdout:Aug 19 14:51:06 smithi032 podman[162727]: 2024-08-19 14:51:06.437740276 +0000 UTC m=+0.319135203 container start 844eaff94f5a9c1a9896be2dce3c86eb7ead269479fdcb7e8da524b3b2a1d188 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-6-deactivate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, RELEASE=squid-de09dd0, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS) 2024-08-19T14:51:06.546 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:06 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:06.527+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:13.297511+0000 front 2024-08-19T14:50:13.297834+0000 (oldest deadline 2024-08-19T14:50:37.997679+0000) 2024-08-19T14:51:06.546 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:06 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:06.527+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.298667+0000 front 2024-08-19T14:50:23.298816+0000 (oldest deadline 2024-08-19T14:50:49.198731+0000) 2024-08-19T14:51:06.547 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:06 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:06.527+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:29.199201+0000 front 2024-08-19T14:50:29.199233+0000 (oldest deadline 2024-08-19T14:50:54.499327+0000) 2024-08-19T14:51:06.547 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:06 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:06.294+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.724961+0000 front 2024-08-19T14:50:12.724727+0000 (oldest deadline 2024-08-19T14:50:36.224802+0000) 2024-08-19T14:51:06.547 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:06 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:06.294+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:20.627490+0000 front 2024-08-19T14:50:20.627552+0000 (oldest deadline 2024-08-19T14:50:46.527500+0000) 2024-08-19T14:51:06.547 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:06 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:06.294+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:30.330088+0000 front 2024-08-19T14:50:30.330151+0000 (oldest deadline 2024-08-19T14:50:53.230021+0000) 2024-08-19T14:51:07.008 INFO:journalctl@ceph.osd.6.smithi032.stdout:Aug 19 14:51:06 smithi032 podman[162727]: 2024-08-19 14:51:06.524785066 +0000 UTC m=+0.406180003 container attach 844eaff94f5a9c1a9896be2dce3c86eb7ead269479fdcb7e8da524b3b2a1d188 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-6-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=squid-de09dd0, io.buildah.version=1.36.0, GIT_BRANCH=HEAD) 2024-08-19T14:51:07.008 INFO:journalctl@ceph.osd.6.smithi032.stdout:Aug 19 14:51:06 smithi032 podman[162727]: 2024-08-19 14:51:06.629718365 +0000 UTC m=+0.511113287 container died 844eaff94f5a9c1a9896be2dce3c86eb7ead269479fdcb7e8da524b3b2a1d188 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-6-deactivate, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, RELEASE=squid-de09dd0) 2024-08-19T14:51:07.261 INFO:journalctl@ceph.osd.6.smithi032.stdout:Aug 19 14:51:07 smithi032 podman[162751]: 2024-08-19 14:51:07.245221939 +0000 UTC m=+0.606399369 container remove 844eaff94f5a9c1a9896be2dce3c86eb7ead269479fdcb7e8da524b3b2a1d188 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-6-deactivate, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-de09dd0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True) 2024-08-19T14:51:07.267 DEBUG:teuthology.orchestra.run.smithi032:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968@osd.6.service' 2024-08-19T14:51:07.296 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:06 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:06.890+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.842003+0000 front 2024-08-19T14:50:12.842211+0000 (oldest deadline 2024-08-19T14:50:36.242616+0000) 2024-08-19T14:51:07.296 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:06 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:06.890+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.744487+0000 front 2024-08-19T14:50:23.744453+0000 (oldest deadline 2024-08-19T14:50:47.844529+0000) 2024-08-19T14:51:07.296 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:06 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:06.890+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:31.345568+0000 front 2024-08-19T14:50:31.345651+0000 (oldest deadline 2024-08-19T14:50:57.745917+0000) 2024-08-19T14:51:07.296 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:06 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:06.890+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6830 osd.3 since back 2024-08-19T14:50:38.846810+0000 front 2024-08-19T14:50:38.846826+0000 (oldest deadline 2024-08-19T14:51:01.746744+0000) 2024-08-19T14:51:07.297 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:51:07 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:51:07.032+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:14.239016+0000 front 2024-08-19T14:50:14.238817+0000 (oldest deadline 2024-08-19T14:50:38.339035+0000) 2024-08-19T14:51:07.297 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:51:07 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:51:07.032+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:22.841332+0000 front 2024-08-19T14:50:22.841705+0000 (oldest deadline 2024-08-19T14:50:48.741518+0000) 2024-08-19T14:51:07.297 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:51:07 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:51:07.032+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.31:6830 osd.3 since back 2024-08-19T14:50:39.645003+0000 front 2024-08-19T14:50:39.645158+0000 (oldest deadline 2024-08-19T14:51:05.545008+0000) 2024-08-19T14:51:07.307 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-19T14:51:07.307 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-08-19T14:51:07.307 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-08-19T14:51:07.308 DEBUG:teuthology.orchestra.run.smithi032:> sudo systemctl stop ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968@osd.7 2024-08-19T14:51:07.640 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:51:07 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7[41448]: 2024-08-19T14:51:07.328+0000 7f1321f92640 -1 osd.7 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.778781+0000 front 2024-08-19T14:50:12.778916+0000 (oldest deadline 2024-08-19T14:50:41.579607+0000) 2024-08-19T14:51:07.641 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:51:07 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7[41448]: 2024-08-19T14:51:07.328+0000 7f1321f92640 -1 osd.7 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:21.580467+0000 front 2024-08-19T14:50:21.580200+0000 (oldest deadline 2024-08-19T14:50:46.280257+0000) 2024-08-19T14:51:07.641 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:51:07 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7[41448]: 2024-08-19T14:51:07.328+0000 7f1321f92640 -1 osd.7 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:29.682438+0000 front 2024-08-19T14:50:29.682226+0000 (oldest deadline 2024-08-19T14:50:58.482746+0000) 2024-08-19T14:51:07.641 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:51:07 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7[41448]: 2024-08-19T14:51:07.328+0000 7f1321f92640 -1 osd.7 983 heartbeat_check: no reply from 172.21.15.31:6830 osd.3 since back 2024-08-19T14:50:38.483316+0000 front 2024-08-19T14:50:38.483244+0000 (oldest deadline 2024-08-19T14:51:03.783258+0000) 2024-08-19T14:51:07.641 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:51:07 smithi032 systemd[1]: Stopping Ceph osd.7 for ed7d1c6e-5e33-11ef-bccf-c7b262605968... 2024-08-19T14:51:07.796 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:07 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:07.552+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:13.297511+0000 front 2024-08-19T14:50:13.297834+0000 (oldest deadline 2024-08-19T14:50:37.997679+0000) 2024-08-19T14:51:07.797 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:07 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:07.552+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.298667+0000 front 2024-08-19T14:50:23.298816+0000 (oldest deadline 2024-08-19T14:50:49.198731+0000) 2024-08-19T14:51:07.797 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:07 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:07.552+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:29.199201+0000 front 2024-08-19T14:50:29.199233+0000 (oldest deadline 2024-08-19T14:50:54.499327+0000) 2024-08-19T14:51:07.797 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:07 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:07.342+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.724961+0000 front 2024-08-19T14:50:12.724727+0000 (oldest deadline 2024-08-19T14:50:36.224802+0000) 2024-08-19T14:51:07.797 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:07 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:07.342+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:20.627490+0000 front 2024-08-19T14:50:20.627552+0000 (oldest deadline 2024-08-19T14:50:46.527500+0000) 2024-08-19T14:51:07.797 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:07 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:07.342+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:30.330088+0000 front 2024-08-19T14:50:30.330151+0000 (oldest deadline 2024-08-19T14:50:53.230021+0000) 2024-08-19T14:51:08.007 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:51:07 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7[41448]: 2024-08-19T14:51:07.639+0000 7f1329984640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-19T14:51:08.008 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:51:07 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7[41448]: 2024-08-19T14:51:07.639+0000 7f1329984640 -1 osd.7 983 *** Got signal Terminated *** 2024-08-19T14:51:08.008 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:51:07 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7[41448]: 2024-08-19T14:51:07.639+0000 7f1329984640 -1 osd.7 983 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-08-19T14:51:08.296 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:07 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:07.858+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.842003+0000 front 2024-08-19T14:50:12.842211+0000 (oldest deadline 2024-08-19T14:50:36.242616+0000) 2024-08-19T14:51:08.296 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:07 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:07.858+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.744487+0000 front 2024-08-19T14:50:23.744453+0000 (oldest deadline 2024-08-19T14:50:47.844529+0000) 2024-08-19T14:51:08.297 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:07 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:07.858+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:31.345568+0000 front 2024-08-19T14:50:31.345651+0000 (oldest deadline 2024-08-19T14:50:57.745917+0000) 2024-08-19T14:51:08.297 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:07 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:07.858+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6830 osd.3 since back 2024-08-19T14:50:38.846810+0000 front 2024-08-19T14:50:38.846826+0000 (oldest deadline 2024-08-19T14:51:01.746744+0000) 2024-08-19T14:51:08.297 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:51:08 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:51:08.024+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:14.239016+0000 front 2024-08-19T14:50:14.238817+0000 (oldest deadline 2024-08-19T14:50:38.339035+0000) 2024-08-19T14:51:08.297 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:51:08 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:51:08.024+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:22.841332+0000 front 2024-08-19T14:50:22.841705+0000 (oldest deadline 2024-08-19T14:50:48.741518+0000) 2024-08-19T14:51:08.297 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:51:08 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:51:08.024+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.31:6830 osd.3 since back 2024-08-19T14:50:39.645003+0000 front 2024-08-19T14:50:39.645158+0000 (oldest deadline 2024-08-19T14:51:05.545008+0000) 2024-08-19T14:51:08.599 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:08 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:08.316+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.724961+0000 front 2024-08-19T14:50:12.724727+0000 (oldest deadline 2024-08-19T14:50:36.224802+0000) 2024-08-19T14:51:08.600 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:08 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:08.316+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:20.627490+0000 front 2024-08-19T14:50:20.627552+0000 (oldest deadline 2024-08-19T14:50:46.527500+0000) 2024-08-19T14:51:08.600 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:08 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:08.316+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:30.330088+0000 front 2024-08-19T14:50:30.330151+0000 (oldest deadline 2024-08-19T14:50:53.230021+0000) 2024-08-19T14:51:08.638 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:51:08 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7[41448]: 2024-08-19T14:51:08.314+0000 7f1321f92640 -1 osd.7 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.778781+0000 front 2024-08-19T14:50:12.778916+0000 (oldest deadline 2024-08-19T14:50:41.579607+0000) 2024-08-19T14:51:08.638 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:51:08 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7[41448]: 2024-08-19T14:51:08.314+0000 7f1321f92640 -1 osd.7 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:21.580467+0000 front 2024-08-19T14:50:21.580200+0000 (oldest deadline 2024-08-19T14:50:46.280257+0000) 2024-08-19T14:51:08.638 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:51:08 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7[41448]: 2024-08-19T14:51:08.314+0000 7f1321f92640 -1 osd.7 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:29.682438+0000 front 2024-08-19T14:50:29.682226+0000 (oldest deadline 2024-08-19T14:50:58.482746+0000) 2024-08-19T14:51:08.638 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:51:08 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7[41448]: 2024-08-19T14:51:08.314+0000 7f1321f92640 -1 osd.7 983 heartbeat_check: no reply from 172.21.15.31:6830 osd.3 since back 2024-08-19T14:50:38.483316+0000 front 2024-08-19T14:50:38.483244+0000 (oldest deadline 2024-08-19T14:51:03.783258+0000) 2024-08-19T14:51:08.859 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:08 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:08.598+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:13.297511+0000 front 2024-08-19T14:50:13.297834+0000 (oldest deadline 2024-08-19T14:50:37.997679+0000) 2024-08-19T14:51:08.859 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:08 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:08.598+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.298667+0000 front 2024-08-19T14:50:23.298816+0000 (oldest deadline 2024-08-19T14:50:49.198731+0000) 2024-08-19T14:51:08.859 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:08 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:08.598+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:29.199201+0000 front 2024-08-19T14:50:29.199233+0000 (oldest deadline 2024-08-19T14:50:54.499327+0000) 2024-08-19T14:51:09.296 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:08 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:08.859+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.842003+0000 front 2024-08-19T14:50:12.842211+0000 (oldest deadline 2024-08-19T14:50:36.242616+0000) 2024-08-19T14:51:09.296 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:08 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:08.859+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.744487+0000 front 2024-08-19T14:50:23.744453+0000 (oldest deadline 2024-08-19T14:50:47.844529+0000) 2024-08-19T14:51:09.297 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:08 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:08.859+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:31.345568+0000 front 2024-08-19T14:50:31.345651+0000 (oldest deadline 2024-08-19T14:50:57.745917+0000) 2024-08-19T14:51:09.297 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:08 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:08.859+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6830 osd.3 since back 2024-08-19T14:50:38.846810+0000 front 2024-08-19T14:50:38.846826+0000 (oldest deadline 2024-08-19T14:51:01.746744+0000) 2024-08-19T14:51:09.297 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:51:09 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:51:09.009+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:14.239016+0000 front 2024-08-19T14:50:14.238817+0000 (oldest deadline 2024-08-19T14:50:38.339035+0000) 2024-08-19T14:51:09.297 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:51:09 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:51:09.009+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:22.841332+0000 front 2024-08-19T14:50:22.841705+0000 (oldest deadline 2024-08-19T14:50:48.741518+0000) 2024-08-19T14:51:09.297 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:51:09 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:51:09.009+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.31:6830 osd.3 since back 2024-08-19T14:50:39.645003+0000 front 2024-08-19T14:50:39.645158+0000 (oldest deadline 2024-08-19T14:51:05.545008+0000) 2024-08-19T14:51:09.567 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:51:09 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7[41448]: 2024-08-19T14:51:09.302+0000 7f1321f92640 -1 osd.7 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.778781+0000 front 2024-08-19T14:50:12.778916+0000 (oldest deadline 2024-08-19T14:50:41.579607+0000) 2024-08-19T14:51:09.567 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:51:09 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7[41448]: 2024-08-19T14:51:09.302+0000 7f1321f92640 -1 osd.7 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:21.580467+0000 front 2024-08-19T14:50:21.580200+0000 (oldest deadline 2024-08-19T14:50:46.280257+0000) 2024-08-19T14:51:09.568 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:51:09 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7[41448]: 2024-08-19T14:51:09.302+0000 7f1321f92640 -1 osd.7 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:29.682438+0000 front 2024-08-19T14:50:29.682226+0000 (oldest deadline 2024-08-19T14:50:58.482746+0000) 2024-08-19T14:51:09.568 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:51:09 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7[41448]: 2024-08-19T14:51:09.302+0000 7f1321f92640 -1 osd.7 983 heartbeat_check: no reply from 172.21.15.31:6830 osd.3 since back 2024-08-19T14:50:38.483316+0000 front 2024-08-19T14:50:38.483244+0000 (oldest deadline 2024-08-19T14:51:03.783258+0000) 2024-08-19T14:51:09.586 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:09 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:09.327+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.724961+0000 front 2024-08-19T14:50:12.724727+0000 (oldest deadline 2024-08-19T14:50:36.224802+0000) 2024-08-19T14:51:09.586 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:09 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:09.327+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:20.627490+0000 front 2024-08-19T14:50:20.627552+0000 (oldest deadline 2024-08-19T14:50:46.527500+0000) 2024-08-19T14:51:09.586 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:09 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:09.327+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:30.330088+0000 front 2024-08-19T14:50:30.330151+0000 (oldest deadline 2024-08-19T14:50:53.230021+0000) 2024-08-19T14:51:09.863 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:09 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:09.584+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:13.297511+0000 front 2024-08-19T14:50:13.297834+0000 (oldest deadline 2024-08-19T14:50:37.997679+0000) 2024-08-19T14:51:09.863 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:09 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:09.584+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.298667+0000 front 2024-08-19T14:50:23.298816+0000 (oldest deadline 2024-08-19T14:50:49.198731+0000) 2024-08-19T14:51:09.863 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:09 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:09.584+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:29.199201+0000 front 2024-08-19T14:50:29.199233+0000 (oldest deadline 2024-08-19T14:50:54.499327+0000) 2024-08-19T14:51:10.296 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:09 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:09.862+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.842003+0000 front 2024-08-19T14:50:12.842211+0000 (oldest deadline 2024-08-19T14:50:36.242616+0000) 2024-08-19T14:51:10.296 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:09 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:09.862+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.744487+0000 front 2024-08-19T14:50:23.744453+0000 (oldest deadline 2024-08-19T14:50:47.844529+0000) 2024-08-19T14:51:10.297 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:09 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:09.862+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:31.345568+0000 front 2024-08-19T14:50:31.345651+0000 (oldest deadline 2024-08-19T14:50:57.745917+0000) 2024-08-19T14:51:10.297 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:09 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:09.862+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6830 osd.3 since back 2024-08-19T14:50:38.846810+0000 front 2024-08-19T14:50:38.846826+0000 (oldest deadline 2024-08-19T14:51:01.746744+0000) 2024-08-19T14:51:10.297 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:51:10 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:51:10.030+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:14.239016+0000 front 2024-08-19T14:50:14.238817+0000 (oldest deadline 2024-08-19T14:50:38.339035+0000) 2024-08-19T14:51:10.297 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:51:10 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:51:10.030+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:22.841332+0000 front 2024-08-19T14:50:22.841705+0000 (oldest deadline 2024-08-19T14:50:48.741518+0000) 2024-08-19T14:51:10.297 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:51:10 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:51:10.030+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.31:6830 osd.3 since back 2024-08-19T14:50:39.645003+0000 front 2024-08-19T14:50:39.645158+0000 (oldest deadline 2024-08-19T14:51:05.545008+0000) 2024-08-19T14:51:10.757 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:51:10 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7[41448]: 2024-08-19T14:51:10.310+0000 7f1321f92640 -1 osd.7 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.778781+0000 front 2024-08-19T14:50:12.778916+0000 (oldest deadline 2024-08-19T14:50:41.579607+0000) 2024-08-19T14:51:10.758 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:51:10 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7[41448]: 2024-08-19T14:51:10.310+0000 7f1321f92640 -1 osd.7 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:21.580467+0000 front 2024-08-19T14:50:21.580200+0000 (oldest deadline 2024-08-19T14:50:46.280257+0000) 2024-08-19T14:51:10.758 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:51:10 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7[41448]: 2024-08-19T14:51:10.310+0000 7f1321f92640 -1 osd.7 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:29.682438+0000 front 2024-08-19T14:50:29.682226+0000 (oldest deadline 2024-08-19T14:50:58.482746+0000) 2024-08-19T14:51:10.758 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:51:10 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7[41448]: 2024-08-19T14:51:10.310+0000 7f1321f92640 -1 osd.7 983 heartbeat_check: no reply from 172.21.15.31:6830 osd.3 since back 2024-08-19T14:50:38.483316+0000 front 2024-08-19T14:50:38.483244+0000 (oldest deadline 2024-08-19T14:51:03.783258+0000) 2024-08-19T14:51:10.758 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:51:10 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7[41448]: 2024-08-19T14:51:10.310+0000 7f1321f92640 -1 osd.7 983 heartbeat_check: no reply from 172.21.15.32:6804 osd.4 since back 2024-08-19T14:50:44.284202+0000 front 2024-08-19T14:50:44.284299+0000 (oldest deadline 2024-08-19T14:51:09.584195+0000) 2024-08-19T14:51:10.796 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:10 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:10.608+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:13.297511+0000 front 2024-08-19T14:50:13.297834+0000 (oldest deadline 2024-08-19T14:50:37.997679+0000) 2024-08-19T14:51:10.796 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:10 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:10.608+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.298667+0000 front 2024-08-19T14:50:23.298816+0000 (oldest deadline 2024-08-19T14:50:49.198731+0000) 2024-08-19T14:51:10.796 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:10 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:10.608+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:29.199201+0000 front 2024-08-19T14:50:29.199233+0000 (oldest deadline 2024-08-19T14:50:54.499327+0000) 2024-08-19T14:51:10.797 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:10 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:10.375+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.724961+0000 front 2024-08-19T14:50:12.724727+0000 (oldest deadline 2024-08-19T14:50:36.224802+0000) 2024-08-19T14:51:10.797 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:10 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:10.375+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:20.627490+0000 front 2024-08-19T14:50:20.627552+0000 (oldest deadline 2024-08-19T14:50:46.527500+0000) 2024-08-19T14:51:10.797 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:10 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:10.375+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:30.330088+0000 front 2024-08-19T14:50:30.330151+0000 (oldest deadline 2024-08-19T14:50:53.230021+0000) 2024-08-19T14:51:11.296 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:10 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:10.819+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.842003+0000 front 2024-08-19T14:50:12.842211+0000 (oldest deadline 2024-08-19T14:50:36.242616+0000) 2024-08-19T14:51:11.297 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:10 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:10.819+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.744487+0000 front 2024-08-19T14:50:23.744453+0000 (oldest deadline 2024-08-19T14:50:47.844529+0000) 2024-08-19T14:51:11.297 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:10 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:10.819+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:31.345568+0000 front 2024-08-19T14:50:31.345651+0000 (oldest deadline 2024-08-19T14:50:57.745917+0000) 2024-08-19T14:51:11.297 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:10 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:10.819+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6830 osd.3 since back 2024-08-19T14:50:38.846810+0000 front 2024-08-19T14:50:38.846826+0000 (oldest deadline 2024-08-19T14:51:01.746744+0000) 2024-08-19T14:51:11.297 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:51:11 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:51:11.052+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:14.239016+0000 front 2024-08-19T14:50:14.238817+0000 (oldest deadline 2024-08-19T14:50:38.339035+0000) 2024-08-19T14:51:11.297 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:51:11 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:51:11.052+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:22.841332+0000 front 2024-08-19T14:50:22.841705+0000 (oldest deadline 2024-08-19T14:50:48.741518+0000) 2024-08-19T14:51:11.297 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:51:11 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:51:11.052+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.31:6830 osd.3 since back 2024-08-19T14:50:39.645003+0000 front 2024-08-19T14:50:39.645158+0000 (oldest deadline 2024-08-19T14:51:05.545008+0000) 2024-08-19T14:51:11.297 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:51:11 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:51:11.052+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.32:6804 osd.4 since back 2024-08-19T14:50:45.545695+0000 front 2024-08-19T14:50:45.545767+0000 (oldest deadline 2024-08-19T14:51:10.845624+0000) 2024-08-19T14:51:11.757 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:51:11 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7[41448]: 2024-08-19T14:51:11.345+0000 7f1321f92640 -1 osd.7 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.778781+0000 front 2024-08-19T14:50:12.778916+0000 (oldest deadline 2024-08-19T14:50:41.579607+0000) 2024-08-19T14:51:11.757 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:51:11 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7[41448]: 2024-08-19T14:51:11.345+0000 7f1321f92640 -1 osd.7 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:21.580467+0000 front 2024-08-19T14:50:21.580200+0000 (oldest deadline 2024-08-19T14:50:46.280257+0000) 2024-08-19T14:51:11.758 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:51:11 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7[41448]: 2024-08-19T14:51:11.345+0000 7f1321f92640 -1 osd.7 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:29.682438+0000 front 2024-08-19T14:50:29.682226+0000 (oldest deadline 2024-08-19T14:50:58.482746+0000) 2024-08-19T14:51:11.758 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:51:11 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7[41448]: 2024-08-19T14:51:11.345+0000 7f1321f92640 -1 osd.7 983 heartbeat_check: no reply from 172.21.15.31:6830 osd.3 since back 2024-08-19T14:50:38.483316+0000 front 2024-08-19T14:50:38.483244+0000 (oldest deadline 2024-08-19T14:51:03.783258+0000) 2024-08-19T14:51:11.758 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:51:11 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7[41448]: 2024-08-19T14:51:11.345+0000 7f1321f92640 -1 osd.7 983 heartbeat_check: no reply from 172.21.15.32:6804 osd.4 since back 2024-08-19T14:50:44.284202+0000 front 2024-08-19T14:50:44.284299+0000 (oldest deadline 2024-08-19T14:51:09.584195+0000) 2024-08-19T14:51:11.787 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:11 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:11.392+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.724961+0000 front 2024-08-19T14:50:12.724727+0000 (oldest deadline 2024-08-19T14:50:36.224802+0000) 2024-08-19T14:51:11.787 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:11 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:11.392+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:20.627490+0000 front 2024-08-19T14:50:20.627552+0000 (oldest deadline 2024-08-19T14:50:46.527500+0000) 2024-08-19T14:51:11.787 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:11 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:11.392+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:30.330088+0000 front 2024-08-19T14:50:30.330151+0000 (oldest deadline 2024-08-19T14:50:53.230021+0000) 2024-08-19T14:51:11.787 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:11 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:11.588+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:13.297511+0000 front 2024-08-19T14:50:13.297834+0000 (oldest deadline 2024-08-19T14:50:37.997679+0000) 2024-08-19T14:51:11.787 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:11 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:11.588+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.298667+0000 front 2024-08-19T14:50:23.298816+0000 (oldest deadline 2024-08-19T14:50:49.198731+0000) 2024-08-19T14:51:11.788 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:11 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:11.588+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:29.199201+0000 front 2024-08-19T14:50:29.199233+0000 (oldest deadline 2024-08-19T14:50:54.499327+0000) 2024-08-19T14:51:11.788 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:11 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:11.588+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.32:6804 osd.4 since back 2024-08-19T14:50:47.901858+0000 front 2024-08-19T14:50:43.801431+0000 (oldest deadline 2024-08-19T14:51:11.401793+0000) 2024-08-19T14:51:12.046 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:11 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:11.786+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.842003+0000 front 2024-08-19T14:50:12.842211+0000 (oldest deadline 2024-08-19T14:50:36.242616+0000) 2024-08-19T14:51:12.046 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:11 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:11.786+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.744487+0000 front 2024-08-19T14:50:23.744453+0000 (oldest deadline 2024-08-19T14:50:47.844529+0000) 2024-08-19T14:51:12.046 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:11 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:11.786+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:31.345568+0000 front 2024-08-19T14:50:31.345651+0000 (oldest deadline 2024-08-19T14:50:57.745917+0000) 2024-08-19T14:51:12.046 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:11 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:11.786+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6830 osd.3 since back 2024-08-19T14:50:38.846810+0000 front 2024-08-19T14:50:38.846826+0000 (oldest deadline 2024-08-19T14:51:01.746744+0000) 2024-08-19T14:51:12.412 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:51:12 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:51:12.075+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:14.239016+0000 front 2024-08-19T14:50:14.238817+0000 (oldest deadline 2024-08-19T14:50:38.339035+0000) 2024-08-19T14:51:12.413 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:51:12 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:51:12.075+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:22.841332+0000 front 2024-08-19T14:50:22.841705+0000 (oldest deadline 2024-08-19T14:50:48.741518+0000) 2024-08-19T14:51:12.413 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:51:12 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:51:12.075+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.31:6830 osd.3 since back 2024-08-19T14:50:39.645003+0000 front 2024-08-19T14:50:39.645158+0000 (oldest deadline 2024-08-19T14:51:05.545008+0000) 2024-08-19T14:51:12.413 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:51:12 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:51:12.075+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.32:6804 osd.4 since back 2024-08-19T14:50:45.545695+0000 front 2024-08-19T14:50:45.545767+0000 (oldest deadline 2024-08-19T14:51:10.845624+0000) 2024-08-19T14:51:12.749 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:12 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:12.412+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.724961+0000 front 2024-08-19T14:50:12.724727+0000 (oldest deadline 2024-08-19T14:50:36.224802+0000) 2024-08-19T14:51:12.749 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:12 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:12.412+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:20.627490+0000 front 2024-08-19T14:50:20.627552+0000 (oldest deadline 2024-08-19T14:50:46.527500+0000) 2024-08-19T14:51:12.749 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:12 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:12.412+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:30.330088+0000 front 2024-08-19T14:50:30.330151+0000 (oldest deadline 2024-08-19T14:50:53.230021+0000) 2024-08-19T14:51:12.750 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:12 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:12.569+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:13.297511+0000 front 2024-08-19T14:50:13.297834+0000 (oldest deadline 2024-08-19T14:50:37.997679+0000) 2024-08-19T14:51:12.750 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:12 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:12.569+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.298667+0000 front 2024-08-19T14:50:23.298816+0000 (oldest deadline 2024-08-19T14:50:49.198731+0000) 2024-08-19T14:51:12.750 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:12 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:12.569+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:29.199201+0000 front 2024-08-19T14:50:29.199233+0000 (oldest deadline 2024-08-19T14:50:54.499327+0000) 2024-08-19T14:51:12.750 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:12 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:12.569+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.32:6804 osd.4 since back 2024-08-19T14:50:47.901858+0000 front 2024-08-19T14:50:43.801431+0000 (oldest deadline 2024-08-19T14:51:11.401793+0000) 2024-08-19T14:51:12.757 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:51:12 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7[41448]: 2024-08-19T14:51:12.350+0000 7f1321f92640 -1 osd.7 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.778781+0000 front 2024-08-19T14:50:12.778916+0000 (oldest deadline 2024-08-19T14:50:41.579607+0000) 2024-08-19T14:51:12.757 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:51:12 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7[41448]: 2024-08-19T14:51:12.350+0000 7f1321f92640 -1 osd.7 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:21.580467+0000 front 2024-08-19T14:50:21.580200+0000 (oldest deadline 2024-08-19T14:50:46.280257+0000) 2024-08-19T14:51:12.757 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:51:12 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7[41448]: 2024-08-19T14:51:12.350+0000 7f1321f92640 -1 osd.7 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:29.682438+0000 front 2024-08-19T14:50:29.682226+0000 (oldest deadline 2024-08-19T14:50:58.482746+0000) 2024-08-19T14:51:12.758 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:51:12 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7[41448]: 2024-08-19T14:51:12.350+0000 7f1321f92640 -1 osd.7 983 heartbeat_check: no reply from 172.21.15.31:6830 osd.3 since back 2024-08-19T14:50:38.483316+0000 front 2024-08-19T14:50:38.483244+0000 (oldest deadline 2024-08-19T14:51:03.783258+0000) 2024-08-19T14:51:12.758 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:51:12 smithi032 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7[41448]: 2024-08-19T14:51:12.350+0000 7f1321f92640 -1 osd.7 983 heartbeat_check: no reply from 172.21.15.32:6804 osd.4 since back 2024-08-19T14:50:44.284202+0000 front 2024-08-19T14:50:44.284299+0000 (oldest deadline 2024-08-19T14:51:09.584195+0000) 2024-08-19T14:51:13.046 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:12 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:12.749+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.842003+0000 front 2024-08-19T14:50:12.842211+0000 (oldest deadline 2024-08-19T14:50:36.242616+0000) 2024-08-19T14:51:13.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:12 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:12.749+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.744487+0000 front 2024-08-19T14:50:23.744453+0000 (oldest deadline 2024-08-19T14:50:47.844529+0000) 2024-08-19T14:51:13.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:12 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:12.749+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:31.345568+0000 front 2024-08-19T14:50:31.345651+0000 (oldest deadline 2024-08-19T14:50:57.745917+0000) 2024-08-19T14:51:13.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:12 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:12.749+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6830 osd.3 since back 2024-08-19T14:50:38.846810+0000 front 2024-08-19T14:50:38.846826+0000 (oldest deadline 2024-08-19T14:51:01.746744+0000) 2024-08-19T14:51:13.366 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:51:13 smithi032 podman[162830]: 2024-08-19 14:51:13.099640111 +0000 UTC m=+5.562671934 container died d3c63b335bac8cdbbcb1e2a1a61c5b4880672814e262b201aaf72414fb284b1b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, RELEASE=squid-de09dd0, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-08-19T14:51:13.463 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:51:13 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:51:13.047+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:14.239016+0000 front 2024-08-19T14:50:14.238817+0000 (oldest deadline 2024-08-19T14:50:38.339035+0000) 2024-08-19T14:51:13.463 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:51:13 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:51:13.047+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:22.841332+0000 front 2024-08-19T14:50:22.841705+0000 (oldest deadline 2024-08-19T14:50:48.741518+0000) 2024-08-19T14:51:13.463 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:51:13 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:51:13.047+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.31:6830 osd.3 since back 2024-08-19T14:50:39.645003+0000 front 2024-08-19T14:50:39.645158+0000 (oldest deadline 2024-08-19T14:51:05.545008+0000) 2024-08-19T14:51:13.463 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:51:13 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:51:13.047+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.32:6804 osd.4 since back 2024-08-19T14:50:45.545695+0000 front 2024-08-19T14:50:45.545767+0000 (oldest deadline 2024-08-19T14:51:10.845624+0000) 2024-08-19T14:51:13.757 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:51:13 smithi032 podman[162830]: 2024-08-19 14:51:13.590342338 +0000 UTC m=+6.053374153 container cleanup d3c63b335bac8cdbbcb1e2a1a61c5b4880672814e262b201aaf72414fb284b1b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=squid-de09dd0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-08-19T14:51:13.757 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:51:13 smithi032 bash[162830]: ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7 2024-08-19T14:51:13.758 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:13 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:13.462+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.724961+0000 front 2024-08-19T14:50:12.724727+0000 (oldest deadline 2024-08-19T14:50:36.224802+0000) 2024-08-19T14:51:13.758 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:13 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:13.462+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:20.627490+0000 front 2024-08-19T14:50:20.627552+0000 (oldest deadline 2024-08-19T14:50:46.527500+0000) 2024-08-19T14:51:13.758 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:13 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:13.462+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:30.330088+0000 front 2024-08-19T14:50:30.330151+0000 (oldest deadline 2024-08-19T14:50:53.230021+0000) 2024-08-19T14:51:13.758 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:13 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:13.462+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.32:6804 osd.4 since back 2024-08-19T14:50:47.232484+0000 front 2024-08-19T14:50:47.232609+0000 (oldest deadline 2024-08-19T14:51:13.132617+0000) 2024-08-19T14:51:13.758 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:13 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:13.544+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:13.297511+0000 front 2024-08-19T14:50:13.297834+0000 (oldest deadline 2024-08-19T14:50:37.997679+0000) 2024-08-19T14:51:13.759 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:13 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:13.544+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.298667+0000 front 2024-08-19T14:50:23.298816+0000 (oldest deadline 2024-08-19T14:50:49.198731+0000) 2024-08-19T14:51:13.759 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:13 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:13.544+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:29.199201+0000 front 2024-08-19T14:50:29.199233+0000 (oldest deadline 2024-08-19T14:50:54.499327+0000) 2024-08-19T14:51:13.759 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:13 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:13.544+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.32:6804 osd.4 since back 2024-08-19T14:50:47.901858+0000 front 2024-08-19T14:50:43.801431+0000 (oldest deadline 2024-08-19T14:51:11.401793+0000) 2024-08-19T14:51:14.046 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:13 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:13.757+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.842003+0000 front 2024-08-19T14:50:12.842211+0000 (oldest deadline 2024-08-19T14:50:36.242616+0000) 2024-08-19T14:51:14.046 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:13 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:13.757+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.744487+0000 front 2024-08-19T14:50:23.744453+0000 (oldest deadline 2024-08-19T14:50:47.844529+0000) 2024-08-19T14:51:14.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:13 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:13.757+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:31.345568+0000 front 2024-08-19T14:50:31.345651+0000 (oldest deadline 2024-08-19T14:50:57.745917+0000) 2024-08-19T14:51:14.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:13 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:13.757+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6830 osd.3 since back 2024-08-19T14:50:38.846810+0000 front 2024-08-19T14:50:38.846826+0000 (oldest deadline 2024-08-19T14:51:01.746744+0000) 2024-08-19T14:51:14.180 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:51:13 smithi032 podman[162859]: 2024-08-19 14:51:13.858119764 +0000 UTC m=+0.760913608 container remove d3c63b335bac8cdbbcb1e2a1a61c5b4880672814e262b201aaf72414fb284b1b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7, GIT_CLEAN=True, ceph=True, RELEASE=squid-de09dd0, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-08-19T14:51:14.433 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:51:14 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:51:14.078+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:14.239016+0000 front 2024-08-19T14:50:14.238817+0000 (oldest deadline 2024-08-19T14:50:38.339035+0000) 2024-08-19T14:51:14.434 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:51:14 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:51:14.078+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:22.841332+0000 front 2024-08-19T14:50:22.841705+0000 (oldest deadline 2024-08-19T14:50:48.741518+0000) 2024-08-19T14:51:14.434 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:51:14 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:51:14.078+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.31:6830 osd.3 since back 2024-08-19T14:50:39.645003+0000 front 2024-08-19T14:50:39.645158+0000 (oldest deadline 2024-08-19T14:51:05.545008+0000) 2024-08-19T14:51:14.434 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:51:14 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:51:14.078+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.32:6804 osd.4 since back 2024-08-19T14:50:45.545695+0000 front 2024-08-19T14:50:45.545767+0000 (oldest deadline 2024-08-19T14:51:10.845624+0000) 2024-08-19T14:51:14.478 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:51:14 smithi032 podman[162945]: 2024-08-19 14:51:14.082459257 +0000 UTC m=+0.017208221 image pull ea7a5189f30bad3aca9524788e2965d2368d7ce786df09d4edcd8c3edf9cefba quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9 2024-08-19T14:51:14.478 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:51:14 smithi032 podman[162945]: 2024-08-19 14:51:14.183766235 +0000 UTC m=+0.118515189 container create 0ff19ed84c02e717b88cb1e33c9762b6486174c42288c3c0056603b2566b9dd0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-de09dd0, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-08-19T14:51:14.478 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:51:14 smithi032 podman[162945]: 2024-08-19 14:51:14.402988393 +0000 UTC m=+0.337737361 container init 0ff19ed84c02e717b88cb1e33c9762b6486174c42288c3c0056603b2566b9dd0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=squid-de09dd0) 2024-08-19T14:51:14.478 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:51:14 smithi032 podman[162945]: 2024-08-19 14:51:14.406355476 +0000 UTC m=+0.341104431 container start 0ff19ed84c02e717b88cb1e33c9762b6486174c42288c3c0056603b2566b9dd0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=squid-de09dd0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731) 2024-08-19T14:51:14.478 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:51:14 smithi032 podman[162945]: 2024-08-19 14:51:14.477850893 +0000 UTC m=+0.412599878 container attach 0ff19ed84c02e717b88cb1e33c9762b6486174c42288c3c0056603b2566b9dd0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7-deactivate, RELEASE=squid-de09dd0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True) 2024-08-19T14:51:14.738 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:14 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:14.432+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.724961+0000 front 2024-08-19T14:50:12.724727+0000 (oldest deadline 2024-08-19T14:50:36.224802+0000) 2024-08-19T14:51:14.739 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:14 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:14.432+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:20.627490+0000 front 2024-08-19T14:50:20.627552+0000 (oldest deadline 2024-08-19T14:50:46.527500+0000) 2024-08-19T14:51:14.739 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:14 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:14.432+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:30.330088+0000 front 2024-08-19T14:50:30.330151+0000 (oldest deadline 2024-08-19T14:50:53.230021+0000) 2024-08-19T14:51:14.739 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:14 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:14.432+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.32:6804 osd.4 since back 2024-08-19T14:50:47.232484+0000 front 2024-08-19T14:50:47.232609+0000 (oldest deadline 2024-08-19T14:51:13.132617+0000) 2024-08-19T14:51:14.739 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:14 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:14.523+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:13.297511+0000 front 2024-08-19T14:50:13.297834+0000 (oldest deadline 2024-08-19T14:50:37.997679+0000) 2024-08-19T14:51:14.739 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:14 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:14.523+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.298667+0000 front 2024-08-19T14:50:23.298816+0000 (oldest deadline 2024-08-19T14:50:49.198731+0000) 2024-08-19T14:51:14.739 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:14 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:14.523+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:29.199201+0000 front 2024-08-19T14:50:29.199233+0000 (oldest deadline 2024-08-19T14:50:54.499327+0000) 2024-08-19T14:51:14.739 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:14 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:14.523+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.32:6804 osd.4 since back 2024-08-19T14:50:47.901858+0000 front 2024-08-19T14:50:43.801431+0000 (oldest deadline 2024-08-19T14:51:11.401793+0000) 2024-08-19T14:51:14.758 INFO:journalctl@ceph.osd.7.smithi032.stdout:Aug 19 14:51:14 smithi032 podman[162945]: 2024-08-19 14:51:14.597395209 +0000 UTC m=+0.532144163 container died 0ff19ed84c02e717b88cb1e33c9762b6486174c42288c3c0056603b2566b9dd0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-de09dd0, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-08-19T14:51:15.046 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:14 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:14.737+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.842003+0000 front 2024-08-19T14:50:12.842211+0000 (oldest deadline 2024-08-19T14:50:36.242616+0000) 2024-08-19T14:51:15.046 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:14 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:14.737+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.744487+0000 front 2024-08-19T14:50:23.744453+0000 (oldest deadline 2024-08-19T14:50:47.844529+0000) 2024-08-19T14:51:15.046 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:14 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:14.737+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:31.345568+0000 front 2024-08-19T14:50:31.345651+0000 (oldest deadline 2024-08-19T14:50:57.745917+0000) 2024-08-19T14:51:15.046 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:14 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:14.737+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6830 osd.3 since back 2024-08-19T14:50:38.846810+0000 front 2024-08-19T14:50:38.846826+0000 (oldest deadline 2024-08-19T14:51:01.746744+0000) 2024-08-19T14:51:15.308 DEBUG:teuthology.orchestra.run.smithi032:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968@osd.7.service' 2024-08-19T14:51:15.349 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-19T14:51:15.349 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-08-19T14:51:15.349 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-08-19T14:51:15.349 DEBUG:teuthology.orchestra.run.smithi045:> sudo systemctl stop ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968@osd.8 2024-08-19T14:51:15.377 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:51:15 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:51:15.080+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:14.239016+0000 front 2024-08-19T14:50:14.238817+0000 (oldest deadline 2024-08-19T14:50:38.339035+0000) 2024-08-19T14:51:15.377 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:51:15 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:51:15.080+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:22.841332+0000 front 2024-08-19T14:50:22.841705+0000 (oldest deadline 2024-08-19T14:50:48.741518+0000) 2024-08-19T14:51:15.377 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:51:15 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:51:15.080+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.31:6830 osd.3 since back 2024-08-19T14:50:39.645003+0000 front 2024-08-19T14:50:39.645158+0000 (oldest deadline 2024-08-19T14:51:05.545008+0000) 2024-08-19T14:51:15.377 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:51:15 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:51:15.080+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.32:6804 osd.4 since back 2024-08-19T14:50:45.545695+0000 front 2024-08-19T14:50:45.545767+0000 (oldest deadline 2024-08-19T14:51:10.845624+0000) 2024-08-19T14:51:15.669 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:51:15 smithi045 systemd[1]: Stopping Ceph osd.8 for ed7d1c6e-5e33-11ef-bccf-c7b262605968... 2024-08-19T14:51:15.669 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:51:15 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:51:15.668+0000 7f99548dc640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-19T14:51:15.670 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:15 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:15.551+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:13.297511+0000 front 2024-08-19T14:50:13.297834+0000 (oldest deadline 2024-08-19T14:50:37.997679+0000) 2024-08-19T14:51:15.670 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:15 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:15.551+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.298667+0000 front 2024-08-19T14:50:23.298816+0000 (oldest deadline 2024-08-19T14:50:49.198731+0000) 2024-08-19T14:51:15.670 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:15 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:15.551+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:29.199201+0000 front 2024-08-19T14:50:29.199233+0000 (oldest deadline 2024-08-19T14:50:54.499327+0000) 2024-08-19T14:51:15.671 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:15 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:15.551+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.32:6804 osd.4 since back 2024-08-19T14:50:47.901858+0000 front 2024-08-19T14:50:43.801431+0000 (oldest deadline 2024-08-19T14:51:11.401793+0000) 2024-08-19T14:51:15.671 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:15 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:15.458+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.724961+0000 front 2024-08-19T14:50:12.724727+0000 (oldest deadline 2024-08-19T14:50:36.224802+0000) 2024-08-19T14:51:15.671 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:15 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:15.458+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:20.627490+0000 front 2024-08-19T14:50:20.627552+0000 (oldest deadline 2024-08-19T14:50:46.527500+0000) 2024-08-19T14:51:15.671 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:15 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:15.458+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:30.330088+0000 front 2024-08-19T14:50:30.330151+0000 (oldest deadline 2024-08-19T14:50:53.230021+0000) 2024-08-19T14:51:15.672 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:15 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:15.458+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.32:6804 osd.4 since back 2024-08-19T14:50:47.232484+0000 front 2024-08-19T14:50:47.232609+0000 (oldest deadline 2024-08-19T14:51:13.132617+0000) 2024-08-19T14:51:16.047 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:51:15 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:51:15.668+0000 7f99548dc640 -1 osd.8 983 *** Got signal Terminated *** 2024-08-19T14:51:16.047 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:51:15 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:51:15.668+0000 7f99548dc640 -1 osd.8 983 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-08-19T14:51:16.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:15 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:15.728+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.842003+0000 front 2024-08-19T14:50:12.842211+0000 (oldest deadline 2024-08-19T14:50:36.242616+0000) 2024-08-19T14:51:16.048 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:15 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:15.728+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.744487+0000 front 2024-08-19T14:50:23.744453+0000 (oldest deadline 2024-08-19T14:50:47.844529+0000) 2024-08-19T14:51:16.048 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:15 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:15.728+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:31.345568+0000 front 2024-08-19T14:50:31.345651+0000 (oldest deadline 2024-08-19T14:50:57.745917+0000) 2024-08-19T14:51:16.048 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:15 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:15.728+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6830 osd.3 since back 2024-08-19T14:50:38.846810+0000 front 2024-08-19T14:50:38.846826+0000 (oldest deadline 2024-08-19T14:51:01.746744+0000) 2024-08-19T14:51:16.434 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:51:16 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:51:16.117+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:14.239016+0000 front 2024-08-19T14:50:14.238817+0000 (oldest deadline 2024-08-19T14:50:38.339035+0000) 2024-08-19T14:51:16.434 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:51:16 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:51:16.117+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:22.841332+0000 front 2024-08-19T14:50:22.841705+0000 (oldest deadline 2024-08-19T14:50:48.741518+0000) 2024-08-19T14:51:16.434 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:51:16 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:51:16.117+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.31:6830 osd.3 since back 2024-08-19T14:50:39.645003+0000 front 2024-08-19T14:50:39.645158+0000 (oldest deadline 2024-08-19T14:51:05.545008+0000) 2024-08-19T14:51:16.434 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:51:16 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:51:16.117+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.32:6804 osd.4 since back 2024-08-19T14:50:45.545695+0000 front 2024-08-19T14:50:45.545767+0000 (oldest deadline 2024-08-19T14:51:10.845624+0000) 2024-08-19T14:51:16.688 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:16 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:16.433+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.724961+0000 front 2024-08-19T14:50:12.724727+0000 (oldest deadline 2024-08-19T14:50:36.224802+0000) 2024-08-19T14:51:16.688 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:16 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:16.433+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:20.627490+0000 front 2024-08-19T14:50:20.627552+0000 (oldest deadline 2024-08-19T14:50:46.527500+0000) 2024-08-19T14:51:16.688 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:16 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:16.433+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:30.330088+0000 front 2024-08-19T14:50:30.330151+0000 (oldest deadline 2024-08-19T14:50:53.230021+0000) 2024-08-19T14:51:16.688 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:16 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:16.433+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.32:6804 osd.4 since back 2024-08-19T14:50:47.232484+0000 front 2024-08-19T14:50:47.232609+0000 (oldest deadline 2024-08-19T14:51:13.132617+0000) 2024-08-19T14:51:16.689 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:16 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:16.576+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:13.297511+0000 front 2024-08-19T14:50:13.297834+0000 (oldest deadline 2024-08-19T14:50:37.997679+0000) 2024-08-19T14:51:16.689 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:16 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:16.576+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.298667+0000 front 2024-08-19T14:50:23.298816+0000 (oldest deadline 2024-08-19T14:50:49.198731+0000) 2024-08-19T14:51:16.689 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:16 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:16.576+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:29.199201+0000 front 2024-08-19T14:50:29.199233+0000 (oldest deadline 2024-08-19T14:50:54.499327+0000) 2024-08-19T14:51:16.689 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:16 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:16.576+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.32:6804 osd.4 since back 2024-08-19T14:50:47.901858+0000 front 2024-08-19T14:50:43.801431+0000 (oldest deadline 2024-08-19T14:51:11.401793+0000) 2024-08-19T14:51:17.046 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:16 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:16.687+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.842003+0000 front 2024-08-19T14:50:12.842211+0000 (oldest deadline 2024-08-19T14:50:36.242616+0000) 2024-08-19T14:51:17.046 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:16 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:16.687+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.744487+0000 front 2024-08-19T14:50:23.744453+0000 (oldest deadline 2024-08-19T14:50:47.844529+0000) 2024-08-19T14:51:17.046 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:16 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:16.687+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:31.345568+0000 front 2024-08-19T14:50:31.345651+0000 (oldest deadline 2024-08-19T14:50:57.745917+0000) 2024-08-19T14:51:17.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:16 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:16.687+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6830 osd.3 since back 2024-08-19T14:50:38.846810+0000 front 2024-08-19T14:50:38.846826+0000 (oldest deadline 2024-08-19T14:51:01.746744+0000) 2024-08-19T14:51:17.439 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:51:17 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:51:17.132+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:14.239016+0000 front 2024-08-19T14:50:14.238817+0000 (oldest deadline 2024-08-19T14:50:38.339035+0000) 2024-08-19T14:51:17.440 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:51:17 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:51:17.132+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:22.841332+0000 front 2024-08-19T14:50:22.841705+0000 (oldest deadline 2024-08-19T14:50:48.741518+0000) 2024-08-19T14:51:17.440 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:51:17 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:51:17.132+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.31:6830 osd.3 since back 2024-08-19T14:50:39.645003+0000 front 2024-08-19T14:50:39.645158+0000 (oldest deadline 2024-08-19T14:51:05.545008+0000) 2024-08-19T14:51:17.440 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:51:17 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:51:17.132+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.32:6804 osd.4 since back 2024-08-19T14:50:45.545695+0000 front 2024-08-19T14:50:45.545767+0000 (oldest deadline 2024-08-19T14:51:10.845624+0000) 2024-08-19T14:51:17.796 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:17 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:17.612+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:13.297511+0000 front 2024-08-19T14:50:13.297834+0000 (oldest deadline 2024-08-19T14:50:37.997679+0000) 2024-08-19T14:51:17.797 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:17 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:17.612+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.298667+0000 front 2024-08-19T14:50:23.298816+0000 (oldest deadline 2024-08-19T14:50:49.198731+0000) 2024-08-19T14:51:17.797 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:17 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:17.612+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:29.199201+0000 front 2024-08-19T14:50:29.199233+0000 (oldest deadline 2024-08-19T14:50:54.499327+0000) 2024-08-19T14:51:17.797 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:17 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:17.612+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.32:6804 osd.4 since back 2024-08-19T14:50:47.901858+0000 front 2024-08-19T14:50:43.801431+0000 (oldest deadline 2024-08-19T14:51:11.401793+0000) 2024-08-19T14:51:17.797 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:17 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:17.647+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.842003+0000 front 2024-08-19T14:50:12.842211+0000 (oldest deadline 2024-08-19T14:50:36.242616+0000) 2024-08-19T14:51:17.797 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:17 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:17.647+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.744487+0000 front 2024-08-19T14:50:23.744453+0000 (oldest deadline 2024-08-19T14:50:47.844529+0000) 2024-08-19T14:51:17.797 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:17 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:17.647+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:31.345568+0000 front 2024-08-19T14:50:31.345651+0000 (oldest deadline 2024-08-19T14:50:57.745917+0000) 2024-08-19T14:51:17.797 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:17 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:17.647+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6830 osd.3 since back 2024-08-19T14:50:38.846810+0000 front 2024-08-19T14:50:38.846826+0000 (oldest deadline 2024-08-19T14:51:01.746744+0000) 2024-08-19T14:51:17.798 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:17 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:17.439+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.724961+0000 front 2024-08-19T14:50:12.724727+0000 (oldest deadline 2024-08-19T14:50:36.224802+0000) 2024-08-19T14:51:17.798 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:17 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:17.439+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:20.627490+0000 front 2024-08-19T14:50:20.627552+0000 (oldest deadline 2024-08-19T14:50:46.527500+0000) 2024-08-19T14:51:17.798 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:17 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:17.439+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:30.330088+0000 front 2024-08-19T14:50:30.330151+0000 (oldest deadline 2024-08-19T14:50:53.230021+0000) 2024-08-19T14:51:17.798 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:17 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:17.439+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.32:6804 osd.4 since back 2024-08-19T14:50:47.232484+0000 front 2024-08-19T14:50:47.232609+0000 (oldest deadline 2024-08-19T14:51:13.132617+0000) 2024-08-19T14:51:18.453 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:51:18 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:51:18.095+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:14.239016+0000 front 2024-08-19T14:50:14.238817+0000 (oldest deadline 2024-08-19T14:50:38.339035+0000) 2024-08-19T14:51:18.453 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:51:18 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:51:18.095+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:22.841332+0000 front 2024-08-19T14:50:22.841705+0000 (oldest deadline 2024-08-19T14:50:48.741518+0000) 2024-08-19T14:51:18.453 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:51:18 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:51:18.095+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.31:6830 osd.3 since back 2024-08-19T14:50:39.645003+0000 front 2024-08-19T14:50:39.645158+0000 (oldest deadline 2024-08-19T14:51:05.545008+0000) 2024-08-19T14:51:18.453 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:51:18 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:51:18.095+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.32:6804 osd.4 since back 2024-08-19T14:50:45.545695+0000 front 2024-08-19T14:50:45.545767+0000 (oldest deadline 2024-08-19T14:51:10.845624+0000) 2024-08-19T14:51:18.797 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:18 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:18.452+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.724961+0000 front 2024-08-19T14:50:12.724727+0000 (oldest deadline 2024-08-19T14:50:36.224802+0000) 2024-08-19T14:51:18.797 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:18 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:18.452+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:20.627490+0000 front 2024-08-19T14:50:20.627552+0000 (oldest deadline 2024-08-19T14:50:46.527500+0000) 2024-08-19T14:51:18.797 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:18 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:18.452+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:30.330088+0000 front 2024-08-19T14:50:30.330151+0000 (oldest deadline 2024-08-19T14:50:53.230021+0000) 2024-08-19T14:51:18.797 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:18 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:18.452+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.32:6804 osd.4 since back 2024-08-19T14:50:47.232484+0000 front 2024-08-19T14:50:47.232609+0000 (oldest deadline 2024-08-19T14:51:13.132617+0000) 2024-08-19T14:51:18.798 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:18 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:18.452+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.32:6812 osd.5 since back 2024-08-19T14:50:54.233709+0000 front 2024-08-19T14:50:54.233586+0000 (oldest deadline 2024-08-19T14:51:17.733482+0000) 2024-08-19T14:51:18.798 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:18 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:18.618+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:13.297511+0000 front 2024-08-19T14:50:13.297834+0000 (oldest deadline 2024-08-19T14:50:37.997679+0000) 2024-08-19T14:51:18.798 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:18 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:18.618+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.298667+0000 front 2024-08-19T14:50:23.298816+0000 (oldest deadline 2024-08-19T14:50:49.198731+0000) 2024-08-19T14:51:18.799 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:18 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:18.618+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:29.199201+0000 front 2024-08-19T14:50:29.199233+0000 (oldest deadline 2024-08-19T14:50:54.499327+0000) 2024-08-19T14:51:18.799 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:18 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:18.618+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.32:6804 osd.4 since back 2024-08-19T14:50:47.901858+0000 front 2024-08-19T14:50:43.801431+0000 (oldest deadline 2024-08-19T14:51:11.401793+0000) 2024-08-19T14:51:18.799 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:18 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:18.689+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.842003+0000 front 2024-08-19T14:50:12.842211+0000 (oldest deadline 2024-08-19T14:50:36.242616+0000) 2024-08-19T14:51:18.799 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:18 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:18.689+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.744487+0000 front 2024-08-19T14:50:23.744453+0000 (oldest deadline 2024-08-19T14:50:47.844529+0000) 2024-08-19T14:51:18.800 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:18 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:18.689+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:31.345568+0000 front 2024-08-19T14:50:31.345651+0000 (oldest deadline 2024-08-19T14:50:57.745917+0000) 2024-08-19T14:51:18.800 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:18 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:18.689+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6830 osd.3 since back 2024-08-19T14:50:38.846810+0000 front 2024-08-19T14:50:38.846826+0000 (oldest deadline 2024-08-19T14:51:01.746744+0000) 2024-08-19T14:51:18.800 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:18 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:18.689+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.32:6812 osd.5 since back 2024-08-19T14:50:52.848618+0000 front 2024-08-19T14:50:52.848721+0000 (oldest deadline 2024-08-19T14:51:18.148567+0000) 2024-08-19T14:51:19.410 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:51:19 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:51:19.107+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:14.239016+0000 front 2024-08-19T14:50:14.238817+0000 (oldest deadline 2024-08-19T14:50:38.339035+0000) 2024-08-19T14:51:19.410 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:51:19 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:51:19.107+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:22.841332+0000 front 2024-08-19T14:50:22.841705+0000 (oldest deadline 2024-08-19T14:50:48.741518+0000) 2024-08-19T14:51:19.410 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:51:19 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:51:19.107+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.31:6830 osd.3 since back 2024-08-19T14:50:39.645003+0000 front 2024-08-19T14:50:39.645158+0000 (oldest deadline 2024-08-19T14:51:05.545008+0000) 2024-08-19T14:51:19.411 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:51:19 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:51:19.107+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.32:6804 osd.4 since back 2024-08-19T14:50:45.545695+0000 front 2024-08-19T14:50:45.545767+0000 (oldest deadline 2024-08-19T14:51:10.845624+0000) 2024-08-19T14:51:19.666 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:19 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:19.410+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.724961+0000 front 2024-08-19T14:50:12.724727+0000 (oldest deadline 2024-08-19T14:50:36.224802+0000) 2024-08-19T14:51:19.666 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:19 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:19.410+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:20.627490+0000 front 2024-08-19T14:50:20.627552+0000 (oldest deadline 2024-08-19T14:50:46.527500+0000) 2024-08-19T14:51:19.666 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:19 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:19.410+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:30.330088+0000 front 2024-08-19T14:50:30.330151+0000 (oldest deadline 2024-08-19T14:50:53.230021+0000) 2024-08-19T14:51:19.666 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:19 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:19.410+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.32:6804 osd.4 since back 2024-08-19T14:50:47.232484+0000 front 2024-08-19T14:50:47.232609+0000 (oldest deadline 2024-08-19T14:51:13.132617+0000) 2024-08-19T14:51:19.667 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:19 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:19.410+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.32:6812 osd.5 since back 2024-08-19T14:50:54.233709+0000 front 2024-08-19T14:50:54.233586+0000 (oldest deadline 2024-08-19T14:51:17.733482+0000) 2024-08-19T14:51:20.046 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:19 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:19.664+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:13.297511+0000 front 2024-08-19T14:50:13.297834+0000 (oldest deadline 2024-08-19T14:50:37.997679+0000) 2024-08-19T14:51:20.047 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:19 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:19.664+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.298667+0000 front 2024-08-19T14:50:23.298816+0000 (oldest deadline 2024-08-19T14:50:49.198731+0000) 2024-08-19T14:51:20.047 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:19 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:19.664+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:29.199201+0000 front 2024-08-19T14:50:29.199233+0000 (oldest deadline 2024-08-19T14:50:54.499327+0000) 2024-08-19T14:51:20.047 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:19 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:19.664+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.32:6804 osd.4 since back 2024-08-19T14:50:47.901858+0000 front 2024-08-19T14:50:43.801431+0000 (oldest deadline 2024-08-19T14:51:11.401793+0000) 2024-08-19T14:51:20.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:19 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:19.678+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.842003+0000 front 2024-08-19T14:50:12.842211+0000 (oldest deadline 2024-08-19T14:50:36.242616+0000) 2024-08-19T14:51:20.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:19 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:19.678+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.744487+0000 front 2024-08-19T14:50:23.744453+0000 (oldest deadline 2024-08-19T14:50:47.844529+0000) 2024-08-19T14:51:20.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:19 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:19.678+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:31.345568+0000 front 2024-08-19T14:50:31.345651+0000 (oldest deadline 2024-08-19T14:50:57.745917+0000) 2024-08-19T14:51:20.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:19 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:19.678+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6830 osd.3 since back 2024-08-19T14:50:38.846810+0000 front 2024-08-19T14:50:38.846826+0000 (oldest deadline 2024-08-19T14:51:01.746744+0000) 2024-08-19T14:51:20.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:19 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:19.678+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.32:6812 osd.5 since back 2024-08-19T14:50:52.848618+0000 front 2024-08-19T14:50:52.848721+0000 (oldest deadline 2024-08-19T14:51:18.148567+0000) 2024-08-19T14:51:20.412 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:51:20 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:51:20.078+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:14.239016+0000 front 2024-08-19T14:50:14.238817+0000 (oldest deadline 2024-08-19T14:50:38.339035+0000) 2024-08-19T14:51:20.413 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:51:20 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:51:20.078+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:22.841332+0000 front 2024-08-19T14:50:22.841705+0000 (oldest deadline 2024-08-19T14:50:48.741518+0000) 2024-08-19T14:51:20.413 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:51:20 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:51:20.078+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.31:6830 osd.3 since back 2024-08-19T14:50:39.645003+0000 front 2024-08-19T14:50:39.645158+0000 (oldest deadline 2024-08-19T14:51:05.545008+0000) 2024-08-19T14:51:20.413 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:51:20 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:51:20.078+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.32:6804 osd.4 since back 2024-08-19T14:50:45.545695+0000 front 2024-08-19T14:50:45.545767+0000 (oldest deadline 2024-08-19T14:51:10.845624+0000) 2024-08-19T14:51:20.413 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:51:20 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8[24878]: 2024-08-19T14:51:20.078+0000 7f994ceea640 -1 osd.8 983 heartbeat_check: no reply from 172.21.15.32:6812 osd.5 since back 2024-08-19T14:50:53.647223+0000 front 2024-08-19T14:50:53.647203+0000 (oldest deadline 2024-08-19T14:51:19.547185+0000) 2024-08-19T14:51:20.665 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:20 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:20.411+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.724961+0000 front 2024-08-19T14:50:12.724727+0000 (oldest deadline 2024-08-19T14:50:36.224802+0000) 2024-08-19T14:51:20.666 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:20 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:20.411+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:20.627490+0000 front 2024-08-19T14:50:20.627552+0000 (oldest deadline 2024-08-19T14:50:46.527500+0000) 2024-08-19T14:51:20.666 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:20 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:20.411+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:30.330088+0000 front 2024-08-19T14:50:30.330151+0000 (oldest deadline 2024-08-19T14:50:53.230021+0000) 2024-08-19T14:51:20.666 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:20 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:20.411+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.32:6804 osd.4 since back 2024-08-19T14:50:47.232484+0000 front 2024-08-19T14:50:47.232609+0000 (oldest deadline 2024-08-19T14:51:13.132617+0000) 2024-08-19T14:51:20.666 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:20 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:20.411+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.32:6812 osd.5 since back 2024-08-19T14:50:54.233709+0000 front 2024-08-19T14:50:54.233586+0000 (oldest deadline 2024-08-19T14:51:17.733482+0000) 2024-08-19T14:51:21.047 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:20 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:20.690+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:13.297511+0000 front 2024-08-19T14:50:13.297834+0000 (oldest deadline 2024-08-19T14:50:37.997679+0000) 2024-08-19T14:51:21.047 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:20 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:20.690+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.298667+0000 front 2024-08-19T14:50:23.298816+0000 (oldest deadline 2024-08-19T14:50:49.198731+0000) 2024-08-19T14:51:21.047 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:20 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:20.690+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:29.199201+0000 front 2024-08-19T14:50:29.199233+0000 (oldest deadline 2024-08-19T14:50:54.499327+0000) 2024-08-19T14:51:21.047 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:20 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:20.690+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.32:6804 osd.4 since back 2024-08-19T14:50:47.901858+0000 front 2024-08-19T14:50:43.801431+0000 (oldest deadline 2024-08-19T14:51:11.401793+0000) 2024-08-19T14:51:21.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:20 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:20.664+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.842003+0000 front 2024-08-19T14:50:12.842211+0000 (oldest deadline 2024-08-19T14:50:36.242616+0000) 2024-08-19T14:51:21.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:20 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:20.664+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.744487+0000 front 2024-08-19T14:50:23.744453+0000 (oldest deadline 2024-08-19T14:50:47.844529+0000) 2024-08-19T14:51:21.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:20 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:20.664+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:31.345568+0000 front 2024-08-19T14:50:31.345651+0000 (oldest deadline 2024-08-19T14:50:57.745917+0000) 2024-08-19T14:51:21.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:20 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:20.664+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6830 osd.3 since back 2024-08-19T14:50:38.846810+0000 front 2024-08-19T14:50:38.846826+0000 (oldest deadline 2024-08-19T14:51:01.746744+0000) 2024-08-19T14:51:21.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:20 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:20.664+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.32:6812 osd.5 since back 2024-08-19T14:50:52.848618+0000 front 2024-08-19T14:50:52.848721+0000 (oldest deadline 2024-08-19T14:51:18.148567+0000) 2024-08-19T14:51:21.546 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:51:21 smithi045 podman[167554]: 2024-08-19 14:51:21.250518953 +0000 UTC m=+5.694368178 container died 3e1b76da34bfd03d6400341d1823e1115e427e7d539eec0fb690e57979b2d079 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-de09dd0, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-08-19T14:51:21.547 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:21 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:21.388+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.724961+0000 front 2024-08-19T14:50:12.724727+0000 (oldest deadline 2024-08-19T14:50:36.224802+0000) 2024-08-19T14:51:21.547 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:21 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:21.388+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:20.627490+0000 front 2024-08-19T14:50:20.627552+0000 (oldest deadline 2024-08-19T14:50:46.527500+0000) 2024-08-19T14:51:21.547 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:21 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:21.388+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:30.330088+0000 front 2024-08-19T14:50:30.330151+0000 (oldest deadline 2024-08-19T14:50:53.230021+0000) 2024-08-19T14:51:21.547 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:21 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:21.388+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.32:6804 osd.4 since back 2024-08-19T14:50:47.232484+0000 front 2024-08-19T14:50:47.232609+0000 (oldest deadline 2024-08-19T14:51:13.132617+0000) 2024-08-19T14:51:21.547 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:21 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:21.388+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.32:6812 osd.5 since back 2024-08-19T14:50:54.233709+0000 front 2024-08-19T14:50:54.233586+0000 (oldest deadline 2024-08-19T14:51:17.733482+0000) 2024-08-19T14:51:22.046 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:51:21 smithi045 podman[167554]: 2024-08-19 14:51:21.818684555 +0000 UTC m=+6.262533817 container cleanup 3e1b76da34bfd03d6400341d1823e1115e427e7d539eec0fb690e57979b2d079 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-de09dd0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, ceph=True, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image) 2024-08-19T14:51:22.047 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:51:21 smithi045 bash[167554]: ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8 2024-08-19T14:51:22.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:21 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:21.659+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.842003+0000 front 2024-08-19T14:50:12.842211+0000 (oldest deadline 2024-08-19T14:50:36.242616+0000) 2024-08-19T14:51:22.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:21 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:21.659+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.744487+0000 front 2024-08-19T14:50:23.744453+0000 (oldest deadline 2024-08-19T14:50:47.844529+0000) 2024-08-19T14:51:22.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:21 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:21.659+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:31.345568+0000 front 2024-08-19T14:50:31.345651+0000 (oldest deadline 2024-08-19T14:50:57.745917+0000) 2024-08-19T14:51:22.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:21 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:21.659+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6830 osd.3 since back 2024-08-19T14:50:38.846810+0000 front 2024-08-19T14:50:38.846826+0000 (oldest deadline 2024-08-19T14:51:01.746744+0000) 2024-08-19T14:51:22.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:21 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:21.659+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.32:6812 osd.5 since back 2024-08-19T14:50:52.848618+0000 front 2024-08-19T14:50:52.848721+0000 (oldest deadline 2024-08-19T14:51:18.148567+0000) 2024-08-19T14:51:22.047 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:21 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:21.730+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:13.297511+0000 front 2024-08-19T14:50:13.297834+0000 (oldest deadline 2024-08-19T14:50:37.997679+0000) 2024-08-19T14:51:22.048 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:21 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:21.730+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.298667+0000 front 2024-08-19T14:50:23.298816+0000 (oldest deadline 2024-08-19T14:50:49.198731+0000) 2024-08-19T14:51:22.048 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:21 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:21.730+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:29.199201+0000 front 2024-08-19T14:50:29.199233+0000 (oldest deadline 2024-08-19T14:50:54.499327+0000) 2024-08-19T14:51:22.048 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:21 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:21.730+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.32:6804 osd.4 since back 2024-08-19T14:50:47.901858+0000 front 2024-08-19T14:50:43.801431+0000 (oldest deadline 2024-08-19T14:51:11.401793+0000) 2024-08-19T14:51:22.048 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:21 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:21.730+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.32:6812 osd.5 since back 2024-08-19T14:50:55.502748+0000 front 2024-08-19T14:50:55.503189+0000 (oldest deadline 2024-08-19T14:51:21.402588+0000) 2024-08-19T14:51:22.382 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:51:22 smithi045 podman[167569]: 2024-08-19 14:51:22.094971439 +0000 UTC m=+0.843130723 container remove 3e1b76da34bfd03d6400341d1823e1115e427e7d539eec0fb690e57979b2d079 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8, org.label-schema.build-date=20240731, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, RELEASE=squid-de09dd0, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-08-19T14:51:22.692 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:51:22 smithi045 podman[167658]: 2024-08-19 14:51:22.361018478 +0000 UTC m=+0.017410233 image pull ea7a5189f30bad3aca9524788e2965d2368d7ce786df09d4edcd8c3edf9cefba quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9 2024-08-19T14:51:22.692 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:51:22 smithi045 podman[167658]: 2024-08-19 14:51:22.479558744 +0000 UTC m=+0.135950530 container create c092923fa2d308e9f88d8697de2f841463001bf513704e5753f98bc7f80ea4ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8-deactivate, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , RELEASE=squid-de09dd0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-08-19T14:51:22.692 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:22 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:22.379+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.724961+0000 front 2024-08-19T14:50:12.724727+0000 (oldest deadline 2024-08-19T14:50:36.224802+0000) 2024-08-19T14:51:22.692 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:22 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:22.379+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:20.627490+0000 front 2024-08-19T14:50:20.627552+0000 (oldest deadline 2024-08-19T14:50:46.527500+0000) 2024-08-19T14:51:22.692 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:22 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:22.379+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:30.330088+0000 front 2024-08-19T14:50:30.330151+0000 (oldest deadline 2024-08-19T14:50:53.230021+0000) 2024-08-19T14:51:22.692 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:22 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:22.379+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.32:6804 osd.4 since back 2024-08-19T14:50:47.232484+0000 front 2024-08-19T14:50:47.232609+0000 (oldest deadline 2024-08-19T14:51:13.132617+0000) 2024-08-19T14:51:22.693 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:22 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:22.379+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.32:6812 osd.5 since back 2024-08-19T14:50:54.233709+0000 front 2024-08-19T14:50:54.233586+0000 (oldest deadline 2024-08-19T14:51:17.733482+0000) 2024-08-19T14:51:22.983 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:51:22 smithi045 podman[167658]: 2024-08-19 14:51:22.720797091 +0000 UTC m=+0.377188837 container init c092923fa2d308e9f88d8697de2f841463001bf513704e5753f98bc7f80ea4ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8-deactivate, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=squid-de09dd0, org.label-schema.schema-version=1.0) 2024-08-19T14:51:22.984 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:51:22 smithi045 podman[167658]: 2024-08-19 14:51:22.72412225 +0000 UTC m=+0.380513997 container start c092923fa2d308e9f88d8697de2f841463001bf513704e5753f98bc7f80ea4ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, RELEASE=squid-de09dd0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image) 2024-08-19T14:51:22.984 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:51:22 smithi045 podman[167658]: 2024-08-19 14:51:22.81570028 +0000 UTC m=+0.472092035 container attach c092923fa2d308e9f88d8697de2f841463001bf513704e5753f98bc7f80ea4ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=squid-de09dd0) 2024-08-19T14:51:22.984 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:22 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:22.772+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:13.297511+0000 front 2024-08-19T14:50:13.297834+0000 (oldest deadline 2024-08-19T14:50:37.997679+0000) 2024-08-19T14:51:22.984 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:22 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:22.772+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.298667+0000 front 2024-08-19T14:50:23.298816+0000 (oldest deadline 2024-08-19T14:50:49.198731+0000) 2024-08-19T14:51:22.984 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:22 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:22.772+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:29.199201+0000 front 2024-08-19T14:50:29.199233+0000 (oldest deadline 2024-08-19T14:50:54.499327+0000) 2024-08-19T14:51:22.984 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:22 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:22.772+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.32:6804 osd.4 since back 2024-08-19T14:50:47.901858+0000 front 2024-08-19T14:50:43.801431+0000 (oldest deadline 2024-08-19T14:51:11.401793+0000) 2024-08-19T14:51:22.984 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:22 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:22.772+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.32:6812 osd.5 since back 2024-08-19T14:50:55.502748+0000 front 2024-08-19T14:50:55.503189+0000 (oldest deadline 2024-08-19T14:51:21.402588+0000) 2024-08-19T14:51:22.985 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:22 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:22.690+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.842003+0000 front 2024-08-19T14:50:12.842211+0000 (oldest deadline 2024-08-19T14:50:36.242616+0000) 2024-08-19T14:51:22.985 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:22 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:22.690+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.744487+0000 front 2024-08-19T14:50:23.744453+0000 (oldest deadline 2024-08-19T14:50:47.844529+0000) 2024-08-19T14:51:22.985 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:22 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:22.690+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:31.345568+0000 front 2024-08-19T14:50:31.345651+0000 (oldest deadline 2024-08-19T14:50:57.745917+0000) 2024-08-19T14:51:22.985 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:22 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:22.690+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6830 osd.3 since back 2024-08-19T14:50:38.846810+0000 front 2024-08-19T14:50:38.846826+0000 (oldest deadline 2024-08-19T14:51:01.746744+0000) 2024-08-19T14:51:22.985 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:22 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:22.690+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.32:6812 osd.5 since back 2024-08-19T14:50:52.848618+0000 front 2024-08-19T14:50:52.848721+0000 (oldest deadline 2024-08-19T14:51:18.148567+0000) 2024-08-19T14:51:23.277 INFO:journalctl@ceph.osd.8.smithi045.stdout:Aug 19 14:51:22 smithi045 podman[167658]: 2024-08-19 14:51:22.983555708 +0000 UTC m=+0.639947457 container died c092923fa2d308e9f88d8697de2f841463001bf513704e5753f98bc7f80ea4ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-8-deactivate, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, RELEASE=squid-de09dd0, org.label-schema.license=GPLv2, io.buildah.version=1.36.0) 2024-08-19T14:51:23.546 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:23 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:23.400+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.724961+0000 front 2024-08-19T14:50:12.724727+0000 (oldest deadline 2024-08-19T14:50:36.224802+0000) 2024-08-19T14:51:23.546 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:23 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:23.400+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:20.627490+0000 front 2024-08-19T14:50:20.627552+0000 (oldest deadline 2024-08-19T14:50:46.527500+0000) 2024-08-19T14:51:23.547 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:23 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:23.400+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:30.330088+0000 front 2024-08-19T14:50:30.330151+0000 (oldest deadline 2024-08-19T14:50:53.230021+0000) 2024-08-19T14:51:23.547 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:23 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:23.400+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.32:6804 osd.4 since back 2024-08-19T14:50:47.232484+0000 front 2024-08-19T14:50:47.232609+0000 (oldest deadline 2024-08-19T14:51:13.132617+0000) 2024-08-19T14:51:23.547 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:23 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:23.400+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.32:6812 osd.5 since back 2024-08-19T14:50:54.233709+0000 front 2024-08-19T14:50:54.233586+0000 (oldest deadline 2024-08-19T14:51:17.733482+0000) 2024-08-19T14:51:23.631 DEBUG:teuthology.orchestra.run.smithi045:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968@osd.8.service' 2024-08-19T14:51:23.713 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-19T14:51:23.713 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-08-19T14:51:23.713 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-08-19T14:51:23.714 DEBUG:teuthology.orchestra.run.smithi045:> sudo systemctl stop ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968@osd.9 2024-08-19T14:51:23.897 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:23 smithi045 systemd[1]: Stopping Ceph osd.9 for ed7d1c6e-5e33-11ef-bccf-c7b262605968... 2024-08-19T14:51:23.897 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:23 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:23.734+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:13.297511+0000 front 2024-08-19T14:50:13.297834+0000 (oldest deadline 2024-08-19T14:50:37.997679+0000) 2024-08-19T14:51:23.897 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:23 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:23.734+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.298667+0000 front 2024-08-19T14:50:23.298816+0000 (oldest deadline 2024-08-19T14:50:49.198731+0000) 2024-08-19T14:51:23.897 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:23 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:23.734+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:29.199201+0000 front 2024-08-19T14:50:29.199233+0000 (oldest deadline 2024-08-19T14:50:54.499327+0000) 2024-08-19T14:51:23.897 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:23 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:23.734+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.32:6804 osd.4 since back 2024-08-19T14:50:47.901858+0000 front 2024-08-19T14:50:43.801431+0000 (oldest deadline 2024-08-19T14:51:11.401793+0000) 2024-08-19T14:51:23.897 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:23 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:23.734+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.32:6812 osd.5 since back 2024-08-19T14:50:55.502748+0000 front 2024-08-19T14:50:55.503189+0000 (oldest deadline 2024-08-19T14:51:21.402588+0000) 2024-08-19T14:51:23.898 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:23 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:23.739+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.842003+0000 front 2024-08-19T14:50:12.842211+0000 (oldest deadline 2024-08-19T14:50:36.242616+0000) 2024-08-19T14:51:23.898 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:23 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:23.739+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.744487+0000 front 2024-08-19T14:50:23.744453+0000 (oldest deadline 2024-08-19T14:50:47.844529+0000) 2024-08-19T14:51:23.898 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:23 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:23.739+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:31.345568+0000 front 2024-08-19T14:50:31.345651+0000 (oldest deadline 2024-08-19T14:50:57.745917+0000) 2024-08-19T14:51:23.898 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:23 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:23.739+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6830 osd.3 since back 2024-08-19T14:50:38.846810+0000 front 2024-08-19T14:50:38.846826+0000 (oldest deadline 2024-08-19T14:51:01.746744+0000) 2024-08-19T14:51:23.898 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:23 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:23.739+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.32:6812 osd.5 since back 2024-08-19T14:50:52.848618+0000 front 2024-08-19T14:50:52.848721+0000 (oldest deadline 2024-08-19T14:51:18.148567+0000) 2024-08-19T14:51:24.296 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:24 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:24.054+0000 7f9989d36640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-19T14:51:24.296 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:24 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:24.054+0000 7f9989d36640 -1 osd.9 983 *** Got signal Terminated *** 2024-08-19T14:51:24.296 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:24 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:24.054+0000 7f9989d36640 -1 osd.9 983 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-08-19T14:51:24.695 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:24 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:24.361+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.724961+0000 front 2024-08-19T14:50:12.724727+0000 (oldest deadline 2024-08-19T14:50:36.224802+0000) 2024-08-19T14:51:24.695 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:24 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:24.361+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:20.627490+0000 front 2024-08-19T14:50:20.627552+0000 (oldest deadline 2024-08-19T14:50:46.527500+0000) 2024-08-19T14:51:24.695 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:24 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:24.361+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:30.330088+0000 front 2024-08-19T14:50:30.330151+0000 (oldest deadline 2024-08-19T14:50:53.230021+0000) 2024-08-19T14:51:24.695 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:24 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:24.361+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.32:6804 osd.4 since back 2024-08-19T14:50:47.232484+0000 front 2024-08-19T14:50:47.232609+0000 (oldest deadline 2024-08-19T14:51:13.132617+0000) 2024-08-19T14:51:24.695 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:24 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:24.361+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.32:6812 osd.5 since back 2024-08-19T14:50:54.233709+0000 front 2024-08-19T14:50:54.233586+0000 (oldest deadline 2024-08-19T14:51:17.733482+0000) 2024-08-19T14:51:25.047 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:24 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:24.693+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:13.297511+0000 front 2024-08-19T14:50:13.297834+0000 (oldest deadline 2024-08-19T14:50:37.997679+0000) 2024-08-19T14:51:25.047 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:24 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:24.693+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.298667+0000 front 2024-08-19T14:50:23.298816+0000 (oldest deadline 2024-08-19T14:50:49.198731+0000) 2024-08-19T14:51:25.047 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:24 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:24.693+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:29.199201+0000 front 2024-08-19T14:50:29.199233+0000 (oldest deadline 2024-08-19T14:50:54.499327+0000) 2024-08-19T14:51:25.047 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:24 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:24.693+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.32:6804 osd.4 since back 2024-08-19T14:50:47.901858+0000 front 2024-08-19T14:50:43.801431+0000 (oldest deadline 2024-08-19T14:51:11.401793+0000) 2024-08-19T14:51:25.047 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:24 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:24.693+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.32:6812 osd.5 since back 2024-08-19T14:50:55.502748+0000 front 2024-08-19T14:50:55.503189+0000 (oldest deadline 2024-08-19T14:51:21.402588+0000) 2024-08-19T14:51:25.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:24 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:24.752+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.842003+0000 front 2024-08-19T14:50:12.842211+0000 (oldest deadline 2024-08-19T14:50:36.242616+0000) 2024-08-19T14:51:25.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:24 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:24.752+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.744487+0000 front 2024-08-19T14:50:23.744453+0000 (oldest deadline 2024-08-19T14:50:47.844529+0000) 2024-08-19T14:51:25.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:24 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:24.752+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:31.345568+0000 front 2024-08-19T14:50:31.345651+0000 (oldest deadline 2024-08-19T14:50:57.745917+0000) 2024-08-19T14:51:25.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:24 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:24.752+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6830 osd.3 since back 2024-08-19T14:50:38.846810+0000 front 2024-08-19T14:50:38.846826+0000 (oldest deadline 2024-08-19T14:51:01.746744+0000) 2024-08-19T14:51:25.048 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:24 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:24.752+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.32:6812 osd.5 since back 2024-08-19T14:50:52.848618+0000 front 2024-08-19T14:50:52.848721+0000 (oldest deadline 2024-08-19T14:51:18.148567+0000) 2024-08-19T14:51:25.697 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:25 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:25.365+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.724961+0000 front 2024-08-19T14:50:12.724727+0000 (oldest deadline 2024-08-19T14:50:36.224802+0000) 2024-08-19T14:51:25.697 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:25 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:25.365+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:20.627490+0000 front 2024-08-19T14:50:20.627552+0000 (oldest deadline 2024-08-19T14:50:46.527500+0000) 2024-08-19T14:51:25.697 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:25 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:25.365+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:30.330088+0000 front 2024-08-19T14:50:30.330151+0000 (oldest deadline 2024-08-19T14:50:53.230021+0000) 2024-08-19T14:51:25.697 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:25 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:25.365+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.32:6804 osd.4 since back 2024-08-19T14:50:47.232484+0000 front 2024-08-19T14:50:47.232609+0000 (oldest deadline 2024-08-19T14:51:13.132617+0000) 2024-08-19T14:51:25.697 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:25 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:25.365+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.32:6812 osd.5 since back 2024-08-19T14:50:54.233709+0000 front 2024-08-19T14:50:54.233586+0000 (oldest deadline 2024-08-19T14:51:17.733482+0000) 2024-08-19T14:51:26.046 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:25 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:25.748+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.842003+0000 front 2024-08-19T14:50:12.842211+0000 (oldest deadline 2024-08-19T14:50:36.242616+0000) 2024-08-19T14:51:26.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:25 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:25.748+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.744487+0000 front 2024-08-19T14:50:23.744453+0000 (oldest deadline 2024-08-19T14:50:47.844529+0000) 2024-08-19T14:51:26.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:25 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:25.748+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:31.345568+0000 front 2024-08-19T14:50:31.345651+0000 (oldest deadline 2024-08-19T14:50:57.745917+0000) 2024-08-19T14:51:26.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:25 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:25.748+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6830 osd.3 since back 2024-08-19T14:50:38.846810+0000 front 2024-08-19T14:50:38.846826+0000 (oldest deadline 2024-08-19T14:51:01.746744+0000) 2024-08-19T14:51:26.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:25 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:25.748+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.32:6812 osd.5 since back 2024-08-19T14:50:52.848618+0000 front 2024-08-19T14:50:52.848721+0000 (oldest deadline 2024-08-19T14:51:18.148567+0000) 2024-08-19T14:51:26.047 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:25 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:25.695+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:13.297511+0000 front 2024-08-19T14:50:13.297834+0000 (oldest deadline 2024-08-19T14:50:37.997679+0000) 2024-08-19T14:51:26.047 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:25 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:25.695+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.298667+0000 front 2024-08-19T14:50:23.298816+0000 (oldest deadline 2024-08-19T14:50:49.198731+0000) 2024-08-19T14:51:26.047 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:25 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:25.695+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:29.199201+0000 front 2024-08-19T14:50:29.199233+0000 (oldest deadline 2024-08-19T14:50:54.499327+0000) 2024-08-19T14:51:26.047 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:25 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:25.695+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.32:6804 osd.4 since back 2024-08-19T14:50:47.901858+0000 front 2024-08-19T14:50:43.801431+0000 (oldest deadline 2024-08-19T14:51:11.401793+0000) 2024-08-19T14:51:26.047 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:25 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:25.695+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.32:6812 osd.5 since back 2024-08-19T14:50:55.502748+0000 front 2024-08-19T14:50:55.503189+0000 (oldest deadline 2024-08-19T14:51:21.402588+0000) 2024-08-19T14:51:26.048 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:25 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:25.695+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.32:6820 osd.6 since back 2024-08-19T14:51:01.403050+0000 front 2024-08-19T14:51:01.403213+0000 (oldest deadline 2024-08-19T14:51:25.503001+0000) 2024-08-19T14:51:26.715 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:26 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:26.322+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.724961+0000 front 2024-08-19T14:50:12.724727+0000 (oldest deadline 2024-08-19T14:50:36.224802+0000) 2024-08-19T14:51:26.715 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:26 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:26.322+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:20.627490+0000 front 2024-08-19T14:50:20.627552+0000 (oldest deadline 2024-08-19T14:50:46.527500+0000) 2024-08-19T14:51:26.715 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:26 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:26.322+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:30.330088+0000 front 2024-08-19T14:50:30.330151+0000 (oldest deadline 2024-08-19T14:50:53.230021+0000) 2024-08-19T14:51:26.715 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:26 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:26.322+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.32:6804 osd.4 since back 2024-08-19T14:50:47.232484+0000 front 2024-08-19T14:50:47.232609+0000 (oldest deadline 2024-08-19T14:51:13.132617+0000) 2024-08-19T14:51:26.715 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:26 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:26.322+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.32:6812 osd.5 since back 2024-08-19T14:50:54.233709+0000 front 2024-08-19T14:50:54.233586+0000 (oldest deadline 2024-08-19T14:51:17.733482+0000) 2024-08-19T14:51:27.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:26 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:26.763+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.842003+0000 front 2024-08-19T14:50:12.842211+0000 (oldest deadline 2024-08-19T14:50:36.242616+0000) 2024-08-19T14:51:27.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:26 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:26.763+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.744487+0000 front 2024-08-19T14:50:23.744453+0000 (oldest deadline 2024-08-19T14:50:47.844529+0000) 2024-08-19T14:51:27.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:26 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:26.763+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:31.345568+0000 front 2024-08-19T14:50:31.345651+0000 (oldest deadline 2024-08-19T14:50:57.745917+0000) 2024-08-19T14:51:27.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:26 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:26.763+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6830 osd.3 since back 2024-08-19T14:50:38.846810+0000 front 2024-08-19T14:50:38.846826+0000 (oldest deadline 2024-08-19T14:51:01.746744+0000) 2024-08-19T14:51:27.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:26 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:26.763+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.32:6812 osd.5 since back 2024-08-19T14:50:52.848618+0000 front 2024-08-19T14:50:52.848721+0000 (oldest deadline 2024-08-19T14:51:18.148567+0000) 2024-08-19T14:51:27.047 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:26 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:26.713+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:13.297511+0000 front 2024-08-19T14:50:13.297834+0000 (oldest deadline 2024-08-19T14:50:37.997679+0000) 2024-08-19T14:51:27.047 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:26 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:26.713+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.298667+0000 front 2024-08-19T14:50:23.298816+0000 (oldest deadline 2024-08-19T14:50:49.198731+0000) 2024-08-19T14:51:27.047 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:26 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:26.713+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:29.199201+0000 front 2024-08-19T14:50:29.199233+0000 (oldest deadline 2024-08-19T14:50:54.499327+0000) 2024-08-19T14:51:27.047 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:26 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:26.713+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.32:6804 osd.4 since back 2024-08-19T14:50:47.901858+0000 front 2024-08-19T14:50:43.801431+0000 (oldest deadline 2024-08-19T14:51:11.401793+0000) 2024-08-19T14:51:27.047 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:26 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:26.713+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.32:6812 osd.5 since back 2024-08-19T14:50:55.502748+0000 front 2024-08-19T14:50:55.503189+0000 (oldest deadline 2024-08-19T14:51:21.402588+0000) 2024-08-19T14:51:27.048 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:26 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:26.713+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.32:6820 osd.6 since back 2024-08-19T14:51:01.403050+0000 front 2024-08-19T14:51:01.403213+0000 (oldest deadline 2024-08-19T14:51:25.503001+0000) 2024-08-19T14:51:27.665 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:27 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:27.315+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.724961+0000 front 2024-08-19T14:50:12.724727+0000 (oldest deadline 2024-08-19T14:50:36.224802+0000) 2024-08-19T14:51:27.666 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:27 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:27.315+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:20.627490+0000 front 2024-08-19T14:50:20.627552+0000 (oldest deadline 2024-08-19T14:50:46.527500+0000) 2024-08-19T14:51:27.666 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:27 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:27.315+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:30.330088+0000 front 2024-08-19T14:50:30.330151+0000 (oldest deadline 2024-08-19T14:50:53.230021+0000) 2024-08-19T14:51:27.666 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:27 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:27.315+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.32:6804 osd.4 since back 2024-08-19T14:50:47.232484+0000 front 2024-08-19T14:50:47.232609+0000 (oldest deadline 2024-08-19T14:51:13.132617+0000) 2024-08-19T14:51:27.666 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:27 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:27.315+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.32:6812 osd.5 since back 2024-08-19T14:50:54.233709+0000 front 2024-08-19T14:50:54.233586+0000 (oldest deadline 2024-08-19T14:51:17.733482+0000) 2024-08-19T14:51:28.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:27 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:27.723+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.842003+0000 front 2024-08-19T14:50:12.842211+0000 (oldest deadline 2024-08-19T14:50:36.242616+0000) 2024-08-19T14:51:28.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:27 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:27.723+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.744487+0000 front 2024-08-19T14:50:23.744453+0000 (oldest deadline 2024-08-19T14:50:47.844529+0000) 2024-08-19T14:51:28.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:27 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:27.723+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:31.345568+0000 front 2024-08-19T14:50:31.345651+0000 (oldest deadline 2024-08-19T14:50:57.745917+0000) 2024-08-19T14:51:28.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:27 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:27.723+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6830 osd.3 since back 2024-08-19T14:50:38.846810+0000 front 2024-08-19T14:50:38.846826+0000 (oldest deadline 2024-08-19T14:51:01.746744+0000) 2024-08-19T14:51:28.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:27 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:27.723+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.32:6812 osd.5 since back 2024-08-19T14:50:52.848618+0000 front 2024-08-19T14:50:52.848721+0000 (oldest deadline 2024-08-19T14:51:18.148567+0000) 2024-08-19T14:51:28.047 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:27 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:27.664+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:13.297511+0000 front 2024-08-19T14:50:13.297834+0000 (oldest deadline 2024-08-19T14:50:37.997679+0000) 2024-08-19T14:51:28.047 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:27 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:27.664+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.298667+0000 front 2024-08-19T14:50:23.298816+0000 (oldest deadline 2024-08-19T14:50:49.198731+0000) 2024-08-19T14:51:28.048 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:27 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:27.664+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:29.199201+0000 front 2024-08-19T14:50:29.199233+0000 (oldest deadline 2024-08-19T14:50:54.499327+0000) 2024-08-19T14:51:28.048 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:27 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:27.664+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.32:6804 osd.4 since back 2024-08-19T14:50:47.901858+0000 front 2024-08-19T14:50:43.801431+0000 (oldest deadline 2024-08-19T14:51:11.401793+0000) 2024-08-19T14:51:28.048 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:27 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:27.664+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.32:6812 osd.5 since back 2024-08-19T14:50:55.502748+0000 front 2024-08-19T14:50:55.503189+0000 (oldest deadline 2024-08-19T14:51:21.402588+0000) 2024-08-19T14:51:28.048 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:27 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:27.664+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.32:6820 osd.6 since back 2024-08-19T14:51:01.403050+0000 front 2024-08-19T14:51:01.403213+0000 (oldest deadline 2024-08-19T14:51:25.503001+0000) 2024-08-19T14:51:28.668 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:28 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:28.333+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.724961+0000 front 2024-08-19T14:50:12.724727+0000 (oldest deadline 2024-08-19T14:50:36.224802+0000) 2024-08-19T14:51:28.668 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:28 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:28.333+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:20.627490+0000 front 2024-08-19T14:50:20.627552+0000 (oldest deadline 2024-08-19T14:50:46.527500+0000) 2024-08-19T14:51:28.668 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:28 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:28.333+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:30.330088+0000 front 2024-08-19T14:50:30.330151+0000 (oldest deadline 2024-08-19T14:50:53.230021+0000) 2024-08-19T14:51:28.668 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:28 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:28.333+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.32:6804 osd.4 since back 2024-08-19T14:50:47.232484+0000 front 2024-08-19T14:50:47.232609+0000 (oldest deadline 2024-08-19T14:51:13.132617+0000) 2024-08-19T14:51:28.668 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:28 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9[30281]: 2024-08-19T14:51:28.333+0000 7f998634c640 -1 osd.9 983 heartbeat_check: no reply from 172.21.15.32:6812 osd.5 since back 2024-08-19T14:50:54.233709+0000 front 2024-08-19T14:50:54.233586+0000 (oldest deadline 2024-08-19T14:51:17.733482+0000) 2024-08-19T14:51:29.047 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:28 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:28.667+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:13.297511+0000 front 2024-08-19T14:50:13.297834+0000 (oldest deadline 2024-08-19T14:50:37.997679+0000) 2024-08-19T14:51:29.047 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:28 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:28.667+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.298667+0000 front 2024-08-19T14:50:23.298816+0000 (oldest deadline 2024-08-19T14:50:49.198731+0000) 2024-08-19T14:51:29.047 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:28 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:28.667+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:29.199201+0000 front 2024-08-19T14:50:29.199233+0000 (oldest deadline 2024-08-19T14:50:54.499327+0000) 2024-08-19T14:51:29.047 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:28 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:28.667+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.32:6804 osd.4 since back 2024-08-19T14:50:47.901858+0000 front 2024-08-19T14:50:43.801431+0000 (oldest deadline 2024-08-19T14:51:11.401793+0000) 2024-08-19T14:51:29.047 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:28 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:28.667+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.32:6812 osd.5 since back 2024-08-19T14:50:55.502748+0000 front 2024-08-19T14:50:55.503189+0000 (oldest deadline 2024-08-19T14:51:21.402588+0000) 2024-08-19T14:51:29.047 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:28 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:28.667+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.32:6820 osd.6 since back 2024-08-19T14:51:01.403050+0000 front 2024-08-19T14:51:01.403213+0000 (oldest deadline 2024-08-19T14:51:25.503001+0000) 2024-08-19T14:51:29.048 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:28 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:28.763+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.842003+0000 front 2024-08-19T14:50:12.842211+0000 (oldest deadline 2024-08-19T14:50:36.242616+0000) 2024-08-19T14:51:29.048 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:28 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:28.763+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.744487+0000 front 2024-08-19T14:50:23.744453+0000 (oldest deadline 2024-08-19T14:50:47.844529+0000) 2024-08-19T14:51:29.048 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:28 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:28.763+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:31.345568+0000 front 2024-08-19T14:50:31.345651+0000 (oldest deadline 2024-08-19T14:50:57.745917+0000) 2024-08-19T14:51:29.048 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:28 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:28.763+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6830 osd.3 since back 2024-08-19T14:50:38.846810+0000 front 2024-08-19T14:50:38.846826+0000 (oldest deadline 2024-08-19T14:51:01.746744+0000) 2024-08-19T14:51:29.048 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:28 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:28.763+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.32:6812 osd.5 since back 2024-08-19T14:50:52.848618+0000 front 2024-08-19T14:50:52.848721+0000 (oldest deadline 2024-08-19T14:51:18.148567+0000) 2024-08-19T14:51:29.048 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:28 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:28.763+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.32:6820 osd.6 since back 2024-08-19T14:51:02.150117+0000 front 2024-08-19T14:51:02.150094+0000 (oldest deadline 2024-08-19T14:51:28.050077+0000) 2024-08-19T14:51:29.797 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:29 smithi045 podman[167762]: 2024-08-19 14:51:29.541759975 +0000 UTC m=+5.580616331 container died 0d390f7b3716e1e06f6d2e098f125b493129e01bda099b1a9f799ec4c66eef72 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240731, RELEASE=squid-de09dd0, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-08-19T14:51:29.798 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:29 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:29.665+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:13.297511+0000 front 2024-08-19T14:50:13.297834+0000 (oldest deadline 2024-08-19T14:50:37.997679+0000) 2024-08-19T14:51:29.798 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:29 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:29.665+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.298667+0000 front 2024-08-19T14:50:23.298816+0000 (oldest deadline 2024-08-19T14:50:49.198731+0000) 2024-08-19T14:51:29.798 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:29 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:29.665+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:29.199201+0000 front 2024-08-19T14:50:29.199233+0000 (oldest deadline 2024-08-19T14:50:54.499327+0000) 2024-08-19T14:51:29.798 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:29 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:29.665+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.32:6804 osd.4 since back 2024-08-19T14:50:47.901858+0000 front 2024-08-19T14:50:43.801431+0000 (oldest deadline 2024-08-19T14:51:11.401793+0000) 2024-08-19T14:51:29.798 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:29 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:29.665+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.32:6812 osd.5 since back 2024-08-19T14:50:55.502748+0000 front 2024-08-19T14:50:55.503189+0000 (oldest deadline 2024-08-19T14:51:21.402588+0000) 2024-08-19T14:51:29.798 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:29 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:29.665+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.32:6820 osd.6 since back 2024-08-19T14:51:01.403050+0000 front 2024-08-19T14:51:01.403213+0000 (oldest deadline 2024-08-19T14:51:25.503001+0000) 2024-08-19T14:51:29.799 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:29 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:29.715+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.842003+0000 front 2024-08-19T14:50:12.842211+0000 (oldest deadline 2024-08-19T14:50:36.242616+0000) 2024-08-19T14:51:29.799 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:29 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:29.715+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.744487+0000 front 2024-08-19T14:50:23.744453+0000 (oldest deadline 2024-08-19T14:50:47.844529+0000) 2024-08-19T14:51:29.799 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:29 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:29.715+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:31.345568+0000 front 2024-08-19T14:50:31.345651+0000 (oldest deadline 2024-08-19T14:50:57.745917+0000) 2024-08-19T14:51:29.799 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:29 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:29.715+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6830 osd.3 since back 2024-08-19T14:50:38.846810+0000 front 2024-08-19T14:50:38.846826+0000 (oldest deadline 2024-08-19T14:51:01.746744+0000) 2024-08-19T14:51:29.799 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:29 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:29.715+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.32:6812 osd.5 since back 2024-08-19T14:50:52.848618+0000 front 2024-08-19T14:50:52.848721+0000 (oldest deadline 2024-08-19T14:51:18.148567+0000) 2024-08-19T14:51:29.799 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:29 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:29.715+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.32:6820 osd.6 since back 2024-08-19T14:51:02.150117+0000 front 2024-08-19T14:51:02.150094+0000 (oldest deadline 2024-08-19T14:51:28.050077+0000) 2024-08-19T14:51:30.199 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:29 smithi045 podman[167762]: 2024-08-19 14:51:29.955932926 +0000 UTC m=+5.994789274 container cleanup 0d390f7b3716e1e06f6d2e098f125b493129e01bda099b1a9f799ec4c66eef72 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240731, RELEASE=squid-de09dd0, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-08-19T14:51:30.199 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:29 smithi045 bash[167762]: ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9 2024-08-19T14:51:30.522 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:30 smithi045 podman[167791]: 2024-08-19 14:51:30.199044635 +0000 UTC m=+0.652682489 container remove 0d390f7b3716e1e06f6d2e098f125b493129e01bda099b1a9f799ec4c66eef72 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9, ceph=True, io.buildah.version=1.36.0, RELEASE=squid-de09dd0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-08-19T14:51:30.786 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:30 smithi045 podman[167879]: 2024-08-19 14:51:30.424800898 +0000 UTC m=+0.017099660 image pull ea7a5189f30bad3aca9524788e2965d2368d7ce786df09d4edcd8c3edf9cefba quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9 2024-08-19T14:51:30.786 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:30 smithi045 podman[167879]: 2024-08-19 14:51:30.591804132 +0000 UTC m=+0.184102880 container create b2ad92471f95549504490dc0c91f74d9c9db82e7818c49b063df0052cfa15e28 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9-deactivate, org.label-schema.build-date=20240731, io.buildah.version=1.36.0, RELEASE=squid-de09dd0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=) 2024-08-19T14:51:30.786 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:30 smithi045 podman[167879]: 2024-08-19 14:51:30.785984466 +0000 UTC m=+0.378283224 container init b2ad92471f95549504490dc0c91f74d9c9db82e7818c49b063df0052cfa15e28 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, RELEASE=squid-de09dd0, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-08-19T14:51:30.786 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:30 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:30.649+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:13.297511+0000 front 2024-08-19T14:50:13.297834+0000 (oldest deadline 2024-08-19T14:50:37.997679+0000) 2024-08-19T14:51:30.787 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:30 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:30.649+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.298667+0000 front 2024-08-19T14:50:23.298816+0000 (oldest deadline 2024-08-19T14:50:49.198731+0000) 2024-08-19T14:51:30.787 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:30 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:30.649+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:29.199201+0000 front 2024-08-19T14:50:29.199233+0000 (oldest deadline 2024-08-19T14:50:54.499327+0000) 2024-08-19T14:51:30.787 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:30 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:30.649+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.32:6804 osd.4 since back 2024-08-19T14:50:47.901858+0000 front 2024-08-19T14:50:43.801431+0000 (oldest deadline 2024-08-19T14:51:11.401793+0000) 2024-08-19T14:51:30.787 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:30 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:30.649+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.32:6812 osd.5 since back 2024-08-19T14:50:55.502748+0000 front 2024-08-19T14:50:55.503189+0000 (oldest deadline 2024-08-19T14:51:21.402588+0000) 2024-08-19T14:51:30.787 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:30 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:30.649+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.32:6820 osd.6 since back 2024-08-19T14:51:01.403050+0000 front 2024-08-19T14:51:01.403213+0000 (oldest deadline 2024-08-19T14:51:25.503001+0000) 2024-08-19T14:51:30.787 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:30 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:30.707+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.842003+0000 front 2024-08-19T14:50:12.842211+0000 (oldest deadline 2024-08-19T14:50:36.242616+0000) 2024-08-19T14:51:30.787 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:30 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:30.707+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.744487+0000 front 2024-08-19T14:50:23.744453+0000 (oldest deadline 2024-08-19T14:50:47.844529+0000) 2024-08-19T14:51:30.787 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:30 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:30.707+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:31.345568+0000 front 2024-08-19T14:50:31.345651+0000 (oldest deadline 2024-08-19T14:50:57.745917+0000) 2024-08-19T14:51:30.787 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:30 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:30.707+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6830 osd.3 since back 2024-08-19T14:50:38.846810+0000 front 2024-08-19T14:50:38.846826+0000 (oldest deadline 2024-08-19T14:51:01.746744+0000) 2024-08-19T14:51:30.787 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:30 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:30.707+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.32:6812 osd.5 since back 2024-08-19T14:50:52.848618+0000 front 2024-08-19T14:50:52.848721+0000 (oldest deadline 2024-08-19T14:51:18.148567+0000) 2024-08-19T14:51:30.787 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:30 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:30.707+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.32:6820 osd.6 since back 2024-08-19T14:51:02.150117+0000 front 2024-08-19T14:51:02.150094+0000 (oldest deadline 2024-08-19T14:51:28.050077+0000) 2024-08-19T14:51:31.046 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:30 smithi045 podman[167879]: 2024-08-19 14:51:30.789467813 +0000 UTC m=+0.381766563 container start b2ad92471f95549504490dc0c91f74d9c9db82e7818c49b063df0052cfa15e28 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=squid-de09dd0, org.label-schema.vendor=CentOS) 2024-08-19T14:51:31.047 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:31 smithi045 podman[167879]: 2024-08-19 14:51:31.002516871 +0000 UTC m=+0.594815621 container attach b2ad92471f95549504490dc0c91f74d9c9db82e7818c49b063df0052cfa15e28 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=squid-de09dd0) 2024-08-19T14:51:31.047 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:31 smithi045 podman[167879]: 2024-08-19 14:51:31.003041069 +0000 UTC m=+0.595339833 container died b2ad92471f95549504490dc0c91f74d9c9db82e7818c49b063df0052cfa15e28 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9-deactivate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-de09dd0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux ) 2024-08-19T14:51:31.662 DEBUG:teuthology.orchestra.run.smithi045:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968@osd.9.service' 2024-08-19T14:51:31.690 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:31 smithi045 podman[167903]: 2024-08-19 14:51:31.631622109 +0000 UTC m=+0.637132168 container remove b2ad92471f95549504490dc0c91f74d9c9db82e7818c49b063df0052cfa15e28 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-9-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=squid-de09dd0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0) 2024-08-19T14:51:31.690 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:31 smithi045 systemd[1]: ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968@osd.9.service: Deactivated successfully. 2024-08-19T14:51:31.690 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:31 smithi045 systemd[1]: Stopped Ceph osd.9 for ed7d1c6e-5e33-11ef-bccf-c7b262605968. 2024-08-19T14:51:31.690 INFO:journalctl@ceph.osd.9.smithi045.stdout:Aug 19 14:51:31 smithi045 systemd[1]: ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968@osd.9.service: Consumed 35min 7.080s CPU time. 2024-08-19T14:51:31.690 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:31 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:31.668+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:13.297511+0000 front 2024-08-19T14:50:13.297834+0000 (oldest deadline 2024-08-19T14:50:37.997679+0000) 2024-08-19T14:51:31.690 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:31 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:31.668+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.298667+0000 front 2024-08-19T14:50:23.298816+0000 (oldest deadline 2024-08-19T14:50:49.198731+0000) 2024-08-19T14:51:31.690 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:31 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:31.668+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:29.199201+0000 front 2024-08-19T14:50:29.199233+0000 (oldest deadline 2024-08-19T14:50:54.499327+0000) 2024-08-19T14:51:31.690 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:31 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:31.668+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.32:6804 osd.4 since back 2024-08-19T14:50:47.901858+0000 front 2024-08-19T14:50:43.801431+0000 (oldest deadline 2024-08-19T14:51:11.401793+0000) 2024-08-19T14:51:31.691 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:31 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:31.668+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.32:6812 osd.5 since back 2024-08-19T14:50:55.502748+0000 front 2024-08-19T14:50:55.503189+0000 (oldest deadline 2024-08-19T14:51:21.402588+0000) 2024-08-19T14:51:31.691 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:31 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:31.668+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.32:6820 osd.6 since back 2024-08-19T14:51:01.403050+0000 front 2024-08-19T14:51:01.403213+0000 (oldest deadline 2024-08-19T14:51:25.503001+0000) 2024-08-19T14:51:31.702 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-19T14:51:31.704 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-08-19T14:51:31.704 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-08-19T14:51:31.705 DEBUG:teuthology.orchestra.run.smithi045:> sudo systemctl stop ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968@osd.10 2024-08-19T14:51:32.046 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:31 smithi045 systemd[1]: Stopping Ceph osd.10 for ed7d1c6e-5e33-11ef-bccf-c7b262605968... 2024-08-19T14:51:32.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:31 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:31.706+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.842003+0000 front 2024-08-19T14:50:12.842211+0000 (oldest deadline 2024-08-19T14:50:36.242616+0000) 2024-08-19T14:51:32.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:31 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:31.706+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.744487+0000 front 2024-08-19T14:50:23.744453+0000 (oldest deadline 2024-08-19T14:50:47.844529+0000) 2024-08-19T14:51:32.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:31 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:31.706+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:31.345568+0000 front 2024-08-19T14:50:31.345651+0000 (oldest deadline 2024-08-19T14:50:57.745917+0000) 2024-08-19T14:51:32.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:31 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:31.706+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6830 osd.3 since back 2024-08-19T14:50:38.846810+0000 front 2024-08-19T14:50:38.846826+0000 (oldest deadline 2024-08-19T14:51:01.746744+0000) 2024-08-19T14:51:32.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:31 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:31.706+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.32:6812 osd.5 since back 2024-08-19T14:50:52.848618+0000 front 2024-08-19T14:50:52.848721+0000 (oldest deadline 2024-08-19T14:51:18.148567+0000) 2024-08-19T14:51:32.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:31 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:31.706+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.32:6820 osd.6 since back 2024-08-19T14:51:02.150117+0000 front 2024-08-19T14:51:02.150094+0000 (oldest deadline 2024-08-19T14:51:28.050077+0000) 2024-08-19T14:51:32.546 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:32 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:32.050+0000 7fbc16d4c640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-19T14:51:32.547 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:32 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:32.050+0000 7fbc16d4c640 -1 osd.10 983 *** Got signal Terminated *** 2024-08-19T14:51:32.547 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:32 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:32.050+0000 7fbc16d4c640 -1 osd.10 983 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-08-19T14:51:33.047 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:32 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:32.620+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:13.297511+0000 front 2024-08-19T14:50:13.297834+0000 (oldest deadline 2024-08-19T14:50:37.997679+0000) 2024-08-19T14:51:33.047 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:32 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:32.620+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.298667+0000 front 2024-08-19T14:50:23.298816+0000 (oldest deadline 2024-08-19T14:50:49.198731+0000) 2024-08-19T14:51:33.047 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:32 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:32.620+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:29.199201+0000 front 2024-08-19T14:50:29.199233+0000 (oldest deadline 2024-08-19T14:50:54.499327+0000) 2024-08-19T14:51:33.047 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:32 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:32.620+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.32:6804 osd.4 since back 2024-08-19T14:50:47.901858+0000 front 2024-08-19T14:50:43.801431+0000 (oldest deadline 2024-08-19T14:51:11.401793+0000) 2024-08-19T14:51:33.047 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:32 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:32.620+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.32:6812 osd.5 since back 2024-08-19T14:50:55.502748+0000 front 2024-08-19T14:50:55.503189+0000 (oldest deadline 2024-08-19T14:51:21.402588+0000) 2024-08-19T14:51:33.047 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:32 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:32.620+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.32:6820 osd.6 since back 2024-08-19T14:51:01.403050+0000 front 2024-08-19T14:51:01.403213+0000 (oldest deadline 2024-08-19T14:51:25.503001+0000) 2024-08-19T14:51:33.048 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:32 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:32.714+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.842003+0000 front 2024-08-19T14:50:12.842211+0000 (oldest deadline 2024-08-19T14:50:36.242616+0000) 2024-08-19T14:51:33.048 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:32 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:32.714+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.744487+0000 front 2024-08-19T14:50:23.744453+0000 (oldest deadline 2024-08-19T14:50:47.844529+0000) 2024-08-19T14:51:33.048 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:32 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:32.714+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:31.345568+0000 front 2024-08-19T14:50:31.345651+0000 (oldest deadline 2024-08-19T14:50:57.745917+0000) 2024-08-19T14:51:33.048 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:32 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:32.714+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6830 osd.3 since back 2024-08-19T14:50:38.846810+0000 front 2024-08-19T14:50:38.846826+0000 (oldest deadline 2024-08-19T14:51:01.746744+0000) 2024-08-19T14:51:33.048 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:32 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:32.714+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.32:6812 osd.5 since back 2024-08-19T14:50:52.848618+0000 front 2024-08-19T14:50:52.848721+0000 (oldest deadline 2024-08-19T14:51:18.148567+0000) 2024-08-19T14:51:33.048 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:32 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:32.714+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.32:6820 osd.6 since back 2024-08-19T14:51:02.150117+0000 front 2024-08-19T14:51:02.150094+0000 (oldest deadline 2024-08-19T14:51:28.050077+0000) 2024-08-19T14:51:34.047 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:33 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:33.648+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:13.297511+0000 front 2024-08-19T14:50:13.297834+0000 (oldest deadline 2024-08-19T14:50:37.997679+0000) 2024-08-19T14:51:34.047 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:33 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:33.648+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.298667+0000 front 2024-08-19T14:50:23.298816+0000 (oldest deadline 2024-08-19T14:50:49.198731+0000) 2024-08-19T14:51:34.047 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:33 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:33.648+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:29.199201+0000 front 2024-08-19T14:50:29.199233+0000 (oldest deadline 2024-08-19T14:50:54.499327+0000) 2024-08-19T14:51:34.047 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:33 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:33.648+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.32:6804 osd.4 since back 2024-08-19T14:50:47.901858+0000 front 2024-08-19T14:50:43.801431+0000 (oldest deadline 2024-08-19T14:51:11.401793+0000) 2024-08-19T14:51:34.047 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:33 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:33.648+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.32:6812 osd.5 since back 2024-08-19T14:50:55.502748+0000 front 2024-08-19T14:50:55.503189+0000 (oldest deadline 2024-08-19T14:51:21.402588+0000) 2024-08-19T14:51:34.047 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:33 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:33.648+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.32:6820 osd.6 since back 2024-08-19T14:51:01.403050+0000 front 2024-08-19T14:51:01.403213+0000 (oldest deadline 2024-08-19T14:51:25.503001+0000) 2024-08-19T14:51:34.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:33 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:33.713+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.842003+0000 front 2024-08-19T14:50:12.842211+0000 (oldest deadline 2024-08-19T14:50:36.242616+0000) 2024-08-19T14:51:34.048 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:33 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:33.713+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.744487+0000 front 2024-08-19T14:50:23.744453+0000 (oldest deadline 2024-08-19T14:50:47.844529+0000) 2024-08-19T14:51:34.048 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:33 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:33.713+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:31.345568+0000 front 2024-08-19T14:50:31.345651+0000 (oldest deadline 2024-08-19T14:50:57.745917+0000) 2024-08-19T14:51:34.048 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:33 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:33.713+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6830 osd.3 since back 2024-08-19T14:50:38.846810+0000 front 2024-08-19T14:50:38.846826+0000 (oldest deadline 2024-08-19T14:51:01.746744+0000) 2024-08-19T14:51:34.048 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:33 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:33.713+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.32:6812 osd.5 since back 2024-08-19T14:50:52.848618+0000 front 2024-08-19T14:50:52.848721+0000 (oldest deadline 2024-08-19T14:51:18.148567+0000) 2024-08-19T14:51:34.048 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:33 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:33.713+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.32:6820 osd.6 since back 2024-08-19T14:51:02.150117+0000 front 2024-08-19T14:51:02.150094+0000 (oldest deadline 2024-08-19T14:51:28.050077+0000) 2024-08-19T14:51:34.048 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:33 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:33.713+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.32:6828 osd.7 since back 2024-08-19T14:51:08.551160+0000 front 2024-08-19T14:51:08.551120+0000 (oldest deadline 2024-08-19T14:51:33.251028+0000) 2024-08-19T14:51:35.047 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:34 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:34.679+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:13.297511+0000 front 2024-08-19T14:50:13.297834+0000 (oldest deadline 2024-08-19T14:50:37.997679+0000) 2024-08-19T14:51:35.047 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:34 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:34.679+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.298667+0000 front 2024-08-19T14:50:23.298816+0000 (oldest deadline 2024-08-19T14:50:49.198731+0000) 2024-08-19T14:51:35.047 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:34 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:34.679+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:29.199201+0000 front 2024-08-19T14:50:29.199233+0000 (oldest deadline 2024-08-19T14:50:54.499327+0000) 2024-08-19T14:51:35.048 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:34 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:34.679+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.32:6804 osd.4 since back 2024-08-19T14:50:47.901858+0000 front 2024-08-19T14:50:43.801431+0000 (oldest deadline 2024-08-19T14:51:11.401793+0000) 2024-08-19T14:51:35.048 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:34 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:34.679+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.32:6812 osd.5 since back 2024-08-19T14:50:55.502748+0000 front 2024-08-19T14:50:55.503189+0000 (oldest deadline 2024-08-19T14:51:21.402588+0000) 2024-08-19T14:51:35.048 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:34 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:34.679+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.32:6820 osd.6 since back 2024-08-19T14:51:01.403050+0000 front 2024-08-19T14:51:01.403213+0000 (oldest deadline 2024-08-19T14:51:25.503001+0000) 2024-08-19T14:51:35.048 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:34 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:34.679+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.32:6828 osd.7 since back 2024-08-19T14:51:08.403986+0000 front 2024-08-19T14:51:08.404017+0000 (oldest deadline 2024-08-19T14:51:34.303826+0000) 2024-08-19T14:51:35.048 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:34 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:34.761+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.842003+0000 front 2024-08-19T14:50:12.842211+0000 (oldest deadline 2024-08-19T14:50:36.242616+0000) 2024-08-19T14:51:35.049 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:34 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:34.761+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.744487+0000 front 2024-08-19T14:50:23.744453+0000 (oldest deadline 2024-08-19T14:50:47.844529+0000) 2024-08-19T14:51:35.049 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:34 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:34.761+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:31.345568+0000 front 2024-08-19T14:50:31.345651+0000 (oldest deadline 2024-08-19T14:50:57.745917+0000) 2024-08-19T14:51:35.049 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:34 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:34.761+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6830 osd.3 since back 2024-08-19T14:50:38.846810+0000 front 2024-08-19T14:50:38.846826+0000 (oldest deadline 2024-08-19T14:51:01.746744+0000) 2024-08-19T14:51:35.049 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:34 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:34.761+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.32:6812 osd.5 since back 2024-08-19T14:50:52.848618+0000 front 2024-08-19T14:50:52.848721+0000 (oldest deadline 2024-08-19T14:51:18.148567+0000) 2024-08-19T14:51:35.049 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:34 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:34.761+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.32:6820 osd.6 since back 2024-08-19T14:51:02.150117+0000 front 2024-08-19T14:51:02.150094+0000 (oldest deadline 2024-08-19T14:51:28.050077+0000) 2024-08-19T14:51:35.050 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:34 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:34.761+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.32:6828 osd.7 since back 2024-08-19T14:51:08.551160+0000 front 2024-08-19T14:51:08.551120+0000 (oldest deadline 2024-08-19T14:51:33.251028+0000) 2024-08-19T14:51:36.047 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:35 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:35.638+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:13.297511+0000 front 2024-08-19T14:50:13.297834+0000 (oldest deadline 2024-08-19T14:50:37.997679+0000) 2024-08-19T14:51:36.047 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:35 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:35.638+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.298667+0000 front 2024-08-19T14:50:23.298816+0000 (oldest deadline 2024-08-19T14:50:49.198731+0000) 2024-08-19T14:51:36.047 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:35 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:35.638+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:29.199201+0000 front 2024-08-19T14:50:29.199233+0000 (oldest deadline 2024-08-19T14:50:54.499327+0000) 2024-08-19T14:51:36.047 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:35 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:35.638+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.32:6804 osd.4 since back 2024-08-19T14:50:47.901858+0000 front 2024-08-19T14:50:43.801431+0000 (oldest deadline 2024-08-19T14:51:11.401793+0000) 2024-08-19T14:51:36.047 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:35 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:35.638+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.32:6812 osd.5 since back 2024-08-19T14:50:55.502748+0000 front 2024-08-19T14:50:55.503189+0000 (oldest deadline 2024-08-19T14:51:21.402588+0000) 2024-08-19T14:51:36.047 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:35 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:35.638+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.32:6820 osd.6 since back 2024-08-19T14:51:01.403050+0000 front 2024-08-19T14:51:01.403213+0000 (oldest deadline 2024-08-19T14:51:25.503001+0000) 2024-08-19T14:51:36.047 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:35 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:35.638+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.32:6828 osd.7 since back 2024-08-19T14:51:08.403986+0000 front 2024-08-19T14:51:08.404017+0000 (oldest deadline 2024-08-19T14:51:34.303826+0000) 2024-08-19T14:51:36.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:35 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:35.796+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.842003+0000 front 2024-08-19T14:50:12.842211+0000 (oldest deadline 2024-08-19T14:50:36.242616+0000) 2024-08-19T14:51:36.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:35 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:35.796+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.744487+0000 front 2024-08-19T14:50:23.744453+0000 (oldest deadline 2024-08-19T14:50:47.844529+0000) 2024-08-19T14:51:36.048 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:35 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:35.796+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:31.345568+0000 front 2024-08-19T14:50:31.345651+0000 (oldest deadline 2024-08-19T14:50:57.745917+0000) 2024-08-19T14:51:36.048 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:35 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:35.796+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6830 osd.3 since back 2024-08-19T14:50:38.846810+0000 front 2024-08-19T14:50:38.846826+0000 (oldest deadline 2024-08-19T14:51:01.746744+0000) 2024-08-19T14:51:36.048 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:35 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:35.796+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.32:6812 osd.5 since back 2024-08-19T14:50:52.848618+0000 front 2024-08-19T14:50:52.848721+0000 (oldest deadline 2024-08-19T14:51:18.148567+0000) 2024-08-19T14:51:36.048 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:35 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:35.796+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.32:6820 osd.6 since back 2024-08-19T14:51:02.150117+0000 front 2024-08-19T14:51:02.150094+0000 (oldest deadline 2024-08-19T14:51:28.050077+0000) 2024-08-19T14:51:36.048 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:35 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:35.796+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.32:6828 osd.7 since back 2024-08-19T14:51:08.551160+0000 front 2024-08-19T14:51:08.551120+0000 (oldest deadline 2024-08-19T14:51:33.251028+0000) 2024-08-19T14:51:36.948 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:36 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:36.660+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:13.297511+0000 front 2024-08-19T14:50:13.297834+0000 (oldest deadline 2024-08-19T14:50:37.997679+0000) 2024-08-19T14:51:36.948 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:36 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:36.660+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.298667+0000 front 2024-08-19T14:50:23.298816+0000 (oldest deadline 2024-08-19T14:50:49.198731+0000) 2024-08-19T14:51:36.948 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:36 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:36.660+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:29.199201+0000 front 2024-08-19T14:50:29.199233+0000 (oldest deadline 2024-08-19T14:50:54.499327+0000) 2024-08-19T14:51:36.948 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:36 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:36.660+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.32:6804 osd.4 since back 2024-08-19T14:50:47.901858+0000 front 2024-08-19T14:50:43.801431+0000 (oldest deadline 2024-08-19T14:51:11.401793+0000) 2024-08-19T14:51:36.948 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:36 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:36.660+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.32:6812 osd.5 since back 2024-08-19T14:50:55.502748+0000 front 2024-08-19T14:50:55.503189+0000 (oldest deadline 2024-08-19T14:51:21.402588+0000) 2024-08-19T14:51:36.948 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:36 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:36.660+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.32:6820 osd.6 since back 2024-08-19T14:51:01.403050+0000 front 2024-08-19T14:51:01.403213+0000 (oldest deadline 2024-08-19T14:51:25.503001+0000) 2024-08-19T14:51:36.948 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:36 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10[102131]: 2024-08-19T14:51:36.660+0000 7fbc0f35a640 -1 osd.10 983 heartbeat_check: no reply from 172.21.15.32:6828 osd.7 since back 2024-08-19T14:51:08.403986+0000 front 2024-08-19T14:51:08.404017+0000 (oldest deadline 2024-08-19T14:51:34.303826+0000) 2024-08-19T14:51:36.949 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:36 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:36.828+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.842003+0000 front 2024-08-19T14:50:12.842211+0000 (oldest deadline 2024-08-19T14:50:36.242616+0000) 2024-08-19T14:51:36.949 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:36 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:36.828+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.744487+0000 front 2024-08-19T14:50:23.744453+0000 (oldest deadline 2024-08-19T14:50:47.844529+0000) 2024-08-19T14:51:36.949 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:36 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:36.828+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:31.345568+0000 front 2024-08-19T14:50:31.345651+0000 (oldest deadline 2024-08-19T14:50:57.745917+0000) 2024-08-19T14:51:36.949 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:36 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:36.828+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6830 osd.3 since back 2024-08-19T14:50:38.846810+0000 front 2024-08-19T14:50:38.846826+0000 (oldest deadline 2024-08-19T14:51:01.746744+0000) 2024-08-19T14:51:36.949 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:36 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:36.828+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.32:6812 osd.5 since back 2024-08-19T14:50:52.848618+0000 front 2024-08-19T14:50:52.848721+0000 (oldest deadline 2024-08-19T14:51:18.148567+0000) 2024-08-19T14:51:36.949 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:36 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:36.828+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.32:6820 osd.6 since back 2024-08-19T14:51:02.150117+0000 front 2024-08-19T14:51:02.150094+0000 (oldest deadline 2024-08-19T14:51:28.050077+0000) 2024-08-19T14:51:36.949 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:36 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:36.829+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.32:6828 osd.7 since back 2024-08-19T14:51:08.551160+0000 front 2024-08-19T14:51:08.551120+0000 (oldest deadline 2024-08-19T14:51:33.251028+0000) 2024-08-19T14:51:37.869 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:37 smithi045 podman[167983]: 2024-08-19 14:51:37.56480763 +0000 UTC m=+5.623512393 container died cf08abdd92bff21b8971545e5c2328af695bc08a4dd9b8117d9789853dd5dcae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-de09dd0, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, org.label-schema.build-date=20240731, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-08-19T14:51:38.168 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:37 smithi045 podman[167983]: 2024-08-19 14:51:37.967347135 +0000 UTC m=+6.026051900 container cleanup cf08abdd92bff21b8971545e5c2328af695bc08a4dd9b8117d9789853dd5dcae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-de09dd0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0) 2024-08-19T14:51:38.174 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:37 smithi045 bash[167983]: ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10 2024-08-19T14:51:38.175 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:37 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:37.873+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.842003+0000 front 2024-08-19T14:50:12.842211+0000 (oldest deadline 2024-08-19T14:50:36.242616+0000) 2024-08-19T14:51:38.175 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:37 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:37.873+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.744487+0000 front 2024-08-19T14:50:23.744453+0000 (oldest deadline 2024-08-19T14:50:47.844529+0000) 2024-08-19T14:51:38.175 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:37 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:37.873+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:31.345568+0000 front 2024-08-19T14:50:31.345651+0000 (oldest deadline 2024-08-19T14:50:57.745917+0000) 2024-08-19T14:51:38.175 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:37 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:37.873+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6830 osd.3 since back 2024-08-19T14:50:38.846810+0000 front 2024-08-19T14:50:38.846826+0000 (oldest deadline 2024-08-19T14:51:01.746744+0000) 2024-08-19T14:51:38.175 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:37 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:37.873+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.32:6812 osd.5 since back 2024-08-19T14:50:52.848618+0000 front 2024-08-19T14:50:52.848721+0000 (oldest deadline 2024-08-19T14:51:18.148567+0000) 2024-08-19T14:51:38.175 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:37 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:37.873+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.32:6820 osd.6 since back 2024-08-19T14:51:02.150117+0000 front 2024-08-19T14:51:02.150094+0000 (oldest deadline 2024-08-19T14:51:28.050077+0000) 2024-08-19T14:51:38.175 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:37 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:37.873+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.32:6828 osd.7 since back 2024-08-19T14:51:08.551160+0000 front 2024-08-19T14:51:08.551120+0000 (oldest deadline 2024-08-19T14:51:33.251028+0000) 2024-08-19T14:51:38.531 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:38 smithi045 podman[168008]: 2024-08-19 14:51:38.19776426 +0000 UTC m=+0.628724601 container remove cf08abdd92bff21b8971545e5c2328af695bc08a4dd9b8117d9789853dd5dcae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=squid-de09dd0, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image) 2024-08-19T14:51:38.796 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:38 smithi045 podman[168097]: 2024-08-19 14:51:38.4338169 +0000 UTC m=+0.017069650 image pull ea7a5189f30bad3aca9524788e2965d2368d7ce786df09d4edcd8c3edf9cefba quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9 2024-08-19T14:51:38.796 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:38 smithi045 podman[168097]: 2024-08-19 14:51:38.544714078 +0000 UTC m=+0.127966824 container create 8ff57f42bbc49c1cc9fa80a53931021b87c290746c6fbdf26e5b1745f47a4bd8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10-deactivate, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=squid-de09dd0, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-08-19T14:51:38.796 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:38 smithi045 podman[168097]: 2024-08-19 14:51:38.772281269 +0000 UTC m=+0.355534029 container init 8ff57f42bbc49c1cc9fa80a53931021b87c290746c6fbdf26e5b1745f47a4bd8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, RELEASE=squid-de09dd0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-08-19T14:51:38.797 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:38 smithi045 podman[168097]: 2024-08-19 14:51:38.775652085 +0000 UTC m=+0.358904833 container start 8ff57f42bbc49c1cc9fa80a53931021b87c290746c6fbdf26e5b1745f47a4bd8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10-deactivate, GIT_CLEAN=True, RELEASE=squid-de09dd0, ceph=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-08-19T14:51:39.250 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:38 smithi045 podman[168097]: 2024-08-19 14:51:38.881018541 +0000 UTC m=+0.464271291 container attach 8ff57f42bbc49c1cc9fa80a53931021b87c290746c6fbdf26e5b1745f47a4bd8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-de09dd0) 2024-08-19T14:51:39.251 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:38 smithi045 podman[168097]: 2024-08-19 14:51:38.964618969 +0000 UTC m=+0.547871717 container died 8ff57f42bbc49c1cc9fa80a53931021b87c290746c6fbdf26e5b1745f47a4bd8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, RELEASE=squid-de09dd0, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-08-19T14:51:39.251 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:38 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:38.849+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.842003+0000 front 2024-08-19T14:50:12.842211+0000 (oldest deadline 2024-08-19T14:50:36.242616+0000) 2024-08-19T14:51:39.251 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:38 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:38.849+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.744487+0000 front 2024-08-19T14:50:23.744453+0000 (oldest deadline 2024-08-19T14:50:47.844529+0000) 2024-08-19T14:51:39.251 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:38 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:38.849+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:31.345568+0000 front 2024-08-19T14:50:31.345651+0000 (oldest deadline 2024-08-19T14:50:57.745917+0000) 2024-08-19T14:51:39.251 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:38 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:38.849+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6830 osd.3 since back 2024-08-19T14:50:38.846810+0000 front 2024-08-19T14:50:38.846826+0000 (oldest deadline 2024-08-19T14:51:01.746744+0000) 2024-08-19T14:51:39.251 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:38 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:38.849+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.32:6812 osd.5 since back 2024-08-19T14:50:52.848618+0000 front 2024-08-19T14:50:52.848721+0000 (oldest deadline 2024-08-19T14:51:18.148567+0000) 2024-08-19T14:51:39.251 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:38 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:38.849+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.32:6820 osd.6 since back 2024-08-19T14:51:02.150117+0000 front 2024-08-19T14:51:02.150094+0000 (oldest deadline 2024-08-19T14:51:28.050077+0000) 2024-08-19T14:51:39.251 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:38 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:38.849+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.32:6828 osd.7 since back 2024-08-19T14:51:08.551160+0000 front 2024-08-19T14:51:08.551120+0000 (oldest deadline 2024-08-19T14:51:33.251028+0000) 2024-08-19T14:51:39.507 INFO:journalctl@ceph.osd.10.smithi045.stdout:Aug 19 14:51:39 smithi045 podman[168120]: 2024-08-19 14:51:39.492922446 +0000 UTC m=+0.521405112 container remove 8ff57f42bbc49c1cc9fa80a53931021b87c290746c6fbdf26e5b1745f47a4bd8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-10-deactivate, GIT_CLEAN=True, RELEASE=squid-de09dd0, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-08-19T14:51:39.512 DEBUG:teuthology.orchestra.run.smithi045:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968@osd.10.service' 2024-08-19T14:51:39.551 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-19T14:51:39.551 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-08-19T14:51:39.551 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-08-19T14:51:39.551 DEBUG:teuthology.orchestra.run.smithi045:> sudo systemctl stop ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968@osd.11 2024-08-19T14:51:39.796 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:39 smithi045 systemd[1]: Stopping Ceph osd.11 for ed7d1c6e-5e33-11ef-bccf-c7b262605968... 2024-08-19T14:51:40.296 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:39 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:39.810+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.842003+0000 front 2024-08-19T14:50:12.842211+0000 (oldest deadline 2024-08-19T14:50:36.242616+0000) 2024-08-19T14:51:40.297 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:39 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:39.810+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.744487+0000 front 2024-08-19T14:50:23.744453+0000 (oldest deadline 2024-08-19T14:50:47.844529+0000) 2024-08-19T14:51:40.297 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:39 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:39.810+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:31.345568+0000 front 2024-08-19T14:50:31.345651+0000 (oldest deadline 2024-08-19T14:50:57.745917+0000) 2024-08-19T14:51:40.297 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:39 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:39.810+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6830 osd.3 since back 2024-08-19T14:50:38.846810+0000 front 2024-08-19T14:50:38.846826+0000 (oldest deadline 2024-08-19T14:51:01.746744+0000) 2024-08-19T14:51:40.297 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:39 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:39.810+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.32:6812 osd.5 since back 2024-08-19T14:50:52.848618+0000 front 2024-08-19T14:50:52.848721+0000 (oldest deadline 2024-08-19T14:51:18.148567+0000) 2024-08-19T14:51:40.297 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:39 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:39.810+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.32:6820 osd.6 since back 2024-08-19T14:51:02.150117+0000 front 2024-08-19T14:51:02.150094+0000 (oldest deadline 2024-08-19T14:51:28.050077+0000) 2024-08-19T14:51:40.297 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:39 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:39.810+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.32:6828 osd.7 since back 2024-08-19T14:51:08.551160+0000 front 2024-08-19T14:51:08.551120+0000 (oldest deadline 2024-08-19T14:51:33.251028+0000) 2024-08-19T14:51:40.297 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:39 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:39.912+0000 7f3e04254640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-19T14:51:40.297 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:39 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:39.912+0000 7f3e04254640 -1 osd.11 983 *** Got signal Terminated *** 2024-08-19T14:51:40.297 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:39 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:39.912+0000 7f3e04254640 -1 osd.11 983 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-08-19T14:51:41.046 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:40 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:40.770+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.842003+0000 front 2024-08-19T14:50:12.842211+0000 (oldest deadline 2024-08-19T14:50:36.242616+0000) 2024-08-19T14:51:41.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:40 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:40.770+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.744487+0000 front 2024-08-19T14:50:23.744453+0000 (oldest deadline 2024-08-19T14:50:47.844529+0000) 2024-08-19T14:51:41.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:40 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:40.770+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:31.345568+0000 front 2024-08-19T14:50:31.345651+0000 (oldest deadline 2024-08-19T14:50:57.745917+0000) 2024-08-19T14:51:41.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:40 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:40.770+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6830 osd.3 since back 2024-08-19T14:50:38.846810+0000 front 2024-08-19T14:50:38.846826+0000 (oldest deadline 2024-08-19T14:51:01.746744+0000) 2024-08-19T14:51:41.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:40 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:40.770+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.32:6812 osd.5 since back 2024-08-19T14:50:52.848618+0000 front 2024-08-19T14:50:52.848721+0000 (oldest deadline 2024-08-19T14:51:18.148567+0000) 2024-08-19T14:51:41.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:40 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:40.770+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.32:6820 osd.6 since back 2024-08-19T14:51:02.150117+0000 front 2024-08-19T14:51:02.150094+0000 (oldest deadline 2024-08-19T14:51:28.050077+0000) 2024-08-19T14:51:41.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:40 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:40.770+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.32:6828 osd.7 since back 2024-08-19T14:51:08.551160+0000 front 2024-08-19T14:51:08.551120+0000 (oldest deadline 2024-08-19T14:51:33.251028+0000) 2024-08-19T14:51:42.046 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:41 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:41.782+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.842003+0000 front 2024-08-19T14:50:12.842211+0000 (oldest deadline 2024-08-19T14:50:36.242616+0000) 2024-08-19T14:51:42.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:41 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:41.782+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.744487+0000 front 2024-08-19T14:50:23.744453+0000 (oldest deadline 2024-08-19T14:50:47.844529+0000) 2024-08-19T14:51:42.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:41 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:41.782+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:31.345568+0000 front 2024-08-19T14:50:31.345651+0000 (oldest deadline 2024-08-19T14:50:57.745917+0000) 2024-08-19T14:51:42.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:41 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:41.782+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6830 osd.3 since back 2024-08-19T14:50:38.846810+0000 front 2024-08-19T14:50:38.846826+0000 (oldest deadline 2024-08-19T14:51:01.746744+0000) 2024-08-19T14:51:42.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:41 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:41.782+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.32:6812 osd.5 since back 2024-08-19T14:50:52.848618+0000 front 2024-08-19T14:50:52.848721+0000 (oldest deadline 2024-08-19T14:51:18.148567+0000) 2024-08-19T14:51:42.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:41 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:41.782+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.32:6820 osd.6 since back 2024-08-19T14:51:02.150117+0000 front 2024-08-19T14:51:02.150094+0000 (oldest deadline 2024-08-19T14:51:28.050077+0000) 2024-08-19T14:51:42.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:41 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:41.782+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.32:6828 osd.7 since back 2024-08-19T14:51:08.551160+0000 front 2024-08-19T14:51:08.551120+0000 (oldest deadline 2024-08-19T14:51:33.251028+0000) 2024-08-19T14:51:43.297 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:42 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:42.810+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.842003+0000 front 2024-08-19T14:50:12.842211+0000 (oldest deadline 2024-08-19T14:50:36.242616+0000) 2024-08-19T14:51:43.297 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:42 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:42.810+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.744487+0000 front 2024-08-19T14:50:23.744453+0000 (oldest deadline 2024-08-19T14:50:47.844529+0000) 2024-08-19T14:51:43.297 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:42 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:42.810+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:31.345568+0000 front 2024-08-19T14:50:31.345651+0000 (oldest deadline 2024-08-19T14:50:57.745917+0000) 2024-08-19T14:51:43.297 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:42 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:42.810+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6830 osd.3 since back 2024-08-19T14:50:38.846810+0000 front 2024-08-19T14:50:38.846826+0000 (oldest deadline 2024-08-19T14:51:01.746744+0000) 2024-08-19T14:51:43.297 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:42 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:42.810+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.32:6812 osd.5 since back 2024-08-19T14:50:52.848618+0000 front 2024-08-19T14:50:52.848721+0000 (oldest deadline 2024-08-19T14:51:18.148567+0000) 2024-08-19T14:51:43.297 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:42 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:42.810+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.32:6820 osd.6 since back 2024-08-19T14:51:02.150117+0000 front 2024-08-19T14:51:02.150094+0000 (oldest deadline 2024-08-19T14:51:28.050077+0000) 2024-08-19T14:51:43.297 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:42 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:42.810+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.32:6828 osd.7 since back 2024-08-19T14:51:08.551160+0000 front 2024-08-19T14:51:08.551120+0000 (oldest deadline 2024-08-19T14:51:33.251028+0000) 2024-08-19T14:51:44.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:43 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:43.767+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.842003+0000 front 2024-08-19T14:50:12.842211+0000 (oldest deadline 2024-08-19T14:50:36.242616+0000) 2024-08-19T14:51:44.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:43 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:43.767+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.744487+0000 front 2024-08-19T14:50:23.744453+0000 (oldest deadline 2024-08-19T14:50:47.844529+0000) 2024-08-19T14:51:44.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:43 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:43.767+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:31.345568+0000 front 2024-08-19T14:50:31.345651+0000 (oldest deadline 2024-08-19T14:50:57.745917+0000) 2024-08-19T14:51:44.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:43 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:43.767+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6830 osd.3 since back 2024-08-19T14:50:38.846810+0000 front 2024-08-19T14:50:38.846826+0000 (oldest deadline 2024-08-19T14:51:01.746744+0000) 2024-08-19T14:51:44.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:43 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:43.767+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.32:6812 osd.5 since back 2024-08-19T14:50:52.848618+0000 front 2024-08-19T14:50:52.848721+0000 (oldest deadline 2024-08-19T14:51:18.148567+0000) 2024-08-19T14:51:44.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:43 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:43.767+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.32:6820 osd.6 since back 2024-08-19T14:51:02.150117+0000 front 2024-08-19T14:51:02.150094+0000 (oldest deadline 2024-08-19T14:51:28.050077+0000) 2024-08-19T14:51:44.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:43 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:43.767+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.32:6828 osd.7 since back 2024-08-19T14:51:08.551160+0000 front 2024-08-19T14:51:08.551120+0000 (oldest deadline 2024-08-19T14:51:33.251028+0000) 2024-08-19T14:51:45.046 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:44 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:44.780+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-08-19T14:50:12.842003+0000 front 2024-08-19T14:50:12.842211+0000 (oldest deadline 2024-08-19T14:50:36.242616+0000) 2024-08-19T14:51:45.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:44 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:44.780+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2024-08-19T14:50:23.744487+0000 front 2024-08-19T14:50:23.744453+0000 (oldest deadline 2024-08-19T14:50:47.844529+0000) 2024-08-19T14:51:45.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:44 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:44.780+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6822 osd.2 since back 2024-08-19T14:50:31.345568+0000 front 2024-08-19T14:50:31.345651+0000 (oldest deadline 2024-08-19T14:50:57.745917+0000) 2024-08-19T14:51:45.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:44 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:44.780+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.31:6830 osd.3 since back 2024-08-19T14:50:38.846810+0000 front 2024-08-19T14:50:38.846826+0000 (oldest deadline 2024-08-19T14:51:01.746744+0000) 2024-08-19T14:51:45.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:44 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:44.780+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.32:6812 osd.5 since back 2024-08-19T14:50:52.848618+0000 front 2024-08-19T14:50:52.848721+0000 (oldest deadline 2024-08-19T14:51:18.148567+0000) 2024-08-19T14:51:45.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:44 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:44.780+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.32:6820 osd.6 since back 2024-08-19T14:51:02.150117+0000 front 2024-08-19T14:51:02.150094+0000 (oldest deadline 2024-08-19T14:51:28.050077+0000) 2024-08-19T14:51:45.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:44 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:44.780+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.32:6828 osd.7 since back 2024-08-19T14:51:08.551160+0000 front 2024-08-19T14:51:08.551120+0000 (oldest deadline 2024-08-19T14:51:33.251028+0000) 2024-08-19T14:51:45.047 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:44 smithi045 ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11[41957]: 2024-08-19T14:51:44.780+0000 7f3dfc862640 -1 osd.11 983 heartbeat_check: no reply from 172.21.15.45:6804 osd.8 since back 2024-08-19T14:51:18.853004+0000 front 2024-08-19T14:51:18.853035+0000 (oldest deadline 2024-08-19T14:51:44.653790+0000) 2024-08-19T14:51:45.627 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:45 smithi045 podman[168199]: 2024-08-19 14:51:45.358404369 +0000 UTC m=+5.578210768 container died 9b146cc9e48321677f6ad5130559b739994f48ea9cec262116ed474ca78c3469 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=squid-de09dd0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0) 2024-08-19T14:51:45.963 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:45 smithi045 podman[168199]: 2024-08-19 14:51:45.728917698 +0000 UTC m=+5.948724103 container cleanup 9b146cc9e48321677f6ad5130559b739994f48ea9cec262116ed474ca78c3469 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, RELEASE=squid-de09dd0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-08-19T14:51:45.964 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:45 smithi045 bash[168199]: ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11 2024-08-19T14:51:46.292 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:45 smithi045 podman[168217]: 2024-08-19 14:51:45.963451458 +0000 UTC m=+0.607859789 container remove 9b146cc9e48321677f6ad5130559b739994f48ea9cec262116ed474ca78c3469 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=squid-de09dd0, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, ceph=True) 2024-08-19T14:51:46.545 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:46 smithi045 podman[168305]: 2024-08-19 14:51:46.19449076 +0000 UTC m=+0.017287452 image pull ea7a5189f30bad3aca9524788e2965d2368d7ce786df09d4edcd8c3edf9cefba quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9 2024-08-19T14:51:46.545 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:46 smithi045 podman[168305]: 2024-08-19 14:51:46.322791908 +0000 UTC m=+0.145588632 container create 57a0091e17d293e9d7ecf556b885513e1c48302f7dec6390c33c100433a52859 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240731, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, RELEASE=squid-de09dd0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-08-19T14:51:46.546 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:46 smithi045 podman[168305]: 2024-08-19 14:51:46.541972098 +0000 UTC m=+0.364768795 container init 57a0091e17d293e9d7ecf556b885513e1c48302f7dec6390c33c100433a52859 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, RELEASE=squid-de09dd0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS) 2024-08-19T14:51:46.546 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:46 smithi045 podman[168305]: 2024-08-19 14:51:46.545453717 +0000 UTC m=+0.368250398 container start 57a0091e17d293e9d7ecf556b885513e1c48302f7dec6390c33c100433a52859 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11-deactivate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=squid-de09dd0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-08-19T14:51:46.796 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:46 smithi045 podman[168305]: 2024-08-19 14:51:46.650289335 +0000 UTC m=+0.473086027 container attach 57a0091e17d293e9d7ecf556b885513e1c48302f7dec6390c33c100433a52859 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=squid-de09dd0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.build-date=20240731) 2024-08-19T14:51:46.796 INFO:journalctl@ceph.osd.11.smithi045.stdout:Aug 19 14:51:46 smithi045 podman[168305]: 2024-08-19 14:51:46.737373689 +0000 UTC m=+0.560170423 container died 57a0091e17d293e9d7ecf556b885513e1c48302f7dec6390c33c100433a52859 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1c9dd28080a631be25c77c36c63b17a3ef340ffb2e21c87d4d7b56b6398436d9, name=ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968-osd-11-deactivate, RELEASE=squid-de09dd0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-08-19T14:51:47.301 DEBUG:teuthology.orchestra.run.smithi045:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ed7d1c6e-5e33-11ef-bccf-c7b262605968@osd.11.service' 2024-08-19T14:51:47.338 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-19T14:51:47.339 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-08-19T14:51:47.339 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 --force --keep-logs 2024-08-19T14:51:47.547 INFO:teuthology.orchestra.run.smithi031.stdout:Deleting cluster with fsid: ed7d1c6e-5e33-11ef-bccf-c7b262605968 2024-08-19T14:51:49.329 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 --force --keep-logs 2024-08-19T14:51:49.540 INFO:teuthology.orchestra.run.smithi032.stdout:Deleting cluster with fsid: ed7d1c6e-5e33-11ef-bccf-c7b262605968 2024-08-19T14:51:51.382 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 --force --keep-logs 2024-08-19T14:51:51.592 INFO:teuthology.orchestra.run.smithi045.stdout:Deleting cluster with fsid: ed7d1c6e-5e33-11ef-bccf-c7b262605968 2024-08-19T14:51:53.127 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-08-19T14:51:53.159 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-08-19T14:51:53.192 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-08-19T14:51:53.219 INFO:tasks.cephadm:Archiving crash dumps... 2024-08-19T14:51:53.221 DEBUG:teuthology.misc:Transferring archived files from smithi031:/var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-08-18_21:00:03-rados-squid-distro-default-smithi/7859709/remote/smithi031/crash 2024-08-19T14:51:53.222 DEBUG:teuthology.orchestra.run.smithi031:> sudo tar c -f - -C /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/crash -- . 2024-08-19T14:51:53.259 INFO:teuthology.orchestra.run.smithi031.stderr:tar: /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/crash: Cannot open: No such file or directory 2024-08-19T14:51:53.259 INFO:teuthology.orchestra.run.smithi031.stderr:tar: Error is not recoverable: exiting now 2024-08-19T14:51:53.262 DEBUG:teuthology.misc:Transferring archived files from smithi032:/var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-08-18_21:00:03-rados-squid-distro-default-smithi/7859709/remote/smithi032/crash 2024-08-19T14:51:53.262 DEBUG:teuthology.orchestra.run.smithi032:> sudo tar c -f - -C /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/crash -- . 2024-08-19T14:51:53.303 INFO:teuthology.orchestra.run.smithi032.stderr:tar: /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/crash: Cannot open: No such file or directory 2024-08-19T14:51:53.303 INFO:teuthology.orchestra.run.smithi032.stderr:tar: Error is not recoverable: exiting now 2024-08-19T14:51:53.305 DEBUG:teuthology.misc:Transferring archived files from smithi045:/var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-08-18_21:00:03-rados-squid-distro-default-smithi/7859709/remote/smithi045/crash 2024-08-19T14:51:53.306 DEBUG:teuthology.orchestra.run.smithi045:> sudo tar c -f - -C /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/crash -- . 2024-08-19T14:51:53.342 INFO:teuthology.orchestra.run.smithi045.stderr:tar: /var/lib/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/crash: Cannot open: No such file or directory 2024-08-19T14:51:53.343 INFO:teuthology.orchestra.run.smithi045.stderr:tar: Error is not recoverable: exiting now 2024-08-19T14:51:53.344 INFO:tasks.cephadm:Checking cluster log for badness... 2024-08-19T14:51:53.344 DEBUG:teuthology.orchestra.run.smithi031:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(MON_DOWN\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | egrep -v noscrub | egrep -v nodeep-scrub | head -n 1 2024-08-19T14:51:53.427 INFO:teuthology.orchestra.run.smithi031.stdout:2024-08-19T14:40:00.000097+0000 mon.a (mon.0) 3045 : cluster [WRN] Health detail: HEALTH_WARN Degraded data redundancy: 406/6 objects degraded (6766.667%), 2 pgs degraded, 3 pgs undersized 2024-08-19T14:51:53.427 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-08-19T14:51:53.427 DEBUG:teuthology.orchestra.run.smithi031:> sudo egrep '\[SEC\]' /var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(MON_DOWN\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | egrep -v noscrub | egrep -v nodeep-scrub | head -n 1 2024-08-19T14:51:53.464 DEBUG:teuthology.orchestra.run.smithi031:> sudo egrep '\[ERR\]' /var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(MON_DOWN\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | egrep -v noscrub | egrep -v nodeep-scrub | head -n 1 2024-08-19T14:51:53.541 DEBUG:teuthology.orchestra.run.smithi031:> sudo egrep '\[WRN\]' /var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(MON_DOWN\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | egrep -v noscrub | egrep -v nodeep-scrub | head -n 1 2024-08-19T14:51:53.616 INFO:teuthology.orchestra.run.smithi031.stdout:2024-08-19T14:40:00.000097+0000 mon.a (mon.0) 3045 : cluster [WRN] Health detail: HEALTH_WARN Degraded data redundancy: 406/6 objects degraded (6766.667%), 2 pgs degraded, 3 pgs undersized 2024-08-19T14:51:53.617 INFO:tasks.cephadm:Compressing logs... 2024-08-19T14:51:53.617 DEBUG:teuthology.orchestra.run.smithi031:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-08-19T14:51:53.659 DEBUG:teuthology.orchestra.run.smithi032:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-08-19T14:51:53.661 DEBUG:teuthology.orchestra.run.smithi045:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-08-19T14:51:53.695 INFO:teuthology.orchestra.run.smithi031.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-08-19T14:51:53.695 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/objectstore_tool.2.log 2024-08-19T14:51:53.695 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/ceph-volume.log 2024-08-19T14:51:53.698 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose --/var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/objectstore_tool.2.log: /var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/ceph-osd.2.log 2024-08-19T14:51:53.698 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/ceph-osd.3.log 2024-08-19T14:51:53.698 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/ceph.cephadm.log 2024-08-19T14:51:53.698 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/ceph.audit.log 2024-08-19T14:51:53.699 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/ceph-mon.a.log 2024-08-19T14:51:53.699 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/ceph-osd.0.log 2024-08-19T14:51:53.699 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/ceph-mgr.y.log 2024-08-19T14:51:53.699 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/ceph-osd.1.log 2024-08-19T14:51:53.699 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/ceph.log 2024-08-19T14:51:53.699 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-08-19T14:51:53.710 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/ceph.log: /var/log/ceph/cephadm.log: 85.5% -- replaced with /var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/ceph.log.gz 2024-08-19T14:51:53.710 INFO:teuthology.orchestra.run.smithi045.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-08-19T14:51:53.710 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/objectstore_tool.2.log 2024-08-19T14:51:53.711 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/ceph-volume.log 2024-08-19T14:51:53.711 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/ceph-osd.11.log 2024-08-19T14:51:53.711 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/objectstore_tool.2.log: /var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/ceph.cephadm.log 2024-08-19T14:51:53.711 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/ceph.audit.log 2024-08-19T14:51:53.714 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/ceph-osd.10.log 2024-08-19T14:51:53.714 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/ceph-osd.8.log 2024-08-19T14:51:53.714 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/ceph-osd.9.log 2024-08-19T14:51:53.715 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/ceph.log 2024-08-19T14:51:53.715 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/ceph-osd.9.log: gzip -5 --verbose -- /var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/ceph-mon.c.log 2024-08-19T14:51:53.715 INFO:teuthology.orchestra.run.smithi045.stderr:gzip/var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/ceph.log: -5 --verbose -- /var/log/ceph/cephadm.log 2024-08-19T14:51:53.716 INFO:teuthology.orchestra.run.smithi032.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-08-19T14:51:53.716 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/ceph-volume.log 2024-08-19T14:51:53.716 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/ceph-mgr.x.log 2024-08-19T14:51:53.716 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/ceph.cephadm.log 2024-08-19T14:51:53.716 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/ceph-volume.log: /var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/ceph.audit.log 2024-08-19T14:51:53.716 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/ceph-osd.4.log 2024-08-19T14:51:53.716 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/ceph-mon.b.log 2024-08-19T14:51:53.717 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/ceph-osd.6.log 2024-08-19T14:51:53.717 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/ceph-osd.7.log 2024-08-19T14:51:53.717 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/ceph-osd.5.log 2024-08-19T14:51:53.717 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/ceph.log 2024-08-19T14:51:53.717 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-08-19T14:51:53.717 INFO:teuthology.orchestra.run.smithi031.stderr: 82.3% -- replaced with /var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/ceph.cephadm.log.gz 2024-08-19T14:51:53.717 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/ceph-mon.c.log: /var/log/ceph/cephadm.log: 82.0% -- replaced with /var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/ceph.cephadm.log.gz 2024-08-19T14:51:53.744 INFO:teuthology.orchestra.run.smithi031.stderr: 95.8% -- replaced with /var/log/ceph/cephadm.log.gz 2024-08-19T14:51:53.770 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/ceph.log: /var/log/ceph/cephadm.log: 95.5% -- replaced with /var/log/ceph/cephadm.log.gz 2024-08-19T14:51:53.778 INFO:teuthology.orchestra.run.smithi045.stderr: 95.5% -- replaced with /var/log/ceph/cephadm.log.gz 2024-08-19T14:51:53.865 INFO:teuthology.orchestra.run.smithi032.stderr: 91.2% -- replaced with /var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/ceph-volume.log.gz 2024-08-19T14:51:53.887 INFO:teuthology.orchestra.run.smithi031.stderr: 91.1% -- replaced with /var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/ceph-volume.log.gz 2024-08-19T14:51:53.887 INFO:teuthology.orchestra.run.smithi032.stderr: 82.1% -- replaced with /var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/ceph.cephadm.log.gz 2024-08-19T14:51:53.895 INFO:teuthology.orchestra.run.smithi032.stderr: 93.3% -- replaced with /var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/ceph-mgr.x.log.gz 2024-08-19T14:51:53.924 INFO:teuthology.orchestra.run.smithi045.stderr: 91.1% -- replaced with /var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/ceph-volume.log.gz 2024-08-19T14:51:53.992 INFO:teuthology.orchestra.run.smithi032.stderr: 89.8% -- replaced with /var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/ceph.audit.log.gz 2024-08-19T14:51:54.105 INFO:teuthology.orchestra.run.smithi045.stderr: 94.7% -- replaced with /var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/objectstore_tool.2.log.gz 2024-08-19T14:51:54.113 INFO:teuthology.orchestra.run.smithi032.stderr: 85.5% -- replaced with /var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/ceph.log.gz 2024-08-19T14:51:54.133 INFO:teuthology.orchestra.run.smithi045.stderr: 85.5% -- replaced with /var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/ceph.log.gz 2024-08-19T14:51:54.174 INFO:teuthology.orchestra.run.smithi031.stderr: 89.8% -- replaced with /var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/ceph.audit.log.gz 2024-08-19T14:51:54.244 INFO:teuthology.orchestra.run.smithi031.stderr: 94.3% -- replaced with /var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/objectstore_tool.2.log.gz 2024-08-19T14:51:54.353 INFO:teuthology.orchestra.run.smithi045.stderr: 89.8% -- replaced with /var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/ceph.audit.log.gz 2024-08-19T14:51:54.473 INFO:teuthology.orchestra.run.smithi031.stderr: 91.5% -- replaced with /var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/ceph-mgr.y.log.gz 2024-08-19T14:52:01.264 INFO:teuthology.orchestra.run.smithi032.stderr: 91.5% -- replaced with /var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/ceph-mon.b.log.gz 2024-08-19T14:52:11.594 INFO:teuthology.orchestra.run.smithi045.stderr: 91.5% -- replaced with /var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/ceph-mon.c.log.gz 2024-08-19T14:52:21.486 INFO:teuthology.orchestra.run.smithi031.stderr: 90.3% -- replaced with /var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/ceph-mon.a.log.gz 2024-08-19T14:52:28.012 INFO:teuthology.orchestra.run.smithi032.stderr: 93.4% -- replaced with /var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/ceph-osd.6.log.gz 2024-08-19T14:53:19.235 INFO:teuthology.orchestra.run.smithi045.stderr: 91.3% -- replaced with /var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/ceph-osd.10.log.gz 2024-08-19T15:04:15.852 INFO:teuthology.orchestra.run.smithi045.stderr: 93.6% -- replaced with /var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/ceph-osd.9.log.gz 2024-08-19T15:09:07.123 INFO:teuthology.orchestra.run.smithi031.stderr: 92.4% -- replaced with /var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/ceph-osd.2.log.gz 2024-08-19T15:14:20.418 INFO:teuthology.orchestra.run.smithi032.stderr: 92.2% -- replaced with /var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/ceph-osd.5.log.gz 2024-08-19T15:14:40.192 INFO:teuthology.orchestra.run.smithi045.stderr: 92.2% -- replaced with /var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/ceph-osd.8.log.gz 2024-08-19T15:15:18.196 INFO:teuthology.orchestra.run.smithi045.stderr: 92.2% -- replaced with /var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/ceph-osd.11.log.gz 2024-08-19T15:15:18.216 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-08-19T15:15:18.216 INFO:teuthology.orchestra.run.smithi045.stderr:real 23m24.539s 2024-08-19T15:15:18.216 INFO:teuthology.orchestra.run.smithi045.stderr:user 9m40.040s 2024-08-19T15:15:18.216 INFO:teuthology.orchestra.run.smithi045.stderr:sys 0m26.433s 2024-08-19T15:23:54.218 INFO:teuthology.orchestra.run.smithi032.stderr: 91.3% -- replaced with /var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/ceph-osd.7.log.gz 2024-08-19T15:24:15.086 INFO:teuthology.orchestra.run.smithi032.stderr: 91.6% -- replaced with /var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/ceph-osd.4.log.gz 2024-08-19T15:24:15.139 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-08-19T15:24:15.140 INFO:teuthology.orchestra.run.smithi032.stderr:real 32m21.462s 2024-08-19T15:24:15.140 INFO:teuthology.orchestra.run.smithi032.stderr:user 12m20.164s 2024-08-19T15:24:15.140 INFO:teuthology.orchestra.run.smithi032.stderr:sys 0m33.540s 2024-08-19T15:43:05.920 INFO:teuthology.orchestra.run.smithi031.stderr: 91.4% -- replaced with /var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/ceph-osd.0.log.gz 2024-08-19T15:46:16.955 INFO:teuthology.orchestra.run.smithi031.stderr: 91.0% -- replaced with /var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/ceph-osd.3.log.gz 2024-08-19T15:47:46.716 INFO:teuthology.orchestra.run.smithi031.stderr: 91.3% -- replaced with /var/log/ceph/ed7d1c6e-5e33-11ef-bccf-c7b262605968/ceph-osd.1.log.gz 2024-08-19T15:47:46.788 INFO:teuthology.orchestra.run.smithi031.stderr: 2024-08-19T15:47:46.788 INFO:teuthology.orchestra.run.smithi031.stderr:real 55m53.115s 2024-08-19T15:47:46.788 INFO:teuthology.orchestra.run.smithi031.stderr:user 19m56.716s 2024-08-19T15:47:46.788 INFO:teuthology.orchestra.run.smithi031.stderr:sys 0m52.300s 2024-08-19T15:47:46.789 INFO:tasks.cephadm:Archiving logs... 2024-08-19T15:47:46.791 DEBUG:teuthology.misc:Transferring archived files from smithi031:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-08-18_21:00:03-rados-squid-distro-default-smithi/7859709/remote/smithi031/log 2024-08-19T15:47:46.792 DEBUG:teuthology.orchestra.run.smithi031:> sudo tar c -f - -C /var/log/ceph -- . 2024-08-19T15:51:08.806 DEBUG:teuthology.misc:Transferring archived files from smithi032:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-08-18_21:00:03-rados-squid-distro-default-smithi/7859709/remote/smithi032/log 2024-08-19T15:51:08.824 DEBUG:teuthology.orchestra.run.smithi032:> sudo tar c -f - -C /var/log/ceph -- . 2024-08-19T15:53:10.982 DEBUG:teuthology.misc:Transferring archived files from smithi045:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-08-18_21:00:03-rados-squid-distro-default-smithi/7859709/remote/smithi045/log 2024-08-19T15:53:10.983 DEBUG:teuthology.orchestra.run.smithi045:> sudo tar c -f - -C /var/log/ceph -- . 2024-08-19T15:54:42.352 INFO:tasks.cephadm:Removing cluster... 2024-08-19T15:54:42.352 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 --force 2024-08-19T15:54:42.920 INFO:teuthology.orchestra.run.smithi031.stdout:Deleting cluster with fsid: ed7d1c6e-5e33-11ef-bccf-c7b262605968 2024-08-19T15:54:44.223 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 --force 2024-08-19T15:54:44.850 INFO:teuthology.orchestra.run.smithi032.stdout:Deleting cluster with fsid: ed7d1c6e-5e33-11ef-bccf-c7b262605968 2024-08-19T15:54:45.863 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid ed7d1c6e-5e33-11ef-bccf-c7b262605968 --force 2024-08-19T15:54:46.559 INFO:teuthology.orchestra.run.smithi045.stdout:Deleting cluster with fsid: ed7d1c6e-5e33-11ef-bccf-c7b262605968 2024-08-19T15:54:47.584 INFO:tasks.cephadm:Removing cephadm ... 2024-08-19T15:54:47.584 DEBUG:teuthology.orchestra.run.smithi031:> rm -rf /home/ubuntu/cephtest/cephadm 2024-08-19T15:54:47.604 DEBUG:teuthology.orchestra.run.smithi032:> rm -rf /home/ubuntu/cephtest/cephadm 2024-08-19T15:54:47.633 DEBUG:teuthology.orchestra.run.smithi045:> rm -rf /home/ubuntu/cephtest/cephadm 2024-08-19T15:54:47.649 INFO:tasks.cephadm:Teardown complete 2024-08-19T15:54:47.649 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-08-19T15:54:47.659 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-08-19T15:54:47.659 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-08-19T15:54:47.662 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-08-19T15:54:47.676 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-08-19T15:54:47.838 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-08-19T15:54:47.849 INFO:teuthology.task.clock:Checking final clock skew... 2024-08-19T15:54:47.849 DEBUG:teuthology.orchestra.run.smithi031:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-19T15:54:47.852 DEBUG:teuthology.orchestra.run.smithi032:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-19T15:54:47.857 DEBUG:teuthology.orchestra.run.smithi045:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-19T15:54:47.867 INFO:teuthology.orchestra.run.smithi031.stderr:bash: line 1: ntpq: command not found 2024-08-19T15:54:47.870 INFO:teuthology.orchestra.run.smithi032.stderr:bash: line 1: ntpq: command not found 2024-08-19T15:54:47.872 INFO:teuthology.orchestra.run.smithi045.stderr:bash: line 1: ntpq: command not found 2024-08-19T15:54:47.931 INFO:teuthology.orchestra.run.smithi032.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-08-19T15:54:47.932 INFO:teuthology.orchestra.run.smithi032.stdout:=============================================================================== 2024-08-19T15:54:47.932 INFO:teuthology.orchestra.run.smithi032.stdout:^* hv01.front.sepia.ceph.com 3 6 377 58 +507us[ +499us] +/- 45ms 2024-08-19T15:54:47.932 INFO:teuthology.orchestra.run.smithi032.stdout:^+ hv02.front.sepia.ceph.com 3 7 377 3 -947us[ -947us] +/- 54ms 2024-08-19T15:54:47.932 INFO:teuthology.orchestra.run.smithi032.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 59 +214us[ +206us] +/- 71ms 2024-08-19T15:54:47.932 INFO:teuthology.orchestra.run.smithi032.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-08-19T15:54:47.943 INFO:teuthology.orchestra.run.smithi045.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-08-19T15:54:47.943 INFO:teuthology.orchestra.run.smithi045.stdout:=============================================================================== 2024-08-19T15:54:47.943 INFO:teuthology.orchestra.run.smithi045.stdout:^* hv01.front.sepia.ceph.com 3 6 377 63 +521us[ +520us] +/- 45ms 2024-08-19T15:54:47.943 INFO:teuthology.orchestra.run.smithi045.stdout:^+ hv02.front.sepia.ceph.com 3 7 377 61 -895us[ -895us] +/- 53ms 2024-08-19T15:54:47.943 INFO:teuthology.orchestra.run.smithi045.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 1 +229us[ +229us] +/- 72ms 2024-08-19T15:54:47.943 INFO:teuthology.orchestra.run.smithi045.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-08-19T15:54:47.943 INFO:teuthology.orchestra.run.smithi031.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-08-19T15:54:47.944 INFO:teuthology.orchestra.run.smithi031.stdout:=============================================================================== 2024-08-19T15:54:47.944 INFO:teuthology.orchestra.run.smithi031.stdout:^* hv01.front.sepia.ceph.com 3 8 377 190 +514us[ +414us] +/- 43ms 2024-08-19T15:54:47.944 INFO:teuthology.orchestra.run.smithi031.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 1 -970us[ -970us] +/- 54ms 2024-08-19T15:54:47.944 INFO:teuthology.orchestra.run.smithi031.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 60 +232us[ +232us] +/- 71ms 2024-08-19T15:54:47.944 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-08-19T15:54:47.960 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-08-19T15:54:47.969 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-08-19T15:54:47.969 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-08-19T15:54:47.981 DEBUG:teuthology.orchestra.run.smithi031:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-08-19T15:54:48.094 DEBUG:teuthology.orchestra.run.smithi032:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-08-19T15:54:48.164 DEBUG:teuthology.orchestra.run.smithi045:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-08-19T15:54:48.226 DEBUG:teuthology.orchestra.run.smithi031:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-08-19T15:54:48.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T15:54:48.256 DEBUG:teuthology.orchestra.run.smithi032:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-08-19T15:54:48.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T15:54:48.286 DEBUG:teuthology.orchestra.run.smithi045:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-08-19T15:54:48.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T15:54:48.313 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-08-19T15:54:48.322 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-08-19T15:54:48.391 INFO:teuthology.task.internal:Duration was 6840.476485 seconds 2024-08-19T15:54:48.391 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-08-19T15:54:48.402 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-08-19T15:54:48.402 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-08-19T15:54:48.405 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-08-19T15:54:48.408 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-08-19T15:54:48.464 INFO:teuthology.orchestra.run.smithi045.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-08-19T15:54:48.480 INFO:teuthology.orchestra.run.smithi031.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-08-19T15:54:48.484 INFO:teuthology.orchestra.run.smithi032.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-08-19T15:54:48.954 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-08-19T15:54:48.955 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi031.front.sepia.ceph.com 2024-08-19T15:54:48.955 DEBUG:teuthology.orchestra.run.smithi031:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-08-19T15:54:48.978 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi032.front.sepia.ceph.com 2024-08-19T15:54:48.978 DEBUG:teuthology.orchestra.run.smithi032:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-08-19T15:54:49.009 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi045.front.sepia.ceph.com 2024-08-19T15:54:49.009 DEBUG:teuthology.orchestra.run.smithi045:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-08-19T15:54:49.042 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-08-19T15:54:49.043 DEBUG:teuthology.orchestra.run.smithi031:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-08-19T15:54:49.045 DEBUG:teuthology.orchestra.run.smithi032:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-08-19T15:54:49.057 DEBUG:teuthology.orchestra.run.smithi045:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-08-19T15:54:49.113 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-08-19T15:54:49.113 DEBUG:teuthology.orchestra.run.smithi031:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-08-19T15:54:49.126 DEBUG:teuthology.orchestra.run.smithi032:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-08-19T15:54:49.131 DEBUG:teuthology.orchestra.run.smithi045:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-08-19T15:54:51.360 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-08-19T15:54:51.371 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-08-19T15:54:51.371 DEBUG:teuthology.orchestra.run.smithi031:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-08-19T15:54:51.410 DEBUG:teuthology.orchestra.run.smithi032:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-08-19T15:54:51.442 DEBUG:teuthology.orchestra.run.smithi045:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-08-19T15:54:51.502 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-08-19T15:54:51.512 DEBUG:teuthology.orchestra.run.smithi031:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-08-19T15:54:51.515 DEBUG:teuthology.orchestra.run.smithi032:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-08-19T15:54:51.518 DEBUG:teuthology.orchestra.run.smithi045:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-08-19T15:54:51.560 INFO:teuthology.orchestra.run.smithi032.stdout:kernel.core_pattern = core 2024-08-19T15:54:51.563 INFO:teuthology.orchestra.run.smithi045.stdout:kernel.core_pattern = core 2024-08-19T15:54:51.574 INFO:teuthology.orchestra.run.smithi031.stdout:kernel.core_pattern = core 2024-08-19T15:54:51.591 DEBUG:teuthology.orchestra.run.smithi031:> test -e /home/ubuntu/cephtest/archive/coredump 2024-08-19T15:54:51.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T15:54:51.606 DEBUG:teuthology.orchestra.run.smithi032:> test -e /home/ubuntu/cephtest/archive/coredump 2024-08-19T15:54:51.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T15:54:51.636 DEBUG:teuthology.orchestra.run.smithi045:> test -e /home/ubuntu/cephtest/archive/coredump 2024-08-19T15:54:51.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T15:54:51.652 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-08-19T15:54:51.669 INFO:teuthology.task.internal:Transferring archived files... 2024-08-19T15:54:51.671 DEBUG:teuthology.misc:Transferring archived files from smithi031:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-08-18_21:00:03-rados-squid-distro-default-smithi/7859709/remote/smithi031 2024-08-19T15:54:51.672 DEBUG:teuthology.orchestra.run.smithi031:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-08-19T15:54:51.752 DEBUG:teuthology.misc:Transferring archived files from smithi032:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-08-18_21:00:03-rados-squid-distro-default-smithi/7859709/remote/smithi032 2024-08-19T15:54:51.752 DEBUG:teuthology.orchestra.run.smithi032:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-08-19T15:54:51.795 DEBUG:teuthology.misc:Transferring archived files from smithi045:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-08-18_21:00:03-rados-squid-distro-default-smithi/7859709/remote/smithi045 2024-08-19T15:54:51.795 DEBUG:teuthology.orchestra.run.smithi045:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-08-19T15:54:51.838 INFO:teuthology.task.internal:Removing archive directory... 2024-08-19T15:54:51.838 DEBUG:teuthology.orchestra.run.smithi031:> rm -rf -- /home/ubuntu/cephtest/archive 2024-08-19T15:54:51.841 DEBUG:teuthology.orchestra.run.smithi032:> rm -rf -- /home/ubuntu/cephtest/archive 2024-08-19T15:54:51.843 DEBUG:teuthology.orchestra.run.smithi045:> rm -rf -- /home/ubuntu/cephtest/archive 2024-08-19T15:54:51.888 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-08-19T15:54:51.900 INFO:teuthology.task.internal:Not uploading archives. 2024-08-19T15:54:51.900 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-08-19T15:54:51.909 INFO:teuthology.task.internal:Tidying up after the test... 2024-08-19T15:54:51.909 DEBUG:teuthology.orchestra.run.smithi031:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-08-19T15:54:51.912 DEBUG:teuthology.orchestra.run.smithi032:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-08-19T15:54:51.914 DEBUG:teuthology.orchestra.run.smithi045:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-08-19T15:54:51.924 INFO:teuthology.orchestra.run.smithi031.stdout: 262218 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Aug 19 15:54 /home/ubuntu/cephtest 2024-08-19T15:54:51.929 INFO:teuthology.orchestra.run.smithi032.stdout: 262218 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Aug 19 15:54 /home/ubuntu/cephtest 2024-08-19T15:54:51.943 INFO:teuthology.orchestra.run.smithi045.stdout: 262218 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Aug 19 15:54 /home/ubuntu/cephtest 2024-08-19T15:54:51.945 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-08-19T15:54:51.955 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-08-19T15:54:51.979 INFO:teuthology.run:Summary data: description: rados/thrash-old-clients/{0-distro$/{centos_9.stream} 0-size-min-size-overrides/3-size-2-min-size 1-install/quincy backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/fastclose rados thrashers/morepggrow thrashosds-health workloads/radosbench} duration: 6840.476484775543 failure_reason: '"2024-08-19T14:40:00.000097+0000 mon.a (mon.0) 3045 : cluster [WRN] Health detail: HEALTH_WARN Degraded data redundancy: 406/6 objects degraded (6766.667%), 2 pgs degraded, 3 pgs undersized" in cluster log' flavor: default owner: scheduled_teuthology@teuthology success: false 2024-08-19T15:54:51.980 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-08-19T15:54:52.072 INFO:teuthology.run:FAIL